2024-02-19T15:04:14.310 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-19T15:04:14.311 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566477 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/strays} --verbose --owner scheduled_leonidus@teuthology --name leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi -- /home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566477/orig.config.yaml 2024-02-19T15:04:14.354 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T15:04:14.434 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566477 branch: wip-lusov-qdb description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/strays} email: null first_in_suite: false job_id: '7566477' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-lusov-qdb ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 9f8908eed2b2d7e8c316bd5be7ed6e49024931ee ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 9f8908eed2b2d7e8c316bd5be7ed6e49024931ee kclient: syntax: v1 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-qdb sha1: 9f8908eed2b2d7e8c316bd5be7ed6e49024931ee owner: scheduled_leonidus@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 2280 sha1: 9f8908eed2b2d7e8c316bd5be7ed6e49024931ee sleep_before_teardown: 0 subset: 111/10000 suite: fs suite_branch: wip-lusov-qdb suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 9f8908eed2b2d7e8c316bd5be7ed6e49024931ee syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi047.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDSeBWtWbirpTGhOQFxv1p6nodVqAgn2Nj5S+d+rpDwo/Q4HD/uW47/wFuybaHwbNC/xXZG1wh20yDi7sJGOLKU= smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKf9i9SekZJBkyR5IzfNKL5GXCmWXK+l0/5forL4I1xhqyzutmv/bot1nC8/sXcsAogerwpcULVffpa+bD1QpRc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_strays teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-19_14:33:01 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-19T15:04:14.434 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/qa; will attempt to use it 2024-02-19T15:04:14.435 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/qa/tasks 2024-02-19T15:04:14.435 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-19T15:04:14.437 INFO:teuthology.task.internal:Checking packages... 2024-02-19T15:04:14.461 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee' 2024-02-19T15:04:14.461 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-19T15:04:14.461 INFO:teuthology.packaging:ref: None 2024-02-19T15:04:14.462 INFO:teuthology.packaging:tag: None 2024-02-19T15:04:14.462 INFO:teuthology.packaging:branch: wip-lusov-qdb 2024-02-19T15:04:14.462 INFO:teuthology.packaging:sha1: 9f8908eed2b2d7e8c316bd5be7ed6e49024931ee 2024-02-19T15:04:14.462 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-lusov-qdb 2024-02-19T15:04:14.607 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1451-g9f8908ee-1jammy 2024-02-19T15:04:14.608 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-19T15:04:14.615 INFO:teuthology.task.internal:no buildpackages task found 2024-02-19T15:04:14.616 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-19T15:04:14.659 INFO:teuthology.task.internal:Saving configuration 2024-02-19T15:04:14.675 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-19T15:04:14.680 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T15:04:14.705 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566477', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 14:58:06.208606', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHXgiT5SJR7KwFVFTI6ij5uh8LvSJbfk1TH91WYDvPuK'} 2024-02-19T15:04:14.726 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566477', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 14:58:06.210160', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPcIkh5M5sJsBXhcSYCFS0+wVBPxna9tJHViF0EEk5Zi'} 2024-02-19T15:04:14.726 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-19T15:04:14.735 INFO:teuthology.task.internal:roles: ubuntu@smithi047.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-02-19T15:04:14.735 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-02-19T15:04:14.735 INFO:teuthology.run_tasks:Running task console_log... 2024-02-19T15:04:14.805 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1496ebdf70>, signals=[15]) 2024-02-19T15:04:14.805 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-19T15:04:14.812 INFO:teuthology.task.internal:Opening connections... 2024-02-19T15:04:14.812 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi047.front.sepia.ceph.com 2024-02-19T15:04:14.814 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T15:04:14.896 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-02-19T15:04:14.897 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T15:04:14.977 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-19T15:04:14.983 DEBUG:teuthology.orchestra.run.smithi047:> uname -m 2024-02-19T15:04:14.989 INFO:teuthology.orchestra.run.smithi047.stdout:x86_64 2024-02-19T15:04:14.989 DEBUG:teuthology.orchestra.run.smithi047:> cat /etc/os-release 2024-02-19T15:04:15.035 INFO:teuthology.orchestra.run.smithi047.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-02-19T15:04:15.035 INFO:teuthology.orchestra.run.smithi047.stdout:NAME="Ubuntu" 2024-02-19T15:04:15.035 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_ID="22.04" 2024-02-19T15:04:15.035 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-02-19T15:04:15.035 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_CODENAME=jammy 2024-02-19T15:04:15.035 INFO:teuthology.orchestra.run.smithi047.stdout:ID=ubuntu 2024-02-19T15:04:15.035 INFO:teuthology.orchestra.run.smithi047.stdout:ID_LIKE=debian 2024-02-19T15:04:15.035 INFO:teuthology.orchestra.run.smithi047.stdout:HOME_URL="https://www.ubuntu.com/" 2024-02-19T15:04:15.035 INFO:teuthology.orchestra.run.smithi047.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-02-19T15:04:15.036 INFO:teuthology.orchestra.run.smithi047.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-02-19T15:04:15.036 INFO:teuthology.orchestra.run.smithi047.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-02-19T15:04:15.036 INFO:teuthology.orchestra.run.smithi047.stdout:UBUNTU_CODENAME=jammy 2024-02-19T15:04:15.036 INFO:teuthology.lock.ops:Updating smithi047.front.sepia.ceph.com on lock server 2024-02-19T15:04:15.059 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-02-19T15:04:15.064 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-02-19T15:04:15.064 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-02-19T15:04:15.112 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-02-19T15:04:15.112 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="Ubuntu" 2024-02-19T15:04:15.112 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="22.04" 2024-02-19T15:04:15.112 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-02-19T15:04:15.112 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_CODENAME=jammy 2024-02-19T15:04:15.113 INFO:teuthology.orchestra.run.smithi116.stdout:ID=ubuntu 2024-02-19T15:04:15.113 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE=debian 2024-02-19T15:04:15.113 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://www.ubuntu.com/" 2024-02-19T15:04:15.113 INFO:teuthology.orchestra.run.smithi116.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-02-19T15:04:15.113 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-02-19T15:04:15.113 INFO:teuthology.orchestra.run.smithi116.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-02-19T15:04:15.113 INFO:teuthology.orchestra.run.smithi116.stdout:UBUNTU_CODENAME=jammy 2024-02-19T15:04:15.113 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-02-19T15:04:15.134 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-19T15:04:15.143 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-19T15:04:15.149 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-19T15:04:15.149 DEBUG:teuthology.orchestra.run.smithi047:> test '!' -e /home/ubuntu/cephtest 2024-02-19T15:04:15.151 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-02-19T15:04:15.159 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-19T15:04:15.165 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-19T15:04:15.165 DEBUG:teuthology.orchestra.run.smithi047:> test -z $(ls -A /var/lib/ceph) 2024-02-19T15:04:15.196 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-02-19T15:04:15.213 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-19T15:04:15.272 INFO:teuthology.run_tasks:Running task kernel... 2024-02-19T15:04:15.286 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-02-19T15:04:15.286 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-02-19T15:04:15.286 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-02-19T15:04:15.287 DEBUG:teuthology.orchestra.run.smithi047:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T15:04:15.287 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T15:04:15.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:04:15.292 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing 2024-02-19T15:04:15.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:04:15.295 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing 2024-02-19T15:04:15.465 DEBUG:teuthology.task.kernel:sha1 for client.1 is 5d6d84d6d8d30cf809a50752f3e48f0e1d8f5996 2024-02-19T15:04:15.465 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "5d6d84d6d8d30cf809a50752f3e48f0e1d8f5996"... 2024-02-19T15:04:15.465 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-02-19T15:04:15.471 INFO:teuthology.orchestra.run.smithi116.stdout:5.15.0-56-generic 2024-02-19T15:04:15.471 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 5d6d84d6d8d30cf809a50752f3e48f0e1d8f5996 2024-02-19T15:04:15.472 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-02-19T15:04:15.472 INFO:teuthology.task.kernel:Installing linux-firmware on client.1... 2024-02-19T15:04:15.472 DEBUG:teuthology.orchestra.run.smithi116:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2024-02-19T15:04:15.528 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2024-02-19T15:04:15.671 INFO:teuthology.orchestra.run.smithi116.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2024-02-19T15:04:15.672 INFO:teuthology.orchestra.run.smithi116.stderr:hint: is subject to change. To configure the initial branch name to use in all 2024-02-19T15:04:15.672 INFO:teuthology.orchestra.run.smithi116.stderr:hint: of your new repositories, which will suppress this warning, call: 2024-02-19T15:04:15.672 INFO:teuthology.orchestra.run.smithi116.stderr:hint: 2024-02-19T15:04:15.672 INFO:teuthology.orchestra.run.smithi116.stderr:hint: git config --global init.defaultBranch 2024-02-19T15:04:15.672 INFO:teuthology.orchestra.run.smithi116.stderr:hint: 2024-02-19T15:04:15.672 INFO:teuthology.orchestra.run.smithi116.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-02-19T15:04:15.672 INFO:teuthology.orchestra.run.smithi116.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2024-02-19T15:04:15.672 INFO:teuthology.orchestra.run.smithi116.stderr:hint: 2024-02-19T15:04:15.672 INFO:teuthology.orchestra.run.smithi116.stderr:hint: git branch -m 2024-02-19T15:04:15.673 INFO:teuthology.orchestra.run.smithi116.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2024-02-19T15:04:15.674 DEBUG:teuthology.orchestra.run.smithi116:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2024-02-19T15:04:15.695 DEBUG:teuthology.task.kernel:sha1 for client.0 is 5d6d84d6d8d30cf809a50752f3e48f0e1d8f5996 2024-02-19T15:04:15.696 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5d6d84d6d8d30cf809a50752f3e48f0e1d8f5996"... 2024-02-19T15:04:15.696 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2024-02-19T15:04:15.701 INFO:teuthology.orchestra.run.smithi047.stdout:5.15.0-56-generic 2024-02-19T15:04:15.701 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 5d6d84d6d8d30cf809a50752f3e48f0e1d8f5996 2024-02-19T15:04:15.701 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-02-19T15:04:15.702 INFO:teuthology.task.kernel:Installing linux-firmware on client.0... 2024-02-19T15:04:15.702 DEBUG:teuthology.orchestra.run.smithi047:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2024-02-19T15:04:15.739 DEBUG:teuthology.orchestra.run.smithi116:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2024-02-19T15:04:15.756 INFO:teuthology.orchestra.run.smithi047.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2024-02-19T15:04:15.794 DEBUG:teuthology.orchestra.run.smithi116:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2024-02-19T15:04:15.888 INFO:teuthology.orchestra.run.smithi047.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2024-02-19T15:04:15.888 INFO:teuthology.orchestra.run.smithi047.stderr:hint: is subject to change. To configure the initial branch name to use in all 2024-02-19T15:04:15.888 INFO:teuthology.orchestra.run.smithi047.stderr:hint: of your new repositories, which will suppress this warning, call: 2024-02-19T15:04:15.888 INFO:teuthology.orchestra.run.smithi047.stderr:hint: 2024-02-19T15:04:15.888 INFO:teuthology.orchestra.run.smithi047.stderr:hint: git config --global init.defaultBranch 2024-02-19T15:04:15.888 INFO:teuthology.orchestra.run.smithi047.stderr:hint: 2024-02-19T15:04:15.888 INFO:teuthology.orchestra.run.smithi047.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-02-19T15:04:15.889 INFO:teuthology.orchestra.run.smithi047.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2024-02-19T15:04:15.889 INFO:teuthology.orchestra.run.smithi047.stderr:hint: 2024-02-19T15:04:15.889 INFO:teuthology.orchestra.run.smithi047.stderr:hint: git branch -m 2024-02-19T15:04:15.889 INFO:teuthology.orchestra.run.smithi047.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2024-02-19T15:04:15.891 DEBUG:teuthology.orchestra.run.smithi047:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2024-02-19T15:04:15.958 DEBUG:teuthology.orchestra.run.smithi047:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2024-02-19T15:04:16.015 DEBUG:teuthology.orchestra.run.smithi047:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2024-02-19T15:07:55.107 INFO:teuthology.orchestra.run.smithi047.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2024-02-19T15:07:55.107 INFO:teuthology.orchestra.run.smithi047.stderr: * [new branch] main -> origin/main 2024-02-19T15:07:55.141 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20190312 -> 20190312 2024-02-19T15:07:55.142 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20190416 -> 20190416 2024-02-19T15:07:55.142 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20190514 -> 20190514 2024-02-19T15:07:55.142 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20190618 -> 20190618 2024-02-19T15:07:55.142 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20190717 -> 20190717 2024-02-19T15:07:55.142 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20190815 -> 20190815 2024-02-19T15:07:55.143 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20190923 -> 20190923 2024-02-19T15:07:55.143 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20191022 -> 20191022 2024-02-19T15:07:55.143 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20191215 -> 20191215 2024-02-19T15:07:55.143 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20200122 -> 20200122 2024-02-19T15:07:55.143 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20200316 -> 20200316 2024-02-19T15:07:55.143 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20200421 -> 20200421 2024-02-19T15:07:55.143 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20200519 -> 20200519 2024-02-19T15:07:55.143 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20200619 -> 20200619 2024-02-19T15:07:55.144 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20200721 -> 20200721 2024-02-19T15:07:55.144 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20200817 -> 20200817 2024-02-19T15:07:55.144 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20200918 -> 20200918 2024-02-19T15:07:55.145 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20201022 -> 20201022 2024-02-19T15:07:55.145 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20201118 -> 20201118 2024-02-19T15:07:55.145 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20201218 -> 20201218 2024-02-19T15:07:55.145 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20210208 -> 20210208 2024-02-19T15:07:55.145 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20210315 -> 20210315 2024-02-19T15:07:55.145 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20210511 -> 20210511 2024-02-19T15:07:55.145 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20210716 -> 20210716 2024-02-19T15:07:55.145 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20210818 -> 20210818 2024-02-19T15:07:55.146 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20210919 -> 20210919 2024-02-19T15:07:55.146 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20211027 -> 20211027 2024-02-19T15:07:55.146 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20211216 -> 20211216 2024-02-19T15:07:55.146 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20220209 -> 20220209 2024-02-19T15:07:55.147 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20220310 -> 20220310 2024-02-19T15:07:55.147 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20220411 -> 20220411 2024-02-19T15:07:55.147 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20220509 -> 20220509 2024-02-19T15:07:55.147 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20220610 -> 20220610 2024-02-19T15:07:55.147 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20220708 -> 20220708 2024-02-19T15:07:55.147 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20220815 -> 20220815 2024-02-19T15:07:55.148 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20220913 -> 20220913 2024-02-19T15:07:55.148 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20221012 -> 20221012 2024-02-19T15:07:55.148 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20221109 -> 20221109 2024-02-19T15:07:55.148 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20221214 -> 20221214 2024-02-19T15:07:55.148 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20230117 -> 20230117 2024-02-19T15:07:55.148 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20230210 -> 20230210 2024-02-19T15:07:55.148 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20230310 -> 20230310 2024-02-19T15:07:55.148 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20230404 -> 20230404 2024-02-19T15:07:55.149 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20230515 -> 20230515 2024-02-19T15:07:55.149 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20230625 -> 20230625 2024-02-19T15:07:55.149 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20230804 -> 20230804 2024-02-19T15:07:55.149 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20230919 -> 20230919 2024-02-19T15:07:55.149 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20231030 -> 20231030 2024-02-19T15:07:55.149 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20231111 -> 20231111 2024-02-19T15:07:55.149 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20231211 -> 20231211 2024-02-19T15:07:55.149 INFO:teuthology.orchestra.run.smithi047.stderr: * [new tag] 20240115 -> 20240115 2024-02-19T15:07:59.468 INFO:teuthology.orchestra.run.smithi047.stderr:Updating files: 49% (1767/3556) Updating files: 50% (1778/3556) Updating files: 51% (1814/3556) Updating files: 52% (1850/3556) Updating files: 53% (1885/3556) Updating files: 54% (1921/3556) Updating files: 55% (1956/3556) Updating files: 56% (1992/3556) Updating files: 57% (2027/3556) Updating files: 58% (2063/3556) Updating files: 59% (2099/3556) Updating files: 60% (2134/3556) Updating files: 61% (2170/3556) Updating files: 62% (2205/3556) Updating files: 63% (2241/3556) Updating files: 64% (2276/3556) Updating files: 64% (2286/3556) Updating files: 65% (2312/3556) Updating files: 66% (2347/3556) Updating files: 67% (2383/3556) Updating files: 68% (2419/3556) Updating files: 69% (2454/3556) Updating files: 70% (2490/3556) Updating files: 71% (2525/3556) Updating files: 72% (2561/3556) Updating files: 73% (2596/3556) Updating files: 74% (2632/3556) Updating files: 75% (2667/3556) Updating files: 76% (2703/3556) Updating files: 77% (2739/3556) Updating files: 77% (2743/3556) Updating files: 78% (2774/3556) Updating files: 79% (2810/3556) Updating files: 80% (2845/3556) Updating files: 81% (2881/3556) Updating files: 82% (2916/3556) Updating files: 83% (2952/3556) Updating files: 83% (2984/3556) Updating files: 84% (2988/3556) Updating files: 85% (3023/3556) Updating files: 86% (3059/3556) Updating files: 87% (3094/3556) Updating files: 88% (3130/3556) Updating files: 89% (3165/3556) Updating files: 90% (3201/3556) Updating files: 91% (3236/3556) Updating files: 92% (3272/3556) Updating files: 93% (3308/3556) Updating files: 94% (3343/3556) Updating files: 95% (3379/3556) Updating files: 96% (3414/3556) Updating files: 97% (3450/3556) Updating files: 98% (3485/3556) Updating files: 99% (3521/3556) Updating files: 100% (3556/3556) Updating files: 100% (3556/3556), done. 2024-02-19T15:07:59.473 INFO:teuthology.orchestra.run.smithi047.stdout:HEAD is now at cebcefff Merge branch 'robot/pr-0-1708118635' into 'main' 2024-02-19T15:07:59.495 INFO:teuthology.task.kernel:Downloading kernel 5d6d84d6d8d30cf809a50752f3e48f0e1d8f5996 on client.0... 2024-02-19T15:07:59.495 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5d6d84d6d8d30cf809a50752f3e48f0e1d8f5996 2024-02-19T15:07:59.643 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/5d6d84d6d8d30cf809a50752f3e48f0e1d8f5996/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-rc4-g5d6d84d6d8d3/ 2024-02-19T15:07:59.643 DEBUG:teuthology.orchestra.run.smithi047:> rm -f /tmp/linux-image.deb && echo linux-image-6.8.0-rc4-g5d6d84d6d8d3_6.8.0-rc4-g5d6d84d6d8d3-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testing/5d6d84d6d8d30cf809a50752f3e48f0e1d8f5996/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-rc4-g5d6d84d6d8d3/ --input-file=- 2024-02-19T15:07:59.646 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi047.front.sepia.ceph.com to complete... 2024-02-19T15:08:01.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19 15:08:01 URL:https://4.chacra.ceph.com/r/kernel/testing/5d6d84d6d8d30cf809a50752f3e48f0e1d8f5996/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-rc4-g5d6d84d6d8d3/linux-image-6.8.0-rc4-g5d6d84d6d8d3_6.8.0-rc4-g5d6d84d6d8d3-1_amd64.deb [39246686/39246686] -> "/tmp/linux-image.deb" [1] 2024-02-19T15:08:01.170 INFO:teuthology.orchestra.run.smithi047.stderr:FINISHED --2024-02-19 15:08:01-- 2024-02-19T15:08:01.170 INFO:teuthology.orchestra.run.smithi047.stderr:Total wall clock time: 1.5s 2024-02-19T15:08:01.170 INFO:teuthology.orchestra.run.smithi047.stderr:Downloaded: 1 files, 37M in 1.4s (27.4 MB/s) 2024-02-19T15:08:01.175 INFO:teuthology.task.kernel:Installing kernel 5d6d84d6d8d30cf809a50752f3e48f0e1d8f5996 on client.0... 2024-02-19T15:08:01.175 DEBUG:teuthology.orchestra.run.smithi047:> sudo dpkg -i /tmp/linux-image.deb 2024-02-19T15:08:01.465 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package linux-image-6.8.0-rc4-g5d6d84d6d8d3. 2024-02-19T15:08:03.432 INFO:teuthology.orchestra.run.smithi047.stdout:(Reading database ... 129542 files and directories currently installed.) 2024-02-19T15:08:03.437 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack /tmp/linux-image.deb ... 2024-02-19T15:08:03.580 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-image-6.8.0-rc4-g5d6d84d6d8d3 (6.8.0-rc4-g5d6d84d6d8d3-1) ... 2024-02-19T15:08:07.349 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-image-6.8.0-rc4-g5d6d84d6d8d3 (6.8.0-rc4-g5d6d84d6d8d3-1) ... 2024-02-19T15:08:07.442 INFO:teuthology.orchestra.run.smithi047.stderr:update-initramfs: Generating /boot/initrd.img-6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:18.411 INFO:teuthology.orchestra.run.smithi116.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2024-02-19T15:08:18.411 INFO:teuthology.orchestra.run.smithi116.stderr: * [new branch] main -> origin/main 2024-02-19T15:08:18.443 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20190312 -> 20190312 2024-02-19T15:08:18.443 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20190416 -> 20190416 2024-02-19T15:08:18.443 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20190514 -> 20190514 2024-02-19T15:08:18.443 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20190618 -> 20190618 2024-02-19T15:08:18.443 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20190717 -> 20190717 2024-02-19T15:08:18.443 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20190815 -> 20190815 2024-02-19T15:08:18.443 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20190923 -> 20190923 2024-02-19T15:08:18.443 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20191022 -> 20191022 2024-02-19T15:08:18.443 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20191215 -> 20191215 2024-02-19T15:08:18.443 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20200122 -> 20200122 2024-02-19T15:08:18.446 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20200316 -> 20200316 2024-02-19T15:08:18.446 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20200421 -> 20200421 2024-02-19T15:08:18.446 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20200519 -> 20200519 2024-02-19T15:08:18.446 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20200619 -> 20200619 2024-02-19T15:08:18.446 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20200721 -> 20200721 2024-02-19T15:08:18.446 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20200817 -> 20200817 2024-02-19T15:08:18.446 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20200918 -> 20200918 2024-02-19T15:08:18.446 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20201022 -> 20201022 2024-02-19T15:08:18.447 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20201118 -> 20201118 2024-02-19T15:08:18.447 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20201218 -> 20201218 2024-02-19T15:08:18.447 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20210208 -> 20210208 2024-02-19T15:08:18.447 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20210315 -> 20210315 2024-02-19T15:08:18.447 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20210511 -> 20210511 2024-02-19T15:08:18.447 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20210716 -> 20210716 2024-02-19T15:08:18.447 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20210818 -> 20210818 2024-02-19T15:08:18.447 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20210919 -> 20210919 2024-02-19T15:08:18.447 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20211027 -> 20211027 2024-02-19T15:08:18.447 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20211216 -> 20211216 2024-02-19T15:08:18.447 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20220209 -> 20220209 2024-02-19T15:08:18.447 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20220310 -> 20220310 2024-02-19T15:08:18.447 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20220411 -> 20220411 2024-02-19T15:08:18.447 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20220509 -> 20220509 2024-02-19T15:08:18.447 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20220610 -> 20220610 2024-02-19T15:08:18.448 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20220708 -> 20220708 2024-02-19T15:08:18.449 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20220815 -> 20220815 2024-02-19T15:08:18.449 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20220913 -> 20220913 2024-02-19T15:08:18.450 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20221012 -> 20221012 2024-02-19T15:08:18.450 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20221109 -> 20221109 2024-02-19T15:08:18.450 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20221214 -> 20221214 2024-02-19T15:08:18.450 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20230117 -> 20230117 2024-02-19T15:08:18.450 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20230210 -> 20230210 2024-02-19T15:08:18.450 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20230310 -> 20230310 2024-02-19T15:08:18.450 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20230404 -> 20230404 2024-02-19T15:08:18.450 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20230515 -> 20230515 2024-02-19T15:08:18.450 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20230625 -> 20230625 2024-02-19T15:08:18.450 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20230804 -> 20230804 2024-02-19T15:08:18.450 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20230919 -> 20230919 2024-02-19T15:08:18.450 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20231030 -> 20231030 2024-02-19T15:08:18.450 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20231111 -> 20231111 2024-02-19T15:08:18.450 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20231211 -> 20231211 2024-02-19T15:08:18.450 INFO:teuthology.orchestra.run.smithi116.stderr: * [new tag] 20240115 -> 20240115 2024-02-19T15:08:22.545 INFO:teuthology.orchestra.run.smithi116.stderr:Updating files: 51% (1829/3556) Updating files: 52% (1850/3556) Updating files: 53% (1885/3556) Updating files: 54% (1921/3556) Updating files: 55% (1956/3556) Updating files: 56% (1992/3556) Updating files: 57% (2027/3556) Updating files: 58% (2063/3556) Updating files: 59% (2099/3556) Updating files: 60% (2134/3556) Updating files: 61% (2170/3556) Updating files: 62% (2205/3556) Updating files: 63% (2241/3556) Updating files: 64% (2276/3556) Updating files: 65% (2312/3556) Updating files: 65% (2341/3556) Updating files: 66% (2347/3556) Updating files: 67% (2383/3556) Updating files: 68% (2419/3556) Updating files: 69% (2454/3556) Updating files: 70% (2490/3556) Updating files: 71% (2525/3556) Updating files: 72% (2561/3556) Updating files: 73% (2596/3556) Updating files: 74% (2632/3556) Updating files: 75% (2667/3556) Updating files: 76% (2703/3556) Updating files: 77% (2739/3556) Updating files: 78% (2774/3556) Updating files: 79% (2810/3556) Updating files: 80% (2845/3556) Updating files: 81% (2881/3556) Updating files: 81% (2909/3556) Updating files: 82% (2916/3556) Updating files: 83% (2952/3556) Updating files: 84% (2988/3556) Updating files: 85% (3023/3556) Updating files: 86% (3059/3556) Updating files: 87% (3094/3556) Updating files: 88% (3130/3556) Updating files: 89% (3165/3556) Updating files: 90% (3201/3556) Updating files: 91% (3236/3556) Updating files: 92% (3272/3556) Updating files: 93% (3308/3556) Updating files: 94% (3343/3556) Updating files: 95% (3379/3556) Updating files: 95% (3403/3556) Updating files: 96% (3414/3556) Updating files: 97% (3450/3556) Updating files: 98% (3485/3556) Updating files: 99% (3521/3556) Updating files: 100% (3556/3556) Updating files: 100% (3556/3556), done. 2024-02-19T15:08:22.550 INFO:teuthology.orchestra.run.smithi116.stdout:HEAD is now at cebcefff Merge branch 'robot/pr-0-1708118635' into 'main' 2024-02-19T15:08:22.571 INFO:teuthology.task.kernel:Downloading kernel 5d6d84d6d8d30cf809a50752f3e48f0e1d8f5996 on client.1... 2024-02-19T15:08:22.572 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5d6d84d6d8d30cf809a50752f3e48f0e1d8f5996 2024-02-19T15:08:22.740 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/5d6d84d6d8d30cf809a50752f3e48f0e1d8f5996/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-rc4-g5d6d84d6d8d3/ 2024-02-19T15:08:22.740 DEBUG:teuthology.orchestra.run.smithi116:> rm -f /tmp/linux-image.deb && echo linux-image-6.8.0-rc4-g5d6d84d6d8d3_6.8.0-rc4-g5d6d84d6d8d3-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testing/5d6d84d6d8d30cf809a50752f3e48f0e1d8f5996/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-rc4-g5d6d84d6d8d3/ --input-file=- 2024-02-19T15:08:22.742 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi116.front.sepia.ceph.com to complete... 2024-02-19T15:08:24.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19 15:08:24 URL:https://4.chacra.ceph.com/r/kernel/testing/5d6d84d6d8d30cf809a50752f3e48f0e1d8f5996/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-rc4-g5d6d84d6d8d3/linux-image-6.8.0-rc4-g5d6d84d6d8d3_6.8.0-rc4-g5d6d84d6d8d3-1_amd64.deb [39246686/39246686] -> "/tmp/linux-image.deb" [1] 2024-02-19T15:08:24.270 INFO:teuthology.orchestra.run.smithi116.stderr:FINISHED --2024-02-19 15:08:24-- 2024-02-19T15:08:24.270 INFO:teuthology.orchestra.run.smithi116.stderr:Total wall clock time: 1.5s 2024-02-19T15:08:24.270 INFO:teuthology.orchestra.run.smithi116.stderr:Downloaded: 1 files, 37M in 1.4s (27.1 MB/s) 2024-02-19T15:08:24.274 INFO:teuthology.task.kernel:Installing kernel 5d6d84d6d8d30cf809a50752f3e48f0e1d8f5996 on client.1... 2024-02-19T15:08:24.275 DEBUG:teuthology.orchestra.run.smithi116:> sudo dpkg -i /tmp/linux-image.deb 2024-02-19T15:08:24.488 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-image-6.8.0-rc4-g5d6d84d6d8d3. 2024-02-19T15:08:26.485 INFO:teuthology.orchestra.run.smithi116.stdout:(Reading database ... 129542 files and directories currently installed.) 2024-02-19T15:08:26.489 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack /tmp/linux-image.deb ... 2024-02-19T15:08:26.643 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-image-6.8.0-rc4-g5d6d84d6d8d3 (6.8.0-rc4-g5d6d84d6d8d3-1) ... 2024-02-19T15:08:29.484 INFO:teuthology.orchestra.run.smithi047.stderr:Sourcing file `/etc/default/grub' 2024-02-19T15:08:29.491 INFO:teuthology.orchestra.run.smithi047.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-02-19T15:08:29.516 INFO:teuthology.orchestra.run.smithi047.stderr:Generating grub configuration file ... 2024-02-19T15:08:29.849 INFO:teuthology.orchestra.run.smithi047.stderr:Found linux image: /boot/vmlinuz-6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:29.862 INFO:teuthology.orchestra.run.smithi047.stderr:Found initrd image: /boot/initrd.img-6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:29.962 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-image-6.8.0-rc4-g5d6d84d6d8d3 (6.8.0-rc4-g5d6d84d6d8d3-1) ... 2024-02-19T15:08:30.065 INFO:teuthology.orchestra.run.smithi116.stderr:update-initramfs: Generating /boot/initrd.img-6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:30.153 INFO:teuthology.orchestra.run.smithi047.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-02-19T15:08:30.165 INFO:teuthology.orchestra.run.smithi047.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-02-19T15:08:30.229 INFO:teuthology.orchestra.run.smithi047.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-02-19T15:08:30.234 INFO:teuthology.orchestra.run.smithi047.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-02-19T15:08:30.401 INFO:teuthology.orchestra.run.smithi047.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-02-19T15:08:30.401 INFO:teuthology.orchestra.run.smithi047.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-02-19T15:08:30.401 INFO:teuthology.orchestra.run.smithi047.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-02-19T15:08:30.445 INFO:teuthology.orchestra.run.smithi047.stderr:done 2024-02-19T15:08:30.659 DEBUG:teuthology.orchestra.run.smithi047:> dpkg-deb -c /tmp/linux-image.deb 2024-02-19T15:08:30.668 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./ 2024-02-19T15:08:30.668 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./boot/ 2024-02-19T15:08:30.668 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 4469316 2024-02-19 07:58 ./boot/System.map-6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:30.681 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 170841 2024-02-19 07:58 ./boot/config-6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:30.681 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11527168 2024-02-19 07:58 ./boot/vmlinuz-6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:30.699 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./etc/ 2024-02-19T15:08:30.699 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./etc/kernel/ 2024-02-19T15:08:30.699 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./etc/kernel/postinst.d/ 2024-02-19T15:08:30.699 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./etc/kernel/postrm.d/ 2024-02-19T15:08:30.699 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./etc/kernel/preinst.d/ 2024-02-19T15:08:30.699 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./etc/kernel/prerm.d/ 2024-02-19T15:08:30.699 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/ 2024-02-19T15:08:30.699 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/ 2024-02-19T15:08:30.699 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/ 2024-02-19T15:08:30.700 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/ 2024-02-19T15:08:30.700 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/ 2024-02-19T15:08:30.700 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/ 2024-02-19T15:08:30.700 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/ 2024-02-19T15:08:30.700 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 490449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/aesni-intel.ko 2024-02-19T15:08:30.700 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-02-19T15:08:30.700 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-02-19T15:08:30.700 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-02-19T15:08:30.700 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 93081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-02-19T15:08:30.703 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 106697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-02-19T15:08:30.703 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 152145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-02-19T15:08:30.703 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-02-19T15:08:30.703 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-02-19T15:08:30.703 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 83857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-02-19T15:08:30.703 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-02-19T15:08:30.703 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 47185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-02-19T15:08:30.703 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 47177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/serpent-avx2.ko 2024-02-19T15:08:30.703 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 51697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-02-19T15:08:30.703 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 39601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-02-19T15:08:30.703 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 42537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-02-19T15:08:30.703 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 58489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-02-19T15:08:30.704 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 50393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-02-19T15:08:30.704 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-02-19T15:08:30.704 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-02-19T15:08:30.704 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/kernel/ 2024-02-19T15:08:30.704 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/kernel/cpu/ 2024-02-19T15:08:30.704 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/kernel/cpu/mce/ 2024-02-19T15:08:30.704 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 41433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-02-19T15:08:30.704 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13593 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/kernel/cpuid.ko 2024-02-19T15:08:30.704 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/kernel/msr.ko 2024-02-19T15:08:30.704 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/kvm/ 2024-02-19T15:08:30.704 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 354001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/kvm/kvm-amd.ko 2024-02-19T15:08:30.704 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 885905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/kvm/kvm-intel.ko 2024-02-19T15:08:30.704 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 2294545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/kvm/kvm.ko 2024-02-19T15:08:30.716 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/ 2024-02-19T15:08:30.716 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 53841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/af_alg.ko 2024-02-19T15:08:30.716 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/algif_hash.ko 2024-02-19T15:08:30.716 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/algif_skcipher.ko 2024-02-19T15:08:30.716 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17113 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/ansi_cprng.ko 2024-02-19T15:08:30.716 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19553 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/anubis.ko 2024-02-19T15:08:30.716 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9585 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/arc4.ko 2024-02-19T15:08:30.716 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/asymmetric_keys/ 2024-02-19T15:08:30.716 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2024-02-19T15:08:30.716 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/async_tx/ 2024-02-19T15:08:30.716 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/async_tx/async_memcpy.ko 2024-02-19T15:08:30.717 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/async_tx/async_pq.ko 2024-02-19T15:08:30.717 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/async_tx/async_raid6_recov.ko 2024-02-19T15:08:30.717 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/async_tx/async_tx.ko 2024-02-19T15:08:30.717 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/async_tx/async_xor.ko 2024-02-19T15:08:30.717 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/async_tx/raid6test.ko 2024-02-19T15:08:30.717 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/authenc.ko 2024-02-19T15:08:30.717 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/authencesn.ko 2024-02-19T15:08:30.717 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/blake2b_generic.ko 2024-02-19T15:08:30.717 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/blowfish_common.ko 2024-02-19T15:08:30.717 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/blowfish_generic.ko 2024-02-19T15:08:30.717 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 38777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/camellia_generic.ko 2024-02-19T15:08:30.717 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 27233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/cast5_generic.ko 2024-02-19T15:08:30.717 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/cast6_generic.ko 2024-02-19T15:08:30.717 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/cast_common.ko 2024-02-19T15:08:30.718 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 35185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/ccm.ko 2024-02-19T15:08:30.718 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/cmac.ko 2024-02-19T15:08:30.718 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/crc32_generic.ko 2024-02-19T15:08:30.718 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 47681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/cryptd.ko 2024-02-19T15:08:30.718 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 39473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/crypto_engine.ko 2024-02-19T15:08:30.718 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26921 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/crypto_simd.ko 2024-02-19T15:08:30.718 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/crypto_user.ko 2024-02-19T15:08:30.718 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/cts.ko 2024-02-19T15:08:30.718 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/des_generic.ko 2024-02-19T15:08:30.718 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15593 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/dh_generic.ko 2024-02-19T15:08:30.718 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10913 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/echainiv.ko 2024-02-19T15:08:30.718 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/essiv.ko 2024-02-19T15:08:30.718 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/fcrypt.ko 2024-02-19T15:08:30.718 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/khazad.ko 2024-02-19T15:08:30.719 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/lrw.ko 2024-02-19T15:08:30.719 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/lz4.ko 2024-02-19T15:08:30.719 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/lz4hc.ko 2024-02-19T15:08:30.719 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/md4.ko 2024-02-19T15:08:30.719 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/michael_mic.ko 2024-02-19T15:08:30.719 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10585 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/pcbc.ko 2024-02-19T15:08:30.719 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/pcrypt.ko 2024-02-19T15:08:30.719 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/rmd160.ko 2024-02-19T15:08:30.719 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/seed.ko 2024-02-19T15:08:30.719 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/serpent_generic.ko 2024-02-19T15:08:30.719 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 143033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/tcrypt.ko 2024-02-19T15:08:30.719 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/tea.ko 2024-02-19T15:08:30.719 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/twofish_common.ko 2024-02-19T15:08:30.719 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/twofish_generic.ko 2024-02-19T15:08:30.719 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/vmac.ko 2024-02-19T15:08:30.720 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/wp512.ko 2024-02-19T15:08:30.720 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/xcbc.ko 2024-02-19T15:08:30.720 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/xor.ko 2024-02-19T15:08:30.720 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/xts.ko 2024-02-19T15:08:30.720 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/xxhash_generic.ko 2024-02-19T15:08:30.720 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ 2024-02-19T15:08:30.720 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/ 2024-02-19T15:08:30.720 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/acpi_extlog.ko 2024-02-19T15:08:30.720 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/acpi_ipmi.ko 2024-02-19T15:08:30.720 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/acpi_pad.ko 2024-02-19T15:08:30.720 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/apei/ 2024-02-19T15:08:30.720 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/apei/einj.ko 2024-02-19T15:08:30.720 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/ec_sys.ko 2024-02-19T15:08:30.721 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/platform_profile.ko 2024-02-19T15:08:30.721 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/sbs.ko 2024-02-19T15:08:30.721 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/sbshc.ko 2024-02-19T15:08:30.721 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 104433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/video.ko 2024-02-19T15:08:30.721 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/ 2024-02-19T15:08:30.721 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/acard-ahci.ko 2024-02-19T15:08:30.721 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 89825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/ahci.ko 2024-02-19T15:08:30.721 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/ahci_platform.ko 2024-02-19T15:08:30.721 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 88313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/libahci.ko 2024-02-19T15:08:30.721 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/libahci_platform.ko 2024-02-19T15:08:30.721 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_acpi.ko 2024-02-19T15:08:30.721 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_ali.ko 2024-02-19T15:08:30.722 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 33481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_amd.ko 2024-02-19T15:08:30.722 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_artop.ko 2024-02-19T15:08:30.722 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_atiixp.ko 2024-02-19T15:08:30.722 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_atp867x.ko 2024-02-19T15:08:30.722 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_cmd640.ko 2024-02-19T15:08:30.722 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_cmd64x.ko 2024-02-19T15:08:30.722 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_cypress.ko 2024-02-19T15:08:30.722 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17753 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_efar.ko 2024-02-19T15:08:30.722 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_hpt366.ko 2024-02-19T15:08:30.722 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 33881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_hpt37x.ko 2024-02-19T15:08:30.722 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_hpt3x2n.ko 2024-02-19T15:08:30.722 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_hpt3x3.ko 2024-02-19T15:08:30.722 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_it8213.ko 2024-02-19T15:08:30.723 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 35177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_it821x.ko 2024-02-19T15:08:30.723 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_jmicron.ko 2024-02-19T15:08:30.723 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 52177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_legacy.ko 2024-02-19T15:08:30.723 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_marvell.ko 2024-02-19T15:08:30.723 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_mpiix.ko 2024-02-19T15:08:30.723 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_netcell.ko 2024-02-19T15:08:30.723 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_ninja32.ko 2024-02-19T15:08:30.723 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_ns87410.ko 2024-02-19T15:08:30.723 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_ns87415.ko 2024-02-19T15:08:30.723 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_oldpiix.ko 2024-02-19T15:08:30.723 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_opti.ko 2024-02-19T15:08:30.723 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20585 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_optidma.ko 2024-02-19T15:08:30.723 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 32489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_pcmcia.ko 2024-02-19T15:08:30.723 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 37697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_pdc2027x.ko 2024-02-19T15:08:30.723 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-02-19T15:08:30.724 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14129 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_piccolo.ko 2024-02-19T15:08:30.724 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_radisys.ko 2024-02-19T15:08:30.724 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_rdc.ko 2024-02-19T15:08:30.724 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_rz1000.ko 2024-02-19T15:08:30.724 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_sch.ko 2024-02-19T15:08:30.724 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_serverworks.ko 2024-02-19T15:08:30.724 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_sil680.ko 2024-02-19T15:08:30.724 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_sl82c105.ko 2024-02-19T15:08:30.724 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15913 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_triflex.ko 2024-02-19T15:08:30.724 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_via.ko 2024-02-19T15:08:30.724 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pdc_adma.ko 2024-02-19T15:08:30.724 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_inic162x.ko 2024-02-19T15:08:30.724 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 83257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_mv.ko 2024-02-19T15:08:30.725 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 70305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_nv.ko 2024-02-19T15:08:30.725 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 37281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_promise.ko 2024-02-19T15:08:30.725 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_qstor.ko 2024-02-19T15:08:30.725 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_sil.ko 2024-02-19T15:08:30.725 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 38177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_sil24.ko 2024-02-19T15:08:30.725 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_sis.ko 2024-02-19T15:08:30.725 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_svw.ko 2024-02-19T15:08:30.725 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 44705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_sx4.ko 2024-02-19T15:08:30.725 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_uli.ko 2024-02-19T15:08:30.725 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_via.ko 2024-02-19T15:08:30.725 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_vsc.ko 2024-02-19T15:08:30.725 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/auxdisplay/ 2024-02-19T15:08:30.725 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/auxdisplay/ks0108.ko 2024-02-19T15:08:30.725 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/bcma/ 2024-02-19T15:08:30.725 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 124457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/bcma/bcma.ko 2024-02-19T15:08:30.726 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/ 2024-02-19T15:08:30.726 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/aoe/ 2024-02-19T15:08:30.726 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 113833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/aoe/aoe.ko 2024-02-19T15:08:30.726 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/drbd/ 2024-02-19T15:08:30.726 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 964817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/drbd/drbd.ko 2024-02-19T15:08:30.726 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 169705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/floppy.ko 2024-02-19T15:08:30.726 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/mtip32xx/ 2024-02-19T15:08:30.726 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 94217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-02-19T15:08:30.726 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 116401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/nbd.ko 2024-02-19T15:08:30.726 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/null_blk/ 2024-02-19T15:08:30.726 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 110329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/null_blk/null_blk.ko 2024-02-19T15:08:30.726 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 98193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/pktcdvd.ko 2024-02-19T15:08:30.726 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 263641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/rbd.ko 2024-02-19T15:08:30.726 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/xen-blkback/ 2024-02-19T15:08:30.726 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 98121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-02-19T15:08:30.727 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/zram/ 2024-02-19T15:08:30.727 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 58777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/zram/zram.ko 2024-02-19T15:08:30.727 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/ 2024-02-19T15:08:30.727 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/applicom.ko 2024-02-19T15:08:30.727 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11913 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/hangcheck-timer.ko 2024-02-19T15:08:30.727 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/hw_random/ 2024-02-19T15:08:30.727 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/hw_random/amd-rng.ko 2024-02-19T15:08:30.727 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/hw_random/intel-rng.ko 2024-02-19T15:08:30.727 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-02-19T15:08:30.727 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/hw_random/via-rng.ko 2024-02-19T15:08:30.727 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/hw_random/virtio-rng.ko 2024-02-19T15:08:30.727 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/ipmi/ 2024-02-19T15:08:30.727 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-02-19T15:08:30.727 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 148521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-02-19T15:08:30.781 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-02-19T15:08:30.781 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 139361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/ipmi/ipmi_si.ko 2024-02-19T15:08:30.781 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 70873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-02-19T15:08:30.781 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 53417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-02-19T15:08:30.781 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/lp.ko 2024-02-19T15:08:30.781 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/mwave/ 2024-02-19T15:08:30.781 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 84025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/mwave/mwave.ko 2024-02-19T15:08:30.781 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/nvram.ko 2024-02-19T15:08:30.782 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 32081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/ppdev.ko 2024-02-19T15:08:30.782 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 51025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/tlclk.ko 2024-02-19T15:08:30.782 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/tpm/ 2024-02-19T15:08:30.782 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/tpm/tpm_atmel.ko 2024-02-19T15:08:30.782 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2024-02-19T15:08:30.782 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2024-02-19T15:08:30.782 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 32769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2024-02-19T15:08:30.782 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/tpm/tpm_infineon.ko 2024-02-19T15:08:30.782 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22609 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/tpm/tpm_nsc.ko 2024-02-19T15:08:30.782 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23465 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/tpm/xen-tpmfront.ko 2024-02-19T15:08:30.782 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/xillybus/ 2024-02-19T15:08:30.782 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/xillybus/xillybus_class.ko 2024-02-19T15:08:30.782 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 50849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/xillybus/xillybus_core.ko 2024-02-19T15:08:30.783 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/xillybus/xillybus_pcie.ko 2024-02-19T15:08:30.783 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/clk/ 2024-02-19T15:08:30.783 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/clk/clk-palmas.ko 2024-02-19T15:08:30.783 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 47729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/clk/clk-si5351.ko 2024-02-19T15:08:30.783 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/clk/clk-twl6040.ko 2024-02-19T15:08:30.783 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/clk/clk-wm831x.ko 2024-02-19T15:08:30.783 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/cpufreq/ 2024-02-19T15:08:30.783 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-02-19T15:08:30.783 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/cpufreq/p4-clockmod.ko 2024-02-19T15:08:30.783 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/cpufreq/speedstep-lib.ko 2024-02-19T15:08:30.783 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/ 2024-02-19T15:08:30.783 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/ccp/ 2024-02-19T15:08:30.783 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 70473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-02-19T15:08:30.784 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 246841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/ccp/ccp.ko 2024-02-19T15:08:30.784 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/intel/ 2024-02-19T15:08:30.784 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/intel/qat/ 2024-02-19T15:08:30.784 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/intel/qat/qat_common/ 2024-02-19T15:08:30.784 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 570497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-02-19T15:08:30.784 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2024-02-19T15:08:30.784 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-02-19T15:08:30.784 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/padlock-aes.ko 2024-02-19T15:08:30.784 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/padlock-sha.ko 2024-02-19T15:08:30.784 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/virtio/ 2024-02-19T15:08:30.784 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 70433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-02-19T15:08:30.784 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/dca/ 2024-02-19T15:08:30.784 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/dca/dca.ko 2024-02-19T15:08:30.784 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/dma/ 2024-02-19T15:08:30.784 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/dma/dw/ 2024-02-19T15:08:30.785 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/dma/dw/dw_dmac.ko 2024-02-19T15:08:30.785 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/dma/ioat/ 2024-02-19T15:08:30.785 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 122817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/dma/ioat/ioatdma.ko 2024-02-19T15:08:30.785 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/ 2024-02-19T15:08:30.785 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 80305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/amd64_edac.ko 2024-02-19T15:08:30.785 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 35673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/e752x_edac.ko 2024-02-19T15:08:30.785 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 41641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/edac_mce_amd.ko 2024-02-19T15:08:30.785 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/i3000_edac.ko 2024-02-19T15:08:30.785 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/i3200_edac.ko 2024-02-19T15:08:30.785 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/i5100_edac.ko 2024-02-19T15:08:30.785 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/i5400_edac.ko 2024-02-19T15:08:30.785 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/i7300_edac.ko 2024-02-19T15:08:30.785 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 51945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/i7core_edac.ko 2024-02-19T15:08:30.785 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/i82975x_edac.ko 2024-02-19T15:08:30.786 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/ie31200_edac.ko 2024-02-19T15:08:30.786 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 56737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/sb_edac.ko 2024-02-19T15:08:30.786 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/x38_edac.ko 2024-02-19T15:08:30.786 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/extcon/ 2024-02-19T15:08:30.786 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/extcon/extcon-gpio.ko 2024-02-19T15:08:30.786 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 27089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/extcon/extcon-max14577.ko 2024-02-19T15:08:30.786 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/extcon/extcon-max77693.ko 2024-02-19T15:08:30.786 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/extcon/extcon-max8997.ko 2024-02-19T15:08:30.786 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/extcon/extcon-palmas.ko 2024-02-19T15:08:30.786 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/extcon/extcon-rt8973a.ko 2024-02-19T15:08:30.786 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 27689 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/extcon/extcon-sm5502.ko 2024-02-19T15:08:30.786 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/firmware/ 2024-02-19T15:08:30.786 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/firmware/dmi-sysfs.ko 2024-02-19T15:08:30.786 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/firmware/efi/ 2024-02-19T15:08:30.787 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/firmware/efi/efi-pstore.ko 2024-02-19T15:08:30.787 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/firmware/iscsi_ibft.ko 2024-02-19T15:08:30.787 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/ 2024-02-19T15:08:30.787 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-adp5520.ko 2024-02-19T15:08:30.787 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-amd8111.ko 2024-02-19T15:08:30.787 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-arizona.ko 2024-02-19T15:08:30.787 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-da9052.ko 2024-02-19T15:08:30.787 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-da9055.ko 2024-02-19T15:08:30.787 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-dln2.ko 2024-02-19T15:08:30.787 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 77753 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-f7188x.ko 2024-02-19T15:08:30.787 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-generic.ko 2024-02-19T15:08:30.787 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-ich.ko 2024-02-19T15:08:30.787 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-janz-ttl.ko 2024-02-19T15:08:30.787 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-kempld.ko 2024-02-19T15:08:30.788 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-lp3943.ko 2024-02-19T15:08:30.788 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-max7300.ko 2024-02-19T15:08:30.788 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-max730x.ko 2024-02-19T15:08:30.788 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-max732x.ko 2024-02-19T15:08:30.788 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-ml-ioh.ko 2024-02-19T15:08:30.788 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 47513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-pca953x.ko 2024-02-19T15:08:30.788 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-pcf857x.ko 2024-02-19T15:08:30.788 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-rdc321x.ko 2024-02-19T15:08:30.788 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-sch.ko 2024-02-19T15:08:30.788 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-sch311x.ko 2024-02-19T15:08:30.788 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-tps65912.ko 2024-02-19T15:08:30.788 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-twl4030.ko 2024-02-19T15:08:30.788 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-twl6040.ko 2024-02-19T15:08:30.789 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-viperboard.ko 2024-02-19T15:08:30.789 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-vx855.ko 2024-02-19T15:08:30.789 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-wm831x.ko 2024-02-19T15:08:30.789 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-wm8350.ko 2024-02-19T15:08:30.789 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-wm8994.ko 2024-02-19T15:08:30.789 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/ 2024-02-19T15:08:30.789 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-a4tech.ko 2024-02-19T15:08:30.789 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 39417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-apple.ko 2024-02-19T15:08:30.789 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-appleir.ko 2024-02-19T15:08:30.789 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-aureal.ko 2024-02-19T15:08:30.789 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-axff.ko 2024-02-19T15:08:30.789 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-belkin.ko 2024-02-19T15:08:30.789 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-cherry.ko 2024-02-19T15:08:30.789 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-chicony.ko 2024-02-19T15:08:30.789 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 60665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-cp2112.ko 2024-02-19T15:08:30.790 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-cypress.ko 2024-02-19T15:08:30.790 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-dr.ko 2024-02-19T15:08:30.790 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-elecom.ko 2024-02-19T15:08:30.790 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-elo.ko 2024-02-19T15:08:30.790 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-emsff.ko 2024-02-19T15:08:30.790 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-ezkey.ko 2024-02-19T15:08:30.790 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-gaff.ko 2024-02-19T15:08:30.790 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-generic.ko 2024-02-19T15:08:30.790 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-gt683r.ko 2024-02-19T15:08:30.790 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12585 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-gyration.ko 2024-02-19T15:08:30.790 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-holtek-kbd.ko 2024-02-19T15:08:30.790 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-holtek-mouse.ko 2024-02-19T15:08:30.790 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-holtekff.ko 2024-02-19T15:08:30.790 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-hyperv.ko 2024-02-19T15:08:30.791 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-icade.ko 2024-02-19T15:08:30.791 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-kensington.ko 2024-02-19T15:08:30.791 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-keytouch.ko 2024-02-19T15:08:30.791 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17113 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-kye.ko 2024-02-19T15:08:30.791 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-lcpower.ko 2024-02-19T15:08:30.791 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-led.ko 2024-02-19T15:08:30.791 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 42849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-lenovo.ko 2024-02-19T15:08:30.791 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-lg-g15.ko 2024-02-19T15:08:30.791 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 57977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-logitech-dj.ko 2024-02-19T15:08:30.791 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 100657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-02-19T15:08:30.791 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 80921 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-logitech.ko 2024-02-19T15:08:30.791 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-magicmouse.ko 2024-02-19T15:08:30.791 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-microsoft.ko 2024-02-19T15:08:30.792 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-monterey.ko 2024-02-19T15:08:30.792 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 50177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-multitouch.ko 2024-02-19T15:08:30.792 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 32497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-ntrig.ko 2024-02-19T15:08:30.792 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-ortek.ko 2024-02-19T15:08:30.792 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-penmount.ko 2024-02-19T15:08:30.792 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-petalynx.ko 2024-02-19T15:08:30.792 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 72865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-picolcd.ko 2024-02-19T15:08:30.792 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-pl.ko 2024-02-19T15:08:30.792 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-plantronics.ko 2024-02-19T15:08:30.792 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-primax.ko 2024-02-19T15:08:30.792 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 35529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-rmi.ko 2024-02-19T15:08:30.792 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-arvo.ko 2024-02-19T15:08:30.792 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-common.ko 2024-02-19T15:08:30.792 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-isku.ko 2024-02-19T15:08:30.793 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-kone.ko 2024-02-19T15:08:30.793 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-02-19T15:08:30.793 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-konepure.ko 2024-02-19T15:08:30.793 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-02-19T15:08:30.793 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-lua.ko 2024-02-19T15:08:30.793 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-pyra.ko 2024-02-19T15:08:30.793 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 27841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-ryos.ko 2024-02-19T15:08:30.793 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-savu.ko 2024-02-19T15:08:30.793 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat.ko 2024-02-19T15:08:30.793 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13585 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-saitek.ko 2024-02-19T15:08:30.793 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-samsung.ko 2024-02-19T15:08:30.793 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-sensor-hub.ko 2024-02-19T15:08:30.793 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-sjoy.ko 2024-02-19T15:08:30.793 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 51161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-sony.ko 2024-02-19T15:08:30.793 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-speedlink.ko 2024-02-19T15:08:30.794 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-steelseries.ko 2024-02-19T15:08:30.794 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-sunplus.ko 2024-02-19T15:08:30.794 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-thrustmaster.ko 2024-02-19T15:08:30.794 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-tivo.ko 2024-02-19T15:08:30.794 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-tmff.ko 2024-02-19T15:08:30.794 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-topseed.ko 2024-02-19T15:08:30.794 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-twinhan.ko 2024-02-19T15:08:30.794 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 70993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-uclogic.ko 2024-02-19T15:08:30.794 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-waltop.ko 2024-02-19T15:08:30.794 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 143393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-wiimote.ko 2024-02-19T15:08:30.794 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8065 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-xinmo.ko 2024-02-19T15:08:30.794 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-zpff.ko 2024-02-19T15:08:30.794 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-zydacron.ko 2024-02-19T15:08:30.794 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 269233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid.ko 2024-02-19T15:08:30.795 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 32233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/uhid.ko 2024-02-19T15:08:30.795 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/usbhid/ 2024-02-19T15:08:30.795 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 125641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/usbhid/usbhid.ko 2024-02-19T15:08:30.795 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/usbhid/usbkbd.ko 2024-02-19T15:08:30.795 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/usbhid/usbmouse.ko 2024-02-19T15:08:30.795 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 218817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/wacom.ko 2024-02-19T15:08:30.795 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hv/ 2024-02-19T15:08:30.795 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 64305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hv/hv_balloon.ko 2024-02-19T15:08:30.795 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 89553 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hv/hv_utils.ko 2024-02-19T15:08:30.795 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 305425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hv/hv_vmbus.ko 2024-02-19T15:08:30.795 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ 2024-02-19T15:08:30.795 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 66801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/abituguru.ko 2024-02-19T15:08:30.795 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 61817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/abituguru3.ko 2024-02-19T15:08:30.795 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/acpi_power_meter.ko 2024-02-19T15:08:30.795 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ad7414.ko 2024-02-19T15:08:30.796 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ad7418.ko 2024-02-19T15:08:30.796 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adc128d818.ko 2024-02-19T15:08:30.796 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adm1025.ko 2024-02-19T15:08:30.796 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 97889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adm1026.ko 2024-02-19T15:08:30.796 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adm1029.ko 2024-02-19T15:08:30.796 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 45465 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adm1031.ko 2024-02-19T15:08:30.796 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adm9240.ko 2024-02-19T15:08:30.796 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ads7828.ko 2024-02-19T15:08:30.796 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adt7410.ko 2024-02-19T15:08:30.796 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adt7411.ko 2024-02-19T15:08:30.796 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 74321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adt7462.ko 2024-02-19T15:08:30.796 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 39497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adt7470.ko 2024-02-19T15:08:30.796 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 69313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adt7475.ko 2024-02-19T15:08:30.797 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adt7x10.ko 2024-02-19T15:08:30.797 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 38985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/amc6821.ko 2024-02-19T15:08:30.797 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 55177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/applesmc.ko 2024-02-19T15:08:30.797 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 49825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/asb100.ko 2024-02-19T15:08:30.797 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 58529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/asc7621.ko 2024-02-19T15:08:30.797 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 40793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/asus_atk0110.ko 2024-02-19T15:08:30.797 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/atxp1.ko 2024-02-19T15:08:30.797 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/coretemp.ko 2024-02-19T15:08:30.797 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/da9052-hwmon.ko 2024-02-19T15:08:30.797 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/da9055-hwmon.ko 2024-02-19T15:08:30.797 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 49241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-02-19T15:08:30.797 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 99513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/dme1737.ko 2024-02-19T15:08:30.797 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ds1621.ko 2024-02-19T15:08:30.797 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ds620.ko 2024-02-19T15:08:30.798 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/emc1403.ko 2024-02-19T15:08:30.798 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 33097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/emc2103.ko 2024-02-19T15:08:30.798 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/emc6w201.ko 2024-02-19T15:08:30.798 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 66897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/f71805f.ko 2024-02-19T15:08:30.798 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 94601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/f71882fg.ko 2024-02-19T15:08:30.798 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 41313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/f75375s.ko 2024-02-19T15:08:30.798 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/fam15h_power.ko 2024-02-19T15:08:30.798 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 55361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/fschmd.ko 2024-02-19T15:08:30.798 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/g760a.ko 2024-02-19T15:08:30.798 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/g762.ko 2024-02-19T15:08:30.798 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 46713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/gl518sm.ko 2024-02-19T15:08:30.798 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 45441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/gl520sm.ko 2024-02-19T15:08:30.798 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/hih6130.ko 2024-02-19T15:08:30.798 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/hwmon-vid.ko 2024-02-19T15:08:30.798 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/i5500_temp.ko 2024-02-19T15:08:30.799 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/i5k_amb.ko 2024-02-19T15:08:30.799 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ibmaem.ko 2024-02-19T15:08:30.799 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ibmpex.ko 2024-02-19T15:08:30.799 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ina209.ko 2024-02-19T15:08:30.799 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ina2xx.ko 2024-02-19T15:08:30.799 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 140137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/it87.ko 2024-02-19T15:08:30.799 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18753 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/jc42.ko 2024-02-19T15:08:30.799 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/k10temp.ko 2024-02-19T15:08:30.799 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/k8temp.ko 2024-02-19T15:08:30.799 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lineage-pem.ko 2024-02-19T15:08:30.799 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 48513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm63.ko 2024-02-19T15:08:30.799 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm73.ko 2024-02-19T15:08:30.799 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 38849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm75.ko 2024-02-19T15:08:30.800 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm77.ko 2024-02-19T15:08:30.800 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm78.ko 2024-02-19T15:08:30.800 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm80.ko 2024-02-19T15:08:30.800 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm83.ko 2024-02-19T15:08:30.800 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 72745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm85.ko 2024-02-19T15:08:30.800 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 46065 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm87.ko 2024-02-19T15:08:30.800 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 37825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm90.ko 2024-02-19T15:08:30.800 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm92.ko 2024-02-19T15:08:30.800 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 108753 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm93.ko 2024-02-19T15:08:30.800 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 33305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm95234.ko 2024-02-19T15:08:30.800 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm95241.ko 2024-02-19T15:08:30.800 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm95245.ko 2024-02-19T15:08:30.800 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ltc2945.ko 2024-02-19T15:08:30.801 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ltc4151.ko 2024-02-19T15:08:30.801 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15105 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ltc4215.ko 2024-02-19T15:08:30.801 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ltc4222.ko 2024-02-19T15:08:30.801 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ltc4245.ko 2024-02-19T15:08:30.801 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ltc4260.ko 2024-02-19T15:08:30.801 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ltc4261.ko 2024-02-19T15:08:30.801 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 33401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/max16065.ko 2024-02-19T15:08:30.801 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/max1619.ko 2024-02-19T15:08:30.801 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/max1668.ko 2024-02-19T15:08:30.801 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/max197.ko 2024-02-19T15:08:30.801 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 27041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/max6639.ko 2024-02-19T15:08:30.801 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/max6650.ko 2024-02-19T15:08:30.802 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 35689 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/max6697.ko 2024-02-19T15:08:30.802 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/mc13783-adc.ko 2024-02-19T15:08:30.802 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/mcp3021.ko 2024-02-19T15:08:30.802 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2024-02-19T15:08:30.802 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 47793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/nct6683.ko 2024-02-19T15:08:30.802 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 119721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/nct6775-core.ko 2024-02-19T15:08:30.802 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 61441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/nct6775.ko 2024-02-19T15:08:30.802 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 59649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/nct7802.ko 2024-02-19T15:08:30.802 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 82721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pc87360.ko 2024-02-19T15:08:30.802 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 54345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pc87427.ko 2024-02-19T15:08:30.802 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pcf8591.ko 2024-02-19T15:08:30.802 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/ 2024-02-19T15:08:30.802 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-02-19T15:08:30.803 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-02-19T15:08:30.803 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-02-19T15:08:30.803 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/max16064.ko 2024-02-19T15:08:30.803 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/max34440.ko 2024-02-19T15:08:30.803 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/max8688.ko 2024-02-19T15:08:30.803 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-02-19T15:08:30.803 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 82481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-02-19T15:08:30.803 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-02-19T15:08:30.803 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-02-19T15:08:30.803 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-02-19T15:08:30.803 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20593 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-02-19T15:08:30.803 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/powr1220.ko 2024-02-19T15:08:30.803 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/sch5627.ko 2024-02-19T15:08:30.803 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/sch5636.ko 2024-02-19T15:08:30.803 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/sch56xx-common.ko 2024-02-19T15:08:30.804 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 32049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/sht15.ko 2024-02-19T15:08:30.804 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/sht21.ko 2024-02-19T15:08:30.804 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/shtc1.ko 2024-02-19T15:08:30.804 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 40489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/sis5595.ko 2024-02-19T15:08:30.804 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/smsc47b397.ko 2024-02-19T15:08:30.804 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 40361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/smsc47m1.ko 2024-02-19T15:08:30.804 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 37289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/smsc47m192.ko 2024-02-19T15:08:30.804 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26465 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/thmc50.ko 2024-02-19T15:08:30.804 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/tmp102.ko 2024-02-19T15:08:30.804 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/tmp103.ko 2024-02-19T15:08:30.804 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/tmp401.ko 2024-02-19T15:08:30.804 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/tmp421.ko 2024-02-19T15:08:30.804 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/via-cputemp.ko 2024-02-19T15:08:30.804 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 43449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/via686a.ko 2024-02-19T15:08:30.805 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 52369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/vt1211.ko 2024-02-19T15:08:30.805 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 53209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/vt8231.ko 2024-02-19T15:08:30.805 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 58265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/w83627ehf.ko 2024-02-19T15:08:30.805 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 79841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/w83627hf.ko 2024-02-19T15:08:30.805 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 71257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/w83781d.ko 2024-02-19T15:08:30.805 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 63201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/w83791d.ko 2024-02-19T15:08:30.805 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 71233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/w83792d.ko 2024-02-19T15:08:30.805 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 101385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/w83793.ko 2024-02-19T15:08:30.805 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 119417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/w83795.ko 2024-02-19T15:08:30.805 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/w83l785ts.ko 2024-02-19T15:08:30.805 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 35529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/w83l786ng.ko 2024-02-19T15:08:30.805 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/wm831x-hwmon.ko 2024-02-19T15:08:30.805 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/wm8350-hwmon.ko 2024-02-19T15:08:30.805 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/ 2024-02-19T15:08:30.805 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/algos/ 2024-02-19T15:08:30.806 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-02-19T15:08:30.806 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-02-19T15:08:30.806 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/ 2024-02-19T15:08:30.806 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-ali1535.ko 2024-02-19T15:08:30.806 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 32481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-ali1563.ko 2024-02-19T15:08:30.806 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2024-02-19T15:08:30.806 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-02-19T15:08:30.806 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-02-19T15:08:30.806 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-02-19T15:08:30.806 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2024-02-19T15:08:30.806 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 6833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-02-19T15:08:30.806 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2024-02-19T15:08:30.806 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 50305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-02-19T15:08:30.806 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-02-19T15:08:30.807 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-02-19T15:08:30.807 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-02-19T15:08:30.807 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-dln2.ko 2024-02-19T15:08:30.807 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-gpio.ko 2024-02-19T15:08:30.807 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 67417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-i801.ko 2024-02-19T15:08:30.807 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-isch.ko 2024-02-19T15:08:30.807 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 47801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-02-19T15:08:30.807 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-kempld.ko 2024-02-19T15:08:30.807 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-02-19T15:08:30.807 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-02-19T15:08:30.807 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-ocores.ko 2024-02-19T15:08:30.807 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-parport.ko 2024-02-19T15:08:30.807 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-02-19T15:08:30.807 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 50977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-02-19T15:08:30.807 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2024-02-19T15:08:30.808 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-02-19T15:08:30.808 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-02-19T15:08:30.808 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-sis5595.ko 2024-02-19T15:08:30.808 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-sis630.ko 2024-02-19T15:08:30.808 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-02-19T15:08:30.808 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2024-02-19T15:08:30.808 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-02-19T15:08:30.808 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-via.ko 2024-02-19T15:08:30.808 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 27625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-02-19T15:08:30.808 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-02-19T15:08:30.808 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 47569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-xiic.ko 2024-02-19T15:08:30.808 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/i2c-mux.ko 2024-02-19T15:08:30.808 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/i2c-smbus.ko 2024-02-19T15:08:30.808 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/i2c-stub.ko 2024-02-19T15:08:30.808 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/muxes/ 2024-02-19T15:08:30.809 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2024-02-19T15:08:30.809 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2024-02-19T15:08:30.809 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2024-02-19T15:08:30.809 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ 2024-02-19T15:08:30.809 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/core/ 2024-02-19T15:08:30.809 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 273529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/core/ib_cm.ko 2024-02-19T15:08:30.809 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 883945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/core/ib_core.ko 2024-02-19T15:08:30.809 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 69185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/core/ib_umad.ko 2024-02-19T15:08:30.809 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 324337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-02-19T15:08:30.809 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 97841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/core/iw_cm.ko 2024-02-19T15:08:30.809 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 282345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/core/rdma_cm.ko 2024-02-19T15:08:30.809 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 66057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-02-19T15:08:30.809 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/ 2024-02-19T15:08:30.809 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/cxgb4/ 2024-02-19T15:08:30.810 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 493177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-02-19T15:08:30.810 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/mlx4/ 2024-02-19T15:08:30.810 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 420593 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-02-19T15:08:30.810 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/mlx5/ 2024-02-19T15:08:30.810 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 741673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-02-19T15:08:30.810 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/mthca/ 2024-02-19T15:08:30.810 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 296881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-02-19T15:08:30.810 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/ocrdma/ 2024-02-19T15:08:30.810 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 153729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2024-02-19T15:08:30.810 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/usnic/ 2024-02-19T15:08:30.810 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 169145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2024-02-19T15:08:30.810 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/ 2024-02-19T15:08:30.810 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/ipoib/ 2024-02-19T15:08:30.810 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 266577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-02-19T15:08:30.810 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/iser/ 2024-02-19T15:08:30.811 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 100281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-02-19T15:08:30.811 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/isert/ 2024-02-19T15:08:30.811 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 111337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-02-19T15:08:30.811 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/srp/ 2024-02-19T15:08:30.811 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 128665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-02-19T15:08:30.811 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/srpt/ 2024-02-19T15:08:30.811 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 129553 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-02-19T15:08:30.811 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/ 2024-02-19T15:08:30.811 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/evbug.ko 2024-02-19T15:08:30.811 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/ff-memless.ko 2024-02-19T15:08:30.811 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/gameport/ 2024-02-19T15:08:30.811 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-02-19T15:08:30.811 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/gameport/fm801-gp.ko 2024-02-19T15:08:30.811 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 41457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/gameport/gameport.ko 2024-02-19T15:08:30.812 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 41713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/joydev.ko 2024-02-19T15:08:30.812 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/ 2024-02-19T15:08:30.812 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/adp5520-keys.ko 2024-02-19T15:08:30.812 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12609 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/adp5588-keys.ko 2024-02-19T15:08:30.812 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/adp5589-keys.ko 2024-02-19T15:08:30.812 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2024-02-19T15:08:30.812 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 37281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/gpio_keys.ko 2024-02-19T15:08:30.812 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2024-02-19T15:08:30.812 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/lkkbd.ko 2024-02-19T15:08:30.812 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/lm8323.ko 2024-02-19T15:08:30.812 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/lm8333.ko 2024-02-19T15:08:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/matrix_keypad.ko 2024-02-19T15:08:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/max7359_keypad.ko 2024-02-19T15:08:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/mcs_touchkey.ko 2024-02-19T15:08:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2024-02-19T15:08:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/newtonkbd.ko 2024-02-19T15:08:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/opencores-kbd.ko 2024-02-19T15:08:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/qt1070.ko 2024-02-19T15:08:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/qt2160.ko 2024-02-19T15:08:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/samsung-keypad.ko 2024-02-19T15:08:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/stowaway.ko 2024-02-19T15:08:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/sunkbd.ko 2024-02-19T15:08:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/tca6416-keypad.ko 2024-02-19T15:08:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/tca8418_keypad.ko 2024-02-19T15:08:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/twl4030_keypad.ko 2024-02-19T15:08:30.814 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/xtkbd.ko 2024-02-19T15:08:30.814 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/matrix-keymap.ko 2024-02-19T15:08:30.814 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/ 2024-02-19T15:08:30.814 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/appletouch.ko 2024-02-19T15:08:30.814 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/bcm5974.ko 2024-02-19T15:08:30.814 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 123201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/cyapatp.ko 2024-02-19T15:08:30.814 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 89393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/elan_i2c.ko 2024-02-19T15:08:30.814 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/gpio_mouse.ko 2024-02-19T15:08:30.814 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 335921 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/psmouse.ko 2024-02-19T15:08:30.814 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14593 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/sermouse.ko 2024-02-19T15:08:30.814 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-02-19T15:08:30.814 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/synaptics_usb.ko 2024-02-19T15:08:30.814 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/vsxxxaa.ko 2024-02-19T15:08:30.814 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/rmi4/ 2024-02-19T15:08:30.815 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 109601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/rmi4/rmi_core.ko 2024-02-19T15:08:30.815 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/serio/ 2024-02-19T15:08:30.815 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/serio/altera_ps2.ko 2024-02-19T15:08:30.815 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/serio/arc_ps2.ko 2024-02-19T15:08:30.815 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17689 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/serio/ct82c710.ko 2024-02-19T15:08:30.815 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-02-19T15:08:30.815 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/serio/parkbd.ko 2024-02-19T15:08:30.815 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/serio/pcips2.ko 2024-02-19T15:08:30.815 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/serio/ps2mult.ko 2024-02-19T15:08:30.815 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/serio/serio_raw.ko 2024-02-19T15:08:30.815 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/serio/serport.ko 2024-02-19T15:08:30.815 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/sparse-keymap.ko 2024-02-19T15:08:30.815 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/ 2024-02-19T15:08:30.815 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-88pm860x.ko 2024-02-19T15:08:30.816 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-adp5520.ko 2024-02-19T15:08:30.816 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 59657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-bd2802.ko 2024-02-19T15:08:30.816 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24689 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-blinkm.ko 2024-02-19T15:08:30.816 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-da903x.ko 2024-02-19T15:08:30.816 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-da9052.ko 2024-02-19T15:08:30.816 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16689 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-gpio.ko 2024-02-19T15:08:30.816 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-lm3530.ko 2024-02-19T15:08:30.816 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-lm3533.ko 2024-02-19T15:08:30.816 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-lm355x.ko 2024-02-19T15:08:30.816 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-lm3642.ko 2024-02-19T15:08:30.816 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-lp3944.ko 2024-02-19T15:08:30.816 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-lp8788.ko 2024-02-19T15:08:30.816 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-max8997.ko 2024-02-19T15:08:30.816 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-mc13783.ko 2024-02-19T15:08:30.817 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-menf21bmc.ko 2024-02-19T15:08:30.817 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-pca9532.ko 2024-02-19T15:08:30.817 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20465 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-pca955x.ko 2024-02-19T15:08:30.817 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-pca963x.ko 2024-02-19T15:08:30.817 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-ss4200.ko 2024-02-19T15:08:30.817 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-tca6507.ko 2024-02-19T15:08:30.817 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-wm831x-status.ko 2024-02-19T15:08:30.817 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11753 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-wm8350.ko 2024-02-19T15:08:30.817 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/trigger/ 2024-02-19T15:08:30.817 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-02-19T15:08:30.817 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-02-19T15:08:30.817 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-02-19T15:08:30.817 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-02-19T15:08:30.817 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-02-19T15:08:30.818 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-02-19T15:08:30.818 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-02-19T15:08:30.818 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-02-19T15:08:30.818 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-02-19T15:08:30.818 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/ 2024-02-19T15:08:30.818 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/bcache/ 2024-02-19T15:08:30.818 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 647121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/bcache/bcache.ko 2024-02-19T15:08:30.818 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 40489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-bio-prison.ko 2024-02-19T15:08:30.818 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 93321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-bufio.ko 2024-02-19T15:08:30.818 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 47441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-cache-smq.ko 2024-02-19T15:08:30.818 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 155825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-cache.ko 2024-02-19T15:08:30.818 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 105473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-crypt.ko 2024-02-19T15:08:30.818 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-delay.ko 2024-02-19T15:08:30.818 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 53769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-era.ko 2024-02-19T15:08:30.819 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-flakey.ko 2024-02-19T15:08:30.819 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 41505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-log-userspace.ko 2024-02-19T15:08:30.819 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 32649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-log.ko 2024-02-19T15:08:30.819 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 49449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-mirror.ko 2024-02-19T15:08:30.819 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 77881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-multipath.ko 2024-02-19T15:08:30.819 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-queue-length.ko 2024-02-19T15:08:30.819 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 70985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-raid.ko 2024-02-19T15:08:30.819 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-region-hash.ko 2024-02-19T15:08:30.819 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-round-robin.ko 2024-02-19T15:08:30.819 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-service-time.ko 2024-02-19T15:08:30.819 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 109521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-snapshot.ko 2024-02-19T15:08:30.819 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-switch.ko 2024-02-19T15:08:30.819 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 181777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-thin-pool.ko 2024-02-19T15:08:30.819 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 50545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-verity.ko 2024-02-19T15:08:30.819 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-zero.ko 2024-02-19T15:08:30.819 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/persistent-data/ 2024-02-19T15:08:30.820 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 199881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-02-19T15:08:30.820 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 37321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/raid0.ko 2024-02-19T15:08:30.820 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 100841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/raid1.ko 2024-02-19T15:08:30.820 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 126761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/raid10.ko 2024-02-19T15:08:30.820 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 369433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/raid456.ko 2024-02-19T15:08:30.820 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/message/ 2024-02-19T15:08:30.820 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/message/fusion/ 2024-02-19T15:08:30.820 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 178049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/message/fusion/mptbase.ko 2024-02-19T15:08:30.820 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 63241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/message/fusion/mptctl.ko 2024-02-19T15:08:30.820 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 46249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/message/fusion/mptfc.ko 2024-02-19T15:08:30.820 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 37121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/message/fusion/mptlan.ko 2024-02-19T15:08:30.820 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 123233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/message/fusion/mptsas.ko 2024-02-19T15:08:30.820 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 80985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/message/fusion/mptscsih.ko 2024-02-19T15:08:30.820 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 52929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/message/fusion/mptspi.ko 2024-02-19T15:08:30.820 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/ 2024-02-19T15:08:30.821 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/88pm800.ko 2024-02-19T15:08:30.821 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/88pm805.ko 2024-02-19T15:08:30.821 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11105 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/88pm80x.ko 2024-02-19T15:08:30.821 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/arizona-i2c.ko 2024-02-19T15:08:30.821 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 123649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/arizona.ko 2024-02-19T15:08:30.821 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/bcm590xx.ko 2024-02-19T15:08:30.821 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/cros_ec_dev.ko 2024-02-19T15:08:30.821 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/dln2.ko 2024-02-19T15:08:30.821 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16593 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/janz-cmodio.ko 2024-02-19T15:08:30.821 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 53489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/kempld-core.ko 2024-02-19T15:08:30.821 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 27433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/lm3533-core.ko 2024-02-19T15:08:30.821 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/lm3533-ctrlbank.ko 2024-02-19T15:08:30.821 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12921 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/lp3943.ko 2024-02-19T15:08:30.821 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 56969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/lpc_ich.ko 2024-02-19T15:08:30.822 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/lpc_sch.ko 2024-02-19T15:08:30.822 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/max8907.ko 2024-02-19T15:08:30.822 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/mc13xxx-core.ko 2024-02-19T15:08:30.822 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/mc13xxx-i2c.ko 2024-02-19T15:08:30.822 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/menf21bmc.ko 2024-02-19T15:08:30.822 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/pcf50633-adc.ko 2024-02-19T15:08:30.822 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/pcf50633-gpio.ko 2024-02-19T15:08:30.822 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/pcf50633.ko 2024-02-19T15:08:30.822 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/rdc321x-southbridge.ko 2024-02-19T15:08:30.822 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/retu-mfd.ko 2024-02-19T15:08:30.822 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 80345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/si476x-core.ko 2024-02-19T15:08:30.822 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 48369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/sm501.ko 2024-02-19T15:08:30.822 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 51209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/tps65010.ko 2024-02-19T15:08:30.822 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/tps6507x.ko 2024-02-19T15:08:30.823 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/viperboard.ko 2024-02-19T15:08:30.823 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/vx855.ko 2024-02-19T15:08:30.823 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/wl1273-core.ko 2024-02-19T15:08:30.823 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 44265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/wm8994.ko 2024-02-19T15:08:30.823 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/ 2024-02-19T15:08:30.823 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/ad525x_dpot-i2c.ko 2024-02-19T15:08:30.823 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 48393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/ad525x_dpot.ko 2024-02-19T15:08:30.823 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/altera-stapl/ 2024-02-19T15:08:30.823 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 46513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-02-19T15:08:30.823 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/apds9802als.ko 2024-02-19T15:08:30.823 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/apds990x.ko 2024-02-19T15:08:30.823 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 45497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/bh1770glc.ko 2024-02-19T15:08:30.823 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/c2port/ 2024-02-19T15:08:30.823 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2024-02-19T15:08:30.823 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 35561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/c2port/core.ko 2024-02-19T15:08:30.824 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/cb710/ 2024-02-19T15:08:30.824 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/cb710/cb710.ko 2024-02-19T15:08:30.824 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/ds1682.ko 2024-02-19T15:08:30.824 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/dummy-irq.ko 2024-02-19T15:08:30.824 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/echo/ 2024-02-19T15:08:30.824 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/echo/echo.ko 2024-02-19T15:08:30.824 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/eeprom/ 2024-02-19T15:08:30.824 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 39025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/eeprom/at24.ko 2024-02-19T15:08:30.824 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-02-19T15:08:30.824 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/eeprom/max6875.ko 2024-02-19T15:08:30.824 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 32913 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/enclosure.ko 2024-02-19T15:08:30.824 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/genwqe/ 2024-02-19T15:08:30.824 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 152073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/genwqe/genwqe_card.ko 2024-02-19T15:08:30.824 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/hmc6352.ko 2024-02-19T15:08:30.825 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/hpilo.ko 2024-02-19T15:08:30.825 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/ibmasm/ 2024-02-19T15:08:30.825 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 72561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/ibmasm/ibmasm.ko 2024-02-19T15:08:30.825 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19689 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/ics932s401.ko 2024-02-19T15:08:30.825 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21593 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/isl29003.ko 2024-02-19T15:08:30.825 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/isl29020.ko 2024-02-19T15:08:30.825 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/lis3lv02d/ 2024-02-19T15:08:30.825 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 42417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-02-19T15:08:30.825 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-02-19T15:08:30.825 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/mei/ 2024-02-19T15:08:30.825 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 125865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/mei/mei-me.ko 2024-02-19T15:08:30.825 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 56377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/mei/mei-txe.ko 2024-02-19T15:08:30.825 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 313601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/mei/mei.ko 2024-02-19T15:08:30.825 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/phantom.ko 2024-02-19T15:08:30.825 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/ti-st/ 2024-02-19T15:08:30.826 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 74305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/ti-st/st_drv.ko 2024-02-19T15:08:30.826 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/tifm_7xx1.ko 2024-02-19T15:08:30.826 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28689 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/tifm_core.ko 2024-02-19T15:08:30.826 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/tsl2550.ko 2024-02-19T15:08:30.826 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 48345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/vmw_balloon.ko 2024-02-19T15:08:30.826 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/vmw_vmci/ 2024-02-19T15:08:30.826 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 183569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-02-19T15:08:30.826 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ 2024-02-19T15:08:30.826 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/bonding/ 2024-02-19T15:08:30.826 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 491721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/bonding/bonding.ko 2024-02-19T15:08:30.826 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/dsa/ 2024-02-19T15:08:30.826 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/dsa/b53/ 2024-02-19T15:08:30.826 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 105937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/dsa/b53/b53_common.ko 2024-02-19T15:08:30.826 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 40225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/dsa/bcm-sf2.ko 2024-02-19T15:08:30.826 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/dsa/mv88e6060.ko 2024-02-19T15:08:30.827 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/dsa/mv88e6xxx/ 2024-02-19T15:08:30.827 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 459065 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-02-19T15:08:30.827 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/dummy.ko 2024-02-19T15:08:30.827 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/eql.ko 2024-02-19T15:08:30.827 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/ 2024-02-19T15:08:30.827 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/3com/ 2024-02-19T15:08:30.827 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 45961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-02-19T15:08:30.827 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 42193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-02-19T15:08:30.827 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 130665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-02-19T15:08:30.827 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 68785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-02-19T15:08:30.827 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/8390/ 2024-02-19T15:08:30.827 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 38049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/8390/8390.ko 2024-02-19T15:08:30.827 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 64617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-02-19T15:08:30.827 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 27089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-02-19T15:08:30.827 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 101681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-02-19T15:08:30.827 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/adaptec/ 2024-02-19T15:08:30.828 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 54625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-02-19T15:08:30.828 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/agere/ 2024-02-19T15:08:30.828 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 69169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/agere/et131x.ko 2024-02-19T15:08:30.828 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/alteon/ 2024-02-19T15:08:30.828 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 59993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-02-19T15:08:30.828 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/altera/ 2024-02-19T15:08:30.828 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 41409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-02-19T15:08:30.828 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/amd/ 2024-02-19T15:08:30.828 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 54617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-02-19T15:08:30.828 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 49193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-02-19T15:08:30.828 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 87457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-02-19T15:08:30.828 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/atheros/ 2024-02-19T15:08:30.828 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/atheros/alx/ 2024-02-19T15:08:30.828 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 97201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-02-19T15:08:30.829 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/atheros/atl1c/ 2024-02-19T15:08:30.829 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 102497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-02-19T15:08:30.829 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/atheros/atl1e/ 2024-02-19T15:08:30.829 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 90209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-02-19T15:08:30.829 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/atheros/atlx/ 2024-02-19T15:08:30.829 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 81321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-02-19T15:08:30.829 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 69545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-02-19T15:08:30.829 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/broadcom/ 2024-02-19T15:08:30.829 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 78385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-02-19T15:08:30.829 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 176697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-02-19T15:08:30.829 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2024-02-19T15:08:30.829 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 1434505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-02-19T15:08:30.829 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 128433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-02-19T15:08:30.829 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/broadcom/genet/ 2024-02-19T15:08:30.830 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 131281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-02-19T15:08:30.830 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 314329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-02-19T15:08:30.830 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/brocade/ 2024-02-19T15:08:30.830 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/brocade/bna/ 2024-02-19T15:08:30.830 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 323737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-02-19T15:08:30.830 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/ 2024-02-19T15:08:30.830 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/cxgb/ 2024-02-19T15:08:30.830 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 193593 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-02-19T15:08:30.830 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2024-02-19T15:08:30.830 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 347217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-02-19T15:08:30.830 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2024-02-19T15:08:30.830 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 1045449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-02-19T15:08:30.830 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2024-02-19T15:08:30.831 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 166145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-02-19T15:08:30.831 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2024-02-19T15:08:30.831 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-02-19T15:08:30.831 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/cisco/ 2024-02-19T15:08:30.831 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/cisco/enic/ 2024-02-19T15:08:30.831 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 204249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-02-19T15:08:30.831 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dec/ 2024-02-19T15:08:30.831 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dec/tulip/ 2024-02-19T15:08:30.831 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 69433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-02-19T15:08:30.831 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 61425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-02-19T15:08:30.831 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 141201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-02-19T15:08:30.831 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 51369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-02-19T15:08:30.831 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 62833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-02-19T15:08:30.832 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-02-19T15:08:30.832 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dlink/ 2024-02-19T15:08:30.832 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 60673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-02-19T15:08:30.832 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 62833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-02-19T15:08:30.832 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 45489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dnet.ko 2024-02-19T15:08:30.832 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/ec_bhf.ko 2024-02-19T15:08:30.832 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/emulex/ 2024-02-19T15:08:30.832 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/emulex/benet/ 2024-02-19T15:08:30.832 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 281217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-02-19T15:08:30.832 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 49457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/ethoc.ko 2024-02-19T15:08:30.832 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 51785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/fealnx.ko 2024-02-19T15:08:30.832 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/fujitsu/ 2024-02-19T15:08:30.832 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 45553 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2024-02-19T15:08:30.832 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/ 2024-02-19T15:08:30.833 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 90305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/e100.ko 2024-02-19T15:08:30.833 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/e1000/ 2024-02-19T15:08:30.833 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 295697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-02-19T15:08:30.833 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/e1000e/ 2024-02-19T15:08:30.833 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 584353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-02-19T15:08:30.833 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/fm10k/ 2024-02-19T15:08:30.833 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 246041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-02-19T15:08:30.833 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/i40e/ 2024-02-19T15:08:30.833 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 944865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-02-19T15:08:30.833 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/iavf/ 2024-02-19T15:08:30.833 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 288529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-02-19T15:08:30.833 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/igb/ 2024-02-19T15:08:30.833 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 513857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-02-19T15:08:30.906 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/igbvf/ 2024-02-19T15:08:30.906 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 93489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-02-19T15:08:30.906 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/ixgbe/ 2024-02-19T15:08:30.906 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 734729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-02-19T15:08:30.906 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/ixgbevf/ 2024-02-19T15:08:30.906 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 151745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-02-19T15:08:30.906 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 86993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/jme.ko 2024-02-19T15:08:30.906 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/marvell/ 2024-02-19T15:08:30.906 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-02-19T15:08:30.907 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 102441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/marvell/skge.ko 2024-02-19T15:08:30.907 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 122609 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-02-19T15:08:30.907 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/mellanox/ 2024-02-19T15:08:30.907 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/mellanox/mlx4/ 2024-02-19T15:08:30.907 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 727481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-02-19T15:08:30.907 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 278881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-02-19T15:08:30.907 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/mellanox/mlx5/ 2024-02-19T15:08:30.907 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2024-02-19T15:08:30.907 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 1121769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-02-19T15:08:30.907 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/micrel/ 2024-02-19T15:08:30.907 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 52985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/micrel/ks8842.ko 2024-02-19T15:08:30.907 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 40881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2024-02-19T15:08:30.907 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2024-02-19T15:08:30.907 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 95577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-02-19T15:08:30.908 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/myricom/ 2024-02-19T15:08:30.908 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/myricom/myri10ge/ 2024-02-19T15:08:30.908 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 108409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-02-19T15:08:30.908 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/natsemi/ 2024-02-19T15:08:30.908 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 73617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-02-19T15:08:30.908 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 47801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-02-19T15:08:30.908 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/neterion/ 2024-02-19T15:08:30.908 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 151097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-02-19T15:08:30.908 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/nvidia/ 2024-02-19T15:08:30.908 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 136801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-02-19T15:08:30.908 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/packetengines/ 2024-02-19T15:08:30.908 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 50433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-02-19T15:08:30.908 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 42985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-02-19T15:08:30.908 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/qlogic/ 2024-02-19T15:08:30.909 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/qlogic/netxen/ 2024-02-19T15:08:30.909 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 227865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-02-19T15:08:30.909 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 87017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-02-19T15:08:30.909 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2024-02-19T15:08:30.909 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 672057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-02-19T15:08:30.909 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/rdc/ 2024-02-19T15:08:30.909 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 42937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-02-19T15:08:30.909 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/realtek/ 2024-02-19T15:08:30.909 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 67281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-02-19T15:08:30.909 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 83809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-02-19T15:08:30.909 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/realtek/atp.ko 2024-02-19T15:08:30.909 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 194073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-02-19T15:08:30.909 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/samsung/ 2024-02-19T15:08:30.909 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/samsung/sxgbe/ 2024-02-19T15:08:30.909 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 104049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2024-02-19T15:08:30.910 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/sfc/ 2024-02-19T15:08:30.910 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 1067001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-02-19T15:08:30.910 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/silan/ 2024-02-19T15:08:30.910 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 47009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-02-19T15:08:30.910 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/sis/ 2024-02-19T15:08:30.910 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 54729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/sis/sis190.ko 2024-02-19T15:08:30.910 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 63201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/sis/sis900.ko 2024-02-19T15:08:30.910 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/smsc/ 2024-02-19T15:08:30.910 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 61425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-02-19T15:08:30.910 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 67297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-02-19T15:08:30.910 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 73257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-02-19T15:08:30.910 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 68049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-02-19T15:08:30.910 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/stmicro/ 2024-02-19T15:08:30.910 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/stmicro/stmmac/ 2024-02-19T15:08:30.910 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2024-02-19T15:08:30.911 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 39385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-02-19T15:08:30.911 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2024-02-19T15:08:30.911 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 497201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-02-19T15:08:30.911 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/sun/ 2024-02-19T15:08:30.911 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 103185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/sun/cassini.ko 2024-02-19T15:08:30.911 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 166049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/sun/niu.ko 2024-02-19T15:08:30.911 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 66273 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/sun/sungem.ko 2024-02-19T15:08:30.911 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 78721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-02-19T15:08:30.911 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/tehuti/ 2024-02-19T15:08:30.911 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 50017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-02-19T15:08:30.911 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/ti/ 2024-02-19T15:08:30.911 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 92345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/ti/tlan.ko 2024-02-19T15:08:30.911 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/via/ 2024-02-19T15:08:30.911 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 80001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-02-19T15:08:30.911 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 83721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-02-19T15:08:30.912 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/wiznet/ 2024-02-19T15:08:30.912 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 46313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-02-19T15:08:30.912 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-02-19T15:08:30.912 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/xircom/ 2024-02-19T15:08:30.912 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 59921 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-02-19T15:08:30.912 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/fddi/ 2024-02-19T15:08:30.912 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 41185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/fddi/defxx.ko 2024-02-19T15:08:30.912 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/fddi/skfp/ 2024-02-19T15:08:30.912 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 217065 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/fddi/skfp/skfp.ko 2024-02-19T15:08:30.912 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 74729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/geneve.ko 2024-02-19T15:08:30.913 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/hyperv/ 2024-02-19T15:08:30.913 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 237265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-02-19T15:08:30.913 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ieee802154/ 2024-02-19T15:08:30.913 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ieee802154/fakelb.ko 2024-02-19T15:08:30.913 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ifb.ko 2024-02-19T15:08:30.913 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ipvlan/ 2024-02-19T15:08:30.913 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 85857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ipvlan/ipvlan.ko 2024-02-19T15:08:30.913 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 62897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/macvlan.ko 2024-02-19T15:08:30.913 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/macvtap.ko 2024-02-19T15:08:30.913 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/mdio/ 2024-02-19T15:08:30.913 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-02-19T15:08:30.913 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-02-19T15:08:30.913 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/mdio/mdio-gpio.ko 2024-02-19T15:08:30.913 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/mdio/mdio-regmap.ko 2024-02-19T15:08:30.914 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/mdio.ko 2024-02-19T15:08:30.914 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/mii.ko 2024-02-19T15:08:30.914 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 45057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/netconsole.ko 2024-02-19T15:08:30.914 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/nlmon.ko 2024-02-19T15:08:30.914 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/pcs/ 2024-02-19T15:08:30.914 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/pcs/pcs-lynx.ko 2024-02-19T15:08:30.914 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 39137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-02-19T15:08:30.914 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/ 2024-02-19T15:08:30.914 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/amd.ko 2024-02-19T15:08:30.914 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 50001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-02-19T15:08:30.914 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 44633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/bcm7xxx.ko 2024-02-19T15:08:30.914 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/bcm87xx.ko 2024-02-19T15:08:30.914 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 44825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/broadcom.ko 2024-02-19T15:08:30.914 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/cicada.ko 2024-02-19T15:08:30.914 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/davicom.ko 2024-02-19T15:08:30.915 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/et1011c.ko 2024-02-19T15:08:30.915 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/icplus.ko 2024-02-19T15:08:30.915 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/lxt.ko 2024-02-19T15:08:30.915 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 81961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/marvell.ko 2024-02-19T15:08:30.915 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 76569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/micrel.ko 2024-02-19T15:08:30.915 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/national.ko 2024-02-19T15:08:30.915 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 99009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/phylink.ko 2024-02-19T15:08:30.915 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/qsemi.ko 2024-02-19T15:08:30.915 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 49977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/realtek.ko 2024-02-19T15:08:30.915 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/smsc.ko 2024-02-19T15:08:30.915 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/ste10Xp.ko 2024-02-19T15:08:30.915 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/vitesse.ko 2024-02-19T15:08:30.915 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/plip/ 2024-02-19T15:08:30.915 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 43721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/plip/plip.ko 2024-02-19T15:08:30.915 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ppp/ 2024-02-19T15:08:30.915 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ppp/bsd_comp.ko 2024-02-19T15:08:30.916 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ppp/ppp_async.ko 2024-02-19T15:08:30.916 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ppp/ppp_deflate.ko 2024-02-19T15:08:30.916 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ppp/ppp_mppe.ko 2024-02-19T15:08:30.916 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ppp/ppp_synctty.ko 2024-02-19T15:08:30.916 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 42673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ppp/pppoe.ko 2024-02-19T15:08:30.916 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ppp/pppox.ko 2024-02-19T15:08:30.916 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 32673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ppp/pptp.ko 2024-02-19T15:08:30.916 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/sb1000.ko 2024-02-19T15:08:30.916 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/slip/ 2024-02-19T15:08:30.916 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 42393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/slip/slip.ko 2024-02-19T15:08:30.916 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/sungem_phy.ko 2024-02-19T15:08:30.916 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 64625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/tap.ko 2024-02-19T15:08:30.916 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/team/ 2024-02-19T15:08:30.916 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 107233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/team/team.ko 2024-02-19T15:08:30.917 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/team/team_mode_activebackup.ko 2024-02-19T15:08:30.917 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/team/team_mode_broadcast.ko 2024-02-19T15:08:30.917 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-02-19T15:08:30.917 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/team/team_mode_random.ko 2024-02-19T15:08:30.917 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-02-19T15:08:30.917 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 87105 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/veth.ko 2024-02-19T15:08:30.917 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/vmxnet3/ 2024-02-19T15:08:30.917 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 144361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-02-19T15:08:30.917 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/vxlan/ 2024-02-19T15:08:30.917 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 307257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/vxlan/vxlan.ko 2024-02-19T15:08:30.917 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/xen-netback/ 2024-02-19T15:08:30.917 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 143905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/xen-netback/xen-netback.ko 2024-02-19T15:08:30.917 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ntb/ 2024-02-19T15:08:30.917 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ntb/ntb.ko 2024-02-19T15:08:30.918 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/nvme/ 2024-02-19T15:08:30.918 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/nvme/host/ 2024-02-19T15:08:30.918 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 267745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/nvme/host/nvme-core.ko 2024-02-19T15:08:30.918 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 103361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/nvme/host/nvme.ko 2024-02-19T15:08:30.918 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/parport/ 2024-02-19T15:08:30.918 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 131329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/parport/parport.ko 2024-02-19T15:08:30.918 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14465 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/parport/parport_cs.ko 2024-02-19T15:08:30.918 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 85217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/parport/parport_pc.ko 2024-02-19T15:08:30.918 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 27569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/parport/parport_serial.ko 2024-02-19T15:08:30.918 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pci/ 2024-02-19T15:08:30.918 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pci/hotplug/ 2024-02-19T15:08:30.918 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24913 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-02-19T15:08:30.918 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2024-02-19T15:08:30.918 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2024-02-19T15:08:30.919 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pci/pci-stub.ko 2024-02-19T15:08:30.919 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 46433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pci/xen-pcifront.ko 2024-02-19T15:08:30.919 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pcmcia/ 2024-02-19T15:08:30.919 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pcmcia/i82092.ko 2024-02-19T15:08:30.919 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 146153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pcmcia/pcmcia.ko 2024-02-19T15:08:30.919 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 62081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pcmcia/pcmcia_core.ko 2024-02-19T15:08:30.919 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2024-02-19T15:08:30.919 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pcmcia/pd6729.ko 2024-02-19T15:08:30.919 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 112793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pcmcia/yenta_socket.ko 2024-02-19T15:08:30.919 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/phy/ 2024-02-19T15:08:30.919 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/phy/broadcom/ 2024-02-19T15:08:30.919 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2024-02-19T15:08:30.919 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/phy/samsung/ 2024-02-19T15:08:30.919 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2024-02-19T15:08:30.920 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pinctrl/ 2024-02-19T15:08:30.920 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pinctrl/intel/ 2024-02-19T15:08:30.920 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 69033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-02-19T15:08:30.920 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/ 2024-02-19T15:08:30.920 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/chrome/ 2024-02-19T15:08:30.920 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 51233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2024-02-19T15:08:30.920 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24585 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/chrome/cros_ec.ko 2024-02-19T15:08:30.920 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2024-02-19T15:08:30.920 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2024-02-19T15:08:30.920 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26105 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2024-02-19T15:08:30.920 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2024-02-19T15:08:30.920 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2024-02-19T15:08:30.920 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/ 2024-02-19T15:08:30.920 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 73601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/acer-wmi.ko 2024-02-19T15:08:30.921 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 39057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/acerhdf.ko 2024-02-19T15:08:30.921 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/apple-gmux.ko 2024-02-19T15:08:30.921 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 59569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/asus-laptop.ko 2024-02-19T15:08:30.921 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-02-19T15:08:30.921 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 131945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/asus-wmi.ko 2024-02-19T15:08:30.921 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/classmate-laptop.ko 2024-02-19T15:08:30.921 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 49025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-02-19T15:08:30.921 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-02-19T15:08:30.921 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 40169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-02-19T15:08:30.921 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-02-19T15:08:30.921 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/hdaps.ko 2024-02-19T15:08:30.921 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/ibm_rtl.ko 2024-02-19T15:08:30.921 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/intel/ 2024-02-19T15:08:30.922 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-02-19T15:08:30.922 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2024-02-19T15:08:30.922 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 44601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/intel_ips.ko 2024-02-19T15:08:30.922 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/msi-wmi.ko 2024-02-19T15:08:30.922 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/mxm-wmi.ko 2024-02-19T15:08:30.922 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-02-19T15:08:30.922 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 42233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/samsung-laptop.ko 2024-02-19T15:08:30.922 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/samsung-q10.ko 2024-02-19T15:08:30.922 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/topstar-laptop.ko 2024-02-19T15:08:30.922 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-02-19T15:08:30.922 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/toshiba_haps.ko 2024-02-19T15:08:30.922 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/wmi-bmof.ko 2024-02-19T15:08:30.922 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 48481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/wmi.ko 2024-02-19T15:08:30.922 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/ 2024-02-19T15:08:30.922 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/ 2024-02-19T15:08:30.923 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/88pm860x_battery.ko 2024-02-19T15:08:30.923 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 33889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/88pm860x_charger.ko 2024-02-19T15:08:30.923 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 54961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/bq2415x_charger.ko 2024-02-19T15:08:30.923 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 59505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/bq24190_charger.ko 2024-02-19T15:08:30.923 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/bq24735-charger.ko 2024-02-19T15:08:30.923 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/cros_peripheral_charger.ko 2024-02-19T15:08:30.923 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/da9030_battery.ko 2024-02-19T15:08:30.923 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/da9052-battery.ko 2024-02-19T15:08:30.923 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/ds2760_battery.ko 2024-02-19T15:08:30.923 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/ds2780_battery.ko 2024-02-19T15:08:30.923 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/ds2781_battery.ko 2024-02-19T15:08:30.923 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23553 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/ds2782_battery.ko 2024-02-19T15:08:30.923 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/gpio-charger.ko 2024-02-19T15:08:30.923 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/isp1704_charger.ko 2024-02-19T15:08:30.924 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/lp8727_charger.ko 2024-02-19T15:08:30.924 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/max14577_charger.ko 2024-02-19T15:08:30.924 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/max17042_battery.ko 2024-02-19T15:08:30.924 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/max8903_charger.ko 2024-02-19T15:08:30.924 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/max8925_power.ko 2024-02-19T15:08:30.924 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/pcf50633-charger.ko 2024-02-19T15:08:30.924 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 37841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/sbs-battery.ko 2024-02-19T15:08:30.924 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/test_power.ko 2024-02-19T15:08:30.924 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/tps65090-charger.ko 2024-02-19T15:08:30.924 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/wm831x_backup.ko 2024-02-19T15:08:30.924 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/wm831x_power.ko 2024-02-19T15:08:30.924 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/wm8350_power.ko 2024-02-19T15:08:30.924 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/powercap/ 2024-02-19T15:08:30.924 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 58105 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/powercap/intel_rapl_common.ko 2024-02-19T15:08:30.925 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/powercap/intel_rapl_msr.ko 2024-02-19T15:08:30.925 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pps/ 2024-02-19T15:08:30.925 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pps/clients/ 2024-02-19T15:08:30.925 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pps/clients/pps-gpio.ko 2024-02-19T15:08:30.925 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pps/clients/pps-ldisc.ko 2024-02-19T15:08:30.925 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pps/clients/pps_parport.ko 2024-02-19T15:08:30.925 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 41801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pps/pps_core.ko 2024-02-19T15:08:30.925 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ptp/ 2024-02-19T15:08:30.925 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 70873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ptp/ptp.ko 2024-02-19T15:08:30.925 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ptp/ptp_kvm.ko 2024-02-19T15:08:30.925 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/ 2024-02-19T15:08:30.925 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-88pm80x.ko 2024-02-19T15:08:30.925 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-88pm860x.ko 2024-02-19T15:08:30.925 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-bq32k.ko 2024-02-19T15:08:30.926 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-da9052.ko 2024-02-19T15:08:30.926 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-da9055.ko 2024-02-19T15:08:30.926 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-da9063.ko 2024-02-19T15:08:30.926 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-ds1286.ko 2024-02-19T15:08:30.926 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 68481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-ds1307.ko 2024-02-19T15:08:30.926 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-ds1374.ko 2024-02-19T15:08:30.926 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-ds1511.ko 2024-02-19T15:08:30.926 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-ds1553.ko 2024-02-19T15:08:30.926 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-ds1672.ko 2024-02-19T15:08:30.926 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13273 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-ds1742.ko 2024-02-19T15:08:30.926 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-ds2404.ko 2024-02-19T15:08:30.926 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23609 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-ds3232.ko 2024-02-19T15:08:30.926 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-em3027.ko 2024-02-19T15:08:30.926 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-fm3130.ko 2024-02-19T15:08:30.927 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-isl12022.ko 2024-02-19T15:08:30.927 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 35873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-isl1208.ko 2024-02-19T15:08:30.927 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-lp8788.ko 2024-02-19T15:08:30.927 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 38001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-m41t80.ko 2024-02-19T15:08:30.927 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-m48t35.ko 2024-02-19T15:08:30.927 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-m48t59.ko 2024-02-19T15:08:30.927 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-m48t86.ko 2024-02-19T15:08:30.927 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-max6900.ko 2024-02-19T15:08:30.927 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-max8907.ko 2024-02-19T15:08:30.927 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-max8925.ko 2024-02-19T15:08:30.927 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-max8997.ko 2024-02-19T15:08:30.927 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16585 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-max8998.ko 2024-02-19T15:08:30.927 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-mc13xxx.ko 2024-02-19T15:08:30.927 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-msm6242.ko 2024-02-19T15:08:30.927 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-palmas.ko 2024-02-19T15:08:30.928 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 46537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-pcf2127.ko 2024-02-19T15:08:30.928 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-pcf50633.ko 2024-02-19T15:08:30.928 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-pcf85063.ko 2024-02-19T15:08:30.928 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-pcf8523.ko 2024-02-19T15:08:30.928 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-pcf8563.ko 2024-02-19T15:08:30.928 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-pcf8583.ko 2024-02-19T15:08:30.928 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-rc5t583.ko 2024-02-19T15:08:30.928 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-rp5c01.ko 2024-02-19T15:08:30.928 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 33657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-rs5c372.ko 2024-02-19T15:08:30.928 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 27737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-02-19T15:08:30.928 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-rx8025.ko 2024-02-19T15:08:30.928 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-rx8581.ko 2024-02-19T15:08:30.928 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-s35390a.ko 2024-02-19T15:08:30.929 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-02-19T15:08:30.929 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-tps6586x.ko 2024-02-19T15:08:30.929 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-tps65910.ko 2024-02-19T15:08:30.929 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-wm831x.ko 2024-02-19T15:08:30.929 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-wm8350.ko 2024-02-19T15:08:30.929 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 27777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-x1205.ko 2024-02-19T15:08:30.929 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/ 2024-02-19T15:08:30.929 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 77529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/3w-9xxx.ko 2024-02-19T15:08:30.929 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 57721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/3w-sas.ko 2024-02-19T15:08:30.929 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 55081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/3w-xxxx.ko 2024-02-19T15:08:30.929 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 135345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/BusLogic.ko 2024-02-19T15:08:30.929 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/a100u2w.ko 2024-02-19T15:08:30.929 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/aacraid/ 2024-02-19T15:08:30.929 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 271441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/aacraid/aacraid.ko 2024-02-19T15:08:30.929 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 100473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/advansys.ko 2024-02-19T15:08:30.930 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/aic7xxx/ 2024-02-19T15:08:30.930 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 356865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-02-19T15:08:30.930 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 295993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-02-19T15:08:30.930 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/aic94xx/ 2024-02-19T15:08:30.930 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 189145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/aic94xx/aic94xx.ko 2024-02-19T15:08:30.930 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/am53c974.ko 2024-02-19T15:08:30.930 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/arcmsr/ 2024-02-19T15:08:30.930 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 108273 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-02-19T15:08:30.930 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 38097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/atp870u.ko 2024-02-19T15:08:30.930 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/be2iscsi/ 2024-02-19T15:08:30.930 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 246937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-02-19T15:08:30.930 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/bfa/ 2024-02-19T15:08:30.930 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 1171953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/bfa/bfa.ko 2024-02-19T15:08:30.930 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/bnx2fc/ 2024-02-19T15:08:30.930 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 238401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-02-19T15:08:30.931 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/bnx2i/ 2024-02-19T15:08:30.931 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 115329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-02-19T15:08:30.931 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 40785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/ch.ko 2024-02-19T15:08:30.931 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/csiostor/ 2024-02-19T15:08:30.931 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 308961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/csiostor/csiostor.ko 2024-02-19T15:08:30.931 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/cxgbi/ 2024-02-19T15:08:30.931 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/cxgbi/cxgb3i/ 2024-02-19T15:08:30.931 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 76041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-02-19T15:08:30.931 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/cxgbi/cxgb4i/ 2024-02-19T15:08:30.931 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 101425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-02-19T15:08:30.931 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 146817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-02-19T15:08:30.931 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 75977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/dc395x.ko 2024-02-19T15:08:30.932 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 38209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/dmx3191d.ko 2024-02-19T15:08:30.932 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/esas2r/ 2024-02-19T15:08:30.932 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 212385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/esas2r/esas2r.ko 2024-02-19T15:08:30.932 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 60505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/esp_scsi.ko 2024-02-19T15:08:30.932 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/fcoe/ 2024-02-19T15:08:30.932 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 78225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/fcoe/fcoe.ko 2024-02-19T15:08:30.932 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 157961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-02-19T15:08:30.932 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/fdomain.ko 2024-02-19T15:08:30.932 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/fnic/ 2024-02-19T15:08:30.932 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 294601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/fnic/fnic.ko 2024-02-19T15:08:30.932 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 201729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/hpsa.ko 2024-02-19T15:08:30.932 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 51121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/hptiop.ko 2024-02-19T15:08:30.932 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 49017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/hv_storvsc.ko 2024-02-19T15:08:30.932 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36129 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/imm.ko 2024-02-19T15:08:30.933 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 43977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/initio.ko 2024-02-19T15:08:30.933 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 265945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/ipr.ko 2024-02-19T15:08:30.933 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 101881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/ips.ko 2024-02-19T15:08:30.933 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/isci/ 2024-02-19T15:08:30.933 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 317193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/isci/isci.ko 2024-02-19T15:08:30.933 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-02-19T15:08:30.933 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 52865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/iscsi_tcp.ko 2024-02-19T15:08:30.933 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/libfc/ 2024-02-19T15:08:30.933 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 315385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/libfc/libfc.ko 2024-02-19T15:08:30.933 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 145497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/libiscsi.ko 2024-02-19T15:08:30.933 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 53809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/libiscsi_tcp.ko 2024-02-19T15:08:30.933 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/libsas/ 2024-02-19T15:08:30.933 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 216401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/libsas/libsas.ko 2024-02-19T15:08:30.933 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/lpfc/ 2024-02-19T15:08:30.934 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 2194089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/lpfc/lpfc.ko 2024-02-19T15:08:30.934 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/megaraid/ 2024-02-19T15:08:30.934 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 73473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-02-19T15:08:30.934 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 33433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-02-19T15:08:30.934 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 310689 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-02-19T15:08:30.934 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 85521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/megaraid.ko 2024-02-19T15:08:30.934 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/mpt3sas/ 2024-02-19T15:08:30.934 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 659569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-02-19T15:08:30.934 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/mvsas/ 2024-02-19T15:08:30.934 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 119489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/mvsas/mvsas.ko 2024-02-19T15:08:30.934 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 78041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/mvumi.ko 2024-02-19T15:08:30.934 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/pcmcia/ 2024-02-19T15:08:30.934 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 81561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2024-02-19T15:08:30.934 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2024-02-19T15:08:30.935 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2024-02-19T15:08:30.935 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2024-02-19T15:08:30.935 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/pm8001/ 2024-02-19T15:08:30.935 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 366337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-02-19T15:08:30.935 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 124601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/pmcraid.ko 2024-02-19T15:08:30.935 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/ppa.ko 2024-02-19T15:08:30.935 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 55337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/qla1280.ko 2024-02-19T15:08:30.935 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/qla2xxx/ 2024-02-19T15:08:30.935 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 1876753 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-02-19T15:08:30.935 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 85193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-02-19T15:08:30.935 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/qla4xxx/ 2024-02-19T15:08:30.935 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 558953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-02-19T15:08:30.935 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/qlogicfas408.ko 2024-02-19T15:08:30.935 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/raid_class.ko 2024-02-19T15:08:30.935 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 261081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/scsi_debug.ko 2024-02-19T15:08:30.936 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 173929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/scsi_transport_fc.ko 2024-02-19T15:08:30.936 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 280113 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-02-19T15:08:30.936 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 97625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/scsi_transport_sas.ko 2024-02-19T15:08:30.936 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 73865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/scsi_transport_spi.ko 2024-02-19T15:08:30.936 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 44889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/scsi_transport_srp.ko 2024-02-19T15:08:30.936 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/ses.ko 2024-02-19T15:08:30.936 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 115713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/st.ko 2024-02-19T15:08:30.936 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 46737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/stex.ko 2024-02-19T15:08:30.936 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/sym53c8xx_2/ 2024-02-19T15:08:30.936 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 164889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-02-19T15:08:30.936 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 44857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/virtio_scsi.ko 2024-02-19T15:08:30.936 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 50233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/vmw_pvscsi.ko 2024-02-19T15:08:30.936 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 39881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/wd719x.ko 2024-02-19T15:08:30.936 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 49929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/xen-scsifront.ko 2024-02-19T15:08:30.937 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ssb/ 2024-02-19T15:08:30.937 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 129393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ssb/ssb.ko 2024-02-19T15:08:30.937 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/ 2024-02-19T15:08:30.937 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/iscsi/ 2024-02-19T15:08:30.937 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 736761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-02-19T15:08:30.937 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/loopback/ 2024-02-19T15:08:30.937 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 49881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/loopback/tcm_loop.ko 2024-02-19T15:08:30.937 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 39617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/target_core_file.ko 2024-02-19T15:08:30.937 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 42873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/target_core_iblock.ko 2024-02-19T15:08:30.937 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 930977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/target_core_mod.ko 2024-02-19T15:08:30.937 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 44137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/target_core_pscsi.ko 2024-02-19T15:08:30.937 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 122489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/target_core_user.ko 2024-02-19T15:08:30.937 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/tcm_fc/ 2024-02-19T15:08:30.937 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 75521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-02-19T15:08:30.937 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/ 2024-02-19T15:08:30.938 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/ 2024-02-19T15:08:30.938 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/ 2024-02-19T15:08:30.938 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-02-19T15:08:30.938 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-02-19T15:08:30.938 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-02-19T15:08:30.938 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-02-19T15:08:30.938 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-02-19T15:08:30.938 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-02-19T15:08:30.938 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-02-19T15:08:30.938 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-02-19T15:08:30.938 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-02-19T15:08:30.938 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-02-19T15:08:30.938 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10753 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-02-19T15:08:30.938 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-02-19T15:08:30.938 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-02-19T15:08:30.939 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-02-19T15:08:30.939 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11465 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-02-19T15:08:30.939 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/intel_powerclamp.ko 2024-02-19T15:08:30.939 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-02-19T15:08:30.939 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-02-19T15:08:30.939 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-02-19T15:08:30.939 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/ 2024-02-19T15:08:30.939 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/ipwireless/ 2024-02-19T15:08:30.939 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 78353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-02-19T15:08:30.939 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 57777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/moxa.ko 2024-02-19T15:08:30.939 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 50865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/mxser.ko 2024-02-19T15:08:30.939 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 108489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/n_gsm.ko 2024-02-19T15:08:30.939 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/n_hdlc.ko 2024-02-19T15:08:30.939 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 67841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/nozomi.ko 2024-02-19T15:08:30.940 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/ 2024-02-19T15:08:30.940 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/8250/ 2024-02-19T15:08:30.940 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/8250/8250_dw.ko 2024-02-19T15:08:30.940 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 59401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-02-19T15:08:30.940 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/altera_jtaguart.ko 2024-02-19T15:08:30.940 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/altera_uart.ko 2024-02-19T15:08:30.940 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/arc_uart.ko 2024-02-19T15:08:30.940 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 72489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/fsl_lpuart.ko 2024-02-19T15:08:30.940 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/jsm/ 2024-02-19T15:08:30.940 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 101993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/jsm/jsm.ko 2024-02-19T15:08:30.940 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/rp2.ko 2024-02-19T15:08:30.940 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 53177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/sc16is7xx.ko 2024-02-19T15:08:30.940 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ufs/ 2024-02-19T15:08:30.941 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ufs/core/ 2024-02-19T15:08:30.941 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 545401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ufs/core/ufshcd-core.ko 2024-02-19T15:08:30.941 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ufs/host/ 2024-02-19T15:08:30.941 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-02-19T15:08:30.941 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2024-02-19T15:08:30.941 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/uio/ 2024-02-19T15:08:30.941 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 44473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/uio/uio.ko 2024-02-19T15:08:30.941 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/uio/uio_aec.ko 2024-02-19T15:08:30.941 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/uio/uio_cif.ko 2024-02-19T15:08:30.941 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/uio/uio_dmem_genirq.ko 2024-02-19T15:08:30.941 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/uio/uio_mf624.ko 2024-02-19T15:08:30.941 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/uio/uio_netx.ko 2024-02-19T15:08:30.941 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/uio/uio_pci_generic.ko 2024-02-19T15:08:30.941 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/uio/uio_pdrv_genirq.ko 2024-02-19T15:08:30.941 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/uio/uio_sercos3.ko 2024-02-19T15:08:30.942 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/ 2024-02-19T15:08:30.942 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/c67x00/ 2024-02-19T15:08:30.942 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 73225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/c67x00/c67x00.ko 2024-02-19T15:08:30.942 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/chipidea/ 2024-02-19T15:08:30.942 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 145793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/chipidea/ci_hdrc.ko 2024-02-19T15:08:30.942 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2024-02-19T15:08:30.942 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/chipidea/ci_hdrc_npcm.ko 2024-02-19T15:08:30.942 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2024-02-19T15:08:30.942 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2024-02-19T15:08:30.942 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/class/ 2024-02-19T15:08:30.942 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 87009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/class/cdc-acm.ko 2024-02-19T15:08:30.942 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 51321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/class/cdc-wdm.ko 2024-02-19T15:08:30.942 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 47217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/class/usblp.ko 2024-02-19T15:08:30.943 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 89769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/class/usbtmc.ko 2024-02-19T15:08:30.943 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/common/ 2024-02-19T15:08:30.943 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/common/ulpi.ko 2024-02-19T15:08:30.943 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/dwc2/ 2024-02-19T15:08:30.943 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/dwc2/dwc2_pci.ko 2024-02-19T15:08:30.943 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/dwc3/ 2024-02-19T15:08:30.943 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/dwc3/dwc3-haps.ko 2024-02-19T15:08:30.943 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/dwc3/dwc3-pci.ko 2024-02-19T15:08:30.943 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 452577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/dwc3/dwc3.ko 2024-02-19T15:08:30.943 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/ 2024-02-19T15:08:30.943 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/ 2024-02-19T15:08:30.943 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 60329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/u_ether.ko 2024-02-19T15:08:30.943 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 51793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/u_serial.ko 2024-02-19T15:08:30.943 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2024-02-19T15:08:30.944 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 42353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2024-02-19T15:08:30.944 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 27857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2024-02-19T15:08:30.944 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2024-02-19T15:08:30.944 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 92409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2024-02-19T15:08:30.944 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 50945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2024-02-19T15:08:30.944 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 122257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2024-02-19T15:08:30.944 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 59801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2024-02-19T15:08:30.944 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2024-02-19T15:08:30.944 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2024-02-19T15:08:30.944 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 57633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2024-02-19T15:08:30.944 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 85521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2024-02-19T15:08:30.944 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2024-02-19T15:08:30.944 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 58025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2024-02-19T15:08:30.944 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 67353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2024-02-19T15:08:30.944 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/ 2024-02-19T15:08:30.945 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2024-02-19T15:08:30.945 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2024-02-19T15:08:30.945 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2024-02-19T15:08:30.945 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_ether.ko 2024-02-19T15:08:30.945 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2024-02-19T15:08:30.945 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18921 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_hid.ko 2024-02-19T15:08:30.945 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2024-02-19T15:08:30.945 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2024-02-19T15:08:30.945 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2024-02-19T15:08:30.945 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_printer.ko 2024-02-19T15:08:30.945 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_serial.ko 2024-02-19T15:08:30.945 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 27113 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_zero.ko 2024-02-19T15:08:30.945 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 62369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2024-02-19T15:08:30.945 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15105 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2024-02-19T15:08:30.945 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 161233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/libcomposite.ko 2024-02-19T15:08:30.946 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/ 2024-02-19T15:08:30.946 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17609 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2024-02-19T15:08:30.946 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/bdc/ 2024-02-19T15:08:30.946 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 157617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2024-02-19T15:08:30.946 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/goku_udc.ko 2024-02-19T15:08:30.946 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 52929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/gr_udc.ko 2024-02-19T15:08:30.946 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 69617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2024-02-19T15:08:30.946 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 55953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/mv_udc.ko 2024-02-19T15:08:30.946 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 75969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/net2272.ko 2024-02-19T15:08:30.946 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 74985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/net2280.ko 2024-02-19T15:08:30.946 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 73017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/pch_udc.ko 2024-02-19T15:08:30.946 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 65881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2024-02-19T15:08:30.946 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 66001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2024-02-19T15:08:30.946 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 81409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2024-02-19T15:08:30.946 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 156337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/udc-core.ko 2024-02-19T15:08:30.947 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/host/ 2024-02-19T15:08:30.947 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/host/bcma-hcd.ko 2024-02-19T15:08:30.947 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 80017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/host/isp116x-hcd.ko 2024-02-19T15:08:30.947 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 87193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/host/oxu210hp-hcd.ko 2024-02-19T15:08:30.947 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 78777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/host/r8a66597-hcd.ko 2024-02-19T15:08:30.947 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 51393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/host/sl811-hcd.ko 2024-02-19T15:08:30.947 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/host/sl811_cs.ko 2024-02-19T15:08:30.947 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/host/ssb-hcd.ko 2024-02-19T15:08:30.947 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/host/xhci-plat-hcd.ko 2024-02-19T15:08:30.947 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/image/ 2024-02-19T15:08:30.947 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/image/mdc800.ko 2024-02-19T15:08:30.947 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/image/microtek.ko 2024-02-19T15:08:30.947 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/ 2024-02-19T15:08:30.947 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 41633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/adutux.ko 2024-02-19T15:08:30.947 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/appledisplay.ko 2024-02-19T15:08:30.947 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/cypress_cy7c63.ko 2024-02-19T15:08:30.948 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/cytherm.ko 2024-02-19T15:08:30.948 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/ehset.ko 2024-02-19T15:08:30.948 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/emi26.ko 2024-02-19T15:08:30.948 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/emi62.ko 2024-02-19T15:08:30.948 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/ezusb.ko 2024-02-19T15:08:30.948 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/idmouse.ko 2024-02-19T15:08:30.948 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/iowarrior.ko 2024-02-19T15:08:30.948 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/isight_firmware.ko 2024-02-19T15:08:30.948 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/ldusb.ko 2024-02-19T15:08:30.948 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/legousbtower.ko 2024-02-19T15:08:30.948 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 27705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/lvstest.ko 2024-02-19T15:08:30.948 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/sisusbvga/ 2024-02-19T15:08:30.948 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 64745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-02-19T15:08:30.948 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/trancevibrator.ko 2024-02-19T15:08:30.949 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/usb3503.ko 2024-02-19T15:08:30.949 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/usblcd.ko 2024-02-19T15:08:30.949 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/usbsevseg.ko 2024-02-19T15:08:30.949 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 70721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/usbtest.ko 2024-02-19T15:08:30.949 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 41817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/uss720.ko 2024-02-19T15:08:30.949 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 27225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/yurex.ko 2024-02-19T15:08:30.949 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/mon/ 2024-02-19T15:08:30.949 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 72825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/mon/usbmon.ko 2024-02-19T15:08:30.949 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/musb/ 2024-02-19T15:08:30.949 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 301633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/musb/musb_hdrc.ko 2024-02-19T15:08:30.949 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/phy/ 2024-02-19T15:08:30.949 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/phy/phy-generic.ko 2024-02-19T15:08:30.949 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2024-02-19T15:08:30.949 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/phy/phy-isp1301.ko 2024-02-19T15:08:30.949 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/phy/phy-tahvo.ko 2024-02-19T15:08:30.949 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/ 2024-02-19T15:08:30.950 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/aircable.ko 2024-02-19T15:08:30.950 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 33241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/ark3116.ko 2024-02-19T15:08:30.950 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/belkin_sa.ko 2024-02-19T15:08:30.950 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 38017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/ch341.ko 2024-02-19T15:08:30.950 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 70345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/cp210x.ko 2024-02-19T15:08:30.950 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/cyberjack.ko 2024-02-19T15:08:30.950 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 58121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/cypress_m8.ko 2024-02-19T15:08:30.950 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 49721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/digi_acceleport.ko 2024-02-19T15:08:30.950 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/empeg.ko 2024-02-19T15:08:30.950 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 44185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/f81232.ko 2024-02-19T15:08:30.950 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 175105 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/ftdi_sio.ko 2024-02-19T15:08:30.950 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 53017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/garmin_gps.ko 2024-02-19T15:08:30.950 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 136985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/io_edgeport.ko 2024-02-19T15:08:30.950 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 124745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/io_ti.ko 2024-02-19T15:08:30.950 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 67681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/ipaq.ko 2024-02-19T15:08:30.950 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/ipw.ko 2024-02-19T15:08:30.951 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/ir-usb.ko 2024-02-19T15:08:30.951 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 58137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-02-19T15:08:30.951 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 107433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/keyspan.ko 2024-02-19T15:08:30.951 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 35521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/keyspan_pda.ko 2024-02-19T15:08:30.951 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/kl5kusb105.ko 2024-02-19T15:08:30.951 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 33593 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/kobil_sct.ko 2024-02-19T15:08:30.951 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 32865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/mct_u232.ko 2024-02-19T15:08:30.951 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/metro-usb.ko 2024-02-19T15:08:30.951 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 66361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/mos7720.ko 2024-02-19T15:08:30.951 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 78129 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/mos7840.ko 2024-02-19T15:08:30.951 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 56249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/mxuport.ko 2024-02-19T15:08:30.951 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/navman.ko 2024-02-19T15:08:30.951 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11921 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/omninet.ko 2024-02-19T15:08:30.952 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/opticon.ko 2024-02-19T15:08:30.952 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 191825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/option.ko 2024-02-19T15:08:30.952 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 38985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/oti6858.ko 2024-02-19T15:08:30.952 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 54265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/pl2303.ko 2024-02-19T15:08:30.952 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/qcaux.ko 2024-02-19T15:08:30.952 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 38025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/qcserial.ko 2024-02-19T15:08:30.952 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/quatech2.ko 2024-02-19T15:08:30.952 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/safe_serial.ko 2024-02-19T15:08:30.952 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 51537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/sierra.ko 2024-02-19T15:08:30.952 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21585 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/spcp8x5.ko 2024-02-19T15:08:30.952 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/ssu100.ko 2024-02-19T15:08:30.952 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/symbolserial.ko 2024-02-19T15:08:30.952 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 61393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-02-19T15:08:30.953 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-02-19T15:08:30.953 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/usb_debug.ko 2024-02-19T15:08:30.953 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 33113 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/usb_wwan.ko 2024-02-19T15:08:30.953 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 125529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/usbserial.ko 2024-02-19T15:08:30.953 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/visor.ko 2024-02-19T15:08:30.953 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 37313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/whiteheat.ko 2024-02-19T15:08:30.953 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/wishbone-serial.ko 2024-02-19T15:08:30.953 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/xsens_mt.ko 2024-02-19T15:08:30.953 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ 2024-02-19T15:08:30.953 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 52761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/uas.ko 2024-02-19T15:08:30.953 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-alauda.ko 2024-02-19T15:08:30.953 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-cypress.ko 2024-02-19T15:08:30.953 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-datafab.ko 2024-02-19T15:08:30.953 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 33401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-02-19T15:08:30.954 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-freecom.ko 2024-02-19T15:08:30.954 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-isd200.ko 2024-02-19T15:08:30.954 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-02-19T15:08:30.954 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-karma.ko 2024-02-19T15:08:30.954 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-onetouch.ko 2024-02-19T15:08:30.954 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-realtek.ko 2024-02-19T15:08:30.954 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-sddr09.ko 2024-02-19T15:08:30.954 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-sddr55.ko 2024-02-19T15:08:30.954 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 35249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-usbat.ko 2024-02-19T15:08:30.954 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 180329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/usb-storage.ko 2024-02-19T15:08:30.954 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/usbip/ 2024-02-19T15:08:30.954 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 67401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/usbip/usbip-core.ko 2024-02-19T15:08:30.954 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 71025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/usbip/usbip-host.ko 2024-02-19T15:08:30.954 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 104065 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-02-19T15:08:30.954 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vfio/ 2024-02-19T15:08:30.955 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vfio/pci/ 2024-02-19T15:08:30.955 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 162865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-02-19T15:08:30.955 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vfio/pci/vfio-pci.ko 2024-02-19T15:08:30.955 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 103673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vfio/vfio.ko 2024-02-19T15:08:30.955 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 76441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-02-19T15:08:30.955 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vhost/ 2024-02-19T15:08:30.955 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 113665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vhost/vhost.ko 2024-02-19T15:08:30.955 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16913 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vhost/vhost_iotlb.ko 2024-02-19T15:08:30.955 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 58001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vhost/vhost_net.ko 2024-02-19T15:08:30.955 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 89057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vhost/vhost_scsi.ko 2024-02-19T15:08:30.955 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/ 2024-02-19T15:08:30.955 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/ 2024-02-19T15:08:30.955 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/88pm860x_bl.ko 2024-02-19T15:08:30.955 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/aat2870_bl.ko 2024-02-19T15:08:30.955 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/adp5520_bl.ko 2024-02-19T15:08:30.955 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 32513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/adp8860_bl.ko 2024-02-19T15:08:30.956 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 39897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/adp8870_bl.ko 2024-02-19T15:08:30.956 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/apple_bl.ko 2024-02-19T15:08:30.956 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/as3711_bl.ko 2024-02-19T15:08:30.956 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/bd6107.ko 2024-02-19T15:08:30.956 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/da903x_bl.ko 2024-02-19T15:08:30.956 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/da9052_bl.ko 2024-02-19T15:08:30.956 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/gpio_backlight.ko 2024-02-19T15:08:30.956 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/kb3886_bl.ko 2024-02-19T15:08:30.956 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/lcd.ko 2024-02-19T15:08:30.956 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/lm3533_bl.ko 2024-02-19T15:08:30.956 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/lm3639_bl.ko 2024-02-19T15:08:30.956 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/lv5207lp.ko 2024-02-19T15:08:30.956 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/max8925_bl.ko 2024-02-19T15:08:30.956 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/pandora_bl.ko 2024-02-19T15:08:30.956 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/pcf50633-backlight.ko 2024-02-19T15:08:30.957 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/platform_lcd.ko 2024-02-19T15:08:30.957 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/wm831x_bl.ko 2024-02-19T15:08:30.957 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/ 2024-02-19T15:08:30.957 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/masters/ 2024-02-19T15:08:30.957 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/masters/ds2482.ko 2024-02-19T15:08:30.957 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 39121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/masters/ds2490.ko 2024-02-19T15:08:30.957 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/masters/matrox_w1.ko 2024-02-19T15:08:30.957 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11273 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/masters/w1-gpio.ko 2024-02-19T15:08:30.957 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/ 2024-02-19T15:08:30.957 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-02-19T15:08:30.957 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-02-19T15:08:30.957 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-02-19T15:08:30.958 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-02-19T15:08:30.958 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-02-19T15:08:30.958 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-02-19T15:08:30.958 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-02-19T15:08:30.958 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-02-19T15:08:30.958 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-02-19T15:08:30.958 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_smem.ko 2024-02-19T15:08:30.958 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 56337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_therm.ko 2024-02-19T15:08:30.958 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 103049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/wire.ko 2024-02-19T15:08:30.958 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/ 2024-02-19T15:08:30.958 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/acquirewdt.ko 2024-02-19T15:08:30.958 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/advantechwdt.ko 2024-02-19T15:08:30.958 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18465 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/alim1535_wdt.ko 2024-02-19T15:08:30.958 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/alim7101_wdt.ko 2024-02-19T15:08:30.959 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/cpu5wdt.ko 2024-02-19T15:08:30.959 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/da9052_wdt.ko 2024-02-19T15:08:30.959 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/da9055_wdt.ko 2024-02-19T15:08:30.959 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/da9063_wdt.ko 2024-02-19T15:08:30.959 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/dw_wdt.ko 2024-02-19T15:08:30.959 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/eurotechwdt.ko 2024-02-19T15:08:30.959 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/f71808e_wdt.ko 2024-02-19T15:08:30.959 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 27393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/hpwdt.ko 2024-02-19T15:08:30.959 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/i6300esb.ko 2024-02-19T15:08:30.959 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-02-19T15:08:30.959 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/iTCO_wdt.ko 2024-02-19T15:08:30.959 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/ib700wdt.ko 2024-02-19T15:08:30.959 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/ibmasr.ko 2024-02-19T15:08:30.959 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-02-19T15:08:30.959 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21129 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/it8712f_wdt.ko 2024-02-19T15:08:30.960 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/it87_wdt.ko 2024-02-19T15:08:30.960 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/kempld_wdt.ko 2024-02-19T15:08:30.960 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/machzwd.ko 2024-02-19T15:08:30.960 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/mena21_wdt.ko 2024-02-19T15:08:30.960 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/menf21bmc_wdt.ko 2024-02-19T15:08:30.960 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24585 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/nv_tco.ko 2024-02-19T15:08:30.960 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/of_xilinx_wdt.ko 2024-02-19T15:08:30.960 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/pc87413_wdt.ko 2024-02-19T15:08:30.960 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 39921 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/pcwd_pci.ko 2024-02-19T15:08:30.960 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/pcwd_usb.ko 2024-02-19T15:08:30.960 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/retu_wdt.ko 2024-02-19T15:08:30.960 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/sbc60xxwdt.ko 2024-02-19T15:08:30.960 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/sbc_epx_c3.ko 2024-02-19T15:08:30.960 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-02-19T15:08:30.961 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/sc1200wdt.ko 2024-02-19T15:08:30.961 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/sch311x_wdt.ko 2024-02-19T15:08:30.961 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/smsc37b787_wdt.ko 2024-02-19T15:08:30.961 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/softdog.ko 2024-02-19T15:08:30.961 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/sp5100_tco.ko 2024-02-19T15:08:30.961 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/twl4030_wdt.ko 2024-02-19T15:08:30.961 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/via_wdt.ko 2024-02-19T15:08:30.961 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-02-19T15:08:30.961 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/w83877f_wdt.ko 2024-02-19T15:08:30.961 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/w83977f_wdt.ko 2024-02-19T15:08:30.961 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/wafer5823wdt.ko 2024-02-19T15:08:30.961 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/wdt_pci.ko 2024-02-19T15:08:30.961 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/wm831x_wdt.ko 2024-02-19T15:08:30.962 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13689 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/wm8350_wdt.ko 2024-02-19T15:08:30.962 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15609 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/xen_wdt.ko 2024-02-19T15:08:30.962 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/xen/ 2024-02-19T15:08:30.962 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/xen/xen-evtchn.ko 2024-02-19T15:08:30.962 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/xen/xen-gntalloc.ko 2024-02-19T15:08:30.962 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 47769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/xen/xen-gntdev.ko 2024-02-19T15:08:30.962 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/xen/xen-pciback/ 2024-02-19T15:08:30.962 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 176385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-02-19T15:08:30.962 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 40097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/xen/xen-privcmd.ko 2024-02-19T15:08:30.962 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 71153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/xen/xen-scsiback.ko 2024-02-19T15:08:30.962 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/xen/xenfs/ 2024-02-19T15:08:30.962 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/xen/xenfs/xenfs.ko 2024-02-19T15:08:30.962 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ 2024-02-19T15:08:30.963 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/9p/ 2024-02-19T15:08:30.963 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 158281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/9p/9p.ko 2024-02-19T15:08:30.963 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/adfs/ 2024-02-19T15:08:30.963 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 61721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/adfs/adfs.ko 2024-02-19T15:08:30.963 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/affs/ 2024-02-19T15:08:30.963 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 163377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/affs/affs.ko 2024-02-19T15:08:30.963 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/afs/ 2024-02-19T15:08:30.963 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 1018985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/afs/kafs.ko 2024-02-19T15:08:30.963 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/autofs/ 2024-02-19T15:08:30.963 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 103225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/autofs/autofs4.ko 2024-02-19T15:08:30.963 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/befs/ 2024-02-19T15:08:30.963 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 64625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/befs/befs.ko 2024-02-19T15:08:30.963 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/bfs/ 2024-02-19T15:08:30.963 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 41017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/bfs/bfs.ko 2024-02-19T15:08:30.963 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 38649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/binfmt_misc.ko 2024-02-19T15:08:30.964 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/btrfs/ 2024-02-19T15:08:30.964 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 3484777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/btrfs/btrfs.ko 2024-02-19T15:08:30.964 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/cachefiles/ 2024-02-19T15:08:30.964 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 305737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/cachefiles/cachefiles.ko 2024-02-19T15:08:30.964 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ceph/ 2024-02-19T15:08:30.964 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 1335465 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ceph/ceph.ko 2024-02-19T15:08:30.964 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/coda/ 2024-02-19T15:08:30.964 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 97769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/coda/coda.ko 2024-02-19T15:08:30.964 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/configfs/ 2024-02-19T15:08:30.964 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 108577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/configfs/configfs.ko 2024-02-19T15:08:30.964 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/cramfs/ 2024-02-19T15:08:30.964 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29593 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/cramfs/cramfs.ko 2024-02-19T15:08:30.964 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/dlm/ 2024-02-19T15:08:30.964 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 644249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/dlm/dlm.ko 2024-02-19T15:08:30.965 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/efs/ 2024-02-19T15:08:30.965 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/efs/efs.ko 2024-02-19T15:08:30.965 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/f2fs/ 2024-02-19T15:08:30.965 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 1527729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/f2fs/f2fs.ko 2024-02-19T15:08:30.965 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/fat/ 2024-02-19T15:08:30.965 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 27841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/fat/msdos.ko 2024-02-19T15:08:30.965 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/freevxfs/ 2024-02-19T15:08:30.965 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 42561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/freevxfs/freevxfs.ko 2024-02-19T15:08:30.965 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/fuse/ 2024-02-19T15:08:30.965 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/fuse/cuse.ko 2024-02-19T15:08:30.965 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/gfs2/ 2024-02-19T15:08:30.965 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 1231361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/gfs2/gfs2.ko 2024-02-19T15:08:30.965 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/hfs/ 2024-02-19T15:08:30.965 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 141713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/hfs/hfs.ko 2024-02-19T15:08:30.965 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/hfsplus/ 2024-02-19T15:08:30.965 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 222897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/hfsplus/hfsplus.ko 2024-02-19T15:08:30.966 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/hpfs/ 2024-02-19T15:08:30.966 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 193545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/hpfs/hpfs.ko 2024-02-19T15:08:30.966 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/isofs/ 2024-02-19T15:08:30.966 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 74737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/isofs/isofs.ko 2024-02-19T15:08:30.966 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/jfs/ 2024-02-19T15:08:30.966 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 420449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/jfs/jfs.ko 2024-02-19T15:08:30.966 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/lockd/ 2024-02-19T15:08:30.966 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 281705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/lockd/lockd.ko 2024-02-19T15:08:30.966 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/minix/ 2024-02-19T15:08:30.966 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 88937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/minix/minix.ko 2024-02-19T15:08:30.966 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/netfs/ 2024-02-19T15:08:30.966 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 490001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/netfs/netfs.ko 2024-02-19T15:08:30.966 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/ 2024-02-19T15:08:30.966 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/blocklayout/ 2024-02-19T15:08:30.966 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 81457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-02-19T15:08:30.967 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/filelayout/ 2024-02-19T15:08:30.967 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 59353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-02-19T15:08:30.967 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/flexfilelayout/ 2024-02-19T15:08:30.967 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 117377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-02-19T15:08:30.967 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 1057985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/nfs.ko 2024-02-19T15:08:30.967 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 66361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/nfsv2.ko 2024-02-19T15:08:30.967 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 120193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/nfsv3.ko 2024-02-19T15:08:30.967 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 2041825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/nfsv4.ko 2024-02-19T15:08:30.967 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs_common/ 2024-02-19T15:08:30.967 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs_common/grace.ko 2024-02-19T15:08:30.967 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs_common/nfs_acl.ko 2024-02-19T15:08:30.967 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfsd/ 2024-02-19T15:08:30.967 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 1511665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfsd/nfsd.ko 2024-02-19T15:08:30.967 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nilfs2/ 2024-02-19T15:08:30.968 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 582529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nilfs2/nilfs2.ko 2024-02-19T15:08:30.968 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/ 2024-02-19T15:08:30.968 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-celtic.ko 2024-02-19T15:08:30.968 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-centeuro.ko 2024-02-19T15:08:30.968 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-croatian.ko 2024-02-19T15:08:30.968 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-cyrillic.ko 2024-02-19T15:08:30.968 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-gaelic.ko 2024-02-19T15:08:30.968 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-greek.ko 2024-02-19T15:08:30.968 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-iceland.ko 2024-02-19T15:08:30.968 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-inuit.ko 2024-02-19T15:08:30.968 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-roman.ko 2024-02-19T15:08:30.968 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-romanian.ko 2024-02-19T15:08:30.968 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-turkish.ko 2024-02-19T15:08:30.968 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_ascii.ko 2024-02-19T15:08:30.968 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp1250.ko 2024-02-19T15:08:30.968 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp1251.ko 2024-02-19T15:08:30.969 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp1255.ko 2024-02-19T15:08:30.969 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp737.ko 2024-02-19T15:08:30.969 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp775.ko 2024-02-19T15:08:30.969 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12129 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp850.ko 2024-02-19T15:08:30.969 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12129 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp852.ko 2024-02-19T15:08:30.969 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12129 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp855.ko 2024-02-19T15:08:30.969 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp857.ko 2024-02-19T15:08:30.969 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12753 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp860.ko 2024-02-19T15:08:30.969 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13065 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp861.ko 2024-02-19T15:08:30.969 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp862.ko 2024-02-19T15:08:30.969 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13065 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp863.ko 2024-02-19T15:08:30.969 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12753 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp864.ko 2024-02-19T15:08:30.969 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13065 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp865.ko 2024-02-19T15:08:30.969 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp866.ko 2024-02-19T15:08:30.969 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12129 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp869.ko 2024-02-19T15:08:30.969 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp874.ko 2024-02-19T15:08:30.970 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 94041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp932.ko 2024-02-19T15:08:30.970 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 142577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp936.ko 2024-02-19T15:08:30.970 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 165649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp949.ko 2024-02-19T15:08:30.970 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 118025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp950.ko 2024-02-19T15:08:30.970 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_euc-jp.ko 2024-02-19T15:08:30.970 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-1.ko 2024-02-19T15:08:30.970 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-13.ko 2024-02-19T15:08:30.970 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-14.ko 2024-02-19T15:08:30.970 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-15.ko 2024-02-19T15:08:30.970 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-2.ko 2024-02-19T15:08:30.970 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-3.ko 2024-02-19T15:08:30.970 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-4.ko 2024-02-19T15:08:30.970 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-5.ko 2024-02-19T15:08:30.970 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-6.ko 2024-02-19T15:08:30.971 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-7.ko 2024-02-19T15:08:30.971 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-9.ko 2024-02-19T15:08:30.971 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_koi8-r.ko 2024-02-19T15:08:30.971 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_koi8-ru.ko 2024-02-19T15:08:30.971 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_koi8-u.ko 2024-02-19T15:08:30.971 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_ucs2_utils.ko 2024-02-19T15:08:30.971 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_utf8.ko 2024-02-19T15:08:30.971 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ntfs/ 2024-02-19T15:08:30.971 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 220545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ntfs/ntfs.ko 2024-02-19T15:08:30.971 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/ 2024-02-19T15:08:30.971 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/cluster/ 2024-02-19T15:08:30.971 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 244673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-02-19T15:08:30.972 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/dlm/ 2024-02-19T15:08:30.972 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 440681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-02-19T15:08:30.972 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/dlmfs/ 2024-02-19T15:08:30.972 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 54177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-02-19T15:08:30.972 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 2590849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/ocfs2.ko 2024-02-19T15:08:30.972 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-02-19T15:08:30.972 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-02-19T15:08:30.972 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 35737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-02-19T15:08:30.972 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/omfs/ 2024-02-19T15:08:30.972 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 53705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/omfs/omfs.ko 2024-02-19T15:08:30.972 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/overlayfs/ 2024-02-19T15:08:30.973 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 361169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/overlayfs/overlay.ko 2024-02-19T15:08:31.014 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/pstore/ 2024-02-19T15:08:31.015 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 56945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/pstore/ramoops.ko 2024-02-19T15:08:31.015 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/qnx4/ 2024-02-19T15:08:31.015 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26913 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/qnx4/qnx4.ko 2024-02-19T15:08:31.015 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/qnx6/ 2024-02-19T15:08:31.015 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 46625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/qnx6/qnx6.ko 2024-02-19T15:08:31.015 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/quota/ 2024-02-19T15:08:31.015 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/quota/quota_tree.ko 2024-02-19T15:08:31.015 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14273 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/quota/quota_v1.ko 2024-02-19T15:08:31.015 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/quota/quota_v2.ko 2024-02-19T15:08:31.015 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/reiserfs/ 2024-02-19T15:08:31.015 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 532289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/reiserfs/reiserfs.ko 2024-02-19T15:08:31.015 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/romfs/ 2024-02-19T15:08:31.015 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/romfs/romfs.ko 2024-02-19T15:08:31.015 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/smb/ 2024-02-19T15:08:31.015 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/smb/client/ 2024-02-19T15:08:31.016 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 2710993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/smb/client/cifs.ko 2024-02-19T15:08:31.016 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/smb/common/ 2024-02-19T15:08:31.016 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7065 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/smb/common/cifs_arc4.ko 2024-02-19T15:08:31.016 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/smb/common/cifs_md4.ko 2024-02-19T15:08:31.016 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/squashfs/ 2024-02-19T15:08:31.016 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 134809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/squashfs/squashfs.ko 2024-02-19T15:08:31.016 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/sysv/ 2024-02-19T15:08:31.016 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 91329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/sysv/sysv.ko 2024-02-19T15:08:31.016 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/udf/ 2024-02-19T15:08:31.016 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 248409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/udf/udf.ko 2024-02-19T15:08:31.016 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ufs/ 2024-02-19T15:08:31.016 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 162785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ufs/ufs.ko 2024-02-19T15:08:31.016 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/xfs/ 2024-02-19T15:08:31.016 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 4036329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/xfs/xfs.ko 2024-02-19T15:08:31.016 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/kernel/ 2024-02-19T15:08:31.017 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/kernel/locking/ 2024-02-19T15:08:31.017 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 80489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/kernel/locking/locktorture.ko 2024-02-19T15:08:31.017 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/kernel/time/ 2024-02-19T15:08:31.017 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/kernel/time/test_udelay.ko 2024-02-19T15:08:31.017 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 60265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/kernel/torture.ko 2024-02-19T15:08:31.017 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/ 2024-02-19T15:08:31.017 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 6737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/crc-itu-t.ko 2024-02-19T15:08:31.017 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 6441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/crc7.ko 2024-02-19T15:08:31.017 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 6937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/crc8.ko 2024-02-19T15:08:31.017 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/crypto/ 2024-02-19T15:08:31.017 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/crypto/libarc4.ko 2024-02-19T15:08:31.017 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/crypto/libdes.ko 2024-02-19T15:08:31.017 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/interval_tree_test.ko 2024-02-19T15:08:31.018 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/libcrc32c.ko 2024-02-19T15:08:31.018 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/lru_cache.ko 2024-02-19T15:08:31.018 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/lz4/ 2024-02-19T15:08:31.018 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 33705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/lz4/lz4_compress.ko 2024-02-19T15:08:31.018 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/lz4/lz4hc_compress.ko 2024-02-19T15:08:31.018 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/math/ 2024-02-19T15:08:31.018 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 6529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/math/cordic.ko 2024-02-19T15:08:31.018 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/memory-notifier-error-inject.ko 2024-02-19T15:08:31.018 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/notifier-error-inject.ko 2024-02-19T15:08:31.018 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/percpu_test.ko 2024-02-19T15:08:31.018 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/pm-notifier-error-inject.ko 2024-02-19T15:08:31.018 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/raid6/ 2024-02-19T15:08:31.018 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 143337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/raid6/raid6_pq.ko 2024-02-19T15:08:31.018 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/rbtree_test.ko 2024-02-19T15:08:31.019 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/reed_solomon/ 2024-02-19T15:08:31.019 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/reed_solomon/reed_solomon.ko 2024-02-19T15:08:31.019 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 38745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/test-kstrtox.ko 2024-02-19T15:08:31.019 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/test-string_helpers.ko 2024-02-19T15:08:31.019 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 4863225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/test_bpf.ko 2024-02-19T15:08:31.019 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 48993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/test_firmware.ko 2024-02-19T15:08:31.019 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 6281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/test_module.ko 2024-02-19T15:08:31.019 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/test_user_copy.ko 2024-02-19T15:08:31.019 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/ts_bm.ko 2024-02-19T15:08:31.019 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/ts_fsm.ko 2024-02-19T15:08:31.019 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/ts_kmp.ko 2024-02-19T15:08:31.019 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/xz/ 2024-02-19T15:08:31.019 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/xz/xz_dec_test.ko 2024-02-19T15:08:31.019 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/mm/ 2024-02-19T15:08:31.020 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18129 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/mm/hwpoison-inject.ko 2024-02-19T15:08:31.020 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ 2024-02-19T15:08:31.020 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/6lowpan/ 2024-02-19T15:08:31.020 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 55665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/6lowpan/6lowpan.ko 2024-02-19T15:08:31.020 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 6961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/6lowpan/nhc_dest.ko 2024-02-19T15:08:31.020 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 6969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/6lowpan/nhc_fragment.ko 2024-02-19T15:08:31.020 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 6961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/6lowpan/nhc_hop.ko 2024-02-19T15:08:31.020 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 6897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/6lowpan/nhc_ipv6.ko 2024-02-19T15:08:31.020 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 6969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/6lowpan/nhc_mobility.ko 2024-02-19T15:08:31.020 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 6969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/6lowpan/nhc_routing.ko 2024-02-19T15:08:31.021 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/6lowpan/nhc_udp.ko 2024-02-19T15:08:31.021 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/802/ 2024-02-19T15:08:31.021 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/802/garp.ko 2024-02-19T15:08:31.021 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/802/mrp.ko 2024-02-19T15:08:31.021 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/802/p8022.ko 2024-02-19T15:08:31.021 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/802/psnap.ko 2024-02-19T15:08:31.021 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/802/stp.ko 2024-02-19T15:08:31.021 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/8021q/ 2024-02-19T15:08:31.021 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 82897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/8021q/8021q.ko 2024-02-19T15:08:31.021 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/9p/ 2024-02-19T15:08:31.021 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 198833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/9p/9pnet.ko 2024-02-19T15:08:31.021 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/9p/9pnet_fd.ko 2024-02-19T15:08:31.022 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/9p/9pnet_rdma.ko 2024-02-19T15:08:31.022 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 32249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/9p/9pnet_virtio.ko 2024-02-19T15:08:31.022 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/batman-adv/ 2024-02-19T15:08:31.022 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 667545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/batman-adv/batman-adv.ko 2024-02-19T15:08:31.022 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/ 2024-02-19T15:08:31.022 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 65953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/br_netfilter.ko 2024-02-19T15:08:31.022 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 791305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/bridge.ko 2024-02-19T15:08:31.022 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ 2024-02-19T15:08:31.022 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-02-19T15:08:31.022 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_among.ko 2024-02-19T15:08:31.022 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_arp.ko 2024-02-19T15:08:31.022 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-02-19T15:08:31.022 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-02-19T15:08:31.022 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_ip.ko 2024-02-19T15:08:31.023 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-02-19T15:08:31.023 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_limit.ko 2024-02-19T15:08:31.023 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_log.ko 2024-02-19T15:08:31.023 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_mark.ko 2024-02-19T15:08:31.023 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-02-19T15:08:31.023 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-02-19T15:08:31.023 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-02-19T15:08:31.023 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-02-19T15:08:31.023 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_snat.ko 2024-02-19T15:08:31.023 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_stp.ko 2024-02-19T15:08:31.023 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-02-19T15:08:31.023 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-02-19T15:08:31.023 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-02-19T15:08:31.023 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-02-19T15:08:31.024 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 55945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebtables.ko 2024-02-19T15:08:31.024 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-02-19T15:08:31.024 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-02-19T15:08:31.024 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ceph/ 2024-02-19T15:08:31.024 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 1046801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ceph/libceph.ko 2024-02-19T15:08:31.024 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/core/ 2024-02-19T15:08:31.024 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 125769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/core/pktgen.ko 2024-02-19T15:08:31.024 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dccp/ 2024-02-19T15:08:31.024 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 194505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dccp/dccp.ko 2024-02-19T15:08:31.024 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dccp/dccp_diag.ko 2024-02-19T15:08:31.024 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 58745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dccp/dccp_ipv4.ko 2024-02-19T15:08:31.024 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 61401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dccp/dccp_ipv6.ko 2024-02-19T15:08:31.024 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dsa/ 2024-02-19T15:08:31.024 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 390089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dsa/dsa_core.ko 2024-02-19T15:08:31.025 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dsa/tag_brcm.ko 2024-02-19T15:08:31.025 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dsa/tag_dsa.ko 2024-02-19T15:08:31.025 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dsa/tag_none.ko 2024-02-19T15:08:31.025 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dsa/tag_trailer.ko 2024-02-19T15:08:31.025 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/hsr/ 2024-02-19T15:08:31.025 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 102073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/hsr/hsr.ko 2024-02-19T15:08:31.025 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ieee802154/ 2024-02-19T15:08:31.025 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ieee802154/6lowpan/ 2024-02-19T15:08:31.025 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 45505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-02-19T15:08:31.025 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 284721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ieee802154/ieee802154.ko 2024-02-19T15:08:31.025 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 47881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ieee802154/ieee802154_socket.ko 2024-02-19T15:08:31.025 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/ 2024-02-19T15:08:31.025 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/ah4.ko 2024-02-19T15:08:31.025 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 38977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/esp4.ko 2024-02-19T15:08:31.026 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 48249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/fou.ko 2024-02-19T15:08:31.026 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/gre.ko 2024-02-19T15:08:31.026 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 51217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/inet_diag.ko 2024-02-19T15:08:31.026 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 61017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/ip_gre.ko 2024-02-19T15:08:31.026 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 61337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/ip_tunnel.ko 2024-02-19T15:08:31.026 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/ip_vti.ko 2024-02-19T15:08:31.026 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/ipcomp.ko 2024-02-19T15:08:31.026 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 29033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/ipip.ko 2024-02-19T15:08:31.026 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/ 2024-02-19T15:08:31.026 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 38009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/arp_tables.ko 2024-02-19T15:08:31.026 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-02-19T15:08:31.026 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-02-19T15:08:31.026 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 42057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/ip_tables.ko 2024-02-19T15:08:31.026 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-02-19T15:08:31.026 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9585 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2024-02-19T15:08:31.027 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-02-19T15:08:31.027 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-02-19T15:08:31.027 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-02-19T15:08:31.027 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/iptable_filter.ko 2024-02-19T15:08:31.027 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-02-19T15:08:31.027 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-02-19T15:08:31.027 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-02-19T15:08:31.027 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/iptable_security.ko 2024-02-19T15:08:31.027 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-02-19T15:08:31.027 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-02-19T15:08:31.027 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 27529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-02-19T15:08:31.027 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-02-19T15:08:31.027 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-02-19T15:08:31.028 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2024-02-19T15:08:31.028 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-02-19T15:08:31.028 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-02-19T15:08:31.028 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-02-19T15:08:31.028 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-02-19T15:08:31.028 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8913 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-02-19T15:08:31.028 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_bic.ko 2024-02-19T15:08:31.028 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_dctcp.ko 2024-02-19T15:08:31.028 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_diag.ko 2024-02-19T15:08:31.028 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_highspeed.ko 2024-02-19T15:08:31.028 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_htcp.ko 2024-02-19T15:08:31.028 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_hybla.ko 2024-02-19T15:08:31.028 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_illinois.ko 2024-02-19T15:08:31.028 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_lp.ko 2024-02-19T15:08:31.028 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_scalable.ko 2024-02-19T15:08:31.029 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_vegas.ko 2024-02-19T15:08:31.029 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_veno.ko 2024-02-19T15:08:31.029 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_westwood.ko 2024-02-19T15:08:31.029 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_yeah.ko 2024-02-19T15:08:31.029 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tunnel4.ko 2024-02-19T15:08:31.029 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/udp_diag.ko 2024-02-19T15:08:31.029 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 45001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/udp_tunnel.ko 2024-02-19T15:08:31.029 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/xfrm4_tunnel.ko 2024-02-19T15:08:31.029 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/ 2024-02-19T15:08:31.029 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/ah6.ko 2024-02-19T15:08:31.029 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 39617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/esp6.ko 2024-02-19T15:08:31.029 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/fou6.ko 2024-02-19T15:08:31.029 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 79929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/ip6_gre.ko 2024-02-19T15:08:31.029 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 94169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/ip6_tunnel.ko 2024-02-19T15:08:31.029 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-02-19T15:08:31.030 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 54273 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/ip6_vti.ko 2024-02-19T15:08:31.030 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/ipcomp6.ko 2024-02-19T15:08:31.030 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/mip6.ko 2024-02-19T15:08:31.030 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ 2024-02-19T15:08:31.030 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 42553 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-02-19T15:08:31.030 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2024-02-19T15:08:31.030 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-02-19T15:08:31.030 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14105 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-02-19T15:08:31.030 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-02-19T15:08:31.030 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-02-19T15:08:31.030 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16113 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-02-19T15:08:31.030 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-02-19T15:08:31.030 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-02-19T15:08:31.030 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-02-19T15:08:31.031 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-02-19T15:08:31.031 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-02-19T15:08:31.031 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-02-19T15:08:31.031 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-02-19T15:08:31.031 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-02-19T15:08:31.031 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-02-19T15:08:31.031 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-02-19T15:08:31.031 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 39137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-02-19T15:08:31.031 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-02-19T15:08:31.031 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-02-19T15:08:31.031 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-02-19T15:08:31.031 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-02-19T15:08:31.032 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-02-19T15:08:31.032 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-02-19T15:08:31.032 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8913 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-02-19T15:08:31.032 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 74833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/sit.ko 2024-02-19T15:08:31.032 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24105 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/tunnel6.ko 2024-02-19T15:08:31.032 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/xfrm6_tunnel.ko 2024-02-19T15:08:31.032 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/key/ 2024-02-19T15:08:31.032 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 84345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/key/af_key.ko 2024-02-19T15:08:31.032 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/l2tp/ 2024-02-19T15:08:31.032 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 125209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/l2tp/l2tp_core.ko 2024-02-19T15:08:31.032 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/l2tp/l2tp_debugfs.ko 2024-02-19T15:08:31.032 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 25817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/l2tp/l2tp_eth.ko 2024-02-19T15:08:31.032 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/l2tp/l2tp_ip.ko 2024-02-19T15:08:31.032 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 37377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/l2tp/l2tp_ip6.ko 2024-02-19T15:08:31.033 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 33449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/l2tp/l2tp_netlink.ko 2024-02-19T15:08:31.033 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 53233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/l2tp/l2tp_ppp.ko 2024-02-19T15:08:31.033 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/lapb/ 2024-02-19T15:08:31.033 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 48121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/lapb/lapb.ko 2024-02-19T15:08:31.033 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/llc/ 2024-02-19T15:08:31.033 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/llc/llc.ko 2024-02-19T15:08:31.033 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 268521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/llc/llc2.ko 2024-02-19T15:08:31.033 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/mac802154/ 2024-02-19T15:08:31.033 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 309417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/mac802154/mac802154.ko 2024-02-19T15:08:31.033 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/mpls/ 2024-02-19T15:08:31.033 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/mpls/mpls_gso.ko 2024-02-19T15:08:31.033 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ 2024-02-19T15:08:31.033 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ 2024-02-19T15:08:31.033 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 119705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set.ko 2024-02-19T15:08:31.033 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-02-19T15:08:31.033 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 27305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-02-19T15:08:31.034 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-02-19T15:08:31.034 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 99993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-02-19T15:08:31.034 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 100281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-02-19T15:08:31.034 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 102553 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-02-19T15:08:31.034 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 102345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-02-19T15:08:31.034 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 111673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-02-19T15:08:31.034 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 59193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-02-19T15:08:31.034 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 108345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-02-19T15:08:31.034 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 112145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-02-19T15:08:31.034 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 112409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-02-19T15:08:31.034 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 110337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-02-19T15:08:31.034 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 113289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-02-19T15:08:31.034 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-02-19T15:08:31.034 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ 2024-02-19T15:08:31.035 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 412513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs.ko 2024-02-19T15:08:31.035 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-02-19T15:08:31.035 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-02-19T15:08:31.035 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-02-19T15:08:31.035 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-02-19T15:08:31.035 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-02-19T15:08:31.035 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-02-19T15:08:31.035 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-02-19T15:08:31.035 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-02-19T15:08:31.035 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-02-19T15:08:31.035 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-02-19T15:08:31.035 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-02-19T15:08:31.035 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-02-19T15:08:31.035 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-02-19T15:08:31.036 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 49169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conncount.ko 2024-02-19T15:08:31.036 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 411745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack.ko 2024-02-19T15:08:31.036 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-02-19T15:08:31.036 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-02-19T15:08:31.036 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 33665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-02-19T15:08:31.036 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 118097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_h323.ko 2024-02-19T15:08:31.036 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_irc.ko 2024-02-19T15:08:31.036 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-02-19T15:08:31.036 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 108313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-02-19T15:08:31.036 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 33577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-02-19T15:08:31.036 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_sane.ko 2024-02-19T15:08:31.036 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 66537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_sip.ko 2024-02-19T15:08:31.036 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-02-19T15:08:31.036 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-02-19T15:08:31.036 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_dup_netdev.ko 2024-02-19T15:08:31.037 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 92209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_flow_table.ko 2024-02-19T15:08:31.037 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9553 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_flow_table_inet.ko 2024-02-19T15:08:31.037 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 38081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_log_syslog.ko 2024-02-19T15:08:31.037 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 96521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_nat.ko 2024-02-19T15:08:31.037 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_nat_amanda.ko 2024-02-19T15:08:31.037 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_nat_ftp.ko 2024-02-19T15:08:31.037 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_nat_irc.ko 2024-02-19T15:08:31.037 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_nat_sip.ko 2024-02-19T15:08:31.037 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_nat_tftp.ko 2024-02-19T15:08:31.037 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 44137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_synproxy_core.ko 2024-02-19T15:08:31.037 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 623417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_tables.ko 2024-02-19T15:08:31.037 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 42577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nfnetlink.ko 2024-02-19T15:08:31.037 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nfnetlink_acct.ko 2024-02-19T15:08:31.037 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nfnetlink_cttimeout.ko 2024-02-19T15:08:31.037 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 39561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nfnetlink_log.ko 2024-02-19T15:08:31.038 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nfnetlink_osf.ko 2024-02-19T15:08:31.038 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 49649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nfnetlink_queue.ko 2024-02-19T15:08:31.038 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_chain_nat.ko 2024-02-19T15:08:31.038 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 33225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_compat.ko 2024-02-19T15:08:31.038 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 40761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_ct.ko 2024-02-19T15:08:31.038 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_dup_netdev.ko 2024-02-19T15:08:31.038 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_fib.ko 2024-02-19T15:08:31.038 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_fwd_netdev.ko 2024-02-19T15:08:31.038 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_hash.ko 2024-02-19T15:08:31.038 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_limit.ko 2024-02-19T15:08:31.038 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_log.ko 2024-02-19T15:08:31.038 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_masq.ko 2024-02-19T15:08:31.038 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_nat.ko 2024-02-19T15:08:31.038 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_numgen.ko 2024-02-19T15:08:31.038 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11753 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_osf.ko 2024-02-19T15:08:31.039 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_queue.ko 2024-02-19T15:08:31.039 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_redir.ko 2024-02-19T15:08:31.039 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_reject.ko 2024-02-19T15:08:31.039 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_reject_inet.ko 2024-02-19T15:08:31.039 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_socket.ko 2024-02-19T15:08:31.039 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_synproxy.ko 2024-02-19T15:08:31.039 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_tproxy.ko 2024-02-19T15:08:31.039 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 89233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/x_tables.ko 2024-02-19T15:08:31.039 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_AUDIT.ko 2024-02-19T15:08:31.039 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_CHECKSUM.ko 2024-02-19T15:08:31.039 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_CLASSIFY.ko 2024-02-19T15:08:31.039 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-02-19T15:08:31.039 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 18081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_CT.ko 2024-02-19T15:08:31.039 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_DSCP.ko 2024-02-19T15:08:31.040 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_HL.ko 2024-02-19T15:08:31.040 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_HMARK.ko 2024-02-19T15:08:31.040 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 38777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_IDLETIMER.ko 2024-02-19T15:08:31.040 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_LED.ko 2024-02-19T15:08:31.040 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_LOG.ko 2024-02-19T15:08:31.040 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_MASQUERADE.ko 2024-02-19T15:08:31.040 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_NETMAP.ko 2024-02-19T15:08:31.040 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_NFLOG.ko 2024-02-19T15:08:31.040 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_NFQUEUE.ko 2024-02-19T15:08:31.040 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_RATEEST.ko 2024-02-19T15:08:31.040 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_REDIRECT.ko 2024-02-19T15:08:31.040 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_SECMARK.ko 2024-02-19T15:08:31.040 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_TCPMSS.ko 2024-02-19T15:08:31.040 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-02-19T15:08:31.040 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_TEE.ko 2024-02-19T15:08:31.041 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_TPROXY.ko 2024-02-19T15:08:31.041 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_TRACE.ko 2024-02-19T15:08:31.041 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_addrtype.ko 2024-02-19T15:08:31.041 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_bpf.ko 2024-02-19T15:08:31.041 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_cgroup.ko 2024-02-19T15:08:31.041 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_cluster.ko 2024-02-19T15:08:31.041 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_comment.ko 2024-02-19T15:08:31.041 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_connbytes.ko 2024-02-19T15:08:31.041 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_connlabel.ko 2024-02-19T15:08:31.041 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_connlimit.ko 2024-02-19T15:08:31.041 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_connmark.ko 2024-02-19T15:08:31.041 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_conntrack.ko 2024-02-19T15:08:31.041 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_cpu.ko 2024-02-19T15:08:31.042 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_dccp.ko 2024-02-19T15:08:31.042 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_devgroup.ko 2024-02-19T15:08:31.042 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_dscp.ko 2024-02-19T15:08:31.042 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_ecn.ko 2024-02-19T15:08:31.042 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_esp.ko 2024-02-19T15:08:31.042 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 43201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_hashlimit.ko 2024-02-19T15:08:31.042 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_helper.ko 2024-02-19T15:08:31.042 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_hl.ko 2024-02-19T15:08:31.042 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_ipcomp.ko 2024-02-19T15:08:31.042 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_iprange.ko 2024-02-19T15:08:31.042 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_ipvs.ko 2024-02-19T15:08:31.042 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_l2tp.ko 2024-02-19T15:08:31.042 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_length.ko 2024-02-19T15:08:31.042 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_limit.ko 2024-02-19T15:08:31.043 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_mac.ko 2024-02-19T15:08:31.043 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8553 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_mark.ko 2024-02-19T15:08:31.043 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_multiport.ko 2024-02-19T15:08:31.043 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_nat.ko 2024-02-19T15:08:31.043 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_nfacct.ko 2024-02-19T15:08:31.043 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_osf.ko 2024-02-19T15:08:31.043 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_owner.ko 2024-02-19T15:08:31.043 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_physdev.ko 2024-02-19T15:08:31.043 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_pkttype.ko 2024-02-19T15:08:31.043 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_policy.ko 2024-02-19T15:08:31.043 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_quota.ko 2024-02-19T15:08:31.043 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_rateest.ko 2024-02-19T15:08:31.043 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_realm.ko 2024-02-19T15:08:31.043 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_recent.ko 2024-02-19T15:08:31.043 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_sctp.ko 2024-02-19T15:08:31.044 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_set.ko 2024-02-19T15:08:31.044 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 16817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_socket.ko 2024-02-19T15:08:31.044 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_state.ko 2024-02-19T15:08:31.044 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_statistic.ko 2024-02-19T15:08:31.044 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_string.ko 2024-02-19T15:08:31.044 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 8457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_tcpmss.ko 2024-02-19T15:08:31.044 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_tcpudp.ko 2024-02-19T15:08:31.044 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_time.ko 2024-02-19T15:08:31.044 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_u32.ko 2024-02-19T15:08:31.044 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netlink/ 2024-02-19T15:08:31.044 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 14441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netlink/netlink_diag.ko 2024-02-19T15:08:31.044 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/nsh/ 2024-02-19T15:08:31.044 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 12337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/nsh/nsh.ko 2024-02-19T15:08:31.044 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/openvswitch/ 2024-02-19T15:08:31.045 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 384097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/openvswitch/openvswitch.ko 2024-02-19T15:08:31.045 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/openvswitch/vport-geneve.ko 2024-02-19T15:08:31.045 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/openvswitch/vport-gre.ko 2024-02-19T15:08:31.045 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/openvswitch/vport-vxlan.ko 2024-02-19T15:08:31.045 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/packet/ 2024-02-19T15:08:31.045 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/packet/af_packet_diag.ko 2024-02-19T15:08:31.045 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/phonet/ 2024-02-19T15:08:31.045 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 112153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/phonet/phonet.ko 2024-02-19T15:08:31.045 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 54729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/phonet/pn_pep.ko 2024-02-19T15:08:31.045 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/rds/ 2024-02-19T15:08:31.045 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 253737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/rds/rds.ko 2024-02-19T15:08:31.045 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 165105 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/rds/rds_rdma.ko 2024-02-19T15:08:31.045 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 68161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/rds/rds_tcp.ko 2024-02-19T15:08:31.045 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/rxrpc/ 2024-02-19T15:08:31.046 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 828297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/rxrpc/rxrpc.ko 2024-02-19T15:08:31.046 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/ 2024-02-19T15:08:31.046 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 32153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/act_csum.ko 2024-02-19T15:08:31.046 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/act_gact.ko 2024-02-19T15:08:31.046 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/act_mirred.ko 2024-02-19T15:08:31.046 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/act_nat.ko 2024-02-19T15:08:31.046 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 43417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/act_pedit.ko 2024-02-19T15:08:31.046 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/act_police.ko 2024-02-19T15:08:31.046 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/act_simple.ko 2024-02-19T15:08:31.046 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 35201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/act_skbedit.ko 2024-02-19T15:08:31.046 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/act_vlan.ko 2024-02-19T15:08:31.046 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/cls_basic.ko 2024-02-19T15:08:31.046 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 33873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/cls_bpf.ko 2024-02-19T15:08:31.046 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 19185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/cls_cgroup.ko 2024-02-19T15:08:31.046 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/cls_flow.ko 2024-02-19T15:08:31.046 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/cls_fw.ko 2024-02-19T15:08:31.047 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 39521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/cls_route.ko 2024-02-19T15:08:31.047 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 62657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/cls_u32.ko 2024-02-19T15:08:31.047 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/em_cmp.ko 2024-02-19T15:08:31.047 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 13321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/em_ipset.ko 2024-02-19T15:08:31.047 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 33249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/em_meta.ko 2024-02-19T15:08:31.047 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/em_nbyte.ko 2024-02-19T15:08:31.047 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/em_text.ko 2024-02-19T15:08:31.047 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/em_u32.ko 2024-02-19T15:08:31.047 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_choke.ko 2024-02-19T15:08:31.047 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_codel.ko 2024-02-19T15:08:31.047 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 34657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_drr.ko 2024-02-19T15:08:31.047 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 35793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_fq.ko 2024-02-19T15:08:31.047 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_fq_codel.ko 2024-02-19T15:08:31.047 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 35457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_gred.ko 2024-02-19T15:08:31.048 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 49849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_hfsc.ko 2024-02-19T15:08:31.048 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 21977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_hhf.ko 2024-02-19T15:08:31.048 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 66449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_htb.ko 2024-02-19T15:08:31.048 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_ingress.ko 2024-02-19T15:08:31.048 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 30225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_mqprio.ko 2024-02-19T15:08:31.048 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_mqprio_lib.ko 2024-02-19T15:08:31.048 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_multiq.ko 2024-02-19T15:08:31.048 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 36153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_netem.ko 2024-02-19T15:08:31.048 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 27409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_pie.ko 2024-02-19T15:08:31.048 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_plug.ko 2024-02-19T15:08:31.048 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 24777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_prio.ko 2024-02-19T15:08:31.048 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 48537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_qfq.ko 2024-02-19T15:08:31.048 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 32817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_red.ko 2024-02-19T15:08:31.048 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 26793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_sfb.ko 2024-02-19T15:08:31.048 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 31321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_sfq.ko 2024-02-19T15:08:31.049 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_tbf.ko 2024-02-19T15:08:31.049 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 28273 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_teql.ko 2024-02-19T15:08:31.049 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sctp/ 2024-02-19T15:08:31.049 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 932945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sctp/sctp.ko 2024-02-19T15:08:31.049 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sctp/sctp_diag.ko 2024-02-19T15:08:31.049 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sunrpc/ 2024-02-19T15:08:31.049 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sunrpc/auth_gss/ 2024-02-19T15:08:31.049 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 337009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-02-19T15:08:31.049 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 55041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-02-19T15:08:31.049 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 1513905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sunrpc/sunrpc.ko 2024-02-19T15:08:31.049 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sunrpc/xprtrdma/ 2024-02-19T15:08:31.049 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 704761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-02-19T15:08:31.049 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/tipc/ 2024-02-19T15:08:31.049 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 10401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/tipc/diag.ko 2024-02-19T15:08:31.050 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 867249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/tipc/tipc.ko 2024-02-19T15:08:31.050 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/unix/ 2024-02-19T15:08:31.050 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/unix/unix_diag.ko 2024-02-19T15:08:31.050 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/vmw_vsock/ 2024-02-19T15:08:31.050 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 87745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-02-19T15:08:31.050 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 67953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-02-19T15:08:31.050 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 98113 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/vmw_vsock/vsock.ko 2024-02-19T15:08:31.050 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 11081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/vmw_vsock/vsock_diag.ko 2024-02-19T15:08:31.050 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 17817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/vmw_vsock/vsock_loopback.ko 2024-02-19T15:08:31.050 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/x25/ 2024-02-19T15:08:31.050 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 131273 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/x25/x25.ko 2024-02-19T15:08:31.050 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/xfrm/ 2024-02-19T15:08:31.050 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 22953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/xfrm/xfrm_algo.ko 2024-02-19T15:08:31.050 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 20553 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/xfrm/xfrm_ipcomp.ko 2024-02-19T15:08:31.051 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 105097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/xfrm/xfrm_user.ko 2024-02-19T15:08:31.051 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/virt/ 2024-02-19T15:08:31.051 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/virt/lib/ 2024-02-19T15:08:31.051 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 15441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/virt/lib/irqbypass.ko 2024-02-19T15:08:31.051 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 642229 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.alias 2024-02-19T15:08:31.051 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 625558 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.alias.bin 2024-02-19T15:08:31.051 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 7677 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.builtin 2024-02-19T15:08:31.051 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 23498 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.builtin.alias.bin 2024-02-19T15:08:31.051 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 9916 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.builtin.bin 2024-02-19T15:08:31.051 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 70627 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.builtin.modinfo 2024-02-19T15:08:31.051 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 147214 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.dep 2024-02-19T15:08:31.051 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 233213 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.dep.bin 2024-02-19T15:08:31.051 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 210 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.devname 2024-02-19T15:08:31.051 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 72614 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.order 2024-02-19T15:08:31.051 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 879 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.softdep 2024-02-19T15:08:31.052 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 257955 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.symbols 2024-02-19T15:08:31.052 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 312782 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.symbols.bin 2024-02-19T15:08:31.052 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./usr/ 2024-02-19T15:08:31.052 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./usr/share/ 2024-02-19T15:08:31.052 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./usr/share/doc/ 2024-02-19T15:08:31.052 INFO:teuthology.orchestra.run.smithi047.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./usr/share/doc/linux-image-6.8.0-rc4-g5d6d84d6d8d3/ 2024-02-19T15:08:31.052 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 173 2024-02-19 07:58 ./usr/share/doc/linux-image-6.8.0-rc4-g5d6d84d6d8d3/changelog.Debian.gz 2024-02-19T15:08:31.052 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r-- root/root 691 2024-02-19 07:58 ./usr/share/doc/linux-image-6.8.0-rc4-g5d6d84d6d8d3/copyright 2024-02-19T15:08:31.057 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:31.057 INFO:teuthology.task.kernel:searching for kernel 6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:31.058 DEBUG:teuthology.orchestra.run.smithi047:> egrep '(submenu|menuentry.*6.8.0-rc4-g5d6d84d6d8d3).*{' /boot/grub/grub.cfg 2024-02-19T15:08:31.064 INFO:teuthology.orchestra.run.smithi047.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-02-19T15:08:31.064 INFO:teuthology.orchestra.run.smithi047.stdout: menuentry 'Ubuntu, with Linux 6.8.0-rc4-g5d6d84d6d8d3' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.8.0-rc4-g5d6d84d6d8d3-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-02-19T15:08:31.064 INFO:teuthology.orchestra.run.smithi047.stdout: menuentry 'Ubuntu, with Linux 6.8.0-rc4-g5d6d84d6d8d3 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.8.0-rc4-g5d6d84d6d8d3-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-02-19T15:08:31.065 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2024-02-19T15:08:31.065 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:31.065 DEBUG:teuthology.orchestra.run.smithi047:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2024-02-19T15:08:31.109 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi047.front.sepia.ceph.com to complete... 2024-02-19T15:08:31.539 INFO:teuthology.orchestra.run.smithi047.stderr:Sourcing file `/etc/default/grub' 2024-02-19T15:08:31.539 INFO:teuthology.orchestra.run.smithi047.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-02-19T15:08:31.549 INFO:teuthology.orchestra.run.smithi047.stderr:Generating grub configuration file ... 2024-02-19T15:08:31.754 INFO:teuthology.orchestra.run.smithi047.stderr:Found linux image: /boot/vmlinuz-6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:31.760 INFO:teuthology.orchestra.run.smithi047.stderr:Found initrd image: /boot/initrd.img-6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:31.972 INFO:teuthology.orchestra.run.smithi047.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-02-19T15:08:31.977 INFO:teuthology.orchestra.run.smithi047.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-02-19T15:08:32.022 INFO:teuthology.orchestra.run.smithi047.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-02-19T15:08:32.028 INFO:teuthology.orchestra.run.smithi047.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-02-19T15:08:32.197 INFO:teuthology.orchestra.run.smithi047.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-02-19T15:08:32.197 INFO:teuthology.orchestra.run.smithi047.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-02-19T15:08:32.198 INFO:teuthology.orchestra.run.smithi047.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-02-19T15:08:32.217 INFO:teuthology.orchestra.run.smithi047.stderr:done 2024-02-19T15:08:51.473 INFO:teuthology.orchestra.run.smithi116.stderr:Sourcing file `/etc/default/grub' 2024-02-19T15:08:51.489 INFO:teuthology.orchestra.run.smithi116.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-02-19T15:08:51.514 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2024-02-19T15:08:51.839 INFO:teuthology.orchestra.run.smithi116.stderr:Found linux image: /boot/vmlinuz-6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:51.859 INFO:teuthology.orchestra.run.smithi116.stderr:Found initrd image: /boot/initrd.img-6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:52.152 INFO:teuthology.orchestra.run.smithi116.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-02-19T15:08:52.164 INFO:teuthology.orchestra.run.smithi116.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-02-19T15:08:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-02-19T15:08:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-02-19T15:08:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-02-19T15:08:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-02-19T15:08:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-02-19T15:08:52.453 INFO:teuthology.orchestra.run.smithi116.stderr:done 2024-02-19T15:08:52.613 DEBUG:teuthology.orchestra.run.smithi116:> dpkg-deb -c /tmp/linux-image.deb 2024-02-19T15:08:52.621 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./ 2024-02-19T15:08:52.622 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./boot/ 2024-02-19T15:08:52.622 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 4469316 2024-02-19 07:58 ./boot/System.map-6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:52.634 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 170841 2024-02-19 07:58 ./boot/config-6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:52.634 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11527168 2024-02-19 07:58 ./boot/vmlinuz-6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:52.655 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./etc/ 2024-02-19T15:08:52.655 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./etc/kernel/ 2024-02-19T15:08:52.655 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./etc/kernel/postinst.d/ 2024-02-19T15:08:52.655 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./etc/kernel/postrm.d/ 2024-02-19T15:08:52.655 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./etc/kernel/preinst.d/ 2024-02-19T15:08:52.655 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./etc/kernel/prerm.d/ 2024-02-19T15:08:52.655 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/ 2024-02-19T15:08:52.655 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/ 2024-02-19T15:08:52.655 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/ 2024-02-19T15:08:52.655 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/ 2024-02-19T15:08:52.655 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/ 2024-02-19T15:08:52.655 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/ 2024-02-19T15:08:52.655 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/ 2024-02-19T15:08:52.656 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 490449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/aesni-intel.ko 2024-02-19T15:08:52.658 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-02-19T15:08:52.658 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-02-19T15:08:52.658 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-02-19T15:08:52.659 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 93081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-02-19T15:08:52.659 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 106697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-02-19T15:08:52.659 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 152145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-02-19T15:08:52.659 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-02-19T15:08:52.659 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-02-19T15:08:52.659 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 83857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-02-19T15:08:52.659 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-02-19T15:08:52.659 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 47185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-02-19T15:08:52.659 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 47177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/serpent-avx2.ko 2024-02-19T15:08:52.659 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 51697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-02-19T15:08:52.659 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 39601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-02-19T15:08:52.659 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 42537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-02-19T15:08:52.659 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 58489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-02-19T15:08:52.659 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 50393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-02-19T15:08:52.660 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-02-19T15:08:52.660 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-02-19T15:08:52.660 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/kernel/ 2024-02-19T15:08:52.660 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/kernel/cpu/ 2024-02-19T15:08:52.660 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/kernel/cpu/mce/ 2024-02-19T15:08:52.660 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 41433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-02-19T15:08:52.660 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13593 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/kernel/cpuid.ko 2024-02-19T15:08:52.660 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/kernel/msr.ko 2024-02-19T15:08:52.660 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/kvm/ 2024-02-19T15:08:52.660 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 354001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/kvm/kvm-amd.ko 2024-02-19T15:08:52.660 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 885905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/kvm/kvm-intel.ko 2024-02-19T15:08:52.661 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 2294545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/arch/x86/kvm/kvm.ko 2024-02-19T15:08:52.714 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/ 2024-02-19T15:08:52.714 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 53841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/af_alg.ko 2024-02-19T15:08:52.714 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/algif_hash.ko 2024-02-19T15:08:52.714 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/algif_skcipher.ko 2024-02-19T15:08:52.714 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17113 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/ansi_cprng.ko 2024-02-19T15:08:52.714 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19553 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/anubis.ko 2024-02-19T15:08:52.715 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9585 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/arc4.ko 2024-02-19T15:08:52.715 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/asymmetric_keys/ 2024-02-19T15:08:52.715 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2024-02-19T15:08:52.715 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/async_tx/ 2024-02-19T15:08:52.715 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/async_tx/async_memcpy.ko 2024-02-19T15:08:52.715 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/async_tx/async_pq.ko 2024-02-19T15:08:52.715 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/async_tx/async_raid6_recov.ko 2024-02-19T15:08:52.715 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/async_tx/async_tx.ko 2024-02-19T15:08:52.715 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/async_tx/async_xor.ko 2024-02-19T15:08:52.715 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/async_tx/raid6test.ko 2024-02-19T15:08:52.715 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/authenc.ko 2024-02-19T15:08:52.715 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/authencesn.ko 2024-02-19T15:08:52.715 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/blake2b_generic.ko 2024-02-19T15:08:52.715 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/blowfish_common.ko 2024-02-19T15:08:52.715 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/blowfish_generic.ko 2024-02-19T15:08:52.716 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 38777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/camellia_generic.ko 2024-02-19T15:08:52.716 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 27233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/cast5_generic.ko 2024-02-19T15:08:52.716 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/cast6_generic.ko 2024-02-19T15:08:52.716 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/cast_common.ko 2024-02-19T15:08:52.716 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 35185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/ccm.ko 2024-02-19T15:08:52.716 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/cmac.ko 2024-02-19T15:08:52.716 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/crc32_generic.ko 2024-02-19T15:08:52.716 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 47681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/cryptd.ko 2024-02-19T15:08:52.716 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 39473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/crypto_engine.ko 2024-02-19T15:08:52.716 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26921 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/crypto_simd.ko 2024-02-19T15:08:52.716 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/crypto_user.ko 2024-02-19T15:08:52.716 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/cts.ko 2024-02-19T15:08:52.716 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/des_generic.ko 2024-02-19T15:08:52.716 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15593 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/dh_generic.ko 2024-02-19T15:08:52.717 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10913 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/echainiv.ko 2024-02-19T15:08:52.717 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/essiv.ko 2024-02-19T15:08:52.717 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/fcrypt.ko 2024-02-19T15:08:52.717 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/khazad.ko 2024-02-19T15:08:52.717 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/lrw.ko 2024-02-19T15:08:52.717 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/lz4.ko 2024-02-19T15:08:52.717 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/lz4hc.ko 2024-02-19T15:08:52.717 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/md4.ko 2024-02-19T15:08:52.717 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/michael_mic.ko 2024-02-19T15:08:52.717 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10585 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/pcbc.ko 2024-02-19T15:08:52.717 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/pcrypt.ko 2024-02-19T15:08:52.717 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/rmd160.ko 2024-02-19T15:08:52.717 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/seed.ko 2024-02-19T15:08:52.717 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/serpent_generic.ko 2024-02-19T15:08:52.717 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 143033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/tcrypt.ko 2024-02-19T15:08:52.717 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/tea.ko 2024-02-19T15:08:52.717 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/twofish_common.ko 2024-02-19T15:08:52.718 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/twofish_generic.ko 2024-02-19T15:08:52.718 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/vmac.ko 2024-02-19T15:08:52.718 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/wp512.ko 2024-02-19T15:08:52.718 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/xcbc.ko 2024-02-19T15:08:52.718 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/xor.ko 2024-02-19T15:08:52.718 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/xts.ko 2024-02-19T15:08:52.718 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/crypto/xxhash_generic.ko 2024-02-19T15:08:52.718 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ 2024-02-19T15:08:52.718 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/ 2024-02-19T15:08:52.718 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/acpi_extlog.ko 2024-02-19T15:08:52.718 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/acpi_ipmi.ko 2024-02-19T15:08:52.718 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/acpi_pad.ko 2024-02-19T15:08:52.718 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/apei/ 2024-02-19T15:08:52.718 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/apei/einj.ko 2024-02-19T15:08:52.718 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/ec_sys.ko 2024-02-19T15:08:52.719 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/platform_profile.ko 2024-02-19T15:08:52.719 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/sbs.ko 2024-02-19T15:08:52.719 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/sbshc.ko 2024-02-19T15:08:52.719 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 104433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/acpi/video.ko 2024-02-19T15:08:52.719 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/ 2024-02-19T15:08:52.719 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/acard-ahci.ko 2024-02-19T15:08:52.719 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 89825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/ahci.ko 2024-02-19T15:08:52.719 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/ahci_platform.ko 2024-02-19T15:08:52.719 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 88313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/libahci.ko 2024-02-19T15:08:52.719 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/libahci_platform.ko 2024-02-19T15:08:52.719 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_acpi.ko 2024-02-19T15:08:52.719 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_ali.ko 2024-02-19T15:08:52.719 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 33481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_amd.ko 2024-02-19T15:08:52.719 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_artop.ko 2024-02-19T15:08:52.719 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_atiixp.ko 2024-02-19T15:08:52.720 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_atp867x.ko 2024-02-19T15:08:52.720 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_cmd640.ko 2024-02-19T15:08:52.720 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_cmd64x.ko 2024-02-19T15:08:52.720 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_cypress.ko 2024-02-19T15:08:52.720 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17753 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_efar.ko 2024-02-19T15:08:52.720 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_hpt366.ko 2024-02-19T15:08:52.720 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 33881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_hpt37x.ko 2024-02-19T15:08:52.720 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_hpt3x2n.ko 2024-02-19T15:08:52.720 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_hpt3x3.ko 2024-02-19T15:08:52.720 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_it8213.ko 2024-02-19T15:08:52.720 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 35177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_it821x.ko 2024-02-19T15:08:52.720 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_jmicron.ko 2024-02-19T15:08:52.720 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 52177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_legacy.ko 2024-02-19T15:08:52.720 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_marvell.ko 2024-02-19T15:08:52.720 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_mpiix.ko 2024-02-19T15:08:52.721 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_netcell.ko 2024-02-19T15:08:52.721 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_ninja32.ko 2024-02-19T15:08:52.721 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_ns87410.ko 2024-02-19T15:08:52.721 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_ns87415.ko 2024-02-19T15:08:52.721 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_oldpiix.ko 2024-02-19T15:08:52.721 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_opti.ko 2024-02-19T15:08:52.721 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20585 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_optidma.ko 2024-02-19T15:08:52.721 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 32489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_pcmcia.ko 2024-02-19T15:08:52.721 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 37697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_pdc2027x.ko 2024-02-19T15:08:52.721 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-02-19T15:08:52.721 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14129 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_piccolo.ko 2024-02-19T15:08:52.721 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_radisys.ko 2024-02-19T15:08:52.721 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_rdc.ko 2024-02-19T15:08:52.721 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_rz1000.ko 2024-02-19T15:08:52.721 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_sch.ko 2024-02-19T15:08:52.722 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_serverworks.ko 2024-02-19T15:08:52.722 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_sil680.ko 2024-02-19T15:08:52.722 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_sl82c105.ko 2024-02-19T15:08:52.722 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15913 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_triflex.ko 2024-02-19T15:08:52.722 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pata_via.ko 2024-02-19T15:08:52.722 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/pdc_adma.ko 2024-02-19T15:08:52.722 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_inic162x.ko 2024-02-19T15:08:52.722 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 83257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_mv.ko 2024-02-19T15:08:52.722 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 70305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_nv.ko 2024-02-19T15:08:52.722 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 37281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_promise.ko 2024-02-19T15:08:52.722 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_qstor.ko 2024-02-19T15:08:52.722 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_sil.ko 2024-02-19T15:08:52.722 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 38177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_sil24.ko 2024-02-19T15:08:52.722 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_sis.ko 2024-02-19T15:08:52.722 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_svw.ko 2024-02-19T15:08:52.723 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 44705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_sx4.ko 2024-02-19T15:08:52.723 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_uli.ko 2024-02-19T15:08:52.723 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_via.ko 2024-02-19T15:08:52.723 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ata/sata_vsc.ko 2024-02-19T15:08:52.723 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/auxdisplay/ 2024-02-19T15:08:52.723 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/auxdisplay/ks0108.ko 2024-02-19T15:08:52.723 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/bcma/ 2024-02-19T15:08:52.723 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 124457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/bcma/bcma.ko 2024-02-19T15:08:52.723 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/ 2024-02-19T15:08:52.723 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/aoe/ 2024-02-19T15:08:52.723 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 113833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/aoe/aoe.ko 2024-02-19T15:08:52.723 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/drbd/ 2024-02-19T15:08:52.723 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 964817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/drbd/drbd.ko 2024-02-19T15:08:52.723 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 169705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/floppy.ko 2024-02-19T15:08:52.724 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/mtip32xx/ 2024-02-19T15:08:52.724 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 94217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-02-19T15:08:52.724 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 116401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/nbd.ko 2024-02-19T15:08:52.724 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/null_blk/ 2024-02-19T15:08:52.724 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 110329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/null_blk/null_blk.ko 2024-02-19T15:08:52.724 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 98193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/pktcdvd.ko 2024-02-19T15:08:52.724 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 263641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/rbd.ko 2024-02-19T15:08:52.724 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/xen-blkback/ 2024-02-19T15:08:52.724 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 98121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-02-19T15:08:52.724 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/zram/ 2024-02-19T15:08:52.724 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 58777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/block/zram/zram.ko 2024-02-19T15:08:52.724 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/ 2024-02-19T15:08:52.724 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/applicom.ko 2024-02-19T15:08:52.724 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11913 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/hangcheck-timer.ko 2024-02-19T15:08:52.725 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/hw_random/ 2024-02-19T15:08:52.725 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/hw_random/amd-rng.ko 2024-02-19T15:08:52.725 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/hw_random/intel-rng.ko 2024-02-19T15:08:52.725 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-02-19T15:08:52.725 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/hw_random/via-rng.ko 2024-02-19T15:08:52.725 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/hw_random/virtio-rng.ko 2024-02-19T15:08:52.725 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/ipmi/ 2024-02-19T15:08:52.725 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-02-19T15:08:52.725 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 148521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-02-19T15:08:52.725 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-02-19T15:08:52.725 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 139361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/ipmi/ipmi_si.ko 2024-02-19T15:08:52.725 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 70873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-02-19T15:08:52.725 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 53417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-02-19T15:08:52.725 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/lp.ko 2024-02-19T15:08:52.725 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/mwave/ 2024-02-19T15:08:52.726 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 84025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/mwave/mwave.ko 2024-02-19T15:08:52.726 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/nvram.ko 2024-02-19T15:08:52.726 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 32081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/ppdev.ko 2024-02-19T15:08:52.726 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 51025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/tlclk.ko 2024-02-19T15:08:52.726 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/tpm/ 2024-02-19T15:08:52.726 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/tpm/tpm_atmel.ko 2024-02-19T15:08:52.726 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2024-02-19T15:08:52.726 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2024-02-19T15:08:52.726 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 32769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2024-02-19T15:08:52.726 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/tpm/tpm_infineon.ko 2024-02-19T15:08:52.726 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22609 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/tpm/tpm_nsc.ko 2024-02-19T15:08:52.726 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23465 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/tpm/xen-tpmfront.ko 2024-02-19T15:08:52.726 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/xillybus/ 2024-02-19T15:08:52.726 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/xillybus/xillybus_class.ko 2024-02-19T15:08:52.726 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 50849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/xillybus/xillybus_core.ko 2024-02-19T15:08:52.727 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/char/xillybus/xillybus_pcie.ko 2024-02-19T15:08:52.727 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/clk/ 2024-02-19T15:08:52.727 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/clk/clk-palmas.ko 2024-02-19T15:08:52.727 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 47729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/clk/clk-si5351.ko 2024-02-19T15:08:52.727 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/clk/clk-twl6040.ko 2024-02-19T15:08:52.727 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/clk/clk-wm831x.ko 2024-02-19T15:08:52.727 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/cpufreq/ 2024-02-19T15:08:52.727 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-02-19T15:08:52.727 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/cpufreq/p4-clockmod.ko 2024-02-19T15:08:52.727 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/cpufreq/speedstep-lib.ko 2024-02-19T15:08:52.727 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/ 2024-02-19T15:08:52.727 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/ccp/ 2024-02-19T15:08:52.727 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 70473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-02-19T15:08:52.727 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 246841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/ccp/ccp.ko 2024-02-19T15:08:52.727 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/intel/ 2024-02-19T15:08:52.727 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/intel/qat/ 2024-02-19T15:08:52.728 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/intel/qat/qat_common/ 2024-02-19T15:08:52.728 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 570497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-02-19T15:08:52.728 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2024-02-19T15:08:52.728 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-02-19T15:08:52.728 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/padlock-aes.ko 2024-02-19T15:08:52.728 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/padlock-sha.ko 2024-02-19T15:08:52.728 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/virtio/ 2024-02-19T15:08:52.728 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 70433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-02-19T15:08:52.728 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/dca/ 2024-02-19T15:08:52.728 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/dca/dca.ko 2024-02-19T15:08:52.728 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/dma/ 2024-02-19T15:08:52.728 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/dma/dw/ 2024-02-19T15:08:52.728 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/dma/dw/dw_dmac.ko 2024-02-19T15:08:52.728 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/dma/ioat/ 2024-02-19T15:08:52.728 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 122817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/dma/ioat/ioatdma.ko 2024-02-19T15:08:52.729 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/ 2024-02-19T15:08:52.729 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 80305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/amd64_edac.ko 2024-02-19T15:08:52.729 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 35673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/e752x_edac.ko 2024-02-19T15:08:52.729 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 41641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/edac_mce_amd.ko 2024-02-19T15:08:52.729 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/i3000_edac.ko 2024-02-19T15:08:52.729 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/i3200_edac.ko 2024-02-19T15:08:52.729 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/i5100_edac.ko 2024-02-19T15:08:52.729 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/i5400_edac.ko 2024-02-19T15:08:52.729 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/i7300_edac.ko 2024-02-19T15:08:52.729 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 51945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/i7core_edac.ko 2024-02-19T15:08:52.729 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/i82975x_edac.ko 2024-02-19T15:08:52.729 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/ie31200_edac.ko 2024-02-19T15:08:52.729 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 56737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/sb_edac.ko 2024-02-19T15:08:52.729 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/edac/x38_edac.ko 2024-02-19T15:08:52.729 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/extcon/ 2024-02-19T15:08:52.729 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/extcon/extcon-gpio.ko 2024-02-19T15:08:52.730 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 27089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/extcon/extcon-max14577.ko 2024-02-19T15:08:52.730 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/extcon/extcon-max77693.ko 2024-02-19T15:08:52.730 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/extcon/extcon-max8997.ko 2024-02-19T15:08:52.730 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/extcon/extcon-palmas.ko 2024-02-19T15:08:52.730 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/extcon/extcon-rt8973a.ko 2024-02-19T15:08:52.730 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 27689 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/extcon/extcon-sm5502.ko 2024-02-19T15:08:52.730 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/firmware/ 2024-02-19T15:08:52.730 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/firmware/dmi-sysfs.ko 2024-02-19T15:08:52.730 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/firmware/efi/ 2024-02-19T15:08:52.730 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/firmware/efi/efi-pstore.ko 2024-02-19T15:08:52.730 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/firmware/iscsi_ibft.ko 2024-02-19T15:08:52.730 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/ 2024-02-19T15:08:52.730 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-adp5520.ko 2024-02-19T15:08:52.730 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-amd8111.ko 2024-02-19T15:08:52.730 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-arizona.ko 2024-02-19T15:08:52.730 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-da9052.ko 2024-02-19T15:08:52.731 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-da9055.ko 2024-02-19T15:08:52.731 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-dln2.ko 2024-02-19T15:08:52.731 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 77753 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-f7188x.ko 2024-02-19T15:08:52.731 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-generic.ko 2024-02-19T15:08:52.731 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-ich.ko 2024-02-19T15:08:52.731 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-janz-ttl.ko 2024-02-19T15:08:52.731 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-kempld.ko 2024-02-19T15:08:52.731 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-lp3943.ko 2024-02-19T15:08:52.731 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-max7300.ko 2024-02-19T15:08:52.731 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-max730x.ko 2024-02-19T15:08:52.731 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-max732x.ko 2024-02-19T15:08:52.731 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-ml-ioh.ko 2024-02-19T15:08:52.731 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 47513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-pca953x.ko 2024-02-19T15:08:52.731 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-pcf857x.ko 2024-02-19T15:08:52.731 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-rdc321x.ko 2024-02-19T15:08:52.731 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-sch.ko 2024-02-19T15:08:52.731 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-sch311x.ko 2024-02-19T15:08:52.732 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-tps65912.ko 2024-02-19T15:08:52.732 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-twl4030.ko 2024-02-19T15:08:52.732 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-twl6040.ko 2024-02-19T15:08:52.732 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-viperboard.ko 2024-02-19T15:08:52.732 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-vx855.ko 2024-02-19T15:08:52.732 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-wm831x.ko 2024-02-19T15:08:52.732 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-wm8350.ko 2024-02-19T15:08:52.732 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/gpio/gpio-wm8994.ko 2024-02-19T15:08:52.732 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/ 2024-02-19T15:08:52.732 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-a4tech.ko 2024-02-19T15:08:52.732 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 39417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-apple.ko 2024-02-19T15:08:52.732 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-appleir.ko 2024-02-19T15:08:52.732 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-aureal.ko 2024-02-19T15:08:52.732 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-axff.ko 2024-02-19T15:08:52.733 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-belkin.ko 2024-02-19T15:08:52.733 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-cherry.ko 2024-02-19T15:08:52.733 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-chicony.ko 2024-02-19T15:08:52.733 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 60665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-cp2112.ko 2024-02-19T15:08:52.733 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-cypress.ko 2024-02-19T15:08:52.733 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-dr.ko 2024-02-19T15:08:52.733 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-elecom.ko 2024-02-19T15:08:52.733 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-elo.ko 2024-02-19T15:08:52.733 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-emsff.ko 2024-02-19T15:08:52.733 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-ezkey.ko 2024-02-19T15:08:52.733 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-gaff.ko 2024-02-19T15:08:52.733 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-generic.ko 2024-02-19T15:08:52.733 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-gt683r.ko 2024-02-19T15:08:52.733 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12585 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-gyration.ko 2024-02-19T15:08:52.734 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-holtek-kbd.ko 2024-02-19T15:08:52.734 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-holtek-mouse.ko 2024-02-19T15:08:52.734 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-holtekff.ko 2024-02-19T15:08:52.734 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-hyperv.ko 2024-02-19T15:08:52.734 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-icade.ko 2024-02-19T15:08:52.734 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-kensington.ko 2024-02-19T15:08:52.734 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-keytouch.ko 2024-02-19T15:08:52.734 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17113 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-kye.ko 2024-02-19T15:08:52.734 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-lcpower.ko 2024-02-19T15:08:52.734 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-led.ko 2024-02-19T15:08:52.734 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 42849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-lenovo.ko 2024-02-19T15:08:52.734 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-lg-g15.ko 2024-02-19T15:08:52.734 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 57977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-logitech-dj.ko 2024-02-19T15:08:52.734 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 100657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-02-19T15:08:52.734 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 80921 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-logitech.ko 2024-02-19T15:08:52.735 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-magicmouse.ko 2024-02-19T15:08:52.735 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-microsoft.ko 2024-02-19T15:08:52.735 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-monterey.ko 2024-02-19T15:08:52.735 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 50177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-multitouch.ko 2024-02-19T15:08:52.735 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 32497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-ntrig.ko 2024-02-19T15:08:52.735 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-ortek.ko 2024-02-19T15:08:52.735 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-penmount.ko 2024-02-19T15:08:52.735 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-petalynx.ko 2024-02-19T15:08:52.735 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 72865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-picolcd.ko 2024-02-19T15:08:52.735 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-pl.ko 2024-02-19T15:08:52.735 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-plantronics.ko 2024-02-19T15:08:52.735 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-primax.ko 2024-02-19T15:08:52.735 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 35529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-rmi.ko 2024-02-19T15:08:52.735 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-arvo.ko 2024-02-19T15:08:52.736 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-common.ko 2024-02-19T15:08:52.736 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-isku.ko 2024-02-19T15:08:52.736 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-kone.ko 2024-02-19T15:08:52.736 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-02-19T15:08:52.736 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-konepure.ko 2024-02-19T15:08:52.736 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-02-19T15:08:52.736 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-lua.ko 2024-02-19T15:08:52.736 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-pyra.ko 2024-02-19T15:08:52.736 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 27841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-ryos.ko 2024-02-19T15:08:52.736 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat-savu.ko 2024-02-19T15:08:52.736 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-roccat.ko 2024-02-19T15:08:52.736 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13585 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-saitek.ko 2024-02-19T15:08:52.736 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-samsung.ko 2024-02-19T15:08:52.736 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-sensor-hub.ko 2024-02-19T15:08:52.736 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-sjoy.ko 2024-02-19T15:08:52.736 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 51161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-sony.ko 2024-02-19T15:08:52.737 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-speedlink.ko 2024-02-19T15:08:52.737 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-steelseries.ko 2024-02-19T15:08:52.737 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-sunplus.ko 2024-02-19T15:08:52.737 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-thrustmaster.ko 2024-02-19T15:08:52.737 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-tivo.ko 2024-02-19T15:08:52.737 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-tmff.ko 2024-02-19T15:08:52.737 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-topseed.ko 2024-02-19T15:08:52.737 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-twinhan.ko 2024-02-19T15:08:52.737 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 70993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-uclogic.ko 2024-02-19T15:08:52.737 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-waltop.ko 2024-02-19T15:08:52.737 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 143393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-wiimote.ko 2024-02-19T15:08:52.737 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8065 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-xinmo.ko 2024-02-19T15:08:52.737 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-zpff.ko 2024-02-19T15:08:52.737 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid-zydacron.ko 2024-02-19T15:08:52.737 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 269233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/hid.ko 2024-02-19T15:08:52.738 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 32233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/uhid.ko 2024-02-19T15:08:52.738 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/usbhid/ 2024-02-19T15:08:52.738 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 125641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/usbhid/usbhid.ko 2024-02-19T15:08:52.738 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/usbhid/usbkbd.ko 2024-02-19T15:08:52.738 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/usbhid/usbmouse.ko 2024-02-19T15:08:52.738 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 218817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hid/wacom.ko 2024-02-19T15:08:52.738 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hv/ 2024-02-19T15:08:52.738 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 64305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hv/hv_balloon.ko 2024-02-19T15:08:52.738 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 89553 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hv/hv_utils.ko 2024-02-19T15:08:52.738 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 305425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hv/hv_vmbus.ko 2024-02-19T15:08:52.738 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ 2024-02-19T15:08:52.738 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 66801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/abituguru.ko 2024-02-19T15:08:52.738 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 61817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/abituguru3.ko 2024-02-19T15:08:52.738 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/acpi_power_meter.ko 2024-02-19T15:08:52.738 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ad7414.ko 2024-02-19T15:08:52.739 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ad7418.ko 2024-02-19T15:08:52.739 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adc128d818.ko 2024-02-19T15:08:52.739 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adm1025.ko 2024-02-19T15:08:52.739 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 97889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adm1026.ko 2024-02-19T15:08:52.739 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adm1029.ko 2024-02-19T15:08:52.739 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 45465 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adm1031.ko 2024-02-19T15:08:52.739 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adm9240.ko 2024-02-19T15:08:52.739 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ads7828.ko 2024-02-19T15:08:52.739 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adt7410.ko 2024-02-19T15:08:52.739 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adt7411.ko 2024-02-19T15:08:52.739 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 74321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adt7462.ko 2024-02-19T15:08:52.739 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 39497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adt7470.ko 2024-02-19T15:08:52.739 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 69313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adt7475.ko 2024-02-19T15:08:52.739 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/adt7x10.ko 2024-02-19T15:08:52.740 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 38985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/amc6821.ko 2024-02-19T15:08:52.740 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 55177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/applesmc.ko 2024-02-19T15:08:52.740 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 49825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/asb100.ko 2024-02-19T15:08:52.740 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 58529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/asc7621.ko 2024-02-19T15:08:52.740 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 40793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/asus_atk0110.ko 2024-02-19T15:08:52.740 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/atxp1.ko 2024-02-19T15:08:52.740 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/coretemp.ko 2024-02-19T15:08:52.740 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/da9052-hwmon.ko 2024-02-19T15:08:52.740 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/da9055-hwmon.ko 2024-02-19T15:08:52.740 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 49241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-02-19T15:08:52.740 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 99513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/dme1737.ko 2024-02-19T15:08:52.740 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ds1621.ko 2024-02-19T15:08:52.740 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ds620.ko 2024-02-19T15:08:52.740 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/emc1403.ko 2024-02-19T15:08:52.740 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 33097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/emc2103.ko 2024-02-19T15:08:52.741 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/emc6w201.ko 2024-02-19T15:08:52.741 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 66897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/f71805f.ko 2024-02-19T15:08:52.741 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 94601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/f71882fg.ko 2024-02-19T15:08:52.741 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 41313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/f75375s.ko 2024-02-19T15:08:52.741 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/fam15h_power.ko 2024-02-19T15:08:52.741 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 55361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/fschmd.ko 2024-02-19T15:08:52.741 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/g760a.ko 2024-02-19T15:08:52.741 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/g762.ko 2024-02-19T15:08:52.741 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 46713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/gl518sm.ko 2024-02-19T15:08:52.741 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 45441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/gl520sm.ko 2024-02-19T15:08:52.741 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/hih6130.ko 2024-02-19T15:08:52.741 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/hwmon-vid.ko 2024-02-19T15:08:52.741 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/i5500_temp.ko 2024-02-19T15:08:52.741 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/i5k_amb.ko 2024-02-19T15:08:52.742 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ibmaem.ko 2024-02-19T15:08:52.742 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ibmpex.ko 2024-02-19T15:08:52.742 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ina209.ko 2024-02-19T15:08:52.742 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ina2xx.ko 2024-02-19T15:08:52.742 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 140137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/it87.ko 2024-02-19T15:08:52.742 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18753 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/jc42.ko 2024-02-19T15:08:52.742 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/k10temp.ko 2024-02-19T15:08:52.742 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/k8temp.ko 2024-02-19T15:08:52.742 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lineage-pem.ko 2024-02-19T15:08:52.742 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 48513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm63.ko 2024-02-19T15:08:52.742 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm73.ko 2024-02-19T15:08:52.742 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 38849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm75.ko 2024-02-19T15:08:52.742 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm77.ko 2024-02-19T15:08:52.742 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm78.ko 2024-02-19T15:08:52.742 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm80.ko 2024-02-19T15:08:52.743 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm83.ko 2024-02-19T15:08:52.743 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 72745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm85.ko 2024-02-19T15:08:52.743 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 46065 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm87.ko 2024-02-19T15:08:52.743 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 37825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm90.ko 2024-02-19T15:08:52.743 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm92.ko 2024-02-19T15:08:52.743 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 108753 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm93.ko 2024-02-19T15:08:52.743 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 33305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm95234.ko 2024-02-19T15:08:52.743 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm95241.ko 2024-02-19T15:08:52.743 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/lm95245.ko 2024-02-19T15:08:52.743 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ltc2945.ko 2024-02-19T15:08:52.743 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ltc4151.ko 2024-02-19T15:08:52.743 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15105 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ltc4215.ko 2024-02-19T15:08:52.743 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ltc4222.ko 2024-02-19T15:08:52.743 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ltc4245.ko 2024-02-19T15:08:52.744 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ltc4260.ko 2024-02-19T15:08:52.744 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/ltc4261.ko 2024-02-19T15:08:52.744 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 33401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/max16065.ko 2024-02-19T15:08:52.744 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/max1619.ko 2024-02-19T15:08:52.744 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/max1668.ko 2024-02-19T15:08:52.744 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/max197.ko 2024-02-19T15:08:52.744 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 27041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/max6639.ko 2024-02-19T15:08:52.744 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/max6650.ko 2024-02-19T15:08:52.744 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 35689 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/max6697.ko 2024-02-19T15:08:52.744 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/mc13783-adc.ko 2024-02-19T15:08:52.744 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/mcp3021.ko 2024-02-19T15:08:52.744 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2024-02-19T15:08:52.744 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 47793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/nct6683.ko 2024-02-19T15:08:52.744 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 119721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/nct6775-core.ko 2024-02-19T15:08:52.745 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 61441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/nct6775.ko 2024-02-19T15:08:52.745 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 59649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/nct7802.ko 2024-02-19T15:08:52.745 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 82721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pc87360.ko 2024-02-19T15:08:52.745 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 54345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pc87427.ko 2024-02-19T15:08:52.745 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pcf8591.ko 2024-02-19T15:08:52.745 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/ 2024-02-19T15:08:52.745 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-02-19T15:08:52.745 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-02-19T15:08:52.745 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-02-19T15:08:52.745 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/max16064.ko 2024-02-19T15:08:52.745 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/max34440.ko 2024-02-19T15:08:52.745 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/max8688.ko 2024-02-19T15:08:52.745 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-02-19T15:08:52.745 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 82481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-02-19T15:08:52.746 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-02-19T15:08:52.746 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-02-19T15:08:52.746 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-02-19T15:08:52.746 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20593 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-02-19T15:08:52.746 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/powr1220.ko 2024-02-19T15:08:52.746 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/sch5627.ko 2024-02-19T15:08:52.746 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/sch5636.ko 2024-02-19T15:08:52.746 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/sch56xx-common.ko 2024-02-19T15:08:52.746 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 32049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/sht15.ko 2024-02-19T15:08:52.746 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/sht21.ko 2024-02-19T15:08:52.746 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/shtc1.ko 2024-02-19T15:08:52.746 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 40489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/sis5595.ko 2024-02-19T15:08:52.746 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/smsc47b397.ko 2024-02-19T15:08:52.746 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 40361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/smsc47m1.ko 2024-02-19T15:08:52.747 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 37289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/smsc47m192.ko 2024-02-19T15:08:52.747 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26465 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/thmc50.ko 2024-02-19T15:08:52.747 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/tmp102.ko 2024-02-19T15:08:52.747 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/tmp103.ko 2024-02-19T15:08:52.747 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/tmp401.ko 2024-02-19T15:08:52.747 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/tmp421.ko 2024-02-19T15:08:52.747 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/via-cputemp.ko 2024-02-19T15:08:52.747 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 43449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/via686a.ko 2024-02-19T15:08:52.747 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 52369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/vt1211.ko 2024-02-19T15:08:52.747 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 53209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/vt8231.ko 2024-02-19T15:08:52.747 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 58265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/w83627ehf.ko 2024-02-19T15:08:52.747 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 79841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/w83627hf.ko 2024-02-19T15:08:52.747 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 71257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/w83781d.ko 2024-02-19T15:08:52.747 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 63201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/w83791d.ko 2024-02-19T15:08:52.747 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 71233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/w83792d.ko 2024-02-19T15:08:52.748 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 101385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/w83793.ko 2024-02-19T15:08:52.748 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 119417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/w83795.ko 2024-02-19T15:08:52.748 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/w83l785ts.ko 2024-02-19T15:08:52.748 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 35529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/w83l786ng.ko 2024-02-19T15:08:52.748 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/wm831x-hwmon.ko 2024-02-19T15:08:52.748 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/hwmon/wm8350-hwmon.ko 2024-02-19T15:08:52.748 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/ 2024-02-19T15:08:52.748 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/algos/ 2024-02-19T15:08:52.748 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-02-19T15:08:52.748 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-02-19T15:08:52.748 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/ 2024-02-19T15:08:52.748 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-ali1535.ko 2024-02-19T15:08:52.748 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 32481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-ali1563.ko 2024-02-19T15:08:52.748 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2024-02-19T15:08:52.748 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-02-19T15:08:52.749 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-02-19T15:08:52.749 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-02-19T15:08:52.749 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2024-02-19T15:08:52.749 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 6833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-02-19T15:08:52.749 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2024-02-19T15:08:52.749 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 50305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-02-19T15:08:52.749 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-02-19T15:08:52.749 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-02-19T15:08:52.749 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-02-19T15:08:52.749 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-dln2.ko 2024-02-19T15:08:52.749 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-gpio.ko 2024-02-19T15:08:52.749 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 67417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-i801.ko 2024-02-19T15:08:52.749 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-isch.ko 2024-02-19T15:08:52.749 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 47801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-02-19T15:08:52.749 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-kempld.ko 2024-02-19T15:08:52.749 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-02-19T15:08:52.749 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-02-19T15:08:52.750 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-ocores.ko 2024-02-19T15:08:52.750 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-parport.ko 2024-02-19T15:08:52.750 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-02-19T15:08:52.750 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 50977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-02-19T15:08:52.750 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2024-02-19T15:08:52.750 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-02-19T15:08:52.750 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-02-19T15:08:52.750 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-sis5595.ko 2024-02-19T15:08:52.750 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-sis630.ko 2024-02-19T15:08:52.750 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-02-19T15:08:52.750 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2024-02-19T15:08:52.750 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-02-19T15:08:52.750 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-via.ko 2024-02-19T15:08:52.750 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 27625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-02-19T15:08:52.750 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-02-19T15:08:52.750 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 47569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/busses/i2c-xiic.ko 2024-02-19T15:08:52.750 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/i2c-mux.ko 2024-02-19T15:08:52.751 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/i2c-smbus.ko 2024-02-19T15:08:52.751 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/i2c-stub.ko 2024-02-19T15:08:52.751 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/muxes/ 2024-02-19T15:08:52.751 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2024-02-19T15:08:52.751 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2024-02-19T15:08:52.751 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2024-02-19T15:08:52.751 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ 2024-02-19T15:08:52.751 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/core/ 2024-02-19T15:08:52.751 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 273529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/core/ib_cm.ko 2024-02-19T15:08:52.751 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 883945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/core/ib_core.ko 2024-02-19T15:08:52.751 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 69185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/core/ib_umad.ko 2024-02-19T15:08:52.751 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 324337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-02-19T15:08:52.751 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 97841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/core/iw_cm.ko 2024-02-19T15:08:52.751 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 282345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/core/rdma_cm.ko 2024-02-19T15:08:52.751 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 66057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-02-19T15:08:52.751 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/ 2024-02-19T15:08:52.752 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/cxgb4/ 2024-02-19T15:08:52.752 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 493177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-02-19T15:08:52.752 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/mlx4/ 2024-02-19T15:08:52.752 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 420593 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-02-19T15:08:52.752 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/mlx5/ 2024-02-19T15:08:52.752 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 741673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-02-19T15:08:52.752 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/mthca/ 2024-02-19T15:08:52.752 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 296881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-02-19T15:08:52.752 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/ocrdma/ 2024-02-19T15:08:52.752 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 153729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2024-02-19T15:08:52.752 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/usnic/ 2024-02-19T15:08:52.752 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 169145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2024-02-19T15:08:52.752 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/ 2024-02-19T15:08:52.752 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/ipoib/ 2024-02-19T15:08:52.753 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 266577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-02-19T15:08:52.812 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/iser/ 2024-02-19T15:08:52.812 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 100281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-02-19T15:08:52.812 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/isert/ 2024-02-19T15:08:52.812 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 111337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-02-19T15:08:52.813 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/srp/ 2024-02-19T15:08:52.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 128665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-02-19T15:08:52.813 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/srpt/ 2024-02-19T15:08:52.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 129553 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-02-19T15:08:52.813 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/ 2024-02-19T15:08:52.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/evbug.ko 2024-02-19T15:08:52.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/ff-memless.ko 2024-02-19T15:08:52.813 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/gameport/ 2024-02-19T15:08:52.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-02-19T15:08:52.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/gameport/fm801-gp.ko 2024-02-19T15:08:52.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 41457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/gameport/gameport.ko 2024-02-19T15:08:52.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 41713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/joydev.ko 2024-02-19T15:08:52.813 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/ 2024-02-19T15:08:52.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/adp5520-keys.ko 2024-02-19T15:08:52.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12609 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/adp5588-keys.ko 2024-02-19T15:08:52.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/adp5589-keys.ko 2024-02-19T15:08:52.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2024-02-19T15:08:52.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 37281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/gpio_keys.ko 2024-02-19T15:08:52.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2024-02-19T15:08:52.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/lkkbd.ko 2024-02-19T15:08:52.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/lm8323.ko 2024-02-19T15:08:52.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/lm8333.ko 2024-02-19T15:08:52.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/matrix_keypad.ko 2024-02-19T15:08:52.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/max7359_keypad.ko 2024-02-19T15:08:52.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/mcs_touchkey.ko 2024-02-19T15:08:52.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2024-02-19T15:08:52.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/newtonkbd.ko 2024-02-19T15:08:52.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/opencores-kbd.ko 2024-02-19T15:08:52.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/qt1070.ko 2024-02-19T15:08:52.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/qt2160.ko 2024-02-19T15:08:52.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/samsung-keypad.ko 2024-02-19T15:08:52.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/stowaway.ko 2024-02-19T15:08:52.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/sunkbd.ko 2024-02-19T15:08:52.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/tca6416-keypad.ko 2024-02-19T15:08:52.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/tca8418_keypad.ko 2024-02-19T15:08:52.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/twl4030_keypad.ko 2024-02-19T15:08:52.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/keyboard/xtkbd.ko 2024-02-19T15:08:52.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/matrix-keymap.ko 2024-02-19T15:08:52.815 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/ 2024-02-19T15:08:52.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/appletouch.ko 2024-02-19T15:08:52.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/bcm5974.ko 2024-02-19T15:08:52.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 123201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/cyapatp.ko 2024-02-19T15:08:52.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 89393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/elan_i2c.ko 2024-02-19T15:08:52.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/gpio_mouse.ko 2024-02-19T15:08:52.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 335921 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/psmouse.ko 2024-02-19T15:08:52.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14593 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/sermouse.ko 2024-02-19T15:08:52.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-02-19T15:08:52.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/synaptics_usb.ko 2024-02-19T15:08:52.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/mouse/vsxxxaa.ko 2024-02-19T15:08:52.816 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/rmi4/ 2024-02-19T15:08:52.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 109601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/rmi4/rmi_core.ko 2024-02-19T15:08:52.816 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/serio/ 2024-02-19T15:08:52.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/serio/altera_ps2.ko 2024-02-19T15:08:52.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/serio/arc_ps2.ko 2024-02-19T15:08:52.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17689 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/serio/ct82c710.ko 2024-02-19T15:08:52.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-02-19T15:08:52.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/serio/parkbd.ko 2024-02-19T15:08:52.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/serio/pcips2.ko 2024-02-19T15:08:52.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/serio/ps2mult.ko 2024-02-19T15:08:52.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/serio/serio_raw.ko 2024-02-19T15:08:52.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/serio/serport.ko 2024-02-19T15:08:52.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/input/sparse-keymap.ko 2024-02-19T15:08:52.817 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/ 2024-02-19T15:08:52.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-88pm860x.ko 2024-02-19T15:08:52.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-adp5520.ko 2024-02-19T15:08:52.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 59657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-bd2802.ko 2024-02-19T15:08:52.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24689 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-blinkm.ko 2024-02-19T15:08:52.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-da903x.ko 2024-02-19T15:08:52.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-da9052.ko 2024-02-19T15:08:52.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16689 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-gpio.ko 2024-02-19T15:08:52.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-lm3530.ko 2024-02-19T15:08:52.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-lm3533.ko 2024-02-19T15:08:52.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-lm355x.ko 2024-02-19T15:08:52.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-lm3642.ko 2024-02-19T15:08:52.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-lp3944.ko 2024-02-19T15:08:52.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-lp8788.ko 2024-02-19T15:08:52.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-max8997.ko 2024-02-19T15:08:52.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-mc13783.ko 2024-02-19T15:08:52.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-menf21bmc.ko 2024-02-19T15:08:52.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-pca9532.ko 2024-02-19T15:08:52.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20465 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-pca955x.ko 2024-02-19T15:08:52.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-pca963x.ko 2024-02-19T15:08:52.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-ss4200.ko 2024-02-19T15:08:52.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-tca6507.ko 2024-02-19T15:08:52.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-wm831x-status.ko 2024-02-19T15:08:52.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11753 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/leds-wm8350.ko 2024-02-19T15:08:52.818 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/trigger/ 2024-02-19T15:08:52.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-02-19T15:08:52.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-02-19T15:08:52.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-02-19T15:08:52.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-02-19T15:08:52.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-02-19T15:08:52.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-02-19T15:08:52.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-02-19T15:08:52.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-02-19T15:08:52.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-02-19T15:08:52.819 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/ 2024-02-19T15:08:52.819 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/bcache/ 2024-02-19T15:08:52.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 647121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/bcache/bcache.ko 2024-02-19T15:08:52.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 40489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-bio-prison.ko 2024-02-19T15:08:52.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 93321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-bufio.ko 2024-02-19T15:08:52.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 47441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-cache-smq.ko 2024-02-19T15:08:52.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 155825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-cache.ko 2024-02-19T15:08:52.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 105473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-crypt.ko 2024-02-19T15:08:52.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-delay.ko 2024-02-19T15:08:52.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 53769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-era.ko 2024-02-19T15:08:52.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-flakey.ko 2024-02-19T15:08:52.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 41505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-log-userspace.ko 2024-02-19T15:08:52.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 32649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-log.ko 2024-02-19T15:08:52.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 49449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-mirror.ko 2024-02-19T15:08:52.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 77881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-multipath.ko 2024-02-19T15:08:52.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-queue-length.ko 2024-02-19T15:08:52.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 70985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-raid.ko 2024-02-19T15:08:52.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-region-hash.ko 2024-02-19T15:08:52.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-round-robin.ko 2024-02-19T15:08:52.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-service-time.ko 2024-02-19T15:08:52.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 109521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-snapshot.ko 2024-02-19T15:08:52.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-switch.ko 2024-02-19T15:08:52.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 181777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-thin-pool.ko 2024-02-19T15:08:52.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 50545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-verity.ko 2024-02-19T15:08:52.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/dm-zero.ko 2024-02-19T15:08:52.821 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/persistent-data/ 2024-02-19T15:08:52.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 199881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-02-19T15:08:52.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 37321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/raid0.ko 2024-02-19T15:08:52.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 100841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/raid1.ko 2024-02-19T15:08:52.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 126761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/raid10.ko 2024-02-19T15:08:52.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 369433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/md/raid456.ko 2024-02-19T15:08:52.821 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/message/ 2024-02-19T15:08:52.821 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/message/fusion/ 2024-02-19T15:08:52.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 178049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/message/fusion/mptbase.ko 2024-02-19T15:08:52.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 63241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/message/fusion/mptctl.ko 2024-02-19T15:08:52.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 46249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/message/fusion/mptfc.ko 2024-02-19T15:08:52.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 37121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/message/fusion/mptlan.ko 2024-02-19T15:08:52.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 123233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/message/fusion/mptsas.ko 2024-02-19T15:08:52.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 80985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/message/fusion/mptscsih.ko 2024-02-19T15:08:52.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 52929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/message/fusion/mptspi.ko 2024-02-19T15:08:52.822 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/ 2024-02-19T15:08:52.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/88pm800.ko 2024-02-19T15:08:52.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/88pm805.ko 2024-02-19T15:08:52.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11105 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/88pm80x.ko 2024-02-19T15:08:52.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/arizona-i2c.ko 2024-02-19T15:08:52.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 123649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/arizona.ko 2024-02-19T15:08:52.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/bcm590xx.ko 2024-02-19T15:08:52.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/cros_ec_dev.ko 2024-02-19T15:08:52.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/dln2.ko 2024-02-19T15:08:52.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16593 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/janz-cmodio.ko 2024-02-19T15:08:52.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 53489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/kempld-core.ko 2024-02-19T15:08:52.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 27433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/lm3533-core.ko 2024-02-19T15:08:52.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/lm3533-ctrlbank.ko 2024-02-19T15:08:52.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12921 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/lp3943.ko 2024-02-19T15:08:52.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 56969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/lpc_ich.ko 2024-02-19T15:08:52.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/lpc_sch.ko 2024-02-19T15:08:52.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/max8907.ko 2024-02-19T15:08:52.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/mc13xxx-core.ko 2024-02-19T15:08:52.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/mc13xxx-i2c.ko 2024-02-19T15:08:52.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/menf21bmc.ko 2024-02-19T15:08:52.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/pcf50633-adc.ko 2024-02-19T15:08:52.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/pcf50633-gpio.ko 2024-02-19T15:08:52.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/pcf50633.ko 2024-02-19T15:08:52.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/rdc321x-southbridge.ko 2024-02-19T15:08:52.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/retu-mfd.ko 2024-02-19T15:08:52.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 80345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/si476x-core.ko 2024-02-19T15:08:52.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 48369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/sm501.ko 2024-02-19T15:08:52.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 51209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/tps65010.ko 2024-02-19T15:08:52.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/tps6507x.ko 2024-02-19T15:08:52.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/viperboard.ko 2024-02-19T15:08:52.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/vx855.ko 2024-02-19T15:08:52.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/wl1273-core.ko 2024-02-19T15:08:52.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 44265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/mfd/wm8994.ko 2024-02-19T15:08:52.824 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/ 2024-02-19T15:08:52.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/ad525x_dpot-i2c.ko 2024-02-19T15:08:52.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 48393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/ad525x_dpot.ko 2024-02-19T15:08:52.824 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/altera-stapl/ 2024-02-19T15:08:52.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 46513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-02-19T15:08:52.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/apds9802als.ko 2024-02-19T15:08:52.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/apds990x.ko 2024-02-19T15:08:52.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 45497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/bh1770glc.ko 2024-02-19T15:08:52.824 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/c2port/ 2024-02-19T15:08:52.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2024-02-19T15:08:52.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 35561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/c2port/core.ko 2024-02-19T15:08:52.825 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/cb710/ 2024-02-19T15:08:52.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/cb710/cb710.ko 2024-02-19T15:08:52.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/ds1682.ko 2024-02-19T15:08:52.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/dummy-irq.ko 2024-02-19T15:08:52.825 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/echo/ 2024-02-19T15:08:52.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/echo/echo.ko 2024-02-19T15:08:52.825 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/eeprom/ 2024-02-19T15:08:52.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 39025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/eeprom/at24.ko 2024-02-19T15:08:52.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-02-19T15:08:52.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/eeprom/max6875.ko 2024-02-19T15:08:52.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 32913 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/enclosure.ko 2024-02-19T15:08:52.825 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/genwqe/ 2024-02-19T15:08:52.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 152073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/genwqe/genwqe_card.ko 2024-02-19T15:08:52.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/hmc6352.ko 2024-02-19T15:08:52.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/hpilo.ko 2024-02-19T15:08:52.826 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/ibmasm/ 2024-02-19T15:08:52.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 72561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/ibmasm/ibmasm.ko 2024-02-19T15:08:52.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19689 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/ics932s401.ko 2024-02-19T15:08:52.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21593 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/isl29003.ko 2024-02-19T15:08:52.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/isl29020.ko 2024-02-19T15:08:52.826 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/lis3lv02d/ 2024-02-19T15:08:52.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 42417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-02-19T15:08:52.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-02-19T15:08:52.826 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/mei/ 2024-02-19T15:08:52.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 125865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/mei/mei-me.ko 2024-02-19T15:08:52.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 56377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/mei/mei-txe.ko 2024-02-19T15:08:52.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 313601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/mei/mei.ko 2024-02-19T15:08:52.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/phantom.ko 2024-02-19T15:08:52.826 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/ti-st/ 2024-02-19T15:08:52.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 74305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/ti-st/st_drv.ko 2024-02-19T15:08:52.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/tifm_7xx1.ko 2024-02-19T15:08:52.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28689 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/tifm_core.ko 2024-02-19T15:08:52.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/tsl2550.ko 2024-02-19T15:08:52.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 48345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/vmw_balloon.ko 2024-02-19T15:08:52.827 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/vmw_vmci/ 2024-02-19T15:08:52.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 183569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-02-19T15:08:52.827 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ 2024-02-19T15:08:52.827 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/bonding/ 2024-02-19T15:08:52.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 491721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/bonding/bonding.ko 2024-02-19T15:08:52.827 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/dsa/ 2024-02-19T15:08:52.827 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/dsa/b53/ 2024-02-19T15:08:52.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 105937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/dsa/b53/b53_common.ko 2024-02-19T15:08:52.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 40225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/dsa/bcm-sf2.ko 2024-02-19T15:08:52.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/dsa/mv88e6060.ko 2024-02-19T15:08:52.827 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/dsa/mv88e6xxx/ 2024-02-19T15:08:52.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 459065 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-02-19T15:08:52.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/dummy.ko 2024-02-19T15:08:52.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/eql.ko 2024-02-19T15:08:52.828 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/ 2024-02-19T15:08:52.828 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/3com/ 2024-02-19T15:08:52.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 45961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-02-19T15:08:52.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 42193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-02-19T15:08:52.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 130665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-02-19T15:08:52.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 68785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-02-19T15:08:52.828 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/8390/ 2024-02-19T15:08:52.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 38049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/8390/8390.ko 2024-02-19T15:08:52.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 64617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-02-19T15:08:52.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 27089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-02-19T15:08:52.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 101681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-02-19T15:08:52.829 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/adaptec/ 2024-02-19T15:08:52.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 54625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-02-19T15:08:52.829 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/agere/ 2024-02-19T15:08:52.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 69169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/agere/et131x.ko 2024-02-19T15:08:52.829 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/alteon/ 2024-02-19T15:08:52.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 59993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-02-19T15:08:52.829 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/altera/ 2024-02-19T15:08:52.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 41409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-02-19T15:08:52.829 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/amd/ 2024-02-19T15:08:52.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 54617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-02-19T15:08:52.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 49193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-02-19T15:08:52.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 87457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-02-19T15:08:52.829 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/atheros/ 2024-02-19T15:08:52.829 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/atheros/alx/ 2024-02-19T15:08:52.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 97201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-02-19T15:08:52.830 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/atheros/atl1c/ 2024-02-19T15:08:52.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 102497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-02-19T15:08:52.830 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/atheros/atl1e/ 2024-02-19T15:08:52.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 90209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-02-19T15:08:52.830 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/atheros/atlx/ 2024-02-19T15:08:52.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 81321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-02-19T15:08:52.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 69545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-02-19T15:08:52.830 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/broadcom/ 2024-02-19T15:08:52.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 78385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-02-19T15:08:52.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 176697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-02-19T15:08:52.830 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2024-02-19T15:08:52.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 1434505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-02-19T15:08:52.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 128433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-02-19T15:08:52.830 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/broadcom/genet/ 2024-02-19T15:08:52.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 131281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-02-19T15:08:52.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 314329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-02-19T15:08:52.831 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/brocade/ 2024-02-19T15:08:52.831 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/brocade/bna/ 2024-02-19T15:08:52.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 323737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-02-19T15:08:52.831 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/ 2024-02-19T15:08:52.831 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/cxgb/ 2024-02-19T15:08:52.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 193593 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-02-19T15:08:52.831 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2024-02-19T15:08:52.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 347217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-02-19T15:08:52.831 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2024-02-19T15:08:52.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 1045449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-02-19T15:08:52.831 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2024-02-19T15:08:52.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 166145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-02-19T15:08:52.831 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2024-02-19T15:08:52.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-02-19T15:08:52.832 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/cisco/ 2024-02-19T15:08:52.832 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/cisco/enic/ 2024-02-19T15:08:52.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 204249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-02-19T15:08:52.832 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dec/ 2024-02-19T15:08:52.832 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dec/tulip/ 2024-02-19T15:08:52.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 69433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-02-19T15:08:52.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 61425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-02-19T15:08:52.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 141201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-02-19T15:08:52.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 51369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-02-19T15:08:52.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 62833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-02-19T15:08:52.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-02-19T15:08:52.832 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dlink/ 2024-02-19T15:08:52.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 60673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-02-19T15:08:52.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 62833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-02-19T15:08:52.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 45489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/dnet.ko 2024-02-19T15:08:52.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/ec_bhf.ko 2024-02-19T15:08:52.833 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/emulex/ 2024-02-19T15:08:52.833 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/emulex/benet/ 2024-02-19T15:08:52.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 281217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-02-19T15:08:52.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 49457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/ethoc.ko 2024-02-19T15:08:52.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 51785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/fealnx.ko 2024-02-19T15:08:52.833 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/fujitsu/ 2024-02-19T15:08:52.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 45553 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2024-02-19T15:08:52.833 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/ 2024-02-19T15:08:52.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 90305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/e100.ko 2024-02-19T15:08:52.833 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/e1000/ 2024-02-19T15:08:52.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 295697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-02-19T15:08:52.834 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/e1000e/ 2024-02-19T15:08:52.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 584353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-02-19T15:08:52.834 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/fm10k/ 2024-02-19T15:08:52.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 246041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-02-19T15:08:52.834 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/i40e/ 2024-02-19T15:08:52.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 944865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-02-19T15:08:52.834 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/iavf/ 2024-02-19T15:08:52.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 288529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-02-19T15:08:52.834 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/igb/ 2024-02-19T15:08:52.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 513857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-02-19T15:08:52.834 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/igbvf/ 2024-02-19T15:08:52.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 93489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-02-19T15:08:52.834 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/ixgbe/ 2024-02-19T15:08:52.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 734729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-02-19T15:08:52.834 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/ixgbevf/ 2024-02-19T15:08:52.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 151745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-02-19T15:08:52.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 86993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/jme.ko 2024-02-19T15:08:52.835 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/marvell/ 2024-02-19T15:08:52.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-02-19T15:08:52.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 102441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/marvell/skge.ko 2024-02-19T15:08:52.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 122609 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-02-19T15:08:52.835 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/mellanox/ 2024-02-19T15:08:52.835 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/mellanox/mlx4/ 2024-02-19T15:08:52.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 727481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-02-19T15:08:52.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 278881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-02-19T15:08:52.835 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/mellanox/mlx5/ 2024-02-19T15:08:52.835 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2024-02-19T15:08:52.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 1121769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-02-19T15:08:52.835 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/micrel/ 2024-02-19T15:08:52.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 52985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/micrel/ks8842.ko 2024-02-19T15:08:52.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 40881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2024-02-19T15:08:52.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2024-02-19T15:08:52.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 95577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-02-19T15:08:52.836 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/myricom/ 2024-02-19T15:08:52.836 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/myricom/myri10ge/ 2024-02-19T15:08:52.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 108409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-02-19T15:08:52.836 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/natsemi/ 2024-02-19T15:08:52.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 73617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-02-19T15:08:52.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 47801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-02-19T15:08:52.836 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/neterion/ 2024-02-19T15:08:52.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 151097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-02-19T15:08:52.836 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/nvidia/ 2024-02-19T15:08:52.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 136801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-02-19T15:08:52.836 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/packetengines/ 2024-02-19T15:08:52.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 50433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-02-19T15:08:52.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 42985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-02-19T15:08:52.837 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/qlogic/ 2024-02-19T15:08:52.837 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/qlogic/netxen/ 2024-02-19T15:08:52.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 227865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-02-19T15:08:52.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 87017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-02-19T15:08:52.837 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2024-02-19T15:08:52.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 672057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-02-19T15:08:52.837 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/rdc/ 2024-02-19T15:08:52.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 42937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-02-19T15:08:52.837 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/realtek/ 2024-02-19T15:08:52.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 67281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-02-19T15:08:52.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 83809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-02-19T15:08:52.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/realtek/atp.ko 2024-02-19T15:08:52.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 194073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-02-19T15:08:52.837 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/samsung/ 2024-02-19T15:08:52.837 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/samsung/sxgbe/ 2024-02-19T15:08:52.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 104049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2024-02-19T15:08:52.837 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/sfc/ 2024-02-19T15:08:52.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 1067001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-02-19T15:08:52.838 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/silan/ 2024-02-19T15:08:52.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 47009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-02-19T15:08:52.838 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/sis/ 2024-02-19T15:08:52.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 54729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/sis/sis190.ko 2024-02-19T15:08:52.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 63201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/sis/sis900.ko 2024-02-19T15:08:52.838 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/smsc/ 2024-02-19T15:08:52.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 61425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-02-19T15:08:52.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 67297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-02-19T15:08:52.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 73257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-02-19T15:08:52.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 68049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-02-19T15:08:52.838 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/stmicro/ 2024-02-19T15:08:52.838 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/stmicro/stmmac/ 2024-02-19T15:08:52.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2024-02-19T15:08:52.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 39385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-02-19T15:08:52.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2024-02-19T15:08:52.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 497201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-02-19T15:08:52.839 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/sun/ 2024-02-19T15:08:52.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 103185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/sun/cassini.ko 2024-02-19T15:08:52.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 166049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/sun/niu.ko 2024-02-19T15:08:52.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 66273 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/sun/sungem.ko 2024-02-19T15:08:52.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 78721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-02-19T15:08:52.839 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/tehuti/ 2024-02-19T15:08:52.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 50017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-02-19T15:08:52.839 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/ti/ 2024-02-19T15:08:52.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 92345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/ti/tlan.ko 2024-02-19T15:08:52.839 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/via/ 2024-02-19T15:08:52.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 80001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-02-19T15:08:52.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 83721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-02-19T15:08:52.840 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/wiznet/ 2024-02-19T15:08:52.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 46313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-02-19T15:08:52.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-02-19T15:08:52.840 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/xircom/ 2024-02-19T15:08:52.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 59921 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-02-19T15:08:52.840 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/fddi/ 2024-02-19T15:08:52.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 41185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/fddi/defxx.ko 2024-02-19T15:08:52.840 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/fddi/skfp/ 2024-02-19T15:08:52.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 217065 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/fddi/skfp/skfp.ko 2024-02-19T15:08:52.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 74729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/geneve.ko 2024-02-19T15:08:52.840 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/hyperv/ 2024-02-19T15:08:52.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 237265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-02-19T15:08:52.840 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ieee802154/ 2024-02-19T15:08:52.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ieee802154/fakelb.ko 2024-02-19T15:08:52.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ifb.ko 2024-02-19T15:08:52.841 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ipvlan/ 2024-02-19T15:08:52.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 85857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ipvlan/ipvlan.ko 2024-02-19T15:08:52.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 62897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/macvlan.ko 2024-02-19T15:08:52.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/macvtap.ko 2024-02-19T15:08:52.841 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/mdio/ 2024-02-19T15:08:52.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-02-19T15:08:52.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-02-19T15:08:52.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/mdio/mdio-gpio.ko 2024-02-19T15:08:52.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/mdio/mdio-regmap.ko 2024-02-19T15:08:52.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/mdio.ko 2024-02-19T15:08:52.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/mii.ko 2024-02-19T15:08:52.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 45057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/netconsole.ko 2024-02-19T15:08:52.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/nlmon.ko 2024-02-19T15:08:52.841 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/pcs/ 2024-02-19T15:08:52.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/pcs/pcs-lynx.ko 2024-02-19T15:08:52.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 39137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-02-19T15:08:52.842 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/ 2024-02-19T15:08:52.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/amd.ko 2024-02-19T15:08:52.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 50001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-02-19T15:08:52.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 44633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/bcm7xxx.ko 2024-02-19T15:08:52.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/bcm87xx.ko 2024-02-19T15:08:52.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 44825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/broadcom.ko 2024-02-19T15:08:52.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/cicada.ko 2024-02-19T15:08:52.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/davicom.ko 2024-02-19T15:08:52.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/et1011c.ko 2024-02-19T15:08:52.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/icplus.ko 2024-02-19T15:08:52.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/lxt.ko 2024-02-19T15:08:52.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 81961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/marvell.ko 2024-02-19T15:08:52.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 76569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/micrel.ko 2024-02-19T15:08:52.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/national.ko 2024-02-19T15:08:52.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 99009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/phylink.ko 2024-02-19T15:08:52.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/qsemi.ko 2024-02-19T15:08:52.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 49977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/realtek.ko 2024-02-19T15:08:52.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/smsc.ko 2024-02-19T15:08:52.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/ste10Xp.ko 2024-02-19T15:08:52.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/phy/vitesse.ko 2024-02-19T15:08:52.843 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/plip/ 2024-02-19T15:08:52.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 43721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/plip/plip.ko 2024-02-19T15:08:52.843 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ppp/ 2024-02-19T15:08:52.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ppp/bsd_comp.ko 2024-02-19T15:08:52.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ppp/ppp_async.ko 2024-02-19T15:08:52.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ppp/ppp_deflate.ko 2024-02-19T15:08:52.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ppp/ppp_mppe.ko 2024-02-19T15:08:52.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ppp/ppp_synctty.ko 2024-02-19T15:08:52.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 42673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ppp/pppoe.ko 2024-02-19T15:08:52.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ppp/pppox.ko 2024-02-19T15:08:52.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 32673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/ppp/pptp.ko 2024-02-19T15:08:52.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/sb1000.ko 2024-02-19T15:08:52.844 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/slip/ 2024-02-19T15:08:52.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 42393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/slip/slip.ko 2024-02-19T15:08:52.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/sungem_phy.ko 2024-02-19T15:08:52.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 64625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/tap.ko 2024-02-19T15:08:52.844 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/team/ 2024-02-19T15:08:52.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 107233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/team/team.ko 2024-02-19T15:08:52.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/team/team_mode_activebackup.ko 2024-02-19T15:08:52.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/team/team_mode_broadcast.ko 2024-02-19T15:08:52.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-02-19T15:08:52.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/team/team_mode_random.ko 2024-02-19T15:08:52.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-02-19T15:08:52.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 87105 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/veth.ko 2024-02-19T15:08:52.844 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/vmxnet3/ 2024-02-19T15:08:52.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 144361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-02-19T15:08:52.845 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/vxlan/ 2024-02-19T15:08:52.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 307257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/vxlan/vxlan.ko 2024-02-19T15:08:52.845 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/xen-netback/ 2024-02-19T15:08:52.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 143905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/net/xen-netback/xen-netback.ko 2024-02-19T15:08:52.845 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ntb/ 2024-02-19T15:08:52.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ntb/ntb.ko 2024-02-19T15:08:52.845 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/nvme/ 2024-02-19T15:08:52.845 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/nvme/host/ 2024-02-19T15:08:52.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 267745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/nvme/host/nvme-core.ko 2024-02-19T15:08:52.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 103361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/nvme/host/nvme.ko 2024-02-19T15:08:52.845 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/parport/ 2024-02-19T15:08:52.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 131329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/parport/parport.ko 2024-02-19T15:08:52.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14465 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/parport/parport_cs.ko 2024-02-19T15:08:52.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 85217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/parport/parport_pc.ko 2024-02-19T15:08:52.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 27569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/parport/parport_serial.ko 2024-02-19T15:08:52.845 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pci/ 2024-02-19T15:08:52.846 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pci/hotplug/ 2024-02-19T15:08:52.846 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24913 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-02-19T15:08:52.846 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2024-02-19T15:08:52.846 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2024-02-19T15:08:52.846 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pci/pci-stub.ko 2024-02-19T15:08:52.846 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 46433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pci/xen-pcifront.ko 2024-02-19T15:08:52.846 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pcmcia/ 2024-02-19T15:08:52.846 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pcmcia/i82092.ko 2024-02-19T15:08:52.846 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 146153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pcmcia/pcmcia.ko 2024-02-19T15:08:52.846 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 62081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pcmcia/pcmcia_core.ko 2024-02-19T15:08:52.846 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2024-02-19T15:08:52.846 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pcmcia/pd6729.ko 2024-02-19T15:08:52.846 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 112793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pcmcia/yenta_socket.ko 2024-02-19T15:08:52.846 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/phy/ 2024-02-19T15:08:52.846 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/phy/broadcom/ 2024-02-19T15:08:52.846 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2024-02-19T15:08:52.847 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/phy/samsung/ 2024-02-19T15:08:52.847 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2024-02-19T15:08:52.847 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pinctrl/ 2024-02-19T15:08:52.847 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pinctrl/intel/ 2024-02-19T15:08:52.847 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 69033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-02-19T15:08:52.847 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/ 2024-02-19T15:08:52.847 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/chrome/ 2024-02-19T15:08:52.847 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 51233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2024-02-19T15:08:52.847 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24585 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/chrome/cros_ec.ko 2024-02-19T15:08:52.847 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2024-02-19T15:08:52.847 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2024-02-19T15:08:52.847 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26105 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2024-02-19T15:08:52.847 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2024-02-19T15:08:52.847 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2024-02-19T15:08:52.847 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/ 2024-02-19T15:08:52.847 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 73601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/acer-wmi.ko 2024-02-19T15:08:52.848 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 39057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/acerhdf.ko 2024-02-19T15:08:52.848 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/apple-gmux.ko 2024-02-19T15:08:52.848 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 59569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/asus-laptop.ko 2024-02-19T15:08:52.848 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-02-19T15:08:52.848 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 131945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/asus-wmi.ko 2024-02-19T15:08:52.848 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/classmate-laptop.ko 2024-02-19T15:08:52.848 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 49025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-02-19T15:08:52.848 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-02-19T15:08:52.848 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 40169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-02-19T15:08:52.848 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-02-19T15:08:52.848 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/hdaps.ko 2024-02-19T15:08:52.848 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/ibm_rtl.ko 2024-02-19T15:08:52.848 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/intel/ 2024-02-19T15:08:52.848 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-02-19T15:08:52.849 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2024-02-19T15:08:52.849 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 44601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/intel_ips.ko 2024-02-19T15:08:52.849 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/msi-wmi.ko 2024-02-19T15:08:52.849 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/mxm-wmi.ko 2024-02-19T15:08:52.849 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-02-19T15:08:52.849 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 42233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/samsung-laptop.ko 2024-02-19T15:08:52.849 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/samsung-q10.ko 2024-02-19T15:08:52.849 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/topstar-laptop.ko 2024-02-19T15:08:52.849 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-02-19T15:08:52.849 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/toshiba_haps.ko 2024-02-19T15:08:52.849 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/wmi-bmof.ko 2024-02-19T15:08:52.849 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 48481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/platform/x86/wmi.ko 2024-02-19T15:08:52.849 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/ 2024-02-19T15:08:52.849 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/ 2024-02-19T15:08:52.849 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/88pm860x_battery.ko 2024-02-19T15:08:52.849 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 33889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/88pm860x_charger.ko 2024-02-19T15:08:52.850 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 54961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/bq2415x_charger.ko 2024-02-19T15:08:52.850 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 59505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/bq24190_charger.ko 2024-02-19T15:08:52.850 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/bq24735-charger.ko 2024-02-19T15:08:52.850 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/cros_peripheral_charger.ko 2024-02-19T15:08:52.850 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/da9030_battery.ko 2024-02-19T15:08:52.850 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/da9052-battery.ko 2024-02-19T15:08:52.850 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/ds2760_battery.ko 2024-02-19T15:08:52.850 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/ds2780_battery.ko 2024-02-19T15:08:52.850 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/ds2781_battery.ko 2024-02-19T15:08:52.850 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23553 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/ds2782_battery.ko 2024-02-19T15:08:52.850 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/gpio-charger.ko 2024-02-19T15:08:52.850 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/isp1704_charger.ko 2024-02-19T15:08:52.850 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/lp8727_charger.ko 2024-02-19T15:08:52.850 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/max14577_charger.ko 2024-02-19T15:08:52.850 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/max17042_battery.ko 2024-02-19T15:08:52.850 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/max8903_charger.ko 2024-02-19T15:08:52.851 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/max8925_power.ko 2024-02-19T15:08:52.851 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/pcf50633-charger.ko 2024-02-19T15:08:52.851 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 37841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/sbs-battery.ko 2024-02-19T15:08:52.851 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/test_power.ko 2024-02-19T15:08:52.851 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/tps65090-charger.ko 2024-02-19T15:08:52.851 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/wm831x_backup.ko 2024-02-19T15:08:52.851 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/wm831x_power.ko 2024-02-19T15:08:52.851 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/power/supply/wm8350_power.ko 2024-02-19T15:08:52.851 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/powercap/ 2024-02-19T15:08:52.851 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 58105 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/powercap/intel_rapl_common.ko 2024-02-19T15:08:52.851 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/powercap/intel_rapl_msr.ko 2024-02-19T15:08:52.851 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pps/ 2024-02-19T15:08:52.851 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pps/clients/ 2024-02-19T15:08:52.851 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pps/clients/pps-gpio.ko 2024-02-19T15:08:52.851 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pps/clients/pps-ldisc.ko 2024-02-19T15:08:52.851 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pps/clients/pps_parport.ko 2024-02-19T15:08:52.852 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 41801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/pps/pps_core.ko 2024-02-19T15:08:52.852 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ptp/ 2024-02-19T15:08:52.852 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 70873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ptp/ptp.ko 2024-02-19T15:08:52.852 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ptp/ptp_kvm.ko 2024-02-19T15:08:52.852 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/ 2024-02-19T15:08:52.852 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-88pm80x.ko 2024-02-19T15:08:52.852 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-88pm860x.ko 2024-02-19T15:08:52.852 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-bq32k.ko 2024-02-19T15:08:52.852 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-da9052.ko 2024-02-19T15:08:52.852 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-da9055.ko 2024-02-19T15:08:52.852 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-da9063.ko 2024-02-19T15:08:52.852 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-ds1286.ko 2024-02-19T15:08:52.852 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 68481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-ds1307.ko 2024-02-19T15:08:52.852 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-ds1374.ko 2024-02-19T15:08:52.852 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-ds1511.ko 2024-02-19T15:08:52.853 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-ds1553.ko 2024-02-19T15:08:52.853 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-ds1672.ko 2024-02-19T15:08:52.853 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13273 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-ds1742.ko 2024-02-19T15:08:52.853 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-ds2404.ko 2024-02-19T15:08:52.853 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23609 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-ds3232.ko 2024-02-19T15:08:52.853 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-em3027.ko 2024-02-19T15:08:52.853 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-fm3130.ko 2024-02-19T15:08:52.853 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-isl12022.ko 2024-02-19T15:08:52.853 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 35873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-isl1208.ko 2024-02-19T15:08:52.853 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-lp8788.ko 2024-02-19T15:08:52.853 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 38001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-m41t80.ko 2024-02-19T15:08:52.853 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-m48t35.ko 2024-02-19T15:08:52.853 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-m48t59.ko 2024-02-19T15:08:52.853 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-m48t86.ko 2024-02-19T15:08:52.853 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-max6900.ko 2024-02-19T15:08:52.853 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-max8907.ko 2024-02-19T15:08:52.854 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-max8925.ko 2024-02-19T15:08:52.854 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-max8997.ko 2024-02-19T15:08:52.854 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16585 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-max8998.ko 2024-02-19T15:08:52.854 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-mc13xxx.ko 2024-02-19T15:08:52.854 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-msm6242.ko 2024-02-19T15:08:52.854 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-palmas.ko 2024-02-19T15:08:52.854 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 46537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-pcf2127.ko 2024-02-19T15:08:52.854 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-pcf50633.ko 2024-02-19T15:08:52.854 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-pcf85063.ko 2024-02-19T15:08:52.854 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-pcf8523.ko 2024-02-19T15:08:52.854 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-pcf8563.ko 2024-02-19T15:08:52.854 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-pcf8583.ko 2024-02-19T15:08:52.854 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-rc5t583.ko 2024-02-19T15:08:52.854 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-rp5c01.ko 2024-02-19T15:08:52.854 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 33657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-rs5c372.ko 2024-02-19T15:08:52.854 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 27737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-02-19T15:08:52.854 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-rx8025.ko 2024-02-19T15:08:52.855 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-rx8581.ko 2024-02-19T15:08:52.855 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-s35390a.ko 2024-02-19T15:08:52.855 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-02-19T15:08:52.855 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-tps6586x.ko 2024-02-19T15:08:52.855 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-tps65910.ko 2024-02-19T15:08:52.855 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-wm831x.ko 2024-02-19T15:08:52.855 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-wm8350.ko 2024-02-19T15:08:52.855 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 27777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/rtc/rtc-x1205.ko 2024-02-19T15:08:52.855 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/ 2024-02-19T15:08:52.855 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 77529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/3w-9xxx.ko 2024-02-19T15:08:52.855 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 57721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/3w-sas.ko 2024-02-19T15:08:52.855 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 55081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/3w-xxxx.ko 2024-02-19T15:08:52.855 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 135345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/BusLogic.ko 2024-02-19T15:08:52.855 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/a100u2w.ko 2024-02-19T15:08:52.855 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/aacraid/ 2024-02-19T15:08:52.855 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 271441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/aacraid/aacraid.ko 2024-02-19T15:08:52.855 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 100473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/advansys.ko 2024-02-19T15:08:52.855 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/aic7xxx/ 2024-02-19T15:08:52.856 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 356865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-02-19T15:08:52.856 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 295993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-02-19T15:08:52.856 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/aic94xx/ 2024-02-19T15:08:52.856 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 189145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/aic94xx/aic94xx.ko 2024-02-19T15:08:52.856 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/am53c974.ko 2024-02-19T15:08:52.856 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/arcmsr/ 2024-02-19T15:08:52.856 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 108273 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-02-19T15:08:52.856 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 38097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/atp870u.ko 2024-02-19T15:08:52.856 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/be2iscsi/ 2024-02-19T15:08:52.856 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 246937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-02-19T15:08:52.856 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/bfa/ 2024-02-19T15:08:52.856 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 1171953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/bfa/bfa.ko 2024-02-19T15:08:52.856 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/bnx2fc/ 2024-02-19T15:08:52.857 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 238401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-02-19T15:08:52.857 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/bnx2i/ 2024-02-19T15:08:52.857 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 115329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-02-19T15:08:52.857 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 40785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/ch.ko 2024-02-19T15:08:52.857 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/csiostor/ 2024-02-19T15:08:52.857 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 308961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/csiostor/csiostor.ko 2024-02-19T15:08:52.857 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/cxgbi/ 2024-02-19T15:08:52.857 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/cxgbi/cxgb3i/ 2024-02-19T15:08:52.857 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 76041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-02-19T15:08:52.857 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/cxgbi/cxgb4i/ 2024-02-19T15:08:52.857 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 101425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-02-19T15:08:52.857 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 146817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-02-19T15:08:52.857 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 75977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/dc395x.ko 2024-02-19T15:08:52.857 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 38209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/dmx3191d.ko 2024-02-19T15:08:52.858 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/esas2r/ 2024-02-19T15:08:52.858 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 212385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/esas2r/esas2r.ko 2024-02-19T15:08:52.858 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 60505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/esp_scsi.ko 2024-02-19T15:08:52.858 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/fcoe/ 2024-02-19T15:08:52.858 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 78225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/fcoe/fcoe.ko 2024-02-19T15:08:52.858 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 157961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-02-19T15:08:52.858 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/fdomain.ko 2024-02-19T15:08:52.858 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/fnic/ 2024-02-19T15:08:52.858 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 294601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/fnic/fnic.ko 2024-02-19T15:08:52.858 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 201729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/hpsa.ko 2024-02-19T15:08:52.858 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 51121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/hptiop.ko 2024-02-19T15:08:52.858 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 49017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/hv_storvsc.ko 2024-02-19T15:08:52.858 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36129 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/imm.ko 2024-02-19T15:08:52.858 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 43977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/initio.ko 2024-02-19T15:08:52.859 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 265945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/ipr.ko 2024-02-19T15:08:52.859 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 101881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/ips.ko 2024-02-19T15:08:52.859 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/isci/ 2024-02-19T15:08:52.859 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 317193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/isci/isci.ko 2024-02-19T15:08:52.859 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-02-19T15:08:52.859 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 52865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/iscsi_tcp.ko 2024-02-19T15:08:52.859 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/libfc/ 2024-02-19T15:08:52.859 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 315385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/libfc/libfc.ko 2024-02-19T15:08:52.859 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 145497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/libiscsi.ko 2024-02-19T15:08:52.859 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 53809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/libiscsi_tcp.ko 2024-02-19T15:08:52.859 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/libsas/ 2024-02-19T15:08:52.859 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 216401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/libsas/libsas.ko 2024-02-19T15:08:52.859 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/lpfc/ 2024-02-19T15:08:52.859 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 2194089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/lpfc/lpfc.ko 2024-02-19T15:08:52.908 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/megaraid/ 2024-02-19T15:08:52.908 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 73473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-02-19T15:08:52.908 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 33433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-02-19T15:08:52.908 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 310689 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-02-19T15:08:52.908 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 85521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/megaraid.ko 2024-02-19T15:08:52.908 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/mpt3sas/ 2024-02-19T15:08:52.908 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 659569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-02-19T15:08:52.909 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/mvsas/ 2024-02-19T15:08:52.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 119489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/mvsas/mvsas.ko 2024-02-19T15:08:52.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 78041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/mvumi.ko 2024-02-19T15:08:52.909 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/pcmcia/ 2024-02-19T15:08:52.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 81561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2024-02-19T15:08:52.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2024-02-19T15:08:52.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2024-02-19T15:08:52.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2024-02-19T15:08:52.909 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/pm8001/ 2024-02-19T15:08:52.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 366337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-02-19T15:08:52.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 124601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/pmcraid.ko 2024-02-19T15:08:52.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/ppa.ko 2024-02-19T15:08:52.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 55337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/qla1280.ko 2024-02-19T15:08:52.910 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/qla2xxx/ 2024-02-19T15:08:52.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 1876753 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-02-19T15:08:52.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 85193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-02-19T15:08:52.910 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/qla4xxx/ 2024-02-19T15:08:52.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 558953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-02-19T15:08:52.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/qlogicfas408.ko 2024-02-19T15:08:52.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/raid_class.ko 2024-02-19T15:08:52.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 261081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/scsi_debug.ko 2024-02-19T15:08:52.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 173929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/scsi_transport_fc.ko 2024-02-19T15:08:52.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 280113 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-02-19T15:08:52.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 97625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/scsi_transport_sas.ko 2024-02-19T15:08:52.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 73865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/scsi_transport_spi.ko 2024-02-19T15:08:52.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 44889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/scsi_transport_srp.ko 2024-02-19T15:08:52.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/ses.ko 2024-02-19T15:08:52.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 115713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/st.ko 2024-02-19T15:08:52.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 46737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/stex.ko 2024-02-19T15:08:52.910 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/sym53c8xx_2/ 2024-02-19T15:08:52.911 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 164889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-02-19T15:08:52.911 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 44857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/virtio_scsi.ko 2024-02-19T15:08:52.911 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 50233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/vmw_pvscsi.ko 2024-02-19T15:08:52.911 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 39881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/wd719x.ko 2024-02-19T15:08:52.911 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 49929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/scsi/xen-scsifront.ko 2024-02-19T15:08:52.911 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ssb/ 2024-02-19T15:08:52.911 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 129393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ssb/ssb.ko 2024-02-19T15:08:52.911 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/ 2024-02-19T15:08:52.911 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/iscsi/ 2024-02-19T15:08:52.911 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 736761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-02-19T15:08:52.911 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/loopback/ 2024-02-19T15:08:52.911 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 49881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/loopback/tcm_loop.ko 2024-02-19T15:08:52.911 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 39617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/target_core_file.ko 2024-02-19T15:08:52.911 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 42873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/target_core_iblock.ko 2024-02-19T15:08:52.912 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 930977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/target_core_mod.ko 2024-02-19T15:08:52.912 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 44137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/target_core_pscsi.ko 2024-02-19T15:08:52.912 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 122489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/target_core_user.ko 2024-02-19T15:08:52.912 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/tcm_fc/ 2024-02-19T15:08:52.912 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 75521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-02-19T15:08:52.912 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/ 2024-02-19T15:08:52.912 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/ 2024-02-19T15:08:52.912 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/ 2024-02-19T15:08:52.912 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-02-19T15:08:52.912 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-02-19T15:08:52.912 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-02-19T15:08:52.912 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-02-19T15:08:52.912 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-02-19T15:08:52.912 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-02-19T15:08:52.913 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-02-19T15:08:52.913 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-02-19T15:08:52.913 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-02-19T15:08:52.913 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-02-19T15:08:52.913 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10753 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-02-19T15:08:52.913 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-02-19T15:08:52.913 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-02-19T15:08:52.913 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-02-19T15:08:52.913 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11465 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-02-19T15:08:52.913 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/intel_powerclamp.ko 2024-02-19T15:08:52.913 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-02-19T15:08:52.913 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-02-19T15:08:52.913 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-02-19T15:08:52.913 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/ 2024-02-19T15:08:52.913 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/ipwireless/ 2024-02-19T15:08:52.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 78353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-02-19T15:08:52.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 57777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/moxa.ko 2024-02-19T15:08:52.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 50865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/mxser.ko 2024-02-19T15:08:52.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 108489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/n_gsm.ko 2024-02-19T15:08:52.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/n_hdlc.ko 2024-02-19T15:08:52.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 67841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/nozomi.ko 2024-02-19T15:08:52.914 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/ 2024-02-19T15:08:52.914 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/8250/ 2024-02-19T15:08:52.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/8250/8250_dw.ko 2024-02-19T15:08:52.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 59401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-02-19T15:08:52.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/altera_jtaguart.ko 2024-02-19T15:08:52.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/altera_uart.ko 2024-02-19T15:08:52.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/arc_uart.ko 2024-02-19T15:08:52.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 72489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/fsl_lpuart.ko 2024-02-19T15:08:52.914 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/jsm/ 2024-02-19T15:08:52.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 101993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/jsm/jsm.ko 2024-02-19T15:08:52.915 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/rp2.ko 2024-02-19T15:08:52.915 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 53177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/tty/serial/sc16is7xx.ko 2024-02-19T15:08:52.915 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ufs/ 2024-02-19T15:08:52.915 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ufs/core/ 2024-02-19T15:08:52.915 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 545401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ufs/core/ufshcd-core.ko 2024-02-19T15:08:52.915 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ufs/host/ 2024-02-19T15:08:52.915 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-02-19T15:08:52.915 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2024-02-19T15:08:52.915 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/uio/ 2024-02-19T15:08:52.915 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 44473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/uio/uio.ko 2024-02-19T15:08:52.915 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/uio/uio_aec.ko 2024-02-19T15:08:52.915 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/uio/uio_cif.ko 2024-02-19T15:08:52.915 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/uio/uio_dmem_genirq.ko 2024-02-19T15:08:52.915 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/uio/uio_mf624.ko 2024-02-19T15:08:52.915 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/uio/uio_netx.ko 2024-02-19T15:08:52.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/uio/uio_pci_generic.ko 2024-02-19T15:08:52.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/uio/uio_pdrv_genirq.ko 2024-02-19T15:08:52.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/uio/uio_sercos3.ko 2024-02-19T15:08:52.916 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/ 2024-02-19T15:08:52.916 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/c67x00/ 2024-02-19T15:08:52.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 73225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/c67x00/c67x00.ko 2024-02-19T15:08:52.916 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/chipidea/ 2024-02-19T15:08:52.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 145793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/chipidea/ci_hdrc.ko 2024-02-19T15:08:52.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2024-02-19T15:08:52.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/chipidea/ci_hdrc_npcm.ko 2024-02-19T15:08:52.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2024-02-19T15:08:52.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2024-02-19T15:08:52.916 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/class/ 2024-02-19T15:08:52.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 87009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/class/cdc-acm.ko 2024-02-19T15:08:52.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 51321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/class/cdc-wdm.ko 2024-02-19T15:08:52.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 47217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/class/usblp.ko 2024-02-19T15:08:52.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 89769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/class/usbtmc.ko 2024-02-19T15:08:52.917 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/common/ 2024-02-19T15:08:52.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/common/ulpi.ko 2024-02-19T15:08:52.917 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/dwc2/ 2024-02-19T15:08:52.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/dwc2/dwc2_pci.ko 2024-02-19T15:08:52.917 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/dwc3/ 2024-02-19T15:08:52.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/dwc3/dwc3-haps.ko 2024-02-19T15:08:52.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/dwc3/dwc3-pci.ko 2024-02-19T15:08:52.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 452577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/dwc3/dwc3.ko 2024-02-19T15:08:52.917 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/ 2024-02-19T15:08:52.917 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/ 2024-02-19T15:08:52.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 60329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/u_ether.ko 2024-02-19T15:08:52.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 51793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/u_serial.ko 2024-02-19T15:08:52.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2024-02-19T15:08:52.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 42353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2024-02-19T15:08:52.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 27857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2024-02-19T15:08:52.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2024-02-19T15:08:52.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 92409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2024-02-19T15:08:52.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 50945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2024-02-19T15:08:52.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 122257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2024-02-19T15:08:52.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 59801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2024-02-19T15:08:52.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2024-02-19T15:08:52.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2024-02-19T15:08:52.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 57633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2024-02-19T15:08:52.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 85521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2024-02-19T15:08:52.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2024-02-19T15:08:52.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 58025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2024-02-19T15:08:52.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 67353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2024-02-19T15:08:52.919 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/ 2024-02-19T15:08:52.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2024-02-19T15:08:52.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2024-02-19T15:08:52.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2024-02-19T15:08:52.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_ether.ko 2024-02-19T15:08:52.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2024-02-19T15:08:52.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18921 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_hid.ko 2024-02-19T15:08:52.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2024-02-19T15:08:52.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2024-02-19T15:08:52.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2024-02-19T15:08:52.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_printer.ko 2024-02-19T15:08:52.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_serial.ko 2024-02-19T15:08:52.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 27113 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/g_zero.ko 2024-02-19T15:08:52.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 62369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2024-02-19T15:08:52.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15105 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2024-02-19T15:08:52.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 161233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/libcomposite.ko 2024-02-19T15:08:52.920 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/ 2024-02-19T15:08:52.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17609 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2024-02-19T15:08:52.920 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/bdc/ 2024-02-19T15:08:52.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 157617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2024-02-19T15:08:52.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/goku_udc.ko 2024-02-19T15:08:52.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 52929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/gr_udc.ko 2024-02-19T15:08:52.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 69617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2024-02-19T15:08:52.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 55953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/mv_udc.ko 2024-02-19T15:08:52.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 75969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/net2272.ko 2024-02-19T15:08:52.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 74985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/net2280.ko 2024-02-19T15:08:52.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 73017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/pch_udc.ko 2024-02-19T15:08:52.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 65881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2024-02-19T15:08:52.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 66001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2024-02-19T15:08:52.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 81409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2024-02-19T15:08:52.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 156337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/gadget/udc/udc-core.ko 2024-02-19T15:08:52.921 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/host/ 2024-02-19T15:08:52.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/host/bcma-hcd.ko 2024-02-19T15:08:52.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 80017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/host/isp116x-hcd.ko 2024-02-19T15:08:52.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 87193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/host/oxu210hp-hcd.ko 2024-02-19T15:08:52.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 78777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/host/r8a66597-hcd.ko 2024-02-19T15:08:52.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 51393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/host/sl811-hcd.ko 2024-02-19T15:08:52.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/host/sl811_cs.ko 2024-02-19T15:08:52.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/host/ssb-hcd.ko 2024-02-19T15:08:52.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/host/xhci-plat-hcd.ko 2024-02-19T15:08:52.921 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/image/ 2024-02-19T15:08:52.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/image/mdc800.ko 2024-02-19T15:08:52.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/image/microtek.ko 2024-02-19T15:08:52.922 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/ 2024-02-19T15:08:52.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 41633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/adutux.ko 2024-02-19T15:08:52.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/appledisplay.ko 2024-02-19T15:08:52.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/cypress_cy7c63.ko 2024-02-19T15:08:52.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/cytherm.ko 2024-02-19T15:08:52.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/ehset.ko 2024-02-19T15:08:52.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/emi26.ko 2024-02-19T15:08:52.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/emi62.ko 2024-02-19T15:08:52.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/ezusb.ko 2024-02-19T15:08:52.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/idmouse.ko 2024-02-19T15:08:52.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/iowarrior.ko 2024-02-19T15:08:52.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/isight_firmware.ko 2024-02-19T15:08:52.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/ldusb.ko 2024-02-19T15:08:52.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/legousbtower.ko 2024-02-19T15:08:52.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 27705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/lvstest.ko 2024-02-19T15:08:52.922 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/sisusbvga/ 2024-02-19T15:08:52.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 64745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-02-19T15:08:52.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/trancevibrator.ko 2024-02-19T15:08:52.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/usb3503.ko 2024-02-19T15:08:52.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/usblcd.ko 2024-02-19T15:08:52.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/usbsevseg.ko 2024-02-19T15:08:52.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 70721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/usbtest.ko 2024-02-19T15:08:52.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 41817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/uss720.ko 2024-02-19T15:08:52.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 27225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/misc/yurex.ko 2024-02-19T15:08:52.923 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/mon/ 2024-02-19T15:08:52.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 72825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/mon/usbmon.ko 2024-02-19T15:08:52.923 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/musb/ 2024-02-19T15:08:52.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 301633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/musb/musb_hdrc.ko 2024-02-19T15:08:52.923 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/phy/ 2024-02-19T15:08:52.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/phy/phy-generic.ko 2024-02-19T15:08:52.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2024-02-19T15:08:52.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/phy/phy-isp1301.ko 2024-02-19T15:08:52.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/phy/phy-tahvo.ko 2024-02-19T15:08:52.924 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/ 2024-02-19T15:08:52.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/aircable.ko 2024-02-19T15:08:52.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 33241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/ark3116.ko 2024-02-19T15:08:52.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/belkin_sa.ko 2024-02-19T15:08:52.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 38017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/ch341.ko 2024-02-19T15:08:52.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 70345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/cp210x.ko 2024-02-19T15:08:52.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/cyberjack.ko 2024-02-19T15:08:52.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 58121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/cypress_m8.ko 2024-02-19T15:08:52.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 49721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/digi_acceleport.ko 2024-02-19T15:08:52.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/empeg.ko 2024-02-19T15:08:52.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 44185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/f81232.ko 2024-02-19T15:08:52.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 175105 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/ftdi_sio.ko 2024-02-19T15:08:52.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 53017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/garmin_gps.ko 2024-02-19T15:08:52.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 136985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/io_edgeport.ko 2024-02-19T15:08:52.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 124745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/io_ti.ko 2024-02-19T15:08:52.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 67681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/ipaq.ko 2024-02-19T15:08:52.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/ipw.ko 2024-02-19T15:08:52.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/ir-usb.ko 2024-02-19T15:08:52.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 58137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-02-19T15:08:52.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 107433 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/keyspan.ko 2024-02-19T15:08:52.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 35521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/keyspan_pda.ko 2024-02-19T15:08:52.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/kl5kusb105.ko 2024-02-19T15:08:52.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 33593 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/kobil_sct.ko 2024-02-19T15:08:52.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 32865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/mct_u232.ko 2024-02-19T15:08:52.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/metro-usb.ko 2024-02-19T15:08:52.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 66361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/mos7720.ko 2024-02-19T15:08:52.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 78129 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/mos7840.ko 2024-02-19T15:08:52.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 56249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/mxuport.ko 2024-02-19T15:08:52.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/navman.ko 2024-02-19T15:08:52.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11921 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/omninet.ko 2024-02-19T15:08:52.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/opticon.ko 2024-02-19T15:08:52.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 191825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/option.ko 2024-02-19T15:08:52.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 38985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/oti6858.ko 2024-02-19T15:08:52.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 54265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/pl2303.ko 2024-02-19T15:08:52.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/qcaux.ko 2024-02-19T15:08:52.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 38025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/qcserial.ko 2024-02-19T15:08:52.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/quatech2.ko 2024-02-19T15:08:52.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/safe_serial.ko 2024-02-19T15:08:52.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 51537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/sierra.ko 2024-02-19T15:08:52.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21585 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/spcp8x5.ko 2024-02-19T15:08:52.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/ssu100.ko 2024-02-19T15:08:52.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/symbolserial.ko 2024-02-19T15:08:52.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 61393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-02-19T15:08:52.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-02-19T15:08:52.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/usb_debug.ko 2024-02-19T15:08:52.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 33113 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/usb_wwan.ko 2024-02-19T15:08:52.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 125529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/usbserial.ko 2024-02-19T15:08:52.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/visor.ko 2024-02-19T15:08:52.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 37313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/whiteheat.ko 2024-02-19T15:08:52.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/wishbone-serial.ko 2024-02-19T15:08:52.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/serial/xsens_mt.ko 2024-02-19T15:08:52.927 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ 2024-02-19T15:08:52.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 52761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/uas.ko 2024-02-19T15:08:52.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-alauda.ko 2024-02-19T15:08:52.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-cypress.ko 2024-02-19T15:08:52.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-datafab.ko 2024-02-19T15:08:52.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 33401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-02-19T15:08:52.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-freecom.ko 2024-02-19T15:08:52.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-isd200.ko 2024-02-19T15:08:52.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-02-19T15:08:52.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-karma.ko 2024-02-19T15:08:52.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-onetouch.ko 2024-02-19T15:08:52.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-realtek.ko 2024-02-19T15:08:52.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-sddr09.ko 2024-02-19T15:08:52.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-sddr55.ko 2024-02-19T15:08:52.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 35249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/ums-usbat.ko 2024-02-19T15:08:52.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 180329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/storage/usb-storage.ko 2024-02-19T15:08:52.928 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/usbip/ 2024-02-19T15:08:52.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 67401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/usbip/usbip-core.ko 2024-02-19T15:08:52.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 71025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/usbip/usbip-host.ko 2024-02-19T15:08:52.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 104065 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-02-19T15:08:52.929 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vfio/ 2024-02-19T15:08:52.929 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vfio/pci/ 2024-02-19T15:08:52.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 162865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-02-19T15:08:52.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vfio/pci/vfio-pci.ko 2024-02-19T15:08:52.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 103673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vfio/vfio.ko 2024-02-19T15:08:52.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 76441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-02-19T15:08:52.929 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vhost/ 2024-02-19T15:08:52.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 113665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vhost/vhost.ko 2024-02-19T15:08:52.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16913 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vhost/vhost_iotlb.ko 2024-02-19T15:08:52.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 58001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vhost/vhost_net.ko 2024-02-19T15:08:52.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 89057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/vhost/vhost_scsi.ko 2024-02-19T15:08:52.929 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/ 2024-02-19T15:08:52.929 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/ 2024-02-19T15:08:52.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/88pm860x_bl.ko 2024-02-19T15:08:52.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/aat2870_bl.ko 2024-02-19T15:08:52.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/adp5520_bl.ko 2024-02-19T15:08:52.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 32513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/adp8860_bl.ko 2024-02-19T15:08:52.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 39897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/adp8870_bl.ko 2024-02-19T15:08:52.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/apple_bl.ko 2024-02-19T15:08:52.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/as3711_bl.ko 2024-02-19T15:08:52.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/bd6107.ko 2024-02-19T15:08:52.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/da903x_bl.ko 2024-02-19T15:08:52.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/da9052_bl.ko 2024-02-19T15:08:52.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/gpio_backlight.ko 2024-02-19T15:08:52.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/kb3886_bl.ko 2024-02-19T15:08:52.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/lcd.ko 2024-02-19T15:08:52.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/lm3533_bl.ko 2024-02-19T15:08:52.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/lm3639_bl.ko 2024-02-19T15:08:52.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/lv5207lp.ko 2024-02-19T15:08:52.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/max8925_bl.ko 2024-02-19T15:08:52.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/pandora_bl.ko 2024-02-19T15:08:52.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/pcf50633-backlight.ko 2024-02-19T15:08:52.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/platform_lcd.ko 2024-02-19T15:08:52.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/video/backlight/wm831x_bl.ko 2024-02-19T15:08:52.931 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/ 2024-02-19T15:08:52.931 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/masters/ 2024-02-19T15:08:52.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/masters/ds2482.ko 2024-02-19T15:08:52.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 39121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/masters/ds2490.ko 2024-02-19T15:08:52.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/masters/matrox_w1.ko 2024-02-19T15:08:52.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11273 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/masters/w1-gpio.ko 2024-02-19T15:08:52.931 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/ 2024-02-19T15:08:52.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-02-19T15:08:52.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-02-19T15:08:52.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-02-19T15:08:52.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-02-19T15:08:52.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-02-19T15:08:52.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-02-19T15:08:52.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-02-19T15:08:52.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-02-19T15:08:52.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-02-19T15:08:52.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_smem.ko 2024-02-19T15:08:52.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 56337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/slaves/w1_therm.ko 2024-02-19T15:08:52.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 103049 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/w1/wire.ko 2024-02-19T15:08:52.932 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/ 2024-02-19T15:08:52.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/acquirewdt.ko 2024-02-19T15:08:52.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/advantechwdt.ko 2024-02-19T15:08:52.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18465 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/alim1535_wdt.ko 2024-02-19T15:08:52.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/alim7101_wdt.ko 2024-02-19T15:08:52.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/cpu5wdt.ko 2024-02-19T15:08:52.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/da9052_wdt.ko 2024-02-19T15:08:52.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/da9055_wdt.ko 2024-02-19T15:08:52.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/da9063_wdt.ko 2024-02-19T15:08:52.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/dw_wdt.ko 2024-02-19T15:08:52.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/eurotechwdt.ko 2024-02-19T15:08:52.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/f71808e_wdt.ko 2024-02-19T15:08:52.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 27393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/hpwdt.ko 2024-02-19T15:08:52.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/i6300esb.ko 2024-02-19T15:08:52.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-02-19T15:08:52.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/iTCO_wdt.ko 2024-02-19T15:08:52.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/ib700wdt.ko 2024-02-19T15:08:52.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/ibmasr.ko 2024-02-19T15:08:52.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-02-19T15:08:52.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21129 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/it8712f_wdt.ko 2024-02-19T15:08:52.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/it87_wdt.ko 2024-02-19T15:08:52.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21393 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/kempld_wdt.ko 2024-02-19T15:08:52.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/machzwd.ko 2024-02-19T15:08:52.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/mena21_wdt.ko 2024-02-19T15:08:52.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/menf21bmc_wdt.ko 2024-02-19T15:08:52.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24585 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/nv_tco.ko 2024-02-19T15:08:52.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/of_xilinx_wdt.ko 2024-02-19T15:08:52.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/pc87413_wdt.ko 2024-02-19T15:08:52.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 39921 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/pcwd_pci.ko 2024-02-19T15:08:52.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/pcwd_usb.ko 2024-02-19T15:08:52.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/retu_wdt.ko 2024-02-19T15:08:52.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/sbc60xxwdt.ko 2024-02-19T15:08:52.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/sbc_epx_c3.ko 2024-02-19T15:08:52.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-02-19T15:08:52.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/sc1200wdt.ko 2024-02-19T15:08:52.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/sch311x_wdt.ko 2024-02-19T15:08:52.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21481 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/smsc37b787_wdt.ko 2024-02-19T15:08:52.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/softdog.ko 2024-02-19T15:08:52.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/sp5100_tco.ko 2024-02-19T15:08:52.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/twl4030_wdt.ko 2024-02-19T15:08:52.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/via_wdt.ko 2024-02-19T15:08:52.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-02-19T15:08:52.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/w83877f_wdt.ko 2024-02-19T15:08:52.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/w83977f_wdt.ko 2024-02-19T15:08:52.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/wafer5823wdt.ko 2024-02-19T15:08:52.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/wdt_pci.ko 2024-02-19T15:08:52.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/wm831x_wdt.ko 2024-02-19T15:08:52.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13689 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/wm8350_wdt.ko 2024-02-19T15:08:52.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15609 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/watchdog/xen_wdt.ko 2024-02-19T15:08:52.935 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/xen/ 2024-02-19T15:08:52.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/xen/xen-evtchn.ko 2024-02-19T15:08:52.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/xen/xen-gntalloc.ko 2024-02-19T15:08:52.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 47769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/xen/xen-gntdev.ko 2024-02-19T15:08:52.936 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/xen/xen-pciback/ 2024-02-19T15:08:52.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 176385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-02-19T15:08:52.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 40097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/xen/xen-privcmd.ko 2024-02-19T15:08:52.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 71153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/xen/xen-scsiback.ko 2024-02-19T15:08:52.936 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/xen/xenfs/ 2024-02-19T15:08:52.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/drivers/xen/xenfs/xenfs.ko 2024-02-19T15:08:52.936 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ 2024-02-19T15:08:52.936 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/9p/ 2024-02-19T15:08:52.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 158281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/9p/9p.ko 2024-02-19T15:08:52.936 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/adfs/ 2024-02-19T15:08:52.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 61721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/adfs/adfs.ko 2024-02-19T15:08:52.937 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/affs/ 2024-02-19T15:08:52.937 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 163377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/affs/affs.ko 2024-02-19T15:08:52.937 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/afs/ 2024-02-19T15:08:52.937 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 1018985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/afs/kafs.ko 2024-02-19T15:08:52.937 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/autofs/ 2024-02-19T15:08:52.937 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 103225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/autofs/autofs4.ko 2024-02-19T15:08:52.937 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/befs/ 2024-02-19T15:08:52.937 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 64625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/befs/befs.ko 2024-02-19T15:08:52.937 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/bfs/ 2024-02-19T15:08:52.937 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 41017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/bfs/bfs.ko 2024-02-19T15:08:52.937 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 38649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/binfmt_misc.ko 2024-02-19T15:08:52.937 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/btrfs/ 2024-02-19T15:08:52.937 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 3484777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/btrfs/btrfs.ko 2024-02-19T15:08:52.937 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/cachefiles/ 2024-02-19T15:08:52.937 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 305737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/cachefiles/cachefiles.ko 2024-02-19T15:08:52.938 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ceph/ 2024-02-19T15:08:52.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 1335465 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ceph/ceph.ko 2024-02-19T15:08:52.938 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/coda/ 2024-02-19T15:08:52.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 97769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/coda/coda.ko 2024-02-19T15:08:52.938 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/configfs/ 2024-02-19T15:08:52.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 108577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/configfs/configfs.ko 2024-02-19T15:08:52.938 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/cramfs/ 2024-02-19T15:08:52.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29593 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/cramfs/cramfs.ko 2024-02-19T15:08:52.938 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/dlm/ 2024-02-19T15:08:52.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 644249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/dlm/dlm.ko 2024-02-19T15:08:52.938 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/efs/ 2024-02-19T15:08:52.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/efs/efs.ko 2024-02-19T15:08:52.938 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/f2fs/ 2024-02-19T15:08:52.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 1527729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/f2fs/f2fs.ko 2024-02-19T15:08:52.938 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/fat/ 2024-02-19T15:08:52.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 27841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/fat/msdos.ko 2024-02-19T15:08:52.939 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/freevxfs/ 2024-02-19T15:08:52.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 42561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/freevxfs/freevxfs.ko 2024-02-19T15:08:52.939 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/fuse/ 2024-02-19T15:08:52.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/fuse/cuse.ko 2024-02-19T15:08:52.939 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/gfs2/ 2024-02-19T15:08:52.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 1231361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/gfs2/gfs2.ko 2024-02-19T15:08:52.939 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/hfs/ 2024-02-19T15:08:52.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 141713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/hfs/hfs.ko 2024-02-19T15:08:52.939 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/hfsplus/ 2024-02-19T15:08:52.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 222897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/hfsplus/hfsplus.ko 2024-02-19T15:08:52.939 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/hpfs/ 2024-02-19T15:08:52.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 193545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/hpfs/hpfs.ko 2024-02-19T15:08:52.939 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/isofs/ 2024-02-19T15:08:52.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 74737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/isofs/isofs.ko 2024-02-19T15:08:52.939 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/jfs/ 2024-02-19T15:08:52.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 420449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/jfs/jfs.ko 2024-02-19T15:08:52.940 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/lockd/ 2024-02-19T15:08:52.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 281705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/lockd/lockd.ko 2024-02-19T15:08:52.940 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/minix/ 2024-02-19T15:08:52.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 88937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/minix/minix.ko 2024-02-19T15:08:52.940 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/netfs/ 2024-02-19T15:08:52.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 490001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/netfs/netfs.ko 2024-02-19T15:08:52.940 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/ 2024-02-19T15:08:52.940 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/blocklayout/ 2024-02-19T15:08:52.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 81457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-02-19T15:08:52.940 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/filelayout/ 2024-02-19T15:08:52.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 59353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-02-19T15:08:52.940 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/flexfilelayout/ 2024-02-19T15:08:52.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 117377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-02-19T15:08:52.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 1057985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/nfs.ko 2024-02-19T15:08:52.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 66361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/nfsv2.ko 2024-02-19T15:08:52.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 120193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/nfsv3.ko 2024-02-19T15:08:52.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 2041825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs/nfsv4.ko 2024-02-19T15:08:52.941 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs_common/ 2024-02-19T15:08:52.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs_common/grace.ko 2024-02-19T15:08:52.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfs_common/nfs_acl.ko 2024-02-19T15:08:52.941 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfsd/ 2024-02-19T15:08:52.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 1511665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nfsd/nfsd.ko 2024-02-19T15:08:52.941 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nilfs2/ 2024-02-19T15:08:52.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 582529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nilfs2/nilfs2.ko 2024-02-19T15:08:52.941 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/ 2024-02-19T15:08:52.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-celtic.ko 2024-02-19T15:08:52.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-centeuro.ko 2024-02-19T15:08:52.942 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-croatian.ko 2024-02-19T15:08:52.942 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-cyrillic.ko 2024-02-19T15:08:52.942 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-gaelic.ko 2024-02-19T15:08:52.942 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-greek.ko 2024-02-19T15:08:52.942 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-iceland.ko 2024-02-19T15:08:52.942 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-inuit.ko 2024-02-19T15:08:52.942 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-roman.ko 2024-02-19T15:08:52.942 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-romanian.ko 2024-02-19T15:08:52.942 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/mac-turkish.ko 2024-02-19T15:08:52.942 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_ascii.ko 2024-02-19T15:08:52.942 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp1250.ko 2024-02-19T15:08:52.942 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp1251.ko 2024-02-19T15:08:52.942 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp1255.ko 2024-02-19T15:08:52.942 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp737.ko 2024-02-19T15:08:52.942 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp775.ko 2024-02-19T15:08:52.942 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12129 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp850.ko 2024-02-19T15:08:52.943 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12129 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp852.ko 2024-02-19T15:08:52.943 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12129 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp855.ko 2024-02-19T15:08:52.943 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp857.ko 2024-02-19T15:08:52.943 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12753 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp860.ko 2024-02-19T15:08:52.943 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13065 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp861.ko 2024-02-19T15:08:52.943 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp862.ko 2024-02-19T15:08:52.943 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13065 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp863.ko 2024-02-19T15:08:52.943 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12753 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp864.ko 2024-02-19T15:08:52.943 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13065 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp865.ko 2024-02-19T15:08:52.943 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp866.ko 2024-02-19T15:08:52.943 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12129 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp869.ko 2024-02-19T15:08:52.943 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp874.ko 2024-02-19T15:08:52.943 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 94041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp932.ko 2024-02-19T15:08:52.943 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 142577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp936.ko 2024-02-19T15:08:52.943 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 165649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp949.ko 2024-02-19T15:08:52.943 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 118025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_cp950.ko 2024-02-19T15:08:52.944 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_euc-jp.ko 2024-02-19T15:08:52.944 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-1.ko 2024-02-19T15:08:52.944 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-13.ko 2024-02-19T15:08:52.944 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-14.ko 2024-02-19T15:08:52.944 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-15.ko 2024-02-19T15:08:52.944 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-2.ko 2024-02-19T15:08:52.944 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-3.ko 2024-02-19T15:08:52.944 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-4.ko 2024-02-19T15:08:52.944 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-5.ko 2024-02-19T15:08:52.944 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-6.ko 2024-02-19T15:08:52.944 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-7.ko 2024-02-19T15:08:52.944 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_iso8859-9.ko 2024-02-19T15:08:52.944 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_koi8-r.ko 2024-02-19T15:08:52.944 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_koi8-ru.ko 2024-02-19T15:08:52.945 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_koi8-u.ko 2024-02-19T15:08:52.945 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_ucs2_utils.ko 2024-02-19T15:08:52.945 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/nls/nls_utf8.ko 2024-02-19T15:08:52.945 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ntfs/ 2024-02-19T15:08:52.945 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 220545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ntfs/ntfs.ko 2024-02-19T15:08:52.945 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/ 2024-02-19T15:08:52.945 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/cluster/ 2024-02-19T15:08:52.945 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 244673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-02-19T15:08:52.945 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/dlm/ 2024-02-19T15:08:52.945 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 440681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-02-19T15:08:52.945 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/dlmfs/ 2024-02-19T15:08:52.945 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 54177 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-02-19T15:08:52.945 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 2590849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/ocfs2.ko 2024-02-19T15:08:52.945 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-02-19T15:08:52.946 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-02-19T15:08:52.946 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 35737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-02-19T15:08:52.946 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/omfs/ 2024-02-19T15:08:52.946 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 53705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/omfs/omfs.ko 2024-02-19T15:08:52.946 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/overlayfs/ 2024-02-19T15:08:52.946 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 361169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/overlayfs/overlay.ko 2024-02-19T15:08:52.946 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/pstore/ 2024-02-19T15:08:52.946 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 56945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/pstore/ramoops.ko 2024-02-19T15:08:52.946 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/qnx4/ 2024-02-19T15:08:52.946 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26913 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/qnx4/qnx4.ko 2024-02-19T15:08:52.946 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/qnx6/ 2024-02-19T15:08:52.946 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 46625 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/qnx6/qnx6.ko 2024-02-19T15:08:52.946 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/quota/ 2024-02-19T15:08:52.946 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/quota/quota_tree.ko 2024-02-19T15:08:52.946 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14273 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/quota/quota_v1.ko 2024-02-19T15:08:52.946 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/quota/quota_v2.ko 2024-02-19T15:08:52.947 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/reiserfs/ 2024-02-19T15:08:52.947 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 532289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/reiserfs/reiserfs.ko 2024-02-19T15:08:52.947 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/romfs/ 2024-02-19T15:08:52.947 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/romfs/romfs.ko 2024-02-19T15:08:52.947 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/smb/ 2024-02-19T15:08:52.947 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/smb/client/ 2024-02-19T15:08:52.947 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 2710993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/smb/client/cifs.ko 2024-02-19T15:08:52.947 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/smb/common/ 2024-02-19T15:08:52.947 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7065 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/smb/common/cifs_arc4.ko 2024-02-19T15:08:52.947 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/smb/common/cifs_md4.ko 2024-02-19T15:08:52.947 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/squashfs/ 2024-02-19T15:08:52.947 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 134809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/squashfs/squashfs.ko 2024-02-19T15:08:52.947 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/sysv/ 2024-02-19T15:08:52.947 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 91329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/sysv/sysv.ko 2024-02-19T15:08:52.947 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/udf/ 2024-02-19T15:08:52.947 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 248409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/udf/udf.ko 2024-02-19T15:08:52.948 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ufs/ 2024-02-19T15:08:52.948 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 162785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/ufs/ufs.ko 2024-02-19T15:08:52.948 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/xfs/ 2024-02-19T15:08:52.948 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 4036329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/fs/xfs/xfs.ko 2024-02-19T15:08:52.988 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/kernel/ 2024-02-19T15:08:52.988 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/kernel/locking/ 2024-02-19T15:08:52.988 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 80489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/kernel/locking/locktorture.ko 2024-02-19T15:08:52.988 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/kernel/time/ 2024-02-19T15:08:52.988 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/kernel/time/test_udelay.ko 2024-02-19T15:08:52.988 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 60265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/kernel/torture.ko 2024-02-19T15:08:52.988 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/ 2024-02-19T15:08:52.988 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 6737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/crc-itu-t.ko 2024-02-19T15:08:52.988 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 6441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/crc7.ko 2024-02-19T15:08:52.988 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 6937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/crc8.ko 2024-02-19T15:08:52.988 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/crypto/ 2024-02-19T15:08:52.988 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/crypto/libarc4.ko 2024-02-19T15:08:52.988 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/crypto/libdes.ko 2024-02-19T15:08:52.988 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/interval_tree_test.ko 2024-02-19T15:08:52.989 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/libcrc32c.ko 2024-02-19T15:08:52.989 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/lru_cache.ko 2024-02-19T15:08:52.989 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/lz4/ 2024-02-19T15:08:52.989 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 33705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/lz4/lz4_compress.ko 2024-02-19T15:08:52.989 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/lz4/lz4hc_compress.ko 2024-02-19T15:08:52.989 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/math/ 2024-02-19T15:08:52.989 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 6529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/math/cordic.ko 2024-02-19T15:08:52.989 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/memory-notifier-error-inject.ko 2024-02-19T15:08:52.989 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/notifier-error-inject.ko 2024-02-19T15:08:52.989 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/percpu_test.ko 2024-02-19T15:08:52.989 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/pm-notifier-error-inject.ko 2024-02-19T15:08:52.989 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/raid6/ 2024-02-19T15:08:52.989 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 143337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/raid6/raid6_pq.ko 2024-02-19T15:08:52.989 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/rbtree_test.ko 2024-02-19T15:08:52.989 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/reed_solomon/ 2024-02-19T15:08:52.989 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/reed_solomon/reed_solomon.ko 2024-02-19T15:08:52.989 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 38745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/test-kstrtox.ko 2024-02-19T15:08:52.990 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/test-string_helpers.ko 2024-02-19T15:08:52.990 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 4863225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/test_bpf.ko 2024-02-19T15:08:52.990 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 48993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/test_firmware.ko 2024-02-19T15:08:52.990 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 6281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/test_module.ko 2024-02-19T15:08:52.990 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/test_user_copy.ko 2024-02-19T15:08:52.990 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/ts_bm.ko 2024-02-19T15:08:52.990 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/ts_fsm.ko 2024-02-19T15:08:52.990 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/ts_kmp.ko 2024-02-19T15:08:52.990 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/xz/ 2024-02-19T15:08:52.990 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/lib/xz/xz_dec_test.ko 2024-02-19T15:08:52.990 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/mm/ 2024-02-19T15:08:52.990 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18129 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/mm/hwpoison-inject.ko 2024-02-19T15:08:52.990 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ 2024-02-19T15:08:52.990 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/6lowpan/ 2024-02-19T15:08:52.990 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 55665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/6lowpan/6lowpan.ko 2024-02-19T15:08:52.991 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 6961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/6lowpan/nhc_dest.ko 2024-02-19T15:08:52.991 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 6969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/6lowpan/nhc_fragment.ko 2024-02-19T15:08:52.991 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 6961 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/6lowpan/nhc_hop.ko 2024-02-19T15:08:52.991 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 6897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/6lowpan/nhc_ipv6.ko 2024-02-19T15:08:52.991 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 6969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/6lowpan/nhc_mobility.ko 2024-02-19T15:08:52.991 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 6969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/6lowpan/nhc_routing.ko 2024-02-19T15:08:52.991 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/6lowpan/nhc_udp.ko 2024-02-19T15:08:52.991 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/802/ 2024-02-19T15:08:52.991 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/802/garp.ko 2024-02-19T15:08:52.991 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/802/mrp.ko 2024-02-19T15:08:52.991 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/802/p8022.ko 2024-02-19T15:08:52.991 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/802/psnap.ko 2024-02-19T15:08:52.991 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/802/stp.ko 2024-02-19T15:08:52.991 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/8021q/ 2024-02-19T15:08:52.991 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 82897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/8021q/8021q.ko 2024-02-19T15:08:52.991 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/9p/ 2024-02-19T15:08:52.992 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 198833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/9p/9pnet.ko 2024-02-19T15:08:52.992 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/9p/9pnet_fd.ko 2024-02-19T15:08:52.992 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/9p/9pnet_rdma.ko 2024-02-19T15:08:52.992 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 32249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/9p/9pnet_virtio.ko 2024-02-19T15:08:52.992 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/batman-adv/ 2024-02-19T15:08:52.992 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 667545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/batman-adv/batman-adv.ko 2024-02-19T15:08:52.992 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/ 2024-02-19T15:08:52.992 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 65953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/br_netfilter.ko 2024-02-19T15:08:52.992 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 791305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/bridge.ko 2024-02-19T15:08:52.992 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ 2024-02-19T15:08:52.992 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-02-19T15:08:52.992 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_among.ko 2024-02-19T15:08:52.992 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_arp.ko 2024-02-19T15:08:52.992 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-02-19T15:08:52.992 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-02-19T15:08:52.993 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_ip.ko 2024-02-19T15:08:52.993 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-02-19T15:08:52.993 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_limit.ko 2024-02-19T15:08:52.993 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_log.ko 2024-02-19T15:08:52.993 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_mark.ko 2024-02-19T15:08:52.993 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-02-19T15:08:52.993 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-02-19T15:08:52.993 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-02-19T15:08:52.993 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-02-19T15:08:52.993 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_snat.ko 2024-02-19T15:08:52.993 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_stp.ko 2024-02-19T15:08:52.993 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-02-19T15:08:52.993 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-02-19T15:08:52.993 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-02-19T15:08:52.993 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-02-19T15:08:52.993 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 55945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/ebtables.ko 2024-02-19T15:08:52.994 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-02-19T15:08:52.994 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-02-19T15:08:52.994 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ceph/ 2024-02-19T15:08:52.994 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 1046801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ceph/libceph.ko 2024-02-19T15:08:52.994 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/core/ 2024-02-19T15:08:52.994 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 125769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/core/pktgen.ko 2024-02-19T15:08:52.994 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dccp/ 2024-02-19T15:08:52.994 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 194505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dccp/dccp.ko 2024-02-19T15:08:52.994 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dccp/dccp_diag.ko 2024-02-19T15:08:52.994 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 58745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dccp/dccp_ipv4.ko 2024-02-19T15:08:52.994 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 61401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dccp/dccp_ipv6.ko 2024-02-19T15:08:52.994 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dsa/ 2024-02-19T15:08:52.994 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 390089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dsa/dsa_core.ko 2024-02-19T15:08:52.994 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13361 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dsa/tag_brcm.ko 2024-02-19T15:08:52.994 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12601 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dsa/tag_dsa.ko 2024-02-19T15:08:52.994 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dsa/tag_none.ko 2024-02-19T15:08:52.994 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/dsa/tag_trailer.ko 2024-02-19T15:08:52.995 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/hsr/ 2024-02-19T15:08:52.995 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 102073 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/hsr/hsr.ko 2024-02-19T15:08:52.995 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ieee802154/ 2024-02-19T15:08:52.995 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ieee802154/6lowpan/ 2024-02-19T15:08:52.995 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 45505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-02-19T15:08:52.995 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 284721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ieee802154/ieee802154.ko 2024-02-19T15:08:52.995 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 47881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ieee802154/ieee802154_socket.ko 2024-02-19T15:08:52.995 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/ 2024-02-19T15:08:52.995 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/ah4.ko 2024-02-19T15:08:52.995 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 38977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/esp4.ko 2024-02-19T15:08:52.995 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 48249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/fou.ko 2024-02-19T15:08:52.995 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/gre.ko 2024-02-19T15:08:52.995 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 51217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/inet_diag.ko 2024-02-19T15:08:52.995 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 61017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/ip_gre.ko 2024-02-19T15:08:52.995 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 61337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/ip_tunnel.ko 2024-02-19T15:08:52.995 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/ip_vti.ko 2024-02-19T15:08:52.996 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/ipcomp.ko 2024-02-19T15:08:52.996 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 29033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/ipip.ko 2024-02-19T15:08:52.996 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/ 2024-02-19T15:08:52.996 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 38009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/arp_tables.ko 2024-02-19T15:08:52.996 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-02-19T15:08:52.996 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-02-19T15:08:52.996 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 42057 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/ip_tables.ko 2024-02-19T15:08:52.996 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-02-19T15:08:52.996 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9585 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2024-02-19T15:08:52.996 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-02-19T15:08:52.996 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-02-19T15:08:52.996 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14785 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-02-19T15:08:52.996 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/iptable_filter.ko 2024-02-19T15:08:52.996 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-02-19T15:08:52.996 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-02-19T15:08:52.997 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-02-19T15:08:52.997 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/iptable_security.ko 2024-02-19T15:08:52.997 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-02-19T15:08:52.997 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-02-19T15:08:52.997 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 27529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-02-19T15:08:52.997 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-02-19T15:08:52.997 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13241 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-02-19T15:08:52.997 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2024-02-19T15:08:52.997 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-02-19T15:08:52.997 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-02-19T15:08:52.997 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-02-19T15:08:52.997 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-02-19T15:08:52.997 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8913 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-02-19T15:08:52.997 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_bic.ko 2024-02-19T15:08:52.997 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_dctcp.ko 2024-02-19T15:08:52.997 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15969 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_diag.ko 2024-02-19T15:08:52.998 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_highspeed.ko 2024-02-19T15:08:52.998 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_htcp.ko 2024-02-19T15:08:52.998 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_hybla.ko 2024-02-19T15:08:52.998 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_illinois.ko 2024-02-19T15:08:52.998 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_lp.ko 2024-02-19T15:08:52.998 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_scalable.ko 2024-02-19T15:08:52.998 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_vegas.ko 2024-02-19T15:08:52.998 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_veno.ko 2024-02-19T15:08:52.998 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10857 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_westwood.ko 2024-02-19T15:08:52.998 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tcp_yeah.ko 2024-02-19T15:08:52.998 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/tunnel4.ko 2024-02-19T15:08:52.998 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/udp_diag.ko 2024-02-19T15:08:52.998 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 45001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/udp_tunnel.ko 2024-02-19T15:08:52.998 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv4/xfrm4_tunnel.ko 2024-02-19T15:08:52.998 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/ 2024-02-19T15:08:52.998 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/ah6.ko 2024-02-19T15:08:52.999 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 39617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/esp6.ko 2024-02-19T15:08:52.999 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/fou6.ko 2024-02-19T15:08:52.999 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 79929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/ip6_gre.ko 2024-02-19T15:08:52.999 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 94169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/ip6_tunnel.ko 2024-02-19T15:08:52.999 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-02-19T15:08:52.999 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 54273 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/ip6_vti.ko 2024-02-19T15:08:52.999 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/ipcomp6.ko 2024-02-19T15:08:52.999 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/mip6.ko 2024-02-19T15:08:52.999 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ 2024-02-19T15:08:52.999 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 42553 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-02-19T15:08:52.999 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2024-02-19T15:08:52.999 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-02-19T15:08:52.999 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14105 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-02-19T15:08:52.999 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-02-19T15:08:52.999 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-02-19T15:08:52.999 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16113 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-02-19T15:08:53.000 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-02-19T15:08:53.000 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-02-19T15:08:53.000 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-02-19T15:08:53.000 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-02-19T15:08:53.000 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-02-19T15:08:53.000 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-02-19T15:08:53.000 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-02-19T15:08:53.000 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15025 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-02-19T15:08:53.000 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-02-19T15:08:53.000 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-02-19T15:08:53.000 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 39137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-02-19T15:08:53.000 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-02-19T15:08:53.000 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-02-19T15:08:53.000 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-02-19T15:08:53.000 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-02-19T15:08:53.001 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-02-19T15:08:53.001 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-02-19T15:08:53.001 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8913 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-02-19T15:08:53.001 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 74833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/sit.ko 2024-02-19T15:08:53.001 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24105 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/tunnel6.ko 2024-02-19T15:08:53.001 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/ipv6/xfrm6_tunnel.ko 2024-02-19T15:08:53.001 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/key/ 2024-02-19T15:08:53.001 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 84345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/key/af_key.ko 2024-02-19T15:08:53.001 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/l2tp/ 2024-02-19T15:08:53.001 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 125209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/l2tp/l2tp_core.ko 2024-02-19T15:08:53.001 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/l2tp/l2tp_debugfs.ko 2024-02-19T15:08:53.001 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 25817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/l2tp/l2tp_eth.ko 2024-02-19T15:08:53.001 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36497 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/l2tp/l2tp_ip.ko 2024-02-19T15:08:53.001 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 37377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/l2tp/l2tp_ip6.ko 2024-02-19T15:08:53.001 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 33449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/l2tp/l2tp_netlink.ko 2024-02-19T15:08:53.001 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 53233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/l2tp/l2tp_ppp.ko 2024-02-19T15:08:53.002 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/lapb/ 2024-02-19T15:08:53.002 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 48121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/lapb/lapb.ko 2024-02-19T15:08:53.002 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/llc/ 2024-02-19T15:08:53.002 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/llc/llc.ko 2024-02-19T15:08:53.002 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 268521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/llc/llc2.ko 2024-02-19T15:08:53.002 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/mac802154/ 2024-02-19T15:08:53.002 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 309417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/mac802154/mac802154.ko 2024-02-19T15:08:53.002 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/mpls/ 2024-02-19T15:08:53.002 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/mpls/mpls_gso.ko 2024-02-19T15:08:53.002 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ 2024-02-19T15:08:53.002 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ 2024-02-19T15:08:53.002 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 119705 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set.ko 2024-02-19T15:08:53.002 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28617 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-02-19T15:08:53.002 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 27305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-02-19T15:08:53.002 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-02-19T15:08:53.003 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 99993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-02-19T15:08:53.003 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 100281 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-02-19T15:08:53.003 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 102553 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-02-19T15:08:53.003 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 102345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-02-19T15:08:53.003 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 111673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-02-19T15:08:53.003 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 59193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-02-19T15:08:53.003 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 108345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-02-19T15:08:53.003 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 112145 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-02-19T15:08:53.003 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 112409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-02-19T15:08:53.003 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 110337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-02-19T15:08:53.003 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 113289 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-02-19T15:08:53.003 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-02-19T15:08:53.003 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ 2024-02-19T15:08:53.003 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 412513 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs.ko 2024-02-19T15:08:53.003 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11993 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-02-19T15:08:53.003 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-02-19T15:08:53.003 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-02-19T15:08:53.004 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-02-19T15:08:53.004 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-02-19T15:08:53.004 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-02-19T15:08:53.004 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-02-19T15:08:53.004 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10641 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-02-19T15:08:53.004 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9265 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-02-19T15:08:53.004 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-02-19T15:08:53.004 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13545 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-02-19T15:08:53.004 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7721 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-02-19T15:08:53.004 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-02-19T15:08:53.004 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 49169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conncount.ko 2024-02-19T15:08:53.004 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 411745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack.ko 2024-02-19T15:08:53.004 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15881 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-02-19T15:08:53.004 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-02-19T15:08:53.004 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 33665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-02-19T15:08:53.004 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 118097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_h323.ko 2024-02-19T15:08:53.005 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_irc.ko 2024-02-19T15:08:53.005 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-02-19T15:08:53.005 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 108313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-02-19T15:08:53.005 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 33577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-02-19T15:08:53.005 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_sane.ko 2024-02-19T15:08:53.005 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 66537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_sip.ko 2024-02-19T15:08:53.005 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-02-19T15:08:53.005 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-02-19T15:08:53.005 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_dup_netdev.ko 2024-02-19T15:08:53.005 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 92209 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_flow_table.ko 2024-02-19T15:08:53.005 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9553 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_flow_table_inet.ko 2024-02-19T15:08:53.005 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 38081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_log_syslog.ko 2024-02-19T15:08:53.005 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 96521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_nat.ko 2024-02-19T15:08:53.005 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_nat_amanda.ko 2024-02-19T15:08:53.005 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12529 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_nat_ftp.ko 2024-02-19T15:08:53.005 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_nat_irc.ko 2024-02-19T15:08:53.006 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_nat_sip.ko 2024-02-19T15:08:53.006 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_nat_tftp.ko 2024-02-19T15:08:53.006 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 44137 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_synproxy_core.ko 2024-02-19T15:08:53.006 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 623417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nf_tables.ko 2024-02-19T15:08:53.006 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 42577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nfnetlink.ko 2024-02-19T15:08:53.006 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nfnetlink_acct.ko 2024-02-19T15:08:53.006 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nfnetlink_cttimeout.ko 2024-02-19T15:08:53.006 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 39561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nfnetlink_log.ko 2024-02-19T15:08:53.006 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nfnetlink_osf.ko 2024-02-19T15:08:53.006 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 49649 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nfnetlink_queue.ko 2024-02-19T15:08:53.006 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_chain_nat.ko 2024-02-19T15:08:53.006 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 33225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_compat.ko 2024-02-19T15:08:53.006 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 40761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_ct.ko 2024-02-19T15:08:53.006 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9505 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_dup_netdev.ko 2024-02-19T15:08:53.006 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_fib.ko 2024-02-19T15:08:53.006 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_fwd_netdev.ko 2024-02-19T15:08:53.006 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_hash.ko 2024-02-19T15:08:53.007 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20033 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_limit.ko 2024-02-19T15:08:53.007 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_log.ko 2024-02-19T15:08:53.007 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_masq.ko 2024-02-19T15:08:53.007 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14329 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_nat.ko 2024-02-19T15:08:53.007 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_numgen.ko 2024-02-19T15:08:53.007 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11753 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_osf.ko 2024-02-19T15:08:53.007 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_queue.ko 2024-02-19T15:08:53.007 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_redir.ko 2024-02-19T15:08:53.007 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_reject.ko 2024-02-19T15:08:53.007 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9985 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_reject_inet.ko 2024-02-19T15:08:53.007 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12841 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_socket.ko 2024-02-19T15:08:53.007 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_synproxy.ko 2024-02-19T15:08:53.007 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/nft_tproxy.ko 2024-02-19T15:08:53.007 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 89233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/x_tables.ko 2024-02-19T15:08:53.007 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_AUDIT.ko 2024-02-19T15:08:53.007 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_CHECKSUM.ko 2024-02-19T15:08:53.007 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7633 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_CLASSIFY.ko 2024-02-19T15:08:53.008 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-02-19T15:08:53.008 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 18081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_CT.ko 2024-02-19T15:08:53.008 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_DSCP.ko 2024-02-19T15:08:53.008 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_HL.ko 2024-02-19T15:08:53.008 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13345 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_HMARK.ko 2024-02-19T15:08:53.008 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 38777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_IDLETIMER.ko 2024-02-19T15:08:53.008 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13801 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_LED.ko 2024-02-19T15:08:53.008 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_LOG.ko 2024-02-19T15:08:53.008 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_MASQUERADE.ko 2024-02-19T15:08:53.008 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11561 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_NETMAP.ko 2024-02-19T15:08:53.008 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_NFLOG.ko 2024-02-19T15:08:53.008 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_NFQUEUE.ko 2024-02-19T15:08:53.008 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_RATEEST.ko 2024-02-19T15:08:53.008 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_REDIRECT.ko 2024-02-19T15:08:53.009 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_SECMARK.ko 2024-02-19T15:08:53.009 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_TCPMSS.ko 2024-02-19T15:08:53.009 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-02-19T15:08:53.009 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_TEE.ko 2024-02-19T15:08:53.009 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_TPROXY.ko 2024-02-19T15:08:53.009 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8233 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_TRACE.ko 2024-02-19T15:08:53.009 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_addrtype.ko 2024-02-19T15:08:53.009 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_bpf.ko 2024-02-19T15:08:53.009 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_cgroup.ko 2024-02-19T15:08:53.009 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11121 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_cluster.ko 2024-02-19T15:08:53.009 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_comment.ko 2024-02-19T15:08:53.009 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_connbytes.ko 2024-02-19T15:08:53.009 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11257 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_connlabel.ko 2024-02-19T15:08:53.009 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_connlimit.ko 2024-02-19T15:08:53.010 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_connmark.ko 2024-02-19T15:08:53.010 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_conntrack.ko 2024-02-19T15:08:53.010 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_cpu.ko 2024-02-19T15:08:53.010 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10889 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_dccp.ko 2024-02-19T15:08:53.010 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_devgroup.ko 2024-02-19T15:08:53.010 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_dscp.ko 2024-02-19T15:08:53.010 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10569 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_ecn.ko 2024-02-19T15:08:53.010 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_esp.ko 2024-02-19T15:08:53.010 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 43201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_hashlimit.ko 2024-02-19T15:08:53.010 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_helper.ko 2024-02-19T15:08:53.010 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_hl.ko 2024-02-19T15:08:53.010 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11825 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_ipcomp.ko 2024-02-19T15:08:53.010 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_iprange.ko 2024-02-19T15:08:53.010 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11697 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_ipvs.ko 2024-02-19T15:08:53.011 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_l2tp.ko 2024-02-19T15:08:53.011 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_length.ko 2024-02-19T15:08:53.011 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_limit.ko 2024-02-19T15:08:53.011 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_mac.ko 2024-02-19T15:08:53.011 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8553 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_mark.ko 2024-02-19T15:08:53.011 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_multiport.ko 2024-02-19T15:08:53.011 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_nat.ko 2024-02-19T15:08:53.011 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_nfacct.ko 2024-02-19T15:08:53.011 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8217 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_osf.ko 2024-02-19T15:08:53.011 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_owner.ko 2024-02-19T15:08:53.011 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10017 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_physdev.ko 2024-02-19T15:08:53.011 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_pkttype.ko 2024-02-19T15:08:53.011 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11897 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_policy.ko 2024-02-19T15:08:53.011 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9385 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_quota.ko 2024-02-19T15:08:53.012 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_rateest.ko 2024-02-19T15:08:53.012 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8169 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_realm.ko 2024-02-19T15:08:53.012 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36425 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_recent.ko 2024-02-19T15:08:53.012 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13305 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_sctp.ko 2024-02-19T15:08:53.012 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30001 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_set.ko 2024-02-19T15:08:53.012 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 16817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_socket.ko 2024-02-19T15:08:53.012 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_state.ko 2024-02-19T15:08:53.012 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_statistic.ko 2024-02-19T15:08:53.012 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8713 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_string.ko 2024-02-19T15:08:53.012 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 8457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_tcpmss.ko 2024-02-19T15:08:53.012 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15681 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_tcpudp.ko 2024-02-19T15:08:53.012 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_time.ko 2024-02-19T15:08:53.012 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9313 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netfilter/xt_u32.ko 2024-02-19T15:08:53.012 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netlink/ 2024-02-19T15:08:53.012 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 14441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/netlink/netlink_diag.ko 2024-02-19T15:08:53.013 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/nsh/ 2024-02-19T15:08:53.013 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 12337 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/nsh/nsh.ko 2024-02-19T15:08:53.013 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/openvswitch/ 2024-02-19T15:08:53.013 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 384097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/openvswitch/openvswitch.ko 2024-02-19T15:08:53.013 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10369 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/openvswitch/vport-geneve.ko 2024-02-19T15:08:53.013 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/openvswitch/vport-gre.ko 2024-02-19T15:08:53.013 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11833 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/openvswitch/vport-vxlan.ko 2024-02-19T15:08:53.013 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/packet/ 2024-02-19T15:08:53.013 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13473 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/packet/af_packet_diag.ko 2024-02-19T15:08:53.013 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/phonet/ 2024-02-19T15:08:53.013 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 112153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/phonet/phonet.ko 2024-02-19T15:08:53.013 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 54729 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/phonet/pn_pep.ko 2024-02-19T15:08:53.013 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/rds/ 2024-02-19T15:08:53.013 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 253737 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/rds/rds.ko 2024-02-19T15:08:53.013 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 165105 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/rds/rds_rdma.ko 2024-02-19T15:08:53.013 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 68161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/rds/rds_tcp.ko 2024-02-19T15:08:53.014 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/rxrpc/ 2024-02-19T15:08:53.014 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 828297 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/rxrpc/rxrpc.ko 2024-02-19T15:08:53.014 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/ 2024-02-19T15:08:53.014 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 32153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/act_csum.ko 2024-02-19T15:08:53.014 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23489 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/act_gact.ko 2024-02-19T15:08:53.014 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36769 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/act_mirred.ko 2024-02-19T15:08:53.014 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23377 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/act_nat.ko 2024-02-19T15:08:53.014 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 43417 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/act_pedit.ko 2024-02-19T15:08:53.014 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31929 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/act_police.ko 2024-02-19T15:08:53.014 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20193 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/act_simple.ko 2024-02-19T15:08:53.014 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 35201 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/act_skbedit.ko 2024-02-19T15:08:53.014 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36577 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/act_vlan.ko 2024-02-19T15:08:53.014 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/cls_basic.ko 2024-02-19T15:08:53.014 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 33873 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/cls_bpf.ko 2024-02-19T15:08:53.014 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 19185 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/cls_cgroup.ko 2024-02-19T15:08:53.015 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/cls_flow.ko 2024-02-19T15:08:53.015 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/cls_fw.ko 2024-02-19T15:08:53.015 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 39521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/cls_route.ko 2024-02-19T15:08:53.015 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 62657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/cls_u32.ko 2024-02-19T15:08:53.015 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7353 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/em_cmp.ko 2024-02-19T15:08:53.015 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 13321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/em_ipset.ko 2024-02-19T15:08:53.015 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 33249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/em_meta.ko 2024-02-19T15:08:53.015 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/em_nbyte.ko 2024-02-19T15:08:53.015 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/em_text.ko 2024-02-19T15:08:53.015 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7161 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/em_u32.ko 2024-02-19T15:08:53.015 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21673 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_choke.ko 2024-02-19T15:08:53.015 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20865 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_codel.ko 2024-02-19T15:08:53.015 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 34657 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_drr.ko 2024-02-19T15:08:53.015 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 35793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_fq.ko 2024-02-19T15:08:53.015 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31521 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_fq_codel.ko 2024-02-19T15:08:53.015 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 35457 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_gred.ko 2024-02-19T15:08:53.015 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 49849 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_hfsc.ko 2024-02-19T15:08:53.016 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 21977 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_hhf.ko 2024-02-19T15:08:53.016 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 66449 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_htb.ko 2024-02-19T15:08:53.016 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_ingress.ko 2024-02-19T15:08:53.016 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 30225 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_mqprio.ko 2024-02-19T15:08:53.016 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10809 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_mqprio_lib.ko 2024-02-19T15:08:53.016 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26089 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_multiq.ko 2024-02-19T15:08:53.016 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 36153 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_netem.ko 2024-02-19T15:08:53.016 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 27409 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_pie.ko 2024-02-19T15:08:53.016 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10665 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_plug.ko 2024-02-19T15:08:53.016 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 24777 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_prio.ko 2024-02-19T15:08:53.016 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 48537 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_qfq.ko 2024-02-19T15:08:53.016 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 32817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_red.ko 2024-02-19T15:08:53.016 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 26793 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_sfb.ko 2024-02-19T15:08:53.016 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 31321 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_sfq.ko 2024-02-19T15:08:53.017 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_tbf.ko 2024-02-19T15:08:53.017 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 28273 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sched/sch_teql.ko 2024-02-19T15:08:53.017 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sctp/ 2024-02-19T15:08:53.017 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 932945 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sctp/sctp.ko 2024-02-19T15:08:53.017 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sctp/sctp_diag.ko 2024-02-19T15:08:53.017 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sunrpc/ 2024-02-19T15:08:53.017 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sunrpc/auth_gss/ 2024-02-19T15:08:53.017 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 337009 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-02-19T15:08:53.017 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 55041 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-02-19T15:08:53.017 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 1513905 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sunrpc/sunrpc.ko 2024-02-19T15:08:53.017 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sunrpc/xprtrdma/ 2024-02-19T15:08:53.017 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 704761 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-02-19T15:08:53.017 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/tipc/ 2024-02-19T15:08:53.017 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 10401 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/tipc/diag.ko 2024-02-19T15:08:53.017 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 867249 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/tipc/tipc.ko 2024-02-19T15:08:53.017 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/unix/ 2024-02-19T15:08:53.018 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15937 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/unix/unix_diag.ko 2024-02-19T15:08:53.018 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/vmw_vsock/ 2024-02-19T15:08:53.018 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 87745 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-02-19T15:08:53.018 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 67953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-02-19T15:08:53.018 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 98113 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/vmw_vsock/vsock.ko 2024-02-19T15:08:53.018 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 11081 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/vmw_vsock/vsock_diag.ko 2024-02-19T15:08:53.018 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 17817 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/vmw_vsock/vsock_loopback.ko 2024-02-19T15:08:53.018 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/x25/ 2024-02-19T15:08:53.018 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 131273 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/x25/x25.ko 2024-02-19T15:08:53.018 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/xfrm/ 2024-02-19T15:08:53.018 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 22953 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/xfrm/xfrm_algo.ko 2024-02-19T15:08:53.018 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 20553 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/xfrm/xfrm_ipcomp.ko 2024-02-19T15:08:53.018 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 105097 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/net/xfrm/xfrm_user.ko 2024-02-19T15:08:53.018 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/virt/ 2024-02-19T15:08:53.018 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/virt/lib/ 2024-02-19T15:08:53.018 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 15441 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/kernel/virt/lib/irqbypass.ko 2024-02-19T15:08:53.018 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 642229 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.alias 2024-02-19T15:08:53.019 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 625558 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.alias.bin 2024-02-19T15:08:53.019 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 7677 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.builtin 2024-02-19T15:08:53.019 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 23498 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.builtin.alias.bin 2024-02-19T15:08:53.019 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 9916 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.builtin.bin 2024-02-19T15:08:53.019 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 70627 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.builtin.modinfo 2024-02-19T15:08:53.019 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 147214 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.dep 2024-02-19T15:08:53.019 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 233213 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.dep.bin 2024-02-19T15:08:53.019 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 210 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.devname 2024-02-19T15:08:53.019 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 72614 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.order 2024-02-19T15:08:53.019 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 879 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.softdep 2024-02-19T15:08:53.019 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 257955 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.symbols 2024-02-19T15:08:53.019 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 312782 2024-02-19 07:58 ./lib/modules/6.8.0-rc4-g5d6d84d6d8d3/modules.symbols.bin 2024-02-19T15:08:53.019 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./usr/ 2024-02-19T15:08:53.019 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./usr/share/ 2024-02-19T15:08:53.019 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./usr/share/doc/ 2024-02-19T15:08:53.019 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x root/root 0 2024-02-19 07:58 ./usr/share/doc/linux-image-6.8.0-rc4-g5d6d84d6d8d3/ 2024-02-19T15:08:53.020 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 173 2024-02-19 07:58 ./usr/share/doc/linux-image-6.8.0-rc4-g5d6d84d6d8d3/changelog.Debian.gz 2024-02-19T15:08:53.020 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r-- root/root 691 2024-02-19 07:58 ./usr/share/doc/linux-image-6.8.0-rc4-g5d6d84d6d8d3/copyright 2024-02-19T15:08:53.024 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:53.024 INFO:teuthology.task.kernel:searching for kernel 6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:53.024 DEBUG:teuthology.orchestra.run.smithi116:> egrep '(submenu|menuentry.*6.8.0-rc4-g5d6d84d6d8d3).*{' /boot/grub/grub.cfg 2024-02-19T15:08:53.036 INFO:teuthology.orchestra.run.smithi116.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-02-19T15:08:53.036 INFO:teuthology.orchestra.run.smithi116.stdout: menuentry 'Ubuntu, with Linux 6.8.0-rc4-g5d6d84d6d8d3' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.8.0-rc4-g5d6d84d6d8d3-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-02-19T15:08:53.036 INFO:teuthology.orchestra.run.smithi116.stdout: menuentry 'Ubuntu, with Linux 6.8.0-rc4-g5d6d84d6d8d3 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.8.0-rc4-g5d6d84d6d8d3-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-02-19T15:08:53.037 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2024-02-19T15:08:53.037 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:53.037 DEBUG:teuthology.orchestra.run.smithi116:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2024-02-19T15:08:53.080 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi116.front.sepia.ceph.com to complete... 2024-02-19T15:08:53.498 INFO:teuthology.orchestra.run.smithi116.stderr:Sourcing file `/etc/default/grub' 2024-02-19T15:08:53.500 INFO:teuthology.orchestra.run.smithi116.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-02-19T15:08:53.505 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2024-02-19T15:08:53.744 INFO:teuthology.orchestra.run.smithi116.stderr:Found linux image: /boot/vmlinuz-6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:53.750 INFO:teuthology.orchestra.run.smithi116.stderr:Found initrd image: /boot/initrd.img-6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:08:53.984 INFO:teuthology.orchestra.run.smithi116.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-02-19T15:08:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-02-19T15:08:54.030 INFO:teuthology.orchestra.run.smithi116.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-02-19T15:08:54.036 INFO:teuthology.orchestra.run.smithi116.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-02-19T15:08:54.149 INFO:teuthology.orchestra.run.smithi116.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-02-19T15:08:54.149 INFO:teuthology.orchestra.run.smithi116.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-02-19T15:08:54.149 INFO:teuthology.orchestra.run.smithi116.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-02-19T15:08:54.181 INFO:teuthology.orchestra.run.smithi116.stderr:done 2024-02-19T15:09:03.365 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-02-19T15:09:03.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-02-19T15:09:03.366 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T15:09:21.852 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.47 2024-02-19T15:09:25.317 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-02-19T15:09:25.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-02-19T15:09:25.318 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T15:09:30.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-02-19T15:09:30.856 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T15:09:43.868 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-02-19T15:09:52.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-02-19T15:09:52.873 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T15:09:55.932 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-02-19T15:10:07.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-02-19T15:10:07.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T15:10:26.364 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-02-19T15:10:30.863 DEBUG:teuthology.orchestra.remote:timed out 2024-02-19T15:10:41.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-02-19T15:10:41.373 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T15:10:41.689 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-02-19T15:10:42.472 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-02-19T15:10:42.472 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.8.0-rc4-g5d6d84d6d8d3"... 2024-02-19T15:10:42.472 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-02-19T15:10:42.523 INFO:teuthology.orchestra.run.smithi116.stdout:6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:10:42.523 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc4-g5d6d84d6d8d3 vs 6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:10:42.523 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T15:10:42.523 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-02-19T15:10:42.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-02-19T15:10:42.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T15:10:43.065 DEBUG:teuthology.orchestra.run.smithi047:> true 2024-02-19T15:10:43.524 DEBUG:teuthology.parallel:result is None 2024-02-19T15:10:43.754 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-02-19T15:10:43.754 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-rc4-g5d6d84d6d8d3"... 2024-02-19T15:10:43.755 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2024-02-19T15:10:43.806 INFO:teuthology.orchestra.run.smithi047.stdout:6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:10:43.806 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc4-g5d6d84d6d8d3 vs 6.8.0-rc4-g5d6d84d6d8d3 2024-02-19T15:10:43.806 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T15:10:43.806 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-02-19T15:10:44.808 DEBUG:teuthology.parallel:result is None 2024-02-19T15:10:44.808 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-19T15:10:44.814 INFO:teuthology.task.internal:Creating test directory... 2024-02-19T15:10:44.815 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T15:10:44.818 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T15:10:44.827 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-19T15:10:44.836 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-19T15:10:44.843 INFO:teuthology.task.internal:Creating archive directory... 2024-02-19T15:10:44.843 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T15:10:44.869 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T15:10:44.902 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-19T15:10:44.908 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-19T15:10:44.908 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T15:10:44.933 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T15:10:45.016 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T15:10:45.032 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T15:10:45.034 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T15:10:45.049 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T15:10:45.052 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-19T15:10:45.058 INFO:teuthology.task.internal:Configuring sudo... 2024-02-19T15:10:45.058 DEBUG:teuthology.orchestra.run.smithi047:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T15:10:45.061 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T15:10:45.082 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-19T15:10:45.091 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-19T15:10:45.091 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T15:10:45.125 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T15:10:45.133 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T15:10:45.183 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T15:10:45.236 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:10:45.236 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T15:10:45.305 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T15:10:45.315 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T15:10:45.364 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:10:45.364 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T15:10:45.438 DEBUG:teuthology.orchestra.run.smithi047:> sudo service rsyslog restart 2024-02-19T15:10:45.441 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-02-19T15:10:45.578 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-19T15:10:45.588 INFO:teuthology.task.internal:Starting timer... 2024-02-19T15:10:45.588 INFO:teuthology.run_tasks:Running task pcp... 2024-02-19T15:10:45.598 INFO:teuthology.run_tasks:Running task selinux... 2024-02-19T15:10:45.608 DEBUG:teuthology.task.selinux:Excluding smithi047: OS 'ubuntu' does not support SELinux 2024-02-19T15:10:45.608 DEBUG:teuthology.task.selinux:Excluding smithi116: OS 'ubuntu' does not support SELinux 2024-02-19T15:10:45.609 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-19T15:10:45.609 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-02-19T15:10:45.609 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-19T15:10:45.609 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-19T15:10:45.617 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-02-19T15:10:45.623 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-02-19T15:10:45.769 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-19T15:10:45.785 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-19T15:10:45.787 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi047.front.sepia.ceph.com,smithi116.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-19T15:18:18.170 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi047.front.sepia.ceph.com'), Remote(name='ubuntu@smithi116.front.sepia.ceph.com')] 2024-02-19T15:18:18.172 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-02-19T15:18:18.173 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T15:18:18.258 DEBUG:teuthology.orchestra.run.smithi047:> true 2024-02-19T15:18:18.379 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-02-19T15:18:18.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-02-19T15:18:18.379 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T15:18:18.457 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-02-19T15:18:18.583 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-02-19T15:18:18.583 INFO:teuthology.run_tasks:Running task clock... 2024-02-19T15:18:18.593 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-19T15:18:18.593 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T15:18:18.593 DEBUG:teuthology.orchestra.run.smithi047:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T15:18:18.597 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T15:18:18.597 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T15:18:18.633 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:18 ntpd[17077]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-02-19T15:18:18.633 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:18 ntpd[17077]: Command line: ntpd -gq 2024-02-19T15:18:18.633 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:18 ntpd[17077]: ---------------------------------------------------- 2024-02-19T15:18:18.633 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:18 ntpd[17077]: ntp-4 is maintained by Network Time Foundation, 2024-02-19T15:18:18.633 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:18 ntpd[17077]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-02-19T15:18:18.633 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:18 ntpd[17077]: corporation. Support and training for ntp-4 are 2024-02-19T15:18:18.633 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:18 ntpd[17077]: available at https://www.nwtime.org/support 2024-02-19T15:18:18.633 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:18 ntpd[17077]: ---------------------------------------------------- 2024-02-19T15:18:18.634 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:18 ntpd[17077]: proto: precision = 0.045 usec (-24) 2024-02-19T15:18:18.634 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:18 ntpd[17077]: basedate set to 2022-02-04 2024-02-19T15:18:18.634 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:18 ntpd[17077]: gps base set to 2022-02-06 (week 2196) 2024-02-19T15:18:18.634 INFO:teuthology.orchestra.run.smithi047.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-19T15:18:18.634 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:18 ntpd[17077]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-19T15:18:18.635 INFO:teuthology.orchestra.run.smithi047.stderr:restrict ::: KOD does nothing without LIMITED. 2024-02-19T15:18:18.635 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:18 ntpd[17077]: restrict ::: KOD does nothing without LIMITED. 2024-02-19T15:18:18.636 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:18 ntpd[17077]: Listen and drop on 0 v6wildcard [::]:123 2024-02-19T15:18:18.636 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:18 ntpd[17077]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-02-19T15:18:18.636 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:18 ntpd[17077]: Listen normally on 2 lo 127.0.0.1:123 2024-02-19T15:18:18.636 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:18 ntpd[17077]: Listen normally on 3 ens1f0 172.21.15.47:123 2024-02-19T15:18:18.636 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:18 ntpd[17077]: Listen normally on 4 lo [::1]:123 2024-02-19T15:18:18.636 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:18 ntpd[17077]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:148e%3]:123 2024-02-19T15:18:18.636 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:18 ntpd[17077]: Listening on routing socket on fd #22 for interface updates 2024-02-19T15:18:18.666 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:18 ntpd[17161]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-02-19T15:18:18.666 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:18 ntpd[17161]: Command line: ntpd -gq 2024-02-19T15:18:18.666 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:18 ntpd[17161]: ---------------------------------------------------- 2024-02-19T15:18:18.666 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:18 ntpd[17161]: ntp-4 is maintained by Network Time Foundation, 2024-02-19T15:18:18.666 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:18 ntpd[17161]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-02-19T15:18:18.666 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:18 ntpd[17161]: corporation. Support and training for ntp-4 are 2024-02-19T15:18:18.666 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:18 ntpd[17161]: available at https://www.nwtime.org/support 2024-02-19T15:18:18.666 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:18 ntpd[17161]: ---------------------------------------------------- 2024-02-19T15:18:18.667 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:18 ntpd[17161]: proto: precision = 0.040 usec (-24) 2024-02-19T15:18:18.667 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:18 ntpd[17161]: basedate set to 2022-02-04 2024-02-19T15:18:18.667 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:18 ntpd[17161]: gps base set to 2022-02-06 (week 2196) 2024-02-19T15:18:18.667 INFO:teuthology.orchestra.run.smithi116.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-19T15:18:18.668 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:18 ntpd[17161]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-19T15:18:18.668 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:18 ntpd[17161]: restrict ::: KOD does nothing without LIMITED. 2024-02-19T15:18:18.668 INFO:teuthology.orchestra.run.smithi116.stderr:restrict ::: KOD does nothing without LIMITED. 2024-02-19T15:18:18.669 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:18 ntpd[17161]: Listen and drop on 0 v6wildcard [::]:123 2024-02-19T15:18:18.669 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:18 ntpd[17161]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-02-19T15:18:18.669 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:18 ntpd[17161]: Listen normally on 2 lo 127.0.0.1:123 2024-02-19T15:18:18.669 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:18 ntpd[17161]: Listen normally on 3 enp3s0f1 172.21.15.116:123 2024-02-19T15:18:18.669 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:18 ntpd[17161]: Listen normally on 4 lo [::1]:123 2024-02-19T15:18:18.670 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:18 ntpd[17161]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:3f67%5]:123 2024-02-19T15:18:18.670 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:18 ntpd[17161]: Listening on routing socket on fd #22 for interface updates 2024-02-19T15:18:19.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:19 ntpd[17077]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:19.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:19 ntpd[17077]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:19.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:19 ntpd[17077]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:19.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:19 ntpd[17077]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:19.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:19 ntpd[17077]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-19T15:18:19.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:19 ntpd[17077]: can't open /var/log/ntpstats/peerstats.20240219: Permission denied 2024-02-19T15:18:19.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:19 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:19.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:19 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:19.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:19 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:19.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:19 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:19.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:19 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:19.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:19 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:19 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:19 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:19 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:19 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:19 ntpd[17161]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-19T15:18:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:19 ntpd[17161]: can't open /var/log/ntpstats/peerstats.20240219: Permission denied 2024-02-19T15:18:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:19 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:19 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:19 ntpd[17161]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-19T15:18:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:19 ntpd[17161]: can't open /var/log/ntpstats/peerstats.20240219: Permission denied 2024-02-19T15:18:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:19 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:19 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:19 ntpd[17161]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-19T15:18:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:19 ntpd[17161]: can't open /var/log/ntpstats/peerstats.20240219: Permission denied 2024-02-19T15:18:21.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:21 ntpd[17077]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:21.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:21 ntpd[17077]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:21.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:21 ntpd[17077]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:21.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:21 ntpd[17077]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:21.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:21 ntpd[17077]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:21.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:21 ntpd[17077]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:21.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:21 ntpd[17077]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:21.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:21 ntpd[17077]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:21.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:21 ntpd[17077]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-19T15:18:21.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:21 ntpd[17077]: can't open /var/log/ntpstats/peerstats.20240219: Permission denied 2024-02-19T15:18:21.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:21 ntpd[17077]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:21.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:21 ntpd[17077]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:21.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:21 ntpd[17077]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-19T15:18:21.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:21 ntpd[17077]: can't open /var/log/ntpstats/peerstats.20240219: Permission denied 2024-02-19T15:18:21.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:21 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:21.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:21 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:21.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:21 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:21.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:21 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:21.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:21 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:21.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:21 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:21.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:21 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:21.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:21 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:21.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:21 ntpd[17161]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-19T15:18:21.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:21 ntpd[17161]: can't open /var/log/ntpstats/peerstats.20240219: Permission denied 2024-02-19T15:18:21.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:21 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:21.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:21 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:21.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:21 ntpd[17161]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-19T15:18:21.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:21 ntpd[17161]: can't open /var/log/ntpstats/peerstats.20240219: Permission denied 2024-02-19T15:18:21.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:21 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:21.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:21 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:21.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:21 ntpd[17161]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-19T15:18:21.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:21 ntpd[17161]: can't open /var/log/ntpstats/peerstats.20240219: Permission denied 2024-02-19T15:18:23.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:23 ntpd[17077]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:23.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:23 ntpd[17077]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:23.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:23 ntpd[17077]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:23.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:23 ntpd[17077]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:23.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:23 ntpd[17077]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:23.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:23 ntpd[17077]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:23.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:23 ntpd[17077]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:23.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:23 ntpd[17077]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:23.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:23 ntpd[17077]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-19T15:18:23.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:23 ntpd[17077]: can't open /var/log/ntpstats/peerstats.20240219: Permission denied 2024-02-19T15:18:23.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:23 ntpd[17077]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:23.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:23 ntpd[17077]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:23.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:23 ntpd[17077]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-19T15:18:23.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:23 ntpd[17077]: can't open /var/log/ntpstats/peerstats.20240219: Permission denied 2024-02-19T15:18:23.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:23 ntpd[17077]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:23.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:23 ntpd[17077]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:23.636 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:23 ntpd[17077]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-19T15:18:23.636 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:23 ntpd[17077]: can't open /var/log/ntpstats/peerstats.20240219: Permission denied 2024-02-19T15:18:23.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:23 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:23.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:23 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:23.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:23 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:23.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:23 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:23.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:23 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:23.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:23 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:23.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:23 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:23.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:23 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:23.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:23 ntpd[17161]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-19T15:18:23.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:23 ntpd[17161]: can't open /var/log/ntpstats/peerstats.20240219: Permission denied 2024-02-19T15:18:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:23 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:23 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:23 ntpd[17161]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-19T15:18:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:23 ntpd[17161]: can't open /var/log/ntpstats/peerstats.20240219: Permission denied 2024-02-19T15:18:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:23 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:23 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:23 ntpd[17161]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-19T15:18:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:23 ntpd[17161]: can't open /var/log/ntpstats/peerstats.20240219: Permission denied 2024-02-19T15:18:25.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:25 ntpd[17077]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:25.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:25 ntpd[17077]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:25.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:25 ntpd[17077]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:25.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:25 ntpd[17077]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:25.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:25 ntpd[17077]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:25.634 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:25 ntpd[17077]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:25 ntpd[17077]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:25 ntpd[17077]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:25 ntpd[17077]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-19T15:18:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:25 ntpd[17077]: can't open /var/log/ntpstats/peerstats.20240219: Permission denied 2024-02-19T15:18:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:25 ntpd[17077]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:25 ntpd[17077]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:25 ntpd[17077]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-19T15:18:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:25 ntpd[17077]: can't open /var/log/ntpstats/peerstats.20240219: Permission denied 2024-02-19T15:18:25.635 INFO:teuthology.orchestra.run.smithi047.stdout:19 Feb 15:18:25 ntpd[17077]: ntpd: time slew -0.000864 s 2024-02-19T15:18:25.635 INFO:teuthology.orchestra.run.smithi047.stdout:ntpd: time slew -0.000864s 2024-02-19T15:18:25.636 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:25 ntpd[17077]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-02-19T15:18:25.636 INFO:teuthology.orchestra.run.smithi047.stderr:19 Feb 15:18:25 ntpd[17077]: can't open /var/log/ntpstats/loopstats.20240219: Permission denied 2024-02-19T15:18:25.667 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:25 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:25.667 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:25 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:25.667 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:25 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:25.667 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:25 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:25.667 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:25 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:25.667 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:25 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:25.667 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:25 ntpd[17161]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-19T15:18:25.667 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:25 ntpd[17161]: can't open /var/log/ntpstats/rawstats.20240219: Permission denied 2024-02-19T15:18:25.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:25 ntpd[17161]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-19T15:18:25.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:25 ntpd[17161]: can't open /var/log/ntpstats/peerstats.20240219: Permission denied 2024-02-19T15:18:25.668 INFO:teuthology.orchestra.run.smithi116.stdout:19 Feb 15:18:25 ntpd[17161]: ntpd: time slew +0.001753 s 2024-02-19T15:18:25.668 INFO:teuthology.orchestra.run.smithi116.stdout:ntpd: time slew +0.001753s 2024-02-19T15:18:25.668 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:25 ntpd[17161]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-02-19T15:18:25.669 INFO:teuthology.orchestra.run.smithi116.stderr:19 Feb 15:18:25 ntpd[17161]: can't open /var/log/ntpstats/loopstats.20240219: Permission denied 2024-02-19T15:18:25.735 INFO:teuthology.orchestra.run.smithi047.stdout: remote refid st t when poll reach delay offset jitter 2024-02-19T15:18:25.735 INFO:teuthology.orchestra.run.smithi047.stdout:============================================================================== 2024-02-19T15:18:25.735 INFO:teuthology.orchestra.run.smithi047.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-19T15:18:25.735 INFO:teuthology.orchestra.run.smithi047.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-19T15:18:25.735 INFO:teuthology.orchestra.run.smithi047.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-19T15:18:25.735 INFO:teuthology.orchestra.run.smithi047.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-19T15:18:25.799 INFO:teuthology.orchestra.run.smithi116.stdout: remote refid st t when poll reach delay offset jitter 2024-02-19T15:18:25.799 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================== 2024-02-19T15:18:25.799 INFO:teuthology.orchestra.run.smithi116.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-19T15:18:25.799 INFO:teuthology.orchestra.run.smithi116.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-19T15:18:25.799 INFO:teuthology.orchestra.run.smithi116.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-19T15:18:25.799 INFO:teuthology.orchestra.run.smithi116.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-19T15:18:25.800 INFO:teuthology.run_tasks:Running task install... 2024-02-19T15:18:25.810 DEBUG:teuthology.task.install:project ceph 2024-02-19T15:18:25.810 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee'}} 2024-02-19T15:18:25.810 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee'} 2024-02-19T15:18:25.810 INFO:teuthology.task.install:Using flavor: default 2024-02-19T15:18:25.816 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-19T15:18:25.816 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-02-19T15:18:25.816 DEBUG:teuthology.orchestra.run.smithi047:> sudo apt-key list | grep Ceph 2024-02-19T15:18:25.817 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-key list | grep Ceph 2024-02-19T15:18:25.922 INFO:teuthology.orchestra.run.smithi047.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-02-19T15:18:25.943 INFO:teuthology.orchestra.run.smithi116.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-02-19T15:18:25.974 INFO:teuthology.orchestra.run.smithi047.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-02-19T15:18:25.975 INFO:teuthology.orchestra.run.smithi047.stdout:uid [ unknown] Ceph.com (release key) 2024-02-19T15:18:25.975 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-02-19T15:18:25.976 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-02-19T15:18:25.976 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9f8908eed2b2d7e8c316bd5be7ed6e49024931ee 2024-02-19T15:18:25.996 INFO:teuthology.orchestra.run.smithi116.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-02-19T15:18:25.996 INFO:teuthology.orchestra.run.smithi116.stdout:uid [ unknown] Ceph.com (release key) 2024-02-19T15:18:25.997 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-02-19T15:18:25.997 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-02-19T15:18:25.997 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9f8908eed2b2d7e8c316bd5be7ed6e49024931ee 2024-02-19T15:18:26.123 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default/ 2024-02-19T15:18:26.123 INFO:teuthology.task.install.deb:Package version is 19.0.0-1451-g9f8908ee-1jammy 2024-02-19T15:18:26.221 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default/ 2024-02-19T15:18:26.221 INFO:teuthology.task.install.deb:Package version is 19.0.0-1451-g9f8908ee-1jammy 2024-02-19T15:18:26.238 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:18:26.238 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-02-19T15:18:26.258 DEBUG:teuthology.orchestra.run.smithi047:> sudo apt-get update 2024-02-19T15:18:26.327 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:18:26.327 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-02-19T15:18:26.346 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-get update 2024-02-19T15:18:26.448 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-02-19T15:18:26.536 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-02-19T15:18:26.559 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-02-19T15:18:26.563 INFO:teuthology.orchestra.run.smithi047.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy InRelease 2024-02-19T15:18:26.579 INFO:teuthology.orchestra.run.smithi116.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy InRelease 2024-02-19T15:18:26.599 INFO:teuthology.orchestra.run.smithi047.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2024-02-19T15:18:26.600 INFO:teuthology.orchestra.run.smithi116.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2024-02-19T15:18:26.622 INFO:teuthology.orchestra.run.smithi116.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy Release.gpg 2024-02-19T15:18:26.637 INFO:teuthology.orchestra.run.smithi047.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy Release.gpg 2024-02-19T15:18:26.641 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-02-19T15:18:26.643 INFO:teuthology.orchestra.run.smithi116.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,417 B] 2024-02-19T15:18:26.657 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-02-19T15:18:26.673 INFO:teuthology.orchestra.run.smithi047.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,417 B] 2024-02-19T15:18:26.677 INFO:teuthology.orchestra.run.smithi116.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-02-19T15:18:26.721 INFO:teuthology.orchestra.run.smithi047.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-02-19T15:18:26.723 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-02-19T15:18:26.729 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-02-19T15:18:26.811 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-02-19T15:18:27.052 INFO:teuthology.orchestra.run.smithi047.stdout:Fetched 28.2 kB in 1s (49.5 kB/s) 2024-02-19T15:18:27.121 INFO:teuthology.orchestra.run.smithi116.stdout:Fetched 28.2 kB in 1s (50.9 kB/s) 2024-02-19T15:18:28.023 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2024-02-19T15:18:28.037 DEBUG:teuthology.orchestra.run.smithi047:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1451-g9f8908ee-1jammy cephadm=19.0.0-1451-g9f8908ee-1jammy ceph-mds=19.0.0-1451-g9f8908ee-1jammy ceph-mgr=19.0.0-1451-g9f8908ee-1jammy ceph-common=19.0.0-1451-g9f8908ee-1jammy ceph-fuse=19.0.0-1451-g9f8908ee-1jammy ceph-test=19.0.0-1451-g9f8908ee-1jammy ceph-volume=19.0.0-1451-g9f8908ee-1jammy radosgw=19.0.0-1451-g9f8908ee-1jammy python3-rados=19.0.0-1451-g9f8908ee-1jammy python3-rgw=19.0.0-1451-g9f8908ee-1jammy python3-cephfs=19.0.0-1451-g9f8908ee-1jammy python3-rbd=19.0.0-1451-g9f8908ee-1jammy libcephfs2=19.0.0-1451-g9f8908ee-1jammy libcephfs-dev=19.0.0-1451-g9f8908ee-1jammy librados2=19.0.0-1451-g9f8908ee-1jammy librbd1=19.0.0-1451-g9f8908ee-1jammy rbd-fuse=19.0.0-1451-g9f8908ee-1jammy python3-cephfs=19.0.0-1451-g9f8908ee-1jammy cephfs-shell=19.0.0-1451-g9f8908ee-1jammy cephfs-top=19.0.0-1451-g9f8908ee-1jammy cephfs-mirror=19.0.0-1451-g9f8908ee-1jammy 2024-02-19T15:18:28.062 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-02-19T15:18:28.076 DEBUG:teuthology.orchestra.run.smithi116:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1451-g9f8908ee-1jammy cephadm=19.0.0-1451-g9f8908ee-1jammy ceph-mds=19.0.0-1451-g9f8908ee-1jammy ceph-mgr=19.0.0-1451-g9f8908ee-1jammy ceph-common=19.0.0-1451-g9f8908ee-1jammy ceph-fuse=19.0.0-1451-g9f8908ee-1jammy ceph-test=19.0.0-1451-g9f8908ee-1jammy ceph-volume=19.0.0-1451-g9f8908ee-1jammy radosgw=19.0.0-1451-g9f8908ee-1jammy python3-rados=19.0.0-1451-g9f8908ee-1jammy python3-rgw=19.0.0-1451-g9f8908ee-1jammy python3-cephfs=19.0.0-1451-g9f8908ee-1jammy python3-rbd=19.0.0-1451-g9f8908ee-1jammy libcephfs2=19.0.0-1451-g9f8908ee-1jammy libcephfs-dev=19.0.0-1451-g9f8908ee-1jammy librados2=19.0.0-1451-g9f8908ee-1jammy librbd1=19.0.0-1451-g9f8908ee-1jammy rbd-fuse=19.0.0-1451-g9f8908ee-1jammy python3-cephfs=19.0.0-1451-g9f8908ee-1jammy cephfs-shell=19.0.0-1451-g9f8908ee-1jammy cephfs-top=19.0.0-1451-g9f8908ee-1jammy cephfs-mirror=19.0.0-1451-g9f8908ee-1jammy 2024-02-19T15:18:28.113 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2024-02-19T15:18:28.148 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-02-19T15:18:28.290 INFO:teuthology.orchestra.run.smithi047.stdout:Building dependency tree... 2024-02-19T15:18:28.291 INFO:teuthology.orchestra.run.smithi047.stdout:Reading state information... 2024-02-19T15:18:28.314 INFO:teuthology.orchestra.run.smithi116.stdout:Building dependency tree... 2024-02-19T15:18:28.314 INFO:teuthology.orchestra.run.smithi116.stdout:Reading state information... 2024-02-19T15:18:28.457 INFO:teuthology.orchestra.run.smithi047.stdout:The following packages were automatically installed and are no longer required: 2024-02-19T15:18:28.457 INFO:teuthology.orchestra.run.smithi047.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-19T15:18:28.457 INFO:teuthology.orchestra.run.smithi047.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-19T15:18:28.457 INFO:teuthology.orchestra.run.smithi047.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-19T15:18:28.458 INFO:teuthology.orchestra.run.smithi047.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-02-19T15:18:28.458 INFO:teuthology.orchestra.run.smithi047.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-02-19T15:18:28.458 INFO:teuthology.orchestra.run.smithi047.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-19T15:18:28.458 INFO:teuthology.orchestra.run.smithi047.stdout:The following additional packages will be installed: 2024-02-19T15:18:28.458 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-02-19T15:18:28.460 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-02-19T15:18:28.460 INFO:teuthology.orchestra.run.smithi047.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-02-19T15:18:28.460 INFO:teuthology.orchestra.run.smithi047.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-02-19T15:18:28.461 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-02-19T15:18:28.461 INFO:teuthology.orchestra.run.smithi047.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-02-19T15:18:28.461 INFO:teuthology.orchestra.run.smithi047.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-02-19T15:18:28.461 INFO:teuthology.orchestra.run.smithi047.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-02-19T15:18:28.461 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-02-19T15:18:28.461 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-02-19T15:18:28.461 INFO:teuthology.orchestra.run.smithi047.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-02-19T15:18:28.461 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-02-19T15:18:28.461 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-02-19T15:18:28.461 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-02-19T15:18:28.461 INFO:teuthology.orchestra.run.smithi047.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-02-19T15:18:28.461 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-02-19T15:18:28.461 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-02-19T15:18:28.461 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-02-19T15:18:28.461 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-02-19T15:18:28.462 INFO:teuthology.orchestra.run.smithi047.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-02-19T15:18:28.462 INFO:teuthology.orchestra.run.smithi047.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-02-19T15:18:28.462 INFO:teuthology.orchestra.run.smithi047.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-02-19T15:18:28.462 INFO:teuthology.orchestra.run.smithi047.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-02-19T15:18:28.462 INFO:teuthology.orchestra.run.smithi047.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-02-19T15:18:28.462 INFO:teuthology.orchestra.run.smithi047.stdout:Suggested packages: 2024-02-19T15:18:28.462 INFO:teuthology.orchestra.run.smithi047.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-02-19T15:18:28.462 INFO:teuthology.orchestra.run.smithi047.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-02-19T15:18:28.462 INFO:teuthology.orchestra.run.smithi047.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-02-19T15:18:28.462 INFO:teuthology.orchestra.run.smithi047.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-02-19T15:18:28.463 INFO:teuthology.orchestra.run.smithi047.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-02-19T15:18:28.463 INFO:teuthology.orchestra.run.smithi047.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-02-19T15:18:28.463 INFO:teuthology.orchestra.run.smithi047.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-02-19T15:18:28.463 INFO:teuthology.orchestra.run.smithi047.stdout:Recommended packages: 2024-02-19T15:18:28.463 INFO:teuthology.orchestra.run.smithi047.stdout: btrfs-tools 2024-02-19T15:18:28.467 INFO:teuthology.orchestra.run.smithi116.stdout:The following packages were automatically installed and are no longer required: 2024-02-19T15:18:28.467 INFO:teuthology.orchestra.run.smithi116.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-19T15:18:28.467 INFO:teuthology.orchestra.run.smithi116.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-19T15:18:28.468 INFO:teuthology.orchestra.run.smithi116.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-19T15:18:28.468 INFO:teuthology.orchestra.run.smithi116.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-02-19T15:18:28.468 INFO:teuthology.orchestra.run.smithi116.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-02-19T15:18:28.468 INFO:teuthology.orchestra.run.smithi116.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-19T15:18:28.469 INFO:teuthology.orchestra.run.smithi116.stdout:The following additional packages will be installed: 2024-02-19T15:18:28.469 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-02-19T15:18:28.469 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-02-19T15:18:28.470 INFO:teuthology.orchestra.run.smithi116.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-02-19T15:18:28.470 INFO:teuthology.orchestra.run.smithi116.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-02-19T15:18:28.470 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-02-19T15:18:28.470 INFO:teuthology.orchestra.run.smithi116.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-02-19T15:18:28.470 INFO:teuthology.orchestra.run.smithi116.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-02-19T15:18:28.470 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-02-19T15:18:28.470 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-02-19T15:18:28.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-02-19T15:18:28.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-02-19T15:18:28.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-02-19T15:18:28.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-02-19T15:18:28.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-02-19T15:18:28.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-02-19T15:18:28.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-02-19T15:18:28.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-02-19T15:18:28.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-02-19T15:18:28.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-02-19T15:18:28.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-02-19T15:18:28.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-02-19T15:18:28.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-02-19T15:18:28.471 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-02-19T15:18:28.471 INFO:teuthology.orchestra.run.smithi116.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-02-19T15:18:28.472 INFO:teuthology.orchestra.run.smithi116.stdout:Suggested packages: 2024-02-19T15:18:28.472 INFO:teuthology.orchestra.run.smithi116.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-02-19T15:18:28.472 INFO:teuthology.orchestra.run.smithi116.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-02-19T15:18:28.472 INFO:teuthology.orchestra.run.smithi116.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-02-19T15:18:28.472 INFO:teuthology.orchestra.run.smithi116.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-02-19T15:18:28.472 INFO:teuthology.orchestra.run.smithi116.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-02-19T15:18:28.472 INFO:teuthology.orchestra.run.smithi116.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-02-19T15:18:28.472 INFO:teuthology.orchestra.run.smithi116.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-02-19T15:18:28.472 INFO:teuthology.orchestra.run.smithi116.stdout:Recommended packages: 2024-02-19T15:18:28.473 INFO:teuthology.orchestra.run.smithi116.stdout: btrfs-tools 2024-02-19T15:18:28.617 INFO:teuthology.orchestra.run.smithi047.stdout:The following NEW packages will be installed: 2024-02-19T15:18:28.617 INFO:teuthology.orchestra.run.smithi047.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-02-19T15:18:28.617 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-02-19T15:18:28.617 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-02-19T15:18:28.617 INFO:teuthology.orchestra.run.smithi047.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-02-19T15:18:28.619 INFO:teuthology.orchestra.run.smithi047.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-02-19T15:18:28.619 INFO:teuthology.orchestra.run.smithi047.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-02-19T15:18:28.619 INFO:teuthology.orchestra.run.smithi047.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-02-19T15:18:28.619 INFO:teuthology.orchestra.run.smithi047.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-02-19T15:18:28.619 INFO:teuthology.orchestra.run.smithi047.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-02-19T15:18:28.619 INFO:teuthology.orchestra.run.smithi047.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-02-19T15:18:28.619 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-02-19T15:18:28.619 INFO:teuthology.orchestra.run.smithi047.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-02-19T15:18:28.619 INFO:teuthology.orchestra.run.smithi047.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-02-19T15:18:28.619 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-02-19T15:18:28.620 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-02-19T15:18:28.620 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-02-19T15:18:28.620 INFO:teuthology.orchestra.run.smithi047.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-02-19T15:18:28.620 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-02-19T15:18:28.620 INFO:teuthology.orchestra.run.smithi047.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-02-19T15:18:28.620 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-02-19T15:18:28.620 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-02-19T15:18:28.620 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-02-19T15:18:28.620 INFO:teuthology.orchestra.run.smithi047.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-02-19T15:18:28.620 INFO:teuthology.orchestra.run.smithi047.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-02-19T15:18:28.620 INFO:teuthology.orchestra.run.smithi047.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-02-19T15:18:28.620 INFO:teuthology.orchestra.run.smithi047.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-02-19T15:18:28.620 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-02-19T15:18:28.620 INFO:teuthology.orchestra.run.smithi047.stdout: socat unzip xmlstarlet zip 2024-02-19T15:18:28.622 INFO:teuthology.orchestra.run.smithi116.stdout:The following NEW packages will be installed: 2024-02-19T15:18:28.622 INFO:teuthology.orchestra.run.smithi116.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-02-19T15:18:28.622 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-02-19T15:18:28.622 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-02-19T15:18:28.622 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-02-19T15:18:28.622 INFO:teuthology.orchestra.run.smithi116.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-02-19T15:18:28.622 INFO:teuthology.orchestra.run.smithi116.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-02-19T15:18:28.623 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-02-19T15:18:28.623 INFO:teuthology.orchestra.run.smithi116.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-02-19T15:18:28.623 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-02-19T15:18:28.623 INFO:teuthology.orchestra.run.smithi116.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-02-19T15:18:28.623 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-02-19T15:18:28.623 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-02-19T15:18:28.623 INFO:teuthology.orchestra.run.smithi116.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-02-19T15:18:28.623 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-02-19T15:18:28.623 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-02-19T15:18:28.623 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-02-19T15:18:28.623 INFO:teuthology.orchestra.run.smithi116.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-02-19T15:18:28.623 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-02-19T15:18:28.623 INFO:teuthology.orchestra.run.smithi116.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-02-19T15:18:28.623 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-02-19T15:18:28.623 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-02-19T15:18:28.623 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-02-19T15:18:28.624 INFO:teuthology.orchestra.run.smithi116.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-02-19T15:18:28.624 INFO:teuthology.orchestra.run.smithi116.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-02-19T15:18:28.624 INFO:teuthology.orchestra.run.smithi116.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-02-19T15:18:28.624 INFO:teuthology.orchestra.run.smithi116.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-02-19T15:18:28.624 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-02-19T15:18:28.624 INFO:teuthology.orchestra.run.smithi116.stdout: socat unzip xmlstarlet zip 2024-02-19T15:18:28.710 INFO:teuthology.orchestra.run.smithi047.stdout:0 upgraded, 124 newly installed, 0 to remove and 277 not upgraded. 2024-02-19T15:18:28.710 INFO:teuthology.orchestra.run.smithi047.stdout:Need to get 177 MB of archives. 2024-02-19T15:18:28.710 INFO:teuthology.orchestra.run.smithi047.stdout:After this operation, 734 MB of additional disk space will be used. 2024-02-19T15:18:28.711 INFO:teuthology.orchestra.run.smithi047.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-02-19T15:18:28.712 INFO:teuthology.orchestra.run.smithi116.stdout:0 upgraded, 124 newly installed, 0 to remove and 277 not upgraded. 2024-02-19T15:18:28.712 INFO:teuthology.orchestra.run.smithi116.stdout:Need to get 177 MB of archives. 2024-02-19T15:18:28.712 INFO:teuthology.orchestra.run.smithi116.stdout:After this operation, 734 MB of additional disk space will be used. 2024-02-19T15:18:28.712 INFO:teuthology.orchestra.run.smithi116.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-02-19T15:18:28.731 INFO:teuthology.orchestra.run.smithi047.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-02-19T15:18:28.732 INFO:teuthology.orchestra.run.smithi116.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-02-19T15:18:28.744 INFO:teuthology.orchestra.run.smithi116.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-02-19T15:18:28.745 INFO:teuthology.orchestra.run.smithi047.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-02-19T15:18:28.792 INFO:teuthology.orchestra.run.smithi047.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-02-19T15:18:28.795 INFO:teuthology.orchestra.run.smithi116.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-02-19T15:18:28.816 INFO:teuthology.orchestra.run.smithi047.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-02-19T15:18:28.818 INFO:teuthology.orchestra.run.smithi116.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-02-19T15:18:28.819 INFO:teuthology.orchestra.run.smithi047.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-02-19T15:18:28.821 INFO:teuthology.orchestra.run.smithi116.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-02-19T15:18:28.833 INFO:teuthology.orchestra.run.smithi047.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-02-19T15:18:28.837 INFO:teuthology.orchestra.run.smithi116.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-02-19T15:18:28.851 INFO:teuthology.orchestra.run.smithi116.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1451-g9f8908ee-1jammy [3,543 kB] 2024-02-19T15:18:28.851 INFO:teuthology.orchestra.run.smithi047.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1451-g9f8908ee-1jammy [3,543 kB] 2024-02-19T15:18:28.886 INFO:teuthology.orchestra.run.smithi047.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-02-19T15:18:28.887 INFO:teuthology.orchestra.run.smithi116.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-02-19T15:18:28.888 INFO:teuthology.orchestra.run.smithi047.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-02-19T15:18:28.891 INFO:teuthology.orchestra.run.smithi116.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-02-19T15:18:28.901 INFO:teuthology.orchestra.run.smithi047.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-02-19T15:18:28.904 INFO:teuthology.orchestra.run.smithi116.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-02-19T15:18:28.906 INFO:teuthology.orchestra.run.smithi047.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-02-19T15:18:28.906 INFO:teuthology.orchestra.run.smithi047.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-02-19T15:18:28.909 INFO:teuthology.orchestra.run.smithi116.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-02-19T15:18:28.909 INFO:teuthology.orchestra.run.smithi116.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-02-19T15:18:28.917 INFO:teuthology.orchestra.run.smithi047.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-02-19T15:18:28.919 INFO:teuthology.orchestra.run.smithi047.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-02-19T15:18:28.920 INFO:teuthology.orchestra.run.smithi116.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-02-19T15:18:28.923 INFO:teuthology.orchestra.run.smithi047.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-02-19T15:18:28.923 INFO:teuthology.orchestra.run.smithi116.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-02-19T15:18:28.924 INFO:teuthology.orchestra.run.smithi047.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-02-19T15:18:28.925 INFO:teuthology.orchestra.run.smithi116.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-02-19T15:18:28.926 INFO:teuthology.orchestra.run.smithi047.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-02-19T15:18:28.926 INFO:teuthology.orchestra.run.smithi116.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-02-19T15:18:28.927 INFO:teuthology.orchestra.run.smithi116.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-02-19T15:18:28.944 INFO:teuthology.orchestra.run.smithi047.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-02-19T15:18:28.946 INFO:teuthology.orchestra.run.smithi047.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-02-19T15:18:28.948 INFO:teuthology.orchestra.run.smithi047.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-02-19T15:18:28.948 INFO:teuthology.orchestra.run.smithi116.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-02-19T15:18:28.949 INFO:teuthology.orchestra.run.smithi047.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-02-19T15:18:28.949 INFO:teuthology.orchestra.run.smithi047.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-02-19T15:18:28.950 INFO:teuthology.orchestra.run.smithi047.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-02-19T15:18:28.950 INFO:teuthology.orchestra.run.smithi116.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-02-19T15:18:28.951 INFO:teuthology.orchestra.run.smithi047.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-02-19T15:18:28.951 INFO:teuthology.orchestra.run.smithi047.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-02-19T15:18:28.951 INFO:teuthology.orchestra.run.smithi047.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-02-19T15:18:28.952 INFO:teuthology.orchestra.run.smithi116.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-02-19T15:18:28.952 INFO:teuthology.orchestra.run.smithi116.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-02-19T15:18:28.953 INFO:teuthology.orchestra.run.smithi116.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-02-19T15:18:28.954 INFO:teuthology.orchestra.run.smithi116.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-02-19T15:18:28.954 INFO:teuthology.orchestra.run.smithi116.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-02-19T15:18:28.954 INFO:teuthology.orchestra.run.smithi116.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-02-19T15:18:28.955 INFO:teuthology.orchestra.run.smithi116.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-02-19T15:18:28.968 INFO:teuthology.orchestra.run.smithi047.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-02-19T15:18:28.970 INFO:teuthology.orchestra.run.smithi116.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-02-19T15:18:28.971 INFO:teuthology.orchestra.run.smithi047.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-02-19T15:18:28.971 INFO:teuthology.orchestra.run.smithi047.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-02-19T15:18:28.974 INFO:teuthology.orchestra.run.smithi047.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-02-19T15:18:28.974 INFO:teuthology.orchestra.run.smithi047.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-02-19T15:18:28.975 INFO:teuthology.orchestra.run.smithi047.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-02-19T15:18:28.975 INFO:teuthology.orchestra.run.smithi047.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-02-19T15:18:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-02-19T15:18:28.976 INFO:teuthology.orchestra.run.smithi116.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-02-19T15:18:28.976 INFO:teuthology.orchestra.run.smithi047.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-02-19T15:18:28.976 INFO:teuthology.orchestra.run.smithi047.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-02-19T15:18:28.978 INFO:teuthology.orchestra.run.smithi116.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-02-19T15:18:28.978 INFO:teuthology.orchestra.run.smithi116.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-02-19T15:18:28.979 INFO:teuthology.orchestra.run.smithi116.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-02-19T15:18:28.979 INFO:teuthology.orchestra.run.smithi116.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-02-19T15:18:28.980 INFO:teuthology.orchestra.run.smithi116.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-02-19T15:18:28.981 INFO:teuthology.orchestra.run.smithi116.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-02-19T15:18:28.988 INFO:teuthology.orchestra.run.smithi047.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-02-19T15:18:28.990 INFO:teuthology.orchestra.run.smithi116.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-02-19T15:18:28.997 INFO:teuthology.orchestra.run.smithi047.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-02-19T15:18:28.997 INFO:teuthology.orchestra.run.smithi047.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-02-19T15:18:28.998 INFO:teuthology.orchestra.run.smithi047.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-02-19T15:18:29.002 INFO:teuthology.orchestra.run.smithi047.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-02-19T15:18:29.002 INFO:teuthology.orchestra.run.smithi047.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-02-19T15:18:29.003 INFO:teuthology.orchestra.run.smithi047.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-02-19T15:18:29.003 INFO:teuthology.orchestra.run.smithi047.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-02-19T15:18:29.004 INFO:teuthology.orchestra.run.smithi116.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-02-19T15:18:29.005 INFO:teuthology.orchestra.run.smithi047.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-02-19T15:18:29.005 INFO:teuthology.orchestra.run.smithi116.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-02-19T15:18:29.006 INFO:teuthology.orchestra.run.smithi116.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-02-19T15:18:29.012 INFO:teuthology.orchestra.run.smithi116.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-02-19T15:18:29.012 INFO:teuthology.orchestra.run.smithi116.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-02-19T15:18:29.012 INFO:teuthology.orchestra.run.smithi116.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-02-19T15:18:29.013 INFO:teuthology.orchestra.run.smithi116.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-02-19T15:18:29.014 INFO:teuthology.orchestra.run.smithi116.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-02-19T15:18:29.015 INFO:teuthology.orchestra.run.smithi047.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-02-19T15:18:29.016 INFO:teuthology.orchestra.run.smithi116.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-02-19T15:18:29.026 INFO:teuthology.orchestra.run.smithi047.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-02-19T15:18:29.027 INFO:teuthology.orchestra.run.smithi047.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-02-19T15:18:29.028 INFO:teuthology.orchestra.run.smithi047.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-02-19T15:18:29.030 INFO:teuthology.orchestra.run.smithi047.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-02-19T15:18:29.030 INFO:teuthology.orchestra.run.smithi047.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-02-19T15:18:29.031 INFO:teuthology.orchestra.run.smithi047.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-02-19T15:18:29.033 INFO:teuthology.orchestra.run.smithi047.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-02-19T15:18:29.040 INFO:teuthology.orchestra.run.smithi116.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-02-19T15:18:29.042 INFO:teuthology.orchestra.run.smithi116.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-02-19T15:18:29.043 INFO:teuthology.orchestra.run.smithi116.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-02-19T15:18:29.046 INFO:teuthology.orchestra.run.smithi116.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-02-19T15:18:29.047 INFO:teuthology.orchestra.run.smithi116.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-02-19T15:18:29.048 INFO:teuthology.orchestra.run.smithi116.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-02-19T15:18:29.051 INFO:teuthology.orchestra.run.smithi116.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-02-19T15:18:29.195 INFO:teuthology.orchestra.run.smithi116.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-02-19T15:18:29.196 INFO:teuthology.orchestra.run.smithi116.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-02-19T15:18:29.198 INFO:teuthology.orchestra.run.smithi116.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-02-19T15:18:29.198 INFO:teuthology.orchestra.run.smithi116.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-02-19T15:18:29.199 INFO:teuthology.orchestra.run.smithi116.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-02-19T15:18:29.218 INFO:teuthology.orchestra.run.smithi116.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-02-19T15:18:29.220 INFO:teuthology.orchestra.run.smithi116.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-02-19T15:18:29.221 INFO:teuthology.orchestra.run.smithi116.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-02-19T15:18:29.231 INFO:teuthology.orchestra.run.smithi047.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-02-19T15:18:29.232 INFO:teuthology.orchestra.run.smithi047.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-02-19T15:18:29.233 INFO:teuthology.orchestra.run.smithi047.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-02-19T15:18:29.233 INFO:teuthology.orchestra.run.smithi047.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-02-19T15:18:29.234 INFO:teuthology.orchestra.run.smithi047.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-02-19T15:18:29.238 INFO:teuthology.orchestra.run.smithi116.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-02-19T15:18:29.238 INFO:teuthology.orchestra.run.smithi116.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-02-19T15:18:29.245 INFO:teuthology.orchestra.run.smithi116.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-02-19T15:18:29.246 INFO:teuthology.orchestra.run.smithi116.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-02-19T15:18:29.246 INFO:teuthology.orchestra.run.smithi116.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-02-19T15:18:29.247 INFO:teuthology.orchestra.run.smithi116.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-02-19T15:18:29.247 INFO:teuthology.orchestra.run.smithi116.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-02-19T15:18:29.248 INFO:teuthology.orchestra.run.smithi116.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-02-19T15:18:29.254 INFO:teuthology.orchestra.run.smithi047.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-02-19T15:18:29.256 INFO:teuthology.orchestra.run.smithi047.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-02-19T15:18:29.257 INFO:teuthology.orchestra.run.smithi047.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-02-19T15:18:29.261 INFO:teuthology.orchestra.run.smithi116.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-02-19T15:18:29.266 INFO:teuthology.orchestra.run.smithi116.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-02-19T15:18:29.269 INFO:teuthology.orchestra.run.smithi116.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-02-19T15:18:29.270 INFO:teuthology.orchestra.run.smithi116.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-02-19T15:18:29.273 INFO:teuthology.orchestra.run.smithi116.stdout:Get:74 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1451-g9f8908ee-1jammy [3,212 kB] 2024-02-19T15:18:29.275 INFO:teuthology.orchestra.run.smithi047.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-02-19T15:18:29.276 INFO:teuthology.orchestra.run.smithi047.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-02-19T15:18:29.280 INFO:teuthology.orchestra.run.smithi047.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-02-19T15:18:29.281 INFO:teuthology.orchestra.run.smithi047.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-02-19T15:18:29.281 INFO:teuthology.orchestra.run.smithi047.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-02-19T15:18:29.282 INFO:teuthology.orchestra.run.smithi047.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-02-19T15:18:29.283 INFO:teuthology.orchestra.run.smithi047.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-02-19T15:18:29.283 INFO:teuthology.orchestra.run.smithi047.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-02-19T15:18:29.287 INFO:teuthology.orchestra.run.smithi047.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-02-19T15:18:29.287 INFO:teuthology.orchestra.run.smithi116.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-02-19T15:18:29.291 INFO:teuthology.orchestra.run.smithi116.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-02-19T15:18:29.291 INFO:teuthology.orchestra.run.smithi116.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-02-19T15:18:29.292 INFO:teuthology.orchestra.run.smithi116.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-02-19T15:18:29.293 INFO:teuthology.orchestra.run.smithi116.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-02-19T15:18:29.297 INFO:teuthology.orchestra.run.smithi116.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-02-19T15:18:29.298 INFO:teuthology.orchestra.run.smithi116.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-02-19T15:18:29.300 INFO:teuthology.orchestra.run.smithi047.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-02-19T15:18:29.302 INFO:teuthology.orchestra.run.smithi047.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-02-19T15:18:29.302 INFO:teuthology.orchestra.run.smithi047.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-02-19T15:18:29.307 INFO:teuthology.orchestra.run.smithi116.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-02-19T15:18:29.307 INFO:teuthology.orchestra.run.smithi116.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-02-19T15:18:29.308 INFO:teuthology.orchestra.run.smithi116.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-02-19T15:18:29.320 INFO:teuthology.orchestra.run.smithi047.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-02-19T15:18:29.322 INFO:teuthology.orchestra.run.smithi047.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-02-19T15:18:29.323 INFO:teuthology.orchestra.run.smithi047.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-02-19T15:18:29.324 INFO:teuthology.orchestra.run.smithi047.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-02-19T15:18:29.324 INFO:teuthology.orchestra.run.smithi047.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-02-19T15:18:29.330 INFO:teuthology.orchestra.run.smithi047.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-02-19T15:18:29.330 INFO:teuthology.orchestra.run.smithi047.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-02-19T15:18:29.332 INFO:teuthology.orchestra.run.smithi116.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-02-19T15:18:29.334 INFO:teuthology.orchestra.run.smithi116.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-02-19T15:18:29.335 INFO:teuthology.orchestra.run.smithi116.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-02-19T15:18:29.342 INFO:teuthology.orchestra.run.smithi116.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-02-19T15:18:29.343 INFO:teuthology.orchestra.run.smithi116.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-02-19T15:18:29.343 INFO:teuthology.orchestra.run.smithi116.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-02-19T15:18:29.343 INFO:teuthology.orchestra.run.smithi047.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-02-19T15:18:29.344 INFO:teuthology.orchestra.run.smithi047.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-02-19T15:18:29.345 INFO:teuthology.orchestra.run.smithi047.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-02-19T15:18:29.346 INFO:teuthology.orchestra.run.smithi116.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-02-19T15:18:29.352 INFO:teuthology.orchestra.run.smithi116.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-02-19T15:18:29.364 INFO:teuthology.orchestra.run.smithi047.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-02-19T15:18:29.366 INFO:teuthology.orchestra.run.smithi047.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-02-19T15:18:29.367 INFO:teuthology.orchestra.run.smithi047.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-02-19T15:18:29.377 INFO:teuthology.orchestra.run.smithi047.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-02-19T15:18:29.377 INFO:teuthology.orchestra.run.smithi047.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-02-19T15:18:29.378 INFO:teuthology.orchestra.run.smithi047.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-02-19T15:18:29.380 INFO:teuthology.orchestra.run.smithi047.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-02-19T15:18:29.388 INFO:teuthology.orchestra.run.smithi047.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-02-19T15:18:29.434 INFO:teuthology.orchestra.run.smithi116.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1451-g9f8908ee-1jammy [710 kB] 2024-02-19T15:18:29.454 INFO:teuthology.orchestra.run.smithi047.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1451-g9f8908ee-1jammy [3,212 kB] 2024-02-19T15:18:29.468 INFO:teuthology.orchestra.run.smithi116.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1451-g9f8908ee-1jammy [356 kB] 2024-02-19T15:18:29.482 INFO:teuthology.orchestra.run.smithi116.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1451-g9f8908ee-1jammy [32.8 kB] 2024-02-19T15:18:29.494 INFO:teuthology.orchestra.run.smithi116.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1451-g9f8908ee-1jammy [180 kB] 2024-02-19T15:18:29.499 INFO:teuthology.orchestra.run.smithi116.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1451-g9f8908ee-1jammy [64.8 kB] 2024-02-19T15:18:29.502 INFO:teuthology.orchestra.run.smithi116.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1451-g9f8908ee-1jammy [332 kB] 2024-02-19T15:18:29.516 INFO:teuthology.orchestra.run.smithi116.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1451-g9f8908ee-1jammy [6,565 kB] 2024-02-19T15:18:29.746 INFO:teuthology.orchestra.run.smithi047.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1451-g9f8908ee-1jammy [710 kB] 2024-02-19T15:18:29.818 INFO:teuthology.orchestra.run.smithi047.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1451-g9f8908ee-1jammy [356 kB] 2024-02-19T15:18:29.829 INFO:teuthology.orchestra.run.smithi047.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1451-g9f8908ee-1jammy [32.8 kB] 2024-02-19T15:18:29.846 INFO:teuthology.orchestra.run.smithi116.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1451-g9f8908ee-1jammy [112 kB] 2024-02-19T15:18:29.848 INFO:teuthology.orchestra.run.smithi116.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1451-g9f8908ee-1jammy [465 kB] 2024-02-19T15:18:29.856 INFO:teuthology.orchestra.run.smithi047.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1451-g9f8908ee-1jammy [180 kB] 2024-02-19T15:18:29.861 INFO:teuthology.orchestra.run.smithi047.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1451-g9f8908ee-1jammy [64.8 kB] 2024-02-19T15:18:29.864 INFO:teuthology.orchestra.run.smithi047.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1451-g9f8908ee-1jammy [332 kB] 2024-02-19T15:18:29.876 INFO:teuthology.orchestra.run.smithi116.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1451-g9f8908ee-1jammy [24.8 MB] 2024-02-19T15:18:29.900 INFO:teuthology.orchestra.run.smithi047.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1451-g9f8908ee-1jammy [6,565 kB] 2024-02-19T15:18:30.511 INFO:teuthology.orchestra.run.smithi047.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1451-g9f8908ee-1jammy [112 kB] 2024-02-19T15:18:30.539 INFO:teuthology.orchestra.run.smithi047.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1451-g9f8908ee-1jammy [465 kB] 2024-02-19T15:18:30.581 INFO:teuthology.orchestra.run.smithi047.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1451-g9f8908ee-1jammy [24.8 MB] 2024-02-19T15:18:31.128 INFO:teuthology.orchestra.run.smithi116.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1451-g9f8908ee-1jammy [5,072 kB] 2024-02-19T15:18:31.381 INFO:teuthology.orchestra.run.smithi116.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1451-g9f8908ee-1jammy [244 kB] 2024-02-19T15:18:31.387 INFO:teuthology.orchestra.run.smithi116.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1451-g9f8908ee-1jammy [124 kB] 2024-02-19T15:18:31.391 INFO:teuthology.orchestra.run.smithi116.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1451-g9f8908ee-1jammy [1,522 kB] 2024-02-19T15:18:31.472 INFO:teuthology.orchestra.run.smithi116.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1451-g9f8908ee-1jammy [6,240 kB] 2024-02-19T15:18:31.789 INFO:teuthology.orchestra.run.smithi116.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1451-g9f8908ee-1jammy [22.7 MB] 2024-02-19T15:18:32.916 INFO:teuthology.orchestra.run.smithi047.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1451-g9f8908ee-1jammy [5,072 kB] 2024-02-19T15:18:32.933 INFO:teuthology.orchestra.run.smithi116.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1451-g9f8908ee-1jammy [14.0 kB] 2024-02-19T15:18:32.934 INFO:teuthology.orchestra.run.smithi116.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1451-g9f8908ee-1jammy [866 kB] 2024-02-19T15:18:32.972 INFO:teuthology.orchestra.run.smithi116.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1451-g9f8908ee-1jammy [2,374 kB] 2024-02-19T15:18:33.083 INFO:teuthology.orchestra.run.smithi116.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1451-g9f8908ee-1jammy [776 kB] 2024-02-19T15:18:33.126 INFO:teuthology.orchestra.run.smithi116.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1451-g9f8908ee-1jammy [144 kB] 2024-02-19T15:18:33.130 INFO:teuthology.orchestra.run.smithi116.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1451-g9f8908ee-1jammy [1,921 kB] 2024-02-19T15:18:33.227 INFO:teuthology.orchestra.run.smithi116.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1451-g9f8908ee-1jammy [8,625 kB] 2024-02-19T15:18:33.394 INFO:teuthology.orchestra.run.smithi047.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1451-g9f8908ee-1jammy [244 kB] 2024-02-19T15:18:33.431 INFO:teuthology.orchestra.run.smithi047.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1451-g9f8908ee-1jammy [124 kB] 2024-02-19T15:18:33.435 INFO:teuthology.orchestra.run.smithi047.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1451-g9f8908ee-1jammy [1,522 kB] 2024-02-19T15:18:33.581 INFO:teuthology.orchestra.run.smithi047.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1451-g9f8908ee-1jammy [6,240 kB] 2024-02-19T15:18:33.665 INFO:teuthology.orchestra.run.smithi116.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1451-g9f8908ee-1jammy [14.2 kB] 2024-02-19T15:18:33.665 INFO:teuthology.orchestra.run.smithi116.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1451-g9f8908ee-1jammy [49.9 MB] 2024-02-19T15:18:34.178 INFO:teuthology.orchestra.run.smithi047.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1451-g9f8908ee-1jammy [22.7 MB] 2024-02-19T15:18:36.268 INFO:teuthology.orchestra.run.smithi116.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1451-g9f8908ee-1jammy [121 kB] 2024-02-19T15:18:36.269 INFO:teuthology.orchestra.run.smithi116.stdout:Get:119 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-1451-g9f8908ee-1jammy [226 kB] 2024-02-19T15:18:36.276 INFO:teuthology.orchestra.run.smithi116.stdout:Get:120 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-1451-g9f8908ee-1jammy [34.0 kB] 2024-02-19T15:18:36.276 INFO:teuthology.orchestra.run.smithi116.stdout:Get:121 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-1451-g9f8908ee-1jammy [23.4 kB] 2024-02-19T15:18:36.277 INFO:teuthology.orchestra.run.smithi116.stdout:Get:122 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1451-g9f8908ee-1jammy [38.4 kB] 2024-02-19T15:18:36.278 INFO:teuthology.orchestra.run.smithi116.stdout:Get:123 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1451-g9f8908ee-1jammy [13.0 MB] 2024-02-19T15:18:36.467 INFO:teuthology.orchestra.run.smithi047.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1451-g9f8908ee-1jammy [14.0 kB] 2024-02-19T15:18:36.467 INFO:teuthology.orchestra.run.smithi047.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1451-g9f8908ee-1jammy [866 kB] 2024-02-19T15:18:36.554 INFO:teuthology.orchestra.run.smithi047.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1451-g9f8908ee-1jammy [2,374 kB] 2024-02-19T15:18:36.767 INFO:teuthology.orchestra.run.smithi047.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1451-g9f8908ee-1jammy [776 kB] 2024-02-19T15:18:36.859 INFO:teuthology.orchestra.run.smithi047.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1451-g9f8908ee-1jammy [144 kB] 2024-02-19T15:18:36.864 INFO:teuthology.orchestra.run.smithi047.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1451-g9f8908ee-1jammy [1,921 kB] 2024-02-19T15:18:36.931 INFO:teuthology.orchestra.run.smithi116.stdout:Get:124 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1451-g9f8908ee-1jammy [91.2 kB] 2024-02-19T15:18:37.016 INFO:teuthology.orchestra.run.smithi047.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1451-g9f8908ee-1jammy [8,625 kB] 2024-02-19T15:18:37.558 INFO:teuthology.orchestra.run.smithi116.stdout:Fetched 177 MB in 8s (21.3 MB/s) 2024-02-19T15:18:37.598 INFO:teuthology.orchestra.run.smithi047.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1451-g9f8908ee-1jammy [14.2 kB] 2024-02-19T15:18:37.598 INFO:teuthology.orchestra.run.smithi047.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1451-g9f8908ee-1jammy [49.9 MB] 2024-02-19T15:18:37.681 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pyperclip. 2024-02-19T15:18:37.748 INFO:teuthology.orchestra.run.smithi116.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132923 files and directories currently installed.) 2024-02-19T15:18:37.751 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-02-19T15:18:37.788 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-02-19T15:18:38.133 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-wcwidth. 2024-02-19T15:18:38.146 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-02-19T15:18:38.191 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-02-19T15:18:38.485 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-cmd2. 2024-02-19T15:18:38.499 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-02-19T15:18:38.569 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-02-19T15:18:38.896 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-02-19T15:18:38.910 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-02-19T15:18:38.997 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-02-19T15:18:39.367 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-02-19T15:18:39.382 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-02-19T15:18:39.425 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-02-19T15:18:39.703 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-02-19T15:18:39.718 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-02-19T15:18:39.761 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-02-19T15:18:40.223 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-02-19T15:18:40.238 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-19T15:18:40.290 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-19T15:18:40.718 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-02-19T15:18:40.733 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-19T15:18:40.776 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-19T15:18:41.084 INFO:teuthology.orchestra.run.smithi047.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1451-g9f8908ee-1jammy [121 kB] 2024-02-19T15:18:41.085 INFO:teuthology.orchestra.run.smithi047.stdout:Get:119 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-1451-g9f8908ee-1jammy [226 kB] 2024-02-19T15:18:41.087 INFO:teuthology.orchestra.run.smithi047.stdout:Get:120 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-1451-g9f8908ee-1jammy [34.0 kB] 2024-02-19T15:18:41.088 INFO:teuthology.orchestra.run.smithi047.stdout:Get:121 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-1451-g9f8908ee-1jammy [23.4 kB] 2024-02-19T15:18:41.093 INFO:teuthology.orchestra.run.smithi047.stdout:Get:122 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1451-g9f8908ee-1jammy [38.4 kB] 2024-02-19T15:18:41.093 INFO:teuthology.orchestra.run.smithi047.stdout:Get:123 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1451-g9f8908ee-1jammy [13.0 MB] 2024-02-19T15:18:41.279 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-02-19T15:18:41.294 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-19T15:18:41.362 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-19T15:18:41.674 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-02-19T15:18:41.688 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-02-19T15:18:41.732 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-02-19T15:18:41.852 INFO:teuthology.orchestra.run.smithi047.stdout:Get:124 https://4.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1451-g9f8908ee-1jammy [91.2 kB] 2024-02-19T15:18:42.068 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package librados2. 2024-02-19T15:18:42.083 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../010-librados2_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:42.126 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking librados2 (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:42.526 INFO:teuthology.orchestra.run.smithi047.stdout:Fetched 177 MB in 13s (13.4 MB/s) 2024-02-19T15:18:42.546 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package librbd1. 2024-02-19T15:18:42.561 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../011-librbd1_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:42.604 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking librbd1 (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:42.636 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pyperclip. 2024-02-19T15:18:42.706 INFO:teuthology.orchestra.run.smithi047.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132923 files and directories currently installed.) 2024-02-19T15:18:42.709 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-02-19T15:18:42.751 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-02-19T15:18:42.990 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libcephfs2. 2024-02-19T15:18:43.004 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:43.040 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libcephfs2 (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:43.079 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-wcwidth. 2024-02-19T15:18:43.093 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-02-19T15:18:43.137 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-02-19T15:18:43.301 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-rados. 2024-02-19T15:18:43.315 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../013-python3-rados_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:43.393 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-rados (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:43.448 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-cmd2. 2024-02-19T15:18:43.463 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-02-19T15:18:43.506 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-02-19T15:18:43.712 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-02-19T15:18:43.727 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-1451-g9f8908ee-1jammy_all.deb ... 2024-02-19T15:18:43.770 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-ceph-argparse (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:43.859 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-02-19T15:18:43.874 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-02-19T15:18:43.960 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-02-19T15:18:44.044 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-cephfs. 2024-02-19T15:18:44.059 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:44.106 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-cephfs (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:44.263 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-02-19T15:18:44.278 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-02-19T15:18:44.313 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-02-19T15:18:44.367 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-ceph-common. 2024-02-19T15:18:44.382 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-1451-g9f8908ee-1jammy_all.deb ... 2024-02-19T15:18:44.417 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-ceph-common (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:44.657 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-02-19T15:18:44.673 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-02-19T15:18:44.712 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-prettytable. 2024-02-19T15:18:44.716 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-02-19T15:18:44.727 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-02-19T15:18:44.778 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-02-19T15:18:45.039 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-rbd. 2024-02-19T15:18:45.054 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:45.097 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-rbd (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:45.153 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-02-19T15:18:45.168 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-19T15:18:45.219 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-19T15:18:45.459 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-02-19T15:18:45.474 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-02-19T15:18:45.517 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-02-19T15:18:45.572 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-02-19T15:18:45.587 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-19T15:18:45.622 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-19T15:18:45.778 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-02-19T15:18:45.793 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-02-19T15:18:45.827 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-02-19T15:18:45.975 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-02-19T15:18:45.991 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-19T15:18:46.034 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-19T15:18:46.139 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-02-19T15:18:46.153 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-02-19T15:18:46.197 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-02-19T15:18:46.328 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-02-19T15:18:46.343 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-02-19T15:18:46.378 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-02-19T15:18:46.526 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package lua5.1. 2024-02-19T15:18:46.540 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-02-19T15:18:46.584 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-02-19T15:18:46.697 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package librados2. 2024-02-19T15:18:46.712 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../010-librados2_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:46.755 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking librados2 (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:46.920 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package lua-any. 2024-02-19T15:18:46.935 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-02-19T15:18:46.978 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking lua-any (27ubuntu1) ... 2024-02-19T15:18:47.197 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package zip. 2024-02-19T15:18:47.212 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-02-19T15:18:47.255 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking zip (3.0-12build2) ... 2024-02-19T15:18:47.334 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package librbd1. 2024-02-19T15:18:47.349 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../011-librbd1_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:47.392 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking librbd1 (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:47.626 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package unzip. 2024-02-19T15:18:47.641 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-02-19T15:18:47.683 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-02-19T15:18:47.803 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libcephfs2. 2024-02-19T15:18:47.817 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:47.853 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libcephfs2 (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:48.029 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package luarocks. 2024-02-19T15:18:48.044 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-02-19T15:18:48.072 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-rados. 2024-02-19T15:18:48.079 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-02-19T15:18:48.086 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../013-python3-rados_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:48.121 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-rados (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:48.416 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-02-19T15:18:48.431 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-1451-g9f8908ee-1jammy_all.deb ... 2024-02-19T15:18:48.474 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-ceph-argparse (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:48.487 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package librgw2. 2024-02-19T15:18:48.501 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../027-librgw2_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:48.545 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking librgw2 (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:48.743 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-cephfs. 2024-02-19T15:18:48.759 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:48.802 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-cephfs (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:49.039 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-rgw. 2024-02-19T15:18:49.053 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:49.097 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-rgw (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:49.121 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-ceph-common. 2024-02-19T15:18:49.136 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-1451-g9f8908ee-1jammy_all.deb ... 2024-02-19T15:18:49.179 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-ceph-common (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:49.408 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package liboath0:amd64. 2024-02-19T15:18:49.422 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-02-19T15:18:49.467 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-02-19T15:18:49.515 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-prettytable. 2024-02-19T15:18:49.531 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-02-19T15:18:49.574 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-02-19T15:18:49.803 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libradosstriper1. 2024-02-19T15:18:49.818 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:49.843 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-rbd. 2024-02-19T15:18:49.859 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:49.861 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libradosstriper1 (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:49.901 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-rbd (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:50.271 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-02-19T15:18:50.286 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-02-19T15:18:50.289 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-common. 2024-02-19T15:18:50.304 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../031-ceph-common_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:50.329 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-02-19T15:18:50.347 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-common (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:50.598 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-02-19T15:18:50.613 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-02-19T15:18:50.656 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-02-19T15:18:50.976 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-02-19T15:18:50.992 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-02-19T15:18:51.034 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-02-19T15:18:51.312 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package lua5.1. 2024-02-19T15:18:51.328 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-02-19T15:18:51.362 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-02-19T15:18:51.653 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-base. 2024-02-19T15:18:51.668 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../032-ceph-base_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:51.698 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package lua-any. 2024-02-19T15:18:51.714 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-02-19T15:18:51.779 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-base (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:51.779 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking lua-any (27ubuntu1) ... 2024-02-19T15:18:52.001 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package zip. 2024-02-19T15:18:52.016 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-02-19T15:18:52.059 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking zip (3.0-12build2) ... 2024-02-19T15:18:52.240 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-02-19T15:18:52.254 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-02-19T15:18:52.298 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-02-19T15:18:52.354 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package unzip. 2024-02-19T15:18:52.370 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-02-19T15:18:52.412 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-02-19T15:18:52.601 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-cheroot. 2024-02-19T15:18:52.615 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-02-19T15:18:52.659 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-02-19T15:18:52.740 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package luarocks. 2024-02-19T15:18:52.756 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-02-19T15:18:52.790 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-02-19T15:18:52.979 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-02-19T15:18:52.993 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-02-19T15:18:53.062 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-02-19T15:18:53.215 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package librgw2. 2024-02-19T15:18:53.229 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../027-librgw2_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:53.273 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking librgw2 (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:53.365 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-jaraco.text. 2024-02-19T15:18:53.379 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-02-19T15:18:53.423 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-02-19T15:18:53.742 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-rgw. 2024-02-19T15:18:53.756 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:53.760 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-02-19T15:18:53.775 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-02-19T15:18:53.800 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-rgw (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:53.809 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-02-19T15:18:54.088 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package liboath0:amd64. 2024-02-19T15:18:54.096 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-tempora. 2024-02-19T15:18:54.102 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-02-19T15:18:54.111 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-02-19T15:18:54.146 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-02-19T15:18:54.154 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-02-19T15:18:54.465 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-portend. 2024-02-19T15:18:54.480 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-02-19T15:18:54.482 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libradosstriper1. 2024-02-19T15:18:54.497 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:54.523 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-02-19T15:18:54.541 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libradosstriper1 (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:54.826 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-02-19T15:18:54.841 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-02-19T15:18:54.884 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-02-19T15:18:54.960 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-common. 2024-02-19T15:18:54.976 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../031-ceph-common_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:55.018 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-common (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:55.187 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-cherrypy3. 2024-02-19T15:18:55.202 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-02-19T15:18:55.237 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-02-19T15:18:55.607 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-natsort. 2024-02-19T15:18:55.622 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-02-19T15:18:55.665 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-02-19T15:18:55.970 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-logutils. 2024-02-19T15:18:55.985 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-02-19T15:18:56.020 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-02-19T15:18:56.024 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-base. 2024-02-19T15:18:56.039 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../032-ceph-base_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:18:56.083 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-base (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:18:56.323 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-markupsafe. 2024-02-19T15:18:56.338 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-02-19T15:18:56.381 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-02-19T15:18:56.636 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-02-19T15:18:56.651 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-02-19T15:18:56.686 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-02-19T15:18:56.717 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-mako. 2024-02-19T15:18:56.733 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-02-19T15:18:56.775 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-02-19T15:18:56.964 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-cheroot. 2024-02-19T15:18:56.979 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-02-19T15:18:57.022 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-02-19T15:18:57.096 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-simplegeneric. 2024-02-19T15:18:57.111 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-02-19T15:18:57.153 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-02-19T15:18:57.481 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-singledispatch. 2024-02-19T15:18:57.497 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-02-19T15:18:57.508 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-02-19T15:18:57.524 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-02-19T15:18:57.531 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-02-19T15:18:57.566 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-02-19T15:18:57.792 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-webob. 2024-02-19T15:18:57.807 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-02-19T15:18:57.850 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-02-19T15:18:57.853 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-jaraco.text. 2024-02-19T15:18:57.868 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-02-19T15:18:57.903 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-02-19T15:18:58.197 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-02-19T15:18:58.212 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-02-19T15:18:58.237 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-waitress. 2024-02-19T15:18:58.252 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-02-19T15:18:58.255 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-02-19T15:18:58.298 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-02-19T15:18:58.542 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-tempora. 2024-02-19T15:18:58.557 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-02-19T15:18:58.599 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-02-19T15:18:58.631 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-tempita. 2024-02-19T15:18:58.646 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-02-19T15:18:58.689 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-02-19T15:18:58.920 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-portend. 2024-02-19T15:18:58.935 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-02-19T15:18:58.976 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-paste. 2024-02-19T15:18:58.977 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-02-19T15:18:58.991 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-02-19T15:18:59.033 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-02-19T15:18:59.247 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-02-19T15:18:59.262 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-02-19T15:18:59.305 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-02-19T15:18:59.371 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-02-19T15:18:59.386 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-02-19T15:18:59.428 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-02-19T15:18:59.625 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-cherrypy3. 2024-02-19T15:18:59.641 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-02-19T15:18:59.683 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-02-19T15:18:59.731 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pastedeploy. 2024-02-19T15:18:59.746 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-02-19T15:18:59.789 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-02-19T15:18:59.995 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-natsort. 2024-02-19T15:19:00.010 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-02-19T15:19:00.053 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-02-19T15:19:00.092 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-webtest. 2024-02-19T15:19:00.107 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-02-19T15:19:00.142 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-02-19T15:19:00.398 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-logutils. 2024-02-19T15:19:00.412 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-02-19T15:19:00.420 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pecan. 2024-02-19T15:19:00.435 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-02-19T15:19:00.456 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-02-19T15:19:00.478 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-02-19T15:19:00.742 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-markupsafe. 2024-02-19T15:19:00.756 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-02-19T15:19:00.800 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-02-19T15:19:00.833 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-certifi. 2024-02-19T15:19:00.847 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-02-19T15:19:00.891 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-02-19T15:19:01.086 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-mako. 2024-02-19T15:19:01.101 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-02-19T15:19:01.136 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-02-19T15:19:01.202 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-idna. 2024-02-19T15:19:01.217 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-02-19T15:19:01.260 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-idna (3.3-1) ... 2024-02-19T15:19:01.431 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-simplegeneric. 2024-02-19T15:19:01.445 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-02-19T15:19:01.481 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-02-19T15:19:02.397 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-urllib3. 2024-02-19T15:19:02.412 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-02-19T15:19:02.447 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-02-19T15:19:02.500 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-singledispatch. 2024-02-19T15:19:02.515 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-02-19T15:19:02.558 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-02-19T15:19:02.759 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-requests. 2024-02-19T15:19:02.773 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-02-19T15:19:02.808 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-02-19T15:19:02.869 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-webob. 2024-02-19T15:19:02.884 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-02-19T15:19:02.928 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-02-19T15:19:03.128 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-werkzeug. 2024-02-19T15:19:03.143 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-02-19T15:19:03.186 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-02-19T15:19:03.306 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-waitress. 2024-02-19T15:19:03.323 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-02-19T15:19:03.367 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-02-19T15:19:03.481 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-02-19T15:19:03.496 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-1451-g9f8908ee-1jammy_all.deb ... 2024-02-19T15:19:03.539 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:03.675 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-tempita. 2024-02-19T15:19:03.690 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-02-19T15:19:03.733 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-02-19T15:19:03.952 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-02-19T15:19:03.967 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:04.002 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:04.044 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-paste. 2024-02-19T15:19:04.060 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-02-19T15:19:04.102 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-02-19T15:19:04.296 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-bcrypt. 2024-02-19T15:19:04.312 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-02-19T15:19:04.363 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-02-19T15:19:04.414 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-02-19T15:19:04.429 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-02-19T15:19:04.497 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-02-19T15:19:04.666 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-openssl. 2024-02-19T15:19:04.681 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-02-19T15:19:04.724 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-02-19T15:19:04.767 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pastedeploy. 2024-02-19T15:19:04.782 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-02-19T15:19:04.817 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-02-19T15:19:05.077 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr. 2024-02-19T15:19:05.092 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:05.119 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-webtest. 2024-02-19T15:19:05.126 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:05.135 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-02-19T15:19:05.177 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-02-19T15:19:05.456 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pecan. 2024-02-19T15:19:05.471 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-02-19T15:19:05.497 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mon. 2024-02-19T15:19:05.505 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-02-19T15:19:05.512 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:05.555 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mon (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:05.868 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-certifi. 2024-02-19T15:19:05.883 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-02-19T15:19:05.935 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-02-19T15:19:06.125 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-osd. 2024-02-19T15:19:06.139 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:06.183 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-osd (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:06.238 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-idna. 2024-02-19T15:19:06.253 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-02-19T15:19:06.296 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-idna (3.3-1) ... 2024-02-19T15:19:06.599 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-urllib3. 2024-02-19T15:19:06.614 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-02-19T15:19:06.649 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-02-19T15:19:06.936 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph. 2024-02-19T15:19:06.950 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../068-ceph_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:06.993 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-requests. 2024-02-19T15:19:06.994 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:07.009 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-02-19T15:19:07.043 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-02-19T15:19:07.322 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-fuse. 2024-02-19T15:19:07.337 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:07.371 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-werkzeug. 2024-02-19T15:19:07.380 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-fuse (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:07.387 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-02-19T15:19:07.429 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-02-19T15:19:07.707 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-02-19T15:19:07.723 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-1451-g9f8908ee-1jammy_all.deb ... 2024-02-19T15:19:07.741 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mds. 2024-02-19T15:19:07.754 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:07.757 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:07.833 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mds (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:08.211 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package cephadm. 2024-02-19T15:19:08.224 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../071-cephadm_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:08.269 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking cephadm (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:08.270 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-02-19T15:19:08.285 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:08.328 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:08.572 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-asyncssh. 2024-02-19T15:19:08.585 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-02-19T15:19:08.621 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-02-19T15:19:08.623 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-bcrypt. 2024-02-19T15:19:08.638 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-02-19T15:19:08.672 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-02-19T15:19:08.933 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python-babel-localedata. 2024-02-19T15:19:08.948 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-02-19T15:19:08.967 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-openssl. 2024-02-19T15:19:08.983 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-02-19T15:19:08.991 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-02-19T15:19:09.025 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-02-19T15:19:09.361 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr. 2024-02-19T15:19:09.377 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:09.419 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:09.798 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mon. 2024-02-19T15:19:09.814 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:09.856 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mon (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:09.954 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-babel. 2024-02-19T15:19:09.967 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-02-19T15:19:10.004 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-02-19T15:19:10.290 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-jinja2. 2024-02-19T15:19:10.302 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-02-19T15:19:10.340 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-02-19T15:19:10.384 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-osd. 2024-02-19T15:19:10.394 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:10.426 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-osd (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:10.701 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-02-19T15:19:10.714 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-1451-g9f8908ee-1jammy_all.deb ... 2024-02-19T15:19:10.751 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:11.096 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-repoze.lru. 2024-02-19T15:19:11.109 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-02-19T15:19:11.146 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-02-19T15:19:11.154 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph. 2024-02-19T15:19:11.167 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../068-ceph_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:11.212 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:11.457 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-routes. 2024-02-19T15:19:11.470 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-02-19T15:19:11.515 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-02-19T15:19:11.523 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-fuse. 2024-02-19T15:19:11.535 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:11.573 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-fuse (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:11.951 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mds. 2024-02-19T15:19:11.963 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:12.000 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mds (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:12.060 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-02-19T15:19:12.074 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-1451-g9f8908ee-1jammy_all.deb ... 2024-02-19T15:19:12.126 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:12.328 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package cephadm. 2024-02-19T15:19:12.340 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../071-cephadm_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:12.370 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking cephadm (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:12.521 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-02-19T15:19:12.535 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-02-19T15:19:12.571 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-02-19T15:19:12.673 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-asyncssh. 2024-02-19T15:19:12.685 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-02-19T15:19:12.722 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-02-19T15:19:13.008 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python-babel-localedata. 2024-02-19T15:19:13.020 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-02-19T15:19:13.066 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-joblib. 2024-02-19T15:19:13.080 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-02-19T15:19:13.083 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-02-19T15:19:13.124 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-02-19T15:19:13.511 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-02-19T15:19:13.525 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-02-19T15:19:13.646 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-02-19T15:19:13.924 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-sklearn. 2024-02-19T15:19:13.939 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-02-19T15:19:13.982 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-02-19T15:19:14.088 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-babel. 2024-02-19T15:19:14.102 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-02-19T15:19:14.146 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-02-19T15:19:14.495 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-jinja2. 2024-02-19T15:19:14.509 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-02-19T15:19:14.549 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-02-19T15:19:14.600 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-02-19T15:19:14.613 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-1451-g9f8908ee-1jammy_all.deb ... 2024-02-19T15:19:14.650 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:14.894 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-02-19T15:19:14.906 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-1451-g9f8908ee-1jammy_all.deb ... 2024-02-19T15:19:14.943 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:15.272 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-repoze.lru. 2024-02-19T15:19:15.285 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-02-19T15:19:15.346 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-02-19T15:19:15.608 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-routes. 2024-02-19T15:19:15.621 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-02-19T15:19:15.628 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-cachetools. 2024-02-19T15:19:15.642 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-02-19T15:19:15.657 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-02-19T15:19:15.685 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-02-19T15:19:15.972 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pyasn1. 2024-02-19T15:19:15.986 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-02-19T15:19:15.994 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-02-19T15:19:16.008 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-1451-g9f8908ee-1jammy_all.deb ... 2024-02-19T15:19:16.030 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-02-19T15:19:16.043 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:16.383 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-02-19T15:19:16.398 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-02-19T15:19:16.441 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-02-19T15:19:16.472 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-02-19T15:19:16.485 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-02-19T15:19:16.522 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-02-19T15:19:16.770 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-rsa. 2024-02-19T15:19:16.784 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-02-19T15:19:16.827 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-rsa (4.8-1) ... 2024-02-19T15:19:17.025 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-joblib. 2024-02-19T15:19:17.039 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-02-19T15:19:17.083 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-02-19T15:19:17.115 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-google-auth. 2024-02-19T15:19:17.129 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-02-19T15:19:17.164 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-02-19T15:19:17.478 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-02-19T15:19:17.484 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-02-19T15:19:17.491 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-02-19T15:19:17.495 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-02-19T15:19:17.534 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-02-19T15:19:17.614 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-02-19T15:19:17.854 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-websocket. 2024-02-19T15:19:17.863 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-02-19T15:19:17.896 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-02-19T15:19:17.916 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-sklearn. 2024-02-19T15:19:17.930 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-02-19T15:19:17.974 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-02-19T15:19:18.199 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-kubernetes. 2024-02-19T15:19:18.210 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-02-19T15:19:18.248 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-02-19T15:19:18.858 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-02-19T15:19:18.874 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-1451-g9f8908ee-1jammy_all.deb ... 2024-02-19T15:19:18.916 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:18.946 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-02-19T15:19:18.954 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-1451-g9f8908ee-1jammy_all.deb ... 2024-02-19T15:19:18.995 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:19.315 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libonig5:amd64. 2024-02-19T15:19:19.324 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-02-19T15:19:19.365 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-02-19T15:19:19.709 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libjq1:amd64. 2024-02-19T15:19:19.719 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-02-19T15:19:19.759 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-02-19T15:19:19.794 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-cachetools. 2024-02-19T15:19:19.809 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-02-19T15:19:19.844 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-02-19T15:19:20.012 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package jq. 2024-02-19T15:19:20.022 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-02-19T15:19:20.070 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-02-19T15:19:20.106 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pyasn1. 2024-02-19T15:19:20.120 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-02-19T15:19:20.189 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-02-19T15:19:20.314 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package socat. 2024-02-19T15:19:20.325 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-02-19T15:19:20.364 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-02-19T15:19:20.509 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-02-19T15:19:20.524 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-02-19T15:19:20.567 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-02-19T15:19:20.642 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package xmlstarlet. 2024-02-19T15:19:20.652 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-02-19T15:19:20.692 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-02-19T15:19:20.861 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-rsa. 2024-02-19T15:19:20.875 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-02-19T15:19:20.919 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-rsa (4.8-1) ... 2024-02-19T15:19:21.046 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-test. 2024-02-19T15:19:21.056 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../099-ceph-test_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:21.096 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-test (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:21.240 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-google-auth. 2024-02-19T15:19:21.253 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-02-19T15:19:21.298 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-02-19T15:19:21.626 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-02-19T15:19:21.640 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-02-19T15:19:21.684 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-02-19T15:19:21.979 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-websocket. 2024-02-19T15:19:21.993 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-02-19T15:19:22.028 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-02-19T15:19:22.324 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-kubernetes. 2024-02-19T15:19:22.338 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-02-19T15:19:22.381 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-02-19T15:19:23.175 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-volume. 2024-02-19T15:19:23.185 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-1451-g9f8908ee-1jammy_all.deb ... 2024-02-19T15:19:23.203 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-02-19T15:19:23.216 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-1451-g9f8908ee-1jammy_all.deb ... 2024-02-19T15:19:23.225 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-volume (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:23.253 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:23.588 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package cephfs-mirror. 2024-02-19T15:19:23.597 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:23.598 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libonig5:amd64. 2024-02-19T15:19:23.611 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-02-19T15:19:23.637 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking cephfs-mirror (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:23.656 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-02-19T15:19:23.976 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libjq1:amd64. 2024-02-19T15:19:23.982 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-colorama. 2024-02-19T15:19:23.989 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-02-19T15:19:23.992 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-02-19T15:19:24.031 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-02-19T15:19:24.034 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-02-19T15:19:24.261 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package jq. 2024-02-19T15:19:24.278 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-02-19T15:19:24.292 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package cephfs-shell. 2024-02-19T15:19:24.302 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-1451-g9f8908ee-1jammy_all.deb ... 2024-02-19T15:19:24.319 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-02-19T15:19:24.342 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking cephfs-shell (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:24.597 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package socat. 2024-02-19T15:19:24.611 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package cephfs-top. 2024-02-19T15:19:24.612 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-02-19T15:19:24.621 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-1451-g9f8908ee-1jammy_all.deb ... 2024-02-19T15:19:24.653 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking cephfs-top (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:24.655 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-02-19T15:19:24.897 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libcephfs-dev. 2024-02-19T15:19:24.908 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:24.947 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libcephfs-dev (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:24.967 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package xmlstarlet. 2024-02-19T15:19:24.989 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-02-19T15:19:25.033 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-02-19T15:19:25.292 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package lua-socket:amd64. 2024-02-19T15:19:25.301 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-02-19T15:19:25.341 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-02-19T15:19:25.395 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-test. 2024-02-19T15:19:25.409 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../099-ceph-test_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:25.453 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-test (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:25.679 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package lua-sec:amd64. 2024-02-19T15:19:25.689 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-02-19T15:19:25.729 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-02-19T15:19:26.083 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package nvme-cli. 2024-02-19T15:19:26.096 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-02-19T15:19:26.132 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-02-19T15:19:26.529 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package pkg-config. 2024-02-19T15:19:26.539 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-02-19T15:19:26.587 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-02-19T15:19:26.849 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-02-19T15:19:26.859 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-02-19T15:19:26.898 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-02-19T15:19:27.333 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-volume. 2024-02-19T15:19:27.349 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-1451-g9f8908ee-1jammy_all.deb ... 2024-02-19T15:19:27.391 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-volume (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:27.543 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-iniconfig. 2024-02-19T15:19:27.553 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-02-19T15:19:27.584 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-02-19T15:19:27.762 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package cephfs-mirror. 2024-02-19T15:19:27.777 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:27.820 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking cephfs-mirror (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:27.920 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pastescript. 2024-02-19T15:19:27.930 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-02-19T15:19:27.970 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-02-19T15:19:28.131 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-colorama. 2024-02-19T15:19:28.147 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-02-19T15:19:28.189 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-02-19T15:19:28.299 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pluggy. 2024-02-19T15:19:28.309 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-02-19T15:19:28.349 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-02-19T15:19:28.450 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package cephfs-shell. 2024-02-19T15:19:28.465 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-1451-g9f8908ee-1jammy_all.deb ... 2024-02-19T15:19:28.585 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking cephfs-shell (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:28.693 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-psutil. 2024-02-19T15:19:28.703 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-02-19T15:19:28.742 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-02-19T15:19:28.978 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package cephfs-top. 2024-02-19T15:19:28.993 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-1451-g9f8908ee-1jammy_all.deb ... 2024-02-19T15:19:29.027 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking cephfs-top (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:29.079 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-py. 2024-02-19T15:19:29.089 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-02-19T15:19:29.137 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-py (1.10.0-1) ... 2024-02-19T15:19:29.255 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libcephfs-dev. 2024-02-19T15:19:29.270 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:29.313 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libcephfs-dev (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:29.482 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pygments. 2024-02-19T15:19:29.492 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-02-19T15:19:29.532 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-02-19T15:19:29.624 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package lua-socket:amd64. 2024-02-19T15:19:29.639 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-02-19T15:19:29.682 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-02-19T15:19:29.994 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pyinotify. 2024-02-19T15:19:30.003 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-02-19T15:19:30.011 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package lua-sec:amd64. 2024-02-19T15:19:30.026 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-02-19T15:19:30.044 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-02-19T15:19:30.069 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-02-19T15:19:30.339 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-toml. 2024-02-19T15:19:30.348 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-02-19T15:19:30.388 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-02-19T15:19:30.465 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package nvme-cli. 2024-02-19T15:19:30.480 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-02-19T15:19:30.515 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-02-19T15:19:30.724 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pytest. 2024-02-19T15:19:30.735 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-02-19T15:19:30.774 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-02-19T15:19:30.861 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package pkg-config. 2024-02-19T15:19:30.876 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-02-19T15:19:30.919 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-02-19T15:19:31.061 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-simplejson. 2024-02-19T15:19:31.071 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-02-19T15:19:31.111 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-02-19T15:19:31.164 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-02-19T15:19:31.178 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-02-19T15:19:31.222 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-02-19T15:19:31.405 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-02-19T15:19:31.416 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-02-19T15:19:31.455 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-02-19T15:19:31.675 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-iniconfig. 2024-02-19T15:19:31.690 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-02-19T15:19:31.733 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-02-19T15:19:32.036 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pastescript. 2024-02-19T15:19:32.052 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-02-19T15:19:32.094 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-02-19T15:19:32.184 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package radosgw. 2024-02-19T15:19:32.193 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../122-radosgw_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:32.234 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking radosgw (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:32.400 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pluggy. 2024-02-19T15:19:32.415 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-02-19T15:19:32.458 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-02-19T15:19:32.736 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-psutil. 2024-02-19T15:19:32.751 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-02-19T15:19:32.793 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-02-19T15:19:32.861 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package rbd-fuse. 2024-02-19T15:19:32.872 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:32.911 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking rbd-fuse (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:33.105 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-py. 2024-02-19T15:19:33.120 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-02-19T15:19:33.154 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-py (1.10.0-1) ... 2024-02-19T15:19:33.193 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-02-19T15:19:33.433 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pygments. 2024-02-19T15:19:33.448 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-02-19T15:19:33.475 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-02-19T15:19:33.482 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-02-19T15:19:33.601 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-02-19T15:19:33.864 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-02-19T15:19:33.928 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pyinotify. 2024-02-19T15:19:33.943 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-02-19T15:19:33.986 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-02-19T15:19:34.270 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-02-19T15:19:34.297 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-toml. 2024-02-19T15:19:34.312 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-02-19T15:19:34.372 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-02-19T15:19:34.692 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pytest. 2024-02-19T15:19:34.707 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-02-19T15:19:34.741 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-02-19T15:19:34.919 INFO:teuthology.orchestra.run.smithi116.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-02-19T15:19:34.971 INFO:teuthology.orchestra.run.smithi116.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2024-02-19T15:19:35.028 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up cephadm (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:35.078 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-simplejson. 2024-02-19T15:19:35.094 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-02-19T15:19:35.136 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-02-19T15:19:35.414 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-02-19T15:19:35.430 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-02-19T15:19:35.472 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-02-19T15:19:35.851 INFO:teuthology.orchestra.run.smithi116.stdout:Adding system user cephadm....done 2024-02-19T15:19:36.011 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-02-19T15:19:36.218 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package radosgw. 2024-02-19T15:19:36.232 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../122-radosgw_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:36.267 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking radosgw (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:36.304 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-02-19T15:19:36.564 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-02-19T15:19:36.673 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-02-19T15:19:36.854 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package rbd-fuse. 2024-02-19T15:19:36.868 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-1451-g9f8908ee-1jammy_amd64.deb ... 2024-02-19T15:19:36.911 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking rbd-fuse (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:36.957 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-02-19T15:19:37.197 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-02-19T15:19:37.225 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-02-19T15:19:37.351 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-py (1.10.0-1) ... 2024-02-19T15:19:37.500 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-02-19T15:19:37.609 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-02-19T15:19:37.678 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-02-19T15:19:37.869 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-02-19T15:19:38.055 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-02-19T15:19:38.266 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-02-19T15:19:38.315 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-02-19T15:19:38.607 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-02-19T15:19:38.866 INFO:teuthology.orchestra.run.smithi047.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-02-19T15:19:38.883 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-02-19T15:19:38.908 INFO:teuthology.orchestra.run.smithi047.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2024-02-19T15:19:38.961 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up cephadm (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:39.235 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-02-19T15:19:39.494 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-02-19T15:19:39.530 INFO:teuthology.orchestra.run.smithi047.stdout:Adding system user cephadm....done 2024-02-19T15:19:39.675 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-02-19T15:19:39.787 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-02-19T15:19:39.951 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-02-19T15:19:40.047 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-ceph-argparse (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:40.219 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-02-19T15:19:40.306 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-02-19T15:19:40.345 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-02-19T15:19:40.424 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-02-19T15:19:40.575 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-02-19T15:19:40.613 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-02-19T15:19:40.834 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-02-19T15:19:40.880 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-02-19T15:19:40.977 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-02-19T15:19:40.998 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-py (1.10.0-1) ... 2024-02-19T15:19:41.236 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-02-19T15:19:41.308 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-02-19T15:19:41.614 INFO:teuthology.orchestra.run.smithi116.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-02-19T15:19:41.641 INFO:teuthology.orchestra.run.smithi116.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-02-19T15:19:41.685 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-02-19T15:19:41.689 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-02-19T15:19:41.832 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-02-19T15:19:41.953 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-02-19T15:19:42.200 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-02-19T15:19:42.220 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-02-19T15:19:42.326 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-02-19T15:19:42.514 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-02-19T15:19:42.807 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-02-19T15:19:42.843 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-02-19T15:19:43.049 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-02-19T15:19:43.119 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-02-19T15:19:43.434 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-02-19T15:19:43.454 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up zip (3.0-12build2) ... 2024-02-19T15:19:43.572 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-02-19T15:19:43.702 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-ceph-argparse (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:43.969 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-02-19T15:19:44.095 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-02-19T15:19:44.221 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-02-19T15:19:44.257 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-02-19T15:19:44.497 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-02-19T15:19:44.525 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-02-19T15:19:44.614 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-02-19T15:19:44.675 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-02-19T15:19:44.802 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-02-19T15:19:44.882 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-02-19T15:19:45.128 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-02-19T15:19:45.343 INFO:teuthology.orchestra.run.smithi047.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-02-19T15:19:45.388 INFO:teuthology.orchestra.run.smithi047.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-02-19T15:19:45.452 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-02-19T15:19:45.529 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-02-19T15:19:45.578 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-02-19T15:19:45.780 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-02-19T15:19:45.962 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-02-19T15:19:46.088 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-02-19T15:19:46.148 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-02-19T15:19:46.365 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-02-19T15:19:46.458 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-02-19T15:19:46.641 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-02-19T15:19:46.776 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-02-19T15:19:46.933 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up zip (3.0-12build2) ... 2024-02-19T15:19:47.043 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-02-19T15:19:47.127 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-02-19T15:19:47.395 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-02-19T15:19:47.521 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-ceph-common (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:47.761 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-02-19T15:19:47.822 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-idna (3.3-1) ... 2024-02-19T15:19:48.037 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-02-19T15:19:48.148 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-02-19T15:19:48.163 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-02-19T15:19:48.289 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-02-19T15:19:48.615 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-02-19T15:19:49.017 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-02-19T15:19:49.293 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-02-19T15:19:49.379 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-02-19T15:19:49.614 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-19T15:19:49.686 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-02-19T15:19:49.756 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-toml (0.10.2-1) ... 2024-02-19T15:19:49.995 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-02-19T15:19:50.033 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-02-19T15:19:50.158 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-02-19T15:19:50.284 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-02-19T15:19:50.355 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-02-19T15:19:50.586 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-02-19T15:19:50.723 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-02-19T15:19:50.878 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-02-19T15:19:50.982 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-02-19T15:19:51.100 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-ceph-common (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:51.138 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-19T15:19:51.256 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-02-19T15:19:51.409 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-idna (3.3-1) ... 2024-02-19T15:19:51.557 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-02-19T15:19:51.786 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-02-19T15:19:51.817 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-02-19T15:19:52.093 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-02-19T15:19:52.360 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-02-19T15:19:52.620 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-02-19T15:19:52.921 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-02-19T15:19:53.047 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-02-19T15:19:53.058 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-02-19T15:19:53.268 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-19T15:19:53.323 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-02-19T15:19:53.402 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-toml (0.10.2-1) ... 2024-02-19T15:19:53.474 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-02-19T15:19:53.678 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-02-19T15:19:53.767 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-02-19T15:19:53.804 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-02-19T15:19:53.930 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-02-19T15:19:54.110 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-02-19T15:19:54.239 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-02-19T15:19:54.377 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-02-19T15:19:54.503 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-02-19T15:19:54.532 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-02-19T15:19:54.629 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-02-19T15:19:54.718 INFO:teuthology.orchestra.run.smithi116.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-02-19T15:19:54.783 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-19T15:19:54.901 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-02-19T15:19:54.964 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-02-19T15:19:55.107 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-02-19T15:19:55.219 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-02-19T15:19:55.487 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-02-19T15:19:55.508 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-02-19T15:19:55.776 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up lua-any (27ubuntu1) ... 2024-02-19T15:19:55.788 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-02-19T15:19:55.902 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-portend (3.0.0-1) ... 2024-02-19T15:19:56.064 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-02-19T15:19:56.187 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-19T15:19:56.295 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-02-19T15:19:56.331 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-02-19T15:19:56.599 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-02-19T15:19:56.613 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-02-19T15:19:56.700 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-02-19T15:19:56.915 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-02-19T15:19:56.984 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-02-19T15:19:57.110 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-02-19T15:19:57.191 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-02-19T15:19:57.402 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-rsa (4.8-1) ... 2024-02-19T15:19:57.445 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-02-19T15:19:57.679 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-02-19T15:19:57.763 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-02-19T15:19:57.954 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-02-19T15:19:58.039 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-02-19T15:19:58.156 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-02-19T15:19:58.239 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-02-19T15:19:58.257 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-02-19T15:19:58.346 INFO:teuthology.orchestra.run.smithi047.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-02-19T15:19:58.592 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-02-19T15:19:58.608 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-02-19T15:19:58.752 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-02-19T15:19:58.939 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-02-19T15:19:59.153 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-02-19T15:19:59.295 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-02-19T15:19:59.404 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up librados2 (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:59.429 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up lua-any (27ubuntu1) ... 2024-02-19T15:19:59.505 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:19:59.563 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-portend (3.0.0-1) ... 2024-02-19T15:19:59.656 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-02-19T15:19:59.831 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-19T15:19:59.948 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-02-19T15:19:59.992 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libcephfs2 (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:00.119 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libradosstriper1 (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:00.245 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-02-19T15:20:00.266 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-02-19T15:20:00.580 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up librbd1 (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:00.585 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-02-19T15:20:00.697 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr-modules-core (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:00.807 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-fuse (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:00.894 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-02-19T15:20:01.017 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-19T15:20:01.017 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-19T15:20:01.105 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-rsa (4.8-1) ... 2024-02-19T15:20:01.381 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-02-19T15:20:01.649 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-02-19T15:20:01.650 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libcephfs-dev (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:01.776 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-rados (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:01.901 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up librgw2 (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:01.917 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-02-19T15:20:02.027 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up cephfs-top (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:02.153 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-02-19T15:20:02.310 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-02-19T15:20:02.612 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-02-19T15:20:02.981 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-02-19T15:20:03.098 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up librados2 (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:03.199 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:03.326 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-02-19T15:20:03.415 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-rbd (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:03.547 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up rbd-fuse (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:03.662 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libcephfs2 (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:03.700 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-rgw (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:03.779 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libradosstriper1 (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:03.827 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-cephfs (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:03.880 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-02-19T15:20:03.952 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up cephfs-shell (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:04.079 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-common (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:04.156 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up librbd1 (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:04.282 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-modules-core (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:04.442 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-fuse (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:04.640 INFO:teuthology.orchestra.run.smithi116.stdout:Adding group ceph....done 2024-02-19T15:20:04.659 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-19T15:20:04.660 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-19T15:20:04.991 INFO:teuthology.orchestra.run.smithi116.stdout:Adding system user ceph....done 2024-02-19T15:20:05.160 INFO:teuthology.orchestra.run.smithi116.stdout:Setting system user ceph properties....done 2024-02-19T15:20:05.175 INFO:teuthology.orchestra.run.smithi116.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-02-19T15:20:05.276 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libcephfs-dev (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:05.318 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-02-19T15:20:05.402 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-rados (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:05.512 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up librgw2 (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:05.613 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up cephfs-top (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:05.703 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-02-19T15:20:05.714 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-02-19T15:20:06.388 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up cephfs-mirror (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:06.613 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-02-19T15:20:06.613 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-02-19T15:20:07.059 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-rbd (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:07.185 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up rbd-fuse (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:07.197 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-test (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:07.324 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up radosgw (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:07.336 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-rgw (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:07.462 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-cephfs (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:07.588 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up cephfs-shell (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:07.714 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-common (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:07.833 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-19T15:20:07.833 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-19T15:20:08.242 INFO:teuthology.orchestra.run.smithi047.stdout:Adding group ceph....done 2024-02-19T15:20:08.451 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-base (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:08.668 INFO:teuthology.orchestra.run.smithi047.stdout:Adding system user ceph....done 2024-02-19T15:20:08.837 INFO:teuthology.orchestra.run.smithi047.stdout:Setting system user ceph properties....done 2024-02-19T15:20:08.842 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-02-19T15:20:08.854 INFO:teuthology.orchestra.run.smithi047.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-02-19T15:20:09.005 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-02-19T15:20:09.410 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-02-19T15:20:09.477 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mds (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:09.701 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-19T15:20:09.701 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-19T15:20:10.082 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up cephfs-mirror (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:10.298 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-02-19T15:20:10.298 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-02-19T15:20:10.312 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:10.533 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-19T15:20:10.533 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-19T15:20:10.891 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-test (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:11.025 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up radosgw (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:11.114 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-osd (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:11.420 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-19T15:20:11.420 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-19T15:20:11.546 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-19T15:20:11.546 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-19T15:20:12.024 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:12.150 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:12.169 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-base (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:12.292 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mon (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:12.501 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-19T15:20:12.501 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-19T15:20:12.514 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-02-19T15:20:13.094 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr-cephadm (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:13.162 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mds (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:13.236 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:13.399 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-19T15:20:13.400 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-19T15:20:13.655 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr-dashboard (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:13.798 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-volume (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:14.039 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:14.142 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-02-19T15:20:14.250 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-19T15:20:14.250 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-19T15:20:14.251 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-02-19T15:20:14.403 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-02-19T15:20:14.916 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-osd (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:15.253 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-19T15:20:15.253 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-19T15:20:15.512 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-02-19T15:20:15.859 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:15.985 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:16.127 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mon (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:16.247 INFO:teuthology.orchestra.run.smithi116.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-02-19T15:20:16.252 DEBUG:teuthology.orchestra.run.smithi116:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-02-19T15:20:16.324 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-02-19T15:20:16.345 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-19T15:20:16.345 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-19T15:20:16.490 INFO:teuthology.orchestra.run.smithi116.stdout:Building dependency tree... 2024-02-19T15:20:16.491 INFO:teuthology.orchestra.run.smithi116.stdout:Reading state information... 2024-02-19T15:20:16.647 INFO:teuthology.orchestra.run.smithi116.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-02-19T15:20:16.647 INFO:teuthology.orchestra.run.smithi116.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-02-19T15:20:16.647 INFO:teuthology.orchestra.run.smithi116.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2024-02-19T15:20:16.647 INFO:teuthology.orchestra.run.smithi116.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2024-02-19T15:20:16.647 INFO:teuthology.orchestra.run.smithi116.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-02-19T15:20:16.647 INFO:teuthology.orchestra.run.smithi116.stdout:The following packages were automatically installed and are no longer required: 2024-02-19T15:20:16.647 INFO:teuthology.orchestra.run.smithi116.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-19T15:20:16.647 INFO:teuthology.orchestra.run.smithi116.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-19T15:20:16.647 INFO:teuthology.orchestra.run.smithi116.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-19T15:20:16.648 INFO:teuthology.orchestra.run.smithi116.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-02-19T15:20:16.648 INFO:teuthology.orchestra.run.smithi116.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-02-19T15:20:16.648 INFO:teuthology.orchestra.run.smithi116.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-19T15:20:16.649 INFO:teuthology.orchestra.run.smithi116.stdout:The following additional packages will be installed: 2024-02-19T15:20:16.649 INFO:teuthology.orchestra.run.smithi116.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-02-19T15:20:16.649 INFO:teuthology.orchestra.run.smithi116.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-02-19T15:20:16.649 INFO:teuthology.orchestra.run.smithi116.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2024-02-19T15:20:16.649 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-02-19T15:20:16.649 INFO:teuthology.orchestra.run.smithi116.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-02-19T15:20:16.649 INFO:teuthology.orchestra.run.smithi116.stdout:Suggested packages: 2024-02-19T15:20:16.649 INFO:teuthology.orchestra.run.smithi116.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-02-19T15:20:16.650 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2024-02-19T15:20:16.650 INFO:teuthology.orchestra.run.smithi116.stdout: libengine-pkcs11-openssl 2024-02-19T15:20:16.698 INFO:teuthology.orchestra.run.smithi116.stdout:The following NEW packages will be installed: 2024-02-19T15:20:16.699 INFO:teuthology.orchestra.run.smithi116.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-02-19T15:20:16.699 INFO:teuthology.orchestra.run.smithi116.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-02-19T15:20:16.700 INFO:teuthology.orchestra.run.smithi116.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-02-19T15:20:16.700 INFO:teuthology.orchestra.run.smithi116.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-02-19T15:20:16.700 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-02-19T15:20:16.700 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-02-19T15:20:16.700 INFO:teuthology.orchestra.run.smithi116.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-02-19T15:20:16.701 INFO:teuthology.orchestra.run.smithi116.stdout:The following packages will be upgraded: 2024-02-19T15:20:16.701 INFO:teuthology.orchestra.run.smithi116.stdout: libssl3 2024-02-19T15:20:16.904 INFO:teuthology.orchestra.run.smithi116.stdout:1 upgraded, 39 newly installed, 0 to remove and 276 not upgraded. 2024-02-19T15:20:16.904 INFO:teuthology.orchestra.run.smithi116.stdout:Need to get 54.6 MB of archives. 2024-02-19T15:20:16.904 INFO:teuthology.orchestra.run.smithi116.stdout:After this operation, 202 MB of additional disk space will be used. 2024-02-19T15:20:16.904 INFO:teuthology.orchestra.run.smithi116.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-02-19T15:20:16.962 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-cephadm (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:17.105 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:17.255 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-dashboard (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:17.390 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-volume (19.0.0-1451-g9f8908ee-1jammy) ... 2024-02-19T15:20:17.396 INFO:teuthology.orchestra.run.smithi116.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.14 [1,902 kB] 2024-02-19T15:20:17.653 INFO:teuthology.orchestra.run.smithi116.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-02-19T15:20:17.687 INFO:teuthology.orchestra.run.smithi116.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-02-19T15:20:17.689 INFO:teuthology.orchestra.run.smithi116.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-02-19T15:20:17.694 INFO:teuthology.orchestra.run.smithi116.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-02-19T15:20:17.697 INFO:teuthology.orchestra.run.smithi116.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-02-19T15:20:17.698 INFO:teuthology.orchestra.run.smithi116.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-02-19T15:20:17.701 INFO:teuthology.orchestra.run.smithi116.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-02-19T15:20:17.703 INFO:teuthology.orchestra.run.smithi116.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-02-19T15:20:17.704 INFO:teuthology.orchestra.run.smithi116.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-02-19T15:20:17.759 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-02-19T15:20:17.860 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-02-19T15:20:17.868 INFO:teuthology.orchestra.run.smithi116.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-02-19T15:20:17.995 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-02-19T15:20:18.203 INFO:teuthology.orchestra.run.smithi116.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-02-19T15:20:19.238 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-02-19T15:20:19.273 INFO:teuthology.orchestra.run.smithi116.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-02-19T15:20:19.275 INFO:teuthology.orchestra.run.smithi116.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-02-19T15:20:19.275 INFO:teuthology.orchestra.run.smithi116.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-02-19T15:20:19.278 INFO:teuthology.orchestra.run.smithi116.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-02-19T15:20:19.278 INFO:teuthology.orchestra.run.smithi116.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-02-19T15:20:19.282 INFO:teuthology.orchestra.run.smithi116.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.10-0ubuntu0.22.04.1 [143 kB] 2024-02-19T15:20:19.284 INFO:teuthology.orchestra.run.smithi116.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-02-19T15:20:19.307 INFO:teuthology.orchestra.run.smithi116.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-02-19T15:20:19.307 INFO:teuthology.orchestra.run.smithi116.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.14 [2,374 kB] 2024-02-19T15:20:19.549 INFO:teuthology.orchestra.run.smithi116.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-02-19T15:20:19.631 INFO:teuthology.orchestra.run.smithi116.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-02-19T15:20:19.914 INFO:teuthology.orchestra.run.smithi047.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-02-19T15:20:19.918 DEBUG:teuthology.orchestra.run.smithi047:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-02-19T15:20:19.995 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2024-02-19T15:20:20.107 INFO:teuthology.orchestra.run.smithi116.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-02-19T15:20:20.174 INFO:teuthology.orchestra.run.smithi047.stdout:Building dependency tree... 2024-02-19T15:20:20.175 INFO:teuthology.orchestra.run.smithi047.stdout:Reading state information... 2024-02-19T15:20:20.236 INFO:teuthology.orchestra.run.smithi116.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-02-19T15:20:20.248 INFO:teuthology.orchestra.run.smithi116.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-02-19T15:20:20.248 INFO:teuthology.orchestra.run.smithi116.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-02-19T15:20:20.249 INFO:teuthology.orchestra.run.smithi116.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-02-19T15:20:20.251 INFO:teuthology.orchestra.run.smithi116.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.10-0ubuntu0.22.04.1 [1,223 kB] 2024-02-19T15:20:20.284 INFO:teuthology.orchestra.run.smithi116.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-02-19T15:20:20.287 INFO:teuthology.orchestra.run.smithi116.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.10-0ubuntu0.22.04.1 [16.1 MB] 2024-02-19T15:20:20.337 INFO:teuthology.orchestra.run.smithi047.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-02-19T15:20:20.338 INFO:teuthology.orchestra.run.smithi047.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-02-19T15:20:20.338 INFO:teuthology.orchestra.run.smithi047.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2024-02-19T15:20:20.338 INFO:teuthology.orchestra.run.smithi047.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2024-02-19T15:20:20.338 INFO:teuthology.orchestra.run.smithi047.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-02-19T15:20:20.338 INFO:teuthology.orchestra.run.smithi047.stdout:The following packages were automatically installed and are no longer required: 2024-02-19T15:20:20.338 INFO:teuthology.orchestra.run.smithi047.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-19T15:20:20.338 INFO:teuthology.orchestra.run.smithi047.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-19T15:20:20.338 INFO:teuthology.orchestra.run.smithi047.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-19T15:20:20.339 INFO:teuthology.orchestra.run.smithi047.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-02-19T15:20:20.339 INFO:teuthology.orchestra.run.smithi047.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-02-19T15:20:20.339 INFO:teuthology.orchestra.run.smithi047.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-19T15:20:20.339 INFO:teuthology.orchestra.run.smithi047.stdout:The following additional packages will be installed: 2024-02-19T15:20:20.339 INFO:teuthology.orchestra.run.smithi047.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-02-19T15:20:20.339 INFO:teuthology.orchestra.run.smithi047.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-02-19T15:20:20.340 INFO:teuthology.orchestra.run.smithi047.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2024-02-19T15:20:20.340 INFO:teuthology.orchestra.run.smithi047.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-02-19T15:20:20.340 INFO:teuthology.orchestra.run.smithi047.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-02-19T15:20:20.341 INFO:teuthology.orchestra.run.smithi047.stdout:Suggested packages: 2024-02-19T15:20:20.341 INFO:teuthology.orchestra.run.smithi047.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-02-19T15:20:20.341 INFO:teuthology.orchestra.run.smithi047.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2024-02-19T15:20:20.341 INFO:teuthology.orchestra.run.smithi047.stdout: libengine-pkcs11-openssl 2024-02-19T15:20:20.392 INFO:teuthology.orchestra.run.smithi047.stdout:The following NEW packages will be installed: 2024-02-19T15:20:20.392 INFO:teuthology.orchestra.run.smithi047.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-02-19T15:20:20.393 INFO:teuthology.orchestra.run.smithi047.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-02-19T15:20:20.393 INFO:teuthology.orchestra.run.smithi047.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-02-19T15:20:20.393 INFO:teuthology.orchestra.run.smithi047.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-02-19T15:20:20.393 INFO:teuthology.orchestra.run.smithi047.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-02-19T15:20:20.393 INFO:teuthology.orchestra.run.smithi047.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-02-19T15:20:20.393 INFO:teuthology.orchestra.run.smithi047.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-02-19T15:20:20.394 INFO:teuthology.orchestra.run.smithi047.stdout:The following packages will be upgraded: 2024-02-19T15:20:20.395 INFO:teuthology.orchestra.run.smithi047.stdout: libssl3 2024-02-19T15:20:20.457 INFO:teuthology.orchestra.run.smithi047.stdout:1 upgraded, 39 newly installed, 0 to remove and 276 not upgraded. 2024-02-19T15:20:20.458 INFO:teuthology.orchestra.run.smithi047.stdout:Need to get 54.6 MB of archives. 2024-02-19T15:20:20.458 INFO:teuthology.orchestra.run.smithi047.stdout:After this operation, 202 MB of additional disk space will be used. 2024-02-19T15:20:20.458 INFO:teuthology.orchestra.run.smithi047.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-02-19T15:20:20.583 INFO:teuthology.orchestra.run.smithi047.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.14 [1,902 kB] 2024-02-19T15:20:20.650 INFO:teuthology.orchestra.run.smithi047.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-02-19T15:20:20.659 INFO:teuthology.orchestra.run.smithi047.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-02-19T15:20:20.660 INFO:teuthology.orchestra.run.smithi047.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-02-19T15:20:20.661 INFO:teuthology.orchestra.run.smithi047.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-02-19T15:20:20.663 INFO:teuthology.orchestra.run.smithi047.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-02-19T15:20:20.663 INFO:teuthology.orchestra.run.smithi047.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-02-19T15:20:20.664 INFO:teuthology.orchestra.run.smithi047.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-02-19T15:20:20.665 INFO:teuthology.orchestra.run.smithi047.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-02-19T15:20:20.665 INFO:teuthology.orchestra.run.smithi047.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-02-19T15:20:20.673 INFO:teuthology.orchestra.run.smithi047.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-02-19T15:20:20.687 INFO:teuthology.orchestra.run.smithi047.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-02-19T15:20:20.729 INFO:teuthology.orchestra.run.smithi116.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-02-19T15:20:20.893 INFO:teuthology.orchestra.run.smithi116.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-02-19T15:20:20.923 INFO:teuthology.orchestra.run.smithi047.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-02-19T15:20:20.925 INFO:teuthology.orchestra.run.smithi047.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-02-19T15:20:20.925 INFO:teuthology.orchestra.run.smithi047.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-02-19T15:20:20.928 INFO:teuthology.orchestra.run.smithi047.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-02-19T15:20:20.928 INFO:teuthology.orchestra.run.smithi047.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-02-19T15:20:20.933 INFO:teuthology.orchestra.run.smithi047.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.10-0ubuntu0.22.04.1 [143 kB] 2024-02-19T15:20:20.935 INFO:teuthology.orchestra.run.smithi047.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-02-19T15:20:20.942 INFO:teuthology.orchestra.run.smithi047.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-02-19T15:20:20.942 INFO:teuthology.orchestra.run.smithi047.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.14 [2,374 kB] 2024-02-19T15:20:20.965 INFO:teuthology.orchestra.run.smithi047.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-02-19T15:20:20.965 INFO:teuthology.orchestra.run.smithi047.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-02-19T15:20:20.976 INFO:teuthology.orchestra.run.smithi116.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-02-19T15:20:20.976 INFO:teuthology.orchestra.run.smithi116.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-02-19T15:20:20.976 INFO:teuthology.orchestra.run.smithi047.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-02-19T15:20:21.000 INFO:teuthology.orchestra.run.smithi047.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-02-19T15:20:21.003 INFO:teuthology.orchestra.run.smithi047.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-02-19T15:20:21.004 INFO:teuthology.orchestra.run.smithi047.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-02-19T15:20:21.004 INFO:teuthology.orchestra.run.smithi047.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-02-19T15:20:21.005 INFO:teuthology.orchestra.run.smithi047.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.10-0ubuntu0.22.04.1 [1,223 kB] 2024-02-19T15:20:21.017 INFO:teuthology.orchestra.run.smithi047.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-02-19T15:20:21.019 INFO:teuthology.orchestra.run.smithi047.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.10-0ubuntu0.22.04.1 [16.1 MB] 2024-02-19T15:20:21.060 INFO:teuthology.orchestra.run.smithi116.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-02-19T15:20:21.067 INFO:teuthology.orchestra.run.smithi116.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-02-19T15:20:21.179 INFO:teuthology.orchestra.run.smithi047.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-02-19T15:20:21.180 INFO:teuthology.orchestra.run.smithi047.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-02-19T15:20:21.180 INFO:teuthology.orchestra.run.smithi047.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-02-19T15:20:21.180 INFO:teuthology.orchestra.run.smithi047.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-02-19T15:20:21.180 INFO:teuthology.orchestra.run.smithi047.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-02-19T15:20:21.180 INFO:teuthology.orchestra.run.smithi047.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-02-19T15:20:21.194 INFO:teuthology.orchestra.run.smithi047.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-02-19T15:20:21.194 INFO:teuthology.orchestra.run.smithi047.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-02-19T15:20:21.537 INFO:teuthology.orchestra.run.smithi116.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-02-19T15:20:21.539 INFO:teuthology.orchestra.run.smithi116.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-02-19T15:20:21.582 INFO:teuthology.orchestra.run.smithi047.stdout:Preconfiguring packages ... 2024-02-19T15:20:21.728 INFO:teuthology.orchestra.run.smithi047.stdout:Fetched 54.6 MB in 1s (69.0 MB/s) 2024-02-19T15:20:21.869 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package flex. 2024-02-19T15:20:21.921 INFO:teuthology.orchestra.run.smithi116.stdout:Preconfiguring packages ... 2024-02-19T15:20:21.943 INFO:teuthology.orchestra.run.smithi047.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140583 files and directories currently installed.) 2024-02-19T15:20:21.946 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-02-19T15:20:21.985 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking flex (2.6.4-8build2) ... 2024-02-19T15:20:22.152 INFO:teuthology.orchestra.run.smithi116.stdout:Fetched 54.6 MB in 5s (11.3 MB/s) 2024-02-19T15:20:22.269 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package flex. 2024-02-19T15:20:22.342 INFO:teuthology.orchestra.run.smithi116.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140583 files and directories currently installed.) 2024-02-19T15:20:22.345 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-02-19T15:20:22.385 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking flex (2.6.4-8build2) ... 2024-02-19T15:20:22.521 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.14_amd64.deb ... 2024-02-19T15:20:22.698 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.14) over (3.0.2-0ubuntu1.7) ... 2024-02-19T15:20:22.904 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.14_amd64.deb ... 2024-02-19T15:20:23.099 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.14) over (3.0.2-0ubuntu1.7) ... 2024-02-19T15:20:23.106 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.14) ... 2024-02-19T15:20:23.504 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.14) ... 2024-02-19T15:20:23.607 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package bison. 2024-02-19T15:20:23.683 INFO:teuthology.orchestra.run.smithi047.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140656 files and directories currently installed.) 2024-02-19T15:20:23.686 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-02-19T15:20:23.734 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-02-19T15:20:24.069 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-02-19T15:20:24.083 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-02-19T15:20:24.109 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package bison. 2024-02-19T15:20:24.127 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-02-19T15:20:24.181 INFO:teuthology.orchestra.run.smithi116.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140656 files and directories currently installed.) 2024-02-19T15:20:24.184 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-02-19T15:20:24.219 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-02-19T15:20:24.497 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package dump. 2024-02-19T15:20:24.511 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-02-19T15:20:24.555 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking dump (0.4b47-3) ... 2024-02-19T15:20:24.603 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-02-19T15:20:24.618 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-02-19T15:20:24.653 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-02-19T15:20:24.841 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package indent. 2024-02-19T15:20:24.856 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-02-19T15:20:24.891 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-02-19T15:20:24.981 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package dump. 2024-02-19T15:20:24.996 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-02-19T15:20:25.073 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking dump (0.4b47-3) ... 2024-02-19T15:20:25.211 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-02-19T15:20:25.227 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-02-19T15:20:25.270 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-02-19T15:20:25.359 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package indent. 2024-02-19T15:20:25.374 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-02-19T15:20:25.418 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-02-19T15:20:25.599 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-02-19T15:20:25.614 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-02-19T15:20:25.656 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-02-19T15:20:25.670 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-02-19T15:20:25.684 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-02-19T15:20:25.754 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-02-19T15:20:25.951 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-02-19T15:20:25.967 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-02-19T15:20:26.009 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-02-19T15:20:26.057 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-02-19T15:20:26.072 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-02-19T15:20:26.107 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-02-19T15:20:26.321 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libfl2:amd64. 2024-02-19T15:20:26.337 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-02-19T15:20:26.343 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-02-19T15:20:26.358 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-02-19T15:20:26.379 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-02-19T15:20:26.452 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-02-19T15:20:26.614 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-02-19T15:20:26.630 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-02-19T15:20:26.673 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-02-19T15:20:26.780 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libfl2:amd64. 2024-02-19T15:20:26.795 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-02-19T15:20:26.838 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-02-19T15:20:26.927 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libjson-perl. 2024-02-19T15:20:26.943 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-02-19T15:20:26.977 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-02-19T15:20:27.112 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-02-19T15:20:27.127 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-02-19T15:20:27.174 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-02-19T15:20:27.289 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libllvm14:amd64. 2024-02-19T15:20:27.305 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-02-19T15:20:27.347 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-02-19T15:20:27.426 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libjson-perl. 2024-02-19T15:20:27.442 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-02-19T15:20:27.476 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-02-19T15:20:27.771 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libllvm14:amd64. 2024-02-19T15:20:27.787 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-02-19T15:20:27.862 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-02-19T15:20:28.700 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-02-19T15:20:28.716 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-02-19T15:20:28.757 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-02-19T15:20:29.002 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libmbim-proxy. 2024-02-19T15:20:29.018 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-02-19T15:20:29.060 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-02-19T15:20:29.174 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-02-19T15:20:29.190 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-02-19T15:20:29.232 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-02-19T15:20:29.371 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-02-19T15:20:29.388 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-02-19T15:20:29.421 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-02-19T15:20:29.468 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libmbim-proxy. 2024-02-19T15:20:29.483 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-02-19T15:20:29.551 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-02-19T15:20:29.740 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libndp0:amd64. 2024-02-19T15:20:29.756 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-02-19T15:20:29.798 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-02-19T15:20:29.879 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-02-19T15:20:29.894 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-02-19T15:20:29.937 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-02-19T15:20:30.143 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libnm0:amd64. 2024-02-19T15:20:30.159 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-02-19T15:20:30.251 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-02-19T15:20:30.306 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libndp0:amd64. 2024-02-19T15:20:30.322 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-02-19T15:20:30.365 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-02-19T15:20:30.563 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libpq5:amd64. 2024-02-19T15:20:30.580 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../16-libpq5_14.10-0ubuntu0.22.04.1_amd64.deb ... 2024-02-19T15:20:30.621 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libpq5:amd64 (14.10-0ubuntu0.22.04.1) ... 2024-02-19T15:20:30.675 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libnm0:amd64. 2024-02-19T15:20:30.691 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-02-19T15:20:30.733 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-02-19T15:20:30.924 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-02-19T15:20:30.941 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-02-19T15:20:30.982 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-02-19T15:20:31.045 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libpq5:amd64. 2024-02-19T15:20:31.060 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../16-libpq5_14.10-0ubuntu0.22.04.1_amd64.deb ... 2024-02-19T15:20:31.103 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libpq5:amd64 (14.10-0ubuntu0.22.04.1) ... 2024-02-19T15:20:31.260 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libqmi-proxy. 2024-02-19T15:20:31.276 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-02-19T15:20:31.318 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-02-19T15:20:31.439 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-02-19T15:20:31.455 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-02-19T15:20:31.497 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-02-19T15:20:31.587 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-02-19T15:20:31.604 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.14_amd64.deb ... 2024-02-19T15:20:31.647 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.14) ... 2024-02-19T15:20:31.825 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libqmi-proxy. 2024-02-19T15:20:31.841 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-02-19T15:20:31.883 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-02-19T15:20:32.107 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-02-19T15:20:32.123 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-02-19T15:20:32.127 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-02-19T15:20:32.143 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.14_amd64.deb ... 2024-02-19T15:20:32.165 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-02-19T15:20:32.185 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.14) ... 2024-02-19T15:20:32.510 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package modemmanager. 2024-02-19T15:20:32.526 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-02-19T15:20:32.568 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-02-19T15:20:32.783 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-02-19T15:20:32.799 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-02-19T15:20:32.841 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-02-19T15:20:33.211 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package modemmanager. 2024-02-19T15:20:33.216 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package network-manager. 2024-02-19T15:20:33.226 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-02-19T15:20:33.232 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-02-19T15:20:33.260 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-02-19T15:20:33.282 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-02-19T15:20:33.869 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package network-manager. 2024-02-19T15:20:33.881 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ppp. 2024-02-19T15:20:33.884 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-02-19T15:20:33.898 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-02-19T15:20:33.934 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-02-19T15:20:34.259 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-02-19T15:20:34.558 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ppp. 2024-02-19T15:20:34.574 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-02-19T15:20:34.619 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package pptp-linux. 2024-02-19T15:20:34.634 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-02-19T15:20:34.677 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-02-19T15:20:34.882 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-02-19T15:20:34.930 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package network-manager-pptp. 2024-02-19T15:20:34.945 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-02-19T15:20:34.988 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-02-19T15:20:35.246 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package pptp-linux. 2024-02-19T15:20:35.260 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-02-19T15:20:35.300 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package postgresql-client-common. 2024-02-19T15:20:35.304 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-02-19T15:20:35.315 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-02-19T15:20:35.409 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking postgresql-client-common (238) ... 2024-02-19T15:20:35.548 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package network-manager-pptp. 2024-02-19T15:20:35.563 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-02-19T15:20:35.598 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-02-19T15:20:35.713 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package postgresql-client-14. 2024-02-19T15:20:35.729 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../27-postgresql-client-14_14.10-0ubuntu0.22.04.1_amd64.deb ... 2024-02-19T15:20:35.791 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking postgresql-client-14 (14.10-0ubuntu0.22.04.1) ... 2024-02-19T15:20:35.943 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package postgresql-client-common. 2024-02-19T15:20:35.958 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-02-19T15:20:36.001 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking postgresql-client-common (238) ... 2024-02-19T15:20:36.313 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package postgresql-client-14. 2024-02-19T15:20:36.328 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../27-postgresql-client-14_14.10-0ubuntu0.22.04.1_amd64.deb ... 2024-02-19T15:20:36.371 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking postgresql-client-14 (14.10-0ubuntu0.22.04.1) ... 2024-02-19T15:20:36.468 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package postgresql-common. 2024-02-19T15:20:36.484 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-02-19T15:20:36.545 INFO:teuthology.orchestra.run.smithi047.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-02-19T15:20:36.567 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking postgresql-common (238) ... 2024-02-19T15:20:36.926 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package postgresql-common. 2024-02-19T15:20:36.941 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-02-19T15:20:36.992 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package postgresql-14. 2024-02-19T15:20:36.993 INFO:teuthology.orchestra.run.smithi116.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-02-19T15:20:37.008 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../29-postgresql-14_14.10-0ubuntu0.22.04.1_amd64.deb ... 2024-02-19T15:20:37.024 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking postgresql-common (238) ... 2024-02-19T15:20:37.044 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking postgresql-14 (14.10-0ubuntu0.22.04.1) ... 2024-02-19T15:20:37.525 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package postgresql-14. 2024-02-19T15:20:37.539 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../29-postgresql-14_14.10-0ubuntu0.22.04.1_amd64.deb ... 2024-02-19T15:20:37.627 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking postgresql-14 (14.10-0ubuntu0.22.04.1) ... 2024-02-19T15:20:38.128 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package postgresql. 2024-02-19T15:20:38.143 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-02-19T15:20:38.178 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking postgresql (14+238) ... 2024-02-19T15:20:38.405 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package postgresql-client. 2024-02-19T15:20:38.420 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-02-19T15:20:38.463 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking postgresql-client (14+238) ... 2024-02-19T15:20:38.724 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package postgresql-contrib. 2024-02-19T15:20:38.740 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-02-19T15:20:38.782 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking postgresql-contrib (14+238) ... 2024-02-19T15:20:38.837 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package postgresql. 2024-02-19T15:20:38.852 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-02-19T15:20:38.895 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking postgresql (14+238) ... 2024-02-19T15:20:39.085 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package tcl8.6. 2024-02-19T15:20:39.101 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-02-19T15:20:39.122 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package postgresql-client. 2024-02-19T15:20:39.137 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-02-19T15:20:39.143 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-02-19T15:20:39.180 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking postgresql-client (14+238) ... 2024-02-19T15:20:39.413 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package tcl. 2024-02-19T15:20:39.429 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-02-19T15:20:39.432 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package postgresql-contrib. 2024-02-19T15:20:39.447 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-02-19T15:20:39.467 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-02-19T15:20:39.490 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking postgresql-contrib (14+238) ... 2024-02-19T15:20:39.790 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package wpasupplicant. 2024-02-19T15:20:39.807 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-02-19T15:20:39.818 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package tcl8.6. 2024-02-19T15:20:39.834 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-02-19T15:20:39.848 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-02-19T15:20:39.876 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-02-19T15:20:40.187 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package tcl. 2024-02-19T15:20:40.204 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-02-19T15:20:40.219 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-02-19T15:20:40.236 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-02-19T15:20:40.250 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-02-19T15:20:40.277 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-02-19T15:20:40.557 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package wpasupplicant. 2024-02-19T15:20:40.573 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-02-19T15:20:40.615 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-02-19T15:20:40.986 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-02-19T15:20:41.002 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-02-19T15:20:41.044 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-02-19T15:20:41.140 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package usb-modeswitch. 2024-02-19T15:20:41.159 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-02-19T15:20:41.207 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-02-19T15:20:41.456 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package usb-modeswitch. 2024-02-19T15:20:41.471 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-02-19T15:20:41.493 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up postgresql-client-common (238) ... 2024-02-19T15:20:41.521 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-02-19T15:20:41.729 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up flex (2.6.4-8build2) ... 2024-02-19T15:20:41.805 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up postgresql-client-common (238) ... 2024-02-19T15:20:41.855 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-02-19T15:20:41.982 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libpq5:amd64 (14.10-0ubuntu0.22.04.1) ... 2024-02-19T15:20:42.011 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up flex (2.6.4-8build2) ... 2024-02-19T15:20:42.107 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-02-19T15:20:42.170 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-02-19T15:20:42.217 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-02-19T15:20:42.296 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libpq5:amd64 (14.10-0ubuntu0.22.04.1) ... 2024-02-19T15:20:42.318 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up postgresql-client-14 (14.10-0ubuntu0.22.04.1) ... 2024-02-19T15:20:42.422 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-02-19T15:20:42.548 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-02-19T15:20:42.674 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up postgresql-client-14 (14.10-0ubuntu0.22.04.1) ... 2024-02-19T15:20:43.263 INFO:teuthology.orchestra.run.smithi047.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-02-19T15:20:43.357 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-02-19T15:20:43.623 INFO:teuthology.orchestra.run.smithi116.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-02-19T15:20:43.713 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-02-19T15:20:43.743 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-02-19T15:20:43.743 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-02-19T15:20:44.091 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-02-19T15:20:44.091 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-02-19T15:20:44.426 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-02-19T15:20:44.552 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.14) ... 2024-02-19T15:20:44.679 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-02-19T15:20:44.732 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-02-19T15:20:44.804 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-02-19T15:20:44.833 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.14) ... 2024-02-19T15:20:44.922 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-02-19T15:20:45.018 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-02-19T15:20:45.102 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-02-19T15:20:45.144 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-02-19T15:20:45.266 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-02-19T15:20:45.278 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-02-19T15:20:45.404 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-02-19T15:20:45.530 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-02-19T15:20:45.971 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-02-19T15:20:46.097 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up dump (0.4b47-3) ... 2024-02-19T15:20:46.184 INFO:teuthology.orchestra.run.smithi047.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-02-19T15:20:46.243 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-02-19T15:20:46.248 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-02-19T15:20:46.375 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-02-19T15:20:46.403 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up dump (0.4b47-3) ... 2024-02-19T15:20:46.498 INFO:teuthology.orchestra.run.smithi116.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-02-19T15:20:46.500 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up tcl (8.6.11+1build2) ... 2024-02-19T15:20:46.562 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-02-19T15:20:46.626 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libjson-perl (4.04000-1) ... 2024-02-19T15:20:46.688 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-02-19T15:20:46.752 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-02-19T15:20:46.814 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up tcl (8.6.11+1build2) ... 2024-02-19T15:20:46.851 INFO:teuthology.orchestra.run.smithi047.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-02-19T15:20:46.921 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-02-19T15:20:46.940 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libjson-perl (4.04000-1) ... 2024-02-19T15:20:47.047 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-02-19T15:20:47.100 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-02-19T15:20:47.181 INFO:teuthology.orchestra.run.smithi116.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-02-19T15:20:47.198 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-02-19T15:20:47.243 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-02-19T15:20:47.332 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-02-19T15:20:47.369 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-02-19T15:20:47.458 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-02-19T15:20:47.545 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-02-19T15:20:47.626 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up postgresql-client (14+238) ... 2024-02-19T15:20:47.696 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-02-19T15:20:47.752 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-02-19T15:20:47.867 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-02-19T15:20:48.379 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:20:48.380 INFO:teuthology.orchestra.run.smithi047.stdout:The following network interfaces were found in /etc/network/interfaces 2024-02-19T15:20:48.380 INFO:teuthology.orchestra.run.smithi047.stdout:which means they are currently configured by ifupdown: 2024-02-19T15:20:48.380 INFO:teuthology.orchestra.run.smithi047.stdout:- ens1f0 2024-02-19T15:20:48.380 INFO:teuthology.orchestra.run.smithi047.stdout:If you want to manage those interfaces with NetworkManager instead 2024-02-19T15:20:48.380 INFO:teuthology.orchestra.run.smithi047.stdout:remove their configuration from /etc/network/interfaces. 2024-02-19T15:20:48.380 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:20:48.519 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-02-19T15:20:48.749 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up postgresql-client (14+238) ... 2024-02-19T15:20:48.874 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-02-19T15:20:48.942 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-02-19T15:20:49.369 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-02-19T15:20:49.420 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:20:49.420 INFO:teuthology.orchestra.run.smithi116.stdout:The following network interfaces were found in /etc/network/interfaces 2024-02-19T15:20:49.420 INFO:teuthology.orchestra.run.smithi116.stdout:which means they are currently configured by ifupdown: 2024-02-19T15:20:49.420 INFO:teuthology.orchestra.run.smithi116.stdout:- enp3s0f1 2024-02-19T15:20:49.420 INFO:teuthology.orchestra.run.smithi116.stdout:If you want to manage those interfaces with NetworkManager instead 2024-02-19T15:20:49.420 INFO:teuthology.orchestra.run.smithi116.stdout:remove their configuration from /etc/network/interfaces. 2024-02-19T15:20:49.420 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:20:49.552 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-02-19T15:20:49.924 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-02-19T15:20:50.088 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-02-19T15:20:50.257 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-02-19T15:20:50.321 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-02-19T15:20:50.366 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-02-19T15:20:50.467 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up postgresql-common (238) ... 2024-02-19T15:20:50.977 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-02-19T15:20:51.120 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-02-19T15:20:51.246 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-02-19T15:20:51.397 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up postgresql-common (238) ... 2024-02-19T15:20:51.696 INFO:teuthology.orchestra.run.smithi047.stdout:Adding user postgres to group ssl-cert 2024-02-19T15:20:51.957 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:20:51.957 INFO:teuthology.orchestra.run.smithi047.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-02-19T15:20:52.230 INFO:teuthology.orchestra.run.smithi047.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-02-19T15:20:52.230 INFO:teuthology.orchestra.run.smithi047.stdout:Removing obsolete dictionary files: 2024-02-19T15:20:52.341 INFO:teuthology.orchestra.run.smithi116.stdout:Adding user postgres to group ssl-cert 2024-02-19T15:20:52.609 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:20:52.609 INFO:teuthology.orchestra.run.smithi116.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-02-19T15:20:52.888 INFO:teuthology.orchestra.run.smithi116.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-02-19T15:20:52.889 INFO:teuthology.orchestra.run.smithi116.stdout:Removing obsolete dictionary files: 2024-02-19T15:20:52.942 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-02-19T15:20:53.493 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up postgresql-14 (14.10-0ubuntu0.22.04.1) ... 2024-02-19T15:20:53.568 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-02-19T15:20:53.822 INFO:teuthology.orchestra.run.smithi047.stdout:Creating new PostgreSQL cluster 14/main ... 2024-02-19T15:20:53.847 INFO:teuthology.orchestra.run.smithi047.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-02-19T15:20:53.899 INFO:teuthology.orchestra.run.smithi047.stdout:The files belonging to this database system will be owned by user "postgres". 2024-02-19T15:20:53.899 INFO:teuthology.orchestra.run.smithi047.stdout:This user must also own the server process. 2024-02-19T15:20:53.899 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:20:53.899 INFO:teuthology.orchestra.run.smithi047.stdout:The database cluster will be initialized with locale "en_US". 2024-02-19T15:20:53.899 INFO:teuthology.orchestra.run.smithi047.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-02-19T15:20:53.899 INFO:teuthology.orchestra.run.smithi047.stdout:The default text search configuration will be set to "english". 2024-02-19T15:20:53.899 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:20:53.900 INFO:teuthology.orchestra.run.smithi047.stdout:Data page checksums are disabled. 2024-02-19T15:20:53.900 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:20:53.900 INFO:teuthology.orchestra.run.smithi047.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-02-19T15:20:53.900 INFO:teuthology.orchestra.run.smithi047.stdout:creating subdirectories ... ok 2024-02-19T15:20:53.900 INFO:teuthology.orchestra.run.smithi047.stdout:selecting dynamic shared memory implementation ... posix 2024-02-19T15:20:53.924 INFO:teuthology.orchestra.run.smithi047.stdout:selecting default max_connections ... 100 2024-02-19T15:20:53.958 INFO:teuthology.orchestra.run.smithi047.stdout:selecting default shared_buffers ... 128MB 2024-02-19T15:20:53.973 INFO:teuthology.orchestra.run.smithi047.stdout:selecting default time zone ... Etc/UTC 2024-02-19T15:20:53.973 INFO:teuthology.orchestra.run.smithi047.stdout:creating configuration files ... ok 2024-02-19T15:20:54.031 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up postgresql-14 (14.10-0ubuntu0.22.04.1) ... 2024-02-19T15:20:54.144 INFO:teuthology.orchestra.run.smithi047.stdout:running bootstrap script ... ok 2024-02-19T15:20:54.402 INFO:teuthology.orchestra.run.smithi116.stdout:Creating new PostgreSQL cluster 14/main ... 2024-02-19T15:20:54.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-02-19T15:20:54.474 INFO:teuthology.orchestra.run.smithi116.stdout:The files belonging to this database system will be owned by user "postgres". 2024-02-19T15:20:54.474 INFO:teuthology.orchestra.run.smithi116.stdout:This user must also own the server process. 2024-02-19T15:20:54.474 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:20:54.475 INFO:teuthology.orchestra.run.smithi116.stdout:The database cluster will be initialized with locale "en_US". 2024-02-19T15:20:54.475 INFO:teuthology.orchestra.run.smithi116.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-02-19T15:20:54.475 INFO:teuthology.orchestra.run.smithi116.stdout:The default text search configuration will be set to "english". 2024-02-19T15:20:54.475 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:20:54.475 INFO:teuthology.orchestra.run.smithi116.stdout:Data page checksums are disabled. 2024-02-19T15:20:54.475 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:20:54.475 INFO:teuthology.orchestra.run.smithi116.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-02-19T15:20:54.475 INFO:teuthology.orchestra.run.smithi116.stdout:creating subdirectories ... ok 2024-02-19T15:20:54.476 INFO:teuthology.orchestra.run.smithi116.stdout:selecting dynamic shared memory implementation ... posix 2024-02-19T15:20:54.499 INFO:teuthology.orchestra.run.smithi116.stdout:selecting default max_connections ... 100 2024-02-19T15:20:54.532 INFO:teuthology.orchestra.run.smithi116.stdout:selecting default shared_buffers ... 128MB 2024-02-19T15:20:54.546 INFO:teuthology.orchestra.run.smithi116.stdout:selecting default time zone ... Etc/UTC 2024-02-19T15:20:54.547 INFO:teuthology.orchestra.run.smithi116.stdout:creating configuration files ... ok 2024-02-19T15:20:54.709 INFO:teuthology.orchestra.run.smithi116.stdout:running bootstrap script ... ok 2024-02-19T15:20:54.789 INFO:teuthology.orchestra.run.smithi047.stdout:performing post-bootstrap initialization ... ok 2024-02-19T15:20:55.315 INFO:teuthology.orchestra.run.smithi047.stdout:syncing data to disk ... ok 2024-02-19T15:20:55.330 INFO:teuthology.orchestra.run.smithi116.stdout:performing post-bootstrap initialization ... ok 2024-02-19T15:20:55.832 INFO:teuthology.orchestra.run.smithi047.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-02-19T15:20:55.898 INFO:teuthology.orchestra.run.smithi116.stdout:syncing data to disk ... ok 2024-02-19T15:20:56.402 INFO:teuthology.orchestra.run.smithi116.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-02-19T15:20:58.784 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-02-19T15:20:58.893 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up postgresql-contrib (14+238) ... 2024-02-19T15:20:59.019 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up postgresql (14+238) ... 2024-02-19T15:20:59.153 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-02-19T15:20:59.288 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-02-19T15:20:59.471 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-02-19T15:20:59.557 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-02-19T15:20:59.557 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-02-19T15:20:59.623 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up postgresql-contrib (14+238) ... 2024-02-19T15:20:59.748 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up postgresql (14+238) ... 2024-02-19T15:20:59.866 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-02-19T15:20:59.967 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-02-19T15:21:00.226 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-02-19T15:21:00.226 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-02-19T15:21:00.348 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-02-19T15:21:00.659 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-02-19T15:21:00.802 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-02-19T15:21:00.944 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-02-19T15:21:01.305 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-02-19T15:21:01.464 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-02-19T15:21:02.247 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-02-19T15:21:02.347 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-02-19T15:21:02.834 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-02-19T15:21:02.886 INFO:teuthology.orchestra.run.smithi047.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-02-19T15:21:02.893 DEBUG:teuthology.parallel:result is None 2024-02-19T15:21:02.917 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-02-19T15:21:03.461 INFO:teuthology.orchestra.run.smithi116.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-02-19T15:21:03.467 DEBUG:teuthology.parallel:result is None 2024-02-19T15:21:03.468 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9f8908eed2b2d7e8c316bd5be7ed6e49024931ee 2024-02-19T15:21:03.686 DEBUG:teuthology.orchestra.run.smithi047:> dpkg-query -W -f '${Version}' ceph 2024-02-19T15:21:03.710 INFO:teuthology.orchestra.run.smithi047.stdout:19.0.0-1451-g9f8908ee-1jammy 2024-02-19T15:21:03.710 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1451-g9f8908ee-1jammy 2024-02-19T15:21:03.711 INFO:teuthology.task.install:The correct ceph version 19.0.0-1451-g9f8908ee-1jammy is installed. 2024-02-19T15:21:03.712 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9f8908eed2b2d7e8c316bd5be7ed6e49024931ee 2024-02-19T15:21:03.927 DEBUG:teuthology.orchestra.run.smithi116:> dpkg-query -W -f '${Version}' ceph 2024-02-19T15:21:03.949 INFO:teuthology.orchestra.run.smithi116.stdout:19.0.0-1451-g9f8908ee-1jammy 2024-02-19T15:21:03.949 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1451-g9f8908ee-1jammy 2024-02-19T15:21:03.949 INFO:teuthology.task.install:The correct ceph version 19.0.0-1451-g9f8908ee-1jammy is installed. 2024-02-19T15:21:03.951 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-19T15:21:03.951 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:03.951 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T15:21:03.973 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:03.973 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T15:21:04.015 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-19T15:21:04.016 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:04.016 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T15:21:04.038 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T15:21:04.102 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:04.102 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T15:21:04.121 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T15:21:04.183 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-19T15:21:04.183 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:04.183 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T15:21:04.204 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T15:21:04.266 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:04.266 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T15:21:04.285 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T15:21:04.350 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-19T15:21:04.351 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:04.351 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T15:21:04.372 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T15:21:04.434 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:04.434 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T15:21:04.454 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T15:21:04.515 INFO:teuthology.run_tasks:Running task ceph... 2024-02-19T15:21:04.592 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-02-19T15:21:04.592 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 777 /var/log/ceph 2024-02-19T15:21:04.595 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /var/log/ceph 2024-02-19T15:21:04.615 INFO:tasks.ceph:Disabling ceph logrotate... 2024-02-19T15:21:04.615 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /etc/logrotate.d/ceph 2024-02-19T15:21:04.660 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/logrotate.d/ceph 2024-02-19T15:21:04.680 INFO:tasks.ceph:Creating extra log directories... 2024-02-19T15:21:04.680 DEBUG:teuthology.orchestra.run.smithi047:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-02-19T15:21:04.724 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-02-19T15:21:04.753 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-02-19T15:21:04.754 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-02-19T15:21:04.754 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-02-19T15:21:04.754 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:04.754 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-02-19T15:21:04.798 DEBUG:teuthology.orchestra.run.smithi047:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-02-19T15:21:04.891 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:04.891 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-02-19T15:21:04.900 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-02-19T15:21:04.992 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-02-19T15:21:04.992 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-02-19T15:21:04.992 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566477', 'branch': 'wip-lusov-qdb', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/strays}', 'email': None, 'first_in_suite': False, 'job_id': '7566477', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-lusov-qdb'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee'}}, 'kclient': {'syntax': 'v1'}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-qdb', 'sha1': '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 2280, 'sha1': '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee', 'sleep_before_teardown': 0, 'subset': '111/10000', 'suite': 'fs', 'suite_branch': 'wip-lusov-qdb', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi047.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDSeBWtWbirpTGhOQFxv1p6nodVqAgn2Nj5S+d+rpDwo/Q4HD/uW47/wFuybaHwbNC/xXZG1wh20yDi7sJGOLKU=', 'smithi116.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKf9i9SekZJBkyR5IzfNKL5GXCmWXK+l0/5forL4I1xhqyzutmv/bot1nC8/sXcsAogerwpcULVffpa+bD1QpRc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_strays']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-02-19_14:33:01', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-02-19T15:21:04.993 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-02-19T15:21:04.996 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-02-19T15:21:05.005 DEBUG:teuthology.orchestra.run.smithi047:> sudo install -d -m0777 -- /var/run/ceph 2024-02-19T15:21:05.052 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/run/ceph 2024-02-19T15:21:05.072 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:05.072 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/scratch_devs of=/dev/stdout 2024-02-19T15:21:05.122 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-19T15:21:05.122 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_1 2024-02-19T15:21:05.174 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-19T15:21:05.174 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T15:21:05.174 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 898 Links: 1 2024-02-19T15:21:05.174 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T15:21:05.174 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-02-19 15:14:50.195500415 +0000 2024-02-19T15:21:05.174 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-02-19 15:14:49.939502700 +0000 2024-02-19T15:21:05.174 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-02-19 15:14:49.939502700 +0000 2024-02-19T15:21:05.174 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-02-19 15:14:49.939502700 +0000 2024-02-19T15:21:05.174 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-19T15:21:05.236 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-02-19T15:21:05.236 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-02-19T15:21:05.236 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000266517 s, 1.9 MB/s 2024-02-19T15:21:05.238 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-19T15:21:05.291 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_2 2024-02-19T15:21:05.342 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-19T15:21:05.342 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T15:21:05.342 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 915 Links: 1 2024-02-19T15:21:05.342 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T15:21:05.342 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-02-19 15:14:50.695495948 +0000 2024-02-19T15:21:05.342 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-02-19 15:14:50.447498163 +0000 2024-02-19T15:21:05.342 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-02-19 15:14:50.447498163 +0000 2024-02-19T15:21:05.342 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-02-19 15:14:50.447498163 +0000 2024-02-19T15:21:05.342 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-19T15:21:05.404 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-02-19T15:21:05.405 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-02-19T15:21:05.405 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000266686 s, 1.9 MB/s 2024-02-19T15:21:05.406 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-19T15:21:05.463 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_3 2024-02-19T15:21:05.514 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-19T15:21:05.514 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T15:21:05.514 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 933 Links: 1 2024-02-19T15:21:05.514 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T15:21:05.514 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-02-19 15:14:51.227491197 +0000 2024-02-19T15:21:05.514 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-02-19 15:14:50.955493626 +0000 2024-02-19T15:21:05.514 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-02-19 15:14:50.955493626 +0000 2024-02-19T15:21:05.514 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-02-19 15:14:50.955493626 +0000 2024-02-19T15:21:05.514 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-19T15:21:05.580 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-02-19T15:21:05.580 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-02-19T15:21:05.580 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000268238 s, 1.9 MB/s 2024-02-19T15:21:05.582 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-19T15:21:05.636 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_4 2024-02-19T15:21:05.686 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-19T15:21:05.686 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T15:21:05.686 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 952 Links: 1 2024-02-19T15:21:05.686 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T15:21:05.686 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-02-19 15:14:51.711486874 +0000 2024-02-19T15:21:05.686 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-02-19 15:14:51.471489016 +0000 2024-02-19T15:21:05.686 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-02-19 15:14:51.471489016 +0000 2024-02-19T15:21:05.686 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-02-19 15:14:51.471489016 +0000 2024-02-19T15:21:05.686 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-19T15:21:05.752 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-02-19T15:21:05.752 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-02-19T15:21:05.752 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000260695 s, 2.0 MB/s 2024-02-19T15:21:05.754 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-19T15:21:05.807 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-02-19T15:21:05.808 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:05.808 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-02-19T15:21:05.816 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-19T15:21:05.817 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2024-02-19T15:21:05.868 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-19T15:21:05.868 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T15:21:05.868 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 888 Links: 1 2024-02-19T15:21:05.868 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T15:21:05.868 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-02-19 15:14:46.394087857 +0000 2024-02-19T15:21:05.869 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-02-19 15:14:46.390087893 +0000 2024-02-19T15:21:05.869 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-02-19 15:14:46.390087893 +0000 2024-02-19T15:21:05.869 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-02-19 15:14:46.390087893 +0000 2024-02-19T15:21:05.869 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-19T15:21:05.930 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-02-19T15:21:05.931 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-02-19T15:21:05.931 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000422538 s, 1.2 MB/s 2024-02-19T15:21:05.933 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-19T15:21:05.990 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2024-02-19T15:21:06.044 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-19T15:21:06.045 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T15:21:06.045 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 908 Links: 1 2024-02-19T15:21:06.045 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T15:21:06.045 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-02-19 15:14:47.178080863 +0000 2024-02-19T15:21:06.045 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-02-19 15:14:46.906083289 +0000 2024-02-19T15:21:06.045 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-02-19 15:14:46.906083289 +0000 2024-02-19T15:21:06.045 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-02-19 15:14:46.906083289 +0000 2024-02-19T15:21:06.045 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-19T15:21:06.107 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-02-19T15:21:06.107 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-02-19T15:21:06.107 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000380706 s, 1.3 MB/s 2024-02-19T15:21:06.108 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-19T15:21:06.162 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2024-02-19T15:21:06.216 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-19T15:21:06.216 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T15:21:06.216 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-02-19T15:21:06.217 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T15:21:06.217 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-02-19 15:14:47.410078794 +0000 2024-02-19T15:21:06.217 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-02-19 15:14:47.406078829 +0000 2024-02-19T15:21:06.217 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-02-19 15:14:47.406078829 +0000 2024-02-19T15:21:06.217 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-02-19 15:14:47.406078829 +0000 2024-02-19T15:21:06.217 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-19T15:21:06.279 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-02-19T15:21:06.279 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-02-19T15:21:06.279 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.00037998 s, 1.3 MB/s 2024-02-19T15:21:06.281 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-19T15:21:06.339 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2024-02-19T15:21:06.388 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-19T15:21:06.389 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T15:21:06.389 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 939 Links: 1 2024-02-19T15:21:06.389 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T15:21:06.389 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-02-19 15:14:48.182071907 +0000 2024-02-19T15:21:06.389 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-02-19 15:14:47.918074261 +0000 2024-02-19T15:21:06.389 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-02-19 15:14:47.918074261 +0000 2024-02-19T15:21:06.389 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-02-19 15:14:47.918074261 +0000 2024-02-19T15:21:06.389 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-19T15:21:06.450 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-02-19T15:21:06.450 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-02-19T15:21:06.450 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000392794 s, 1.3 MB/s 2024-02-19T15:21:06.452 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-19T15:21:06.507 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-02-19T15:21:06.507 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi047.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-02-19T15:21:06.507 INFO:tasks.ceph:Generating config... 2024-02-19T15:21:06.508 INFO:tasks.ceph:[client] client mount timeout = 600 2024-02-19T15:21:06.508 INFO:tasks.ceph:[client] debug client = 20 2024-02-19T15:21:06.508 INFO:tasks.ceph:[client] debug ms = 1 2024-02-19T15:21:06.508 INFO:tasks.ceph:[client] pid file = 2024-02-19T15:21:06.509 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-02-19T15:21:06.509 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-02-19T15:21:06.509 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-02-19T15:21:06.509 INFO:tasks.ceph:[mds] debug mds = 20 2024-02-19T15:21:06.509 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-02-19T15:21:06.509 INFO:tasks.ceph:[mds] debug ms = 1 2024-02-19T15:21:06.509 INFO:tasks.ceph:[mds] mds debug frag = True 2024-02-19T15:21:06.509 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-02-19T15:21:06.509 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-02-19T15:21:06.509 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-02-19T15:21:06.509 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-02-19T15:21:06.509 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-02-19T15:21:06.509 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-02-19T15:21:06.510 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-02-19T15:21:06.510 INFO:tasks.ceph:[mgr] debug ms = 1 2024-02-19T15:21:06.510 INFO:tasks.ceph:[mon] debug mon = 20 2024-02-19T15:21:06.510 INFO:tasks.ceph:[mon] debug ms = 1 2024-02-19T15:21:06.510 INFO:tasks.ceph:[mon] debug paxos = 20 2024-02-19T15:21:06.510 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-02-19T15:21:06.510 INFO:tasks.ceph:[osd] bdev async discard = True 2024-02-19T15:21:06.510 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-02-19T15:21:06.510 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-02-19T15:21:06.510 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-02-19T15:21:06.510 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-02-19T15:21:06.510 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-02-19T15:21:06.510 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-02-19T15:21:06.510 INFO:tasks.ceph:[osd] debug ms = 1 2024-02-19T15:21:06.510 INFO:tasks.ceph:[osd] debug osd = 20 2024-02-19T15:21:06.510 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-02-19T15:21:06.511 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-02-19T15:21:06.511 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-02-19T15:21:06.511 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-02-19T15:21:06.511 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-02-19T15:21:06.511 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-02-19T15:21:06.511 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-02-19T15:21:06.511 INFO:tasks.ceph:Setting up mon.a... 2024-02-19T15:21:06.511 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-02-19T15:21:06.558 INFO:teuthology.orchestra.run.smithi047.stdout:creating /etc/ceph/ceph.keyring 2024-02-19T15:21:06.573 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-02-19T15:21:06.621 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-02-19T15:21:06.684 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.47'), ('mon.b', '172.21.15.116'), ('mon.c', '[v2:172.21.15.116:3301,v1:172.21.15.116:6790]')] 2024-02-19T15:21:06.684 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.47,172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-02-19T15:21:06.685 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:06.685 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-02-19T15:21:06.737 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.47 --add b 172.21.15.116 --addv c '[v2:172.21.15.116:3301,v1:172.21.15.116:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-02-19T15:21:06.828 INFO:teuthology.orchestra.run.smithi047.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-02-19T15:21:06.829 INFO:teuthology.orchestra.run.smithi047.stdout:monmaptool: generated fsid ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e 2024-02-19T15:21:06.829 INFO:teuthology.orchestra.run.smithi047.stdout:setting min_mon_release = quincy 2024-02-19T15:21:06.829 INFO:teuthology.orchestra.run.smithi047.stdout:epoch 0 2024-02-19T15:21:06.829 INFO:teuthology.orchestra.run.smithi047.stdout:fsid ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e 2024-02-19T15:21:06.829 INFO:teuthology.orchestra.run.smithi047.stdout:last_changed 2024-02-19T15:21:06.830077+0000 2024-02-19T15:21:06.829 INFO:teuthology.orchestra.run.smithi047.stdout:created 2024-02-19T15:21:06.830077+0000 2024-02-19T15:21:06.829 INFO:teuthology.orchestra.run.smithi047.stdout:min_mon_release 17 (quincy) 2024-02-19T15:21:06.829 INFO:teuthology.orchestra.run.smithi047.stdout:election_strategy: 1 2024-02-19T15:21:06.829 INFO:teuthology.orchestra.run.smithi047.stdout:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-02-19T15:21:06.829 INFO:teuthology.orchestra.run.smithi047.stdout:1: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-02-19T15:21:06.829 INFO:teuthology.orchestra.run.smithi047.stdout:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-02-19T15:21:06.829 INFO:teuthology.orchestra.run.smithi047.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-02-19T15:21:06.832 DEBUG:teuthology.orchestra.run.smithi047:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-02-19T15:21:06.840 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e... 2024-02-19T15:21:06.841 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-19T15:21:06.888 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-19T15:21:06.927 INFO:teuthology.orchestra.run.smithi047.stdout:[global] 2024-02-19T15:21:06.927 INFO:teuthology.orchestra.run.smithi047.stdout: chdir = "" 2024-02-19T15:21:06.928 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-19T15:21:06.928 INFO:teuthology.orchestra.run.smithi047.stdout: auth supported = cephx 2024-02-19T15:21:06.928 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.928 INFO:teuthology.orchestra.run.smithi047.stdout: filestore xattr use omap = true 2024-02-19T15:21:06.928 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.928 INFO:teuthology.orchestra.run.smithi047.stdout: mon clock drift allowed = 1.000 2024-02-19T15:21:06.928 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.928 INFO:teuthology.orchestra.run.smithi047.stdout: osd crush chooseleaf type = 0 2024-02-19T15:21:06.928 INFO:teuthology.orchestra.run.smithi047.stdout: auth debug = true 2024-02-19T15:21:06.928 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.928 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on old message = true 2024-02-19T15:21:06.929 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on bug = true 2024-02-19T15:21:06.929 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.929 INFO:teuthology.orchestra.run.smithi047.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-19T15:21:06.929 INFO:teuthology.orchestra.run.smithi047.stdout: mon pg warn max object skew = 0 2024-02-19T15:21:06.929 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.929 INFO:teuthology.orchestra.run.smithi047.stdout: # disable pg_autoscaler by default for new pools 2024-02-19T15:21:06.929 INFO:teuthology.orchestra.run.smithi047.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-19T15:21:06.929 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.929 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default size = 2 2024-02-19T15:21:06.929 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.929 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow primary affinity = true 2024-02-19T15:21:06.929 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow pg remap = true 2024-02-19T15:21:06.929 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on legacy crush tunables = false 2024-02-19T15:21:06.929 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on crush straw calc version zero = false 2024-02-19T15:21:06.930 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on no sortbitwise = false 2024-02-19T15:21:06.930 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on osd down out interval zero = false 2024-02-19T15:21:06.930 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on too few osds = false 2024-02-19T15:21:06.930 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-19T15:21:06.930 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-19T15:21:06.930 INFO:teuthology.orchestra.run.smithi047.stdout: mon_allow_pool_size_one = true 2024-02-19T15:21:06.930 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.930 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-19T15:21:06.930 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.930 INFO:teuthology.orchestra.run.smithi047.stdout: osd default data pool replay window = 5 2024-02-19T15:21:06.930 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.930 INFO:teuthology.orchestra.run.smithi047.stdout: mon allow pool delete = true 2024-02-19T15:21:06.930 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.930 INFO:teuthology.orchestra.run.smithi047.stdout: mon cluster log file level = debug 2024-02-19T15:21:06.930 INFO:teuthology.orchestra.run.smithi047.stdout: debug asserts on shutdown = true 2024-02-19T15:21:06.930 INFO:teuthology.orchestra.run.smithi047.stdout: mon health detail to clog = false 2024-02-19T15:21:06.930 INFO:teuthology.orchestra.run.smithi047.stdout: mon host = "172.21.15.47,172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790]" 2024-02-19T15:21:06.931 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on skipped message = False 2024-02-19T15:21:06.931 INFO:teuthology.orchestra.run.smithi047.stdout: fsid = ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e 2024-02-19T15:21:06.931 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.931 INFO:teuthology.orchestra.run.smithi047.stdout:[osd] 2024-02-19T15:21:06.931 INFO:teuthology.orchestra.run.smithi047.stdout: osd journal size = 100 2024-02-19T15:21:06.931 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.931 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub load threshold = 5.0 2024-02-19T15:21:06.931 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub max interval = 600 2024-02-19T15:21:06.931 INFO:teuthology.orchestra.run.smithi047.stdout: osd mclock profile = high_recovery_ops 2024-02-19T15:21:06.931 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.931 INFO:teuthology.orchestra.run.smithi047.stdout: osd recover clone overlap = true 2024-02-19T15:21:06.931 INFO:teuthology.orchestra.run.smithi047.stdout: osd recovery max chunk = 1048576 2024-02-19T15:21:06.931 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.931 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug shutdown = true 2024-02-19T15:21:06.931 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug op order = true 2024-02-19T15:21:06.932 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify stray on activate = true 2024-02-19T15:21:06.932 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug trim objects = true 2024-02-19T15:21:06.932 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.932 INFO:teuthology.orchestra.run.smithi047.stdout: osd open classes on start = true 2024-02-19T15:21:06.932 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug pg log writeout = true 2024-02-19T15:21:06.932 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.932 INFO:teuthology.orchestra.run.smithi047.stdout: osd deep scrub update digest min age = 30 2024-02-19T15:21:06.932 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.932 INFO:teuthology.orchestra.run.smithi047.stdout: osd map max advance = 10 2024-02-19T15:21:06.932 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.932 INFO:teuthology.orchestra.run.smithi047.stdout: journal zero on create = true 2024-02-19T15:21:06.932 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.932 INFO:teuthology.orchestra.run.smithi047.stdout: filestore ondisk finisher threads = 3 2024-02-19T15:21:06.933 INFO:teuthology.orchestra.run.smithi047.stdout: filestore apply finisher threads = 3 2024-02-19T15:21:06.933 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.933 INFO:teuthology.orchestra.run.smithi047.stdout: bdev debug aio = true 2024-02-19T15:21:06.933 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug misdirected ops = true 2024-02-19T15:21:06.933 INFO:teuthology.orchestra.run.smithi047.stdout: bdev async discard = True 2024-02-19T15:21:06.933 INFO:teuthology.orchestra.run.smithi047.stdout: bdev enable discard = True 2024-02-19T15:21:06.933 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore allocator = bitmap 2024-02-19T15:21:06.933 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore block size = 96636764160 2024-02-19T15:21:06.933 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore fsck on mount = True 2024-02-19T15:21:06.933 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluefs = 1/20 2024-02-19T15:21:06.933 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluestore = 1/20 2024-02-19T15:21:06.933 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:21:06.933 INFO:teuthology.orchestra.run.smithi047.stdout: debug osd = 20 2024-02-19T15:21:06.933 INFO:teuthology.orchestra.run.smithi047.stdout: debug rocksdb = 4/10 2024-02-19T15:21:06.934 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd backfillfull_ratio = 0.85 2024-02-19T15:21:06.934 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd full ratio = 0.9 2024-02-19T15:21:06.934 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd nearfull ratio = 0.8 2024-02-19T15:21:06.934 INFO:teuthology.orchestra.run.smithi047.stdout: osd failsafe full ratio = 0.95 2024-02-19T15:21:06.934 INFO:teuthology.orchestra.run.smithi047.stdout: osd objectstore = bluestore 2024-02-19T15:21:06.934 INFO:teuthology.orchestra.run.smithi047.stdout: osd op complaint time = 180 2024-02-19T15:21:06.934 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.934 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr] 2024-02-19T15:21:06.934 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:21:06.934 INFO:teuthology.orchestra.run.smithi047.stdout: debug mgr = 20 2024-02-19T15:21:06.934 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-02-19T15:21:06.934 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-02-19T15:21:06.934 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:21:06.934 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:21:06.934 INFO:teuthology.orchestra.run.smithi047.stdout: mgr/telemetry/nag = false 2024-02-19T15:21:06.934 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.934 INFO:teuthology.orchestra.run.smithi047.stdout:[mon] 2024-02-19T15:21:06.935 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:21:06.935 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-02-19T15:21:06.935 INFO:teuthology.orchestra.run.smithi047.stdout: debug paxos = 20 2024-02-19T15:21:06.935 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-02-19T15:21:06.935 INFO:teuthology.orchestra.run.smithi047.stdout: mon data avail warn = 5 2024-02-19T15:21:06.935 INFO:teuthology.orchestra.run.smithi047.stdout: mon mgr mkfs grace = 240 2024-02-19T15:21:06.935 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:21:06.935 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd reporter subtree level = osd 2024-02-19T15:21:06.935 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd prime pg temp = true 2024-02-19T15:21:06.935 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:21:06.935 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.935 INFO:teuthology.orchestra.run.smithi047.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-19T15:21:06.935 INFO:teuthology.orchestra.run.smithi047.stdout: auth mon ticket ttl = 660 # 11m 2024-02-19T15:21:06.935 INFO:teuthology.orchestra.run.smithi047.stdout: auth service ticket ttl = 240 # 4m 2024-02-19T15:21:06.935 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.935 INFO:teuthology.orchestra.run.smithi047.stdout: # don't complain about insecure global_id in the test suite 2024-02-19T15:21:06.935 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-19T15:21:06.936 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-19T15:21:06.936 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.936 INFO:teuthology.orchestra.run.smithi047.stdout: # 1m isn't quite enough 2024-02-19T15:21:06.936 INFO:teuthology.orchestra.run.smithi047.stdout: mon_down_mkfs_grace = 2m 2024-02-19T15:21:06.936 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.936 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_filestore_osds = false 2024-02-19T15:21:06.936 INFO:teuthology.orchestra.run.smithi047.stdout: mon op complaint time = 120 2024-02-19T15:21:06.936 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:06.936 INFO:teuthology.orchestra.run.smithi047.stdout:[client] 2024-02-19T15:21:06.936 INFO:teuthology.orchestra.run.smithi047.stdout: rgw cache enabled = true 2024-02-19T15:21:06.936 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable ops log = true 2024-02-19T15:21:06.936 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable usage log = true 2024-02-19T15:21:06.936 INFO:teuthology.orchestra.run.smithi047.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-19T15:21:06.936 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-19T15:21:06.936 INFO:teuthology.orchestra.run.smithi047.stdout: client mount timeout = 600 2024-02-19T15:21:06.937 INFO:teuthology.orchestra.run.smithi047.stdout: debug client = 20 2024-02-19T15:21:06.937 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:21:06.937 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = "" 2024-02-19T15:21:06.937 INFO:teuthology.orchestra.run.smithi047.stdout: rados mon op timeout = 900 2024-02-19T15:21:06.937 INFO:teuthology.orchestra.run.smithi047.stdout: rados osd op timeout = 900 2024-02-19T15:21:06.937 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.a] 2024-02-19T15:21:06.937 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.b] 2024-02-19T15:21:06.937 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.c] 2024-02-19T15:21:06.937 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.a] 2024-02-19T15:21:06.937 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.c] 2024-02-19T15:21:06.937 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.b] 2024-02-19T15:21:06.937 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.d] 2024-02-19T15:21:06.937 INFO:teuthology.orchestra.run.smithi047.stdout:[mds] 2024-02-19T15:21:06.937 INFO:teuthology.orchestra.run.smithi047.stdout: debug mds = 20 2024-02-19T15:21:06.937 INFO:teuthology.orchestra.run.smithi047.stdout: debug mds balancer = 20 2024-02-19T15:21:06.937 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:21:06.937 INFO:teuthology.orchestra.run.smithi047.stdout: mds debug frag = True 2024-02-19T15:21:06.937 INFO:teuthology.orchestra.run.smithi047.stdout: mds debug scatterstat = True 2024-02-19T15:21:06.938 INFO:teuthology.orchestra.run.smithi047.stdout: mds op complaint time = 180 2024-02-19T15:21:06.938 INFO:teuthology.orchestra.run.smithi047.stdout: mds verify scatter = True 2024-02-19T15:21:06.938 INFO:teuthology.orchestra.run.smithi047.stdout: osd op complaint time = 180 2024-02-19T15:21:06.938 INFO:teuthology.orchestra.run.smithi047.stdout: rados mon op timeout = 900 2024-02-19T15:21:06.938 INFO:teuthology.orchestra.run.smithi047.stdout: rados osd op timeout = 900 2024-02-19T15:21:06.938 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-02-19T15:21:06.938 INFO:teuthology.orchestra.run.smithi116.stdout: chdir = "" 2024-02-19T15:21:06.938 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-19T15:21:06.938 INFO:teuthology.orchestra.run.smithi116.stdout: auth supported = cephx 2024-02-19T15:21:06.938 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.939 INFO:teuthology.orchestra.run.smithi116.stdout: filestore xattr use omap = true 2024-02-19T15:21:06.939 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.939 INFO:teuthology.orchestra.run.smithi116.stdout: mon clock drift allowed = 1.000 2024-02-19T15:21:06.939 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.939 INFO:teuthology.orchestra.run.smithi116.stdout: osd crush chooseleaf type = 0 2024-02-19T15:21:06.939 INFO:teuthology.orchestra.run.smithi116.stdout: auth debug = true 2024-02-19T15:21:06.939 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.939 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on old message = true 2024-02-19T15:21:06.939 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on bug = true 2024-02-19T15:21:06.939 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.939 INFO:teuthology.orchestra.run.smithi116.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-19T15:21:06.939 INFO:teuthology.orchestra.run.smithi116.stdout: mon pg warn max object skew = 0 2024-02-19T15:21:06.939 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.939 INFO:teuthology.orchestra.run.smithi116.stdout: # disable pg_autoscaler by default for new pools 2024-02-19T15:21:06.939 INFO:teuthology.orchestra.run.smithi116.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-19T15:21:06.939 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.940 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default size = 2 2024-02-19T15:21:06.940 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.940 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow primary affinity = true 2024-02-19T15:21:06.940 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow pg remap = true 2024-02-19T15:21:06.940 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on legacy crush tunables = false 2024-02-19T15:21:06.940 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on crush straw calc version zero = false 2024-02-19T15:21:06.940 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on no sortbitwise = false 2024-02-19T15:21:06.940 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on osd down out interval zero = false 2024-02-19T15:21:06.940 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on too few osds = false 2024-02-19T15:21:06.940 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-19T15:21:06.940 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-19T15:21:06.940 INFO:teuthology.orchestra.run.smithi116.stdout: mon_allow_pool_size_one = true 2024-02-19T15:21:06.940 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.940 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-19T15:21:06.940 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.941 INFO:teuthology.orchestra.run.smithi116.stdout: osd default data pool replay window = 5 2024-02-19T15:21:06.941 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.941 INFO:teuthology.orchestra.run.smithi116.stdout: mon allow pool delete = true 2024-02-19T15:21:06.941 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.941 INFO:teuthology.orchestra.run.smithi116.stdout: mon cluster log file level = debug 2024-02-19T15:21:06.941 INFO:teuthology.orchestra.run.smithi116.stdout: debug asserts on shutdown = true 2024-02-19T15:21:06.941 INFO:teuthology.orchestra.run.smithi116.stdout: mon health detail to clog = false 2024-02-19T15:21:06.941 INFO:teuthology.orchestra.run.smithi116.stdout: mon host = "172.21.15.47,172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790]" 2024-02-19T15:21:06.941 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on skipped message = False 2024-02-19T15:21:06.941 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e 2024-02-19T15:21:06.941 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.941 INFO:teuthology.orchestra.run.smithi116.stdout:[osd] 2024-02-19T15:21:06.941 INFO:teuthology.orchestra.run.smithi116.stdout: osd journal size = 100 2024-02-19T15:21:06.942 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.942 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub load threshold = 5.0 2024-02-19T15:21:06.942 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub max interval = 600 2024-02-19T15:21:06.942 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock profile = high_recovery_ops 2024-02-19T15:21:06.942 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.942 INFO:teuthology.orchestra.run.smithi116.stdout: osd recover clone overlap = true 2024-02-19T15:21:06.942 INFO:teuthology.orchestra.run.smithi116.stdout: osd recovery max chunk = 1048576 2024-02-19T15:21:06.942 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.942 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug shutdown = true 2024-02-19T15:21:06.942 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug op order = true 2024-02-19T15:21:06.942 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify stray on activate = true 2024-02-19T15:21:06.942 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug trim objects = true 2024-02-19T15:21:06.942 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.942 INFO:teuthology.orchestra.run.smithi116.stdout: osd open classes on start = true 2024-02-19T15:21:06.942 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug pg log writeout = true 2024-02-19T15:21:06.942 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.943 INFO:teuthology.orchestra.run.smithi116.stdout: osd deep scrub update digest min age = 30 2024-02-19T15:21:06.943 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.943 INFO:teuthology.orchestra.run.smithi116.stdout: osd map max advance = 10 2024-02-19T15:21:06.943 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.943 INFO:teuthology.orchestra.run.smithi116.stdout: journal zero on create = true 2024-02-19T15:21:06.943 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.943 INFO:teuthology.orchestra.run.smithi116.stdout: filestore ondisk finisher threads = 3 2024-02-19T15:21:06.943 INFO:teuthology.orchestra.run.smithi116.stdout: filestore apply finisher threads = 3 2024-02-19T15:21:06.943 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.943 INFO:teuthology.orchestra.run.smithi116.stdout: bdev debug aio = true 2024-02-19T15:21:06.943 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug misdirected ops = true 2024-02-19T15:21:06.943 INFO:teuthology.orchestra.run.smithi116.stdout: bdev async discard = True 2024-02-19T15:21:06.943 INFO:teuthology.orchestra.run.smithi116.stdout: bdev enable discard = True 2024-02-19T15:21:06.943 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore allocator = bitmap 2024-02-19T15:21:06.943 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore block size = 96636764160 2024-02-19T15:21:06.943 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore fsck on mount = True 2024-02-19T15:21:06.943 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluefs = 1/20 2024-02-19T15:21:06.944 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluestore = 1/20 2024-02-19T15:21:06.944 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:21:06.944 INFO:teuthology.orchestra.run.smithi116.stdout: debug osd = 20 2024-02-19T15:21:06.944 INFO:teuthology.orchestra.run.smithi116.stdout: debug rocksdb = 4/10 2024-02-19T15:21:06.944 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd backfillfull_ratio = 0.85 2024-02-19T15:21:06.944 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd full ratio = 0.9 2024-02-19T15:21:06.944 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd nearfull ratio = 0.8 2024-02-19T15:21:06.944 INFO:teuthology.orchestra.run.smithi116.stdout: osd failsafe full ratio = 0.95 2024-02-19T15:21:06.944 INFO:teuthology.orchestra.run.smithi116.stdout: osd objectstore = bluestore 2024-02-19T15:21:06.944 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2024-02-19T15:21:06.944 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.944 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr] 2024-02-19T15:21:06.944 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:21:06.944 INFO:teuthology.orchestra.run.smithi116.stdout: debug mgr = 20 2024-02-19T15:21:06.944 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-02-19T15:21:06.944 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-02-19T15:21:06.944 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:21:06.945 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:21:06.945 INFO:teuthology.orchestra.run.smithi116.stdout: mgr/telemetry/nag = false 2024-02-19T15:21:06.945 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.945 INFO:teuthology.orchestra.run.smithi116.stdout:[mon] 2024-02-19T15:21:06.945 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:21:06.945 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-02-19T15:21:06.945 INFO:teuthology.orchestra.run.smithi116.stdout: debug paxos = 20 2024-02-19T15:21:06.945 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-02-19T15:21:06.945 INFO:teuthology.orchestra.run.smithi116.stdout: mon data avail warn = 5 2024-02-19T15:21:06.945 INFO:teuthology.orchestra.run.smithi116.stdout: mon mgr mkfs grace = 240 2024-02-19T15:21:06.945 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:21:06.945 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd reporter subtree level = osd 2024-02-19T15:21:06.945 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd prime pg temp = true 2024-02-19T15:21:06.945 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:21:06.945 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.945 INFO:teuthology.orchestra.run.smithi116.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-19T15:21:06.945 INFO:teuthology.orchestra.run.smithi116.stdout: auth mon ticket ttl = 660 # 11m 2024-02-19T15:21:06.946 INFO:teuthology.orchestra.run.smithi116.stdout: auth service ticket ttl = 240 # 4m 2024-02-19T15:21:06.946 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.946 INFO:teuthology.orchestra.run.smithi116.stdout: # don't complain about insecure global_id in the test suite 2024-02-19T15:21:06.946 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-19T15:21:06.946 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-19T15:21:06.946 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.946 INFO:teuthology.orchestra.run.smithi116.stdout: # 1m isn't quite enough 2024-02-19T15:21:06.946 INFO:teuthology.orchestra.run.smithi116.stdout: mon_down_mkfs_grace = 2m 2024-02-19T15:21:06.946 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.946 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_filestore_osds = false 2024-02-19T15:21:06.946 INFO:teuthology.orchestra.run.smithi116.stdout: mon op complaint time = 120 2024-02-19T15:21:06.946 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:21:06.946 INFO:teuthology.orchestra.run.smithi116.stdout:[client] 2024-02-19T15:21:06.946 INFO:teuthology.orchestra.run.smithi116.stdout: rgw cache enabled = true 2024-02-19T15:21:06.946 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable ops log = true 2024-02-19T15:21:06.947 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable usage log = true 2024-02-19T15:21:06.947 INFO:teuthology.orchestra.run.smithi116.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-19T15:21:06.947 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-19T15:21:06.947 INFO:teuthology.orchestra.run.smithi116.stdout: client mount timeout = 600 2024-02-19T15:21:06.947 INFO:teuthology.orchestra.run.smithi116.stdout: debug client = 20 2024-02-19T15:21:06.947 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:21:06.947 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = "" 2024-02-19T15:21:06.947 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2024-02-19T15:21:06.947 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2024-02-19T15:21:06.947 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.a] 2024-02-19T15:21:06.947 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.b] 2024-02-19T15:21:06.947 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.c] 2024-02-19T15:21:06.947 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a] 2024-02-19T15:21:06.947 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c] 2024-02-19T15:21:06.947 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b] 2024-02-19T15:21:06.947 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d] 2024-02-19T15:21:06.947 INFO:teuthology.orchestra.run.smithi116.stdout:[mds] 2024-02-19T15:21:06.947 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds = 20 2024-02-19T15:21:06.948 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds balancer = 20 2024-02-19T15:21:06.948 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:21:06.948 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug frag = True 2024-02-19T15:21:06.948 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug scatterstat = True 2024-02-19T15:21:06.948 INFO:teuthology.orchestra.run.smithi116.stdout: mds op complaint time = 180 2024-02-19T15:21:06.948 INFO:teuthology.orchestra.run.smithi116.stdout: mds verify scatter = True 2024-02-19T15:21:06.948 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2024-02-19T15:21:06.948 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2024-02-19T15:21:06.948 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2024-02-19T15:21:06.949 INFO:tasks.ceph:Creating admin key on mon.a... 2024-02-19T15:21:06.949 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-02-19T15:21:07.050 INFO:tasks.ceph:Copying monmap to all nodes... 2024-02-19T15:21:07.050 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:07.050 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-02-19T15:21:07.058 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:07.058 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-02-19T15:21:07.109 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi047.front.sepia.ceph.com 2024-02-19T15:21:07.109 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:07.109 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/ceph/ceph.keyring 2024-02-19T15:21:07.109 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-02-19T15:21:07.180 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:07.180 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-02-19T15:21:07.229 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi116.front.sepia.ceph.com 2024-02-19T15:21:07.229 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:07.229 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.keyring 2024-02-19T15:21:07.229 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-02-19T15:21:07.263 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:07.263 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-02-19T15:21:07.316 INFO:tasks.ceph:Setting up mon nodes... 2024-02-19T15:21:07.316 INFO:tasks.ceph:Setting up mgr nodes... 2024-02-19T15:21:07.317 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-02-19T15:21:07.371 INFO:teuthology.orchestra.run.smithi047.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-02-19T15:21:07.377 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-02-19T15:21:07.433 INFO:teuthology.orchestra.run.smithi116.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-02-19T15:21:07.445 INFO:tasks.ceph:Setting up mds nodes... 2024-02-19T15:21:07.446 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-02-19T15:21:07.501 INFO:teuthology.orchestra.run.smithi047.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-02-19T15:21:07.506 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-02-19T15:21:07.526 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-02-19T15:21:07.628 INFO:teuthology.orchestra.run.smithi047.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-02-19T15:21:07.632 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-02-19T15:21:07.652 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-02-19T15:21:07.710 INFO:teuthology.orchestra.run.smithi116.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-02-19T15:21:07.715 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-02-19T15:21:07.735 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-02-19T15:21:07.830 INFO:teuthology.orchestra.run.smithi116.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-02-19T15:21:07.835 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-02-19T15:21:07.854 INFO:tasks.ceph_client:Setting up client nodes... 2024-02-19T15:21:07.854 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-19T15:21:07.897 INFO:teuthology.orchestra.run.smithi047.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-02-19T15:21:07.914 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-02-19T15:21:08.002 INFO:teuthology.orchestra.run.smithi047.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-02-19T15:21:08.019 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-19T15:21:08.060 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-02-19T15:21:08.077 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-02-19T15:21:08.161 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-02-19T15:21:08.177 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-02-19T15:21:08.177 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi047.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-02-19T15:21:08.177 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-02-19T15:21:08.196 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-02-19T15:21:08.196 INFO:tasks.ceph:role: osd.0 2024-02-19T15:21:08.197 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi047.front.sepia.ceph.com 2024-02-19T15:21:08.197 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-02-19T15:21:08.259 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-02-19T15:21:08.260 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-19T15:21:08.260 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-19T15:21:08.260 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-02-19T15:21:08.260 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-19T15:21:08.260 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-02-19T15:21:08.260 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-19T15:21:08.260 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-02-19T15:21:08.260 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-19T15:21:08.260 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-19T15:21:08.286 INFO:teuthology.orchestra.run.smithi047.stdout:Discarding blocks...Done. 2024-02-19T15:21:08.289 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-02-19T15:21:08.289 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-02-19T15:21:08.361 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-02-19T15:21:08.420 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: /sbin/restorecon: command not found 2024-02-19T15:21:08.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:21:08.422 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-02-19T15:21:08.488 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-02-19T15:21:08.488 INFO:tasks.ceph:role: osd.1 2024-02-19T15:21:08.488 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi047.front.sepia.ceph.com 2024-02-19T15:21:08.488 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-02-19T15:21:08.547 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-02-19T15:21:08.547 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-19T15:21:08.548 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-19T15:21:08.548 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-02-19T15:21:08.548 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-19T15:21:08.548 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-02-19T15:21:08.548 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-19T15:21:08.548 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-02-19T15:21:08.548 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-19T15:21:08.548 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-19T15:21:08.575 INFO:teuthology.orchestra.run.smithi047.stdout:Discarding blocks...Done. 2024-02-19T15:21:08.577 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-02-19T15:21:08.577 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-02-19T15:21:08.645 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-02-19T15:21:08.700 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: /sbin/restorecon: command not found 2024-02-19T15:21:08.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:21:08.701 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-02-19T15:21:08.764 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-02-19T15:21:08.764 INFO:tasks.ceph:role: osd.2 2024-02-19T15:21:08.764 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi047.front.sepia.ceph.com 2024-02-19T15:21:08.764 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-02-19T15:21:08.828 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-02-19T15:21:08.828 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-19T15:21:08.828 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-19T15:21:08.828 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-02-19T15:21:08.828 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-19T15:21:08.828 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-02-19T15:21:08.828 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-19T15:21:08.828 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-02-19T15:21:08.828 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-19T15:21:08.828 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-19T15:21:08.854 INFO:teuthology.orchestra.run.smithi047.stdout:Discarding blocks...Done. 2024-02-19T15:21:08.857 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-02-19T15:21:08.857 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-02-19T15:21:08.925 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-02-19T15:21:08.984 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: /sbin/restorecon: command not found 2024-02-19T15:21:08.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:21:08.985 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-02-19T15:21:09.048 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-02-19T15:21:09.048 INFO:tasks.ceph:role: osd.3 2024-02-19T15:21:09.048 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi047.front.sepia.ceph.com 2024-02-19T15:21:09.048 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-02-19T15:21:09.108 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-02-19T15:21:09.108 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-19T15:21:09.108 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-19T15:21:09.108 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-02-19T15:21:09.108 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-19T15:21:09.108 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-02-19T15:21:09.108 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-19T15:21:09.108 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-02-19T15:21:09.108 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-19T15:21:09.108 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-19T15:21:09.134 INFO:teuthology.orchestra.run.smithi047.stdout:Discarding blocks...Done. 2024-02-19T15:21:09.137 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-02-19T15:21:09.137 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-02-19T15:21:09.201 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-02-19T15:21:09.261 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: /sbin/restorecon: command not found 2024-02-19T15:21:09.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:21:09.262 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-19T15:21:09.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:09.376+0000 7f3adaa2e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-02-19T15:21:09.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:09.376+0000 7f3adaa2e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-02-19T15:21:09.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:09.376+0000 7f3adaa2e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-02-19T15:21:09.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:09.376+0000 7f3adaa2e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-02-19T15:21:09.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:09.376+0000 7f3adaa2e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-02-19T15:21:09.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:09.376+0000 7f3adaa2e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-02-19T15:21:12.360 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-19T15:21:12.381 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-19T15:21:12.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:12.476+0000 7f861f3947c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-02-19T15:21:12.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:12.476+0000 7f861f3947c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-02-19T15:21:12.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:12.476+0000 7f861f3947c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-02-19T15:21:12.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:12.476+0000 7f861f3947c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-02-19T15:21:12.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:12.476+0000 7f861f3947c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-02-19T15:21:12.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:12.476+0000 7f861f3947c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-02-19T15:21:15.464 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-19T15:21:15.486 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-19T15:21:15.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:15.580+0000 7f8f3f17e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-02-19T15:21:15.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:15.580+0000 7f8f3f17e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-02-19T15:21:15.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:15.580+0000 7f8f3f17e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-02-19T15:21:15.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:15.580+0000 7f8f3f17e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-02-19T15:21:15.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:15.580+0000 7f8f3f17e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-02-19T15:21:15.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:15.580+0000 7f8f3f17e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-02-19T15:21:18.560 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-19T15:21:18.581 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-19T15:21:18.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:18.676+0000 7fd3f5ccd7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-02-19T15:21:18.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:18.676+0000 7fd3f5ccd7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-02-19T15:21:18.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:18.676+0000 7fd3f5ccd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-02-19T15:21:18.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:18.676+0000 7fd3f5ccd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-02-19T15:21:18.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:18.676+0000 7fd3f5ccd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-02-19T15:21:18.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:18.676+0000 7fd3f5ccd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-02-19T15:21:21.673 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-19T15:21:21.695 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-02-19T15:21:21.717 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-02-19T15:21:21.717 INFO:tasks.ceph:role: osd.4 2024-02-19T15:21:21.717 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com 2024-02-19T15:21:21.717 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-02-19T15:21:21.791 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-02-19T15:21:21.792 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-19T15:21:21.792 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-19T15:21:21.792 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-02-19T15:21:21.792 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-19T15:21:21.792 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-02-19T15:21:21.792 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-19T15:21:21.792 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-02-19T15:21:21.792 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-19T15:21:21.792 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-19T15:21:22.253 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-02-19T15:21:22.256 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-02-19T15:21:22.256 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-02-19T15:21:22.374 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-02-19T15:21:22.391 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: /sbin/restorecon: command not found 2024-02-19T15:21:22.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:21:22.392 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-02-19T15:21:22.455 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-02-19T15:21:22.456 INFO:tasks.ceph:role: osd.5 2024-02-19T15:21:22.456 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com 2024-02-19T15:21:22.456 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-02-19T15:21:22.523 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-02-19T15:21:22.523 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-19T15:21:22.523 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-19T15:21:22.524 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-02-19T15:21:22.524 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-19T15:21:22.524 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-02-19T15:21:22.524 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-19T15:21:22.524 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-02-19T15:21:22.524 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-19T15:21:22.524 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-19T15:21:23.001 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-02-19T15:21:23.004 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-02-19T15:21:23.004 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-02-19T15:21:23.181 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-02-19T15:21:23.197 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: /sbin/restorecon: command not found 2024-02-19T15:21:23.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:21:23.198 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-02-19T15:21:23.264 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-02-19T15:21:23.264 INFO:tasks.ceph:role: osd.6 2024-02-19T15:21:23.264 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com 2024-02-19T15:21:23.264 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-02-19T15:21:23.336 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-02-19T15:21:23.336 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-19T15:21:23.336 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-19T15:21:23.336 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-02-19T15:21:23.336 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-19T15:21:23.336 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-02-19T15:21:23.336 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-19T15:21:23.336 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-02-19T15:21:23.336 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-19T15:21:23.336 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-19T15:21:23.812 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-02-19T15:21:23.815 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-02-19T15:21:23.815 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-02-19T15:21:23.926 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-02-19T15:21:23.942 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: /sbin/restorecon: command not found 2024-02-19T15:21:23.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:21:23.943 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-02-19T15:21:24.004 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-02-19T15:21:24.004 INFO:tasks.ceph:role: osd.7 2024-02-19T15:21:24.004 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi116.front.sepia.ceph.com 2024-02-19T15:21:24.005 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-02-19T15:21:24.075 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-02-19T15:21:24.075 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-19T15:21:24.075 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-19T15:21:24.075 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-02-19T15:21:24.076 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-19T15:21:24.076 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-02-19T15:21:24.076 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-19T15:21:24.076 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-02-19T15:21:24.076 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-19T15:21:24.076 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-19T15:21:24.511 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-02-19T15:21:24.514 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-02-19T15:21:24.514 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-02-19T15:21:24.608 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-02-19T15:21:24.624 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: /sbin/restorecon: command not found 2024-02-19T15:21:24.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:21:24.626 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-19T15:21:24.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:24.744+0000 7f2bfda2b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-02-19T15:21:24.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:24.744+0000 7f2bfda2b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-02-19T15:21:24.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:24.744+0000 7f2bfda2b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-02-19T15:21:24.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:24.744+0000 7f2bfda2b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-02-19T15:21:24.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:24.744+0000 7f2bfda2b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-02-19T15:21:24.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:24.748+0000 7f2bfda2b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-02-19T15:21:27.726 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-19T15:21:27.746 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-19T15:21:27.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:27.836+0000 7fb96e62b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-02-19T15:21:27.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:27.836+0000 7fb96e62b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-02-19T15:21:27.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:27.836+0000 7fb96e62b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-02-19T15:21:27.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:27.836+0000 7fb96e62b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-02-19T15:21:27.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:27.836+0000 7fb96e62b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-02-19T15:21:27.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:27.840+0000 7fb96e62b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-02-19T15:21:30.843 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-19T15:21:30.863 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-19T15:21:30.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:30.956+0000 7f38af13c7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-02-19T15:21:30.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:30.956+0000 7f38af13c7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-02-19T15:21:30.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:30.956+0000 7f38af13c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-02-19T15:21:30.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:30.956+0000 7f38af13c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-02-19T15:21:30.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:30.956+0000 7f38af13c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-02-19T15:21:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:30.960+0000 7f38af13c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-02-19T15:21:33.931 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-19T15:21:33.951 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-19T15:21:34.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:34.040+0000 7f3ac76597c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-02-19T15:21:34.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:34.040+0000 7f3ac76597c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-02-19T15:21:34.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:34.040+0000 7f3ac76597c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-02-19T15:21:34.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:34.040+0000 7f3ac76597c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-02-19T15:21:34.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:34.040+0000 7f3ac76597c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-02-19T15:21:34.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:34.044+0000 7f3ac76597c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-02-19T15:21:34.994 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:21:34.998 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:21:37.009 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-19T15:21:37.030 INFO:tasks.ceph:Reading keys from all nodes... 2024-02-19T15:21:37.030 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:37.030 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-02-19T15:21:37.053 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:37.053 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-02-19T15:21:37.117 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:37.117 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-02-19T15:21:37.181 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:37.181 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-02-19T15:21:37.245 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:37.246 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-02-19T15:21:37.309 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:37.309 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-02-19T15:21:37.373 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:37.373 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-02-19T15:21:37.437 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:37.437 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-02-19T15:21:37.458 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:37.458 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-02-19T15:21:37.520 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:37.520 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-02-19T15:21:37.584 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:37.584 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-02-19T15:21:37.648 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:37.648 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-02-19T15:21:37.712 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:37.712 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-02-19T15:21:37.780 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:37.780 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-02-19T15:21:37.844 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:37.844 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-02-19T15:21:37.853 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:37.853 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-02-19T15:21:37.905 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:37.905 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-02-19T15:21:37.914 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:37.914 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-02-19T15:21:37.964 INFO:tasks.ceph:Adding keys to all mons... 2024-02-19T15:21:37.964 DEBUG:teuthology.orchestra.run.smithi047:> sudo tee -a /etc/ceph/ceph.keyring 2024-02-19T15:21:37.968 DEBUG:teuthology.orchestra.run.smithi116:> sudo tee -a /etc/ceph/ceph.keyring 2024-02-19T15:21:38.012 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr.y] 2024-02-19T15:21:38.012 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDjcdNlwxs9FhAAWqP7YoKio/bbdx7SGiCJ9Q== 2024-02-19T15:21:38.012 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.a] 2024-02-19T15:21:38.012 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDjcdNl8BP+HRAAlJ3Jk+1k2XcZAnICzy++Ow== 2024-02-19T15:21:38.012 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.c] 2024-02-19T15:21:38.012 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDjcdNl48SHJRAAn4GCT5KnP+2PD8IXbGClCg== 2024-02-19T15:21:38.012 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.0] 2024-02-19T15:21:38.012 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDlcdNl/bycFhAAmzYi/OxEPUW3tcRJC0DNEQ== 2024-02-19T15:21:38.012 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.1] 2024-02-19T15:21:38.012 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDocdNluquhHBAA88aLHCvVol7JoZB0MtoutA== 2024-02-19T15:21:38.012 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.2] 2024-02-19T15:21:38.012 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDrcdNlGAnWIhAAyELvCPKjtfgF0+kX6NQFIA== 2024-02-19T15:21:38.013 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.3] 2024-02-19T15:21:38.013 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDucdNlROZ6KBAAxTCDuYsw5SRIZBcclzwxhg== 2024-02-19T15:21:38.013 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr.x] 2024-02-19T15:21:38.013 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDjcdNliM78GRAABWBg4hMcKjV2pK5WVBjLMQ== 2024-02-19T15:21:38.013 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.b] 2024-02-19T15:21:38.013 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDjcdNlIOh3KhAAlnU2ZFY4iv/XEw0WqxyZpw== 2024-02-19T15:21:38.013 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.d] 2024-02-19T15:21:38.013 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDjcdNljcekMRAA414655SzH43fBA0AtjpxZw== 2024-02-19T15:21:38.013 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.4] 2024-02-19T15:21:38.013 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQD0cdNluOGILBAA66R9RVXD2vdyIw+oZ+iM1g== 2024-02-19T15:21:38.013 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.5] 2024-02-19T15:21:38.013 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQD3cdNlHg8dMhAA2LUzfpvvHR4i8e5RBCeGpQ== 2024-02-19T15:21:38.013 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.6] 2024-02-19T15:21:38.013 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQD6cdNlXD1EORAAhKGIt0wFeCyoE6QkkZ3caA== 2024-02-19T15:21:38.013 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.7] 2024-02-19T15:21:38.013 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQD+cdNlfW65AhAAVWj3ptC8uQHPh/UXcb/uDg== 2024-02-19T15:21:38.014 INFO:teuthology.orchestra.run.smithi047.stdout:[client.0] 2024-02-19T15:21:38.014 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDjcdNlb7eRNRAAk6ZLGfjhcdlR9GoXFCd9eA== 2024-02-19T15:21:38.014 INFO:teuthology.orchestra.run.smithi047.stdout:[client.2] 2024-02-19T15:21:38.014 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDkcdNlyDk9ABAAIAjKSd4qJO5oHhVnDc1Rgw== 2024-02-19T15:21:38.014 INFO:teuthology.orchestra.run.smithi047.stdout:[client.1] 2024-02-19T15:21:38.014 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDkcdNlKBu+AxAAlLRU70SAs3ZJo5wFwRWa4w== 2024-02-19T15:21:38.014 INFO:teuthology.orchestra.run.smithi047.stdout:[client.3] 2024-02-19T15:21:38.014 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDkcdNlUqK3CRAABmSIzoJvl0ymME6swrvnSQ== 2024-02-19T15:21:38.026 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr.y] 2024-02-19T15:21:38.026 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQDjcdNlwxs9FhAAWqP7YoKio/bbdx7SGiCJ9Q== 2024-02-19T15:21:38.026 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a] 2024-02-19T15:21:38.026 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQDjcdNl8BP+HRAAlJ3Jk+1k2XcZAnICzy++Ow== 2024-02-19T15:21:38.026 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c] 2024-02-19T15:21:38.026 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQDjcdNl48SHJRAAn4GCT5KnP+2PD8IXbGClCg== 2024-02-19T15:21:38.026 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.0] 2024-02-19T15:21:38.026 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQDlcdNl/bycFhAAmzYi/OxEPUW3tcRJC0DNEQ== 2024-02-19T15:21:38.027 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.1] 2024-02-19T15:21:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQDocdNluquhHBAA88aLHCvVol7JoZB0MtoutA== 2024-02-19T15:21:38.027 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.2] 2024-02-19T15:21:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQDrcdNlGAnWIhAAyELvCPKjtfgF0+kX6NQFIA== 2024-02-19T15:21:38.027 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.3] 2024-02-19T15:21:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQDucdNlROZ6KBAAxTCDuYsw5SRIZBcclzwxhg== 2024-02-19T15:21:38.027 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr.x] 2024-02-19T15:21:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQDjcdNliM78GRAABWBg4hMcKjV2pK5WVBjLMQ== 2024-02-19T15:21:38.027 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b] 2024-02-19T15:21:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQDjcdNlIOh3KhAAlnU2ZFY4iv/XEw0WqxyZpw== 2024-02-19T15:21:38.027 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d] 2024-02-19T15:21:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQDjcdNljcekMRAA414655SzH43fBA0AtjpxZw== 2024-02-19T15:21:38.027 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.4] 2024-02-19T15:21:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQD0cdNluOGILBAA66R9RVXD2vdyIw+oZ+iM1g== 2024-02-19T15:21:38.027 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.5] 2024-02-19T15:21:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQD3cdNlHg8dMhAA2LUzfpvvHR4i8e5RBCeGpQ== 2024-02-19T15:21:38.027 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.6] 2024-02-19T15:21:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQD6cdNlXD1EORAAhKGIt0wFeCyoE6QkkZ3caA== 2024-02-19T15:21:38.028 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.7] 2024-02-19T15:21:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQD+cdNlfW65AhAAVWj3ptC8uQHPh/UXcb/uDg== 2024-02-19T15:21:38.028 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2024-02-19T15:21:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQDjcdNlb7eRNRAAk6ZLGfjhcdlR9GoXFCd9eA== 2024-02-19T15:21:38.028 INFO:teuthology.orchestra.run.smithi116.stdout:[client.2] 2024-02-19T15:21:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQDkcdNlyDk9ABAAIAjKSd4qJO5oHhVnDc1Rgw== 2024-02-19T15:21:38.028 INFO:teuthology.orchestra.run.smithi116.stdout:[client.1] 2024-02-19T15:21:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQDkcdNlKBu+AxAAlLRU70SAs3ZJo5wFwRWa4w== 2024-02-19T15:21:38.028 INFO:teuthology.orchestra.run.smithi116.stdout:[client.3] 2024-02-19T15:21:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQDkcdNlUqK3CRAABmSIzoJvl0ymME6swrvnSQ== 2024-02-19T15:21:38.029 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-19T15:21:38.060 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-19T15:21:38.132 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-19T15:21:38.135 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-19T15:21:38.194 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-19T15:21:38.197 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-19T15:21:38.253 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T15:21:38.256 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T15:21:38.353 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T15:21:38.356 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T15:21:38.409 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T15:21:38.413 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T15:21:38.495 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T15:21:38.498 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T15:21:38.550 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-19T15:21:38.553 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-19T15:21:38.605 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-19T15:21:38.608 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-19T15:21:38.663 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-19T15:21:38.666 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-19T15:21:38.722 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T15:21:38.725 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T15:21:38.777 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T15:21:38.780 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T15:21:38.832 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T15:21:38.835 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T15:21:38.887 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T15:21:38.890 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T15:21:38.943 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-19T15:21:38.946 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-19T15:21:39.002 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-19T15:21:39.005 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-19T15:21:39.061 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-19T15:21:39.064 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-19T15:21:39.119 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-19T15:21:39.122 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-19T15:21:39.177 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-02-19T15:21:39.177 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-02-19T15:21:39.197 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-02-19T15:21:39.309 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-02-19T15:21:39.329 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-02-19T15:21:39.349 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-02-19T15:21:39.456 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-02-19T15:21:39.475 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-02-19T15:21:39.534 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-02-19T15:21:39.641 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-02-19T15:21:39.660 DEBUG:teuthology.orchestra.run.smithi047:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-02-19T15:21:39.663 DEBUG:teuthology.orchestra.run.smithi116:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-02-19T15:21:39.711 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-02-19T15:21:39.711 INFO:tasks.ceph.mon.a:Restarting daemon 2024-02-19T15:21:39.712 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-02-19T15:21:39.714 INFO:tasks.ceph.mon.a:Started 2024-02-19T15:21:39.715 INFO:tasks.ceph.mon.b:Restarting daemon 2024-02-19T15:21:39.715 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-02-19T15:21:39.754 INFO:tasks.ceph.mon.b:Started 2024-02-19T15:21:39.755 INFO:tasks.ceph.mon.c:Restarting daemon 2024-02-19T15:21:39.755 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-02-19T15:21:39.758 INFO:tasks.ceph.mon.c:Started 2024-02-19T15:21:39.758 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-02-19T15:21:39.758 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-02-19T15:21:39.758 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-02-19T15:21:39.761 INFO:tasks.ceph.mgr.y:Started 2024-02-19T15:21:39.761 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-02-19T15:21:39.761 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-02-19T15:21:39.764 INFO:tasks.ceph.mgr.x:Started 2024-02-19T15:21:39.764 INFO:tasks.ceph:Setting crush tunables to default 2024-02-19T15:21:39.765 DEBUG:teuthology.orchestra.run.smithi047:> sudo ceph --cluster ceph osd crush tunables default 2024-02-19T15:21:39.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:39.900+0000 7f693ba00640 1 Processor -- start 2024-02-19T15:21:39.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:39.900+0000 7f693ba00640 1 -- start start 2024-02-19T15:21:39.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:39.904+0000 7f693ba00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f693415bd90 0x7f6934070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:39.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:39.904+0000 7f693ba00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6934072850 0x7f693406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:39.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:39.904+0000 7f693ba00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693415fc90 con 0x7f69340cb490 2024-02-19T15:21:39.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:39.904+0000 7f693ba00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f693415ff70 con 0x7f6934072850 2024-02-19T15:21:39.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:39.904+0000 7f693ba00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6934160250 con 0x7f693415bd90 2024-02-19T15:21:39.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:39.904+0000 7f6939a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6934072850 0x7f693406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:39.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:39.904+0000 7f6939a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6934072850 0x7f693406cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57434/0 (socket says 172.21.15.47:57434) 2024-02-19T15:21:39.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:39.904+0000 7f6939a00640 1 -- 172.21.15.47:0/1016170362 learned_addr learned my addr 172.21.15.47:0/1016170362 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:21:39.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:39.904+0000 7f6933e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f69340cb490 0x7f6934072140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34476/0 (socket says 172.21.15.47:34476) 2024-02-19T15:21:39.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:39.904+0000 7f6939000640 1 --2- 172.21.15.47:0/1016170362 >> v2:172.21.15.116:3300/0 conn(0x7f693415bd90 0x7f6934070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:39.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:39.904+0000 7f6939a00640 1 -- 172.21.15.47:0/1016170362 >> v2:172.21.15.47:3300/0 conn(0x7f6934072850 msgr2=0x7f693406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-02-19T15:21:39.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:39.904+0000 7f6939a00640 1 -- 172.21.15.47:0/1016170362 >> v2:172.21.15.47:3300/0 conn(0x7f6934072850 msgr2=0x7f693406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-19T15:21:39.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:39.904+0000 7f6939a00640 1 --2- 172.21.15.47:0/1016170362 >> v2:172.21.15.47:3300/0 conn(0x7f6934072850 0x7f693406cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-19T15:21:39.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:39.904+0000 7f6939a00640 1 --2- 172.21.15.47:0/1016170362 >> v2:172.21.15.47:3300/0 conn(0x7f6934072850 0x7f693406cc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T15:21:39.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:39.904+0000 7f6939000640 1 -- 172.21.15.47:0/1016170362 >> v2:172.21.15.116:3300/0 conn(0x7f693415bd90 msgr2=0x7f6934070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-02-19T15:21:39.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:39.904+0000 7f6939000640 1 -- 172.21.15.47:0/1016170362 >> v2:172.21.15.116:3300/0 conn(0x7f693415bd90 msgr2=0x7f6934070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-19T15:21:39.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:39.904+0000 7f6939000640 1 --2- 172.21.15.47:0/1016170362 >> v2:172.21.15.116:3300/0 conn(0x7f693415bd90 0x7f6934070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-19T15:21:39.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:39.904+0000 7f6939000640 1 --2- 172.21.15.47:0/1016170362 >> v2:172.21.15.116:3300/0 conn(0x7f693415bd90 0x7f6934070960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T15:21:40.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6933400640 1 -- 172.21.15.47:0/1016170362 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055514379 0 0) 0x7f693415fc90 con 0x7f69340cb490 2024-02-19T15:21:40.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6933400640 1 -- 172.21.15.47:0/1016170362 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f691c003530 con 0x7f69340cb490 2024-02-19T15:21:40.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6933400640 1 -- 172.21.15.47:0/1016170362 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1503831896 0 0) 0x7f691c003530 con 0x7f69340cb490 2024-02-19T15:21:40.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6933400640 1 -- 172.21.15.47:0/1016170362 >> v2:172.21.15.116:3300/0 conn(0x7f693415bd90 msgr2=0x7f6934070960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-19T15:21:40.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6933400640 1 --2- 172.21.15.47:0/1016170362 >> v2:172.21.15.116:3300/0 conn(0x7f693415bd90 0x7f6934070960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:40.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6933400640 1 -- 172.21.15.47:0/1016170362 >> v2:172.21.15.47:3300/0 conn(0x7f6934072850 msgr2=0x7f693406cc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-19T15:21:40.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6933400640 1 --2- 172.21.15.47:0/1016170362 >> v2:172.21.15.47:3300/0 conn(0x7f6934072850 0x7f693406cc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:40.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6933400640 1 -- 172.21.15.47:0/1016170362 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6934160530 con 0x7f69340cb490 2024-02-19T15:21:40.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6933400640 1 -- 172.21.15.47:0/1016170362 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f6920001680 con 0x7f69340cb490 2024-02-19T15:21:40.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6933400640 1 -- 172.21.15.47:0/1016170362 >> v1:172.21.15.47:6789/0 conn(0x7f69340cb490 legacy=0x7f6934072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:40.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6933400640 1 --2- 172.21.15.47:0/1016170362 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f691c0039d0 0x7f691c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:40.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6933400640 1 --2- 172.21.15.47:0/1016170362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f691c0047d0 0x7f691c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:40.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6933400640 1 --2- 172.21.15.47:0/1016170362 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f691c0052b0 0x7f691c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:40.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6933400640 1 -- 172.21.15.47:0/1016170362 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693415ff70 con 0x7f691c0052b0 2024-02-19T15:21:40.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6933400640 1 -- 172.21.15.47:0/1016170362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6934160250 con 0x7f691c0047d0 2024-02-19T15:21:40.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6933400640 1 -- 172.21.15.47:0/1016170362 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691c00c200 con 0x7f691c0039d0 2024-02-19T15:21:40.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6933e00640 1 --2- 172.21.15.47:0/1016170362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f691c0047d0 0x7f691c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6939000640 1 --2- 172.21.15.47:0/1016170362 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f691c0039d0 0x7f691c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6939a00640 1 --2- 172.21.15.47:0/1016170362 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f691c0052b0 0x7f691c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6939000640 1 -- 172.21.15.47:0/1016170362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f691c0047d0 msgr2=0x7f691c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:21:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6939000640 1 --2- 172.21.15.47:0/1016170362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f691c0047d0 0x7f691c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6939000640 1 -- 172.21.15.47:0/1016170362 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f691c0052b0 msgr2=0x7f691c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:21:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6939000640 1 --2- 172.21.15.47:0/1016170362 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f691c0052b0 0x7f691c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6939000640 1 -- 172.21.15.47:0/1016170362 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c003530 con 0x7f691c0039d0 2024-02-19T15:21:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.048+0000 7f6939a00640 1 -- 172.21.15.47:0/1016170362 reap_dead start 2024-02-19T15:21:40.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.052+0000 7f6939000640 1 --2- 172.21.15.47:0/1016170362 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f691c0039d0 0x7f691c003df0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f692800c820 tx=0x7f692801c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1a44ecd6dd4b34f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:40.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.052+0000 7f6933400640 1 -- 172.21.15.47:0/1016170362 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69280ab600 con 0x7f691c0039d0 2024-02-19T15:21:40.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.052+0000 7f6933400640 1 -- 172.21.15.47:0/1016170362 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69280a7050 con 0x7f691c0039d0 2024-02-19T15:21:40.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.052+0000 7f6933400640 1 -- 172.21.15.47:0/1016170362 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69280ab9a0 con 0x7f691c0039d0 2024-02-19T15:21:40.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.052+0000 7f693ba00640 1 -- 172.21.15.47:0/1016170362 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f691c0039d0 msgr2=0x7f691c003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:40.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.052+0000 7f693ba00640 1 --2- 172.21.15.47:0/1016170362 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f691c0039d0 0x7f691c003df0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f692800c820 tx=0x7f692801c510 comp rx=0 tx=0).stop 2024-02-19T15:21:40.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.052+0000 7f693ba00640 1 -- 172.21.15.47:0/1016170362 shutdown_connections 2024-02-19T15:21:40.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.052+0000 7f693ba00640 1 --2- 172.21.15.47:0/1016170362 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f691c0039d0 0x7f691c003df0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:40.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.052+0000 7f693ba00640 1 -- 172.21.15.47:0/1016170362 >> 172.21.15.47:0/1016170362 conn(0x7f69340cfc20 msgr2=0x7f69340d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:40.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.052+0000 7f693ba00640 1 -- 172.21.15.47:0/1016170362 shutdown_connections 2024-02-19T15:21:40.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.052+0000 7f693ba00640 1 -- 172.21.15.47:0/1016170362 wait complete. 2024-02-19T15:21:40.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.056+0000 7f693ba00640 1 Processor -- start 2024-02-19T15:21:40.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.056+0000 7f693ba00640 1 -- start start 2024-02-19T15:21:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.056+0000 7f693ba00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f691c0039d0 0x7f69341558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.056+0000 7f693ba00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6934155de0 0x7f69340c5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.056+0000 7f693ba00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69340c58b0 0x7f6934157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.056+0000 7f693ba00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69340c5d30 con 0x7f69340c58b0 2024-02-19T15:21:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.056+0000 7f693ba00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6934152ad0 con 0x7f691c0039d0 2024-02-19T15:21:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.056+0000 7f693ba00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69341584e0 con 0x7f6934155de0 2024-02-19T15:21:40.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.056+0000 7f6939000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f691c0039d0 0x7f69341558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:40.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.056+0000 7f6939000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f691c0039d0 0x7f69341558a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:41374/0 (socket says 172.21.15.47:41374) 2024-02-19T15:21:40.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.056+0000 7f6939000640 1 -- 172.21.15.47:0/2208575760 learned_addr learned my addr 172.21.15.47:0/2208575760 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:21:40.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.056+0000 7f6939a00640 1 --2- 172.21.15.47:0/2208575760 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69340c58b0 0x7f6934157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:40.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.056+0000 7f6933e00640 1 --2- 172.21.15.47:0/2208575760 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6934155de0 0x7f69340c5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:40.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.056+0000 7f6939000640 1 -- 172.21.15.47:0/2208575760 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6934155de0 msgr2=0x7f69340c5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:40.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.056+0000 7f6939000640 1 --2- 172.21.15.47:0/2208575760 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6934155de0 0x7f69340c5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:40.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.056+0000 7f6939000640 1 -- 172.21.15.47:0/2208575760 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69340c58b0 msgr2=0x7f6934157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:40.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.056+0000 7f6939000640 1 --2- 172.21.15.47:0/2208575760 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69340c58b0 0x7f6934157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:40.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.056+0000 7f6939000640 1 -- 172.21.15.47:0/2208575760 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69341587c0 con 0x7f691c0039d0 2024-02-19T15:21:40.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.056+0000 7f6939000640 1 --2- 172.21.15.47:0/2208575760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f691c0039d0 0x7f69341558a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f692801c340 tx=0x7f691c005800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:40.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.060+0000 7f6930c00640 1 -- 172.21.15.47:0/2208575760 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69280abc50 con 0x7f691c0039d0 2024-02-19T15:21:40.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.060+0000 7f6930c00640 1 -- 172.21.15.47:0/2208575760 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69280abdd0 con 0x7f691c0039d0 2024-02-19T15:21:40.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.060+0000 7f693ba00640 1 -- 172.21.15.47:0/2208575760 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6934158aa0 con 0x7f691c0039d0 2024-02-19T15:21:40.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.060+0000 7f6930c00640 1 -- 172.21.15.47:0/2208575760 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69280b6660 con 0x7f691c0039d0 2024-02-19T15:21:40.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.060+0000 7f693ba00640 1 -- 172.21.15.47:0/2208575760 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6934157db0 con 0x7f691c0039d0 2024-02-19T15:21:40.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.060+0000 7f6930c00640 1 -- 172.21.15.47:0/2208575760 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f69280b6b50 con 0x7f691c0039d0 2024-02-19T15:21:40.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.060+0000 7f6930c00640 1 -- 172.21.15.47:0/2208575760 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f69280be3c0 con 0x7f691c0039d0 2024-02-19T15:21:40.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.060+0000 7f693ba00640 1 -- 172.21.15.47:0/2208575760 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6934158c30 con 0x7f691c0039d0 2024-02-19T15:21:40.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.092+0000 7f6930c00640 1 -- 172.21.15.47:0/2208575760 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f69280c4060 con 0x7f691c0039d0 2024-02-19T15:21:40.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.156+0000 7f693ba00640 1 -- 172.21.15.47:0/2208575760 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f6934075360 con 0x7f691c0039d0 2024-02-19T15:21:40.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.164+0000 7f6930c00640 1 -- 172.21.15.47:0/2208575760 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f69280ba030 con 0x7f691c0039d0 2024-02-19T15:21:40.167 INFO:teuthology.orchestra.run.smithi047.stderr:adjusted tunables profile to default 2024-02-19T15:21:40.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.164+0000 7f693ba00640 1 -- 172.21.15.47:0/2208575760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f691c0039d0 msgr2=0x7f69341558a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:40.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.164+0000 7f693ba00640 1 --2- 172.21.15.47:0/2208575760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f691c0039d0 0x7f69341558a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f692801c340 tx=0x7f691c005800 comp rx=0 tx=0).stop 2024-02-19T15:21:40.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.164+0000 7f693ba00640 1 -- 172.21.15.47:0/2208575760 shutdown_connections 2024-02-19T15:21:40.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.164+0000 7f693ba00640 1 --2- 172.21.15.47:0/2208575760 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69340c58b0 0x7f6934157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:40.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.164+0000 7f693ba00640 1 --2- 172.21.15.47:0/2208575760 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6934155de0 0x7f69340c5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:40.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.164+0000 7f693ba00640 1 --2- 172.21.15.47:0/2208575760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f691c0039d0 0x7f69341558a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:40.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.164+0000 7f693ba00640 1 -- 172.21.15.47:0/2208575760 >> 172.21.15.47:0/2208575760 conn(0x7f69340cfc20 msgr2=0x7f6934158e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:40.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.164+0000 7f693ba00640 1 -- 172.21.15.47:0/2208575760 shutdown_connections 2024-02-19T15:21:40.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:40.164+0000 7f693ba00640 1 -- 172.21.15.47:0/2208575760 wait complete. 2024-02-19T15:21:40.185 INFO:tasks.ceph:check_enable_crimson: False 2024-02-19T15:21:40.186 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-02-19T15:21:40.186 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:40.186 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-02-19T15:21:40.205 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:40.205 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-02-19T15:21:40.269 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:40.269 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-02-19T15:21:40.292 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:40.288+0000 7f67eb02f180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-19T15:21:40.333 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:21:40.333 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-02-19T15:21:40.383 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:40.380+0000 7fad823d4180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-19T15:21:40.397 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:40.397 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-02-19T15:21:40.417 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:40.417 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-02-19T15:21:40.483 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:40.483 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-02-19T15:21:40.546 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:21:40.546 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-02-19T15:21:40.607 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new 98fb351e-d895-42c6-8746-924f0c4d0776 0 2024-02-19T15:21:40.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.780+0000 7f3480800640 1 Processor -- start 2024-02-19T15:21:40.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.780+0000 7f3480800640 1 -- start start 2024-02-19T15:21:40.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.780+0000 7f3480800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f34780c3ab0 0x7f3478072bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:40.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.780+0000 7f3480800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3478066cd0 con 0x7f34780727e0 2024-02-19T15:21:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.780+0000 7f3480800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34780d1300 con 0x7f34780ad0d0 2024-02-19T15:21:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.780+0000 7f3480800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34780b1040 con 0x7f34780c3ab0 2024-02-19T15:21:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.780+0000 7f347d400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f34780c3ab0 0x7f3478072bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.780+0000 7f347d400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f34780c3ab0 0x7f3478072bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33044/0 (socket says 172.21.15.116:33044) 2024-02-19T15:21:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.780+0000 7f347d400640 1 -- 172.21.15.116:0/3182770486 learned_addr learned my addr 172.21.15.116:0/3182770486 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-02-19T15:21:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.780+0000 7f347d400640 1 -- 172.21.15.116:0/3182770486 >> v1:172.21.15.116:6789/0 conn(0x7f34780ad0d0 legacy=0x7f34780ad520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.780+0000 7f347d400640 1 -- 172.21.15.116:0/3182770486 >> v1:172.21.15.47:6789/0 conn(0x7f34780727e0 legacy=0x7f34780b17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.780+0000 7f347d400640 1 -- 172.21.15.116:0/3182770486 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3478066f20 con 0x7f34780c3ab0 2024-02-19T15:21:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.784+0000 7f347d400640 1 --2- 172.21.15.116:0/3182770486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34780c3ab0 0x7f3478072bc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3468009870 tx=0x7f346801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a2f1b06cc72d1f2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.784+0000 7f347ca00640 1 -- 172.21.15.116:0/3182770486 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34680a6040 con 0x7f34780c3ab0 2024-02-19T15:21:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.784+0000 7f347ca00640 1 -- 172.21.15.116:0/3182770486 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34680aa600 con 0x7f34780c3ab0 2024-02-19T15:21:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.784+0000 7f347ca00640 1 -- 172.21.15.116:0/3182770486 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34680aa990 con 0x7f34780c3ab0 2024-02-19T15:21:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.784+0000 7f3480800640 1 -- 172.21.15.116:0/3182770486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34780c3ab0 msgr2=0x7f3478072bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.784+0000 7f3480800640 1 --2- 172.21.15.116:0/3182770486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34780c3ab0 0x7f3478072bc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3468009870 tx=0x7f346801c4a0 comp rx=0 tx=0).stop 2024-02-19T15:21:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.784+0000 7f3480800640 1 -- 172.21.15.116:0/3182770486 shutdown_connections 2024-02-19T15:21:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.784+0000 7f3480800640 1 --2- 172.21.15.116:0/3182770486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34780c3ab0 0x7f3478072bc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.784+0000 7f3480800640 1 -- 172.21.15.116:0/3182770486 >> 172.21.15.116:0/3182770486 conn(0x7f34780d0420 msgr2=0x7f34780d0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.784+0000 7f3480800640 1 -- 172.21.15.116:0/3182770486 shutdown_connections 2024-02-19T15:21:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.784+0000 7f3480800640 1 -- 172.21.15.116:0/3182770486 wait complete. 2024-02-19T15:21:40.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f3480800640 1 Processor -- start 2024-02-19T15:21:40.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f3480800640 1 -- start start 2024-02-19T15:21:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f3480800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34780727e0 0x7f34781522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f3480800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34780ad0d0 0x7f3478099200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f3480800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34780c3ab0 0x7f3478151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f3480800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34780cea00 con 0x7f34780c3ab0 2024-02-19T15:21:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f3480800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347814ce70 con 0x7f34780ad0d0 2024-02-19T15:21:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f3480800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3478099740 con 0x7f34780727e0 2024-02-19T15:21:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f347d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34780ad0d0 0x7f3478099200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f347d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34780ad0d0 0x7f3478099200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33054/0 (socket says 172.21.15.116:33054) 2024-02-19T15:21:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f347d400640 1 -- 172.21.15.116:0/601416812 learned_addr learned my addr 172.21.15.116:0/601416812 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-02-19T15:21:40.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f347de00640 1 --2- 172.21.15.116:0/601416812 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34780727e0 0x7f34781522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:40.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f347e800640 1 --2- 172.21.15.116:0/601416812 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34780c3ab0 0x7f3478151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:40.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f347d400640 1 -- 172.21.15.116:0/601416812 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34780727e0 msgr2=0x7f34781522d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:40.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f347d400640 1 --2- 172.21.15.116:0/601416812 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34780727e0 0x7f34781522d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:40.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f347d400640 1 -- 172.21.15.116:0/601416812 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34780c3ab0 msgr2=0x7f3478151570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:40.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f347d400640 1 --2- 172.21.15.116:0/601416812 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34780c3ab0 0x7f3478151570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:40.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f347d400640 1 -- 172.21.15.116:0/601416812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34680094d0 con 0x7f34780ad0d0 2024-02-19T15:21:40.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f347d400640 1 --2- 172.21.15.116:0/601416812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34780ad0d0 0x7f3478099200 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f346801a0d0 tx=0x7f3468019240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:40.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f3472000640 1 -- 172.21.15.116:0/601416812 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34680b3800 con 0x7f34780ad0d0 2024-02-19T15:21:40.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f3472000640 1 -- 172.21.15.116:0/601416812 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34680b3980 con 0x7f34780ad0d0 2024-02-19T15:21:40.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f3472000640 1 -- 172.21.15.116:0/601416812 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34680b3d30 con 0x7f34780ad0d0 2024-02-19T15:21:40.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.788+0000 7f3480800640 1 -- 172.21.15.116:0/601416812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34780cec50 con 0x7f34780ad0d0 2024-02-19T15:21:40.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.792+0000 7f3480800640 1 -- 172.21.15.116:0/601416812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34780cf210 con 0x7f34780ad0d0 2024-02-19T15:21:40.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.792+0000 7f3472000640 1 -- 172.21.15.116:0/601416812 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f34680bd3f0 con 0x7f34780ad0d0 2024-02-19T15:21:40.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.792+0000 7f3472000640 1 -- 172.21.15.116:0/601416812 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f34680b25b0 con 0x7f34780ad0d0 2024-02-19T15:21:40.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.792+0000 7f3480800640 1 -- 172.21.15.116:0/601416812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3478075360 con 0x7f34780ad0d0 2024-02-19T15:21:40.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.792+0000 7f3472000640 1 -- 172.21.15.116:0/601416812 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f34680c3060 con 0x7f34780ad0d0 2024-02-19T15:21:40.810 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:40.808+0000 7f67eb02f180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-19T15:21:40.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.860+0000 7f3480800640 1 -- 172.21.15.116:0/601416812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd new", "uuid": "98fb351e-d895-42c6-8746-924f0c4d0776", "id": 0} v 0) v1 -- 0x7f347806ba40 con 0x7f34780ad0d0 2024-02-19T15:21:40.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.864+0000 7f3472000640 1 -- 172.21.15.116:0/601416812 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "98fb351e-d895-42c6-8746-924f0c4d0776", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f34680bd7e0 con 0x7f34780ad0d0 2024-02-19T15:21:40.866 INFO:teuthology.orchestra.run.smithi116.stdout:0 2024-02-19T15:21:40.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.864+0000 7f3480800640 1 -- 172.21.15.116:0/601416812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34780ad0d0 msgr2=0x7f3478099200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:40.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.864+0000 7f3480800640 1 --2- 172.21.15.116:0/601416812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34780ad0d0 0x7f3478099200 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f346801a0d0 tx=0x7f3468019240 comp rx=0 tx=0).stop 2024-02-19T15:21:40.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.864+0000 7f3480800640 1 -- 172.21.15.116:0/601416812 shutdown_connections 2024-02-19T15:21:40.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.864+0000 7f3480800640 1 --2- 172.21.15.116:0/601416812 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34780c3ab0 0x7f3478151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:40.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.864+0000 7f3480800640 1 --2- 172.21.15.116:0/601416812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34780ad0d0 0x7f3478099200 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:40.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.864+0000 7f3480800640 1 --2- 172.21.15.116:0/601416812 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34780727e0 0x7f34781522d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:40.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.864+0000 7f3480800640 1 -- 172.21.15.116:0/601416812 >> 172.21.15.116:0/601416812 conn(0x7f34780d0420 msgr2=0x7f347806dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:40.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.864+0000 7f3480800640 1 -- 172.21.15.116:0/601416812 shutdown_connections 2024-02-19T15:21:40.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:40.864+0000 7f3480800640 1 -- 172.21.15.116:0/601416812 wait complete. 2024-02-19T15:21:40.885 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new 5ffa7095-5c4c-4975-b8c7-a3d9c6613018 1 2024-02-19T15:21:40.909 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:40.908+0000 7fad823d4180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-19T15:21:41.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.016+0000 7fc3f3a00640 1 Processor -- start 2024-02-19T15:21:41.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.016+0000 7fc3f3a00640 1 -- start start 2024-02-19T15:21:41.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.020+0000 7fc3f3a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc3ec1458c0 0x7fc3ec0cd400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.020+0000 7fc3f3a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc3ec06f430 0x7fc3ec06aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.020+0000 7fc3f3a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc3ec06b3f0 0x7fc3ec071850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.020+0000 7fc3f3a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3ec06b880 con 0x7fc3ec06b3f0 2024-02-19T15:21:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.020+0000 7fc3f3a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3ec0cdc30 con 0x7fc3ec1458c0 2024-02-19T15:21:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.020+0000 7fc3f3a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec0cfb50 con 0x7fc3ec06f430 2024-02-19T15:21:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.020+0000 7fc3f1000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc3ec1458c0 0x7fc3ec0cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.020+0000 7fc3ebe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc3ec06f430 0x7fc3ec06aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.020+0000 7fc3f1000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc3ec1458c0 0x7fc3ec0cd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39058/0 (socket says 172.21.15.116:39058) 2024-02-19T15:21:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.020+0000 7fc3f1000640 1 -- 172.21.15.116:0/3104960099 learned_addr learned my addr 172.21.15.116:0/3104960099 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-02-19T15:21:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.020+0000 7fc3f1a00640 1 --2- 172.21.15.116:0/3104960099 >> v2:172.21.15.47:3300/0 conn(0x7fc3ec06b3f0 0x7fc3ec071850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.020+0000 7fc3f1000640 1 -- 172.21.15.116:0/3104960099 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc3ec06f430 msgr2=0x7fc3ec06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:21:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.020+0000 7fc3f1000640 1 --2- 172.21.15.116:0/3104960099 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc3ec06f430 0x7fc3ec06aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.020+0000 7fc3f1000640 1 -- 172.21.15.116:0/3104960099 >> v2:172.21.15.47:3300/0 conn(0x7fc3ec06b3f0 msgr2=0x7fc3ec071850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:21:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.020+0000 7fc3f1000640 1 --2- 172.21.15.116:0/3104960099 >> v2:172.21.15.47:3300/0 conn(0x7fc3ec06b3f0 0x7fc3ec071850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.020+0000 7fc3f1000640 1 -- 172.21.15.116:0/3104960099 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ec06d810 con 0x7fc3ec1458c0 2024-02-19T15:21:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.020+0000 7fc3ebe00640 1 --2- 172.21.15.116:0/3104960099 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc3ec06f430 0x7fc3ec06aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:21:41.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3f1000640 1 --2- 172.21.15.116:0/3104960099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3ec1458c0 0x7fc3ec0cd400 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec0b5e50 tx=0x7fc3e001c1d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29beed23448a233c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:41.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3eb400640 1 -- 172.21.15.116:0/3104960099 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e00ab600 con 0x7fc3ec1458c0 2024-02-19T15:21:41.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3eb400640 1 -- 172.21.15.116:0/3104960099 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e00a7050 con 0x7fc3ec1458c0 2024-02-19T15:21:41.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3eb400640 1 -- 172.21.15.116:0/3104960099 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e00ab980 con 0x7fc3ec1458c0 2024-02-19T15:21:41.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3f3a00640 1 -- 172.21.15.116:0/3104960099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3ec1458c0 msgr2=0x7fc3ec0cd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3f3a00640 1 --2- 172.21.15.116:0/3104960099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3ec1458c0 0x7fc3ec0cd400 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec0b5e50 tx=0x7fc3e001c1d0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3f3a00640 1 -- 172.21.15.116:0/3104960099 shutdown_connections 2024-02-19T15:21:41.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3f3a00640 1 --2- 172.21.15.116:0/3104960099 >> v2:172.21.15.47:3300/0 conn(0x7fc3ec06b3f0 0x7fc3ec071850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3f3a00640 1 --2- 172.21.15.116:0/3104960099 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc3ec06f430 0x7fc3ec06aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3f3a00640 1 --2- 172.21.15.116:0/3104960099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3ec1458c0 0x7fc3ec0cd400 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3f3a00640 1 -- 172.21.15.116:0/3104960099 >> 172.21.15.116:0/3104960099 conn(0x7fc3ec0b2640 msgr2=0x7fc3ec0b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:41.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3f3a00640 1 -- 172.21.15.116:0/3104960099 shutdown_connections 2024-02-19T15:21:41.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3f3a00640 1 -- 172.21.15.116:0/3104960099 wait complete. 2024-02-19T15:21:41.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3f3a00640 1 Processor -- start 2024-02-19T15:21:41.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3f3a00640 1 -- start start 2024-02-19T15:21:41.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3f3a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc3ec06b3f0 0x7fc3ec15ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3f3a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3ec06f430 0x7fc3ec1f6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3f3a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3ec1458c0 0x7fc3ec13e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3f3a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec15f470 con 0x7fc3ec1458c0 2024-02-19T15:21:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3f3a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec1f6e60 con 0x7fc3ec06f430 2024-02-19T15:21:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.024+0000 7fc3f3a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec15c350 con 0x7fc3ec06b3f0 2024-02-19T15:21:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3ebe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3ec06f430 0x7fc3ec1f6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3f1000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc3ec06b3f0 0x7fc3ec15ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3ebe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3ec06f430 0x7fc3ec1f6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39064/0 (socket says 172.21.15.116:39064) 2024-02-19T15:21:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3ebe00640 1 -- 172.21.15.116:0/4215026563 learned_addr learned my addr 172.21.15.116:0/4215026563 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-02-19T15:21:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3f1a00640 1 --2- 172.21.15.116:0/4215026563 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3ec1458c0 0x7fc3ec13e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3ebe00640 1 -- 172.21.15.116:0/4215026563 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc3ec06b3f0 msgr2=0x7fc3ec15ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3ebe00640 1 --2- 172.21.15.116:0/4215026563 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc3ec06b3f0 0x7fc3ec15ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3ebe00640 1 -- 172.21.15.116:0/4215026563 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3ec1458c0 msgr2=0x7fc3ec13e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3ebe00640 1 --2- 172.21.15.116:0/4215026563 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3ec1458c0 0x7fc3ec13e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3ebe00640 1 -- 172.21.15.116:0/4215026563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d8009420 con 0x7fc3ec06f430 2024-02-19T15:21:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3f1000640 1 --2- 172.21.15.116:0/4215026563 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc3ec06b3f0 0x7fc3ec15ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:21:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3f1a00640 1 --2- 172.21.15.116:0/4215026563 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3ec1458c0 0x7fc3ec13e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:21:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3ebe00640 1 --2- 172.21.15.116:0/4215026563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3ec06f430 0x7fc3ec1f6920 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc3d800ec10 tx=0x7fc3d800c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3e8c00640 1 -- 172.21.15.116:0/4215026563 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d800cc30 con 0x7fc3ec06f430 2024-02-19T15:21:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3f3a00640 1 -- 172.21.15.116:0/4215026563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e00094d0 con 0x7fc3ec06f430 2024-02-19T15:21:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3e8c00640 1 -- 172.21.15.116:0/4215026563 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d800cd90 con 0x7fc3ec06f430 2024-02-19T15:21:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3f3a00640 1 -- 172.21.15.116:0/4215026563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ec15cbf0 con 0x7fc3ec06f430 2024-02-19T15:21:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3e8c00640 1 -- 172.21.15.116:0/4215026563 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8013670 con 0x7fc3ec06f430 2024-02-19T15:21:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3e8c00640 1 -- 172.21.15.116:0/4215026563 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fc3d80059a0 con 0x7fc3ec06f430 2024-02-19T15:21:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3e8c00640 1 -- 172.21.15.116:0/4215026563 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fc3d80063d0 con 0x7fc3ec06f430 2024-02-19T15:21:41.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.028+0000 7fc3f3a00640 1 -- 172.21.15.116:0/4215026563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3ec075360 con 0x7fc3ec06f430 2024-02-19T15:21:41.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.032+0000 7fc3e8c00640 1 -- 172.21.15.116:0/4215026563 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc3d8021060 con 0x7fc3ec06f430 2024-02-19T15:21:41.060 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:41.056+0000 7f67eb02f180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-19T15:21:41.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.100+0000 7fc3f3a00640 1 -- 172.21.15.116:0/4215026563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5ffa7095-5c4c-4975-b8c7-a3d9c6613018", "id": 1} v 0) v1 -- 0x7fc3ec144eb0 con 0x7fc3ec06f430 2024-02-19T15:21:41.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.104+0000 7fc3e8c00640 1 -- 172.21.15.116:0/4215026563 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5ffa7095-5c4c-4975-b8c7-a3d9c6613018", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc3d802b0b0 con 0x7fc3ec06f430 2024-02-19T15:21:41.108 INFO:teuthology.orchestra.run.smithi116.stdout:1 2024-02-19T15:21:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.108+0000 7fc3f3a00640 1 -- 172.21.15.116:0/4215026563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3ec06f430 msgr2=0x7fc3ec1f6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.108+0000 7fc3f3a00640 1 --2- 172.21.15.116:0/4215026563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3ec06f430 0x7fc3ec1f6920 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc3d800ec10 tx=0x7fc3d800c520 comp rx=0 tx=0).stop 2024-02-19T15:21:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.108+0000 7fc3f3a00640 1 -- 172.21.15.116:0/4215026563 shutdown_connections 2024-02-19T15:21:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.108+0000 7fc3f3a00640 1 --2- 172.21.15.116:0/4215026563 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3ec1458c0 0x7fc3ec13e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.108+0000 7fc3f3a00640 1 --2- 172.21.15.116:0/4215026563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3ec06f430 0x7fc3ec1f6920 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.108+0000 7fc3f3a00640 1 --2- 172.21.15.116:0/4215026563 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc3ec06b3f0 0x7fc3ec15ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.108+0000 7fc3f3a00640 1 -- 172.21.15.116:0/4215026563 >> 172.21.15.116:0/4215026563 conn(0x7fc3ec0b2640 msgr2=0x7fc3ec144020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.108+0000 7fc3f3a00640 1 -- 172.21.15.116:0/4215026563 shutdown_connections 2024-02-19T15:21:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.108+0000 7fc3f3a00640 1 -- 172.21.15.116:0/4215026563 wait complete. 2024-02-19T15:21:41.110 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:41.108+0000 7fad823d4180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-19T15:21:41.127 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new dd82bd48-437f-45d7-9633-c8afd0ace368 2 2024-02-19T15:21:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.256+0000 7fd935c00640 1 Processor -- start 2024-02-19T15:21:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.256+0000 7fd935c00640 1 -- start start 2024-02-19T15:21:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.256+0000 7fd935c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd930159c60 0x7fd930070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.256+0000 7fd935c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd9300cb740 0x7fd9300caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.256+0000 7fd935c00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9300cbe70 con 0x7fd9300cafe0 2024-02-19T15:21:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.256+0000 7fd935c00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9300af750 con 0x7fd9300cb740 2024-02-19T15:21:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.256+0000 7fd935c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9300cc150 con 0x7fd930159c60 2024-02-19T15:21:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.260+0000 7fd92ea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd930159c60 0x7fd930070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.260+0000 7fd92ea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd930159c60 0x7fd930070be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:56002/0 (socket says 172.21.15.116:56002) 2024-02-19T15:21:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.260+0000 7fd92ea00640 1 -- 172.21.15.116:0/421516109 learned_addr learned my addr 172.21.15.116:0/421516109 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-02-19T15:21:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.260+0000 7fd92e000640 1 --2- 172.21.15.116:0/421516109 >> v2:172.21.15.116:3300/0 conn(0x7fd9300cb740 0x7fd9300caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.260+0000 7fd92ea00640 1 -- 172.21.15.116:0/421516109 >> v2:172.21.15.116:3300/0 conn(0x7fd9300cb740 msgr2=0x7fd9300caaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:21:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.260+0000 7fd92ea00640 1 --2- 172.21.15.116:0/421516109 >> v2:172.21.15.116:3300/0 conn(0x7fd9300cb740 0x7fd9300caaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.260+0000 7fd92ea00640 1 -- 172.21.15.116:0/421516109 >> v1:172.21.15.47:6789/0 conn(0x7fd9300cafe0 legacy=0x7fd9300cdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.260+0000 7fd92ea00640 1 -- 172.21.15.116:0/421516109 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9300cc430 con 0x7fd930159c60 2024-02-19T15:21:41.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.260+0000 7fd92ea00640 1 --2- 172.21.15.116:0/421516109 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd930159c60 0x7fd930070be0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd920009870 tx=0x7fd92001c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1a407bfc9654780 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:41.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.260+0000 7fd92d600640 1 -- 172.21.15.116:0/421516109 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9200ab670 con 0x7fd930159c60 2024-02-19T15:21:41.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.260+0000 7fd92d600640 1 -- 172.21.15.116:0/421516109 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9200ab7d0 con 0x7fd930159c60 2024-02-19T15:21:41.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.260+0000 7fd92d600640 1 -- 172.21.15.116:0/421516109 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9200abb60 con 0x7fd930159c60 2024-02-19T15:21:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd935c00640 1 -- 172.21.15.116:0/421516109 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd930159c60 msgr2=0x7fd930070be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd935c00640 1 --2- 172.21.15.116:0/421516109 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd930159c60 0x7fd930070be0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd920009870 tx=0x7fd92001c560 comp rx=0 tx=0).stop 2024-02-19T15:21:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd935c00640 1 -- 172.21.15.116:0/421516109 shutdown_connections 2024-02-19T15:21:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd935c00640 1 --2- 172.21.15.116:0/421516109 >> v2:172.21.15.116:3300/0 conn(0x7fd9300cb740 0x7fd9300caaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd935c00640 1 --2- 172.21.15.116:0/421516109 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd930159c60 0x7fd930070be0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd935c00640 1 -- 172.21.15.116:0/421516109 >> 172.21.15.116:0/421516109 conn(0x7fd9300cfdb0 msgr2=0x7fd9300d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd935c00640 1 -- 172.21.15.116:0/421516109 shutdown_connections 2024-02-19T15:21:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd935c00640 1 -- 172.21.15.116:0/421516109 wait complete. 2024-02-19T15:21:41.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd935c00640 1 Processor -- start 2024-02-19T15:21:41.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd935c00640 1 -- start start 2024-02-19T15:21:41.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd935c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9300cafe0 0x7fd93009a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd935c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9300cb740 0x7fd93009c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd935c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd930159c60 0x7fd930153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd935c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93009b060 con 0x7fd9300cb740 2024-02-19T15:21:41.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd935c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93009b340 con 0x7fd930159c60 2024-02-19T15:21:41.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd935c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93009b620 con 0x7fd9300cafe0 2024-02-19T15:21:41.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd92ea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9300cafe0 0x7fd93009a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd92f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd930159c60 0x7fd930153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd92ea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9300cafe0 0x7fd93009a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:56018/0 (socket says 172.21.15.116:56018) 2024-02-19T15:21:41.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd92ea00640 1 -- 172.21.15.116:0/3378668530 learned_addr learned my addr 172.21.15.116:0/3378668530 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-02-19T15:21:41.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd92e000640 1 --2- 172.21.15.116:0/3378668530 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9300cb740 0x7fd93009c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd92ea00640 1 -- 172.21.15.116:0/3378668530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd930159c60 msgr2=0x7fd930153bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd92ea00640 1 --2- 172.21.15.116:0/3378668530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd930159c60 0x7fd930153bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd92ea00640 1 -- 172.21.15.116:0/3378668530 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9300cb740 msgr2=0x7fd93009c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd92ea00640 1 --2- 172.21.15.116:0/3378668530 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9300cb740 0x7fd93009c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.264+0000 7fd92ea00640 1 -- 172.21.15.116:0/3378668530 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9200094d0 con 0x7fd9300cafe0 2024-02-19T15:21:41.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.268+0000 7fd92ea00640 1 --2- 172.21.15.116:0/3378668530 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9300cafe0 0x7fd93009a9d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd9200ad040 tx=0x7fd920019300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:41.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.268+0000 7fd926a00640 1 -- 172.21.15.116:0/3378668530 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9200b5810 con 0x7fd9300cafe0 2024-02-19T15:21:41.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.268+0000 7fd926a00640 1 -- 172.21.15.116:0/3378668530 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9200b59b0 con 0x7fd9300cafe0 2024-02-19T15:21:41.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.268+0000 7fd926a00640 1 -- 172.21.15.116:0/3378668530 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9200b5d60 con 0x7fd9300cafe0 2024-02-19T15:21:41.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.268+0000 7fd935c00640 1 -- 172.21.15.116:0/3378668530 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd93009b900 con 0x7fd9300cafe0 2024-02-19T15:21:41.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.268+0000 7fd935c00640 1 -- 172.21.15.116:0/3378668530 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd93009d720 con 0x7fd9300cafe0 2024-02-19T15:21:41.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.268+0000 7fd926a00640 1 -- 172.21.15.116:0/3378668530 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd9200bf7a0 con 0x7fd9300cafe0 2024-02-19T15:21:41.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.268+0000 7fd926a00640 1 -- 172.21.15.116:0/3378668530 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fd9200b4660 con 0x7fd9300cafe0 2024-02-19T15:21:41.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.268+0000 7fd935c00640 1 -- 172.21.15.116:0/3378668530 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd930068500 con 0x7fd9300cafe0 2024-02-19T15:21:41.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.272+0000 7fd926a00640 1 -- 172.21.15.116:0/3378668530 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd9200a7070 con 0x7fd9300cafe0 2024-02-19T15:21:41.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.336+0000 7fd935c00640 1 -- 172.21.15.116:0/3378668530 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd new", "uuid": "dd82bd48-437f-45d7-9633-c8afd0ace368", "id": 2} v 0) v1 -- 0x7fd930075360 con 0x7fd9300cafe0 2024-02-19T15:21:41.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.344+0000 7fd926a00640 1 -- 172.21.15.116:0/3378668530 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dd82bd48-437f-45d7-9633-c8afd0ace368", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd9200bfa50 con 0x7fd9300cafe0 2024-02-19T15:21:41.345 INFO:teuthology.orchestra.run.smithi116.stdout:2 2024-02-19T15:21:41.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.344+0000 7fd935c00640 1 -- 172.21.15.116:0/3378668530 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9300cafe0 msgr2=0x7fd93009a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.344+0000 7fd935c00640 1 --2- 172.21.15.116:0/3378668530 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9300cafe0 0x7fd93009a9d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd9200ad040 tx=0x7fd920019300 comp rx=0 tx=0).stop 2024-02-19T15:21:41.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.344+0000 7fd935c00640 1 -- 172.21.15.116:0/3378668530 shutdown_connections 2024-02-19T15:21:41.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.344+0000 7fd935c00640 1 --2- 172.21.15.116:0/3378668530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd930159c60 0x7fd930153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.344+0000 7fd935c00640 1 --2- 172.21.15.116:0/3378668530 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9300cb740 0x7fd93009c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.344+0000 7fd935c00640 1 --2- 172.21.15.116:0/3378668530 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9300cafe0 0x7fd93009a9d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.344+0000 7fd935c00640 1 -- 172.21.15.116:0/3378668530 >> 172.21.15.116:0/3378668530 conn(0x7fd9300cfdb0 msgr2=0x7fd9300b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:41.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.344+0000 7fd935c00640 1 -- 172.21.15.116:0/3378668530 shutdown_connections 2024-02-19T15:21:41.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.344+0000 7fd935c00640 1 -- 172.21.15.116:0/3378668530 wait complete. 2024-02-19T15:21:41.364 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new 07f72bca-9e3d-4b97-8a28-f7c711c1620a 3 2024-02-19T15:21:41.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.496+0000 7f8e26600640 1 Processor -- start 2024-02-19T15:21:41.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.496+0000 7f8e26600640 1 -- start start 2024-02-19T15:21:41.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.496+0000 7f8e26600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8e20164c10 0x7f8e20167000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.496+0000 7f8e26600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e2015cf80 con 0x7f8e200d1900 2024-02-19T15:21:41.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.496+0000 7f8e26600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e2006f950 con 0x7f8e2006aeb0 2024-02-19T15:21:41.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.496+0000 7f8e26600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e201608a0 con 0x7f8e20164c10 2024-02-19T15:21:41.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.496+0000 7f8e1fe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8e20164c10 0x7f8e20167000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.496+0000 7f8e1fe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8e20164c10 0x7f8e20167000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39088/0 (socket says 172.21.15.116:39088) 2024-02-19T15:21:41.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.496+0000 7f8e1fe00640 1 -- 172.21.15.116:0/524867247 learned_addr learned my addr 172.21.15.116:0/524867247 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-02-19T15:21:41.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.496+0000 7f8e1fe00640 1 -- 172.21.15.116:0/524867247 >> v1:172.21.15.116:6789/0 conn(0x7f8e2006aeb0 legacy=0x7f8e2006b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.496+0000 7f8e1fe00640 1 -- 172.21.15.116:0/524867247 >> v1:172.21.15.47:6789/0 conn(0x7f8e200d1900 legacy=0x7f8e2006f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.496+0000 7f8e1fe00640 1 -- 172.21.15.116:0/524867247 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e2015c3b0 con 0x7f8e20164c10 2024-02-19T15:21:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.500+0000 7f8e1fe00640 1 --2- 172.21.15.116:0/524867247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e20164c10 0x7f8e20167000 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8e18009e80 tx=0x7f8e1801c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de53ccd7a60f32fb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.500+0000 7f8e1e000640 1 -- 172.21.15.116:0/524867247 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e180a5810 con 0x7f8e20164c10 2024-02-19T15:21:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.500+0000 7f8e1e000640 1 -- 172.21.15.116:0/524867247 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e180a5970 con 0x7f8e20164c10 2024-02-19T15:21:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.500+0000 7f8e1e000640 1 -- 172.21.15.116:0/524867247 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e180a5d00 con 0x7f8e20164c10 2024-02-19T15:21:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.500+0000 7f8e26600640 1 -- 172.21.15.116:0/524867247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e20164c10 msgr2=0x7f8e20167000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.500+0000 7f8e26600640 1 --2- 172.21.15.116:0/524867247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e20164c10 0x7f8e20167000 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8e18009e80 tx=0x7f8e1801c950 comp rx=0 tx=0).stop 2024-02-19T15:21:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.500+0000 7f8e26600640 1 -- 172.21.15.116:0/524867247 shutdown_connections 2024-02-19T15:21:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.500+0000 7f8e26600640 1 --2- 172.21.15.116:0/524867247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e20164c10 0x7f8e20167000 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.500+0000 7f8e26600640 1 -- 172.21.15.116:0/524867247 >> 172.21.15.116:0/524867247 conn(0x7f8e200d0d30 msgr2=0x7f8e200d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.500+0000 7f8e26600640 1 -- 172.21.15.116:0/524867247 shutdown_connections 2024-02-19T15:21:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.500+0000 7f8e26600640 1 -- 172.21.15.116:0/524867247 wait complete. 2024-02-19T15:21:41.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e26600640 1 Processor -- start 2024-02-19T15:21:41.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e26600640 1 -- start start 2024-02-19T15:21:41.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e26600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e2006aeb0 0x7f8e200cf310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e26600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e200d1900 0x7f8e2015ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e26600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8e20164c10 0x7f8e20158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e26600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2015f140 con 0x7f8e20164c10 2024-02-19T15:21:41.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e26600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e200c52f0 con 0x7f8e200d1900 2024-02-19T15:21:41.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e26600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e200d0220 con 0x7f8e2006aeb0 2024-02-19T15:21:41.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e1f400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e2006aeb0 0x7f8e200cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e1ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e200d1900 0x7f8e2015ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e1f400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e2006aeb0 0x7f8e200cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:56030/0 (socket says 172.21.15.116:56030) 2024-02-19T15:21:41.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e1f400640 1 -- 172.21.15.116:0/3796417047 learned_addr learned my addr 172.21.15.116:0/3796417047 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-02-19T15:21:41.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e1fe00640 1 --2- 172.21.15.116:0/3796417047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8e20164c10 0x7f8e20158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e1ea00640 1 -- 172.21.15.116:0/3796417047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e2006aeb0 msgr2=0x7f8e200cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e1ea00640 1 --2- 172.21.15.116:0/3796417047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e2006aeb0 0x7f8e200cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e1ea00640 1 -- 172.21.15.116:0/3796417047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8e20164c10 msgr2=0x7f8e20158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e1ea00640 1 --2- 172.21.15.116:0/3796417047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8e20164c10 0x7f8e20158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e1ea00640 1 -- 172.21.15.116:0/3796417047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e18009b30 con 0x7f8e200d1900 2024-02-19T15:21:41.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e1f400640 1 --2- 172.21.15.116:0/3796417047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e2006aeb0 0x7f8e200cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:21:41.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e1ea00640 1 --2- 172.21.15.116:0/3796417047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e200d1900 0x7f8e2015ec00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c000f70 tx=0x7f8e0c00c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:41.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e17400640 1 -- 172.21.15.116:0/3796417047 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c0052d0 con 0x7f8e200d1900 2024-02-19T15:21:41.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e17400640 1 -- 172.21.15.116:0/3796417047 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0c005430 con 0x7f8e200d1900 2024-02-19T15:21:41.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e17400640 1 -- 172.21.15.116:0/3796417047 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c003840 con 0x7f8e200d1900 2024-02-19T15:21:41.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e26600640 1 -- 172.21.15.116:0/3796417047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e200cf980 con 0x7f8e200d1900 2024-02-19T15:21:41.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e26600640 1 -- 172.21.15.116:0/3796417047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e200cff40 con 0x7f8e200d1900 2024-02-19T15:21:41.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e17400640 1 -- 172.21.15.116:0/3796417047 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f8e0c0039a0 con 0x7f8e200d1900 2024-02-19T15:21:41.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.504+0000 7f8e17400640 1 -- 172.21.15.116:0/3796417047 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f8e0c011a20 con 0x7f8e200d1900 2024-02-19T15:21:41.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.508+0000 7f8e26600640 1 -- 172.21.15.116:0/3796417047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e20075360 con 0x7f8e200d1900 2024-02-19T15:21:41.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.508+0000 7f8e17400640 1 -- 172.21.15.116:0/3796417047 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f8e0c017070 con 0x7f8e200d1900 2024-02-19T15:21:41.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.576+0000 7f8e26600640 1 -- 172.21.15.116:0/3796417047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd new", "uuid": "07f72bca-9e3d-4b97-8a28-f7c711c1620a", "id": 3} v 0) v1 -- 0x7f8e200c5430 con 0x7f8e200d1900 2024-02-19T15:21:41.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.584+0000 7f8e17400640 1 -- 172.21.15.116:0/3796417047 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "07f72bca-9e3d-4b97-8a28-f7c711c1620a", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8e0c003c50 con 0x7f8e200d1900 2024-02-19T15:21:41.584 INFO:teuthology.orchestra.run.smithi116.stdout:3 2024-02-19T15:21:41.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.584+0000 7f8e26600640 1 -- 172.21.15.116:0/3796417047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e200d1900 msgr2=0x7f8e2015ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.584+0000 7f8e26600640 1 --2- 172.21.15.116:0/3796417047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e200d1900 0x7f8e2015ec00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c000f70 tx=0x7f8e0c00c890 comp rx=0 tx=0).stop 2024-02-19T15:21:41.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.584+0000 7f8e26600640 1 -- 172.21.15.116:0/3796417047 shutdown_connections 2024-02-19T15:21:41.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.584+0000 7f8e26600640 1 --2- 172.21.15.116:0/3796417047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8e20164c10 0x7f8e20158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.584+0000 7f8e26600640 1 --2- 172.21.15.116:0/3796417047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e200d1900 0x7f8e2015ec00 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.584+0000 7f8e26600640 1 --2- 172.21.15.116:0/3796417047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e2006aeb0 0x7f8e200cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.584+0000 7f8e26600640 1 -- 172.21.15.116:0/3796417047 >> 172.21.15.116:0/3796417047 conn(0x7f8e200d0d30 msgr2=0x7f8e200b0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:41.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.584+0000 7f8e26600640 1 -- 172.21.15.116:0/3796417047 shutdown_connections 2024-02-19T15:21:41.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.584+0000 7f8e26600640 1 -- 172.21.15.116:0/3796417047 wait complete. 2024-02-19T15:21:41.603 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new 9e9b38e5-8b24-4be2-b072-d46ae037c3e4 4 2024-02-19T15:21:41.688 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:41.684+0000 7f67eb02f180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-19T15:21:41.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.736+0000 7f72db200640 1 Processor -- start 2024-02-19T15:21:41.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.736+0000 7f72db200640 1 -- start start 2024-02-19T15:21:41.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.736+0000 7f72db200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72d40ceb60 0x7f72d40d0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.736+0000 7f72db200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d4160ea0 con 0x7f72d40ce780 2024-02-19T15:21:41.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.736+0000 7f72db200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d40d2840 con 0x7f72d40c6720 2024-02-19T15:21:41.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.736+0000 7f72db200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72d40cda20 con 0x7f72d40ceb60 2024-02-19T15:21:41.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.736+0000 7f72d3400640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f72d40c6720 0x7f72d4160790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58728/0 (socket says 172.21.15.116:58728) 2024-02-19T15:21:41.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.736+0000 7f72d3400640 1 -- 172.21.15.116:0/3439512952 learned_addr learned my addr 172.21.15.116:0/3439512952 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-02-19T15:21:41.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.736+0000 7f72d9200640 1 --2- 172.21.15.116:0/3439512952 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72d40ceb60 0x7f72d40d0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.736+0000 7f72d2a00640 1 -- 172.21.15.116:0/3439512952 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047187998 0 0) 0x7f72d40d2840 con 0x7f72d40c6720 2024-02-19T15:21:41.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.736+0000 7f72d2a00640 1 -- 172.21.15.116:0/3439512952 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72b8003530 con 0x7f72d40c6720 2024-02-19T15:21:41.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.736+0000 7f72d2a00640 1 -- 172.21.15.116:0/3439512952 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 947151309 0 0) 0x7f72d4160ea0 con 0x7f72d40ce780 2024-02-19T15:21:41.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.736+0000 7f72d9200640 1 -- 172.21.15.116:0/3439512952 >> v1:172.21.15.116:6789/0 conn(0x7f72d40c6720 legacy=0x7f72d4160790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.736+0000 7f72d9200640 1 -- 172.21.15.116:0/3439512952 >> v1:172.21.15.47:6789/0 conn(0x7f72d40ce780 legacy=0x7f72d4160080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.736+0000 7f72d9200640 1 -- 172.21.15.116:0/3439512952 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d40cd360 con 0x7f72d40ceb60 2024-02-19T15:21:41.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.740+0000 7f72d9200640 1 --2- 172.21.15.116:0/3439512952 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72d40ceb60 0x7f72d40d0710 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009dc0 tx=0x7f72cc01c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71a2ed0e38686cda server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:41.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.740+0000 7f72d2a00640 1 -- 172.21.15.116:0/3439512952 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc0a58d0 con 0x7f72d40ceb60 2024-02-19T15:21:41.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.740+0000 7f72d2a00640 1 -- 172.21.15.116:0/3439512952 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72cc0a5a30 con 0x7f72d40ceb60 2024-02-19T15:21:41.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.740+0000 7f72d2a00640 1 -- 172.21.15.116:0/3439512952 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc0a5dc0 con 0x7f72d40ceb60 2024-02-19T15:21:41.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.740+0000 7f72db200640 1 -- 172.21.15.116:0/3439512952 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72d40ceb60 msgr2=0x7f72d40d0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.740+0000 7f72db200640 1 --2- 172.21.15.116:0/3439512952 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72d40ceb60 0x7f72d40d0710 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009dc0 tx=0x7f72cc01c900 comp rx=0 tx=0).stop 2024-02-19T15:21:41.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.740+0000 7f72db200640 1 -- 172.21.15.116:0/3439512952 shutdown_connections 2024-02-19T15:21:41.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.740+0000 7f72db200640 1 --2- 172.21.15.116:0/3439512952 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72d40ceb60 0x7f72d40d0710 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72db200640 1 -- 172.21.15.116:0/3439512952 >> 172.21.15.116:0/3439512952 conn(0x7f72d40d1b30 msgr2=0x7f72d40d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:41.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72db200640 1 -- 172.21.15.116:0/3439512952 shutdown_connections 2024-02-19T15:21:41.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72db200640 1 -- 172.21.15.116:0/3439512952 wait complete. 2024-02-19T15:21:41.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72db200640 1 Processor -- start 2024-02-19T15:21:41.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72db200640 1 -- start start 2024-02-19T15:21:41.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72db200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72d40c6720 0x7f72d40c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72db200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72d40ce780 0x7f72d41581e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72db200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72d40ceb60 0x7f72d4160060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72db200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d4158720 con 0x7f72d40ceb60 2024-02-19T15:21:41.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72db200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d414b0e0 con 0x7f72d40ce780 2024-02-19T15:21:41.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72db200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72d414b3c0 con 0x7f72d40c6720 2024-02-19T15:21:41.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72d3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72d40ce780 0x7f72d41581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72d3e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72d40c6720 0x7f72d40c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72d3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72d40ce780 0x7f72d41581e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39118/0 (socket says 172.21.15.116:39118) 2024-02-19T15:21:41.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72d3400640 1 -- 172.21.15.116:0/1530306 learned_addr learned my addr 172.21.15.116:0/1530306 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-02-19T15:21:41.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72d9200640 1 --2- 172.21.15.116:0/1530306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72d40ceb60 0x7f72d4160060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72d3400640 1 -- 172.21.15.116:0/1530306 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72d40c6720 msgr2=0x7f72d40c9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72d3400640 1 --2- 172.21.15.116:0/1530306 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72d40c6720 0x7f72d40c9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72d3400640 1 -- 172.21.15.116:0/1530306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72d40ceb60 msgr2=0x7f72d4160060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72d3400640 1 --2- 172.21.15.116:0/1530306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72d40ceb60 0x7f72d4160060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72d3400640 1 -- 172.21.15.116:0/1530306 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72cc009a70 con 0x7f72d40ce780 2024-02-19T15:21:41.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72d3400640 1 --2- 172.21.15.116:0/1530306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72d40ce780 0x7f72d41581e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f72c000c3f0 tx=0x7f72c000c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:41.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72cbe00640 1 -- 172.21.15.116:0/1530306 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c0005cf0 con 0x7f72d40ce780 2024-02-19T15:21:41.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72cbe00640 1 -- 172.21.15.116:0/1530306 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c0005e50 con 0x7f72d40ce780 2024-02-19T15:21:41.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72cbe00640 1 -- 172.21.15.116:0/1530306 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c00072f0 con 0x7f72d40ce780 2024-02-19T15:21:41.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.744+0000 7f72db200640 1 -- 172.21.15.116:0/1530306 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72d414b6a0 con 0x7f72d40ce780 2024-02-19T15:21:41.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.748+0000 7f72db200640 1 -- 172.21.15.116:0/1530306 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72d414bc60 con 0x7f72d40ce780 2024-02-19T15:21:41.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.748+0000 7f72cbe00640 1 -- 172.21.15.116:0/1530306 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f72c0007450 con 0x7f72d40ce780 2024-02-19T15:21:41.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.748+0000 7f72cbe00640 1 -- 172.21.15.116:0/1530306 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f72c0007700 con 0x7f72d40ce780 2024-02-19T15:21:41.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.748+0000 7f72db200640 1 -- 172.21.15.116:0/1530306 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72d414be50 con 0x7f72d40ce780 2024-02-19T15:21:41.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.748+0000 7f72cbe00640 1 -- 172.21.15.116:0/1530306 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f72c0017070 con 0x7f72d40ce780 2024-02-19T15:21:41.752 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:41.748+0000 7fad823d4180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-19T15:21:41.755 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:41.752+0000 7f67eb02f180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-19T15:21:41.816 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:41.812+0000 7f67eb02f180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-19T15:21:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.816+0000 7f72db200640 1 -- 172.21.15.116:0/1530306 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9e9b38e5-8b24-4be2-b072-d46ae037c3e4", "id": 4} v 0) v1 -- 0x7f72d4075360 con 0x7f72d40ce780 2024-02-19T15:21:41.818 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:41.816+0000 7fad823d4180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-19T15:21:41.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.820+0000 7f72cbe00640 1 -- 172.21.15.116:0/1530306 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9e9b38e5-8b24-4be2-b072-d46ae037c3e4", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f72c001c5d0 con 0x7f72d40ce780 2024-02-19T15:21:41.824 INFO:teuthology.orchestra.run.smithi116.stdout:4 2024-02-19T15:21:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.824+0000 7f72db200640 1 -- 172.21.15.116:0/1530306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72d40ce780 msgr2=0x7f72d41581e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.824+0000 7f72db200640 1 --2- 172.21.15.116:0/1530306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72d40ce780 0x7f72d41581e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f72c000c3f0 tx=0x7f72c000c8b0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.824+0000 7f72db200640 1 -- 172.21.15.116:0/1530306 shutdown_connections 2024-02-19T15:21:41.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.824+0000 7f72db200640 1 --2- 172.21.15.116:0/1530306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72d40ceb60 0x7f72d4160060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.824+0000 7f72db200640 1 --2- 172.21.15.116:0/1530306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72d40ce780 0x7f72d41581e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.824+0000 7f72db200640 1 --2- 172.21.15.116:0/1530306 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72d40c6720 0x7f72d40c9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.824+0000 7f72db200640 1 -- 172.21.15.116:0/1530306 >> 172.21.15.116:0/1530306 conn(0x7f72d40d1b30 msgr2=0x7f72d406ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:41.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.824+0000 7f72db200640 1 -- 172.21.15.116:0/1530306 shutdown_connections 2024-02-19T15:21:41.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.824+0000 7f72db200640 1 -- 172.21.15.116:0/1530306 wait complete. 2024-02-19T15:21:41.843 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new f1585559-4b9e-4272-85d4-b8a5ecb0c1aa 5 2024-02-19T15:21:41.883 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:41.880+0000 7fad823d4180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-19T15:21:41.908 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:41.908+0000 7f67eb02f180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-19T15:21:41.970 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:41.968+0000 7f67eb02f180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-19T15:21:41.975 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:41.972+0000 7fad823d4180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-19T15:21:41.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.976+0000 7f81e3800640 1 Processor -- start 2024-02-19T15:21:41.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.976+0000 7f81e3800640 1 -- start start 2024-02-19T15:21:41.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.976+0000 7f81e3800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f81dc06b110 0x7f81dc162520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.976+0000 7f81e3800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81dc161ad0 con 0x7f81dc0b5700 2024-02-19T15:21:41.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.976+0000 7f81e3800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81dc162090 con 0x7f81dc15edf0 2024-02-19T15:21:41.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.976+0000 7f81e3800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81dc0af5b0 con 0x7f81dc06b110 2024-02-19T15:21:41.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.976+0000 7f81e0e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f81dc06b110 0x7f81dc162520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.976+0000 7f81e0e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f81dc06b110 0x7f81dc162520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39120/0 (socket says 172.21.15.116:39120) 2024-02-19T15:21:41.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.976+0000 7f81e0e00640 1 -- 172.21.15.116:0/3279180741 learned_addr learned my addr 172.21.15.116:0/3279180741 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-02-19T15:21:41.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.976+0000 7f81e0e00640 1 -- 172.21.15.116:0/3279180741 >> v1:172.21.15.116:6789/0 conn(0x7f81dc15edf0 legacy=0x7f81dc071850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.976+0000 7f81e0e00640 1 -- 172.21.15.116:0/3279180741 >> v1:172.21.15.47:6789/0 conn(0x7f81dc0b5700 legacy=0x7f81dc06b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.976+0000 7f81e0e00640 1 -- 172.21.15.116:0/3279180741 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc162a60 con 0x7f81dc06b110 2024-02-19T15:21:41.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.980+0000 7f81e0e00640 1 --2- 172.21.15.116:0/3279180741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81dc06b110 0x7f81dc162520 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f81d0009870 tx=0x7f81d001c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbcfa39a0abfe65a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:41.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.980+0000 7f81db400640 1 -- 172.21.15.116:0/3279180741 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d00ab650 con 0x7f81dc06b110 2024-02-19T15:21:41.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.980+0000 7f81db400640 1 -- 172.21.15.116:0/3279180741 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d00ab7b0 con 0x7f81dc06b110 2024-02-19T15:21:41.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.980+0000 7f81db400640 1 -- 172.21.15.116:0/3279180741 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d00abb40 con 0x7f81dc06b110 2024-02-19T15:21:41.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.980+0000 7f81e3800640 1 -- 172.21.15.116:0/3279180741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81dc06b110 msgr2=0x7f81dc162520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.980+0000 7f81e3800640 1 --2- 172.21.15.116:0/3279180741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81dc06b110 0x7f81dc162520 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f81d0009870 tx=0x7f81d001c560 comp rx=0 tx=0).stop 2024-02-19T15:21:41.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.980+0000 7f81e3800640 1 -- 172.21.15.116:0/3279180741 shutdown_connections 2024-02-19T15:21:41.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.980+0000 7f81e3800640 1 --2- 172.21.15.116:0/3279180741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81dc06b110 0x7f81dc162520 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.980+0000 7f81e3800640 1 -- 172.21.15.116:0/3279180741 >> 172.21.15.116:0/3279180741 conn(0x7f81dc0b2640 msgr2=0x7f81dc0b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:41.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.980+0000 7f81e3800640 1 -- 172.21.15.116:0/3279180741 shutdown_connections 2024-02-19T15:21:41.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.980+0000 7f81e3800640 1 -- 172.21.15.116:0/3279180741 wait complete. 2024-02-19T15:21:41.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.980+0000 7f81e3800640 1 Processor -- start 2024-02-19T15:21:41.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.980+0000 7f81e3800640 1 -- start start 2024-02-19T15:21:41.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81e3800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81dc06b110 0x7f81dc15b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81e3800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81dc0b5700 0x7f81dc15d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81e3800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81dc15edf0 0x7f81dc15bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:41.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81e3800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc15de00 con 0x7f81dc15edf0 2024-02-19T15:21:41.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81e3800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc1551c0 con 0x7f81dc06b110 2024-02-19T15:21:41.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81e3800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc154890 con 0x7f81dc0b5700 2024-02-19T15:21:41.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81e0e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81dc06b110 0x7f81dc15b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81e0e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81dc06b110 0x7f81dc15b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39134/0 (socket says 172.21.15.116:39134) 2024-02-19T15:21:41.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81e0e00640 1 -- 172.21.15.116:0/2775033170 learned_addr learned my addr 172.21.15.116:0/2775033170 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-02-19T15:21:41.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81dbe00640 1 --2- 172.21.15.116:0/2775033170 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81dc0b5700 0x7f81dc15d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81e1800640 1 --2- 172.21.15.116:0/2775033170 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81dc15edf0 0x7f81dc15bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:41.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81e0e00640 1 -- 172.21.15.116:0/2775033170 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81dc0b5700 msgr2=0x7f81dc15d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81e0e00640 1 --2- 172.21.15.116:0/2775033170 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81dc0b5700 0x7f81dc15d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81e0e00640 1 -- 172.21.15.116:0/2775033170 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81dc15edf0 msgr2=0x7f81dc15bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:41.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81e0e00640 1 --2- 172.21.15.116:0/2775033170 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81dc15edf0 0x7f81dc15bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:41.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81e0e00640 1 -- 172.21.15.116:0/2775033170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d00094d0 con 0x7f81dc06b110 2024-02-19T15:21:41.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81e0e00640 1 --2- 172.21.15.116:0/2775033170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81dc06b110 0x7f81dc15b3c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f81d00ad040 tx=0x7f81d001ada0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:41.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81d8c00640 1 -- 172.21.15.116:0/2775033170 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d00b4800 con 0x7f81dc06b110 2024-02-19T15:21:41.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81d8c00640 1 -- 172.21.15.116:0/2775033170 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d00b4980 con 0x7f81dc06b110 2024-02-19T15:21:41.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81e3800640 1 -- 172.21.15.116:0/2775033170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81dc154b70 con 0x7f81dc06b110 2024-02-19T15:21:41.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81d8c00640 1 -- 172.21.15.116:0/2775033170 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d00b4d30 con 0x7f81dc06b110 2024-02-19T15:21:41.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81e3800640 1 -- 172.21.15.116:0/2775033170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81dc150ea0 con 0x7f81dc06b110 2024-02-19T15:21:41.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81d8c00640 1 -- 172.21.15.116:0/2775033170 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f81d00be780 con 0x7f81dc06b110 2024-02-19T15:21:41.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81d8c00640 1 -- 172.21.15.116:0/2775033170 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f81d00c7390 con 0x7f81dc06b110 2024-02-19T15:21:41.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.984+0000 7f81e3800640 1 -- 172.21.15.116:0/2775033170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81dc0b4af0 con 0x7f81dc06b110 2024-02-19T15:21:41.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:41.988+0000 7f81d8c00640 1 -- 172.21.15.116:0/2775033170 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f81d00c78e0 con 0x7f81dc06b110 2024-02-19T15:21:42.039 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:42.036+0000 7fad823d4180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-19T15:21:42.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.056+0000 7f81e3800640 1 -- 172.21.15.116:0/2775033170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f1585559-4b9e-4272-85d4-b8a5ecb0c1aa", "id": 5} v 0) v1 -- 0x7f81dc068500 con 0x7f81dc06b110 2024-02-19T15:21:42.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.064+0000 7f81d8c00640 1 -- 172.21.15.116:0/2775033170 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f1585559-4b9e-4272-85d4-b8a5ecb0c1aa", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f81d00c7a80 con 0x7f81dc06b110 2024-02-19T15:21:42.065 INFO:teuthology.orchestra.run.smithi116.stdout:5 2024-02-19T15:21:42.065 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:42.064+0000 7f67eb02f180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-19T15:21:42.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.064+0000 7f81e3800640 1 -- 172.21.15.116:0/2775033170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81dc06b110 msgr2=0x7f81dc15b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:42.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.064+0000 7f81e3800640 1 --2- 172.21.15.116:0/2775033170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81dc06b110 0x7f81dc15b3c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f81d00ad040 tx=0x7f81d001ada0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.064+0000 7f81e3800640 1 -- 172.21.15.116:0/2775033170 shutdown_connections 2024-02-19T15:21:42.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.064+0000 7f81e3800640 1 --2- 172.21.15.116:0/2775033170 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81dc15edf0 0x7f81dc15bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.064+0000 7f81e3800640 1 --2- 172.21.15.116:0/2775033170 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81dc0b5700 0x7f81dc15d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.064+0000 7f81e3800640 1 --2- 172.21.15.116:0/2775033170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81dc06b110 0x7f81dc15b3c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.064+0000 7f81e3800640 1 -- 172.21.15.116:0/2775033170 >> 172.21.15.116:0/2775033170 conn(0x7f81dc0b2640 msgr2=0x7f81dc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:42.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.064+0000 7f81e3800640 1 -- 172.21.15.116:0/2775033170 shutdown_connections 2024-02-19T15:21:42.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.064+0000 7f81e3800640 1 -- 172.21.15.116:0/2775033170 wait complete. 2024-02-19T15:21:42.084 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new 10295c58-e6c4-494b-bc8f-5c6d085d7fc4 6 2024-02-19T15:21:42.131 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:42.128+0000 7f67eb02f180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-19T15:21:42.134 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:42.132+0000 7fad823d4180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-19T15:21:42.200 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:42.196+0000 7fad823d4180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-19T15:21:42.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.216+0000 7f432e400640 1 Processor -- start 2024-02-19T15:21:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.216+0000 7f432e400640 1 -- start start 2024-02-19T15:21:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.216+0000 7f432e400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f432806f430 0x7f432806aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.216+0000 7f432e400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43280cdc30 con 0x7f43281458c0 2024-02-19T15:21:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.216+0000 7f432e400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43280cfb50 con 0x7f432806b3f0 2024-02-19T15:21:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.216+0000 7f432e400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432806b880 con 0x7f432806f430 2024-02-19T15:21:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.216+0000 7f431ea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f432806f430 0x7f432806aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.216+0000 7f431ea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f432806f430 0x7f432806aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:56058/0 (socket says 172.21.15.116:56058) 2024-02-19T15:21:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.216+0000 7f4327e00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f432806b3f0 0x7f432806dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58740/0 (socket says 172.21.15.116:58740) 2024-02-19T15:21:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.216+0000 7f431ea00640 1 -- 172.21.15.116:0/1024810225 learned_addr learned my addr 172.21.15.116:0/1024810225 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-02-19T15:21:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.216+0000 7f431ea00640 1 -- 172.21.15.116:0/1024810225 >> v1:172.21.15.116:6789/0 conn(0x7f432806b3f0 legacy=0x7f432806dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.216+0000 7f431ea00640 1 -- 172.21.15.116:0/1024810225 >> v1:172.21.15.47:6789/0 conn(0x7f43281458c0 legacy=0x7f43280cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.216+0000 7f431ea00640 1 -- 172.21.15.116:0/1024810225 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432806d810 con 0x7f432806f430 2024-02-19T15:21:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.220+0000 7f431ea00640 1 --2- 172.21.15.116:0/1024810225 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f432806f430 0x7f432806aeb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f43100098a0 tx=0x7f431001c4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eac8366851bab506 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.220+0000 7f4326a00640 1 -- 172.21.15.116:0/1024810225 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43100ac640 con 0x7f432806f430 2024-02-19T15:21:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.220+0000 7f4326a00640 1 -- 172.21.15.116:0/1024810225 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43100ac7a0 con 0x7f432806f430 2024-02-19T15:21:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.220+0000 7f4326a00640 1 -- 172.21.15.116:0/1024810225 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43100acb30 con 0x7f432806f430 2024-02-19T15:21:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.220+0000 7f432e400640 1 -- 172.21.15.116:0/1024810225 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f432806f430 msgr2=0x7f432806aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.220+0000 7f432e400640 1 --2- 172.21.15.116:0/1024810225 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f432806f430 0x7f432806aeb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f43100098a0 tx=0x7f431001c4f0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.220+0000 7f432e400640 1 -- 172.21.15.116:0/1024810225 shutdown_connections 2024-02-19T15:21:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.220+0000 7f432e400640 1 --2- 172.21.15.116:0/1024810225 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f432806f430 0x7f432806aeb0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.220+0000 7f432e400640 1 -- 172.21.15.116:0/1024810225 >> 172.21.15.116:0/1024810225 conn(0x7f43280b2640 msgr2=0x7f43280b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.220+0000 7f432e400640 1 -- 172.21.15.116:0/1024810225 shutdown_connections 2024-02-19T15:21:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.220+0000 7f432e400640 1 -- 172.21.15.116:0/1024810225 wait complete. 2024-02-19T15:21:42.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f432e400640 1 Processor -- start 2024-02-19T15:21:42.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f432e400640 1 -- start start 2024-02-19T15:21:42.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f432e400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f432806b3f0 0x7f432815b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:42.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f432e400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f432806f430 0x7f432815ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:42.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f432e400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43281458c0 0x7f43280b1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:42.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f432e400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4328144800 con 0x7f43281458c0 2024-02-19T15:21:42.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f432e400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432815be90 con 0x7f432806f430 2024-02-19T15:21:42.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f432e400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4328143c80 con 0x7f432806b3f0 2024-02-19T15:21:42.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f4327400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f432806b3f0 0x7f432815b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:42.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f4327400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f432806b3f0 0x7f432815b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:56060/0 (socket says 172.21.15.116:56060) 2024-02-19T15:21:42.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f4327400640 1 -- 172.21.15.116:0/2062984979 learned_addr learned my addr 172.21.15.116:0/2062984979 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-02-19T15:21:42.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f431ea00640 1 --2- 172.21.15.116:0/2062984979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f432806f430 0x7f432815ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:42.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f4327e00640 1 --2- 172.21.15.116:0/2062984979 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43281458c0 0x7f43280b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:42.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f431ea00640 1 -- 172.21.15.116:0/2062984979 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f432806b3f0 msgr2=0x7f432815b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:42.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f431ea00640 1 --2- 172.21.15.116:0/2062984979 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f432806b3f0 0x7f432815b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f431ea00640 1 -- 172.21.15.116:0/2062984979 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43281458c0 msgr2=0x7f43280b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:42.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f431ea00640 1 --2- 172.21.15.116:0/2062984979 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43281458c0 0x7f43280b1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f431ea00640 1 -- 172.21.15.116:0/2062984979 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43100094d0 con 0x7f432806f430 2024-02-19T15:21:42.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f4327400640 1 --2- 172.21.15.116:0/2062984979 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f432806b3f0 0x7f432815b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:21:42.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f431ea00640 1 --2- 172.21.15.116:0/2062984979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f432806f430 0x7f432815ed40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f43100ac000 tx=0x7f431001ad30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:42.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f431fe00640 1 -- 172.21.15.116:0/2062984979 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43100b7860 con 0x7f432806f430 2024-02-19T15:21:42.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f431fe00640 1 -- 172.21.15.116:0/2062984979 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43100b7a00 con 0x7f432806f430 2024-02-19T15:21:42.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f431fe00640 1 -- 172.21.15.116:0/2062984979 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43100b7db0 con 0x7f432806f430 2024-02-19T15:21:42.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f432e400640 1 -- 172.21.15.116:0/2062984979 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4328143f60 con 0x7f432806f430 2024-02-19T15:21:42.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f432e400640 1 -- 172.21.15.116:0/2062984979 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4328144520 con 0x7f432806f430 2024-02-19T15:21:42.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.224+0000 7f431fe00640 1 -- 172.21.15.116:0/2062984979 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f43100b3070 con 0x7f432806f430 2024-02-19T15:21:42.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.228+0000 7f431fe00640 1 -- 172.21.15.116:0/2062984979 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f43100c0d80 con 0x7f432806f430 2024-02-19T15:21:42.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.228+0000 7f432e400640 1 -- 172.21.15.116:0/2062984979 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4328068500 con 0x7f432806f430 2024-02-19T15:21:42.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.228+0000 7f431fe00640 1 -- 172.21.15.116:0/2062984979 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f43100bb020 con 0x7f432806f430 2024-02-19T15:21:42.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.296+0000 7f432e400640 1 -- 172.21.15.116:0/2062984979 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd new", "uuid": "10295c58-e6c4-494b-bc8f-5c6d085d7fc4", "id": 6} v 0) v1 -- 0x7f43280cd770 con 0x7f432806f430 2024-02-19T15:21:42.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.300+0000 7f431fe00640 1 -- 172.21.15.116:0/2062984979 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "10295c58-e6c4-494b-bc8f-5c6d085d7fc4", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f43100b1070 con 0x7f432806f430 2024-02-19T15:21:42.301 INFO:teuthology.orchestra.run.smithi116.stdout:6 2024-02-19T15:21:42.301 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:42.300+0000 7f67eb02f180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-19T15:21:42.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.300+0000 7f432e400640 1 -- 172.21.15.116:0/2062984979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f432806f430 msgr2=0x7f432815ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:42.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.300+0000 7f432e400640 1 --2- 172.21.15.116:0/2062984979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f432806f430 0x7f432815ed40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f43100ac000 tx=0x7f431001ad30 comp rx=0 tx=0).stop 2024-02-19T15:21:42.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.300+0000 7f432e400640 1 -- 172.21.15.116:0/2062984979 shutdown_connections 2024-02-19T15:21:42.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.300+0000 7f432e400640 1 --2- 172.21.15.116:0/2062984979 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43281458c0 0x7f43280b1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.300+0000 7f432e400640 1 --2- 172.21.15.116:0/2062984979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f432806f430 0x7f432815ed40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.300+0000 7f432e400640 1 --2- 172.21.15.116:0/2062984979 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f432806b3f0 0x7f432815b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.300+0000 7f432e400640 1 -- 172.21.15.116:0/2062984979 >> 172.21.15.116:0/2062984979 conn(0x7f43280b2640 msgr2=0x7f43280b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:42.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.300+0000 7f432e400640 1 -- 172.21.15.116:0/2062984979 shutdown_connections 2024-02-19T15:21:42.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.300+0000 7f432e400640 1 -- 172.21.15.116:0/2062984979 wait complete. 2024-02-19T15:21:42.320 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new a472a117-2914-48bb-a15d-e07c9349d453 7 2024-02-19T15:21:42.374 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:42.372+0000 7fad823d4180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-19T15:21:42.376 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:42.376+0000 7f67eb02f180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-19T15:21:42.448 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:42.444+0000 7fad823d4180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-19T15:21:42.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.452+0000 7f49b2c00640 1 Processor -- start 2024-02-19T15:21:42.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.452+0000 7f49b2c00640 1 -- start start 2024-02-19T15:21:42.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.452+0000 7f49b2c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f49ac0ae290 0x7f49ac06c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:42.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.452+0000 7f49b2c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f49ac0acec0 0x7f49ac0ad2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:42.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.452+0000 7f49b2c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac0ad830 con 0x7f49ac09dc10 2024-02-19T15:21:42.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.452+0000 7f49b2c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49ac0ae670 con 0x7f49ac0ae290 2024-02-19T15:21:42.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.452+0000 7f49b2c00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49ac15c200 con 0x7f49ac0acec0 2024-02-19T15:21:42.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.452+0000 7f49b0c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f49ac0acec0 0x7f49ac0ad2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:42.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.452+0000 7f49b0c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f49ac0acec0 0x7f49ac0ad2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39148/0 (socket says 172.21.15.116:39148) 2024-02-19T15:21:42.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.452+0000 7f49b0c00640 1 -- 172.21.15.116:0/2017133742 learned_addr learned my addr 172.21.15.116:0/2017133742 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-02-19T15:21:42.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.452+0000 7f49abe00640 1 --2- 172.21.15.116:0/2017133742 >> v2:172.21.15.47:3300/0 conn(0x7f49ac0ae290 0x7f49ac06c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:42.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.452+0000 7f49b0c00640 1 -- 172.21.15.116:0/2017133742 >> v2:172.21.15.47:3300/0 conn(0x7f49ac0ae290 msgr2=0x7f49ac06c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:21:42.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.452+0000 7f49b0c00640 1 --2- 172.21.15.116:0/2017133742 >> v2:172.21.15.47:3300/0 conn(0x7f49ac0ae290 0x7f49ac06c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.452+0000 7f49b0c00640 1 -- 172.21.15.116:0/2017133742 >> v1:172.21.15.116:6789/0 conn(0x7f49ac09dc10 legacy=0x7f49ac06b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:42.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.452+0000 7f49b0c00640 1 -- 172.21.15.116:0/2017133742 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ac06bf40 con 0x7f49ac0acec0 2024-02-19T15:21:42.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.456+0000 7f49b0c00640 1 --2- 172.21.15.116:0/2017133742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac0acec0 0x7f49ac0ad2f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f49ac0c7230 tx=0x7f49a401c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ac6960cfc30b5ef server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:42.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.456+0000 7f49ab400640 1 -- 172.21.15.116:0/2017133742 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a40a5880 con 0x7f49ac0acec0 2024-02-19T15:21:42.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.456+0000 7f49ab400640 1 -- 172.21.15.116:0/2017133742 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a40a59e0 con 0x7f49ac0acec0 2024-02-19T15:21:42.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.456+0000 7f49ab400640 1 -- 172.21.15.116:0/2017133742 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a40a5d70 con 0x7f49ac0acec0 2024-02-19T15:21:42.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.456+0000 7f49b2c00640 1 -- 172.21.15.116:0/2017133742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac0acec0 msgr2=0x7f49ac0ad2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:42.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.456+0000 7f49b2c00640 1 --2- 172.21.15.116:0/2017133742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac0acec0 0x7f49ac0ad2f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f49ac0c7230 tx=0x7f49a401c9c0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.456+0000 7f49b2c00640 1 -- 172.21.15.116:0/2017133742 shutdown_connections 2024-02-19T15:21:42.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.456+0000 7f49b2c00640 1 --2- 172.21.15.116:0/2017133742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac0acec0 0x7f49ac0ad2f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.456+0000 7f49b2c00640 1 --2- 172.21.15.116:0/2017133742 >> v2:172.21.15.47:3300/0 conn(0x7f49ac0ae290 0x7f49ac06c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.456+0000 7f49b2c00640 1 -- 172.21.15.116:0/2017133742 >> 172.21.15.116:0/2017133742 conn(0x7f49ac0ccc50 msgr2=0x7f49ac0cd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:42.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.456+0000 7f49b2c00640 1 -- 172.21.15.116:0/2017133742 shutdown_connections 2024-02-19T15:21:42.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.456+0000 7f49b2c00640 1 -- 172.21.15.116:0/2017133742 wait complete. 2024-02-19T15:21:42.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49b2c00640 1 Processor -- start 2024-02-19T15:21:42.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49b2c00640 1 -- start start 2024-02-19T15:21:42.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49b2c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac09dc10 0x7f49ac0c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:42.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49b2c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49ac0acec0 0x7f49ac15da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:42.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49b2c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49ac0ae290 0x7f49ac0cf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:42.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49b2c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0c4b60 con 0x7f49ac0ae290 2024-02-19T15:21:42.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49b2c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac15dfb0 con 0x7f49ac09dc10 2024-02-19T15:21:42.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49b2c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0c4d90 con 0x7f49ac0acec0 2024-02-19T15:21:42.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49abe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac09dc10 0x7f49ac0c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:42.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49abe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac09dc10 0x7f49ac0c5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39150/0 (socket says 172.21.15.116:39150) 2024-02-19T15:21:42.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49abe00640 1 -- 172.21.15.116:0/1503889664 learned_addr learned my addr 172.21.15.116:0/1503889664 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-02-19T15:21:42.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49a3e00640 1 --2- 172.21.15.116:0/1503889664 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49ac0acec0 0x7f49ac15da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:42.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49b0c00640 1 --2- 172.21.15.116:0/1503889664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49ac0ae290 0x7f49ac0cf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:42.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49abe00640 1 -- 172.21.15.116:0/1503889664 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49ac0acec0 msgr2=0x7f49ac15da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:42.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49abe00640 1 --2- 172.21.15.116:0/1503889664 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49ac0acec0 0x7f49ac15da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49abe00640 1 -- 172.21.15.116:0/1503889664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49ac0ae290 msgr2=0x7f49ac0cf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:42.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49abe00640 1 --2- 172.21.15.116:0/1503889664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49ac0ae290 0x7f49ac0cf4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49abe00640 1 -- 172.21.15.116:0/1503889664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a4009b30 con 0x7f49ac09dc10 2024-02-19T15:21:42.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49abe00640 1 --2- 172.21.15.116:0/1503889664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac09dc10 0x7f49ac0c5e30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f499c00e8a0 tx=0x7f499c00eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:42.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49a8c00640 1 -- 172.21.15.116:0/1503889664 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c009510 con 0x7f49ac09dc10 2024-02-19T15:21:42.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49a8c00640 1 -- 172.21.15.116:0/1503889664 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499c00c940 con 0x7f49ac09dc10 2024-02-19T15:21:42.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49a8c00640 1 -- 172.21.15.116:0/1503889664 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c00ccb0 con 0x7f49ac09dc10 2024-02-19T15:21:42.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49b2c00640 1 -- 172.21.15.116:0/1503889664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ac0c5070 con 0x7f49ac09dc10 2024-02-19T15:21:42.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.460+0000 7f49b2c00640 1 -- 172.21.15.116:0/1503889664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ac0c5630 con 0x7f49ac09dc10 2024-02-19T15:21:42.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.464+0000 7f49a8c00640 1 -- 172.21.15.116:0/1503889664 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f499c0059a0 con 0x7f49ac09dc10 2024-02-19T15:21:42.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.464+0000 7f49a8c00640 1 -- 172.21.15.116:0/1503889664 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f499c00ce10 con 0x7f49ac09dc10 2024-02-19T15:21:42.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.464+0000 7f49b2c00640 1 -- 172.21.15.116:0/1503889664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ac075360 con 0x7f49ac09dc10 2024-02-19T15:21:42.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.464+0000 7f49a8c00640 1 -- 172.21.15.116:0/1503889664 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f499c021060 con 0x7f49ac09dc10 2024-02-19T15:21:42.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.548+0000 7f49b2c00640 1 -- 172.21.15.116:0/1503889664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a472a117-2914-48bb-a15d-e07c9349d453", "id": 7} v 0) v1 -- 0x7f49ac06ee60 con 0x7f49ac09dc10 2024-02-19T15:21:42.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.552+0000 7f49a8c00640 1 -- 172.21.15.116:0/1503889664 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a472a117-2914-48bb-a15d-e07c9349d453", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f499c029020 con 0x7f49ac09dc10 2024-02-19T15:21:42.554 INFO:teuthology.orchestra.run.smithi116.stdout:7 2024-02-19T15:21:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.552+0000 7f49b2c00640 1 -- 172.21.15.116:0/1503889664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac09dc10 msgr2=0x7f49ac0c5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.552+0000 7f49b2c00640 1 --2- 172.21.15.116:0/1503889664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac09dc10 0x7f49ac0c5e30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f499c00e8a0 tx=0x7f499c00eca0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.552+0000 7f49b2c00640 1 -- 172.21.15.116:0/1503889664 shutdown_connections 2024-02-19T15:21:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.552+0000 7f49b2c00640 1 --2- 172.21.15.116:0/1503889664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49ac0ae290 0x7f49ac0cf4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.552+0000 7f49b2c00640 1 --2- 172.21.15.116:0/1503889664 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49ac0acec0 0x7f49ac15da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.552+0000 7f49b2c00640 1 --2- 172.21.15.116:0/1503889664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac09dc10 0x7f49ac0c5e30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.552+0000 7f49b2c00640 1 -- 172.21.15.116:0/1503889664 >> 172.21.15.116:0/1503889664 conn(0x7f49ac0ccc50 msgr2=0x7f49ac0b2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:42.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.552+0000 7f49b2c00640 1 -- 172.21.15.116:0/1503889664 shutdown_connections 2024-02-19T15:21:42.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-02-19T15:21:42.552+0000 7f49b2c00640 1 -- 172.21.15.116:0/1503889664 wait complete. 2024-02-19T15:21:42.573 INFO:tasks.ceph.osd.0:Restarting daemon 2024-02-19T15:21:42.573 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-02-19T15:21:42.576 INFO:tasks.ceph.osd.0:Started 2024-02-19T15:21:42.576 INFO:tasks.ceph.osd.1:Restarting daemon 2024-02-19T15:21:42.576 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-02-19T15:21:42.579 INFO:tasks.ceph.osd.1:Started 2024-02-19T15:21:42.579 INFO:tasks.ceph.osd.2:Restarting daemon 2024-02-19T15:21:42.580 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-02-19T15:21:42.582 INFO:tasks.ceph.osd.2:Started 2024-02-19T15:21:42.583 INFO:tasks.ceph.osd.3:Restarting daemon 2024-02-19T15:21:42.583 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-02-19T15:21:42.586 INFO:tasks.ceph.osd.3:Started 2024-02-19T15:21:42.587 INFO:tasks.ceph.osd.4:Restarting daemon 2024-02-19T15:21:42.587 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-02-19T15:21:42.589 INFO:tasks.ceph.osd.4:Started 2024-02-19T15:21:42.590 INFO:tasks.ceph.osd.5:Restarting daemon 2024-02-19T15:21:42.590 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-02-19T15:21:42.592 INFO:tasks.ceph.osd.5:Started 2024-02-19T15:21:42.593 INFO:tasks.ceph.osd.6:Restarting daemon 2024-02-19T15:21:42.593 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-02-19T15:21:42.595 INFO:tasks.ceph.osd.6:Started 2024-02-19T15:21:42.596 INFO:tasks.ceph.osd.7:Restarting daemon 2024-02-19T15:21:42.596 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-02-19T15:21:42.600 INFO:tasks.ceph.osd.7:Started 2024-02-19T15:21:42.600 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:21:42.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.768+0000 7fb0b3e00640 1 Processor -- start 2024-02-19T15:21:42.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.768+0000 7fb0b3e00640 1 -- start start 2024-02-19T15:21:42.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.772+0000 7fb0b3e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb0b40aee90 0x7fb0b40709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:42.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.772+0000 7fb0b3e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b40712c0 con 0x7fb0b4070ee0 2024-02-19T15:21:42.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.772+0000 7fb0aa000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb0b40aee90 0x7fb0b40709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:42.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.772+0000 7fb0b2a00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb0b4159ce0 0x7fb0b40cae30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:45412/0 (socket says 172.21.15.47:45412) 2024-02-19T15:21:42.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.772+0000 7fb0b2a00640 1 -- 172.21.15.47:0/41717348 learned_addr learned my addr 172.21.15.47:0/41717348 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:21:42.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.772+0000 7fb0b3e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b40cb5f0 con 0x7fb0b4159ce0 2024-02-19T15:21:42.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.772+0000 7fb0b3e00640 1 -- 172.21.15.47:0/41717348 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b40cb8d0 con 0x7fb0b40aee90 2024-02-19T15:21:42.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.776+0000 7fb0b2000640 1 -- 172.21.15.47:0/41717348 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130580056 0 0) 0x7fb0b40712c0 con 0x7fb0b4070ee0 2024-02-19T15:21:42.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.776+0000 7fb0b2000640 1 -- 172.21.15.47:0/41717348 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb098003450 con 0x7fb0b4070ee0 2024-02-19T15:21:42.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.776+0000 7fb0aa000640 1 -- 172.21.15.47:0/41717348 >> v1:172.21.15.116:6789/0 conn(0x7fb0b4159ce0 legacy=0x7fb0b40cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:42.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.776+0000 7fb0aa000640 1 -- 172.21.15.47:0/41717348 >> v1:172.21.15.47:6789/0 conn(0x7fb0b4070ee0 legacy=0x7fb0b40cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:42.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.776+0000 7fb0aa000640 1 -- 172.21.15.47:0/41717348 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b40cd3a0 con 0x7fb0b40aee90 2024-02-19T15:21:42.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.780+0000 7fb0aa000640 1 --2- 172.21.15.47:0/41717348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b40aee90 0x7fb0b40709a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb0a40097c0 tx=0x7fb0a401c4c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78404f812b480387 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:42.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.784+0000 7fb0b2000640 1 -- 172.21.15.47:0/41717348 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a40a8050 con 0x7fb0b40aee90 2024-02-19T15:21:42.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.784+0000 7fb0b2000640 1 -- 172.21.15.47:0/41717348 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a40ac5f0 con 0x7fb0b40aee90 2024-02-19T15:21:42.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.784+0000 7fb0b2000640 1 -- 172.21.15.47:0/41717348 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a40ac980 con 0x7fb0b40aee90 2024-02-19T15:21:42.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.784+0000 7fb0b3e00640 1 -- 172.21.15.47:0/41717348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b40aee90 msgr2=0x7fb0b40709a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:42.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.784+0000 7fb0b3e00640 1 --2- 172.21.15.47:0/41717348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b40aee90 0x7fb0b40709a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb0a40097c0 tx=0x7fb0a401c4c0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.784+0000 7fb0b3e00640 1 -- 172.21.15.47:0/41717348 shutdown_connections 2024-02-19T15:21:42.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.784+0000 7fb0b3e00640 1 --2- 172.21.15.47:0/41717348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b40aee90 0x7fb0b40709a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.784+0000 7fb0b3e00640 1 -- 172.21.15.47:0/41717348 >> 172.21.15.47:0/41717348 conn(0x7fb0b406aeb0 msgr2=0x7fb0b406b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:42.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.784+0000 7fb0b3e00640 1 -- 172.21.15.47:0/41717348 shutdown_connections 2024-02-19T15:21:42.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.784+0000 7fb0b3e00640 1 -- 172.21.15.47:0/41717348 wait complete. 2024-02-19T15:21:42.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.788+0000 7fb0b3e00640 1 Processor -- start 2024-02-19T15:21:42.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.788+0000 7fb0b3e00640 1 -- start start 2024-02-19T15:21:42.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.788+0000 7fb0b3e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb0b4070ee0 0x7fb0b414b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:42.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.788+0000 7fb0b3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b40aee90 0x7fb0b406dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:42.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.788+0000 7fb0b3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0b4159ce0 0x7fb0b41515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:42.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0b2a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb0b4070ee0 0x7fb0b414b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:42.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0b2a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb0b4070ee0 0x7fb0b414b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:40456/0 (socket says 172.21.15.47:40456) 2024-02-19T15:21:42.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0b3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0b4159ce0 0x7fb0b41515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:42.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0b3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0b4159ce0 0x7fb0b41515b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57506/0 (socket says 172.21.15.47:57506) 2024-02-19T15:21:42.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0b2a00640 1 -- 172.21.15.47:0/3480231558 learned_addr learned my addr 172.21.15.47:0/3480231558 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:21:42.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0b3e00640 1 -- 172.21.15.47:0/3480231558 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b414b990 con 0x7fb0b4159ce0 2024-02-19T15:21:42.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0b3e00640 1 -- 172.21.15.47:0/3480231558 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b40afbe0 con 0x7fb0b40aee90 2024-02-19T15:21:42.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0b3e00640 1 -- 172.21.15.47:0/3480231558 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b40afec0 con 0x7fb0b4070ee0 2024-02-19T15:21:42.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0b2a00640 1 -- 172.21.15.47:0/3480231558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b40aee90 msgr2=0x7fb0b406dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:42.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0b2a00640 1 --2- 172.21.15.47:0/3480231558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b40aee90 0x7fb0b406dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0b2a00640 1 -- 172.21.15.47:0/3480231558 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0b4159ce0 msgr2=0x7fb0b41515b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:42.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0b2a00640 1 --2- 172.21.15.47:0/3480231558 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0b4159ce0 0x7fb0b41515b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0b2a00640 1 -- 172.21.15.47:0/3480231558 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a4009420 con 0x7fb0b4070ee0 2024-02-19T15:21:42.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0b3400640 1 --2- 172.21.15.47:0/3480231558 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0b4159ce0 0x7fb0b41515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:21:42.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0b2a00640 1 --2- 172.21.15.47:0/3480231558 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb0b4070ee0 0x7fb0b414b450 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00d620 tx=0x7fb09c00dae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:42.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0abe00640 1 -- 172.21.15.47:0/3480231558 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c0017d0 con 0x7fb0b4070ee0 2024-02-19T15:21:42.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0abe00640 1 -- 172.21.15.47:0/3480231558 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09c001930 con 0x7fb0b4070ee0 2024-02-19T15:21:42.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0abe00640 1 -- 172.21.15.47:0/3480231558 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c001d00 con 0x7fb0b4070ee0 2024-02-19T15:21:42.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0b3e00640 1 -- 172.21.15.47:0/3480231558 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b40b01a0 con 0x7fb0b4070ee0 2024-02-19T15:21:42.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0b3e00640 1 -- 172.21.15.47:0/3480231558 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b406d860 con 0x7fb0b4070ee0 2024-02-19T15:21:42.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.792+0000 7fb0abe00640 1 -- 172.21.15.47:0/3480231558 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fb09c019450 con 0x7fb0b4070ee0 2024-02-19T15:21:42.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.796+0000 7fb0abe00640 1 -- 172.21.15.47:0/3480231558 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fb09c009040 con 0x7fb0b4070ee0 2024-02-19T15:21:42.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.796+0000 7fb0b3e00640 1 -- 172.21.15.47:0/3480231558 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b4075360 con 0x7fb0b4070ee0 2024-02-19T15:21:42.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.800+0000 7fb0abe00640 1 -- 172.21.15.47:0/3480231558 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fb09c01e060 con 0x7fb0b4070ee0 2024-02-19T15:21:42.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.871+0000 7fb0b3e00640 1 -- 172.21.15.47:0/3480231558 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0b40740a0 con 0x7fb0b4070ee0 2024-02-19T15:21:42.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.875+0000 7fb0abe00640 1 -- 172.21.15.47:0/3480231558 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fb09c014070 con 0x7fb0b4070ee0 2024-02-19T15:21:42.878 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:42.878 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":10,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:21:42.552244+0000","last_up_change":"0.000000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:21:42.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.875+0000 7fb0b3e00640 1 -- 172.21.15.47:0/3480231558 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb0b4070ee0 msgr2=0x7fb0b414b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:42.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.875+0000 7fb0b3e00640 1 --2- 172.21.15.47:0/3480231558 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb0b4070ee0 0x7fb0b414b450 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00d620 tx=0x7fb09c00dae0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.875+0000 7fb0b3e00640 1 -- 172.21.15.47:0/3480231558 shutdown_connections 2024-02-19T15:21:42.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.875+0000 7fb0b3e00640 1 --2- 172.21.15.47:0/3480231558 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0b4159ce0 0x7fb0b41515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.875+0000 7fb0b3e00640 1 --2- 172.21.15.47:0/3480231558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b40aee90 0x7fb0b406dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.875+0000 7fb0b3e00640 1 --2- 172.21.15.47:0/3480231558 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb0b4070ee0 0x7fb0b414b450 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:42.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.875+0000 7fb0b3e00640 1 -- 172.21.15.47:0/3480231558 >> 172.21.15.47:0/3480231558 conn(0x7fb0b406aeb0 msgr2=0x7fb0b41572e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:42.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.875+0000 7fb0b3e00640 1 -- 172.21.15.47:0/3480231558 shutdown_connections 2024-02-19T15:21:42.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:42.875+0000 7fb0b3e00640 1 -- 172.21.15.47:0/3480231558 wait complete. 2024-02-19T15:21:42.896 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-02-19T15:21:42.897 INFO:tasks.ceph:Waiting for OSDs to come up 2024-02-19T15:21:43.041 INFO:tasks.ceph.mgr.x.smithi116.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-02-19T15:21:43.041 INFO:tasks.ceph.mgr.x.smithi116.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-02-19T15:21:43.041 INFO:tasks.ceph.mgr.x.smithi116.stderr: from numpy import show_config as show_numpy_config 2024-02-19T15:21:43.094 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:43.092+0000 7f67eb02f180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-19T15:21:43.108 INFO:tasks.ceph.mgr.y.smithi047.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-02-19T15:21:43.108 INFO:tasks.ceph.mgr.y.smithi047.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-02-19T15:21:43.108 INFO:tasks.ceph.mgr.y.smithi047.stderr: from numpy import show_config as show_numpy_config 2024-02-19T15:21:43.148 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:43.143+0000 7fad823d4180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-19T15:21:43.249 INFO:tasks.ceph.osd.7.smithi116.stderr:2024-02-19T15:21:43.248+0000 7f736949e7c0 -1 Falling back to public interface 2024-02-19T15:21:43.250 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-02-19T15:21:43.247+0000 7f5f3f7137c0 -1 Falling back to public interface 2024-02-19T15:21:43.254 INFO:tasks.ceph.osd.4.smithi116.stderr:2024-02-19T15:21:43.252+0000 7f90996047c0 -1 Falling back to public interface 2024-02-19T15:21:43.254 INFO:tasks.ceph.osd.5.smithi116.stderr:2024-02-19T15:21:43.252+0000 7f0f2d9967c0 -1 Falling back to public interface 2024-02-19T15:21:43.261 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-02-19T15:21:43.260+0000 7f73280d47c0 -1 Falling back to public interface 2024-02-19T15:21:43.262 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-02-19T15:21:43.259+0000 7fbb578f37c0 -1 Falling back to public interface 2024-02-19T15:21:43.262 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-02-19T15:21:43.259+0000 7f83c2c3b7c0 -1 Falling back to public interface 2024-02-19T15:21:43.263 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-02-19T15:21:43.259+0000 7fbb56400640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6812/0: (98) Address already in use 2024-02-19T15:21:43.270 INFO:tasks.ceph.osd.3.smithi047.stderr:2024-02-19T15:21:43.267+0000 7f73b1fda7c0 -1 Falling back to public interface 2024-02-19T15:21:43.290 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:43.288+0000 7f67eb02f180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-19T15:21:43.304 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:43.299+0000 7fad823d4180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-19T15:21:43.468 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:43.468+0000 7f67eb02f180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-19T15:21:43.483 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:43.479+0000 7fad823d4180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-19T15:21:43.541 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:43.540+0000 7f67eb02f180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-19T15:21:43.560 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:43.555+0000 7fad823d4180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-19T15:21:43.690 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:43.688+0000 7f67eb02f180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-19T15:21:43.703 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-02-19T15:21:43.717 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:43.715+0000 7fad823d4180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-19T15:21:43.836 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.831+0000 7f6571400640 1 Processor -- start 2024-02-19T15:21:43.836 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.831+0000 7f6571400640 1 -- start start 2024-02-19T15:21:43.838 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.835+0000 7f6571400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f656c15df70 0x7f656c0cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:43.838 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.835+0000 7f6571400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f656c06af90 0x7f656c06b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:43.838 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.835+0000 7f6571400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656c0ced30 con 0x7f656c15e350 2024-02-19T15:21:43.838 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.835+0000 7f6571400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f656c06b8e0 con 0x7f656c15df70 2024-02-19T15:21:43.838 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.835+0000 7f6571400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f656c0ce770 con 0x7f656c06af90 2024-02-19T15:21:43.838 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.835+0000 7f656a800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f656c15df70 0x7f656c0cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:43.838 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.835+0000 7f656a800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f656c15df70 0x7f656c0cefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44934/0 (socket says 172.21.15.47:44934) 2024-02-19T15:21:43.838 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.835+0000 7f656a800640 1 -- 172.21.15.47:0/1181791547 learned_addr learned my addr 172.21.15.47:0/1181791547 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:21:43.839 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.835+0000 7f656b200640 1 --2- 172.21.15.47:0/1181791547 >> v2:172.21.15.116:3300/0 conn(0x7f656c06af90 0x7f656c06b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:43.839 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.835+0000 7f656a800640 1 -- 172.21.15.47:0/1181791547 >> v2:172.21.15.116:3300/0 conn(0x7f656c06af90 msgr2=0x7f656c06b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:21:43.839 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.835+0000 7f656a800640 1 --2- 172.21.15.47:0/1181791547 >> v2:172.21.15.116:3300/0 conn(0x7f656c06af90 0x7f656c06b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:43.839 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.835+0000 7f656a800640 1 -- 172.21.15.47:0/1181791547 >> v1:172.21.15.116:6789/0 conn(0x7f656c15e350 legacy=0x7f656c0c6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:43.839 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.835+0000 7f656a800640 1 -- 172.21.15.47:0/1181791547 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656c0cea50 con 0x7f656c15df70 2024-02-19T15:21:43.841 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f656a800640 1 --2- 172.21.15.47:0/1181791547 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f656c15df70 0x7f656c0cefe0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f656c0b0600 tx=0x7f655c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64fcb8f983b6d474 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:43.841 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f6569e00640 1 -- 172.21.15.47:0/1181791547 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c0a8a10 con 0x7f656c15df70 2024-02-19T15:21:43.841 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f6569e00640 1 -- 172.21.15.47:0/1181791547 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f655c0a8b70 con 0x7f656c15df70 2024-02-19T15:21:43.841 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f6569e00640 1 -- 172.21.15.47:0/1181791547 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c0b0440 con 0x7f656c15df70 2024-02-19T15:21:43.842 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f6571400640 1 -- 172.21.15.47:0/1181791547 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f656c15df70 msgr2=0x7f656c0cefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:43.842 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f6571400640 1 --2- 172.21.15.47:0/1181791547 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f656c15df70 0x7f656c0cefe0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f656c0b0600 tx=0x7f655c01ed10 comp rx=0 tx=0).stop 2024-02-19T15:21:43.842 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f6571400640 1 -- 172.21.15.47:0/1181791547 shutdown_connections 2024-02-19T15:21:43.842 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f6571400640 1 --2- 172.21.15.47:0/1181791547 >> v2:172.21.15.116:3300/0 conn(0x7f656c06af90 0x7f656c06b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:43.842 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f6571400640 1 --2- 172.21.15.47:0/1181791547 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f656c15df70 0x7f656c0cefe0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:43.843 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f6571400640 1 -- 172.21.15.47:0/1181791547 >> 172.21.15.47:0/1181791547 conn(0x7f656c06c0a0 msgr2=0x7f656c06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:43.843 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f6571400640 1 -- 172.21.15.47:0/1181791547 shutdown_connections 2024-02-19T15:21:43.843 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f6571400640 1 -- 172.21.15.47:0/1181791547 wait complete. 2024-02-19T15:21:43.844 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f6571400640 1 Processor -- start 2024-02-19T15:21:43.844 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f6571400640 1 -- start start 2024-02-19T15:21:43.845 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f6571400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f656c06af90 0x7f656c0cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:43.845 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f6571400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f656c15df70 0x7f656c0b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:43.846 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f6571400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f656c15e350 0x7f656c150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:43.846 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f6571400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c0c7ad0 con 0x7f656c06af90 2024-02-19T15:21:43.846 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f6571400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c0c7db0 con 0x7f656c15e350 2024-02-19T15:21:43.846 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f6571400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f656c0c8090 con 0x7f656c15df70 2024-02-19T15:21:43.846 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.839+0000 7f656a800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f656c06af90 0x7f656c0cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:43.846 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.843+0000 7f656a800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f656c06af90 0x7f656c0cc8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44936/0 (socket says 172.21.15.47:44936) 2024-02-19T15:21:43.846 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.843+0000 7f656a800640 1 -- 172.21.15.47:0/61004760 learned_addr learned my addr 172.21.15.47:0/61004760 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:21:43.846 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.843+0000 7f6563e00640 1 --2- 172.21.15.47:0/61004760 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f656c15df70 0x7f656c0b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:43.846 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.843+0000 7f656b200640 1 --2- 172.21.15.47:0/61004760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f656c15e350 0x7f656c150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:43.846 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.843+0000 7f656a800640 1 -- 172.21.15.47:0/61004760 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f656c15df70 msgr2=0x7f656c0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:43.846 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.843+0000 7f656a800640 1 --2- 172.21.15.47:0/61004760 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f656c15df70 0x7f656c0b1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:43.846 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.843+0000 7f656a800640 1 -- 172.21.15.47:0/61004760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f656c15e350 msgr2=0x7f656c150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:43.846 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.843+0000 7f656a800640 1 --2- 172.21.15.47:0/61004760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f656c15e350 0x7f656c150010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:43.846 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.843+0000 7f656a800640 1 -- 172.21.15.47:0/61004760 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f655c0094d0 con 0x7f656c06af90 2024-02-19T15:21:43.846 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.843+0000 7f656b200640 1 --2- 172.21.15.47:0/61004760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f656c15e350 0x7f656c150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:21:43.847 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.843+0000 7f656a800640 1 --2- 172.21.15.47:0/61004760 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f656c06af90 0x7f656c0cc8e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f655c01b920 tx=0x7f655c01bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:43.848 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.843+0000 7f6562a00640 1 -- 172.21.15.47:0/61004760 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c0ba600 con 0x7f656c06af90 2024-02-19T15:21:43.848 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.843+0000 7f6562a00640 1 -- 172.21.15.47:0/61004760 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f655c0ba760 con 0x7f656c06af90 2024-02-19T15:21:43.848 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.843+0000 7f6562a00640 1 -- 172.21.15.47:0/61004760 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c0bab10 con 0x7f656c06af90 2024-02-19T15:21:43.848 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.843+0000 7f6571400640 1 -- 172.21.15.47:0/61004760 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656c0c8370 con 0x7f656c06af90 2024-02-19T15:21:43.848 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.843+0000 7f6571400640 1 -- 172.21.15.47:0/61004760 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656c0c8930 con 0x7f656c06af90 2024-02-19T15:21:43.848 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.843+0000 7f6562a00640 1 -- 172.21.15.47:0/61004760 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f655c0bac70 con 0x7f656c06af90 2024-02-19T15:21:43.848 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.843+0000 7f6562a00640 1 -- 172.21.15.47:0/61004760 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f655c0c3a90 con 0x7f656c06af90 2024-02-19T15:21:43.848 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.843+0000 7f6571400640 1 -- 172.21.15.47:0/61004760 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656c0684b0 con 0x7f656c06af90 2024-02-19T15:21:43.856 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.851+0000 7f6562a00640 1 -- 172.21.15.47:0/61004760 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f655c0b5070 con 0x7f656c06af90 2024-02-19T15:21:43.918 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.915+0000 7f6571400640 1 -- 172.21.15.47:0/61004760 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f656c0c8b20 con 0x7f656c06af90 2024-02-19T15:21:43.918 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.915+0000 7f6562a00640 1 -- 172.21.15.47:0/61004760 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f655c0c3da0 con 0x7f656c06af90 2024-02-19T15:21:43.919 INFO:teuthology.misc.health.smithi047.stdout: 2024-02-19T15:21:43.919 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:21:42.552244+0000","last_up_change":"0.000000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:21:43.920 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.915+0000 7f6571400640 1 -- 172.21.15.47:0/61004760 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f656c06af90 msgr2=0x7f656c0cc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:43.920 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.915+0000 7f6571400640 1 --2- 172.21.15.47:0/61004760 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f656c06af90 0x7f656c0cc8e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f655c01b920 tx=0x7f655c01bab0 comp rx=0 tx=0).stop 2024-02-19T15:21:43.920 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.915+0000 7f6571400640 1 -- 172.21.15.47:0/61004760 shutdown_connections 2024-02-19T15:21:43.920 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.915+0000 7f6571400640 1 --2- 172.21.15.47:0/61004760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f656c15e350 0x7f656c150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:43.920 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.915+0000 7f6571400640 1 --2- 172.21.15.47:0/61004760 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f656c15df70 0x7f656c0b1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:43.920 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.915+0000 7f6571400640 1 --2- 172.21.15.47:0/61004760 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f656c06af90 0x7f656c0cc8e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:43.920 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.915+0000 7f6571400640 1 -- 172.21.15.47:0/61004760 >> 172.21.15.47:0/61004760 conn(0x7f656c06c0a0 msgr2=0x7f656c0cba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:43.920 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.915+0000 7f6571400640 1 -- 172.21.15.47:0/61004760 shutdown_connections 2024-02-19T15:21:43.920 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:43.915+0000 7f6571400640 1 -- 172.21.15.47:0/61004760 wait complete. 2024-02-19T15:21:43.935 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-02-19T15:21:44.364 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:44.360+0000 7f67eb02f180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-19T15:21:44.446 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:44.443+0000 7fad823d4180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-19T15:21:44.449 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:44.448+0000 7f67eb02f180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-19T15:21:44.538 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:44.535+0000 7fad823d4180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-19T15:21:44.649 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:44.648+0000 7f67eb02f180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-19T15:21:44.733 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:44.732+0000 7f67eb02f180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-19T15:21:44.749 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:44.743+0000 7fad823d4180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-19T15:21:44.838 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:44.835+0000 7fad823d4180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-19T15:21:45.059 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:45.056+0000 7f67eb02f180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-19T15:21:45.130 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:45.128+0000 7f67eb02f180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-19T15:21:45.194 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:45.191+0000 7fad823d4180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-19T15:21:45.270 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-19T15:21:45.267+0000 7fad823d4180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-19T15:21:46.151 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:46.148+0000 7f6790200640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-02-19T15:21:46.151 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-19T15:21:46.148+0000 7f6790200640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-02-19T15:21:46.970 INFO:tasks.ceph.osd.7.smithi116.stderr:2024-02-19T15:21:46.968+0000 7f736949e7c0 -1 osd.7 0 log_to_monitors true 2024-02-19T15:21:47.014 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-02-19T15:21:47.011+0000 7f5f3f7137c0 -1 osd.1 0 log_to_monitors true 2024-02-19T15:21:47.105 INFO:tasks.ceph.osd.4.smithi116.stderr:2024-02-19T15:21:47.104+0000 7f90996047c0 -1 osd.4 0 log_to_monitors true 2024-02-19T15:21:47.124 INFO:tasks.ceph.osd.3.smithi047.stderr:2024-02-19T15:21:47.119+0000 7f73b1fda7c0 -1 osd.3 0 log_to_monitors true 2024-02-19T15:21:47.135 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-02-19T15:21:47.131+0000 7fbb578f37c0 -1 osd.2 0 log_to_monitors true 2024-02-19T15:21:47.178 INFO:tasks.ceph.osd.5.smithi116.stderr:2024-02-19T15:21:47.176+0000 7f0f2d9967c0 -1 osd.5 0 log_to_monitors true 2024-02-19T15:21:47.194 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-02-19T15:21:47.192+0000 7f73280d47c0 -1 osd.6 0 log_to_monitors true 2024-02-19T15:21:47.245 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-02-19T15:21:47.239+0000 7f83c2c3b7c0 -1 osd.0 0 log_to_monitors true 2024-02-19T15:21:49.696 INFO:tasks.ceph.osd.3.smithi047.stderr:2024-02-19T15:21:49.691+0000 7f73ac400640 -1 osd.3 0 waiting for initial osdmap 2024-02-19T15:21:49.696 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-02-19T15:21:49.691+0000 7fbb51e00640 -1 osd.2 0 waiting for initial osdmap 2024-02-19T15:21:49.721 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-02-19T15:21:49.715+0000 7f5f39c00640 -1 osd.1 0 waiting for initial osdmap 2024-02-19T15:21:49.722 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-02-19T15:21:49.719+0000 7fbb48800640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T15:21:49.727 INFO:tasks.ceph.osd.3.smithi047.stderr:2024-02-19T15:21:49.723+0000 7f73a2e00640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T15:21:49.755 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-02-19T15:21:49.751+0000 7f5f30600640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T15:21:50.069 INFO:tasks.ceph.osd.4.smithi116.stderr:2024-02-19T15:21:50.068+0000 7f9093a00640 -1 osd.4 0 waiting for initial osdmap 2024-02-19T15:21:50.113 INFO:tasks.ceph.osd.4.smithi116.stderr:2024-02-19T15:21:50.112+0000 7f908a400640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T15:21:50.128 INFO:tasks.ceph.osd.7.smithi116.stderr:2024-02-19T15:21:50.124+0000 7f7363a00640 -1 osd.7 0 waiting for initial osdmap 2024-02-19T15:21:50.162 INFO:tasks.ceph.osd.7.smithi116.stderr:2024-02-19T15:21:50.160+0000 7f735a400640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T15:21:50.168 INFO:tasks.ceph.osd.5.smithi116.stderr:2024-02-19T15:21:50.164+0000 7f0f29c00640 -1 osd.5 0 waiting for initial osdmap 2024-02-19T15:21:50.214 INFO:tasks.ceph.osd.5.smithi116.stderr:2024-02-19T15:21:50.212+0000 7f0f1e800640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T15:21:50.478 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-02-19T15:21:50.475+0000 7f83bd000640 -1 osd.0 0 waiting for initial osdmap 2024-02-19T15:21:50.498 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-02-19T15:21:50.495+0000 7f83b3a00640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T15:21:50.674 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-02-19T15:21:50.671+0000 7f7322600640 -1 osd.6 0 waiting for initial osdmap 2024-02-19T15:21:50.706 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-02-19T15:21:50.703+0000 7f7319000640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T15:21:50.740 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-02-19T15:21:50.870 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.867+0000 7f806c200640 1 Processor -- start 2024-02-19T15:21:50.871 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.867+0000 7f806c200640 1 -- start start 2024-02-19T15:21:50.872 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.867+0000 7f806c200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f80640d1140 0x7f806415b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:50.872 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.867+0000 7f806c200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8064144a60 con 0x7f806415ba30 2024-02-19T15:21:50.872 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.867+0000 7f806c200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f806415c260 con 0x7f8064144680 2024-02-19T15:21:50.872 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.867+0000 7f806c200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f806415e9c0 con 0x7f80640d1140 2024-02-19T15:21:50.872 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.867+0000 7f8068e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f80640d1140 0x7f806415b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:50.872 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.867+0000 7f806a200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f806415ba30 0x7f806406e8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39224/0 (socket says 172.21.15.47:39224) 2024-02-19T15:21:50.872 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.867+0000 7f806a200640 1 -- 172.21.15.47:0/84993098 learned_addr learned my addr 172.21.15.47:0/84993098 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:21:50.873 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.867+0000 7f8063e00640 1 -- 172.21.15.47:0/84993098 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812306367 0 0) 0x7f8064144a60 con 0x7f806415ba30 2024-02-19T15:21:50.873 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.867+0000 7f8063e00640 1 -- 172.21.15.47:0/84993098 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f804c003530 con 0x7f806415ba30 2024-02-19T15:21:50.873 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.867+0000 7f8068e00640 1 -- 172.21.15.47:0/84993098 >> v1:172.21.15.116:6789/0 conn(0x7f8064144680 legacy=0x7f80640d0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:50.873 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.867+0000 7f8063e00640 1 -- 172.21.15.47:0/84993098 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454480952 0 0) 0x7f806415c260 con 0x7f8064144680 2024-02-19T15:21:50.873 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.867+0000 7f8068e00640 1 -- 172.21.15.47:0/84993098 >> v1:172.21.15.47:6789/0 conn(0x7f806415ba30 legacy=0x7f806406e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:50.873 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.867+0000 7f8068e00640 1 -- 172.21.15.47:0/84993098 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f806415eca0 con 0x7f80640d1140 2024-02-19T15:21:50.875 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.871+0000 7f8068e00640 1 --2- 172.21.15.47:0/84993098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80640d1140 0x7f806415b4f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f80500097c0 tx=0x7f805001c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e64b165cfe2da50 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:50.877 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.871+0000 7f8063e00640 1 -- 172.21.15.47:0/84993098 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80500ab670 con 0x7f80640d1140 2024-02-19T15:21:50.877 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.871+0000 7f8063e00640 1 -- 172.21.15.47:0/84993098 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80500ab7d0 con 0x7f80640d1140 2024-02-19T15:21:50.877 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.871+0000 7f8063e00640 1 -- 172.21.15.47:0/84993098 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80500abb60 con 0x7f80640d1140 2024-02-19T15:21:50.877 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.871+0000 7f806c200640 1 -- 172.21.15.47:0/84993098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80640d1140 msgr2=0x7f806415b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:50.877 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.871+0000 7f806c200640 1 --2- 172.21.15.47:0/84993098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80640d1140 0x7f806415b4f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f80500097c0 tx=0x7f805001c470 comp rx=0 tx=0).stop 2024-02-19T15:21:50.877 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.871+0000 7f806c200640 1 -- 172.21.15.47:0/84993098 shutdown_connections 2024-02-19T15:21:50.877 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.871+0000 7f806c200640 1 --2- 172.21.15.47:0/84993098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80640d1140 0x7f806415b4f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:50.877 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.871+0000 7f806c200640 1 -- 172.21.15.47:0/84993098 >> 172.21.15.47:0/84993098 conn(0x7f806406c270 msgr2=0x7f806406c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:50.877 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.871+0000 7f806c200640 1 -- 172.21.15.47:0/84993098 shutdown_connections 2024-02-19T15:21:50.877 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.871+0000 7f806c200640 1 -- 172.21.15.47:0/84993098 wait complete. 2024-02-19T15:21:50.878 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f806c200640 1 Processor -- start 2024-02-19T15:21:50.878 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f806c200640 1 -- start start 2024-02-19T15:21:50.881 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f806c200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80640d1140 0x7f806413d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:50.881 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f806c200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8064144680 0x7f8064164650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:50.881 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f806c200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f806415ba30 0x7f8064164b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:50.881 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f806c200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806413d910 con 0x7f8064144680 2024-02-19T15:21:50.881 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f806c200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8064141410 con 0x7f80640d1140 2024-02-19T15:21:50.881 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f806c200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8064141660 con 0x7f806415ba30 2024-02-19T15:21:50.881 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f8068e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8064144680 0x7f8064164650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:50.881 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f8068e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8064144680 0x7f8064164650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44986/0 (socket says 172.21.15.47:44986) 2024-02-19T15:21:50.881 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f8068e00640 1 -- 172.21.15.47:0/3050752696 learned_addr learned my addr 172.21.15.47:0/3050752696 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:21:50.881 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f806a200640 1 --2- 172.21.15.47:0/3050752696 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f806415ba30 0x7f8064164b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:50.881 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f8069800640 1 --2- 172.21.15.47:0/3050752696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80640d1140 0x7f806413d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:50.881 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f8068e00640 1 -- 172.21.15.47:0/3050752696 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f806415ba30 msgr2=0x7f8064164b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:50.881 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f8068e00640 1 --2- 172.21.15.47:0/3050752696 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f806415ba30 0x7f8064164b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:50.882 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f8068e00640 1 -- 172.21.15.47:0/3050752696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80640d1140 msgr2=0x7f806413d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:50.882 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f8068e00640 1 --2- 172.21.15.47:0/3050752696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80640d1140 0x7f806413d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:50.883 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f8068e00640 1 -- 172.21.15.47:0/3050752696 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8050009420 con 0x7f8064144680 2024-02-19T15:21:50.883 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f8069800640 1 --2- 172.21.15.47:0/3050752696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80640d1140 0x7f806413d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:21:50.883 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f8068e00640 1 --2- 172.21.15.47:0/3050752696 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8064144680 0x7f8064164650 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f80500097c0 tx=0x7f805001acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:50.883 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f8061600640 1 -- 172.21.15.47:0/3050752696 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8050004220 con 0x7f8064144680 2024-02-19T15:21:50.883 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f8061600640 1 -- 172.21.15.47:0/3050752696 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8050004380 con 0x7f8064144680 2024-02-19T15:21:50.883 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f806c200640 1 -- 172.21.15.47:0/3050752696 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8064159000 con 0x7f8064144680 2024-02-19T15:21:50.883 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f8061600640 1 -- 172.21.15.47:0/3050752696 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8050003820 con 0x7f8064144680 2024-02-19T15:21:50.883 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.875+0000 7f806c200640 1 -- 172.21.15.47:0/3050752696 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f806413eec0 con 0x7f8064144680 2024-02-19T15:21:50.884 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.879+0000 7f806c200640 1 -- 172.21.15.47:0/3050752696 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80640b4060 con 0x7f8064144680 2024-02-19T15:21:50.884 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.879+0000 7f8061600640 1 -- 172.21.15.47:0/3050752696 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 4) v1 ==== 86353+0+0 (secure 0 0 0) 0x7f8050003980 con 0x7f8064144680 2024-02-19T15:21:50.884 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.879+0000 7f8061600640 1 --2- 172.21.15.47:0/3050752696 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8044066de0 0x7f80440692a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:50.884 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.879+0000 7f8061600640 1 -- 172.21.15.47:0/3050752696 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4191+0+0 (secure 0 0 0) 0x7f805011ee40 con 0x7f8064144680 2024-02-19T15:21:50.884 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.879+0000 7f8069800640 1 --2- 172.21.15.47:0/3050752696 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8044066de0 0x7f80440692a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:50.884 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.879+0000 7f8069800640 1 --2- 172.21.15.47:0/3050752696 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8044066de0 0x7f80440692a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8058007fd0 tx=0x7f8058009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:50.888 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:50.883+0000 7f8061600640 1 -- 172.21.15.47:0/3050752696 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f80500ec440 con 0x7f8064144680 2024-02-19T15:21:51.049 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:51.047+0000 7f806c200640 1 -- 172.21.15.47:0/3050752696 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80640684b0 con 0x7f8064144680 2024-02-19T15:21:51.050 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:51.047+0000 7f8061600640 1 -- 172.21.15.47:0/3050752696 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10435 (secure 0 0 0) 0x7f80500c4090 con 0x7f8064144680 2024-02-19T15:21:51.050 INFO:teuthology.misc.health.smithi047.stdout: 2024-02-19T15:21:51.050 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":14,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:21:50.708404+0000","last_up_change":"2024-02-19T15:21:50.708404+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:21:51.052 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:51.047+0000 7f806c200640 1 -- 172.21.15.47:0/3050752696 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8044066de0 msgr2=0x7f80440692a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:51.052 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:51.047+0000 7f806c200640 1 --2- 172.21.15.47:0/3050752696 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8044066de0 0x7f80440692a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8058007fd0 tx=0x7f8058009040 comp rx=0 tx=0).stop 2024-02-19T15:21:51.052 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:51.047+0000 7f806c200640 1 -- 172.21.15.47:0/3050752696 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8064144680 msgr2=0x7f8064164650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:51.052 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:51.047+0000 7f806c200640 1 --2- 172.21.15.47:0/3050752696 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8064144680 0x7f8064164650 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f80500097c0 tx=0x7f805001acb0 comp rx=0 tx=0).stop 2024-02-19T15:21:51.053 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:51.047+0000 7f806c200640 1 -- 172.21.15.47:0/3050752696 shutdown_connections 2024-02-19T15:21:51.053 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:51.047+0000 7f806c200640 1 --2- 172.21.15.47:0/3050752696 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8044066de0 0x7f80440692a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:51.053 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:51.047+0000 7f806c200640 1 --2- 172.21.15.47:0/3050752696 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f806415ba30 0x7f8064164b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:51.053 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:51.047+0000 7f806c200640 1 --2- 172.21.15.47:0/3050752696 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8064144680 0x7f8064164650 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:51.053 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:51.047+0000 7f806c200640 1 --2- 172.21.15.47:0/3050752696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80640d1140 0x7f806413d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:51.053 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:51.047+0000 7f806c200640 1 -- 172.21.15.47:0/3050752696 >> 172.21.15.47:0/3050752696 conn(0x7f806406c270 msgr2=0x7f806406c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:51.053 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:51.047+0000 7f806c200640 1 -- 172.21.15.47:0/3050752696 shutdown_connections 2024-02-19T15:21:51.053 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:51.047+0000 7f806c200640 1 -- 172.21.15.47:0/3050752696 wait complete. 2024-02-19T15:21:51.069 DEBUG:teuthology.misc:7 of 8 OSDs are up 2024-02-19T15:21:57.875 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-02-19T15:21:58.016 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.014+0000 7fad93200640 1 Processor -- start 2024-02-19T15:21:58.016 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.014+0000 7fad93200640 1 -- start start 2024-02-19T15:21:58.017 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.014+0000 7fad93200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad940cb030 0x7fad9415eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:58.017 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.014+0000 7fad93200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad9406df20 con 0x7fad9406e3b0 2024-02-19T15:21:58.017 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.014+0000 7fad93200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad94158a60 con 0x7fad9406af90 2024-02-19T15:21:58.017 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.014+0000 7fad93200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad940cb8d0 con 0x7fad940cb030 2024-02-19T15:21:58.017 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.014+0000 7fad92800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fad9406e3b0 0x7fad9406e840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37038/0 (socket says 172.21.15.47:37038) 2024-02-19T15:21:58.017 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.014+0000 7fad92800640 1 -- 172.21.15.47:0/228358755 learned_addr learned my addr 172.21.15.47:0/228358755 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:21:58.018 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.018+0000 7fad91400640 1 --2- 172.21.15.47:0/228358755 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad940cb030 0x7fad9415eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:58.019 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.018+0000 7fad90a00640 1 -- 172.21.15.47:0/228358755 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452508486 0 0) 0x7fad9406df20 con 0x7fad9406e3b0 2024-02-19T15:21:58.019 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.018+0000 7fad90a00640 1 -- 172.21.15.47:0/228358755 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad78003530 con 0x7fad9406e3b0 2024-02-19T15:21:58.019 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.018+0000 7fad90a00640 1 -- 172.21.15.47:0/228358755 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167893291 0 0) 0x7fad94158a60 con 0x7fad9406af90 2024-02-19T15:21:58.019 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.018+0000 7fad90a00640 1 -- 172.21.15.47:0/228358755 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad9406df20 con 0x7fad9406af90 2024-02-19T15:21:58.019 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.018+0000 7fad90a00640 1 -- 172.21.15.47:0/228358755 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1878273088 0 0) 0x7fad78003530 con 0x7fad9406e3b0 2024-02-19T15:21:58.019 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.018+0000 7fad91400640 1 -- 172.21.15.47:0/228358755 >> v1:172.21.15.116:6789/0 conn(0x7fad9406af90 legacy=0x7fad9406b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:58.019 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.018+0000 7fad91400640 1 -- 172.21.15.47:0/228358755 >> v1:172.21.15.47:6789/0 conn(0x7fad9406e3b0 legacy=0x7fad9406e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:58.019 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.018+0000 7fad91400640 1 -- 172.21.15.47:0/228358755 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9406d8d0 con 0x7fad940cb030 2024-02-19T15:21:58.021 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.018+0000 7fad91400640 1 --2- 172.21.15.47:0/228358755 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad940cb030 0x7fad9415eea0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fad800097f0 tx=0x7fad8001c3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78aad134ca705c57 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:58.022 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.018+0000 7fad90a00640 1 -- 172.21.15.47:0/228358755 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c001a00 con 0x7fad940cb030 2024-02-19T15:21:58.022 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.018+0000 7fad90a00640 1 -- 172.21.15.47:0/228358755 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad800a7050 con 0x7fad940cb030 2024-02-19T15:21:58.022 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.018+0000 7fad90a00640 1 -- 172.21.15.47:0/228358755 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad800ab620 con 0x7fad940cb030 2024-02-19T15:21:58.022 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad93200640 1 -- 172.21.15.47:0/228358755 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad940cb030 msgr2=0x7fad9415eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:58.022 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad93200640 1 --2- 172.21.15.47:0/228358755 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad940cb030 0x7fad9415eea0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fad800097f0 tx=0x7fad8001c3d0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.022 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad93200640 1 -- 172.21.15.47:0/228358755 shutdown_connections 2024-02-19T15:21:58.023 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad93200640 1 --2- 172.21.15.47:0/228358755 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad940cb030 0x7fad9415eea0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.023 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad93200640 1 -- 172.21.15.47:0/228358755 >> 172.21.15.47:0/228358755 conn(0x7fad940d09e0 msgr2=0x7fad940d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:58.023 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad93200640 1 -- 172.21.15.47:0/228358755 shutdown_connections 2024-02-19T15:21:58.023 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad93200640 1 -- 172.21.15.47:0/228358755 wait complete. 2024-02-19T15:21:58.024 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad93200640 1 Processor -- start 2024-02-19T15:21:58.024 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad93200640 1 -- start start 2024-02-19T15:21:58.026 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad93200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad9406af90 0x7fad940c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:58.026 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad93200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad9406e3b0 0x7fad94153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:58.026 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad93200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad940cb030 0x7fad940b5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:58.026 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad93200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad940c87a0 con 0x7fad940cb030 2024-02-19T15:21:58.026 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad93200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad940c8a80 con 0x7fad9406af90 2024-02-19T15:21:58.026 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad93200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad940c8d60 con 0x7fad9406e3b0 2024-02-19T15:21:58.026 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad92800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad940cb030 0x7fad940b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:58.027 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad92800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad940cb030 0x7fad940b5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42706/0 (socket says 172.21.15.47:42706) 2024-02-19T15:21:58.027 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad92800640 1 -- 172.21.15.47:0/3919077068 learned_addr learned my addr 172.21.15.47:0/3919077068 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:21:58.027 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad91400640 1 --2- 172.21.15.47:0/3919077068 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad9406e3b0 0x7fad94153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:58.027 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad91e00640 1 --2- 172.21.15.47:0/3919077068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad9406af90 0x7fad940c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:58.027 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad92800640 1 -- 172.21.15.47:0/3919077068 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad9406e3b0 msgr2=0x7fad94153670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:58.027 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad92800640 1 --2- 172.21.15.47:0/3919077068 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad9406e3b0 0x7fad94153670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.027 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad92800640 1 -- 172.21.15.47:0/3919077068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad9406af90 msgr2=0x7fad940c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:58.027 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad92800640 1 --2- 172.21.15.47:0/3919077068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad9406af90 0x7fad940c9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.027 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.022+0000 7fad92800640 1 -- 172.21.15.47:0/3919077068 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad80009420 con 0x7fad940cb030 2024-02-19T15:21:58.027 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.026+0000 7fad92800640 1 --2- 172.21.15.47:0/3919077068 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad940cb030 0x7fad940b5310 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fad8c001f90 tx=0x7fad8c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:58.027 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.026+0000 7fad8a000640 1 -- 172.21.15.47:0/3919077068 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c012400 con 0x7fad940cb030 2024-02-19T15:21:58.027 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.026+0000 7fad8a000640 1 -- 172.21.15.47:0/3919077068 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8c012580 con 0x7fad940cb030 2024-02-19T15:21:58.027 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.026+0000 7fad8a000640 1 -- 172.21.15.47:0/3919077068 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c012930 con 0x7fad940cb030 2024-02-19T15:21:58.027 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.026+0000 7fad93200640 1 -- 172.21.15.47:0/3919077068 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad940c9040 con 0x7fad940cb030 2024-02-19T15:21:58.027 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.026+0000 7fad93200640 1 -- 172.21.15.47:0/3919077068 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad94153bb0 con 0x7fad940cb030 2024-02-19T15:21:58.028 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.026+0000 7fad93200640 1 -- 172.21.15.47:0/3919077068 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad940c9280 con 0x7fad940cb030 2024-02-19T15:21:58.028 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.026+0000 7fad8a000640 1 -- 172.21.15.47:0/3919077068 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fad8c012a90 con 0x7fad940cb030 2024-02-19T15:21:58.030 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.030+0000 7fad8a000640 1 --2- 172.21.15.47:0/3919077068 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fad74066e40 0x7fad74069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:58.030 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.030+0000 7fad8a000640 1 -- 172.21.15.47:0/3919077068 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fad8c084de0 con 0x7fad940cb030 2024-02-19T15:21:58.030 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.030+0000 7fad91e00640 1 --2- 172.21.15.47:0/3919077068 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fad74066e40 0x7fad74069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:58.031 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.030+0000 7fad91e00640 1 --2- 172.21.15.47:0/3919077068 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fad74066e40 0x7fad74069300 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fad840011f0 tx=0x7fad84009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:58.034 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.034+0000 7fad8a000640 1 -- 172.21.15.47:0/3919077068 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fad8c051f30 con 0x7fad940cb030 2024-02-19T15:21:58.199 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.198+0000 7fad93200640 1 -- 172.21.15.47:0/3919077068 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad940c88e0 con 0x7fad940cb030 2024-02-19T15:21:58.200 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.198+0000 7fad8a000640 1 -- 172.21.15.47:0/3919077068 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12896 (secure 0 0 0) 0x7fad940c88e0 con 0x7fad940cb030 2024-02-19T15:21:58.200 INFO:teuthology.misc.health.smithi047.stdout: 2024-02-19T15:21:58.200 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":17,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:21:53.719262+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:21:58.202 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.202+0000 7fad93200640 1 -- 172.21.15.47:0/3919077068 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fad74066e40 msgr2=0x7fad74069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:58.202 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.202+0000 7fad93200640 1 --2- 172.21.15.47:0/3919077068 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fad74066e40 0x7fad74069300 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fad840011f0 tx=0x7fad84009040 comp rx=0 tx=0).stop 2024-02-19T15:21:58.202 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.202+0000 7fad93200640 1 -- 172.21.15.47:0/3919077068 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad940cb030 msgr2=0x7fad940b5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:58.202 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.202+0000 7fad93200640 1 --2- 172.21.15.47:0/3919077068 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad940cb030 0x7fad940b5310 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fad8c001f90 tx=0x7fad8c00a430 comp rx=0 tx=0).stop 2024-02-19T15:21:58.202 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.202+0000 7fad93200640 1 -- 172.21.15.47:0/3919077068 shutdown_connections 2024-02-19T15:21:58.203 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.202+0000 7fad93200640 1 --2- 172.21.15.47:0/3919077068 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fad74066e40 0x7fad74069300 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.203 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.202+0000 7fad93200640 1 --2- 172.21.15.47:0/3919077068 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad940cb030 0x7fad940b5310 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.203 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.202+0000 7fad93200640 1 --2- 172.21.15.47:0/3919077068 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad9406e3b0 0x7fad94153670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.203 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.202+0000 7fad93200640 1 --2- 172.21.15.47:0/3919077068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad9406af90 0x7fad940c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.203 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.202+0000 7fad93200640 1 -- 172.21.15.47:0/3919077068 >> 172.21.15.47:0/3919077068 conn(0x7fad940d09e0 msgr2=0x7fad940cf2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:58.203 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.202+0000 7fad93200640 1 -- 172.21.15.47:0/3919077068 shutdown_connections 2024-02-19T15:21:58.203 INFO:teuthology.misc.health.smithi047.stderr:2024-02-19T15:21:58.202+0000 7fad93200640 1 -- 172.21.15.47:0/3919077068 wait complete. 2024-02-19T15:21:58.219 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-02-19T15:21:58.219 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-02-19T15:21:58.220 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T15:21:58.220 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T15:21:58.223 INFO:tasks.ceph.mds.a:Started 2024-02-19T15:21:58.223 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T15:21:58.223 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T15:21:58.226 INFO:tasks.ceph.mds.c:Started 2024-02-19T15:21:58.226 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T15:21:58.226 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T15:21:58.230 INFO:tasks.ceph.mds.b:Started 2024-02-19T15:21:58.230 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T15:21:58.230 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T15:21:58.233 INFO:tasks.ceph.mds.d:Started 2024-02-19T15:21:58.233 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-02-19T15:21:58.234 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:21:58.322 INFO:tasks.ceph.mds.a.smithi047.stdout:starting mds.a at 2024-02-19T15:21:58.323 INFO:tasks.ceph.mds.c.smithi047.stdout:starting mds.c at 2024-02-19T15:21:58.337 INFO:tasks.ceph.mds.b.smithi116.stdout:starting mds.b at 2024-02-19T15:21:58.337 INFO:tasks.ceph.mds.d.smithi116.stdout:starting mds.d at 2024-02-19T15:21:58.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.378+0000 7fe21b600640 1 Processor -- start 2024-02-19T15:21:58.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.378+0000 7fe21b600640 1 -- start start 2024-02-19T15:21:58.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.378+0000 7fe21b600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe2140cded0 0x7fe2140ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:58.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.378+0000 7fe21b600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe2140ceb40 0x7fe21406bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:58.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.378+0000 7fe21b600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2140cce30 con 0x7fe21406d650 2024-02-19T15:21:58.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.378+0000 7fe21b600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2140cefd0 con 0x7fe2140ceb40 2024-02-19T15:21:58.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.378+0000 7fe21b600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2140cd110 con 0x7fe2140cded0 2024-02-19T15:21:58.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.378+0000 7fe219600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe2140ceb40 0x7fe21406bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:58.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.378+0000 7fe219600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe2140ceb40 0x7fe21406bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42714/0 (socket says 172.21.15.47:42714) 2024-02-19T15:21:58.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.378+0000 7fe219600640 1 -- 172.21.15.47:0/3115297926 learned_addr learned my addr 172.21.15.47:0/3115297926 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:21:58.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.378+0000 7fe213e00640 1 --2- 172.21.15.47:0/3115297926 >> v2:172.21.15.116:3300/0 conn(0x7fe2140cded0 0x7fe2140ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:58.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.378+0000 7fe219600640 1 -- 172.21.15.47:0/3115297926 >> v2:172.21.15.116:3300/0 conn(0x7fe2140cded0 msgr2=0x7fe2140ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:21:58.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.378+0000 7fe219600640 1 --2- 172.21.15.47:0/3115297926 >> v2:172.21.15.116:3300/0 conn(0x7fe2140cded0 0x7fe2140ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.378+0000 7fe219600640 1 -- 172.21.15.47:0/3115297926 >> v1:172.21.15.116:6789/0 conn(0x7fe21406d650 legacy=0x7fe2140d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:58.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.378+0000 7fe219600640 1 -- 172.21.15.47:0/3115297926 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2140cd3f0 con 0x7fe2140ceb40 2024-02-19T15:21:58.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.378+0000 7fe213400640 1 -- 172.21.15.47:0/3115297926 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2980730727 0 0) 0x7fe2140cce30 con 0x7fe21406d650 2024-02-19T15:21:58.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.382+0000 7fe219600640 1 --2- 172.21.15.47:0/3115297926 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2140ceb40 0x7fe21406bd60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe21406b950 tx=0x7fe20c01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d2c754eaf0206 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:58.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.382+0000 7fe213400640 1 -- 172.21.15.47:0/3115297926 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20c0a57f0 con 0x7fe2140ceb40 2024-02-19T15:21:58.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.382+0000 7fe213400640 1 -- 172.21.15.47:0/3115297926 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe20c0a5950 con 0x7fe2140ceb40 2024-02-19T15:21:58.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.382+0000 7fe213400640 1 -- 172.21.15.47:0/3115297926 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20c0a5ce0 con 0x7fe2140ceb40 2024-02-19T15:21:58.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.382+0000 7fe21b600640 1 -- 172.21.15.47:0/3115297926 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2140ceb40 msgr2=0x7fe21406bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:58.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.382+0000 7fe21b600640 1 --2- 172.21.15.47:0/3115297926 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2140ceb40 0x7fe21406bd60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe21406b950 tx=0x7fe20c01c930 comp rx=0 tx=0).stop 2024-02-19T15:21:58.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.382+0000 7fe21b600640 1 -- 172.21.15.47:0/3115297926 shutdown_connections 2024-02-19T15:21:58.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.382+0000 7fe21b600640 1 --2- 172.21.15.47:0/3115297926 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2140ceb40 0x7fe21406bd60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.382+0000 7fe21b600640 1 --2- 172.21.15.47:0/3115297926 >> v2:172.21.15.116:3300/0 conn(0x7fe2140cded0 0x7fe2140ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.382+0000 7fe21b600640 1 -- 172.21.15.47:0/3115297926 >> 172.21.15.47:0/3115297926 conn(0x7fe2140d1bf0 msgr2=0x7fe2140d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:58.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.382+0000 7fe21b600640 1 -- 172.21.15.47:0/3115297926 shutdown_connections 2024-02-19T15:21:58.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.382+0000 7fe21b600640 1 -- 172.21.15.47:0/3115297926 wait complete. 2024-02-19T15:21:58.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.382+0000 7fe21b600640 1 Processor -- start 2024-02-19T15:21:58.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.382+0000 7fe21b600640 1 -- start start 2024-02-19T15:21:58.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe21b600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe21406d650 0x7fe2140c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:58.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe21b600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2140cded0 0x7fe21415d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:58.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe21b600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2140ceb40 0x7fe21406fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:58.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe21b600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2140c9510 con 0x7fe2140ceb40 2024-02-19T15:21:58.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe21b600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2140c97f0 con 0x7fe21406d650 2024-02-19T15:21:58.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe21b600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2140c9ad0 con 0x7fe2140cded0 2024-02-19T15:21:58.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe219600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2140ceb40 0x7fe21406fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:58.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe219600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2140ceb40 0x7fe21406fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42726/0 (socket says 172.21.15.47:42726) 2024-02-19T15:21:58.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe219600640 1 -- 172.21.15.47:0/1138755323 learned_addr learned my addr 172.21.15.47:0/1138755323 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:21:58.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe213e00640 1 --2- 172.21.15.47:0/1138755323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2140cded0 0x7fe21415d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:58.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe218c00640 1 --2- 172.21.15.47:0/1138755323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe21406d650 0x7fe2140c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:58.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe219600640 1 -- 172.21.15.47:0/1138755323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2140cded0 msgr2=0x7fe21415d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:58.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe219600640 1 --2- 172.21.15.47:0/1138755323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2140cded0 0x7fe21415d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe219600640 1 -- 172.21.15.47:0/1138755323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe21406d650 msgr2=0x7fe2140c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:58.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe219600640 1 --2- 172.21.15.47:0/1138755323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe21406d650 0x7fe2140c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe219600640 1 -- 172.21.15.47:0/1138755323 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20c009b30 con 0x7fe2140ceb40 2024-02-19T15:21:58.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe213e00640 1 --2- 172.21.15.47:0/1138755323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2140cded0 0x7fe21415d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:21:58.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe218c00640 1 --2- 172.21.15.47:0/1138755323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe21406d650 0x7fe2140c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:21:58.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe219600640 1 --2- 172.21.15.47:0/1138755323 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2140ceb40 0x7fe21406fb00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe20c0af040 tx=0x7fe20c0196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:58.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe210c00640 1 -- 172.21.15.47:0/1138755323 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20c0b8670 con 0x7fe2140ceb40 2024-02-19T15:21:58.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe210c00640 1 -- 172.21.15.47:0/1138755323 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe20c0b87d0 con 0x7fe2140ceb40 2024-02-19T15:21:58.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe21b600640 1 -- 172.21.15.47:0/1138755323 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2140c9db0 con 0x7fe2140ceb40 2024-02-19T15:21:58.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe210c00640 1 -- 172.21.15.47:0/1138755323 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20c0b8b80 con 0x7fe2140ceb40 2024-02-19T15:21:58.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe21b600640 1 -- 172.21.15.47:0/1138755323 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2140c9200 con 0x7fe2140ceb40 2024-02-19T15:21:58.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe21b600640 1 -- 172.21.15.47:0/1138755323 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2140c9c10 con 0x7fe2140ceb40 2024-02-19T15:21:58.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.386+0000 7fe210c00640 1 -- 172.21.15.47:0/1138755323 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe20c0b8ce0 con 0x7fe2140ceb40 2024-02-19T15:21:58.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.390+0000 7fe210c00640 1 --2- 172.21.15.47:0/1138755323 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe1f4066e90 0x7fe1f4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:58.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.390+0000 7fe210c00640 1 -- 172.21.15.47:0/1138755323 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe20c11fd00 con 0x7fe2140ceb40 2024-02-19T15:21:58.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.390+0000 7fe218c00640 1 --2- 172.21.15.47:0/1138755323 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe1f4066e90 0x7fe1f4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:58.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.390+0000 7fe218c00640 1 --2- 172.21.15.47:0/1138755323 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe1f4066e90 0x7fe1f4069350 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe208000f70 tx=0x7fe208009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:58.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.394+0000 7fe210c00640 1 -- 172.21.15.47:0/1138755323 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe2140c9c10 con 0x7fe2140ceb40 2024-02-19T15:21:58.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.558+0000 7fe21b600640 1 -- 172.21.15.47:0/1138755323 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2140c9c10 con 0x7fe2140ceb40 2024-02-19T15:21:58.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.558+0000 7fe210c00640 1 -- 172.21.15.47:0/1138755323 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12896 (secure 0 0 0) 0x7fe2140c9c10 con 0x7fe2140ceb40 2024-02-19T15:21:58.561 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:58.561 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":17,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:21:53.719262+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:21:58.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.562+0000 7fe21b600640 1 -- 172.21.15.47:0/1138755323 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe1f4066e90 msgr2=0x7fe1f4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:58.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.562+0000 7fe21b600640 1 --2- 172.21.15.47:0/1138755323 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe1f4066e90 0x7fe1f4069350 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe208000f70 tx=0x7fe208009040 comp rx=0 tx=0).stop 2024-02-19T15:21:58.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.562+0000 7fe21b600640 1 -- 172.21.15.47:0/1138755323 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2140ceb40 msgr2=0x7fe21406fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:58.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.562+0000 7fe21b600640 1 --2- 172.21.15.47:0/1138755323 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2140ceb40 0x7fe21406fb00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe20c0af040 tx=0x7fe20c0196d0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.562+0000 7fe21b600640 1 -- 172.21.15.47:0/1138755323 shutdown_connections 2024-02-19T15:21:58.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.562+0000 7fe21b600640 1 --2- 172.21.15.47:0/1138755323 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe1f4066e90 0x7fe1f4069350 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.562+0000 7fe21b600640 1 --2- 172.21.15.47:0/1138755323 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2140ceb40 0x7fe21406fb00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.562+0000 7fe21b600640 1 --2- 172.21.15.47:0/1138755323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2140cded0 0x7fe21415d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.562+0000 7fe21b600640 1 --2- 172.21.15.47:0/1138755323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe21406d650 0x7fe2140c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.562+0000 7fe21b600640 1 -- 172.21.15.47:0/1138755323 >> 172.21.15.47:0/1138755323 conn(0x7fe2140d1bf0 msgr2=0x7fe2140d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:58.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.562+0000 7fe21b600640 1 -- 172.21.15.47:0/1138755323 shutdown_connections 2024-02-19T15:21:58.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.562+0000 7fe21b600640 1 -- 172.21.15.47:0/1138755323 wait complete. 2024-02-19T15:21:58.582 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:21:58.582 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:21:58.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.722+0000 7fde0b400640 1 Processor -- start 2024-02-19T15:21:58.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.722+0000 7fde0b400640 1 -- start start 2024-02-19T15:21:58.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.726+0000 7fde0b400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fde040ce3b0 0x7fde040ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:58.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.726+0000 7fde0b400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde0415e280 0x7fde040cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:58.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.726+0000 7fde0b400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde040cc620 con 0x7fde040d2650 2024-02-19T15:21:58.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.726+0000 7fde0b400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde040d12e0 con 0x7fde040ce3b0 2024-02-19T15:21:58.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.726+0000 7fde0b400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde040d1940 con 0x7fde0415e280 2024-02-19T15:21:58.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.726+0000 7fde08a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fde040ce3b0 0x7fde040ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:58.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.726+0000 7fde03e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fde040d2650 0x7fde040ccab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37054/0 (socket says 172.21.15.47:37054) 2024-02-19T15:21:58.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.726+0000 7fde03e00640 1 -- 172.21.15.47:0/871633455 learned_addr learned my addr 172.21.15.47:0/871633455 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:21:58.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.726+0000 7fde09400640 1 --2- 172.21.15.47:0/871633455 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde0415e280 0x7fde040cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:58.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.726+0000 7fde08a00640 1 -- 172.21.15.47:0/871633455 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde0415e280 msgr2=0x7fde040cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:21:58.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.726+0000 7fde08a00640 1 --2- 172.21.15.47:0/871633455 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde0415e280 0x7fde040cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.726+0000 7fde08a00640 1 -- 172.21.15.47:0/871633455 >> v1:172.21.15.47:6789/0 conn(0x7fde040d2650 legacy=0x7fde040ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:58.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.726+0000 7fde08a00640 1 -- 172.21.15.47:0/871633455 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde040cecd0 con 0x7fde040ce3b0 2024-02-19T15:21:58.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.730+0000 7fde08a00640 1 --2- 172.21.15.47:0/871633455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde040ce3b0 0x7fde040ce790 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fddf80097c0 tx=0x7fddf801c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e438eb3518f8845 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:58.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.730+0000 7fde03400640 1 -- 172.21.15.47:0/871633455 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf80ab600 con 0x7fde040ce3b0 2024-02-19T15:21:58.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.730+0000 7fde03400640 1 -- 172.21.15.47:0/871633455 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf80a7050 con 0x7fde040ce3b0 2024-02-19T15:21:58.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.730+0000 7fde03400640 1 -- 172.21.15.47:0/871633455 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf80ab980 con 0x7fde040ce3b0 2024-02-19T15:21:58.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.730+0000 7fde0b400640 1 -- 172.21.15.47:0/871633455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde040ce3b0 msgr2=0x7fde040ce790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:58.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.730+0000 7fde0b400640 1 --2- 172.21.15.47:0/871633455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde040ce3b0 0x7fde040ce790 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fddf80097c0 tx=0x7fddf801c380 comp rx=0 tx=0).stop 2024-02-19T15:21:58.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.730+0000 7fde0b400640 1 -- 172.21.15.47:0/871633455 shutdown_connections 2024-02-19T15:21:58.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.730+0000 7fde0b400640 1 --2- 172.21.15.47:0/871633455 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde0415e280 0x7fde040cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.730+0000 7fde0b400640 1 --2- 172.21.15.47:0/871633455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde040ce3b0 0x7fde040ce790 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.730+0000 7fde0b400640 1 -- 172.21.15.47:0/871633455 >> 172.21.15.47:0/871633455 conn(0x7fde040d1b30 msgr2=0x7fde040d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:58.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.730+0000 7fde0b400640 1 -- 172.21.15.47:0/871633455 shutdown_connections 2024-02-19T15:21:58.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.730+0000 7fde0b400640 1 -- 172.21.15.47:0/871633455 wait complete. 2024-02-19T15:21:58.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.730+0000 7fde0b400640 1 Processor -- start 2024-02-19T15:21:58.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.730+0000 7fde0b400640 1 -- start start 2024-02-19T15:21:58.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.730+0000 7fde0b400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde040ce3b0 0x7fde040cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:58.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.730+0000 7fde0b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde040d2650 0x7fde04157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:58.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.730+0000 7fde0b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0415e280 0x7fde04157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:58.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.730+0000 7fde0b400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde04153920 con 0x7fde040d2650 2024-02-19T15:21:58.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.730+0000 7fde0b400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde04160bc0 con 0x7fde0415e280 2024-02-19T15:21:58.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.730+0000 7fde0b400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0409d680 con 0x7fde040ce3b0 2024-02-19T15:21:58.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.734+0000 7fde03e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde040d2650 0x7fde04157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:58.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.734+0000 7fde03e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde040d2650 0x7fde04157040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42740/0 (socket says 172.21.15.47:42740) 2024-02-19T15:21:58.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.734+0000 7fde03e00640 1 -- 172.21.15.47:0/48899257 learned_addr learned my addr 172.21.15.47:0/48899257 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:21:58.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.734+0000 7fde08a00640 1 --2- 172.21.15.47:0/48899257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde040ce3b0 0x7fde040cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:58.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.734+0000 7fde09400640 1 --2- 172.21.15.47:0/48899257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0415e280 0x7fde04157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:58.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.734+0000 7fde03e00640 1 -- 172.21.15.47:0/48899257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde040ce3b0 msgr2=0x7fde040cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:58.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.734+0000 7fde03e00640 1 --2- 172.21.15.47:0/48899257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde040ce3b0 0x7fde040cb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.734+0000 7fde03e00640 1 -- 172.21.15.47:0/48899257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0415e280 msgr2=0x7fde04157580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:58.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.734+0000 7fde03e00640 1 --2- 172.21.15.47:0/48899257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0415e280 0x7fde04157580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.734+0000 7fde03e00640 1 -- 172.21.15.47:0/48899257 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf8009420 con 0x7fde040d2650 2024-02-19T15:21:58.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.734+0000 7fde03e00640 1 --2- 172.21.15.47:0/48899257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde040d2650 0x7fde04157040 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fddf000c3a0 tx=0x7fddf000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:58.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.734+0000 7fde00c00640 1 -- 172.21.15.47:0/48899257 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf0005cf0 con 0x7fde040d2650 2024-02-19T15:21:58.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.734+0000 7fde00c00640 1 -- 172.21.15.47:0/48899257 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf0005e50 con 0x7fde040d2650 2024-02-19T15:21:58.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.734+0000 7fde0b400640 1 -- 172.21.15.47:0/48899257 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde0409cad0 con 0x7fde040d2650 2024-02-19T15:21:58.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.734+0000 7fde00c00640 1 -- 172.21.15.47:0/48899257 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf00072f0 con 0x7fde040d2650 2024-02-19T15:21:58.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.734+0000 7fde0b400640 1 -- 172.21.15.47:0/48899257 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde0409cd50 con 0x7fde040d2650 2024-02-19T15:21:58.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.734+0000 7fde0b400640 1 -- 172.21.15.47:0/48899257 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0409d830 con 0x7fde040d2650 2024-02-19T15:21:58.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.734+0000 7fde00c00640 1 -- 172.21.15.47:0/48899257 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fddf0007450 con 0x7fde040d2650 2024-02-19T15:21:58.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.738+0000 7fde00c00640 1 --2- 172.21.15.47:0/48899257 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdde4066e40 0x7fdde4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:58.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.738+0000 7fde00c00640 1 -- 172.21.15.47:0/48899257 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fddf0083c20 con 0x7fde040d2650 2024-02-19T15:21:58.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.738+0000 7fde08a00640 1 --2- 172.21.15.47:0/48899257 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdde4066e40 0x7fdde4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:58.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.738+0000 7fde08a00640 1 --2- 172.21.15.47:0/48899257 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdde4066e40 0x7fdde4069300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fddf80ad000 tx=0x7fddf8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:58.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.742+0000 7fde00c00640 1 -- 172.21.15.47:0/48899257 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fde0409d830 con 0x7fde040d2650 2024-02-19T15:21:58.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.906+0000 7fde0b400640 1 -- 172.21.15.47:0/48899257 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fde0409d7f0 con 0x7fde040d2650 2024-02-19T15:21:58.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.906+0000 7fde00c00640 1 -- 172.21.15.47:0/48899257 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7fddf0017020 con 0x7fde040d2650 2024-02-19T15:21:58.909 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:21:58.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.910+0000 7fde0b400640 1 -- 172.21.15.47:0/48899257 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdde4066e40 msgr2=0x7fdde4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:58.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.910+0000 7fde0b400640 1 --2- 172.21.15.47:0/48899257 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdde4066e40 0x7fdde4069300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fddf80ad000 tx=0x7fddf8008040 comp rx=0 tx=0).stop 2024-02-19T15:21:58.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.910+0000 7fde0b400640 1 -- 172.21.15.47:0/48899257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde040d2650 msgr2=0x7fde04157040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:58.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.910+0000 7fde0b400640 1 --2- 172.21.15.47:0/48899257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde040d2650 0x7fde04157040 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fddf000c3a0 tx=0x7fddf000c860 comp rx=0 tx=0).stop 2024-02-19T15:21:58.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.910+0000 7fde0b400640 1 -- 172.21.15.47:0/48899257 shutdown_connections 2024-02-19T15:21:58.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.910+0000 7fde0b400640 1 --2- 172.21.15.47:0/48899257 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdde4066e40 0x7fdde4069300 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.910+0000 7fde0b400640 1 --2- 172.21.15.47:0/48899257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0415e280 0x7fde04157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.910+0000 7fde0b400640 1 --2- 172.21.15.47:0/48899257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde040d2650 0x7fde04157040 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.910+0000 7fde0b400640 1 --2- 172.21.15.47:0/48899257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde040ce3b0 0x7fde040cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:58.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.910+0000 7fde0b400640 1 -- 172.21.15.47:0/48899257 >> 172.21.15.47:0/48899257 conn(0x7fde040d1b30 msgr2=0x7fde040c5620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:58.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.910+0000 7fde0b400640 1 -- 172.21.15.47:0/48899257 shutdown_connections 2024-02-19T15:21:58.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:58.910+0000 7fde0b400640 1 -- 172.21.15.47:0/48899257 wait complete. 2024-02-19T15:21:58.929 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:21:59.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.066+0000 7fbcd1400640 1 Processor -- start 2024-02-19T15:21:59.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.066+0000 7fbcd1400640 1 -- start start 2024-02-19T15:21:59.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.066+0000 7fbcd1400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbccc15d6e0 0x7fbccc162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:59.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.066+0000 7fbcd1400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbccc071350 0x7fbccc164c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:59.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.066+0000 7fbcd1400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbccc0718c0 con 0x7fbccc15b1a0 2024-02-19T15:21:59.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.066+0000 7fbcd1400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbccc071ba0 con 0x7fbccc15d6e0 2024-02-19T15:21:59.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.066+0000 7fbcd1400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc071e80 con 0x7fbccc071350 2024-02-19T15:21:59.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.070+0000 7fbcc1e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbccc15d6e0 0x7fbccc162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:59.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.070+0000 7fbcc1e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbccc15d6e0 0x7fbccc162740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42746/0 (socket says 172.21.15.47:42746) 2024-02-19T15:21:59.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.070+0000 7fbcc1e00640 1 -- 172.21.15.47:0/3065467345 learned_addr learned my addr 172.21.15.47:0/3065467345 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:21:59.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.070+0000 7fbcca800640 1 --1- 172.21.15.47:0/3065467345 >> v1:172.21.15.116:6789/0 conn(0x7fbccc15b1a0 0x7fbccc15b580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:33800/0 (socket says 172.21.15.47:33800) 2024-02-19T15:21:59.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.070+0000 7fbccb200640 1 --2- 172.21.15.47:0/3065467345 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbccc071350 0x7fbccc164c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:59.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.070+0000 7fbcc1e00640 1 -- 172.21.15.47:0/3065467345 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbccc071350 msgr2=0x7fbccc164c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:21:59.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.070+0000 7fbcc1e00640 1 --2- 172.21.15.47:0/3065467345 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbccc071350 0x7fbccc164c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:59.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.070+0000 7fbcc1e00640 1 -- 172.21.15.47:0/3065467345 >> v1:172.21.15.116:6789/0 conn(0x7fbccc15b1a0 legacy=0x7fbccc15b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:59.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.070+0000 7fbcc1e00640 1 -- 172.21.15.47:0/3065467345 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbccc072160 con 0x7fbccc15d6e0 2024-02-19T15:21:59.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.070+0000 7fbcc9e00640 1 -- 172.21.15.47:0/3065467345 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187012647 0 0) 0x7fbccc0718c0 con 0x7fbccc15b1a0 2024-02-19T15:21:59.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.070+0000 7fbcc1e00640 1 --2- 172.21.15.47:0/3065467345 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbccc15d6e0 0x7fbccc162740 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbcb40098a0 tx=0x7fbcb401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef44a6454b04623 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:59.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.070+0000 7fbcc9e00640 1 -- 172.21.15.47:0/3065467345 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb40a9620 con 0x7fbccc15d6e0 2024-02-19T15:21:59.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.070+0000 7fbcc9e00640 1 -- 172.21.15.47:0/3065467345 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb40a9780 con 0x7fbccc15d6e0 2024-02-19T15:21:59.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.070+0000 7fbcc9e00640 1 -- 172.21.15.47:0/3065467345 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb40a9b10 con 0x7fbccc15d6e0 2024-02-19T15:21:59.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbcd1400640 1 -- 172.21.15.47:0/3065467345 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbccc15d6e0 msgr2=0x7fbccc162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:59.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbcd1400640 1 --2- 172.21.15.47:0/3065467345 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbccc15d6e0 0x7fbccc162740 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbcb40098a0 tx=0x7fbcb401c540 comp rx=0 tx=0).stop 2024-02-19T15:21:59.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbcd1400640 1 -- 172.21.15.47:0/3065467345 shutdown_connections 2024-02-19T15:21:59.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbcd1400640 1 --2- 172.21.15.47:0/3065467345 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbccc071350 0x7fbccc164c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:59.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbcd1400640 1 --2- 172.21.15.47:0/3065467345 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbccc15d6e0 0x7fbccc162740 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:59.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbcd1400640 1 -- 172.21.15.47:0/3065467345 >> 172.21.15.47:0/3065467345 conn(0x7fbccc0b16a0 msgr2=0x7fbccc0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:59.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbcd1400640 1 -- 172.21.15.47:0/3065467345 shutdown_connections 2024-02-19T15:21:59.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbcd1400640 1 -- 172.21.15.47:0/3065467345 wait complete. 2024-02-19T15:21:59.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbcd1400640 1 Processor -- start 2024-02-19T15:21:59.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbcd1400640 1 -- start start 2024-02-19T15:21:59.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbcd1400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbccc071350 0x7fbccc14b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:59.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbcd1400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbccc15b1a0 0x7fbccc148800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:59.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbcd1400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbccc15d6e0 0x7fbccc0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:59.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbcd1400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc0cf040 con 0x7fbccc15d6e0 2024-02-19T15:21:59.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbcd1400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc0cda60 con 0x7fbccc071350 2024-02-19T15:21:59.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbcd1400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc0cdd40 con 0x7fbccc15b1a0 2024-02-19T15:21:59.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbccb200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbccc15d6e0 0x7fbccc0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:59.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbccb200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbccc15d6e0 0x7fbccc0ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42750/0 (socket says 172.21.15.47:42750) 2024-02-19T15:21:59.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbccb200640 1 -- 172.21.15.47:0/1143089623 learned_addr learned my addr 172.21.15.47:0/1143089623 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:21:59.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbcc1e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbccc15b1a0 0x7fbccc148800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:59.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbcca800640 1 --2- 172.21.15.47:0/1143089623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbccc071350 0x7fbccc14b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:59.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbccb200640 1 -- 172.21.15.47:0/1143089623 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbccc15b1a0 msgr2=0x7fbccc148800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:59.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbccb200640 1 --2- 172.21.15.47:0/1143089623 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbccc15b1a0 0x7fbccc148800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:59.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbccb200640 1 -- 172.21.15.47:0/1143089623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbccc071350 msgr2=0x7fbccc14b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:59.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbccb200640 1 --2- 172.21.15.47:0/1143089623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbccc071350 0x7fbccc14b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:59.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbccb200640 1 -- 172.21.15.47:0/1143089623 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb40094d0 con 0x7fbccc15d6e0 2024-02-19T15:21:59.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.074+0000 7fbcca800640 1 --2- 172.21.15.47:0/1143089623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbccc071350 0x7fbccc14b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:21:59.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.078+0000 7fbccb200640 1 --2- 172.21.15.47:0/1143089623 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbccc15d6e0 0x7fbccc0ceb00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbcc400e7f0 tx=0x7fbcc400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:59.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.078+0000 7fbcc3400640 1 -- 172.21.15.47:0/1143089623 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4009530 con 0x7fbccc15d6e0 2024-02-19T15:21:59.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.078+0000 7fbcc3400640 1 -- 172.21.15.47:0/1143089623 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc400eea0 con 0x7fbccc15d6e0 2024-02-19T15:21:59.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.078+0000 7fbcd1400640 1 -- 172.21.15.47:0/1143089623 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbccc0ce020 con 0x7fbccc15d6e0 2024-02-19T15:21:59.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.078+0000 7fbcc3400640 1 -- 172.21.15.47:0/1143089623 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc400cb60 con 0x7fbccc15d6e0 2024-02-19T15:21:59.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.078+0000 7fbcd1400640 1 -- 172.21.15.47:0/1143089623 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbccc0ce5e0 con 0x7fbccc15d6e0 2024-02-19T15:21:59.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.078+0000 7fbcd1400640 1 -- 172.21.15.47:0/1143089623 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbccc0ce7d0 con 0x7fbccc15d6e0 2024-02-19T15:21:59.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.078+0000 7fbcc3400640 1 -- 172.21.15.47:0/1143089623 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fbcc400ccc0 con 0x7fbccc15d6e0 2024-02-19T15:21:59.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.078+0000 7fbcc3400640 1 --2- 172.21.15.47:0/1143089623 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbca8066e40 0x7fbca8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:21:59.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.078+0000 7fbcc3400640 1 -- 172.21.15.47:0/1143089623 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbcc40847a0 con 0x7fbccc15d6e0 2024-02-19T15:21:59.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.078+0000 7fbcca800640 1 --2- 172.21.15.47:0/1143089623 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbca8066e40 0x7fbca8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:21:59.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.082+0000 7fbcca800640 1 --2- 172.21.15.47:0/1143089623 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbca8066e40 0x7fbca8069300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc005fd0 tx=0x7fbcbc005b80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:21:59.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.082+0000 7fbcc3400640 1 -- 172.21.15.47:0/1143089623 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fbcc40518f0 con 0x7fbccc15d6e0 2024-02-19T15:21:59.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.282+0000 7fbcd1400640 1 -- 172.21.15.47:0/1143089623 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbccc075360 con 0x7fbccc15d6e0 2024-02-19T15:21:59.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.286+0000 7fbcc3400640 1 -- 172.21.15.47:0/1143089623 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1321 (secure 0 0 0) 0x7fbcc4017540 con 0x7fbccc15d6e0 2024-02-19T15:21:59.286 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:21:59.286 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/284820350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":284820350},{"type":"v1","addr":"172.21.15.47:6834","nonce":284820350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-02-19T15:21:59.286 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 2 2024-02-19T15:21:59.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.286+0000 7fbcd1400640 1 -- 172.21.15.47:0/1143089623 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbca8066e40 msgr2=0x7fbca8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:59.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.286+0000 7fbcd1400640 1 --2- 172.21.15.47:0/1143089623 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbca8066e40 0x7fbca8069300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc005fd0 tx=0x7fbcbc005b80 comp rx=0 tx=0).stop 2024-02-19T15:21:59.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.286+0000 7fbcd1400640 1 -- 172.21.15.47:0/1143089623 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbccc15d6e0 msgr2=0x7fbccc0ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:21:59.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.286+0000 7fbcd1400640 1 --2- 172.21.15.47:0/1143089623 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbccc15d6e0 0x7fbccc0ceb00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbcc400e7f0 tx=0x7fbcc400ecb0 comp rx=0 tx=0).stop 2024-02-19T15:21:59.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.286+0000 7fbcd1400640 1 -- 172.21.15.47:0/1143089623 shutdown_connections 2024-02-19T15:21:59.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.286+0000 7fbcd1400640 1 --2- 172.21.15.47:0/1143089623 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbca8066e40 0x7fbca8069300 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:59.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.286+0000 7fbcd1400640 1 --2- 172.21.15.47:0/1143089623 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbccc15d6e0 0x7fbccc0ceb00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:59.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.286+0000 7fbcd1400640 1 --2- 172.21.15.47:0/1143089623 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbccc15b1a0 0x7fbccc148800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:59.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.286+0000 7fbcd1400640 1 --2- 172.21.15.47:0/1143089623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbccc071350 0x7fbccc14b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:21:59.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.286+0000 7fbcd1400640 1 -- 172.21.15.47:0/1143089623 >> 172.21.15.47:0/1143089623 conn(0x7fbccc0b16a0 msgr2=0x7fbccc0b0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:21:59.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.286+0000 7fbcd1400640 1 -- 172.21.15.47:0/1143089623 shutdown_connections 2024-02-19T15:21:59.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:21:59.286+0000 7fbcd1400640 1 -- 172.21.15.47:0/1143089623 wait complete. 2024-02-19T15:22:01.307 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:22:01.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.458+0000 7f46e0c00640 1 Processor -- start 2024-02-19T15:22:01.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.458+0000 7f46e0c00640 1 -- start start 2024-02-19T15:22:01.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.458+0000 7f46e0c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f46dc15dfa0 0x7f46dc06fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:01.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.458+0000 7f46e0c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f46dc0b4240 0x7f46dc0d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:01.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.458+0000 7f46e0c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46dc06ff60 con 0x7f46dc0ce400 2024-02-19T15:22:01.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.458+0000 7f46e0c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46dc0d1070 con 0x7f46dc15dfa0 2024-02-19T15:22:01.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.458+0000 7f46e0c00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46dc06ec60 con 0x7f46dc0b4240 2024-02-19T15:22:01.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.458+0000 7f46d9e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f46dc15dfa0 0x7f46dc06fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:01.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.458+0000 7f46d9e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f46dc15dfa0 0x7f46dc06fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42752/0 (socket says 172.21.15.47:42752) 2024-02-19T15:22:01.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.458+0000 7f46d9e00640 1 -- 172.21.15.47:0/850455667 learned_addr learned my addr 172.21.15.47:0/850455667 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:01.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.458+0000 7f46d9e00640 1 -- 172.21.15.47:0/850455667 >> v2:172.21.15.116:3300/0 conn(0x7f46dc0b4240 msgr2=0x7f46dc0d0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:01.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.458+0000 7f46d9e00640 1 --2- 172.21.15.47:0/850455667 >> v2:172.21.15.116:3300/0 conn(0x7f46dc0b4240 0x7f46dc0d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:01.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.458+0000 7f46d9e00640 1 -- 172.21.15.47:0/850455667 >> v1:172.21.15.116:6789/0 conn(0x7f46dc0ce400 legacy=0x7f46dc0cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:01.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.458+0000 7f46d9e00640 1 -- 172.21.15.47:0/850455667 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc06ef40 con 0x7f46dc15dfa0 2024-02-19T15:22:01.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.462+0000 7f46d9e00640 1 --2- 172.21.15.47:0/850455667 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46dc15dfa0 0x7f46dc06fa20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f46dc072110 tx=0x7f46cc01c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90788158ed084e76 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:01.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.462+0000 7f46d9400640 1 -- 172.21.15.47:0/850455667 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc0aa610 con 0x7f46dc15dfa0 2024-02-19T15:22:01.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.462+0000 7f46d9400640 1 -- 172.21.15.47:0/850455667 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46cc0a6040 con 0x7f46dc15dfa0 2024-02-19T15:22:01.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.462+0000 7f46d9400640 1 -- 172.21.15.47:0/850455667 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc0aa990 con 0x7f46dc15dfa0 2024-02-19T15:22:01.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.462+0000 7f46e0c00640 1 -- 172.21.15.47:0/850455667 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46dc15dfa0 msgr2=0x7f46dc06fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:01.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.462+0000 7f46e0c00640 1 --2- 172.21.15.47:0/850455667 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46dc15dfa0 0x7f46dc06fa20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f46dc072110 tx=0x7f46cc01c4b0 comp rx=0 tx=0).stop 2024-02-19T15:22:01.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.462+0000 7f46e0c00640 1 -- 172.21.15.47:0/850455667 shutdown_connections 2024-02-19T15:22:01.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.462+0000 7f46e0c00640 1 --2- 172.21.15.47:0/850455667 >> v2:172.21.15.116:3300/0 conn(0x7f46dc0b4240 0x7f46dc0d0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:01.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.462+0000 7f46e0c00640 1 --2- 172.21.15.47:0/850455667 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46dc15dfa0 0x7f46dc06fa20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:01.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.462+0000 7f46e0c00640 1 -- 172.21.15.47:0/850455667 >> 172.21.15.47:0/850455667 conn(0x7f46dc0b3690 msgr2=0x7f46dc0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:01.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.462+0000 7f46e0c00640 1 -- 172.21.15.47:0/850455667 shutdown_connections 2024-02-19T15:22:01.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.462+0000 7f46e0c00640 1 -- 172.21.15.47:0/850455667 wait complete. 2024-02-19T15:22:01.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46e0c00640 1 Processor -- start 2024-02-19T15:22:01.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46e0c00640 1 -- start start 2024-02-19T15:22:01.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46e0c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46dc0b4240 0x7f46dc0b1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:01.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46e0c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46dc0ce400 0x7f46dc15ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:01.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46e0c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc15dfa0 0x7f46dc15e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:01.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46e0c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc15f8e0 con 0x7f46dc0b4240 2024-02-19T15:22:01.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46e0c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc15f2b0 con 0x7f46dc0ce400 2024-02-19T15:22:01.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46e0c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc0b1e70 con 0x7f46dc15dfa0 2024-02-19T15:22:01.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46d9e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46dc0b4240 0x7f46dc0b1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:01.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46d9e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46dc0b4240 0x7f46dc0b1890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42760/0 (socket says 172.21.15.47:42760) 2024-02-19T15:22:01.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46d9e00640 1 -- 172.21.15.47:0/4266812997 learned_addr learned my addr 172.21.15.47:0/4266812997 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:01.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46d3e00640 1 --2- 172.21.15.47:0/4266812997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46dc0ce400 0x7f46dc15ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:01.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46da800640 1 --2- 172.21.15.47:0/4266812997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc15dfa0 0x7f46dc15e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:01.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46d9e00640 1 -- 172.21.15.47:0/4266812997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc15dfa0 msgr2=0x7f46dc15e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:01.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46d9e00640 1 --2- 172.21.15.47:0/4266812997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc15dfa0 0x7f46dc15e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:01.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46d9e00640 1 -- 172.21.15.47:0/4266812997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46dc0ce400 msgr2=0x7f46dc15ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:01.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46d9e00640 1 --2- 172.21.15.47:0/4266812997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46dc0ce400 0x7f46dc15ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:01.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46d9e00640 1 -- 172.21.15.47:0/4266812997 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc0094d0 con 0x7f46dc0b4240 2024-02-19T15:22:01.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46d3e00640 1 --2- 172.21.15.47:0/4266812997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46dc0ce400 0x7f46dc15ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:01.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46da800640 1 --2- 172.21.15.47:0/4266812997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc15dfa0 0x7f46dc15e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:01.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46d9e00640 1 --2- 172.21.15.47:0/4266812997 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46dc0b4240 0x7f46dc0b1890 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f46cc01a0e0 tx=0x7f46cc01acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:01.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46d2000640 1 -- 172.21.15.47:0/4266812997 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc0bd5f0 con 0x7f46dc0b4240 2024-02-19T15:22:01.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46d2000640 1 -- 172.21.15.47:0/4266812997 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46cc0bd770 con 0x7f46dc0b4240 2024-02-19T15:22:01.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46d2000640 1 -- 172.21.15.47:0/4266812997 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc0bdb20 con 0x7f46dc0b4240 2024-02-19T15:22:01.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46e0c00640 1 -- 172.21.15.47:0/4266812997 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46dc0b2150 con 0x7f46dc0b4240 2024-02-19T15:22:01.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.466+0000 7f46e0c00640 1 -- 172.21.15.47:0/4266812997 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46dc15d680 con 0x7f46dc0b4240 2024-02-19T15:22:01.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.470+0000 7f46e0c00640 1 -- 172.21.15.47:0/4266812997 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46dc075360 con 0x7f46dc0b4240 2024-02-19T15:22:01.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.470+0000 7f46d2000640 1 -- 172.21.15.47:0/4266812997 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f46cc0b33f0 con 0x7f46dc0b4240 2024-02-19T15:22:01.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.470+0000 7f46d2000640 1 --2- 172.21.15.47:0/4266812997 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f46b8066e40 0x7f46b8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:01.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.470+0000 7f46d2000640 1 -- 172.21.15.47:0/4266812997 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f46cc11f3b0 con 0x7f46dc0b4240 2024-02-19T15:22:01.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.470+0000 7f46d3e00640 1 --2- 172.21.15.47:0/4266812997 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f46b8066e40 0x7f46b8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:01.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.470+0000 7f46d3e00640 1 --2- 172.21.15.47:0/4266812997 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f46b8066e40 0x7f46b8069300 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f46c4006fd0 tx=0x7f46c4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:01.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.474+0000 7f46d2000640 1 -- 172.21.15.47:0/4266812997 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f46cc0ec590 con 0x7f46dc0b4240 2024-02-19T15:22:01.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.670+0000 7f46e0c00640 1 -- 172.21.15.47:0/4266812997 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46dc0b7b00 con 0x7f46dc0b4240 2024-02-19T15:22:01.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.670+0000 7f46d2000640 1 -- 172.21.15.47:0/4266812997 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3697 (secure 0 0 0) 0x7f46cc0efc50 con 0x7f46dc0b4240 2024-02-19T15:22:01.673 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:01.673 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3873760209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3873760209},{"type":"v1","addr":"172.21.15.116:6836","nonce":3873760209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/284820350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":284820350},{"type":"v1","addr":"172.21.15.47:6834","nonce":284820350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2764089359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2764089359},{"type":"v1","addr":"172.21.15.47:6835","nonce":2764089359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4284,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3920452343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3920452343},{"type":"v1","addr":"172.21.15.116:6837","nonce":3920452343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-02-19T15:22:01.674 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 3 2024-02-19T15:22:01.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.674+0000 7f46e0c00640 1 -- 172.21.15.47:0/4266812997 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f46b8066e40 msgr2=0x7f46b8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:01.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.674+0000 7f46e0c00640 1 --2- 172.21.15.47:0/4266812997 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f46b8066e40 0x7f46b8069300 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f46c4006fd0 tx=0x7f46c4008040 comp rx=0 tx=0).stop 2024-02-19T15:22:01.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.674+0000 7f46e0c00640 1 -- 172.21.15.47:0/4266812997 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46dc0b4240 msgr2=0x7f46dc0b1890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:01.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.674+0000 7f46e0c00640 1 --2- 172.21.15.47:0/4266812997 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46dc0b4240 0x7f46dc0b1890 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f46cc01a0e0 tx=0x7f46cc01acf0 comp rx=0 tx=0).stop 2024-02-19T15:22:01.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.674+0000 7f46e0c00640 1 -- 172.21.15.47:0/4266812997 shutdown_connections 2024-02-19T15:22:01.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.674+0000 7f46e0c00640 1 --2- 172.21.15.47:0/4266812997 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f46b8066e40 0x7f46b8069300 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:01.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.674+0000 7f46e0c00640 1 --2- 172.21.15.47:0/4266812997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc15dfa0 0x7f46dc15e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:01.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.674+0000 7f46e0c00640 1 --2- 172.21.15.47:0/4266812997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46dc0ce400 0x7f46dc15ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:01.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.674+0000 7f46e0c00640 1 --2- 172.21.15.47:0/4266812997 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46dc0b4240 0x7f46dc0b1890 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:01.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.674+0000 7f46e0c00640 1 -- 172.21.15.47:0/4266812997 >> 172.21.15.47:0/4266812997 conn(0x7f46dc0b3690 msgr2=0x7f46dc0cede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:01.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.674+0000 7f46e0c00640 1 -- 172.21.15.47:0/4266812997 shutdown_connections 2024-02-19T15:22:01.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.674+0000 7f46e0c00640 1 -- 172.21.15.47:0/4266812997 wait complete. 2024-02-19T15:22:01.695 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:01.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.834+0000 7fd653e00640 1 Processor -- start 2024-02-19T15:22:01.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.834+0000 7fd653e00640 1 -- start start 2024-02-19T15:22:01.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.838+0000 7fd653e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd64c15b1a0 0x7fd64c15b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:01.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.838+0000 7fd653e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd64c071350 0x7fd64c15bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:01.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.838+0000 7fd653e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64c15c000 con 0x7fd64c06c9a0 2024-02-19T15:22:01.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.838+0000 7fd653e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd64c0718c0 con 0x7fd64c15b1a0 2024-02-19T15:22:01.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.838+0000 7fd653e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd64c071ba0 con 0x7fd64c071350 2024-02-19T15:22:01.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.838+0000 7fd651400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd64c15b1a0 0x7fd64c15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:01.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.838+0000 7fd651e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd64c06c9a0 0x7fd64c15c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37056/0 (socket says 172.21.15.47:37056) 2024-02-19T15:22:01.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.838+0000 7fd651e00640 1 -- 172.21.15.47:0/132269666 learned_addr learned my addr 172.21.15.47:0/132269666 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:01.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.838+0000 7fd650a00640 1 --2- 172.21.15.47:0/132269666 >> v2:172.21.15.116:3300/0 conn(0x7fd64c071350 0x7fd64c15bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:01.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.838+0000 7fd651400640 1 -- 172.21.15.47:0/132269666 >> v2:172.21.15.116:3300/0 conn(0x7fd64c071350 msgr2=0x7fd64c15bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:01.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.838+0000 7fd651400640 1 --2- 172.21.15.47:0/132269666 >> v2:172.21.15.116:3300/0 conn(0x7fd64c071350 0x7fd64c15bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:01.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.838+0000 7fd651400640 1 -- 172.21.15.47:0/132269666 >> v1:172.21.15.47:6789/0 conn(0x7fd64c06c9a0 legacy=0x7fd64c15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:01.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.838+0000 7fd651400640 1 -- 172.21.15.47:0/132269666 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64c071e80 con 0x7fd64c15b1a0 2024-02-19T15:22:01.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.838+0000 7fd64be00640 1 -- 172.21.15.47:0/132269666 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3178592148 0 0) 0x7fd64c15c000 con 0x7fd64c06c9a0 2024-02-19T15:22:01.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.838+0000 7fd651400640 1 --2- 172.21.15.47:0/132269666 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd64c15b1a0 0x7fd64c15b580 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd6400097c0 tx=0x7fd64001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73008aad85209cdc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:01.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.838+0000 7fd64be00640 1 -- 172.21.15.47:0/132269666 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6400ab650 con 0x7fd64c15b1a0 2024-02-19T15:22:01.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.838+0000 7fd64be00640 1 -- 172.21.15.47:0/132269666 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6400ab7b0 con 0x7fd64c15b1a0 2024-02-19T15:22:01.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.838+0000 7fd64be00640 1 -- 172.21.15.47:0/132269666 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6400abb40 con 0x7fd64c15b1a0 2024-02-19T15:22:01.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd653e00640 1 -- 172.21.15.47:0/132269666 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd64c15b1a0 msgr2=0x7fd64c15b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:01.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd653e00640 1 --2- 172.21.15.47:0/132269666 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd64c15b1a0 0x7fd64c15b580 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd6400097c0 tx=0x7fd64001c430 comp rx=0 tx=0).stop 2024-02-19T15:22:01.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd653e00640 1 -- 172.21.15.47:0/132269666 shutdown_connections 2024-02-19T15:22:01.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd653e00640 1 --2- 172.21.15.47:0/132269666 >> v2:172.21.15.116:3300/0 conn(0x7fd64c071350 0x7fd64c15bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:01.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd653e00640 1 --2- 172.21.15.47:0/132269666 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd64c15b1a0 0x7fd64c15b580 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:01.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd653e00640 1 -- 172.21.15.47:0/132269666 >> 172.21.15.47:0/132269666 conn(0x7fd64c0b16a0 msgr2=0x7fd64c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:01.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd653e00640 1 -- 172.21.15.47:0/132269666 shutdown_connections 2024-02-19T15:22:01.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd653e00640 1 -- 172.21.15.47:0/132269666 wait complete. 2024-02-19T15:22:01.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd653e00640 1 Processor -- start 2024-02-19T15:22:01.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd653e00640 1 -- start start 2024-02-19T15:22:01.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd653e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd64c06c9a0 0x7fd64c0cbb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:01.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd653e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd64c071350 0x7fd64c157960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:01.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd653e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd64c15b1a0 0x7fd64c157ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:01.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd653e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c0cc0d0 con 0x7fd64c071350 2024-02-19T15:22:01.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd653e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c0b3630 con 0x7fd64c15b1a0 2024-02-19T15:22:01.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd653e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c14f7c0 con 0x7fd64c06c9a0 2024-02-19T15:22:01.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd650a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd64c071350 0x7fd64c157960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:01.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd650a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd64c071350 0x7fd64c157960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42770/0 (socket says 172.21.15.47:42770) 2024-02-19T15:22:01.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd650a00640 1 -- 172.21.15.47:0/192219135 learned_addr learned my addr 172.21.15.47:0/192219135 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:01.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd651400640 1 --2- 172.21.15.47:0/192219135 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd64c06c9a0 0x7fd64c0cbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:01.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd651e00640 1 --2- 172.21.15.47:0/192219135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd64c15b1a0 0x7fd64c157ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:01.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd650a00640 1 -- 172.21.15.47:0/192219135 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd64c06c9a0 msgr2=0x7fd64c0cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:01.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd650a00640 1 --2- 172.21.15.47:0/192219135 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd64c06c9a0 0x7fd64c0cbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:01.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd650a00640 1 -- 172.21.15.47:0/192219135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd64c15b1a0 msgr2=0x7fd64c157ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:01.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd650a00640 1 --2- 172.21.15.47:0/192219135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd64c15b1a0 0x7fd64c157ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:01.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd650a00640 1 -- 172.21.15.47:0/192219135 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd640009420 con 0x7fd64c071350 2024-02-19T15:22:01.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd651400640 1 --2- 172.21.15.47:0/192219135 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd64c06c9a0 0x7fd64c0cbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:01.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.842+0000 7fd650a00640 1 --2- 172.21.15.47:0/192219135 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd64c071350 0x7fd64c157960 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd63800d800 tx=0x7fd63800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:01.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.846+0000 7fd649600640 1 -- 172.21.15.47:0/192219135 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63800b6c0 con 0x7fd64c071350 2024-02-19T15:22:01.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.846+0000 7fd649600640 1 -- 172.21.15.47:0/192219135 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd63800b820 con 0x7fd64c071350 2024-02-19T15:22:01.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.846+0000 7fd649600640 1 -- 172.21.15.47:0/192219135 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63800bb90 con 0x7fd64c071350 2024-02-19T15:22:01.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.846+0000 7fd653e00640 1 -- 172.21.15.47:0/192219135 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd64c0b3350 con 0x7fd64c071350 2024-02-19T15:22:01.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.846+0000 7fd653e00640 1 -- 172.21.15.47:0/192219135 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd64c14f450 con 0x7fd64c071350 2024-02-19T15:22:01.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.846+0000 7fd653e00640 1 -- 172.21.15.47:0/192219135 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd64c075360 con 0x7fd64c071350 2024-02-19T15:22:01.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.846+0000 7fd649600640 1 -- 172.21.15.47:0/192219135 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd63800bcf0 con 0x7fd64c071350 2024-02-19T15:22:01.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.846+0000 7fd649600640 1 --2- 172.21.15.47:0/192219135 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd62c066e40 0x7fd62c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:01.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.846+0000 7fd649600640 1 -- 172.21.15.47:0/192219135 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd638084c70 con 0x7fd64c071350 2024-02-19T15:22:01.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.846+0000 7fd651400640 1 --2- 172.21.15.47:0/192219135 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd62c066e40 0x7fd62c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:01.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.850+0000 7fd651400640 1 --2- 172.21.15.47:0/192219135 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd62c066e40 0x7fd62c069300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd6400af000 tx=0x7fd6400a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:01.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:01.850+0000 7fd649600640 1 -- 172.21.15.47:0/192219135 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd638051da0 con 0x7fd64c071350 2024-02-19T15:22:02.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.014+0000 7fd653e00640 1 -- 172.21.15.47:0/192219135 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd64c0cc270 con 0x7fd64c071350 2024-02-19T15:22:02.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.014+0000 7fd649600640 1 -- 172.21.15.47:0/192219135 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12896 (secure 0 0 0) 0x7fd638055460 con 0x7fd64c071350 2024-02-19T15:22:02.018 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:02.018 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":17,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:21:53.719262+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:02.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.018+0000 7fd653e00640 1 -- 172.21.15.47:0/192219135 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd62c066e40 msgr2=0x7fd62c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:02.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.018+0000 7fd653e00640 1 --2- 172.21.15.47:0/192219135 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd62c066e40 0x7fd62c069300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd6400af000 tx=0x7fd6400a7000 comp rx=0 tx=0).stop 2024-02-19T15:22:02.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.018+0000 7fd653e00640 1 -- 172.21.15.47:0/192219135 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd64c071350 msgr2=0x7fd64c157960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:02.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.018+0000 7fd653e00640 1 --2- 172.21.15.47:0/192219135 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd64c071350 0x7fd64c157960 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd63800d800 tx=0x7fd63800dcc0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.018+0000 7fd653e00640 1 -- 172.21.15.47:0/192219135 shutdown_connections 2024-02-19T15:22:02.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.018+0000 7fd653e00640 1 --2- 172.21.15.47:0/192219135 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd62c066e40 0x7fd62c069300 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.018+0000 7fd653e00640 1 --2- 172.21.15.47:0/192219135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd64c15b1a0 0x7fd64c157ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.018+0000 7fd653e00640 1 --2- 172.21.15.47:0/192219135 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd64c071350 0x7fd64c157960 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.018+0000 7fd653e00640 1 --2- 172.21.15.47:0/192219135 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd64c06c9a0 0x7fd64c0cbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.018+0000 7fd653e00640 1 -- 172.21.15.47:0/192219135 >> 172.21.15.47:0/192219135 conn(0x7fd64c0b16a0 msgr2=0x7fd64c0af6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:02.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.018+0000 7fd653e00640 1 -- 172.21.15.47:0/192219135 shutdown_connections 2024-02-19T15:22:02.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.018+0000 7fd653e00640 1 -- 172.21.15.47:0/192219135 wait complete. 2024-02-19T15:22:02.038 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:22:02.039 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:22:02.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.178+0000 7fb6d7e00640 1 Processor -- start 2024-02-19T15:22:02.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.178+0000 7fb6d7e00640 1 -- start start 2024-02-19T15:22:02.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.182+0000 7fb6d7e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d00ce3b0 0x7fb6d00ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:02.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.182+0000 7fb6d7e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb6d015e280 0x7fb6d00cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:02.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.182+0000 7fb6d7e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d00cc620 con 0x7fb6d00d2650 2024-02-19T15:22:02.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.182+0000 7fb6d7e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6d00d12e0 con 0x7fb6d015e280 2024-02-19T15:22:02.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.182+0000 7fb6d7e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d00d1940 con 0x7fb6d00ce3b0 2024-02-19T15:22:02.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.182+0000 7fb6d5e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb6d015e280 0x7fb6d00cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:02.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.182+0000 7fb6d4a00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb6d00d2650 0x7fb6d00ccab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:33822/0 (socket says 172.21.15.47:33822) 2024-02-19T15:22:02.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.182+0000 7fb6d5400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d00ce3b0 0x7fb6d00ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:02.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.182+0000 7fb6d4a00640 1 -- 172.21.15.47:0/1839336014 learned_addr learned my addr 172.21.15.47:0/1839336014 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:02.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.182+0000 7fb6d5e00640 1 -- 172.21.15.47:0/1839336014 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d00ce3b0 msgr2=0x7fb6d00ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:02.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.182+0000 7fb6d5e00640 1 --2- 172.21.15.47:0/1839336014 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d00ce3b0 0x7fb6d00ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.182+0000 7fb6d5e00640 1 -- 172.21.15.47:0/1839336014 >> v1:172.21.15.116:6789/0 conn(0x7fb6d00d2650 legacy=0x7fb6d00ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:02.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.182+0000 7fb6d5e00640 1 -- 172.21.15.47:0/1839336014 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d00cecd0 con 0x7fb6d015e280 2024-02-19T15:22:02.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.182+0000 7fb6d5e00640 1 --2- 172.21.15.47:0/1839336014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6d015e280 0x7fb6d00cd1c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8009d90 tx=0x7fb6c801c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b958360bb72a8ec7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:02.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6cfe00640 1 -- 172.21.15.47:0/1839336014 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c80a5800 con 0x7fb6d015e280 2024-02-19T15:22:02.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6cfe00640 1 -- 172.21.15.47:0/1839336014 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6c80a5960 con 0x7fb6d015e280 2024-02-19T15:22:02.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6cfe00640 1 -- 172.21.15.47:0/1839336014 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c80a5cf0 con 0x7fb6d015e280 2024-02-19T15:22:02.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d7e00640 1 -- 172.21.15.47:0/1839336014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6d015e280 msgr2=0x7fb6d00cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:02.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d7e00640 1 --2- 172.21.15.47:0/1839336014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6d015e280 0x7fb6d00cd1c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8009d90 tx=0x7fb6c801c850 comp rx=0 tx=0).stop 2024-02-19T15:22:02.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d7e00640 1 -- 172.21.15.47:0/1839336014 shutdown_connections 2024-02-19T15:22:02.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d7e00640 1 --2- 172.21.15.47:0/1839336014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6d015e280 0x7fb6d00cd1c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d7e00640 1 --2- 172.21.15.47:0/1839336014 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d00ce3b0 0x7fb6d00ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d7e00640 1 -- 172.21.15.47:0/1839336014 >> 172.21.15.47:0/1839336014 conn(0x7fb6d00d1b30 msgr2=0x7fb6d00d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:02.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d7e00640 1 -- 172.21.15.47:0/1839336014 shutdown_connections 2024-02-19T15:22:02.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d7e00640 1 -- 172.21.15.47:0/1839336014 wait complete. 2024-02-19T15:22:02.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d7e00640 1 Processor -- start 2024-02-19T15:22:02.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d7e00640 1 -- start start 2024-02-19T15:22:02.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6d00ce3b0 0x7fb6d014fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:02.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d7e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d00d2650 0x7fb6d00cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:02.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6d015e280 0x7fb6d00cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:02.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d7e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d00cbef0 con 0x7fb6d015e280 2024-02-19T15:22:02.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d7e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d0150300 con 0x7fb6d00ce3b0 2024-02-19T15:22:02.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d7e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d01511b0 con 0x7fb6d00d2650 2024-02-19T15:22:02.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d5e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6d015e280 0x7fb6d00cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:02.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d5e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6d015e280 0x7fb6d00cb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42774/0 (socket says 172.21.15.47:42774) 2024-02-19T15:22:02.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d5e00640 1 -- 172.21.15.47:0/1687828820 learned_addr learned my addr 172.21.15.47:0/1687828820 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:02.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d5400640 1 --2- 172.21.15.47:0/1687828820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6d00ce3b0 0x7fb6d014fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:02.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.186+0000 7fb6d4a00640 1 --2- 172.21.15.47:0/1687828820 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d00d2650 0x7fb6d00cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:02.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.190+0000 7fb6d5e00640 1 -- 172.21.15.47:0/1687828820 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d00d2650 msgr2=0x7fb6d00cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:02.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.190+0000 7fb6d5e00640 1 --2- 172.21.15.47:0/1687828820 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d00d2650 0x7fb6d00cb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.190+0000 7fb6d5e00640 1 -- 172.21.15.47:0/1687828820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6d00ce3b0 msgr2=0x7fb6d014fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:02.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.190+0000 7fb6d5e00640 1 --2- 172.21.15.47:0/1687828820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6d00ce3b0 0x7fb6d014fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.190+0000 7fb6d5e00640 1 -- 172.21.15.47:0/1687828820 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c8009a70 con 0x7fb6d015e280 2024-02-19T15:22:02.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.190+0000 7fb6d5e00640 1 --2- 172.21.15.47:0/1687828820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6d015e280 0x7fb6d00cb9b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb6c80b0040 tx=0x7fb6c80195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:02.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.190+0000 7fb6cd600640 1 -- 172.21.15.47:0/1687828820 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c80b9650 con 0x7fb6d015e280 2024-02-19T15:22:02.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.190+0000 7fb6cd600640 1 -- 172.21.15.47:0/1687828820 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6c80b97b0 con 0x7fb6d015e280 2024-02-19T15:22:02.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.190+0000 7fb6d7e00640 1 -- 172.21.15.47:0/1687828820 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6d00b2c40 con 0x7fb6d015e280 2024-02-19T15:22:02.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.190+0000 7fb6cd600640 1 -- 172.21.15.47:0/1687828820 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c80b9b60 con 0x7fb6d015e280 2024-02-19T15:22:02.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.190+0000 7fb6d7e00640 1 -- 172.21.15.47:0/1687828820 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d0159df0 con 0x7fb6d015e280 2024-02-19T15:22:02.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.190+0000 7fb6d7e00640 1 -- 172.21.15.47:0/1687828820 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d0159f80 con 0x7fb6d015e280 2024-02-19T15:22:02.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.190+0000 7fb6cd600640 1 -- 172.21.15.47:0/1687828820 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb6c80b9cc0 con 0x7fb6d015e280 2024-02-19T15:22:02.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.190+0000 7fb6cd600640 1 --2- 172.21.15.47:0/1687828820 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb6b0066e40 0x7fb6b0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:02.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.190+0000 7fb6cd600640 1 -- 172.21.15.47:0/1687828820 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb6c811f8c0 con 0x7fb6d015e280 2024-02-19T15:22:02.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.190+0000 7fb6d5400640 1 --2- 172.21.15.47:0/1687828820 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb6b0066e40 0x7fb6b0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:02.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.194+0000 7fb6d5400640 1 --2- 172.21.15.47:0/1687828820 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb6b0066e40 0x7fb6b0069300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb6d009fab0 tx=0x7fb6c4009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:02.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.194+0000 7fb6cd600640 1 -- 172.21.15.47:0/1687828820 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb6c80eca10 con 0x7fb6d015e280 2024-02-19T15:22:02.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.358+0000 7fb6d7e00640 1 -- 172.21.15.47:0/1687828820 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb6d0075360 con 0x7fb6d015e280 2024-02-19T15:22:02.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.358+0000 7fb6cd600640 1 -- 172.21.15.47:0/1687828820 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb6c80bd020 con 0x7fb6d015e280 2024-02-19T15:22:02.360 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:22:02.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.362+0000 7fb6d7e00640 1 -- 172.21.15.47:0/1687828820 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb6b0066e40 msgr2=0x7fb6b0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:02.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.362+0000 7fb6d7e00640 1 --2- 172.21.15.47:0/1687828820 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb6b0066e40 0x7fb6b0069300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb6d009fab0 tx=0x7fb6c4009190 comp rx=0 tx=0).stop 2024-02-19T15:22:02.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.362+0000 7fb6d7e00640 1 -- 172.21.15.47:0/1687828820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6d015e280 msgr2=0x7fb6d00cb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:02.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.362+0000 7fb6d7e00640 1 --2- 172.21.15.47:0/1687828820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6d015e280 0x7fb6d00cb9b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb6c80b0040 tx=0x7fb6c80195f0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.362+0000 7fb6d7e00640 1 -- 172.21.15.47:0/1687828820 shutdown_connections 2024-02-19T15:22:02.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.362+0000 7fb6d7e00640 1 --2- 172.21.15.47:0/1687828820 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb6b0066e40 0x7fb6b0069300 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.362+0000 7fb6d7e00640 1 --2- 172.21.15.47:0/1687828820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6d015e280 0x7fb6d00cb9b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.362+0000 7fb6d7e00640 1 --2- 172.21.15.47:0/1687828820 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d00d2650 0x7fb6d00cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.362+0000 7fb6d7e00640 1 --2- 172.21.15.47:0/1687828820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6d00ce3b0 0x7fb6d014fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.362+0000 7fb6d7e00640 1 -- 172.21.15.47:0/1687828820 >> 172.21.15.47:0/1687828820 conn(0x7fb6d00d1b30 msgr2=0x7fb6d00c72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:02.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.362+0000 7fb6d7e00640 1 -- 172.21.15.47:0/1687828820 shutdown_connections 2024-02-19T15:22:02.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.362+0000 7fb6d7e00640 1 -- 172.21.15.47:0/1687828820 wait complete. 2024-02-19T15:22:02.381 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T15:22:02.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.518+0000 7fcbd4800640 1 Processor -- start 2024-02-19T15:22:02.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.518+0000 7fcbd4800640 1 -- start start 2024-02-19T15:22:02.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.518+0000 7fcbd4800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcbcc15b1a0 0x7fcbcc15b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:02.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.518+0000 7fcbd4800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcbcc06c9a0 0x7fcbcc15bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:02.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.518+0000 7fcbd4800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbcc15c000 con 0x7fcbcc071350 2024-02-19T15:22:02.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.522+0000 7fcbd4800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbcc0718c0 con 0x7fcbcc06c9a0 2024-02-19T15:22:02.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.522+0000 7fcbd4800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc071ba0 con 0x7fcbcc15b1a0 2024-02-19T15:22:02.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.522+0000 7fcbd1400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fcbcc071350 0x7fcbcc15c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37058/0 (socket says 172.21.15.47:37058) 2024-02-19T15:22:02.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.522+0000 7fcbd1400640 1 -- 172.21.15.47:0/2884842437 learned_addr learned my addr 172.21.15.47:0/2884842437 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:02.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.522+0000 7fcbd1e00640 1 --2- 172.21.15.47:0/2884842437 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcbcc15b1a0 0x7fcbcc15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:02.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.522+0000 7fcbd2800640 1 --2- 172.21.15.47:0/2884842437 >> v2:172.21.15.116:3300/0 conn(0x7fcbcc06c9a0 0x7fcbcc15bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:02.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.522+0000 7fcbd0a00640 1 -- 172.21.15.47:0/2884842437 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2513805940 0 0) 0x7fcbcc15c000 con 0x7fcbcc071350 2024-02-19T15:22:02.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.522+0000 7fcbd0a00640 1 -- 172.21.15.47:0/2884842437 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbb4003530 con 0x7fcbcc071350 2024-02-19T15:22:02.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.522+0000 7fcbd1e00640 1 -- 172.21.15.47:0/2884842437 >> v2:172.21.15.116:3300/0 conn(0x7fcbcc06c9a0 msgr2=0x7fcbcc15bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:02.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.522+0000 7fcbd1e00640 1 --2- 172.21.15.47:0/2884842437 >> v2:172.21.15.116:3300/0 conn(0x7fcbcc06c9a0 0x7fcbcc15bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.522+0000 7fcbd1e00640 1 -- 172.21.15.47:0/2884842437 >> v1:172.21.15.47:6789/0 conn(0x7fcbcc071350 legacy=0x7fcbcc15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:02.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.522+0000 7fcbd1e00640 1 -- 172.21.15.47:0/2884842437 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbcc071e80 con 0x7fcbcc15b1a0 2024-02-19T15:22:02.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.522+0000 7fcbd2800640 1 --2- 172.21.15.47:0/2884842437 >> v2:172.21.15.116:3300/0 conn(0x7fcbcc06c9a0 0x7fcbcc15bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:02.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.522+0000 7fcbd1e00640 1 --2- 172.21.15.47:0/2884842437 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcbcc15b1a0 0x7fcbcc15b580 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00097c0 tx=0x7fcbc001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b0e9be68c00c401 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:02.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.522+0000 7fcbd0a00640 1 -- 172.21.15.47:0/2884842437 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc00aa5f0 con 0x7fcbcc15b1a0 2024-02-19T15:22:02.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd0a00640 1 -- 172.21.15.47:0/2884842437 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc00aa750 con 0x7fcbcc15b1a0 2024-02-19T15:22:02.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd0a00640 1 -- 172.21.15.47:0/2884842437 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc00aaae0 con 0x7fcbcc15b1a0 2024-02-19T15:22:02.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd4800640 1 -- 172.21.15.47:0/2884842437 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcbcc15b1a0 msgr2=0x7fcbcc15b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:02.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd4800640 1 --2- 172.21.15.47:0/2884842437 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcbcc15b1a0 0x7fcbcc15b580 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00097c0 tx=0x7fcbc001c470 comp rx=0 tx=0).stop 2024-02-19T15:22:02.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd4800640 1 -- 172.21.15.47:0/2884842437 shutdown_connections 2024-02-19T15:22:02.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd4800640 1 --2- 172.21.15.47:0/2884842437 >> v2:172.21.15.116:3300/0 conn(0x7fcbcc06c9a0 0x7fcbcc15bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd4800640 1 --2- 172.21.15.47:0/2884842437 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcbcc15b1a0 0x7fcbcc15b580 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd4800640 1 -- 172.21.15.47:0/2884842437 >> 172.21.15.47:0/2884842437 conn(0x7fcbcc0b16a0 msgr2=0x7fcbcc0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:02.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd4800640 1 -- 172.21.15.47:0/2884842437 shutdown_connections 2024-02-19T15:22:02.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd4800640 1 -- 172.21.15.47:0/2884842437 wait complete. 2024-02-19T15:22:02.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd4800640 1 Processor -- start 2024-02-19T15:22:02.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd4800640 1 -- start start 2024-02-19T15:22:02.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd4800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbcc06c9a0 0x7fcbcc152cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:02.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd4800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbcc071350 0x7fcbcc14fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:02.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd4800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcbcc15b1a0 0x7fcbcc150180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:02.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd4800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc157e10 con 0x7fcbcc06c9a0 2024-02-19T15:22:02.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd4800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc1580f0 con 0x7fcbcc071350 2024-02-19T15:22:02.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd4800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc1583d0 con 0x7fcbcc15b1a0 2024-02-19T15:22:02.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd1e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbcc06c9a0 0x7fcbcc152cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:02.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd1e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbcc06c9a0 0x7fcbcc152cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42786/0 (socket says 172.21.15.47:42786) 2024-02-19T15:22:02.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd1e00640 1 -- 172.21.15.47:0/1770544177 learned_addr learned my addr 172.21.15.47:0/1770544177 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:02.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd1400640 1 --2- 172.21.15.47:0/1770544177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbcc071350 0x7fcbcc14fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:02.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd2800640 1 --2- 172.21.15.47:0/1770544177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcbcc15b1a0 0x7fcbcc150180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:02.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd1e00640 1 -- 172.21.15.47:0/1770544177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcbcc15b1a0 msgr2=0x7fcbcc150180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:02.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd1e00640 1 --2- 172.21.15.47:0/1770544177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcbcc15b1a0 0x7fcbcc150180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd1e00640 1 -- 172.21.15.47:0/1770544177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbcc071350 msgr2=0x7fcbcc14fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:02.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd1e00640 1 --2- 172.21.15.47:0/1770544177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbcc071350 0x7fcbcc14fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd1e00640 1 -- 172.21.15.47:0/1770544177 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc8009a70 con 0x7fcbcc06c9a0 2024-02-19T15:22:02.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.526+0000 7fcbd1400640 1 --2- 172.21.15.47:0/1770544177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbcc071350 0x7fcbcc14fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:02.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.530+0000 7fcbd1e00640 1 --2- 172.21.15.47:0/1770544177 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbcc06c9a0 0x7fcbcc152cc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcbc001ab20 tx=0x7fcbc0019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:02.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.530+0000 7fcbc6000640 1 -- 172.21.15.47:0/1770544177 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc00aad50 con 0x7fcbcc06c9a0 2024-02-19T15:22:02.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.530+0000 7fcbc6000640 1 -- 172.21.15.47:0/1770544177 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc00be450 con 0x7fcbcc06c9a0 2024-02-19T15:22:02.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.530+0000 7fcbd4800640 1 -- 172.21.15.47:0/1770544177 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc0009420 con 0x7fcbcc06c9a0 2024-02-19T15:22:02.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.530+0000 7fcbc6000640 1 -- 172.21.15.47:0/1770544177 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc00be800 con 0x7fcbcc06c9a0 2024-02-19T15:22:02.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.530+0000 7fcbd4800640 1 -- 172.21.15.47:0/1770544177 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbcc156310 con 0x7fcbcc06c9a0 2024-02-19T15:22:02.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.530+0000 7fcbd4800640 1 -- 172.21.15.47:0/1770544177 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbcc158560 con 0x7fcbcc06c9a0 2024-02-19T15:22:02.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.530+0000 7fcbc6000640 1 -- 172.21.15.47:0/1770544177 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fcbc00be960 con 0x7fcbcc06c9a0 2024-02-19T15:22:02.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.530+0000 7fcbc6000640 1 --2- 172.21.15.47:0/1770544177 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcbb0066e40 0x7fcbb0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:02.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.530+0000 7fcbc6000640 1 -- 172.21.15.47:0/1770544177 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcbc00a7070 con 0x7fcbcc06c9a0 2024-02-19T15:22:02.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.530+0000 7fcbd1400640 1 --2- 172.21.15.47:0/1770544177 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcbb0066e40 0x7fcbb0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:02.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.534+0000 7fcbd1400640 1 --2- 172.21.15.47:0/1770544177 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcbb0066e40 0x7fcbb0069300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc06e1e0 tx=0x7fcbbc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:02.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.534+0000 7fcbc6000640 1 -- 172.21.15.47:0/1770544177 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fcbc00ec280 con 0x7fcbcc06c9a0 2024-02-19T15:22:02.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.698+0000 7fcbd4800640 1 -- 172.21.15.47:0/1770544177 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fcbcc06d1b0 con 0x7fcbcc06c9a0 2024-02-19T15:22:02.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.698+0000 7fcbc6000640 1 -- 172.21.15.47:0/1770544177 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7fcbc00ef940 con 0x7fcbcc06c9a0 2024-02-19T15:22:02.702 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:02.702 INFO:teuthology.orchestra.run.smithi047.stdout:[ 2024-02-19T15:22:02.702 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:02.702 INFO:teuthology.orchestra.run.smithi047.stdout: "poolnum": 1, 2024-02-19T15:22:02.702 INFO:teuthology.orchestra.run.smithi047.stdout: "poolname": ".mgr" 2024-02-19T15:22:02.702 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:02.702 INFO:teuthology.orchestra.run.smithi047.stdout:] 2024-02-19T15:22:02.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.702+0000 7fcbd4800640 1 -- 172.21.15.47:0/1770544177 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcbb0066e40 msgr2=0x7fcbb0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:02.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.702+0000 7fcbd4800640 1 --2- 172.21.15.47:0/1770544177 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcbb0066e40 0x7fcbb0069300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc06e1e0 tx=0x7fcbbc009040 comp rx=0 tx=0).stop 2024-02-19T15:22:02.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.702+0000 7fcbd4800640 1 -- 172.21.15.47:0/1770544177 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbcc06c9a0 msgr2=0x7fcbcc152cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:02.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.702+0000 7fcbd4800640 1 --2- 172.21.15.47:0/1770544177 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbcc06c9a0 0x7fcbcc152cc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcbc001ab20 tx=0x7fcbc0019210 comp rx=0 tx=0).stop 2024-02-19T15:22:02.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.702+0000 7fcbd4800640 1 -- 172.21.15.47:0/1770544177 shutdown_connections 2024-02-19T15:22:02.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.702+0000 7fcbd4800640 1 --2- 172.21.15.47:0/1770544177 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcbb0066e40 0x7fcbb0069300 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.702+0000 7fcbd4800640 1 --2- 172.21.15.47:0/1770544177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcbcc15b1a0 0x7fcbcc150180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.702+0000 7fcbd4800640 1 --2- 172.21.15.47:0/1770544177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbcc071350 0x7fcbcc14fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.702+0000 7fcbd4800640 1 --2- 172.21.15.47:0/1770544177 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbcc06c9a0 0x7fcbcc152cc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.702+0000 7fcbd4800640 1 -- 172.21.15.47:0/1770544177 >> 172.21.15.47:0/1770544177 conn(0x7fcbcc0b16a0 msgr2=0x7fcbcc15db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:02.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.702+0000 7fcbd4800640 1 -- 172.21.15.47:0/1770544177 shutdown_connections 2024-02-19T15:22:02.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.702+0000 7fcbd4800640 1 -- 172.21.15.47:0/1770544177 wait complete. 2024-02-19T15:22:02.722 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T15:22:02.722 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T15:22:02.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.862+0000 7f5e85a00640 1 Processor -- start 2024-02-19T15:22:02.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.862+0000 7f5e85a00640 1 -- start start 2024-02-19T15:22:02.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.866+0000 7f5e85a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5e8006e570 0x7f5e80071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:02.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.866+0000 7f5e85a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e800ca520 con 0x7f5e8015abf0 2024-02-19T15:22:02.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.866+0000 7f5e85a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e800ca800 con 0x7f5e80071050 2024-02-19T15:22:02.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.866+0000 7f5e85a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e800caae0 con 0x7f5e8006e570 2024-02-19T15:22:02.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.866+0000 7f5e7ea00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5e8015abf0 0x7f5e8006de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37074/0 (socket says 172.21.15.47:37074) 2024-02-19T15:22:02.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.866+0000 7f5e7ea00640 1 -- 172.21.15.47:0/3841653600 learned_addr learned my addr 172.21.15.47:0/3841653600 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:02.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.866+0000 7f5e76000640 1 --2- 172.21.15.47:0/3841653600 >> v2:172.21.15.116:3300/0 conn(0x7f5e8006e570 0x7f5e80071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:02.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.866+0000 7f5e7e000640 1 -- 172.21.15.47:0/3841653600 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4163832563 0 0) 0x7f5e800ca520 con 0x7f5e8015abf0 2024-02-19T15:22:02.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.866+0000 7f5e7e000640 1 -- 172.21.15.47:0/3841653600 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e64005740 con 0x7f5e8015abf0 2024-02-19T15:22:02.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.866+0000 7f5e7e000640 1 -- 172.21.15.47:0/3841653600 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584449715 0 0) 0x7f5e800ca800 con 0x7f5e80071050 2024-02-19T15:22:02.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.866+0000 7f5e7e000640 1 -- 172.21.15.47:0/3841653600 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e800ca520 con 0x7f5e80071050 2024-02-19T15:22:02.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.866+0000 7f5e7e000640 1 -- 172.21.15.47:0/3841653600 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2658863238 0 0) 0x7f5e64005740 con 0x7f5e8015abf0 2024-02-19T15:22:02.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.866+0000 7f5e76000640 1 -- 172.21.15.47:0/3841653600 >> v1:172.21.15.116:6789/0 conn(0x7f5e80071050 legacy=0x7f5e800ae490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:02.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.866+0000 7f5e76000640 1 -- 172.21.15.47:0/3841653600 >> v1:172.21.15.47:6789/0 conn(0x7f5e8015abf0 legacy=0x7f5e8006de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:02.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.866+0000 7f5e76000640 1 -- 172.21.15.47:0/3841653600 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e800cadc0 con 0x7f5e8006e570 2024-02-19T15:22:02.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.870+0000 7f5e76000640 1 --2- 172.21.15.47:0/3841653600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e8006e570 0x7f5e80071f00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5e680097c0 tx=0x7f5e6801c390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a653596abae81d9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:02.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.870+0000 7f5e7e000640 1 -- 172.21.15.47:0/3841653600 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e70001680 con 0x7f5e8006e570 2024-02-19T15:22:02.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.870+0000 7f5e7e000640 1 -- 172.21.15.47:0/3841653600 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e680a7050 con 0x7f5e8006e570 2024-02-19T15:22:02.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.870+0000 7f5e7e000640 1 -- 172.21.15.47:0/3841653600 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e680ab620 con 0x7f5e8006e570 2024-02-19T15:22:02.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.870+0000 7f5e85a00640 1 -- 172.21.15.47:0/3841653600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e8006e570 msgr2=0x7f5e80071f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:02.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.870+0000 7f5e85a00640 1 --2- 172.21.15.47:0/3841653600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e8006e570 0x7f5e80071f00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5e680097c0 tx=0x7f5e6801c390 comp rx=0 tx=0).stop 2024-02-19T15:22:02.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.870+0000 7f5e85a00640 1 -- 172.21.15.47:0/3841653600 shutdown_connections 2024-02-19T15:22:02.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.870+0000 7f5e85a00640 1 --2- 172.21.15.47:0/3841653600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e8006e570 0x7f5e80071f00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.870+0000 7f5e85a00640 1 -- 172.21.15.47:0/3841653600 >> 172.21.15.47:0/3841653600 conn(0x7f5e800cc3f0 msgr2=0x7f5e800cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:02.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.870+0000 7f5e85a00640 1 -- 172.21.15.47:0/3841653600 shutdown_connections 2024-02-19T15:22:02.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.870+0000 7f5e85a00640 1 -- 172.21.15.47:0/3841653600 wait complete. 2024-02-19T15:22:02.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.870+0000 7f5e85a00640 1 Processor -- start 2024-02-19T15:22:02.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.870+0000 7f5e85a00640 1 -- start start 2024-02-19T15:22:02.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e85a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e8006e570 0x7f5e80154160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:02.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e85a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e80071050 0x7f5e800af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:02.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e85a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e8015abf0 0x7f5e800afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:02.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e85a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e80155780 con 0x7f5e8006e570 2024-02-19T15:22:02.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e85a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e80155a60 con 0x7f5e8015abf0 2024-02-19T15:22:02.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e85a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e80155d40 con 0x7f5e80071050 2024-02-19T15:22:02.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e7ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e8006e570 0x7f5e80154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:02.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e7ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e8006e570 0x7f5e80154160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42802/0 (socket says 172.21.15.47:42802) 2024-02-19T15:22:02.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e7ea00640 1 -- 172.21.15.47:0/4033083288 learned_addr learned my addr 172.21.15.47:0/4033083288 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:02.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e76000640 1 --2- 172.21.15.47:0/4033083288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e80071050 0x7f5e800af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:02.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e7f400640 1 --2- 172.21.15.47:0/4033083288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e8015abf0 0x7f5e800afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:02.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e7ea00640 1 -- 172.21.15.47:0/4033083288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e80071050 msgr2=0x7f5e800af750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:02.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e7ea00640 1 --2- 172.21.15.47:0/4033083288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e80071050 0x7f5e800af750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e7ea00640 1 -- 172.21.15.47:0/4033083288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e8015abf0 msgr2=0x7f5e800afc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:02.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e7ea00640 1 --2- 172.21.15.47:0/4033083288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e8015abf0 0x7f5e800afc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:02.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e7ea00640 1 -- 172.21.15.47:0/4033083288 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e68009420 con 0x7f5e8006e570 2024-02-19T15:22:02.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e76000640 1 --2- 172.21.15.47:0/4033083288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e80071050 0x7f5e800af750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:22:02.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e7f400640 1 --2- 172.21.15.47:0/4033083288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e8015abf0 0x7f5e800afc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:22:02.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e7ea00640 1 --2- 172.21.15.47:0/4033083288 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e8006e570 0x7f5e80154160 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5e7000c750 tx=0x7f5e7000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:02.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e77400640 1 -- 172.21.15.47:0/4033083288 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e70005af0 con 0x7f5e8006e570 2024-02-19T15:22:02.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e77400640 1 -- 172.21.15.47:0/4033083288 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e70005c50 con 0x7f5e8006e570 2024-02-19T15:22:02.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e77400640 1 -- 172.21.15.47:0/4033083288 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e70007070 con 0x7f5e8006e570 2024-02-19T15:22:02.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e85a00640 1 -- 172.21.15.47:0/4033083288 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e80156020 con 0x7f5e8006e570 2024-02-19T15:22:02.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e85a00640 1 -- 172.21.15.47:0/4033083288 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e801546a0 con 0x7f5e8006e570 2024-02-19T15:22:02.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.874+0000 7f5e85a00640 1 -- 172.21.15.47:0/4033083288 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e80075360 con 0x7f5e8006e570 2024-02-19T15:22:02.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.878+0000 7f5e77400640 1 -- 172.21.15.47:0/4033083288 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5e700071d0 con 0x7f5e8006e570 2024-02-19T15:22:02.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.878+0000 7f5e77400640 1 --2- 172.21.15.47:0/4033083288 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5e5c066e40 0x7f5e5c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:02.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.878+0000 7f5e77400640 1 -- 172.21.15.47:0/4033083288 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5e70083e10 con 0x7f5e8006e570 2024-02-19T15:22:02.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.878+0000 7f5e76000640 1 --2- 172.21.15.47:0/4033083288 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5e5c066e40 0x7f5e5c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:02.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.878+0000 7f5e76000640 1 --2- 172.21.15.47:0/4033083288 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5e5c066e40 0x7f5e5c069300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5e680ad000 tx=0x7f5e68008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:02.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:02.882+0000 7f5e77400640 1 -- 172.21.15.47:0/4033083288 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f5e70087030 con 0x7f5e8006e570 2024-02-19T15:22:03.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.042+0000 7f5e85a00640 1 -- 172.21.15.47:0/4033083288 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5e8015a420 con 0x7f5e8006e570 2024-02-19T15:22:03.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.810+0000 7f5e77400640 1 -- 172.21.15.47:0/4033083288 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5e700170b0 con 0x7f5e8006e570 2024-02-19T15:22:03.811 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_metadata' created 2024-02-19T15:22:03.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.814+0000 7f5e85a00640 1 -- 172.21.15.47:0/4033083288 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5e5c066e40 msgr2=0x7f5e5c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:03.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.814+0000 7f5e85a00640 1 --2- 172.21.15.47:0/4033083288 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5e5c066e40 0x7f5e5c069300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5e680ad000 tx=0x7f5e68008040 comp rx=0 tx=0).stop 2024-02-19T15:22:03.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.814+0000 7f5e85a00640 1 -- 172.21.15.47:0/4033083288 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e8006e570 msgr2=0x7f5e80154160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:03.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.814+0000 7f5e85a00640 1 --2- 172.21.15.47:0/4033083288 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e8006e570 0x7f5e80154160 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5e7000c750 tx=0x7f5e7000cc10 comp rx=0 tx=0).stop 2024-02-19T15:22:03.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.814+0000 7f5e85a00640 1 -- 172.21.15.47:0/4033083288 shutdown_connections 2024-02-19T15:22:03.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.814+0000 7f5e85a00640 1 --2- 172.21.15.47:0/4033083288 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5e5c066e40 0x7f5e5c069300 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5e680ad000 tx=0x7f5e68008040 comp rx=0 tx=0).stop 2024-02-19T15:22:03.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.814+0000 7f5e85a00640 1 --2- 172.21.15.47:0/4033083288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e8015abf0 0x7f5e800afc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:03.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.814+0000 7f5e85a00640 1 --2- 172.21.15.47:0/4033083288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e80071050 0x7f5e800af750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:03.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.814+0000 7f5e85a00640 1 --2- 172.21.15.47:0/4033083288 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e8006e570 0x7f5e80154160 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5e7000c750 tx=0x7f5e7000cc10 comp rx=0 tx=0).stop 2024-02-19T15:22:03.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.814+0000 7f5e85a00640 1 -- 172.21.15.47:0/4033083288 >> 172.21.15.47:0/4033083288 conn(0x7f5e800cc3f0 msgr2=0x7f5e8015df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:03.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.814+0000 7f5e85a00640 1 -- 172.21.15.47:0/4033083288 shutdown_connections 2024-02-19T15:22:03.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.818+0000 7f5e85a00640 1 -- 172.21.15.47:0/4033083288 wait complete. 2024-02-19T15:22:03.844 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T15:22:03.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.994+0000 7f7190000640 1 Processor -- start 2024-02-19T15:22:03.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.994+0000 7f7190000640 1 -- start start 2024-02-19T15:22:03.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.994+0000 7f7190000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f71880cec80 0x7f71880d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:03.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.994+0000 7f7190000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f718815f8f0 0x7f718815fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:03.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.994+0000 7f7190000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71880d0a50 con 0x7f718815f190 2024-02-19T15:22:03.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.994+0000 7f7190000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f718806d470 con 0x7f718815f8f0 2024-02-19T15:22:03.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.994+0000 7f7190000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71880cfe80 con 0x7f71880cec80 2024-02-19T15:22:03.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.994+0000 7f718e000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f718815f8f0 0x7f718815fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:03.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.994+0000 7f718e000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f718815f8f0 0x7f718815fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52770/0 (socket says 172.21.15.47:52770) 2024-02-19T15:22:03.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.994+0000 7f718e000640 1 -- 172.21.15.47:0/1548536605 learned_addr learned my addr 172.21.15.47:0/1548536605 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:03.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.994+0000 7f718d600640 1 --2- 172.21.15.47:0/1548536605 >> v2:172.21.15.116:3300/0 conn(0x7f71880cec80 0x7f71880d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:03.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.994+0000 7f718e000640 1 -- 172.21.15.47:0/1548536605 >> v2:172.21.15.116:3300/0 conn(0x7f71880cec80 msgr2=0x7f71880d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:03.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.994+0000 7f718e000640 1 --2- 172.21.15.47:0/1548536605 >> v2:172.21.15.116:3300/0 conn(0x7f71880cec80 0x7f71880d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:03.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.994+0000 7f718e000640 1 -- 172.21.15.47:0/1548536605 >> v1:172.21.15.116:6789/0 conn(0x7f718815f190 legacy=0x7f718806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:03.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.994+0000 7f718e000640 1 -- 172.21.15.47:0/1548536605 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71880d0120 con 0x7f718815f8f0 2024-02-19T15:22:03.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.994+0000 7f718d600640 1 --2- 172.21.15.47:0/1548536605 >> v2:172.21.15.116:3300/0 conn(0x7f71880cec80 0x7f71880d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:22:03.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.998+0000 7f718e000640 1 --2- 172.21.15.47:0/1548536605 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f718815f8f0 0x7f718815fd80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f718809fa10 tx=0x7f718001c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a599da9f9af92b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:04.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.998+0000 7f7187e00640 1 -- 172.21.15.47:0/1548536605 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71800a5750 con 0x7f718815f8f0 2024-02-19T15:22:04.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.998+0000 7f7187e00640 1 -- 172.21.15.47:0/1548536605 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71800a58b0 con 0x7f718815f8f0 2024-02-19T15:22:04.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.998+0000 7f7187e00640 1 -- 172.21.15.47:0/1548536605 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71800a5c40 con 0x7f718815f8f0 2024-02-19T15:22:04.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.998+0000 7f7190000640 1 -- 172.21.15.47:0/1548536605 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f718815f8f0 msgr2=0x7f718815fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:04.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.998+0000 7f7190000640 1 --2- 172.21.15.47:0/1548536605 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f718815f8f0 0x7f718815fd80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f718809fa10 tx=0x7f718001c980 comp rx=0 tx=0).stop 2024-02-19T15:22:04.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.998+0000 7f7190000640 1 -- 172.21.15.47:0/1548536605 shutdown_connections 2024-02-19T15:22:04.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.998+0000 7f7190000640 1 --2- 172.21.15.47:0/1548536605 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f718815f8f0 0x7f718815fd80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:04.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.998+0000 7f7190000640 1 --2- 172.21.15.47:0/1548536605 >> v2:172.21.15.116:3300/0 conn(0x7f71880cec80 0x7f71880d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:04.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.998+0000 7f7190000640 1 -- 172.21.15.47:0/1548536605 >> 172.21.15.47:0/1548536605 conn(0x7f718809aed0 msgr2=0x7f71880d1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:04.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.998+0000 7f7190000640 1 -- 172.21.15.47:0/1548536605 shutdown_connections 2024-02-19T15:22:04.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:03.998+0000 7f7190000640 1 -- 172.21.15.47:0/1548536605 wait complete. 2024-02-19T15:22:04.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f7190000640 1 Processor -- start 2024-02-19T15:22:04.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f7190000640 1 -- start start 2024-02-19T15:22:04.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f7190000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71880cec80 0x7f718809e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:04.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f7190000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f718815f190 0x7f7188156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:04.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f7190000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f718815f8f0 0x7f718809eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:04.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f7190000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f718809da10 con 0x7f718815f8f0 2024-02-19T15:22:04.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f7190000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f718809d170 con 0x7f71880cec80 2024-02-19T15:22:04.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f7190000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f718809d450 con 0x7f718815f190 2024-02-19T15:22:04.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f718e000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f718815f8f0 0x7f718809eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:04.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f718e000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f718815f8f0 0x7f718809eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52778/0 (socket says 172.21.15.47:52778) 2024-02-19T15:22:04.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f718e000640 1 -- 172.21.15.47:0/2185597336 learned_addr learned my addr 172.21.15.47:0/2185597336 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:04.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f718d600640 1 --2- 172.21.15.47:0/2185597336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71880cec80 0x7f718809e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:04.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f718cc00640 1 --2- 172.21.15.47:0/2185597336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f718815f190 0x7f7188156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:04.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f718e000640 1 -- 172.21.15.47:0/2185597336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f718815f190 msgr2=0x7f7188156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:04.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f718e000640 1 --2- 172.21.15.47:0/2185597336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f718815f190 0x7f7188156f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:04.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f718e000640 1 -- 172.21.15.47:0/2185597336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71880cec80 msgr2=0x7f718809e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:04.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f718e000640 1 --2- 172.21.15.47:0/2185597336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71880cec80 0x7f718809e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:04.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f718e000640 1 -- 172.21.15.47:0/2185597336 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7180009b30 con 0x7f718815f8f0 2024-02-19T15:22:04.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f718e000640 1 --2- 172.21.15.47:0/2185597336 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f718815f8f0 0x7f718809eb80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f718001a5b0 tx=0x7f7180019720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:04.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f7185600640 1 -- 172.21.15.47:0/2185597336 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71800b8670 con 0x7f718815f8f0 2024-02-19T15:22:04.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f7185600640 1 -- 172.21.15.47:0/2185597336 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71800b87f0 con 0x7f718815f8f0 2024-02-19T15:22:04.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f7185600640 1 -- 172.21.15.47:0/2185597336 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71800b8ba0 con 0x7f718815f8f0 2024-02-19T15:22:04.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f7190000640 1 -- 172.21.15.47:0/2185597336 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f718809d730 con 0x7f718815f8f0 2024-02-19T15:22:04.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f7190000640 1 -- 172.21.15.47:0/2185597336 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7188159fa0 con 0x7f718815f8f0 2024-02-19T15:22:04.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.002+0000 7f7190000640 1 -- 172.21.15.47:0/2185597336 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7188075360 con 0x7f718815f8f0 2024-02-19T15:22:04.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.006+0000 7f7185600640 1 -- 172.21.15.47:0/2185597336 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f71800b8d00 con 0x7f718815f8f0 2024-02-19T15:22:04.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.006+0000 7f7185600640 1 --2- 172.21.15.47:0/2185597336 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7168066e70 0x7f7168069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:04.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.006+0000 7f7185600640 1 -- 172.21.15.47:0/2185597336 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f718011fe30 con 0x7f718815f8f0 2024-02-19T15:22:04.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.006+0000 7f718d600640 1 --2- 172.21.15.47:0/2185597336 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7168066e70 0x7f7168069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:04.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.006+0000 7f718d600640 1 --2- 172.21.15.47:0/2185597336 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7168066e70 0x7f7168069330 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f717c00ab60 tx=0x7f717c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:04.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.010+0000 7f7185600640 1 -- 172.21.15.47:0/2185597336 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f71800ecfc0 con 0x7f718815f8f0 2024-02-19T15:22:04.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.174+0000 7f7190000640 1 -- 172.21.15.47:0/2185597336 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f718815bb60 con 0x7f718815f8f0 2024-02-19T15:22:04.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.810+0000 7f7185600640 1 -- 172.21.15.47:0/2185597336 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7f71800f0680 con 0x7f718815f8f0 2024-02-19T15:22:04.814 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_data' created 2024-02-19T15:22:04.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.818+0000 7f7190000640 1 -- 172.21.15.47:0/2185597336 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7168066e70 msgr2=0x7f7168069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:04.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.818+0000 7f7190000640 1 --2- 172.21.15.47:0/2185597336 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7168066e70 0x7f7168069330 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f717c00ab60 tx=0x7f717c008040 comp rx=0 tx=0).stop 2024-02-19T15:22:04.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.818+0000 7f7190000640 1 -- 172.21.15.47:0/2185597336 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f718815f8f0 msgr2=0x7f718809eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:04.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.818+0000 7f7190000640 1 --2- 172.21.15.47:0/2185597336 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f718815f8f0 0x7f718809eb80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f718001a5b0 tx=0x7f7180019720 comp rx=0 tx=0).stop 2024-02-19T15:22:04.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.818+0000 7f7190000640 1 -- 172.21.15.47:0/2185597336 shutdown_connections 2024-02-19T15:22:04.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.818+0000 7f7190000640 1 --2- 172.21.15.47:0/2185597336 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7168066e70 0x7f7168069330 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:04.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.818+0000 7f7190000640 1 --2- 172.21.15.47:0/2185597336 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f718815f8f0 0x7f718809eb80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:04.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.818+0000 7f7190000640 1 --2- 172.21.15.47:0/2185597336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f718815f190 0x7f7188156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:04.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.818+0000 7f7190000640 1 --2- 172.21.15.47:0/2185597336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71880cec80 0x7f718809e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:04.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.818+0000 7f7190000640 1 -- 172.21.15.47:0/2185597336 >> 172.21.15.47:0/2185597336 conn(0x7f718809aed0 msgr2=0x7f718806f420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:04.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.818+0000 7f7190000640 1 -- 172.21.15.47:0/2185597336 shutdown_connections 2024-02-19T15:22:04.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:04.818+0000 7f7190000640 1 -- 172.21.15.47:0/2185597336 wait complete. 2024-02-19T15:22:04.867 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T15:22:05.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.010+0000 7f8ed0000640 1 Processor -- start 2024-02-19T15:22:05.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.010+0000 7f8ed0000640 1 -- start start 2024-02-19T15:22:05.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.010+0000 7f8ed0000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8ec815bd90 0x7f8ec8070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:05.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.010+0000 7f8ed0000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ec80cb490 0x7f8ec8072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:05.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.010+0000 7f8ed0000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec815fc90 con 0x7f8ec8072680 2024-02-19T15:22:05.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.010+0000 7f8ed0000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ec815ff70 con 0x7f8ec815bd90 2024-02-19T15:22:05.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.010+0000 7f8ed0000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec8160250 con 0x7f8ec80cb490 2024-02-19T15:22:05.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.014+0000 7f8ecd600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8ec815bd90 0x7f8ec8070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:05.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.014+0000 7f8ece000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8ec8072680 0x7f8ec80ca270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43958/0 (socket says 172.21.15.47:43958) 2024-02-19T15:22:05.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.014+0000 7f8ece000640 1 -- 172.21.15.47:0/3083421306 learned_addr learned my addr 172.21.15.47:0/3083421306 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:05.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.014+0000 7f8ecd600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8ec815bd90 0x7f8ec8070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52782/0 (socket says 172.21.15.47:52782) 2024-02-19T15:22:05.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.014+0000 7f8eccc00640 1 --2- 172.21.15.47:0/3083421306 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ec80cb490 0x7f8ec8072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:05.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.014+0000 7f8ecd600640 1 -- 172.21.15.47:0/3083421306 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ec80cb490 msgr2=0x7f8ec8072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:05.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.014+0000 7f8ecd600640 1 --2- 172.21.15.47:0/3083421306 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ec80cb490 0x7f8ec8072140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:05.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.014+0000 7f8ecd600640 1 -- 172.21.15.47:0/3083421306 >> v1:172.21.15.47:6789/0 conn(0x7f8ec8072680 legacy=0x7f8ec80ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:05.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.014+0000 7f8ecd600640 1 -- 172.21.15.47:0/3083421306 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec8160530 con 0x7f8ec815bd90 2024-02-19T15:22:05.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.014+0000 7f8ec7e00640 1 -- 172.21.15.47:0/3083421306 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103001617 0 0) 0x7f8ec815fc90 con 0x7f8ec8072680 2024-02-19T15:22:05.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.014+0000 7f8ecd600640 1 --2- 172.21.15.47:0/3083421306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ec815bd90 0x7f8ec8070960 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc0097c0 tx=0x7f8ebc01c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9985fec7f21e66d5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:05.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.014+0000 7f8ec7e00640 1 -- 172.21.15.47:0/3083421306 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc0ab600 con 0x7f8ec815bd90 2024-02-19T15:22:05.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.014+0000 7f8ec7e00640 1 -- 172.21.15.47:0/3083421306 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ebc0a7050 con 0x7f8ec815bd90 2024-02-19T15:22:05.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.014+0000 7f8ec7e00640 1 -- 172.21.15.47:0/3083421306 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc0ab980 con 0x7f8ec815bd90 2024-02-19T15:22:05.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8ed0000640 1 -- 172.21.15.47:0/3083421306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ec815bd90 msgr2=0x7f8ec8070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:05.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8ed0000640 1 --2- 172.21.15.47:0/3083421306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ec815bd90 0x7f8ec8070960 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc0097c0 tx=0x7f8ebc01c340 comp rx=0 tx=0).stop 2024-02-19T15:22:05.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8ed0000640 1 -- 172.21.15.47:0/3083421306 shutdown_connections 2024-02-19T15:22:05.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8ed0000640 1 --2- 172.21.15.47:0/3083421306 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ec80cb490 0x7f8ec8072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:05.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8ed0000640 1 --2- 172.21.15.47:0/3083421306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ec815bd90 0x7f8ec8070960 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:05.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8ed0000640 1 -- 172.21.15.47:0/3083421306 >> 172.21.15.47:0/3083421306 conn(0x7f8ec80cfc20 msgr2=0x7f8ec80d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:05.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8ed0000640 1 -- 172.21.15.47:0/3083421306 shutdown_connections 2024-02-19T15:22:05.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8ed0000640 1 -- 172.21.15.47:0/3083421306 wait complete. 2024-02-19T15:22:05.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8ed0000640 1 Processor -- start 2024-02-19T15:22:05.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8ed0000640 1 -- start start 2024-02-19T15:22:05.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8ed0000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ec8072680 0x7f8ec80ce930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:05.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8ed0000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ec80cb490 0x7f8ec8152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:05.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8ed0000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ec815bd90 0x7f8ec81554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:05.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8ed0000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec80cd890 con 0x7f8ec80cb490 2024-02-19T15:22:05.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8ed0000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec80cdb70 con 0x7f8ec815bd90 2024-02-19T15:22:05.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8ed0000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec80cde50 con 0x7f8ec8072680 2024-02-19T15:22:05.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8eccc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ec80cb490 0x7f8ec8152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:05.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8eccc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ec80cb490 0x7f8ec8152620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52786/0 (socket says 172.21.15.47:52786) 2024-02-19T15:22:05.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8eccc00640 1 -- 172.21.15.47:0/3923682244 learned_addr learned my addr 172.21.15.47:0/3923682244 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:05.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8ecd600640 1 --2- 172.21.15.47:0/3923682244 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ec8072680 0x7f8ec80ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:05.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8ece000640 1 --2- 172.21.15.47:0/3923682244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ec815bd90 0x7f8ec81554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:05.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8eccc00640 1 -- 172.21.15.47:0/3923682244 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ec8072680 msgr2=0x7f8ec80ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:05.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8eccc00640 1 --2- 172.21.15.47:0/3923682244 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ec8072680 0x7f8ec80ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:05.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8eccc00640 1 -- 172.21.15.47:0/3923682244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ec815bd90 msgr2=0x7f8ec81554e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:05.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8eccc00640 1 --2- 172.21.15.47:0/3923682244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ec815bd90 0x7f8ec81554e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:05.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8eccc00640 1 -- 172.21.15.47:0/3923682244 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc009420 con 0x7f8ec80cb490 2024-02-19T15:22:05.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8eccc00640 1 --2- 172.21.15.47:0/3923682244 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ec80cb490 0x7f8ec8152620 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8eb400d5f0 tx=0x7f8eb400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:05.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.018+0000 7f8ec5600640 1 -- 172.21.15.47:0/3923682244 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb40125f0 con 0x7f8ec80cb490 2024-02-19T15:22:05.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.022+0000 7f8ec5600640 1 -- 172.21.15.47:0/3923682244 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb4012750 con 0x7f8ec80cb490 2024-02-19T15:22:05.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.022+0000 7f8ec5600640 1 -- 172.21.15.47:0/3923682244 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4012ae0 con 0x7f8ec80cb490 2024-02-19T15:22:05.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.022+0000 7f8ed0000640 1 -- 172.21.15.47:0/3923682244 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec80ce130 con 0x7f8ec80cb490 2024-02-19T15:22:05.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.022+0000 7f8ed0000640 1 -- 172.21.15.47:0/3923682244 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec8152b60 con 0x7f8ec80cb490 2024-02-19T15:22:05.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.022+0000 7f8ed0000640 1 -- 172.21.15.47:0/3923682244 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec8075360 con 0x7f8ec80cb490 2024-02-19T15:22:05.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.022+0000 7f8ec5600640 1 -- 172.21.15.47:0/3923682244 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8eb4012c40 con 0x7f8ec80cb490 2024-02-19T15:22:05.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.022+0000 7f8ec5600640 1 --2- 172.21.15.47:0/3923682244 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ea8066e40 0x7f8ea8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:05.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.022+0000 7f8ec5600640 1 -- 172.21.15.47:0/3923682244 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f8eb4084f60 con 0x7f8ec80cb490 2024-02-19T15:22:05.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.022+0000 7f8ecd600640 1 --2- 172.21.15.47:0/3923682244 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ea8066e40 0x7f8ea8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:05.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.026+0000 7f8ecd600640 1 --2- 172.21.15.47:0/3923682244 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ea8066e40 0x7f8ea8069300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc0ad000 tx=0x7f8ebc0a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:05.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.030+0000 7f8ec5600640 1 -- 172.21.15.47:0/3923682244 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8eb4051d10 con 0x7f8ec80cb490 2024-02-19T15:22:05.093 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:22:05.096 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:22:05.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.226+0000 7f8ed0000640 1 -- 172.21.15.47:0/3923682244 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f8ec806e800 con 0x7f8ec80cb490 2024-02-19T15:22:05.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.826+0000 7f8ec5600640 1 -- 172.21.15.47:0/3923682244 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f8eb40553d0 con 0x7f8ec80cb490 2024-02-19T15:22:05.829 INFO:teuthology.orchestra.run.smithi047.stderr:new fs with metadata pool 2 and data pool 3 2024-02-19T15:22:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.830+0000 7f8ed0000640 1 -- 172.21.15.47:0/3923682244 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ea8066e40 msgr2=0x7f8ea8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.830+0000 7f8ed0000640 1 --2- 172.21.15.47:0/3923682244 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ea8066e40 0x7f8ea8069300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc0ad000 tx=0x7f8ebc0a6000 comp rx=0 tx=0).stop 2024-02-19T15:22:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.830+0000 7f8ed0000640 1 -- 172.21.15.47:0/3923682244 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ec80cb490 msgr2=0x7f8ec8152620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.830+0000 7f8ed0000640 1 --2- 172.21.15.47:0/3923682244 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ec80cb490 0x7f8ec8152620 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8eb400d5f0 tx=0x7f8eb400dab0 comp rx=0 tx=0).stop 2024-02-19T15:22:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.830+0000 7f8ed0000640 1 -- 172.21.15.47:0/3923682244 shutdown_connections 2024-02-19T15:22:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.830+0000 7f8ed0000640 1 --2- 172.21.15.47:0/3923682244 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ea8066e40 0x7f8ea8069300 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.830+0000 7f8ed0000640 1 --2- 172.21.15.47:0/3923682244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ec815bd90 0x7f8ec81554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.830+0000 7f8ed0000640 1 --2- 172.21.15.47:0/3923682244 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ec80cb490 0x7f8ec8152620 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.830+0000 7f8ed0000640 1 --2- 172.21.15.47:0/3923682244 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ec8072680 0x7f8ec80ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.830+0000 7f8ed0000640 1 -- 172.21.15.47:0/3923682244 >> 172.21.15.47:0/3923682244 conn(0x7f8ec80cfc20 msgr2=0x7f8ec80c6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:05.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.830+0000 7f8ed0000640 1 -- 172.21.15.47:0/3923682244 shutdown_connections 2024-02-19T15:22:05.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:05.830+0000 7f8ed0000640 1 -- 172.21.15.47:0/3923682244 wait complete. 2024-02-19T15:22:05.878 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:06.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f104a400640 1 Processor -- start 2024-02-19T15:22:06.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f104a400640 1 -- start start 2024-02-19T15:22:06.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f104a400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f104415d6e0 0x7f1044162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:06.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f104a400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10440718c0 con 0x7f104415b1a0 2024-02-19T15:22:06.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f104a400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1044071ba0 con 0x7f1044071350 2024-02-19T15:22:06.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f104a400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1044071e80 con 0x7f104415d6e0 2024-02-19T15:22:06.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1043400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f104415b1a0 0x7f104415b580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43966/0 (socket says 172.21.15.47:43966) 2024-02-19T15:22:06.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1043400640 1 -- 172.21.15.47:0/837725645 learned_addr learned my addr 172.21.15.47:0/837725645 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:06.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1042a00640 1 --2- 172.21.15.47:0/837725645 >> v2:172.21.15.116:3300/0 conn(0x7f104415d6e0 0x7f1044162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:06.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1042000640 1 -- 172.21.15.47:0/837725645 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573492522 0 0) 0x7f10440718c0 con 0x7f104415b1a0 2024-02-19T15:22:06.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1042000640 1 -- 172.21.15.47:0/837725645 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1028003530 con 0x7f104415b1a0 2024-02-19T15:22:06.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1042000640 1 -- 172.21.15.47:0/837725645 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4093321513 0 0) 0x7f1028003530 con 0x7f104415b1a0 2024-02-19T15:22:06.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1042000640 1 -- 172.21.15.47:0/837725645 >> v2:172.21.15.116:3300/0 conn(0x7f104415d6e0 msgr2=0x7f1044162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:06.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1042000640 1 --2- 172.21.15.47:0/837725645 >> v2:172.21.15.116:3300/0 conn(0x7f104415d6e0 0x7f1044162740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1042000640 1 -- 172.21.15.47:0/837725645 >> v1:172.21.15.116:6789/0 conn(0x7f1044071350 legacy=0x7f1044164c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:06.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1042000640 1 -- 172.21.15.47:0/837725645 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1044072160 con 0x7f104415b1a0 2024-02-19T15:22:06.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1042000640 1 -- 172.21.15.47:0/837725645 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f1034001680 con 0x7f104415b1a0 2024-02-19T15:22:06.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1042000640 1 -- 172.21.15.47:0/837725645 >> v1:172.21.15.47:6789/0 conn(0x7f104415b1a0 legacy=0x7f104415b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:06.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1042000640 1 --2- 172.21.15.47:0/837725645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1028003a00 0x7f1028003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:06.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1042000640 1 --2- 172.21.15.47:0/837725645 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f10280047d0 0x7f1028004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:06.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1042000640 1 --2- 172.21.15.47:0/837725645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1028005190 0x7f102800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:06.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1042000640 1 -- 172.21.15.47:0/837725645 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1044071e80 con 0x7f1028003a00 2024-02-19T15:22:06.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1042000640 1 -- 172.21.15.47:0/837725645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102800c0c0 con 0x7f1028005190 2024-02-19T15:22:06.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1042000640 1 -- 172.21.15.47:0/837725645 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102800c280 con 0x7f10280047d0 2024-02-19T15:22:06.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1043400640 1 --2- 172.21.15.47:0/837725645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1028003a00 0x7f1028003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:06.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1042a00640 1 --2- 172.21.15.47:0/837725645 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f10280047d0 0x7f1028004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:06.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1043e00640 1 --2- 172.21.15.47:0/837725645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1028005190 0x7f102800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:06.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1043400640 1 -- 172.21.15.47:0/837725645 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f10280047d0 msgr2=0x7f1028004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:06.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1043400640 1 --2- 172.21.15.47:0/837725645 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f10280047d0 0x7f1028004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1043400640 1 -- 172.21.15.47:0/837725645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1028005190 msgr2=0x7f102800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:06.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1043400640 1 --2- 172.21.15.47:0/837725645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1028005190 0x7f102800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1043400640 1 -- 172.21.15.47:0/837725645 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1028003530 con 0x7f1028003a00 2024-02-19T15:22:06.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1043e00640 1 --2- 172.21.15.47:0/837725645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1028005190 0x7f102800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:06.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.018+0000 7f1043e00640 1 -- 172.21.15.47:0/837725645 reap_dead start 2024-02-19T15:22:06.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.022+0000 7f1043400640 1 --2- 172.21.15.47:0/837725645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1028003a00 0x7f1028003e20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f103400c0e0 tx=0x7f103401cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc9ca49bcd8689d1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:06.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.022+0000 7f1042000640 1 -- 172.21.15.47:0/837725645 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1034005ae0 con 0x7f1028003a00 2024-02-19T15:22:06.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.022+0000 7f1042000640 1 -- 172.21.15.47:0/837725645 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1034005c40 con 0x7f1028003a00 2024-02-19T15:22:06.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.022+0000 7f1042000640 1 -- 172.21.15.47:0/837725645 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10340a5ac0 con 0x7f1028003a00 2024-02-19T15:22:06.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.022+0000 7f104a400640 1 -- 172.21.15.47:0/837725645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1028003a00 msgr2=0x7f1028003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:06.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.022+0000 7f104a400640 1 --2- 172.21.15.47:0/837725645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1028003a00 0x7f1028003e20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f103400c0e0 tx=0x7f103401cb80 comp rx=0 tx=0).stop 2024-02-19T15:22:06.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.022+0000 7f104a400640 1 -- 172.21.15.47:0/837725645 shutdown_connections 2024-02-19T15:22:06.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.022+0000 7f104a400640 1 --2- 172.21.15.47:0/837725645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1028003a00 0x7f1028003e20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.022+0000 7f104a400640 1 -- 172.21.15.47:0/837725645 >> 172.21.15.47:0/837725645 conn(0x7f10440b16a0 msgr2=0x7f10440b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:06.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.022+0000 7f104a400640 1 -- 172.21.15.47:0/837725645 shutdown_connections 2024-02-19T15:22:06.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.022+0000 7f104a400640 1 -- 172.21.15.47:0/837725645 wait complete. 2024-02-19T15:22:06.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f104a400640 1 Processor -- start 2024-02-19T15:22:06.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f104a400640 1 -- start start 2024-02-19T15:22:06.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f104a400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1028003a00 0x7f104413ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:06.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f104a400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1044159ed0 0x7f104415a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:06.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f104a400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1044150940 0x7f1044150dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:06.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f104a400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1044141880 con 0x7f1028003a00 2024-02-19T15:22:06.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f104a400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104415a890 con 0x7f1044159ed0 2024-02-19T15:22:06.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f104a400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f104413f7f0 con 0x7f1044150940 2024-02-19T15:22:06.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f1043400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1028003a00 0x7f104413ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:06.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f1043400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1028003a00 0x7f104413ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52792/0 (socket says 172.21.15.47:52792) 2024-02-19T15:22:06.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f1043400640 1 -- 172.21.15.47:0/2460289789 learned_addr learned my addr 172.21.15.47:0/2460289789 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:06.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f1043e00640 1 --2- 172.21.15.47:0/2460289789 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1044150940 0x7f1044150dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:06.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f1042a00640 1 --2- 172.21.15.47:0/2460289789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1044159ed0 0x7f104415a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:06.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f1043400640 1 -- 172.21.15.47:0/2460289789 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1044150940 msgr2=0x7f1044150dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:06.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f1043400640 1 --2- 172.21.15.47:0/2460289789 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1044150940 0x7f1044150dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f1043400640 1 -- 172.21.15.47:0/2460289789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1044159ed0 msgr2=0x7f104415a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:06.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f1043400640 1 --2- 172.21.15.47:0/2460289789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1044159ed0 0x7f104415a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f1043400640 1 -- 172.21.15.47:0/2460289789 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1044141a90 con 0x7f1028003a00 2024-02-19T15:22:06.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f1043400640 1 --2- 172.21.15.47:0/2460289789 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1028003a00 0x7f104413ef70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f103401c9b0 tx=0x7f1028005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:06.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f103b400640 1 -- 172.21.15.47:0/2460289789 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10340b4070 con 0x7f1028003a00 2024-02-19T15:22:06.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f103b400640 1 -- 172.21.15.47:0/2460289789 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10340a5880 con 0x7f1028003a00 2024-02-19T15:22:06.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f104a400640 1 -- 172.21.15.47:0/2460289789 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1044141d70 con 0x7f1028003a00 2024-02-19T15:22:06.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f103b400640 1 -- 172.21.15.47:0/2460289789 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10340a5c30 con 0x7f1028003a00 2024-02-19T15:22:06.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f104a400640 1 -- 172.21.15.47:0/2460289789 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1044142330 con 0x7f1028003a00 2024-02-19T15:22:06.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.026+0000 7f104a400640 1 -- 172.21.15.47:0/2460289789 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1044075360 con 0x7f1028003a00 2024-02-19T15:22:06.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.030+0000 7f103b400640 1 -- 172.21.15.47:0/2460289789 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f10340b7070 con 0x7f1028003a00 2024-02-19T15:22:06.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.030+0000 7f103b400640 1 --2- 172.21.15.47:0/2460289789 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1024066e70 0x7f1024069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:06.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.030+0000 7f103b400640 1 -- 172.21.15.47:0/2460289789 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f103411e210 con 0x7f1028003a00 2024-02-19T15:22:06.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.030+0000 7f1042a00640 1 --2- 172.21.15.47:0/2460289789 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1024066e70 0x7f1024069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:06.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.030+0000 7f1042a00640 1 --2- 172.21.15.47:0/2460289789 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1024066e70 0x7f1024069330 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f10441e6fb0 tx=0x7f1030008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:06.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.034+0000 7f103b400640 1 -- 172.21.15.47:0/2460289789 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f10340eaf60 con 0x7f1028003a00 2024-02-19T15:22:06.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.198+0000 7f104a400640 1 -- 172.21.15.47:0/2460289789 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f104415e040 con 0x7f1028003a00 2024-02-19T15:22:06.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.198+0000 7f103b400640 1 -- 172.21.15.47:0/2460289789 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16630 (secure 0 0 0) 0x7f10340ee620 con 0x7f1028003a00 2024-02-19T15:22:06.200 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:06.200 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":20,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:05.228072+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:03.091360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:04.222965+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:06.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.198+0000 7f104a400640 1 -- 172.21.15.47:0/2460289789 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1024066e70 msgr2=0x7f1024069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:06.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.198+0000 7f104a400640 1 --2- 172.21.15.47:0/2460289789 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1024066e70 0x7f1024069330 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f10441e6fb0 tx=0x7f1030008040 comp rx=0 tx=0).stop 2024-02-19T15:22:06.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.198+0000 7f104a400640 1 -- 172.21.15.47:0/2460289789 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1028003a00 msgr2=0x7f104413ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:06.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.198+0000 7f104a400640 1 --2- 172.21.15.47:0/2460289789 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1028003a00 0x7f104413ef70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f103401c9b0 tx=0x7f1028005800 comp rx=0 tx=0).stop 2024-02-19T15:22:06.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.198+0000 7f104a400640 1 -- 172.21.15.47:0/2460289789 shutdown_connections 2024-02-19T15:22:06.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.198+0000 7f104a400640 1 --2- 172.21.15.47:0/2460289789 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1024066e70 0x7f1024069330 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.198+0000 7f104a400640 1 --2- 172.21.15.47:0/2460289789 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1044150940 0x7f1044150dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.198+0000 7f104a400640 1 --2- 172.21.15.47:0/2460289789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1044159ed0 0x7f104415a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.198+0000 7f104a400640 1 --2- 172.21.15.47:0/2460289789 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1028003a00 0x7f104413ef70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.198+0000 7f104a400640 1 -- 172.21.15.47:0/2460289789 >> 172.21.15.47:0/2460289789 conn(0x7f10440b16a0 msgr2=0x7f1044070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:06.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.198+0000 7f104a400640 1 -- 172.21.15.47:0/2460289789 shutdown_connections 2024-02-19T15:22:06.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.198+0000 7f104a400640 1 -- 172.21.15.47:0/2460289789 wait complete. 2024-02-19T15:22:06.220 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:06.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.358+0000 7fa498400640 1 Processor -- start 2024-02-19T15:22:06.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.358+0000 7fa498400640 1 -- start start 2024-02-19T15:22:06.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.358+0000 7fa498400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa49015b1a0 0x7fa49015b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:06.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.358+0000 7fa498400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa490071350 0x7fa49015bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:06.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.358+0000 7fa498400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49015c000 con 0x7fa49006c9a0 2024-02-19T15:22:06.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.358+0000 7fa498400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4900718c0 con 0x7fa49015b1a0 2024-02-19T15:22:06.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.358+0000 7fa498400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa490071ba0 con 0x7fa490071350 2024-02-19T15:22:06.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.362+0000 7fa495a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa49015b1a0 0x7fa49015b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:06.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.362+0000 7fa495000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa490071350 0x7fa49015bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:06.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.362+0000 7fa495a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa49015b1a0 0x7fa49015b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:45360/0 (socket says 172.21.15.47:45360) 2024-02-19T15:22:06.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.362+0000 7fa495a00640 1 -- 172.21.15.47:0/621658308 learned_addr learned my addr 172.21.15.47:0/621658308 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:06.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.362+0000 7fa495000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa490071350 0x7fa49015bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:50518/0 (socket says 172.21.15.47:50518) 2024-02-19T15:22:06.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.362+0000 7fa496400640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa49006c9a0 0x7fa49015c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:46168/0 (socket says 172.21.15.47:46168) 2024-02-19T15:22:06.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.362+0000 7fa495a00640 1 -- 172.21.15.47:0/621658308 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa490071350 msgr2=0x7fa49015bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:06.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.362+0000 7fa495a00640 1 --2- 172.21.15.47:0/621658308 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa490071350 0x7fa49015bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.362+0000 7fa495a00640 1 -- 172.21.15.47:0/621658308 >> v1:172.21.15.116:6789/0 conn(0x7fa49006c9a0 legacy=0x7fa49015c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:06.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.362+0000 7fa495a00640 1 -- 172.21.15.47:0/621658308 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa490071e80 con 0x7fa49015b1a0 2024-02-19T15:22:06.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.362+0000 7fa495000640 1 --2- 172.21.15.47:0/621658308 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa490071350 0x7fa49015bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:06.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.362+0000 7fa495a00640 1 --2- 172.21.15.47:0/621658308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa49015b1a0 0x7fa49015b580 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa484009870 tx=0x7fa48401c530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3237ec35224287d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:06.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.362+0000 7fa48fe00640 1 -- 172.21.15.47:0/621658308 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4840a9610 con 0x7fa49015b1a0 2024-02-19T15:22:06.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa48fe00640 1 -- 172.21.15.47:0/621658308 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4840a9770 con 0x7fa49015b1a0 2024-02-19T15:22:06.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa48fe00640 1 -- 172.21.15.47:0/621658308 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4840a9b00 con 0x7fa49015b1a0 2024-02-19T15:22:06.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa498400640 1 -- 172.21.15.47:0/621658308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa49015b1a0 msgr2=0x7fa49015b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:06.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa498400640 1 --2- 172.21.15.47:0/621658308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa49015b1a0 0x7fa49015b580 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa484009870 tx=0x7fa48401c530 comp rx=0 tx=0).stop 2024-02-19T15:22:06.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa498400640 1 -- 172.21.15.47:0/621658308 shutdown_connections 2024-02-19T15:22:06.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa498400640 1 --2- 172.21.15.47:0/621658308 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa490071350 0x7fa49015bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa498400640 1 --2- 172.21.15.47:0/621658308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa49015b1a0 0x7fa49015b580 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa498400640 1 -- 172.21.15.47:0/621658308 >> 172.21.15.47:0/621658308 conn(0x7fa4900b16a0 msgr2=0x7fa4900b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:06.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa498400640 1 -- 172.21.15.47:0/621658308 shutdown_connections 2024-02-19T15:22:06.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa498400640 1 -- 172.21.15.47:0/621658308 wait complete. 2024-02-19T15:22:06.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa498400640 1 Processor -- start 2024-02-19T15:22:06.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa498400640 1 -- start start 2024-02-19T15:22:06.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa498400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa49006c9a0 0x7fa4901f6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:06.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa498400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa490071350 0x7fa490156b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:06.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa498400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa49015b1a0 0x7fa4900b0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:06.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa498400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4901f7c40 con 0x7fa490071350 2024-02-19T15:22:06.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa498400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4901f7030 con 0x7fa49006c9a0 2024-02-19T15:22:06.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa498400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4901f7310 con 0x7fa49015b1a0 2024-02-19T15:22:06.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa495000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa490071350 0x7fa490156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:06.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa495000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa490071350 0x7fa490156b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52802/0 (socket says 172.21.15.47:52802) 2024-02-19T15:22:06.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa495000640 1 -- 172.21.15.47:0/619957085 learned_addr learned my addr 172.21.15.47:0/619957085 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:06.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa495a00640 1 --2- 172.21.15.47:0/619957085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa49006c9a0 0x7fa4901f6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:06.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa496400640 1 --2- 172.21.15.47:0/619957085 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa49015b1a0 0x7fa4900b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:06.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa495000640 1 -- 172.21.15.47:0/619957085 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa49015b1a0 msgr2=0x7fa4900b0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:06.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa495000640 1 --2- 172.21.15.47:0/619957085 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa49015b1a0 0x7fa4900b0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa495000640 1 -- 172.21.15.47:0/619957085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa49006c9a0 msgr2=0x7fa4901f6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:06.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa495000640 1 --2- 172.21.15.47:0/619957085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa49006c9a0 0x7fa4901f6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.366+0000 7fa495000640 1 -- 172.21.15.47:0/619957085 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47c00b6c0 con 0x7fa490071350 2024-02-19T15:22:06.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.370+0000 7fa495000640 1 --2- 172.21.15.47:0/619957085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa490071350 0x7fa490156b80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa47c00ed40 tx=0x7fa47c00d720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:06.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.370+0000 7fa48d600640 1 -- 172.21.15.47:0/619957085 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c010850 con 0x7fa490071350 2024-02-19T15:22:06.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.370+0000 7fa48d600640 1 -- 172.21.15.47:0/619957085 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47c0109b0 con 0x7fa490071350 2024-02-19T15:22:06.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.370+0000 7fa498400640 1 -- 172.21.15.47:0/619957085 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4840094d0 con 0x7fa490071350 2024-02-19T15:22:06.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.370+0000 7fa48d600640 1 -- 172.21.15.47:0/619957085 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c010d20 con 0x7fa490071350 2024-02-19T15:22:06.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.370+0000 7fa498400640 1 -- 172.21.15.47:0/619957085 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4901f78d0 con 0x7fa490071350 2024-02-19T15:22:06.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.370+0000 7fa498400640 1 -- 172.21.15.47:0/619957085 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4900af380 con 0x7fa490071350 2024-02-19T15:22:06.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.370+0000 7fa48d600640 1 -- 172.21.15.47:0/619957085 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa47c00f3d0 con 0x7fa490071350 2024-02-19T15:22:06.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.370+0000 7fa48d600640 1 --2- 172.21.15.47:0/619957085 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa470066e40 0x7fa470069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:06.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.370+0000 7fa48d600640 1 -- 172.21.15.47:0/619957085 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa47c085360 con 0x7fa490071350 2024-02-19T15:22:06.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.370+0000 7fa495a00640 1 --2- 172.21.15.47:0/619957085 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa470066e40 0x7fa470069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:06.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.374+0000 7fa495a00640 1 --2- 172.21.15.47:0/619957085 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa470066e40 0x7fa470069300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa4840ae000 tx=0x7fa4840b3040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:06.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.374+0000 7fa48d600640 1 -- 172.21.15.47:0/619957085 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa47c0521f0 con 0x7fa490071350 2024-02-19T15:22:06.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.550+0000 7fa498400640 1 -- 172.21.15.47:0/619957085 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa49006d1b0 con 0x7fa490071350 2024-02-19T15:22:06.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.554+0000 7fa48d600640 1 -- 172.21.15.47:0/619957085 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16630 (secure 0 0 0) 0x7fa47c0558b0 con 0x7fa490071350 2024-02-19T15:22:06.554 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:06.555 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":20,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:05.228072+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:03.091360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:04.222965+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:06.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.554+0000 7fa498400640 1 -- 172.21.15.47:0/619957085 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa470066e40 msgr2=0x7fa470069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:06.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.554+0000 7fa498400640 1 --2- 172.21.15.47:0/619957085 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa470066e40 0x7fa470069300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa4840ae000 tx=0x7fa4840b3040 comp rx=0 tx=0).stop 2024-02-19T15:22:06.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.554+0000 7fa498400640 1 -- 172.21.15.47:0/619957085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa490071350 msgr2=0x7fa490156b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:06.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.554+0000 7fa498400640 1 --2- 172.21.15.47:0/619957085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa490071350 0x7fa490156b80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa47c00ed40 tx=0x7fa47c00d720 comp rx=0 tx=0).stop 2024-02-19T15:22:06.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.554+0000 7fa498400640 1 -- 172.21.15.47:0/619957085 shutdown_connections 2024-02-19T15:22:06.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.554+0000 7fa498400640 1 --2- 172.21.15.47:0/619957085 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa470066e40 0x7fa470069300 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.554+0000 7fa498400640 1 --2- 172.21.15.47:0/619957085 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa49015b1a0 0x7fa4900b0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.554+0000 7fa498400640 1 --2- 172.21.15.47:0/619957085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa490071350 0x7fa490156b80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.554+0000 7fa498400640 1 --2- 172.21.15.47:0/619957085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa49006c9a0 0x7fa4901f6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.554+0000 7fa498400640 1 -- 172.21.15.47:0/619957085 >> 172.21.15.47:0/619957085 conn(0x7fa4900b16a0 msgr2=0x7fa49015db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:06.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.554+0000 7fa498400640 1 -- 172.21.15.47:0/619957085 shutdown_connections 2024-02-19T15:22:06.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.554+0000 7fa498400640 1 -- 172.21.15.47:0/619957085 wait complete. 2024-02-19T15:22:06.575 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T15:22:06.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.718+0000 7fac03e00640 1 Processor -- start 2024-02-19T15:22:06.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.718+0000 7fac03e00640 1 -- start start 2024-02-19T15:22:06.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.718+0000 7fac03e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fabfc15af10 0x7fabfc06e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:06.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.718+0000 7fac03e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fabfc0ca8f0 0x7fabfc0cad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:06.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.718+0000 7fac03e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabfc071c70 con 0x7fabfc0cb290 2024-02-19T15:22:06.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.718+0000 7fac03e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabfc06e8a0 con 0x7fabfc15af10 2024-02-19T15:22:06.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.718+0000 7fac03e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabfc06cef0 con 0x7fabfc0ca8f0 2024-02-19T15:22:06.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.718+0000 7fac01400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fabfc15af10 0x7fabfc06e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:06.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.718+0000 7fac01e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fabfc0cb290 0x7fabfc15eaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43970/0 (socket says 172.21.15.47:43970) 2024-02-19T15:22:06.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.718+0000 7fac01400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fabfc15af10 0x7fabfc06e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52804/0 (socket says 172.21.15.47:52804) 2024-02-19T15:22:06.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.718+0000 7fac01e00640 1 -- 172.21.15.47:0/3823543638 learned_addr learned my addr 172.21.15.47:0/3823543638 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:06.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.718+0000 7fac00a00640 1 --2- 172.21.15.47:0/3823543638 >> v2:172.21.15.116:3300/0 conn(0x7fabfc0ca8f0 0x7fabfc0cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:06.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.718+0000 7fac01400640 1 -- 172.21.15.47:0/3823543638 >> v2:172.21.15.116:3300/0 conn(0x7fabfc0ca8f0 msgr2=0x7fabfc0cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:06.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.718+0000 7fac01400640 1 --2- 172.21.15.47:0/3823543638 >> v2:172.21.15.116:3300/0 conn(0x7fabfc0ca8f0 0x7fabfc0cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.718+0000 7fac01400640 1 -- 172.21.15.47:0/3823543638 >> v1:172.21.15.47:6789/0 conn(0x7fabfc0cb290 legacy=0x7fabfc15eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:06.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.718+0000 7fac01400640 1 -- 172.21.15.47:0/3823543638 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabfc0ca610 con 0x7fabfc15af10 2024-02-19T15:22:06.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.718+0000 7fabfbe00640 1 -- 172.21.15.47:0/3823543638 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568005774 0 0) 0x7fabfc071c70 con 0x7fabfc0cb290 2024-02-19T15:22:06.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.722+0000 7fac01400640 1 --2- 172.21.15.47:0/3823543638 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fabfc15af10 0x7fabfc06e360 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0b27b0 tx=0x7fabf001ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=906870662fb70f27 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:06.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.722+0000 7fabfbe00640 1 -- 172.21.15.47:0/3823543638 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf00a7b60 con 0x7fabfc15af10 2024-02-19T15:22:06.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.722+0000 7fabfbe00640 1 -- 172.21.15.47:0/3823543638 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf00a7cc0 con 0x7fabfc15af10 2024-02-19T15:22:06.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.722+0000 7fabfbe00640 1 -- 172.21.15.47:0/3823543638 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf00af660 con 0x7fabfc15af10 2024-02-19T15:22:06.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.722+0000 7fac03e00640 1 -- 172.21.15.47:0/3823543638 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fabfc15af10 msgr2=0x7fabfc06e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:06.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.722+0000 7fac03e00640 1 --2- 172.21.15.47:0/3823543638 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fabfc15af10 0x7fabfc06e360 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0b27b0 tx=0x7fabf001ea90 comp rx=0 tx=0).stop 2024-02-19T15:22:06.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.722+0000 7fac03e00640 1 -- 172.21.15.47:0/3823543638 shutdown_connections 2024-02-19T15:22:06.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.722+0000 7fac03e00640 1 --2- 172.21.15.47:0/3823543638 >> v2:172.21.15.116:3300/0 conn(0x7fabfc0ca8f0 0x7fabfc0cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.722+0000 7fac03e00640 1 --2- 172.21.15.47:0/3823543638 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fabfc15af10 0x7fabfc06e360 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.722+0000 7fac03e00640 1 -- 172.21.15.47:0/3823543638 >> 172.21.15.47:0/3823543638 conn(0x7fabfc0cfc20 msgr2=0x7fabfc0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:06.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.722+0000 7fac03e00640 1 -- 172.21.15.47:0/3823543638 shutdown_connections 2024-02-19T15:22:06.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.722+0000 7fac03e00640 1 -- 172.21.15.47:0/3823543638 wait complete. 2024-02-19T15:22:06.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.722+0000 7fac03e00640 1 Processor -- start 2024-02-19T15:22:06.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac03e00640 1 -- start start 2024-02-19T15:22:06.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac03e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabfc0ca8f0 0x7fabfc0c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:06.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac03e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fabfc0cb290 0x7fabfc1560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:06.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac03e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabfc15af10 0x7fabfc158e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:06.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac03e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc0c9c30 con 0x7fabfc0cb290 2024-02-19T15:22:06.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac03e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc0c8650 con 0x7fabfc0ca8f0 2024-02-19T15:22:06.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac03e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc0c8930 con 0x7fabfc15af10 2024-02-19T15:22:06.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac00a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fabfc0cb290 0x7fabfc1560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:06.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac00a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fabfc0cb290 0x7fabfc1560a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52816/0 (socket says 172.21.15.47:52816) 2024-02-19T15:22:06.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac00a00640 1 -- 172.21.15.47:0/538897125 learned_addr learned my addr 172.21.15.47:0/538897125 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:06.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac01400640 1 --2- 172.21.15.47:0/538897125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabfc0ca8f0 0x7fabfc0c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:06.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac01e00640 1 --2- 172.21.15.47:0/538897125 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabfc15af10 0x7fabfc158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:06.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac00a00640 1 -- 172.21.15.47:0/538897125 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabfc15af10 msgr2=0x7fabfc158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:06.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac00a00640 1 --2- 172.21.15.47:0/538897125 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabfc15af10 0x7fabfc158e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac00a00640 1 -- 172.21.15.47:0/538897125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabfc0ca8f0 msgr2=0x7fabfc0c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:06.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac00a00640 1 --2- 172.21.15.47:0/538897125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabfc0ca8f0 0x7fabfc0c96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:06.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac00a00640 1 -- 172.21.15.47:0/538897125 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf000b6c0 con 0x7fabfc0cb290 2024-02-19T15:22:06.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac01400640 1 --2- 172.21.15.47:0/538897125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabfc0ca8f0 0x7fabfc0c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:06.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac00a00640 1 --2- 172.21.15.47:0/538897125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fabfc0cb290 0x7fabfc1560a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fabe800d800 tx=0x7fabe800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:06.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fabf9600640 1 -- 172.21.15.47:0/538897125 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe800b6c0 con 0x7fabfc0cb290 2024-02-19T15:22:06.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fabf9600640 1 -- 172.21.15.47:0/538897125 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe800b820 con 0x7fabfc0cb290 2024-02-19T15:22:06.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac03e00640 1 -- 172.21.15.47:0/538897125 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabfc0c8c10 con 0x7fabfc0cb290 2024-02-19T15:22:06.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fabf9600640 1 -- 172.21.15.47:0/538897125 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe800bb90 con 0x7fabfc0cb290 2024-02-19T15:22:06.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac03e00640 1 -- 172.21.15.47:0/538897125 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabfc0c91d0 con 0x7fabfc0cb290 2024-02-19T15:22:06.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.726+0000 7fac03e00640 1 -- 172.21.15.47:0/538897125 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabfc06d1d0 con 0x7fabfc0cb290 2024-02-19T15:22:06.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.730+0000 7fabf9600640 1 -- 172.21.15.47:0/538897125 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fabe800bcf0 con 0x7fabfc0cb290 2024-02-19T15:22:06.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.730+0000 7fabf9600640 1 --2- 172.21.15.47:0/538897125 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fabdc066e40 0x7fabdc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:06.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.730+0000 7fabf9600640 1 -- 172.21.15.47:0/538897125 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fabe8084ea0 con 0x7fabfc0cb290 2024-02-19T15:22:06.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.730+0000 7fac01400640 1 --2- 172.21.15.47:0/538897125 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fabdc066e40 0x7fabdc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:06.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.730+0000 7fac01400640 1 --2- 172.21.15.47:0/538897125 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fabdc066e40 0x7fabdc069300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fabf00b1000 tx=0x7fabf0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:06.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.734+0000 7fabf9600640 1 -- 172.21.15.47:0/538897125 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fabe8051d30 con 0x7fabfc0cb290 2024-02-19T15:22:06.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:06.934+0000 7fac03e00640 1 -- 172.21.15.47:0/538897125 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fabfc075360 con 0x7fabfc0cb290 2024-02-19T15:22:07.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:07.850+0000 7fabf9600640 1 -- 172.21.15.47:0/538897125 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7fabe80553f0 con 0x7fabfc0cb290 2024-02-19T15:22:07.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:07.854+0000 7fac03e00640 1 -- 172.21.15.47:0/538897125 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fabdc066e40 msgr2=0x7fabdc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:07.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:07.854+0000 7fac03e00640 1 --2- 172.21.15.47:0/538897125 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fabdc066e40 0x7fabdc069300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fabf00b1000 tx=0x7fabf0008040 comp rx=0 tx=0).stop 2024-02-19T15:22:07.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:07.854+0000 7fac03e00640 1 -- 172.21.15.47:0/538897125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fabfc0cb290 msgr2=0x7fabfc1560a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:07.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:07.854+0000 7fac03e00640 1 --2- 172.21.15.47:0/538897125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fabfc0cb290 0x7fabfc1560a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fabe800d800 tx=0x7fabe800dcc0 comp rx=0 tx=0).stop 2024-02-19T15:22:07.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:07.854+0000 7fac03e00640 1 -- 172.21.15.47:0/538897125 shutdown_connections 2024-02-19T15:22:07.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:07.854+0000 7fac03e00640 1 --2- 172.21.15.47:0/538897125 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fabdc066e40 0x7fabdc069300 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:07.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:07.854+0000 7fac03e00640 1 --2- 172.21.15.47:0/538897125 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabfc15af10 0x7fabfc158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:07.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:07.854+0000 7fac03e00640 1 --2- 172.21.15.47:0/538897125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fabfc0cb290 0x7fabfc1560a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:07.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:07.854+0000 7fac03e00640 1 --2- 172.21.15.47:0/538897125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabfc0ca8f0 0x7fabfc0c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:07.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:07.854+0000 7fac03e00640 1 -- 172.21.15.47:0/538897125 >> 172.21.15.47:0/538897125 conn(0x7fabfc0cfc20 msgr2=0x7fabfc0c4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:07.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:07.854+0000 7fac03e00640 1 -- 172.21.15.47:0/538897125 shutdown_connections 2024-02-19T15:22:07.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:07.854+0000 7fac03e00640 1 -- 172.21.15.47:0/538897125 wait complete. 2024-02-19T15:22:07.880 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T15:22:07.880 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T15:22:08.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.026+0000 7fd851200640 1 Processor -- start 2024-02-19T15:22:08.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.026+0000 7fd851200640 1 -- start start 2024-02-19T15:22:08.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.026+0000 7fd851200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd84c15bd90 0x7fd84c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:08.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.026+0000 7fd851200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd84c0cb490 0x7fd84c072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:08.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.026+0000 7fd851200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd84c072680 0x7fd84c0ca270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:08.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.026+0000 7fd851200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd84c15fc90 con 0x7fd84c15bd90 2024-02-19T15:22:08.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.026+0000 7fd851200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd84c15ff70 con 0x7fd84c0cb490 2024-02-19T15:22:08.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.026+0000 7fd851200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c160250 con 0x7fd84c072680 2024-02-19T15:22:08.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.026+0000 7fd849e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd84c15bd90 0x7fd84c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:08.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.026+0000 7fd841400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd84c0cb490 0x7fd84c072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:08.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.026+0000 7fd849e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd84c15bd90 0x7fd84c070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52820/0 (socket says 172.21.15.47:52820) 2024-02-19T15:22:08.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.026+0000 7fd849e00640 1 -- 172.21.15.47:0/4071443127 learned_addr learned my addr 172.21.15.47:0/4071443127 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:08.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.026+0000 7fd84a800640 1 --2- 172.21.15.47:0/4071443127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd84c072680 0x7fd84c0ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:08.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.026+0000 7fd849e00640 1 -- 172.21.15.47:0/4071443127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd84c072680 msgr2=0x7fd84c0ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:08.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.026+0000 7fd849e00640 1 --2- 172.21.15.47:0/4071443127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd84c072680 0x7fd84c0ca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:08.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.026+0000 7fd849e00640 1 -- 172.21.15.47:0/4071443127 >> v2:172.21.15.116:3300/0 conn(0x7fd84c0cb490 msgr2=0x7fd84c072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:08.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.026+0000 7fd849e00640 1 --2- 172.21.15.47:0/4071443127 >> v2:172.21.15.116:3300/0 conn(0x7fd84c0cb490 0x7fd84c072140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:08.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.026+0000 7fd849e00640 1 -- 172.21.15.47:0/4071443127 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84c160530 con 0x7fd84c15bd90 2024-02-19T15:22:08.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.030+0000 7fd849e00640 1 --2- 172.21.15.47:0/4071443127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd84c15bd90 0x7fd84c070960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd84c06c890 tx=0x7fd83c01c100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c010eee02f36f5b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:08.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.030+0000 7fd849400640 1 -- 172.21.15.47:0/4071443127 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0ab600 con 0x7fd84c15bd90 2024-02-19T15:22:08.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.030+0000 7fd849400640 1 -- 172.21.15.47:0/4071443127 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83c0a7050 con 0x7fd84c15bd90 2024-02-19T15:22:08.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.030+0000 7fd849400640 1 -- 172.21.15.47:0/4071443127 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0ab980 con 0x7fd84c15bd90 2024-02-19T15:22:08.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.030+0000 7fd851200640 1 -- 172.21.15.47:0/4071443127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd84c15bd90 msgr2=0x7fd84c070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:08.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.030+0000 7fd851200640 1 --2- 172.21.15.47:0/4071443127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd84c15bd90 0x7fd84c070960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd84c06c890 tx=0x7fd83c01c100 comp rx=0 tx=0).stop 2024-02-19T15:22:08.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.030+0000 7fd851200640 1 -- 172.21.15.47:0/4071443127 shutdown_connections 2024-02-19T15:22:08.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.030+0000 7fd851200640 1 --2- 172.21.15.47:0/4071443127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd84c072680 0x7fd84c0ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:08.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.030+0000 7fd851200640 1 --2- 172.21.15.47:0/4071443127 >> v2:172.21.15.116:3300/0 conn(0x7fd84c0cb490 0x7fd84c072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:08.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.030+0000 7fd851200640 1 --2- 172.21.15.47:0/4071443127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd84c15bd90 0x7fd84c070960 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:08.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.030+0000 7fd851200640 1 -- 172.21.15.47:0/4071443127 >> 172.21.15.47:0/4071443127 conn(0x7fd84c0cfc20 msgr2=0x7fd84c0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:08.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.030+0000 7fd851200640 1 -- 172.21.15.47:0/4071443127 shutdown_connections 2024-02-19T15:22:08.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.030+0000 7fd851200640 1 -- 172.21.15.47:0/4071443127 wait complete. 2024-02-19T15:22:08.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd851200640 1 Processor -- start 2024-02-19T15:22:08.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd851200640 1 -- start start 2024-02-19T15:22:08.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd851200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd84c072680 0x7fd84c15cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:08.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd851200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd84c0cb490 0x7fd84c15a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd851200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd84c15bd90 0x7fd84c0ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd851200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c15d080 con 0x7fd84c072680 2024-02-19T15:22:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd851200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c0cd7f0 con 0x7fd84c15bd90 2024-02-19T15:22:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd851200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c15aae0 con 0x7fd84c0cb490 2024-02-19T15:22:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd849e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd84c072680 0x7fd84c15cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd849e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd84c072680 0x7fd84c15cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52830/0 (socket says 172.21.15.47:52830) 2024-02-19T15:22:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd849e00640 1 -- 172.21.15.47:0/1569265111 learned_addr learned my addr 172.21.15.47:0/1569265111 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd84a800640 1 --2- 172.21.15.47:0/1569265111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd84c15bd90 0x7fd84c0ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd841400640 1 --2- 172.21.15.47:0/1569265111 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd84c0cb490 0x7fd84c15a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd849e00640 1 -- 172.21.15.47:0/1569265111 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd84c0cb490 msgr2=0x7fd84c15a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd849e00640 1 --2- 172.21.15.47:0/1569265111 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd84c0cb490 0x7fd84c15a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd849e00640 1 -- 172.21.15.47:0/1569265111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd84c15bd90 msgr2=0x7fd84c0ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd849e00640 1 --2- 172.21.15.47:0/1569265111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd84c15bd90 0x7fd84c0ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd849e00640 1 -- 172.21.15.47:0/1569265111 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83c0094d0 con 0x7fd84c072680 2024-02-19T15:22:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd849e00640 1 --2- 172.21.15.47:0/1569265111 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd84c072680 0x7fd84c15cb40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd83c01a7b0 tx=0x7fd83c018ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd842a00640 1 -- 172.21.15.47:0/1569265111 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0b4420 con 0x7fd84c072680 2024-02-19T15:22:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd842a00640 1 -- 172.21.15.47:0/1569265111 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83c0b4580 con 0x7fd84c072680 2024-02-19T15:22:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd842a00640 1 -- 172.21.15.47:0/1569265111 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0b4930 con 0x7fd84c072680 2024-02-19T15:22:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd851200640 1 -- 172.21.15.47:0/1569265111 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd84c0cda40 con 0x7fd84c072680 2024-02-19T15:22:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.034+0000 7fd851200640 1 -- 172.21.15.47:0/1569265111 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd84c0ce000 con 0x7fd84c072680 2024-02-19T15:22:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.038+0000 7fd851200640 1 -- 172.21.15.47:0/1569265111 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd84c075360 con 0x7fd84c072680 2024-02-19T15:22:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.038+0000 7fd842a00640 1 -- 172.21.15.47:0/1569265111 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd83c0b2070 con 0x7fd84c072680 2024-02-19T15:22:08.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.038+0000 7fd842a00640 1 --2- 172.21.15.47:0/1569265111 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd828066e40 0x7fd828069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:08.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.038+0000 7fd842a00640 1 -- 172.21.15.47:0/1569265111 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd83c11e400 con 0x7fd84c072680 2024-02-19T15:22:08.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.038+0000 7fd841400640 1 --2- 172.21.15.47:0/1569265111 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd828066e40 0x7fd828069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:08.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.038+0000 7fd841400640 1 --2- 172.21.15.47:0/1569265111 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd828066e40 0x7fd828069300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd84c0ac200 tx=0x7fd834009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:08.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.042+0000 7fd842a00640 1 -- 172.21.15.47:0/1569265111 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd83c0eb2a0 con 0x7fd84c072680 2024-02-19T15:22:08.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.238+0000 7fd851200640 1 -- 172.21.15.47:0/1569265111 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd84c06e800 con 0x7fd84c072680 2024-02-19T15:22:08.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.854+0000 7fd842a00640 1 -- 172.21.15.47:0/1569265111 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd83c0ee960 con 0x7fd84c072680 2024-02-19T15:22:08.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.854+0000 7fd851200640 1 -- 172.21.15.47:0/1569265111 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd828066e40 msgr2=0x7fd828069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:08.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.854+0000 7fd851200640 1 --2- 172.21.15.47:0/1569265111 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd828066e40 0x7fd828069300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd84c0ac200 tx=0x7fd834009190 comp rx=0 tx=0).stop 2024-02-19T15:22:08.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.854+0000 7fd851200640 1 -- 172.21.15.47:0/1569265111 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd84c072680 msgr2=0x7fd84c15cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:08.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.854+0000 7fd851200640 1 --2- 172.21.15.47:0/1569265111 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd84c072680 0x7fd84c15cb40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd83c01a7b0 tx=0x7fd83c018ea0 comp rx=0 tx=0).stop 2024-02-19T15:22:08.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.858+0000 7fd851200640 1 -- 172.21.15.47:0/1569265111 shutdown_connections 2024-02-19T15:22:08.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.858+0000 7fd851200640 1 --2- 172.21.15.47:0/1569265111 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd828066e40 0x7fd828069300 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:08.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.858+0000 7fd851200640 1 --2- 172.21.15.47:0/1569265111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd84c15bd90 0x7fd84c0ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:08.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.858+0000 7fd851200640 1 --2- 172.21.15.47:0/1569265111 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd84c0cb490 0x7fd84c15a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:08.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.858+0000 7fd851200640 1 --2- 172.21.15.47:0/1569265111 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd84c072680 0x7fd84c15cb40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:08.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.858+0000 7fd851200640 1 -- 172.21.15.47:0/1569265111 >> 172.21.15.47:0/1569265111 conn(0x7fd84c0cfc20 msgr2=0x7fd84c0c9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:08.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.858+0000 7fd851200640 1 -- 172.21.15.47:0/1569265111 shutdown_connections 2024-02-19T15:22:08.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:08.858+0000 7fd851200640 1 -- 172.21.15.47:0/1569265111 wait complete. 2024-02-19T15:22:08.883 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T15:22:08.883 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:22:09.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.026+0000 7fe3b7c00640 1 Processor -- start 2024-02-19T15:22:09.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.026+0000 7fe3b7c00640 1 -- start start 2024-02-19T15:22:09.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.026+0000 7fe3b7c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe3b015d6e0 0x7fe3b0162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:09.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.026+0000 7fe3b7c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe3b0071350 0x7fe3b0164c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:09.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.026+0000 7fe3b7c00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3b00718c0 con 0x7fe3b015b1a0 2024-02-19T15:22:09.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.026+0000 7fe3b7c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3b0071ba0 con 0x7fe3b0071350 2024-02-19T15:22:09.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.026+0000 7fe3b7c00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3b0071e80 con 0x7fe3b015d6e0 2024-02-19T15:22:09.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.026+0000 7fe3b5200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fe3b015b1a0 0x7fe3b015b580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43980/0 (socket says 172.21.15.47:43980) 2024-02-19T15:22:09.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.026+0000 7fe3b5200640 1 -- 172.21.15.47:0/67399592 learned_addr learned my addr 172.21.15.47:0/67399592 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:09.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.026+0000 7fe3b5c00640 1 --2- 172.21.15.47:0/67399592 >> v2:172.21.15.47:3300/0 conn(0x7fe3b0071350 0x7fe3b0164c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:09.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.026+0000 7fe3afe00640 1 --2- 172.21.15.47:0/67399592 >> v2:172.21.15.116:3300/0 conn(0x7fe3b015d6e0 0x7fe3b0162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:09.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.030+0000 7fe3af400640 1 -- 172.21.15.47:0/67399592 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331156425 0 0) 0x7fe3b00718c0 con 0x7fe3b015b1a0 2024-02-19T15:22:09.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.030+0000 7fe3af400640 1 -- 172.21.15.47:0/67399592 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe398005740 con 0x7fe3b015b1a0 2024-02-19T15:22:09.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.030+0000 7fe3b5c00640 1 -- 172.21.15.47:0/67399592 >> v2:172.21.15.116:3300/0 conn(0x7fe3b015d6e0 msgr2=0x7fe3b0162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:09.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.030+0000 7fe3b5c00640 1 --2- 172.21.15.47:0/67399592 >> v2:172.21.15.116:3300/0 conn(0x7fe3b015d6e0 0x7fe3b0162740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.030+0000 7fe3b5c00640 1 -- 172.21.15.47:0/67399592 >> v1:172.21.15.47:6789/0 conn(0x7fe3b015b1a0 legacy=0x7fe3b015b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:09.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.030+0000 7fe3b5c00640 1 -- 172.21.15.47:0/67399592 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b0072160 con 0x7fe3b0071350 2024-02-19T15:22:09.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.030+0000 7fe3afe00640 1 --2- 172.21.15.47:0/67399592 >> v2:172.21.15.116:3300/0 conn(0x7fe3b015d6e0 0x7fe3b0162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:09.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.030+0000 7fe3b5c00640 1 --2- 172.21.15.47:0/67399592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3b0071350 0x7fe3b0164c90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8009d90 tx=0x7fe3a801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4d61e6b57799ada server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:09.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.030+0000 7fe3af400640 1 -- 172.21.15.47:0/67399592 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a80a5910 con 0x7fe3b0071350 2024-02-19T15:22:09.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.030+0000 7fe3af400640 1 -- 172.21.15.47:0/67399592 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a80a5a70 con 0x7fe3b0071350 2024-02-19T15:22:09.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.030+0000 7fe3af400640 1 -- 172.21.15.47:0/67399592 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a80a5e00 con 0x7fe3b0071350 2024-02-19T15:22:09.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.030+0000 7fe3b7c00640 1 -- 172.21.15.47:0/67399592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3b0071350 msgr2=0x7fe3b0164c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:09.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.030+0000 7fe3b7c00640 1 --2- 172.21.15.47:0/67399592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3b0071350 0x7fe3b0164c90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8009d90 tx=0x7fe3a801c930 comp rx=0 tx=0).stop 2024-02-19T15:22:09.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.030+0000 7fe3b7c00640 1 -- 172.21.15.47:0/67399592 shutdown_connections 2024-02-19T15:22:09.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.030+0000 7fe3b7c00640 1 --2- 172.21.15.47:0/67399592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3b0071350 0x7fe3b0164c90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.030+0000 7fe3b7c00640 1 --2- 172.21.15.47:0/67399592 >> v2:172.21.15.116:3300/0 conn(0x7fe3b015d6e0 0x7fe3b0162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.030+0000 7fe3b7c00640 1 -- 172.21.15.47:0/67399592 >> 172.21.15.47:0/67399592 conn(0x7fe3b00b16a0 msgr2=0x7fe3b00b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:09.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.030+0000 7fe3b7c00640 1 -- 172.21.15.47:0/67399592 shutdown_connections 2024-02-19T15:22:09.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3b7c00640 1 -- 172.21.15.47:0/67399592 wait complete. 2024-02-19T15:22:09.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3b7c00640 1 Processor -- start 2024-02-19T15:22:09.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3b7c00640 1 -- start start 2024-02-19T15:22:09.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3b7c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3b0071350 0x7fe3b0147e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:09.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3b7c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3b015b1a0 0x7fe3b00b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:09.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3b7c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe3b015d6e0 0x7fe3b0146270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:09.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3b7c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b01467b0 con 0x7fe3b0071350 2024-02-19T15:22:09.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3b7c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b0149280 con 0x7fe3b015b1a0 2024-02-19T15:22:09.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3b7c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b01485f0 con 0x7fe3b015d6e0 2024-02-19T15:22:09.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3b5200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3b0071350 0x7fe3b0147e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:09.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3b5200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3b0071350 0x7fe3b0147e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52848/0 (socket says 172.21.15.47:52848) 2024-02-19T15:22:09.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3afe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3b015b1a0 0x7fe3b00b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:09.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3b5200640 1 -- 172.21.15.47:0/3235341440 learned_addr learned my addr 172.21.15.47:0/3235341440 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:09.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3b5c00640 1 --2- 172.21.15.47:0/3235341440 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe3b015d6e0 0x7fe3b0146270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:09.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3b5200640 1 -- 172.21.15.47:0/3235341440 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe3b015d6e0 msgr2=0x7fe3b0146270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:09.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3b5200640 1 --2- 172.21.15.47:0/3235341440 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe3b015d6e0 0x7fe3b0146270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3b5200640 1 -- 172.21.15.47:0/3235341440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3b015b1a0 msgr2=0x7fe3b00b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:09.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3b5200640 1 --2- 172.21.15.47:0/3235341440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3b015b1a0 0x7fe3b00b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3b5200640 1 -- 172.21.15.47:0/3235341440 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39c009420 con 0x7fe3b0071350 2024-02-19T15:22:09.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3b5200640 1 --2- 172.21.15.47:0/3235341440 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3b0071350 0x7fe3b0147e00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe3a400c3c0 tx=0x7fe3a400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:09.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3acc00640 1 -- 172.21.15.47:0/3235341440 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4005cf0 con 0x7fe3b0071350 2024-02-19T15:22:09.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3acc00640 1 -- 172.21.15.47:0/3235341440 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a4005e50 con 0x7fe3b0071350 2024-02-19T15:22:09.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3acc00640 1 -- 172.21.15.47:0/3235341440 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a40072f0 con 0x7fe3b0071350 2024-02-19T15:22:09.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.034+0000 7fe3b7c00640 1 -- 172.21.15.47:0/3235341440 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a8009a70 con 0x7fe3b0071350 2024-02-19T15:22:09.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.038+0000 7fe3b7c00640 1 -- 172.21.15.47:0/3235341440 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b0149520 con 0x7fe3b0071350 2024-02-19T15:22:09.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.038+0000 7fe3b7c00640 1 -- 172.21.15.47:0/3235341440 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3b0075360 con 0x7fe3b0071350 2024-02-19T15:22:09.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.038+0000 7fe3acc00640 1 -- 172.21.15.47:0/3235341440 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe3a4007450 con 0x7fe3b0071350 2024-02-19T15:22:09.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.038+0000 7fe3acc00640 1 --2- 172.21.15.47:0/3235341440 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe390066e40 0x7fe390069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:09.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.038+0000 7fe3acc00640 1 -- 172.21.15.47:0/3235341440 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3a4084050 con 0x7fe3b0071350 2024-02-19T15:22:09.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.038+0000 7fe3afe00640 1 --2- 172.21.15.47:0/3235341440 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe390066e40 0x7fe390069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:09.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.038+0000 7fe3afe00640 1 --2- 172.21.15.47:0/3235341440 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe390066e40 0x7fe390069300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0148370 tx=0x7fe39c012040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:09.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.042+0000 7fe3acc00640 1 -- 172.21.15.47:0/3235341440 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe3a40b61c0 con 0x7fe3b0071350 2024-02-19T15:22:09.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.246+0000 7fe3b7c00640 1 -- 172.21.15.47:0/3235341440 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3b0151f60 con 0x7fe3b0071350 2024-02-19T15:22:09.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.246+0000 7fe3acc00640 1 -- 172.21.15.47:0/3235341440 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4935 (secure 0 0 0) 0x7fe3a4050e50 con 0x7fe3b0071350 2024-02-19T15:22:09.249 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:09.249 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3873760209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3873760209},{"type":"v1","addr":"172.21.15.116:6836","nonce":3873760209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/284820350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":284820350},{"type":"v1","addr":"172.21.15.47:6834","nonce":284820350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2764089359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2764089359},{"type":"v1","addr":"172.21.15.47:6835","nonce":2764089359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:22:05.228517+0000","modified":"2024-02-19T15:22:08.242600+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6837/3920452343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3920452343},{"type":"v1","addr":"172.21.15.116:6837","nonce":3920452343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-19T15:22:09.249 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 8 2024-02-19T15:22:09.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.250+0000 7fe3b7c00640 1 -- 172.21.15.47:0/3235341440 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe390066e40 msgr2=0x7fe390069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:09.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.250+0000 7fe3b7c00640 1 --2- 172.21.15.47:0/3235341440 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe390066e40 0x7fe390069300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0148370 tx=0x7fe39c012040 comp rx=0 tx=0).stop 2024-02-19T15:22:09.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.250+0000 7fe3b7c00640 1 -- 172.21.15.47:0/3235341440 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3b0071350 msgr2=0x7fe3b0147e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:09.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.250+0000 7fe3b7c00640 1 --2- 172.21.15.47:0/3235341440 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3b0071350 0x7fe3b0147e00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe3a400c3c0 tx=0x7fe3a400c880 comp rx=0 tx=0).stop 2024-02-19T15:22:09.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.250+0000 7fe3b7c00640 1 -- 172.21.15.47:0/3235341440 shutdown_connections 2024-02-19T15:22:09.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.250+0000 7fe3b7c00640 1 --2- 172.21.15.47:0/3235341440 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe390066e40 0x7fe390069300 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.250+0000 7fe3b7c00640 1 --2- 172.21.15.47:0/3235341440 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe3b015d6e0 0x7fe3b0146270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.250+0000 7fe3b7c00640 1 --2- 172.21.15.47:0/3235341440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3b015b1a0 0x7fe3b00b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.250+0000 7fe3b7c00640 1 --2- 172.21.15.47:0/3235341440 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3b0071350 0x7fe3b0147e00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.250+0000 7fe3b7c00640 1 -- 172.21.15.47:0/3235341440 >> 172.21.15.47:0/3235341440 conn(0x7fe3b00b16a0 msgr2=0x7fe3b015dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:09.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.250+0000 7fe3b7c00640 1 -- 172.21.15.47:0/3235341440 shutdown_connections 2024-02-19T15:22:09.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.250+0000 7fe3b7c00640 1 -- 172.21.15.47:0/3235341440 wait complete. 2024-02-19T15:22:09.270 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:09.340 INFO:tasks.ceph.mds.d.smithi116.stderr:2024-02-19T15:22:09.338+0000 7fa135c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T15:22:09.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.410+0000 7f4b87000640 1 Processor -- start 2024-02-19T15:22:09.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.410+0000 7f4b87000640 1 -- start start 2024-02-19T15:22:09.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.410+0000 7f4b87000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4b800b4240 0x7f4b800d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:09.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.410+0000 7f4b87000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b800d1070 con 0x7f4b800ce400 2024-02-19T15:22:09.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.410+0000 7f4b87000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b8006ec60 con 0x7f4b8015dfa0 2024-02-19T15:22:09.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.410+0000 7f4b87000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b8006ef40 con 0x7f4b800b4240 2024-02-19T15:22:09.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.410+0000 7f4b77400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4b800b4240 0x7f4b800d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:09.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.410+0000 7f4b85000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f4b800ce400 0x7f4b800cc1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43982/0 (socket says 172.21.15.47:43982) 2024-02-19T15:22:09.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.410+0000 7f4b85000640 1 -- 172.21.15.47:0/4009655789 learned_addr learned my addr 172.21.15.47:0/4009655789 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:09.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.410+0000 7f4b77400640 1 -- 172.21.15.47:0/4009655789 >> v1:172.21.15.116:6789/0 conn(0x7f4b8015dfa0 legacy=0x7f4b8006fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:09.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.410+0000 7f4b77400640 1 -- 172.21.15.47:0/4009655789 >> v1:172.21.15.47:6789/0 conn(0x7f4b800ce400 legacy=0x7f4b800cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:09.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.410+0000 7f4b77400640 1 -- 172.21.15.47:0/4009655789 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8006f220 con 0x7f4b800b4240 2024-02-19T15:22:09.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.414+0000 7f4b77400640 1 --2- 172.21.15.47:0/4009655789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b800b4240 0x7f4b800d0820 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4b680097c0 tx=0x7f4b6801c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=480941518f1ae68c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:09.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.414+0000 7f4b7f400640 1 -- 172.21.15.47:0/4009655789 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b680a7070 con 0x7f4b800b4240 2024-02-19T15:22:09.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.414+0000 7f4b7f400640 1 -- 172.21.15.47:0/4009655789 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b680ab430 con 0x7f4b800b4240 2024-02-19T15:22:09.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.414+0000 7f4b7f400640 1 -- 172.21.15.47:0/4009655789 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b680ab7c0 con 0x7f4b800b4240 2024-02-19T15:22:09.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.414+0000 7f4b87000640 1 -- 172.21.15.47:0/4009655789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b800b4240 msgr2=0x7f4b800d0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:09.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.414+0000 7f4b87000640 1 --2- 172.21.15.47:0/4009655789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b800b4240 0x7f4b800d0820 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4b680097c0 tx=0x7f4b6801c470 comp rx=0 tx=0).stop 2024-02-19T15:22:09.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.414+0000 7f4b87000640 1 -- 172.21.15.47:0/4009655789 shutdown_connections 2024-02-19T15:22:09.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.414+0000 7f4b87000640 1 --2- 172.21.15.47:0/4009655789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b800b4240 0x7f4b800d0820 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.414+0000 7f4b87000640 1 -- 172.21.15.47:0/4009655789 >> 172.21.15.47:0/4009655789 conn(0x7f4b800b3690 msgr2=0x7f4b800b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:09.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.414+0000 7f4b87000640 1 -- 172.21.15.47:0/4009655789 shutdown_connections 2024-02-19T15:22:09.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.414+0000 7f4b87000640 1 -- 172.21.15.47:0/4009655789 wait complete. 2024-02-19T15:22:09.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b87000640 1 Processor -- start 2024-02-19T15:22:09.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b87000640 1 -- start start 2024-02-19T15:22:09.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b87000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b800b4240 0x7f4b80152740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:09.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b87000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b800ce400 0x7f4b800c7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:09.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b87000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b8015dfa0 0x7f4b800c79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:09.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b87000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b800c7090 con 0x7f4b8015dfa0 2024-02-19T15:22:09.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b87000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b80152c80 con 0x7f4b800b4240 2024-02-19T15:22:09.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b87000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b800c6db0 con 0x7f4b800ce400 2024-02-19T15:22:09.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b85000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b8015dfa0 0x7f4b800c79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:09.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b7fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b800b4240 0x7f4b80152740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:09.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b85000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b8015dfa0 0x7f4b800c79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52860/0 (socket says 172.21.15.47:52860) 2024-02-19T15:22:09.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b85000640 1 -- 172.21.15.47:0/153994335 learned_addr learned my addr 172.21.15.47:0/153994335 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:09.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b77400640 1 --2- 172.21.15.47:0/153994335 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b800ce400 0x7f4b800c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:09.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b85000640 1 -- 172.21.15.47:0/153994335 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b800ce400 msgr2=0x7f4b800c7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:09.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b85000640 1 --2- 172.21.15.47:0/153994335 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b800ce400 0x7f4b800c7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b85000640 1 -- 172.21.15.47:0/153994335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b800b4240 msgr2=0x7f4b80152740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:09.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b85000640 1 --2- 172.21.15.47:0/153994335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b800b4240 0x7f4b80152740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b85000640 1 -- 172.21.15.47:0/153994335 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b68009420 con 0x7f4b8015dfa0 2024-02-19T15:22:09.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b7fe00640 1 --2- 172.21.15.47:0/153994335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b800b4240 0x7f4b80152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:09.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b85000640 1 --2- 172.21.15.47:0/153994335 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b8015dfa0 0x7f4b800c79b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4b7800c400 tx=0x7f4b7800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:09.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b7cc00640 1 -- 172.21.15.47:0/153994335 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b78005cf0 con 0x7f4b8015dfa0 2024-02-19T15:22:09.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b7cc00640 1 -- 172.21.15.47:0/153994335 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b78005e50 con 0x7f4b8015dfa0 2024-02-19T15:22:09.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b87000640 1 -- 172.21.15.47:0/153994335 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b800ceda0 con 0x7f4b8015dfa0 2024-02-19T15:22:09.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b7cc00640 1 -- 172.21.15.47:0/153994335 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b780072d0 con 0x7f4b8015dfa0 2024-02-19T15:22:09.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.418+0000 7f4b87000640 1 -- 172.21.15.47:0/153994335 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8015e980 con 0x7f4b8015dfa0 2024-02-19T15:22:09.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.422+0000 7f4b87000640 1 -- 172.21.15.47:0/153994335 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b80075360 con 0x7f4b8015dfa0 2024-02-19T15:22:09.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.422+0000 7f4b7cc00640 1 -- 172.21.15.47:0/153994335 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4b78007430 con 0x7f4b8015dfa0 2024-02-19T15:22:09.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.422+0000 7f4b7cc00640 1 --2- 172.21.15.47:0/153994335 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4b5c066e40 0x7f4b5c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:09.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.422+0000 7f4b7cc00640 1 -- 172.21.15.47:0/153994335 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b78083ef0 con 0x7f4b8015dfa0 2024-02-19T15:22:09.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.422+0000 7f4b7fe00640 1 --2- 172.21.15.47:0/153994335 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4b5c066e40 0x7f4b5c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:09.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.422+0000 7f4b7fe00640 1 --2- 172.21.15.47:0/153994335 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4b5c066e40 0x7f4b5c069300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4b70002200 tx=0x7f4b70009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:09.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.426+0000 7f4b7cc00640 1 -- 172.21.15.47:0/153994335 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f4b78050d90 con 0x7f4b8015dfa0 2024-02-19T15:22:09.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.590+0000 7f4b87000640 1 -- 172.21.15.47:0/153994335 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b800cc6c0 con 0x7f4b8015dfa0 2024-02-19T15:22:09.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.590+0000 7f4b7cc00640 1 -- 172.21.15.47:0/153994335 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f4b78017090 con 0x7f4b8015dfa0 2024-02-19T15:22:09.594 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:09.594 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":21,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:06.843155+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:03.091360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:04.222965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:09.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.594+0000 7f4b87000640 1 -- 172.21.15.47:0/153994335 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4b5c066e40 msgr2=0x7f4b5c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:09.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.594+0000 7f4b87000640 1 --2- 172.21.15.47:0/153994335 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4b5c066e40 0x7f4b5c069300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4b70002200 tx=0x7f4b70009040 comp rx=0 tx=0).stop 2024-02-19T15:22:09.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.594+0000 7f4b87000640 1 -- 172.21.15.47:0/153994335 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b8015dfa0 msgr2=0x7f4b800c79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:09.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.594+0000 7f4b87000640 1 --2- 172.21.15.47:0/153994335 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b8015dfa0 0x7f4b800c79b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4b7800c400 tx=0x7f4b7800c8c0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.594+0000 7f4b87000640 1 -- 172.21.15.47:0/153994335 shutdown_connections 2024-02-19T15:22:09.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.594+0000 7f4b87000640 1 --2- 172.21.15.47:0/153994335 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4b5c066e40 0x7f4b5c069300 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.594+0000 7f4b87000640 1 --2- 172.21.15.47:0/153994335 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b8015dfa0 0x7f4b800c79b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.594+0000 7f4b87000640 1 --2- 172.21.15.47:0/153994335 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b800ce400 0x7f4b800c7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.594+0000 7f4b87000640 1 --2- 172.21.15.47:0/153994335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b800b4240 0x7f4b80152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.594+0000 7f4b87000640 1 -- 172.21.15.47:0/153994335 >> 172.21.15.47:0/153994335 conn(0x7f4b800b3690 msgr2=0x7f4b800d1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:09.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.594+0000 7f4b87000640 1 -- 172.21.15.47:0/153994335 shutdown_connections 2024-02-19T15:22:09.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.594+0000 7f4b87000640 1 -- 172.21.15.47:0/153994335 wait complete. 2024-02-19T15:22:09.615 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T15:22:09.615 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:22:09.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.754+0000 7f02c3e00640 1 Processor -- start 2024-02-19T15:22:09.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.754+0000 7f02c3e00640 1 -- start start 2024-02-19T15:22:09.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.754+0000 7f02c3e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f02bc15bcd0 0x7f02bc0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:09.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.754+0000 7f02c3e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f02bc0d0360 0x7f02bc15d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:09.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.754+0000 7f02c3e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02bc0cc700 con 0x7f02bc0cedb0 2024-02-19T15:22:09.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.754+0000 7f02c3e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02bc0cf5e0 con 0x7f02bc15bcd0 2024-02-19T15:22:09.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.754+0000 7f02c3e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc0cf300 con 0x7f02bc0d0360 2024-02-19T15:22:09.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.754+0000 7f02c0a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f02bc0cedb0 0x7f02bc0cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43984/0 (socket says 172.21.15.47:43984) 2024-02-19T15:22:09.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.754+0000 7f02c0a00640 1 -- 172.21.15.47:0/4260249724 learned_addr learned my addr 172.21.15.47:0/4260249724 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:09.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.754+0000 7f02c1e00640 1 --2- 172.21.15.47:0/4260249724 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f02bc0d0360 0x7f02bc15d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:09.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.754+0000 7f02c1400640 1 --2- 172.21.15.47:0/4260249724 >> v2:172.21.15.116:3300/0 conn(0x7f02bc15bcd0 0x7f02bc0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:09.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.754+0000 7f02bbe00640 1 -- 172.21.15.47:0/4260249724 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185648080 0 0) 0x7f02bc0cc700 con 0x7f02bc0cedb0 2024-02-19T15:22:09.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.754+0000 7f02bbe00640 1 -- 172.21.15.47:0/4260249724 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02a4003530 con 0x7f02bc0cedb0 2024-02-19T15:22:09.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.754+0000 7f02c1e00640 1 -- 172.21.15.47:0/4260249724 >> v2:172.21.15.116:3300/0 conn(0x7f02bc15bcd0 msgr2=0x7f02bc0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:09.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.754+0000 7f02c1e00640 1 --2- 172.21.15.47:0/4260249724 >> v2:172.21.15.116:3300/0 conn(0x7f02bc15bcd0 0x7f02bc0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.754+0000 7f02c1e00640 1 -- 172.21.15.47:0/4260249724 >> v1:172.21.15.47:6789/0 conn(0x7f02bc0cedb0 legacy=0x7f02bc0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:09.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.754+0000 7f02c1e00640 1 -- 172.21.15.47:0/4260249724 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02bc0cc9a0 con 0x7f02bc0d0360 2024-02-19T15:22:09.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.754+0000 7f02c1400640 1 --2- 172.21.15.47:0/4260249724 >> v2:172.21.15.116:3300/0 conn(0x7f02bc15bcd0 0x7f02bc0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:09.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.758+0000 7f02c1e00640 1 --2- 172.21.15.47:0/4260249724 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f02bc0d0360 0x7f02bc15d4f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f02b4009d90 tx=0x7f02b401c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ba4ecc8a2d80d95 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:09.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.758+0000 7f02bbe00640 1 -- 172.21.15.47:0/4260249724 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b40a5a00 con 0x7f02bc0d0360 2024-02-19T15:22:09.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.758+0000 7f02bbe00640 1 -- 172.21.15.47:0/4260249724 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b40a5b60 con 0x7f02bc0d0360 2024-02-19T15:22:09.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.758+0000 7f02bbe00640 1 -- 172.21.15.47:0/4260249724 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b40ad440 con 0x7f02bc0d0360 2024-02-19T15:22:09.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.758+0000 7f02c3e00640 1 -- 172.21.15.47:0/4260249724 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f02bc0d0360 msgr2=0x7f02bc15d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:09.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.758+0000 7f02c3e00640 1 --2- 172.21.15.47:0/4260249724 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f02bc0d0360 0x7f02bc15d4f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f02b4009d90 tx=0x7f02b401c940 comp rx=0 tx=0).stop 2024-02-19T15:22:09.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.758+0000 7f02c3e00640 1 -- 172.21.15.47:0/4260249724 shutdown_connections 2024-02-19T15:22:09.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.758+0000 7f02c3e00640 1 --2- 172.21.15.47:0/4260249724 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f02bc0d0360 0x7f02bc15d4f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.758+0000 7f02c3e00640 1 --2- 172.21.15.47:0/4260249724 >> v2:172.21.15.116:3300/0 conn(0x7f02bc15bcd0 0x7f02bc0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.758+0000 7f02c3e00640 1 -- 172.21.15.47:0/4260249724 >> 172.21.15.47:0/4260249724 conn(0x7f02bc0d1bf0 msgr2=0x7f02bc0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:09.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.758+0000 7f02c3e00640 1 -- 172.21.15.47:0/4260249724 shutdown_connections 2024-02-19T15:22:09.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.758+0000 7f02c3e00640 1 -- 172.21.15.47:0/4260249724 wait complete. 2024-02-19T15:22:09.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c3e00640 1 Processor -- start 2024-02-19T15:22:09.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c3e00640 1 -- start start 2024-02-19T15:22:09.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02bc0cedb0 0x7f02bc15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:09.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c3e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f02bc0d0360 0x7f02bc153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:09.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02bc15bcd0 0x7f02bc158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:09.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c3e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc0c8dc0 con 0x7f02bc0cedb0 2024-02-19T15:22:09.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c3e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc1545f0 con 0x7f02bc15bcd0 2024-02-19T15:22:09.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c3e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc1548d0 con 0x7f02bc0d0360 2024-02-19T15:22:09.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c1400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02bc0cedb0 0x7f02bc15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:09.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c1400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02bc0cedb0 0x7f02bc15ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52868/0 (socket says 172.21.15.47:52868) 2024-02-19T15:22:09.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c1400640 1 -- 172.21.15.47:0/2769134656 learned_addr learned my addr 172.21.15.47:0/2769134656 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:09.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c0a00640 1 --2- 172.21.15.47:0/2769134656 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f02bc0d0360 0x7f02bc153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:09.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c1e00640 1 --2- 172.21.15.47:0/2769134656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02bc15bcd0 0x7f02bc158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:09.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c1400640 1 -- 172.21.15.47:0/2769134656 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f02bc0d0360 msgr2=0x7f02bc153c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:09.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c1400640 1 --2- 172.21.15.47:0/2769134656 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f02bc0d0360 0x7f02bc153c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c1400640 1 -- 172.21.15.47:0/2769134656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02bc15bcd0 msgr2=0x7f02bc158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:09.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c1400640 1 --2- 172.21.15.47:0/2769134656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02bc15bcd0 0x7f02bc158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c1400640 1 -- 172.21.15.47:0/2769134656 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b0009420 con 0x7f02bc0cedb0 2024-02-19T15:22:09.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c1400640 1 --2- 172.21.15.47:0/2769134656 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02bc0cedb0 0x7f02bc15ab70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f02b000e980 tx=0x7f02b000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:09.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02b9600640 1 -- 172.21.15.47:0/2769134656 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b000ca00 con 0x7f02bc0cedb0 2024-02-19T15:22:09.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02b9600640 1 -- 172.21.15.47:0/2769134656 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b000cb60 con 0x7f02bc0cedb0 2024-02-19T15:22:09.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02b9600640 1 -- 172.21.15.47:0/2769134656 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b0013470 con 0x7f02bc0cedb0 2024-02-19T15:22:09.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c3e00640 1 -- 172.21.15.47:0/2769134656 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b4009a70 con 0x7f02bc0cedb0 2024-02-19T15:22:09.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c3e00640 1 -- 172.21.15.47:0/2769134656 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02bc0cdb40 con 0x7f02bc0cedb0 2024-02-19T15:22:09.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.762+0000 7f02c3e00640 1 -- 172.21.15.47:0/2769134656 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02bc0c9030 con 0x7f02bc0cedb0 2024-02-19T15:22:09.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.766+0000 7f02b9600640 1 -- 172.21.15.47:0/2769134656 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f02b00135d0 con 0x7f02bc0cedb0 2024-02-19T15:22:09.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.766+0000 7f02b9600640 1 --2- 172.21.15.47:0/2769134656 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f029c066e40 0x7f029c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:09.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.766+0000 7f02b9600640 1 -- 172.21.15.47:0/2769134656 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02b0084c30 con 0x7f02bc0cedb0 2024-02-19T15:22:09.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.766+0000 7f02c0a00640 1 --2- 172.21.15.47:0/2769134656 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f029c066e40 0x7f029c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:09.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.766+0000 7f02c0a00640 1 --2- 172.21.15.47:0/2769134656 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f029c066e40 0x7f029c069300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f02bc1f8360 tx=0x7f02a8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:09.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.770+0000 7f02b9600640 1 -- 172.21.15.47:0/2769134656 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f02b0051ac0 con 0x7f02bc0cedb0 2024-02-19T15:22:09.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.930+0000 7f02c3e00640 1 -- 172.21.15.47:0/2769134656 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f02bc0cd9c0 con 0x7f029c066e40 2024-02-19T15:22:09.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.938+0000 7f02b9600640 1 -- 172.21.15.47:0/2769134656 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264584 (secure 0 0 0) 0x7f02a8001460 con 0x7f029c066e40 2024-02-19T15:22:09.940 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:09.944 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:22:09.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.942+0000 7f02c3e00640 1 -- 172.21.15.47:0/2769134656 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f029c066e40 msgr2=0x7f029c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:09.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.942+0000 7f02c3e00640 1 --2- 172.21.15.47:0/2769134656 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f029c066e40 0x7f029c069300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f02bc1f8360 tx=0x7f02a8009040 comp rx=0 tx=0).stop 2024-02-19T15:22:09.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.942+0000 7f02c3e00640 1 -- 172.21.15.47:0/2769134656 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02bc0cedb0 msgr2=0x7f02bc15ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:09.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.942+0000 7f02c3e00640 1 --2- 172.21.15.47:0/2769134656 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02bc0cedb0 0x7f02bc15ab70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f02b000e980 tx=0x7f02b000ee40 comp rx=0 tx=0).stop 2024-02-19T15:22:09.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.942+0000 7f02c3e00640 1 -- 172.21.15.47:0/2769134656 shutdown_connections 2024-02-19T15:22:09.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.942+0000 7f02c3e00640 1 --2- 172.21.15.47:0/2769134656 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f029c066e40 0x7f029c069300 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.942+0000 7f02c3e00640 1 --2- 172.21.15.47:0/2769134656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02bc15bcd0 0x7f02bc158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.942+0000 7f02c3e00640 1 --2- 172.21.15.47:0/2769134656 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f02bc0d0360 0x7f02bc153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.942+0000 7f02c3e00640 1 --2- 172.21.15.47:0/2769134656 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02bc0cedb0 0x7f02bc15ab70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:09.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.942+0000 7f02c3e00640 1 -- 172.21.15.47:0/2769134656 >> 172.21.15.47:0/2769134656 conn(0x7f02bc0d1bf0 msgr2=0x7f02bc0702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:09.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.942+0000 7f02c3e00640 1 -- 172.21.15.47:0/2769134656 shutdown_connections 2024-02-19T15:22:09.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:09.942+0000 7f02c3e00640 1 -- 172.21.15.47:0/2769134656 wait complete. 2024-02-19T15:22:09.962 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-02-19T15:22:09.150908+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235568,"kb_used_data":2752,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739152,"statfs":{"total":773094113280,"available":772852891648,"internally_reserved":0,"allocated":2818048,"data_stored":1711467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.338037"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.873472+0000","last_change":"2024-02-19T15:22:05.868047+0000","last_active":"2024-02-19T15:22:05.873472+0000","last_peered":"2024-02-19T15:22:05.873472+0000","last_clean":"2024-02-19T15:22:05.873472+0000","last_became_active":"2024-02-19T15:22:05.867538+0000","last_became_peered":"2024-02-19T15:22:05.867538+0000","last_unstale":"2024-02-19T15:22:05.873472+0000","last_undegraded":"2024-02-19T15:22:05.873472+0000","last_fullsized":"2024-02-19T15:22:05.873472+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:33.203920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.866600+0000","last_change":"2024-02-19T15:22:05.861729+0000","last_active":"2024-02-19T15:22:05.866600+0000","last_peered":"2024-02-19T15:22:05.866600+0000","last_clean":"2024-02-19T15:22:05.866600+0000","last_became_active":"2024-02-19T15:22:05.861455+0000","last_became_peered":"2024-02-19T15:22:05.861455+0000","last_unstale":"2024-02-19T15:22:05.866600+0000","last_undegraded":"2024-02-19T15:22:05.866600+0000","last_fullsized":"2024-02-19T15:22:05.866600+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:43:34.980001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.866483+0000","last_change":"2024-02-19T15:22:05.857715+0000","last_active":"2024-02-19T15:22:05.866483+0000","last_peered":"2024-02-19T15:22:05.866483+0000","last_clean":"2024-02-19T15:22:05.866483+0000","last_became_active":"2024-02-19T15:22:05.857360+0000","last_became_peered":"2024-02-19T15:22:05.857360+0000","last_unstale":"2024-02-19T15:22:05.866483+0000","last_undegraded":"2024-02-19T15:22:05.866483+0000","last_fullsized":"2024-02-19T15:22:05.866483+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:50:49.643232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.860650+0000","last_change":"2024-02-19T15:22:05.850537+0000","last_active":"2024-02-19T15:22:05.860650+0000","last_peered":"2024-02-19T15:22:05.860650+0000","last_clean":"2024-02-19T15:22:05.860650+0000","last_became_active":"2024-02-19T15:22:05.850288+0000","last_became_peered":"2024-02-19T15:22:05.850288+0000","last_unstale":"2024-02-19T15:22:05.860650+0000","last_undegraded":"2024-02-19T15:22:05.860650+0000","last_fullsized":"2024-02-19T15:22:05.860650+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:03:09.391902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.876968+0000","last_change":"2024-02-19T15:22:05.872936+0000","last_active":"2024-02-19T15:22:05.876968+0000","last_peered":"2024-02-19T15:22:05.876968+0000","last_clean":"2024-02-19T15:22:05.876968+0000","last_became_active":"2024-02-19T15:22:05.871969+0000","last_became_peered":"2024-02-19T15:22:05.871969+0000","last_unstale":"2024-02-19T15:22:05.876968+0000","last_undegraded":"2024-02-19T15:22:05.876968+0000","last_fullsized":"2024-02-19T15:22:05.876968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:21:49.261015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.860703+0000","last_change":"2024-02-19T15:22:05.853731+0000","last_active":"2024-02-19T15:22:05.860703+0000","last_peered":"2024-02-19T15:22:05.860703+0000","last_clean":"2024-02-19T15:22:05.860703+0000","last_became_active":"2024-02-19T15:22:05.853423+0000","last_became_peered":"2024-02-19T15:22:05.853423+0000","last_unstale":"2024-02-19T15:22:05.860703+0000","last_undegraded":"2024-02-19T15:22:05.860703+0000","last_fullsized":"2024-02-19T15:22:05.860703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:48:20.147250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.866634+0000","last_change":"2024-02-19T15:22:05.860498+0000","last_active":"2024-02-19T15:22:05.866634+0000","last_peered":"2024-02-19T15:22:05.866634+0000","last_clean":"2024-02-19T15:22:05.866634+0000","last_became_active":"2024-02-19T15:22:05.860283+0000","last_became_peered":"2024-02-19T15:22:05.860283+0000","last_unstale":"2024-02-19T15:22:05.866634+0000","last_undegraded":"2024-02-19T15:22:05.866634+0000","last_fullsized":"2024-02-19T15:22:05.866634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:03:09.391902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.870717+0000","last_change":"2024-02-19T15:22:05.860461+0000","last_active":"2024-02-19T15:22:05.870717+0000","last_peered":"2024-02-19T15:22:05.870717+0000","last_clean":"2024-02-19T15:22:05.870717+0000","last_became_active":"2024-02-19T15:22:05.860119+0000","last_became_peered":"2024-02-19T15:22:05.860119+0000","last_unstale":"2024-02-19T15:22:05.870717+0000","last_undegraded":"2024-02-19T15:22:05.870717+0000","last_fullsized":"2024-02-19T15:22:05.870717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:22.856662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.868344+0000","last_change":"2024-02-19T15:22:05.863575+0000","last_active":"2024-02-19T15:22:05.868344+0000","last_peered":"2024-02-19T15:22:05.868344+0000","last_clean":"2024-02-19T15:22:05.868344+0000","last_became_active":"2024-02-19T15:22:05.863306+0000","last_became_peered":"2024-02-19T15:22:05.863306+0000","last_unstale":"2024-02-19T15:22:05.868344+0000","last_undegraded":"2024-02-19T15:22:05.868344+0000","last_fullsized":"2024-02-19T15:22:05.868344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:43:34.980001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854624+0000","last_change":"2024-02-19T15:22:05.867622+0000","last_active":"2024-02-19T15:22:06.854624+0000","last_peered":"2024-02-19T15:22:06.854624+0000","last_clean":"2024-02-19T15:22:06.854624+0000","last_became_active":"2024-02-19T15:22:05.867352+0000","last_became_peered":"2024-02-19T15:22:05.867352+0000","last_unstale":"2024-02-19T15:22:06.854624+0000","last_undegraded":"2024-02-19T15:22:06.854624+0000","last_fullsized":"2024-02-19T15:22:06.854624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:08.091823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854743+0000","last_change":"2024-02-19T15:22:05.859648+0000","last_active":"2024-02-19T15:22:06.854743+0000","last_peered":"2024-02-19T15:22:06.854743+0000","last_clean":"2024-02-19T15:22:06.854743+0000","last_became_active":"2024-02-19T15:22:05.859381+0000","last_became_peered":"2024-02-19T15:22:05.859381+0000","last_unstale":"2024-02-19T15:22:06.854743+0000","last_undegraded":"2024-02-19T15:22:06.854743+0000","last_fullsized":"2024-02-19T15:22:06.854743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:35:11.566219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.874393+0000","last_change":"2024-02-19T15:22:05.869009+0000","last_active":"2024-02-19T15:22:05.874393+0000","last_peered":"2024-02-19T15:22:05.874393+0000","last_clean":"2024-02-19T15:22:05.874393+0000","last_became_active":"2024-02-19T15:22:05.868709+0000","last_became_peered":"2024-02-19T15:22:05.868709+0000","last_unstale":"2024-02-19T15:22:05.874393+0000","last_undegraded":"2024-02-19T15:22:05.874393+0000","last_fullsized":"2024-02-19T15:22:05.874393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:28.331381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.852782+0000","last_change":"2024-02-19T15:22:05.848558+0000","last_active":"2024-02-19T15:22:05.852782+0000","last_peered":"2024-02-19T15:22:05.852782+0000","last_clean":"2024-02-19T15:22:05.852782+0000","last_became_active":"2024-02-19T15:22:05.848288+0000","last_became_peered":"2024-02-19T15:22:05.848288+0000","last_unstale":"2024-02-19T15:22:05.852782+0000","last_undegraded":"2024-02-19T15:22:05.852782+0000","last_fullsized":"2024-02-19T15:22:05.852782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:41.785448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.857444+0000","last_change":"2024-02-19T15:22:05.846265+0000","last_active":"2024-02-19T15:22:05.857444+0000","last_peered":"2024-02-19T15:22:05.857444+0000","last_clean":"2024-02-19T15:22:05.857444+0000","last_became_active":"2024-02-19T15:22:05.841757+0000","last_became_peered":"2024-02-19T15:22:05.841757+0000","last_unstale":"2024-02-19T15:22:05.857444+0000","last_undegraded":"2024-02-19T15:22:05.857444+0000","last_fullsized":"2024-02-19T15:22:05.857444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:16:22.481692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.858006+0000","last_change":"2024-02-19T15:22:05.850786+0000","last_active":"2024-02-19T15:22:05.858006+0000","last_peered":"2024-02-19T15:22:05.858006+0000","last_clean":"2024-02-19T15:22:05.858006+0000","last_became_active":"2024-02-19T15:22:05.850136+0000","last_became_peered":"2024-02-19T15:22:05.850136+0000","last_unstale":"2024-02-19T15:22:05.858006+0000","last_undegraded":"2024-02-19T15:22:05.858006+0000","last_fullsized":"2024-02-19T15:22:05.858006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:02.917816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.865077+0000","last_change":"2024-02-19T15:22:05.862076+0000","last_active":"2024-02-19T15:22:05.865077+0000","last_peered":"2024-02-19T15:22:05.865077+0000","last_clean":"2024-02-19T15:22:05.865077+0000","last_became_active":"2024-02-19T15:22:05.861826+0000","last_became_peered":"2024-02-19T15:22:05.861826+0000","last_unstale":"2024-02-19T15:22:05.865077+0000","last_undegraded":"2024-02-19T15:22:05.865077+0000","last_fullsized":"2024-02-19T15:22:05.865077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:43:34.980001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.875480+0000","last_change":"2024-02-19T15:22:05.869845+0000","last_active":"2024-02-19T15:22:05.875480+0000","last_peered":"2024-02-19T15:22:05.875480+0000","last_clean":"2024-02-19T15:22:05.875480+0000","last_became_active":"2024-02-19T15:22:05.865534+0000","last_became_peered":"2024-02-19T15:22:05.865534+0000","last_unstale":"2024-02-19T15:22:05.875480+0000","last_undegraded":"2024-02-19T15:22:05.875480+0000","last_fullsized":"2024-02-19T15:22:05.875480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:33:20.316638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.857111+0000","last_change":"2024-02-19T15:22:05.852133+0000","last_active":"2024-02-19T15:22:05.857111+0000","last_peered":"2024-02-19T15:22:05.857111+0000","last_clean":"2024-02-19T15:22:05.857111+0000","last_became_active":"2024-02-19T15:22:05.851823+0000","last_became_peered":"2024-02-19T15:22:05.851823+0000","last_unstale":"2024-02-19T15:22:05.857111+0000","last_undegraded":"2024-02-19T15:22:05.857111+0000","last_fullsized":"2024-02-19T15:22:05.857111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:31:18.431106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.871476+0000","last_change":"2024-02-19T15:22:05.866370+0000","last_active":"2024-02-19T15:22:05.871476+0000","last_peered":"2024-02-19T15:22:05.871476+0000","last_clean":"2024-02-19T15:22:05.871476+0000","last_became_active":"2024-02-19T15:22:05.866059+0000","last_became_peered":"2024-02-19T15:22:05.866059+0000","last_unstale":"2024-02-19T15:22:05.871476+0000","last_undegraded":"2024-02-19T15:22:05.871476+0000","last_fullsized":"2024-02-19T15:22:05.871476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:41.123853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854536+0000","last_change":"2024-02-19T15:22:05.858350+0000","last_active":"2024-02-19T15:22:06.854536+0000","last_peered":"2024-02-19T15:22:06.854536+0000","last_clean":"2024-02-19T15:22:06.854536+0000","last_became_active":"2024-02-19T15:22:05.858055+0000","last_became_peered":"2024-02-19T15:22:05.858055+0000","last_unstale":"2024-02-19T15:22:06.854536+0000","last_undegraded":"2024-02-19T15:22:06.854536+0000","last_fullsized":"2024-02-19T15:22:06.854536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:53:17.517796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.870157+0000","last_change":"2024-02-19T15:22:05.867437+0000","last_active":"2024-02-19T15:22:05.870157+0000","last_peered":"2024-02-19T15:22:05.870157+0000","last_clean":"2024-02-19T15:22:05.870157+0000","last_became_active":"2024-02-19T15:22:05.867104+0000","last_became_peered":"2024-02-19T15:22:05.867104+0000","last_unstale":"2024-02-19T15:22:05.870157+0000","last_undegraded":"2024-02-19T15:22:05.870157+0000","last_fullsized":"2024-02-19T15:22:05.870157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:43:50.173776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.876782+0000","last_change":"2024-02-19T15:22:05.873978+0000","last_active":"2024-02-19T15:22:05.876782+0000","last_peered":"2024-02-19T15:22:05.876782+0000","last_clean":"2024-02-19T15:22:05.876782+0000","last_became_active":"2024-02-19T15:22:05.870210+0000","last_became_peered":"2024-02-19T15:22:05.870210+0000","last_unstale":"2024-02-19T15:22:05.876782+0000","last_undegraded":"2024-02-19T15:22:05.876782+0000","last_fullsized":"2024-02-19T15:22:05.876782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:48:05.400374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.871490+0000","last_change":"2024-02-19T15:22:05.869630+0000","last_active":"2024-02-19T15:22:05.871490+0000","last_peered":"2024-02-19T15:22:05.871490+0000","last_clean":"2024-02-19T15:22:05.871490+0000","last_became_active":"2024-02-19T15:22:05.869363+0000","last_became_peered":"2024-02-19T15:22:05.869363+0000","last_unstale":"2024-02-19T15:22:05.871490+0000","last_undegraded":"2024-02-19T15:22:05.871490+0000","last_fullsized":"2024-02-19T15:22:05.871490+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:03:09.391902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.858535+0000","last_change":"2024-02-19T15:22:05.846649+0000","last_active":"2024-02-19T15:22:05.858535+0000","last_peered":"2024-02-19T15:22:05.858535+0000","last_clean":"2024-02-19T15:22:05.858535+0000","last_became_active":"2024-02-19T15:22:05.846335+0000","last_became_peered":"2024-02-19T15:22:05.846335+0000","last_unstale":"2024-02-19T15:22:05.858535+0000","last_undegraded":"2024-02-19T15:22:05.858535+0000","last_fullsized":"2024-02-19T15:22:05.858535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:37:51.996719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.872265+0000","last_change":"2024-02-19T15:22:05.870492+0000","last_active":"2024-02-19T15:22:05.872265+0000","last_peered":"2024-02-19T15:22:05.872265+0000","last_clean":"2024-02-19T15:22:05.872265+0000","last_became_active":"2024-02-19T15:22:05.868145+0000","last_became_peered":"2024-02-19T15:22:05.868145+0000","last_unstale":"2024-02-19T15:22:05.872265+0000","last_undegraded":"2024-02-19T15:22:05.872265+0000","last_fullsized":"2024-02-19T15:22:05.872265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:48:20.147250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854818+0000","last_change":"2024-02-19T15:22:05.869349+0000","last_active":"2024-02-19T15:22:06.854818+0000","last_peered":"2024-02-19T15:22:06.854818+0000","last_clean":"2024-02-19T15:22:06.854818+0000","last_became_active":"2024-02-19T15:22:05.869042+0000","last_became_peered":"2024-02-19T15:22:05.869042+0000","last_unstale":"2024-02-19T15:22:06.854818+0000","last_undegraded":"2024-02-19T15:22:06.854818+0000","last_fullsized":"2024-02-19T15:22:06.854818+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:14:16.136182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.859704+0000","last_change":"2024-02-19T15:22:05.856951+0000","last_active":"2024-02-19T15:22:05.859704+0000","last_peered":"2024-02-19T15:22:05.859704+0000","last_clean":"2024-02-19T15:22:05.859704+0000","last_became_active":"2024-02-19T15:22:05.856670+0000","last_became_peered":"2024-02-19T15:22:05.856670+0000","last_unstale":"2024-02-19T15:22:05.859704+0000","last_undegraded":"2024-02-19T15:22:05.859704+0000","last_fullsized":"2024-02-19T15:22:05.859704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:03:09.391902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.872296+0000","last_change":"2024-02-19T15:22:05.868491+0000","last_active":"2024-02-19T15:22:05.872296+0000","last_peered":"2024-02-19T15:22:05.872296+0000","last_clean":"2024-02-19T15:22:05.872296+0000","last_became_active":"2024-02-19T15:22:05.868184+0000","last_became_peered":"2024-02-19T15:22:05.868184+0000","last_unstale":"2024-02-19T15:22:05.872296+0000","last_undegraded":"2024-02-19T15:22:05.872296+0000","last_fullsized":"2024-02-19T15:22:05.872296+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:32:56.909524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.861731+0000","last_change":"2024-02-19T15:22:05.847847+0000","last_active":"2024-02-19T15:22:05.861731+0000","last_peered":"2024-02-19T15:22:05.861731+0000","last_clean":"2024-02-19T15:22:05.861731+0000","last_became_active":"2024-02-19T15:22:05.847538+0000","last_became_peered":"2024-02-19T15:22:05.847538+0000","last_unstale":"2024-02-19T15:22:05.861731+0000","last_undegraded":"2024-02-19T15:22:05.861731+0000","last_fullsized":"2024-02-19T15:22:05.861731+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:26:20.732058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.860681+0000","last_change":"2024-02-19T15:22:05.847222+0000","last_active":"2024-02-19T15:22:05.860681+0000","last_peered":"2024-02-19T15:22:05.860681+0000","last_clean":"2024-02-19T15:22:05.860681+0000","last_became_active":"2024-02-19T15:22:05.846957+0000","last_became_peered":"2024-02-19T15:22:05.846957+0000","last_unstale":"2024-02-19T15:22:05.860681+0000","last_undegraded":"2024-02-19T15:22:05.860681+0000","last_fullsized":"2024-02-19T15:22:05.860681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:02.917816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.869164+0000","last_change":"2024-02-19T15:22:05.862263+0000","last_active":"2024-02-19T15:22:05.869164+0000","last_peered":"2024-02-19T15:22:05.869164+0000","last_clean":"2024-02-19T15:22:05.869164+0000","last_became_active":"2024-02-19T15:22:05.861932+0000","last_became_peered":"2024-02-19T15:22:05.861932+0000","last_unstale":"2024-02-19T15:22:05.869164+0000","last_undegraded":"2024-02-19T15:22:05.869164+0000","last_fullsized":"2024-02-19T15:22:05.869164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:22.856662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854810+0000","last_change":"2024-02-19T15:22:05.852998+0000","last_active":"2024-02-19T15:22:06.854810+0000","last_peered":"2024-02-19T15:22:06.854810+0000","last_clean":"2024-02-19T15:22:06.854810+0000","last_became_active":"2024-02-19T15:22:05.852660+0000","last_became_peered":"2024-02-19T15:22:05.852660+0000","last_unstale":"2024-02-19T15:22:06.854810+0000","last_undegraded":"2024-02-19T15:22:06.854810+0000","last_fullsized":"2024-02-19T15:22:06.854810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:54:25.604351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.875056+0000","last_change":"2024-02-19T15:22:05.862188+0000","last_active":"2024-02-19T15:22:05.875056+0000","last_peered":"2024-02-19T15:22:05.875056+0000","last_clean":"2024-02-19T15:22:05.875056+0000","last_became_active":"2024-02-19T15:22:05.861789+0000","last_became_peered":"2024-02-19T15:22:05.861789+0000","last_unstale":"2024-02-19T15:22:05.875056+0000","last_undegraded":"2024-02-19T15:22:05.875056+0000","last_fullsized":"2024-02-19T15:22:05.875056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:52:38.926167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.876093+0000","last_change":"2024-02-19T15:22:04.848427+0000","last_active":"2024-02-19T15:22:05.876093+0000","last_peered":"2024-02-19T15:22:05.876093+0000","last_clean":"2024-02-19T15:22:05.876093+0000","last_became_active":"2024-02-19T15:22:04.848111+0000","last_became_peered":"2024-02-19T15:22:04.848111+0000","last_unstale":"2024-02-19T15:22:05.876093+0000","last_undegraded":"2024-02-19T15:22:05.876093+0000","last_fullsized":"2024-02-19T15:22:05.876093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:17.404136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.861706+0000","last_change":"2024-02-19T15:22:05.845098+0000","last_active":"2024-02-19T15:22:05.861706+0000","last_peered":"2024-02-19T15:22:05.861706+0000","last_clean":"2024-02-19T15:22:05.861706+0000","last_became_active":"2024-02-19T15:22:05.844805+0000","last_became_peered":"2024-02-19T15:22:05.844805+0000","last_unstale":"2024-02-19T15:22:05.861706+0000","last_undegraded":"2024-02-19T15:22:05.861706+0000","last_fullsized":"2024-02-19T15:22:05.861706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:33:05.203089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854248+0000","last_change":"2024-02-19T15:22:04.842285+0000","last_active":"2024-02-19T15:22:06.854248+0000","last_peered":"2024-02-19T15:22:06.854248+0000","last_clean":"2024-02-19T15:22:06.854248+0000","last_became_active":"2024-02-19T15:22:04.840126+0000","last_became_peered":"2024-02-19T15:22:04.840126+0000","last_unstale":"2024-02-19T15:22:06.854248+0000","last_undegraded":"2024-02-19T15:22:06.854248+0000","last_fullsized":"2024-02-19T15:22:06.854248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:11:04.017098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.837921+0000","last_change":"2024-02-19T15:22:04.839208+0000","last_active":"2024-02-19T15:22:05.837921+0000","last_peered":"2024-02-19T15:22:05.837921+0000","last_clean":"2024-02-19T15:22:05.837921+0000","last_became_active":"2024-02-19T15:22:04.838857+0000","last_became_peered":"2024-02-19T15:22:04.838857+0000","last_unstale":"2024-02-19T15:22:05.837921+0000","last_undegraded":"2024-02-19T15:22:05.837921+0000","last_fullsized":"2024-02-19T15:22:05.837921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:52:37.920564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.867560+0000","last_change":"2024-02-19T15:22:05.859786+0000","last_active":"2024-02-19T15:22:05.867560+0000","last_peered":"2024-02-19T15:22:05.867560+0000","last_clean":"2024-02-19T15:22:05.867560+0000","last_became_active":"2024-02-19T15:22:05.854215+0000","last_became_peered":"2024-02-19T15:22:05.854215+0000","last_unstale":"2024-02-19T15:22:05.867560+0000","last_undegraded":"2024-02-19T15:22:05.867560+0000","last_fullsized":"2024-02-19T15:22:05.867560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:31:18.431106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.838455+0000","last_change":"2024-02-19T15:22:04.843918+0000","last_active":"2024-02-19T15:22:05.838455+0000","last_peered":"2024-02-19T15:22:05.838455+0000","last_clean":"2024-02-19T15:22:05.838455+0000","last_became_active":"2024-02-19T15:22:04.842002+0000","last_became_peered":"2024-02-19T15:22:04.842002+0000","last_unstale":"2024-02-19T15:22:05.838455+0000","last_undegraded":"2024-02-19T15:22:05.838455+0000","last_fullsized":"2024-02-19T15:22:05.838455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:27.325777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.858027+0000","last_change":"2024-02-19T15:22:05.852203+0000","last_active":"2024-02-19T15:22:05.858027+0000","last_peered":"2024-02-19T15:22:05.858027+0000","last_clean":"2024-02-19T15:22:05.858027+0000","last_became_active":"2024-02-19T15:22:05.850442+0000","last_became_peered":"2024-02-19T15:22:05.850442+0000","last_unstale":"2024-02-19T15:22:05.858027+0000","last_undegraded":"2024-02-19T15:22:05.858027+0000","last_fullsized":"2024-02-19T15:22:05.858027+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:05.756626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854530+0000","last_change":"2024-02-19T15:22:04.839817+0000","last_active":"2024-02-19T15:22:06.854530+0000","last_peered":"2024-02-19T15:22:06.854530+0000","last_clean":"2024-02-19T15:22:06.854530+0000","last_became_active":"2024-02-19T15:22:04.839034+0000","last_became_peered":"2024-02-19T15:22:04.839034+0000","last_unstale":"2024-02-19T15:22:06.854530+0000","last_undegraded":"2024-02-19T15:22:06.854530+0000","last_fullsized":"2024-02-19T15:22:06.854530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:16:21.476088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.875539+0000","last_change":"2024-02-19T15:22:05.869563+0000","last_active":"2024-02-19T15:22:05.875539+0000","last_peered":"2024-02-19T15:22:05.875539+0000","last_clean":"2024-02-19T15:22:05.875539+0000","last_became_active":"2024-02-19T15:22:05.869260+0000","last_became_peered":"2024-02-19T15:22:05.869260+0000","last_unstale":"2024-02-19T15:22:05.875539+0000","last_undegraded":"2024-02-19T15:22:05.875539+0000","last_fullsized":"2024-02-19T15:22:05.875539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:33.203920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.840836+0000","last_change":"2024-02-19T15:22:04.843586+0000","last_active":"2024-02-19T15:22:05.840836+0000","last_peered":"2024-02-19T15:22:05.840836+0000","last_clean":"2024-02-19T15:22:05.840836+0000","last_became_active":"2024-02-19T15:22:04.842983+0000","last_became_peered":"2024-02-19T15:22:04.842983+0000","last_unstale":"2024-02-19T15:22:05.840836+0000","last_undegraded":"2024-02-19T15:22:05.840836+0000","last_fullsized":"2024-02-19T15:22:05.840836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:21.851058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.867806+0000","last_change":"2024-02-19T15:22:05.861506+0000","last_active":"2024-02-19T15:22:05.867806+0000","last_peered":"2024-02-19T15:22:05.867806+0000","last_clean":"2024-02-19T15:22:05.867806+0000","last_became_active":"2024-02-19T15:22:05.861172+0000","last_became_peered":"2024-02-19T15:22:05.861172+0000","last_unstale":"2024-02-19T15:22:05.867806+0000","last_undegraded":"2024-02-19T15:22:05.867806+0000","last_fullsized":"2024-02-19T15:22:05.867806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:25:37.290835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.860919+0000","last_change":"2024-02-19T15:22:04.833984+0000","last_active":"2024-02-19T15:22:05.860919+0000","last_peered":"2024-02-19T15:22:05.860919+0000","last_clean":"2024-02-19T15:22:05.860919+0000","last_became_active":"2024-02-19T15:22:04.833418+0000","last_became_peered":"2024-02-19T15:22:04.833418+0000","last_unstale":"2024-02-19T15:22:05.860919+0000","last_undegraded":"2024-02-19T15:22:05.860919+0000","last_fullsized":"2024-02-19T15:22:05.860919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:31:17.425502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.867888+0000","last_change":"2024-02-19T15:22:05.865117+0000","last_active":"2024-02-19T15:22:05.867888+0000","last_peered":"2024-02-19T15:22:05.867888+0000","last_clean":"2024-02-19T15:22:05.867888+0000","last_became_active":"2024-02-19T15:22:05.864502+0000","last_became_peered":"2024-02-19T15:22:05.864502+0000","last_unstale":"2024-02-19T15:22:05.867888+0000","last_undegraded":"2024-02-19T15:22:05.867888+0000","last_fullsized":"2024-02-19T15:22:05.867888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:18.409739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854661+0000","last_change":"2024-02-19T15:22:04.832285+0000","last_active":"2024-02-19T15:22:06.854661+0000","last_peered":"2024-02-19T15:22:06.854661+0000","last_clean":"2024-02-19T15:22:06.854661+0000","last_became_active":"2024-02-19T15:22:04.831992+0000","last_became_peered":"2024-02-19T15:22:04.831992+0000","last_unstale":"2024-02-19T15:22:06.854661+0000","last_undegraded":"2024-02-19T15:22:06.854661+0000","last_fullsized":"2024-02-19T15:22:06.854661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:50:48.637629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.877939+0000","last_change":"2024-02-19T15:22:05.874836+0000","last_active":"2024-02-19T15:22:05.877939+0000","last_peered":"2024-02-19T15:22:05.877939+0000","last_clean":"2024-02-19T15:22:05.877939+0000","last_became_active":"2024-02-19T15:22:05.874522+0000","last_became_peered":"2024-02-19T15:22:05.874522+0000","last_unstale":"2024-02-19T15:22:05.877939+0000","last_undegraded":"2024-02-19T15:22:05.877939+0000","last_fullsized":"2024-02-19T15:22:05.877939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:43.283192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.869059+0000","last_change":"2024-02-19T15:22:04.837502+0000","last_active":"2024-02-19T15:22:05.869059+0000","last_peered":"2024-02-19T15:22:05.869059+0000","last_clean":"2024-02-19T15:22:05.869059+0000","last_became_active":"2024-02-19T15:22:04.836913+0000","last_became_peered":"2024-02-19T15:22:04.836913+0000","last_unstale":"2024-02-19T15:22:05.869059+0000","last_undegraded":"2024-02-19T15:22:05.869059+0000","last_fullsized":"2024-02-19T15:22:05.869059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:20:00.631607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.867649+0000","last_change":"2024-02-19T15:22:05.864869+0000","last_active":"2024-02-19T15:22:05.867649+0000","last_peered":"2024-02-19T15:22:05.867649+0000","last_clean":"2024-02-19T15:22:05.867649+0000","last_became_active":"2024-02-19T15:22:05.864424+0000","last_became_peered":"2024-02-19T15:22:05.864424+0000","last_unstale":"2024-02-19T15:22:05.867649+0000","last_undegraded":"2024-02-19T15:22:05.867649+0000","last_fullsized":"2024-02-19T15:22:05.867649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:19:29.439405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.838773+0000","last_change":"2024-02-19T15:22:04.847248+0000","last_active":"2024-02-19T15:22:05.838773+0000","last_peered":"2024-02-19T15:22:05.838773+0000","last_clean":"2024-02-19T15:22:05.838773+0000","last_became_active":"2024-02-19T15:22:04.846883+0000","last_became_peered":"2024-02-19T15:22:04.846883+0000","last_unstale":"2024-02-19T15:22:05.838773+0000","last_undegraded":"2024-02-19T15:22:05.838773+0000","last_fullsized":"2024-02-19T15:22:05.838773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:40.779845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.876987+0000","last_change":"2024-02-19T15:22:05.873228+0000","last_active":"2024-02-19T15:22:05.876987+0000","last_peered":"2024-02-19T15:22:05.876987+0000","last_clean":"2024-02-19T15:22:05.876987+0000","last_became_active":"2024-02-19T15:22:05.872458+0000","last_became_peered":"2024-02-19T15:22:05.872458+0000","last_unstale":"2024-02-19T15:22:05.876987+0000","last_undegraded":"2024-02-19T15:22:05.876987+0000","last_fullsized":"2024-02-19T15:22:05.876987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:48:05.400374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854298+0000","last_change":"2024-02-19T15:22:04.843476+0000","last_active":"2024-02-19T15:22:06.854298+0000","last_peered":"2024-02-19T15:22:06.854298+0000","last_clean":"2024-02-19T15:22:06.854298+0000","last_became_active":"2024-02-19T15:22:04.843159+0000","last_became_peered":"2024-02-19T15:22:04.843159+0000","last_unstale":"2024-02-19T15:22:06.854298+0000","last_undegraded":"2024-02-19T15:22:06.854298+0000","last_fullsized":"2024-02-19T15:22:06.854298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:21:58.456669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854827+0000","last_change":"2024-02-19T15:22:05.852049+0000","last_active":"2024-02-19T15:22:06.854827+0000","last_peered":"2024-02-19T15:22:06.854827+0000","last_clean":"2024-02-19T15:22:06.854827+0000","last_became_active":"2024-02-19T15:22:05.851786+0000","last_became_peered":"2024-02-19T15:22:05.851786+0000","last_unstale":"2024-02-19T15:22:06.854827+0000","last_undegraded":"2024-02-19T15:22:06.854827+0000","last_fullsized":"2024-02-19T15:22:06.854827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:31:59.866248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.873203+0000","last_change":"2024-02-19T15:22:04.844209+0000","last_active":"2024-02-19T15:22:05.873203+0000","last_peered":"2024-02-19T15:22:05.873203+0000","last_clean":"2024-02-19T15:22:05.873203+0000","last_became_active":"2024-02-19T15:22:04.842885+0000","last_became_peered":"2024-02-19T15:22:04.842885+0000","last_unstale":"2024-02-19T15:22:05.873203+0000","last_undegraded":"2024-02-19T15:22:05.873203+0000","last_fullsized":"2024-02-19T15:22:05.873203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:27.325777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.878606+0000","last_change":"2024-02-19T15:22:05.877219+0000","last_active":"2024-02-19T15:22:05.878606+0000","last_peered":"2024-02-19T15:22:05.878606+0000","last_clean":"2024-02-19T15:22:05.878606+0000","last_became_active":"2024-02-19T15:22:05.876953+0000","last_became_peered":"2024-02-19T15:22:05.876953+0000","last_unstale":"2024-02-19T15:22:05.878606+0000","last_undegraded":"2024-02-19T15:22:05.878606+0000","last_fullsized":"2024-02-19T15:22:05.878606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:03:40.323060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.868901+0000","last_change":"2024-02-19T15:22:04.837084+0000","last_active":"2024-02-19T15:22:05.868901+0000","last_peered":"2024-02-19T15:22:05.868901+0000","last_clean":"2024-02-19T15:22:05.868901+0000","last_became_active":"2024-02-19T15:22:04.836783+0000","last_became_peered":"2024-02-19T15:22:04.836783+0000","last_unstale":"2024-02-19T15:22:05.868901+0000","last_undegraded":"2024-02-19T15:22:05.868901+0000","last_fullsized":"2024-02-19T15:22:05.868901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:34:14.847696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.878539+0000","last_change":"2024-02-19T15:22:05.876111+0000","last_active":"2024-02-19T15:22:05.878539+0000","last_peered":"2024-02-19T15:22:05.878539+0000","last_clean":"2024-02-19T15:22:05.878539+0000","last_became_active":"2024-02-19T15:22:05.875843+0000","last_became_peered":"2024-02-19T15:22:05.875843+0000","last_unstale":"2024-02-19T15:22:05.878539+0000","last_undegraded":"2024-02-19T15:22:05.878539+0000","last_fullsized":"2024-02-19T15:22:05.878539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:14:40.204009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.871629+0000","last_change":"2024-02-19T15:22:04.839982+0000","last_active":"2024-02-19T15:22:05.871629+0000","last_peered":"2024-02-19T15:22:05.871629+0000","last_clean":"2024-02-19T15:22:05.871629+0000","last_became_active":"2024-02-19T15:22:04.839617+0000","last_became_peered":"2024-02-19T15:22:04.839617+0000","last_unstale":"2024-02-19T15:22:05.871629+0000","last_undegraded":"2024-02-19T15:22:05.871629+0000","last_fullsized":"2024-02-19T15:22:05.871629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:21.137278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.875958+0000","last_change":"2024-02-19T15:22:05.870359+0000","last_active":"2024-02-19T15:22:05.875958+0000","last_peered":"2024-02-19T15:22:05.875958+0000","last_clean":"2024-02-19T15:22:05.875958+0000","last_became_active":"2024-02-19T15:22:05.870083+0000","last_became_peered":"2024-02-19T15:22:05.870083+0000","last_unstale":"2024-02-19T15:22:05.875958+0000","last_undegraded":"2024-02-19T15:22:05.875958+0000","last_fullsized":"2024-02-19T15:22:05.875958+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:33:20.316638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.870508+0000","last_change":"2024-02-19T15:22:04.828045+0000","last_active":"2024-02-19T15:22:05.870508+0000","last_peered":"2024-02-19T15:22:05.870508+0000","last_clean":"2024-02-19T15:22:05.870508+0000","last_became_active":"2024-02-19T15:22:04.827744+0000","last_became_peered":"2024-02-19T15:22:04.827744+0000","last_unstale":"2024-02-19T15:22:05.870508+0000","last_undegraded":"2024-02-19T15:22:05.870508+0000","last_fullsized":"2024-02-19T15:22:05.870508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:21.851058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.860629+0000","last_change":"2024-02-19T15:22:05.849002+0000","last_active":"2024-02-19T15:22:05.860629+0000","last_peered":"2024-02-19T15:22:05.860629+0000","last_clean":"2024-02-19T15:22:05.860629+0000","last_became_active":"2024-02-19T15:22:05.848692+0000","last_became_peered":"2024-02-19T15:22:05.848692+0000","last_unstale":"2024-02-19T15:22:05.860629+0000","last_undegraded":"2024-02-19T15:22:05.860629+0000","last_fullsized":"2024-02-19T15:22:05.860629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:05.756626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.845964+0000","last_change":"2024-02-19T15:22:04.851700+0000","last_active":"2024-02-19T15:22:05.845964+0000","last_peered":"2024-02-19T15:22:05.845964+0000","last_clean":"2024-02-19T15:22:05.845964+0000","last_became_active":"2024-02-19T15:22:04.851270+0000","last_became_peered":"2024-02-19T15:22:04.851270+0000","last_unstale":"2024-02-19T15:22:05.845964+0000","last_undegraded":"2024-02-19T15:22:05.845964+0000","last_fullsized":"2024-02-19T15:22:05.845964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:40.779845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.867560+0000","last_change":"2024-02-19T15:22:05.862153+0000","last_active":"2024-02-19T15:22:05.867560+0000","last_peered":"2024-02-19T15:22:05.867560+0000","last_clean":"2024-02-19T15:22:05.867560+0000","last_became_active":"2024-02-19T15:22:05.861836+0000","last_became_peered":"2024-02-19T15:22:05.861836+0000","last_unstale":"2024-02-19T15:22:05.867560+0000","last_undegraded":"2024-02-19T15:22:05.867560+0000","last_fullsized":"2024-02-19T15:22:05.867560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:48:20.147250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":168,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.850934+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:22:05.850934+0000","last_peered":"2024-02-19T15:22:05.850934+0000","last_clean":"2024-02-19T15:22:05.850934+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:22:05.850934+0000","last_undegraded":"2024-02-19T15:22:05.850934+0000","last_fullsized":"2024-02-19T15:22:05.850934+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.840015+0000","last_change":"2024-02-19T15:22:04.846635+0000","last_active":"2024-02-19T15:22:05.840015+0000","last_peered":"2024-02-19T15:22:05.840015+0000","last_clean":"2024-02-19T15:22:05.840015+0000","last_became_active":"2024-02-19T15:22:04.845191+0000","last_became_peered":"2024-02-19T15:22:04.845191+0000","last_unstale":"2024-02-19T15:22:05.840015+0000","last_undegraded":"2024-02-19T15:22:05.840015+0000","last_fullsized":"2024-02-19T15:22:05.840015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:31:17.425502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.871032+0000","last_change":"2024-02-19T15:22:05.863647+0000","last_active":"2024-02-19T15:22:05.871032+0000","last_peered":"2024-02-19T15:22:05.871032+0000","last_clean":"2024-02-19T15:22:05.871032+0000","last_became_active":"2024-02-19T15:22:05.863370+0000","last_became_peered":"2024-02-19T15:22:05.863370+0000","last_unstale":"2024-02-19T15:22:05.871032+0000","last_undegraded":"2024-02-19T15:22:05.871032+0000","last_fullsized":"2024-02-19T15:22:05.871032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:02.917816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.850879+0000","last_change":"2024-02-19T15:22:04.849295+0000","last_active":"2024-02-19T15:22:05.850879+0000","last_peered":"2024-02-19T15:22:05.850879+0000","last_clean":"2024-02-19T15:22:05.850879+0000","last_became_active":"2024-02-19T15:22:04.849004+0000","last_became_peered":"2024-02-19T15:22:04.849004+0000","last_unstale":"2024-02-19T15:22:05.850879+0000","last_undegraded":"2024-02-19T15:22:05.850879+0000","last_fullsized":"2024-02-19T15:22:05.850879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:37.601588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.867078+0000","last_change":"2024-02-19T15:22:05.861044+0000","last_active":"2024-02-19T15:22:05.867078+0000","last_peered":"2024-02-19T15:22:05.867078+0000","last_clean":"2024-02-19T15:22:05.867078+0000","last_became_active":"2024-02-19T15:22:05.860784+0000","last_became_peered":"2024-02-19T15:22:05.860784+0000","last_unstale":"2024-02-19T15:22:05.867078+0000","last_undegraded":"2024-02-19T15:22:05.867078+0000","last_fullsized":"2024-02-19T15:22:05.867078+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:21:59.462272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.843126+0000","last_change":"2024-02-19T15:22:04.842486+0000","last_active":"2024-02-19T15:22:05.843126+0000","last_peered":"2024-02-19T15:22:05.843126+0000","last_clean":"2024-02-19T15:22:05.843126+0000","last_became_active":"2024-02-19T15:22:04.840062+0000","last_became_peered":"2024-02-19T15:22:04.840062+0000","last_unstale":"2024-02-19T15:22:05.843126+0000","last_undegraded":"2024-02-19T15:22:05.843126+0000","last_fullsized":"2024-02-19T15:22:05.843126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:17.404136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.858264+0000","last_change":"2024-02-19T15:22:05.853794+0000","last_active":"2024-02-19T15:22:05.858264+0000","last_peered":"2024-02-19T15:22:05.858264+0000","last_clean":"2024-02-19T15:22:05.858264+0000","last_became_active":"2024-02-19T15:22:05.851972+0000","last_became_peered":"2024-02-19T15:22:05.851972+0000","last_unstale":"2024-02-19T15:22:05.858264+0000","last_undegraded":"2024-02-19T15:22:05.858264+0000","last_fullsized":"2024-02-19T15:22:05.858264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:32:56.909524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.842890+0000","last_change":"2024-02-19T15:22:04.851077+0000","last_active":"2024-02-19T15:22:05.842890+0000","last_peered":"2024-02-19T15:22:05.842890+0000","last_clean":"2024-02-19T15:22:05.842890+0000","last_became_active":"2024-02-19T15:22:04.850734+0000","last_became_peered":"2024-02-19T15:22:04.850734+0000","last_unstale":"2024-02-19T15:22:05.842890+0000","last_undegraded":"2024-02-19T15:22:05.842890+0000","last_fullsized":"2024-02-19T15:22:05.842890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:37.601588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.864456+0000","last_change":"2024-02-19T15:22:05.859004+0000","last_active":"2024-02-19T15:22:05.864456+0000","last_peered":"2024-02-19T15:22:05.864456+0000","last_clean":"2024-02-19T15:22:05.864456+0000","last_became_active":"2024-02-19T15:22:05.858524+0000","last_became_peered":"2024-02-19T15:22:05.858524+0000","last_unstale":"2024-02-19T15:22:05.864456+0000","last_undegraded":"2024-02-19T15:22:05.864456+0000","last_fullsized":"2024-02-19T15:22:05.864456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:34:37.414912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.849782+0000","last_change":"2024-02-19T15:22:04.832556+0000","last_active":"2024-02-19T15:22:05.849782+0000","last_peered":"2024-02-19T15:22:05.849782+0000","last_clean":"2024-02-19T15:22:05.849782+0000","last_became_active":"2024-02-19T15:22:04.832261+0000","last_became_peered":"2024-02-19T15:22:04.832261+0000","last_unstale":"2024-02-19T15:22:05.849782+0000","last_undegraded":"2024-02-19T15:22:05.849782+0000","last_fullsized":"2024-02-19T15:22:05.849782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:37.601588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.859005+0000","last_change":"2024-02-19T15:22:05.850119+0000","last_active":"2024-02-19T15:22:05.859005+0000","last_peered":"2024-02-19T15:22:05.859005+0000","last_clean":"2024-02-19T15:22:05.859005+0000","last_became_active":"2024-02-19T15:22:05.849831+0000","last_became_peered":"2024-02-19T15:22:05.849831+0000","last_unstale":"2024-02-19T15:22:05.859005+0000","last_undegraded":"2024-02-19T15:22:05.859005+0000","last_fullsized":"2024-02-19T15:22:05.859005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:32:56.909524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.854977+0000","last_change":"2024-02-19T15:22:04.850106+0000","last_active":"2024-02-19T15:22:05.854977+0000","last_peered":"2024-02-19T15:22:05.854977+0000","last_clean":"2024-02-19T15:22:05.854977+0000","last_became_active":"2024-02-19T15:22:04.849467+0000","last_became_peered":"2024-02-19T15:22:04.849467+0000","last_unstale":"2024-02-19T15:22:05.854977+0000","last_undegraded":"2024-02-19T15:22:05.854977+0000","last_fullsized":"2024-02-19T15:22:05.854977+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:32:55.903921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.860264+0000","last_change":"2024-02-19T15:22:05.857834+0000","last_active":"2024-02-19T15:22:05.860264+0000","last_peered":"2024-02-19T15:22:05.860264+0000","last_clean":"2024-02-19T15:22:05.860264+0000","last_became_active":"2024-02-19T15:22:05.857488+0000","last_became_peered":"2024-02-19T15:22:05.857488+0000","last_unstale":"2024-02-19T15:22:05.860264+0000","last_undegraded":"2024-02-19T15:22:05.860264+0000","last_fullsized":"2024-02-19T15:22:05.860264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:48:20.147250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.849966+0000","last_change":"2024-02-19T15:22:04.830568+0000","last_active":"2024-02-19T15:22:05.849966+0000","last_peered":"2024-02-19T15:22:05.849966+0000","last_clean":"2024-02-19T15:22:05.849966+0000","last_became_active":"2024-02-19T15:22:04.829131+0000","last_became_peered":"2024-02-19T15:22:04.829131+0000","last_unstale":"2024-02-19T15:22:05.849966+0000","last_undegraded":"2024-02-19T15:22:05.849966+0000","last_fullsized":"2024-02-19T15:22:05.849966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:37:50.991116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.871054+0000","last_change":"2024-02-19T15:22:05.866131+0000","last_active":"2024-02-19T15:22:05.871054+0000","last_peered":"2024-02-19T15:22:05.871054+0000","last_clean":"2024-02-19T15:22:05.871054+0000","last_became_active":"2024-02-19T15:22:05.865820+0000","last_became_peered":"2024-02-19T15:22:05.865820+0000","last_unstale":"2024-02-19T15:22:05.871054+0000","last_undegraded":"2024-02-19T15:22:05.871054+0000","last_fullsized":"2024-02-19T15:22:05.871054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:05.756626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854644+0000","last_change":"2024-02-19T15:22:04.840658+0000","last_active":"2024-02-19T15:22:06.854644+0000","last_peered":"2024-02-19T15:22:06.854644+0000","last_clean":"2024-02-19T15:22:06.854644+0000","last_became_active":"2024-02-19T15:22:04.840167+0000","last_became_peered":"2024-02-19T15:22:04.840167+0000","last_unstale":"2024-02-19T15:22:06.854644+0000","last_undegraded":"2024-02-19T15:22:06.854644+0000","last_fullsized":"2024-02-19T15:22:06.854644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:26:19.726455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.859322+0000","last_change":"2024-02-19T15:22:05.854827+0000","last_active":"2024-02-19T15:22:05.859322+0000","last_peered":"2024-02-19T15:22:05.859322+0000","last_clean":"2024-02-19T15:22:05.859322+0000","last_became_active":"2024-02-19T15:22:05.854579+0000","last_became_peered":"2024-02-19T15:22:05.854579+0000","last_unstale":"2024-02-19T15:22:05.859322+0000","last_undegraded":"2024-02-19T15:22:05.859322+0000","last_fullsized":"2024-02-19T15:22:05.859322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:11:05.022701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.849999+0000","last_change":"2024-02-19T15:22:04.848481+0000","last_active":"2024-02-19T15:22:05.849999+0000","last_peered":"2024-02-19T15:22:05.849999+0000","last_clean":"2024-02-19T15:22:05.849999+0000","last_became_active":"2024-02-19T15:22:04.848224+0000","last_became_peered":"2024-02-19T15:22:04.848224+0000","last_unstale":"2024-02-19T15:22:05.849999+0000","last_undegraded":"2024-02-19T15:22:05.849999+0000","last_fullsized":"2024-02-19T15:22:05.849999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:21.851058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.874326+0000","last_change":"2024-02-19T15:22:05.868543+0000","last_active":"2024-02-19T15:22:05.874326+0000","last_peered":"2024-02-19T15:22:05.874326+0000","last_clean":"2024-02-19T15:22:05.874326+0000","last_became_active":"2024-02-19T15:22:05.868307+0000","last_became_peered":"2024-02-19T15:22:05.868307+0000","last_unstale":"2024-02-19T15:22:05.874326+0000","last_undegraded":"2024-02-19T15:22:05.874326+0000","last_fullsized":"2024-02-19T15:22:05.874326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:22.142882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.844375+0000","last_change":"2024-02-19T15:22:04.841547+0000","last_active":"2024-02-19T15:22:05.844375+0000","last_peered":"2024-02-19T15:22:05.844375+0000","last_clean":"2024-02-19T15:22:05.844375+0000","last_became_active":"2024-02-19T15:22:04.841280+0000","last_became_peered":"2024-02-19T15:22:04.841280+0000","last_unstale":"2024-02-19T15:22:05.844375+0000","last_undegraded":"2024-02-19T15:22:05.844375+0000","last_fullsized":"2024-02-19T15:22:05.844375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:40.779845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.870739+0000","last_change":"2024-02-19T15:22:05.863084+0000","last_active":"2024-02-19T15:22:05.870739+0000","last_peered":"2024-02-19T15:22:05.870739+0000","last_clean":"2024-02-19T15:22:05.870739+0000","last_became_active":"2024-02-19T15:22:05.862820+0000","last_became_peered":"2024-02-19T15:22:05.862820+0000","last_unstale":"2024-02-19T15:22:05.870739+0000","last_undegraded":"2024-02-19T15:22:05.870739+0000","last_fullsized":"2024-02-19T15:22:05.870739+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:37:51.996719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.842582+0000","last_change":"2024-02-19T15:22:04.840145+0000","last_active":"2024-02-19T15:22:05.842582+0000","last_peered":"2024-02-19T15:22:05.842582+0000","last_clean":"2024-02-19T15:22:05.842582+0000","last_became_active":"2024-02-19T15:22:04.839832+0000","last_became_peered":"2024-02-19T15:22:04.839832+0000","last_unstale":"2024-02-19T15:22:05.842582+0000","last_undegraded":"2024-02-19T15:22:05.842582+0000","last_fullsized":"2024-02-19T15:22:05.842582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:27.325777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.870898+0000","last_change":"2024-02-19T15:22:05.862437+0000","last_active":"2024-02-19T15:22:05.870898+0000","last_peered":"2024-02-19T15:22:05.870898+0000","last_clean":"2024-02-19T15:22:05.870898+0000","last_became_active":"2024-02-19T15:22:05.855551+0000","last_became_peered":"2024-02-19T15:22:05.855551+0000","last_unstale":"2024-02-19T15:22:05.870898+0000","last_undegraded":"2024-02-19T15:22:05.870898+0000","last_fullsized":"2024-02-19T15:22:05.870898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:38.607191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.872781+0000","last_change":"2024-02-19T15:22:04.849761+0000","last_active":"2024-02-19T15:22:05.872781+0000","last_peered":"2024-02-19T15:22:05.872781+0000","last_clean":"2024-02-19T15:22:05.872781+0000","last_became_active":"2024-02-19T15:22:04.849413+0000","last_became_peered":"2024-02-19T15:22:04.849413+0000","last_unstale":"2024-02-19T15:22:05.872781+0000","last_undegraded":"2024-02-19T15:22:05.872781+0000","last_fullsized":"2024-02-19T15:22:05.872781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:37:50.991116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854363+0000","last_change":"2024-02-19T15:22:05.871270+0000","last_active":"2024-02-19T15:22:06.854363+0000","last_peered":"2024-02-19T15:22:06.854363+0000","last_clean":"2024-02-19T15:22:06.854363+0000","last_became_active":"2024-02-19T15:22:05.870986+0000","last_became_peered":"2024-02-19T15:22:05.870986+0000","last_unstale":"2024-02-19T15:22:06.854363+0000","last_undegraded":"2024-02-19T15:22:06.854363+0000","last_fullsized":"2024-02-19T15:22:06.854363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:22:17.882758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.872436+0000","last_change":"2024-02-19T15:22:05.864283+0000","last_active":"2024-02-19T15:22:05.872436+0000","last_peered":"2024-02-19T15:22:05.872436+0000","last_clean":"2024-02-19T15:22:05.872436+0000","last_became_active":"2024-02-19T15:22:05.863996+0000","last_became_peered":"2024-02-19T15:22:05.863996+0000","last_unstale":"2024-02-19T15:22:05.872436+0000","last_undegraded":"2024-02-19T15:22:05.872436+0000","last_fullsized":"2024-02-19T15:22:05.872436+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:41.123853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.875933+0000","last_change":"2024-02-19T15:22:04.855753+0000","last_active":"2024-02-19T15:22:05.875933+0000","last_peered":"2024-02-19T15:22:05.875933+0000","last_clean":"2024-02-19T15:22:05.875933+0000","last_became_active":"2024-02-19T15:22:04.855303+0000","last_became_peered":"2024-02-19T15:22:04.855303+0000","last_unstale":"2024-02-19T15:22:05.875933+0000","last_undegraded":"2024-02-19T15:22:05.875933+0000","last_fullsized":"2024-02-19T15:22:05.875933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:04.751023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.875438+0000","last_change":"2024-02-19T15:22:05.870398+0000","last_active":"2024-02-19T15:22:05.875438+0000","last_peered":"2024-02-19T15:22:05.875438+0000","last_clean":"2024-02-19T15:22:05.875438+0000","last_became_active":"2024-02-19T15:22:05.869963+0000","last_became_peered":"2024-02-19T15:22:05.869963+0000","last_unstale":"2024-02-19T15:22:05.875438+0000","last_undegraded":"2024-02-19T15:22:05.875438+0000","last_fullsized":"2024-02-19T15:22:05.875438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:21:49.261015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.868588+0000","last_change":"2024-02-19T15:22:04.849798+0000","last_active":"2024-02-19T15:22:05.868588+0000","last_peered":"2024-02-19T15:22:05.868588+0000","last_clean":"2024-02-19T15:22:05.868588+0000","last_became_active":"2024-02-19T15:22:04.849115+0000","last_became_peered":"2024-02-19T15:22:04.849115+0000","last_unstale":"2024-02-19T15:22:05.868588+0000","last_undegraded":"2024-02-19T15:22:05.868588+0000","last_fullsized":"2024-02-19T15:22:05.868588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:01.912213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.867825+0000","last_change":"2024-02-19T15:22:05.861837+0000","last_active":"2024-02-19T15:22:05.867825+0000","last_peered":"2024-02-19T15:22:05.867825+0000","last_clean":"2024-02-19T15:22:05.867825+0000","last_became_active":"2024-02-19T15:22:05.861570+0000","last_became_peered":"2024-02-19T15:22:05.861570+0000","last_unstale":"2024-02-19T15:22:05.867825+0000","last_undegraded":"2024-02-19T15:22:05.867825+0000","last_fullsized":"2024-02-19T15:22:05.867825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:42:49.483797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.870819+0000","last_change":"2024-02-19T15:22:04.841218+0000","last_active":"2024-02-19T15:22:05.870819+0000","last_peered":"2024-02-19T15:22:05.870819+0000","last_clean":"2024-02-19T15:22:05.870819+0000","last_became_active":"2024-02-19T15:22:04.840740+0000","last_became_peered":"2024-02-19T15:22:04.840740+0000","last_unstale":"2024-02-19T15:22:05.870819+0000","last_undegraded":"2024-02-19T15:22:05.870819+0000","last_fullsized":"2024-02-19T15:22:05.870819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:21.137278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.850921+0000","last_change":"2024-02-19T15:22:05.847219+0000","last_active":"2024-02-19T15:22:05.850921+0000","last_peered":"2024-02-19T15:22:05.850921+0000","last_clean":"2024-02-19T15:22:05.850921+0000","last_became_active":"2024-02-19T15:22:05.846948+0000","last_became_peered":"2024-02-19T15:22:05.846948+0000","last_unstale":"2024-02-19T15:22:05.850921+0000","last_undegraded":"2024-02-19T15:22:05.850921+0000","last_fullsized":"2024-02-19T15:22:05.850921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:28.331381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.870264+0000","last_change":"2024-02-19T15:22:04.843469+0000","last_active":"2024-02-19T15:22:05.870264+0000","last_peered":"2024-02-19T15:22:05.870264+0000","last_clean":"2024-02-19T15:22:05.870264+0000","last_became_active":"2024-02-19T15:22:04.842098+0000","last_became_peered":"2024-02-19T15:22:04.842098+0000","last_unstale":"2024-02-19T15:22:05.870264+0000","last_undegraded":"2024-02-19T15:22:05.870264+0000","last_fullsized":"2024-02-19T15:22:05.870264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:32:33.319050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509445,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]}]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":798720,"data_stored":658533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]}]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":60808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":67709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]}]},{"osd":3,"up_from":14,"seq":60129542148,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":212992,"data_stored":69195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]}]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31364,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340476,"statfs":{"total":96636764160,"available":96604647424,"internally_reserved":0,"allocated":188416,"data_stored":60808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1040000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":208896,"data_stored":67743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.997}]}]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31944,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339896,"statfs":{"total":96636764160,"available":96604053504,"internally_reserved":0,"allocated":782336,"data_stored":651176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:22:09.967 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:22:10.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.142+0000 7f2688400640 1 Processor -- start 2024-02-19T15:22:10.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.142+0000 7f2688400640 1 -- start start 2024-02-19T15:22:10.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.142+0000 7f2688400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f268015b1a0 0x7f268015b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:10.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.146+0000 7f2688400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f268006c9a0 0x7f268015bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:10.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.146+0000 7f2688400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f268015c000 con 0x7f2680071350 2024-02-19T15:22:10.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.146+0000 7f2688400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26800718c0 con 0x7f268015b1a0 2024-02-19T15:22:10.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.146+0000 7f2688400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2680071ba0 con 0x7f268006c9a0 2024-02-19T15:22:10.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.146+0000 7f2685000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f2680071350 0x7f268015c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43988/0 (socket says 172.21.15.47:43988) 2024-02-19T15:22:10.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.146+0000 7f2685000640 1 -- 172.21.15.47:0/956080671 learned_addr learned my addr 172.21.15.47:0/956080671 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:10.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.146+0000 7f2685a00640 1 --2- 172.21.15.47:0/956080671 >> v2:172.21.15.116:3300/0 conn(0x7f268015b1a0 0x7f268015b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:10.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.146+0000 7f2686400640 1 --2- 172.21.15.47:0/956080671 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f268006c9a0 0x7f268015bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:10.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.146+0000 7f2686400640 1 -- 172.21.15.47:0/956080671 >> v2:172.21.15.116:3300/0 conn(0x7f268015b1a0 msgr2=0x7f268015b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:10.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.146+0000 7f2686400640 1 --2- 172.21.15.47:0/956080671 >> v2:172.21.15.116:3300/0 conn(0x7f268015b1a0 0x7f268015b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.146+0000 7f2686400640 1 -- 172.21.15.47:0/956080671 >> v1:172.21.15.47:6789/0 conn(0x7f2680071350 legacy=0x7f268015c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:10.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.146+0000 7f2686400640 1 -- 172.21.15.47:0/956080671 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2680071e80 con 0x7f268006c9a0 2024-02-19T15:22:10.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.146+0000 7f2685a00640 1 --2- 172.21.15.47:0/956080671 >> v2:172.21.15.116:3300/0 conn(0x7f268015b1a0 0x7f268015b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:10.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.146+0000 7f267fe00640 1 -- 172.21.15.47:0/956080671 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1851071725 0 0) 0x7f268015c000 con 0x7f2680071350 2024-02-19T15:22:10.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.146+0000 7f2686400640 1 --2- 172.21.15.47:0/956080671 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f268006c9a0 0x7f268015bac0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2678009dc0 tx=0x7f267801c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c28d614c95ab9daa server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:10.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.146+0000 7f267fe00640 1 -- 172.21.15.47:0/956080671 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26780a5a00 con 0x7f268006c9a0 2024-02-19T15:22:10.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f267fe00640 1 -- 172.21.15.47:0/956080671 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26780a5b60 con 0x7f268006c9a0 2024-02-19T15:22:10.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f267fe00640 1 -- 172.21.15.47:0/956080671 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26780ad440 con 0x7f268006c9a0 2024-02-19T15:22:10.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2688400640 1 -- 172.21.15.47:0/956080671 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f268006c9a0 msgr2=0x7f268015bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:10.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2688400640 1 --2- 172.21.15.47:0/956080671 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f268006c9a0 0x7f268015bac0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2678009dc0 tx=0x7f267801c920 comp rx=0 tx=0).stop 2024-02-19T15:22:10.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2688400640 1 -- 172.21.15.47:0/956080671 shutdown_connections 2024-02-19T15:22:10.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2688400640 1 --2- 172.21.15.47:0/956080671 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f268006c9a0 0x7f268015bac0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2688400640 1 --2- 172.21.15.47:0/956080671 >> v2:172.21.15.116:3300/0 conn(0x7f268015b1a0 0x7f268015b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2688400640 1 -- 172.21.15.47:0/956080671 >> 172.21.15.47:0/956080671 conn(0x7f26800b16a0 msgr2=0x7f26800b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:10.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2688400640 1 -- 172.21.15.47:0/956080671 shutdown_connections 2024-02-19T15:22:10.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2688400640 1 -- 172.21.15.47:0/956080671 wait complete. 2024-02-19T15:22:10.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2688400640 1 Processor -- start 2024-02-19T15:22:10.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2688400640 1 -- start start 2024-02-19T15:22:10.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2688400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f268006c9a0 0x7f268014c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:10.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2688400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2680071350 0x7f268014cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:10.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2688400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f268015b1a0 0x7f26800cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:10.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2688400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2680148cf0 con 0x7f2680071350 2024-02-19T15:22:10.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2688400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2680148fd0 con 0x7f268006c9a0 2024-02-19T15:22:10.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2688400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26801492b0 con 0x7f268015b1a0 2024-02-19T15:22:10.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2685000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2680071350 0x7f268014cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:10.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2685a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f268006c9a0 0x7f268014c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:10.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2685000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2680071350 0x7f268014cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52878/0 (socket says 172.21.15.47:52878) 2024-02-19T15:22:10.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2685000640 1 -- 172.21.15.47:0/4092822445 learned_addr learned my addr 172.21.15.47:0/4092822445 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:10.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2686400640 1 --2- 172.21.15.47:0/4092822445 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f268015b1a0 0x7f26800cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:10.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2685000640 1 -- 172.21.15.47:0/4092822445 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f268015b1a0 msgr2=0x7f26800cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:10.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2685000640 1 --2- 172.21.15.47:0/4092822445 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f268015b1a0 0x7f26800cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2685000640 1 -- 172.21.15.47:0/4092822445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f268006c9a0 msgr2=0x7f268014c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:10.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2685000640 1 --2- 172.21.15.47:0/4092822445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f268006c9a0 0x7f268014c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2685000640 1 -- 172.21.15.47:0/4092822445 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2674009420 con 0x7f2680071350 2024-02-19T15:22:10.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.150+0000 7f2685a00640 1 --2- 172.21.15.47:0/4092822445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f268006c9a0 0x7f268014c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:10.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.154+0000 7f2685000640 1 --2- 172.21.15.47:0/4092822445 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2680071350 0x7f268014cf30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f266c0011f0 tx=0x7f266c00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:10.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.154+0000 7f267d600640 1 -- 172.21.15.47:0/4092822445 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f266c0074e0 con 0x7f2680071350 2024-02-19T15:22:10.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.154+0000 7f267d600640 1 -- 172.21.15.47:0/4092822445 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f266c007640 con 0x7f2680071350 2024-02-19T15:22:10.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.154+0000 7f2688400640 1 -- 172.21.15.47:0/4092822445 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2678009a70 con 0x7f2680071350 2024-02-19T15:22:10.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.154+0000 7f267d600640 1 -- 172.21.15.47:0/4092822445 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f266c004a10 con 0x7f2680071350 2024-02-19T15:22:10.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.154+0000 7f2688400640 1 -- 172.21.15.47:0/4092822445 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2680148910 con 0x7f2680071350 2024-02-19T15:22:10.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.154+0000 7f2688400640 1 -- 172.21.15.47:0/4092822445 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2680075360 con 0x7f2680071350 2024-02-19T15:22:10.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.154+0000 7f267d600640 1 -- 172.21.15.47:0/4092822445 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f266c004b70 con 0x7f2680071350 2024-02-19T15:22:10.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.154+0000 7f267d600640 1 --2- 172.21.15.47:0/4092822445 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2660066e70 0x7f2660069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:10.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.154+0000 7f267d600640 1 -- 172.21.15.47:0/4092822445 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f266c083f40 con 0x7f2680071350 2024-02-19T15:22:10.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.154+0000 7f2685a00640 1 --2- 172.21.15.47:0/4092822445 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2660066e70 0x7f2660069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:10.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.158+0000 7f2685a00640 1 --2- 172.21.15.47:0/4092822445 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2660066e70 0x7f2660069330 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2674009fd0 tx=0x7f2674009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:10.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.158+0000 7f267d600640 1 -- 172.21.15.47:0/4092822445 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f266c050d40 con 0x7f2680071350 2024-02-19T15:22:10.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.318+0000 7f2688400640 1 -- 172.21.15.47:0/4092822445 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2680149420 con 0x7f2660066e70 2024-02-19T15:22:10.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.326+0000 7f267d600640 1 -- 172.21.15.47:0/4092822445 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264584 (secure 0 0 0) 0x7f2680149420 con 0x7f2660066e70 2024-02-19T15:22:10.330 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:10.332 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:22:10.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.330+0000 7f2688400640 1 -- 172.21.15.47:0/4092822445 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2660066e70 msgr2=0x7f2660069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:10.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.330+0000 7f2688400640 1 --2- 172.21.15.47:0/4092822445 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2660066e70 0x7f2660069330 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2674009fd0 tx=0x7f2674009190 comp rx=0 tx=0).stop 2024-02-19T15:22:10.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.330+0000 7f2688400640 1 -- 172.21.15.47:0/4092822445 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2680071350 msgr2=0x7f268014cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:10.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.330+0000 7f2688400640 1 --2- 172.21.15.47:0/4092822445 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2680071350 0x7f268014cf30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f266c0011f0 tx=0x7f266c00c9b0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.330+0000 7f2688400640 1 -- 172.21.15.47:0/4092822445 shutdown_connections 2024-02-19T15:22:10.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.330+0000 7f2688400640 1 --2- 172.21.15.47:0/4092822445 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2660066e70 0x7f2660069330 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.330+0000 7f2688400640 1 --2- 172.21.15.47:0/4092822445 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f268015b1a0 0x7f26800cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.330+0000 7f2688400640 1 --2- 172.21.15.47:0/4092822445 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2680071350 0x7f268014cf30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.330+0000 7f2688400640 1 --2- 172.21.15.47:0/4092822445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f268006c9a0 0x7f268014c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.330+0000 7f2688400640 1 -- 172.21.15.47:0/4092822445 >> 172.21.15.47:0/4092822445 conn(0x7f26800b16a0 msgr2=0x7f26800709d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:10.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.330+0000 7f2688400640 1 -- 172.21.15.47:0/4092822445 shutdown_connections 2024-02-19T15:22:10.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.330+0000 7f2688400640 1 -- 172.21.15.47:0/4092822445 wait complete. 2024-02-19T15:22:10.351 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-02-19T15:22:09.150908+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235568,"kb_used_data":2752,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739152,"statfs":{"total":773094113280,"available":772852891648,"internally_reserved":0,"allocated":2818048,"data_stored":1711467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.338037"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.873472+0000","last_change":"2024-02-19T15:22:05.868047+0000","last_active":"2024-02-19T15:22:05.873472+0000","last_peered":"2024-02-19T15:22:05.873472+0000","last_clean":"2024-02-19T15:22:05.873472+0000","last_became_active":"2024-02-19T15:22:05.867538+0000","last_became_peered":"2024-02-19T15:22:05.867538+0000","last_unstale":"2024-02-19T15:22:05.873472+0000","last_undegraded":"2024-02-19T15:22:05.873472+0000","last_fullsized":"2024-02-19T15:22:05.873472+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:33.203920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.866600+0000","last_change":"2024-02-19T15:22:05.861729+0000","last_active":"2024-02-19T15:22:05.866600+0000","last_peered":"2024-02-19T15:22:05.866600+0000","last_clean":"2024-02-19T15:22:05.866600+0000","last_became_active":"2024-02-19T15:22:05.861455+0000","last_became_peered":"2024-02-19T15:22:05.861455+0000","last_unstale":"2024-02-19T15:22:05.866600+0000","last_undegraded":"2024-02-19T15:22:05.866600+0000","last_fullsized":"2024-02-19T15:22:05.866600+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:43:34.980001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.866483+0000","last_change":"2024-02-19T15:22:05.857715+0000","last_active":"2024-02-19T15:22:05.866483+0000","last_peered":"2024-02-19T15:22:05.866483+0000","last_clean":"2024-02-19T15:22:05.866483+0000","last_became_active":"2024-02-19T15:22:05.857360+0000","last_became_peered":"2024-02-19T15:22:05.857360+0000","last_unstale":"2024-02-19T15:22:05.866483+0000","last_undegraded":"2024-02-19T15:22:05.866483+0000","last_fullsized":"2024-02-19T15:22:05.866483+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:50:49.643232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.860650+0000","last_change":"2024-02-19T15:22:05.850537+0000","last_active":"2024-02-19T15:22:05.860650+0000","last_peered":"2024-02-19T15:22:05.860650+0000","last_clean":"2024-02-19T15:22:05.860650+0000","last_became_active":"2024-02-19T15:22:05.850288+0000","last_became_peered":"2024-02-19T15:22:05.850288+0000","last_unstale":"2024-02-19T15:22:05.860650+0000","last_undegraded":"2024-02-19T15:22:05.860650+0000","last_fullsized":"2024-02-19T15:22:05.860650+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:03:09.391902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.876968+0000","last_change":"2024-02-19T15:22:05.872936+0000","last_active":"2024-02-19T15:22:05.876968+0000","last_peered":"2024-02-19T15:22:05.876968+0000","last_clean":"2024-02-19T15:22:05.876968+0000","last_became_active":"2024-02-19T15:22:05.871969+0000","last_became_peered":"2024-02-19T15:22:05.871969+0000","last_unstale":"2024-02-19T15:22:05.876968+0000","last_undegraded":"2024-02-19T15:22:05.876968+0000","last_fullsized":"2024-02-19T15:22:05.876968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:21:49.261015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.860703+0000","last_change":"2024-02-19T15:22:05.853731+0000","last_active":"2024-02-19T15:22:05.860703+0000","last_peered":"2024-02-19T15:22:05.860703+0000","last_clean":"2024-02-19T15:22:05.860703+0000","last_became_active":"2024-02-19T15:22:05.853423+0000","last_became_peered":"2024-02-19T15:22:05.853423+0000","last_unstale":"2024-02-19T15:22:05.860703+0000","last_undegraded":"2024-02-19T15:22:05.860703+0000","last_fullsized":"2024-02-19T15:22:05.860703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:48:20.147250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.866634+0000","last_change":"2024-02-19T15:22:05.860498+0000","last_active":"2024-02-19T15:22:05.866634+0000","last_peered":"2024-02-19T15:22:05.866634+0000","last_clean":"2024-02-19T15:22:05.866634+0000","last_became_active":"2024-02-19T15:22:05.860283+0000","last_became_peered":"2024-02-19T15:22:05.860283+0000","last_unstale":"2024-02-19T15:22:05.866634+0000","last_undegraded":"2024-02-19T15:22:05.866634+0000","last_fullsized":"2024-02-19T15:22:05.866634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:03:09.391902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.870717+0000","last_change":"2024-02-19T15:22:05.860461+0000","last_active":"2024-02-19T15:22:05.870717+0000","last_peered":"2024-02-19T15:22:05.870717+0000","last_clean":"2024-02-19T15:22:05.870717+0000","last_became_active":"2024-02-19T15:22:05.860119+0000","last_became_peered":"2024-02-19T15:22:05.860119+0000","last_unstale":"2024-02-19T15:22:05.870717+0000","last_undegraded":"2024-02-19T15:22:05.870717+0000","last_fullsized":"2024-02-19T15:22:05.870717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:22.856662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.868344+0000","last_change":"2024-02-19T15:22:05.863575+0000","last_active":"2024-02-19T15:22:05.868344+0000","last_peered":"2024-02-19T15:22:05.868344+0000","last_clean":"2024-02-19T15:22:05.868344+0000","last_became_active":"2024-02-19T15:22:05.863306+0000","last_became_peered":"2024-02-19T15:22:05.863306+0000","last_unstale":"2024-02-19T15:22:05.868344+0000","last_undegraded":"2024-02-19T15:22:05.868344+0000","last_fullsized":"2024-02-19T15:22:05.868344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:43:34.980001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854624+0000","last_change":"2024-02-19T15:22:05.867622+0000","last_active":"2024-02-19T15:22:06.854624+0000","last_peered":"2024-02-19T15:22:06.854624+0000","last_clean":"2024-02-19T15:22:06.854624+0000","last_became_active":"2024-02-19T15:22:05.867352+0000","last_became_peered":"2024-02-19T15:22:05.867352+0000","last_unstale":"2024-02-19T15:22:06.854624+0000","last_undegraded":"2024-02-19T15:22:06.854624+0000","last_fullsized":"2024-02-19T15:22:06.854624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:08.091823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854743+0000","last_change":"2024-02-19T15:22:05.859648+0000","last_active":"2024-02-19T15:22:06.854743+0000","last_peered":"2024-02-19T15:22:06.854743+0000","last_clean":"2024-02-19T15:22:06.854743+0000","last_became_active":"2024-02-19T15:22:05.859381+0000","last_became_peered":"2024-02-19T15:22:05.859381+0000","last_unstale":"2024-02-19T15:22:06.854743+0000","last_undegraded":"2024-02-19T15:22:06.854743+0000","last_fullsized":"2024-02-19T15:22:06.854743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:35:11.566219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.874393+0000","last_change":"2024-02-19T15:22:05.869009+0000","last_active":"2024-02-19T15:22:05.874393+0000","last_peered":"2024-02-19T15:22:05.874393+0000","last_clean":"2024-02-19T15:22:05.874393+0000","last_became_active":"2024-02-19T15:22:05.868709+0000","last_became_peered":"2024-02-19T15:22:05.868709+0000","last_unstale":"2024-02-19T15:22:05.874393+0000","last_undegraded":"2024-02-19T15:22:05.874393+0000","last_fullsized":"2024-02-19T15:22:05.874393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:28.331381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.852782+0000","last_change":"2024-02-19T15:22:05.848558+0000","last_active":"2024-02-19T15:22:05.852782+0000","last_peered":"2024-02-19T15:22:05.852782+0000","last_clean":"2024-02-19T15:22:05.852782+0000","last_became_active":"2024-02-19T15:22:05.848288+0000","last_became_peered":"2024-02-19T15:22:05.848288+0000","last_unstale":"2024-02-19T15:22:05.852782+0000","last_undegraded":"2024-02-19T15:22:05.852782+0000","last_fullsized":"2024-02-19T15:22:05.852782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:41.785448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.857444+0000","last_change":"2024-02-19T15:22:05.846265+0000","last_active":"2024-02-19T15:22:05.857444+0000","last_peered":"2024-02-19T15:22:05.857444+0000","last_clean":"2024-02-19T15:22:05.857444+0000","last_became_active":"2024-02-19T15:22:05.841757+0000","last_became_peered":"2024-02-19T15:22:05.841757+0000","last_unstale":"2024-02-19T15:22:05.857444+0000","last_undegraded":"2024-02-19T15:22:05.857444+0000","last_fullsized":"2024-02-19T15:22:05.857444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:16:22.481692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.858006+0000","last_change":"2024-02-19T15:22:05.850786+0000","last_active":"2024-02-19T15:22:05.858006+0000","last_peered":"2024-02-19T15:22:05.858006+0000","last_clean":"2024-02-19T15:22:05.858006+0000","last_became_active":"2024-02-19T15:22:05.850136+0000","last_became_peered":"2024-02-19T15:22:05.850136+0000","last_unstale":"2024-02-19T15:22:05.858006+0000","last_undegraded":"2024-02-19T15:22:05.858006+0000","last_fullsized":"2024-02-19T15:22:05.858006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:02.917816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.865077+0000","last_change":"2024-02-19T15:22:05.862076+0000","last_active":"2024-02-19T15:22:05.865077+0000","last_peered":"2024-02-19T15:22:05.865077+0000","last_clean":"2024-02-19T15:22:05.865077+0000","last_became_active":"2024-02-19T15:22:05.861826+0000","last_became_peered":"2024-02-19T15:22:05.861826+0000","last_unstale":"2024-02-19T15:22:05.865077+0000","last_undegraded":"2024-02-19T15:22:05.865077+0000","last_fullsized":"2024-02-19T15:22:05.865077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:43:34.980001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.875480+0000","last_change":"2024-02-19T15:22:05.869845+0000","last_active":"2024-02-19T15:22:05.875480+0000","last_peered":"2024-02-19T15:22:05.875480+0000","last_clean":"2024-02-19T15:22:05.875480+0000","last_became_active":"2024-02-19T15:22:05.865534+0000","last_became_peered":"2024-02-19T15:22:05.865534+0000","last_unstale":"2024-02-19T15:22:05.875480+0000","last_undegraded":"2024-02-19T15:22:05.875480+0000","last_fullsized":"2024-02-19T15:22:05.875480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:33:20.316638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.857111+0000","last_change":"2024-02-19T15:22:05.852133+0000","last_active":"2024-02-19T15:22:05.857111+0000","last_peered":"2024-02-19T15:22:05.857111+0000","last_clean":"2024-02-19T15:22:05.857111+0000","last_became_active":"2024-02-19T15:22:05.851823+0000","last_became_peered":"2024-02-19T15:22:05.851823+0000","last_unstale":"2024-02-19T15:22:05.857111+0000","last_undegraded":"2024-02-19T15:22:05.857111+0000","last_fullsized":"2024-02-19T15:22:05.857111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:31:18.431106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.871476+0000","last_change":"2024-02-19T15:22:05.866370+0000","last_active":"2024-02-19T15:22:05.871476+0000","last_peered":"2024-02-19T15:22:05.871476+0000","last_clean":"2024-02-19T15:22:05.871476+0000","last_became_active":"2024-02-19T15:22:05.866059+0000","last_became_peered":"2024-02-19T15:22:05.866059+0000","last_unstale":"2024-02-19T15:22:05.871476+0000","last_undegraded":"2024-02-19T15:22:05.871476+0000","last_fullsized":"2024-02-19T15:22:05.871476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:41.123853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854536+0000","last_change":"2024-02-19T15:22:05.858350+0000","last_active":"2024-02-19T15:22:06.854536+0000","last_peered":"2024-02-19T15:22:06.854536+0000","last_clean":"2024-02-19T15:22:06.854536+0000","last_became_active":"2024-02-19T15:22:05.858055+0000","last_became_peered":"2024-02-19T15:22:05.858055+0000","last_unstale":"2024-02-19T15:22:06.854536+0000","last_undegraded":"2024-02-19T15:22:06.854536+0000","last_fullsized":"2024-02-19T15:22:06.854536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:53:17.517796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.870157+0000","last_change":"2024-02-19T15:22:05.867437+0000","last_active":"2024-02-19T15:22:05.870157+0000","last_peered":"2024-02-19T15:22:05.870157+0000","last_clean":"2024-02-19T15:22:05.870157+0000","last_became_active":"2024-02-19T15:22:05.867104+0000","last_became_peered":"2024-02-19T15:22:05.867104+0000","last_unstale":"2024-02-19T15:22:05.870157+0000","last_undegraded":"2024-02-19T15:22:05.870157+0000","last_fullsized":"2024-02-19T15:22:05.870157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:43:50.173776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.876782+0000","last_change":"2024-02-19T15:22:05.873978+0000","last_active":"2024-02-19T15:22:05.876782+0000","last_peered":"2024-02-19T15:22:05.876782+0000","last_clean":"2024-02-19T15:22:05.876782+0000","last_became_active":"2024-02-19T15:22:05.870210+0000","last_became_peered":"2024-02-19T15:22:05.870210+0000","last_unstale":"2024-02-19T15:22:05.876782+0000","last_undegraded":"2024-02-19T15:22:05.876782+0000","last_fullsized":"2024-02-19T15:22:05.876782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:48:05.400374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.871490+0000","last_change":"2024-02-19T15:22:05.869630+0000","last_active":"2024-02-19T15:22:05.871490+0000","last_peered":"2024-02-19T15:22:05.871490+0000","last_clean":"2024-02-19T15:22:05.871490+0000","last_became_active":"2024-02-19T15:22:05.869363+0000","last_became_peered":"2024-02-19T15:22:05.869363+0000","last_unstale":"2024-02-19T15:22:05.871490+0000","last_undegraded":"2024-02-19T15:22:05.871490+0000","last_fullsized":"2024-02-19T15:22:05.871490+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:03:09.391902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.858535+0000","last_change":"2024-02-19T15:22:05.846649+0000","last_active":"2024-02-19T15:22:05.858535+0000","last_peered":"2024-02-19T15:22:05.858535+0000","last_clean":"2024-02-19T15:22:05.858535+0000","last_became_active":"2024-02-19T15:22:05.846335+0000","last_became_peered":"2024-02-19T15:22:05.846335+0000","last_unstale":"2024-02-19T15:22:05.858535+0000","last_undegraded":"2024-02-19T15:22:05.858535+0000","last_fullsized":"2024-02-19T15:22:05.858535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:37:51.996719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.872265+0000","last_change":"2024-02-19T15:22:05.870492+0000","last_active":"2024-02-19T15:22:05.872265+0000","last_peered":"2024-02-19T15:22:05.872265+0000","last_clean":"2024-02-19T15:22:05.872265+0000","last_became_active":"2024-02-19T15:22:05.868145+0000","last_became_peered":"2024-02-19T15:22:05.868145+0000","last_unstale":"2024-02-19T15:22:05.872265+0000","last_undegraded":"2024-02-19T15:22:05.872265+0000","last_fullsized":"2024-02-19T15:22:05.872265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:48:20.147250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854818+0000","last_change":"2024-02-19T15:22:05.869349+0000","last_active":"2024-02-19T15:22:06.854818+0000","last_peered":"2024-02-19T15:22:06.854818+0000","last_clean":"2024-02-19T15:22:06.854818+0000","last_became_active":"2024-02-19T15:22:05.869042+0000","last_became_peered":"2024-02-19T15:22:05.869042+0000","last_unstale":"2024-02-19T15:22:06.854818+0000","last_undegraded":"2024-02-19T15:22:06.854818+0000","last_fullsized":"2024-02-19T15:22:06.854818+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:14:16.136182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.859704+0000","last_change":"2024-02-19T15:22:05.856951+0000","last_active":"2024-02-19T15:22:05.859704+0000","last_peered":"2024-02-19T15:22:05.859704+0000","last_clean":"2024-02-19T15:22:05.859704+0000","last_became_active":"2024-02-19T15:22:05.856670+0000","last_became_peered":"2024-02-19T15:22:05.856670+0000","last_unstale":"2024-02-19T15:22:05.859704+0000","last_undegraded":"2024-02-19T15:22:05.859704+0000","last_fullsized":"2024-02-19T15:22:05.859704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:03:09.391902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.872296+0000","last_change":"2024-02-19T15:22:05.868491+0000","last_active":"2024-02-19T15:22:05.872296+0000","last_peered":"2024-02-19T15:22:05.872296+0000","last_clean":"2024-02-19T15:22:05.872296+0000","last_became_active":"2024-02-19T15:22:05.868184+0000","last_became_peered":"2024-02-19T15:22:05.868184+0000","last_unstale":"2024-02-19T15:22:05.872296+0000","last_undegraded":"2024-02-19T15:22:05.872296+0000","last_fullsized":"2024-02-19T15:22:05.872296+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:32:56.909524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.861731+0000","last_change":"2024-02-19T15:22:05.847847+0000","last_active":"2024-02-19T15:22:05.861731+0000","last_peered":"2024-02-19T15:22:05.861731+0000","last_clean":"2024-02-19T15:22:05.861731+0000","last_became_active":"2024-02-19T15:22:05.847538+0000","last_became_peered":"2024-02-19T15:22:05.847538+0000","last_unstale":"2024-02-19T15:22:05.861731+0000","last_undegraded":"2024-02-19T15:22:05.861731+0000","last_fullsized":"2024-02-19T15:22:05.861731+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:26:20.732058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.860681+0000","last_change":"2024-02-19T15:22:05.847222+0000","last_active":"2024-02-19T15:22:05.860681+0000","last_peered":"2024-02-19T15:22:05.860681+0000","last_clean":"2024-02-19T15:22:05.860681+0000","last_became_active":"2024-02-19T15:22:05.846957+0000","last_became_peered":"2024-02-19T15:22:05.846957+0000","last_unstale":"2024-02-19T15:22:05.860681+0000","last_undegraded":"2024-02-19T15:22:05.860681+0000","last_fullsized":"2024-02-19T15:22:05.860681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:02.917816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.869164+0000","last_change":"2024-02-19T15:22:05.862263+0000","last_active":"2024-02-19T15:22:05.869164+0000","last_peered":"2024-02-19T15:22:05.869164+0000","last_clean":"2024-02-19T15:22:05.869164+0000","last_became_active":"2024-02-19T15:22:05.861932+0000","last_became_peered":"2024-02-19T15:22:05.861932+0000","last_unstale":"2024-02-19T15:22:05.869164+0000","last_undegraded":"2024-02-19T15:22:05.869164+0000","last_fullsized":"2024-02-19T15:22:05.869164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:22.856662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854810+0000","last_change":"2024-02-19T15:22:05.852998+0000","last_active":"2024-02-19T15:22:06.854810+0000","last_peered":"2024-02-19T15:22:06.854810+0000","last_clean":"2024-02-19T15:22:06.854810+0000","last_became_active":"2024-02-19T15:22:05.852660+0000","last_became_peered":"2024-02-19T15:22:05.852660+0000","last_unstale":"2024-02-19T15:22:06.854810+0000","last_undegraded":"2024-02-19T15:22:06.854810+0000","last_fullsized":"2024-02-19T15:22:06.854810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:54:25.604351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.875056+0000","last_change":"2024-02-19T15:22:05.862188+0000","last_active":"2024-02-19T15:22:05.875056+0000","last_peered":"2024-02-19T15:22:05.875056+0000","last_clean":"2024-02-19T15:22:05.875056+0000","last_became_active":"2024-02-19T15:22:05.861789+0000","last_became_peered":"2024-02-19T15:22:05.861789+0000","last_unstale":"2024-02-19T15:22:05.875056+0000","last_undegraded":"2024-02-19T15:22:05.875056+0000","last_fullsized":"2024-02-19T15:22:05.875056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:52:38.926167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.876093+0000","last_change":"2024-02-19T15:22:04.848427+0000","last_active":"2024-02-19T15:22:05.876093+0000","last_peered":"2024-02-19T15:22:05.876093+0000","last_clean":"2024-02-19T15:22:05.876093+0000","last_became_active":"2024-02-19T15:22:04.848111+0000","last_became_peered":"2024-02-19T15:22:04.848111+0000","last_unstale":"2024-02-19T15:22:05.876093+0000","last_undegraded":"2024-02-19T15:22:05.876093+0000","last_fullsized":"2024-02-19T15:22:05.876093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:17.404136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.861706+0000","last_change":"2024-02-19T15:22:05.845098+0000","last_active":"2024-02-19T15:22:05.861706+0000","last_peered":"2024-02-19T15:22:05.861706+0000","last_clean":"2024-02-19T15:22:05.861706+0000","last_became_active":"2024-02-19T15:22:05.844805+0000","last_became_peered":"2024-02-19T15:22:05.844805+0000","last_unstale":"2024-02-19T15:22:05.861706+0000","last_undegraded":"2024-02-19T15:22:05.861706+0000","last_fullsized":"2024-02-19T15:22:05.861706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:33:05.203089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854248+0000","last_change":"2024-02-19T15:22:04.842285+0000","last_active":"2024-02-19T15:22:06.854248+0000","last_peered":"2024-02-19T15:22:06.854248+0000","last_clean":"2024-02-19T15:22:06.854248+0000","last_became_active":"2024-02-19T15:22:04.840126+0000","last_became_peered":"2024-02-19T15:22:04.840126+0000","last_unstale":"2024-02-19T15:22:06.854248+0000","last_undegraded":"2024-02-19T15:22:06.854248+0000","last_fullsized":"2024-02-19T15:22:06.854248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:11:04.017098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.837921+0000","last_change":"2024-02-19T15:22:04.839208+0000","last_active":"2024-02-19T15:22:05.837921+0000","last_peered":"2024-02-19T15:22:05.837921+0000","last_clean":"2024-02-19T15:22:05.837921+0000","last_became_active":"2024-02-19T15:22:04.838857+0000","last_became_peered":"2024-02-19T15:22:04.838857+0000","last_unstale":"2024-02-19T15:22:05.837921+0000","last_undegraded":"2024-02-19T15:22:05.837921+0000","last_fullsized":"2024-02-19T15:22:05.837921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:52:37.920564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.867560+0000","last_change":"2024-02-19T15:22:05.859786+0000","last_active":"2024-02-19T15:22:05.867560+0000","last_peered":"2024-02-19T15:22:05.867560+0000","last_clean":"2024-02-19T15:22:05.867560+0000","last_became_active":"2024-02-19T15:22:05.854215+0000","last_became_peered":"2024-02-19T15:22:05.854215+0000","last_unstale":"2024-02-19T15:22:05.867560+0000","last_undegraded":"2024-02-19T15:22:05.867560+0000","last_fullsized":"2024-02-19T15:22:05.867560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:31:18.431106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.838455+0000","last_change":"2024-02-19T15:22:04.843918+0000","last_active":"2024-02-19T15:22:05.838455+0000","last_peered":"2024-02-19T15:22:05.838455+0000","last_clean":"2024-02-19T15:22:05.838455+0000","last_became_active":"2024-02-19T15:22:04.842002+0000","last_became_peered":"2024-02-19T15:22:04.842002+0000","last_unstale":"2024-02-19T15:22:05.838455+0000","last_undegraded":"2024-02-19T15:22:05.838455+0000","last_fullsized":"2024-02-19T15:22:05.838455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:27.325777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.858027+0000","last_change":"2024-02-19T15:22:05.852203+0000","last_active":"2024-02-19T15:22:05.858027+0000","last_peered":"2024-02-19T15:22:05.858027+0000","last_clean":"2024-02-19T15:22:05.858027+0000","last_became_active":"2024-02-19T15:22:05.850442+0000","last_became_peered":"2024-02-19T15:22:05.850442+0000","last_unstale":"2024-02-19T15:22:05.858027+0000","last_undegraded":"2024-02-19T15:22:05.858027+0000","last_fullsized":"2024-02-19T15:22:05.858027+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:05.756626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854530+0000","last_change":"2024-02-19T15:22:04.839817+0000","last_active":"2024-02-19T15:22:06.854530+0000","last_peered":"2024-02-19T15:22:06.854530+0000","last_clean":"2024-02-19T15:22:06.854530+0000","last_became_active":"2024-02-19T15:22:04.839034+0000","last_became_peered":"2024-02-19T15:22:04.839034+0000","last_unstale":"2024-02-19T15:22:06.854530+0000","last_undegraded":"2024-02-19T15:22:06.854530+0000","last_fullsized":"2024-02-19T15:22:06.854530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:16:21.476088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.875539+0000","last_change":"2024-02-19T15:22:05.869563+0000","last_active":"2024-02-19T15:22:05.875539+0000","last_peered":"2024-02-19T15:22:05.875539+0000","last_clean":"2024-02-19T15:22:05.875539+0000","last_became_active":"2024-02-19T15:22:05.869260+0000","last_became_peered":"2024-02-19T15:22:05.869260+0000","last_unstale":"2024-02-19T15:22:05.875539+0000","last_undegraded":"2024-02-19T15:22:05.875539+0000","last_fullsized":"2024-02-19T15:22:05.875539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:33.203920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.840836+0000","last_change":"2024-02-19T15:22:04.843586+0000","last_active":"2024-02-19T15:22:05.840836+0000","last_peered":"2024-02-19T15:22:05.840836+0000","last_clean":"2024-02-19T15:22:05.840836+0000","last_became_active":"2024-02-19T15:22:04.842983+0000","last_became_peered":"2024-02-19T15:22:04.842983+0000","last_unstale":"2024-02-19T15:22:05.840836+0000","last_undegraded":"2024-02-19T15:22:05.840836+0000","last_fullsized":"2024-02-19T15:22:05.840836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:21.851058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.867806+0000","last_change":"2024-02-19T15:22:05.861506+0000","last_active":"2024-02-19T15:22:05.867806+0000","last_peered":"2024-02-19T15:22:05.867806+0000","last_clean":"2024-02-19T15:22:05.867806+0000","last_became_active":"2024-02-19T15:22:05.861172+0000","last_became_peered":"2024-02-19T15:22:05.861172+0000","last_unstale":"2024-02-19T15:22:05.867806+0000","last_undegraded":"2024-02-19T15:22:05.867806+0000","last_fullsized":"2024-02-19T15:22:05.867806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:25:37.290835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.860919+0000","last_change":"2024-02-19T15:22:04.833984+0000","last_active":"2024-02-19T15:22:05.860919+0000","last_peered":"2024-02-19T15:22:05.860919+0000","last_clean":"2024-02-19T15:22:05.860919+0000","last_became_active":"2024-02-19T15:22:04.833418+0000","last_became_peered":"2024-02-19T15:22:04.833418+0000","last_unstale":"2024-02-19T15:22:05.860919+0000","last_undegraded":"2024-02-19T15:22:05.860919+0000","last_fullsized":"2024-02-19T15:22:05.860919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:31:17.425502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.867888+0000","last_change":"2024-02-19T15:22:05.865117+0000","last_active":"2024-02-19T15:22:05.867888+0000","last_peered":"2024-02-19T15:22:05.867888+0000","last_clean":"2024-02-19T15:22:05.867888+0000","last_became_active":"2024-02-19T15:22:05.864502+0000","last_became_peered":"2024-02-19T15:22:05.864502+0000","last_unstale":"2024-02-19T15:22:05.867888+0000","last_undegraded":"2024-02-19T15:22:05.867888+0000","last_fullsized":"2024-02-19T15:22:05.867888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:18.409739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854661+0000","last_change":"2024-02-19T15:22:04.832285+0000","last_active":"2024-02-19T15:22:06.854661+0000","last_peered":"2024-02-19T15:22:06.854661+0000","last_clean":"2024-02-19T15:22:06.854661+0000","last_became_active":"2024-02-19T15:22:04.831992+0000","last_became_peered":"2024-02-19T15:22:04.831992+0000","last_unstale":"2024-02-19T15:22:06.854661+0000","last_undegraded":"2024-02-19T15:22:06.854661+0000","last_fullsized":"2024-02-19T15:22:06.854661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:50:48.637629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.877939+0000","last_change":"2024-02-19T15:22:05.874836+0000","last_active":"2024-02-19T15:22:05.877939+0000","last_peered":"2024-02-19T15:22:05.877939+0000","last_clean":"2024-02-19T15:22:05.877939+0000","last_became_active":"2024-02-19T15:22:05.874522+0000","last_became_peered":"2024-02-19T15:22:05.874522+0000","last_unstale":"2024-02-19T15:22:05.877939+0000","last_undegraded":"2024-02-19T15:22:05.877939+0000","last_fullsized":"2024-02-19T15:22:05.877939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:43.283192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.869059+0000","last_change":"2024-02-19T15:22:04.837502+0000","last_active":"2024-02-19T15:22:05.869059+0000","last_peered":"2024-02-19T15:22:05.869059+0000","last_clean":"2024-02-19T15:22:05.869059+0000","last_became_active":"2024-02-19T15:22:04.836913+0000","last_became_peered":"2024-02-19T15:22:04.836913+0000","last_unstale":"2024-02-19T15:22:05.869059+0000","last_undegraded":"2024-02-19T15:22:05.869059+0000","last_fullsized":"2024-02-19T15:22:05.869059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:20:00.631607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.867649+0000","last_change":"2024-02-19T15:22:05.864869+0000","last_active":"2024-02-19T15:22:05.867649+0000","last_peered":"2024-02-19T15:22:05.867649+0000","last_clean":"2024-02-19T15:22:05.867649+0000","last_became_active":"2024-02-19T15:22:05.864424+0000","last_became_peered":"2024-02-19T15:22:05.864424+0000","last_unstale":"2024-02-19T15:22:05.867649+0000","last_undegraded":"2024-02-19T15:22:05.867649+0000","last_fullsized":"2024-02-19T15:22:05.867649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:19:29.439405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.838773+0000","last_change":"2024-02-19T15:22:04.847248+0000","last_active":"2024-02-19T15:22:05.838773+0000","last_peered":"2024-02-19T15:22:05.838773+0000","last_clean":"2024-02-19T15:22:05.838773+0000","last_became_active":"2024-02-19T15:22:04.846883+0000","last_became_peered":"2024-02-19T15:22:04.846883+0000","last_unstale":"2024-02-19T15:22:05.838773+0000","last_undegraded":"2024-02-19T15:22:05.838773+0000","last_fullsized":"2024-02-19T15:22:05.838773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:40.779845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.876987+0000","last_change":"2024-02-19T15:22:05.873228+0000","last_active":"2024-02-19T15:22:05.876987+0000","last_peered":"2024-02-19T15:22:05.876987+0000","last_clean":"2024-02-19T15:22:05.876987+0000","last_became_active":"2024-02-19T15:22:05.872458+0000","last_became_peered":"2024-02-19T15:22:05.872458+0000","last_unstale":"2024-02-19T15:22:05.876987+0000","last_undegraded":"2024-02-19T15:22:05.876987+0000","last_fullsized":"2024-02-19T15:22:05.876987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:48:05.400374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854298+0000","last_change":"2024-02-19T15:22:04.843476+0000","last_active":"2024-02-19T15:22:06.854298+0000","last_peered":"2024-02-19T15:22:06.854298+0000","last_clean":"2024-02-19T15:22:06.854298+0000","last_became_active":"2024-02-19T15:22:04.843159+0000","last_became_peered":"2024-02-19T15:22:04.843159+0000","last_unstale":"2024-02-19T15:22:06.854298+0000","last_undegraded":"2024-02-19T15:22:06.854298+0000","last_fullsized":"2024-02-19T15:22:06.854298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:21:58.456669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854827+0000","last_change":"2024-02-19T15:22:05.852049+0000","last_active":"2024-02-19T15:22:06.854827+0000","last_peered":"2024-02-19T15:22:06.854827+0000","last_clean":"2024-02-19T15:22:06.854827+0000","last_became_active":"2024-02-19T15:22:05.851786+0000","last_became_peered":"2024-02-19T15:22:05.851786+0000","last_unstale":"2024-02-19T15:22:06.854827+0000","last_undegraded":"2024-02-19T15:22:06.854827+0000","last_fullsized":"2024-02-19T15:22:06.854827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:31:59.866248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.873203+0000","last_change":"2024-02-19T15:22:04.844209+0000","last_active":"2024-02-19T15:22:05.873203+0000","last_peered":"2024-02-19T15:22:05.873203+0000","last_clean":"2024-02-19T15:22:05.873203+0000","last_became_active":"2024-02-19T15:22:04.842885+0000","last_became_peered":"2024-02-19T15:22:04.842885+0000","last_unstale":"2024-02-19T15:22:05.873203+0000","last_undegraded":"2024-02-19T15:22:05.873203+0000","last_fullsized":"2024-02-19T15:22:05.873203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:27.325777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.878606+0000","last_change":"2024-02-19T15:22:05.877219+0000","last_active":"2024-02-19T15:22:05.878606+0000","last_peered":"2024-02-19T15:22:05.878606+0000","last_clean":"2024-02-19T15:22:05.878606+0000","last_became_active":"2024-02-19T15:22:05.876953+0000","last_became_peered":"2024-02-19T15:22:05.876953+0000","last_unstale":"2024-02-19T15:22:05.878606+0000","last_undegraded":"2024-02-19T15:22:05.878606+0000","last_fullsized":"2024-02-19T15:22:05.878606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:03:40.323060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.868901+0000","last_change":"2024-02-19T15:22:04.837084+0000","last_active":"2024-02-19T15:22:05.868901+0000","last_peered":"2024-02-19T15:22:05.868901+0000","last_clean":"2024-02-19T15:22:05.868901+0000","last_became_active":"2024-02-19T15:22:04.836783+0000","last_became_peered":"2024-02-19T15:22:04.836783+0000","last_unstale":"2024-02-19T15:22:05.868901+0000","last_undegraded":"2024-02-19T15:22:05.868901+0000","last_fullsized":"2024-02-19T15:22:05.868901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:34:14.847696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.878539+0000","last_change":"2024-02-19T15:22:05.876111+0000","last_active":"2024-02-19T15:22:05.878539+0000","last_peered":"2024-02-19T15:22:05.878539+0000","last_clean":"2024-02-19T15:22:05.878539+0000","last_became_active":"2024-02-19T15:22:05.875843+0000","last_became_peered":"2024-02-19T15:22:05.875843+0000","last_unstale":"2024-02-19T15:22:05.878539+0000","last_undegraded":"2024-02-19T15:22:05.878539+0000","last_fullsized":"2024-02-19T15:22:05.878539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:14:40.204009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.871629+0000","last_change":"2024-02-19T15:22:04.839982+0000","last_active":"2024-02-19T15:22:05.871629+0000","last_peered":"2024-02-19T15:22:05.871629+0000","last_clean":"2024-02-19T15:22:05.871629+0000","last_became_active":"2024-02-19T15:22:04.839617+0000","last_became_peered":"2024-02-19T15:22:04.839617+0000","last_unstale":"2024-02-19T15:22:05.871629+0000","last_undegraded":"2024-02-19T15:22:05.871629+0000","last_fullsized":"2024-02-19T15:22:05.871629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:21.137278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.875958+0000","last_change":"2024-02-19T15:22:05.870359+0000","last_active":"2024-02-19T15:22:05.875958+0000","last_peered":"2024-02-19T15:22:05.875958+0000","last_clean":"2024-02-19T15:22:05.875958+0000","last_became_active":"2024-02-19T15:22:05.870083+0000","last_became_peered":"2024-02-19T15:22:05.870083+0000","last_unstale":"2024-02-19T15:22:05.875958+0000","last_undegraded":"2024-02-19T15:22:05.875958+0000","last_fullsized":"2024-02-19T15:22:05.875958+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:33:20.316638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.870508+0000","last_change":"2024-02-19T15:22:04.828045+0000","last_active":"2024-02-19T15:22:05.870508+0000","last_peered":"2024-02-19T15:22:05.870508+0000","last_clean":"2024-02-19T15:22:05.870508+0000","last_became_active":"2024-02-19T15:22:04.827744+0000","last_became_peered":"2024-02-19T15:22:04.827744+0000","last_unstale":"2024-02-19T15:22:05.870508+0000","last_undegraded":"2024-02-19T15:22:05.870508+0000","last_fullsized":"2024-02-19T15:22:05.870508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:21.851058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.860629+0000","last_change":"2024-02-19T15:22:05.849002+0000","last_active":"2024-02-19T15:22:05.860629+0000","last_peered":"2024-02-19T15:22:05.860629+0000","last_clean":"2024-02-19T15:22:05.860629+0000","last_became_active":"2024-02-19T15:22:05.848692+0000","last_became_peered":"2024-02-19T15:22:05.848692+0000","last_unstale":"2024-02-19T15:22:05.860629+0000","last_undegraded":"2024-02-19T15:22:05.860629+0000","last_fullsized":"2024-02-19T15:22:05.860629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:05.756626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.845964+0000","last_change":"2024-02-19T15:22:04.851700+0000","last_active":"2024-02-19T15:22:05.845964+0000","last_peered":"2024-02-19T15:22:05.845964+0000","last_clean":"2024-02-19T15:22:05.845964+0000","last_became_active":"2024-02-19T15:22:04.851270+0000","last_became_peered":"2024-02-19T15:22:04.851270+0000","last_unstale":"2024-02-19T15:22:05.845964+0000","last_undegraded":"2024-02-19T15:22:05.845964+0000","last_fullsized":"2024-02-19T15:22:05.845964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:40.779845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.867560+0000","last_change":"2024-02-19T15:22:05.862153+0000","last_active":"2024-02-19T15:22:05.867560+0000","last_peered":"2024-02-19T15:22:05.867560+0000","last_clean":"2024-02-19T15:22:05.867560+0000","last_became_active":"2024-02-19T15:22:05.861836+0000","last_became_peered":"2024-02-19T15:22:05.861836+0000","last_unstale":"2024-02-19T15:22:05.867560+0000","last_undegraded":"2024-02-19T15:22:05.867560+0000","last_fullsized":"2024-02-19T15:22:05.867560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:48:20.147250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":168,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.850934+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:22:05.850934+0000","last_peered":"2024-02-19T15:22:05.850934+0000","last_clean":"2024-02-19T15:22:05.850934+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:22:05.850934+0000","last_undegraded":"2024-02-19T15:22:05.850934+0000","last_fullsized":"2024-02-19T15:22:05.850934+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.840015+0000","last_change":"2024-02-19T15:22:04.846635+0000","last_active":"2024-02-19T15:22:05.840015+0000","last_peered":"2024-02-19T15:22:05.840015+0000","last_clean":"2024-02-19T15:22:05.840015+0000","last_became_active":"2024-02-19T15:22:04.845191+0000","last_became_peered":"2024-02-19T15:22:04.845191+0000","last_unstale":"2024-02-19T15:22:05.840015+0000","last_undegraded":"2024-02-19T15:22:05.840015+0000","last_fullsized":"2024-02-19T15:22:05.840015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:31:17.425502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.871032+0000","last_change":"2024-02-19T15:22:05.863647+0000","last_active":"2024-02-19T15:22:05.871032+0000","last_peered":"2024-02-19T15:22:05.871032+0000","last_clean":"2024-02-19T15:22:05.871032+0000","last_became_active":"2024-02-19T15:22:05.863370+0000","last_became_peered":"2024-02-19T15:22:05.863370+0000","last_unstale":"2024-02-19T15:22:05.871032+0000","last_undegraded":"2024-02-19T15:22:05.871032+0000","last_fullsized":"2024-02-19T15:22:05.871032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:02.917816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.850879+0000","last_change":"2024-02-19T15:22:04.849295+0000","last_active":"2024-02-19T15:22:05.850879+0000","last_peered":"2024-02-19T15:22:05.850879+0000","last_clean":"2024-02-19T15:22:05.850879+0000","last_became_active":"2024-02-19T15:22:04.849004+0000","last_became_peered":"2024-02-19T15:22:04.849004+0000","last_unstale":"2024-02-19T15:22:05.850879+0000","last_undegraded":"2024-02-19T15:22:05.850879+0000","last_fullsized":"2024-02-19T15:22:05.850879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:37.601588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.867078+0000","last_change":"2024-02-19T15:22:05.861044+0000","last_active":"2024-02-19T15:22:05.867078+0000","last_peered":"2024-02-19T15:22:05.867078+0000","last_clean":"2024-02-19T15:22:05.867078+0000","last_became_active":"2024-02-19T15:22:05.860784+0000","last_became_peered":"2024-02-19T15:22:05.860784+0000","last_unstale":"2024-02-19T15:22:05.867078+0000","last_undegraded":"2024-02-19T15:22:05.867078+0000","last_fullsized":"2024-02-19T15:22:05.867078+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:21:59.462272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.843126+0000","last_change":"2024-02-19T15:22:04.842486+0000","last_active":"2024-02-19T15:22:05.843126+0000","last_peered":"2024-02-19T15:22:05.843126+0000","last_clean":"2024-02-19T15:22:05.843126+0000","last_became_active":"2024-02-19T15:22:04.840062+0000","last_became_peered":"2024-02-19T15:22:04.840062+0000","last_unstale":"2024-02-19T15:22:05.843126+0000","last_undegraded":"2024-02-19T15:22:05.843126+0000","last_fullsized":"2024-02-19T15:22:05.843126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:17.404136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.858264+0000","last_change":"2024-02-19T15:22:05.853794+0000","last_active":"2024-02-19T15:22:05.858264+0000","last_peered":"2024-02-19T15:22:05.858264+0000","last_clean":"2024-02-19T15:22:05.858264+0000","last_became_active":"2024-02-19T15:22:05.851972+0000","last_became_peered":"2024-02-19T15:22:05.851972+0000","last_unstale":"2024-02-19T15:22:05.858264+0000","last_undegraded":"2024-02-19T15:22:05.858264+0000","last_fullsized":"2024-02-19T15:22:05.858264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:32:56.909524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.842890+0000","last_change":"2024-02-19T15:22:04.851077+0000","last_active":"2024-02-19T15:22:05.842890+0000","last_peered":"2024-02-19T15:22:05.842890+0000","last_clean":"2024-02-19T15:22:05.842890+0000","last_became_active":"2024-02-19T15:22:04.850734+0000","last_became_peered":"2024-02-19T15:22:04.850734+0000","last_unstale":"2024-02-19T15:22:05.842890+0000","last_undegraded":"2024-02-19T15:22:05.842890+0000","last_fullsized":"2024-02-19T15:22:05.842890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:37.601588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.864456+0000","last_change":"2024-02-19T15:22:05.859004+0000","last_active":"2024-02-19T15:22:05.864456+0000","last_peered":"2024-02-19T15:22:05.864456+0000","last_clean":"2024-02-19T15:22:05.864456+0000","last_became_active":"2024-02-19T15:22:05.858524+0000","last_became_peered":"2024-02-19T15:22:05.858524+0000","last_unstale":"2024-02-19T15:22:05.864456+0000","last_undegraded":"2024-02-19T15:22:05.864456+0000","last_fullsized":"2024-02-19T15:22:05.864456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:34:37.414912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.849782+0000","last_change":"2024-02-19T15:22:04.832556+0000","last_active":"2024-02-19T15:22:05.849782+0000","last_peered":"2024-02-19T15:22:05.849782+0000","last_clean":"2024-02-19T15:22:05.849782+0000","last_became_active":"2024-02-19T15:22:04.832261+0000","last_became_peered":"2024-02-19T15:22:04.832261+0000","last_unstale":"2024-02-19T15:22:05.849782+0000","last_undegraded":"2024-02-19T15:22:05.849782+0000","last_fullsized":"2024-02-19T15:22:05.849782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:37.601588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.859005+0000","last_change":"2024-02-19T15:22:05.850119+0000","last_active":"2024-02-19T15:22:05.859005+0000","last_peered":"2024-02-19T15:22:05.859005+0000","last_clean":"2024-02-19T15:22:05.859005+0000","last_became_active":"2024-02-19T15:22:05.849831+0000","last_became_peered":"2024-02-19T15:22:05.849831+0000","last_unstale":"2024-02-19T15:22:05.859005+0000","last_undegraded":"2024-02-19T15:22:05.859005+0000","last_fullsized":"2024-02-19T15:22:05.859005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:32:56.909524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.854977+0000","last_change":"2024-02-19T15:22:04.850106+0000","last_active":"2024-02-19T15:22:05.854977+0000","last_peered":"2024-02-19T15:22:05.854977+0000","last_clean":"2024-02-19T15:22:05.854977+0000","last_became_active":"2024-02-19T15:22:04.849467+0000","last_became_peered":"2024-02-19T15:22:04.849467+0000","last_unstale":"2024-02-19T15:22:05.854977+0000","last_undegraded":"2024-02-19T15:22:05.854977+0000","last_fullsized":"2024-02-19T15:22:05.854977+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:32:55.903921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.860264+0000","last_change":"2024-02-19T15:22:05.857834+0000","last_active":"2024-02-19T15:22:05.860264+0000","last_peered":"2024-02-19T15:22:05.860264+0000","last_clean":"2024-02-19T15:22:05.860264+0000","last_became_active":"2024-02-19T15:22:05.857488+0000","last_became_peered":"2024-02-19T15:22:05.857488+0000","last_unstale":"2024-02-19T15:22:05.860264+0000","last_undegraded":"2024-02-19T15:22:05.860264+0000","last_fullsized":"2024-02-19T15:22:05.860264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:48:20.147250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.849966+0000","last_change":"2024-02-19T15:22:04.830568+0000","last_active":"2024-02-19T15:22:05.849966+0000","last_peered":"2024-02-19T15:22:05.849966+0000","last_clean":"2024-02-19T15:22:05.849966+0000","last_became_active":"2024-02-19T15:22:04.829131+0000","last_became_peered":"2024-02-19T15:22:04.829131+0000","last_unstale":"2024-02-19T15:22:05.849966+0000","last_undegraded":"2024-02-19T15:22:05.849966+0000","last_fullsized":"2024-02-19T15:22:05.849966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:37:50.991116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.871054+0000","last_change":"2024-02-19T15:22:05.866131+0000","last_active":"2024-02-19T15:22:05.871054+0000","last_peered":"2024-02-19T15:22:05.871054+0000","last_clean":"2024-02-19T15:22:05.871054+0000","last_became_active":"2024-02-19T15:22:05.865820+0000","last_became_peered":"2024-02-19T15:22:05.865820+0000","last_unstale":"2024-02-19T15:22:05.871054+0000","last_undegraded":"2024-02-19T15:22:05.871054+0000","last_fullsized":"2024-02-19T15:22:05.871054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:05.756626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854644+0000","last_change":"2024-02-19T15:22:04.840658+0000","last_active":"2024-02-19T15:22:06.854644+0000","last_peered":"2024-02-19T15:22:06.854644+0000","last_clean":"2024-02-19T15:22:06.854644+0000","last_became_active":"2024-02-19T15:22:04.840167+0000","last_became_peered":"2024-02-19T15:22:04.840167+0000","last_unstale":"2024-02-19T15:22:06.854644+0000","last_undegraded":"2024-02-19T15:22:06.854644+0000","last_fullsized":"2024-02-19T15:22:06.854644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:26:19.726455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.859322+0000","last_change":"2024-02-19T15:22:05.854827+0000","last_active":"2024-02-19T15:22:05.859322+0000","last_peered":"2024-02-19T15:22:05.859322+0000","last_clean":"2024-02-19T15:22:05.859322+0000","last_became_active":"2024-02-19T15:22:05.854579+0000","last_became_peered":"2024-02-19T15:22:05.854579+0000","last_unstale":"2024-02-19T15:22:05.859322+0000","last_undegraded":"2024-02-19T15:22:05.859322+0000","last_fullsized":"2024-02-19T15:22:05.859322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:11:05.022701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.849999+0000","last_change":"2024-02-19T15:22:04.848481+0000","last_active":"2024-02-19T15:22:05.849999+0000","last_peered":"2024-02-19T15:22:05.849999+0000","last_clean":"2024-02-19T15:22:05.849999+0000","last_became_active":"2024-02-19T15:22:04.848224+0000","last_became_peered":"2024-02-19T15:22:04.848224+0000","last_unstale":"2024-02-19T15:22:05.849999+0000","last_undegraded":"2024-02-19T15:22:05.849999+0000","last_fullsized":"2024-02-19T15:22:05.849999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:21.851058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.874326+0000","last_change":"2024-02-19T15:22:05.868543+0000","last_active":"2024-02-19T15:22:05.874326+0000","last_peered":"2024-02-19T15:22:05.874326+0000","last_clean":"2024-02-19T15:22:05.874326+0000","last_became_active":"2024-02-19T15:22:05.868307+0000","last_became_peered":"2024-02-19T15:22:05.868307+0000","last_unstale":"2024-02-19T15:22:05.874326+0000","last_undegraded":"2024-02-19T15:22:05.874326+0000","last_fullsized":"2024-02-19T15:22:05.874326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:22.142882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.844375+0000","last_change":"2024-02-19T15:22:04.841547+0000","last_active":"2024-02-19T15:22:05.844375+0000","last_peered":"2024-02-19T15:22:05.844375+0000","last_clean":"2024-02-19T15:22:05.844375+0000","last_became_active":"2024-02-19T15:22:04.841280+0000","last_became_peered":"2024-02-19T15:22:04.841280+0000","last_unstale":"2024-02-19T15:22:05.844375+0000","last_undegraded":"2024-02-19T15:22:05.844375+0000","last_fullsized":"2024-02-19T15:22:05.844375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:40.779845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.870739+0000","last_change":"2024-02-19T15:22:05.863084+0000","last_active":"2024-02-19T15:22:05.870739+0000","last_peered":"2024-02-19T15:22:05.870739+0000","last_clean":"2024-02-19T15:22:05.870739+0000","last_became_active":"2024-02-19T15:22:05.862820+0000","last_became_peered":"2024-02-19T15:22:05.862820+0000","last_unstale":"2024-02-19T15:22:05.870739+0000","last_undegraded":"2024-02-19T15:22:05.870739+0000","last_fullsized":"2024-02-19T15:22:05.870739+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:37:51.996719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.842582+0000","last_change":"2024-02-19T15:22:04.840145+0000","last_active":"2024-02-19T15:22:05.842582+0000","last_peered":"2024-02-19T15:22:05.842582+0000","last_clean":"2024-02-19T15:22:05.842582+0000","last_became_active":"2024-02-19T15:22:04.839832+0000","last_became_peered":"2024-02-19T15:22:04.839832+0000","last_unstale":"2024-02-19T15:22:05.842582+0000","last_undegraded":"2024-02-19T15:22:05.842582+0000","last_fullsized":"2024-02-19T15:22:05.842582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:27.325777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.870898+0000","last_change":"2024-02-19T15:22:05.862437+0000","last_active":"2024-02-19T15:22:05.870898+0000","last_peered":"2024-02-19T15:22:05.870898+0000","last_clean":"2024-02-19T15:22:05.870898+0000","last_became_active":"2024-02-19T15:22:05.855551+0000","last_became_peered":"2024-02-19T15:22:05.855551+0000","last_unstale":"2024-02-19T15:22:05.870898+0000","last_undegraded":"2024-02-19T15:22:05.870898+0000","last_fullsized":"2024-02-19T15:22:05.870898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:38.607191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.872781+0000","last_change":"2024-02-19T15:22:04.849761+0000","last_active":"2024-02-19T15:22:05.872781+0000","last_peered":"2024-02-19T15:22:05.872781+0000","last_clean":"2024-02-19T15:22:05.872781+0000","last_became_active":"2024-02-19T15:22:04.849413+0000","last_became_peered":"2024-02-19T15:22:04.849413+0000","last_unstale":"2024-02-19T15:22:05.872781+0000","last_undegraded":"2024-02-19T15:22:05.872781+0000","last_fullsized":"2024-02-19T15:22:05.872781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:37:50.991116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854363+0000","last_change":"2024-02-19T15:22:05.871270+0000","last_active":"2024-02-19T15:22:06.854363+0000","last_peered":"2024-02-19T15:22:06.854363+0000","last_clean":"2024-02-19T15:22:06.854363+0000","last_became_active":"2024-02-19T15:22:05.870986+0000","last_became_peered":"2024-02-19T15:22:05.870986+0000","last_unstale":"2024-02-19T15:22:06.854363+0000","last_undegraded":"2024-02-19T15:22:06.854363+0000","last_fullsized":"2024-02-19T15:22:06.854363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:22:17.882758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.872436+0000","last_change":"2024-02-19T15:22:05.864283+0000","last_active":"2024-02-19T15:22:05.872436+0000","last_peered":"2024-02-19T15:22:05.872436+0000","last_clean":"2024-02-19T15:22:05.872436+0000","last_became_active":"2024-02-19T15:22:05.863996+0000","last_became_peered":"2024-02-19T15:22:05.863996+0000","last_unstale":"2024-02-19T15:22:05.872436+0000","last_undegraded":"2024-02-19T15:22:05.872436+0000","last_fullsized":"2024-02-19T15:22:05.872436+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:41.123853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.875933+0000","last_change":"2024-02-19T15:22:04.855753+0000","last_active":"2024-02-19T15:22:05.875933+0000","last_peered":"2024-02-19T15:22:05.875933+0000","last_clean":"2024-02-19T15:22:05.875933+0000","last_became_active":"2024-02-19T15:22:04.855303+0000","last_became_peered":"2024-02-19T15:22:04.855303+0000","last_unstale":"2024-02-19T15:22:05.875933+0000","last_undegraded":"2024-02-19T15:22:05.875933+0000","last_fullsized":"2024-02-19T15:22:05.875933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:04.751023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.875438+0000","last_change":"2024-02-19T15:22:05.870398+0000","last_active":"2024-02-19T15:22:05.875438+0000","last_peered":"2024-02-19T15:22:05.875438+0000","last_clean":"2024-02-19T15:22:05.875438+0000","last_became_active":"2024-02-19T15:22:05.869963+0000","last_became_peered":"2024-02-19T15:22:05.869963+0000","last_unstale":"2024-02-19T15:22:05.875438+0000","last_undegraded":"2024-02-19T15:22:05.875438+0000","last_fullsized":"2024-02-19T15:22:05.875438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:21:49.261015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.868588+0000","last_change":"2024-02-19T15:22:04.849798+0000","last_active":"2024-02-19T15:22:05.868588+0000","last_peered":"2024-02-19T15:22:05.868588+0000","last_clean":"2024-02-19T15:22:05.868588+0000","last_became_active":"2024-02-19T15:22:04.849115+0000","last_became_peered":"2024-02-19T15:22:04.849115+0000","last_unstale":"2024-02-19T15:22:05.868588+0000","last_undegraded":"2024-02-19T15:22:05.868588+0000","last_fullsized":"2024-02-19T15:22:05.868588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:01.912213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.867825+0000","last_change":"2024-02-19T15:22:05.861837+0000","last_active":"2024-02-19T15:22:05.867825+0000","last_peered":"2024-02-19T15:22:05.867825+0000","last_clean":"2024-02-19T15:22:05.867825+0000","last_became_active":"2024-02-19T15:22:05.861570+0000","last_became_peered":"2024-02-19T15:22:05.861570+0000","last_unstale":"2024-02-19T15:22:05.867825+0000","last_undegraded":"2024-02-19T15:22:05.867825+0000","last_fullsized":"2024-02-19T15:22:05.867825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:42:49.483797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.870819+0000","last_change":"2024-02-19T15:22:04.841218+0000","last_active":"2024-02-19T15:22:05.870819+0000","last_peered":"2024-02-19T15:22:05.870819+0000","last_clean":"2024-02-19T15:22:05.870819+0000","last_became_active":"2024-02-19T15:22:04.840740+0000","last_became_peered":"2024-02-19T15:22:04.840740+0000","last_unstale":"2024-02-19T15:22:05.870819+0000","last_undegraded":"2024-02-19T15:22:05.870819+0000","last_fullsized":"2024-02-19T15:22:05.870819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:21.137278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.850921+0000","last_change":"2024-02-19T15:22:05.847219+0000","last_active":"2024-02-19T15:22:05.850921+0000","last_peered":"2024-02-19T15:22:05.850921+0000","last_clean":"2024-02-19T15:22:05.850921+0000","last_became_active":"2024-02-19T15:22:05.846948+0000","last_became_peered":"2024-02-19T15:22:05.846948+0000","last_unstale":"2024-02-19T15:22:05.850921+0000","last_undegraded":"2024-02-19T15:22:05.850921+0000","last_fullsized":"2024-02-19T15:22:05.850921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:28.331381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-02-19T15:22:05.870264+0000","last_change":"2024-02-19T15:22:04.843469+0000","last_active":"2024-02-19T15:22:05.870264+0000","last_peered":"2024-02-19T15:22:05.870264+0000","last_clean":"2024-02-19T15:22:05.870264+0000","last_became_active":"2024-02-19T15:22:04.842098+0000","last_became_peered":"2024-02-19T15:22:04.842098+0000","last_unstale":"2024-02-19T15:22:05.870264+0000","last_undegraded":"2024-02-19T15:22:05.870264+0000","last_fullsized":"2024-02-19T15:22:05.870264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:32:33.319050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509445,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]}]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":798720,"data_stored":658533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]}]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":60808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":67709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]}]},{"osd":3,"up_from":14,"seq":60129542148,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":212992,"data_stored":69195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]}]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31364,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340476,"statfs":{"total":96636764160,"available":96604647424,"internally_reserved":0,"allocated":188416,"data_stored":60808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1040000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":208896,"data_stored":67743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.997}]}]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31944,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339896,"statfs":{"total":96636764160,"available":96604053504,"internally_reserved":0,"allocated":782336,"data_stored":651176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:22:10.354 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T15:22:10.355 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-19T15:22:10.355 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-02-19T15:22:10.355 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-02-19T15:22:10.355 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-02-19T15:22:10.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.530+0000 7fe1f2800640 1 Processor -- start 2024-02-19T15:22:10.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.530+0000 7fe1f2800640 1 -- start start 2024-02-19T15:22:10.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.530+0000 7fe1f2800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe1ec15b1a0 0x7fe1ec15b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:10.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.530+0000 7fe1f2800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1ec0718c0 con 0x7fe1ec06c9a0 2024-02-19T15:22:10.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.530+0000 7fe1f2800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1ec071ba0 con 0x7fe1ec071350 2024-02-19T15:22:10.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.530+0000 7fe1f2800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1ec071e80 con 0x7fe1ec15b1a0 2024-02-19T15:22:10.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.530+0000 7fe1eb400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe1ec15b1a0 0x7fe1ec15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:10.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.530+0000 7fe1e3e00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe1ec071350 0x7fe1ec15c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:46198/0 (socket says 172.21.15.47:46198) 2024-02-19T15:22:10.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.530+0000 7fe1ebe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fe1ec06c9a0 0x7fe1ec15cde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:44000/0 (socket says 172.21.15.47:44000) 2024-02-19T15:22:10.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.530+0000 7fe1e3e00640 1 -- 172.21.15.47:0/817109884 learned_addr learned my addr 172.21.15.47:0/817109884 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:10.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.534+0000 7fe1eb400640 1 -- 172.21.15.47:0/817109884 >> v1:172.21.15.116:6789/0 conn(0x7fe1ec071350 legacy=0x7fe1ec15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:10.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.534+0000 7fe1eb400640 1 -- 172.21.15.47:0/817109884 >> v1:172.21.15.47:6789/0 conn(0x7fe1ec06c9a0 legacy=0x7fe1ec15cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:10.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.534+0000 7fe1eb400640 1 -- 172.21.15.47:0/817109884 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ec072160 con 0x7fe1ec15b1a0 2024-02-19T15:22:10.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.534+0000 7fe1eb400640 1 --2- 172.21.15.47:0/817109884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1ec15b1a0 0x7fe1ec15b580 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc0097f0 tx=0x7fe1dc01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa1a7300d48293f8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:10.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.534+0000 7fe1eaa00640 1 -- 172.21.15.47:0/817109884 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc0a7070 con 0x7fe1ec15b1a0 2024-02-19T15:22:10.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.534+0000 7fe1eaa00640 1 -- 172.21.15.47:0/817109884 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1dc0ab430 con 0x7fe1ec15b1a0 2024-02-19T15:22:10.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.534+0000 7fe1eaa00640 1 -- 172.21.15.47:0/817109884 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc0ab7c0 con 0x7fe1ec15b1a0 2024-02-19T15:22:10.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.534+0000 7fe1f2800640 1 -- 172.21.15.47:0/817109884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1ec15b1a0 msgr2=0x7fe1ec15b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:10.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.534+0000 7fe1f2800640 1 --2- 172.21.15.47:0/817109884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1ec15b1a0 0x7fe1ec15b580 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc0097f0 tx=0x7fe1dc01c4a0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.534+0000 7fe1f2800640 1 -- 172.21.15.47:0/817109884 shutdown_connections 2024-02-19T15:22:10.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.534+0000 7fe1f2800640 1 --2- 172.21.15.47:0/817109884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1ec15b1a0 0x7fe1ec15b580 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.534+0000 7fe1f2800640 1 -- 172.21.15.47:0/817109884 >> 172.21.15.47:0/817109884 conn(0x7fe1ec0b16a0 msgr2=0x7fe1ec0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:10.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1f2800640 1 -- 172.21.15.47:0/817109884 shutdown_connections 2024-02-19T15:22:10.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1f2800640 1 -- 172.21.15.47:0/817109884 wait complete. 2024-02-19T15:22:10.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1f2800640 1 Processor -- start 2024-02-19T15:22:10.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1f2800640 1 -- start start 2024-02-19T15:22:10.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1f2800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1ec06c9a0 0x7fe1ec147440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:10.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1f2800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1ec071350 0x7fe1ec156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:10.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1f2800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1ec15b1a0 0x7fe1ec0b37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:10.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1f2800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec157d40 con 0x7fe1ec06c9a0 2024-02-19T15:22:10.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1f2800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec147980 con 0x7fe1ec071350 2024-02-19T15:22:10.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1f2800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec1570c0 con 0x7fe1ec15b1a0 2024-02-19T15:22:10.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1eb400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1ec06c9a0 0x7fe1ec147440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:10.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1eb400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1ec06c9a0 0x7fe1ec147440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52892/0 (socket says 172.21.15.47:52892) 2024-02-19T15:22:10.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1eb400640 1 -- 172.21.15.47:0/1363511016 learned_addr learned my addr 172.21.15.47:0/1363511016 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:10.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1e3e00640 1 --2- 172.21.15.47:0/1363511016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1ec071350 0x7fe1ec156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:10.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1ebe00640 1 --2- 172.21.15.47:0/1363511016 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1ec15b1a0 0x7fe1ec0b37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:10.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1eb400640 1 -- 172.21.15.47:0/1363511016 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1ec15b1a0 msgr2=0x7fe1ec0b37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:10.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1eb400640 1 --2- 172.21.15.47:0/1363511016 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1ec15b1a0 0x7fe1ec0b37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1eb400640 1 -- 172.21.15.47:0/1363511016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1ec071350 msgr2=0x7fe1ec156600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:10.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1eb400640 1 --2- 172.21.15.47:0/1363511016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1ec071350 0x7fe1ec156600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1eb400640 1 -- 172.21.15.47:0/1363511016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc009420 con 0x7fe1ec06c9a0 2024-02-19T15:22:10.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1eb400640 1 --2- 172.21.15.47:0/1363511016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1ec06c9a0 0x7fe1ec147440 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc0ad040 tx=0x7fe1dc01ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:10.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1e3400640 1 -- 172.21.15.47:0/1363511016 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc0b5810 con 0x7fe1ec06c9a0 2024-02-19T15:22:10.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1e3400640 1 -- 172.21.15.47:0/1363511016 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1dc0b5970 con 0x7fe1ec06c9a0 2024-02-19T15:22:10.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1f2800640 1 -- 172.21.15.47:0/1363511016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1ec1573a0 con 0x7fe1ec06c9a0 2024-02-19T15:22:10.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.538+0000 7fe1e3400640 1 -- 172.21.15.47:0/1363511016 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc0b5d20 con 0x7fe1ec06c9a0 2024-02-19T15:22:10.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.542+0000 7fe1f2800640 1 -- 172.21.15.47:0/1363511016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1ec157960 con 0x7fe1ec06c9a0 2024-02-19T15:22:10.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.542+0000 7fe1f2800640 1 -- 172.21.15.47:0/1363511016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1ec157fb0 con 0x7fe1ec06c9a0 2024-02-19T15:22:10.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.542+0000 7fe1e3400640 1 -- 172.21.15.47:0/1363511016 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe1dc0bf410 con 0x7fe1ec06c9a0 2024-02-19T15:22:10.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.542+0000 7fe1e3400640 1 --2- 172.21.15.47:0/1363511016 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe1c8066e40 0x7fe1c8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:10.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.542+0000 7fe1e3400640 1 -- 172.21.15.47:0/1363511016 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1dc11f5b0 con 0x7fe1ec06c9a0 2024-02-19T15:22:10.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.542+0000 7fe1e3e00640 1 --2- 172.21.15.47:0/1363511016 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe1c8066e40 0x7fe1c8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:10.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.542+0000 7fe1e3e00640 1 --2- 172.21.15.47:0/1363511016 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe1c8066e40 0x7fe1c8069300 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec154c80 tx=0x7fe1d4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:10.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.546+0000 7fe1e3400640 1 -- 172.21.15.47:0/1363511016 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe1dc0ec3b0 con 0x7fe1ec06c9a0 2024-02-19T15:22:10.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.746+0000 7fe1f2800640 1 -- 172.21.15.47:0/1363511016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fe1ec166550 con 0x7fe1ec06c9a0 2024-02-19T15:22:10.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.750+0000 7fe1e3400640 1 -- 172.21.15.47:0/1363511016 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+165201 (secure 0 0 0) 0x7fe1dc11de60 con 0x7fe1ec06c9a0 2024-02-19T15:22:10.752 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:10.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.750+0000 7fe1f2800640 1 -- 172.21.15.47:0/1363511016 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe1c8066e40 msgr2=0x7fe1c8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:10.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.750+0000 7fe1f2800640 1 --2- 172.21.15.47:0/1363511016 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe1c8066e40 0x7fe1c8069300 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec154c80 tx=0x7fe1d4009040 comp rx=0 tx=0).stop 2024-02-19T15:22:10.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.750+0000 7fe1f2800640 1 -- 172.21.15.47:0/1363511016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1ec06c9a0 msgr2=0x7fe1ec147440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:10.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.750+0000 7fe1f2800640 1 --2- 172.21.15.47:0/1363511016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1ec06c9a0 0x7fe1ec147440 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc0ad040 tx=0x7fe1dc01ace0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.750+0000 7fe1f2800640 1 -- 172.21.15.47:0/1363511016 shutdown_connections 2024-02-19T15:22:10.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.750+0000 7fe1f2800640 1 --2- 172.21.15.47:0/1363511016 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe1c8066e40 0x7fe1c8069300 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.750+0000 7fe1f2800640 1 --2- 172.21.15.47:0/1363511016 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1ec15b1a0 0x7fe1ec0b37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.750+0000 7fe1f2800640 1 --2- 172.21.15.47:0/1363511016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1ec071350 0x7fe1ec156600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.750+0000 7fe1f2800640 1 --2- 172.21.15.47:0/1363511016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1ec06c9a0 0x7fe1ec147440 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.750+0000 7fe1f2800640 1 -- 172.21.15.47:0/1363511016 >> 172.21.15.47:0/1363511016 conn(0x7fe1ec0b16a0 msgr2=0x7fe1ec0aeca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:10.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.750+0000 7fe1f2800640 1 -- 172.21.15.47:0/1363511016 shutdown_connections 2024-02-19T15:22:10.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.750+0000 7fe1f2800640 1 -- 172.21.15.47:0/1363511016 wait complete. 2024-02-19T15:22:10.772 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2412897884},{"type":"v1","addr":"172.21.15.116:6833","nonce":2412897884}]},"active_addr":"172.21.15.116:6833/2412897884","active_change":"2024-02-19T15:21:45.134946+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":2641171848}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":829603042}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":3921525327}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":4056772039}]}]} 2024-02-19T15:22:10.775 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-02-19T15:22:10.775 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-02-19T15:22:10.776 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:10.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.914+0000 7f5d7b000640 1 Processor -- start 2024-02-19T15:22:10.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.914+0000 7f5d7b000640 1 -- start start 2024-02-19T15:22:10.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.914+0000 7f5d7b000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5d740cff60 0x7f5d740d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:10.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.914+0000 7f5d7b000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5d740d1760 0x7f5d740ce320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:10.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.914+0000 7f5d7b000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d740b3240 con 0x7f5d7415e270 2024-02-19T15:22:10.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.914+0000 7f5d7b000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d740b3520 con 0x7f5d740d1760 2024-02-19T15:22:10.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.914+0000 7f5d7b000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d7406cb50 con 0x7f5d740cff60 2024-02-19T15:22:10.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.914+0000 7f5d79000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5d740d1760 0x7f5d740ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:10.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.914+0000 7f5d79000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5d740d1760 0x7f5d740ce320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52900/0 (socket says 172.21.15.47:52900) 2024-02-19T15:22:10.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.918+0000 7f5d79000640 1 -- 172.21.15.47:0/3178892519 learned_addr learned my addr 172.21.15.47:0/3178892519 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:10.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.918+0000 7f5d73e00640 1 --2- 172.21.15.47:0/3178892519 >> v2:172.21.15.116:3300/0 conn(0x7f5d740cff60 0x7f5d740d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:10.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.914+0000 7f5d6b400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5d7415e270 0x7f5d740d0340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:44006/0 (socket says 172.21.15.47:44006) 2024-02-19T15:22:10.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.918+0000 7f5d79000640 1 -- 172.21.15.47:0/3178892519 >> v2:172.21.15.116:3300/0 conn(0x7f5d740cff60 msgr2=0x7f5d740d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:10.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.918+0000 7f5d79000640 1 --2- 172.21.15.47:0/3178892519 >> v2:172.21.15.116:3300/0 conn(0x7f5d740cff60 0x7f5d740d0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.918+0000 7f5d79000640 1 -- 172.21.15.47:0/3178892519 >> v1:172.21.15.47:6789/0 conn(0x7f5d7415e270 legacy=0x7f5d740d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:10.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.918+0000 7f5d79000640 1 -- 172.21.15.47:0/3178892519 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7406ce30 con 0x7f5d740d1760 2024-02-19T15:22:10.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.918+0000 7f5d79000640 1 --2- 172.21.15.47:0/3178892519 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d740d1760 0x7f5d740ce320 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5d74071d30 tx=0x7f5d6c01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d877cc1be016c2a0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:10.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.918+0000 7f5d73400640 1 -- 172.21.15.47:0/3178892519 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c0a5840 con 0x7f5d740d1760 2024-02-19T15:22:10.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.918+0000 7f5d73400640 1 -- 172.21.15.47:0/3178892519 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d6c0a59a0 con 0x7f5d740d1760 2024-02-19T15:22:10.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.918+0000 7f5d73400640 1 -- 172.21.15.47:0/3178892519 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c0a5d30 con 0x7f5d740d1760 2024-02-19T15:22:10.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.918+0000 7f5d7b000640 1 -- 172.21.15.47:0/3178892519 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d740d1760 msgr2=0x7f5d740ce320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:10.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.918+0000 7f5d7b000640 1 --2- 172.21.15.47:0/3178892519 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d740d1760 0x7f5d740ce320 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5d74071d30 tx=0x7f5d6c01c960 comp rx=0 tx=0).stop 2024-02-19T15:22:10.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d7b000640 1 -- 172.21.15.47:0/3178892519 shutdown_connections 2024-02-19T15:22:10.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d7b000640 1 --2- 172.21.15.47:0/3178892519 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d740d1760 0x7f5d740ce320 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d7b000640 1 --2- 172.21.15.47:0/3178892519 >> v2:172.21.15.116:3300/0 conn(0x7f5d740cff60 0x7f5d740d0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d7b000640 1 -- 172.21.15.47:0/3178892519 >> 172.21.15.47:0/3178892519 conn(0x7f5d740d1b90 msgr2=0x7f5d740d1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:10.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d7b000640 1 -- 172.21.15.47:0/3178892519 shutdown_connections 2024-02-19T15:22:10.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d7b000640 1 -- 172.21.15.47:0/3178892519 wait complete. 2024-02-19T15:22:10.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d7b000640 1 Processor -- start 2024-02-19T15:22:10.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d7b000640 1 -- start start 2024-02-19T15:22:10.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d7b000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d740cff60 0x7f5d74160e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:10.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d7b000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d740d1760 0x7f5d740c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:10.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d7b000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d7415e270 0x7f5d740c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:10.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d7b000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d74150380 con 0x7f5d740d1760 2024-02-19T15:22:10.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d7b000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d741582f0 con 0x7f5d740cff60 2024-02-19T15:22:10.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d7b000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d741585d0 con 0x7f5d7415e270 2024-02-19T15:22:10.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d6b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d740d1760 0x7f5d740c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:10.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d6b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d740d1760 0x7f5d740c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52902/0 (socket says 172.21.15.47:52902) 2024-02-19T15:22:10.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d6b400640 1 -- 172.21.15.47:0/1617352686 learned_addr learned my addr 172.21.15.47:0/1617352686 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:10.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d73e00640 1 --2- 172.21.15.47:0/1617352686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d740cff60 0x7f5d74160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:10.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d79000640 1 --2- 172.21.15.47:0/1617352686 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d7415e270 0x7f5d740c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:10.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d6b400640 1 -- 172.21.15.47:0/1617352686 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d7415e270 msgr2=0x7f5d740c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:10.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d6b400640 1 --2- 172.21.15.47:0/1617352686 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d7415e270 0x7f5d740c7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d6b400640 1 -- 172.21.15.47:0/1617352686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d740cff60 msgr2=0x7f5d74160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:10.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d6b400640 1 --2- 172.21.15.47:0/1617352686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d740cff60 0x7f5d74160e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:10.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d6b400640 1 -- 172.21.15.47:0/1617352686 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d6c009b30 con 0x7f5d740d1760 2024-02-19T15:22:10.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d6b400640 1 --2- 172.21.15.47:0/1617352686 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d740d1760 0x7f5d740c74c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00c370 tx=0x7f5d5c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:10.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d70c00640 1 -- 172.21.15.47:0/1617352686 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c0074e0 con 0x7f5d740d1760 2024-02-19T15:22:10.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d70c00640 1 -- 172.21.15.47:0/1617352686 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d5c007640 con 0x7f5d740d1760 2024-02-19T15:22:10.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d7b000640 1 -- 172.21.15.47:0/1617352686 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7414db00 con 0x7f5d740d1760 2024-02-19T15:22:10.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d70c00640 1 -- 172.21.15.47:0/1617352686 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c004a30 con 0x7f5d740d1760 2024-02-19T15:22:10.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.922+0000 7f5d7b000640 1 -- 172.21.15.47:0/1617352686 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7414e0c0 con 0x7f5d740d1760 2024-02-19T15:22:10.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.926+0000 7f5d7b000640 1 -- 172.21.15.47:0/1617352686 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d74068500 con 0x7f5d740d1760 2024-02-19T15:22:10.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.926+0000 7f5d70c00640 1 -- 172.21.15.47:0/1617352686 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5d5c004b90 con 0x7f5d740d1760 2024-02-19T15:22:10.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.926+0000 7f5d70c00640 1 --2- 172.21.15.47:0/1617352686 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5d50066e40 0x7f5d50069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:10.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.926+0000 7f5d70c00640 1 -- 172.21.15.47:0/1617352686 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d5c0842a0 con 0x7f5d740d1760 2024-02-19T15:22:10.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.926+0000 7f5d73e00640 1 --2- 172.21.15.47:0/1617352686 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5d50066e40 0x7f5d50069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:10.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.930+0000 7f5d73e00640 1 --2- 172.21.15.47:0/1617352686 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5d50066e40 0x7f5d50069300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5d64009b60 tx=0x7f5d64009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:10.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:10.930+0000 7f5d70c00640 1 -- 172.21.15.47:0/1617352686 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f5d5c051120 con 0x7f5d740d1760 2024-02-19T15:22:11.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.098+0000 7f5d7b000640 1 -- 172.21.15.47:0/1617352686 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d74075360 con 0x7f5d740d1760 2024-02-19T15:22:11.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.098+0000 7f5d70c00640 1 -- 172.21.15.47:0/1617352686 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f5d5c0547e0 con 0x7f5d740d1760 2024-02-19T15:22:11.100 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:11.100 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":21,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:06.843155+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:03.091360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:04.222965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:11.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.098+0000 7f5d7b000640 1 -- 172.21.15.47:0/1617352686 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5d50066e40 msgr2=0x7f5d50069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.098+0000 7f5d7b000640 1 --2- 172.21.15.47:0/1617352686 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5d50066e40 0x7f5d50069300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5d64009b60 tx=0x7f5d64009190 comp rx=0 tx=0).stop 2024-02-19T15:22:11.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.098+0000 7f5d7b000640 1 -- 172.21.15.47:0/1617352686 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d740d1760 msgr2=0x7f5d740c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.098+0000 7f5d7b000640 1 --2- 172.21.15.47:0/1617352686 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d740d1760 0x7f5d740c74c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00c370 tx=0x7f5d5c00c830 comp rx=0 tx=0).stop 2024-02-19T15:22:11.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.098+0000 7f5d7b000640 1 -- 172.21.15.47:0/1617352686 shutdown_connections 2024-02-19T15:22:11.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.098+0000 7f5d7b000640 1 --2- 172.21.15.47:0/1617352686 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5d50066e40 0x7f5d50069300 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.098+0000 7f5d7b000640 1 --2- 172.21.15.47:0/1617352686 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d7415e270 0x7f5d740c7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.098+0000 7f5d7b000640 1 --2- 172.21.15.47:0/1617352686 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d740d1760 0x7f5d740c74c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.098+0000 7f5d7b000640 1 --2- 172.21.15.47:0/1617352686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d740cff60 0x7f5d74160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.098+0000 7f5d7b000640 1 -- 172.21.15.47:0/1617352686 >> 172.21.15.47:0/1617352686 conn(0x7f5d740d1b90 msgr2=0x7f5d74159150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:11.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.098+0000 7f5d7b000640 1 -- 172.21.15.47:0/1617352686 shutdown_connections 2024-02-19T15:22:11.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.098+0000 7f5d7b000640 1 -- 172.21.15.47:0/1617352686 wait complete. 2024-02-19T15:22:11.121 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-02-19T15:22:11.121 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:11.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.258+0000 7f49ac600640 1 Processor -- start 2024-02-19T15:22:11.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.258+0000 7f49ac600640 1 -- start start 2024-02-19T15:22:11.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.262+0000 7f49ac600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f49a415b1a0 0x7f49a415b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.262+0000 7f49ac600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a4071350 0x7f49a415bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.262+0000 7f49ac600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f49a406c9a0 0x7f49a415c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.262+0000 7f49ac600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49a415c000 con 0x7f49a406c9a0 2024-02-19T15:22:11.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.262+0000 7f49ac600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49a40718c0 con 0x7f49a415b1a0 2024-02-19T15:22:11.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.262+0000 7f49ac600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a4071ba0 con 0x7f49a4071350 2024-02-19T15:22:11.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.262+0000 7f49aa600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f49a406c9a0 0x7f49a415c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.262+0000 7f49a9c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f49a415b1a0 0x7f49a415b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.262+0000 7f49aa600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f49a406c9a0 0x7f49a415c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52916/0 (socket says 172.21.15.47:52916) 2024-02-19T15:22:11.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.262+0000 7f49aa600640 1 -- 172.21.15.47:0/2167507601 learned_addr learned my addr 172.21.15.47:0/2167507601 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:11.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.262+0000 7f49a9200640 1 --2- 172.21.15.47:0/2167507601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a4071350 0x7f49a415bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.262+0000 7f49aa600640 1 -- 172.21.15.47:0/2167507601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a4071350 msgr2=0x7f49a415bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:11.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.262+0000 7f49aa600640 1 --2- 172.21.15.47:0/2167507601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a4071350 0x7f49a415bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.262+0000 7f49aa600640 1 -- 172.21.15.47:0/2167507601 >> v2:172.21.15.116:3300/0 conn(0x7f49a415b1a0 msgr2=0x7f49a415b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:11.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.262+0000 7f49aa600640 1 --2- 172.21.15.47:0/2167507601 >> v2:172.21.15.116:3300/0 conn(0x7f49a415b1a0 0x7f49a415b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.262+0000 7f49aa600640 1 -- 172.21.15.47:0/2167507601 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a4071e80 con 0x7f49a406c9a0 2024-02-19T15:22:11.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.262+0000 7f49a9c00640 1 --2- 172.21.15.47:0/2167507601 >> v2:172.21.15.116:3300/0 conn(0x7f49a415b1a0 0x7f49a415b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:11.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.266+0000 7f49aa600640 1 --2- 172.21.15.47:0/2167507601 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a406c9a0 0x7f49a415c6d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f49a4067b40 tx=0x7f499c01ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a70ab70cc4dc508 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.266+0000 7f49a3e00640 1 -- 172.21.15.47:0/2167507601 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c00b2d0 con 0x7f49a406c9a0 2024-02-19T15:22:11.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.266+0000 7f49a3e00640 1 -- 172.21.15.47:0/2167507601 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499c00b430 con 0x7f49a406c9a0 2024-02-19T15:22:11.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.266+0000 7f49a3e00640 1 -- 172.21.15.47:0/2167507601 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c0a7c90 con 0x7f49a406c9a0 2024-02-19T15:22:11.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.266+0000 7f49ac600640 1 -- 172.21.15.47:0/2167507601 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a406c9a0 msgr2=0x7f49a415c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.266+0000 7f49ac600640 1 --2- 172.21.15.47:0/2167507601 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a406c9a0 0x7f49a415c6d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f49a4067b40 tx=0x7f499c01ebc0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.266+0000 7f49ac600640 1 -- 172.21.15.47:0/2167507601 shutdown_connections 2024-02-19T15:22:11.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.266+0000 7f49ac600640 1 --2- 172.21.15.47:0/2167507601 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a406c9a0 0x7f49a415c6d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.266+0000 7f49ac600640 1 --2- 172.21.15.47:0/2167507601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a4071350 0x7f49a415bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.266+0000 7f49ac600640 1 --2- 172.21.15.47:0/2167507601 >> v2:172.21.15.116:3300/0 conn(0x7f49a415b1a0 0x7f49a415b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.266+0000 7f49ac600640 1 -- 172.21.15.47:0/2167507601 >> 172.21.15.47:0/2167507601 conn(0x7f49a40b16a0 msgr2=0x7f49a40b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:11.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.266+0000 7f49ac600640 1 -- 172.21.15.47:0/2167507601 shutdown_connections 2024-02-19T15:22:11.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.266+0000 7f49ac600640 1 -- 172.21.15.47:0/2167507601 wait complete. 2024-02-19T15:22:11.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.266+0000 7f49ac600640 1 Processor -- start 2024-02-19T15:22:11.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.266+0000 7f49ac600640 1 -- start start 2024-02-19T15:22:11.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49ac600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a406c9a0 0x7f49a41fa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49ac600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a4071350 0x7f49a40b3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49ac600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a415b1a0 0x7f49a4160dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49ac600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a41fac00 con 0x7f49a415b1a0 2024-02-19T15:22:11.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49ac600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a40b4520 con 0x7f49a4071350 2024-02-19T15:22:11.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49ac600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a4163460 con 0x7f49a406c9a0 2024-02-19T15:22:11.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49a9c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a406c9a0 0x7f49a41fa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49aa600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a415b1a0 0x7f49a4160dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49a9c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a406c9a0 0x7f49a41fa6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:50646/0 (socket says 172.21.15.47:50646) 2024-02-19T15:22:11.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49aa600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a415b1a0 0x7f49a4160dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52928/0 (socket says 172.21.15.47:52928) 2024-02-19T15:22:11.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49a9c00640 1 -- 172.21.15.47:0/2033608089 learned_addr learned my addr 172.21.15.47:0/2033608089 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:11.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49a9200640 1 --2- 172.21.15.47:0/2033608089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a4071350 0x7f49a40b3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49aa600640 1 -- 172.21.15.47:0/2033608089 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a406c9a0 msgr2=0x7f49a41fa6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49aa600640 1 --2- 172.21.15.47:0/2033608089 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a406c9a0 0x7f49a41fa6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49aa600640 1 -- 172.21.15.47:0/2033608089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a4071350 msgr2=0x7f49a40b3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49aa600640 1 --2- 172.21.15.47:0/2033608089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a4071350 0x7f49a40b3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49aa600640 1 -- 172.21.15.47:0/2033608089 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4998009420 con 0x7f49a415b1a0 2024-02-19T15:22:11.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49aa600640 1 --2- 172.21.15.47:0/2033608089 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a415b1a0 0x7f49a4160dd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f499c0ad000 tx=0x7f499c01b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49a1600640 1 -- 172.21.15.47:0/2033608089 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c009800 con 0x7f49a415b1a0 2024-02-19T15:22:11.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49a1600640 1 -- 172.21.15.47:0/2033608089 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499c009980 con 0x7f49a415b1a0 2024-02-19T15:22:11.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49a1600640 1 -- 172.21.15.47:0/2033608089 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c009d30 con 0x7f49a415b1a0 2024-02-19T15:22:11.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49ac600640 1 -- 172.21.15.47:0/2033608089 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499c00bd40 con 0x7f49a415b1a0 2024-02-19T15:22:11.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49ac600640 1 -- 172.21.15.47:0/2033608089 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a4163a20 con 0x7f49a415b1a0 2024-02-19T15:22:11.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49ac600640 1 -- 172.21.15.47:0/2033608089 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a4075360 con 0x7f49a415b1a0 2024-02-19T15:22:11.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.270+0000 7f49a1600640 1 -- 172.21.15.47:0/2033608089 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f499c0c3460 con 0x7f49a415b1a0 2024-02-19T15:22:11.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.274+0000 7f49a1600640 1 --2- 172.21.15.47:0/2033608089 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4984066e40 0x7f4984069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.274+0000 7f49a1600640 1 -- 172.21.15.47:0/2033608089 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f499c11ff80 con 0x7f49a415b1a0 2024-02-19T15:22:11.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.274+0000 7f49a9c00640 1 --2- 172.21.15.47:0/2033608089 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4984066e40 0x7f4984069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.274+0000 7f49a9c00640 1 --2- 172.21.15.47:0/2033608089 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4984066e40 0x7f4984069300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4998009df0 tx=0x7f4998009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.278+0000 7f49a1600640 1 -- 172.21.15.47:0/2033608089 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f499c0ecd80 con 0x7f49a415b1a0 2024-02-19T15:22:11.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.438+0000 7f49ac600640 1 -- 172.21.15.47:0/2033608089 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49a4159170 con 0x7f49a415b1a0 2024-02-19T15:22:11.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.442+0000 7f49a1600640 1 -- 172.21.15.47:0/2033608089 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f499c0f0440 con 0x7f49a415b1a0 2024-02-19T15:22:11.443 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:11.443 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":21,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:06.843155+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:03.091360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:04.222965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:11.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.442+0000 7f49ac600640 1 -- 172.21.15.47:0/2033608089 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4984066e40 msgr2=0x7f4984069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.442+0000 7f49ac600640 1 --2- 172.21.15.47:0/2033608089 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4984066e40 0x7f4984069300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4998009df0 tx=0x7f4998009190 comp rx=0 tx=0).stop 2024-02-19T15:22:11.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.442+0000 7f49ac600640 1 -- 172.21.15.47:0/2033608089 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a415b1a0 msgr2=0x7f49a4160dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.442+0000 7f49ac600640 1 --2- 172.21.15.47:0/2033608089 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a415b1a0 0x7f49a4160dd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f499c0ad000 tx=0x7f499c01b960 comp rx=0 tx=0).stop 2024-02-19T15:22:11.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.442+0000 7f49ac600640 1 -- 172.21.15.47:0/2033608089 shutdown_connections 2024-02-19T15:22:11.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.442+0000 7f49ac600640 1 --2- 172.21.15.47:0/2033608089 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4984066e40 0x7f4984069300 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.442+0000 7f49ac600640 1 --2- 172.21.15.47:0/2033608089 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a415b1a0 0x7f49a4160dd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.442+0000 7f49ac600640 1 --2- 172.21.15.47:0/2033608089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a4071350 0x7f49a40b3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.442+0000 7f49ac600640 1 --2- 172.21.15.47:0/2033608089 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a406c9a0 0x7f49a41fa6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.442+0000 7f49ac600640 1 -- 172.21.15.47:0/2033608089 >> 172.21.15.47:0/2033608089 conn(0x7f49a40b16a0 msgr2=0x7f49a415b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:11.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.442+0000 7f49ac600640 1 -- 172.21.15.47:0/2033608089 shutdown_connections 2024-02-19T15:22:11.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.442+0000 7f49ac600640 1 -- 172.21.15.47:0/2033608089 wait complete. 2024-02-19T15:22:11.464 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-02-19T15:22:11.464 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-02-19T15:22:11.465 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-02-19T15:22:11.465 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-02-19T15:22:11.466 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-02-19T15:22:11.466 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-02-19T15:22:11.466 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-02-19T15:22:11.467 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-02-19T15:22:11.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.710+0000 7f0397e00640 1 Processor -- start 2024-02-19T15:22:11.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.710+0000 7f9f8f600640 1 Processor -- start 2024-02-19T15:22:11.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.710+0000 7f9f8f600640 1 -- start start 2024-02-19T15:22:11.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.710+0000 7f0397e00640 1 -- start start 2024-02-19T15:22:11.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f0397e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f039815bed0 0x7f03980d0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f0397e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f039806fbf0 0x7f03980d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f0397e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0398070050 0x7f03980d2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f0397e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0398070ab0 con 0x7f0398070050 2024-02-19T15:22:11.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f0397e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03980aed70 con 0x7f039815bed0 2024-02-19T15:22:11.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f0397e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03980704f0 con 0x7f039806fbf0 2024-02-19T15:22:11.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f9f8f600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9f800ddca0 0x7f9f80054c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f9f8f600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9f800e4000 0x7f9f80034d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f9f8f600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f800e4a40 con 0x7f9f800e3ba0 2024-02-19T15:22:11.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f9f8f600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f800514a0 con 0x7f9f800ddca0 2024-02-19T15:22:11.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f9f8f600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f8003ba90 con 0x7f9f800e4000 2024-02-19T15:22:11.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f9f87e00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9f800e3ba0 0x7f9f80034660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:46212/0 (socket says 172.21.15.47:46212) 2024-02-19T15:22:11.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f9f87e00640 1 -- 172.21.15.47:0/3079401172 learned_addr learned my addr 172.21.15.47:0/3079401172 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:11.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f0396a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f039815bed0 0x7f03980d0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f0396a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f039815bed0 0x7f03980d0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:45512/0 (socket says 172.21.15.47:45512) 2024-02-19T15:22:11.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f0396a00640 1 -- 172.21.15.47:0/4194548752 learned_addr learned my addr 172.21.15.47:0/4194548752 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:11.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f0396000640 1 --2- 172.21.15.47:0/4194548752 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f039806fbf0 0x7f03980d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f9f8d600640 1 --2- 172.21.15.47:0/3079401172 >> v2:172.21.15.116:3300/0 conn(0x7f9f800e4000 0x7f9f80034d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f9f87400640 1 -- 172.21.15.47:0/3079401172 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665888735 0 0) 0x7f9f74001120 con 0x7f9f800e3ba0 2024-02-19T15:22:11.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f9f8d600640 1 -- 172.21.15.47:0/3079401172 >> v2:172.21.15.47:3300/0 conn(0x7f9f800ddca0 msgr2=0x7f9f80054c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:11.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f9f8cc00640 1 --2- 172.21.15.47:0/3079401172 >> v2:172.21.15.47:3300/0 conn(0x7f9f800ddca0 0x7f9f80054c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f9f8d600640 1 --2- 172.21.15.47:0/3079401172 >> v2:172.21.15.47:3300/0 conn(0x7f9f800ddca0 0x7f9f80054c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f9f8d600640 1 -- 172.21.15.47:0/3079401172 >> v1:172.21.15.116:6789/0 conn(0x7f9f800e3ba0 legacy=0x7f9f80034660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f9f8d600640 1 -- 172.21.15.47:0/3079401172 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f80050820 con 0x7f9f800e4000 2024-02-19T15:22:11.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.718+0000 7f9f8cc00640 1 --2- 172.21.15.47:0/3079401172 >> v2:172.21.15.47:3300/0 conn(0x7f9f800ddca0 0x7f9f80054c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:22:11.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.718+0000 7f9426600640 1 Processor -- start 2024-02-19T15:22:11.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.718+0000 7f6ef1e00640 1 Processor -- start 2024-02-19T15:22:11.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f0397400640 1 --2- 172.21.15.47:0/4194548752 >> v2:172.21.15.47:3300/0 conn(0x7f0398070050 0x7f03980d2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f0396a00640 1 -- 172.21.15.47:0/4194548752 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f039806fbf0 msgr2=0x7f03980d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:11.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f0396a00640 1 --2- 172.21.15.47:0/4194548752 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f039806fbf0 0x7f03980d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f0396a00640 1 -- 172.21.15.47:0/4194548752 >> v2:172.21.15.47:3300/0 conn(0x7f0398070050 msgr2=0x7f03980d2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:11.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.718+0000 7f064d400640 1 Processor -- start 2024-02-19T15:22:11.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f0396a00640 1 --2- 172.21.15.47:0/4194548752 >> v2:172.21.15.47:3300/0 conn(0x7f0398070050 0x7f03980d2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f0396a00640 1 -- 172.21.15.47:0/4194548752 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03980707d0 con 0x7f039815bed0 2024-02-19T15:22:11.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.714+0000 7f0397400640 1 --2- 172.21.15.47:0/4194548752 >> v2:172.21.15.47:3300/0 conn(0x7f0398070050 0x7f03980d2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:22:11.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.718+0000 7f6ef1e00640 1 -- start start 2024-02-19T15:22:11.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.718+0000 7f9426600640 1 -- start start 2024-02-19T15:22:11.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.718+0000 7f7598000640 1 Processor -- start 2024-02-19T15:22:11.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.718+0000 7f7598000640 1 -- start start 2024-02-19T15:22:11.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.718+0000 7f6ef1e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6ee40339c0 0x7f6ee4038640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.718+0000 7f6ef1e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee4039e60 0x7f6ee403a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.718+0000 7f6ef1e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee4050d20 con 0x7f6ee404e760 2024-02-19T15:22:11.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.718+0000 7f6ef1e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee4051000 con 0x7f6ee40339c0 2024-02-19T15:22:11.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.718+0000 7f6ef1e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee40512e0 con 0x7f6ee4039e60 2024-02-19T15:22:11.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.718+0000 7f064d400640 1 -- start start 2024-02-19T15:22:11.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.718+0000 7f064d400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f06400e32e0 0x7f064003e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.718+0000 7f064d400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0640007810 0x7f06400540b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.718+0000 7f064d400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f064003bfd0 con 0x7f0640053c20 2024-02-19T15:22:11.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.718+0000 7f064d400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0640058690 con 0x7f06400e32e0 2024-02-19T15:22:11.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.718+0000 7f064d400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06400545f0 con 0x7f0640007810 2024-02-19T15:22:11.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f064ca00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0640007810 0x7f06400540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f0647400640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0640053c20 0x7f06400e6c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:46222/0 (socket says 172.21.15.47:46222) 2024-02-19T15:22:11.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f0647400640 1 -- 172.21.15.47:0/3551854613 learned_addr learned my addr 172.21.15.47:0/3551854613 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:11.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f7598000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75901585c0 0x7f759006ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f0647e00640 1 --2- 172.21.15.47:0/3551854613 >> v2:172.21.15.116:3300/0 conn(0x7f06400e32e0 0x7f064003e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f6eebe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6ee40339c0 0x7f6ee4038640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f6eebe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6ee40339c0 0x7f6ee4038640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52960/0 (socket says 172.21.15.47:52960) 2024-02-19T15:22:11.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f6eebe00640 1 -- 172.21.15.47:0/2764843942 learned_addr learned my addr 172.21.15.47:0/2764843942 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:11.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f6ef1400640 1 --2- 172.21.15.47:0/2764843942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee4039e60 0x7f6ee403a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f9f8d600640 1 --2- 172.21.15.47:0/3079401172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f800e4000 0x7f9f80034d70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9f88000b70 tx=0x7f9f88097710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0dd792915095dd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f7598000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f75900728c0 0x7f75900712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f7598000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7590070e10 con 0x7f75900b29a0 2024-02-19T15:22:11.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f7598000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75901589a0 con 0x7f75900728c0 2024-02-19T15:22:11.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f7598000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7590070ae0 con 0x7f75901585c0 2024-02-19T15:22:11.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f6eebe00640 1 -- 172.21.15.47:0/2764843942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee4039e60 msgr2=0x7f6ee403a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:11.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f6eebe00640 1 --2- 172.21.15.47:0/2764843942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee4039e60 0x7f6ee403a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f6eebe00640 1 -- 172.21.15.47:0/2764843942 >> v1:172.21.15.47:6789/0 conn(0x7f6ee404e760 legacy=0x7f6ee40500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f6eeb400640 1 -- 172.21.15.47:0/2764843942 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629153099 0 0) 0x7f6ee4050d20 con 0x7f6ee404e760 2024-02-19T15:22:11.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f6eebe00640 1 -- 172.21.15.47:0/2764843942 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee404f5f0 con 0x7f6ee40339c0 2024-02-19T15:22:11.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f064ca00640 1 -- 172.21.15.47:0/3551854613 >> v2:172.21.15.116:3300/0 conn(0x7f06400e32e0 msgr2=0x7f064003e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:11.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f064ca00640 1 --2- 172.21.15.47:0/3551854613 >> v2:172.21.15.116:3300/0 conn(0x7f06400e32e0 0x7f064003e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f064ca00640 1 -- 172.21.15.47:0/3551854613 >> v1:172.21.15.116:6789/0 conn(0x7f0640053c20 legacy=0x7f06400e6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:11.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f064ca00640 1 -- 172.21.15.47:0/3551854613 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0640057df0 con 0x7f0640007810 2024-02-19T15:22:11.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f9f87400640 1 -- 172.21.15.47:0/3079401172 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f88126600 con 0x7f9f800e4000 2024-02-19T15:22:11.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f7595600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75901585c0 0x7f759006ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f7595600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75901585c0 0x7f759006ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:50692/0 (socket says 172.21.15.47:50692) 2024-02-19T15:22:11.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f7595600640 1 -- 172.21.15.47:0/2968540550 learned_addr learned my addr 172.21.15.47:0/2968540550 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:11.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f7596000640 1 --2- 172.21.15.47:0/2968540550 >> v2:172.21.15.47:3300/0 conn(0x7f75900728c0 0x7f75900712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f9f87400640 1 -- 172.21.15.47:0/3079401172 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f88122050 con 0x7f9f800e4000 2024-02-19T15:22:11.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f9f87400640 1 -- 172.21.15.47:0/3079401172 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f881269a0 con 0x7f9f800e4000 2024-02-19T15:22:11.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f7595600640 1 -- 172.21.15.47:0/2968540550 >> v2:172.21.15.47:3300/0 conn(0x7f75900728c0 msgr2=0x7f75900712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:11.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f7595600640 1 --2- 172.21.15.47:0/2968540550 >> v2:172.21.15.47:3300/0 conn(0x7f75900728c0 0x7f75900712a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f7595600640 1 -- 172.21.15.47:0/2968540550 >> v1:172.21.15.47:6789/0 conn(0x7f75900b29a0 legacy=0x7f75900721b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:11.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f7595600640 1 -- 172.21.15.47:0/2968540550 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75900b3340 con 0x7f75901585c0 2024-02-19T15:22:11.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f9426600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f94200cf0e0 0x7f94200b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f9f8f600640 1 -- 172.21.15.47:0/3079401172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f800e4000 msgr2=0x7f9f80034d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f9f8f600640 1 --2- 172.21.15.47:0/3079401172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f800e4000 0x7f9f80034d70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9f88000b70 tx=0x7f9f88097710 comp rx=0 tx=0).stop 2024-02-19T15:22:11.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f0396a00640 1 --2- 172.21.15.47:0/4194548752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f039815bed0 0x7f03980d0620 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f03980cf650 tx=0x7f038801c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a151c6c57a82675 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f9426600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f94200b3610 0x7f9420070b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f9f8f600640 1 -- 172.21.15.47:0/3079401172 shutdown_connections 2024-02-19T15:22:11.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f9f8f600640 1 --2- 172.21.15.47:0/3079401172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f800e4000 0x7f9f80034d70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f9f8f600640 1 --2- 172.21.15.47:0/3079401172 >> v2:172.21.15.47:3300/0 conn(0x7f9f800ddca0 0x7f9f80054c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f9f8f600640 1 -- 172.21.15.47:0/3079401172 >> 172.21.15.47:0/3079401172 conn(0x7f9f80055be0 msgr2=0x7f9f80055ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:11.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f0ec2e00640 1 Processor -- start 2024-02-19T15:22:11.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f9f8f600640 1 -- 172.21.15.47:0/3079401172 shutdown_connections 2024-02-19T15:22:11.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f0395600640 1 -- 172.21.15.47:0/4194548752 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03880ab600 con 0x7f039815bed0 2024-02-19T15:22:11.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.722+0000 7f941f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f94200cf0e0 0x7f94200b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f941f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f94200cf0e0 0x7f94200b30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52978/0 (socket says 172.21.15.47:52978) 2024-02-19T15:22:11.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f941f400640 1 -- 172.21.15.47:0/1578331960 learned_addr learned my addr 172.21.15.47:0/1578331960 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:11.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0ec2e00640 1 -- start start 2024-02-19T15:22:11.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0395600640 1 -- 172.21.15.47:0/4194548752 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03880a7050 con 0x7f039815bed0 2024-02-19T15:22:11.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0395600640 1 -- 172.21.15.47:0/4194548752 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03880ab980 con 0x7f039815bed0 2024-02-19T15:22:11.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f9426600640 1 --2- 172.21.15.47:0/1578331960 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9420071090 0x7f942015dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f9426600640 1 -- 172.21.15.47:0/1578331960 --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9420071520 con 0x7f94200cf0e0 2024-02-19T15:22:11.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f9426600640 1 -- 172.21.15.47:0/1578331960 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94200cf4c0 con 0x7f94200b3610 2024-02-19T15:22:11.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f9426600640 1 -- 172.21.15.47:0/1578331960 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94200b3ab0 con 0x7f9420071090 2024-02-19T15:22:11.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f9f8f600640 1 -- 172.21.15.47:0/3079401172 wait complete. 2024-02-19T15:22:11.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0397e00640 1 -- 172.21.15.47:0/4194548752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f039815bed0 msgr2=0x7f03980d0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0397e00640 1 --2- 172.21.15.47:0/4194548752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f039815bed0 0x7f03980d0620 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f03980cf650 tx=0x7f038801c460 comp rx=0 tx=0).stop 2024-02-19T15:22:11.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0397e00640 1 -- 172.21.15.47:0/4194548752 shutdown_connections 2024-02-19T15:22:11.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0397e00640 1 --2- 172.21.15.47:0/4194548752 >> v2:172.21.15.47:3300/0 conn(0x7f0398070050 0x7f03980d2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0397e00640 1 --2- 172.21.15.47:0/4194548752 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f039806fbf0 0x7f03980d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f941ea00640 1 --2- 172.21.15.47:0/1578331960 >> v2:172.21.15.116:3300/0 conn(0x7f94200b3610 0x7f9420070b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f941fe00640 1 --2- 172.21.15.47:0/1578331960 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9420071090 0x7f942015dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0397e00640 1 --2- 172.21.15.47:0/4194548752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f039815bed0 0x7f03980d0620 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0397e00640 1 -- 172.21.15.47:0/4194548752 >> 172.21.15.47:0/4194548752 conn(0x7f039809a9d0 msgr2=0x7f039809ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:11.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0ec2e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0eb4033e40 0x7f0eb40eb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0397e00640 1 -- 172.21.15.47:0/4194548752 shutdown_connections 2024-02-19T15:22:11.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0ec2e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0eb4005ea0 0x7f0eb40062d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0397e00640 1 -- 172.21.15.47:0/4194548752 wait complete. 2024-02-19T15:22:11.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f064ca00640 1 --2- 172.21.15.47:0/3551854613 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0640007810 0x7f06400540b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f063c0097c0 tx=0x7f063c01c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75672a27ba66683c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0ec2e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb4037bc0 con 0x7f0eb4047c20 2024-02-19T15:22:11.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0ec2e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eb4050210 con 0x7f0eb4033e40 2024-02-19T15:22:11.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0ec2e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb404f090 con 0x7f0eb4005ea0 2024-02-19T15:22:11.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f6eebe00640 1 --2- 172.21.15.47:0/2764843942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ee40339c0 0x7f6ee4038640 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6ed8009870 tx=0x7f6ed801c580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1df9be82f6ae7b44 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0646a00640 1 -- 172.21.15.47:0/3551854613 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063c0a4ea0 con 0x7f0640007810 2024-02-19T15:22:11.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0646a00640 1 -- 172.21.15.47:0/3551854613 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f063c0a7050 con 0x7f0640007810 2024-02-19T15:22:11.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0646a00640 1 -- 172.21.15.47:0/3551854613 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063c0ab820 con 0x7f0640007810 2024-02-19T15:22:11.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0ebbe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0eb4033e40 0x7f0eb40eb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f7595600640 1 --2- 172.21.15.47:0/2968540550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75901585c0 0x7f759006ce70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7588009e50 tx=0x7f758801c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16154d2348a1cf11 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f941f400640 1 -- 172.21.15.47:0/1578331960 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9420071090 msgr2=0x7f942015dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:11.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f941f400640 1 --2- 172.21.15.47:0/1578331960 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9420071090 0x7f942015dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f941f400640 1 -- 172.21.15.47:0/1578331960 >> v2:172.21.15.116:3300/0 conn(0x7f94200b3610 msgr2=0x7f9420070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:11.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f941f400640 1 --2- 172.21.15.47:0/1578331960 >> v2:172.21.15.116:3300/0 conn(0x7f94200b3610 0x7f9420070b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f941f400640 1 -- 172.21.15.47:0/1578331960 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94200b3d90 con 0x7f94200cf0e0 2024-02-19T15:22:11.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f941fe00640 1 --2- 172.21.15.47:0/1578331960 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9420071090 0x7f942015dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:11.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0ebb400640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0eb4047c20 0x7f0eb40eb6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:46228/0 (socket says 172.21.15.47:46228) 2024-02-19T15:22:11.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0ebb400640 1 -- 172.21.15.47:0/3852139518 learned_addr learned my addr 172.21.15.47:0/3852139518 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:11.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0ec0e00640 1 --2- 172.21.15.47:0/3852139518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0eb4005ea0 0x7f0eb40062d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f064d400640 1 -- 172.21.15.47:0/3551854613 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0640007810 msgr2=0x7f06400540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f064d400640 1 --2- 172.21.15.47:0/3551854613 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0640007810 0x7f06400540b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f063c0097c0 tx=0x7f063c01c450 comp rx=0 tx=0).stop 2024-02-19T15:22:11.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f6eeb400640 1 -- 172.21.15.47:0/2764843942 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed80ab600 con 0x7f6ee40339c0 2024-02-19T15:22:11.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f6eeb400640 1 -- 172.21.15.47:0/2764843942 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed80a7050 con 0x7f6ee40339c0 2024-02-19T15:22:11.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f6eeb400640 1 -- 172.21.15.47:0/2764843942 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed80ab980 con 0x7f6ee40339c0 2024-02-19T15:22:11.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f0ebaa00640 1 -- 172.21.15.47:0/3852139518 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680551484 0 0) 0x7f0eb4037bc0 con 0x7f0eb4047c20 2024-02-19T15:22:11.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f758fe00640 1 -- 172.21.15.47:0/2968540550 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75880a5750 con 0x7f75901585c0 2024-02-19T15:22:11.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f758fe00640 1 -- 172.21.15.47:0/2968540550 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75880a58b0 con 0x7f75901585c0 2024-02-19T15:22:11.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.726+0000 7f758fe00640 1 -- 172.21.15.47:0/2968540550 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75880a5c40 con 0x7f75901585c0 2024-02-19T15:22:11.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f064d400640 1 -- 172.21.15.47:0/3551854613 shutdown_connections 2024-02-19T15:22:11.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f0ebaa00640 1 -- 172.21.15.47:0/3852139518 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ea4003530 con 0x7f0eb4047c20 2024-02-19T15:22:11.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f064d400640 1 --2- 172.21.15.47:0/3551854613 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0640007810 0x7f06400540b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f064d400640 1 --2- 172.21.15.47:0/3551854613 >> v2:172.21.15.116:3300/0 conn(0x7f06400e32e0 0x7f064003e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f064d400640 1 -- 172.21.15.47:0/3551854613 >> 172.21.15.47:0/3551854613 conn(0x7f06400559f0 msgr2=0x7f0640055e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:11.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f0ec0e00640 1 -- 172.21.15.47:0/3852139518 >> v2:172.21.15.47:3300/0 conn(0x7f0eb4033e40 msgr2=0x7f0eb40eb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:11.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f0ec0e00640 1 --2- 172.21.15.47:0/3852139518 >> v2:172.21.15.47:3300/0 conn(0x7f0eb4033e40 0x7f0eb40eb180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f0ec0e00640 1 -- 172.21.15.47:0/3852139518 >> v1:172.21.15.116:6789/0 conn(0x7f0eb4047c20 legacy=0x7f0eb40eb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f0ec0e00640 1 -- 172.21.15.47:0/3852139518 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb404fd80 con 0x7f0eb4005ea0 2024-02-19T15:22:11.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f9f8f600640 1 Processor -- start 2024-02-19T15:22:11.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f9f8f600640 1 -- start start 2024-02-19T15:22:11.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f6ef1e00640 1 -- 172.21.15.47:0/2764843942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ee40339c0 msgr2=0x7f6ee4038640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f6ef1e00640 1 --2- 172.21.15.47:0/2764843942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ee40339c0 0x7f6ee4038640 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6ed8009870 tx=0x7f6ed801c580 comp rx=0 tx=0).stop 2024-02-19T15:22:11.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f6ef1e00640 1 -- 172.21.15.47:0/2764843942 shutdown_connections 2024-02-19T15:22:11.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f6ef1e00640 1 --2- 172.21.15.47:0/2764843942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee4039e60 0x7f6ee403a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f6ef1e00640 1 --2- 172.21.15.47:0/2764843942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ee40339c0 0x7f6ee4038640 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f6ef1e00640 1 -- 172.21.15.47:0/2764843942 >> 172.21.15.47:0/2764843942 conn(0x7f6ee4009fc0 msgr2=0x7f6ee400a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:11.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f6ef1e00640 1 -- 172.21.15.47:0/2764843942 shutdown_connections 2024-02-19T15:22:11.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f064d400640 1 -- 172.21.15.47:0/3551854613 shutdown_connections 2024-02-19T15:22:11.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f6ef1e00640 1 -- 172.21.15.47:0/2764843942 wait complete. 2024-02-19T15:22:11.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f25c5600640 1 Processor -- start 2024-02-19T15:22:11.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f9f8f600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f800ddca0 0x7f9f800dd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f9f8f600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f800e3ba0 0x7f9f8004f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f9f8f600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f800e4000 0x7f9f800e2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f9f8f600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f800dd8b0 con 0x7f9f800e3ba0 2024-02-19T15:22:11.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f9f8f600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8004ac60 con 0x7f9f800e4000 2024-02-19T15:22:11.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f9f8f600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8004b530 con 0x7f9f800ddca0 2024-02-19T15:22:11.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f7598000640 1 -- 172.21.15.47:0/2968540550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75901585c0 msgr2=0x7f759006ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f7598000640 1 --2- 172.21.15.47:0/2968540550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75901585c0 0x7f759006ce70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7588009e50 tx=0x7f758801c930 comp rx=0 tx=0).stop 2024-02-19T15:22:11.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f25c5600640 1 -- start start 2024-02-19T15:22:11.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f7598000640 1 -- 172.21.15.47:0/2968540550 shutdown_connections 2024-02-19T15:22:11.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f7598000640 1 --2- 172.21.15.47:0/2968540550 >> v2:172.21.15.47:3300/0 conn(0x7f75900728c0 0x7f75900712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f7598000640 1 --2- 172.21.15.47:0/2968540550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75901585c0 0x7f759006ce70 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7588009e50 tx=0x7f758801c930 comp rx=0 tx=0).stop 2024-02-19T15:22:11.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f7598000640 1 -- 172.21.15.47:0/2968540550 >> 172.21.15.47:0/2968540550 conn(0x7f75900b16a0 msgr2=0x7f75900b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:11.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f9f8cc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f800ddca0 0x7f9f800dd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f9f8d600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f800e4000 0x7f9f800e2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f9f8cc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f800ddca0 0x7f9f800dd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:50724/0 (socket says 172.21.15.47:50724) 2024-02-19T15:22:11.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f9f8cc00640 1 -- 172.21.15.47:0/3108666576 learned_addr learned my addr 172.21.15.47:0/3108666576 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:11.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f9f87e00640 1 --2- 172.21.15.47:0/3108666576 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f800e3ba0 0x7f9f8004f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f064d400640 1 -- 172.21.15.47:0/3551854613 wait complete. 2024-02-19T15:22:11.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f0397e00640 1 Processor -- start 2024-02-19T15:22:11.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f9f8cc00640 1 -- 172.21.15.47:0/3108666576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f800e4000 msgr2=0x7f9f800e2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f9f8cc00640 1 --2- 172.21.15.47:0/3108666576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f800e4000 0x7f9f800e2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f9f8cc00640 1 -- 172.21.15.47:0/3108666576 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f800e3ba0 msgr2=0x7f9f8004f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f9f8cc00640 1 --2- 172.21.15.47:0/3108666576 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f800e3ba0 0x7f9f8004f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f9f8cc00640 1 -- 172.21.15.47:0/3108666576 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f88073870 con 0x7f9f800ddca0 2024-02-19T15:22:11.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.730+0000 7f6ef1e00640 1 Processor -- start 2024-02-19T15:22:11.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f6ef1e00640 1 -- start start 2024-02-19T15:22:11.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f064d400640 1 Processor -- start 2024-02-19T15:22:11.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0397e00640 1 -- start start 2024-02-19T15:22:11.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f9f8cc00640 1 --2- 172.21.15.47:0/3108666576 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f800ddca0 0x7f9f800dd370 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c00c7c0 tx=0x7f9f7c00cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f7598000640 1 -- 172.21.15.47:0/2968540550 shutdown_connections 2024-02-19T15:22:11.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25c5600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f25c0071050 0x7f25c01628c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f6ef1e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee40339c0 0x7f6ee40d3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f6ef1e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ee4039e60 0x7f6ee40e1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f6ef1e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ee40e2f00 0x7f6ee40cf630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f6ef1e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee40e1810 con 0x7f6ee40e2f00 2024-02-19T15:22:11.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f6ef1e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee40d6290 con 0x7f6ee4039e60 2024-02-19T15:22:11.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f6ef1e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee40e3900 con 0x7f6ee40339c0 2024-02-19T15:22:11.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f6ef0a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee40339c0 0x7f6ee40d3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f6eebe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ee4039e60 0x7f6ee40e1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f6ef1400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ee40e2f00 0x7f6ee40cf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f6ef0a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee40339c0 0x7f6ee40d3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:50738/0 (socket says 172.21.15.47:50738) 2024-02-19T15:22:11.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f6ef0a00640 1 -- 172.21.15.47:0/3280151261 learned_addr learned my addr 172.21.15.47:0/3280151261 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:11.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f6ef0a00640 1 -- 172.21.15.47:0/3280151261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ee4039e60 msgr2=0x7f6ee40e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f6ef0a00640 1 --2- 172.21.15.47:0/3280151261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ee4039e60 0x7f6ee40e1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f6ef0a00640 1 -- 172.21.15.47:0/3280151261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ee40e2f00 msgr2=0x7f6ee40cf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f6ef0a00640 1 --2- 172.21.15.47:0/3280151261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ee40e2f00 0x7f6ee40cf630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f6ef0a00640 1 -- 172.21.15.47:0/3280151261 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed80094d0 con 0x7f6ee40339c0 2024-02-19T15:22:11.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0397e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f039806fbf0 0x7f039806f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0397e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0398070050 0x7f03980b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0397e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f03980cb2b0 0x7f03981fe200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0397e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03981477d0 con 0x7f039806fbf0 2024-02-19T15:22:11.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0397e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0398147a20 con 0x7f0398070050 2024-02-19T15:22:11.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0397e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0398147d00 con 0x7f03980cb2b0 2024-02-19T15:22:11.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0396000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0398070050 0x7f03980b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0396000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0398070050 0x7f03980b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:45574/0 (socket says 172.21.15.47:45574) 2024-02-19T15:22:11.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0396000640 1 -- 172.21.15.47:0/4141416363 learned_addr learned my addr 172.21.15.47:0/4141416363 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:11.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0397400640 1 --2- 172.21.15.47:0/4141416363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f03980cb2b0 0x7f03981fe200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f9f84c00640 1 -- 172.21.15.47:0/3108666576 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c007a30 con 0x7f9f800ddca0 2024-02-19T15:22:11.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f9f8f600640 1 -- 172.21.15.47:0/3108666576 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f8004b810 con 0x7f9f800ddca0 2024-02-19T15:22:11.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f9f8f600640 1 -- 172.21.15.47:0/3108666576 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f8004bdd0 con 0x7f9f800ddca0 2024-02-19T15:22:11.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f9f84c00640 1 -- 172.21.15.47:0/3108666576 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7c007b90 con 0x7f9f800ddca0 2024-02-19T15:22:11.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f9f84c00640 1 -- 172.21.15.47:0/3108666576 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c007a30 con 0x7f9f800ddca0 2024-02-19T15:22:11.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f7598000640 1 -- 172.21.15.47:0/2968540550 wait complete. 2024-02-19T15:22:11.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f7598000640 1 Processor -- start 2024-02-19T15:22:11.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f064d400640 1 -- start start 2024-02-19T15:22:11.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0ec0e00640 1 --2- 172.21.15.47:0/3852139518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0eb4005ea0 0x7f0eb40062d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0eac0097c0 tx=0x7f0eac01ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b76ec9d63751e4da server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0ebaa00640 1 -- 172.21.15.47:0/3852139518 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac0a8890 con 0x7f0eb4005ea0 2024-02-19T15:22:11.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0ebaa00640 1 -- 172.21.15.47:0/3852139518 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eac0a89f0 con 0x7f0eb4005ea0 2024-02-19T15:22:11.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0ebaa00640 1 -- 172.21.15.47:0/3852139518 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac0a8d80 con 0x7f0eb4005ea0 2024-02-19T15:22:11.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0ec2e00640 1 -- 172.21.15.47:0/3852139518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0eb4005ea0 msgr2=0x7f0eb40062d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0ec2e00640 1 --2- 172.21.15.47:0/3852139518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0eb4005ea0 0x7f0eb40062d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0eac0097c0 tx=0x7f0eac01ec90 comp rx=0 tx=0).stop 2024-02-19T15:22:11.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0ec2e00640 1 -- 172.21.15.47:0/3852139518 shutdown_connections 2024-02-19T15:22:11.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0ec2e00640 1 --2- 172.21.15.47:0/3852139518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0eb4005ea0 0x7f0eb40062d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0ec2e00640 1 --2- 172.21.15.47:0/3852139518 >> v2:172.21.15.47:3300/0 conn(0x7f0eb4033e40 0x7f0eb40eb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0ec2e00640 1 -- 172.21.15.47:0/3852139518 >> 172.21.15.47:0/3852139518 conn(0x7f0eb4009fc0 msgr2=0x7f0eb400a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:11.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0ec2e00640 1 -- 172.21.15.47:0/3852139518 shutdown_connections 2024-02-19T15:22:11.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0ec2e00640 1 -- 172.21.15.47:0/3852139518 wait complete. 2024-02-19T15:22:11.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25c5600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c00b3e20 con 0x7f25c01635c0 2024-02-19T15:22:11.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25bf400640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f25c00b2cc0 0x7f25c006c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:46236/0 (socket says 172.21.15.47:46236) 2024-02-19T15:22:11.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25bf400640 1 -- 172.21.15.47:0/2405224457 learned_addr learned my addr 172.21.15.47:0/2405224457 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:11.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25c5600640 1 -- 172.21.15.47:0/2405224457 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c00b3210 con 0x7f25c00b2cc0 2024-02-19T15:22:11.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25c5600640 1 -- 172.21.15.47:0/2405224457 --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c00b34f0 con 0x7f25c0071050 2024-02-19T15:22:11.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25bd600640 1 -- 172.21.15.47:0/2405224457 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875780524 0 0) 0x7f25c00b3210 con 0x7f25c00b2cc0 2024-02-19T15:22:11.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25bd600640 1 -- 172.21.15.47:0/2405224457 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a4003530 con 0x7f25c00b2cc0 2024-02-19T15:22:11.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25bd600640 1 -- 172.21.15.47:0/2405224457 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2948187046 0 0) 0x7f25a4003530 con 0x7f25c00b2cc0 2024-02-19T15:22:11.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25bd600640 1 -- 172.21.15.47:0/2405224457 >> v2:172.21.15.47:3300/0 conn(0x7f25c0071050 msgr2=0x7f25c01628c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:11.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25bd600640 1 --2- 172.21.15.47:0/2405224457 >> v2:172.21.15.47:3300/0 conn(0x7f25c0071050 0x7f25c01628c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25bd600640 1 -- 172.21.15.47:0/2405224457 >> v1:172.21.15.47:6789/0 conn(0x7f25c01635c0 legacy=0x7f25c0070940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:11.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25bd600640 1 -- 172.21.15.47:0/2405224457 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c00b37d0 con 0x7f25c00b2cc0 2024-02-19T15:22:11.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25bd600640 1 -- 172.21.15.47:0/2405224457 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f25b0001680 con 0x7f25c00b2cc0 2024-02-19T15:22:11.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25bd600640 1 -- 172.21.15.47:0/2405224457 >> v1:172.21.15.116:6789/0 conn(0x7f25c00b2cc0 legacy=0x7f25c006c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25bd600640 1 --2- 172.21.15.47:0/2405224457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25a4003980 0x7f25a4003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25bd600640 1 --2- 172.21.15.47:0/2405224457 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25a40047d0 0x7f25a4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25bd600640 1 --2- 172.21.15.47:0/2405224457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25a40052b0 0x7f25a400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25bd600640 1 -- 172.21.15.47:0/2405224457 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c00b34f0 con 0x7f25a40047d0 2024-02-19T15:22:11.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25bd600640 1 -- 172.21.15.47:0/2405224457 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a400c1b0 con 0x7f25a40052b0 2024-02-19T15:22:11.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25bd600640 1 -- 172.21.15.47:0/2405224457 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a400c370 con 0x7f25a4003980 2024-02-19T15:22:11.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f25bf400640 1 --2- 172.21.15.47:0/2405224457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25a40052b0 0x7f25a400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f9f8f600640 1 -- 172.21.15.47:0/3108666576 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9f80055940 con 0x7f9f800ddca0 2024-02-19T15:22:11.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f0396000640 1 -- 172.21.15.47:0/4141416363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f03980cb2b0 msgr2=0x7f03981fe200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f6ef0a00640 1 --2- 172.21.15.47:0/3280151261 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee40339c0 0x7f6ee40d3390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6ee000c3e0 tx=0x7f6ee000c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f064d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0640007810 0x7f06400d8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f064d400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0640053c20 0x7f06400d9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.734+0000 7f064d400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06400defb0 0x7f06400de2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f064d400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06400da570 con 0x7f06400defb0 2024-02-19T15:22:11.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f064d400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06400da850 con 0x7f0640007810 2024-02-19T15:22:11.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f064d400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06400dab30 con 0x7f0640053c20 2024-02-19T15:22:11.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f6ee8c00640 1 -- 172.21.15.47:0/3280151261 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee00052d0 con 0x7f6ee40339c0 2024-02-19T15:22:11.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f6ee8c00640 1 -- 172.21.15.47:0/3280151261 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee0005450 con 0x7f6ee40339c0 2024-02-19T15:22:11.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f6ee8c00640 1 -- 172.21.15.47:0/3280151261 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee0003840 con 0x7f6ee40339c0 2024-02-19T15:22:11.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0396000640 1 --2- 172.21.15.47:0/4141416363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f03980cb2b0 0x7f03981fe200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0396000640 1 -- 172.21.15.47:0/4141416363 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f039806fbf0 msgr2=0x7f039806f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0396000640 1 --2- 172.21.15.47:0/4141416363 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f039806fbf0 0x7f039806f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0396000640 1 -- 172.21.15.47:0/4141416363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03880094d0 con 0x7f0398070050 2024-02-19T15:22:11.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0397400640 1 --2- 172.21.15.47:0/4141416363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f03980cb2b0 0x7f03981fe200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:11.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0647e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0640007810 0x7f06400d8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0647e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0640007810 0x7f06400d8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:45590/0 (socket says 172.21.15.47:45590) 2024-02-19T15:22:11.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0647e00640 1 -- 172.21.15.47:0/2102309375 learned_addr learned my addr 172.21.15.47:0/2102309375 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:11.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f9f84c00640 1 -- 172.21.15.47:0/3108666576 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9f7c007d00 con 0x7f9f800ddca0 2024-02-19T15:22:11.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f6ef1e00640 1 -- 172.21.15.47:0/3280151261 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee40e3ba0 con 0x7f6ee40339c0 2024-02-19T15:22:11.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f6ef1e00640 1 -- 172.21.15.47:0/3280151261 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee40e2480 con 0x7f6ee40339c0 2024-02-19T15:22:11.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0647400640 1 --2- 172.21.15.47:0/2102309375 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0640053c20 0x7f06400d9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0396000640 1 --2- 172.21.15.47:0/4141416363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0398070050 0x7f03980b29c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f038000d800 tx=0x7f038000dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f038ea00640 1 -- 172.21.15.47:0/4141416363 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03800096a0 con 0x7f0398070050 2024-02-19T15:22:11.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f038ea00640 1 -- 172.21.15.47:0/4141416363 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0380009800 con 0x7f0398070050 2024-02-19T15:22:11.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f038ea00640 1 -- 172.21.15.47:0/4141416363 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0380009b70 con 0x7f0398070050 2024-02-19T15:22:11.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0397e00640 1 -- 172.21.15.47:0/4141416363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0398147fe0 con 0x7f0398070050 2024-02-19T15:22:11.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0397e00640 1 -- 172.21.15.47:0/4141416363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03980cbb90 con 0x7f0398070050 2024-02-19T15:22:11.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f7598000640 1 -- start start 2024-02-19T15:22:11.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f941f400640 1 --2- 172.21.15.47:0/1578331960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94200cf0e0 0x7f94200b30d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f942006be20 tx=0x7f941001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e7fb294295df0d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0ec2e00640 1 Processor -- start 2024-02-19T15:22:11.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f941e000640 1 -- 172.21.15.47:0/1578331960 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94100aa610 con 0x7f94200cf0e0 2024-02-19T15:22:11.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0ec2e00640 1 -- start start 2024-02-19T15:22:11.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f25bf400640 1 -- 172.21.15.47:0/2405224457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25a4003980 msgr2=0x7f25a4003d80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-02-19T15:22:11.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f25bf400640 1 --2- 172.21.15.47:0/2405224457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25a4003980 0x7f25a4003d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f25bf400640 1 -- 172.21.15.47:0/2405224457 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25a40047d0 msgr2=0x7f25a4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:11.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f25bf400640 1 --2- 172.21.15.47:0/2405224457 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25a40047d0 0x7f25a4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f25bf400640 1 -- 172.21.15.47:0/2405224457 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c00b37d0 con 0x7f25a40052b0 2024-02-19T15:22:11.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f6ef1e00640 1 -- 172.21.15.47:0/3280151261 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6ee40053a0 con 0x7f6ee40339c0 2024-02-19T15:22:11.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f9f84c00640 1 --2- 172.21.15.47:0/3108666576 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9f68066ea0 0x7f9f68069360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f9f84c00640 1 -- 172.21.15.47:0/3108666576 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f7c084c00 con 0x7f9f800ddca0 2024-02-19T15:22:11.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f9f84c00640 1 --2- 172.21.15.47:0/3108666576 >> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] conn(0x7f9f6806f0a0 0x7f9f680714c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f9f84c00640 1 -- 172.21.15.47:0/3108666576 --> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f68071b70 con 0x7f9f6806f0a0 2024-02-19T15:22:11.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f9f84c00640 1 -- 172.21.15.47:0/3108666576 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f7c089240 con 0x7f9f800ddca0 2024-02-19T15:22:11.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f9f87e00640 1 --2- 172.21.15.47:0/3108666576 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9f68066ea0 0x7f9f68069360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f9f8d600640 1 --2- 172.21.15.47:0/3108666576 >> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] conn(0x7f9f6806f0a0 0x7f9f680714c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f7598000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75900728c0 0x7f759013e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f7598000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75900b29a0 0x7f7590151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0ec2e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eb4005ea0 0x7f0eb40dc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0ec2e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0eb4033e40 0x7f0eb4036b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0ec2e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0eb4047c20 0x7f0eb40e97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f941e000640 1 -- 172.21.15.47:0/1578331960 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94100a6040 con 0x7f94200cf0e0 2024-02-19T15:22:11.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f941e000640 1 -- 172.21.15.47:0/1578331960 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94100aa990 con 0x7f94200cf0e0 2024-02-19T15:22:11.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f7595600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75900728c0 0x7f759013e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f7595600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75900728c0 0x7f759013e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:53060/0 (socket says 172.21.15.47:53060) 2024-02-19T15:22:11.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0647e00640 1 -- 172.21.15.47:0/2102309375 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0640053c20 msgr2=0x7f06400d9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0647e00640 1 --2- 172.21.15.47:0/2102309375 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0640053c20 0x7f06400d9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0647e00640 1 -- 172.21.15.47:0/2102309375 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06400defb0 msgr2=0x7f06400de2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0647e00640 1 --2- 172.21.15.47:0/2102309375 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06400defb0 0x7f06400de2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.738+0000 7f0647e00640 1 -- 172.21.15.47:0/2102309375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063c009420 con 0x7f0640007810 2024-02-19T15:22:11.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f9426600640 1 -- 172.21.15.47:0/1578331960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94200cf0e0 msgr2=0x7f94200b30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f9426600640 1 --2- 172.21.15.47:0/1578331960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94200cf0e0 0x7f94200b30d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f942006be20 tx=0x7f941001c4f0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f6ee8c00640 1 -- 172.21.15.47:0/3280151261 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6ee00039a0 con 0x7f6ee40339c0 2024-02-19T15:22:11.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0397e00640 1 -- 172.21.15.47:0/4141416363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0370001040 con 0x7f0398070050 2024-02-19T15:22:11.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0647e00640 1 --2- 172.21.15.47:0/2102309375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0640007810 0x7f06400d8d70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0648050ba0 tx=0x7f0648068470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f9f8d600640 1 --2- 172.21.15.47:0/3108666576 >> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] conn(0x7f9f6806f0a0 0x7f9f680714c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f7595600640 1 -- 172.21.15.47:0/902184944 learned_addr learned my addr 172.21.15.47:0/902184944 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:11.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f9426600640 1 -- 172.21.15.47:0/1578331960 shutdown_connections 2024-02-19T15:22:11.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f7594c00640 1 --2- 172.21.15.47:0/902184944 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75900b29a0 0x7f7590151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0ec2e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb4036690 con 0x7f0eb4047c20 2024-02-19T15:22:11.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0ec2e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb40dda30 con 0x7f0eb4005ea0 2024-02-19T15:22:11.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0ec2e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb40dd470 con 0x7f0eb4033e40 2024-02-19T15:22:11.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f7598000640 1 --2- 172.21.15.47:0/902184944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7590151790 0x7f759014c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0ebbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eb4005ea0 0x7f0eb40dc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f7598000640 1 -- 172.21.15.47:0/902184944 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75901565c0 con 0x7f75900728c0 2024-02-19T15:22:11.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0ebbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eb4005ea0 0x7f0eb40dc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:45602/0 (socket says 172.21.15.47:45602) 2024-02-19T15:22:11.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0ebbe00640 1 -- 172.21.15.47:0/653973550 learned_addr learned my addr 172.21.15.47:0/653973550 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:11.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f7598000640 1 -- 172.21.15.47:0/902184944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7590151c20 con 0x7f7590151790 2024-02-19T15:22:11.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f7598000640 1 -- 172.21.15.47:0/902184944 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759015c7d0 con 0x7f75900b29a0 2024-02-19T15:22:11.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f9426600640 1 --2- 172.21.15.47:0/1578331960 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9420071090 0x7f942015dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f9426600640 1 --2- 172.21.15.47:0/1578331960 >> v2:172.21.15.116:3300/0 conn(0x7f94200b3610 0x7f9420070b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f9426600640 1 --2- 172.21.15.47:0/1578331960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94200cf0e0 0x7f94200b30d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f9426600640 1 -- 172.21.15.47:0/1578331960 >> 172.21.15.47:0/1578331960 conn(0x7f94200b16a0 msgr2=0x7f94200b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:11.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f25bf400640 1 -- 172.21.15.47:0/2405224457 reap_dead start 2024-02-19T15:22:11.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f9f87e00640 1 --2- 172.21.15.47:0/3108666576 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9f68066ea0 0x7f9f68069360 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9f74007fd0 tx=0x7f9f740079c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0ec0e00640 1 --2- 172.21.15.47:0/653973550 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0eb4047c20 0x7f0eb40e97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0ebbe00640 1 -- 172.21.15.47:0/653973550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0eb4033e40 msgr2=0x7f0eb4036b20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-19T15:22:11.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0ebb400640 1 --2- 172.21.15.47:0/653973550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0eb4033e40 0x7f0eb4036b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f25bf400640 1 --2- 172.21.15.47:0/2405224457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25a40052b0 0x7f25a400b990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f25b000c260 tx=0x7f25b001f3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3744974c137949f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f9426600640 1 -- 172.21.15.47:0/1578331960 shutdown_connections 2024-02-19T15:22:11.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f9426600640 1 -- 172.21.15.47:0/1578331960 wait complete. 2024-02-19T15:22:11.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f038ea00640 1 -- 172.21.15.47:0/4141416363 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0380009cd0 con 0x7f0398070050 2024-02-19T15:22:11.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f25bd600640 1 -- 172.21.15.47:0/2405224457 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b0005ae0 con 0x7f25a40052b0 2024-02-19T15:22:11.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f6ee8c00640 1 --2- 172.21.15.47:0/3280151261 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ecc066f10 0x7f6ecc0693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f9f84c00640 1 -- 172.21.15.47:0/3108666576 <== osd.2 v2:172.21.15.47:6808/2881016873 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9f68071b70 con 0x7f9f6806f0a0 2024-02-19T15:22:11.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f6ee8c00640 1 -- 172.21.15.47:0/3280151261 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ee0083fe0 con 0x7f6ee40339c0 2024-02-19T15:22:11.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f25bd600640 1 -- 172.21.15.47:0/2405224457 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b00ab070 con 0x7f25a40052b0 2024-02-19T15:22:11.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f7595600640 1 -- 172.21.15.47:0/902184944 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75900b29a0 msgr2=0x7f7590151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f25bd600640 1 -- 172.21.15.47:0/2405224457 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b0005e80 con 0x7f25a40052b0 2024-02-19T15:22:11.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f7595600640 1 --2- 172.21.15.47:0/902184944 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75900b29a0 0x7f7590151250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f7595600640 1 -- 172.21.15.47:0/902184944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7590151790 msgr2=0x7f759014c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f7595600640 1 --2- 172.21.15.47:0/902184944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7590151790 0x7f759014c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f7595600640 1 -- 172.21.15.47:0/902184944 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7588009b30 con 0x7f75900728c0 2024-02-19T15:22:11.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f7594c00640 1 --2- 172.21.15.47:0/902184944 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75900b29a0 0x7f7590151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:11.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f6ee8c00640 1 --2- 172.21.15.47:0/3280151261 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x7f6ecc06f130 0x7f6ecc071550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f6ee8c00640 1 -- 172.21.15.47:0/3280151261 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ecc071c00 con 0x7f6ecc06f130 2024-02-19T15:22:11.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f6ee8c00640 1 -- 172.21.15.47:0/3280151261 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ee0086600 con 0x7f6ee40339c0 2024-02-19T15:22:11.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f063be00640 1 -- 172.21.15.47:0/2102309375 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0648093430 con 0x7f0640007810 2024-02-19T15:22:11.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f063be00640 1 -- 172.21.15.47:0/2102309375 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0648093590 con 0x7f0640007810 2024-02-19T15:22:11.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f6eebe00640 1 --2- 172.21.15.47:0/3280151261 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ecc066f10 0x7f6ecc0693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f6ef1400640 1 --2- 172.21.15.47:0/3280151261 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x7f6ecc06f130 0x7f6ecc071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f038ea00640 1 --2- 172.21.15.47:0/4141416363 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0374066e90 0x7f0374069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f038ea00640 1 -- 172.21.15.47:0/4141416363 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0380084ff0 con 0x7f0398070050 2024-02-19T15:22:11.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f038ea00640 1 --2- 172.21.15.47:0/4141416363 >> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] conn(0x7f037406f110 0x7f0374071530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f038ea00640 1 -- 172.21.15.47:0/4141416363 --> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0374071be0 con 0x7f037406f110 2024-02-19T15:22:11.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f038ea00640 1 -- 172.21.15.47:0/4141416363 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03800875d0 con 0x7f0398070050 2024-02-19T15:22:11.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0396a00640 1 --2- 172.21.15.47:0/4141416363 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0374066e90 0x7f0374069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f7595600640 1 --2- 172.21.15.47:0/902184944 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75900728c0 0x7f759013e180 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f75880af040 tx=0x7f75880196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f063be00640 1 -- 172.21.15.47:0/2102309375 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0648093940 con 0x7f0640007810 2024-02-19T15:22:11.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f25c5600640 1 -- 172.21.15.47:0/2405224457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25a40052b0 msgr2=0x7f25a400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f25c5600640 1 --2- 172.21.15.47:0/2405224457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25a40052b0 0x7f25a400b990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f25b000c260 tx=0x7f25b001f3f0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f758d600640 1 -- 172.21.15.47:0/902184944 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75880b8670 con 0x7f75900728c0 2024-02-19T15:22:11.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f7598000640 1 -- 172.21.15.47:0/902184944 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f759014d0c0 con 0x7f75900728c0 2024-02-19T15:22:11.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f758d600640 1 -- 172.21.15.47:0/902184944 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75880b87d0 con 0x7f75900728c0 2024-02-19T15:22:11.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f7598000640 1 -- 172.21.15.47:0/902184944 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f759014d680 con 0x7f75900728c0 2024-02-19T15:22:11.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f758d600640 1 -- 172.21.15.47:0/902184944 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75880b8670 con 0x7f75900728c0 2024-02-19T15:22:11.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f9426600640 1 Processor -- start 2024-02-19T15:22:11.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0ebbe00640 1 --2- 172.21.15.47:0/653973550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0eb4033e40 0x7f0eb4036b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0ebbe00640 1 -- 172.21.15.47:0/653973550 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0eb4047c20 msgr2=0x7f0eb40e97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0ebbe00640 1 --2- 172.21.15.47:0/653973550 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0eb4047c20 0x7f0eb40e97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0ebbe00640 1 -- 172.21.15.47:0/653973550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eac009420 con 0x7f0eb4005ea0 2024-02-19T15:22:11.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0ebbe00640 1 --2- 172.21.15.47:0/653973550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eb4005ea0 0x7f0eb40dc9b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc0693d0 tx=0x7f0ebc068ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0eabe00640 1 -- 172.21.15.47:0/653973550 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc093630 con 0x7f0eb4005ea0 2024-02-19T15:22:11.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0eabe00640 1 -- 172.21.15.47:0/653973550 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ebc093790 con 0x7f0eb4005ea0 2024-02-19T15:22:11.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0eabe00640 1 -- 172.21.15.47:0/653973550 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc093b00 con 0x7f0eb4005ea0 2024-02-19T15:22:11.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0ec2e00640 1 -- 172.21.15.47:0/653973550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb40dd750 con 0x7f0eb4005ea0 2024-02-19T15:22:11.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f0ec0e00640 1 --2- 172.21.15.47:0/653973550 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0eb4047c20 0x7f0eb40e97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:11.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f0ebb400640 1 --2- 172.21.15.47:0/653973550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0eb4033e40 0x7f0eb4036b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:22:11.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.742+0000 7f0397400640 1 --2- 172.21.15.47:0/4141416363 >> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] conn(0x7f037406f110 0x7f0374071530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f0396a00640 1 --2- 172.21.15.47:0/4141416363 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0374066e90 0x7f0374069350 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f03880ad000 tx=0x7f0388004740 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f0397400640 1 --2- 172.21.15.47:0/4141416363 >> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] conn(0x7f037406f110 0x7f0374071530 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f7598000640 1 -- 172.21.15.47:0/902184944 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f756c001040 con 0x7f75900728c0 2024-02-19T15:22:11.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f064d400640 1 -- 172.21.15.47:0/2102309375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06400dae10 con 0x7f0640007810 2024-02-19T15:22:11.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f064d400640 1 -- 172.21.15.47:0/2102309375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06400d92b0 con 0x7f0640007810 2024-02-19T15:22:11.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f0ec2e00640 1 -- 172.21.15.47:0/653973550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb40dc320 con 0x7f0eb4005ea0 2024-02-19T15:22:11.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f6eebe00640 1 --2- 172.21.15.47:0/3280151261 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ecc066f10 0x7f6ecc0693d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6ed80ad000 tx=0x7f6ed80a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f064d400640 1 -- 172.21.15.47:0/2102309375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f06400da9c0 con 0x7f0640007810 2024-02-19T15:22:11.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f9426600640 1 -- start start 2024-02-19T15:22:11.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f9426600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9420071090 0x7f9420169d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f9426600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94200b3610 0x7f9420156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f9426600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94200cf0e0 0x7f9420157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f9426600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942016ac70 con 0x7f94200b3610 2024-02-19T15:22:11.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f9426600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9420156c80 con 0x7f9420071090 2024-02-19T15:22:11.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f9426600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f942016a3d0 con 0x7f94200cf0e0 2024-02-19T15:22:11.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f0ea9600640 1 -- 172.21.15.47:0/653973550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0e94001040 con 0x7f0eb4005ea0 2024-02-19T15:22:11.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f25c5600640 1 -- 172.21.15.47:0/2405224457 shutdown_connections 2024-02-19T15:22:11.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f25c5600640 1 --2- 172.21.15.47:0/2405224457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25a40052b0 0x7f25a400b990 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f25c5600640 1 -- 172.21.15.47:0/2405224457 >> 172.21.15.47:0/2405224457 conn(0x7f25c00b16a0 msgr2=0x7f25c00b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:11.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f25c5600640 1 -- 172.21.15.47:0/2405224457 shutdown_connections 2024-02-19T15:22:11.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f941f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9420071090 0x7f9420169d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f941f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9420071090 0x7f9420169d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:45616/0 (socket says 172.21.15.47:45616) 2024-02-19T15:22:11.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f941fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94200cf0e0 0x7f9420157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f941f400640 1 -- 172.21.15.47:0/1407262476 learned_addr learned my addr 172.21.15.47:0/1407262476 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:11.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f941ea00640 1 --2- 172.21.15.47:0/1407262476 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94200b3610 0x7f9420156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f941fe00640 1 -- 172.21.15.47:0/1407262476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9420071090 msgr2=0x7f9420169d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f941fe00640 1 --2- 172.21.15.47:0/1407262476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9420071090 0x7f9420169d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f941fe00640 1 -- 172.21.15.47:0/1407262476 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94200b3610 msgr2=0x7f9420156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f941fe00640 1 --2- 172.21.15.47:0/1407262476 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94200b3610 0x7f9420156740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f941fe00640 1 -- 172.21.15.47:0/1407262476 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9418005d20 con 0x7f94200cf0e0 2024-02-19T15:22:11.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f941ea00640 1 --2- 172.21.15.47:0/1407262476 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94200b3610 0x7f9420156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:22:11.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f6ef1400640 1 --2- 172.21.15.47:0/3280151261 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x7f6ecc06f130 0x7f6ecc071550 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f063be00640 1 -- 172.21.15.47:0/2102309375 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0648093aa0 con 0x7f0640007810 2024-02-19T15:22:11.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.746+0000 7f25c5600640 1 -- 172.21.15.47:0/2405224457 wait complete. 2024-02-19T15:22:11.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f941fe00640 1 --2- 172.21.15.47:0/1407262476 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94200cf0e0 0x7f9420157770 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f941800fb10 tx=0x7f941800d150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f0eabe00640 1 -- 172.21.15.47:0/653973550 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0ebc093c60 con 0x7f0eb4005ea0 2024-02-19T15:22:11.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f758d600640 1 -- 172.21.15.47:0/902184944 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f75880b8940 con 0x7f75900728c0 2024-02-19T15:22:11.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f9417400640 1 -- 172.21.15.47:0/1407262476 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941800dc20 con 0x7f94200cf0e0 2024-02-19T15:22:11.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f9417400640 1 -- 172.21.15.47:0/1407262476 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f941800dda0 con 0x7f94200cf0e0 2024-02-19T15:22:11.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f9417400640 1 -- 172.21.15.47:0/1407262476 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9418003be0 con 0x7f94200cf0e0 2024-02-19T15:22:11.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f038ea00640 1 -- 172.21.15.47:0/4141416363 <== osd.1 v2:172.21.15.47:6800/1591346632 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0374071be0 con 0x7f037406f110 2024-02-19T15:22:11.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f9426600640 1 -- 172.21.15.47:0/1407262476 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94100094d0 con 0x7f94200cf0e0 2024-02-19T15:22:11.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f9426600640 1 -- 172.21.15.47:0/1407262476 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f942016a990 con 0x7f94200cf0e0 2024-02-19T15:22:11.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f063be00640 1 --2- 172.21.15.47:0/2102309375 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0628066f40 0x7f0628069400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f0eabe00640 1 --2- 172.21.15.47:0/653973550 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0ea0066e90 0x7f0ea0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f6ee8c00640 1 -- 172.21.15.47:0/3280151261 <== osd.0 v2:172.21.15.47:6809/1968337531 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6ecc071c00 con 0x7f6ecc06f130 2024-02-19T15:22:11.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f0eabe00640 1 -- 172.21.15.47:0/653973550 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ebc0ff4a0 con 0x7f0eb4005ea0 2024-02-19T15:22:11.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f758d600640 1 --2- 172.21.15.47:0/902184944 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7570066ea0 0x7f7570069360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f758d600640 1 -- 172.21.15.47:0/902184944 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f758811fe10 con 0x7f75900728c0 2024-02-19T15:22:11.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f758d600640 1 --2- 172.21.15.47:0/902184944 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x7f757006f0c0 0x7f75700714e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f758d600640 1 -- 172.21.15.47:0/902184944 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7570071b90 con 0x7f757006f0c0 2024-02-19T15:22:11.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f758d600640 1 -- 172.21.15.47:0/902184944 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75881201c0 con 0x7f75900728c0 2024-02-19T15:22:11.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f0647400640 1 --2- 172.21.15.47:0/2102309375 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0628066f40 0x7f0628069400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f0eabe00640 1 --2- 172.21.15.47:0/653973550 >> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] conn(0x7f0ea006f0b0 0x7f0ea00714d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f0eabe00640 1 -- 172.21.15.47:0/653973550 --> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ea0071b80 con 0x7f0ea006f0b0 2024-02-19T15:22:11.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f0eabe00640 1 -- 172.21.15.47:0/653973550 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ebc0ff860 con 0x7f0eb4005ea0 2024-02-19T15:22:11.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f7594c00640 1 --2- 172.21.15.47:0/902184944 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7570066ea0 0x7f7570069360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f0ebb400640 1 --2- 172.21.15.47:0/653973550 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0ea0066e90 0x7f0ea0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f0ec0e00640 1 --2- 172.21.15.47:0/653973550 >> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] conn(0x7f0ea006f0b0 0x7f0ea00714d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f9426600640 1 -- 172.21.15.47:0/1407262476 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f94200b34a0 con 0x7f94200cf0e0 2024-02-19T15:22:11.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f7596000640 1 --2- 172.21.15.47:0/902184944 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x7f757006f0c0 0x7f75700714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f063be00640 1 -- 172.21.15.47:0/2102309375 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06480ff6f0 con 0x7f0640007810 2024-02-19T15:22:11.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f0ec0e00640 1 --2- 172.21.15.47:0/653973550 >> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] conn(0x7f0ea006f0b0 0x7f0ea00714d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.750+0000 7f0647400640 1 --2- 172.21.15.47:0/2102309375 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0628066f40 0x7f0628069400 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0634007ca0 tx=0x7f0634007bf0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f0ebb400640 1 --2- 172.21.15.47:0/653973550 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0ea0066e90 0x7f0ea0069350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4047b80 tx=0x7f0eb0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f7594c00640 1 --2- 172.21.15.47:0/902184944 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7570066ea0 0x7f7570069360 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f75901fae70 tx=0x7f757c009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f063be00640 1 --2- 172.21.15.47:0/2102309375 >> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] conn(0x7f062806f160 0x7f0628071580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f063be00640 1 -- 172.21.15.47:0/2102309375 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0628071c30 con 0x7f062806f160 2024-02-19T15:22:11.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f063be00640 1 -- 172.21.15.47:0/2102309375 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0648101d10 con 0x7f0640007810 2024-02-19T15:22:11.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f7596000640 1 --2- 172.21.15.47:0/902184944 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x7f757006f0c0 0x7f75700714e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f064ca00640 1 --2- 172.21.15.47:0/2102309375 >> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] conn(0x7f062806f160 0x7f0628071580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f25c5600640 1 Processor -- start 2024-02-19T15:22:11.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f9417400640 1 -- 172.21.15.47:0/1407262476 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9418003d40 con 0x7f94200cf0e0 2024-02-19T15:22:11.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f25c5600640 1 -- start start 2024-02-19T15:22:11.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f0eabe00640 1 -- 172.21.15.47:0/653973550 <== osd.6 v2:172.21.15.116:6824/2921645124 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0ea0071b80 con 0x7f0ea006f0b0 2024-02-19T15:22:11.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f9417400640 1 --2- 172.21.15.47:0/1407262476 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9400066e90 0x7f9400069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f9417400640 1 -- 172.21.15.47:0/1407262476 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9418085020 con 0x7f94200cf0e0 2024-02-19T15:22:11.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f758d600640 1 -- 172.21.15.47:0/902184944 <== osd.3 v2:172.21.15.47:6824/4149334728 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7570071b90 con 0x7f757006f0c0 2024-02-19T15:22:11.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f25c5600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25a40052b0 0x7f25c00a96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f25c5600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25c0155a30 0x7f25c00ae920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f25c5600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25c0155280 0x7f25c00abd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f25c5600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c00cd410 con 0x7f25c0155a30 2024-02-19T15:22:11.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f25c5600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c00aee60 con 0x7f25c0155280 2024-02-19T15:22:11.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f25c5600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c00af100 con 0x7f25a40052b0 2024-02-19T15:22:11.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f9417400640 1 --2- 172.21.15.47:0/1407262476 >> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] conn(0x7f940006f110 0x7f9400071530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f9417400640 1 -- 172.21.15.47:0/1407262476 --> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9400071be0 con 0x7f940006f110 2024-02-19T15:22:11.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f9417400640 1 -- 172.21.15.47:0/1407262476 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9418087730 con 0x7f94200cf0e0 2024-02-19T15:22:11.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f064ca00640 1 --2- 172.21.15.47:0/2102309375 >> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] conn(0x7f062806f160 0x7f0628071580 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f25bea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25a40052b0 0x7f25c00a96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f25be000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25c0155a30 0x7f25c00ae920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f25bea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25a40052b0 0x7f25c00a96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:50788/0 (socket says 172.21.15.47:50788) 2024-02-19T15:22:11.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f25bea00640 1 -- 172.21.15.47:0/535210725 learned_addr learned my addr 172.21.15.47:0/535210725 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:11.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f941ea00640 1 --2- 172.21.15.47:0/1407262476 >> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] conn(0x7f940006f110 0x7f9400071530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f941f400640 1 --2- 172.21.15.47:0/1407262476 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9400066e90 0x7f9400069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f25bf400640 1 --2- 172.21.15.47:0/535210725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25c0155280 0x7f25c00abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f25be000640 1 -- 172.21.15.47:0/535210725 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25a40052b0 msgr2=0x7f25c00a96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f25be000640 1 --2- 172.21.15.47:0/535210725 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25a40052b0 0x7f25c00a96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f25be000640 1 -- 172.21.15.47:0/535210725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25c0155280 msgr2=0x7f25c00abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f25be000640 1 --2- 172.21.15.47:0/535210725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25c0155280 0x7f25c00abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.754+0000 7f25be000640 1 -- 172.21.15.47:0/535210725 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c00cd620 con 0x7f25c0155a30 2024-02-19T15:22:11.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.758+0000 7f941ea00640 1 --2- 172.21.15.47:0/1407262476 >> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] conn(0x7f940006f110 0x7f9400071530 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.758+0000 7f25bf400640 1 --2- 172.21.15.47:0/535210725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25c0155280 0x7f25c00abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:11.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.758+0000 7f941f400640 1 --2- 172.21.15.47:0/1407262476 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9400066e90 0x7f9400069350 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9410000c10 tx=0x7f94100b5040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.758+0000 7f063be00640 1 -- 172.21.15.47:0/2102309375 <== osd.4 v2:172.21.15.116:6809/3366344088 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0628071c30 con 0x7f062806f160 2024-02-19T15:22:11.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.758+0000 7f25be000640 1 --2- 172.21.15.47:0/535210725 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25c0155a30 0x7f25c00ae920 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f25ac00b3e0 tx=0x7f25ac009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.758+0000 7f25b6a00640 1 -- 172.21.15.47:0/535210725 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac011830 con 0x7f25c0155a30 2024-02-19T15:22:11.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.758+0000 7f25b6a00640 1 -- 172.21.15.47:0/535210725 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ac011990 con 0x7f25c0155a30 2024-02-19T15:22:11.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.758+0000 7f25b6a00640 1 -- 172.21.15.47:0/535210725 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac011d20 con 0x7f25c0155a30 2024-02-19T15:22:11.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.758+0000 7f25c5600640 1 -- 172.21.15.47:0/535210725 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c00cd900 con 0x7f25c0155a30 2024-02-19T15:22:11.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.758+0000 7f25c5600640 1 -- 172.21.15.47:0/535210725 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c0155700 con 0x7f25c0155a30 2024-02-19T15:22:11.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.758+0000 7f9417400640 1 -- 172.21.15.47:0/1407262476 <== osd.5 v2:172.21.15.116:6808/3478542733 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9400071be0 con 0x7f940006f110 2024-02-19T15:22:11.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.766+0000 7f25c5600640 1 -- 172.21.15.47:0/535210725 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f25c01628c0 con 0x7f25c0155a30 2024-02-19T15:22:11.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.766+0000 7f25b6a00640 1 -- 172.21.15.47:0/535210725 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f25ac01a400 con 0x7f25c0155a30 2024-02-19T15:22:11.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.766+0000 7f25b6a00640 1 --2- 172.21.15.47:0/535210725 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f25a0066e40 0x7f25a0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.766+0000 7f25b6a00640 1 -- 172.21.15.47:0/535210725 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25ac055cb0 con 0x7f25c0155a30 2024-02-19T15:22:11.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.766+0000 7f25b6a00640 1 --2- 172.21.15.47:0/535210725 >> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] conn(0x7f25a006f040 0x7f25a0071460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.766+0000 7f25b6a00640 1 -- 172.21.15.47:0/535210725 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f25a0071b10 con 0x7f25a006f040 2024-02-19T15:22:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.766+0000 7f25b6a00640 1 -- 172.21.15.47:0/535210725 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f25ac011e80 con 0x7f25c0155a30 2024-02-19T15:22:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.766+0000 7f25bea00640 1 --2- 172.21.15.47:0/535210725 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f25a0066e40 0x7f25a0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.766+0000 7f25bf400640 1 --2- 172.21.15.47:0/535210725 >> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] conn(0x7f25a006f040 0x7f25a0071460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.770+0000 7f25bea00640 1 --2- 172.21.15.47:0/535210725 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f25a0066e40 0x7f25a0069300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f25b8001f50 tx=0x7f25b8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.770+0000 7f25bf400640 1 --2- 172.21.15.47:0/535210725 >> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] conn(0x7f25a006f040 0x7f25a0071460 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.770+0000 7f25b6a00640 1 -- 172.21.15.47:0/535210725 <== osd.7 v2:172.21.15.116:6800/3212743377 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f25a0071b10 con 0x7f25a006f040 2024-02-19T15:22:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.770+0000 7f9f8f600640 1 -- 172.21.15.47:0/3108666576 --> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9f800e0df0 con 0x7f9f6806f0a0 2024-02-19T15:22:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.774+0000 7f9f84c00640 1 -- 172.21.15.47:0/3108666576 <== osd.2 v2:172.21.15.47:6808/2881016873 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9f800e0df0 con 0x7f9f6806f0a0 2024-02-19T15:22:11.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.778+0000 7f0397e00640 1 -- 172.21.15.47:0/4141416363 --> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0370002d40 con 0x7f037406f110 2024-02-19T15:22:11.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.778+0000 7f9f8f600640 1 -- 172.21.15.47:0/3108666576 >> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] conn(0x7f9f6806f0a0 msgr2=0x7f9f680714c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.778+0000 7f9f8f600640 1 --2- 172.21.15.47:0/3108666576 >> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] conn(0x7f9f6806f0a0 0x7f9f680714c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.778+0000 7f9f8f600640 1 -- 172.21.15.47:0/3108666576 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9f68066ea0 msgr2=0x7f9f68069360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.778+0000 7f9f8f600640 1 --2- 172.21.15.47:0/3108666576 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9f68066ea0 0x7f9f68069360 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9f74007fd0 tx=0x7f9f740079c0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.778+0000 7f9f8f600640 1 -- 172.21.15.47:0/3108666576 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f800ddca0 msgr2=0x7f9f800dd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.778+0000 7f9f8f600640 1 --2- 172.21.15.47:0/3108666576 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f800ddca0 0x7f9f800dd370 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c00c7c0 tx=0x7f9f7c00cc80 comp rx=0 tx=0).stop 2024-02-19T15:22:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.778+0000 7f9f8d600640 1 -- 172.21.15.47:0/3108666576 reap_dead start 2024-02-19T15:22:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.782+0000 7f9f8f600640 1 -- 172.21.15.47:0/3108666576 shutdown_connections 2024-02-19T15:22:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.782+0000 7f9f8f600640 1 -- 172.21.15.47:0/3108666576 >> 172.21.15.47:0/3108666576 conn(0x7f9f80055be0 msgr2=0x7f9f800df620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.782+0000 7f9f8f600640 1 -- 172.21.15.47:0/3108666576 shutdown_connections 2024-02-19T15:22:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.782+0000 7f038ea00640 1 -- 172.21.15.47:0/4141416363 <== osd.1 v2:172.21.15.47:6800/1591346632 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0370002d40 con 0x7f037406f110 2024-02-19T15:22:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.782+0000 7f9f8f600640 1 -- 172.21.15.47:0/3108666576 wait complete. 2024-02-19T15:22:11.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.782+0000 7f0397e00640 1 -- 172.21.15.47:0/4141416363 >> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] conn(0x7f037406f110 msgr2=0x7f0374071530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.782+0000 7f0397e00640 1 --2- 172.21.15.47:0/4141416363 >> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] conn(0x7f037406f110 0x7f0374071530 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.782+0000 7f0397e00640 1 -- 172.21.15.47:0/4141416363 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0374066e90 msgr2=0x7f0374069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.782+0000 7f0397e00640 1 --2- 172.21.15.47:0/4141416363 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0374066e90 0x7f0374069350 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f03880ad000 tx=0x7f0388004740 comp rx=0 tx=0).stop 2024-02-19T15:22:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.782+0000 7f0397e00640 1 -- 172.21.15.47:0/4141416363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0398070050 msgr2=0x7f03980b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.782+0000 7f0397e00640 1 --2- 172.21.15.47:0/4141416363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0398070050 0x7f03980b29c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f038000d800 tx=0x7f038000dcc0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.782+0000 7f0397400640 1 -- 172.21.15.47:0/4141416363 reap_dead start 2024-02-19T15:22:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.782+0000 7f0397e00640 1 -- 172.21.15.47:0/4141416363 shutdown_connections 2024-02-19T15:22:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.782+0000 7f0397e00640 1 -- 172.21.15.47:0/4141416363 >> 172.21.15.47:0/4141416363 conn(0x7f039809a9d0 msgr2=0x7f03980d1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.782+0000 7f0397e00640 1 -- 172.21.15.47:0/4141416363 shutdown_connections 2024-02-19T15:22:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.782+0000 7f0397e00640 1 -- 172.21.15.47:0/4141416363 wait complete. 2024-02-19T15:22:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.782+0000 7f6ef1e00640 1 -- 172.21.15.47:0/3280151261 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6ee40e1bf0 con 0x7f6ecc06f130 2024-02-19T15:22:11.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f6ee8c00640 1 -- 172.21.15.47:0/3280151261 <== osd.0 v2:172.21.15.47:6809/1968337531 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6ee40e1bf0 con 0x7f6ecc06f130 2024-02-19T15:22:11.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f0ea9600640 1 -- 172.21.15.47:0/653973550 --> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0e94002d40 con 0x7f0ea006f0b0 2024-02-19T15:22:11.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f0eabe00640 1 -- 172.21.15.47:0/653973550 <== osd.6 v2:172.21.15.116:6824/2921645124 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0e94002d40 con 0x7f0ea006f0b0 2024-02-19T15:22:11.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f0ea9600640 1 -- 172.21.15.47:0/653973550 >> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] conn(0x7f0ea006f0b0 msgr2=0x7f0ea00714d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f0ea9600640 1 --2- 172.21.15.47:0/653973550 >> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] conn(0x7f0ea006f0b0 0x7f0ea00714d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f6ef1e00640 1 -- 172.21.15.47:0/3280151261 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x7f6ecc06f130 msgr2=0x7f6ecc071550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f0ea9600640 1 -- 172.21.15.47:0/653973550 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0ea0066e90 msgr2=0x7f0ea0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f0ea9600640 1 --2- 172.21.15.47:0/653973550 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0ea0066e90 0x7f0ea0069350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4047b80 tx=0x7f0eb0009040 comp rx=0 tx=0).stop 2024-02-19T15:22:11.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f6ef1e00640 1 --2- 172.21.15.47:0/3280151261 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x7f6ecc06f130 0x7f6ecc071550 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f6ef1e00640 1 -- 172.21.15.47:0/3280151261 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ecc066f10 msgr2=0x7f6ecc0693d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f6ef1e00640 1 --2- 172.21.15.47:0/3280151261 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ecc066f10 0x7f6ecc0693d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6ed80ad000 tx=0x7f6ed80a6000 comp rx=0 tx=0).stop 2024-02-19T15:22:11.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f6ef1e00640 1 -- 172.21.15.47:0/3280151261 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee40339c0 msgr2=0x7f6ee40d3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f6ef1e00640 1 --2- 172.21.15.47:0/3280151261 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee40339c0 0x7f6ee40d3390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6ee000c3e0 tx=0x7f6ee000c8a0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f6ef1400640 1 -- 172.21.15.47:0/3280151261 reap_dead start 2024-02-19T15:22:11.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f6ef1e00640 1 -- 172.21.15.47:0/3280151261 shutdown_connections 2024-02-19T15:22:11.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f6ef1e00640 1 -- 172.21.15.47:0/3280151261 >> 172.21.15.47:0/3280151261 conn(0x7f6ee4009fc0 msgr2=0x7f6ee40e4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:11.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f7598000640 1 -- 172.21.15.47:0/902184944 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f756c002d40 con 0x7f757006f0c0 2024-02-19T15:22:11.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f0ea9600640 1 -- 172.21.15.47:0/653973550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eb4005ea0 msgr2=0x7f0eb40dc9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f0ea9600640 1 --2- 172.21.15.47:0/653973550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eb4005ea0 0x7f0eb40dc9b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc0693d0 tx=0x7f0ebc068ac0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f0ec0e00640 1 -- 172.21.15.47:0/653973550 reap_dead start 2024-02-19T15:22:11.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f0ea9600640 1 -- 172.21.15.47:0/653973550 shutdown_connections 2024-02-19T15:22:11.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f0ea9600640 1 -- 172.21.15.47:0/653973550 >> 172.21.15.47:0/653973550 conn(0x7f0eb4009fc0 msgr2=0x7f0eb4052f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:11.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f6ef1e00640 1 -- 172.21.15.47:0/3280151261 shutdown_connections 2024-02-19T15:22:11.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f0ea9600640 1 -- 172.21.15.47:0/653973550 shutdown_connections 2024-02-19T15:22:11.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f6ef1e00640 1 -- 172.21.15.47:0/3280151261 wait complete. 2024-02-19T15:22:11.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f0ea9600640 1 -- 172.21.15.47:0/653973550 wait complete. 2024-02-19T15:22:11.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f758d600640 1 -- 172.21.15.47:0/902184944 <== osd.3 v2:172.21.15.47:6824/4149334728 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f756c002d40 con 0x7f757006f0c0 2024-02-19T15:22:11.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.786+0000 7f9426600640 1 -- 172.21.15.47:0/1407262476 --> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f942006cbe0 con 0x7f940006f110 2024-02-19T15:22:11.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f7598000640 1 -- 172.21.15.47:0/902184944 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x7f757006f0c0 msgr2=0x7f75700714e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f7598000640 1 --2- 172.21.15.47:0/902184944 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x7f757006f0c0 0x7f75700714e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f9417400640 1 -- 172.21.15.47:0/1407262476 <== osd.5 v2:172.21.15.116:6808/3478542733 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f940c00a850 con 0x7f940006f110 2024-02-19T15:22:11.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f7598000640 1 -- 172.21.15.47:0/902184944 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7570066ea0 msgr2=0x7f7570069360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f7598000640 1 --2- 172.21.15.47:0/902184944 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7570066ea0 0x7f7570069360 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f75901fae70 tx=0x7f757c009060 comp rx=0 tx=0).stop 2024-02-19T15:22:11.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f7598000640 1 -- 172.21.15.47:0/902184944 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75900728c0 msgr2=0x7f759013e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f7598000640 1 --2- 172.21.15.47:0/902184944 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75900728c0 0x7f759013e180 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f75880af040 tx=0x7f75880196d0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f7596000640 1 -- 172.21.15.47:0/902184944 reap_dead start 2024-02-19T15:22:11.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f7598000640 1 -- 172.21.15.47:0/902184944 shutdown_connections 2024-02-19T15:22:11.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f7598000640 1 -- 172.21.15.47:0/902184944 >> 172.21.15.47:0/902184944 conn(0x7f75900b16a0 msgr2=0x7f75900ad840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:11.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f7598000640 1 -- 172.21.15.47:0/902184944 shutdown_connections 2024-02-19T15:22:11.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f7598000640 1 -- 172.21.15.47:0/902184944 wait complete. 2024-02-19T15:22:11.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f9426600640 1 -- 172.21.15.47:0/1407262476 >> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] conn(0x7f940006f110 msgr2=0x7f9400071530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f9426600640 1 --2- 172.21.15.47:0/1407262476 >> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] conn(0x7f940006f110 0x7f9400071530 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f9426600640 1 -- 172.21.15.47:0/1407262476 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9400066e90 msgr2=0x7f9400069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f9426600640 1 --2- 172.21.15.47:0/1407262476 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9400066e90 0x7f9400069350 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9410000c10 tx=0x7f94100b5040 comp rx=0 tx=0).stop 2024-02-19T15:22:11.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f9426600640 1 -- 172.21.15.47:0/1407262476 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94200cf0e0 msgr2=0x7f9420157770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f9426600640 1 --2- 172.21.15.47:0/1407262476 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94200cf0e0 0x7f9420157770 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f941800fb10 tx=0x7f941800d150 comp rx=0 tx=0).stop 2024-02-19T15:22:11.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f941fe00640 1 -- 172.21.15.47:0/1407262476 reap_dead start 2024-02-19T15:22:11.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f9426600640 1 -- 172.21.15.47:0/1407262476 shutdown_connections 2024-02-19T15:22:11.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f9426600640 1 -- 172.21.15.47:0/1407262476 >> 172.21.15.47:0/1407262476 conn(0x7f94200b16a0 msgr2=0x7f94200af6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:11.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f9426600640 1 -- 172.21.15.47:0/1407262476 shutdown_connections 2024-02-19T15:22:11.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f9426600640 1 -- 172.21.15.47:0/1407262476 wait complete. 2024-02-19T15:22:11.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f064d400640 1 -- 172.21.15.47:0/2102309375 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f064003ed90 con 0x7f062806f160 2024-02-19T15:22:11.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.790+0000 7f063be00640 1 -- 172.21.15.47:0/2102309375 <== osd.4 v2:172.21.15.116:6809/3366344088 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f063c004040 con 0x7f062806f160 2024-02-19T15:22:11.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.794+0000 7f064d400640 1 -- 172.21.15.47:0/2102309375 >> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] conn(0x7f062806f160 msgr2=0x7f0628071580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.794+0000 7f064d400640 1 --2- 172.21.15.47:0/2102309375 >> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] conn(0x7f062806f160 0x7f0628071580 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.798+0000 7f064d400640 1 -- 172.21.15.47:0/2102309375 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0628066f40 msgr2=0x7f0628069400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.798+0000 7f064d400640 1 --2- 172.21.15.47:0/2102309375 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0628066f40 0x7f0628069400 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0634007ca0 tx=0x7f0634007bf0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.798+0000 7f064d400640 1 -- 172.21.15.47:0/2102309375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0640007810 msgr2=0x7f06400d8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.798+0000 7f064d400640 1 --2- 172.21.15.47:0/2102309375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0640007810 0x7f06400d8d70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0648050ba0 tx=0x7f0648068470 comp rx=0 tx=0).stop 2024-02-19T15:22:11.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.798+0000 7f064ca00640 1 -- 172.21.15.47:0/2102309375 reap_dead start 2024-02-19T15:22:11.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.798+0000 7f064d400640 1 -- 172.21.15.47:0/2102309375 shutdown_connections 2024-02-19T15:22:11.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.798+0000 7f064d400640 1 -- 172.21.15.47:0/2102309375 >> 172.21.15.47:0/2102309375 conn(0x7f06400559f0 msgr2=0x7f06400e0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:11.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.802+0000 7f064d400640 1 -- 172.21.15.47:0/2102309375 shutdown_connections 2024-02-19T15:22:11.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.802+0000 7f25c5600640 1 -- 172.21.15.47:0/535210725 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f25c00b4290 con 0x7f25a006f040 2024-02-19T15:22:11.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.802+0000 7f064d400640 1 -- 172.21.15.47:0/2102309375 wait complete. 2024-02-19T15:22:11.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.802+0000 7f25b6a00640 1 -- 172.21.15.47:0/535210725 <== osd.7 v2:172.21.15.116:6800/3212743377 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f25c00b4290 con 0x7f25a006f040 2024-02-19T15:22:11.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.802+0000 7f25c5600640 1 -- 172.21.15.47:0/535210725 >> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] conn(0x7f25a006f040 msgr2=0x7f25a0071460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.802+0000 7f25c5600640 1 --2- 172.21.15.47:0/535210725 >> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] conn(0x7f25a006f040 0x7f25a0071460 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:11.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.806+0000 7f25c5600640 1 -- 172.21.15.47:0/535210725 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f25a0066e40 msgr2=0x7f25a0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.806+0000 7f25c5600640 1 --2- 172.21.15.47:0/535210725 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f25a0066e40 0x7f25a0069300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f25b8001f50 tx=0x7f25b8009040 comp rx=0 tx=0).stop 2024-02-19T15:22:11.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.806+0000 7f25c5600640 1 -- 172.21.15.47:0/535210725 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25c0155a30 msgr2=0x7f25c00ae920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:11.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.806+0000 7f25c5600640 1 --2- 172.21.15.47:0/535210725 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25c0155a30 0x7f25c00ae920 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f25ac00b3e0 tx=0x7f25ac009f90 comp rx=0 tx=0).stop 2024-02-19T15:22:11.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.806+0000 7f25bf400640 1 -- 172.21.15.47:0/535210725 reap_dead start 2024-02-19T15:22:11.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.806+0000 7f25c5600640 1 -- 172.21.15.47:0/535210725 shutdown_connections 2024-02-19T15:22:11.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.806+0000 7f25c5600640 1 -- 172.21.15.47:0/535210725 >> 172.21.15.47:0/535210725 conn(0x7f25c00b16a0 msgr2=0x7f25c00af5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:11.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.806+0000 7f25c5600640 1 -- 172.21.15.47:0/535210725 shutdown_connections 2024-02-19T15:22:11.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:11.806+0000 7f25c5600640 1 -- 172.21.15.47:0/535210725 wait complete. 2024-02-19T15:22:11.824 INFO:teuthology.orchestra.run.smithi047.stdout:60129542152 2024-02-19T15:22:11.824 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-02-19T15:22:11.827 INFO:teuthology.orchestra.run.smithi047.stdout:60129542151 2024-02-19T15:22:11.827 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-02-19T15:22:11.828 INFO:teuthology.orchestra.run.smithi047.stdout:60129542150 2024-02-19T15:22:11.828 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-02-19T15:22:11.832 INFO:teuthology.orchestra.run.smithi047.stdout:64424509446 2024-02-19T15:22:11.832 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-02-19T15:22:11.838 INFO:teuthology.orchestra.run.smithi047.stdout:60129542150 2024-02-19T15:22:11.838 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-02-19T15:22:11.842 INFO:teuthology.orchestra.run.smithi047.stdout:60129542150 2024-02-19T15:22:11.843 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-02-19T15:22:11.846 INFO:teuthology.orchestra.run.smithi047.stdout:60129542152 2024-02-19T15:22:11.846 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-02-19T15:22:11.849 INFO:teuthology.orchestra.run.smithi047.stdout:60129542151 2024-02-19T15:22:11.849 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-02-19T15:22:12.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.066+0000 7ffbc5a00640 1 Processor -- start 2024-02-19T15:22:12.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.066+0000 7ffbc5a00640 1 -- start start 2024-02-19T15:22:12.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.066+0000 7ffbc5a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffbc006aeb0 0x7ffbc00b3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.066+0000 7ffbc5a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffbc0072590 0x7ffbc006eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.066+0000 7ffbc5a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc006da80 con 0x7ffbc015bf50 2024-02-19T15:22:12.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.066+0000 7ffbc5a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbc006dd60 con 0x7ffbc0072590 2024-02-19T15:22:12.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.066+0000 7ffbc5a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc006e040 con 0x7ffbc006aeb0 2024-02-19T15:22:12.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.066+0000 7ffbbf400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffbc0072590 0x7ffbc006eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.066+0000 7ffbbf400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffbc0072590 0x7ffbc006eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:45640/0 (socket says 172.21.15.47:45640) 2024-02-19T15:22:12.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.066+0000 7ffbbf400640 1 -- 172.21.15.47:0/2612374115 learned_addr learned my addr 172.21.15.47:0/2612374115 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:12.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.066+0000 7ffbbea00640 1 --2- 172.21.15.47:0/2612374115 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffbc006aeb0 0x7ffbc00b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.070+0000 7ffbbf400640 1 -- 172.21.15.47:0/2612374115 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffbc006aeb0 msgr2=0x7ffbc00b3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:12.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.070+0000 7ffbbf400640 1 --2- 172.21.15.47:0/2612374115 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffbc006aeb0 0x7ffbc00b3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.070+0000 7ffbbd600640 1 -- 172.21.15.47:0/2612374115 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274369222 0 0) 0x7ffbc006da80 con 0x7ffbc015bf50 2024-02-19T15:22:12.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.070+0000 7ffbbf400640 1 -- 172.21.15.47:0/2612374115 >> v1:172.21.15.116:6789/0 conn(0x7ffbc015bf50 legacy=0x7ffbc006b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.070+0000 7ffbbf400640 1 -- 172.21.15.47:0/2612374115 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc006e320 con 0x7ffbc0072590 2024-02-19T15:22:12.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.074+0000 7ffbbf400640 1 --2- 172.21.15.47:0/2612374115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbc0072590 0x7ffbc006eb20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009e50 tx=0x7ffbb801ca00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b151fb7d35666c35 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.074+0000 7ffbbd600640 1 -- 172.21.15.47:0/2612374115 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb80a58f0 con 0x7ffbc0072590 2024-02-19T15:22:12.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.074+0000 7ffbbd600640 1 -- 172.21.15.47:0/2612374115 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb80a5a50 con 0x7ffbc0072590 2024-02-19T15:22:12.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.074+0000 7ffbbd600640 1 -- 172.21.15.47:0/2612374115 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb80a5de0 con 0x7ffbc0072590 2024-02-19T15:22:12.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.074+0000 7ffbc5a00640 1 -- 172.21.15.47:0/2612374115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbc0072590 msgr2=0x7ffbc006eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.074+0000 7ffbc5a00640 1 --2- 172.21.15.47:0/2612374115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbc0072590 0x7ffbc006eb20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009e50 tx=0x7ffbb801ca00 comp rx=0 tx=0).stop 2024-02-19T15:22:12.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.074+0000 7ffbc5a00640 1 -- 172.21.15.47:0/2612374115 shutdown_connections 2024-02-19T15:22:12.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.074+0000 7ffbc5a00640 1 --2- 172.21.15.47:0/2612374115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbc0072590 0x7ffbc006eb20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.074+0000 7ffbc5a00640 1 --2- 172.21.15.47:0/2612374115 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffbc006aeb0 0x7ffbc00b3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.074+0000 7ffbc5a00640 1 -- 172.21.15.47:0/2612374115 >> 172.21.15.47:0/2612374115 conn(0x7ffbc00b2560 msgr2=0x7ffbc00b2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:12.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.074+0000 7ffbc5a00640 1 -- 172.21.15.47:0/2612374115 shutdown_connections 2024-02-19T15:22:12.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.074+0000 7ffbc5a00640 1 -- 172.21.15.47:0/2612374115 wait complete. 2024-02-19T15:22:12.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.078+0000 7ffbc5a00640 1 Processor -- start 2024-02-19T15:22:12.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.078+0000 7ffbc5a00640 1 -- start start 2024-02-19T15:22:12.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.078+0000 7ffbc5a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbc006aeb0 0x7ffbc00c4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.078+0000 7ffbc5a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffbc0072590 0x7ffbc014ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.078+0000 7ffbc5a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffbc015bf50 0x7ffbc00cc670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.078+0000 7ffbc5a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc00c7140 con 0x7ffbc0072590 2024-02-19T15:22:12.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.078+0000 7ffbc5a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc015b2f0 con 0x7ffbc006aeb0 2024-02-19T15:22:12.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.078+0000 7ffbc5a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc015b5d0 con 0x7ffbc015bf50 2024-02-19T15:22:12.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7f54d6200640 1 Processor -- start 2024-02-19T15:22:12.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7f54d6200640 1 -- start start 2024-02-19T15:22:12.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7ffbbe000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffbc0072590 0x7ffbc014ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7ffbbe000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffbc0072590 0x7ffbc014ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:53096/0 (socket says 172.21.15.47:53096) 2024-02-19T15:22:12.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7ffbbe000640 1 -- 172.21.15.47:0/601435597 learned_addr learned my addr 172.21.15.47:0/601435597 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:12.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7ffbbe000640 1 -- 172.21.15.47:0/601435597 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffbc015bf50 msgr2=0x7ffbc00cc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7ffbbea00640 1 --2- 172.21.15.47:0/601435597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbc006aeb0 0x7ffbc00c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7ffbbf400640 1 --2- 172.21.15.47:0/601435597 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffbc015bf50 0x7ffbc00cc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7ffbbe000640 1 --2- 172.21.15.47:0/601435597 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffbc015bf50 0x7ffbc00cc670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7ffbbe000640 1 -- 172.21.15.47:0/601435597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbc006aeb0 msgr2=0x7ffbc00c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7ffbbe000640 1 --2- 172.21.15.47:0/601435597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbc006aeb0 0x7ffbc00c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7ffbbe000640 1 -- 172.21.15.47:0/601435597 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb8009b30 con 0x7ffbc0072590 2024-02-19T15:22:12.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7ffbbea00640 1 --2- 172.21.15.47:0/601435597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbc006aeb0 0x7ffbc00c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:22:12.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7ffbbe000640 1 --2- 172.21.15.47:0/601435597 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffbc0072590 0x7ffbc014ec30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0001ec0 tx=0x7ffbb000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7ffbb6a00640 1 -- 172.21.15.47:0/601435597 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb00054e0 con 0x7ffbc0072590 2024-02-19T15:22:12.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7ffbb6a00640 1 -- 172.21.15.47:0/601435597 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb0003600 con 0x7ffbc0072590 2024-02-19T15:22:12.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7ffbb6a00640 1 -- 172.21.15.47:0/601435597 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb00054e0 con 0x7ffbc0072590 2024-02-19T15:22:12.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7f54d6200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f54d0157c00 0x7f54d0072680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7ffbc5a00640 1 -- 172.21.15.47:0/601435597 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc014f170 con 0x7ffbc0072590 2024-02-19T15:22:12.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7ffbc5a00640 1 -- 172.21.15.47:0/601435597 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc00c6e70 con 0x7ffbc0072590 2024-02-19T15:22:12.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f54d6200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54d00cfbc0 0x7f54d00cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f54d6200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d00d0530 con 0x7f54d00ce8b0 2024-02-19T15:22:12.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f54d6200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54d00ce4c0 con 0x7f54d0157c00 2024-02-19T15:22:12.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f54d6200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d00ced70 con 0x7f54d00cfbc0 2024-02-19T15:22:12.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f54cfe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54d00cfbc0 0x7f54d00cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f54cfe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54d00cfbc0 0x7f54d00cfff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:50808/0 (socket says 172.21.15.47:50808) 2024-02-19T15:22:12.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f54cfe00640 1 -- 172.21.15.47:0/3523131288 learned_addr learned my addr 172.21.15.47:0/3523131288 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:12.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f54cf400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f54d0157c00 0x7f54d0072680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7ffbc5a00640 1 -- 172.21.15.47:0/601435597 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc015b7f0 con 0x7ffbc0072590 2024-02-19T15:22:12.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f54cfe00640 1 -- 172.21.15.47:0/3523131288 >> v2:172.21.15.116:3300/0 conn(0x7f54d0157c00 msgr2=0x7f54d0072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:12.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f54cfe00640 1 --2- 172.21.15.47:0/3523131288 >> v2:172.21.15.116:3300/0 conn(0x7f54d0157c00 0x7f54d0072680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f54cfe00640 1 -- 172.21.15.47:0/3523131288 >> v1:172.21.15.47:6789/0 conn(0x7f54d00ce8b0 legacy=0x7f54d015aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:12.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f54cfe00640 1 -- 172.21.15.47:0/3523131288 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d0071ba0 con 0x7f54d00cfbc0 2024-02-19T15:22:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7f8e4f200640 1 Processor -- start 2024-02-19T15:22:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.082+0000 7f8e4f200640 1 -- start start 2024-02-19T15:22:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e4f200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8e48166680 0x7f8e480b6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e47400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8e480d3460 0x7f8e48165f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:44044/0 (socket says 172.21.15.47:44044) 2024-02-19T15:22:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e47400640 1 -- 172.21.15.47:0/1241178948 learned_addr learned my addr 172.21.15.47:0/1241178948 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e4f200640 1 -- 172.21.15.47:0/1241178948 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4809bcd0 con 0x7f8e480d3460 2024-02-19T15:22:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e4f200640 1 -- 172.21.15.47:0/1241178948 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4809bfb0 con 0x7f8e48072ef0 2024-02-19T15:22:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e4d200640 1 --2- 172.21.15.47:0/1241178948 >> v2:172.21.15.116:3300/0 conn(0x7f8e48166680 0x7f8e480b6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e4f200640 1 -- 172.21.15.47:0/1241178948 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e4809c290 con 0x7f8e48166680 2024-02-19T15:22:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e46a00640 1 -- 172.21.15.47:0/1241178948 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191448515 0 0) 0x7f8e4809bfb0 con 0x7f8e48072ef0 2024-02-19T15:22:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e46a00640 1 -- 172.21.15.47:0/1241178948 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e2c003530 con 0x7f8e48072ef0 2024-02-19T15:22:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e46a00640 1 -- 172.21.15.47:0/1241178948 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3457079844 0 0) 0x7f8e2c003530 con 0x7f8e48072ef0 2024-02-19T15:22:12.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e46a00640 1 -- 172.21.15.47:0/1241178948 >> v2:172.21.15.116:3300/0 conn(0x7f8e48166680 msgr2=0x7f8e480b6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:12.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e46a00640 1 --2- 172.21.15.47:0/1241178948 >> v2:172.21.15.116:3300/0 conn(0x7f8e48166680 0x7f8e480b6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e46a00640 1 -- 172.21.15.47:0/1241178948 >> v1:172.21.15.47:6789/0 conn(0x7f8e480d3460 legacy=0x7f8e48165f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e46a00640 1 -- 172.21.15.47:0/1241178948 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4809c570 con 0x7f8e48072ef0 2024-02-19T15:22:12.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e46a00640 1 -- 172.21.15.47:0/1241178948 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f8e38001680 con 0x7f8e48072ef0 2024-02-19T15:22:12.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e46a00640 1 -- 172.21.15.47:0/1241178948 >> v1:172.21.15.116:6789/0 conn(0x7f8e48072ef0 legacy=0x7f8e4809d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e46a00640 1 --2- 172.21.15.47:0/1241178948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e2c0039d0 0x7f8e2c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e46a00640 1 --2- 172.21.15.47:0/1241178948 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e2c0047d0 0x7f8e2c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e46a00640 1 --2- 172.21.15.47:0/1241178948 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8e2c0052b0 0x7f8e2c02d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e46a00640 1 -- 172.21.15.47:0/1241178948 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4809c290 con 0x7f8e2c0052b0 2024-02-19T15:22:12.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e46a00640 1 -- 172.21.15.47:0/1241178948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c02e1e0 con 0x7f8e2c0039d0 2024-02-19T15:22:12.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e46a00640 1 -- 172.21.15.47:0/1241178948 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c02e3a0 con 0x7f8e2c0047d0 2024-02-19T15:22:12.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e47e00640 1 --2- 172.21.15.47:0/1241178948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e2c0039d0 0x7f8e2c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e47400640 1 --2- 172.21.15.47:0/1241178948 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e2c0047d0 0x7f8e2c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e47400640 1 -- 172.21.15.47:0/1241178948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e2c0039d0 msgr2=0x7f8e2c003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:12.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e47400640 1 --2- 172.21.15.47:0/1241178948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e2c0039d0 0x7f8e2c003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e47400640 1 -- 172.21.15.47:0/1241178948 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8e2c0052b0 msgr2=0x7f8e2c02d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:12.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e47400640 1 --2- 172.21.15.47:0/1241178948 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8e2c0052b0 0x7f8e2c02d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7f8e47400640 1 -- 172.21.15.47:0/1241178948 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e2c003530 con 0x7f8e2c0047d0 2024-02-19T15:22:12.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.090+0000 7f8e4d200640 1 -- 172.21.15.47:0/1241178948 reap_dead start 2024-02-19T15:22:12.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.086+0000 7ffbb6a00640 1 -- 172.21.15.47:0/601435597 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ffbb0003770 con 0x7ffbc0072590 2024-02-19T15:22:12.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.090+0000 7ffbb6a00640 1 --2- 172.21.15.47:0/601435597 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffba0066ed0 0x7ffba0069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.090+0000 7ffbb6a00640 1 -- 172.21.15.47:0/601435597 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffbb0083b40 con 0x7ffbc0072590 2024-02-19T15:22:12.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.090+0000 7ffbbea00640 1 --2- 172.21.15.47:0/601435597 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffba0066ed0 0x7ffba0069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.090+0000 7f54cfe00640 1 --2- 172.21.15.47:0/3523131288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54d00cfbc0 0x7f54d00cfff0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f54c0009870 tx=0x7f54c001c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4ba375aa82aa9f1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.090+0000 7efcbe400640 1 Processor -- start 2024-02-19T15:22:12.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.090+0000 7efcbe400640 1 -- start start 2024-02-19T15:22:12.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7ffbbea00640 1 --2- 172.21.15.47:0/601435597 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffba0066ed0 0x7ffba0069390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ffbac009ae0 tx=0x7ffbac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f54ce000640 1 -- 172.21.15.47:0/3523131288 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c00ac630 con 0x7f54d00cfbc0 2024-02-19T15:22:12.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f54ce000640 1 -- 172.21.15.47:0/3523131288 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c00ac790 con 0x7f54d00cfbc0 2024-02-19T15:22:12.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f54ce000640 1 -- 172.21.15.47:0/3523131288 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c00ac630 con 0x7f54d00cfbc0 2024-02-19T15:22:12.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7efcbe400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcb00540b0 0x7efcb0006f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7efcbe400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb003bfd0 con 0x7efcb00e32e0 2024-02-19T15:22:12.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f8e47400640 1 --2- 172.21.15.47:0/1241178948 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e2c0047d0 0x7f8e2c004c50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8e34001850 tx=0x7f8e3401c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db39e98e01449e68 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f8e46a00640 1 -- 172.21.15.47:0/1241178948 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e34004ab0 con 0x7f8e2c0047d0 2024-02-19T15:22:12.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f8e46a00640 1 -- 172.21.15.47:0/1241178948 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e34004c10 con 0x7f8e2c0047d0 2024-02-19T15:22:12.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f54d6200640 1 -- 172.21.15.47:0/3523131288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54d00cfbc0 msgr2=0x7f54d00cfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f54d6200640 1 --2- 172.21.15.47:0/3523131288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54d00cfbc0 0x7f54d00cfff0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f54c0009870 tx=0x7f54c001c520 comp rx=0 tx=0).stop 2024-02-19T15:22:12.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f8e4f200640 1 -- 172.21.15.47:0/1241178948 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e2c0047d0 msgr2=0x7f8e2c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f8e4f200640 1 --2- 172.21.15.47:0/1241178948 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e2c0047d0 0x7f8e2c004c50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8e34001850 tx=0x7f8e3401c4e0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f54d6200640 1 -- 172.21.15.47:0/3523131288 shutdown_connections 2024-02-19T15:22:12.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f54d6200640 1 --2- 172.21.15.47:0/3523131288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54d00cfbc0 0x7f54d00cfff0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f54d6200640 1 --2- 172.21.15.47:0/3523131288 >> v2:172.21.15.116:3300/0 conn(0x7f54d0157c00 0x7f54d0072680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f54d6200640 1 -- 172.21.15.47:0/3523131288 >> 172.21.15.47:0/3523131288 conn(0x7f54d00b1950 msgr2=0x7f54d00b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:12.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7efcbe400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb003ef60 con 0x7efcb0053c20 2024-02-19T15:22:12.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7efcbe400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb0058690 con 0x7efcb00540b0 2024-02-19T15:22:12.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7efcbda00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcb00540b0 0x7efcb0006f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7efcbda00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcb00540b0 0x7efcb0006f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:50828/0 (socket says 172.21.15.47:50828) 2024-02-19T15:22:12.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7efcbda00640 1 -- 172.21.15.47:0/3782859453 learned_addr learned my addr 172.21.15.47:0/3782859453 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:12.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f54d6200640 1 -- 172.21.15.47:0/3523131288 shutdown_connections 2024-02-19T15:22:12.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f54d6200640 1 -- 172.21.15.47:0/3523131288 wait complete. 2024-02-19T15:22:12.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.098+0000 7f54d6200640 1 Processor -- start 2024-02-19T15:22:12.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f8e4f200640 1 -- 172.21.15.47:0/1241178948 shutdown_connections 2024-02-19T15:22:12.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f8e4f200640 1 --2- 172.21.15.47:0/1241178948 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e2c0047d0 0x7f8e2c004c50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f8e4f200640 1 -- 172.21.15.47:0/1241178948 >> 172.21.15.47:0/1241178948 conn(0x7f8e480d3de0 msgr2=0x7f8e480d41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:12.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f8e4f200640 1 -- 172.21.15.47:0/1241178948 shutdown_connections 2024-02-19T15:22:12.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f8e4f200640 1 -- 172.21.15.47:0/1241178948 wait complete. 2024-02-19T15:22:12.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f0ab6a00640 1 Processor -- start 2024-02-19T15:22:12.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7f0ab6a00640 1 -- start start 2024-02-19T15:22:12.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7ffbb6a00640 1 -- 172.21.15.47:0/601435597 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ffbb00509d0 con 0x7ffbc0072590 2024-02-19T15:22:12.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7efcbda00640 1 -- 172.21.15.47:0/3782859453 >> v1:172.21.15.116:6789/0 conn(0x7efcb0053c20 legacy=0x7efcb0006800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:12.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7efcbda00640 1 -- 172.21.15.47:0/3782859453 >> v1:172.21.15.47:6789/0 conn(0x7efcb00e32e0 legacy=0x7efcb003e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.098+0000 7f8e4f200640 1 Processor -- start 2024-02-19T15:22:12.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.098+0000 7f8e4f200640 1 -- start start 2024-02-19T15:22:12.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.098+0000 7f54d6200640 1 -- start start 2024-02-19T15:22:12.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.098+0000 7f9563200640 1 Processor -- start 2024-02-19T15:22:12.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.098+0000 7f9563200640 1 -- start start 2024-02-19T15:22:12.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.094+0000 7efcbda00640 1 -- 172.21.15.47:0/3782859453 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb0054540 con 0x7efcb00540b0 2024-02-19T15:22:12.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.098+0000 7efcbda00640 1 --2- 172.21.15.47:0/3782859453 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcb00540b0 0x7efcb0006f10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efcb8069cf0 tx=0x7efcb8097770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=527a2853d9f3ff7b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f9563200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f955c009d80 0x7f955c00a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f54d6200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54d00ce8b0 0x7f54d0153790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e4f200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e480b4060 0x7f8e480b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e4f200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8e48086b80 0x7f8e48162a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e4f200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e480a1160 0x7f8e480a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f54d6200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54d00cfbc0 0x7f54d015b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e47e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e480b4060 0x7f8e480b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e47e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e480b4060 0x7f8e480b4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:50832/0 (socket says 172.21.15.47:50832) 2024-02-19T15:22:12.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f9563200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955c00a720 con 0x7f955c0557f0 2024-02-19T15:22:12.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f9563200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955c039e20 con 0x7f955c0e10f0 2024-02-19T15:22:12.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f9563200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f955c0e14d0 con 0x7f955c009d80 2024-02-19T15:22:12.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f9562800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f955c0557f0 0x7f955c055c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:44074/0 (socket says 172.21.15.47:44074) 2024-02-19T15:22:12.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f9562800640 1 -- 172.21.15.47:0/3913728142 learned_addr learned my addr 172.21.15.47:0/3913728142 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:12.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7efcb7400640 1 -- 172.21.15.47:0/3782859453 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb8124720 con 0x7efcb00540b0 2024-02-19T15:22:12.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f9561400640 1 --2- 172.21.15.47:0/3913728142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f955c009d80 0x7f955c00a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f54cf400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54d00ce8b0 0x7f54d0153790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f54cf400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54d00ce8b0 0x7f54d0153790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:45694/0 (socket says 172.21.15.47:45694) 2024-02-19T15:22:12.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f54cf400640 1 -- 172.21.15.47:0/2013005081 learned_addr learned my addr 172.21.15.47:0/2013005081 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:12.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f54d6200640 1 --2- 172.21.15.47:0/2013005081 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54d0157c00 0x7f54d014b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f54d6200640 1 -- 172.21.15.47:0/2013005081 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d015c6c0 con 0x7f54d0157c00 2024-02-19T15:22:12.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f54d6200640 1 -- 172.21.15.47:0/2013005081 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d015b780 con 0x7f54d00ce8b0 2024-02-19T15:22:12.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f54d6200640 1 -- 172.21.15.47:0/2013005081 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d015ba40 con 0x7f54d00cfbc0 2024-02-19T15:22:12.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f54cfe00640 1 --2- 172.21.15.47:0/2013005081 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54d0157c00 0x7f54d014b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f54cea00640 1 --2- 172.21.15.47:0/2013005081 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54d00cfbc0 0x7f54d015b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f54cfe00640 1 -- 172.21.15.47:0/2013005081 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54d00cfbc0 msgr2=0x7f54d015b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f54cfe00640 1 --2- 172.21.15.47:0/2013005081 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54d00cfbc0 0x7f54d015b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f54cfe00640 1 -- 172.21.15.47:0/2013005081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54d00ce8b0 msgr2=0x7f54d0153790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f54cfe00640 1 --2- 172.21.15.47:0/2013005081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54d00ce8b0 0x7f54d0153790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f54cfe00640 1 -- 172.21.15.47:0/2013005081 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c00094d0 con 0x7f54d0157c00 2024-02-19T15:22:12.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f54cfe00640 1 --2- 172.21.15.47:0/2013005081 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54d0157c00 0x7f54d014b620 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f54c00ac000 tx=0x7f54c00192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e47400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8e48086b80 0x7f8e48162a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e47400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8e48086b80 0x7f8e48162a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:53106/0 (socket says 172.21.15.47:53106) 2024-02-19T15:22:12.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e4f200640 1 -- 172.21.15.47:0/798005913 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e480a0cd0 con 0x7f8e48086b80 2024-02-19T15:22:12.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e47e00640 1 -- 172.21.15.47:0/798005913 learned_addr learned my addr 172.21.15.47:0/798005913 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:12.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e4f200640 1 -- 172.21.15.47:0/798005913 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e480a1b20 con 0x7f8e480a1160 2024-02-19T15:22:12.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e4f200640 1 -- 172.21.15.47:0/798005913 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e480b4980 con 0x7f8e480b4060 2024-02-19T15:22:12.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e4d200640 1 --2- 172.21.15.47:0/798005913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e480a1160 0x7f8e480a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e47e00640 1 -- 172.21.15.47:0/798005913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e480a1160 msgr2=0x7f8e480a15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e47e00640 1 --2- 172.21.15.47:0/798005913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e480a1160 0x7f8e480a15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e47e00640 1 -- 172.21.15.47:0/798005913 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8e48086b80 msgr2=0x7f8e48162a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e47e00640 1 --2- 172.21.15.47:0/798005913 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8e48086b80 0x7f8e48162a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e47e00640 1 -- 172.21.15.47:0/798005913 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e48163300 con 0x7f8e480b4060 2024-02-19T15:22:12.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e47e00640 1 --2- 172.21.15.47:0/798005913 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e480b4060 0x7f8e480b4440 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8e380038e0 tx=0x7f8e38003da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e3fe00640 1 -- 172.21.15.47:0/798005913 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e38010a10 con 0x7f8e480b4060 2024-02-19T15:22:12.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e3fe00640 1 -- 172.21.15.47:0/798005913 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e38010b70 con 0x7f8e480b4060 2024-02-19T15:22:12.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e3fe00640 1 -- 172.21.15.47:0/798005913 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e38001ad0 con 0x7f8e480b4060 2024-02-19T15:22:12.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e4f200640 1 -- 172.21.15.47:0/798005913 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e481635e0 con 0x7f8e480b4060 2024-02-19T15:22:12.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f8e4f200640 1 -- 172.21.15.47:0/798005913 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e480a08f0 con 0x7f8e480b4060 2024-02-19T15:22:12.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f0ab6a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0ab006d7e0 0x7f0ab016d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f0ab6a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab014bf20 con 0x7f0ab014b6a0 2024-02-19T15:22:12.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f0ab6a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab00ce2f0 con 0x7f0ab014b230 2024-02-19T15:22:12.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f0ab6a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab0070a40 con 0x7f0ab006d7e0 2024-02-19T15:22:12.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f0ab5600640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0ab014b230 0x7f0ab00cd4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:46280/0 (socket says 172.21.15.47:46280) 2024-02-19T15:22:12.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f0ab5600640 1 -- 172.21.15.47:0/792238587 learned_addr learned my addr 172.21.15.47:0/792238587 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:12.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f9560a00640 1 -- 172.21.15.47:0/3913728142 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634735736 0 0) 0x7f955c00a720 con 0x7f955c0557f0 2024-02-19T15:22:12.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f9560a00640 1 -- 172.21.15.47:0/3913728142 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9548003530 con 0x7f955c0557f0 2024-02-19T15:22:12.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f9561400640 1 -- 172.21.15.47:0/3913728142 >> v1:172.21.15.116:6789/0 conn(0x7f955c0e10f0 legacy=0x7f955c0e4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f9561400640 1 -- 172.21.15.47:0/3913728142 >> v1:172.21.15.47:6789/0 conn(0x7f955c0557f0 legacy=0x7f955c055c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f9561400640 1 -- 172.21.15.47:0/3913728142 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955c0e5600 con 0x7f955c009d80 2024-02-19T15:22:12.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7efcb7400640 1 -- 172.21.15.47:0/3782859453 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb8124880 con 0x7efcb00540b0 2024-02-19T15:22:12.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7efcb7400640 1 -- 172.21.15.47:0/3782859453 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb8124cf0 con 0x7efcb00540b0 2024-02-19T15:22:12.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7efcbe400640 1 -- 172.21.15.47:0/3782859453 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcb00540b0 msgr2=0x7efcb0006f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7efcbe400640 1 --2- 172.21.15.47:0/3782859453 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcb00540b0 0x7efcb0006f10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efcb8069cf0 tx=0x7efcb8097770 comp rx=0 tx=0).stop 2024-02-19T15:22:12.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7efcbe400640 1 -- 172.21.15.47:0/3782859453 shutdown_connections 2024-02-19T15:22:12.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7efcbe400640 1 --2- 172.21.15.47:0/3782859453 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcb00540b0 0x7efcb0006f10 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7efcbe400640 1 -- 172.21.15.47:0/3782859453 >> 172.21.15.47:0/3782859453 conn(0x7efcb00559f0 msgr2=0x7efcb0055e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:12.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.102+0000 7f0aafe00640 1 -- 172.21.15.47:0/792238587 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3595185863 0 0) 0x7f0ab00ce2f0 con 0x7f0ab014b230 2024-02-19T15:22:12.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0aafe00640 1 -- 172.21.15.47:0/792238587 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a98003530 con 0x7f0ab014b230 2024-02-19T15:22:12.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0ab6000640 1 --2- 172.21.15.47:0/792238587 >> v2:172.21.15.116:3300/0 conn(0x7f0ab006d7e0 0x7f0ab016d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7efcbe400640 1 -- 172.21.15.47:0/3782859453 shutdown_connections 2024-02-19T15:22:12.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7efcbe400640 1 -- 172.21.15.47:0/3782859453 wait complete. 2024-02-19T15:22:12.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0aafe00640 1 -- 172.21.15.47:0/792238587 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2792787155 0 0) 0x7f0a98003530 con 0x7f0ab014b230 2024-02-19T15:22:12.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f80f3e00640 1 Processor -- start 2024-02-19T15:22:12.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0aafe00640 1 -- 172.21.15.47:0/792238587 >> v2:172.21.15.116:3300/0 conn(0x7f0ab006d7e0 msgr2=0x7f0ab016d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:12.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f80f3e00640 1 -- start start 2024-02-19T15:22:12.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f8e4f200640 1 -- 172.21.15.47:0/798005913 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e48075360 con 0x7f8e480b4060 2024-02-19T15:22:12.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0aafe00640 1 --2- 172.21.15.47:0/792238587 >> v2:172.21.15.116:3300/0 conn(0x7f0ab006d7e0 0x7f0ab016d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0aafe00640 1 -- 172.21.15.47:0/792238587 >> v1:172.21.15.47:6789/0 conn(0x7f0ab014b6a0 legacy=0x7f0ab00cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0aafe00640 1 -- 172.21.15.47:0/792238587 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab00cefd0 con 0x7f0ab014b230 2024-02-19T15:22:12.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0aafe00640 1 -- 172.21.15.47:0/792238587 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f0aa4001680 con 0x7f0ab014b230 2024-02-19T15:22:12.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0aafe00640 1 -- 172.21.15.47:0/792238587 >> v1:172.21.15.116:6789/0 conn(0x7f0ab014b230 legacy=0x7f0ab00cd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0aafe00640 1 --2- 172.21.15.47:0/792238587 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a98003980 0x7f0a98003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0aafe00640 1 --2- 172.21.15.47:0/792238587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a980047d0 0x7f0a98004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0aafe00640 1 --2- 172.21.15.47:0/792238587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a98005190 0x7f0a9800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0aafe00640 1 -- 172.21.15.47:0/792238587 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab0070a40 con 0x7f0a98005190 2024-02-19T15:22:12.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0aafe00640 1 -- 172.21.15.47:0/792238587 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9800c130 con 0x7f0a980047d0 2024-02-19T15:22:12.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0aafe00640 1 -- 172.21.15.47:0/792238587 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9800c2f0 con 0x7f0a98003980 2024-02-19T15:22:12.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0ab6000640 1 --2- 172.21.15.47:0/792238587 >> v2:172.21.15.116:3300/0 conn(0x7f0ab006d7e0 0x7f0ab016d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:12.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0ab5600640 1 --2- 172.21.15.47:0/792238587 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a98003980 0x7f0a98003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0ab4c00640 1 --2- 172.21.15.47:0/792238587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a980047d0 0x7f0a98004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0ab6000640 1 --2- 172.21.15.47:0/792238587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a98005190 0x7f0a9800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f9561400640 1 --2- 172.21.15.47:0/3913728142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f955c009d80 0x7f955c00a1e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f95500097f0 tx=0x7f955001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea033f4c4e57f224 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f8e3fe00640 1 -- 172.21.15.47:0/798005913 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8e3801f460 con 0x7f8e480b4060 2024-02-19T15:22:12.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f80f3e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f80f40b57b0 0x7f80f40ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f80f3e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80f41585d0 0x7f80f40addf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f8e3fe00640 1 --2- 172.21.15.47:0/798005913 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8e28066ec0 0x7f8e28069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0ab6000640 1 -- 172.21.15.47:0/792238587 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a98003980 msgr2=0x7f0a98003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:12.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0ab6000640 1 --2- 172.21.15.47:0/792238587 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a98003980 0x7f0a98003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0ab6000640 1 -- 172.21.15.47:0/792238587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a980047d0 msgr2=0x7f0a98004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:12.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0ab6000640 1 --2- 172.21.15.47:0/792238587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a980047d0 0x7f0a98004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f0ab6000640 1 -- 172.21.15.47:0/792238587 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab00cefd0 con 0x7f0a98005190 2024-02-19T15:22:12.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.106+0000 7f80f2000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f80f40b57b0 0x7f80f40ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f80f3400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80f41585d0 0x7f80f40addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f80f3400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80f41585d0 0x7f80f40addf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:50874/0 (socket says 172.21.15.47:50874) 2024-02-19T15:22:12.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f80f2000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f80f40b57b0 0x7f80f40ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:45736/0 (socket says 172.21.15.47:45736) 2024-02-19T15:22:12.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f80f3400640 1 -- 172.21.15.47:0/3469490690 learned_addr learned my addr 172.21.15.47:0/3469490690 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:12.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f0ab5600640 1 --2- 172.21.15.47:0/792238587 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a98003980 0x7f0a98003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:12.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f54c7400640 1 -- 172.21.15.47:0/2013005081 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c00acb00 con 0x7f54d0157c00 2024-02-19T15:22:12.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f54c7400640 1 -- 172.21.15.47:0/2013005081 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c00acc60 con 0x7f54d0157c00 2024-02-19T15:22:12.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f80f3e00640 1 -- 172.21.15.47:0/3469490690 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80f4161aa0 con 0x7f80f41581f0 2024-02-19T15:22:12.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f54c7400640 1 -- 172.21.15.47:0/2013005081 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c0003620 con 0x7f54d0157c00 2024-02-19T15:22:12.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f8e3fe00640 1 -- 172.21.15.47:0/798005913 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e38087760 con 0x7f8e480b4060 2024-02-19T15:22:12.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f80f3e00640 1 -- 172.21.15.47:0/3469490690 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80f4162400 con 0x7f80f40b57b0 2024-02-19T15:22:12.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f80f3e00640 1 -- 172.21.15.47:0/3469490690 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f40b37f0 con 0x7f80f41585d0 2024-02-19T15:22:12.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f8e47400640 1 --2- 172.21.15.47:0/798005913 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8e28066ec0 0x7f8e28069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f54d6200640 1 -- 172.21.15.47:0/2013005081 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d015bd20 con 0x7f54d0157c00 2024-02-19T15:22:12.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f54d6200640 1 -- 172.21.15.47:0/2013005081 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d015c2e0 con 0x7f54d0157c00 2024-02-19T15:22:12.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7efcbe400640 1 Processor -- start 2024-02-19T15:22:12.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f8e47400640 1 --2- 172.21.15.47:0/798005913 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8e28066ec0 0x7f8e28069380 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8e340aa000 tx=0x7f8e340011a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f80f1600640 1 -- 172.21.15.47:0/3469490690 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016359776 0 0) 0x7f80f4161aa0 con 0x7f80f41581f0 2024-02-19T15:22:12.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f80f1600640 1 -- 172.21.15.47:0/3469490690 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80d8003530 con 0x7f80f41581f0 2024-02-19T15:22:12.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f80f3400640 1 -- 172.21.15.47:0/3469490690 >> v2:172.21.15.116:3300/0 conn(0x7f80f40b57b0 msgr2=0x7f80f40ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:12.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f80f3400640 1 --2- 172.21.15.47:0/3469490690 >> v2:172.21.15.116:3300/0 conn(0x7f80f40b57b0 0x7f80f40ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f80f3400640 1 -- 172.21.15.47:0/3469490690 >> v1:172.21.15.47:6789/0 conn(0x7f80f41581f0 legacy=0x7f80f40b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7efcbe400640 1 -- start start 2024-02-19T15:22:12.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f80f3400640 1 -- 172.21.15.47:0/3469490690 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f415f310 con 0x7f80f41585d0 2024-02-19T15:22:12.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7efcbe400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcb0053c20 0x7efcb00e2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7efcbe400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcb00540b0 0x7efcb00d9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7efcbe400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb00e4b80 0x7efcb00da120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7efcbe400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb00d1f00 con 0x7efcb0053c20 2024-02-19T15:22:12.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7efcbe400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb00e55b0 con 0x7efcb00e4b80 2024-02-19T15:22:12.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7efcbe400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb00e5890 con 0x7efcb00540b0 2024-02-19T15:22:12.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7f54d6200640 1 -- 172.21.15.47:0/2013005081 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a80052a0 con 0x7f54d0157c00 2024-02-19T15:22:12.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7efcb7e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcb00540b0 0x7efcb00d9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7efcb7e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcb00540b0 0x7efcb00d9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:50890/0 (socket says 172.21.15.47:50890) 2024-02-19T15:22:12.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7efcb7e00640 1 -- 172.21.15.47:0/1480217012 learned_addr learned my addr 172.21.15.47:0/1480217012 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:12.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7efcbda00640 1 --2- 172.21.15.47:0/1480217012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb00e4b80 0x7efcb00da120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7efcb7e00640 1 -- 172.21.15.47:0/1480217012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb00e4b80 msgr2=0x7efcb00da120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7efcb7e00640 1 --2- 172.21.15.47:0/1480217012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb00e4b80 0x7efcb00da120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7efcb7e00640 1 -- 172.21.15.47:0/1480217012 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcb0053c20 msgr2=0x7efcb00e2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7efcb7e00640 1 --2- 172.21.15.47:0/1480217012 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcb0053c20 0x7efcb00e2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7efcb7e00640 1 -- 172.21.15.47:0/1480217012 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb8073870 con 0x7efcb00540b0 2024-02-19T15:22:12.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.110+0000 7fb43ee00640 1 Processor -- start 2024-02-19T15:22:12.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7fb43ee00640 1 -- start start 2024-02-19T15:22:12.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7f8e3fe00640 1 -- 172.21.15.47:0/798005913 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8e38054670 con 0x7f8e480b4060 2024-02-19T15:22:12.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7f0ab6000640 1 -- 172.21.15.47:0/792238587 reap_dead start 2024-02-19T15:22:12.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7f0ab6000640 1 --2- 172.21.15.47:0/792238587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a98005190 0x7f0a9800b990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800e350 tx=0x7f0aa801ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655947be7b86e8c5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7f9560a00640 1 -- 172.21.15.47:0/3913728142 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95500ac620 con 0x7f955c009d80 2024-02-19T15:22:12.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7f9560a00640 1 -- 172.21.15.47:0/3913728142 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95500ac780 con 0x7f955c009d80 2024-02-19T15:22:12.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7f9560a00640 1 -- 172.21.15.47:0/3913728142 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95500acb10 con 0x7f955c009d80 2024-02-19T15:22:12.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7f9563200640 1 -- 172.21.15.47:0/3913728142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f955c009d80 msgr2=0x7f955c00a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7f9563200640 1 --2- 172.21.15.47:0/3913728142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f955c009d80 0x7f955c00a1e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f95500097f0 tx=0x7f955001c3e0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7f9563200640 1 -- 172.21.15.47:0/3913728142 shutdown_connections 2024-02-19T15:22:12.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7f54c7400640 1 -- 172.21.15.47:0/2013005081 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f54c00ba070 con 0x7f54d0157c00 2024-02-19T15:22:12.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7f9563200640 1 --2- 172.21.15.47:0/3913728142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f955c009d80 0x7f955c00a1e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7f9563200640 1 -- 172.21.15.47:0/3913728142 >> 172.21.15.47:0/3913728142 conn(0x7f955c056cf0 msgr2=0x7f955c057100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:12.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7fb43ee00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb43815de20 0x7fb4380cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7fb43ee00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb438072890 0x7fb4380cbe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7fb43ee00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4380cc7f0 con 0x7fb4380cc3a0 2024-02-19T15:22:12.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7fb43ee00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4380cf630 con 0x7fb43815de20 2024-02-19T15:22:12.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7fb43ee00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb43815e200 con 0x7fb438072890 2024-02-19T15:22:12.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7f9563200640 1 -- 172.21.15.47:0/3913728142 shutdown_connections 2024-02-19T15:22:12.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7fb437400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb438072890 0x7fb4380cbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7fb437400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb438072890 0x7fb4380cbe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:45754/0 (socket says 172.21.15.47:45754) 2024-02-19T15:22:12.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7fb437400640 1 -- 172.21.15.47:0/420089722 learned_addr learned my addr 172.21.15.47:0/420089722 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:12.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7f80f3400640 1 --2- 172.21.15.47:0/3469490690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80f41585d0 0x7f80f40addf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f80ec009870 tx=0x7f80ec01c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c386e00a89bc4e5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7fb437400640 1 -- 172.21.15.47:0/420089722 >> v2:172.21.15.47:3300/0 conn(0x7fb43815de20 msgr2=0x7fb4380cf0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:12.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7fb437400640 1 --2- 172.21.15.47:0/420089722 >> v2:172.21.15.47:3300/0 conn(0x7fb43815de20 0x7fb4380cf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7fb437400640 1 -- 172.21.15.47:0/420089722 >> v1:172.21.15.47:6789/0 conn(0x7fb4380cc3a0 legacy=0x7fb4380ad840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:12.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7f54c7400640 1 --2- 172.21.15.47:0/2013005081 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f54b0066e90 0x7f54b0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7f54c7400640 1 -- 172.21.15.47:0/2013005081 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54c0120bb0 con 0x7f54d0157c00 2024-02-19T15:22:12.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7f54cf400640 1 --2- 172.21.15.47:0/2013005081 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f54b0066e90 0x7f54b0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7efcb7e00640 1 --2- 172.21.15.47:0/1480217012 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcb00540b0 0x7efcb00d9be0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efca400c450 tx=0x7efca400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7fb437400640 1 -- 172.21.15.47:0/420089722 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4381626c0 con 0x7fb438072890 2024-02-19T15:22:12.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.114+0000 7f9563200640 1 -- 172.21.15.47:0/3913728142 wait complete. 2024-02-19T15:22:12.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7efcb4c00640 1 -- 172.21.15.47:0/1480217012 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4012830 con 0x7efcb00540b0 2024-02-19T15:22:12.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f54cf400640 1 --2- 172.21.15.47:0/2013005081 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f54b0066e90 0x7f54b0069350 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f54c8008bc0 tx=0x7f54c8007730 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f80f1600640 1 -- 172.21.15.47:0/3469490690 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec0aa610 con 0x7f80f41585d0 2024-02-19T15:22:12.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f80f1600640 1 -- 172.21.15.47:0/3469490690 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ec0a6040 con 0x7f80f41585d0 2024-02-19T15:22:12.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f80f1600640 1 -- 172.21.15.47:0/3469490690 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec0aa990 con 0x7f80f41585d0 2024-02-19T15:22:12.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7efcb4c00640 1 -- 172.21.15.47:0/1480217012 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca40129b0 con 0x7efcb00540b0 2024-02-19T15:22:12.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7efcb4c00640 1 -- 172.21.15.47:0/1480217012 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4012d60 con 0x7efcb00540b0 2024-02-19T15:22:12.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7efcbe400640 1 -- 172.21.15.47:0/1480217012 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb00e5b70 con 0x7efcb00540b0 2024-02-19T15:22:12.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f9563200640 1 Processor -- start 2024-02-19T15:22:12.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7efcbe400640 1 -- 172.21.15.47:0/1480217012 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb0050380 con 0x7efcb00540b0 2024-02-19T15:22:12.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f9563200640 1 -- start start 2024-02-19T15:22:12.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f80f3e00640 1 -- 172.21.15.47:0/3469490690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80f41585d0 msgr2=0x7f80f40addf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f80f3e00640 1 --2- 172.21.15.47:0/3469490690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80f41585d0 0x7f80f40addf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f80ec009870 tx=0x7f80ec01c4d0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f9563200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f955c009d80 0x7f955c17c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f9563200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f955c0557f0 0x7f955c0e2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f9563200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f955c0e10f0 0x7f955c188560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f9563200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955c0e2960 con 0x7f955c009d80 2024-02-19T15:22:12.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f9563200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955c0e2c30 con 0x7f955c0557f0 2024-02-19T15:22:12.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f9563200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f955c0e2f10 con 0x7f955c0e10f0 2024-02-19T15:22:12.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f9561e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f955c009d80 0x7f955c17c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f9561e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f955c009d80 0x7f955c17c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:53160/0 (socket says 172.21.15.47:53160) 2024-02-19T15:22:12.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f9562800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f955c0e10f0 0x7f955c188560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f9561e00640 1 -- 172.21.15.47:0/972515436 learned_addr learned my addr 172.21.15.47:0/972515436 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:12.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f9561400640 1 --2- 172.21.15.47:0/972515436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f955c0557f0 0x7f955c0e2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f9561e00640 1 -- 172.21.15.47:0/972515436 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f955c0e10f0 msgr2=0x7f955c188560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f9561e00640 1 --2- 172.21.15.47:0/972515436 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f955c0e10f0 0x7f955c188560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f9561e00640 1 -- 172.21.15.47:0/972515436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f955c0557f0 msgr2=0x7f955c0e2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f9561e00640 1 --2- 172.21.15.47:0/972515436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f955c0557f0 0x7f955c0e2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f9561e00640 1 -- 172.21.15.47:0/972515436 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9550009420 con 0x7f955c009d80 2024-02-19T15:22:12.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7fb437400640 1 --2- 172.21.15.47:0/420089722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb438072890 0x7fb4380cbe60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb4380b46b0 tx=0x7fb42401c420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8945c3566156633 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7efcb4c00640 1 -- 172.21.15.47:0/1480217012 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efca401c460 con 0x7efcb00540b0 2024-02-19T15:22:12.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7f80f3e00640 1 -- 172.21.15.47:0/3469490690 shutdown_connections 2024-02-19T15:22:12.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7f80f3e00640 1 --2- 172.21.15.47:0/3469490690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80f41585d0 0x7f80f40addf0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7f80f3e00640 1 --2- 172.21.15.47:0/3469490690 >> v2:172.21.15.116:3300/0 conn(0x7f80f40b57b0 0x7f80f40ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7f80f3e00640 1 -- 172.21.15.47:0/3469490690 >> 172.21.15.47:0/3469490690 conn(0x7f80f40b24f0 msgr2=0x7f80f40b2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:12.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7efcbe400640 1 -- 172.21.15.47:0/1480217012 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb00e56f0 con 0x7efcb00540b0 2024-02-19T15:22:12.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7fb436a00640 1 -- 172.21.15.47:0/420089722 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4240ab600 con 0x7fb438072890 2024-02-19T15:22:12.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7fb436a00640 1 -- 172.21.15.47:0/420089722 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4240a7050 con 0x7fb438072890 2024-02-19T15:22:12.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7fb436a00640 1 -- 172.21.15.47:0/420089722 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4240ab980 con 0x7fb438072890 2024-02-19T15:22:12.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7efcb4c00640 1 --2- 172.21.15.47:0/1480217012 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc98066f10 0x7efc980693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7efcb4c00640 1 -- 172.21.15.47:0/1480217012 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efca4084800 con 0x7efcb00540b0 2024-02-19T15:22:12.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7efcbd000640 1 --2- 172.21.15.47:0/1480217012 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc98066f10 0x7efc980693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.118+0000 7f9561e00640 1 --2- 172.21.15.47:0/972515436 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f955c009d80 0x7f955c17c850 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f955400c420 tx=0x7f955400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7fb43ee00640 1 -- 172.21.15.47:0/420089722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb438072890 msgr2=0x7fb4380cbe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7fb43ee00640 1 --2- 172.21.15.47:0/420089722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb438072890 0x7fb4380cbe60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb4380b46b0 tx=0x7fb42401c420 comp rx=0 tx=0).stop 2024-02-19T15:22:12.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7fb43ee00640 1 -- 172.21.15.47:0/420089722 shutdown_connections 2024-02-19T15:22:12.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7fb43ee00640 1 --2- 172.21.15.47:0/420089722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb438072890 0x7fb4380cbe60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7fb43ee00640 1 --2- 172.21.15.47:0/420089722 >> v2:172.21.15.47:3300/0 conn(0x7fb43815de20 0x7fb4380cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7fb43ee00640 1 -- 172.21.15.47:0/420089722 >> 172.21.15.47:0/420089722 conn(0x7fb4380b1950 msgr2=0x7fb4380b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:12.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7f955a000640 1 -- 172.21.15.47:0/972515436 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95540052d0 con 0x7f955c009d80 2024-02-19T15:22:12.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7f955a000640 1 -- 172.21.15.47:0/972515436 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9554005430 con 0x7f955c009d80 2024-02-19T15:22:12.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7f955a000640 1 -- 172.21.15.47:0/972515436 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9554007260 con 0x7f955c009d80 2024-02-19T15:22:12.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7f9563200640 1 -- 172.21.15.47:0/972515436 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f955c0e31f0 con 0x7f955c009d80 2024-02-19T15:22:12.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7f0aafe00640 1 -- 172.21.15.47:0/792238587 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa80a5d60 con 0x7f0a98005190 2024-02-19T15:22:12.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7f9563200640 1 -- 172.21.15.47:0/972515436 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f955c0e37b0 con 0x7f955c009d80 2024-02-19T15:22:12.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7efcbd000640 1 --2- 172.21.15.47:0/1480217012 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc98066f10 0x7efc980693d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efcac000f70 tx=0x7efcac0079c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7f80f3e00640 1 -- 172.21.15.47:0/3469490690 shutdown_connections 2024-02-19T15:22:12.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7f80f3e00640 1 -- 172.21.15.47:0/3469490690 wait complete. 2024-02-19T15:22:12.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7f80f3e00640 1 Processor -- start 2024-02-19T15:22:12.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7f80f3e00640 1 -- start start 2024-02-19T15:22:12.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7f0aafe00640 1 -- 172.21.15.47:0/792238587 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa800f040 con 0x7f0a98005190 2024-02-19T15:22:12.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7f0aafe00640 1 -- 172.21.15.47:0/792238587 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8003be0 con 0x7f0a98005190 2024-02-19T15:22:12.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f9563200640 1 -- 172.21.15.47:0/972515436 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f955c0073e0 con 0x7f955c009d80 2024-02-19T15:22:12.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f0ab6a00640 1 -- 172.21.15.47:0/792238587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a98005190 msgr2=0x7f0a9800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f0ab6a00640 1 --2- 172.21.15.47:0/792238587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a98005190 0x7f0a9800b990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800e350 tx=0x7f0aa801ce60 comp rx=0 tx=0).stop 2024-02-19T15:22:12.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f0ab6a00640 1 -- 172.21.15.47:0/792238587 shutdown_connections 2024-02-19T15:22:12.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f0ab6a00640 1 --2- 172.21.15.47:0/792238587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a98005190 0x7f0a9800b990 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f0ab6a00640 1 -- 172.21.15.47:0/792238587 >> 172.21.15.47:0/792238587 conn(0x7f0ab00cf900 msgr2=0x7f0ab00cfce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:12.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f54c7400640 1 -- 172.21.15.47:0/2013005081 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f54c00edb10 con 0x7f54d0157c00 2024-02-19T15:22:12.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f0ab6a00640 1 -- 172.21.15.47:0/792238587 shutdown_connections 2024-02-19T15:22:12.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7fb43ee00640 1 -- 172.21.15.47:0/420089722 shutdown_connections 2024-02-19T15:22:12.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.122+0000 7fb43ee00640 1 -- 172.21.15.47:0/420089722 wait complete. 2024-02-19T15:22:12.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7fb43ee00640 1 Processor -- start 2024-02-19T15:22:12.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f0ab6a00640 1 -- 172.21.15.47:0/792238587 wait complete. 2024-02-19T15:22:12.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f80f3e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80f40b57b0 0x7f80f406f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f80f3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80f41581f0 0x7f80f406f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f80f3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80f41585d0 0x7f80f41536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f80f3e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f4155760 con 0x7f80f41581f0 2024-02-19T15:22:12.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f80f3e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f40d0af0 con 0x7f80f41585d0 2024-02-19T15:22:12.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f80f3e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f41551a0 con 0x7f80f40b57b0 2024-02-19T15:22:12.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f80f2a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80f40b57b0 0x7f80f406f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7efcb4c00640 1 -- 172.21.15.47:0/1480217012 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7efcb00e56f0 con 0x7efcb00540b0 2024-02-19T15:22:12.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f80f2a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80f40b57b0 0x7f80f406f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:50908/0 (socket says 172.21.15.47:50908) 2024-02-19T15:22:12.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f80f2a00640 1 -- 172.21.15.47:0/4029544555 learned_addr learned my addr 172.21.15.47:0/4029544555 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:12.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f80f3400640 1 --2- 172.21.15.47:0/4029544555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80f41585d0 0x7f80f41536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f80f2a00640 1 -- 172.21.15.47:0/4029544555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80f41585d0 msgr2=0x7f80f41536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f80f2a00640 1 --2- 172.21.15.47:0/4029544555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80f41585d0 0x7f80f41536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f80f2a00640 1 -- 172.21.15.47:0/4029544555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80f41581f0 msgr2=0x7f80f406f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f80f2a00640 1 --2- 172.21.15.47:0/4029544555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80f41581f0 0x7f80f406f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f80f2a00640 1 -- 172.21.15.47:0/4029544555 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ec0094d0 con 0x7f80f40b57b0 2024-02-19T15:22:12.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f80f2a00640 1 --2- 172.21.15.47:0/4029544555 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80f40b57b0 0x7f80f406f1b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f80e400c800 tx=0x7f80e400ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.130+0000 7f0ab6a00640 1 Processor -- start 2024-02-19T15:22:12.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f80eaa00640 1 -- 172.21.15.47:0/4029544555 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e4003b80 con 0x7f80f40b57b0 2024-02-19T15:22:12.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f80eaa00640 1 -- 172.21.15.47:0/4029544555 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e4003d00 con 0x7f80f40b57b0 2024-02-19T15:22:12.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.126+0000 7f80eaa00640 1 -- 172.21.15.47:0/4029544555 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e400aa90 con 0x7f80f40b57b0 2024-02-19T15:22:12.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.130+0000 7f80f3e00640 1 -- 172.21.15.47:0/4029544555 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f4155480 con 0x7f80f40b57b0 2024-02-19T15:22:12.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.130+0000 7f80f3e00640 1 -- 172.21.15.47:0/4029544555 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f406ec90 con 0x7f80f40b57b0 2024-02-19T15:22:12.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.130+0000 7f80f3e00640 1 -- 172.21.15.47:0/4029544555 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80f4075360 con 0x7f80f40b57b0 2024-02-19T15:22:12.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.130+0000 7f955a000640 1 -- 172.21.15.47:0/972515436 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f95540073c0 con 0x7f955c009d80 2024-02-19T15:22:12.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.130+0000 7f80eaa00640 1 -- 172.21.15.47:0/4029544555 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f80e400abf0 con 0x7f80f40b57b0 2024-02-19T15:22:12.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f80eaa00640 1 --2- 172.21.15.47:0/4029544555 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f80d0066e70 0x7f80d0069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f80eaa00640 1 -- 172.21.15.47:0/4029544555 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f80e4084910 con 0x7f80f40b57b0 2024-02-19T15:22:12.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.130+0000 7f0ab6a00640 1 -- start start 2024-02-19T15:22:12.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f0ab6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a98005190 0x7f0ab0144b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f0ab6a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ab00b2ef0 0x7f0ab0163bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f0ab6a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ab0145320 0x7f0ab01457b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f80f2000640 1 --2- 172.21.15.47:0/4029544555 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f80d0066e70 0x7f80d0069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f0ab6a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab0145cf0 con 0x7f0a98005190 2024-02-19T15:22:12.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f0ab6a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab01654c0 con 0x7f0ab00b2ef0 2024-02-19T15:22:12.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f0ab6000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ab0145320 0x7f0ab01457b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f0ab6000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ab0145320 0x7f0ab01457b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:50918/0 (socket says 172.21.15.47:50918) 2024-02-19T15:22:12.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f0ab6000640 1 -- 172.21.15.47:0/3213141509 learned_addr learned my addr 172.21.15.47:0/3213141509 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:12.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f0ab6a00640 1 -- 172.21.15.47:0/3213141509 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab01450c0 con 0x7f0ab0145320 2024-02-19T15:22:12.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f0ab5600640 1 --2- 172.21.15.47:0/3213141509 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a98005190 0x7f0ab0144b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f0ab4c00640 1 --2- 172.21.15.47:0/3213141509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ab00b2ef0 0x7f0ab0163bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f0ab6000640 1 -- 172.21.15.47:0/3213141509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ab00b2ef0 msgr2=0x7f0ab0163bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f0ab6000640 1 --2- 172.21.15.47:0/3213141509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ab00b2ef0 0x7f0ab0163bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f0ab6000640 1 -- 172.21.15.47:0/3213141509 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a98005190 msgr2=0x7f0ab0144b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f0ab6000640 1 --2- 172.21.15.47:0/3213141509 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a98005190 0x7f0ab0144b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f0ab6000640 1 -- 172.21.15.47:0/3213141509 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab0164420 con 0x7f0ab0145320 2024-02-19T15:22:12.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f955a000640 1 --2- 172.21.15.47:0/972515436 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9544066e90 0x7f9544069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f0ab6000640 1 --2- 172.21.15.47:0/3213141509 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ab0145320 0x7f0ab01457b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8098880 tx=0x7f0aa8009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f955a000640 1 -- 172.21.15.47:0/972515436 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f95540844c0 con 0x7f955c009d80 2024-02-19T15:22:12.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f80f2000640 1 --2- 172.21.15.47:0/4029544555 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f80d0066e70 0x7f80d0069330 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f80dc006aa0 tx=0x7f80dc006a30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f9561400640 1 --2- 172.21.15.47:0/972515436 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9544066e90 0x7f9544069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7fb43ee00640 1 -- start start 2024-02-19T15:22:12.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7fb43ee00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb438072890 0x7fb43814d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7fb43ee00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4380cc3a0 0x7fb43806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7fb43ee00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb43815de20 0x7fb43806d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7f9561400640 1 --2- 172.21.15.47:0/972515436 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9544066e90 0x7f9544069350 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f95500ae000 tx=0x7f95500a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7fb43ee00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb438156050 con 0x7fb4380cc3a0 2024-02-19T15:22:12.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7fb43ee00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43814d9e0 con 0x7fb43815de20 2024-02-19T15:22:12.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7fb43ee00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb438155440 con 0x7fb438072890 2024-02-19T15:22:12.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7fb43ce00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb43815de20 0x7fb43806d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7fb43ce00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb43815de20 0x7fb43806d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:45784/0 (socket says 172.21.15.47:45784) 2024-02-19T15:22:12.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7fb43ce00640 1 -- 172.21.15.47:0/2850814164 learned_addr learned my addr 172.21.15.47:0/2850814164 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:12.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7fb43ce00640 1 -- 172.21.15.47:0/2850814164 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb438072890 msgr2=0x7fb43814d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7fb43ce00640 1 --2- 172.21.15.47:0/2850814164 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb438072890 0x7fb43814d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.134+0000 7fb43ce00640 1 -- 172.21.15.47:0/2850814164 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4380cc3a0 msgr2=0x7fb43806d1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T15:22:12.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.138+0000 7fb437400640 1 --2- 172.21.15.47:0/2850814164 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4380cc3a0 0x7fb43806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.138+0000 7fb43ce00640 1 --2- 172.21.15.47:0/2850814164 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4380cc3a0 0x7fb43806d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.138+0000 7fb43ce00640 1 -- 172.21.15.47:0/2850814164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4240094d0 con 0x7fb43815de20 2024-02-19T15:22:12.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.138+0000 7f0aad600640 1 -- 172.21.15.47:0/3213141509 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8004430 con 0x7f0ab0145320 2024-02-19T15:22:12.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.138+0000 7f0aad600640 1 -- 172.21.15.47:0/3213141509 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa8004590 con 0x7f0ab0145320 2024-02-19T15:22:12.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.138+0000 7f0ab6a00640 1 -- 172.21.15.47:0/3213141509 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab0164700 con 0x7f0ab0145320 2024-02-19T15:22:12.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.138+0000 7f0ab6a00640 1 -- 172.21.15.47:0/3213141509 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab0164cc0 con 0x7f0ab0145320 2024-02-19T15:22:12.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.138+0000 7f0aad600640 1 -- 172.21.15.47:0/3213141509 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa80b6480 con 0x7f0ab0145320 2024-02-19T15:22:12.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.138+0000 7f955a000640 1 -- 172.21.15.47:0/972515436 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f95540514d0 con 0x7f955c009d80 2024-02-19T15:22:12.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.138+0000 7f80eaa00640 1 -- 172.21.15.47:0/4029544555 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f80e4051670 con 0x7f80f40b57b0 2024-02-19T15:22:12.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.138+0000 7fb437400640 1 --2- 172.21.15.47:0/2850814164 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4380cc3a0 0x7fb43806d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:22:12.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.138+0000 7fb43ce00640 1 --2- 172.21.15.47:0/2850814164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb43815de20 0x7fb43806d710 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb43000c780 tx=0x7fb43000cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.138+0000 7fb42fe00640 1 -- 172.21.15.47:0/2850814164 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb430003d90 con 0x7fb43815de20 2024-02-19T15:22:12.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.138+0000 7fb42fe00640 1 -- 172.21.15.47:0/2850814164 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb43000ce90 con 0x7fb43815de20 2024-02-19T15:22:12.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.138+0000 7fb42fe00640 1 -- 172.21.15.47:0/2850814164 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4300052b0 con 0x7fb43815de20 2024-02-19T15:22:12.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.138+0000 7fb43ee00640 1 -- 172.21.15.47:0/2850814164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb438155720 con 0x7fb43815de20 2024-02-19T15:22:12.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.142+0000 7f0aad600640 1 -- 172.21.15.47:0/3213141509 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0aa80b65e0 con 0x7f0ab0145320 2024-02-19T15:22:12.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.142+0000 7fb43ee00640 1 -- 172.21.15.47:0/2850814164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb438155ce0 con 0x7fb43815de20 2024-02-19T15:22:12.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.142+0000 7f0aad600640 1 --2- 172.21.15.47:0/3213141509 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a90066f40 0x7f0a90069400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.142+0000 7f0aad600640 1 -- 172.21.15.47:0/3213141509 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0aa811f690 con 0x7f0ab0145320 2024-02-19T15:22:12.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.142+0000 7f0ab5600640 1 --2- 172.21.15.47:0/3213141509 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a90066f40 0x7f0a90069400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.142+0000 7fb42fe00640 1 -- 172.21.15.47:0/2850814164 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb430005410 con 0x7fb43815de20 2024-02-19T15:22:12.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.142+0000 7fb42fe00640 1 --2- 172.21.15.47:0/2850814164 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb418066e90 0x7fb418069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:12.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.142+0000 7f0ab5600640 1 --2- 172.21.15.47:0/3213141509 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a90066f40 0x7f0a90069400 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4000fc0 tx=0x7f0aa4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.142+0000 7fb437e00640 1 --2- 172.21.15.47:0/2850814164 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb418066e90 0x7fb418069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:12.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.146+0000 7f0ab6a00640 1 -- 172.21.15.47:0/3213141509 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a8c0052a0 con 0x7f0ab0145320 2024-02-19T15:22:12.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.146+0000 7fb42fe00640 1 -- 172.21.15.47:0/2850814164 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb430055310 con 0x7fb43815de20 2024-02-19T15:22:12.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.146+0000 7fb43ee00640 1 -- 172.21.15.47:0/2850814164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb438075360 con 0x7fb43815de20 2024-02-19T15:22:12.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.146+0000 7fb437e00640 1 --2- 172.21.15.47:0/2850814164 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb418066e90 0x7fb418069350 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb428009730 tx=0x7fb428006ca0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:12.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.150+0000 7f0aad600640 1 -- 172.21.15.47:0/3213141509 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0aa80ec3d0 con 0x7f0ab0145320 2024-02-19T15:22:12.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.150+0000 7fb42fe00640 1 -- 172.21.15.47:0/2850814164 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb430050eb0 con 0x7fb43815de20 2024-02-19T15:22:12.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.446+0000 7ffbc5a00640 1 -- 172.21.15.47:0/601435597 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ffbc0075360 con 0x7ffbc0072590 2024-02-19T15:22:12.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.446+0000 7ffbb6a00640 1 -- 172.21.15.47:0/601435597 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffbb0017070 con 0x7ffbc0072590 2024-02-19T15:22:12.448 INFO:teuthology.orchestra.run.smithi047.stdout:60129542150 2024-02-19T15:22:12.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.450+0000 7ffbc5a00640 1 -- 172.21.15.47:0/601435597 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffba0066ed0 msgr2=0x7ffba0069390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.450+0000 7ffbc5a00640 1 --2- 172.21.15.47:0/601435597 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffba0066ed0 0x7ffba0069390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ffbac009ae0 tx=0x7ffbac008040 comp rx=0 tx=0).stop 2024-02-19T15:22:12.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.450+0000 7ffbc5a00640 1 -- 172.21.15.47:0/601435597 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffbc0072590 msgr2=0x7ffbc014ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.450+0000 7ffbc5a00640 1 --2- 172.21.15.47:0/601435597 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffbc0072590 0x7ffbc014ec30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0001ec0 tx=0x7ffbb000c8e0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.450+0000 7ffbc5a00640 1 -- 172.21.15.47:0/601435597 shutdown_connections 2024-02-19T15:22:12.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.450+0000 7ffbc5a00640 1 --2- 172.21.15.47:0/601435597 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffba0066ed0 0x7ffba0069390 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.450+0000 7ffbc5a00640 1 --2- 172.21.15.47:0/601435597 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffbc015bf50 0x7ffbc00cc670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.450+0000 7ffbc5a00640 1 --2- 172.21.15.47:0/601435597 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffbc0072590 0x7ffbc014ec30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.450+0000 7ffbc5a00640 1 --2- 172.21.15.47:0/601435597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbc006aeb0 0x7ffbc00c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.450+0000 7ffbc5a00640 1 -- 172.21.15.47:0/601435597 >> 172.21.15.47:0/601435597 conn(0x7ffbc00b2560 msgr2=0x7ffbc00c9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:12.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.450+0000 7ffbc5a00640 1 -- 172.21.15.47:0/601435597 shutdown_connections 2024-02-19T15:22:12.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.450+0000 7ffbc5a00640 1 -- 172.21.15.47:0/601435597 wait complete. 2024-02-19T15:22:12.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.454+0000 7f8e4f200640 1 -- 172.21.15.47:0/798005913 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f8e4806bdf0 con 0x7f8e480b4060 2024-02-19T15:22:12.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.458+0000 7f8e3fe00640 1 -- 172.21.15.47:0/798005913 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8e38057d30 con 0x7f8e480b4060 2024-02-19T15:22:12.462 INFO:teuthology.orchestra.run.smithi047.stdout:60129542149 2024-02-19T15:22:12.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.462+0000 7f8e4f200640 1 -- 172.21.15.47:0/798005913 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8e28066ec0 msgr2=0x7f8e28069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.462+0000 7f8e4f200640 1 --2- 172.21.15.47:0/798005913 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8e28066ec0 0x7f8e28069380 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8e340aa000 tx=0x7f8e340011a0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.462+0000 7f8e4f200640 1 -- 172.21.15.47:0/798005913 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e480b4060 msgr2=0x7f8e480b4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.462+0000 7f8e4f200640 1 --2- 172.21.15.47:0/798005913 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e480b4060 0x7f8e480b4440 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8e380038e0 tx=0x7f8e38003da0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.462+0000 7f8e4f200640 1 -- 172.21.15.47:0/798005913 shutdown_connections 2024-02-19T15:22:12.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.462+0000 7f8e4f200640 1 --2- 172.21.15.47:0/798005913 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8e28066ec0 0x7f8e28069380 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.462+0000 7f8e4f200640 1 --2- 172.21.15.47:0/798005913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e480a1160 0x7f8e480a15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.466+0000 7f8e4f200640 1 --2- 172.21.15.47:0/798005913 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8e48086b80 0x7f8e48162a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.466+0000 7f8e4f200640 1 --2- 172.21.15.47:0/798005913 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e480b4060 0x7f8e480b4440 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.466+0000 7f8e4f200640 1 -- 172.21.15.47:0/798005913 >> 172.21.15.47:0/798005913 conn(0x7f8e480d3de0 msgr2=0x7f8e480b09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:12.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.466+0000 7f8e4f200640 1 -- 172.21.15.47:0/798005913 shutdown_connections 2024-02-19T15:22:12.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.466+0000 7f8e4f200640 1 -- 172.21.15.47:0/798005913 wait complete. 2024-02-19T15:22:12.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.474+0000 7f54d6200640 1 -- 172.21.15.47:0/2013005081 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f54a8005820 con 0x7f54d0157c00 2024-02-19T15:22:12.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.478+0000 7f54c7400640 1 -- 172.21.15.47:0/2013005081 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f54c00b1090 con 0x7f54d0157c00 2024-02-19T15:22:12.479 INFO:teuthology.orchestra.run.smithi047.stdout:64424509445 2024-02-19T15:22:12.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.478+0000 7f80f3e00640 1 -- 172.21.15.47:0/4029544555 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f80f4161060 con 0x7f80f40b57b0 2024-02-19T15:22:12.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.482+0000 7f80eaa00640 1 -- 172.21.15.47:0/4029544555 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f80e4054d30 con 0x7f80f40b57b0 2024-02-19T15:22:12.483 INFO:teuthology.orchestra.run.smithi047.stdout:60129542150 2024-02-19T15:22:12.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.482+0000 7f54d6200640 1 -- 172.21.15.47:0/2013005081 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f54b0066e90 msgr2=0x7f54b0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.482+0000 7f54d6200640 1 --2- 172.21.15.47:0/2013005081 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f54b0066e90 0x7f54b0069350 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f54c8008bc0 tx=0x7f54c8007730 comp rx=0 tx=0).stop 2024-02-19T15:22:12.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.482+0000 7f54d6200640 1 -- 172.21.15.47:0/2013005081 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54d0157c00 msgr2=0x7f54d014b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.482+0000 7f54d6200640 1 --2- 172.21.15.47:0/2013005081 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54d0157c00 0x7f54d014b620 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f54c00ac000 tx=0x7f54c00192c0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.482+0000 7f9563200640 1 -- 172.21.15.47:0/972515436 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f955c036800 con 0x7f955c009d80 2024-02-19T15:22:12.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.482+0000 7f955a000640 1 -- 172.21.15.47:0/972515436 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9554020090 con 0x7f955c009d80 2024-02-19T15:22:12.484 INFO:teuthology.orchestra.run.smithi047.stdout:60129542148 2024-02-19T15:22:12.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.482+0000 7f54d6200640 1 -- 172.21.15.47:0/2013005081 shutdown_connections 2024-02-19T15:22:12.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.482+0000 7f54d6200640 1 --2- 172.21.15.47:0/2013005081 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f54b0066e90 0x7f54b0069350 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.482+0000 7f54d6200640 1 --2- 172.21.15.47:0/2013005081 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54d0157c00 0x7f54d014b620 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.482+0000 7f54d6200640 1 --2- 172.21.15.47:0/2013005081 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54d00cfbc0 0x7f54d015b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.482+0000 7f54d6200640 1 --2- 172.21.15.47:0/2013005081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54d00ce8b0 0x7f54d0153790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.482+0000 7f54d6200640 1 -- 172.21.15.47:0/2013005081 >> 172.21.15.47:0/2013005081 conn(0x7f54d00b1950 msgr2=0x7f54d0154900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:12.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.482+0000 7f80f3e00640 1 -- 172.21.15.47:0/4029544555 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f80d0066e70 msgr2=0x7f80d0069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.482+0000 7f80f3e00640 1 --2- 172.21.15.47:0/4029544555 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f80d0066e70 0x7f80d0069330 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f80dc006aa0 tx=0x7f80dc006a30 comp rx=0 tx=0).stop 2024-02-19T15:22:12.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.482+0000 7f80f3e00640 1 -- 172.21.15.47:0/4029544555 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80f40b57b0 msgr2=0x7f80f406f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.482+0000 7f80f3e00640 1 --2- 172.21.15.47:0/4029544555 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80f40b57b0 0x7f80f406f1b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f80e400c800 tx=0x7f80e400ccc0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.486+0000 7f80f3e00640 1 -- 172.21.15.47:0/4029544555 shutdown_connections 2024-02-19T15:22:12.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.486+0000 7f80f3e00640 1 --2- 172.21.15.47:0/4029544555 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f80d0066e70 0x7f80d0069330 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.486+0000 7f80f3e00640 1 --2- 172.21.15.47:0/4029544555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80f41585d0 0x7f80f41536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.486+0000 7f80f3e00640 1 --2- 172.21.15.47:0/4029544555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80f41581f0 0x7f80f406f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.486+0000 7f80f3e00640 1 --2- 172.21.15.47:0/4029544555 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80f40b57b0 0x7f80f406f1b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.486+0000 7f80f3e00640 1 -- 172.21.15.47:0/4029544555 >> 172.21.15.47:0/4029544555 conn(0x7f80f40b24f0 msgr2=0x7f80f415f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:12.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.486+0000 7f80f3e00640 1 -- 172.21.15.47:0/4029544555 shutdown_connections 2024-02-19T15:22:12.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.486+0000 7f80f3e00640 1 -- 172.21.15.47:0/4029544555 wait complete. 2024-02-19T15:22:12.489 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.1 2024-02-19T15:22:12.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.486+0000 7efcbe400640 1 -- 172.21.15.47:0/1480217012 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7efcb00057f0 con 0x7efcb00540b0 2024-02-19T15:22:12.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.486+0000 7f9563200640 1 -- 172.21.15.47:0/972515436 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9544066e90 msgr2=0x7f9544069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.486+0000 7f9563200640 1 --2- 172.21.15.47:0/972515436 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9544066e90 0x7f9544069350 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f95500ae000 tx=0x7f95500a7000 comp rx=0 tx=0).stop 2024-02-19T15:22:12.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.486+0000 7f9563200640 1 -- 172.21.15.47:0/972515436 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f955c009d80 msgr2=0x7f955c17c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.486+0000 7f9563200640 1 --2- 172.21.15.47:0/972515436 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f955c009d80 0x7f955c17c850 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f955400c420 tx=0x7f955400c8e0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.486+0000 7f54d6200640 1 -- 172.21.15.47:0/2013005081 shutdown_connections 2024-02-19T15:22:12.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.486+0000 7efcb4c00640 1 -- 172.21.15.47:0/1480217012 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efcb00e56f0 con 0x7efcb00540b0 2024-02-19T15:22:12.491 INFO:teuthology.orchestra.run.smithi047.stdout:60129542148 2024-02-19T15:22:12.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.486+0000 7f54d6200640 1 -- 172.21.15.47:0/2013005081 wait complete. 2024-02-19T15:22:12.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7f0ab6a00640 1 -- 172.21.15.47:0/3213141509 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f0a8c005550 con 0x7f0ab0145320 2024-02-19T15:22:12.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.486+0000 7f9563200640 1 -- 172.21.15.47:0/972515436 shutdown_connections 2024-02-19T15:22:12.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7f9563200640 1 --2- 172.21.15.47:0/972515436 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9544066e90 0x7f9544069350 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7f9563200640 1 --2- 172.21.15.47:0/972515436 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f955c0e10f0 0x7f955c188560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7f9563200640 1 --2- 172.21.15.47:0/972515436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f955c0557f0 0x7f955c0e2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7f9563200640 1 --2- 172.21.15.47:0/972515436 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f955c009d80 0x7f955c17c850 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7f9563200640 1 -- 172.21.15.47:0/972515436 >> 172.21.15.47:0/972515436 conn(0x7f955c056cf0 msgr2=0x7f955c0376c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:12.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7f9563200640 1 -- 172.21.15.47:0/972515436 shutdown_connections 2024-02-19T15:22:12.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7f9563200640 1 -- 172.21.15.47:0/972515436 wait complete. 2024-02-19T15:22:12.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7f0aad600640 1 -- 172.21.15.47:0/3213141509 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0aa8009dc0 con 0x7f0ab0145320 2024-02-19T15:22:12.493 INFO:teuthology.orchestra.run.smithi047.stdout:60129542148 2024-02-19T15:22:12.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7fb43ee00640 1 -- 172.21.15.47:0/2850814164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fb43815ef10 con 0x7fb43815de20 2024-02-19T15:22:12.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7efcbe400640 1 -- 172.21.15.47:0/1480217012 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc98066f10 msgr2=0x7efc980693d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7efcbe400640 1 --2- 172.21.15.47:0/1480217012 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc98066f10 0x7efc980693d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efcac000f70 tx=0x7efcac0079c0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7efcbe400640 1 -- 172.21.15.47:0/1480217012 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcb00540b0 msgr2=0x7efcb00d9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7efcbe400640 1 --2- 172.21.15.47:0/1480217012 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcb00540b0 0x7efcb00d9be0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efca400c450 tx=0x7efca400c910 comp rx=0 tx=0).stop 2024-02-19T15:22:12.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7efcbe400640 1 -- 172.21.15.47:0/1480217012 shutdown_connections 2024-02-19T15:22:12.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7efcbe400640 1 --2- 172.21.15.47:0/1480217012 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc98066f10 0x7efc980693d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7efcbe400640 1 --2- 172.21.15.47:0/1480217012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb00e4b80 0x7efcb00da120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7efcbe400640 1 --2- 172.21.15.47:0/1480217012 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcb00540b0 0x7efcb00d9be0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7efcbe400640 1 --2- 172.21.15.47:0/1480217012 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcb0053c20 0x7efcb00e2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7efcbe400640 1 -- 172.21.15.47:0/1480217012 >> 172.21.15.47:0/1480217012 conn(0x7efcb00559f0 msgr2=0x7efcb0039f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:12.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7efcbe400640 1 -- 172.21.15.47:0/1480217012 shutdown_connections 2024-02-19T15:22:12.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7fb42fe00640 1 -- 172.21.15.47:0/2850814164 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb430017020 con 0x7fb43815de20 2024-02-19T15:22:12.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7f0ab6a00640 1 -- 172.21.15.47:0/3213141509 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a90066f40 msgr2=0x7f0a90069400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7f0ab6a00640 1 --2- 172.21.15.47:0/3213141509 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a90066f40 0x7f0a90069400 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4000fc0 tx=0x7f0aa4009040 comp rx=0 tx=0).stop 2024-02-19T15:22:12.495 INFO:teuthology.orchestra.run.smithi047.stdout:60129542149 2024-02-19T15:22:12.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7efcbe400640 1 -- 172.21.15.47:0/1480217012 wait complete. 2024-02-19T15:22:12.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7f0ab6a00640 1 -- 172.21.15.47:0/3213141509 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ab0145320 msgr2=0x7f0ab01457b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.490+0000 7f0ab6a00640 1 --2- 172.21.15.47:0/3213141509 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ab0145320 0x7f0ab01457b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8098880 tx=0x7f0aa8009750 comp rx=0 tx=0).stop 2024-02-19T15:22:12.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.494+0000 7f0ab6a00640 1 -- 172.21.15.47:0/3213141509 shutdown_connections 2024-02-19T15:22:12.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.494+0000 7f0ab6a00640 1 --2- 172.21.15.47:0/3213141509 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a90066f40 0x7f0a90069400 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.494+0000 7f0ab6a00640 1 --2- 172.21.15.47:0/3213141509 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ab0145320 0x7f0ab01457b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.494+0000 7f0ab6a00640 1 --2- 172.21.15.47:0/3213141509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ab00b2ef0 0x7f0ab0163bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.494+0000 7f0ab6a00640 1 --2- 172.21.15.47:0/3213141509 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a98005190 0x7f0ab0144b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.494+0000 7f0ab6a00640 1 -- 172.21.15.47:0/3213141509 >> 172.21.15.47:0/3213141509 conn(0x7f0ab00cf900 msgr2=0x7f0ab0147f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:12.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.494+0000 7f0ab6a00640 1 -- 172.21.15.47:0/3213141509 shutdown_connections 2024-02-19T15:22:12.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.494+0000 7f0ab6a00640 1 -- 172.21.15.47:0/3213141509 wait complete. 2024-02-19T15:22:12.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.494+0000 7fb43ee00640 1 -- 172.21.15.47:0/2850814164 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb418066e90 msgr2=0x7fb418069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.494+0000 7fb43ee00640 1 --2- 172.21.15.47:0/2850814164 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb418066e90 0x7fb418069350 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb428009730 tx=0x7fb428006ca0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.494+0000 7fb43ee00640 1 -- 172.21.15.47:0/2850814164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb43815de20 msgr2=0x7fb43806d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:12.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.494+0000 7fb43ee00640 1 --2- 172.21.15.47:0/2850814164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb43815de20 0x7fb43806d710 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb43000c780 tx=0x7fb43000cc40 comp rx=0 tx=0).stop 2024-02-19T15:22:12.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.498+0000 7fb43ee00640 1 -- 172.21.15.47:0/2850814164 shutdown_connections 2024-02-19T15:22:12.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.498+0000 7fb43ee00640 1 --2- 172.21.15.47:0/2850814164 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb418066e90 0x7fb418069350 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.498+0000 7fb43ee00640 1 --2- 172.21.15.47:0/2850814164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb43815de20 0x7fb43806d710 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.498+0000 7fb43ee00640 1 --2- 172.21.15.47:0/2850814164 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4380cc3a0 0x7fb43806d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.498+0000 7fb43ee00640 1 --2- 172.21.15.47:0/2850814164 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb438072890 0x7fb43814d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:12.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.498+0000 7fb43ee00640 1 -- 172.21.15.47:0/2850814164 >> 172.21.15.47:0/2850814164 conn(0x7fb4380b1950 msgr2=0x7fb4380ad840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:12.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.498+0000 7fb43ee00640 1 -- 172.21.15.47:0/2850814164 shutdown_connections 2024-02-19T15:22:12.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:12.498+0000 7fb43ee00640 1 -- 172.21.15.47:0/2850814164 wait complete. 2024-02-19T15:22:12.500 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.2 2024-02-19T15:22:12.520 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.5 2024-02-19T15:22:12.528 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.7 2024-02-19T15:22:12.535 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.6 2024-02-19T15:22:12.535 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.4 2024-02-19T15:22:12.537 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.0 2024-02-19T15:22:12.539 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.3 2024-02-19T15:22:13.490 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-02-19T15:22:13.501 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-02-19T15:22:13.520 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-02-19T15:22:13.529 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-02-19T15:22:13.535 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-02-19T15:22:13.536 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-02-19T15:22:13.538 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-02-19T15:22:13.539 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-02-19T15:22:13.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.702+0000 7fb823e00640 1 Processor -- start 2024-02-19T15:22:13.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.702+0000 7fb823e00640 1 -- start start 2024-02-19T15:22:13.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.706+0000 7fb823e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb8241431e0 0x7fb8240708a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.706+0000 7fb823e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb824070de0 0x7fb8240cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.706+0000 7fb822a00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb8241404b0 0x7fb82406d030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:55824/0 (socket says 172.21.15.47:55824) 2024-02-19T15:22:13.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.706+0000 7fb822000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb8241431e0 0x7fb8240708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.710+0000 7fb822000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb8241431e0 0x7fb8240708a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:36554/0 (socket says 172.21.15.47:36554) 2024-02-19T15:22:13.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.710+0000 7fb822000640 1 -- 172.21.15.47:0/3382170641 learned_addr learned my addr 172.21.15.47:0/3382170641 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.706+0000 7fb823400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb824070de0 0x7fb8240cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.710+0000 7fb823e00640 1 -- 172.21.15.47:0/3382170641 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb824140890 con 0x7fb8241404b0 2024-02-19T15:22:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.710+0000 7fb823e00640 1 -- 172.21.15.47:0/3382170641 --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb82406e1d0 con 0x7fb824070de0 2024-02-19T15:22:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.710+0000 7fb823e00640 1 -- 172.21.15.47:0/3382170641 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb82406d740 con 0x7fb8241431e0 2024-02-19T15:22:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.710+0000 7fb822000640 1 -- 172.21.15.47:0/3382170641 >> v2:172.21.15.47:3300/0 conn(0x7fb824070de0 msgr2=0x7fb8240cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.710+0000 7fb822000640 1 --2- 172.21.15.47:0/3382170641 >> v2:172.21.15.47:3300/0 conn(0x7fb824070de0 0x7fb8240cdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.710+0000 7fb822000640 1 -- 172.21.15.47:0/3382170641 >> v1:172.21.15.116:6789/0 conn(0x7fb8241404b0 legacy=0x7fb82406d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.710+0000 7fb822000640 1 -- 172.21.15.47:0/3382170641 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82406d930 con 0x7fb8241431e0 2024-02-19T15:22:13.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.714+0000 7fb822000640 1 --2- 172.21.15.47:0/3382170641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8241431e0 0x7fb8240708a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb824074ff0 tx=0x7fb81401ced0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61f22eec3e0600ee server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.714+0000 7fb821600640 1 -- 172.21.15.47:0/3382170641 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8140a5e70 con 0x7fb8241431e0 2024-02-19T15:22:13.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.714+0000 7fb821600640 1 -- 172.21.15.47:0/3382170641 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8140ad440 con 0x7fb8241431e0 2024-02-19T15:22:13.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.714+0000 7fb821600640 1 -- 172.21.15.47:0/3382170641 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8140ad7d0 con 0x7fb8241431e0 2024-02-19T15:22:13.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.714+0000 7fb823e00640 1 -- 172.21.15.47:0/3382170641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8241431e0 msgr2=0x7fb8240708a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.714+0000 7fb823e00640 1 --2- 172.21.15.47:0/3382170641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8241431e0 0x7fb8240708a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb824074ff0 tx=0x7fb81401ced0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.714+0000 7fb823e00640 1 -- 172.21.15.47:0/3382170641 shutdown_connections 2024-02-19T15:22:13.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.714+0000 7fb823e00640 1 --2- 172.21.15.47:0/3382170641 >> v2:172.21.15.47:3300/0 conn(0x7fb824070de0 0x7fb8240cdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.714+0000 7fb823e00640 1 --2- 172.21.15.47:0/3382170641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8241431e0 0x7fb8240708a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.714+0000 7fb823e00640 1 -- 172.21.15.47:0/3382170641 >> 172.21.15.47:0/3382170641 conn(0x7fb8240cf900 msgr2=0x7fb8240cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:13.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.714+0000 7fb823e00640 1 -- 172.21.15.47:0/3382170641 shutdown_connections 2024-02-19T15:22:13.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.714+0000 7fb823e00640 1 -- 172.21.15.47:0/3382170641 wait complete. 2024-02-19T15:22:13.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.718+0000 7fb823e00640 1 Processor -- start 2024-02-19T15:22:13.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.718+0000 7fb823e00640 1 -- start start 2024-02-19T15:22:13.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.722+0000 7fb823e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb824070de0 0x7fb82415b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.722+0000 7fb823e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8241404b0 0x7fb82415be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.722+0000 7fb823e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8241431e0 0x7fb824206a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.722+0000 7fb823e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82413d4d0 con 0x7fb8241404b0 2024-02-19T15:22:13.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.722+0000 7fb823e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82415e000 con 0x7fb8241431e0 2024-02-19T15:22:13.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.722+0000 7fb823e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb82415b2d0 con 0x7fb824070de0 2024-02-19T15:22:13.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.722+0000 7fb823400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8241431e0 0x7fb824206a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.722+0000 7fb822a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb824070de0 0x7fb82415b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.722+0000 7fb823400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8241431e0 0x7fb824206a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:36568/0 (socket says 172.21.15.47:36568) 2024-02-19T15:22:13.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.722+0000 7fb823400640 1 -- 172.21.15.47:0/3483540180 learned_addr learned my addr 172.21.15.47:0/3483540180 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:13.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.722+0000 7fb822000640 1 --2- 172.21.15.47:0/3483540180 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8241404b0 0x7fb82415be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.722+0000 7fb822a00640 1 -- 172.21.15.47:0/3483540180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8241431e0 msgr2=0x7fb824206a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.722+0000 7fb822a00640 1 --2- 172.21.15.47:0/3483540180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8241431e0 0x7fb824206a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.722+0000 7fb822a00640 1 -- 172.21.15.47:0/3483540180 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8241404b0 msgr2=0x7fb82415be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.722+0000 7fb822a00640 1 --2- 172.21.15.47:0/3483540180 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8241404b0 0x7fb82415be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.722+0000 7fb822a00640 1 -- 172.21.15.47:0/3483540180 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb814007550 con 0x7fb824070de0 2024-02-19T15:22:13.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.722+0000 7fb822a00640 1 --2- 172.21.15.47:0/3483540180 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb824070de0 0x7fb82415b950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00c700 tx=0x7fb81c00cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.722+0000 7fb81aa00640 1 -- 172.21.15.47:0/3483540180 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c00ce10 con 0x7fb824070de0 2024-02-19T15:22:13.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.722+0000 7fb81aa00640 1 -- 172.21.15.47:0/3483540180 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81c00a640 con 0x7fb824070de0 2024-02-19T15:22:13.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.722+0000 7fb81aa00640 1 -- 172.21.15.47:0/3483540180 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c00a9d0 con 0x7fb824070de0 2024-02-19T15:22:13.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.726+0000 7fb823e00640 1 -- 172.21.15.47:0/3483540180 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb82415b5b0 con 0x7fb824070de0 2024-02-19T15:22:13.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.726+0000 7fb823e00640 1 -- 172.21.15.47:0/3483540180 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb82415dc90 con 0x7fb824070de0 2024-02-19T15:22:13.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.726+0000 7fb823e00640 1 -- 172.21.15.47:0/3483540180 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb824075360 con 0x7fb824070de0 2024-02-19T15:22:13.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.730+0000 7fb81aa00640 1 -- 172.21.15.47:0/3483540180 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb81c00ab30 con 0x7fb824070de0 2024-02-19T15:22:13.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.730+0000 7fb81aa00640 1 --2- 172.21.15.47:0/3483540180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb804066ec0 0x7fb804069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.730+0000 7fb822000640 1 --2- 172.21.15.47:0/3483540180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb804066ec0 0x7fb804069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.730+0000 7fb81aa00640 1 -- 172.21.15.47:0/3483540180 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb81c054e00 con 0x7fb824070de0 2024-02-19T15:22:13.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.734+0000 7fb822000640 1 --2- 172.21.15.47:0/3483540180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb804066ec0 0x7fb804069380 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb8140af000 tx=0x7fb8140a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.738+0000 7fb81aa00640 1 -- 172.21.15.47:0/3483540180 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb81c051830 con 0x7fb824070de0 2024-02-19T15:22:13.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.782+0000 7f2ce6600640 1 Processor -- start 2024-02-19T15:22:13.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.782+0000 7f2ce6600640 1 -- start start 2024-02-19T15:22:13.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f7691e00640 1 Processor -- start 2024-02-19T15:22:13.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f2ce6600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2cd8051820 0x7f2cd8007fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f2ce6600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cd80381b0 0x7f2cd8038640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f2ce6600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd8050660 con 0x7f2cd80e7550 2024-02-19T15:22:13.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f2ce6600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd8050c20 con 0x7f2cd8051820 2024-02-19T15:22:13.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f2ce6600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd8050940 con 0x7f2cd80381b0 2024-02-19T15:22:13.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f2cdea00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2cd8051820 0x7f2cd8007fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f2cdf400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f2cd80e7550 0x7f2cd8054040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35318/0 (socket says 172.21.15.47:35318) 2024-02-19T15:22:13.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f2cdf400640 1 -- 172.21.15.47:0/535681885 learned_addr learned my addr 172.21.15.47:0/535681885 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:13.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f2cdfe00640 1 --2- 172.21.15.47:0/535681885 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cd80381b0 0x7f2cd8038640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f2cdea00640 1 -- 172.21.15.47:0/535681885 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cd80381b0 msgr2=0x7f2cd8038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:13.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f2cdea00640 1 --2- 172.21.15.47:0/535681885 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cd80381b0 0x7f2cd8038640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f2cdea00640 1 -- 172.21.15.47:0/535681885 >> v1:172.21.15.47:6789/0 conn(0x7f2cd80e7550 legacy=0x7f2cd8054040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f2cdea00640 1 -- 172.21.15.47:0/535681885 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd8048760 con 0x7f2cd8051820 2024-02-19T15:22:13.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f7691e00640 1 -- start start 2024-02-19T15:22:13.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7fa82be00640 1 Processor -- start 2024-02-19T15:22:13.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f18c7000640 1 Processor -- start 2024-02-19T15:22:13.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7fa82be00640 1 -- start start 2024-02-19T15:22:13.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f7691e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f768c0b21c0 0x7f768c070f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f7691e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c0714d0 con 0x7f768c0cf0e0 2024-02-19T15:22:13.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f7691e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c0cf4c0 con 0x7f768c070b50 2024-02-19T15:22:13.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f7691e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f768c0b37e0 con 0x7f768c0b21c0 2024-02-19T15:22:13.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.786+0000 7f18c7000640 1 -- start start 2024-02-19T15:22:13.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f768aa00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f768c0cf0e0 0x7f768c0b30d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35320/0 (socket says 172.21.15.47:35320) 2024-02-19T15:22:13.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f768aa00640 1 -- 172.21.15.47:0/1733382224 learned_addr learned my addr 172.21.15.47:0/1733382224 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:13.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f18c7000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18c01635c0 0x7f18c01628c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f18c7000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18c00b2cc0 0x7f18c0070940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f18c7000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c00b3e20 con 0x7f18c0070e80 2024-02-19T15:22:13.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f18c7000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c0071310 con 0x7f18c01635c0 2024-02-19T15:22:13.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f18c7000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c00b3210 con 0x7f18c00b2cc0 2024-02-19T15:22:13.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f7689600640 1 -- 172.21.15.47:0/1733382224 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942450650 0 0) 0x7f768c0714d0 con 0x7f768c0cf0e0 2024-02-19T15:22:13.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f7689600640 1 -- 172.21.15.47:0/1733382224 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7670003530 con 0x7f768c0cf0e0 2024-02-19T15:22:13.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7fa82be00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa8240b5d90 0x7fa8240cc350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f768b400640 1 --2- 172.21.15.47:0/1733382224 >> v2:172.21.15.47:3300/0 conn(0x7f768c0b21c0 0x7f768c070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7fa829400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa82415d9d0 0x7fa82406bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35334/0 (socket says 172.21.15.47:35334) 2024-02-19T15:22:13.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7fa829400640 1 -- 172.21.15.47:0/3122662049 learned_addr learned my addr 172.21.15.47:0/3122662049 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:13.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7fa82be00640 1 -- 172.21.15.47:0/3122662049 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82406bac0 con 0x7fa82415d9d0 2024-02-19T15:22:13.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f18bfe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18c01635c0 0x7f18c01628c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7fa82be00640 1 -- 172.21.15.47:0/3122662049 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82406aeb0 con 0x7fa8240ad140 2024-02-19T15:22:13.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7fa828a00640 1 --2- 172.21.15.47:0/3122662049 >> v2:172.21.15.116:3300/0 conn(0x7fa8240b5d90 0x7fa8240cc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7fa82be00640 1 -- 172.21.15.47:0/3122662049 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa82406b190 con 0x7fa8240b5d90 2024-02-19T15:22:13.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f18bfe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18c01635c0 0x7f18c01628c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:36600/0 (socket says 172.21.15.47:36600) 2024-02-19T15:22:13.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f18bfe00640 1 -- 172.21.15.47:0/1681053671 learned_addr learned my addr 172.21.15.47:0/1681053671 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:13.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f18bf400640 1 --2- 172.21.15.47:0/1681053671 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18c00b2cc0 0x7f18c0070940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f7689600640 1 -- 172.21.15.47:0/1733382224 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265039962 0 0) 0x7f768c0cf4c0 con 0x7f768c070b50 2024-02-19T15:22:13.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f7689600640 1 -- 172.21.15.47:0/1733382224 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f768c0714d0 con 0x7f768c070b50 2024-02-19T15:22:13.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f7689600640 1 -- 172.21.15.47:0/1733382224 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2703115456 0 0) 0x7f7670003530 con 0x7f768c0cf0e0 2024-02-19T15:22:13.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f7689600640 1 -- 172.21.15.47:0/1733382224 >> v2:172.21.15.47:3300/0 conn(0x7f768c0b21c0 msgr2=0x7f768c070f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:13.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f7689600640 1 --2- 172.21.15.47:0/1733382224 >> v2:172.21.15.47:3300/0 conn(0x7f768c0b21c0 0x7f768c070f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f7689600640 1 -- 172.21.15.47:0/1733382224 >> v1:172.21.15.116:6789/0 conn(0x7f768c070b50 legacy=0x7f768c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f7689600640 1 -- 172.21.15.47:0/1733382224 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c0b3ab0 con 0x7f768c0cf0e0 2024-02-19T15:22:13.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f7689600640 1 -- 172.21.15.47:0/1733382224 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f767c001a00 con 0x7f768c0cf0e0 2024-02-19T15:22:13.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f7689600640 1 -- 172.21.15.47:0/1733382224 >> v1:172.21.15.47:6789/0 conn(0x7f768c0cf0e0 legacy=0x7f768c0b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f7689600640 1 --2- 172.21.15.47:0/1733382224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7670003b60 0x7f7670003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f7689600640 1 --2- 172.21.15.47:0/1733382224 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76700047d0 0x7f7670004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f7689600640 1 --2- 172.21.15.47:0/1733382224 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76700051e0 0x7f767000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f7689600640 1 -- 172.21.15.47:0/1733382224 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c0b37e0 con 0x7f76700047d0 2024-02-19T15:22:13.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f7689600640 1 -- 172.21.15.47:0/1733382224 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767000c110 con 0x7f7670003b60 2024-02-19T15:22:13.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f7689600640 1 -- 172.21.15.47:0/1733382224 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f767000c2d0 con 0x7f76700051e0 2024-02-19T15:22:13.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f2cdea00640 1 --2- 172.21.15.47:0/535681885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cd8051820 0x7f2cd8007fe0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2cd40097c0 tx=0x7f2cd401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40132e09561b31ec server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f768aa00640 1 --2- 172.21.15.47:0/1733382224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7670003b60 0x7f7670003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f768b400640 1 --2- 172.21.15.47:0/1733382224 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76700051e0 0x7f767000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f768a000640 1 --2- 172.21.15.47:0/1733382224 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76700047d0 0x7f7670004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f2cde000640 1 -- 172.21.15.47:0/535681885 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd40ab600 con 0x7f2cd8051820 2024-02-19T15:22:13.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f2cde000640 1 -- 172.21.15.47:0/535681885 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd40a7050 con 0x7f2cd8051820 2024-02-19T15:22:13.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f2cde000640 1 -- 172.21.15.47:0/535681885 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd40ab980 con 0x7f2cd8051820 2024-02-19T15:22:13.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7fa828a00640 1 -- 172.21.15.47:0/3122662049 >> v1:172.21.15.116:6789/0 conn(0x7fa8240ad140 legacy=0x7fa8240b1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7fa828a00640 1 -- 172.21.15.47:0/3122662049 >> v1:172.21.15.47:6789/0 conn(0x7fa82415d9d0 legacy=0x7fa82406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7fa828a00640 1 -- 172.21.15.47:0/3122662049 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82406b470 con 0x7fa8240b5d90 2024-02-19T15:22:13.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f2ce6600640 1 -- 172.21.15.47:0/535681885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cd8051820 msgr2=0x7f2cd8007fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f2ce6600640 1 --2- 172.21.15.47:0/535681885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cd8051820 0x7f2cd8007fe0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2cd40097c0 tx=0x7f2cd401c4a0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f2ce6600640 1 -- 172.21.15.47:0/535681885 shutdown_connections 2024-02-19T15:22:13.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f2ce6600640 1 --2- 172.21.15.47:0/535681885 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cd80381b0 0x7f2cd8038640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f2ce6600640 1 --2- 172.21.15.47:0/535681885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cd8051820 0x7f2cd8007fe0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f2ce6600640 1 -- 172.21.15.47:0/535681885 >> 172.21.15.47:0/535681885 conn(0x7f2cd8055360 msgr2=0x7f2cd8055770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:13.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f2ce6600640 1 -- 172.21.15.47:0/535681885 shutdown_connections 2024-02-19T15:22:13.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f2ce6600640 1 -- 172.21.15.47:0/535681885 wait complete. 2024-02-19T15:22:13.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f18bfe00640 1 -- 172.21.15.47:0/1681053671 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18c00b2cc0 msgr2=0x7f18c0070940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:13.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f18bfe00640 1 --2- 172.21.15.47:0/1681053671 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18c00b2cc0 0x7f18c0070940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f18bfe00640 1 -- 172.21.15.47:0/1681053671 >> v1:172.21.15.116:6789/0 conn(0x7f18c0070e80 legacy=0x7f18c0165eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:13.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.790+0000 7f18bfe00640 1 -- 172.21.15.47:0/1681053671 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c00b34f0 con 0x7f18c01635c0 2024-02-19T15:22:13.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f768aa00640 1 -- 172.21.15.47:0/1733382224 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76700051e0 msgr2=0x7f767000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:13.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f768aa00640 1 --2- 172.21.15.47:0/1733382224 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76700051e0 0x7f767000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f768aa00640 1 -- 172.21.15.47:0/1733382224 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76700047d0 msgr2=0x7f7670004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:13.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f768aa00640 1 --2- 172.21.15.47:0/1733382224 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76700047d0 0x7f7670004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f768aa00640 1 -- 172.21.15.47:0/1733382224 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7670003530 con 0x7f7670003b60 2024-02-19T15:22:13.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f768a000640 1 --2- 172.21.15.47:0/1733382224 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76700047d0 0x7f7670004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:22:13.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.794+0000 7f768b400640 1 -- 172.21.15.47:0/1733382224 reap_dead start 2024-02-19T15:22:13.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f18bfe00640 1 --2- 172.21.15.47:0/1681053671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c01635c0 0x7f18c01628c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f18ac009870 tx=0x7f18ac01c4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d433e9efe7d2f48 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f18bea00640 1 -- 172.21.15.47:0/1681053671 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ac0ac630 con 0x7f18c01635c0 2024-02-19T15:22:13.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f5a03800640 1 Processor -- start 2024-02-19T15:22:13.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f18bea00640 1 -- 172.21.15.47:0/1681053671 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18ac0ac790 con 0x7f18c01635c0 2024-02-19T15:22:13.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f18bea00640 1 -- 172.21.15.47:0/1681053671 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ac0acb20 con 0x7f18c01635c0 2024-02-19T15:22:13.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f768aa00640 1 --2- 172.21.15.47:0/1733382224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7670003b60 0x7f7670003f60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f767c002240 tx=0x7f767c01d220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68ee20e3afaba023 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f5a03800640 1 -- start start 2024-02-19T15:22:13.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f2ce6600640 1 Processor -- start 2024-02-19T15:22:13.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f18c7000640 1 -- 172.21.15.47:0/1681053671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c01635c0 msgr2=0x7f18c01628c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f18c7000640 1 --2- 172.21.15.47:0/1681053671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c01635c0 0x7f18c01628c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f18ac009870 tx=0x7f18ac01c4f0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f7689600640 1 -- 172.21.15.47:0/1733382224 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c0ad630 con 0x7f7670003b60 2024-02-19T15:22:13.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f7689600640 1 -- 172.21.15.47:0/1733382224 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767c0a8070 con 0x7f7670003b60 2024-02-19T15:22:13.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f7689600640 1 -- 172.21.15.47:0/1733382224 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c0ada70 con 0x7f7670003b60 2024-02-19T15:22:13.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f2ce6600640 1 -- start start 2024-02-19T15:22:13.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f5a03800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f59fc0b2cc0 0x7f59fc1628c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f7691e00640 1 -- 172.21.15.47:0/1733382224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7670003b60 msgr2=0x7f7670003f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f7691e00640 1 --2- 172.21.15.47:0/1733382224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7670003b60 0x7f7670003f60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f767c002240 tx=0x7f767c01d220 comp rx=0 tx=0).stop 2024-02-19T15:22:13.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f5a03800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59fc0b3e20 con 0x7f59fc071050 2024-02-19T15:22:13.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f5a03800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59fc0b3210 con 0x7f59fc1635c0 2024-02-19T15:22:13.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f5a03800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59fc0b34f0 con 0x7f59fc0b2cc0 2024-02-19T15:22:13.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f18c7000640 1 -- 172.21.15.47:0/1681053671 shutdown_connections 2024-02-19T15:22:13.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f18c7000640 1 --2- 172.21.15.47:0/1681053671 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18c00b2cc0 0x7f18c0070940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f18c7000640 1 --2- 172.21.15.47:0/1681053671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c01635c0 0x7f18c01628c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f18c7000640 1 -- 172.21.15.47:0/1681053671 >> 172.21.15.47:0/1681053671 conn(0x7f18c00b16a0 msgr2=0x7f18c00b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:13.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f18c7000640 1 -- 172.21.15.47:0/1681053671 shutdown_connections 2024-02-19T15:22:13.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f7691e00640 1 -- 172.21.15.47:0/1733382224 shutdown_connections 2024-02-19T15:22:13.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7fa828a00640 1 --2- 172.21.15.47:0/3122662049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8240b5d90 0x7fa8240cc350 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa81800a960 tx=0x7fa81801d720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f814bc81d278d42f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f7691e00640 1 --2- 172.21.15.47:0/1733382224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7670003b60 0x7f7670003f60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f7691e00640 1 -- 172.21.15.47:0/1733382224 >> 172.21.15.47:0/1733382224 conn(0x7f768c0b16a0 msgr2=0x7f768c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:13.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f18c7000640 1 -- 172.21.15.47:0/1681053671 wait complete. 2024-02-19T15:22:13.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f7691e00640 1 -- 172.21.15.47:0/1733382224 shutdown_connections 2024-02-19T15:22:13.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f5a00e00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f59fc1635c0 0x7f59fc070940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:55860/0 (socket says 172.21.15.47:55860) 2024-02-19T15:22:13.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7fa823e00640 1 -- 172.21.15.47:0/3122662049 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8180a6810 con 0x7fa8240b5d90 2024-02-19T15:22:13.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f7691e00640 1 -- 172.21.15.47:0/1733382224 wait complete. 2024-02-19T15:22:13.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7fa823e00640 1 -- 172.21.15.47:0/3122662049 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8180a6970 con 0x7fa8240b5d90 2024-02-19T15:22:13.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7fa823e00640 1 -- 172.21.15.47:0/3122662049 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8180a6d00 con 0x7fa8240b5d90 2024-02-19T15:22:13.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f5a00e00640 1 -- 172.21.15.47:0/2950582399 learned_addr learned my addr 172.21.15.47:0/2950582399 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:13.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.798+0000 7f5a01800640 1 --2- 172.21.15.47:0/2950582399 >> v2:172.21.15.47:3300/0 conn(0x7f59fc0b2cc0 0x7f59fc1628c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f5a01800640 1 -- 172.21.15.47:0/2950582399 >> v1:172.21.15.116:6789/0 conn(0x7f59fc1635c0 legacy=0x7f59fc070940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f5a01800640 1 -- 172.21.15.47:0/2950582399 >> v1:172.21.15.47:6789/0 conn(0x7f59fc071050 legacy=0x7f59fc06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f5a01800640 1 -- 172.21.15.47:0/2950582399 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc0b37d0 con 0x7f59fc0b2cc0 2024-02-19T15:22:13.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f2ce6600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cd80381b0 0x7f2cd80e9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7fa82be00640 1 -- 172.21.15.47:0/3122662049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8240b5d90 msgr2=0x7fa8240cc350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7fa82be00640 1 --2- 172.21.15.47:0/3122662049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8240b5d90 0x7fa8240cc350 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa81800a960 tx=0x7fa81801d720 comp rx=0 tx=0).stop 2024-02-19T15:22:13.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f2ce6600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2cd8051820 0x7f2cd80d6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f2cdf400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cd80381b0 0x7f2cd80e9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f2cdf400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cd80381b0 0x7f2cd80e9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:47434/0 (socket says 172.21.15.47:47434) 2024-02-19T15:22:13.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f2cdf400640 1 -- 172.21.15.47:0/4242113231 learned_addr learned my addr 172.21.15.47:0/4242113231 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:13.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f41c1600640 1 Processor -- start 2024-02-19T15:22:13.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f2ce6600640 1 --2- 172.21.15.47:0/4242113231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cd80e7550 0x7f2cd804edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7fa82be00640 1 -- 172.21.15.47:0/3122662049 shutdown_connections 2024-02-19T15:22:13.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7fa82be00640 1 --2- 172.21.15.47:0/3122662049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8240b5d90 0x7fa8240cc350 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7fa82be00640 1 -- 172.21.15.47:0/3122662049 >> 172.21.15.47:0/3122662049 conn(0x7fa8240b4fc0 msgr2=0x7fa8240b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:13.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f2cdfe00640 1 --2- 172.21.15.47:0/4242113231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cd80e7550 0x7f2cd804edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f2ce6600640 1 -- 172.21.15.47:0/4242113231 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd80e7fd0 con 0x7f2cd8051820 2024-02-19T15:22:13.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f2ce6600640 1 -- 172.21.15.47:0/4242113231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd80e82b0 con 0x7f2cd80e7550 2024-02-19T15:22:13.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f2ce6600640 1 -- 172.21.15.47:0/4242113231 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd80e8590 con 0x7f2cd80381b0 2024-02-19T15:22:13.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f2cdfe00640 1 -- 172.21.15.47:0/4242113231 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cd80381b0 msgr2=0x7f2cd80e9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f2cdfe00640 1 --2- 172.21.15.47:0/4242113231 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cd80381b0 0x7f2cd80e9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f2cdfe00640 1 -- 172.21.15.47:0/4242113231 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2cd8051820 msgr2=0x7f2cd80d6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f2cdfe00640 1 --2- 172.21.15.47:0/4242113231 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2cd8051820 0x7f2cd80d6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f2cdfe00640 1 -- 172.21.15.47:0/4242113231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd4009420 con 0x7f2cd80e7550 2024-02-19T15:22:13.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f7691e00640 1 Processor -- start 2024-02-19T15:22:13.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f7691e00640 1 -- start start 2024-02-19T15:22:13.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f41c1600640 1 -- start start 2024-02-19T15:22:13.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f41c1600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f41bc0b1ee0 0x7f41bc0720c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f41c1600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f41bc0cf000 0x7f41bc072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f41c1600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41bc06e6a0 con 0x7f41bc15ba70 2024-02-19T15:22:13.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f41c1600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41bc06e4a0 con 0x7f41bc0cf000 2024-02-19T15:22:13.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f41c1600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41bc06e980 con 0x7f41bc0b1ee0 2024-02-19T15:22:13.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f41c0c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f41bc0cf000 0x7f41bc072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f41c0c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f41bc0cf000 0x7f41bc072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57824/0 (socket says 172.21.15.47:57824) 2024-02-19T15:22:13.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7f41c0c00640 1 -- 172.21.15.47:0/1696377076 learned_addr learned my addr 172.21.15.47:0/1696377076 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:13.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.802+0000 7fa82be00640 1 -- 172.21.15.47:0/3122662049 shutdown_connections 2024-02-19T15:22:13.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f41bb400640 1 --2- 172.21.15.47:0/1696377076 >> v2:172.21.15.116:3300/0 conn(0x7f41bc0b1ee0 0x7f41bc0720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f2cdfe00640 1 --2- 172.21.15.47:0/4242113231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cd80e7550 0x7f2cd804edc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2ce00500b0 tx=0x7f2ce0068ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f2cd3400640 1 -- 172.21.15.47:0/4242113231 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0093630 con 0x7f2cd80e7550 2024-02-19T15:22:13.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f2cd3400640 1 -- 172.21.15.47:0/4242113231 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce0093790 con 0x7f2cd80e7550 2024-02-19T15:22:13.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f2cd3400640 1 -- 172.21.15.47:0/4242113231 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0093b20 con 0x7f2cd80e7550 2024-02-19T15:22:13.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f7691e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7670003b60 0x7f768c150cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f7691e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f768c143080 0x7f768c1f63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f7691e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f768c143500 0x7f768c1f6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f2ce6600640 1 -- 172.21.15.47:0/4242113231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd80e8870 con 0x7f2cd80e7550 2024-02-19T15:22:13.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f2ce6600640 1 -- 172.21.15.47:0/4242113231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd804f300 con 0x7f2cd80e7550 2024-02-19T15:22:13.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f768aa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7670003b60 0x7f768c150cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f768aa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7670003b60 0x7f768c150cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57828/0 (socket says 172.21.15.47:57828) 2024-02-19T15:22:13.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f7691e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c16a9c0 con 0x7f7670003b60 2024-02-19T15:22:13.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f7691e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c16aca0 con 0x7f768c143080 2024-02-19T15:22:13.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f7691e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768c16af80 con 0x7f768c143500 2024-02-19T15:22:13.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f768b400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f768c143500 0x7f768c1f6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f768b400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f768c143500 0x7f768c1f6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:47438/0 (socket says 172.21.15.47:47438) 2024-02-19T15:22:13.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f768b400640 1 -- 172.21.15.47:0/1209948131 learned_addr learned my addr 172.21.15.47:0/1209948131 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:13.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f768a000640 1 --2- 172.21.15.47:0/1209948131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f768c143080 0x7f768c1f63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f5a01800640 1 --2- 172.21.15.47:0/2950582399 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59fc0b2cc0 0x7f59fc1628c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0097c0 tx=0x7f59ec01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bd50dd251aa7d9d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7fa82be00640 1 -- 172.21.15.47:0/3122662049 wait complete. 2024-02-19T15:22:13.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f41bb400640 1 -- 172.21.15.47:0/1696377076 >> v2:172.21.15.47:3300/0 conn(0x7f41bc0cf000 msgr2=0x7f41bc072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:13.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f41bb400640 1 --2- 172.21.15.47:0/1696377076 >> v2:172.21.15.47:3300/0 conn(0x7f41bc0cf000 0x7f41bc072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f41bb400640 1 -- 172.21.15.47:0/1696377076 >> v1:172.21.15.47:6789/0 conn(0x7f41bc15ba70 legacy=0x7f41bc06dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f41bb400640 1 -- 172.21.15.47:0/1696377076 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41bc0b1400 con 0x7f41bc0b1ee0 2024-02-19T15:22:13.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7fa82be00640 1 Processor -- start 2024-02-19T15:22:13.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f59fb400640 1 -- 172.21.15.47:0/2950582399 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ec0ab600 con 0x7f59fc0b2cc0 2024-02-19T15:22:13.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7fa82be00640 1 -- start start 2024-02-19T15:22:13.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f59fb400640 1 -- 172.21.15.47:0/2950582399 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59ec0ab760 con 0x7f59fc0b2cc0 2024-02-19T15:22:13.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f59fb400640 1 -- 172.21.15.47:0/2950582399 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ec0ab600 con 0x7f59fc0b2cc0 2024-02-19T15:22:13.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f18c7000640 1 Processor -- start 2024-02-19T15:22:13.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f18c7000640 1 -- start start 2024-02-19T15:22:13.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f768b400640 1 -- 172.21.15.47:0/1209948131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f768c143080 msgr2=0x7f768c1f63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f768b400640 1 --2- 172.21.15.47:0/1209948131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f768c143080 0x7f768c1f63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f768b400640 1 -- 172.21.15.47:0/1209948131 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7670003b60 msgr2=0x7f768c150cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f768b400640 1 --2- 172.21.15.47:0/1209948131 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7670003b60 0x7f768c150cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f768b400640 1 -- 172.21.15.47:0/1209948131 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c16b260 con 0x7f768c143500 2024-02-19T15:22:13.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f768aa00640 1 --2- 172.21.15.47:0/1209948131 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7670003b60 0x7f768c150cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:13.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f768b400640 1 --2- 172.21.15.47:0/1209948131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f768c143500 0x7f768c1f6930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f768c072920 tx=0x7f768400b950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.806+0000 7f2ce6600640 1 -- 172.21.15.47:0/4242113231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd80e8d90 con 0x7f2cd80e7550 2024-02-19T15:22:13.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f2cd3400640 1 -- 172.21.15.47:0/4242113231 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2ce0093c80 con 0x7f2cd80e7550 2024-02-19T15:22:13.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f7683400640 1 -- 172.21.15.47:0/1209948131 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7684014680 con 0x7f768c143500 2024-02-19T15:22:13.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f7683400640 1 -- 172.21.15.47:0/1209948131 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76840147e0 con 0x7f768c143500 2024-02-19T15:22:13.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f7683400640 1 -- 172.21.15.47:0/1209948131 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7684014bd0 con 0x7f768c143500 2024-02-19T15:22:13.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f7691e00640 1 -- 172.21.15.47:0/1209948131 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f768c15cb00 con 0x7f768c143500 2024-02-19T15:22:13.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f5a03800640 1 -- 172.21.15.47:0/2950582399 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59fc0b2cc0 msgr2=0x7f59fc1628c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f5a03800640 1 --2- 172.21.15.47:0/2950582399 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59fc0b2cc0 0x7f59fc1628c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0097c0 tx=0x7f59ec01c470 comp rx=0 tx=0).stop 2024-02-19T15:22:13.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7fa82be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8240ad140 0x7fa82414ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f5a03800640 1 -- 172.21.15.47:0/2950582399 shutdown_connections 2024-02-19T15:22:13.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f5a03800640 1 --2- 172.21.15.47:0/2950582399 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59fc0b2cc0 0x7f59fc1628c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f5a03800640 1 -- 172.21.15.47:0/2950582399 >> 172.21.15.47:0/2950582399 conn(0x7f59fc0b16a0 msgr2=0x7f59fc0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:13.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f7691e00640 1 -- 172.21.15.47:0/1209948131 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768c16c870 con 0x7f768c143500 2024-02-19T15:22:13.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f41bb400640 1 --2- 172.21.15.47:0/1696377076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41bc0b1ee0 0x7f41bc0720c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f41bc0b59a0 tx=0x7f41a801c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccaea213076df4ef server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7fa82be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8240b5d90 0x7fa824203120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f5a03800640 1 -- 172.21.15.47:0/2950582399 shutdown_connections 2024-02-19T15:22:13.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f5a03800640 1 -- 172.21.15.47:0/2950582399 wait complete. 2024-02-19T15:22:13.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7fa82be00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa82415d9d0 0x7fa8241529d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7fa82be00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa824151ca0 con 0x7fa8240b5d90 2024-02-19T15:22:13.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7fa82be00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa824207e00 con 0x7fa8240ad140 2024-02-19T15:22:13.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7fa82be00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa824151ef0 con 0x7fa82415d9d0 2024-02-19T15:22:13.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7fa829400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8240ad140 0x7fa82414ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7fa829400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8240ad140 0x7fa82414ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:36674/0 (socket says 172.21.15.47:36674) 2024-02-19T15:22:13.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7fa829400640 1 -- 172.21.15.47:0/658778839 learned_addr learned my addr 172.21.15.47:0/658778839 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:13.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f18c7000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c0070e80 0x7f18c00a9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f18c7000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18c00b2cc0 0x7f18c0156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f18c7000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18c01635c0 0x7f18c0156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f18c7000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c00ab180 con 0x7f18c01635c0 2024-02-19T15:22:13.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f18c7000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c00ab700 con 0x7f18c0070e80 2024-02-19T15:22:13.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f18c7000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c00aa8e0 con 0x7f18c00b2cc0 2024-02-19T15:22:13.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f2cd3400640 1 --2- 172.21.15.47:0/4242113231 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2cc0066ec0 0x7f2cc0069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f2cd3400640 1 -- 172.21.15.47:0/4242113231 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ce00ff520 con 0x7f2cd80e7550 2024-02-19T15:22:13.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f18bf400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18c00b2cc0 0x7f18c0156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f18bf400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18c00b2cc0 0x7f18c0156670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:47444/0 (socket says 172.21.15.47:47444) 2024-02-19T15:22:13.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f18bf400640 1 -- 172.21.15.47:0/936909365 learned_addr learned my addr 172.21.15.47:0/936909365 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:13.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7fa829e00640 1 --2- 172.21.15.47:0/658778839 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa82415d9d0 0x7fa8241529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7fa829400640 1 -- 172.21.15.47:0/658778839 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa82415d9d0 msgr2=0x7fa8241529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7fa828a00640 1 --2- 172.21.15.47:0/658778839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8240b5d90 0x7fa824203120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f18bfe00640 1 --2- 172.21.15.47:0/936909365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c0070e80 0x7f18c00a9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f7680c00640 1 -- 172.21.15.47:0/1209948131 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76600052a0 con 0x7f768c143500 2024-02-19T15:22:13.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f2cdf400640 1 --2- 172.21.15.47:0/4242113231 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2cc0066ec0 0x7f2cc0069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7fa829400640 1 --2- 172.21.15.47:0/658778839 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa82415d9d0 0x7fa8241529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7fa829400640 1 -- 172.21.15.47:0/658778839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8240b5d90 msgr2=0x7fa824203120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7fa829400640 1 --2- 172.21.15.47:0/658778839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8240b5d90 0x7fa824203120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7fa829400640 1 -- 172.21.15.47:0/658778839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81800a5c0 con 0x7fa8240ad140 2024-02-19T15:22:13.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7fa828a00640 1 --2- 172.21.15.47:0/658778839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8240b5d90 0x7fa824203120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:22:13.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f18c5000640 1 --2- 172.21.15.47:0/936909365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18c01635c0 0x7f18c0156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f18bf400640 1 -- 172.21.15.47:0/936909365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c0070e80 msgr2=0x7f18c00a9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f18bf400640 1 --2- 172.21.15.47:0/936909365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c0070e80 0x7f18c00a9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f18bf400640 1 -- 172.21.15.47:0/936909365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18c01635c0 msgr2=0x7f18c0156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f18bf400640 1 --2- 172.21.15.47:0/936909365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18c01635c0 0x7f18c0156bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7f18bf400640 1 -- 172.21.15.47:0/936909365 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ac0094d0 con 0x7f18c00b2cc0 2024-02-19T15:22:13.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7fa829e00640 1 --2- 172.21.15.47:0/658778839 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa82415d9d0 0x7fa8241529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:22:13.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.810+0000 7fa829400640 1 --2- 172.21.15.47:0/658778839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8240ad140 0x7fa82414ffd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa81000c400 tx=0x7fa81000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f18bf400640 1 --2- 172.21.15.47:0/936909365 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18c00b2cc0 0x7f18c0156670 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f18b000e930 tx=0x7f18b000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f2cdf400640 1 --2- 172.21.15.47:0/4242113231 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2cc0066ec0 0x7f2cc0069380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc007fd0 tx=0x7f2ccc007f20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f18b7e00640 1 -- 172.21.15.47:0/936909365 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b0009690 con 0x7f18c00b2cc0 2024-02-19T15:22:13.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f18b7e00640 1 -- 172.21.15.47:0/936909365 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b000ca00 con 0x7f18c00b2cc0 2024-02-19T15:22:13.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f18b7e00640 1 -- 172.21.15.47:0/936909365 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b000cd70 con 0x7f18c00b2cc0 2024-02-19T15:22:13.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f18c7000640 1 -- 172.21.15.47:0/936909365 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c00aabc0 con 0x7f18c00b2cc0 2024-02-19T15:22:13.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f18c7000640 1 -- 172.21.15.47:0/936909365 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c0157880 con 0x7f18c00b2cc0 2024-02-19T15:22:13.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f5a03800640 1 Processor -- start 2024-02-19T15:22:13.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f5a03800640 1 -- start start 2024-02-19T15:22:13.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f7683400640 1 -- 172.21.15.47:0/1209948131 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7684014d30 con 0x7f768c143500 2024-02-19T15:22:13.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f18c7000640 1 -- 172.21.15.47:0/936909365 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c00ab8b0 con 0x7f18c00b2cc0 2024-02-19T15:22:13.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f7683400640 1 --2- 172.21.15.47:0/1209948131 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f766c066f10 0x7f766c0693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f7683400640 1 -- 172.21.15.47:0/1209948131 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7684084e00 con 0x7f768c143500 2024-02-19T15:22:13.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f768aa00640 1 --2- 172.21.15.47:0/1209948131 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f766c066f10 0x7f766c0693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f5a03800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59fc071050 0x7f59fc163e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f5a03800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59fc0b2cc0 0x7f59fc150710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f18b7e00640 1 -- 172.21.15.47:0/936909365 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f18b0013470 con 0x7f18c00b2cc0 2024-02-19T15:22:13.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f5a03800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f59fc1635c0 0x7f59fc150c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f5a03800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0cc710 con 0x7f59fc0b2cc0 2024-02-19T15:22:13.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f5a03800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0ccdd0 con 0x7f59fc071050 2024-02-19T15:22:13.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f5a03800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0cc3c0 con 0x7f59fc1635c0 2024-02-19T15:22:13.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f59fbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59fc0b2cc0 0x7f59fc150710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f5a00e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59fc071050 0x7f59fc163e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f59fbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59fc0b2cc0 0x7f59fc150710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57852/0 (socket says 172.21.15.47:57852) 2024-02-19T15:22:13.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f59fbe00640 1 -- 172.21.15.47:0/1595215607 learned_addr learned my addr 172.21.15.47:0/1595215607 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:13.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f5a01800640 1 --2- 172.21.15.47:0/1595215607 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f59fc1635c0 0x7f59fc150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.814+0000 7f768aa00640 1 --2- 172.21.15.47:0/1209948131 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f766c066f10 0x7f766c0693d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f767c0ad000 tx=0x7f767c0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f5a00e00640 1 -- 172.21.15.47:0/1595215607 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f59fc1635c0 msgr2=0x7f59fc150c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f5a00e00640 1 --2- 172.21.15.47:0/1595215607 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f59fc1635c0 0x7f59fc150c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f5a00e00640 1 -- 172.21.15.47:0/1595215607 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59fc0b2cc0 msgr2=0x7f59fc150710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f5a00e00640 1 --2- 172.21.15.47:0/1595215607 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59fc0b2cc0 0x7f59fc150710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f5a00e00640 1 -- 172.21.15.47:0/1595215607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ec009420 con 0x7f59fc071050 2024-02-19T15:22:13.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f18b7e00640 1 --2- 172.21.15.47:0/936909365 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f18a0066e40 0x7f18a0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f5a00e00640 1 --2- 172.21.15.47:0/1595215607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59fc071050 0x7f59fc163e40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f59f400cad0 tx=0x7f59f400ced0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f18b7e00640 1 -- 172.21.15.47:0/936909365 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18b0084d00 con 0x7f18c00b2cc0 2024-02-19T15:22:13.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f18bfe00640 1 --2- 172.21.15.47:0/936909365 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f18a0066e40 0x7f18a0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f18bfe00640 1 --2- 172.21.15.47:0/936909365 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f18a0066e40 0x7f18a0069300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f18ac0ae000 tx=0x7f18ac0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f59f8c00640 1 -- 172.21.15.47:0/1595215607 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f400ab70 con 0x7f59fc071050 2024-02-19T15:22:13.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f59f8c00640 1 -- 172.21.15.47:0/1595215607 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f400acd0 con 0x7f59fc071050 2024-02-19T15:22:13.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f59f8c00640 1 -- 172.21.15.47:0/1595215607 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4012620 con 0x7f59fc071050 2024-02-19T15:22:13.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7fa821600640 1 -- 172.21.15.47:0/658778839 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa810005cf0 con 0x7fa8240ad140 2024-02-19T15:22:13.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7fa821600640 1 -- 172.21.15.47:0/658778839 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa810005e50 con 0x7fa8240ad140 2024-02-19T15:22:13.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7fa821600640 1 -- 172.21.15.47:0/658778839 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8100072d0 con 0x7fa8240ad140 2024-02-19T15:22:13.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f2cd3400640 1 -- 172.21.15.47:0/4242113231 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2cd80e8d90 con 0x7f2cd80e7550 2024-02-19T15:22:13.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f41baa00640 1 -- 172.21.15.47:0/1696377076 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a80ab600 con 0x7f41bc0b1ee0 2024-02-19T15:22:13.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f41baa00640 1 -- 172.21.15.47:0/1696377076 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a80a7050 con 0x7f41bc0b1ee0 2024-02-19T15:22:13.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f41baa00640 1 -- 172.21.15.47:0/1696377076 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a80ab980 con 0x7f41bc0b1ee0 2024-02-19T15:22:13.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f7683400640 1 -- 172.21.15.47:0/1209948131 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7684051bd0 con 0x7f768c143500 2024-02-19T15:22:13.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7fa82be00640 1 -- 172.21.15.47:0/658778839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8241521d0 con 0x7fa8240ad140 2024-02-19T15:22:13.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7fa82be00640 1 -- 172.21.15.47:0/658778839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa824150510 con 0x7fa8240ad140 2024-02-19T15:22:13.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f5a03800640 1 -- 172.21.15.47:0/1595215607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59fc0ccfe0 con 0x7f59fc071050 2024-02-19T15:22:13.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f5a03800640 1 -- 172.21.15.47:0/1595215607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59fc0ccb30 con 0x7f59fc071050 2024-02-19T15:22:13.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f41c1600640 1 -- 172.21.15.47:0/1696377076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41bc0b1ee0 msgr2=0x7f41bc0720c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f41c1600640 1 --2- 172.21.15.47:0/1696377076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41bc0b1ee0 0x7f41bc0720c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f41bc0b59a0 tx=0x7f41a801c3c0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f41c1600640 1 -- 172.21.15.47:0/1696377076 shutdown_connections 2024-02-19T15:22:13.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f41c1600640 1 --2- 172.21.15.47:0/1696377076 >> v2:172.21.15.47:3300/0 conn(0x7f41bc0cf000 0x7f41bc072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.818+0000 7f41c1600640 1 --2- 172.21.15.47:0/1696377076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41bc0b1ee0 0x7f41bc0720c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.822+0000 7f41c1600640 1 -- 172.21.15.47:0/1696377076 >> 172.21.15.47:0/1696377076 conn(0x7f41bc0cf7c0 msgr2=0x7f41bc0cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:13.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.822+0000 7f41c1600640 1 -- 172.21.15.47:0/1696377076 shutdown_connections 2024-02-19T15:22:13.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.822+0000 7f41c1600640 1 -- 172.21.15.47:0/1696377076 wait complete. 2024-02-19T15:22:13.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.822+0000 7fa82be00640 1 -- 172.21.15.47:0/658778839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa824075360 con 0x7fa8240ad140 2024-02-19T15:22:13.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.822+0000 7f59f8c00640 1 -- 172.21.15.47:0/1595215607 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f59f4007070 con 0x7f59fc071050 2024-02-19T15:22:13.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.822+0000 7f5a03800640 1 -- 172.21.15.47:0/1595215607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59fc075360 con 0x7f59fc071050 2024-02-19T15:22:13.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.822+0000 7f41c1600640 1 Processor -- start 2024-02-19T15:22:13.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f59f8c00640 1 --2- 172.21.15.47:0/1595215607 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f59d8066e40 0x7f59d8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f59f8c00640 1 -- 172.21.15.47:0/1595215607 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f59f40847f0 con 0x7f59fc071050 2024-02-19T15:22:13.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.822+0000 7f41c1600640 1 -- start start 2024-02-19T15:22:13.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.822+0000 7fa821600640 1 -- 172.21.15.47:0/658778839 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa810007430 con 0x7fa8240ad140 2024-02-19T15:22:13.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7fa821600640 1 --2- 172.21.15.47:0/658778839 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa804066f10 0x7fa8040693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f59fbe00640 1 --2- 172.21.15.47:0/1595215607 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f59d8066e40 0x7f59d8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f41c1600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41bc0b1ee0 0x7f41bc13f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f41c1600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f41bc0cf000 0x7f41bc0712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f41c1600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41bc15ba70 0x7f41bc0717e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7fa821600640 1 -- 172.21.15.47:0/658778839 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa810083f70 con 0x7fa8240ad140 2024-02-19T15:22:13.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7fa828a00640 1 --2- 172.21.15.47:0/658778839 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa804066f10 0x7fa8040693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f41c1600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc0b2e50 con 0x7f41bc0b1ee0 2024-02-19T15:22:13.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f41c1600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc0b2560 con 0x7f41bc15ba70 2024-02-19T15:22:13.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f41c1600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc15e9e0 con 0x7f41bc0cf000 2024-02-19T15:22:13.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f41bb400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f41bc0cf000 0x7f41bc0712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f41bb400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f41bc0cf000 0x7f41bc0712a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:47454/0 (socket says 172.21.15.47:47454) 2024-02-19T15:22:13.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f41bb400640 1 -- 172.21.15.47:0/1591647318 learned_addr learned my addr 172.21.15.47:0/1591647318 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:13.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f41c0c00640 1 --2- 172.21.15.47:0/1591647318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41bc15ba70 0x7f41bc0717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f59fbe00640 1 --2- 172.21.15.47:0/1595215607 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f59d8066e40 0x7f59d8069300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f59fc1597b0 tx=0x7f59e4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f18b7e00640 1 -- 172.21.15.47:0/936909365 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f18c00ab8b0 con 0x7f18c00b2cc0 2024-02-19T15:22:13.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f41c0c00640 1 -- 172.21.15.47:0/1591647318 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f41bc0cf000 msgr2=0x7f41bc0712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f41c0c00640 1 --2- 172.21.15.47:0/1591647318 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f41bc0cf000 0x7f41bc0712a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f41c0c00640 1 -- 172.21.15.47:0/1591647318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41bc0b1ee0 msgr2=0x7f41bc13f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f41c0c00640 1 --2- 172.21.15.47:0/1591647318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41bc0b1ee0 0x7f41bc13f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f41c0c00640 1 -- 172.21.15.47:0/1591647318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a8009420 con 0x7f41bc15ba70 2024-02-19T15:22:13.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7fa828a00640 1 --2- 172.21.15.47:0/658778839 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa804066f10 0x7fa8040693d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa8180b0000 tx=0x7fa8180a8070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.826+0000 7f41c0c00640 1 --2- 172.21.15.47:0/1591647318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41bc15ba70 0x7f41bc0717e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f41b400efc0 tx=0x7f41b400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.830+0000 7fa821600640 1 -- 172.21.15.47:0/658778839 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa810050df0 con 0x7fa8240ad140 2024-02-19T15:22:13.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.838+0000 7f41b3e00640 1 -- 172.21.15.47:0/1591647318 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b400cd20 con 0x7f41bc15ba70 2024-02-19T15:22:13.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.838+0000 7f41b3e00640 1 -- 172.21.15.47:0/1591647318 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b400f040 con 0x7f41bc15ba70 2024-02-19T15:22:13.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.838+0000 7f41b3e00640 1 -- 172.21.15.47:0/1591647318 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b4003be0 con 0x7f41bc15ba70 2024-02-19T15:22:13.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.838+0000 7f41c1600640 1 -- 172.21.15.47:0/1591647318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41bc0b2800 con 0x7f41bc15ba70 2024-02-19T15:22:13.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.838+0000 7f59f8c00640 1 -- 172.21.15.47:0/1595215607 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f59f4051680 con 0x7f59fc071050 2024-02-19T15:22:13.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.838+0000 7f41c1600640 1 -- 172.21.15.47:0/1591647318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41bc14fe50 con 0x7f41bc15ba70 2024-02-19T15:22:13.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.842+0000 7f41b3e00640 1 -- 172.21.15.47:0/1591647318 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f41b4003d40 con 0x7f41bc15ba70 2024-02-19T15:22:13.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.842+0000 7f41c1600640 1 -- 172.21.15.47:0/1591647318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41bc075360 con 0x7f41bc15ba70 2024-02-19T15:22:13.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.842+0000 7f41b3e00640 1 --2- 172.21.15.47:0/1591647318 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f419c066f10 0x7f419c0693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.842+0000 7f41b3e00640 1 -- 172.21.15.47:0/1591647318 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41b4085030 con 0x7f41bc15ba70 2024-02-19T15:22:13.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.846+0000 7f41bbe00640 1 --2- 172.21.15.47:0/1591647318 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f419c066f10 0x7f419c0693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.846+0000 7f41bbe00640 1 --2- 172.21.15.47:0/1591647318 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f419c066f10 0x7f419c0693d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f41ac000f70 tx=0x7f41ac0079e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.850+0000 7f41b3e00640 1 -- 172.21.15.47:0/1591647318 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f41b4051e00 con 0x7f41bc15ba70 2024-02-19T15:22:13.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.858+0000 7f8ce7800640 1 Processor -- start 2024-02-19T15:22:13.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.858+0000 7f8ce7800640 1 -- start start 2024-02-19T15:22:13.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.862+0000 7f8ce7800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ce0071260 0x7f8ce00cf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.862+0000 7f8ce7800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8ce006cbe0 0x7f8ce006d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.862+0000 7f8ce7800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce00b3ab0 con 0x7f8ce00b21c0 2024-02-19T15:22:13.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.862+0000 7f8ce7800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ce00b3d90 con 0x7f8ce006cbe0 2024-02-19T15:22:13.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.862+0000 7f8ce7800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce00b4070 con 0x7f8ce0071260 2024-02-19T15:22:13.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.862+0000 7f8ce5800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8ce006cbe0 0x7f8ce006d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.862+0000 7f8ce5800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8ce006cbe0 0x7f8ce006d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:36720/0 (socket says 172.21.15.47:36720) 2024-02-19T15:22:13.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.862+0000 7f8ce5800640 1 -- 172.21.15.47:0/4273721576 learned_addr learned my addr 172.21.15.47:0/4273721576 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:13.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.862+0000 7f8cdfe00640 1 --2- 172.21.15.47:0/4273721576 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ce0071260 0x7f8ce00cf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.862+0000 7f8ce5800640 1 -- 172.21.15.47:0/4273721576 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ce0071260 msgr2=0x7f8ce00cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:13.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.862+0000 7f8ce5800640 1 --2- 172.21.15.47:0/4273721576 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ce0071260 0x7f8ce00cf0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.862+0000 7f8ce5800640 1 -- 172.21.15.47:0/4273721576 >> v1:172.21.15.116:6789/0 conn(0x7f8ce00b21c0 legacy=0x7f8ce0070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:13.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.862+0000 7f8ce5800640 1 -- 172.21.15.47:0/4273721576 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce00b4350 con 0x7f8ce006cbe0 2024-02-19T15:22:13.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.870+0000 7f8ce5800640 1 --2- 172.21.15.47:0/4273721576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce006cbe0 0x7f8ce006d090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc009870 tx=0x7f8ccc01c4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf3e8a1bc8c31174 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.870+0000 7f8cdf400640 1 -- 172.21.15.47:0/4273721576 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc0aa430 con 0x7f8ce006cbe0 2024-02-19T15:22:13.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.870+0000 7f8cdf400640 1 -- 172.21.15.47:0/4273721576 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ccc0aa590 con 0x7f8ce006cbe0 2024-02-19T15:22:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.870+0000 7f8ce7800640 1 -- 172.21.15.47:0/4273721576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce006cbe0 msgr2=0x7f8ce006d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.870+0000 7f8cdf400640 1 -- 172.21.15.47:0/4273721576 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc0aa430 con 0x7f8ce006cbe0 2024-02-19T15:22:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.870+0000 7f8ce7800640 1 --2- 172.21.15.47:0/4273721576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce006cbe0 0x7f8ce006d090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc009870 tx=0x7f8ccc01c4f0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.870+0000 7f8ce7800640 1 -- 172.21.15.47:0/4273721576 shutdown_connections 2024-02-19T15:22:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.870+0000 7f8ce7800640 1 --2- 172.21.15.47:0/4273721576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce006cbe0 0x7f8ce006d090 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.870+0000 7f8ce7800640 1 --2- 172.21.15.47:0/4273721576 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ce0071260 0x7f8ce00cf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.870+0000 7f8ce7800640 1 -- 172.21.15.47:0/4273721576 >> 172.21.15.47:0/4273721576 conn(0x7f8ce00b16a0 msgr2=0x7f8ce00b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.874+0000 7f8ce7800640 1 -- 172.21.15.47:0/4273721576 shutdown_connections 2024-02-19T15:22:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.874+0000 7f8ce7800640 1 -- 172.21.15.47:0/4273721576 wait complete. 2024-02-19T15:22:13.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.874+0000 7f8ce7800640 1 Processor -- start 2024-02-19T15:22:13.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.878+0000 7f8ce7800640 1 -- start start 2024-02-19T15:22:13.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.878+0000 7f8ce7800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ce006cbe0 0x7f8ce00b0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.878+0000 7f8ce7800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0071260 0x7f8ce00b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.878+0000 7f8ce7800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ce00b21c0 0x7f8ce0206ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.878+0000 7f8ce7800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce0143470 con 0x7f8ce006cbe0 2024-02-19T15:22:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.878+0000 7f8ce5800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ce00b21c0 0x7f8ce0206ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.878+0000 7f8cdfe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0071260 0x7f8ce00b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.878+0000 7f8ce5800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ce00b21c0 0x7f8ce0206ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:47472/0 (socket says 172.21.15.47:47472) 2024-02-19T15:22:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.878+0000 7f8ce5800640 1 -- 172.21.15.47:0/2444972880 learned_addr learned my addr 172.21.15.47:0/2444972880 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.878+0000 7f8ce7800640 1 -- 172.21.15.47:0/2444972880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce0207da0 con 0x7f8ce0071260 2024-02-19T15:22:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.878+0000 7f8ce7800640 1 -- 172.21.15.47:0/2444972880 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce0207ac0 con 0x7f8ce00b21c0 2024-02-19T15:22:13.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.878+0000 7f8cdfe00640 1 -- 172.21.15.47:0/2444972880 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ce00b21c0 msgr2=0x7f8ce0206ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.878+0000 7f8cdfe00640 1 --2- 172.21.15.47:0/2444972880 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ce00b21c0 0x7f8ce0206ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.878+0000 7f8cdfe00640 1 -- 172.21.15.47:0/2444972880 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ce006cbe0 msgr2=0x7f8ce00b0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:13.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.878+0000 7f8cdfe00640 1 --2- 172.21.15.47:0/2444972880 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ce006cbe0 0x7f8ce00b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:13.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.878+0000 7f8cdfe00640 1 -- 172.21.15.47:0/2444972880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ccc0094d0 con 0x7f8ce0071260 2024-02-19T15:22:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.882+0000 7f8cdfe00640 1 --2- 172.21.15.47:0/2444972880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0071260 0x7f8ce00b0a60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8cd000d5f0 tx=0x7f8cd000dab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.882+0000 7f8cdcc00640 1 -- 172.21.15.47:0/2444972880 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd00125f0 con 0x7f8ce0071260 2024-02-19T15:22:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.882+0000 7f8cdcc00640 1 -- 172.21.15.47:0/2444972880 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd0012750 con 0x7f8ce0071260 2024-02-19T15:22:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.882+0000 7f8cdcc00640 1 -- 172.21.15.47:0/2444972880 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0012ae0 con 0x7f8ce0071260 2024-02-19T15:22:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.882+0000 7f8ce7800640 1 -- 172.21.15.47:0/2444972880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce0204b50 con 0x7f8ce0071260 2024-02-19T15:22:13.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.882+0000 7f8ce7800640 1 -- 172.21.15.47:0/2444972880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce0205110 con 0x7f8ce0071260 2024-02-19T15:22:13.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.886+0000 7f8cdcc00640 1 -- 172.21.15.47:0/2444972880 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8cd0012c40 con 0x7f8ce0071260 2024-02-19T15:22:13.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.886+0000 7f8cdcc00640 1 --2- 172.21.15.47:0/2444972880 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8cc0066e90 0x7f8cc0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:13.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.886+0000 7f8cdcc00640 1 -- 172.21.15.47:0/2444972880 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8cd0084eb0 con 0x7f8ce0071260 2024-02-19T15:22:13.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.886+0000 7f8ce7800640 1 -- 172.21.15.47:0/2444972880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce006e4e0 con 0x7f8ce0071260 2024-02-19T15:22:13.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.890+0000 7f8ce4e00640 1 --2- 172.21.15.47:0/2444972880 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8cc0066e90 0x7f8cc0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:13.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.890+0000 7f8ce4e00640 1 --2- 172.21.15.47:0/2444972880 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8cc0066e90 0x7f8cc0069350 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8cd800ab90 tx=0x7f8cd8009230 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:13.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:13.894+0000 7f8cdcc00640 1 -- 172.21.15.47:0/2444972880 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8cd0051c80 con 0x7f8ce0071260 2024-02-19T15:22:14.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.082+0000 7fb823e00640 1 -- 172.21.15.47:0/3483540180 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb82406e310 con 0x7fb824070de0 2024-02-19T15:22:14.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.086+0000 7fb81aa00640 1 -- 172.21.15.47:0/3483540180 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb81c0519d0 con 0x7fb824070de0 2024-02-19T15:22:14.090 INFO:teuthology.orchestra.run.smithi047.stdout:60129542152 2024-02-19T15:22:14.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.090+0000 7fb823e00640 1 -- 172.21.15.47:0/3483540180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb804066ec0 msgr2=0x7fb804069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.090+0000 7fb823e00640 1 --2- 172.21.15.47:0/3483540180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb804066ec0 0x7fb804069380 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb8140af000 tx=0x7fb8140a7070 comp rx=0 tx=0).stop 2024-02-19T15:22:14.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.090+0000 7fb823e00640 1 -- 172.21.15.47:0/3483540180 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb824070de0 msgr2=0x7fb82415b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.090+0000 7fb823e00640 1 --2- 172.21.15.47:0/3483540180 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb824070de0 0x7fb82415b950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00c700 tx=0x7fb81c00cbc0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.090+0000 7fb823e00640 1 -- 172.21.15.47:0/3483540180 shutdown_connections 2024-02-19T15:22:14.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.090+0000 7fb823e00640 1 --2- 172.21.15.47:0/3483540180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb804066ec0 0x7fb804069380 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.090+0000 7fb823e00640 1 --2- 172.21.15.47:0/3483540180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8241431e0 0x7fb824206a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.090+0000 7fb823e00640 1 --2- 172.21.15.47:0/3483540180 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8241404b0 0x7fb82415be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.094+0000 7fb823e00640 1 --2- 172.21.15.47:0/3483540180 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb824070de0 0x7fb82415b950 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.094+0000 7fb823e00640 1 -- 172.21.15.47:0/3483540180 >> 172.21.15.47:0/3483540180 conn(0x7fb8240cf900 msgr2=0x7fb824141c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:14.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.094+0000 7fb823e00640 1 -- 172.21.15.47:0/3483540180 shutdown_connections 2024-02-19T15:22:14.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.094+0000 7fb823e00640 1 -- 172.21.15.47:0/3483540180 wait complete. 2024-02-19T15:22:14.124 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.1 2024-02-19T15:22:14.124 DEBUG:teuthology.parallel:result is None 2024-02-19T15:22:14.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.138+0000 7f7680c00640 1 -- 172.21.15.47:0/1209948131 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f7660005550 con 0x7f768c143500 2024-02-19T15:22:14.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.138+0000 7f7683400640 1 -- 172.21.15.47:0/1209948131 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7684055290 con 0x7f768c143500 2024-02-19T15:22:14.140 INFO:teuthology.orchestra.run.smithi047.stdout:64424509447 2024-02-19T15:22:14.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.142+0000 7f7680c00640 1 -- 172.21.15.47:0/1209948131 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f766c066f10 msgr2=0x7f766c0693d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.142+0000 7f7680c00640 1 --2- 172.21.15.47:0/1209948131 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f766c066f10 0x7f766c0693d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f767c0ad000 tx=0x7f767c0a7000 comp rx=0 tx=0).stop 2024-02-19T15:22:14.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.142+0000 7f7680c00640 1 -- 172.21.15.47:0/1209948131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f768c143500 msgr2=0x7f768c1f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.142+0000 7f7680c00640 1 --2- 172.21.15.47:0/1209948131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f768c143500 0x7f768c1f6930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f768c072920 tx=0x7f768400b950 comp rx=0 tx=0).stop 2024-02-19T15:22:14.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.142+0000 7f7680c00640 1 -- 172.21.15.47:0/1209948131 shutdown_connections 2024-02-19T15:22:14.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.142+0000 7f7680c00640 1 --2- 172.21.15.47:0/1209948131 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f766c066f10 0x7f766c0693d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.142+0000 7f7680c00640 1 --2- 172.21.15.47:0/1209948131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f768c143500 0x7f768c1f6930 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.142+0000 7f7680c00640 1 --2- 172.21.15.47:0/1209948131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f768c143080 0x7f768c1f63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.142+0000 7f7680c00640 1 --2- 172.21.15.47:0/1209948131 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7670003b60 0x7f768c150cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.142+0000 7f7680c00640 1 -- 172.21.15.47:0/1209948131 >> 172.21.15.47:0/1209948131 conn(0x7f768c0b16a0 msgr2=0x7f768c139990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:14.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.142+0000 7f7680c00640 1 -- 172.21.15.47:0/1209948131 shutdown_connections 2024-02-19T15:22:14.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.142+0000 7f7680c00640 1 -- 172.21.15.47:0/1209948131 wait complete. 2024-02-19T15:22:14.163 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509447 for osd.6 2024-02-19T15:22:14.163 DEBUG:teuthology.parallel:result is None 2024-02-19T15:22:14.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.178+0000 7f2ce6600640 1 -- 172.21.15.47:0/4242113231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f2cd80e8d90 con 0x7f2cd80e7550 2024-02-19T15:22:14.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.178+0000 7f2cd3400640 1 -- 172.21.15.47:0/4242113231 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2cd80e8d90 con 0x7f2cd80e7550 2024-02-19T15:22:14.179 INFO:teuthology.orchestra.run.smithi047.stdout:60129542152 2024-02-19T15:22:14.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.178+0000 7f18c7000640 1 -- 172.21.15.47:0/936909365 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f18c00ab8b0 con 0x7f18c00b2cc0 2024-02-19T15:22:14.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.178+0000 7f18b7e00640 1 -- 172.21.15.47:0/936909365 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f18c00ab8b0 con 0x7f18c00b2cc0 2024-02-19T15:22:14.182 INFO:teuthology.orchestra.run.smithi047.stdout:60129542151 2024-02-19T15:22:14.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f2ce6600640 1 -- 172.21.15.47:0/4242113231 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2cc0066ec0 msgr2=0x7f2cc0069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f2ce6600640 1 --2- 172.21.15.47:0/4242113231 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2cc0066ec0 0x7f2cc0069380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc007fd0 tx=0x7f2ccc007f20 comp rx=0 tx=0).stop 2024-02-19T15:22:14.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f2ce6600640 1 -- 172.21.15.47:0/4242113231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cd80e7550 msgr2=0x7f2cd804edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f2ce6600640 1 --2- 172.21.15.47:0/4242113231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cd80e7550 0x7f2cd804edc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2ce00500b0 tx=0x7f2ce0068ac0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f2ce6600640 1 -- 172.21.15.47:0/4242113231 shutdown_connections 2024-02-19T15:22:14.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f2ce6600640 1 --2- 172.21.15.47:0/4242113231 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2cc0066ec0 0x7f2cc0069380 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f2ce6600640 1 --2- 172.21.15.47:0/4242113231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cd80e7550 0x7f2cd804edc0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f2ce6600640 1 --2- 172.21.15.47:0/4242113231 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2cd8051820 0x7f2cd80d6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f2ce6600640 1 --2- 172.21.15.47:0/4242113231 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cd80381b0 0x7f2cd80e9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f2ce6600640 1 -- 172.21.15.47:0/4242113231 >> 172.21.15.47:0/4242113231 conn(0x7f2cd8055360 msgr2=0x7f2cd8055740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:14.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f2ce6600640 1 -- 172.21.15.47:0/4242113231 shutdown_connections 2024-02-19T15:22:14.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f2ce6600640 1 -- 172.21.15.47:0/4242113231 wait complete. 2024-02-19T15:22:14.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f18c7000640 1 -- 172.21.15.47:0/936909365 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f18a0066e40 msgr2=0x7f18a0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f18c7000640 1 --2- 172.21.15.47:0/936909365 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f18a0066e40 0x7f18a0069300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f18ac0ae000 tx=0x7f18ac0a7000 comp rx=0 tx=0).stop 2024-02-19T15:22:14.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f18c7000640 1 -- 172.21.15.47:0/936909365 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18c00b2cc0 msgr2=0x7f18c0156670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f18c7000640 1 --2- 172.21.15.47:0/936909365 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18c00b2cc0 0x7f18c0156670 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f18b000e930 tx=0x7f18b000edf0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f18c7000640 1 -- 172.21.15.47:0/936909365 shutdown_connections 2024-02-19T15:22:14.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f18c7000640 1 --2- 172.21.15.47:0/936909365 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f18a0066e40 0x7f18a0069300 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f18c7000640 1 --2- 172.21.15.47:0/936909365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18c01635c0 0x7f18c0156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f18c7000640 1 --2- 172.21.15.47:0/936909365 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18c00b2cc0 0x7f18c0156670 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f18c7000640 1 --2- 172.21.15.47:0/936909365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c0070e80 0x7f18c00a9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f18c7000640 1 -- 172.21.15.47:0/936909365 >> 172.21.15.47:0/936909365 conn(0x7f18c00b16a0 msgr2=0x7f18c01655f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:14.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f18c7000640 1 -- 172.21.15.47:0/936909365 shutdown_connections 2024-02-19T15:22:14.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.182+0000 7f18c7000640 1 -- 172.21.15.47:0/936909365 wait complete. 2024-02-19T15:22:14.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.190+0000 7fa82be00640 1 -- 172.21.15.47:0/658778839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fa8240b20a0 con 0x7fa8240ad140 2024-02-19T15:22:14.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.190+0000 7fa821600640 1 -- 172.21.15.47:0/658778839 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa810017020 con 0x7fa8240ad140 2024-02-19T15:22:14.192 INFO:teuthology.orchestra.run.smithi047.stdout:60129542151 2024-02-19T15:22:14.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.190+0000 7f5a03800640 1 -- 172.21.15.47:0/1595215607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f59fc0b2850 con 0x7f59fc071050 2024-02-19T15:22:14.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.190+0000 7f59f8c00640 1 -- 172.21.15.47:0/1595215607 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f59f4054d40 con 0x7f59fc071050 2024-02-19T15:22:14.193 INFO:teuthology.orchestra.run.smithi047.stdout:60129542150 2024-02-19T15:22:14.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7fa82be00640 1 -- 172.21.15.47:0/658778839 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa804066f10 msgr2=0x7fa8040693d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7fa82be00640 1 --2- 172.21.15.47:0/658778839 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa804066f10 0x7fa8040693d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa8180b0000 tx=0x7fa8180a8070 comp rx=0 tx=0).stop 2024-02-19T15:22:14.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7fa82be00640 1 -- 172.21.15.47:0/658778839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8240ad140 msgr2=0x7fa82414ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7fa82be00640 1 --2- 172.21.15.47:0/658778839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8240ad140 0x7fa82414ffd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa81000c400 tx=0x7fa81000c8c0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7fa82be00640 1 -- 172.21.15.47:0/658778839 shutdown_connections 2024-02-19T15:22:14.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7fa82be00640 1 --2- 172.21.15.47:0/658778839 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa804066f10 0x7fa8040693d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7fa82be00640 1 --2- 172.21.15.47:0/658778839 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa82415d9d0 0x7fa8241529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7fa82be00640 1 --2- 172.21.15.47:0/658778839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8240b5d90 0x7fa824203120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7fa82be00640 1 --2- 172.21.15.47:0/658778839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8240ad140 0x7fa82414ffd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7fa82be00640 1 -- 172.21.15.47:0/658778839 >> 172.21.15.47:0/658778839 conn(0x7fa8240b4fc0 msgr2=0x7fa82406e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:14.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7fa82be00640 1 -- 172.21.15.47:0/658778839 shutdown_connections 2024-02-19T15:22:14.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7fa82be00640 1 -- 172.21.15.47:0/658778839 wait complete. 2024-02-19T15:22:14.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7f5a03800640 1 -- 172.21.15.47:0/1595215607 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f59d8066e40 msgr2=0x7f59d8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7f5a03800640 1 --2- 172.21.15.47:0/1595215607 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f59d8066e40 0x7f59d8069300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f59fc1597b0 tx=0x7f59e4009040 comp rx=0 tx=0).stop 2024-02-19T15:22:14.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7f5a03800640 1 -- 172.21.15.47:0/1595215607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59fc071050 msgr2=0x7f59fc163e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7f5a03800640 1 --2- 172.21.15.47:0/1595215607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59fc071050 0x7f59fc163e40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f59f400cad0 tx=0x7f59f400ced0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7f5a03800640 1 -- 172.21.15.47:0/1595215607 shutdown_connections 2024-02-19T15:22:14.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7f5a03800640 1 --2- 172.21.15.47:0/1595215607 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f59d8066e40 0x7f59d8069300 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7f5a03800640 1 --2- 172.21.15.47:0/1595215607 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f59fc1635c0 0x7f59fc150c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7f5a03800640 1 --2- 172.21.15.47:0/1595215607 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59fc0b2cc0 0x7f59fc150710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7f5a03800640 1 --2- 172.21.15.47:0/1595215607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59fc071050 0x7f59fc163e40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.194+0000 7f5a03800640 1 -- 172.21.15.47:0/1595215607 >> 172.21.15.47:0/1595215607 conn(0x7f59fc0b16a0 msgr2=0x7f59fc154e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:14.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.198+0000 7f5a03800640 1 -- 172.21.15.47:0/1595215607 shutdown_connections 2024-02-19T15:22:14.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.198+0000 7f5a03800640 1 -- 172.21.15.47:0/1595215607 wait complete. 2024-02-19T15:22:14.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.198+0000 7f41c1600640 1 -- 172.21.15.47:0/1591647318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f41bc1558b0 con 0x7f41bc15ba70 2024-02-19T15:22:14.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.202+0000 7f41b3e00640 1 -- 172.21.15.47:0/1591647318 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f41b40554c0 con 0x7f41bc15ba70 2024-02-19T15:22:14.203 INFO:teuthology.orchestra.run.smithi047.stdout:60129542150 2024-02-19T15:22:14.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.206+0000 7f41c1600640 1 -- 172.21.15.47:0/1591647318 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f419c066f10 msgr2=0x7f419c0693d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.206+0000 7f41c1600640 1 --2- 172.21.15.47:0/1591647318 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f419c066f10 0x7f419c0693d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f41ac000f70 tx=0x7f41ac0079e0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.206+0000 7f41c1600640 1 -- 172.21.15.47:0/1591647318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41bc15ba70 msgr2=0x7f41bc0717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.206+0000 7f41c1600640 1 --2- 172.21.15.47:0/1591647318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41bc15ba70 0x7f41bc0717e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f41b400efc0 tx=0x7f41b400c490 comp rx=0 tx=0).stop 2024-02-19T15:22:14.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.206+0000 7f41c1600640 1 -- 172.21.15.47:0/1591647318 shutdown_connections 2024-02-19T15:22:14.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.206+0000 7f41c1600640 1 --2- 172.21.15.47:0/1591647318 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f419c066f10 0x7f419c0693d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.206+0000 7f41c1600640 1 --2- 172.21.15.47:0/1591647318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41bc15ba70 0x7f41bc0717e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.206+0000 7f41c1600640 1 --2- 172.21.15.47:0/1591647318 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f41bc0cf000 0x7f41bc0712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.206+0000 7f41c1600640 1 --2- 172.21.15.47:0/1591647318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41bc0b1ee0 0x7f41bc13f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.206+0000 7f41c1600640 1 -- 172.21.15.47:0/1591647318 >> 172.21.15.47:0/1591647318 conn(0x7f41bc0cf7c0 msgr2=0x7f41bc1595d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:14.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.206+0000 7f41c1600640 1 -- 172.21.15.47:0/1591647318 shutdown_connections 2024-02-19T15:22:14.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.206+0000 7f41c1600640 1 -- 172.21.15.47:0/1591647318 wait complete. 2024-02-19T15:22:14.215 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.7 2024-02-19T15:22:14.215 DEBUG:teuthology.parallel:result is None 2024-02-19T15:22:14.216 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.5 2024-02-19T15:22:14.216 DEBUG:teuthology.parallel:result is None 2024-02-19T15:22:14.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.222+0000 7f8ce7800640 1 -- 172.21.15.47:0/2444972880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f8ce0075360 con 0x7f8ce0071260 2024-02-19T15:22:14.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.222+0000 7f8cdcc00640 1 -- 172.21.15.47:0/2444972880 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8cd0055340 con 0x7f8ce0071260 2024-02-19T15:22:14.223 INFO:teuthology.orchestra.run.smithi047.stdout:60129542150 2024-02-19T15:22:14.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.222+0000 7f8ce7800640 1 -- 172.21.15.47:0/2444972880 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8cc0066e90 msgr2=0x7f8cc0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.222+0000 7f8ce7800640 1 --2- 172.21.15.47:0/2444972880 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8cc0066e90 0x7f8cc0069350 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8cd800ab90 tx=0x7f8cd8009230 comp rx=0 tx=0).stop 2024-02-19T15:22:14.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.222+0000 7f8ce7800640 1 -- 172.21.15.47:0/2444972880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0071260 msgr2=0x7f8ce00b0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.222+0000 7f8ce7800640 1 --2- 172.21.15.47:0/2444972880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0071260 0x7f8ce00b0a60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8cd000d5f0 tx=0x7f8cd000dab0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.222+0000 7f8ce7800640 1 -- 172.21.15.47:0/2444972880 shutdown_connections 2024-02-19T15:22:14.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.222+0000 7f8ce7800640 1 --2- 172.21.15.47:0/2444972880 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8cc0066e90 0x7f8cc0069350 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.222+0000 7f8ce7800640 1 --2- 172.21.15.47:0/2444972880 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ce00b21c0 0x7f8ce0206ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.222+0000 7f8ce7800640 1 --2- 172.21.15.47:0/2444972880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce0071260 0x7f8ce00b0a60 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.222+0000 7f8ce7800640 1 --2- 172.21.15.47:0/2444972880 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ce006cbe0 0x7f8ce00b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.222+0000 7f8ce7800640 1 -- 172.21.15.47:0/2444972880 >> 172.21.15.47:0/2444972880 conn(0x7f8ce00b16a0 msgr2=0x7f8ce00b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:14.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.222+0000 7f8ce7800640 1 -- 172.21.15.47:0/2444972880 shutdown_connections 2024-02-19T15:22:14.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.222+0000 7f8ce7800640 1 -- 172.21.15.47:0/2444972880 wait complete. 2024-02-19T15:22:14.229 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.4 2024-02-19T15:22:14.229 DEBUG:teuthology.parallel:result is None 2024-02-19T15:22:14.231 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.2 2024-02-19T15:22:14.231 DEBUG:teuthology.parallel:result is None 2024-02-19T15:22:14.239 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.0 2024-02-19T15:22:14.239 DEBUG:teuthology.parallel:result is None 2024-02-19T15:22:14.247 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.3 2024-02-19T15:22:14.247 DEBUG:teuthology.parallel:result is None 2024-02-19T15:22:14.247 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-02-19T15:22:14.247 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:22:14.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.434+0000 7f57d2800640 1 Processor -- start 2024-02-19T15:22:14.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.434+0000 7f57d2800640 1 -- start start 2024-02-19T15:22:14.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.434+0000 7f57d2800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57cc15ab30 0x7f57cc0726d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:14.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.434+0000 7f57d2800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57cc15e7e0 con 0x7f57cc0cb040 2024-02-19T15:22:14.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.434+0000 7f57d2800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57cc15ea30 con 0x7f57cc0cce50 2024-02-19T15:22:14.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.434+0000 7f57d2800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc15f3d0 con 0x7f57cc15ab30 2024-02-19T15:22:14.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.434+0000 7f57d1e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f57cc0cb040 0x7f57cc0cdce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35354/0 (socket says 172.21.15.47:35354) 2024-02-19T15:22:14.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.434+0000 7f57d1e00640 1 -- 172.21.15.47:0/2862764066 learned_addr learned my addr 172.21.15.47:0/2862764066 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:14.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.434+0000 7f57d1400640 1 --2- 172.21.15.47:0/2862764066 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57cc15ab30 0x7f57cc0726d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:14.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.434+0000 7f57cbe00640 1 -- 172.21.15.47:0/2862764066 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2257985762 0 0) 0x7f57cc15e7e0 con 0x7f57cc0cb040 2024-02-19T15:22:14.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.434+0000 7f57cbe00640 1 -- 172.21.15.47:0/2862764066 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57b4005740 con 0x7f57cc0cb040 2024-02-19T15:22:14.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.434+0000 7f57cbe00640 1 -- 172.21.15.47:0/2862764066 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838414411 0 0) 0x7f57cc15ea30 con 0x7f57cc0cce50 2024-02-19T15:22:14.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.434+0000 7f57d1400640 1 -- 172.21.15.47:0/2862764066 >> v1:172.21.15.116:6789/0 conn(0x7f57cc0cce50 legacy=0x7f57cc0ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.434+0000 7f57d1400640 1 -- 172.21.15.47:0/2862764066 >> v1:172.21.15.47:6789/0 conn(0x7f57cc0cb040 legacy=0x7f57cc0cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.434+0000 7f57d1400640 1 -- 172.21.15.47:0/2862764066 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57cc15ed10 con 0x7f57cc15ab30 2024-02-19T15:22:14.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.438+0000 7f57d1400640 1 --2- 172.21.15.47:0/2862764066 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57cc15ab30 0x7f57cc0726d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f57c00097f0 tx=0x7f57c001c390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4637762fad83c6da server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:14.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.438+0000 7f57cbe00640 1 -- 172.21.15.47:0/2862764066 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c0004230 con 0x7f57cc15ab30 2024-02-19T15:22:14.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.438+0000 7f57cbe00640 1 -- 172.21.15.47:0/2862764066 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c00a7050 con 0x7f57cc15ab30 2024-02-19T15:22:14.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.438+0000 7f57cbe00640 1 -- 172.21.15.47:0/2862764066 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c0003620 con 0x7f57cc15ab30 2024-02-19T15:22:14.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.438+0000 7f57d2800640 1 -- 172.21.15.47:0/2862764066 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57cc15ab30 msgr2=0x7f57cc0726d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.438+0000 7f57d2800640 1 --2- 172.21.15.47:0/2862764066 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57cc15ab30 0x7f57cc0726d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f57c00097f0 tx=0x7f57c001c390 comp rx=0 tx=0).stop 2024-02-19T15:22:14.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.438+0000 7f57d2800640 1 -- 172.21.15.47:0/2862764066 shutdown_connections 2024-02-19T15:22:14.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.438+0000 7f57d2800640 1 --2- 172.21.15.47:0/2862764066 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57cc15ab30 0x7f57cc0726d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.438+0000 7f57d2800640 1 -- 172.21.15.47:0/2862764066 >> 172.21.15.47:0/2862764066 conn(0x7f57cc0b1840 msgr2=0x7f57cc0b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:14.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.438+0000 7f57d2800640 1 -- 172.21.15.47:0/2862764066 shutdown_connections 2024-02-19T15:22:14.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.438+0000 7f57d2800640 1 -- 172.21.15.47:0/2862764066 wait complete. 2024-02-19T15:22:14.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.438+0000 7f57d2800640 1 Processor -- start 2024-02-19T15:22:14.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.438+0000 7f57d2800640 1 -- start start 2024-02-19T15:22:14.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57d2800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57cc0cb040 0x7f57cc152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:14.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57d2800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc0cce50 0x7f57cc06e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:14.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57d2800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57cc15ab30 0x7f57cc0c7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:14.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57d2800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc0c8200 con 0x7f57cc15ab30 2024-02-19T15:22:14.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57d2800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc09c6b0 con 0x7f57cc0cce50 2024-02-19T15:22:14.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57d2800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc09c990 con 0x7f57cc0cb040 2024-02-19T15:22:14.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57d1e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57cc15ab30 0x7f57cc0c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:14.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57d1e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57cc15ab30 0x7f57cc0c7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57878/0 (socket says 172.21.15.47:57878) 2024-02-19T15:22:14.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57d1400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57cc0cb040 0x7f57cc152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:14.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57d1e00640 1 -- 172.21.15.47:0/266746541 learned_addr learned my addr 172.21.15.47:0/266746541 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:14.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57d0a00640 1 --2- 172.21.15.47:0/266746541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc0cce50 0x7f57cc06e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:14.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57d1e00640 1 -- 172.21.15.47:0/266746541 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57cc0cb040 msgr2=0x7f57cc152c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57d1e00640 1 --2- 172.21.15.47:0/266746541 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57cc0cb040 0x7f57cc152c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57d1e00640 1 -- 172.21.15.47:0/266746541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc0cce50 msgr2=0x7f57cc06e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57d1e00640 1 --2- 172.21.15.47:0/266746541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc0cce50 0x7f57cc06e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57d1e00640 1 -- 172.21.15.47:0/266746541 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c0009420 con 0x7f57cc15ab30 2024-02-19T15:22:14.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57d1e00640 1 --2- 172.21.15.47:0/266746541 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57cc15ab30 0x7f57cc0c7430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f57c400ca70 tx=0x7f57c400cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:14.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57c9600640 1 -- 172.21.15.47:0/266746541 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c400abd0 con 0x7f57cc15ab30 2024-02-19T15:22:14.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57c9600640 1 -- 172.21.15.47:0/266746541 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c400ad30 con 0x7f57cc15ab30 2024-02-19T15:22:14.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57d2800640 1 -- 172.21.15.47:0/266746541 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57cc09cc70 con 0x7f57cc15ab30 2024-02-19T15:22:14.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57c9600640 1 -- 172.21.15.47:0/266746541 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c4012600 con 0x7f57cc15ab30 2024-02-19T15:22:14.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57d2800640 1 -- 172.21.15.47:0/266746541 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57cc09d230 con 0x7f57cc15ab30 2024-02-19T15:22:14.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57d2800640 1 -- 172.21.15.47:0/266746541 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57cc09d450 con 0x7f57cc15ab30 2024-02-19T15:22:14.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.442+0000 7f57c9600640 1 -- 172.21.15.47:0/266746541 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f57c4007070 con 0x7f57cc15ab30 2024-02-19T15:22:14.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.446+0000 7f57c9600640 1 --2- 172.21.15.47:0/266746541 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f57ac066e40 0x7f57ac069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:14.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.446+0000 7f57c9600640 1 -- 172.21.15.47:0/266746541 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f57c40848b0 con 0x7f57cc15ab30 2024-02-19T15:22:14.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.446+0000 7f57d1400640 1 --2- 172.21.15.47:0/266746541 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f57ac066e40 0x7f57ac069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:14.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.446+0000 7f57d1400640 1 --2- 172.21.15.47:0/266746541 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f57ac066e40 0x7f57ac069300 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f57c00ad000 tx=0x7f57c00a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:14.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.450+0000 7f57c9600640 1 -- 172.21.15.47:0/266746541 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f57c40516b0 con 0x7f57cc15ab30 2024-02-19T15:22:14.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.606+0000 7f57d2800640 1 -- 172.21.15.47:0/266746541 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f57cc075360 con 0x7f57ac066e40 2024-02-19T15:22:14.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.614+0000 7f57c9600640 1 -- 172.21.15.47:0/266746541 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264599 (secure 0 0 0) 0x7f57cc075360 con 0x7f57ac066e40 2024-02-19T15:22:14.619 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:14.621 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:22:14.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.618+0000 7f57d2800640 1 -- 172.21.15.47:0/266746541 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f57ac066e40 msgr2=0x7f57ac069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.618+0000 7f57d2800640 1 --2- 172.21.15.47:0/266746541 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f57ac066e40 0x7f57ac069300 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f57c00ad000 tx=0x7f57c00a6000 comp rx=0 tx=0).stop 2024-02-19T15:22:14.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.618+0000 7f57d2800640 1 -- 172.21.15.47:0/266746541 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57cc15ab30 msgr2=0x7f57cc0c7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.618+0000 7f57d2800640 1 --2- 172.21.15.47:0/266746541 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57cc15ab30 0x7f57cc0c7430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f57c400ca70 tx=0x7f57c400cf30 comp rx=0 tx=0).stop 2024-02-19T15:22:14.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.618+0000 7f57d2800640 1 -- 172.21.15.47:0/266746541 shutdown_connections 2024-02-19T15:22:14.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.618+0000 7f57d2800640 1 --2- 172.21.15.47:0/266746541 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f57ac066e40 0x7f57ac069300 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.618+0000 7f57d2800640 1 --2- 172.21.15.47:0/266746541 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57cc15ab30 0x7f57cc0c7430 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.618+0000 7f57d2800640 1 --2- 172.21.15.47:0/266746541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc0cce50 0x7f57cc06e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.618+0000 7f57d2800640 1 --2- 172.21.15.47:0/266746541 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57cc0cb040 0x7f57cc152c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.618+0000 7f57d2800640 1 -- 172.21.15.47:0/266746541 >> 172.21.15.47:0/266746541 conn(0x7f57cc0b1840 msgr2=0x7f57cc072030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:14.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.618+0000 7f57d2800640 1 -- 172.21.15.47:0/266746541 shutdown_connections 2024-02-19T15:22:14.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.618+0000 7f57d2800640 1 -- 172.21.15.47:0/266746541 wait complete. 2024-02-19T15:22:14.639 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-02-19T15:22:13.151480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235712,"kb_used_data":2888,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739008,"statfs":{"total":773094113280,"available":772852744192,"internally_reserved":0,"allocated":2957312,"data_stored":1776620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001247"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181542+0000","last_change":"2024-02-19T15:22:05.868047+0000","last_active":"2024-02-19T15:22:07.181542+0000","last_peered":"2024-02-19T15:22:07.181542+0000","last_clean":"2024-02-19T15:22:07.181542+0000","last_became_active":"2024-02-19T15:22:05.867538+0000","last_became_peered":"2024-02-19T15:22:05.867538+0000","last_unstale":"2024-02-19T15:22:07.181542+0000","last_undegraded":"2024-02-19T15:22:07.181542+0000","last_fullsized":"2024-02-19T15:22:07.181542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:47:52.212576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853548+0000","last_change":"2024-02-19T15:22:05.861729+0000","last_active":"2024-02-19T15:22:06.853548+0000","last_peered":"2024-02-19T15:22:06.853548+0000","last_clean":"2024-02-19T15:22:06.853548+0000","last_became_active":"2024-02-19T15:22:05.861455+0000","last_became_peered":"2024-02-19T15:22:05.861455+0000","last_unstale":"2024-02-19T15:22:06.853548+0000","last_undegraded":"2024-02-19T15:22:06.853548+0000","last_fullsized":"2024-02-19T15:22:06.853548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:43.283192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424456+0000","last_change":"2024-02-19T15:22:05.857715+0000","last_active":"2024-02-19T15:22:07.424456+0000","last_peered":"2024-02-19T15:22:07.424456+0000","last_clean":"2024-02-19T15:22:07.424456+0000","last_became_active":"2024-02-19T15:22:05.857360+0000","last_became_peered":"2024-02-19T15:22:05.857360+0000","last_unstale":"2024-02-19T15:22:07.424456+0000","last_undegraded":"2024-02-19T15:22:07.424456+0000","last_fullsized":"2024-02-19T15:22:07.424456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:35:11.566219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424408+0000","last_change":"2024-02-19T15:22:05.850537+0000","last_active":"2024-02-19T15:22:07.424408+0000","last_peered":"2024-02-19T15:22:07.424408+0000","last_clean":"2024-02-19T15:22:07.424408+0000","last_became_active":"2024-02-19T15:22:05.850288+0000","last_became_peered":"2024-02-19T15:22:05.850288+0000","last_unstale":"2024-02-19T15:22:07.424408+0000","last_undegraded":"2024-02-19T15:22:07.424408+0000","last_fullsized":"2024-02-19T15:22:07.424408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:20:55.654033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181346+0000","last_change":"2024-02-19T15:22:05.872936+0000","last_active":"2024-02-19T15:22:07.181346+0000","last_peered":"2024-02-19T15:22:07.181346+0000","last_clean":"2024-02-19T15:22:07.181346+0000","last_became_active":"2024-02-19T15:22:05.871969+0000","last_became_peered":"2024-02-19T15:22:05.871969+0000","last_unstale":"2024-02-19T15:22:07.181346+0000","last_undegraded":"2024-02-19T15:22:07.181346+0000","last_fullsized":"2024-02-19T15:22:07.181346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:14:10.262240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424104+0000","last_change":"2024-02-19T15:22:05.853731+0000","last_active":"2024-02-19T15:22:07.424104+0000","last_peered":"2024-02-19T15:22:07.424104+0000","last_clean":"2024-02-19T15:22:07.424104+0000","last_became_active":"2024-02-19T15:22:05.853423+0000","last_became_peered":"2024-02-19T15:22:05.853423+0000","last_unstale":"2024-02-19T15:22:07.424104+0000","last_undegraded":"2024-02-19T15:22:07.424104+0000","last_fullsized":"2024-02-19T15:22:07.424104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:14:40.204009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181151+0000","last_change":"2024-02-19T15:22:05.860498+0000","last_active":"2024-02-19T15:22:07.181151+0000","last_peered":"2024-02-19T15:22:07.181151+0000","last_clean":"2024-02-19T15:22:07.181151+0000","last_became_active":"2024-02-19T15:22:05.860283+0000","last_became_peered":"2024-02-19T15:22:05.860283+0000","last_unstale":"2024-02-19T15:22:07.181151+0000","last_undegraded":"2024-02-19T15:22:07.181151+0000","last_fullsized":"2024-02-19T15:22:07.181151+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:20:55.654033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851236+0000","last_change":"2024-02-19T15:22:05.860461+0000","last_active":"2024-02-19T15:22:06.851236+0000","last_peered":"2024-02-19T15:22:06.851236+0000","last_clean":"2024-02-19T15:22:06.851236+0000","last_became_active":"2024-02-19T15:22:05.860119+0000","last_became_peered":"2024-02-19T15:22:05.860119+0000","last_unstale":"2024-02-19T15:22:06.851236+0000","last_undegraded":"2024-02-19T15:22:06.851236+0000","last_fullsized":"2024-02-19T15:22:06.851236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:43:34.980001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181680+0000","last_change":"2024-02-19T15:22:05.863575+0000","last_active":"2024-02-19T15:22:07.181680+0000","last_peered":"2024-02-19T15:22:07.181680+0000","last_clean":"2024-02-19T15:22:07.181680+0000","last_became_active":"2024-02-19T15:22:05.863306+0000","last_became_peered":"2024-02-19T15:22:05.863306+0000","last_unstale":"2024-02-19T15:22:07.181680+0000","last_undegraded":"2024-02-19T15:22:07.181680+0000","last_fullsized":"2024-02-19T15:22:07.181680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:03:59.289702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854624+0000","last_change":"2024-02-19T15:22:05.867622+0000","last_active":"2024-02-19T15:22:06.854624+0000","last_peered":"2024-02-19T15:22:06.854624+0000","last_clean":"2024-02-19T15:22:06.854624+0000","last_became_active":"2024-02-19T15:22:05.867352+0000","last_became_peered":"2024-02-19T15:22:05.867352+0000","last_unstale":"2024-02-19T15:22:06.854624+0000","last_undegraded":"2024-02-19T15:22:06.854624+0000","last_fullsized":"2024-02-19T15:22:06.854624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:08.091823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854743+0000","last_change":"2024-02-19T15:22:05.859648+0000","last_active":"2024-02-19T15:22:06.854743+0000","last_peered":"2024-02-19T15:22:06.854743+0000","last_clean":"2024-02-19T15:22:06.854743+0000","last_became_active":"2024-02-19T15:22:05.859381+0000","last_became_peered":"2024-02-19T15:22:05.859381+0000","last_unstale":"2024-02-19T15:22:06.854743+0000","last_undegraded":"2024-02-19T15:22:06.854743+0000","last_fullsized":"2024-02-19T15:22:06.854743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:35:11.566219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424319+0000","last_change":"2024-02-19T15:22:05.869009+0000","last_active":"2024-02-19T15:22:07.424319+0000","last_peered":"2024-02-19T15:22:07.424319+0000","last_clean":"2024-02-19T15:22:07.424319+0000","last_became_active":"2024-02-19T15:22:05.868709+0000","last_became_peered":"2024-02-19T15:22:05.868709+0000","last_unstale":"2024-02-19T15:22:07.424319+0000","last_undegraded":"2024-02-19T15:22:07.424319+0000","last_fullsized":"2024-02-19T15:22:07.424319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:22.856662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851443+0000","last_change":"2024-02-19T15:22:05.848558+0000","last_active":"2024-02-19T15:22:06.851443+0000","last_peered":"2024-02-19T15:22:06.851443+0000","last_clean":"2024-02-19T15:22:06.851443+0000","last_became_active":"2024-02-19T15:22:05.848288+0000","last_became_peered":"2024-02-19T15:22:05.848288+0000","last_unstale":"2024-02-19T15:22:06.851443+0000","last_undegraded":"2024-02-19T15:22:06.851443+0000","last_fullsized":"2024-02-19T15:22:06.851443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:33:20.316638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424183+0000","last_change":"2024-02-19T15:22:05.846265+0000","last_active":"2024-02-19T15:22:07.424183+0000","last_peered":"2024-02-19T15:22:07.424183+0000","last_clean":"2024-02-19T15:22:07.424183+0000","last_became_active":"2024-02-19T15:22:05.841757+0000","last_became_peered":"2024-02-19T15:22:05.841757+0000","last_unstale":"2024-02-19T15:22:07.424183+0000","last_undegraded":"2024-02-19T15:22:07.424183+0000","last_fullsized":"2024-02-19T15:22:07.424183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:14:58.397614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853190+0000","last_change":"2024-02-19T15:22:05.850786+0000","last_active":"2024-02-19T15:22:06.853190+0000","last_peered":"2024-02-19T15:22:06.853190+0000","last_clean":"2024-02-19T15:22:06.853190+0000","last_became_active":"2024-02-19T15:22:05.850136+0000","last_became_peered":"2024-02-19T15:22:05.850136+0000","last_unstale":"2024-02-19T15:22:06.853190+0000","last_undegraded":"2024-02-19T15:22:06.853190+0000","last_fullsized":"2024-02-19T15:22:06.853190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:33.203920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.423964+0000","last_change":"2024-02-19T15:22:05.862076+0000","last_active":"2024-02-19T15:22:07.423964+0000","last_peered":"2024-02-19T15:22:07.423964+0000","last_clean":"2024-02-19T15:22:07.423964+0000","last_became_active":"2024-02-19T15:22:05.861826+0000","last_became_peered":"2024-02-19T15:22:05.861826+0000","last_unstale":"2024-02-19T15:22:07.423964+0000","last_undegraded":"2024-02-19T15:22:07.423964+0000","last_fullsized":"2024-02-19T15:22:07.423964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:43.283192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.425028+0000","last_change":"2024-02-19T15:22:05.869845+0000","last_active":"2024-02-19T15:22:07.425028+0000","last_peered":"2024-02-19T15:22:07.425028+0000","last_clean":"2024-02-19T15:22:07.425028+0000","last_became_active":"2024-02-19T15:22:05.865534+0000","last_became_peered":"2024-02-19T15:22:05.865534+0000","last_unstale":"2024-02-19T15:22:07.425028+0000","last_undegraded":"2024-02-19T15:22:07.425028+0000","last_fullsized":"2024-02-19T15:22:07.425028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:29:35.769837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.423831+0000","last_change":"2024-02-19T15:22:05.852133+0000","last_active":"2024-02-19T15:22:07.423831+0000","last_peered":"2024-02-19T15:22:07.423831+0000","last_clean":"2024-02-19T15:22:07.423831+0000","last_became_active":"2024-02-19T15:22:05.851823+0000","last_became_peered":"2024-02-19T15:22:05.851823+0000","last_unstale":"2024-02-19T15:22:07.423831+0000","last_undegraded":"2024-02-19T15:22:07.423831+0000","last_fullsized":"2024-02-19T15:22:07.423831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:38.607191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181615+0000","last_change":"2024-02-19T15:22:05.866370+0000","last_active":"2024-02-19T15:22:07.181615+0000","last_peered":"2024-02-19T15:22:07.181615+0000","last_clean":"2024-02-19T15:22:07.181615+0000","last_became_active":"2024-02-19T15:22:05.866059+0000","last_became_peered":"2024-02-19T15:22:05.866059+0000","last_unstale":"2024-02-19T15:22:07.181615+0000","last_undegraded":"2024-02-19T15:22:07.181615+0000","last_fullsized":"2024-02-19T15:22:07.181615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:31:00.196963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854536+0000","last_change":"2024-02-19T15:22:05.858350+0000","last_active":"2024-02-19T15:22:06.854536+0000","last_peered":"2024-02-19T15:22:06.854536+0000","last_clean":"2024-02-19T15:22:06.854536+0000","last_became_active":"2024-02-19T15:22:05.858055+0000","last_became_peered":"2024-02-19T15:22:05.858055+0000","last_unstale":"2024-02-19T15:22:06.854536+0000","last_undegraded":"2024-02-19T15:22:06.854536+0000","last_fullsized":"2024-02-19T15:22:06.854536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:53:17.517796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424367+0000","last_change":"2024-02-19T15:22:05.867437+0000","last_active":"2024-02-19T15:22:07.424367+0000","last_peered":"2024-02-19T15:22:07.424367+0000","last_clean":"2024-02-19T15:22:07.424367+0000","last_became_active":"2024-02-19T15:22:05.867104+0000","last_became_peered":"2024-02-19T15:22:05.867104+0000","last_unstale":"2024-02-19T15:22:07.424367+0000","last_undegraded":"2024-02-19T15:22:07.424367+0000","last_fullsized":"2024-02-19T15:22:07.424367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:53:17.517796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424285+0000","last_change":"2024-02-19T15:22:05.873978+0000","last_active":"2024-02-19T15:22:07.424285+0000","last_peered":"2024-02-19T15:22:07.424285+0000","last_clean":"2024-02-19T15:22:07.424285+0000","last_became_active":"2024-02-19T15:22:05.870210+0000","last_became_peered":"2024-02-19T15:22:05.870210+0000","last_unstale":"2024-02-19T15:22:07.424285+0000","last_undegraded":"2024-02-19T15:22:07.424285+0000","last_fullsized":"2024-02-19T15:22:07.424285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:46:15.823647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851320+0000","last_change":"2024-02-19T15:22:05.869630+0000","last_active":"2024-02-19T15:22:06.851320+0000","last_peered":"2024-02-19T15:22:06.851320+0000","last_clean":"2024-02-19T15:22:06.851320+0000","last_became_active":"2024-02-19T15:22:05.869363+0000","last_became_peered":"2024-02-19T15:22:05.869363+0000","last_unstale":"2024-02-19T15:22:06.851320+0000","last_undegraded":"2024-02-19T15:22:06.851320+0000","last_fullsized":"2024-02-19T15:22:06.851320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:33.203920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424119+0000","last_change":"2024-02-19T15:22:05.846649+0000","last_active":"2024-02-19T15:22:07.424119+0000","last_peered":"2024-02-19T15:22:07.424119+0000","last_clean":"2024-02-19T15:22:07.424119+0000","last_became_active":"2024-02-19T15:22:05.846335+0000","last_became_peered":"2024-02-19T15:22:05.846335+0000","last_unstale":"2024-02-19T15:22:07.424119+0000","last_undegraded":"2024-02-19T15:22:07.424119+0000","last_fullsized":"2024-02-19T15:22:07.424119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:03:40.323060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851859+0000","last_change":"2024-02-19T15:22:05.870492+0000","last_active":"2024-02-19T15:22:06.851859+0000","last_peered":"2024-02-19T15:22:06.851859+0000","last_clean":"2024-02-19T15:22:06.851859+0000","last_became_active":"2024-02-19T15:22:05.868145+0000","last_became_peered":"2024-02-19T15:22:05.868145+0000","last_unstale":"2024-02-19T15:22:06.851859+0000","last_undegraded":"2024-02-19T15:22:06.851859+0000","last_fullsized":"2024-02-19T15:22:06.851859+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:03:40.323060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854818+0000","last_change":"2024-02-19T15:22:05.869349+0000","last_active":"2024-02-19T15:22:06.854818+0000","last_peered":"2024-02-19T15:22:06.854818+0000","last_clean":"2024-02-19T15:22:06.854818+0000","last_became_active":"2024-02-19T15:22:05.869042+0000","last_became_peered":"2024-02-19T15:22:05.869042+0000","last_unstale":"2024-02-19T15:22:06.854818+0000","last_undegraded":"2024-02-19T15:22:06.854818+0000","last_fullsized":"2024-02-19T15:22:06.854818+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:14:16.136182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853505+0000","last_change":"2024-02-19T15:22:05.856951+0000","last_active":"2024-02-19T15:22:06.853505+0000","last_peered":"2024-02-19T15:22:06.853505+0000","last_clean":"2024-02-19T15:22:06.853505+0000","last_became_active":"2024-02-19T15:22:05.856670+0000","last_became_peered":"2024-02-19T15:22:05.856670+0000","last_unstale":"2024-02-19T15:22:06.853505+0000","last_undegraded":"2024-02-19T15:22:06.853505+0000","last_fullsized":"2024-02-19T15:22:06.853505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:48:05.400374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851719+0000","last_change":"2024-02-19T15:22:05.868491+0000","last_active":"2024-02-19T15:22:06.851719+0000","last_peered":"2024-02-19T15:22:06.851719+0000","last_clean":"2024-02-19T15:22:06.851719+0000","last_became_active":"2024-02-19T15:22:05.868184+0000","last_became_peered":"2024-02-19T15:22:05.868184+0000","last_unstale":"2024-02-19T15:22:06.851719+0000","last_undegraded":"2024-02-19T15:22:06.851719+0000","last_fullsized":"2024-02-19T15:22:06.851719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:14:40.204009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424406+0000","last_change":"2024-02-19T15:22:05.847847+0000","last_active":"2024-02-19T15:22:07.424406+0000","last_peered":"2024-02-19T15:22:07.424406+0000","last_clean":"2024-02-19T15:22:07.424406+0000","last_became_active":"2024-02-19T15:22:05.847538+0000","last_became_peered":"2024-02-19T15:22:05.847538+0000","last_unstale":"2024-02-19T15:22:07.424406+0000","last_undegraded":"2024-02-19T15:22:07.424406+0000","last_fullsized":"2024-02-19T15:22:07.424406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:08.091823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424277+0000","last_change":"2024-02-19T15:22:05.847222+0000","last_active":"2024-02-19T15:22:07.424277+0000","last_peered":"2024-02-19T15:22:07.424277+0000","last_clean":"2024-02-19T15:22:07.424277+0000","last_became_active":"2024-02-19T15:22:05.846957+0000","last_became_peered":"2024-02-19T15:22:05.846957+0000","last_unstale":"2024-02-19T15:22:07.424277+0000","last_undegraded":"2024-02-19T15:22:07.424277+0000","last_fullsized":"2024-02-19T15:22:07.424277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:08:56.380125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.423913+0000","last_change":"2024-02-19T15:22:05.862263+0000","last_active":"2024-02-19T15:22:07.423913+0000","last_peered":"2024-02-19T15:22:07.423913+0000","last_clean":"2024-02-19T15:22:07.423913+0000","last_became_active":"2024-02-19T15:22:05.861932+0000","last_became_peered":"2024-02-19T15:22:05.861932+0000","last_unstale":"2024-02-19T15:22:07.423913+0000","last_undegraded":"2024-02-19T15:22:07.423913+0000","last_fullsized":"2024-02-19T15:22:07.423913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:37:51.996719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854810+0000","last_change":"2024-02-19T15:22:05.852998+0000","last_active":"2024-02-19T15:22:06.854810+0000","last_peered":"2024-02-19T15:22:06.854810+0000","last_clean":"2024-02-19T15:22:06.854810+0000","last_became_active":"2024-02-19T15:22:05.852660+0000","last_became_peered":"2024-02-19T15:22:05.852660+0000","last_unstale":"2024-02-19T15:22:06.854810+0000","last_undegraded":"2024-02-19T15:22:06.854810+0000","last_fullsized":"2024-02-19T15:22:06.854810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:54:25.604351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.423850+0000","last_change":"2024-02-19T15:22:05.862188+0000","last_active":"2024-02-19T15:22:07.423850+0000","last_peered":"2024-02-19T15:22:07.423850+0000","last_clean":"2024-02-19T15:22:07.423850+0000","last_became_active":"2024-02-19T15:22:05.861789+0000","last_became_peered":"2024-02-19T15:22:05.861789+0000","last_unstale":"2024-02-19T15:22:07.423850+0000","last_undegraded":"2024-02-19T15:22:07.423850+0000","last_fullsized":"2024-02-19T15:22:07.423850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:41.785448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851552+0000","last_change":"2024-02-19T15:22:04.848427+0000","last_active":"2024-02-19T15:22:06.851552+0000","last_peered":"2024-02-19T15:22:06.851552+0000","last_clean":"2024-02-19T15:22:06.851552+0000","last_became_active":"2024-02-19T15:22:04.848111+0000","last_became_peered":"2024-02-19T15:22:04.848111+0000","last_unstale":"2024-02-19T15:22:06.851552+0000","last_undegraded":"2024-02-19T15:22:06.851552+0000","last_fullsized":"2024-02-19T15:22:06.851552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:17.404136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424555+0000","last_change":"2024-02-19T15:22:05.845098+0000","last_active":"2024-02-19T15:22:07.424555+0000","last_peered":"2024-02-19T15:22:07.424555+0000","last_clean":"2024-02-19T15:22:07.424555+0000","last_became_active":"2024-02-19T15:22:05.844805+0000","last_became_peered":"2024-02-19T15:22:05.844805+0000","last_unstale":"2024-02-19T15:22:07.424555+0000","last_undegraded":"2024-02-19T15:22:07.424555+0000","last_fullsized":"2024-02-19T15:22:07.424555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:14:16.136182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854248+0000","last_change":"2024-02-19T15:22:04.842285+0000","last_active":"2024-02-19T15:22:06.854248+0000","last_peered":"2024-02-19T15:22:06.854248+0000","last_clean":"2024-02-19T15:22:06.854248+0000","last_became_active":"2024-02-19T15:22:04.840126+0000","last_became_peered":"2024-02-19T15:22:04.840126+0000","last_unstale":"2024-02-19T15:22:06.854248+0000","last_undegraded":"2024-02-19T15:22:06.854248+0000","last_fullsized":"2024-02-19T15:22:06.854248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:11:04.017098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851307+0000","last_change":"2024-02-19T15:22:04.839208+0000","last_active":"2024-02-19T15:22:06.851307+0000","last_peered":"2024-02-19T15:22:06.851307+0000","last_clean":"2024-02-19T15:22:06.851307+0000","last_became_active":"2024-02-19T15:22:04.838857+0000","last_became_peered":"2024-02-19T15:22:04.838857+0000","last_unstale":"2024-02-19T15:22:06.851307+0000","last_undegraded":"2024-02-19T15:22:06.851307+0000","last_fullsized":"2024-02-19T15:22:06.851307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:52:37.920564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851309+0000","last_change":"2024-02-19T15:22:05.859786+0000","last_active":"2024-02-19T15:22:06.851309+0000","last_peered":"2024-02-19T15:22:06.851309+0000","last_clean":"2024-02-19T15:22:06.851309+0000","last_became_active":"2024-02-19T15:22:05.854215+0000","last_became_peered":"2024-02-19T15:22:05.854215+0000","last_unstale":"2024-02-19T15:22:06.851309+0000","last_undegraded":"2024-02-19T15:22:06.851309+0000","last_fullsized":"2024-02-19T15:22:06.851309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:41.123853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424205+0000","last_change":"2024-02-19T15:22:04.843918+0000","last_active":"2024-02-19T15:22:07.424205+0000","last_peered":"2024-02-19T15:22:07.424205+0000","last_clean":"2024-02-19T15:22:07.424205+0000","last_became_active":"2024-02-19T15:22:04.842002+0000","last_became_peered":"2024-02-19T15:22:04.842002+0000","last_unstale":"2024-02-19T15:22:07.424205+0000","last_undegraded":"2024-02-19T15:22:07.424205+0000","last_fullsized":"2024-02-19T15:22:07.424205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:27.325777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853182+0000","last_change":"2024-02-19T15:22:05.852203+0000","last_active":"2024-02-19T15:22:06.853182+0000","last_peered":"2024-02-19T15:22:06.853182+0000","last_clean":"2024-02-19T15:22:06.853182+0000","last_became_active":"2024-02-19T15:22:05.850442+0000","last_became_peered":"2024-02-19T15:22:05.850442+0000","last_unstale":"2024-02-19T15:22:06.853182+0000","last_undegraded":"2024-02-19T15:22:06.853182+0000","last_fullsized":"2024-02-19T15:22:06.853182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:41.123853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854530+0000","last_change":"2024-02-19T15:22:04.839817+0000","last_active":"2024-02-19T15:22:06.854530+0000","last_peered":"2024-02-19T15:22:06.854530+0000","last_clean":"2024-02-19T15:22:06.854530+0000","last_became_active":"2024-02-19T15:22:04.839034+0000","last_became_peered":"2024-02-19T15:22:04.839034+0000","last_unstale":"2024-02-19T15:22:06.854530+0000","last_undegraded":"2024-02-19T15:22:06.854530+0000","last_fullsized":"2024-02-19T15:22:06.854530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:16:21.476088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424743+0000","last_change":"2024-02-19T15:22:05.869563+0000","last_active":"2024-02-19T15:22:07.424743+0000","last_peered":"2024-02-19T15:22:07.424743+0000","last_clean":"2024-02-19T15:22:07.424743+0000","last_became_active":"2024-02-19T15:22:05.869260+0000","last_became_peered":"2024-02-19T15:22:05.869260+0000","last_unstale":"2024-02-19T15:22:07.424743+0000","last_undegraded":"2024-02-19T15:22:07.424743+0000","last_fullsized":"2024-02-19T15:22:07.424743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:26:42.865617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424823+0000","last_change":"2024-02-19T15:22:04.843586+0000","last_active":"2024-02-19T15:22:07.424823+0000","last_peered":"2024-02-19T15:22:07.424823+0000","last_clean":"2024-02-19T15:22:07.424823+0000","last_became_active":"2024-02-19T15:22:04.842983+0000","last_became_peered":"2024-02-19T15:22:04.842983+0000","last_unstale":"2024-02-19T15:22:07.424823+0000","last_undegraded":"2024-02-19T15:22:07.424823+0000","last_fullsized":"2024-02-19T15:22:07.424823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:21.851058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424603+0000","last_change":"2024-02-19T15:22:05.861506+0000","last_active":"2024-02-19T15:22:07.424603+0000","last_peered":"2024-02-19T15:22:07.424603+0000","last_clean":"2024-02-19T15:22:07.424603+0000","last_became_active":"2024-02-19T15:22:05.861172+0000","last_became_peered":"2024-02-19T15:22:05.861172+0000","last_unstale":"2024-02-19T15:22:07.424603+0000","last_undegraded":"2024-02-19T15:22:07.424603+0000","last_fullsized":"2024-02-19T15:22:07.424603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:26:48.255672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853456+0000","last_change":"2024-02-19T15:22:04.833984+0000","last_active":"2024-02-19T15:22:06.853456+0000","last_peered":"2024-02-19T15:22:06.853456+0000","last_clean":"2024-02-19T15:22:06.853456+0000","last_became_active":"2024-02-19T15:22:04.833418+0000","last_became_peered":"2024-02-19T15:22:04.833418+0000","last_unstale":"2024-02-19T15:22:06.853456+0000","last_undegraded":"2024-02-19T15:22:06.853456+0000","last_fullsized":"2024-02-19T15:22:06.853456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:31:17.425502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424400+0000","last_change":"2024-02-19T15:22:05.865117+0000","last_active":"2024-02-19T15:22:07.424400+0000","last_peered":"2024-02-19T15:22:07.424400+0000","last_clean":"2024-02-19T15:22:07.424400+0000","last_became_active":"2024-02-19T15:22:05.864502+0000","last_became_peered":"2024-02-19T15:22:05.864502+0000","last_unstale":"2024-02-19T15:22:07.424400+0000","last_undegraded":"2024-02-19T15:22:07.424400+0000","last_fullsized":"2024-02-19T15:22:07.424400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:38.607191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854661+0000","last_change":"2024-02-19T15:22:04.832285+0000","last_active":"2024-02-19T15:22:06.854661+0000","last_peered":"2024-02-19T15:22:06.854661+0000","last_clean":"2024-02-19T15:22:06.854661+0000","last_became_active":"2024-02-19T15:22:04.831992+0000","last_became_peered":"2024-02-19T15:22:04.831992+0000","last_unstale":"2024-02-19T15:22:06.854661+0000","last_undegraded":"2024-02-19T15:22:06.854661+0000","last_fullsized":"2024-02-19T15:22:06.854661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:50:48.637629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181470+0000","last_change":"2024-02-19T15:22:05.874836+0000","last_active":"2024-02-19T15:22:07.181470+0000","last_peered":"2024-02-19T15:22:07.181470+0000","last_clean":"2024-02-19T15:22:07.181470+0000","last_became_active":"2024-02-19T15:22:05.874522+0000","last_became_peered":"2024-02-19T15:22:05.874522+0000","last_unstale":"2024-02-19T15:22:07.181470+0000","last_undegraded":"2024-02-19T15:22:07.181470+0000","last_fullsized":"2024-02-19T15:22:07.181470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:26:42.865617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424505+0000","last_change":"2024-02-19T15:22:04.837502+0000","last_active":"2024-02-19T15:22:07.424505+0000","last_peered":"2024-02-19T15:22:07.424505+0000","last_clean":"2024-02-19T15:22:07.424505+0000","last_became_active":"2024-02-19T15:22:04.836913+0000","last_became_peered":"2024-02-19T15:22:04.836913+0000","last_unstale":"2024-02-19T15:22:07.424505+0000","last_undegraded":"2024-02-19T15:22:07.424505+0000","last_fullsized":"2024-02-19T15:22:07.424505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:20:00.631607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424276+0000","last_change":"2024-02-19T15:22:05.864869+0000","last_active":"2024-02-19T15:22:07.424276+0000","last_peered":"2024-02-19T15:22:07.424276+0000","last_clean":"2024-02-19T15:22:07.424276+0000","last_became_active":"2024-02-19T15:22:05.864424+0000","last_became_peered":"2024-02-19T15:22:05.864424+0000","last_unstale":"2024-02-19T15:22:07.424276+0000","last_undegraded":"2024-02-19T15:22:07.424276+0000","last_fullsized":"2024-02-19T15:22:07.424276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:22:17.882758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424358+0000","last_change":"2024-02-19T15:22:04.847248+0000","last_active":"2024-02-19T15:22:07.424358+0000","last_peered":"2024-02-19T15:22:07.424358+0000","last_clean":"2024-02-19T15:22:07.424358+0000","last_became_active":"2024-02-19T15:22:04.846883+0000","last_became_peered":"2024-02-19T15:22:04.846883+0000","last_unstale":"2024-02-19T15:22:07.424358+0000","last_undegraded":"2024-02-19T15:22:07.424358+0000","last_fullsized":"2024-02-19T15:22:07.424358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:40.779845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181361+0000","last_change":"2024-02-19T15:22:05.873228+0000","last_active":"2024-02-19T15:22:07.181361+0000","last_peered":"2024-02-19T15:22:07.181361+0000","last_clean":"2024-02-19T15:22:07.181361+0000","last_became_active":"2024-02-19T15:22:05.872458+0000","last_became_peered":"2024-02-19T15:22:05.872458+0000","last_unstale":"2024-02-19T15:22:07.181361+0000","last_undegraded":"2024-02-19T15:22:07.181361+0000","last_fullsized":"2024-02-19T15:22:07.181361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:59:01.655185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854298+0000","last_change":"2024-02-19T15:22:04.843476+0000","last_active":"2024-02-19T15:22:06.854298+0000","last_peered":"2024-02-19T15:22:06.854298+0000","last_clean":"2024-02-19T15:22:06.854298+0000","last_became_active":"2024-02-19T15:22:04.843159+0000","last_became_peered":"2024-02-19T15:22:04.843159+0000","last_unstale":"2024-02-19T15:22:06.854298+0000","last_undegraded":"2024-02-19T15:22:06.854298+0000","last_fullsized":"2024-02-19T15:22:06.854298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:21:58.456669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854827+0000","last_change":"2024-02-19T15:22:05.852049+0000","last_active":"2024-02-19T15:22:06.854827+0000","last_peered":"2024-02-19T15:22:06.854827+0000","last_clean":"2024-02-19T15:22:06.854827+0000","last_became_active":"2024-02-19T15:22:05.851786+0000","last_became_peered":"2024-02-19T15:22:05.851786+0000","last_unstale":"2024-02-19T15:22:06.854827+0000","last_undegraded":"2024-02-19T15:22:06.854827+0000","last_fullsized":"2024-02-19T15:22:06.854827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:31:59.866248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.423650+0000","last_change":"2024-02-19T15:22:04.844209+0000","last_active":"2024-02-19T15:22:07.423650+0000","last_peered":"2024-02-19T15:22:07.423650+0000","last_clean":"2024-02-19T15:22:07.423650+0000","last_became_active":"2024-02-19T15:22:04.842885+0000","last_became_peered":"2024-02-19T15:22:04.842885+0000","last_unstale":"2024-02-19T15:22:07.423650+0000","last_undegraded":"2024-02-19T15:22:07.423650+0000","last_fullsized":"2024-02-19T15:22:07.423650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:27.325777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181171+0000","last_change":"2024-02-19T15:22:05.877219+0000","last_active":"2024-02-19T15:22:07.181171+0000","last_peered":"2024-02-19T15:22:07.181171+0000","last_clean":"2024-02-19T15:22:07.181171+0000","last_became_active":"2024-02-19T15:22:05.876953+0000","last_became_peered":"2024-02-19T15:22:05.876953+0000","last_unstale":"2024-02-19T15:22:07.181171+0000","last_undegraded":"2024-02-19T15:22:07.181171+0000","last_fullsized":"2024-02-19T15:22:07.181171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:55:27.572569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424947+0000","last_change":"2024-02-19T15:22:04.837084+0000","last_active":"2024-02-19T15:22:07.424947+0000","last_peered":"2024-02-19T15:22:07.424947+0000","last_clean":"2024-02-19T15:22:07.424947+0000","last_became_active":"2024-02-19T15:22:04.836783+0000","last_became_peered":"2024-02-19T15:22:04.836783+0000","last_unstale":"2024-02-19T15:22:07.424947+0000","last_undegraded":"2024-02-19T15:22:07.424947+0000","last_fullsized":"2024-02-19T15:22:07.424947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:34:14.847696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181498+0000","last_change":"2024-02-19T15:22:05.876111+0000","last_active":"2024-02-19T15:22:07.181498+0000","last_peered":"2024-02-19T15:22:07.181498+0000","last_clean":"2024-02-19T15:22:07.181498+0000","last_became_active":"2024-02-19T15:22:05.875843+0000","last_became_peered":"2024-02-19T15:22:05.875843+0000","last_unstale":"2024-02-19T15:22:07.181498+0000","last_undegraded":"2024-02-19T15:22:07.181498+0000","last_fullsized":"2024-02-19T15:22:07.181498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:29:35.769837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.423944+0000","last_change":"2024-02-19T15:22:04.839982+0000","last_active":"2024-02-19T15:22:07.423944+0000","last_peered":"2024-02-19T15:22:07.423944+0000","last_clean":"2024-02-19T15:22:07.423944+0000","last_became_active":"2024-02-19T15:22:04.839617+0000","last_became_peered":"2024-02-19T15:22:04.839617+0000","last_unstale":"2024-02-19T15:22:07.423944+0000","last_undegraded":"2024-02-19T15:22:07.423944+0000","last_fullsized":"2024-02-19T15:22:07.423944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:21.137278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181744+0000","last_change":"2024-02-19T15:22:05.870359+0000","last_active":"2024-02-19T15:22:07.181744+0000","last_peered":"2024-02-19T15:22:07.181744+0000","last_clean":"2024-02-19T15:22:07.181744+0000","last_became_active":"2024-02-19T15:22:05.870083+0000","last_became_peered":"2024-02-19T15:22:05.870083+0000","last_unstale":"2024-02-19T15:22:07.181744+0000","last_undegraded":"2024-02-19T15:22:07.181744+0000","last_fullsized":"2024-02-19T15:22:07.181744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:41:29.186489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853581+0000","last_change":"2024-02-19T15:22:04.828045+0000","last_active":"2024-02-19T15:22:06.853581+0000","last_peered":"2024-02-19T15:22:06.853581+0000","last_clean":"2024-02-19T15:22:06.853581+0000","last_became_active":"2024-02-19T15:22:04.827744+0000","last_became_peered":"2024-02-19T15:22:04.827744+0000","last_unstale":"2024-02-19T15:22:06.853581+0000","last_undegraded":"2024-02-19T15:22:06.853581+0000","last_fullsized":"2024-02-19T15:22:06.853581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:21.851058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424587+0000","last_change":"2024-02-19T15:22:05.849002+0000","last_active":"2024-02-19T15:22:07.424587+0000","last_peered":"2024-02-19T15:22:07.424587+0000","last_clean":"2024-02-19T15:22:07.424587+0000","last_became_active":"2024-02-19T15:22:05.848692+0000","last_became_peered":"2024-02-19T15:22:05.848692+0000","last_unstale":"2024-02-19T15:22:07.424587+0000","last_undegraded":"2024-02-19T15:22:07.424587+0000","last_fullsized":"2024-02-19T15:22:07.424587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:14:10.262240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.423816+0000","last_change":"2024-02-19T15:22:04.851700+0000","last_active":"2024-02-19T15:22:07.423816+0000","last_peered":"2024-02-19T15:22:07.423816+0000","last_clean":"2024-02-19T15:22:07.423816+0000","last_became_active":"2024-02-19T15:22:04.851270+0000","last_became_peered":"2024-02-19T15:22:04.851270+0000","last_unstale":"2024-02-19T15:22:07.423816+0000","last_undegraded":"2024-02-19T15:22:07.423816+0000","last_fullsized":"2024-02-19T15:22:07.423816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:40.779845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181370+0000","last_change":"2024-02-19T15:22:05.862153+0000","last_active":"2024-02-19T15:22:07.181370+0000","last_peered":"2024-02-19T15:22:07.181370+0000","last_clean":"2024-02-19T15:22:07.181370+0000","last_became_active":"2024-02-19T15:22:05.861836+0000","last_became_peered":"2024-02-19T15:22:05.861836+0000","last_unstale":"2024-02-19T15:22:07.181370+0000","last_undegraded":"2024-02-19T15:22:07.181370+0000","last_fullsized":"2024-02-19T15:22:07.181370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:07:38.614353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181103+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:22:07.181103+0000","last_peered":"2024-02-19T15:22:07.181103+0000","last_clean":"2024-02-19T15:22:07.181103+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:22:07.181103+0000","last_undegraded":"2024-02-19T15:22:07.181103+0000","last_fullsized":"2024-02-19T15:22:07.181103+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424087+0000","last_change":"2024-02-19T15:22:04.846635+0000","last_active":"2024-02-19T15:22:07.424087+0000","last_peered":"2024-02-19T15:22:07.424087+0000","last_clean":"2024-02-19T15:22:07.424087+0000","last_became_active":"2024-02-19T15:22:04.845191+0000","last_became_peered":"2024-02-19T15:22:04.845191+0000","last_unstale":"2024-02-19T15:22:07.424087+0000","last_undegraded":"2024-02-19T15:22:07.424087+0000","last_fullsized":"2024-02-19T15:22:07.424087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:31:17.425502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851571+0000","last_change":"2024-02-19T15:22:05.863647+0000","last_active":"2024-02-19T15:22:06.851571+0000","last_peered":"2024-02-19T15:22:06.851571+0000","last_clean":"2024-02-19T15:22:06.851571+0000","last_became_active":"2024-02-19T15:22:05.863370+0000","last_became_peered":"2024-02-19T15:22:05.863370+0000","last_unstale":"2024-02-19T15:22:06.851571+0000","last_undegraded":"2024-02-19T15:22:06.851571+0000","last_fullsized":"2024-02-19T15:22:06.851571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:48:05.400374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181034+0000","last_change":"2024-02-19T15:22:04.849295+0000","last_active":"2024-02-19T15:22:07.181034+0000","last_peered":"2024-02-19T15:22:07.181034+0000","last_clean":"2024-02-19T15:22:07.181034+0000","last_became_active":"2024-02-19T15:22:04.849004+0000","last_became_peered":"2024-02-19T15:22:04.849004+0000","last_unstale":"2024-02-19T15:22:07.181034+0000","last_undegraded":"2024-02-19T15:22:07.181034+0000","last_fullsized":"2024-02-19T15:22:07.181034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:37.601588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424163+0000","last_change":"2024-02-19T15:22:05.861044+0000","last_active":"2024-02-19T15:22:07.424163+0000","last_peered":"2024-02-19T15:22:07.424163+0000","last_clean":"2024-02-19T15:22:07.424163+0000","last_became_active":"2024-02-19T15:22:05.860784+0000","last_became_peered":"2024-02-19T15:22:05.860784+0000","last_unstale":"2024-02-19T15:22:07.424163+0000","last_undegraded":"2024-02-19T15:22:07.424163+0000","last_fullsized":"2024-02-19T15:22:07.424163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:16:26.878453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.423856+0000","last_change":"2024-02-19T15:22:04.842486+0000","last_active":"2024-02-19T15:22:07.423856+0000","last_peered":"2024-02-19T15:22:07.423856+0000","last_clean":"2024-02-19T15:22:07.423856+0000","last_became_active":"2024-02-19T15:22:04.840062+0000","last_became_peered":"2024-02-19T15:22:04.840062+0000","last_unstale":"2024-02-19T15:22:07.423856+0000","last_undegraded":"2024-02-19T15:22:07.423856+0000","last_fullsized":"2024-02-19T15:22:07.423856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:17.404136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853201+0000","last_change":"2024-02-19T15:22:05.853794+0000","last_active":"2024-02-19T15:22:06.853201+0000","last_peered":"2024-02-19T15:22:06.853201+0000","last_clean":"2024-02-19T15:22:06.853201+0000","last_became_active":"2024-02-19T15:22:05.851972+0000","last_became_peered":"2024-02-19T15:22:05.851972+0000","last_unstale":"2024-02-19T15:22:06.853201+0000","last_undegraded":"2024-02-19T15:22:06.853201+0000","last_fullsized":"2024-02-19T15:22:06.853201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:33:20.316638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424384+0000","last_change":"2024-02-19T15:22:04.851077+0000","last_active":"2024-02-19T15:22:07.424384+0000","last_peered":"2024-02-19T15:22:07.424384+0000","last_clean":"2024-02-19T15:22:07.424384+0000","last_became_active":"2024-02-19T15:22:04.850734+0000","last_became_peered":"2024-02-19T15:22:04.850734+0000","last_unstale":"2024-02-19T15:22:07.424384+0000","last_undegraded":"2024-02-19T15:22:07.424384+0000","last_fullsized":"2024-02-19T15:22:07.424384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:37.601588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424264+0000","last_change":"2024-02-19T15:22:05.859004+0000","last_active":"2024-02-19T15:22:07.424264+0000","last_peered":"2024-02-19T15:22:07.424264+0000","last_clean":"2024-02-19T15:22:07.424264+0000","last_became_active":"2024-02-19T15:22:05.858524+0000","last_became_peered":"2024-02-19T15:22:05.858524+0000","last_unstale":"2024-02-19T15:22:07.424264+0000","last_undegraded":"2024-02-19T15:22:07.424264+0000","last_fullsized":"2024-02-19T15:22:07.424264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:31:18.431106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853431+0000","last_change":"2024-02-19T15:22:04.832556+0000","last_active":"2024-02-19T15:22:06.853431+0000","last_peered":"2024-02-19T15:22:06.853431+0000","last_clean":"2024-02-19T15:22:06.853431+0000","last_became_active":"2024-02-19T15:22:04.832261+0000","last_became_peered":"2024-02-19T15:22:04.832261+0000","last_unstale":"2024-02-19T15:22:06.853431+0000","last_undegraded":"2024-02-19T15:22:06.853431+0000","last_fullsized":"2024-02-19T15:22:06.853431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:37.601588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424743+0000","last_change":"2024-02-19T15:22:05.850119+0000","last_active":"2024-02-19T15:22:07.424743+0000","last_peered":"2024-02-19T15:22:07.424743+0000","last_clean":"2024-02-19T15:22:07.424743+0000","last_became_active":"2024-02-19T15:22:05.849831+0000","last_became_peered":"2024-02-19T15:22:05.849831+0000","last_unstale":"2024-02-19T15:22:07.424743+0000","last_undegraded":"2024-02-19T15:22:07.424743+0000","last_fullsized":"2024-02-19T15:22:07.424743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:07:38.614353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181613+0000","last_change":"2024-02-19T15:22:04.850106+0000","last_active":"2024-02-19T15:22:07.181613+0000","last_peered":"2024-02-19T15:22:07.181613+0000","last_clean":"2024-02-19T15:22:07.181613+0000","last_became_active":"2024-02-19T15:22:04.849467+0000","last_became_peered":"2024-02-19T15:22:04.849467+0000","last_unstale":"2024-02-19T15:22:07.181613+0000","last_undegraded":"2024-02-19T15:22:07.181613+0000","last_fullsized":"2024-02-19T15:22:07.181613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:32:55.903921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853330+0000","last_change":"2024-02-19T15:22:05.857834+0000","last_active":"2024-02-19T15:22:06.853330+0000","last_peered":"2024-02-19T15:22:06.853330+0000","last_clean":"2024-02-19T15:22:06.853330+0000","last_became_active":"2024-02-19T15:22:05.857488+0000","last_became_peered":"2024-02-19T15:22:05.857488+0000","last_unstale":"2024-02-19T15:22:06.853330+0000","last_undegraded":"2024-02-19T15:22:06.853330+0000","last_fullsized":"2024-02-19T15:22:06.853330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:21:49.261015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853569+0000","last_change":"2024-02-19T15:22:04.830568+0000","last_active":"2024-02-19T15:22:06.853569+0000","last_peered":"2024-02-19T15:22:06.853569+0000","last_clean":"2024-02-19T15:22:06.853569+0000","last_became_active":"2024-02-19T15:22:04.829131+0000","last_became_peered":"2024-02-19T15:22:04.829131+0000","last_unstale":"2024-02-19T15:22:06.853569+0000","last_undegraded":"2024-02-19T15:22:06.853569+0000","last_fullsized":"2024-02-19T15:22:06.853569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:37:50.991116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.852021+0000","last_change":"2024-02-19T15:22:05.866131+0000","last_active":"2024-02-19T15:22:06.852021+0000","last_peered":"2024-02-19T15:22:06.852021+0000","last_clean":"2024-02-19T15:22:06.852021+0000","last_became_active":"2024-02-19T15:22:05.865820+0000","last_became_peered":"2024-02-19T15:22:05.865820+0000","last_unstale":"2024-02-19T15:22:06.852021+0000","last_undegraded":"2024-02-19T15:22:06.852021+0000","last_fullsized":"2024-02-19T15:22:06.852021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:46:15.823647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854644+0000","last_change":"2024-02-19T15:22:04.840658+0000","last_active":"2024-02-19T15:22:06.854644+0000","last_peered":"2024-02-19T15:22:06.854644+0000","last_clean":"2024-02-19T15:22:06.854644+0000","last_became_active":"2024-02-19T15:22:04.840167+0000","last_became_peered":"2024-02-19T15:22:04.840167+0000","last_unstale":"2024-02-19T15:22:06.854644+0000","last_undegraded":"2024-02-19T15:22:06.854644+0000","last_fullsized":"2024-02-19T15:22:06.854644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:26:19.726455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424579+0000","last_change":"2024-02-19T15:22:05.854827+0000","last_active":"2024-02-19T15:22:07.424579+0000","last_peered":"2024-02-19T15:22:07.424579+0000","last_clean":"2024-02-19T15:22:07.424579+0000","last_became_active":"2024-02-19T15:22:05.854579+0000","last_became_peered":"2024-02-19T15:22:05.854579+0000","last_unstale":"2024-02-19T15:22:07.424579+0000","last_undegraded":"2024-02-19T15:22:07.424579+0000","last_fullsized":"2024-02-19T15:22:07.424579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:31:59.866248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181002+0000","last_change":"2024-02-19T15:22:04.848481+0000","last_active":"2024-02-19T15:22:07.181002+0000","last_peered":"2024-02-19T15:22:07.181002+0000","last_clean":"2024-02-19T15:22:07.181002+0000","last_became_active":"2024-02-19T15:22:04.848224+0000","last_became_peered":"2024-02-19T15:22:04.848224+0000","last_unstale":"2024-02-19T15:22:07.181002+0000","last_undegraded":"2024-02-19T15:22:07.181002+0000","last_fullsized":"2024-02-19T15:22:07.181002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:21.851058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424492+0000","last_change":"2024-02-19T15:22:05.868543+0000","last_active":"2024-02-19T15:22:07.424492+0000","last_peered":"2024-02-19T15:22:07.424492+0000","last_clean":"2024-02-19T15:22:07.424492+0000","last_became_active":"2024-02-19T15:22:05.868307+0000","last_became_peered":"2024-02-19T15:22:05.868307+0000","last_unstale":"2024-02-19T15:22:07.424492+0000","last_undegraded":"2024-02-19T15:22:07.424492+0000","last_fullsized":"2024-02-19T15:22:07.424492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:37:51.996719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853286+0000","last_change":"2024-02-19T15:22:04.841547+0000","last_active":"2024-02-19T15:22:06.853286+0000","last_peered":"2024-02-19T15:22:06.853286+0000","last_clean":"2024-02-19T15:22:06.853286+0000","last_became_active":"2024-02-19T15:22:04.841280+0000","last_became_peered":"2024-02-19T15:22:04.841280+0000","last_unstale":"2024-02-19T15:22:06.853286+0000","last_undegraded":"2024-02-19T15:22:06.853286+0000","last_fullsized":"2024-02-19T15:22:06.853286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:40.779845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851683+0000","last_change":"2024-02-19T15:22:05.863084+0000","last_active":"2024-02-19T15:22:06.851683+0000","last_peered":"2024-02-19T15:22:06.851683+0000","last_clean":"2024-02-19T15:22:06.851683+0000","last_became_active":"2024-02-19T15:22:05.862820+0000","last_became_peered":"2024-02-19T15:22:05.862820+0000","last_unstale":"2024-02-19T15:22:06.851683+0000","last_undegraded":"2024-02-19T15:22:06.851683+0000","last_fullsized":"2024-02-19T15:22:06.851683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:43.283192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853263+0000","last_change":"2024-02-19T15:22:04.840145+0000","last_active":"2024-02-19T15:22:06.853263+0000","last_peered":"2024-02-19T15:22:06.853263+0000","last_clean":"2024-02-19T15:22:06.853263+0000","last_became_active":"2024-02-19T15:22:04.839832+0000","last_became_peered":"2024-02-19T15:22:04.839832+0000","last_unstale":"2024-02-19T15:22:06.853263+0000","last_undegraded":"2024-02-19T15:22:06.853263+0000","last_fullsized":"2024-02-19T15:22:06.853263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:27.325777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851519+0000","last_change":"2024-02-19T15:22:05.862437+0000","last_active":"2024-02-19T15:22:06.851519+0000","last_peered":"2024-02-19T15:22:06.851519+0000","last_clean":"2024-02-19T15:22:06.851519+0000","last_became_active":"2024-02-19T15:22:05.855551+0000","last_became_peered":"2024-02-19T15:22:05.855551+0000","last_unstale":"2024-02-19T15:22:06.851519+0000","last_undegraded":"2024-02-19T15:22:06.851519+0000","last_fullsized":"2024-02-19T15:22:06.851519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:21:49.261015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181446+0000","last_change":"2024-02-19T15:22:04.849761+0000","last_active":"2024-02-19T15:22:07.181446+0000","last_peered":"2024-02-19T15:22:07.181446+0000","last_clean":"2024-02-19T15:22:07.181446+0000","last_became_active":"2024-02-19T15:22:04.849413+0000","last_became_peered":"2024-02-19T15:22:04.849413+0000","last_unstale":"2024-02-19T15:22:07.181446+0000","last_undegraded":"2024-02-19T15:22:07.181446+0000","last_fullsized":"2024-02-19T15:22:07.181446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:37:50.991116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854363+0000","last_change":"2024-02-19T15:22:05.871270+0000","last_active":"2024-02-19T15:22:06.854363+0000","last_peered":"2024-02-19T15:22:06.854363+0000","last_clean":"2024-02-19T15:22:06.854363+0000","last_became_active":"2024-02-19T15:22:05.870986+0000","last_became_peered":"2024-02-19T15:22:05.870986+0000","last_unstale":"2024-02-19T15:22:06.854363+0000","last_undegraded":"2024-02-19T15:22:06.854363+0000","last_fullsized":"2024-02-19T15:22:06.854363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:22:17.882758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424727+0000","last_change":"2024-02-19T15:22:05.864283+0000","last_active":"2024-02-19T15:22:07.424727+0000","last_peered":"2024-02-19T15:22:07.424727+0000","last_clean":"2024-02-19T15:22:07.424727+0000","last_became_active":"2024-02-19T15:22:05.863996+0000","last_became_peered":"2024-02-19T15:22:05.863996+0000","last_unstale":"2024-02-19T15:22:07.424727+0000","last_undegraded":"2024-02-19T15:22:07.424727+0000","last_fullsized":"2024-02-19T15:22:07.424727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:59:01.655185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181379+0000","last_change":"2024-02-19T15:22:04.855753+0000","last_active":"2024-02-19T15:22:07.181379+0000","last_peered":"2024-02-19T15:22:07.181379+0000","last_clean":"2024-02-19T15:22:07.181379+0000","last_became_active":"2024-02-19T15:22:04.855303+0000","last_became_peered":"2024-02-19T15:22:04.855303+0000","last_unstale":"2024-02-19T15:22:07.181379+0000","last_undegraded":"2024-02-19T15:22:07.181379+0000","last_fullsized":"2024-02-19T15:22:07.181379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:04.751023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424554+0000","last_change":"2024-02-19T15:22:05.870398+0000","last_active":"2024-02-19T15:22:07.424554+0000","last_peered":"2024-02-19T15:22:07.424554+0000","last_clean":"2024-02-19T15:22:07.424554+0000","last_became_active":"2024-02-19T15:22:05.869963+0000","last_became_peered":"2024-02-19T15:22:05.869963+0000","last_unstale":"2024-02-19T15:22:07.424554+0000","last_undegraded":"2024-02-19T15:22:07.424554+0000","last_fullsized":"2024-02-19T15:22:07.424554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:55:27.572569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181517+0000","last_change":"2024-02-19T15:22:04.849798+0000","last_active":"2024-02-19T15:22:07.181517+0000","last_peered":"2024-02-19T15:22:07.181517+0000","last_clean":"2024-02-19T15:22:07.181517+0000","last_became_active":"2024-02-19T15:22:04.849115+0000","last_became_peered":"2024-02-19T15:22:04.849115+0000","last_unstale":"2024-02-19T15:22:07.181517+0000","last_undegraded":"2024-02-19T15:22:07.181517+0000","last_fullsized":"2024-02-19T15:22:07.181517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:01.912213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424542+0000","last_change":"2024-02-19T15:22:05.861837+0000","last_active":"2024-02-19T15:22:07.424542+0000","last_peered":"2024-02-19T15:22:07.424542+0000","last_clean":"2024-02-19T15:22:07.424542+0000","last_became_active":"2024-02-19T15:22:05.861570+0000","last_became_peered":"2024-02-19T15:22:05.861570+0000","last_unstale":"2024-02-19T15:22:07.424542+0000","last_undegraded":"2024-02-19T15:22:07.424542+0000","last_fullsized":"2024-02-19T15:22:07.424542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:54:25.604351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851906+0000","last_change":"2024-02-19T15:22:04.841218+0000","last_active":"2024-02-19T15:22:06.851906+0000","last_peered":"2024-02-19T15:22:06.851906+0000","last_clean":"2024-02-19T15:22:06.851906+0000","last_became_active":"2024-02-19T15:22:04.840740+0000","last_became_peered":"2024-02-19T15:22:04.840740+0000","last_unstale":"2024-02-19T15:22:06.851906+0000","last_undegraded":"2024-02-19T15:22:06.851906+0000","last_fullsized":"2024-02-19T15:22:06.851906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:21.137278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851910+0000","last_change":"2024-02-19T15:22:05.847219+0000","last_active":"2024-02-19T15:22:06.851910+0000","last_peered":"2024-02-19T15:22:06.851910+0000","last_clean":"2024-02-19T15:22:06.851910+0000","last_became_active":"2024-02-19T15:22:05.846948+0000","last_became_peered":"2024-02-19T15:22:05.846948+0000","last_unstale":"2024-02-19T15:22:06.851910+0000","last_undegraded":"2024-02-19T15:22:06.851910+0000","last_fullsized":"2024-02-19T15:22:06.851910+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:08:56.380125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424663+0000","last_change":"2024-02-19T15:22:04.843469+0000","last_active":"2024-02-19T15:22:07.424663+0000","last_peered":"2024-02-19T15:22:07.424663+0000","last_clean":"2024-02-19T15:22:07.424663+0000","last_became_active":"2024-02-19T15:22:04.842098+0000","last_became_peered":"2024-02-19T15:22:04.842098+0000","last_unstale":"2024-02-19T15:22:07.424663+0000","last_undegraded":"2024-02-19T15:22:07.424663+0000","last_fullsized":"2024-02-19T15:22:07.424663+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:32:33.319050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509447,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]}]},{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":664833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]}]},{"osd":5,"up_from":14,"seq":60129542152,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":73973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]}]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":74009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":75495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":73883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.7160000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.1620000000000008}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.7679999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.3810000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4039999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5220000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4829999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":221184,"data_stored":74043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":819200,"data_stored":664889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:22:14.644 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:22:14.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.822+0000 7f8ddc600640 1 Processor -- start 2024-02-19T15:22:14.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.822+0000 7f8ddc600640 1 -- start start 2024-02-19T15:22:14.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.822+0000 7f8ddc600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd40ced10 0x7f8dd41690a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:14.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.822+0000 7f8ddc600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd40cce30 con 0x7f8dd406d650 2024-02-19T15:22:14.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.822+0000 7f8ddc600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd40cd110 con 0x7f8dd40cded0 2024-02-19T15:22:14.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.822+0000 7f8ddc600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd40cd3f0 con 0x7f8dd40ced10 2024-02-19T15:22:14.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.822+0000 7f8dd9c00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8dd406d650 0x7f8dd40d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35356/0 (socket says 172.21.15.47:35356) 2024-02-19T15:22:14.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.822+0000 7f8dd9c00640 1 -- 172.21.15.47:0/789930614 learned_addr learned my addr 172.21.15.47:0/789930614 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:14.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.822+0000 7f8dda600640 1 --2- 172.21.15.47:0/789930614 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd40ced10 0x7f8dd41690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:14.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.822+0000 7f8dd3e00640 1 -- 172.21.15.47:0/789930614 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165486532 0 0) 0x7f8dd40cce30 con 0x7f8dd406d650 2024-02-19T15:22:14.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.826+0000 7f8dd3e00640 1 -- 172.21.15.47:0/789930614 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dbc003530 con 0x7f8dd406d650 2024-02-19T15:22:14.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.826+0000 7f8dd3e00640 1 -- 172.21.15.47:0/789930614 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035752771 0 0) 0x7f8dd40cd110 con 0x7f8dd40cded0 2024-02-19T15:22:14.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.826+0000 7f8dda600640 1 -- 172.21.15.47:0/789930614 >> v1:172.21.15.116:6789/0 conn(0x7f8dd40cded0 legacy=0x7f8dd40ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.826+0000 7f8dda600640 1 -- 172.21.15.47:0/789930614 >> v1:172.21.15.47:6789/0 conn(0x7f8dd406d650 legacy=0x7f8dd40d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.826+0000 7f8dda600640 1 -- 172.21.15.47:0/789930614 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd40cd6d0 con 0x7f8dd40ced10 2024-02-19T15:22:14.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.826+0000 7f8dda600640 1 --2- 172.21.15.47:0/789930614 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd40ced10 0x7f8dd41690a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc009dc0 tx=0x7f8dcc01c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8639098bfc61bbb1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:14.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.826+0000 7f8dd3e00640 1 -- 172.21.15.47:0/789930614 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc0a59b0 con 0x7f8dd40ced10 2024-02-19T15:22:14.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.826+0000 7f8dd3e00640 1 -- 172.21.15.47:0/789930614 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dcc0a5b10 con 0x7f8dd40ced10 2024-02-19T15:22:14.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.826+0000 7f8dd3e00640 1 -- 172.21.15.47:0/789930614 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc0a5ea0 con 0x7f8dd40ced10 2024-02-19T15:22:14.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.826+0000 7f8ddc600640 1 -- 172.21.15.47:0/789930614 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd40ced10 msgr2=0x7f8dd41690a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.826+0000 7f8ddc600640 1 --2- 172.21.15.47:0/789930614 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd40ced10 0x7f8dd41690a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc009dc0 tx=0x7f8dcc01c8d0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.826+0000 7f8ddc600640 1 -- 172.21.15.47:0/789930614 shutdown_connections 2024-02-19T15:22:14.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.826+0000 7f8ddc600640 1 --2- 172.21.15.47:0/789930614 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd40ced10 0x7f8dd41690a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.826+0000 7f8ddc600640 1 -- 172.21.15.47:0/789930614 >> 172.21.15.47:0/789930614 conn(0x7f8dd40d1bf0 msgr2=0x7f8dd40d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:14.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.826+0000 7f8ddc600640 1 -- 172.21.15.47:0/789930614 shutdown_connections 2024-02-19T15:22:14.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8ddc600640 1 -- 172.21.15.47:0/789930614 wait complete. 2024-02-19T15:22:14.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8ddc600640 1 Processor -- start 2024-02-19T15:22:14.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8ddc600640 1 -- start start 2024-02-19T15:22:14.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8ddc600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dd406d650 0x7f8dd415c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:14.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8ddc600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dd40cded0 0x7f8dd41611c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:14.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8ddc600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd40ced10 0x7f8dd414c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:14.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8ddc600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd415e9c0 con 0x7f8dd406d650 2024-02-19T15:22:14.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8ddc600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd415daf0 con 0x7f8dd40cded0 2024-02-19T15:22:14.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8ddc600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd4161700 con 0x7f8dd40ced10 2024-02-19T15:22:14.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8dd9c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dd406d650 0x7f8dd415c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:14.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8dd9c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dd406d650 0x7f8dd415c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57888/0 (socket says 172.21.15.47:57888) 2024-02-19T15:22:14.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8dd9c00640 1 -- 172.21.15.47:0/3847285423 learned_addr learned my addr 172.21.15.47:0/3847285423 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:14.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8dda600640 1 --2- 172.21.15.47:0/3847285423 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd40ced10 0x7f8dd414c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:14.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8dd9200640 1 --2- 172.21.15.47:0/3847285423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dd40cded0 0x7f8dd41611c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:14.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8dd9c00640 1 -- 172.21.15.47:0/3847285423 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd40ced10 msgr2=0x7f8dd414c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8dd9c00640 1 --2- 172.21.15.47:0/3847285423 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd40ced10 0x7f8dd414c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8dd9c00640 1 -- 172.21.15.47:0/3847285423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dd40cded0 msgr2=0x7f8dd41611c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:14.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8dd9c00640 1 --2- 172.21.15.47:0/3847285423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dd40cded0 0x7f8dd41611c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:14.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8dd9c00640 1 -- 172.21.15.47:0/3847285423 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dcc009a70 con 0x7f8dd406d650 2024-02-19T15:22:14.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8dd9200640 1 --2- 172.21.15.47:0/3847285423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dd40cded0 0x7f8dd41611c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:14.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8dd9c00640 1 --2- 172.21.15.47:0/3847285423 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dd406d650 0x7f8dd415c7f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8dc800c3c0 tx=0x7f8dc800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:14.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8dd1600640 1 -- 172.21.15.47:0/3847285423 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8005cf0 con 0x7f8dd406d650 2024-02-19T15:22:14.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8dd1600640 1 -- 172.21.15.47:0/3847285423 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc8005e50 con 0x7f8dd406d650 2024-02-19T15:22:14.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8dd1600640 1 -- 172.21.15.47:0/3847285423 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc80072f0 con 0x7f8dd406d650 2024-02-19T15:22:14.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8ddc600640 1 -- 172.21.15.47:0/3847285423 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd415dd40 con 0x7f8dd406d650 2024-02-19T15:22:14.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.830+0000 7f8ddc600640 1 -- 172.21.15.47:0/3847285423 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd415e300 con 0x7f8dd406d650 2024-02-19T15:22:14.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.834+0000 7f8ddc600640 1 -- 172.21.15.47:0/3847285423 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd4075360 con 0x7f8dd406d650 2024-02-19T15:22:14.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.834+0000 7f8dd1600640 1 -- 172.21.15.47:0/3847285423 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8dc8007450 con 0x7f8dd406d650 2024-02-19T15:22:14.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.834+0000 7f8dd1600640 1 --2- 172.21.15.47:0/3847285423 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8db4066e40 0x7f8db4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:14.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.834+0000 7f8dd1600640 1 -- 172.21.15.47:0/3847285423 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8dc8083ff0 con 0x7f8dd406d650 2024-02-19T15:22:14.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.834+0000 7f8dd9200640 1 --2- 172.21.15.47:0/3847285423 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8db4066e40 0x7f8db4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:14.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.834+0000 7f8dd9200640 1 --2- 172.21.15.47:0/3847285423 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8db4066e40 0x7f8db4069300 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0000f70 tx=0x7f8dc0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:14.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.838+0000 7f8dd1600640 1 -- 172.21.15.47:0/3847285423 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8dc80b61c0 con 0x7f8dd406d650 2024-02-19T15:22:15.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:14.998+0000 7f8ddc600640 1 -- 172.21.15.47:0/3847285423 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8dd416be00 con 0x7f8db4066e40 2024-02-19T15:22:15.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.006+0000 7f8dd1600640 1 -- 172.21.15.47:0/3847285423 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264599 (secure 0 0 0) 0x7f8dc00013b0 con 0x7f8db4066e40 2024-02-19T15:22:15.010 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:15.011 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:22:15.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.010+0000 7f8ddc600640 1 -- 172.21.15.47:0/3847285423 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8db4066e40 msgr2=0x7f8db4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.010+0000 7f8ddc600640 1 --2- 172.21.15.47:0/3847285423 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8db4066e40 0x7f8db4069300 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0000f70 tx=0x7f8dc0009040 comp rx=0 tx=0).stop 2024-02-19T15:22:15.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.010+0000 7f8ddc600640 1 -- 172.21.15.47:0/3847285423 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dd406d650 msgr2=0x7f8dd415c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.010+0000 7f8ddc600640 1 --2- 172.21.15.47:0/3847285423 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dd406d650 0x7f8dd415c7f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8dc800c3c0 tx=0x7f8dc800c880 comp rx=0 tx=0).stop 2024-02-19T15:22:15.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.010+0000 7f8ddc600640 1 -- 172.21.15.47:0/3847285423 shutdown_connections 2024-02-19T15:22:15.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.010+0000 7f8ddc600640 1 --2- 172.21.15.47:0/3847285423 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8db4066e40 0x7f8db4069300 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.010+0000 7f8ddc600640 1 --2- 172.21.15.47:0/3847285423 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd40ced10 0x7f8dd414c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.010+0000 7f8ddc600640 1 --2- 172.21.15.47:0/3847285423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dd40cded0 0x7f8dd41611c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.010+0000 7f8ddc600640 1 --2- 172.21.15.47:0/3847285423 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dd406d650 0x7f8dd415c7f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.010+0000 7f8ddc600640 1 -- 172.21.15.47:0/3847285423 >> 172.21.15.47:0/3847285423 conn(0x7f8dd40d1bf0 msgr2=0x7f8dd40b7320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:15.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.010+0000 7f8ddc600640 1 -- 172.21.15.47:0/3847285423 shutdown_connections 2024-02-19T15:22:15.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.010+0000 7f8ddc600640 1 -- 172.21.15.47:0/3847285423 wait complete. 2024-02-19T15:22:15.029 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-02-19T15:22:13.151480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235712,"kb_used_data":2888,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739008,"statfs":{"total":773094113280,"available":772852744192,"internally_reserved":0,"allocated":2957312,"data_stored":1776620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001247"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181542+0000","last_change":"2024-02-19T15:22:05.868047+0000","last_active":"2024-02-19T15:22:07.181542+0000","last_peered":"2024-02-19T15:22:07.181542+0000","last_clean":"2024-02-19T15:22:07.181542+0000","last_became_active":"2024-02-19T15:22:05.867538+0000","last_became_peered":"2024-02-19T15:22:05.867538+0000","last_unstale":"2024-02-19T15:22:07.181542+0000","last_undegraded":"2024-02-19T15:22:07.181542+0000","last_fullsized":"2024-02-19T15:22:07.181542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:47:52.212576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853548+0000","last_change":"2024-02-19T15:22:05.861729+0000","last_active":"2024-02-19T15:22:06.853548+0000","last_peered":"2024-02-19T15:22:06.853548+0000","last_clean":"2024-02-19T15:22:06.853548+0000","last_became_active":"2024-02-19T15:22:05.861455+0000","last_became_peered":"2024-02-19T15:22:05.861455+0000","last_unstale":"2024-02-19T15:22:06.853548+0000","last_undegraded":"2024-02-19T15:22:06.853548+0000","last_fullsized":"2024-02-19T15:22:06.853548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:43.283192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424456+0000","last_change":"2024-02-19T15:22:05.857715+0000","last_active":"2024-02-19T15:22:07.424456+0000","last_peered":"2024-02-19T15:22:07.424456+0000","last_clean":"2024-02-19T15:22:07.424456+0000","last_became_active":"2024-02-19T15:22:05.857360+0000","last_became_peered":"2024-02-19T15:22:05.857360+0000","last_unstale":"2024-02-19T15:22:07.424456+0000","last_undegraded":"2024-02-19T15:22:07.424456+0000","last_fullsized":"2024-02-19T15:22:07.424456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:35:11.566219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424408+0000","last_change":"2024-02-19T15:22:05.850537+0000","last_active":"2024-02-19T15:22:07.424408+0000","last_peered":"2024-02-19T15:22:07.424408+0000","last_clean":"2024-02-19T15:22:07.424408+0000","last_became_active":"2024-02-19T15:22:05.850288+0000","last_became_peered":"2024-02-19T15:22:05.850288+0000","last_unstale":"2024-02-19T15:22:07.424408+0000","last_undegraded":"2024-02-19T15:22:07.424408+0000","last_fullsized":"2024-02-19T15:22:07.424408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:20:55.654033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181346+0000","last_change":"2024-02-19T15:22:05.872936+0000","last_active":"2024-02-19T15:22:07.181346+0000","last_peered":"2024-02-19T15:22:07.181346+0000","last_clean":"2024-02-19T15:22:07.181346+0000","last_became_active":"2024-02-19T15:22:05.871969+0000","last_became_peered":"2024-02-19T15:22:05.871969+0000","last_unstale":"2024-02-19T15:22:07.181346+0000","last_undegraded":"2024-02-19T15:22:07.181346+0000","last_fullsized":"2024-02-19T15:22:07.181346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:14:10.262240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424104+0000","last_change":"2024-02-19T15:22:05.853731+0000","last_active":"2024-02-19T15:22:07.424104+0000","last_peered":"2024-02-19T15:22:07.424104+0000","last_clean":"2024-02-19T15:22:07.424104+0000","last_became_active":"2024-02-19T15:22:05.853423+0000","last_became_peered":"2024-02-19T15:22:05.853423+0000","last_unstale":"2024-02-19T15:22:07.424104+0000","last_undegraded":"2024-02-19T15:22:07.424104+0000","last_fullsized":"2024-02-19T15:22:07.424104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:14:40.204009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181151+0000","last_change":"2024-02-19T15:22:05.860498+0000","last_active":"2024-02-19T15:22:07.181151+0000","last_peered":"2024-02-19T15:22:07.181151+0000","last_clean":"2024-02-19T15:22:07.181151+0000","last_became_active":"2024-02-19T15:22:05.860283+0000","last_became_peered":"2024-02-19T15:22:05.860283+0000","last_unstale":"2024-02-19T15:22:07.181151+0000","last_undegraded":"2024-02-19T15:22:07.181151+0000","last_fullsized":"2024-02-19T15:22:07.181151+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:20:55.654033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851236+0000","last_change":"2024-02-19T15:22:05.860461+0000","last_active":"2024-02-19T15:22:06.851236+0000","last_peered":"2024-02-19T15:22:06.851236+0000","last_clean":"2024-02-19T15:22:06.851236+0000","last_became_active":"2024-02-19T15:22:05.860119+0000","last_became_peered":"2024-02-19T15:22:05.860119+0000","last_unstale":"2024-02-19T15:22:06.851236+0000","last_undegraded":"2024-02-19T15:22:06.851236+0000","last_fullsized":"2024-02-19T15:22:06.851236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:43:34.980001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181680+0000","last_change":"2024-02-19T15:22:05.863575+0000","last_active":"2024-02-19T15:22:07.181680+0000","last_peered":"2024-02-19T15:22:07.181680+0000","last_clean":"2024-02-19T15:22:07.181680+0000","last_became_active":"2024-02-19T15:22:05.863306+0000","last_became_peered":"2024-02-19T15:22:05.863306+0000","last_unstale":"2024-02-19T15:22:07.181680+0000","last_undegraded":"2024-02-19T15:22:07.181680+0000","last_fullsized":"2024-02-19T15:22:07.181680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:03:59.289702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854624+0000","last_change":"2024-02-19T15:22:05.867622+0000","last_active":"2024-02-19T15:22:06.854624+0000","last_peered":"2024-02-19T15:22:06.854624+0000","last_clean":"2024-02-19T15:22:06.854624+0000","last_became_active":"2024-02-19T15:22:05.867352+0000","last_became_peered":"2024-02-19T15:22:05.867352+0000","last_unstale":"2024-02-19T15:22:06.854624+0000","last_undegraded":"2024-02-19T15:22:06.854624+0000","last_fullsized":"2024-02-19T15:22:06.854624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:08.091823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854743+0000","last_change":"2024-02-19T15:22:05.859648+0000","last_active":"2024-02-19T15:22:06.854743+0000","last_peered":"2024-02-19T15:22:06.854743+0000","last_clean":"2024-02-19T15:22:06.854743+0000","last_became_active":"2024-02-19T15:22:05.859381+0000","last_became_peered":"2024-02-19T15:22:05.859381+0000","last_unstale":"2024-02-19T15:22:06.854743+0000","last_undegraded":"2024-02-19T15:22:06.854743+0000","last_fullsized":"2024-02-19T15:22:06.854743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:35:11.566219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424319+0000","last_change":"2024-02-19T15:22:05.869009+0000","last_active":"2024-02-19T15:22:07.424319+0000","last_peered":"2024-02-19T15:22:07.424319+0000","last_clean":"2024-02-19T15:22:07.424319+0000","last_became_active":"2024-02-19T15:22:05.868709+0000","last_became_peered":"2024-02-19T15:22:05.868709+0000","last_unstale":"2024-02-19T15:22:07.424319+0000","last_undegraded":"2024-02-19T15:22:07.424319+0000","last_fullsized":"2024-02-19T15:22:07.424319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:22.856662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851443+0000","last_change":"2024-02-19T15:22:05.848558+0000","last_active":"2024-02-19T15:22:06.851443+0000","last_peered":"2024-02-19T15:22:06.851443+0000","last_clean":"2024-02-19T15:22:06.851443+0000","last_became_active":"2024-02-19T15:22:05.848288+0000","last_became_peered":"2024-02-19T15:22:05.848288+0000","last_unstale":"2024-02-19T15:22:06.851443+0000","last_undegraded":"2024-02-19T15:22:06.851443+0000","last_fullsized":"2024-02-19T15:22:06.851443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:33:20.316638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424183+0000","last_change":"2024-02-19T15:22:05.846265+0000","last_active":"2024-02-19T15:22:07.424183+0000","last_peered":"2024-02-19T15:22:07.424183+0000","last_clean":"2024-02-19T15:22:07.424183+0000","last_became_active":"2024-02-19T15:22:05.841757+0000","last_became_peered":"2024-02-19T15:22:05.841757+0000","last_unstale":"2024-02-19T15:22:07.424183+0000","last_undegraded":"2024-02-19T15:22:07.424183+0000","last_fullsized":"2024-02-19T15:22:07.424183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:14:58.397614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853190+0000","last_change":"2024-02-19T15:22:05.850786+0000","last_active":"2024-02-19T15:22:06.853190+0000","last_peered":"2024-02-19T15:22:06.853190+0000","last_clean":"2024-02-19T15:22:06.853190+0000","last_became_active":"2024-02-19T15:22:05.850136+0000","last_became_peered":"2024-02-19T15:22:05.850136+0000","last_unstale":"2024-02-19T15:22:06.853190+0000","last_undegraded":"2024-02-19T15:22:06.853190+0000","last_fullsized":"2024-02-19T15:22:06.853190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:33.203920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.423964+0000","last_change":"2024-02-19T15:22:05.862076+0000","last_active":"2024-02-19T15:22:07.423964+0000","last_peered":"2024-02-19T15:22:07.423964+0000","last_clean":"2024-02-19T15:22:07.423964+0000","last_became_active":"2024-02-19T15:22:05.861826+0000","last_became_peered":"2024-02-19T15:22:05.861826+0000","last_unstale":"2024-02-19T15:22:07.423964+0000","last_undegraded":"2024-02-19T15:22:07.423964+0000","last_fullsized":"2024-02-19T15:22:07.423964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:43.283192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.425028+0000","last_change":"2024-02-19T15:22:05.869845+0000","last_active":"2024-02-19T15:22:07.425028+0000","last_peered":"2024-02-19T15:22:07.425028+0000","last_clean":"2024-02-19T15:22:07.425028+0000","last_became_active":"2024-02-19T15:22:05.865534+0000","last_became_peered":"2024-02-19T15:22:05.865534+0000","last_unstale":"2024-02-19T15:22:07.425028+0000","last_undegraded":"2024-02-19T15:22:07.425028+0000","last_fullsized":"2024-02-19T15:22:07.425028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:29:35.769837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.423831+0000","last_change":"2024-02-19T15:22:05.852133+0000","last_active":"2024-02-19T15:22:07.423831+0000","last_peered":"2024-02-19T15:22:07.423831+0000","last_clean":"2024-02-19T15:22:07.423831+0000","last_became_active":"2024-02-19T15:22:05.851823+0000","last_became_peered":"2024-02-19T15:22:05.851823+0000","last_unstale":"2024-02-19T15:22:07.423831+0000","last_undegraded":"2024-02-19T15:22:07.423831+0000","last_fullsized":"2024-02-19T15:22:07.423831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:38.607191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181615+0000","last_change":"2024-02-19T15:22:05.866370+0000","last_active":"2024-02-19T15:22:07.181615+0000","last_peered":"2024-02-19T15:22:07.181615+0000","last_clean":"2024-02-19T15:22:07.181615+0000","last_became_active":"2024-02-19T15:22:05.866059+0000","last_became_peered":"2024-02-19T15:22:05.866059+0000","last_unstale":"2024-02-19T15:22:07.181615+0000","last_undegraded":"2024-02-19T15:22:07.181615+0000","last_fullsized":"2024-02-19T15:22:07.181615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:31:00.196963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854536+0000","last_change":"2024-02-19T15:22:05.858350+0000","last_active":"2024-02-19T15:22:06.854536+0000","last_peered":"2024-02-19T15:22:06.854536+0000","last_clean":"2024-02-19T15:22:06.854536+0000","last_became_active":"2024-02-19T15:22:05.858055+0000","last_became_peered":"2024-02-19T15:22:05.858055+0000","last_unstale":"2024-02-19T15:22:06.854536+0000","last_undegraded":"2024-02-19T15:22:06.854536+0000","last_fullsized":"2024-02-19T15:22:06.854536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:53:17.517796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424367+0000","last_change":"2024-02-19T15:22:05.867437+0000","last_active":"2024-02-19T15:22:07.424367+0000","last_peered":"2024-02-19T15:22:07.424367+0000","last_clean":"2024-02-19T15:22:07.424367+0000","last_became_active":"2024-02-19T15:22:05.867104+0000","last_became_peered":"2024-02-19T15:22:05.867104+0000","last_unstale":"2024-02-19T15:22:07.424367+0000","last_undegraded":"2024-02-19T15:22:07.424367+0000","last_fullsized":"2024-02-19T15:22:07.424367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:53:17.517796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424285+0000","last_change":"2024-02-19T15:22:05.873978+0000","last_active":"2024-02-19T15:22:07.424285+0000","last_peered":"2024-02-19T15:22:07.424285+0000","last_clean":"2024-02-19T15:22:07.424285+0000","last_became_active":"2024-02-19T15:22:05.870210+0000","last_became_peered":"2024-02-19T15:22:05.870210+0000","last_unstale":"2024-02-19T15:22:07.424285+0000","last_undegraded":"2024-02-19T15:22:07.424285+0000","last_fullsized":"2024-02-19T15:22:07.424285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:46:15.823647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851320+0000","last_change":"2024-02-19T15:22:05.869630+0000","last_active":"2024-02-19T15:22:06.851320+0000","last_peered":"2024-02-19T15:22:06.851320+0000","last_clean":"2024-02-19T15:22:06.851320+0000","last_became_active":"2024-02-19T15:22:05.869363+0000","last_became_peered":"2024-02-19T15:22:05.869363+0000","last_unstale":"2024-02-19T15:22:06.851320+0000","last_undegraded":"2024-02-19T15:22:06.851320+0000","last_fullsized":"2024-02-19T15:22:06.851320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:33.203920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424119+0000","last_change":"2024-02-19T15:22:05.846649+0000","last_active":"2024-02-19T15:22:07.424119+0000","last_peered":"2024-02-19T15:22:07.424119+0000","last_clean":"2024-02-19T15:22:07.424119+0000","last_became_active":"2024-02-19T15:22:05.846335+0000","last_became_peered":"2024-02-19T15:22:05.846335+0000","last_unstale":"2024-02-19T15:22:07.424119+0000","last_undegraded":"2024-02-19T15:22:07.424119+0000","last_fullsized":"2024-02-19T15:22:07.424119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:03:40.323060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851859+0000","last_change":"2024-02-19T15:22:05.870492+0000","last_active":"2024-02-19T15:22:06.851859+0000","last_peered":"2024-02-19T15:22:06.851859+0000","last_clean":"2024-02-19T15:22:06.851859+0000","last_became_active":"2024-02-19T15:22:05.868145+0000","last_became_peered":"2024-02-19T15:22:05.868145+0000","last_unstale":"2024-02-19T15:22:06.851859+0000","last_undegraded":"2024-02-19T15:22:06.851859+0000","last_fullsized":"2024-02-19T15:22:06.851859+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:03:40.323060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854818+0000","last_change":"2024-02-19T15:22:05.869349+0000","last_active":"2024-02-19T15:22:06.854818+0000","last_peered":"2024-02-19T15:22:06.854818+0000","last_clean":"2024-02-19T15:22:06.854818+0000","last_became_active":"2024-02-19T15:22:05.869042+0000","last_became_peered":"2024-02-19T15:22:05.869042+0000","last_unstale":"2024-02-19T15:22:06.854818+0000","last_undegraded":"2024-02-19T15:22:06.854818+0000","last_fullsized":"2024-02-19T15:22:06.854818+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:14:16.136182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853505+0000","last_change":"2024-02-19T15:22:05.856951+0000","last_active":"2024-02-19T15:22:06.853505+0000","last_peered":"2024-02-19T15:22:06.853505+0000","last_clean":"2024-02-19T15:22:06.853505+0000","last_became_active":"2024-02-19T15:22:05.856670+0000","last_became_peered":"2024-02-19T15:22:05.856670+0000","last_unstale":"2024-02-19T15:22:06.853505+0000","last_undegraded":"2024-02-19T15:22:06.853505+0000","last_fullsized":"2024-02-19T15:22:06.853505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:48:05.400374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851719+0000","last_change":"2024-02-19T15:22:05.868491+0000","last_active":"2024-02-19T15:22:06.851719+0000","last_peered":"2024-02-19T15:22:06.851719+0000","last_clean":"2024-02-19T15:22:06.851719+0000","last_became_active":"2024-02-19T15:22:05.868184+0000","last_became_peered":"2024-02-19T15:22:05.868184+0000","last_unstale":"2024-02-19T15:22:06.851719+0000","last_undegraded":"2024-02-19T15:22:06.851719+0000","last_fullsized":"2024-02-19T15:22:06.851719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:14:40.204009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424406+0000","last_change":"2024-02-19T15:22:05.847847+0000","last_active":"2024-02-19T15:22:07.424406+0000","last_peered":"2024-02-19T15:22:07.424406+0000","last_clean":"2024-02-19T15:22:07.424406+0000","last_became_active":"2024-02-19T15:22:05.847538+0000","last_became_peered":"2024-02-19T15:22:05.847538+0000","last_unstale":"2024-02-19T15:22:07.424406+0000","last_undegraded":"2024-02-19T15:22:07.424406+0000","last_fullsized":"2024-02-19T15:22:07.424406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:08.091823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424277+0000","last_change":"2024-02-19T15:22:05.847222+0000","last_active":"2024-02-19T15:22:07.424277+0000","last_peered":"2024-02-19T15:22:07.424277+0000","last_clean":"2024-02-19T15:22:07.424277+0000","last_became_active":"2024-02-19T15:22:05.846957+0000","last_became_peered":"2024-02-19T15:22:05.846957+0000","last_unstale":"2024-02-19T15:22:07.424277+0000","last_undegraded":"2024-02-19T15:22:07.424277+0000","last_fullsized":"2024-02-19T15:22:07.424277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:08:56.380125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.423913+0000","last_change":"2024-02-19T15:22:05.862263+0000","last_active":"2024-02-19T15:22:07.423913+0000","last_peered":"2024-02-19T15:22:07.423913+0000","last_clean":"2024-02-19T15:22:07.423913+0000","last_became_active":"2024-02-19T15:22:05.861932+0000","last_became_peered":"2024-02-19T15:22:05.861932+0000","last_unstale":"2024-02-19T15:22:07.423913+0000","last_undegraded":"2024-02-19T15:22:07.423913+0000","last_fullsized":"2024-02-19T15:22:07.423913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:37:51.996719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854810+0000","last_change":"2024-02-19T15:22:05.852998+0000","last_active":"2024-02-19T15:22:06.854810+0000","last_peered":"2024-02-19T15:22:06.854810+0000","last_clean":"2024-02-19T15:22:06.854810+0000","last_became_active":"2024-02-19T15:22:05.852660+0000","last_became_peered":"2024-02-19T15:22:05.852660+0000","last_unstale":"2024-02-19T15:22:06.854810+0000","last_undegraded":"2024-02-19T15:22:06.854810+0000","last_fullsized":"2024-02-19T15:22:06.854810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:54:25.604351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.423850+0000","last_change":"2024-02-19T15:22:05.862188+0000","last_active":"2024-02-19T15:22:07.423850+0000","last_peered":"2024-02-19T15:22:07.423850+0000","last_clean":"2024-02-19T15:22:07.423850+0000","last_became_active":"2024-02-19T15:22:05.861789+0000","last_became_peered":"2024-02-19T15:22:05.861789+0000","last_unstale":"2024-02-19T15:22:07.423850+0000","last_undegraded":"2024-02-19T15:22:07.423850+0000","last_fullsized":"2024-02-19T15:22:07.423850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:41.785448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851552+0000","last_change":"2024-02-19T15:22:04.848427+0000","last_active":"2024-02-19T15:22:06.851552+0000","last_peered":"2024-02-19T15:22:06.851552+0000","last_clean":"2024-02-19T15:22:06.851552+0000","last_became_active":"2024-02-19T15:22:04.848111+0000","last_became_peered":"2024-02-19T15:22:04.848111+0000","last_unstale":"2024-02-19T15:22:06.851552+0000","last_undegraded":"2024-02-19T15:22:06.851552+0000","last_fullsized":"2024-02-19T15:22:06.851552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:17.404136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424555+0000","last_change":"2024-02-19T15:22:05.845098+0000","last_active":"2024-02-19T15:22:07.424555+0000","last_peered":"2024-02-19T15:22:07.424555+0000","last_clean":"2024-02-19T15:22:07.424555+0000","last_became_active":"2024-02-19T15:22:05.844805+0000","last_became_peered":"2024-02-19T15:22:05.844805+0000","last_unstale":"2024-02-19T15:22:07.424555+0000","last_undegraded":"2024-02-19T15:22:07.424555+0000","last_fullsized":"2024-02-19T15:22:07.424555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:14:16.136182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854248+0000","last_change":"2024-02-19T15:22:04.842285+0000","last_active":"2024-02-19T15:22:06.854248+0000","last_peered":"2024-02-19T15:22:06.854248+0000","last_clean":"2024-02-19T15:22:06.854248+0000","last_became_active":"2024-02-19T15:22:04.840126+0000","last_became_peered":"2024-02-19T15:22:04.840126+0000","last_unstale":"2024-02-19T15:22:06.854248+0000","last_undegraded":"2024-02-19T15:22:06.854248+0000","last_fullsized":"2024-02-19T15:22:06.854248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:11:04.017098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851307+0000","last_change":"2024-02-19T15:22:04.839208+0000","last_active":"2024-02-19T15:22:06.851307+0000","last_peered":"2024-02-19T15:22:06.851307+0000","last_clean":"2024-02-19T15:22:06.851307+0000","last_became_active":"2024-02-19T15:22:04.838857+0000","last_became_peered":"2024-02-19T15:22:04.838857+0000","last_unstale":"2024-02-19T15:22:06.851307+0000","last_undegraded":"2024-02-19T15:22:06.851307+0000","last_fullsized":"2024-02-19T15:22:06.851307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:52:37.920564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851309+0000","last_change":"2024-02-19T15:22:05.859786+0000","last_active":"2024-02-19T15:22:06.851309+0000","last_peered":"2024-02-19T15:22:06.851309+0000","last_clean":"2024-02-19T15:22:06.851309+0000","last_became_active":"2024-02-19T15:22:05.854215+0000","last_became_peered":"2024-02-19T15:22:05.854215+0000","last_unstale":"2024-02-19T15:22:06.851309+0000","last_undegraded":"2024-02-19T15:22:06.851309+0000","last_fullsized":"2024-02-19T15:22:06.851309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:41.123853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424205+0000","last_change":"2024-02-19T15:22:04.843918+0000","last_active":"2024-02-19T15:22:07.424205+0000","last_peered":"2024-02-19T15:22:07.424205+0000","last_clean":"2024-02-19T15:22:07.424205+0000","last_became_active":"2024-02-19T15:22:04.842002+0000","last_became_peered":"2024-02-19T15:22:04.842002+0000","last_unstale":"2024-02-19T15:22:07.424205+0000","last_undegraded":"2024-02-19T15:22:07.424205+0000","last_fullsized":"2024-02-19T15:22:07.424205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:27.325777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853182+0000","last_change":"2024-02-19T15:22:05.852203+0000","last_active":"2024-02-19T15:22:06.853182+0000","last_peered":"2024-02-19T15:22:06.853182+0000","last_clean":"2024-02-19T15:22:06.853182+0000","last_became_active":"2024-02-19T15:22:05.850442+0000","last_became_peered":"2024-02-19T15:22:05.850442+0000","last_unstale":"2024-02-19T15:22:06.853182+0000","last_undegraded":"2024-02-19T15:22:06.853182+0000","last_fullsized":"2024-02-19T15:22:06.853182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:41.123853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854530+0000","last_change":"2024-02-19T15:22:04.839817+0000","last_active":"2024-02-19T15:22:06.854530+0000","last_peered":"2024-02-19T15:22:06.854530+0000","last_clean":"2024-02-19T15:22:06.854530+0000","last_became_active":"2024-02-19T15:22:04.839034+0000","last_became_peered":"2024-02-19T15:22:04.839034+0000","last_unstale":"2024-02-19T15:22:06.854530+0000","last_undegraded":"2024-02-19T15:22:06.854530+0000","last_fullsized":"2024-02-19T15:22:06.854530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:16:21.476088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424743+0000","last_change":"2024-02-19T15:22:05.869563+0000","last_active":"2024-02-19T15:22:07.424743+0000","last_peered":"2024-02-19T15:22:07.424743+0000","last_clean":"2024-02-19T15:22:07.424743+0000","last_became_active":"2024-02-19T15:22:05.869260+0000","last_became_peered":"2024-02-19T15:22:05.869260+0000","last_unstale":"2024-02-19T15:22:07.424743+0000","last_undegraded":"2024-02-19T15:22:07.424743+0000","last_fullsized":"2024-02-19T15:22:07.424743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:26:42.865617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424823+0000","last_change":"2024-02-19T15:22:04.843586+0000","last_active":"2024-02-19T15:22:07.424823+0000","last_peered":"2024-02-19T15:22:07.424823+0000","last_clean":"2024-02-19T15:22:07.424823+0000","last_became_active":"2024-02-19T15:22:04.842983+0000","last_became_peered":"2024-02-19T15:22:04.842983+0000","last_unstale":"2024-02-19T15:22:07.424823+0000","last_undegraded":"2024-02-19T15:22:07.424823+0000","last_fullsized":"2024-02-19T15:22:07.424823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:21.851058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424603+0000","last_change":"2024-02-19T15:22:05.861506+0000","last_active":"2024-02-19T15:22:07.424603+0000","last_peered":"2024-02-19T15:22:07.424603+0000","last_clean":"2024-02-19T15:22:07.424603+0000","last_became_active":"2024-02-19T15:22:05.861172+0000","last_became_peered":"2024-02-19T15:22:05.861172+0000","last_unstale":"2024-02-19T15:22:07.424603+0000","last_undegraded":"2024-02-19T15:22:07.424603+0000","last_fullsized":"2024-02-19T15:22:07.424603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:26:48.255672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853456+0000","last_change":"2024-02-19T15:22:04.833984+0000","last_active":"2024-02-19T15:22:06.853456+0000","last_peered":"2024-02-19T15:22:06.853456+0000","last_clean":"2024-02-19T15:22:06.853456+0000","last_became_active":"2024-02-19T15:22:04.833418+0000","last_became_peered":"2024-02-19T15:22:04.833418+0000","last_unstale":"2024-02-19T15:22:06.853456+0000","last_undegraded":"2024-02-19T15:22:06.853456+0000","last_fullsized":"2024-02-19T15:22:06.853456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:31:17.425502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424400+0000","last_change":"2024-02-19T15:22:05.865117+0000","last_active":"2024-02-19T15:22:07.424400+0000","last_peered":"2024-02-19T15:22:07.424400+0000","last_clean":"2024-02-19T15:22:07.424400+0000","last_became_active":"2024-02-19T15:22:05.864502+0000","last_became_peered":"2024-02-19T15:22:05.864502+0000","last_unstale":"2024-02-19T15:22:07.424400+0000","last_undegraded":"2024-02-19T15:22:07.424400+0000","last_fullsized":"2024-02-19T15:22:07.424400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:38.607191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854661+0000","last_change":"2024-02-19T15:22:04.832285+0000","last_active":"2024-02-19T15:22:06.854661+0000","last_peered":"2024-02-19T15:22:06.854661+0000","last_clean":"2024-02-19T15:22:06.854661+0000","last_became_active":"2024-02-19T15:22:04.831992+0000","last_became_peered":"2024-02-19T15:22:04.831992+0000","last_unstale":"2024-02-19T15:22:06.854661+0000","last_undegraded":"2024-02-19T15:22:06.854661+0000","last_fullsized":"2024-02-19T15:22:06.854661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:50:48.637629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181470+0000","last_change":"2024-02-19T15:22:05.874836+0000","last_active":"2024-02-19T15:22:07.181470+0000","last_peered":"2024-02-19T15:22:07.181470+0000","last_clean":"2024-02-19T15:22:07.181470+0000","last_became_active":"2024-02-19T15:22:05.874522+0000","last_became_peered":"2024-02-19T15:22:05.874522+0000","last_unstale":"2024-02-19T15:22:07.181470+0000","last_undegraded":"2024-02-19T15:22:07.181470+0000","last_fullsized":"2024-02-19T15:22:07.181470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:26:42.865617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424505+0000","last_change":"2024-02-19T15:22:04.837502+0000","last_active":"2024-02-19T15:22:07.424505+0000","last_peered":"2024-02-19T15:22:07.424505+0000","last_clean":"2024-02-19T15:22:07.424505+0000","last_became_active":"2024-02-19T15:22:04.836913+0000","last_became_peered":"2024-02-19T15:22:04.836913+0000","last_unstale":"2024-02-19T15:22:07.424505+0000","last_undegraded":"2024-02-19T15:22:07.424505+0000","last_fullsized":"2024-02-19T15:22:07.424505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:20:00.631607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424276+0000","last_change":"2024-02-19T15:22:05.864869+0000","last_active":"2024-02-19T15:22:07.424276+0000","last_peered":"2024-02-19T15:22:07.424276+0000","last_clean":"2024-02-19T15:22:07.424276+0000","last_became_active":"2024-02-19T15:22:05.864424+0000","last_became_peered":"2024-02-19T15:22:05.864424+0000","last_unstale":"2024-02-19T15:22:07.424276+0000","last_undegraded":"2024-02-19T15:22:07.424276+0000","last_fullsized":"2024-02-19T15:22:07.424276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:22:17.882758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424358+0000","last_change":"2024-02-19T15:22:04.847248+0000","last_active":"2024-02-19T15:22:07.424358+0000","last_peered":"2024-02-19T15:22:07.424358+0000","last_clean":"2024-02-19T15:22:07.424358+0000","last_became_active":"2024-02-19T15:22:04.846883+0000","last_became_peered":"2024-02-19T15:22:04.846883+0000","last_unstale":"2024-02-19T15:22:07.424358+0000","last_undegraded":"2024-02-19T15:22:07.424358+0000","last_fullsized":"2024-02-19T15:22:07.424358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:40.779845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181361+0000","last_change":"2024-02-19T15:22:05.873228+0000","last_active":"2024-02-19T15:22:07.181361+0000","last_peered":"2024-02-19T15:22:07.181361+0000","last_clean":"2024-02-19T15:22:07.181361+0000","last_became_active":"2024-02-19T15:22:05.872458+0000","last_became_peered":"2024-02-19T15:22:05.872458+0000","last_unstale":"2024-02-19T15:22:07.181361+0000","last_undegraded":"2024-02-19T15:22:07.181361+0000","last_fullsized":"2024-02-19T15:22:07.181361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:59:01.655185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854298+0000","last_change":"2024-02-19T15:22:04.843476+0000","last_active":"2024-02-19T15:22:06.854298+0000","last_peered":"2024-02-19T15:22:06.854298+0000","last_clean":"2024-02-19T15:22:06.854298+0000","last_became_active":"2024-02-19T15:22:04.843159+0000","last_became_peered":"2024-02-19T15:22:04.843159+0000","last_unstale":"2024-02-19T15:22:06.854298+0000","last_undegraded":"2024-02-19T15:22:06.854298+0000","last_fullsized":"2024-02-19T15:22:06.854298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:21:58.456669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854827+0000","last_change":"2024-02-19T15:22:05.852049+0000","last_active":"2024-02-19T15:22:06.854827+0000","last_peered":"2024-02-19T15:22:06.854827+0000","last_clean":"2024-02-19T15:22:06.854827+0000","last_became_active":"2024-02-19T15:22:05.851786+0000","last_became_peered":"2024-02-19T15:22:05.851786+0000","last_unstale":"2024-02-19T15:22:06.854827+0000","last_undegraded":"2024-02-19T15:22:06.854827+0000","last_fullsized":"2024-02-19T15:22:06.854827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:31:59.866248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.423650+0000","last_change":"2024-02-19T15:22:04.844209+0000","last_active":"2024-02-19T15:22:07.423650+0000","last_peered":"2024-02-19T15:22:07.423650+0000","last_clean":"2024-02-19T15:22:07.423650+0000","last_became_active":"2024-02-19T15:22:04.842885+0000","last_became_peered":"2024-02-19T15:22:04.842885+0000","last_unstale":"2024-02-19T15:22:07.423650+0000","last_undegraded":"2024-02-19T15:22:07.423650+0000","last_fullsized":"2024-02-19T15:22:07.423650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:27.325777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181171+0000","last_change":"2024-02-19T15:22:05.877219+0000","last_active":"2024-02-19T15:22:07.181171+0000","last_peered":"2024-02-19T15:22:07.181171+0000","last_clean":"2024-02-19T15:22:07.181171+0000","last_became_active":"2024-02-19T15:22:05.876953+0000","last_became_peered":"2024-02-19T15:22:05.876953+0000","last_unstale":"2024-02-19T15:22:07.181171+0000","last_undegraded":"2024-02-19T15:22:07.181171+0000","last_fullsized":"2024-02-19T15:22:07.181171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:55:27.572569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424947+0000","last_change":"2024-02-19T15:22:04.837084+0000","last_active":"2024-02-19T15:22:07.424947+0000","last_peered":"2024-02-19T15:22:07.424947+0000","last_clean":"2024-02-19T15:22:07.424947+0000","last_became_active":"2024-02-19T15:22:04.836783+0000","last_became_peered":"2024-02-19T15:22:04.836783+0000","last_unstale":"2024-02-19T15:22:07.424947+0000","last_undegraded":"2024-02-19T15:22:07.424947+0000","last_fullsized":"2024-02-19T15:22:07.424947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:34:14.847696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181498+0000","last_change":"2024-02-19T15:22:05.876111+0000","last_active":"2024-02-19T15:22:07.181498+0000","last_peered":"2024-02-19T15:22:07.181498+0000","last_clean":"2024-02-19T15:22:07.181498+0000","last_became_active":"2024-02-19T15:22:05.875843+0000","last_became_peered":"2024-02-19T15:22:05.875843+0000","last_unstale":"2024-02-19T15:22:07.181498+0000","last_undegraded":"2024-02-19T15:22:07.181498+0000","last_fullsized":"2024-02-19T15:22:07.181498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:29:35.769837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.423944+0000","last_change":"2024-02-19T15:22:04.839982+0000","last_active":"2024-02-19T15:22:07.423944+0000","last_peered":"2024-02-19T15:22:07.423944+0000","last_clean":"2024-02-19T15:22:07.423944+0000","last_became_active":"2024-02-19T15:22:04.839617+0000","last_became_peered":"2024-02-19T15:22:04.839617+0000","last_unstale":"2024-02-19T15:22:07.423944+0000","last_undegraded":"2024-02-19T15:22:07.423944+0000","last_fullsized":"2024-02-19T15:22:07.423944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:21.137278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181744+0000","last_change":"2024-02-19T15:22:05.870359+0000","last_active":"2024-02-19T15:22:07.181744+0000","last_peered":"2024-02-19T15:22:07.181744+0000","last_clean":"2024-02-19T15:22:07.181744+0000","last_became_active":"2024-02-19T15:22:05.870083+0000","last_became_peered":"2024-02-19T15:22:05.870083+0000","last_unstale":"2024-02-19T15:22:07.181744+0000","last_undegraded":"2024-02-19T15:22:07.181744+0000","last_fullsized":"2024-02-19T15:22:07.181744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:41:29.186489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853581+0000","last_change":"2024-02-19T15:22:04.828045+0000","last_active":"2024-02-19T15:22:06.853581+0000","last_peered":"2024-02-19T15:22:06.853581+0000","last_clean":"2024-02-19T15:22:06.853581+0000","last_became_active":"2024-02-19T15:22:04.827744+0000","last_became_peered":"2024-02-19T15:22:04.827744+0000","last_unstale":"2024-02-19T15:22:06.853581+0000","last_undegraded":"2024-02-19T15:22:06.853581+0000","last_fullsized":"2024-02-19T15:22:06.853581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:21.851058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424587+0000","last_change":"2024-02-19T15:22:05.849002+0000","last_active":"2024-02-19T15:22:07.424587+0000","last_peered":"2024-02-19T15:22:07.424587+0000","last_clean":"2024-02-19T15:22:07.424587+0000","last_became_active":"2024-02-19T15:22:05.848692+0000","last_became_peered":"2024-02-19T15:22:05.848692+0000","last_unstale":"2024-02-19T15:22:07.424587+0000","last_undegraded":"2024-02-19T15:22:07.424587+0000","last_fullsized":"2024-02-19T15:22:07.424587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:14:10.262240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.423816+0000","last_change":"2024-02-19T15:22:04.851700+0000","last_active":"2024-02-19T15:22:07.423816+0000","last_peered":"2024-02-19T15:22:07.423816+0000","last_clean":"2024-02-19T15:22:07.423816+0000","last_became_active":"2024-02-19T15:22:04.851270+0000","last_became_peered":"2024-02-19T15:22:04.851270+0000","last_unstale":"2024-02-19T15:22:07.423816+0000","last_undegraded":"2024-02-19T15:22:07.423816+0000","last_fullsized":"2024-02-19T15:22:07.423816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:40.779845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181370+0000","last_change":"2024-02-19T15:22:05.862153+0000","last_active":"2024-02-19T15:22:07.181370+0000","last_peered":"2024-02-19T15:22:07.181370+0000","last_clean":"2024-02-19T15:22:07.181370+0000","last_became_active":"2024-02-19T15:22:05.861836+0000","last_became_peered":"2024-02-19T15:22:05.861836+0000","last_unstale":"2024-02-19T15:22:07.181370+0000","last_undegraded":"2024-02-19T15:22:07.181370+0000","last_fullsized":"2024-02-19T15:22:07.181370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:07:38.614353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181103+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:22:07.181103+0000","last_peered":"2024-02-19T15:22:07.181103+0000","last_clean":"2024-02-19T15:22:07.181103+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:22:07.181103+0000","last_undegraded":"2024-02-19T15:22:07.181103+0000","last_fullsized":"2024-02-19T15:22:07.181103+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424087+0000","last_change":"2024-02-19T15:22:04.846635+0000","last_active":"2024-02-19T15:22:07.424087+0000","last_peered":"2024-02-19T15:22:07.424087+0000","last_clean":"2024-02-19T15:22:07.424087+0000","last_became_active":"2024-02-19T15:22:04.845191+0000","last_became_peered":"2024-02-19T15:22:04.845191+0000","last_unstale":"2024-02-19T15:22:07.424087+0000","last_undegraded":"2024-02-19T15:22:07.424087+0000","last_fullsized":"2024-02-19T15:22:07.424087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:31:17.425502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851571+0000","last_change":"2024-02-19T15:22:05.863647+0000","last_active":"2024-02-19T15:22:06.851571+0000","last_peered":"2024-02-19T15:22:06.851571+0000","last_clean":"2024-02-19T15:22:06.851571+0000","last_became_active":"2024-02-19T15:22:05.863370+0000","last_became_peered":"2024-02-19T15:22:05.863370+0000","last_unstale":"2024-02-19T15:22:06.851571+0000","last_undegraded":"2024-02-19T15:22:06.851571+0000","last_fullsized":"2024-02-19T15:22:06.851571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:48:05.400374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181034+0000","last_change":"2024-02-19T15:22:04.849295+0000","last_active":"2024-02-19T15:22:07.181034+0000","last_peered":"2024-02-19T15:22:07.181034+0000","last_clean":"2024-02-19T15:22:07.181034+0000","last_became_active":"2024-02-19T15:22:04.849004+0000","last_became_peered":"2024-02-19T15:22:04.849004+0000","last_unstale":"2024-02-19T15:22:07.181034+0000","last_undegraded":"2024-02-19T15:22:07.181034+0000","last_fullsized":"2024-02-19T15:22:07.181034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:37.601588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424163+0000","last_change":"2024-02-19T15:22:05.861044+0000","last_active":"2024-02-19T15:22:07.424163+0000","last_peered":"2024-02-19T15:22:07.424163+0000","last_clean":"2024-02-19T15:22:07.424163+0000","last_became_active":"2024-02-19T15:22:05.860784+0000","last_became_peered":"2024-02-19T15:22:05.860784+0000","last_unstale":"2024-02-19T15:22:07.424163+0000","last_undegraded":"2024-02-19T15:22:07.424163+0000","last_fullsized":"2024-02-19T15:22:07.424163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:16:26.878453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.423856+0000","last_change":"2024-02-19T15:22:04.842486+0000","last_active":"2024-02-19T15:22:07.423856+0000","last_peered":"2024-02-19T15:22:07.423856+0000","last_clean":"2024-02-19T15:22:07.423856+0000","last_became_active":"2024-02-19T15:22:04.840062+0000","last_became_peered":"2024-02-19T15:22:04.840062+0000","last_unstale":"2024-02-19T15:22:07.423856+0000","last_undegraded":"2024-02-19T15:22:07.423856+0000","last_fullsized":"2024-02-19T15:22:07.423856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:17.404136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853201+0000","last_change":"2024-02-19T15:22:05.853794+0000","last_active":"2024-02-19T15:22:06.853201+0000","last_peered":"2024-02-19T15:22:06.853201+0000","last_clean":"2024-02-19T15:22:06.853201+0000","last_became_active":"2024-02-19T15:22:05.851972+0000","last_became_peered":"2024-02-19T15:22:05.851972+0000","last_unstale":"2024-02-19T15:22:06.853201+0000","last_undegraded":"2024-02-19T15:22:06.853201+0000","last_fullsized":"2024-02-19T15:22:06.853201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:33:20.316638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424384+0000","last_change":"2024-02-19T15:22:04.851077+0000","last_active":"2024-02-19T15:22:07.424384+0000","last_peered":"2024-02-19T15:22:07.424384+0000","last_clean":"2024-02-19T15:22:07.424384+0000","last_became_active":"2024-02-19T15:22:04.850734+0000","last_became_peered":"2024-02-19T15:22:04.850734+0000","last_unstale":"2024-02-19T15:22:07.424384+0000","last_undegraded":"2024-02-19T15:22:07.424384+0000","last_fullsized":"2024-02-19T15:22:07.424384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:37.601588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424264+0000","last_change":"2024-02-19T15:22:05.859004+0000","last_active":"2024-02-19T15:22:07.424264+0000","last_peered":"2024-02-19T15:22:07.424264+0000","last_clean":"2024-02-19T15:22:07.424264+0000","last_became_active":"2024-02-19T15:22:05.858524+0000","last_became_peered":"2024-02-19T15:22:05.858524+0000","last_unstale":"2024-02-19T15:22:07.424264+0000","last_undegraded":"2024-02-19T15:22:07.424264+0000","last_fullsized":"2024-02-19T15:22:07.424264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:31:18.431106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853431+0000","last_change":"2024-02-19T15:22:04.832556+0000","last_active":"2024-02-19T15:22:06.853431+0000","last_peered":"2024-02-19T15:22:06.853431+0000","last_clean":"2024-02-19T15:22:06.853431+0000","last_became_active":"2024-02-19T15:22:04.832261+0000","last_became_peered":"2024-02-19T15:22:04.832261+0000","last_unstale":"2024-02-19T15:22:06.853431+0000","last_undegraded":"2024-02-19T15:22:06.853431+0000","last_fullsized":"2024-02-19T15:22:06.853431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:37.601588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424743+0000","last_change":"2024-02-19T15:22:05.850119+0000","last_active":"2024-02-19T15:22:07.424743+0000","last_peered":"2024-02-19T15:22:07.424743+0000","last_clean":"2024-02-19T15:22:07.424743+0000","last_became_active":"2024-02-19T15:22:05.849831+0000","last_became_peered":"2024-02-19T15:22:05.849831+0000","last_unstale":"2024-02-19T15:22:07.424743+0000","last_undegraded":"2024-02-19T15:22:07.424743+0000","last_fullsized":"2024-02-19T15:22:07.424743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:07:38.614353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181613+0000","last_change":"2024-02-19T15:22:04.850106+0000","last_active":"2024-02-19T15:22:07.181613+0000","last_peered":"2024-02-19T15:22:07.181613+0000","last_clean":"2024-02-19T15:22:07.181613+0000","last_became_active":"2024-02-19T15:22:04.849467+0000","last_became_peered":"2024-02-19T15:22:04.849467+0000","last_unstale":"2024-02-19T15:22:07.181613+0000","last_undegraded":"2024-02-19T15:22:07.181613+0000","last_fullsized":"2024-02-19T15:22:07.181613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:32:55.903921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853330+0000","last_change":"2024-02-19T15:22:05.857834+0000","last_active":"2024-02-19T15:22:06.853330+0000","last_peered":"2024-02-19T15:22:06.853330+0000","last_clean":"2024-02-19T15:22:06.853330+0000","last_became_active":"2024-02-19T15:22:05.857488+0000","last_became_peered":"2024-02-19T15:22:05.857488+0000","last_unstale":"2024-02-19T15:22:06.853330+0000","last_undegraded":"2024-02-19T15:22:06.853330+0000","last_fullsized":"2024-02-19T15:22:06.853330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:21:49.261015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853569+0000","last_change":"2024-02-19T15:22:04.830568+0000","last_active":"2024-02-19T15:22:06.853569+0000","last_peered":"2024-02-19T15:22:06.853569+0000","last_clean":"2024-02-19T15:22:06.853569+0000","last_became_active":"2024-02-19T15:22:04.829131+0000","last_became_peered":"2024-02-19T15:22:04.829131+0000","last_unstale":"2024-02-19T15:22:06.853569+0000","last_undegraded":"2024-02-19T15:22:06.853569+0000","last_fullsized":"2024-02-19T15:22:06.853569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:37:50.991116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.852021+0000","last_change":"2024-02-19T15:22:05.866131+0000","last_active":"2024-02-19T15:22:06.852021+0000","last_peered":"2024-02-19T15:22:06.852021+0000","last_clean":"2024-02-19T15:22:06.852021+0000","last_became_active":"2024-02-19T15:22:05.865820+0000","last_became_peered":"2024-02-19T15:22:05.865820+0000","last_unstale":"2024-02-19T15:22:06.852021+0000","last_undegraded":"2024-02-19T15:22:06.852021+0000","last_fullsized":"2024-02-19T15:22:06.852021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:46:15.823647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854644+0000","last_change":"2024-02-19T15:22:04.840658+0000","last_active":"2024-02-19T15:22:06.854644+0000","last_peered":"2024-02-19T15:22:06.854644+0000","last_clean":"2024-02-19T15:22:06.854644+0000","last_became_active":"2024-02-19T15:22:04.840167+0000","last_became_peered":"2024-02-19T15:22:04.840167+0000","last_unstale":"2024-02-19T15:22:06.854644+0000","last_undegraded":"2024-02-19T15:22:06.854644+0000","last_fullsized":"2024-02-19T15:22:06.854644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:26:19.726455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424579+0000","last_change":"2024-02-19T15:22:05.854827+0000","last_active":"2024-02-19T15:22:07.424579+0000","last_peered":"2024-02-19T15:22:07.424579+0000","last_clean":"2024-02-19T15:22:07.424579+0000","last_became_active":"2024-02-19T15:22:05.854579+0000","last_became_peered":"2024-02-19T15:22:05.854579+0000","last_unstale":"2024-02-19T15:22:07.424579+0000","last_undegraded":"2024-02-19T15:22:07.424579+0000","last_fullsized":"2024-02-19T15:22:07.424579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:31:59.866248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181002+0000","last_change":"2024-02-19T15:22:04.848481+0000","last_active":"2024-02-19T15:22:07.181002+0000","last_peered":"2024-02-19T15:22:07.181002+0000","last_clean":"2024-02-19T15:22:07.181002+0000","last_became_active":"2024-02-19T15:22:04.848224+0000","last_became_peered":"2024-02-19T15:22:04.848224+0000","last_unstale":"2024-02-19T15:22:07.181002+0000","last_undegraded":"2024-02-19T15:22:07.181002+0000","last_fullsized":"2024-02-19T15:22:07.181002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:21.851058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424492+0000","last_change":"2024-02-19T15:22:05.868543+0000","last_active":"2024-02-19T15:22:07.424492+0000","last_peered":"2024-02-19T15:22:07.424492+0000","last_clean":"2024-02-19T15:22:07.424492+0000","last_became_active":"2024-02-19T15:22:05.868307+0000","last_became_peered":"2024-02-19T15:22:05.868307+0000","last_unstale":"2024-02-19T15:22:07.424492+0000","last_undegraded":"2024-02-19T15:22:07.424492+0000","last_fullsized":"2024-02-19T15:22:07.424492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:37:51.996719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853286+0000","last_change":"2024-02-19T15:22:04.841547+0000","last_active":"2024-02-19T15:22:06.853286+0000","last_peered":"2024-02-19T15:22:06.853286+0000","last_clean":"2024-02-19T15:22:06.853286+0000","last_became_active":"2024-02-19T15:22:04.841280+0000","last_became_peered":"2024-02-19T15:22:04.841280+0000","last_unstale":"2024-02-19T15:22:06.853286+0000","last_undegraded":"2024-02-19T15:22:06.853286+0000","last_fullsized":"2024-02-19T15:22:06.853286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:40.779845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851683+0000","last_change":"2024-02-19T15:22:05.863084+0000","last_active":"2024-02-19T15:22:06.851683+0000","last_peered":"2024-02-19T15:22:06.851683+0000","last_clean":"2024-02-19T15:22:06.851683+0000","last_became_active":"2024-02-19T15:22:05.862820+0000","last_became_peered":"2024-02-19T15:22:05.862820+0000","last_unstale":"2024-02-19T15:22:06.851683+0000","last_undegraded":"2024-02-19T15:22:06.851683+0000","last_fullsized":"2024-02-19T15:22:06.851683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:43.283192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.853263+0000","last_change":"2024-02-19T15:22:04.840145+0000","last_active":"2024-02-19T15:22:06.853263+0000","last_peered":"2024-02-19T15:22:06.853263+0000","last_clean":"2024-02-19T15:22:06.853263+0000","last_became_active":"2024-02-19T15:22:04.839832+0000","last_became_peered":"2024-02-19T15:22:04.839832+0000","last_unstale":"2024-02-19T15:22:06.853263+0000","last_undegraded":"2024-02-19T15:22:06.853263+0000","last_fullsized":"2024-02-19T15:22:06.853263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:27.325777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851519+0000","last_change":"2024-02-19T15:22:05.862437+0000","last_active":"2024-02-19T15:22:06.851519+0000","last_peered":"2024-02-19T15:22:06.851519+0000","last_clean":"2024-02-19T15:22:06.851519+0000","last_became_active":"2024-02-19T15:22:05.855551+0000","last_became_peered":"2024-02-19T15:22:05.855551+0000","last_unstale":"2024-02-19T15:22:06.851519+0000","last_undegraded":"2024-02-19T15:22:06.851519+0000","last_fullsized":"2024-02-19T15:22:06.851519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:21:49.261015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181446+0000","last_change":"2024-02-19T15:22:04.849761+0000","last_active":"2024-02-19T15:22:07.181446+0000","last_peered":"2024-02-19T15:22:07.181446+0000","last_clean":"2024-02-19T15:22:07.181446+0000","last_became_active":"2024-02-19T15:22:04.849413+0000","last_became_peered":"2024-02-19T15:22:04.849413+0000","last_unstale":"2024-02-19T15:22:07.181446+0000","last_undegraded":"2024-02-19T15:22:07.181446+0000","last_fullsized":"2024-02-19T15:22:07.181446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:37:50.991116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.854363+0000","last_change":"2024-02-19T15:22:05.871270+0000","last_active":"2024-02-19T15:22:06.854363+0000","last_peered":"2024-02-19T15:22:06.854363+0000","last_clean":"2024-02-19T15:22:06.854363+0000","last_became_active":"2024-02-19T15:22:05.870986+0000","last_became_peered":"2024-02-19T15:22:05.870986+0000","last_unstale":"2024-02-19T15:22:06.854363+0000","last_undegraded":"2024-02-19T15:22:06.854363+0000","last_fullsized":"2024-02-19T15:22:06.854363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:22:17.882758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424727+0000","last_change":"2024-02-19T15:22:05.864283+0000","last_active":"2024-02-19T15:22:07.424727+0000","last_peered":"2024-02-19T15:22:07.424727+0000","last_clean":"2024-02-19T15:22:07.424727+0000","last_became_active":"2024-02-19T15:22:05.863996+0000","last_became_peered":"2024-02-19T15:22:05.863996+0000","last_unstale":"2024-02-19T15:22:07.424727+0000","last_undegraded":"2024-02-19T15:22:07.424727+0000","last_fullsized":"2024-02-19T15:22:07.424727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:59:01.655185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181379+0000","last_change":"2024-02-19T15:22:04.855753+0000","last_active":"2024-02-19T15:22:07.181379+0000","last_peered":"2024-02-19T15:22:07.181379+0000","last_clean":"2024-02-19T15:22:07.181379+0000","last_became_active":"2024-02-19T15:22:04.855303+0000","last_became_peered":"2024-02-19T15:22:04.855303+0000","last_unstale":"2024-02-19T15:22:07.181379+0000","last_undegraded":"2024-02-19T15:22:07.181379+0000","last_fullsized":"2024-02-19T15:22:07.181379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:04.751023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424554+0000","last_change":"2024-02-19T15:22:05.870398+0000","last_active":"2024-02-19T15:22:07.424554+0000","last_peered":"2024-02-19T15:22:07.424554+0000","last_clean":"2024-02-19T15:22:07.424554+0000","last_became_active":"2024-02-19T15:22:05.869963+0000","last_became_peered":"2024-02-19T15:22:05.869963+0000","last_unstale":"2024-02-19T15:22:07.424554+0000","last_undegraded":"2024-02-19T15:22:07.424554+0000","last_fullsized":"2024-02-19T15:22:07.424554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:55:27.572569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.181517+0000","last_change":"2024-02-19T15:22:04.849798+0000","last_active":"2024-02-19T15:22:07.181517+0000","last_peered":"2024-02-19T15:22:07.181517+0000","last_clean":"2024-02-19T15:22:07.181517+0000","last_became_active":"2024-02-19T15:22:04.849115+0000","last_became_peered":"2024-02-19T15:22:04.849115+0000","last_unstale":"2024-02-19T15:22:07.181517+0000","last_undegraded":"2024-02-19T15:22:07.181517+0000","last_fullsized":"2024-02-19T15:22:07.181517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:01.912213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424542+0000","last_change":"2024-02-19T15:22:05.861837+0000","last_active":"2024-02-19T15:22:07.424542+0000","last_peered":"2024-02-19T15:22:07.424542+0000","last_clean":"2024-02-19T15:22:07.424542+0000","last_became_active":"2024-02-19T15:22:05.861570+0000","last_became_peered":"2024-02-19T15:22:05.861570+0000","last_unstale":"2024-02-19T15:22:07.424542+0000","last_undegraded":"2024-02-19T15:22:07.424542+0000","last_fullsized":"2024-02-19T15:22:07.424542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:54:25.604351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851906+0000","last_change":"2024-02-19T15:22:04.841218+0000","last_active":"2024-02-19T15:22:06.851906+0000","last_peered":"2024-02-19T15:22:06.851906+0000","last_clean":"2024-02-19T15:22:06.851906+0000","last_became_active":"2024-02-19T15:22:04.840740+0000","last_became_peered":"2024-02-19T15:22:04.840740+0000","last_unstale":"2024-02-19T15:22:06.851906+0000","last_undegraded":"2024-02-19T15:22:06.851906+0000","last_fullsized":"2024-02-19T15:22:06.851906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:21.137278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:06.851910+0000","last_change":"2024-02-19T15:22:05.847219+0000","last_active":"2024-02-19T15:22:06.851910+0000","last_peered":"2024-02-19T15:22:06.851910+0000","last_clean":"2024-02-19T15:22:06.851910+0000","last_became_active":"2024-02-19T15:22:05.846948+0000","last_became_peered":"2024-02-19T15:22:05.846948+0000","last_unstale":"2024-02-19T15:22:06.851910+0000","last_undegraded":"2024-02-19T15:22:06.851910+0000","last_fullsized":"2024-02-19T15:22:06.851910+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:04.811516+0000","last_clean_scrub_stamp":"2024-02-19T15:22:04.811516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:08:56.380125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-02-19T15:22:07.424663+0000","last_change":"2024-02-19T15:22:04.843469+0000","last_active":"2024-02-19T15:22:07.424663+0000","last_peered":"2024-02-19T15:22:07.424663+0000","last_clean":"2024-02-19T15:22:07.424663+0000","last_became_active":"2024-02-19T15:22:04.842098+0000","last_became_peered":"2024-02-19T15:22:04.842098+0000","last_unstale":"2024-02-19T15:22:07.424663+0000","last_undegraded":"2024-02-19T15:22:07.424663+0000","last_fullsized":"2024-02-19T15:22:07.424663+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:03.805912+0000","last_clean_scrub_stamp":"2024-02-19T15:22:03.805912+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:32:33.319050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509447,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]}]},{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":664833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]}]},{"osd":5,"up_from":14,"seq":60129542152,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":73973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]}]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":74009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":75495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":73883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.7160000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.1620000000000008}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.7679999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.3810000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4039999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5220000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4829999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":221184,"data_stored":74043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":819200,"data_stored":664889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:22:15.033 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-02-19T15:22:15.033 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-19T15:22:15.033 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-02-19T15:22:15.033 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-02-19T15:22:15.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.210+0000 7f8b63a00640 1 Processor -- start 2024-02-19T15:22:15.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.210+0000 7f8b63a00640 1 -- start start 2024-02-19T15:22:15.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.210+0000 7f8b63a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8b5c15d6e0 0x7f8b5c162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:15.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.210+0000 7f8b63a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5c0718c0 con 0x7f8b5c071350 2024-02-19T15:22:15.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.210+0000 7f8b63a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5c071ba0 con 0x7f8b5c15b1a0 2024-02-19T15:22:15.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.210+0000 7f8b63a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b5c071e80 con 0x7f8b5c15d6e0 2024-02-19T15:22:15.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.210+0000 7f8b53e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8b5c15d6e0 0x7f8b5c162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:15.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.210+0000 7f8b53e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8b5c15d6e0 0x7f8b5c162740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57890/0 (socket says 172.21.15.47:57890) 2024-02-19T15:22:15.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.210+0000 7f8b53e00640 1 -- 172.21.15.47:0/1127654091 learned_addr learned my addr 172.21.15.47:0/1127654091 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:15.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.210+0000 7f8b53e00640 1 -- 172.21.15.47:0/1127654091 >> v1:172.21.15.116:6789/0 conn(0x7f8b5c15b1a0 legacy=0x7f8b5c15b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.210+0000 7f8b53e00640 1 -- 172.21.15.47:0/1127654091 >> v1:172.21.15.47:6789/0 conn(0x7f8b5c071350 legacy=0x7f8b5c164c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.210+0000 7f8b53e00640 1 -- 172.21.15.47:0/1127654091 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5c072160 con 0x7f8b5c15d6e0 2024-02-19T15:22:15.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.214+0000 7f8b53e00640 1 --2- 172.21.15.47:0/1127654091 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b5c15d6e0 0x7f8b5c162740 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8b440098a0 tx=0x7f8b4401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16194ae7d4a4b762 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:15.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.214+0000 7f8b5be00640 1 -- 172.21.15.47:0/1127654091 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b440ab660 con 0x7f8b5c15d6e0 2024-02-19T15:22:15.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.214+0000 7f8b5be00640 1 -- 172.21.15.47:0/1127654091 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b440ab7c0 con 0x7f8b5c15d6e0 2024-02-19T15:22:15.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.214+0000 7f8b5be00640 1 -- 172.21.15.47:0/1127654091 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b440abb50 con 0x7f8b5c15d6e0 2024-02-19T15:22:15.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.214+0000 7f8b63a00640 1 -- 172.21.15.47:0/1127654091 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b5c15d6e0 msgr2=0x7f8b5c162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.214+0000 7f8b63a00640 1 --2- 172.21.15.47:0/1127654091 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b5c15d6e0 0x7f8b5c162740 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8b440098a0 tx=0x7f8b4401c540 comp rx=0 tx=0).stop 2024-02-19T15:22:15.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.214+0000 7f8b63a00640 1 -- 172.21.15.47:0/1127654091 shutdown_connections 2024-02-19T15:22:15.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.214+0000 7f8b63a00640 1 --2- 172.21.15.47:0/1127654091 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b5c15d6e0 0x7f8b5c162740 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.214+0000 7f8b63a00640 1 -- 172.21.15.47:0/1127654091 >> 172.21.15.47:0/1127654091 conn(0x7f8b5c0b16a0 msgr2=0x7f8b5c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:15.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.214+0000 7f8b63a00640 1 -- 172.21.15.47:0/1127654091 shutdown_connections 2024-02-19T15:22:15.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.214+0000 7f8b63a00640 1 -- 172.21.15.47:0/1127654091 wait complete. 2024-02-19T15:22:15.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b63a00640 1 Processor -- start 2024-02-19T15:22:15.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b63a00640 1 -- start start 2024-02-19T15:22:15.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b63a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b5c071350 0x7f8b5c15a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:15.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b63a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b5c15b1a0 0x7f8b5c14eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:15.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b63a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b5c15d6e0 0x7f8b5c0b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:15.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b63a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c150f20 con 0x7f8b5c15d6e0 2024-02-19T15:22:15.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b63a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c1528f0 con 0x7f8b5c15b1a0 2024-02-19T15:22:15.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b63a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c158490 con 0x7f8b5c071350 2024-02-19T15:22:15.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b61a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b5c15d6e0 0x7f8b5c0b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:15.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b61a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b5c15d6e0 0x7f8b5c0b0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57894/0 (socket says 172.21.15.47:57894) 2024-02-19T15:22:15.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b61a00640 1 -- 172.21.15.47:0/903065172 learned_addr learned my addr 172.21.15.47:0/903065172 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:15.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b61000640 1 --2- 172.21.15.47:0/903065172 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b5c071350 0x7f8b5c15a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:15.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b53e00640 1 --2- 172.21.15.47:0/903065172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b5c15b1a0 0x7f8b5c14eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:15.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b61a00640 1 -- 172.21.15.47:0/903065172 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b5c071350 msgr2=0x7f8b5c15a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b61a00640 1 --2- 172.21.15.47:0/903065172 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b5c071350 0x7f8b5c15a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b61a00640 1 -- 172.21.15.47:0/903065172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b5c15b1a0 msgr2=0x7f8b5c14eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b61a00640 1 --2- 172.21.15.47:0/903065172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b5c15b1a0 0x7f8b5c14eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b61a00640 1 -- 172.21.15.47:0/903065172 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b440094d0 con 0x7f8b5c15d6e0 2024-02-19T15:22:15.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b61a00640 1 --2- 172.21.15.47:0/903065172 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b5c15d6e0 0x7f8b5c0b0430 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400c760 tx=0x7f8b5400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:15.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b59600640 1 -- 172.21.15.47:0/903065172 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b54003b80 con 0x7f8b5c15d6e0 2024-02-19T15:22:15.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b59600640 1 -- 172.21.15.47:0/903065172 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b54003ce0 con 0x7f8b5c15d6e0 2024-02-19T15:22:15.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b59600640 1 -- 172.21.15.47:0/903065172 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5400aa30 con 0x7f8b5c15d6e0 2024-02-19T15:22:15.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b63a00640 1 -- 172.21.15.47:0/903065172 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b5c151130 con 0x7f8b5c15d6e0 2024-02-19T15:22:15.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.218+0000 7f8b63a00640 1 -- 172.21.15.47:0/903065172 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b5c152e00 con 0x7f8b5c15d6e0 2024-02-19T15:22:15.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.222+0000 7f8b63a00640 1 -- 172.21.15.47:0/903065172 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b5c152ff0 con 0x7f8b5c15d6e0 2024-02-19T15:22:15.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.222+0000 7f8b59600640 1 -- 172.21.15.47:0/903065172 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8b5400ab90 con 0x7f8b5c15d6e0 2024-02-19T15:22:15.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.222+0000 7f8b59600640 1 --2- 172.21.15.47:0/903065172 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8b38066e40 0x7f8b38069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:15.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.222+0000 7f8b59600640 1 -- 172.21.15.47:0/903065172 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8b54084700 con 0x7f8b5c15d6e0 2024-02-19T15:22:15.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.222+0000 7f8b61000640 1 --2- 172.21.15.47:0/903065172 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8b38066e40 0x7f8b38069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:15.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.222+0000 7f8b61000640 1 --2- 172.21.15.47:0/903065172 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8b38066e40 0x7f8b38069300 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c007fd0 tx=0x7f8b4c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:15.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.226+0000 7f8b59600640 1 -- 172.21.15.47:0/903065172 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8b54051500 con 0x7f8b5c15d6e0 2024-02-19T15:22:15.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.422+0000 7f8b63a00640 1 -- 172.21.15.47:0/903065172 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8b5c14f0d0 con 0x7f8b5c15d6e0 2024-02-19T15:22:15.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.422+0000 7f8b59600640 1 -- 172.21.15.47:0/903065172 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f8b54054bc0 con 0x7f8b5c15d6e0 2024-02-19T15:22:15.423 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:15.423 INFO:teuthology.orchestra.run.smithi047.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-19T15:22:15.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.422+0000 7f8b63a00640 1 -- 172.21.15.47:0/903065172 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8b38066e40 msgr2=0x7f8b38069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.422+0000 7f8b63a00640 1 --2- 172.21.15.47:0/903065172 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8b38066e40 0x7f8b38069300 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c007fd0 tx=0x7f8b4c009040 comp rx=0 tx=0).stop 2024-02-19T15:22:15.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.422+0000 7f8b63a00640 1 -- 172.21.15.47:0/903065172 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b5c15d6e0 msgr2=0x7f8b5c0b0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.422+0000 7f8b63a00640 1 --2- 172.21.15.47:0/903065172 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b5c15d6e0 0x7f8b5c0b0430 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400c760 tx=0x7f8b5400cc20 comp rx=0 tx=0).stop 2024-02-19T15:22:15.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.422+0000 7f8b63a00640 1 -- 172.21.15.47:0/903065172 shutdown_connections 2024-02-19T15:22:15.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.422+0000 7f8b63a00640 1 --2- 172.21.15.47:0/903065172 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8b38066e40 0x7f8b38069300 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.422+0000 7f8b63a00640 1 --2- 172.21.15.47:0/903065172 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b5c15d6e0 0x7f8b5c0b0430 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.422+0000 7f8b63a00640 1 --2- 172.21.15.47:0/903065172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b5c15b1a0 0x7f8b5c14eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.422+0000 7f8b63a00640 1 --2- 172.21.15.47:0/903065172 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b5c071350 0x7f8b5c15a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.422+0000 7f8b63a00640 1 -- 172.21.15.47:0/903065172 >> 172.21.15.47:0/903065172 conn(0x7f8b5c0b16a0 msgr2=0x7f8b5c0cf0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:15.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.422+0000 7f8b63a00640 1 -- 172.21.15.47:0/903065172 shutdown_connections 2024-02-19T15:22:15.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.422+0000 7f8b63a00640 1 -- 172.21.15.47:0/903065172 wait complete. 2024-02-19T15:22:15.445 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-02-19T15:22:15.446 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:15.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.582+0000 7fa712e00640 1 Processor -- start 2024-02-19T15:22:15.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.582+0000 7fa712e00640 1 -- start start 2024-02-19T15:22:15.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.586+0000 7fa712e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa70c06cfc0 0x7fa70c0b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:15.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.586+0000 7fa712e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa70c06c660 con 0x7fa70c15e180 2024-02-19T15:22:15.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.586+0000 7fa712e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa70c0b2270 con 0x7fa70c06c1d0 2024-02-19T15:22:15.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.586+0000 7fa712e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa70c0b2550 con 0x7fa70c06cfc0 2024-02-19T15:22:15.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.586+0000 7fa703e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa70c06cfc0 0x7fa70c0b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:15.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.586+0000 7fa70be00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa70c15e180 0x7fa70c06c8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35380/0 (socket says 172.21.15.47:35380) 2024-02-19T15:22:15.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.586+0000 7fa703e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa70c06cfc0 0x7fa70c0b1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57904/0 (socket says 172.21.15.47:57904) 2024-02-19T15:22:15.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.586+0000 7fa70be00640 1 -- 172.21.15.47:0/1796048794 learned_addr learned my addr 172.21.15.47:0/1796048794 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:15.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.586+0000 7fa703e00640 1 -- 172.21.15.47:0/1796048794 >> v1:172.21.15.116:6789/0 conn(0x7fa70c06c1d0 legacy=0x7fa70c0ae870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.586+0000 7fa703e00640 1 -- 172.21.15.47:0/1796048794 >> v1:172.21.15.47:6789/0 conn(0x7fa70c15e180 legacy=0x7fa70c06c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.586+0000 7fa703e00640 1 -- 172.21.15.47:0/1796048794 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa70c0b2830 con 0x7fa70c06cfc0 2024-02-19T15:22:15.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.586+0000 7fa70b400640 1 -- 172.21.15.47:0/1796048794 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857767962 0 0) 0x7fa70c06c660 con 0x7fa70c15e180 2024-02-19T15:22:15.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.586+0000 7fa703e00640 1 --2- 172.21.15.47:0/1796048794 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa70c06cfc0 0x7fa70c0b1c20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa6f40097c0 tx=0x7fa6f401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5768adf9d574a6e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:15.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa70b400640 1 -- 172.21.15.47:0/1796048794 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f40ab670 con 0x7fa70c06cfc0 2024-02-19T15:22:15.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa70b400640 1 -- 172.21.15.47:0/1796048794 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f40ab7d0 con 0x7fa70c06cfc0 2024-02-19T15:22:15.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa70b400640 1 -- 172.21.15.47:0/1796048794 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f40abb60 con 0x7fa70c06cfc0 2024-02-19T15:22:15.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa712e00640 1 -- 172.21.15.47:0/1796048794 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa70c06cfc0 msgr2=0x7fa70c0b1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa712e00640 1 --2- 172.21.15.47:0/1796048794 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa70c06cfc0 0x7fa70c0b1c20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa6f40097c0 tx=0x7fa6f401c470 comp rx=0 tx=0).stop 2024-02-19T15:22:15.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa712e00640 1 -- 172.21.15.47:0/1796048794 shutdown_connections 2024-02-19T15:22:15.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa712e00640 1 --2- 172.21.15.47:0/1796048794 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa70c06cfc0 0x7fa70c0b1c20 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa712e00640 1 -- 172.21.15.47:0/1796048794 >> 172.21.15.47:0/1796048794 conn(0x7fa70c0d1ba0 msgr2=0x7fa70c0d1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:15.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa712e00640 1 -- 172.21.15.47:0/1796048794 shutdown_connections 2024-02-19T15:22:15.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa712e00640 1 -- 172.21.15.47:0/1796048794 wait complete. 2024-02-19T15:22:15.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa712e00640 1 Processor -- start 2024-02-19T15:22:15.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa712e00640 1 -- start start 2024-02-19T15:22:15.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa712e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa70c06c1d0 0x7fa70c0b6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:15.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa712e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70c06cfc0 0x7fa70c152480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:15.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa712e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa70c15e180 0x7fa70c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:15.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa712e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c0b4c80 con 0x7fa70c06c1d0 2024-02-19T15:22:15.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa712e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c0b7190 con 0x7fa70c06cfc0 2024-02-19T15:22:15.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa712e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c0c5a20 con 0x7fa70c15e180 2024-02-19T15:22:15.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa70be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa70c06c1d0 0x7fa70c0b6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:15.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa70be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa70c06c1d0 0x7fa70c0b6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57918/0 (socket says 172.21.15.47:57918) 2024-02-19T15:22:15.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa70be00640 1 -- 172.21.15.47:0/367871983 learned_addr learned my addr 172.21.15.47:0/367871983 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:15.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa703e00640 1 --2- 172.21.15.47:0/367871983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70c06cfc0 0x7fa70c152480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:15.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.590+0000 7fa710e00640 1 --2- 172.21.15.47:0/367871983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa70c15e180 0x7fa70c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:15.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.594+0000 7fa70be00640 1 -- 172.21.15.47:0/367871983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa70c15e180 msgr2=0x7fa70c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.594+0000 7fa70be00640 1 --2- 172.21.15.47:0/367871983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa70c15e180 0x7fa70c1529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.594+0000 7fa70be00640 1 -- 172.21.15.47:0/367871983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70c06cfc0 msgr2=0x7fa70c152480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.594+0000 7fa70be00640 1 --2- 172.21.15.47:0/367871983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70c06cfc0 0x7fa70c152480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.594+0000 7fa70be00640 1 -- 172.21.15.47:0/367871983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f4009420 con 0x7fa70c06c1d0 2024-02-19T15:22:15.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.594+0000 7fa703e00640 1 --2- 172.21.15.47:0/367871983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70c06cfc0 0x7fa70c152480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:15.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.594+0000 7fa70be00640 1 --2- 172.21.15.47:0/367871983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa70c06c1d0 0x7fa70c0b6c50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0011f0 tx=0x7fa6fc00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:15.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.594+0000 7fa708c00640 1 -- 172.21.15.47:0/367871983 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc0074e0 con 0x7fa70c06c1d0 2024-02-19T15:22:15.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.594+0000 7fa708c00640 1 -- 172.21.15.47:0/367871983 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6fc007640 con 0x7fa70c06c1d0 2024-02-19T15:22:15.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.594+0000 7fa708c00640 1 -- 172.21.15.47:0/367871983 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc004a10 con 0x7fa70c06c1d0 2024-02-19T15:22:15.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.594+0000 7fa712e00640 1 -- 172.21.15.47:0/367871983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa70c0c56d0 con 0x7fa70c06c1d0 2024-02-19T15:22:15.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.594+0000 7fa712e00640 1 -- 172.21.15.47:0/367871983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa70c0b76a0 con 0x7fa70c06c1d0 2024-02-19T15:22:15.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.594+0000 7fa712e00640 1 -- 172.21.15.47:0/367871983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa70c068500 con 0x7fa70c06c1d0 2024-02-19T15:22:15.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.594+0000 7fa708c00640 1 -- 172.21.15.47:0/367871983 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa6fc004b70 con 0x7fa70c06c1d0 2024-02-19T15:22:15.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.594+0000 7fa708c00640 1 --2- 172.21.15.47:0/367871983 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa6e8066e40 0x7fa6e8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:15.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.594+0000 7fa708c00640 1 -- 172.21.15.47:0/367871983 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6fc083d20 con 0x7fa70c06c1d0 2024-02-19T15:22:15.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.594+0000 7fa703e00640 1 --2- 172.21.15.47:0/367871983 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa6e8066e40 0x7fa6e8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:15.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.598+0000 7fa703e00640 1 --2- 172.21.15.47:0/367871983 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa6e8066e40 0x7fa6e8069300 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa6f40af000 tx=0x7fa6f40a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:15.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.598+0000 7fa708c00640 1 -- 172.21.15.47:0/367871983 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa6fc050b20 con 0x7fa70c06c1d0 2024-02-19T15:22:15.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.762+0000 7fa712e00640 1 -- 172.21.15.47:0/367871983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa70c075360 con 0x7fa70c06c1d0 2024-02-19T15:22:15.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.766+0000 7fa708c00640 1 -- 172.21.15.47:0/367871983 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7fa6fc018070 con 0x7fa70c06c1d0 2024-02-19T15:22:15.767 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:15.767 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":21,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:06.843155+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:03.091360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:04.222965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:15.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.766+0000 7fa712e00640 1 -- 172.21.15.47:0/367871983 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa6e8066e40 msgr2=0x7fa6e8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.766+0000 7fa712e00640 1 --2- 172.21.15.47:0/367871983 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa6e8066e40 0x7fa6e8069300 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa6f40af000 tx=0x7fa6f40a7000 comp rx=0 tx=0).stop 2024-02-19T15:22:15.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.766+0000 7fa712e00640 1 -- 172.21.15.47:0/367871983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa70c06c1d0 msgr2=0x7fa70c0b6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.766+0000 7fa712e00640 1 --2- 172.21.15.47:0/367871983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa70c06c1d0 0x7fa70c0b6c50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0011f0 tx=0x7fa6fc00c9b0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.766+0000 7fa712e00640 1 -- 172.21.15.47:0/367871983 shutdown_connections 2024-02-19T15:22:15.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.766+0000 7fa712e00640 1 --2- 172.21.15.47:0/367871983 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa6e8066e40 0x7fa6e8069300 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.766+0000 7fa712e00640 1 --2- 172.21.15.47:0/367871983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa70c15e180 0x7fa70c1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.766+0000 7fa712e00640 1 --2- 172.21.15.47:0/367871983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70c06cfc0 0x7fa70c152480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.766+0000 7fa712e00640 1 --2- 172.21.15.47:0/367871983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa70c06c1d0 0x7fa70c0b6c50 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.766+0000 7fa712e00640 1 -- 172.21.15.47:0/367871983 >> 172.21.15.47:0/367871983 conn(0x7fa70c0d1ba0 msgr2=0x7fa70c0ae870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:15.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.766+0000 7fa712e00640 1 -- 172.21.15.47:0/367871983 shutdown_connections 2024-02-19T15:22:15.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.766+0000 7fa712e00640 1 -- 172.21.15.47:0/367871983 wait complete. 2024-02-19T15:22:15.788 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:22:03.091360+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:22:04.222965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:22:15.788 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:22:15.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.934+0000 7f386ac00640 1 Processor -- start 2024-02-19T15:22:15.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.934+0000 7f386ac00640 1 -- start start 2024-02-19T15:22:15.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.934+0000 7f386ac00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f38640d2650 0x7f386406c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:15.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.934+0000 7f386ac00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38640cc620 con 0x7f386415e280 2024-02-19T15:22:15.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.934+0000 7f386ac00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38640d12e0 con 0x7f38640ce3b0 2024-02-19T15:22:15.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.934+0000 7f386ac00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38640d1940 con 0x7f38640d2650 2024-02-19T15:22:15.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.934+0000 7f3868c00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f386415e280 0x7f386406c710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35388/0 (socket says 172.21.15.47:35388) 2024-02-19T15:22:15.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.934+0000 7f3868c00640 1 -- 172.21.15.47:0/695235415 learned_addr learned my addr 172.21.15.47:0/695235415 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:15.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.934+0000 7f385b400640 1 --2- 172.21.15.47:0/695235415 >> v2:172.21.15.116:3300/0 conn(0x7f38640d2650 0x7f386406c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:15.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.934+0000 7f3863400640 1 -- 172.21.15.47:0/695235415 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736308451 0 0) 0x7f38640cc620 con 0x7f386415e280 2024-02-19T15:22:15.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.934+0000 7f3863400640 1 -- 172.21.15.47:0/695235415 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3848003530 con 0x7f386415e280 2024-02-19T15:22:15.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.934+0000 7f3863400640 1 -- 172.21.15.47:0/695235415 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615274529 0 0) 0x7f38640d12e0 con 0x7f38640ce3b0 2024-02-19T15:22:15.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.934+0000 7f385b400640 1 -- 172.21.15.47:0/695235415 >> v1:172.21.15.116:6789/0 conn(0x7f38640ce3b0 legacy=0x7f38640ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.934+0000 7f385b400640 1 -- 172.21.15.47:0/695235415 >> v1:172.21.15.47:6789/0 conn(0x7f386415e280 legacy=0x7f386406c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.934+0000 7f385b400640 1 -- 172.21.15.47:0/695235415 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38640d1000 con 0x7f38640d2650 2024-02-19T15:22:15.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.938+0000 7f385b400640 1 --2- 172.21.15.47:0/695235415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38640d2650 0x7f386406c1d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f384c0097c0 tx=0x7f384c01c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3a33fc176d43e7b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:15.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.938+0000 7f3863400640 1 -- 172.21.15.47:0/695235415 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384c0a4ea0 con 0x7f38640d2650 2024-02-19T15:22:15.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.938+0000 7f3863400640 1 -- 172.21.15.47:0/695235415 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384c0a7050 con 0x7f38640d2650 2024-02-19T15:22:15.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.938+0000 7f3863400640 1 -- 172.21.15.47:0/695235415 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384c0ab820 con 0x7f38640d2650 2024-02-19T15:22:15.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.938+0000 7f386ac00640 1 -- 172.21.15.47:0/695235415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38640d2650 msgr2=0x7f386406c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.938+0000 7f386ac00640 1 --2- 172.21.15.47:0/695235415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38640d2650 0x7f386406c1d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f384c0097c0 tx=0x7f384c01c380 comp rx=0 tx=0).stop 2024-02-19T15:22:15.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.938+0000 7f386ac00640 1 -- 172.21.15.47:0/695235415 shutdown_connections 2024-02-19T15:22:15.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.938+0000 7f386ac00640 1 --2- 172.21.15.47:0/695235415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38640d2650 0x7f386406c1d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.938+0000 7f386ac00640 1 -- 172.21.15.47:0/695235415 >> 172.21.15.47:0/695235415 conn(0x7f38640d1b30 msgr2=0x7f38640d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:15.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.938+0000 7f386ac00640 1 -- 172.21.15.47:0/695235415 shutdown_connections 2024-02-19T15:22:15.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.938+0000 7f386ac00640 1 -- 172.21.15.47:0/695235415 wait complete. 2024-02-19T15:22:15.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.938+0000 7f386ac00640 1 Processor -- start 2024-02-19T15:22:15.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.938+0000 7f386ac00640 1 -- start start 2024-02-19T15:22:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f386ac00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38640ce3b0 0x7f38641530b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f386ac00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38640d2650 0x7f386415d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f386ac00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f386415e280 0x7f3864155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f386ac00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3864153bd0 con 0x7f38640d2650 2024-02-19T15:22:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f386ac00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38641535f0 con 0x7f38640ce3b0 2024-02-19T15:22:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f386ac00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f386415da90 con 0x7f386415e280 2024-02-19T15:22:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f385b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38640d2650 0x7f386415d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f385b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38640d2650 0x7f386415d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57920/0 (socket says 172.21.15.47:57920) 2024-02-19T15:22:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f385b400640 1 -- 172.21.15.47:0/1996308217 learned_addr learned my addr 172.21.15.47:0/1996308217 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f3863e00640 1 --2- 172.21.15.47:0/1996308217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38640ce3b0 0x7f38641530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f3868c00640 1 --2- 172.21.15.47:0/1996308217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f386415e280 0x7f3864155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f385b400640 1 -- 172.21.15.47:0/1996308217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f386415e280 msgr2=0x7f3864155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f385b400640 1 --2- 172.21.15.47:0/1996308217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f386415e280 0x7f3864155fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f385b400640 1 -- 172.21.15.47:0/1996308217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38640ce3b0 msgr2=0x7f38641530b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:15.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f385b400640 1 --2- 172.21.15.47:0/1996308217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38640ce3b0 0x7f38641530b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:15.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f385b400640 1 -- 172.21.15.47:0/1996308217 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f384c009420 con 0x7f38640d2650 2024-02-19T15:22:15.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f3863e00640 1 --2- 172.21.15.47:0/1996308217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38640ce3b0 0x7f38641530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:15.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f385b400640 1 --2- 172.21.15.47:0/1996308217 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38640d2650 0x7f386415d550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f384c0097c0 tx=0x7f384c019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:15.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f3860c00640 1 -- 172.21.15.47:0/1996308217 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384c006e20 con 0x7f38640d2650 2024-02-19T15:22:15.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f3860c00640 1 -- 172.21.15.47:0/1996308217 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384c004020 con 0x7f38640d2650 2024-02-19T15:22:15.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f386ac00640 1 -- 172.21.15.47:0/1996308217 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f386415aed0 con 0x7f38640d2650 2024-02-19T15:22:15.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f3860c00640 1 -- 172.21.15.47:0/1996308217 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384c0043b0 con 0x7f38640d2650 2024-02-19T15:22:15.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f386ac00640 1 -- 172.21.15.47:0/1996308217 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f386415b190 con 0x7f38640d2650 2024-02-19T15:22:15.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.942+0000 7f386ac00640 1 -- 172.21.15.47:0/1996308217 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3864153f80 con 0x7f38640d2650 2024-02-19T15:22:15.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.946+0000 7f3860c00640 1 -- 172.21.15.47:0/1996308217 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f384c0b2040 con 0x7f38640d2650 2024-02-19T15:22:15.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.946+0000 7f3860c00640 1 --2- 172.21.15.47:0/1996308217 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3840066e40 0x7f3840069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:15.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.946+0000 7f3860c00640 1 -- 172.21.15.47:0/1996308217 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f384c11e830 con 0x7f38640d2650 2024-02-19T15:22:15.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.946+0000 7f3863e00640 1 --2- 172.21.15.47:0/1996308217 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3840066e40 0x7f3840069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:15.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.946+0000 7f3863e00640 1 --2- 172.21.15.47:0/1996308217 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3840066e40 0x7f3840069300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f38540012f0 tx=0x7f3854009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:15.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:15.950+0000 7f3860c00640 1 -- 172.21.15.47:0/1996308217 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f384c0eb630 con 0x7f38640d2650 2024-02-19T15:22:16.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.114+0000 7f386ac00640 1 -- 172.21.15.47:0/1996308217 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3864075360 con 0x7f38640d2650 2024-02-19T15:22:16.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.114+0000 7f3860c00640 1 -- 172.21.15.47:0/1996308217 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f384c0eecf0 con 0x7f38640d2650 2024-02-19T15:22:16.115 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:22:16.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.114+0000 7f386ac00640 1 -- 172.21.15.47:0/1996308217 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3840066e40 msgr2=0x7f3840069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:16.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.114+0000 7f386ac00640 1 --2- 172.21.15.47:0/1996308217 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3840066e40 0x7f3840069300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f38540012f0 tx=0x7f3854009040 comp rx=0 tx=0).stop 2024-02-19T15:22:16.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.114+0000 7f386ac00640 1 -- 172.21.15.47:0/1996308217 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38640d2650 msgr2=0x7f386415d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:16.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.114+0000 7f386ac00640 1 --2- 172.21.15.47:0/1996308217 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38640d2650 0x7f386415d550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f384c0097c0 tx=0x7f384c019120 comp rx=0 tx=0).stop 2024-02-19T15:22:16.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.114+0000 7f386ac00640 1 -- 172.21.15.47:0/1996308217 shutdown_connections 2024-02-19T15:22:16.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.114+0000 7f386ac00640 1 --2- 172.21.15.47:0/1996308217 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3840066e40 0x7f3840069300 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.114+0000 7f386ac00640 1 --2- 172.21.15.47:0/1996308217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f386415e280 0x7f3864155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.114+0000 7f386ac00640 1 --2- 172.21.15.47:0/1996308217 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38640d2650 0x7f386415d550 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.114+0000 7f386ac00640 1 --2- 172.21.15.47:0/1996308217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38640ce3b0 0x7f38641530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.114+0000 7f386ac00640 1 -- 172.21.15.47:0/1996308217 >> 172.21.15.47:0/1996308217 conn(0x7f38640d1b30 msgr2=0x7f38640c77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:16.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.114+0000 7f386ac00640 1 -- 172.21.15.47:0/1996308217 shutdown_connections 2024-02-19T15:22:16.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.114+0000 7f386ac00640 1 -- 172.21.15.47:0/1996308217 wait complete. 2024-02-19T15:22:16.137 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:22:16.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.278+0000 7fb4b5a00640 1 Processor -- start 2024-02-19T15:22:16.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.278+0000 7fb4b5a00640 1 -- start start 2024-02-19T15:22:16.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.278+0000 7fb4b5a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb4b015af10 0x7fb4b006e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:16.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.278+0000 7fb4b5a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb4b00ca8f0 0x7fb4b00cad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:16.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.278+0000 7fb4b5a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b0071c70 con 0x7fb4b00cb290 2024-02-19T15:22:16.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.278+0000 7fb4b5a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4b006e8a0 con 0x7fb4b015af10 2024-02-19T15:22:16.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.278+0000 7fb4b5a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4b006cef0 con 0x7fb4b00ca8f0 2024-02-19T15:22:16.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.278+0000 7fb4aea00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb4b015af10 0x7fb4b006e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:16.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.278+0000 7fb4a6000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb4b00ca8f0 0x7fb4b00cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:16.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.278+0000 7fb4a6000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb4b00ca8f0 0x7fb4b00cad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:36780/0 (socket says 172.21.15.47:36780) 2024-02-19T15:22:16.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.278+0000 7fb4aea00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb4b015af10 0x7fb4b006e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57932/0 (socket says 172.21.15.47:57932) 2024-02-19T15:22:16.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.278+0000 7fb4a6000640 1 -- 172.21.15.47:0/473066530 learned_addr learned my addr 172.21.15.47:0/473066530 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:16.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.278+0000 7fb4a6000640 1 -- 172.21.15.47:0/473066530 >> v2:172.21.15.47:3300/0 conn(0x7fb4b015af10 msgr2=0x7fb4b006e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:16.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.278+0000 7fb4a6000640 1 --2- 172.21.15.47:0/473066530 >> v2:172.21.15.47:3300/0 conn(0x7fb4b015af10 0x7fb4b006e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.278+0000 7fb4a6000640 1 -- 172.21.15.47:0/473066530 >> v1:172.21.15.116:6789/0 conn(0x7fb4b00cb290 legacy=0x7fb4b015eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:16.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.278+0000 7fb4a6000640 1 -- 172.21.15.47:0/473066530 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b00ca610 con 0x7fb4b00ca8f0 2024-02-19T15:22:16.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.282+0000 7fb4a6000640 1 --2- 172.21.15.47:0/473066530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4b00ca8f0 0x7fb4b00cad50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb4b00b27b0 tx=0x7fb49801c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=373caacc02f208e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:16.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.282+0000 7fb4ae000640 1 -- 172.21.15.47:0/473066530 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4980ab660 con 0x7fb4b00ca8f0 2024-02-19T15:22:16.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.282+0000 7fb4ae000640 1 -- 172.21.15.47:0/473066530 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4980ab7c0 con 0x7fb4b00ca8f0 2024-02-19T15:22:16.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.282+0000 7fb4ae000640 1 -- 172.21.15.47:0/473066530 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4980abb50 con 0x7fb4b00ca8f0 2024-02-19T15:22:16.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.282+0000 7fb4b5a00640 1 -- 172.21.15.47:0/473066530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4b00ca8f0 msgr2=0x7fb4b00cad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:16.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.282+0000 7fb4b5a00640 1 --2- 172.21.15.47:0/473066530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4b00ca8f0 0x7fb4b00cad50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb4b00b27b0 tx=0x7fb49801c460 comp rx=0 tx=0).stop 2024-02-19T15:22:16.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.282+0000 7fb4b5a00640 1 -- 172.21.15.47:0/473066530 shutdown_connections 2024-02-19T15:22:16.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.282+0000 7fb4b5a00640 1 --2- 172.21.15.47:0/473066530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4b00ca8f0 0x7fb4b00cad50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.282+0000 7fb4b5a00640 1 --2- 172.21.15.47:0/473066530 >> v2:172.21.15.47:3300/0 conn(0x7fb4b015af10 0x7fb4b006e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.282+0000 7fb4b5a00640 1 -- 172.21.15.47:0/473066530 >> 172.21.15.47:0/473066530 conn(0x7fb4b00cfc20 msgr2=0x7fb4b00d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:16.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.282+0000 7fb4b5a00640 1 -- 172.21.15.47:0/473066530 shutdown_connections 2024-02-19T15:22:16.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.282+0000 7fb4b5a00640 1 -- 172.21.15.47:0/473066530 wait complete. 2024-02-19T15:22:16.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4b5a00640 1 Processor -- start 2024-02-19T15:22:16.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4b5a00640 1 -- start start 2024-02-19T15:22:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4b5a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4b00ca8f0 0x7fb4b00c28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4b5a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4b00cb290 0x7fb4b0158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4b5a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b015af10 0x7fb4b01451d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4b5a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b0144d40 con 0x7fb4b015af10 2024-02-19T15:22:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4b5a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b0159780 con 0x7fb4b00cb290 2024-02-19T15:22:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4b5a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b01591c0 con 0x7fb4b00ca8f0 2024-02-19T15:22:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4af400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b015af10 0x7fb4b01451d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:16.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4aea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4b00ca8f0 0x7fb4b00c28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:16.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4af400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b015af10 0x7fb4b01451d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57938/0 (socket says 172.21.15.47:57938) 2024-02-19T15:22:16.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4af400640 1 -- 172.21.15.47:0/42710127 learned_addr learned my addr 172.21.15.47:0/42710127 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:16.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4a6000640 1 --2- 172.21.15.47:0/42710127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4b00cb290 0x7fb4b0158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:16.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4af400640 1 -- 172.21.15.47:0/42710127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4b00ca8f0 msgr2=0x7fb4b00c28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:16.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4af400640 1 --2- 172.21.15.47:0/42710127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4b00ca8f0 0x7fb4b00c28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4af400640 1 -- 172.21.15.47:0/42710127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4b00cb290 msgr2=0x7fb4b0158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:16.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4af400640 1 --2- 172.21.15.47:0/42710127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4b00cb290 0x7fb4b0158770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4af400640 1 -- 172.21.15.47:0/42710127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4980094d0 con 0x7fb4b015af10 2024-02-19T15:22:16.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4af400640 1 --2- 172.21.15.47:0/42710127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b015af10 0x7fb4b01451d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb4a800c730 tx=0x7fb4a800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:16.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4a7400640 1 -- 172.21.15.47:0/42710127 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a8003b80 con 0x7fb4b015af10 2024-02-19T15:22:16.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4a7400640 1 -- 172.21.15.47:0/42710127 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a8003ce0 con 0x7fb4b015af10 2024-02-19T15:22:16.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4a7400640 1 -- 172.21.15.47:0/42710127 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a8005310 con 0x7fb4b015af10 2024-02-19T15:22:16.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4b5a00640 1 -- 172.21.15.47:0/42710127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b01594a0 con 0x7fb4b015af10 2024-02-19T15:22:16.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.286+0000 7fb4b5a00640 1 -- 172.21.15.47:0/42710127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b0153bf0 con 0x7fb4b015af10 2024-02-19T15:22:16.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.290+0000 7fb4b5a00640 1 -- 172.21.15.47:0/42710127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b0144eb0 con 0x7fb4b015af10 2024-02-19T15:22:16.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.290+0000 7fb4a7400640 1 -- 172.21.15.47:0/42710127 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb4a8005680 con 0x7fb4b015af10 2024-02-19T15:22:16.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.290+0000 7fb4a7400640 1 --2- 172.21.15.47:0/42710127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb48c066e70 0x7fb48c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:16.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.290+0000 7fb4a7400640 1 -- 172.21.15.47:0/42710127 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4a8084870 con 0x7fb4b015af10 2024-02-19T15:22:16.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.290+0000 7fb4aea00640 1 --2- 172.21.15.47:0/42710127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb48c066e70 0x7fb48c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:16.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.290+0000 7fb4aea00640 1 --2- 172.21.15.47:0/42710127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb48c066e70 0x7fb48c069330 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009b60 tx=0x7fb4a0009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:16.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.294+0000 7fb4a7400640 1 -- 172.21.15.47:0/42710127 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb4a8051700 con 0x7fb4b015af10 2024-02-19T15:22:16.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.458+0000 7fb4b5a00640 1 -- 172.21.15.47:0/42710127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb4b0075360 con 0x7fb4b015af10 2024-02-19T15:22:16.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.458+0000 7fb4a7400640 1 -- 172.21.15.47:0/42710127 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb4a8054dc0 con 0x7fb4b015af10 2024-02-19T15:22:16.459 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:22:16.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.458+0000 7fb4b5a00640 1 -- 172.21.15.47:0/42710127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb48c066e70 msgr2=0x7fb48c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:16.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.458+0000 7fb4b5a00640 1 --2- 172.21.15.47:0/42710127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb48c066e70 0x7fb48c069330 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009b60 tx=0x7fb4a0009190 comp rx=0 tx=0).stop 2024-02-19T15:22:16.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.458+0000 7fb4b5a00640 1 -- 172.21.15.47:0/42710127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b015af10 msgr2=0x7fb4b01451d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:16.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.458+0000 7fb4b5a00640 1 --2- 172.21.15.47:0/42710127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b015af10 0x7fb4b01451d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb4a800c730 tx=0x7fb4a800cbf0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.458+0000 7fb4b5a00640 1 -- 172.21.15.47:0/42710127 shutdown_connections 2024-02-19T15:22:16.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.458+0000 7fb4b5a00640 1 --2- 172.21.15.47:0/42710127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb48c066e70 0x7fb48c069330 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.458+0000 7fb4b5a00640 1 --2- 172.21.15.47:0/42710127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b015af10 0x7fb4b01451d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.458+0000 7fb4b5a00640 1 --2- 172.21.15.47:0/42710127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4b00cb290 0x7fb4b0158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.458+0000 7fb4b5a00640 1 --2- 172.21.15.47:0/42710127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4b00ca8f0 0x7fb4b00c28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.458+0000 7fb4b5a00640 1 -- 172.21.15.47:0/42710127 >> 172.21.15.47:0/42710127 conn(0x7fb4b00cfc20 msgr2=0x7fb4b00d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:16.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.458+0000 7fb4b5a00640 1 -- 172.21.15.47:0/42710127 shutdown_connections 2024-02-19T15:22:16.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.458+0000 7fb4b5a00640 1 -- 172.21.15.47:0/42710127 wait complete. 2024-02-19T15:22:16.481 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:22:16.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.622+0000 7f5e52200640 1 Processor -- start 2024-02-19T15:22:16.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.622+0000 7f5e52200640 1 -- start start 2024-02-19T15:22:16.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.622+0000 7f5e52200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5e4c0c3100 0x7f5e4c0cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:16.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.622+0000 7f5e52200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5e4c0ca270 0x7f5e4c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:16.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.622+0000 7f5e52200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e4c06d420 con 0x7f5e4c0cd7f0 2024-02-19T15:22:16.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.622+0000 7f5e52200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e4c0cc4e0 con 0x7f5e4c0c3100 2024-02-19T15:22:16.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.622+0000 7f5e52200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e4c06caf0 con 0x7f5e4c0ca270 2024-02-19T15:22:16.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.622+0000 7f5e4b400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5e4c0c3100 0x7f5e4c0cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:16.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.622+0000 7f5e4b400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5e4c0c3100 0x7f5e4c0cbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57944/0 (socket says 172.21.15.47:57944) 2024-02-19T15:22:16.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.622+0000 7f5e4b400640 1 -- 172.21.15.47:0/3958464575 learned_addr learned my addr 172.21.15.47:0/3958464575 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:16.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.622+0000 7f5e4be00640 1 --1- 172.21.15.47:0/3958464575 >> v1:172.21.15.47:6789/0 conn(0x7f5e4c0cd7f0 0x7f5e4c06d8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35404/0 (socket says 172.21.15.47:35404) 2024-02-19T15:22:16.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.622+0000 7f5e4aa00640 1 --2- 172.21.15.47:0/3958464575 >> v2:172.21.15.116:3300/0 conn(0x7f5e4c0ca270 0x7f5e4c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:16.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.622+0000 7f5e4b400640 1 -- 172.21.15.47:0/3958464575 >> v2:172.21.15.116:3300/0 conn(0x7f5e4c0ca270 msgr2=0x7f5e4c070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:16.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.622+0000 7f5e4b400640 1 --2- 172.21.15.47:0/3958464575 >> v2:172.21.15.116:3300/0 conn(0x7f5e4c0ca270 0x7f5e4c070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.622+0000 7f5e4b400640 1 -- 172.21.15.47:0/3958464575 >> v1:172.21.15.47:6789/0 conn(0x7f5e4c0cd7f0 legacy=0x7f5e4c06d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:16.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.622+0000 7f5e4b400640 1 -- 172.21.15.47:0/3958464575 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4c06cdd0 con 0x7f5e4c0c3100 2024-02-19T15:22:16.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.622+0000 7f5e4a000640 1 -- 172.21.15.47:0/3958464575 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806602808 0 0) 0x7f5e4c06d420 con 0x7f5e4c0cd7f0 2024-02-19T15:22:16.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.622+0000 7f5e4aa00640 1 --2- 172.21.15.47:0/3958464575 >> v2:172.21.15.116:3300/0 conn(0x7f5e4c0ca270 0x7f5e4c070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:16.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.626+0000 7f5e4b400640 1 --2- 172.21.15.47:0/3958464575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e4c0c3100 0x7f5e4c0cbc40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c073290 tx=0x7f5e3c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=789431eeb0658004 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:16.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.626+0000 7f5e4a000640 1 -- 172.21.15.47:0/3958464575 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c0ab600 con 0x7f5e4c0c3100 2024-02-19T15:22:16.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.626+0000 7f5e4a000640 1 -- 172.21.15.47:0/3958464575 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3c0a7050 con 0x7f5e4c0c3100 2024-02-19T15:22:16.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.626+0000 7f5e4a000640 1 -- 172.21.15.47:0/3958464575 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c0ab980 con 0x7f5e4c0c3100 2024-02-19T15:22:16.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.626+0000 7f5e52200640 1 -- 172.21.15.47:0/3958464575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e4c0c3100 msgr2=0x7f5e4c0cbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:16.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.626+0000 7f5e52200640 1 --2- 172.21.15.47:0/3958464575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e4c0c3100 0x7f5e4c0cbc40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c073290 tx=0x7f5e3c01c460 comp rx=0 tx=0).stop 2024-02-19T15:22:16.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.626+0000 7f5e52200640 1 -- 172.21.15.47:0/3958464575 shutdown_connections 2024-02-19T15:22:16.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.626+0000 7f5e52200640 1 --2- 172.21.15.47:0/3958464575 >> v2:172.21.15.116:3300/0 conn(0x7f5e4c0ca270 0x7f5e4c070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.626+0000 7f5e52200640 1 --2- 172.21.15.47:0/3958464575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e4c0c3100 0x7f5e4c0cbc40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.626+0000 7f5e52200640 1 -- 172.21.15.47:0/3958464575 >> 172.21.15.47:0/3958464575 conn(0x7f5e4c0cfc20 msgr2=0x7f5e4c0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:16.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.626+0000 7f5e52200640 1 -- 172.21.15.47:0/3958464575 shutdown_connections 2024-02-19T15:22:16.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.626+0000 7f5e52200640 1 -- 172.21.15.47:0/3958464575 wait complete. 2024-02-19T15:22:16.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.626+0000 7f5e52200640 1 Processor -- start 2024-02-19T15:22:16.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.626+0000 7f5e52200640 1 -- start start 2024-02-19T15:22:16.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e52200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e4c0c3100 0x7f5e4c0c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:16.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e52200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e4c0ca270 0x7f5e4c0c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:16.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e52200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e4c0cd7f0 0x7f5e4c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:16.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e52200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c0c7700 con 0x7f5e4c0cd7f0 2024-02-19T15:22:16.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e52200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c0c9c00 con 0x7f5e4c0c3100 2024-02-19T15:22:16.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e52200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c0c7950 con 0x7f5e4c0ca270 2024-02-19T15:22:16.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e4be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e4c0cd7f0 0x7f5e4c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:16.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e4be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e4c0cd7f0 0x7f5e4c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57950/0 (socket says 172.21.15.47:57950) 2024-02-19T15:22:16.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e4be00640 1 -- 172.21.15.47:0/1171790988 learned_addr learned my addr 172.21.15.47:0/1171790988 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:16.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e4aa00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e4c0ca270 0x7f5e4c0c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:16.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e4b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e4c0c3100 0x7f5e4c0c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:16.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e4be00640 1 -- 172.21.15.47:0/1171790988 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e4c0ca270 msgr2=0x7f5e4c0c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:16.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e4be00640 1 --2- 172.21.15.47:0/1171790988 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e4c0ca270 0x7f5e4c0c5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e4be00640 1 -- 172.21.15.47:0/1171790988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e4c0c3100 msgr2=0x7f5e4c0c96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:16.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e4be00640 1 --2- 172.21.15.47:0/1171790988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e4c0c3100 0x7f5e4c0c96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e4be00640 1 -- 172.21.15.47:0/1171790988 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e38009420 con 0x7f5e4c0cd7f0 2024-02-19T15:22:16.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e4be00640 1 --2- 172.21.15.47:0/1171790988 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e4c0cd7f0 0x7f5e4c15d6f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5e4400c730 tx=0x7f5e4400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:16.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e43400640 1 -- 172.21.15.47:0/1171790988 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4400ce40 con 0x7f5e4c0cd7f0 2024-02-19T15:22:16.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e43400640 1 -- 172.21.15.47:0/1171790988 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e4400a640 con 0x7f5e4c0cd7f0 2024-02-19T15:22:16.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e43400640 1 -- 172.21.15.47:0/1171790988 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4400aa30 con 0x7f5e4c0cd7f0 2024-02-19T15:22:16.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e52200640 1 -- 172.21.15.47:0/1171790988 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3c0094d0 con 0x7f5e4c0cd7f0 2024-02-19T15:22:16.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e52200640 1 -- 172.21.15.47:0/1171790988 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e4c0ce190 con 0x7f5e4c0cd7f0 2024-02-19T15:22:16.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e52200640 1 -- 172.21.15.47:0/1171790988 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e4c075360 con 0x7f5e4c0cd7f0 2024-02-19T15:22:16.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.630+0000 7f5e43400640 1 -- 172.21.15.47:0/1171790988 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5e4400ab90 con 0x7f5e4c0cd7f0 2024-02-19T15:22:16.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.634+0000 7f5e43400640 1 --2- 172.21.15.47:0/1171790988 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5e2c066e40 0x7f5e2c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:16.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.634+0000 7f5e43400640 1 -- 172.21.15.47:0/1171790988 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e44083ec0 con 0x7f5e4c0cd7f0 2024-02-19T15:22:16.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.634+0000 7f5e4b400640 1 --2- 172.21.15.47:0/1171790988 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5e2c066e40 0x7f5e2c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:16.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.634+0000 7f5e4b400640 1 --2- 172.21.15.47:0/1171790988 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5e2c066e40 0x7f5e2c069300 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c0ad000 tx=0x7f5e3c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:16.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.638+0000 7f5e43400640 1 -- 172.21.15.47:0/1171790988 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f5e44050cc0 con 0x7f5e4c0cd7f0 2024-02-19T15:22:16.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.798+0000 7f5e52200640 1 -- 172.21.15.47:0/1171790988 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5e4c0c6ab0 con 0x7f5e4c0cd7f0 2024-02-19T15:22:16.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.798+0000 7f5e43400640 1 -- 172.21.15.47:0/1171790988 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5e44018070 con 0x7f5e4c0cd7f0 2024-02-19T15:22:16.800 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:22:16.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.802+0000 7f5e52200640 1 -- 172.21.15.47:0/1171790988 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5e2c066e40 msgr2=0x7f5e2c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:16.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.802+0000 7f5e52200640 1 --2- 172.21.15.47:0/1171790988 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5e2c066e40 0x7f5e2c069300 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c0ad000 tx=0x7f5e3c008040 comp rx=0 tx=0).stop 2024-02-19T15:22:16.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.802+0000 7f5e52200640 1 -- 172.21.15.47:0/1171790988 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e4c0cd7f0 msgr2=0x7f5e4c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:16.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.802+0000 7f5e52200640 1 --2- 172.21.15.47:0/1171790988 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e4c0cd7f0 0x7f5e4c15d6f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5e4400c730 tx=0x7f5e4400cbf0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.802+0000 7f5e52200640 1 -- 172.21.15.47:0/1171790988 shutdown_connections 2024-02-19T15:22:16.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.802+0000 7f5e52200640 1 --2- 172.21.15.47:0/1171790988 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5e2c066e40 0x7f5e2c069300 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.802+0000 7f5e52200640 1 --2- 172.21.15.47:0/1171790988 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e4c0cd7f0 0x7f5e4c15d6f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.802+0000 7f5e52200640 1 --2- 172.21.15.47:0/1171790988 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e4c0ca270 0x7f5e4c0c5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.802+0000 7f5e52200640 1 --2- 172.21.15.47:0/1171790988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e4c0c3100 0x7f5e4c0c96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.802+0000 7f5e52200640 1 -- 172.21.15.47:0/1171790988 >> 172.21.15.47:0/1171790988 conn(0x7f5e4c0cfc20 msgr2=0x7f5e4c06d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:16.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.802+0000 7f5e52200640 1 -- 172.21.15.47:0/1171790988 shutdown_connections 2024-02-19T15:22:16.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.802+0000 7f5e52200640 1 -- 172.21.15.47:0/1171790988 wait complete. 2024-02-19T15:22:16.821 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:22:16.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.958+0000 7f93c9c00640 1 Processor -- start 2024-02-19T15:22:16.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.958+0000 7f93c9c00640 1 -- start start 2024-02-19T15:22:16.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.962+0000 7f93c9c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93c40b2e20 0x7f93c40ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:16.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.962+0000 7f93c9c00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93c40d2ad0 con 0x7f93c40ce3c0 2024-02-19T15:22:16.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.962+0000 7f93c9c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93c40cc190 con 0x7f93c415e570 2024-02-19T15:22:16.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.962+0000 7f93c9c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c40d0ac0 con 0x7f93c40b2e20 2024-02-19T15:22:16.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.962+0000 7f93c2a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f93c40ce3c0 0x7f93c40ce7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35412/0 (socket says 172.21.15.47:35412) 2024-02-19T15:22:16.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.962+0000 7f93c2a00640 1 -- 172.21.15.47:0/3336528675 learned_addr learned my addr 172.21.15.47:0/3336528675 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:16.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.962+0000 7f93c3400640 1 --2- 172.21.15.47:0/3336528675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93c40b2e20 0x7f93c40ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:16.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.962+0000 7f93c2000640 1 -- 172.21.15.47:0/3336528675 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026345179 0 0) 0x7f93c40cc190 con 0x7f93c415e570 2024-02-19T15:22:16.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.962+0000 7f93c2000640 1 -- 172.21.15.47:0/3336528675 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93a8003530 con 0x7f93c415e570 2024-02-19T15:22:16.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.962+0000 7f93c2000640 1 -- 172.21.15.47:0/3336528675 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562440274 0 0) 0x7f93b4001120 con 0x7f93c40ce3c0 2024-02-19T15:22:16.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.962+0000 7f93c2000640 1 -- 172.21.15.47:0/3336528675 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93c40cc190 con 0x7f93c40ce3c0 2024-02-19T15:22:16.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.962+0000 7f93c3400640 1 -- 172.21.15.47:0/3336528675 >> v1:172.21.15.116:6789/0 conn(0x7f93c415e570 legacy=0x7f93c40cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:16.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.962+0000 7f93c3400640 1 -- 172.21.15.47:0/3336528675 >> v1:172.21.15.47:6789/0 conn(0x7f93c40ce3c0 legacy=0x7f93c40ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:16.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.962+0000 7f93c3400640 1 -- 172.21.15.47:0/3336528675 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c40d0da0 con 0x7f93c40b2e20 2024-02-19T15:22:16.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.966+0000 7f93c3400640 1 --2- 172.21.15.47:0/3336528675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93c40b2e20 0x7f93c40ccd30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009dc0 tx=0x7f93bc01c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b2f6d43a0be86 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:16.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.966+0000 7f93c2000640 1 -- 172.21.15.47:0/3336528675 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c40cf510 con 0x7f93c40b2e20 2024-02-19T15:22:16.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.966+0000 7f93c2000640 1 -- 172.21.15.47:0/3336528675 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93bc0a5960 con 0x7f93c40b2e20 2024-02-19T15:22:16.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.966+0000 7f93c2000640 1 -- 172.21.15.47:0/3336528675 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93bc0a5cf0 con 0x7f93c40b2e20 2024-02-19T15:22:16.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.966+0000 7f93c9c00640 1 -- 172.21.15.47:0/3336528675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93c40b2e20 msgr2=0x7f93c40ccd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:16.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.966+0000 7f93c9c00640 1 --2- 172.21.15.47:0/3336528675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93c40b2e20 0x7f93c40ccd30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009dc0 tx=0x7f93bc01c990 comp rx=0 tx=0).stop 2024-02-19T15:22:16.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.966+0000 7f93c9c00640 1 -- 172.21.15.47:0/3336528675 shutdown_connections 2024-02-19T15:22:16.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.966+0000 7f93c9c00640 1 --2- 172.21.15.47:0/3336528675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93c40b2e20 0x7f93c40ccd30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.966+0000 7f93c9c00640 1 -- 172.21.15.47:0/3336528675 >> 172.21.15.47:0/3336528675 conn(0x7f93c40d1b40 msgr2=0x7f93c40d1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:16.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.966+0000 7f93c9c00640 1 -- 172.21.15.47:0/3336528675 shutdown_connections 2024-02-19T15:22:16.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.966+0000 7f93c9c00640 1 -- 172.21.15.47:0/3336528675 wait complete. 2024-02-19T15:22:16.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.966+0000 7f93c9c00640 1 Processor -- start 2024-02-19T15:22:16.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.966+0000 7f93c9c00640 1 -- start start 2024-02-19T15:22:16.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93c9c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93c40b2e20 0x7f93c414f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:16.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93c9c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c40ce3c0 0x7f93c414c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:16.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93c9c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93c415e570 0x7f93c4160c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:16.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93c9c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c4161b00 con 0x7f93c40b2e20 2024-02-19T15:22:16.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93c9c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c414cba0 con 0x7f93c40ce3c0 2024-02-19T15:22:16.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93c9c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c4161540 con 0x7f93c415e570 2024-02-19T15:22:16.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93c2a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93c40b2e20 0x7f93c414f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:16.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93c2a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93c40b2e20 0x7f93c414f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57958/0 (socket says 172.21.15.47:57958) 2024-02-19T15:22:16.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93c2a00640 1 -- 172.21.15.47:0/1997780599 learned_addr learned my addr 172.21.15.47:0/1997780599 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:16.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93c3400640 1 --2- 172.21.15.47:0/1997780599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93c415e570 0x7f93c4160c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:16.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93bbe00640 1 --2- 172.21.15.47:0/1997780599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c40ce3c0 0x7f93c414c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:16.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93c2a00640 1 -- 172.21.15.47:0/1997780599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93c415e570 msgr2=0x7f93c4160c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93c2a00640 1 --2- 172.21.15.47:0/1997780599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93c415e570 0x7f93c4160c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93c2a00640 1 -- 172.21.15.47:0/1997780599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c40ce3c0 msgr2=0x7f93c414c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93c2a00640 1 --2- 172.21.15.47:0/1997780599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c40ce3c0 0x7f93c414c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93c2a00640 1 -- 172.21.15.47:0/1997780599 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93bc009a70 con 0x7f93c40b2e20 2024-02-19T15:22:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93c2a00640 1 --2- 172.21.15.47:0/1997780599 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93c40b2e20 0x7f93c414f330 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f93b4005890 tx=0x7f93b400c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93baa00640 1 -- 172.21.15.47:0/1997780599 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b4005cf0 con 0x7f93c40b2e20 2024-02-19T15:22:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93baa00640 1 -- 172.21.15.47:0/1997780599 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b4005e50 con 0x7f93c40b2e20 2024-02-19T15:22:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93c9c00640 1 -- 172.21.15.47:0/1997780599 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c4161820 con 0x7f93c40b2e20 2024-02-19T15:22:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93baa00640 1 -- 172.21.15.47:0/1997780599 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b4007290 con 0x7f93c40b2e20 2024-02-19T15:22:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93c9c00640 1 -- 172.21.15.47:0/1997780599 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93c40c99d0 con 0x7f93c40b2e20 2024-02-19T15:22:16.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93c9c00640 1 -- 172.21.15.47:0/1997780599 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c40cbf50 con 0x7f93c40b2e20 2024-02-19T15:22:16.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.970+0000 7f93baa00640 1 -- 172.21.15.47:0/1997780599 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f93b40073f0 con 0x7f93c40b2e20 2024-02-19T15:22:16.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.974+0000 7f93baa00640 1 --2- 172.21.15.47:0/1997780599 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f93a0066e40 0x7f93a0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:16.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.974+0000 7f93baa00640 1 -- 172.21.15.47:0/1997780599 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f93b4083e10 con 0x7f93c40b2e20 2024-02-19T15:22:16.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.974+0000 7f93bbe00640 1 --2- 172.21.15.47:0/1997780599 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f93a0066e40 0x7f93a0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:16.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.974+0000 7f93bbe00640 1 --2- 172.21.15.47:0/1997780599 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f93a0066e40 0x7f93a0069300 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f93c40b2df0 tx=0x7f93ac009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:16.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:16.978+0000 7f93baa00640 1 -- 172.21.15.47:0/1997780599 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f93b4087030 con 0x7f93c40b2e20 2024-02-19T15:22:17.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.174+0000 7f93c9c00640 1 -- 172.21.15.47:0/1997780599 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93c40d2ad0 con 0x7f93c40b2e20 2024-02-19T15:22:17.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.174+0000 7f93baa00640 1 -- 172.21.15.47:0/1997780599 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4935 (secure 0 0 0) 0x7f93c40d2ad0 con 0x7f93c40b2e20 2024-02-19T15:22:17.177 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:17.178 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3873760209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3873760209},{"type":"v1","addr":"172.21.15.116:6836","nonce":3873760209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/284820350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":284820350},{"type":"v1","addr":"172.21.15.47:6834","nonce":284820350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2764089359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2764089359},{"type":"v1","addr":"172.21.15.47:6835","nonce":2764089359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:22:05.228517+0000","modified":"2024-02-19T15:22:08.242600+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6837/3920452343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3920452343},{"type":"v1","addr":"172.21.15.116:6837","nonce":3920452343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-19T15:22:17.178 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 8 2024-02-19T15:22:17.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.178+0000 7f93c9c00640 1 -- 172.21.15.47:0/1997780599 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f93a0066e40 msgr2=0x7f93a0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:17.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.178+0000 7f93c9c00640 1 --2- 172.21.15.47:0/1997780599 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f93a0066e40 0x7f93a0069300 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f93c40b2df0 tx=0x7f93ac009040 comp rx=0 tx=0).stop 2024-02-19T15:22:17.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.178+0000 7f93c9c00640 1 -- 172.21.15.47:0/1997780599 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93c40b2e20 msgr2=0x7f93c414f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:17.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.178+0000 7f93c9c00640 1 --2- 172.21.15.47:0/1997780599 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93c40b2e20 0x7f93c414f330 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f93b4005890 tx=0x7f93b400c980 comp rx=0 tx=0).stop 2024-02-19T15:22:17.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.178+0000 7f93c9c00640 1 -- 172.21.15.47:0/1997780599 shutdown_connections 2024-02-19T15:22:17.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.178+0000 7f93c9c00640 1 --2- 172.21.15.47:0/1997780599 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f93a0066e40 0x7f93a0069300 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.178+0000 7f93c9c00640 1 --2- 172.21.15.47:0/1997780599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93c415e570 0x7f93c4160c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.178+0000 7f93c9c00640 1 --2- 172.21.15.47:0/1997780599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c40ce3c0 0x7f93c414c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.178+0000 7f93c9c00640 1 --2- 172.21.15.47:0/1997780599 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93c40b2e20 0x7f93c414f330 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.178+0000 7f93c9c00640 1 -- 172.21.15.47:0/1997780599 >> 172.21.15.47:0/1997780599 conn(0x7f93c40d1b40 msgr2=0x7f93c415d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:17.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.178+0000 7f93c9c00640 1 -- 172.21.15.47:0/1997780599 shutdown_connections 2024-02-19T15:22:17.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.178+0000 7f93c9c00640 1 -- 172.21.15.47:0/1997780599 wait complete. 2024-02-19T15:22:17.198 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:17.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.334+0000 7f0035400640 1 Processor -- start 2024-02-19T15:22:17.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.334+0000 7f0035400640 1 -- start start 2024-02-19T15:22:17.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.338+0000 7f0035400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f003006d650 0x7f00300d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:17.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.338+0000 7f0035400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f00300cded0 0x7f00300ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:17.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.338+0000 7f0035400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00300ceb40 0x7f0030161d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:17.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.338+0000 7f0035400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00300cefd0 con 0x7f003006d650 2024-02-19T15:22:17.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.338+0000 7f0035400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00300cce30 con 0x7f00300cded0 2024-02-19T15:22:17.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.338+0000 7f0035400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00300cd110 con 0x7f00300ceb40 2024-02-19T15:22:17.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.338+0000 7f002e800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f003006d650 0x7f00300d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:17.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.338+0000 7f002e800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f003006d650 0x7f00300d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57964/0 (socket says 172.21.15.47:57964) 2024-02-19T15:22:17.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.338+0000 7f002e800640 1 -- 172.21.15.47:0/4286450476 learned_addr learned my addr 172.21.15.47:0/4286450476 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:17.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.338+0000 7f002f200640 1 --2- 172.21.15.47:0/4286450476 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00300ceb40 0x7f0030161d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:17.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.338+0000 7f0027e00640 1 --2- 172.21.15.47:0/4286450476 >> v2:172.21.15.116:3300/0 conn(0x7f00300cded0 0x7f00300ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:17.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.338+0000 7f002e800640 1 -- 172.21.15.47:0/4286450476 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00300ceb40 msgr2=0x7f0030161d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:17.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.338+0000 7f002e800640 1 --2- 172.21.15.47:0/4286450476 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00300ceb40 0x7f0030161d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.338+0000 7f002e800640 1 -- 172.21.15.47:0/4286450476 >> v2:172.21.15.116:3300/0 conn(0x7f00300cded0 msgr2=0x7f00300ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:17.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.338+0000 7f002e800640 1 --2- 172.21.15.47:0/4286450476 >> v2:172.21.15.116:3300/0 conn(0x7f00300cded0 0x7f00300ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.338+0000 7f002e800640 1 -- 172.21.15.47:0/4286450476 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00300cd3f0 con 0x7f003006d650 2024-02-19T15:22:17.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.338+0000 7f0027e00640 1 --2- 172.21.15.47:0/4286450476 >> v2:172.21.15.116:3300/0 conn(0x7f00300cded0 0x7f00300ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:22:17.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f002e800640 1 --2- 172.21.15.47:0/4286450476 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f003006d650 0x7f00300d08b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f003006b950 tx=0x7f002001c1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5464405c31eb4d67 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:17.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f002de00640 1 -- 172.21.15.47:0/4286450476 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00200ab600 con 0x7f003006d650 2024-02-19T15:22:17.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f002de00640 1 -- 172.21.15.47:0/4286450476 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00200a7050 con 0x7f003006d650 2024-02-19T15:22:17.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f002de00640 1 -- 172.21.15.47:0/4286450476 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00200ab980 con 0x7f003006d650 2024-02-19T15:22:17.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f0035400640 1 -- 172.21.15.47:0/4286450476 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f003006d650 msgr2=0x7f00300d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:17.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f0035400640 1 --2- 172.21.15.47:0/4286450476 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f003006d650 0x7f00300d08b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f003006b950 tx=0x7f002001c1a0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f0035400640 1 -- 172.21.15.47:0/4286450476 shutdown_connections 2024-02-19T15:22:17.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f0035400640 1 --2- 172.21.15.47:0/4286450476 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00300ceb40 0x7f0030161d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f0035400640 1 --2- 172.21.15.47:0/4286450476 >> v2:172.21.15.116:3300/0 conn(0x7f00300cded0 0x7f00300ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f0035400640 1 --2- 172.21.15.47:0/4286450476 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f003006d650 0x7f00300d08b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f0035400640 1 -- 172.21.15.47:0/4286450476 >> 172.21.15.47:0/4286450476 conn(0x7f00300d1bf0 msgr2=0x7f00300d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:17.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f0035400640 1 -- 172.21.15.47:0/4286450476 shutdown_connections 2024-02-19T15:22:17.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f0035400640 1 -- 172.21.15.47:0/4286450476 wait complete. 2024-02-19T15:22:17.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f0035400640 1 Processor -- start 2024-02-19T15:22:17.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f0035400640 1 -- start start 2024-02-19T15:22:17.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f0035400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003006d650 0x7f003014d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:17.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f0035400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00300cded0 0x7f00300b5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:17.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f0035400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00300ceb40 0x7f00300b5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:17.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f0035400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00300b6fb0 con 0x7f00300ceb40 2024-02-19T15:22:17.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f0035400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00300b63a0 con 0x7f003006d650 2024-02-19T15:22:17.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.342+0000 7f0035400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00300b6680 con 0x7f00300cded0 2024-02-19T15:22:17.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.346+0000 7f002f200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00300ceb40 0x7f00300b5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:17.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.346+0000 7f002f200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00300ceb40 0x7f00300b5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57974/0 (socket says 172.21.15.47:57974) 2024-02-19T15:22:17.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.346+0000 7f002f200640 1 -- 172.21.15.47:0/4083114574 learned_addr learned my addr 172.21.15.47:0/4083114574 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:17.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.346+0000 7f0027e00640 1 --2- 172.21.15.47:0/4083114574 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00300cded0 0x7f00300b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:17.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.346+0000 7f002e800640 1 --2- 172.21.15.47:0/4083114574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003006d650 0x7f003014d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:17.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.346+0000 7f002f200640 1 -- 172.21.15.47:0/4083114574 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00300cded0 msgr2=0x7f00300b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:17.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.346+0000 7f002f200640 1 --2- 172.21.15.47:0/4083114574 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00300cded0 0x7f00300b5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.346+0000 7f002f200640 1 -- 172.21.15.47:0/4083114574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003006d650 msgr2=0x7f003014d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:17.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.346+0000 7f002f200640 1 --2- 172.21.15.47:0/4083114574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003006d650 0x7f003014d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.346+0000 7f002f200640 1 -- 172.21.15.47:0/4083114574 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00200094d0 con 0x7f00300ceb40 2024-02-19T15:22:17.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.346+0000 7f002e800640 1 --2- 172.21.15.47:0/4083114574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003006d650 0x7f003014d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:17.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.346+0000 7f002f200640 1 --2- 172.21.15.47:0/4083114574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00300ceb40 0x7f00300b5d90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0028007e60 tx=0x7f002800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:17.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.346+0000 7f0026a00640 1 -- 172.21.15.47:0/4083114574 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0028014420 con 0x7f00300ceb40 2024-02-19T15:22:17.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.346+0000 7f0026a00640 1 -- 172.21.15.47:0/4083114574 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00280145a0 con 0x7f00300ceb40 2024-02-19T15:22:17.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.346+0000 7f0035400640 1 -- 172.21.15.47:0/4083114574 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00300b6960 con 0x7f00300ceb40 2024-02-19T15:22:17.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.346+0000 7f0026a00640 1 -- 172.21.15.47:0/4083114574 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0028014950 con 0x7f00300ceb40 2024-02-19T15:22:17.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.346+0000 7f0035400640 1 -- 172.21.15.47:0/4083114574 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f003014dbc0 con 0x7f00300ceb40 2024-02-19T15:22:17.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.346+0000 7f0035400640 1 -- 172.21.15.47:0/4083114574 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0030075360 con 0x7f00300ceb40 2024-02-19T15:22:17.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.346+0000 7f0026a00640 1 -- 172.21.15.47:0/4083114574 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0028014ab0 con 0x7f00300ceb40 2024-02-19T15:22:17.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.350+0000 7f0026a00640 1 --2- 172.21.15.47:0/4083114574 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f000c066e40 0x7f000c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:17.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.350+0000 7f0026a00640 1 -- 172.21.15.47:0/4083114574 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00280852d0 con 0x7f00300ceb40 2024-02-19T15:22:17.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.350+0000 7f002e800640 1 --2- 172.21.15.47:0/4083114574 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f000c066e40 0x7f000c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:17.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.350+0000 7f002e800640 1 --2- 172.21.15.47:0/4083114574 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f000c066e40 0x7f000c069300 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f00200ad000 tx=0x7f0020008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:17.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.354+0000 7f0026a00640 1 -- 172.21.15.47:0/4083114574 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f00280520d0 con 0x7f00300ceb40 2024-02-19T15:22:17.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.514+0000 7f0035400640 1 -- 172.21.15.47:0/4083114574 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00300cf580 con 0x7f00300ceb40 2024-02-19T15:22:17.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.514+0000 7f0026a00640 1 -- 172.21.15.47:0/4083114574 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f0028055790 con 0x7f00300ceb40 2024-02-19T15:22:17.517 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:17.517 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":21,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:06.843155+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:03.091360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:04.222965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:17.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.518+0000 7f0035400640 1 -- 172.21.15.47:0/4083114574 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f000c066e40 msgr2=0x7f000c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:17.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.518+0000 7f0035400640 1 --2- 172.21.15.47:0/4083114574 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f000c066e40 0x7f000c069300 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f00200ad000 tx=0x7f0020008040 comp rx=0 tx=0).stop 2024-02-19T15:22:17.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.518+0000 7f0035400640 1 -- 172.21.15.47:0/4083114574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00300ceb40 msgr2=0x7f00300b5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:17.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.518+0000 7f0035400640 1 --2- 172.21.15.47:0/4083114574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00300ceb40 0x7f00300b5d90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0028007e60 tx=0x7f002800e4b0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.518+0000 7f0035400640 1 -- 172.21.15.47:0/4083114574 shutdown_connections 2024-02-19T15:22:17.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.518+0000 7f0035400640 1 --2- 172.21.15.47:0/4083114574 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f000c066e40 0x7f000c069300 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.518+0000 7f0035400640 1 --2- 172.21.15.47:0/4083114574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00300ceb40 0x7f00300b5d90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.518+0000 7f0035400640 1 --2- 172.21.15.47:0/4083114574 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00300cded0 0x7f00300b5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.518+0000 7f0035400640 1 --2- 172.21.15.47:0/4083114574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003006d650 0x7f003014d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.518+0000 7f0035400640 1 -- 172.21.15.47:0/4083114574 >> 172.21.15.47:0/4083114574 conn(0x7f00300d1bf0 msgr2=0x7f003006c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:17.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.518+0000 7f0035400640 1 -- 172.21.15.47:0/4083114574 shutdown_connections 2024-02-19T15:22:17.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.518+0000 7f0035400640 1 -- 172.21.15.47:0/4083114574 wait complete. 2024-02-19T15:22:17.538 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:22:03.091360+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:22:04.222965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:22:17.539 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:22:17.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.678+0000 7f019b600640 1 Processor -- start 2024-02-19T15:22:17.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.678+0000 7f019b600640 1 -- start start 2024-02-19T15:22:17.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.682+0000 7f019b600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f019415bd90 0x7f0194070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:17.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.682+0000 7f019b600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f01940cb490 0x7f0194072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:17.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.682+0000 7f019b600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019415fc90 con 0x7f0194072680 2024-02-19T15:22:17.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.682+0000 7f019b600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f019415ff70 con 0x7f01940cb490 2024-02-19T15:22:17.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.682+0000 7f019b600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0194160250 con 0x7f019415bd90 2024-02-19T15:22:17.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.682+0000 7f018be00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f01940cb490 0x7f0194072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:17.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.682+0000 7f0198c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f019415bd90 0x7f0194070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:17.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.682+0000 7f018be00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f01940cb490 0x7f0194072140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:36848/0 (socket says 172.21.15.47:36848) 2024-02-19T15:22:17.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.682+0000 7f018be00640 1 -- 172.21.15.47:0/2292591546 learned_addr learned my addr 172.21.15.47:0/2292591546 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:17.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.682+0000 7f0198c00640 1 -- 172.21.15.47:0/2292591546 >> v2:172.21.15.116:3300/0 conn(0x7f01940cb490 msgr2=0x7f0194072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:17.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.682+0000 7f0198c00640 1 --2- 172.21.15.47:0/2292591546 >> v2:172.21.15.116:3300/0 conn(0x7f01940cb490 0x7f0194072140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.682+0000 7f0198c00640 1 -- 172.21.15.47:0/2292591546 >> v1:172.21.15.47:6789/0 conn(0x7f0194072680 legacy=0x7f01940ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:17.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.682+0000 7f0198c00640 1 -- 172.21.15.47:0/2292591546 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0194160530 con 0x7f019415bd90 2024-02-19T15:22:17.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f0198c00640 1 --2- 172.21.15.47:0/2292591546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f019415bd90 0x7f0194070960 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f01840097c0 tx=0x7f018401c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbeccabd1e34b34c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:17.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f0193e00640 1 -- 172.21.15.47:0/2292591546 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01840ab600 con 0x7f019415bd90 2024-02-19T15:22:17.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f0193e00640 1 -- 172.21.15.47:0/2292591546 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01840a7050 con 0x7f019415bd90 2024-02-19T15:22:17.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f0193e00640 1 -- 172.21.15.47:0/2292591546 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01840ab980 con 0x7f019415bd90 2024-02-19T15:22:17.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f019b600640 1 -- 172.21.15.47:0/2292591546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f019415bd90 msgr2=0x7f0194070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:17.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f019b600640 1 --2- 172.21.15.47:0/2292591546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f019415bd90 0x7f0194070960 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f01840097c0 tx=0x7f018401c340 comp rx=0 tx=0).stop 2024-02-19T15:22:17.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f019b600640 1 -- 172.21.15.47:0/2292591546 shutdown_connections 2024-02-19T15:22:17.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f019b600640 1 --2- 172.21.15.47:0/2292591546 >> v2:172.21.15.116:3300/0 conn(0x7f01940cb490 0x7f0194072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f019b600640 1 --2- 172.21.15.47:0/2292591546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f019415bd90 0x7f0194070960 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f019b600640 1 -- 172.21.15.47:0/2292591546 >> 172.21.15.47:0/2292591546 conn(0x7f01940cfc20 msgr2=0x7f01940d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:17.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f019b600640 1 -- 172.21.15.47:0/2292591546 shutdown_connections 2024-02-19T15:22:17.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f019b600640 1 -- 172.21.15.47:0/2292591546 wait complete. 2024-02-19T15:22:17.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f019b600640 1 Processor -- start 2024-02-19T15:22:17.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f019b600640 1 -- start start 2024-02-19T15:22:17.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f019b600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0194072680 0x7f01940b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:17.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f019b600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01940cb490 0x7f0194154890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:17.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f019b600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f019415bd90 0x7f01940c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:17.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f019b600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0194155de0 con 0x7f0194072680 2024-02-19T15:22:17.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f019b600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019415c420 con 0x7f01940cb490 2024-02-19T15:22:17.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f019b600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019415c700 con 0x7f019415bd90 2024-02-19T15:22:17.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.686+0000 7f0198c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0194072680 0x7f01940b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:17.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.690+0000 7f0198c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0194072680 0x7f01940b4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57978/0 (socket says 172.21.15.47:57978) 2024-02-19T15:22:17.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.690+0000 7f0198c00640 1 -- 172.21.15.47:0/2718131686 learned_addr learned my addr 172.21.15.47:0/2718131686 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:17.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.690+0000 7f0199600640 1 --2- 172.21.15.47:0/2718131686 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f019415bd90 0x7f01940c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:17.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.690+0000 7f018be00640 1 --2- 172.21.15.47:0/2718131686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01940cb490 0x7f0194154890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:17.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.690+0000 7f0198c00640 1 -- 172.21.15.47:0/2718131686 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f019415bd90 msgr2=0x7f01940c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:17.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.690+0000 7f0198c00640 1 --2- 172.21.15.47:0/2718131686 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f019415bd90 0x7f01940c94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.690+0000 7f0198c00640 1 -- 172.21.15.47:0/2718131686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01940cb490 msgr2=0x7f0194154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:17.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.690+0000 7f0198c00640 1 --2- 172.21.15.47:0/2718131686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01940cb490 0x7f0194154890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.690+0000 7f0198c00640 1 -- 172.21.15.47:0/2718131686 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0184009420 con 0x7f0194072680 2024-02-19T15:22:17.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.690+0000 7f0198c00640 1 --2- 172.21.15.47:0/2718131686 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0194072680 0x7f01940b4e20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0184019f70 tx=0x7f01840190e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:17.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.690+0000 7f0191600640 1 -- 172.21.15.47:0/2718131686 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01840b4410 con 0x7f0194072680 2024-02-19T15:22:17.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.690+0000 7f0191600640 1 -- 172.21.15.47:0/2718131686 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01840b4570 con 0x7f0194072680 2024-02-19T15:22:17.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.690+0000 7f0191600640 1 -- 172.21.15.47:0/2718131686 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01840b4920 con 0x7f0194072680 2024-02-19T15:22:17.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.690+0000 7f019b600640 1 -- 172.21.15.47:0/2718131686 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0194155a70 con 0x7f0194072680 2024-02-19T15:22:17.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.690+0000 7f019b600640 1 -- 172.21.15.47:0/2718131686 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0194154dd0 con 0x7f0194072680 2024-02-19T15:22:17.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.690+0000 7f019b600640 1 -- 172.21.15.47:0/2718131686 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0194075360 con 0x7f0194072680 2024-02-19T15:22:17.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.690+0000 7f0191600640 1 -- 172.21.15.47:0/2718131686 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f01840b2040 con 0x7f0194072680 2024-02-19T15:22:17.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.694+0000 7f0191600640 1 --2- 172.21.15.47:0/2718131686 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0170066e40 0x7f0170069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:17.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.694+0000 7f0191600640 1 -- 172.21.15.47:0/2718131686 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f018411eb40 con 0x7f0194072680 2024-02-19T15:22:17.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.694+0000 7f018be00640 1 --2- 172.21.15.47:0/2718131686 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0170066e40 0x7f0170069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:17.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.694+0000 7f018be00640 1 --2- 172.21.15.47:0/2718131686 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0170066e40 0x7f0170069300 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f019415dcb0 tx=0x7f017c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:17.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.698+0000 7f0191600640 1 -- 172.21.15.47:0/2718131686 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f01840eb940 con 0x7f0194072680 2024-02-19T15:22:17.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.858+0000 7f019b600640 1 -- 172.21.15.47:0/2718131686 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f019406e800 con 0x7f0194072680 2024-02-19T15:22:17.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.858+0000 7f0191600640 1 -- 172.21.15.47:0/2718131686 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f01840b00a0 con 0x7f0194072680 2024-02-19T15:22:17.860 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:22:17.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.858+0000 7f019b600640 1 -- 172.21.15.47:0/2718131686 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0170066e40 msgr2=0x7f0170069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:17.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.858+0000 7f019b600640 1 --2- 172.21.15.47:0/2718131686 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0170066e40 0x7f0170069300 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f019415dcb0 tx=0x7f017c009240 comp rx=0 tx=0).stop 2024-02-19T15:22:17.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.858+0000 7f019b600640 1 -- 172.21.15.47:0/2718131686 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0194072680 msgr2=0x7f01940b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:17.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.858+0000 7f019b600640 1 --2- 172.21.15.47:0/2718131686 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0194072680 0x7f01940b4e20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0184019f70 tx=0x7f01840190e0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.858+0000 7f019b600640 1 -- 172.21.15.47:0/2718131686 shutdown_connections 2024-02-19T15:22:17.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.858+0000 7f019b600640 1 --2- 172.21.15.47:0/2718131686 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0170066e40 0x7f0170069300 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.858+0000 7f019b600640 1 --2- 172.21.15.47:0/2718131686 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f019415bd90 0x7f01940c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.858+0000 7f019b600640 1 --2- 172.21.15.47:0/2718131686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01940cb490 0x7f0194154890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.858+0000 7f019b600640 1 --2- 172.21.15.47:0/2718131686 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0194072680 0x7f01940b4e20 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:17.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.858+0000 7f019b600640 1 -- 172.21.15.47:0/2718131686 >> 172.21.15.47:0/2718131686 conn(0x7f01940cfc20 msgr2=0x7f01940c6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:17.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.858+0000 7f019b600640 1 -- 172.21.15.47:0/2718131686 shutdown_connections 2024-02-19T15:22:17.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:17.858+0000 7f019b600640 1 -- 172.21.15.47:0/2718131686 wait complete. 2024-02-19T15:22:17.881 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:22:18.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.022+0000 7f4122000640 1 Processor -- start 2024-02-19T15:22:18.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.022+0000 7f4122000640 1 -- start start 2024-02-19T15:22:18.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.022+0000 7f4122000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f411c15bd90 0x7f411c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:18.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.022+0000 7f4122000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f411c0cb490 0x7f411c072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:18.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.022+0000 7f4122000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c15fc90 con 0x7f411c072680 2024-02-19T15:22:18.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.022+0000 7f4122000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f411c15ff70 con 0x7f411c0cb490 2024-02-19T15:22:18.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.022+0000 7f4122000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f411c160250 con 0x7f411c15bd90 2024-02-19T15:22:18.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.022+0000 7f4113e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f411c0cb490 0x7f411c072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:18.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.022+0000 7f411be00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f411c072680 0x7f411c0ca270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35434/0 (socket says 172.21.15.47:35434) 2024-02-19T15:22:18.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.022+0000 7f411be00640 1 -- 172.21.15.47:0/3842522579 learned_addr learned my addr 172.21.15.47:0/3842522579 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:18.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.022+0000 7f411b400640 1 --2- 172.21.15.47:0/3842522579 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f411c15bd90 0x7f411c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:18.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.026+0000 7f4113e00640 1 -- 172.21.15.47:0/3842522579 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f411c15bd90 msgr2=0x7f411c070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:18.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.026+0000 7f4113e00640 1 --2- 172.21.15.47:0/3842522579 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f411c15bd90 0x7f411c070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.026+0000 7f4113e00640 1 -- 172.21.15.47:0/3842522579 >> v1:172.21.15.47:6789/0 conn(0x7f411c072680 legacy=0x7f411c0ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:18.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.026+0000 7f4113e00640 1 -- 172.21.15.47:0/3842522579 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411c160530 con 0x7f411c0cb490 2024-02-19T15:22:18.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.026+0000 7f4113e00640 1 --2- 172.21.15.47:0/3842522579 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f411c0cb490 0x7f411c072140 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f41040097c0 tx=0x7f410401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c5594943bc5b19d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:18.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.026+0000 7f411aa00640 1 -- 172.21.15.47:0/3842522579 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41040ac600 con 0x7f411c0cb490 2024-02-19T15:22:18.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.026+0000 7f411aa00640 1 -- 172.21.15.47:0/3842522579 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41040a8050 con 0x7f411c0cb490 2024-02-19T15:22:18.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.026+0000 7f411aa00640 1 -- 172.21.15.47:0/3842522579 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41040ac980 con 0x7f411c0cb490 2024-02-19T15:22:18.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.026+0000 7f4122000640 1 -- 172.21.15.47:0/3842522579 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f411c0cb490 msgr2=0x7f411c072140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:18.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.026+0000 7f4122000640 1 --2- 172.21.15.47:0/3842522579 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f411c0cb490 0x7f411c072140 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f41040097c0 tx=0x7f410401c340 comp rx=0 tx=0).stop 2024-02-19T15:22:18.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.026+0000 7f4122000640 1 -- 172.21.15.47:0/3842522579 shutdown_connections 2024-02-19T15:22:18.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.026+0000 7f4122000640 1 --2- 172.21.15.47:0/3842522579 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f411c0cb490 0x7f411c072140 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.026+0000 7f4122000640 1 --2- 172.21.15.47:0/3842522579 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f411c15bd90 0x7f411c070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.026+0000 7f4122000640 1 -- 172.21.15.47:0/3842522579 >> 172.21.15.47:0/3842522579 conn(0x7f411c0cfc20 msgr2=0x7f411c0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:18.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.026+0000 7f4122000640 1 -- 172.21.15.47:0/3842522579 shutdown_connections 2024-02-19T15:22:18.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.026+0000 7f4122000640 1 -- 172.21.15.47:0/3842522579 wait complete. 2024-02-19T15:22:18.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4122000640 1 Processor -- start 2024-02-19T15:22:18.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4122000640 1 -- start start 2024-02-19T15:22:18.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4122000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f411c072680 0x7f411c0c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:18.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4122000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f411c0cb490 0x7f411c143df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:18.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4122000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f411c15bd90 0x7f411c145e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:18.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4122000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411c158080 con 0x7f411c0cb490 2024-02-19T15:22:18.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4122000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411c1468d0 con 0x7f411c15bd90 2024-02-19T15:22:18.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4122000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f411c146bb0 con 0x7f411c072680 2024-02-19T15:22:18.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4113e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f411c0cb490 0x7f411c143df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:18.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4113e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f411c0cb490 0x7f411c143df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57986/0 (socket says 172.21.15.47:57986) 2024-02-19T15:22:18.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4113e00640 1 -- 172.21.15.47:0/600696602 learned_addr learned my addr 172.21.15.47:0/600696602 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:18.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f411b400640 1 --2- 172.21.15.47:0/600696602 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f411c072680 0x7f411c0c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:18.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f411be00640 1 --2- 172.21.15.47:0/600696602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f411c15bd90 0x7f411c145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:18.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4113e00640 1 -- 172.21.15.47:0/600696602 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f411c072680 msgr2=0x7f411c0c5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:18.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4113e00640 1 --2- 172.21.15.47:0/600696602 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f411c072680 0x7f411c0c5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4113e00640 1 -- 172.21.15.47:0/600696602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f411c15bd90 msgr2=0x7f411c145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:18.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4113e00640 1 --2- 172.21.15.47:0/600696602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f411c15bd90 0x7f411c145e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4113e00640 1 -- 172.21.15.47:0/600696602 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4104009420 con 0x7f411c0cb490 2024-02-19T15:22:18.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4113e00640 1 --2- 172.21.15.47:0/600696602 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f411c0cb490 0x7f411c143df0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f410401a9f0 tx=0x7f410401ab80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:18.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4113400640 1 -- 172.21.15.47:0/600696602 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41040b7450 con 0x7f411c0cb490 2024-02-19T15:22:18.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4113400640 1 -- 172.21.15.47:0/600696602 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41040b75b0 con 0x7f411c0cb490 2024-02-19T15:22:18.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4122000640 1 -- 172.21.15.47:0/600696602 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f411c158340 con 0x7f411c0cb490 2024-02-19T15:22:18.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4113400640 1 -- 172.21.15.47:0/600696602 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41040b7960 con 0x7f411c0cb490 2024-02-19T15:22:18.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.030+0000 7f4122000640 1 -- 172.21.15.47:0/600696602 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f411c0c63b0 con 0x7f411c0cb490 2024-02-19T15:22:18.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.034+0000 7f4122000640 1 -- 172.21.15.47:0/600696602 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f411c075360 con 0x7f411c0cb490 2024-02-19T15:22:18.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.034+0000 7f4113400640 1 -- 172.21.15.47:0/600696602 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f41040b3070 con 0x7f411c0cb490 2024-02-19T15:22:18.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.034+0000 7f4113400640 1 --2- 172.21.15.47:0/600696602 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f40f8066e40 0x7f40f8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:18.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.034+0000 7f4113400640 1 -- 172.21.15.47:0/600696602 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f410411ef40 con 0x7f411c0cb490 2024-02-19T15:22:18.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.034+0000 7f411b400640 1 --2- 172.21.15.47:0/600696602 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f40f8066e40 0x7f40f8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:18.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.034+0000 7f411b400640 1 --2- 172.21.15.47:0/600696602 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f40f8066e40 0x7f40f8069300 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f410c009b60 tx=0x7f410c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:18.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.038+0000 7f4113400640 1 -- 172.21.15.47:0/600696602 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f41040ebdd0 con 0x7f411c0cb490 2024-02-19T15:22:18.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.202+0000 7f4122000640 1 -- 172.21.15.47:0/600696602 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f411c06e800 con 0x7f411c0cb490 2024-02-19T15:22:18.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.202+0000 7f4113400640 1 -- 172.21.15.47:0/600696602 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f41040b1020 con 0x7f411c0cb490 2024-02-19T15:22:18.204 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:22:18.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.202+0000 7f4122000640 1 -- 172.21.15.47:0/600696602 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f40f8066e40 msgr2=0x7f40f8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:18.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.202+0000 7f4122000640 1 --2- 172.21.15.47:0/600696602 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f40f8066e40 0x7f40f8069300 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f410c009b60 tx=0x7f410c009190 comp rx=0 tx=0).stop 2024-02-19T15:22:18.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.202+0000 7f4122000640 1 -- 172.21.15.47:0/600696602 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f411c0cb490 msgr2=0x7f411c143df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:18.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.202+0000 7f4122000640 1 --2- 172.21.15.47:0/600696602 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f411c0cb490 0x7f411c143df0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f410401a9f0 tx=0x7f410401ab80 comp rx=0 tx=0).stop 2024-02-19T15:22:18.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.202+0000 7f4122000640 1 -- 172.21.15.47:0/600696602 shutdown_connections 2024-02-19T15:22:18.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.202+0000 7f4122000640 1 --2- 172.21.15.47:0/600696602 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f40f8066e40 0x7f40f8069300 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.202+0000 7f4122000640 1 --2- 172.21.15.47:0/600696602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f411c15bd90 0x7f411c145e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.202+0000 7f4122000640 1 --2- 172.21.15.47:0/600696602 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f411c0cb490 0x7f411c143df0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.202+0000 7f4122000640 1 --2- 172.21.15.47:0/600696602 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f411c072680 0x7f411c0c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.202+0000 7f4122000640 1 -- 172.21.15.47:0/600696602 >> 172.21.15.47:0/600696602 conn(0x7f411c0cfc20 msgr2=0x7f411c0c6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:18.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.202+0000 7f4122000640 1 -- 172.21.15.47:0/600696602 shutdown_connections 2024-02-19T15:22:18.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.202+0000 7f4122000640 1 -- 172.21.15.47:0/600696602 wait complete. 2024-02-19T15:22:18.226 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:22:18.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.366+0000 7ff0c9400640 1 Processor -- start 2024-02-19T15:22:18.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.366+0000 7ff0c9400640 1 -- start start 2024-02-19T15:22:18.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.370+0000 7ff0c9400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff0c40ce3b0 0x7ff0c40ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:18.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.370+0000 7ff0c9400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff0c40d2650 0x7ff0c40ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:18.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.370+0000 7ff0c9400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c40cc620 con 0x7ff0c415e280 2024-02-19T15:22:18.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.370+0000 7ff0c9400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c40d12e0 con 0x7ff0c40d2650 2024-02-19T15:22:18.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.370+0000 7ff0c9400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c40d1940 con 0x7ff0c40ce3b0 2024-02-19T15:22:18.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.370+0000 7ff0b9e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff0c40d2650 0x7ff0c40ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:18.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.370+0000 7ff0c3200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff0c415e280 0x7ff0c40ccff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35444/0 (socket says 172.21.15.47:35444) 2024-02-19T15:22:18.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.370+0000 7ff0c3200640 1 -- 172.21.15.47:0/1316642504 learned_addr learned my addr 172.21.15.47:0/1316642504 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:18.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.370+0000 7ff0c2800640 1 --2- 172.21.15.47:0/1316642504 >> v2:172.21.15.116:3300/0 conn(0x7ff0c40ce3b0 0x7ff0c40ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:18.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.370+0000 7ff0b9e00640 1 -- 172.21.15.47:0/1316642504 >> v2:172.21.15.116:3300/0 conn(0x7ff0c40ce3b0 msgr2=0x7ff0c40ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:18.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.370+0000 7ff0b9e00640 1 --2- 172.21.15.47:0/1316642504 >> v2:172.21.15.116:3300/0 conn(0x7ff0c40ce3b0 0x7ff0c40ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.370+0000 7ff0b9e00640 1 -- 172.21.15.47:0/1316642504 >> v1:172.21.15.47:6789/0 conn(0x7ff0c415e280 legacy=0x7ff0c40ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:18.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.370+0000 7ff0b9e00640 1 -- 172.21.15.47:0/1316642504 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c40cecd0 con 0x7ff0c40d2650 2024-02-19T15:22:18.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0b9e00640 1 --2- 172.21.15.47:0/1316642504 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0c40d2650 0x7ff0c40ccab0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4071220 tx=0x7ff0ac01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a297cc88ada7102 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:18.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c1e00640 1 -- 172.21.15.47:0/1316642504 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac0ab600 con 0x7ff0c40d2650 2024-02-19T15:22:18.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c1e00640 1 -- 172.21.15.47:0/1316642504 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ac0a7050 con 0x7ff0c40d2650 2024-02-19T15:22:18.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c1e00640 1 -- 172.21.15.47:0/1316642504 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac0ab980 con 0x7ff0c40d2650 2024-02-19T15:22:18.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c9400640 1 -- 172.21.15.47:0/1316642504 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0c40d2650 msgr2=0x7ff0c40ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:18.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c9400640 1 --2- 172.21.15.47:0/1316642504 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0c40d2650 0x7ff0c40ccab0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4071220 tx=0x7ff0ac01c360 comp rx=0 tx=0).stop 2024-02-19T15:22:18.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c9400640 1 -- 172.21.15.47:0/1316642504 shutdown_connections 2024-02-19T15:22:18.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c9400640 1 --2- 172.21.15.47:0/1316642504 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0c40d2650 0x7ff0c40ccab0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c9400640 1 --2- 172.21.15.47:0/1316642504 >> v2:172.21.15.116:3300/0 conn(0x7ff0c40ce3b0 0x7ff0c40ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c9400640 1 -- 172.21.15.47:0/1316642504 >> 172.21.15.47:0/1316642504 conn(0x7ff0c40d1b30 msgr2=0x7ff0c40d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:18.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c9400640 1 -- 172.21.15.47:0/1316642504 shutdown_connections 2024-02-19T15:22:18.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c9400640 1 -- 172.21.15.47:0/1316642504 wait complete. 2024-02-19T15:22:18.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c9400640 1 Processor -- start 2024-02-19T15:22:18.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c9400640 1 -- start start 2024-02-19T15:22:18.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c9400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0c40ce3b0 0x7ff0c40b34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:18.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c9400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0c40d2650 0x7ff0c415b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:18.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c9400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0c415e280 0x7ff0c4148fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:18.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c9400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c41494e0 con 0x7ff0c415e280 2024-02-19T15:22:18.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c9400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c40b46e0 con 0x7ff0c40d2650 2024-02-19T15:22:18.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c9400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c415a850 con 0x7ff0c40ce3b0 2024-02-19T15:22:18.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c3200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0c415e280 0x7ff0c4148fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:18.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c3200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0c415e280 0x7ff0c4148fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58008/0 (socket says 172.21.15.47:58008) 2024-02-19T15:22:18.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c3200640 1 -- 172.21.15.47:0/298364470 learned_addr learned my addr 172.21.15.47:0/298364470 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:18.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0c2800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0c40ce3b0 0x7ff0c40b34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:18.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.374+0000 7ff0b9e00640 1 --2- 172.21.15.47:0/298364470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0c40d2650 0x7ff0c415b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:18.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.378+0000 7ff0c3200640 1 -- 172.21.15.47:0/298364470 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0c40ce3b0 msgr2=0x7ff0c40b34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:18.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.378+0000 7ff0c3200640 1 --2- 172.21.15.47:0/298364470 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0c40ce3b0 0x7ff0c40b34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.378+0000 7ff0c3200640 1 -- 172.21.15.47:0/298364470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0c40d2650 msgr2=0x7ff0c415b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:18.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.378+0000 7ff0c3200640 1 --2- 172.21.15.47:0/298364470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0c40d2650 0x7ff0c415b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.378+0000 7ff0c3200640 1 -- 172.21.15.47:0/298364470 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ac009420 con 0x7ff0c415e280 2024-02-19T15:22:18.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.378+0000 7ff0c3200640 1 --2- 172.21.15.47:0/298364470 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0c415e280 0x7ff0c4148fa0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc00c3f0 tx=0x7ff0bc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:18.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.378+0000 7ff0bb400640 1 -- 172.21.15.47:0/298364470 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc005cf0 con 0x7ff0c415e280 2024-02-19T15:22:18.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.378+0000 7ff0bb400640 1 -- 172.21.15.47:0/298364470 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0bc005e50 con 0x7ff0c415e280 2024-02-19T15:22:18.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.378+0000 7ff0bb400640 1 -- 172.21.15.47:0/298364470 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc0072f0 con 0x7ff0c415e280 2024-02-19T15:22:18.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.378+0000 7ff0c9400640 1 -- 172.21.15.47:0/298364470 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c40b3ad0 con 0x7ff0c415e280 2024-02-19T15:22:18.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.378+0000 7ff0c9400640 1 -- 172.21.15.47:0/298364470 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c40b4090 con 0x7ff0c415e280 2024-02-19T15:22:18.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.378+0000 7ff0c9400640 1 -- 172.21.15.47:0/298364470 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c4075360 con 0x7ff0c415e280 2024-02-19T15:22:18.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.378+0000 7ff0bb400640 1 -- 172.21.15.47:0/298364470 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff0bc007450 con 0x7ff0c415e280 2024-02-19T15:22:18.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.378+0000 7ff0bb400640 1 --2- 172.21.15.47:0/298364470 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff0a0066e40 0x7ff0a0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:18.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.378+0000 7ff0bb400640 1 -- 172.21.15.47:0/298364470 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0bc083ee0 con 0x7ff0c415e280 2024-02-19T15:22:18.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.378+0000 7ff0c2800640 1 --2- 172.21.15.47:0/298364470 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff0a0066e40 0x7ff0a0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:18.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.382+0000 7ff0c2800640 1 --2- 172.21.15.47:0/298364470 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff0a0066e40 0x7ff0a0069300 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4011340 tx=0x7ff0b400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:18.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.382+0000 7ff0bb400640 1 -- 172.21.15.47:0/298364470 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff0bc050d60 con 0x7ff0c415e280 2024-02-19T15:22:18.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.546+0000 7ff0c9400640 1 -- 172.21.15.47:0/298364470 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff0c40b4820 con 0x7ff0c415e280 2024-02-19T15:22:18.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.546+0000 7ff0bb400640 1 -- 172.21.15.47:0/298364470 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff0bc0170a0 con 0x7ff0c415e280 2024-02-19T15:22:18.548 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:22:18.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.546+0000 7ff0c9400640 1 -- 172.21.15.47:0/298364470 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff0a0066e40 msgr2=0x7ff0a0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:18.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.546+0000 7ff0c9400640 1 --2- 172.21.15.47:0/298364470 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff0a0066e40 0x7ff0a0069300 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4011340 tx=0x7ff0b400b040 comp rx=0 tx=0).stop 2024-02-19T15:22:18.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.546+0000 7ff0c9400640 1 -- 172.21.15.47:0/298364470 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0c415e280 msgr2=0x7ff0c4148fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:18.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.546+0000 7ff0c9400640 1 --2- 172.21.15.47:0/298364470 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0c415e280 0x7ff0c4148fa0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc00c3f0 tx=0x7ff0bc00c8b0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.546+0000 7ff0c9400640 1 -- 172.21.15.47:0/298364470 shutdown_connections 2024-02-19T15:22:18.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.546+0000 7ff0c9400640 1 --2- 172.21.15.47:0/298364470 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff0a0066e40 0x7ff0a0069300 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.546+0000 7ff0c9400640 1 --2- 172.21.15.47:0/298364470 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0c415e280 0x7ff0c4148fa0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.546+0000 7ff0c9400640 1 --2- 172.21.15.47:0/298364470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0c40d2650 0x7ff0c415b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.546+0000 7ff0c9400640 1 --2- 172.21.15.47:0/298364470 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0c40ce3b0 0x7ff0c40b34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.546+0000 7ff0c9400640 1 -- 172.21.15.47:0/298364470 >> 172.21.15.47:0/298364470 conn(0x7ff0c40d1b30 msgr2=0x7ff0c40c7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:18.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.546+0000 7ff0c9400640 1 -- 172.21.15.47:0/298364470 shutdown_connections 2024-02-19T15:22:18.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.546+0000 7ff0c9400640 1 -- 172.21.15.47:0/298364470 wait complete. 2024-02-19T15:22:18.569 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:22:18.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.706+0000 7fdf79400640 1 Processor -- start 2024-02-19T15:22:18.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.706+0000 7fdf79400640 1 -- start start 2024-02-19T15:22:18.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.710+0000 7fdf79400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf7415e110 0x7fdf740ce6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:18.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.710+0000 7fdf79400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdf7406f120 0x7fdf7406aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:18.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.710+0000 7fdf79400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf740b3120 con 0x7fdf7406b3f0 2024-02-19T15:22:18.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.710+0000 7fdf79400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf7406b880 con 0x7fdf7406f120 2024-02-19T15:22:18.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.710+0000 7fdf79400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf740b2510 con 0x7fdf7415e110 2024-02-19T15:22:18.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.710+0000 7fdf6be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdf7406f120 0x7fdf7406aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:18.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.710+0000 7fdf72800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf7415e110 0x7fdf740ce6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:18.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.710+0000 7fdf73200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fdf7406b3f0 0x7fdf7406d8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35446/0 (socket says 172.21.15.47:35446) 2024-02-19T15:22:18.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.710+0000 7fdf73200640 1 -- 172.21.15.47:0/406664756 learned_addr learned my addr 172.21.15.47:0/406664756 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:18.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.710+0000 7fdf6be00640 1 -- 172.21.15.47:0/406664756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf7415e110 msgr2=0x7fdf740ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:18.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.710+0000 7fdf6be00640 1 --2- 172.21.15.47:0/406664756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf7415e110 0x7fdf740ce6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.710+0000 7fdf6be00640 1 -- 172.21.15.47:0/406664756 >> v1:172.21.15.47:6789/0 conn(0x7fdf7406b3f0 legacy=0x7fdf7406d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:18.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.710+0000 7fdf71e00640 1 -- 172.21.15.47:0/406664756 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258170885 0 0) 0x7fdf740b3120 con 0x7fdf7406b3f0 2024-02-19T15:22:18.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.710+0000 7fdf6be00640 1 -- 172.21.15.47:0/406664756 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf740b27f0 con 0x7fdf7406f120 2024-02-19T15:22:18.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.714+0000 7fdf6be00640 1 --2- 172.21.15.47:0/406664756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf7406f120 0x7fdf7406aeb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0097c0 tx=0x7fdf5c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96810b568a1cf06d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:18.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.714+0000 7fdf71e00640 1 -- 172.21.15.47:0/406664756 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0ac620 con 0x7fdf7406f120 2024-02-19T15:22:18.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.714+0000 7fdf71e00640 1 -- 172.21.15.47:0/406664756 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf5c0ac780 con 0x7fdf7406f120 2024-02-19T15:22:18.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.714+0000 7fdf71e00640 1 -- 172.21.15.47:0/406664756 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0acb10 con 0x7fdf7406f120 2024-02-19T15:22:18.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.714+0000 7fdf79400640 1 -- 172.21.15.47:0/406664756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf7406f120 msgr2=0x7fdf7406aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:18.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.714+0000 7fdf79400640 1 --2- 172.21.15.47:0/406664756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf7406f120 0x7fdf7406aeb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0097c0 tx=0x7fdf5c01c4a0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.714+0000 7fdf79400640 1 -- 172.21.15.47:0/406664756 shutdown_connections 2024-02-19T15:22:18.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.714+0000 7fdf79400640 1 --2- 172.21.15.47:0/406664756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf7406f120 0x7fdf7406aeb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.714+0000 7fdf79400640 1 --2- 172.21.15.47:0/406664756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf7415e110 0x7fdf740ce6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.714+0000 7fdf79400640 1 -- 172.21.15.47:0/406664756 >> 172.21.15.47:0/406664756 conn(0x7fdf740cd510 msgr2=0x7fdf740cd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:18.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.714+0000 7fdf79400640 1 -- 172.21.15.47:0/406664756 shutdown_connections 2024-02-19T15:22:18.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.714+0000 7fdf79400640 1 -- 172.21.15.47:0/406664756 wait complete. 2024-02-19T15:22:18.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.714+0000 7fdf79400640 1 Processor -- start 2024-02-19T15:22:18.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.714+0000 7fdf79400640 1 -- start start 2024-02-19T15:22:18.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.714+0000 7fdf79400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf7406b3f0 0x7fdf7415e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:18.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.714+0000 7fdf79400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf7406f120 0x7fdf7415c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:18.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf79400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf7415e110 0x7fdf7415c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:18.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf79400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf74158970 con 0x7fdf7406f120 2024-02-19T15:22:18.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf79400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf74158c10 con 0x7fdf7415e110 2024-02-19T15:22:18.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf79400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf74158ef0 con 0x7fdf7406b3f0 2024-02-19T15:22:18.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf6be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf7406f120 0x7fdf7415c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:18.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf6be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf7406f120 0x7fdf7415c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58026/0 (socket says 172.21.15.47:58026) 2024-02-19T15:22:18.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf6be00640 1 -- 172.21.15.47:0/1012611407 learned_addr learned my addr 172.21.15.47:0/1012611407 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:18.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf72800640 1 --2- 172.21.15.47:0/1012611407 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf7406b3f0 0x7fdf7415e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:18.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf73200640 1 --2- 172.21.15.47:0/1012611407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf7415e110 0x7fdf7415c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:18.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf6be00640 1 -- 172.21.15.47:0/1012611407 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf7406b3f0 msgr2=0x7fdf7415e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:18.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf6be00640 1 --2- 172.21.15.47:0/1012611407 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf7406b3f0 0x7fdf7415e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf6be00640 1 -- 172.21.15.47:0/1012611407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf7415e110 msgr2=0x7fdf7415c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:18.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf6be00640 1 --2- 172.21.15.47:0/1012611407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf7415e110 0x7fdf7415c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf6be00640 1 -- 172.21.15.47:0/1012611407 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c009420 con 0x7fdf7406f120 2024-02-19T15:22:18.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf73200640 1 --2- 172.21.15.47:0/1012611407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf7415e110 0x7fdf7415c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:18.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf6be00640 1 --2- 172.21.15.47:0/1012611407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf7406f120 0x7fdf7415c160 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0af040 tx=0x7fdf5c019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:18.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf6aa00640 1 -- 172.21.15.47:0/1012611407 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0b5410 con 0x7fdf7406f120 2024-02-19T15:22:18.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf6aa00640 1 -- 172.21.15.47:0/1012611407 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf5c0b5590 con 0x7fdf7406f120 2024-02-19T15:22:18.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf79400640 1 -- 172.21.15.47:0/1012611407 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf741591d0 con 0x7fdf7406f120 2024-02-19T15:22:18.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf6aa00640 1 -- 172.21.15.47:0/1012611407 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0b5940 con 0x7fdf7406f120 2024-02-19T15:22:18.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf79400640 1 -- 172.21.15.47:0/1012611407 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf741561a0 con 0x7fdf7406f120 2024-02-19T15:22:18.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf79400640 1 -- 172.21.15.47:0/1012611407 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf74068500 con 0x7fdf7406f120 2024-02-19T15:22:18.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.718+0000 7fdf6aa00640 1 -- 172.21.15.47:0/1012611407 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fdf5c0b3040 con 0x7fdf7406f120 2024-02-19T15:22:18.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.722+0000 7fdf6aa00640 1 --2- 172.21.15.47:0/1012611407 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdf50066e40 0x7fdf50069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:18.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.722+0000 7fdf6aa00640 1 -- 172.21.15.47:0/1012611407 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf5c11e850 con 0x7fdf7406f120 2024-02-19T15:22:18.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.722+0000 7fdf72800640 1 --2- 172.21.15.47:0/1012611407 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdf50066e40 0x7fdf50069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:18.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.722+0000 7fdf72800640 1 --2- 172.21.15.47:0/1012611407 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdf50066e40 0x7fdf50069300 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdf740b6b80 tx=0x7fdf64009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:18.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.726+0000 7fdf6aa00640 1 -- 172.21.15.47:0/1012611407 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fdf5c0eb5a0 con 0x7fdf7406f120 2024-02-19T15:22:18.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.921+0000 7fdf79400640 1 -- 172.21.15.47:0/1012611407 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf74075360 con 0x7fdf7406f120 2024-02-19T15:22:18.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.921+0000 7fdf6aa00640 1 -- 172.21.15.47:0/1012611407 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4935 (secure 0 0 0) 0x7fdf5c0b1090 con 0x7fdf7406f120 2024-02-19T15:22:18.924 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:18.924 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3873760209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3873760209},{"type":"v1","addr":"172.21.15.116:6836","nonce":3873760209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/284820350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":284820350},{"type":"v1","addr":"172.21.15.47:6834","nonce":284820350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2764089359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2764089359},{"type":"v1","addr":"172.21.15.47:6835","nonce":2764089359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:22:05.228517+0000","modified":"2024-02-19T15:22:08.242600+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6837/3920452343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3920452343},{"type":"v1","addr":"172.21.15.116:6837","nonce":3920452343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-19T15:22:18.924 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 8 2024-02-19T15:22:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.921+0000 7fdf79400640 1 -- 172.21.15.47:0/1012611407 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdf50066e40 msgr2=0x7fdf50069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.921+0000 7fdf79400640 1 --2- 172.21.15.47:0/1012611407 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdf50066e40 0x7fdf50069300 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdf740b6b80 tx=0x7fdf64009190 comp rx=0 tx=0).stop 2024-02-19T15:22:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.921+0000 7fdf79400640 1 -- 172.21.15.47:0/1012611407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf7406f120 msgr2=0x7fdf7415c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.921+0000 7fdf79400640 1 --2- 172.21.15.47:0/1012611407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf7406f120 0x7fdf7415c160 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0af040 tx=0x7fdf5c019240 comp rx=0 tx=0).stop 2024-02-19T15:22:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.921+0000 7fdf79400640 1 -- 172.21.15.47:0/1012611407 shutdown_connections 2024-02-19T15:22:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.921+0000 7fdf79400640 1 --2- 172.21.15.47:0/1012611407 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdf50066e40 0x7fdf50069300 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.921+0000 7fdf79400640 1 --2- 172.21.15.47:0/1012611407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf7415e110 0x7fdf7415c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.921+0000 7fdf79400640 1 --2- 172.21.15.47:0/1012611407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf7406f120 0x7fdf7415c160 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.921+0000 7fdf79400640 1 --2- 172.21.15.47:0/1012611407 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf7406b3f0 0x7fdf7415e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:18.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.921+0000 7fdf79400640 1 -- 172.21.15.47:0/1012611407 >> 172.21.15.47:0/1012611407 conn(0x7fdf740cd510 msgr2=0x7fdf740b1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:18.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.921+0000 7fdf79400640 1 -- 172.21.15.47:0/1012611407 shutdown_connections 2024-02-19T15:22:18.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:18.921+0000 7fdf79400640 1 -- 172.21.15.47:0/1012611407 wait complete. 2024-02-19T15:22:18.944 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:19.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.081+0000 7fceaf000640 1 Processor -- start 2024-02-19T15:22:19.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.081+0000 7fceaf000640 1 -- start start 2024-02-19T15:22:19.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.081+0000 7fceaf000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcea80b4240 0x7fcea80d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:19.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.081+0000 7fceaf000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcea80ce400 0x7fcea80ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:19.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.081+0000 7fceaf000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea80ced90 con 0x7fcea815dfa0 2024-02-19T15:22:19.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.081+0000 7fceaf000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea80d1070 con 0x7fcea80ce400 2024-02-19T15:22:19.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.081+0000 7fceaf000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea806ec60 con 0x7fcea80b4240 2024-02-19T15:22:19.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.081+0000 7fcead000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcea80ce400 0x7fcea80ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:19.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.081+0000 7fcea7e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fcea815dfa0 0x7fcea806fa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35448/0 (socket says 172.21.15.47:35448) 2024-02-19T15:22:19.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.081+0000 7fcea7e00640 1 -- 172.21.15.47:0/3005466955 learned_addr learned my addr 172.21.15.47:0/3005466955 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:19.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.081+0000 7fce9fe00640 1 --2- 172.21.15.47:0/3005466955 >> v2:172.21.15.116:3300/0 conn(0x7fcea80b4240 0x7fcea80d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:19.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.085+0000 7fcead000640 1 -- 172.21.15.47:0/3005466955 >> v2:172.21.15.116:3300/0 conn(0x7fcea80b4240 msgr2=0x7fcea80d0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:19.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.085+0000 7fcead000640 1 --2- 172.21.15.47:0/3005466955 >> v2:172.21.15.116:3300/0 conn(0x7fcea80b4240 0x7fcea80d0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:19.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.085+0000 7fcead000640 1 -- 172.21.15.47:0/3005466955 >> v1:172.21.15.47:6789/0 conn(0x7fcea815dfa0 legacy=0x7fcea806fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:19.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.085+0000 7fcead000640 1 -- 172.21.15.47:0/3005466955 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea806ef40 con 0x7fcea80ce400 2024-02-19T15:22:19.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.085+0000 7fcea7400640 1 -- 172.21.15.47:0/3005466955 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 541864565 0 0) 0x7fce98001120 con 0x7fcea815dfa0 2024-02-19T15:22:19.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.085+0000 7fcead000640 1 --2- 172.21.15.47:0/3005466955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcea80ce400 0x7fcea80ce850 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcea8072110 tx=0x7fcea001c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c832c0f4d76c4204 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:19.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.085+0000 7fcea7400640 1 -- 172.21.15.47:0/3005466955 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea00a58d0 con 0x7fcea80ce400 2024-02-19T15:22:19.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.085+0000 7fcea7400640 1 -- 172.21.15.47:0/3005466955 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea00a5a30 con 0x7fcea80ce400 2024-02-19T15:22:19.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.085+0000 7fcea7400640 1 -- 172.21.15.47:0/3005466955 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea00a5dc0 con 0x7fcea80ce400 2024-02-19T15:22:19.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.085+0000 7fceaf000640 1 -- 172.21.15.47:0/3005466955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcea80ce400 msgr2=0x7fcea80ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:19.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.085+0000 7fceaf000640 1 --2- 172.21.15.47:0/3005466955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcea80ce400 0x7fcea80ce850 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcea8072110 tx=0x7fcea001c900 comp rx=0 tx=0).stop 2024-02-19T15:22:19.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fceaf000640 1 -- 172.21.15.47:0/3005466955 shutdown_connections 2024-02-19T15:22:19.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fceaf000640 1 --2- 172.21.15.47:0/3005466955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcea80ce400 0x7fcea80ce850 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:19.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fceaf000640 1 --2- 172.21.15.47:0/3005466955 >> v2:172.21.15.116:3300/0 conn(0x7fcea80b4240 0x7fcea80d0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:19.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fceaf000640 1 -- 172.21.15.47:0/3005466955 >> 172.21.15.47:0/3005466955 conn(0x7fcea80b3690 msgr2=0x7fcea80b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:19.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fceaf000640 1 -- 172.21.15.47:0/3005466955 shutdown_connections 2024-02-19T15:22:19.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fceaf000640 1 -- 172.21.15.47:0/3005466955 wait complete. 2024-02-19T15:22:19.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fceaf000640 1 Processor -- start 2024-02-19T15:22:19.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fceaf000640 1 -- start start 2024-02-19T15:22:19.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fceaf000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcea80b4240 0x7fcea815afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:19.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fceaf000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcea80ce400 0x7fcea8153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:19.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fceaf000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcea815dfa0 0x7fcea8153f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:19.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fceaf000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea80cd060 con 0x7fcea80b4240 2024-02-19T15:22:19.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fceaf000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea80cd340 con 0x7fcea815dfa0 2024-02-19T15:22:19.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fceaf000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea80cd620 con 0x7fcea80ce400 2024-02-19T15:22:19.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fcea7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcea80b4240 0x7fcea815afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:19.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fcea7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcea80b4240 0x7fcea815afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58040/0 (socket says 172.21.15.47:58040) 2024-02-19T15:22:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fcea7e00640 1 -- 172.21.15.47:0/3414634675 learned_addr learned my addr 172.21.15.47:0/3414634675 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fcead000640 1 --2- 172.21.15.47:0/3414634675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcea815dfa0 0x7fcea8153f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fce9fe00640 1 --2- 172.21.15.47:0/3414634675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcea80ce400 0x7fcea8153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fcea7e00640 1 -- 172.21.15.47:0/3414634675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcea80ce400 msgr2=0x7fcea8153a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fcea7e00640 1 --2- 172.21.15.47:0/3414634675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcea80ce400 0x7fcea8153a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fcea7e00640 1 -- 172.21.15.47:0/3414634675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcea815dfa0 msgr2=0x7fcea8153f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fcea7e00640 1 --2- 172.21.15.47:0/3414634675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcea815dfa0 0x7fcea8153f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fcea7e00640 1 -- 172.21.15.47:0/3414634675 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea0009a70 con 0x7fcea80b4240 2024-02-19T15:22:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fcea7e00640 1 --2- 172.21.15.47:0/3414634675 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcea80b4240 0x7fcea815afa0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fce9800c480 tx=0x7fce9800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.089+0000 7fcea4c00640 1 -- 172.21.15.47:0/3414634675 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce980074e0 con 0x7fcea80b4240 2024-02-19T15:22:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.093+0000 7fcea4c00640 1 -- 172.21.15.47:0/3414634675 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce98007660 con 0x7fcea80b4240 2024-02-19T15:22:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.093+0000 7fceaf000640 1 -- 172.21.15.47:0/3414634675 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea80cd900 con 0x7fcea80b4240 2024-02-19T15:22:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.093+0000 7fcea4c00640 1 -- 172.21.15.47:0/3414634675 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce98004a30 con 0x7fcea80b4240 2024-02-19T15:22:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.093+0000 7fceaf000640 1 -- 172.21.15.47:0/3414634675 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea814e740 con 0x7fcea80b4240 2024-02-19T15:22:19.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.093+0000 7fceaf000640 1 -- 172.21.15.47:0/3414634675 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea8075360 con 0x7fcea80b4240 2024-02-19T15:22:19.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.093+0000 7fcea4c00640 1 -- 172.21.15.47:0/3414634675 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fce98004b90 con 0x7fcea80b4240 2024-02-19T15:22:19.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.093+0000 7fcea4c00640 1 --2- 172.21.15.47:0/3414634675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fce84066e40 0x7fce84069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:19.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.093+0000 7fcea4c00640 1 -- 172.21.15.47:0/3414634675 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce98083d90 con 0x7fcea80b4240 2024-02-19T15:22:19.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.093+0000 7fce9fe00640 1 --2- 172.21.15.47:0/3414634675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fce84066e40 0x7fce84069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:19.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.097+0000 7fce9fe00640 1 --2- 172.21.15.47:0/3414634675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fce84066e40 0x7fce84069300 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcea80b7ad0 tx=0x7fce90009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:19.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.097+0000 7fcea4c00640 1 -- 172.21.15.47:0/3414634675 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fce98050c20 con 0x7fcea80b4240 2024-02-19T15:22:19.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.265+0000 7fceaf000640 1 -- 172.21.15.47:0/3414634675 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcea80ced90 con 0x7fcea80b4240 2024-02-19T15:22:19.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.269+0000 7fcea4c00640 1 -- 172.21.15.47:0/3414634675 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7fcea80ced90 con 0x7fcea80b4240 2024-02-19T15:22:19.271 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:19.271 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":21,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:06.843155+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:03.091360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:04.222965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:19.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.269+0000 7fceaf000640 1 -- 172.21.15.47:0/3414634675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fce84066e40 msgr2=0x7fce84069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:19.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.269+0000 7fceaf000640 1 --2- 172.21.15.47:0/3414634675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fce84066e40 0x7fce84069300 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcea80b7ad0 tx=0x7fce90009190 comp rx=0 tx=0).stop 2024-02-19T15:22:19.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.269+0000 7fceaf000640 1 -- 172.21.15.47:0/3414634675 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcea80b4240 msgr2=0x7fcea815afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:19.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.269+0000 7fceaf000640 1 --2- 172.21.15.47:0/3414634675 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcea80b4240 0x7fcea815afa0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fce9800c480 tx=0x7fce9800c940 comp rx=0 tx=0).stop 2024-02-19T15:22:19.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.269+0000 7fceaf000640 1 -- 172.21.15.47:0/3414634675 shutdown_connections 2024-02-19T15:22:19.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.269+0000 7fceaf000640 1 --2- 172.21.15.47:0/3414634675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fce84066e40 0x7fce84069300 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:19.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.269+0000 7fceaf000640 1 --2- 172.21.15.47:0/3414634675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcea815dfa0 0x7fcea8153f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:19.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.269+0000 7fceaf000640 1 --2- 172.21.15.47:0/3414634675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcea80ce400 0x7fcea8153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:19.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.269+0000 7fceaf000640 1 --2- 172.21.15.47:0/3414634675 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcea80b4240 0x7fcea815afa0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:19.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.269+0000 7fceaf000640 1 -- 172.21.15.47:0/3414634675 >> 172.21.15.47:0/3414634675 conn(0x7fcea80b3690 msgr2=0x7fcea806fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:19.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.269+0000 7fceaf000640 1 -- 172.21.15.47:0/3414634675 shutdown_connections 2024-02-19T15:22:19.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.269+0000 7fceaf000640 1 -- 172.21.15.47:0/3414634675 wait complete. 2024-02-19T15:22:19.291 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:22:19.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.429+0000 7f8fa9e00640 1 Processor -- start 2024-02-19T15:22:19.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.429+0000 7f8fa9e00640 1 -- start start 2024-02-19T15:22:19.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.429+0000 7f8fa9e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8fa415de20 0x7f8fa40d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:19.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.429+0000 7f8fa9e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8fa406cd90 0x7f8fa406bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:19.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.429+0000 7f8fa9e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa406eb80 con 0x7f8fa415e200 2024-02-19T15:22:19.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.429+0000 7f8fa9e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fa406c500 con 0x7f8fa406cd90 2024-02-19T15:22:19.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.429+0000 7f8fa9e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa40d0a80 con 0x7f8fa415de20 2024-02-19T15:22:19.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.429+0000 7f8fa3400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8fa406cd90 0x7f8fa406bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:19.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.429+0000 7f8f9be00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8fa415e200 0x7f8fa406c680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35452/0 (socket says 172.21.15.47:35452) 2024-02-19T15:22:19.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.429+0000 7f8f9be00640 1 -- 172.21.15.47:0/4030199923 learned_addr learned my addr 172.21.15.47:0/4030199923 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:19.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.429+0000 7f8fa2a00640 1 --2- 172.21.15.47:0/4030199923 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8fa415de20 0x7f8fa40d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:19.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.429+0000 7f8fa3400640 1 -- 172.21.15.47:0/4030199923 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8fa415de20 msgr2=0x7f8fa40d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:19.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.429+0000 7f8fa3400640 1 --2- 172.21.15.47:0/4030199923 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8fa415de20 0x7f8fa40d2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:19.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.429+0000 7f8fa3400640 1 -- 172.21.15.47:0/4030199923 >> v1:172.21.15.47:6789/0 conn(0x7f8fa415e200 legacy=0x7f8fa406c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:19.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.429+0000 7f8fa3400640 1 -- 172.21.15.47:0/4030199923 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa40d0d60 con 0x7f8fa406cd90 2024-02-19T15:22:19.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.429+0000 7f8fa2000640 1 -- 172.21.15.47:0/4030199923 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295664028 0 0) 0x7f8fa406eb80 con 0x7f8fa415e200 2024-02-19T15:22:19.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.433+0000 7f8fa3400640 1 --2- 172.21.15.47:0/4030199923 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fa406cd90 0x7f8fa406bfc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c009d90 tx=0x7f8f9c01c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d8383e1d4aa6eb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:19.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.433+0000 7f8fa2000640 1 -- 172.21.15.47:0/4030199923 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c0a5a00 con 0x7f8fa406cd90 2024-02-19T15:22:19.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.433+0000 7f8fa2000640 1 -- 172.21.15.47:0/4030199923 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9c0a5b60 con 0x7f8fa406cd90 2024-02-19T15:22:19.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.433+0000 7f8fa2000640 1 -- 172.21.15.47:0/4030199923 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c0ad440 con 0x7f8fa406cd90 2024-02-19T15:22:19.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.433+0000 7f8fa9e00640 1 -- 172.21.15.47:0/4030199923 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fa406cd90 msgr2=0x7f8fa406bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:19.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.433+0000 7f8fa9e00640 1 --2- 172.21.15.47:0/4030199923 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fa406cd90 0x7f8fa406bfc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c009d90 tx=0x7f8f9c01c940 comp rx=0 tx=0).stop 2024-02-19T15:22:19.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.433+0000 7f8fa9e00640 1 -- 172.21.15.47:0/4030199923 shutdown_connections 2024-02-19T15:22:19.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.433+0000 7f8fa9e00640 1 --2- 172.21.15.47:0/4030199923 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fa406cd90 0x7f8fa406bfc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:19.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.433+0000 7f8fa9e00640 1 --2- 172.21.15.47:0/4030199923 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8fa415de20 0x7f8fa40d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:19.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.433+0000 7f8fa9e00640 1 -- 172.21.15.47:0/4030199923 >> 172.21.15.47:0/4030199923 conn(0x7f8fa40d1b70 msgr2=0x7f8fa40d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:19.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.433+0000 7f8fa9e00640 1 -- 172.21.15.47:0/4030199923 shutdown_connections 2024-02-19T15:22:19.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.433+0000 7f8fa9e00640 1 -- 172.21.15.47:0/4030199923 wait complete. 2024-02-19T15:22:19.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8fa9e00640 1 Processor -- start 2024-02-19T15:22:19.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8fa9e00640 1 -- start start 2024-02-19T15:22:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8fa9e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8fa406cd90 0x7f8fa40ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8fa9e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fa415de20 0x7f8fa40d1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8fa9e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa415e200 0x7f8fa40d2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8fa9e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa40cf020 con 0x7f8fa415de20 2024-02-19T15:22:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8fa9e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa40cedf0 con 0x7f8fa415e200 2024-02-19T15:22:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8fa9e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa40cc890 con 0x7f8fa406cd90 2024-02-19T15:22:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8f9be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fa415de20 0x7f8fa40d1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8f9be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fa415de20 0x7f8fa40d1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58048/0 (socket says 172.21.15.47:58048) 2024-02-19T15:22:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8f9be00640 1 -- 172.21.15.47:0/1365720848 learned_addr learned my addr 172.21.15.47:0/1365720848 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8fa2a00640 1 --2- 172.21.15.47:0/1365720848 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8fa406cd90 0x7f8fa40ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8fa3400640 1 --2- 172.21.15.47:0/1365720848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa415e200 0x7f8fa40d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8f9be00640 1 -- 172.21.15.47:0/1365720848 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8fa406cd90 msgr2=0x7f8fa40ce8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8f9be00640 1 --2- 172.21.15.47:0/1365720848 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8fa406cd90 0x7f8fa40ce8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8f9be00640 1 -- 172.21.15.47:0/1365720848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa415e200 msgr2=0x7f8fa40d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8f9be00640 1 --2- 172.21.15.47:0/1365720848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa415e200 0x7f8fa40d2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8f9be00640 1 -- 172.21.15.47:0/1365720848 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9c009a70 con 0x7f8fa415de20 2024-02-19T15:22:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8fa3400640 1 --2- 172.21.15.47:0/1365720848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa415e200 0x7f8fa40d2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8fa2a00640 1 --2- 172.21.15.47:0/1365720848 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8fa406cd90 0x7f8fa40ce8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8f9be00640 1 --2- 172.21.15.47:0/1365720848 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fa415de20 0x7f8fa40d1f50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c00c480 tx=0x7f8f8c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8f9b400640 1 -- 172.21.15.47:0/1365720848 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c0074e0 con 0x7f8fa415de20 2024-02-19T15:22:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8f9b400640 1 -- 172.21.15.47:0/1365720848 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f8c007660 con 0x7f8fa415de20 2024-02-19T15:22:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8fa9e00640 1 -- 172.21.15.47:0/1365720848 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa40cc4b0 con 0x7f8fa415de20 2024-02-19T15:22:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8f9b400640 1 -- 172.21.15.47:0/1365720848 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c004a30 con 0x7f8fa415de20 2024-02-19T15:22:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8fa9e00640 1 -- 172.21.15.47:0/1365720848 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa4155cb0 con 0x7f8fa415de20 2024-02-19T15:22:19.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.437+0000 7f8fa9e00640 1 -- 172.21.15.47:0/1365720848 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa4068500 con 0x7f8fa415de20 2024-02-19T15:22:19.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.441+0000 7f8f9b400640 1 -- 172.21.15.47:0/1365720848 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8f8c004b90 con 0x7f8fa415de20 2024-02-19T15:22:19.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.441+0000 7f8f9b400640 1 --2- 172.21.15.47:0/1365720848 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8f80066e40 0x7f8f80069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:19.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.441+0000 7f8f9b400640 1 -- 172.21.15.47:0/1365720848 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f8c083fd0 con 0x7f8fa415de20 2024-02-19T15:22:19.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.441+0000 7f8fa2a00640 1 --2- 172.21.15.47:0/1365720848 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8f80066e40 0x7f8f80069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:19.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.441+0000 7f8fa2a00640 1 --2- 172.21.15.47:0/1365720848 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8f80066e40 0x7f8f80069300 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8f94009bb0 tx=0x7f8f94008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:19.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.445+0000 7f8f9b400640 1 -- 172.21.15.47:0/1365720848 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8f8c0548e0 con 0x7f8fa415de20 2024-02-19T15:22:19.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.641+0000 7f8fa9e00640 1 -- 172.21.15.47:0/1365720848 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fa40cca00 con 0x7f8fa415de20 2024-02-19T15:22:19.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.645+0000 7f8f9b400640 1 -- 172.21.15.47:0/1365720848 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4935 (secure 0 0 0) 0x7f8f8c018090 con 0x7f8fa415de20 2024-02-19T15:22:19.646 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:19.646 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3873760209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3873760209},{"type":"v1","addr":"172.21.15.116:6836","nonce":3873760209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/284820350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":284820350},{"type":"v1","addr":"172.21.15.47:6834","nonce":284820350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2764089359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2764089359},{"type":"v1","addr":"172.21.15.47:6835","nonce":2764089359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:22:05.228517+0000","modified":"2024-02-19T15:22:08.242600+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6837/3920452343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3920452343},{"type":"v1","addr":"172.21.15.116:6837","nonce":3920452343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-19T15:22:19.647 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 8 2024-02-19T15:22:19.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.645+0000 7f8fa9e00640 1 -- 172.21.15.47:0/1365720848 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8f80066e40 msgr2=0x7f8f80069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:19.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.645+0000 7f8fa9e00640 1 --2- 172.21.15.47:0/1365720848 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8f80066e40 0x7f8f80069300 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8f94009bb0 tx=0x7f8f94008040 comp rx=0 tx=0).stop 2024-02-19T15:22:19.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.645+0000 7f8fa9e00640 1 -- 172.21.15.47:0/1365720848 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fa415de20 msgr2=0x7f8fa40d1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:19.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.645+0000 7f8fa9e00640 1 --2- 172.21.15.47:0/1365720848 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fa415de20 0x7f8fa40d1f50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c00c480 tx=0x7f8f8c00c940 comp rx=0 tx=0).stop 2024-02-19T15:22:19.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.645+0000 7f8fa9e00640 1 -- 172.21.15.47:0/1365720848 shutdown_connections 2024-02-19T15:22:19.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.645+0000 7f8fa9e00640 1 --2- 172.21.15.47:0/1365720848 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8f80066e40 0x7f8f80069300 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:19.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.645+0000 7f8fa9e00640 1 --2- 172.21.15.47:0/1365720848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa415e200 0x7f8fa40d2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:19.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.645+0000 7f8fa9e00640 1 --2- 172.21.15.47:0/1365720848 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fa415de20 0x7f8fa40d1f50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:19.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.645+0000 7f8fa9e00640 1 --2- 172.21.15.47:0/1365720848 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8fa406cd90 0x7f8fa40ce8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:19.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.645+0000 7f8fa9e00640 1 -- 172.21.15.47:0/1365720848 >> 172.21.15.47:0/1365720848 conn(0x7f8fa40d1b70 msgr2=0x7f8fa4074000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:19.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.645+0000 7f8fa9e00640 1 -- 172.21.15.47:0/1365720848 shutdown_connections 2024-02-19T15:22:19.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:19.645+0000 7f8fa9e00640 1 -- 172.21.15.47:0/1365720848 wait complete. 2024-02-19T15:22:20.069 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:22:05.228517+0000', 'modified': '2024-02-19T15:22:08.242600+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4284}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4284': {'gid': 4284, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.116:6837/3920452343', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 3920452343}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 3920452343}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:22:20.069 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T15:22:20.070 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-19T15:22:20.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.209+0000 7efc5da00640 1 Processor -- start 2024-02-19T15:22:20.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.209+0000 7efc5da00640 1 -- start start 2024-02-19T15:22:20.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.213+0000 7efc5da00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efc5815d6e0 0x7efc58162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.213+0000 7efc5da00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc58071350 0x7efc58164c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.213+0000 7efc5da00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc580718c0 con 0x7efc5815b1a0 2024-02-19T15:22:20.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.213+0000 7efc5da00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc58071ba0 con 0x7efc5815d6e0 2024-02-19T15:22:20.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.213+0000 7efc5da00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc58071e80 con 0x7efc58071350 2024-02-19T15:22:20.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.213+0000 7efc4fe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efc5815d6e0 0x7efc58162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.213+0000 7efc57400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc58071350 0x7efc58164c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.213+0000 7efc57400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc58071350 0x7efc58164c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:47670/0 (socket says 172.21.15.47:47670) 2024-02-19T15:22:20.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.213+0000 7efc4fe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efc5815d6e0 0x7efc58162740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:36940/0 (socket says 172.21.15.47:36940) 2024-02-19T15:22:20.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.213+0000 7efc57400640 1 -- 172.21.15.47:0/831326437 learned_addr learned my addr 172.21.15.47:0/831326437 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:20.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.213+0000 7efc56a00640 1 --1- 172.21.15.47:0/831326437 >> v1:172.21.15.116:6789/0 conn(0x7efc5815b1a0 0x7efc5815b580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:55952/0 (socket says 172.21.15.47:55952) 2024-02-19T15:22:20.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.213+0000 7efc56000640 1 -- 172.21.15.47:0/831326437 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291629831 0 0) 0x7efc580718c0 con 0x7efc5815b1a0 2024-02-19T15:22:20.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.213+0000 7efc4fe00640 1 -- 172.21.15.47:0/831326437 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc58071350 msgr2=0x7efc58164c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:20.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.213+0000 7efc4fe00640 1 --2- 172.21.15.47:0/831326437 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc58071350 0x7efc58164c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.213+0000 7efc4fe00640 1 -- 172.21.15.47:0/831326437 >> v1:172.21.15.116:6789/0 conn(0x7efc5815b1a0 legacy=0x7efc5815b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:20.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.213+0000 7efc4fe00640 1 -- 172.21.15.47:0/831326437 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc58072160 con 0x7efc5815d6e0 2024-02-19T15:22:20.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.213+0000 7efc57400640 1 --2- 172.21.15.47:0/831326437 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc58071350 0x7efc58164c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:20.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.217+0000 7efc4fe00640 1 --2- 172.21.15.47:0/831326437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc5815d6e0 0x7efc58162740 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efc400097f0 tx=0x7efc4001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0eabaeba184436b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:20.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.217+0000 7efc56000640 1 -- 172.21.15.47:0/831326437 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc400aa620 con 0x7efc5815d6e0 2024-02-19T15:22:20.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.217+0000 7efc56000640 1 -- 172.21.15.47:0/831326437 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc400aa780 con 0x7efc5815d6e0 2024-02-19T15:22:20.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.217+0000 7efc56000640 1 -- 172.21.15.47:0/831326437 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc400aab10 con 0x7efc5815d6e0 2024-02-19T15:22:20.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.217+0000 7efc5da00640 1 -- 172.21.15.47:0/831326437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc5815d6e0 msgr2=0x7efc58162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:20.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.217+0000 7efc5da00640 1 --2- 172.21.15.47:0/831326437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc5815d6e0 0x7efc58162740 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efc400097f0 tx=0x7efc4001c4a0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.217+0000 7efc5da00640 1 -- 172.21.15.47:0/831326437 shutdown_connections 2024-02-19T15:22:20.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.217+0000 7efc5da00640 1 --2- 172.21.15.47:0/831326437 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc58071350 0x7efc58164c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.217+0000 7efc5da00640 1 --2- 172.21.15.47:0/831326437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc5815d6e0 0x7efc58162740 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.217+0000 7efc5da00640 1 -- 172.21.15.47:0/831326437 >> 172.21.15.47:0/831326437 conn(0x7efc580b16a0 msgr2=0x7efc580b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:20.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.217+0000 7efc5da00640 1 -- 172.21.15.47:0/831326437 shutdown_connections 2024-02-19T15:22:20.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.217+0000 7efc5da00640 1 -- 172.21.15.47:0/831326437 wait complete. 2024-02-19T15:22:20.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.217+0000 7efc5da00640 1 Processor -- start 2024-02-19T15:22:20.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.217+0000 7efc5da00640 1 -- start start 2024-02-19T15:22:20.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.217+0000 7efc5da00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc58071350 0x7efc58146e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.217+0000 7efc5da00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc5815b1a0 0x7efc581572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.217+0000 7efc5da00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc5815d6e0 0x7efc58157820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.217+0000 7efc5da00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc58156e50 con 0x7efc58071350 2024-02-19T15:22:20.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.217+0000 7efc5da00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc58156240 con 0x7efc5815b1a0 2024-02-19T15:22:20.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.217+0000 7efc5da00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc58156520 con 0x7efc5815d6e0 2024-02-19T15:22:20.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.221+0000 7efc57400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc5815d6e0 0x7efc58157820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.221+0000 7efc4fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc5815b1a0 0x7efc581572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.221+0000 7efc56a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc58071350 0x7efc58146e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.221+0000 7efc56a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc58071350 0x7efc58146e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58058/0 (socket says 172.21.15.47:58058) 2024-02-19T15:22:20.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.221+0000 7efc4fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc5815b1a0 0x7efc581572e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:36952/0 (socket says 172.21.15.47:36952) 2024-02-19T15:22:20.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.221+0000 7efc56a00640 1 -- 172.21.15.47:0/3699214042 learned_addr learned my addr 172.21.15.47:0/3699214042 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:20.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.221+0000 7efc4fe00640 1 -- 172.21.15.47:0/3699214042 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc5815d6e0 msgr2=0x7efc58157820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:20.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.221+0000 7efc4fe00640 1 --2- 172.21.15.47:0/3699214042 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc5815d6e0 0x7efc58157820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.221+0000 7efc4fe00640 1 -- 172.21.15.47:0/3699214042 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc58071350 msgr2=0x7efc58146e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:20.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.221+0000 7efc4fe00640 1 --2- 172.21.15.47:0/3699214042 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc58071350 0x7efc58146e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.221+0000 7efc4fe00640 1 -- 172.21.15.47:0/3699214042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc50009b30 con 0x7efc5815b1a0 2024-02-19T15:22:20.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.221+0000 7efc4fe00640 1 --2- 172.21.15.47:0/3699214042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc5815b1a0 0x7efc581572e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efc400ab000 tx=0x7efc40019240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:20.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.221+0000 7efc4ea00640 1 -- 172.21.15.47:0/3699214042 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc400aabb0 con 0x7efc5815b1a0 2024-02-19T15:22:20.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.221+0000 7efc4ea00640 1 -- 172.21.15.47:0/3699214042 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc400aad10 con 0x7efc5815b1a0 2024-02-19T15:22:20.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.221+0000 7efc5da00640 1 -- 172.21.15.47:0/3699214042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc40009420 con 0x7efc5815b1a0 2024-02-19T15:22:20.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.221+0000 7efc4ea00640 1 -- 172.21.15.47:0/3699214042 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc40004250 con 0x7efc5815b1a0 2024-02-19T15:22:20.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.221+0000 7efc5da00640 1 -- 172.21.15.47:0/3699214042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc58156ae0 con 0x7efc5815b1a0 2024-02-19T15:22:20.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.221+0000 7efc4ea00640 1 -- 172.21.15.47:0/3699214042 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efc400bb070 con 0x7efc5815b1a0 2024-02-19T15:22:20.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc4ea00640 1 --2- 172.21.15.47:0/3699214042 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc34066e70 0x7efc34069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc4ea00640 1 -- 172.21.15.47:0/3699214042 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc4011edf0 con 0x7efc5815b1a0 2024-02-19T15:22:20.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc56a00640 1 --2- 172.21.15.47:0/3699214042 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc34066e70 0x7efc34069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc60ea1000 1 Processor -- start 2024-02-19T15:22:20.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc60ea1000 1 -- start start 2024-02-19T15:22:20.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc56a00640 1 --2- 172.21.15.47:0/3699214042 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc34066e70 0x7efc34069330 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efc48000f70 tx=0x7efc48007e20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:20.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc60ea1000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56081427f920 0x56081427fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc60ea1000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560814280240 0x5608142359f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc60ea1000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814235f30 0x560814236310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc60ea1000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608140fdd00 con 0x56081427f920 2024-02-19T15:22:20.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc60ea1000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56081408f430 con 0x560814280240 2024-02-19T15:22:20.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc60ea1000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x56081408c960 con 0x560814235f30 2024-02-19T15:22:20.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc56a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56081427f920 0x56081427fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc56a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56081427f920 0x56081427fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58062/0 (socket says 172.21.15.47:58062) 2024-02-19T15:22:20.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc56a00640 1 -- 172.21.15.47:0/3459886146 learned_addr learned my addr 172.21.15.47:0/3459886146 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:20.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc4fe00640 1 --2- 172.21.15.47:0/3459886146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560814280240 0x5608142359f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc57400640 1 --2- 172.21.15.47:0/3459886146 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814235f30 0x560814236310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc56a00640 1 -- 172.21.15.47:0/3459886146 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814235f30 msgr2=0x560814236310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:20.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc56a00640 1 --2- 172.21.15.47:0/3459886146 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814235f30 0x560814236310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc56a00640 1 -- 172.21.15.47:0/3459886146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560814280240 msgr2=0x5608142359f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:20.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc56a00640 1 --2- 172.21.15.47:0/3459886146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560814280240 0x5608142359f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc56a00640 1 -- 172.21.15.47:0/3459886146 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560813d7e890 con 0x56081427f920 2024-02-19T15:22:20.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc57400640 1 --2- 172.21.15.47:0/3459886146 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814235f30 0x560814236310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:20.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc56a00640 1 --2- 172.21.15.47:0/3459886146 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56081427f920 0x56081427fd00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efc48007470 tx=0x7efc48013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f44150f00767db6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:20.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc47400640 1 -- 172.21.15.47:0/3459886146 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc48013b80 con 0x56081427f920 2024-02-19T15:22:20.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc47400640 1 -- 172.21.15.47:0/3459886146 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc48013ce0 con 0x56081427f920 2024-02-19T15:22:20.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc47400640 1 -- 172.21.15.47:0/3459886146 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc48012550 con 0x56081427f920 2024-02-19T15:22:20.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc60ea1000 1 -- 172.21.15.47:0/3459886146 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56081427f920 msgr2=0x56081427fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:20.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc60ea1000 1 --2- 172.21.15.47:0/3459886146 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56081427f920 0x56081427fd00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efc48007470 tx=0x7efc48013000 comp rx=0 tx=0).stop 2024-02-19T15:22:20.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc60ea1000 1 -- 172.21.15.47:0/3459886146 shutdown_connections 2024-02-19T15:22:20.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc60ea1000 1 --2- 172.21.15.47:0/3459886146 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814235f30 0x560814236310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc60ea1000 1 --2- 172.21.15.47:0/3459886146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560814280240 0x5608142359f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc60ea1000 1 --2- 172.21.15.47:0/3459886146 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56081427f920 0x56081427fd00 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc60ea1000 1 -- 172.21.15.47:0/3459886146 >> 172.21.15.47:0/3459886146 conn(0x56081423c2a0 msgr2=0x5608142342d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:20.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc60ea1000 1 -- 172.21.15.47:0/3459886146 shutdown_connections 2024-02-19T15:22:20.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.225+0000 7efc60ea1000 1 -- 172.21.15.47:0/3459886146 wait complete. 2024-02-19T15:22:20.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc60ea1000 1 -- start start 2024-02-19T15:22:20.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc60ea1000 1 Processor -- start 2024-02-19T15:22:20.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc60ea1000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 0x56081423a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc60ea1000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56081427f920 0x560814280830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc60ea1000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814280240 0x560814280d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc60ea1000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x560814088140 con 0x560814235f30 2024-02-19T15:22:20.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc60ea1000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560814022630 con 0x56081427f920 2024-02-19T15:22:20.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc60ea1000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x5608140222d0 con 0x560814280240 2024-02-19T15:22:20.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc60ea1000 10 client.0 _ll_register_callbacks cb 0x5608141381c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:22:20.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc56a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 0x56081423a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc56a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 0x56081423a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58070/0 (socket says 172.21.15.47:58070) 2024-02-19T15:22:20.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc56a00640 1 -- 172.21.15.47:0/3806161244 learned_addr learned my addr 172.21.15.47:0/3806161244 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:20.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc57400640 1 --2- 172.21.15.47:0/3806161244 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814280240 0x560814280d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc4fe00640 1 --2- 172.21.15.47:0/3806161244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56081427f920 0x560814280830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc56a00640 1 -- 172.21.15.47:0/3806161244 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814280240 msgr2=0x560814280d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:20.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc56a00640 1 --2- 172.21.15.47:0/3806161244 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814280240 0x560814280d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc56a00640 1 -- 172.21.15.47:0/3806161244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56081427f920 msgr2=0x560814280830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:20.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc56a00640 1 --2- 172.21.15.47:0/3806161244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56081427f920 0x560814280830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc56a00640 1 -- 172.21.15.47:0/3806161244 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc48004c80 con 0x560814235f30 2024-02-19T15:22:20.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc56a00640 1 --2- 172.21.15.47:0/3806161244 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 0x56081423a6d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efc48001ec0 tx=0x7efc48000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67e736a3d9cd2895 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:20.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc3be00640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:22:20.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc3be00640 1 -- 172.21.15.47:0/3806161244 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc48020a60 con 0x560814235f30 2024-02-19T15:22:20.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc3be00640 1 -- 172.21.15.47:0/3806161244 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc4801b050 con 0x560814235f30 2024-02-19T15:22:20.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc60ea1000 1 -- 172.21.15.47:0/3806161244 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56081403b850 con 0x560814235f30 2024-02-19T15:22:20.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc3be00640 1 -- 172.21.15.47:0/3806161244 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc48020bc0 con 0x560814235f30 2024-02-19T15:22:20.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc60ea1000 10 client.4632 fetch_fsmap learned FSMap version 8 2024-02-19T15:22:20.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc60ea1000 1 -- 172.21.15.47:0/3806161244 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x560814047e70 con 0x560814235f30 2024-02-19T15:22:20.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc3be00640 1 -- 172.21.15.47:0/3806161244 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56081403b850 con 0x560814235f30 2024-02-19T15:22:20.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc3be00640 1 -- 172.21.15.47:0/3806161244 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7efc48012070 con 0x560814235f30 2024-02-19T15:22:20.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc60ea1000 10 client.4632 fetch_fsmap finished waiting for FSMap version 8 2024-02-19T15:22:20.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc60ea1000 10 client.4632 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T15:22:20.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc60ea1000 20 client.4632 populate_metadata read hostname 'smithi047' 2024-02-19T15:22:20.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc60ea1000 1 --2- 172.21.15.47:0/3806161244 >> [v2:172.21.15.116:6835/3920452343,v1:172.21.15.116:6837/3920452343] conn(0x56081423ac10 0x560814290bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc60ea1000 4 client.4632 mds_command: new command op to 4284 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T15:22:20.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc60ea1000 1 -- 172.21.15.47:0/3806161244 --> [v2:172.21.15.116:6835/3920452343,v1:172.21.15.116:6837/3920452343] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560814084f20 con 0x56081423ac10 2024-02-19T15:22:20.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.229+0000 7efc4fe00640 1 --2- 172.21.15.47:0/3806161244 >> [v2:172.21.15.116:6835/3920452343,v1:172.21.15.116:6837/3920452343] conn(0x56081423ac10 0x560814290bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.233+0000 7efc4fe00640 1 --2- 172.21.15.47:0/3806161244 >> [v2:172.21.15.116:6835/3920452343,v1:172.21.15.116:6837/3920452343] conn(0x56081423ac10 0x560814290bd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d54fe3cc0aaca843 server_cookie=c59feaf1f5779e8 in_seq=0 out_seq=0 2024-02-19T15:22:20.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.233+0000 7efc3be00640 10 client.4632 ms_handle_connect on v2:172.21.15.116:6835/3920452343 2024-02-19T15:22:20.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.233+0000 7efc3be00640 1 -- 172.21.15.47:0/3806161244 <== mds.0 v2:172.21.15.116:6835/3920452343 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560814084f20 con 0x56081423ac10 2024-02-19T15:22:20.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.233+0000 7efc3be00640 10 client.4632 handle_command_reply: tid=0 2024-02-19T15:22:20.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.233+0000 7efc60ea1000 1 client.4632 shutdown 2024-02-19T15:22:20.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.233+0000 7efc60ea1000 1 -- 172.21.15.47:0/3806161244 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 msgr2=0x56081423a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:20.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.233+0000 7efc60ea1000 1 --2- 172.21.15.47:0/3806161244 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 0x56081423a6d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efc48001ec0 tx=0x7efc48000ee0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.233+0000 7efc60ea1000 1 -- 172.21.15.47:0/3806161244 shutdown_connections 2024-02-19T15:22:20.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.233+0000 7efc60ea1000 1 --2- 172.21.15.47:0/3806161244 >> [v2:172.21.15.116:6835/3920452343,v1:172.21.15.116:6837/3920452343] conn(0x56081423ac10 0x560814290bd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.233+0000 7efc60ea1000 1 --2- 172.21.15.47:0/3806161244 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814280240 0x560814280d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.233+0000 7efc60ea1000 1 --2- 172.21.15.47:0/3806161244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56081427f920 0x560814280830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.233+0000 7efc3be00640 0 client.4632 ms_handle_reset on v2:172.21.15.116:6835/3920452343 2024-02-19T15:22:20.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.233+0000 7efc60ea1000 1 --2- 172.21.15.47:0/3806161244 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 0x56081423a6d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.233+0000 7efc60ea1000 1 -- 172.21.15.47:0/3806161244 >> 172.21.15.47:0/3806161244 conn(0x56081423c2a0 msgr2=0x5608142349e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:20.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.233+0000 7efc60ea1000 1 -- 172.21.15.47:0/3806161244 shutdown_connections 2024-02-19T15:22:20.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.233+0000 7efc60ea1000 1 -- 172.21.15.47:0/3806161244 wait complete. 2024-02-19T15:22:20.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.233+0000 7efc60ea1000 20 client.4632 trim_cache size 0 max 16384 2024-02-19T15:22:20.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.249+0000 7efc60ea1000 1 Processor -- start 2024-02-19T15:22:20.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.249+0000 7efc60ea1000 1 -- start start 2024-02-19T15:22:20.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.249+0000 7efc60ea1000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 0x5608142934e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.249+0000 7efc60ea1000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56081427f920 0x560814293a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.249+0000 7efc60ea1000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814280240 0x5608142c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.249+0000 7efc60ea1000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x560813d3f940 con 0x560814235f30 2024-02-19T15:22:20.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.249+0000 7efc60ea1000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560814020660 con 0x56081427f920 2024-02-19T15:22:20.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.249+0000 7efc60ea1000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x56081404aa60 con 0x560814280240 2024-02-19T15:22:20.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.249+0000 7efc56a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 0x5608142934e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.249+0000 7efc56a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 0x5608142934e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58086/0 (socket says 172.21.15.47:58086) 2024-02-19T15:22:20.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.249+0000 7efc56a00640 1 -- 172.21.15.47:0/2215354041 learned_addr learned my addr 172.21.15.47:0/2215354041 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:20.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.249+0000 7efc57400640 1 --2- 172.21.15.47:0/2215354041 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814280240 0x5608142c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.249+0000 7efc4fe00640 1 --2- 172.21.15.47:0/2215354041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56081427f920 0x560814293a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.249+0000 7efc56a00640 1 -- 172.21.15.47:0/2215354041 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814280240 msgr2=0x5608142c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:20.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.249+0000 7efc56a00640 1 --2- 172.21.15.47:0/2215354041 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814280240 0x5608142c4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.249+0000 7efc56a00640 1 -- 172.21.15.47:0/2215354041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56081427f920 msgr2=0x560814293a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:20.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.249+0000 7efc56a00640 1 --2- 172.21.15.47:0/2215354041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56081427f920 0x560814293a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.249+0000 7efc56a00640 1 -- 172.21.15.47:0/2215354041 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56081402de10 con 0x560814235f30 2024-02-19T15:22:20.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc57400640 1 --2- 172.21.15.47:0/2215354041 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814280240 0x5608142c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:20.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc56a00640 1 --2- 172.21.15.47:0/2215354041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 0x5608142934e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efc48007360 tx=0x7efc4802a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57d032106bce4af1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:20.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc46a00640 1 -- 172.21.15.47:0/2215354041 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc48013e90 con 0x560814235f30 2024-02-19T15:22:20.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc46a00640 1 -- 172.21.15.47:0/2215354041 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4800b040 con 0x560814235f30 2024-02-19T15:22:20.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc46a00640 1 -- 172.21.15.47:0/2215354041 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4802b440 con 0x560814235f30 2024-02-19T15:22:20.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc60ea1000 1 -- 172.21.15.47:0/2215354041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 msgr2=0x5608142934e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:20.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc60ea1000 1 --2- 172.21.15.47:0/2215354041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 0x5608142934e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efc48007360 tx=0x7efc4802a000 comp rx=0 tx=0).stop 2024-02-19T15:22:20.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc60ea1000 1 -- 172.21.15.47:0/2215354041 shutdown_connections 2024-02-19T15:22:20.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc60ea1000 1 --2- 172.21.15.47:0/2215354041 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814280240 0x5608142c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc60ea1000 1 --2- 172.21.15.47:0/2215354041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56081427f920 0x560814293a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc60ea1000 1 --2- 172.21.15.47:0/2215354041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 0x5608142934e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc60ea1000 1 -- 172.21.15.47:0/2215354041 >> 172.21.15.47:0/2215354041 conn(0x56081423c2a0 msgr2=0x560814291dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:20.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc60ea1000 1 -- 172.21.15.47:0/2215354041 shutdown_connections 2024-02-19T15:22:20.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc60ea1000 1 -- 172.21.15.47:0/2215354041 wait complete. 2024-02-19T15:22:20.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc60ea1000 1 -- start start 2024-02-19T15:22:20.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc60ea1000 1 Processor -- start 2024-02-19T15:22:20.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc60ea1000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 0x5608142c8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc60ea1000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56081427f920 0x5608142c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc60ea1000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814280240 0x5608142c9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc60ea1000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x56081408c140 con 0x560814235f30 2024-02-19T15:22:20.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc60ea1000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608140c81a0 con 0x56081427f920 2024-02-19T15:22:20.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc60ea1000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x56081408fe80 con 0x560814280240 2024-02-19T15:22:20.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc60ea1000 10 client.0 _ll_register_callbacks cb 0x7efc48024070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:22:20.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc56a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 0x5608142c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc56a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 0x5608142c8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58098/0 (socket says 172.21.15.47:58098) 2024-02-19T15:22:20.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc56a00640 1 -- 172.21.15.47:0/842333107 learned_addr learned my addr 172.21.15.47:0/842333107 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:20.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc57400640 1 --2- 172.21.15.47:0/842333107 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814280240 0x5608142c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc4fe00640 1 --2- 172.21.15.47:0/842333107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56081427f920 0x5608142c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc56a00640 1 -- 172.21.15.47:0/842333107 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814280240 msgr2=0x5608142c9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:20.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc56a00640 1 --2- 172.21.15.47:0/842333107 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814280240 0x5608142c9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc56a00640 1 -- 172.21.15.47:0/842333107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56081427f920 msgr2=0x5608142c8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:20.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc56a00640 1 --2- 172.21.15.47:0/842333107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56081427f920 0x5608142c8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc56a00640 1 -- 172.21.15.47:0/842333107 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560814047e70 con 0x560814235f30 2024-02-19T15:22:20.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc56a00640 1 --2- 172.21.15.47:0/842333107 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 0x5608142c8570 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efc480137c0 tx=0x7efc48009030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaf41e3cd60cb12e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:20.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.253+0000 7efc3be00640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:22:20.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc3be00640 1 -- 172.21.15.47:0/842333107 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc48049070 con 0x560814235f30 2024-02-19T15:22:20.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 1 -- 172.21.15.47:0/842333107 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56081400de80 con 0x560814235f30 2024-02-19T15:22:20.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc3be00640 1 -- 172.21.15.47:0/842333107 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc4801b070 con 0x560814235f30 2024-02-19T15:22:20.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc3be00640 1 -- 172.21.15.47:0/842333107 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc4802b440 con 0x560814235f30 2024-02-19T15:22:20.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 10 client.4638 fetch_fsmap learned FSMap version 8 2024-02-19T15:22:20.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 1 -- 172.21.15.47:0/842333107 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x560813d7ecc0 con 0x560814235f30 2024-02-19T15:22:20.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc3be00640 1 -- 172.21.15.47:0/842333107 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56081400de80 con 0x560814235f30 2024-02-19T15:22:20.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc3be00640 1 -- 172.21.15.47:0/842333107 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7efc48012040 con 0x560814235f30 2024-02-19T15:22:20.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 10 client.4638 fetch_fsmap finished waiting for FSMap version 8 2024-02-19T15:22:20.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 10 client.4638 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T15:22:20.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 20 client.4638 populate_metadata read hostname 'smithi047' 2024-02-19T15:22:20.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 1 --2- 172.21.15.47:0/842333107 >> [v2:172.21.15.116:6835/3920452343,v1:172.21.15.116:6837/3920452343] conn(0x5608142c9a70 0x5608142c9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 4 client.4638 mds_command: new command op to 4284 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T15:22:20.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 1 -- 172.21.15.47:0/842333107 --> [v2:172.21.15.116:6835/3920452343,v1:172.21.15.116:6837/3920452343] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560814050fa0 con 0x5608142c9a70 2024-02-19T15:22:20.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc4fe00640 1 --2- 172.21.15.47:0/842333107 >> [v2:172.21.15.116:6835/3920452343,v1:172.21.15.116:6837/3920452343] conn(0x5608142c9a70 0x5608142c9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc3be00640 10 client.4638 ms_handle_connect on v2:172.21.15.116:6835/3920452343 2024-02-19T15:22:20.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc4fe00640 1 --2- 172.21.15.47:0/842333107 >> [v2:172.21.15.116:6835/3920452343,v1:172.21.15.116:6837/3920452343] conn(0x5608142c9a70 0x5608142c9e50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d77995478095246 server_cookie=9830851a522570d0 in_seq=0 out_seq=0 2024-02-19T15:22:20.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc3be00640 1 -- 172.21.15.47:0/842333107 <== mds.0 v2:172.21.15.116:6835/3920452343 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x560814050fa0 con 0x5608142c9a70 2024-02-19T15:22:20.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc3be00640 10 client.4638 handle_command_reply: tid=0 2024-02-19T15:22:20.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 1 client.4638 shutdown 2024-02-19T15:22:20.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 1 -- 172.21.15.47:0/842333107 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 msgr2=0x5608142c8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:20.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 1 --2- 172.21.15.47:0/842333107 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 0x5608142c8570 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efc480137c0 tx=0x7efc48009030 comp rx=0 tx=0).stop 2024-02-19T15:22:20.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 1 -- 172.21.15.47:0/842333107 shutdown_connections 2024-02-19T15:22:20.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 1 --2- 172.21.15.47:0/842333107 >> [v2:172.21.15.116:6835/3920452343,v1:172.21.15.116:6837/3920452343] conn(0x5608142c9a70 0x5608142c9e50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 1 --2- 172.21.15.47:0/842333107 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560814280240 0x5608142c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 1 --2- 172.21.15.47:0/842333107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56081427f920 0x5608142c8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc3be00640 0 client.4638 ms_handle_reset on v2:172.21.15.116:6835/3920452343 2024-02-19T15:22:20.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 1 --2- 172.21.15.47:0/842333107 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560814235f30 0x5608142c8570 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 1 -- 172.21.15.47:0/842333107 >> 172.21.15.47:0/842333107 conn(0x56081423c2a0 msgr2=0x5608142924d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:20.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 1 -- 172.21.15.47:0/842333107 shutdown_connections 2024-02-19T15:22:20.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 1 -- 172.21.15.47:0/842333107 wait complete. 2024-02-19T15:22:20.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc60ea1000 20 client.4638 trim_cache size 0 max 16384 2024-02-19T15:22:20.264 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:22:20.264 INFO:teuthology.orchestra.run.smithi047.stdout: "cluster_fsid": "ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e", 2024-02-19T15:22:20.264 INFO:teuthology.orchestra.run.smithi047.stdout: "whoami": 0, 2024-02-19T15:22:20.264 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 4284, 2024-02-19T15:22:20.264 INFO:teuthology.orchestra.run.smithi047.stdout: "want_state": "up:active", 2024-02-19T15:22:20.264 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "up:active", 2024-02-19T15:22:20.264 INFO:teuthology.orchestra.run.smithi047.stdout: "fs_name": "cephfs", 2024-02-19T15:22:20.265 INFO:teuthology.orchestra.run.smithi047.stdout: "rank_uptime": 14.4300076, 2024-02-19T15:22:20.265 INFO:teuthology.orchestra.run.smithi047.stdout: "mdsmap_epoch": 8, 2024-02-19T15:22:20.265 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch": 21, 2024-02-19T15:22:20.265 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch_barrier": 21, 2024-02-19T15:22:20.265 INFO:teuthology.orchestra.run.smithi047.stdout: "uptime": 21.918555937000001 2024-02-19T15:22:20.265 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:22:20.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc5da00640 1 -- 172.21.15.47:0/3699214042 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc34066e70 msgr2=0x7efc34069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:20.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc5da00640 1 --2- 172.21.15.47:0/3699214042 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc34066e70 0x7efc34069330 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efc48000f70 tx=0x7efc48007e20 comp rx=0 tx=0).stop 2024-02-19T15:22:20.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc5da00640 1 -- 172.21.15.47:0/3699214042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc5815b1a0 msgr2=0x7efc581572e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:20.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc5da00640 1 --2- 172.21.15.47:0/3699214042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc5815b1a0 0x7efc581572e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efc400ab000 tx=0x7efc40019240 comp rx=0 tx=0).stop 2024-02-19T15:22:20.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc5da00640 1 -- 172.21.15.47:0/3699214042 shutdown_connections 2024-02-19T15:22:20.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc5da00640 1 --2- 172.21.15.47:0/3699214042 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc34066e70 0x7efc34069330 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc5da00640 1 --2- 172.21.15.47:0/3699214042 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc5815d6e0 0x7efc58157820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc5da00640 1 --2- 172.21.15.47:0/3699214042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc5815b1a0 0x7efc581572e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc5da00640 1 --2- 172.21.15.47:0/3699214042 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc58071350 0x7efc58146e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc5da00640 1 -- 172.21.15.47:0/3699214042 >> 172.21.15.47:0/3699214042 conn(0x7efc580b16a0 msgr2=0x7efc58070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:20.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc5da00640 1 -- 172.21.15.47:0/3699214042 shutdown_connections 2024-02-19T15:22:20.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.257+0000 7efc5da00640 1 -- 172.21.15.47:0/3699214042 wait complete. 2024-02-19T15:22:20.280 INFO:teuthology.run_tasks:Running task kclient... 2024-02-19T15:22:20.302 INFO:tasks.kclient:Mounting kernel clients... 2024-02-19T15:22:20.302 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-02-19T15:22:20.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:20.302 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list 2024-02-19T15:22:20.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:20.312 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link delete ceph-brx 2024-02-19T15:22:20.376 INFO:teuthology.orchestra.run.smithi047.stderr:Cannot find device "ceph-brx" 2024-02-19T15:22:20.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:22:20.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:20.378 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list 2024-02-19T15:22:20.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:20.430 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link delete ceph-brx 2024-02-19T15:22:20.500 INFO:teuthology.orchestra.run.smithi047.stderr:Cannot find device "ceph-brx" 2024-02-19T15:22:20.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:22:20.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:20.502 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-02-19T15:22:20.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:20.512 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link delete ceph-brx 2024-02-19T15:22:20.576 INFO:teuthology.orchestra.run.smithi116.stderr:Cannot find device "ceph-brx" 2024-02-19T15:22:20.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:22:20.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:20.578 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-02-19T15:22:20.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:20.630 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link delete ceph-brx 2024-02-19T15:22:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:Cannot find device "ceph-brx" 2024-02-19T15:22:20.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:22:20.697 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-02-19T15:22:20.698 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-19T15:22:20.698 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-19T15:22:20.698 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-02-19T15:22:20.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.833+0000 7f801ec00640 1 Processor -- start 2024-02-19T15:22:20.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.833+0000 7f801ec00640 1 -- start start 2024-02-19T15:22:20.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.837+0000 7f801ec00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80180cd430 0x7f801806fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.837+0000 7f801ec00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f801806dc70 0x7f801806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.837+0000 7f801ec00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8018071910 con 0x7f801806aeb0 2024-02-19T15:22:20.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.837+0000 7f801ec00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8018071bf0 con 0x7f801806dc70 2024-02-19T15:22:20.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.837+0000 7f801ec00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8018071ed0 con 0x7f80180cd430 2024-02-19T15:22:20.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.837+0000 7f801e200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f801806dc70 0x7f801806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.837+0000 7f801d800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f801806aeb0 0x7f801806b290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35454/0 (socket says 172.21.15.47:35454) 2024-02-19T15:22:20.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.837+0000 7f801e200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f801806dc70 0x7f801806e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58110/0 (socket says 172.21.15.47:58110) 2024-02-19T15:22:20.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.837+0000 7f801d800640 1 -- 172.21.15.47:0/3699423841 learned_addr learned my addr 172.21.15.47:0/3699423841 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:20.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.837+0000 7f801ce00640 1 --2- 172.21.15.47:0/3699423841 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80180cd430 0x7f801806fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.837+0000 7f801e200640 1 -- 172.21.15.47:0/3699423841 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80180cd430 msgr2=0x7f801806fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:20.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.837+0000 7f801e200640 1 --2- 172.21.15.47:0/3699423841 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80180cd430 0x7f801806fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.837+0000 7f801e200640 1 -- 172.21.15.47:0/3699423841 >> v1:172.21.15.47:6789/0 conn(0x7f801806aeb0 legacy=0x7f801806b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:20.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.837+0000 7f801e200640 1 -- 172.21.15.47:0/3699423841 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80180721b0 con 0x7f801806dc70 2024-02-19T15:22:20.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.841+0000 7f801e200640 1 --2- 172.21.15.47:0/3699423841 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f801806dc70 0x7f801806e0c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8010009d90 tx=0x7f801001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ed758175c5547a5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:20.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.841+0000 7f8017e00640 1 -- 172.21.15.47:0/3699423841 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80100a5910 con 0x7f801806dc70 2024-02-19T15:22:20.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.841+0000 7f8017e00640 1 -- 172.21.15.47:0/3699423841 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80100a5a70 con 0x7f801806dc70 2024-02-19T15:22:20.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.841+0000 7f8017e00640 1 -- 172.21.15.47:0/3699423841 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80100a5e00 con 0x7f801806dc70 2024-02-19T15:22:20.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.841+0000 7f801ec00640 1 -- 172.21.15.47:0/3699423841 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f801806dc70 msgr2=0x7f801806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:20.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.841+0000 7f801ec00640 1 --2- 172.21.15.47:0/3699423841 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f801806dc70 0x7f801806e0c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8010009d90 tx=0x7f801001c8b0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.841+0000 7f801ec00640 1 -- 172.21.15.47:0/3699423841 shutdown_connections 2024-02-19T15:22:20.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.841+0000 7f801ec00640 1 --2- 172.21.15.47:0/3699423841 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f801806dc70 0x7f801806e0c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.841+0000 7f801ec00640 1 --2- 172.21.15.47:0/3699423841 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80180cd430 0x7f801806fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.841+0000 7f801ec00640 1 -- 172.21.15.47:0/3699423841 >> 172.21.15.47:0/3699423841 conn(0x7f80180b24f0 msgr2=0x7f80180b2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:20.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.841+0000 7f801ec00640 1 -- 172.21.15.47:0/3699423841 shutdown_connections 2024-02-19T15:22:20.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.841+0000 7f801ec00640 1 -- 172.21.15.47:0/3699423841 wait complete. 2024-02-19T15:22:20.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.841+0000 7f801ec00640 1 Processor -- start 2024-02-19T15:22:20.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.841+0000 7f801ec00640 1 -- start start 2024-02-19T15:22:20.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f801ec00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f801806aeb0 0x7f80180b0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f801ec00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f801806dc70 0x7f8018151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f801ec00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f801814ef00 0x7f8018150e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f801ec00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8018153ce0 con 0x7f801806dc70 2024-02-19T15:22:20.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f801ec00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8018156f80 con 0x7f801806aeb0 2024-02-19T15:22:20.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f801ec00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80180b1680 con 0x7f801814ef00 2024-02-19T15:22:20.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f801ce00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f801806dc70 0x7f8018151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f801ce00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f801806dc70 0x7f8018151ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58116/0 (socket says 172.21.15.47:58116) 2024-02-19T15:22:20.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f801ce00640 1 -- 172.21.15.47:0/2250512201 learned_addr learned my addr 172.21.15.47:0/2250512201 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:20.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f801d800640 1 --2- 172.21.15.47:0/2250512201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f801806aeb0 0x7f80180b0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f801e200640 1 --2- 172.21.15.47:0/2250512201 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f801814ef00 0x7f8018150e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f801ce00640 1 -- 172.21.15.47:0/2250512201 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f801814ef00 msgr2=0x7f8018150e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:20.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f801ce00640 1 --2- 172.21.15.47:0/2250512201 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f801814ef00 0x7f8018150e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f801ce00640 1 -- 172.21.15.47:0/2250512201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f801806aeb0 msgr2=0x7f80180b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:20.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f801ce00640 1 --2- 172.21.15.47:0/2250512201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f801806aeb0 0x7f80180b0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:20.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f801ce00640 1 -- 172.21.15.47:0/2250512201 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8010009a70 con 0x7f801806dc70 2024-02-19T15:22:20.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f801ce00640 1 --2- 172.21.15.47:0/2250512201 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f801806dc70 0x7f8018151ba0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f800400e870 tx=0x7f800400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:20.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f8015600640 1 -- 172.21.15.47:0/2250512201 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8004009510 con 0x7f801806dc70 2024-02-19T15:22:20.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f8015600640 1 -- 172.21.15.47:0/2250512201 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f800400eec0 con 0x7f801806dc70 2024-02-19T15:22:20.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f8015600640 1 -- 172.21.15.47:0/2250512201 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800400cb40 con 0x7f801806dc70 2024-02-19T15:22:20.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f801ec00640 1 -- 172.21.15.47:0/2250512201 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8018153340 con 0x7f801806dc70 2024-02-19T15:22:20.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f801ec00640 1 -- 172.21.15.47:0/2250512201 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8018153900 con 0x7f801806dc70 2024-02-19T15:22:20.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.845+0000 7f801ec00640 1 -- 172.21.15.47:0/2250512201 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8018153ac0 con 0x7f801806dc70 2024-02-19T15:22:20.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.849+0000 7f8015600640 1 -- 172.21.15.47:0/2250512201 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f800400ccc0 con 0x7f801806dc70 2024-02-19T15:22:20.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.849+0000 7f8015600640 1 --2- 172.21.15.47:0/2250512201 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ff8066e90 0x7f7ff8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:20.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.849+0000 7f8015600640 1 -- 172.21.15.47:0/2250512201 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f80040849a0 con 0x7f801806dc70 2024-02-19T15:22:20.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.849+0000 7f801d800640 1 --2- 172.21.15.47:0/2250512201 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ff8066e90 0x7f7ff8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:20.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.849+0000 7f801d800640 1 --2- 172.21.15.47:0/2250512201 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ff8066e90 0x7f7ff8069350 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f80180b5b10 tx=0x7f800c007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:20.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:20.853+0000 7f8015600640 1 -- 172.21.15.47:0/2250512201 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8004051920 con 0x7f801806dc70 2024-02-19T15:22:21.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:21.053+0000 7f801ec00640 1 -- 172.21.15.47:0/2250512201 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8018153e20 con 0x7f801806dc70 2024-02-19T15:22:21.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:21.053+0000 7f8015600640 1 -- 172.21.15.47:0/2250512201 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8018153e20 con 0x7f801806dc70 2024-02-19T15:22:21.058 INFO:teuthology.orchestra.run.smithi047.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-02-19T15:22:21.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:21.057+0000 7f801ec00640 1 -- 172.21.15.47:0/2250512201 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ff8066e90 msgr2=0x7f7ff8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:21.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:21.057+0000 7f801ec00640 1 --2- 172.21.15.47:0/2250512201 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ff8066e90 0x7f7ff8069350 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f80180b5b10 tx=0x7f800c007e90 comp rx=0 tx=0).stop 2024-02-19T15:22:21.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:21.057+0000 7f801ec00640 1 -- 172.21.15.47:0/2250512201 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f801806dc70 msgr2=0x7f8018151ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:21.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:21.057+0000 7f801ec00640 1 --2- 172.21.15.47:0/2250512201 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f801806dc70 0x7f8018151ba0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f800400e870 tx=0x7f800400ec70 comp rx=0 tx=0).stop 2024-02-19T15:22:21.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:21.057+0000 7f801ec00640 1 -- 172.21.15.47:0/2250512201 shutdown_connections 2024-02-19T15:22:21.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:21.057+0000 7f801ec00640 1 --2- 172.21.15.47:0/2250512201 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ff8066e90 0x7f7ff8069350 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:21.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:21.057+0000 7f801ec00640 1 --2- 172.21.15.47:0/2250512201 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f801814ef00 0x7f8018150e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:21.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:21.057+0000 7f801ec00640 1 --2- 172.21.15.47:0/2250512201 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f801806dc70 0x7f8018151ba0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:21.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:21.057+0000 7f801ec00640 1 --2- 172.21.15.47:0/2250512201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f801806aeb0 0x7f80180b0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:21.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:21.057+0000 7f801ec00640 1 -- 172.21.15.47:0/2250512201 >> 172.21.15.47:0/2250512201 conn(0x7f80180b24f0 msgr2=0x7f801806b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:21.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:21.057+0000 7f801ec00640 1 -- 172.21.15.47:0/2250512201 shutdown_connections 2024-02-19T15:22:21.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:21.057+0000 7f801ec00640 1 -- 172.21.15.47:0/2250512201 wait complete. 2024-02-19T15:22:21.079 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-19T15:22:21.079 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T15:22:21.079 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi047.front.sepia.ceph.com 2024-02-19T15:22:21.079 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-19T15:22:21.079 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-19T15:22:21.079 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-19T15:22:21.079 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-19T15:22:21.079 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T15:22:21.080 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:22:21.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:21.080 DEBUG:teuthology.orchestra.run.smithi047:> ip addr 2024-02-19T15:22:21.090 INFO:teuthology.orchestra.run.smithi047.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T15:22:21.090 INFO:teuthology.orchestra.run.smithi047.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T15:22:21.090 INFO:teuthology.orchestra.run.smithi047.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T15:22:21.090 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:21.090 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 ::1/128 scope host 2024-02-19T15:22:21.090 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:21.090 INFO:teuthology.orchestra.run.smithi047.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:22:21.090 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:78 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:21.090 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f0 2024-02-19T15:22:21.090 INFO:teuthology.orchestra.run.smithi047.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T15:22:21.090 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8e brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:21.090 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f0 2024-02-19T15:22:21.090 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.21.15.47/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-02-19T15:22:21.091 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft 42474sec preferred_lft 42474sec 2024-02-19T15:22:21.091 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::ec4:7aff:febd:148e/64 scope link 2024-02-19T15:22:21.091 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:21.091 INFO:teuthology.orchestra.run.smithi047.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:22:21.091 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:79 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:21.091 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f1 2024-02-19T15:22:21.091 INFO:teuthology.orchestra.run.smithi047.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:22:21.091 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8f brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:21.091 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f1 2024-02-19T15:22:21.091 INFO:teuthology.orchestra.run.smithi047.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-02-19T15:22:21.091 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 02:42:47:cf:b1:62 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:21.091 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-02-19T15:22:21.091 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:21.092 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-19T15:22:21.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:21.092 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2024-02-19T15:22:21.092 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2024-02-19T15:22:21.092 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link add name ceph-brx type bridge 2024-02-19T15:22:21.092 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip addr flush dev ceph-brx 2024-02-19T15:22:21.092 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set ceph-brx up 2024-02-19T15:22:21.092 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-19T15:22:21.093 DEBUG:teuthology.orchestra.run.smithi047:> ') 2024-02-19T15:22:21.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:21.200 DEBUG:teuthology.orchestra.run.smithi047:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T15:22:21.219 INFO:teuthology.orchestra.run.smithi047.stdout:1 2024-02-19T15:22:21.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:21.221 DEBUG:teuthology.orchestra.run.smithi047:> route 2024-02-19T15:22:21.284 INFO:teuthology.orchestra.run.smithi047.stdout:Kernel IP routing table 2024-02-19T15:22:21.284 INFO:teuthology.orchestra.run.smithi047.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T15:22:21.284 INFO:teuthology.orchestra.run.smithi047.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-02-19T15:22:21.284 INFO:teuthology.orchestra.run.smithi047.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-02-19T15:22:21.285 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-02-19T15:22:21.285 INFO:teuthology.orchestra.run.smithi047.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T15:22:21.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:21.285 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2024-02-19T15:22:21.285 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2024-02-19T15:22:21.285 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-02-19T15:22:21.285 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-02-19T15:22:21.285 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-02-19T15:22:21.285 DEBUG:teuthology.orchestra.run.smithi047:> ') 2024-02-19T15:22:21.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:21.376 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list 2024-02-19T15:22:21.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:21.385 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list-id 2024-02-19T15:22:21.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:21.434 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2024-02-19T15:22:21.434 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2024-02-19T15:22:21.434 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-19T15:22:21.434 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-02-19T15:22:21.434 DEBUG:teuthology.orchestra.run.smithi047:> ') 2024-02-19T15:22:21.518 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-02-19T15:22:21.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:21.518 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2024-02-19T15:22:21.518 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2024-02-19T15:22:21.518 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-02-19T15:22:21.518 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-19T15:22:21.518 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-02-19T15:22:21.518 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-02-19T15:22:21.518 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-02-19T15:22:21.518 DEBUG:teuthology.orchestra.run.smithi047:> ') 2024-02-19T15:22:21.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:21.682 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2024-02-19T15:22:21.682 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2024-02-19T15:22:21.682 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set brx.0 up 2024-02-19T15:22:21.682 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set dev brx.0 master ceph-brx 2024-02-19T15:22:21.682 DEBUG:teuthology.orchestra.run.smithi047:> ') 2024-02-19T15:22:21.721 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:22:21.722 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-19T15:22:21.774 INFO:teuthology.orchestra.run.smithi047.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:22:21.774 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:22:21.774 DEBUG:teuthology.orchestra.run.smithi047:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:22:21.825 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-19T15:22:21.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:21.825 DEBUG:teuthology.orchestra.run.smithi047:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:22:21.923 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:22:21.923 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:22:21.923 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:22:22.489 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:22:22.490 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:22:22.490 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:22:22.490 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:22:22.490 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:22:22.502 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:22:22.502 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:22:22.502 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:22:22.502 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:22:22.502 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:22:22.502 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790" 2024-02-19T15:22:22.502 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790:/ 2024-02-19T15:22:22.502 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e" will pass to kernel 2024-02-19T15:22:22.506 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T15:22:22.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:22.506 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:22:22.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:22.528 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:22:22.601 INFO:teuthology.orchestra.run.smithi047.stdout:client4650 2024-02-19T15:22:22.601 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:22:22.601 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:22:22.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:22.654 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:22:22.718 INFO:teuthology.orchestra.run.smithi047.stdout:client4650 2024-02-19T15:22:22.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:22.718 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:22:22.783 INFO:teuthology.orchestra.run.smithi047.stdout:client4650 2024-02-19T15:22:22.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:22.783 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e.client4650/status') 2024-02-19T15:22:22.856 INFO:teuthology.orchestra.run.smithi047.stdout:instance: client.4650 (3)192.168.0.1:0/4048368170 2024-02-19T15:22:22.856 INFO:teuthology.orchestra.run.smithi047.stdout:blocklisted: false 2024-02-19T15:22:22.857 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2024-02-19T15:22:22.857 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2024-02-19T15:22:22.857 INFO:teuthology.orchestra.run.smithi047.stderr:69 bytes copied, 7.3631e-05 s, 937 kB/s 2024-02-19T15:22:22.859 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2024-02-19T15:22:22.859 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-19T15:22:22.859 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-02-19T15:22:22.859 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-02-19T15:22:23.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.037+0000 7f7cc4800640 1 Processor -- start 2024-02-19T15:22:23.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.037+0000 7f7cc4800640 1 -- start start 2024-02-19T15:22:23.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.041+0000 7f7cc4800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7cbc15d6e0 0x7f7cbc162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:23.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.041+0000 7f7cc4800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7cbc071350 0x7f7cbc164c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:23.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.041+0000 7f7cc4800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cbc0718c0 con 0x7f7cbc15b1a0 2024-02-19T15:22:23.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.041+0000 7f7cc4800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cbc071ba0 con 0x7f7cbc071350 2024-02-19T15:22:23.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.041+0000 7f7cc4800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc071e80 con 0x7f7cbc15d6e0 2024-02-19T15:22:23.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.041+0000 7f7cc2800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7cbc071350 0x7f7cbc164c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:23.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.041+0000 7f7cc1e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f7cbc15b1a0 0x7f7cbc15b580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35470/0 (socket says 172.21.15.47:35470) 2024-02-19T15:22:23.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.041+0000 7f7cc1e00640 1 -- 172.21.15.47:0/749899977 learned_addr learned my addr 172.21.15.47:0/749899977 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:23.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.041+0000 7f7cc1400640 1 --2- 172.21.15.47:0/749899977 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7cbc15d6e0 0x7f7cbc162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:23.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.041+0000 7f7cc2800640 1 -- 172.21.15.47:0/749899977 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7cbc15d6e0 msgr2=0x7f7cbc162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:23.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.041+0000 7f7cc2800640 1 --2- 172.21.15.47:0/749899977 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7cbc15d6e0 0x7f7cbc162740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:23.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.041+0000 7f7cc2800640 1 -- 172.21.15.47:0/749899977 >> v1:172.21.15.47:6789/0 conn(0x7f7cbc15b1a0 legacy=0x7f7cbc15b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:23.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.041+0000 7f7cc2800640 1 -- 172.21.15.47:0/749899977 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cbc072160 con 0x7f7cbc071350 2024-02-19T15:22:23.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.041+0000 7f7cc0a00640 1 -- 172.21.15.47:0/749899977 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583324713 0 0) 0x7f7cbc0718c0 con 0x7f7cbc15b1a0 2024-02-19T15:22:23.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.045+0000 7f7cc2800640 1 --2- 172.21.15.47:0/749899977 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cbc071350 0x7f7cbc164c90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8009d90 tx=0x7f7cb801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9d5c06e39454951 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:23.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.045+0000 7f7cc0a00640 1 -- 172.21.15.47:0/749899977 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb80a5a50 con 0x7f7cbc071350 2024-02-19T15:22:23.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.045+0000 7f7cc0a00640 1 -- 172.21.15.47:0/749899977 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb80a5bb0 con 0x7f7cbc071350 2024-02-19T15:22:23.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.045+0000 7f7cc0a00640 1 -- 172.21.15.47:0/749899977 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb80ad440 con 0x7f7cbc071350 2024-02-19T15:22:23.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.045+0000 7f7cc4800640 1 -- 172.21.15.47:0/749899977 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cbc071350 msgr2=0x7f7cbc164c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:23.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.045+0000 7f7cc4800640 1 --2- 172.21.15.47:0/749899977 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cbc071350 0x7f7cbc164c90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8009d90 tx=0x7f7cb801c970 comp rx=0 tx=0).stop 2024-02-19T15:22:23.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.045+0000 7f7cc4800640 1 -- 172.21.15.47:0/749899977 shutdown_connections 2024-02-19T15:22:23.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.045+0000 7f7cc4800640 1 --2- 172.21.15.47:0/749899977 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cbc071350 0x7f7cbc164c90 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:23.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.045+0000 7f7cc4800640 1 --2- 172.21.15.47:0/749899977 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7cbc15d6e0 0x7f7cbc162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:23.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.045+0000 7f7cc4800640 1 -- 172.21.15.47:0/749899977 >> 172.21.15.47:0/749899977 conn(0x7f7cbc0b16a0 msgr2=0x7f7cbc0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:23.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.045+0000 7f7cc4800640 1 -- 172.21.15.47:0/749899977 shutdown_connections 2024-02-19T15:22:23.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.045+0000 7f7cc4800640 1 -- 172.21.15.47:0/749899977 wait complete. 2024-02-19T15:22:23.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.045+0000 7f7cc4800640 1 Processor -- start 2024-02-19T15:22:23.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.045+0000 7f7cc4800640 1 -- start start 2024-02-19T15:22:23.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cc4800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cbc071350 0x7f7cbc0aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:23.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cc4800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cbc15b1a0 0x7f7cbc0b4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:23.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cc4800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7cbc15d6e0 0x7f7cbc15a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:23.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cc4800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc0b3160 con 0x7f7cbc15b1a0 2024-02-19T15:22:23.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cc4800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc15aac0 con 0x7f7cbc071350 2024-02-19T15:22:23.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cc4800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc0b2830 con 0x7f7cbc15d6e0 2024-02-19T15:22:23.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cc1400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cbc15b1a0 0x7f7cbc0b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:23.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cc1400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cbc15b1a0 0x7f7cbc0b4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58126/0 (socket says 172.21.15.47:58126) 2024-02-19T15:22:23.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cc1400640 1 -- 172.21.15.47:0/957553841 learned_addr learned my addr 172.21.15.47:0/957553841 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:23.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cc1e00640 1 --2- 172.21.15.47:0/957553841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cbc071350 0x7f7cbc0aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:23.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cc2800640 1 --2- 172.21.15.47:0/957553841 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7cbc15d6e0 0x7f7cbc15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:23.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cc1400640 1 -- 172.21.15.47:0/957553841 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7cbc15d6e0 msgr2=0x7f7cbc15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:23.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cc1400640 1 --2- 172.21.15.47:0/957553841 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7cbc15d6e0 0x7f7cbc15a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:23.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cc1400640 1 -- 172.21.15.47:0/957553841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cbc071350 msgr2=0x7f7cbc0aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:23.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cc1400640 1 --2- 172.21.15.47:0/957553841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cbc071350 0x7f7cbc0aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:23.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cc1400640 1 -- 172.21.15.47:0/957553841 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb8009a70 con 0x7f7cbc15b1a0 2024-02-19T15:22:23.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cc1400640 1 --2- 172.21.15.47:0/957553841 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cbc15b1a0 0x7f7cbc0b4020 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7cac00e930 tx=0x7f7cac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:23.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cb6000640 1 -- 172.21.15.47:0/957553841 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac009690 con 0x7f7cbc15b1a0 2024-02-19T15:22:23.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cb6000640 1 -- 172.21.15.47:0/957553841 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cac00ca00 con 0x7f7cbc15b1a0 2024-02-19T15:22:23.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cb6000640 1 -- 172.21.15.47:0/957553841 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac00cd70 con 0x7f7cbc15b1a0 2024-02-19T15:22:23.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cc4800640 1 -- 172.21.15.47:0/957553841 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cbc0b2b10 con 0x7f7cbc15b1a0 2024-02-19T15:22:23.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cc4800640 1 -- 172.21.15.47:0/957553841 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cbc0b46b0 con 0x7f7cbc15b1a0 2024-02-19T15:22:23.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cc4800640 1 -- 172.21.15.47:0/957553841 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cbc075360 con 0x7f7cbc15b1a0 2024-02-19T15:22:23.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.049+0000 7f7cb6000640 1 -- 172.21.15.47:0/957553841 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7cac013470 con 0x7f7cbc15b1a0 2024-02-19T15:22:23.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.053+0000 7f7cb6000640 1 --2- 172.21.15.47:0/957553841 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ca0066e70 0x7f7ca0069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:23.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.053+0000 7f7cb6000640 1 -- 172.21.15.47:0/957553841 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7cac084c30 con 0x7f7cbc15b1a0 2024-02-19T15:22:23.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.053+0000 7f7cc1e00640 1 --2- 172.21.15.47:0/957553841 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ca0066e70 0x7f7ca0069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:23.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.053+0000 7f7cc1e00640 1 --2- 172.21.15.47:0/957553841 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ca0066e70 0x7f7ca0069330 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0007fd0 tx=0x7f7cb0007d60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:23.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.057+0000 7f7cb6000640 1 -- 172.21.15.47:0/957553841 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7cac051af0 con 0x7f7cbc15b1a0 2024-02-19T15:22:23.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.253+0000 7f7cc4800640 1 -- 172.21.15.47:0/957553841 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7cbc164c40 con 0x7f7cbc15b1a0 2024-02-19T15:22:23.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.253+0000 7f7cb6000640 1 -- 172.21.15.47:0/957553841 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7cac017070 con 0x7f7cbc15b1a0 2024-02-19T15:22:23.258 INFO:teuthology.orchestra.run.smithi047.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-02-19T15:22:23.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.257+0000 7f7cc4800640 1 -- 172.21.15.47:0/957553841 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ca0066e70 msgr2=0x7f7ca0069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:23.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.257+0000 7f7cc4800640 1 --2- 172.21.15.47:0/957553841 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ca0066e70 0x7f7ca0069330 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0007fd0 tx=0x7f7cb0007d60 comp rx=0 tx=0).stop 2024-02-19T15:22:23.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.257+0000 7f7cc4800640 1 -- 172.21.15.47:0/957553841 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cbc15b1a0 msgr2=0x7f7cbc0b4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:23.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.257+0000 7f7cc4800640 1 --2- 172.21.15.47:0/957553841 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cbc15b1a0 0x7f7cbc0b4020 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7cac00e930 tx=0x7f7cac00edf0 comp rx=0 tx=0).stop 2024-02-19T15:22:23.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.257+0000 7f7cc4800640 1 -- 172.21.15.47:0/957553841 shutdown_connections 2024-02-19T15:22:23.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.257+0000 7f7cc4800640 1 --2- 172.21.15.47:0/957553841 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ca0066e70 0x7f7ca0069330 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:23.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.257+0000 7f7cc4800640 1 --2- 172.21.15.47:0/957553841 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7cbc15d6e0 0x7f7cbc15a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:23.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.257+0000 7f7cc4800640 1 --2- 172.21.15.47:0/957553841 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cbc15b1a0 0x7f7cbc0b4020 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:23.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.257+0000 7f7cc4800640 1 --2- 172.21.15.47:0/957553841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cbc071350 0x7f7cbc0aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:23.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.257+0000 7f7cc4800640 1 -- 172.21.15.47:0/957553841 >> 172.21.15.47:0/957553841 conn(0x7f7cbc0b16a0 msgr2=0x7f7cbc0cbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:23.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.257+0000 7f7cc4800640 1 -- 172.21.15.47:0/957553841 shutdown_connections 2024-02-19T15:22:23.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:23.257+0000 7f7cc4800640 1 -- 172.21.15.47:0/957553841 wait complete. 2024-02-19T15:22:23.279 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-19T15:22:23.279 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T15:22:23.279 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi047.front.sepia.ceph.com 2024-02-19T15:22:23.279 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-02-19T15:22:23.280 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-02-19T15:22:23.280 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-19T15:22:23.280 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-19T15:22:23.280 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T15:22:23.280 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-02-19T15:22:23.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:23.280 DEBUG:teuthology.orchestra.run.smithi047:> ip addr 2024-02-19T15:22:23.290 INFO:teuthology.orchestra.run.smithi047.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T15:22:23.290 INFO:teuthology.orchestra.run.smithi047.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T15:22:23.290 INFO:teuthology.orchestra.run.smithi047.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T15:22:23.290 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:23.290 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 ::1/128 scope host 2024-02-19T15:22:23.290 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:23.290 INFO:teuthology.orchestra.run.smithi047.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:22:23.290 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:78 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:23.290 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f0 2024-02-19T15:22:23.290 INFO:teuthology.orchestra.run.smithi047.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T15:22:23.290 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8e brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:23.290 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f0 2024-02-19T15:22:23.291 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.21.15.47/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-02-19T15:22:23.291 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft 42472sec preferred_lft 42472sec 2024-02-19T15:22:23.291 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::ec4:7aff:febd:148e/64 scope link 2024-02-19T15:22:23.291 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:23.291 INFO:teuthology.orchestra.run.smithi047.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:22:23.291 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:79 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:23.291 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f1 2024-02-19T15:22:23.291 INFO:teuthology.orchestra.run.smithi047.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:22:23.291 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8f brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:23.291 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f1 2024-02-19T15:22:23.291 INFO:teuthology.orchestra.run.smithi047.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-02-19T15:22:23.291 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 02:42:47:cf:b1:62 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:23.291 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-02-19T15:22:23.291 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:23.291 INFO:teuthology.orchestra.run.smithi047.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-02-19T15:22:23.292 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:23.292 INFO:teuthology.orchestra.run.smithi047.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-02-19T15:22:23.292 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:23.292 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link tentative 2024-02-19T15:22:23.292 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:23.292 INFO:teuthology.orchestra.run.smithi047.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:22:23.292 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-19T15:22:23.292 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link tentative 2024-02-19T15:22:23.292 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:23.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:23.292 DEBUG:teuthology.orchestra.run.smithi047:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T15:22:23.356 INFO:teuthology.orchestra.run.smithi047.stdout:1 2024-02-19T15:22:23.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:23.358 DEBUG:teuthology.orchestra.run.smithi047:> route 2024-02-19T15:22:23.415 INFO:teuthology.orchestra.run.smithi047.stdout:Kernel IP routing table 2024-02-19T15:22:23.415 INFO:teuthology.orchestra.run.smithi047.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T15:22:23.415 INFO:teuthology.orchestra.run.smithi047.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-02-19T15:22:23.415 INFO:teuthology.orchestra.run.smithi047.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-02-19T15:22:23.415 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-02-19T15:22:23.415 INFO:teuthology.orchestra.run.smithi047.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T15:22:23.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:23.416 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2024-02-19T15:22:23.416 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2024-02-19T15:22:23.416 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-02-19T15:22:23.416 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-02-19T15:22:23.416 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-02-19T15:22:23.416 DEBUG:teuthology.orchestra.run.smithi047:> ') 2024-02-19T15:22:23.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:23.507 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list 2024-02-19T15:22:23.516 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-02-19T15:22:23.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:23.517 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list-id 2024-02-19T15:22:23.566 INFO:teuthology.orchestra.run.smithi047.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-02-19T15:22:23.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:23.567 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2024-02-19T15:22:23.567 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2024-02-19T15:22:23.567 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-02-19T15:22:23.567 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-02-19T15:22:23.567 DEBUG:teuthology.orchestra.run.smithi047:> ') 2024-02-19T15:22:23.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:23.647 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-02-19T15:22:23.714 INFO:teuthology.orchestra.run.smithi047.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T15:22:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T15:22:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T15:22:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 ::1/128 scope host 2024-02-19T15:22:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:23.714 INFO:teuthology.orchestra.run.smithi047.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-02-19T15:22:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 66:f6:4a:bc:8d:fc brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-02-19T15:22:23.715 INFO:teuthology.orchestra.run.smithi047.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-02-19T15:22:23.715 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:23.715 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::64f6:4aff:febc:8dfc/64 scope link 2024-02-19T15:22:23.715 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:23.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:23.716 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-02-19T15:22:23.778 INFO:teuthology.orchestra.run.smithi047.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T15:22:23.778 INFO:teuthology.orchestra.run.smithi047.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T15:22:23.778 INFO:teuthology.orchestra.run.smithi047.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T15:22:23.778 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:23.778 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 ::1/128 scope host 2024-02-19T15:22:23.778 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:23.778 INFO:teuthology.orchestra.run.smithi047.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-02-19T15:22:23.778 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 66:f6:4a:bc:8d:fc brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-02-19T15:22:23.778 INFO:teuthology.orchestra.run.smithi047.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-02-19T15:22:23.778 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:23.778 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::64f6:4aff:febc:8dfc/64 scope link 2024-02-19T15:22:23.778 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:23.780 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-02-19T15:22:23.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:23.780 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2024-02-19T15:22:23.780 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2024-02-19T15:22:23.780 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-02-19T15:22:23.780 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-02-19T15:22:23.781 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-02-19T15:22:23.781 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-02-19T15:22:23.781 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-02-19T15:22:23.781 DEBUG:teuthology.orchestra.run.smithi047:> ') 2024-02-19T15:22:23.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:23.922 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2024-02-19T15:22:23.922 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2024-02-19T15:22:23.922 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set brx.1 up 2024-02-19T15:22:23.922 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set dev brx.1 master ceph-brx 2024-02-19T15:22:23.922 DEBUG:teuthology.orchestra.run.smithi047:> ') 2024-02-19T15:22:23.960 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:22:23.960 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-02-19T15:22:24.014 INFO:teuthology.orchestra.run.smithi047.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-02-19T15:22:24.014 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:22:24.014 DEBUG:teuthology.orchestra.run.smithi047:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-02-19T15:22:24.065 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-19T15:22:24.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:24.066 DEBUG:teuthology.orchestra.run.smithi047:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-02-19T15:22:24.154 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-02-19T15:22:24.154 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-02-19T15:22:24.155 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:22:24.170 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-02-19T15:22:24.170 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-02-19T15:22:24.170 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:22:24.170 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:22:24.170 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:22:24.178 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-02-19T15:22:24.178 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-02-19T15:22:24.178 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:22:24.178 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:22:24.178 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:22:24.178 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790" 2024-02-19T15:22:24.178 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790:/ 2024-02-19T15:22:24.178 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e" will pass to kernel 2024-02-19T15:22:24.182 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T15:22:24.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:24.182 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-02-19T15:22:24.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:24.202 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:22:24.257 INFO:teuthology.orchestra.run.smithi047.stdout:client4523 2024-02-19T15:22:24.257 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:22:24.257 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:22:24.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:24.306 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:22:24.357 INFO:teuthology.orchestra.run.smithi047.stdout:client4523 2024-02-19T15:22:24.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:24.357 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:22:24.409 INFO:teuthology.orchestra.run.smithi047.stdout:client4523 2024-02-19T15:22:24.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:24.409 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e.client4523/status') 2024-02-19T15:22:24.469 INFO:teuthology.orchestra.run.smithi047.stdout:instance: client.4523 (3)172.21.15.47:0/4136357741 2024-02-19T15:22:24.470 INFO:teuthology.orchestra.run.smithi047.stdout:blocklisted: false 2024-02-19T15:22:24.470 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2024-02-19T15:22:24.470 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2024-02-19T15:22:24.470 INFO:teuthology.orchestra.run.smithi047.stderr:70 bytes copied, 5.6569e-05 s, 1.2 MB/s 2024-02-19T15:22:24.472 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-02-19T15:22:24.472 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-19T15:22:24.472 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-19T15:22:24.472 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-02-19T15:22:24.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.653+0000 7f4fa2000640 1 Processor -- start 2024-02-19T15:22:24.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.653+0000 7f4fa2000640 1 -- start start 2024-02-19T15:22:24.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.653+0000 7f4fa2000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f9c15f510 0x7f4f9c070700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:24.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.653+0000 7f4fa2000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4f9c15fa90 0x7f4f9c15ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:24.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.653+0000 7f4fa2000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4f9c160450 0x7f4f9c06ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:24.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.653+0000 7f4fa2000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f9c0cffc0 con 0x7f4f9c160450 2024-02-19T15:22:24.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.653+0000 7f4fa2000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f9c0ae970 con 0x7f4f9c15fa90 2024-02-19T15:22:24.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.653+0000 7f4fa2000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c1608e0 con 0x7f4f9c15f510 2024-02-19T15:22:24.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.657+0000 7f4f9be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4f9c160450 0x7f4f9c06ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:24.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.657+0000 7f4f93e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4f9c15fa90 0x7f4f9c15ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:24.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.657+0000 7f4f9b400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f9c15f510 0x7f4f9c070700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:24.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.657+0000 7f4f9be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4f9c160450 0x7f4f9c06ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60546/0 (socket says 172.21.15.47:60546) 2024-02-19T15:22:24.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.657+0000 7f4f9be00640 1 -- 172.21.15.47:0/2923583483 learned_addr learned my addr 172.21.15.47:0/2923583483 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:24.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.657+0000 7f4f9be00640 1 -- 172.21.15.47:0/2923583483 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f9c15f510 msgr2=0x7f4f9c070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:24.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.657+0000 7f4f9be00640 1 --2- 172.21.15.47:0/2923583483 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f9c15f510 0x7f4f9c070700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:24.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.657+0000 7f4f9be00640 1 -- 172.21.15.47:0/2923583483 >> v2:172.21.15.116:3300/0 conn(0x7f4f9c15fa90 msgr2=0x7f4f9c15ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:24.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.657+0000 7f4f9be00640 1 --2- 172.21.15.47:0/2923583483 >> v2:172.21.15.116:3300/0 conn(0x7f4f9c15fa90 0x7f4f9c15ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:24.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.657+0000 7f4f9be00640 1 -- 172.21.15.47:0/2923583483 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f9c070c40 con 0x7f4f9c160450 2024-02-19T15:22:24.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.657+0000 7f4f93e00640 1 --2- 172.21.15.47:0/2923583483 >> v2:172.21.15.116:3300/0 conn(0x7f4f9c15fa90 0x7f4f9c15ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:24.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.657+0000 7f4f9be00640 1 --2- 172.21.15.47:0/2923583483 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f9c160450 0x7f4f9c06ebc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c06be90 tx=0x7f4f9401c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa4ddb7011ce7a41 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:24.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.657+0000 7f4f9aa00640 1 -- 172.21.15.47:0/2923583483 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f940a6420 con 0x7f4f9c160450 2024-02-19T15:22:24.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.657+0000 7f4f9aa00640 1 -- 172.21.15.47:0/2923583483 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f940a6580 con 0x7f4f9c160450 2024-02-19T15:22:24.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.657+0000 7f4f9aa00640 1 -- 172.21.15.47:0/2923583483 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f940a6910 con 0x7f4f9c160450 2024-02-19T15:22:24.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4fa2000640 1 -- 172.21.15.47:0/2923583483 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f9c160450 msgr2=0x7f4f9c06ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:24.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4fa2000640 1 --2- 172.21.15.47:0/2923583483 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f9c160450 0x7f4f9c06ebc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c06be90 tx=0x7f4f9401c6a0 comp rx=0 tx=0).stop 2024-02-19T15:22:24.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4fa2000640 1 -- 172.21.15.47:0/2923583483 shutdown_connections 2024-02-19T15:22:24.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4fa2000640 1 --2- 172.21.15.47:0/2923583483 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f9c160450 0x7f4f9c06ebc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:24.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4fa2000640 1 --2- 172.21.15.47:0/2923583483 >> v2:172.21.15.116:3300/0 conn(0x7f4f9c15fa90 0x7f4f9c15ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:24.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4fa2000640 1 --2- 172.21.15.47:0/2923583483 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f9c15f510 0x7f4f9c070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:24.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4fa2000640 1 -- 172.21.15.47:0/2923583483 >> 172.21.15.47:0/2923583483 conn(0x7f4f9c0b3690 msgr2=0x7f4f9c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:24.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4fa2000640 1 -- 172.21.15.47:0/2923583483 shutdown_connections 2024-02-19T15:22:24.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4fa2000640 1 -- 172.21.15.47:0/2923583483 wait complete. 2024-02-19T15:22:24.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4fa2000640 1 Processor -- start 2024-02-19T15:22:24.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4fa2000640 1 -- start start 2024-02-19T15:22:24.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4fa2000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f9c15f510 0x7f4f9c0ce010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:24.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4fa2000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f9c15fa90 0x7f4f9c153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:24.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4fa2000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f9c160450 0x7f4f9c15d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:24.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4fa2000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c0cf470 con 0x7f4f9c15fa90 2024-02-19T15:22:24.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4fa2000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c0ce860 con 0x7f4f9c15f510 2024-02-19T15:22:24.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4fa2000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c0ceb40 con 0x7f4f9c160450 2024-02-19T15:22:24.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4f93e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f9c15fa90 0x7f4f9c153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:24.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4f93e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f9c15fa90 0x7f4f9c153fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60556/0 (socket says 172.21.15.47:60556) 2024-02-19T15:22:24.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4f93e00640 1 -- 172.21.15.47:0/3977970654 learned_addr learned my addr 172.21.15.47:0/3977970654 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:24.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4f9b400640 1 --2- 172.21.15.47:0/3977970654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f9c15f510 0x7f4f9c0ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:24.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4f9be00640 1 --2- 172.21.15.47:0/3977970654 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f9c160450 0x7f4f9c15d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:24.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4f93e00640 1 -- 172.21.15.47:0/3977970654 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f9c160450 msgr2=0x7f4f9c15d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:24.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4f93e00640 1 --2- 172.21.15.47:0/3977970654 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f9c160450 0x7f4f9c15d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:24.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4f93e00640 1 -- 172.21.15.47:0/3977970654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f9c15f510 msgr2=0x7f4f9c0ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:24.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4f93e00640 1 --2- 172.21.15.47:0/3977970654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f9c15f510 0x7f4f9c0ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:24.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4f93e00640 1 -- 172.21.15.47:0/3977970654 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f84009420 con 0x7f4f9c15fa90 2024-02-19T15:22:24.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4f9b400640 1 --2- 172.21.15.47:0/3977970654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f9c15f510 0x7f4f9c0ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:24.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.661+0000 7f4f93e00640 1 --2- 172.21.15.47:0/3977970654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f9c15fa90 0x7f4f9c153fa0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4f8400ebe0 tx=0x7f4f8400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:24.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.665+0000 7f4f93400640 1 -- 172.21.15.47:0/3977970654 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8400eee0 con 0x7f4f9c15fa90 2024-02-19T15:22:24.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.665+0000 7f4f93400640 1 -- 172.21.15.47:0/3977970654 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8400cc30 con 0x7f4f9c15fa90 2024-02-19T15:22:24.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.665+0000 7f4fa2000640 1 -- 172.21.15.47:0/3977970654 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f94009b30 con 0x7f4f9c15fa90 2024-02-19T15:22:24.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.665+0000 7f4f93400640 1 -- 172.21.15.47:0/3977970654 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f84013470 con 0x7f4f9c15fa90 2024-02-19T15:22:24.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.665+0000 7f4fa2000640 1 -- 172.21.15.47:0/3977970654 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f9c0cf100 con 0x7f4f9c15fa90 2024-02-19T15:22:24.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.665+0000 7f4fa2000640 1 -- 172.21.15.47:0/3977970654 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f9c0cf5e0 con 0x7f4f9c15fa90 2024-02-19T15:22:24.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.665+0000 7f4f93400640 1 -- 172.21.15.47:0/3977970654 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4f840135d0 con 0x7f4f9c15fa90 2024-02-19T15:22:24.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.665+0000 7f4f93400640 1 --2- 172.21.15.47:0/3977970654 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4f78066e40 0x7f4f78069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:24.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.665+0000 7f4f93400640 1 -- 172.21.15.47:0/3977970654 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f84084bf0 con 0x7f4f9c15fa90 2024-02-19T15:22:24.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.665+0000 7f4f9b400640 1 --2- 172.21.15.47:0/3977970654 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4f78066e40 0x7f4f78069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:24.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.665+0000 7f4f9b400640 1 --2- 172.21.15.47:0/3977970654 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4f78066e40 0x7f4f78069300 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c009f50 tx=0x7f4f8c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:24.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.669+0000 7f4f93400640 1 -- 172.21.15.47:0/3977970654 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f4f84051b20 con 0x7f4f9c15fa90 2024-02-19T15:22:24.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.869+0000 7f4fa2000640 1 -- 172.21.15.47:0/3977970654 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4f9c075360 con 0x7f4f9c15fa90 2024-02-19T15:22:24.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.869+0000 7f4f93400640 1 -- 172.21.15.47:0/3977970654 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f4f84017060 con 0x7f4f9c15fa90 2024-02-19T15:22:24.873 INFO:teuthology.orchestra.run.smithi047.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-02-19T15:22:24.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.873+0000 7f4fa2000640 1 -- 172.21.15.47:0/3977970654 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4f78066e40 msgr2=0x7f4f78069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:24.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.873+0000 7f4fa2000640 1 --2- 172.21.15.47:0/3977970654 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4f78066e40 0x7f4f78069300 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c009f50 tx=0x7f4f8c009190 comp rx=0 tx=0).stop 2024-02-19T15:22:24.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.873+0000 7f4fa2000640 1 -- 172.21.15.47:0/3977970654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f9c15fa90 msgr2=0x7f4f9c153fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:24.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.873+0000 7f4fa2000640 1 --2- 172.21.15.47:0/3977970654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f9c15fa90 0x7f4f9c153fa0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4f8400ebe0 tx=0x7f4f8400c520 comp rx=0 tx=0).stop 2024-02-19T15:22:24.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.873+0000 7f4fa2000640 1 -- 172.21.15.47:0/3977970654 shutdown_connections 2024-02-19T15:22:24.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.873+0000 7f4fa2000640 1 --2- 172.21.15.47:0/3977970654 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4f78066e40 0x7f4f78069300 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:24.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.873+0000 7f4fa2000640 1 --2- 172.21.15.47:0/3977970654 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f9c160450 0x7f4f9c15d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:24.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.873+0000 7f4fa2000640 1 --2- 172.21.15.47:0/3977970654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f9c15fa90 0x7f4f9c153fa0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:24.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.873+0000 7f4fa2000640 1 --2- 172.21.15.47:0/3977970654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f9c15f510 0x7f4f9c0ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:24.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.873+0000 7f4fa2000640 1 -- 172.21.15.47:0/3977970654 >> 172.21.15.47:0/3977970654 conn(0x7f4f9c0b3690 msgr2=0x7f4f9c0c5690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:24.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.873+0000 7f4fa2000640 1 -- 172.21.15.47:0/3977970654 shutdown_connections 2024-02-19T15:22:24.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:24.873+0000 7f4fa2000640 1 -- 172.21.15.47:0/3977970654 wait complete. 2024-02-19T15:22:24.895 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-19T15:22:24.895 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T15:22:24.895 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2024-02-19T15:22:24.895 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-02-19T15:22:24.895 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-19T15:22:24.895 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-19T15:22:24.895 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-19T15:22:24.895 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T15:22:24.895 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-02-19T15:22:24.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:24.895 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2024-02-19T15:22:24.904 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T15:22:24.904 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T15:22:24.904 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T15:22:24.904 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:24.905 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2024-02-19T15:22:24.905 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:24.905 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:22:24.905 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:24.905 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2024-02-19T15:22:24.905 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:22:24.905 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:24.905 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2024-02-19T15:22:24.905 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:22:24.905 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:24.905 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T15:22:24.905 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:24.905 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-02-19T15:22:24.905 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 42492sec preferred_lft 42492sec 2024-02-19T15:22:24.905 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2024-02-19T15:22:24.905 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:24.906 INFO:teuthology.orchestra.run.smithi116.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-02-19T15:22:24.906 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 02:42:ca:fc:5f:53 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:24.906 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-02-19T15:22:24.906 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:24.906 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-19T15:22:24.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:24.906 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-02-19T15:22:24.906 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-02-19T15:22:24.906 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add name ceph-brx type bridge 2024-02-19T15:22:24.906 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr flush dev ceph-brx 2024-02-19T15:22:24.906 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set ceph-brx up 2024-02-19T15:22:24.906 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-19T15:22:24.906 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-02-19T15:22:25.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:25.012 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T15:22:25.030 INFO:teuthology.orchestra.run.smithi116.stdout:1 2024-02-19T15:22:25.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:25.032 DEBUG:teuthology.orchestra.run.smithi116:> route 2024-02-19T15:22:25.102 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2024-02-19T15:22:25.102 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T15:22:25.103 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-02-19T15:22:25.103 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-19T15:22:25.103 INFO:teuthology.orchestra.run.smithi116.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-02-19T15:22:25.103 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-19T15:22:25.103 INFO:teuthology.orchestra.run.smithi116.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-02-19T15:22:25.103 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-02-19T15:22:25.103 INFO:teuthology.orchestra.run.smithi116.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-02-19T15:22:25.103 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T15:22:25.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:25.103 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-02-19T15:22:25.103 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-02-19T15:22:25.104 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-19T15:22:25.104 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-19T15:22:25.104 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-19T15:22:25.104 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-02-19T15:22:25.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:25.194 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-02-19T15:22:25.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:25.204 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list-id 2024-02-19T15:22:25.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:25.254 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-02-19T15:22:25.254 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-02-19T15:22:25.255 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-19T15:22:25.255 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-02-19T15:22:25.255 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-02-19T15:22:25.336 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-02-19T15:22:25.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:25.336 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-02-19T15:22:25.336 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-02-19T15:22:25.336 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-02-19T15:22:25.336 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-19T15:22:25.336 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-02-19T15:22:25.336 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-02-19T15:22:25.336 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-02-19T15:22:25.336 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-02-19T15:22:25.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:25.498 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-02-19T15:22:25.498 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-02-19T15:22:25.498 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.0 up 2024-02-19T15:22:25.498 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set dev brx.0 master ceph-brx 2024-02-19T15:22:25.498 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-02-19T15:22:25.538 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:22:25.539 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-02-19T15:22:25.590 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-02-19T15:22:25.590 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:22:25.590 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-02-19T15:22:25.641 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-19T15:22:25.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:25.642 DEBUG:teuthology.orchestra.run.smithi116:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-19T15:22:25.742 INFO:teuthology.orchestra.run.smithi116.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-19T15:22:25.742 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-19T15:22:25.742 INFO:teuthology.orchestra.run.smithi116.stdout:invalid new device string format 2024-02-19T15:22:26.296 INFO:teuthology.orchestra.run.smithi116.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-19T15:22:26.297 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-19T15:22:26.297 INFO:teuthology.orchestra.run.smithi116.stdout:invalid new device string format 2024-02-19T15:22:26.297 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:22:26.297 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:22:26.308 INFO:teuthology.orchestra.run.smithi116.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-19T15:22:26.308 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-19T15:22:26.308 INFO:teuthology.orchestra.run.smithi116.stdout:invalid new device string format 2024-02-19T15:22:26.308 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:22:26.308 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:22:26.308 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: resolved to: "172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790" 2024-02-19T15:22:26.309 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790:/ 2024-02-19T15:22:26.309 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e" will pass to kernel 2024-02-19T15:22:26.312 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T15:22:26.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:26.312 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-02-19T15:22:26.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:26.332 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:22:26.399 INFO:teuthology.orchestra.run.smithi116.stdout:client4668 2024-02-19T15:22:26.399 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:22:26.399 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:22:26.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:26.449 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:22:26.499 INFO:teuthology.orchestra.run.smithi116.stdout:client4668 2024-02-19T15:22:26.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:26.499 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:22:26.552 INFO:teuthology.orchestra.run.smithi116.stdout:client4668 2024-02-19T15:22:26.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:26.552 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e.client4668/status') 2024-02-19T15:22:26.612 INFO:teuthology.orchestra.run.smithi116.stdout:instance: client.4668 (3)172.21.15.116:0/600256071 2024-02-19T15:22:26.612 INFO:teuthology.orchestra.run.smithi116.stdout:blocklisted: false 2024-02-19T15:22:26.612 INFO:teuthology.orchestra.run.smithi116.stderr:0+1 records in 2024-02-19T15:22:26.613 INFO:teuthology.orchestra.run.smithi116.stderr:0+1 records out 2024-02-19T15:22:26.613 INFO:teuthology.orchestra.run.smithi116.stderr:70 bytes copied, 6.8573e-05 s, 1.0 MB/s 2024-02-19T15:22:26.614 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2024-02-19T15:22:26.614 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-19T15:22:26.614 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-02-19T15:22:26.615 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-02-19T15:22:26.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.765+0000 7ff197800640 1 Processor -- start 2024-02-19T15:22:26.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.765+0000 7ff197800640 1 -- start start 2024-02-19T15:22:26.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.765+0000 7ff197800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff1900b4240 0x7ff1900d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:26.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.765+0000 7ff197800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1900ce400 0x7ff1900ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:26.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.765+0000 7ff197800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1900d1070 con 0x7ff19015dfa0 2024-02-19T15:22:26.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.765+0000 7ff197800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1900cedd0 con 0x7ff1900b4240 2024-02-19T15:22:26.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.765+0000 7ff197800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff19006ec60 con 0x7ff1900ce400 2024-02-19T15:22:26.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.765+0000 7ff194e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff19015dfa0 0x7ff19006fa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34098/0 (socket says 172.21.15.47:34098) 2024-02-19T15:22:26.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.765+0000 7ff194e00640 1 -- 172.21.15.47:0/152165434 learned_addr learned my addr 172.21.15.47:0/152165434 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:26.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.765+0000 7ff187e00640 1 --2- 172.21.15.47:0/152165434 >> v2:172.21.15.116:3300/0 conn(0x7ff1900b4240 0x7ff1900d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:26.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.765+0000 7ff195800640 1 --2- 172.21.15.47:0/152165434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1900ce400 0x7ff1900ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:26.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.765+0000 7ff187e00640 1 -- 172.21.15.47:0/152165434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1900ce400 msgr2=0x7ff1900ce890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:26.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.765+0000 7ff187e00640 1 --2- 172.21.15.47:0/152165434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1900ce400 0x7ff1900ce890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:26.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.765+0000 7ff18fe00640 1 -- 172.21.15.47:0/152165434 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178129704 0 0) 0x7ff1900d1070 con 0x7ff19015dfa0 2024-02-19T15:22:26.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.765+0000 7ff187e00640 1 -- 172.21.15.47:0/152165434 >> v1:172.21.15.47:6789/0 conn(0x7ff19015dfa0 legacy=0x7ff19006fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:26.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.765+0000 7ff187e00640 1 -- 172.21.15.47:0/152165434 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19006ef40 con 0x7ff1900b4240 2024-02-19T15:22:26.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.769+0000 7ff187e00640 1 --2- 172.21.15.47:0/152165434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1900b4240 0x7ff1900d0820 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff1780097c0 tx=0x7ff17801c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7e690c764bf5e1e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:26.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.769+0000 7ff18fe00640 1 -- 172.21.15.47:0/152165434 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1780ab670 con 0x7ff1900b4240 2024-02-19T15:22:26.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.769+0000 7ff18fe00640 1 -- 172.21.15.47:0/152165434 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1780ab7d0 con 0x7ff1900b4240 2024-02-19T15:22:26.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.769+0000 7ff18fe00640 1 -- 172.21.15.47:0/152165434 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1780abb60 con 0x7ff1900b4240 2024-02-19T15:22:26.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.769+0000 7ff197800640 1 -- 172.21.15.47:0/152165434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1900b4240 msgr2=0x7ff1900d0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:26.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.769+0000 7ff197800640 1 --2- 172.21.15.47:0/152165434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1900b4240 0x7ff1900d0820 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff1780097c0 tx=0x7ff17801c470 comp rx=0 tx=0).stop 2024-02-19T15:22:26.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.769+0000 7ff197800640 1 -- 172.21.15.47:0/152165434 shutdown_connections 2024-02-19T15:22:26.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.769+0000 7ff197800640 1 --2- 172.21.15.47:0/152165434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1900ce400 0x7ff1900ce890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:26.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.769+0000 7ff197800640 1 --2- 172.21.15.47:0/152165434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1900b4240 0x7ff1900d0820 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:26.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.769+0000 7ff197800640 1 -- 172.21.15.47:0/152165434 >> 172.21.15.47:0/152165434 conn(0x7ff1900b3690 msgr2=0x7ff1900b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:26.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.769+0000 7ff197800640 1 -- 172.21.15.47:0/152165434 shutdown_connections 2024-02-19T15:22:26.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.769+0000 7ff197800640 1 -- 172.21.15.47:0/152165434 wait complete. 2024-02-19T15:22:26.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff197800640 1 Processor -- start 2024-02-19T15:22:26.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff197800640 1 -- start start 2024-02-19T15:22:26.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff197800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1900b4240 0x7ff1901543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:26.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff197800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1900ce400 0x7ff190154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:26.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff197800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff19015dfa0 0x7ff1900cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:26.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff197800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff190155ee0 con 0x7ff19015dfa0 2024-02-19T15:22:26.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff197800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff190155640 con 0x7ff1900ce400 2024-02-19T15:22:26.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff197800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff190155920 con 0x7ff1900b4240 2024-02-19T15:22:26.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff195800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff19015dfa0 0x7ff1900cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:26.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff187e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1900ce400 0x7ff190154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:26.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff187e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1900ce400 0x7ff190154bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:33266/0 (socket says 172.21.15.47:33266) 2024-02-19T15:22:26.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff194e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1900b4240 0x7ff1901543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:26.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff187e00640 1 -- 172.21.15.47:0/3098953012 learned_addr learned my addr 172.21.15.47:0/3098953012 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:26.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff187e00640 1 -- 172.21.15.47:0/3098953012 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1900b4240 msgr2=0x7ff1901543e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:26.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff187e00640 1 --2- 172.21.15.47:0/3098953012 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1900b4240 0x7ff1901543e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:26.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff187e00640 1 -- 172.21.15.47:0/3098953012 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff19015dfa0 msgr2=0x7ff1900cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:26.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff187e00640 1 --2- 172.21.15.47:0/3098953012 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff19015dfa0 0x7ff1900cadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:26.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff187e00640 1 -- 172.21.15.47:0/3098953012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff178009420 con 0x7ff1900ce400 2024-02-19T15:22:26.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff187e00640 1 --2- 172.21.15.47:0/3098953012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1900ce400 0x7ff190154bf0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff17801ab20 tx=0x7ff17801acb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:26.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff18d600640 1 -- 172.21.15.47:0/3098953012 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff178004220 con 0x7ff1900ce400 2024-02-19T15:22:26.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff18d600640 1 -- 172.21.15.47:0/3098953012 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1780b9070 con 0x7ff1900ce400 2024-02-19T15:22:26.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff18d600640 1 -- 172.21.15.47:0/3098953012 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff178003620 con 0x7ff1900ce400 2024-02-19T15:22:26.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff197800640 1 -- 172.21.15.47:0/3098953012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff190155c00 con 0x7ff1900ce400 2024-02-19T15:22:26.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.773+0000 7ff197800640 1 -- 172.21.15.47:0/3098953012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff190154920 con 0x7ff1900ce400 2024-02-19T15:22:26.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.777+0000 7ff197800640 1 -- 172.21.15.47:0/3098953012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff190075360 con 0x7ff1900ce400 2024-02-19T15:22:26.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.777+0000 7ff18d600640 1 -- 172.21.15.47:0/3098953012 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff178003780 con 0x7ff1900ce400 2024-02-19T15:22:26.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.777+0000 7ff18d600640 1 --2- 172.21.15.47:0/3098953012 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff16c066e40 0x7ff16c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:26.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.777+0000 7ff18d600640 1 -- 172.21.15.47:0/3098953012 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1780bb070 con 0x7ff1900ce400 2024-02-19T15:22:26.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.777+0000 7ff194e00640 1 --2- 172.21.15.47:0/3098953012 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff16c066e40 0x7ff16c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:26.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.777+0000 7ff194e00640 1 --2- 172.21.15.47:0/3098953012 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff16c066e40 0x7ff16c069300 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff180000f70 tx=0x7ff180007df0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:26.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.789+0000 7ff18d600640 1 -- 172.21.15.47:0/3098953012 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff1780ec2b0 con 0x7ff1900ce400 2024-02-19T15:22:26.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.985+0000 7ff197800640 1 -- 172.21.15.47:0/3098953012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff190155a60 con 0x7ff1900ce400 2024-02-19T15:22:26.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.985+0000 7ff18d600640 1 -- 172.21.15.47:0/3098953012 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff1780c4060 con 0x7ff1900ce400 2024-02-19T15:22:26.988 INFO:teuthology.orchestra.run.smithi047.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-02-19T15:22:26.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.985+0000 7ff197800640 1 -- 172.21.15.47:0/3098953012 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff16c066e40 msgr2=0x7ff16c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:26.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.985+0000 7ff197800640 1 --2- 172.21.15.47:0/3098953012 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff16c066e40 0x7ff16c069300 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff180000f70 tx=0x7ff180007df0 comp rx=0 tx=0).stop 2024-02-19T15:22:26.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.985+0000 7ff197800640 1 -- 172.21.15.47:0/3098953012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1900ce400 msgr2=0x7ff190154bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:26.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.985+0000 7ff197800640 1 --2- 172.21.15.47:0/3098953012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1900ce400 0x7ff190154bf0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff17801ab20 tx=0x7ff17801acb0 comp rx=0 tx=0).stop 2024-02-19T15:22:26.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.985+0000 7ff197800640 1 -- 172.21.15.47:0/3098953012 shutdown_connections 2024-02-19T15:22:26.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.985+0000 7ff197800640 1 --2- 172.21.15.47:0/3098953012 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff16c066e40 0x7ff16c069300 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:26.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.985+0000 7ff197800640 1 --2- 172.21.15.47:0/3098953012 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff19015dfa0 0x7ff1900cadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:26.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.985+0000 7ff197800640 1 --2- 172.21.15.47:0/3098953012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1900ce400 0x7ff190154bf0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:26.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.985+0000 7ff197800640 1 --2- 172.21.15.47:0/3098953012 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1900b4240 0x7ff1901543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:26.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.985+0000 7ff197800640 1 -- 172.21.15.47:0/3098953012 >> 172.21.15.47:0/3098953012 conn(0x7ff1900b3690 msgr2=0x7ff1900cef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:26.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.985+0000 7ff197800640 1 -- 172.21.15.47:0/3098953012 shutdown_connections 2024-02-19T15:22:26.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:26.989+0000 7ff197800640 1 -- 172.21.15.47:0/3098953012 wait complete. 2024-02-19T15:22:27.010 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-19T15:22:27.010 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T15:22:27.010 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2024-02-19T15:22:27.010 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-02-19T15:22:27.010 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-02-19T15:22:27.010 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-19T15:22:27.010 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-19T15:22:27.010 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T15:22:27.010 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-02-19T15:22:27.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:27.010 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2024-02-19T15:22:27.019 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T15:22:27.019 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T15:22:27.019 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T15:22:27.019 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:27.019 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2024-02-19T15:22:27.020 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:27.020 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:22:27.020 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:27.020 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2024-02-19T15:22:27.020 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:22:27.020 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:27.020 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2024-02-19T15:22:27.020 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:22:27.020 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:27.020 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T15:22:27.020 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:27.020 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-02-19T15:22:27.020 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 42490sec preferred_lft 42490sec 2024-02-19T15:22:27.021 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2024-02-19T15:22:27.021 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:27.021 INFO:teuthology.orchestra.run.smithi116.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-02-19T15:22:27.021 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 02:42:ca:fc:5f:53 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:27.021 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-02-19T15:22:27.021 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:27.021 INFO:teuthology.orchestra.run.smithi116.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-02-19T15:22:27.021 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-02-19T15:22:27.021 INFO:teuthology.orchestra.run.smithi116.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-02-19T15:22:27.021 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:27.021 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-02-19T15:22:27.021 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:27.021 INFO:teuthology.orchestra.run.smithi116.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:22:27.021 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-19T15:22:27.021 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link tentative 2024-02-19T15:22:27.022 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:27.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:27.022 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T15:22:27.083 INFO:teuthology.orchestra.run.smithi116.stdout:1 2024-02-19T15:22:27.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:27.085 DEBUG:teuthology.orchestra.run.smithi116:> route 2024-02-19T15:22:27.136 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2024-02-19T15:22:27.136 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T15:22:27.136 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-02-19T15:22:27.136 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-19T15:22:27.136 INFO:teuthology.orchestra.run.smithi116.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-02-19T15:22:27.136 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-19T15:22:27.136 INFO:teuthology.orchestra.run.smithi116.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-02-19T15:22:27.136 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-02-19T15:22:27.136 INFO:teuthology.orchestra.run.smithi116.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-02-19T15:22:27.136 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T15:22:27.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:27.137 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-02-19T15:22:27.137 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-02-19T15:22:27.137 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-19T15:22:27.137 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-19T15:22:27.137 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-19T15:22:27.137 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-02-19T15:22:27.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:27.227 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-02-19T15:22:27.236 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-02-19T15:22:27.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:27.237 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list-id 2024-02-19T15:22:27.285 INFO:teuthology.orchestra.run.smithi116.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-02-19T15:22:27.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:27.286 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-02-19T15:22:27.286 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-02-19T15:22:27.286 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-02-19T15:22:27.286 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-02-19T15:22:27.286 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-02-19T15:22:27.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:27.367 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-02-19T15:22:27.432 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T15:22:27.432 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T15:22:27.432 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T15:22:27.433 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:27.433 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2024-02-19T15:22:27.433 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:27.433 INFO:teuthology.orchestra.run.smithi116.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-02-19T15:22:27.433 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 36:99:55:cc:06:e1 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-02-19T15:22:27.433 INFO:teuthology.orchestra.run.smithi116.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-02-19T15:22:27.433 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:27.433 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::3499:55ff:fecc:6e1/64 scope link 2024-02-19T15:22:27.433 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:27.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:27.434 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-02-19T15:22:27.500 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T15:22:27.500 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T15:22:27.501 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T15:22:27.501 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:27.501 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2024-02-19T15:22:27.501 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:27.501 INFO:teuthology.orchestra.run.smithi116.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-02-19T15:22:27.501 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 36:99:55:cc:06:e1 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-02-19T15:22:27.501 INFO:teuthology.orchestra.run.smithi116.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-02-19T15:22:27.501 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:27.501 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::3499:55ff:fecc:6e1/64 scope link 2024-02-19T15:22:27.501 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:22:27.502 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-02-19T15:22:27.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:27.502 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-02-19T15:22:27.503 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-02-19T15:22:27.503 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-02-19T15:22:27.503 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-02-19T15:22:27.503 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-02-19T15:22:27.503 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-02-19T15:22:27.503 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-02-19T15:22:27.503 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-02-19T15:22:27.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:27.640 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-02-19T15:22:27.640 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-02-19T15:22:27.640 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.1 up 2024-02-19T15:22:27.640 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set dev brx.1 master ceph-brx 2024-02-19T15:22:27.640 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-02-19T15:22:27.679 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:22:27.679 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-02-19T15:22:27.733 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-02-19T15:22:27.733 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:22:27.733 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-02-19T15:22:27.784 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-19T15:22:27.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:27.784 DEBUG:teuthology.orchestra.run.smithi116:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-02-19T15:22:27.877 INFO:teuthology.orchestra.run.smithi116.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-02-19T15:22:27.877 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-02-19T15:22:27.877 INFO:teuthology.orchestra.run.smithi116.stdout:invalid new device string format 2024-02-19T15:22:27.892 INFO:teuthology.orchestra.run.smithi116.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-02-19T15:22:27.892 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-02-19T15:22:27.892 INFO:teuthology.orchestra.run.smithi116.stdout:invalid new device string format 2024-02-19T15:22:27.892 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:22:27.892 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:22:27.900 INFO:teuthology.orchestra.run.smithi116.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-02-19T15:22:27.901 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-02-19T15:22:27.901 INFO:teuthology.orchestra.run.smithi116.stdout:invalid new device string format 2024-02-19T15:22:27.901 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:22:27.901 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:22:27.901 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: resolved to: "172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790" 2024-02-19T15:22:27.901 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790:/ 2024-02-19T15:22:27.901 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e" will pass to kernel 2024-02-19T15:22:27.905 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T15:22:27.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:27.905 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-02-19T15:22:27.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:27.964 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:22:28.015 INFO:teuthology.orchestra.run.smithi116.stdout:client4538 2024-02-19T15:22:28.016 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:22:28.016 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:22:28.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:28.069 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:22:28.119 INFO:teuthology.orchestra.run.smithi116.stdout:client4538 2024-02-19T15:22:28.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:28.120 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:22:28.171 INFO:teuthology.orchestra.run.smithi116.stdout:client4538 2024-02-19T15:22:28.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:28.172 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e.client4538/status') 2024-02-19T15:22:28.232 INFO:teuthology.orchestra.run.smithi116.stdout:instance: client.4538 (3)192.168.0.2:0/575810023 2024-02-19T15:22:28.232 INFO:teuthology.orchestra.run.smithi116.stdout:blocklisted: false 2024-02-19T15:22:28.232 INFO:teuthology.orchestra.run.smithi116.stderr:0+1 records in 2024-02-19T15:22:28.232 INFO:teuthology.orchestra.run.smithi116.stderr:0+1 records out 2024-02-19T15:22:28.232 INFO:teuthology.orchestra.run.smithi116.stderr:68 bytes copied, 6.5201e-05 s, 1.0 MB/s 2024-02-19T15:22:28.234 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-02-19T15:22:28.249 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:28.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.397+0000 7f76d5800640 1 Processor -- start 2024-02-19T15:22:28.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.397+0000 7f76d5800640 1 -- start start 2024-02-19T15:22:28.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.401+0000 7f76d5800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76d015d6e0 0x7f76d0162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:28.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.401+0000 7f76d5800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f76d0071350 0x7f76d0164c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:28.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.401+0000 7f76d5800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d00718c0 con 0x7f76d015b1a0 2024-02-19T15:22:28.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.401+0000 7f76d5800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d0071ba0 con 0x7f76d0071350 2024-02-19T15:22:28.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.401+0000 7f76d5800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d0071e80 con 0x7f76d015d6e0 2024-02-19T15:22:28.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.401+0000 7f76cea00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f76d015b1a0 0x7f76d015b580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:40342/0 (socket says 172.21.15.47:40342) 2024-02-19T15:22:28.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.401+0000 7f76cea00640 1 -- 172.21.15.47:0/655862343 learned_addr learned my addr 172.21.15.47:0/655862343 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:28.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.401+0000 7f76cf400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f76d0071350 0x7f76d0164c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:28.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.401+0000 7f76c6000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76d015d6e0 0x7f76d0162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:28.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.401+0000 7f76cf400640 1 -- 172.21.15.47:0/655862343 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76d015d6e0 msgr2=0x7f76d0162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:28.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.401+0000 7f76cf400640 1 --2- 172.21.15.47:0/655862343 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76d015d6e0 0x7f76d0162740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.401+0000 7f76cf400640 1 -- 172.21.15.47:0/655862343 >> v1:172.21.15.116:6789/0 conn(0x7f76d015b1a0 legacy=0x7f76d015b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:28.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.401+0000 7f76cf400640 1 -- 172.21.15.47:0/655862343 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d0072160 con 0x7f76d0071350 2024-02-19T15:22:28.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.401+0000 7f76ce000640 1 -- 172.21.15.47:0/655862343 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723232172 0 0) 0x7f76d00718c0 con 0x7f76d015b1a0 2024-02-19T15:22:28.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76cf400640 1 --2- 172.21.15.47:0/655862343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d0071350 0x7f76d0164c90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009d90 tx=0x7f76c801c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95ffb748b6b7bb82 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:28.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76ce000640 1 -- 172.21.15.47:0/655862343 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c80a5a50 con 0x7f76d0071350 2024-02-19T15:22:28.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76ce000640 1 -- 172.21.15.47:0/655862343 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c80a5bb0 con 0x7f76d0071350 2024-02-19T15:22:28.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76ce000640 1 -- 172.21.15.47:0/655862343 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c80ad440 con 0x7f76d0071350 2024-02-19T15:22:28.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76d5800640 1 -- 172.21.15.47:0/655862343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d0071350 msgr2=0x7f76d0164c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:28.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76d5800640 1 --2- 172.21.15.47:0/655862343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d0071350 0x7f76d0164c90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009d90 tx=0x7f76c801c970 comp rx=0 tx=0).stop 2024-02-19T15:22:28.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76d5800640 1 -- 172.21.15.47:0/655862343 shutdown_connections 2024-02-19T15:22:28.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76d5800640 1 --2- 172.21.15.47:0/655862343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d0071350 0x7f76d0164c90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76d5800640 1 --2- 172.21.15.47:0/655862343 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76d015d6e0 0x7f76d0162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76d5800640 1 -- 172.21.15.47:0/655862343 >> 172.21.15.47:0/655862343 conn(0x7f76d00b16a0 msgr2=0x7f76d00b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:28.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76d5800640 1 -- 172.21.15.47:0/655862343 shutdown_connections 2024-02-19T15:22:28.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76d5800640 1 -- 172.21.15.47:0/655862343 wait complete. 2024-02-19T15:22:28.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76d5800640 1 Processor -- start 2024-02-19T15:22:28.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76d5800640 1 -- start start 2024-02-19T15:22:28.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76d5800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d0071350 0x7f76d01554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:28.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76d5800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76d015b1a0 0x7f76d0151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:28.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76d5800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76d015d6e0 0x7f76d0148e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:28.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76d5800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d01493a0 con 0x7f76d015d6e0 2024-02-19T15:22:28.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76d5800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d0149de0 con 0x7f76d0071350 2024-02-19T15:22:28.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76d5800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d0155b80 con 0x7f76d015b1a0 2024-02-19T15:22:28.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76cea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d0071350 0x7f76d01554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:28.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.405+0000 7f76cf400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76d015d6e0 0x7f76d0148e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:28.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.409+0000 7f76cea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d0071350 0x7f76d01554f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:33284/0 (socket says 172.21.15.47:33284) 2024-02-19T15:22:28.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.409+0000 7f76c6000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76d015b1a0 0x7f76d0151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:28.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.409+0000 7f76cea00640 1 -- 172.21.15.47:0/901148329 learned_addr learned my addr 172.21.15.47:0/901148329 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:28.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.409+0000 7f76cf400640 1 -- 172.21.15.47:0/901148329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76d015b1a0 msgr2=0x7f76d0151230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:28.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.409+0000 7f76cf400640 1 --2- 172.21.15.47:0/901148329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76d015b1a0 0x7f76d0151230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.409+0000 7f76cf400640 1 -- 172.21.15.47:0/901148329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d0071350 msgr2=0x7f76d01554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:28.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.409+0000 7f76cf400640 1 --2- 172.21.15.47:0/901148329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d0071350 0x7f76d01554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.409+0000 7f76cf400640 1 -- 172.21.15.47:0/901148329 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c8009a70 con 0x7f76d015d6e0 2024-02-19T15:22:28.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.409+0000 7f76cea00640 1 --2- 172.21.15.47:0/901148329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d0071350 0x7f76d01554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:28.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.409+0000 7f76cf400640 1 --2- 172.21.15.47:0/901148329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76d015d6e0 0x7f76d0148e60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f76c80ac000 tx=0x7f76c8019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:28.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.409+0000 7f76c7400640 1 -- 172.21.15.47:0/901148329 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c80ade30 con 0x7f76d015d6e0 2024-02-19T15:22:28.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.409+0000 7f76c7400640 1 -- 172.21.15.47:0/901148329 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c8009450 con 0x7f76d015d6e0 2024-02-19T15:22:28.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.409+0000 7f76d5800640 1 -- 172.21.15.47:0/901148329 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d0155e60 con 0x7f76d015d6e0 2024-02-19T15:22:28.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.409+0000 7f76c7400640 1 -- 172.21.15.47:0/901148329 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c8003ba0 con 0x7f76d015d6e0 2024-02-19T15:22:28.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.409+0000 7f76d5800640 1 -- 172.21.15.47:0/901148329 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d014ef60 con 0x7f76d015d6e0 2024-02-19T15:22:28.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.409+0000 7f76d5800640 1 -- 172.21.15.47:0/901148329 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d0156310 con 0x7f76d015d6e0 2024-02-19T15:22:28.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.409+0000 7f76c7400640 1 -- 172.21.15.47:0/901148329 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f76c80b9450 con 0x7f76d015d6e0 2024-02-19T15:22:28.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.409+0000 7f76c7400640 1 --2- 172.21.15.47:0/901148329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f76ac066e40 0x7f76ac069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:28.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.409+0000 7f76c7400640 1 -- 172.21.15.47:0/901148329 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f76c811f550 con 0x7f76d015d6e0 2024-02-19T15:22:28.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.413+0000 7f76cea00640 1 --2- 172.21.15.47:0/901148329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f76ac066e40 0x7f76ac069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:28.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.413+0000 7f76cea00640 1 --2- 172.21.15.47:0/901148329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f76ac066e40 0x7f76ac069300 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f76c0000f70 tx=0x7f76c0007e20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:28.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.417+0000 7f76c7400640 1 -- 172.21.15.47:0/901148329 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f76c80ec2d0 con 0x7f76d015d6e0 2024-02-19T15:22:28.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.581+0000 7f76d5800640 1 -- 172.21.15.47:0/901148329 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76d014f150 con 0x7f76d015d6e0 2024-02-19T15:22:28.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.581+0000 7f76c7400640 1 -- 172.21.15.47:0/901148329 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f76c80ef990 con 0x7f76d015d6e0 2024-02-19T15:22:28.585 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:28.585 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":21,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:06.843155+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:03.091360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:04.222965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:28.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.585+0000 7f76d5800640 1 -- 172.21.15.47:0/901148329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f76ac066e40 msgr2=0x7f76ac069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:28.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.585+0000 7f76d5800640 1 --2- 172.21.15.47:0/901148329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f76ac066e40 0x7f76ac069300 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f76c0000f70 tx=0x7f76c0007e20 comp rx=0 tx=0).stop 2024-02-19T15:22:28.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.585+0000 7f76d5800640 1 -- 172.21.15.47:0/901148329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76d015d6e0 msgr2=0x7f76d0148e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:28.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.585+0000 7f76d5800640 1 --2- 172.21.15.47:0/901148329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76d015d6e0 0x7f76d0148e60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f76c80ac000 tx=0x7f76c8019710 comp rx=0 tx=0).stop 2024-02-19T15:22:28.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.585+0000 7f76d5800640 1 -- 172.21.15.47:0/901148329 shutdown_connections 2024-02-19T15:22:28.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.585+0000 7f76d5800640 1 --2- 172.21.15.47:0/901148329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f76ac066e40 0x7f76ac069300 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.585+0000 7f76d5800640 1 --2- 172.21.15.47:0/901148329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76d015d6e0 0x7f76d0148e60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.585+0000 7f76d5800640 1 --2- 172.21.15.47:0/901148329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76d015b1a0 0x7f76d0151230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.585+0000 7f76d5800640 1 --2- 172.21.15.47:0/901148329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d0071350 0x7f76d01554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.585+0000 7f76d5800640 1 -- 172.21.15.47:0/901148329 >> 172.21.15.47:0/901148329 conn(0x7f76d00b16a0 msgr2=0x7f76d015a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:28.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.585+0000 7f76d5800640 1 -- 172.21.15.47:0/901148329 shutdown_connections 2024-02-19T15:22:28.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.585+0000 7f76d5800640 1 -- 172.21.15.47:0/901148329 wait complete. 2024-02-19T15:22:28.606 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:22:03.091360+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:22:04.222965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:22:28.606 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:22:28.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.745+0000 7f0a74600640 1 Processor -- start 2024-02-19T15:22:28.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.745+0000 7f0a74600640 1 -- start start 2024-02-19T15:22:28.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.745+0000 7f0a74600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0a6c15bd90 0x7f0a6c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:28.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.745+0000 7f0a74600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a6c0cb490 0x7f0a6c072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:28.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.745+0000 7f0a74600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0a6c072680 0x7f0a6c0ca270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:28.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.745+0000 7f0a74600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a6c15fc90 con 0x7f0a6c15bd90 2024-02-19T15:22:28.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.745+0000 7f0a74600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a6c15ff70 con 0x7f0a6c072680 2024-02-19T15:22:28.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.745+0000 7f0a74600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c160250 con 0x7f0a6c0cb490 2024-02-19T15:22:28.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.745+0000 7f0a71c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0a6c15bd90 0x7f0a6c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:28.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.745+0000 7f0a71c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0a6c15bd90 0x7f0a6c070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60572/0 (socket says 172.21.15.47:60572) 2024-02-19T15:22:28.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.745+0000 7f0a71c00640 1 -- 172.21.15.47:0/2842147074 learned_addr learned my addr 172.21.15.47:0/2842147074 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:28.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.745+0000 7f0a71200640 1 --2- 172.21.15.47:0/2842147074 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a6c0cb490 0x7f0a6c072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:28.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.745+0000 7f0a72600640 1 --2- 172.21.15.47:0/2842147074 >> v2:172.21.15.116:3300/0 conn(0x7f0a6c072680 0x7f0a6c0ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:28.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.749+0000 7f0a71c00640 1 -- 172.21.15.47:0/2842147074 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a6c0cb490 msgr2=0x7f0a6c072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:28.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.749+0000 7f0a71c00640 1 --2- 172.21.15.47:0/2842147074 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a6c0cb490 0x7f0a6c072140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.749+0000 7f0a71c00640 1 -- 172.21.15.47:0/2842147074 >> v2:172.21.15.116:3300/0 conn(0x7f0a6c072680 msgr2=0x7f0a6c0ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:28.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.749+0000 7f0a71c00640 1 --2- 172.21.15.47:0/2842147074 >> v2:172.21.15.116:3300/0 conn(0x7f0a6c072680 0x7f0a6c0ca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.749+0000 7f0a71c00640 1 -- 172.21.15.47:0/2842147074 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c160530 con 0x7f0a6c15bd90 2024-02-19T15:22:28.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.749+0000 7f0a71c00640 1 --2- 172.21.15.47:0/2842147074 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a6c15bd90 0x7f0a6c070960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0b5620 tx=0x7f0a6001c100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4501bfafd7d43e2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:28.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.749+0000 7f0a6be00640 1 -- 172.21.15.47:0/2842147074 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a600ab600 con 0x7f0a6c15bd90 2024-02-19T15:22:28.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.749+0000 7f0a6be00640 1 -- 172.21.15.47:0/2842147074 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a600a7050 con 0x7f0a6c15bd90 2024-02-19T15:22:28.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.749+0000 7f0a6be00640 1 -- 172.21.15.47:0/2842147074 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a600ab980 con 0x7f0a6c15bd90 2024-02-19T15:22:28.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a74600640 1 -- 172.21.15.47:0/2842147074 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a6c15bd90 msgr2=0x7f0a6c070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:28.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a74600640 1 --2- 172.21.15.47:0/2842147074 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a6c15bd90 0x7f0a6c070960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0b5620 tx=0x7f0a6001c100 comp rx=0 tx=0).stop 2024-02-19T15:22:28.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a74600640 1 -- 172.21.15.47:0/2842147074 shutdown_connections 2024-02-19T15:22:28.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a74600640 1 --2- 172.21.15.47:0/2842147074 >> v2:172.21.15.116:3300/0 conn(0x7f0a6c072680 0x7f0a6c0ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a74600640 1 --2- 172.21.15.47:0/2842147074 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a6c0cb490 0x7f0a6c072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a74600640 1 --2- 172.21.15.47:0/2842147074 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a6c15bd90 0x7f0a6c070960 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a74600640 1 -- 172.21.15.47:0/2842147074 >> 172.21.15.47:0/2842147074 conn(0x7f0a6c0cfc20 msgr2=0x7f0a6c0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:28.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a74600640 1 -- 172.21.15.47:0/2842147074 shutdown_connections 2024-02-19T15:22:28.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a74600640 1 -- 172.21.15.47:0/2842147074 wait complete. 2024-02-19T15:22:28.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a74600640 1 Processor -- start 2024-02-19T15:22:28.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a74600640 1 -- start start 2024-02-19T15:22:28.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a74600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a6c072680 0x7f0a6c0aa260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:28.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a74600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a6c0cb490 0x7f0a6c15b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:28.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a74600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a6c15bd90 0x7f0a6c0ab040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:28.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a74600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0b1480 con 0x7f0a6c072680 2024-02-19T15:22:28.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a74600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0a94e0 con 0x7f0a6c15bd90 2024-02-19T15:22:28.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a74600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0a9780 con 0x7f0a6c0cb490 2024-02-19T15:22:28.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a71c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a6c072680 0x7f0a6c0aa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:28.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a71200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a6c0cb490 0x7f0a6c15b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:28.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a71c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a6c072680 0x7f0a6c0aa260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60578/0 (socket says 172.21.15.47:60578) 2024-02-19T15:22:28.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a71c00640 1 -- 172.21.15.47:0/41634822 learned_addr learned my addr 172.21.15.47:0/41634822 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:28.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a72600640 1 --2- 172.21.15.47:0/41634822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a6c15bd90 0x7f0a6c0ab040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:28.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a71c00640 1 -- 172.21.15.47:0/41634822 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a6c0cb490 msgr2=0x7f0a6c15b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:28.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a71c00640 1 --2- 172.21.15.47:0/41634822 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a6c0cb490 0x7f0a6c15b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a71c00640 1 -- 172.21.15.47:0/41634822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a6c15bd90 msgr2=0x7f0a6c0ab040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:28.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a71c00640 1 --2- 172.21.15.47:0/41634822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a6c15bd90 0x7f0a6c0ab040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.753+0000 7f0a71c00640 1 -- 172.21.15.47:0/41634822 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a600094d0 con 0x7f0a6c072680 2024-02-19T15:22:28.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.757+0000 7f0a71c00640 1 --2- 172.21.15.47:0/41634822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a6c072680 0x7f0a6c0aa260 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0a6001a7b0 tx=0x7f0a60018ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:28.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.757+0000 7f0a69600640 1 -- 172.21.15.47:0/41634822 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a600b4420 con 0x7f0a6c072680 2024-02-19T15:22:28.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.757+0000 7f0a69600640 1 -- 172.21.15.47:0/41634822 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a600b4580 con 0x7f0a6c072680 2024-02-19T15:22:28.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.757+0000 7f0a69600640 1 -- 172.21.15.47:0/41634822 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a600b4930 con 0x7f0a6c072680 2024-02-19T15:22:28.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.757+0000 7f0a74600640 1 -- 172.21.15.47:0/41634822 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6c0a9a60 con 0x7f0a6c072680 2024-02-19T15:22:28.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.757+0000 7f0a74600640 1 -- 172.21.15.47:0/41634822 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6c0aa7a0 con 0x7f0a6c072680 2024-02-19T15:22:28.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.757+0000 7f0a74600640 1 -- 172.21.15.47:0/41634822 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a6c075360 con 0x7f0a6c072680 2024-02-19T15:22:28.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.757+0000 7f0a69600640 1 -- 172.21.15.47:0/41634822 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0a600b2070 con 0x7f0a6c072680 2024-02-19T15:22:28.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.757+0000 7f0a69600640 1 --2- 172.21.15.47:0/41634822 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a4c066e40 0x7f0a4c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:28.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.757+0000 7f0a69600640 1 -- 172.21.15.47:0/41634822 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a6011e410 con 0x7f0a6c072680 2024-02-19T15:22:28.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.757+0000 7f0a71200640 1 --2- 172.21.15.47:0/41634822 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a4c066e40 0x7f0a4c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:28.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.761+0000 7f0a71200640 1 --2- 172.21.15.47:0/41634822 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a4c066e40 0x7f0a4c069300 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0b0880 tx=0x7f0a58009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:28.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.761+0000 7f0a69600640 1 -- 172.21.15.47:0/41634822 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0a600eb2a0 con 0x7f0a6c072680 2024-02-19T15:22:28.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.929+0000 7f0a74600640 1 -- 172.21.15.47:0/41634822 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0a6c06e800 con 0x7f0a6c072680 2024-02-19T15:22:28.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.929+0000 7f0a69600640 1 -- 172.21.15.47:0/41634822 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0a600ee960 con 0x7f0a6c072680 2024-02-19T15:22:28.931 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:22:28.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.929+0000 7f0a74600640 1 -- 172.21.15.47:0/41634822 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a4c066e40 msgr2=0x7f0a4c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:28.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.929+0000 7f0a74600640 1 --2- 172.21.15.47:0/41634822 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a4c066e40 0x7f0a4c069300 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0b0880 tx=0x7f0a58009190 comp rx=0 tx=0).stop 2024-02-19T15:22:28.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.929+0000 7f0a74600640 1 -- 172.21.15.47:0/41634822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a6c072680 msgr2=0x7f0a6c0aa260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:28.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.929+0000 7f0a74600640 1 --2- 172.21.15.47:0/41634822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a6c072680 0x7f0a6c0aa260 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0a6001a7b0 tx=0x7f0a60018ea0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.929+0000 7f0a74600640 1 -- 172.21.15.47:0/41634822 shutdown_connections 2024-02-19T15:22:28.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.929+0000 7f0a74600640 1 --2- 172.21.15.47:0/41634822 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a4c066e40 0x7f0a4c069300 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.929+0000 7f0a74600640 1 --2- 172.21.15.47:0/41634822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a6c15bd90 0x7f0a6c0ab040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.929+0000 7f0a74600640 1 --2- 172.21.15.47:0/41634822 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a6c0cb490 0x7f0a6c15b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.929+0000 7f0a74600640 1 --2- 172.21.15.47:0/41634822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a6c072680 0x7f0a6c0aa260 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:28.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.929+0000 7f0a74600640 1 -- 172.21.15.47:0/41634822 >> 172.21.15.47:0/41634822 conn(0x7f0a6c0cfc20 msgr2=0x7f0a6c0c7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:28.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.929+0000 7f0a74600640 1 -- 172.21.15.47:0/41634822 shutdown_connections 2024-02-19T15:22:28.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:28.929+0000 7f0a74600640 1 -- 172.21.15.47:0/41634822 wait complete. 2024-02-19T15:22:28.952 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:22:29.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.093+0000 7fec19e00640 1 Processor -- start 2024-02-19T15:22:29.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.093+0000 7fec19e00640 1 -- start start 2024-02-19T15:22:29.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.093+0000 7fec19e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fec1406e360 0x7fec140cb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:29.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.093+0000 7fec19e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fec140cd880 0x7fec140cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:29.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.093+0000 7fec19e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec140cb540 con 0x7fec1415af10 2024-02-19T15:22:29.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.093+0000 7fec19e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec14071c70 con 0x7fec1406e360 2024-02-19T15:22:29.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.093+0000 7fec19e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec1406cef0 con 0x7fec140cd880 2024-02-19T15:22:29.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.093+0000 7fec0a000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fec1406e360 0x7fec140cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:29.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.093+0000 7fec0a000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fec1406e360 0x7fec140cb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60594/0 (socket says 172.21.15.47:60594) 2024-02-19T15:22:29.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.093+0000 7fec0a000640 1 -- 172.21.15.47:0/1762383241 learned_addr learned my addr 172.21.15.47:0/1762383241 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:29.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.093+0000 7fec13400640 1 --2- 172.21.15.47:0/1762383241 >> v2:172.21.15.116:3300/0 conn(0x7fec140cd880 0x7fec140cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:29.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.093+0000 7fec0a000640 1 -- 172.21.15.47:0/1762383241 >> v2:172.21.15.116:3300/0 conn(0x7fec140cd880 msgr2=0x7fec140cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:29.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.093+0000 7fec0a000640 1 --2- 172.21.15.47:0/1762383241 >> v2:172.21.15.116:3300/0 conn(0x7fec140cd880 0x7fec140cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.093+0000 7fec0a000640 1 -- 172.21.15.47:0/1762383241 >> v1:172.21.15.47:6789/0 conn(0x7fec1415af10 legacy=0x7fec140ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:29.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.093+0000 7fec0a000640 1 -- 172.21.15.47:0/1762383241 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec140ca610 con 0x7fec1406e360 2024-02-19T15:22:29.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.093+0000 7fec12000640 1 -- 172.21.15.47:0/1762383241 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821519953 0 0) 0x7fec140cb540 con 0x7fec1415af10 2024-02-19T15:22:29.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.097+0000 7fec0a000640 1 --2- 172.21.15.47:0/1762383241 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1406e360 0x7fec140cb000 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fec14067c60 tx=0x7febfc01eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8e197c6c2177da server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:29.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.097+0000 7fec12000640 1 -- 172.21.15.47:0/1762383241 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc0a7c00 con 0x7fec1406e360 2024-02-19T15:22:29.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.097+0000 7fec12000640 1 -- 172.21.15.47:0/1762383241 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febfc0a7d60 con 0x7fec1406e360 2024-02-19T15:22:29.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.097+0000 7fec12000640 1 -- 172.21.15.47:0/1762383241 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc0af660 con 0x7fec1406e360 2024-02-19T15:22:29.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.097+0000 7fec19e00640 1 -- 172.21.15.47:0/1762383241 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1406e360 msgr2=0x7fec140cb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:29.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.097+0000 7fec19e00640 1 --2- 172.21.15.47:0/1762383241 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1406e360 0x7fec140cb000 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fec14067c60 tx=0x7febfc01eb30 comp rx=0 tx=0).stop 2024-02-19T15:22:29.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.097+0000 7fec19e00640 1 -- 172.21.15.47:0/1762383241 shutdown_connections 2024-02-19T15:22:29.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.097+0000 7fec19e00640 1 --2- 172.21.15.47:0/1762383241 >> v2:172.21.15.116:3300/0 conn(0x7fec140cd880 0x7fec140cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.097+0000 7fec19e00640 1 --2- 172.21.15.47:0/1762383241 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1406e360 0x7fec140cb000 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.097+0000 7fec19e00640 1 -- 172.21.15.47:0/1762383241 >> 172.21.15.47:0/1762383241 conn(0x7fec140cfc20 msgr2=0x7fec140d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:29.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.097+0000 7fec19e00640 1 -- 172.21.15.47:0/1762383241 shutdown_connections 2024-02-19T15:22:29.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.097+0000 7fec19e00640 1 -- 172.21.15.47:0/1762383241 wait complete. 2024-02-19T15:22:29.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec19e00640 1 Processor -- start 2024-02-19T15:22:29.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec19e00640 1 -- start start 2024-02-19T15:22:29.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec19e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec1406e360 0x7fec14156230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:29.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec19e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec140cd880 0x7fec140ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:29.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec19e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1415af10 0x7fec140b38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:29.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec19e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1415d3d0 con 0x7fec1415af10 2024-02-19T15:22:29.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec19e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec141f7a80 con 0x7fec140cd880 2024-02-19T15:22:29.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec19e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec141f77a0 con 0x7fec1406e360 2024-02-19T15:22:29.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec13400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1415af10 0x7fec140b38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:29.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec13400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1415af10 0x7fec140b38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60598/0 (socket says 172.21.15.47:60598) 2024-02-19T15:22:29.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec13400640 1 -- 172.21.15.47:0/671844690 learned_addr learned my addr 172.21.15.47:0/671844690 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:29.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec12a00640 1 --2- 172.21.15.47:0/671844690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec1406e360 0x7fec14156230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:29.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec0a000640 1 --2- 172.21.15.47:0/671844690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec140cd880 0x7fec140ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:29.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec13400640 1 -- 172.21.15.47:0/671844690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec1406e360 msgr2=0x7fec14156230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:29.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec13400640 1 --2- 172.21.15.47:0/671844690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec1406e360 0x7fec14156230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec13400640 1 -- 172.21.15.47:0/671844690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec140cd880 msgr2=0x7fec140ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:29.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec13400640 1 --2- 172.21.15.47:0/671844690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec140cd880 0x7fec140ad880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec13400640 1 -- 172.21.15.47:0/671844690 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febfc00b770 con 0x7fec1415af10 2024-02-19T15:22:29.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec13400640 1 --2- 172.21.15.47:0/671844690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1415af10 0x7fec140b38c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fec0c009dd0 tx=0x7fec0c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:29.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec0be00640 1 -- 172.21.15.47:0/671844690 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c014480 con 0x7fec1415af10 2024-02-19T15:22:29.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec0be00640 1 -- 172.21.15.47:0/671844690 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0c00f040 con 0x7fec1415af10 2024-02-19T15:22:29.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec0be00640 1 -- 172.21.15.47:0/671844690 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c014840 con 0x7fec1415af10 2024-02-19T15:22:29.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec19e00640 1 -- 172.21.15.47:0/671844690 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec1415cd80 con 0x7fec1415af10 2024-02-19T15:22:29.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.101+0000 7fec19e00640 1 -- 172.21.15.47:0/671844690 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec14156770 con 0x7fec1415af10 2024-02-19T15:22:29.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.105+0000 7fec19e00640 1 -- 172.21.15.47:0/671844690 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec1415d580 con 0x7fec1415af10 2024-02-19T15:22:29.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.105+0000 7fec0be00640 1 -- 172.21.15.47:0/671844690 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fec0c0149a0 con 0x7fec1415af10 2024-02-19T15:22:29.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.105+0000 7fec0be00640 1 --2- 172.21.15.47:0/671844690 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7febf0066e40 0x7febf0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:29.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.105+0000 7fec0be00640 1 -- 172.21.15.47:0/671844690 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec0c085500 con 0x7fec1415af10 2024-02-19T15:22:29.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.105+0000 7fec12a00640 1 --2- 172.21.15.47:0/671844690 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7febf0066e40 0x7febf0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:29.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.105+0000 7fec12a00640 1 --2- 172.21.15.47:0/671844690 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7febf0066e40 0x7febf0069300 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fec04007fd0 tx=0x7fec04009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:29.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.109+0000 7fec0be00640 1 -- 172.21.15.47:0/671844690 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fec1415d580 con 0x7fec1415af10 2024-02-19T15:22:29.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.273+0000 7fec19e00640 1 -- 172.21.15.47:0/671844690 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fec1415d580 con 0x7fec1415af10 2024-02-19T15:22:29.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.273+0000 7fec0be00640 1 -- 172.21.15.47:0/671844690 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fec1415d580 con 0x7fec1415af10 2024-02-19T15:22:29.275 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:22:29.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.273+0000 7fec19e00640 1 -- 172.21.15.47:0/671844690 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7febf0066e40 msgr2=0x7febf0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:29.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.273+0000 7fec19e00640 1 --2- 172.21.15.47:0/671844690 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7febf0066e40 0x7febf0069300 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fec04007fd0 tx=0x7fec04009040 comp rx=0 tx=0).stop 2024-02-19T15:22:29.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.273+0000 7fec19e00640 1 -- 172.21.15.47:0/671844690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1415af10 msgr2=0x7fec140b38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:29.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.273+0000 7fec19e00640 1 --2- 172.21.15.47:0/671844690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1415af10 0x7fec140b38c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fec0c009dd0 tx=0x7fec0c00c490 comp rx=0 tx=0).stop 2024-02-19T15:22:29.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.273+0000 7fec19e00640 1 -- 172.21.15.47:0/671844690 shutdown_connections 2024-02-19T15:22:29.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.273+0000 7fec19e00640 1 --2- 172.21.15.47:0/671844690 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7febf0066e40 0x7febf0069300 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.273+0000 7fec19e00640 1 --2- 172.21.15.47:0/671844690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1415af10 0x7fec140b38c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.273+0000 7fec19e00640 1 --2- 172.21.15.47:0/671844690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec140cd880 0x7fec140ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.273+0000 7fec19e00640 1 --2- 172.21.15.47:0/671844690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec1406e360 0x7fec14156230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.273+0000 7fec19e00640 1 -- 172.21.15.47:0/671844690 >> 172.21.15.47:0/671844690 conn(0x7fec140cfc20 msgr2=0x7fec140d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:29.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.273+0000 7fec19e00640 1 -- 172.21.15.47:0/671844690 shutdown_connections 2024-02-19T15:22:29.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.273+0000 7fec19e00640 1 -- 172.21.15.47:0/671844690 wait complete. 2024-02-19T15:22:29.305 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:22:29.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.445+0000 7f8ee8800640 1 Processor -- start 2024-02-19T15:22:29.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.445+0000 7f8ee8800640 1 -- start start 2024-02-19T15:22:29.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.445+0000 7f8ee8800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8ee00cb490 0x7f8ee0070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:29.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.445+0000 7f8ee8800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee015fc90 con 0x7f8ee015bd90 2024-02-19T15:22:29.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.445+0000 7f8ee8800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee015ff70 con 0x7f8ee0072850 2024-02-19T15:22:29.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.445+0000 7f8ee8800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ee0160250 con 0x7f8ee00cb490 2024-02-19T15:22:29.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.445+0000 7f8ee5400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8ee00cb490 0x7f8ee0070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:29.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.445+0000 7f8ee5e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8ee015bd90 0x7f8ee0072140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34126/0 (socket says 172.21.15.47:34126) 2024-02-19T15:22:29.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.449+0000 7f8ee5e00640 1 -- 172.21.15.47:0/2991533741 learned_addr learned my addr 172.21.15.47:0/2991533741 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:29.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.449+0000 7f8ee5400640 1 -- 172.21.15.47:0/2991533741 >> v1:172.21.15.116:6789/0 conn(0x7f8ee0072850 legacy=0x7f8ee006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:29.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.449+0000 7f8ee5400640 1 -- 172.21.15.47:0/2991533741 >> v1:172.21.15.47:6789/0 conn(0x7f8ee015bd90 legacy=0x7f8ee0072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:29.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.449+0000 7f8ee4a00640 1 -- 172.21.15.47:0/2991533741 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255499424 0 0) 0x7f8ee015ff70 con 0x7f8ee0072850 2024-02-19T15:22:29.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.449+0000 7f8ee5400640 1 -- 172.21.15.47:0/2991533741 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee0160530 con 0x7f8ee00cb490 2024-02-19T15:22:29.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.449+0000 7f8ee5400640 1 --2- 172.21.15.47:0/2991533741 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ee00cb490 0x7f8ee0070960 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00097c0 tx=0x7f8ed001c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e8958141c04e6a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:29.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.449+0000 7f8ee4a00640 1 -- 172.21.15.47:0/2991533741 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed00ab600 con 0x7f8ee00cb490 2024-02-19T15:22:29.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.449+0000 7f8ee4a00640 1 -- 172.21.15.47:0/2991533741 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed00a7050 con 0x7f8ee00cb490 2024-02-19T15:22:29.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.449+0000 7f8ee4a00640 1 -- 172.21.15.47:0/2991533741 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed00ab980 con 0x7f8ee00cb490 2024-02-19T15:22:29.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.449+0000 7f8ee8800640 1 -- 172.21.15.47:0/2991533741 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ee00cb490 msgr2=0x7f8ee0070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:29.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.449+0000 7f8ee8800640 1 --2- 172.21.15.47:0/2991533741 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ee00cb490 0x7f8ee0070960 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00097c0 tx=0x7f8ed001c380 comp rx=0 tx=0).stop 2024-02-19T15:22:29.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee8800640 1 -- 172.21.15.47:0/2991533741 shutdown_connections 2024-02-19T15:22:29.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee8800640 1 --2- 172.21.15.47:0/2991533741 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ee00cb490 0x7f8ee0070960 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee8800640 1 -- 172.21.15.47:0/2991533741 >> 172.21.15.47:0/2991533741 conn(0x7f8ee00cfc20 msgr2=0x7f8ee00d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:29.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee8800640 1 -- 172.21.15.47:0/2991533741 shutdown_connections 2024-02-19T15:22:29.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee8800640 1 -- 172.21.15.47:0/2991533741 wait complete. 2024-02-19T15:22:29.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee8800640 1 Processor -- start 2024-02-19T15:22:29.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee8800640 1 -- start start 2024-02-19T15:22:29.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee8800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ee0072850 0x7f8ee00ce020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:29.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee8800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ee00cb490 0x7f8ee015c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:29.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee8800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ee015bd90 0x7f8ee01525e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:29.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee8800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee015ce00 con 0x7f8ee0072850 2024-02-19T15:22:29.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee8800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee015d0e0 con 0x7f8ee00cb490 2024-02-19T15:22:29.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee8800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee015d3c0 con 0x7f8ee015bd90 2024-02-19T15:22:29.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee5e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ee0072850 0x7f8ee00ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:29.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee5e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ee0072850 0x7f8ee00ce020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60614/0 (socket says 172.21.15.47:60614) 2024-02-19T15:22:29.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee5e00640 1 -- 172.21.15.47:0/291913746 learned_addr learned my addr 172.21.15.47:0/291913746 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:29.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee6800640 1 --2- 172.21.15.47:0/291913746 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ee015bd90 0x7f8ee01525e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:29.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee5400640 1 --2- 172.21.15.47:0/291913746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ee00cb490 0x7f8ee015c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:29.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee5e00640 1 -- 172.21.15.47:0/291913746 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ee015bd90 msgr2=0x7f8ee01525e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:29.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee5e00640 1 --2- 172.21.15.47:0/291913746 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ee015bd90 0x7f8ee01525e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee5e00640 1 -- 172.21.15.47:0/291913746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ee00cb490 msgr2=0x7f8ee015c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:29.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee5e00640 1 --2- 172.21.15.47:0/291913746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ee00cb490 0x7f8ee015c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee5e00640 1 -- 172.21.15.47:0/291913746 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed0009420 con 0x7f8ee0072850 2024-02-19T15:22:29.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee5e00640 1 --2- 172.21.15.47:0/291913746 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ee0072850 0x7f8ee00ce020 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8ed400c400 tx=0x7f8ed400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:29.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8eda000640 1 -- 172.21.15.47:0/291913746 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed4005cf0 con 0x7f8ee0072850 2024-02-19T15:22:29.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8eda000640 1 -- 172.21.15.47:0/291913746 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed4005e50 con 0x7f8ee0072850 2024-02-19T15:22:29.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee8800640 1 -- 172.21.15.47:0/291913746 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ee015d6a0 con 0x7f8ee0072850 2024-02-19T15:22:29.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8eda000640 1 -- 172.21.15.47:0/291913746 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed40072d0 con 0x7f8ee0072850 2024-02-19T15:22:29.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.453+0000 7f8ee8800640 1 -- 172.21.15.47:0/291913746 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ee0153400 con 0x7f8ee0072850 2024-02-19T15:22:29.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.457+0000 7f8ee8800640 1 -- 172.21.15.47:0/291913746 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ee0075360 con 0x7f8ee0072850 2024-02-19T15:22:29.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.457+0000 7f8eda000640 1 -- 172.21.15.47:0/291913746 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8ed4007430 con 0x7f8ee0072850 2024-02-19T15:22:29.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.457+0000 7f8eda000640 1 --2- 172.21.15.47:0/291913746 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ec4066e40 0x7f8ec4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:29.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.457+0000 7f8eda000640 1 -- 172.21.15.47:0/291913746 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ed4083ed0 con 0x7f8ee0072850 2024-02-19T15:22:29.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.457+0000 7f8ee5400640 1 --2- 172.21.15.47:0/291913746 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ec4066e40 0x7f8ec4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:29.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.457+0000 7f8ee5400640 1 --2- 172.21.15.47:0/291913746 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ec4066e40 0x7f8ec4069300 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00ad000 tx=0x7f8ed0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:29.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.461+0000 7f8eda000640 1 -- 172.21.15.47:0/291913746 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8ed4050d70 con 0x7f8ee0072850 2024-02-19T15:22:29.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.625+0000 7f8ee8800640 1 -- 172.21.15.47:0/291913746 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8ee015db50 con 0x7f8ee0072850 2024-02-19T15:22:29.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.625+0000 7f8eda000640 1 -- 172.21.15.47:0/291913746 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8ed40170a0 con 0x7f8ee0072850 2024-02-19T15:22:29.629 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:22:29.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.629+0000 7f8ee8800640 1 -- 172.21.15.47:0/291913746 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ec4066e40 msgr2=0x7f8ec4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:29.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.629+0000 7f8ee8800640 1 --2- 172.21.15.47:0/291913746 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ec4066e40 0x7f8ec4069300 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00ad000 tx=0x7f8ed0008040 comp rx=0 tx=0).stop 2024-02-19T15:22:29.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.629+0000 7f8ee8800640 1 -- 172.21.15.47:0/291913746 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ee0072850 msgr2=0x7f8ee00ce020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:29.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.629+0000 7f8ee8800640 1 --2- 172.21.15.47:0/291913746 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ee0072850 0x7f8ee00ce020 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8ed400c400 tx=0x7f8ed400c8c0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.629+0000 7f8ee8800640 1 -- 172.21.15.47:0/291913746 shutdown_connections 2024-02-19T15:22:29.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.629+0000 7f8ee8800640 1 --2- 172.21.15.47:0/291913746 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ec4066e40 0x7f8ec4069300 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.629+0000 7f8ee8800640 1 --2- 172.21.15.47:0/291913746 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ee015bd90 0x7f8ee01525e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.629+0000 7f8ee8800640 1 --2- 172.21.15.47:0/291913746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ee00cb490 0x7f8ee015c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.629+0000 7f8ee8800640 1 --2- 172.21.15.47:0/291913746 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ee0072850 0x7f8ee00ce020 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.629+0000 7f8ee8800640 1 -- 172.21.15.47:0/291913746 >> 172.21.15.47:0/291913746 conn(0x7f8ee00cfc20 msgr2=0x7f8ee00b4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:29.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.629+0000 7f8ee8800640 1 -- 172.21.15.47:0/291913746 shutdown_connections 2024-02-19T15:22:29.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.629+0000 7f8ee8800640 1 -- 172.21.15.47:0/291913746 wait complete. 2024-02-19T15:22:29.651 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:29.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.789+0000 7f3bc9200640 1 Processor -- start 2024-02-19T15:22:29.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.789+0000 7f3bc9200640 1 -- start start 2024-02-19T15:22:29.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.789+0000 7f3bc9200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3bc40cea80 0x7f3bc40727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:29.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.789+0000 7f3bc9200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bc415c390 0x7f3bc415c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:29.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.789+0000 7f3bc9200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bc415d280 con 0x7f3bc4145590 2024-02-19T15:22:29.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.789+0000 7f3bc9200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bc4070ff0 con 0x7f3bc40cea80 2024-02-19T15:22:29.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.789+0000 7f3bc9200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc415cd60 con 0x7f3bc415c390 2024-02-19T15:22:29.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.789+0000 7f3bc1e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f3bc4145590 0x7f3bc406c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34130/0 (socket says 172.21.15.47:34130) 2024-02-19T15:22:29.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.789+0000 7f3bc1e00640 1 -- 172.21.15.47:0/989202165 learned_addr learned my addr 172.21.15.47:0/989202165 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:29.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.789+0000 7f3bc2800640 1 --2- 172.21.15.47:0/989202165 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bc415c390 0x7f3bc415c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:29.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.789+0000 7f3bc1400640 1 --2- 172.21.15.47:0/989202165 >> v2:172.21.15.116:3300/0 conn(0x7f3bc40cea80 0x7f3bc40727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:29.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.789+0000 7f3bc2800640 1 -- 172.21.15.47:0/989202165 >> v2:172.21.15.116:3300/0 conn(0x7f3bc40cea80 msgr2=0x7f3bc40727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:29.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.789+0000 7f3bc2800640 1 --2- 172.21.15.47:0/989202165 >> v2:172.21.15.116:3300/0 conn(0x7f3bc40cea80 0x7f3bc40727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.789+0000 7f3bc2800640 1 -- 172.21.15.47:0/989202165 >> v1:172.21.15.47:6789/0 conn(0x7f3bc4145590 legacy=0x7f3bc406c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:29.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.789+0000 7f3bc2800640 1 -- 172.21.15.47:0/989202165 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc40709e0 con 0x7f3bc415c390 2024-02-19T15:22:29.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.789+0000 7f3bc0a00640 1 -- 172.21.15.47:0/989202165 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062076027 0 0) 0x7f3bc415d280 con 0x7f3bc4145590 2024-02-19T15:22:29.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.793+0000 7f3bc2800640 1 --2- 172.21.15.47:0/989202165 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bc415c390 0x7f3bc415c820 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009d90 tx=0x7f3bbc01c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cedd919b2d91cc7c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:29.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.793+0000 7f3bc0a00640 1 -- 172.21.15.47:0/989202165 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc0a5920 con 0x7f3bc415c390 2024-02-19T15:22:29.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.793+0000 7f3bc0a00640 1 -- 172.21.15.47:0/989202165 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bbc0a5a80 con 0x7f3bc415c390 2024-02-19T15:22:29.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.793+0000 7f3bc0a00640 1 -- 172.21.15.47:0/989202165 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc0a5e10 con 0x7f3bc415c390 2024-02-19T15:22:29.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.793+0000 7f3bc9200640 1 -- 172.21.15.47:0/989202165 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bc415c390 msgr2=0x7f3bc415c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:29.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.793+0000 7f3bc9200640 1 --2- 172.21.15.47:0/989202165 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bc415c390 0x7f3bc415c820 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009d90 tx=0x7f3bbc01c970 comp rx=0 tx=0).stop 2024-02-19T15:22:29.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.793+0000 7f3bc9200640 1 -- 172.21.15.47:0/989202165 shutdown_connections 2024-02-19T15:22:29.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.793+0000 7f3bc9200640 1 --2- 172.21.15.47:0/989202165 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bc415c390 0x7f3bc415c820 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.793+0000 7f3bc9200640 1 --2- 172.21.15.47:0/989202165 >> v2:172.21.15.116:3300/0 conn(0x7f3bc40cea80 0x7f3bc40727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.793+0000 7f3bc9200640 1 -- 172.21.15.47:0/989202165 >> 172.21.15.47:0/989202165 conn(0x7f3bc40b16a0 msgr2=0x7f3bc40b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:29.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.793+0000 7f3bc9200640 1 -- 172.21.15.47:0/989202165 shutdown_connections 2024-02-19T15:22:29.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.793+0000 7f3bc9200640 1 -- 172.21.15.47:0/989202165 wait complete. 2024-02-19T15:22:29.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.793+0000 7f3bc9200640 1 Processor -- start 2024-02-19T15:22:29.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.793+0000 7f3bc9200640 1 -- start start 2024-02-19T15:22:29.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bc9200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bc40cea80 0x7f3bc41414d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:29.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bc9200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc4145590 0x7f3bc415af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:29.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bc9200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bc415c390 0x7f3bc4139ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:29.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bc9200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc413af20 con 0x7f3bc415c390 2024-02-19T15:22:29.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bc9200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc413a8f0 con 0x7f3bc4145590 2024-02-19T15:22:29.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bc9200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc4140ed0 con 0x7f3bc40cea80 2024-02-19T15:22:29.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bc2800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bc415c390 0x7f3bc4139ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:29.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bc2800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bc415c390 0x7f3bc4139ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60626/0 (socket says 172.21.15.47:60626) 2024-02-19T15:22:29.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bc2800640 1 -- 172.21.15.47:0/1718849180 learned_addr learned my addr 172.21.15.47:0/1718849180 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:29.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bc1400640 1 --2- 172.21.15.47:0/1718849180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc4145590 0x7f3bc415af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:29.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bc1e00640 1 --2- 172.21.15.47:0/1718849180 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bc40cea80 0x7f3bc41414d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:29.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bc2800640 1 -- 172.21.15.47:0/1718849180 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bc40cea80 msgr2=0x7f3bc41414d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:29.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bc2800640 1 --2- 172.21.15.47:0/1718849180 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bc40cea80 0x7f3bc41414d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bc2800640 1 -- 172.21.15.47:0/1718849180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc4145590 msgr2=0x7f3bc415af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:29.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bc2800640 1 --2- 172.21.15.47:0/1718849180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc4145590 0x7f3bc415af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bc2800640 1 -- 172.21.15.47:0/1718849180 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bbc009a70 con 0x7f3bc415c390 2024-02-19T15:22:29.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bc2800640 1 --2- 172.21.15.47:0/1718849180 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bc415c390 0x7f3bc4139ca0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc0b0040 tx=0x7f3bbc019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:29.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bba000640 1 -- 172.21.15.47:0/1718849180 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc0add90 con 0x7f3bc415c390 2024-02-19T15:22:29.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bba000640 1 -- 172.21.15.47:0/1718849180 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bbc009450 con 0x7f3bc415c390 2024-02-19T15:22:29.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bc9200640 1 -- 172.21.15.47:0/1718849180 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bc41411b0 con 0x7f3bc415c390 2024-02-19T15:22:29.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bba000640 1 -- 172.21.15.47:0/1718849180 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc0b9690 con 0x7f3bc415c390 2024-02-19T15:22:29.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bc9200640 1 -- 172.21.15.47:0/1718849180 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bc4140b60 con 0x7f3bc415c390 2024-02-19T15:22:29.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bc9200640 1 -- 172.21.15.47:0/1718849180 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bc413b090 con 0x7f3bc415c390 2024-02-19T15:22:29.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.797+0000 7f3bba000640 1 -- 172.21.15.47:0/1718849180 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3bbc0b97f0 con 0x7f3bc415c390 2024-02-19T15:22:29.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.801+0000 7f3bba000640 1 --2- 172.21.15.47:0/1718849180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ba4066e70 0x7f3ba4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:29.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.801+0000 7f3bba000640 1 -- 172.21.15.47:0/1718849180 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3bbc1201c0 con 0x7f3bc415c390 2024-02-19T15:22:29.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.801+0000 7f3bc1e00640 1 --2- 172.21.15.47:0/1718849180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ba4066e70 0x7f3ba4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:29.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.801+0000 7f3bc1e00640 1 --2- 172.21.15.47:0/1718849180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ba4066e70 0x7f3ba4069330 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4007fd0 tx=0x7f3bb4007d60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:29.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.805+0000 7f3bba000640 1 -- 172.21.15.47:0/1718849180 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f3bbc0ecf40 con 0x7f3bc415c390 2024-02-19T15:22:29.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.969+0000 7f3bc9200640 1 -- 172.21.15.47:0/1718849180 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bc4075360 con 0x7f3bc415c390 2024-02-19T15:22:29.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.973+0000 7f3bba000640 1 -- 172.21.15.47:0/1718849180 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f3bbc0f0600 con 0x7f3bc415c390 2024-02-19T15:22:29.975 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:29.975 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":21,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:06.843155+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:03.091360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:04.222965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:29.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.973+0000 7f3bc9200640 1 -- 172.21.15.47:0/1718849180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ba4066e70 msgr2=0x7f3ba4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:29.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.973+0000 7f3bc9200640 1 --2- 172.21.15.47:0/1718849180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ba4066e70 0x7f3ba4069330 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4007fd0 tx=0x7f3bb4007d60 comp rx=0 tx=0).stop 2024-02-19T15:22:29.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.973+0000 7f3bc9200640 1 -- 172.21.15.47:0/1718849180 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bc415c390 msgr2=0x7f3bc4139ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:29.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.973+0000 7f3bc9200640 1 --2- 172.21.15.47:0/1718849180 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bc415c390 0x7f3bc4139ca0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc0b0040 tx=0x7f3bbc019710 comp rx=0 tx=0).stop 2024-02-19T15:22:29.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.973+0000 7f3bc9200640 1 -- 172.21.15.47:0/1718849180 shutdown_connections 2024-02-19T15:22:29.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.973+0000 7f3bc9200640 1 --2- 172.21.15.47:0/1718849180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ba4066e70 0x7f3ba4069330 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.973+0000 7f3bc9200640 1 --2- 172.21.15.47:0/1718849180 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bc415c390 0x7f3bc4139ca0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.973+0000 7f3bc9200640 1 --2- 172.21.15.47:0/1718849180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc4145590 0x7f3bc415af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.973+0000 7f3bc9200640 1 --2- 172.21.15.47:0/1718849180 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bc40cea80 0x7f3bc41414d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:29.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.973+0000 7f3bc9200640 1 -- 172.21.15.47:0/1718849180 >> 172.21.15.47:0/1718849180 conn(0x7f3bc40b16a0 msgr2=0x7f3bc40b0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:29.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.973+0000 7f3bc9200640 1 -- 172.21.15.47:0/1718849180 shutdown_connections 2024-02-19T15:22:29.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:29.973+0000 7f3bc9200640 1 -- 172.21.15.47:0/1718849180 wait complete. 2024-02-19T15:22:29.997 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:22:03.091360+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:22:04.222965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:22:29.997 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:22:30.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.137+0000 7fa249400640 1 Processor -- start 2024-02-19T15:22:30.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.137+0000 7fa249400640 1 -- start start 2024-02-19T15:22:30.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.141+0000 7fa249400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa2440716d0 0x7fa24406cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:30.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.141+0000 7fa249400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa24406d110 0x7fa2440ca530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:30.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.141+0000 7fa249400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa24406e510 con 0x7fa24415a830 2024-02-19T15:22:30.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.141+0000 7fa249400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa24406e9a0 con 0x7fa24406d110 2024-02-19T15:22:30.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.141+0000 7fa249400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa24406d900 con 0x7fa2440716d0 2024-02-19T15:22:30.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.141+0000 7fa248a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa24406d110 0x7fa2440ca530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:30.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.141+0000 7fa248a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa24406d110 0x7fa2440ca530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60642/0 (socket says 172.21.15.47:60642) 2024-02-19T15:22:30.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.141+0000 7fa248a00640 1 -- 172.21.15.47:0/3888686051 learned_addr learned my addr 172.21.15.47:0/3888686051 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:30.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.141+0000 7fa243e00640 1 --1- 172.21.15.47:0/3888686051 >> v1:172.21.15.116:6789/0 conn(0x7fa24415a830 0x7fa244070fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:40364/0 (socket says 172.21.15.47:40364) 2024-02-19T15:22:30.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.141+0000 7fa23b400640 1 --2- 172.21.15.47:0/3888686051 >> v2:172.21.15.116:3300/0 conn(0x7fa2440716d0 0x7fa24406cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:30.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.141+0000 7fa248a00640 1 -- 172.21.15.47:0/3888686051 >> v2:172.21.15.116:3300/0 conn(0x7fa2440716d0 msgr2=0x7fa24406cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:30.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.141+0000 7fa248a00640 1 --2- 172.21.15.47:0/3888686051 >> v2:172.21.15.116:3300/0 conn(0x7fa2440716d0 0x7fa24406cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.141+0000 7fa248a00640 1 -- 172.21.15.47:0/3888686051 >> v1:172.21.15.116:6789/0 conn(0x7fa24415a830 legacy=0x7fa244070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:30.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.141+0000 7fa248a00640 1 -- 172.21.15.47:0/3888686051 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24406dbe0 con 0x7fa24406d110 2024-02-19T15:22:30.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.141+0000 7fa243400640 1 -- 172.21.15.47:0/3888686051 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 761727985 0 0) 0x7fa24406e510 con 0x7fa24415a830 2024-02-19T15:22:30.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.141+0000 7fa248a00640 1 --2- 172.21.15.47:0/3888686051 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa24406d110 0x7fa2440ca530 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa2440ccf30 tx=0x7fa23c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6644e38ad05ce0a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:30.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.141+0000 7fa243400640 1 -- 172.21.15.47:0/3888686051 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c0a57a0 con 0x7fa24406d110 2024-02-19T15:22:30.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.141+0000 7fa243400640 1 -- 172.21.15.47:0/3888686051 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa23c0a5900 con 0x7fa24406d110 2024-02-19T15:22:30.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.141+0000 7fa243400640 1 -- 172.21.15.47:0/3888686051 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c0a5c90 con 0x7fa24406d110 2024-02-19T15:22:30.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa249400640 1 -- 172.21.15.47:0/3888686051 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa24406d110 msgr2=0x7fa2440ca530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:30.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa249400640 1 --2- 172.21.15.47:0/3888686051 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa24406d110 0x7fa2440ca530 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa2440ccf30 tx=0x7fa23c01c9f0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa249400640 1 -- 172.21.15.47:0/3888686051 shutdown_connections 2024-02-19T15:22:30.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa249400640 1 --2- 172.21.15.47:0/3888686051 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa24406d110 0x7fa2440ca530 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa249400640 1 --2- 172.21.15.47:0/3888686051 >> v2:172.21.15.116:3300/0 conn(0x7fa2440716d0 0x7fa24406cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa249400640 1 -- 172.21.15.47:0/3888686051 >> 172.21.15.47:0/3888686051 conn(0x7fa2440b12d0 msgr2=0x7fa2440b16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:30.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa249400640 1 -- 172.21.15.47:0/3888686051 shutdown_connections 2024-02-19T15:22:30.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa249400640 1 -- 172.21.15.47:0/3888686051 wait complete. 2024-02-19T15:22:30.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa249400640 1 Processor -- start 2024-02-19T15:22:30.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa249400640 1 -- start start 2024-02-19T15:22:30.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa249400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa24406d110 0x7fa2441534f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:30.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa249400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2440716d0 0x7fa2440b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:30.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa249400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa24415a830 0x7fa2440cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:30.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa249400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2440c6120 con 0x7fa24415a830 2024-02-19T15:22:30.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa249400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2440b4ce0 con 0x7fa24406d110 2024-02-19T15:22:30.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa249400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2440b43b0 con 0x7fa2440716d0 2024-02-19T15:22:30.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa248a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa24415a830 0x7fa2440cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:30.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa243e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa24406d110 0x7fa2441534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:30.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa248a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa24415a830 0x7fa2440cdf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60654/0 (socket says 172.21.15.47:60654) 2024-02-19T15:22:30.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa248a00640 1 -- 172.21.15.47:0/364058329 learned_addr learned my addr 172.21.15.47:0/364058329 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:30.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa23b400640 1 --2- 172.21.15.47:0/364058329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2440716d0 0x7fa2440b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:30.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa248a00640 1 -- 172.21.15.47:0/364058329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2440716d0 msgr2=0x7fa2440b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:30.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa248a00640 1 --2- 172.21.15.47:0/364058329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2440716d0 0x7fa2440b23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa248a00640 1 -- 172.21.15.47:0/364058329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa24406d110 msgr2=0x7fa2441534f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:30.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa248a00640 1 --2- 172.21.15.47:0/364058329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa24406d110 0x7fa2441534f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.145+0000 7fa248a00640 1 -- 172.21.15.47:0/364058329 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23c009b30 con 0x7fa24415a830 2024-02-19T15:22:30.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.149+0000 7fa248a00640 1 --2- 172.21.15.47:0/364058329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa24415a830 0x7fa2440cdf00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa23c0af040 tx=0x7fa23c019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:30.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.149+0000 7fa240c00640 1 -- 172.21.15.47:0/364058329 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c0b8670 con 0x7fa24415a830 2024-02-19T15:22:30.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.149+0000 7fa240c00640 1 -- 172.21.15.47:0/364058329 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa23c0b87d0 con 0x7fa24415a830 2024-02-19T15:22:30.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.149+0000 7fa240c00640 1 -- 172.21.15.47:0/364058329 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c0b8b80 con 0x7fa24415a830 2024-02-19T15:22:30.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.149+0000 7fa249400640 1 -- 172.21.15.47:0/364058329 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2440b4690 con 0x7fa24415a830 2024-02-19T15:22:30.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.149+0000 7fa249400640 1 -- 172.21.15.47:0/364058329 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa244153a30 con 0x7fa24415a830 2024-02-19T15:22:30.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.149+0000 7fa249400640 1 -- 172.21.15.47:0/364058329 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2440684d0 con 0x7fa24415a830 2024-02-19T15:22:30.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.149+0000 7fa240c00640 1 -- 172.21.15.47:0/364058329 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa23c0b8ce0 con 0x7fa24415a830 2024-02-19T15:22:30.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.149+0000 7fa240c00640 1 --2- 172.21.15.47:0/364058329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa220066e40 0x7fa220069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:30.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.149+0000 7fa240c00640 1 -- 172.21.15.47:0/364058329 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa23c11fe80 con 0x7fa24415a830 2024-02-19T15:22:30.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.149+0000 7fa243e00640 1 --2- 172.21.15.47:0/364058329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa220066e40 0x7fa220069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:30.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.153+0000 7fa243e00640 1 --2- 172.21.15.47:0/364058329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa220066e40 0x7fa220069300 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa234000f70 tx=0x7fa234007df0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:30.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.157+0000 7fa240c00640 1 -- 172.21.15.47:0/364058329 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa23c0ecc80 con 0x7fa24415a830 2024-02-19T15:22:30.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.321+0000 7fa249400640 1 -- 172.21.15.47:0/364058329 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa24404f260 con 0x7fa24415a830 2024-02-19T15:22:30.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.321+0000 7fa240c00640 1 -- 172.21.15.47:0/364058329 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa23c0f0340 con 0x7fa24415a830 2024-02-19T15:22:30.324 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:22:30.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.321+0000 7fa249400640 1 -- 172.21.15.47:0/364058329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa220066e40 msgr2=0x7fa220069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:30.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.321+0000 7fa249400640 1 --2- 172.21.15.47:0/364058329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa220066e40 0x7fa220069300 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa234000f70 tx=0x7fa234007df0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.321+0000 7fa249400640 1 -- 172.21.15.47:0/364058329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa24415a830 msgr2=0x7fa2440cdf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:30.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.321+0000 7fa249400640 1 --2- 172.21.15.47:0/364058329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa24415a830 0x7fa2440cdf00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa23c0af040 tx=0x7fa23c019790 comp rx=0 tx=0).stop 2024-02-19T15:22:30.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.321+0000 7fa249400640 1 -- 172.21.15.47:0/364058329 shutdown_connections 2024-02-19T15:22:30.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.321+0000 7fa249400640 1 --2- 172.21.15.47:0/364058329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa220066e40 0x7fa220069300 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.321+0000 7fa249400640 1 --2- 172.21.15.47:0/364058329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa24415a830 0x7fa2440cdf00 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.321+0000 7fa249400640 1 --2- 172.21.15.47:0/364058329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2440716d0 0x7fa2440b23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.321+0000 7fa249400640 1 --2- 172.21.15.47:0/364058329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa24406d110 0x7fa2441534f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.321+0000 7fa249400640 1 -- 172.21.15.47:0/364058329 >> 172.21.15.47:0/364058329 conn(0x7fa2440b12d0 msgr2=0x7fa2440ad850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:30.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.325+0000 7fa249400640 1 -- 172.21.15.47:0/364058329 shutdown_connections 2024-02-19T15:22:30.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.325+0000 7fa249400640 1 -- 172.21.15.47:0/364058329 wait complete. 2024-02-19T15:22:30.346 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:22:30.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.485+0000 7f2a9b800640 1 Processor -- start 2024-02-19T15:22:30.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.485+0000 7f2a9b800640 1 -- start start 2024-02-19T15:22:30.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.485+0000 7f2a9b800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2a940ce3b0 0x7f2a940ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:30.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.485+0000 7f2a9b800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2a940d2650 0x7f2a940ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:30.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.485+0000 7f2a9b800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a940cc620 con 0x7f2a9415e280 2024-02-19T15:22:30.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.485+0000 7f2a9b800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a940d12e0 con 0x7f2a940ce3b0 2024-02-19T15:22:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.485+0000 7f2a9b800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a940d1940 con 0x7f2a940d2650 2024-02-19T15:22:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.485+0000 7f2a98e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2a940ce3b0 0x7f2a940ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.485+0000 7f2a93e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2a940d2650 0x7f2a940ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.485+0000 7f2a98e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2a940ce3b0 0x7f2a940ce790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60656/0 (socket says 172.21.15.47:60656) 2024-02-19T15:22:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.485+0000 7f2a98e00640 1 -- 172.21.15.47:0/3295408364 learned_addr learned my addr 172.21.15.47:0/3295408364 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.485+0000 7f2a93e00640 1 -- 172.21.15.47:0/3295408364 >> v2:172.21.15.47:3300/0 conn(0x7f2a940ce3b0 msgr2=0x7f2a940ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.485+0000 7f2a93e00640 1 --2- 172.21.15.47:0/3295408364 >> v2:172.21.15.47:3300/0 conn(0x7f2a940ce3b0 0x7f2a940ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.485+0000 7f2a93e00640 1 -- 172.21.15.47:0/3295408364 >> v1:172.21.15.116:6789/0 conn(0x7f2a9415e280 legacy=0x7f2a940ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.485+0000 7f2a93e00640 1 -- 172.21.15.47:0/3295408364 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a940cecd0 con 0x7f2a940d2650 2024-02-19T15:22:30.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.489+0000 7f2a93e00640 1 --2- 172.21.15.47:0/3295408364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a940d2650 0x7f2a940ccab0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2a94071dd0 tx=0x7f2a8001c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d95a2135f30fab0c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:30.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.489+0000 7f2a93400640 1 -- 172.21.15.47:0/3295408364 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a800ab600 con 0x7f2a940d2650 2024-02-19T15:22:30.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.489+0000 7f2a93400640 1 -- 172.21.15.47:0/3295408364 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a800a7050 con 0x7f2a940d2650 2024-02-19T15:22:30.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.489+0000 7f2a93400640 1 -- 172.21.15.47:0/3295408364 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a800ab980 con 0x7f2a940d2650 2024-02-19T15:22:30.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.489+0000 7f2a9b800640 1 -- 172.21.15.47:0/3295408364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a940d2650 msgr2=0x7f2a940ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:30.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.489+0000 7f2a9b800640 1 --2- 172.21.15.47:0/3295408364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a940d2650 0x7f2a940ccab0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2a94071dd0 tx=0x7f2a8001c360 comp rx=0 tx=0).stop 2024-02-19T15:22:30.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.489+0000 7f2a9b800640 1 -- 172.21.15.47:0/3295408364 shutdown_connections 2024-02-19T15:22:30.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.489+0000 7f2a9b800640 1 --2- 172.21.15.47:0/3295408364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a940d2650 0x7f2a940ccab0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.489+0000 7f2a9b800640 1 --2- 172.21.15.47:0/3295408364 >> v2:172.21.15.47:3300/0 conn(0x7f2a940ce3b0 0x7f2a940ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.489+0000 7f2a9b800640 1 -- 172.21.15.47:0/3295408364 >> 172.21.15.47:0/3295408364 conn(0x7f2a940d1b30 msgr2=0x7f2a940d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:30.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.489+0000 7f2a9b800640 1 -- 172.21.15.47:0/3295408364 shutdown_connections 2024-02-19T15:22:30.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.489+0000 7f2a9b800640 1 -- 172.21.15.47:0/3295408364 wait complete. 2024-02-19T15:22:30.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a9b800640 1 Processor -- start 2024-02-19T15:22:30.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a9b800640 1 -- start start 2024-02-19T15:22:30.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a9b800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a940ce3b0 0x7f2a94159e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:30.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a9b800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a940d2650 0x7f2a940b4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:30.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a9b800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a9415e280 0x7f2a9415b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:30.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a9b800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a940b4280 con 0x7f2a940d2650 2024-02-19T15:22:30.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a9b800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a940b3670 con 0x7f2a9415e280 2024-02-19T15:22:30.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a9b800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a940b3950 con 0x7f2a940ce3b0 2024-02-19T15:22:30.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a93e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a940d2650 0x7f2a940b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:30.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a93e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a940d2650 0x7f2a940b4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60660/0 (socket says 172.21.15.47:60660) 2024-02-19T15:22:30.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a93e00640 1 -- 172.21.15.47:0/2910248553 learned_addr learned my addr 172.21.15.47:0/2910248553 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:30.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a99800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a9415e280 0x7f2a9415b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:30.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a98e00640 1 --2- 172.21.15.47:0/2910248553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a940ce3b0 0x7f2a94159e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:30.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a93e00640 1 -- 172.21.15.47:0/2910248553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a940ce3b0 msgr2=0x7f2a94159e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:30.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a93e00640 1 --2- 172.21.15.47:0/2910248553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a940ce3b0 0x7f2a94159e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a93e00640 1 -- 172.21.15.47:0/2910248553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a9415e280 msgr2=0x7f2a9415b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:30.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a93e00640 1 --2- 172.21.15.47:0/2910248553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a9415e280 0x7f2a9415b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a93e00640 1 -- 172.21.15.47:0/2910248553 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a80009420 con 0x7f2a940d2650 2024-02-19T15:22:30.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a93e00640 1 --2- 172.21.15.47:0/2910248553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a940d2650 0x7f2a940b4710 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2a800ae040 tx=0x7f2a8001aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:30.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a90c00640 1 -- 172.21.15.47:0/2910248553 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a800b7450 con 0x7f2a940d2650 2024-02-19T15:22:30.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a90c00640 1 -- 172.21.15.47:0/2910248553 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a800b75d0 con 0x7f2a940d2650 2024-02-19T15:22:30.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a90c00640 1 -- 172.21.15.47:0/2910248553 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a800b7980 con 0x7f2a940d2650 2024-02-19T15:22:30.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a9b800640 1 -- 172.21.15.47:0/2910248553 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a940b3c30 con 0x7f2a940d2650 2024-02-19T15:22:30.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.493+0000 7f2a9b800640 1 -- 172.21.15.47:0/2910248553 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a9415a350 con 0x7f2a940d2650 2024-02-19T15:22:30.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.497+0000 7f2a9b800640 1 -- 172.21.15.47:0/2910248553 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a940b43f0 con 0x7f2a940d2650 2024-02-19T15:22:30.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.497+0000 7f2a90c00640 1 -- 172.21.15.47:0/2910248553 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2a800b7ae0 con 0x7f2a940d2650 2024-02-19T15:22:30.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.497+0000 7f2a90c00640 1 --2- 172.21.15.47:0/2910248553 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2a74066e40 0x7f2a74069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:30.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.497+0000 7f2a90c00640 1 -- 172.21.15.47:0/2910248553 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a8011f0a0 con 0x7f2a940d2650 2024-02-19T15:22:30.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.497+0000 7f2a98e00640 1 --2- 172.21.15.47:0/2910248553 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2a74066e40 0x7f2a74069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:30.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.497+0000 7f2a98e00640 1 --2- 172.21.15.47:0/2910248553 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2a74066e40 0x7f2a74069300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2a88009ab0 tx=0x7f2a88009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:30.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.501+0000 7f2a90c00640 1 -- 172.21.15.47:0/2910248553 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2a800ebea0 con 0x7f2a940d2650 2024-02-19T15:22:30.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.665+0000 7f2a9b800640 1 -- 172.21.15.47:0/2910248553 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2a94075360 con 0x7f2a940d2650 2024-02-19T15:22:30.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.665+0000 7f2a90c00640 1 -- 172.21.15.47:0/2910248553 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2a800ef560 con 0x7f2a940d2650 2024-02-19T15:22:30.668 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:22:30.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.665+0000 7f2a9b800640 1 -- 172.21.15.47:0/2910248553 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2a74066e40 msgr2=0x7f2a74069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:30.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.665+0000 7f2a9b800640 1 --2- 172.21.15.47:0/2910248553 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2a74066e40 0x7f2a74069300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2a88009ab0 tx=0x7f2a88009240 comp rx=0 tx=0).stop 2024-02-19T15:22:30.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.665+0000 7f2a9b800640 1 -- 172.21.15.47:0/2910248553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a940d2650 msgr2=0x7f2a940b4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:30.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.665+0000 7f2a9b800640 1 --2- 172.21.15.47:0/2910248553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a940d2650 0x7f2a940b4710 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2a800ae040 tx=0x7f2a8001aba0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.665+0000 7f2a9b800640 1 -- 172.21.15.47:0/2910248553 shutdown_connections 2024-02-19T15:22:30.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.665+0000 7f2a9b800640 1 --2- 172.21.15.47:0/2910248553 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2a74066e40 0x7f2a74069300 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.665+0000 7f2a9b800640 1 --2- 172.21.15.47:0/2910248553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a9415e280 0x7f2a9415b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.665+0000 7f2a9b800640 1 --2- 172.21.15.47:0/2910248553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a940d2650 0x7f2a940b4710 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.665+0000 7f2a9b800640 1 --2- 172.21.15.47:0/2910248553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a940ce3b0 0x7f2a94159e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.665+0000 7f2a9b800640 1 -- 172.21.15.47:0/2910248553 >> 172.21.15.47:0/2910248553 conn(0x7f2a940d1b30 msgr2=0x7f2a940cc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:30.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.665+0000 7f2a9b800640 1 -- 172.21.15.47:0/2910248553 shutdown_connections 2024-02-19T15:22:30.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.665+0000 7f2a9b800640 1 -- 172.21.15.47:0/2910248553 wait complete. 2024-02-19T15:22:30.689 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:22:30.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.829+0000 7f80d5400640 1 Processor -- start 2024-02-19T15:22:30.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.829+0000 7f80d5400640 1 -- start start 2024-02-19T15:22:30.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.829+0000 7f80d5400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80d015e160 0x7f80d00d1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:30.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.829+0000 7f80d5400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f80d006e1e0 0x7f80d015e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:30.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.829+0000 7f80d5400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d0071a70 con 0x7f80d006f220 2024-02-19T15:22:30.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.829+0000 7f80d5400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80d006f680 con 0x7f80d006e1e0 2024-02-19T15:22:30.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.829+0000 7f80d5400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80d00ced30 con 0x7f80d015e160 2024-02-19T15:22:30.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.829+0000 7f80cf200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f80d006e1e0 0x7f80d015e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:30.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.829+0000 7f80c7e00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f80d006f220 0x7f80d006dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:40394/0 (socket says 172.21.15.47:40394) 2024-02-19T15:22:30.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.829+0000 7f80c7e00640 1 -- 172.21.15.47:0/387409985 learned_addr learned my addr 172.21.15.47:0/387409985 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:30.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.829+0000 7f80ce800640 1 --2- 172.21.15.47:0/387409985 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80d015e160 0x7f80d00d1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:30.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.829+0000 7f80cde00640 1 -- 172.21.15.47:0/387409985 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004906250 0 0) 0x7f80d0071a70 con 0x7f80d006f220 2024-02-19T15:22:30.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.829+0000 7f80cde00640 1 -- 172.21.15.47:0/387409985 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b4003530 con 0x7f80d006f220 2024-02-19T15:22:30.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.829+0000 7f80cf200640 1 -- 172.21.15.47:0/387409985 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80d015e160 msgr2=0x7f80d00d1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:30.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.829+0000 7f80cf200640 1 --2- 172.21.15.47:0/387409985 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80d015e160 0x7f80d00d1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.829+0000 7f80cf200640 1 -- 172.21.15.47:0/387409985 >> v1:172.21.15.116:6789/0 conn(0x7f80d006f220 legacy=0x7f80d006dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:30.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.829+0000 7f80cf200640 1 -- 172.21.15.47:0/387409985 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d0162a00 con 0x7f80d006e1e0 2024-02-19T15:22:30.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.829+0000 7f80ce800640 1 --2- 172.21.15.47:0/387409985 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80d015e160 0x7f80d00d1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:30.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.833+0000 7f80cf200640 1 --2- 172.21.15.47:0/387409985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80d006e1e0 0x7f80d015e540 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f80c8009d90 tx=0x7f80c801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=767cd3c4a0748615 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:30.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.833+0000 7f80cde00640 1 -- 172.21.15.47:0/387409985 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c80a5a00 con 0x7f80d006e1e0 2024-02-19T15:22:30.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.833+0000 7f80cde00640 1 -- 172.21.15.47:0/387409985 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c80a5b60 con 0x7f80d006e1e0 2024-02-19T15:22:30.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.833+0000 7f80cde00640 1 -- 172.21.15.47:0/387409985 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c80ad440 con 0x7f80d006e1e0 2024-02-19T15:22:30.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.833+0000 7f80d5400640 1 -- 172.21.15.47:0/387409985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80d006e1e0 msgr2=0x7f80d015e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:30.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.833+0000 7f80d5400640 1 --2- 172.21.15.47:0/387409985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80d006e1e0 0x7f80d015e540 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f80c8009d90 tx=0x7f80c801c940 comp rx=0 tx=0).stop 2024-02-19T15:22:30.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.833+0000 7f80d5400640 1 -- 172.21.15.47:0/387409985 shutdown_connections 2024-02-19T15:22:30.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.833+0000 7f80d5400640 1 --2- 172.21.15.47:0/387409985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80d006e1e0 0x7f80d015e540 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.833+0000 7f80d5400640 1 --2- 172.21.15.47:0/387409985 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80d015e160 0x7f80d00d1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.833+0000 7f80d5400640 1 -- 172.21.15.47:0/387409985 >> 172.21.15.47:0/387409985 conn(0x7f80d00d0d30 msgr2=0x7f80d00d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:30.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.833+0000 7f80d5400640 1 -- 172.21.15.47:0/387409985 shutdown_connections 2024-02-19T15:22:30.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.833+0000 7f80d5400640 1 -- 172.21.15.47:0/387409985 wait complete. 2024-02-19T15:22:30.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80d5400640 1 Processor -- start 2024-02-19T15:22:30.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80d5400640 1 -- start start 2024-02-19T15:22:30.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80d5400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d006e1e0 0x7f80d00b09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:30.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80d5400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80d006f220 0x7f80d014f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:30.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80d5400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80d015e160 0x7f80d014fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:30.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80d5400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d0157c80 con 0x7f80d015e160 2024-02-19T15:22:30.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80d5400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d01570d0 con 0x7f80d006e1e0 2024-02-19T15:22:30.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80d5400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80d0150350 con 0x7f80d006f220 2024-02-19T15:22:30.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80cf200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80d015e160 0x7f80d014fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:30.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80cf200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80d015e160 0x7f80d014fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60666/0 (socket says 172.21.15.47:60666) 2024-02-19T15:22:30.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80cf200640 1 -- 172.21.15.47:0/2107636594 learned_addr learned my addr 172.21.15.47:0/2107636594 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:30.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80ce800640 1 --2- 172.21.15.47:0/2107636594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d006e1e0 0x7f80d00b09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:30.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80c7e00640 1 --2- 172.21.15.47:0/2107636594 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80d006f220 0x7f80d014f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:30.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80cf200640 1 -- 172.21.15.47:0/2107636594 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80d006f220 msgr2=0x7f80d014f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:30.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80cf200640 1 --2- 172.21.15.47:0/2107636594 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80d006f220 0x7f80d014f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80cf200640 1 -- 172.21.15.47:0/2107636594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d006e1e0 msgr2=0x7f80d00b09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:30.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80cf200640 1 --2- 172.21.15.47:0/2107636594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d006e1e0 0x7f80d00b09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:30.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80cf200640 1 -- 172.21.15.47:0/2107636594 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c0009420 con 0x7f80d015e160 2024-02-19T15:22:30.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80cf200640 1 --2- 172.21.15.47:0/2107636594 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80d015e160 0x7f80d014fe10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f80c801a570 tx=0x7f80c80196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:30.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80c6a00640 1 -- 172.21.15.47:0/2107636594 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c80aec30 con 0x7f80d015e160 2024-02-19T15:22:30.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80c6a00640 1 -- 172.21.15.47:0/2107636594 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c80aed90 con 0x7f80d015e160 2024-02-19T15:22:30.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80c6a00640 1 -- 172.21.15.47:0/2107636594 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c80b7650 con 0x7f80d015e160 2024-02-19T15:22:30.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80d5400640 1 -- 172.21.15.47:0/2107636594 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c8009a70 con 0x7f80d015e160 2024-02-19T15:22:30.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.837+0000 7f80d5400640 1 -- 172.21.15.47:0/2107636594 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d01575c0 con 0x7f80d015e160 2024-02-19T15:22:30.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.841+0000 7f80d5400640 1 -- 172.21.15.47:0/2107636594 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d0157dc0 con 0x7f80d015e160 2024-02-19T15:22:30.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.841+0000 7f80c6a00640 1 -- 172.21.15.47:0/2107636594 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f80c80b4050 con 0x7f80d015e160 2024-02-19T15:22:30.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.841+0000 7f80c6a00640 1 --2- 172.21.15.47:0/2107636594 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f80ac066e40 0x7f80ac069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:30.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.841+0000 7f80c6a00640 1 -- 172.21.15.47:0/2107636594 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f80c811f890 con 0x7f80d015e160 2024-02-19T15:22:30.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.841+0000 7f80ce800640 1 --2- 172.21.15.47:0/2107636594 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f80ac066e40 0x7f80ac069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:30.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.841+0000 7f80ce800640 1 --2- 172.21.15.47:0/2107636594 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f80ac066e40 0x7f80ac069300 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f80d00b23d0 tx=0x7f80c0009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:30.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:30.845+0000 7f80c6a00640 1 -- 172.21.15.47:0/2107636594 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f80d0157dc0 con 0x7f80d015e160 2024-02-19T15:22:31.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.005+0000 7f80d5400640 1 -- 172.21.15.47:0/2107636594 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f80d0157dc0 con 0x7f80d015e160 2024-02-19T15:22:31.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.009+0000 7f80c6a00640 1 -- 172.21.15.47:0/2107636594 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f80d0157dc0 con 0x7f80d015e160 2024-02-19T15:22:31.010 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:22:31.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.009+0000 7f80d5400640 1 -- 172.21.15.47:0/2107636594 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f80ac066e40 msgr2=0x7f80ac069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:31.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.009+0000 7f80d5400640 1 --2- 172.21.15.47:0/2107636594 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f80ac066e40 0x7f80ac069300 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f80d00b23d0 tx=0x7f80c0009190 comp rx=0 tx=0).stop 2024-02-19T15:22:31.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.009+0000 7f80d5400640 1 -- 172.21.15.47:0/2107636594 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80d015e160 msgr2=0x7f80d014fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:31.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.009+0000 7f80d5400640 1 --2- 172.21.15.47:0/2107636594 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80d015e160 0x7f80d014fe10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f80c801a570 tx=0x7f80c80196e0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.009+0000 7f80d5400640 1 -- 172.21.15.47:0/2107636594 shutdown_connections 2024-02-19T15:22:31.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.009+0000 7f80d5400640 1 --2- 172.21.15.47:0/2107636594 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f80ac066e40 0x7f80ac069300 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.009+0000 7f80d5400640 1 --2- 172.21.15.47:0/2107636594 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80d015e160 0x7f80d014fe10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.009+0000 7f80d5400640 1 --2- 172.21.15.47:0/2107636594 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80d006f220 0x7f80d014f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.009+0000 7f80d5400640 1 --2- 172.21.15.47:0/2107636594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d006e1e0 0x7f80d00b09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.009+0000 7f80d5400640 1 -- 172.21.15.47:0/2107636594 >> 172.21.15.47:0/2107636594 conn(0x7f80d00d0d30 msgr2=0x7f80d015d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:31.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.009+0000 7f80d5400640 1 -- 172.21.15.47:0/2107636594 shutdown_connections 2024-02-19T15:22:31.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.009+0000 7f80d5400640 1 -- 172.21.15.47:0/2107636594 wait complete. 2024-02-19T15:22:31.031 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:31.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.169+0000 7feaa4000640 1 Processor -- start 2024-02-19T15:22:31.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.169+0000 7feaa4000640 1 -- start start 2024-02-19T15:22:31.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.169+0000 7feaa4000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fea9c0cded0 0x7fea9c0ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:31.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.169+0000 7feaa4000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea9c0ceb40 0x7fea9c06bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:31.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.169+0000 7feaa4000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea9c0cce30 con 0x7fea9c06d650 2024-02-19T15:22:31.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.169+0000 7feaa4000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea9c0cefd0 con 0x7fea9c0cded0 2024-02-19T15:22:31.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.169+0000 7feaa4000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea9c0cd110 con 0x7fea9c0ceb40 2024-02-19T15:22:31.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.169+0000 7feaa0c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fea9c0cded0 0x7fea9c0ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:31.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.169+0000 7feaa1600640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fea9c06d650 0x7fea9c0d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:40408/0 (socket says 172.21.15.47:40408) 2024-02-19T15:22:31.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.169+0000 7feaa1600640 1 -- 172.21.15.47:0/1561651453 learned_addr learned my addr 172.21.15.47:0/1561651453 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:31.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.169+0000 7feaa2000640 1 --2- 172.21.15.47:0/1561651453 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea9c0ceb40 0x7fea9c06bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:31.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.169+0000 7feaa0c00640 1 -- 172.21.15.47:0/1561651453 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea9c0ceb40 msgr2=0x7fea9c06bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:31.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.169+0000 7feaa0c00640 1 --2- 172.21.15.47:0/1561651453 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea9c0ceb40 0x7fea9c06bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.169+0000 7feaa0c00640 1 -- 172.21.15.47:0/1561651453 >> v1:172.21.15.116:6789/0 conn(0x7fea9c06d650 legacy=0x7fea9c0d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:31.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.169+0000 7feaa0c00640 1 -- 172.21.15.47:0/1561651453 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea9c0cd3f0 con 0x7fea9c0cded0 2024-02-19T15:22:31.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.169+0000 7fea9be00640 1 -- 172.21.15.47:0/1561651453 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900037152 0 0) 0x7fea9c0cce30 con 0x7fea9c06d650 2024-02-19T15:22:31.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.173+0000 7feaa0c00640 1 --2- 172.21.15.47:0/1561651453 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea9c0cded0 0x7fea9c0ce600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fea880097c0 tx=0x7fea8801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7f87c09f5676ae server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:31.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.173+0000 7fea9be00640 1 -- 172.21.15.47:0/1561651453 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea880ac640 con 0x7fea9c0cded0 2024-02-19T15:22:31.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.173+0000 7fea9be00640 1 -- 172.21.15.47:0/1561651453 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea880ac7a0 con 0x7fea9c0cded0 2024-02-19T15:22:31.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.173+0000 7fea9be00640 1 -- 172.21.15.47:0/1561651453 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea880acb30 con 0x7fea9c0cded0 2024-02-19T15:22:31.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.173+0000 7feaa4000640 1 -- 172.21.15.47:0/1561651453 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea9c0cded0 msgr2=0x7fea9c0ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:31.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.173+0000 7feaa4000640 1 --2- 172.21.15.47:0/1561651453 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea9c0cded0 0x7fea9c0ce600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fea880097c0 tx=0x7fea8801c3e0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.173+0000 7feaa4000640 1 -- 172.21.15.47:0/1561651453 shutdown_connections 2024-02-19T15:22:31.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.173+0000 7feaa4000640 1 --2- 172.21.15.47:0/1561651453 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea9c0ceb40 0x7fea9c06bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.173+0000 7feaa4000640 1 --2- 172.21.15.47:0/1561651453 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea9c0cded0 0x7fea9c0ce600 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.173+0000 7feaa4000640 1 -- 172.21.15.47:0/1561651453 >> 172.21.15.47:0/1561651453 conn(0x7fea9c0d1bf0 msgr2=0x7fea9c0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:31.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.173+0000 7feaa4000640 1 -- 172.21.15.47:0/1561651453 shutdown_connections 2024-02-19T15:22:31.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.173+0000 7feaa4000640 1 -- 172.21.15.47:0/1561651453 wait complete. 2024-02-19T15:22:31.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa4000640 1 Processor -- start 2024-02-19T15:22:31.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa4000640 1 -- start start 2024-02-19T15:22:31.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa4000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea9c06d650 0x7fea9c0ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:31.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa4000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea9c0cded0 0x7fea9c0b51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:31.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa4000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea9c0ceb40 0x7fea9c1600a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:31.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa4000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9c0caea0 con 0x7fea9c0ceb40 2024-02-19T15:22:31.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa4000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9c0cb180 con 0x7fea9c06d650 2024-02-19T15:22:31.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa4000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea9c0cb460 con 0x7fea9c0cded0 2024-02-19T15:22:31.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa2000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea9c0ceb40 0x7fea9c1600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:31.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa1600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea9c06d650 0x7fea9c0ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:31.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa2000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea9c0ceb40 0x7fea9c1600a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60692/0 (socket says 172.21.15.47:60692) 2024-02-19T15:22:31.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa2000640 1 -- 172.21.15.47:0/23986391 learned_addr learned my addr 172.21.15.47:0/23986391 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:31.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa0c00640 1 --2- 172.21.15.47:0/23986391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea9c0cded0 0x7fea9c0b51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:31.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa2000640 1 -- 172.21.15.47:0/23986391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea9c0cded0 msgr2=0x7fea9c0b51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:31.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa2000640 1 --2- 172.21.15.47:0/23986391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea9c0cded0 0x7fea9c0b51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa2000640 1 -- 172.21.15.47:0/23986391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea9c06d650 msgr2=0x7fea9c0ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:31.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa2000640 1 --2- 172.21.15.47:0/23986391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea9c06d650 0x7fea9c0ca8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa2000640 1 -- 172.21.15.47:0/23986391 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea88009420 con 0x7fea9c0ceb40 2024-02-19T15:22:31.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa2000640 1 --2- 172.21.15.47:0/23986391 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea9c0ceb40 0x7fea9c1600a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fea9400efc0 tx=0x7fea9400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:31.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7fea99600640 1 -- 172.21.15.47:0/23986391 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea94014480 con 0x7fea9c0ceb40 2024-02-19T15:22:31.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7fea99600640 1 -- 172.21.15.47:0/23986391 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea9400f040 con 0x7fea9c0ceb40 2024-02-19T15:22:31.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7fea99600640 1 -- 172.21.15.47:0/23986391 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea94014820 con 0x7fea9c0ceb40 2024-02-19T15:22:31.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa4000640 1 -- 172.21.15.47:0/23986391 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea9c0cb740 con 0x7fea9c0ceb40 2024-02-19T15:22:31.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa4000640 1 -- 172.21.15.47:0/23986391 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea9c15b300 con 0x7fea9c0ceb40 2024-02-19T15:22:31.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.177+0000 7feaa4000640 1 -- 172.21.15.47:0/23986391 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea9c075360 con 0x7fea9c0ceb40 2024-02-19T15:22:31.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.181+0000 7fea99600640 1 -- 172.21.15.47:0/23986391 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fea94014980 con 0x7fea9c0ceb40 2024-02-19T15:22:31.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.181+0000 7fea99600640 1 --2- 172.21.15.47:0/23986391 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fea7c066e40 0x7fea7c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:31.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.181+0000 7fea99600640 1 -- 172.21.15.47:0/23986391 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea94085540 con 0x7fea9c0ceb40 2024-02-19T15:22:31.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.181+0000 7feaa1600640 1 --2- 172.21.15.47:0/23986391 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fea7c066e40 0x7fea7c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:31.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.181+0000 7feaa1600640 1 --2- 172.21.15.47:0/23986391 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fea7c066e40 0x7fea7c069300 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fea90000f70 tx=0x7fea90007df0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:31.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.185+0000 7fea99600640 1 -- 172.21.15.47:0/23986391 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fea94052340 con 0x7fea9c0ceb40 2024-02-19T15:22:31.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.349+0000 7feaa4000640 1 -- 172.21.15.47:0/23986391 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea9c0cbbf0 con 0x7fea9c0ceb40 2024-02-19T15:22:31.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.349+0000 7fea99600640 1 -- 172.21.15.47:0/23986391 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7fea94055a00 con 0x7fea9c0ceb40 2024-02-19T15:22:31.352 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:31.352 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":21,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:06.843155+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:03.091360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:04.222965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:31.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.349+0000 7feaa4000640 1 -- 172.21.15.47:0/23986391 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fea7c066e40 msgr2=0x7fea7c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:31.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.349+0000 7feaa4000640 1 --2- 172.21.15.47:0/23986391 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fea7c066e40 0x7fea7c069300 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fea90000f70 tx=0x7fea90007df0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.349+0000 7feaa4000640 1 -- 172.21.15.47:0/23986391 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea9c0ceb40 msgr2=0x7fea9c1600a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:31.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.349+0000 7feaa4000640 1 --2- 172.21.15.47:0/23986391 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea9c0ceb40 0x7fea9c1600a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fea9400efc0 tx=0x7fea9400c490 comp rx=0 tx=0).stop 2024-02-19T15:22:31.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.349+0000 7feaa4000640 1 -- 172.21.15.47:0/23986391 shutdown_connections 2024-02-19T15:22:31.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.349+0000 7feaa4000640 1 --2- 172.21.15.47:0/23986391 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fea7c066e40 0x7fea7c069300 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.349+0000 7feaa4000640 1 --2- 172.21.15.47:0/23986391 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea9c0ceb40 0x7fea9c1600a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.349+0000 7feaa4000640 1 --2- 172.21.15.47:0/23986391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea9c0cded0 0x7fea9c0b51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.349+0000 7feaa4000640 1 --2- 172.21.15.47:0/23986391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea9c06d650 0x7fea9c0ca8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.349+0000 7feaa4000640 1 -- 172.21.15.47:0/23986391 >> 172.21.15.47:0/23986391 conn(0x7fea9c0d1bf0 msgr2=0x7fea9c0d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:31.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.349+0000 7feaa4000640 1 -- 172.21.15.47:0/23986391 shutdown_connections 2024-02-19T15:22:31.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.349+0000 7feaa4000640 1 -- 172.21.15.47:0/23986391 wait complete. 2024-02-19T15:22:31.373 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:22:03.091360+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:22:04.222965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:22:31.373 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:22:31.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.513+0000 7fe4aaa00640 1 Processor -- start 2024-02-19T15:22:31.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.513+0000 7fe4aaa00640 1 -- start start 2024-02-19T15:22:31.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.513+0000 7fe4aaa00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe4a415e160 0x7fe4a40d1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:31.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.513+0000 7fe4aaa00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4a406f220 0x7fe4a406dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:31.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.513+0000 7fe4aaa00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe4a406e010 0x7fe4a406aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:31.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.513+0000 7fe4aaa00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a4071a70 con 0x7fe4a415e160 2024-02-19T15:22:31.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.513+0000 7fe4aaa00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a406e460 con 0x7fe4a406e010 2024-02-19T15:22:31.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.513+0000 7fe4aaa00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a406f680 con 0x7fe4a406f220 2024-02-19T15:22:31.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.513+0000 7fe4a3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe4a415e160 0x7fe4a40d1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:31.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.513+0000 7fe4a3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe4a415e160 0x7fe4a40d1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60704/0 (socket says 172.21.15.47:60704) 2024-02-19T15:22:31.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.513+0000 7fe4a3e00640 1 -- 172.21.15.47:0/1171976863 learned_addr learned my addr 172.21.15.47:0/1171976863 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:31.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.513+0000 7fe4a8a00640 1 --2- 172.21.15.47:0/1171976863 >> v2:172.21.15.116:3300/0 conn(0x7fe4a406e010 0x7fe4a406aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:31.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.513+0000 7fe49b400640 1 --2- 172.21.15.47:0/1171976863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4a406f220 0x7fe4a406dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:31.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.513+0000 7fe4a3e00640 1 -- 172.21.15.47:0/1171976863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4a406f220 msgr2=0x7fe4a406dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:31.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.513+0000 7fe4a3e00640 1 --2- 172.21.15.47:0/1171976863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4a406f220 0x7fe4a406dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.513+0000 7fe4a3e00640 1 -- 172.21.15.47:0/1171976863 >> v2:172.21.15.116:3300/0 conn(0x7fe4a406e010 msgr2=0x7fe4a406aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:31.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.513+0000 7fe4a3e00640 1 --2- 172.21.15.47:0/1171976863 >> v2:172.21.15.116:3300/0 conn(0x7fe4a406e010 0x7fe4a406aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.513+0000 7fe4a3e00640 1 -- 172.21.15.47:0/1171976863 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a40ced30 con 0x7fe4a415e160 2024-02-19T15:22:31.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.513+0000 7fe4a8a00640 1 --2- 172.21.15.47:0/1171976863 >> v2:172.21.15.116:3300/0 conn(0x7fe4a406e010 0x7fe4a406aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:31.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.517+0000 7fe4a3e00640 1 --2- 172.21.15.47:0/1171976863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe4a415e160 0x7fe4a40d1820 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40b05c0 tx=0x7fe49401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f80663f0fad14b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:31.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.517+0000 7fe4a3400640 1 -- 172.21.15.47:0/1171976863 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4940a5850 con 0x7fe4a415e160 2024-02-19T15:22:31.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.517+0000 7fe4a3400640 1 -- 172.21.15.47:0/1171976863 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4940a59b0 con 0x7fe4a415e160 2024-02-19T15:22:31.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.517+0000 7fe4a3400640 1 -- 172.21.15.47:0/1171976863 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4940a5d40 con 0x7fe4a415e160 2024-02-19T15:22:31.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.517+0000 7fe4aaa00640 1 -- 172.21.15.47:0/1171976863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe4a415e160 msgr2=0x7fe4a40d1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:31.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.517+0000 7fe4aaa00640 1 --2- 172.21.15.47:0/1171976863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe4a415e160 0x7fe4a40d1820 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40b05c0 tx=0x7fe49401c9d0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.517+0000 7fe4aaa00640 1 -- 172.21.15.47:0/1171976863 shutdown_connections 2024-02-19T15:22:31.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.517+0000 7fe4aaa00640 1 --2- 172.21.15.47:0/1171976863 >> v2:172.21.15.116:3300/0 conn(0x7fe4a406e010 0x7fe4a406aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.517+0000 7fe4aaa00640 1 --2- 172.21.15.47:0/1171976863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4a406f220 0x7fe4a406dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.517+0000 7fe4aaa00640 1 --2- 172.21.15.47:0/1171976863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe4a415e160 0x7fe4a40d1820 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.517+0000 7fe4aaa00640 1 -- 172.21.15.47:0/1171976863 >> 172.21.15.47:0/1171976863 conn(0x7fe4a40d0d30 msgr2=0x7fe4a40d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:31.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.517+0000 7fe4aaa00640 1 -- 172.21.15.47:0/1171976863 shutdown_connections 2024-02-19T15:22:31.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.517+0000 7fe4aaa00640 1 -- 172.21.15.47:0/1171976863 wait complete. 2024-02-19T15:22:31.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe4aaa00640 1 Processor -- start 2024-02-19T15:22:31.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe4aaa00640 1 -- start start 2024-02-19T15:22:31.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe4aaa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4a406e010 0x7fe4a414cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:31.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe4aaa00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4a406f220 0x7fe4a40b0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:31.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe4aaa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe4a415e160 0x7fe4a40af930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:31.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe4aaa00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a40b36d0 con 0x7fe4a415e160 2024-02-19T15:22:31.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe4aaa00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a414ca30 con 0x7fe4a406e010 2024-02-19T15:22:31.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe4aaa00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a414c100 con 0x7fe4a406f220 2024-02-19T15:22:31.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe4a3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4a406e010 0x7fe4a414cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:31.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe4a8a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe4a415e160 0x7fe4a40af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:31.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe4a3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4a406e010 0x7fe4a414cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:33406/0 (socket says 172.21.15.47:33406) 2024-02-19T15:22:31.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe4a3e00640 1 -- 172.21.15.47:0/1944381839 learned_addr learned my addr 172.21.15.47:0/1944381839 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:31.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe49b400640 1 --2- 172.21.15.47:0/1944381839 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4a406f220 0x7fe4a40b0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:31.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe49b400640 1 -- 172.21.15.47:0/1944381839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4a406e010 msgr2=0x7fe4a414cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:31.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe49b400640 1 --2- 172.21.15.47:0/1944381839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4a406e010 0x7fe4a414cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe49b400640 1 -- 172.21.15.47:0/1944381839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe4a415e160 msgr2=0x7fe4a40af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:31.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe49b400640 1 --2- 172.21.15.47:0/1944381839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe4a415e160 0x7fe4a40af930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe49b400640 1 -- 172.21.15.47:0/1944381839 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe494009b30 con 0x7fe4a406f220 2024-02-19T15:22:31.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe4a8a00640 1 --2- 172.21.15.47:0/1944381839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe4a415e160 0x7fe4a40af930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:31.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe4a3e00640 1 --2- 172.21.15.47:0/1944381839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4a406e010 0x7fe4a414cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:31.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe49b400640 1 --2- 172.21.15.47:0/1944381839 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4a406f220 0x7fe4a40b0d40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe48c00e8a0 tx=0x7fe48c00eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:31.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe4a0c00640 1 -- 172.21.15.47:0/1944381839 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c0095d0 con 0x7fe4a406f220 2024-02-19T15:22:31.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe4a0c00640 1 -- 172.21.15.47:0/1944381839 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48c00c940 con 0x7fe4a406f220 2024-02-19T15:22:31.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe4a0c00640 1 -- 172.21.15.47:0/1944381839 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c00ccb0 con 0x7fe4a406f220 2024-02-19T15:22:31.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe4aaa00640 1 -- 172.21.15.47:0/1944381839 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a414c3e0 con 0x7fe4a406f220 2024-02-19T15:22:31.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.521+0000 7fe4aaa00640 1 -- 172.21.15.47:0/1944381839 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a40b1280 con 0x7fe4a406f220 2024-02-19T15:22:31.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.525+0000 7fe4aaa00640 1 -- 172.21.15.47:0/1944381839 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a40b1500 con 0x7fe4a406f220 2024-02-19T15:22:31.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.525+0000 7fe4a0c00640 1 -- 172.21.15.47:0/1944381839 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe48c0059a0 con 0x7fe4a406f220 2024-02-19T15:22:31.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.525+0000 7fe4a0c00640 1 --2- 172.21.15.47:0/1944381839 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe480066e40 0x7fe480069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:31.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.525+0000 7fe4a0c00640 1 -- 172.21.15.47:0/1944381839 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe48c084930 con 0x7fe4a406f220 2024-02-19T15:22:31.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.525+0000 7fe4a3e00640 1 --2- 172.21.15.47:0/1944381839 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe480066e40 0x7fe480069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:31.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.525+0000 7fe4a3e00640 1 --2- 172.21.15.47:0/1944381839 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe480066e40 0x7fe480069300 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe494000f90 tx=0x7fe4940a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:31.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.529+0000 7fe4a0c00640 1 -- 172.21.15.47:0/1944381839 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe48c0517b0 con 0x7fe4a406f220 2024-02-19T15:22:31.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.693+0000 7fe4aaa00640 1 -- 172.21.15.47:0/1944381839 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe4a40b1500 con 0x7fe4a406f220 2024-02-19T15:22:31.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.693+0000 7fe4a0c00640 1 -- 172.21.15.47:0/1944381839 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe48c021020 con 0x7fe4a406f220 2024-02-19T15:22:31.695 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:22:31.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.693+0000 7fe4aaa00640 1 -- 172.21.15.47:0/1944381839 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe480066e40 msgr2=0x7fe480069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:31.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.693+0000 7fe4aaa00640 1 --2- 172.21.15.47:0/1944381839 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe480066e40 0x7fe480069300 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe494000f90 tx=0x7fe4940a7070 comp rx=0 tx=0).stop 2024-02-19T15:22:31.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.693+0000 7fe4aaa00640 1 -- 172.21.15.47:0/1944381839 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4a406f220 msgr2=0x7fe4a40b0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:31.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.693+0000 7fe4aaa00640 1 --2- 172.21.15.47:0/1944381839 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4a406f220 0x7fe4a40b0d40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe48c00e8a0 tx=0x7fe48c00eca0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.693+0000 7fe4aaa00640 1 -- 172.21.15.47:0/1944381839 shutdown_connections 2024-02-19T15:22:31.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.693+0000 7fe4aaa00640 1 --2- 172.21.15.47:0/1944381839 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe480066e40 0x7fe480069300 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.693+0000 7fe4aaa00640 1 --2- 172.21.15.47:0/1944381839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe4a415e160 0x7fe4a40af930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.693+0000 7fe4aaa00640 1 --2- 172.21.15.47:0/1944381839 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4a406f220 0x7fe4a40b0d40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.693+0000 7fe4aaa00640 1 --2- 172.21.15.47:0/1944381839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4a406e010 0x7fe4a414cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.693+0000 7fe4aaa00640 1 -- 172.21.15.47:0/1944381839 >> 172.21.15.47:0/1944381839 conn(0x7fe4a40d0d30 msgr2=0x7fe4a4160ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:31.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.693+0000 7fe4aaa00640 1 -- 172.21.15.47:0/1944381839 shutdown_connections 2024-02-19T15:22:31.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.693+0000 7fe4aaa00640 1 -- 172.21.15.47:0/1944381839 wait complete. 2024-02-19T15:22:31.716 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:22:31.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.853+0000 7f7c20200640 1 Processor -- start 2024-02-19T15:22:31.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.857+0000 7f7c20200640 1 -- start start 2024-02-19T15:22:31.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.857+0000 7f7c20200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c1806e360 0x7f7c180cb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:31.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.857+0000 7f7c20200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7c180cd880 0x7f7c180cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:31.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.857+0000 7f7c20200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c180cb540 con 0x7f7c1815af10 2024-02-19T15:22:31.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.857+0000 7f7c20200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c18071c70 con 0x7f7c180cd880 2024-02-19T15:22:31.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.857+0000 7f7c20200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1806cef0 con 0x7f7c1806e360 2024-02-19T15:22:31.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.857+0000 7f7c1ce00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c1806e360 0x7f7c180cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:31.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.857+0000 7f7c1ce00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c1806e360 0x7f7c180cb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:32830/0 (socket says 172.21.15.47:32830) 2024-02-19T15:22:31.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.857+0000 7f7c1ce00640 1 -- 172.21.15.47:0/2647304788 learned_addr learned my addr 172.21.15.47:0/2647304788 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:31.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.857+0000 7f7c1e200640 1 --2- 172.21.15.47:0/2647304788 >> v2:172.21.15.116:3300/0 conn(0x7f7c180cd880 0x7f7c180cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:31.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.857+0000 7f7c1ce00640 1 -- 172.21.15.47:0/2647304788 >> v2:172.21.15.116:3300/0 conn(0x7f7c180cd880 msgr2=0x7f7c180cdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:31.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.857+0000 7f7c1ce00640 1 --2- 172.21.15.47:0/2647304788 >> v2:172.21.15.116:3300/0 conn(0x7f7c180cd880 0x7f7c180cdd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.857+0000 7f7c1ce00640 1 -- 172.21.15.47:0/2647304788 >> v1:172.21.15.116:6789/0 conn(0x7f7c1815af10 legacy=0x7f7c180ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:31.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.857+0000 7f7c1ce00640 1 -- 172.21.15.47:0/2647304788 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c180ca610 con 0x7f7c1806e360 2024-02-19T15:22:31.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.857+0000 7f7c1e200640 1 --2- 172.21.15.47:0/2647304788 >> v2:172.21.15.116:3300/0 conn(0x7f7c180cd880 0x7f7c180cdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:31.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.861+0000 7f7c1ce00640 1 --2- 172.21.15.47:0/2647304788 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c1806e360 0x7f7c180cb000 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400bb10 tx=0x7f7c0401eb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c787b1ac1d2739a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:31.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.861+0000 7f7c17e00640 1 -- 172.21.15.47:0/2647304788 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c040a7c40 con 0x7f7c1806e360 2024-02-19T15:22:31.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.861+0000 7f7c17e00640 1 -- 172.21.15.47:0/2647304788 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c040a7da0 con 0x7f7c1806e360 2024-02-19T15:22:31.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.861+0000 7f7c17e00640 1 -- 172.21.15.47:0/2647304788 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c040af660 con 0x7f7c1806e360 2024-02-19T15:22:31.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.861+0000 7f7c20200640 1 -- 172.21.15.47:0/2647304788 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c1806e360 msgr2=0x7f7c180cb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:31.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.861+0000 7f7c20200640 1 --2- 172.21.15.47:0/2647304788 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c1806e360 0x7f7c180cb000 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400bb10 tx=0x7f7c0401eb70 comp rx=0 tx=0).stop 2024-02-19T15:22:31.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.861+0000 7f7c20200640 1 -- 172.21.15.47:0/2647304788 shutdown_connections 2024-02-19T15:22:31.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.861+0000 7f7c20200640 1 --2- 172.21.15.47:0/2647304788 >> v2:172.21.15.116:3300/0 conn(0x7f7c180cd880 0x7f7c180cdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.861+0000 7f7c20200640 1 --2- 172.21.15.47:0/2647304788 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c1806e360 0x7f7c180cb000 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.861+0000 7f7c20200640 1 -- 172.21.15.47:0/2647304788 >> 172.21.15.47:0/2647304788 conn(0x7f7c180cfc20 msgr2=0x7f7c180d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:31.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.861+0000 7f7c20200640 1 -- 172.21.15.47:0/2647304788 shutdown_connections 2024-02-19T15:22:31.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.861+0000 7f7c20200640 1 -- 172.21.15.47:0/2647304788 wait complete. 2024-02-19T15:22:31.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.861+0000 7f7c20200640 1 Processor -- start 2024-02-19T15:22:31.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.861+0000 7f7c20200640 1 -- start start 2024-02-19T15:22:31.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c20200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c1806e360 0x7f7c180b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:31.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c20200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c180cd880 0x7f7c18154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:31.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c20200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c1815af10 0x7f7c1814fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:31.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c20200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c18155f40 con 0x7f7c1815af10 2024-02-19T15:22:31.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c20200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c18154ff0 con 0x7f7c180cd880 2024-02-19T15:22:31.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c20200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c181552c0 con 0x7f7c1806e360 2024-02-19T15:22:31.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c1e200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c1815af10 0x7f7c1814fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:31.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c1e200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c1815af10 0x7f7c1814fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60726/0 (socket says 172.21.15.47:60726) 2024-02-19T15:22:31.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c1e200640 1 -- 172.21.15.47:0/4238135410 learned_addr learned my addr 172.21.15.47:0/4238135410 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:31.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c1ce00640 1 --2- 172.21.15.47:0/4238135410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c180cd880 0x7f7c18154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:31.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c1d800640 1 --2- 172.21.15.47:0/4238135410 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c1806e360 0x7f7c180b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:31.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c1e200640 1 -- 172.21.15.47:0/4238135410 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c1806e360 msgr2=0x7f7c180b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:31.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c1e200640 1 --2- 172.21.15.47:0/4238135410 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c1806e360 0x7f7c180b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c1e200640 1 -- 172.21.15.47:0/4238135410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c180cd880 msgr2=0x7f7c18154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:31.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c1e200640 1 --2- 172.21.15.47:0/4238135410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c180cd880 0x7f7c18154ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:31.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c1e200640 1 -- 172.21.15.47:0/4238135410 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c10009a70 con 0x7f7c1815af10 2024-02-19T15:22:31.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c1ce00640 1 --2- 172.21.15.47:0/4238135410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c180cd880 0x7f7c18154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:31.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c1e200640 1 --2- 172.21.15.47:0/4238135410 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c1815af10 0x7f7c1814fa10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7c1000efc0 tx=0x7f7c10009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:31.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c15600640 1 -- 172.21.15.47:0/4238135410 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c10009d70 con 0x7f7c1815af10 2024-02-19T15:22:31.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c15600640 1 -- 172.21.15.47:0/4238135410 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1000f040 con 0x7f7c1815af10 2024-02-19T15:22:31.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c15600640 1 -- 172.21.15.47:0/4238135410 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c100039a0 con 0x7f7c1815af10 2024-02-19T15:22:31.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c20200640 1 -- 172.21.15.47:0/4238135410 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c0400b770 con 0x7f7c1815af10 2024-02-19T15:22:31.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c20200640 1 -- 172.21.15.47:0/4238135410 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c18155880 con 0x7f7c1815af10 2024-02-19T15:22:31.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c20200640 1 -- 172.21.15.47:0/4238135410 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c18156080 con 0x7f7c1815af10 2024-02-19T15:22:31.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.865+0000 7f7c15600640 1 -- 172.21.15.47:0/4238135410 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7c100144a0 con 0x7f7c1815af10 2024-02-19T15:22:31.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.869+0000 7f7c15600640 1 --2- 172.21.15.47:0/4238135410 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7bf8066e40 0x7f7bf8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:31.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.869+0000 7f7c15600640 1 -- 172.21.15.47:0/4238135410 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c100855a0 con 0x7f7c1815af10 2024-02-19T15:22:31.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.869+0000 7f7c1d800640 1 --2- 172.21.15.47:0/4238135410 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7bf8066e40 0x7f7bf8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:31.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.869+0000 7f7c1d800640 1 --2- 172.21.15.47:0/4238135410 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7bf8066e40 0x7f7bf8069300 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7c18069270 tx=0x7f7c0c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:31.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:31.873+0000 7f7c15600640 1 -- 172.21.15.47:0/4238135410 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7c18156080 con 0x7f7c1815af10 2024-02-19T15:22:32.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.041+0000 7f7c20200640 1 -- 172.21.15.47:0/4238135410 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7c18156080 con 0x7f7c1815af10 2024-02-19T15:22:32.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.041+0000 7f7c15600640 1 -- 172.21.15.47:0/4238135410 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7c18156080 con 0x7f7c1815af10 2024-02-19T15:22:32.043 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:22:32.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.041+0000 7f7c20200640 1 -- 172.21.15.47:0/4238135410 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7bf8066e40 msgr2=0x7f7bf8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:32.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.041+0000 7f7c20200640 1 --2- 172.21.15.47:0/4238135410 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7bf8066e40 0x7f7bf8069300 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7c18069270 tx=0x7f7c0c009040 comp rx=0 tx=0).stop 2024-02-19T15:22:32.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.041+0000 7f7c20200640 1 -- 172.21.15.47:0/4238135410 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c1815af10 msgr2=0x7f7c1814fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:32.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.041+0000 7f7c20200640 1 --2- 172.21.15.47:0/4238135410 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c1815af10 0x7f7c1814fa10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7c1000efc0 tx=0x7f7c10009f90 comp rx=0 tx=0).stop 2024-02-19T15:22:32.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.041+0000 7f7c20200640 1 -- 172.21.15.47:0/4238135410 shutdown_connections 2024-02-19T15:22:32.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.041+0000 7f7c20200640 1 --2- 172.21.15.47:0/4238135410 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7bf8066e40 0x7f7bf8069300 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.041+0000 7f7c20200640 1 --2- 172.21.15.47:0/4238135410 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c1815af10 0x7f7c1814fa10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.041+0000 7f7c20200640 1 --2- 172.21.15.47:0/4238135410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c180cd880 0x7f7c18154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.041+0000 7f7c20200640 1 --2- 172.21.15.47:0/4238135410 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c1806e360 0x7f7c180b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.041+0000 7f7c20200640 1 -- 172.21.15.47:0/4238135410 >> 172.21.15.47:0/4238135410 conn(0x7f7c180cfc20 msgr2=0x7f7c180d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:32.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.041+0000 7f7c20200640 1 -- 172.21.15.47:0/4238135410 shutdown_connections 2024-02-19T15:22:32.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.041+0000 7f7c20200640 1 -- 172.21.15.47:0/4238135410 wait complete. 2024-02-19T15:22:32.064 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:22:32.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.201+0000 7fc370400640 1 Processor -- start 2024-02-19T15:22:32.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.201+0000 7fc370400640 1 -- start start 2024-02-19T15:22:32.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.205+0000 7fc370400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc36806e1e0 0x7fc3680d1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:32.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.205+0000 7fc370400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc368071a70 con 0x7fc36806f220 2024-02-19T15:22:32.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.205+0000 7fc370400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc36806f690 con 0x7fc36815e160 2024-02-19T15:22:32.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.205+0000 7fc370400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3680ced30 con 0x7fc36806e1e0 2024-02-19T15:22:32.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.205+0000 7fc36e400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc36806f220 0x7fc36815e540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34136/0 (socket says 172.21.15.47:34136) 2024-02-19T15:22:32.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.205+0000 7fc36e400640 1 -- 172.21.15.47:0/2341669448 learned_addr learned my addr 172.21.15.47:0/2341669448 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:32.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.205+0000 7fc36d000640 1 --2- 172.21.15.47:0/2341669448 >> v2:172.21.15.116:3300/0 conn(0x7fc36806e1e0 0x7fc3680d1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:32.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.205+0000 7fc367e00640 1 -- 172.21.15.47:0/2341669448 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982586094 0 0) 0x7fc368071a70 con 0x7fc36806f220 2024-02-19T15:22:32.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.205+0000 7fc367e00640 1 -- 172.21.15.47:0/2341669448 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc350003530 con 0x7fc36806f220 2024-02-19T15:22:32.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.205+0000 7fc367e00640 1 -- 172.21.15.47:0/2341669448 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165615163 0 0) 0x7fc36806f690 con 0x7fc36815e160 2024-02-19T15:22:32.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.205+0000 7fc36d000640 1 -- 172.21.15.47:0/2341669448 >> v1:172.21.15.116:6789/0 conn(0x7fc36815e160 legacy=0x7fc36806dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:32.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.205+0000 7fc36d000640 1 -- 172.21.15.47:0/2341669448 >> v1:172.21.15.47:6789/0 conn(0x7fc36806f220 legacy=0x7fc36815e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:32.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.205+0000 7fc36d000640 1 -- 172.21.15.47:0/2341669448 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc368162a00 con 0x7fc36806e1e0 2024-02-19T15:22:32.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.209+0000 7fc36d000640 1 --2- 172.21.15.47:0/2341669448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc36806e1e0 0x7fc3680d1820 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc3540097c0 tx=0x7fc35401c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10e450f4376c9582 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:32.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.209+0000 7fc367e00640 1 -- 172.21.15.47:0/2341669448 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3540ab670 con 0x7fc36806e1e0 2024-02-19T15:22:32.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.209+0000 7fc367e00640 1 -- 172.21.15.47:0/2341669448 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3540ab7d0 con 0x7fc36806e1e0 2024-02-19T15:22:32.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.209+0000 7fc367e00640 1 -- 172.21.15.47:0/2341669448 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3540abb60 con 0x7fc36806e1e0 2024-02-19T15:22:32.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.209+0000 7fc370400640 1 -- 172.21.15.47:0/2341669448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc36806e1e0 msgr2=0x7fc3680d1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:32.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.209+0000 7fc370400640 1 --2- 172.21.15.47:0/2341669448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc36806e1e0 0x7fc3680d1820 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc3540097c0 tx=0x7fc35401c470 comp rx=0 tx=0).stop 2024-02-19T15:22:32.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.209+0000 7fc370400640 1 -- 172.21.15.47:0/2341669448 shutdown_connections 2024-02-19T15:22:32.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.209+0000 7fc370400640 1 --2- 172.21.15.47:0/2341669448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc36806e1e0 0x7fc3680d1820 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.209+0000 7fc370400640 1 -- 172.21.15.47:0/2341669448 >> 172.21.15.47:0/2341669448 conn(0x7fc3680d0d30 msgr2=0x7fc3680d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:32.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.209+0000 7fc370400640 1 -- 172.21.15.47:0/2341669448 shutdown_connections 2024-02-19T15:22:32.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.209+0000 7fc370400640 1 -- 172.21.15.47:0/2341669448 wait complete. 2024-02-19T15:22:32.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.209+0000 7fc370400640 1 Processor -- start 2024-02-19T15:22:32.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.209+0000 7fc370400640 1 -- start start 2024-02-19T15:22:32.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc370400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc36806e1e0 0x7fc36815ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:32.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc370400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc36806f220 0x7fc3680b0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:32.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc370400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc36815e160 0x7fc3680b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:32.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc370400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3680b1ef0 con 0x7fc36806e1e0 2024-02-19T15:22:32.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc370400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3680b2c40 con 0x7fc36806f220 2024-02-19T15:22:32.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc370400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3680b1370 con 0x7fc36815e160 2024-02-19T15:22:32.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc36da00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc36806e1e0 0x7fc36815ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:32.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc36da00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc36806e1e0 0x7fc36815ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60742/0 (socket says 172.21.15.47:60742) 2024-02-19T15:22:32.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc36da00640 1 -- 172.21.15.47:0/947187072 learned_addr learned my addr 172.21.15.47:0/947187072 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:32.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc36d000640 1 --2- 172.21.15.47:0/947187072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc36806f220 0x7fc3680b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:32.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc36e400640 1 --2- 172.21.15.47:0/947187072 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc36815e160 0x7fc3680b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:32.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc36da00640 1 -- 172.21.15.47:0/947187072 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc36815e160 msgr2=0x7fc3680b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:32.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc36da00640 1 --2- 172.21.15.47:0/947187072 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc36815e160 0x7fc3680b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc36da00640 1 -- 172.21.15.47:0/947187072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc36806f220 msgr2=0x7fc3680b0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:32.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc36da00640 1 --2- 172.21.15.47:0/947187072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc36806f220 0x7fc3680b0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc36da00640 1 -- 172.21.15.47:0/947187072 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc354009420 con 0x7fc36806e1e0 2024-02-19T15:22:32.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc36d000640 1 --2- 172.21.15.47:0/947187072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc36806f220 0x7fc3680b0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:32.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc36da00640 1 --2- 172.21.15.47:0/947187072 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc36806e1e0 0x7fc36815ff90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc35c001ec0 tx=0x7fc35c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:32.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc365600640 1 -- 172.21.15.47:0/947187072 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c0054e0 con 0x7fc36806e1e0 2024-02-19T15:22:32.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc365600640 1 -- 172.21.15.47:0/947187072 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35c003600 con 0x7fc36806e1e0 2024-02-19T15:22:32.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc365600640 1 -- 172.21.15.47:0/947187072 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c0039b0 con 0x7fc36806e1e0 2024-02-19T15:22:32.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc370400640 1 -- 172.21.15.47:0/947187072 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3680b1650 con 0x7fc36806e1e0 2024-02-19T15:22:32.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc370400640 1 -- 172.21.15.47:0/947187072 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3680b1c10 con 0x7fc36806e1e0 2024-02-19T15:22:32.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc370400640 1 -- 172.21.15.47:0/947187072 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc368075360 con 0x7fc36806e1e0 2024-02-19T15:22:32.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.213+0000 7fc365600640 1 -- 172.21.15.47:0/947187072 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc35c003b10 con 0x7fc36806e1e0 2024-02-19T15:22:32.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.217+0000 7fc365600640 1 --2- 172.21.15.47:0/947187072 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc348066e40 0x7fc348069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:32.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.217+0000 7fc365600640 1 -- 172.21.15.47:0/947187072 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc35c083fa0 con 0x7fc36806e1e0 2024-02-19T15:22:32.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.217+0000 7fc36d000640 1 --2- 172.21.15.47:0/947187072 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc348066e40 0x7fc348069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:32.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.217+0000 7fc36d000640 1 --2- 172.21.15.47:0/947187072 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc348066e40 0x7fc348069300 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc3540af000 tx=0x7fc3540a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:32.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.221+0000 7fc365600640 1 -- 172.21.15.47:0/947187072 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc35c050e30 con 0x7fc36806e1e0 2024-02-19T15:22:32.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.381+0000 7fc370400640 1 -- 172.21.15.47:0/947187072 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc3680c93b0 con 0x7fc36806e1e0 2024-02-19T15:22:32.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.381+0000 7fc365600640 1 -- 172.21.15.47:0/947187072 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc35c001130 con 0x7fc36806e1e0 2024-02-19T15:22:32.385 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:22:32.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.385+0000 7fc370400640 1 -- 172.21.15.47:0/947187072 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc348066e40 msgr2=0x7fc348069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:32.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.385+0000 7fc370400640 1 --2- 172.21.15.47:0/947187072 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc348066e40 0x7fc348069300 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc3540af000 tx=0x7fc3540a7000 comp rx=0 tx=0).stop 2024-02-19T15:22:32.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.385+0000 7fc370400640 1 -- 172.21.15.47:0/947187072 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc36806e1e0 msgr2=0x7fc36815ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:32.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.385+0000 7fc370400640 1 --2- 172.21.15.47:0/947187072 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc36806e1e0 0x7fc36815ff90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc35c001ec0 tx=0x7fc35c00c8e0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.385+0000 7fc370400640 1 -- 172.21.15.47:0/947187072 shutdown_connections 2024-02-19T15:22:32.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.385+0000 7fc370400640 1 --2- 172.21.15.47:0/947187072 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc348066e40 0x7fc348069300 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.385+0000 7fc370400640 1 --2- 172.21.15.47:0/947187072 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc36815e160 0x7fc3680b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.385+0000 7fc370400640 1 --2- 172.21.15.47:0/947187072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc36806f220 0x7fc3680b0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.385+0000 7fc370400640 1 --2- 172.21.15.47:0/947187072 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc36806e1e0 0x7fc36815ff90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.385+0000 7fc370400640 1 -- 172.21.15.47:0/947187072 >> 172.21.15.47:0/947187072 conn(0x7fc3680d0d30 msgr2=0x7fc36815d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:32.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.385+0000 7fc370400640 1 -- 172.21.15.47:0/947187072 shutdown_connections 2024-02-19T15:22:32.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.385+0000 7fc370400640 1 -- 172.21.15.47:0/947187072 wait complete. 2024-02-19T15:22:32.406 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:32.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.541+0000 7f62e3e00640 1 Processor -- start 2024-02-19T15:22:32.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.545+0000 7f62e3e00640 1 -- start start 2024-02-19T15:22:32.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.545+0000 7f62e3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f62e415f180 0x7f62e40cca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:32.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.545+0000 7f62e3e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62e4067100 0x7f62e40ccf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:32.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.545+0000 7f62e3e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e406bf20 con 0x7f62e40ae2f0 2024-02-19T15:22:32.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.545+0000 7f62e3e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62e406c520 con 0x7f62e415f180 2024-02-19T15:22:32.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.545+0000 7f62e3e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62e40cfd80 con 0x7f62e4067100 2024-02-19T15:22:32.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.545+0000 7f62e2a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f62e415f180 0x7f62e40cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:32.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.545+0000 7f62e2a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f62e415f180 0x7f62e40cca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60758/0 (socket says 172.21.15.47:60758) 2024-02-19T15:22:32.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.545+0000 7f62e2a00640 1 -- 172.21.15.47:0/2174268717 learned_addr learned my addr 172.21.15.47:0/2174268717 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:32.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.545+0000 7f62dbe00640 1 --2- 172.21.15.47:0/2174268717 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62e4067100 0x7f62e40ccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:32.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.545+0000 7f62e2a00640 1 -- 172.21.15.47:0/2174268717 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62e4067100 msgr2=0x7f62e40ccf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:32.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.545+0000 7f62e2a00640 1 --2- 172.21.15.47:0/2174268717 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62e4067100 0x7f62e40ccf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.545+0000 7f62e2a00640 1 -- 172.21.15.47:0/2174268717 >> v1:172.21.15.116:6789/0 conn(0x7f62e40ae2f0 legacy=0x7f62e415f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:32.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.545+0000 7f62e2000640 1 -- 172.21.15.47:0/2174268717 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563139412 0 0) 0x7f62e406bf20 con 0x7f62e40ae2f0 2024-02-19T15:22:32.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.545+0000 7f62e2a00640 1 -- 172.21.15.47:0/2174268717 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e406c7c0 con 0x7f62e415f180 2024-02-19T15:22:32.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.549+0000 7f62e2a00640 1 --2- 172.21.15.47:0/2174268717 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62e415f180 0x7f62e40cca50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009870 tx=0x7f62d401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=375a91ded8d4038a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:32.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.549+0000 7f62e2000640 1 -- 172.21.15.47:0/2174268717 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d40ab600 con 0x7f62e415f180 2024-02-19T15:22:32.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.549+0000 7f62e2000640 1 -- 172.21.15.47:0/2174268717 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d40a7050 con 0x7f62e415f180 2024-02-19T15:22:32.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.549+0000 7f62e2000640 1 -- 172.21.15.47:0/2174268717 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d40ab980 con 0x7f62e415f180 2024-02-19T15:22:32.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.549+0000 7f62e3e00640 1 -- 172.21.15.47:0/2174268717 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62e415f180 msgr2=0x7f62e40cca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:32.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.549+0000 7f62e3e00640 1 --2- 172.21.15.47:0/2174268717 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62e415f180 0x7f62e40cca50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009870 tx=0x7f62d401c4d0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.549+0000 7f62e3e00640 1 -- 172.21.15.47:0/2174268717 shutdown_connections 2024-02-19T15:22:32.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.549+0000 7f62e3e00640 1 --2- 172.21.15.47:0/2174268717 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62e4067100 0x7f62e40ccf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.549+0000 7f62e3e00640 1 --2- 172.21.15.47:0/2174268717 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62e415f180 0x7f62e40cca50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.549+0000 7f62e3e00640 1 -- 172.21.15.47:0/2174268717 >> 172.21.15.47:0/2174268717 conn(0x7f62e40b1be0 msgr2=0x7f62e40b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:32.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.549+0000 7f62e3e00640 1 -- 172.21.15.47:0/2174268717 shutdown_connections 2024-02-19T15:22:32.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.549+0000 7f62e3e00640 1 -- 172.21.15.47:0/2174268717 wait complete. 2024-02-19T15:22:32.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.549+0000 7f62e3e00640 1 Processor -- start 2024-02-19T15:22:32.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.549+0000 7f62e3e00640 1 -- start start 2024-02-19T15:22:32.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62e3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62e4067100 0x7f62e4071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:32.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62e3e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62e40ae2f0 0x7f62e409d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:32.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62e3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62e415f180 0x7f62e409b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:32.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62e3e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e40723e0 con 0x7f62e415f180 2024-02-19T15:22:32.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62e3e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e4072610 con 0x7f62e4067100 2024-02-19T15:22:32.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62e3e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62e40728f0 con 0x7f62e40ae2f0 2024-02-19T15:22:32.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62dbe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62e40ae2f0 0x7f62e409d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:32.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62e3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62e415f180 0x7f62e409b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:32.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62e2a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62e4067100 0x7f62e4071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:32.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62dbe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62e40ae2f0 0x7f62e409d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:32872/0 (socket says 172.21.15.47:32872) 2024-02-19T15:22:32.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62dbe00640 1 -- 172.21.15.47:0/4013415028 learned_addr learned my addr 172.21.15.47:0/4013415028 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:32.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62dbe00640 1 -- 172.21.15.47:0/4013415028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62e4067100 msgr2=0x7f62e4071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:32.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62dbe00640 1 --2- 172.21.15.47:0/4013415028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62e4067100 0x7f62e4071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62dbe00640 1 -- 172.21.15.47:0/4013415028 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62e415f180 msgr2=0x7f62e409b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:32.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62dbe00640 1 --2- 172.21.15.47:0/4013415028 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62e415f180 0x7f62e409b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62dbe00640 1 -- 172.21.15.47:0/4013415028 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d40094d0 con 0x7f62e40ae2f0 2024-02-19T15:22:32.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62e3400640 1 --2- 172.21.15.47:0/4013415028 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62e415f180 0x7f62e409b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:32.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62dbe00640 1 --2- 172.21.15.47:0/4013415028 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62e40ae2f0 0x7f62e409d810 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00e840 tx=0x7f62cc00ec40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:32.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62daa00640 1 -- 172.21.15.47:0/4013415028 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc009510 con 0x7f62e40ae2f0 2024-02-19T15:22:32.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62daa00640 1 -- 172.21.15.47:0/4013415028 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62cc00ee90 con 0x7f62e40ae2f0 2024-02-19T15:22:32.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62e3e00640 1 -- 172.21.15.47:0/4013415028 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e40c3ba0 con 0x7f62e40ae2f0 2024-02-19T15:22:32.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62daa00640 1 -- 172.21.15.47:0/4013415028 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc00cb80 con 0x7f62e40ae2f0 2024-02-19T15:22:32.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62e3e00640 1 -- 172.21.15.47:0/4013415028 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e40c4160 con 0x7f62e40ae2f0 2024-02-19T15:22:32.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62e3e00640 1 -- 172.21.15.47:0/4013415028 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e4075360 con 0x7f62e40ae2f0 2024-02-19T15:22:32.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.553+0000 7f62daa00640 1 -- 172.21.15.47:0/4013415028 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f62cc00cce0 con 0x7f62e40ae2f0 2024-02-19T15:22:32.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.557+0000 7f62daa00640 1 --2- 172.21.15.47:0/4013415028 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f62c0066e40 0x7f62c0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:32.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.557+0000 7f62daa00640 1 -- 172.21.15.47:0/4013415028 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62cc084af0 con 0x7f62e40ae2f0 2024-02-19T15:22:32.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.557+0000 7f62e2a00640 1 --2- 172.21.15.47:0/4013415028 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f62c0066e40 0x7f62c0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:32.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.557+0000 7f62e2a00640 1 --2- 172.21.15.47:0/4013415028 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f62c0066e40 0x7f62c0069300 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f62d40ad000 tx=0x7f62d4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:32.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.561+0000 7f62daa00640 1 -- 172.21.15.47:0/4013415028 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f62cc0518f0 con 0x7f62e40ae2f0 2024-02-19T15:22:32.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.721+0000 7f62e3e00640 1 -- 172.21.15.47:0/4013415028 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62e40cda00 con 0x7f62e40ae2f0 2024-02-19T15:22:32.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.725+0000 7f62daa00640 1 -- 172.21.15.47:0/4013415028 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f62cc017090 con 0x7f62e40ae2f0 2024-02-19T15:22:32.728 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:32.728 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":21,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:06.843155+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:03.091360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:04.222965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:32.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.725+0000 7f62e3e00640 1 -- 172.21.15.47:0/4013415028 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f62c0066e40 msgr2=0x7f62c0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:32.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.725+0000 7f62e3e00640 1 --2- 172.21.15.47:0/4013415028 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f62c0066e40 0x7f62c0069300 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f62d40ad000 tx=0x7f62d4008040 comp rx=0 tx=0).stop 2024-02-19T15:22:32.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.725+0000 7f62e3e00640 1 -- 172.21.15.47:0/4013415028 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62e40ae2f0 msgr2=0x7f62e409d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:32.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.725+0000 7f62e3e00640 1 --2- 172.21.15.47:0/4013415028 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62e40ae2f0 0x7f62e409d810 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00e840 tx=0x7f62cc00ec40 comp rx=0 tx=0).stop 2024-02-19T15:22:32.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.725+0000 7f62e3e00640 1 -- 172.21.15.47:0/4013415028 shutdown_connections 2024-02-19T15:22:32.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.725+0000 7f62e3e00640 1 --2- 172.21.15.47:0/4013415028 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f62c0066e40 0x7f62c0069300 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.725+0000 7f62e3e00640 1 --2- 172.21.15.47:0/4013415028 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62e415f180 0x7f62e409b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.725+0000 7f62e3e00640 1 --2- 172.21.15.47:0/4013415028 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62e40ae2f0 0x7f62e409d810 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.725+0000 7f62e3e00640 1 --2- 172.21.15.47:0/4013415028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62e4067100 0x7f62e4071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.725+0000 7f62e3e00640 1 -- 172.21.15.47:0/4013415028 >> 172.21.15.47:0/4013415028 conn(0x7f62e40b1be0 msgr2=0x7f62e40b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:32.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.725+0000 7f62e3e00640 1 -- 172.21.15.47:0/4013415028 shutdown_connections 2024-02-19T15:22:32.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.725+0000 7f62e3e00640 1 -- 172.21.15.47:0/4013415028 wait complete. 2024-02-19T15:22:32.749 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:22:03.091360+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:22:04.222965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:22:32.750 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:22:32.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.889+0000 7f1d28e00640 1 Processor -- start 2024-02-19T15:22:32.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.889+0000 7f1d28e00640 1 -- start start 2024-02-19T15:22:32.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.889+0000 7f1d28e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1d240cb490 0x7f1d24070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:32.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.889+0000 7f1d28e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2415fc90 con 0x7f1d24072850 2024-02-19T15:22:32.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.889+0000 7f1d28e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2415ff70 con 0x7f1d2415bd90 2024-02-19T15:22:32.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.889+0000 7f1d28e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d24160250 con 0x7f1d240cb490 2024-02-19T15:22:32.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.893+0000 7f1d1be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1d240cb490 0x7f1d24070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:32.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.893+0000 7f1d22800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1d24072850 0x7f1d2406cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34144/0 (socket says 172.21.15.47:34144) 2024-02-19T15:22:32.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.893+0000 7f1d22800640 1 -- 172.21.15.47:0/3058674638 learned_addr learned my addr 172.21.15.47:0/3058674638 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:32.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.893+0000 7f1d1be00640 1 -- 172.21.15.47:0/3058674638 >> v1:172.21.15.116:6789/0 conn(0x7f1d2415bd90 legacy=0x7f1d24072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:32.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.893+0000 7f1d1be00640 1 -- 172.21.15.47:0/3058674638 >> v1:172.21.15.47:6789/0 conn(0x7f1d24072850 legacy=0x7f1d2406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:32.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.893+0000 7f1d1be00640 1 -- 172.21.15.47:0/3058674638 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d24160530 con 0x7f1d240cb490 2024-02-19T15:22:32.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.893+0000 7f1d21400640 1 -- 172.21.15.47:0/3058674638 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542415184 0 0) 0x7f1d2415fc90 con 0x7f1d24072850 2024-02-19T15:22:32.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.893+0000 7f1d1be00640 1 --2- 172.21.15.47:0/3058674638 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d240cb490 0x7f1d24070960 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c0097c0 tx=0x7f1d0c01c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=378dd9712546ab98 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:32.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.893+0000 7f1d21400640 1 -- 172.21.15.47:0/3058674638 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c0ab600 con 0x7f1d240cb490 2024-02-19T15:22:32.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.893+0000 7f1d21400640 1 -- 172.21.15.47:0/3058674638 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0c0a7050 con 0x7f1d240cb490 2024-02-19T15:22:32.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.893+0000 7f1d21400640 1 -- 172.21.15.47:0/3058674638 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c0ab980 con 0x7f1d240cb490 2024-02-19T15:22:32.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d28e00640 1 -- 172.21.15.47:0/3058674638 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d240cb490 msgr2=0x7f1d24070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:32.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d28e00640 1 --2- 172.21.15.47:0/3058674638 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d240cb490 0x7f1d24070960 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c0097c0 tx=0x7f1d0c01c380 comp rx=0 tx=0).stop 2024-02-19T15:22:32.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d28e00640 1 -- 172.21.15.47:0/3058674638 shutdown_connections 2024-02-19T15:22:32.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d28e00640 1 --2- 172.21.15.47:0/3058674638 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d240cb490 0x7f1d24070960 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d28e00640 1 -- 172.21.15.47:0/3058674638 >> 172.21.15.47:0/3058674638 conn(0x7f1d240cfc20 msgr2=0x7f1d240d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:32.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d28e00640 1 -- 172.21.15.47:0/3058674638 shutdown_connections 2024-02-19T15:22:32.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d28e00640 1 -- 172.21.15.47:0/3058674638 wait complete. 2024-02-19T15:22:32.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d28e00640 1 Processor -- start 2024-02-19T15:22:32.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d28e00640 1 -- start start 2024-02-19T15:22:32.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d28e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d24072850 0x7f1d24167580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:32.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d28e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d240cb490 0x7f1d24152720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:32.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d28e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d2415bd90 0x7f1d240c88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:32.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d28e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d24158680 con 0x7f1d240cb490 2024-02-19T15:22:32.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d28e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2415aa70 con 0x7f1d2415bd90 2024-02-19T15:22:32.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d28e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2415ad10 con 0x7f1d24072850 2024-02-19T15:22:32.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d1be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d240cb490 0x7f1d24152720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:32.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d1be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d240cb490 0x7f1d24152720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60782/0 (socket says 172.21.15.47:60782) 2024-02-19T15:22:32.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d1be00640 1 -- 172.21.15.47:0/3773756085 learned_addr learned my addr 172.21.15.47:0/3773756085 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:32.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d22800640 1 --2- 172.21.15.47:0/3773756085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d2415bd90 0x7f1d240c88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:32.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d21e00640 1 --2- 172.21.15.47:0/3773756085 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d24072850 0x7f1d24167580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:32.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d1be00640 1 -- 172.21.15.47:0/3773756085 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d24072850 msgr2=0x7f1d24167580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:32.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d1be00640 1 --2- 172.21.15.47:0/3773756085 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d24072850 0x7f1d24167580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d1be00640 1 -- 172.21.15.47:0/3773756085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d2415bd90 msgr2=0x7f1d240c88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:32.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d1be00640 1 --2- 172.21.15.47:0/3773756085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d2415bd90 0x7f1d240c88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:32.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d1be00640 1 -- 172.21.15.47:0/3773756085 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0c009420 con 0x7f1d240cb490 2024-02-19T15:22:32.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.897+0000 7f1d1be00640 1 --2- 172.21.15.47:0/3773756085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d240cb490 0x7f1d24152720 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c018f90 tx=0x7f1d0c019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:32.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.901+0000 7f1d1a000640 1 -- 172.21.15.47:0/3773756085 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c0b4840 con 0x7f1d240cb490 2024-02-19T15:22:32.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.901+0000 7f1d1a000640 1 -- 172.21.15.47:0/3773756085 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0c0b49c0 con 0x7f1d240cb490 2024-02-19T15:22:32.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.901+0000 7f1d1a000640 1 -- 172.21.15.47:0/3773756085 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c0b4d70 con 0x7f1d240cb490 2024-02-19T15:22:32.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.901+0000 7f1d28e00640 1 -- 172.21.15.47:0/3773756085 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d2415afb0 con 0x7f1d240cb490 2024-02-19T15:22:32.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.901+0000 7f1d28e00640 1 -- 172.21.15.47:0/3773756085 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d2415b570 con 0x7f1d240cb490 2024-02-19T15:22:32.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.901+0000 7f1d28e00640 1 -- 172.21.15.47:0/3773756085 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d2415b760 con 0x7f1d240cb490 2024-02-19T15:22:32.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.901+0000 7f1d1a000640 1 -- 172.21.15.47:0/3773756085 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1d0c0b2040 con 0x7f1d240cb490 2024-02-19T15:22:32.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.901+0000 7f1d1a000640 1 --2- 172.21.15.47:0/3773756085 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d00066e70 0x7f1d00069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:32.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.901+0000 7f1d1a000640 1 -- 172.21.15.47:0/3773756085 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d0c11ebe0 con 0x7f1d240cb490 2024-02-19T15:22:32.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.901+0000 7f1d21e00640 1 --2- 172.21.15.47:0/3773756085 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d00066e70 0x7f1d00069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:32.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.901+0000 7f1d21e00640 1 --2- 172.21.15.47:0/3773756085 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d00066e70 0x7f1d00069330 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1d240cae60 tx=0x7f1d14009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:32.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:32.905+0000 7f1d1a000640 1 -- 172.21.15.47:0/3773756085 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1d0c0eba70 con 0x7f1d240cb490 2024-02-19T15:22:33.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.069+0000 7f1d28e00640 1 -- 172.21.15.47:0/3773756085 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1d24075360 con 0x7f1d240cb490 2024-02-19T15:22:33.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.069+0000 7f1d1a000640 1 -- 172.21.15.47:0/3773756085 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1d0c0b80a0 con 0x7f1d240cb490 2024-02-19T15:22:33.072 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:22:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.073+0000 7f1d28e00640 1 -- 172.21.15.47:0/3773756085 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d00066e70 msgr2=0x7f1d00069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.073+0000 7f1d28e00640 1 --2- 172.21.15.47:0/3773756085 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d00066e70 0x7f1d00069330 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1d240cae60 tx=0x7f1d14009040 comp rx=0 tx=0).stop 2024-02-19T15:22:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.073+0000 7f1d28e00640 1 -- 172.21.15.47:0/3773756085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d240cb490 msgr2=0x7f1d24152720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.073+0000 7f1d28e00640 1 --2- 172.21.15.47:0/3773756085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d240cb490 0x7f1d24152720 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c018f90 tx=0x7f1d0c019120 comp rx=0 tx=0).stop 2024-02-19T15:22:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.073+0000 7f1d28e00640 1 -- 172.21.15.47:0/3773756085 shutdown_connections 2024-02-19T15:22:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.073+0000 7f1d28e00640 1 --2- 172.21.15.47:0/3773756085 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d00066e70 0x7f1d00069330 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.073+0000 7f1d28e00640 1 --2- 172.21.15.47:0/3773756085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d2415bd90 0x7f1d240c88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.073+0000 7f1d28e00640 1 --2- 172.21.15.47:0/3773756085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d240cb490 0x7f1d24152720 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.073+0000 7f1d28e00640 1 --2- 172.21.15.47:0/3773756085 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d24072850 0x7f1d24167580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.073+0000 7f1d28e00640 1 -- 172.21.15.47:0/3773756085 >> 172.21.15.47:0/3773756085 conn(0x7f1d240cfc20 msgr2=0x7f1d240ca230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.073+0000 7f1d28e00640 1 -- 172.21.15.47:0/3773756085 shutdown_connections 2024-02-19T15:22:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.073+0000 7f1d28e00640 1 -- 172.21.15.47:0/3773756085 wait complete. 2024-02-19T15:22:33.094 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:22:33.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.233+0000 7ff14e200640 1 Processor -- start 2024-02-19T15:22:33.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.233+0000 7ff14e200640 1 -- start start 2024-02-19T15:22:33.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.233+0000 7ff14e200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff14815af10 0x7ff14806e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:33.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.233+0000 7ff14e200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff1480cd880 0x7ff1480cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:33.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.233+0000 7ff14e200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff148071c70 con 0x7ff1480ca8f0 2024-02-19T15:22:33.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.233+0000 7ff14e200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff14806e8a0 con 0x7ff14815af10 2024-02-19T15:22:33.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.233+0000 7ff14e200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff14806cef0 con 0x7ff1480cd880 2024-02-19T15:22:33.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.233+0000 7ff147400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff14815af10 0x7ff14806e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:33.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.233+0000 7ff147400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff14815af10 0x7ff14806e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60792/0 (socket says 172.21.15.47:60792) 2024-02-19T15:22:33.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.233+0000 7ff147400640 1 -- 172.21.15.47:0/2634907253 learned_addr learned my addr 172.21.15.47:0/2634907253 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:33.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.233+0000 7ff147e00640 1 --2- 172.21.15.47:0/2634907253 >> v2:172.21.15.116:3300/0 conn(0x7ff1480cd880 0x7ff1480cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:33.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.233+0000 7ff147400640 1 -- 172.21.15.47:0/2634907253 >> v2:172.21.15.116:3300/0 conn(0x7ff1480cd880 msgr2=0x7ff1480cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:33.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.233+0000 7ff147400640 1 --2- 172.21.15.47:0/2634907253 >> v2:172.21.15.116:3300/0 conn(0x7ff1480cd880 0x7ff1480cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.233+0000 7ff147400640 1 -- 172.21.15.47:0/2634907253 >> v1:172.21.15.116:6789/0 conn(0x7ff1480ca8f0 legacy=0x7ff1480cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:33.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.233+0000 7ff147400640 1 -- 172.21.15.47:0/2634907253 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1480ca610 con 0x7ff14815af10 2024-02-19T15:22:33.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.237+0000 7ff147400640 1 --2- 172.21.15.47:0/2634907253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff14815af10 0x7ff14806e360 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff148067c60 tx=0x7ff13801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c105bd693b98d1d5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:33.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.237+0000 7ff146a00640 1 -- 172.21.15.47:0/2634907253 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1380ab660 con 0x7ff14815af10 2024-02-19T15:22:33.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.237+0000 7ff146a00640 1 -- 172.21.15.47:0/2634907253 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1380ab7c0 con 0x7ff14815af10 2024-02-19T15:22:33.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.237+0000 7ff146a00640 1 -- 172.21.15.47:0/2634907253 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1380abb50 con 0x7ff14815af10 2024-02-19T15:22:33.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.237+0000 7ff14e200640 1 -- 172.21.15.47:0/2634907253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff14815af10 msgr2=0x7ff14806e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:33.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.237+0000 7ff14e200640 1 --2- 172.21.15.47:0/2634907253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff14815af10 0x7ff14806e360 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff148067c60 tx=0x7ff13801c460 comp rx=0 tx=0).stop 2024-02-19T15:22:33.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.237+0000 7ff14e200640 1 -- 172.21.15.47:0/2634907253 shutdown_connections 2024-02-19T15:22:33.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.237+0000 7ff14e200640 1 --2- 172.21.15.47:0/2634907253 >> v2:172.21.15.116:3300/0 conn(0x7ff1480cd880 0x7ff1480cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.237+0000 7ff14e200640 1 --2- 172.21.15.47:0/2634907253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff14815af10 0x7ff14806e360 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.237+0000 7ff14e200640 1 -- 172.21.15.47:0/2634907253 >> 172.21.15.47:0/2634907253 conn(0x7ff1480cfc20 msgr2=0x7ff1480d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:33.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.237+0000 7ff14e200640 1 -- 172.21.15.47:0/2634907253 shutdown_connections 2024-02-19T15:22:33.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.237+0000 7ff14e200640 1 -- 172.21.15.47:0/2634907253 wait complete. 2024-02-19T15:22:33.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff14e200640 1 Processor -- start 2024-02-19T15:22:33.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff14e200640 1 -- start start 2024-02-19T15:22:33.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff14e200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1480ca8f0 0x7ff1481541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:33.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff14e200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1480cd880 0x7ff148157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:33.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff14e200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff14815af10 0x7ff14806d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:33.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff14e200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1480c6cf0 con 0x7ff1480ca8f0 2024-02-19T15:22:33.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff14e200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14815ce00 con 0x7ff1480cd880 2024-02-19T15:22:33.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff14e200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff148154730 con 0x7ff14815af10 2024-02-19T15:22:33.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff147400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1480ca8f0 0x7ff1481541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:33.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff147400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1480ca8f0 0x7ff1481541f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60808/0 (socket says 172.21.15.47:60808) 2024-02-19T15:22:33.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff13fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1480cd880 0x7ff148157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:33.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff147400640 1 -- 172.21.15.47:0/3939931658 learned_addr learned my addr 172.21.15.47:0/3939931658 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:33.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff147e00640 1 --2- 172.21.15.47:0/3939931658 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff14815af10 0x7ff14806d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:33.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff147400640 1 -- 172.21.15.47:0/3939931658 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff14815af10 msgr2=0x7ff14806d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:33.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff147400640 1 --2- 172.21.15.47:0/3939931658 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff14815af10 0x7ff14806d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff147400640 1 -- 172.21.15.47:0/3939931658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1480cd880 msgr2=0x7ff148157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:33.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff147400640 1 --2- 172.21.15.47:0/3939931658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1480cd880 0x7ff148157ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff147400640 1 -- 172.21.15.47:0/3939931658 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1380094d0 con 0x7ff1480ca8f0 2024-02-19T15:22:33.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff13fe00640 1 --2- 172.21.15.47:0/3939931658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1480cd880 0x7ff148157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:33.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff147400640 1 --2- 172.21.15.47:0/3939931658 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1480ca8f0 0x7ff1481541f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff138019070 tx=0x7ff13801aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:33.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff13f400640 1 -- 172.21.15.47:0/3939931658 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1380ab660 con 0x7ff1480ca8f0 2024-02-19T15:22:33.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff13f400640 1 -- 172.21.15.47:0/3939931658 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1380b4800 con 0x7ff1480ca8f0 2024-02-19T15:22:33.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff13f400640 1 -- 172.21.15.47:0/3939931658 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1380b4b90 con 0x7ff1480ca8f0 2024-02-19T15:22:33.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff14e200640 1 -- 172.21.15.47:0/3939931658 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1481586f0 con 0x7ff1480ca8f0 2024-02-19T15:22:33.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff14e200640 1 -- 172.21.15.47:0/3939931658 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff148158cb0 con 0x7ff1480ca8f0 2024-02-19T15:22:33.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.241+0000 7ff14e200640 1 -- 172.21.15.47:0/3939931658 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff148075360 con 0x7ff1480ca8f0 2024-02-19T15:22:33.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.245+0000 7ff13f400640 1 -- 172.21.15.47:0/3939931658 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff1380a9070 con 0x7ff1480ca8f0 2024-02-19T15:22:33.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.245+0000 7ff13f400640 1 --2- 172.21.15.47:0/3939931658 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff124066e40 0x7ff124069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:33.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.245+0000 7ff13f400640 1 -- 172.21.15.47:0/3939931658 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff13811ef60 con 0x7ff1480ca8f0 2024-02-19T15:22:33.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.245+0000 7ff13fe00640 1 --2- 172.21.15.47:0/3939931658 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff124066e40 0x7ff124069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:33.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.245+0000 7ff13fe00640 1 --2- 172.21.15.47:0/3939931658 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff124066e40 0x7ff124069300 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff148155fb0 tx=0x7ff130009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:33.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.249+0000 7ff13f400640 1 -- 172.21.15.47:0/3939931658 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff1380ebe80 con 0x7ff1480ca8f0 2024-02-19T15:22:33.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.413+0000 7ff14e200640 1 -- 172.21.15.47:0/3939931658 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff1480d0740 con 0x7ff1480ca8f0 2024-02-19T15:22:33.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.413+0000 7ff13f400640 1 -- 172.21.15.47:0/3939931658 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff1380ef540 con 0x7ff1480ca8f0 2024-02-19T15:22:33.415 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:22:33.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.413+0000 7ff14e200640 1 -- 172.21.15.47:0/3939931658 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff124066e40 msgr2=0x7ff124069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:33.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.413+0000 7ff14e200640 1 --2- 172.21.15.47:0/3939931658 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff124066e40 0x7ff124069300 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff148155fb0 tx=0x7ff130009040 comp rx=0 tx=0).stop 2024-02-19T15:22:33.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.413+0000 7ff14e200640 1 -- 172.21.15.47:0/3939931658 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1480ca8f0 msgr2=0x7ff1481541f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:33.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.413+0000 7ff14e200640 1 --2- 172.21.15.47:0/3939931658 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1480ca8f0 0x7ff1481541f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff138019070 tx=0x7ff13801aca0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.413+0000 7ff14e200640 1 -- 172.21.15.47:0/3939931658 shutdown_connections 2024-02-19T15:22:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.413+0000 7ff14e200640 1 --2- 172.21.15.47:0/3939931658 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff124066e40 0x7ff124069300 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.413+0000 7ff14e200640 1 --2- 172.21.15.47:0/3939931658 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff14815af10 0x7ff14806d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.413+0000 7ff14e200640 1 --2- 172.21.15.47:0/3939931658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1480cd880 0x7ff148157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.413+0000 7ff14e200640 1 --2- 172.21.15.47:0/3939931658 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1480ca8f0 0x7ff1481541f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.413+0000 7ff14e200640 1 -- 172.21.15.47:0/3939931658 >> 172.21.15.47:0/3939931658 conn(0x7ff1480cfc20 msgr2=0x7ff1480b5010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.413+0000 7ff14e200640 1 -- 172.21.15.47:0/3939931658 shutdown_connections 2024-02-19T15:22:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.413+0000 7ff14e200640 1 -- 172.21.15.47:0/3939931658 wait complete. 2024-02-19T15:22:33.436 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:22:33.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.573+0000 7f96aa200640 1 Processor -- start 2024-02-19T15:22:33.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.573+0000 7f96aa200640 1 -- start start 2024-02-19T15:22:33.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.577+0000 7f96aa200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f96a415bd90 0x7f96a4070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:33.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.577+0000 7f96aa200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f96a40cb490 0x7f96a4072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:33.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.577+0000 7f96aa200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a415fc90 con 0x7f96a4072680 2024-02-19T15:22:33.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.577+0000 7f96aa200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96a415ff70 con 0x7f96a40cb490 2024-02-19T15:22:33.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.577+0000 7f96aa200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96a4160250 con 0x7f96a415bd90 2024-02-19T15:22:33.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.577+0000 7f969aa00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f96a40cb490 0x7f96a4072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:33.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.577+0000 7f96a3e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f96a4072680 0x7f96a40ca270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38080/0 (socket says 172.21.15.47:38080) 2024-02-19T15:22:33.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.577+0000 7f96a3e00640 1 -- 172.21.15.47:0/1088096536 learned_addr learned my addr 172.21.15.47:0/1088096536 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:33.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.577+0000 7f969aa00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f96a40cb490 0x7f96a4072140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44596/0 (socket says 172.21.15.47:44596) 2024-02-19T15:22:33.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.577+0000 7f96a3400640 1 --2- 172.21.15.47:0/1088096536 >> v2:172.21.15.116:3300/0 conn(0x7f96a415bd90 0x7f96a4070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:33.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.577+0000 7f969aa00640 1 -- 172.21.15.47:0/1088096536 >> v2:172.21.15.116:3300/0 conn(0x7f96a415bd90 msgr2=0x7f96a4070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:33.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.577+0000 7f969aa00640 1 --2- 172.21.15.47:0/1088096536 >> v2:172.21.15.116:3300/0 conn(0x7f96a415bd90 0x7f96a4070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.577+0000 7f969aa00640 1 -- 172.21.15.47:0/1088096536 >> v1:172.21.15.47:6789/0 conn(0x7f96a4072680 legacy=0x7f96a40ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:33.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.577+0000 7f969aa00640 1 -- 172.21.15.47:0/1088096536 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a4160530 con 0x7f96a40cb490 2024-02-19T15:22:33.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.581+0000 7f969aa00640 1 --2- 172.21.15.47:0/1088096536 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96a40cb490 0x7f96a4072140 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f96a406c300 tx=0x7f968c01c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db26132dd4fe0c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:33.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.581+0000 7f96a2a00640 1 -- 172.21.15.47:0/1088096536 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c0ad680 con 0x7f96a40cb490 2024-02-19T15:22:33.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.581+0000 7f96a2a00640 1 -- 172.21.15.47:0/1088096536 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968c0ad7e0 con 0x7f96a40cb490 2024-02-19T15:22:33.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.581+0000 7f96a2a00640 1 -- 172.21.15.47:0/1088096536 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c0adb70 con 0x7f96a40cb490 2024-02-19T15:22:33.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.581+0000 7f96aa200640 1 -- 172.21.15.47:0/1088096536 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96a40cb490 msgr2=0x7f96a4072140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:33.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.581+0000 7f96aa200640 1 --2- 172.21.15.47:0/1088096536 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96a40cb490 0x7f96a4072140 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f96a406c300 tx=0x7f968c01c320 comp rx=0 tx=0).stop 2024-02-19T15:22:33.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.581+0000 7f96aa200640 1 -- 172.21.15.47:0/1088096536 shutdown_connections 2024-02-19T15:22:33.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.581+0000 7f96aa200640 1 --2- 172.21.15.47:0/1088096536 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96a40cb490 0x7f96a4072140 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.581+0000 7f96aa200640 1 --2- 172.21.15.47:0/1088096536 >> v2:172.21.15.116:3300/0 conn(0x7f96a415bd90 0x7f96a4070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.581+0000 7f96aa200640 1 -- 172.21.15.47:0/1088096536 >> 172.21.15.47:0/1088096536 conn(0x7f96a40cfc20 msgr2=0x7f96a40d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:33.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.581+0000 7f96aa200640 1 -- 172.21.15.47:0/1088096536 shutdown_connections 2024-02-19T15:22:33.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.581+0000 7f96aa200640 1 -- 172.21.15.47:0/1088096536 wait complete. 2024-02-19T15:22:33.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.581+0000 7f96aa200640 1 Processor -- start 2024-02-19T15:22:33.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.581+0000 7f96aa200640 1 -- start start 2024-02-19T15:22:33.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f96aa200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96a4072680 0x7f96a40c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:33.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f96aa200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a40cb490 0x7f96a4159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:33.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f96aa200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96a415bd90 0x7f96a415eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:33.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f96aa200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a415ea30 con 0x7f96a4072680 2024-02-19T15:22:33.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f96aa200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a415de20 con 0x7f96a40cb490 2024-02-19T15:22:33.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f96aa200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96a415e100 con 0x7f96a415bd90 2024-02-19T15:22:33.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f96a3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96a4072680 0x7f96a40c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:33.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f96a3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96a4072680 0x7f96a40c55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44598/0 (socket says 172.21.15.47:44598) 2024-02-19T15:22:33.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f96a3400640 1 -- 172.21.15.47:0/279060428 learned_addr learned my addr 172.21.15.47:0/279060428 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:33.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f96a3e00640 1 --2- 172.21.15.47:0/279060428 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96a415bd90 0x7f96a415eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:33.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f969aa00640 1 --2- 172.21.15.47:0/279060428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a40cb490 0x7f96a4159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:33.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f96a3400640 1 -- 172.21.15.47:0/279060428 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96a415bd90 msgr2=0x7f96a415eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:33.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f96a3400640 1 --2- 172.21.15.47:0/279060428 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96a415bd90 0x7f96a415eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f96a3400640 1 -- 172.21.15.47:0/279060428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a40cb490 msgr2=0x7f96a4159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:33.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f96a3400640 1 --2- 172.21.15.47:0/279060428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a40cb490 0x7f96a4159490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f96a3400640 1 -- 172.21.15.47:0/279060428 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968c009420 con 0x7f96a4072680 2024-02-19T15:22:33.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f96a3400640 1 --2- 172.21.15.47:0/279060428 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96a4072680 0x7f96a40c55e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f969400e870 tx=0x7f969400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:33.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f969be00640 1 -- 172.21.15.47:0/279060428 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9694009510 con 0x7f96a4072680 2024-02-19T15:22:33.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f969be00640 1 -- 172.21.15.47:0/279060428 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969400eec0 con 0x7f96a4072680 2024-02-19T15:22:33.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f96aa200640 1 -- 172.21.15.47:0/279060428 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96a415e3e0 con 0x7f96a4072680 2024-02-19T15:22:33.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f969be00640 1 -- 172.21.15.47:0/279060428 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969400cb40 con 0x7f96a4072680 2024-02-19T15:22:33.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f96aa200640 1 -- 172.21.15.47:0/279060428 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96a40c5b20 con 0x7f96a4072680 2024-02-19T15:22:33.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f96aa200640 1 -- 172.21.15.47:0/279060428 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96a4075360 con 0x7f96a4072680 2024-02-19T15:22:33.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.585+0000 7f969be00640 1 -- 172.21.15.47:0/279060428 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f969400cca0 con 0x7f96a4072680 2024-02-19T15:22:33.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.589+0000 7f969be00640 1 --2- 172.21.15.47:0/279060428 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9680066e40 0x7f9680069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:33.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.589+0000 7f969be00640 1 -- 172.21.15.47:0/279060428 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9694084b40 con 0x7f96a4072680 2024-02-19T15:22:33.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.589+0000 7f969aa00640 1 --2- 172.21.15.47:0/279060428 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9680066e40 0x7f9680069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:33.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.589+0000 7f969aa00640 1 --2- 172.21.15.47:0/279060428 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9680066e40 0x7f9680069300 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f968c0af000 tx=0x7f968c0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:33.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.593+0000 7f969be00640 1 -- 172.21.15.47:0/279060428 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f9694051940 con 0x7f96a4072680 2024-02-19T15:22:33.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.749+0000 7f96aa200640 1 -- 172.21.15.47:0/279060428 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f96a406e800 con 0x7f96a4072680 2024-02-19T15:22:33.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.753+0000 7f969be00640 1 -- 172.21.15.47:0/279060428 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f96940170a0 con 0x7f96a4072680 2024-02-19T15:22:33.754 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:22:33.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.753+0000 7f96aa200640 1 -- 172.21.15.47:0/279060428 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9680066e40 msgr2=0x7f9680069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:33.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.753+0000 7f96aa200640 1 --2- 172.21.15.47:0/279060428 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9680066e40 0x7f9680069300 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f968c0af000 tx=0x7f968c0a7000 comp rx=0 tx=0).stop 2024-02-19T15:22:33.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.753+0000 7f96aa200640 1 -- 172.21.15.47:0/279060428 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96a4072680 msgr2=0x7f96a40c55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:33.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.753+0000 7f96aa200640 1 --2- 172.21.15.47:0/279060428 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96a4072680 0x7f96a40c55e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f969400e870 tx=0x7f969400ec70 comp rx=0 tx=0).stop 2024-02-19T15:22:33.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.753+0000 7f96aa200640 1 -- 172.21.15.47:0/279060428 shutdown_connections 2024-02-19T15:22:33.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.753+0000 7f96aa200640 1 --2- 172.21.15.47:0/279060428 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9680066e40 0x7f9680069300 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.753+0000 7f96aa200640 1 --2- 172.21.15.47:0/279060428 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96a415bd90 0x7f96a415eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.753+0000 7f96aa200640 1 --2- 172.21.15.47:0/279060428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a40cb490 0x7f96a4159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.753+0000 7f96aa200640 1 --2- 172.21.15.47:0/279060428 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96a4072680 0x7f96a40c55e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.753+0000 7f96aa200640 1 -- 172.21.15.47:0/279060428 >> 172.21.15.47:0/279060428 conn(0x7f96a40cfc20 msgr2=0x7f96a40b47c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:33.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.753+0000 7f96aa200640 1 -- 172.21.15.47:0/279060428 shutdown_connections 2024-02-19T15:22:33.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.753+0000 7f96aa200640 1 -- 172.21.15.47:0/279060428 wait complete. 2024-02-19T15:22:33.785 INFO:tasks.cephfs_test_runner:Starting test: test_dir_deletion (tasks.cephfs.test_strays.TestStrays) That when deleting a bunch of dentries and the containing 2024-02-19T15:22:33.785 INFO:tasks.cephfs_test_runner:test_dir_deletion (tasks.cephfs.test_strays.TestStrays) 2024-02-19T15:22:33.785 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:33.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.921+0000 7f5873400640 1 Processor -- start 2024-02-19T15:22:33.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.921+0000 7f5873400640 1 -- start start 2024-02-19T15:22:33.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.921+0000 7f5873400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f586c15b1a0 0x7f586c15b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:33.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.921+0000 7f5873400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586c0718c0 con 0x7f586c071350 2024-02-19T15:22:33.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.921+0000 7f5873400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586c071ba0 con 0x7f586c06c9a0 2024-02-19T15:22:33.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.921+0000 7f5873400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f586c071e80 con 0x7f586c15b1a0 2024-02-19T15:22:33.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.921+0000 7f5870a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f586c15b1a0 0x7f586c15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:33.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.925+0000 7f5870a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f586c15b1a0 0x7f586c15b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44608/0 (socket says 172.21.15.47:44608) 2024-02-19T15:22:33.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.925+0000 7f5870a00640 1 -- 172.21.15.47:0/4017475250 learned_addr learned my addr 172.21.15.47:0/4017475250 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:33.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.925+0000 7f5870a00640 1 -- 172.21.15.47:0/4017475250 >> v1:172.21.15.116:6789/0 conn(0x7f586c06c9a0 legacy=0x7f586c15cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:33.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.925+0000 7f5870a00640 1 -- 172.21.15.47:0/4017475250 >> v1:172.21.15.47:6789/0 conn(0x7f586c071350 legacy=0x7f586c15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:33.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.925+0000 7f5870a00640 1 -- 172.21.15.47:0/4017475250 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586c072160 con 0x7f586c15b1a0 2024-02-19T15:22:33.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.925+0000 7f5870a00640 1 --2- 172.21.15.47:0/4017475250 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f586c15b1a0 0x7f586c15b580 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f585c0098a0 tx=0x7f585c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23587974fdbaa260 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:33.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.925+0000 7f586be00640 1 -- 172.21.15.47:0/4017475250 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f585c0aa430 con 0x7f586c15b1a0 2024-02-19T15:22:33.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.925+0000 7f586be00640 1 -- 172.21.15.47:0/4017475250 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f585c0aa590 con 0x7f586c15b1a0 2024-02-19T15:22:33.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.925+0000 7f586be00640 1 -- 172.21.15.47:0/4017475250 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f585c0aa920 con 0x7f586c15b1a0 2024-02-19T15:22:33.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.925+0000 7f5873400640 1 -- 172.21.15.47:0/4017475250 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f586c15b1a0 msgr2=0x7f586c15b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:33.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.925+0000 7f5873400640 1 --2- 172.21.15.47:0/4017475250 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f586c15b1a0 0x7f586c15b580 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f585c0098a0 tx=0x7f585c01c540 comp rx=0 tx=0).stop 2024-02-19T15:22:33.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5873400640 1 -- 172.21.15.47:0/4017475250 shutdown_connections 2024-02-19T15:22:33.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5873400640 1 --2- 172.21.15.47:0/4017475250 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f586c15b1a0 0x7f586c15b580 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5873400640 1 -- 172.21.15.47:0/4017475250 >> 172.21.15.47:0/4017475250 conn(0x7f586c0b16a0 msgr2=0x7f586c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:33.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5873400640 1 -- 172.21.15.47:0/4017475250 shutdown_connections 2024-02-19T15:22:33.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5873400640 1 -- 172.21.15.47:0/4017475250 wait complete. 2024-02-19T15:22:33.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5873400640 1 Processor -- start 2024-02-19T15:22:33.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5873400640 1 -- start start 2024-02-19T15:22:33.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5873400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f586c06c9a0 0x7f586c155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:33.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5873400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f586c071350 0x7f586c0cb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:33.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5873400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f586c15b1a0 0x7f586c0cdf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:33.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5873400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586c154cd0 con 0x7f586c071350 2024-02-19T15:22:33.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5873400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586c155290 con 0x7f586c15b1a0 2024-02-19T15:22:33.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5873400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f586c1f7a20 con 0x7f586c06c9a0 2024-02-19T15:22:33.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5863e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f586c071350 0x7f586c0cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:33.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5870a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f586c06c9a0 0x7f586c155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:33.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5863e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f586c071350 0x7f586c0cb160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44618/0 (socket says 172.21.15.47:44618) 2024-02-19T15:22:33.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5863e00640 1 -- 172.21.15.47:0/2953140325 learned_addr learned my addr 172.21.15.47:0/2953140325 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:33.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5871400640 1 --2- 172.21.15.47:0/2953140325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f586c15b1a0 0x7f586c0cdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:33.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5863e00640 1 -- 172.21.15.47:0/2953140325 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f586c06c9a0 msgr2=0x7f586c155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:33.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5863e00640 1 --2- 172.21.15.47:0/2953140325 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f586c06c9a0 0x7f586c155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5863e00640 1 -- 172.21.15.47:0/2953140325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f586c15b1a0 msgr2=0x7f586c0cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:33.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5863e00640 1 --2- 172.21.15.47:0/2953140325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f586c15b1a0 0x7f586c0cdf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:33.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5863e00640 1 -- 172.21.15.47:0/2953140325 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f585c0094d0 con 0x7f586c071350 2024-02-19T15:22:33.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.929+0000 7f5863e00640 1 --2- 172.21.15.47:0/2953140325 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f586c071350 0x7f586c0cb160 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f585400c420 tx=0x7f585400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:33.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.933+0000 7f5869600640 1 -- 172.21.15.47:0/2953140325 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58540052d0 con 0x7f586c071350 2024-02-19T15:22:33.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.933+0000 7f5869600640 1 -- 172.21.15.47:0/2953140325 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5854005430 con 0x7f586c071350 2024-02-19T15:22:33.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.933+0000 7f5869600640 1 -- 172.21.15.47:0/2953140325 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5854007260 con 0x7f586c071350 2024-02-19T15:22:33.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.933+0000 7f5873400640 1 -- 172.21.15.47:0/2953140325 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f586c1f7d00 con 0x7f586c071350 2024-02-19T15:22:33.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.933+0000 7f5873400640 1 -- 172.21.15.47:0/2953140325 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f586c155c60 con 0x7f586c071350 2024-02-19T15:22:33.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.933+0000 7f5873400640 1 -- 172.21.15.47:0/2953140325 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f586c0b4b50 con 0x7f586c071350 2024-02-19T15:22:33.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.933+0000 7f5869600640 1 -- 172.21.15.47:0/2953140325 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f58540073c0 con 0x7f586c071350 2024-02-19T15:22:33.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.933+0000 7f5869600640 1 --2- 172.21.15.47:0/2953140325 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5848066e70 0x7f5848069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:33.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.933+0000 7f5869600640 1 -- 172.21.15.47:0/2953140325 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5854083e30 con 0x7f586c071350 2024-02-19T15:22:33.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.933+0000 7f5870a00640 1 --2- 172.21.15.47:0/2953140325 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5848066e70 0x7f5848069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:33.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.937+0000 7f5870a00640 1 --2- 172.21.15.47:0/2953140325 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5848066e70 0x7f5848069330 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f585c0ae000 tx=0x7f585c0b5040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:33.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:33.937+0000 7f5869600640 1 -- 172.21.15.47:0/2953140325 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f5854050c30 con 0x7f586c071350 2024-02-19T15:22:34.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.105+0000 7f5873400640 1 -- 172.21.15.47:0/2953140325 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f586c075360 con 0x7f586c071350 2024-02-19T15:22:34.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.105+0000 7f5869600640 1 -- 172.21.15.47:0/2953140325 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f585407be60 con 0x7f586c071350 2024-02-19T15:22:34.109 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:34.109 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":21,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:06.843155+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:03.091360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:04.222965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:34.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.109+0000 7f5873400640 1 -- 172.21.15.47:0/2953140325 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5848066e70 msgr2=0x7f5848069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:34.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.109+0000 7f5873400640 1 --2- 172.21.15.47:0/2953140325 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5848066e70 0x7f5848069330 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f585c0ae000 tx=0x7f585c0b5040 comp rx=0 tx=0).stop 2024-02-19T15:22:34.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.109+0000 7f5873400640 1 -- 172.21.15.47:0/2953140325 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f586c071350 msgr2=0x7f586c0cb160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:34.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.109+0000 7f5873400640 1 --2- 172.21.15.47:0/2953140325 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f586c071350 0x7f586c0cb160 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f585400c420 tx=0x7f585400c8e0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.109+0000 7f5873400640 1 -- 172.21.15.47:0/2953140325 shutdown_connections 2024-02-19T15:22:34.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.109+0000 7f5873400640 1 --2- 172.21.15.47:0/2953140325 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5848066e70 0x7f5848069330 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.109+0000 7f5873400640 1 --2- 172.21.15.47:0/2953140325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f586c15b1a0 0x7f586c0cdf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.109+0000 7f5873400640 1 --2- 172.21.15.47:0/2953140325 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f586c071350 0x7f586c0cb160 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.109+0000 7f5873400640 1 --2- 172.21.15.47:0/2953140325 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f586c06c9a0 0x7f586c155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.109+0000 7f5873400640 1 -- 172.21.15.47:0/2953140325 >> 172.21.15.47:0/2953140325 conn(0x7f586c0b16a0 msgr2=0x7f586c0ad840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:34.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.109+0000 7f5873400640 1 -- 172.21.15.47:0/2953140325 shutdown_connections 2024-02-19T15:22:34.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.109+0000 7f5873400640 1 -- 172.21.15.47:0/2953140325 wait complete. 2024-02-19T15:22:34.131 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:22:03.091360+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:22:04.222965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:22:34.131 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:22:34.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.269+0000 7fbbf7e00640 1 Processor -- start 2024-02-19T15:22:34.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.273+0000 7fbbf7e00640 1 -- start start 2024-02-19T15:22:34.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.273+0000 7fbbf7e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbbf00d2650 0x7fbbf006c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:34.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.273+0000 7fbbf7e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbbf015e280 0x7fbbf006c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:34.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.273+0000 7fbbf7e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf00cc620 con 0x7fbbf00ce3b0 2024-02-19T15:22:34.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.273+0000 7fbbf7e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbf00d12e0 con 0x7fbbf00d2650 2024-02-19T15:22:34.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.273+0000 7fbbf7e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf00d1940 con 0x7fbbf015e280 2024-02-19T15:22:34.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.273+0000 7fbbf4a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbbf00d2650 0x7fbbf006c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:34.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.273+0000 7fbbf4a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbbf00d2650 0x7fbbf006c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44626/0 (socket says 172.21.15.47:44626) 2024-02-19T15:22:34.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.273+0000 7fbbf5400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbbf00ce3b0 0x7fbbf00ce790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38100/0 (socket says 172.21.15.47:38100) 2024-02-19T15:22:34.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.273+0000 7fbbf4a00640 1 -- 172.21.15.47:0/3009675036 learned_addr learned my addr 172.21.15.47:0/3009675036 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:34.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.273+0000 7fbbf5e00640 1 --2- 172.21.15.47:0/3009675036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbbf015e280 0x7fbbf006c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:34.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.273+0000 7fbbf4a00640 1 -- 172.21.15.47:0/3009675036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbbf015e280 msgr2=0x7fbbf006c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:34.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.273+0000 7fbbf4a00640 1 --2- 172.21.15.47:0/3009675036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbbf015e280 0x7fbbf006c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.273+0000 7fbbf4a00640 1 -- 172.21.15.47:0/3009675036 >> v1:172.21.15.47:6789/0 conn(0x7fbbf00ce3b0 legacy=0x7fbbf00ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:34.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.273+0000 7fbbf4a00640 1 -- 172.21.15.47:0/3009675036 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf00d1000 con 0x7fbbf00d2650 2024-02-19T15:22:34.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.277+0000 7fbbf4a00640 1 --2- 172.21.15.47:0/3009675036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbf00d2650 0x7fbbf006c1d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc009870 tx=0x7fbbdc01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10bada6e73ca0241 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:34.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.277+0000 7fbbefe00640 1 -- 172.21.15.47:0/3009675036 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc004230 con 0x7fbbf00d2650 2024-02-19T15:22:34.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.277+0000 7fbbefe00640 1 -- 172.21.15.47:0/3009675036 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbdc0a7050 con 0x7fbbf00d2650 2024-02-19T15:22:34.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.277+0000 7fbbefe00640 1 -- 172.21.15.47:0/3009675036 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc003620 con 0x7fbbf00d2650 2024-02-19T15:22:34.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.277+0000 7fbbf7e00640 1 -- 172.21.15.47:0/3009675036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbf00d2650 msgr2=0x7fbbf006c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:34.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.277+0000 7fbbf7e00640 1 --2- 172.21.15.47:0/3009675036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbf00d2650 0x7fbbf006c1d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc009870 tx=0x7fbbdc01c440 comp rx=0 tx=0).stop 2024-02-19T15:22:34.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.277+0000 7fbbf7e00640 1 -- 172.21.15.47:0/3009675036 shutdown_connections 2024-02-19T15:22:34.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.277+0000 7fbbf7e00640 1 --2- 172.21.15.47:0/3009675036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbbf015e280 0x7fbbf006c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.277+0000 7fbbf7e00640 1 --2- 172.21.15.47:0/3009675036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbf00d2650 0x7fbbf006c1d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.277+0000 7fbbf7e00640 1 -- 172.21.15.47:0/3009675036 >> 172.21.15.47:0/3009675036 conn(0x7fbbf00d1b30 msgr2=0x7fbbf00d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:34.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.277+0000 7fbbf7e00640 1 -- 172.21.15.47:0/3009675036 shutdown_connections 2024-02-19T15:22:34.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.277+0000 7fbbf7e00640 1 -- 172.21.15.47:0/3009675036 wait complete. 2024-02-19T15:22:34.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.277+0000 7fbbf7e00640 1 Processor -- start 2024-02-19T15:22:34.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.277+0000 7fbbf7e00640 1 -- start start 2024-02-19T15:22:34.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbf00ce3b0 0x7fbbf0159e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:34.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf7e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbbf00d2650 0x7fbbf006fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:34.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbf015e280 0x7fbbf0070080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:34.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf7e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf0154cd0 con 0x7fbbf00ce3b0 2024-02-19T15:22:34.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf7e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf00c9ee0 con 0x7fbbf015e280 2024-02-19T15:22:34.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf7e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf0156b40 con 0x7fbbf00d2650 2024-02-19T15:22:34.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf5400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbf00ce3b0 0x7fbbf0159e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:34.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf5400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbf00ce3b0 0x7fbbf0159e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44630/0 (socket says 172.21.15.47:44630) 2024-02-19T15:22:34.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf5400640 1 -- 172.21.15.47:0/2125774677 learned_addr learned my addr 172.21.15.47:0/2125774677 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:34.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf4a00640 1 --2- 172.21.15.47:0/2125774677 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbbf00d2650 0x7fbbf006fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:34.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf5e00640 1 --2- 172.21.15.47:0/2125774677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbf015e280 0x7fbbf0070080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:34.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf5400640 1 -- 172.21.15.47:0/2125774677 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbbf00d2650 msgr2=0x7fbbf006fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:34.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf5400640 1 --2- 172.21.15.47:0/2125774677 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbbf00d2650 0x7fbbf006fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf5400640 1 -- 172.21.15.47:0/2125774677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbf015e280 msgr2=0x7fbbf0070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:34.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf5400640 1 --2- 172.21.15.47:0/2125774677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbf015e280 0x7fbbf0070080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf5400640 1 -- 172.21.15.47:0/2125774677 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbdc0094d0 con 0x7fbbf00ce3b0 2024-02-19T15:22:34.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf4a00640 1 --2- 172.21.15.47:0/2125774677 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbbf00d2650 0x7fbbf006fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:34.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf5400640 1 --2- 172.21.15.47:0/2125774677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbf00ce3b0 0x7fbbf0159e10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400c370 tx=0x7fbbe400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:34.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbed600640 1 -- 172.21.15.47:0/2125774677 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe40074e0 con 0x7fbbf00ce3b0 2024-02-19T15:22:34.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbed600640 1 -- 172.21.15.47:0/2125774677 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe4007640 con 0x7fbbf00ce3b0 2024-02-19T15:22:34.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbed600640 1 -- 172.21.15.47:0/2125774677 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe4004a30 con 0x7fbbf00ce3b0 2024-02-19T15:22:34.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf7e00640 1 -- 172.21.15.47:0/2125774677 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf0156e20 con 0x7fbbf00ce3b0 2024-02-19T15:22:34.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf7e00640 1 -- 172.21.15.47:0/2125774677 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf015a350 con 0x7fbbf00ce3b0 2024-02-19T15:22:34.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbf7e00640 1 -- 172.21.15.47:0/2125774677 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf0159750 con 0x7fbbf00ce3b0 2024-02-19T15:22:34.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.281+0000 7fbbed600640 1 -- 172.21.15.47:0/2125774677 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fbbe4004b90 con 0x7fbbf00ce3b0 2024-02-19T15:22:34.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.285+0000 7fbbed600640 1 --2- 172.21.15.47:0/2125774677 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbbd0066e40 0x7fbbd0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:34.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.285+0000 7fbbed600640 1 -- 172.21.15.47:0/2125774677 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbbe4084290 con 0x7fbbf00ce3b0 2024-02-19T15:22:34.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.285+0000 7fbbf4a00640 1 --2- 172.21.15.47:0/2125774677 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbbd0066e40 0x7fbbd0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:34.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.285+0000 7fbbf4a00640 1 --2- 172.21.15.47:0/2125774677 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbbd0066e40 0x7fbbd0069300 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0ad000 tx=0x7fbbdc0a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:34.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.289+0000 7fbbed600640 1 -- 172.21.15.47:0/2125774677 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fbbf0159750 con 0x7fbbf00ce3b0 2024-02-19T15:22:34.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.453+0000 7fbbf7e00640 1 -- 172.21.15.47:0/2125774677 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbbf0159750 con 0x7fbbf00ce3b0 2024-02-19T15:22:34.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.453+0000 7fbbed600640 1 -- 172.21.15.47:0/2125774677 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbbf0159750 con 0x7fbbf00ce3b0 2024-02-19T15:22:34.455 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:22:34.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.453+0000 7fbbf7e00640 1 -- 172.21.15.47:0/2125774677 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbbd0066e40 msgr2=0x7fbbd0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:34.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.453+0000 7fbbf7e00640 1 --2- 172.21.15.47:0/2125774677 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbbd0066e40 0x7fbbd0069300 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0ad000 tx=0x7fbbdc0a6000 comp rx=0 tx=0).stop 2024-02-19T15:22:34.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.453+0000 7fbbf7e00640 1 -- 172.21.15.47:0/2125774677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbf00ce3b0 msgr2=0x7fbbf0159e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:34.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.453+0000 7fbbf7e00640 1 --2- 172.21.15.47:0/2125774677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbf00ce3b0 0x7fbbf0159e10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400c370 tx=0x7fbbe400c830 comp rx=0 tx=0).stop 2024-02-19T15:22:34.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.453+0000 7fbbf7e00640 1 -- 172.21.15.47:0/2125774677 shutdown_connections 2024-02-19T15:22:34.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.453+0000 7fbbf7e00640 1 --2- 172.21.15.47:0/2125774677 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbbd0066e40 0x7fbbd0069300 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.453+0000 7fbbf7e00640 1 --2- 172.21.15.47:0/2125774677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbf015e280 0x7fbbf0070080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.453+0000 7fbbf7e00640 1 --2- 172.21.15.47:0/2125774677 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbbf00d2650 0x7fbbf006fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.453+0000 7fbbf7e00640 1 --2- 172.21.15.47:0/2125774677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbf00ce3b0 0x7fbbf0159e10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.453+0000 7fbbf7e00640 1 -- 172.21.15.47:0/2125774677 >> 172.21.15.47:0/2125774677 conn(0x7fbbf00d1b30 msgr2=0x7fbbf00c65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:34.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.453+0000 7fbbf7e00640 1 -- 172.21.15.47:0/2125774677 shutdown_connections 2024-02-19T15:22:34.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.453+0000 7fbbf7e00640 1 -- 172.21.15.47:0/2125774677 wait complete. 2024-02-19T15:22:34.476 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:22:34.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.617+0000 7fe2ae400640 1 Processor -- start 2024-02-19T15:22:34.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.617+0000 7fe2ae400640 1 -- start start 2024-02-19T15:22:34.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.617+0000 7fe2ae400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe2a806e1e0 0x7fe2a80d1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:34.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.617+0000 7fe2ae400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe2a806f220 0x7fe2a815e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:34.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.617+0000 7fe2ae400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a8071a70 con 0x7fe2a815e160 2024-02-19T15:22:34.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.617+0000 7fe2ae400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2a806f690 con 0x7fe2a806f220 2024-02-19T15:22:34.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.617+0000 7fe2ae400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2a80ced30 con 0x7fe2a806e1e0 2024-02-19T15:22:34.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.617+0000 7fe2a7400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fe2a815e160 0x7fe2a806dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38108/0 (socket says 172.21.15.47:38108) 2024-02-19T15:22:34.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.617+0000 7fe2a7400640 1 -- 172.21.15.47:0/3387058078 learned_addr learned my addr 172.21.15.47:0/3387058078 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:34.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.617+0000 7fe2a7e00640 1 --2- 172.21.15.47:0/3387058078 >> v2:172.21.15.47:3300/0 conn(0x7fe2a806f220 0x7fe2a815e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:34.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.617+0000 7fe29fe00640 1 --2- 172.21.15.47:0/3387058078 >> v2:172.21.15.116:3300/0 conn(0x7fe2a806e1e0 0x7fe2a80d1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:34.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.617+0000 7fe2a7e00640 1 -- 172.21.15.47:0/3387058078 >> v2:172.21.15.116:3300/0 conn(0x7fe2a806e1e0 msgr2=0x7fe2a80d1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:34.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.617+0000 7fe2a7e00640 1 --2- 172.21.15.47:0/3387058078 >> v2:172.21.15.116:3300/0 conn(0x7fe2a806e1e0 0x7fe2a80d1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.617+0000 7fe2a7e00640 1 -- 172.21.15.47:0/3387058078 >> v1:172.21.15.47:6789/0 conn(0x7fe2a815e160 legacy=0x7fe2a806dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:34.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.617+0000 7fe2a7e00640 1 -- 172.21.15.47:0/3387058078 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a8162a00 con 0x7fe2a806f220 2024-02-19T15:22:34.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.617+0000 7fe2a6a00640 1 -- 172.21.15.47:0/3387058078 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025694931 0 0) 0x7fe2a8071a70 con 0x7fe2a815e160 2024-02-19T15:22:34.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.621+0000 7fe2a7e00640 1 --2- 172.21.15.47:0/3387058078 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2a806f220 0x7fe2a815e540 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe2a80c7ae0 tx=0x7fe2a001c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e33854558173dc6c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:34.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.621+0000 7fe2a6a00640 1 -- 172.21.15.47:0/3387058078 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a00a59c0 con 0x7fe2a806f220 2024-02-19T15:22:34.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.621+0000 7fe2a6a00640 1 -- 172.21.15.47:0/3387058078 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a00a5b20 con 0x7fe2a806f220 2024-02-19T15:22:34.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.621+0000 7fe2a6a00640 1 -- 172.21.15.47:0/3387058078 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a00ad440 con 0x7fe2a806f220 2024-02-19T15:22:34.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.621+0000 7fe2ae400640 1 -- 172.21.15.47:0/3387058078 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2a806f220 msgr2=0x7fe2a815e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:34.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.621+0000 7fe2ae400640 1 --2- 172.21.15.47:0/3387058078 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2a806f220 0x7fe2a815e540 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe2a80c7ae0 tx=0x7fe2a001c900 comp rx=0 tx=0).stop 2024-02-19T15:22:34.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.621+0000 7fe2ae400640 1 -- 172.21.15.47:0/3387058078 shutdown_connections 2024-02-19T15:22:34.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.621+0000 7fe2ae400640 1 --2- 172.21.15.47:0/3387058078 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2a806f220 0x7fe2a815e540 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.621+0000 7fe2ae400640 1 --2- 172.21.15.47:0/3387058078 >> v2:172.21.15.116:3300/0 conn(0x7fe2a806e1e0 0x7fe2a80d1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.621+0000 7fe2ae400640 1 -- 172.21.15.47:0/3387058078 >> 172.21.15.47:0/3387058078 conn(0x7fe2a80d0d30 msgr2=0x7fe2a80d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:34.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.621+0000 7fe2ae400640 1 -- 172.21.15.47:0/3387058078 shutdown_connections 2024-02-19T15:22:34.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.621+0000 7fe2ae400640 1 -- 172.21.15.47:0/3387058078 wait complete. 2024-02-19T15:22:34.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.621+0000 7fe2ae400640 1 Processor -- start 2024-02-19T15:22:34.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.621+0000 7fe2ae400640 1 -- start start 2024-02-19T15:22:34.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe2ae400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2a806e1e0 0x7fe2a8154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:34.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe2ae400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a806f220 0x7fe2a80b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:34.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe2ae400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2a815e160 0x7fe2a815a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:34.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe2ae400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a80af6e0 con 0x7fe2a815e160 2024-02-19T15:22:34.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe2ae400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a80b31a0 con 0x7fe2a806f220 2024-02-19T15:22:34.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe2ae400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a80af1b0 con 0x7fe2a806e1e0 2024-02-19T15:22:34.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe2a7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2a815e160 0x7fe2a815a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:34.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe2a7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2a815e160 0x7fe2a815a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44654/0 (socket says 172.21.15.47:44654) 2024-02-19T15:22:34.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe2a7e00640 1 -- 172.21.15.47:0/2166427550 learned_addr learned my addr 172.21.15.47:0/2166427550 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:34.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe2a7400640 1 --2- 172.21.15.47:0/2166427550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2a806e1e0 0x7fe2a8154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:34.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe29fe00640 1 --2- 172.21.15.47:0/2166427550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a806f220 0x7fe2a80b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:34.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe2a7e00640 1 -- 172.21.15.47:0/2166427550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2a806e1e0 msgr2=0x7fe2a8154030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:34.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe2a7e00640 1 --2- 172.21.15.47:0/2166427550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2a806e1e0 0x7fe2a8154030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe2a7e00640 1 -- 172.21.15.47:0/2166427550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a806f220 msgr2=0x7fe2a80b2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:34.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe2a7e00640 1 --2- 172.21.15.47:0/2166427550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a806f220 0x7fe2a80b2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe2a7e00640 1 -- 172.21.15.47:0/2166427550 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a0009a70 con 0x7fe2a815e160 2024-02-19T15:22:34.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe2a7e00640 1 --2- 172.21.15.47:0/2166427550 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2a815e160 0x7fe2a815a380 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe2a001cde0 tx=0x7fe2a00196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:34.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe29f400640 1 -- 172.21.15.47:0/2166427550 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a00bf070 con 0x7fe2a815e160 2024-02-19T15:22:34.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe29f400640 1 -- 172.21.15.47:0/2166427550 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a0009450 con 0x7fe2a815e160 2024-02-19T15:22:34.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe2ae400640 1 -- 172.21.15.47:0/2166427550 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a80af400 con 0x7fe2a815e160 2024-02-19T15:22:34.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe29f400640 1 -- 172.21.15.47:0/2166427550 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a00addd0 con 0x7fe2a815e160 2024-02-19T15:22:34.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe2ae400640 1 -- 172.21.15.47:0/2166427550 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a81592f0 con 0x7fe2a815e160 2024-02-19T15:22:34.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe2ae400640 1 -- 172.21.15.47:0/2166427550 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a8158e70 con 0x7fe2a815e160 2024-02-19T15:22:34.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.625+0000 7fe29f400640 1 -- 172.21.15.47:0/2166427550 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe2a00b4050 con 0x7fe2a815e160 2024-02-19T15:22:34.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.629+0000 7fe29f400640 1 --2- 172.21.15.47:0/2166427550 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe284066e40 0x7fe284069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:34.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.629+0000 7fe29f400640 1 -- 172.21.15.47:0/2166427550 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2a011f3c0 con 0x7fe2a815e160 2024-02-19T15:22:34.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.629+0000 7fe2a7400640 1 --2- 172.21.15.47:0/2166427550 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe284066e40 0x7fe284069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:34.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.629+0000 7fe2a7400640 1 --2- 172.21.15.47:0/2166427550 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe284066e40 0x7fe284069300 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe298000f70 tx=0x7fe298007df0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:34.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.633+0000 7fe29f400640 1 -- 172.21.15.47:0/2166427550 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe2a8158e70 con 0x7fe2a815e160 2024-02-19T15:22:34.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.793+0000 7fe2ae400640 1 -- 172.21.15.47:0/2166427550 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe2a8158e70 con 0x7fe2a815e160 2024-02-19T15:22:34.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.793+0000 7fe29f400640 1 -- 172.21.15.47:0/2166427550 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe2a8158e70 con 0x7fe2a815e160 2024-02-19T15:22:34.797 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:22:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.797+0000 7fe2ae400640 1 -- 172.21.15.47:0/2166427550 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe284066e40 msgr2=0x7fe284069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.797+0000 7fe2ae400640 1 --2- 172.21.15.47:0/2166427550 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe284066e40 0x7fe284069300 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe298000f70 tx=0x7fe298007df0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.797+0000 7fe2ae400640 1 -- 172.21.15.47:0/2166427550 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2a815e160 msgr2=0x7fe2a815a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.797+0000 7fe2ae400640 1 --2- 172.21.15.47:0/2166427550 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2a815e160 0x7fe2a815a380 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe2a001cde0 tx=0x7fe2a00196a0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.797+0000 7fe2ae400640 1 -- 172.21.15.47:0/2166427550 shutdown_connections 2024-02-19T15:22:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.797+0000 7fe2ae400640 1 --2- 172.21.15.47:0/2166427550 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe284066e40 0x7fe284069300 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.797+0000 7fe2ae400640 1 --2- 172.21.15.47:0/2166427550 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2a815e160 0x7fe2a815a380 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.797+0000 7fe2ae400640 1 --2- 172.21.15.47:0/2166427550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a806f220 0x7fe2a80b2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.797+0000 7fe2ae400640 1 --2- 172.21.15.47:0/2166427550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2a806e1e0 0x7fe2a8154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.797+0000 7fe2ae400640 1 -- 172.21.15.47:0/2166427550 >> 172.21.15.47:0/2166427550 conn(0x7fe2a80d0d30 msgr2=0x7fe2a80ae0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.797+0000 7fe2ae400640 1 -- 172.21.15.47:0/2166427550 shutdown_connections 2024-02-19T15:22:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.797+0000 7fe2ae400640 1 -- 172.21.15.47:0/2166427550 wait complete. 2024-02-19T15:22:34.817 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:22:34.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.957+0000 7f2f5aa00640 1 Processor -- start 2024-02-19T15:22:34.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.957+0000 7f2f5aa00640 1 -- start start 2024-02-19T15:22:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.957+0000 7f2f5aa00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2f540c7e80 0x7f2f540d0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.957+0000 7f2f5aa00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2f540c8260 0x7f2f540d0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.957+0000 7f2f5aa00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540cd750 con 0x7f2f540cfbf0 2024-02-19T15:22:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.957+0000 7f2f5aa00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f540ceee0 con 0x7f2f540c8260 2024-02-19T15:22:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.957+0000 7f2f5aa00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f540cdc60 con 0x7f2f540c7e80 2024-02-19T15:22:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.957+0000 7f2f4be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2f540c8260 0x7f2f540d0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.957+0000 7f2f4be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2f540c8260 0x7f2f540d0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44668/0 (socket says 172.21.15.47:44668) 2024-02-19T15:22:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.957+0000 7f2f4be00640 1 -- 172.21.15.47:0/4012317444 learned_addr learned my addr 172.21.15.47:0/4012317444 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.957+0000 7f2f53e00640 1 --2- 172.21.15.47:0/4012317444 >> v2:172.21.15.116:3300/0 conn(0x7f2f540c7e80 0x7f2f540d0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:34.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.957+0000 7f2f4be00640 1 -- 172.21.15.47:0/4012317444 >> v2:172.21.15.116:3300/0 conn(0x7f2f540c7e80 msgr2=0x7f2f540d0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:34.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.957+0000 7f2f4be00640 1 --2- 172.21.15.47:0/4012317444 >> v2:172.21.15.116:3300/0 conn(0x7f2f540c7e80 0x7f2f540d0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.957+0000 7f2f4be00640 1 -- 172.21.15.47:0/4012317444 >> v1:172.21.15.116:6789/0 conn(0x7f2f540cfbf0 legacy=0x7f2f540c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:34.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.957+0000 7f2f4be00640 1 -- 172.21.15.47:0/4012317444 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f540d2a40 con 0x7f2f540c8260 2024-02-19T15:22:34.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.961+0000 7f2f4be00640 1 --2- 172.21.15.47:0/4012317444 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2f540c8260 0x7f2f540d0650 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2f54071b30 tx=0x7f2f3c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b581e4b5efab835 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:34.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.961+0000 7f2f53400640 1 -- 172.21.15.47:0/4012317444 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c0aa650 con 0x7f2f540c8260 2024-02-19T15:22:34.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.961+0000 7f2f53400640 1 -- 172.21.15.47:0/4012317444 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3c0aa7b0 con 0x7f2f540c8260 2024-02-19T15:22:34.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.961+0000 7f2f53400640 1 -- 172.21.15.47:0/4012317444 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c0aab40 con 0x7f2f540c8260 2024-02-19T15:22:34.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.961+0000 7f2f5aa00640 1 -- 172.21.15.47:0/4012317444 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2f540c8260 msgr2=0x7f2f540d0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:34.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.961+0000 7f2f5aa00640 1 --2- 172.21.15.47:0/4012317444 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2f540c8260 0x7f2f540d0650 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2f54071b30 tx=0x7f2f3c01c4f0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.961+0000 7f2f5aa00640 1 -- 172.21.15.47:0/4012317444 shutdown_connections 2024-02-19T15:22:34.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.961+0000 7f2f5aa00640 1 --2- 172.21.15.47:0/4012317444 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2f540c8260 0x7f2f540d0650 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.961+0000 7f2f5aa00640 1 --2- 172.21.15.47:0/4012317444 >> v2:172.21.15.116:3300/0 conn(0x7f2f540c7e80 0x7f2f540d0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.961+0000 7f2f5aa00640 1 -- 172.21.15.47:0/4012317444 >> 172.21.15.47:0/4012317444 conn(0x7f2f540d1b30 msgr2=0x7f2f540d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:34.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.961+0000 7f2f5aa00640 1 -- 172.21.15.47:0/4012317444 shutdown_connections 2024-02-19T15:22:34.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.961+0000 7f2f5aa00640 1 -- 172.21.15.47:0/4012317444 wait complete. 2024-02-19T15:22:34.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f5aa00640 1 Processor -- start 2024-02-19T15:22:34.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f5aa00640 1 -- start start 2024-02-19T15:22:34.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f5aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f540c7e80 0x7f2f54148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:34.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f5aa00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f540c8260 0x7f2f5415a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:34.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f5aa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2f540cfbf0 0x7f2f5415ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:34.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f5aa00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f541f50f0 con 0x7f2f540cfbf0 2024-02-19T15:22:34.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f5aa00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f540ae760 con 0x7f2f540c7e80 2024-02-19T15:22:34.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f5aa00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f54159800 con 0x7f2f540c8260 2024-02-19T15:22:34.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f58a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2f540cfbf0 0x7f2f5415ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:34.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f53e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f540c7e80 0x7f2f54148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:34.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f53e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f540c7e80 0x7f2f54148e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:47274/0 (socket says 172.21.15.47:47274) 2024-02-19T15:22:34.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f53e00640 1 -- 172.21.15.47:0/2002930156 learned_addr learned my addr 172.21.15.47:0/2002930156 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:34.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f4be00640 1 --2- 172.21.15.47:0/2002930156 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f540c8260 0x7f2f5415a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:34.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f53e00640 1 -- 172.21.15.47:0/2002930156 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f540c8260 msgr2=0x7f2f5415a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:34.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f53e00640 1 --2- 172.21.15.47:0/2002930156 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f540c8260 0x7f2f5415a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f53e00640 1 -- 172.21.15.47:0/2002930156 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2f540cfbf0 msgr2=0x7f2f5415ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:34.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f53e00640 1 --2- 172.21.15.47:0/2002930156 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2f540cfbf0 0x7f2f5415ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:34.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f53e00640 1 -- 172.21.15.47:0/2002930156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3c0094d0 con 0x7f2f540c7e80 2024-02-19T15:22:34.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f53e00640 1 --2- 172.21.15.47:0/2002930156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f540c7e80 0x7f2f54148e00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400e870 tx=0x7f2f4400ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:34.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f50c00640 1 -- 172.21.15.47:0/2002930156 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f44009510 con 0x7f2f540c7e80 2024-02-19T15:22:34.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f50c00640 1 -- 172.21.15.47:0/2002930156 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4400eec0 con 0x7f2f540c7e80 2024-02-19T15:22:34.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f50c00640 1 -- 172.21.15.47:0/2002930156 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4400cb40 con 0x7f2f540c7e80 2024-02-19T15:22:34.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f5aa00640 1 -- 172.21.15.47:0/2002930156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f54159ae0 con 0x7f2f540c7e80 2024-02-19T15:22:34.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.965+0000 7f2f5aa00640 1 -- 172.21.15.47:0/2002930156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f540ae9b0 con 0x7f2f540c7e80 2024-02-19T15:22:34.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.969+0000 7f2f5aa00640 1 -- 172.21.15.47:0/2002930156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f540d1180 con 0x7f2f540c7e80 2024-02-19T15:22:34.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.969+0000 7f2f50c00640 1 -- 172.21.15.47:0/2002930156 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2f4400cca0 con 0x7f2f540c7e80 2024-02-19T15:22:34.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.969+0000 7f2f50c00640 1 --2- 172.21.15.47:0/2002930156 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2f30066e40 0x7f2f30069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:34.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.969+0000 7f2f50c00640 1 -- 172.21.15.47:0/2002930156 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f44084b40 con 0x7f2f540c7e80 2024-02-19T15:22:34.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.969+0000 7f2f4be00640 1 --2- 172.21.15.47:0/2002930156 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2f30066e40 0x7f2f30069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:34.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.969+0000 7f2f4be00640 1 --2- 172.21.15.47:0/2002930156 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2f30066e40 0x7f2f30069300 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c0ae000 tx=0x7f2f3c0b5040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:34.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:34.973+0000 7f2f50c00640 1 -- 172.21.15.47:0/2002930156 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2f44051940 con 0x7f2f540c7e80 2024-02-19T15:22:35.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.133+0000 7f2f5aa00640 1 -- 172.21.15.47:0/2002930156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2f54075360 con 0x7f2f540c7e80 2024-02-19T15:22:35.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.133+0000 7f2f50c00640 1 -- 172.21.15.47:0/2002930156 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2f440170a0 con 0x7f2f540c7e80 2024-02-19T15:22:35.137 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:22:35.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.137+0000 7f2f5aa00640 1 -- 172.21.15.47:0/2002930156 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2f30066e40 msgr2=0x7f2f30069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:35.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.137+0000 7f2f5aa00640 1 --2- 172.21.15.47:0/2002930156 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2f30066e40 0x7f2f30069300 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c0ae000 tx=0x7f2f3c0b5040 comp rx=0 tx=0).stop 2024-02-19T15:22:35.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.137+0000 7f2f5aa00640 1 -- 172.21.15.47:0/2002930156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f540c7e80 msgr2=0x7f2f54148e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:35.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.137+0000 7f2f5aa00640 1 --2- 172.21.15.47:0/2002930156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f540c7e80 0x7f2f54148e00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400e870 tx=0x7f2f4400ec70 comp rx=0 tx=0).stop 2024-02-19T15:22:35.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.137+0000 7f2f5aa00640 1 -- 172.21.15.47:0/2002930156 shutdown_connections 2024-02-19T15:22:35.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.137+0000 7f2f5aa00640 1 --2- 172.21.15.47:0/2002930156 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2f30066e40 0x7f2f30069300 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:35.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.137+0000 7f2f5aa00640 1 --2- 172.21.15.47:0/2002930156 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2f540cfbf0 0x7f2f5415ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:35.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.137+0000 7f2f5aa00640 1 --2- 172.21.15.47:0/2002930156 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f540c8260 0x7f2f5415a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:35.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.137+0000 7f2f5aa00640 1 --2- 172.21.15.47:0/2002930156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f540c7e80 0x7f2f54148e00 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:35.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.137+0000 7f2f5aa00640 1 -- 172.21.15.47:0/2002930156 >> 172.21.15.47:0/2002930156 conn(0x7f2f540d1b30 msgr2=0x7f2f540cde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:35.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.137+0000 7f2f5aa00640 1 -- 172.21.15.47:0/2002930156 shutdown_connections 2024-02-19T15:22:35.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.137+0000 7f2f5aa00640 1 -- 172.21.15.47:0/2002930156 wait complete. 2024-02-19T15:22:35.158 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-02-19T15:22:35.168 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:22:35.208 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:22:35.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.297+0000 7fb94a600640 1 Processor -- start 2024-02-19T15:22:35.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.297+0000 7fb94a600640 1 -- start start 2024-02-19T15:22:35.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.297+0000 7fb94a600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9440cd020 0x7fb9440ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:35.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.297+0000 7fb94a600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9440ce6e0 con 0x7fb94415dd20 2024-02-19T15:22:35.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.297+0000 7fb94a600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9440acec0 con 0x7fb94415e100 2024-02-19T15:22:35.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.297+0000 7fb94a600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94406c300 con 0x7fb9440cd020 2024-02-19T15:22:35.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.297+0000 7fb949200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb94415dd20 0x7fb9440cc910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38114/0 (socket says 172.21.15.47:38114) 2024-02-19T15:22:35.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.297+0000 7fb949200640 1 -- 172.21.15.47:0/1631529470 learned_addr learned my addr 172.21.15.47:0/1631529470 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:35.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.297+0000 7fb943e00640 1 --2- 172.21.15.47:0/1631529470 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9440cd020 0x7fb9440ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:35.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.297+0000 7fb943400640 1 -- 172.21.15.47:0/1631529470 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578818242 0 0) 0x7fb9440ce6e0 con 0x7fb94415dd20 2024-02-19T15:22:35.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.297+0000 7fb943400640 1 -- 172.21.15.47:0/1631529470 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb92c003530 con 0x7fb94415dd20 2024-02-19T15:22:35.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.297+0000 7fb943400640 1 -- 172.21.15.47:0/1631529470 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438428135 0 0) 0x7fb9440acec0 con 0x7fb94415e100 2024-02-19T15:22:35.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.297+0000 7fb943400640 1 -- 172.21.15.47:0/1631529470 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9440ce6e0 con 0x7fb94415e100 2024-02-19T15:22:35.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.297+0000 7fb943e00640 1 -- 172.21.15.47:0/1631529470 >> v1:172.21.15.116:6789/0 conn(0x7fb94415e100 legacy=0x7fb9440ad3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:35.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.297+0000 7fb943e00640 1 -- 172.21.15.47:0/1631529470 >> v1:172.21.15.47:6789/0 conn(0x7fb94415dd20 legacy=0x7fb9440cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:35.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.297+0000 7fb943e00640 1 -- 172.21.15.47:0/1631529470 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94406c5e0 con 0x7fb9440cd020 2024-02-19T15:22:35.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.301+0000 7fb943e00640 1 --2- 172.21.15.47:0/1631529470 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9440cd020 0x7fb9440ca6e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb9300097f0 tx=0x7fb93001c390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8698c99f1758fa91 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:35.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.301+0000 7fb943400640 1 -- 172.21.15.47:0/1631529470 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9300ac410 con 0x7fb9440cd020 2024-02-19T15:22:35.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.301+0000 7fb943400640 1 -- 172.21.15.47:0/1631529470 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9300ac570 con 0x7fb9440cd020 2024-02-19T15:22:35.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.301+0000 7fb943400640 1 -- 172.21.15.47:0/1631529470 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9300ac8e0 con 0x7fb9440cd020 2024-02-19T15:22:35.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.301+0000 7fb94a600640 1 -- 172.21.15.47:0/1631529470 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9440cd020 msgr2=0x7fb9440ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:35.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.301+0000 7fb94a600640 1 --2- 172.21.15.47:0/1631529470 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9440cd020 0x7fb9440ca6e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb9300097f0 tx=0x7fb93001c390 comp rx=0 tx=0).stop 2024-02-19T15:22:35.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.301+0000 7fb94a600640 1 -- 172.21.15.47:0/1631529470 shutdown_connections 2024-02-19T15:22:35.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.301+0000 7fb94a600640 1 --2- 172.21.15.47:0/1631529470 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9440cd020 0x7fb9440ca6e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:35.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.301+0000 7fb94a600640 1 -- 172.21.15.47:0/1631529470 >> 172.21.15.47:0/1631529470 conn(0x7fb9440b1be0 msgr2=0x7fb9440b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:35.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.301+0000 7fb94a600640 1 -- 172.21.15.47:0/1631529470 shutdown_connections 2024-02-19T15:22:35.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.301+0000 7fb94a600640 1 -- 172.21.15.47:0/1631529470 wait complete. 2024-02-19T15:22:35.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb94a600640 1 Processor -- start 2024-02-19T15:22:35.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb94a600640 1 -- start start 2024-02-19T15:22:35.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb94a600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9440cd020 0x7fb9440c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:35.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb94a600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb94415dd20 0x7fb9440c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:35.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb94a600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb94415e100 0x7fb94409aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:35.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb94a600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94415f560 con 0x7fb9440cd020 2024-02-19T15:22:35.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb94a600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9440c8d80 con 0x7fb94415dd20 2024-02-19T15:22:35.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb94a600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9440c9fd0 con 0x7fb94415e100 2024-02-19T15:22:35.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb949200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9440cd020 0x7fb9440c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:35.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb949200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9440cd020 0x7fb9440c7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44692/0 (socket says 172.21.15.47:44692) 2024-02-19T15:22:35.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb949200640 1 -- 172.21.15.47:0/1252467944 learned_addr learned my addr 172.21.15.47:0/1252467944 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:35.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb943e00640 1 --2- 172.21.15.47:0/1252467944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb94415dd20 0x7fb9440c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:35.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb949c00640 1 --2- 172.21.15.47:0/1252467944 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb94415e100 0x7fb94409aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:35.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb949200640 1 -- 172.21.15.47:0/1252467944 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb94415e100 msgr2=0x7fb94409aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:35.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb949200640 1 --2- 172.21.15.47:0/1252467944 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb94415e100 0x7fb94409aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:35.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb949200640 1 -- 172.21.15.47:0/1252467944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb94415dd20 msgr2=0x7fb9440c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:35.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb949200640 1 --2- 172.21.15.47:0/1252467944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb94415dd20 0x7fb9440c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:35.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb949200640 1 -- 172.21.15.47:0/1252467944 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb930009420 con 0x7fb9440cd020 2024-02-19T15:22:35.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb949200640 1 --2- 172.21.15.47:0/1252467944 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9440cd020 0x7fb9440c7b70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb93800c420 tx=0x7fb93800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:35.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb940c00640 1 -- 172.21.15.47:0/1252467944 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb938005cf0 con 0x7fb9440cd020 2024-02-19T15:22:35.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb940c00640 1 -- 172.21.15.47:0/1252467944 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb938005e50 con 0x7fb9440cd020 2024-02-19T15:22:35.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb94a600640 1 -- 172.21.15.47:0/1252467944 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9440c8200 con 0x7fb9440cd020 2024-02-19T15:22:35.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb940c00640 1 -- 172.21.15.47:0/1252467944 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb938007310 con 0x7fb9440cd020 2024-02-19T15:22:35.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb94a600640 1 -- 172.21.15.47:0/1252467944 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9440c87c0 con 0x7fb9440cd020 2024-02-19T15:22:35.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.305+0000 7fb94a600640 1 -- 172.21.15.47:0/1252467944 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94415f6d0 con 0x7fb9440cd020 2024-02-19T15:22:35.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.309+0000 7fb940c00640 1 -- 172.21.15.47:0/1252467944 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb938007470 con 0x7fb9440cd020 2024-02-19T15:22:35.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.309+0000 7fb940c00640 1 --2- 172.21.15.47:0/1252467944 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb924066e40 0x7fb924069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:35.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.309+0000 7fb940c00640 1 -- 172.21.15.47:0/1252467944 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb938083ff0 con 0x7fb9440cd020 2024-02-19T15:22:35.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.309+0000 7fb943e00640 1 --2- 172.21.15.47:0/1252467944 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb924066e40 0x7fb924069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:35.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.309+0000 7fb943e00640 1 --2- 172.21.15.47:0/1252467944 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb924066e40 0x7fb924069300 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb9300ae000 tx=0x7fb930008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:35.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.313+0000 7fb940c00640 1 -- 172.21.15.47:0/1252467944 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb938050e80 con 0x7fb9440cd020 2024-02-19T15:22:35.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.537+0000 7fb94a600640 1 -- 172.21.15.47:0/1252467944 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7fb9440ca190 con 0x7fb9440cd020 2024-02-19T15:22:35.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.949+0000 7fb940c00640 1 -- 172.21.15.47:0/1252467944 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v54) v1 ==== 135+0+0 (secure 0 0 0) 0x7fb938054540 con 0x7fb9440cd020 2024-02-19T15:22:35.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.953+0000 7fb94a600640 1 -- 172.21.15.47:0/1252467944 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb924066e40 msgr2=0x7fb924069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:35.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.953+0000 7fb94a600640 1 --2- 172.21.15.47:0/1252467944 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb924066e40 0x7fb924069300 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb9300ae000 tx=0x7fb930008040 comp rx=0 tx=0).stop 2024-02-19T15:22:35.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.953+0000 7fb94a600640 1 -- 172.21.15.47:0/1252467944 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9440cd020 msgr2=0x7fb9440c7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:35.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.953+0000 7fb94a600640 1 --2- 172.21.15.47:0/1252467944 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9440cd020 0x7fb9440c7b70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb93800c420 tx=0x7fb93800c8e0 comp rx=0 tx=0).stop 2024-02-19T15:22:35.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.953+0000 7fb94a600640 1 -- 172.21.15.47:0/1252467944 shutdown_connections 2024-02-19T15:22:35.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.953+0000 7fb94a600640 1 --2- 172.21.15.47:0/1252467944 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb924066e40 0x7fb924069300 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:35.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.953+0000 7fb94a600640 1 --2- 172.21.15.47:0/1252467944 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb94415e100 0x7fb94409aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:35.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.953+0000 7fb94a600640 1 --2- 172.21.15.47:0/1252467944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb94415dd20 0x7fb9440c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:35.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.953+0000 7fb94a600640 1 --2- 172.21.15.47:0/1252467944 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9440cd020 0x7fb9440c7b70 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:35.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.953+0000 7fb94a600640 1 -- 172.21.15.47:0/1252467944 >> 172.21.15.47:0/1252467944 conn(0x7fb9440b1be0 msgr2=0x7fb9440cb430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:35.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.953+0000 7fb94a600640 1 -- 172.21.15.47:0/1252467944 shutdown_connections 2024-02-19T15:22:35.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:35.953+0000 7fb94a600640 1 -- 172.21.15.47:0/1252467944 wait complete. 2024-02-19T15:22:35.976 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-19T15:22:36.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.117+0000 7fef61800640 1 Processor -- start 2024-02-19T15:22:36.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.117+0000 7fef61800640 1 -- start start 2024-02-19T15:22:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.121+0000 7fef61800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fef5c15bd90 0x7fef5c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.121+0000 7fef61800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c0cb490 0x7fef5c072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.121+0000 7fef61800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5c15fc90 con 0x7fef5c072680 2024-02-19T15:22:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.121+0000 7fef61800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef5c15ff70 con 0x7fef5c15bd90 2024-02-19T15:22:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.121+0000 7fef61800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c160250 con 0x7fef5c0cb490 2024-02-19T15:22:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.121+0000 7fef5aa00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fef5c15bd90 0x7fef5c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.121+0000 7fef5aa00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fef5c15bd90 0x7fef5c070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44694/0 (socket says 172.21.15.47:44694) 2024-02-19T15:22:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.121+0000 7fef5aa00640 1 -- 172.21.15.47:0/1198121885 learned_addr learned my addr 172.21.15.47:0/1198121885 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.121+0000 7fef5a000640 1 --2- 172.21.15.47:0/1198121885 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c0cb490 0x7fef5c072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.121+0000 7fef5aa00640 1 -- 172.21.15.47:0/1198121885 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c0cb490 msgr2=0x7fef5c072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.121+0000 7fef5aa00640 1 --2- 172.21.15.47:0/1198121885 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c0cb490 0x7fef5c072140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:36.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.121+0000 7fef5aa00640 1 -- 172.21.15.47:0/1198121885 >> v1:172.21.15.116:6789/0 conn(0x7fef5c072680 legacy=0x7fef5c0ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:36.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.121+0000 7fef5aa00640 1 -- 172.21.15.47:0/1198121885 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5c160530 con 0x7fef5c15bd90 2024-02-19T15:22:36.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.121+0000 7fef5aa00640 1 --2- 172.21.15.47:0/1198121885 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c15bd90 0x7fef5c070960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fef4c009870 tx=0x7fef4c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f38c0c6d10e52ac server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:36.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.121+0000 7fef59600640 1 -- 172.21.15.47:0/1198121885 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4c004230 con 0x7fef5c15bd90 2024-02-19T15:22:36.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.121+0000 7fef59600640 1 -- 172.21.15.47:0/1198121885 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4c0a7050 con 0x7fef5c15bd90 2024-02-19T15:22:36.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.121+0000 7fef59600640 1 -- 172.21.15.47:0/1198121885 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4c003620 con 0x7fef5c15bd90 2024-02-19T15:22:36.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef61800640 1 -- 172.21.15.47:0/1198121885 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c15bd90 msgr2=0x7fef5c070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:36.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef61800640 1 --2- 172.21.15.47:0/1198121885 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c15bd90 0x7fef5c070960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fef4c009870 tx=0x7fef4c01c400 comp rx=0 tx=0).stop 2024-02-19T15:22:36.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef61800640 1 -- 172.21.15.47:0/1198121885 shutdown_connections 2024-02-19T15:22:36.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef61800640 1 --2- 172.21.15.47:0/1198121885 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c0cb490 0x7fef5c072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:36.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef61800640 1 --2- 172.21.15.47:0/1198121885 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c15bd90 0x7fef5c070960 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:36.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef61800640 1 -- 172.21.15.47:0/1198121885 >> 172.21.15.47:0/1198121885 conn(0x7fef5c0cfc20 msgr2=0x7fef5c0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:36.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef61800640 1 -- 172.21.15.47:0/1198121885 shutdown_connections 2024-02-19T15:22:36.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef61800640 1 -- 172.21.15.47:0/1198121885 wait complete. 2024-02-19T15:22:36.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef61800640 1 Processor -- start 2024-02-19T15:22:36.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef61800640 1 -- start start 2024-02-19T15:22:36.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef61800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c072680 0x7fef5c0b4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:36.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef61800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c0cb490 0x7fef5c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:36.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef61800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c15bd90 0x7fef5c15d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:36.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef61800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c15e6a0 con 0x7fef5c0cb490 2024-02-19T15:22:36.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef61800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c15e0e0 con 0x7fef5c072680 2024-02-19T15:22:36.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef61800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c15e3c0 con 0x7fef5c15bd90 2024-02-19T15:22:36.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef5a000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c0cb490 0x7fef5c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:36.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef5a000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c0cb490 0x7fef5c0b4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44698/0 (socket says 172.21.15.47:44698) 2024-02-19T15:22:36.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef5aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c072680 0x7fef5c0b4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:36.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef5a000640 1 -- 172.21.15.47:0/2552800206 learned_addr learned my addr 172.21.15.47:0/2552800206 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:36.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef5b400640 1 --2- 172.21.15.47:0/2552800206 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c15bd90 0x7fef5c15d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:36.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef5a000640 1 -- 172.21.15.47:0/2552800206 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c15bd90 msgr2=0x7fef5c15d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:36.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef5a000640 1 --2- 172.21.15.47:0/2552800206 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c15bd90 0x7fef5c15d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:36.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef5a000640 1 -- 172.21.15.47:0/2552800206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c072680 msgr2=0x7fef5c0b4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:36.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef5a000640 1 --2- 172.21.15.47:0/2552800206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c072680 0x7fef5c0b4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:36.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.125+0000 7fef5a000640 1 -- 172.21.15.47:0/2552800206 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4c0094d0 con 0x7fef5c0cb490 2024-02-19T15:22:36.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.129+0000 7fef5a000640 1 --2- 172.21.15.47:0/2552800206 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c0cb490 0x7fef5c0b4d90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fef4800e870 tx=0x7fef4800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:36.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.129+0000 7fef52a00640 1 -- 172.21.15.47:0/2552800206 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef48009510 con 0x7fef5c0cb490 2024-02-19T15:22:36.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.129+0000 7fef52a00640 1 -- 172.21.15.47:0/2552800206 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4800eec0 con 0x7fef5c0cb490 2024-02-19T15:22:36.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.129+0000 7fef52a00640 1 -- 172.21.15.47:0/2552800206 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4800cb40 con 0x7fef5c0cb490 2024-02-19T15:22:36.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.129+0000 7fef61800640 1 -- 172.21.15.47:0/2552800206 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef5c0ac990 con 0x7fef5c0cb490 2024-02-19T15:22:36.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.129+0000 7fef61800640 1 -- 172.21.15.47:0/2552800206 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5c15b480 con 0x7fef5c0cb490 2024-02-19T15:22:36.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.129+0000 7fef61800640 1 -- 172.21.15.47:0/2552800206 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef5c075360 con 0x7fef5c0cb490 2024-02-19T15:22:36.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.129+0000 7fef52a00640 1 -- 172.21.15.47:0/2552800206 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fef4800cca0 con 0x7fef5c0cb490 2024-02-19T15:22:36.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.129+0000 7fef52a00640 1 --2- 172.21.15.47:0/2552800206 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef3c066e40 0x7fef3c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:36.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.129+0000 7fef52a00640 1 -- 172.21.15.47:0/2552800206 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef48084ab0 con 0x7fef5c0cb490 2024-02-19T15:22:36.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.129+0000 7fef5aa00640 1 --2- 172.21.15.47:0/2552800206 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef3c066e40 0x7fef3c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:36.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.133+0000 7fef5aa00640 1 --2- 172.21.15.47:0/2552800206 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef3c066e40 0x7fef3c069300 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fef4c0ad000 tx=0x7fef4c0a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:36.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.133+0000 7fef52a00640 1 -- 172.21.15.47:0/2552800206 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fef48051940 con 0x7fef5c0cb490 2024-02-19T15:22:36.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.297+0000 7fef61800640 1 -- 172.21.15.47:0/2552800206 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fef5c15f190 con 0x7fef5c0cb490 2024-02-19T15:22:36.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.301+0000 7fef52a00640 1 -- 172.21.15.47:0/2552800206 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fef480170c0 con 0x7fef5c0cb490 2024-02-19T15:22:36.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.301+0000 7fef61800640 1 -- 172.21.15.47:0/2552800206 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef3c066e40 msgr2=0x7fef3c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:36.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.301+0000 7fef61800640 1 --2- 172.21.15.47:0/2552800206 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef3c066e40 0x7fef3c069300 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fef4c0ad000 tx=0x7fef4c0a6000 comp rx=0 tx=0).stop 2024-02-19T15:22:36.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.301+0000 7fef61800640 1 -- 172.21.15.47:0/2552800206 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c0cb490 msgr2=0x7fef5c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:36.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.301+0000 7fef61800640 1 --2- 172.21.15.47:0/2552800206 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c0cb490 0x7fef5c0b4d90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fef4800e870 tx=0x7fef4800ec70 comp rx=0 tx=0).stop 2024-02-19T15:22:36.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.305+0000 7fef61800640 1 -- 172.21.15.47:0/2552800206 shutdown_connections 2024-02-19T15:22:36.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.305+0000 7fef61800640 1 --2- 172.21.15.47:0/2552800206 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef3c066e40 0x7fef3c069300 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:36.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.305+0000 7fef61800640 1 --2- 172.21.15.47:0/2552800206 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c15bd90 0x7fef5c15d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:36.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.305+0000 7fef61800640 1 --2- 172.21.15.47:0/2552800206 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c0cb490 0x7fef5c0b4d90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:36.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.305+0000 7fef61800640 1 --2- 172.21.15.47:0/2552800206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c072680 0x7fef5c0b4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:36.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.305+0000 7fef61800640 1 -- 172.21.15.47:0/2552800206 >> 172.21.15.47:0/2552800206 conn(0x7fef5c0cfc20 msgr2=0x7fef5c0b1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:36.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.305+0000 7fef61800640 1 -- 172.21.15.47:0/2552800206 shutdown_connections 2024-02-19T15:22:36.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.305+0000 7fef61800640 1 -- 172.21.15.47:0/2552800206 wait complete. 2024-02-19T15:22:36.326 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T15:22:36.326 DEBUG:teuthology.orchestra.run.smithi047:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:22:36.412 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:22:36.457 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T15:22:36.457 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:22:36.457 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:22:36.465 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:22:36.465 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:22:36.514 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:22:36.514 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:22:36.566 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:36.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.749+0000 7f5d19800640 1 Processor -- start 2024-02-19T15:22:36.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.749+0000 7f5d19800640 1 -- start start 2024-02-19T15:22:36.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.753+0000 7f5d19800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5d14071350 0x7f5d14165e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:36.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.753+0000 7f5d19800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d140718c0 con 0x7f5d1415d6e0 2024-02-19T15:22:36.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.753+0000 7f5d19800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d14071ba0 con 0x7f5d1415b1a0 2024-02-19T15:22:36.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.753+0000 7f5d19800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d14071e80 con 0x7f5d14071350 2024-02-19T15:22:36.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.753+0000 7f5d0a000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5d1415d6e0 0x7f5d14162740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38120/0 (socket says 172.21.15.47:38120) 2024-02-19T15:22:36.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.753+0000 7f5d0a000640 1 -- 172.21.15.47:0/557239464 learned_addr learned my addr 172.21.15.47:0/557239464 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:36.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.753+0000 7f5d13400640 1 --2- 172.21.15.47:0/557239464 >> v2:172.21.15.47:3300/0 conn(0x7f5d14071350 0x7f5d14165e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:36.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.753+0000 7f5d13400640 1 -- 172.21.15.47:0/557239464 >> v1:172.21.15.116:6789/0 conn(0x7f5d1415b1a0 legacy=0x7f5d1415b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:36.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.753+0000 7f5d13400640 1 -- 172.21.15.47:0/557239464 >> v1:172.21.15.47:6789/0 conn(0x7f5d1415d6e0 legacy=0x7f5d14162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:36.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.753+0000 7f5d13400640 1 -- 172.21.15.47:0/557239464 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d14072160 con 0x7f5d14071350 2024-02-19T15:22:36.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.753+0000 7f5d13400640 1 --2- 172.21.15.47:0/557239464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d14071350 0x7f5d14165e70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c009dc0 tx=0x7f5d0c01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5471f8d13b467e47 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:36.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.753+0000 7f5d12000640 1 -- 172.21.15.47:0/557239464 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c0a5a30 con 0x7f5d14071350 2024-02-19T15:22:36.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.753+0000 7f5d12000640 1 -- 172.21.15.47:0/557239464 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0c0a5b90 con 0x7f5d14071350 2024-02-19T15:22:36.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.753+0000 7f5d12000640 1 -- 172.21.15.47:0/557239464 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c0ad440 con 0x7f5d14071350 2024-02-19T15:22:36.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d19800640 1 -- 172.21.15.47:0/557239464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d14071350 msgr2=0x7f5d14165e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:36.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d19800640 1 --2- 172.21.15.47:0/557239464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d14071350 0x7f5d14165e70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c009dc0 tx=0x7f5d0c01c950 comp rx=0 tx=0).stop 2024-02-19T15:22:36.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d19800640 1 -- 172.21.15.47:0/557239464 shutdown_connections 2024-02-19T15:22:36.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d19800640 1 --2- 172.21.15.47:0/557239464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d14071350 0x7f5d14165e70 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:36.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d19800640 1 -- 172.21.15.47:0/557239464 >> 172.21.15.47:0/557239464 conn(0x7f5d140b16a0 msgr2=0x7f5d140b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:36.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d19800640 1 -- 172.21.15.47:0/557239464 shutdown_connections 2024-02-19T15:22:36.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d19800640 1 -- 172.21.15.47:0/557239464 wait complete. 2024-02-19T15:22:36.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d19800640 1 Processor -- start 2024-02-19T15:22:36.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d19800640 1 -- start start 2024-02-19T15:22:36.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d19800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d14071350 0x7f5d14161490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:36.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d19800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d1415b1a0 0x7f5d14147020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:36.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d19800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d1415d6e0 0x7f5d14147560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:36.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d19800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d14163cf0 con 0x7f5d1415b1a0 2024-02-19T15:22:36.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d19800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d141633c0 con 0x7f5d1415d6e0 2024-02-19T15:22:36.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d19800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d141636a0 con 0x7f5d14071350 2024-02-19T15:22:36.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d0a000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d1415b1a0 0x7f5d14147020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:36.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d0a000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d1415b1a0 0x7f5d14147020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44720/0 (socket says 172.21.15.47:44720) 2024-02-19T15:22:36.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d0a000640 1 -- 172.21.15.47:0/2056174818 learned_addr learned my addr 172.21.15.47:0/2056174818 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:36.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d12a00640 1 --2- 172.21.15.47:0/2056174818 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d14071350 0x7f5d14161490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:36.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d13400640 1 --2- 172.21.15.47:0/2056174818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d1415d6e0 0x7f5d14147560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:36.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d0a000640 1 -- 172.21.15.47:0/2056174818 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d14071350 msgr2=0x7f5d14161490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:36.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d0a000640 1 --2- 172.21.15.47:0/2056174818 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d14071350 0x7f5d14161490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:36.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d0a000640 1 -- 172.21.15.47:0/2056174818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d1415d6e0 msgr2=0x7f5d14147560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:36.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d0a000640 1 --2- 172.21.15.47:0/2056174818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d1415d6e0 0x7f5d14147560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:36.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.757+0000 7f5d0a000640 1 -- 172.21.15.47:0/2056174818 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0c009a70 con 0x7f5d1415b1a0 2024-02-19T15:22:36.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.761+0000 7f5d0a000640 1 --2- 172.21.15.47:0/2056174818 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d1415b1a0 0x7f5d14147020 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00c4d0 tx=0x7f5cfc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:36.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.761+0000 7f5d0b400640 1 -- 172.21.15.47:0/2056174818 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc005cf0 con 0x7f5d1415b1a0 2024-02-19T15:22:36.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.761+0000 7f5d0b400640 1 -- 172.21.15.47:0/2056174818 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cfc005e50 con 0x7f5d1415b1a0 2024-02-19T15:22:36.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.761+0000 7f5d0b400640 1 -- 172.21.15.47:0/2056174818 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc007270 con 0x7f5d1415b1a0 2024-02-19T15:22:36.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.761+0000 7f5d19800640 1 -- 172.21.15.47:0/2056174818 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d14163980 con 0x7f5d1415b1a0 2024-02-19T15:22:36.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.761+0000 7f5d19800640 1 -- 172.21.15.47:0/2056174818 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d140b4060 con 0x7f5d1415b1a0 2024-02-19T15:22:36.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.761+0000 7f5d19800640 1 -- 172.21.15.47:0/2056174818 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d14163e60 con 0x7f5d1415b1a0 2024-02-19T15:22:36.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.761+0000 7f5d0b400640 1 -- 172.21.15.47:0/2056174818 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5cfc0073d0 con 0x7f5d1415b1a0 2024-02-19T15:22:36.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.761+0000 7f5d0b400640 1 --2- 172.21.15.47:0/2056174818 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5cf0066e40 0x7f5cf0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:36.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.765+0000 7f5d0b400640 1 -- 172.21.15.47:0/2056174818 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5cfc083f80 con 0x7f5d1415b1a0 2024-02-19T15:22:36.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.765+0000 7f5d12a00640 1 --2- 172.21.15.47:0/2056174818 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5cf0066e40 0x7f5cf0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:36.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.765+0000 7f5d12a00640 1 --2- 172.21.15.47:0/2056174818 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5cf0066e40 0x7f5cf0069300 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5d04007fd0 tx=0x7f5d04009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:36.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.769+0000 7f5d0b400640 1 -- 172.21.15.47:0/2056174818 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f5cfc050cf0 con 0x7f5d1415b1a0 2024-02-19T15:22:36.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.933+0000 7f5d19800640 1 -- 172.21.15.47:0/2056174818 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d14163b10 con 0x7f5d1415b1a0 2024-02-19T15:22:36.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.933+0000 7f5d0b400640 1 -- 172.21.15.47:0/2056174818 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f5cfc0543b0 con 0x7f5d1415b1a0 2024-02-19T15:22:36.937 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:36.937 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":21,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:06.843155+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:03.091360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:04.222965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:36.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.937+0000 7f5d19800640 1 -- 172.21.15.47:0/2056174818 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5cf0066e40 msgr2=0x7f5cf0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:36.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.937+0000 7f5d19800640 1 --2- 172.21.15.47:0/2056174818 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5cf0066e40 0x7f5cf0069300 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5d04007fd0 tx=0x7f5d04009040 comp rx=0 tx=0).stop 2024-02-19T15:22:36.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.937+0000 7f5d19800640 1 -- 172.21.15.47:0/2056174818 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d1415b1a0 msgr2=0x7f5d14147020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:36.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.937+0000 7f5d19800640 1 --2- 172.21.15.47:0/2056174818 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d1415b1a0 0x7f5d14147020 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00c4d0 tx=0x7f5cfc00c990 comp rx=0 tx=0).stop 2024-02-19T15:22:36.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.937+0000 7f5d19800640 1 -- 172.21.15.47:0/2056174818 shutdown_connections 2024-02-19T15:22:36.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.937+0000 7f5d19800640 1 --2- 172.21.15.47:0/2056174818 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5cf0066e40 0x7f5cf0069300 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:36.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.937+0000 7f5d19800640 1 --2- 172.21.15.47:0/2056174818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d1415d6e0 0x7f5d14147560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:36.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.937+0000 7f5d19800640 1 --2- 172.21.15.47:0/2056174818 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d1415b1a0 0x7f5d14147020 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:36.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.937+0000 7f5d19800640 1 --2- 172.21.15.47:0/2056174818 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d14071350 0x7f5d14161490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:36.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.937+0000 7f5d19800640 1 -- 172.21.15.47:0/2056174818 >> 172.21.15.47:0/2056174818 conn(0x7f5d140b16a0 msgr2=0x7f5d140cf940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:36.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.937+0000 7f5d19800640 1 -- 172.21.15.47:0/2056174818 shutdown_connections 2024-02-19T15:22:36.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:36.937+0000 7f5d19800640 1 -- 172.21.15.47:0/2056174818 wait complete. 2024-02-19T15:22:36.958 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:22:03.091360+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:22:04.222965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:22:36.958 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:22:37.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.097+0000 7f2752000640 1 Processor -- start 2024-02-19T15:22:37.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.097+0000 7f2752000640 1 -- start start 2024-02-19T15:22:37.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.097+0000 7f2752000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f274c06e360 0x7f274c0cb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:37.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.097+0000 7f2752000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f274c0cd880 0x7f274c0cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:37.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.097+0000 7f2752000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274c0cb540 con 0x7f274c15af10 2024-02-19T15:22:37.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.097+0000 7f2752000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f274c071c70 con 0x7f274c0cd880 2024-02-19T15:22:37.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.097+0000 7f2752000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f274c06cef0 con 0x7f274c06e360 2024-02-19T15:22:37.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.097+0000 7f274b400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f274c15af10 0x7f274c0ca8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38126/0 (socket says 172.21.15.47:38126) 2024-02-19T15:22:37.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.097+0000 7f274b400640 1 -- 172.21.15.47:0/2616839325 learned_addr learned my addr 172.21.15.47:0/2616839325 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:37.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.097+0000 7f274be00640 1 --2- 172.21.15.47:0/2616839325 >> v2:172.21.15.47:3300/0 conn(0x7f274c0cd880 0x7f274c0cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:37.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.097+0000 7f2743e00640 1 --2- 172.21.15.47:0/2616839325 >> v2:172.21.15.116:3300/0 conn(0x7f274c06e360 0x7f274c0cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:37.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.101+0000 7f2743e00640 1 -- 172.21.15.47:0/2616839325 >> v2:172.21.15.47:3300/0 conn(0x7f274c0cd880 msgr2=0x7f274c0cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:37.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.101+0000 7f2743e00640 1 --2- 172.21.15.47:0/2616839325 >> v2:172.21.15.47:3300/0 conn(0x7f274c0cd880 0x7f274c0cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.101+0000 7f2743e00640 1 -- 172.21.15.47:0/2616839325 >> v1:172.21.15.47:6789/0 conn(0x7f274c15af10 legacy=0x7f274c0ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:37.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.101+0000 7f2743e00640 1 -- 172.21.15.47:0/2616839325 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274c0ca610 con 0x7f274c06e360 2024-02-19T15:22:37.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.101+0000 7f274aa00640 1 -- 172.21.15.47:0/2616839325 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441860171 0 0) 0x7f274c0cb540 con 0x7f274c15af10 2024-02-19T15:22:37.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.101+0000 7f2743e00640 1 --2- 172.21.15.47:0/2616839325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f274c06e360 0x7f274c0cb000 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f274c0b27b0 tx=0x7f273401c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e7ab6453abeb1eb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:37.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.101+0000 7f274aa00640 1 -- 172.21.15.47:0/2616839325 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27340ac620 con 0x7f274c06e360 2024-02-19T15:22:37.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.101+0000 7f274aa00640 1 -- 172.21.15.47:0/2616839325 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27340ac780 con 0x7f274c06e360 2024-02-19T15:22:37.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.101+0000 7f274aa00640 1 -- 172.21.15.47:0/2616839325 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27340acb10 con 0x7f274c06e360 2024-02-19T15:22:37.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.101+0000 7f2752000640 1 -- 172.21.15.47:0/2616839325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f274c06e360 msgr2=0x7f274c0cb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:37.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.101+0000 7f2752000640 1 --2- 172.21.15.47:0/2616839325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f274c06e360 0x7f274c0cb000 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f274c0b27b0 tx=0x7f273401c3c0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.101+0000 7f2752000640 1 -- 172.21.15.47:0/2616839325 shutdown_connections 2024-02-19T15:22:37.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.101+0000 7f2752000640 1 --2- 172.21.15.47:0/2616839325 >> v2:172.21.15.47:3300/0 conn(0x7f274c0cd880 0x7f274c0cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.101+0000 7f2752000640 1 --2- 172.21.15.47:0/2616839325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f274c06e360 0x7f274c0cb000 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.101+0000 7f2752000640 1 -- 172.21.15.47:0/2616839325 >> 172.21.15.47:0/2616839325 conn(0x7f274c0cfc20 msgr2=0x7f274c0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:37.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f2752000640 1 -- 172.21.15.47:0/2616839325 shutdown_connections 2024-02-19T15:22:37.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f2752000640 1 -- 172.21.15.47:0/2616839325 wait complete. 2024-02-19T15:22:37.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f2752000640 1 Processor -- start 2024-02-19T15:22:37.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f2752000640 1 -- start start 2024-02-19T15:22:37.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f2752000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f274c06e360 0x7f274c151f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:37.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f2752000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f274c0cd880 0x7f274c0c3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:37.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f2752000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f274c15af10 0x7f274c0c5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:37.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f2752000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274c152470 con 0x7f274c06e360 2024-02-19T15:22:37.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f2752000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274c0c52c0 con 0x7f274c0cd880 2024-02-19T15:22:37.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f2752000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f274c0c46b0 con 0x7f274c15af10 2024-02-19T15:22:37.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f274b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f274c06e360 0x7f274c151f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:37.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f274b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f274c06e360 0x7f274c151f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44734/0 (socket says 172.21.15.47:44734) 2024-02-19T15:22:37.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f274b400640 1 -- 172.21.15.47:0/4268337908 learned_addr learned my addr 172.21.15.47:0/4268337908 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:37.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f274be00640 1 --2- 172.21.15.47:0/4268337908 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f274c15af10 0x7f274c0c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:37.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f2743e00640 1 --2- 172.21.15.47:0/4268337908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f274c0cd880 0x7f274c0c3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:37.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f274b400640 1 -- 172.21.15.47:0/4268337908 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f274c15af10 msgr2=0x7f274c0c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:37.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f274b400640 1 --2- 172.21.15.47:0/4268337908 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f274c15af10 0x7f274c0c5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f274b400640 1 -- 172.21.15.47:0/4268337908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f274c0cd880 msgr2=0x7f274c0c3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:37.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f274b400640 1 --2- 172.21.15.47:0/4268337908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f274c0cd880 0x7f274c0c3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f274b400640 1 -- 172.21.15.47:0/4268337908 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2734009420 con 0x7f274c06e360 2024-02-19T15:22:37.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f274b400640 1 --2- 172.21.15.47:0/4268337908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f274c06e360 0x7f274c151f30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f273c00c480 tx=0x7f273c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:37.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f2743400640 1 -- 172.21.15.47:0/4268337908 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273c0074e0 con 0x7f274c06e360 2024-02-19T15:22:37.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f2743400640 1 -- 172.21.15.47:0/4268337908 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f273c007660 con 0x7f274c06e360 2024-02-19T15:22:37.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f2752000640 1 -- 172.21.15.47:0/4268337908 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f274c0c4990 con 0x7f274c06e360 2024-02-19T15:22:37.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f2743400640 1 -- 172.21.15.47:0/4268337908 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273c004a30 con 0x7f274c06e360 2024-02-19T15:22:37.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.105+0000 7f2752000640 1 -- 172.21.15.47:0/4268337908 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f274c0c4f50 con 0x7f274c06e360 2024-02-19T15:22:37.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.109+0000 7f2752000640 1 -- 172.21.15.47:0/4268337908 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f274c075360 con 0x7f274c06e360 2024-02-19T15:22:37.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.109+0000 7f2743400640 1 -- 172.21.15.47:0/4268337908 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f273c004b90 con 0x7f274c06e360 2024-02-19T15:22:37.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.109+0000 7f2743400640 1 --2- 172.21.15.47:0/4268337908 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2728066e40 0x7f2728069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:37.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.109+0000 7f2743400640 1 -- 172.21.15.47:0/4268337908 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f273c083fa0 con 0x7f274c06e360 2024-02-19T15:22:37.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.109+0000 7f2743e00640 1 --2- 172.21.15.47:0/4268337908 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2728066e40 0x7f2728069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:37.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.113+0000 7f2743e00640 1 --2- 172.21.15.47:0/4268337908 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2728066e40 0x7f2728069300 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f27340ae000 tx=0x7f2734008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:37.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.113+0000 7f2743400640 1 -- 172.21.15.47:0/4268337908 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f273c050da0 con 0x7f274c06e360 2024-02-19T15:22:37.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.277+0000 7f2752000640 1 -- 172.21.15.47:0/4268337908 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f274c0d0740 con 0x7f274c06e360 2024-02-19T15:22:37.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.277+0000 7f2743400640 1 -- 172.21.15.47:0/4268337908 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f273c018020 con 0x7f274c06e360 2024-02-19T15:22:37.281 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:22:37.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.281+0000 7f2752000640 1 -- 172.21.15.47:0/4268337908 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2728066e40 msgr2=0x7f2728069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:37.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.281+0000 7f2752000640 1 --2- 172.21.15.47:0/4268337908 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2728066e40 0x7f2728069300 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f27340ae000 tx=0x7f2734008040 comp rx=0 tx=0).stop 2024-02-19T15:22:37.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.281+0000 7f2752000640 1 -- 172.21.15.47:0/4268337908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f274c06e360 msgr2=0x7f274c151f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:37.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.281+0000 7f2752000640 1 --2- 172.21.15.47:0/4268337908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f274c06e360 0x7f274c151f30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f273c00c480 tx=0x7f273c00c940 comp rx=0 tx=0).stop 2024-02-19T15:22:37.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.281+0000 7f2752000640 1 -- 172.21.15.47:0/4268337908 shutdown_connections 2024-02-19T15:22:37.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.281+0000 7f2752000640 1 --2- 172.21.15.47:0/4268337908 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2728066e40 0x7f2728069300 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.281+0000 7f2752000640 1 --2- 172.21.15.47:0/4268337908 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f274c15af10 0x7f274c0c5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.281+0000 7f2752000640 1 --2- 172.21.15.47:0/4268337908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f274c0cd880 0x7f274c0c3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.281+0000 7f2752000640 1 --2- 172.21.15.47:0/4268337908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f274c06e360 0x7f274c151f30 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.281+0000 7f2752000640 1 -- 172.21.15.47:0/4268337908 >> 172.21.15.47:0/4268337908 conn(0x7f274c0cfc20 msgr2=0x7f274c071080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:37.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.281+0000 7f2752000640 1 -- 172.21.15.47:0/4268337908 shutdown_connections 2024-02-19T15:22:37.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.281+0000 7f2752000640 1 -- 172.21.15.47:0/4268337908 wait complete. 2024-02-19T15:22:37.302 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:22:37.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.441+0000 7fa155800640 1 Processor -- start 2024-02-19T15:22:37.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.441+0000 7fa155800640 1 -- start start 2024-02-19T15:22:37.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.441+0000 7fa155800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa15015dfb0 0x7fa15015e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:37.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.441+0000 7fa155800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1500ce8f0 0x7fa1500cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:37.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.441+0000 7fa155800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1500cbfb0 con 0x7fa1500d2480 2024-02-19T15:22:37.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.441+0000 7fa155800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1500d0980 con 0x7fa15015dfb0 2024-02-19T15:22:37.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.441+0000 7fa155800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1500d0c60 con 0x7fa1500ce8f0 2024-02-19T15:22:37.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.441+0000 7fa14fe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa15015dfb0 0x7fa15015e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:37.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.445+0000 7fa147400640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa1500d2480 0x7fa1500ce1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:42324/0 (socket says 172.21.15.47:42324) 2024-02-19T15:22:37.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.445+0000 7fa147400640 1 -- 172.21.15.47:0/2289247809 learned_addr learned my addr 172.21.15.47:0/2289247809 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:37.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.445+0000 7fa154e00640 1 --2- 172.21.15.47:0/2289247809 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1500ce8f0 0x7fa1500cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:37.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.445+0000 7fa14f400640 1 -- 172.21.15.47:0/2289247809 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232106323 0 0) 0x7fa1500cbfb0 con 0x7fa1500d2480 2024-02-19T15:22:37.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.445+0000 7fa14fe00640 1 -- 172.21.15.47:0/2289247809 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1500ce8f0 msgr2=0x7fa1500cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:37.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.445+0000 7fa14fe00640 1 --2- 172.21.15.47:0/2289247809 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1500ce8f0 0x7fa1500cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.445+0000 7fa14fe00640 1 -- 172.21.15.47:0/2289247809 >> v1:172.21.15.116:6789/0 conn(0x7fa1500d2480 legacy=0x7fa1500ce1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:37.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.445+0000 7fa14fe00640 1 -- 172.21.15.47:0/2289247809 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1500d0f40 con 0x7fa15015dfb0 2024-02-19T15:22:37.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.445+0000 7fa14fe00640 1 --2- 172.21.15.47:0/2289247809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa15015dfb0 0x7fa15015e390 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa1380097c0 tx=0x7fa13801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f57f9b09c33cbcd9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:37.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.445+0000 7fa14f400640 1 -- 172.21.15.47:0/2289247809 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1380aa620 con 0x7fa15015dfb0 2024-02-19T15:22:37.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.445+0000 7fa14f400640 1 -- 172.21.15.47:0/2289247809 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1380aa780 con 0x7fa15015dfb0 2024-02-19T15:22:37.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.445+0000 7fa14f400640 1 -- 172.21.15.47:0/2289247809 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1380aab10 con 0x7fa15015dfb0 2024-02-19T15:22:37.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa155800640 1 -- 172.21.15.47:0/2289247809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa15015dfb0 msgr2=0x7fa15015e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:37.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa155800640 1 --2- 172.21.15.47:0/2289247809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa15015dfb0 0x7fa15015e390 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa1380097c0 tx=0x7fa13801c4a0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa155800640 1 -- 172.21.15.47:0/2289247809 shutdown_connections 2024-02-19T15:22:37.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa155800640 1 --2- 172.21.15.47:0/2289247809 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1500ce8f0 0x7fa1500cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa155800640 1 --2- 172.21.15.47:0/2289247809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa15015dfb0 0x7fa15015e390 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa155800640 1 -- 172.21.15.47:0/2289247809 >> 172.21.15.47:0/2289247809 conn(0x7fa1500d1960 msgr2=0x7fa1500d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:37.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa155800640 1 -- 172.21.15.47:0/2289247809 shutdown_connections 2024-02-19T15:22:37.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa155800640 1 -- 172.21.15.47:0/2289247809 wait complete. 2024-02-19T15:22:37.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa155800640 1 Processor -- start 2024-02-19T15:22:37.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa155800640 1 -- start start 2024-02-19T15:22:37.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa155800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1500ce8f0 0x7fa150159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:37.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa155800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa15015dfb0 0x7fa1500c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:37.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa155800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1500ca220 0x7fa150150500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:37.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa155800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1500c8f70 con 0x7fa1500ce8f0 2024-02-19T15:22:37.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa155800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1500c9180 con 0x7fa1500ca220 2024-02-19T15:22:37.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa155800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1500c9460 con 0x7fa15015dfb0 2024-02-19T15:22:37.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa14fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1500ce8f0 0x7fa150159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:37.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa14fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1500ce8f0 0x7fa150159950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44750/0 (socket says 172.21.15.47:44750) 2024-02-19T15:22:37.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa14fe00640 1 -- 172.21.15.47:0/1905369361 learned_addr learned my addr 172.21.15.47:0/1905369361 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:37.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa147400640 1 --2- 172.21.15.47:0/1905369361 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa15015dfb0 0x7fa1500c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:37.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa154e00640 1 --2- 172.21.15.47:0/1905369361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1500ca220 0x7fa150150500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:37.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa14fe00640 1 -- 172.21.15.47:0/1905369361 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa15015dfb0 msgr2=0x7fa1500c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:37.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa14fe00640 1 --2- 172.21.15.47:0/1905369361 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa15015dfb0 0x7fa1500c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa14fe00640 1 -- 172.21.15.47:0/1905369361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1500ca220 msgr2=0x7fa150150500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:37.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa14fe00640 1 --2- 172.21.15.47:0/1905369361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1500ca220 0x7fa150150500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa14fe00640 1 -- 172.21.15.47:0/1905369361 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa138009420 con 0x7fa1500ce8f0 2024-02-19T15:22:37.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa147400640 1 --2- 172.21.15.47:0/1905369361 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa15015dfb0 0x7fa1500c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:37.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.449+0000 7fa14fe00640 1 --2- 172.21.15.47:0/1905369361 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1500ce8f0 0x7fa150159950 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa1380ac040 tx=0x7fa13801ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:37.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.453+0000 7fa14cc00640 1 -- 172.21.15.47:0/1905369361 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1380be610 con 0x7fa1500ce8f0 2024-02-19T15:22:37.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.453+0000 7fa14cc00640 1 -- 172.21.15.47:0/1905369361 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1380be790 con 0x7fa1500ce8f0 2024-02-19T15:22:37.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.453+0000 7fa14cc00640 1 -- 172.21.15.47:0/1905369361 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1380beb40 con 0x7fa1500ce8f0 2024-02-19T15:22:37.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.453+0000 7fa155800640 1 -- 172.21.15.47:0/1905369361 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1500c9740 con 0x7fa1500ce8f0 2024-02-19T15:22:37.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.453+0000 7fa155800640 1 -- 172.21.15.47:0/1905369361 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1500c9d00 con 0x7fa1500ce8f0 2024-02-19T15:22:37.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.453+0000 7fa155800640 1 -- 172.21.15.47:0/1905369361 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa15004f260 con 0x7fa1500ce8f0 2024-02-19T15:22:37.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.453+0000 7fa14cc00640 1 -- 172.21.15.47:0/1905369361 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa1380b4400 con 0x7fa1500ce8f0 2024-02-19T15:22:37.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.453+0000 7fa14cc00640 1 --2- 172.21.15.47:0/1905369361 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa12c066e40 0x7fa12c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:37.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.453+0000 7fa14cc00640 1 -- 172.21.15.47:0/1905369361 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa13811f400 con 0x7fa1500ce8f0 2024-02-19T15:22:37.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.453+0000 7fa147400640 1 --2- 172.21.15.47:0/1905369361 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa12c066e40 0x7fa12c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:37.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.457+0000 7fa147400640 1 --2- 172.21.15.47:0/1905369361 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa12c066e40 0x7fa12c069300 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa150161bb0 tx=0x7fa1400079e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:37.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.457+0000 7fa14cc00640 1 -- 172.21.15.47:0/1905369361 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa1380ec290 con 0x7fa1500ce8f0 2024-02-19T15:22:37.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.617+0000 7fa155800640 1 -- 172.21.15.47:0/1905369361 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa1500cdb60 con 0x7fa1500ce8f0 2024-02-19T15:22:37.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.617+0000 7fa14cc00640 1 -- 172.21.15.47:0/1905369361 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa1380ef950 con 0x7fa1500ce8f0 2024-02-19T15:22:37.622 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:22:37.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.621+0000 7fa155800640 1 -- 172.21.15.47:0/1905369361 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa12c066e40 msgr2=0x7fa12c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:37.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.621+0000 7fa155800640 1 --2- 172.21.15.47:0/1905369361 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa12c066e40 0x7fa12c069300 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa150161bb0 tx=0x7fa1400079e0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.621+0000 7fa155800640 1 -- 172.21.15.47:0/1905369361 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1500ce8f0 msgr2=0x7fa150159950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:37.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.621+0000 7fa155800640 1 --2- 172.21.15.47:0/1905369361 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1500ce8f0 0x7fa150159950 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa1380ac040 tx=0x7fa13801ace0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.621+0000 7fa155800640 1 -- 172.21.15.47:0/1905369361 shutdown_connections 2024-02-19T15:22:37.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.621+0000 7fa155800640 1 --2- 172.21.15.47:0/1905369361 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa12c066e40 0x7fa12c069300 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.621+0000 7fa155800640 1 --2- 172.21.15.47:0/1905369361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1500ca220 0x7fa150150500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.621+0000 7fa155800640 1 --2- 172.21.15.47:0/1905369361 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa15015dfb0 0x7fa1500c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.621+0000 7fa155800640 1 --2- 172.21.15.47:0/1905369361 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1500ce8f0 0x7fa150159950 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.621+0000 7fa155800640 1 -- 172.21.15.47:0/1905369361 >> 172.21.15.47:0/1905369361 conn(0x7fa1500d1960 msgr2=0x7fa1500b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:37.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.621+0000 7fa155800640 1 -- 172.21.15.47:0/1905369361 shutdown_connections 2024-02-19T15:22:37.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.621+0000 7fa155800640 1 -- 172.21.15.47:0/1905369361 wait complete. 2024-02-19T15:22:37.643 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:22:37.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.781+0000 7fc675e00640 1 Processor -- start 2024-02-19T15:22:37.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.781+0000 7fc675e00640 1 -- start start 2024-02-19T15:22:37.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.785+0000 7fc675e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc67015dfb0 0x7fc67015e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:37.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.785+0000 7fc675e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc6700d2480 0x7fc6700ce1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:37.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.785+0000 7fc675e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6700cbfb0 con 0x7fc6700b2d70 2024-02-19T15:22:37.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.785+0000 7fc675e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6700d0980 con 0x7fc67015dfb0 2024-02-19T15:22:37.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.785+0000 7fc675e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6700d0c60 con 0x7fc6700d2480 2024-02-19T15:22:37.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.785+0000 7fc66fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc6700d2480 0x7fc6700ce1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:37.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.785+0000 7fc674a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc67015dfb0 0x7fc67015e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:37.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.785+0000 7fc66fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc6700d2480 0x7fc6700ce1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:50976/0 (socket says 172.21.15.47:50976) 2024-02-19T15:22:37.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.785+0000 7fc66fe00640 1 -- 172.21.15.47:0/2392911928 learned_addr learned my addr 172.21.15.47:0/2392911928 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:37.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.785+0000 7fc674a00640 1 --2- 172.21.15.47:0/2392911928 >> v2:172.21.15.116:3300/0 conn(0x7fc67015dfb0 0x7fc67015e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:47354/0 (socket says 172.21.15.47:47354) 2024-02-19T15:22:37.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.785+0000 7fc675400640 1 --1- 172.21.15.47:0/2392911928 >> v1:172.21.15.116:6789/0 conn(0x7fc6700b2d70 0x7fc67006c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:42330/0 (socket says 172.21.15.47:42330) 2024-02-19T15:22:37.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.785+0000 7fc66fe00640 1 -- 172.21.15.47:0/2392911928 >> v2:172.21.15.116:3300/0 conn(0x7fc67015dfb0 msgr2=0x7fc67015e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:37.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.785+0000 7fc66fe00640 1 --2- 172.21.15.47:0/2392911928 >> v2:172.21.15.116:3300/0 conn(0x7fc67015dfb0 0x7fc67015e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.785+0000 7fc66fe00640 1 -- 172.21.15.47:0/2392911928 >> v1:172.21.15.116:6789/0 conn(0x7fc6700b2d70 legacy=0x7fc67006c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:37.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.785+0000 7fc66fe00640 1 -- 172.21.15.47:0/2392911928 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6700d0f40 con 0x7fc6700d2480 2024-02-19T15:22:37.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.785+0000 7fc66f400640 1 -- 172.21.15.47:0/2392911928 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704533783 0 0) 0x7fc6700cbfb0 con 0x7fc6700b2d70 2024-02-19T15:22:37.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.785+0000 7fc66fe00640 1 --2- 172.21.15.47:0/2392911928 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc6700d2480 0x7fc6700ce1e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc65c009870 tx=0x7fc65c01c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d96434631dd485a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:37.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc66f400640 1 -- 172.21.15.47:0/2392911928 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c0a9640 con 0x7fc6700d2480 2024-02-19T15:22:37.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc66f400640 1 -- 172.21.15.47:0/2392911928 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65c0a97a0 con 0x7fc6700d2480 2024-02-19T15:22:37.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc66f400640 1 -- 172.21.15.47:0/2392911928 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c0a9b30 con 0x7fc6700d2480 2024-02-19T15:22:37.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc675e00640 1 -- 172.21.15.47:0/2392911928 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc6700d2480 msgr2=0x7fc6700ce1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:37.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc675e00640 1 --2- 172.21.15.47:0/2392911928 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc6700d2480 0x7fc6700ce1e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc65c009870 tx=0x7fc65c01c560 comp rx=0 tx=0).stop 2024-02-19T15:22:37.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc675e00640 1 -- 172.21.15.47:0/2392911928 shutdown_connections 2024-02-19T15:22:37.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc675e00640 1 --2- 172.21.15.47:0/2392911928 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc6700d2480 0x7fc6700ce1e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc675e00640 1 --2- 172.21.15.47:0/2392911928 >> v2:172.21.15.116:3300/0 conn(0x7fc67015dfb0 0x7fc67015e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc675e00640 1 -- 172.21.15.47:0/2392911928 >> 172.21.15.47:0/2392911928 conn(0x7fc6700d1960 msgr2=0x7fc6700d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:37.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc675e00640 1 -- 172.21.15.47:0/2392911928 shutdown_connections 2024-02-19T15:22:37.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc675e00640 1 -- 172.21.15.47:0/2392911928 wait complete. 2024-02-19T15:22:37.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc675e00640 1 Processor -- start 2024-02-19T15:22:37.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc675e00640 1 -- start start 2024-02-19T15:22:37.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc675e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc6700b2d70 0x7fc670155c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:37.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc675e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc67015dfb0 0x7fc6700c7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:37.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc675e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6700b5c40 0x7fc670153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:37.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc675e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6700b5820 con 0x7fc6700b5c40 2024-02-19T15:22:37.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc675e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6700c71b0 con 0x7fc67015dfb0 2024-02-19T15:22:37.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc675e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6700c78d0 con 0x7fc6700b2d70 2024-02-19T15:22:37.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc675400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6700b5c40 0x7fc670153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:37.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc66fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc67015dfb0 0x7fc6700c7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:37.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc674a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc6700b2d70 0x7fc670155c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:37.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc66fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc67015dfb0 0x7fc6700c7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:47366/0 (socket says 172.21.15.47:47366) 2024-02-19T15:22:37.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc66fe00640 1 -- 172.21.15.47:0/2210017531 learned_addr learned my addr 172.21.15.47:0/2210017531 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:37.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.789+0000 7fc675400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6700b5c40 0x7fc670153f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44756/0 (socket says 172.21.15.47:44756) 2024-02-19T15:22:37.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.793+0000 7fc674a00640 1 -- 172.21.15.47:0/2210017531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc67015dfb0 msgr2=0x7fc6700c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:37.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.793+0000 7fc674a00640 1 --2- 172.21.15.47:0/2210017531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc67015dfb0 0x7fc6700c7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.793+0000 7fc674a00640 1 -- 172.21.15.47:0/2210017531 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6700b5c40 msgr2=0x7fc670153f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:37.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.793+0000 7fc674a00640 1 --2- 172.21.15.47:0/2210017531 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6700b5c40 0x7fc670153f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.793+0000 7fc674a00640 1 -- 172.21.15.47:0/2210017531 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c0094d0 con 0x7fc6700b2d70 2024-02-19T15:22:37.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.793+0000 7fc66fe00640 1 --2- 172.21.15.47:0/2210017531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc67015dfb0 0x7fc6700c7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:37.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.793+0000 7fc675400640 1 --2- 172.21.15.47:0/2210017531 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6700b5c40 0x7fc670153f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:37.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.793+0000 7fc674a00640 1 --2- 172.21.15.47:0/2210017531 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc6700b2d70 0x7fc670155c00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc66400e8a0 tx=0x7fc66400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:37.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.793+0000 7fc66cc00640 1 -- 172.21.15.47:0/2210017531 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc664009e80 con 0x7fc6700b2d70 2024-02-19T15:22:37.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.793+0000 7fc66cc00640 1 -- 172.21.15.47:0/2210017531 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66400c940 con 0x7fc6700b2d70 2024-02-19T15:22:37.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.793+0000 7fc66cc00640 1 -- 172.21.15.47:0/2210017531 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66400ccb0 con 0x7fc6700b2d70 2024-02-19T15:22:37.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.793+0000 7fc675e00640 1 -- 172.21.15.47:0/2210017531 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6700b51a0 con 0x7fc6700b2d70 2024-02-19T15:22:37.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.793+0000 7fc675e00640 1 -- 172.21.15.47:0/2210017531 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6700b5440 con 0x7fc6700b2d70 2024-02-19T15:22:37.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.793+0000 7fc675e00640 1 -- 172.21.15.47:0/2210017531 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc67004f260 con 0x7fc6700b2d70 2024-02-19T15:22:37.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.793+0000 7fc66cc00640 1 -- 172.21.15.47:0/2210017531 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc664013470 con 0x7fc6700b2d70 2024-02-19T15:22:37.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.793+0000 7fc66cc00640 1 --2- 172.21.15.47:0/2210017531 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc650066e70 0x7fc650069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:37.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.797+0000 7fc66cc00640 1 -- 172.21.15.47:0/2210017531 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6640849d0 con 0x7fc6700b2d70 2024-02-19T15:22:37.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.797+0000 7fc66fe00640 1 --2- 172.21.15.47:0/2210017531 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc650066e70 0x7fc650069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:37.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.797+0000 7fc66fe00640 1 --2- 172.21.15.47:0/2210017531 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc650066e70 0x7fc650069330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc65c01c5a0 tx=0x7fc65c0b3040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:37.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.801+0000 7fc66cc00640 1 -- 172.21.15.47:0/2210017531 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc664051860 con 0x7fc6700b2d70 2024-02-19T15:22:37.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.965+0000 7fc675e00640 1 -- 172.21.15.47:0/2210017531 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc6700c8780 con 0x7fc6700b2d70 2024-02-19T15:22:37.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.969+0000 7fc66cc00640 1 -- 172.21.15.47:0/2210017531 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc6640170a0 con 0x7fc6700b2d70 2024-02-19T15:22:37.971 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:22:37.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.969+0000 7fc675e00640 1 -- 172.21.15.47:0/2210017531 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc650066e70 msgr2=0x7fc650069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:37.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.969+0000 7fc675e00640 1 --2- 172.21.15.47:0/2210017531 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc650066e70 0x7fc650069330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc65c01c5a0 tx=0x7fc65c0b3040 comp rx=0 tx=0).stop 2024-02-19T15:22:37.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.969+0000 7fc675e00640 1 -- 172.21.15.47:0/2210017531 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc6700b2d70 msgr2=0x7fc670155c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:37.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.969+0000 7fc675e00640 1 --2- 172.21.15.47:0/2210017531 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc6700b2d70 0x7fc670155c00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc66400e8a0 tx=0x7fc66400ed60 comp rx=0 tx=0).stop 2024-02-19T15:22:37.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.969+0000 7fc675e00640 1 -- 172.21.15.47:0/2210017531 shutdown_connections 2024-02-19T15:22:37.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.969+0000 7fc675e00640 1 --2- 172.21.15.47:0/2210017531 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc650066e70 0x7fc650069330 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.969+0000 7fc675e00640 1 --2- 172.21.15.47:0/2210017531 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6700b5c40 0x7fc670153f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.969+0000 7fc675e00640 1 --2- 172.21.15.47:0/2210017531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc67015dfb0 0x7fc6700c7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.969+0000 7fc675e00640 1 --2- 172.21.15.47:0/2210017531 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc6700b2d70 0x7fc670155c00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:37.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.969+0000 7fc675e00640 1 -- 172.21.15.47:0/2210017531 >> 172.21.15.47:0/2210017531 conn(0x7fc6700d1960 msgr2=0x7fc6700b3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:37.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.969+0000 7fc675e00640 1 -- 172.21.15.47:0/2210017531 shutdown_connections 2024-02-19T15:22:37.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:37.969+0000 7fc675e00640 1 -- 172.21.15.47:0/2210017531 wait complete. 2024-02-19T15:22:37.993 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-19T15:22:38.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.133+0000 7f168ac00640 1 Processor -- start 2024-02-19T15:22:38.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.133+0000 7f168ac00640 1 -- start start 2024-02-19T15:22:38.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.133+0000 7f168ac00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f16840ce400 0x7f16840ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:38.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.133+0000 7f168ac00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16840ced90 con 0x7f16840b4240 2024-02-19T15:22:38.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.133+0000 7f168ac00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16840d1070 con 0x7f168415dfa0 2024-02-19T15:22:38.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.133+0000 7f168ac00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f168406ec60 con 0x7f16840ce400 2024-02-19T15:22:38.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.133+0000 7f1688c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f16840ce400 0x7f16840ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:38.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.133+0000 7f1688c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f16840ce400 0x7f16840ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44764/0 (socket says 172.21.15.47:44764) 2024-02-19T15:22:38.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.133+0000 7f1688c00640 1 -- 172.21.15.47:0/1214440019 learned_addr learned my addr 172.21.15.47:0/1214440019 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:38.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.133+0000 7f1688c00640 1 -- 172.21.15.47:0/1214440019 >> v1:172.21.15.116:6789/0 conn(0x7f168415dfa0 legacy=0x7f168406fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:38.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.133+0000 7f1688c00640 1 -- 172.21.15.47:0/1214440019 >> v1:172.21.15.47:6789/0 conn(0x7f16840b4240 legacy=0x7f16840d0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:38.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.133+0000 7f1688c00640 1 -- 172.21.15.47:0/1214440019 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f168406ef40 con 0x7f16840ce400 2024-02-19T15:22:38.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.133+0000 7f1683400640 1 -- 172.21.15.47:0/1214440019 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286852350 0 0) 0x7f166c001120 con 0x7f16840b4240 2024-02-19T15:22:38.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.137+0000 7f1688c00640 1 --2- 172.21.15.47:0/1214440019 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16840ce400 0x7f16840ce850 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f167c009e50 tx=0x7f167c01ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c44353d5af5fd605 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:38.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.137+0000 7f1683400640 1 -- 172.21.15.47:0/1214440019 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167c0a57e0 con 0x7f16840ce400 2024-02-19T15:22:38.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.137+0000 7f1683400640 1 -- 172.21.15.47:0/1214440019 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167c0a5940 con 0x7f16840ce400 2024-02-19T15:22:38.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.137+0000 7f1683400640 1 -- 172.21.15.47:0/1214440019 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167c0a5cd0 con 0x7f16840ce400 2024-02-19T15:22:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.137+0000 7f168ac00640 1 -- 172.21.15.47:0/1214440019 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16840ce400 msgr2=0x7f16840ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.137+0000 7f168ac00640 1 --2- 172.21.15.47:0/1214440019 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16840ce400 0x7f16840ce850 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f167c009e50 tx=0x7f167c01ca00 comp rx=0 tx=0).stop 2024-02-19T15:22:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.137+0000 7f168ac00640 1 -- 172.21.15.47:0/1214440019 shutdown_connections 2024-02-19T15:22:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.137+0000 7f168ac00640 1 --2- 172.21.15.47:0/1214440019 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16840ce400 0x7f16840ce850 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.137+0000 7f168ac00640 1 -- 172.21.15.47:0/1214440019 >> 172.21.15.47:0/1214440019 conn(0x7f16840b3690 msgr2=0x7f16840b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.137+0000 7f168ac00640 1 -- 172.21.15.47:0/1214440019 shutdown_connections 2024-02-19T15:22:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.137+0000 7f168ac00640 1 -- 172.21.15.47:0/1214440019 wait complete. 2024-02-19T15:22:38.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.137+0000 7f168ac00640 1 Processor -- start 2024-02-19T15:22:38.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f168ac00640 1 -- start start 2024-02-19T15:22:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f168ac00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16840b4240 0x7f1684151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f168ac00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16840ce400 0x7f1684160860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f168ac00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f168415dfa0 0x7f1684160da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f168ac00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16840c9970 con 0x7f16840b4240 2024-02-19T15:22:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f168ac00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16841612e0 con 0x7f16840ce400 2024-02-19T15:22:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f168ac00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1684152160 con 0x7f168415dfa0 2024-02-19T15:22:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f1683e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16840b4240 0x7f1684151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f1683e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16840b4240 0x7f1684151c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44776/0 (socket says 172.21.15.47:44776) 2024-02-19T15:22:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f1683e00640 1 -- 172.21.15.47:0/2860075922 learned_addr learned my addr 172.21.15.47:0/2860075922 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f1688c00640 1 --2- 172.21.15.47:0/2860075922 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f168415dfa0 0x7f1684160da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f167b400640 1 --2- 172.21.15.47:0/2860075922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16840ce400 0x7f1684160860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f1683e00640 1 -- 172.21.15.47:0/2860075922 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f168415dfa0 msgr2=0x7f1684160da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f1683e00640 1 --2- 172.21.15.47:0/2860075922 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f168415dfa0 0x7f1684160da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:38.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f1683e00640 1 -- 172.21.15.47:0/2860075922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16840ce400 msgr2=0x7f1684160860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:38.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f1683e00640 1 --2- 172.21.15.47:0/2860075922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16840ce400 0x7f1684160860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:38.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f1683e00640 1 -- 172.21.15.47:0/2860075922 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c009b30 con 0x7f16840b4240 2024-02-19T15:22:38.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f1683e00640 1 --2- 172.21.15.47:0/2860075922 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16840b4240 0x7f1684151c20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f167400c530 tx=0x7f167400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:38.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f1680c00640 1 -- 172.21.15.47:0/2860075922 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1674005220 con 0x7f16840b4240 2024-02-19T15:22:38.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f1680c00640 1 -- 172.21.15.47:0/2860075922 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1674005380 con 0x7f16840b4240 2024-02-19T15:22:38.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f168ac00640 1 -- 172.21.15.47:0/2860075922 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16840c8cf0 con 0x7f16840b4240 2024-02-19T15:22:38.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f1680c00640 1 -- 172.21.15.47:0/2860075922 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1674007270 con 0x7f16840b4240 2024-02-19T15:22:38.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f168ac00640 1 -- 172.21.15.47:0/2860075922 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16840c9080 con 0x7f16840b4240 2024-02-19T15:22:38.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.141+0000 7f168ac00640 1 -- 172.21.15.47:0/2860075922 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16840c9ab0 con 0x7f16840b4240 2024-02-19T15:22:38.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.145+0000 7f1680c00640 1 -- 172.21.15.47:0/2860075922 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f16740073d0 con 0x7f16840b4240 2024-02-19T15:22:38.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.145+0000 7f1680c00640 1 --2- 172.21.15.47:0/2860075922 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1660066e40 0x7f1660069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:38.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.145+0000 7f1680c00640 1 -- 172.21.15.47:0/2860075922 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1674084210 con 0x7f16840b4240 2024-02-19T15:22:38.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.145+0000 7f167b400640 1 --2- 172.21.15.47:0/2860075922 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1660066e40 0x7f1660069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:38.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.145+0000 7f167b400640 1 --2- 172.21.15.47:0/2860075922 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1660066e40 0x7f1660069300 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f168406d640 tx=0x7f166c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:38.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.149+0000 7f1680c00640 1 -- 172.21.15.47:0/2860075922 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f16740510a0 con 0x7f16840b4240 2024-02-19T15:22:38.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.321+0000 7f168ac00640 1 -- 172.21.15.47:0/2860075922 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1684075360 con 0x7f16840b4240 2024-02-19T15:22:38.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.321+0000 7f1680c00640 1 -- 172.21.15.47:0/2860075922 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1674054760 con 0x7f16840b4240 2024-02-19T15:22:38.326 INFO:teuthology.orchestra.run.smithi047.stderr:listed 0 entries 2024-02-19T15:22:38.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.325+0000 7f168ac00640 1 -- 172.21.15.47:0/2860075922 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1660066e40 msgr2=0x7f1660069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:38.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.325+0000 7f168ac00640 1 --2- 172.21.15.47:0/2860075922 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1660066e40 0x7f1660069300 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f168406d640 tx=0x7f166c009040 comp rx=0 tx=0).stop 2024-02-19T15:22:38.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.325+0000 7f168ac00640 1 -- 172.21.15.47:0/2860075922 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16840b4240 msgr2=0x7f1684151c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:38.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.325+0000 7f168ac00640 1 --2- 172.21.15.47:0/2860075922 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16840b4240 0x7f1684151c20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f167400c530 tx=0x7f167400c9f0 comp rx=0 tx=0).stop 2024-02-19T15:22:38.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.325+0000 7f168ac00640 1 -- 172.21.15.47:0/2860075922 shutdown_connections 2024-02-19T15:22:38.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.325+0000 7f168ac00640 1 --2- 172.21.15.47:0/2860075922 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1660066e40 0x7f1660069300 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:38.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.325+0000 7f168ac00640 1 --2- 172.21.15.47:0/2860075922 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f168415dfa0 0x7f1684160da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:38.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.325+0000 7f168ac00640 1 --2- 172.21.15.47:0/2860075922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16840ce400 0x7f1684160860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:38.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.325+0000 7f168ac00640 1 --2- 172.21.15.47:0/2860075922 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16840b4240 0x7f1684151c20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:38.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.325+0000 7f168ac00640 1 -- 172.21.15.47:0/2860075922 >> 172.21.15.47:0/2860075922 conn(0x7f16840b3690 msgr2=0x7f168406d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:38.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.325+0000 7f168ac00640 1 -- 172.21.15.47:0/2860075922 shutdown_connections 2024-02-19T15:22:38.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.325+0000 7f168ac00640 1 -- 172.21.15.47:0/2860075922 wait complete. 2024-02-19T15:22:38.347 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-02-19T15:22:38.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:38.347 DEBUG:teuthology.orchestra.run.smithi047:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-02-19T15:22:38.380 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi047.front.sepia.ceph.com 2024-02-19T15:22:38.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:38.381 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-19T15:22:38.436 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:22:38.437 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:22:38.445 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:22:38.445 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:22:38.493 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:22:38.493 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:22:38.541 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:38.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.677+0000 7f619d000640 1 Processor -- start 2024-02-19T15:22:38.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.677+0000 7f619d000640 1 -- start start 2024-02-19T15:22:38.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.677+0000 7f619d000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f619815b1a0 0x7f619815b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:38.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.677+0000 7f619d000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6198071350 0x7f619815bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:38.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.677+0000 7f619d000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619815c000 con 0x7f619806c9a0 2024-02-19T15:22:38.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.677+0000 7f619d000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61980718c0 con 0x7f619815b1a0 2024-02-19T15:22:38.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.677+0000 7f619d000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6198071ba0 con 0x7f6198071350 2024-02-19T15:22:38.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.677+0000 7f6195e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f619815b1a0 0x7f619815b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:38.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.677+0000 7f6195e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f619815b1a0 0x7f619815b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44782/0 (socket says 172.21.15.47:44782) 2024-02-19T15:22:38.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.677+0000 7f6195e00640 1 -- 172.21.15.47:0/1324702505 learned_addr learned my addr 172.21.15.47:0/1324702505 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:38.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.677+0000 7f618fe00640 1 --2- 172.21.15.47:0/1324702505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6198071350 0x7f619815bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:38.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.681+0000 7f6195e00640 1 -- 172.21.15.47:0/1324702505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6198071350 msgr2=0x7f619815bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:38.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.681+0000 7f6195e00640 1 --2- 172.21.15.47:0/1324702505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6198071350 0x7f619815bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:38.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.681+0000 7f6195e00640 1 -- 172.21.15.47:0/1324702505 >> v1:172.21.15.116:6789/0 conn(0x7f619806c9a0 legacy=0x7f619815c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:38.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.681+0000 7f6195e00640 1 -- 172.21.15.47:0/1324702505 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6198071e80 con 0x7f619815b1a0 2024-02-19T15:22:38.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.681+0000 7f6195e00640 1 --2- 172.21.15.47:0/1324702505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f619815b1a0 0x7f619815b580 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6188009870 tx=0x7f618801e740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f627269ab016486f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:38.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.681+0000 7f6195400640 1 -- 172.21.15.47:0/1324702505 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61880a7720 con 0x7f619815b1a0 2024-02-19T15:22:38.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.681+0000 7f6195400640 1 -- 172.21.15.47:0/1324702505 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61880a7880 con 0x7f619815b1a0 2024-02-19T15:22:38.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.681+0000 7f6195400640 1 -- 172.21.15.47:0/1324702505 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61880a7c10 con 0x7f619815b1a0 2024-02-19T15:22:38.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.681+0000 7f619d000640 1 -- 172.21.15.47:0/1324702505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f619815b1a0 msgr2=0x7f619815b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:38.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.681+0000 7f619d000640 1 --2- 172.21.15.47:0/1324702505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f619815b1a0 0x7f619815b580 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6188009870 tx=0x7f618801e740 comp rx=0 tx=0).stop 2024-02-19T15:22:38.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.681+0000 7f619d000640 1 -- 172.21.15.47:0/1324702505 shutdown_connections 2024-02-19T15:22:38.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.681+0000 7f619d000640 1 --2- 172.21.15.47:0/1324702505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6198071350 0x7f619815bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:38.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.681+0000 7f619d000640 1 --2- 172.21.15.47:0/1324702505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f619815b1a0 0x7f619815b580 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:38.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.681+0000 7f619d000640 1 -- 172.21.15.47:0/1324702505 >> 172.21.15.47:0/1324702505 conn(0x7f61980b16a0 msgr2=0x7f61980b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:38.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.681+0000 7f619d000640 1 -- 172.21.15.47:0/1324702505 shutdown_connections 2024-02-19T15:22:38.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.681+0000 7f619d000640 1 -- 172.21.15.47:0/1324702505 wait complete. 2024-02-19T15:22:38.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f619d000640 1 Processor -- start 2024-02-19T15:22:38.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f619d000640 1 -- start start 2024-02-19T15:22:38.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f619d000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f619806c9a0 0x7f61981481c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:38.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f619d000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6198071350 0x7f6198153a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:38.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f619d000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619815b1a0 0x7f6198152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:38.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f619d000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6198153f50 con 0x7f619806c9a0 2024-02-19T15:22:38.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f619d000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6198154ef0 con 0x7f6198071350 2024-02-19T15:22:38.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f619d000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61980ae7b0 con 0x7f619815b1a0 2024-02-19T15:22:38.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f6195e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f619806c9a0 0x7f61981481c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:38.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f6195e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f619806c9a0 0x7f61981481c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44798/0 (socket says 172.21.15.47:44798) 2024-02-19T15:22:38.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f6195e00640 1 -- 172.21.15.47:0/601775235 learned_addr learned my addr 172.21.15.47:0/601775235 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:38.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f618fe00640 1 --2- 172.21.15.47:0/601775235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6198071350 0x7f6198153a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:38.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f6196800640 1 --2- 172.21.15.47:0/601775235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619815b1a0 0x7f6198152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:38.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f6195e00640 1 -- 172.21.15.47:0/601775235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619815b1a0 msgr2=0x7f6198152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:38.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f6195e00640 1 --2- 172.21.15.47:0/601775235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619815b1a0 0x7f6198152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:38.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f6195e00640 1 -- 172.21.15.47:0/601775235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6198071350 msgr2=0x7f6198153a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:38.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f6195e00640 1 --2- 172.21.15.47:0/601775235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6198071350 0x7f6198153a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:38.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f6195e00640 1 -- 172.21.15.47:0/601775235 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61880094d0 con 0x7f619806c9a0 2024-02-19T15:22:38.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f6196800640 1 --2- 172.21.15.47:0/601775235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619815b1a0 0x7f6198152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:38.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f6195e00640 1 --2- 172.21.15.47:0/601775235 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f619806c9a0 0x7f61981481c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f61880b2040 tx=0x7f618801b4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:38.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f618e000640 1 -- 172.21.15.47:0/601775235 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61880af850 con 0x7f619806c9a0 2024-02-19T15:22:38.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f618e000640 1 -- 172.21.15.47:0/601775235 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61880af9b0 con 0x7f619806c9a0 2024-02-19T15:22:38.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f618e000640 1 -- 172.21.15.47:0/601775235 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61880afd60 con 0x7f619806c9a0 2024-02-19T15:22:38.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f619d000640 1 -- 172.21.15.47:0/601775235 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61980aea90 con 0x7f619806c9a0 2024-02-19T15:22:38.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.685+0000 7f619d000640 1 -- 172.21.15.47:0/601775235 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61980af050 con 0x7f619806c9a0 2024-02-19T15:22:38.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.689+0000 7f619d000640 1 -- 172.21.15.47:0/601775235 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6198070d60 con 0x7f619806c9a0 2024-02-19T15:22:38.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.689+0000 7f618e000640 1 -- 172.21.15.47:0/601775235 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f61880c2360 con 0x7f619806c9a0 2024-02-19T15:22:38.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.689+0000 7f618e000640 1 --2- 172.21.15.47:0/601775235 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6174066e40 0x7f6174069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:38.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.689+0000 7f618e000640 1 -- 172.21.15.47:0/601775235 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f618811ec10 con 0x7f619806c9a0 2024-02-19T15:22:38.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.689+0000 7f618fe00640 1 --2- 172.21.15.47:0/601775235 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6174066e40 0x7f6174069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:38.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.689+0000 7f618fe00640 1 --2- 172.21.15.47:0/601775235 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6174066e40 0x7f6174069300 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6198148d80 tx=0x7f6180009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:38.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.693+0000 7f618e000640 1 -- 172.21.15.47:0/601775235 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f61880eba10 con 0x7f619806c9a0 2024-02-19T15:22:38.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.857+0000 7f619d000640 1 -- 172.21.15.47:0/601775235 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61980af240 con 0x7f619806c9a0 2024-02-19T15:22:38.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.857+0000 7f618e000640 1 -- 172.21.15.47:0/601775235 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f61880bf020 con 0x7f619806c9a0 2024-02-19T15:22:38.861 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:38.861 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":21,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:06.843155+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:03.091360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:04.222965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:38.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.861+0000 7f619d000640 1 -- 172.21.15.47:0/601775235 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6174066e40 msgr2=0x7f6174069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:38.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.861+0000 7f619d000640 1 --2- 172.21.15.47:0/601775235 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6174066e40 0x7f6174069300 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6198148d80 tx=0x7f6180009190 comp rx=0 tx=0).stop 2024-02-19T15:22:38.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.861+0000 7f619d000640 1 -- 172.21.15.47:0/601775235 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f619806c9a0 msgr2=0x7f61981481c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:38.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.861+0000 7f619d000640 1 --2- 172.21.15.47:0/601775235 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f619806c9a0 0x7f61981481c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f61880b2040 tx=0x7f618801b4e0 comp rx=0 tx=0).stop 2024-02-19T15:22:38.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.861+0000 7f619d000640 1 -- 172.21.15.47:0/601775235 shutdown_connections 2024-02-19T15:22:38.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.861+0000 7f619d000640 1 --2- 172.21.15.47:0/601775235 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6174066e40 0x7f6174069300 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:38.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.861+0000 7f619d000640 1 --2- 172.21.15.47:0/601775235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619815b1a0 0x7f6198152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:38.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.861+0000 7f619d000640 1 --2- 172.21.15.47:0/601775235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6198071350 0x7f6198153a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:38.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.861+0000 7f619d000640 1 --2- 172.21.15.47:0/601775235 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f619806c9a0 0x7f61981481c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:38.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.861+0000 7f619d000640 1 -- 172.21.15.47:0/601775235 >> 172.21.15.47:0/601775235 conn(0x7f61980b16a0 msgr2=0x7f619815b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:38.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.861+0000 7f619d000640 1 -- 172.21.15.47:0/601775235 shutdown_connections 2024-02-19T15:22:38.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:38.861+0000 7f619d000640 1 -- 172.21.15.47:0/601775235 wait complete. 2024-02-19T15:22:38.882 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:22:03.091360+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:22:04.222965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:22:38.883 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:22:39.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.021+0000 7f6512400640 1 Processor -- start 2024-02-19T15:22:39.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.021+0000 7f6512400640 1 -- start start 2024-02-19T15:22:39.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.025+0000 7f6512400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f650c15bd90 0x7f650c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:39.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.025+0000 7f6512400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f650c0cb490 0x7f650c072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:39.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.025+0000 7f6512400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f650c072680 0x7f650c0ca270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:39.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.025+0000 7f6512400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f650c15fc90 con 0x7f650c072680 2024-02-19T15:22:39.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.025+0000 7f6512400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f650c15ff70 con 0x7f650c0cb490 2024-02-19T15:22:39.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.025+0000 7f6512400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f650c160250 con 0x7f650c15bd90 2024-02-19T15:22:39.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.025+0000 7f650be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f650c072680 0x7f650c0ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:39.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.025+0000 7f650be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f650c072680 0x7f650c0ca270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44810/0 (socket says 172.21.15.47:44810) 2024-02-19T15:22:39.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.025+0000 7f650be00640 1 -- 172.21.15.47:0/774348580 learned_addr learned my addr 172.21.15.47:0/774348580 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:39.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.025+0000 7f650b400640 1 --2- 172.21.15.47:0/774348580 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f650c15bd90 0x7f650c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:39.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.025+0000 7f6502a00640 1 --2- 172.21.15.47:0/774348580 >> v2:172.21.15.116:3300/0 conn(0x7f650c0cb490 0x7f650c072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:39.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.025+0000 7f650be00640 1 -- 172.21.15.47:0/774348580 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f650c15bd90 msgr2=0x7f650c070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:39.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.025+0000 7f650be00640 1 --2- 172.21.15.47:0/774348580 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f650c15bd90 0x7f650c070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.025+0000 7f650be00640 1 -- 172.21.15.47:0/774348580 >> v2:172.21.15.116:3300/0 conn(0x7f650c0cb490 msgr2=0x7f650c072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:39.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.025+0000 7f650be00640 1 --2- 172.21.15.47:0/774348580 >> v2:172.21.15.116:3300/0 conn(0x7f650c0cb490 0x7f650c072140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.025+0000 7f650be00640 1 -- 172.21.15.47:0/774348580 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f650c160530 con 0x7f650c072680 2024-02-19T15:22:39.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.029+0000 7f650be00640 1 --2- 172.21.15.47:0/774348580 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f650c072680 0x7f650c0ca270 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f650c0b2810 tx=0x7f650401c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc867aa68fa65bf8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:39.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.029+0000 7f650aa00640 1 -- 172.21.15.47:0/774348580 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65040a8410 con 0x7f650c072680 2024-02-19T15:22:39.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.029+0000 7f650aa00640 1 -- 172.21.15.47:0/774348580 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65040a8570 con 0x7f650c072680 2024-02-19T15:22:39.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.029+0000 7f650aa00640 1 -- 172.21.15.47:0/774348580 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65040a8900 con 0x7f650c072680 2024-02-19T15:22:39.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.029+0000 7f6512400640 1 -- 172.21.15.47:0/774348580 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f650c072680 msgr2=0x7f650c0ca270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:39.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.029+0000 7f6512400640 1 --2- 172.21.15.47:0/774348580 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f650c072680 0x7f650c0ca270 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f650c0b2810 tx=0x7f650401c5d0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.029+0000 7f6512400640 1 -- 172.21.15.47:0/774348580 shutdown_connections 2024-02-19T15:22:39.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.029+0000 7f6512400640 1 --2- 172.21.15.47:0/774348580 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f650c072680 0x7f650c0ca270 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.029+0000 7f6512400640 1 --2- 172.21.15.47:0/774348580 >> v2:172.21.15.116:3300/0 conn(0x7f650c0cb490 0x7f650c072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.029+0000 7f6512400640 1 --2- 172.21.15.47:0/774348580 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f650c15bd90 0x7f650c070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.029+0000 7f6512400640 1 -- 172.21.15.47:0/774348580 >> 172.21.15.47:0/774348580 conn(0x7f650c0cfc20 msgr2=0x7f650c0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:39.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.029+0000 7f6512400640 1 -- 172.21.15.47:0/774348580 shutdown_connections 2024-02-19T15:22:39.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.029+0000 7f6512400640 1 -- 172.21.15.47:0/774348580 wait complete. 2024-02-19T15:22:39.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.029+0000 7f6512400640 1 Processor -- start 2024-02-19T15:22:39.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.029+0000 7f6512400640 1 -- start start 2024-02-19T15:22:39.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f6512400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f650c072680 0x7f650c0c9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:39.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f6512400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c0cb490 0x7f650c0a9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:39.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f6512400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f650c15bd90 0x7f650c15ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:39.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f6512400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650c0c90e0 con 0x7f650c072680 2024-02-19T15:22:39.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f6512400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650c0c8460 con 0x7f650c0cb490 2024-02-19T15:22:39.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f6512400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f650c0c8740 con 0x7f650c15bd90 2024-02-19T15:22:39.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f650b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f650c072680 0x7f650c0c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:39.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f650b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f650c072680 0x7f650c0c9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44820/0 (socket says 172.21.15.47:44820) 2024-02-19T15:22:39.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f650b400640 1 -- 172.21.15.47:0/3754949315 learned_addr learned my addr 172.21.15.47:0/3754949315 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:39.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f6502a00640 1 --2- 172.21.15.47:0/3754949315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c0cb490 0x7f650c0a9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:39.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f650be00640 1 --2- 172.21.15.47:0/3754949315 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f650c15bd90 0x7f650c15ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:39.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f650b400640 1 -- 172.21.15.47:0/3754949315 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f650c15bd90 msgr2=0x7f650c15ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:39.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f650b400640 1 --2- 172.21.15.47:0/3754949315 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f650c15bd90 0x7f650c15ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f650b400640 1 -- 172.21.15.47:0/3754949315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c0cb490 msgr2=0x7f650c0a9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:39.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f650b400640 1 --2- 172.21.15.47:0/3754949315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c0cb490 0x7f650c0a9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f650b400640 1 -- 172.21.15.47:0/3754949315 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6504009b30 con 0x7f650c072680 2024-02-19T15:22:39.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f650b400640 1 --2- 172.21.15.47:0/3754949315 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f650c072680 0x7f650c0c9500 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f64fc00e870 tx=0x7f64fc00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:39.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f6503e00640 1 -- 172.21.15.47:0/3754949315 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64fc009510 con 0x7f650c072680 2024-02-19T15:22:39.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f6503e00640 1 -- 172.21.15.47:0/3754949315 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64fc00eec0 con 0x7f650c072680 2024-02-19T15:22:39.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f6503e00640 1 -- 172.21.15.47:0/3754949315 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64fc00cb40 con 0x7f650c072680 2024-02-19T15:22:39.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f6512400640 1 -- 172.21.15.47:0/3754949315 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f650c0c8a20 con 0x7f650c072680 2024-02-19T15:22:39.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f6512400640 1 -- 172.21.15.47:0/3754949315 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f650c0a9b60 con 0x7f650c072680 2024-02-19T15:22:39.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f6512400640 1 -- 172.21.15.47:0/3754949315 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f650c0c9350 con 0x7f650c072680 2024-02-19T15:22:39.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.033+0000 7f6503e00640 1 -- 172.21.15.47:0/3754949315 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f64fc00ccc0 con 0x7f650c072680 2024-02-19T15:22:39.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.037+0000 7f6503e00640 1 --2- 172.21.15.47:0/3754949315 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f64e8066e40 0x7f64e8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:39.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.037+0000 7f6503e00640 1 -- 172.21.15.47:0/3754949315 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f64fc084a80 con 0x7f650c072680 2024-02-19T15:22:39.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.037+0000 7f6502a00640 1 --2- 172.21.15.47:0/3754949315 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f64e8066e40 0x7f64e8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:39.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.037+0000 7f6502a00640 1 --2- 172.21.15.47:0/3754949315 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f64e8066e40 0x7f64e8069300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f650c0c7ef0 tx=0x7f64f4009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:39.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.041+0000 7f6503e00640 1 -- 172.21.15.47:0/3754949315 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f64fc051910 con 0x7f650c072680 2024-02-19T15:22:39.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.201+0000 7f6512400640 1 -- 172.21.15.47:0/3754949315 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f650c075360 con 0x7f650c072680 2024-02-19T15:22:39.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.201+0000 7f6503e00640 1 -- 172.21.15.47:0/3754949315 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f64fc0170a0 con 0x7f650c072680 2024-02-19T15:22:39.205 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:22:39.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.205+0000 7f6512400640 1 -- 172.21.15.47:0/3754949315 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f64e8066e40 msgr2=0x7f64e8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:39.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.205+0000 7f6512400640 1 --2- 172.21.15.47:0/3754949315 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f64e8066e40 0x7f64e8069300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f650c0c7ef0 tx=0x7f64f4009190 comp rx=0 tx=0).stop 2024-02-19T15:22:39.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.205+0000 7f6512400640 1 -- 172.21.15.47:0/3754949315 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f650c072680 msgr2=0x7f650c0c9500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:39.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.205+0000 7f6512400640 1 --2- 172.21.15.47:0/3754949315 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f650c072680 0x7f650c0c9500 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f64fc00e870 tx=0x7f64fc00ec70 comp rx=0 tx=0).stop 2024-02-19T15:22:39.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.205+0000 7f6512400640 1 -- 172.21.15.47:0/3754949315 shutdown_connections 2024-02-19T15:22:39.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.205+0000 7f6512400640 1 --2- 172.21.15.47:0/3754949315 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f64e8066e40 0x7f64e8069300 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.205+0000 7f6512400640 1 --2- 172.21.15.47:0/3754949315 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f650c15bd90 0x7f650c15ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.205+0000 7f6512400640 1 --2- 172.21.15.47:0/3754949315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c0cb490 0x7f650c0a9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.205+0000 7f6512400640 1 --2- 172.21.15.47:0/3754949315 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f650c072680 0x7f650c0c9500 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.205+0000 7f6512400640 1 -- 172.21.15.47:0/3754949315 >> 172.21.15.47:0/3754949315 conn(0x7f650c0cfc20 msgr2=0x7f650c0b47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:39.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.205+0000 7f6512400640 1 -- 172.21.15.47:0/3754949315 shutdown_connections 2024-02-19T15:22:39.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.205+0000 7f6512400640 1 -- 172.21.15.47:0/3754949315 wait complete. 2024-02-19T15:22:39.225 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:22:39.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.365+0000 7fb4bb800640 1 Processor -- start 2024-02-19T15:22:39.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.365+0000 7fb4bb800640 1 -- start start 2024-02-19T15:22:39.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.365+0000 7fb4bb800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb4b415bd90 0x7fb4b4070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:39.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.365+0000 7fb4bb800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4b4072850 0x7fb4b406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:39.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.365+0000 7fb4bb800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b415fc90 con 0x7fb4b40cb490 2024-02-19T15:22:39.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.365+0000 7fb4bb800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4b415ff70 con 0x7fb4b415bd90 2024-02-19T15:22:39.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.365+0000 7fb4bb800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b4160250 con 0x7fb4b4072850 2024-02-19T15:22:39.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.365+0000 7fb4b8e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb4b415bd90 0x7fb4b4070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:39.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.365+0000 7fb4b9800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4b4072850 0x7fb4b406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:39.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.365+0000 7fb4b9800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4b4072850 0x7fb4b406cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:51026/0 (socket says 172.21.15.47:51026) 2024-02-19T15:22:39.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.365+0000 7fb4b9800640 1 -- 172.21.15.47:0/3020035185 learned_addr learned my addr 172.21.15.47:0/3020035185 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:39.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.365+0000 7fb4abe00640 1 --1- 172.21.15.47:0/3020035185 >> v1:172.21.15.116:6789/0 conn(0x7fb4b40cb490 0x7fb4b4072140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:42366/0 (socket says 172.21.15.47:42366) 2024-02-19T15:22:39.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.369+0000 7fb4b9800640 1 -- 172.21.15.47:0/3020035185 >> v2:172.21.15.116:3300/0 conn(0x7fb4b415bd90 msgr2=0x7fb4b4070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:39.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.369+0000 7fb4b9800640 1 --2- 172.21.15.47:0/3020035185 >> v2:172.21.15.116:3300/0 conn(0x7fb4b415bd90 0x7fb4b4070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.369+0000 7fb4b9800640 1 -- 172.21.15.47:0/3020035185 >> v1:172.21.15.116:6789/0 conn(0x7fb4b40cb490 legacy=0x7fb4b4072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:39.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.369+0000 7fb4b9800640 1 -- 172.21.15.47:0/3020035185 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b4160530 con 0x7fb4b4072850 2024-02-19T15:22:39.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.369+0000 7fb4b3e00640 1 -- 172.21.15.47:0/3020035185 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419038875 0 0) 0x7fb4b415fc90 con 0x7fb4b40cb490 2024-02-19T15:22:39.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.369+0000 7fb4b9800640 1 --2- 172.21.15.47:0/3020035185 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4b4072850 0x7fb4b406cc60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009e50 tx=0x7fb4ac01c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6cdbd10ff3b1234 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:39.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.369+0000 7fb4b3e00640 1 -- 172.21.15.47:0/3020035185 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac0a56e0 con 0x7fb4b4072850 2024-02-19T15:22:39.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.369+0000 7fb4b3e00640 1 -- 172.21.15.47:0/3020035185 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ac0a5840 con 0x7fb4b4072850 2024-02-19T15:22:39.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.369+0000 7fb4b3e00640 1 -- 172.21.15.47:0/3020035185 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac0a5bd0 con 0x7fb4b4072850 2024-02-19T15:22:39.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.369+0000 7fb4bb800640 1 -- 172.21.15.47:0/3020035185 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4b4072850 msgr2=0x7fb4b406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:39.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.369+0000 7fb4bb800640 1 --2- 172.21.15.47:0/3020035185 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4b4072850 0x7fb4b406cc60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009e50 tx=0x7fb4ac01c910 comp rx=0 tx=0).stop 2024-02-19T15:22:39.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4bb800640 1 -- 172.21.15.47:0/3020035185 shutdown_connections 2024-02-19T15:22:39.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4bb800640 1 --2- 172.21.15.47:0/3020035185 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4b4072850 0x7fb4b406cc60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4bb800640 1 --2- 172.21.15.47:0/3020035185 >> v2:172.21.15.116:3300/0 conn(0x7fb4b415bd90 0x7fb4b4070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4bb800640 1 -- 172.21.15.47:0/3020035185 >> 172.21.15.47:0/3020035185 conn(0x7fb4b40cfc20 msgr2=0x7fb4b40d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:39.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4bb800640 1 -- 172.21.15.47:0/3020035185 shutdown_connections 2024-02-19T15:22:39.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4bb800640 1 -- 172.21.15.47:0/3020035185 wait complete. 2024-02-19T15:22:39.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4bb800640 1 Processor -- start 2024-02-19T15:22:39.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4bb800640 1 -- start start 2024-02-19T15:22:39.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4bb800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4b4072850 0x7fb4b40c5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:39.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4bb800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4b40cb490 0x7fb4b41586e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:39.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4bb800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b415bd90 0x7fb4b4158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:39.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4bb800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b4156590 con 0x7fb4b415bd90 2024-02-19T15:22:39.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4bb800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b40c8870 con 0x7fb4b40cb490 2024-02-19T15:22:39.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4bb800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b4155f40 con 0x7fb4b4072850 2024-02-19T15:22:39.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4b9800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b415bd90 0x7fb4b4158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:39.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4b8e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4b4072850 0x7fb4b40c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:39.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4abe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4b40cb490 0x7fb4b41586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:39.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4b9800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b415bd90 0x7fb4b4158c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44832/0 (socket says 172.21.15.47:44832) 2024-02-19T15:22:39.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4b9800640 1 -- 172.21.15.47:0/1959743385 learned_addr learned my addr 172.21.15.47:0/1959743385 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:39.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4b8e00640 1 -- 172.21.15.47:0/1959743385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4b40cb490 msgr2=0x7fb4b41586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:39.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4b8e00640 1 --2- 172.21.15.47:0/1959743385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4b40cb490 0x7fb4b41586e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4b8e00640 1 -- 172.21.15.47:0/1959743385 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b415bd90 msgr2=0x7fb4b4158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:39.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4b8e00640 1 --2- 172.21.15.47:0/1959743385 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b415bd90 0x7fb4b4158c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4b8e00640 1 -- 172.21.15.47:0/1959743385 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ac009b30 con 0x7fb4b4072850 2024-02-19T15:22:39.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4b9800640 1 --2- 172.21.15.47:0/1959743385 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b415bd90 0x7fb4b4158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:39.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4abe00640 1 --2- 172.21.15.47:0/1959743385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4b40cb490 0x7fb4b41586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:39.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.373+0000 7fb4b8e00640 1 --2- 172.21.15.47:0/1959743385 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4b4072850 0x7fb4b40c5890 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb4a400e870 tx=0x7fb4a400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:39.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.377+0000 7fb4b1600640 1 -- 172.21.15.47:0/1959743385 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a40095d0 con 0x7fb4b4072850 2024-02-19T15:22:39.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.377+0000 7fb4b1600640 1 -- 172.21.15.47:0/1959743385 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a400c940 con 0x7fb4b4072850 2024-02-19T15:22:39.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.377+0000 7fb4b1600640 1 -- 172.21.15.47:0/1959743385 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a400ccb0 con 0x7fb4b4072850 2024-02-19T15:22:39.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.377+0000 7fb4bb800640 1 -- 172.21.15.47:0/1959743385 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b4156220 con 0x7fb4b4072850 2024-02-19T15:22:39.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.377+0000 7fb4bb800640 1 -- 172.21.15.47:0/1959743385 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b40c5dd0 con 0x7fb4b4072850 2024-02-19T15:22:39.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.377+0000 7fb4bb800640 1 -- 172.21.15.47:0/1959743385 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b4075360 con 0x7fb4b4072850 2024-02-19T15:22:39.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.377+0000 7fb4b1600640 1 -- 172.21.15.47:0/1959743385 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb4a40059a0 con 0x7fb4b4072850 2024-02-19T15:22:39.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.377+0000 7fb4b1600640 1 --2- 172.21.15.47:0/1959743385 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb490066e40 0x7fb490069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:39.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.377+0000 7fb4b1600640 1 -- 172.21.15.47:0/1959743385 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4a4084a30 con 0x7fb4b4072850 2024-02-19T15:22:39.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.377+0000 7fb4abe00640 1 --2- 172.21.15.47:0/1959743385 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb490066e40 0x7fb490069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:39.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.381+0000 7fb4abe00640 1 --2- 172.21.15.47:0/1959743385 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb490066e40 0x7fb490069300 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb49c007bc0 tx=0x7fb49c007b10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:39.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.381+0000 7fb4b1600640 1 -- 172.21.15.47:0/1959743385 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb4a4051a40 con 0x7fb4b4072850 2024-02-19T15:22:39.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.541+0000 7fb4bb800640 1 -- 172.21.15.47:0/1959743385 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb4b4071180 con 0x7fb4b4072850 2024-02-19T15:22:39.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.541+0000 7fb4b1600640 1 -- 172.21.15.47:0/1959743385 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb4a40170a0 con 0x7fb4b4072850 2024-02-19T15:22:39.545 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:22:39.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.545+0000 7fb4bb800640 1 -- 172.21.15.47:0/1959743385 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb490066e40 msgr2=0x7fb490069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:39.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.545+0000 7fb4bb800640 1 --2- 172.21.15.47:0/1959743385 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb490066e40 0x7fb490069300 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb49c007bc0 tx=0x7fb49c007b10 comp rx=0 tx=0).stop 2024-02-19T15:22:39.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.545+0000 7fb4bb800640 1 -- 172.21.15.47:0/1959743385 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4b4072850 msgr2=0x7fb4b40c5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:39.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.545+0000 7fb4bb800640 1 --2- 172.21.15.47:0/1959743385 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4b4072850 0x7fb4b40c5890 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb4a400e870 tx=0x7fb4a400ed30 comp rx=0 tx=0).stop 2024-02-19T15:22:39.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.545+0000 7fb4bb800640 1 -- 172.21.15.47:0/1959743385 shutdown_connections 2024-02-19T15:22:39.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.545+0000 7fb4bb800640 1 --2- 172.21.15.47:0/1959743385 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb490066e40 0x7fb490069300 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.545+0000 7fb4bb800640 1 --2- 172.21.15.47:0/1959743385 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4b415bd90 0x7fb4b4158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.545+0000 7fb4bb800640 1 --2- 172.21.15.47:0/1959743385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4b40cb490 0x7fb4b41586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.545+0000 7fb4bb800640 1 --2- 172.21.15.47:0/1959743385 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4b4072850 0x7fb4b40c5890 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.545+0000 7fb4bb800640 1 -- 172.21.15.47:0/1959743385 >> 172.21.15.47:0/1959743385 conn(0x7fb4b40cfc20 msgr2=0x7fb4b415ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:39.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.545+0000 7fb4bb800640 1 -- 172.21.15.47:0/1959743385 shutdown_connections 2024-02-19T15:22:39.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.545+0000 7fb4bb800640 1 -- 172.21.15.47:0/1959743385 wait complete. 2024-02-19T15:22:39.566 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:22:39.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.705+0000 7fbf9aa00640 1 Processor -- start 2024-02-19T15:22:39.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.705+0000 7fbf9aa00640 1 -- start start 2024-02-19T15:22:39.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.705+0000 7fbf9aa00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbf9415bd90 0x7fbf94070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:39.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.705+0000 7fbf9aa00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbf940cb490 0x7fbf94072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:39.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.705+0000 7fbf9aa00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbf94072680 0x7fbf940ca270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:39.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.705+0000 7fbf9aa00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf9415fc90 con 0x7fbf94072680 2024-02-19T15:22:39.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.705+0000 7fbf9aa00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf9415ff70 con 0x7fbf9415bd90 2024-02-19T15:22:39.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.705+0000 7fbf9aa00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf94160250 con 0x7fbf940cb490 2024-02-19T15:22:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.705+0000 7fbf98a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbf94072680 0x7fbf940ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.705+0000 7fbf98a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbf94072680 0x7fbf940ca270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44838/0 (socket says 172.21.15.47:44838) 2024-02-19T15:22:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.705+0000 7fbf98a00640 1 -- 172.21.15.47:0/1612934526 learned_addr learned my addr 172.21.15.47:0/1612934526 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.705+0000 7fbf93e00640 1 --2- 172.21.15.47:0/1612934526 >> v2:172.21.15.116:3300/0 conn(0x7fbf9415bd90 0x7fbf94070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.705+0000 7fbf8b400640 1 --2- 172.21.15.47:0/1612934526 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbf940cb490 0x7fbf94072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.705+0000 7fbf98a00640 1 -- 172.21.15.47:0/1612934526 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbf940cb490 msgr2=0x7fbf94072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.705+0000 7fbf98a00640 1 --2- 172.21.15.47:0/1612934526 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbf940cb490 0x7fbf94072140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.705+0000 7fbf98a00640 1 -- 172.21.15.47:0/1612934526 >> v2:172.21.15.116:3300/0 conn(0x7fbf9415bd90 msgr2=0x7fbf94070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.705+0000 7fbf98a00640 1 --2- 172.21.15.47:0/1612934526 >> v2:172.21.15.116:3300/0 conn(0x7fbf9415bd90 0x7fbf94070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.705+0000 7fbf98a00640 1 -- 172.21.15.47:0/1612934526 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf94160530 con 0x7fbf94072680 2024-02-19T15:22:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.705+0000 7fbf93e00640 1 --2- 172.21.15.47:0/1612934526 >> v2:172.21.15.116:3300/0 conn(0x7fbf9415bd90 0x7fbf94070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:39.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.709+0000 7fbf98a00640 1 --2- 172.21.15.47:0/1612934526 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf94072680 0x7fbf940ca270 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbf9406c300 tx=0x7fbf8c01c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ea88ac7bbbce753 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:39.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.709+0000 7fbf93400640 1 -- 172.21.15.47:0/1612934526 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c0a8410 con 0x7fbf94072680 2024-02-19T15:22:39.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.709+0000 7fbf93400640 1 -- 172.21.15.47:0/1612934526 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8c0a8570 con 0x7fbf94072680 2024-02-19T15:22:39.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.709+0000 7fbf93400640 1 -- 172.21.15.47:0/1612934526 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c0a8900 con 0x7fbf94072680 2024-02-19T15:22:39.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.709+0000 7fbf9aa00640 1 -- 172.21.15.47:0/1612934526 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf94072680 msgr2=0x7fbf940ca270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:39.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.709+0000 7fbf9aa00640 1 --2- 172.21.15.47:0/1612934526 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf94072680 0x7fbf940ca270 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbf9406c300 tx=0x7fbf8c01c5d0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.709+0000 7fbf9aa00640 1 -- 172.21.15.47:0/1612934526 shutdown_connections 2024-02-19T15:22:39.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.709+0000 7fbf9aa00640 1 --2- 172.21.15.47:0/1612934526 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf94072680 0x7fbf940ca270 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.709+0000 7fbf9aa00640 1 --2- 172.21.15.47:0/1612934526 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbf940cb490 0x7fbf94072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.709+0000 7fbf9aa00640 1 --2- 172.21.15.47:0/1612934526 >> v2:172.21.15.116:3300/0 conn(0x7fbf9415bd90 0x7fbf94070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.709+0000 7fbf9aa00640 1 -- 172.21.15.47:0/1612934526 >> 172.21.15.47:0/1612934526 conn(0x7fbf940cfc20 msgr2=0x7fbf940d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:39.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.709+0000 7fbf9aa00640 1 -- 172.21.15.47:0/1612934526 shutdown_connections 2024-02-19T15:22:39.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.709+0000 7fbf9aa00640 1 -- 172.21.15.47:0/1612934526 wait complete. 2024-02-19T15:22:39.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf9aa00640 1 Processor -- start 2024-02-19T15:22:39.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf9aa00640 1 -- start start 2024-02-19T15:22:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf9aa00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbf94072680 0x7fbf94153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf9aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf940cb490 0x7fbf940c57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf9aa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf9415bd90 0x7fbf94157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf9aa00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf941539a0 con 0x7fbf9415bd90 2024-02-19T15:22:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf9aa00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf94157bf0 con 0x7fbf940cb490 2024-02-19T15:22:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf9aa00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf94153000 con 0x7fbf94072680 2024-02-19T15:22:39.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf98a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf9415bd90 0x7fbf94157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:39.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf98a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf9415bd90 0x7fbf94157220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44854/0 (socket says 172.21.15.47:44854) 2024-02-19T15:22:39.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf98a00640 1 -- 172.21.15.47:0/4146591710 learned_addr learned my addr 172.21.15.47:0/4146591710 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:39.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf93e00640 1 --2- 172.21.15.47:0/4146591710 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbf94072680 0x7fbf94153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:39.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf8b400640 1 --2- 172.21.15.47:0/4146591710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf940cb490 0x7fbf940c57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:39.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf98a00640 1 -- 172.21.15.47:0/4146591710 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbf94072680 msgr2=0x7fbf94153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:39.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf98a00640 1 --2- 172.21.15.47:0/4146591710 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbf94072680 0x7fbf94153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf98a00640 1 -- 172.21.15.47:0/4146591710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf940cb490 msgr2=0x7fbf940c57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:39.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf98a00640 1 --2- 172.21.15.47:0/4146591710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf940cb490 0x7fbf940c57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf98a00640 1 -- 172.21.15.47:0/4146591710 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf84009420 con 0x7fbf9415bd90 2024-02-19T15:22:39.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf98a00640 1 --2- 172.21.15.47:0/4146591710 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf9415bd90 0x7fbf94157220 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c01ac80 tx=0x7fbf8c019370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:39.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf90c00640 1 -- 172.21.15.47:0/4146591710 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c0b6070 con 0x7fbf9415bd90 2024-02-19T15:22:39.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf90c00640 1 -- 172.21.15.47:0/4146591710 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8c0a8d10 con 0x7fbf9415bd90 2024-02-19T15:22:39.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf9aa00640 1 -- 172.21.15.47:0/4146591710 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf8c009b30 con 0x7fbf9415bd90 2024-02-19T15:22:39.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf90c00640 1 -- 172.21.15.47:0/4146591710 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c0a8410 con 0x7fbf9415bd90 2024-02-19T15:22:39.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.713+0000 7fbf9aa00640 1 -- 172.21.15.47:0/4146591710 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf941535c0 con 0x7fbf9415bd90 2024-02-19T15:22:39.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.717+0000 7fbf9aa00640 1 -- 172.21.15.47:0/4146591710 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf94153ae0 con 0x7fbf9415bd90 2024-02-19T15:22:39.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.717+0000 7fbf90c00640 1 -- 172.21.15.47:0/4146591710 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fbf8c0ba070 con 0x7fbf9415bd90 2024-02-19T15:22:39.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.717+0000 7fbf90c00640 1 --2- 172.21.15.47:0/4146591710 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbf70066e70 0x7fbf70069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:39.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.717+0000 7fbf90c00640 1 -- 172.21.15.47:0/4146591710 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf8c11eb00 con 0x7fbf9415bd90 2024-02-19T15:22:39.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.717+0000 7fbf93e00640 1 --2- 172.21.15.47:0/4146591710 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbf70066e70 0x7fbf70069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:39.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.717+0000 7fbf93e00640 1 --2- 172.21.15.47:0/4146591710 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbf70066e70 0x7fbf70069330 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009c50 tx=0x7fbf84009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:39.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.721+0000 7fbf90c00640 1 -- 172.21.15.47:0/4146591710 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fbf94153ae0 con 0x7fbf9415bd90 2024-02-19T15:22:39.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.885+0000 7fbf9aa00640 1 -- 172.21.15.47:0/4146591710 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbf94153ae0 con 0x7fbf9415bd90 2024-02-19T15:22:39.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.885+0000 7fbf90c00640 1 -- 172.21.15.47:0/4146591710 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbf8c0ac020 con 0x7fbf9415bd90 2024-02-19T15:22:39.887 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:22:39.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.885+0000 7fbf9aa00640 1 -- 172.21.15.47:0/4146591710 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbf70066e70 msgr2=0x7fbf70069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:39.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.885+0000 7fbf9aa00640 1 --2- 172.21.15.47:0/4146591710 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbf70066e70 0x7fbf70069330 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009c50 tx=0x7fbf84009190 comp rx=0 tx=0).stop 2024-02-19T15:22:39.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.885+0000 7fbf9aa00640 1 -- 172.21.15.47:0/4146591710 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf9415bd90 msgr2=0x7fbf94157220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:39.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.885+0000 7fbf9aa00640 1 --2- 172.21.15.47:0/4146591710 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf9415bd90 0x7fbf94157220 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c01ac80 tx=0x7fbf8c019370 comp rx=0 tx=0).stop 2024-02-19T15:22:39.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.885+0000 7fbf9aa00640 1 -- 172.21.15.47:0/4146591710 shutdown_connections 2024-02-19T15:22:39.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.885+0000 7fbf9aa00640 1 --2- 172.21.15.47:0/4146591710 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbf70066e70 0x7fbf70069330 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.885+0000 7fbf9aa00640 1 --2- 172.21.15.47:0/4146591710 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf9415bd90 0x7fbf94157220 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.885+0000 7fbf9aa00640 1 --2- 172.21.15.47:0/4146591710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf940cb490 0x7fbf940c57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.885+0000 7fbf9aa00640 1 --2- 172.21.15.47:0/4146591710 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbf94072680 0x7fbf94153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:39.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.885+0000 7fbf9aa00640 1 -- 172.21.15.47:0/4146591710 >> 172.21.15.47:0/4146591710 conn(0x7fbf940cfc20 msgr2=0x7fbf940c7670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:39.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.885+0000 7fbf9aa00640 1 -- 172.21.15.47:0/4146591710 shutdown_connections 2024-02-19T15:22:39.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:39.885+0000 7fbf9aa00640 1 -- 172.21.15.47:0/4146591710 wait complete. 2024-02-19T15:22:39.908 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-19T15:22:40.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.045+0000 7fb9d4200640 1 Processor -- start 2024-02-19T15:22:40.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.045+0000 7fb9d4200640 1 -- start start 2024-02-19T15:22:40.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.049+0000 7fb9d4200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9cc15dfa0 0x7fb9cc06fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:40.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.049+0000 7fb9d4200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9cc06ff60 con 0x7fb9cc0ce400 2024-02-19T15:22:40.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.049+0000 7fb9d4200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9cc0d1070 con 0x7fb9cc0b4240 2024-02-19T15:22:40.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.049+0000 7fb9d4200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc06ec60 con 0x7fb9cc15dfa0 2024-02-19T15:22:40.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.049+0000 7fb9d2200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb9cc0ce400 0x7fb9cc0cc1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38142/0 (socket says 172.21.15.47:38142) 2024-02-19T15:22:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.049+0000 7fb9d2200640 1 -- 172.21.15.47:0/2729647722 learned_addr learned my addr 172.21.15.47:0/2729647722 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.049+0000 7fb9d1800640 1 --2- 172.21.15.47:0/2729647722 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9cc15dfa0 0x7fb9cc06fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.049+0000 7fb9cbe00640 1 -- 172.21.15.47:0/2729647722 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567566262 0 0) 0x7fb9cc06ff60 con 0x7fb9cc0ce400 2024-02-19T15:22:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.049+0000 7fb9cbe00640 1 -- 172.21.15.47:0/2729647722 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b4003530 con 0x7fb9cc0ce400 2024-02-19T15:22:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.049+0000 7fb9cbe00640 1 -- 172.21.15.47:0/2729647722 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053049756 0 0) 0x7fb9cc0d1070 con 0x7fb9cc0b4240 2024-02-19T15:22:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.049+0000 7fb9cbe00640 1 -- 172.21.15.47:0/2729647722 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9cc06ff60 con 0x7fb9cc0b4240 2024-02-19T15:22:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.049+0000 7fb9d1800640 1 -- 172.21.15.47:0/2729647722 >> v1:172.21.15.116:6789/0 conn(0x7fb9cc0b4240 legacy=0x7fb9cc0d0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.049+0000 7fb9d1800640 1 -- 172.21.15.47:0/2729647722 >> v1:172.21.15.47:6789/0 conn(0x7fb9cc0ce400 legacy=0x7fb9cc0cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.049+0000 7fb9cbe00640 1 -- 172.21.15.47:0/2729647722 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3282149927 0 0) 0x7fb9b4003530 con 0x7fb9cc0ce400 2024-02-19T15:22:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.049+0000 7fb9d1800640 1 -- 172.21.15.47:0/2729647722 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9cc06ef40 con 0x7fb9cc15dfa0 2024-02-19T15:22:40.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.053+0000 7fb9d1800640 1 --2- 172.21.15.47:0/2729647722 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9cc15dfa0 0x7fb9cc06fa20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00097f0 tx=0x7fb9c001c3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1e054f25653bb4f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:40.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.053+0000 7fb9cbe00640 1 -- 172.21.15.47:0/2729647722 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4001a00 con 0x7fb9cc15dfa0 2024-02-19T15:22:40.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.053+0000 7fb9cbe00640 1 -- 172.21.15.47:0/2729647722 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c00a7050 con 0x7fb9cc15dfa0 2024-02-19T15:22:40.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.053+0000 7fb9cbe00640 1 -- 172.21.15.47:0/2729647722 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c00ab620 con 0x7fb9cc15dfa0 2024-02-19T15:22:40.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.053+0000 7fb9d4200640 1 -- 172.21.15.47:0/2729647722 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9cc15dfa0 msgr2=0x7fb9cc06fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:40.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.053+0000 7fb9d4200640 1 --2- 172.21.15.47:0/2729647722 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9cc15dfa0 0x7fb9cc06fa20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00097f0 tx=0x7fb9c001c3d0 comp rx=0 tx=0).stop 2024-02-19T15:22:40.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.053+0000 7fb9d4200640 1 -- 172.21.15.47:0/2729647722 shutdown_connections 2024-02-19T15:22:40.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.053+0000 7fb9d4200640 1 --2- 172.21.15.47:0/2729647722 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9cc15dfa0 0x7fb9cc06fa20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:40.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.053+0000 7fb9d4200640 1 -- 172.21.15.47:0/2729647722 >> 172.21.15.47:0/2729647722 conn(0x7fb9cc0b3690 msgr2=0x7fb9cc0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:40.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.053+0000 7fb9d4200640 1 -- 172.21.15.47:0/2729647722 shutdown_connections 2024-02-19T15:22:40.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.053+0000 7fb9d4200640 1 -- 172.21.15.47:0/2729647722 wait complete. 2024-02-19T15:22:40.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.053+0000 7fb9d4200640 1 Processor -- start 2024-02-19T15:22:40.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.053+0000 7fb9d4200640 1 -- start start 2024-02-19T15:22:40.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9d4200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9cc0b4240 0x7fb9cc155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:40.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9d4200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9cc0ce400 0x7fb9cc156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:40.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9d4200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9cc15dfa0 0x7fb9cc0c7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:40.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9d4200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc160b40 con 0x7fb9cc0ce400 2024-02-19T15:22:40.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9d4200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc1561b0 con 0x7fb9cc15dfa0 2024-02-19T15:22:40.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9d4200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc156a90 con 0x7fb9cc0b4240 2024-02-19T15:22:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9d0e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9cc0ce400 0x7fb9cc156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9d1800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9cc0b4240 0x7fb9cc155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9d0e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9cc0ce400 0x7fb9cc156550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44870/0 (socket says 172.21.15.47:44870) 2024-02-19T15:22:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9d0e00640 1 -- 172.21.15.47:0/2039116995 learned_addr learned my addr 172.21.15.47:0/2039116995 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9d2200640 1 --2- 172.21.15.47:0/2039116995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9cc15dfa0 0x7fb9cc0c7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9d0e00640 1 -- 172.21.15.47:0/2039116995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9cc0b4240 msgr2=0x7fb9cc155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9d0e00640 1 --2- 172.21.15.47:0/2039116995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9cc0b4240 0x7fb9cc155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9d0e00640 1 -- 172.21.15.47:0/2039116995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9cc15dfa0 msgr2=0x7fb9cc0c7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9d0e00640 1 --2- 172.21.15.47:0/2039116995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9cc15dfa0 0x7fb9cc0c7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9d0e00640 1 -- 172.21.15.47:0/2039116995 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c0009420 con 0x7fb9cc0ce400 2024-02-19T15:22:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9d0e00640 1 --2- 172.21.15.47:0/2039116995 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9cc0ce400 0x7fb9cc156550 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb9b800c490 tx=0x7fb9b800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9c9600640 1 -- 172.21.15.47:0/2039116995 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b80052d0 con 0x7fb9cc0ce400 2024-02-19T15:22:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9c9600640 1 -- 172.21.15.47:0/2039116995 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b8005430 con 0x7fb9cc0ce400 2024-02-19T15:22:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9d4200640 1 -- 172.21.15.47:0/2039116995 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9cc161450 con 0x7fb9cc0ce400 2024-02-19T15:22:40.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9c9600640 1 -- 172.21.15.47:0/2039116995 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8012870 con 0x7fb9cc0ce400 2024-02-19T15:22:40.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9d4200640 1 -- 172.21.15.47:0/2039116995 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9cc0c70d0 con 0x7fb9cc0ce400 2024-02-19T15:22:40.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9d4200640 1 -- 172.21.15.47:0/2039116995 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9cc075360 con 0x7fb9cc0ce400 2024-02-19T15:22:40.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.057+0000 7fb9c9600640 1 -- 172.21.15.47:0/2039116995 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb9b80129d0 con 0x7fb9cc0ce400 2024-02-19T15:22:40.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.061+0000 7fb9c9600640 1 --2- 172.21.15.47:0/2039116995 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb9ac066e70 0x7fb9ac069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:40.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.061+0000 7fb9c9600640 1 -- 172.21.15.47:0/2039116995 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb9b8084470 con 0x7fb9cc0ce400 2024-02-19T15:22:40.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.061+0000 7fb9d1800640 1 --2- 172.21.15.47:0/2039116995 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb9ac066e70 0x7fb9ac069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:40.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.061+0000 7fb9d1800640 1 --2- 172.21.15.47:0/2039116995 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb9ac066e70 0x7fb9ac069330 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00ad000 tx=0x7fb9c00a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:40.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.065+0000 7fb9c9600640 1 -- 172.21.15.47:0/2039116995 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb9b8054e40 con 0x7fb9cc0ce400 2024-02-19T15:22:40.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.241+0000 7fb9d4200640 1 -- 172.21.15.47:0/2039116995 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb9cc0c7ab0 con 0x7fb9cc0ce400 2024-02-19T15:22:40.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.241+0000 7fb9c9600640 1 -- 172.21.15.47:0/2039116995 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb9b80549d0 con 0x7fb9cc0ce400 2024-02-19T15:22:40.243 INFO:teuthology.orchestra.run.smithi047.stderr:listed 0 entries 2024-02-19T15:22:40.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.241+0000 7fb9d4200640 1 -- 172.21.15.47:0/2039116995 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb9ac066e70 msgr2=0x7fb9ac069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:40.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.241+0000 7fb9d4200640 1 --2- 172.21.15.47:0/2039116995 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb9ac066e70 0x7fb9ac069330 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00ad000 tx=0x7fb9c00a6000 comp rx=0 tx=0).stop 2024-02-19T15:22:40.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.241+0000 7fb9d4200640 1 -- 172.21.15.47:0/2039116995 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9cc0ce400 msgr2=0x7fb9cc156550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:40.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.241+0000 7fb9d4200640 1 --2- 172.21.15.47:0/2039116995 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9cc0ce400 0x7fb9cc156550 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb9b800c490 tx=0x7fb9b800c950 comp rx=0 tx=0).stop 2024-02-19T15:22:40.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.241+0000 7fb9d4200640 1 -- 172.21.15.47:0/2039116995 shutdown_connections 2024-02-19T15:22:40.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.241+0000 7fb9d4200640 1 --2- 172.21.15.47:0/2039116995 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb9ac066e70 0x7fb9ac069330 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:40.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.241+0000 7fb9d4200640 1 --2- 172.21.15.47:0/2039116995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9cc15dfa0 0x7fb9cc0c7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:40.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.241+0000 7fb9d4200640 1 --2- 172.21.15.47:0/2039116995 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9cc0ce400 0x7fb9cc156550 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:40.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.241+0000 7fb9d4200640 1 --2- 172.21.15.47:0/2039116995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9cc0b4240 0x7fb9cc155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:40.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.241+0000 7fb9d4200640 1 -- 172.21.15.47:0/2039116995 >> 172.21.15.47:0/2039116995 conn(0x7fb9cc0b3690 msgr2=0x7fb9cc06d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:40.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.241+0000 7fb9d4200640 1 -- 172.21.15.47:0/2039116995 shutdown_connections 2024-02-19T15:22:40.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.241+0000 7fb9d4200640 1 -- 172.21.15.47:0/2039116995 wait complete. 2024-02-19T15:22:40.264 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-02-19T15:22:40.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:40.265 DEBUG:teuthology.orchestra.run.smithi116:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-02-19T15:22:40.312 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2024-02-19T15:22:40.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:40.312 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-02-19T15:22:40.327 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:22:40.327 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:22:40.337 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:22:40.337 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:22:40.386 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:22:40.386 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:22:40.438 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:40.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.617+0000 7f0e11000640 1 Processor -- start 2024-02-19T15:22:40.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.617+0000 7f0e11000640 1 -- start start 2024-02-19T15:22:40.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.617+0000 7f0e11000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e0c0ccbd0 0x7f0e0c15be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:40.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.617+0000 7f0e11000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e0c0cd040 con 0x7f0e0c0d19e0 2024-02-19T15:22:40.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.617+0000 7f0e11000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e0c15cd40 con 0x7f0e0c0cec90 2024-02-19T15:22:40.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.617+0000 7f0e11000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c0ce280 con 0x7f0e0c0ccbd0 2024-02-19T15:22:40.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.617+0000 7f0e0b400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0e0c0d19e0 0x7f0e0c0d1dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38154/0 (socket says 172.21.15.47:38154) 2024-02-19T15:22:40.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.617+0000 7f0e0b400640 1 -- 172.21.15.47:0/3276087994 learned_addr learned my addr 172.21.15.47:0/3276087994 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:40.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.617+0000 7f0e0be00640 1 --2- 172.21.15.47:0/3276087994 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e0c0ccbd0 0x7f0e0c15be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:40.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.621+0000 7f0e0aa00640 1 -- 172.21.15.47:0/3276087994 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041400977 0 0) 0x7f0e0c0cd040 con 0x7f0e0c0d19e0 2024-02-19T15:22:40.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.621+0000 7f0e0aa00640 1 -- 172.21.15.47:0/3276087994 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0df0003530 con 0x7f0e0c0d19e0 2024-02-19T15:22:40.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.621+0000 7f0e0aa00640 1 -- 172.21.15.47:0/3276087994 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314627408 0 0) 0x7f0e0c15cd40 con 0x7f0e0c0cec90 2024-02-19T15:22:40.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.621+0000 7f0e0aa00640 1 -- 172.21.15.47:0/3276087994 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e0c0cd040 con 0x7f0e0c0cec90 2024-02-19T15:22:40.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.621+0000 7f0e0be00640 1 -- 172.21.15.47:0/3276087994 >> v1:172.21.15.116:6789/0 conn(0x7f0e0c0cec90 legacy=0x7f0e0c0cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:40.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.621+0000 7f0e0be00640 1 -- 172.21.15.47:0/3276087994 >> v1:172.21.15.47:6789/0 conn(0x7f0e0c0d19e0 legacy=0x7f0e0c0d1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:40.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.621+0000 7f0e0be00640 1 -- 172.21.15.47:0/3276087994 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0c15d020 con 0x7f0e0c0ccbd0 2024-02-19T15:22:40.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.621+0000 7f0e0be00640 1 --2- 172.21.15.47:0/3276087994 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e0c0ccbd0 0x7f0e0c15be50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0e04009dc0 tx=0x7f0e0401c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=382e6e57ffe3aa5d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:40.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.621+0000 7f0e0aa00640 1 -- 172.21.15.47:0/3276087994 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e040a5920 con 0x7f0e0c0ccbd0 2024-02-19T15:22:40.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.621+0000 7f0e0aa00640 1 -- 172.21.15.47:0/3276087994 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e040a5a80 con 0x7f0e0c0ccbd0 2024-02-19T15:22:40.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.621+0000 7f0e0aa00640 1 -- 172.21.15.47:0/3276087994 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e040a5e10 con 0x7f0e0c0ccbd0 2024-02-19T15:22:40.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.621+0000 7f0e11000640 1 -- 172.21.15.47:0/3276087994 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e0c0ccbd0 msgr2=0x7f0e0c15be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:40.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.621+0000 7f0e11000640 1 --2- 172.21.15.47:0/3276087994 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e0c0ccbd0 0x7f0e0c15be50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0e04009dc0 tx=0x7f0e0401c960 comp rx=0 tx=0).stop 2024-02-19T15:22:40.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e11000640 1 -- 172.21.15.47:0/3276087994 shutdown_connections 2024-02-19T15:22:40.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e11000640 1 --2- 172.21.15.47:0/3276087994 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e0c0ccbd0 0x7f0e0c15be50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:40.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e11000640 1 -- 172.21.15.47:0/3276087994 >> 172.21.15.47:0/3276087994 conn(0x7f0e0c09a9d0 msgr2=0x7f0e0c09ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:40.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e11000640 1 -- 172.21.15.47:0/3276087994 shutdown_connections 2024-02-19T15:22:40.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e11000640 1 -- 172.21.15.47:0/3276087994 wait complete. 2024-02-19T15:22:40.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e11000640 1 Processor -- start 2024-02-19T15:22:40.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e11000640 1 -- start start 2024-02-19T15:22:40.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e11000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e0c0ccbd0 0x7f0e0c0c9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:40.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e11000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e0c0cec90 0x7f0e0c0cb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:40.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e11000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e0c0d19e0 0x7f0e0c14f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:40.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e11000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c14f8d0 con 0x7f0e0c0ccbd0 2024-02-19T15:22:40.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e11000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c153a30 con 0x7f0e0c0cec90 2024-02-19T15:22:40.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e11000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c153120 con 0x7f0e0c0d19e0 2024-02-19T15:22:40.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e0b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e0c0ccbd0 0x7f0e0c0c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:40.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e0be00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e0c0d19e0 0x7f0e0c14f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:40.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e0b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e0c0ccbd0 0x7f0e0c0c9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44886/0 (socket says 172.21.15.47:44886) 2024-02-19T15:22:40.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e0b400640 1 -- 172.21.15.47:0/4153215212 learned_addr learned my addr 172.21.15.47:0/4153215212 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:40.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e03e00640 1 --2- 172.21.15.47:0/4153215212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e0c0cec90 0x7f0e0c0cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:40.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e0b400640 1 -- 172.21.15.47:0/4153215212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e0c0d19e0 msgr2=0x7f0e0c14f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:40.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e0b400640 1 --2- 172.21.15.47:0/4153215212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e0c0d19e0 0x7f0e0c14f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:40.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e0b400640 1 -- 172.21.15.47:0/4153215212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e0c0cec90 msgr2=0x7f0e0c0cb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:40.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e0b400640 1 --2- 172.21.15.47:0/4153215212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e0c0cec90 0x7f0e0c0cb110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:40.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e0b400640 1 -- 172.21.15.47:0/4153215212 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e04009a70 con 0x7f0e0c0ccbd0 2024-02-19T15:22:40.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e0be00640 1 --2- 172.21.15.47:0/4153215212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e0c0d19e0 0x7f0e0c14f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:40.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e0b400640 1 --2- 172.21.15.47:0/4153215212 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e0c0ccbd0 0x7f0e0c0c9500 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00c4d0 tx=0x7f0dfc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:40.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e03400640 1 -- 172.21.15.47:0/4153215212 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc005cf0 con 0x7f0e0c0ccbd0 2024-02-19T15:22:40.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e03400640 1 -- 172.21.15.47:0/4153215212 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dfc005e50 con 0x7f0e0c0ccbd0 2024-02-19T15:22:40.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e11000640 1 -- 172.21.15.47:0/4153215212 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e0c0c9af0 con 0x7f0e0c0ccbd0 2024-02-19T15:22:40.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.625+0000 7f0e03400640 1 -- 172.21.15.47:0/4153215212 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc0072b0 con 0x7f0e0c0ccbd0 2024-02-19T15:22:40.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.629+0000 7f0e11000640 1 -- 172.21.15.47:0/4153215212 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0c1536c0 con 0x7f0e0c0ccbd0 2024-02-19T15:22:40.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.629+0000 7f0e11000640 1 -- 172.21.15.47:0/4153215212 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e0c075360 con 0x7f0e0c0ccbd0 2024-02-19T15:22:40.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.629+0000 7f0e03400640 1 -- 172.21.15.47:0/4153215212 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0dfc007410 con 0x7f0e0c0ccbd0 2024-02-19T15:22:40.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.629+0000 7f0e03400640 1 --2- 172.21.15.47:0/4153215212 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0de8066e40 0x7f0de8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:40.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.629+0000 7f0e03400640 1 -- 172.21.15.47:0/4153215212 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0dfc084080 con 0x7f0e0c0ccbd0 2024-02-19T15:22:40.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.629+0000 7f0e03e00640 1 --2- 172.21.15.47:0/4153215212 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0de8066e40 0x7f0de8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:40.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.629+0000 7f0e03e00640 1 --2- 172.21.15.47:0/4153215212 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0de8066e40 0x7f0de8069300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c15b0c0 tx=0x7f0df4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:40.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.633+0000 7f0e03400640 1 -- 172.21.15.47:0/4153215212 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0dfc0b61c0 con 0x7f0e0c0ccbd0 2024-02-19T15:22:40.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.797+0000 7f0e11000640 1 -- 172.21.15.47:0/4153215212 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e0c167980 con 0x7f0e0c0ccbd0 2024-02-19T15:22:40.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.797+0000 7f0e03400640 1 -- 172.21.15.47:0/4153215212 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f0dfc017090 con 0x7f0e0c0ccbd0 2024-02-19T15:22:40.800 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:40.800 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":21,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:06.843155+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:03.091360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:04.222965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:40.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.797+0000 7f0e11000640 1 -- 172.21.15.47:0/4153215212 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0de8066e40 msgr2=0x7f0de8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:40.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.797+0000 7f0e11000640 1 --2- 172.21.15.47:0/4153215212 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0de8066e40 0x7f0de8069300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c15b0c0 tx=0x7f0df4009040 comp rx=0 tx=0).stop 2024-02-19T15:22:40.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.797+0000 7f0e11000640 1 -- 172.21.15.47:0/4153215212 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e0c0ccbd0 msgr2=0x7f0e0c0c9500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:40.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.797+0000 7f0e11000640 1 --2- 172.21.15.47:0/4153215212 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e0c0ccbd0 0x7f0e0c0c9500 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00c4d0 tx=0x7f0dfc00c990 comp rx=0 tx=0).stop 2024-02-19T15:22:40.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.797+0000 7f0e11000640 1 -- 172.21.15.47:0/4153215212 shutdown_connections 2024-02-19T15:22:40.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.797+0000 7f0e11000640 1 --2- 172.21.15.47:0/4153215212 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0de8066e40 0x7f0de8069300 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:40.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.797+0000 7f0e11000640 1 --2- 172.21.15.47:0/4153215212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e0c0d19e0 0x7f0e0c14f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:40.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.797+0000 7f0e11000640 1 --2- 172.21.15.47:0/4153215212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e0c0cec90 0x7f0e0c0cb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:40.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.797+0000 7f0e11000640 1 --2- 172.21.15.47:0/4153215212 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e0c0ccbd0 0x7f0e0c0c9500 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:40.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.797+0000 7f0e11000640 1 -- 172.21.15.47:0/4153215212 >> 172.21.15.47:0/4153215212 conn(0x7f0e0c09a9d0 msgr2=0x7f0e0c09adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:40.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.797+0000 7f0e11000640 1 -- 172.21.15.47:0/4153215212 shutdown_connections 2024-02-19T15:22:40.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.797+0000 7f0e11000640 1 -- 172.21.15.47:0/4153215212 wait complete. 2024-02-19T15:22:40.823 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:22:03.091360+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:22:04.222965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:22:40.823 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:22:40.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.965+0000 7f6b18e00640 1 Processor -- start 2024-02-19T15:22:40.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.965+0000 7f6b18e00640 1 -- start start 2024-02-19T15:22:40.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.965+0000 7f6b18e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6b14072850 0x7f6b14070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:40.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.965+0000 7f6b18e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1415fc90 con 0x7f6b1415bd90 2024-02-19T15:22:40.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.965+0000 7f6b18e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1415ff70 con 0x7f6b140cb490 2024-02-19T15:22:40.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.965+0000 7f6b18e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b14160250 con 0x7f6b14072850 2024-02-19T15:22:40.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.965+0000 7f6b11e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6b1415bd90 0x7f6b14072140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38164/0 (socket says 172.21.15.47:38164) 2024-02-19T15:22:40.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.965+0000 7f6b11e00640 1 -- 172.21.15.47:0/2575079536 learned_addr learned my addr 172.21.15.47:0/2575079536 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:40.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.965+0000 7f6b12800640 1 --2- 172.21.15.47:0/2575079536 >> v2:172.21.15.116:3300/0 conn(0x7f6b14072850 0x7f6b14070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:40.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.965+0000 7f6b12800640 1 -- 172.21.15.47:0/2575079536 >> v1:172.21.15.116:6789/0 conn(0x7f6b140cb490 legacy=0x7f6b1406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:40.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.965+0000 7f6b12800640 1 -- 172.21.15.47:0/2575079536 >> v1:172.21.15.47:6789/0 conn(0x7f6b1415bd90 legacy=0x7f6b14072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:40.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.965+0000 7f6b12800640 1 -- 172.21.15.47:0/2575079536 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b14160530 con 0x7f6b14072850 2024-02-19T15:22:40.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.969+0000 7f6b12800640 1 --2- 172.21.15.47:0/2575079536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b14072850 0x7f6b14070960 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c009d90 tx=0x7f6b0c01c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80dee76ed0dec12c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:40.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.969+0000 7f6b11400640 1 -- 172.21.15.47:0/2575079536 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c0a5820 con 0x7f6b14072850 2024-02-19T15:22:40.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.969+0000 7f6b11400640 1 -- 172.21.15.47:0/2575079536 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0c0a5980 con 0x7f6b14072850 2024-02-19T15:22:40.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.969+0000 7f6b11400640 1 -- 172.21.15.47:0/2575079536 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c0a5d10 con 0x7f6b14072850 2024-02-19T15:22:40.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.969+0000 7f6b18e00640 1 -- 172.21.15.47:0/2575079536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b14072850 msgr2=0x7f6b14070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:40.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.969+0000 7f6b18e00640 1 --2- 172.21.15.47:0/2575079536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b14072850 0x7f6b14070960 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c009d90 tx=0x7f6b0c01c850 comp rx=0 tx=0).stop 2024-02-19T15:22:40.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.969+0000 7f6b18e00640 1 -- 172.21.15.47:0/2575079536 shutdown_connections 2024-02-19T15:22:40.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.969+0000 7f6b18e00640 1 --2- 172.21.15.47:0/2575079536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b14072850 0x7f6b14070960 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:40.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.969+0000 7f6b18e00640 1 -- 172.21.15.47:0/2575079536 >> 172.21.15.47:0/2575079536 conn(0x7f6b140cfc20 msgr2=0x7f6b140d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:40.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.969+0000 7f6b18e00640 1 -- 172.21.15.47:0/2575079536 shutdown_connections 2024-02-19T15:22:40.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.969+0000 7f6b18e00640 1 -- 172.21.15.47:0/2575079536 wait complete. 2024-02-19T15:22:40.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.969+0000 7f6b18e00640 1 Processor -- start 2024-02-19T15:22:40.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.969+0000 7f6b18e00640 1 -- start start 2024-02-19T15:22:40.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b18e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b14072850 0x7f6b1414ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:40.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b18e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b140cb490 0x7f6b1415f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:40.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b18e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1415bd90 0x7f6b14150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:40.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b18e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1414e410 con 0x7f6b14072850 2024-02-19T15:22:40.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b18e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1415f5b0 con 0x7f6b1415bd90 2024-02-19T15:22:40.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b18e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b140cd830 con 0x7f6b140cb490 2024-02-19T15:22:40.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b11e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b14072850 0x7f6b1414ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:40.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b11e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b14072850 0x7f6b1414ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44888/0 (socket says 172.21.15.47:44888) 2024-02-19T15:22:40.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b11e00640 1 -- 172.21.15.47:0/403467963 learned_addr learned my addr 172.21.15.47:0/403467963 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:40.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b12800640 1 --2- 172.21.15.47:0/403467963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1415bd90 0x7f6b14150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:40.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b0be00640 1 --2- 172.21.15.47:0/403467963 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b140cb490 0x7f6b1415f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:40.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b11e00640 1 -- 172.21.15.47:0/403467963 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b140cb490 msgr2=0x7f6b1415f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:40.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b11e00640 1 --2- 172.21.15.47:0/403467963 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b140cb490 0x7f6b1415f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:40.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b11e00640 1 -- 172.21.15.47:0/403467963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1415bd90 msgr2=0x7f6b14150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:40.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b11e00640 1 --2- 172.21.15.47:0/403467963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1415bd90 0x7f6b14150970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:40.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b11e00640 1 -- 172.21.15.47:0/403467963 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0c009a70 con 0x7f6b14072850 2024-02-19T15:22:40.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b12800640 1 --2- 172.21.15.47:0/403467963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1415bd90 0x7f6b14150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:40.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b11e00640 1 --2- 172.21.15.47:0/403467963 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b14072850 0x7f6b1414ded0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6b0400c400 tx=0x7f6b0400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:40.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b0a000640 1 -- 172.21.15.47:0/403467963 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b04005cf0 con 0x7f6b14072850 2024-02-19T15:22:40.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b0a000640 1 -- 172.21.15.47:0/403467963 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b04005e50 con 0x7f6b14072850 2024-02-19T15:22:40.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b0a000640 1 -- 172.21.15.47:0/403467963 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b040072d0 con 0x7f6b14072850 2024-02-19T15:22:40.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b18e00640 1 -- 172.21.15.47:0/403467963 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b140cdb10 con 0x7f6b14072850 2024-02-19T15:22:40.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b18e00640 1 -- 172.21.15.47:0/403467963 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b140ce0d0 con 0x7f6b14072850 2024-02-19T15:22:40.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.973+0000 7f6b18e00640 1 -- 172.21.15.47:0/403467963 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b14075360 con 0x7f6b14072850 2024-02-19T15:22:40.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.977+0000 7f6b0a000640 1 -- 172.21.15.47:0/403467963 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6b04007430 con 0x7f6b14072850 2024-02-19T15:22:40.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.977+0000 7f6b0a000640 1 --2- 172.21.15.47:0/403467963 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6af0066e90 0x7f6af0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:40.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.977+0000 7f6b0a000640 1 -- 172.21.15.47:0/403467963 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b04083eb0 con 0x7f6b14072850 2024-02-19T15:22:40.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.977+0000 7f6b0be00640 1 --2- 172.21.15.47:0/403467963 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6af0066e90 0x7f6af0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:40.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.977+0000 7f6b0be00640 1 --2- 172.21.15.47:0/403467963 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6af0066e90 0x7f6af0069350 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6b140cb1d0 tx=0x7f6afc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:40.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:40.981+0000 7f6b0a000640 1 -- 172.21.15.47:0/403467963 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6b04050dc0 con 0x7f6b14072850 2024-02-19T15:22:41.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.145+0000 7f6b18e00640 1 -- 172.21.15.47:0/403467963 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6b14156ac0 con 0x7f6b14072850 2024-02-19T15:22:41.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.145+0000 7f6b0a000640 1 -- 172.21.15.47:0/403467963 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6b040170a0 con 0x7f6b14072850 2024-02-19T15:22:41.147 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:22:41.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.145+0000 7f6b18e00640 1 -- 172.21.15.47:0/403467963 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6af0066e90 msgr2=0x7f6af0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:41.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.145+0000 7f6b18e00640 1 --2- 172.21.15.47:0/403467963 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6af0066e90 0x7f6af0069350 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6b140cb1d0 tx=0x7f6afc009040 comp rx=0 tx=0).stop 2024-02-19T15:22:41.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.145+0000 7f6b18e00640 1 -- 172.21.15.47:0/403467963 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b14072850 msgr2=0x7f6b1414ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:41.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.145+0000 7f6b18e00640 1 --2- 172.21.15.47:0/403467963 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b14072850 0x7f6b1414ded0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6b0400c400 tx=0x7f6b0400c8c0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.145+0000 7f6b18e00640 1 -- 172.21.15.47:0/403467963 shutdown_connections 2024-02-19T15:22:41.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.145+0000 7f6b18e00640 1 --2- 172.21.15.47:0/403467963 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6af0066e90 0x7f6af0069350 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.145+0000 7f6b18e00640 1 --2- 172.21.15.47:0/403467963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1415bd90 0x7f6b14150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.145+0000 7f6b18e00640 1 --2- 172.21.15.47:0/403467963 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b140cb490 0x7f6b1415f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.145+0000 7f6b18e00640 1 --2- 172.21.15.47:0/403467963 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b14072850 0x7f6b1414ded0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.145+0000 7f6b18e00640 1 -- 172.21.15.47:0/403467963 >> 172.21.15.47:0/403467963 conn(0x7f6b140cfc20 msgr2=0x7f6b14159d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:41.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.145+0000 7f6b18e00640 1 -- 172.21.15.47:0/403467963 shutdown_connections 2024-02-19T15:22:41.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.145+0000 7f6b18e00640 1 -- 172.21.15.47:0/403467963 wait complete. 2024-02-19T15:22:41.168 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:22:41.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.309+0000 7fc2b2600640 1 Processor -- start 2024-02-19T15:22:41.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.309+0000 7fc2b2600640 1 -- start start 2024-02-19T15:22:41.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.309+0000 7fc2b2600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc2ac0ce3b0 0x7fc2ac0ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:41.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.309+0000 7fc2b2600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc2ac15e280 0x7fc2ac0cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:41.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.309+0000 7fc2b2600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2ac0cc620 con 0x7fc2ac0d2650 2024-02-19T15:22:41.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.309+0000 7fc2b2600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2ac0d12e0 con 0x7fc2ac15e280 2024-02-19T15:22:41.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.309+0000 7fc2b2600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2ac0d1940 con 0x7fc2ac0ce3b0 2024-02-19T15:22:41.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.309+0000 7fc2abe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc2ac15e280 0x7fc2ac0cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:41.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.309+0000 7fc2ab400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc2ac0ce3b0 0x7fc2ac0ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:41.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.309+0000 7fc2abe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc2ac15e280 0x7fc2ac0cd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44902/0 (socket says 172.21.15.47:44902) 2024-02-19T15:22:41.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.309+0000 7fc2abe00640 1 -- 172.21.15.47:0/1808694829 learned_addr learned my addr 172.21.15.47:0/1808694829 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:41.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.309+0000 7fc2ab400640 1 -- 172.21.15.47:0/1808694829 >> v2:172.21.15.47:3300/0 conn(0x7fc2ac15e280 msgr2=0x7fc2ac0cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:41.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.309+0000 7fc2ab400640 1 --2- 172.21.15.47:0/1808694829 >> v2:172.21.15.47:3300/0 conn(0x7fc2ac15e280 0x7fc2ac0cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.309+0000 7fc2ab400640 1 -- 172.21.15.47:0/1808694829 >> v1:172.21.15.116:6789/0 conn(0x7fc2ac0d2650 legacy=0x7fc2ac0ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:41.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.309+0000 7fc2ab400640 1 -- 172.21.15.47:0/1808694829 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ac0cecd0 con 0x7fc2ac0ce3b0 2024-02-19T15:22:41.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.309+0000 7fc2abe00640 1 --2- 172.21.15.47:0/1808694829 >> v2:172.21.15.47:3300/0 conn(0x7fc2ac15e280 0x7fc2ac0cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:41.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.309+0000 7fc2aaa00640 1 -- 172.21.15.47:0/1808694829 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 884711674 0 0) 0x7fc2ac0cc620 con 0x7fc2ac0d2650 2024-02-19T15:22:41.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.313+0000 7fc2ab400640 1 --2- 172.21.15.47:0/1808694829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2ac0ce3b0 0x7fc2ac0ce790 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac071220 tx=0x7fc29c01c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56555845a0d873be server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:41.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.313+0000 7fc2aaa00640 1 -- 172.21.15.47:0/1808694829 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c0ab600 con 0x7fc2ac0ce3b0 2024-02-19T15:22:41.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.313+0000 7fc2aaa00640 1 -- 172.21.15.47:0/1808694829 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29c0a7050 con 0x7fc2ac0ce3b0 2024-02-19T15:22:41.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.313+0000 7fc2aaa00640 1 -- 172.21.15.47:0/1808694829 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c0ab980 con 0x7fc2ac0ce3b0 2024-02-19T15:22:41.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.313+0000 7fc2b2600640 1 -- 172.21.15.47:0/1808694829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2ac0ce3b0 msgr2=0x7fc2ac0ce790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:41.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.313+0000 7fc2b2600640 1 --2- 172.21.15.47:0/1808694829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2ac0ce3b0 0x7fc2ac0ce790 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac071220 tx=0x7fc29c01c360 comp rx=0 tx=0).stop 2024-02-19T15:22:41.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.313+0000 7fc2b2600640 1 -- 172.21.15.47:0/1808694829 shutdown_connections 2024-02-19T15:22:41.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.313+0000 7fc2b2600640 1 --2- 172.21.15.47:0/1808694829 >> v2:172.21.15.47:3300/0 conn(0x7fc2ac15e280 0x7fc2ac0cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.313+0000 7fc2b2600640 1 --2- 172.21.15.47:0/1808694829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2ac0ce3b0 0x7fc2ac0ce790 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.313+0000 7fc2b2600640 1 -- 172.21.15.47:0/1808694829 >> 172.21.15.47:0/1808694829 conn(0x7fc2ac0d1b30 msgr2=0x7fc2ac0d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:41.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.313+0000 7fc2b2600640 1 -- 172.21.15.47:0/1808694829 shutdown_connections 2024-02-19T15:22:41.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.313+0000 7fc2b2600640 1 -- 172.21.15.47:0/1808694829 wait complete. 2024-02-19T15:22:41.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2b2600640 1 Processor -- start 2024-02-19T15:22:41.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2b2600640 1 -- start start 2024-02-19T15:22:41.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2b2600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2ac0ce3b0 0x7fc2ac0b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:41.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2b2600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2ac0d2650 0x7fc2ac158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:41.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2b2600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2ac15e280 0x7fc2ac15af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:41.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2b2600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ac0b7150 con 0x7fc2ac0d2650 2024-02-19T15:22:41.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2b2600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ac159e80 con 0x7fc2ac15e280 2024-02-19T15:22:41.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2b2600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ac15a160 con 0x7fc2ac0ce3b0 2024-02-19T15:22:41.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2a2a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2ac0d2650 0x7fc2ac158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:41.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2a2a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2ac0d2650 0x7fc2ac158de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44912/0 (socket says 172.21.15.47:44912) 2024-02-19T15:22:41.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2a2a00640 1 -- 172.21.15.47:0/3666564153 learned_addr learned my addr 172.21.15.47:0/3666564153 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:41.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2ab400640 1 --2- 172.21.15.47:0/3666564153 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2ac0ce3b0 0x7fc2ac0b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:41.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2abe00640 1 --2- 172.21.15.47:0/3666564153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2ac15e280 0x7fc2ac15af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:41.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2a2a00640 1 -- 172.21.15.47:0/3666564153 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2ac0ce3b0 msgr2=0x7fc2ac0b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:41.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2a2a00640 1 --2- 172.21.15.47:0/3666564153 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2ac0ce3b0 0x7fc2ac0b33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2a2a00640 1 -- 172.21.15.47:0/3666564153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2ac15e280 msgr2=0x7fc2ac15af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:41.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2a2a00640 1 --2- 172.21.15.47:0/3666564153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2ac15e280 0x7fc2ac15af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2a2a00640 1 -- 172.21.15.47:0/3666564153 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a4009b30 con 0x7fc2ac0d2650 2024-02-19T15:22:41.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2abe00640 1 --2- 172.21.15.47:0/3666564153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2ac15e280 0x7fc2ac15af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:41.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2a2a00640 1 --2- 172.21.15.47:0/3666564153 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2ac0d2650 0x7fc2ac158de0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc294001ec0 tx=0x7fc29400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:41.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2a3e00640 1 -- 172.21.15.47:0/3666564153 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2940054e0 con 0x7fc2ac0d2650 2024-02-19T15:22:41.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2a3e00640 1 -- 172.21.15.47:0/3666564153 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc294003600 con 0x7fc2ac0d2650 2024-02-19T15:22:41.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2a3e00640 1 -- 172.21.15.47:0/3666564153 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2940039b0 con 0x7fc2ac0d2650 2024-02-19T15:22:41.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2b2600640 1 -- 172.21.15.47:0/3666564153 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc29c009420 con 0x7fc2ac0d2650 2024-02-19T15:22:41.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2b2600640 1 -- 172.21.15.47:0/3666564153 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2ac15a720 con 0x7fc2ac0d2650 2024-02-19T15:22:41.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.317+0000 7fc2b2600640 1 -- 172.21.15.47:0/3666564153 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2ac075360 con 0x7fc2ac0d2650 2024-02-19T15:22:41.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.321+0000 7fc2a3e00640 1 -- 172.21.15.47:0/3666564153 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc294003b10 con 0x7fc2ac0d2650 2024-02-19T15:22:41.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.321+0000 7fc2a3e00640 1 --2- 172.21.15.47:0/3666564153 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc288066e40 0x7fc288069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:41.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.321+0000 7fc2a3e00640 1 -- 172.21.15.47:0/3666564153 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc294084050 con 0x7fc2ac0d2650 2024-02-19T15:22:41.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.321+0000 7fc2ab400640 1 --2- 172.21.15.47:0/3666564153 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc288066e40 0x7fc288069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:41.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.321+0000 7fc2ab400640 1 --2- 172.21.15.47:0/3666564153 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc288066e40 0x7fc288069300 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc29c0ad000 tx=0x7fc29c0a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:41.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.325+0000 7fc2a3e00640 1 -- 172.21.15.47:0/3666564153 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc2940b61c0 con 0x7fc2ac0d2650 2024-02-19T15:22:41.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.485+0000 7fc2b2600640 1 -- 172.21.15.47:0/3666564153 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc2ac0cdf80 con 0x7fc2ac0d2650 2024-02-19T15:22:41.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.485+0000 7fc2a3e00640 1 -- 172.21.15.47:0/3666564153 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc294050e50 con 0x7fc2ac0d2650 2024-02-19T15:22:41.490 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:22:41.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.489+0000 7fc2b2600640 1 -- 172.21.15.47:0/3666564153 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc288066e40 msgr2=0x7fc288069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:41.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.489+0000 7fc2b2600640 1 --2- 172.21.15.47:0/3666564153 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc288066e40 0x7fc288069300 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc29c0ad000 tx=0x7fc29c0a6000 comp rx=0 tx=0).stop 2024-02-19T15:22:41.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.489+0000 7fc2b2600640 1 -- 172.21.15.47:0/3666564153 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2ac0d2650 msgr2=0x7fc2ac158de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:41.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.489+0000 7fc2b2600640 1 --2- 172.21.15.47:0/3666564153 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2ac0d2650 0x7fc2ac158de0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc294001ec0 tx=0x7fc29400c8e0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.489+0000 7fc2b2600640 1 -- 172.21.15.47:0/3666564153 shutdown_connections 2024-02-19T15:22:41.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.489+0000 7fc2b2600640 1 --2- 172.21.15.47:0/3666564153 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc288066e40 0x7fc288069300 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.489+0000 7fc2b2600640 1 --2- 172.21.15.47:0/3666564153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2ac15e280 0x7fc2ac15af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.489+0000 7fc2b2600640 1 --2- 172.21.15.47:0/3666564153 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2ac0d2650 0x7fc2ac158de0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.489+0000 7fc2b2600640 1 --2- 172.21.15.47:0/3666564153 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2ac0ce3b0 0x7fc2ac0b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.489+0000 7fc2b2600640 1 -- 172.21.15.47:0/3666564153 >> 172.21.15.47:0/3666564153 conn(0x7fc2ac0d1b30 msgr2=0x7fc2ac0c6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:41.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.489+0000 7fc2b2600640 1 -- 172.21.15.47:0/3666564153 shutdown_connections 2024-02-19T15:22:41.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.489+0000 7fc2b2600640 1 -- 172.21.15.47:0/3666564153 wait complete. 2024-02-19T15:22:41.512 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:22:41.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.649+0000 7fa409e00640 1 Processor -- start 2024-02-19T15:22:41.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.649+0000 7fa409e00640 1 -- start start 2024-02-19T15:22:41.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.653+0000 7fa409e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa40415bd90 0x7fa404070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:41.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.653+0000 7fa409e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa40415fc90 con 0x7fa4040cb490 2024-02-19T15:22:41.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.653+0000 7fa409e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa40415ff70 con 0x7fa404072850 2024-02-19T15:22:41.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.653+0000 7fa409e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa404160250 con 0x7fa40415bd90 2024-02-19T15:22:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.653+0000 7fa402a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa40415bd90 0x7fa404070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.653+0000 7fa402a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa40415bd90 0x7fa404070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44914/0 (socket says 172.21.15.47:44914) 2024-02-19T15:22:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.653+0000 7fa3fa000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa4040cb490 0x7fa404072140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38180/0 (socket says 172.21.15.47:38180) 2024-02-19T15:22:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.653+0000 7fa402a00640 1 -- 172.21.15.47:0/115975864 learned_addr learned my addr 172.21.15.47:0/115975864 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.653+0000 7fa402a00640 1 -- 172.21.15.47:0/115975864 >> v1:172.21.15.116:6789/0 conn(0x7fa404072850 legacy=0x7fa40406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.653+0000 7fa402a00640 1 -- 172.21.15.47:0/115975864 >> v1:172.21.15.47:6789/0 conn(0x7fa4040cb490 legacy=0x7fa404072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.653+0000 7fa402a00640 1 -- 172.21.15.47:0/115975864 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa404160530 con 0x7fa40415bd90 2024-02-19T15:22:41.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.657+0000 7fa402a00640 1 --2- 172.21.15.47:0/115975864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa40415bd90 0x7fa404070960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4009e50 tx=0x7fa3f401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=685d4917675fda97 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:41.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.657+0000 7fa402000640 1 -- 172.21.15.47:0/115975864 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f40a56e0 con 0x7fa40415bd90 2024-02-19T15:22:41.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.657+0000 7fa402000640 1 -- 172.21.15.47:0/115975864 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f40a5840 con 0x7fa40415bd90 2024-02-19T15:22:41.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.657+0000 7fa402000640 1 -- 172.21.15.47:0/115975864 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f40a5bd0 con 0x7fa40415bd90 2024-02-19T15:22:41.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.657+0000 7fa409e00640 1 -- 172.21.15.47:0/115975864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa40415bd90 msgr2=0x7fa404070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:41.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.657+0000 7fa409e00640 1 --2- 172.21.15.47:0/115975864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa40415bd90 0x7fa404070960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4009e50 tx=0x7fa3f401c910 comp rx=0 tx=0).stop 2024-02-19T15:22:41.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.657+0000 7fa409e00640 1 -- 172.21.15.47:0/115975864 shutdown_connections 2024-02-19T15:22:41.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.657+0000 7fa409e00640 1 --2- 172.21.15.47:0/115975864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa40415bd90 0x7fa404070960 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.657+0000 7fa409e00640 1 -- 172.21.15.47:0/115975864 >> 172.21.15.47:0/115975864 conn(0x7fa4040cfc20 msgr2=0x7fa4040d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:41.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.657+0000 7fa409e00640 1 -- 172.21.15.47:0/115975864 shutdown_connections 2024-02-19T15:22:41.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.657+0000 7fa409e00640 1 -- 172.21.15.47:0/115975864 wait complete. 2024-02-19T15:22:41.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.657+0000 7fa409e00640 1 Processor -- start 2024-02-19T15:22:41.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.657+0000 7fa409e00640 1 -- start start 2024-02-19T15:22:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.657+0000 7fa409e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa404072850 0x7fa4041509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.657+0000 7fa409e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4040cb490 0x7fa4040b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa409e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa40415bd90 0x7fa40415e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa409e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4041651f0 con 0x7fa404072850 2024-02-19T15:22:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa409e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4040c9040 con 0x7fa4040cb490 2024-02-19T15:22:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa409e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4040b4be0 con 0x7fa40415bd90 2024-02-19T15:22:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa402a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa404072850 0x7fa4041509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa402a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa404072850 0x7fa4041509b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44928/0 (socket says 172.21.15.47:44928) 2024-02-19T15:22:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa402a00640 1 -- 172.21.15.47:0/3282459108 learned_addr learned my addr 172.21.15.47:0/3282459108 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa3fa000640 1 --2- 172.21.15.47:0/3282459108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4040cb490 0x7fa4040b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa403400640 1 --2- 172.21.15.47:0/3282459108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa40415bd90 0x7fa40415e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa402a00640 1 -- 172.21.15.47:0/3282459108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa40415bd90 msgr2=0x7fa40415e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa402a00640 1 --2- 172.21.15.47:0/3282459108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa40415bd90 0x7fa40415e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa402a00640 1 -- 172.21.15.47:0/3282459108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4040cb490 msgr2=0x7fa4040b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa402a00640 1 --2- 172.21.15.47:0/3282459108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4040cb490 0x7fa4040b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa402a00640 1 -- 172.21.15.47:0/3282459108 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f4009b30 con 0x7fa404072850 2024-02-19T15:22:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa402a00640 1 --2- 172.21.15.47:0/3282459108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa404072850 0x7fa4041509b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa3f401a540 tx=0x7fa3f40196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:41.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa3fbe00640 1 -- 172.21.15.47:0/3282459108 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f40b8680 con 0x7fa404072850 2024-02-19T15:22:41.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa3fbe00640 1 -- 172.21.15.47:0/3282459108 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f40b87e0 con 0x7fa404072850 2024-02-19T15:22:41.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa3fbe00640 1 -- 172.21.15.47:0/3282459108 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f40b8b70 con 0x7fa404072850 2024-02-19T15:22:41.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa409e00640 1 -- 172.21.15.47:0/3282459108 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4040c9290 con 0x7fa404072850 2024-02-19T15:22:41.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa409e00640 1 -- 172.21.15.47:0/3282459108 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4040c9850 con 0x7fa404072850 2024-02-19T15:22:41.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa409e00640 1 -- 172.21.15.47:0/3282459108 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa404166610 con 0x7fa404072850 2024-02-19T15:22:41.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.661+0000 7fa3fbe00640 1 -- 172.21.15.47:0/3282459108 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa3f40b8cd0 con 0x7fa404072850 2024-02-19T15:22:41.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.665+0000 7fa3fbe00640 1 --2- 172.21.15.47:0/3282459108 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa3e0066e40 0x7fa3e0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:41.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.665+0000 7fa3fbe00640 1 -- 172.21.15.47:0/3282459108 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3f411fc90 con 0x7fa404072850 2024-02-19T15:22:41.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.665+0000 7fa3fa000640 1 --2- 172.21.15.47:0/3282459108 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa3e0066e40 0x7fa3e0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:41.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.665+0000 7fa3fa000640 1 --2- 172.21.15.47:0/3282459108 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa3e0066e40 0x7fa3e0069300 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa404151830 tx=0x7fa3ec007e50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:41.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.669+0000 7fa3fbe00640 1 -- 172.21.15.47:0/3282459108 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa3f40ecb20 con 0x7fa404072850 2024-02-19T15:22:41.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.829+0000 7fa409e00640 1 -- 172.21.15.47:0/3282459108 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa40406e800 con 0x7fa404072850 2024-02-19T15:22:41.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.829+0000 7fa3fbe00640 1 -- 172.21.15.47:0/3282459108 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa3f40f01e0 con 0x7fa404072850 2024-02-19T15:22:41.833 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:22:41.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.833+0000 7fa409e00640 1 -- 172.21.15.47:0/3282459108 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa3e0066e40 msgr2=0x7fa3e0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:41.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.833+0000 7fa409e00640 1 --2- 172.21.15.47:0/3282459108 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa3e0066e40 0x7fa3e0069300 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa404151830 tx=0x7fa3ec007e50 comp rx=0 tx=0).stop 2024-02-19T15:22:41.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.833+0000 7fa409e00640 1 -- 172.21.15.47:0/3282459108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa404072850 msgr2=0x7fa4041509b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:41.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.833+0000 7fa409e00640 1 --2- 172.21.15.47:0/3282459108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa404072850 0x7fa4041509b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa3f401a540 tx=0x7fa3f40196b0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.833+0000 7fa409e00640 1 -- 172.21.15.47:0/3282459108 shutdown_connections 2024-02-19T15:22:41.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.833+0000 7fa409e00640 1 --2- 172.21.15.47:0/3282459108 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa3e0066e40 0x7fa3e0069300 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.833+0000 7fa409e00640 1 --2- 172.21.15.47:0/3282459108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa40415bd90 0x7fa40415e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.833+0000 7fa409e00640 1 --2- 172.21.15.47:0/3282459108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4040cb490 0x7fa4040b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.833+0000 7fa409e00640 1 --2- 172.21.15.47:0/3282459108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa404072850 0x7fa4041509b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.833+0000 7fa409e00640 1 -- 172.21.15.47:0/3282459108 >> 172.21.15.47:0/3282459108 conn(0x7fa4040cfc20 msgr2=0x7fa4040caae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:41.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.833+0000 7fa409e00640 1 -- 172.21.15.47:0/3282459108 shutdown_connections 2024-02-19T15:22:41.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.833+0000 7fa409e00640 1 -- 172.21.15.47:0/3282459108 wait complete. 2024-02-19T15:22:41.853 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-19T15:22:41.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.989+0000 7f6243a00640 1 Processor -- start 2024-02-19T15:22:41.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.993+0000 7f6243a00640 1 -- start start 2024-02-19T15:22:41.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.993+0000 7f6243a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f623c15b1a0 0x7f623c15b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:41.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.993+0000 7f6243a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f623c071350 0x7f623c15bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:41.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.993+0000 7f6243a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f623c15c000 con 0x7f623c06c9a0 2024-02-19T15:22:41.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.993+0000 7f6243a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f623c0718c0 con 0x7f623c15b1a0 2024-02-19T15:22:41.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.993+0000 7f6243a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f623c071ba0 con 0x7f623c071350 2024-02-19T15:22:41.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.993+0000 7f6241000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f623c15b1a0 0x7f623c15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:41.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.993+0000 7f6241a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f623c06c9a0 0x7f623c15c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38190/0 (socket says 172.21.15.47:38190) 2024-02-19T15:22:41.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.993+0000 7f6241a00640 1 -- 172.21.15.47:0/3770217501 learned_addr learned my addr 172.21.15.47:0/3770217501 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:41.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.993+0000 7f623be00640 1 --2- 172.21.15.47:0/3770217501 >> v2:172.21.15.116:3300/0 conn(0x7f623c071350 0x7f623c15bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:41.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.993+0000 7f6241000640 1 -- 172.21.15.47:0/3770217501 >> v2:172.21.15.116:3300/0 conn(0x7f623c071350 msgr2=0x7f623c15bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:41.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.993+0000 7f6241000640 1 --2- 172.21.15.47:0/3770217501 >> v2:172.21.15.116:3300/0 conn(0x7f623c071350 0x7f623c15bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:41.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.993+0000 7f6241000640 1 -- 172.21.15.47:0/3770217501 >> v1:172.21.15.47:6789/0 conn(0x7f623c06c9a0 legacy=0x7f623c15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:41.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.993+0000 7f6241000640 1 -- 172.21.15.47:0/3770217501 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623c071e80 con 0x7f623c15b1a0 2024-02-19T15:22:42.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.997+0000 7f6241000640 1 --2- 172.21.15.47:0/3770217501 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f623c15b1a0 0x7f623c15b580 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f62300097c0 tx=0x7f623001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7499eb6c77f852b6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:42.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.997+0000 7f623b400640 1 -- 172.21.15.47:0/3770217501 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62300ab650 con 0x7f623c15b1a0 2024-02-19T15:22:42.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.997+0000 7f623b400640 1 -- 172.21.15.47:0/3770217501 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62300ab7b0 con 0x7f623c15b1a0 2024-02-19T15:22:42.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.997+0000 7f623b400640 1 -- 172.21.15.47:0/3770217501 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62300abb40 con 0x7f623c15b1a0 2024-02-19T15:22:42.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.997+0000 7f6243a00640 1 -- 172.21.15.47:0/3770217501 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f623c15b1a0 msgr2=0x7f623c15b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:42.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.997+0000 7f6243a00640 1 --2- 172.21.15.47:0/3770217501 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f623c15b1a0 0x7f623c15b580 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f62300097c0 tx=0x7f623001c430 comp rx=0 tx=0).stop 2024-02-19T15:22:42.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.997+0000 7f6243a00640 1 -- 172.21.15.47:0/3770217501 shutdown_connections 2024-02-19T15:22:42.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.997+0000 7f6243a00640 1 --2- 172.21.15.47:0/3770217501 >> v2:172.21.15.116:3300/0 conn(0x7f623c071350 0x7f623c15bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.997+0000 7f6243a00640 1 --2- 172.21.15.47:0/3770217501 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f623c15b1a0 0x7f623c15b580 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.997+0000 7f6243a00640 1 -- 172.21.15.47:0/3770217501 >> 172.21.15.47:0/3770217501 conn(0x7f623c0b16a0 msgr2=0x7f623c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:42.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.997+0000 7f6243a00640 1 -- 172.21.15.47:0/3770217501 shutdown_connections 2024-02-19T15:22:42.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.997+0000 7f6243a00640 1 -- 172.21.15.47:0/3770217501 wait complete. 2024-02-19T15:22:42.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.997+0000 7f6243a00640 1 Processor -- start 2024-02-19T15:22:42.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:41.997+0000 7f6243a00640 1 -- start start 2024-02-19T15:22:42.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6243a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c06c9a0 0x7f623c1493d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:42.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6243a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f623c071350 0x7f623c14d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:42.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6243a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f623c15b1a0 0x7f623c14d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:42.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6243a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623c1510d0 con 0x7f623c15b1a0 2024-02-19T15:22:42.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6243a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623c1507a0 con 0x7f623c071350 2024-02-19T15:22:42.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6243a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f623c150a80 con 0x7f623c06c9a0 2024-02-19T15:22:42.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6241a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f623c15b1a0 0x7f623c14d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:42.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6241a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f623c15b1a0 0x7f623c14d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44946/0 (socket says 172.21.15.47:44946) 2024-02-19T15:22:42.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f623be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f623c071350 0x7f623c14d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:42.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6241a00640 1 -- 172.21.15.47:0/3999029417 learned_addr learned my addr 172.21.15.47:0/3999029417 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:42.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6241000640 1 --2- 172.21.15.47:0/3999029417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c06c9a0 0x7f623c1493d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:42.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6241a00640 1 -- 172.21.15.47:0/3999029417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c06c9a0 msgr2=0x7f623c1493d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:42.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6241a00640 1 --2- 172.21.15.47:0/3999029417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c06c9a0 0x7f623c1493d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6241a00640 1 -- 172.21.15.47:0/3999029417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f623c071350 msgr2=0x7f623c14d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:42.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6241a00640 1 --2- 172.21.15.47:0/3999029417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f623c071350 0x7f623c14d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6241a00640 1 -- 172.21.15.47:0/3999029417 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6230009420 con 0x7f623c15b1a0 2024-02-19T15:22:42.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f623be00640 1 --2- 172.21.15.47:0/3999029417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f623c071350 0x7f623c14d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:42.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6241a00640 1 --2- 172.21.15.47:0/3999029417 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f623c15b1a0 0x7f623c14d740 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f623400ca80 tx=0x7f623400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:42.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6238c00640 1 -- 172.21.15.47:0/3999029417 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623400ac30 con 0x7f623c15b1a0 2024-02-19T15:22:42.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6238c00640 1 -- 172.21.15.47:0/3999029417 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f623400ad90 con 0x7f623c15b1a0 2024-02-19T15:22:42.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6238c00640 1 -- 172.21.15.47:0/3999029417 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6234012600 con 0x7f623c15b1a0 2024-02-19T15:22:42.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6243a00640 1 -- 172.21.15.47:0/3999029417 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f623c150d60 con 0x7f623c15b1a0 2024-02-19T15:22:42.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6243a00640 1 -- 172.21.15.47:0/3999029417 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f623c150450 con 0x7f623c15b1a0 2024-02-19T15:22:42.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.001+0000 7f6243a00640 1 -- 172.21.15.47:0/3999029417 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f623c151400 con 0x7f623c15b1a0 2024-02-19T15:22:42.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.005+0000 7f6238c00640 1 -- 172.21.15.47:0/3999029417 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6234007070 con 0x7f623c15b1a0 2024-02-19T15:22:42.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.005+0000 7f6238c00640 1 --2- 172.21.15.47:0/3999029417 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f621c066e70 0x7f621c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:42.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.005+0000 7f6238c00640 1 -- 172.21.15.47:0/3999029417 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6234084880 con 0x7f623c15b1a0 2024-02-19T15:22:42.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.005+0000 7f6241000640 1 --2- 172.21.15.47:0/3999029417 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f621c066e70 0x7f621c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:42.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.005+0000 7f6241000640 1 --2- 172.21.15.47:0/3999029417 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f621c066e70 0x7f621c069330 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f62300af000 tx=0x7f62300a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:42.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.009+0000 7f6238c00640 1 -- 172.21.15.47:0/3999029417 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6234051680 con 0x7f623c15b1a0 2024-02-19T15:22:42.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.181+0000 7f6243a00640 1 -- 172.21.15.47:0/3999029417 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f623c075360 con 0x7f623c15b1a0 2024-02-19T15:22:42.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.181+0000 7f6238c00640 1 -- 172.21.15.47:0/3999029417 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6234017080 con 0x7f623c15b1a0 2024-02-19T15:22:42.185 INFO:teuthology.orchestra.run.smithi047.stderr:listed 0 entries 2024-02-19T15:22:42.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.185+0000 7f6243a00640 1 -- 172.21.15.47:0/3999029417 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f621c066e70 msgr2=0x7f621c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:42.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.185+0000 7f6243a00640 1 --2- 172.21.15.47:0/3999029417 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f621c066e70 0x7f621c069330 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f62300af000 tx=0x7f62300a7000 comp rx=0 tx=0).stop 2024-02-19T15:22:42.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.185+0000 7f6243a00640 1 -- 172.21.15.47:0/3999029417 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f623c15b1a0 msgr2=0x7f623c14d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:42.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.185+0000 7f6243a00640 1 --2- 172.21.15.47:0/3999029417 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f623c15b1a0 0x7f623c14d740 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f623400ca80 tx=0x7f623400cf40 comp rx=0 tx=0).stop 2024-02-19T15:22:42.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.185+0000 7f6243a00640 1 -- 172.21.15.47:0/3999029417 shutdown_connections 2024-02-19T15:22:42.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.185+0000 7f6243a00640 1 --2- 172.21.15.47:0/3999029417 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f621c066e70 0x7f621c069330 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.185+0000 7f6243a00640 1 --2- 172.21.15.47:0/3999029417 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f623c15b1a0 0x7f623c14d740 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.185+0000 7f6243a00640 1 --2- 172.21.15.47:0/3999029417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f623c071350 0x7f623c14d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.185+0000 7f6243a00640 1 --2- 172.21.15.47:0/3999029417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c06c9a0 0x7f623c1493d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.185+0000 7f6243a00640 1 -- 172.21.15.47:0/3999029417 >> 172.21.15.47:0/3999029417 conn(0x7f623c0b16a0 msgr2=0x7f623c0b0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:42.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.185+0000 7f6243a00640 1 -- 172.21.15.47:0/3999029417 shutdown_connections 2024-02-19T15:22:42.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.185+0000 7f6243a00640 1 -- 172.21.15.47:0/3999029417 wait complete. 2024-02-19T15:22:42.207 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-02-19T15:22:42.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:42.207 DEBUG:teuthology.orchestra.run.smithi047:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-02-19T15:22:42.230 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi047.front.sepia.ceph.com 2024-02-19T15:22:42.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:42.231 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-02-19T15:22:42.283 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:22:42.283 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:22:42.292 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:22:42.292 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:22:42.341 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:22:42.341 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:22:42.389 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:42.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.525+0000 7fb46a400640 1 Processor -- start 2024-02-19T15:22:42.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.525+0000 7fb46a400640 1 -- start start 2024-02-19T15:22:42.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.525+0000 7fb46a400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb46415b1a0 0x7fb46415b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:42.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.525+0000 7fb46a400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb46406c9a0 0x7fb46415bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:42.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.525+0000 7fb46a400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb46415c000 con 0x7fb464071350 2024-02-19T15:22:42.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.525+0000 7fb46a400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4640718c0 con 0x7fb46415b1a0 2024-02-19T15:22:42.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.525+0000 7fb46a400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb464071ba0 con 0x7fb46406c9a0 2024-02-19T15:22:42.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.529+0000 7fb463e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb46406c9a0 0x7fb46415bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:42.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.529+0000 7fb463400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb46415b1a0 0x7fb46415b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:42.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.529+0000 7fb463e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb46406c9a0 0x7fb46415bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:51112/0 (socket says 172.21.15.47:51112) 2024-02-19T15:22:42.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.529+0000 7fb463e00640 1 -- 172.21.15.47:0/1908358041 learned_addr learned my addr 172.21.15.47:0/1908358041 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:42.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.529+0000 7fb463400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb46415b1a0 0x7fb46415b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:47514/0 (socket says 172.21.15.47:47514) 2024-02-19T15:22:42.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.529+0000 7fb463e00640 1 -- 172.21.15.47:0/1908358041 >> v2:172.21.15.116:3300/0 conn(0x7fb46415b1a0 msgr2=0x7fb46415b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:42.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.529+0000 7fb463e00640 1 --2- 172.21.15.47:0/1908358041 >> v2:172.21.15.116:3300/0 conn(0x7fb46415b1a0 0x7fb46415b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.529+0000 7fb463e00640 1 -- 172.21.15.47:0/1908358041 >> v1:172.21.15.116:6789/0 conn(0x7fb464071350 legacy=0x7fb46415c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:42.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.529+0000 7fb463e00640 1 -- 172.21.15.47:0/1908358041 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb464071e80 con 0x7fb46406c9a0 2024-02-19T15:22:42.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.529+0000 7fb462a00640 1 -- 172.21.15.47:0/1908358041 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561671406 0 0) 0x7fb46415c000 con 0x7fb464071350 2024-02-19T15:22:42.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.529+0000 7fb463e00640 1 --2- 172.21.15.47:0/1908358041 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb46406c9a0 0x7fb46415bac0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb45c009e80 tx=0x7fb45c01c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c419cacae23d97ac server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:42.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.529+0000 7fb462a00640 1 -- 172.21.15.47:0/1908358041 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c0a58d0 con 0x7fb46406c9a0 2024-02-19T15:22:42.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.529+0000 7fb462a00640 1 -- 172.21.15.47:0/1908358041 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45c0a5a30 con 0x7fb46406c9a0 2024-02-19T15:22:42.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.529+0000 7fb462a00640 1 -- 172.21.15.47:0/1908358041 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c0a5dc0 con 0x7fb46406c9a0 2024-02-19T15:22:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb46a400640 1 -- 172.21.15.47:0/1908358041 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb46406c9a0 msgr2=0x7fb46415bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb46a400640 1 --2- 172.21.15.47:0/1908358041 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb46406c9a0 0x7fb46415bac0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb45c009e80 tx=0x7fb45c01c9e0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb46a400640 1 -- 172.21.15.47:0/1908358041 shutdown_connections 2024-02-19T15:22:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb46a400640 1 --2- 172.21.15.47:0/1908358041 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb46406c9a0 0x7fb46415bac0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb46a400640 1 --2- 172.21.15.47:0/1908358041 >> v2:172.21.15.116:3300/0 conn(0x7fb46415b1a0 0x7fb46415b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb46a400640 1 -- 172.21.15.47:0/1908358041 >> 172.21.15.47:0/1908358041 conn(0x7fb4640b16a0 msgr2=0x7fb4640b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb46a400640 1 -- 172.21.15.47:0/1908358041 shutdown_connections 2024-02-19T15:22:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb46a400640 1 -- 172.21.15.47:0/1908358041 wait complete. 2024-02-19T15:22:42.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb46a400640 1 Processor -- start 2024-02-19T15:22:42.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb46a400640 1 -- start start 2024-02-19T15:22:42.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb46a400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb46406c9a0 0x7fb46413d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:42.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb46a400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb464071350 0x7fb46406d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:42.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb46a400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb46415b1a0 0x7fb46406d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:42.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb46a400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46413b770 con 0x7fb46406c9a0 2024-02-19T15:22:42.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb46a400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4640ce950 con 0x7fb46415b1a0 2024-02-19T15:22:42.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb46a400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb46413add0 con 0x7fb464071350 2024-02-19T15:22:42.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb463400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb46406c9a0 0x7fb46413d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:42.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb463400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb46406c9a0 0x7fb46413d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44954/0 (socket says 172.21.15.47:44954) 2024-02-19T15:22:42.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb463400640 1 -- 172.21.15.47:0/1931995468 learned_addr learned my addr 172.21.15.47:0/1931995468 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:42.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb463e00640 1 --2- 172.21.15.47:0/1931995468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb46415b1a0 0x7fb46406d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:42.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb45be00640 1 --2- 172.21.15.47:0/1931995468 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb464071350 0x7fb46406d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:42.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb463400640 1 -- 172.21.15.47:0/1931995468 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb464071350 msgr2=0x7fb46406d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:42.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb463400640 1 --2- 172.21.15.47:0/1931995468 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb464071350 0x7fb46406d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb463400640 1 -- 172.21.15.47:0/1931995468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb46415b1a0 msgr2=0x7fb46406d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:42.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb463400640 1 --2- 172.21.15.47:0/1931995468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb46415b1a0 0x7fb46406d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.533+0000 7fb463400640 1 -- 172.21.15.47:0/1931995468 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c009b30 con 0x7fb46406c9a0 2024-02-19T15:22:42.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.537+0000 7fb463400640 1 --2- 172.21.15.47:0/1931995468 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb46406c9a0 0x7fb46413d810 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb45400e930 tx=0x7fb45400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:42.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.537+0000 7fb45b400640 1 -- 172.21.15.47:0/1931995468 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb454009690 con 0x7fb46406c9a0 2024-02-19T15:22:42.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.537+0000 7fb45b400640 1 -- 172.21.15.47:0/1931995468 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45400ca00 con 0x7fb46406c9a0 2024-02-19T15:22:42.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.537+0000 7fb45b400640 1 -- 172.21.15.47:0/1931995468 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45400cd70 con 0x7fb46406c9a0 2024-02-19T15:22:42.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.537+0000 7fb46a400640 1 -- 172.21.15.47:0/1931995468 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46413b0b0 con 0x7fb46406c9a0 2024-02-19T15:22:42.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.537+0000 7fb46a400640 1 -- 172.21.15.47:0/1931995468 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46413dd50 con 0x7fb46406c9a0 2024-02-19T15:22:42.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.537+0000 7fb46a400640 1 -- 172.21.15.47:0/1931995468 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb46413b9e0 con 0x7fb46406c9a0 2024-02-19T15:22:42.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.537+0000 7fb45b400640 1 -- 172.21.15.47:0/1931995468 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb454013470 con 0x7fb46406c9a0 2024-02-19T15:22:42.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.537+0000 7fb45b400640 1 --2- 172.21.15.47:0/1931995468 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb440066e70 0x7fb440069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:42.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.537+0000 7fb45b400640 1 -- 172.21.15.47:0/1931995468 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb454084ce0 con 0x7fb46406c9a0 2024-02-19T15:22:42.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.541+0000 7fb45be00640 1 --2- 172.21.15.47:0/1931995468 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb440066e70 0x7fb440069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:42.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.541+0000 7fb45be00640 1 --2- 172.21.15.47:0/1931995468 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb440066e70 0x7fb440069330 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb464152080 tx=0x7fb44c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:42.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.541+0000 7fb45b400640 1 -- 172.21.15.47:0/1931995468 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb454051ba0 con 0x7fb46406c9a0 2024-02-19T15:22:42.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.709+0000 7fb46a400640 1 -- 172.21.15.47:0/1931995468 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb464075360 con 0x7fb46406c9a0 2024-02-19T15:22:42.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.709+0000 7fb45b400640 1 -- 172.21.15.47:0/1931995468 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7fb454017090 con 0x7fb46406c9a0 2024-02-19T15:22:42.712 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:42.712 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":21,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:06.843155+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:03.091360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:04.222965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:42.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.709+0000 7fb46a400640 1 -- 172.21.15.47:0/1931995468 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb440066e70 msgr2=0x7fb440069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:42.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.709+0000 7fb46a400640 1 --2- 172.21.15.47:0/1931995468 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb440066e70 0x7fb440069330 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb464152080 tx=0x7fb44c009040 comp rx=0 tx=0).stop 2024-02-19T15:22:42.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.709+0000 7fb46a400640 1 -- 172.21.15.47:0/1931995468 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb46406c9a0 msgr2=0x7fb46413d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:42.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.709+0000 7fb46a400640 1 --2- 172.21.15.47:0/1931995468 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb46406c9a0 0x7fb46413d810 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb45400e930 tx=0x7fb45400edf0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.709+0000 7fb46a400640 1 -- 172.21.15.47:0/1931995468 shutdown_connections 2024-02-19T15:22:42.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.709+0000 7fb46a400640 1 --2- 172.21.15.47:0/1931995468 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb440066e70 0x7fb440069330 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.709+0000 7fb46a400640 1 --2- 172.21.15.47:0/1931995468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb46415b1a0 0x7fb46406d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.709+0000 7fb46a400640 1 --2- 172.21.15.47:0/1931995468 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb464071350 0x7fb46406d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.709+0000 7fb46a400640 1 --2- 172.21.15.47:0/1931995468 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb46406c9a0 0x7fb46413d810 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.709+0000 7fb46a400640 1 -- 172.21.15.47:0/1931995468 >> 172.21.15.47:0/1931995468 conn(0x7fb4640b16a0 msgr2=0x7fb464157f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:42.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.709+0000 7fb46a400640 1 -- 172.21.15.47:0/1931995468 shutdown_connections 2024-02-19T15:22:42.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.709+0000 7fb46a400640 1 -- 172.21.15.47:0/1931995468 wait complete. 2024-02-19T15:22:42.733 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:22:03.091360+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:22:04.222965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:22:42.734 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:22:42.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.873+0000 7f4cd6800640 1 Processor -- start 2024-02-19T15:22:42.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.873+0000 7f4cd6800640 1 -- start start 2024-02-19T15:22:42.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.873+0000 7f4cd6800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4cd00c7e80 0x7f4cd00d0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:42.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.873+0000 7f4cd6800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4cd00c8260 0x7f4cd00d0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:42.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.873+0000 7f4cd6800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd00cd750 con 0x7f4cd00cfbf0 2024-02-19T15:22:42.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.873+0000 7f4cd6800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd00ceee0 con 0x7f4cd00c8260 2024-02-19T15:22:42.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.873+0000 7f4cd6800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd00cdc60 con 0x7f4cd00c7e80 2024-02-19T15:22:42.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.873+0000 7f4cc6a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4cd00c8260 0x7f4cd00d0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:42.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.873+0000 7f4ccf400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4cd00c7e80 0x7f4cd00d0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:42.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.873+0000 7f4cc6a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4cd00c8260 0x7f4cd00d0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:47522/0 (socket says 172.21.15.47:47522) 2024-02-19T15:22:42.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.873+0000 7f4cc6a00640 1 -- 172.21.15.47:0/543595427 learned_addr learned my addr 172.21.15.47:0/543595427 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:42.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.873+0000 7f4ccfe00640 1 --1- 172.21.15.47:0/543595427 >> v1:172.21.15.116:6789/0 conn(0x7f4cd00cfbf0 0x7f4cd00c5680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:42416/0 (socket says 172.21.15.47:42416) 2024-02-19T15:22:42.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.873+0000 7f4cc6a00640 1 -- 172.21.15.47:0/543595427 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4cd00c7e80 msgr2=0x7f4cd00d0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:42.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.873+0000 7f4cc6a00640 1 --2- 172.21.15.47:0/543595427 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4cd00c7e80 0x7f4cd00d0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.873+0000 7f4cc6a00640 1 -- 172.21.15.47:0/543595427 >> v1:172.21.15.116:6789/0 conn(0x7f4cd00cfbf0 legacy=0x7f4cd00c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:42.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.873+0000 7f4cc6a00640 1 -- 172.21.15.47:0/543595427 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd00d2a40 con 0x7f4cd00c8260 2024-02-19T15:22:42.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.877+0000 7f4cc6a00640 1 --2- 172.21.15.47:0/543595427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cd00c8260 0x7f4cd00d0650 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8009870 tx=0x7f4cb801c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77528c6c3bb9df81 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:42.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.877+0000 7f4ccea00640 1 -- 172.21.15.47:0/543595427 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8004230 con 0x7f4cd00c8260 2024-02-19T15:22:42.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.877+0000 7f4ccea00640 1 -- 172.21.15.47:0/543595427 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb80a7050 con 0x7f4cd00c8260 2024-02-19T15:22:42.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.877+0000 7f4ccea00640 1 -- 172.21.15.47:0/543595427 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8003620 con 0x7f4cd00c8260 2024-02-19T15:22:42.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.877+0000 7f4cd6800640 1 -- 172.21.15.47:0/543595427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cd00c8260 msgr2=0x7f4cd00d0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.877+0000 7f4cd6800640 1 --2- 172.21.15.47:0/543595427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cd00c8260 0x7f4cd00d0650 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8009870 tx=0x7f4cb801c3f0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.877+0000 7f4cd6800640 1 -- 172.21.15.47:0/543595427 shutdown_connections 2024-02-19T15:22:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.877+0000 7f4cd6800640 1 --2- 172.21.15.47:0/543595427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cd00c8260 0x7f4cd00d0650 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.877+0000 7f4cd6800640 1 --2- 172.21.15.47:0/543595427 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4cd00c7e80 0x7f4cd00d0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.877+0000 7f4cd6800640 1 -- 172.21.15.47:0/543595427 >> 172.21.15.47:0/543595427 conn(0x7f4cd00d1b30 msgr2=0x7f4cd00d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.877+0000 7f4cd6800640 1 -- 172.21.15.47:0/543595427 shutdown_connections 2024-02-19T15:22:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.877+0000 7f4cd6800640 1 -- 172.21.15.47:0/543595427 wait complete. 2024-02-19T15:22:42.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cd6800640 1 Processor -- start 2024-02-19T15:22:42.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cd6800640 1 -- start start 2024-02-19T15:22:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cd6800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4cd00c7e80 0x7f4cd015db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cd6800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4cd00c8260 0x7f4cd015a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cd6800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cd00cfbf0 0x7f4cd01579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cd6800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd015c820 con 0x7f4cd00c8260 2024-02-19T15:22:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cd6800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd015ca70 con 0x7f4cd00cfbf0 2024-02-19T15:22:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cd6800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd015cd50 con 0x7f4cd00c7e80 2024-02-19T15:22:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cc6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4cd00c8260 0x7f4cd015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4ccf400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4cd00c7e80 0x7f4cd015db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cc6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4cd00c8260 0x7f4cd015a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44962/0 (socket says 172.21.15.47:44962) 2024-02-19T15:22:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cc6a00640 1 -- 172.21.15.47:0/394293367 learned_addr learned my addr 172.21.15.47:0/394293367 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4ccfe00640 1 --2- 172.21.15.47:0/394293367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cd00cfbf0 0x7f4cd01579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cc6a00640 1 -- 172.21.15.47:0/394293367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4cd00c7e80 msgr2=0x7f4cd015db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cc6a00640 1 --2- 172.21.15.47:0/394293367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4cd00c7e80 0x7f4cd015db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cc6a00640 1 -- 172.21.15.47:0/394293367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cd00cfbf0 msgr2=0x7f4cd01579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:42.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cc6a00640 1 --2- 172.21.15.47:0/394293367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cd00cfbf0 0x7f4cd01579f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:42.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cc6a00640 1 -- 172.21.15.47:0/394293367 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb80094d0 con 0x7f4cd00c8260 2024-02-19T15:22:42.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cc6a00640 1 --2- 172.21.15.47:0/394293367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4cd00c8260 0x7f4cd015a580 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4cb801aaa0 tx=0x7f4cb8019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:42.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cc7e00640 1 -- 172.21.15.47:0/394293367 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8004530 con 0x7f4cd00c8260 2024-02-19T15:22:42.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cc7e00640 1 -- 172.21.15.47:0/394293367 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb80b6640 con 0x7f4cd00c8260 2024-02-19T15:22:42.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cc7e00640 1 -- 172.21.15.47:0/394293367 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb80b69d0 con 0x7f4cd00c8260 2024-02-19T15:22:42.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cd6800640 1 -- 172.21.15.47:0/394293367 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd015d030 con 0x7f4cd00c8260 2024-02-19T15:22:42.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.881+0000 7f4cd6800640 1 -- 172.21.15.47:0/394293367 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd015d5f0 con 0x7f4cd00c8260 2024-02-19T15:22:42.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.885+0000 7f4cd6800640 1 -- 172.21.15.47:0/394293367 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd0158760 con 0x7f4cd00c8260 2024-02-19T15:22:42.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.885+0000 7f4cc7e00640 1 -- 172.21.15.47:0/394293367 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4cb80b2040 con 0x7f4cd00c8260 2024-02-19T15:22:42.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.885+0000 7f4cc7e00640 1 --2- 172.21.15.47:0/394293367 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4cac066e40 0x7f4cac069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:42.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.885+0000 7f4cc7e00640 1 -- 172.21.15.47:0/394293367 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4cb811f0e0 con 0x7f4cd00c8260 2024-02-19T15:22:42.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.885+0000 7f4ccf400640 1 --2- 172.21.15.47:0/394293367 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4cac066e40 0x7f4cac069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:42.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.885+0000 7f4ccf400640 1 --2- 172.21.15.47:0/394293367 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4cac066e40 0x7f4cac069300 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0009b60 tx=0x7f4cc0009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:42.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:42.889+0000 7f4cc7e00640 1 -- 172.21.15.47:0/394293367 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f4cb80ebef0 con 0x7f4cd00c8260 2024-02-19T15:22:43.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.053+0000 7f4cd6800640 1 -- 172.21.15.47:0/394293367 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4cd00d1180 con 0x7f4cd00c8260 2024-02-19T15:22:43.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.053+0000 7f4cc7e00640 1 -- 172.21.15.47:0/394293367 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4cb80ef5b0 con 0x7f4cd00c8260 2024-02-19T15:22:43.057 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:22:43.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.057+0000 7f4cd6800640 1 -- 172.21.15.47:0/394293367 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4cac066e40 msgr2=0x7f4cac069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:43.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.057+0000 7f4cd6800640 1 --2- 172.21.15.47:0/394293367 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4cac066e40 0x7f4cac069300 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0009b60 tx=0x7f4cc0009190 comp rx=0 tx=0).stop 2024-02-19T15:22:43.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.057+0000 7f4cd6800640 1 -- 172.21.15.47:0/394293367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4cd00c8260 msgr2=0x7f4cd015a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:43.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.057+0000 7f4cd6800640 1 --2- 172.21.15.47:0/394293367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4cd00c8260 0x7f4cd015a580 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4cb801aaa0 tx=0x7f4cb8019190 comp rx=0 tx=0).stop 2024-02-19T15:22:43.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.057+0000 7f4cd6800640 1 -- 172.21.15.47:0/394293367 shutdown_connections 2024-02-19T15:22:43.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.057+0000 7f4cd6800640 1 --2- 172.21.15.47:0/394293367 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4cac066e40 0x7f4cac069300 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.057+0000 7f4cd6800640 1 --2- 172.21.15.47:0/394293367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cd00cfbf0 0x7f4cd01579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.057+0000 7f4cd6800640 1 --2- 172.21.15.47:0/394293367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4cd00c8260 0x7f4cd015a580 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.057+0000 7f4cd6800640 1 --2- 172.21.15.47:0/394293367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4cd00c7e80 0x7f4cd015db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.057+0000 7f4cd6800640 1 -- 172.21.15.47:0/394293367 >> 172.21.15.47:0/394293367 conn(0x7f4cd00d1b30 msgr2=0x7f4cd0070270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:43.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.057+0000 7f4cd6800640 1 -- 172.21.15.47:0/394293367 shutdown_connections 2024-02-19T15:22:43.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.057+0000 7f4cd6800640 1 -- 172.21.15.47:0/394293367 wait complete. 2024-02-19T15:22:43.078 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:22:43.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.217+0000 7f88a4800640 1 Processor -- start 2024-02-19T15:22:43.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.217+0000 7f88a4800640 1 -- start start 2024-02-19T15:22:43.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.217+0000 7f88a4800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f889c15bd90 0x7f889c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:43.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.221+0000 7f88a4800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f889c15fc90 con 0x7f889c0cb490 2024-02-19T15:22:43.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.221+0000 7f88a4800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f889c15ff70 con 0x7f889c072850 2024-02-19T15:22:43.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.221+0000 7f88a4800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f889c160250 con 0x7f889c15bd90 2024-02-19T15:22:43.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.221+0000 7f88a1e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f889c15bd90 0x7f889c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:43.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.221+0000 7f88a1400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f889c0cb490 0x7f889c072140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38206/0 (socket says 172.21.15.47:38206) 2024-02-19T15:22:43.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.221+0000 7f88a1400640 1 -- 172.21.15.47:0/2369317320 learned_addr learned my addr 172.21.15.47:0/2369317320 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:43.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.221+0000 7f88a1e00640 1 -- 172.21.15.47:0/2369317320 >> v1:172.21.15.116:6789/0 conn(0x7f889c072850 legacy=0x7f889c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:43.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.221+0000 7f88a1e00640 1 -- 172.21.15.47:0/2369317320 >> v1:172.21.15.47:6789/0 conn(0x7f889c0cb490 legacy=0x7f889c072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:43.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.221+0000 7f88a1e00640 1 -- 172.21.15.47:0/2369317320 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c160530 con 0x7f889c15bd90 2024-02-19T15:22:43.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.221+0000 7f88a0a00640 1 -- 172.21.15.47:0/2369317320 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564066452 0 0) 0x7f889c15ff70 con 0x7f889c072850 2024-02-19T15:22:43.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.221+0000 7f88a1e00640 1 --2- 172.21.15.47:0/2369317320 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f889c15bd90 0x7f889c070960 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f88900097c0 tx=0x7f889001c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57f62714e3286be5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:43.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.221+0000 7f88a0a00640 1 -- 172.21.15.47:0/2369317320 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88900ab600 con 0x7f889c15bd90 2024-02-19T15:22:43.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.221+0000 7f88a0a00640 1 -- 172.21.15.47:0/2369317320 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88900a7050 con 0x7f889c15bd90 2024-02-19T15:22:43.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.221+0000 7f88a0a00640 1 -- 172.21.15.47:0/2369317320 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88900ab980 con 0x7f889c15bd90 2024-02-19T15:22:43.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a4800640 1 -- 172.21.15.47:0/2369317320 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f889c15bd90 msgr2=0x7f889c070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:43.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a4800640 1 --2- 172.21.15.47:0/2369317320 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f889c15bd90 0x7f889c070960 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f88900097c0 tx=0x7f889001c380 comp rx=0 tx=0).stop 2024-02-19T15:22:43.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a4800640 1 -- 172.21.15.47:0/2369317320 shutdown_connections 2024-02-19T15:22:43.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a4800640 1 --2- 172.21.15.47:0/2369317320 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f889c15bd90 0x7f889c070960 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a4800640 1 -- 172.21.15.47:0/2369317320 >> 172.21.15.47:0/2369317320 conn(0x7f889c0cfc20 msgr2=0x7f889c0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:43.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a4800640 1 -- 172.21.15.47:0/2369317320 shutdown_connections 2024-02-19T15:22:43.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a4800640 1 -- 172.21.15.47:0/2369317320 wait complete. 2024-02-19T15:22:43.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a4800640 1 Processor -- start 2024-02-19T15:22:43.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a4800640 1 -- start start 2024-02-19T15:22:43.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a4800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f889c072850 0x7f889c1542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:43.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a4800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f889c0cb490 0x7f889c0ca510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:43.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a4800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f889c15bd90 0x7f889c0c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:43.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a4800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c0c6cf0 con 0x7f889c072850 2024-02-19T15:22:43.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a4800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c0c6f00 con 0x7f889c15bd90 2024-02-19T15:22:43.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a4800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889c0c71e0 con 0x7f889c0cb490 2024-02-19T15:22:43.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a1e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f889c072850 0x7f889c1542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:43.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a1e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f889c072850 0x7f889c1542d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:44982/0 (socket says 172.21.15.47:44982) 2024-02-19T15:22:43.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a1e00640 1 -- 172.21.15.47:0/295699810 learned_addr learned my addr 172.21.15.47:0/295699810 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:43.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a1400640 1 --2- 172.21.15.47:0/295699810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f889c0cb490 0x7f889c0ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:43.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a2800640 1 --2- 172.21.15.47:0/295699810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f889c15bd90 0x7f889c0c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:43.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a1e00640 1 -- 172.21.15.47:0/295699810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f889c0cb490 msgr2=0x7f889c0ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:43.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a1e00640 1 --2- 172.21.15.47:0/295699810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f889c0cb490 0x7f889c0ca510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a1e00640 1 -- 172.21.15.47:0/295699810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f889c15bd90 msgr2=0x7f889c0c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:43.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a1e00640 1 --2- 172.21.15.47:0/295699810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f889c15bd90 0x7f889c0c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a1e00640 1 -- 172.21.15.47:0/295699810 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8890009420 con 0x7f889c072850 2024-02-19T15:22:43.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.225+0000 7f88a2800640 1 --2- 172.21.15.47:0/295699810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f889c15bd90 0x7f889c0c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:43.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.229+0000 7f88a1e00640 1 --2- 172.21.15.47:0/295699810 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f889c072850 0x7f889c1542d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f88900ae040 tx=0x7f889001abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:43.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.229+0000 7f8896000640 1 -- 172.21.15.47:0/295699810 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890004030 con 0x7f889c072850 2024-02-19T15:22:43.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.229+0000 7f8896000640 1 -- 172.21.15.47:0/295699810 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8890004190 con 0x7f889c072850 2024-02-19T15:22:43.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.229+0000 7f88a4800640 1 -- 172.21.15.47:0/295699810 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889c0c74c0 con 0x7f889c072850 2024-02-19T15:22:43.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.229+0000 7f8896000640 1 -- 172.21.15.47:0/295699810 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890003620 con 0x7f889c072850 2024-02-19T15:22:43.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.229+0000 7f88a4800640 1 -- 172.21.15.47:0/295699810 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889c0c7a80 con 0x7f889c072850 2024-02-19T15:22:43.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.229+0000 7f88a4800640 1 -- 172.21.15.47:0/295699810 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f889c075360 con 0x7f889c072850 2024-02-19T15:22:43.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.229+0000 7f8896000640 1 -- 172.21.15.47:0/295699810 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f88900b2050 con 0x7f889c072850 2024-02-19T15:22:43.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.229+0000 7f8896000640 1 --2- 172.21.15.47:0/295699810 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8880066e40 0x7f8880069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:43.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.229+0000 7f8896000640 1 -- 172.21.15.47:0/295699810 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f889011ebc0 con 0x7f889c072850 2024-02-19T15:22:43.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.229+0000 7f88a1400640 1 --2- 172.21.15.47:0/295699810 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8880066e40 0x7f8880069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:43.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.233+0000 7f88a1400640 1 --2- 172.21.15.47:0/295699810 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8880066e40 0x7f8880069300 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f889c1f7cb0 tx=0x7f888c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:43.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.233+0000 7f8896000640 1 -- 172.21.15.47:0/295699810 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f88900eba50 con 0x7f889c072850 2024-02-19T15:22:43.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.397+0000 7f88a4800640 1 -- 172.21.15.47:0/295699810 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f889c06e800 con 0x7f889c072850 2024-02-19T15:22:43.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.397+0000 7f8896000640 1 -- 172.21.15.47:0/295699810 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f88900b0020 con 0x7f889c072850 2024-02-19T15:22:43.400 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:22:43.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.397+0000 7f88a4800640 1 -- 172.21.15.47:0/295699810 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8880066e40 msgr2=0x7f8880069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:43.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.397+0000 7f88a4800640 1 --2- 172.21.15.47:0/295699810 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8880066e40 0x7f8880069300 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f889c1f7cb0 tx=0x7f888c009040 comp rx=0 tx=0).stop 2024-02-19T15:22:43.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.397+0000 7f88a4800640 1 -- 172.21.15.47:0/295699810 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f889c072850 msgr2=0x7f889c1542d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:43.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.397+0000 7f88a4800640 1 --2- 172.21.15.47:0/295699810 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f889c072850 0x7f889c1542d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f88900ae040 tx=0x7f889001abc0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.397+0000 7f88a4800640 1 -- 172.21.15.47:0/295699810 shutdown_connections 2024-02-19T15:22:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.397+0000 7f88a4800640 1 --2- 172.21.15.47:0/295699810 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8880066e40 0x7f8880069300 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.397+0000 7f88a4800640 1 --2- 172.21.15.47:0/295699810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f889c15bd90 0x7f889c0c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.397+0000 7f88a4800640 1 --2- 172.21.15.47:0/295699810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f889c0cb490 0x7f889c0ca510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.397+0000 7f88a4800640 1 --2- 172.21.15.47:0/295699810 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f889c072850 0x7f889c1542d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.397+0000 7f88a4800640 1 -- 172.21.15.47:0/295699810 >> 172.21.15.47:0/295699810 conn(0x7f889c0cfc20 msgr2=0x7f889c0b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.401+0000 7f88a4800640 1 -- 172.21.15.47:0/295699810 shutdown_connections 2024-02-19T15:22:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.401+0000 7f88a4800640 1 -- 172.21.15.47:0/295699810 wait complete. 2024-02-19T15:22:43.421 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:22:43.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.561+0000 7fcce5600640 1 Processor -- start 2024-02-19T15:22:43.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.561+0000 7fcce5600640 1 -- start start 2024-02-19T15:22:43.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.561+0000 7fcce5600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce00d2650 0x7fcce006c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:43.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.565+0000 7fcce5600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce00cc620 con 0x7fcce015e280 2024-02-19T15:22:43.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.565+0000 7fcce5600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce00d12e0 con 0x7fcce00ce3b0 2024-02-19T15:22:43.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.565+0000 7fcce5600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce00d1940 con 0x7fcce00d2650 2024-02-19T15:22:43.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.565+0000 7fccdf400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fcce015e280 0x7fcce006c710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57122/0 (socket says 172.21.15.47:57122) 2024-02-19T15:22:43.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.565+0000 7fccdf400640 1 -- 172.21.15.47:0/2210219697 learned_addr learned my addr 172.21.15.47:0/2210219697 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:43.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.565+0000 7fccd7e00640 1 --2- 172.21.15.47:0/2210219697 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce00d2650 0x7fcce006c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:43.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.565+0000 7fccde000640 1 -- 172.21.15.47:0/2210219697 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483669201 0 0) 0x7fcce00d12e0 con 0x7fcce00ce3b0 2024-02-19T15:22:43.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.565+0000 7fccde000640 1 -- 172.21.15.47:0/2210219697 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc4003530 con 0x7fcce00ce3b0 2024-02-19T15:22:43.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.565+0000 7fccde000640 1 -- 172.21.15.47:0/2210219697 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1546799196 0 0) 0x7fcce00cc620 con 0x7fcce015e280 2024-02-19T15:22:43.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.565+0000 7fccd7e00640 1 -- 172.21.15.47:0/2210219697 >> v1:172.21.15.116:6789/0 conn(0x7fcce00ce3b0 legacy=0x7fcce00ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:43.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.565+0000 7fccd7e00640 1 -- 172.21.15.47:0/2210219697 >> v1:172.21.15.47:6789/0 conn(0x7fcce015e280 legacy=0x7fcce006c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:43.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.565+0000 7fccd7e00640 1 -- 172.21.15.47:0/2210219697 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce00d1000 con 0x7fcce00d2650 2024-02-19T15:22:43.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.565+0000 7fccd7e00640 1 --2- 172.21.15.47:0/2210219697 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce00d2650 0x7fcce006c1d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fccc80097f0 tx=0x7fccc801c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b437ba0d1c5334ce server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:43.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fccde000640 1 -- 172.21.15.47:0/2210219697 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc80ab600 con 0x7fcce00d2650 2024-02-19T15:22:43.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fccde000640 1 -- 172.21.15.47:0/2210219697 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc80a7050 con 0x7fcce00d2650 2024-02-19T15:22:43.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fccde000640 1 -- 172.21.15.47:0/2210219697 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc80ab980 con 0x7fcce00d2650 2024-02-19T15:22:43.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fcce5600640 1 -- 172.21.15.47:0/2210219697 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce00d2650 msgr2=0x7fcce006c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:43.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fcce5600640 1 --2- 172.21.15.47:0/2210219697 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce00d2650 0x7fcce006c1d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fccc80097f0 tx=0x7fccc801c3c0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fcce5600640 1 -- 172.21.15.47:0/2210219697 shutdown_connections 2024-02-19T15:22:43.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fcce5600640 1 --2- 172.21.15.47:0/2210219697 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce00d2650 0x7fcce006c1d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fcce5600640 1 -- 172.21.15.47:0/2210219697 >> 172.21.15.47:0/2210219697 conn(0x7fcce00d1b30 msgr2=0x7fcce00d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:43.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fcce5600640 1 -- 172.21.15.47:0/2210219697 shutdown_connections 2024-02-19T15:22:43.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fcce5600640 1 -- 172.21.15.47:0/2210219697 wait complete. 2024-02-19T15:22:43.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fcce5600640 1 Processor -- start 2024-02-19T15:22:43.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fcce5600640 1 -- start start 2024-02-19T15:22:43.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fcce5600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce00ce3b0 0x7fcce0150c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:43.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fcce5600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcce00d2650 0x7fcce0154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:43.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fcce5600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcce015e280 0x7fcce0160030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:43.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fcce5600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce00ca540 con 0x7fcce00d2650 2024-02-19T15:22:43.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fcce5600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce015f290 con 0x7fcce015e280 2024-02-19T15:22:43.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fcce5600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce015f570 con 0x7fcce00ce3b0 2024-02-19T15:22:43.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fccd7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcce00d2650 0x7fcce0154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:43.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fccd7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcce00d2650 0x7fcce0154df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36198/0 (socket says 172.21.15.47:36198) 2024-02-19T15:22:43.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fccd7e00640 1 -- 172.21.15.47:0/2161824007 learned_addr learned my addr 172.21.15.47:0/2161824007 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:43.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fccdea00640 1 --2- 172.21.15.47:0/2161824007 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce00ce3b0 0x7fcce0150c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:43.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.569+0000 7fccdf400640 1 --2- 172.21.15.47:0/2161824007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcce015e280 0x7fcce0160030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:43.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.573+0000 7fccd7e00640 1 -- 172.21.15.47:0/2161824007 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce00ce3b0 msgr2=0x7fcce0150c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:43.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.573+0000 7fccd7e00640 1 --2- 172.21.15.47:0/2161824007 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce00ce3b0 0x7fcce0150c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.573+0000 7fccd7e00640 1 -- 172.21.15.47:0/2161824007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcce015e280 msgr2=0x7fcce0160030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:43.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.573+0000 7fccd7e00640 1 --2- 172.21.15.47:0/2161824007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcce015e280 0x7fcce0160030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.573+0000 7fccd7e00640 1 -- 172.21.15.47:0/2161824007 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc8009420 con 0x7fcce00d2650 2024-02-19T15:22:43.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.573+0000 7fccd7e00640 1 --2- 172.21.15.47:0/2161824007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcce00d2650 0x7fcce0154df0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fccc80ae040 tx=0x7fccc8019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:43.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.573+0000 7fccd6a00640 1 -- 172.21.15.47:0/2161824007 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc8004030 con 0x7fcce00d2650 2024-02-19T15:22:43.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.573+0000 7fccd6a00640 1 -- 172.21.15.47:0/2161824007 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc8004190 con 0x7fcce00d2650 2024-02-19T15:22:43.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.573+0000 7fccd6a00640 1 -- 172.21.15.47:0/2161824007 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc8004500 con 0x7fcce00d2650 2024-02-19T15:22:43.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.573+0000 7fcce5600640 1 -- 172.21.15.47:0/2161824007 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce015f850 con 0x7fcce00d2650 2024-02-19T15:22:43.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.573+0000 7fcce5600640 1 -- 172.21.15.47:0/2161824007 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce00cbcc0 con 0x7fcce00d2650 2024-02-19T15:22:43.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.573+0000 7fcce5600640 1 -- 172.21.15.47:0/2161824007 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce0075360 con 0x7fcce00d2650 2024-02-19T15:22:43.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.573+0000 7fccd6a00640 1 -- 172.21.15.47:0/2161824007 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fccc80b2040 con 0x7fcce00d2650 2024-02-19T15:22:43.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.573+0000 7fccd6a00640 1 --2- 172.21.15.47:0/2161824007 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fccbc066e40 0x7fccbc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:43.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.573+0000 7fccd6a00640 1 -- 172.21.15.47:0/2161824007 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fccc811eac0 con 0x7fcce00d2650 2024-02-19T15:22:43.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.573+0000 7fccdea00640 1 --2- 172.21.15.47:0/2161824007 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fccbc066e40 0x7fccbc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:43.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.577+0000 7fccdea00640 1 --2- 172.21.15.47:0/2161824007 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fccbc066e40 0x7fccbc069300 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fccd00011f0 tx=0x7fccd0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:43.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.577+0000 7fccd6a00640 1 -- 172.21.15.47:0/2161824007 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fccc80eb950 con 0x7fcce00d2650 2024-02-19T15:22:43.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.741+0000 7fcce5600640 1 -- 172.21.15.47:0/2161824007 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcce00cdf80 con 0x7fcce00d2650 2024-02-19T15:22:43.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.741+0000 7fccd6a00640 1 -- 172.21.15.47:0/2161824007 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fccc80c10a0 con 0x7fcce00d2650 2024-02-19T15:22:43.744 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:22:43.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.741+0000 7fcce5600640 1 -- 172.21.15.47:0/2161824007 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fccbc066e40 msgr2=0x7fccbc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.741+0000 7fcce5600640 1 --2- 172.21.15.47:0/2161824007 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fccbc066e40 0x7fccbc069300 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fccd00011f0 tx=0x7fccd0009040 comp rx=0 tx=0).stop 2024-02-19T15:22:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.741+0000 7fcce5600640 1 -- 172.21.15.47:0/2161824007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcce00d2650 msgr2=0x7fcce0154df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.741+0000 7fcce5600640 1 --2- 172.21.15.47:0/2161824007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcce00d2650 0x7fcce0154df0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fccc80ae040 tx=0x7fccc8019160 comp rx=0 tx=0).stop 2024-02-19T15:22:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.745+0000 7fcce5600640 1 -- 172.21.15.47:0/2161824007 shutdown_connections 2024-02-19T15:22:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.745+0000 7fcce5600640 1 --2- 172.21.15.47:0/2161824007 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fccbc066e40 0x7fccbc069300 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.745+0000 7fcce5600640 1 --2- 172.21.15.47:0/2161824007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcce015e280 0x7fcce0160030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.745+0000 7fcce5600640 1 --2- 172.21.15.47:0/2161824007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcce00d2650 0x7fcce0154df0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.745+0000 7fcce5600640 1 --2- 172.21.15.47:0/2161824007 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce00ce3b0 0x7fcce0150c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.745+0000 7fcce5600640 1 -- 172.21.15.47:0/2161824007 >> 172.21.15.47:0/2161824007 conn(0x7fcce00d1b30 msgr2=0x7fcce00c78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.745+0000 7fcce5600640 1 -- 172.21.15.47:0/2161824007 shutdown_connections 2024-02-19T15:22:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.745+0000 7fcce5600640 1 -- 172.21.15.47:0/2161824007 wait complete. 2024-02-19T15:22:43.765 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-19T15:22:43.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.901+0000 7f9343e00640 1 Processor -- start 2024-02-19T15:22:43.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.901+0000 7f9343e00640 1 -- start start 2024-02-19T15:22:43.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.905+0000 7f9343e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f934415dd20 0x7f93440ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:43.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.905+0000 7f9343e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f93440cd020 0x7f934406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:43.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.905+0000 7f9343e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93440ce6e0 con 0x7f934415e100 2024-02-19T15:22:43.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.905+0000 7f9343e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93440acec0 con 0x7f934415dd20 2024-02-19T15:22:43.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.905+0000 7f9343e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f934406c300 con 0x7f93440cd020 2024-02-19T15:22:43.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.905+0000 7f9342a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f934415dd20 0x7f93440ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:43.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.905+0000 7f9343400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f93440cd020 0x7f934406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:43.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.905+0000 7f9342a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f934415dd20 0x7f93440ca6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36202/0 (socket says 172.21.15.47:36202) 2024-02-19T15:22:43.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.905+0000 7f933a000640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f934415e100 0x7f93440cc910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:43032/0 (socket says 172.21.15.47:43032) 2024-02-19T15:22:43.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.905+0000 7f9342a00640 1 -- 172.21.15.47:0/1306686682 learned_addr learned my addr 172.21.15.47:0/1306686682 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:43.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.905+0000 7f9342a00640 1 -- 172.21.15.47:0/1306686682 >> v2:172.21.15.116:3300/0 conn(0x7f93440cd020 msgr2=0x7f934406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:43.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.905+0000 7f9342a00640 1 --2- 172.21.15.47:0/1306686682 >> v2:172.21.15.116:3300/0 conn(0x7f93440cd020 0x7f934406b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.905+0000 7f9342a00640 1 -- 172.21.15.47:0/1306686682 >> v1:172.21.15.116:6789/0 conn(0x7f934415e100 legacy=0x7f93440cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:43.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.905+0000 7f9342a00640 1 -- 172.21.15.47:0/1306686682 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934406c5e0 con 0x7f934415dd20 2024-02-19T15:22:43.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.905+0000 7f9342000640 1 -- 172.21.15.47:0/1306686682 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3340942927 0 0) 0x7f93440ce6e0 con 0x7f934415e100 2024-02-19T15:22:43.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.909+0000 7f9342a00640 1 --2- 172.21.15.47:0/1306686682 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f934415dd20 0x7f93440ca6e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9344161a20 tx=0x7f933401c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ab4fd0235cd8fa server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:43.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.909+0000 7f9342000640 1 -- 172.21.15.47:0/1306686682 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93340a56f0 con 0x7f934415dd20 2024-02-19T15:22:43.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.909+0000 7f9342000640 1 -- 172.21.15.47:0/1306686682 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93340a5850 con 0x7f934415dd20 2024-02-19T15:22:43.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.909+0000 7f9342000640 1 -- 172.21.15.47:0/1306686682 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93340a5be0 con 0x7f934415dd20 2024-02-19T15:22:43.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.909+0000 7f9343e00640 1 -- 172.21.15.47:0/1306686682 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f934415dd20 msgr2=0x7f93440ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:43.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.909+0000 7f9343e00640 1 --2- 172.21.15.47:0/1306686682 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f934415dd20 0x7f93440ca6e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9344161a20 tx=0x7f933401c8a0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.909+0000 7f9343e00640 1 -- 172.21.15.47:0/1306686682 shutdown_connections 2024-02-19T15:22:43.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.909+0000 7f9343e00640 1 --2- 172.21.15.47:0/1306686682 >> v2:172.21.15.116:3300/0 conn(0x7f93440cd020 0x7f934406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.909+0000 7f9343e00640 1 --2- 172.21.15.47:0/1306686682 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f934415dd20 0x7f93440ca6e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.909+0000 7f9343e00640 1 -- 172.21.15.47:0/1306686682 >> 172.21.15.47:0/1306686682 conn(0x7f93440b1be0 msgr2=0x7f93440b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:43.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.909+0000 7f9343e00640 1 -- 172.21.15.47:0/1306686682 shutdown_connections 2024-02-19T15:22:43.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.909+0000 7f9343e00640 1 -- 172.21.15.47:0/1306686682 wait complete. 2024-02-19T15:22:43.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.909+0000 7f9343e00640 1 Processor -- start 2024-02-19T15:22:43.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.909+0000 7f9343e00640 1 -- start start 2024-02-19T15:22:43.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.909+0000 7f9343e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93440cd020 0x7f9344155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:43.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.909+0000 7f9343e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f934415dd20 0x7f93440cb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:43.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.909+0000 7f9343e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f934415e100 0x7f934409a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:43.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.909+0000 7f9343e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9344159780 con 0x7f934415e100 2024-02-19T15:22:43.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.909+0000 7f9343e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934415b310 con 0x7f934415dd20 2024-02-19T15:22:43.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.909+0000 7f9343e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f934415b5f0 con 0x7f93440cd020 2024-02-19T15:22:43.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.913+0000 7f9343400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f934415e100 0x7f934409a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:43.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.913+0000 7f9343400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f934415e100 0x7f934409a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36204/0 (socket says 172.21.15.47:36204) 2024-02-19T15:22:43.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.913+0000 7f9343400640 1 -- 172.21.15.47:0/3827018626 learned_addr learned my addr 172.21.15.47:0/3827018626 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:43.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.913+0000 7f9342a00640 1 --2- 172.21.15.47:0/3827018626 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93440cd020 0x7f9344155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:43.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.913+0000 7f933a000640 1 --2- 172.21.15.47:0/3827018626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f934415dd20 0x7f93440cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:43.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.913+0000 7f9343400640 1 -- 172.21.15.47:0/3827018626 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93440cd020 msgr2=0x7f9344155570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:43.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.913+0000 7f9343400640 1 --2- 172.21.15.47:0/3827018626 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93440cd020 0x7f9344155570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.913+0000 7f9343400640 1 -- 172.21.15.47:0/3827018626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f934415dd20 msgr2=0x7f93440cb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:43.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.913+0000 7f9343400640 1 --2- 172.21.15.47:0/3827018626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f934415dd20 0x7f93440cb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:43.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.913+0000 7f9343400640 1 -- 172.21.15.47:0/3827018626 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9334009b30 con 0x7f934415e100 2024-02-19T15:22:43.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.913+0000 7f9343400640 1 --2- 172.21.15.47:0/3827018626 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f934415e100 0x7f934409a9d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f933c00e7f0 tx=0x7f933c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:43.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.913+0000 7f933b400640 1 -- 172.21.15.47:0/3827018626 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933c009530 con 0x7f934415e100 2024-02-19T15:22:43.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.913+0000 7f933b400640 1 -- 172.21.15.47:0/3827018626 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933c00eea0 con 0x7f934415e100 2024-02-19T15:22:43.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.913+0000 7f933b400640 1 -- 172.21.15.47:0/3827018626 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933c00cb60 con 0x7f934415e100 2024-02-19T15:22:43.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.913+0000 7f9343e00640 1 -- 172.21.15.47:0/3827018626 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9344159a40 con 0x7f934415e100 2024-02-19T15:22:43.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.913+0000 7f9343e00640 1 -- 172.21.15.47:0/3827018626 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9344155db0 con 0x7f934415e100 2024-02-19T15:22:43.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.913+0000 7f9343e00640 1 -- 172.21.15.47:0/3827018626 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93440cfe90 con 0x7f934415e100 2024-02-19T15:22:43.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.913+0000 7f933b400640 1 -- 172.21.15.47:0/3827018626 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f933c00ccc0 con 0x7f934415e100 2024-02-19T15:22:43.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.917+0000 7f933b400640 1 --2- 172.21.15.47:0/3827018626 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9320066e40 0x7f9320069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:43.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.917+0000 7f933b400640 1 -- 172.21.15.47:0/3827018626 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f933c084a60 con 0x7f934415e100 2024-02-19T15:22:43.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.917+0000 7f9342a00640 1 --2- 172.21.15.47:0/3827018626 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9320066e40 0x7f9320069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:43.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.917+0000 7f9342a00640 1 --2- 172.21.15.47:0/3827018626 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9320066e40 0x7f9320069300 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f93340ae000 tx=0x7f93340a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:43.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:43.921+0000 7f933b400640 1 -- 172.21.15.47:0/3827018626 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f933c0518f0 con 0x7f934415e100 2024-02-19T15:22:44.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.097+0000 7f9343e00640 1 -- 172.21.15.47:0/3827018626 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9344159c30 con 0x7f934415e100 2024-02-19T15:22:44.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.097+0000 7f933b400640 1 -- 172.21.15.47:0/3827018626 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f933c017080 con 0x7f934415e100 2024-02-19T15:22:44.102 INFO:teuthology.orchestra.run.smithi047.stderr:listed 0 entries 2024-02-19T15:22:44.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.101+0000 7f9343e00640 1 -- 172.21.15.47:0/3827018626 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9320066e40 msgr2=0x7f9320069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:44.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.101+0000 7f9343e00640 1 --2- 172.21.15.47:0/3827018626 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9320066e40 0x7f9320069300 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f93340ae000 tx=0x7f93340a7070 comp rx=0 tx=0).stop 2024-02-19T15:22:44.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.101+0000 7f9343e00640 1 -- 172.21.15.47:0/3827018626 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f934415e100 msgr2=0x7f934409a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:44.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.101+0000 7f9343e00640 1 --2- 172.21.15.47:0/3827018626 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f934415e100 0x7f934409a9d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f933c00e7f0 tx=0x7f933c00ecb0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.101+0000 7f9343e00640 1 -- 172.21.15.47:0/3827018626 shutdown_connections 2024-02-19T15:22:44.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.101+0000 7f9343e00640 1 --2- 172.21.15.47:0/3827018626 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9320066e40 0x7f9320069300 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.101+0000 7f9343e00640 1 --2- 172.21.15.47:0/3827018626 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f934415e100 0x7f934409a9d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.101+0000 7f9343e00640 1 --2- 172.21.15.47:0/3827018626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f934415dd20 0x7f93440cb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.101+0000 7f9343e00640 1 --2- 172.21.15.47:0/3827018626 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93440cd020 0x7f9344155570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.101+0000 7f9343e00640 1 -- 172.21.15.47:0/3827018626 >> 172.21.15.47:0/3827018626 conn(0x7f93440b1be0 msgr2=0x7f93440cbd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:44.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.101+0000 7f9343e00640 1 -- 172.21.15.47:0/3827018626 shutdown_connections 2024-02-19T15:22:44.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.101+0000 7f9343e00640 1 -- 172.21.15.47:0/3827018626 wait complete. 2024-02-19T15:22:44.122 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-02-19T15:22:44.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:44.123 DEBUG:teuthology.orchestra.run.smithi116:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-02-19T15:22:44.146 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2024-02-19T15:22:44.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:22:44.146 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-02-19T15:22:44.202 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-02-19T15:22:44.202 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:22:44.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.341+0000 7fd1d8e00640 1 Processor -- start 2024-02-19T15:22:44.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.341+0000 7fd1d8e00640 1 -- start start 2024-02-19T15:22:44.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.341+0000 7fd1d8e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd1d40cea80 0x7fd1d40727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:44.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.341+0000 7fd1d8e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d415d280 con 0x7fd1d4145590 2024-02-19T15:22:44.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.341+0000 7fd1d8e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d4070ff0 con 0x7fd1d415c390 2024-02-19T15:22:44.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.341+0000 7fd1d8e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d40709e0 con 0x7fd1d40cea80 2024-02-19T15:22:44.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.341+0000 7fd1d1e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd1d4145590 0x7fd1d406c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57136/0 (socket says 172.21.15.47:57136) 2024-02-19T15:22:44.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.341+0000 7fd1d1e00640 1 -- 172.21.15.47:0/3043126598 learned_addr learned my addr 172.21.15.47:0/3043126598 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:44.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.341+0000 7fd1d1400640 1 --2- 172.21.15.47:0/3043126598 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd1d40cea80 0x7fd1d40727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:44.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.341+0000 7fd1d0a00640 1 -- 172.21.15.47:0/3043126598 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438118007 0 0) 0x7fd1d415d280 con 0x7fd1d4145590 2024-02-19T15:22:44.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.341+0000 7fd1d0a00640 1 -- 172.21.15.47:0/3043126598 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1b8003530 con 0x7fd1d4145590 2024-02-19T15:22:44.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.341+0000 7fd1d0a00640 1 -- 172.21.15.47:0/3043126598 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567342754 0 0) 0x7fd1d4070ff0 con 0x7fd1d415c390 2024-02-19T15:22:44.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.341+0000 7fd1d0a00640 1 -- 172.21.15.47:0/3043126598 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1d415d280 con 0x7fd1d415c390 2024-02-19T15:22:44.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.341+0000 7fd1d1400640 1 -- 172.21.15.47:0/3043126598 >> v1:172.21.15.116:6789/0 conn(0x7fd1d415c390 legacy=0x7fd1d415c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:44.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.341+0000 7fd1d1400640 1 -- 172.21.15.47:0/3043126598 >> v1:172.21.15.47:6789/0 conn(0x7fd1d4145590 legacy=0x7fd1d406c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:44.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.341+0000 7fd1d1400640 1 -- 172.21.15.47:0/3043126598 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d40cf020 con 0x7fd1d40cea80 2024-02-19T15:22:44.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.345+0000 7fd1d1400640 1 --2- 172.21.15.47:0/3043126598 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd1d40cea80 0x7fd1d40727b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd1c40097f0 tx=0x7fd1c401c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd1080a8189f3fed server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:44.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.345+0000 7fd1d0a00640 1 -- 172.21.15.47:0/3043126598 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c40ab5f0 con 0x7fd1d40cea80 2024-02-19T15:22:44.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.345+0000 7fd1d0a00640 1 -- 172.21.15.47:0/3043126598 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c40ab750 con 0x7fd1d40cea80 2024-02-19T15:22:44.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.345+0000 7fd1d0a00640 1 -- 172.21.15.47:0/3043126598 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c40abae0 con 0x7fd1d40cea80 2024-02-19T15:22:44.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.345+0000 7fd1d8e00640 1 -- 172.21.15.47:0/3043126598 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd1d40cea80 msgr2=0x7fd1d40727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:44.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.345+0000 7fd1d8e00640 1 --2- 172.21.15.47:0/3043126598 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd1d40cea80 0x7fd1d40727b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd1c40097f0 tx=0x7fd1c401c4e0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.345+0000 7fd1d8e00640 1 -- 172.21.15.47:0/3043126598 shutdown_connections 2024-02-19T15:22:44.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.345+0000 7fd1d8e00640 1 --2- 172.21.15.47:0/3043126598 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd1d40cea80 0x7fd1d40727b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.345+0000 7fd1d8e00640 1 -- 172.21.15.47:0/3043126598 >> 172.21.15.47:0/3043126598 conn(0x7fd1d40b16a0 msgr2=0x7fd1d40b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:44.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.345+0000 7fd1d8e00640 1 -- 172.21.15.47:0/3043126598 shutdown_connections 2024-02-19T15:22:44.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.345+0000 7fd1d8e00640 1 -- 172.21.15.47:0/3043126598 wait complete. 2024-02-19T15:22:44.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.345+0000 7fd1d8e00640 1 Processor -- start 2024-02-19T15:22:44.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.345+0000 7fd1d8e00640 1 -- start start 2024-02-19T15:22:44.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d8e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1d40cea80 0x7fd1d40b40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:44.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d8e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd1d4145590 0x7fd1d415b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:44.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d8e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d415c390 0x7fd1d413bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:44.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d8e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d4143b90 con 0x7fd1d415c390 2024-02-19T15:22:44.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d8e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d40b2e00 con 0x7fd1d40cea80 2024-02-19T15:22:44.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d8e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d40b3050 con 0x7fd1d4145590 2024-02-19T15:22:44.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d1e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1d40cea80 0x7fd1d40b40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:44.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d2800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d415c390 0x7fd1d413bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:44.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d1400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd1d4145590 0x7fd1d415b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:44.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d2800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d415c390 0x7fd1d413bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36208/0 (socket says 172.21.15.47:36208) 2024-02-19T15:22:44.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d2800640 1 -- 172.21.15.47:0/2110079183 learned_addr learned my addr 172.21.15.47:0/2110079183 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:44.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d1400640 1 -- 172.21.15.47:0/2110079183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1d40cea80 msgr2=0x7fd1d40b40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:44.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d1400640 1 --2- 172.21.15.47:0/2110079183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1d40cea80 0x7fd1d40b40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d1400640 1 -- 172.21.15.47:0/2110079183 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d415c390 msgr2=0x7fd1d413bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:44.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d1400640 1 --2- 172.21.15.47:0/2110079183 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d415c390 0x7fd1d413bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d1400640 1 -- 172.21.15.47:0/2110079183 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c4009420 con 0x7fd1d4145590 2024-02-19T15:22:44.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d2800640 1 --2- 172.21.15.47:0/2110079183 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d415c390 0x7fd1d413bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:44.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d1400640 1 --2- 172.21.15.47:0/2110079183 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd1d4145590 0x7fd1d415b4d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd1c40ac000 tx=0x7fd1c401ad20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:44.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1ca000640 1 -- 172.21.15.47:0/2110079183 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c40bf610 con 0x7fd1d4145590 2024-02-19T15:22:44.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1ca000640 1 -- 172.21.15.47:0/2110079183 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c40bf7b0 con 0x7fd1d4145590 2024-02-19T15:22:44.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d8e00640 1 -- 172.21.15.47:0/2110079183 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d40b3330 con 0x7fd1d4145590 2024-02-19T15:22:44.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1ca000640 1 -- 172.21.15.47:0/2110079183 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c40bfb60 con 0x7fd1d4145590 2024-02-19T15:22:44.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d8e00640 1 -- 172.21.15.47:0/2110079183 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d40b38f0 con 0x7fd1d4145590 2024-02-19T15:22:44.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.349+0000 7fd1d8e00640 1 -- 172.21.15.47:0/2110079183 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d4075360 con 0x7fd1d4145590 2024-02-19T15:22:44.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.353+0000 7fd1ca000640 1 -- 172.21.15.47:0/2110079183 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd1c40b5400 con 0x7fd1d4145590 2024-02-19T15:22:44.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.353+0000 7fd1ca000640 1 --2- 172.21.15.47:0/2110079183 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd1b4066e40 0x7fd1b4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:44.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.353+0000 7fd1ca000640 1 -- 172.21.15.47:0/2110079183 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd1c40a8050 con 0x7fd1d4145590 2024-02-19T15:22:44.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.353+0000 7fd1d1e00640 1 --2- 172.21.15.47:0/2110079183 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd1b4066e40 0x7fd1b4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:44.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.353+0000 7fd1d1e00640 1 --2- 172.21.15.47:0/2110079183 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd1b4066e40 0x7fd1b4069300 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc0012f0 tx=0x7fd1cc001ac0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:44.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.357+0000 7fd1ca000640 1 -- 172.21.15.47:0/2110079183 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd1c40ec070 con 0x7fd1d4145590 2024-02-19T15:22:44.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.553+0000 7fd1d8e00640 1 -- 172.21.15.47:0/2110079183 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1d41410e0 con 0x7fd1d4145590 2024-02-19T15:22:44.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.553+0000 7fd1ca000640 1 -- 172.21.15.47:0/2110079183 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4935 (secure 0 0 0) 0x7fd1c40ef730 con 0x7fd1d4145590 2024-02-19T15:22:44.558 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:44.558 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3873760209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3873760209},{"type":"v1","addr":"172.21.15.116:6836","nonce":3873760209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/284820350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":284820350},{"type":"v1","addr":"172.21.15.47:6834","nonce":284820350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2764089359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2764089359},{"type":"v1","addr":"172.21.15.47:6835","nonce":2764089359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:22:05.228517+0000","modified":"2024-02-19T15:22:08.242600+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6837/3920452343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3920452343},{"type":"v1","addr":"172.21.15.116:6837","nonce":3920452343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-19T15:22:44.558 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 8 2024-02-19T15:22:44.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.557+0000 7fd1d8e00640 1 -- 172.21.15.47:0/2110079183 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd1b4066e40 msgr2=0x7fd1b4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:44.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.557+0000 7fd1d8e00640 1 --2- 172.21.15.47:0/2110079183 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd1b4066e40 0x7fd1b4069300 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc0012f0 tx=0x7fd1cc001ac0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.557+0000 7fd1d8e00640 1 -- 172.21.15.47:0/2110079183 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd1d4145590 msgr2=0x7fd1d415b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:44.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.557+0000 7fd1d8e00640 1 --2- 172.21.15.47:0/2110079183 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd1d4145590 0x7fd1d415b4d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd1c40ac000 tx=0x7fd1c401ad20 comp rx=0 tx=0).stop 2024-02-19T15:22:44.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.557+0000 7fd1d8e00640 1 -- 172.21.15.47:0/2110079183 shutdown_connections 2024-02-19T15:22:44.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.557+0000 7fd1d8e00640 1 --2- 172.21.15.47:0/2110079183 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd1b4066e40 0x7fd1b4069300 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.557+0000 7fd1d8e00640 1 --2- 172.21.15.47:0/2110079183 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1d415c390 0x7fd1d413bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.557+0000 7fd1d8e00640 1 --2- 172.21.15.47:0/2110079183 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd1d4145590 0x7fd1d415b4d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.557+0000 7fd1d8e00640 1 --2- 172.21.15.47:0/2110079183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1d40cea80 0x7fd1d40b40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.557+0000 7fd1d8e00640 1 -- 172.21.15.47:0/2110079183 >> 172.21.15.47:0/2110079183 conn(0x7fd1d40b16a0 msgr2=0x7fd1d41395b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:44.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.557+0000 7fd1d8e00640 1 -- 172.21.15.47:0/2110079183 shutdown_connections 2024-02-19T15:22:44.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.557+0000 7fd1d8e00640 1 -- 172.21.15.47:0/2110079183 wait complete. 2024-02-19T15:22:44.579 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:44.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.717+0000 7f1471a00640 1 Processor -- start 2024-02-19T15:22:44.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.717+0000 7f1471a00640 1 -- start start 2024-02-19T15:22:44.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.717+0000 7f1471a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f146c071990 0x7f146c1600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:44.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.717+0000 7f1471a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f146c160600 0x7f146c071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:44.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.717+0000 7f1471a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f146c15fc30 con 0x7f146c0b0010 2024-02-19T15:22:44.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.717+0000 7f1471a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f146c072880 con 0x7f146c160600 2024-02-19T15:22:44.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.717+0000 7f1471a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f146c06b9f0 con 0x7f146c071990 2024-02-19T15:22:44.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.717+0000 7f146b400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f146c0b0010 0x7f146c06edf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57148/0 (socket says 172.21.15.47:57148) 2024-02-19T15:22:44.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.717+0000 7f146b400640 1 -- 172.21.15.47:0/1954706676 learned_addr learned my addr 172.21.15.47:0/1954706676 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:44.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.717+0000 7f146a000640 1 --2- 172.21.15.47:0/1954706676 >> v2:172.21.15.116:3300/0 conn(0x7f146c160600 0x7f146c071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:44.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.717+0000 7f146aa00640 1 --2- 172.21.15.47:0/1954706676 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f146c071990 0x7f146c1600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:44.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.717+0000 7f1469600640 1 -- 172.21.15.47:0/1954706676 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951521269 0 0) 0x7f146c15fc30 con 0x7f146c0b0010 2024-02-19T15:22:44.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.717+0000 7f1469600640 1 -- 172.21.15.47:0/1954706676 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1450003530 con 0x7f146c0b0010 2024-02-19T15:22:44.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.717+0000 7f146a000640 1 -- 172.21.15.47:0/1954706676 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f146c071990 msgr2=0x7f146c1600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:44.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.717+0000 7f146a000640 1 --2- 172.21.15.47:0/1954706676 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f146c071990 0x7f146c1600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.717+0000 7f146a000640 1 -- 172.21.15.47:0/1954706676 >> v1:172.21.15.47:6789/0 conn(0x7f146c0b0010 legacy=0x7f146c06edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:44.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.717+0000 7f146a000640 1 -- 172.21.15.47:0/1954706676 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146c06bc90 con 0x7f146c160600 2024-02-19T15:22:44.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.721+0000 7f146a000640 1 --2- 172.21.15.47:0/1954706676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f146c160600 0x7f146c071d70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f14580097f0 tx=0x7f145801c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6ad28dd49faa5a2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:44.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.721+0000 7f1469600640 1 -- 172.21.15.47:0/1954706676 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14580ab600 con 0x7f146c160600 2024-02-19T15:22:44.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.721+0000 7f1469600640 1 -- 172.21.15.47:0/1954706676 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14580a7050 con 0x7f146c160600 2024-02-19T15:22:44.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.721+0000 7f1469600640 1 -- 172.21.15.47:0/1954706676 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14580ab980 con 0x7f146c160600 2024-02-19T15:22:44.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.721+0000 7f1471a00640 1 -- 172.21.15.47:0/1954706676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f146c160600 msgr2=0x7f146c071d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:44.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.721+0000 7f1471a00640 1 --2- 172.21.15.47:0/1954706676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f146c160600 0x7f146c071d70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f14580097f0 tx=0x7f145801c3e0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.721+0000 7f1471a00640 1 -- 172.21.15.47:0/1954706676 shutdown_connections 2024-02-19T15:22:44.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.721+0000 7f1471a00640 1 --2- 172.21.15.47:0/1954706676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f146c160600 0x7f146c071d70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.721+0000 7f1471a00640 1 --2- 172.21.15.47:0/1954706676 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f146c071990 0x7f146c1600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.721+0000 7f1471a00640 1 -- 172.21.15.47:0/1954706676 >> 172.21.15.47:0/1954706676 conn(0x7f146c0d0b10 msgr2=0x7f146c0d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:44.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.721+0000 7f1471a00640 1 -- 172.21.15.47:0/1954706676 shutdown_connections 2024-02-19T15:22:44.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.721+0000 7f1471a00640 1 -- 172.21.15.47:0/1954706676 wait complete. 2024-02-19T15:22:44.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.721+0000 7f1471a00640 1 Processor -- start 2024-02-19T15:22:44.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.721+0000 7f1471a00640 1 -- start start 2024-02-19T15:22:44.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f1471a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f146c071990 0x7f146c0cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:44.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f1471a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f146c0b0010 0x7f146c0b62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:44.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f1471a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f146c160600 0x7f146c0cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:44.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f1471a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146c0b4df0 con 0x7f146c160600 2024-02-19T15:22:44.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f1471a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146c0b55d0 con 0x7f146c071990 2024-02-19T15:22:44.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f1471a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f146c1fb5b0 con 0x7f146c0b0010 2024-02-19T15:22:44.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f146b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f146c160600 0x7f146c0cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:44.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f146b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f146c160600 0x7f146c0cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36212/0 (socket says 172.21.15.47:36212) 2024-02-19T15:22:44.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f146b400640 1 -- 172.21.15.47:0/3111588828 learned_addr learned my addr 172.21.15.47:0/3111588828 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:44.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f146aa00640 1 --2- 172.21.15.47:0/3111588828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f146c071990 0x7f146c0cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:44.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f146a000640 1 --2- 172.21.15.47:0/3111588828 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f146c0b0010 0x7f146c0b62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:44.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f146b400640 1 -- 172.21.15.47:0/3111588828 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f146c0b0010 msgr2=0x7f146c0b62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:44.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f146b400640 1 --2- 172.21.15.47:0/3111588828 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f146c0b0010 0x7f146c0b62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f146b400640 1 -- 172.21.15.47:0/3111588828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f146c071990 msgr2=0x7f146c0cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:44.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f146b400640 1 --2- 172.21.15.47:0/3111588828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f146c071990 0x7f146c0cdac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f146b400640 1 -- 172.21.15.47:0/3111588828 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1458009420 con 0x7f146c160600 2024-02-19T15:22:44.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f146b400640 1 --2- 172.21.15.47:0/3111588828 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f146c160600 0x7f146c0cc730 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f146400cb70 tx=0x7f146400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:44.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f1462a00640 1 -- 172.21.15.47:0/3111588828 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146400ad40 con 0x7f146c160600 2024-02-19T15:22:44.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f1462a00640 1 -- 172.21.15.47:0/3111588828 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1464012400 con 0x7f146c160600 2024-02-19T15:22:44.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f1471a00640 1 -- 172.21.15.47:0/3111588828 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f146c0b5e50 con 0x7f146c160600 2024-02-19T15:22:44.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f1462a00640 1 -- 172.21.15.47:0/3111588828 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146400aea0 con 0x7f146c160600 2024-02-19T15:22:44.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f1471a00640 1 -- 172.21.15.47:0/3111588828 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f146c0ce000 con 0x7f146c160600 2024-02-19T15:22:44.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f1471a00640 1 -- 172.21.15.47:0/3111588828 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f146c0b5fe0 con 0x7f146c160600 2024-02-19T15:22:44.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.725+0000 7f1462a00640 1 -- 172.21.15.47:0/3111588828 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1464007070 con 0x7f146c160600 2024-02-19T15:22:44.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.729+0000 7f1462a00640 1 --2- 172.21.15.47:0/3111588828 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f144c066e40 0x7f144c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:44.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.729+0000 7f1462a00640 1 -- 172.21.15.47:0/3111588828 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1464084a60 con 0x7f146c160600 2024-02-19T15:22:44.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.729+0000 7f146aa00640 1 --2- 172.21.15.47:0/3111588828 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f144c066e40 0x7f144c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:44.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.729+0000 7f146aa00640 1 --2- 172.21.15.47:0/3111588828 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f144c066e40 0x7f144c069300 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f145c009b60 tx=0x7f145c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:44.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.733+0000 7f1462a00640 1 -- 172.21.15.47:0/3111588828 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1464051860 con 0x7f146c160600 2024-02-19T15:22:44.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.897+0000 7f1471a00640 1 -- 172.21.15.47:0/3111588828 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f146c075360 con 0x7f146c160600 2024-02-19T15:22:44.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.897+0000 7f1462a00640 1 -- 172.21.15.47:0/3111588828 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f1464017020 con 0x7f146c160600 2024-02-19T15:22:44.901 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:44.901 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":21,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:06.843155+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:03.091360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:04.222965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:44.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.901+0000 7f1471a00640 1 -- 172.21.15.47:0/3111588828 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f144c066e40 msgr2=0x7f144c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:44.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.901+0000 7f1471a00640 1 --2- 172.21.15.47:0/3111588828 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f144c066e40 0x7f144c069300 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f145c009b60 tx=0x7f145c009190 comp rx=0 tx=0).stop 2024-02-19T15:22:44.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.901+0000 7f1471a00640 1 -- 172.21.15.47:0/3111588828 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f146c160600 msgr2=0x7f146c0cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:44.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.901+0000 7f1471a00640 1 --2- 172.21.15.47:0/3111588828 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f146c160600 0x7f146c0cc730 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f146400cb70 tx=0x7f146400a430 comp rx=0 tx=0).stop 2024-02-19T15:22:44.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.901+0000 7f1471a00640 1 -- 172.21.15.47:0/3111588828 shutdown_connections 2024-02-19T15:22:44.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.901+0000 7f1471a00640 1 --2- 172.21.15.47:0/3111588828 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f144c066e40 0x7f144c069300 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.901+0000 7f1471a00640 1 --2- 172.21.15.47:0/3111588828 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f146c160600 0x7f146c0cc730 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.901+0000 7f1471a00640 1 --2- 172.21.15.47:0/3111588828 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f146c0b0010 0x7f146c0b62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.901+0000 7f1471a00640 1 --2- 172.21.15.47:0/3111588828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f146c071990 0x7f146c0cdac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:44.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.901+0000 7f1471a00640 1 -- 172.21.15.47:0/3111588828 >> 172.21.15.47:0/3111588828 conn(0x7f146c0d0b10 msgr2=0x7f146c166390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:44.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.901+0000 7f1471a00640 1 -- 172.21.15.47:0/3111588828 shutdown_connections 2024-02-19T15:22:44.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:44.901+0000 7f1471a00640 1 -- 172.21.15.47:0/3111588828 wait complete. 2024-02-19T15:22:44.923 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:22:03.091360+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:22:04.222965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:22:44.923 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:22:45.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.061+0000 7fb04ec00640 1 Processor -- start 2024-02-19T15:22:45.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.065+0000 7fb04ec00640 1 -- start start 2024-02-19T15:22:45.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.065+0000 7fb04ec00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb04815dfb0 0x7fb04815e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:45.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.065+0000 7fb04ec00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb0480d2480 0x7fb0480ce1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:45.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.065+0000 7fb04ec00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb0480b2d70 0x7fb0480ce720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:45.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.065+0000 7fb04ec00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0480cbfb0 con 0x7fb0480b2d70 2024-02-19T15:22:45.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.065+0000 7fb04ec00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0480d0980 con 0x7fb0480d2480 2024-02-19T15:22:45.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.065+0000 7fb04ec00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0480d0c60 con 0x7fb04815dfb0 2024-02-19T15:22:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.065+0000 7fb04e200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb0480b2d70 0x7fb0480ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.065+0000 7fb04e200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb0480b2d70 0x7fb0480ce720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36216/0 (socket says 172.21.15.47:36216) 2024-02-19T15:22:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.065+0000 7fb04e200640 1 -- 172.21.15.47:0/4253044796 learned_addr learned my addr 172.21.15.47:0/4253044796 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.065+0000 7fb04ce00640 1 --2- 172.21.15.47:0/4253044796 >> v2:172.21.15.116:3300/0 conn(0x7fb0480d2480 0x7fb0480ce1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.065+0000 7fb04d800640 1 --2- 172.21.15.47:0/4253044796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb04815dfb0 0x7fb04815e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.065+0000 7fb04e200640 1 -- 172.21.15.47:0/4253044796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb04815dfb0 msgr2=0x7fb04815e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.065+0000 7fb04e200640 1 --2- 172.21.15.47:0/4253044796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb04815dfb0 0x7fb04815e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.065+0000 7fb04e200640 1 -- 172.21.15.47:0/4253044796 >> v2:172.21.15.116:3300/0 conn(0x7fb0480d2480 msgr2=0x7fb0480ce1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.065+0000 7fb04e200640 1 --2- 172.21.15.47:0/4253044796 >> v2:172.21.15.116:3300/0 conn(0x7fb0480d2480 0x7fb0480ce1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.065+0000 7fb04e200640 1 -- 172.21.15.47:0/4253044796 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0480d0f40 con 0x7fb0480b2d70 2024-02-19T15:22:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.069+0000 7fb04e200640 1 --2- 172.21.15.47:0/4253044796 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0480b2d70 0x7fb0480ce720 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb048072420 tx=0x7fb04001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=836ee5d4bdac7f49 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.069+0000 7fb047e00640 1 -- 172.21.15.47:0/4253044796 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0400a5890 con 0x7fb0480b2d70 2024-02-19T15:22:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.069+0000 7fb047e00640 1 -- 172.21.15.47:0/4253044796 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0400a59f0 con 0x7fb0480b2d70 2024-02-19T15:22:45.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.069+0000 7fb047e00640 1 -- 172.21.15.47:0/4253044796 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0400a5d80 con 0x7fb0480b2d70 2024-02-19T15:22:45.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.069+0000 7fb04ec00640 1 -- 172.21.15.47:0/4253044796 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0480b2d70 msgr2=0x7fb0480ce720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:45.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.069+0000 7fb04ec00640 1 --2- 172.21.15.47:0/4253044796 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0480b2d70 0x7fb0480ce720 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb048072420 tx=0x7fb04001c9f0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.069+0000 7fb04ec00640 1 -- 172.21.15.47:0/4253044796 shutdown_connections 2024-02-19T15:22:45.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.069+0000 7fb04ec00640 1 --2- 172.21.15.47:0/4253044796 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0480b2d70 0x7fb0480ce720 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.069+0000 7fb04ec00640 1 --2- 172.21.15.47:0/4253044796 >> v2:172.21.15.116:3300/0 conn(0x7fb0480d2480 0x7fb0480ce1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.069+0000 7fb04ec00640 1 --2- 172.21.15.47:0/4253044796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb04815dfb0 0x7fb04815e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.069+0000 7fb04ec00640 1 -- 172.21.15.47:0/4253044796 >> 172.21.15.47:0/4253044796 conn(0x7fb0480d1960 msgr2=0x7fb0480d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:45.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.069+0000 7fb04ec00640 1 -- 172.21.15.47:0/4253044796 shutdown_connections 2024-02-19T15:22:45.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.069+0000 7fb04ec00640 1 -- 172.21.15.47:0/4253044796 wait complete. 2024-02-19T15:22:45.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.069+0000 7fb04ec00640 1 Processor -- start 2024-02-19T15:22:45.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.069+0000 7fb04ec00640 1 -- start start 2024-02-19T15:22:45.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb04ec00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0480b2d70 0x7fb04815d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:45.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb04ec00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb04815dfb0 0x7fb048158d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:45.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb04ec00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb048159280 0x7fb04815bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:45.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb04ec00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb048159710 con 0x7fb0480b2d70 2024-02-19T15:22:45.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb04ec00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb048152130 con 0x7fb048159280 2024-02-19T15:22:45.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb04ec00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb048152f90 con 0x7fb04815dfb0 2024-02-19T15:22:45.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb04d800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0480b2d70 0x7fb04815d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:45.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb04d800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0480b2d70 0x7fb04815d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36224/0 (socket says 172.21.15.47:36224) 2024-02-19T15:22:45.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb04d800640 1 -- 172.21.15.47:0/2795603260 learned_addr learned my addr 172.21.15.47:0/2795603260 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:45.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb04ce00640 1 --2- 172.21.15.47:0/2795603260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb04815dfb0 0x7fb048158d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:45.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb04e200640 1 --2- 172.21.15.47:0/2795603260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb048159280 0x7fb04815bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:45.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb04d800640 1 -- 172.21.15.47:0/2795603260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb04815dfb0 msgr2=0x7fb048158d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:45.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb04d800640 1 --2- 172.21.15.47:0/2795603260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb04815dfb0 0x7fb048158d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb04d800640 1 -- 172.21.15.47:0/2795603260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb048159280 msgr2=0x7fb04815bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:45.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb04d800640 1 --2- 172.21.15.47:0/2795603260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb048159280 0x7fb04815bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb04d800640 1 -- 172.21.15.47:0/2795603260 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb040009b30 con 0x7fb0480b2d70 2024-02-19T15:22:45.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb04d800640 1 --2- 172.21.15.47:0/2795603260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0480b2d70 0x7fb04815d230 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00e960 tx=0x7fb03c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:45.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb045600640 1 -- 172.21.15.47:0/2795603260 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c009690 con 0x7fb0480b2d70 2024-02-19T15:22:45.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb045600640 1 -- 172.21.15.47:0/2795603260 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03c00ca00 con 0x7fb0480b2d70 2024-02-19T15:22:45.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb045600640 1 -- 172.21.15.47:0/2795603260 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c00cd70 con 0x7fb0480b2d70 2024-02-19T15:22:45.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb04ec00640 1 -- 172.21.15.47:0/2795603260 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb048152380 con 0x7fb0480b2d70 2024-02-19T15:22:45.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb04ec00640 1 -- 172.21.15.47:0/2795603260 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb048152940 con 0x7fb0480b2d70 2024-02-19T15:22:45.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb04ec00640 1 -- 172.21.15.47:0/2795603260 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb04804f260 con 0x7fb0480b2d70 2024-02-19T15:22:45.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.073+0000 7fb045600640 1 -- 172.21.15.47:0/2795603260 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb03c013470 con 0x7fb0480b2d70 2024-02-19T15:22:45.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.077+0000 7fb045600640 1 --2- 172.21.15.47:0/2795603260 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb028066e70 0x7fb028069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:45.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.077+0000 7fb045600640 1 -- 172.21.15.47:0/2795603260 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb03c084bb0 con 0x7fb0480b2d70 2024-02-19T15:22:45.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.077+0000 7fb04ce00640 1 --2- 172.21.15.47:0/2795603260 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb028066e70 0x7fb028069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:45.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.077+0000 7fb04ce00640 1 --2- 172.21.15.47:0/2795603260 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb028066e70 0x7fb028069330 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb04815ba20 tx=0x7fb034009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:45.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.081+0000 7fb045600640 1 -- 172.21.15.47:0/2795603260 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb03c051a50 con 0x7fb0480b2d70 2024-02-19T15:22:45.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.241+0000 7fb04ec00640 1 -- 172.21.15.47:0/2795603260 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb0480d2160 con 0x7fb0480b2d70 2024-02-19T15:22:45.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.241+0000 7fb045600640 1 -- 172.21.15.47:0/2795603260 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb03c0170a0 con 0x7fb0480b2d70 2024-02-19T15:22:45.246 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:22:45.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.245+0000 7fb04ec00640 1 -- 172.21.15.47:0/2795603260 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb028066e70 msgr2=0x7fb028069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:45.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.245+0000 7fb04ec00640 1 --2- 172.21.15.47:0/2795603260 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb028066e70 0x7fb028069330 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb04815ba20 tx=0x7fb034009190 comp rx=0 tx=0).stop 2024-02-19T15:22:45.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.245+0000 7fb04ec00640 1 -- 172.21.15.47:0/2795603260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0480b2d70 msgr2=0x7fb04815d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:45.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.245+0000 7fb04ec00640 1 --2- 172.21.15.47:0/2795603260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0480b2d70 0x7fb04815d230 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00e960 tx=0x7fb03c00ee20 comp rx=0 tx=0).stop 2024-02-19T15:22:45.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.245+0000 7fb04ec00640 1 -- 172.21.15.47:0/2795603260 shutdown_connections 2024-02-19T15:22:45.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.245+0000 7fb04ec00640 1 --2- 172.21.15.47:0/2795603260 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb028066e70 0x7fb028069330 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.245+0000 7fb04ec00640 1 --2- 172.21.15.47:0/2795603260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb048159280 0x7fb04815bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.245+0000 7fb04ec00640 1 --2- 172.21.15.47:0/2795603260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb04815dfb0 0x7fb048158d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.245+0000 7fb04ec00640 1 --2- 172.21.15.47:0/2795603260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0480b2d70 0x7fb04815d230 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.245+0000 7fb04ec00640 1 -- 172.21.15.47:0/2795603260 >> 172.21.15.47:0/2795603260 conn(0x7fb0480d1960 msgr2=0x7fb04806ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:45.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.245+0000 7fb04ec00640 1 -- 172.21.15.47:0/2795603260 shutdown_connections 2024-02-19T15:22:45.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.245+0000 7fb04ec00640 1 -- 172.21.15.47:0/2795603260 wait complete. 2024-02-19T15:22:45.267 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:22:45.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.409+0000 7f6ea1200640 1 Processor -- start 2024-02-19T15:22:45.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.409+0000 7f6ea1200640 1 -- start start 2024-02-19T15:22:45.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.409+0000 7f6ea1200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6e9c0b1280 0x7f6e9c06aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:45.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.409+0000 7f6ea1200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e9c15e2b0 0x7f6e9c06b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:45.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.409+0000 7f6ea1200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9c0b04c0 con 0x7f6e9c15ded0 2024-02-19T15:22:45.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.409+0000 7f6ea1200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e9c0b07a0 con 0x7f6e9c0b1280 2024-02-19T15:22:45.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.409+0000 7f6ea1200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c0b0a80 con 0x7f6e9c15e2b0 2024-02-19T15:22:45.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.409+0000 7f6e99400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6e9c0b1280 0x7f6e9c06aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:45.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.409+0000 7f6e99400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6e9c0b1280 0x7f6e9c06aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36230/0 (socket says 172.21.15.47:36230) 2024-02-19T15:22:45.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.409+0000 7f6e99400640 1 -- 172.21.15.47:0/2777204835 learned_addr learned my addr 172.21.15.47:0/2777204835 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:45.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.409+0000 7f6e9a800640 1 --2- 172.21.15.47:0/2777204835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e9c15e2b0 0x7f6e9c06b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:45.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.409+0000 7f6e99400640 1 -- 172.21.15.47:0/2777204835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e9c15e2b0 msgr2=0x7f6e9c06b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:45.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.409+0000 7f6e99400640 1 --2- 172.21.15.47:0/2777204835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e9c15e2b0 0x7f6e9c06b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.409+0000 7f6e99400640 1 -- 172.21.15.47:0/2777204835 >> v1:172.21.15.116:6789/0 conn(0x7f6e9c15ded0 legacy=0x7f6e9c066cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:45.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.409+0000 7f6e99400640 1 -- 172.21.15.47:0/2777204835 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e9c0b0d60 con 0x7f6e9c0b1280 2024-02-19T15:22:45.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.409+0000 7f6e98a00640 1 -- 172.21.15.47:0/2777204835 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933268441 0 0) 0x7f6e9c0b04c0 con 0x7f6e9c15ded0 2024-02-19T15:22:45.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.413+0000 7f6e99400640 1 --2- 172.21.15.47:0/2777204835 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e9c0b1280 0x7f6e9c06aeb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6e88009870 tx=0x7f6e8801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e32f18bfc54ae106 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:45.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.413+0000 7f6e98a00640 1 -- 172.21.15.47:0/2777204835 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e880a9640 con 0x7f6e9c0b1280 2024-02-19T15:22:45.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.413+0000 7f6e98a00640 1 -- 172.21.15.47:0/2777204835 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e880a97a0 con 0x7f6e9c0b1280 2024-02-19T15:22:45.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.413+0000 7f6e98a00640 1 -- 172.21.15.47:0/2777204835 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e880a9b30 con 0x7f6e9c0b1280 2024-02-19T15:22:45.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.413+0000 7f6ea1200640 1 -- 172.21.15.47:0/2777204835 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e9c0b1280 msgr2=0x7f6e9c06aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:45.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.413+0000 7f6ea1200640 1 --2- 172.21.15.47:0/2777204835 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e9c0b1280 0x7f6e9c06aeb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6e88009870 tx=0x7f6e8801c560 comp rx=0 tx=0).stop 2024-02-19T15:22:45.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.413+0000 7f6ea1200640 1 -- 172.21.15.47:0/2777204835 shutdown_connections 2024-02-19T15:22:45.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.413+0000 7f6ea1200640 1 --2- 172.21.15.47:0/2777204835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e9c15e2b0 0x7f6e9c06b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.413+0000 7f6ea1200640 1 --2- 172.21.15.47:0/2777204835 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e9c0b1280 0x7f6e9c06aeb0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.413+0000 7f6ea1200640 1 -- 172.21.15.47:0/2777204835 >> 172.21.15.47:0/2777204835 conn(0x7f6e9c0b32f0 msgr2=0x7f6e9c0b36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:45.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.413+0000 7f6ea1200640 1 -- 172.21.15.47:0/2777204835 shutdown_connections 2024-02-19T15:22:45.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.413+0000 7f6ea1200640 1 -- 172.21.15.47:0/2777204835 wait complete. 2024-02-19T15:22:45.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.413+0000 7f6ea1200640 1 Processor -- start 2024-02-19T15:22:45.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.413+0000 7f6ea1200640 1 -- start start 2024-02-19T15:22:45.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6ea1200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e9c0b1280 0x7f6e9c154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:45.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6ea1200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e9c15ded0 0x7f6e9c0c4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:45.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6ea1200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e9c15e2b0 0x7f6e9c0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:45.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6ea1200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c154c80 con 0x7f6e9c15ded0 2024-02-19T15:22:45.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6ea1200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c154eb0 con 0x7f6e9c0b1280 2024-02-19T15:22:45.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6ea1200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c155190 con 0x7f6e9c15e2b0 2024-02-19T15:22:45.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6e99400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e9c15ded0 0x7f6e9c0c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:45.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6e99400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e9c15ded0 0x7f6e9c0c4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36240/0 (socket says 172.21.15.47:36240) 2024-02-19T15:22:45.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6e99400640 1 -- 172.21.15.47:0/3426795224 learned_addr learned my addr 172.21.15.47:0/3426795224 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:45.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6e9a800640 1 --2- 172.21.15.47:0/3426795224 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e9c15e2b0 0x7f6e9c0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:45.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6e99e00640 1 --2- 172.21.15.47:0/3426795224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e9c0b1280 0x7f6e9c154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:45.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6e99400640 1 -- 172.21.15.47:0/3426795224 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e9c15e2b0 msgr2=0x7f6e9c0c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:45.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6e99400640 1 --2- 172.21.15.47:0/3426795224 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e9c15e2b0 0x7f6e9c0c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6e99400640 1 -- 172.21.15.47:0/3426795224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e9c0b1280 msgr2=0x7f6e9c154740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:45.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6e99400640 1 --2- 172.21.15.47:0/3426795224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e9c0b1280 0x7f6e9c154740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6e99400640 1 -- 172.21.15.47:0/3426795224 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e880094d0 con 0x7f6e9c15ded0 2024-02-19T15:22:45.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6e99400640 1 --2- 172.21.15.47:0/3426795224 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e9c15ded0 0x7f6e9c0c4750 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6e880ab040 tx=0x7f6e8801ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:45.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6e92000640 1 -- 172.21.15.47:0/3426795224 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e880b1880 con 0x7f6e9c15ded0 2024-02-19T15:22:45.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6e92000640 1 -- 172.21.15.47:0/3426795224 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e880b1a00 con 0x7f6e9c15ded0 2024-02-19T15:22:45.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6e92000640 1 -- 172.21.15.47:0/3426795224 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e880b1db0 con 0x7f6e9c15ded0 2024-02-19T15:22:45.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6ea1200640 1 -- 172.21.15.47:0/3426795224 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e9c155470 con 0x7f6e9c15ded0 2024-02-19T15:22:45.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6ea1200640 1 -- 172.21.15.47:0/3426795224 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e9c155a30 con 0x7f6e9c15ded0 2024-02-19T15:22:45.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6ea1200640 1 -- 172.21.15.47:0/3426795224 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e9c155c20 con 0x7f6e9c15ded0 2024-02-19T15:22:45.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.417+0000 7f6e92000640 1 -- 172.21.15.47:0/3426795224 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6e880a7070 con 0x7f6e9c15ded0 2024-02-19T15:22:45.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.421+0000 7f6e92000640 1 --2- 172.21.15.47:0/3426795224 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6e7c066e40 0x7f6e7c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:45.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.421+0000 7f6e92000640 1 -- 172.21.15.47:0/3426795224 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e8811f250 con 0x7f6e9c15ded0 2024-02-19T15:22:45.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.421+0000 7f6e99e00640 1 --2- 172.21.15.47:0/3426795224 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6e7c066e40 0x7f6e7c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:45.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.421+0000 7f6e99e00640 1 --2- 172.21.15.47:0/3426795224 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6e7c066e40 0x7f6e7c069300 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c000f70 tx=0x7f6e8c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:45.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.425+0000 7f6e92000640 1 -- 172.21.15.47:0/3426795224 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6e880ec170 con 0x7f6e9c15ded0 2024-02-19T15:22:45.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.585+0000 7f6ea1200640 1 -- 172.21.15.47:0/3426795224 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6e9c0d09b0 con 0x7f6e9c15ded0 2024-02-19T15:22:45.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.585+0000 7f6e92000640 1 -- 172.21.15.47:0/3426795224 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6e880ef830 con 0x7f6e9c15ded0 2024-02-19T15:22:45.588 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:22:45.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.585+0000 7f6ea1200640 1 -- 172.21.15.47:0/3426795224 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6e7c066e40 msgr2=0x7f6e7c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:45.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.585+0000 7f6ea1200640 1 --2- 172.21.15.47:0/3426795224 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6e7c066e40 0x7f6e7c069300 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c000f70 tx=0x7f6e8c009040 comp rx=0 tx=0).stop 2024-02-19T15:22:45.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.585+0000 7f6ea1200640 1 -- 172.21.15.47:0/3426795224 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e9c15ded0 msgr2=0x7f6e9c0c4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:45.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.585+0000 7f6ea1200640 1 --2- 172.21.15.47:0/3426795224 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e9c15ded0 0x7f6e9c0c4750 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6e880ab040 tx=0x7f6e8801ada0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.589+0000 7f6ea1200640 1 -- 172.21.15.47:0/3426795224 shutdown_connections 2024-02-19T15:22:45.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.589+0000 7f6ea1200640 1 --2- 172.21.15.47:0/3426795224 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6e7c066e40 0x7f6e7c069300 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.589+0000 7f6ea1200640 1 --2- 172.21.15.47:0/3426795224 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e9c15e2b0 0x7f6e9c0c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.589+0000 7f6ea1200640 1 --2- 172.21.15.47:0/3426795224 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e9c15ded0 0x7f6e9c0c4750 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.589+0000 7f6ea1200640 1 --2- 172.21.15.47:0/3426795224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e9c0b1280 0x7f6e9c154740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.589+0000 7f6ea1200640 1 -- 172.21.15.47:0/3426795224 >> 172.21.15.47:0/3426795224 conn(0x7f6e9c0b32f0 msgr2=0x7f6e9c06d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:45.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.589+0000 7f6ea1200640 1 -- 172.21.15.47:0/3426795224 shutdown_connections 2024-02-19T15:22:45.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.589+0000 7f6ea1200640 1 -- 172.21.15.47:0/3426795224 wait complete. 2024-02-19T15:22:45.610 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:22:45.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.749+0000 7fea7b800640 1 Processor -- start 2024-02-19T15:22:45.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.749+0000 7fea7b800640 1 -- start start 2024-02-19T15:22:45.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.749+0000 7fea7b800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fea740d2650 0x7fea7406c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:45.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.749+0000 7fea7b800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fea7415e280 0x7fea7406c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:45.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.749+0000 7fea7b800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea740cc620 con 0x7fea740ce3b0 2024-02-19T15:22:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.749+0000 7fea7b800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea740d12e0 con 0x7fea7415e280 2024-02-19T15:22:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.749+0000 7fea7b800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea740d1940 con 0x7fea740d2650 2024-02-19T15:22:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.753+0000 7fea79800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fea7415e280 0x7fea7406c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.753+0000 7fea73e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fea740d2650 0x7fea7406c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.753+0000 7fea79800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fea7415e280 0x7fea7406c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36244/0 (socket says 172.21.15.47:36244) 2024-02-19T15:22:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.753+0000 7fea79800640 1 -- 172.21.15.47:0/2726187415 learned_addr learned my addr 172.21.15.47:0/2726187415 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.753+0000 7fea73e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fea740d2650 0x7fea7406c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:45568/0 (socket says 172.21.15.47:45568) 2024-02-19T15:22:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.753+0000 7fea78e00640 1 --1- 172.21.15.47:0/2726187415 >> v1:172.21.15.116:6789/0 conn(0x7fea740ce3b0 0x7fea740ce790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:43068/0 (socket says 172.21.15.47:43068) 2024-02-19T15:22:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.753+0000 7fea79800640 1 -- 172.21.15.47:0/2726187415 >> v2:172.21.15.116:3300/0 conn(0x7fea740d2650 msgr2=0x7fea7406c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.753+0000 7fea79800640 1 --2- 172.21.15.47:0/2726187415 >> v2:172.21.15.116:3300/0 conn(0x7fea740d2650 0x7fea7406c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.753+0000 7fea79800640 1 -- 172.21.15.47:0/2726187415 >> v1:172.21.15.116:6789/0 conn(0x7fea740ce3b0 legacy=0x7fea740ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.753+0000 7fea79800640 1 -- 172.21.15.47:0/2726187415 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea740d1000 con 0x7fea7415e280 2024-02-19T15:22:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.753+0000 7fea73400640 1 -- 172.21.15.47:0/2726187415 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487556641 0 0) 0x7fea740cc620 con 0x7fea740ce3b0 2024-02-19T15:22:45.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.753+0000 7fea79800640 1 --2- 172.21.15.47:0/2726187415 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea7415e280 0x7fea7406c710 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fea74071dd0 tx=0x7fea6c01c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77efdfa2eac8d163 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:45.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.753+0000 7fea73400640 1 -- 172.21.15.47:0/2726187415 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6c0a56a0 con 0x7fea7415e280 2024-02-19T15:22:45.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.753+0000 7fea73400640 1 -- 172.21.15.47:0/2726187415 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6c0a5800 con 0x7fea7415e280 2024-02-19T15:22:45.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.753+0000 7fea73400640 1 -- 172.21.15.47:0/2726187415 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6c0a5b90 con 0x7fea7415e280 2024-02-19T15:22:45.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.753+0000 7fea7b800640 1 -- 172.21.15.47:0/2726187415 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea7415e280 msgr2=0x7fea7406c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:45.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.753+0000 7fea7b800640 1 --2- 172.21.15.47:0/2726187415 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea7415e280 0x7fea7406c710 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fea74071dd0 tx=0x7fea6c01c8d0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea7b800640 1 -- 172.21.15.47:0/2726187415 shutdown_connections 2024-02-19T15:22:45.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea7b800640 1 --2- 172.21.15.47:0/2726187415 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea7415e280 0x7fea7406c710 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea7b800640 1 --2- 172.21.15.47:0/2726187415 >> v2:172.21.15.116:3300/0 conn(0x7fea740d2650 0x7fea7406c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea7b800640 1 -- 172.21.15.47:0/2726187415 >> 172.21.15.47:0/2726187415 conn(0x7fea740d1b30 msgr2=0x7fea740d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:45.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea7b800640 1 -- 172.21.15.47:0/2726187415 shutdown_connections 2024-02-19T15:22:45.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea7b800640 1 -- 172.21.15.47:0/2726187415 wait complete. 2024-02-19T15:22:45.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea7b800640 1 Processor -- start 2024-02-19T15:22:45.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea7b800640 1 -- start start 2024-02-19T15:22:45.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea7b800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea740ce3b0 0x7fea74152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:45.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea7b800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea740d2650 0x7fea74155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:45.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea7b800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea7415e280 0x7fea74156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:45.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea7b800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea74153fd0 con 0x7fea7415e280 2024-02-19T15:22:45.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea7b800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea741542b0 con 0x7fea740ce3b0 2024-02-19T15:22:45.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea7b800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea74154590 con 0x7fea740d2650 2024-02-19T15:22:45.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea79800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea7415e280 0x7fea74156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:45.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea79800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea7415e280 0x7fea74156320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36246/0 (socket says 172.21.15.47:36246) 2024-02-19T15:22:45.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea79800640 1 -- 172.21.15.47:0/3262170559 learned_addr learned my addr 172.21.15.47:0/3262170559 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:45.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea73e00640 1 --2- 172.21.15.47:0/3262170559 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea740d2650 0x7fea74155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:45.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea78e00640 1 --2- 172.21.15.47:0/3262170559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea740ce3b0 0x7fea74152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:45.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea79800640 1 -- 172.21.15.47:0/3262170559 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea740d2650 msgr2=0x7fea74155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:45.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea79800640 1 --2- 172.21.15.47:0/3262170559 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea740d2650 0x7fea74155de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea79800640 1 -- 172.21.15.47:0/3262170559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea740ce3b0 msgr2=0x7fea74152cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:45.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea79800640 1 --2- 172.21.15.47:0/3262170559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea740ce3b0 0x7fea74152cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea79800640 1 -- 172.21.15.47:0/3262170559 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6c009b30 con 0x7fea7415e280 2024-02-19T15:22:45.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.757+0000 7fea79800640 1 --2- 172.21.15.47:0/3262170559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea7415e280 0x7fea74156320 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fea6c0194e0 tx=0x7fea6c019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:45.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.761+0000 7fea70c00640 1 -- 172.21.15.47:0/3262170559 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6c0b8670 con 0x7fea7415e280 2024-02-19T15:22:45.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.761+0000 7fea70c00640 1 -- 172.21.15.47:0/3262170559 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6c0b87f0 con 0x7fea7415e280 2024-02-19T15:22:45.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.761+0000 7fea70c00640 1 -- 172.21.15.47:0/3262170559 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6c0b8ba0 con 0x7fea7415e280 2024-02-19T15:22:45.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.761+0000 7fea7b800640 1 -- 172.21.15.47:0/3262170559 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea740c8ca0 con 0x7fea7415e280 2024-02-19T15:22:45.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.761+0000 7fea7b800640 1 -- 172.21.15.47:0/3262170559 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea7415f200 con 0x7fea7415e280 2024-02-19T15:22:45.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.761+0000 7fea7b800640 1 -- 172.21.15.47:0/3262170559 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea7415f3f0 con 0x7fea7415e280 2024-02-19T15:22:45.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.761+0000 7fea70c00640 1 -- 172.21.15.47:0/3262170559 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fea740cdcd0 con 0x7fea7415e280 2024-02-19T15:22:45.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.761+0000 7fea70c00640 1 --2- 172.21.15.47:0/3262170559 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fea54066e70 0x7fea54069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:45.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.761+0000 7fea70c00640 1 -- 172.21.15.47:0/3262170559 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea6c11f7c0 con 0x7fea7415e280 2024-02-19T15:22:45.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.761+0000 7fea78e00640 1 --2- 172.21.15.47:0/3262170559 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fea54066e70 0x7fea54069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:45.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.765+0000 7fea78e00640 1 --2- 172.21.15.47:0/3262170559 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fea54066e70 0x7fea54069330 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fea68000f70 tx=0x7fea68007df0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:45.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.765+0000 7fea70c00640 1 -- 172.21.15.47:0/3262170559 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fea6c0ec650 con 0x7fea7415e280 2024-02-19T15:22:45.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.933+0000 7fea7b800640 1 -- 172.21.15.47:0/3262170559 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fea74075360 con 0x7fea7415e280 2024-02-19T15:22:45.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.933+0000 7fea70c00640 1 -- 172.21.15.47:0/3262170559 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fea6c009510 con 0x7fea7415e280 2024-02-19T15:22:45.938 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:22:45.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.937+0000 7fea7b800640 1 -- 172.21.15.47:0/3262170559 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fea54066e70 msgr2=0x7fea54069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:45.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.937+0000 7fea7b800640 1 --2- 172.21.15.47:0/3262170559 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fea54066e70 0x7fea54069330 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fea68000f70 tx=0x7fea68007df0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.937+0000 7fea7b800640 1 -- 172.21.15.47:0/3262170559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea7415e280 msgr2=0x7fea74156320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:45.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.937+0000 7fea7b800640 1 --2- 172.21.15.47:0/3262170559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea7415e280 0x7fea74156320 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fea6c0194e0 tx=0x7fea6c019670 comp rx=0 tx=0).stop 2024-02-19T15:22:45.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.937+0000 7fea7b800640 1 -- 172.21.15.47:0/3262170559 shutdown_connections 2024-02-19T15:22:45.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.937+0000 7fea7b800640 1 --2- 172.21.15.47:0/3262170559 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fea54066e70 0x7fea54069330 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.937+0000 7fea7b800640 1 --2- 172.21.15.47:0/3262170559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fea7415e280 0x7fea74156320 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.937+0000 7fea7b800640 1 --2- 172.21.15.47:0/3262170559 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea740d2650 0x7fea74155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.937+0000 7fea7b800640 1 --2- 172.21.15.47:0/3262170559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea740ce3b0 0x7fea74152cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:45.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.937+0000 7fea7b800640 1 -- 172.21.15.47:0/3262170559 >> 172.21.15.47:0/3262170559 conn(0x7fea740d1b30 msgr2=0x7fea740c6270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:45.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.937+0000 7fea7b800640 1 -- 172.21.15.47:0/3262170559 shutdown_connections 2024-02-19T15:22:45.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:45.937+0000 7fea7b800640 1 -- 172.21.15.47:0/3262170559 wait complete. 2024-02-19T15:22:45.961 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:22:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.101+0000 7f61e9600640 1 Processor -- start 2024-02-19T15:22:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.101+0000 7f61e9600640 1 -- start start 2024-02-19T15:22:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.101+0000 7f61e9600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f61e415a390 0x7f61e40b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.101+0000 7f61e9600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e40b3360 con 0x7f61e406d940 2024-02-19T15:22:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.101+0000 7f61e9600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e40b3600 con 0x7f61e40cf180 2024-02-19T15:22:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.101+0000 7f61e9600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e40b38e0 con 0x7f61e415a390 2024-02-19T15:22:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.101+0000 7f61e2a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f61e415a390 0x7f61e40b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.101+0000 7f61e2a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f61e415a390 0x7f61e40b46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36254/0 (socket says 172.21.15.47:36254) 2024-02-19T15:22:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.101+0000 7f61e3400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f61e406d940 0x7f61e40cc540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57162/0 (socket says 172.21.15.47:57162) 2024-02-19T15:22:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.101+0000 7f61e2a00640 1 -- 172.21.15.47:0/2761202542 learned_addr learned my addr 172.21.15.47:0/2761202542 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.105+0000 7f61e2a00640 1 -- 172.21.15.47:0/2761202542 >> v1:172.21.15.116:6789/0 conn(0x7f61e40cf180 legacy=0x7f61e406d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.105+0000 7f61e2a00640 1 -- 172.21.15.47:0/2761202542 >> v1:172.21.15.47:6789/0 conn(0x7f61e406d940 legacy=0x7f61e40cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.105+0000 7f61e2a00640 1 -- 172.21.15.47:0/2761202542 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e40b3bc0 con 0x7f61e415a390 2024-02-19T15:22:46.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.105+0000 7f61e2a00640 1 --2- 172.21.15.47:0/2761202542 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e415a390 0x7f61e40b46a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f61d400aa10 tx=0x7f61d4020120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b18322c36c33b8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:46.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.105+0000 7f61e2000640 1 -- 172.21.15.47:0/2761202542 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d40b0600 con 0x7f61e415a390 2024-02-19T15:22:46.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.105+0000 7f61e2000640 1 -- 172.21.15.47:0/2761202542 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d40ac050 con 0x7f61e415a390 2024-02-19T15:22:46.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.105+0000 7f61e2000640 1 -- 172.21.15.47:0/2761202542 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d40b0980 con 0x7f61e415a390 2024-02-19T15:22:46.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.105+0000 7f61e9600640 1 -- 172.21.15.47:0/2761202542 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e415a390 msgr2=0x7f61e40b46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:46.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.105+0000 7f61e9600640 1 --2- 172.21.15.47:0/2761202542 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e415a390 0x7f61e40b46a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f61d400aa10 tx=0x7f61d4020120 comp rx=0 tx=0).stop 2024-02-19T15:22:46.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.105+0000 7f61e9600640 1 -- 172.21.15.47:0/2761202542 shutdown_connections 2024-02-19T15:22:46.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.105+0000 7f61e9600640 1 --2- 172.21.15.47:0/2761202542 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e415a390 0x7f61e40b46a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.105+0000 7f61e9600640 1 -- 172.21.15.47:0/2761202542 >> 172.21.15.47:0/2761202542 conn(0x7f61e40b16a0 msgr2=0x7f61e40b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:46.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.105+0000 7f61e9600640 1 -- 172.21.15.47:0/2761202542 shutdown_connections 2024-02-19T15:22:46.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e9600640 1 -- 172.21.15.47:0/2761202542 wait complete. 2024-02-19T15:22:46.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e9600640 1 Processor -- start 2024-02-19T15:22:46.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e9600640 1 -- start start 2024-02-19T15:22:46.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e9600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e406d940 0x7f61e414e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:46.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e9600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61e40cf180 0x7f61e415d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:46.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e9600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61e415a390 0x7f61e41f75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:46.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e9600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e415d560 con 0x7f61e406d940 2024-02-19T15:22:46.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e9600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e41e4270 con 0x7f61e40cf180 2024-02-19T15:22:46.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e9600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e41e4550 con 0x7f61e415a390 2024-02-19T15:22:46.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e2a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e406d940 0x7f61e414e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:46.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e2a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e406d940 0x7f61e414e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36258/0 (socket says 172.21.15.47:36258) 2024-02-19T15:22:46.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e2a00640 1 -- 172.21.15.47:0/1371162815 learned_addr learned my addr 172.21.15.47:0/1371162815 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61da000640 1 --2- 172.21.15.47:0/1371162815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61e40cf180 0x7f61e415d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e3400640 1 --2- 172.21.15.47:0/1371162815 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61e415a390 0x7f61e41f75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e2a00640 1 -- 172.21.15.47:0/1371162815 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61e415a390 msgr2=0x7f61e41f75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e2a00640 1 --2- 172.21.15.47:0/1371162815 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61e415a390 0x7f61e41f75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e2a00640 1 -- 172.21.15.47:0/1371162815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61e40cf180 msgr2=0x7f61e415d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e2a00640 1 --2- 172.21.15.47:0/1371162815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61e40cf180 0x7f61e415d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e2a00640 1 -- 172.21.15.47:0/1371162815 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d400a670 con 0x7f61e406d940 2024-02-19T15:22:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e2a00640 1 --2- 172.21.15.47:0/1371162815 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e406d940 0x7f61e414e5b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f61d400a1c0 tx=0x7f61d401e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61db400640 1 -- 172.21.15.47:0/1371162815 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d40bf070 con 0x7f61e406d940 2024-02-19T15:22:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61db400640 1 -- 172.21.15.47:0/1371162815 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d40b96f0 con 0x7f61e406d940 2024-02-19T15:22:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e9600640 1 -- 172.21.15.47:0/1371162815 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e41e4830 con 0x7f61e406d940 2024-02-19T15:22:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61db400640 1 -- 172.21.15.47:0/1371162815 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d40b9ac0 con 0x7f61e406d940 2024-02-19T15:22:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.109+0000 7f61e9600640 1 -- 172.21.15.47:0/1371162815 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e41e4df0 con 0x7f61e406d940 2024-02-19T15:22:46.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.113+0000 7f61e9600640 1 -- 172.21.15.47:0/1371162815 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e4075360 con 0x7f61e406d940 2024-02-19T15:22:46.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.113+0000 7f61db400640 1 -- 172.21.15.47:0/1371162815 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f61d40b7070 con 0x7f61e406d940 2024-02-19T15:22:46.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.113+0000 7f61db400640 1 --2- 172.21.15.47:0/1371162815 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f61c0066e40 0x7f61c0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:46.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.113+0000 7f61db400640 1 -- 172.21.15.47:0/1371162815 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f61d4122890 con 0x7f61e406d940 2024-02-19T15:22:46.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.113+0000 7f61da000640 1 --2- 172.21.15.47:0/1371162815 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f61c0066e40 0x7f61c0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:46.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.113+0000 7f61da000640 1 --2- 172.21.15.47:0/1371162815 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f61c0066e40 0x7f61c0069300 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f61e41e4030 tx=0x7f61cc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:46.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.117+0000 7f61db400640 1 -- 172.21.15.47:0/1371162815 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f61d40ef690 con 0x7f61e406d940 2024-02-19T15:22:46.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.313+0000 7f61e9600640 1 -- 172.21.15.47:0/1371162815 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61e40b1480 con 0x7f61e406d940 2024-02-19T15:22:46.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.313+0000 7f61db400640 1 -- 172.21.15.47:0/1371162815 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4935 (secure 0 0 0) 0x7f61d40b5090 con 0x7f61e406d940 2024-02-19T15:22:46.317 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:46.317 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3873760209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3873760209},{"type":"v1","addr":"172.21.15.116:6836","nonce":3873760209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/284820350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":284820350},{"type":"v1","addr":"172.21.15.47:6834","nonce":284820350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2764089359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2764089359},{"type":"v1","addr":"172.21.15.47:6835","nonce":2764089359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:22:05.228517+0000","modified":"2024-02-19T15:22:08.242600+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6837/3920452343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3920452343},{"type":"v1","addr":"172.21.15.116:6837","nonce":3920452343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-19T15:22:46.317 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 8 2024-02-19T15:22:46.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.317+0000 7f61e9600640 1 -- 172.21.15.47:0/1371162815 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f61c0066e40 msgr2=0x7f61c0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:46.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.317+0000 7f61e9600640 1 --2- 172.21.15.47:0/1371162815 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f61c0066e40 0x7f61c0069300 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f61e41e4030 tx=0x7f61cc009040 comp rx=0 tx=0).stop 2024-02-19T15:22:46.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.317+0000 7f61e9600640 1 -- 172.21.15.47:0/1371162815 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e406d940 msgr2=0x7f61e414e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:46.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.317+0000 7f61e9600640 1 --2- 172.21.15.47:0/1371162815 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e406d940 0x7f61e414e5b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f61d400a1c0 tx=0x7f61d401e960 comp rx=0 tx=0).stop 2024-02-19T15:22:46.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.317+0000 7f61e9600640 1 -- 172.21.15.47:0/1371162815 shutdown_connections 2024-02-19T15:22:46.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.317+0000 7f61e9600640 1 --2- 172.21.15.47:0/1371162815 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f61c0066e40 0x7f61c0069300 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.317+0000 7f61e9600640 1 --2- 172.21.15.47:0/1371162815 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61e415a390 0x7f61e41f75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.317+0000 7f61e9600640 1 --2- 172.21.15.47:0/1371162815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61e40cf180 0x7f61e415d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.317+0000 7f61e9600640 1 --2- 172.21.15.47:0/1371162815 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61e406d940 0x7f61e414e5b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.317+0000 7f61e9600640 1 -- 172.21.15.47:0/1371162815 >> 172.21.15.47:0/1371162815 conn(0x7f61e40b16a0 msgr2=0x7f61e40cf6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:46.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.317+0000 7f61e9600640 1 -- 172.21.15.47:0/1371162815 shutdown_connections 2024-02-19T15:22:46.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.317+0000 7f61e9600640 1 -- 172.21.15.47:0/1371162815 wait complete. 2024-02-19T15:22:46.338 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:46.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.477+0000 7f8c1cc00640 1 Processor -- start 2024-02-19T15:22:46.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.477+0000 7f8c1cc00640 1 -- start start 2024-02-19T15:22:46.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.477+0000 7f8c1cc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c1815bcd0 0x7f8c180cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:46.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.477+0000 7f8c1cc00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8c180cedb0 0x7f8c180cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:46.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.477+0000 7f8c1cc00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c180cc700 con 0x7f8c180d2710 2024-02-19T15:22:46.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.477+0000 7f8c1cc00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c180cf5e0 con 0x7f8c180cedb0 2024-02-19T15:22:46.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.477+0000 7f8c1cc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c180cf300 con 0x7f8c1815bcd0 2024-02-19T15:22:46.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.477+0000 7f8c0d400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8c180cedb0 0x7f8c180cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:46.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.477+0000 7f8c0d400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8c180cedb0 0x7f8c180cfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36260/0 (socket says 172.21.15.47:36260) 2024-02-19T15:22:46.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.477+0000 7f8c0d400640 1 -- 172.21.15.47:0/3540163911 learned_addr learned my addr 172.21.15.47:0/3540163911 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:46.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.477+0000 7f8c15e00640 1 --2- 172.21.15.47:0/3540163911 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c1815bcd0 0x7f8c180cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:46.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.477+0000 7f8c0d400640 1 -- 172.21.15.47:0/3540163911 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c1815bcd0 msgr2=0x7f8c180cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:46.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.477+0000 7f8c0d400640 1 --2- 172.21.15.47:0/3540163911 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c1815bcd0 0x7f8c180cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.477+0000 7f8c0d400640 1 -- 172.21.15.47:0/3540163911 >> v1:172.21.15.116:6789/0 conn(0x7f8c180d2710 legacy=0x7f8c1815d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:46.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.477+0000 7f8c0d400640 1 -- 172.21.15.47:0/3540163911 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c180cc9a0 con 0x7f8c180cedb0 2024-02-19T15:22:46.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.477+0000 7f8c15400640 1 -- 172.21.15.47:0/3540163911 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911977563 0 0) 0x7f8c180cc700 con 0x7f8c180d2710 2024-02-19T15:22:46.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.481+0000 7f8c0d400640 1 --2- 172.21.15.47:0/3540163911 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c180cedb0 0x7f8c180cfc50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8c00009870 tx=0x7f8c0001c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae345f7b2829c85 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:46.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.481+0000 7f8c15400640 1 -- 172.21.15.47:0/3540163911 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c000ab640 con 0x7f8c180cedb0 2024-02-19T15:22:46.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.481+0000 7f8c15400640 1 -- 172.21.15.47:0/3540163911 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c000a7070 con 0x7f8c180cedb0 2024-02-19T15:22:46.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.481+0000 7f8c15400640 1 -- 172.21.15.47:0/3540163911 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c000ab9c0 con 0x7f8c180cedb0 2024-02-19T15:22:46.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.481+0000 7f8c1cc00640 1 -- 172.21.15.47:0/3540163911 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c180cedb0 msgr2=0x7f8c180cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:46.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.481+0000 7f8c1cc00640 1 --2- 172.21.15.47:0/3540163911 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c180cedb0 0x7f8c180cfc50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8c00009870 tx=0x7f8c0001c530 comp rx=0 tx=0).stop 2024-02-19T15:22:46.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.481+0000 7f8c1cc00640 1 -- 172.21.15.47:0/3540163911 shutdown_connections 2024-02-19T15:22:46.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.481+0000 7f8c1cc00640 1 --2- 172.21.15.47:0/3540163911 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c180cedb0 0x7f8c180cfc50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.481+0000 7f8c1cc00640 1 --2- 172.21.15.47:0/3540163911 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c1815bcd0 0x7f8c180cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.481+0000 7f8c1cc00640 1 -- 172.21.15.47:0/3540163911 >> 172.21.15.47:0/3540163911 conn(0x7f8c180d1bf0 msgr2=0x7f8c180d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:46.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.481+0000 7f8c1cc00640 1 -- 172.21.15.47:0/3540163911 shutdown_connections 2024-02-19T15:22:46.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.481+0000 7f8c1cc00640 1 -- 172.21.15.47:0/3540163911 wait complete. 2024-02-19T15:22:46.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.481+0000 7f8c1cc00640 1 Processor -- start 2024-02-19T15:22:46.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.481+0000 7f8c1cc00640 1 -- start start 2024-02-19T15:22:46.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c1cc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c180cedb0 0x7f8c1814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:46.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c1cc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c180d2710 0x7f8c1814f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:46.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c1cc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c1815bcd0 0x7f8c180b5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:46.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c1cc00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c180c74c0 con 0x7f8c1815bcd0 2024-02-19T15:22:46.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c1cc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c180cb430 con 0x7f8c180d2710 2024-02-19T15:22:46.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c1cc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c180ca160 con 0x7f8c180cedb0 2024-02-19T15:22:46.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c16800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c1815bcd0 0x7f8c180b5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:46.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c16800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c1815bcd0 0x7f8c180b5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36264/0 (socket says 172.21.15.47:36264) 2024-02-19T15:22:46.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c16800640 1 -- 172.21.15.47:0/558998803 learned_addr learned my addr 172.21.15.47:0/558998803 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:46.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c15e00640 1 --2- 172.21.15.47:0/558998803 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c180cedb0 0x7f8c1814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:46.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c0d400640 1 --2- 172.21.15.47:0/558998803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c180d2710 0x7f8c1814f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:46.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c16800640 1 -- 172.21.15.47:0/558998803 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c180cedb0 msgr2=0x7f8c1814fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:46.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c16800640 1 --2- 172.21.15.47:0/558998803 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c180cedb0 0x7f8c1814fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c16800640 1 -- 172.21.15.47:0/558998803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c180d2710 msgr2=0x7f8c1814f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:46.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c16800640 1 --2- 172.21.15.47:0/558998803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c180d2710 0x7f8c1814f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c16800640 1 -- 172.21.15.47:0/558998803 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c000094d0 con 0x7f8c1815bcd0 2024-02-19T15:22:46.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c15e00640 1 --2- 172.21.15.47:0/558998803 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c180cedb0 0x7f8c1814fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:46.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c16800640 1 --2- 172.21.15.47:0/558998803 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c1815bcd0 0x7f8c180b5820 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8c1000c780 tx=0x7f8c1000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:46.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c0ea00640 1 -- 172.21.15.47:0/558998803 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c10003db0 con 0x7f8c1815bcd0 2024-02-19T15:22:46.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c0ea00640 1 -- 172.21.15.47:0/558998803 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1000ce30 con 0x7f8c1815bcd0 2024-02-19T15:22:46.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c1cc00640 1 -- 172.21.15.47:0/558998803 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c180cdb40 con 0x7f8c1815bcd0 2024-02-19T15:22:46.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c0ea00640 1 -- 172.21.15.47:0/558998803 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c100052b0 con 0x7f8c1815bcd0 2024-02-19T15:22:46.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c1cc00640 1 -- 172.21.15.47:0/558998803 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c180cb9d0 con 0x7f8c1815bcd0 2024-02-19T15:22:46.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.485+0000 7f8c1cc00640 1 -- 172.21.15.47:0/558998803 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c18075360 con 0x7f8c1815bcd0 2024-02-19T15:22:46.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.489+0000 7f8c0ea00640 1 -- 172.21.15.47:0/558998803 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8c10005410 con 0x7f8c1815bcd0 2024-02-19T15:22:46.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.489+0000 7f8c0ea00640 1 --2- 172.21.15.47:0/558998803 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8bf4066e70 0x7f8bf4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:46.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.489+0000 7f8c0ea00640 1 -- 172.21.15.47:0/558998803 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f8c10084820 con 0x7f8c1815bcd0 2024-02-19T15:22:46.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.489+0000 7f8c15e00640 1 --2- 172.21.15.47:0/558998803 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8bf4066e70 0x7f8bf4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:46.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.489+0000 7f8c15e00640 1 --2- 172.21.15.47:0/558998803 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8bf4066e70 0x7f8bf4069330 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8c08009f50 tx=0x7f8c08009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:46.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.493+0000 7f8c0ea00640 1 -- 172.21.15.47:0/558998803 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8c10051510 con 0x7f8c1815bcd0 2024-02-19T15:22:46.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.657+0000 7f8c1cc00640 1 -- 172.21.15.47:0/558998803 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c180cb210 con 0x7f8c1815bcd0 2024-02-19T15:22:46.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.657+0000 7f8c0ea00640 1 -- 172.21.15.47:0/558998803 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16914 (secure 0 0 0) 0x7f8c10054bd0 con 0x7f8c1815bcd0 2024-02-19T15:22:46.660 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:46.660 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":22,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:45.982769+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:03.091360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:04.222965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:46.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.657+0000 7f8c1cc00640 1 -- 172.21.15.47:0/558998803 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8bf4066e70 msgr2=0x7f8bf4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:46.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.657+0000 7f8c1cc00640 1 --2- 172.21.15.47:0/558998803 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8bf4066e70 0x7f8bf4069330 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8c08009f50 tx=0x7f8c08009190 comp rx=0 tx=0).stop 2024-02-19T15:22:46.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.657+0000 7f8c1cc00640 1 -- 172.21.15.47:0/558998803 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c1815bcd0 msgr2=0x7f8c180b5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:46.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.657+0000 7f8c1cc00640 1 --2- 172.21.15.47:0/558998803 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c1815bcd0 0x7f8c180b5820 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8c1000c780 tx=0x7f8c1000cc40 comp rx=0 tx=0).stop 2024-02-19T15:22:46.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.657+0000 7f8c1cc00640 1 -- 172.21.15.47:0/558998803 shutdown_connections 2024-02-19T15:22:46.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.657+0000 7f8c1cc00640 1 --2- 172.21.15.47:0/558998803 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8bf4066e70 0x7f8bf4069330 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.657+0000 7f8c1cc00640 1 --2- 172.21.15.47:0/558998803 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c1815bcd0 0x7f8c180b5820 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.657+0000 7f8c1cc00640 1 --2- 172.21.15.47:0/558998803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c180d2710 0x7f8c1814f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.657+0000 7f8c1cc00640 1 --2- 172.21.15.47:0/558998803 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c180cedb0 0x7f8c1814fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.657+0000 7f8c1cc00640 1 -- 172.21.15.47:0/558998803 >> 172.21.15.47:0/558998803 conn(0x7f8c180d1bf0 msgr2=0x7f8c1806cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:46.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.657+0000 7f8c1cc00640 1 -- 172.21.15.47:0/558998803 shutdown_connections 2024-02-19T15:22:46.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.657+0000 7f8c1cc00640 1 -- 172.21.15.47:0/558998803 wait complete. 2024-02-19T15:22:46.681 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T15:22:46.681 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:22:46.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.817+0000 7f7e55200640 1 Processor -- start 2024-02-19T15:22:46.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.817+0000 7f7e55200640 1 -- start start 2024-02-19T15:22:46.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.821+0000 7f7e55200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7e5015b1a0 0x7f7e5015b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:46.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.821+0000 7f7e55200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7e5006c9a0 0x7f7e5015bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:46.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.821+0000 7f7e55200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e5015c000 con 0x7f7e50071350 2024-02-19T15:22:46.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.821+0000 7f7e55200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e500718c0 con 0x7f7e5015b1a0 2024-02-19T15:22:46.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.821+0000 7f7e55200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e50071ba0 con 0x7f7e5006c9a0 2024-02-19T15:22:46.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.821+0000 7f7e4de00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7e5015b1a0 0x7f7e5015b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:46.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.821+0000 7f7e4de00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7e5015b1a0 0x7f7e5015b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36278/0 (socket says 172.21.15.47:36278) 2024-02-19T15:22:46.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.821+0000 7f7e4de00640 1 -- 172.21.15.47:0/2644235674 learned_addr learned my addr 172.21.15.47:0/2644235674 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:46.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.821+0000 7f7e4e800640 1 --2- 172.21.15.47:0/2644235674 >> v2:172.21.15.116:3300/0 conn(0x7f7e5006c9a0 0x7f7e5015bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:46.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.821+0000 7f7e4de00640 1 -- 172.21.15.47:0/2644235674 >> v2:172.21.15.116:3300/0 conn(0x7f7e5006c9a0 msgr2=0x7f7e5015bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:46.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.821+0000 7f7e4de00640 1 --2- 172.21.15.47:0/2644235674 >> v2:172.21.15.116:3300/0 conn(0x7f7e5006c9a0 0x7f7e5015bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.821+0000 7f7e4de00640 1 -- 172.21.15.47:0/2644235674 >> v1:172.21.15.116:6789/0 conn(0x7f7e50071350 legacy=0x7f7e5015c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:46.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.821+0000 7f7e4de00640 1 -- 172.21.15.47:0/2644235674 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e50071e80 con 0x7f7e5015b1a0 2024-02-19T15:22:46.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.821+0000 7f7e4de00640 1 --2- 172.21.15.47:0/2644235674 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e5015b1a0 0x7f7e5015b580 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7e40009e50 tx=0x7f7e4001c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38f9c1a44649e4cb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:46.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e4d400640 1 -- 172.21.15.47:0/2644235674 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e400a58a0 con 0x7f7e5015b1a0 2024-02-19T15:22:46.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e4d400640 1 -- 172.21.15.47:0/2644235674 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e400a5a00 con 0x7f7e5015b1a0 2024-02-19T15:22:46.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e4d400640 1 -- 172.21.15.47:0/2644235674 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e400a5d90 con 0x7f7e5015b1a0 2024-02-19T15:22:46.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e55200640 1 -- 172.21.15.47:0/2644235674 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e5015b1a0 msgr2=0x7f7e5015b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:46.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e55200640 1 --2- 172.21.15.47:0/2644235674 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e5015b1a0 0x7f7e5015b580 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7e40009e50 tx=0x7f7e4001c9c0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e55200640 1 -- 172.21.15.47:0/2644235674 shutdown_connections 2024-02-19T15:22:46.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e55200640 1 --2- 172.21.15.47:0/2644235674 >> v2:172.21.15.116:3300/0 conn(0x7f7e5006c9a0 0x7f7e5015bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e55200640 1 --2- 172.21.15.47:0/2644235674 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e5015b1a0 0x7f7e5015b580 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e55200640 1 -- 172.21.15.47:0/2644235674 >> 172.21.15.47:0/2644235674 conn(0x7f7e500b16a0 msgr2=0x7f7e500b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:46.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e55200640 1 -- 172.21.15.47:0/2644235674 shutdown_connections 2024-02-19T15:22:46.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e55200640 1 -- 172.21.15.47:0/2644235674 wait complete. 2024-02-19T15:22:46.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e55200640 1 Processor -- start 2024-02-19T15:22:46.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e55200640 1 -- start start 2024-02-19T15:22:46.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e55200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e5006c9a0 0x7f7e5014f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:46.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e55200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e50071350 0x7f7e50164fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:46.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e55200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e5015b1a0 0x7f7e50149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:46.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e55200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5014faa0 con 0x7f7e50071350 2024-02-19T15:22:46.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e55200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e50165510 con 0x7f7e5015b1a0 2024-02-19T15:22:46.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e55200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5014fcb0 con 0x7f7e5006c9a0 2024-02-19T15:22:46.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e45400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e50071350 0x7f7e50164fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:46.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.825+0000 7f7e45400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e50071350 0x7f7e50164fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36292/0 (socket says 172.21.15.47:36292) 2024-02-19T15:22:46.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.829+0000 7f7e45400640 1 -- 172.21.15.47:0/312576730 learned_addr learned my addr 172.21.15.47:0/312576730 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:46.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.829+0000 7f7e4de00640 1 --2- 172.21.15.47:0/312576730 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e5006c9a0 0x7f7e5014f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:46.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.829+0000 7f7e4e800640 1 --2- 172.21.15.47:0/312576730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e5015b1a0 0x7f7e50149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:46.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.829+0000 7f7e45400640 1 -- 172.21.15.47:0/312576730 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e5006c9a0 msgr2=0x7f7e5014f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:46.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.829+0000 7f7e45400640 1 --2- 172.21.15.47:0/312576730 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e5006c9a0 0x7f7e5014f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.829+0000 7f7e45400640 1 -- 172.21.15.47:0/312576730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e5015b1a0 msgr2=0x7f7e50149940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:46.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.829+0000 7f7e45400640 1 --2- 172.21.15.47:0/312576730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e5015b1a0 0x7f7e50149940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:46.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.829+0000 7f7e45400640 1 -- 172.21.15.47:0/312576730 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e40009b30 con 0x7f7e50071350 2024-02-19T15:22:46.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.829+0000 7f7e4e800640 1 --2- 172.21.15.47:0/312576730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e5015b1a0 0x7f7e50149940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:46.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.829+0000 7f7e45400640 1 --2- 172.21.15.47:0/312576730 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e50071350 0x7f7e50164fd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7e3800c420 tx=0x7f7e3800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:46.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.829+0000 7f7e46a00640 1 -- 172.21.15.47:0/312576730 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e380052d0 con 0x7f7e50071350 2024-02-19T15:22:46.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.829+0000 7f7e46a00640 1 -- 172.21.15.47:0/312576730 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e38005430 con 0x7f7e50071350 2024-02-19T15:22:46.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.829+0000 7f7e55200640 1 -- 172.21.15.47:0/312576730 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e5014ff90 con 0x7f7e50071350 2024-02-19T15:22:46.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.829+0000 7f7e46a00640 1 -- 172.21.15.47:0/312576730 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e38007260 con 0x7f7e50071350 2024-02-19T15:22:46.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.829+0000 7f7e55200640 1 -- 172.21.15.47:0/312576730 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e50167810 con 0x7f7e50071350 2024-02-19T15:22:46.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.829+0000 7f7e55200640 1 -- 172.21.15.47:0/312576730 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e5006d1b0 con 0x7f7e50071350 2024-02-19T15:22:46.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.829+0000 7f7e46a00640 1 -- 172.21.15.47:0/312576730 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7e380073c0 con 0x7f7e50071350 2024-02-19T15:22:46.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.829+0000 7f7e46a00640 1 --2- 172.21.15.47:0/312576730 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7e2c066e40 0x7f7e2c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:46.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.829+0000 7f7e46a00640 1 -- 172.21.15.47:0/312576730 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f7e38083e90 con 0x7f7e50071350 2024-02-19T15:22:46.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.833+0000 7f7e4de00640 1 --2- 172.21.15.47:0/312576730 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7e2c066e40 0x7f7e2c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:46.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.833+0000 7f7e4de00640 1 --2- 172.21.15.47:0/312576730 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7e2c066e40 0x7f7e2c069300 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7e400ae000 tx=0x7f7e400a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:46.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:46.837+0000 7f7e46a00640 1 -- 172.21.15.47:0/312576730 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7e38050c00 con 0x7f7e50071350 2024-02-19T15:22:47.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.037+0000 7f7e55200640 1 -- 172.21.15.47:0/312576730 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e50075360 con 0x7f7e50071350 2024-02-19T15:22:47.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.037+0000 7f7e46a00640 1 -- 172.21.15.47:0/312576730 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4935 (secure 0 0 0) 0x7f7e38085090 con 0x7f7e50071350 2024-02-19T15:22:47.040 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:47.040 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3873760209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3873760209},{"type":"v1","addr":"172.21.15.116:6836","nonce":3873760209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/284820350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":284820350},{"type":"v1","addr":"172.21.15.47:6834","nonce":284820350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2764089359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2764089359},{"type":"v1","addr":"172.21.15.47:6835","nonce":2764089359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:22:05.228517+0000","modified":"2024-02-19T15:22:08.242600+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6837/3920452343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3920452343},{"type":"v1","addr":"172.21.15.116:6837","nonce":3920452343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-19T15:22:47.040 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 8 2024-02-19T15:22:47.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.037+0000 7f7e55200640 1 -- 172.21.15.47:0/312576730 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7e2c066e40 msgr2=0x7f7e2c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:47.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.037+0000 7f7e55200640 1 --2- 172.21.15.47:0/312576730 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7e2c066e40 0x7f7e2c069300 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7e400ae000 tx=0x7f7e400a7070 comp rx=0 tx=0).stop 2024-02-19T15:22:47.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.037+0000 7f7e55200640 1 -- 172.21.15.47:0/312576730 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e50071350 msgr2=0x7f7e50164fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:47.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.037+0000 7f7e55200640 1 --2- 172.21.15.47:0/312576730 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e50071350 0x7f7e50164fd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7e3800c420 tx=0x7f7e3800c8e0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.037+0000 7f7e55200640 1 -- 172.21.15.47:0/312576730 shutdown_connections 2024-02-19T15:22:47.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.037+0000 7f7e55200640 1 --2- 172.21.15.47:0/312576730 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7e2c066e40 0x7f7e2c069300 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.037+0000 7f7e55200640 1 --2- 172.21.15.47:0/312576730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e5015b1a0 0x7f7e50149940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.037+0000 7f7e55200640 1 --2- 172.21.15.47:0/312576730 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e50071350 0x7f7e50164fd0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.037+0000 7f7e55200640 1 --2- 172.21.15.47:0/312576730 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e5006c9a0 0x7f7e5014f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.037+0000 7f7e55200640 1 -- 172.21.15.47:0/312576730 >> 172.21.15.47:0/312576730 conn(0x7f7e500b16a0 msgr2=0x7f7e500b0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:47.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.037+0000 7f7e55200640 1 -- 172.21.15.47:0/312576730 shutdown_connections 2024-02-19T15:22:47.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.037+0000 7f7e55200640 1 -- 172.21.15.47:0/312576730 wait complete. 2024-02-19T15:22:47.061 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:22:47.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.201+0000 7f8c66800640 1 Processor -- start 2024-02-19T15:22:47.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.201+0000 7f8c66800640 1 -- start start 2024-02-19T15:22:47.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.201+0000 7f8c66800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c600d19e0 0x7f8c600d1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:47.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.201+0000 7f8c66800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8c600cec90 0x7f8c600cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:47.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.201+0000 7f8c66800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8c600cf0f0 0x7f8c600cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:47.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.201+0000 7f8c66800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c600ce280 con 0x7f8c600cec90 2024-02-19T15:22:47.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.201+0000 7f8c66800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c600d2300 con 0x7f8c600cf0f0 2024-02-19T15:22:47.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.201+0000 7f8c66800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6015cd40 con 0x7f8c600d19e0 2024-02-19T15:22:47.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.201+0000 7f8c64a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8c600cec90 0x7f8c600cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:47.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.201+0000 7f8c64a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8c600cec90 0x7f8c600cc4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36300/0 (socket says 172.21.15.47:36300) 2024-02-19T15:22:47.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.201+0000 7f8c64a00640 1 -- 172.21.15.47:0/4099885163 learned_addr learned my addr 172.21.15.47:0/4099885163 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:47.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.201+0000 7f8c65e00640 1 --2- 172.21.15.47:0/4099885163 >> v2:172.21.15.116:3300/0 conn(0x7f8c600cf0f0 0x7f8c600cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:47.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.201+0000 7f8c65400640 1 --2- 172.21.15.47:0/4099885163 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c600d19e0 0x7f8c600d1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:47.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.201+0000 7f8c64a00640 1 -- 172.21.15.47:0/4099885163 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c600d19e0 msgr2=0x7f8c600d1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:47.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.201+0000 7f8c64a00640 1 --2- 172.21.15.47:0/4099885163 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c600d19e0 0x7f8c600d1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.201+0000 7f8c64a00640 1 -- 172.21.15.47:0/4099885163 >> v2:172.21.15.116:3300/0 conn(0x7f8c600cf0f0 msgr2=0x7f8c600cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:47.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.201+0000 7f8c64a00640 1 --2- 172.21.15.47:0/4099885163 >> v2:172.21.15.116:3300/0 conn(0x7f8c600cf0f0 0x7f8c600cca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.201+0000 7f8c64a00640 1 -- 172.21.15.47:0/4099885163 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6015d020 con 0x7f8c600cec90 2024-02-19T15:22:47.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.205+0000 7f8c64a00640 1 --2- 172.21.15.47:0/4099885163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c600cec90 0x7f8c600cc4c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8c600d2a70 tx=0x7f8c4c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b3a4a3a72d62476 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:47.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.205+0000 7f8c5fe00640 1 -- 172.21.15.47:0/4099885163 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c0aa610 con 0x7f8c600cec90 2024-02-19T15:22:47.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.205+0000 7f8c5fe00640 1 -- 172.21.15.47:0/4099885163 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c4c0a6040 con 0x7f8c600cec90 2024-02-19T15:22:47.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.205+0000 7f8c5fe00640 1 -- 172.21.15.47:0/4099885163 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c0aa990 con 0x7f8c600cec90 2024-02-19T15:22:47.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.205+0000 7f8c66800640 1 -- 172.21.15.47:0/4099885163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c600cec90 msgr2=0x7f8c600cc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:47.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.205+0000 7f8c66800640 1 --2- 172.21.15.47:0/4099885163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c600cec90 0x7f8c600cc4c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8c600d2a70 tx=0x7f8c4c01c4f0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.205+0000 7f8c66800640 1 -- 172.21.15.47:0/4099885163 shutdown_connections 2024-02-19T15:22:47.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.205+0000 7f8c66800640 1 --2- 172.21.15.47:0/4099885163 >> v2:172.21.15.116:3300/0 conn(0x7f8c600cf0f0 0x7f8c600cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.205+0000 7f8c66800640 1 --2- 172.21.15.47:0/4099885163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c600cec90 0x7f8c600cc4c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.205+0000 7f8c66800640 1 --2- 172.21.15.47:0/4099885163 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c600d19e0 0x7f8c600d1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.205+0000 7f8c66800640 1 -- 172.21.15.47:0/4099885163 >> 172.21.15.47:0/4099885163 conn(0x7f8c6009a9d0 msgr2=0x7f8c6009ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:47.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.205+0000 7f8c66800640 1 -- 172.21.15.47:0/4099885163 shutdown_connections 2024-02-19T15:22:47.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.205+0000 7f8c66800640 1 -- 172.21.15.47:0/4099885163 wait complete. 2024-02-19T15:22:47.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.205+0000 7f8c66800640 1 Processor -- start 2024-02-19T15:22:47.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.205+0000 7f8c66800640 1 -- start start 2024-02-19T15:22:47.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c66800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c600cec90 0x7f8c600b0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:47.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c66800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c600d19e0 0x7f8c600b06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:47.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c66800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c60153c40 0x7f8c600c6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:47.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c66800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6006cc70 con 0x7f8c600cec90 2024-02-19T15:22:47.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c66800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6006c060 con 0x7f8c60153c40 2024-02-19T15:22:47.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c66800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6006c340 con 0x7f8c600d19e0 2024-02-19T15:22:47.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c65400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c600cec90 0x7f8c600b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:47.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c65400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c600cec90 0x7f8c600b0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36304/0 (socket says 172.21.15.47:36304) 2024-02-19T15:22:47.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c65400640 1 -- 172.21.15.47:0/895921573 learned_addr learned my addr 172.21.15.47:0/895921573 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:47.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c64a00640 1 --2- 172.21.15.47:0/895921573 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c600d19e0 0x7f8c600b06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:47.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c65e00640 1 --2- 172.21.15.47:0/895921573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c60153c40 0x7f8c600c6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:47.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c65400640 1 -- 172.21.15.47:0/895921573 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c600d19e0 msgr2=0x7f8c600b06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:47.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c65400640 1 --2- 172.21.15.47:0/895921573 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c600d19e0 0x7f8c600b06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c65400640 1 -- 172.21.15.47:0/895921573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c60153c40 msgr2=0x7f8c600c6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:47.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c65400640 1 --2- 172.21.15.47:0/895921573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c60153c40 0x7f8c600c6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c65400640 1 -- 172.21.15.47:0/895921573 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4c0094d0 con 0x7f8c600cec90 2024-02-19T15:22:47.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c65400640 1 --2- 172.21.15.47:0/895921573 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c600cec90 0x7f8c600b0180 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400e870 tx=0x7f8c5400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:47.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c5d600640 1 -- 172.21.15.47:0/895921573 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c54009510 con 0x7f8c600cec90 2024-02-19T15:22:47.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c5d600640 1 -- 172.21.15.47:0/895921573 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c5400eec0 con 0x7f8c600cec90 2024-02-19T15:22:47.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c5d600640 1 -- 172.21.15.47:0/895921573 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5400cb40 con 0x7f8c600cec90 2024-02-19T15:22:47.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c66800640 1 -- 172.21.15.47:0/895921573 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c6006c620 con 0x7f8c600cec90 2024-02-19T15:22:47.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c66800640 1 -- 172.21.15.47:0/895921573 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c600c6890 con 0x7f8c600cec90 2024-02-19T15:22:47.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c66800640 1 -- 172.21.15.47:0/895921573 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c60075360 con 0x7f8c600cec90 2024-02-19T15:22:47.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.209+0000 7f8c5d600640 1 -- 172.21.15.47:0/895921573 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8c5400cca0 con 0x7f8c600cec90 2024-02-19T15:22:47.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.213+0000 7f8c5d600640 1 --2- 172.21.15.47:0/895921573 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8c40066e40 0x7f8c40069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:47.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.213+0000 7f8c5d600640 1 -- 172.21.15.47:0/895921573 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f8c54084b40 con 0x7f8c600cec90 2024-02-19T15:22:47.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.213+0000 7f8c64a00640 1 --2- 172.21.15.47:0/895921573 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8c40066e40 0x7f8c40069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:47.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.213+0000 7f8c64a00640 1 --2- 172.21.15.47:0/895921573 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8c40066e40 0x7f8c40069300 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c0ae000 tx=0x7f8c4c0b5040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:47.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.217+0000 7f8c5d600640 1 -- 172.21.15.47:0/895921573 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8c54051940 con 0x7f8c600cec90 2024-02-19T15:22:47.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.413+0000 7f8c66800640 1 -- 172.21.15.47:0/895921573 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c600c6fb0 con 0x7f8c600cec90 2024-02-19T15:22:47.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.417+0000 7f8c5d600640 1 -- 172.21.15.47:0/895921573 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4935 (secure 0 0 0) 0x7f8c54017090 con 0x7f8c600cec90 2024-02-19T15:22:47.418 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:47.418 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3873760209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3873760209},{"type":"v1","addr":"172.21.15.116:6836","nonce":3873760209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/284820350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":284820350},{"type":"v1","addr":"172.21.15.47:6834","nonce":284820350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2764089359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2764089359},{"type":"v1","addr":"172.21.15.47:6835","nonce":2764089359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:22:05.228517+0000","modified":"2024-02-19T15:22:08.242600+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6837/3920452343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3920452343},{"type":"v1","addr":"172.21.15.116:6837","nonce":3920452343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-19T15:22:47.419 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 8 2024-02-19T15:22:47.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.417+0000 7f8c66800640 1 -- 172.21.15.47:0/895921573 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8c40066e40 msgr2=0x7f8c40069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:47.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.417+0000 7f8c66800640 1 --2- 172.21.15.47:0/895921573 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8c40066e40 0x7f8c40069300 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c0ae000 tx=0x7f8c4c0b5040 comp rx=0 tx=0).stop 2024-02-19T15:22:47.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.417+0000 7f8c66800640 1 -- 172.21.15.47:0/895921573 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c600cec90 msgr2=0x7f8c600b0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:47.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.417+0000 7f8c66800640 1 --2- 172.21.15.47:0/895921573 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c600cec90 0x7f8c600b0180 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400e870 tx=0x7f8c5400ec70 comp rx=0 tx=0).stop 2024-02-19T15:22:47.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.417+0000 7f8c66800640 1 -- 172.21.15.47:0/895921573 shutdown_connections 2024-02-19T15:22:47.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.417+0000 7f8c66800640 1 --2- 172.21.15.47:0/895921573 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8c40066e40 0x7f8c40069300 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.417+0000 7f8c66800640 1 --2- 172.21.15.47:0/895921573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c60153c40 0x7f8c600c6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.417+0000 7f8c66800640 1 --2- 172.21.15.47:0/895921573 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c600d19e0 0x7f8c600b06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.417+0000 7f8c66800640 1 --2- 172.21.15.47:0/895921573 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c600cec90 0x7f8c600b0180 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.417+0000 7f8c66800640 1 -- 172.21.15.47:0/895921573 >> 172.21.15.47:0/895921573 conn(0x7f8c6009a9d0 msgr2=0x7f8c6015adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:47.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.417+0000 7f8c66800640 1 -- 172.21.15.47:0/895921573 shutdown_connections 2024-02-19T15:22:47.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.417+0000 7f8c66800640 1 -- 172.21.15.47:0/895921573 wait complete. 2024-02-19T15:22:47.441 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:47.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.577+0000 7f5312200640 1 Processor -- start 2024-02-19T15:22:47.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.577+0000 7f5312200640 1 -- start start 2024-02-19T15:22:47.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.577+0000 7f5312200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f530c06f750 0x7f530c166dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:47.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.577+0000 7f5312200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530c0ae970 con 0x7f530c15f510 2024-02-19T15:22:47.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.577+0000 7f5312200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530c0d0200 con 0x7f530c06ebc0 2024-02-19T15:22:47.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.577+0000 7f5312200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f530c0cffc0 con 0x7f530c06f750 2024-02-19T15:22:47.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.577+0000 7f530b400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f530c15f510 0x7f530c070700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57164/0 (socket says 172.21.15.47:57164) 2024-02-19T15:22:47.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.577+0000 7f530b400640 1 -- 172.21.15.47:0/4223688195 learned_addr learned my addr 172.21.15.47:0/4223688195 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:47.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.577+0000 7f530be00640 1 --2- 172.21.15.47:0/4223688195 >> v2:172.21.15.116:3300/0 conn(0x7f530c06f750 0x7f530c166dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:47.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.581+0000 7f530aa00640 1 -- 172.21.15.47:0/4223688195 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3251220598 0 0) 0x7f530c0ae970 con 0x7f530c15f510 2024-02-19T15:22:47.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.581+0000 7f530aa00640 1 -- 172.21.15.47:0/4223688195 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52f0003530 con 0x7f530c15f510 2024-02-19T15:22:47.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.581+0000 7f530be00640 1 -- 172.21.15.47:0/4223688195 >> v1:172.21.15.116:6789/0 conn(0x7f530c06ebc0 legacy=0x7f530c06f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:47.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.581+0000 7f530aa00640 1 -- 172.21.15.47:0/4223688195 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955212070 0 0) 0x7f530c0d0200 con 0x7f530c06ebc0 2024-02-19T15:22:47.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.581+0000 7f530be00640 1 -- 172.21.15.47:0/4223688195 >> v1:172.21.15.47:6789/0 conn(0x7f530c15f510 legacy=0x7f530c070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:47.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.581+0000 7f530be00640 1 -- 172.21.15.47:0/4223688195 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530c0d04e0 con 0x7f530c06f750 2024-02-19T15:22:47.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.581+0000 7f530be00640 1 --2- 172.21.15.47:0/4223688195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f530c06f750 0x7f530c166dc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5304009d90 tx=0x7f530401c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5897121ee9f6129 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:47.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.581+0000 7f530aa00640 1 -- 172.21.15.47:0/4223688195 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53040a5890 con 0x7f530c06f750 2024-02-19T15:22:47.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.581+0000 7f530aa00640 1 -- 172.21.15.47:0/4223688195 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53040a59f0 con 0x7f530c06f750 2024-02-19T15:22:47.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.581+0000 7f530aa00640 1 -- 172.21.15.47:0/4223688195 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53040a5d80 con 0x7f530c06f750 2024-02-19T15:22:47.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f5312200640 1 -- 172.21.15.47:0/4223688195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f530c06f750 msgr2=0x7f530c166dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:47.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f5312200640 1 --2- 172.21.15.47:0/4223688195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f530c06f750 0x7f530c166dc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5304009d90 tx=0x7f530401c8e0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f5312200640 1 -- 172.21.15.47:0/4223688195 shutdown_connections 2024-02-19T15:22:47.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f5312200640 1 --2- 172.21.15.47:0/4223688195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f530c06f750 0x7f530c166dc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f5312200640 1 -- 172.21.15.47:0/4223688195 >> 172.21.15.47:0/4223688195 conn(0x7f530c0b3690 msgr2=0x7f530c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:47.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f5312200640 1 -- 172.21.15.47:0/4223688195 shutdown_connections 2024-02-19T15:22:47.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f5312200640 1 -- 172.21.15.47:0/4223688195 wait complete. 2024-02-19T15:22:47.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f5312200640 1 Processor -- start 2024-02-19T15:22:47.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f5312200640 1 -- start start 2024-02-19T15:22:47.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f5312200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f530c06ebc0 0x7f530c15a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:47.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f5312200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f530c06f750 0x7f530c15a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:47.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f5312200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f530c15f510 0x7f530c0ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:47.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f5312200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530c154b20 con 0x7f530c15f510 2024-02-19T15:22:47.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f5312200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530c1f9f50 con 0x7f530c06ebc0 2024-02-19T15:22:47.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f5312200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f530c1faad0 con 0x7f530c06f750 2024-02-19T15:22:47.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f530be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f530c15f510 0x7f530c0ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:47.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f530b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f530c06ebc0 0x7f530c15a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:47.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f530b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f530c06ebc0 0x7f530c15a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:45646/0 (socket says 172.21.15.47:45646) 2024-02-19T15:22:47.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f530b400640 1 -- 172.21.15.47:0/2887609977 learned_addr learned my addr 172.21.15.47:0/2887609977 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:47.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f530be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f530c15f510 0x7f530c0ca7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36318/0 (socket says 172.21.15.47:36318) 2024-02-19T15:22:47.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f5303e00640 1 --2- 172.21.15.47:0/2887609977 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f530c06f750 0x7f530c15a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:47.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f530b400640 1 -- 172.21.15.47:0/2887609977 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f530c06f750 msgr2=0x7f530c15a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:47.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f530b400640 1 --2- 172.21.15.47:0/2887609977 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f530c06f750 0x7f530c15a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f530b400640 1 -- 172.21.15.47:0/2887609977 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f530c15f510 msgr2=0x7f530c0ca7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:47.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f530b400640 1 --2- 172.21.15.47:0/2887609977 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f530c15f510 0x7f530c0ca7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.585+0000 7f530b400640 1 -- 172.21.15.47:0/2887609977 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5304009a70 con 0x7f530c06ebc0 2024-02-19T15:22:47.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.589+0000 7f530b400640 1 --2- 172.21.15.47:0/2887609977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f530c06ebc0 0x7f530c15a420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f52fc00c480 tx=0x7f52fc00c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:47.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.589+0000 7f5303400640 1 -- 172.21.15.47:0/2887609977 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52fc005cf0 con 0x7f530c06ebc0 2024-02-19T15:22:47.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.589+0000 7f5303400640 1 -- 172.21.15.47:0/2887609977 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52fc005e50 con 0x7f530c06ebc0 2024-02-19T15:22:47.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.589+0000 7f5312200640 1 -- 172.21.15.47:0/2887609977 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f530c1fa1a0 con 0x7f530c06ebc0 2024-02-19T15:22:47.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.589+0000 7f5303400640 1 -- 172.21.15.47:0/2887609977 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52fc007290 con 0x7f530c06ebc0 2024-02-19T15:22:47.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.589+0000 7f5312200640 1 -- 172.21.15.47:0/2887609977 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f530c1fa760 con 0x7f530c06ebc0 2024-02-19T15:22:47.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.589+0000 7f5312200640 1 -- 172.21.15.47:0/2887609977 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f530c075360 con 0x7f530c06ebc0 2024-02-19T15:22:47.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.589+0000 7f5303400640 1 -- 172.21.15.47:0/2887609977 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f52fc0073f0 con 0x7f530c06ebc0 2024-02-19T15:22:47.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.589+0000 7f5303400640 1 --2- 172.21.15.47:0/2887609977 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52e8066e70 0x7f52e8069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:47.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.589+0000 7f5303400640 1 -- 172.21.15.47:0/2887609977 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f52fc084070 con 0x7f530c06ebc0 2024-02-19T15:22:47.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.589+0000 7f5303e00640 1 --2- 172.21.15.47:0/2887609977 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52e8066e70 0x7f52e8069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:47.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.593+0000 7f5303e00640 1 --2- 172.21.15.47:0/2887609977 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52e8066e70 0x7f52e8069330 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f530c06e1a0 tx=0x7f52f4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:47.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.593+0000 7f5303400640 1 -- 172.21.15.47:0/2887609977 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f52fc0b61c0 con 0x7f530c06ebc0 2024-02-19T15:22:47.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.757+0000 7f5312200640 1 -- 172.21.15.47:0/2887609977 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f530c0b7b00 con 0x7f530c06ebc0 2024-02-19T15:22:47.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.761+0000 7f5303400640 1 -- 172.21.15.47:0/2887609977 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16914 (secure 0 0 0) 0x7f52fc050e70 con 0x7f530c06ebc0 2024-02-19T15:22:47.764 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:47.764 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":23,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:46.986956+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:03.091360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:04.222965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:47.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.761+0000 7f5312200640 1 -- 172.21.15.47:0/2887609977 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52e8066e70 msgr2=0x7f52e8069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:47.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.761+0000 7f5312200640 1 --2- 172.21.15.47:0/2887609977 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52e8066e70 0x7f52e8069330 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f530c06e1a0 tx=0x7f52f4009040 comp rx=0 tx=0).stop 2024-02-19T15:22:47.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.761+0000 7f5312200640 1 -- 172.21.15.47:0/2887609977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f530c06ebc0 msgr2=0x7f530c15a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:47.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.761+0000 7f5312200640 1 --2- 172.21.15.47:0/2887609977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f530c06ebc0 0x7f530c15a420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f52fc00c480 tx=0x7f52fc00c940 comp rx=0 tx=0).stop 2024-02-19T15:22:47.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.761+0000 7f5312200640 1 -- 172.21.15.47:0/2887609977 shutdown_connections 2024-02-19T15:22:47.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.761+0000 7f5312200640 1 --2- 172.21.15.47:0/2887609977 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52e8066e70 0x7f52e8069330 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.761+0000 7f5312200640 1 --2- 172.21.15.47:0/2887609977 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f530c15f510 0x7f530c0ca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.761+0000 7f5312200640 1 --2- 172.21.15.47:0/2887609977 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f530c06f750 0x7f530c15a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.761+0000 7f5312200640 1 --2- 172.21.15.47:0/2887609977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f530c06ebc0 0x7f530c15a420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.761+0000 7f5312200640 1 -- 172.21.15.47:0/2887609977 >> 172.21.15.47:0/2887609977 conn(0x7f530c0b3690 msgr2=0x7f530c0d1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:47.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.761+0000 7f5312200640 1 -- 172.21.15.47:0/2887609977 shutdown_connections 2024-02-19T15:22:47.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.761+0000 7f5312200640 1 -- 172.21.15.47:0/2887609977 wait complete. 2024-02-19T15:22:47.785 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T15:22:47.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.925+0000 7fab6e800640 1 Processor -- start 2024-02-19T15:22:47.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.925+0000 7fab6e800640 1 -- start start 2024-02-19T15:22:47.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.925+0000 7fab6e800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fab6815b1a0 0x7fab6815b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:47.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.925+0000 7fab6e800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fab68071350 0x7fab6815bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:47.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.925+0000 7fab6e800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab6815c000 con 0x7fab6806c9a0 2024-02-19T15:22:47.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.925+0000 7fab6e800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab680718c0 con 0x7fab68071350 2024-02-19T15:22:47.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.925+0000 7fab6e800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab68071ba0 con 0x7fab6815b1a0 2024-02-19T15:22:47.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.925+0000 7fab5fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fab68071350 0x7fab6815bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:47.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.925+0000 7fab5fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fab68071350 0x7fab6815bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36328/0 (socket says 172.21.15.47:36328) 2024-02-19T15:22:47.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.925+0000 7fab67e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fab6806c9a0 0x7fab6815c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57166/0 (socket says 172.21.15.47:57166) 2024-02-19T15:22:47.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.925+0000 7fab5fe00640 1 -- 172.21.15.47:0/3146417145 learned_addr learned my addr 172.21.15.47:0/3146417145 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:47.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.925+0000 7fab67400640 1 --2- 172.21.15.47:0/3146417145 >> v2:172.21.15.116:3300/0 conn(0x7fab6815b1a0 0x7fab6815b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:47.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.925+0000 7fab5fe00640 1 -- 172.21.15.47:0/3146417145 >> v2:172.21.15.116:3300/0 conn(0x7fab6815b1a0 msgr2=0x7fab6815b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:47.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.925+0000 7fab5fe00640 1 --2- 172.21.15.47:0/3146417145 >> v2:172.21.15.116:3300/0 conn(0x7fab6815b1a0 0x7fab6815b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.925+0000 7fab5fe00640 1 -- 172.21.15.47:0/3146417145 >> v1:172.21.15.47:6789/0 conn(0x7fab6806c9a0 legacy=0x7fab6815c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:47.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.925+0000 7fab5fe00640 1 -- 172.21.15.47:0/3146417145 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab68071e80 con 0x7fab68071350 2024-02-19T15:22:47.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.929+0000 7fab5fe00640 1 --2- 172.21.15.47:0/3146417145 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab68071350 0x7fab6815bac0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fab50009870 tx=0x7fab5001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7619ac7ec6b29b19 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:47.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.929+0000 7fab66a00640 1 -- 172.21.15.47:0/3146417145 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab500aa670 con 0x7fab68071350 2024-02-19T15:22:47.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.929+0000 7fab66a00640 1 -- 172.21.15.47:0/3146417145 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab500aa7d0 con 0x7fab68071350 2024-02-19T15:22:47.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.929+0000 7fab66a00640 1 -- 172.21.15.47:0/3146417145 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab500aab60 con 0x7fab68071350 2024-02-19T15:22:47.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.929+0000 7fab6e800640 1 -- 172.21.15.47:0/3146417145 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab68071350 msgr2=0x7fab6815bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:47.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.929+0000 7fab6e800640 1 --2- 172.21.15.47:0/3146417145 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab68071350 0x7fab6815bac0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fab50009870 tx=0x7fab5001c4f0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.929+0000 7fab6e800640 1 -- 172.21.15.47:0/3146417145 shutdown_connections 2024-02-19T15:22:47.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.929+0000 7fab6e800640 1 --2- 172.21.15.47:0/3146417145 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab68071350 0x7fab6815bac0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.929+0000 7fab6e800640 1 --2- 172.21.15.47:0/3146417145 >> v2:172.21.15.116:3300/0 conn(0x7fab6815b1a0 0x7fab6815b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.929+0000 7fab6e800640 1 -- 172.21.15.47:0/3146417145 >> 172.21.15.47:0/3146417145 conn(0x7fab680b16a0 msgr2=0x7fab680b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:47.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.929+0000 7fab6e800640 1 -- 172.21.15.47:0/3146417145 shutdown_connections 2024-02-19T15:22:47.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.929+0000 7fab6e800640 1 -- 172.21.15.47:0/3146417145 wait complete. 2024-02-19T15:22:47.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.929+0000 7fab6e800640 1 Processor -- start 2024-02-19T15:22:47.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.929+0000 7fab6e800640 1 -- start start 2024-02-19T15:22:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab6e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab6806c9a0 0x7fab68143910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab6e800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab68071350 0x7fab681645d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab6e800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab6815b1a0 0x7fab68141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab6e800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab68142870 con 0x7fab68071350 2024-02-19T15:22:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab6e800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab68142b50 con 0x7fab6806c9a0 2024-02-19T15:22:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab6e800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab68142e30 con 0x7fab6815b1a0 2024-02-19T15:22:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab5fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab68071350 0x7fab681645d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab5fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab68071350 0x7fab681645d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36344/0 (socket says 172.21.15.47:36344) 2024-02-19T15:22:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab5fe00640 1 -- 172.21.15.47:0/2783231240 learned_addr learned my addr 172.21.15.47:0/2783231240 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab67400640 1 --2- 172.21.15.47:0/2783231240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab6806c9a0 0x7fab68143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab67e00640 1 --2- 172.21.15.47:0/2783231240 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab6815b1a0 0x7fab68141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab5fe00640 1 -- 172.21.15.47:0/2783231240 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab6815b1a0 msgr2=0x7fab68141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:47.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab5fe00640 1 --2- 172.21.15.47:0/2783231240 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab6815b1a0 0x7fab68141ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab5fe00640 1 -- 172.21.15.47:0/2783231240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab6806c9a0 msgr2=0x7fab68143910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:47.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab5fe00640 1 --2- 172.21.15.47:0/2783231240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab6806c9a0 0x7fab68143910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:47.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab5fe00640 1 -- 172.21.15.47:0/2783231240 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab500094d0 con 0x7fab68071350 2024-02-19T15:22:47.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab67400640 1 --2- 172.21.15.47:0/2783231240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab6806c9a0 0x7fab68143910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:47.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab5fe00640 1 --2- 172.21.15.47:0/2783231240 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab68071350 0x7fab681645d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fab50019100 tx=0x7fab5001ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:47.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab5f400640 1 -- 172.21.15.47:0/2783231240 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab500aa670 con 0x7fab68071350 2024-02-19T15:22:47.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab5f400640 1 -- 172.21.15.47:0/2783231240 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab500b3800 con 0x7fab68071350 2024-02-19T15:22:47.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab6e800640 1 -- 172.21.15.47:0/2783231240 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab68143110 con 0x7fab68071350 2024-02-19T15:22:47.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab5f400640 1 -- 172.21.15.47:0/2783231240 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab500b3bb0 con 0x7fab68071350 2024-02-19T15:22:47.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab6e800640 1 -- 172.21.15.47:0/2783231240 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab68154660 con 0x7fab68071350 2024-02-19T15:22:47.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.933+0000 7fab6e800640 1 -- 172.21.15.47:0/2783231240 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab6806d1b0 con 0x7fab68071350 2024-02-19T15:22:47.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.937+0000 7fab5f400640 1 -- 172.21.15.47:0/2783231240 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fab500b3d10 con 0x7fab68071350 2024-02-19T15:22:47.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.937+0000 7fab5f400640 1 --2- 172.21.15.47:0/2783231240 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fab44066e40 0x7fab44069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:47.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.937+0000 7fab5f400640 1 -- 172.21.15.47:0/2783231240 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fab5011f610 con 0x7fab68071350 2024-02-19T15:22:47.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.937+0000 7fab67400640 1 --2- 172.21.15.47:0/2783231240 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fab44066e40 0x7fab44069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:47.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.937+0000 7fab67400640 1 --2- 172.21.15.47:0/2783231240 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fab44066e40 0x7fab44069300 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fab58006fd0 tx=0x7fab58008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:47.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:47.941+0000 7fab5f400640 1 -- 172.21.15.47:0/2783231240 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fab500ec3c0 con 0x7fab68071350 2024-02-19T15:22:48.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.141+0000 7fab6e800640 1 -- 172.21.15.47:0/2783231240 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fab68075360 con 0x7fab68071350 2024-02-19T15:22:48.144 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-02-19T15:22:48.141+0000 7fc100600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T15:22:48.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.141+0000 7fab5f400640 1 -- 172.21.15.47:0/2783231240 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7fab500efa80 con 0x7fab68071350 2024-02-19T15:22:48.146 INFO:teuthology.orchestra.run.smithi047.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T15:22:48.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.145+0000 7fab6e800640 1 -- 172.21.15.47:0/2783231240 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fab44066e40 msgr2=0x7fab44069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:48.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.145+0000 7fab6e800640 1 --2- 172.21.15.47:0/2783231240 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fab44066e40 0x7fab44069300 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fab58006fd0 tx=0x7fab58008040 comp rx=0 tx=0).stop 2024-02-19T15:22:48.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.145+0000 7fab6e800640 1 -- 172.21.15.47:0/2783231240 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab68071350 msgr2=0x7fab681645d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:48.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.145+0000 7fab6e800640 1 --2- 172.21.15.47:0/2783231240 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab68071350 0x7fab681645d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fab50019100 tx=0x7fab5001ad30 comp rx=0 tx=0).stop 2024-02-19T15:22:48.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.145+0000 7fab6e800640 1 -- 172.21.15.47:0/2783231240 shutdown_connections 2024-02-19T15:22:48.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.145+0000 7fab6e800640 1 --2- 172.21.15.47:0/2783231240 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fab44066e40 0x7fab44069300 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:48.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.145+0000 7fab6e800640 1 --2- 172.21.15.47:0/2783231240 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab6815b1a0 0x7fab68141ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:48.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.145+0000 7fab6e800640 1 --2- 172.21.15.47:0/2783231240 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab68071350 0x7fab681645d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:48.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.145+0000 7fab6e800640 1 --2- 172.21.15.47:0/2783231240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab6806c9a0 0x7fab68143910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:48.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.145+0000 7fab6e800640 1 -- 172.21.15.47:0/2783231240 >> 172.21.15.47:0/2783231240 conn(0x7fab680b16a0 msgr2=0x7fab681596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:48.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.145+0000 7fab6e800640 1 -- 172.21.15.47:0/2783231240 shutdown_connections 2024-02-19T15:22:48.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.145+0000 7fab6e800640 1 -- 172.21.15.47:0/2783231240 wait complete. 2024-02-19T15:22:48.160 INFO:tasks.ceph.mds.d.smithi116.stderr: -6982> 2024-02-19T15:22:09.338+0000 7fa135c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T15:22:48.169 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T15:22:48.208 INFO:tasks.ceph.mds.d.smithi116.stdout:starting mds.d at 2024-02-19T15:22:48.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.309+0000 7f69cfe00640 1 Processor -- start 2024-02-19T15:22:48.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.309+0000 7f69cfe00640 1 -- start start 2024-02-19T15:22:48.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.309+0000 7f69cfe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f69c80cb490 0x7f69c8070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:48.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.309+0000 7f69cfe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69c8072850 0x7f69c806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:48.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.309+0000 7f69cfe00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69c815fc90 con 0x7f69c815bd90 2024-02-19T15:22:48.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.309+0000 7f69cfe00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69c815ff70 con 0x7f69c80cb490 2024-02-19T15:22:48.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.309+0000 7f69cfe00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c8160250 con 0x7f69c8072850 2024-02-19T15:22:48.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.309+0000 7f69cca00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f69c80cb490 0x7f69c8070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:48.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.309+0000 7f69cd400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f69c815bd90 0x7f69c8072140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57168/0 (socket says 172.21.15.47:57168) 2024-02-19T15:22:48.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.309+0000 7f69cca00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f69c80cb490 0x7f69c8070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36346/0 (socket says 172.21.15.47:36346) 2024-02-19T15:22:48.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.309+0000 7f69cd400640 1 -- 172.21.15.47:0/2093475108 learned_addr learned my addr 172.21.15.47:0/2093475108 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:48.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.309+0000 7f69cde00640 1 --2- 172.21.15.47:0/2093475108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69c8072850 0x7f69c806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:48.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.309+0000 7f69cca00640 1 -- 172.21.15.47:0/2093475108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69c8072850 msgr2=0x7f69c806cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:48.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.309+0000 7f69cca00640 1 --2- 172.21.15.47:0/2093475108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69c8072850 0x7f69c806cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:48.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.309+0000 7f69cca00640 1 -- 172.21.15.47:0/2093475108 >> v1:172.21.15.47:6789/0 conn(0x7f69c815bd90 legacy=0x7f69c8072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:48.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.309+0000 7f69cca00640 1 -- 172.21.15.47:0/2093475108 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c8160530 con 0x7f69c80cb490 2024-02-19T15:22:48.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.309+0000 7f69c7e00640 1 -- 172.21.15.47:0/2093475108 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745413161 0 0) 0x7f69c815fc90 con 0x7f69c815bd90 2024-02-19T15:22:48.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.313+0000 7f69cca00640 1 --2- 172.21.15.47:0/2093475108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69c80cb490 0x7f69c8070960 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f69b40097c0 tx=0x7f69b401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5d95363c6810ea1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:48.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.313+0000 7f69c7e00640 1 -- 172.21.15.47:0/2093475108 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b40ab600 con 0x7f69c80cb490 2024-02-19T15:22:48.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.313+0000 7f69c7e00640 1 -- 172.21.15.47:0/2093475108 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b40a7050 con 0x7f69c80cb490 2024-02-19T15:22:48.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.313+0000 7f69c7e00640 1 -- 172.21.15.47:0/2093475108 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b40ab980 con 0x7f69c80cb490 2024-02-19T15:22:48.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.313+0000 7f69cfe00640 1 -- 172.21.15.47:0/2093475108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69c80cb490 msgr2=0x7f69c8070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:48.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.313+0000 7f69cfe00640 1 --2- 172.21.15.47:0/2093475108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69c80cb490 0x7f69c8070960 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f69b40097c0 tx=0x7f69b401c340 comp rx=0 tx=0).stop 2024-02-19T15:22:48.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.313+0000 7f69cfe00640 1 -- 172.21.15.47:0/2093475108 shutdown_connections 2024-02-19T15:22:48.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.313+0000 7f69cfe00640 1 --2- 172.21.15.47:0/2093475108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69c8072850 0x7f69c806cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:48.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.313+0000 7f69cfe00640 1 --2- 172.21.15.47:0/2093475108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69c80cb490 0x7f69c8070960 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:48.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.313+0000 7f69cfe00640 1 -- 172.21.15.47:0/2093475108 >> 172.21.15.47:0/2093475108 conn(0x7f69c80cfc20 msgr2=0x7f69c80d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:48.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.313+0000 7f69cfe00640 1 -- 172.21.15.47:0/2093475108 shutdown_connections 2024-02-19T15:22:48.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.313+0000 7f69cfe00640 1 -- 172.21.15.47:0/2093475108 wait complete. 2024-02-19T15:22:48.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cfe00640 1 Processor -- start 2024-02-19T15:22:48.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cfe00640 1 -- start start 2024-02-19T15:22:48.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cfe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69c8072850 0x7f69c815b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:48.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cfe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69c80cb490 0x7f69c8149f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:48.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69c815bd90 0x7f69c814da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:48.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cfe00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c815b990 con 0x7f69c815bd90 2024-02-19T15:22:48.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cfe00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c814bae0 con 0x7f69c80cb490 2024-02-19T15:22:48.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cfe00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c814a760 con 0x7f69c8072850 2024-02-19T15:22:48.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cde00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69c815bd90 0x7f69c814da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:48.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cde00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69c815bd90 0x7f69c814da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36356/0 (socket says 172.21.15.47:36356) 2024-02-19T15:22:48.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cde00640 1 -- 172.21.15.47:0/755517539 learned_addr learned my addr 172.21.15.47:0/755517539 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:48.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cd400640 1 --2- 172.21.15.47:0/755517539 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69c8072850 0x7f69c815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:48.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cca00640 1 --2- 172.21.15.47:0/755517539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69c80cb490 0x7f69c8149f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:48.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cde00640 1 -- 172.21.15.47:0/755517539 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69c8072850 msgr2=0x7f69c815b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:48.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cde00640 1 --2- 172.21.15.47:0/755517539 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69c8072850 0x7f69c815b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:48.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cde00640 1 -- 172.21.15.47:0/755517539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69c80cb490 msgr2=0x7f69c8149f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:48.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cde00640 1 --2- 172.21.15.47:0/755517539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69c80cb490 0x7f69c8149f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:48.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cde00640 1 -- 172.21.15.47:0/755517539 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b4009420 con 0x7f69c815bd90 2024-02-19T15:22:48.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cd400640 1 --2- 172.21.15.47:0/755517539 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69c8072850 0x7f69c815b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:48.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cde00640 1 --2- 172.21.15.47:0/755517539 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69c815bd90 0x7f69c814da80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f69c0007e60 tx=0x7f69c000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:48.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69c5600640 1 -- 172.21.15.47:0/755517539 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c0014420 con 0x7f69c815bd90 2024-02-19T15:22:48.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69c5600640 1 -- 172.21.15.47:0/755517539 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c00145a0 con 0x7f69c815bd90 2024-02-19T15:22:48.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cfe00640 1 -- 172.21.15.47:0/755517539 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c814aa40 con 0x7f69c815bd90 2024-02-19T15:22:48.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69c5600640 1 -- 172.21.15.47:0/755517539 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c0014950 con 0x7f69c815bd90 2024-02-19T15:22:48.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.317+0000 7f69cfe00640 1 -- 172.21.15.47:0/755517539 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c814b000 con 0x7f69c815bd90 2024-02-19T15:22:48.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.321+0000 7f69cfe00640 1 -- 172.21.15.47:0/755517539 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c8075360 con 0x7f69c815bd90 2024-02-19T15:22:48.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.321+0000 7f69c5600640 1 -- 172.21.15.47:0/755517539 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f69c0014ab0 con 0x7f69c815bd90 2024-02-19T15:22:48.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.321+0000 7f69c5600640 1 --2- 172.21.15.47:0/755517539 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f69a8066e70 0x7f69a8069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:48.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.321+0000 7f69c5600640 1 -- 172.21.15.47:0/755517539 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f69c0085350 con 0x7f69c815bd90 2024-02-19T15:22:48.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.321+0000 7f69cd400640 1 --2- 172.21.15.47:0/755517539 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f69a8066e70 0x7f69a8069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:48.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.321+0000 7f69cd400640 1 --2- 172.21.15.47:0/755517539 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f69a8066e70 0x7f69a8069330 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f69bc007fd0 tx=0x7f69bc007d60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:48.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.325+0000 7f69c5600640 1 -- 172.21.15.47:0/755517539 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f69c0052060 con 0x7f69c815bd90 2024-02-19T15:22:48.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:48.525+0000 7f69cfe00640 1 -- 172.21.15.47:0/755517539 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f69c806cc90 con 0x7f69c815bd90 2024-02-19T15:22:49.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.145+0000 7f69c5600640 1 -- 172.21.15.47:0/755517539 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f69c0055720 con 0x7f69c815bd90 2024-02-19T15:22:49.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.149+0000 7f69cfe00640 1 -- 172.21.15.47:0/755517539 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f69a8066e70 msgr2=0x7f69a8069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:49.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.149+0000 7f69cfe00640 1 --2- 172.21.15.47:0/755517539 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f69a8066e70 0x7f69a8069330 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f69bc007fd0 tx=0x7f69bc007d60 comp rx=0 tx=0).stop 2024-02-19T15:22:49.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.149+0000 7f69cfe00640 1 -- 172.21.15.47:0/755517539 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69c815bd90 msgr2=0x7f69c814da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:49.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.149+0000 7f69cfe00640 1 --2- 172.21.15.47:0/755517539 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69c815bd90 0x7f69c814da80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f69c0007e60 tx=0x7f69c000e4b0 comp rx=0 tx=0).stop 2024-02-19T15:22:49.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.149+0000 7f69cfe00640 1 -- 172.21.15.47:0/755517539 shutdown_connections 2024-02-19T15:22:49.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.149+0000 7f69cfe00640 1 --2- 172.21.15.47:0/755517539 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f69a8066e70 0x7f69a8069330 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:49.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.149+0000 7f69cfe00640 1 --2- 172.21.15.47:0/755517539 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69c815bd90 0x7f69c814da80 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:49.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.149+0000 7f69cfe00640 1 --2- 172.21.15.47:0/755517539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69c80cb490 0x7f69c8149f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:49.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.149+0000 7f69cfe00640 1 --2- 172.21.15.47:0/755517539 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69c8072850 0x7f69c815b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:49.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.149+0000 7f69cfe00640 1 -- 172.21.15.47:0/755517539 >> 172.21.15.47:0/755517539 conn(0x7f69c80cfc20 msgr2=0x7f69c8158a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:49.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.149+0000 7f69cfe00640 1 -- 172.21.15.47:0/755517539 shutdown_connections 2024-02-19T15:22:49.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.149+0000 7f69cfe00640 1 -- 172.21.15.47:0/755517539 wait complete. 2024-02-19T15:22:49.174 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T15:22:49.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.317+0000 7f1c10400640 1 Processor -- start 2024-02-19T15:22:49.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.317+0000 7f1c10400640 1 -- start start 2024-02-19T15:22:49.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.317+0000 7f1c10400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c0806e570 0x7f1c08071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:49.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.317+0000 7f1c10400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c080ca520 con 0x7f1c0815abf0 2024-02-19T15:22:49.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.317+0000 7f1c10400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c080ca800 con 0x7f1c08071050 2024-02-19T15:22:49.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.317+0000 7f1c10400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c080caae0 con 0x7f1c0806e570 2024-02-19T15:22:49.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.317+0000 7f1c0da00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1c0815abf0 0x7f1c0806de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57182/0 (socket says 172.21.15.47:57182) 2024-02-19T15:22:49.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.317+0000 7f1c0da00640 1 -- 172.21.15.47:0/3576676275 learned_addr learned my addr 172.21.15.47:0/3576676275 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:49.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.317+0000 7f1c0d000640 1 --2- 172.21.15.47:0/3576676275 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c0806e570 0x7f1c08071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:49.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.317+0000 7f1c07e00640 1 -- 172.21.15.47:0/3576676275 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600822158 0 0) 0x7f1c080ca520 con 0x7f1c0815abf0 2024-02-19T15:22:49.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.317+0000 7f1c07e00640 1 -- 172.21.15.47:0/3576676275 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf0003530 con 0x7f1c0815abf0 2024-02-19T15:22:49.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.317+0000 7f1c07e00640 1 -- 172.21.15.47:0/3576676275 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263355774 0 0) 0x7f1c080ca800 con 0x7f1c08071050 2024-02-19T15:22:49.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.317+0000 7f1c07e00640 1 -- 172.21.15.47:0/3576676275 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c080ca520 con 0x7f1c08071050 2024-02-19T15:22:49.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.321+0000 7f1c0d000640 1 -- 172.21.15.47:0/3576676275 >> v1:172.21.15.116:6789/0 conn(0x7f1c08071050 legacy=0x7f1c080ae490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:49.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.321+0000 7f1c0d000640 1 -- 172.21.15.47:0/3576676275 >> v1:172.21.15.47:6789/0 conn(0x7f1c0815abf0 legacy=0x7f1c0806de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:49.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.321+0000 7f1c0d000640 1 -- 172.21.15.47:0/3576676275 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c080cadc0 con 0x7f1c0806e570 2024-02-19T15:22:49.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.321+0000 7f1c0d000640 1 --2- 172.21.15.47:0/3576676275 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c0806e570 0x7f1c08071f00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40097f0 tx=0x7f1bf401c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8eea0f1f15679a19 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:49.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.321+0000 7f1c07e00640 1 -- 172.21.15.47:0/3576676275 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf40aa430 con 0x7f1c0806e570 2024-02-19T15:22:49.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.321+0000 7f1c07e00640 1 -- 172.21.15.47:0/3576676275 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf40aa590 con 0x7f1c0806e570 2024-02-19T15:22:49.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.321+0000 7f1c07e00640 1 -- 172.21.15.47:0/3576676275 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf40aa920 con 0x7f1c0806e570 2024-02-19T15:22:49.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.321+0000 7f1c10400640 1 -- 172.21.15.47:0/3576676275 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c0806e570 msgr2=0x7f1c08071f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:49.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.321+0000 7f1c10400640 1 --2- 172.21.15.47:0/3576676275 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c0806e570 0x7f1c08071f00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40097f0 tx=0x7f1bf401c4b0 comp rx=0 tx=0).stop 2024-02-19T15:22:49.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.321+0000 7f1c10400640 1 -- 172.21.15.47:0/3576676275 shutdown_connections 2024-02-19T15:22:49.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.321+0000 7f1c10400640 1 --2- 172.21.15.47:0/3576676275 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c0806e570 0x7f1c08071f00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:49.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.321+0000 7f1c10400640 1 -- 172.21.15.47:0/3576676275 >> 172.21.15.47:0/3576676275 conn(0x7f1c080cc3f0 msgr2=0x7f1c080cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:49.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.321+0000 7f1c10400640 1 -- 172.21.15.47:0/3576676275 shutdown_connections 2024-02-19T15:22:49.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c10400640 1 -- 172.21.15.47:0/3576676275 wait complete. 2024-02-19T15:22:49.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c10400640 1 Processor -- start 2024-02-19T15:22:49.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c10400640 1 -- start start 2024-02-19T15:22:49.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c10400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c0806e570 0x7f1c08148ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:49.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c10400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c08071050 0x7f1c0806cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:49.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c10400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0815abf0 0x7f1c081f5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:49.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c10400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0815e7f0 con 0x7f1c0806e570 2024-02-19T15:22:49.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c10400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c081469f0 con 0x7f1c0815abf0 2024-02-19T15:22:49.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c10400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c081466c0 con 0x7f1c08071050 2024-02-19T15:22:49.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c0da00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c0806e570 0x7f1c08148ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:49.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c0da00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c0806e570 0x7f1c08148ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36368/0 (socket says 172.21.15.47:36368) 2024-02-19T15:22:49.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c0da00640 1 -- 172.21.15.47:0/1770871796 learned_addr learned my addr 172.21.15.47:0/1770871796 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:49.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c0d000640 1 --2- 172.21.15.47:0/1770871796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c08071050 0x7f1c0806cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:49.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c0e400640 1 --2- 172.21.15.47:0/1770871796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0815abf0 0x7f1c081f5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:49.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c0da00640 1 -- 172.21.15.47:0/1770871796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c08071050 msgr2=0x7f1c0806cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:49.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c0da00640 1 --2- 172.21.15.47:0/1770871796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c08071050 0x7f1c0806cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:49.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c0da00640 1 -- 172.21.15.47:0/1770871796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0815abf0 msgr2=0x7f1c081f5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:49.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c0da00640 1 --2- 172.21.15.47:0/1770871796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0815abf0 0x7f1c081f5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:49.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c0da00640 1 -- 172.21.15.47:0/1770871796 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf4009420 con 0x7f1c0806e570 2024-02-19T15:22:49.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c0e400640 1 --2- 172.21.15.47:0/1770871796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0815abf0 0x7f1c081f5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:49.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c0da00640 1 --2- 172.21.15.47:0/1770871796 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c0806e570 0x7f1c08148ff0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc00c450 tx=0x7f1bfc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:49.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c05600640 1 -- 172.21.15.47:0/1770871796 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc005cf0 con 0x7f1c0806e570 2024-02-19T15:22:49.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c05600640 1 -- 172.21.15.47:0/1770871796 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bfc005e50 con 0x7f1c0806e570 2024-02-19T15:22:49.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c10400640 1 -- 172.21.15.47:0/1770871796 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c08149530 con 0x7f1c0806e570 2024-02-19T15:22:49.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c05600640 1 -- 172.21.15.47:0/1770871796 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc0072b0 con 0x7f1c0806e570 2024-02-19T15:22:49.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.325+0000 7f1c10400640 1 -- 172.21.15.47:0/1770871796 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c0815ece0 con 0x7f1c0806e570 2024-02-19T15:22:49.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.329+0000 7f1c10400640 1 -- 172.21.15.47:0/1770871796 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c08075360 con 0x7f1c0806e570 2024-02-19T15:22:49.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.329+0000 7f1c05600640 1 -- 172.21.15.47:0/1770871796 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1bfc007410 con 0x7f1c0806e570 2024-02-19T15:22:49.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.329+0000 7f1c05600640 1 --2- 172.21.15.47:0/1770871796 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1be8066e40 0x7f1be8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:49.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.329+0000 7f1c05600640 1 -- 172.21.15.47:0/1770871796 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f1bfc083ff0 con 0x7f1c0806e570 2024-02-19T15:22:49.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.329+0000 7f1c0d000640 1 --2- 172.21.15.47:0/1770871796 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1be8066e40 0x7f1be8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:49.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.329+0000 7f1c0d000640 1 --2- 172.21.15.47:0/1770871796 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1be8066e40 0x7f1be8069300 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40ae000 tx=0x7f1bf40a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:49.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.333+0000 7f1c05600640 1 -- 172.21.15.47:0/1770871796 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1bfc050d90 con 0x7f1c0806e570 2024-02-19T15:22:49.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:49.497+0000 7f1c10400640 1 -- 172.21.15.47:0/1770871796 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1c080cd300 con 0x7f1c0806e570 2024-02-19T15:22:50.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.149+0000 7f1c05600640 1 -- 172.21.15.47:0/1770871796 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v25) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1bfc0170d0 con 0x7f1c0806e570 2024-02-19T15:22:50.154 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_metadata' removed 2024-02-19T15:22:50.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.153+0000 7f1c10400640 1 -- 172.21.15.47:0/1770871796 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1be8066e40 msgr2=0x7f1be8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:50.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.153+0000 7f1c10400640 1 --2- 172.21.15.47:0/1770871796 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1be8066e40 0x7f1be8069300 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40ae000 tx=0x7f1bf40a7000 comp rx=0 tx=0).stop 2024-02-19T15:22:50.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.153+0000 7f1c10400640 1 -- 172.21.15.47:0/1770871796 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c0806e570 msgr2=0x7f1c08148ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:50.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.153+0000 7f1c10400640 1 --2- 172.21.15.47:0/1770871796 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c0806e570 0x7f1c08148ff0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc00c450 tx=0x7f1bfc00c910 comp rx=0 tx=0).stop 2024-02-19T15:22:50.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.153+0000 7f1c10400640 1 -- 172.21.15.47:0/1770871796 shutdown_connections 2024-02-19T15:22:50.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.153+0000 7f1c10400640 1 --2- 172.21.15.47:0/1770871796 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1be8066e40 0x7f1be8069300 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:50.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.153+0000 7f1c10400640 1 --2- 172.21.15.47:0/1770871796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0815abf0 0x7f1c081f5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:50.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.153+0000 7f1c10400640 1 --2- 172.21.15.47:0/1770871796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c08071050 0x7f1c0806cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:50.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.153+0000 7f1c10400640 1 --2- 172.21.15.47:0/1770871796 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c0806e570 0x7f1c08148ff0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:50.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.153+0000 7f1c10400640 1 -- 172.21.15.47:0/1770871796 >> 172.21.15.47:0/1770871796 conn(0x7f1c080cc3f0 msgr2=0x7f1c080af450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:50.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.153+0000 7f1c10400640 1 -- 172.21.15.47:0/1770871796 shutdown_connections 2024-02-19T15:22:50.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.153+0000 7f1c10400640 1 -- 172.21.15.47:0/1770871796 wait complete. 2024-02-19T15:22:50.182 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T15:22:50.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.325+0000 7f5f5fe00640 1 Processor -- start 2024-02-19T15:22:50.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.325+0000 7f5f5fe00640 1 -- start start 2024-02-19T15:22:50.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.325+0000 7f5f5fe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5f600b33b0 0x7f5f600b3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:50.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.325+0000 7f5f5fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f6006cf80 0x7f5f6006d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:50.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.325+0000 7f5f5fe00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f600d0140 con 0x7f5f600cf5f0 2024-02-19T15:22:50.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.325+0000 7f5f5fe00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f6015f180 con 0x7f5f600b33b0 2024-02-19T15:22:50.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.325+0000 7f5f5fe00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6015e5b0 con 0x7f5f6006cf80 2024-02-19T15:22:50.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.325+0000 7f5f5f400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f6006cf80 0x7f5f6006d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:50.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.325+0000 7f5f5ea00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5f600b33b0 0x7f5f600b3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:50.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.325+0000 7f5f56000640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5f600cf5f0 0x7f5f600cfa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:43134/0 (socket says 172.21.15.47:43134) 2024-02-19T15:22:50.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.325+0000 7f5f56000640 1 -- 172.21.15.47:0/1199639 learned_addr learned my addr 172.21.15.47:0/1199639 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:50.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.325+0000 7f5f5f400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f6006cf80 0x7f5f6006d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:39188/0 (socket says 172.21.15.47:39188) 2024-02-19T15:22:50.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.329+0000 7f5f5f400640 1 -- 172.21.15.47:0/1199639 >> v2:172.21.15.116:3300/0 conn(0x7f5f600b33b0 msgr2=0x7f5f600b3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:50.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.329+0000 7f5f5f400640 1 --2- 172.21.15.47:0/1199639 >> v2:172.21.15.116:3300/0 conn(0x7f5f600b33b0 0x7f5f600b3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:50.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.329+0000 7f5f5f400640 1 -- 172.21.15.47:0/1199639 >> v1:172.21.15.116:6789/0 conn(0x7f5f600cf5f0 legacy=0x7f5f600cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:50.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.329+0000 7f5f5f400640 1 -- 172.21.15.47:0/1199639 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f600b3cd0 con 0x7f5f6006cf80 2024-02-19T15:22:50.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.329+0000 7f5f5e000640 1 -- 172.21.15.47:0/1199639 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865942408 0 0) 0x7f5f600d0140 con 0x7f5f600cf5f0 2024-02-19T15:22:50.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.329+0000 7f5f5ea00640 1 --2- 172.21.15.47:0/1199639 >> v2:172.21.15.116:3300/0 conn(0x7f5f600b33b0 0x7f5f600b3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:50.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.329+0000 7f5f5f400640 1 --2- 172.21.15.47:0/1199639 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f6006cf80 0x7f5f6006d410 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5f58009d90 tx=0x7f5f5801c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b50f80e145c7d9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:50.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.329+0000 7f5f5e000640 1 -- 172.21.15.47:0/1199639 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f580a5a00 con 0x7f5f6006cf80 2024-02-19T15:22:50.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.329+0000 7f5f5e000640 1 -- 172.21.15.47:0/1199639 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f580a5b60 con 0x7f5f6006cf80 2024-02-19T15:22:50.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.329+0000 7f5f5e000640 1 -- 172.21.15.47:0/1199639 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f580ad440 con 0x7f5f6006cf80 2024-02-19T15:22:50.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.329+0000 7f5f5fe00640 1 -- 172.21.15.47:0/1199639 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f6006cf80 msgr2=0x7f5f6006d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:50.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.329+0000 7f5f5fe00640 1 --2- 172.21.15.47:0/1199639 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f6006cf80 0x7f5f6006d410 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5f58009d90 tx=0x7f5f5801c940 comp rx=0 tx=0).stop 2024-02-19T15:22:50.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.329+0000 7f5f5fe00640 1 -- 172.21.15.47:0/1199639 shutdown_connections 2024-02-19T15:22:50.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.329+0000 7f5f5fe00640 1 --2- 172.21.15.47:0/1199639 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f6006cf80 0x7f5f6006d410 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:50.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.329+0000 7f5f5fe00640 1 --2- 172.21.15.47:0/1199639 >> v2:172.21.15.116:3300/0 conn(0x7f5f600b33b0 0x7f5f600b3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:50.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.329+0000 7f5f5fe00640 1 -- 172.21.15.47:0/1199639 >> 172.21.15.47:0/1199639 conn(0x7f5f6006aeb0 msgr2=0x7f5f6006b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:50.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5fe00640 1 -- 172.21.15.47:0/1199639 shutdown_connections 2024-02-19T15:22:50.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5fe00640 1 -- 172.21.15.47:0/1199639 wait complete. 2024-02-19T15:22:50.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5fe00640 1 Processor -- start 2024-02-19T15:22:50.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5fe00640 1 -- start start 2024-02-19T15:22:50.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5f6006cf80 0x7f5f60159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:50.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f600b33b0 0x7f5f60146d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:50.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f600cf5f0 0x7f5f6015b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:50.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5fe00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f601597f0 con 0x7f5f6006cf80 2024-02-19T15:22:50.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5fe00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f60158be0 con 0x7f5f600b33b0 2024-02-19T15:22:50.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5fe00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f60158ec0 con 0x7f5f600cf5f0 2024-02-19T15:22:50.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5f6006cf80 0x7f5f60159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:50.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5f6006cf80 0x7f5f60159c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36372/0 (socket says 172.21.15.47:36372) 2024-02-19T15:22:50.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5ea00640 1 -- 172.21.15.47:0/1793333786 learned_addr learned my addr 172.21.15.47:0/1793333786 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:50.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f56000640 1 --2- 172.21.15.47:0/1793333786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f600b33b0 0x7f5f60146d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:50.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5f400640 1 --2- 172.21.15.47:0/1793333786 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f600cf5f0 0x7f5f6015b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:50.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5ea00640 1 -- 172.21.15.47:0/1793333786 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f600cf5f0 msgr2=0x7f5f6015b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:50.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5ea00640 1 --2- 172.21.15.47:0/1793333786 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f600cf5f0 0x7f5f6015b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:50.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5ea00640 1 -- 172.21.15.47:0/1793333786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f600b33b0 msgr2=0x7f5f60146d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:50.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5ea00640 1 --2- 172.21.15.47:0/1793333786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f600b33b0 0x7f5f60146d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:50.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5ea00640 1 -- 172.21.15.47:0/1793333786 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f50009420 con 0x7f5f6006cf80 2024-02-19T15:22:50.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5f400640 1 --2- 172.21.15.47:0/1793333786 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f600cf5f0 0x7f5f6015b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:50.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5ea00640 1 --2- 172.21.15.47:0/1793333786 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5f6006cf80 0x7f5f60159c80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000ebe0 tx=0x7f5f5000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:50.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f57400640 1 -- 172.21.15.47:0/1793333786 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5000eee0 con 0x7f5f6006cf80 2024-02-19T15:22:50.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f57400640 1 -- 172.21.15.47:0/1793333786 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5000cc30 con 0x7f5f6006cf80 2024-02-19T15:22:50.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f57400640 1 -- 172.21.15.47:0/1793333786 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f50013470 con 0x7f5f6006cf80 2024-02-19T15:22:50.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.333+0000 7f5f5fe00640 1 -- 172.21.15.47:0/1793333786 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f58009a70 con 0x7f5f6006cf80 2024-02-19T15:22:50.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.337+0000 7f5f5fe00640 1 -- 172.21.15.47:0/1793333786 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f60159480 con 0x7f5f6006cf80 2024-02-19T15:22:50.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.337+0000 7f5f5fe00640 1 -- 172.21.15.47:0/1793333786 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f60069140 con 0x7f5f6006cf80 2024-02-19T15:22:50.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.337+0000 7f5f57400640 1 -- 172.21.15.47:0/1793333786 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5f500135d0 con 0x7f5f6006cf80 2024-02-19T15:22:50.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.337+0000 7f5f57400640 1 --2- 172.21.15.47:0/1793333786 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5f3c066e90 0x7f5f3c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:50.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.337+0000 7f5f57400640 1 -- 172.21.15.47:0/1793333786 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f5f50084a10 con 0x7f5f6006cf80 2024-02-19T15:22:50.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.337+0000 7f5f56000640 1 --2- 172.21.15.47:0/1793333786 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5f3c066e90 0x7f5f3c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:50.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.337+0000 7f5f56000640 1 --2- 172.21.15.47:0/1793333786 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5f3c066e90 0x7f5f3c069350 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5f601f9de0 tx=0x7f5f48007df0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:50.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.341+0000 7f5f57400640 1 -- 172.21.15.47:0/1793333786 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f5f50051af0 con 0x7f5f6006cf80 2024-02-19T15:22:50.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:50.505+0000 7f5f5fe00640 1 -- 172.21.15.47:0/1793333786 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5f6004f260 con 0x7f5f6006cf80 2024-02-19T15:22:51.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.153+0000 7f5f57400640 1 -- 172.21.15.47:0/1793333786 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v26) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5f500170d0 con 0x7f5f6006cf80 2024-02-19T15:22:51.158 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_data' removed 2024-02-19T15:22:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.157+0000 7f5f5fe00640 1 -- 172.21.15.47:0/1793333786 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5f3c066e90 msgr2=0x7f5f3c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.157+0000 7f5f5fe00640 1 --2- 172.21.15.47:0/1793333786 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5f3c066e90 0x7f5f3c069350 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5f601f9de0 tx=0x7f5f48007df0 comp rx=0 tx=0).stop 2024-02-19T15:22:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.157+0000 7f5f5fe00640 1 -- 172.21.15.47:0/1793333786 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5f6006cf80 msgr2=0x7f5f60159c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.157+0000 7f5f5fe00640 1 --2- 172.21.15.47:0/1793333786 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5f6006cf80 0x7f5f60159c80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000ebe0 tx=0x7f5f5000c520 comp rx=0 tx=0).stop 2024-02-19T15:22:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.157+0000 7f5f5fe00640 1 -- 172.21.15.47:0/1793333786 shutdown_connections 2024-02-19T15:22:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.157+0000 7f5f5fe00640 1 --2- 172.21.15.47:0/1793333786 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5f3c066e90 0x7f5f3c069350 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.157+0000 7f5f5fe00640 1 --2- 172.21.15.47:0/1793333786 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f600cf5f0 0x7f5f6015b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.157+0000 7f5f5fe00640 1 --2- 172.21.15.47:0/1793333786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f600b33b0 0x7f5f60146d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.157+0000 7f5f5fe00640 1 --2- 172.21.15.47:0/1793333786 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5f6006cf80 0x7f5f60159c80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.157+0000 7f5f5fe00640 1 -- 172.21.15.47:0/1793333786 >> 172.21.15.47:0/1793333786 conn(0x7f5f6006aeb0 msgr2=0x7f5f6006b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:51.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.157+0000 7f5f5fe00640 1 -- 172.21.15.47:0/1793333786 shutdown_connections 2024-02-19T15:22:51.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.157+0000 7f5f5fe00640 1 -- 172.21.15.47:0/1793333786 wait complete. 2024-02-19T15:22:51.187 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T15:22:51.187 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-19T15:22:51.188 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T15:22:51.188 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:22:51.188 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T15:22:51.188 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-19T15:22:51.188 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T15:22:51.188 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:22:51.189 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T15:22:51.189 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-19T15:22:51.189 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T15:22:51.189 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:22:51.189 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T15:22:51.189 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-19T15:22:51.190 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T15:22:51.190 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:22:51.199 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T15:22:51.200 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T15:22:51.201 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T15:22:51.201 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T15:22:51.202 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T15:22:51.202 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T15:22:51.204 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T15:22:51.204 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T15:22:51.248 INFO:tasks.ceph.mds.a:Started 2024-02-19T15:22:51.248 DEBUG:teuthology.parallel:result is None 2024-02-19T15:22:51.249 INFO:tasks.ceph.mds.b:Started 2024-02-19T15:22:51.249 DEBUG:teuthology.parallel:result is None 2024-02-19T15:22:51.250 INFO:tasks.ceph.mds.c:Started 2024-02-19T15:22:51.250 DEBUG:teuthology.parallel:result is None 2024-02-19T15:22:51.251 INFO:tasks.ceph.mds.d:Started 2024-02-19T15:22:51.251 DEBUG:teuthology.parallel:result is None 2024-02-19T15:22:51.252 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-19T15:22:51.333 INFO:tasks.ceph.mds.b.smithi116.stdout:starting mds.b at 2024-02-19T15:22:51.334 INFO:tasks.ceph.mds.d.smithi116.stdout:starting mds.d at 2024-02-19T15:22:51.335 INFO:tasks.ceph.mds.a.smithi047.stdout:starting mds.a at 2024-02-19T15:22:51.336 INFO:tasks.ceph.mds.c.smithi047.stdout:starting mds.c at 2024-02-19T15:22:51.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.393+0000 7fc75d600640 1 Processor -- start 2024-02-19T15:22:51.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.393+0000 7fc75d600640 1 -- start start 2024-02-19T15:22:51.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.397+0000 7fc75d600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7581456a0 0x7fc7580cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:51.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.397+0000 7fc75d600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc7580cf4a0 0x7fc75806aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:51.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.397+0000 7fc75d600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc75806b3f0 0x7fc75815f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:51.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.397+0000 7fc75d600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7580d03c0 con 0x7fc75806b3f0 2024-02-19T15:22:51.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.397+0000 7fc75d600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7580cf900 con 0x7fc7580cf4a0 2024-02-19T15:22:51.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.397+0000 7fc75d600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75806b850 con 0x7fc7581456a0 2024-02-19T15:22:51.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.397+0000 7fc757400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc75806b3f0 0x7fc75815f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:51.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.397+0000 7fc757400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc75806b3f0 0x7fc75815f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36398/0 (socket says 172.21.15.47:36398) 2024-02-19T15:22:51.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.397+0000 7fc757400640 1 -- 172.21.15.47:0/1747815857 learned_addr learned my addr 172.21.15.47:0/1747815857 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:51.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.397+0000 7fc74fe00640 1 --2- 172.21.15.47:0/1747815857 >> v2:172.21.15.116:3300/0 conn(0x7fc7580cf4a0 0x7fc75806aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:51.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.397+0000 7fc756a00640 1 --2- 172.21.15.47:0/1747815857 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7581456a0 0x7fc7580cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:51.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.397+0000 7fc757400640 1 -- 172.21.15.47:0/1747815857 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7581456a0 msgr2=0x7fc7580cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:51.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.397+0000 7fc757400640 1 --2- 172.21.15.47:0/1747815857 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7581456a0 0x7fc7580cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:51.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.397+0000 7fc757400640 1 -- 172.21.15.47:0/1747815857 >> v2:172.21.15.116:3300/0 conn(0x7fc7580cf4a0 msgr2=0x7fc75806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:51.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.397+0000 7fc757400640 1 --2- 172.21.15.47:0/1747815857 >> v2:172.21.15.116:3300/0 conn(0x7fc7580cf4a0 0x7fc75806aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:51.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.397+0000 7fc757400640 1 -- 172.21.15.47:0/1747815857 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc758145a80 con 0x7fc75806b3f0 2024-02-19T15:22:51.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.401+0000 7fc757400640 1 --2- 172.21.15.47:0/1747815857 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc75806b3f0 0x7fc75815f830 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc7580b5e50 tx=0x7fc75001c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df024d7b24540d71 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:51.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.401+0000 7fc756000640 1 -- 172.21.15.47:0/1747815857 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7500a8420 con 0x7fc75806b3f0 2024-02-19T15:22:51.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.401+0000 7fc756000640 1 -- 172.21.15.47:0/1747815857 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7500a8580 con 0x7fc75806b3f0 2024-02-19T15:22:51.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.401+0000 7fc756000640 1 -- 172.21.15.47:0/1747815857 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7500a8910 con 0x7fc75806b3f0 2024-02-19T15:22:51.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.401+0000 7fc75d600640 1 -- 172.21.15.47:0/1747815857 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc75806b3f0 msgr2=0x7fc75815f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:51.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.401+0000 7fc75d600640 1 --2- 172.21.15.47:0/1747815857 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc75806b3f0 0x7fc75815f830 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc7580b5e50 tx=0x7fc75001c730 comp rx=0 tx=0).stop 2024-02-19T15:22:51.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.401+0000 7fc75d600640 1 -- 172.21.15.47:0/1747815857 shutdown_connections 2024-02-19T15:22:51.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.401+0000 7fc75d600640 1 --2- 172.21.15.47:0/1747815857 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc75806b3f0 0x7fc75815f830 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:51.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.401+0000 7fc75d600640 1 --2- 172.21.15.47:0/1747815857 >> v2:172.21.15.116:3300/0 conn(0x7fc7580cf4a0 0x7fc75806aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:51.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.401+0000 7fc75d600640 1 --2- 172.21.15.47:0/1747815857 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7581456a0 0x7fc7580cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:51.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.401+0000 7fc75d600640 1 -- 172.21.15.47:0/1747815857 >> 172.21.15.47:0/1747815857 conn(0x7fc7580d0ad0 msgr2=0x7fc7580d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:51.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.401+0000 7fc75d600640 1 -- 172.21.15.47:0/1747815857 shutdown_connections 2024-02-19T15:22:51.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.401+0000 7fc75d600640 1 -- 172.21.15.47:0/1747815857 wait complete. 2024-02-19T15:22:51.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.401+0000 7fc75d600640 1 Processor -- start 2024-02-19T15:22:51.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.401+0000 7fc75d600640 1 -- start start 2024-02-19T15:22:51.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.401+0000 7fc75d600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc75806b3f0 0x7fc758143d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:51.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.401+0000 7fc75d600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7580cf4a0 0x7fc758141ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:51.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.401+0000 7fc75d600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7581456a0 0x7fc758142220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:51.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.401+0000 7fc75d600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75815c8c0 con 0x7fc75806b3f0 2024-02-19T15:22:51.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.405+0000 7fc75d600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc758145de0 con 0x7fc7581456a0 2024-02-19T15:22:51.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.405+0000 7fc75d600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7581460c0 con 0x7fc7580cf4a0 2024-02-19T15:22:51.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.405+0000 7fc756a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc75806b3f0 0x7fc758143d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:51.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.405+0000 7fc756a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc75806b3f0 0x7fc758143d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36406/0 (socket says 172.21.15.47:36406) 2024-02-19T15:22:51.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.405+0000 7fc756a00640 1 -- 172.21.15.47:0/2550348605 learned_addr learned my addr 172.21.15.47:0/2550348605 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:51.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.405+0000 7fc74fe00640 1 --2- 172.21.15.47:0/2550348605 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7580cf4a0 0x7fc758141ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:51.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.405+0000 7fc757400640 1 --2- 172.21.15.47:0/2550348605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7581456a0 0x7fc758142220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:51.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.405+0000 7fc756a00640 1 -- 172.21.15.47:0/2550348605 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7580cf4a0 msgr2=0x7fc758141ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:51.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.405+0000 7fc756a00640 1 --2- 172.21.15.47:0/2550348605 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7580cf4a0 0x7fc758141ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:51.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.405+0000 7fc756a00640 1 -- 172.21.15.47:0/2550348605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7581456a0 msgr2=0x7fc758142220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:51.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.405+0000 7fc756a00640 1 --2- 172.21.15.47:0/2550348605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7581456a0 0x7fc758142220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:51.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.405+0000 7fc756a00640 1 -- 172.21.15.47:0/2550348605 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc750009b30 con 0x7fc75806b3f0 2024-02-19T15:22:51.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.405+0000 7fc756a00640 1 --2- 172.21.15.47:0/2550348605 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc75806b3f0 0x7fc758143d60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc74800d5f0 tx=0x7fc74800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:51.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.405+0000 7fc74ea00640 1 -- 172.21.15.47:0/2550348605 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7480125f0 con 0x7fc75806b3f0 2024-02-19T15:22:51.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.405+0000 7fc74ea00640 1 -- 172.21.15.47:0/2550348605 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc748012750 con 0x7fc75806b3f0 2024-02-19T15:22:51.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.405+0000 7fc75d600640 1 -- 172.21.15.47:0/2550348605 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7581463a0 con 0x7fc75806b3f0 2024-02-19T15:22:51.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.405+0000 7fc74ea00640 1 -- 172.21.15.47:0/2550348605 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc748012ae0 con 0x7fc75806b3f0 2024-02-19T15:22:51.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.405+0000 7fc75d600640 1 -- 172.21.15.47:0/2550348605 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc758146960 con 0x7fc75806b3f0 2024-02-19T15:22:51.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.405+0000 7fc75d600640 1 -- 172.21.15.47:0/2550348605 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc758075360 con 0x7fc75806b3f0 2024-02-19T15:22:51.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.405+0000 7fc74ea00640 1 -- 172.21.15.47:0/2550348605 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc748012c40 con 0x7fc75806b3f0 2024-02-19T15:22:51.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.409+0000 7fc74ea00640 1 --2- 172.21.15.47:0/2550348605 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc734066e40 0x7fc734069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:51.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.409+0000 7fc74ea00640 1 -- 172.21.15.47:0/2550348605 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc748084b40 con 0x7fc75806b3f0 2024-02-19T15:22:51.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.409+0000 7fc74fe00640 1 --2- 172.21.15.47:0/2550348605 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc734066e40 0x7fc734069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:51.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.409+0000 7fc74fe00640 1 --2- 172.21.15.47:0/2550348605 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc734066e40 0x7fc734069300 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc7580701c0 tx=0x7fc740008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:51.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.409+0000 7fc74ea00640 1 -- 172.21.15.47:0/2550348605 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc748051c40 con 0x7fc75806b3f0 2024-02-19T15:22:51.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:51.585+0000 7fc75d600640 1 -- 172.21.15.47:0/2550348605 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc758072b90 con 0x7fc75806b3f0 2024-02-19T15:22:52.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.164+0000 7fc74ea00640 1 -- 172.21.15.47:0/2550348605 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v28) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc748055300 con 0x7fc75806b3f0 2024-02-19T15:22:52.171 INFO:teuthology.orchestra.run.smithi047.stderr: removed all blocklist entries 2024-02-19T15:22:52.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.168+0000 7fc75d600640 1 -- 172.21.15.47:0/2550348605 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc734066e40 msgr2=0x7fc734069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:52.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.168+0000 7fc75d600640 1 --2- 172.21.15.47:0/2550348605 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc734066e40 0x7fc734069300 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc7580701c0 tx=0x7fc740008040 comp rx=0 tx=0).stop 2024-02-19T15:22:52.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.168+0000 7fc75d600640 1 -- 172.21.15.47:0/2550348605 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc75806b3f0 msgr2=0x7fc758143d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:52.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.168+0000 7fc75d600640 1 --2- 172.21.15.47:0/2550348605 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc75806b3f0 0x7fc758143d60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc74800d5f0 tx=0x7fc74800dab0 comp rx=0 tx=0).stop 2024-02-19T15:22:52.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.168+0000 7fc75d600640 1 -- 172.21.15.47:0/2550348605 shutdown_connections 2024-02-19T15:22:52.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.168+0000 7fc75d600640 1 --2- 172.21.15.47:0/2550348605 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc734066e40 0x7fc734069300 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:52.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.168+0000 7fc75d600640 1 --2- 172.21.15.47:0/2550348605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7581456a0 0x7fc758142220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:52.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.168+0000 7fc75d600640 1 --2- 172.21.15.47:0/2550348605 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7580cf4a0 0x7fc758141ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:52.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.168+0000 7fc75d600640 1 --2- 172.21.15.47:0/2550348605 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc75806b3f0 0x7fc758143d60 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:52.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.168+0000 7fc75d600640 1 -- 172.21.15.47:0/2550348605 >> 172.21.15.47:0/2550348605 conn(0x7fc7580d0ad0 msgr2=0x7fc7580cd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:52.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.168+0000 7fc75d600640 1 -- 172.21.15.47:0/2550348605 shutdown_connections 2024-02-19T15:22:52.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.168+0000 7fc75d600640 1 -- 172.21.15.47:0/2550348605 wait complete. 2024-02-19T15:22:52.200 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-19T15:22:52.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.340+0000 7fd956a00640 1 Processor -- start 2024-02-19T15:22:52.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.340+0000 7fd956a00640 1 -- start start 2024-02-19T15:22:52.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.340+0000 7fd956a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd95015b1a0 0x7fd95015b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:52.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.340+0000 7fd956a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd950071350 0x7fd95015bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:52.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.344+0000 7fd956a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95015c000 con 0x7fd95006c9a0 2024-02-19T15:22:52.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.344+0000 7fd956a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9500718c0 con 0x7fd95015b1a0 2024-02-19T15:22:52.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.344+0000 7fd956a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd950071ba0 con 0x7fd950071350 2024-02-19T15:22:52.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.344+0000 7fd94fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd95015b1a0 0x7fd95015b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:52.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.344+0000 7fd954a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd95006c9a0 0x7fd95015c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57208/0 (socket says 172.21.15.47:57208) 2024-02-19T15:22:52.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.344+0000 7fd954a00640 1 -- 172.21.15.47:0/1499710184 learned_addr learned my addr 172.21.15.47:0/1499710184 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:52.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.344+0000 7fd947e00640 1 --2- 172.21.15.47:0/1499710184 >> v2:172.21.15.116:3300/0 conn(0x7fd950071350 0x7fd95015bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:52.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.344+0000 7fd94fe00640 1 -- 172.21.15.47:0/1499710184 >> v2:172.21.15.116:3300/0 conn(0x7fd950071350 msgr2=0x7fd95015bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:52.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.344+0000 7fd94fe00640 1 --2- 172.21.15.47:0/1499710184 >> v2:172.21.15.116:3300/0 conn(0x7fd950071350 0x7fd95015bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:52.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.344+0000 7fd94fe00640 1 -- 172.21.15.47:0/1499710184 >> v1:172.21.15.47:6789/0 conn(0x7fd95006c9a0 legacy=0x7fd95015c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:52.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.344+0000 7fd94f400640 1 -- 172.21.15.47:0/1499710184 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4223047212 0 0) 0x7fd95015c000 con 0x7fd95006c9a0 2024-02-19T15:22:52.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.344+0000 7fd94fe00640 1 -- 172.21.15.47:0/1499710184 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd950071e80 con 0x7fd95015b1a0 2024-02-19T15:22:52.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.344+0000 7fd94fe00640 1 --2- 172.21.15.47:0/1499710184 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd95015b1a0 0x7fd95015b580 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd9400097c0 tx=0x7fd94001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc939d260c1bc17 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:52.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.344+0000 7fd94f400640 1 -- 172.21.15.47:0/1499710184 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9400ab650 con 0x7fd95015b1a0 2024-02-19T15:22:52.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.344+0000 7fd94f400640 1 -- 172.21.15.47:0/1499710184 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9400ab7b0 con 0x7fd95015b1a0 2024-02-19T15:22:52.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.344+0000 7fd94f400640 1 -- 172.21.15.47:0/1499710184 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9400abb40 con 0x7fd95015b1a0 2024-02-19T15:22:52.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd956a00640 1 -- 172.21.15.47:0/1499710184 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd95015b1a0 msgr2=0x7fd95015b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:52.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd956a00640 1 --2- 172.21.15.47:0/1499710184 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd95015b1a0 0x7fd95015b580 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd9400097c0 tx=0x7fd94001c430 comp rx=0 tx=0).stop 2024-02-19T15:22:52.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd956a00640 1 -- 172.21.15.47:0/1499710184 shutdown_connections 2024-02-19T15:22:52.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd956a00640 1 --2- 172.21.15.47:0/1499710184 >> v2:172.21.15.116:3300/0 conn(0x7fd950071350 0x7fd95015bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:52.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd956a00640 1 --2- 172.21.15.47:0/1499710184 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd95015b1a0 0x7fd95015b580 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:52.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd956a00640 1 -- 172.21.15.47:0/1499710184 >> 172.21.15.47:0/1499710184 conn(0x7fd9500b16a0 msgr2=0x7fd9500b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:52.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd956a00640 1 -- 172.21.15.47:0/1499710184 shutdown_connections 2024-02-19T15:22:52.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd956a00640 1 -- 172.21.15.47:0/1499710184 wait complete. 2024-02-19T15:22:52.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd956a00640 1 Processor -- start 2024-02-19T15:22:52.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd956a00640 1 -- start start 2024-02-19T15:22:52.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd956a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd95006c9a0 0x7fd9500b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:52.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd956a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd950071350 0x7fd9500a9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:52.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd956a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd95015b1a0 0x7fd95014e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:52.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd956a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9500b3000 con 0x7fd950071350 2024-02-19T15:22:52.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd956a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9500a9970 con 0x7fd95006c9a0 2024-02-19T15:22:52.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd956a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9500aa1d0 con 0x7fd95015b1a0 2024-02-19T15:22:52.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd947e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd950071350 0x7fd9500a9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:52.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd947e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd950071350 0x7fd9500a9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36432/0 (socket says 172.21.15.47:36432) 2024-02-19T15:22:52.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd947e00640 1 -- 172.21.15.47:0/3202446402 learned_addr learned my addr 172.21.15.47:0/3202446402 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:52.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd94fe00640 1 --2- 172.21.15.47:0/3202446402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd95006c9a0 0x7fd9500b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:52.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.348+0000 7fd954a00640 1 --2- 172.21.15.47:0/3202446402 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd95015b1a0 0x7fd95014e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:52.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.352+0000 7fd947e00640 1 -- 172.21.15.47:0/3202446402 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd95015b1a0 msgr2=0x7fd95014e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:52.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.352+0000 7fd947e00640 1 --2- 172.21.15.47:0/3202446402 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd95015b1a0 0x7fd95014e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:52.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.352+0000 7fd947e00640 1 -- 172.21.15.47:0/3202446402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd95006c9a0 msgr2=0x7fd9500b47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:52.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.352+0000 7fd947e00640 1 --2- 172.21.15.47:0/3202446402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd95006c9a0 0x7fd9500b47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:52.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.352+0000 7fd947e00640 1 -- 172.21.15.47:0/3202446402 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd940009420 con 0x7fd950071350 2024-02-19T15:22:52.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.352+0000 7fd954a00640 1 --2- 172.21.15.47:0/3202446402 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd95015b1a0 0x7fd95014e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:52.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.352+0000 7fd947e00640 1 --2- 172.21.15.47:0/3202446402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd950071350 0x7fd9500a9430 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd93800e930 tx=0x7fd93800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:52.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.352+0000 7fd94cc00640 1 -- 172.21.15.47:0/3202446402 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd938009690 con 0x7fd950071350 2024-02-19T15:22:52.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.352+0000 7fd94cc00640 1 -- 172.21.15.47:0/3202446402 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93800ca00 con 0x7fd950071350 2024-02-19T15:22:52.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.352+0000 7fd94cc00640 1 -- 172.21.15.47:0/3202446402 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93800cd70 con 0x7fd950071350 2024-02-19T15:22:52.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.352+0000 7fd956a00640 1 -- 172.21.15.47:0/3202446402 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9500b3d00 con 0x7fd950071350 2024-02-19T15:22:52.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.352+0000 7fd956a00640 1 -- 172.21.15.47:0/3202446402 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9500b42c0 con 0x7fd950071350 2024-02-19T15:22:52.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.352+0000 7fd956a00640 1 -- 172.21.15.47:0/3202446402 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd950075360 con 0x7fd950071350 2024-02-19T15:22:52.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.352+0000 7fd94cc00640 1 -- 172.21.15.47:0/3202446402 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd938013470 con 0x7fd950071350 2024-02-19T15:22:52.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.352+0000 7fd94cc00640 1 --2- 172.21.15.47:0/3202446402 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd92c066e70 0x7fd92c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:52.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.352+0000 7fd94cc00640 1 -- 172.21.15.47:0/3202446402 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd938084810 con 0x7fd950071350 2024-02-19T15:22:52.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.352+0000 7fd94fe00640 1 --2- 172.21.15.47:0/3202446402 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd92c066e70 0x7fd92c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:52.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.356+0000 7fd94fe00640 1 --2- 172.21.15.47:0/3202446402 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd92c066e70 0x7fd92c069330 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd9400af000 tx=0x7fd9400a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:52.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.356+0000 7fd94cc00640 1 -- 172.21.15.47:0/3202446402 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd938051b50 con 0x7fd950071350 2024-02-19T15:22:52.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.576+0000 7fd956a00640 1 -- 172.21.15.47:0/3202446402 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd950157dc0 con 0x7fd950071350 2024-02-19T15:22:52.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.576+0000 7fd94cc00640 1 -- 172.21.15.47:0/3202446402 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fd938017090 con 0x7fd950071350 2024-02-19T15:22:52.582 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:52.582 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:22:52.582 INFO:teuthology.orchestra.run.smithi047.stdout: "auth_dump": [ 2024-02-19T15:22:52.583 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.583 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.a", 2024-02-19T15:22:52.583 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNl8BP+HRAAlJ3Jk+1k2XcZAnICzy++Ow==", 2024-02-19T15:22:52.583 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.583 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:22:52.583 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:22:52.584 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:22:52.584 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:22:52.584 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.584 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.584 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.584 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.b", 2024-02-19T15:22:52.584 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlIOh3KhAAlnU2ZFY4iv/XEw0WqxyZpw==", 2024-02-19T15:22:52.585 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.585 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:22:52.585 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:22:52.585 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:22:52.585 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:22:52.585 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.586 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.586 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.586 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.c", 2024-02-19T15:22:52.586 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNl48SHJRAAn4GCT5KnP+2PD8IXbGClCg==", 2024-02-19T15:22:52.586 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.586 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:22:52.586 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:22:52.587 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:22:52.587 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:22:52.587 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.587 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.587 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.587 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.d", 2024-02-19T15:22:52.587 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNljcekMRAA414655SzH43fBA0AtjpxZw==", 2024-02-19T15:22:52.588 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.588 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:22:52.588 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:22:52.588 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:22:52.588 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:22:52.588 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.588 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.589 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.589 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.0", 2024-02-19T15:22:52.589 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDlcdNl/bycFhAAmzYi/OxEPUW3tcRJC0DNEQ==", 2024-02-19T15:22:52.589 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.589 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:22:52.589 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:22:52.589 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:22:52.590 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.590 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.590 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.590 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.1", 2024-02-19T15:22:52.590 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDocdNluquhHBAA88aLHCvVol7JoZB0MtoutA==", 2024-02-19T15:22:52.590 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.590 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:22:52.591 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:22:52.591 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:22:52.591 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.591 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.591 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.591 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.2", 2024-02-19T15:22:52.591 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDrcdNlGAnWIhAAyELvCPKjtfgF0+kX6NQFIA==", 2024-02-19T15:22:52.592 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.592 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:22:52.592 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:22:52.592 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:22:52.592 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.592 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.592 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.593 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.3", 2024-02-19T15:22:52.593 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDucdNlROZ6KBAAxTCDuYsw5SRIZBcclzwxhg==", 2024-02-19T15:22:52.593 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.593 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:22:52.593 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:22:52.593 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:22:52.593 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.594 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.594 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.594 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.4", 2024-02-19T15:22:52.594 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD0cdNluOGILBAA66R9RVXD2vdyIw+oZ+iM1g==", 2024-02-19T15:22:52.594 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.594 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:22:52.595 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:22:52.595 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:22:52.595 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.595 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.595 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.595 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.5", 2024-02-19T15:22:52.595 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD3cdNlHg8dMhAA2LUzfpvvHR4i8e5RBCeGpQ==", 2024-02-19T15:22:52.596 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.596 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:22:52.596 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:22:52.596 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:22:52.596 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.596 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.596 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.597 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.6", 2024-02-19T15:22:52.597 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD6cdNlXD1EORAAhKGIt0wFeCyoE6QkkZ3caA==", 2024-02-19T15:22:52.597 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.597 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:22:52.597 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:22:52.597 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:22:52.597 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.598 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.598 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.598 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.7", 2024-02-19T15:22:52.598 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD+cdNlfW65AhAAVWj3ptC8uQHPh/UXcb/uDg==", 2024-02-19T15:22:52.598 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.598 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:22:52.599 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:22:52.599 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:22:52.599 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.599 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.599 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.599 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.0", 2024-02-19T15:22:52.599 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlb7eRNRAAk6ZLGfjhcdlR9GoXFCd9eA==", 2024-02-19T15:22:52.600 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.600 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:22:52.600 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:22:52.600 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:22:52.600 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:22:52.601 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.601 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.601 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.601 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.1", 2024-02-19T15:22:52.601 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlKBu+AxAAlLRU70SAs3ZJo5wFwRWa4w==", 2024-02-19T15:22:52.602 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.602 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:22:52.602 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:22:52.602 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:22:52.602 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:22:52.602 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.603 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.603 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.603 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.2", 2024-02-19T15:22:52.603 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlyDk9ABAAIAjKSd4qJO5oHhVnDc1Rgw==", 2024-02-19T15:22:52.603 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.603 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:22:52.603 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:22:52.604 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:22:52.604 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:22:52.604 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.604 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.604 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.605 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.3", 2024-02-19T15:22:52.605 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlUqK3CRAABmSIzoJvl0ymME6swrvnSQ==", 2024-02-19T15:22:52.605 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.605 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:22:52.605 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:22:52.605 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:22:52.605 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:22:52.606 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.606 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.606 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.606 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.admin", 2024-02-19T15:22:52.606 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNls3vPAhAA4x2yQTUCz4Dv9vqEWnSTyQ==", 2024-02-19T15:22:52.606 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.607 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:22:52.607 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:22:52.607 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:22:52.607 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:22:52.607 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.607 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.608 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.608 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.bootstrap-mds", 2024-02-19T15:22:52.608 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQAEctNlGWMjAhAAjjHmgGzcySzF6/QK2wCRCg==", 2024-02-19T15:22:52.608 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.609 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile bootstrap-mds" 2024-02-19T15:22:52.609 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.609 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.609 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.609 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.bootstrap-mgr", 2024-02-19T15:22:52.610 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQAEctNl1WwjAhAAx1Wre/WvDIShvhRVFSqt4Q==", 2024-02-19T15:22:52.610 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.610 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile bootstrap-mgr" 2024-02-19T15:22:52.610 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.610 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.611 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.611 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.bootstrap-osd", 2024-02-19T15:22:52.611 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQAEctNlGnUjAhAAAUzDL3pV2WfNF5ONoGcVSA==", 2024-02-19T15:22:52.611 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.612 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile bootstrap-osd" 2024-02-19T15:22:52.612 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.612 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.612 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.612 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.bootstrap-rbd", 2024-02-19T15:22:52.613 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQAEctNlPX0jAhAA3QVKZ/7gglPi5rOVTRRJ2Q==", 2024-02-19T15:22:52.613 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.613 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile bootstrap-rbd" 2024-02-19T15:22:52.613 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.614 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.614 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.614 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-02-19T15:22:52.614 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQAEctNloIUjAhAAlE7fv/pduTtDMVvmIU60jQ==", 2024-02-19T15:22:52.614 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.615 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-02-19T15:22:52.615 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.615 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.615 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.615 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.bootstrap-rgw", 2024-02-19T15:22:52.616 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQAEctNlJo4jAhAAqI3Pfgl2saBwbEJcuQ1P/A==", 2024-02-19T15:22:52.616 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.616 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile bootstrap-rgw" 2024-02-19T15:22:52.616 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.617 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.617 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.617 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mgr.x", 2024-02-19T15:22:52.617 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNliM78GRAABWBg4hMcKjV2pK5WVBjLMQ==", 2024-02-19T15:22:52.617 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.618 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:22:52.618 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile mgr", 2024-02-19T15:22:52.618 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:22:52.618 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.618 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:22:52.619 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:52.619 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mgr.y", 2024-02-19T15:22:52.619 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlwxs9FhAAWqP7YoKio/bbdx7SGiCJ9Q==", 2024-02-19T15:22:52.619 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:22:52.619 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:22:52.620 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile mgr", 2024-02-19T15:22:52.620 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:22:52.620 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.620 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:52.620 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-02-19T15:22:52.621 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:22:52.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.580+0000 7fd956a00640 1 -- 172.21.15.47:0/3202446402 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd92c066e70 msgr2=0x7fd92c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:52.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.580+0000 7fd956a00640 1 --2- 172.21.15.47:0/3202446402 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd92c066e70 0x7fd92c069330 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd9400af000 tx=0x7fd9400a7000 comp rx=0 tx=0).stop 2024-02-19T15:22:52.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.580+0000 7fd956a00640 1 -- 172.21.15.47:0/3202446402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd950071350 msgr2=0x7fd9500a9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:52.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.580+0000 7fd956a00640 1 --2- 172.21.15.47:0/3202446402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd950071350 0x7fd9500a9430 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd93800e930 tx=0x7fd93800edf0 comp rx=0 tx=0).stop 2024-02-19T15:22:52.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.580+0000 7fd956a00640 1 -- 172.21.15.47:0/3202446402 shutdown_connections 2024-02-19T15:22:52.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.580+0000 7fd956a00640 1 --2- 172.21.15.47:0/3202446402 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd92c066e70 0x7fd92c069330 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:52.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.580+0000 7fd956a00640 1 --2- 172.21.15.47:0/3202446402 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd95015b1a0 0x7fd95014e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:52.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.580+0000 7fd956a00640 1 --2- 172.21.15.47:0/3202446402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd950071350 0x7fd9500a9430 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:52.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.580+0000 7fd956a00640 1 --2- 172.21.15.47:0/3202446402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd95006c9a0 0x7fd9500b47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:52.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.580+0000 7fd956a00640 1 -- 172.21.15.47:0/3202446402 >> 172.21.15.47:0/3202446402 conn(0x7fd9500b16a0 msgr2=0x7fd95014f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:52.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.580+0000 7fd956a00640 1 -- 172.21.15.47:0/3202446402 shutdown_connections 2024-02-19T15:22:52.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.580+0000 7fd956a00640 1 -- 172.21.15.47:0/3202446402 wait complete. 2024-02-19T15:22:52.627 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-02-19T15:22:52.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.764+0000 7f9c44000640 1 Processor -- start 2024-02-19T15:22:52.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.764+0000 7f9c44000640 1 -- start start 2024-02-19T15:22:52.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.764+0000 7f9c44000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9c3c15caa0 0x7f9c3c0727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:52.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.764+0000 7f9c44000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c3c15d280 con 0x7f9c3c0cea80 2024-02-19T15:22:52.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.764+0000 7f9c44000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c3c070ff0 con 0x7f9c3c145590 2024-02-19T15:22:52.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.764+0000 7f9c44000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c3c0709e0 con 0x7f9c3c15caa0 2024-02-19T15:22:52.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.764+0000 7f9c40c00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f9c3c0cea80 0x7f9c3c15c390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57214/0 (socket says 172.21.15.47:57214) 2024-02-19T15:22:52.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.764+0000 7f9c40c00640 1 -- 172.21.15.47:0/3233922710 learned_addr learned my addr 172.21.15.47:0/3233922710 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.764+0000 7f9c42000640 1 --2- 172.21.15.47:0/3233922710 >> v2:172.21.15.116:3300/0 conn(0x7f9c3c15caa0 0x7f9c3c0727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.764+0000 7f9c3be00640 1 -- 172.21.15.47:0/3233922710 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128216865 0 0) 0x7f9c3c15d280 con 0x7f9c3c0cea80 2024-02-19T15:22:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.764+0000 7f9c3be00640 1 -- 172.21.15.47:0/3233922710 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c24003530 con 0x7f9c3c0cea80 2024-02-19T15:22:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.768+0000 7f9c42000640 1 -- 172.21.15.47:0/3233922710 >> v1:172.21.15.116:6789/0 conn(0x7f9c3c145590 legacy=0x7f9c3c06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.768+0000 7f9c42000640 1 -- 172.21.15.47:0/3233922710 >> v1:172.21.15.47:6789/0 conn(0x7f9c3c0cea80 legacy=0x7f9c3c15c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.768+0000 7f9c3be00640 1 -- 172.21.15.47:0/3233922710 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4104106510 0 0) 0x7f9c24003530 con 0x7f9c3c0cea80 2024-02-19T15:22:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.768+0000 7f9c42000640 1 -- 172.21.15.47:0/3233922710 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c3c0cf020 con 0x7f9c3c15caa0 2024-02-19T15:22:52.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.768+0000 7f9c42000640 1 --2- 172.21.15.47:0/3233922710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c15caa0 0x7f9c3c0727b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9c34009d90 tx=0x7f9c3401c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e786e42673c16641 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:52.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.768+0000 7f9c3be00640 1 -- 172.21.15.47:0/3233922710 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c28001680 con 0x7f9c3c15caa0 2024-02-19T15:22:52.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.768+0000 7f9c3be00640 1 -- 172.21.15.47:0/3233922710 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c340a55e0 con 0x7f9c3c15caa0 2024-02-19T15:22:52.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.768+0000 7f9c3be00640 1 -- 172.21.15.47:0/3233922710 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c340a5970 con 0x7f9c3c15caa0 2024-02-19T15:22:52.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.768+0000 7f9c44000640 1 -- 172.21.15.47:0/3233922710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c15caa0 msgr2=0x7f9c3c0727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:52.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.768+0000 7f9c44000640 1 --2- 172.21.15.47:0/3233922710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c15caa0 0x7f9c3c0727b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9c34009d90 tx=0x7f9c3401c890 comp rx=0 tx=0).stop 2024-02-19T15:22:52.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.768+0000 7f9c44000640 1 -- 172.21.15.47:0/3233922710 shutdown_connections 2024-02-19T15:22:52.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.768+0000 7f9c44000640 1 --2- 172.21.15.47:0/3233922710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c15caa0 0x7f9c3c0727b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:52.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.768+0000 7f9c44000640 1 -- 172.21.15.47:0/3233922710 >> 172.21.15.47:0/3233922710 conn(0x7f9c3c0b16a0 msgr2=0x7f9c3c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:52.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.768+0000 7f9c44000640 1 -- 172.21.15.47:0/3233922710 shutdown_connections 2024-02-19T15:22:52.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.768+0000 7f9c44000640 1 -- 172.21.15.47:0/3233922710 wait complete. 2024-02-19T15:22:52.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c44000640 1 Processor -- start 2024-02-19T15:22:52.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c44000640 1 -- start start 2024-02-19T15:22:52.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c44000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c3c0cea80 0x7f9c3c15b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:52.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c44000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c3c145590 0x7f9c3c0ab190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:52.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c44000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c15caa0 0x7f9c3c0ab6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:52.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c44000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c0abce0 con 0x7f9c3c145590 2024-02-19T15:22:52.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c44000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c0abfc0 con 0x7f9c3c15caa0 2024-02-19T15:22:52.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c44000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c0ac2a0 con 0x7f9c3c0cea80 2024-02-19T15:22:52.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c40c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c3c145590 0x7f9c3c0ab190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:52.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c40c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c3c145590 0x7f9c3c0ab190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36434/0 (socket says 172.21.15.47:36434) 2024-02-19T15:22:52.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c40c00640 1 -- 172.21.15.47:0/4244649685 learned_addr learned my addr 172.21.15.47:0/4244649685 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:52.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c42000640 1 --2- 172.21.15.47:0/4244649685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c15caa0 0x7f9c3c0ab6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:52.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c41600640 1 --2- 172.21.15.47:0/4244649685 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c3c0cea80 0x7f9c3c15b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:52.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c40c00640 1 -- 172.21.15.47:0/4244649685 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c3c0cea80 msgr2=0x7f9c3c15b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:52.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c40c00640 1 --2- 172.21.15.47:0/4244649685 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c3c0cea80 0x7f9c3c15b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:52.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c40c00640 1 -- 172.21.15.47:0/4244649685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c15caa0 msgr2=0x7f9c3c0ab6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:52.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c40c00640 1 --2- 172.21.15.47:0/4244649685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c15caa0 0x7f9c3c0ab6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:52.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c40c00640 1 -- 172.21.15.47:0/4244649685 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c34009a70 con 0x7f9c3c145590 2024-02-19T15:22:52.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c42000640 1 --2- 172.21.15.47:0/4244649685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c15caa0 0x7f9c3c0ab6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:52.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c40c00640 1 --2- 172.21.15.47:0/4244649685 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c3c145590 0x7f9c3c0ab190 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9c280011f0 tx=0x7f9c2800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:52.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c39600640 1 -- 172.21.15.47:0/4244649685 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c28005ae0 con 0x7f9c3c145590 2024-02-19T15:22:52.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c39600640 1 -- 172.21.15.47:0/4244649685 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c28005c40 con 0x7f9c3c145590 2024-02-19T15:22:52.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c39600640 1 -- 172.21.15.47:0/4244649685 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c280072b0 con 0x7f9c3c145590 2024-02-19T15:22:52.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c44000640 1 -- 172.21.15.47:0/4244649685 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c3c0ac580 con 0x7f9c3c145590 2024-02-19T15:22:52.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.772+0000 7f9c44000640 1 -- 172.21.15.47:0/4244649685 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c3c15b940 con 0x7f9c3c145590 2024-02-19T15:22:52.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.776+0000 7f9c44000640 1 -- 172.21.15.47:0/4244649685 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c3c0723d0 con 0x7f9c3c145590 2024-02-19T15:22:52.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.776+0000 7f9c39600640 1 -- 172.21.15.47:0/4244649685 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9c28007410 con 0x7f9c3c145590 2024-02-19T15:22:52.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.776+0000 7f9c39600640 1 --2- 172.21.15.47:0/4244649685 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9c1c066e40 0x7f9c1c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:52.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.776+0000 7f9c39600640 1 -- 172.21.15.47:0/4244649685 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9c28083dd0 con 0x7f9c3c145590 2024-02-19T15:22:52.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.776+0000 7f9c41600640 1 --2- 172.21.15.47:0/4244649685 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9c1c066e40 0x7f9c1c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:52.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.776+0000 7f9c41600640 1 --2- 172.21.15.47:0/4244649685 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9c1c066e40 0x7f9c1c069300 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c144130 tx=0x7f9c30009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:52.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:52.780+0000 7f9c39600640 1 -- 172.21.15.47:0/4244649685 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f9c28050fb0 con 0x7f9c3c145590 2024-02-19T15:22:53.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.000+0000 7f9c44000640 1 -- 172.21.15.47:0/4244649685 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f9c3c075360 con 0x7f9c3c145590 2024-02-19T15:22:53.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.000+0000 7f9c39600640 1 -- 172.21.15.47:0/4244649685 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f9c280170a0 con 0x7f9c3c145590 2024-02-19T15:22:53.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.004+0000 7f9c44000640 1 -- 172.21.15.47:0/4244649685 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9c1c066e40 msgr2=0x7f9c1c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:53.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.004+0000 7f9c44000640 1 --2- 172.21.15.47:0/4244649685 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9c1c066e40 0x7f9c1c069300 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c144130 tx=0x7f9c30009040 comp rx=0 tx=0).stop 2024-02-19T15:22:53.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.004+0000 7f9c44000640 1 -- 172.21.15.47:0/4244649685 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c3c145590 msgr2=0x7f9c3c0ab190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:53.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.004+0000 7f9c44000640 1 --2- 172.21.15.47:0/4244649685 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c3c145590 0x7f9c3c0ab190 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9c280011f0 tx=0x7f9c2800cbe0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.004+0000 7f9c44000640 1 -- 172.21.15.47:0/4244649685 shutdown_connections 2024-02-19T15:22:53.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.004+0000 7f9c44000640 1 --2- 172.21.15.47:0/4244649685 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9c1c066e40 0x7f9c1c069300 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.004+0000 7f9c44000640 1 --2- 172.21.15.47:0/4244649685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c15caa0 0x7f9c3c0ab6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.004+0000 7f9c44000640 1 --2- 172.21.15.47:0/4244649685 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c3c145590 0x7f9c3c0ab190 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.004+0000 7f9c44000640 1 --2- 172.21.15.47:0/4244649685 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c3c0cea80 0x7f9c3c15b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.004+0000 7f9c44000640 1 -- 172.21.15.47:0/4244649685 >> 172.21.15.47:0/4244649685 conn(0x7f9c3c0b16a0 msgr2=0x7f9c3c1420a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:53.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.004+0000 7f9c44000640 1 -- 172.21.15.47:0/4244649685 shutdown_connections 2024-02-19T15:22:53.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.004+0000 7f9c44000640 1 -- 172.21.15.47:0/4244649685 wait complete. 2024-02-19T15:22:53.027 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-02-19T15:22:53.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.168+0000 7fdebe400640 1 Processor -- start 2024-02-19T15:22:53.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.168+0000 7fdebe400640 1 -- start start 2024-02-19T15:22:53.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.168+0000 7fdebe400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdeb81613a0 0x7fdeb80adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:53.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.168+0000 7fdebe400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdeb806e7c0 0x7fdeb806aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:53.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.168+0000 7fdebe400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb8161780 con 0x7fdeb806fdf0 2024-02-19T15:22:53.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.168+0000 7fdebe400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb80b01a0 con 0x7fdeb81613a0 2024-02-19T15:22:53.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.168+0000 7fdebe400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb80ae3c0 con 0x7fdeb806e7c0 2024-02-19T15:22:53.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.168+0000 7fdeb7400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdeb81613a0 0x7fdeb80adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:53.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.168+0000 7fdeb7400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdeb81613a0 0x7fdeb80adbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36444/0 (socket says 172.21.15.47:36444) 2024-02-19T15:22:53.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.168+0000 7fdeb7400640 1 -- 172.21.15.47:0/2420358309 learned_addr learned my addr 172.21.15.47:0/2420358309 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:53.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.168+0000 7fdeb7e00640 1 --2- 172.21.15.47:0/2420358309 >> v2:172.21.15.116:3300/0 conn(0x7fdeb806e7c0 0x7fdeb806aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:53.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.168+0000 7fdeb7400640 1 -- 172.21.15.47:0/2420358309 >> v2:172.21.15.116:3300/0 conn(0x7fdeb806e7c0 msgr2=0x7fdeb806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:53.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.168+0000 7fdeb7400640 1 --2- 172.21.15.47:0/2420358309 >> v2:172.21.15.116:3300/0 conn(0x7fdeb806e7c0 0x7fdeb806aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.168+0000 7fdeb7400640 1 -- 172.21.15.47:0/2420358309 >> v1:172.21.15.116:6789/0 conn(0x7fdeb806fdf0 legacy=0x7fdeb806e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:53.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.168+0000 7fdeb7400640 1 -- 172.21.15.47:0/2420358309 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb80ae6a0 con 0x7fdeb81613a0 2024-02-19T15:22:53.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.168+0000 7fdeb7e00640 1 --2- 172.21.15.47:0/2420358309 >> v2:172.21.15.116:3300/0 conn(0x7fdeb806e7c0 0x7fdeb806aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:53.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.172+0000 7fdeb7400640 1 --2- 172.21.15.47:0/2420358309 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdeb81613a0 0x7fdeb80adbb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8068400 tx=0x7fdea801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f210e35e1d3e9d9b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:53.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.172+0000 7fdeb6000640 1 -- 172.21.15.47:0/2420358309 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea80ac600 con 0x7fdeb81613a0 2024-02-19T15:22:53.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.172+0000 7fdeb6000640 1 -- 172.21.15.47:0/2420358309 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea80a8050 con 0x7fdeb81613a0 2024-02-19T15:22:53.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.172+0000 7fdeb6000640 1 -- 172.21.15.47:0/2420358309 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea80ac980 con 0x7fdeb81613a0 2024-02-19T15:22:53.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.172+0000 7fdebe400640 1 -- 172.21.15.47:0/2420358309 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdeb81613a0 msgr2=0x7fdeb80adbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:53.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.172+0000 7fdebe400640 1 --2- 172.21.15.47:0/2420358309 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdeb81613a0 0x7fdeb80adbb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8068400 tx=0x7fdea801c490 comp rx=0 tx=0).stop 2024-02-19T15:22:53.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.172+0000 7fdebe400640 1 -- 172.21.15.47:0/2420358309 shutdown_connections 2024-02-19T15:22:53.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.172+0000 7fdebe400640 1 --2- 172.21.15.47:0/2420358309 >> v2:172.21.15.116:3300/0 conn(0x7fdeb806e7c0 0x7fdeb806aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.172+0000 7fdebe400640 1 --2- 172.21.15.47:0/2420358309 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdeb81613a0 0x7fdeb80adbb0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.172+0000 7fdebe400640 1 -- 172.21.15.47:0/2420358309 >> 172.21.15.47:0/2420358309 conn(0x7fdeb80d0ad0 msgr2=0x7fdeb80d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:53.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.172+0000 7fdebe400640 1 -- 172.21.15.47:0/2420358309 shutdown_connections 2024-02-19T15:22:53.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.172+0000 7fdebe400640 1 -- 172.21.15.47:0/2420358309 wait complete. 2024-02-19T15:22:53.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdebe400640 1 Processor -- start 2024-02-19T15:22:53.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdebe400640 1 -- start start 2024-02-19T15:22:53.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdebe400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdeb806e7c0 0x7fdeb8150e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:53.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdebe400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdeb806fdf0 0x7fdeb8151350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:53.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdebe400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdeb81613a0 0x7fdeb814ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:53.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdebe400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb80cd300 con 0x7fdeb806e7c0 2024-02-19T15:22:53.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdebe400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb80cd5e0 con 0x7fdeb81613a0 2024-02-19T15:22:53.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdebe400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb80cd8c0 con 0x7fdeb806fdf0 2024-02-19T15:22:53.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdeb7400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdeb806e7c0 0x7fdeb8150e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:53.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdeb7400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdeb806e7c0 0x7fdeb8150e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36448/0 (socket says 172.21.15.47:36448) 2024-02-19T15:22:53.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdeb7400640 1 -- 172.21.15.47:0/3505884798 learned_addr learned my addr 172.21.15.47:0/3505884798 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:53.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdeb7400640 1 -- 172.21.15.47:0/3505884798 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdeb806fdf0 msgr2=0x7fdeb8151350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:53.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdeb7400640 1 --2- 172.21.15.47:0/3505884798 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdeb806fdf0 0x7fdeb8151350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdeb7400640 1 -- 172.21.15.47:0/3505884798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdeb81613a0 msgr2=0x7fdeb814ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:53.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdeb7400640 1 --2- 172.21.15.47:0/3505884798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdeb81613a0 0x7fdeb814ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdeb7400640 1 -- 172.21.15.47:0/3505884798 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea80094d0 con 0x7fdeb806e7c0 2024-02-19T15:22:53.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdeb7400640 1 --2- 172.21.15.47:0/3505884798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdeb806e7c0 0x7fdeb8150e10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fdea80190a0 tx=0x7fdea801acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:53.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdeaf400640 1 -- 172.21.15.47:0/3505884798 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea80b7620 con 0x7fdeb806e7c0 2024-02-19T15:22:53.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdeaf400640 1 -- 172.21.15.47:0/3505884798 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea80b7780 con 0x7fdeb806e7c0 2024-02-19T15:22:53.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdeaf400640 1 -- 172.21.15.47:0/3505884798 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea80b7af0 con 0x7fdeb806e7c0 2024-02-19T15:22:53.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdebe400640 1 -- 172.21.15.47:0/3505884798 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb80cdba0 con 0x7fdeb806e7c0 2024-02-19T15:22:53.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdebe400640 1 -- 172.21.15.47:0/3505884798 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb80ce3a0 con 0x7fdeb806e7c0 2024-02-19T15:22:53.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.176+0000 7fdebe400640 1 -- 172.21.15.47:0/3505884798 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb81549d0 con 0x7fdeb806e7c0 2024-02-19T15:22:53.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.180+0000 7fdeaf400640 1 -- 172.21.15.47:0/3505884798 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fdea80b3040 con 0x7fdeb806e7c0 2024-02-19T15:22:53.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.180+0000 7fdeaf400640 1 --2- 172.21.15.47:0/3505884798 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fde98066e40 0x7fde98069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:53.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.180+0000 7fdeaf400640 1 -- 172.21.15.47:0/3505884798 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdea811e880 con 0x7fdeb806e7c0 2024-02-19T15:22:53.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.180+0000 7fdeb6a00640 1 --2- 172.21.15.47:0/3505884798 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fde98066e40 0x7fde98069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:53.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.180+0000 7fdeb6a00640 1 --2- 172.21.15.47:0/3505884798 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fde98066e40 0x7fde98069300 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdea4000f70 tx=0x7fdea40079e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:53.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.184+0000 7fdeaf400640 1 -- 172.21.15.47:0/3505884798 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fdea80eb9b0 con 0x7fdeb806e7c0 2024-02-19T15:22:53.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.404+0000 7fdebe400640 1 -- 172.21.15.47:0/3505884798 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7fdeb8075360 con 0x7fdeb806e7c0 2024-02-19T15:22:53.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.408+0000 7fdeaf400640 1 -- 172.21.15.47:0/3505884798 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7fdea80ef070 con 0x7fdeb806e7c0 2024-02-19T15:22:53.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.408+0000 7fdebe400640 1 -- 172.21.15.47:0/3505884798 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fde98066e40 msgr2=0x7fde98069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:53.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.408+0000 7fdebe400640 1 --2- 172.21.15.47:0/3505884798 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fde98066e40 0x7fde98069300 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdea4000f70 tx=0x7fdea40079e0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.408+0000 7fdebe400640 1 -- 172.21.15.47:0/3505884798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdeb806e7c0 msgr2=0x7fdeb8150e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:53.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.408+0000 7fdebe400640 1 --2- 172.21.15.47:0/3505884798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdeb806e7c0 0x7fdeb8150e10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fdea80190a0 tx=0x7fdea801acd0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.408+0000 7fdebe400640 1 -- 172.21.15.47:0/3505884798 shutdown_connections 2024-02-19T15:22:53.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.408+0000 7fdebe400640 1 --2- 172.21.15.47:0/3505884798 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fde98066e40 0x7fde98069300 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.408+0000 7fdebe400640 1 --2- 172.21.15.47:0/3505884798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdeb81613a0 0x7fdeb814ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.408+0000 7fdebe400640 1 --2- 172.21.15.47:0/3505884798 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdeb806fdf0 0x7fdeb8151350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.408+0000 7fdebe400640 1 --2- 172.21.15.47:0/3505884798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdeb806e7c0 0x7fdeb8150e10 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.408+0000 7fdebe400640 1 -- 172.21.15.47:0/3505884798 >> 172.21.15.47:0/3505884798 conn(0x7fdeb80d0ad0 msgr2=0x7fdeb80b3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:53.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.408+0000 7fdebe400640 1 -- 172.21.15.47:0/3505884798 shutdown_connections 2024-02-19T15:22:53.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.408+0000 7fdebe400640 1 -- 172.21.15.47:0/3505884798 wait complete. 2024-02-19T15:22:53.432 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-02-19T15:22:53.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.568+0000 7fd0e6800640 1 Processor -- start 2024-02-19T15:22:53.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.568+0000 7fd0e6800640 1 -- start start 2024-02-19T15:22:53.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.568+0000 7fd0e6800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e00cded0 0x7fd0e00ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:53.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.568+0000 7fd0e6800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd0e00ceb40 0x7fd0e006bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:53.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.568+0000 7fd0e6800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e00cce30 con 0x7fd0e006d650 2024-02-19T15:22:53.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.568+0000 7fd0e6800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0e00cefd0 con 0x7fd0e00ceb40 2024-02-19T15:22:53.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.568+0000 7fd0e6800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e00cd110 con 0x7fd0e00cded0 2024-02-19T15:22:53.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.568+0000 7fd0df400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd0e006d650 0x7fd0e00d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53788/0 (socket says 172.21.15.47:53788) 2024-02-19T15:22:53.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.568+0000 7fd0df400640 1 -- 172.21.15.47:0/523881409 learned_addr learned my addr 172.21.15.47:0/523881409 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:53.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0d6a00640 1 --2- 172.21.15.47:0/523881409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e00cded0 0x7fd0e00ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:53.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dfe00640 1 --2- 172.21.15.47:0/523881409 >> v2:172.21.15.116:3300/0 conn(0x7fd0e00ceb40 0x7fd0e006bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:53.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dea00640 1 -- 172.21.15.47:0/523881409 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3622258820 0 0) 0x7fd0e00cce30 con 0x7fd0e006d650 2024-02-19T15:22:53.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dea00640 1 -- 172.21.15.47:0/523881409 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0c4003530 con 0x7fd0e006d650 2024-02-19T15:22:53.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dea00640 1 -- 172.21.15.47:0/523881409 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2559648025 0 0) 0x7fd0c4003530 con 0x7fd0e006d650 2024-02-19T15:22:53.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dea00640 1 -- 172.21.15.47:0/523881409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e00cded0 msgr2=0x7fd0e00ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:53.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dea00640 1 --2- 172.21.15.47:0/523881409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e00cded0 0x7fd0e00ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dea00640 1 -- 172.21.15.47:0/523881409 >> v2:172.21.15.116:3300/0 conn(0x7fd0e00ceb40 msgr2=0x7fd0e006bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:53.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dea00640 1 --2- 172.21.15.47:0/523881409 >> v2:172.21.15.116:3300/0 conn(0x7fd0e00ceb40 0x7fd0e006bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dea00640 1 -- 172.21.15.47:0/523881409 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e00cd3f0 con 0x7fd0e006d650 2024-02-19T15:22:53.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dea00640 1 -- 172.21.15.47:0/523881409 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7fd0d0001680 con 0x7fd0e006d650 2024-02-19T15:22:53.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dea00640 1 -- 172.21.15.47:0/523881409 >> v1:172.21.15.47:6789/0 conn(0x7fd0e006d650 legacy=0x7fd0e00d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:53.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dea00640 1 --2- 172.21.15.47:0/523881409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e00cded0 0x7fd0c4003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:53.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dea00640 1 --2- 172.21.15.47:0/523881409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0c4003f00 0x7fd0c40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:53.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dea00640 1 --2- 172.21.15.47:0/523881409 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0c4004d10 0x7fd0c4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:53.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dea00640 1 -- 172.21.15.47:0/523881409 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e00cefd0 con 0x7fd0c4004d10 2024-02-19T15:22:53.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dea00640 1 -- 172.21.15.47:0/523881409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e00cd110 con 0x7fd0c4003f00 2024-02-19T15:22:53.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dea00640 1 -- 172.21.15.47:0/523881409 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c400b9e0 con 0x7fd0e00cded0 2024-02-19T15:22:53.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dfe00640 1 --2- 172.21.15.47:0/523881409 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0c4004d10 0x7fd0c4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:53.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0d6a00640 1 --2- 172.21.15.47:0/523881409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0c4003f00 0x7fd0c40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:53.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0df400640 1 --2- 172.21.15.47:0/523881409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e00cded0 0x7fd0c4003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:53.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dfe00640 1 -- 172.21.15.47:0/523881409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e00cded0 msgr2=0x7fd0c4003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:53.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dfe00640 1 --2- 172.21.15.47:0/523881409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e00cded0 0x7fd0c4003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dfe00640 1 -- 172.21.15.47:0/523881409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0c4003f00 msgr2=0x7fd0c40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:53.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dfe00640 1 --2- 172.21.15.47:0/523881409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0c4003f00 0x7fd0c40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0dfe00640 1 -- 172.21.15.47:0/523881409 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c4003530 con 0x7fd0c4004d10 2024-02-19T15:22:53.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.572+0000 7fd0d6a00640 1 --2- 172.21.15.47:0/523881409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0c4003f00 0x7fd0c40047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:53.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0dfe00640 1 --2- 172.21.15.47:0/523881409 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0c4004d10 0x7fd0c4005190 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800e170 tx=0x7fd0d801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=635e4fbdfc12f518 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:53.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0dea00640 1 -- 172.21.15.47:0/523881409 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d80a5ba0 con 0x7fd0c4004d10 2024-02-19T15:22:53.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0dea00640 1 -- 172.21.15.47:0/523881409 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d800f040 con 0x7fd0c4004d10 2024-02-19T15:22:53.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0dea00640 1 -- 172.21.15.47:0/523881409 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d80ad440 con 0x7fd0c4004d10 2024-02-19T15:22:53.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0e6800640 1 -- 172.21.15.47:0/523881409 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0c4004d10 msgr2=0x7fd0c4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:53.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0e6800640 1 --2- 172.21.15.47:0/523881409 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0c4004d10 0x7fd0c4005190 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800e170 tx=0x7fd0d801cca0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0dfe00640 1 -- 172.21.15.47:0/523881409 reap_dead start 2024-02-19T15:22:53.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0e6800640 1 -- 172.21.15.47:0/523881409 shutdown_connections 2024-02-19T15:22:53.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0e6800640 1 -- 172.21.15.47:0/523881409 >> 172.21.15.47:0/523881409 conn(0x7fd0e00d1bf0 msgr2=0x7fd0e00d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:53.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0e6800640 1 -- 172.21.15.47:0/523881409 shutdown_connections 2024-02-19T15:22:53.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0e6800640 1 -- 172.21.15.47:0/523881409 wait complete. 2024-02-19T15:22:53.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0e6800640 1 Processor -- start 2024-02-19T15:22:53.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0e6800640 1 -- start start 2024-02-19T15:22:53.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0e6800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0c4004d10 0x7fd0e0154f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:53.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0e6800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0e006f090 0x7fd0e006f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:53.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0e6800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0e00c7b50 0x7fd0e00c7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:53.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0e6800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e006fa50 con 0x7fd0e00c7b50 2024-02-19T15:22:53.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0e6800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e00c6060 con 0x7fd0e006f090 2024-02-19T15:22:53.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0e6800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e00c6be0 con 0x7fd0c4004d10 2024-02-19T15:22:53.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0dfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0e00c7b50 0x7fd0e00c7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:53.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0dfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0e00c7b50 0x7fd0e00c7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39312/0 (socket says 172.21.15.47:39312) 2024-02-19T15:22:53.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.576+0000 7fd0dfe00640 1 -- 172.21.15.47:0/608553406 learned_addr learned my addr 172.21.15.47:0/608553406 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:53.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.580+0000 7fd0d6a00640 1 --2- 172.21.15.47:0/608553406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0e006f090 0x7fd0e006f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:53.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.580+0000 7fd0df400640 1 --2- 172.21.15.47:0/608553406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0c4004d10 0x7fd0e0154f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:53.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.580+0000 7fd0dfe00640 1 -- 172.21.15.47:0/608553406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0c4004d10 msgr2=0x7fd0e0154f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:53.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.580+0000 7fd0dfe00640 1 --2- 172.21.15.47:0/608553406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0c4004d10 0x7fd0e0154f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.580+0000 7fd0dfe00640 1 -- 172.21.15.47:0/608553406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0e006f090 msgr2=0x7fd0e006f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:53.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.580+0000 7fd0dfe00640 1 --2- 172.21.15.47:0/608553406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0e006f090 0x7fd0e006f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.580+0000 7fd0dfe00640 1 -- 172.21.15.47:0/608553406 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e01554c0 con 0x7fd0e00c7b50 2024-02-19T15:22:53.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.580+0000 7fd0d6a00640 1 --2- 172.21.15.47:0/608553406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0e006f090 0x7fd0e006f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:22:53.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.580+0000 7fd0df400640 1 --2- 172.21.15.47:0/608553406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0c4004d10 0x7fd0e0154f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:22:53.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.580+0000 7fd0dfe00640 1 --2- 172.21.15.47:0/608553406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0e00c7b50 0x7fd0e00c7fd0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800e170 tx=0x7fd0d8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:53.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.580+0000 7fd0d7e00640 1 -- 172.21.15.47:0/608553406 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d80bc070 con 0x7fd0e00c7b50 2024-02-19T15:22:53.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.580+0000 7fd0d7e00640 1 -- 172.21.15.47:0/608553406 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d80039a0 con 0x7fd0e00c7b50 2024-02-19T15:22:53.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.580+0000 7fd0d7e00640 1 -- 172.21.15.47:0/608553406 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d80a5ea0 con 0x7fd0e00c7b50 2024-02-19T15:22:53.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.580+0000 7fd0e6800640 1 -- 172.21.15.47:0/608553406 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e00c62b0 con 0x7fd0e00c7b50 2024-02-19T15:22:53.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.580+0000 7fd0e6800640 1 -- 172.21.15.47:0/608553406 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e00c6870 con 0x7fd0e00c7b50 2024-02-19T15:22:53.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.580+0000 7fd0e6800640 1 -- 172.21.15.47:0/608553406 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e0075360 con 0x7fd0e00c7b50 2024-02-19T15:22:53.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.580+0000 7fd0d7e00640 1 -- 172.21.15.47:0/608553406 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd0d80b4040 con 0x7fd0e00c7b50 2024-02-19T15:22:53.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.580+0000 7fd0d7e00640 1 --2- 172.21.15.47:0/608553406 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd0bc066e90 0x7fd0bc069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:53.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.580+0000 7fd0d7e00640 1 -- 172.21.15.47:0/608553406 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd0d811e4a0 con 0x7fd0e00c7b50 2024-02-19T15:22:53.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.580+0000 7fd0df400640 1 --2- 172.21.15.47:0/608553406 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd0bc066e90 0x7fd0bc069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:53.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.584+0000 7fd0df400640 1 --2- 172.21.15.47:0/608553406 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd0bc066e90 0x7fd0bc069350 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd0d00017b0 tx=0x7fd0d0001c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:53.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.584+0000 7fd0d7e00640 1 -- 172.21.15.47:0/608553406 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd0d80eb5c0 con 0x7fd0e00c7b50 2024-02-19T15:22:53.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.804+0000 7fd0e6800640 1 -- 172.21.15.47:0/608553406 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fd0e00cf580 con 0x7fd0e00c7b50 2024-02-19T15:22:53.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.808+0000 7fd0d7e00640 1 -- 172.21.15.47:0/608553406 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7fd0d80b20a0 con 0x7fd0e00c7b50 2024-02-19T15:22:53.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.808+0000 7fd0e6800640 1 -- 172.21.15.47:0/608553406 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd0bc066e90 msgr2=0x7fd0bc069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:53.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.808+0000 7fd0e6800640 1 --2- 172.21.15.47:0/608553406 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd0bc066e90 0x7fd0bc069350 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd0d00017b0 tx=0x7fd0d0001c90 comp rx=0 tx=0).stop 2024-02-19T15:22:53.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.808+0000 7fd0e6800640 1 -- 172.21.15.47:0/608553406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0e00c7b50 msgr2=0x7fd0e00c7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:53.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.808+0000 7fd0e6800640 1 --2- 172.21.15.47:0/608553406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0e00c7b50 0x7fd0e00c7fd0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800e170 tx=0x7fd0d8009750 comp rx=0 tx=0).stop 2024-02-19T15:22:53.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.808+0000 7fd0e6800640 1 -- 172.21.15.47:0/608553406 shutdown_connections 2024-02-19T15:22:53.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.808+0000 7fd0e6800640 1 --2- 172.21.15.47:0/608553406 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd0bc066e90 0x7fd0bc069350 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.808+0000 7fd0e6800640 1 --2- 172.21.15.47:0/608553406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0e00c7b50 0x7fd0e00c7fd0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.808+0000 7fd0e6800640 1 --2- 172.21.15.47:0/608553406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0e006f090 0x7fd0e006f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.808+0000 7fd0e6800640 1 --2- 172.21.15.47:0/608553406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0c4004d10 0x7fd0e0154f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.808+0000 7fd0e6800640 1 -- 172.21.15.47:0/608553406 >> 172.21.15.47:0/608553406 conn(0x7fd0e00d1bf0 msgr2=0x7fd0e006ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:53.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.808+0000 7fd0e6800640 1 -- 172.21.15.47:0/608553406 shutdown_connections 2024-02-19T15:22:53.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.808+0000 7fd0e6800640 1 -- 172.21.15.47:0/608553406 wait complete. 2024-02-19T15:22:53.833 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-02-19T15:22:53.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.972+0000 7f32f4600640 1 Processor -- start 2024-02-19T15:22:53.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.972+0000 7f32f4600640 1 -- start start 2024-02-19T15:22:53.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.972+0000 7f32f4600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f32ec15b1a0 0x7f32ec15b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:53.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.972+0000 7f32f4600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32ec0718c0 con 0x7f32ec06c9a0 2024-02-19T15:22:53.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.972+0000 7f32f4600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32ec071ba0 con 0x7f32ec071350 2024-02-19T15:22:53.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.972+0000 7f32f4600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32ec071e80 con 0x7f32ec15b1a0 2024-02-19T15:22:53.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.972+0000 7f32f1c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f32ec15b1a0 0x7f32ec15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:53.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.972+0000 7f32f2600640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f32ec06c9a0 0x7f32ec15cde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53798/0 (socket says 172.21.15.47:53798) 2024-02-19T15:22:53.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.972+0000 7f32f2600640 1 -- 172.21.15.47:0/1773425642 learned_addr learned my addr 172.21.15.47:0/1773425642 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:53.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.972+0000 7f32f1c00640 1 -- 172.21.15.47:0/1773425642 >> v1:172.21.15.116:6789/0 conn(0x7f32ec071350 legacy=0x7f32ec15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:53.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.972+0000 7f32ebe00640 1 -- 172.21.15.47:0/1773425642 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158909622 0 0) 0x7f32ec0718c0 con 0x7f32ec06c9a0 2024-02-19T15:22:53.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.972+0000 7f32f1c00640 1 -- 172.21.15.47:0/1773425642 >> v1:172.21.15.47:6789/0 conn(0x7f32ec06c9a0 legacy=0x7f32ec15cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:53.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.972+0000 7f32f1c00640 1 -- 172.21.15.47:0/1773425642 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ec072160 con 0x7f32ec15b1a0 2024-02-19T15:22:53.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.976+0000 7f32f1c00640 1 --2- 172.21.15.47:0/1773425642 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32ec15b1a0 0x7f32ec15b580 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f32e00097f0 tx=0x7f32e001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=153f3f01018610bc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:53.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.976+0000 7f32ebe00640 1 -- 172.21.15.47:0/1773425642 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e00a7070 con 0x7f32ec15b1a0 2024-02-19T15:22:53.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.976+0000 7f32ebe00640 1 -- 172.21.15.47:0/1773425642 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e00ab430 con 0x7f32ec15b1a0 2024-02-19T15:22:53.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.976+0000 7f32ebe00640 1 -- 172.21.15.47:0/1773425642 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e00ab7c0 con 0x7f32ec15b1a0 2024-02-19T15:22:53.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.976+0000 7f32f4600640 1 -- 172.21.15.47:0/1773425642 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32ec15b1a0 msgr2=0x7f32ec15b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:53.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.976+0000 7f32f4600640 1 --2- 172.21.15.47:0/1773425642 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32ec15b1a0 0x7f32ec15b580 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f32e00097f0 tx=0x7f32e001c4a0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.976+0000 7f32f4600640 1 -- 172.21.15.47:0/1773425642 shutdown_connections 2024-02-19T15:22:53.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.976+0000 7f32f4600640 1 --2- 172.21.15.47:0/1773425642 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32ec15b1a0 0x7f32ec15b580 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.976+0000 7f32f4600640 1 -- 172.21.15.47:0/1773425642 >> 172.21.15.47:0/1773425642 conn(0x7f32ec0b16a0 msgr2=0x7f32ec0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:53.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.976+0000 7f32f4600640 1 -- 172.21.15.47:0/1773425642 shutdown_connections 2024-02-19T15:22:53.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.976+0000 7f32f4600640 1 -- 172.21.15.47:0/1773425642 wait complete. 2024-02-19T15:22:53.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f4600640 1 Processor -- start 2024-02-19T15:22:53.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f4600640 1 -- start start 2024-02-19T15:22:53.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f4600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32ec06c9a0 0x7f32ec1f6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:53.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f4600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32ec071350 0x7f32ec1f7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:53.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f4600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32ec15b1a0 0x7f32ec0b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:53.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f4600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec0b37f0 con 0x7f32ec06c9a0 2024-02-19T15:22:53.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f4600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec1f6da0 con 0x7f32ec15b1a0 2024-02-19T15:22:53.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f4600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec1f7ec0 con 0x7f32ec071350 2024-02-19T15:22:53.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f1c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32ec06c9a0 0x7f32ec1f6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:53.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f1c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32ec06c9a0 0x7f32ec1f6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39324/0 (socket says 172.21.15.47:39324) 2024-02-19T15:22:53.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f1c00640 1 -- 172.21.15.47:0/2268366435 learned_addr learned my addr 172.21.15.47:0/2268366435 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:53.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f2600640 1 --2- 172.21.15.47:0/2268366435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32ec15b1a0 0x7f32ec0b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:53.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f1200640 1 --2- 172.21.15.47:0/2268366435 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32ec071350 0x7f32ec1f7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:53.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f1c00640 1 -- 172.21.15.47:0/2268366435 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32ec071350 msgr2=0x7f32ec1f7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:53.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f1c00640 1 --2- 172.21.15.47:0/2268366435 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32ec071350 0x7f32ec1f7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f1c00640 1 -- 172.21.15.47:0/2268366435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32ec15b1a0 msgr2=0x7f32ec0b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:53.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f1c00640 1 --2- 172.21.15.47:0/2268366435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32ec15b1a0 0x7f32ec0b3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:53.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f1c00640 1 -- 172.21.15.47:0/2268366435 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e0009420 con 0x7f32ec06c9a0 2024-02-19T15:22:53.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f1c00640 1 --2- 172.21.15.47:0/2268366435 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32ec06c9a0 0x7f32ec1f6860 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f32e00190b0 tx=0x7f32e001ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:53.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32e9600640 1 -- 172.21.15.47:0/2268366435 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e0004220 con 0x7f32ec06c9a0 2024-02-19T15:22:53.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32e9600640 1 -- 172.21.15.47:0/2268366435 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e00b9070 con 0x7f32ec06c9a0 2024-02-19T15:22:53.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32e9600640 1 -- 172.21.15.47:0/2268366435 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e0003620 con 0x7f32ec06c9a0 2024-02-19T15:22:53.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f4600640 1 -- 172.21.15.47:0/2268366435 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32ec0b2be0 con 0x7f32ec06c9a0 2024-02-19T15:22:53.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f4600640 1 -- 172.21.15.47:0/2268366435 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32ec0b31a0 con 0x7f32ec06c9a0 2024-02-19T15:22:53.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.980+0000 7f32f4600640 1 -- 172.21.15.47:0/2268366435 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32ec075360 con 0x7f32ec06c9a0 2024-02-19T15:22:53.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.984+0000 7f32e9600640 1 -- 172.21.15.47:0/2268366435 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f32e0003780 con 0x7f32ec06c9a0 2024-02-19T15:22:53.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.984+0000 7f32e9600640 1 --2- 172.21.15.47:0/2268366435 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f32cc066e40 0x7f32cc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:53.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.984+0000 7f32e9600640 1 -- 172.21.15.47:0/2268366435 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f32e00bb070 con 0x7f32ec06c9a0 2024-02-19T15:22:53.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.984+0000 7f32f1200640 1 --2- 172.21.15.47:0/2268366435 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f32cc066e40 0x7f32cc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:53.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.984+0000 7f32f1200640 1 --2- 172.21.15.47:0/2268366435 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f32cc066e40 0x7f32cc069300 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f32ec0afb60 tx=0x7f32d8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:53.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:53.988+0000 7f32e9600640 1 -- 172.21.15.47:0/2268366435 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f32e00ebd10 con 0x7f32ec06c9a0 2024-02-19T15:22:54.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.208+0000 7f32f4600640 1 -- 172.21.15.47:0/2268366435 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f32ec06eb90 con 0x7f32ec06c9a0 2024-02-19T15:22:54.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.212+0000 7f32e9600640 1 -- 172.21.15.47:0/2268366435 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f32e00ef3d0 con 0x7f32ec06c9a0 2024-02-19T15:22:54.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.212+0000 7f32f4600640 1 -- 172.21.15.47:0/2268366435 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f32cc066e40 msgr2=0x7f32cc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:54.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.212+0000 7f32f4600640 1 --2- 172.21.15.47:0/2268366435 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f32cc066e40 0x7f32cc069300 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f32ec0afb60 tx=0x7f32d8009040 comp rx=0 tx=0).stop 2024-02-19T15:22:54.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.212+0000 7f32f4600640 1 -- 172.21.15.47:0/2268366435 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32ec06c9a0 msgr2=0x7f32ec1f6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:54.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.212+0000 7f32f4600640 1 --2- 172.21.15.47:0/2268366435 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32ec06c9a0 0x7f32ec1f6860 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f32e00190b0 tx=0x7f32e001ace0 comp rx=0 tx=0).stop 2024-02-19T15:22:54.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.216+0000 7f32f4600640 1 -- 172.21.15.47:0/2268366435 shutdown_connections 2024-02-19T15:22:54.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.216+0000 7f32f4600640 1 --2- 172.21.15.47:0/2268366435 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f32cc066e40 0x7f32cc069300 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:54.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.216+0000 7f32f4600640 1 --2- 172.21.15.47:0/2268366435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32ec15b1a0 0x7f32ec0b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:54.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.216+0000 7f32f4600640 1 --2- 172.21.15.47:0/2268366435 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32ec071350 0x7f32ec1f7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:54.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.216+0000 7f32f4600640 1 --2- 172.21.15.47:0/2268366435 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32ec06c9a0 0x7f32ec1f6860 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:54.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.216+0000 7f32f4600640 1 -- 172.21.15.47:0/2268366435 >> 172.21.15.47:0/2268366435 conn(0x7f32ec0b16a0 msgr2=0x7f32ec15a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:54.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.216+0000 7f32f4600640 1 -- 172.21.15.47:0/2268366435 shutdown_connections 2024-02-19T15:22:54.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.216+0000 7f32f4600640 1 -- 172.21.15.47:0/2268366435 wait complete. 2024-02-19T15:22:54.237 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-02-19T15:22:54.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.372+0000 7fa462a00640 1 Processor -- start 2024-02-19T15:22:54.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.372+0000 7fa462a00640 1 -- start start 2024-02-19T15:22:54.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.372+0000 7fa462a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa45c0cea80 0x7fa45c0727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:54.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.372+0000 7fa462a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa45c15c390 0x7fa45c15c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:54.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.372+0000 7fa462a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa45c15d280 con 0x7fa45c145590 2024-02-19T15:22:54.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.372+0000 7fa462a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa45c070ff0 con 0x7fa45c0cea80 2024-02-19T15:22:54.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.372+0000 7fa462a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c15cd60 con 0x7fa45c15c390 2024-02-19T15:22:54.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.372+0000 7fa453e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa45c0cea80 0x7fa45c0727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:54.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.372+0000 7fa460a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa45c15c390 0x7fa45c15c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:54.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.372+0000 7fa453e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa45c0cea80 0x7fa45c0727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:47240/0 (socket says 172.21.15.47:47240) 2024-02-19T15:22:54.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.372+0000 7fa45be00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa45c145590 0x7fa45c06c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:37390/0 (socket says 172.21.15.47:37390) 2024-02-19T15:22:54.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.372+0000 7fa453e00640 1 -- 172.21.15.47:0/1678755029 learned_addr learned my addr 172.21.15.47:0/1678755029 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:54.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.372+0000 7fa460a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa45c15c390 0x7fa45c15c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:49296/0 (socket says 172.21.15.47:49296) 2024-02-19T15:22:54.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.376+0000 7fa453e00640 1 -- 172.21.15.47:0/1678755029 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa45c15c390 msgr2=0x7fa45c15c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:54.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.376+0000 7fa453e00640 1 --2- 172.21.15.47:0/1678755029 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa45c15c390 0x7fa45c15c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:54.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.376+0000 7fa453e00640 1 -- 172.21.15.47:0/1678755029 >> v1:172.21.15.116:6789/0 conn(0x7fa45c145590 legacy=0x7fa45c06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:54.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.376+0000 7fa453e00640 1 -- 172.21.15.47:0/1678755029 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45c0709e0 con 0x7fa45c0cea80 2024-02-19T15:22:54.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.376+0000 7fa453e00640 1 --2- 172.21.15.47:0/1678755029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa45c0cea80 0x7fa45c0727b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa444009870 tx=0x7fa44401c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3065958278dfb42a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:54.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.376+0000 7fa45b400640 1 -- 172.21.15.47:0/1678755029 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4440ab650 con 0x7fa45c0cea80 2024-02-19T15:22:54.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.376+0000 7fa45b400640 1 -- 172.21.15.47:0/1678755029 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4440ab7b0 con 0x7fa45c0cea80 2024-02-19T15:22:54.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.376+0000 7fa45b400640 1 -- 172.21.15.47:0/1678755029 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4440abb40 con 0x7fa45c0cea80 2024-02-19T15:22:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa462a00640 1 -- 172.21.15.47:0/1678755029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa45c0cea80 msgr2=0x7fa45c0727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa462a00640 1 --2- 172.21.15.47:0/1678755029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa45c0cea80 0x7fa45c0727b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa444009870 tx=0x7fa44401c560 comp rx=0 tx=0).stop 2024-02-19T15:22:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa462a00640 1 -- 172.21.15.47:0/1678755029 shutdown_connections 2024-02-19T15:22:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa462a00640 1 --2- 172.21.15.47:0/1678755029 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa45c15c390 0x7fa45c15c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa462a00640 1 --2- 172.21.15.47:0/1678755029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa45c0cea80 0x7fa45c0727b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa462a00640 1 -- 172.21.15.47:0/1678755029 >> 172.21.15.47:0/1678755029 conn(0x7fa45c0b16a0 msgr2=0x7fa45c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa462a00640 1 -- 172.21.15.47:0/1678755029 shutdown_connections 2024-02-19T15:22:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa462a00640 1 -- 172.21.15.47:0/1678755029 wait complete. 2024-02-19T15:22:54.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa462a00640 1 Processor -- start 2024-02-19T15:22:54.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa462a00640 1 -- start start 2024-02-19T15:22:54.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa462a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa45c0cea80 0x7fa45c13bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:54.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa462a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa45c145590 0x7fa45c0aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:54.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa462a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa45c15c390 0x7fa45c15ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:54.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa462a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c0af330 con 0x7fa45c145590 2024-02-19T15:22:54.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa462a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c13cf20 con 0x7fa45c0cea80 2024-02-19T15:22:54.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa462a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c13c310 con 0x7fa45c15c390 2024-02-19T15:22:54.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa453e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa45c145590 0x7fa45c0aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:54.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa453e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa45c145590 0x7fa45c0aedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39328/0 (socket says 172.21.15.47:39328) 2024-02-19T15:22:54.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa453e00640 1 -- 172.21.15.47:0/216052888 learned_addr learned my addr 172.21.15.47:0/216052888 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa460a00640 1 --2- 172.21.15.47:0/216052888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa45c15c390 0x7fa45c15ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa45be00640 1 --2- 172.21.15.47:0/216052888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa45c0cea80 0x7fa45c13bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa453e00640 1 -- 172.21.15.47:0/216052888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa45c15c390 msgr2=0x7fa45c15ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa453e00640 1 --2- 172.21.15.47:0/216052888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa45c15c390 0x7fa45c15ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa453e00640 1 -- 172.21.15.47:0/216052888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa45c0cea80 msgr2=0x7fa45c13bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa453e00640 1 --2- 172.21.15.47:0/216052888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa45c0cea80 0x7fa45c13bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa453e00640 1 -- 172.21.15.47:0/216052888 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4440094d0 con 0x7fa45c145590 2024-02-19T15:22:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa460a00640 1 --2- 172.21.15.47:0/216052888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa45c15c390 0x7fa45c15ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:22:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa45be00640 1 --2- 172.21.15.47:0/216052888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa45c0cea80 0x7fa45c13bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:22:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.380+0000 7fa453e00640 1 --2- 172.21.15.47:0/216052888 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa45c145590 0x7fa45c0aedf0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa4440ad040 tx=0x7fa44401ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:54.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.384+0000 7fa458c00640 1 -- 172.21.15.47:0/216052888 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4440b4800 con 0x7fa45c145590 2024-02-19T15:22:54.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.384+0000 7fa458c00640 1 -- 172.21.15.47:0/216052888 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4440b4960 con 0x7fa45c145590 2024-02-19T15:22:54.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.384+0000 7fa458c00640 1 -- 172.21.15.47:0/216052888 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4440b4cf0 con 0x7fa45c145590 2024-02-19T15:22:54.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.384+0000 7fa462a00640 1 -- 172.21.15.47:0/216052888 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa45c13c5f0 con 0x7fa45c145590 2024-02-19T15:22:54.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.384+0000 7fa462a00640 1 -- 172.21.15.47:0/216052888 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45c13cbb0 con 0x7fa45c145590 2024-02-19T15:22:54.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.384+0000 7fa462a00640 1 -- 172.21.15.47:0/216052888 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa45c13d0b0 con 0x7fa45c145590 2024-02-19T15:22:54.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.384+0000 7fa458c00640 1 -- 172.21.15.47:0/216052888 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa4440b3380 con 0x7fa45c145590 2024-02-19T15:22:54.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.384+0000 7fa458c00640 1 --2- 172.21.15.47:0/216052888 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa438066e40 0x7fa438069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:54.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.384+0000 7fa458c00640 1 -- 172.21.15.47:0/216052888 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa44411ef40 con 0x7fa45c145590 2024-02-19T15:22:54.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.384+0000 7fa45be00640 1 --2- 172.21.15.47:0/216052888 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa438066e40 0x7fa438069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:54.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.384+0000 7fa45be00640 1 --2- 172.21.15.47:0/216052888 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa438066e40 0x7fa438069300 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa44c007fd0 tx=0x7fa44c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:54.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.388+0000 7fa458c00640 1 -- 172.21.15.47:0/216052888 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa4440ec000 con 0x7fa45c145590 2024-02-19T15:22:54.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.608+0000 7fa462a00640 1 -- 172.21.15.47:0/216052888 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fa45c068310 con 0x7fa45c145590 2024-02-19T15:22:54.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.612+0000 7fa458c00640 1 -- 172.21.15.47:0/216052888 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa4440c4070 con 0x7fa45c145590 2024-02-19T15:22:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.612+0000 7fa462a00640 1 -- 172.21.15.47:0/216052888 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa438066e40 msgr2=0x7fa438069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.612+0000 7fa462a00640 1 --2- 172.21.15.47:0/216052888 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa438066e40 0x7fa438069300 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa44c007fd0 tx=0x7fa44c009040 comp rx=0 tx=0).stop 2024-02-19T15:22:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.612+0000 7fa462a00640 1 -- 172.21.15.47:0/216052888 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa45c145590 msgr2=0x7fa45c0aedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.612+0000 7fa462a00640 1 --2- 172.21.15.47:0/216052888 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa45c145590 0x7fa45c0aedf0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa4440ad040 tx=0x7fa44401ada0 comp rx=0 tx=0).stop 2024-02-19T15:22:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.612+0000 7fa462a00640 1 -- 172.21.15.47:0/216052888 shutdown_connections 2024-02-19T15:22:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.612+0000 7fa462a00640 1 --2- 172.21.15.47:0/216052888 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa438066e40 0x7fa438069300 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:54.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.612+0000 7fa462a00640 1 --2- 172.21.15.47:0/216052888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa45c15c390 0x7fa45c15ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:54.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.612+0000 7fa462a00640 1 --2- 172.21.15.47:0/216052888 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa45c145590 0x7fa45c0aedf0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:54.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.612+0000 7fa462a00640 1 --2- 172.21.15.47:0/216052888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa45c0cea80 0x7fa45c13bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:54.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.612+0000 7fa462a00640 1 -- 172.21.15.47:0/216052888 >> 172.21.15.47:0/216052888 conn(0x7fa45c0b16a0 msgr2=0x7fa45c0af510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:54.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.612+0000 7fa462a00640 1 -- 172.21.15.47:0/216052888 shutdown_connections 2024-02-19T15:22:54.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.612+0000 7fa462a00640 1 -- 172.21.15.47:0/216052888 wait complete. 2024-02-19T15:22:54.638 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-02-19T15:22:54.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.772+0000 7f0016e00640 1 Processor -- start 2024-02-19T15:22:54.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.772+0000 7f0016e00640 1 -- start start 2024-02-19T15:22:54.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.776+0000 7f0016e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0010145590 0x7f00100727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:54.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.776+0000 7f0016e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f00100cea80 0x7f001006c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:54.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.776+0000 7f0016e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001015d280 con 0x7f001015c390 2024-02-19T15:22:54.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.776+0000 7f0016e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0010070ff0 con 0x7f00100cea80 2024-02-19T15:22:54.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.776+0000 7f0016e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00100709e0 con 0x7f0010145590 2024-02-19T15:22:54.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.776+0000 7f0007400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f00100cea80 0x7f001006c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:54.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.776+0000 7f000fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0010145590 0x7f00100727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:54.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.776+0000 7f0007400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f00100cea80 0x7f001006c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:47256/0 (socket says 172.21.15.47:47256) 2024-02-19T15:22:54.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.776+0000 7f0007400640 1 -- 172.21.15.47:0/1140669208 learned_addr learned my addr 172.21.15.47:0/1140669208 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:54.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.776+0000 7f0014e00640 1 --1- 172.21.15.47:0/1140669208 >> v1:172.21.15.116:6789/0 conn(0x7f001015c390 0x7f001015c820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:37398/0 (socket says 172.21.15.47:37398) 2024-02-19T15:22:54.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.776+0000 7f0007400640 1 -- 172.21.15.47:0/1140669208 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0010145590 msgr2=0x7f00100727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:54.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.776+0000 7f0007400640 1 --2- 172.21.15.47:0/1140669208 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0010145590 0x7f00100727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:54.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.776+0000 7f0007400640 1 -- 172.21.15.47:0/1140669208 >> v1:172.21.15.116:6789/0 conn(0x7f001015c390 legacy=0x7f001015c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:54.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.776+0000 7f0007400640 1 -- 172.21.15.47:0/1140669208 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001006cee0 con 0x7f00100cea80 2024-02-19T15:22:54.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.780+0000 7f0007400640 1 --2- 172.21.15.47:0/1140669208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00100cea80 0x7f001006c9a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7efff80098a0 tx=0x7efff801c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ac6f26eeb933afb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:54.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.780+0000 7f000f400640 1 -- 172.21.15.47:0/1140669208 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff80a7070 con 0x7f00100cea80 2024-02-19T15:22:54.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.780+0000 7f000f400640 1 -- 172.21.15.47:0/1140669208 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff80ab430 con 0x7f00100cea80 2024-02-19T15:22:54.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.780+0000 7f000f400640 1 -- 172.21.15.47:0/1140669208 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff80ab7c0 con 0x7f00100cea80 2024-02-19T15:22:54.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.780+0000 7f0016e00640 1 -- 172.21.15.47:0/1140669208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00100cea80 msgr2=0x7f001006c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.780+0000 7f0016e00640 1 --2- 172.21.15.47:0/1140669208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00100cea80 0x7f001006c9a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7efff80098a0 tx=0x7efff801c540 comp rx=0 tx=0).stop 2024-02-19T15:22:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.780+0000 7f0016e00640 1 -- 172.21.15.47:0/1140669208 shutdown_connections 2024-02-19T15:22:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.780+0000 7f0016e00640 1 --2- 172.21.15.47:0/1140669208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00100cea80 0x7f001006c9a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.780+0000 7f0016e00640 1 --2- 172.21.15.47:0/1140669208 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0010145590 0x7f00100727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.780+0000 7f0016e00640 1 -- 172.21.15.47:0/1140669208 >> 172.21.15.47:0/1140669208 conn(0x7f00100b16a0 msgr2=0x7f00100b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.780+0000 7f0016e00640 1 -- 172.21.15.47:0/1140669208 shutdown_connections 2024-02-19T15:22:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.780+0000 7f0016e00640 1 -- 172.21.15.47:0/1140669208 wait complete. 2024-02-19T15:22:54.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.780+0000 7f0016e00640 1 Processor -- start 2024-02-19T15:22:54.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.780+0000 7f0016e00640 1 -- start start 2024-02-19T15:22:54.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f0016e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00100cea80 0x7f00100b4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:54.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f0016e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0010145590 0x7f001013f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:54.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f0016e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f001015c390 0x7f001015add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:54.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f0016e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00101439b0 con 0x7f001015c390 2024-02-19T15:22:54.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f0016e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0010142da0 con 0x7f00100cea80 2024-02-19T15:22:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f0016e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0010143080 con 0x7f0010145590 2024-02-19T15:22:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f0014e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f001015c390 0x7f001015add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f0014e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f001015c390 0x7f001015add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39340/0 (socket says 172.21.15.47:39340) 2024-02-19T15:22:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f0014e00640 1 -- 172.21.15.47:0/619647197 learned_addr learned my addr 172.21.15.47:0/619647197 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f000fe00640 1 --2- 172.21.15.47:0/619647197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00100cea80 0x7f00100b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f0007400640 1 --2- 172.21.15.47:0/619647197 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0010145590 0x7f001013f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f0014e00640 1 -- 172.21.15.47:0/619647197 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0010145590 msgr2=0x7f001013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f0014e00640 1 --2- 172.21.15.47:0/619647197 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0010145590 0x7f001013f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f0014e00640 1 -- 172.21.15.47:0/619647197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00100cea80 msgr2=0x7f00100b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f0014e00640 1 --2- 172.21.15.47:0/619647197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00100cea80 0x7f00100b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f0014e00640 1 -- 172.21.15.47:0/619647197 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff80094d0 con 0x7f001015c390 2024-02-19T15:22:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f0014e00640 1 --2- 172.21.15.47:0/619647197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f001015c390 0x7f001015add0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f000800c700 tx=0x7f000800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f000cc00640 1 -- 172.21.15.47:0/619647197 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000800ce10 con 0x7f001015c390 2024-02-19T15:22:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f000cc00640 1 -- 172.21.15.47:0/619647197 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f000800a640 con 0x7f001015c390 2024-02-19T15:22:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f000cc00640 1 -- 172.21.15.47:0/619647197 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000800a9b0 con 0x7f001015c390 2024-02-19T15:22:54.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f0016e00640 1 -- 172.21.15.47:0/619647197 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0010143360 con 0x7f001015c390 2024-02-19T15:22:54.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f0016e00640 1 -- 172.21.15.47:0/619647197 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f001015bcd0 con 0x7f001015c390 2024-02-19T15:22:54.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f0016e00640 1 -- 172.21.15.47:0/619647197 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0010143b20 con 0x7f001015c390 2024-02-19T15:22:54.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.784+0000 7f000cc00640 1 -- 172.21.15.47:0/619647197 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f000800ab10 con 0x7f001015c390 2024-02-19T15:22:54.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.788+0000 7f000cc00640 1 --2- 172.21.15.47:0/619647197 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7effec066e70 0x7effec069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:54.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.788+0000 7f000cc00640 1 -- 172.21.15.47:0/619647197 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0008083e70 con 0x7f001015c390 2024-02-19T15:22:54.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.788+0000 7f000fe00640 1 --2- 172.21.15.47:0/619647197 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7effec066e70 0x7effec069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:54.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.788+0000 7f000fe00640 1 --2- 172.21.15.47:0/619647197 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7effec066e70 0x7effec069330 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f001013fb00 tx=0x7f0000009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:54.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:54.792+0000 7f000cc00640 1 -- 172.21.15.47:0/619647197 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0008050fc0 con 0x7f001015c390 2024-02-19T15:22:55.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.008+0000 7f0016e00640 1 -- 172.21.15.47:0/619647197 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f0010075360 con 0x7f001015c390 2024-02-19T15:22:55.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.012+0000 7f000cc00640 1 -- 172.21.15.47:0/619647197 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f00080170a0 con 0x7f001015c390 2024-02-19T15:22:55.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.012+0000 7f0016e00640 1 -- 172.21.15.47:0/619647197 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7effec066e70 msgr2=0x7effec069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:55.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.012+0000 7f0016e00640 1 --2- 172.21.15.47:0/619647197 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7effec066e70 0x7effec069330 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f001013fb00 tx=0x7f0000009190 comp rx=0 tx=0).stop 2024-02-19T15:22:55.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.012+0000 7f0016e00640 1 -- 172.21.15.47:0/619647197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f001015c390 msgr2=0x7f001015add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:55.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.012+0000 7f0016e00640 1 --2- 172.21.15.47:0/619647197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f001015c390 0x7f001015add0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f000800c700 tx=0x7f000800cbc0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.012+0000 7f0016e00640 1 -- 172.21.15.47:0/619647197 shutdown_connections 2024-02-19T15:22:55.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.012+0000 7f0016e00640 1 --2- 172.21.15.47:0/619647197 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7effec066e70 0x7effec069330 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.012+0000 7f0016e00640 1 --2- 172.21.15.47:0/619647197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f001015c390 0x7f001015add0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.012+0000 7f0016e00640 1 --2- 172.21.15.47:0/619647197 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0010145590 0x7f001013f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.012+0000 7f0016e00640 1 --2- 172.21.15.47:0/619647197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00100cea80 0x7f00100b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.012+0000 7f0016e00640 1 -- 172.21.15.47:0/619647197 >> 172.21.15.47:0/619647197 conn(0x7f00100b16a0 msgr2=0x7f00100ae250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:55.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.012+0000 7f0016e00640 1 -- 172.21.15.47:0/619647197 shutdown_connections 2024-02-19T15:22:55.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.012+0000 7f0016e00640 1 -- 172.21.15.47:0/619647197 wait complete. 2024-02-19T15:22:55.038 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:55.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.176+0000 7fa9cfc00640 1 Processor -- start 2024-02-19T15:22:55.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.176+0000 7fa9cfc00640 1 -- start start 2024-02-19T15:22:55.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.176+0000 7fa9cfc00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa9c815b1a0 0x7fa9c815b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:55.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.176+0000 7fa9cfc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9c8071350 0x7fa9c815bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:55.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.176+0000 7fa9cfc00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c815c000 con 0x7fa9c806c9a0 2024-02-19T15:22:55.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.176+0000 7fa9cfc00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9c80718c0 con 0x7fa9c815b1a0 2024-02-19T15:22:55.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.176+0000 7fa9cfc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c8071ba0 con 0x7fa9c8071350 2024-02-19T15:22:55.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.176+0000 7fa9cd200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa9c815b1a0 0x7fa9c815b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:55.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.176+0000 7fa9cdc00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa9c806c9a0 0x7fa9c815c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53810/0 (socket says 172.21.15.47:53810) 2024-02-19T15:22:55.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.176+0000 7fa9cdc00640 1 -- 172.21.15.47:0/1733465852 learned_addr learned my addr 172.21.15.47:0/1733465852 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:55.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.176+0000 7fa9c7e00640 1 --2- 172.21.15.47:0/1733465852 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9c8071350 0x7fa9c815bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:55.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.176+0000 7fa9cd200640 1 -- 172.21.15.47:0/1733465852 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9c8071350 msgr2=0x7fa9c815bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:55.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.176+0000 7fa9cd200640 1 --2- 172.21.15.47:0/1733465852 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9c8071350 0x7fa9c815bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.176+0000 7fa9cd200640 1 -- 172.21.15.47:0/1733465852 >> v1:172.21.15.47:6789/0 conn(0x7fa9c806c9a0 legacy=0x7fa9c815c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:55.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.176+0000 7fa9cd200640 1 -- 172.21.15.47:0/1733465852 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c8071e80 con 0x7fa9c815b1a0 2024-02-19T15:22:55.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.176+0000 7fa9c7400640 1 -- 172.21.15.47:0/1733465852 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2589664457 0 0) 0x7fa9c815c000 con 0x7fa9c806c9a0 2024-02-19T15:22:55.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.180+0000 7fa9cd200640 1 --2- 172.21.15.47:0/1733465852 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9c815b1a0 0x7fa9c815b580 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc0097c0 tx=0x7fa9bc01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=affde6d24c853210 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:55.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.180+0000 7fa9c7400640 1 -- 172.21.15.47:0/1733465852 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc0aa5f0 con 0x7fa9c815b1a0 2024-02-19T15:22:55.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.180+0000 7fa9c7400640 1 -- 172.21.15.47:0/1733465852 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9bc0aa750 con 0x7fa9c815b1a0 2024-02-19T15:22:55.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.180+0000 7fa9c7400640 1 -- 172.21.15.47:0/1733465852 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc0aaae0 con 0x7fa9c815b1a0 2024-02-19T15:22:55.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.180+0000 7fa9cfc00640 1 -- 172.21.15.47:0/1733465852 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9c815b1a0 msgr2=0x7fa9c815b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:55.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.180+0000 7fa9cfc00640 1 --2- 172.21.15.47:0/1733465852 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9c815b1a0 0x7fa9c815b580 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc0097c0 tx=0x7fa9bc01c470 comp rx=0 tx=0).stop 2024-02-19T15:22:55.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.180+0000 7fa9cfc00640 1 -- 172.21.15.47:0/1733465852 shutdown_connections 2024-02-19T15:22:55.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.180+0000 7fa9cfc00640 1 --2- 172.21.15.47:0/1733465852 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9c8071350 0x7fa9c815bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.180+0000 7fa9cfc00640 1 --2- 172.21.15.47:0/1733465852 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9c815b1a0 0x7fa9c815b580 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.180+0000 7fa9cfc00640 1 -- 172.21.15.47:0/1733465852 >> 172.21.15.47:0/1733465852 conn(0x7fa9c80b16a0 msgr2=0x7fa9c80b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:55.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.180+0000 7fa9cfc00640 1 -- 172.21.15.47:0/1733465852 shutdown_connections 2024-02-19T15:22:55.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.180+0000 7fa9cfc00640 1 -- 172.21.15.47:0/1733465852 wait complete. 2024-02-19T15:22:55.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.180+0000 7fa9cfc00640 1 Processor -- start 2024-02-19T15:22:55.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.180+0000 7fa9cfc00640 1 -- start start 2024-02-19T15:22:55.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9cfc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9c806c9a0 0x7fa9c8146ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:55.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9cfc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9c8071350 0x7fa9c8147d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:55.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9cfc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9c815b1a0 0x7fa9c8148710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:55.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9cfc00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c8150530 con 0x7fa9c806c9a0 2024-02-19T15:22:55.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9cfc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c81507b0 con 0x7fa9c815b1a0 2024-02-19T15:22:55.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9cfc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c8150a90 con 0x7fa9c8071350 2024-02-19T15:22:55.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9cd200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9c806c9a0 0x7fa9c8146ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:55.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9cd200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9c806c9a0 0x7fa9c8146ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39372/0 (socket says 172.21.15.47:39372) 2024-02-19T15:22:55.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9cd200640 1 -- 172.21.15.47:0/3895749849 learned_addr learned my addr 172.21.15.47:0/3895749849 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:55.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9c7e00640 1 --2- 172.21.15.47:0/3895749849 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9c8071350 0x7fa9c8147d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:55.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9cdc00640 1 --2- 172.21.15.47:0/3895749849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9c815b1a0 0x7fa9c8148710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:55.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9cd200640 1 -- 172.21.15.47:0/3895749849 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9c8071350 msgr2=0x7fa9c8147d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:55.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9cd200640 1 --2- 172.21.15.47:0/3895749849 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9c8071350 0x7fa9c8147d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9cd200640 1 -- 172.21.15.47:0/3895749849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9c815b1a0 msgr2=0x7fa9c8148710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:55.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9cd200640 1 --2- 172.21.15.47:0/3895749849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9c815b1a0 0x7fa9c8148710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9cd200640 1 -- 172.21.15.47:0/3895749849 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9bc009420 con 0x7fa9c806c9a0 2024-02-19T15:22:55.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9c7e00640 1 --2- 172.21.15.47:0/3895749849 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9c8071350 0x7fa9c8147d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:22:55.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9cdc00640 1 --2- 172.21.15.47:0/3895749849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9c815b1a0 0x7fa9c8148710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:22:55.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9cd200640 1 --2- 172.21.15.47:0/3895749849 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9c806c9a0 0x7fa9c8146ca0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc0ac040 tx=0x7fa9bc01acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:55.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9c4c00640 1 -- 172.21.15.47:0/3895749849 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc004220 con 0x7fa9c806c9a0 2024-02-19T15:22:55.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9c4c00640 1 -- 172.21.15.47:0/3895749849 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9bc0b8070 con 0x7fa9c806c9a0 2024-02-19T15:22:55.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9c4c00640 1 -- 172.21.15.47:0/3895749849 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc003620 con 0x7fa9c806c9a0 2024-02-19T15:22:55.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9cfc00640 1 -- 172.21.15.47:0/3895749849 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c8150d70 con 0x7fa9c806c9a0 2024-02-19T15:22:55.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9cfc00640 1 -- 172.21.15.47:0/3895749849 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c8151330 con 0x7fa9c806c9a0 2024-02-19T15:22:55.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9cfc00640 1 -- 172.21.15.47:0/3895749849 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c8150bd0 con 0x7fa9c806c9a0 2024-02-19T15:22:55.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.184+0000 7fa9c4c00640 1 -- 172.21.15.47:0/3895749849 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa9bc003780 con 0x7fa9c806c9a0 2024-02-19T15:22:55.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.188+0000 7fa9c4c00640 1 --2- 172.21.15.47:0/3895749849 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa9a8066e70 0x7fa9a8069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:55.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.188+0000 7fa9c4c00640 1 -- 172.21.15.47:0/3895749849 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa9bc0ba070 con 0x7fa9c806c9a0 2024-02-19T15:22:55.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.188+0000 7fa9c7e00640 1 --2- 172.21.15.47:0/3895749849 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa9a8066e70 0x7fa9a8069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:55.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.188+0000 7fa9c7e00640 1 --2- 172.21.15.47:0/3895749849 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa9a8066e70 0x7fa9a8069330 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa9c814b530 tx=0x7fa9b4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:55.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.192+0000 7fa9c4c00640 1 -- 172.21.15.47:0/3895749849 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa9c8150bd0 con 0x7fa9c806c9a0 2024-02-19T15:22:55.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.352+0000 7fa9cfc00640 1 -- 172.21.15.47:0/3895749849 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9c8150bd0 con 0x7fa9c806c9a0 2024-02-19T15:22:55.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.356+0000 7fa9c4c00640 1 -- 172.21.15.47:0/3895749849 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+12903 (secure 0 0 0) 0x7fa9c8150bd0 con 0x7fa9c806c9a0 2024-02-19T15:22:55.359 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:55.359 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":30,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:52.177874+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:55.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.356+0000 7fa9cfc00640 1 -- 172.21.15.47:0/3895749849 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa9a8066e70 msgr2=0x7fa9a8069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:55.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.356+0000 7fa9cfc00640 1 --2- 172.21.15.47:0/3895749849 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa9a8066e70 0x7fa9a8069330 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa9c814b530 tx=0x7fa9b4008040 comp rx=0 tx=0).stop 2024-02-19T15:22:55.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.356+0000 7fa9cfc00640 1 -- 172.21.15.47:0/3895749849 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9c806c9a0 msgr2=0x7fa9c8146ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:55.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.356+0000 7fa9cfc00640 1 --2- 172.21.15.47:0/3895749849 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9c806c9a0 0x7fa9c8146ca0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc0ac040 tx=0x7fa9bc01acb0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.356+0000 7fa9cfc00640 1 -- 172.21.15.47:0/3895749849 shutdown_connections 2024-02-19T15:22:55.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.356+0000 7fa9cfc00640 1 --2- 172.21.15.47:0/3895749849 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa9a8066e70 0x7fa9a8069330 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.356+0000 7fa9cfc00640 1 --2- 172.21.15.47:0/3895749849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9c815b1a0 0x7fa9c8148710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.356+0000 7fa9cfc00640 1 --2- 172.21.15.47:0/3895749849 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9c8071350 0x7fa9c8147d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.356+0000 7fa9cfc00640 1 --2- 172.21.15.47:0/3895749849 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9c806c9a0 0x7fa9c8146ca0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.356+0000 7fa9cfc00640 1 -- 172.21.15.47:0/3895749849 >> 172.21.15.47:0/3895749849 conn(0x7fa9c80b16a0 msgr2=0x7fa9c815a550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:55.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.356+0000 7fa9cfc00640 1 -- 172.21.15.47:0/3895749849 shutdown_connections 2024-02-19T15:22:55.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.356+0000 7fa9cfc00640 1 -- 172.21.15.47:0/3895749849 wait complete. 2024-02-19T15:22:55.381 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:22:55.381 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:22:55.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.520+0000 7f98efa00640 1 Processor -- start 2024-02-19T15:22:55.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.520+0000 7f98efa00640 1 -- start start 2024-02-19T15:22:55.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.520+0000 7f98efa00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f98e80ce3b0 0x7f98e80ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:55.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.520+0000 7f98efa00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98e815e280 0x7f98e80cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:55.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.520+0000 7f98efa00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e80cc620 con 0x7f98e80d2650 2024-02-19T15:22:55.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.520+0000 7f98efa00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98e80d12e0 con 0x7f98e80ce3b0 2024-02-19T15:22:55.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.520+0000 7f98efa00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e80d1940 con 0x7f98e815e280 2024-02-19T15:22:55.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.520+0000 7f98ed000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f98e80ce3b0 0x7f98e80ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:55.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.520+0000 7f98ed000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f98e80ce3b0 0x7f98e80ce790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39380/0 (socket says 172.21.15.47:39380) 2024-02-19T15:22:55.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.520+0000 7f98ed000640 1 -- 172.21.15.47:0/2766948037 learned_addr learned my addr 172.21.15.47:0/2766948037 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:55.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.520+0000 7f98eda00640 1 --2- 172.21.15.47:0/2766948037 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98e815e280 0x7f98e80cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:55.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.520+0000 7f98ed000640 1 -- 172.21.15.47:0/2766948037 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98e815e280 msgr2=0x7f98e80cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:55.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.520+0000 7f98ed000640 1 --2- 172.21.15.47:0/2766948037 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98e815e280 0x7f98e80cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.520+0000 7f98ed000640 1 -- 172.21.15.47:0/2766948037 >> v1:172.21.15.116:6789/0 conn(0x7f98e80d2650 legacy=0x7f98e80ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:55.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.520+0000 7f98ed000640 1 -- 172.21.15.47:0/2766948037 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e80cecd0 con 0x7f98e80ce3b0 2024-02-19T15:22:55.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.524+0000 7f98ed000640 1 --2- 172.21.15.47:0/2766948037 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98e80ce3b0 0x7f98e80ce790 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f98dc009870 tx=0x7f98dc01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7811414f89a4cac server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.524+0000 7f98e7400640 1 -- 172.21.15.47:0/2766948037 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc0ab600 con 0x7f98e80ce3b0 2024-02-19T15:22:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.524+0000 7f98e7400640 1 -- 172.21.15.47:0/2766948037 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98dc0a7050 con 0x7f98e80ce3b0 2024-02-19T15:22:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.524+0000 7f98e7400640 1 -- 172.21.15.47:0/2766948037 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc0ab980 con 0x7f98e80ce3b0 2024-02-19T15:22:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.524+0000 7f98efa00640 1 -- 172.21.15.47:0/2766948037 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98e80ce3b0 msgr2=0x7f98e80ce790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.524+0000 7f98efa00640 1 --2- 172.21.15.47:0/2766948037 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98e80ce3b0 0x7f98e80ce790 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f98dc009870 tx=0x7f98dc01c440 comp rx=0 tx=0).stop 2024-02-19T15:22:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.524+0000 7f98efa00640 1 -- 172.21.15.47:0/2766948037 shutdown_connections 2024-02-19T15:22:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.524+0000 7f98efa00640 1 --2- 172.21.15.47:0/2766948037 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98e815e280 0x7f98e80cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.524+0000 7f98efa00640 1 --2- 172.21.15.47:0/2766948037 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98e80ce3b0 0x7f98e80ce790 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.524+0000 7f98efa00640 1 -- 172.21.15.47:0/2766948037 >> 172.21.15.47:0/2766948037 conn(0x7f98e80d1b30 msgr2=0x7f98e80d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.524+0000 7f98efa00640 1 -- 172.21.15.47:0/2766948037 shutdown_connections 2024-02-19T15:22:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.524+0000 7f98efa00640 1 -- 172.21.15.47:0/2766948037 wait complete. 2024-02-19T15:22:55.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98efa00640 1 Processor -- start 2024-02-19T15:22:55.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98efa00640 1 -- start start 2024-02-19T15:22:55.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98efa00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98e80ce3b0 0x7f98e814ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:55.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98efa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98e80d2650 0x7f98e815ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:55.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98efa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e815e280 0x7f98e8155e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:55.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98efa00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e814c820 con 0x7f98e80d2650 2024-02-19T15:22:55.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98efa00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e8158390 con 0x7f98e815e280 2024-02-19T15:22:55.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98efa00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e81fb000 con 0x7f98e80ce3b0 2024-02-19T15:22:55.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98e7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98e80d2650 0x7f98e815ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:55.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98e7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98e80d2650 0x7f98e815ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39388/0 (socket says 172.21.15.47:39388) 2024-02-19T15:22:55.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98e7e00640 1 -- 172.21.15.47:0/1514110117 learned_addr learned my addr 172.21.15.47:0/1514110117 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:55.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98ed000640 1 --2- 172.21.15.47:0/1514110117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98e80ce3b0 0x7f98e814ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:55.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98eda00640 1 --2- 172.21.15.47:0/1514110117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e815e280 0x7f98e8155e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:55.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98e7e00640 1 -- 172.21.15.47:0/1514110117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98e80ce3b0 msgr2=0x7f98e814ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:55.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98e7e00640 1 --2- 172.21.15.47:0/1514110117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98e80ce3b0 0x7f98e814ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98e7e00640 1 -- 172.21.15.47:0/1514110117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e815e280 msgr2=0x7f98e8155e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:55.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98e7e00640 1 --2- 172.21.15.47:0/1514110117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e815e280 0x7f98e8155e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98e7e00640 1 -- 172.21.15.47:0/1514110117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc0094d0 con 0x7f98e80d2650 2024-02-19T15:22:55.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98e7e00640 1 --2- 172.21.15.47:0/1514110117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98e80d2650 0x7f98e815ff30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f98d400c370 tx=0x7f98d400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:55.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98e4c00640 1 -- 172.21.15.47:0/1514110117 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d40052d0 con 0x7f98e80d2650 2024-02-19T15:22:55.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98e4c00640 1 -- 172.21.15.47:0/1514110117 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d4005430 con 0x7f98e80d2650 2024-02-19T15:22:55.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98e4c00640 1 -- 172.21.15.47:0/1514110117 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d4007260 con 0x7f98e80d2650 2024-02-19T15:22:55.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98efa00640 1 -- 172.21.15.47:0/1514110117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e81fb2e0 con 0x7f98e80d2650 2024-02-19T15:22:55.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.528+0000 7f98efa00640 1 -- 172.21.15.47:0/1514110117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e81fb8a0 con 0x7f98e80d2650 2024-02-19T15:22:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.532+0000 7f98efa00640 1 -- 172.21.15.47:0/1514110117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e81fba60 con 0x7f98e80d2650 2024-02-19T15:22:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.532+0000 7f98e4c00640 1 -- 172.21.15.47:0/1514110117 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f98d40073c0 con 0x7f98e80d2650 2024-02-19T15:22:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.532+0000 7f98e4c00640 1 --2- 172.21.15.47:0/1514110117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f98c8066e90 0x7f98c8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.532+0000 7f98e4c00640 1 -- 172.21.15.47:0/1514110117 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f98d4083820 con 0x7f98e80d2650 2024-02-19T15:22:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.532+0000 7f98ed000640 1 --2- 172.21.15.47:0/1514110117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f98c8066e90 0x7f98c8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.532+0000 7f98ed000640 1 --2- 172.21.15.47:0/1514110117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f98c8066e90 0x7f98c8069350 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f98dc0ad000 tx=0x7f98dc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:55.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.536+0000 7f98e4c00640 1 -- 172.21.15.47:0/1514110117 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f98d4050b80 con 0x7f98e80d2650 2024-02-19T15:22:55.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.696+0000 7f98efa00640 1 -- 172.21.15.47:0/1514110117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f98e8075360 con 0x7f98e80d2650 2024-02-19T15:22:55.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.696+0000 7f98e4c00640 1 -- 172.21.15.47:0/1514110117 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7f98d4054240 con 0x7f98e80d2650 2024-02-19T15:22:55.702 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:22:55.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.700+0000 7f98efa00640 1 -- 172.21.15.47:0/1514110117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f98c8066e90 msgr2=0x7f98c8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:55.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.700+0000 7f98efa00640 1 --2- 172.21.15.47:0/1514110117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f98c8066e90 0x7f98c8069350 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f98dc0ad000 tx=0x7f98dc008040 comp rx=0 tx=0).stop 2024-02-19T15:22:55.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.700+0000 7f98efa00640 1 -- 172.21.15.47:0/1514110117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98e80d2650 msgr2=0x7f98e815ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:55.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.700+0000 7f98efa00640 1 --2- 172.21.15.47:0/1514110117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98e80d2650 0x7f98e815ff30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f98d400c370 tx=0x7f98d400c830 comp rx=0 tx=0).stop 2024-02-19T15:22:55.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.700+0000 7f98efa00640 1 -- 172.21.15.47:0/1514110117 shutdown_connections 2024-02-19T15:22:55.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.700+0000 7f98efa00640 1 --2- 172.21.15.47:0/1514110117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f98c8066e90 0x7f98c8069350 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.700+0000 7f98efa00640 1 --2- 172.21.15.47:0/1514110117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e815e280 0x7f98e8155e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.700+0000 7f98efa00640 1 --2- 172.21.15.47:0/1514110117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98e80d2650 0x7f98e815ff30 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.700+0000 7f98efa00640 1 --2- 172.21.15.47:0/1514110117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98e80ce3b0 0x7f98e814ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.700+0000 7f98efa00640 1 -- 172.21.15.47:0/1514110117 >> 172.21.15.47:0/1514110117 conn(0x7f98e80d1b30 msgr2=0x7f98e80c6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:55.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.700+0000 7f98efa00640 1 -- 172.21.15.47:0/1514110117 shutdown_connections 2024-02-19T15:22:55.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.700+0000 7f98efa00640 1 -- 172.21.15.47:0/1514110117 wait complete. 2024-02-19T15:22:55.724 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T15:22:55.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.860+0000 7feecfe00640 1 Processor -- start 2024-02-19T15:22:55.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.860+0000 7feecfe00640 1 -- start start 2024-02-19T15:22:55.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.860+0000 7feecfe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feec815d6e0 0x7feec8162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:55.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.860+0000 7feecfe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feec8071350 0x7feec8164c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:55.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.860+0000 7feecfe00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec80718c0 con 0x7feec815b1a0 2024-02-19T15:22:55.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.860+0000 7feecfe00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feec8071ba0 con 0x7feec815d6e0 2024-02-19T15:22:55.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.860+0000 7feecfe00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feec8071e80 con 0x7feec8071350 2024-02-19T15:22:55.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.860+0000 7feecca00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feec815d6e0 0x7feec8162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:55.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.860+0000 7feecde00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feec8071350 0x7feec8164c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:55.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.860+0000 7feecca00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feec815d6e0 0x7feec8162740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:47290/0 (socket says 172.21.15.47:47290) 2024-02-19T15:22:55.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.860+0000 7feecd400640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7feec815b1a0 0x7feec815b580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:37424/0 (socket says 172.21.15.47:37424) 2024-02-19T15:22:55.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.860+0000 7feecca00640 1 -- 172.21.15.47:0/2624019225 learned_addr learned my addr 172.21.15.47:0/2624019225 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:55.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.860+0000 7feecca00640 1 -- 172.21.15.47:0/2624019225 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feec8071350 msgr2=0x7feec8164c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:55.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.860+0000 7feecca00640 1 --2- 172.21.15.47:0/2624019225 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feec8071350 0x7feec8164c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.860+0000 7feecca00640 1 -- 172.21.15.47:0/2624019225 >> v1:172.21.15.116:6789/0 conn(0x7feec815b1a0 legacy=0x7feec815b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:55.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.860+0000 7feec7e00640 1 -- 172.21.15.47:0/2624019225 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636023558 0 0) 0x7feec80718c0 con 0x7feec815b1a0 2024-02-19T15:22:55.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.860+0000 7feecca00640 1 -- 172.21.15.47:0/2624019225 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec8072160 con 0x7feec815d6e0 2024-02-19T15:22:55.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.864+0000 7feecca00640 1 --2- 172.21.15.47:0/2624019225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec815d6e0 0x7feec8162740 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7feeb40098a0 tx=0x7feeb401c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89452792913fc032 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:55.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.864+0000 7feec7e00640 1 -- 172.21.15.47:0/2624019225 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb40a9620 con 0x7feec815d6e0 2024-02-19T15:22:55.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.864+0000 7feec7e00640 1 -- 172.21.15.47:0/2624019225 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb40a9780 con 0x7feec815d6e0 2024-02-19T15:22:55.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.864+0000 7feec7e00640 1 -- 172.21.15.47:0/2624019225 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb40a9b10 con 0x7feec815d6e0 2024-02-19T15:22:55.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.864+0000 7feecfe00640 1 -- 172.21.15.47:0/2624019225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec815d6e0 msgr2=0x7feec8162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:55.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.864+0000 7feecfe00640 1 --2- 172.21.15.47:0/2624019225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec815d6e0 0x7feec8162740 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7feeb40098a0 tx=0x7feeb401c540 comp rx=0 tx=0).stop 2024-02-19T15:22:55.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.864+0000 7feecfe00640 1 -- 172.21.15.47:0/2624019225 shutdown_connections 2024-02-19T15:22:55.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.864+0000 7feecfe00640 1 --2- 172.21.15.47:0/2624019225 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feec8071350 0x7feec8164c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.864+0000 7feecfe00640 1 --2- 172.21.15.47:0/2624019225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec815d6e0 0x7feec8162740 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.864+0000 7feecfe00640 1 -- 172.21.15.47:0/2624019225 >> 172.21.15.47:0/2624019225 conn(0x7feec80b16a0 msgr2=0x7feec80b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:55.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.864+0000 7feecfe00640 1 -- 172.21.15.47:0/2624019225 shutdown_connections 2024-02-19T15:22:55.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.864+0000 7feecfe00640 1 -- 172.21.15.47:0/2624019225 wait complete. 2024-02-19T15:22:55.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecfe00640 1 Processor -- start 2024-02-19T15:22:55.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecfe00640 1 -- start start 2024-02-19T15:22:55.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feec8071350 0x7feec81509c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:55.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecfe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feec815b1a0 0x7feec80af740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:55.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecfe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec815d6e0 0x7feec81552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:55.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecfe00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec80afc80 con 0x7feec8071350 2024-02-19T15:22:55.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecfe00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec8151610 con 0x7feec815d6e0 2024-02-19T15:22:55.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecfe00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feec80afe00 con 0x7feec815b1a0 2024-02-19T15:22:55.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecd400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feec8071350 0x7feec81509c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecd400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feec8071350 0x7feec81509c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39398/0 (socket says 172.21.15.47:39398) 2024-02-19T15:22:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecd400640 1 -- 172.21.15.47:0/946773452 learned_addr learned my addr 172.21.15.47:0/946773452 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecca00640 1 --2- 172.21.15.47:0/946773452 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feec815b1a0 0x7feec80af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecde00640 1 --2- 172.21.15.47:0/946773452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec815d6e0 0x7feec81552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecd400640 1 -- 172.21.15.47:0/946773452 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feec815b1a0 msgr2=0x7feec80af740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecd400640 1 --2- 172.21.15.47:0/946773452 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feec815b1a0 0x7feec80af740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecd400640 1 -- 172.21.15.47:0/946773452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec815d6e0 msgr2=0x7feec81552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecd400640 1 --2- 172.21.15.47:0/946773452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec815d6e0 0x7feec81552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecd400640 1 -- 172.21.15.47:0/946773452 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb40094d0 con 0x7feec8071350 2024-02-19T15:22:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecd400640 1 --2- 172.21.15.47:0/946773452 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feec8071350 0x7feec81509c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7feebc0011f0 tx=0x7feebc00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feec5600640 1 -- 172.21.15.47:0/946773452 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feebc0074e0 con 0x7feec8071350 2024-02-19T15:22:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feec5600640 1 -- 172.21.15.47:0/946773452 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feebc007640 con 0x7feec8071350 2024-02-19T15:22:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feec5600640 1 -- 172.21.15.47:0/946773452 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feebc004a10 con 0x7feec8071350 2024-02-19T15:22:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecfe00640 1 -- 172.21.15.47:0/946773452 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec81557f0 con 0x7feec8071350 2024-02-19T15:22:55.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.868+0000 7feecfe00640 1 -- 172.21.15.47:0/946773452 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec80b0050 con 0x7feec8071350 2024-02-19T15:22:55.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.872+0000 7feecfe00640 1 -- 172.21.15.47:0/946773452 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec81510b0 con 0x7feec8071350 2024-02-19T15:22:55.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.872+0000 7feec5600640 1 -- 172.21.15.47:0/946773452 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7feebc004b70 con 0x7feec8071350 2024-02-19T15:22:55.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.872+0000 7feec5600640 1 --2- 172.21.15.47:0/946773452 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feea8066e40 0x7feea8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:55.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.872+0000 7feec5600640 1 -- 172.21.15.47:0/946773452 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7feebc083a70 con 0x7feec8071350 2024-02-19T15:22:55.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.872+0000 7feecca00640 1 --2- 172.21.15.47:0/946773452 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feea8066e40 0x7feea8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:55.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.872+0000 7feecca00640 1 --2- 172.21.15.47:0/946773452 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feea8066e40 0x7feea8069300 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7feeb40ae000 tx=0x7feeb40a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:55.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:55.876+0000 7feec5600640 1 -- 172.21.15.47:0/946773452 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7feebc050b10 con 0x7feec8071350 2024-02-19T15:22:56.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.040+0000 7feecfe00640 1 -- 172.21.15.47:0/946773452 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7feec815a860 con 0x7feec8071350 2024-02-19T15:22:56.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.044+0000 7feec5600640 1 -- 172.21.15.47:0/946773452 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) v1 ==== 84+0+65 (secure 0 0 0) 0x7feebc018070 con 0x7feec8071350 2024-02-19T15:22:56.047 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:56.047 INFO:teuthology.orchestra.run.smithi047.stdout:[ 2024-02-19T15:22:56.047 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:22:56.047 INFO:teuthology.orchestra.run.smithi047.stdout: "poolnum": 1, 2024-02-19T15:22:56.047 INFO:teuthology.orchestra.run.smithi047.stdout: "poolname": ".mgr" 2024-02-19T15:22:56.047 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:22:56.047 INFO:teuthology.orchestra.run.smithi047.stdout:] 2024-02-19T15:22:56.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.044+0000 7feecfe00640 1 -- 172.21.15.47:0/946773452 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feea8066e40 msgr2=0x7feea8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:56.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.044+0000 7feecfe00640 1 --2- 172.21.15.47:0/946773452 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feea8066e40 0x7feea8069300 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7feeb40ae000 tx=0x7feeb40a7000 comp rx=0 tx=0).stop 2024-02-19T15:22:56.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.044+0000 7feecfe00640 1 -- 172.21.15.47:0/946773452 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feec8071350 msgr2=0x7feec81509c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:56.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.044+0000 7feecfe00640 1 --2- 172.21.15.47:0/946773452 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feec8071350 0x7feec81509c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7feebc0011f0 tx=0x7feebc00c9b0 comp rx=0 tx=0).stop 2024-02-19T15:22:56.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.044+0000 7feecfe00640 1 -- 172.21.15.47:0/946773452 shutdown_connections 2024-02-19T15:22:56.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.044+0000 7feecfe00640 1 --2- 172.21.15.47:0/946773452 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feea8066e40 0x7feea8069300 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:56.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.044+0000 7feecfe00640 1 --2- 172.21.15.47:0/946773452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec815d6e0 0x7feec81552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:56.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.044+0000 7feecfe00640 1 --2- 172.21.15.47:0/946773452 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feec815b1a0 0x7feec80af740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:56.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.044+0000 7feecfe00640 1 --2- 172.21.15.47:0/946773452 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feec8071350 0x7feec81509c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:56.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.044+0000 7feecfe00640 1 -- 172.21.15.47:0/946773452 >> 172.21.15.47:0/946773452 conn(0x7feec80b16a0 msgr2=0x7feec815db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:56.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.044+0000 7feecfe00640 1 -- 172.21.15.47:0/946773452 shutdown_connections 2024-02-19T15:22:56.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.044+0000 7feecfe00640 1 -- 172.21.15.47:0/946773452 wait complete. 2024-02-19T15:22:56.068 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T15:22:56.069 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T15:22:56.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.208+0000 7f176e200640 1 Processor -- start 2024-02-19T15:22:56.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.208+0000 7f176e200640 1 -- start start 2024-02-19T15:22:56.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.212+0000 7f176e200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f17680cec80 0x7f17680d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:56.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.212+0000 7f176e200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f176815f8f0 0x7f176815fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:56.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.212+0000 7f176e200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17680d0a50 con 0x7f176815f190 2024-02-19T15:22:56.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.212+0000 7f176e200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f176806d470 con 0x7f176815f8f0 2024-02-19T15:22:56.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.212+0000 7f176e200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17680cfe80 con 0x7f17680cec80 2024-02-19T15:22:56.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.212+0000 7f1767e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f176815f8f0 0x7f176815fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:56.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.212+0000 7f1767e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f176815f8f0 0x7f176815fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39412/0 (socket says 172.21.15.47:39412) 2024-02-19T15:22:56.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.212+0000 7f1767e00640 1 -- 172.21.15.47:0/4134312370 learned_addr learned my addr 172.21.15.47:0/4134312370 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:56.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.212+0000 7f1767e00640 1 -- 172.21.15.47:0/4134312370 >> v2:172.21.15.116:3300/0 conn(0x7f17680cec80 msgr2=0x7f17680d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:56.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.212+0000 7f1767e00640 1 --2- 172.21.15.47:0/4134312370 >> v2:172.21.15.116:3300/0 conn(0x7f17680cec80 0x7f17680d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:56.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.212+0000 7f1767e00640 1 -- 172.21.15.47:0/4134312370 >> v1:172.21.15.116:6789/0 conn(0x7f176815f190 legacy=0x7f176806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:56.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.212+0000 7f1767e00640 1 -- 172.21.15.47:0/4134312370 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17680d0120 con 0x7f176815f8f0 2024-02-19T15:22:56.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.212+0000 7f1767e00640 1 --2- 172.21.15.47:0/4134312370 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f176815f8f0 0x7f176815fd80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f176809fa10 tx=0x7f176001c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64f700b094c23454 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:56.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.212+0000 7f1766a00640 1 -- 172.21.15.47:0/4134312370 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17600ac620 con 0x7f176815f8f0 2024-02-19T15:22:56.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f1766a00640 1 -- 172.21.15.47:0/4134312370 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17600ac780 con 0x7f176815f8f0 2024-02-19T15:22:56.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f1766a00640 1 -- 172.21.15.47:0/4134312370 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17600acb10 con 0x7f176815f8f0 2024-02-19T15:22:56.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f176e200640 1 -- 172.21.15.47:0/4134312370 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f176815f8f0 msgr2=0x7f176815fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:56.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f176e200640 1 --2- 172.21.15.47:0/4134312370 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f176815f8f0 0x7f176815fd80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f176809fa10 tx=0x7f176001c4b0 comp rx=0 tx=0).stop 2024-02-19T15:22:56.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f176e200640 1 -- 172.21.15.47:0/4134312370 shutdown_connections 2024-02-19T15:22:56.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f176e200640 1 --2- 172.21.15.47:0/4134312370 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f176815f8f0 0x7f176815fd80 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:56.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f176e200640 1 --2- 172.21.15.47:0/4134312370 >> v2:172.21.15.116:3300/0 conn(0x7f17680cec80 0x7f17680d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:56.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f176e200640 1 -- 172.21.15.47:0/4134312370 >> 172.21.15.47:0/4134312370 conn(0x7f176809aed0 msgr2=0x7f17680d1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:56.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f176e200640 1 -- 172.21.15.47:0/4134312370 shutdown_connections 2024-02-19T15:22:56.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f176e200640 1 -- 172.21.15.47:0/4134312370 wait complete. 2024-02-19T15:22:56.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f176e200640 1 Processor -- start 2024-02-19T15:22:56.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f176e200640 1 -- start start 2024-02-19T15:22:56.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f176e200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17680cec80 0x7f17680b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:56.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f176e200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f176815f190 0x7f1768151000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:56.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f176e200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f176815f8f0 0x7f1768159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:56.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f176e200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17680b4f90 con 0x7f176815f190 2024-02-19T15:22:56.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f176e200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17680b5d80 con 0x7f17680cec80 2024-02-19T15:22:56.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f176e200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17680cdf60 con 0x7f176815f8f0 2024-02-19T15:22:56.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f175ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f176815f190 0x7f1768151000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:56.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f175ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f176815f190 0x7f1768151000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39418/0 (socket says 172.21.15.47:39418) 2024-02-19T15:22:56.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f175ea00640 1 -- 172.21.15.47:0/165577182 learned_addr learned my addr 172.21.15.47:0/165577182 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:56.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f1767e00640 1 --2- 172.21.15.47:0/165577182 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f176815f8f0 0x7f1768159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:56.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.216+0000 7f1767400640 1 --2- 172.21.15.47:0/165577182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17680cec80 0x7f17680b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:56.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.220+0000 7f175ea00640 1 -- 172.21.15.47:0/165577182 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f176815f8f0 msgr2=0x7f1768159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:56.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.220+0000 7f175ea00640 1 --2- 172.21.15.47:0/165577182 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f176815f8f0 0x7f1768159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:56.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.220+0000 7f175ea00640 1 -- 172.21.15.47:0/165577182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17680cec80 msgr2=0x7f17680b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:56.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.220+0000 7f175ea00640 1 --2- 172.21.15.47:0/165577182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17680cec80 0x7f17680b4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:56.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.220+0000 7f175ea00640 1 -- 172.21.15.47:0/165577182 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17600094d0 con 0x7f176815f190 2024-02-19T15:22:56.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.220+0000 7f175ea00640 1 --2- 172.21.15.47:0/165577182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f176815f190 0x7f1768151000 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f175000c450 tx=0x7f175000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:56.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.220+0000 7f175fe00640 1 -- 172.21.15.47:0/165577182 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1750012830 con 0x7f176815f190 2024-02-19T15:22:56.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.220+0000 7f175fe00640 1 -- 172.21.15.47:0/165577182 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17500129b0 con 0x7f176815f190 2024-02-19T15:22:56.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.220+0000 7f176e200640 1 -- 172.21.15.47:0/165577182 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1768151540 con 0x7f176815f190 2024-02-19T15:22:56.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.220+0000 7f175fe00640 1 -- 172.21.15.47:0/165577182 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1750012d60 con 0x7f176815f190 2024-02-19T15:22:56.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.220+0000 7f176e200640 1 -- 172.21.15.47:0/165577182 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17680b5450 con 0x7f176815f190 2024-02-19T15:22:56.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.220+0000 7f176e200640 1 -- 172.21.15.47:0/165577182 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17680b5ef0 con 0x7f176815f190 2024-02-19T15:22:56.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.220+0000 7f175fe00640 1 -- 172.21.15.47:0/165577182 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f175001c460 con 0x7f176815f190 2024-02-19T15:22:56.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.220+0000 7f175fe00640 1 --2- 172.21.15.47:0/165577182 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1744066e40 0x7f1744069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:56.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.220+0000 7f175fe00640 1 -- 172.21.15.47:0/165577182 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f17500843f0 con 0x7f176815f190 2024-02-19T15:22:56.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.220+0000 7f1767400640 1 --2- 172.21.15.47:0/165577182 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1744066e40 0x7f1744069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:56.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.224+0000 7f1767400640 1 --2- 172.21.15.47:0/165577182 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1744066e40 0x7f1744069300 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f176815d750 tx=0x7f175800d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:56.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.224+0000 7f175fe00640 1 -- 172.21.15.47:0/165577182 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f17500515d0 con 0x7f176815f190 2024-02-19T15:22:56.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:56.388+0000 7f176e200640 1 -- 172.21.15.47:0/165577182 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f17680cf0c0 con 0x7f176815f190 2024-02-19T15:22:57.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.176+0000 7f175fe00640 1 -- 172.21.15.47:0/165577182 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1750054c90 con 0x7f176815f190 2024-02-19T15:22:57.179 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_metadata' created 2024-02-19T15:22:57.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.176+0000 7f176e200640 1 -- 172.21.15.47:0/165577182 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1744066e40 msgr2=0x7f1744069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:57.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.176+0000 7f176e200640 1 --2- 172.21.15.47:0/165577182 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1744066e40 0x7f1744069300 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f176815d750 tx=0x7f175800d040 comp rx=0 tx=0).stop 2024-02-19T15:22:57.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.176+0000 7f176e200640 1 -- 172.21.15.47:0/165577182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f176815f190 msgr2=0x7f1768151000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:57.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.176+0000 7f176e200640 1 --2- 172.21.15.47:0/165577182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f176815f190 0x7f1768151000 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f175000c450 tx=0x7f175000c910 comp rx=0 tx=0).stop 2024-02-19T15:22:57.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.180+0000 7f176e200640 1 -- 172.21.15.47:0/165577182 shutdown_connections 2024-02-19T15:22:57.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.180+0000 7f176e200640 1 --2- 172.21.15.47:0/165577182 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1744066e40 0x7f1744069300 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:57.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.180+0000 7f176e200640 1 --2- 172.21.15.47:0/165577182 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f176815f8f0 0x7f1768159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:57.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.180+0000 7f176e200640 1 --2- 172.21.15.47:0/165577182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f176815f190 0x7f1768151000 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:57.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.180+0000 7f176e200640 1 --2- 172.21.15.47:0/165577182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17680cec80 0x7f17680b4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:57.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.180+0000 7f176e200640 1 -- 172.21.15.47:0/165577182 >> 172.21.15.47:0/165577182 conn(0x7f176809aed0 msgr2=0x7f17680d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:57.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.180+0000 7f176e200640 1 -- 172.21.15.47:0/165577182 shutdown_connections 2024-02-19T15:22:57.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.180+0000 7f176e200640 1 -- 172.21.15.47:0/165577182 wait complete. 2024-02-19T15:22:57.210 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T15:22:57.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.360+0000 7f7614200640 1 Processor -- start 2024-02-19T15:22:57.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.360+0000 7f7614200640 1 -- start start 2024-02-19T15:22:57.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.360+0000 7f7614200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f760c06c4c0 0x7f760c06e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:57.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.360+0000 7f7614200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f760c0b5650 con 0x7f760c0b5270 2024-02-19T15:22:57.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.360+0000 7f7614200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f760c071560 con 0x7f760c06c920 2024-02-19T15:22:57.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.360+0000 7f7614200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f760c15f7b0 con 0x7f760c06c4c0 2024-02-19T15:22:57.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.360+0000 7f7611800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f760c0b5270 0x7f760c0719f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53822/0 (socket says 172.21.15.47:53822) 2024-02-19T15:22:57.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.360+0000 7f7611800640 1 -- 172.21.15.47:0/260290130 learned_addr learned my addr 172.21.15.47:0/260290130 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:57.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.360+0000 7f7610e00640 1 --2- 172.21.15.47:0/260290130 >> v2:172.21.15.116:3300/0 conn(0x7f760c06c4c0 0x7f760c06e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:57.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.360+0000 7f760be00640 1 -- 172.21.15.47:0/260290130 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265688211 0 0) 0x7f760c0b5650 con 0x7f760c0b5270 2024-02-19T15:22:57.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.360+0000 7f760be00640 1 -- 172.21.15.47:0/260290130 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f4003530 con 0x7f760c0b5270 2024-02-19T15:22:57.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.360+0000 7f760be00640 1 -- 172.21.15.47:0/260290130 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395734256 0 0) 0x7f760c071560 con 0x7f760c06c920 2024-02-19T15:22:57.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.360+0000 7f760be00640 1 -- 172.21.15.47:0/260290130 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f760c0b5650 con 0x7f760c06c920 2024-02-19T15:22:57.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.360+0000 7f7610e00640 1 -- 172.21.15.47:0/260290130 >> v1:172.21.15.116:6789/0 conn(0x7f760c06c920 legacy=0x7f760c0c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:57.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.360+0000 7f7610e00640 1 -- 172.21.15.47:0/260290130 >> v1:172.21.15.47:6789/0 conn(0x7f760c0b5270 legacy=0x7f760c0719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:57.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.360+0000 7f760be00640 1 -- 172.21.15.47:0/260290130 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 53709504 0 0) 0x7f75f4003530 con 0x7f760c0b5270 2024-02-19T15:22:57.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.360+0000 7f7610e00640 1 -- 172.21.15.47:0/260290130 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f760c072320 con 0x7f760c06c4c0 2024-02-19T15:22:57.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.364+0000 7f7610e00640 1 --2- 172.21.15.47:0/260290130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f760c06c4c0 0x7f760c06e020 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f75f80097c0 tx=0x7f75f801c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6ce95b5ce56d570 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:57.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.364+0000 7f760be00640 1 -- 172.21.15.47:0/260290130 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7600001680 con 0x7f760c06c4c0 2024-02-19T15:22:57.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.364+0000 7f760be00640 1 -- 172.21.15.47:0/260290130 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f80a7050 con 0x7f760c06c4c0 2024-02-19T15:22:57.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.364+0000 7f760be00640 1 -- 172.21.15.47:0/260290130 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f80ab620 con 0x7f760c06c4c0 2024-02-19T15:22:57.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.364+0000 7f7614200640 1 -- 172.21.15.47:0/260290130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f760c06c4c0 msgr2=0x7f760c06e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:57.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.364+0000 7f7614200640 1 --2- 172.21.15.47:0/260290130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f760c06c4c0 0x7f760c06e020 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f75f80097c0 tx=0x7f75f801c3c0 comp rx=0 tx=0).stop 2024-02-19T15:22:57.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.364+0000 7f7614200640 1 -- 172.21.15.47:0/260290130 shutdown_connections 2024-02-19T15:22:57.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.364+0000 7f7614200640 1 --2- 172.21.15.47:0/260290130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f760c06c4c0 0x7f760c06e020 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:57.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.364+0000 7f7614200640 1 -- 172.21.15.47:0/260290130 >> 172.21.15.47:0/260290130 conn(0x7f760c0cc3f0 msgr2=0x7f760c0cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:57.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.364+0000 7f7614200640 1 -- 172.21.15.47:0/260290130 shutdown_connections 2024-02-19T15:22:57.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.364+0000 7f7614200640 1 -- 172.21.15.47:0/260290130 wait complete. 2024-02-19T15:22:57.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.364+0000 7f7614200640 1 Processor -- start 2024-02-19T15:22:57.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.364+0000 7f7614200640 1 -- start start 2024-02-19T15:22:57.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7614200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f760c06c4c0 0x7f760c0c5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:57.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7614200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f760c06c920 0x7f760c0c2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:57.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7614200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f760c0b5270 0x7f760c0cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:57.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7614200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c0c5180 con 0x7f760c06c920 2024-02-19T15:22:57.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7614200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c0c5a00 con 0x7f760c0b5270 2024-02-19T15:22:57.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7614200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f760c0c3120 con 0x7f760c06c4c0 2024-02-19T15:22:57.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7610e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f760c06c920 0x7f760c0c2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:57.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7610e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f760c06c920 0x7f760c0c2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39424/0 (socket says 172.21.15.47:39424) 2024-02-19T15:22:57.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7610e00640 1 -- 172.21.15.47:0/2167251572 learned_addr learned my addr 172.21.15.47:0/2167251572 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:57.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7612200640 1 --2- 172.21.15.47:0/2167251572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f760c0b5270 0x7f760c0cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:57.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7611800640 1 --2- 172.21.15.47:0/2167251572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f760c06c4c0 0x7f760c0c5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:57.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7610e00640 1 -- 172.21.15.47:0/2167251572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f760c06c4c0 msgr2=0x7f760c0c5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:57.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7610e00640 1 --2- 172.21.15.47:0/2167251572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f760c06c4c0 0x7f760c0c5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:57.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7610e00640 1 -- 172.21.15.47:0/2167251572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f760c0b5270 msgr2=0x7f760c0cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:57.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7610e00640 1 --2- 172.21.15.47:0/2167251572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f760c0b5270 0x7f760c0cdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:57.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7610e00640 1 -- 172.21.15.47:0/2167251572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f8009420 con 0x7f760c06c920 2024-02-19T15:22:57.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7611800640 1 --2- 172.21.15.47:0/2167251572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f760c06c4c0 0x7f760c0c5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:22:57.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7610e00640 1 --2- 172.21.15.47:0/2167251572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f760c06c920 0x7f760c0c2be0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f75f80ae040 tx=0x7f75f8019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:57.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7609600640 1 -- 172.21.15.47:0/2167251572 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f8004020 con 0x7f760c06c920 2024-02-19T15:22:57.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7609600640 1 -- 172.21.15.47:0/2167251572 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f8004180 con 0x7f760c06c920 2024-02-19T15:22:57.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7614200640 1 -- 172.21.15.47:0/2167251572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f760c0c39d0 con 0x7f760c06c920 2024-02-19T15:22:57.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7609600640 1 -- 172.21.15.47:0/2167251572 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f80044f0 con 0x7f760c06c920 2024-02-19T15:22:57.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7614200640 1 -- 172.21.15.47:0/2167251572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f760c0c5690 con 0x7f760c06c920 2024-02-19T15:22:57.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.368+0000 7f7614200640 1 -- 172.21.15.47:0/2167251572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f760c0c5b70 con 0x7f760c06c920 2024-02-19T15:22:57.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.372+0000 7f7609600640 1 -- 172.21.15.47:0/2167251572 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f75f80b2040 con 0x7f760c06c920 2024-02-19T15:22:57.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.372+0000 7f7609600640 1 --2- 172.21.15.47:0/2167251572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f75ec066e40 0x7f75ec069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:57.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.372+0000 7f7609600640 1 -- 172.21.15.47:0/2167251572 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f75f811d8a0 con 0x7f760c06c920 2024-02-19T15:22:57.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.372+0000 7f7611800640 1 --2- 172.21.15.47:0/2167251572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f75ec066e40 0x7f75ec069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:57.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.372+0000 7f7611800640 1 --2- 172.21.15.47:0/2167251572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f75ec066e40 0x7f75ec069300 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7600001000 tx=0x7f7600009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:57.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.376+0000 7f7609600640 1 -- 172.21.15.47:0/2167251572 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f75f80ea7d0 con 0x7f760c06c920 2024-02-19T15:22:57.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:57.536+0000 7f7614200640 1 -- 172.21.15.47:0/2167251572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f760c075360 con 0x7f760c06c920 2024-02-19T15:22:58.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.176+0000 7f7609600640 1 -- 172.21.15.47:0/2167251572 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) v1 ==== 170+0+0 (secure 0 0 0) 0x7f75f80bb0d0 con 0x7f760c06c920 2024-02-19T15:22:58.182 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_data' created 2024-02-19T15:22:58.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.180+0000 7f7614200640 1 -- 172.21.15.47:0/2167251572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f75ec066e40 msgr2=0x7f75ec069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:58.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.180+0000 7f7614200640 1 --2- 172.21.15.47:0/2167251572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f75ec066e40 0x7f75ec069300 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7600001000 tx=0x7f7600009040 comp rx=0 tx=0).stop 2024-02-19T15:22:58.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.180+0000 7f7614200640 1 -- 172.21.15.47:0/2167251572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f760c06c920 msgr2=0x7f760c0c2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:58.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.180+0000 7f7614200640 1 --2- 172.21.15.47:0/2167251572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f760c06c920 0x7f760c0c2be0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f75f80ae040 tx=0x7f75f8019160 comp rx=0 tx=0).stop 2024-02-19T15:22:58.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.180+0000 7f7614200640 1 -- 172.21.15.47:0/2167251572 shutdown_connections 2024-02-19T15:22:58.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.180+0000 7f7614200640 1 --2- 172.21.15.47:0/2167251572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f75ec066e40 0x7f75ec069300 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:58.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.180+0000 7f7614200640 1 --2- 172.21.15.47:0/2167251572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f760c0b5270 0x7f760c0cdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:58.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.180+0000 7f7614200640 1 --2- 172.21.15.47:0/2167251572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f760c06c920 0x7f760c0c2be0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:58.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.180+0000 7f7614200640 1 --2- 172.21.15.47:0/2167251572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f760c06c4c0 0x7f760c0c5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:58.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.180+0000 7f7614200640 1 -- 172.21.15.47:0/2167251572 >> 172.21.15.47:0/2167251572 conn(0x7f760c0cc3f0 msgr2=0x7f760c0af760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:58.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.180+0000 7f7614200640 1 -- 172.21.15.47:0/2167251572 shutdown_connections 2024-02-19T15:22:58.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.180+0000 7f7614200640 1 -- 172.21.15.47:0/2167251572 wait complete. 2024-02-19T15:22:58.226 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T15:22:58.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.368+0000 7f4430200640 1 Processor -- start 2024-02-19T15:22:58.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.368+0000 7f4430200640 1 -- start start 2024-02-19T15:22:58.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.368+0000 7f4430200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f442815af10 0x7f442806e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:58.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.368+0000 7f4430200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f44280cd880 0x7f44280cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:58.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.368+0000 7f4430200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4428071c70 con 0x7f44280ca8f0 2024-02-19T15:22:58.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.368+0000 7f4430200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f442806e8a0 con 0x7f44280cd880 2024-02-19T15:22:58.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.368+0000 7f4430200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f442806cef0 con 0x7f442815af10 2024-02-19T15:22:58.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.372+0000 7f442e200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f44280cd880 0x7f44280cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:58.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.372+0000 7f442d800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f442815af10 0x7f442806e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:58.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.372+0000 7f442d800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f442815af10 0x7f442806e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:49422/0 (socket says 172.21.15.47:49422) 2024-02-19T15:22:58.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.372+0000 7f442e200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f44280cd880 0x7f44280cdd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:47354/0 (socket says 172.21.15.47:47354) 2024-02-19T15:22:58.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.372+0000 7f442d800640 1 -- 172.21.15.47:0/2508136826 learned_addr learned my addr 172.21.15.47:0/2508136826 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:58.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.372+0000 7f442d800640 1 -- 172.21.15.47:0/2508136826 >> v2:172.21.15.116:3300/0 conn(0x7f44280cd880 msgr2=0x7f44280cdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:58.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.372+0000 7f442d800640 1 --2- 172.21.15.47:0/2508136826 >> v2:172.21.15.116:3300/0 conn(0x7f44280cd880 0x7f44280cdd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:58.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.372+0000 7f442d800640 1 -- 172.21.15.47:0/2508136826 >> v1:172.21.15.116:6789/0 conn(0x7f44280ca8f0 legacy=0x7f44280cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:58.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.372+0000 7f442d800640 1 -- 172.21.15.47:0/2508136826 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44280ca610 con 0x7f442815af10 2024-02-19T15:22:58.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.372+0000 7f442e200640 1 --2- 172.21.15.47:0/2508136826 >> v2:172.21.15.116:3300/0 conn(0x7f44280cd880 0x7f44280cdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:58.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.372+0000 7f442d800640 1 --2- 172.21.15.47:0/2508136826 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f442815af10 0x7f442806e360 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f441c009870 tx=0x7f441c01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0606a59924df09a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:58.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.376+0000 7f4427e00640 1 -- 172.21.15.47:0/2508136826 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441c0a6040 con 0x7f442815af10 2024-02-19T15:22:58.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.376+0000 7f4427e00640 1 -- 172.21.15.47:0/2508136826 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f441c0aa600 con 0x7f442815af10 2024-02-19T15:22:58.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.376+0000 7f4427e00640 1 -- 172.21.15.47:0/2508136826 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441c0aa990 con 0x7f442815af10 2024-02-19T15:22:58.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.376+0000 7f4430200640 1 -- 172.21.15.47:0/2508136826 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f442815af10 msgr2=0x7f442806e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:58.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.376+0000 7f4430200640 1 --2- 172.21.15.47:0/2508136826 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f442815af10 0x7f442806e360 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f441c009870 tx=0x7f441c01c4a0 comp rx=0 tx=0).stop 2024-02-19T15:22:58.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.376+0000 7f4430200640 1 -- 172.21.15.47:0/2508136826 shutdown_connections 2024-02-19T15:22:58.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.376+0000 7f4430200640 1 --2- 172.21.15.47:0/2508136826 >> v2:172.21.15.116:3300/0 conn(0x7f44280cd880 0x7f44280cdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:58.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.376+0000 7f4430200640 1 --2- 172.21.15.47:0/2508136826 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f442815af10 0x7f442806e360 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:58.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.376+0000 7f4430200640 1 -- 172.21.15.47:0/2508136826 >> 172.21.15.47:0/2508136826 conn(0x7f44280cfc20 msgr2=0x7f44280d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:58.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.376+0000 7f4430200640 1 -- 172.21.15.47:0/2508136826 shutdown_connections 2024-02-19T15:22:58.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.376+0000 7f4430200640 1 -- 172.21.15.47:0/2508136826 wait complete. 2024-02-19T15:22:58.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.376+0000 7f4430200640 1 Processor -- start 2024-02-19T15:22:58.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.376+0000 7f4430200640 1 -- start start 2024-02-19T15:22:58.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.376+0000 7f4430200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44280ca8f0 0x7f44280b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:58.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.376+0000 7f4430200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44280cd880 0x7f44280c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:58.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.376+0000 7f4430200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f442815af10 0x7f44281536c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:58.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.376+0000 7f4430200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44280c78c0 con 0x7f44280cd880 2024-02-19T15:22:58.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.376+0000 7f4430200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44280c7ba0 con 0x7f44280ca8f0 2024-02-19T15:22:58.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.376+0000 7f4430200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44280c7e80 con 0x7f442815af10 2024-02-19T15:22:58.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.380+0000 7f442d800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44280ca8f0 0x7f44280b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:58.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.380+0000 7f442ce00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44280cd880 0x7f44280c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:58.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.380+0000 7f442d800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44280ca8f0 0x7f44280b4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:47368/0 (socket says 172.21.15.47:47368) 2024-02-19T15:22:58.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.380+0000 7f442d800640 1 -- 172.21.15.47:0/3895271389 learned_addr learned my addr 172.21.15.47:0/3895271389 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:58.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.380+0000 7f442e200640 1 --2- 172.21.15.47:0/3895271389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f442815af10 0x7f44281536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:58.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.380+0000 7f442d800640 1 -- 172.21.15.47:0/3895271389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f442815af10 msgr2=0x7f44281536c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:58.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.380+0000 7f442d800640 1 --2- 172.21.15.47:0/3895271389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f442815af10 0x7f44281536c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:58.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.380+0000 7f442d800640 1 -- 172.21.15.47:0/3895271389 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44280cd880 msgr2=0x7f44280c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:58.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.380+0000 7f442d800640 1 --2- 172.21.15.47:0/3895271389 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44280cd880 0x7f44280c9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:58.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.380+0000 7f442d800640 1 -- 172.21.15.47:0/3895271389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4420009a70 con 0x7f44280ca8f0 2024-02-19T15:22:58.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.380+0000 7f442ce00640 1 --2- 172.21.15.47:0/3895271389 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44280cd880 0x7f44280c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:58.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.380+0000 7f442e200640 1 --2- 172.21.15.47:0/3895271389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f442815af10 0x7f44281536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:22:58.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.380+0000 7f442d800640 1 --2- 172.21.15.47:0/3895271389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44280ca8f0 0x7f44280b4ea0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f441c01ab50 tx=0x7f441c01ace0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:58.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.380+0000 7f4425600640 1 -- 172.21.15.47:0/3895271389 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441c0b3600 con 0x7f44280ca8f0 2024-02-19T15:22:58.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.380+0000 7f4430200640 1 -- 172.21.15.47:0/3895271389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441c0094d0 con 0x7f44280ca8f0 2024-02-19T15:22:58.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.380+0000 7f4425600640 1 -- 172.21.15.47:0/3895271389 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f441c0b3780 con 0x7f44280ca8f0 2024-02-19T15:22:58.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.380+0000 7f4425600640 1 -- 172.21.15.47:0/3895271389 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441c0a6020 con 0x7f44280ca8f0 2024-02-19T15:22:58.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.380+0000 7f4430200640 1 -- 172.21.15.47:0/3895271389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4428153c00 con 0x7f44280ca8f0 2024-02-19T15:22:58.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.380+0000 7f4430200640 1 -- 172.21.15.47:0/3895271389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44280c9150 con 0x7f44280ca8f0 2024-02-19T15:22:58.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.380+0000 7f4425600640 1 -- 172.21.15.47:0/3895271389 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f44280cea60 con 0x7f44280ca8f0 2024-02-19T15:22:58.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.384+0000 7f4425600640 1 --2- 172.21.15.47:0/3895271389 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4408066e40 0x7f4408069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:58.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.384+0000 7f4425600640 1 -- 172.21.15.47:0/3895271389 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f441c11eae0 con 0x7f44280ca8f0 2024-02-19T15:22:58.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.384+0000 7f442ce00640 1 --2- 172.21.15.47:0/3895271389 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4408066e40 0x7f4408069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:58.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.384+0000 7f442ce00640 1 --2- 172.21.15.47:0/3895271389 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4408066e40 0x7f4408069300 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f442815e460 tx=0x7f4414009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:58.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.388+0000 7f4425600640 1 -- 172.21.15.47:0/3895271389 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f441c0eb8c0 con 0x7f44280ca8f0 2024-02-19T15:22:58.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:58.584+0000 7f4430200640 1 -- 172.21.15.47:0/3895271389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f44280c7ce0 con 0x7f44280ca8f0 2024-02-19T15:22:59.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.224+0000 7f4425600640 1 -- 172.21.15.47:0/3895271389 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7f44280c7ce0 con 0x7f44280ca8f0 2024-02-19T15:22:59.229 INFO:teuthology.orchestra.run.smithi047.stderr:new fs with metadata pool 4 and data pool 5 2024-02-19T15:22:59.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.228+0000 7f4430200640 1 -- 172.21.15.47:0/3895271389 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4408066e40 msgr2=0x7f4408069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:59.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.228+0000 7f4430200640 1 --2- 172.21.15.47:0/3895271389 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4408066e40 0x7f4408069300 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f442815e460 tx=0x7f4414009040 comp rx=0 tx=0).stop 2024-02-19T15:22:59.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.228+0000 7f4430200640 1 -- 172.21.15.47:0/3895271389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44280ca8f0 msgr2=0x7f44280b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:59.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.228+0000 7f4430200640 1 --2- 172.21.15.47:0/3895271389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44280ca8f0 0x7f44280b4ea0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f441c01ab50 tx=0x7f441c01ace0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.228+0000 7f4430200640 1 -- 172.21.15.47:0/3895271389 shutdown_connections 2024-02-19T15:22:59.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.228+0000 7f4430200640 1 --2- 172.21.15.47:0/3895271389 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4408066e40 0x7f4408069300 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.228+0000 7f4430200640 1 --2- 172.21.15.47:0/3895271389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f442815af10 0x7f44281536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.228+0000 7f4430200640 1 --2- 172.21.15.47:0/3895271389 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44280cd880 0x7f44280c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.228+0000 7f4430200640 1 --2- 172.21.15.47:0/3895271389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44280ca8f0 0x7f44280b4ea0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.228+0000 7f4430200640 1 -- 172.21.15.47:0/3895271389 >> 172.21.15.47:0/3895271389 conn(0x7f44280cfc20 msgr2=0x7f44280c5c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:59.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.228+0000 7f4430200640 1 -- 172.21.15.47:0/3895271389 shutdown_connections 2024-02-19T15:22:59.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.228+0000 7f4430200640 1 -- 172.21.15.47:0/3895271389 wait complete. 2024-02-19T15:22:59.261 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:59.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.400+0000 7f2c81800640 1 Processor -- start 2024-02-19T15:22:59.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.400+0000 7f2c81800640 1 -- start start 2024-02-19T15:22:59.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.400+0000 7f2c81800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2c7c15b1a0 0x7f2c7c15b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:59.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.400+0000 7f2c81800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2c7c071350 0x7f2c7c15bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:59.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.400+0000 7f2c81800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7c15c000 con 0x7f2c7c06c9a0 2024-02-19T15:22:59.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.400+0000 7f2c81800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c7c0718c0 con 0x7f2c7c071350 2024-02-19T15:22:59.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.400+0000 7f2c81800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c7c071ba0 con 0x7f2c7c15b1a0 2024-02-19T15:22:59.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.400+0000 7f2c7a000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2c7c071350 0x7f2c7c15bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:59.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.400+0000 7f2c7a000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2c7c071350 0x7f2c7c15bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39438/0 (socket says 172.21.15.47:39438) 2024-02-19T15:22:59.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.400+0000 7f2c7a000640 1 -- 172.21.15.47:0/1656578536 learned_addr learned my addr 172.21.15.47:0/1656578536 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:59.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.400+0000 7f2c7aa00640 1 --2- 172.21.15.47:0/1656578536 >> v2:172.21.15.116:3300/0 conn(0x7f2c7c15b1a0 0x7f2c7c15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:59.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.400+0000 7f2c7a000640 1 -- 172.21.15.47:0/1656578536 >> v2:172.21.15.116:3300/0 conn(0x7f2c7c15b1a0 msgr2=0x7f2c7c15b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:22:59.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.400+0000 7f2c7a000640 1 --2- 172.21.15.47:0/1656578536 >> v2:172.21.15.116:3300/0 conn(0x7f2c7c15b1a0 0x7f2c7c15b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.400+0000 7f2c7a000640 1 -- 172.21.15.47:0/1656578536 >> v1:172.21.15.116:6789/0 conn(0x7f2c7c06c9a0 legacy=0x7f2c7c15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:59.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.400+0000 7f2c7a000640 1 -- 172.21.15.47:0/1656578536 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c071e80 con 0x7f2c7c071350 2024-02-19T15:22:59.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.404+0000 7f2c7a000640 1 --2- 172.21.15.47:0/1656578536 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c7c071350 0x7f2c7c15bac0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2c68009870 tx=0x7f2c6801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=798c0cf04d1c93f1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:59.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.404+0000 7f2c79600640 1 -- 172.21.15.47:0/1656578536 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c680aa670 con 0x7f2c7c071350 2024-02-19T15:22:59.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.404+0000 7f2c79600640 1 -- 172.21.15.47:0/1656578536 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c680aa7d0 con 0x7f2c7c071350 2024-02-19T15:22:59.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.404+0000 7f2c79600640 1 -- 172.21.15.47:0/1656578536 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c680aab60 con 0x7f2c7c071350 2024-02-19T15:22:59.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.404+0000 7f2c81800640 1 -- 172.21.15.47:0/1656578536 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c7c071350 msgr2=0x7f2c7c15bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:59.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.404+0000 7f2c81800640 1 --2- 172.21.15.47:0/1656578536 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c7c071350 0x7f2c7c15bac0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2c68009870 tx=0x7f2c6801c4f0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.404+0000 7f2c81800640 1 -- 172.21.15.47:0/1656578536 shutdown_connections 2024-02-19T15:22:59.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.404+0000 7f2c81800640 1 --2- 172.21.15.47:0/1656578536 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c7c071350 0x7f2c7c15bac0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.404+0000 7f2c81800640 1 --2- 172.21.15.47:0/1656578536 >> v2:172.21.15.116:3300/0 conn(0x7f2c7c15b1a0 0x7f2c7c15b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.404+0000 7f2c81800640 1 -- 172.21.15.47:0/1656578536 >> 172.21.15.47:0/1656578536 conn(0x7f2c7c0b16a0 msgr2=0x7f2c7c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:59.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.404+0000 7f2c81800640 1 -- 172.21.15.47:0/1656578536 shutdown_connections 2024-02-19T15:22:59.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.404+0000 7f2c81800640 1 -- 172.21.15.47:0/1656578536 wait complete. 2024-02-19T15:22:59.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.404+0000 7f2c81800640 1 Processor -- start 2024-02-19T15:22:59.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c81800640 1 -- start start 2024-02-19T15:22:59.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c81800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c7c06c9a0 0x7f2c7c155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:59.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c81800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c7c071350 0x7f2c7c0aff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:59.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c81800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c7c15b1a0 0x7f2c7c0b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:59.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c81800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c15ded0 con 0x7f2c7c06c9a0 2024-02-19T15:22:59.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c81800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c0ad260 con 0x7f2c7c071350 2024-02-19T15:22:59.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c81800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c0ad540 con 0x7f2c7c15b1a0 2024-02-19T15:22:59.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c7aa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c7c06c9a0 0x7f2c7c155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:59.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c7aa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c7c06c9a0 0x7f2c7c155820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39452/0 (socket says 172.21.15.47:39452) 2024-02-19T15:22:59.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c7aa00640 1 -- 172.21.15.47:0/2461916484 learned_addr learned my addr 172.21.15.47:0/2461916484 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:59.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c7a000640 1 --2- 172.21.15.47:0/2461916484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c7c071350 0x7f2c7c0aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:59.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c7b400640 1 --2- 172.21.15.47:0/2461916484 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c7c15b1a0 0x7f2c7c0b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:59.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c7aa00640 1 -- 172.21.15.47:0/2461916484 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c7c15b1a0 msgr2=0x7f2c7c0b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:59.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c7aa00640 1 --2- 172.21.15.47:0/2461916484 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c7c15b1a0 0x7f2c7c0b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c7aa00640 1 -- 172.21.15.47:0/2461916484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c7c071350 msgr2=0x7f2c7c0aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:59.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c7aa00640 1 --2- 172.21.15.47:0/2461916484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c7c071350 0x7f2c7c0aff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c7aa00640 1 -- 172.21.15.47:0/2461916484 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c680094d0 con 0x7f2c7c06c9a0 2024-02-19T15:22:59.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c7aa00640 1 --2- 172.21.15.47:0/2461916484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c7c06c9a0 0x7f2c7c155820 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c00d800 tx=0x7f2c6c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:59.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c72a00640 1 -- 172.21.15.47:0/2461916484 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c00b6c0 con 0x7f2c7c06c9a0 2024-02-19T15:22:59.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c72a00640 1 -- 172.21.15.47:0/2461916484 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c6c00b820 con 0x7f2c7c06c9a0 2024-02-19T15:22:59.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c72a00640 1 -- 172.21.15.47:0/2461916484 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c00bb90 con 0x7f2c7c06c9a0 2024-02-19T15:22:59.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c81800640 1 -- 172.21.15.47:0/2461916484 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7c0ad820 con 0x7f2c7c06c9a0 2024-02-19T15:22:59.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c81800640 1 -- 172.21.15.47:0/2461916484 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7c155d60 con 0x7f2c7c06c9a0 2024-02-19T15:22:59.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.408+0000 7f2c81800640 1 -- 172.21.15.47:0/2461916484 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7c075360 con 0x7f2c7c06c9a0 2024-02-19T15:22:59.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.412+0000 7f2c72a00640 1 -- 172.21.15.47:0/2461916484 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2c6c00bcf0 con 0x7f2c7c06c9a0 2024-02-19T15:22:59.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.412+0000 7f2c72a00640 1 --2- 172.21.15.47:0/2461916484 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2c5c066e70 0x7f2c5c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:59.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.412+0000 7f2c72a00640 1 -- 172.21.15.47:0/2461916484 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c6c084d90 con 0x7f2c7c06c9a0 2024-02-19T15:22:59.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.412+0000 7f2c7a000640 1 --2- 172.21.15.47:0/2461916484 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2c5c066e70 0x7f2c5c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:59.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.412+0000 7f2c7a000640 1 --2- 172.21.15.47:0/2461916484 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2c5c066e70 0x7f2c5c069330 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2c680ae000 tx=0x7f2c680b5040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:59.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.416+0000 7f2c72a00640 1 -- 172.21.15.47:0/2461916484 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2c6c051cc0 con 0x7f2c7c06c9a0 2024-02-19T15:22:59.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.580+0000 7f2c81800640 1 -- 172.21.15.47:0/2461916484 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c7c0cf660 con 0x7f2c7c06c9a0 2024-02-19T15:22:59.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.580+0000 7f2c72a00640 1 -- 172.21.15.47:0/2461916484 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16694 (secure 0 0 0) 0x7f2c6c055380 con 0x7f2c7c06c9a0 2024-02-19T15:22:59.586 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:59.587 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":33,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:58.589746+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:56.436273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:57.586982+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.584+0000 7f2c81800640 1 -- 172.21.15.47:0/2461916484 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2c5c066e70 msgr2=0x7f2c5c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.584+0000 7f2c81800640 1 --2- 172.21.15.47:0/2461916484 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2c5c066e70 0x7f2c5c069330 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2c680ae000 tx=0x7f2c680b5040 comp rx=0 tx=0).stop 2024-02-19T15:22:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.584+0000 7f2c81800640 1 -- 172.21.15.47:0/2461916484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c7c06c9a0 msgr2=0x7f2c7c155820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.584+0000 7f2c81800640 1 --2- 172.21.15.47:0/2461916484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c7c06c9a0 0x7f2c7c155820 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c00d800 tx=0x7f2c6c00dcc0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.584+0000 7f2c81800640 1 -- 172.21.15.47:0/2461916484 shutdown_connections 2024-02-19T15:22:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.584+0000 7f2c81800640 1 --2- 172.21.15.47:0/2461916484 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2c5c066e70 0x7f2c5c069330 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.584+0000 7f2c81800640 1 --2- 172.21.15.47:0/2461916484 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c7c15b1a0 0x7f2c7c0b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.584+0000 7f2c81800640 1 --2- 172.21.15.47:0/2461916484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c7c071350 0x7f2c7c0aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.584+0000 7f2c81800640 1 --2- 172.21.15.47:0/2461916484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c7c06c9a0 0x7f2c7c155820 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.584+0000 7f2c81800640 1 -- 172.21.15.47:0/2461916484 >> 172.21.15.47:0/2461916484 conn(0x7f2c7c0b16a0 msgr2=0x7f2c7c0b0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.584+0000 7f2c81800640 1 -- 172.21.15.47:0/2461916484 shutdown_connections 2024-02-19T15:22:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.584+0000 7f2c81800640 1 -- 172.21.15.47:0/2461916484 wait complete. 2024-02-19T15:22:59.608 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:22:59.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.744+0000 7fdd9fa00640 1 Processor -- start 2024-02-19T15:22:59.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.744+0000 7fdd9fa00640 1 -- start start 2024-02-19T15:22:59.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.744+0000 7fdd9fa00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9806b330 0x7fdd980b4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:59.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.744+0000 7fdd9fa00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd9806b7c0 con 0x7fdd9806aeb0 2024-02-19T15:22:59.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.744+0000 7fdd9fa00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd98071a00 con 0x7fdd98145740 2024-02-19T15:22:59.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.744+0000 7fdd9fa00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd98071ce0 con 0x7fdd9806b330 2024-02-19T15:22:59.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.744+0000 7fdd97e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fdd9806aeb0 0x7fdd980b3a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53832/0 (socket says 172.21.15.47:53832) 2024-02-19T15:22:59.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.744+0000 7fdd97e00640 1 -- 172.21.15.47:0/1047655431 learned_addr learned my addr 172.21.15.47:0/1047655431 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:22:59.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.744+0000 7fdd9da00640 1 --2- 172.21.15.47:0/1047655431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9806b330 0x7fdd980b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:59.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.744+0000 7fdd97400640 1 -- 172.21.15.47:0/1047655431 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008193362 0 0) 0x7fdd9806b7c0 con 0x7fdd9806aeb0 2024-02-19T15:22:59.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.744+0000 7fdd97400640 1 -- 172.21.15.47:0/1047655431 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd80003530 con 0x7fdd9806aeb0 2024-02-19T15:22:59.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.744+0000 7fdd97400640 1 -- 172.21.15.47:0/1047655431 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467162971 0 0) 0x7fdd98071a00 con 0x7fdd98145740 2024-02-19T15:22:59.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.744+0000 7fdd97400640 1 -- 172.21.15.47:0/1047655431 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd9806b7c0 con 0x7fdd98145740 2024-02-19T15:22:59.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.744+0000 7fdd9da00640 1 -- 172.21.15.47:0/1047655431 >> v1:172.21.15.116:6789/0 conn(0x7fdd98145740 legacy=0x7fdd980cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:59.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.744+0000 7fdd9da00640 1 -- 172.21.15.47:0/1047655431 >> v1:172.21.15.47:6789/0 conn(0x7fdd9806aeb0 legacy=0x7fdd980b3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:59.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.744+0000 7fdd9da00640 1 -- 172.21.15.47:0/1047655431 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd98071fc0 con 0x7fdd9806b330 2024-02-19T15:22:59.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.748+0000 7fdd9da00640 1 --2- 172.21.15.47:0/1047655431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9806b330 0x7fdd980b4130 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdd90009dc0 tx=0x7fdd9001c840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fd676697296994e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:59.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.748+0000 7fdd97400640 1 -- 172.21.15.47:0/1047655431 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd900a5900 con 0x7fdd9806b330 2024-02-19T15:22:59.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.748+0000 7fdd97400640 1 -- 172.21.15.47:0/1047655431 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd900a5a60 con 0x7fdd9806b330 2024-02-19T15:22:59.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.748+0000 7fdd97400640 1 -- 172.21.15.47:0/1047655431 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd900a5df0 con 0x7fdd9806b330 2024-02-19T15:22:59.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.748+0000 7fdd9fa00640 1 -- 172.21.15.47:0/1047655431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9806b330 msgr2=0x7fdd980b4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:59.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.748+0000 7fdd9fa00640 1 --2- 172.21.15.47:0/1047655431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9806b330 0x7fdd980b4130 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdd90009dc0 tx=0x7fdd9001c840 comp rx=0 tx=0).stop 2024-02-19T15:22:59.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.748+0000 7fdd9fa00640 1 -- 172.21.15.47:0/1047655431 shutdown_connections 2024-02-19T15:22:59.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.748+0000 7fdd9fa00640 1 --2- 172.21.15.47:0/1047655431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9806b330 0x7fdd980b4130 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.748+0000 7fdd9fa00640 1 -- 172.21.15.47:0/1047655431 >> 172.21.15.47:0/1047655431 conn(0x7fdd980d0a50 msgr2=0x7fdd980d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:59.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.748+0000 7fdd9fa00640 1 -- 172.21.15.47:0/1047655431 shutdown_connections 2024-02-19T15:22:59.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.748+0000 7fdd9fa00640 1 -- 172.21.15.47:0/1047655431 wait complete. 2024-02-19T15:22:59.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9fa00640 1 Processor -- start 2024-02-19T15:22:59.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9fa00640 1 -- start start 2024-02-19T15:22:59.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9fa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd9806aeb0 0x7fdd980cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9fa00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9806b330 0x7fdd980cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9fa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd98145740 0x7fdd9815a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9fa00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd98141580 con 0x7fdd98145740 2024-02-19T15:22:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9fa00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9806f350 con 0x7fdd9806aeb0 2024-02-19T15:22:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9fa00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd98140c70 con 0x7fdd9806b330 2024-02-19T15:22:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9da00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd98145740 0x7fdd9815a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9da00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd98145740 0x7fdd9815a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39456/0 (socket says 172.21.15.47:39456) 2024-02-19T15:22:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9da00640 1 -- 172.21.15.47:0/4153401278 learned_addr learned my addr 172.21.15.47:0/4153401278 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:22:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd97e00640 1 --2- 172.21.15.47:0/4153401278 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9806b330 0x7fdd980cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9d000640 1 --2- 172.21.15.47:0/4153401278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd9806aeb0 0x7fdd980cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9da00640 1 -- 172.21.15.47:0/4153401278 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9806b330 msgr2=0x7fdd980cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9da00640 1 --2- 172.21.15.47:0/4153401278 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9806b330 0x7fdd980cceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9da00640 1 -- 172.21.15.47:0/4153401278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd9806aeb0 msgr2=0x7fdd980cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9da00640 1 --2- 172.21.15.47:0/4153401278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd9806aeb0 0x7fdd980cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9da00640 1 -- 172.21.15.47:0/4153401278 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd90009a70 con 0x7fdd98145740 2024-02-19T15:22:59.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9d000640 1 --2- 172.21.15.47:0/4153401278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd9806aeb0 0x7fdd980cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:22:59.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9da00640 1 --2- 172.21.15.47:0/4153401278 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd98145740 0x7fdd9815a7e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fdd900b0040 tx=0x7fdd900195e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:59.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd94c00640 1 -- 172.21.15.47:0/4153401278 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd900b9650 con 0x7fdd98145740 2024-02-19T15:22:59.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd94c00640 1 -- 172.21.15.47:0/4153401278 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd900b97b0 con 0x7fdd98145740 2024-02-19T15:22:59.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9fa00640 1 -- 172.21.15.47:0/4153401278 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9813f8d0 con 0x7fdd98145740 2024-02-19T15:22:59.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd94c00640 1 -- 172.21.15.47:0/4153401278 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd900b9b60 con 0x7fdd98145740 2024-02-19T15:22:59.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9fa00640 1 -- 172.21.15.47:0/4153401278 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd98140ec0 con 0x7fdd98145740 2024-02-19T15:22:59.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.752+0000 7fdd9fa00640 1 -- 172.21.15.47:0/4153401278 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd981416c0 con 0x7fdd98145740 2024-02-19T15:22:59.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.756+0000 7fdd94c00640 1 -- 172.21.15.47:0/4153401278 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fdd900b9cc0 con 0x7fdd98145740 2024-02-19T15:22:59.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.756+0000 7fdd94c00640 1 --2- 172.21.15.47:0/4153401278 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdd78066e40 0x7fdd78069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:22:59.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.756+0000 7fdd94c00640 1 -- 172.21.15.47:0/4153401278 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd9011fc60 con 0x7fdd98145740 2024-02-19T15:22:59.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.756+0000 7fdd9d000640 1 --2- 172.21.15.47:0/4153401278 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdd78066e40 0x7fdd78069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:22:59.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.756+0000 7fdd9d000640 1 --2- 172.21.15.47:0/4153401278 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdd78066e40 0x7fdd78069300 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0011f0 tx=0x7fdd8c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:22:59.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.760+0000 7fdd94c00640 1 -- 172.21.15.47:0/4153401278 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fdd981416c0 con 0x7fdd98145740 2024-02-19T15:22:59.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.924+0000 7fdd9fa00640 1 -- 172.21.15.47:0/4153401278 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd981416c0 con 0x7fdd98145740 2024-02-19T15:22:59.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.924+0000 7fdd94c00640 1 -- 172.21.15.47:0/4153401278 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16694 (secure 0 0 0) 0x7fdd981416c0 con 0x7fdd98145740 2024-02-19T15:22:59.929 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:22:59.929 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":33,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:22:58.589746+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:56.436273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:57.586982+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:22:59.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.928+0000 7fdd9fa00640 1 -- 172.21.15.47:0/4153401278 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdd78066e40 msgr2=0x7fdd78069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:59.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.928+0000 7fdd9fa00640 1 --2- 172.21.15.47:0/4153401278 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdd78066e40 0x7fdd78069300 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0011f0 tx=0x7fdd8c009040 comp rx=0 tx=0).stop 2024-02-19T15:22:59.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.928+0000 7fdd9fa00640 1 -- 172.21.15.47:0/4153401278 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd98145740 msgr2=0x7fdd9815a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:22:59.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.928+0000 7fdd9fa00640 1 --2- 172.21.15.47:0/4153401278 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd98145740 0x7fdd9815a7e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fdd900b0040 tx=0x7fdd900195e0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.928+0000 7fdd9fa00640 1 -- 172.21.15.47:0/4153401278 shutdown_connections 2024-02-19T15:22:59.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.928+0000 7fdd9fa00640 1 --2- 172.21.15.47:0/4153401278 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdd78066e40 0x7fdd78069300 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.928+0000 7fdd9fa00640 1 --2- 172.21.15.47:0/4153401278 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd98145740 0x7fdd9815a7e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.928+0000 7fdd9fa00640 1 --2- 172.21.15.47:0/4153401278 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9806b330 0x7fdd980cceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.928+0000 7fdd9fa00640 1 --2- 172.21.15.47:0/4153401278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd9806aeb0 0x7fdd980cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:22:59.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.928+0000 7fdd9fa00640 1 -- 172.21.15.47:0/4153401278 >> 172.21.15.47:0/4153401278 conn(0x7fdd980d0a50 msgr2=0x7fdd980c9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:22:59.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.928+0000 7fdd9fa00640 1 -- 172.21.15.47:0/4153401278 shutdown_connections 2024-02-19T15:22:59.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:22:59.928+0000 7fdd9fa00640 1 -- 172.21.15.47:0/4153401278 wait complete. 2024-02-19T15:22:59.958 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T15:23:00.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.096+0000 7f116c800640 1 Processor -- start 2024-02-19T15:23:00.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.096+0000 7f116c800640 1 -- start start 2024-02-19T15:23:00.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.100+0000 7f116c800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f116415bd90 0x7f1164070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:00.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.100+0000 7f116c800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11640cb490 0x7f1164072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:00.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.100+0000 7f116c800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f116415fc90 con 0x7f1164072680 2024-02-19T15:23:00.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.100+0000 7f116c800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f116415ff70 con 0x7f116415bd90 2024-02-19T15:23:00.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.100+0000 7f116c800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1164160250 con 0x7f11640cb490 2024-02-19T15:23:00.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.100+0000 7f1169e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f116415bd90 0x7f1164070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:00.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.100+0000 7f1169e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f116415bd90 0x7f1164070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39458/0 (socket says 172.21.15.47:39458) 2024-02-19T15:23:00.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.100+0000 7f1169e00640 1 -- 172.21.15.47:0/3468207927 learned_addr learned my addr 172.21.15.47:0/3468207927 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:00.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.100+0000 7f1169400640 1 --2- 172.21.15.47:0/3468207927 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11640cb490 0x7f1164072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:00.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.100+0000 7f1169e00640 1 -- 172.21.15.47:0/3468207927 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11640cb490 msgr2=0x7f1164072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:00.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.100+0000 7f1169e00640 1 --2- 172.21.15.47:0/3468207927 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11640cb490 0x7f1164072140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:00.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.100+0000 7f1169e00640 1 -- 172.21.15.47:0/3468207927 >> v1:172.21.15.116:6789/0 conn(0x7f1164072680 legacy=0x7f11640ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:00.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.100+0000 7f1169e00640 1 -- 172.21.15.47:0/3468207927 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1164160530 con 0x7f116415bd90 2024-02-19T15:23:00.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.104+0000 7f1169e00640 1 --2- 172.21.15.47:0/3468207927 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f116415bd90 0x7f1164070960 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1158009870 tx=0x7f115801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c46fb96ff7f5f1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:00.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.104+0000 7f1168a00640 1 -- 172.21.15.47:0/3468207927 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1158004230 con 0x7f116415bd90 2024-02-19T15:23:00.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.104+0000 7f1168a00640 1 -- 172.21.15.47:0/3468207927 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11580a7050 con 0x7f116415bd90 2024-02-19T15:23:00.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.104+0000 7f1168a00640 1 -- 172.21.15.47:0/3468207927 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1158003620 con 0x7f116415bd90 2024-02-19T15:23:00.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.104+0000 7f116c800640 1 -- 172.21.15.47:0/3468207927 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f116415bd90 msgr2=0x7f1164070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:00.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.104+0000 7f116c800640 1 --2- 172.21.15.47:0/3468207927 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f116415bd90 0x7f1164070960 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1158009870 tx=0x7f115801c400 comp rx=0 tx=0).stop 2024-02-19T15:23:00.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.104+0000 7f116c800640 1 -- 172.21.15.47:0/3468207927 shutdown_connections 2024-02-19T15:23:00.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.104+0000 7f116c800640 1 --2- 172.21.15.47:0/3468207927 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11640cb490 0x7f1164072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:00.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.104+0000 7f116c800640 1 --2- 172.21.15.47:0/3468207927 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f116415bd90 0x7f1164070960 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:00.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.104+0000 7f116c800640 1 -- 172.21.15.47:0/3468207927 >> 172.21.15.47:0/3468207927 conn(0x7f11640cfc20 msgr2=0x7f11640d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:00.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.104+0000 7f116c800640 1 -- 172.21.15.47:0/3468207927 shutdown_connections 2024-02-19T15:23:00.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.104+0000 7f116c800640 1 -- 172.21.15.47:0/3468207927 wait complete. 2024-02-19T15:23:00.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.108+0000 7f116c800640 1 Processor -- start 2024-02-19T15:23:00.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.108+0000 7f116c800640 1 -- start start 2024-02-19T15:23:00.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.108+0000 7f116c800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1164072680 0x7f11641650e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:00.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.108+0000 7f116c800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11640cb490 0x7f116414cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:00.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.108+0000 7f116c800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f116415bd90 0x7f116414d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:00.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.108+0000 7f116c800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11640cf480 con 0x7f11640cb490 2024-02-19T15:23:00.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.108+0000 7f116c800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11640aab40 con 0x7f1164072680 2024-02-19T15:23:00.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.108+0000 7f116c800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11640ceae0 con 0x7f116415bd90 2024-02-19T15:23:00.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.108+0000 7f1169400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11640cb490 0x7f116414cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:00.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.108+0000 7f1169400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11640cb490 0x7f116414cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39460/0 (socket says 172.21.15.47:39460) 2024-02-19T15:23:00.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.108+0000 7f1169400640 1 -- 172.21.15.47:0/1808552388 learned_addr learned my addr 172.21.15.47:0/1808552388 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:00.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.108+0000 7f1169e00640 1 --2- 172.21.15.47:0/1808552388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1164072680 0x7f11641650e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:00.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.108+0000 7f116a800640 1 --2- 172.21.15.47:0/1808552388 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f116415bd90 0x7f116414d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:00.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.108+0000 7f1169400640 1 -- 172.21.15.47:0/1808552388 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f116415bd90 msgr2=0x7f116414d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:00.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.108+0000 7f1169400640 1 --2- 172.21.15.47:0/1808552388 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f116415bd90 0x7f116414d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:00.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.108+0000 7f1169400640 1 -- 172.21.15.47:0/1808552388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1164072680 msgr2=0x7f11641650e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:00.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.108+0000 7f1169400640 1 --2- 172.21.15.47:0/1808552388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1164072680 0x7f11641650e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:00.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.108+0000 7f1169400640 1 -- 172.21.15.47:0/1808552388 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11580094d0 con 0x7f11640cb490 2024-02-19T15:23:00.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.108+0000 7f1169e00640 1 --2- 172.21.15.47:0/1808552388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1164072680 0x7f11641650e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:00.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.108+0000 7f1169400640 1 --2- 172.21.15.47:0/1808552388 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11640cb490 0x7f116414cad0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f115400d620 tx=0x7f115400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:00.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.112+0000 7f115e000640 1 -- 172.21.15.47:0/1808552388 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115400b620 con 0x7f11640cb490 2024-02-19T15:23:00.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.112+0000 7f115e000640 1 -- 172.21.15.47:0/1808552388 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f115400b780 con 0x7f11640cb490 2024-02-19T15:23:00.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.112+0000 7f116c800640 1 -- 172.21.15.47:0/1808552388 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11640cedc0 con 0x7f11640cb490 2024-02-19T15:23:00.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.112+0000 7f115e000640 1 -- 172.21.15.47:0/1808552388 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115400bb10 con 0x7f11640cb490 2024-02-19T15:23:00.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.112+0000 7f116c800640 1 -- 172.21.15.47:0/1808552388 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1164165620 con 0x7f11640cb490 2024-02-19T15:23:00.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.112+0000 7f116c800640 1 -- 172.21.15.47:0/1808552388 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1164075360 con 0x7f11640cb490 2024-02-19T15:23:00.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.112+0000 7f115e000640 1 -- 172.21.15.47:0/1808552388 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f115400bc70 con 0x7f11640cb490 2024-02-19T15:23:00.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.112+0000 7f115e000640 1 --2- 172.21.15.47:0/1808552388 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1148066e40 0x7f1148069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:00.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.112+0000 7f115e000640 1 -- 172.21.15.47:0/1808552388 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1154084f30 con 0x7f11640cb490 2024-02-19T15:23:00.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.112+0000 7f1169e00640 1 --2- 172.21.15.47:0/1808552388 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1148066e40 0x7f1148069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:00.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.116+0000 7f1169e00640 1 --2- 172.21.15.47:0/1808552388 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1148066e40 0x7f1148069300 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f11580ad000 tx=0x7f1158009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:00.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.116+0000 7f115e000640 1 -- 172.21.15.47:0/1808552388 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1154051d10 con 0x7f11640cb490 2024-02-19T15:23:00.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:00.320+0000 7f116c800640 1 -- 172.21.15.47:0/1808552388 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f116406e800 con 0x7f11640cb490 2024-02-19T15:23:01.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.216+0000 7f115e000640 1 -- 172.21.15.47:0/1808552388 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1154007380 con 0x7f11640cb490 2024-02-19T15:23:01.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.216+0000 7f116c800640 1 -- 172.21.15.47:0/1808552388 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1148066e40 msgr2=0x7f1148069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:01.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.216+0000 7f116c800640 1 --2- 172.21.15.47:0/1808552388 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1148066e40 0x7f1148069300 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f11580ad000 tx=0x7f1158009040 comp rx=0 tx=0).stop 2024-02-19T15:23:01.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.216+0000 7f116c800640 1 -- 172.21.15.47:0/1808552388 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11640cb490 msgr2=0x7f116414cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:01.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.216+0000 7f116c800640 1 --2- 172.21.15.47:0/1808552388 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11640cb490 0x7f116414cad0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f115400d620 tx=0x7f115400dae0 comp rx=0 tx=0).stop 2024-02-19T15:23:01.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.216+0000 7f116c800640 1 -- 172.21.15.47:0/1808552388 shutdown_connections 2024-02-19T15:23:01.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.216+0000 7f116c800640 1 --2- 172.21.15.47:0/1808552388 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1148066e40 0x7f1148069300 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:01.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.216+0000 7f116c800640 1 --2- 172.21.15.47:0/1808552388 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f116415bd90 0x7f116414d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:01.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.216+0000 7f116c800640 1 --2- 172.21.15.47:0/1808552388 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11640cb490 0x7f116414cad0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:01.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.216+0000 7f116c800640 1 --2- 172.21.15.47:0/1808552388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1164072680 0x7f11641650e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:01.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.216+0000 7f116c800640 1 -- 172.21.15.47:0/1808552388 >> 172.21.15.47:0/1808552388 conn(0x7f11640cfc20 msgr2=0x7f11640c6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:01.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.216+0000 7f116c800640 1 -- 172.21.15.47:0/1808552388 shutdown_connections 2024-02-19T15:23:01.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.216+0000 7f116c800640 1 -- 172.21.15.47:0/1808552388 wait complete. 2024-02-19T15:23:01.246 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T15:23:01.246 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T15:23:01.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.384+0000 7f855ae00640 1 Processor -- start 2024-02-19T15:23:01.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.384+0000 7f855ae00640 1 -- start start 2024-02-19T15:23:01.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.388+0000 7f855ae00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85540ce3b0 0x7f85540ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:01.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.388+0000 7f855ae00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f855415e280 0x7f85540cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:01.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.388+0000 7f855ae00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85540cc620 con 0x7f85540d2650 2024-02-19T15:23:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.388+0000 7f855ae00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85540d12e0 con 0x7f855415e280 2024-02-19T15:23:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.388+0000 7f855ae00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85540d1940 con 0x7f85540ce3b0 2024-02-19T15:23:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.388+0000 7f8558e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f855415e280 0x7f85540cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.388+0000 7f8558e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f855415e280 0x7f85540cd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39470/0 (socket says 172.21.15.47:39470) 2024-02-19T15:23:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.388+0000 7f8558e00640 1 -- 172.21.15.47:0/2709855450 learned_addr learned my addr 172.21.15.47:0/2709855450 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.388+0000 7f8553e00640 1 --2- 172.21.15.47:0/2709855450 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85540ce3b0 0x7f85540ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.388+0000 7f8558e00640 1 -- 172.21.15.47:0/2709855450 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85540ce3b0 msgr2=0x7f85540ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.388+0000 7f8558e00640 1 --2- 172.21.15.47:0/2709855450 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85540ce3b0 0x7f85540ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.388+0000 7f8558e00640 1 -- 172.21.15.47:0/2709855450 >> v1:172.21.15.116:6789/0 conn(0x7f85540d2650 legacy=0x7f85540ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.388+0000 7f8558e00640 1 -- 172.21.15.47:0/2709855450 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85540cecd0 con 0x7f855415e280 2024-02-19T15:23:01.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.388+0000 7f8558e00640 1 --2- 172.21.15.47:0/2709855450 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f855415e280 0x7f85540cd1c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f854c009e50 tx=0x7f854c01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b30b04e2667216 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:01.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.392+0000 7f8553400640 1 -- 172.21.15.47:0/2709855450 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c0a56c0 con 0x7f855415e280 2024-02-19T15:23:01.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.392+0000 7f8553400640 1 -- 172.21.15.47:0/2709855450 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854c0a5820 con 0x7f855415e280 2024-02-19T15:23:01.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.392+0000 7f8553400640 1 -- 172.21.15.47:0/2709855450 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c0a5bb0 con 0x7f855415e280 2024-02-19T15:23:01.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.392+0000 7f855ae00640 1 -- 172.21.15.47:0/2709855450 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f855415e280 msgr2=0x7f85540cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:01.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.392+0000 7f855ae00640 1 --2- 172.21.15.47:0/2709855450 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f855415e280 0x7f85540cd1c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f854c009e50 tx=0x7f854c01c910 comp rx=0 tx=0).stop 2024-02-19T15:23:01.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.392+0000 7f855ae00640 1 -- 172.21.15.47:0/2709855450 shutdown_connections 2024-02-19T15:23:01.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.392+0000 7f855ae00640 1 --2- 172.21.15.47:0/2709855450 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f855415e280 0x7f85540cd1c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:01.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.392+0000 7f855ae00640 1 --2- 172.21.15.47:0/2709855450 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85540ce3b0 0x7f85540ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:01.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.392+0000 7f855ae00640 1 -- 172.21.15.47:0/2709855450 >> 172.21.15.47:0/2709855450 conn(0x7f85540d1b30 msgr2=0x7f85540d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:01.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.392+0000 7f855ae00640 1 -- 172.21.15.47:0/2709855450 shutdown_connections 2024-02-19T15:23:01.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.392+0000 7f855ae00640 1 -- 172.21.15.47:0/2709855450 wait complete. 2024-02-19T15:23:01.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.392+0000 7f855ae00640 1 Processor -- start 2024-02-19T15:23:01.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.392+0000 7f855ae00640 1 -- start start 2024-02-19T15:23:01.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f855ae00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85540ce3b0 0x7f855414c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:01.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f855ae00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85540d2650 0x7f855415cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:01.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f855ae00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f855415e280 0x7f855415d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:01.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f855ae00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855415a060 con 0x7f85540d2650 2024-02-19T15:23:01.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f855ae00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855415a2b0 con 0x7f855415e280 2024-02-19T15:23:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f855ae00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f855415a590 con 0x7f85540ce3b0 2024-02-19T15:23:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f854b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85540d2650 0x7f855415cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f854b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85540d2650 0x7f855415cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39474/0 (socket says 172.21.15.47:39474) 2024-02-19T15:23:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f854b400640 1 -- 172.21.15.47:0/2908496409 learned_addr learned my addr 172.21.15.47:0/2908496409 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f8553e00640 1 --2- 172.21.15.47:0/2908496409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85540ce3b0 0x7f855414c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f8558e00640 1 --2- 172.21.15.47:0/2908496409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f855415e280 0x7f855415d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f854b400640 1 -- 172.21.15.47:0/2908496409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85540ce3b0 msgr2=0x7f855414c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f854b400640 1 --2- 172.21.15.47:0/2908496409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85540ce3b0 0x7f855414c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f854b400640 1 -- 172.21.15.47:0/2908496409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f855415e280 msgr2=0x7f855415d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f854b400640 1 --2- 172.21.15.47:0/2908496409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f855415e280 0x7f855415d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f854b400640 1 -- 172.21.15.47:0/2908496409 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c009b30 con 0x7f85540d2650 2024-02-19T15:23:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f854b400640 1 --2- 172.21.15.47:0/2908496409 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85540d2650 0x7f855415cd50 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f853c00c370 tx=0x7f853c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f8550c00640 1 -- 172.21.15.47:0/2908496409 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c0052d0 con 0x7f85540d2650 2024-02-19T15:23:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f8550c00640 1 -- 172.21.15.47:0/2908496409 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853c005430 con 0x7f85540d2650 2024-02-19T15:23:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f855ae00640 1 -- 172.21.15.47:0/2908496409 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f855415a870 con 0x7f85540d2650 2024-02-19T15:23:01.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f8550c00640 1 -- 172.21.15.47:0/2908496409 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c007260 con 0x7f85540d2650 2024-02-19T15:23:01.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f855ae00640 1 -- 172.21.15.47:0/2908496409 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f855415ae30 con 0x7f85540d2650 2024-02-19T15:23:01.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.396+0000 7f855ae00640 1 -- 172.21.15.47:0/2908496409 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8554075360 con 0x7f85540d2650 2024-02-19T15:23:01.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.400+0000 7f8550c00640 1 -- 172.21.15.47:0/2908496409 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f853c0073c0 con 0x7f85540d2650 2024-02-19T15:23:01.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.400+0000 7f8550c00640 1 --2- 172.21.15.47:0/2908496409 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8530066e40 0x7f8530069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:01.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.400+0000 7f8550c00640 1 -- 172.21.15.47:0/2908496409 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f853c083dc0 con 0x7f85540d2650 2024-02-19T15:23:01.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.400+0000 7f8553e00640 1 --2- 172.21.15.47:0/2908496409 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8530066e40 0x7f8530069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:01.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.400+0000 7f8553e00640 1 --2- 172.21.15.47:0/2908496409 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8530066e40 0x7f8530069300 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f85540700c0 tx=0x7f8544008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:01.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.404+0000 7f8550c00640 1 -- 172.21.15.47:0/2908496409 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f853c050bc0 con 0x7f85540d2650 2024-02-19T15:23:01.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:01.652+0000 7f855ae00640 1 -- 172.21.15.47:0/2908496409 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f85540cdf80 con 0x7f85540d2650 2024-02-19T15:23:02.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.216+0000 7f8550c00640 1 -- 172.21.15.47:0/2908496409 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f853c054280 con 0x7f85540d2650 2024-02-19T15:23:02.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.220+0000 7f855ae00640 1 -- 172.21.15.47:0/2908496409 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8530066e40 msgr2=0x7f8530069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:02.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.220+0000 7f855ae00640 1 --2- 172.21.15.47:0/2908496409 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8530066e40 0x7f8530069300 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f85540700c0 tx=0x7f8544008040 comp rx=0 tx=0).stop 2024-02-19T15:23:02.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.220+0000 7f855ae00640 1 -- 172.21.15.47:0/2908496409 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85540d2650 msgr2=0x7f855415cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:02.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.220+0000 7f855ae00640 1 --2- 172.21.15.47:0/2908496409 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85540d2650 0x7f855415cd50 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f853c00c370 tx=0x7f853c00c830 comp rx=0 tx=0).stop 2024-02-19T15:23:02.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.220+0000 7f855ae00640 1 -- 172.21.15.47:0/2908496409 shutdown_connections 2024-02-19T15:23:02.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.220+0000 7f855ae00640 1 --2- 172.21.15.47:0/2908496409 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8530066e40 0x7f8530069300 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:02.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.220+0000 7f855ae00640 1 --2- 172.21.15.47:0/2908496409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f855415e280 0x7f855415d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:02.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.220+0000 7f855ae00640 1 --2- 172.21.15.47:0/2908496409 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85540d2650 0x7f855415cd50 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:02.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.220+0000 7f855ae00640 1 --2- 172.21.15.47:0/2908496409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85540ce3b0 0x7f855414c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:02.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.220+0000 7f855ae00640 1 -- 172.21.15.47:0/2908496409 >> 172.21.15.47:0/2908496409 conn(0x7f85540d1b30 msgr2=0x7f85540c5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:02.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.220+0000 7f855ae00640 1 -- 172.21.15.47:0/2908496409 shutdown_connections 2024-02-19T15:23:02.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.220+0000 7f855ae00640 1 -- 172.21.15.47:0/2908496409 wait complete. 2024-02-19T15:23:02.243 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T15:23:02.243 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:23:02.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.400+0000 7f2845e00640 1 Processor -- start 2024-02-19T15:23:02.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.400+0000 7f2845e00640 1 -- start start 2024-02-19T15:23:02.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.400+0000 7f2845e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f284015d6e0 0x7f2840162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:02.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.400+0000 7f2845e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2840071350 0x7f2840164c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:02.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.400+0000 7f2845e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28400718c0 con 0x7f284015b1a0 2024-02-19T15:23:02.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.400+0000 7f2845e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2840071ba0 con 0x7f2840071350 2024-02-19T15:23:02.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.400+0000 7f2845e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2840071e80 con 0x7f284015d6e0 2024-02-19T15:23:02.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.400+0000 7f283f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2840071350 0x7f2840164c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:02.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.400+0000 7f283f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2840071350 0x7f2840164c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39482/0 (socket says 172.21.15.47:39482) 2024-02-19T15:23:02.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.400+0000 7f283f400640 1 -- 172.21.15.47:0/3588277001 learned_addr learned my addr 172.21.15.47:0/3588277001 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:02.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.400+0000 7f283ea00640 1 --1- 172.21.15.47:0/3588277001 >> v1:172.21.15.47:6789/0 conn(0x7f284015b1a0 0x7f284015b580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53834/0 (socket says 172.21.15.47:53834) 2024-02-19T15:23:02.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.404+0000 7f2836000640 1 --2- 172.21.15.47:0/3588277001 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f284015d6e0 0x7f2840162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:02.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.404+0000 7f283f400640 1 -- 172.21.15.47:0/3588277001 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f284015d6e0 msgr2=0x7f2840162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:02.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.404+0000 7f283f400640 1 --2- 172.21.15.47:0/3588277001 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f284015d6e0 0x7f2840162740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:02.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.404+0000 7f283f400640 1 -- 172.21.15.47:0/3588277001 >> v1:172.21.15.47:6789/0 conn(0x7f284015b1a0 legacy=0x7f284015b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:02.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.404+0000 7f283f400640 1 -- 172.21.15.47:0/3588277001 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2840072160 con 0x7f2840071350 2024-02-19T15:23:02.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.404+0000 7f283f400640 1 --2- 172.21.15.47:0/3588277001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2840071350 0x7f2840164c90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f283800d150 tx=0x7f283801ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=356a2183bfd4f63f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:02.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.404+0000 7f283e000640 1 -- 172.21.15.47:0/3588277001 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283800b2d0 con 0x7f2840071350 2024-02-19T15:23:02.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.404+0000 7f283e000640 1 -- 172.21.15.47:0/3588277001 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283800b430 con 0x7f2840071350 2024-02-19T15:23:02.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.404+0000 7f283e000640 1 -- 172.21.15.47:0/3588277001 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28380a7c40 con 0x7f2840071350 2024-02-19T15:23:02.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.404+0000 7f2845e00640 1 -- 172.21.15.47:0/3588277001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2840071350 msgr2=0x7f2840164c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:02.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.404+0000 7f2845e00640 1 --2- 172.21.15.47:0/3588277001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2840071350 0x7f2840164c90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f283800d150 tx=0x7f283801ec30 comp rx=0 tx=0).stop 2024-02-19T15:23:02.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2845e00640 1 -- 172.21.15.47:0/3588277001 shutdown_connections 2024-02-19T15:23:02.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2845e00640 1 --2- 172.21.15.47:0/3588277001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2840071350 0x7f2840164c90 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:02.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2845e00640 1 --2- 172.21.15.47:0/3588277001 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f284015d6e0 0x7f2840162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:02.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2845e00640 1 -- 172.21.15.47:0/3588277001 >> 172.21.15.47:0/3588277001 conn(0x7f28400b16a0 msgr2=0x7f28400b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:02.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2845e00640 1 -- 172.21.15.47:0/3588277001 shutdown_connections 2024-02-19T15:23:02.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2845e00640 1 -- 172.21.15.47:0/3588277001 wait complete. 2024-02-19T15:23:02.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2845e00640 1 Processor -- start 2024-02-19T15:23:02.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2845e00640 1 -- start start 2024-02-19T15:23:02.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2845e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2840071350 0x7f2840147340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:02.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2845e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f284015b1a0 0x7f284014c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:02.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2845e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f284015d6e0 0x7f28400ce500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:02.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2845e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284014ee90 con 0x7f284015b1a0 2024-02-19T15:23:02.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2845e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284014f6f0 con 0x7f284015d6e0 2024-02-19T15:23:02.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2845e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f284014cb70 con 0x7f2840071350 2024-02-19T15:23:02.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2836000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f284015b1a0 0x7f284014c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:02.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2836000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f284015b1a0 0x7f284014c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39488/0 (socket says 172.21.15.47:39488) 2024-02-19T15:23:02.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2836000640 1 -- 172.21.15.47:0/2574439045 learned_addr learned my addr 172.21.15.47:0/2574439045 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:02.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f283f400640 1 --2- 172.21.15.47:0/2574439045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f284015d6e0 0x7f28400ce500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:02.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f283ea00640 1 --2- 172.21.15.47:0/2574439045 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2840071350 0x7f2840147340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:02.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2836000640 1 -- 172.21.15.47:0/2574439045 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2840071350 msgr2=0x7f2840147340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:02.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2836000640 1 --2- 172.21.15.47:0/2574439045 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2840071350 0x7f2840147340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:02.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2836000640 1 -- 172.21.15.47:0/2574439045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f284015d6e0 msgr2=0x7f28400ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:02.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2836000640 1 --2- 172.21.15.47:0/2574439045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f284015d6e0 0x7f28400ce500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:02.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2836000640 1 -- 172.21.15.47:0/2574439045 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f283800bd40 con 0x7f284015b1a0 2024-02-19T15:23:02.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.408+0000 7f2836000640 1 --2- 172.21.15.47:0/2574439045 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f284015b1a0 0x7f284014c320 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f282800e930 tx=0x7f282800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:02.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.412+0000 7f2837e00640 1 -- 172.21.15.47:0/2574439045 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2828009690 con 0x7f284015b1a0 2024-02-19T15:23:02.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.412+0000 7f2837e00640 1 -- 172.21.15.47:0/2574439045 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f282800ca00 con 0x7f284015b1a0 2024-02-19T15:23:02.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.412+0000 7f2837e00640 1 -- 172.21.15.47:0/2574439045 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282800cd70 con 0x7f284015b1a0 2024-02-19T15:23:02.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.412+0000 7f2845e00640 1 -- 172.21.15.47:0/2574439045 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f284014f0a0 con 0x7f284015b1a0 2024-02-19T15:23:02.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.412+0000 7f2845e00640 1 -- 172.21.15.47:0/2574439045 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f284014c860 con 0x7f284015b1a0 2024-02-19T15:23:02.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.412+0000 7f2845e00640 1 -- 172.21.15.47:0/2574439045 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f284014f880 con 0x7f284015b1a0 2024-02-19T15:23:02.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.412+0000 7f2837e00640 1 -- 172.21.15.47:0/2574439045 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2828013470 con 0x7f284015b1a0 2024-02-19T15:23:02.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.412+0000 7f2837e00640 1 --2- 172.21.15.47:0/2574439045 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f281c066e40 0x7f281c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:02.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.412+0000 7f2837e00640 1 -- 172.21.15.47:0/2574439045 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2828084d00 con 0x7f284015b1a0 2024-02-19T15:23:02.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.412+0000 7f283ea00640 1 --2- 172.21.15.47:0/2574439045 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f281c066e40 0x7f281c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:02.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.416+0000 7f283ea00640 1 --2- 172.21.15.47:0/2574439045 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f281c066e40 0x7f281c069300 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2830000f70 tx=0x7f2830007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:02.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.416+0000 7f2837e00640 1 -- 172.21.15.47:0/2574439045 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2828051bc0 con 0x7f284015b1a0 2024-02-19T15:23:02.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.612+0000 7f2845e00640 1 -- 172.21.15.47:0/2574439045 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2840075360 con 0x7f284015b1a0 2024-02-19T15:23:02.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.612+0000 7f2837e00640 1 -- 172.21.15.47:0/2574439045 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f2828017090 con 0x7f284015b1a0 2024-02-19T15:23:02.693 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:02.693 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1616905430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1616905430},{"type":"v1","addr":"172.21.15.116:6836","nonce":1616905430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4814,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2845273225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2845273225},{"type":"v1","addr":"172.21.15.116:6837","nonce":2845273225}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5064,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/20043732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":20043732},{"type":"v1","addr":"172.21.15.47:6834","nonce":20043732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:22:58.590074+0000","modified":"2024-02-19T15:23:01.656160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3700339993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3700339993},{"type":"v1","addr":"172.21.15.47:6835","nonce":3700339993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T15:23:02.697 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 16 2024-02-19T15:23:02.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.616+0000 7f2845e00640 1 -- 172.21.15.47:0/2574439045 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f281c066e40 msgr2=0x7f281c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:02.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.616+0000 7f2845e00640 1 --2- 172.21.15.47:0/2574439045 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f281c066e40 0x7f281c069300 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2830000f70 tx=0x7f2830007e90 comp rx=0 tx=0).stop 2024-02-19T15:23:02.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.616+0000 7f2845e00640 1 -- 172.21.15.47:0/2574439045 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f284015b1a0 msgr2=0x7f284014c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:02.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.616+0000 7f2845e00640 1 --2- 172.21.15.47:0/2574439045 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f284015b1a0 0x7f284014c320 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f282800e930 tx=0x7f282800edf0 comp rx=0 tx=0).stop 2024-02-19T15:23:02.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.616+0000 7f2845e00640 1 -- 172.21.15.47:0/2574439045 shutdown_connections 2024-02-19T15:23:02.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.616+0000 7f2845e00640 1 --2- 172.21.15.47:0/2574439045 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f281c066e40 0x7f281c069300 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:02.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.616+0000 7f2845e00640 1 --2- 172.21.15.47:0/2574439045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f284015d6e0 0x7f28400ce500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:02.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.616+0000 7f2845e00640 1 --2- 172.21.15.47:0/2574439045 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f284015b1a0 0x7f284014c320 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:02.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.616+0000 7f2845e00640 1 --2- 172.21.15.47:0/2574439045 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2840071350 0x7f2840147340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:02.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.616+0000 7f2845e00640 1 -- 172.21.15.47:0/2574439045 >> 172.21.15.47:0/2574439045 conn(0x7f28400b16a0 msgr2=0x7f284014e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:02.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.616+0000 7f2845e00640 1 -- 172.21.15.47:0/2574439045 shutdown_connections 2024-02-19T15:23:02.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.616+0000 7f2845e00640 1 -- 172.21.15.47:0/2574439045 wait complete. 2024-02-19T15:23:02.698 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:23:02.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.832+0000 7fdbd9a00640 1 Processor -- start 2024-02-19T15:23:02.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.832+0000 7fdbd9a00640 1 -- start start 2024-02-19T15:23:02.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.836+0000 7fdbd9a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdbd406fdf0 0x7fdbd40adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:02.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.836+0000 7fdbd9a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd4161780 con 0x7fdbd406e7c0 2024-02-19T15:23:02.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.836+0000 7fdbd9a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd40b01a0 con 0x7fdbd41613a0 2024-02-19T15:23:02.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.836+0000 7fdbd9a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbd40ae3c0 con 0x7fdbd406fdf0 2024-02-19T15:23:02.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.836+0000 7fdbd2000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdbd406fdf0 0x7fdbd40adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:02.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.836+0000 7fdbd3400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fdbd406e7c0 0x7fdbd406aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53848/0 (socket says 172.21.15.47:53848) 2024-02-19T15:23:02.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.836+0000 7fdbd3400640 1 -- 172.21.15.47:0/111729620 learned_addr learned my addr 172.21.15.47:0/111729620 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:02.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.836+0000 7fdbd2000640 1 -- 172.21.15.47:0/111729620 >> v1:172.21.15.116:6789/0 conn(0x7fdbd41613a0 legacy=0x7fdbd406e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:02.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.836+0000 7fdbd2000640 1 -- 172.21.15.47:0/111729620 >> v1:172.21.15.47:6789/0 conn(0x7fdbd406e7c0 legacy=0x7fdbd406aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:02.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.836+0000 7fdbd1600640 1 -- 172.21.15.47:0/111729620 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547259237 0 0) 0x7fdbd4161780 con 0x7fdbd406e7c0 2024-02-19T15:23:02.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.836+0000 7fdbd2000640 1 -- 172.21.15.47:0/111729620 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd40ae6a0 con 0x7fdbd406fdf0 2024-02-19T15:23:02.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.836+0000 7fdbd2000640 1 --2- 172.21.15.47:0/111729620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd406fdf0 0x7fdbd40adbb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdbc00097c0 tx=0x7fdbc001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7313e94a74625c4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:02.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd1600640 1 -- 172.21.15.47:0/111729620 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc00ab600 con 0x7fdbd406fdf0 2024-02-19T15:23:02.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd1600640 1 -- 172.21.15.47:0/111729620 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc00a7050 con 0x7fdbd406fdf0 2024-02-19T15:23:02.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd1600640 1 -- 172.21.15.47:0/111729620 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc00ab980 con 0x7fdbd406fdf0 2024-02-19T15:23:02.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd9a00640 1 -- 172.21.15.47:0/111729620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd406fdf0 msgr2=0x7fdbd40adbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:02.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd9a00640 1 --2- 172.21.15.47:0/111729620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd406fdf0 0x7fdbd40adbb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdbc00097c0 tx=0x7fdbc001c410 comp rx=0 tx=0).stop 2024-02-19T15:23:02.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd9a00640 1 -- 172.21.15.47:0/111729620 shutdown_connections 2024-02-19T15:23:02.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd9a00640 1 --2- 172.21.15.47:0/111729620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd406fdf0 0x7fdbd40adbb0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:02.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd9a00640 1 -- 172.21.15.47:0/111729620 >> 172.21.15.47:0/111729620 conn(0x7fdbd40d0ad0 msgr2=0x7fdbd40d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:02.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd9a00640 1 -- 172.21.15.47:0/111729620 shutdown_connections 2024-02-19T15:23:02.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd9a00640 1 -- 172.21.15.47:0/111729620 wait complete. 2024-02-19T15:23:02.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd9a00640 1 Processor -- start 2024-02-19T15:23:02.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd9a00640 1 -- start start 2024-02-19T15:23:02.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd9a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd406e7c0 0x7fdbd4150890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:02.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd9a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd406fdf0 0x7fdbd4159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:02.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd9a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd41613a0 0x7fdbd4159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:02.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd9a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd4152030 con 0x7fdbd41613a0 2024-02-19T15:23:02.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd9a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd415c350 con 0x7fdbd406fdf0 2024-02-19T15:23:02.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd9a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd415c630 con 0x7fdbd406e7c0 2024-02-19T15:23:02.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd41613a0 0x7fdbd4159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:02.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd41613a0 0x7fdbd4159fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39504/0 (socket says 172.21.15.47:39504) 2024-02-19T15:23:02.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.840+0000 7fdbd3400640 1 -- 172.21.15.47:0/1494561518 learned_addr learned my addr 172.21.15.47:0/1494561518 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:02.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.844+0000 7fdbd2000640 1 --2- 172.21.15.47:0/1494561518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd406fdf0 0x7fdbd4159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:02.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.844+0000 7fdbd2a00640 1 --2- 172.21.15.47:0/1494561518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd406e7c0 0x7fdbd4150890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:02.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.844+0000 7fdbd3400640 1 -- 172.21.15.47:0/1494561518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd406e7c0 msgr2=0x7fdbd4150890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:02.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.844+0000 7fdbd3400640 1 --2- 172.21.15.47:0/1494561518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd406e7c0 0x7fdbd4150890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:02.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.844+0000 7fdbd3400640 1 -- 172.21.15.47:0/1494561518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd406fdf0 msgr2=0x7fdbd4159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:02.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.844+0000 7fdbd3400640 1 --2- 172.21.15.47:0/1494561518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd406fdf0 0x7fdbd4159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:02.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.844+0000 7fdbd3400640 1 -- 172.21.15.47:0/1494561518 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc0009420 con 0x7fdbd41613a0 2024-02-19T15:23:02.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.844+0000 7fdbd2a00640 1 --2- 172.21.15.47:0/1494561518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd406e7c0 0x7fdbd4150890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:02.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.844+0000 7fdbd2000640 1 --2- 172.21.15.47:0/1494561518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd406fdf0 0x7fdbd4159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:02.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.844+0000 7fdbd3400640 1 --2- 172.21.15.47:0/1494561518 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd41613a0 0x7fdbd4159fa0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc00c7a0 tx=0x7fdbcc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:02.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.844+0000 7fdbcaa00640 1 -- 172.21.15.47:0/1494561518 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc00a640 con 0x7fdbd41613a0 2024-02-19T15:23:02.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.844+0000 7fdbcaa00640 1 -- 172.21.15.47:0/1494561518 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbcc00a7a0 con 0x7fdbd41613a0 2024-02-19T15:23:02.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.844+0000 7fdbd9a00640 1 -- 172.21.15.47:0/1494561518 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd415c910 con 0x7fdbd41613a0 2024-02-19T15:23:02.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.844+0000 7fdbcaa00640 1 -- 172.21.15.47:0/1494561518 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc00abd0 con 0x7fdbd41613a0 2024-02-19T15:23:02.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.844+0000 7fdbd9a00640 1 -- 172.21.15.47:0/1494561518 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd4150dd0 con 0x7fdbd41613a0 2024-02-19T15:23:02.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.844+0000 7fdbd9a00640 1 -- 172.21.15.47:0/1494561518 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbd4075360 con 0x7fdbd41613a0 2024-02-19T15:23:02.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.844+0000 7fdbcaa00640 1 -- 172.21.15.47:0/1494561518 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fdbcc00ad30 con 0x7fdbd41613a0 2024-02-19T15:23:02.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.848+0000 7fdbcaa00640 1 --2- 172.21.15.47:0/1494561518 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbb4066e40 0x7fdbb4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:02.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.848+0000 7fdbcaa00640 1 -- 172.21.15.47:0/1494561518 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdbcc084360 con 0x7fdbd41613a0 2024-02-19T15:23:02.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.848+0000 7fdbd2a00640 1 --2- 172.21.15.47:0/1494561518 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbb4066e40 0x7fdbb4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:02.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.848+0000 7fdbd2a00640 1 --2- 172.21.15.47:0/1494561518 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbb4066e40 0x7fdbb4069300 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4000f70 tx=0x7fdbc4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:02.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:02.852+0000 7fdbcaa00640 1 -- 172.21.15.47:0/1494561518 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fdbcc0510b0 con 0x7fdbd41613a0 2024-02-19T15:23:03.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.032+0000 7fdbd9a00640 1 -- 172.21.15.47:0/1494561518 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbd415f870 con 0x7fdbd41613a0 2024-02-19T15:23:03.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.032+0000 7fdbcaa00640 1 -- 172.21.15.47:0/1494561518 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16681 (secure 0 0 0) 0x7fdbcc054770 con 0x7fdbd41613a0 2024-02-19T15:23:03.037 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:03.037 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":34,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:23:00.195724+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:56.436273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:57.586982+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:23:03.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.036+0000 7fdbd9a00640 1 -- 172.21.15.47:0/1494561518 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbb4066e40 msgr2=0x7fdbb4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:03.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.036+0000 7fdbd9a00640 1 --2- 172.21.15.47:0/1494561518 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbb4066e40 0x7fdbb4069300 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4000f70 tx=0x7fdbc4009040 comp rx=0 tx=0).stop 2024-02-19T15:23:03.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.036+0000 7fdbd9a00640 1 -- 172.21.15.47:0/1494561518 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd41613a0 msgr2=0x7fdbd4159fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:03.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.036+0000 7fdbd9a00640 1 --2- 172.21.15.47:0/1494561518 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd41613a0 0x7fdbd4159fa0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc00c7a0 tx=0x7fdbcc00cc60 comp rx=0 tx=0).stop 2024-02-19T15:23:03.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.036+0000 7fdbd9a00640 1 -- 172.21.15.47:0/1494561518 shutdown_connections 2024-02-19T15:23:03.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.036+0000 7fdbd9a00640 1 --2- 172.21.15.47:0/1494561518 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbb4066e40 0x7fdbb4069300 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:03.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.036+0000 7fdbd9a00640 1 --2- 172.21.15.47:0/1494561518 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd41613a0 0x7fdbd4159fa0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:03.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.036+0000 7fdbd9a00640 1 --2- 172.21.15.47:0/1494561518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd406fdf0 0x7fdbd4159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:03.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.036+0000 7fdbd9a00640 1 --2- 172.21.15.47:0/1494561518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd406e7c0 0x7fdbd4150890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:03.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.036+0000 7fdbd9a00640 1 -- 172.21.15.47:0/1494561518 >> 172.21.15.47:0/1494561518 conn(0x7fdbd40d0ad0 msgr2=0x7fdbd40b16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:03.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.036+0000 7fdbd9a00640 1 -- 172.21.15.47:0/1494561518 shutdown_connections 2024-02-19T15:23:03.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.036+0000 7fdbd9a00640 1 -- 172.21.15.47:0/1494561518 wait complete. 2024-02-19T15:23:03.066 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T15:23:03.067 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:23:03.175 INFO:tasks.ceph.mds.c.smithi047.stderr:2024-02-19T15:23:03.172+0000 7f7979c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T15:23:03.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.204+0000 7fa63c800640 1 Processor -- start 2024-02-19T15:23:03.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.204+0000 7fa63c800640 1 -- start start 2024-02-19T15:23:03.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.204+0000 7fa63c800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa63415caa0 0x7fa6340727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:03.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.204+0000 7fa63c800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa63415d280 con 0x7fa634145590 2024-02-19T15:23:03.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.204+0000 7fa63c800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa634070ff0 con 0x7fa6340cea80 2024-02-19T15:23:03.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.204+0000 7fa63c800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6340709e0 con 0x7fa63415caa0 2024-02-19T15:23:03.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.204+0000 7fa639e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa634145590 0x7fa63406c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53858/0 (socket says 172.21.15.47:53858) 2024-02-19T15:23:03.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.204+0000 7fa639e00640 1 -- 172.21.15.47:0/1536487083 learned_addr learned my addr 172.21.15.47:0/1536487083 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:03.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.204+0000 7fa63a800640 1 --2- 172.21.15.47:0/1536487083 >> v2:172.21.15.116:3300/0 conn(0x7fa63415caa0 0x7fa6340727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:03.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.204+0000 7fa638a00640 1 -- 172.21.15.47:0/1536487083 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244252099 0 0) 0x7fa63415d280 con 0x7fa634145590 2024-02-19T15:23:03.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.204+0000 7fa638a00640 1 -- 172.21.15.47:0/1536487083 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa61c003530 con 0x7fa634145590 2024-02-19T15:23:03.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.204+0000 7fa638a00640 1 -- 172.21.15.47:0/1536487083 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213704603 0 0) 0x7fa634070ff0 con 0x7fa6340cea80 2024-02-19T15:23:03.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.204+0000 7fa638a00640 1 -- 172.21.15.47:0/1536487083 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa63415d280 con 0x7fa6340cea80 2024-02-19T15:23:03.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.204+0000 7fa63a800640 1 -- 172.21.15.47:0/1536487083 >> v1:172.21.15.116:6789/0 conn(0x7fa6340cea80 legacy=0x7fa63415c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:03.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.204+0000 7fa63a800640 1 -- 172.21.15.47:0/1536487083 >> v1:172.21.15.47:6789/0 conn(0x7fa634145590 legacy=0x7fa63406c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:03.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.204+0000 7fa638a00640 1 -- 172.21.15.47:0/1536487083 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2737997531 0 0) 0x7fa61c003530 con 0x7fa634145590 2024-02-19T15:23:03.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.204+0000 7fa63a800640 1 -- 172.21.15.47:0/1536487083 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6340cf020 con 0x7fa63415caa0 2024-02-19T15:23:03.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.208+0000 7fa63a800640 1 --2- 172.21.15.47:0/1536487083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa63415caa0 0x7fa6340727b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa630009d90 tx=0x7fa63001c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e54d0eef36ee5b6d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:03.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.208+0000 7fa638a00640 1 -- 172.21.15.47:0/1536487083 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628001680 con 0x7fa63415caa0 2024-02-19T15:23:03.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.208+0000 7fa638a00640 1 -- 172.21.15.47:0/1536487083 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6300a55e0 con 0x7fa63415caa0 2024-02-19T15:23:03.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.208+0000 7fa638a00640 1 -- 172.21.15.47:0/1536487083 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6300a5970 con 0x7fa63415caa0 2024-02-19T15:23:03.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.208+0000 7fa63c800640 1 -- 172.21.15.47:0/1536487083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa63415caa0 msgr2=0x7fa6340727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:03.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.208+0000 7fa63c800640 1 --2- 172.21.15.47:0/1536487083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa63415caa0 0x7fa6340727b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa630009d90 tx=0x7fa63001c890 comp rx=0 tx=0).stop 2024-02-19T15:23:03.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.208+0000 7fa63c800640 1 -- 172.21.15.47:0/1536487083 shutdown_connections 2024-02-19T15:23:03.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.208+0000 7fa63c800640 1 --2- 172.21.15.47:0/1536487083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa63415caa0 0x7fa6340727b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:03.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.208+0000 7fa63c800640 1 -- 172.21.15.47:0/1536487083 >> 172.21.15.47:0/1536487083 conn(0x7fa6340b16a0 msgr2=0x7fa6340b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:03.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.208+0000 7fa63c800640 1 -- 172.21.15.47:0/1536487083 shutdown_connections 2024-02-19T15:23:03.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.208+0000 7fa63c800640 1 -- 172.21.15.47:0/1536487083 wait complete. 2024-02-19T15:23:03.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.208+0000 7fa63c800640 1 Processor -- start 2024-02-19T15:23:03.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa63c800640 1 -- start start 2024-02-19T15:23:03.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa63c800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6340cea80 0x7fa63413b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:03.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa63c800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa634145590 0x7fa63415a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:03.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa63c800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa63415caa0 0x7fa634142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:03.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa63c800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa634141d20 con 0x7fa6340cea80 2024-02-19T15:23:03.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa63c800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63413bcc0 con 0x7fa634145590 2024-02-19T15:23:03.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa63c800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa634158460 con 0x7fa63415caa0 2024-02-19T15:23:03.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa639e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6340cea80 0x7fa63413b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:03.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa639e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6340cea80 0x7fa63413b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39514/0 (socket says 172.21.15.47:39514) 2024-02-19T15:23:03.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa639e00640 1 -- 172.21.15.47:0/2580674020 learned_addr learned my addr 172.21.15.47:0/2580674020 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:03.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa63a800640 1 --2- 172.21.15.47:0/2580674020 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa63415caa0 0x7fa634142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:03.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa639400640 1 --2- 172.21.15.47:0/2580674020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa634145590 0x7fa63415a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:03.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa639e00640 1 -- 172.21.15.47:0/2580674020 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa63415caa0 msgr2=0x7fa634142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:03.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa639e00640 1 --2- 172.21.15.47:0/2580674020 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa63415caa0 0x7fa634142210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:03.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa639e00640 1 -- 172.21.15.47:0/2580674020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa634145590 msgr2=0x7fa63415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:03.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa639e00640 1 --2- 172.21.15.47:0/2580674020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa634145590 0x7fa63415a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:03.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa639e00640 1 -- 172.21.15.47:0/2580674020 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa630009a70 con 0x7fa6340cea80 2024-02-19T15:23:03.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa63a800640 1 --2- 172.21.15.47:0/2580674020 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa63415caa0 0x7fa634142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:03.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa639e00640 1 --2- 172.21.15.47:0/2580674020 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6340cea80 0x7fa63413b780 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa62800c750 tx=0x7fa62800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:03.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa62e000640 1 -- 172.21.15.47:0/2580674020 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628005af0 con 0x7fa6340cea80 2024-02-19T15:23:03.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa62e000640 1 -- 172.21.15.47:0/2580674020 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa628005c50 con 0x7fa6340cea80 2024-02-19T15:23:03.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa62e000640 1 -- 172.21.15.47:0/2580674020 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628007070 con 0x7fa6340cea80 2024-02-19T15:23:03.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa63c800640 1 -- 172.21.15.47:0/2580674020 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa634158740 con 0x7fa6340cea80 2024-02-19T15:23:03.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa63c800640 1 -- 172.21.15.47:0/2580674020 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa634158d00 con 0x7fa6340cea80 2024-02-19T15:23:03.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.212+0000 7fa63c800640 1 -- 172.21.15.47:0/2580674020 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa634075360 con 0x7fa6340cea80 2024-02-19T15:23:03.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.216+0000 7fa62e000640 1 -- 172.21.15.47:0/2580674020 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa6280071d0 con 0x7fa6340cea80 2024-02-19T15:23:03.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.216+0000 7fa62e000640 1 --2- 172.21.15.47:0/2580674020 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa618066e40 0x7fa618069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:03.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.216+0000 7fa62e000640 1 -- 172.21.15.47:0/2580674020 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa628084160 con 0x7fa6340cea80 2024-02-19T15:23:03.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.216+0000 7fa639400640 1 --2- 172.21.15.47:0/2580674020 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa618066e40 0x7fa618069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:03.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.216+0000 7fa639400640 1 --2- 172.21.15.47:0/2580674020 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa618066e40 0x7fa618069300 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa634072250 tx=0x7fa624009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:03.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.220+0000 7fa62e000640 1 -- 172.21.15.47:0/2580674020 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa628050f60 con 0x7fa6340cea80 2024-02-19T15:23:03.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.380+0000 7fa63c800640 1 -- 172.21.15.47:0/2580674020 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa63415a8f0 con 0x7fa618066e40 2024-02-19T15:23:03.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.392+0000 7fa62e000640 1 -- 172.21.15.47:0/2580674020 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289331 (secure 0 0 0) 0x7fa63415a8f0 con 0x7fa618066e40 2024-02-19T15:23:03.398 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:03.400 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:23:03.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.396+0000 7fa63c800640 1 -- 172.21.15.47:0/2580674020 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa618066e40 msgr2=0x7fa618069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:03.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.396+0000 7fa63c800640 1 --2- 172.21.15.47:0/2580674020 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa618066e40 0x7fa618069300 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa634072250 tx=0x7fa624009040 comp rx=0 tx=0).stop 2024-02-19T15:23:03.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.396+0000 7fa63c800640 1 -- 172.21.15.47:0/2580674020 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6340cea80 msgr2=0x7fa63413b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:03.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.396+0000 7fa63c800640 1 --2- 172.21.15.47:0/2580674020 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6340cea80 0x7fa63413b780 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa62800c750 tx=0x7fa62800cc10 comp rx=0 tx=0).stop 2024-02-19T15:23:03.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.396+0000 7fa63c800640 1 -- 172.21.15.47:0/2580674020 shutdown_connections 2024-02-19T15:23:03.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.396+0000 7fa63c800640 1 --2- 172.21.15.47:0/2580674020 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa618066e40 0x7fa618069300 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:03.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.396+0000 7fa63c800640 1 --2- 172.21.15.47:0/2580674020 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa63415caa0 0x7fa634142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:03.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.396+0000 7fa63c800640 1 --2- 172.21.15.47:0/2580674020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa634145590 0x7fa63415a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:03.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.396+0000 7fa63c800640 1 --2- 172.21.15.47:0/2580674020 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6340cea80 0x7fa63413b780 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:03.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.396+0000 7fa63c800640 1 -- 172.21.15.47:0/2580674020 >> 172.21.15.47:0/2580674020 conn(0x7fa6340b16a0 msgr2=0x7fa6340b0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:03.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.396+0000 7fa63c800640 1 -- 172.21.15.47:0/2580674020 shutdown_connections 2024-02-19T15:23:03.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.396+0000 7fa63c800640 1 -- 172.21.15.47:0/2580674020 wait complete. 2024-02-19T15:23:03.419 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-02-19T15:23:03.161177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":102,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":241140,"kb_used_data":4164,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754733580,"statfs":{"total":773094113280,"available":772847185920,"internally_reserved":0,"allocated":4263936,"data_stored":2364536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":242535986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.980496"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201483+0000","last_change":"2024-02-19T15:22:59.212722+0000","last_active":"2024-02-19T15:23:00.201483+0000","last_peered":"2024-02-19T15:23:00.201483+0000","last_clean":"2024-02-19T15:23:00.201483+0000","last_became_active":"2024-02-19T15:22:59.212414+0000","last_became_peered":"2024-02-19T15:22:59.212414+0000","last_unstale":"2024-02-19T15:23:00.201483+0000","last_undegraded":"2024-02-19T15:23:00.201483+0000","last_fullsized":"2024-02-19T15:23:00.201483+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:54:31.030957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662311+0000","last_change":"2024-02-19T15:22:59.209934+0000","last_active":"2024-02-19T15:23:00.662311+0000","last_peered":"2024-02-19T15:23:00.662311+0000","last_clean":"2024-02-19T15:23:00.662311+0000","last_became_active":"2024-02-19T15:22:59.209661+0000","last_became_peered":"2024-02-19T15:22:59.209661+0000","last_unstale":"2024-02-19T15:23:00.662311+0000","last_undegraded":"2024-02-19T15:23:00.662311+0000","last_fullsized":"2024-02-19T15:23:00.662311+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:54:08.451277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663450+0000","last_change":"2024-02-19T15:22:59.223748+0000","last_active":"2024-02-19T15:23:00.663450+0000","last_peered":"2024-02-19T15:23:00.663450+0000","last_clean":"2024-02-19T15:23:00.663450+0000","last_became_active":"2024-02-19T15:22:59.223178+0000","last_became_peered":"2024-02-19T15:22:59.223178+0000","last_unstale":"2024-02-19T15:23:00.663450+0000","last_undegraded":"2024-02-19T15:23:00.663450+0000","last_fullsized":"2024-02-19T15:23:00.663450+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:08.692279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201889+0000","last_change":"2024-02-19T15:22:59.234029+0000","last_active":"2024-02-19T15:23:00.201889+0000","last_peered":"2024-02-19T15:23:00.201889+0000","last_clean":"2024-02-19T15:23:00.201889+0000","last_became_active":"2024-02-19T15:22:59.233761+0000","last_became_peered":"2024-02-19T15:22:59.233761+0000","last_unstale":"2024-02-19T15:23:00.201889+0000","last_undegraded":"2024-02-19T15:23:00.201889+0000","last_fullsized":"2024-02-19T15:23:00.201889+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:21:24.942760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.200768+0000","last_change":"2024-02-19T15:22:59.226653+0000","last_active":"2024-02-19T15:23:00.200768+0000","last_peered":"2024-02-19T15:23:00.200768+0000","last_clean":"2024-02-19T15:23:00.200768+0000","last_became_active":"2024-02-19T15:22:59.226387+0000","last_became_peered":"2024-02-19T15:22:59.226387+0000","last_unstale":"2024-02-19T15:23:00.200768+0000","last_undegraded":"2024-02-19T15:23:00.200768+0000","last_fullsized":"2024-02-19T15:23:00.200768+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:00:08.809218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.661839+0000","last_change":"2024-02-19T15:22:59.216353+0000","last_active":"2024-02-19T15:23:00.661839+0000","last_peered":"2024-02-19T15:23:00.661839+0000","last_clean":"2024-02-19T15:23:00.661839+0000","last_became_active":"2024-02-19T15:22:59.216050+0000","last_became_peered":"2024-02-19T15:22:59.216050+0000","last_unstale":"2024-02-19T15:23:00.661839+0000","last_undegraded":"2024-02-19T15:23:00.661839+0000","last_fullsized":"2024-02-19T15:23:00.661839+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:46.146658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664160+0000","last_change":"2024-02-19T15:22:59.223166+0000","last_active":"2024-02-19T15:23:00.664160+0000","last_peered":"2024-02-19T15:23:00.664160+0000","last_clean":"2024-02-19T15:23:00.664160+0000","last_became_active":"2024-02-19T15:22:59.222718+0000","last_became_peered":"2024-02-19T15:22:59.222718+0000","last_unstale":"2024-02-19T15:23:00.664160+0000","last_undegraded":"2024-02-19T15:23:00.664160+0000","last_fullsized":"2024-02-19T15:23:00.664160+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:25:08.609489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662633+0000","last_change":"2024-02-19T15:22:59.228518+0000","last_active":"2024-02-19T15:23:00.662633+0000","last_peered":"2024-02-19T15:23:00.662633+0000","last_clean":"2024-02-19T15:23:00.662633+0000","last_became_active":"2024-02-19T15:22:59.227787+0000","last_became_peered":"2024-02-19T15:22:59.227787+0000","last_unstale":"2024-02-19T15:23:00.662633+0000","last_undegraded":"2024-02-19T15:23:00.662633+0000","last_fullsized":"2024-02-19T15:23:00.662633+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:05:30.246561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664709+0000","last_change":"2024-02-19T15:22:59.222925+0000","last_active":"2024-02-19T15:23:00.664709+0000","last_peered":"2024-02-19T15:23:00.664709+0000","last_clean":"2024-02-19T15:23:00.664709+0000","last_became_active":"2024-02-19T15:22:59.222493+0000","last_became_peered":"2024-02-19T15:22:59.222493+0000","last_unstale":"2024-02-19T15:23:00.664709+0000","last_undegraded":"2024-02-19T15:23:00.664709+0000","last_fullsized":"2024-02-19T15:23:00.664709+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:48:02.181707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.202102+0000","last_change":"2024-02-19T15:22:59.230647+0000","last_active":"2024-02-19T15:23:00.202102+0000","last_peered":"2024-02-19T15:23:00.202102+0000","last_clean":"2024-02-19T15:23:00.202102+0000","last_became_active":"2024-02-19T15:22:59.229997+0000","last_became_peered":"2024-02-19T15:22:59.229997+0000","last_unstale":"2024-02-19T15:23:00.202102+0000","last_undegraded":"2024-02-19T15:23:00.202102+0000","last_fullsized":"2024-02-19T15:23:00.202102+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:06:58.745577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663898+0000","last_change":"2024-02-19T15:22:59.228636+0000","last_active":"2024-02-19T15:23:00.663898+0000","last_peered":"2024-02-19T15:23:00.663898+0000","last_clean":"2024-02-19T15:23:00.663898+0000","last_became_active":"2024-02-19T15:22:59.227852+0000","last_became_peered":"2024-02-19T15:22:59.227852+0000","last_unstale":"2024-02-19T15:23:00.663898+0000","last_undegraded":"2024-02-19T15:23:00.663898+0000","last_fullsized":"2024-02-19T15:23:00.663898+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:12:52.929727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662959+0000","last_change":"2024-02-19T15:22:59.221574+0000","last_active":"2024-02-19T15:23:00.662959+0000","last_peered":"2024-02-19T15:23:00.662959+0000","last_clean":"2024-02-19T15:23:00.662959+0000","last_became_active":"2024-02-19T15:22:59.221223+0000","last_became_peered":"2024-02-19T15:22:59.221223+0000","last_unstale":"2024-02-19T15:23:00.662959+0000","last_undegraded":"2024-02-19T15:23:00.662959+0000","last_fullsized":"2024-02-19T15:23:00.662959+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:35:59.446260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.200424+0000","last_change":"2024-02-19T15:22:59.215552+0000","last_active":"2024-02-19T15:23:00.200424+0000","last_peered":"2024-02-19T15:23:00.200424+0000","last_clean":"2024-02-19T15:23:00.200424+0000","last_became_active":"2024-02-19T15:22:59.215185+0000","last_became_peered":"2024-02-19T15:22:59.215185+0000","last_unstale":"2024-02-19T15:23:00.200424+0000","last_undegraded":"2024-02-19T15:23:00.200424+0000","last_fullsized":"2024-02-19T15:23:00.200424+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:41:07.184621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.661979+0000","last_change":"2024-02-19T15:22:59.209918+0000","last_active":"2024-02-19T15:23:00.661979+0000","last_peered":"2024-02-19T15:23:00.661979+0000","last_clean":"2024-02-19T15:23:00.661979+0000","last_became_active":"2024-02-19T15:22:59.209558+0000","last_became_peered":"2024-02-19T15:22:59.209558+0000","last_unstale":"2024-02-19T15:23:00.661979+0000","last_undegraded":"2024-02-19T15:23:00.661979+0000","last_fullsized":"2024-02-19T15:23:00.661979+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:35:59.446260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662383+0000","last_change":"2024-02-19T15:22:59.213739+0000","last_active":"2024-02-19T15:23:00.662383+0000","last_peered":"2024-02-19T15:23:00.662383+0000","last_clean":"2024-02-19T15:23:00.662383+0000","last_became_active":"2024-02-19T15:22:59.213370+0000","last_became_peered":"2024-02-19T15:22:59.213370+0000","last_unstale":"2024-02-19T15:23:00.662383+0000","last_undegraded":"2024-02-19T15:23:00.662383+0000","last_fullsized":"2024-02-19T15:23:00.662383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:46.146658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663044+0000","last_change":"2024-02-19T15:22:59.228970+0000","last_active":"2024-02-19T15:23:00.663044+0000","last_peered":"2024-02-19T15:23:00.663044+0000","last_clean":"2024-02-19T15:23:00.663044+0000","last_became_active":"2024-02-19T15:22:59.228152+0000","last_became_peered":"2024-02-19T15:22:59.228152+0000","last_unstale":"2024-02-19T15:23:00.663044+0000","last_undegraded":"2024-02-19T15:23:00.663044+0000","last_fullsized":"2024-02-19T15:23:00.663044+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:42.769874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664244+0000","last_change":"2024-02-19T15:22:59.226022+0000","last_active":"2024-02-19T15:23:00.664244+0000","last_peered":"2024-02-19T15:23:00.664244+0000","last_clean":"2024-02-19T15:23:00.664244+0000","last_became_active":"2024-02-19T15:22:59.225628+0000","last_became_peered":"2024-02-19T15:22:59.225628+0000","last_unstale":"2024-02-19T15:23:00.664244+0000","last_undegraded":"2024-02-19T15:23:00.664244+0000","last_fullsized":"2024-02-19T15:23:00.664244+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:05:22.259533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662360+0000","last_change":"2024-02-19T15:22:59.217500+0000","last_active":"2024-02-19T15:23:00.662360+0000","last_peered":"2024-02-19T15:23:00.662360+0000","last_clean":"2024-02-19T15:23:00.662360+0000","last_became_active":"2024-02-19T15:22:59.217191+0000","last_became_peered":"2024-02-19T15:22:59.217191+0000","last_unstale":"2024-02-19T15:23:00.662360+0000","last_undegraded":"2024-02-19T15:23:00.662360+0000","last_fullsized":"2024-02-19T15:23:00.662360+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:42.769874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662032+0000","last_change":"2024-02-19T15:22:59.220277+0000","last_active":"2024-02-19T15:23:00.662032+0000","last_peered":"2024-02-19T15:23:00.662032+0000","last_clean":"2024-02-19T15:23:00.662032+0000","last_became_active":"2024-02-19T15:22:59.219964+0000","last_became_peered":"2024-02-19T15:22:59.219964+0000","last_unstale":"2024-02-19T15:23:00.662032+0000","last_undegraded":"2024-02-19T15:23:00.662032+0000","last_fullsized":"2024-02-19T15:23:00.662032+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:05:30.246561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.200694+0000","last_change":"2024-02-19T15:22:59.213027+0000","last_active":"2024-02-19T15:23:00.200694+0000","last_peered":"2024-02-19T15:23:00.200694+0000","last_clean":"2024-02-19T15:23:00.200694+0000","last_became_active":"2024-02-19T15:22:59.212762+0000","last_became_peered":"2024-02-19T15:22:59.212762+0000","last_unstale":"2024-02-19T15:23:00.200694+0000","last_undegraded":"2024-02-19T15:23:00.200694+0000","last_fullsized":"2024-02-19T15:23:00.200694+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:54.369825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663713+0000","last_change":"2024-02-19T15:22:59.209802+0000","last_active":"2024-02-19T15:23:00.663713+0000","last_peered":"2024-02-19T15:23:00.663713+0000","last_clean":"2024-02-19T15:23:00.663713+0000","last_became_active":"2024-02-19T15:22:59.209421+0000","last_became_peered":"2024-02-19T15:22:59.209421+0000","last_unstale":"2024-02-19T15:23:00.663713+0000","last_undegraded":"2024-02-19T15:23:00.663713+0000","last_fullsized":"2024-02-19T15:23:00.663713+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:41.612341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663929+0000","last_change":"2024-02-19T15:22:59.222070+0000","last_active":"2024-02-19T15:23:00.663929+0000","last_peered":"2024-02-19T15:23:00.663929+0000","last_clean":"2024-02-19T15:23:00.663929+0000","last_became_active":"2024-02-19T15:22:59.221680+0000","last_became_peered":"2024-02-19T15:22:59.221680+0000","last_unstale":"2024-02-19T15:23:00.663929+0000","last_undegraded":"2024-02-19T15:23:00.663929+0000","last_fullsized":"2024-02-19T15:23:00.663929+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:06:09.543916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.226469+0000","last_change":"2024-02-19T15:22:59.224876+0000","last_active":"2024-02-19T15:23:00.226469+0000","last_peered":"2024-02-19T15:23:00.226469+0000","last_clean":"2024-02-19T15:23:00.226469+0000","last_became_active":"2024-02-19T15:22:59.224559+0000","last_became_peered":"2024-02-19T15:22:59.224559+0000","last_unstale":"2024-02-19T15:23:00.226469+0000","last_undegraded":"2024-02-19T15:23:00.226469+0000","last_fullsized":"2024-02-19T15:23:00.226469+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:42:37.450463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662291+0000","last_change":"2024-02-19T15:22:59.209467+0000","last_active":"2024-02-19T15:23:00.662291+0000","last_peered":"2024-02-19T15:23:00.662291+0000","last_clean":"2024-02-19T15:23:00.662291+0000","last_became_active":"2024-02-19T15:22:59.209175+0000","last_became_peered":"2024-02-19T15:22:59.209175+0000","last_unstale":"2024-02-19T15:23:00.662291+0000","last_undegraded":"2024-02-19T15:23:00.662291+0000","last_fullsized":"2024-02-19T15:23:00.662291+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:23:35.768395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.202330+0000","last_change":"2024-02-19T15:22:59.232815+0000","last_active":"2024-02-19T15:23:00.202330+0000","last_peered":"2024-02-19T15:23:00.202330+0000","last_clean":"2024-02-19T15:23:00.202330+0000","last_became_active":"2024-02-19T15:22:59.232553+0000","last_became_peered":"2024-02-19T15:22:59.232553+0000","last_unstale":"2024-02-19T15:23:00.202330+0000","last_undegraded":"2024-02-19T15:23:00.202330+0000","last_fullsized":"2024-02-19T15:23:00.202330+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:46.146658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662466+0000","last_change":"2024-02-19T15:22:59.233785+0000","last_active":"2024-02-19T15:23:00.662466+0000","last_peered":"2024-02-19T15:23:00.662466+0000","last_clean":"2024-02-19T15:23:00.662466+0000","last_became_active":"2024-02-19T15:22:59.233468+0000","last_became_peered":"2024-02-19T15:22:59.233468+0000","last_unstale":"2024-02-19T15:23:00.662466+0000","last_undegraded":"2024-02-19T15:23:00.662466+0000","last_fullsized":"2024-02-19T15:23:00.662466+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:53.399488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.202138+0000","last_change":"2024-02-19T15:22:59.234009+0000","last_active":"2024-02-19T15:23:00.202138+0000","last_peered":"2024-02-19T15:23:00.202138+0000","last_clean":"2024-02-19T15:23:00.202138+0000","last_became_active":"2024-02-19T15:22:59.233727+0000","last_became_peered":"2024-02-19T15:22:59.233727+0000","last_unstale":"2024-02-19T15:23:00.202138+0000","last_undegraded":"2024-02-19T15:23:00.202138+0000","last_fullsized":"2024-02-19T15:23:00.202138+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:27:04.060833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662972+0000","last_change":"2024-02-19T15:22:59.217810+0000","last_active":"2024-02-19T15:23:00.662972+0000","last_peered":"2024-02-19T15:23:00.662972+0000","last_clean":"2024-02-19T15:23:00.662972+0000","last_became_active":"2024-02-19T15:22:59.217362+0000","last_became_peered":"2024-02-19T15:22:59.217362+0000","last_unstale":"2024-02-19T15:23:00.662972+0000","last_undegraded":"2024-02-19T15:23:00.662972+0000","last_fullsized":"2024-02-19T15:23:00.662972+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:05:30.246561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663872+0000","last_change":"2024-02-19T15:22:59.222425+0000","last_active":"2024-02-19T15:23:00.663872+0000","last_peered":"2024-02-19T15:23:00.663872+0000","last_clean":"2024-02-19T15:23:00.663872+0000","last_became_active":"2024-02-19T15:22:59.221865+0000","last_became_peered":"2024-02-19T15:22:59.221865+0000","last_unstale":"2024-02-19T15:23:00.663872+0000","last_undegraded":"2024-02-19T15:23:00.663872+0000","last_fullsized":"2024-02-19T15:23:00.663872+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:01:10.613097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663858+0000","last_change":"2024-02-19T15:22:59.225668+0000","last_active":"2024-02-19T15:23:00.663858+0000","last_peered":"2024-02-19T15:23:00.663858+0000","last_clean":"2024-02-19T15:23:00.663858+0000","last_became_active":"2024-02-19T15:22:59.225322+0000","last_became_peered":"2024-02-19T15:22:59.225322+0000","last_unstale":"2024-02-19T15:23:00.663858+0000","last_undegraded":"2024-02-19T15:23:00.663858+0000","last_fullsized":"2024-02-19T15:23:00.663858+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:39:07.808039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201689+0000","last_change":"2024-02-19T15:22:59.218650+0000","last_active":"2024-02-19T15:23:00.201689+0000","last_peered":"2024-02-19T15:23:00.201689+0000","last_clean":"2024-02-19T15:23:00.201689+0000","last_became_active":"2024-02-19T15:22:59.218254+0000","last_became_peered":"2024-02-19T15:22:59.218254+0000","last_unstale":"2024-02-19T15:23:00.201689+0000","last_undegraded":"2024-02-19T15:23:00.201689+0000","last_fullsized":"2024-02-19T15:23:00.201689+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:04:26.085669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662736+0000","last_change":"2024-02-19T15:22:59.232336+0000","last_active":"2024-02-19T15:23:00.662736+0000","last_peered":"2024-02-19T15:23:00.662736+0000","last_clean":"2024-02-19T15:23:00.662736+0000","last_became_active":"2024-02-19T15:22:59.231664+0000","last_became_peered":"2024-02-19T15:22:59.231664+0000","last_unstale":"2024-02-19T15:23:00.662736+0000","last_undegraded":"2024-02-19T15:23:00.662736+0000","last_fullsized":"2024-02-19T15:23:00.662736+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:52:10.521631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.226696+0000","last_change":"2024-02-19T15:22:59.222179+0000","last_active":"2024-02-19T15:23:00.226696+0000","last_peered":"2024-02-19T15:23:00.226696+0000","last_clean":"2024-02-19T15:23:00.226696+0000","last_became_active":"2024-02-19T15:22:59.221907+0000","last_became_peered":"2024-02-19T15:22:59.221907+0000","last_unstale":"2024-02-19T15:23:00.226696+0000","last_undegraded":"2024-02-19T15:23:00.226696+0000","last_fullsized":"2024-02-19T15:23:00.226696+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:23:51.466880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201537+0000","last_change":"2024-02-19T15:22:58.202088+0000","last_active":"2024-02-19T15:23:00.201537+0000","last_peered":"2024-02-19T15:23:00.201537+0000","last_clean":"2024-02-19T15:23:00.201537+0000","last_became_active":"2024-02-19T15:22:58.201781+0000","last_became_peered":"2024-02-19T15:22:58.201781+0000","last_unstale":"2024-02-19T15:23:00.201537+0000","last_undegraded":"2024-02-19T15:23:00.201537+0000","last_fullsized":"2024-02-19T15:23:00.201537+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:01:28.252912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201510+0000","last_change":"2024-02-19T15:22:59.220518+0000","last_active":"2024-02-19T15:23:00.201510+0000","last_peered":"2024-02-19T15:23:00.201510+0000","last_clean":"2024-02-19T15:23:00.201510+0000","last_became_active":"2024-02-19T15:22:59.220215+0000","last_became_peered":"2024-02-19T15:22:59.220215+0000","last_unstale":"2024-02-19T15:23:00.201510+0000","last_undegraded":"2024-02-19T15:23:00.201510+0000","last_fullsized":"2024-02-19T15:23:00.201510+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:51:49.244632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.226747+0000","last_change":"2024-02-19T15:22:58.212160+0000","last_active":"2024-02-19T15:23:00.226747+0000","last_peered":"2024-02-19T15:23:00.226747+0000","last_clean":"2024-02-19T15:23:00.226747+0000","last_became_active":"2024-02-19T15:22:58.211748+0000","last_became_peered":"2024-02-19T15:22:58.211748+0000","last_unstale":"2024-02-19T15:23:00.226747+0000","last_undegraded":"2024-02-19T15:23:00.226747+0000","last_fullsized":"2024-02-19T15:23:00.226747+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:48:14.739039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664821+0000","last_change":"2024-02-19T15:22:59.219546+0000","last_active":"2024-02-19T15:23:00.664821+0000","last_peered":"2024-02-19T15:23:00.664821+0000","last_clean":"2024-02-19T15:23:00.664821+0000","last_became_active":"2024-02-19T15:22:59.219303+0000","last_became_peered":"2024-02-19T15:22:59.219303+0000","last_unstale":"2024-02-19T15:23:00.664821+0000","last_undegraded":"2024-02-19T15:23:00.664821+0000","last_fullsized":"2024-02-19T15:23:00.664821+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:30:20.556027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662098+0000","last_change":"2024-02-19T15:22:58.210229+0000","last_active":"2024-02-19T15:23:00.662098+0000","last_peered":"2024-02-19T15:23:00.662098+0000","last_clean":"2024-02-19T15:23:00.662098+0000","last_became_active":"2024-02-19T15:22:58.209101+0000","last_became_peered":"2024-02-19T15:22:58.209101+0000","last_unstale":"2024-02-19T15:23:00.662098+0000","last_undegraded":"2024-02-19T15:23:00.662098+0000","last_fullsized":"2024-02-19T15:23:00.662098+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:32:40.017025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.202261+0000","last_change":"2024-02-19T15:22:59.233156+0000","last_active":"2024-02-19T15:23:00.202261+0000","last_peered":"2024-02-19T15:23:00.202261+0000","last_clean":"2024-02-19T15:23:00.202261+0000","last_became_active":"2024-02-19T15:22:59.232879+0000","last_became_peered":"2024-02-19T15:22:59.232879+0000","last_unstale":"2024-02-19T15:23:00.202261+0000","last_undegraded":"2024-02-19T15:23:00.202261+0000","last_fullsized":"2024-02-19T15:23:00.202261+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:23:35.768395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.200800+0000","last_change":"2024-02-19T15:22:58.207825+0000","last_active":"2024-02-19T15:23:00.200800+0000","last_peered":"2024-02-19T15:23:00.200800+0000","last_clean":"2024-02-19T15:23:00.200800+0000","last_became_active":"2024-02-19T15:22:58.207526+0000","last_became_peered":"2024-02-19T15:22:58.207526+0000","last_unstale":"2024-02-19T15:23:00.200800+0000","last_undegraded":"2024-02-19T15:23:00.200800+0000","last_fullsized":"2024-02-19T15:23:00.200800+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:47:08.185258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663719+0000","last_change":"2024-02-19T15:22:59.233487+0000","last_active":"2024-02-19T15:23:00.663719+0000","last_peered":"2024-02-19T15:23:00.663719+0000","last_clean":"2024-02-19T15:23:00.663719+0000","last_became_active":"2024-02-19T15:22:59.233142+0000","last_became_peered":"2024-02-19T15:22:59.233142+0000","last_unstale":"2024-02-19T15:23:00.663719+0000","last_undegraded":"2024-02-19T15:23:00.663719+0000","last_fullsized":"2024-02-19T15:23:00.663719+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:53.399488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664252+0000","last_change":"2024-02-19T15:22:58.204949+0000","last_active":"2024-02-19T15:23:00.664252+0000","last_peered":"2024-02-19T15:23:00.664252+0000","last_clean":"2024-02-19T15:23:00.664252+0000","last_became_active":"2024-02-19T15:22:58.204693+0000","last_became_peered":"2024-02-19T15:22:58.204693+0000","last_unstale":"2024-02-19T15:23:00.664252+0000","last_undegraded":"2024-02-19T15:23:00.664252+0000","last_fullsized":"2024-02-19T15:23:00.664252+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:31:52.558573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663098+0000","last_change":"2024-02-19T15:22:59.210037+0000","last_active":"2024-02-19T15:23:00.663098+0000","last_peered":"2024-02-19T15:23:00.663098+0000","last_clean":"2024-02-19T15:23:00.663098+0000","last_became_active":"2024-02-19T15:22:59.209168+0000","last_became_peered":"2024-02-19T15:22:59.209168+0000","last_unstale":"2024-02-19T15:23:00.663098+0000","last_undegraded":"2024-02-19T15:23:00.663098+0000","last_fullsized":"2024-02-19T15:23:00.663098+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:52:10.521631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664063+0000","last_change":"2024-02-19T15:22:58.207343+0000","last_active":"2024-02-19T15:23:00.664063+0000","last_peered":"2024-02-19T15:23:00.664063+0000","last_clean":"2024-02-19T15:23:00.664063+0000","last_became_active":"2024-02-19T15:22:58.207010+0000","last_became_peered":"2024-02-19T15:22:58.207010+0000","last_unstale":"2024-02-19T15:23:00.664063+0000","last_undegraded":"2024-02-19T15:23:00.664063+0000","last_fullsized":"2024-02-19T15:23:00.664063+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:53.363347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663291+0000","last_change":"2024-02-19T15:22:59.210486+0000","last_active":"2024-02-19T15:23:00.663291+0000","last_peered":"2024-02-19T15:23:00.663291+0000","last_clean":"2024-02-19T15:23:00.663291+0000","last_became_active":"2024-02-19T15:22:59.210163+0000","last_became_peered":"2024-02-19T15:22:59.210163+0000","last_unstale":"2024-02-19T15:23:00.663291+0000","last_undegraded":"2024-02-19T15:23:00.663291+0000","last_fullsized":"2024-02-19T15:23:00.663291+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:13:34.151422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.661850+0000","last_change":"2024-02-19T15:22:58.203400+0000","last_active":"2024-02-19T15:23:00.661850+0000","last_peered":"2024-02-19T15:23:00.661850+0000","last_clean":"2024-02-19T15:23:00.661850+0000","last_became_active":"2024-02-19T15:22:58.203078+0000","last_became_peered":"2024-02-19T15:22:58.203078+0000","last_unstale":"2024-02-19T15:23:00.661850+0000","last_undegraded":"2024-02-19T15:23:00.661850+0000","last_fullsized":"2024-02-19T15:23:00.661850+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:48.695445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201704+0000","last_change":"2024-02-19T15:22:59.224076+0000","last_active":"2024-02-19T15:23:00.201704+0000","last_peered":"2024-02-19T15:23:00.201704+0000","last_clean":"2024-02-19T15:23:00.201704+0000","last_became_active":"2024-02-19T15:22:59.223783+0000","last_became_peered":"2024-02-19T15:22:59.223783+0000","last_unstale":"2024-02-19T15:23:00.201704+0000","last_undegraded":"2024-02-19T15:23:00.201704+0000","last_fullsized":"2024-02-19T15:23:00.201704+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:39:07.966403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662823+0000","last_change":"2024-02-19T15:22:58.197186+0000","last_active":"2024-02-19T15:23:00.662823+0000","last_peered":"2024-02-19T15:23:00.662823+0000","last_clean":"2024-02-19T15:23:00.662823+0000","last_became_active":"2024-02-19T15:22:58.196405+0000","last_became_peered":"2024-02-19T15:22:58.196405+0000","last_unstale":"2024-02-19T15:23:00.662823+0000","last_undegraded":"2024-02-19T15:23:00.662823+0000","last_fullsized":"2024-02-19T15:23:00.662823+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:42:21.548099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663211+0000","last_change":"2024-02-19T15:22:59.228875+0000","last_active":"2024-02-19T15:23:00.663211+0000","last_peered":"2024-02-19T15:23:00.663211+0000","last_clean":"2024-02-19T15:23:00.663211+0000","last_became_active":"2024-02-19T15:22:59.228548+0000","last_became_peered":"2024-02-19T15:22:59.228548+0000","last_unstale":"2024-02-19T15:23:00.663211+0000","last_undegraded":"2024-02-19T15:23:00.663211+0000","last_fullsized":"2024-02-19T15:23:00.663211+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:54:08.451277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662414+0000","last_change":"2024-02-19T15:22:58.209590+0000","last_active":"2024-02-19T15:23:00.662414+0000","last_peered":"2024-02-19T15:23:00.662414+0000","last_clean":"2024-02-19T15:23:00.662414+0000","last_became_active":"2024-02-19T15:22:58.209253+0000","last_became_peered":"2024-02-19T15:22:58.209253+0000","last_unstale":"2024-02-19T15:23:00.662414+0000","last_undegraded":"2024-02-19T15:23:00.662414+0000","last_fullsized":"2024-02-19T15:23:00.662414+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:40.605863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662430+0000","last_change":"2024-02-19T15:22:59.219447+0000","last_active":"2024-02-19T15:23:00.662430+0000","last_peered":"2024-02-19T15:23:00.662430+0000","last_clean":"2024-02-19T15:23:00.662430+0000","last_became_active":"2024-02-19T15:22:59.219172+0000","last_became_peered":"2024-02-19T15:22:59.219172+0000","last_unstale":"2024-02-19T15:23:00.662430+0000","last_undegraded":"2024-02-19T15:23:00.662430+0000","last_fullsized":"2024-02-19T15:23:00.662430+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:28:04.641517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.226864+0000","last_change":"2024-02-19T15:22:58.209938+0000","last_active":"2024-02-19T15:23:00.226864+0000","last_peered":"2024-02-19T15:23:00.226864+0000","last_clean":"2024-02-19T15:23:00.226864+0000","last_became_active":"2024-02-19T15:22:58.209429+0000","last_became_peered":"2024-02-19T15:22:58.209429+0000","last_unstale":"2024-02-19T15:23:00.226864+0000","last_undegraded":"2024-02-19T15:23:00.226864+0000","last_fullsized":"2024-02-19T15:23:00.226864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:54:21.839011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662382+0000","last_change":"2024-02-19T15:22:59.228204+0000","last_active":"2024-02-19T15:23:00.662382+0000","last_peered":"2024-02-19T15:23:00.662382+0000","last_clean":"2024-02-19T15:23:00.662382+0000","last_became_active":"2024-02-19T15:22:59.227900+0000","last_became_peered":"2024-02-19T15:22:59.227900+0000","last_unstale":"2024-02-19T15:23:00.662382+0000","last_undegraded":"2024-02-19T15:23:00.662382+0000","last_fullsized":"2024-02-19T15:23:00.662382+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:08.692279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":17,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663464+0000","last_change":"2024-02-19T15:22:58.208360+0000","last_active":"2024-02-19T15:23:00.663464+0000","last_peered":"2024-02-19T15:23:00.663464+0000","last_clean":"2024-02-19T15:23:00.663464+0000","last_became_active":"2024-02-19T15:22:58.207987+0000","last_became_peered":"2024-02-19T15:22:58.207987+0000","last_unstale":"2024-02-19T15:23:00.663464+0000","last_undegraded":"2024-02-19T15:23:00.663464+0000","last_fullsized":"2024-02-19T15:23:00.663464+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:01:09.606619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663309+0000","last_change":"2024-02-19T15:22:59.224735+0000","last_active":"2024-02-19T15:23:00.663309+0000","last_peered":"2024-02-19T15:23:00.663309+0000","last_clean":"2024-02-19T15:23:00.663309+0000","last_became_active":"2024-02-19T15:22:59.223502+0000","last_became_peered":"2024-02-19T15:22:59.223502+0000","last_unstale":"2024-02-19T15:23:00.663309+0000","last_undegraded":"2024-02-19T15:23:00.663309+0000","last_fullsized":"2024-02-19T15:23:00.663309+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:42.769874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662412+0000","last_change":"2024-02-19T15:22:58.211326+0000","last_active":"2024-02-19T15:23:00.662412+0000","last_peered":"2024-02-19T15:23:00.662412+0000","last_clean":"2024-02-19T15:23:00.662412+0000","last_became_active":"2024-02-19T15:22:58.211004+0000","last_became_peered":"2024-02-19T15:22:58.211004+0000","last_unstale":"2024-02-19T15:23:00.662412+0000","last_undegraded":"2024-02-19T15:23:00.662412+0000","last_fullsized":"2024-02-19T15:23:00.662412+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:40:31.134963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201927+0000","last_change":"2024-02-19T15:22:59.222270+0000","last_active":"2024-02-19T15:23:00.201927+0000","last_peered":"2024-02-19T15:23:00.201927+0000","last_clean":"2024-02-19T15:23:00.201927+0000","last_became_active":"2024-02-19T15:22:59.221994+0000","last_became_peered":"2024-02-19T15:22:59.221994+0000","last_unstale":"2024-02-19T15:23:00.201927+0000","last_undegraded":"2024-02-19T15:23:00.201927+0000","last_fullsized":"2024-02-19T15:23:00.201927+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:22:26.006732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664312+0000","last_change":"2024-02-19T15:22:58.198232+0000","last_active":"2024-02-19T15:23:00.664312+0000","last_peered":"2024-02-19T15:23:00.664312+0000","last_clean":"2024-02-19T15:23:00.664312+0000","last_became_active":"2024-02-19T15:22:58.197783+0000","last_became_peered":"2024-02-19T15:22:58.197783+0000","last_unstale":"2024-02-19T15:23:00.664312+0000","last_undegraded":"2024-02-19T15:23:00.664312+0000","last_fullsized":"2024-02-19T15:23:00.664312+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:04:51.651313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662050+0000","last_change":"2024-02-19T15:22:59.210621+0000","last_active":"2024-02-19T15:23:00.662050+0000","last_peered":"2024-02-19T15:23:00.662050+0000","last_clean":"2024-02-19T15:23:00.662050+0000","last_became_active":"2024-02-19T15:22:59.210387+0000","last_became_peered":"2024-02-19T15:22:59.210387+0000","last_unstale":"2024-02-19T15:23:00.662050+0000","last_undegraded":"2024-02-19T15:23:00.662050+0000","last_fullsized":"2024-02-19T15:23:00.662050+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:52:10.521631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.202290+0000","last_change":"2024-02-19T15:22:58.207165+0000","last_active":"2024-02-19T15:23:00.202290+0000","last_peered":"2024-02-19T15:23:00.202290+0000","last_clean":"2024-02-19T15:23:00.202290+0000","last_became_active":"2024-02-19T15:22:58.206682+0000","last_became_peered":"2024-02-19T15:22:58.206682+0000","last_unstale":"2024-02-19T15:23:00.202290+0000","last_undegraded":"2024-02-19T15:23:00.202290+0000","last_fullsized":"2024-02-19T15:23:00.202290+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:41:12.383107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663012+0000","last_change":"2024-02-19T15:22:59.232469+0000","last_active":"2024-02-19T15:23:00.663012+0000","last_peered":"2024-02-19T15:23:00.663012+0000","last_clean":"2024-02-19T15:23:00.663012+0000","last_became_active":"2024-02-19T15:22:59.232016+0000","last_became_peered":"2024-02-19T15:22:59.232016+0000","last_unstale":"2024-02-19T15:23:00.663012+0000","last_undegraded":"2024-02-19T15:23:00.663012+0000","last_fullsized":"2024-02-19T15:23:00.663012+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:54:08.451277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.661755+0000","last_change":"2024-02-19T15:22:58.209673+0000","last_active":"2024-02-19T15:23:00.661755+0000","last_peered":"2024-02-19T15:23:00.661755+0000","last_clean":"2024-02-19T15:23:00.661755+0000","last_became_active":"2024-02-19T15:22:58.209178+0000","last_became_peered":"2024-02-19T15:22:58.209178+0000","last_unstale":"2024-02-19T15:23:00.661755+0000","last_undegraded":"2024-02-19T15:23:00.661755+0000","last_fullsized":"2024-02-19T15:23:00.661755+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:29:15.629841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201631+0000","last_change":"2024-02-19T15:22:59.220850+0000","last_active":"2024-02-19T15:23:00.201631+0000","last_peered":"2024-02-19T15:23:00.201631+0000","last_clean":"2024-02-19T15:23:00.201631+0000","last_became_active":"2024-02-19T15:22:59.220526+0000","last_became_peered":"2024-02-19T15:22:59.220526+0000","last_unstale":"2024-02-19T15:23:00.201631+0000","last_undegraded":"2024-02-19T15:23:00.201631+0000","last_fullsized":"2024-02-19T15:23:00.201631+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:42:37.450463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":22,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.661908+0000","last_change":"2024-02-19T15:22:58.207005+0000","last_active":"2024-02-19T15:23:00.661908+0000","last_peered":"2024-02-19T15:23:00.661908+0000","last_clean":"2024-02-19T15:23:00.661908+0000","last_became_active":"2024-02-19T15:22:58.206744+0000","last_became_peered":"2024-02-19T15:22:58.206744+0000","last_unstale":"2024-02-19T15:23:00.661908+0000","last_undegraded":"2024-02-19T15:23:00.661908+0000","last_fullsized":"2024-02-19T15:23:00.661908+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:01:09.606619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664561+0000","last_change":"2024-02-19T15:22:59.217007+0000","last_active":"2024-02-19T15:23:00.664561+0000","last_peered":"2024-02-19T15:23:00.664561+0000","last_clean":"2024-02-19T15:23:00.664561+0000","last_became_active":"2024-02-19T15:22:59.216654+0000","last_became_peered":"2024-02-19T15:22:59.216654+0000","last_unstale":"2024-02-19T15:23:00.664561+0000","last_undegraded":"2024-02-19T15:23:00.664561+0000","last_fullsized":"2024-02-19T15:23:00.664561+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:40:17.123906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.202455+0000","last_change":"2024-02-19T15:22:58.213906+0000","last_active":"2024-02-19T15:23:00.202455+0000","last_peered":"2024-02-19T15:23:00.202455+0000","last_clean":"2024-02-19T15:23:00.202455+0000","last_became_active":"2024-02-19T15:22:58.213115+0000","last_became_peered":"2024-02-19T15:22:58.213115+0000","last_unstale":"2024-02-19T15:23:00.202455+0000","last_undegraded":"2024-02-19T15:23:00.202455+0000","last_fullsized":"2024-02-19T15:23:00.202455+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:48.695445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663732+0000","last_change":"2024-02-19T15:22:59.220627+0000","last_active":"2024-02-19T15:23:00.663732+0000","last_peered":"2024-02-19T15:23:00.663732+0000","last_clean":"2024-02-19T15:23:00.663732+0000","last_became_active":"2024-02-19T15:22:59.220362+0000","last_became_peered":"2024-02-19T15:22:59.220362+0000","last_unstale":"2024-02-19T15:23:00.663732+0000","last_undegraded":"2024-02-19T15:23:00.663732+0000","last_fullsized":"2024-02-19T15:23:00.663732+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:53.399488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664592+0000","last_change":"2024-02-19T15:22:58.198617+0000","last_active":"2024-02-19T15:23:00.664592+0000","last_peered":"2024-02-19T15:23:00.664592+0000","last_clean":"2024-02-19T15:23:00.664592+0000","last_became_active":"2024-02-19T15:22:58.198222+0000","last_became_peered":"2024-02-19T15:22:58.198222+0000","last_unstale":"2024-02-19T15:23:00.664592+0000","last_undegraded":"2024-02-19T15:23:00.664592+0000","last_fullsized":"2024-02-19T15:23:00.664592+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:09:55.343795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201960+0000","last_change":"2024-02-19T15:22:59.230410+0000","last_active":"2024-02-19T15:23:00.201960+0000","last_peered":"2024-02-19T15:23:00.201960+0000","last_clean":"2024-02-19T15:23:00.201960+0000","last_became_active":"2024-02-19T15:22:59.229960+0000","last_became_peered":"2024-02-19T15:22:59.229960+0000","last_unstale":"2024-02-19T15:23:00.201960+0000","last_undegraded":"2024-02-19T15:23:00.201960+0000","last_fullsized":"2024-02-19T15:23:00.201960+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:16:04.810073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664541+0000","last_change":"2024-02-19T15:22:58.205977+0000","last_active":"2024-02-19T15:23:00.664541+0000","last_peered":"2024-02-19T15:23:00.664541+0000","last_clean":"2024-02-19T15:23:00.664541+0000","last_became_active":"2024-02-19T15:22:58.199517+0000","last_became_peered":"2024-02-19T15:22:58.199517+0000","last_unstale":"2024-02-19T15:23:00.664541+0000","last_undegraded":"2024-02-19T15:23:00.664541+0000","last_fullsized":"2024-02-19T15:23:00.664541+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:47:09.696439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.200495+0000","last_change":"2024-02-19T15:22:59.229038+0000","last_active":"2024-02-19T15:23:00.200495+0000","last_peered":"2024-02-19T15:23:00.200495+0000","last_clean":"2024-02-19T15:23:00.200495+0000","last_became_active":"2024-02-19T15:22:59.228259+0000","last_became_peered":"2024-02-19T15:22:59.228259+0000","last_unstale":"2024-02-19T15:23:00.200495+0000","last_undegraded":"2024-02-19T15:23:00.200495+0000","last_fullsized":"2024-02-19T15:23:00.200495+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:47:10.702917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664528+0000","last_change":"2024-02-19T15:22:58.206492+0000","last_active":"2024-02-19T15:23:00.664528+0000","last_peered":"2024-02-19T15:23:00.664528+0000","last_clean":"2024-02-19T15:23:00.664528+0000","last_became_active":"2024-02-19T15:22:58.206214+0000","last_became_peered":"2024-02-19T15:22:58.206214+0000","last_unstale":"2024-02-19T15:23:00.664528+0000","last_undegraded":"2024-02-19T15:23:00.664528+0000","last_fullsized":"2024-02-19T15:23:00.664528+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:41:06.178143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.200413+0000","last_change":"2024-02-19T15:22:59.221429+0000","last_active":"2024-02-19T15:23:00.200413+0000","last_peered":"2024-02-19T15:23:00.200413+0000","last_clean":"2024-02-19T15:23:00.200413+0000","last_became_active":"2024-02-19T15:22:59.221098+0000","last_became_peered":"2024-02-19T15:22:59.221098+0000","last_unstale":"2024-02-19T15:23:00.200413+0000","last_undegraded":"2024-02-19T15:23:00.200413+0000","last_fullsized":"2024-02-19T15:23:00.200413+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:42:22.554577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.226693+0000","last_change":"2024-02-19T15:22:58.214753+0000","last_active":"2024-02-19T15:23:00.226693+0000","last_peered":"2024-02-19T15:23:00.226693+0000","last_clean":"2024-02-19T15:23:00.226693+0000","last_became_active":"2024-02-19T15:22:58.214415+0000","last_became_peered":"2024-02-19T15:22:58.214415+0000","last_unstale":"2024-02-19T15:23:00.226693+0000","last_undegraded":"2024-02-19T15:23:00.226693+0000","last_fullsized":"2024-02-19T15:23:00.226693+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:41:10.198513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.200582+0000","last_change":"2024-02-19T15:22:59.211355+0000","last_active":"2024-02-19T15:23:00.200582+0000","last_peered":"2024-02-19T15:23:00.200582+0000","last_clean":"2024-02-19T15:23:00.200582+0000","last_became_active":"2024-02-19T15:22:59.208205+0000","last_became_peered":"2024-02-19T15:22:59.208205+0000","last_unstale":"2024-02-19T15:23:00.200582+0000","last_undegraded":"2024-02-19T15:23:00.200582+0000","last_fullsized":"2024-02-19T15:23:00.200582+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:09:56.350273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201456+0000","last_change":"2024-02-19T15:22:58.207613+0000","last_active":"2024-02-19T15:23:00.201456+0000","last_peered":"2024-02-19T15:23:00.201456+0000","last_clean":"2024-02-19T15:23:00.201456+0000","last_became_active":"2024-02-19T15:22:58.207284+0000","last_became_peered":"2024-02-19T15:22:58.207284+0000","last_unstale":"2024-02-19T15:23:00.201456+0000","last_undegraded":"2024-02-19T15:23:00.201456+0000","last_fullsized":"2024-02-19T15:23:00.201456+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:45:00.304846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.200184+0000","last_change":"2024-02-19T15:22:59.208581+0000","last_active":"2024-02-19T15:23:00.200184+0000","last_peered":"2024-02-19T15:23:00.200184+0000","last_clean":"2024-02-19T15:23:00.200184+0000","last_became_active":"2024-02-19T15:22:59.208136+0000","last_became_peered":"2024-02-19T15:22:59.208136+0000","last_unstale":"2024-02-19T15:23:00.200184+0000","last_undegraded":"2024-02-19T15:23:00.200184+0000","last_fullsized":"2024-02-19T15:23:00.200184+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:04:52.657791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663008+0000","last_change":"2024-02-19T15:22:58.203803+0000","last_active":"2024-02-19T15:23:00.663008+0000","last_peered":"2024-02-19T15:23:00.663008+0000","last_clean":"2024-02-19T15:23:00.663008+0000","last_became_active":"2024-02-19T15:22:58.203518+0000","last_became_peered":"2024-02-19T15:22:58.203518+0000","last_unstale":"2024-02-19T15:23:00.663008+0000","last_undegraded":"2024-02-19T15:23:00.663008+0000","last_fullsized":"2024-02-19T15:23:00.663008+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:30:28.131448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663316+0000","last_change":"2024-02-19T15:22:59.222928+0000","last_active":"2024-02-19T15:23:00.663316+0000","last_peered":"2024-02-19T15:23:00.663316+0000","last_clean":"2024-02-19T15:23:00.663316+0000","last_became_active":"2024-02-19T15:22:59.222291+0000","last_became_peered":"2024-02-19T15:22:59.222291+0000","last_unstale":"2024-02-19T15:23:00.663316+0000","last_undegraded":"2024-02-19T15:23:00.663316+0000","last_fullsized":"2024-02-19T15:23:00.663316+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:08.692279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662562+0000","last_change":"2024-02-19T15:22:58.209230+0000","last_active":"2024-02-19T15:23:00.662562+0000","last_peered":"2024-02-19T15:23:00.662562+0000","last_clean":"2024-02-19T15:23:00.662562+0000","last_became_active":"2024-02-19T15:22:58.208914+0000","last_became_peered":"2024-02-19T15:22:58.208914+0000","last_unstale":"2024-02-19T15:23:00.662562+0000","last_undegraded":"2024-02-19T15:23:00.662562+0000","last_fullsized":"2024-02-19T15:23:00.662562+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:40.605863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.200924+0000","last_change":"2024-02-19T15:22:59.212386+0000","last_active":"2024-02-19T15:23:00.200924+0000","last_peered":"2024-02-19T15:23:00.200924+0000","last_clean":"2024-02-19T15:23:00.200924+0000","last_became_active":"2024-02-19T15:22:59.212114+0000","last_became_peered":"2024-02-19T15:22:59.212114+0000","last_unstale":"2024-02-19T15:23:00.200924+0000","last_undegraded":"2024-02-19T15:23:00.200924+0000","last_fullsized":"2024-02-19T15:23:00.200924+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:22:57.295384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.202503+0000","last_change":"2024-02-19T15:22:58.215574+0000","last_active":"2024-02-19T15:23:00.202503+0000","last_peered":"2024-02-19T15:23:00.202503+0000","last_clean":"2024-02-19T15:23:00.202503+0000","last_became_active":"2024-02-19T15:22:58.215320+0000","last_became_peered":"2024-02-19T15:22:58.215320+0000","last_unstale":"2024-02-19T15:23:00.202503+0000","last_undegraded":"2024-02-19T15:23:00.202503+0000","last_fullsized":"2024-02-19T15:23:00.202503+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:32:40.017025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663591+0000","last_change":"2024-02-19T15:22:59.222837+0000","last_active":"2024-02-19T15:23:00.663591+0000","last_peered":"2024-02-19T15:23:00.663591+0000","last_clean":"2024-02-19T15:23:00.663591+0000","last_became_active":"2024-02-19T15:22:59.222528+0000","last_became_peered":"2024-02-19T15:22:59.222528+0000","last_unstale":"2024-02-19T15:23:00.663591+0000","last_undegraded":"2024-02-19T15:23:00.663591+0000","last_fullsized":"2024-02-19T15:23:00.663591+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:28:04.641517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"34'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.228480+0000","last_change":"2024-02-19T15:22:58.214998+0000","last_active":"2024-02-19T15:23:00.228480+0000","last_peered":"2024-02-19T15:23:00.228480+0000","last_clean":"2024-02-19T15:23:00.228480+0000","last_became_active":"2024-02-19T15:22:58.214534+0000","last_became_peered":"2024-02-19T15:22:58.214534+0000","last_unstale":"2024-02-19T15:23:00.228480+0000","last_undegraded":"2024-02-19T15:23:00.228480+0000","last_fullsized":"2024-02-19T15:23:00.228480+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:41:23.269776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.226625+0000","last_change":"2024-02-19T15:22:59.221728+0000","last_active":"2024-02-19T15:23:00.226625+0000","last_peered":"2024-02-19T15:23:00.226625+0000","last_clean":"2024-02-19T15:23:00.226625+0000","last_became_active":"2024-02-19T15:22:59.219985+0000","last_became_peered":"2024-02-19T15:22:59.219985+0000","last_unstale":"2024-02-19T15:23:00.226625+0000","last_undegraded":"2024-02-19T15:23:00.226625+0000","last_fullsized":"2024-02-19T15:23:00.226625+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:04:26.085669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201987+0000","last_change":"2024-02-19T15:22:58.207764+0000","last_active":"2024-02-19T15:23:00.201987+0000","last_peered":"2024-02-19T15:23:00.201987+0000","last_clean":"2024-02-19T15:23:00.201987+0000","last_became_active":"2024-02-19T15:22:58.206673+0000","last_became_peered":"2024-02-19T15:22:58.206673+0000","last_unstale":"2024-02-19T15:23:00.201987+0000","last_undegraded":"2024-02-19T15:23:00.201987+0000","last_fullsized":"2024-02-19T15:23:00.201987+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:29:15.629841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201484+0000","last_change":"2024-02-19T15:22:59.217684+0000","last_active":"2024-02-19T15:23:00.201484+0000","last_peered":"2024-02-19T15:23:00.201484+0000","last_clean":"2024-02-19T15:23:00.201484+0000","last_became_active":"2024-02-19T15:22:59.217378+0000","last_became_peered":"2024-02-19T15:22:59.217378+0000","last_unstale":"2024-02-19T15:23:00.201484+0000","last_undegraded":"2024-02-19T15:23:00.201484+0000","last_fullsized":"2024-02-19T15:23:00.201484+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:26.125796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.226801+0000","last_change":"2024-02-19T15:22:58.205664+0000","last_active":"2024-02-19T15:23:00.226801+0000","last_peered":"2024-02-19T15:23:00.226801+0000","last_clean":"2024-02-19T15:23:00.226801+0000","last_became_active":"2024-02-19T15:22:58.205359+0000","last_became_peered":"2024-02-19T15:22:58.205359+0000","last_unstale":"2024-02-19T15:23:00.226801+0000","last_undegraded":"2024-02-19T15:23:00.226801+0000","last_fullsized":"2024-02-19T15:23:00.226801+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:01:28.252912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.226687+0000","last_change":"2024-02-19T15:22:59.210920+0000","last_active":"2024-02-19T15:23:00.226687+0000","last_peered":"2024-02-19T15:23:00.226687+0000","last_clean":"2024-02-19T15:23:00.226687+0000","last_became_active":"2024-02-19T15:22:59.210531+0000","last_became_peered":"2024-02-19T15:22:59.210531+0000","last_unstale":"2024-02-19T15:23:00.226687+0000","last_undegraded":"2024-02-19T15:23:00.226687+0000","last_fullsized":"2024-02-19T15:23:00.226687+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:56:32.688788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.202216+0000","last_change":"2024-02-19T15:22:58.215032+0000","last_active":"2024-02-19T15:23:00.202216+0000","last_peered":"2024-02-19T15:23:00.202216+0000","last_clean":"2024-02-19T15:23:00.202216+0000","last_became_active":"2024-02-19T15:22:58.214755+0000","last_became_peered":"2024-02-19T15:22:58.214755+0000","last_unstale":"2024-02-19T15:23:00.202216+0000","last_undegraded":"2024-02-19T15:23:00.202216+0000","last_fullsized":"2024-02-19T15:23:00.202216+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:49:11.624620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.661788+0000","last_change":"2024-02-19T15:22:59.221371+0000","last_active":"2024-02-19T15:23:00.661788+0000","last_peered":"2024-02-19T15:23:00.661788+0000","last_clean":"2024-02-19T15:23:00.661788+0000","last_became_active":"2024-02-19T15:22:59.218613+0000","last_became_peered":"2024-02-19T15:22:59.218613+0000","last_unstale":"2024-02-19T15:23:00.661788+0000","last_undegraded":"2024-02-19T15:23:00.661788+0000","last_fullsized":"2024-02-19T15:23:00.661788+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:23:35.768395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663117+0000","last_change":"2024-02-19T15:22:58.196152+0000","last_active":"2024-02-19T15:23:00.663117+0000","last_peered":"2024-02-19T15:23:00.663117+0000","last_clean":"2024-02-19T15:23:00.663117+0000","last_became_active":"2024-02-19T15:22:58.195852+0000","last_became_peered":"2024-02-19T15:22:58.195852+0000","last_unstale":"2024-02-19T15:23:00.663117+0000","last_undegraded":"2024-02-19T15:23:00.663117+0000","last_fullsized":"2024-02-19T15:23:00.663117+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:48:44.574187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662908+0000","last_change":"2024-02-19T15:22:59.211695+0000","last_active":"2024-02-19T15:23:00.662908+0000","last_peered":"2024-02-19T15:23:00.662908+0000","last_clean":"2024-02-19T15:23:00.662908+0000","last_became_active":"2024-02-19T15:22:59.211460+0000","last_became_peered":"2024-02-19T15:22:59.211460+0000","last_unstale":"2024-02-19T15:23:00.662908+0000","last_undegraded":"2024-02-19T15:23:00.662908+0000","last_fullsized":"2024-02-19T15:23:00.662908+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:40:32.141441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.661840+0000","last_change":"2024-02-19T15:22:58.210252+0000","last_active":"2024-02-19T15:23:00.661840+0000","last_peered":"2024-02-19T15:23:00.661840+0000","last_clean":"2024-02-19T15:23:00.661840+0000","last_became_active":"2024-02-19T15:22:58.209971+0000","last_became_peered":"2024-02-19T15:22:58.209971+0000","last_unstale":"2024-02-19T15:23:00.661840+0000","last_undegraded":"2024-02-19T15:23:00.661840+0000","last_fullsized":"2024-02-19T15:23:00.661840+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:13:33.144944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663685+0000","last_change":"2024-02-19T15:22:59.226673+0000","last_active":"2024-02-19T15:23:00.663685+0000","last_peered":"2024-02-19T15:23:00.663685+0000","last_clean":"2024-02-19T15:23:00.663685+0000","last_became_active":"2024-02-19T15:22:59.226367+0000","last_became_peered":"2024-02-19T15:22:59.226367+0000","last_unstale":"2024-02-19T15:23:00.663685+0000","last_undegraded":"2024-02-19T15:23:00.663685+0000","last_fullsized":"2024-02-19T15:23:00.663685+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:28:04.641517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201170+0000","last_change":"2024-02-19T15:22:58.202003+0000","last_active":"2024-02-19T15:23:00.201170+0000","last_peered":"2024-02-19T15:23:00.201170+0000","last_clean":"2024-02-19T15:23:00.201170+0000","last_became_active":"2024-02-19T15:22:58.201654+0000","last_became_peered":"2024-02-19T15:22:58.201654+0000","last_unstale":"2024-02-19T15:23:00.201170+0000","last_undegraded":"2024-02-19T15:23:00.201170+0000","last_fullsized":"2024-02-19T15:23:00.201170+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:53:52.672122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":193,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663080+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:23:00.663080+0000","last_peered":"2024-02-19T15:23:00.663080+0000","last_clean":"2024-02-19T15:23:00.663080+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:23:00.663080+0000","last_undegraded":"2024-02-19T15:23:00.663080+0000","last_fullsized":"2024-02-19T15:23:00.663080+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509457,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31560,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340280,"statfs":{"total":96636764160,"available":96604446720,"internally_reserved":0,"allocated":389120,"data_stored":150351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":4.6029999999999998,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.50800000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.442},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.71499999999999997}]},{"osd":2,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.53100000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.64800000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.66400000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.63200000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.70299999999999996}]}]},{"osd":7,"up_from":14,"seq":60129542161,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":966656,"data_stored":733893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.56100000000000005}]},{"osd":1,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":2.968,"5min":2.968,"15min":2.968},"last":0.72699999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":0.65500000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":3.1720000000000002,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":0.87}]},{"osd":4,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.77200000000000002}]},{"osd":5,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.81499999999999995}]},{"osd":6,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.88800000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542162,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":144107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.77600000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.84999999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.80200000000000005}]},{"osd":3,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.69899999999999995}]},{"osd":4,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":0.58599999999999997}]},{"osd":6,"last update":"Mon Feb 19 15:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.81799999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.60299999999999998}]}]},{"osd":4,"up_from":14,"seq":60129542160,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":149839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.66400000000000003}]},{"osd":1,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.61599999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.71299999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.60499999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.80100000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.68999999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.745},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.73099999999999998}]}]},{"osd":3,"up_from":14,"seq":60129542160,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31556,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":385024,"data_stored":145069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.2450000000000001,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.41199999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.313},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":4.0220000000000002,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.56200000000000006}]},{"osd":2,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.38100000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.625},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":4.2039999999999997,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.47899999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":4.1589999999999998,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.64100000000000001}]},{"osd":6,"last update":"Mon Feb 19 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":4.0940000000000003,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.52800000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":3.9660000000000002,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.69599999999999995}]}]},{"osd":2,"up_from":14,"seq":60129542161,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31560,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340280,"statfs":{"total":96636764160,"available":96604446720,"internally_reserved":0,"allocated":385024,"data_stored":149805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.7160000000000002,"5min":4.7160000000000002,"15min":4.7160000000000002},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":8.2129999999999992,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.377}]},{"osd":1,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":8.1620000000000008,"5min":8.1620000000000008,"15min":8.1620000000000008},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.53300000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":4.7679999999999998,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":4.9909999999999997,"5min":4.9909999999999997,"15min":4.9909999999999997},"last":0.502}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":4.3810000000000002,"5min":4.3810000000000002,"15min":4.3810000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":4.3440000000000003,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.626}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":4.4039999999999999,"5min":4.4039999999999999,"15min":4.4039999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":4.3259999999999996,"5min":4.3259999999999996,"15min":4.3259999999999996},"last":0.748}]},{"osd":6,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":4.5220000000000002,"5min":4.5220000000000002,"15min":4.5220000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":4.4320000000000004,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.60099999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":4.4829999999999997,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":4.4989999999999997,"5min":4.4989999999999997,"15min":4.4989999999999997},"last":0.57499999999999996}]}]},{"osd":1,"up_from":14,"seq":60129542162,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":397312,"data_stored":151369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.50800000000000001}]},{"osd":2,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.55800000000000005}]},{"osd":3,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.69999999999999996}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":1.127},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.92700000000000005}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.877}]},{"osd":6,"last update":"Mon Feb 19 15:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":1.0840000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":1.107},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":1.032}]}]},{"osd":0,"up_from":14,"seq":60129542160,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":974848,"data_stored":740103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.0199999999999996,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.313},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.36199999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":5.6669999999999998,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.4649999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.44700000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.9539999999999997,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.345,"5min":1.345,"15min":1.345},"last":0.47899999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":4.7880000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.85799999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.96099999999999997}]},{"osd":6,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":4.8460000000000001,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.83599999999999997}]},{"osd":7,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":4.9809999999999999,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":4.915,"5min":4.915,"15min":4.915},"last":0.91500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:23:03.424 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:23:03.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.596+0000 7faed6000640 1 Processor -- start 2024-02-19T15:23:03.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.596+0000 7faed6000640 1 -- start start 2024-02-19T15:23:03.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.596+0000 7faed6000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faed00cedb0 0x7faed00cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:03.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.596+0000 7faed6000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faed00cc700 con 0x7faed015bcd0 2024-02-19T15:23:03.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.596+0000 7faed6000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faed00cf300 con 0x7faed00d2710 2024-02-19T15:23:03.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.596+0000 7faed6000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed00cf5e0 con 0x7faed00cedb0 2024-02-19T15:23:03.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.596+0000 7faecf400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7faed015bcd0 0x7faed00cd480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42986/0 (socket says 172.21.15.47:42986) 2024-02-19T15:23:03.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.596+0000 7faecf400640 1 -- 172.21.15.47:0/1451491293 learned_addr learned my addr 172.21.15.47:0/1451491293 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:03.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.596+0000 7faec7e00640 1 --2- 172.21.15.47:0/1451491293 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faed00cedb0 0x7faed00cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:03.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.600+0000 7faecea00640 1 -- 172.21.15.47:0/1451491293 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540632229 0 0) 0x7faed00cc700 con 0x7faed015bcd0 2024-02-19T15:23:03.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.600+0000 7faecea00640 1 -- 172.21.15.47:0/1451491293 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faeb4003530 con 0x7faed015bcd0 2024-02-19T15:23:03.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.600+0000 7faecea00640 1 -- 172.21.15.47:0/1451491293 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729604378 0 0) 0x7faed00cf300 con 0x7faed00d2710 2024-02-19T15:23:03.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.600+0000 7faecea00640 1 -- 172.21.15.47:0/1451491293 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faed00cc700 con 0x7faed00d2710 2024-02-19T15:23:03.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.600+0000 7faec7e00640 1 -- 172.21.15.47:0/1451491293 >> v1:172.21.15.116:6789/0 conn(0x7faed00d2710 legacy=0x7faed015d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:03.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.600+0000 7faecea00640 1 -- 172.21.15.47:0/1451491293 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2738605992 0 0) 0x7faeb4003530 con 0x7faed015bcd0 2024-02-19T15:23:03.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.600+0000 7faec7e00640 1 -- 172.21.15.47:0/1451491293 >> v1:172.21.15.47:6789/0 conn(0x7faed015bcd0 legacy=0x7faed00cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:03.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.600+0000 7faec7e00640 1 -- 172.21.15.47:0/1451491293 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed00cc9a0 con 0x7faed00cedb0 2024-02-19T15:23:03.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.604+0000 7faec7e00640 1 --2- 172.21.15.47:0/1451491293 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faed00cedb0 0x7faed00cfc50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faeb80097f0 tx=0x7faeb801c3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a55fc919d19ae4b1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:03.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.604+0000 7faecea00640 1 -- 172.21.15.47:0/1451491293 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec0001680 con 0x7faed00cedb0 2024-02-19T15:23:03.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.604+0000 7faecea00640 1 -- 172.21.15.47:0/1451491293 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb80a7050 con 0x7faed00cedb0 2024-02-19T15:23:03.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.604+0000 7faecea00640 1 -- 172.21.15.47:0/1451491293 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb80ab600 con 0x7faed00cedb0 2024-02-19T15:23:03.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.604+0000 7faed6000640 1 -- 172.21.15.47:0/1451491293 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faed00cedb0 msgr2=0x7faed00cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:03.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.604+0000 7faed6000640 1 --2- 172.21.15.47:0/1451491293 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faed00cedb0 0x7faed00cfc50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faeb80097f0 tx=0x7faeb801c3d0 comp rx=0 tx=0).stop 2024-02-19T15:23:03.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.604+0000 7faed6000640 1 -- 172.21.15.47:0/1451491293 shutdown_connections 2024-02-19T15:23:03.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.604+0000 7faed6000640 1 --2- 172.21.15.47:0/1451491293 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faed00cedb0 0x7faed00cfc50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:03.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.604+0000 7faed6000640 1 -- 172.21.15.47:0/1451491293 >> 172.21.15.47:0/1451491293 conn(0x7faed00d1bf0 msgr2=0x7faed00d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:03.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.604+0000 7faed6000640 1 -- 172.21.15.47:0/1451491293 shutdown_connections 2024-02-19T15:23:03.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.604+0000 7faed6000640 1 -- 172.21.15.47:0/1451491293 wait complete. 2024-02-19T15:23:03.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faed6000640 1 Processor -- start 2024-02-19T15:23:03.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faed6000640 1 -- start start 2024-02-19T15:23:03.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faed6000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faed00cedb0 0x7faed00cb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:03.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faed6000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faed00d2710 0x7faed00b4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:03.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faed6000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faed015bcd0 0x7faed00c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:03.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faed6000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed00cdb90 con 0x7faed00d2710 2024-02-19T15:23:03.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faed6000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed00c84d0 con 0x7faed015bcd0 2024-02-19T15:23:03.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faed6000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed0158240 con 0x7faed00cedb0 2024-02-19T15:23:03.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faec7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faed00d2710 0x7faed00b4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:03.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faec7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faed00d2710 0x7faed00b4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48052/0 (socket says 172.21.15.47:48052) 2024-02-19T15:23:03.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faec7e00640 1 -- 172.21.15.47:0/1190773586 learned_addr learned my addr 172.21.15.47:0/1190773586 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:03.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faecfe00640 1 --2- 172.21.15.47:0/1190773586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faed015bcd0 0x7faed00c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:03.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faecf400640 1 --2- 172.21.15.47:0/1190773586 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faed00cedb0 0x7faed00cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:03.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faec7e00640 1 -- 172.21.15.47:0/1190773586 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faed00cedb0 msgr2=0x7faed00cb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:03.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faec7e00640 1 --2- 172.21.15.47:0/1190773586 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faed00cedb0 0x7faed00cb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:03.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faec7e00640 1 -- 172.21.15.47:0/1190773586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faed015bcd0 msgr2=0x7faed00c7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:03.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faec7e00640 1 --2- 172.21.15.47:0/1190773586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faed015bcd0 0x7faed00c7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:03.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faec7e00640 1 -- 172.21.15.47:0/1190773586 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb8009420 con 0x7faed00d2710 2024-02-19T15:23:03.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faec7e00640 1 --2- 172.21.15.47:0/1190773586 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faed00d2710 0x7faed00b4ae0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7faeb80ae040 tx=0x7faeb8019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:03.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faec7400640 1 -- 172.21.15.47:0/1190773586 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb80b8070 con 0x7faed00d2710 2024-02-19T15:23:03.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faec7400640 1 -- 172.21.15.47:0/1190773586 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb8003830 con 0x7faed00d2710 2024-02-19T15:23:03.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faed6000640 1 -- 172.21.15.47:0/1190773586 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed015d240 con 0x7faed00d2710 2024-02-19T15:23:03.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faec7400640 1 -- 172.21.15.47:0/1190773586 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb8003ba0 con 0x7faed00d2710 2024-02-19T15:23:03.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faed6000640 1 -- 172.21.15.47:0/1190773586 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed0158790 con 0x7faed00d2710 2024-02-19T15:23:03.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.608+0000 7faed6000640 1 -- 172.21.15.47:0/1190773586 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed0075360 con 0x7faed00d2710 2024-02-19T15:23:03.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.612+0000 7faec7400640 1 -- 172.21.15.47:0/1190773586 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7faeb80b2040 con 0x7faed00d2710 2024-02-19T15:23:03.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.612+0000 7faec7400640 1 --2- 172.21.15.47:0/1190773586 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faeac066e40 0x7faeac069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:03.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.612+0000 7faec7400640 1 -- 172.21.15.47:0/1190773586 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faeb811dfe0 con 0x7faed00d2710 2024-02-19T15:23:03.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.612+0000 7faecf400640 1 --2- 172.21.15.47:0/1190773586 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faeac066e40 0x7faeac069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:03.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.612+0000 7faecf400640 1 --2- 172.21.15.47:0/1190773586 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faeac066e40 0x7faeac069300 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faec0000fe0 tx=0x7faec0001680 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:03.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.616+0000 7faec7400640 1 -- 172.21.15.47:0/1190773586 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7faeb80eade0 con 0x7faed00d2710 2024-02-19T15:23:03.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.776+0000 7faed6000640 1 -- 172.21.15.47:0/1190773586 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faed015d0c0 con 0x7faeac066e40 2024-02-19T15:23:03.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.784+0000 7faec7400640 1 -- 172.21.15.47:0/1190773586 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289331 (secure 0 0 0) 0x7faec000f9a0 con 0x7faeac066e40 2024-02-19T15:23:03.791 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:03.793 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:23:03.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.788+0000 7faed6000640 1 -- 172.21.15.47:0/1190773586 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faeac066e40 msgr2=0x7faeac069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:03.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.788+0000 7faed6000640 1 --2- 172.21.15.47:0/1190773586 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faeac066e40 0x7faeac069300 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faec0000fe0 tx=0x7faec0001680 comp rx=0 tx=0).stop 2024-02-19T15:23:03.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.788+0000 7faed6000640 1 -- 172.21.15.47:0/1190773586 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faed00d2710 msgr2=0x7faed00b4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:03.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.788+0000 7faed6000640 1 --2- 172.21.15.47:0/1190773586 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faed00d2710 0x7faed00b4ae0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7faeb80ae040 tx=0x7faeb8019170 comp rx=0 tx=0).stop 2024-02-19T15:23:03.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.788+0000 7faed6000640 1 -- 172.21.15.47:0/1190773586 shutdown_connections 2024-02-19T15:23:03.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.788+0000 7faed6000640 1 --2- 172.21.15.47:0/1190773586 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faeac066e40 0x7faeac069300 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:03.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.788+0000 7faed6000640 1 --2- 172.21.15.47:0/1190773586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faed015bcd0 0x7faed00c7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:03.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.788+0000 7faed6000640 1 --2- 172.21.15.47:0/1190773586 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faed00d2710 0x7faed00b4ae0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:03.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.788+0000 7faed6000640 1 --2- 172.21.15.47:0/1190773586 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faed00cedb0 0x7faed00cb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:03.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.788+0000 7faed6000640 1 -- 172.21.15.47:0/1190773586 >> 172.21.15.47:0/1190773586 conn(0x7faed00d1bf0 msgr2=0x7faed00ce150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:03.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.788+0000 7faed6000640 1 -- 172.21.15.47:0/1190773586 shutdown_connections 2024-02-19T15:23:03.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:03.788+0000 7faed6000640 1 -- 172.21.15.47:0/1190773586 wait complete. 2024-02-19T15:23:03.811 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-02-19T15:23:03.161177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":102,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":241140,"kb_used_data":4164,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754733580,"statfs":{"total":773094113280,"available":772847185920,"internally_reserved":0,"allocated":4263936,"data_stored":2364536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":242535986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.980496"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201483+0000","last_change":"2024-02-19T15:22:59.212722+0000","last_active":"2024-02-19T15:23:00.201483+0000","last_peered":"2024-02-19T15:23:00.201483+0000","last_clean":"2024-02-19T15:23:00.201483+0000","last_became_active":"2024-02-19T15:22:59.212414+0000","last_became_peered":"2024-02-19T15:22:59.212414+0000","last_unstale":"2024-02-19T15:23:00.201483+0000","last_undegraded":"2024-02-19T15:23:00.201483+0000","last_fullsized":"2024-02-19T15:23:00.201483+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:54:31.030957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662311+0000","last_change":"2024-02-19T15:22:59.209934+0000","last_active":"2024-02-19T15:23:00.662311+0000","last_peered":"2024-02-19T15:23:00.662311+0000","last_clean":"2024-02-19T15:23:00.662311+0000","last_became_active":"2024-02-19T15:22:59.209661+0000","last_became_peered":"2024-02-19T15:22:59.209661+0000","last_unstale":"2024-02-19T15:23:00.662311+0000","last_undegraded":"2024-02-19T15:23:00.662311+0000","last_fullsized":"2024-02-19T15:23:00.662311+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:54:08.451277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663450+0000","last_change":"2024-02-19T15:22:59.223748+0000","last_active":"2024-02-19T15:23:00.663450+0000","last_peered":"2024-02-19T15:23:00.663450+0000","last_clean":"2024-02-19T15:23:00.663450+0000","last_became_active":"2024-02-19T15:22:59.223178+0000","last_became_peered":"2024-02-19T15:22:59.223178+0000","last_unstale":"2024-02-19T15:23:00.663450+0000","last_undegraded":"2024-02-19T15:23:00.663450+0000","last_fullsized":"2024-02-19T15:23:00.663450+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:08.692279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201889+0000","last_change":"2024-02-19T15:22:59.234029+0000","last_active":"2024-02-19T15:23:00.201889+0000","last_peered":"2024-02-19T15:23:00.201889+0000","last_clean":"2024-02-19T15:23:00.201889+0000","last_became_active":"2024-02-19T15:22:59.233761+0000","last_became_peered":"2024-02-19T15:22:59.233761+0000","last_unstale":"2024-02-19T15:23:00.201889+0000","last_undegraded":"2024-02-19T15:23:00.201889+0000","last_fullsized":"2024-02-19T15:23:00.201889+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:21:24.942760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.200768+0000","last_change":"2024-02-19T15:22:59.226653+0000","last_active":"2024-02-19T15:23:00.200768+0000","last_peered":"2024-02-19T15:23:00.200768+0000","last_clean":"2024-02-19T15:23:00.200768+0000","last_became_active":"2024-02-19T15:22:59.226387+0000","last_became_peered":"2024-02-19T15:22:59.226387+0000","last_unstale":"2024-02-19T15:23:00.200768+0000","last_undegraded":"2024-02-19T15:23:00.200768+0000","last_fullsized":"2024-02-19T15:23:00.200768+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:00:08.809218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.661839+0000","last_change":"2024-02-19T15:22:59.216353+0000","last_active":"2024-02-19T15:23:00.661839+0000","last_peered":"2024-02-19T15:23:00.661839+0000","last_clean":"2024-02-19T15:23:00.661839+0000","last_became_active":"2024-02-19T15:22:59.216050+0000","last_became_peered":"2024-02-19T15:22:59.216050+0000","last_unstale":"2024-02-19T15:23:00.661839+0000","last_undegraded":"2024-02-19T15:23:00.661839+0000","last_fullsized":"2024-02-19T15:23:00.661839+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:46.146658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664160+0000","last_change":"2024-02-19T15:22:59.223166+0000","last_active":"2024-02-19T15:23:00.664160+0000","last_peered":"2024-02-19T15:23:00.664160+0000","last_clean":"2024-02-19T15:23:00.664160+0000","last_became_active":"2024-02-19T15:22:59.222718+0000","last_became_peered":"2024-02-19T15:22:59.222718+0000","last_unstale":"2024-02-19T15:23:00.664160+0000","last_undegraded":"2024-02-19T15:23:00.664160+0000","last_fullsized":"2024-02-19T15:23:00.664160+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:25:08.609489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662633+0000","last_change":"2024-02-19T15:22:59.228518+0000","last_active":"2024-02-19T15:23:00.662633+0000","last_peered":"2024-02-19T15:23:00.662633+0000","last_clean":"2024-02-19T15:23:00.662633+0000","last_became_active":"2024-02-19T15:22:59.227787+0000","last_became_peered":"2024-02-19T15:22:59.227787+0000","last_unstale":"2024-02-19T15:23:00.662633+0000","last_undegraded":"2024-02-19T15:23:00.662633+0000","last_fullsized":"2024-02-19T15:23:00.662633+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:05:30.246561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664709+0000","last_change":"2024-02-19T15:22:59.222925+0000","last_active":"2024-02-19T15:23:00.664709+0000","last_peered":"2024-02-19T15:23:00.664709+0000","last_clean":"2024-02-19T15:23:00.664709+0000","last_became_active":"2024-02-19T15:22:59.222493+0000","last_became_peered":"2024-02-19T15:22:59.222493+0000","last_unstale":"2024-02-19T15:23:00.664709+0000","last_undegraded":"2024-02-19T15:23:00.664709+0000","last_fullsized":"2024-02-19T15:23:00.664709+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:48:02.181707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.202102+0000","last_change":"2024-02-19T15:22:59.230647+0000","last_active":"2024-02-19T15:23:00.202102+0000","last_peered":"2024-02-19T15:23:00.202102+0000","last_clean":"2024-02-19T15:23:00.202102+0000","last_became_active":"2024-02-19T15:22:59.229997+0000","last_became_peered":"2024-02-19T15:22:59.229997+0000","last_unstale":"2024-02-19T15:23:00.202102+0000","last_undegraded":"2024-02-19T15:23:00.202102+0000","last_fullsized":"2024-02-19T15:23:00.202102+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:06:58.745577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663898+0000","last_change":"2024-02-19T15:22:59.228636+0000","last_active":"2024-02-19T15:23:00.663898+0000","last_peered":"2024-02-19T15:23:00.663898+0000","last_clean":"2024-02-19T15:23:00.663898+0000","last_became_active":"2024-02-19T15:22:59.227852+0000","last_became_peered":"2024-02-19T15:22:59.227852+0000","last_unstale":"2024-02-19T15:23:00.663898+0000","last_undegraded":"2024-02-19T15:23:00.663898+0000","last_fullsized":"2024-02-19T15:23:00.663898+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:12:52.929727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662959+0000","last_change":"2024-02-19T15:22:59.221574+0000","last_active":"2024-02-19T15:23:00.662959+0000","last_peered":"2024-02-19T15:23:00.662959+0000","last_clean":"2024-02-19T15:23:00.662959+0000","last_became_active":"2024-02-19T15:22:59.221223+0000","last_became_peered":"2024-02-19T15:22:59.221223+0000","last_unstale":"2024-02-19T15:23:00.662959+0000","last_undegraded":"2024-02-19T15:23:00.662959+0000","last_fullsized":"2024-02-19T15:23:00.662959+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:35:59.446260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.200424+0000","last_change":"2024-02-19T15:22:59.215552+0000","last_active":"2024-02-19T15:23:00.200424+0000","last_peered":"2024-02-19T15:23:00.200424+0000","last_clean":"2024-02-19T15:23:00.200424+0000","last_became_active":"2024-02-19T15:22:59.215185+0000","last_became_peered":"2024-02-19T15:22:59.215185+0000","last_unstale":"2024-02-19T15:23:00.200424+0000","last_undegraded":"2024-02-19T15:23:00.200424+0000","last_fullsized":"2024-02-19T15:23:00.200424+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:41:07.184621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.661979+0000","last_change":"2024-02-19T15:22:59.209918+0000","last_active":"2024-02-19T15:23:00.661979+0000","last_peered":"2024-02-19T15:23:00.661979+0000","last_clean":"2024-02-19T15:23:00.661979+0000","last_became_active":"2024-02-19T15:22:59.209558+0000","last_became_peered":"2024-02-19T15:22:59.209558+0000","last_unstale":"2024-02-19T15:23:00.661979+0000","last_undegraded":"2024-02-19T15:23:00.661979+0000","last_fullsized":"2024-02-19T15:23:00.661979+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:35:59.446260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662383+0000","last_change":"2024-02-19T15:22:59.213739+0000","last_active":"2024-02-19T15:23:00.662383+0000","last_peered":"2024-02-19T15:23:00.662383+0000","last_clean":"2024-02-19T15:23:00.662383+0000","last_became_active":"2024-02-19T15:22:59.213370+0000","last_became_peered":"2024-02-19T15:22:59.213370+0000","last_unstale":"2024-02-19T15:23:00.662383+0000","last_undegraded":"2024-02-19T15:23:00.662383+0000","last_fullsized":"2024-02-19T15:23:00.662383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:46.146658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663044+0000","last_change":"2024-02-19T15:22:59.228970+0000","last_active":"2024-02-19T15:23:00.663044+0000","last_peered":"2024-02-19T15:23:00.663044+0000","last_clean":"2024-02-19T15:23:00.663044+0000","last_became_active":"2024-02-19T15:22:59.228152+0000","last_became_peered":"2024-02-19T15:22:59.228152+0000","last_unstale":"2024-02-19T15:23:00.663044+0000","last_undegraded":"2024-02-19T15:23:00.663044+0000","last_fullsized":"2024-02-19T15:23:00.663044+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:42.769874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664244+0000","last_change":"2024-02-19T15:22:59.226022+0000","last_active":"2024-02-19T15:23:00.664244+0000","last_peered":"2024-02-19T15:23:00.664244+0000","last_clean":"2024-02-19T15:23:00.664244+0000","last_became_active":"2024-02-19T15:22:59.225628+0000","last_became_peered":"2024-02-19T15:22:59.225628+0000","last_unstale":"2024-02-19T15:23:00.664244+0000","last_undegraded":"2024-02-19T15:23:00.664244+0000","last_fullsized":"2024-02-19T15:23:00.664244+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:05:22.259533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662360+0000","last_change":"2024-02-19T15:22:59.217500+0000","last_active":"2024-02-19T15:23:00.662360+0000","last_peered":"2024-02-19T15:23:00.662360+0000","last_clean":"2024-02-19T15:23:00.662360+0000","last_became_active":"2024-02-19T15:22:59.217191+0000","last_became_peered":"2024-02-19T15:22:59.217191+0000","last_unstale":"2024-02-19T15:23:00.662360+0000","last_undegraded":"2024-02-19T15:23:00.662360+0000","last_fullsized":"2024-02-19T15:23:00.662360+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:42.769874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662032+0000","last_change":"2024-02-19T15:22:59.220277+0000","last_active":"2024-02-19T15:23:00.662032+0000","last_peered":"2024-02-19T15:23:00.662032+0000","last_clean":"2024-02-19T15:23:00.662032+0000","last_became_active":"2024-02-19T15:22:59.219964+0000","last_became_peered":"2024-02-19T15:22:59.219964+0000","last_unstale":"2024-02-19T15:23:00.662032+0000","last_undegraded":"2024-02-19T15:23:00.662032+0000","last_fullsized":"2024-02-19T15:23:00.662032+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:05:30.246561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.200694+0000","last_change":"2024-02-19T15:22:59.213027+0000","last_active":"2024-02-19T15:23:00.200694+0000","last_peered":"2024-02-19T15:23:00.200694+0000","last_clean":"2024-02-19T15:23:00.200694+0000","last_became_active":"2024-02-19T15:22:59.212762+0000","last_became_peered":"2024-02-19T15:22:59.212762+0000","last_unstale":"2024-02-19T15:23:00.200694+0000","last_undegraded":"2024-02-19T15:23:00.200694+0000","last_fullsized":"2024-02-19T15:23:00.200694+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:54.369825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663713+0000","last_change":"2024-02-19T15:22:59.209802+0000","last_active":"2024-02-19T15:23:00.663713+0000","last_peered":"2024-02-19T15:23:00.663713+0000","last_clean":"2024-02-19T15:23:00.663713+0000","last_became_active":"2024-02-19T15:22:59.209421+0000","last_became_peered":"2024-02-19T15:22:59.209421+0000","last_unstale":"2024-02-19T15:23:00.663713+0000","last_undegraded":"2024-02-19T15:23:00.663713+0000","last_fullsized":"2024-02-19T15:23:00.663713+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:41.612341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663929+0000","last_change":"2024-02-19T15:22:59.222070+0000","last_active":"2024-02-19T15:23:00.663929+0000","last_peered":"2024-02-19T15:23:00.663929+0000","last_clean":"2024-02-19T15:23:00.663929+0000","last_became_active":"2024-02-19T15:22:59.221680+0000","last_became_peered":"2024-02-19T15:22:59.221680+0000","last_unstale":"2024-02-19T15:23:00.663929+0000","last_undegraded":"2024-02-19T15:23:00.663929+0000","last_fullsized":"2024-02-19T15:23:00.663929+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:06:09.543916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.226469+0000","last_change":"2024-02-19T15:22:59.224876+0000","last_active":"2024-02-19T15:23:00.226469+0000","last_peered":"2024-02-19T15:23:00.226469+0000","last_clean":"2024-02-19T15:23:00.226469+0000","last_became_active":"2024-02-19T15:22:59.224559+0000","last_became_peered":"2024-02-19T15:22:59.224559+0000","last_unstale":"2024-02-19T15:23:00.226469+0000","last_undegraded":"2024-02-19T15:23:00.226469+0000","last_fullsized":"2024-02-19T15:23:00.226469+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:42:37.450463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662291+0000","last_change":"2024-02-19T15:22:59.209467+0000","last_active":"2024-02-19T15:23:00.662291+0000","last_peered":"2024-02-19T15:23:00.662291+0000","last_clean":"2024-02-19T15:23:00.662291+0000","last_became_active":"2024-02-19T15:22:59.209175+0000","last_became_peered":"2024-02-19T15:22:59.209175+0000","last_unstale":"2024-02-19T15:23:00.662291+0000","last_undegraded":"2024-02-19T15:23:00.662291+0000","last_fullsized":"2024-02-19T15:23:00.662291+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:23:35.768395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.202330+0000","last_change":"2024-02-19T15:22:59.232815+0000","last_active":"2024-02-19T15:23:00.202330+0000","last_peered":"2024-02-19T15:23:00.202330+0000","last_clean":"2024-02-19T15:23:00.202330+0000","last_became_active":"2024-02-19T15:22:59.232553+0000","last_became_peered":"2024-02-19T15:22:59.232553+0000","last_unstale":"2024-02-19T15:23:00.202330+0000","last_undegraded":"2024-02-19T15:23:00.202330+0000","last_fullsized":"2024-02-19T15:23:00.202330+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:46.146658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662466+0000","last_change":"2024-02-19T15:22:59.233785+0000","last_active":"2024-02-19T15:23:00.662466+0000","last_peered":"2024-02-19T15:23:00.662466+0000","last_clean":"2024-02-19T15:23:00.662466+0000","last_became_active":"2024-02-19T15:22:59.233468+0000","last_became_peered":"2024-02-19T15:22:59.233468+0000","last_unstale":"2024-02-19T15:23:00.662466+0000","last_undegraded":"2024-02-19T15:23:00.662466+0000","last_fullsized":"2024-02-19T15:23:00.662466+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:53.399488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.202138+0000","last_change":"2024-02-19T15:22:59.234009+0000","last_active":"2024-02-19T15:23:00.202138+0000","last_peered":"2024-02-19T15:23:00.202138+0000","last_clean":"2024-02-19T15:23:00.202138+0000","last_became_active":"2024-02-19T15:22:59.233727+0000","last_became_peered":"2024-02-19T15:22:59.233727+0000","last_unstale":"2024-02-19T15:23:00.202138+0000","last_undegraded":"2024-02-19T15:23:00.202138+0000","last_fullsized":"2024-02-19T15:23:00.202138+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:27:04.060833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662972+0000","last_change":"2024-02-19T15:22:59.217810+0000","last_active":"2024-02-19T15:23:00.662972+0000","last_peered":"2024-02-19T15:23:00.662972+0000","last_clean":"2024-02-19T15:23:00.662972+0000","last_became_active":"2024-02-19T15:22:59.217362+0000","last_became_peered":"2024-02-19T15:22:59.217362+0000","last_unstale":"2024-02-19T15:23:00.662972+0000","last_undegraded":"2024-02-19T15:23:00.662972+0000","last_fullsized":"2024-02-19T15:23:00.662972+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:05:30.246561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663872+0000","last_change":"2024-02-19T15:22:59.222425+0000","last_active":"2024-02-19T15:23:00.663872+0000","last_peered":"2024-02-19T15:23:00.663872+0000","last_clean":"2024-02-19T15:23:00.663872+0000","last_became_active":"2024-02-19T15:22:59.221865+0000","last_became_peered":"2024-02-19T15:22:59.221865+0000","last_unstale":"2024-02-19T15:23:00.663872+0000","last_undegraded":"2024-02-19T15:23:00.663872+0000","last_fullsized":"2024-02-19T15:23:00.663872+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:01:10.613097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663858+0000","last_change":"2024-02-19T15:22:59.225668+0000","last_active":"2024-02-19T15:23:00.663858+0000","last_peered":"2024-02-19T15:23:00.663858+0000","last_clean":"2024-02-19T15:23:00.663858+0000","last_became_active":"2024-02-19T15:22:59.225322+0000","last_became_peered":"2024-02-19T15:22:59.225322+0000","last_unstale":"2024-02-19T15:23:00.663858+0000","last_undegraded":"2024-02-19T15:23:00.663858+0000","last_fullsized":"2024-02-19T15:23:00.663858+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:39:07.808039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201689+0000","last_change":"2024-02-19T15:22:59.218650+0000","last_active":"2024-02-19T15:23:00.201689+0000","last_peered":"2024-02-19T15:23:00.201689+0000","last_clean":"2024-02-19T15:23:00.201689+0000","last_became_active":"2024-02-19T15:22:59.218254+0000","last_became_peered":"2024-02-19T15:22:59.218254+0000","last_unstale":"2024-02-19T15:23:00.201689+0000","last_undegraded":"2024-02-19T15:23:00.201689+0000","last_fullsized":"2024-02-19T15:23:00.201689+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:04:26.085669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662736+0000","last_change":"2024-02-19T15:22:59.232336+0000","last_active":"2024-02-19T15:23:00.662736+0000","last_peered":"2024-02-19T15:23:00.662736+0000","last_clean":"2024-02-19T15:23:00.662736+0000","last_became_active":"2024-02-19T15:22:59.231664+0000","last_became_peered":"2024-02-19T15:22:59.231664+0000","last_unstale":"2024-02-19T15:23:00.662736+0000","last_undegraded":"2024-02-19T15:23:00.662736+0000","last_fullsized":"2024-02-19T15:23:00.662736+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:52:10.521631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.226696+0000","last_change":"2024-02-19T15:22:59.222179+0000","last_active":"2024-02-19T15:23:00.226696+0000","last_peered":"2024-02-19T15:23:00.226696+0000","last_clean":"2024-02-19T15:23:00.226696+0000","last_became_active":"2024-02-19T15:22:59.221907+0000","last_became_peered":"2024-02-19T15:22:59.221907+0000","last_unstale":"2024-02-19T15:23:00.226696+0000","last_undegraded":"2024-02-19T15:23:00.226696+0000","last_fullsized":"2024-02-19T15:23:00.226696+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:23:51.466880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201537+0000","last_change":"2024-02-19T15:22:58.202088+0000","last_active":"2024-02-19T15:23:00.201537+0000","last_peered":"2024-02-19T15:23:00.201537+0000","last_clean":"2024-02-19T15:23:00.201537+0000","last_became_active":"2024-02-19T15:22:58.201781+0000","last_became_peered":"2024-02-19T15:22:58.201781+0000","last_unstale":"2024-02-19T15:23:00.201537+0000","last_undegraded":"2024-02-19T15:23:00.201537+0000","last_fullsized":"2024-02-19T15:23:00.201537+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:01:28.252912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201510+0000","last_change":"2024-02-19T15:22:59.220518+0000","last_active":"2024-02-19T15:23:00.201510+0000","last_peered":"2024-02-19T15:23:00.201510+0000","last_clean":"2024-02-19T15:23:00.201510+0000","last_became_active":"2024-02-19T15:22:59.220215+0000","last_became_peered":"2024-02-19T15:22:59.220215+0000","last_unstale":"2024-02-19T15:23:00.201510+0000","last_undegraded":"2024-02-19T15:23:00.201510+0000","last_fullsized":"2024-02-19T15:23:00.201510+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:51:49.244632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.226747+0000","last_change":"2024-02-19T15:22:58.212160+0000","last_active":"2024-02-19T15:23:00.226747+0000","last_peered":"2024-02-19T15:23:00.226747+0000","last_clean":"2024-02-19T15:23:00.226747+0000","last_became_active":"2024-02-19T15:22:58.211748+0000","last_became_peered":"2024-02-19T15:22:58.211748+0000","last_unstale":"2024-02-19T15:23:00.226747+0000","last_undegraded":"2024-02-19T15:23:00.226747+0000","last_fullsized":"2024-02-19T15:23:00.226747+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:48:14.739039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664821+0000","last_change":"2024-02-19T15:22:59.219546+0000","last_active":"2024-02-19T15:23:00.664821+0000","last_peered":"2024-02-19T15:23:00.664821+0000","last_clean":"2024-02-19T15:23:00.664821+0000","last_became_active":"2024-02-19T15:22:59.219303+0000","last_became_peered":"2024-02-19T15:22:59.219303+0000","last_unstale":"2024-02-19T15:23:00.664821+0000","last_undegraded":"2024-02-19T15:23:00.664821+0000","last_fullsized":"2024-02-19T15:23:00.664821+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:30:20.556027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662098+0000","last_change":"2024-02-19T15:22:58.210229+0000","last_active":"2024-02-19T15:23:00.662098+0000","last_peered":"2024-02-19T15:23:00.662098+0000","last_clean":"2024-02-19T15:23:00.662098+0000","last_became_active":"2024-02-19T15:22:58.209101+0000","last_became_peered":"2024-02-19T15:22:58.209101+0000","last_unstale":"2024-02-19T15:23:00.662098+0000","last_undegraded":"2024-02-19T15:23:00.662098+0000","last_fullsized":"2024-02-19T15:23:00.662098+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:32:40.017025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.202261+0000","last_change":"2024-02-19T15:22:59.233156+0000","last_active":"2024-02-19T15:23:00.202261+0000","last_peered":"2024-02-19T15:23:00.202261+0000","last_clean":"2024-02-19T15:23:00.202261+0000","last_became_active":"2024-02-19T15:22:59.232879+0000","last_became_peered":"2024-02-19T15:22:59.232879+0000","last_unstale":"2024-02-19T15:23:00.202261+0000","last_undegraded":"2024-02-19T15:23:00.202261+0000","last_fullsized":"2024-02-19T15:23:00.202261+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:23:35.768395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.200800+0000","last_change":"2024-02-19T15:22:58.207825+0000","last_active":"2024-02-19T15:23:00.200800+0000","last_peered":"2024-02-19T15:23:00.200800+0000","last_clean":"2024-02-19T15:23:00.200800+0000","last_became_active":"2024-02-19T15:22:58.207526+0000","last_became_peered":"2024-02-19T15:22:58.207526+0000","last_unstale":"2024-02-19T15:23:00.200800+0000","last_undegraded":"2024-02-19T15:23:00.200800+0000","last_fullsized":"2024-02-19T15:23:00.200800+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:47:08.185258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663719+0000","last_change":"2024-02-19T15:22:59.233487+0000","last_active":"2024-02-19T15:23:00.663719+0000","last_peered":"2024-02-19T15:23:00.663719+0000","last_clean":"2024-02-19T15:23:00.663719+0000","last_became_active":"2024-02-19T15:22:59.233142+0000","last_became_peered":"2024-02-19T15:22:59.233142+0000","last_unstale":"2024-02-19T15:23:00.663719+0000","last_undegraded":"2024-02-19T15:23:00.663719+0000","last_fullsized":"2024-02-19T15:23:00.663719+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:53.399488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664252+0000","last_change":"2024-02-19T15:22:58.204949+0000","last_active":"2024-02-19T15:23:00.664252+0000","last_peered":"2024-02-19T15:23:00.664252+0000","last_clean":"2024-02-19T15:23:00.664252+0000","last_became_active":"2024-02-19T15:22:58.204693+0000","last_became_peered":"2024-02-19T15:22:58.204693+0000","last_unstale":"2024-02-19T15:23:00.664252+0000","last_undegraded":"2024-02-19T15:23:00.664252+0000","last_fullsized":"2024-02-19T15:23:00.664252+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:31:52.558573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663098+0000","last_change":"2024-02-19T15:22:59.210037+0000","last_active":"2024-02-19T15:23:00.663098+0000","last_peered":"2024-02-19T15:23:00.663098+0000","last_clean":"2024-02-19T15:23:00.663098+0000","last_became_active":"2024-02-19T15:22:59.209168+0000","last_became_peered":"2024-02-19T15:22:59.209168+0000","last_unstale":"2024-02-19T15:23:00.663098+0000","last_undegraded":"2024-02-19T15:23:00.663098+0000","last_fullsized":"2024-02-19T15:23:00.663098+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:52:10.521631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664063+0000","last_change":"2024-02-19T15:22:58.207343+0000","last_active":"2024-02-19T15:23:00.664063+0000","last_peered":"2024-02-19T15:23:00.664063+0000","last_clean":"2024-02-19T15:23:00.664063+0000","last_became_active":"2024-02-19T15:22:58.207010+0000","last_became_peered":"2024-02-19T15:22:58.207010+0000","last_unstale":"2024-02-19T15:23:00.664063+0000","last_undegraded":"2024-02-19T15:23:00.664063+0000","last_fullsized":"2024-02-19T15:23:00.664063+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:53.363347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663291+0000","last_change":"2024-02-19T15:22:59.210486+0000","last_active":"2024-02-19T15:23:00.663291+0000","last_peered":"2024-02-19T15:23:00.663291+0000","last_clean":"2024-02-19T15:23:00.663291+0000","last_became_active":"2024-02-19T15:22:59.210163+0000","last_became_peered":"2024-02-19T15:22:59.210163+0000","last_unstale":"2024-02-19T15:23:00.663291+0000","last_undegraded":"2024-02-19T15:23:00.663291+0000","last_fullsized":"2024-02-19T15:23:00.663291+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:13:34.151422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.661850+0000","last_change":"2024-02-19T15:22:58.203400+0000","last_active":"2024-02-19T15:23:00.661850+0000","last_peered":"2024-02-19T15:23:00.661850+0000","last_clean":"2024-02-19T15:23:00.661850+0000","last_became_active":"2024-02-19T15:22:58.203078+0000","last_became_peered":"2024-02-19T15:22:58.203078+0000","last_unstale":"2024-02-19T15:23:00.661850+0000","last_undegraded":"2024-02-19T15:23:00.661850+0000","last_fullsized":"2024-02-19T15:23:00.661850+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:48.695445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201704+0000","last_change":"2024-02-19T15:22:59.224076+0000","last_active":"2024-02-19T15:23:00.201704+0000","last_peered":"2024-02-19T15:23:00.201704+0000","last_clean":"2024-02-19T15:23:00.201704+0000","last_became_active":"2024-02-19T15:22:59.223783+0000","last_became_peered":"2024-02-19T15:22:59.223783+0000","last_unstale":"2024-02-19T15:23:00.201704+0000","last_undegraded":"2024-02-19T15:23:00.201704+0000","last_fullsized":"2024-02-19T15:23:00.201704+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:39:07.966403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662823+0000","last_change":"2024-02-19T15:22:58.197186+0000","last_active":"2024-02-19T15:23:00.662823+0000","last_peered":"2024-02-19T15:23:00.662823+0000","last_clean":"2024-02-19T15:23:00.662823+0000","last_became_active":"2024-02-19T15:22:58.196405+0000","last_became_peered":"2024-02-19T15:22:58.196405+0000","last_unstale":"2024-02-19T15:23:00.662823+0000","last_undegraded":"2024-02-19T15:23:00.662823+0000","last_fullsized":"2024-02-19T15:23:00.662823+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:42:21.548099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663211+0000","last_change":"2024-02-19T15:22:59.228875+0000","last_active":"2024-02-19T15:23:00.663211+0000","last_peered":"2024-02-19T15:23:00.663211+0000","last_clean":"2024-02-19T15:23:00.663211+0000","last_became_active":"2024-02-19T15:22:59.228548+0000","last_became_peered":"2024-02-19T15:22:59.228548+0000","last_unstale":"2024-02-19T15:23:00.663211+0000","last_undegraded":"2024-02-19T15:23:00.663211+0000","last_fullsized":"2024-02-19T15:23:00.663211+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:54:08.451277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662414+0000","last_change":"2024-02-19T15:22:58.209590+0000","last_active":"2024-02-19T15:23:00.662414+0000","last_peered":"2024-02-19T15:23:00.662414+0000","last_clean":"2024-02-19T15:23:00.662414+0000","last_became_active":"2024-02-19T15:22:58.209253+0000","last_became_peered":"2024-02-19T15:22:58.209253+0000","last_unstale":"2024-02-19T15:23:00.662414+0000","last_undegraded":"2024-02-19T15:23:00.662414+0000","last_fullsized":"2024-02-19T15:23:00.662414+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:40.605863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662430+0000","last_change":"2024-02-19T15:22:59.219447+0000","last_active":"2024-02-19T15:23:00.662430+0000","last_peered":"2024-02-19T15:23:00.662430+0000","last_clean":"2024-02-19T15:23:00.662430+0000","last_became_active":"2024-02-19T15:22:59.219172+0000","last_became_peered":"2024-02-19T15:22:59.219172+0000","last_unstale":"2024-02-19T15:23:00.662430+0000","last_undegraded":"2024-02-19T15:23:00.662430+0000","last_fullsized":"2024-02-19T15:23:00.662430+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:28:04.641517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.226864+0000","last_change":"2024-02-19T15:22:58.209938+0000","last_active":"2024-02-19T15:23:00.226864+0000","last_peered":"2024-02-19T15:23:00.226864+0000","last_clean":"2024-02-19T15:23:00.226864+0000","last_became_active":"2024-02-19T15:22:58.209429+0000","last_became_peered":"2024-02-19T15:22:58.209429+0000","last_unstale":"2024-02-19T15:23:00.226864+0000","last_undegraded":"2024-02-19T15:23:00.226864+0000","last_fullsized":"2024-02-19T15:23:00.226864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:54:21.839011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662382+0000","last_change":"2024-02-19T15:22:59.228204+0000","last_active":"2024-02-19T15:23:00.662382+0000","last_peered":"2024-02-19T15:23:00.662382+0000","last_clean":"2024-02-19T15:23:00.662382+0000","last_became_active":"2024-02-19T15:22:59.227900+0000","last_became_peered":"2024-02-19T15:22:59.227900+0000","last_unstale":"2024-02-19T15:23:00.662382+0000","last_undegraded":"2024-02-19T15:23:00.662382+0000","last_fullsized":"2024-02-19T15:23:00.662382+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:08.692279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":17,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663464+0000","last_change":"2024-02-19T15:22:58.208360+0000","last_active":"2024-02-19T15:23:00.663464+0000","last_peered":"2024-02-19T15:23:00.663464+0000","last_clean":"2024-02-19T15:23:00.663464+0000","last_became_active":"2024-02-19T15:22:58.207987+0000","last_became_peered":"2024-02-19T15:22:58.207987+0000","last_unstale":"2024-02-19T15:23:00.663464+0000","last_undegraded":"2024-02-19T15:23:00.663464+0000","last_fullsized":"2024-02-19T15:23:00.663464+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:01:09.606619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663309+0000","last_change":"2024-02-19T15:22:59.224735+0000","last_active":"2024-02-19T15:23:00.663309+0000","last_peered":"2024-02-19T15:23:00.663309+0000","last_clean":"2024-02-19T15:23:00.663309+0000","last_became_active":"2024-02-19T15:22:59.223502+0000","last_became_peered":"2024-02-19T15:22:59.223502+0000","last_unstale":"2024-02-19T15:23:00.663309+0000","last_undegraded":"2024-02-19T15:23:00.663309+0000","last_fullsized":"2024-02-19T15:23:00.663309+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:42.769874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662412+0000","last_change":"2024-02-19T15:22:58.211326+0000","last_active":"2024-02-19T15:23:00.662412+0000","last_peered":"2024-02-19T15:23:00.662412+0000","last_clean":"2024-02-19T15:23:00.662412+0000","last_became_active":"2024-02-19T15:22:58.211004+0000","last_became_peered":"2024-02-19T15:22:58.211004+0000","last_unstale":"2024-02-19T15:23:00.662412+0000","last_undegraded":"2024-02-19T15:23:00.662412+0000","last_fullsized":"2024-02-19T15:23:00.662412+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:40:31.134963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201927+0000","last_change":"2024-02-19T15:22:59.222270+0000","last_active":"2024-02-19T15:23:00.201927+0000","last_peered":"2024-02-19T15:23:00.201927+0000","last_clean":"2024-02-19T15:23:00.201927+0000","last_became_active":"2024-02-19T15:22:59.221994+0000","last_became_peered":"2024-02-19T15:22:59.221994+0000","last_unstale":"2024-02-19T15:23:00.201927+0000","last_undegraded":"2024-02-19T15:23:00.201927+0000","last_fullsized":"2024-02-19T15:23:00.201927+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:22:26.006732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664312+0000","last_change":"2024-02-19T15:22:58.198232+0000","last_active":"2024-02-19T15:23:00.664312+0000","last_peered":"2024-02-19T15:23:00.664312+0000","last_clean":"2024-02-19T15:23:00.664312+0000","last_became_active":"2024-02-19T15:22:58.197783+0000","last_became_peered":"2024-02-19T15:22:58.197783+0000","last_unstale":"2024-02-19T15:23:00.664312+0000","last_undegraded":"2024-02-19T15:23:00.664312+0000","last_fullsized":"2024-02-19T15:23:00.664312+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:04:51.651313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662050+0000","last_change":"2024-02-19T15:22:59.210621+0000","last_active":"2024-02-19T15:23:00.662050+0000","last_peered":"2024-02-19T15:23:00.662050+0000","last_clean":"2024-02-19T15:23:00.662050+0000","last_became_active":"2024-02-19T15:22:59.210387+0000","last_became_peered":"2024-02-19T15:22:59.210387+0000","last_unstale":"2024-02-19T15:23:00.662050+0000","last_undegraded":"2024-02-19T15:23:00.662050+0000","last_fullsized":"2024-02-19T15:23:00.662050+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:52:10.521631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.202290+0000","last_change":"2024-02-19T15:22:58.207165+0000","last_active":"2024-02-19T15:23:00.202290+0000","last_peered":"2024-02-19T15:23:00.202290+0000","last_clean":"2024-02-19T15:23:00.202290+0000","last_became_active":"2024-02-19T15:22:58.206682+0000","last_became_peered":"2024-02-19T15:22:58.206682+0000","last_unstale":"2024-02-19T15:23:00.202290+0000","last_undegraded":"2024-02-19T15:23:00.202290+0000","last_fullsized":"2024-02-19T15:23:00.202290+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:41:12.383107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663012+0000","last_change":"2024-02-19T15:22:59.232469+0000","last_active":"2024-02-19T15:23:00.663012+0000","last_peered":"2024-02-19T15:23:00.663012+0000","last_clean":"2024-02-19T15:23:00.663012+0000","last_became_active":"2024-02-19T15:22:59.232016+0000","last_became_peered":"2024-02-19T15:22:59.232016+0000","last_unstale":"2024-02-19T15:23:00.663012+0000","last_undegraded":"2024-02-19T15:23:00.663012+0000","last_fullsized":"2024-02-19T15:23:00.663012+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:54:08.451277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.661755+0000","last_change":"2024-02-19T15:22:58.209673+0000","last_active":"2024-02-19T15:23:00.661755+0000","last_peered":"2024-02-19T15:23:00.661755+0000","last_clean":"2024-02-19T15:23:00.661755+0000","last_became_active":"2024-02-19T15:22:58.209178+0000","last_became_peered":"2024-02-19T15:22:58.209178+0000","last_unstale":"2024-02-19T15:23:00.661755+0000","last_undegraded":"2024-02-19T15:23:00.661755+0000","last_fullsized":"2024-02-19T15:23:00.661755+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:29:15.629841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201631+0000","last_change":"2024-02-19T15:22:59.220850+0000","last_active":"2024-02-19T15:23:00.201631+0000","last_peered":"2024-02-19T15:23:00.201631+0000","last_clean":"2024-02-19T15:23:00.201631+0000","last_became_active":"2024-02-19T15:22:59.220526+0000","last_became_peered":"2024-02-19T15:22:59.220526+0000","last_unstale":"2024-02-19T15:23:00.201631+0000","last_undegraded":"2024-02-19T15:23:00.201631+0000","last_fullsized":"2024-02-19T15:23:00.201631+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:42:37.450463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":22,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.661908+0000","last_change":"2024-02-19T15:22:58.207005+0000","last_active":"2024-02-19T15:23:00.661908+0000","last_peered":"2024-02-19T15:23:00.661908+0000","last_clean":"2024-02-19T15:23:00.661908+0000","last_became_active":"2024-02-19T15:22:58.206744+0000","last_became_peered":"2024-02-19T15:22:58.206744+0000","last_unstale":"2024-02-19T15:23:00.661908+0000","last_undegraded":"2024-02-19T15:23:00.661908+0000","last_fullsized":"2024-02-19T15:23:00.661908+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:01:09.606619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664561+0000","last_change":"2024-02-19T15:22:59.217007+0000","last_active":"2024-02-19T15:23:00.664561+0000","last_peered":"2024-02-19T15:23:00.664561+0000","last_clean":"2024-02-19T15:23:00.664561+0000","last_became_active":"2024-02-19T15:22:59.216654+0000","last_became_peered":"2024-02-19T15:22:59.216654+0000","last_unstale":"2024-02-19T15:23:00.664561+0000","last_undegraded":"2024-02-19T15:23:00.664561+0000","last_fullsized":"2024-02-19T15:23:00.664561+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:40:17.123906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.202455+0000","last_change":"2024-02-19T15:22:58.213906+0000","last_active":"2024-02-19T15:23:00.202455+0000","last_peered":"2024-02-19T15:23:00.202455+0000","last_clean":"2024-02-19T15:23:00.202455+0000","last_became_active":"2024-02-19T15:22:58.213115+0000","last_became_peered":"2024-02-19T15:22:58.213115+0000","last_unstale":"2024-02-19T15:23:00.202455+0000","last_undegraded":"2024-02-19T15:23:00.202455+0000","last_fullsized":"2024-02-19T15:23:00.202455+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:48.695445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663732+0000","last_change":"2024-02-19T15:22:59.220627+0000","last_active":"2024-02-19T15:23:00.663732+0000","last_peered":"2024-02-19T15:23:00.663732+0000","last_clean":"2024-02-19T15:23:00.663732+0000","last_became_active":"2024-02-19T15:22:59.220362+0000","last_became_peered":"2024-02-19T15:22:59.220362+0000","last_unstale":"2024-02-19T15:23:00.663732+0000","last_undegraded":"2024-02-19T15:23:00.663732+0000","last_fullsized":"2024-02-19T15:23:00.663732+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:53.399488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664592+0000","last_change":"2024-02-19T15:22:58.198617+0000","last_active":"2024-02-19T15:23:00.664592+0000","last_peered":"2024-02-19T15:23:00.664592+0000","last_clean":"2024-02-19T15:23:00.664592+0000","last_became_active":"2024-02-19T15:22:58.198222+0000","last_became_peered":"2024-02-19T15:22:58.198222+0000","last_unstale":"2024-02-19T15:23:00.664592+0000","last_undegraded":"2024-02-19T15:23:00.664592+0000","last_fullsized":"2024-02-19T15:23:00.664592+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:09:55.343795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201960+0000","last_change":"2024-02-19T15:22:59.230410+0000","last_active":"2024-02-19T15:23:00.201960+0000","last_peered":"2024-02-19T15:23:00.201960+0000","last_clean":"2024-02-19T15:23:00.201960+0000","last_became_active":"2024-02-19T15:22:59.229960+0000","last_became_peered":"2024-02-19T15:22:59.229960+0000","last_unstale":"2024-02-19T15:23:00.201960+0000","last_undegraded":"2024-02-19T15:23:00.201960+0000","last_fullsized":"2024-02-19T15:23:00.201960+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:16:04.810073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664541+0000","last_change":"2024-02-19T15:22:58.205977+0000","last_active":"2024-02-19T15:23:00.664541+0000","last_peered":"2024-02-19T15:23:00.664541+0000","last_clean":"2024-02-19T15:23:00.664541+0000","last_became_active":"2024-02-19T15:22:58.199517+0000","last_became_peered":"2024-02-19T15:22:58.199517+0000","last_unstale":"2024-02-19T15:23:00.664541+0000","last_undegraded":"2024-02-19T15:23:00.664541+0000","last_fullsized":"2024-02-19T15:23:00.664541+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:47:09.696439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.200495+0000","last_change":"2024-02-19T15:22:59.229038+0000","last_active":"2024-02-19T15:23:00.200495+0000","last_peered":"2024-02-19T15:23:00.200495+0000","last_clean":"2024-02-19T15:23:00.200495+0000","last_became_active":"2024-02-19T15:22:59.228259+0000","last_became_peered":"2024-02-19T15:22:59.228259+0000","last_unstale":"2024-02-19T15:23:00.200495+0000","last_undegraded":"2024-02-19T15:23:00.200495+0000","last_fullsized":"2024-02-19T15:23:00.200495+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:47:10.702917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.664528+0000","last_change":"2024-02-19T15:22:58.206492+0000","last_active":"2024-02-19T15:23:00.664528+0000","last_peered":"2024-02-19T15:23:00.664528+0000","last_clean":"2024-02-19T15:23:00.664528+0000","last_became_active":"2024-02-19T15:22:58.206214+0000","last_became_peered":"2024-02-19T15:22:58.206214+0000","last_unstale":"2024-02-19T15:23:00.664528+0000","last_undegraded":"2024-02-19T15:23:00.664528+0000","last_fullsized":"2024-02-19T15:23:00.664528+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:41:06.178143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.200413+0000","last_change":"2024-02-19T15:22:59.221429+0000","last_active":"2024-02-19T15:23:00.200413+0000","last_peered":"2024-02-19T15:23:00.200413+0000","last_clean":"2024-02-19T15:23:00.200413+0000","last_became_active":"2024-02-19T15:22:59.221098+0000","last_became_peered":"2024-02-19T15:22:59.221098+0000","last_unstale":"2024-02-19T15:23:00.200413+0000","last_undegraded":"2024-02-19T15:23:00.200413+0000","last_fullsized":"2024-02-19T15:23:00.200413+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:42:22.554577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.226693+0000","last_change":"2024-02-19T15:22:58.214753+0000","last_active":"2024-02-19T15:23:00.226693+0000","last_peered":"2024-02-19T15:23:00.226693+0000","last_clean":"2024-02-19T15:23:00.226693+0000","last_became_active":"2024-02-19T15:22:58.214415+0000","last_became_peered":"2024-02-19T15:22:58.214415+0000","last_unstale":"2024-02-19T15:23:00.226693+0000","last_undegraded":"2024-02-19T15:23:00.226693+0000","last_fullsized":"2024-02-19T15:23:00.226693+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:41:10.198513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.200582+0000","last_change":"2024-02-19T15:22:59.211355+0000","last_active":"2024-02-19T15:23:00.200582+0000","last_peered":"2024-02-19T15:23:00.200582+0000","last_clean":"2024-02-19T15:23:00.200582+0000","last_became_active":"2024-02-19T15:22:59.208205+0000","last_became_peered":"2024-02-19T15:22:59.208205+0000","last_unstale":"2024-02-19T15:23:00.200582+0000","last_undegraded":"2024-02-19T15:23:00.200582+0000","last_fullsized":"2024-02-19T15:23:00.200582+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:09:56.350273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201456+0000","last_change":"2024-02-19T15:22:58.207613+0000","last_active":"2024-02-19T15:23:00.201456+0000","last_peered":"2024-02-19T15:23:00.201456+0000","last_clean":"2024-02-19T15:23:00.201456+0000","last_became_active":"2024-02-19T15:22:58.207284+0000","last_became_peered":"2024-02-19T15:22:58.207284+0000","last_unstale":"2024-02-19T15:23:00.201456+0000","last_undegraded":"2024-02-19T15:23:00.201456+0000","last_fullsized":"2024-02-19T15:23:00.201456+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:45:00.304846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.200184+0000","last_change":"2024-02-19T15:22:59.208581+0000","last_active":"2024-02-19T15:23:00.200184+0000","last_peered":"2024-02-19T15:23:00.200184+0000","last_clean":"2024-02-19T15:23:00.200184+0000","last_became_active":"2024-02-19T15:22:59.208136+0000","last_became_peered":"2024-02-19T15:22:59.208136+0000","last_unstale":"2024-02-19T15:23:00.200184+0000","last_undegraded":"2024-02-19T15:23:00.200184+0000","last_fullsized":"2024-02-19T15:23:00.200184+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:04:52.657791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663008+0000","last_change":"2024-02-19T15:22:58.203803+0000","last_active":"2024-02-19T15:23:00.663008+0000","last_peered":"2024-02-19T15:23:00.663008+0000","last_clean":"2024-02-19T15:23:00.663008+0000","last_became_active":"2024-02-19T15:22:58.203518+0000","last_became_peered":"2024-02-19T15:22:58.203518+0000","last_unstale":"2024-02-19T15:23:00.663008+0000","last_undegraded":"2024-02-19T15:23:00.663008+0000","last_fullsized":"2024-02-19T15:23:00.663008+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:30:28.131448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663316+0000","last_change":"2024-02-19T15:22:59.222928+0000","last_active":"2024-02-19T15:23:00.663316+0000","last_peered":"2024-02-19T15:23:00.663316+0000","last_clean":"2024-02-19T15:23:00.663316+0000","last_became_active":"2024-02-19T15:22:59.222291+0000","last_became_peered":"2024-02-19T15:22:59.222291+0000","last_unstale":"2024-02-19T15:23:00.663316+0000","last_undegraded":"2024-02-19T15:23:00.663316+0000","last_fullsized":"2024-02-19T15:23:00.663316+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:08.692279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662562+0000","last_change":"2024-02-19T15:22:58.209230+0000","last_active":"2024-02-19T15:23:00.662562+0000","last_peered":"2024-02-19T15:23:00.662562+0000","last_clean":"2024-02-19T15:23:00.662562+0000","last_became_active":"2024-02-19T15:22:58.208914+0000","last_became_peered":"2024-02-19T15:22:58.208914+0000","last_unstale":"2024-02-19T15:23:00.662562+0000","last_undegraded":"2024-02-19T15:23:00.662562+0000","last_fullsized":"2024-02-19T15:23:00.662562+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:40.605863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.200924+0000","last_change":"2024-02-19T15:22:59.212386+0000","last_active":"2024-02-19T15:23:00.200924+0000","last_peered":"2024-02-19T15:23:00.200924+0000","last_clean":"2024-02-19T15:23:00.200924+0000","last_became_active":"2024-02-19T15:22:59.212114+0000","last_became_peered":"2024-02-19T15:22:59.212114+0000","last_unstale":"2024-02-19T15:23:00.200924+0000","last_undegraded":"2024-02-19T15:23:00.200924+0000","last_fullsized":"2024-02-19T15:23:00.200924+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:22:57.295384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.202503+0000","last_change":"2024-02-19T15:22:58.215574+0000","last_active":"2024-02-19T15:23:00.202503+0000","last_peered":"2024-02-19T15:23:00.202503+0000","last_clean":"2024-02-19T15:23:00.202503+0000","last_became_active":"2024-02-19T15:22:58.215320+0000","last_became_peered":"2024-02-19T15:22:58.215320+0000","last_unstale":"2024-02-19T15:23:00.202503+0000","last_undegraded":"2024-02-19T15:23:00.202503+0000","last_fullsized":"2024-02-19T15:23:00.202503+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:32:40.017025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663591+0000","last_change":"2024-02-19T15:22:59.222837+0000","last_active":"2024-02-19T15:23:00.663591+0000","last_peered":"2024-02-19T15:23:00.663591+0000","last_clean":"2024-02-19T15:23:00.663591+0000","last_became_active":"2024-02-19T15:22:59.222528+0000","last_became_peered":"2024-02-19T15:22:59.222528+0000","last_unstale":"2024-02-19T15:23:00.663591+0000","last_undegraded":"2024-02-19T15:23:00.663591+0000","last_fullsized":"2024-02-19T15:23:00.663591+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:28:04.641517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"34'1","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.228480+0000","last_change":"2024-02-19T15:22:58.214998+0000","last_active":"2024-02-19T15:23:00.228480+0000","last_peered":"2024-02-19T15:23:00.228480+0000","last_clean":"2024-02-19T15:23:00.228480+0000","last_became_active":"2024-02-19T15:22:58.214534+0000","last_became_peered":"2024-02-19T15:22:58.214534+0000","last_unstale":"2024-02-19T15:23:00.228480+0000","last_undegraded":"2024-02-19T15:23:00.228480+0000","last_fullsized":"2024-02-19T15:23:00.228480+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:41:23.269776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.226625+0000","last_change":"2024-02-19T15:22:59.221728+0000","last_active":"2024-02-19T15:23:00.226625+0000","last_peered":"2024-02-19T15:23:00.226625+0000","last_clean":"2024-02-19T15:23:00.226625+0000","last_became_active":"2024-02-19T15:22:59.219985+0000","last_became_peered":"2024-02-19T15:22:59.219985+0000","last_unstale":"2024-02-19T15:23:00.226625+0000","last_undegraded":"2024-02-19T15:23:00.226625+0000","last_fullsized":"2024-02-19T15:23:00.226625+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:04:26.085669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201987+0000","last_change":"2024-02-19T15:22:58.207764+0000","last_active":"2024-02-19T15:23:00.201987+0000","last_peered":"2024-02-19T15:23:00.201987+0000","last_clean":"2024-02-19T15:23:00.201987+0000","last_became_active":"2024-02-19T15:22:58.206673+0000","last_became_peered":"2024-02-19T15:22:58.206673+0000","last_unstale":"2024-02-19T15:23:00.201987+0000","last_undegraded":"2024-02-19T15:23:00.201987+0000","last_fullsized":"2024-02-19T15:23:00.201987+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:29:15.629841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201484+0000","last_change":"2024-02-19T15:22:59.217684+0000","last_active":"2024-02-19T15:23:00.201484+0000","last_peered":"2024-02-19T15:23:00.201484+0000","last_clean":"2024-02-19T15:23:00.201484+0000","last_became_active":"2024-02-19T15:22:59.217378+0000","last_became_peered":"2024-02-19T15:22:59.217378+0000","last_unstale":"2024-02-19T15:23:00.201484+0000","last_undegraded":"2024-02-19T15:23:00.201484+0000","last_fullsized":"2024-02-19T15:23:00.201484+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:26.125796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.226801+0000","last_change":"2024-02-19T15:22:58.205664+0000","last_active":"2024-02-19T15:23:00.226801+0000","last_peered":"2024-02-19T15:23:00.226801+0000","last_clean":"2024-02-19T15:23:00.226801+0000","last_became_active":"2024-02-19T15:22:58.205359+0000","last_became_peered":"2024-02-19T15:22:58.205359+0000","last_unstale":"2024-02-19T15:23:00.226801+0000","last_undegraded":"2024-02-19T15:23:00.226801+0000","last_fullsized":"2024-02-19T15:23:00.226801+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:01:28.252912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.226687+0000","last_change":"2024-02-19T15:22:59.210920+0000","last_active":"2024-02-19T15:23:00.226687+0000","last_peered":"2024-02-19T15:23:00.226687+0000","last_clean":"2024-02-19T15:23:00.226687+0000","last_became_active":"2024-02-19T15:22:59.210531+0000","last_became_peered":"2024-02-19T15:22:59.210531+0000","last_unstale":"2024-02-19T15:23:00.226687+0000","last_undegraded":"2024-02-19T15:23:00.226687+0000","last_fullsized":"2024-02-19T15:23:00.226687+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:56:32.688788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.202216+0000","last_change":"2024-02-19T15:22:58.215032+0000","last_active":"2024-02-19T15:23:00.202216+0000","last_peered":"2024-02-19T15:23:00.202216+0000","last_clean":"2024-02-19T15:23:00.202216+0000","last_became_active":"2024-02-19T15:22:58.214755+0000","last_became_peered":"2024-02-19T15:22:58.214755+0000","last_unstale":"2024-02-19T15:23:00.202216+0000","last_undegraded":"2024-02-19T15:23:00.202216+0000","last_fullsized":"2024-02-19T15:23:00.202216+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:49:11.624620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.661788+0000","last_change":"2024-02-19T15:22:59.221371+0000","last_active":"2024-02-19T15:23:00.661788+0000","last_peered":"2024-02-19T15:23:00.661788+0000","last_clean":"2024-02-19T15:23:00.661788+0000","last_became_active":"2024-02-19T15:22:59.218613+0000","last_became_peered":"2024-02-19T15:22:59.218613+0000","last_unstale":"2024-02-19T15:23:00.661788+0000","last_undegraded":"2024-02-19T15:23:00.661788+0000","last_fullsized":"2024-02-19T15:23:00.661788+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:23:35.768395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663117+0000","last_change":"2024-02-19T15:22:58.196152+0000","last_active":"2024-02-19T15:23:00.663117+0000","last_peered":"2024-02-19T15:23:00.663117+0000","last_clean":"2024-02-19T15:23:00.663117+0000","last_became_active":"2024-02-19T15:22:58.195852+0000","last_became_peered":"2024-02-19T15:22:58.195852+0000","last_unstale":"2024-02-19T15:23:00.663117+0000","last_undegraded":"2024-02-19T15:23:00.663117+0000","last_fullsized":"2024-02-19T15:23:00.663117+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:48:44.574187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.662908+0000","last_change":"2024-02-19T15:22:59.211695+0000","last_active":"2024-02-19T15:23:00.662908+0000","last_peered":"2024-02-19T15:23:00.662908+0000","last_clean":"2024-02-19T15:23:00.662908+0000","last_became_active":"2024-02-19T15:22:59.211460+0000","last_became_peered":"2024-02-19T15:22:59.211460+0000","last_unstale":"2024-02-19T15:23:00.662908+0000","last_undegraded":"2024-02-19T15:23:00.662908+0000","last_fullsized":"2024-02-19T15:23:00.662908+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:40:32.141441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.661840+0000","last_change":"2024-02-19T15:22:58.210252+0000","last_active":"2024-02-19T15:23:00.661840+0000","last_peered":"2024-02-19T15:23:00.661840+0000","last_clean":"2024-02-19T15:23:00.661840+0000","last_became_active":"2024-02-19T15:22:58.209971+0000","last_became_peered":"2024-02-19T15:22:58.209971+0000","last_unstale":"2024-02-19T15:23:00.661840+0000","last_undegraded":"2024-02-19T15:23:00.661840+0000","last_fullsized":"2024-02-19T15:23:00.661840+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:13:33.144944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663685+0000","last_change":"2024-02-19T15:22:59.226673+0000","last_active":"2024-02-19T15:23:00.663685+0000","last_peered":"2024-02-19T15:23:00.663685+0000","last_clean":"2024-02-19T15:23:00.663685+0000","last_became_active":"2024-02-19T15:22:59.226367+0000","last_became_peered":"2024-02-19T15:22:59.226367+0000","last_unstale":"2024-02-19T15:23:00.663685+0000","last_undegraded":"2024-02-19T15:23:00.663685+0000","last_fullsized":"2024-02-19T15:23:00.663685+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:58.179604+0000","last_clean_scrub_stamp":"2024-02-19T15:22:58.179604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:28:04.641517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.201170+0000","last_change":"2024-02-19T15:22:58.202003+0000","last_active":"2024-02-19T15:23:00.201170+0000","last_peered":"2024-02-19T15:23:00.201170+0000","last_clean":"2024-02-19T15:23:00.201170+0000","last_became_active":"2024-02-19T15:22:58.201654+0000","last_became_peered":"2024-02-19T15:22:58.201654+0000","last_unstale":"2024-02-19T15:23:00.201170+0000","last_undegraded":"2024-02-19T15:23:00.201170+0000","last_fullsized":"2024-02-19T15:23:00.201170+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:22:57.173126+0000","last_clean_scrub_stamp":"2024-02-19T15:22:57.173126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:53:52.672122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":193,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-02-19T15:23:00.663080+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:23:00.663080+0000","last_peered":"2024-02-19T15:23:00.663080+0000","last_clean":"2024-02-19T15:23:00.663080+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:23:00.663080+0000","last_undegraded":"2024-02-19T15:23:00.663080+0000","last_fullsized":"2024-02-19T15:23:00.663080+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509457,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31560,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340280,"statfs":{"total":96636764160,"available":96604446720,"internally_reserved":0,"allocated":389120,"data_stored":150351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":4.6029999999999998,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.50800000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.442},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.71499999999999997}]},{"osd":2,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.53100000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.64800000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.66400000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.63200000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.70299999999999996}]}]},{"osd":7,"up_from":14,"seq":60129542161,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":966656,"data_stored":733893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.56100000000000005}]},{"osd":1,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":2.968,"5min":2.968,"15min":2.968},"last":0.72699999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":0.65500000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":3.1720000000000002,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":0.87}]},{"osd":4,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.77200000000000002}]},{"osd":5,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.81499999999999995}]},{"osd":6,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.88800000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542162,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":144107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.77600000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.84999999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.80200000000000005}]},{"osd":3,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.69899999999999995}]},{"osd":4,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":0.58599999999999997}]},{"osd":6,"last update":"Mon Feb 19 15:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.81799999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.60299999999999998}]}]},{"osd":4,"up_from":14,"seq":60129542160,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":149839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.66400000000000003}]},{"osd":1,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.61599999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.71299999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.60499999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.80100000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.68999999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.745},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.73099999999999998}]}]},{"osd":3,"up_from":14,"seq":60129542160,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31556,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":385024,"data_stored":145069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.2450000000000001,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.41199999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.313},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":4.0220000000000002,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.56200000000000006}]},{"osd":2,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.38100000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.625},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":4.2039999999999997,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.47899999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":4.1589999999999998,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.64100000000000001}]},{"osd":6,"last update":"Mon Feb 19 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":4.0940000000000003,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.52800000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":3.9660000000000002,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.69599999999999995}]}]},{"osd":2,"up_from":14,"seq":60129542161,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31560,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340280,"statfs":{"total":96636764160,"available":96604446720,"internally_reserved":0,"allocated":385024,"data_stored":149805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.7160000000000002,"5min":4.7160000000000002,"15min":4.7160000000000002},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":8.2129999999999992,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.377}]},{"osd":1,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":8.1620000000000008,"5min":8.1620000000000008,"15min":8.1620000000000008},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.53300000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":4.7679999999999998,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":4.9909999999999997,"5min":4.9909999999999997,"15min":4.9909999999999997},"last":0.502}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":4.3810000000000002,"5min":4.3810000000000002,"15min":4.3810000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":4.3440000000000003,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.626}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":4.4039999999999999,"5min":4.4039999999999999,"15min":4.4039999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":4.3259999999999996,"5min":4.3259999999999996,"15min":4.3259999999999996},"last":0.748}]},{"osd":6,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":4.5220000000000002,"5min":4.5220000000000002,"15min":4.5220000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":4.4320000000000004,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.60099999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":4.4829999999999997,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":4.4989999999999997,"5min":4.4989999999999997,"15min":4.4989999999999997},"last":0.57499999999999996}]}]},{"osd":1,"up_from":14,"seq":60129542162,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":397312,"data_stored":151369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.50800000000000001}]},{"osd":2,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.55800000000000005}]},{"osd":3,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.69999999999999996}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":1.127},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.92700000000000005}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.877}]},{"osd":6,"last update":"Mon Feb 19 15:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":1.0840000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":1.107},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":1.032}]}]},{"osd":0,"up_from":14,"seq":60129542160,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":974848,"data_stored":740103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.0199999999999996,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.313},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.36199999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":5.6669999999999998,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.4649999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.44700000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.9539999999999997,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.345,"5min":1.345,"15min":1.345},"last":0.47899999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":4.7880000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.85799999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.96099999999999997}]},{"osd":6,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":4.8460000000000001,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.83599999999999997}]},{"osd":7,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":4.9809999999999999,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":4.915,"5min":4.915,"15min":4.915},"last":0.91500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:23:03.817 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T15:23:03.817 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-19T15:23:04.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.000+0000 7fbdf7200640 1 Processor -- start 2024-02-19T15:23:04.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.000+0000 7fbdf7200640 1 -- start start 2024-02-19T15:23:04.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.000+0000 7fbdf7200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdf00cec80 0x7fbdf00d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:04.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.000+0000 7fbdf7200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbdf015f8f0 0x7fbdf015fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:04.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.000+0000 7fbdf7200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf00d0a50 con 0x7fbdf015f190 2024-02-19T15:23:04.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.000+0000 7fbdf7200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf006d470 con 0x7fbdf015f8f0 2024-02-19T15:23:04.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.000+0000 7fbdf7200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf00cfe80 con 0x7fbdf00cec80 2024-02-19T15:23:04.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.000+0000 7fbdf5200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbdf015f8f0 0x7fbdf015fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:04.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.000+0000 7fbdf5200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbdf015f8f0 0x7fbdf015fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48064/0 (socket says 172.21.15.47:48064) 2024-02-19T15:23:04.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.000+0000 7fbdf5200640 1 -- 172.21.15.47:0/3845652259 learned_addr learned my addr 172.21.15.47:0/3845652259 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:04.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.000+0000 7fbdefe00640 1 --2- 172.21.15.47:0/3845652259 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdf00cec80 0x7fbdf00d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:04.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.004+0000 7fbdf5200640 1 -- 172.21.15.47:0/3845652259 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdf00cec80 msgr2=0x7fbdf00d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:04.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.004+0000 7fbdf5200640 1 --2- 172.21.15.47:0/3845652259 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdf00cec80 0x7fbdf00d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.004+0000 7fbdf5200640 1 -- 172.21.15.47:0/3845652259 >> v1:172.21.15.116:6789/0 conn(0x7fbdf015f190 legacy=0x7fbdf006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:04.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.004+0000 7fbdf5200640 1 -- 172.21.15.47:0/3845652259 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf00d0120 con 0x7fbdf015f8f0 2024-02-19T15:23:04.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.004+0000 7fbdefe00640 1 --2- 172.21.15.47:0/3845652259 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdf00cec80 0x7fbdf00d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:04.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.004+0000 7fbdf5200640 1 --2- 172.21.15.47:0/3845652259 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbdf015f8f0 0x7fbdf015fd50 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fbde8009e50 tx=0x7fbde801ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fadf6892d7308fd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:04.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.004+0000 7fbdef400640 1 -- 172.21.15.47:0/3845652259 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde80a57e0 con 0x7fbdf015f8f0 2024-02-19T15:23:04.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.004+0000 7fbdef400640 1 -- 172.21.15.47:0/3845652259 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde80a5940 con 0x7fbdf015f8f0 2024-02-19T15:23:04.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.004+0000 7fbdef400640 1 -- 172.21.15.47:0/3845652259 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde80a5cd0 con 0x7fbdf015f8f0 2024-02-19T15:23:04.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.004+0000 7fbdf7200640 1 -- 172.21.15.47:0/3845652259 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbdf015f8f0 msgr2=0x7fbdf015fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:04.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.004+0000 7fbdf7200640 1 --2- 172.21.15.47:0/3845652259 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbdf015f8f0 0x7fbdf015fd50 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fbde8009e50 tx=0x7fbde801ca00 comp rx=0 tx=0).stop 2024-02-19T15:23:04.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.004+0000 7fbdf7200640 1 -- 172.21.15.47:0/3845652259 shutdown_connections 2024-02-19T15:23:04.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf7200640 1 --2- 172.21.15.47:0/3845652259 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbdf015f8f0 0x7fbdf015fd50 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf7200640 1 --2- 172.21.15.47:0/3845652259 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdf00cec80 0x7fbdf00d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf7200640 1 -- 172.21.15.47:0/3845652259 >> 172.21.15.47:0/3845652259 conn(0x7fbdf009aed0 msgr2=0x7fbdf00d1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:04.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf7200640 1 -- 172.21.15.47:0/3845652259 shutdown_connections 2024-02-19T15:23:04.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf7200640 1 -- 172.21.15.47:0/3845652259 wait complete. 2024-02-19T15:23:04.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf7200640 1 Processor -- start 2024-02-19T15:23:04.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf7200640 1 -- start start 2024-02-19T15:23:04.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf7200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdf00cec80 0x7fbdf00ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:04.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf7200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdf015f190 0x7fbdf00c6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:04.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf7200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbdf015f8f0 0x7fbdf00c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:04.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf7200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf00cac80 con 0x7fbdf015f8f0 2024-02-19T15:23:04.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf7200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf00caf60 con 0x7fbdf015f190 2024-02-19T15:23:04.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf7200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf00cb240 con 0x7fbdf00cec80 2024-02-19T15:23:04.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf5200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbdf015f8f0 0x7fbdf00c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:04.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf5200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbdf015f8f0 0x7fbdf00c6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48078/0 (socket says 172.21.15.47:48078) 2024-02-19T15:23:04.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf5200640 1 -- 172.21.15.47:0/1981086127 learned_addr learned my addr 172.21.15.47:0/1981086127 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:04.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdefe00640 1 --2- 172.21.15.47:0/1981086127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdf00cec80 0x7fbdf00ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:04.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbde7e00640 1 --2- 172.21.15.47:0/1981086127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdf015f190 0x7fbdf00c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:04.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf5200640 1 -- 172.21.15.47:0/1981086127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdf00cec80 msgr2=0x7fbdf00ca2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:04.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf5200640 1 --2- 172.21.15.47:0/1981086127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdf00cec80 0x7fbdf00ca2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf5200640 1 -- 172.21.15.47:0/1981086127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdf015f190 msgr2=0x7fbdf00c6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:04.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf5200640 1 --2- 172.21.15.47:0/1981086127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdf015f190 0x7fbdf00c6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.008+0000 7fbdf5200640 1 -- 172.21.15.47:0/1981086127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde8009b30 con 0x7fbdf015f8f0 2024-02-19T15:23:04.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.012+0000 7fbdf5200640 1 --2- 172.21.15.47:0/1981086127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbdf015f8f0 0x7fbdf00c6a60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fbde80af040 tx=0x7fbde80197a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:04.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.012+0000 7fbdecc00640 1 -- 172.21.15.47:0/1981086127 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde80b8640 con 0x7fbdf015f8f0 2024-02-19T15:23:04.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.012+0000 7fbdecc00640 1 -- 172.21.15.47:0/1981086127 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde80b87a0 con 0x7fbdf015f8f0 2024-02-19T15:23:04.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.012+0000 7fbdf7200640 1 -- 172.21.15.47:0/1981086127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf00cb520 con 0x7fbdf015f8f0 2024-02-19T15:23:04.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.012+0000 7fbdecc00640 1 -- 172.21.15.47:0/1981086127 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde80b8b50 con 0x7fbdf015f8f0 2024-02-19T15:23:04.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.012+0000 7fbdf7200640 1 -- 172.21.15.47:0/1981086127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf00cbd20 con 0x7fbdf015f8f0 2024-02-19T15:23:04.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.012+0000 7fbdf7200640 1 -- 172.21.15.47:0/1981086127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf0075360 con 0x7fbdf015f8f0 2024-02-19T15:23:04.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.012+0000 7fbdecc00640 1 -- 172.21.15.47:0/1981086127 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fbde80b8cb0 con 0x7fbdf015f8f0 2024-02-19T15:23:04.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.012+0000 7fbdecc00640 1 --2- 172.21.15.47:0/1981086127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbdcc066e40 0x7fbdcc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:04.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.012+0000 7fbdecc00640 1 -- 172.21.15.47:0/1981086127 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbde811fb40 con 0x7fbdf015f8f0 2024-02-19T15:23:04.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.012+0000 7fbdefe00640 1 --2- 172.21.15.47:0/1981086127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbdcc066e40 0x7fbdcc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:04.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.016+0000 7fbdefe00640 1 --2- 172.21.15.47:0/1981086127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbdcc066e40 0x7fbdcc069300 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009950 tx=0x7fbde0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:04.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.016+0000 7fbdecc00640 1 -- 172.21.15.47:0/1981086127 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fbde80eca60 con 0x7fbdf015f8f0 2024-02-19T15:23:04.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.240+0000 7fbdf7200640 1 -- 172.21.15.47:0/1981086127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fbdf00cb9d0 con 0x7fbdf015f8f0 2024-02-19T15:23:04.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.240+0000 7fbdecc00640 1 -- 172.21.15.47:0/1981086127 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7fbde80f0120 con 0x7fbdf015f8f0 2024-02-19T15:23:04.246 INFO:teuthology.orchestra.run.smithi047.stdout:[client.0] 2024-02-19T15:23:04.247 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDjcdNlb7eRNRAAk6ZLGfjhcdlR9GoXFCd9eA== 2024-02-19T15:23:04.247 INFO:teuthology.orchestra.run.smithi047.stdout: caps mds = "allow" 2024-02-19T15:23:04.247 INFO:teuthology.orchestra.run.smithi047.stdout: caps mon = "allow r" 2024-02-19T15:23:04.247 INFO:teuthology.orchestra.run.smithi047.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-19T15:23:04.247 INFO:teuthology.orchestra.run.smithi047.stderr:updated caps for client.0 2024-02-19T15:23:04.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.244+0000 7fbdf7200640 1 -- 172.21.15.47:0/1981086127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbdcc066e40 msgr2=0x7fbdcc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:04.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.244+0000 7fbdf7200640 1 --2- 172.21.15.47:0/1981086127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbdcc066e40 0x7fbdcc069300 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009950 tx=0x7fbde0008040 comp rx=0 tx=0).stop 2024-02-19T15:23:04.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.244+0000 7fbdf7200640 1 -- 172.21.15.47:0/1981086127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbdf015f8f0 msgr2=0x7fbdf00c6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.244+0000 7fbdf7200640 1 --2- 172.21.15.47:0/1981086127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbdf015f8f0 0x7fbdf00c6a60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fbde80af040 tx=0x7fbde80197a0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.244+0000 7fbdf7200640 1 -- 172.21.15.47:0/1981086127 shutdown_connections 2024-02-19T15:23:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.244+0000 7fbdf7200640 1 --2- 172.21.15.47:0/1981086127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbdcc066e40 0x7fbdcc069300 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.244+0000 7fbdf7200640 1 --2- 172.21.15.47:0/1981086127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbdf015f8f0 0x7fbdf00c6a60 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.244+0000 7fbdf7200640 1 --2- 172.21.15.47:0/1981086127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdf015f190 0x7fbdf00c6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.244+0000 7fbdf7200640 1 --2- 172.21.15.47:0/1981086127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdf00cec80 0x7fbdf00ca2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.244+0000 7fbdf7200640 1 -- 172.21.15.47:0/1981086127 >> 172.21.15.47:0/1981086127 conn(0x7fbdf009aed0 msgr2=0x7fbdf006f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.244+0000 7fbdf7200640 1 -- 172.21.15.47:0/1981086127 shutdown_connections 2024-02-19T15:23:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.244+0000 7fbdf7200640 1 -- 172.21.15.47:0/1981086127 wait complete. 2024-02-19T15:23:04.269 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:23:04.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.404+0000 7f5d1a000640 1 Processor -- start 2024-02-19T15:23:04.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.404+0000 7f5d1a000640 1 -- start start 2024-02-19T15:23:04.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.404+0000 7f5d1a000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d141456a0 0x7f5d140cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:04.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.404+0000 7f5d1a000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d14145a80 con 0x7f5d140cf4a0 2024-02-19T15:23:04.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.404+0000 7f5d1a000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d140d1880 con 0x7f5d1406b5c0 2024-02-19T15:23:04.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.404+0000 7f5d1a000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d140d03c0 con 0x7f5d141456a0 2024-02-19T15:23:04.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.404+0000 7f5d0be00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5d140cf4a0 0x7f5d1406aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42990/0 (socket says 172.21.15.47:42990) 2024-02-19T15:23:04.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.404+0000 7f5d0be00640 1 -- 172.21.15.47:0/4255245381 learned_addr learned my addr 172.21.15.47:0/4255245381 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:04.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.404+0000 7f5d13400640 1 --2- 172.21.15.47:0/4255245381 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d141456a0 0x7f5d140cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:04.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.404+0000 7f5d12a00640 1 -- 172.21.15.47:0/4255245381 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027692031 0 0) 0x7f5d14145a80 con 0x7f5d140cf4a0 2024-02-19T15:23:04.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.404+0000 7f5d12a00640 1 -- 172.21.15.47:0/4255245381 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cf8003530 con 0x7f5d140cf4a0 2024-02-19T15:23:04.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d12a00640 1 -- 172.21.15.47:0/4255245381 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2884825086 0 0) 0x7f5cf8003530 con 0x7f5d140cf4a0 2024-02-19T15:23:04.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d12a00640 1 -- 172.21.15.47:0/4255245381 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d141456a0 msgr2=0x7f5d140cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:04.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d12a00640 1 --2- 172.21.15.47:0/4255245381 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d141456a0 0x7f5d140cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d12a00640 1 -- 172.21.15.47:0/4255245381 >> v1:172.21.15.116:6789/0 conn(0x7f5d1406b5c0 legacy=0x7f5d1406dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:04.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d12a00640 1 -- 172.21.15.47:0/4255245381 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d140cfb20 con 0x7f5d140cf4a0 2024-02-19T15:23:04.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d12a00640 1 -- 172.21.15.47:0/4255245381 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f5cfc001680 con 0x7f5d140cf4a0 2024-02-19T15:23:04.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d12a00640 1 -- 172.21.15.47:0/4255245381 >> v1:172.21.15.47:6789/0 conn(0x7f5d140cf4a0 legacy=0x7f5d1406aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:04.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d12a00640 1 --2- 172.21.15.47:0/4255245381 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cf80039b0 0x7f5cf8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:04.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d12a00640 1 --2- 172.21.15.47:0/4255245381 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d141456a0 0x7f5cf80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:04.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d12a00640 1 --2- 172.21.15.47:0/4255245381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf8004d10 0x7f5cf8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:04.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d12a00640 1 -- 172.21.15.47:0/4255245381 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d140d03c0 con 0x7f5cf80039b0 2024-02-19T15:23:04.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d12a00640 1 -- 172.21.15.47:0/4255245381 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf800b990 con 0x7f5cf8004d10 2024-02-19T15:23:04.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d12a00640 1 -- 172.21.15.47:0/4255245381 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf800bb50 con 0x7f5d141456a0 2024-02-19T15:23:04.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d13400640 1 --2- 172.21.15.47:0/4255245381 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cf80039b0 0x7f5cf8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:04.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d0be00640 1 --2- 172.21.15.47:0/4255245381 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d141456a0 0x7f5cf80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:04.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d13e00640 1 --2- 172.21.15.47:0/4255245381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf8004d10 0x7f5cf8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:04.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d13400640 1 -- 172.21.15.47:0/4255245381 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d141456a0 msgr2=0x7f5cf80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:04.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d13400640 1 --2- 172.21.15.47:0/4255245381 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d141456a0 0x7f5cf80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d13400640 1 -- 172.21.15.47:0/4255245381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf8004d10 msgr2=0x7f5cf8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:04.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d13400640 1 --2- 172.21.15.47:0/4255245381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf8004d10 0x7f5cf8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d13400640 1 -- 172.21.15.47:0/4255245381 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf8003530 con 0x7f5cf80039b0 2024-02-19T15:23:04.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.408+0000 7f5d13400640 1 --2- 172.21.15.47:0/4255245381 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cf80039b0 0x7f5cf8003d90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5d04009ae0 tx=0x7f5d0401c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7e5866adc5b676 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:04.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d12a00640 1 -- 172.21.15.47:0/4255245381 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d040a55a0 con 0x7f5cf80039b0 2024-02-19T15:23:04.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d12a00640 1 -- 172.21.15.47:0/4255245381 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d040a5700 con 0x7f5cf80039b0 2024-02-19T15:23:04.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d12a00640 1 -- 172.21.15.47:0/4255245381 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d040a5ab0 con 0x7f5cf80039b0 2024-02-19T15:23:04.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d1a000640 1 -- 172.21.15.47:0/4255245381 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cf80039b0 msgr2=0x7f5cf8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:04.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d1a000640 1 --2- 172.21.15.47:0/4255245381 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cf80039b0 0x7f5cf8003d90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5d04009ae0 tx=0x7f5d0401c9b0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d13e00640 1 -- 172.21.15.47:0/4255245381 reap_dead start 2024-02-19T15:23:04.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d1a000640 1 -- 172.21.15.47:0/4255245381 shutdown_connections 2024-02-19T15:23:04.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d1a000640 1 -- 172.21.15.47:0/4255245381 >> 172.21.15.47:0/4255245381 conn(0x7f5d140d0ad0 msgr2=0x7f5d140d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:04.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d1a000640 1 -- 172.21.15.47:0/4255245381 shutdown_connections 2024-02-19T15:23:04.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d1a000640 1 -- 172.21.15.47:0/4255245381 wait complete. 2024-02-19T15:23:04.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d1a000640 1 Processor -- start 2024-02-19T15:23:04.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d1a000640 1 -- start start 2024-02-19T15:23:04.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d1a000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf80039b0 0x7f5d141e2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:04.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d1a000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d140aee10 0x7f5d140b53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:04.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d1a000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d140ad840 0x7f5d14143b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:04.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d1a000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d140add70 con 0x7f5d140aee10 2024-02-19T15:23:04.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d1a000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d140ae050 con 0x7f5cf80039b0 2024-02-19T15:23:04.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d1a000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d140ae330 con 0x7f5d140ad840 2024-02-19T15:23:04.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d0be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d140aee10 0x7f5d140b53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:04.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d0be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d140aee10 0x7f5d140b53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48088/0 (socket says 172.21.15.47:48088) 2024-02-19T15:23:04.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d0be00640 1 -- 172.21.15.47:0/1852453670 learned_addr learned my addr 172.21.15.47:0/1852453670 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:04.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.412+0000 7f5d13400640 1 --2- 172.21.15.47:0/1852453670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf80039b0 0x7f5d141e2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:04.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.416+0000 7f5d13e00640 1 --2- 172.21.15.47:0/1852453670 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d140ad840 0x7f5d14143b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:04.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.416+0000 7f5d0be00640 1 -- 172.21.15.47:0/1852453670 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d140ad840 msgr2=0x7f5d14143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:04.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.416+0000 7f5d0be00640 1 --2- 172.21.15.47:0/1852453670 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d140ad840 0x7f5d14143b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.416+0000 7f5d0be00640 1 -- 172.21.15.47:0/1852453670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf80039b0 msgr2=0x7f5d141e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:04.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.416+0000 7f5d0be00640 1 --2- 172.21.15.47:0/1852453670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf80039b0 0x7f5d141e2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.416+0000 7f5d0be00640 1 -- 172.21.15.47:0/1852453670 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d140ae610 con 0x7f5d140aee10 2024-02-19T15:23:04.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.416+0000 7f5d0be00640 1 --2- 172.21.15.47:0/1852453670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d140aee10 0x7f5d140b53e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc007290 tx=0x7f5cfc007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:04.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.416+0000 7f5d0b400640 1 -- 172.21.15.47:0/1852453670 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc00f8c0 con 0x7f5d140aee10 2024-02-19T15:23:04.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.416+0000 7f5d0b400640 1 -- 172.21.15.47:0/1852453670 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cfc00fa60 con 0x7f5d140aee10 2024-02-19T15:23:04.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.416+0000 7f5d0b400640 1 -- 172.21.15.47:0/1852453670 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc00fe10 con 0x7f5d140aee10 2024-02-19T15:23:04.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.416+0000 7f5d1a000640 1 -- 172.21.15.47:0/1852453670 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d140ae8f0 con 0x7f5d140aee10 2024-02-19T15:23:04.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.416+0000 7f5d1a000640 1 -- 172.21.15.47:0/1852453670 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d140b5920 con 0x7f5d140aee10 2024-02-19T15:23:04.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.416+0000 7f5d1a000640 1 -- 172.21.15.47:0/1852453670 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d14075360 con 0x7f5d140aee10 2024-02-19T15:23:04.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.416+0000 7f5d0b400640 1 -- 172.21.15.47:0/1852453670 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5cfc01d460 con 0x7f5d140aee10 2024-02-19T15:23:04.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.420+0000 7f5d0b400640 1 --2- 172.21.15.47:0/1852453670 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5cf0066e40 0x7f5cf0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:04.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.420+0000 7f5d0b400640 1 -- 172.21.15.47:0/1852453670 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5cfc084aa0 con 0x7f5d140aee10 2024-02-19T15:23:04.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.420+0000 7f5d13400640 1 --2- 172.21.15.47:0/1852453670 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5cf0066e40 0x7f5cf0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:04.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.420+0000 7f5d13400640 1 --2- 172.21.15.47:0/1852453670 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5cf0066e40 0x7f5cf0069300 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5d040af000 tx=0x7f5d040046d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:04.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.420+0000 7f5d0b400640 1 -- 172.21.15.47:0/1852453670 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f5cfc0518a0 con 0x7f5d140aee10 2024-02-19T15:23:04.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.616+0000 7f5d1a000640 1 -- 172.21.15.47:0/1852453670 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d14072b90 con 0x7f5d140aee10 2024-02-19T15:23:04.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.616+0000 7f5d0b400640 1 -- 172.21.15.47:0/1852453670 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f5cfc054f60 con 0x7f5d140aee10 2024-02-19T15:23:04.621 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:04.621 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1616905430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1616905430},{"type":"v1","addr":"172.21.15.116:6836","nonce":1616905430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4814,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2845273225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2845273225},{"type":"v1","addr":"172.21.15.116:6837","nonce":2845273225}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5064,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/20043732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":20043732},{"type":"v1","addr":"172.21.15.47:6834","nonce":20043732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:22:58.590074+0000","modified":"2024-02-19T15:23:01.656160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3700339993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3700339993},{"type":"v1","addr":"172.21.15.47:6835","nonce":3700339993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T15:23:04.622 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 16 2024-02-19T15:23:04.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.620+0000 7f5d1a000640 1 -- 172.21.15.47:0/1852453670 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5cf0066e40 msgr2=0x7f5cf0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:04.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.620+0000 7f5d1a000640 1 --2- 172.21.15.47:0/1852453670 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5cf0066e40 0x7f5cf0069300 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5d040af000 tx=0x7f5d040046d0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.620+0000 7f5d1a000640 1 -- 172.21.15.47:0/1852453670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d140aee10 msgr2=0x7f5d140b53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:04.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.620+0000 7f5d1a000640 1 --2- 172.21.15.47:0/1852453670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d140aee10 0x7f5d140b53e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc007290 tx=0x7f5cfc007750 comp rx=0 tx=0).stop 2024-02-19T15:23:04.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.620+0000 7f5d1a000640 1 -- 172.21.15.47:0/1852453670 shutdown_connections 2024-02-19T15:23:04.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.620+0000 7f5d1a000640 1 --2- 172.21.15.47:0/1852453670 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5cf0066e40 0x7f5cf0069300 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.620+0000 7f5d1a000640 1 --2- 172.21.15.47:0/1852453670 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d140ad840 0x7f5d14143b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.620+0000 7f5d1a000640 1 --2- 172.21.15.47:0/1852453670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d140aee10 0x7f5d140b53e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.620+0000 7f5d1a000640 1 --2- 172.21.15.47:0/1852453670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf80039b0 0x7f5d141e2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:04.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.620+0000 7f5d1a000640 1 -- 172.21.15.47:0/1852453670 >> 172.21.15.47:0/1852453670 conn(0x7f5d140d0ad0 msgr2=0x7f5d141e57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:04.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.620+0000 7f5d1a000640 1 -- 172.21.15.47:0/1852453670 shutdown_connections 2024-02-19T15:23:04.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:04.620+0000 7f5d1a000640 1 -- 172.21.15.47:0/1852453670 wait complete. 2024-02-19T15:23:05.046 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:22:58.590074+0000', 'modified': '2024-02-19T15:23:01.656160+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5067}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5067': {'gid': 5067, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/3700339993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 3700339993}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 3700339993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:23:05.046 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T15:23:05.046 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T15:23:05.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.184+0000 7f9975e00640 1 Processor -- start 2024-02-19T15:23:05.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.184+0000 7f9975e00640 1 -- start start 2024-02-19T15:23:05.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.184+0000 7f9975e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f99700b2fc0 0x7f99700d0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.184+0000 7f9975e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f99700ceb00 0x7f99700cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.184+0000 7f9975e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99700d10a0 con 0x7f99700cef90 2024-02-19T15:23:05.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.184+0000 7f9975e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99700b33a0 con 0x7f99700ceb00 2024-02-19T15:23:05.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.184+0000 7f9975e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99700cc2f0 con 0x7f99700b2fc0 2024-02-19T15:23:05.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.184+0000 7f9967e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f99700cef90 0x7f99700ce3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42998/0 (socket says 172.21.15.47:42998) 2024-02-19T15:23:05.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.184+0000 7f9967e00640 1 -- 172.21.15.47:0/3613143114 learned_addr learned my addr 172.21.15.47:0/3613143114 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:05.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.184+0000 7f996f400640 1 --2- 172.21.15.47:0/3613143114 >> v2:172.21.15.47:3300/0 conn(0x7f99700ceb00 0x7f99700cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.184+0000 7f996ea00640 1 --2- 172.21.15.47:0/3613143114 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f99700b2fc0 0x7f99700d0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.184+0000 7f996f400640 1 -- 172.21.15.47:0/3613143114 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f99700b2fc0 msgr2=0x7f99700d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:05.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.184+0000 7f996f400640 1 --2- 172.21.15.47:0/3613143114 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f99700b2fc0 0x7f99700d0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.184+0000 7f996f400640 1 -- 172.21.15.47:0/3613143114 >> v1:172.21.15.47:6789/0 conn(0x7f99700cef90 legacy=0x7f99700ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:05.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.184+0000 7f996f400640 1 -- 172.21.15.47:0/3613143114 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99700cc5d0 con 0x7f99700ceb00 2024-02-19T15:23:05.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.188+0000 7f996f400640 1 --2- 172.21.15.47:0/3613143114 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99700ceb00 0x7f99700cd0b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9968009d90 tx=0x7f996801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca24b318b644ae6b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:05.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.188+0000 7f996e000640 1 -- 172.21.15.47:0/3613143114 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99680a5970 con 0x7f99700ceb00 2024-02-19T15:23:05.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.188+0000 7f996e000640 1 -- 172.21.15.47:0/3613143114 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99680a5ad0 con 0x7f99700ceb00 2024-02-19T15:23:05.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.188+0000 7f996e000640 1 -- 172.21.15.47:0/3613143114 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99680a5e60 con 0x7f99700ceb00 2024-02-19T15:23:05.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.188+0000 7f9975e00640 1 -- 172.21.15.47:0/3613143114 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99700ceb00 msgr2=0x7f99700cd0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:05.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.188+0000 7f9975e00640 1 --2- 172.21.15.47:0/3613143114 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99700ceb00 0x7f99700cd0b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9968009d90 tx=0x7f996801c8b0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.188+0000 7f9975e00640 1 -- 172.21.15.47:0/3613143114 shutdown_connections 2024-02-19T15:23:05.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.188+0000 7f9975e00640 1 --2- 172.21.15.47:0/3613143114 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99700ceb00 0x7f99700cd0b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.188+0000 7f9975e00640 1 --2- 172.21.15.47:0/3613143114 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f99700b2fc0 0x7f99700d0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.188+0000 7f9975e00640 1 -- 172.21.15.47:0/3613143114 >> 172.21.15.47:0/3613143114 conn(0x7f99700d1b70 msgr2=0x7f99700d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:05.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.188+0000 7f9975e00640 1 -- 172.21.15.47:0/3613143114 shutdown_connections 2024-02-19T15:23:05.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.188+0000 7f9975e00640 1 -- 172.21.15.47:0/3613143114 wait complete. 2024-02-19T15:23:05.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9975e00640 1 Processor -- start 2024-02-19T15:23:05.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9975e00640 1 -- start start 2024-02-19T15:23:05.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9975e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f99700b2fc0 0x7f99700b6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9975e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99700ceb00 0x7f99700af460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9975e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99700cef90 0x7f99700c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9975e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99700b7450 con 0x7f99700ceb00 2024-02-19T15:23:05.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9975e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99700b6150 con 0x7f99700cef90 2024-02-19T15:23:05.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9975e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99700b6430 con 0x7f99700b2fc0 2024-02-19T15:23:05.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9967e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99700ceb00 0x7f99700af460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9967e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99700ceb00 0x7f99700af460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48112/0 (socket says 172.21.15.47:48112) 2024-02-19T15:23:05.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9967e00640 1 -- 172.21.15.47:0/1553643873 learned_addr learned my addr 172.21.15.47:0/1553643873 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:05.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f996ea00640 1 --2- 172.21.15.47:0/1553643873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f99700b2fc0 0x7f99700b6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f996f400640 1 --2- 172.21.15.47:0/1553643873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99700cef90 0x7f99700c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9967e00640 1 -- 172.21.15.47:0/1553643873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f99700b2fc0 msgr2=0x7f99700b6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:05.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9967e00640 1 --2- 172.21.15.47:0/1553643873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f99700b2fc0 0x7f99700b6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9967e00640 1 -- 172.21.15.47:0/1553643873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99700cef90 msgr2=0x7f99700c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:05.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9967e00640 1 --2- 172.21.15.47:0/1553643873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99700cef90 0x7f99700c8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9967e00640 1 -- 172.21.15.47:0/1553643873 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9968009a70 con 0x7f99700ceb00 2024-02-19T15:23:05.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9967e00640 1 --2- 172.21.15.47:0/1553643873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99700ceb00 0x7f99700af460 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f995800c430 tx=0x7f995800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:05.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9967400640 1 -- 172.21.15.47:0/1553643873 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9958005cf0 con 0x7f99700ceb00 2024-02-19T15:23:05.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9967400640 1 -- 172.21.15.47:0/1553643873 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9958005e50 con 0x7f99700ceb00 2024-02-19T15:23:05.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9975e00640 1 -- 172.21.15.47:0/1553643873 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99700b6710 con 0x7f99700ceb00 2024-02-19T15:23:05.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.192+0000 7f9967400640 1 -- 172.21.15.47:0/1553643873 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99580072f0 con 0x7f99700ceb00 2024-02-19T15:23:05.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.196+0000 7f9975e00640 1 -- 172.21.15.47:0/1553643873 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99700b5e00 con 0x7f99700ceb00 2024-02-19T15:23:05.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.196+0000 7f9967400640 1 -- 172.21.15.47:0/1553643873 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9958007510 con 0x7f99700ceb00 2024-02-19T15:23:05.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.196+0000 7f9967400640 1 --2- 172.21.15.47:0/1553643873 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f994c066e40 0x7f994c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.196+0000 7f9967400640 1 -- 172.21.15.47:0/1553643873 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9958083fa0 con 0x7f99700ceb00 2024-02-19T15:23:05.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.196+0000 7f996ea00640 1 --2- 172.21.15.47:0/1553643873 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f994c066e40 0x7f994c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.196+0000 7f9979304000 1 Processor -- start 2024-02-19T15:23:05.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.196+0000 7f9979304000 1 -- start start 2024-02-19T15:23:05.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f996ea00640 1 --2- 172.21.15.47:0/1553643873 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f994c066e40 0x7f994c069300 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f99600079c0 tx=0x7f9960008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:05.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f9979304000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b33496c920 0x55b33496cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f9979304000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b33496d240 0x55b3349229f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f9979304000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b334922f30 0x55b334923310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f9979304000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3347ead00 con 0x55b334922f30 2024-02-19T15:23:05.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f9979304000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b33477c430 con 0x55b33496d240 2024-02-19T15:23:05.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f9979304000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b334779960 con 0x55b33496c920 2024-02-19T15:23:05.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f996f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b334922f30 0x55b334923310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f996f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b334922f30 0x55b334923310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48122/0 (socket says 172.21.15.47:48122) 2024-02-19T15:23:05.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f996f400640 1 -- 172.21.15.47:0/956067811 learned_addr learned my addr 172.21.15.47:0/956067811 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:05.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f996ea00640 1 --2- 172.21.15.47:0/956067811 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b33496c920 0x55b33496cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f9967e00640 1 --2- 172.21.15.47:0/956067811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b33496d240 0x55b3349229f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f996f400640 1 -- 172.21.15.47:0/956067811 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b33496c920 msgr2=0x55b33496cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:05.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f996f400640 1 --2- 172.21.15.47:0/956067811 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b33496c920 0x55b33496cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f996f400640 1 -- 172.21.15.47:0/956067811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b33496d240 msgr2=0x55b3349229f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:05.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f996f400640 1 --2- 172.21.15.47:0/956067811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b33496d240 0x55b3349229f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f996f400640 1 -- 172.21.15.47:0/956067811 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b33446b890 con 0x55b334922f30 2024-02-19T15:23:05.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f996ea00640 1 --2- 172.21.15.47:0/956067811 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b33496c920 0x55b33496cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:05.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f996f400640 1 --2- 172.21.15.47:0/956067811 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b334922f30 0x55b334923310 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f996801af60 tx=0x7f9968004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bce7fdfb520f4ed server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:05.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f995fe00640 1 -- 172.21.15.47:0/956067811 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99680ad640 con 0x55b334922f30 2024-02-19T15:23:05.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f995fe00640 1 -- 172.21.15.47:0/956067811 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99680ad7a0 con 0x55b334922f30 2024-02-19T15:23:05.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f995fe00640 1 -- 172.21.15.47:0/956067811 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99680adb50 con 0x55b334922f30 2024-02-19T15:23:05.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f9979304000 1 -- 172.21.15.47:0/956067811 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b334922f30 msgr2=0x55b334923310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:05.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f9979304000 1 --2- 172.21.15.47:0/956067811 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b334922f30 0x55b334923310 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f996801af60 tx=0x7f9968004ac0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f9979304000 1 -- 172.21.15.47:0/956067811 shutdown_connections 2024-02-19T15:23:05.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f9979304000 1 --2- 172.21.15.47:0/956067811 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b334922f30 0x55b334923310 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f9979304000 1 --2- 172.21.15.47:0/956067811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b33496d240 0x55b3349229f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f9979304000 1 --2- 172.21.15.47:0/956067811 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b33496c920 0x55b33496cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f9979304000 1 -- 172.21.15.47:0/956067811 >> 172.21.15.47:0/956067811 conn(0x55b3349292a0 msgr2=0x55b3349212d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:05.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f9979304000 1 -- 172.21.15.47:0/956067811 shutdown_connections 2024-02-19T15:23:05.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f9979304000 1 -- 172.21.15.47:0/956067811 wait complete. 2024-02-19T15:23:05.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f9979304000 1 -- start start 2024-02-19T15:23:05.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.200+0000 7f9979304000 1 Processor -- start 2024-02-19T15:23:05.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f9979304000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b334922f30 0x55b33496d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f9979304000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b33496c920 0x55b33496dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f9979304000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b33496d240 0x55b33496e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f9979304000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b334775140 con 0x55b33496d240 2024-02-19T15:23:05.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f9979304000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b33470f630 con 0x55b33496c920 2024-02-19T15:23:05.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f9979304000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b33470f2d0 con 0x55b334922f30 2024-02-19T15:23:05.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f9979304000 10 client.0 _ll_register_callbacks cb 0x55b3348251c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:23:05.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f996f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b33496d240 0x55b33496e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f996f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b33496d240 0x55b33496e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48132/0 (socket says 172.21.15.47:48132) 2024-02-19T15:23:05.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f996ea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b334922f30 0x55b33496d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f996f400640 1 -- 172.21.15.47:0/2226672757 learned_addr learned my addr 172.21.15.47:0/2226672757 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:05.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f9967e00640 1 --2- 172.21.15.47:0/2226672757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b33496c920 0x55b33496dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f996f400640 1 -- 172.21.15.47:0/2226672757 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b334922f30 msgr2=0x55b33496d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:05.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f996f400640 1 --2- 172.21.15.47:0/2226672757 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b334922f30 0x55b33496d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f996f400640 1 -- 172.21.15.47:0/2226672757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b33496c920 msgr2=0x55b33496dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:05.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f996f400640 1 --2- 172.21.15.47:0/2226672757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b33496c920 0x55b33496dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f996f400640 1 -- 172.21.15.47:0/2226672757 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99700cc5d0 con 0x55b33496d240 2024-02-19T15:23:05.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f996f400640 1 --2- 172.21.15.47:0/2226672757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b33496d240 0x55b33496e2b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f996801af60 tx=0x7f996801b0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e592bd8af36ca8cd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:05.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f995cc00640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:23:05.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f995cc00640 1 -- 172.21.15.47:0/2226672757 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9968004240 con 0x55b33496d240 2024-02-19T15:23:05.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f9979304000 1 -- 172.21.15.47:0/2226672757 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b334728850 con 0x55b33496d240 2024-02-19T15:23:05.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f995cc00640 1 -- 172.21.15.47:0/2226672757 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99680043a0 con 0x55b33496d240 2024-02-19T15:23:05.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f995cc00640 1 -- 172.21.15.47:0/2226672757 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9968003e90 con 0x55b33496d240 2024-02-19T15:23:05.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f9979304000 10 client.5229 fetch_fsmap learned FSMap version 16 2024-02-19T15:23:05.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f9979304000 1 -- 172.21.15.47:0/2226672757 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55b334734e70 con 0x55b33496d240 2024-02-19T15:23:05.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f995cc00640 1 -- 172.21.15.47:0/2226672757 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b334728850 con 0x55b33496d240 2024-02-19T15:23:05.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f995cc00640 1 -- 172.21.15.47:0/2226672757 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 16) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9968004560 con 0x55b33496d240 2024-02-19T15:23:05.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f9979304000 10 client.5229 fetch_fsmap finished waiting for FSMap version 16 2024-02-19T15:23:05.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f9979304000 10 client.5229 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T15:23:05.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f9979304000 20 client.5229 populate_metadata read hostname 'smithi047' 2024-02-19T15:23:05.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f9979304000 1 --2- 172.21.15.47:0/2226672757 >> [v2:172.21.15.47:6833/3700339993,v1:172.21.15.47:6835/3700339993] conn(0x55b334955990 0x55b33496ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f9979304000 4 client.5229 mds_command: new command op to 5067 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T15:23:05.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f9979304000 1 -- 172.21.15.47:0/2226672757 --> [v2:172.21.15.47:6833/3700339993,v1:172.21.15.47:6835/3700339993] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b334771f20 con 0x55b334955990 2024-02-19T15:23:05.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f996ea00640 1 --2- 172.21.15.47:0/2226672757 >> [v2:172.21.15.47:6833/3700339993,v1:172.21.15.47:6835/3700339993] conn(0x55b334955990 0x55b33496ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f996ea00640 1 --2- 172.21.15.47:0/2226672757 >> [v2:172.21.15.47:6833/3700339993,v1:172.21.15.47:6835/3700339993] conn(0x55b334955990 0x55b33496ecc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcdf2b004b87852a server_cookie=fbd22805d53e782d in_seq=0 out_seq=0 2024-02-19T15:23:05.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.204+0000 7f995cc00640 10 client.5229 ms_handle_connect on v2:172.21.15.47:6833/3700339993 2024-02-19T15:23:05.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.208+0000 7f995cc00640 1 -- 172.21.15.47:0/2226672757 <== mds.0 v2:172.21.15.47:6833/3700339993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b334771f20 con 0x55b334955990 2024-02-19T15:23:05.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.208+0000 7f995cc00640 10 client.5229 handle_command_reply: tid=0 2024-02-19T15:23:05.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.208+0000 7f9979304000 1 client.5229 shutdown 2024-02-19T15:23:05.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.208+0000 7f9979304000 1 -- 172.21.15.47:0/2226672757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b33496d240 msgr2=0x55b33496e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:05.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.208+0000 7f9979304000 1 --2- 172.21.15.47:0/2226672757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b33496d240 0x55b33496e2b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f996801af60 tx=0x7f996801b0f0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.208+0000 7f9979304000 1 -- 172.21.15.47:0/2226672757 shutdown_connections 2024-02-19T15:23:05.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.208+0000 7f9979304000 1 --2- 172.21.15.47:0/2226672757 >> [v2:172.21.15.47:6833/3700339993,v1:172.21.15.47:6835/3700339993] conn(0x55b334955990 0x55b33496ecc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.208+0000 7f9979304000 1 --2- 172.21.15.47:0/2226672757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b33496d240 0x55b33496e2b0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.208+0000 7f9979304000 1 --2- 172.21.15.47:0/2226672757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b33496c920 0x55b33496dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.208+0000 7f995cc00640 0 client.5229 ms_handle_reset on v2:172.21.15.47:6833/3700339993 2024-02-19T15:23:05.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.208+0000 7f9979304000 1 --2- 172.21.15.47:0/2226672757 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b334922f30 0x55b33496d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.208+0000 7f9979304000 1 -- 172.21.15.47:0/2226672757 >> 172.21.15.47:0/2226672757 conn(0x55b3349292a0 msgr2=0x55b3349219e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:05.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.208+0000 7f9979304000 1 -- 172.21.15.47:0/2226672757 shutdown_connections 2024-02-19T15:23:05.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.208+0000 7f9979304000 1 -- 172.21.15.47:0/2226672757 wait complete. 2024-02-19T15:23:05.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.208+0000 7f9979304000 20 client.5229 trim_cache size 0 max 16384 2024-02-19T15:23:05.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.224+0000 7f9979304000 1 Processor -- start 2024-02-19T15:23:05.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.224+0000 7f9979304000 1 -- start start 2024-02-19T15:23:05.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.224+0000 7f9979304000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b33496e410 0x55b3349cd230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.224+0000 7f9979304000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3349292a0 0x55b3349cd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.224+0000 7f9979304000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b334922f30 0x55b3349cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.224+0000 7f9979304000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b33442c940 con 0x55b334922f30 2024-02-19T15:23:05.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.224+0000 7f9979304000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b334770d70 con 0x55b3349292a0 2024-02-19T15:23:05.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.224+0000 7f9979304000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b3347348c0 con 0x55b33496e410 2024-02-19T15:23:05.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.224+0000 7f996f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b334922f30 0x55b3349cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.224+0000 7f996f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b334922f30 0x55b3349cdcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48134/0 (socket says 172.21.15.47:48134) 2024-02-19T15:23:05.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.224+0000 7f996f400640 1 -- 172.21.15.47:0/3622054744 learned_addr learned my addr 172.21.15.47:0/3622054744 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:05.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.224+0000 7f996ea00640 1 --2- 172.21.15.47:0/3622054744 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b33496e410 0x55b3349cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.224+0000 7f9967e00640 1 --2- 172.21.15.47:0/3622054744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3349292a0 0x55b3349cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f996f400640 1 -- 172.21.15.47:0/3622054744 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b33496e410 msgr2=0x55b3349cd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:05.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f996f400640 1 --2- 172.21.15.47:0/3622054744 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b33496e410 0x55b3349cd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f996f400640 1 -- 172.21.15.47:0/3622054744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3349292a0 msgr2=0x55b3349cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:05.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f996f400640 1 --2- 172.21.15.47:0/3622054744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3349292a0 0x55b3349cd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f996f400640 1 -- 172.21.15.47:0/3622054744 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b33471ae10 con 0x55b334922f30 2024-02-19T15:23:05.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f996f400640 1 --2- 172.21.15.47:0/3622054744 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b334922f30 0x55b3349cdcb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f99680070b0 tx=0x7f99680c5040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8abfde69ca48a0dc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:05.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f995f400640 1 -- 172.21.15.47:0/3622054744 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9968003eb0 con 0x55b334922f30 2024-02-19T15:23:05.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f995f400640 1 -- 172.21.15.47:0/3622054744 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9968004010 con 0x55b334922f30 2024-02-19T15:23:05.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f995f400640 1 -- 172.21.15.47:0/3622054744 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99680043c0 con 0x55b334922f30 2024-02-19T15:23:05.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f9979304000 1 -- 172.21.15.47:0/3622054744 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b334922f30 msgr2=0x55b3349cdcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:05.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f9979304000 1 --2- 172.21.15.47:0/3622054744 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b334922f30 0x55b3349cdcb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f99680070b0 tx=0x7f99680c5040 comp rx=0 tx=0).stop 2024-02-19T15:23:05.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f9979304000 1 -- 172.21.15.47:0/3622054744 shutdown_connections 2024-02-19T15:23:05.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f9979304000 1 --2- 172.21.15.47:0/3622054744 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b334922f30 0x55b3349cdcb0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f9979304000 1 --2- 172.21.15.47:0/3622054744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3349292a0 0x55b3349cd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f9979304000 1 --2- 172.21.15.47:0/3622054744 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b33496e410 0x55b3349cd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f9979304000 1 -- 172.21.15.47:0/3622054744 >> 172.21.15.47:0/3622054744 conn(0x55b3349829f0 msgr2=0x55b334954b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:05.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f9979304000 1 -- 172.21.15.47:0/3622054744 shutdown_connections 2024-02-19T15:23:05.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f9979304000 1 -- 172.21.15.47:0/3622054744 wait complete. 2024-02-19T15:23:05.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f9979304000 1 -- start start 2024-02-19T15:23:05.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f9979304000 1 Processor -- start 2024-02-19T15:23:05.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f9979304000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b334922f30 0x55b3349d2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f9979304000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b3349292a0 0x55b3349d2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f9979304000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b33496e410 0x55b3349b5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f9979304000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b33473fc80 con 0x55b33496e410 2024-02-19T15:23:05.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f9979304000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b334773fd0 con 0x55b334922f30 2024-02-19T15:23:05.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f9979304000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b33476f2c0 con 0x55b3349292a0 2024-02-19T15:23:05.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f9979304000 10 client.0 _ll_register_callbacks cb 0x7f99680bc070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:23:05.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f996f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b33496e410 0x55b3349b5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f996f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b33496e410 0x55b3349b5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48136/0 (socket says 172.21.15.47:48136) 2024-02-19T15:23:05.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f996f400640 1 -- 172.21.15.47:0/3347779572 learned_addr learned my addr 172.21.15.47:0/3347779572 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:05.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f996ea00640 1 --2- 172.21.15.47:0/3347779572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b334922f30 0x55b3349d2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.228+0000 7f9967e00640 1 --2- 172.21.15.47:0/3347779572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b3349292a0 0x55b3349d2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f996f400640 1 -- 172.21.15.47:0/3347779572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b3349292a0 msgr2=0x55b3349d2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:05.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f996f400640 1 --2- 172.21.15.47:0/3347779572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b3349292a0 0x55b3349d2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f996f400640 1 -- 172.21.15.47:0/3347779572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b334922f30 msgr2=0x55b3349d2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:05.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f996f400640 1 --2- 172.21.15.47:0/3347779572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b334922f30 0x55b3349d2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f996f400640 1 -- 172.21.15.47:0/3347779572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b334734e70 con 0x55b33496e410 2024-02-19T15:23:05.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f996f400640 1 --2- 172.21.15.47:0/3347779572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b33496e410 0x55b3349b5e70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9968009d90 tx=0x7f99680a7070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca3dfdefaee055ab server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:05.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f995cc00640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:23:05.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f995cc00640 1 -- 172.21.15.47:0/3347779572 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99680dc070 con 0x55b33496e410 2024-02-19T15:23:05.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f995cc00640 1 -- 172.21.15.47:0/3347779572 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99680cc440 con 0x55b33496e410 2024-02-19T15:23:05.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f9979304000 1 -- 172.21.15.47:0/3347779572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b3346fae80 con 0x55b33496e410 2024-02-19T15:23:05.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f995cc00640 1 -- 172.21.15.47:0/3347779572 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99680046a0 con 0x55b33496e410 2024-02-19T15:23:05.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f9979304000 10 client.5235 fetch_fsmap learned FSMap version 16 2024-02-19T15:23:05.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f9979304000 1 -- 172.21.15.47:0/3347779572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55b33446bcc0 con 0x55b33496e410 2024-02-19T15:23:05.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f995cc00640 1 -- 172.21.15.47:0/3347779572 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b3346fae80 con 0x55b33496e410 2024-02-19T15:23:05.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f995cc00640 1 -- 172.21.15.47:0/3347779572 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 16) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f99680df020 con 0x55b33496e410 2024-02-19T15:23:05.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f9979304000 10 client.5235 fetch_fsmap finished waiting for FSMap version 16 2024-02-19T15:23:05.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f9979304000 10 client.5235 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T15:23:05.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f9979304000 20 client.5235 populate_metadata read hostname 'smithi047' 2024-02-19T15:23:05.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f9979304000 1 --2- 172.21.15.47:0/3347779572 >> [v2:172.21.15.47:6833/3700339993,v1:172.21.15.47:6835/3700339993] conn(0x55b33496c920 0x55b3349b63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f9979304000 4 client.5235 mds_command: new command op to 5067 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T15:23:05.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f9979304000 1 -- 172.21.15.47:0/3347779572 --> [v2:172.21.15.47:6833/3700339993,v1:172.21.15.47:6835/3700339993] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b33473dfa0 con 0x55b33496c920 2024-02-19T15:23:05.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f996ea00640 1 --2- 172.21.15.47:0/3347779572 >> [v2:172.21.15.47:6833/3700339993,v1:172.21.15.47:6835/3700339993] conn(0x55b33496c920 0x55b3349b63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f996ea00640 1 --2- 172.21.15.47:0/3347779572 >> [v2:172.21.15.47:6833/3700339993,v1:172.21.15.47:6835/3700339993] conn(0x55b33496c920 0x55b3349b63b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a7f14c5705dd0e4 server_cookie=168b244b37cb791a in_seq=0 out_seq=0 2024-02-19T15:23:05.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f995cc00640 10 client.5235 ms_handle_connect on v2:172.21.15.47:6833/3700339993 2024-02-19T15:23:05.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f995cc00640 1 -- 172.21.15.47:0/3347779572 <== mds.0 v2:172.21.15.47:6833/3700339993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55b33473dfa0 con 0x55b33496c920 2024-02-19T15:23:05.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f995cc00640 10 client.5235 handle_command_reply: tid=0 2024-02-19T15:23:05.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.232+0000 7f9979304000 1 client.5235 shutdown 2024-02-19T15:23:05.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9979304000 1 -- 172.21.15.47:0/3347779572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b33496e410 msgr2=0x55b3349b5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:05.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9979304000 1 --2- 172.21.15.47:0/3347779572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b33496e410 0x55b3349b5e70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9968009d90 tx=0x7f99680a7070 comp rx=0 tx=0).stop 2024-02-19T15:23:05.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9979304000 1 -- 172.21.15.47:0/3347779572 shutdown_connections 2024-02-19T15:23:05.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9979304000 1 --2- 172.21.15.47:0/3347779572 >> [v2:172.21.15.47:6833/3700339993,v1:172.21.15.47:6835/3700339993] conn(0x55b33496c920 0x55b3349b63b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9979304000 1 --2- 172.21.15.47:0/3347779572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b33496e410 0x55b3349b5e70 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9979304000 1 --2- 172.21.15.47:0/3347779572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55b3349292a0 0x55b3349d2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9979304000 1 --2- 172.21.15.47:0/3347779572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b334922f30 0x55b3349d2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f995cc00640 0 client.5235 ms_handle_reset on v2:172.21.15.47:6833/3700339993 2024-02-19T15:23:05.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9979304000 1 -- 172.21.15.47:0/3347779572 >> 172.21.15.47:0/3347779572 conn(0x55b3349829f0 msgr2=0x55b334955230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:05.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9979304000 1 -- 172.21.15.47:0/3347779572 shutdown_connections 2024-02-19T15:23:05.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9979304000 1 -- 172.21.15.47:0/3347779572 wait complete. 2024-02-19T15:23:05.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9979304000 20 client.5235 trim_cache size 0 max 16384 2024-02-19T15:23:05.241 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:23:05.241 INFO:teuthology.orchestra.run.smithi047.stdout: "cluster_fsid": "ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e", 2024-02-19T15:23:05.242 INFO:teuthology.orchestra.run.smithi047.stdout: "whoami": 0, 2024-02-19T15:23:05.242 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 5067, 2024-02-19T15:23:05.242 INFO:teuthology.orchestra.run.smithi047.stdout: "want_state": "up:active", 2024-02-19T15:23:05.242 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "up:active", 2024-02-19T15:23:05.242 INFO:teuthology.orchestra.run.smithi047.stdout: "fs_name": "cephfs", 2024-02-19T15:23:05.242 INFO:teuthology.orchestra.run.smithi047.stdout: "rank_uptime": 6.0254441170000002, 2024-02-19T15:23:05.242 INFO:teuthology.orchestra.run.smithi047.stdout: "mdsmap_epoch": 16, 2024-02-19T15:23:05.242 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch": 34, 2024-02-19T15:23:05.242 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch_barrier": 34, 2024-02-19T15:23:05.242 INFO:teuthology.orchestra.run.smithi047.stdout: "uptime": 13.897653585 2024-02-19T15:23:05.242 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:23:05.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9975e00640 1 -- 172.21.15.47:0/1553643873 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f994c066e40 msgr2=0x7f994c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:05.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9975e00640 1 --2- 172.21.15.47:0/1553643873 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f994c066e40 0x7f994c069300 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f99600079c0 tx=0x7f9960008040 comp rx=0 tx=0).stop 2024-02-19T15:23:05.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9975e00640 1 -- 172.21.15.47:0/1553643873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99700ceb00 msgr2=0x7f99700af460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:05.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9975e00640 1 --2- 172.21.15.47:0/1553643873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99700ceb00 0x7f99700af460 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f995800c430 tx=0x7f995800c8f0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9975e00640 1 -- 172.21.15.47:0/1553643873 shutdown_connections 2024-02-19T15:23:05.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9975e00640 1 --2- 172.21.15.47:0/1553643873 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f994c066e40 0x7f994c069300 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9975e00640 1 --2- 172.21.15.47:0/1553643873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99700cef90 0x7f99700c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9975e00640 1 --2- 172.21.15.47:0/1553643873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99700ceb00 0x7f99700af460 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9975e00640 1 --2- 172.21.15.47:0/1553643873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f99700b2fc0 0x7f99700b6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9975e00640 1 -- 172.21.15.47:0/1553643873 >> 172.21.15.47:0/1553643873 conn(0x7f99700d1b70 msgr2=0x7f99700ce3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:05.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9975e00640 1 -- 172.21.15.47:0/1553643873 shutdown_connections 2024-02-19T15:23:05.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.236+0000 7f9975e00640 1 -- 172.21.15.47:0/1553643873 wait complete. 2024-02-19T15:23:05.250 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:23:05.254 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:23:05.261 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-02-19T15:23:05.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.472+0000 7fc7da200640 1 Processor -- start 2024-02-19T15:23:05.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.472+0000 7fc7da200640 1 -- start start 2024-02-19T15:23:05.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7da200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7d40cedb0 0x7fc7d40cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7da200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d40cc700 con 0x7fc7d40d2710 2024-02-19T15:23:05.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7da200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d40cf300 con 0x7fc7d415bcd0 2024-02-19T15:23:05.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7da200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d40cf5e0 con 0x7fc7d40cedb0 2024-02-19T15:23:05.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d3e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc7d40d2710 0x7fc7d415d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43002/0 (socket says 172.21.15.47:43002) 2024-02-19T15:23:05.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d3e00640 1 -- 172.21.15.47:0/1948685310 learned_addr learned my addr 172.21.15.47:0/1948685310 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:05.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7cbe00640 1 --2- 172.21.15.47:0/1948685310 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7d40cedb0 0x7fc7d40cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d2a00640 1 -- 172.21.15.47:0/1948685310 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290506931 0 0) 0x7fc7d40cc700 con 0x7fc7d40d2710 2024-02-19T15:23:05.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d2a00640 1 -- 172.21.15.47:0/1948685310 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7b8003530 con 0x7fc7d40d2710 2024-02-19T15:23:05.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d2a00640 1 -- 172.21.15.47:0/1948685310 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25329635 0 0) 0x7fc7d40cf300 con 0x7fc7d415bcd0 2024-02-19T15:23:05.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d2a00640 1 -- 172.21.15.47:0/1948685310 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7d40cc700 con 0x7fc7d415bcd0 2024-02-19T15:23:05.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d2a00640 1 -- 172.21.15.47:0/1948685310 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2215383308 0 0) 0x7fc7b8003530 con 0x7fc7d40d2710 2024-02-19T15:23:05.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d2a00640 1 -- 172.21.15.47:0/1948685310 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7d40cedb0 msgr2=0x7fc7d40cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:05.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d2a00640 1 --2- 172.21.15.47:0/1948685310 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7d40cedb0 0x7fc7d40cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d2a00640 1 -- 172.21.15.47:0/1948685310 >> v1:172.21.15.116:6789/0 conn(0x7fc7d415bcd0 legacy=0x7fc7d40cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:05.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d2a00640 1 -- 172.21.15.47:0/1948685310 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d40cc9a0 con 0x7fc7d40d2710 2024-02-19T15:23:05.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d2a00640 1 -- 172.21.15.47:0/1948685310 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7fc7cc001a00 con 0x7fc7d40d2710 2024-02-19T15:23:05.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d2a00640 1 -- 172.21.15.47:0/1948685310 >> v1:172.21.15.47:6789/0 conn(0x7fc7d40d2710 legacy=0x7fc7d415d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:05.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d2a00640 1 --2- 172.21.15.47:0/1948685310 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7b8003b40 0x7fc7b8003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d2a00640 1 --2- 172.21.15.47:0/1948685310 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7b80047d0 0x7fc7b8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d2a00640 1 --2- 172.21.15.47:0/1948685310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7b8005190 0x7fc7b800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d2a00640 1 -- 172.21.15.47:0/1948685310 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d40cf5e0 con 0x7fc7b8003b40 2024-02-19T15:23:05.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d2a00640 1 -- 172.21.15.47:0/1948685310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b800bf80 con 0x7fc7b8005190 2024-02-19T15:23:05.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d2a00640 1 -- 172.21.15.47:0/1948685310 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b800c140 con 0x7fc7b80047d0 2024-02-19T15:23:05.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7cbe00640 1 --2- 172.21.15.47:0/1948685310 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7d40cedb0 0x7fc7d40cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:23:05.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d3400640 1 --2- 172.21.15.47:0/1948685310 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7b8003b40 0x7fc7b8003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d3e00640 1 --2- 172.21.15.47:0/1948685310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7b8005190 0x7fc7b800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7cbe00640 1 --2- 172.21.15.47:0/1948685310 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7b80047d0 0x7fc7b8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d3400640 1 -- 172.21.15.47:0/1948685310 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7b80047d0 msgr2=0x7fc7b8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:05.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d3400640 1 --2- 172.21.15.47:0/1948685310 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7b80047d0 0x7fc7b8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d3400640 1 -- 172.21.15.47:0/1948685310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7b8005190 msgr2=0x7fc7b800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:05.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d3400640 1 --2- 172.21.15.47:0/1948685310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7b8005190 0x7fc7b800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.476+0000 7fc7d3400640 1 -- 172.21.15.47:0/1948685310 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b8003530 con 0x7fc7b8003b40 2024-02-19T15:23:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.480+0000 7fc7d3400640 1 --2- 172.21.15.47:0/1948685310 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7b8003b40 0x7fc7b8003f40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc7c40018c0 tx=0x7fc7c401c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=123c319612c66bb7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.480+0000 7fc7d2a00640 1 -- 172.21.15.47:0/1948685310 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c40052d0 con 0x7fc7b8003b40 2024-02-19T15:23:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.480+0000 7fc7d2a00640 1 -- 172.21.15.47:0/1948685310 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c4005430 con 0x7fc7b8003b40 2024-02-19T15:23:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.480+0000 7fc7d2a00640 1 -- 172.21.15.47:0/1948685310 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4003820 con 0x7fc7b8003b40 2024-02-19T15:23:05.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.480+0000 7fc7da200640 1 -- 172.21.15.47:0/1948685310 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7b8003b40 msgr2=0x7fc7b8003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:05.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.480+0000 7fc7da200640 1 --2- 172.21.15.47:0/1948685310 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7b8003b40 0x7fc7b8003f40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc7c40018c0 tx=0x7fc7c401c550 comp rx=0 tx=0).stop 2024-02-19T15:23:05.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.480+0000 7fc7d3e00640 1 -- 172.21.15.47:0/1948685310 reap_dead start 2024-02-19T15:23:05.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.480+0000 7fc7da200640 1 -- 172.21.15.47:0/1948685310 shutdown_connections 2024-02-19T15:23:05.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.480+0000 7fc7da200640 1 -- 172.21.15.47:0/1948685310 >> 172.21.15.47:0/1948685310 conn(0x7fc7d40d1bf0 msgr2=0x7fc7d40d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:05.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.480+0000 7fc7da200640 1 -- 172.21.15.47:0/1948685310 shutdown_connections 2024-02-19T15:23:05.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.480+0000 7fc7da200640 1 -- 172.21.15.47:0/1948685310 wait complete. 2024-02-19T15:23:05.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.480+0000 7fc7da200640 1 Processor -- start 2024-02-19T15:23:05.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7da200640 1 -- start start 2024-02-19T15:23:05.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7da200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7b8003b40 0x7fc7d414f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7da200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7d40c7160 0x7fc7d415c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7da200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7d40caea0 0x7fc7d415c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7da200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d40cdb90 con 0x7fc7d40caea0 2024-02-19T15:23:05.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7da200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d40c9e40 con 0x7fc7b8003b40 2024-02-19T15:23:05.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7da200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d40ca0e0 con 0x7fc7d40c7160 2024-02-19T15:23:05.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7d3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7d40caea0 0x7fc7d415c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7d3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7d40caea0 0x7fc7d415c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48142/0 (socket says 172.21.15.47:48142) 2024-02-19T15:23:05.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7d3e00640 1 -- 172.21.15.47:0/2864607819 learned_addr learned my addr 172.21.15.47:0/2864607819 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:05.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7d3400640 1 --2- 172.21.15.47:0/2864607819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7b8003b40 0x7fc7d414f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7cbe00640 1 --2- 172.21.15.47:0/2864607819 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7d40c7160 0x7fc7d415c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7d3e00640 1 -- 172.21.15.47:0/2864607819 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7d40c7160 msgr2=0x7fc7d415c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:05.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7d3e00640 1 --2- 172.21.15.47:0/2864607819 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7d40c7160 0x7fc7d415c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7d3e00640 1 -- 172.21.15.47:0/2864607819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7b8003b40 msgr2=0x7fc7d414f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:05.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7d3e00640 1 --2- 172.21.15.47:0/2864607819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7b8003b40 0x7fc7d414f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7d3e00640 1 -- 172.21.15.47:0/2864607819 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d415d240 con 0x7fc7d40caea0 2024-02-19T15:23:05.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7d3e00640 1 --2- 172.21.15.47:0/2864607819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7d40caea0 0x7fc7d415c5f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc000f90 tx=0x7fc7cc00a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:05.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7cb400640 1 -- 172.21.15.47:0/2864607819 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc00ad00 con 0x7fc7d40caea0 2024-02-19T15:23:05.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7cb400640 1 -- 172.21.15.47:0/2864607819 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7cc00ae60 con 0x7fc7d40caea0 2024-02-19T15:23:05.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7cb400640 1 -- 172.21.15.47:0/2864607819 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc013630 con 0x7fc7d40caea0 2024-02-19T15:23:05.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7da200640 1 -- 172.21.15.47:0/2864607819 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d40ca3c0 con 0x7fc7d40caea0 2024-02-19T15:23:05.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7da200640 1 -- 172.21.15.47:0/2864607819 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d40ca980 con 0x7fc7d40caea0 2024-02-19T15:23:05.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.484+0000 7fc7da200640 1 -- 172.21.15.47:0/2864607819 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d40cab70 con 0x7fc7d40caea0 2024-02-19T15:23:05.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.488+0000 7fc7cb400640 1 -- 172.21.15.47:0/2864607819 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc7cc013790 con 0x7fc7d40caea0 2024-02-19T15:23:05.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.488+0000 7fc7cb400640 1 --2- 172.21.15.47:0/2864607819 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc7b0066e70 0x7fc7b0069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:05.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.488+0000 7fc7cb400640 1 -- 172.21.15.47:0/2864607819 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7cc084fe0 con 0x7fc7d40caea0 2024-02-19T15:23:05.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.488+0000 7fc7d3400640 1 --2- 172.21.15.47:0/2864607819 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc7b0066e70 0x7fc7b0069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:05.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.488+0000 7fc7d3400640 1 --2- 172.21.15.47:0/2864607819 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc7b0066e70 0x7fc7b0069330 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc7c40ad000 tx=0x7fc7c4001350 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:05.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.496+0000 7fc7cb400640 1 -- 172.21.15.47:0/2864607819 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc7cc051e20 con 0x7fc7d40caea0 2024-02-19T15:23:05.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.880+0000 7fc7da200640 1 -- 172.21.15.47:0/2864607819 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc7d4075360 con 0x7fc7d40caea0 2024-02-19T15:23:05.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.880+0000 7fc7cb400640 1 -- 172.21.15.47:0/2864607819 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc7cc0554e0 con 0x7fc7d40caea0 2024-02-19T15:23:05.885 INFO:teuthology.orchestra.run.smithi047.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-02-19T15:23:05.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.884+0000 7fc7da200640 1 -- 172.21.15.47:0/2864607819 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc7b0066e70 msgr2=0x7fc7b0069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:05.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.884+0000 7fc7da200640 1 --2- 172.21.15.47:0/2864607819 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc7b0066e70 0x7fc7b0069330 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc7c40ad000 tx=0x7fc7c4001350 comp rx=0 tx=0).stop 2024-02-19T15:23:05.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.884+0000 7fc7da200640 1 -- 172.21.15.47:0/2864607819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7d40caea0 msgr2=0x7fc7d415c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:05.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.884+0000 7fc7da200640 1 --2- 172.21.15.47:0/2864607819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7d40caea0 0x7fc7d415c5f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc000f90 tx=0x7fc7cc00a6a0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.884+0000 7fc7da200640 1 -- 172.21.15.47:0/2864607819 shutdown_connections 2024-02-19T15:23:05.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.884+0000 7fc7da200640 1 --2- 172.21.15.47:0/2864607819 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc7b0066e70 0x7fc7b0069330 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.884+0000 7fc7da200640 1 --2- 172.21.15.47:0/2864607819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7d40caea0 0x7fc7d415c5f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.884+0000 7fc7da200640 1 --2- 172.21.15.47:0/2864607819 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7d40c7160 0x7fc7d415c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.884+0000 7fc7da200640 1 --2- 172.21.15.47:0/2864607819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7b8003b40 0x7fc7d414f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:05.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.884+0000 7fc7da200640 1 -- 172.21.15.47:0/2864607819 >> 172.21.15.47:0/2864607819 conn(0x7fc7d40d1bf0 msgr2=0x7fc7d4070370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:05.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.884+0000 7fc7da200640 1 -- 172.21.15.47:0/2864607819 shutdown_connections 2024-02-19T15:23:05.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:05.884+0000 7fc7da200640 1 -- 172.21.15.47:0/2864607819 wait complete. 2024-02-19T15:23:05.913 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-02-19T15:23:05.913 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T15:23:05.913 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi047.front.sepia.ceph.com 2024-02-19T15:23:05.913 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-19T15:23:05.913 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-19T15:23:05.913 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-02-19T15:23:05.913 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-02-19T15:23:05.913 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T15:23:05.913 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:23:05.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:05.913 DEBUG:teuthology.orchestra.run.smithi047:> ip addr 2024-02-19T15:23:05.923 INFO:teuthology.orchestra.run.smithi047.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T15:23:05.923 INFO:teuthology.orchestra.run.smithi047.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T15:23:05.923 INFO:teuthology.orchestra.run.smithi047.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T15:23:05.923 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:23:05.923 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 ::1/128 scope host 2024-02-19T15:23:05.923 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:23:05.923 INFO:teuthology.orchestra.run.smithi047.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:23:05.923 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:78 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:23:05.923 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f0 2024-02-19T15:23:05.923 INFO:teuthology.orchestra.run.smithi047.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T15:23:05.923 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8e brd ff:ff:ff:ff:ff:ff 2024-02-19T15:23:05.923 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f0 2024-02-19T15:23:05.924 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.21.15.47/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-02-19T15:23:05.924 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft 42429sec preferred_lft 42429sec 2024-02-19T15:23:05.924 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::ec4:7aff:febd:148e/64 scope link 2024-02-19T15:23:05.924 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:23:05.924 INFO:teuthology.orchestra.run.smithi047.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:23:05.924 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:79 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:23:05.924 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f1 2024-02-19T15:23:05.924 INFO:teuthology.orchestra.run.smithi047.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:23:05.924 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8f brd ff:ff:ff:ff:ff:ff 2024-02-19T15:23:05.924 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f1 2024-02-19T15:23:05.924 INFO:teuthology.orchestra.run.smithi047.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-02-19T15:23:05.924 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 02:42:47:cf:b1:62 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:23:05.924 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-02-19T15:23:05.925 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:23:05.925 INFO:teuthology.orchestra.run.smithi047.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-02-19T15:23:05.925 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-02-19T15:23:05.925 INFO:teuthology.orchestra.run.smithi047.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-02-19T15:23:05.925 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:23:05.925 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-02-19T15:23:05.925 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:23:05.925 INFO:teuthology.orchestra.run.smithi047.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:23:05.925 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-19T15:23:05.925 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-02-19T15:23:05.925 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:23:05.925 INFO:teuthology.orchestra.run.smithi047.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:23:05.925 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-02-19T15:23:05.925 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-02-19T15:23:05.925 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:23:05.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:05.926 DEBUG:teuthology.orchestra.run.smithi047:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T15:23:05.987 INFO:teuthology.orchestra.run.smithi047.stdout:1 2024-02-19T15:23:05.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:05.990 DEBUG:teuthology.orchestra.run.smithi047:> route 2024-02-19T15:23:06.039 INFO:teuthology.orchestra.run.smithi047.stdout:Kernel IP routing table 2024-02-19T15:23:06.039 INFO:teuthology.orchestra.run.smithi047.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T15:23:06.039 INFO:teuthology.orchestra.run.smithi047.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-02-19T15:23:06.039 INFO:teuthology.orchestra.run.smithi047.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-02-19T15:23:06.039 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-02-19T15:23:06.039 INFO:teuthology.orchestra.run.smithi047.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T15:23:06.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:06.039 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2024-02-19T15:23:06.040 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2024-02-19T15:23:06.040 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-02-19T15:23:06.040 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-02-19T15:23:06.040 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-02-19T15:23:06.040 DEBUG:teuthology.orchestra.run.smithi047:> ') 2024-02-19T15:23:06.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:06.132 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list 2024-02-19T15:23:06.140 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-02-19T15:23:06.140 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-02-19T15:23:06.141 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:23:06.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:06.141 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set brx.0 up 2024-02-19T15:23:06.206 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:23:06.206 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-19T15:23:06.256 INFO:teuthology.orchestra.run.smithi047.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:23:06.256 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:23:06.256 DEBUG:teuthology.orchestra.run.smithi047:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:23:06.306 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-19T15:23:06.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:06.306 DEBUG:teuthology.orchestra.run.smithi047:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:23:06.394 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:23:06.394 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:23:06.394 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:23:06.409 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:23:06.409 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:23:06.409 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:23:06.410 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:23:06.410 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:23:06.418 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:23:06.419 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:23:06.419 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:23:06.419 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:23:06.419 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:23:06.419 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790" 2024-02-19T15:23:06.419 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790:/ 2024-02-19T15:23:06.419 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e" will pass to kernel 2024-02-19T15:23:06.422 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T15:23:06.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:06.423 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:23:06.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:06.443 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:23:06.505 INFO:teuthology.orchestra.run.smithi047.stdout:client5244 2024-02-19T15:23:06.505 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:23:06.505 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:23:06.554 DEBUG:teuthology.orchestra.run.smithi047:> stdin-killer -- python3 -c ' 2024-02-19T15:23:06.555 DEBUG:teuthology.orchestra.run.smithi047:> import os 2024-02-19T15:23:06.555 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:23:06.555 DEBUG:teuthology.orchestra.run.smithi047:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-02-19T15:23:06.555 DEBUG:teuthology.orchestra.run.smithi047:> subdir = "delete_me" 2024-02-19T15:23:06.555 DEBUG:teuthology.orchestra.run.smithi047:> size = 1024 2024-02-19T15:23:06.555 DEBUG:teuthology.orchestra.run.smithi047:> file_count = 1000 2024-02-19T15:23:06.555 DEBUG:teuthology.orchestra.run.smithi047:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-02-19T15:23:06.555 DEBUG:teuthology.orchestra.run.smithi047:> for i in range(0, file_count): 2024-02-19T15:23:06.555 DEBUG:teuthology.orchestra.run.smithi047:> filename = "{0}_{1}.bin".format(i, size) 2024-02-19T15:23:06.555 DEBUG:teuthology.orchestra.run.smithi047:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-02-19T15:23:06.555 DEBUG:teuthology.orchestra.run.smithi047:> f.write(size * '"'"'x'"'"') 2024-02-19T15:23:06.555 DEBUG:teuthology.orchestra.run.smithi047:> ' 2024-02-19T15:23:07.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-02-19T15:23:07.667 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:23:07.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.848+0000 7f40c3e00640 1 Processor -- start 2024-02-19T15:23:07.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.848+0000 7f40c3e00640 1 -- start start 2024-02-19T15:23:07.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.852+0000 7f40c3e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f40c40acec0 0x7f40c406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:07.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.852+0000 7f40c3e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40c40cdf90 0x7f40c40ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:07.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.852+0000 7f40c3e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c406e290 con 0x7f40c406e470 2024-02-19T15:23:07.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.852+0000 7f40c3e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c409e010 con 0x7f40c40acec0 2024-02-19T15:23:07.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.852+0000 7f40c3e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c409dd30 con 0x7f40c40cdf90 2024-02-19T15:23:07.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.852+0000 7f40c2000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f40c406e470 0x7f40c40ad2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43004/0 (socket says 172.21.15.47:43004) 2024-02-19T15:23:07.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.852+0000 7f40c2000640 1 -- 172.21.15.47:0/4257210631 learned_addr learned my addr 172.21.15.47:0/4257210631 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:07.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.852+0000 7f40c3400640 1 --2- 172.21.15.47:0/4257210631 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40c40cdf90 0x7f40c40ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:07.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.852+0000 7f40c2a00640 1 --2- 172.21.15.47:0/4257210631 >> v2:172.21.15.116:3300/0 conn(0x7f40c40acec0 0x7f40c406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:07.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.852+0000 7f40c1600640 1 -- 172.21.15.47:0/4257210631 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264831262 0 0) 0x7f40c406e290 con 0x7f40c406e470 2024-02-19T15:23:07.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.852+0000 7f40c1600640 1 -- 172.21.15.47:0/4257210631 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40a8003530 con 0x7f40c406e470 2024-02-19T15:23:07.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.852+0000 7f40c3400640 1 -- 172.21.15.47:0/4257210631 >> v2:172.21.15.116:3300/0 conn(0x7f40c40acec0 msgr2=0x7f40c406dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:07.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.852+0000 7f40c3400640 1 --2- 172.21.15.47:0/4257210631 >> v2:172.21.15.116:3300/0 conn(0x7f40c40acec0 0x7f40c406dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:07.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.852+0000 7f40c3400640 1 -- 172.21.15.47:0/4257210631 >> v1:172.21.15.47:6789/0 conn(0x7f40c406e470 legacy=0x7f40c40ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:07.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.852+0000 7f40c1600640 1 -- 172.21.15.47:0/4257210631 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 169067340 0 0) 0x7f40a8003530 con 0x7f40c406e470 2024-02-19T15:23:07.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.852+0000 7f40c3400640 1 -- 172.21.15.47:0/4257210631 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c406cf90 con 0x7f40c40cdf90 2024-02-19T15:23:07.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.852+0000 7f40c2a00640 1 --2- 172.21.15.47:0/4257210631 >> v2:172.21.15.116:3300/0 conn(0x7f40c40acec0 0x7f40c406dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:23:07.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.856+0000 7f40c3400640 1 --2- 172.21.15.47:0/4257210631 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40c40cdf90 0x7f40c40ce410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f40bc009d90 tx=0x7f40bc01c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88bee00bb3d4fda2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:07.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.856+0000 7f40c1600640 1 -- 172.21.15.47:0/4257210631 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b00016a0 con 0x7f40c40cdf90 2024-02-19T15:23:07.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.856+0000 7f40c1600640 1 -- 172.21.15.47:0/4257210631 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40bc0a6480 con 0x7f40c40cdf90 2024-02-19T15:23:07.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.856+0000 7f40c1600640 1 -- 172.21.15.47:0/4257210631 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc0a6810 con 0x7f40c40cdf90 2024-02-19T15:23:07.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.856+0000 7f40c3e00640 1 -- 172.21.15.47:0/4257210631 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40c40cdf90 msgr2=0x7f40c40ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:07.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.856+0000 7f40c3e00640 1 --2- 172.21.15.47:0/4257210631 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40c40cdf90 0x7f40c40ce410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f40bc009d90 tx=0x7f40bc01c860 comp rx=0 tx=0).stop 2024-02-19T15:23:07.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.856+0000 7f40c3e00640 1 -- 172.21.15.47:0/4257210631 shutdown_connections 2024-02-19T15:23:07.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.856+0000 7f40c3e00640 1 --2- 172.21.15.47:0/4257210631 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40c40cdf90 0x7f40c40ce410 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:07.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.856+0000 7f40c3e00640 1 --2- 172.21.15.47:0/4257210631 >> v2:172.21.15.116:3300/0 conn(0x7f40c40acec0 0x7f40c406dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:07.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.856+0000 7f40c3e00640 1 -- 172.21.15.47:0/4257210631 >> 172.21.15.47:0/4257210631 conn(0x7f40c40b1be0 msgr2=0x7f40c40b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:07.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.856+0000 7f40c3e00640 1 -- 172.21.15.47:0/4257210631 shutdown_connections 2024-02-19T15:23:07.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.856+0000 7f40c3e00640 1 -- 172.21.15.47:0/4257210631 wait complete. 2024-02-19T15:23:07.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.860+0000 7f40c3e00640 1 Processor -- start 2024-02-19T15:23:07.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.860+0000 7f40c3e00640 1 -- start start 2024-02-19T15:23:07.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40c3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40c406e470 0x7f40c40c8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:07.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40c3e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40c40acec0 0x7f40c40c5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:07.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40c3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40c40cdf90 0x7f40c415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:07.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40c3e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c40c5430 con 0x7f40c406e470 2024-02-19T15:23:07.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40c3e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c4153ea0 con 0x7f40c40cdf90 2024-02-19T15:23:07.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40c3e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c40c98f0 con 0x7f40c40acec0 2024-02-19T15:23:07.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40c2a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40c406e470 0x7f40c40c8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:07.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40c2a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40c406e470 0x7f40c40c8f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48148/0 (socket says 172.21.15.47:48148) 2024-02-19T15:23:07.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40c2a00640 1 -- 172.21.15.47:0/3363777339 learned_addr learned my addr 172.21.15.47:0/3363777339 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:07.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40c3400640 1 --2- 172.21.15.47:0/3363777339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40c40cdf90 0x7f40c415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:07.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40c2000640 1 --2- 172.21.15.47:0/3363777339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40c40acec0 0x7f40c40c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:07.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40c2a00640 1 -- 172.21.15.47:0/3363777339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40c40acec0 msgr2=0x7f40c40c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:07.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40c2a00640 1 --2- 172.21.15.47:0/3363777339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40c40acec0 0x7f40c40c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:07.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40c2a00640 1 -- 172.21.15.47:0/3363777339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40c40cdf90 msgr2=0x7f40c415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:07.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40c2a00640 1 --2- 172.21.15.47:0/3363777339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40c40cdf90 0x7f40c415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:07.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40c2a00640 1 -- 172.21.15.47:0/3363777339 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b4009420 con 0x7f40c406e470 2024-02-19T15:23:07.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40c2a00640 1 --2- 172.21.15.47:0/3363777339 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40c406e470 0x7f40c40c8f30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f40b400ebe0 tx=0x7f40b400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:07.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40baa00640 1 -- 172.21.15.47:0/3363777339 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b400eee0 con 0x7f40c406e470 2024-02-19T15:23:07.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40baa00640 1 -- 172.21.15.47:0/3363777339 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b400cbd0 con 0x7f40c406e470 2024-02-19T15:23:07.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40c3e00640 1 -- 172.21.15.47:0/3363777339 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40bc009a70 con 0x7f40c406e470 2024-02-19T15:23:07.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40baa00640 1 -- 172.21.15.47:0/3363777339 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b4013470 con 0x7f40c406e470 2024-02-19T15:23:07.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40c3e00640 1 -- 172.21.15.47:0/3363777339 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c40c4d70 con 0x7f40c406e470 2024-02-19T15:23:07.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.864+0000 7f40c3e00640 1 -- 172.21.15.47:0/3363777339 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c40c56a0 con 0x7f40c406e470 2024-02-19T15:23:07.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.868+0000 7f40baa00640 1 -- 172.21.15.47:0/3363777339 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f40b40135d0 con 0x7f40c406e470 2024-02-19T15:23:07.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.868+0000 7f40baa00640 1 --2- 172.21.15.47:0/3363777339 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f40a4066e70 0x7f40a4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:07.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.868+0000 7f40baa00640 1 -- 172.21.15.47:0/3363777339 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40b4084be0 con 0x7f40c406e470 2024-02-19T15:23:07.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.868+0000 7f40c2000640 1 --2- 172.21.15.47:0/3363777339 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f40a4066e70 0x7f40a4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:07.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.868+0000 7f40c2000640 1 --2- 172.21.15.47:0/3363777339 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f40a4066e70 0x7f40a4069330 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f40b0001330 tx=0x7f40b00012c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:07.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:07.876+0000 7f40baa00640 1 -- 172.21.15.47:0/3363777339 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f40b4051a70 con 0x7f40c406e470 2024-02-19T15:23:08.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.072+0000 7f40c3e00640 1 -- 172.21.15.47:0/3363777339 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40c40c4bb0 con 0x7f40c406e470 2024-02-19T15:23:08.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.072+0000 7f40baa00640 1 -- 172.21.15.47:0/3363777339 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f40b4055130 con 0x7f40c406e470 2024-02-19T15:23:08.076 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:08.076 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1616905430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1616905430},{"type":"v1","addr":"172.21.15.116:6836","nonce":1616905430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4814,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2845273225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2845273225},{"type":"v1","addr":"172.21.15.116:6837","nonce":2845273225}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5064,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/20043732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":20043732},{"type":"v1","addr":"172.21.15.47:6834","nonce":20043732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:22:58.590074+0000","modified":"2024-02-19T15:23:01.656160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3700339993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3700339993},{"type":"v1","addr":"172.21.15.47:6835","nonce":3700339993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T15:23:08.076 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 16 2024-02-19T15:23:08.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.072+0000 7f40c3e00640 1 -- 172.21.15.47:0/3363777339 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f40a4066e70 msgr2=0x7f40a4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:08.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.072+0000 7f40c3e00640 1 --2- 172.21.15.47:0/3363777339 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f40a4066e70 0x7f40a4069330 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f40b0001330 tx=0x7f40b00012c0 comp rx=0 tx=0).stop 2024-02-19T15:23:08.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.072+0000 7f40c3e00640 1 -- 172.21.15.47:0/3363777339 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40c406e470 msgr2=0x7f40c40c8f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:08.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.072+0000 7f40c3e00640 1 --2- 172.21.15.47:0/3363777339 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40c406e470 0x7f40c40c8f30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f40b400ebe0 tx=0x7f40b400c520 comp rx=0 tx=0).stop 2024-02-19T15:23:08.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.072+0000 7f40c3e00640 1 -- 172.21.15.47:0/3363777339 shutdown_connections 2024-02-19T15:23:08.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.072+0000 7f40c3e00640 1 --2- 172.21.15.47:0/3363777339 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f40a4066e70 0x7f40a4069330 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:08.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.072+0000 7f40c3e00640 1 --2- 172.21.15.47:0/3363777339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40c40cdf90 0x7f40c415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:08.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.072+0000 7f40c3e00640 1 --2- 172.21.15.47:0/3363777339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40c40acec0 0x7f40c40c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:08.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.072+0000 7f40c3e00640 1 --2- 172.21.15.47:0/3363777339 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40c406e470 0x7f40c40c8f30 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:08.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.072+0000 7f40c3e00640 1 -- 172.21.15.47:0/3363777339 >> 172.21.15.47:0/3363777339 conn(0x7f40c40b1be0 msgr2=0x7f40c4074120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:08.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.072+0000 7f40c3e00640 1 -- 172.21.15.47:0/3363777339 shutdown_connections 2024-02-19T15:23:08.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.072+0000 7f40c3e00640 1 -- 172.21.15.47:0/3363777339 wait complete. 2024-02-19T15:23:08.097 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-02-19T15:23:08.613 INFO:teuthology.orchestra.run.smithi047.stdout:{"message":"","return_code":0} 2024-02-19T15:23:08.614 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-02-19T15:23:08.614 DEBUG:teuthology.orchestra.run.smithi047:> stdin-killer -- python3 -c ' 2024-02-19T15:23:08.614 DEBUG:teuthology.orchestra.run.smithi047:> import os 2024-02-19T15:23:08.614 DEBUG:teuthology.orchestra.run.smithi047:> import stat 2024-02-19T15:23:08.615 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:23:08.615 DEBUG:teuthology.orchestra.run.smithi047:> print(os.stat("/home/ubuntu/cephtest/mnt.0/delete_me").st_ino) 2024-02-19T15:23:08.615 DEBUG:teuthology.orchestra.run.smithi047:> ' 2024-02-19T15:23:08.682 INFO:teuthology.orchestra.run.smithi047.stdout:1099511627776 2024-02-19T15:23:08.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-02-19T15:23:08.691 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-02-19T15:23:08.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f92939a0080 1 Processor -- start 2024-02-19T15:23:08.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f92939a0080 1 -- start start 2024-02-19T15:23:08.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f92939a0080 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d259ca7ab0 0x55d259ca1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:08.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f92939a0080 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d259ca7e90 0x55d259ca22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:08.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f92939a0080 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55d259d33bd0 0x55d259ca3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:08.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f92939a0080 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d259c0bf90 con 0x55d259ca7ab0 2024-02-19T15:23:08.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f92939a0080 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d259ba31e0 con 0x55d259ca7e90 2024-02-19T15:23:08.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f92939a0080 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d259ba2e80 con 0x55d259d33bd0 2024-02-19T15:23:08.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f9290c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d259ca7ab0 0x55d259ca1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:08.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f9290c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d259ca7ab0 0x55d259ca1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48150/0 (socket says 172.21.15.47:48150) 2024-02-19T15:23:08.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f9290c00640 1 -- 172.21.15.47:0/743760917 learned_addr learned my addr 172.21.15.47:0/743760917 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:08.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f928be00640 1 --2- 172.21.15.47:0/743760917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d259ca7e90 0x55d259ca22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:08.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f9291600640 1 --2- 172.21.15.47:0/743760917 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55d259d33bd0 0x55d259ca3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:08.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f9290c00640 1 -- 172.21.15.47:0/743760917 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55d259d33bd0 msgr2=0x55d259ca3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:08.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f9290c00640 1 --2- 172.21.15.47:0/743760917 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55d259d33bd0 0x55d259ca3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:08.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f9290c00640 1 -- 172.21.15.47:0/743760917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d259ca7e90 msgr2=0x55d259ca22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:08.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f9290c00640 1 --2- 172.21.15.47:0/743760917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d259ca7e90 0x55d259ca22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:08.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f9290c00640 1 -- 172.21.15.47:0/743760917 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9284005d30 con 0x55d259ca7ab0 2024-02-19T15:23:08.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f928be00640 1 --2- 172.21.15.47:0/743760917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d259ca7e90 0x55d259ca22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:08.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f9290c00640 1 --2- 172.21.15.47:0/743760917 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d259ca7ab0 0x55d259ca1d90 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f928400a530 tx=0x7f9284003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=954c7b324b28d8c9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:08.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f928aa00640 1 -- 172.21.15.47:0/743760917 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928400a240 con 0x55d259ca7ab0 2024-02-19T15:23:08.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f928aa00640 1 -- 172.21.15.47:0/743760917 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92840b9070 con 0x55d259ca7ab0 2024-02-19T15:23:08.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.776+0000 7f928aa00640 1 -- 172.21.15.47:0/743760917 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92840aab00 con 0x55d259ca7ab0 2024-02-19T15:23:08.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f92939a0080 1 -- 172.21.15.47:0/743760917 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d259ca7ab0 msgr2=0x55d259ca1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:08.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f92939a0080 1 --2- 172.21.15.47:0/743760917 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d259ca7ab0 0x55d259ca1d90 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f928400a530 tx=0x7f9284003030 comp rx=0 tx=0).stop 2024-02-19T15:23:08.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f92939a0080 1 -- 172.21.15.47:0/743760917 shutdown_connections 2024-02-19T15:23:08.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f92939a0080 1 --2- 172.21.15.47:0/743760917 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55d259d33bd0 0x55d259ca3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:08.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f92939a0080 1 --2- 172.21.15.47:0/743760917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d259ca7e90 0x55d259ca22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:08.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f92939a0080 1 --2- 172.21.15.47:0/743760917 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d259ca7ab0 0x55d259ca1d90 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:08.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f92939a0080 1 -- 172.21.15.47:0/743760917 >> 172.21.15.47:0/743760917 conn(0x55d259c8a680 msgr2=0x55d259d12490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:08.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f92939a0080 1 -- 172.21.15.47:0/743760917 shutdown_connections 2024-02-19T15:23:08.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f92939a0080 1 -- 172.21.15.47:0/743760917 wait complete. 2024-02-19T15:23:08.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f92939a0080 1 Processor -- start 2024-02-19T15:23:08.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f92939a0080 1 -- start start 2024-02-19T15:23:08.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f92939a0080 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d259ca7ab0 0x55d259d30630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:08.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f92939a0080 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55d259ca7e90 0x55d259c8ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:08.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f92939a0080 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d259d33bd0 0x55d259c8f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:08.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f92939a0080 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d259bc6610 con 0x55d259ca7ab0 2024-02-19T15:23:08.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f92939a0080 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d259b96740 con 0x55d259d33bd0 2024-02-19T15:23:08.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f92939a0080 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d259d2c7f0 con 0x55d259ca7e90 2024-02-19T15:23:08.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f9290c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d259ca7ab0 0x55d259d30630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:08.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f9290c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d259ca7ab0 0x55d259d30630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48160/0 (socket says 172.21.15.47:48160) 2024-02-19T15:23:08.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f9290c00640 1 -- 172.21.15.47:0/3275794035 learned_addr learned my addr 172.21.15.47:0/3275794035 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:08.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f9291600640 1 --2- 172.21.15.47:0/3275794035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d259d33bd0 0x55d259c8f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:08.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f928be00640 1 --2- 172.21.15.47:0/3275794035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55d259ca7e90 0x55d259c8ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:08.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f9290c00640 1 -- 172.21.15.47:0/3275794035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55d259ca7e90 msgr2=0x55d259c8ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:08.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f9290c00640 1 --2- 172.21.15.47:0/3275794035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55d259ca7e90 0x55d259c8ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:08.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f9290c00640 1 -- 172.21.15.47:0/3275794035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d259d33bd0 msgr2=0x55d259c8f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:08.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f9290c00640 1 --2- 172.21.15.47:0/3275794035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d259d33bd0 0x55d259c8f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:08.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f9290c00640 1 -- 172.21.15.47:0/3275794035 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d259baed80 con 0x55d259ca7ab0 2024-02-19T15:23:08.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f9290c00640 1 --2- 172.21.15.47:0/3275794035 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d259ca7ab0 0x55d259d30630 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f928400a530 tx=0x7f9284018190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:08.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f9289600640 1 -- 172.21.15.47:0/3275794035 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92840aa8e0 con 0x55d259ca7ab0 2024-02-19T15:23:08.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f9289600640 1 -- 172.21.15.47:0/3275794035 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92840aaa60 con 0x55d259ca7ab0 2024-02-19T15:23:08.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f92939a0080 1 -- 172.21.15.47:0/3275794035 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d259ba1f30 con 0x55d259ca7ab0 2024-02-19T15:23:08.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f9289600640 1 -- 172.21.15.47:0/3275794035 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92840aae10 con 0x55d259ca7ab0 2024-02-19T15:23:08.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.780+0000 7f92939a0080 1 -- 172.21.15.47:0/3275794035 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d259ba1990 con 0x55d259ca7ab0 2024-02-19T15:23:08.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.784+0000 7f9289600640 1 -- 172.21.15.47:0/3275794035 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f92840bb030 con 0x55d259ca7ab0 2024-02-19T15:23:08.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.784+0000 7f9289600640 1 --2- 172.21.15.47:0/3275794035 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9270065850 0x7f9270067d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:08.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.784+0000 7f9289600640 1 -- 172.21.15.47:0/3275794035 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f928411f690 con 0x55d259ca7ab0 2024-02-19T15:23:08.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.784+0000 7f92939a0080 1 --2- 172.21.15.47:0/3275794035 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x55d259ca47a0 0x55d259ca4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:08.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.784+0000 7f92939a0080 1 -- 172.21.15.47:0/3275794035 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) v8 -- 0x55d259ca50f0 con 0x55d259ca47a0 2024-02-19T15:23:08.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.784+0000 7f928be00640 1 --2- 172.21.15.47:0/3275794035 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9270065850 0x7f9270067d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:08.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.784+0000 7f9291600640 1 --2- 172.21.15.47:0/3275794035 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x55d259ca47a0 0x55d259ca4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:08.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.788+0000 7f928be00640 1 --2- 172.21.15.47:0/3275794035 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9270065850 0x7f9270067d10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f927c004180 tx=0x7f927c000bd0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:08.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.788+0000 7f9291600640 1 --2- 172.21.15.47:0/3275794035 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x55d259ca47a0 0x55d259ca4bb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:08.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.788+0000 7f9291600640 1 -- 172.21.15.47:0/3275794035 <== osd.3 v2:172.21.15.47:6824/4149334728 1 ==== osd_op_reply(1 10000000000.00000000 [stat out=16b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x7f928c006020 con 0x55d259ca47a0 2024-02-19T15:23:08.792 INFO:teuthology.orchestra.run.smithi047.stdout:cephfs_metadata/10000000000.00000000 mtime 2024-02-19T15:23:08.000000+0000, size 0 2024-02-19T15:23:08.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.788+0000 7f92939a0080 1 -- 172.21.15.47:0/3275794035 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x55d259ca47a0 msgr2=0x55d259ca4bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:08.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.788+0000 7f92939a0080 1 --2- 172.21.15.47:0/3275794035 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x55d259ca47a0 0x55d259ca4bb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:08.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.788+0000 7f92939a0080 1 -- 172.21.15.47:0/3275794035 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9270065850 msgr2=0x7f9270067d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:08.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.788+0000 7f92939a0080 1 --2- 172.21.15.47:0/3275794035 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9270065850 0x7f9270067d10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f927c004180 tx=0x7f927c000bd0 comp rx=0 tx=0).stop 2024-02-19T15:23:08.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.788+0000 7f92939a0080 1 -- 172.21.15.47:0/3275794035 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d259ca7ab0 msgr2=0x55d259d30630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:08.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.788+0000 7f92939a0080 1 --2- 172.21.15.47:0/3275794035 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d259ca7ab0 0x55d259d30630 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f928400a530 tx=0x7f9284018190 comp rx=0 tx=0).stop 2024-02-19T15:23:08.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.788+0000 7f9291600640 1 -- 172.21.15.47:0/3275794035 reap_dead start 2024-02-19T15:23:08.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.788+0000 7f92939a0080 1 -- 172.21.15.47:0/3275794035 shutdown_connections 2024-02-19T15:23:08.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.788+0000 7f92939a0080 1 -- 172.21.15.47:0/3275794035 >> 172.21.15.47:0/3275794035 conn(0x55d259c8a680 msgr2=0x55d259ca1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:08.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.788+0000 7f92939a0080 1 -- 172.21.15.47:0/3275794035 shutdown_connections 2024-02-19T15:23:08.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:08.788+0000 7f92939a0080 1 -- 172.21.15.47:0/3275794035 wait complete. 2024-02-19T15:23:08.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:08.796 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-02-19T15:23:10.335 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:23:10.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.472+0000 7fbac5400640 1 Processor -- start 2024-02-19T15:23:10.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.472+0000 7fbac5400640 1 -- start start 2024-02-19T15:23:10.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.472+0000 7fbac5400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbac0071350 0x7fbac0165e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:10.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.472+0000 7fbac5400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac00718c0 con 0x7fbac015b1a0 2024-02-19T15:23:10.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.472+0000 7fbac5400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac0071ba0 con 0x7fbac015d6e0 2024-02-19T15:23:10.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.472+0000 7fbac5400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac0071e80 con 0x7fbac0071350 2024-02-19T15:23:10.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.472+0000 7fbabe800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbac015b1a0 0x7fbac015b580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43024/0 (socket says 172.21.15.47:43024) 2024-02-19T15:23:10.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.472+0000 7fbabe800640 1 -- 172.21.15.47:0/1102381616 learned_addr learned my addr 172.21.15.47:0/1102381616 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:10.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.472+0000 7fbabf200640 1 --2- 172.21.15.47:0/1102381616 >> v2:172.21.15.47:3300/0 conn(0x7fbac0071350 0x7fbac0165e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:10.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.472+0000 7fbabde00640 1 -- 172.21.15.47:0/1102381616 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174825553 0 0) 0x7fbac00718c0 con 0x7fbac015b1a0 2024-02-19T15:23:10.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.472+0000 7fbabde00640 1 -- 172.21.15.47:0/1102381616 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaa4003530 con 0x7fbac015b1a0 2024-02-19T15:23:10.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.472+0000 7fbabf200640 1 -- 172.21.15.47:0/1102381616 >> v1:172.21.15.116:6789/0 conn(0x7fbac015d6e0 legacy=0x7fbac0162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:10.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.472+0000 7fbabf200640 1 -- 172.21.15.47:0/1102381616 >> v1:172.21.15.47:6789/0 conn(0x7fbac015b1a0 legacy=0x7fbac015b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:10.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.472+0000 7fbabf200640 1 -- 172.21.15.47:0/1102381616 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac0072160 con 0x7fbac0071350 2024-02-19T15:23:10.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.476+0000 7fbabf200640 1 --2- 172.21.15.47:0/1102381616 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbac0071350 0x7fbac0165e70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fbab8009dc0 tx=0x7fbab801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b96cd752cba88262 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:10.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.476+0000 7fbabde00640 1 -- 172.21.15.47:0/1102381616 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab80a5a30 con 0x7fbac0071350 2024-02-19T15:23:10.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.476+0000 7fbabde00640 1 -- 172.21.15.47:0/1102381616 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab80a5b90 con 0x7fbac0071350 2024-02-19T15:23:10.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.476+0000 7fbabde00640 1 -- 172.21.15.47:0/1102381616 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab80ad440 con 0x7fbac0071350 2024-02-19T15:23:10.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.476+0000 7fbac5400640 1 -- 172.21.15.47:0/1102381616 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbac0071350 msgr2=0x7fbac0165e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:10.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.476+0000 7fbac5400640 1 --2- 172.21.15.47:0/1102381616 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbac0071350 0x7fbac0165e70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fbab8009dc0 tx=0x7fbab801c950 comp rx=0 tx=0).stop 2024-02-19T15:23:10.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.476+0000 7fbac5400640 1 -- 172.21.15.47:0/1102381616 shutdown_connections 2024-02-19T15:23:10.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.476+0000 7fbac5400640 1 --2- 172.21.15.47:0/1102381616 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbac0071350 0x7fbac0165e70 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:10.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.476+0000 7fbac5400640 1 -- 172.21.15.47:0/1102381616 >> 172.21.15.47:0/1102381616 conn(0x7fbac00b16a0 msgr2=0x7fbac00b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:10.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.476+0000 7fbac5400640 1 -- 172.21.15.47:0/1102381616 shutdown_connections 2024-02-19T15:23:10.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.476+0000 7fbac5400640 1 -- 172.21.15.47:0/1102381616 wait complete. 2024-02-19T15:23:10.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbac5400640 1 Processor -- start 2024-02-19T15:23:10.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbac5400640 1 -- start start 2024-02-19T15:23:10.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbac5400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbac0071350 0x7fbac0150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:10.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbac5400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbac015b1a0 0x7fbac014c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:10.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbac5400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbac015d6e0 0x7fbac01555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:10.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbac5400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac01514b0 con 0x7fbac015b1a0 2024-02-19T15:23:10.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbac5400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac0154520 con 0x7fbac0071350 2024-02-19T15:23:10.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbac5400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac0154800 con 0x7fbac015d6e0 2024-02-19T15:23:10.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbab7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbac015b1a0 0x7fbac014c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:10.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbab7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbac015b1a0 0x7fbac014c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48170/0 (socket says 172.21.15.47:48170) 2024-02-19T15:23:10.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbab7e00640 1 -- 172.21.15.47:0/1349372971 learned_addr learned my addr 172.21.15.47:0/1349372971 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:10.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbabe800640 1 --2- 172.21.15.47:0/1349372971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbac0071350 0x7fbac0150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:10.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbabf200640 1 --2- 172.21.15.47:0/1349372971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbac015d6e0 0x7fbac01555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:10.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbab7e00640 1 -- 172.21.15.47:0/1349372971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbac015d6e0 msgr2=0x7fbac01555c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:10.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbab7e00640 1 --2- 172.21.15.47:0/1349372971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbac015d6e0 0x7fbac01555c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:10.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbab7e00640 1 -- 172.21.15.47:0/1349372971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbac0071350 msgr2=0x7fbac0150f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:10.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbab7e00640 1 --2- 172.21.15.47:0/1349372971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbac0071350 0x7fbac0150f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:10.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbab7e00640 1 -- 172.21.15.47:0/1349372971 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab8009a70 con 0x7fbac015b1a0 2024-02-19T15:23:10.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbabf200640 1 --2- 172.21.15.47:0/1349372971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbac015d6e0 0x7fbac01555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:10.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbab7e00640 1 --2- 172.21.15.47:0/1349372971 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbac015b1a0 0x7fbac014c820 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fbaa800c4d0 tx=0x7fbaa800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:10.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbab6a00640 1 -- 172.21.15.47:0/1349372971 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa80052d0 con 0x7fbac015b1a0 2024-02-19T15:23:10.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbab6a00640 1 -- 172.21.15.47:0/1349372971 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa8005430 con 0x7fbac015b1a0 2024-02-19T15:23:10.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbab6a00640 1 -- 172.21.15.47:0/1349372971 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8012830 con 0x7fbac015b1a0 2024-02-19T15:23:10.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbac5400640 1 -- 172.21.15.47:0/1349372971 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac0154ae0 con 0x7fbac015b1a0 2024-02-19T15:23:10.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.480+0000 7fbac5400640 1 -- 172.21.15.47:0/1349372971 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac01550a0 con 0x7fbac015b1a0 2024-02-19T15:23:10.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.484+0000 7fbac5400640 1 -- 172.21.15.47:0/1349372971 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac0075360 con 0x7fbac015b1a0 2024-02-19T15:23:10.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.484+0000 7fbab6a00640 1 -- 172.21.15.47:0/1349372971 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fbaa8012990 con 0x7fbac015b1a0 2024-02-19T15:23:10.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.484+0000 7fbab6a00640 1 --2- 172.21.15.47:0/1349372971 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fba9c066e70 0x7fba9c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:10.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.484+0000 7fbab6a00640 1 -- 172.21.15.47:0/1349372971 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbaa8084460 con 0x7fbac015b1a0 2024-02-19T15:23:10.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.484+0000 7fbabe800640 1 --2- 172.21.15.47:0/1349372971 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fba9c066e70 0x7fba9c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:10.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.484+0000 7fbabe800640 1 --2- 172.21.15.47:0/1349372971 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fba9c066e70 0x7fba9c069330 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbab00011f0 tx=0x7fbab0007d40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:10.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.488+0000 7fbab6a00640 1 -- 172.21.15.47:0/1349372971 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fbaa8051240 con 0x7fbac015b1a0 2024-02-19T15:23:10.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.684+0000 7fbac5400640 1 -- 172.21.15.47:0/1349372971 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbac006df00 con 0x7fbac015b1a0 2024-02-19T15:23:10.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.684+0000 7fbab6a00640 1 -- 172.21.15.47:0/1349372971 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4938 (secure 0 0 0) 0x7fbaa8054900 con 0x7fbac015b1a0 2024-02-19T15:23:10.691 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:10.691 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1616905430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1616905430},{"type":"v1","addr":"172.21.15.116:6836","nonce":1616905430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4814,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2845273225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2845273225},{"type":"v1","addr":"172.21.15.116:6837","nonce":2845273225}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5064,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/20043732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":20043732},{"type":"v1","addr":"172.21.15.47:6834","nonce":20043732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:22:58.590074+0000","modified":"2024-02-19T15:23:01.656160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3700339993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3700339993},{"type":"v1","addr":"172.21.15.47:6835","nonce":3700339993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T15:23:10.691 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 16 2024-02-19T15:23:10.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.688+0000 7fbac5400640 1 -- 172.21.15.47:0/1349372971 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fba9c066e70 msgr2=0x7fba9c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:10.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.688+0000 7fbac5400640 1 --2- 172.21.15.47:0/1349372971 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fba9c066e70 0x7fba9c069330 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbab00011f0 tx=0x7fbab0007d40 comp rx=0 tx=0).stop 2024-02-19T15:23:10.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.688+0000 7fbac5400640 1 -- 172.21.15.47:0/1349372971 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbac015b1a0 msgr2=0x7fbac014c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:10.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.688+0000 7fbac5400640 1 --2- 172.21.15.47:0/1349372971 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbac015b1a0 0x7fbac014c820 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fbaa800c4d0 tx=0x7fbaa800c990 comp rx=0 tx=0).stop 2024-02-19T15:23:10.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.688+0000 7fbac5400640 1 -- 172.21.15.47:0/1349372971 shutdown_connections 2024-02-19T15:23:10.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.688+0000 7fbac5400640 1 --2- 172.21.15.47:0/1349372971 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fba9c066e70 0x7fba9c069330 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:10.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.688+0000 7fbac5400640 1 --2- 172.21.15.47:0/1349372971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbac015d6e0 0x7fbac01555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:10.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.688+0000 7fbac5400640 1 --2- 172.21.15.47:0/1349372971 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbac015b1a0 0x7fbac014c820 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:10.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.688+0000 7fbac5400640 1 --2- 172.21.15.47:0/1349372971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbac0071350 0x7fbac0150f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:10.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.688+0000 7fbac5400640 1 -- 172.21.15.47:0/1349372971 >> 172.21.15.47:0/1349372971 conn(0x7fbac00b16a0 msgr2=0x7fbac0070b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:10.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.688+0000 7fbac5400640 1 -- 172.21.15.47:0/1349372971 shutdown_connections 2024-02-19T15:23:10.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:10.688+0000 7fbac5400640 1 -- 172.21.15.47:0/1349372971 wait complete. 2024-02-19T15:23:10.711 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-02-19T15:23:11.057 INFO:teuthology.orchestra.run.smithi047.stdout:{"message":"","return_code":0} 2024-02-19T15:23:11.057 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-02-19T15:23:11.058 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:23:11.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.196+0000 7f1c15a00640 1 Processor -- start 2024-02-19T15:23:11.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.196+0000 7f1c15a00640 1 -- start start 2024-02-19T15:23:11.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.196+0000 7f1c15a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c1015d6e0 0x7f1c10162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:11.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.196+0000 7f1c15a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1c10071350 0x7f1c10164c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:11.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.196+0000 7f1c15a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c100718c0 con 0x7f1c1015b1a0 2024-02-19T15:23:11.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.196+0000 7f1c15a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c10071ba0 con 0x7f1c10071350 2024-02-19T15:23:11.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.196+0000 7f1c15a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c10071e80 con 0x7f1c1015d6e0 2024-02-19T15:23:11.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.196+0000 7f1c0ea00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1c1015b1a0 0x7f1c1015b580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43030/0 (socket says 172.21.15.47:43030) 2024-02-19T15:23:11.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.196+0000 7f1c0ea00640 1 -- 172.21.15.47:0/3501050819 learned_addr learned my addr 172.21.15.47:0/3501050819 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:11.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.196+0000 7f1c0f400640 1 --2- 172.21.15.47:0/3501050819 >> v2:172.21.15.47:3300/0 conn(0x7f1c10071350 0x7f1c10164c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:11.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.196+0000 7f1c06000640 1 --2- 172.21.15.47:0/3501050819 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c1015d6e0 0x7f1c10162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:11.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.196+0000 7f1c0f400640 1 -- 172.21.15.47:0/3501050819 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c1015d6e0 msgr2=0x7f1c10162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:11.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.196+0000 7f1c0f400640 1 --2- 172.21.15.47:0/3501050819 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c1015d6e0 0x7f1c10162740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:11.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.196+0000 7f1c0f400640 1 -- 172.21.15.47:0/3501050819 >> v1:172.21.15.47:6789/0 conn(0x7f1c1015b1a0 legacy=0x7f1c1015b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:11.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.196+0000 7f1c0e000640 1 -- 172.21.15.47:0/3501050819 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223144143 0 0) 0x7f1c100718c0 con 0x7f1c1015b1a0 2024-02-19T15:23:11.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.196+0000 7f1c0f400640 1 -- 172.21.15.47:0/3501050819 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c10072160 con 0x7f1c10071350 2024-02-19T15:23:11.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.200+0000 7f1c0f400640 1 --2- 172.21.15.47:0/3501050819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c10071350 0x7f1c10164c90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1c080097c0 tx=0x7f1c0801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ea13a451b745a5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:11.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.200+0000 7f1c0e000640 1 -- 172.21.15.47:0/3501050819 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c080aa620 con 0x7f1c10071350 2024-02-19T15:23:11.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.200+0000 7f1c0e000640 1 -- 172.21.15.47:0/3501050819 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c080aa780 con 0x7f1c10071350 2024-02-19T15:23:11.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.200+0000 7f1c0e000640 1 -- 172.21.15.47:0/3501050819 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c080aab10 con 0x7f1c10071350 2024-02-19T15:23:11.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.200+0000 7f1c15a00640 1 -- 172.21.15.47:0/3501050819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c10071350 msgr2=0x7f1c10164c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:11.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.200+0000 7f1c15a00640 1 --2- 172.21.15.47:0/3501050819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c10071350 0x7f1c10164c90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1c080097c0 tx=0x7f1c0801c4a0 comp rx=0 tx=0).stop 2024-02-19T15:23:11.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.200+0000 7f1c15a00640 1 -- 172.21.15.47:0/3501050819 shutdown_connections 2024-02-19T15:23:11.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.200+0000 7f1c15a00640 1 --2- 172.21.15.47:0/3501050819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c10071350 0x7f1c10164c90 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:11.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.200+0000 7f1c15a00640 1 --2- 172.21.15.47:0/3501050819 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c1015d6e0 0x7f1c10162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:11.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.200+0000 7f1c15a00640 1 -- 172.21.15.47:0/3501050819 >> 172.21.15.47:0/3501050819 conn(0x7f1c100b16a0 msgr2=0x7f1c100b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:11.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.200+0000 7f1c15a00640 1 -- 172.21.15.47:0/3501050819 shutdown_connections 2024-02-19T15:23:11.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.200+0000 7f1c15a00640 1 -- 172.21.15.47:0/3501050819 wait complete. 2024-02-19T15:23:11.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c15a00640 1 Processor -- start 2024-02-19T15:23:11.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c15a00640 1 -- start start 2024-02-19T15:23:11.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c15a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c10071350 0x7f1c10154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:11.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c15a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c1015b1a0 0x7f1c101550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:11.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c15a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c1015d6e0 0x7f1c100b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:11.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c15a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c100b29d0 con 0x7f1c10071350 2024-02-19T15:23:11.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c15a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c101f6fd0 con 0x7f1c1015d6e0 2024-02-19T15:23:11.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c15a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c101f72b0 con 0x7f1c1015b1a0 2024-02-19T15:23:11.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c0ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c10071350 0x7f1c10154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:11.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c0ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c10071350 0x7f1c10154bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48194/0 (socket says 172.21.15.47:48194) 2024-02-19T15:23:11.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c0ea00640 1 -- 172.21.15.47:0/2467183475 learned_addr learned my addr 172.21.15.47:0/2467183475 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:11.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c06000640 1 --2- 172.21.15.47:0/2467183475 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c1015b1a0 0x7f1c101550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:11.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c0f400640 1 --2- 172.21.15.47:0/2467183475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c1015d6e0 0x7f1c100b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:11.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c0ea00640 1 -- 172.21.15.47:0/2467183475 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c1015b1a0 msgr2=0x7f1c101550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:11.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c0ea00640 1 --2- 172.21.15.47:0/2467183475 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c1015b1a0 0x7f1c101550f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:11.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c0ea00640 1 -- 172.21.15.47:0/2467183475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c1015d6e0 msgr2=0x7f1c100b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:11.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c0ea00640 1 --2- 172.21.15.47:0/2467183475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c1015d6e0 0x7f1c100b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:11.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c0ea00640 1 -- 172.21.15.47:0/2467183475 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c08009420 con 0x7f1c10071350 2024-02-19T15:23:11.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c0ea00640 1 --2- 172.21.15.47:0/2467183475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c10071350 0x7f1c10154bb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1c00001570 tx=0x7f1c0000cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:11.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c07400640 1 -- 172.21.15.47:0/2467183475 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0000a640 con 0x7f1c10071350 2024-02-19T15:23:11.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c07400640 1 -- 172.21.15.47:0/2467183475 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0000a7c0 con 0x7f1c10071350 2024-02-19T15:23:11.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c15a00640 1 -- 172.21.15.47:0/2467183475 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c101f7590 con 0x7f1c10071350 2024-02-19T15:23:11.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c07400640 1 -- 172.21.15.47:0/2467183475 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0000ab70 con 0x7f1c10071350 2024-02-19T15:23:11.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c15a00640 1 -- 172.21.15.47:0/2467183475 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c101f7b50 con 0x7f1c10071350 2024-02-19T15:23:11.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.204+0000 7f1c15a00640 1 -- 172.21.15.47:0/2467183475 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c10075360 con 0x7f1c10071350 2024-02-19T15:23:11.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.208+0000 7f1c07400640 1 -- 172.21.15.47:0/2467183475 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1c0000acd0 con 0x7f1c10071350 2024-02-19T15:23:11.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.208+0000 7f1c07400640 1 --2- 172.21.15.47:0/2467183475 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1bec066e40 0x7f1bec069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:11.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.208+0000 7f1c07400640 1 -- 172.21.15.47:0/2467183475 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c00084390 con 0x7f1c10071350 2024-02-19T15:23:11.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.208+0000 7f1c06000640 1 --2- 172.21.15.47:0/2467183475 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1bec066e40 0x7f1bec069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:11.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.208+0000 7f1c06000640 1 --2- 172.21.15.47:0/2467183475 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1bec066e40 0x7f1bec069300 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1c101f47e0 tx=0x7f1bf8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:11.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.212+0000 7f1c07400640 1 -- 172.21.15.47:0/2467183475 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1c00051190 con 0x7f1c10071350 2024-02-19T15:23:11.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.412+0000 7f1c15a00640 1 -- 172.21.15.47:0/2467183475 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c100cbfc0 con 0x7f1c10071350 2024-02-19T15:23:11.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.412+0000 7f1c07400640 1 -- 172.21.15.47:0/2467183475 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f1c00058360 con 0x7f1c10071350 2024-02-19T15:23:11.418 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:11.418 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1616905430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1616905430},{"type":"v1","addr":"172.21.15.116:6836","nonce":1616905430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4814,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2845273225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2845273225},{"type":"v1","addr":"172.21.15.116:6837","nonce":2845273225}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5064,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/20043732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":20043732},{"type":"v1","addr":"172.21.15.47:6834","nonce":20043732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:22:58.590074+0000","modified":"2024-02-19T15:23:01.656160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3700339993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3700339993},{"type":"v1","addr":"172.21.15.47:6835","nonce":3700339993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T15:23:11.418 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 16 2024-02-19T15:23:11.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.416+0000 7f1c15a00640 1 -- 172.21.15.47:0/2467183475 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1bec066e40 msgr2=0x7f1bec069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:11.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.416+0000 7f1c15a00640 1 --2- 172.21.15.47:0/2467183475 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1bec066e40 0x7f1bec069300 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1c101f47e0 tx=0x7f1bf8008040 comp rx=0 tx=0).stop 2024-02-19T15:23:11.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.416+0000 7f1c15a00640 1 -- 172.21.15.47:0/2467183475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c10071350 msgr2=0x7f1c10154bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:11.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.416+0000 7f1c15a00640 1 --2- 172.21.15.47:0/2467183475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c10071350 0x7f1c10154bb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1c00001570 tx=0x7f1c0000cd20 comp rx=0 tx=0).stop 2024-02-19T15:23:11.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.416+0000 7f1c15a00640 1 -- 172.21.15.47:0/2467183475 shutdown_connections 2024-02-19T15:23:11.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.416+0000 7f1c15a00640 1 --2- 172.21.15.47:0/2467183475 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1bec066e40 0x7f1bec069300 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:11.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.416+0000 7f1c15a00640 1 --2- 172.21.15.47:0/2467183475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c1015d6e0 0x7f1c100b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:11.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.416+0000 7f1c15a00640 1 --2- 172.21.15.47:0/2467183475 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c1015b1a0 0x7f1c101550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:11.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.416+0000 7f1c15a00640 1 --2- 172.21.15.47:0/2467183475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c10071350 0x7f1c10154bb0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:11.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.416+0000 7f1c15a00640 1 -- 172.21.15.47:0/2467183475 >> 172.21.15.47:0/2467183475 conn(0x7f1c100b16a0 msgr2=0x7f1c1006d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:11.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.416+0000 7f1c15a00640 1 -- 172.21.15.47:0/2467183475 shutdown_connections 2024-02-19T15:23:11.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.416+0000 7f1c15a00640 1 -- 172.21.15.47:0/2467183475 wait complete. 2024-02-19T15:23:11.439 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-02-19T15:23:11.598 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"strays_created":1001}} 2024-02-19T15:23:11.599 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1001 } } 2024-02-19T15:23:11.599 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:23:11.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.744+0000 7f983b000640 1 Processor -- start 2024-02-19T15:23:11.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.744+0000 7f983b000640 1 -- start start 2024-02-19T15:23:11.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.744+0000 7f983b000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f98340b21c0 0x7f98340cef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:11.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.744+0000 7f983b000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f98340b30d0 0x7f9834070b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:11.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.748+0000 7f983b000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9834160cd0 con 0x7f98340b3510 2024-02-19T15:23:11.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.748+0000 7f983b000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98340b3d90 con 0x7f98340b30d0 2024-02-19T15:23:11.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.748+0000 7f983b000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98340b4070 con 0x7f98340b21c0 2024-02-19T15:23:11.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.748+0000 7f9833400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f98340b30d0 0x7f9834070b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:11.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.748+0000 7f9833400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f98340b30d0 0x7f9834070b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48208/0 (socket says 172.21.15.47:48208) 2024-02-19T15:23:11.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.748+0000 7f9833400640 1 -- 172.21.15.47:0/2734716126 learned_addr learned my addr 172.21.15.47:0/2734716126 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:11.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.756+0000 7f9833400640 1 -- 172.21.15.47:0/2734716126 >> v2:172.21.15.116:3300/0 conn(0x7f98340b21c0 msgr2=0x7f98340cef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:11.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.756+0000 7f9833400640 1 --2- 172.21.15.47:0/2734716126 >> v2:172.21.15.116:3300/0 conn(0x7f98340b21c0 0x7f98340cef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:11.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.756+0000 7f9833400640 1 -- 172.21.15.47:0/2734716126 >> v1:172.21.15.47:6789/0 conn(0x7f98340b3510 legacy=0x7f983415dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:11.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.756+0000 7f9833400640 1 -- 172.21.15.47:0/2734716126 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98340b4350 con 0x7f98340b30d0 2024-02-19T15:23:11.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.792+0000 7f9833400640 1 --2- 172.21.15.47:0/2734716126 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98340b30d0 0x7f9834070b50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9834062c40 tx=0x7f982001c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77d91490ba47e818 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:11.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.796+0000 7f9832a00640 1 -- 172.21.15.47:0/2734716126 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98200ab680 con 0x7f98340b30d0 2024-02-19T15:23:11.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.796+0000 7f9832a00640 1 -- 172.21.15.47:0/2734716126 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98200ab7e0 con 0x7f98340b30d0 2024-02-19T15:23:11.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.796+0000 7f9832a00640 1 -- 172.21.15.47:0/2734716126 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98200abb70 con 0x7f98340b30d0 2024-02-19T15:23:11.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.796+0000 7f983b000640 1 -- 172.21.15.47:0/2734716126 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98340b30d0 msgr2=0x7f9834070b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.796+0000 7f983b000640 1 --2- 172.21.15.47:0/2734716126 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98340b30d0 0x7f9834070b50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9834062c40 tx=0x7f982001c530 comp rx=0 tx=0).stop 2024-02-19T15:23:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.800+0000 7f983b000640 1 -- 172.21.15.47:0/2734716126 shutdown_connections 2024-02-19T15:23:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.800+0000 7f983b000640 1 --2- 172.21.15.47:0/2734716126 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98340b30d0 0x7f9834070b50 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.800+0000 7f983b000640 1 --2- 172.21.15.47:0/2734716126 >> v2:172.21.15.116:3300/0 conn(0x7f98340b21c0 0x7f98340cef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.800+0000 7f983b000640 1 -- 172.21.15.47:0/2734716126 >> 172.21.15.47:0/2734716126 conn(0x7f98340b16a0 msgr2=0x7f98340b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.804+0000 7f983b000640 1 -- 172.21.15.47:0/2734716126 shutdown_connections 2024-02-19T15:23:11.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.804+0000 7f983b000640 1 -- 172.21.15.47:0/2734716126 wait complete. 2024-02-19T15:23:11.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.808+0000 7f983b000640 1 Processor -- start 2024-02-19T15:23:11.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.824+0000 7f983b000640 1 -- start start 2024-02-19T15:23:11.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.828+0000 7f983b000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98340b21c0 0x7f9834168330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:11.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.828+0000 7f983b000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98340b30d0 0x7f983413d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:11.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.828+0000 7f983b000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98340b3510 0x7f9834152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:11.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.828+0000 7f983b000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9834169700 con 0x7f98340b30d0 2024-02-19T15:23:11.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.828+0000 7f983b000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9834168870 con 0x7f98340b3510 2024-02-19T15:23:11.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.828+0000 7f983b000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f983413d830 con 0x7f98340b21c0 2024-02-19T15:23:11.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.832+0000 7f9833400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98340b30d0 0x7f983413d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:11.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.832+0000 7f9833400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98340b30d0 0x7f983413d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48216/0 (socket says 172.21.15.47:48216) 2024-02-19T15:23:11.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.832+0000 7f9833400640 1 -- 172.21.15.47:0/400558404 learned_addr learned my addr 172.21.15.47:0/400558404 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:11.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.832+0000 7f9833e00640 1 --2- 172.21.15.47:0/400558404 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98340b21c0 0x7f9834168330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:11.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.832+0000 7f9839000640 1 --2- 172.21.15.47:0/400558404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98340b3510 0x7f9834152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:11.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.832+0000 7f9833e00640 1 -- 172.21.15.47:0/400558404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98340b3510 msgr2=0x7f9834152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:11.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.832+0000 7f9833e00640 1 --2- 172.21.15.47:0/400558404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98340b3510 0x7f9834152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:11.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.832+0000 7f9833e00640 1 -- 172.21.15.47:0/400558404 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98340b30d0 msgr2=0x7f983413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:11.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.832+0000 7f9833e00640 1 --2- 172.21.15.47:0/400558404 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98340b30d0 0x7f983413d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:11.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.832+0000 7f9833e00640 1 -- 172.21.15.47:0/400558404 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98200094d0 con 0x7f98340b21c0 2024-02-19T15:23:11.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.836+0000 7f9833e00640 1 --2- 172.21.15.47:0/400558404 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98340b21c0 0x7f9834168330 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f982400b420 tx=0x7f982400b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:11.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.836+0000 7f982be00640 1 -- 172.21.15.47:0/400558404 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9824011830 con 0x7f98340b21c0 2024-02-19T15:23:11.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.840+0000 7f982be00640 1 -- 172.21.15.47:0/400558404 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9824011990 con 0x7f98340b21c0 2024-02-19T15:23:11.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.840+0000 7f982be00640 1 -- 172.21.15.47:0/400558404 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9824011d40 con 0x7f98340b21c0 2024-02-19T15:23:11.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.844+0000 7f983b000640 1 -- 172.21.15.47:0/400558404 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9834168af0 con 0x7f98340b21c0 2024-02-19T15:23:11.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.844+0000 7f983b000640 1 -- 172.21.15.47:0/400558404 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98341690b0 con 0x7f98340b21c0 2024-02-19T15:23:11.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.848+0000 7f982be00640 1 -- 172.21.15.47:0/400558404 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f982401b460 con 0x7f98340b21c0 2024-02-19T15:23:11.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.852+0000 7f982be00640 1 --2- 172.21.15.47:0/400558404 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9814066f10 0x7f98140693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:11.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.852+0000 7f983b000640 1 -- 172.21.15.47:0/400558404 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9834075360 con 0x7f98340b21c0 2024-02-19T15:23:11.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.856+0000 7f982be00640 1 -- 172.21.15.47:0/400558404 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98240846a0 con 0x7f98340b21c0 2024-02-19T15:23:11.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.856+0000 7f9833400640 1 --2- 172.21.15.47:0/400558404 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9814066f10 0x7f98140693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:11.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.856+0000 7f9833400640 1 --2- 172.21.15.47:0/400558404 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9814066f10 0x7f98140693d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f982001c570 tx=0x7f98200b6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:11.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:11.872+0000 7f982be00640 1 -- 172.21.15.47:0/400558404 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f9824051520 con 0x7f98340b21c0 2024-02-19T15:23:12.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:12.180+0000 7f983b000640 1 -- 172.21.15.47:0/400558404 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9834062c40 con 0x7f98340b21c0 2024-02-19T15:23:12.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:12.180+0000 7f982be00640 1 -- 172.21.15.47:0/400558404 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f9834062c40 con 0x7f98340b21c0 2024-02-19T15:23:12.186 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:12.186 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1616905430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1616905430},{"type":"v1","addr":"172.21.15.116:6836","nonce":1616905430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4814,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2845273225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2845273225},{"type":"v1","addr":"172.21.15.116:6837","nonce":2845273225}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5064,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/20043732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":20043732},{"type":"v1","addr":"172.21.15.47:6834","nonce":20043732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:22:58.590074+0000","modified":"2024-02-19T15:23:01.656160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3700339993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3700339993},{"type":"v1","addr":"172.21.15.47:6835","nonce":3700339993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T15:23:12.186 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 16 2024-02-19T15:23:12.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:12.184+0000 7f983b000640 1 -- 172.21.15.47:0/400558404 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9814066f10 msgr2=0x7f98140693d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:12.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:12.184+0000 7f983b000640 1 --2- 172.21.15.47:0/400558404 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9814066f10 0x7f98140693d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f982001c570 tx=0x7f98200b6040 comp rx=0 tx=0).stop 2024-02-19T15:23:12.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:12.184+0000 7f983b000640 1 -- 172.21.15.47:0/400558404 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98340b21c0 msgr2=0x7f9834168330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:12.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:12.184+0000 7f983b000640 1 --2- 172.21.15.47:0/400558404 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98340b21c0 0x7f9834168330 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f982400b420 tx=0x7f982400b8e0 comp rx=0 tx=0).stop 2024-02-19T15:23:12.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:12.184+0000 7f983b000640 1 -- 172.21.15.47:0/400558404 shutdown_connections 2024-02-19T15:23:12.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:12.184+0000 7f983b000640 1 --2- 172.21.15.47:0/400558404 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9814066f10 0x7f98140693d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:12.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:12.184+0000 7f983b000640 1 --2- 172.21.15.47:0/400558404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98340b3510 0x7f9834152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:12.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:12.184+0000 7f983b000640 1 --2- 172.21.15.47:0/400558404 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98340b30d0 0x7f983413d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:12.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:12.184+0000 7f983b000640 1 --2- 172.21.15.47:0/400558404 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98340b21c0 0x7f9834168330 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:12.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:12.184+0000 7f983b000640 1 -- 172.21.15.47:0/400558404 >> 172.21.15.47:0/400558404 conn(0x7f98340b16a0 msgr2=0x7f98340b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:12.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:12.184+0000 7f983b000640 1 -- 172.21.15.47:0/400558404 shutdown_connections 2024-02-19T15:23:12.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:12.184+0000 7f983b000640 1 -- 172.21.15.47:0/400558404 wait complete. 2024-02-19T15:23:12.207 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-02-19T15:23:12.493 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"strays_enqueued":500}} 2024-02-19T15:23:12.493 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 500 } } 2024-02-19T15:23:12.493 DEBUG:tasks.ceph_test_case:wait_until_equal: 500 != 1001, waiting (timeout=600)... 2024-02-19T15:23:17.494 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:23:17.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.627+0000 7f28db200640 1 Processor -- start 2024-02-19T15:23:17.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.631+0000 7f28db200640 1 -- start start 2024-02-19T15:23:17.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.631+0000 7f28db200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f28dc0cc0d0 0x7f28dc06d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:17.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.631+0000 7f28db200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28dc06dff0 0x7f28dc071fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:17.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.631+0000 7f28db200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f28dc0cc680 0x7f28dc0ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:17.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.631+0000 7f28db200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28dc159c80 con 0x7f28dc0cc0d0 2024-02-19T15:23:17.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.631+0000 7f28db200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28dc0cc4b0 con 0x7f28dc0cc680 2024-02-19T15:23:17.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.631+0000 7f28db200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc0cd050 con 0x7f28dc06dff0 2024-02-19T15:23:17.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.631+0000 7f28d9e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f28dc0cc0d0 0x7f28dc06d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:17.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.631+0000 7f28d9e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f28dc0cc0d0 0x7f28dc06d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58926/0 (socket says 172.21.15.47:58926) 2024-02-19T15:23:17.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.631+0000 7f28d9e00640 1 -- 172.21.15.47:0/3542623062 learned_addr learned my addr 172.21.15.47:0/3542623062 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:17.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.631+0000 7f28da800640 1 --2- 172.21.15.47:0/3542623062 >> v2:172.21.15.116:3300/0 conn(0x7f28dc0cc680 0x7f28dc0ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:17.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.631+0000 7f28d9e00640 1 -- 172.21.15.47:0/3542623062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28dc06dff0 msgr2=0x7f28dc071fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:17.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.631+0000 7f28d3e00640 1 --2- 172.21.15.47:0/3542623062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28dc06dff0 0x7f28dc071fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:17.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.631+0000 7f28d9e00640 1 --2- 172.21.15.47:0/3542623062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28dc06dff0 0x7f28dc071fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:17.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.631+0000 7f28d9e00640 1 -- 172.21.15.47:0/3542623062 >> v2:172.21.15.116:3300/0 conn(0x7f28dc0cc680 msgr2=0x7f28dc0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:17.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.631+0000 7f28d9e00640 1 --2- 172.21.15.47:0/3542623062 >> v2:172.21.15.116:3300/0 conn(0x7f28dc0cc680 0x7f28dc0ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:17.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.631+0000 7f28d9e00640 1 -- 172.21.15.47:0/3542623062 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28dc06c880 con 0x7f28dc0cc0d0 2024-02-19T15:23:17.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.631+0000 7f28da800640 1 --2- 172.21.15.47:0/3542623062 >> v2:172.21.15.116:3300/0 conn(0x7f28dc0cc680 0x7f28dc0ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:23:17.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.631+0000 7f28d3e00640 1 --2- 172.21.15.47:0/3542623062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28dc06dff0 0x7f28dc071fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:23:17.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.635+0000 7f28d9e00640 1 --2- 172.21.15.47:0/3542623062 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28dc0cc0d0 0x7f28dc06d2d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f28dc0cb5a0 tx=0x7f28cc01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea008607e90281 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:17.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.635+0000 7f28d9400640 1 -- 172.21.15.47:0/3542623062 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28cc0ab620 con 0x7f28dc0cc0d0 2024-02-19T15:23:17.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.635+0000 7f28d9400640 1 -- 172.21.15.47:0/3542623062 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28cc0ab780 con 0x7f28dc0cc0d0 2024-02-19T15:23:17.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.635+0000 7f28d9400640 1 -- 172.21.15.47:0/3542623062 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28cc0abb10 con 0x7f28dc0cc0d0 2024-02-19T15:23:17.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.635+0000 7f28db200640 1 -- 172.21.15.47:0/3542623062 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28dc0cc0d0 msgr2=0x7f28dc06d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:17.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.635+0000 7f28db200640 1 --2- 172.21.15.47:0/3542623062 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28dc0cc0d0 0x7f28dc06d2d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f28dc0cb5a0 tx=0x7f28cc01c460 comp rx=0 tx=0).stop 2024-02-19T15:23:17.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.635+0000 7f28db200640 1 -- 172.21.15.47:0/3542623062 shutdown_connections 2024-02-19T15:23:17.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.635+0000 7f28db200640 1 --2- 172.21.15.47:0/3542623062 >> v2:172.21.15.116:3300/0 conn(0x7f28dc0cc680 0x7f28dc0ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:17.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.635+0000 7f28db200640 1 --2- 172.21.15.47:0/3542623062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28dc06dff0 0x7f28dc071fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:17.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.635+0000 7f28db200640 1 --2- 172.21.15.47:0/3542623062 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28dc0cc0d0 0x7f28dc06d2d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:17.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.635+0000 7f28db200640 1 -- 172.21.15.47:0/3542623062 >> 172.21.15.47:0/3542623062 conn(0x7f28dc0b1840 msgr2=0x7f28dc0b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:17.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.635+0000 7f28db200640 1 -- 172.21.15.47:0/3542623062 shutdown_connections 2024-02-19T15:23:17.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.635+0000 7f28db200640 1 -- 172.21.15.47:0/3542623062 wait complete. 2024-02-19T15:23:17.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.635+0000 7f28db200640 1 Processor -- start 2024-02-19T15:23:17.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.635+0000 7f28db200640 1 -- start start 2024-02-19T15:23:17.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28db200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28dc06dff0 0x7f28dc151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:17.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28db200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28dc0cc0d0 0x7f28dc1514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:17.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28db200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28dc0cc680 0x7f28dc09b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:17.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28db200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc1530b0 con 0x7f28dc0cc0d0 2024-02-19T15:23:17.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28db200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc152530 con 0x7f28dc0cc680 2024-02-19T15:23:17.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28db200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc152810 con 0x7f28dc06dff0 2024-02-19T15:23:17.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28d3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28dc0cc0d0 0x7f28dc1514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:17.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28d3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28dc0cc0d0 0x7f28dc1514a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58938/0 (socket says 172.21.15.47:58938) 2024-02-19T15:23:17.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28d3e00640 1 -- 172.21.15.47:0/1908743420 learned_addr learned my addr 172.21.15.47:0/1908743420 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:17.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28d9e00640 1 --2- 172.21.15.47:0/1908743420 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28dc06dff0 0x7f28dc151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:17.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28da800640 1 --2- 172.21.15.47:0/1908743420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28dc0cc680 0x7f28dc09b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:17.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28d3e00640 1 -- 172.21.15.47:0/1908743420 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28dc06dff0 msgr2=0x7f28dc151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:17.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28d3e00640 1 --2- 172.21.15.47:0/1908743420 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28dc06dff0 0x7f28dc151ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:17.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28d3e00640 1 -- 172.21.15.47:0/1908743420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28dc0cc680 msgr2=0x7f28dc09b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:17.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28d3e00640 1 --2- 172.21.15.47:0/1908743420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28dc0cc680 0x7f28dc09b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:17.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28d3e00640 1 -- 172.21.15.47:0/1908743420 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28cc0094d0 con 0x7f28dc0cc0d0 2024-02-19T15:23:17.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28da800640 1 --2- 172.21.15.47:0/1908743420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28dc0cc680 0x7f28dc09b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:23:17.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28d3e00640 1 --2- 172.21.15.47:0/1908743420 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28dc0cc0d0 0x7f28dc1514a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f28c400c7c0 tx=0x7f28c400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:17.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28d2000640 1 -- 172.21.15.47:0/1908743420 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c4007a30 con 0x7f28dc0cc0d0 2024-02-19T15:23:17.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28d2000640 1 -- 172.21.15.47:0/1908743420 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c4007b90 con 0x7f28dc0cc0d0 2024-02-19T15:23:17.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28d2000640 1 -- 172.21.15.47:0/1908743420 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c400ce70 con 0x7f28dc0cc0d0 2024-02-19T15:23:17.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28db200640 1 -- 172.21.15.47:0/1908743420 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28dc152af0 con 0x7f28dc0cc0d0 2024-02-19T15:23:17.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28db200640 1 -- 172.21.15.47:0/1908743420 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28dc152220 con 0x7f28dc0cc0d0 2024-02-19T15:23:17.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.639+0000 7f28db200640 1 -- 172.21.15.47:0/1908743420 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28dc075360 con 0x7f28dc0cc0d0 2024-02-19T15:23:17.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.643+0000 7f28d2000640 1 -- 172.21.15.47:0/1908743420 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f28c40113f0 con 0x7f28dc0cc0d0 2024-02-19T15:23:17.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.643+0000 7f28d2000640 1 --2- 172.21.15.47:0/1908743420 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f28b8066e40 0x7f28b8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:17.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.643+0000 7f28d2000640 1 -- 172.21.15.47:0/1908743420 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28c4084ca0 con 0x7f28dc0cc0d0 2024-02-19T15:23:17.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.643+0000 7f28d9e00640 1 --2- 172.21.15.47:0/1908743420 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f28b8066e40 0x7f28b8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:17.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.643+0000 7f28d9e00640 1 --2- 172.21.15.47:0/1908743420 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f28b8066e40 0x7f28b8069300 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f28cc0af000 tx=0x7f28cc0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:17.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.647+0000 7f28d2000640 1 -- 172.21.15.47:0/1908743420 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f28c4051960 con 0x7f28dc0cc0d0 2024-02-19T15:23:17.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.843+0000 7f28db200640 1 -- 172.21.15.47:0/1908743420 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28dc06dba0 con 0x7f28dc0cc0d0 2024-02-19T15:23:17.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.843+0000 7f28d2000640 1 -- 172.21.15.47:0/1908743420 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f28c401f020 con 0x7f28dc0cc0d0 2024-02-19T15:23:17.848 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:17.848 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1616905430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1616905430},{"type":"v1","addr":"172.21.15.116:6836","nonce":1616905430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4814,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2845273225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2845273225},{"type":"v1","addr":"172.21.15.116:6837","nonce":2845273225}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5064,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/20043732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":20043732},{"type":"v1","addr":"172.21.15.47:6834","nonce":20043732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:22:58.590074+0000","modified":"2024-02-19T15:23:01.656160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3700339993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3700339993},{"type":"v1","addr":"172.21.15.47:6835","nonce":3700339993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T15:23:17.849 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 16 2024-02-19T15:23:17.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.843+0000 7f28db200640 1 -- 172.21.15.47:0/1908743420 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f28b8066e40 msgr2=0x7f28b8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:17.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.843+0000 7f28db200640 1 --2- 172.21.15.47:0/1908743420 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f28b8066e40 0x7f28b8069300 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f28cc0af000 tx=0x7f28cc0a7000 comp rx=0 tx=0).stop 2024-02-19T15:23:17.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.843+0000 7f28db200640 1 -- 172.21.15.47:0/1908743420 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28dc0cc0d0 msgr2=0x7f28dc1514a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:17.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.843+0000 7f28db200640 1 --2- 172.21.15.47:0/1908743420 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28dc0cc0d0 0x7f28dc1514a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f28c400c7c0 tx=0x7f28c400cc80 comp rx=0 tx=0).stop 2024-02-19T15:23:17.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.843+0000 7f28db200640 1 -- 172.21.15.47:0/1908743420 shutdown_connections 2024-02-19T15:23:17.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.843+0000 7f28db200640 1 --2- 172.21.15.47:0/1908743420 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f28b8066e40 0x7f28b8069300 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:17.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.843+0000 7f28db200640 1 --2- 172.21.15.47:0/1908743420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28dc0cc680 0x7f28dc09b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:17.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.843+0000 7f28db200640 1 --2- 172.21.15.47:0/1908743420 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28dc0cc0d0 0x7f28dc1514a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:17.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.843+0000 7f28db200640 1 --2- 172.21.15.47:0/1908743420 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28dc06dff0 0x7f28dc151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:17.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.843+0000 7f28db200640 1 -- 172.21.15.47:0/1908743420 >> 172.21.15.47:0/1908743420 conn(0x7f28dc0b1840 msgr2=0x7f28dc0ae530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:17.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.843+0000 7f28db200640 1 -- 172.21.15.47:0/1908743420 shutdown_connections 2024-02-19T15:23:17.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:17.843+0000 7f28db200640 1 -- 172.21.15.47:0/1908743420 wait complete. 2024-02-19T15:23:17.870 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-02-19T15:23:18.029 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"strays_enqueued":1001}} 2024-02-19T15:23:18.030 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1001 } } 2024-02-19T15:23:18.030 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:23:18.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.163+0000 7f826fc00640 1 Processor -- start 2024-02-19T15:23:18.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.163+0000 7f826fc00640 1 -- start start 2024-02-19T15:23:18.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.167+0000 7f826fc00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f826806aeb0 0x7f82680b3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:18.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.167+0000 7f826fc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f826815bf50 0x7f826806b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:18.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.167+0000 7f826fc00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826806b7d0 con 0x7f8268072590 2024-02-19T15:23:18.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.167+0000 7f826fc00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f826806da80 con 0x7f826806aeb0 2024-02-19T15:23:18.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.167+0000 7f826fc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826806dd60 con 0x7f826815bf50 2024-02-19T15:23:18.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.167+0000 7f826d200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f826806aeb0 0x7f82680b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:18.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.167+0000 7f826dc00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8268072590 0x7f826806eb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:47554/0 (socket says 172.21.15.47:47554) 2024-02-19T15:23:18.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.167+0000 7f826dc00640 1 -- 172.21.15.47:0/932773321 learned_addr learned my addr 172.21.15.47:0/932773321 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:18.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.167+0000 7f8267e00640 1 --2- 172.21.15.47:0/932773321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f826815bf50 0x7f826806b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:18.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.167+0000 7f826d200640 1 -- 172.21.15.47:0/932773321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f826815bf50 msgr2=0x7f826806b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:18.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.167+0000 7f826d200640 1 --2- 172.21.15.47:0/932773321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f826815bf50 0x7f826806b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.167+0000 7f826d200640 1 -- 172.21.15.47:0/932773321 >> v1:172.21.15.47:6789/0 conn(0x7f8268072590 legacy=0x7f826806eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:18.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.167+0000 7f826d200640 1 -- 172.21.15.47:0/932773321 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826806e040 con 0x7f826806aeb0 2024-02-19T15:23:18.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.167+0000 7f8267400640 1 -- 172.21.15.47:0/932773321 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401144673 0 0) 0x7f826806b7d0 con 0x7f8268072590 2024-02-19T15:23:18.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.167+0000 7f826d200640 1 --2- 172.21.15.47:0/932773321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f826806aeb0 0x7f82680b3860 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f825c0097c0 tx=0x7f825c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7d38958c271bd03 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:18.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f8267400640 1 -- 172.21.15.47:0/932773321 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c0ab670 con 0x7f826806aeb0 2024-02-19T15:23:18.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f8267400640 1 -- 172.21.15.47:0/932773321 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825c0ab7d0 con 0x7f826806aeb0 2024-02-19T15:23:18.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f8267400640 1 -- 172.21.15.47:0/932773321 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c0abb60 con 0x7f826806aeb0 2024-02-19T15:23:18.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f826fc00640 1 -- 172.21.15.47:0/932773321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f826806aeb0 msgr2=0x7f82680b3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:18.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f826fc00640 1 --2- 172.21.15.47:0/932773321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f826806aeb0 0x7f82680b3860 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f825c0097c0 tx=0x7f825c01c470 comp rx=0 tx=0).stop 2024-02-19T15:23:18.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f826fc00640 1 -- 172.21.15.47:0/932773321 shutdown_connections 2024-02-19T15:23:18.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f826fc00640 1 --2- 172.21.15.47:0/932773321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f826815bf50 0x7f826806b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f826fc00640 1 --2- 172.21.15.47:0/932773321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f826806aeb0 0x7f82680b3860 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f826fc00640 1 -- 172.21.15.47:0/932773321 >> 172.21.15.47:0/932773321 conn(0x7f82680b2560 msgr2=0x7f82680b2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:18.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f826fc00640 1 -- 172.21.15.47:0/932773321 shutdown_connections 2024-02-19T15:23:18.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f826fc00640 1 -- 172.21.15.47:0/932773321 wait complete. 2024-02-19T15:23:18.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f826fc00640 1 Processor -- start 2024-02-19T15:23:18.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f826fc00640 1 -- start start 2024-02-19T15:23:18.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f826fc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f826806aeb0 0x7f826814ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:18.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f826fc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8268072590 0x7f82681524c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:18.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f826fc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f826815bf50 0x7f82680b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:18.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f826fc00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82680ceba0 con 0x7f8268072590 2024-02-19T15:23:18.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f826fc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82680cee40 con 0x7f826815bf50 2024-02-19T15:23:18.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f826fc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82680cf120 con 0x7f826806aeb0 2024-02-19T15:23:18.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f8267e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8268072590 0x7f82681524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:18.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f8267e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8268072590 0x7f82681524c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58964/0 (socket says 172.21.15.47:58964) 2024-02-19T15:23:18.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.171+0000 7f8267e00640 1 -- 172.21.15.47:0/2562126087 learned_addr learned my addr 172.21.15.47:0/2562126087 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:18.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.175+0000 7f8267e00640 1 -- 172.21.15.47:0/2562126087 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f826806aeb0 msgr2=0x7f826814ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:18.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.175+0000 7f8267e00640 1 --2- 172.21.15.47:0/2562126087 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f826806aeb0 0x7f826814ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.175+0000 7f8267e00640 1 -- 172.21.15.47:0/2562126087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f826815bf50 msgr2=0x7f82680b18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:18.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.175+0000 7f8267e00640 1 --2- 172.21.15.47:0/2562126087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f826815bf50 0x7f82680b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.175+0000 7f8267e00640 1 -- 172.21.15.47:0/2562126087 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825c009420 con 0x7f8268072590 2024-02-19T15:23:18.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.175+0000 7f8267e00640 1 --2- 172.21.15.47:0/2562126087 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8268072590 0x7f82681524c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f825400e870 tx=0x7f825400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:18.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.175+0000 7f8264c00640 1 -- 172.21.15.47:0/2562126087 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8254009510 con 0x7f8268072590 2024-02-19T15:23:18.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.175+0000 7f8264c00640 1 -- 172.21.15.47:0/2562126087 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825400eec0 con 0x7f8268072590 2024-02-19T15:23:18.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.175+0000 7f8264c00640 1 -- 172.21.15.47:0/2562126087 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825400cb40 con 0x7f8268072590 2024-02-19T15:23:18.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.175+0000 7f826fc00640 1 -- 172.21.15.47:0/2562126087 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82680cf400 con 0x7f8268072590 2024-02-19T15:23:18.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.175+0000 7f826fc00640 1 -- 172.21.15.47:0/2562126087 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82680cf9c0 con 0x7f8268072590 2024-02-19T15:23:18.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.175+0000 7f826fc00640 1 -- 172.21.15.47:0/2562126087 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8268075360 con 0x7f8268072590 2024-02-19T15:23:18.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.175+0000 7f8264c00640 1 -- 172.21.15.47:0/2562126087 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f825400cca0 con 0x7f8268072590 2024-02-19T15:23:18.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.179+0000 7f8264c00640 1 --2- 172.21.15.47:0/2562126087 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8248066e40 0x7f8248069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:18.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.179+0000 7f8264c00640 1 -- 172.21.15.47:0/2562126087 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8254084bf0 con 0x7f8268072590 2024-02-19T15:23:18.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.179+0000 7f826d200640 1 --2- 172.21.15.47:0/2562126087 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8248066e40 0x7f8248069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:18.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.179+0000 7f826d200640 1 --2- 172.21.15.47:0/2562126087 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8248066e40 0x7f8248069300 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f825c0af000 tx=0x7f825c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:18.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.183+0000 7f8264c00640 1 -- 172.21.15.47:0/2562126087 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8254051940 con 0x7f8268072590 2024-02-19T15:23:18.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.379+0000 7f826fc00640 1 -- 172.21.15.47:0/2562126087 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8268071e60 con 0x7f8268072590 2024-02-19T15:23:18.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.383+0000 7f8264c00640 1 -- 172.21.15.47:0/2562126087 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f8254017090 con 0x7f8268072590 2024-02-19T15:23:18.387 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:18.387 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1616905430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1616905430},{"type":"v1","addr":"172.21.15.116:6836","nonce":1616905430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4814,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2845273225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2845273225},{"type":"v1","addr":"172.21.15.116:6837","nonce":2845273225}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5064,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/20043732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":20043732},{"type":"v1","addr":"172.21.15.47:6834","nonce":20043732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:22:58.590074+0000","modified":"2024-02-19T15:23:01.656160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3700339993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3700339993},{"type":"v1","addr":"172.21.15.47:6835","nonce":3700339993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T15:23:18.387 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 16 2024-02-19T15:23:18.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.383+0000 7f826fc00640 1 -- 172.21.15.47:0/2562126087 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8248066e40 msgr2=0x7f8248069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:18.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.383+0000 7f826fc00640 1 --2- 172.21.15.47:0/2562126087 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8248066e40 0x7f8248069300 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f825c0af000 tx=0x7f825c008040 comp rx=0 tx=0).stop 2024-02-19T15:23:18.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.383+0000 7f826fc00640 1 -- 172.21.15.47:0/2562126087 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8268072590 msgr2=0x7f82681524c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:18.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.383+0000 7f826fc00640 1 --2- 172.21.15.47:0/2562126087 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8268072590 0x7f82681524c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f825400e870 tx=0x7f825400ec70 comp rx=0 tx=0).stop 2024-02-19T15:23:18.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.383+0000 7f826fc00640 1 -- 172.21.15.47:0/2562126087 shutdown_connections 2024-02-19T15:23:18.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.383+0000 7f826fc00640 1 --2- 172.21.15.47:0/2562126087 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8248066e40 0x7f8248069300 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.383+0000 7f826fc00640 1 --2- 172.21.15.47:0/2562126087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f826815bf50 0x7f82680b18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.383+0000 7f826fc00640 1 --2- 172.21.15.47:0/2562126087 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8268072590 0x7f82681524c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.383+0000 7f826fc00640 1 --2- 172.21.15.47:0/2562126087 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f826806aeb0 0x7f826814ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.383+0000 7f826fc00640 1 -- 172.21.15.47:0/2562126087 >> 172.21.15.47:0/2562126087 conn(0x7f82680b2560 msgr2=0x7f82680af880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:18.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.383+0000 7f826fc00640 1 -- 172.21.15.47:0/2562126087 shutdown_connections 2024-02-19T15:23:18.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.383+0000 7f826fc00640 1 -- 172.21.15.47:0/2562126087 wait complete. 2024-02-19T15:23:18.414 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue pq_executed 2024-02-19T15:23:18.574 INFO:teuthology.orchestra.run.smithi047.stdout:{"purge_queue":{"pq_executed":1001}} 2024-02-19T15:23:18.574 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1001 } } 2024-02-19T15:23:18.574 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-02-19T15:23:18.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc093ab080 1 Processor -- start 2024-02-19T15:23:18.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc093ab080 1 -- start start 2024-02-19T15:23:18.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc093ab080 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x559d90cb2ab0 0x559d90c9a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:18.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc093ab080 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d90cb2e90 0x559d90c983e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:18.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc093ab080 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559d90d3ea00 0x559d90c98920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:18.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc093ab080 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d90bae1e0 con 0x559d90d3ea00 2024-02-19T15:23:18.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc093ab080 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d90bade80 con 0x559d90cb2e90 2024-02-19T15:23:18.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc093ab080 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d90bd1610 con 0x559d90cb2ab0 2024-02-19T15:23:18.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc07000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559d90d3ea00 0x559d90c98920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:18.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc07000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559d90d3ea00 0x559d90c98920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58988/0 (socket says 172.21.15.47:58988) 2024-02-19T15:23:18.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc07000640 1 -- 172.21.15.47:0/186337991 learned_addr learned my addr 172.21.15.47:0/186337991 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:18.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc05c00640 1 --2- 172.21.15.47:0/186337991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d90cb2e90 0x559d90c983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:18.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc06600640 1 --2- 172.21.15.47:0/186337991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x559d90cb2ab0 0x559d90c9a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:18.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc07000640 1 -- 172.21.15.47:0/186337991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x559d90cb2ab0 msgr2=0x559d90c9a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:18.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc07000640 1 --2- 172.21.15.47:0/186337991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x559d90cb2ab0 0x559d90c9a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc07000640 1 -- 172.21.15.47:0/186337991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d90cb2e90 msgr2=0x559d90c983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:18.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc07000640 1 --2- 172.21.15.47:0/186337991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d90cb2e90 0x559d90c983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc07000640 1 -- 172.21.15.47:0/186337991 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf4005df0 con 0x559d90d3ea00 2024-02-19T15:23:18.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc05c00640 1 --2- 172.21.15.47:0/186337991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d90cb2e90 0x559d90c983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:23:18.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc06600640 1 --2- 172.21.15.47:0/186337991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x559d90cb2ab0 0x559d90c9a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:23:18.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc07000640 1 --2- 172.21.15.47:0/186337991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559d90d3ea00 0x559d90c98920 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efc0000a4b0 tx=0x7efc0000a8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c478a6ce083efb91 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:18.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efbffe00640 1 -- 172.21.15.47:0/186337991 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0000e710 con 0x559d90d3ea00 2024-02-19T15:23:18.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efbffe00640 1 -- 172.21.15.47:0/186337991 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0000e890 con 0x559d90d3ea00 2024-02-19T15:23:18.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efbffe00640 1 -- 172.21.15.47:0/186337991 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0000ec40 con 0x559d90d3ea00 2024-02-19T15:23:18.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc093ab080 1 -- 172.21.15.47:0/186337991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559d90d3ea00 msgr2=0x559d90c98920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:18.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc093ab080 1 --2- 172.21.15.47:0/186337991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559d90d3ea00 0x559d90c98920 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efc0000a4b0 tx=0x7efc0000a8b0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc093ab080 1 -- 172.21.15.47:0/186337991 shutdown_connections 2024-02-19T15:23:18.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc093ab080 1 --2- 172.21.15.47:0/186337991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559d90d3ea00 0x559d90c98920 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc093ab080 1 --2- 172.21.15.47:0/186337991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d90cb2e90 0x559d90c983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc093ab080 1 --2- 172.21.15.47:0/186337991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x559d90cb2ab0 0x559d90c9a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc093ab080 1 -- 172.21.15.47:0/186337991 >> 172.21.15.47:0/186337991 conn(0x559d90c95680 msgr2=0x559d90c67b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:18.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc093ab080 1 -- 172.21.15.47:0/186337991 shutdown_connections 2024-02-19T15:23:18.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.615+0000 7efc093ab080 1 -- 172.21.15.47:0/186337991 wait complete. 2024-02-19T15:23:18.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc093ab080 1 Processor -- start 2024-02-19T15:23:18.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc093ab080 1 -- start start 2024-02-19T15:23:18.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc093ab080 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d90cb2ab0 0x559d90d44b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:18.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc093ab080 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559d90cb2e90 0x559d90c9a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:18.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc093ab080 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x559d90d3ea00 0x559d90d3d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:18.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc093ab080 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d90ba1740 con 0x559d90cb2e90 2024-02-19T15:23:18.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc093ab080 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d90d3c480 con 0x559d90cb2ab0 2024-02-19T15:23:18.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc093ab080 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d90d3c720 con 0x559d90d3ea00 2024-02-19T15:23:18.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc05c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559d90cb2e90 0x559d90c9a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:18.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc05c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559d90cb2e90 0x559d90c9a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58994/0 (socket says 172.21.15.47:58994) 2024-02-19T15:23:18.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc05c00640 1 -- 172.21.15.47:0/318376677 learned_addr learned my addr 172.21.15.47:0/318376677 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:18.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc06600640 1 --2- 172.21.15.47:0/318376677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d90cb2ab0 0x559d90d44b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:18.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc07000640 1 --2- 172.21.15.47:0/318376677 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x559d90d3ea00 0x559d90d3d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:18.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc05c00640 1 -- 172.21.15.47:0/318376677 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x559d90d3ea00 msgr2=0x559d90d3d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:18.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc05c00640 1 --2- 172.21.15.47:0/318376677 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x559d90d3ea00 0x559d90d3d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc05c00640 1 -- 172.21.15.47:0/318376677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d90cb2ab0 msgr2=0x559d90d44b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:18.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc05c00640 1 --2- 172.21.15.47:0/318376677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d90cb2ab0 0x559d90d44b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc05c00640 1 -- 172.21.15.47:0/318376677 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0000a0e0 con 0x559d90cb2e90 2024-02-19T15:23:18.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc07000640 1 --2- 172.21.15.47:0/318376677 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x559d90d3ea00 0x559d90d3d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:18.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc05c00640 1 --2- 172.21.15.47:0/318376677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559d90cb2e90 0x559d90c9a2b0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7efbf40ac040 tx=0x7efbf4018190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:18.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efbfea00640 1 -- 172.21.15.47:0/318376677 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf40b8070 con 0x559d90cb2e90 2024-02-19T15:23:18.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efbfea00640 1 -- 172.21.15.47:0/318376677 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf40aaaf0 con 0x559d90cb2e90 2024-02-19T15:23:18.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efbfea00640 1 -- 172.21.15.47:0/318376677 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf40aae80 con 0x559d90cb2e90 2024-02-19T15:23:18.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc093ab080 1 -- 172.21.15.47:0/318376677 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x559d90bb9d80 con 0x559d90cb2e90 2024-02-19T15:23:18.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.619+0000 7efc093ab080 1 -- 172.21.15.47:0/318376677 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d90bacf30 con 0x559d90cb2e90 2024-02-19T15:23:18.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efbfea00640 1 -- 172.21.15.47:0/318376677 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efbf40c5400 con 0x559d90cb2e90 2024-02-19T15:23:18.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efbfea00640 1 --2- 172.21.15.47:0/318376677 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efbe8065820 0x7efbe8067ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:18.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efbfea00640 1 -- 172.21.15.47:0/318376677 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efbf411e5e0 con 0x559d90cb2e90 2024-02-19T15:23:18.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efc093ab080 1 --2- 172.21.15.47:0/318376677 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x559d90d3ddf0 0x559d90d36d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:18.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efc093ab080 1 -- 172.21.15.47:0/318376677 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) v8 -- 0x559d90c9a7f0 con 0x559d90d3ddf0 2024-02-19T15:23:18.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efc06600640 1 --2- 172.21.15.47:0/318376677 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efbe8065820 0x7efbe8067ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:18.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efc07000640 1 --2- 172.21.15.47:0/318376677 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x559d90d3ddf0 0x559d90d36d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:18.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efc07000640 1 --2- 172.21.15.47:0/318376677 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x559d90d3ddf0 0x559d90d36d70 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:18.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efc06600640 1 --2- 172.21.15.47:0/318376677 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efbe8065820 0x7efbe8067ce0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efbf800b8b0 tx=0x7efbf800d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:18.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efc07000640 1 -- 172.21.15.47:0/318376677 <== osd.3 v2:172.21.15.47:6824/4149334728 1 ==== osd_op_reply(1 10000000000.00000000 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x7efc0000eb00 con 0x559d90d3ddf0 2024-02-19T15:23:18.631 INFO:teuthology.orchestra.run.smithi047.stderr: error stat-ing cephfs_metadata/10000000000.00000000: (2) No such file or directory 2024-02-19T15:23:18.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efc093ab080 1 -- 172.21.15.47:0/318376677 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x559d90d3ddf0 msgr2=0x559d90d36d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:18.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efc093ab080 1 --2- 172.21.15.47:0/318376677 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x559d90d3ddf0 0x559d90d36d70 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efc093ab080 1 -- 172.21.15.47:0/318376677 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efbe8065820 msgr2=0x7efbe8067ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:18.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efc093ab080 1 --2- 172.21.15.47:0/318376677 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efbe8065820 0x7efbe8067ce0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efbf800b8b0 tx=0x7efbf800d040 comp rx=0 tx=0).stop 2024-02-19T15:23:18.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efc093ab080 1 -- 172.21.15.47:0/318376677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559d90cb2e90 msgr2=0x559d90c9a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:18.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efc093ab080 1 --2- 172.21.15.47:0/318376677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559d90cb2e90 0x559d90c9a2b0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7efbf40ac040 tx=0x7efbf4018190 comp rx=0 tx=0).stop 2024-02-19T15:23:18.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efc07000640 1 -- 172.21.15.47:0/318376677 reap_dead start 2024-02-19T15:23:18.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efc093ab080 1 -- 172.21.15.47:0/318376677 shutdown_connections 2024-02-19T15:23:18.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efc093ab080 1 -- 172.21.15.47:0/318376677 >> 172.21.15.47:0/318376677 conn(0x559d90c95680 msgr2=0x559d90d21b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:18.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efc093ab080 1 -- 172.21.15.47:0/318376677 shutdown_connections 2024-02-19T15:23:18.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.623+0000 7efc093ab080 1 -- 172.21.15.47:0/318376677 wait complete. 2024-02-19T15:23:18.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:23:18.635 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-02-19T15:23:18.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.811+0000 7f9d7b200640 1 Processor -- start 2024-02-19T15:23:18.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.811+0000 7f9d7b200640 1 -- start start 2024-02-19T15:23:18.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.815+0000 7f9d7b200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9d7c15dc90 0x7f9d7c0cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:18.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.815+0000 7f9d7b200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9d7c06b5c0 0x7f9d7c0ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:18.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.815+0000 7f9d7b200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d7c0ce9e0 con 0x7f9d7c15e070 2024-02-19T15:23:18.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.815+0000 7f9d7b200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d7c072ae0 con 0x7f9d7c15dc90 2024-02-19T15:23:18.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.815+0000 7f9d7b200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d7c161a10 con 0x7f9d7c06b5c0 2024-02-19T15:23:18.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.815+0000 7f9d79e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9d7c15dc90 0x7f9d7c0cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:18.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.815+0000 7f9d79e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9d7c15dc90 0x7f9d7c0cfc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59000/0 (socket says 172.21.15.47:59000) 2024-02-19T15:23:18.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.815+0000 7f9d79e00640 1 -- 172.21.15.47:0/1745646650 learned_addr learned my addr 172.21.15.47:0/1745646650 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:18.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.815+0000 7f9d7a800640 1 --2- 172.21.15.47:0/1745646650 >> v2:172.21.15.116:3300/0 conn(0x7f9d7c06b5c0 0x7f9d7c0ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:18.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.815+0000 7f9d79e00640 1 -- 172.21.15.47:0/1745646650 >> v2:172.21.15.116:3300/0 conn(0x7f9d7c06b5c0 msgr2=0x7f9d7c0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:18.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.815+0000 7f9d79e00640 1 --2- 172.21.15.47:0/1745646650 >> v2:172.21.15.116:3300/0 conn(0x7f9d7c06b5c0 0x7f9d7c0ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.815+0000 7f9d79e00640 1 -- 172.21.15.47:0/1745646650 >> v1:172.21.15.116:6789/0 conn(0x7f9d7c15e070 legacy=0x7f9d7c06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:18.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.815+0000 7f9d79e00640 1 -- 172.21.15.47:0/1745646650 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c161cf0 con 0x7f9d7c15dc90 2024-02-19T15:23:18.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.815+0000 7f9d79e00640 1 --2- 172.21.15.47:0/1745646650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d7c15dc90 0x7f9d7c0cfc10 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c0afe30 tx=0x7f9d6c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f4d2c0e69dc863a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:18.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d78a00640 1 -- 172.21.15.47:0/1745646650 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c0a8a10 con 0x7f9d7c15dc90 2024-02-19T15:23:18.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d78a00640 1 -- 172.21.15.47:0/1745646650 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6c0a8b70 con 0x7f9d7c15dc90 2024-02-19T15:23:18.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d78a00640 1 -- 172.21.15.47:0/1745646650 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c0b0440 con 0x7f9d7c15dc90 2024-02-19T15:23:18.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d7b200640 1 -- 172.21.15.47:0/1745646650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d7c15dc90 msgr2=0x7f9d7c0cfc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:18.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d7b200640 1 --2- 172.21.15.47:0/1745646650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d7c15dc90 0x7f9d7c0cfc10 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c0afe30 tx=0x7f9d6c01ed10 comp rx=0 tx=0).stop 2024-02-19T15:23:18.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d7b200640 1 -- 172.21.15.47:0/1745646650 shutdown_connections 2024-02-19T15:23:18.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d7b200640 1 --2- 172.21.15.47:0/1745646650 >> v2:172.21.15.116:3300/0 conn(0x7f9d7c06b5c0 0x7f9d7c0ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d7b200640 1 --2- 172.21.15.47:0/1745646650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d7c15dc90 0x7f9d7c0cfc10 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d7b200640 1 -- 172.21.15.47:0/1745646650 >> 172.21.15.47:0/1745646650 conn(0x7f9d7c0d08d0 msgr2=0x7f9d7c0d0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:18.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d7b200640 1 -- 172.21.15.47:0/1745646650 shutdown_connections 2024-02-19T15:23:18.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d7b200640 1 -- 172.21.15.47:0/1745646650 wait complete. 2024-02-19T15:23:18.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d7b200640 1 Processor -- start 2024-02-19T15:23:18.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d7b200640 1 -- start start 2024-02-19T15:23:18.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d7b200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d7c06b5c0 0x7f9d7c155430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:18.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d7b200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d7c15dc90 0x7f9d7c0b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:18.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d7b200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d7c15e070 0x7f9d7c0c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:18.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d7b200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c15e910 con 0x7f9d7c06b5c0 2024-02-19T15:23:18.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d7b200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c155970 con 0x7f9d7c15dc90 2024-02-19T15:23:18.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d7b200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c15e5e0 con 0x7f9d7c15e070 2024-02-19T15:23:18.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d79e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d7c06b5c0 0x7f9d7c155430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:18.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d79e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d7c06b5c0 0x7f9d7c155430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59002/0 (socket says 172.21.15.47:59002) 2024-02-19T15:23:18.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d79e00640 1 -- 172.21.15.47:0/2459114521 learned_addr learned my addr 172.21.15.47:0/2459114521 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:18.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d79400640 1 --2- 172.21.15.47:0/2459114521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d7c15dc90 0x7f9d7c0b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:18.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.819+0000 7f9d7a800640 1 --2- 172.21.15.47:0/2459114521 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d7c15e070 0x7f9d7c0c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:18.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.823+0000 7f9d79e00640 1 -- 172.21.15.47:0/2459114521 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d7c15e070 msgr2=0x7f9d7c0c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:18.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.823+0000 7f9d79e00640 1 --2- 172.21.15.47:0/2459114521 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d7c15e070 0x7f9d7c0c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.823+0000 7f9d79e00640 1 -- 172.21.15.47:0/2459114521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d7c15dc90 msgr2=0x7f9d7c0b3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:18.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.823+0000 7f9d79e00640 1 --2- 172.21.15.47:0/2459114521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d7c15dc90 0x7f9d7c0b3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:18.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.823+0000 7f9d79e00640 1 -- 172.21.15.47:0/2459114521 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6c0094d0 con 0x7f9d7c06b5c0 2024-02-19T15:23:18.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.823+0000 7f9d7a800640 1 --2- 172.21.15.47:0/2459114521 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d7c15e070 0x7f9d7c0c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:18.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.823+0000 7f9d79e00640 1 --2- 172.21.15.47:0/2459114521 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d7c06b5c0 0x7f9d7c155430 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c0b3040 tx=0x7f9d6c01bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:18.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.823+0000 7f9d72000640 1 -- 172.21.15.47:0/2459114521 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c0ba600 con 0x7f9d7c06b5c0 2024-02-19T15:23:18.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.823+0000 7f9d72000640 1 -- 172.21.15.47:0/2459114521 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6c0ba760 con 0x7f9d7c06b5c0 2024-02-19T15:23:18.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.823+0000 7f9d72000640 1 -- 172.21.15.47:0/2459114521 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c0bab10 con 0x7f9d7c06b5c0 2024-02-19T15:23:18.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.823+0000 7f9d7b200640 1 -- 172.21.15.47:0/2459114521 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d7c0b3fd0 con 0x7f9d7c06b5c0 2024-02-19T15:23:18.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.823+0000 7f9d7b200640 1 -- 172.21.15.47:0/2459114521 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d7c0b4590 con 0x7f9d7c06b5c0 2024-02-19T15:23:18.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.823+0000 7f9d7b200640 1 -- 172.21.15.47:0/2459114521 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d7c075360 con 0x7f9d7c06b5c0 2024-02-19T15:23:18.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.823+0000 7f9d72000640 1 -- 172.21.15.47:0/2459114521 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9d6c0bac70 con 0x7f9d7c06b5c0 2024-02-19T15:23:18.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.823+0000 7f9d72000640 1 --2- 172.21.15.47:0/2459114521 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9d5c066e40 0x7f9d5c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:18.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.823+0000 7f9d72000640 1 -- 172.21.15.47:0/2459114521 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d6c122fc0 con 0x7f9d7c06b5c0 2024-02-19T15:23:18.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.827+0000 7f9d79400640 1 --2- 172.21.15.47:0/2459114521 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9d5c066e40 0x7f9d5c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:18.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.827+0000 7f9d79400640 1 --2- 172.21.15.47:0/2459114521 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9d5c066e40 0x7f9d5c069300 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c0b6650 tx=0x7f9d68009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:18.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:18.827+0000 7f9d72000640 1 -- 172.21.15.47:0/2459114521 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f9d6c0efe40 con 0x7f9d7c06b5c0 2024-02-19T15:23:19.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:19.047+0000 7f9d7b200640 1 -- 172.21.15.47:0/2459114521 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f9d7c0b0280 con 0x7f9d7c06b5c0 2024-02-19T15:23:19.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:19.047+0000 7f9d72000640 1 -- 172.21.15.47:0/2459114521 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1691 (secure 0 0 0) 0x7f9d6c0b56b0 con 0x7f9d7c06b5c0 2024-02-19T15:23:19.054 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:19.054 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:23:19.054 INFO:teuthology.orchestra.run.smithi047.stdout: "stats": { 2024-02-19T15:23:19.054 INFO:teuthology.orchestra.run.smithi047.stdout: "total_bytes": 773094113280, 2024-02-19T15:23:19.054 INFO:teuthology.orchestra.run.smithi047.stdout: "total_avail_bytes": 772834353152, 2024-02-19T15:23:19.054 INFO:teuthology.orchestra.run.smithi047.stdout: "total_used_bytes": 259760128, 2024-02-19T15:23:19.054 INFO:teuthology.orchestra.run.smithi047.stdout: "total_used_raw_bytes": 259760128, 2024-02-19T15:23:19.054 INFO:teuthology.orchestra.run.smithi047.stdout: "total_used_raw_ratio": 0.00033600066672079265, 2024-02-19T15:23:19.054 INFO:teuthology.orchestra.run.smithi047.stdout: "num_osds": 8, 2024-02-19T15:23:19.054 INFO:teuthology.orchestra.run.smithi047.stdout: "num_per_pool_osds": 8, 2024-02-19T15:23:19.054 INFO:teuthology.orchestra.run.smithi047.stdout: "num_per_pool_omap_osds": 8 2024-02-19T15:23:19.054 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:19.054 INFO:teuthology.orchestra.run.smithi047.stdout: "stats_by_class": { 2024-02-19T15:23:19.055 INFO:teuthology.orchestra.run.smithi047.stdout: "ssd": { 2024-02-19T15:23:19.055 INFO:teuthology.orchestra.run.smithi047.stdout: "total_bytes": 773094113280, 2024-02-19T15:23:19.055 INFO:teuthology.orchestra.run.smithi047.stdout: "total_avail_bytes": 772834353152, 2024-02-19T15:23:19.055 INFO:teuthology.orchestra.run.smithi047.stdout: "total_used_bytes": 259760128, 2024-02-19T15:23:19.055 INFO:teuthology.orchestra.run.smithi047.stdout: "total_used_raw_bytes": 259760128, 2024-02-19T15:23:19.055 INFO:teuthology.orchestra.run.smithi047.stdout: "total_used_raw_ratio": 0.00033600066672079265 2024-02-19T15:23:19.055 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:19.055 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:19.055 INFO:teuthology.orchestra.run.smithi047.stdout: "pools": [ 2024-02-19T15:23:19.055 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:19.055 INFO:teuthology.orchestra.run.smithi047.stdout: "name": ".mgr", 2024-02-19T15:23:19.055 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 1, 2024-02-19T15:23:19.055 INFO:teuthology.orchestra.run.smithi047.stdout: "stats": { 2024-02-19T15:23:19.055 INFO:teuthology.orchestra.run.smithi047.stdout: "stored": 590368, 2024-02-19T15:23:19.055 INFO:teuthology.orchestra.run.smithi047.stdout: "objects": 2, 2024-02-19T15:23:19.056 INFO:teuthology.orchestra.run.smithi047.stdout: "kb_used": 1160, 2024-02-19T15:23:19.056 INFO:teuthology.orchestra.run.smithi047.stdout: "bytes_used": 1187840, 2024-02-19T15:23:19.056 INFO:teuthology.orchestra.run.smithi047.stdout: "percent_used": 1.6179947124328464e-06, 2024-02-19T15:23:19.056 INFO:teuthology.orchestra.run.smithi047.stdout: "max_avail": 367071068160 2024-02-19T15:23:19.056 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:19.056 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:19.056 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:19.056 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "cephfs_metadata", 2024-02-19T15:23:19.056 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 4, 2024-02-19T15:23:19.056 INFO:teuthology.orchestra.run.smithi047.stdout: "stats": { 2024-02-19T15:23:19.056 INFO:teuthology.orchestra.run.smithi047.stdout: "stored": 4296875, 2024-02-19T15:23:19.056 INFO:teuthology.orchestra.run.smithi047.stdout: "objects": 23, 2024-02-19T15:23:19.056 INFO:teuthology.orchestra.run.smithi047.stdout: "kb_used": 8448, 2024-02-19T15:23:19.056 INFO:teuthology.orchestra.run.smithi047.stdout: "bytes_used": 8650752, 2024-02-19T15:23:19.056 INFO:teuthology.orchestra.run.smithi047.stdout: "percent_used": 1.1783345144067425e-05, 2024-02-19T15:23:19.057 INFO:teuthology.orchestra.run.smithi047.stdout: "max_avail": 367071068160 2024-02-19T15:23:19.057 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:19.057 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:19.057 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:19.057 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "cephfs_data", 2024-02-19T15:23:19.057 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 5, 2024-02-19T15:23:19.057 INFO:teuthology.orchestra.run.smithi047.stdout: "stats": { 2024-02-19T15:23:19.057 INFO:teuthology.orchestra.run.smithi047.stdout: "stored": 85504, 2024-02-19T15:23:19.057 INFO:teuthology.orchestra.run.smithi047.stdout: "objects": 72, 2024-02-19T15:23:19.057 INFO:teuthology.orchestra.run.smithi047.stdout: "kb_used": 668, 2024-02-19T15:23:19.057 INFO:teuthology.orchestra.run.smithi047.stdout: "bytes_used": 684032, 2024-02-19T15:23:19.057 INFO:teuthology.orchestra.run.smithi047.stdout: "percent_used": 9.3174242010718444e-07, 2024-02-19T15:23:19.057 INFO:teuthology.orchestra.run.smithi047.stdout: "max_avail": 367071068160 2024-02-19T15:23:19.057 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:19.057 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:19.057 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-02-19T15:23:19.058 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:23:19.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:19.051+0000 7f9d7b200640 1 -- 172.21.15.47:0/2459114521 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9d5c066e40 msgr2=0x7f9d5c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:19.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:19.051+0000 7f9d7b200640 1 --2- 172.21.15.47:0/2459114521 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9d5c066e40 0x7f9d5c069300 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c0b6650 tx=0x7f9d68009040 comp rx=0 tx=0).stop 2024-02-19T15:23:19.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:19.051+0000 7f9d7b200640 1 -- 172.21.15.47:0/2459114521 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d7c06b5c0 msgr2=0x7f9d7c155430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:19.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:19.051+0000 7f9d7b200640 1 --2- 172.21.15.47:0/2459114521 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d7c06b5c0 0x7f9d7c155430 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c0b3040 tx=0x7f9d6c01bab0 comp rx=0 tx=0).stop 2024-02-19T15:23:19.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:19.051+0000 7f9d7b200640 1 -- 172.21.15.47:0/2459114521 shutdown_connections 2024-02-19T15:23:19.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:19.051+0000 7f9d7b200640 1 --2- 172.21.15.47:0/2459114521 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9d5c066e40 0x7f9d5c069300 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:19.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:19.051+0000 7f9d7b200640 1 --2- 172.21.15.47:0/2459114521 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d7c15e070 0x7f9d7c0c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:19.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:19.051+0000 7f9d7b200640 1 --2- 172.21.15.47:0/2459114521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d7c15dc90 0x7f9d7c0b3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:19.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:19.051+0000 7f9d7b200640 1 --2- 172.21.15.47:0/2459114521 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d7c06b5c0 0x7f9d7c155430 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:19.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:19.051+0000 7f9d7b200640 1 -- 172.21.15.47:0/2459114521 >> 172.21.15.47:0/2459114521 conn(0x7f9d7c0d08d0 msgr2=0x7f9d7c0cbdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:19.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:19.051+0000 7f9d7b200640 1 -- 172.21.15.47:0/2459114521 shutdown_connections 2024-02-19T15:23:19.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:19.051+0000 7f9d7b200640 1 -- 172.21.15.47:0/2459114521 wait complete. 2024-02-19T15:23:19.073 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-02-19T15:23:24.074 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-02-19T15:23:24.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.207+0000 7f74fca00640 1 Processor -- start 2024-02-19T15:23:24.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.207+0000 7f74fca00640 1 -- start start 2024-02-19T15:23:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.211+0000 7f74fca00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74f80cea80 0x7f74f80727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.211+0000 7f74fca00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f74f815c390 0x7f74f815c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.211+0000 7f74fca00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f815d280 con 0x7f74f8145590 2024-02-19T15:23:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.211+0000 7f74fca00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74f8070ff0 con 0x7f74f815c390 2024-02-19T15:23:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.211+0000 7f74fca00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f815cd60 con 0x7f74f80cea80 2024-02-19T15:23:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.211+0000 7f74f5e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f74f8145590 0x7f74f806c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:32946/0 (socket says 172.21.15.47:32946) 2024-02-19T15:23:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.211+0000 7f74f5e00640 1 -- 172.21.15.47:0/4055418319 learned_addr learned my addr 172.21.15.47:0/4055418319 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:24.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.211+0000 7f74f6800640 1 --2- 172.21.15.47:0/4055418319 >> v2:172.21.15.47:3300/0 conn(0x7f74f815c390 0x7f74f815c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:24.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.211+0000 7f74ed400640 1 --2- 172.21.15.47:0/4055418319 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74f80cea80 0x7f74f80727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:24.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.211+0000 7f74f5400640 1 -- 172.21.15.47:0/4055418319 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645893566 0 0) 0x7f74f815d280 con 0x7f74f8145590 2024-02-19T15:23:24.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.211+0000 7f74f5400640 1 -- 172.21.15.47:0/4055418319 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74dc003530 con 0x7f74f8145590 2024-02-19T15:23:24.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.211+0000 7f74f6800640 1 -- 172.21.15.47:0/4055418319 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74f80cea80 msgr2=0x7f74f80727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:24.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.211+0000 7f74f6800640 1 --2- 172.21.15.47:0/4055418319 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74f80cea80 0x7f74f80727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:24.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.211+0000 7f74f6800640 1 -- 172.21.15.47:0/4055418319 >> v1:172.21.15.47:6789/0 conn(0x7f74f8145590 legacy=0x7f74f806c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:24.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.211+0000 7f74f6800640 1 -- 172.21.15.47:0/4055418319 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f80709e0 con 0x7f74f815c390 2024-02-19T15:23:24.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.211+0000 7f74f6800640 1 --2- 172.21.15.47:0/4055418319 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74f815c390 0x7f74f815c820 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009d90 tx=0x7f74f001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89d8e239d4772a0f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:24.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74f5400640 1 -- 172.21.15.47:0/4055418319 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f00a5940 con 0x7f74f815c390 2024-02-19T15:23:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74f5400640 1 -- 172.21.15.47:0/4055418319 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f00a5aa0 con 0x7f74f815c390 2024-02-19T15:23:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74f5400640 1 -- 172.21.15.47:0/4055418319 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f00a5e30 con 0x7f74f815c390 2024-02-19T15:23:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74fca00640 1 -- 172.21.15.47:0/4055418319 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74f815c390 msgr2=0x7f74f815c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74fca00640 1 --2- 172.21.15.47:0/4055418319 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74f815c390 0x7f74f815c820 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009d90 tx=0x7f74f001c970 comp rx=0 tx=0).stop 2024-02-19T15:23:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74fca00640 1 -- 172.21.15.47:0/4055418319 shutdown_connections 2024-02-19T15:23:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74fca00640 1 --2- 172.21.15.47:0/4055418319 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74f815c390 0x7f74f815c820 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74fca00640 1 --2- 172.21.15.47:0/4055418319 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74f80cea80 0x7f74f80727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74fca00640 1 -- 172.21.15.47:0/4055418319 >> 172.21.15.47:0/4055418319 conn(0x7f74f80b16a0 msgr2=0x7f74f80b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74fca00640 1 -- 172.21.15.47:0/4055418319 shutdown_connections 2024-02-19T15:23:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74fca00640 1 -- 172.21.15.47:0/4055418319 wait complete. 2024-02-19T15:23:24.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74fca00640 1 Processor -- start 2024-02-19T15:23:24.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74fca00640 1 -- start start 2024-02-19T15:23:24.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74fca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74f80cea80 0x7f74f81365c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74fca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74f8145590 0x7f74f815b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74fca00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74f815c390 0x7f74f8134b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74fca00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f815bae0 con 0x7f74f8145590 2024-02-19T15:23:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74fca00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f81377b0 con 0x7f74f80cea80 2024-02-19T15:23:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74fca00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f815bf70 con 0x7f74f815c390 2024-02-19T15:23:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74ed400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74f8145590 0x7f74f815b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74ed400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74f8145590 0x7f74f815b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60118/0 (socket says 172.21.15.47:60118) 2024-02-19T15:23:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74ed400640 1 -- 172.21.15.47:0/637414117 learned_addr learned my addr 172.21.15.47:0/637414117 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74f6800640 1 --2- 172.21.15.47:0/637414117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74f815c390 0x7f74f8134b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.215+0000 7f74f5e00640 1 --2- 172.21.15.47:0/637414117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74f80cea80 0x7f74f81365c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.219+0000 7f74ed400640 1 -- 172.21.15.47:0/637414117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74f815c390 msgr2=0x7f74f8134b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.219+0000 7f74ed400640 1 --2- 172.21.15.47:0/637414117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74f815c390 0x7f74f8134b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.219+0000 7f74ed400640 1 -- 172.21.15.47:0/637414117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74f80cea80 msgr2=0x7f74f81365c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.219+0000 7f74ed400640 1 --2- 172.21.15.47:0/637414117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74f80cea80 0x7f74f81365c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:24.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.219+0000 7f74ed400640 1 -- 172.21.15.47:0/637414117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f0009a70 con 0x7f74f8145590 2024-02-19T15:23:24.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.219+0000 7f74ed400640 1 --2- 172.21.15.47:0/637414117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74f8145590 0x7f74f815b5a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f74e000e930 tx=0x7f74e000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:24.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.219+0000 7f74eea00640 1 -- 172.21.15.47:0/637414117 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e0009690 con 0x7f74f8145590 2024-02-19T15:23:24.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.219+0000 7f74eea00640 1 -- 172.21.15.47:0/637414117 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e000ca00 con 0x7f74f8145590 2024-02-19T15:23:24.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.219+0000 7f74eea00640 1 -- 172.21.15.47:0/637414117 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e000cd70 con 0x7f74f8145590 2024-02-19T15:23:24.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.219+0000 7f74fca00640 1 -- 172.21.15.47:0/637414117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f8136ba0 con 0x7f74f8145590 2024-02-19T15:23:24.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.219+0000 7f74fca00640 1 -- 172.21.15.47:0/637414117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f8137160 con 0x7f74f8145590 2024-02-19T15:23:24.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.219+0000 7f74fca00640 1 -- 172.21.15.47:0/637414117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f8137960 con 0x7f74f8145590 2024-02-19T15:23:24.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.219+0000 7f74eea00640 1 -- 172.21.15.47:0/637414117 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f74e0013470 con 0x7f74f8145590 2024-02-19T15:23:24.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.219+0000 7f74eea00640 1 --2- 172.21.15.47:0/637414117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f74d4066e40 0x7f74d4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:24.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.219+0000 7f74eea00640 1 -- 172.21.15.47:0/637414117 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74e0084cc0 con 0x7f74f8145590 2024-02-19T15:23:24.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.223+0000 7f74f5e00640 1 --2- 172.21.15.47:0/637414117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f74d4066e40 0x7f74d4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:24.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.223+0000 7f74f5e00640 1 --2- 172.21.15.47:0/637414117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f74d4066e40 0x7f74d4069300 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f74e8005890 tx=0x7f74e8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:24.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.227+0000 7f74eea00640 1 -- 172.21.15.47:0/637414117 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f74f8137960 con 0x7f74f8145590 2024-02-19T15:23:24.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.447+0000 7f74fca00640 1 -- 172.21.15.47:0/637414117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f74f8137920 con 0x7f74f8145590 2024-02-19T15:23:24.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.447+0000 7f74eea00640 1 -- 172.21.15.47:0/637414117 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1658 (secure 0 0 0) 0x7f74e0021690 con 0x7f74f8145590 2024-02-19T15:23:24.452 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:24.452 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:23:24.453 INFO:teuthology.orchestra.run.smithi047.stdout: "stats": { 2024-02-19T15:23:24.453 INFO:teuthology.orchestra.run.smithi047.stdout: "total_bytes": 773094113280, 2024-02-19T15:23:24.453 INFO:teuthology.orchestra.run.smithi047.stdout: "total_avail_bytes": 772833574912, 2024-02-19T15:23:24.453 INFO:teuthology.orchestra.run.smithi047.stdout: "total_used_bytes": 260538368, 2024-02-19T15:23:24.453 INFO:teuthology.orchestra.run.smithi047.stdout: "total_used_raw_bytes": 260538368, 2024-02-19T15:23:24.453 INFO:teuthology.orchestra.run.smithi047.stdout: "total_used_raw_ratio": 0.00033700731000863016, 2024-02-19T15:23:24.453 INFO:teuthology.orchestra.run.smithi047.stdout: "num_osds": 8, 2024-02-19T15:23:24.453 INFO:teuthology.orchestra.run.smithi047.stdout: "num_per_pool_osds": 8, 2024-02-19T15:23:24.453 INFO:teuthology.orchestra.run.smithi047.stdout: "num_per_pool_omap_osds": 8 2024-02-19T15:23:24.453 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:24.453 INFO:teuthology.orchestra.run.smithi047.stdout: "stats_by_class": { 2024-02-19T15:23:24.453 INFO:teuthology.orchestra.run.smithi047.stdout: "ssd": { 2024-02-19T15:23:24.453 INFO:teuthology.orchestra.run.smithi047.stdout: "total_bytes": 773094113280, 2024-02-19T15:23:24.454 INFO:teuthology.orchestra.run.smithi047.stdout: "total_avail_bytes": 772833574912, 2024-02-19T15:23:24.454 INFO:teuthology.orchestra.run.smithi047.stdout: "total_used_bytes": 260538368, 2024-02-19T15:23:24.454 INFO:teuthology.orchestra.run.smithi047.stdout: "total_used_raw_bytes": 260538368, 2024-02-19T15:23:24.454 INFO:teuthology.orchestra.run.smithi047.stdout: "total_used_raw_ratio": 0.00033700731000863016 2024-02-19T15:23:24.454 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:24.454 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:24.454 INFO:teuthology.orchestra.run.smithi047.stdout: "pools": [ 2024-02-19T15:23:24.454 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:24.454 INFO:teuthology.orchestra.run.smithi047.stdout: "name": ".mgr", 2024-02-19T15:23:24.454 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 1, 2024-02-19T15:23:24.454 INFO:teuthology.orchestra.run.smithi047.stdout: "stats": { 2024-02-19T15:23:24.454 INFO:teuthology.orchestra.run.smithi047.stdout: "stored": 590368, 2024-02-19T15:23:24.454 INFO:teuthology.orchestra.run.smithi047.stdout: "objects": 2, 2024-02-19T15:23:24.454 INFO:teuthology.orchestra.run.smithi047.stdout: "kb_used": 1160, 2024-02-19T15:23:24.454 INFO:teuthology.orchestra.run.smithi047.stdout: "bytes_used": 1187840, 2024-02-19T15:23:24.455 INFO:teuthology.orchestra.run.smithi047.stdout: "percent_used": 1.6180016473299474e-06, 2024-02-19T15:23:24.455 INFO:teuthology.orchestra.run.smithi047.stdout: "max_avail": 367069495296 2024-02-19T15:23:24.455 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:24.455 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:24.455 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:24.455 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "cephfs_metadata", 2024-02-19T15:23:24.455 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 4, 2024-02-19T15:23:24.455 INFO:teuthology.orchestra.run.smithi047.stdout: "stats": { 2024-02-19T15:23:24.455 INFO:teuthology.orchestra.run.smithi047.stdout: "stored": 4683142, 2024-02-19T15:23:24.455 INFO:teuthology.orchestra.run.smithi047.stdout: "objects": 24, 2024-02-19T15:23:24.455 INFO:teuthology.orchestra.run.smithi047.stdout: "kb_used": 9208, 2024-02-19T15:23:24.455 INFO:teuthology.orchestra.run.smithi047.stdout: "bytes_used": 9428992, 2024-02-19T15:23:24.455 INFO:teuthology.orchestra.run.smithi047.stdout: "percent_used": 1.2843441254517529e-05, 2024-02-19T15:23:24.455 INFO:teuthology.orchestra.run.smithi047.stdout: "max_avail": 367069495296 2024-02-19T15:23:24.455 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:24.456 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:24.456 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:24.456 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "cephfs_data", 2024-02-19T15:23:24.456 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 5, 2024-02-19T15:23:24.456 INFO:teuthology.orchestra.run.smithi047.stdout: "stats": { 2024-02-19T15:23:24.456 INFO:teuthology.orchestra.run.smithi047.stdout: "stored": 0, 2024-02-19T15:23:24.456 INFO:teuthology.orchestra.run.smithi047.stdout: "objects": 0, 2024-02-19T15:23:24.456 INFO:teuthology.orchestra.run.smithi047.stdout: "kb_used": 0, 2024-02-19T15:23:24.456 INFO:teuthology.orchestra.run.smithi047.stdout: "bytes_used": 0, 2024-02-19T15:23:24.456 INFO:teuthology.orchestra.run.smithi047.stdout: "percent_used": 0, 2024-02-19T15:23:24.456 INFO:teuthology.orchestra.run.smithi047.stdout: "max_avail": 367069495296 2024-02-19T15:23:24.456 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:24.457 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:24.457 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-02-19T15:23:24.457 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:23:24.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.447+0000 7f74fca00640 1 -- 172.21.15.47:0/637414117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f74d4066e40 msgr2=0x7f74d4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:24.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.447+0000 7f74fca00640 1 --2- 172.21.15.47:0/637414117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f74d4066e40 0x7f74d4069300 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f74e8005890 tx=0x7f74e8009040 comp rx=0 tx=0).stop 2024-02-19T15:23:24.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.447+0000 7f74fca00640 1 -- 172.21.15.47:0/637414117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74f8145590 msgr2=0x7f74f815b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:24.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.447+0000 7f74fca00640 1 --2- 172.21.15.47:0/637414117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74f8145590 0x7f74f815b5a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f74e000e930 tx=0x7f74e000edf0 comp rx=0 tx=0).stop 2024-02-19T15:23:24.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.447+0000 7f74fca00640 1 -- 172.21.15.47:0/637414117 shutdown_connections 2024-02-19T15:23:24.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.447+0000 7f74fca00640 1 --2- 172.21.15.47:0/637414117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f74d4066e40 0x7f74d4069300 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:24.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.447+0000 7f74fca00640 1 --2- 172.21.15.47:0/637414117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74f815c390 0x7f74f8134b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:24.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.447+0000 7f74fca00640 1 --2- 172.21.15.47:0/637414117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74f8145590 0x7f74f815b5a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:24.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.447+0000 7f74fca00640 1 --2- 172.21.15.47:0/637414117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74f80cea80 0x7f74f81365c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:24.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.447+0000 7f74fca00640 1 -- 172.21.15.47:0/637414117 >> 172.21.15.47:0/637414117 conn(0x7f74f80b16a0 msgr2=0x7f74f806d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:24.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.447+0000 7f74fca00640 1 -- 172.21.15.47:0/637414117 shutdown_connections 2024-02-19T15:23:24.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.447+0000 7f74fca00640 1 -- 172.21.15.47:0/637414117 wait complete. 2024-02-19T15:23:24.473 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-02-19T15:23:24.474 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T15:23:24.474 DEBUG:teuthology.orchestra.run.smithi047:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:23:24.534 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:23:24.625 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T15:23:24.625 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:23:24.626 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:23:24.634 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:23:24.634 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:23:24.682 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:23:24.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.855+0000 7f5877000640 1 Processor -- start 2024-02-19T15:23:24.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.855+0000 7f5877000640 1 -- start start 2024-02-19T15:23:24.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f5877000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58700d0460 0x7f587006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:24.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f5877000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5870071e20 0x7f58700b4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:24.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f5877000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f587006f650 con 0x7f5870071a40 2024-02-19T15:23:24.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f5877000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58700b4ad0 con 0x7f5870071e20 2024-02-19T15:23:24.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f5877000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58700ce0c0 con 0x7f58700d0460 2024-02-19T15:23:24.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f5875c00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5870071a40 0x7f58700cfd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:32954/0 (socket says 172.21.15.47:32954) 2024-02-19T15:23:24.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f5875c00640 1 -- 172.21.15.47:0/435573321 learned_addr learned my addr 172.21.15.47:0/435573321 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:24.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f586fe00640 1 -- 172.21.15.47:0/435573321 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669830316 0 0) 0x7f587006f650 con 0x7f5870071a40 2024-02-19T15:23:24.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f586fe00640 1 -- 172.21.15.47:0/435573321 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5858005740 con 0x7f5870071a40 2024-02-19T15:23:24.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f5875200640 1 --2- 172.21.15.47:0/435573321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58700d0460 0x7f587006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:24.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f5876600640 1 --2- 172.21.15.47:0/435573321 >> v2:172.21.15.116:3300/0 conn(0x7f5870071e20 0x7f58700b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:24.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f586fe00640 1 -- 172.21.15.47:0/435573321 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2070578717 0 0) 0x7f5858005740 con 0x7f5870071a40 2024-02-19T15:23:24.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f586fe00640 1 -- 172.21.15.47:0/435573321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58700d0460 msgr2=0x7f587006ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:24.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f586fe00640 1 --2- 172.21.15.47:0/435573321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58700d0460 0x7f587006ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:24.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f586fe00640 1 -- 172.21.15.47:0/435573321 >> v2:172.21.15.116:3300/0 conn(0x7f5870071e20 msgr2=0x7f58700b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:24.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f586fe00640 1 --2- 172.21.15.47:0/435573321 >> v2:172.21.15.116:3300/0 conn(0x7f5870071e20 0x7f58700b4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:24.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f586fe00640 1 -- 172.21.15.47:0/435573321 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587006f090 con 0x7f5870071a40 2024-02-19T15:23:24.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f586fe00640 1 -- 172.21.15.47:0/435573321 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f5864001680 con 0x7f5870071a40 2024-02-19T15:23:24.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f5876600640 1 --2- 172.21.15.47:0/435573321 >> v2:172.21.15.116:3300/0 conn(0x7f5870071e20 0x7f58700b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:24.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f586fe00640 1 -- 172.21.15.47:0/435573321 >> v1:172.21.15.47:6789/0 conn(0x7f5870071a40 legacy=0x7f58700cfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:24.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f586fe00640 1 --2- 172.21.15.47:0/435573321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5858005b90 0x7f5858005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:24.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f586fe00640 1 --2- 172.21.15.47:0/435573321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58700d0460 0x7f58580069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:24.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f586fe00640 1 --2- 172.21.15.47:0/435573321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5858006f20 0x7f58580073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:24.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f586fe00640 1 -- 172.21.15.47:0/435573321 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58700b4ad0 con 0x7f5858006f20 2024-02-19T15:23:24.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f586fe00640 1 -- 172.21.15.47:0/435573321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58700ce0c0 con 0x7f5858005b90 2024-02-19T15:23:24.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f586fe00640 1 -- 172.21.15.47:0/435573321 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f585800dc50 con 0x7f58700d0460 2024-02-19T15:23:24.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f5876600640 1 --2- 172.21.15.47:0/435573321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5858006f20 0x7f58580073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:24.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f5875c00640 1 --2- 172.21.15.47:0/435573321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5858005b90 0x7f5858005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:24.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f5875200640 1 --2- 172.21.15.47:0/435573321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58700d0460 0x7f58580069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:24.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f5876600640 1 -- 172.21.15.47:0/435573321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58700d0460 msgr2=0x7f58580069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:24.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f5876600640 1 --2- 172.21.15.47:0/435573321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58700d0460 0x7f58580069e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:24.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f5876600640 1 -- 172.21.15.47:0/435573321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5858005b90 msgr2=0x7f5858005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:24.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f5876600640 1 --2- 172.21.15.47:0/435573321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5858005b90 0x7f5858005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:24.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.859+0000 7f5876600640 1 -- 172.21.15.47:0/435573321 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5858005740 con 0x7f5858006f20 2024-02-19T15:23:24.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.863+0000 7f5876600640 1 --2- 172.21.15.47:0/435573321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5858006f20 0x7f58580073a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f58680073e0 tx=0x7f586801cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96c26f661337cbab server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:24.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.863+0000 7f586fe00640 1 -- 172.21.15.47:0/435573321 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58680a5e40 con 0x7f5858006f20 2024-02-19T15:23:24.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.863+0000 7f586fe00640 1 -- 172.21.15.47:0/435573321 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58680ad440 con 0x7f5858006f20 2024-02-19T15:23:24.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.863+0000 7f586fe00640 1 -- 172.21.15.47:0/435573321 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58680ad7b0 con 0x7f5858006f20 2024-02-19T15:23:24.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.863+0000 7f5877000640 1 -- 172.21.15.47:0/435573321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5858006f20 msgr2=0x7f58580073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:24.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.863+0000 7f5877000640 1 --2- 172.21.15.47:0/435573321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5858006f20 0x7f58580073a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f58680073e0 tx=0x7f586801cf40 comp rx=0 tx=0).stop 2024-02-19T15:23:24.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.863+0000 7f5876600640 1 -- 172.21.15.47:0/435573321 reap_dead start 2024-02-19T15:23:24.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.863+0000 7f5877000640 1 -- 172.21.15.47:0/435573321 shutdown_connections 2024-02-19T15:23:24.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.863+0000 7f5877000640 1 -- 172.21.15.47:0/435573321 >> 172.21.15.47:0/435573321 conn(0x7f58700b24f0 msgr2=0x7f58700b2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:24.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.863+0000 7f5877000640 1 -- 172.21.15.47:0/435573321 shutdown_connections 2024-02-19T15:23:24.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.863+0000 7f5877000640 1 -- 172.21.15.47:0/435573321 wait complete. 2024-02-19T15:23:24.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.863+0000 7f5877000640 1 Processor -- start 2024-02-19T15:23:24.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.863+0000 7f5877000640 1 -- start start 2024-02-19T15:23:24.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5877000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58701524a0 0x7f58700b15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:24.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5877000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5870152880 0x7f58700b1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:24.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5877000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58700c66d0 0x7f58700c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:24.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5877000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58700c70c0 con 0x7f58701524a0 2024-02-19T15:23:24.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5877000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58700c7e30 con 0x7f5870152880 2024-02-19T15:23:24.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5877000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58700c88c0 con 0x7f58700c66d0 2024-02-19T15:23:24.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5875c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58701524a0 0x7f58700b15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:24.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5875c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58701524a0 0x7f58700b15f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60138/0 (socket says 172.21.15.47:60138) 2024-02-19T15:23:24.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5875c00640 1 -- 172.21.15.47:0/43236800 learned_addr learned my addr 172.21.15.47:0/43236800 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:24.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5875200640 1 --2- 172.21.15.47:0/43236800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5870152880 0x7f58700b1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:24.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5876600640 1 --2- 172.21.15.47:0/43236800 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58700c66d0 0x7f58700c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:24.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5875c00640 1 -- 172.21.15.47:0/43236800 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58700c66d0 msgr2=0x7f58700c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:24.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5875c00640 1 --2- 172.21.15.47:0/43236800 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58700c66d0 0x7f58700c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:24.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5875c00640 1 -- 172.21.15.47:0/43236800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5870152880 msgr2=0x7f58700b1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:24.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5875c00640 1 --2- 172.21.15.47:0/43236800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5870152880 0x7f58700b1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:24.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5875c00640 1 -- 172.21.15.47:0/43236800 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58700ca660 con 0x7f58701524a0 2024-02-19T15:23:24.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5876600640 1 --2- 172.21.15.47:0/43236800 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58700c66d0 0x7f58700c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:24.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5875200640 1 --2- 172.21.15.47:0/43236800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5870152880 0x7f58700b1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:24.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5875c00640 1 --2- 172.21.15.47:0/43236800 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58701524a0 0x7f58700b15f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5864012570 tx=0x7f5864012a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:24.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f586d600640 1 -- 172.21.15.47:0/43236800 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5864014810 con 0x7f58701524a0 2024-02-19T15:23:24.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f586d600640 1 -- 172.21.15.47:0/43236800 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5864014970 con 0x7f58701524a0 2024-02-19T15:23:24.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f586d600640 1 -- 172.21.15.47:0/43236800 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5864014d20 con 0x7f58701524a0 2024-02-19T15:23:24.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5877000640 1 -- 172.21.15.47:0/43236800 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58700c72b0 con 0x7f58701524a0 2024-02-19T15:23:24.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5877000640 1 -- 172.21.15.47:0/43236800 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58700c7870 con 0x7f58701524a0 2024-02-19T15:23:24.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f5877000640 1 -- 172.21.15.47:0/43236800 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5870068500 con 0x7f58701524a0 2024-02-19T15:23:24.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.867+0000 7f586d600640 1 -- 172.21.15.47:0/43236800 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5864016480 con 0x7f58701524a0 2024-02-19T15:23:24.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.871+0000 7f586d600640 1 --2- 172.21.15.47:0/43236800 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5850066e40 0x7f5850069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:24.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.871+0000 7f586d600640 1 -- 172.21.15.47:0/43236800 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5864087f20 con 0x7f58701524a0 2024-02-19T15:23:24.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.871+0000 7f5875200640 1 --2- 172.21.15.47:0/43236800 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5850066e40 0x7f5850069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:24.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.871+0000 7f5875200640 1 --2- 172.21.15.47:0/43236800 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5850066e40 0x7f5850069300 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5870159430 tx=0x7f585c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:24.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:24.875+0000 7f586d600640 1 -- 172.21.15.47:0/43236800 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f5864054d10 con 0x7f58701524a0 2024-02-19T15:23:25.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.039+0000 7f5877000640 1 -- 172.21.15.47:0/43236800 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5870075360 con 0x7f58701524a0 2024-02-19T15:23:25.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.039+0000 7f586d600640 1 -- 172.21.15.47:0/43236800 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16681 (secure 0 0 0) 0x7f58640583d0 con 0x7f58701524a0 2024-02-19T15:23:25.046 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:25.046 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":34,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:23:00.195724+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:56.436273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:57.586982+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:23:25.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.043+0000 7f5877000640 1 -- 172.21.15.47:0/43236800 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5850066e40 msgr2=0x7f5850069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:25.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.043+0000 7f5877000640 1 --2- 172.21.15.47:0/43236800 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5850066e40 0x7f5850069300 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5870159430 tx=0x7f585c008040 comp rx=0 tx=0).stop 2024-02-19T15:23:25.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.043+0000 7f5877000640 1 -- 172.21.15.47:0/43236800 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58701524a0 msgr2=0x7f58700b15f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:25.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.043+0000 7f5877000640 1 --2- 172.21.15.47:0/43236800 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58701524a0 0x7f58700b15f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5864012570 tx=0x7f5864012a30 comp rx=0 tx=0).stop 2024-02-19T15:23:25.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.043+0000 7f5877000640 1 -- 172.21.15.47:0/43236800 shutdown_connections 2024-02-19T15:23:25.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.043+0000 7f5877000640 1 --2- 172.21.15.47:0/43236800 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5850066e40 0x7f5850069300 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.043+0000 7f5877000640 1 --2- 172.21.15.47:0/43236800 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58700c66d0 0x7f58700c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.043+0000 7f5877000640 1 --2- 172.21.15.47:0/43236800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5870152880 0x7f58700b1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.043+0000 7f5877000640 1 --2- 172.21.15.47:0/43236800 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58701524a0 0x7f58700b15f0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.043+0000 7f5877000640 1 -- 172.21.15.47:0/43236800 >> 172.21.15.47:0/43236800 conn(0x7f58700b24f0 msgr2=0x7f58700cf450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:25.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.043+0000 7f5877000640 1 -- 172.21.15.47:0/43236800 shutdown_connections 2024-02-19T15:23:25.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.043+0000 7f5877000640 1 -- 172.21.15.47:0/43236800 wait complete. 2024-02-19T15:23:25.068 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:22:56.436273+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:22:57.586982+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:23:25.068 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:23:25.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.207+0000 7fee61a00640 1 Processor -- start 2024-02-19T15:23:25.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.207+0000 7fee61a00640 1 -- start start 2024-02-19T15:23:25.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.207+0000 7fee61a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fee5c0d0820 0x7fee5c0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:25.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.207+0000 7fee61a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee5c0c8260 0x7fee5c0ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:25.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.207+0000 7fee61a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5c0ceee0 con 0x7fee5c0c7e80 2024-02-19T15:23:25.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.207+0000 7fee61a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee5c0cdc60 con 0x7fee5c0d0820 2024-02-19T15:23:25.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.207+0000 7fee61a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c0d2a40 con 0x7fee5c0c8260 2024-02-19T15:23:25.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.207+0000 7fee53e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fee5c0d0820 0x7fee5c0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:25.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.207+0000 7fee53e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fee5c0d0820 0x7fee5c0cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60154/0 (socket says 172.21.15.47:60154) 2024-02-19T15:23:25.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.207+0000 7fee53e00640 1 -- 172.21.15.47:0/2624251243 learned_addr learned my addr 172.21.15.47:0/2624251243 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:25.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.207+0000 7fee5b400640 1 --2- 172.21.15.47:0/2624251243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee5c0c8260 0x7fee5c0ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:25.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.207+0000 7fee53e00640 1 -- 172.21.15.47:0/2624251243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee5c0c8260 msgr2=0x7fee5c0ccc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:25.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.207+0000 7fee53e00640 1 --2- 172.21.15.47:0/2624251243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee5c0c8260 0x7fee5c0ccc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.207+0000 7fee53e00640 1 -- 172.21.15.47:0/2624251243 >> v1:172.21.15.116:6789/0 conn(0x7fee5c0c7e80 legacy=0x7fee5c0d0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:25.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.207+0000 7fee53e00640 1 -- 172.21.15.47:0/2624251243 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5c0cd980 con 0x7fee5c0d0820 2024-02-19T15:23:25.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.211+0000 7fee53e00640 1 --2- 172.21.15.47:0/2624251243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee5c0d0820 0x7fee5c0cc740 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fee44009870 tx=0x7fee4401c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ba8a3872a6eae3a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:25.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.211+0000 7fee5a000640 1 -- 172.21.15.47:0/2624251243 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee440aa5f0 con 0x7fee5c0d0820 2024-02-19T15:23:25.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.211+0000 7fee5a000640 1 -- 172.21.15.47:0/2624251243 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee440aa750 con 0x7fee5c0d0820 2024-02-19T15:23:25.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.211+0000 7fee5a000640 1 -- 172.21.15.47:0/2624251243 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee440aaae0 con 0x7fee5c0d0820 2024-02-19T15:23:25.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.211+0000 7fee61a00640 1 -- 172.21.15.47:0/2624251243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee5c0d0820 msgr2=0x7fee5c0cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:25.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.211+0000 7fee61a00640 1 --2- 172.21.15.47:0/2624251243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee5c0d0820 0x7fee5c0cc740 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fee44009870 tx=0x7fee4401c530 comp rx=0 tx=0).stop 2024-02-19T15:23:25.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.211+0000 7fee61a00640 1 -- 172.21.15.47:0/2624251243 shutdown_connections 2024-02-19T15:23:25.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.211+0000 7fee61a00640 1 --2- 172.21.15.47:0/2624251243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee5c0c8260 0x7fee5c0ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.211+0000 7fee61a00640 1 --2- 172.21.15.47:0/2624251243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee5c0d0820 0x7fee5c0cc740 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.211+0000 7fee61a00640 1 -- 172.21.15.47:0/2624251243 >> 172.21.15.47:0/2624251243 conn(0x7fee5c0d1b30 msgr2=0x7fee5c0d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:25.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.211+0000 7fee61a00640 1 -- 172.21.15.47:0/2624251243 shutdown_connections 2024-02-19T15:23:25.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.211+0000 7fee61a00640 1 -- 172.21.15.47:0/2624251243 wait complete. 2024-02-19T15:23:25.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee61a00640 1 Processor -- start 2024-02-19T15:23:25.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee61a00640 1 -- start start 2024-02-19T15:23:25.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee61a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee5c0c7e80 0x7fee5c0aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:25.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee61a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee5c0c8260 0x7fee5c0af190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:25.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee61a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee5c0d0820 0x7fee5c06d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:25.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee61a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c1fadb0 con 0x7fee5c0c8260 2024-02-19T15:23:25.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee61a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c1483e0 con 0x7fee5c0c7e80 2024-02-19T15:23:25.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee61a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c1fa480 con 0x7fee5c0d0820 2024-02-19T15:23:25.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee53e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee5c0c8260 0x7fee5c0af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:25.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee53e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee5c0c8260 0x7fee5c0af190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60160/0 (socket says 172.21.15.47:60160) 2024-02-19T15:23:25.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee53e00640 1 -- 172.21.15.47:0/4220471411 learned_addr learned my addr 172.21.15.47:0/4220471411 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:25.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee5aa00640 1 --2- 172.21.15.47:0/4220471411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee5c0c7e80 0x7fee5c0aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:25.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee5b400640 1 --2- 172.21.15.47:0/4220471411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee5c0d0820 0x7fee5c06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:25.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee53e00640 1 -- 172.21.15.47:0/4220471411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee5c0d0820 msgr2=0x7fee5c06d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:25.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee53e00640 1 --2- 172.21.15.47:0/4220471411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee5c0d0820 0x7fee5c06d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee53e00640 1 -- 172.21.15.47:0/4220471411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee5c0c7e80 msgr2=0x7fee5c0aec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:25.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee53e00640 1 --2- 172.21.15.47:0/4220471411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee5c0c7e80 0x7fee5c0aec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee53e00640 1 -- 172.21.15.47:0/4220471411 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee440094d0 con 0x7fee5c0c8260 2024-02-19T15:23:25.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee5b400640 1 --2- 172.21.15.47:0/4220471411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee5c0d0820 0x7fee5c06d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:25.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee5aa00640 1 --2- 172.21.15.47:0/4220471411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee5c0c7e80 0x7fee5c0aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:25.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee53e00640 1 --2- 172.21.15.47:0/4220471411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee5c0c8260 0x7fee5c0af190 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fee4401a160 tx=0x7fee4401ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:25.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee52a00640 1 -- 172.21.15.47:0/4220471411 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee440bd5f0 con 0x7fee5c0c8260 2024-02-19T15:23:25.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee52a00640 1 -- 172.21.15.47:0/4220471411 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee440bd770 con 0x7fee5c0c8260 2024-02-19T15:23:25.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee52a00640 1 -- 172.21.15.47:0/4220471411 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee440bdb20 con 0x7fee5c0c8260 2024-02-19T15:23:25.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee61a00640 1 -- 172.21.15.47:0/4220471411 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee5c1fa760 con 0x7fee5c0c8260 2024-02-19T15:23:25.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.215+0000 7fee61a00640 1 -- 172.21.15.47:0/4220471411 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee5c1fb240 con 0x7fee5c0c8260 2024-02-19T15:23:25.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.219+0000 7fee61a00640 1 -- 172.21.15.47:0/4220471411 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee5c0d1180 con 0x7fee5c0c8260 2024-02-19T15:23:25.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.219+0000 7fee52a00640 1 -- 172.21.15.47:0/4220471411 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fee440b3600 con 0x7fee5c0c8260 2024-02-19T15:23:25.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.219+0000 7fee52a00640 1 --2- 172.21.15.47:0/4220471411 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fee38066e40 0x7fee38069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:25.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.219+0000 7fee52a00640 1 -- 172.21.15.47:0/4220471411 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee440a7070 con 0x7fee5c0c8260 2024-02-19T15:23:25.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.219+0000 7fee5aa00640 1 --2- 172.21.15.47:0/4220471411 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fee38066e40 0x7fee38069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:25.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.219+0000 7fee5aa00640 1 --2- 172.21.15.47:0/4220471411 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fee38066e40 0x7fee38069300 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fee4c001710 tx=0x7fee4c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:25.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.223+0000 7fee52a00640 1 -- 172.21.15.47:0/4220471411 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fee440ec5a0 con 0x7fee5c0c8260 2024-02-19T15:23:25.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.387+0000 7fee61a00640 1 -- 172.21.15.47:0/4220471411 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fee5c075360 con 0x7fee5c0c8260 2024-02-19T15:23:25.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.387+0000 7fee52a00640 1 -- 172.21.15.47:0/4220471411 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7fee440efc60 con 0x7fee5c0c8260 2024-02-19T15:23:25.392 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:23:25.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.387+0000 7fee61a00640 1 -- 172.21.15.47:0/4220471411 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fee38066e40 msgr2=0x7fee38069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:25.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.387+0000 7fee61a00640 1 --2- 172.21.15.47:0/4220471411 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fee38066e40 0x7fee38069300 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fee4c001710 tx=0x7fee4c009040 comp rx=0 tx=0).stop 2024-02-19T15:23:25.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.387+0000 7fee61a00640 1 -- 172.21.15.47:0/4220471411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee5c0c8260 msgr2=0x7fee5c0af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:25.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.387+0000 7fee61a00640 1 --2- 172.21.15.47:0/4220471411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee5c0c8260 0x7fee5c0af190 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fee4401a160 tx=0x7fee4401ad70 comp rx=0 tx=0).stop 2024-02-19T15:23:25.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.387+0000 7fee61a00640 1 -- 172.21.15.47:0/4220471411 shutdown_connections 2024-02-19T15:23:25.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.387+0000 7fee61a00640 1 --2- 172.21.15.47:0/4220471411 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fee38066e40 0x7fee38069300 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.387+0000 7fee61a00640 1 --2- 172.21.15.47:0/4220471411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee5c0d0820 0x7fee5c06d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.387+0000 7fee61a00640 1 --2- 172.21.15.47:0/4220471411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee5c0c8260 0x7fee5c0af190 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.387+0000 7fee61a00640 1 --2- 172.21.15.47:0/4220471411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee5c0c7e80 0x7fee5c0aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.387+0000 7fee61a00640 1 -- 172.21.15.47:0/4220471411 >> 172.21.15.47:0/4220471411 conn(0x7fee5c0d1b30 msgr2=0x7fee5c0c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:25.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.387+0000 7fee61a00640 1 -- 172.21.15.47:0/4220471411 shutdown_connections 2024-02-19T15:23:25.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.387+0000 7fee61a00640 1 -- 172.21.15.47:0/4220471411 wait complete. 2024-02-19T15:23:25.413 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:23:25.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.551+0000 7fba3c800640 1 Processor -- start 2024-02-19T15:23:25.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.551+0000 7fba3c800640 1 -- start start 2024-02-19T15:23:25.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.551+0000 7fba3c800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fba340cd7f0 0x7fba340cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:25.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.551+0000 7fba3c800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fba3415f340 0x7fba340ad850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:25.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.551+0000 7fba3c800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba3406d700 con 0x7fba3406e380 2024-02-19T15:23:25.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.551+0000 7fba3c800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba340ce110 con 0x7fba3415f340 2024-02-19T15:23:25.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.551+0000 7fba3c800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3406caf0 con 0x7fba340cd7f0 2024-02-19T15:23:25.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.551+0000 7fba39400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fba3406e380 0x7fba3415ec30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:32966/0 (socket says 172.21.15.47:32966) 2024-02-19T15:23:25.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.551+0000 7fba39400640 1 -- 172.21.15.47:0/2779688634 learned_addr learned my addr 172.21.15.47:0/2779688634 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:25.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.551+0000 7fba39e00640 1 --2- 172.21.15.47:0/2779688634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fba340cd7f0 0x7fba340cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:25.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.551+0000 7fba3a800640 1 --2- 172.21.15.47:0/2779688634 >> v2:172.21.15.116:3300/0 conn(0x7fba3415f340 0x7fba340ad850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:25.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.551+0000 7fba38a00640 1 -- 172.21.15.47:0/2779688634 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624152508 0 0) 0x7fba3406d700 con 0x7fba3406e380 2024-02-19T15:23:25.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.551+0000 7fba38a00640 1 -- 172.21.15.47:0/2779688634 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba1c003530 con 0x7fba3406e380 2024-02-19T15:23:25.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.555+0000 7fba38a00640 1 -- 172.21.15.47:0/2779688634 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 812240837 0 0) 0x7fba1c003530 con 0x7fba3406e380 2024-02-19T15:23:25.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.555+0000 7fba39e00640 1 -- 172.21.15.47:0/2779688634 >> v2:172.21.15.116:3300/0 conn(0x7fba3415f340 msgr2=0x7fba340ad850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:25.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.555+0000 7fba39e00640 1 --2- 172.21.15.47:0/2779688634 >> v2:172.21.15.116:3300/0 conn(0x7fba3415f340 0x7fba340ad850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.555+0000 7fba39e00640 1 -- 172.21.15.47:0/2779688634 >> v1:172.21.15.47:6789/0 conn(0x7fba3406e380 legacy=0x7fba3415ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:25.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.555+0000 7fba39e00640 1 -- 172.21.15.47:0/2779688634 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3406cdd0 con 0x7fba340cd7f0 2024-02-19T15:23:25.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.555+0000 7fba39e00640 1 --2- 172.21.15.47:0/2779688634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fba340cd7f0 0x7fba340cdbd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fba280097c0 tx=0x7fba2801c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79b6489975d94df6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:25.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.555+0000 7fba38a00640 1 -- 172.21.15.47:0/2779688634 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba24001680 con 0x7fba340cd7f0 2024-02-19T15:23:25.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.555+0000 7fba38a00640 1 -- 172.21.15.47:0/2779688634 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba280a7050 con 0x7fba340cd7f0 2024-02-19T15:23:25.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.555+0000 7fba38a00640 1 -- 172.21.15.47:0/2779688634 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba280ab620 con 0x7fba340cd7f0 2024-02-19T15:23:25.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba3c800640 1 -- 172.21.15.47:0/2779688634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fba340cd7f0 msgr2=0x7fba340cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:25.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba3c800640 1 --2- 172.21.15.47:0/2779688634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fba340cd7f0 0x7fba340cdbd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fba280097c0 tx=0x7fba2801c300 comp rx=0 tx=0).stop 2024-02-19T15:23:25.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba3c800640 1 -- 172.21.15.47:0/2779688634 shutdown_connections 2024-02-19T15:23:25.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba3c800640 1 --2- 172.21.15.47:0/2779688634 >> v2:172.21.15.116:3300/0 conn(0x7fba3415f340 0x7fba340ad850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba3c800640 1 --2- 172.21.15.47:0/2779688634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fba340cd7f0 0x7fba340cdbd0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba3c800640 1 -- 172.21.15.47:0/2779688634 >> 172.21.15.47:0/2779688634 conn(0x7fba340cfc20 msgr2=0x7fba340d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:25.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba3c800640 1 -- 172.21.15.47:0/2779688634 shutdown_connections 2024-02-19T15:23:25.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba3c800640 1 -- 172.21.15.47:0/2779688634 wait complete. 2024-02-19T15:23:25.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba3c800640 1 Processor -- start 2024-02-19T15:23:25.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba3c800640 1 -- start start 2024-02-19T15:23:25.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba3c800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fba3406e380 0x7fba34154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:25.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba3c800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba340cd7f0 0x7fba34157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:25.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba3c800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba3415f340 0x7fba341524c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:25.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba3c800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba34156010 con 0x7fba340cd7f0 2024-02-19T15:23:25.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba3c800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba34155770 con 0x7fba3415f340 2024-02-19T15:23:25.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba3c800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba34155a50 con 0x7fba3406e380 2024-02-19T15:23:25.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba39400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba340cd7f0 0x7fba34157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:25.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba39400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba340cd7f0 0x7fba34157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60172/0 (socket says 172.21.15.47:60172) 2024-02-19T15:23:25.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba39400640 1 -- 172.21.15.47:0/3449297982 learned_addr learned my addr 172.21.15.47:0/3449297982 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:25.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba39e00640 1 --2- 172.21.15.47:0/3449297982 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fba3406e380 0x7fba34154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:25.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba3a800640 1 --2- 172.21.15.47:0/3449297982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba3415f340 0x7fba341524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:25.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba39400640 1 -- 172.21.15.47:0/3449297982 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fba3406e380 msgr2=0x7fba34154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:25.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba39400640 1 --2- 172.21.15.47:0/3449297982 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fba3406e380 0x7fba34154e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba39400640 1 -- 172.21.15.47:0/3449297982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba3415f340 msgr2=0x7fba341524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:25.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba39400640 1 --2- 172.21.15.47:0/3449297982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba3415f340 0x7fba341524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.559+0000 7fba39400640 1 -- 172.21.15.47:0/3449297982 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba28009420 con 0x7fba340cd7f0 2024-02-19T15:23:25.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.563+0000 7fba39400640 1 --2- 172.21.15.47:0/3449297982 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba340cd7f0 0x7fba34157a90 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fba2400c750 tx=0x7fba2400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:25.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.563+0000 7fba2e000640 1 -- 172.21.15.47:0/3449297982 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba24005ae0 con 0x7fba340cd7f0 2024-02-19T15:23:25.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.563+0000 7fba2e000640 1 -- 172.21.15.47:0/3449297982 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba24005c40 con 0x7fba340cd7f0 2024-02-19T15:23:25.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.563+0000 7fba2e000640 1 -- 172.21.15.47:0/3449297982 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba24007070 con 0x7fba340cd7f0 2024-02-19T15:23:25.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.563+0000 7fba3c800640 1 -- 172.21.15.47:0/3449297982 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba34155d30 con 0x7fba340cd7f0 2024-02-19T15:23:25.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.563+0000 7fba3c800640 1 -- 172.21.15.47:0/3449297982 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba341553b0 con 0x7fba340cd7f0 2024-02-19T15:23:25.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.563+0000 7fba3c800640 1 -- 172.21.15.47:0/3449297982 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba3413fb90 con 0x7fba340cd7f0 2024-02-19T15:23:25.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.563+0000 7fba2e000640 1 -- 172.21.15.47:0/3449297982 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fba240071d0 con 0x7fba340cd7f0 2024-02-19T15:23:25.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.563+0000 7fba2e000640 1 --2- 172.21.15.47:0/3449297982 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fba18066e40 0x7fba18069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:25.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.563+0000 7fba2e000640 1 -- 172.21.15.47:0/3449297982 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba24084840 con 0x7fba340cd7f0 2024-02-19T15:23:25.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.563+0000 7fba39e00640 1 --2- 172.21.15.47:0/3449297982 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fba18066e40 0x7fba18069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:25.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.567+0000 7fba39e00640 1 --2- 172.21.15.47:0/3449297982 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fba18066e40 0x7fba18069300 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fba280ad000 tx=0x7fba280a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:25.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.567+0000 7fba2e000640 1 -- 172.21.15.47:0/3449297982 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fba3413fb90 con 0x7fba340cd7f0 2024-02-19T15:23:25.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.727+0000 7fba3c800640 1 -- 172.21.15.47:0/3449297982 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fba3413fb90 con 0x7fba340cd7f0 2024-02-19T15:23:25.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.731+0000 7fba2e000640 1 -- 172.21.15.47:0/3449297982 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) v1 ==== 104+0+11 (secure 0 0 0) 0x7fba3413fb90 con 0x7fba340cd7f0 2024-02-19T15:23:25.735 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:23:25.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.731+0000 7fba3c800640 1 -- 172.21.15.47:0/3449297982 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fba18066e40 msgr2=0x7fba18069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:25.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.731+0000 7fba3c800640 1 --2- 172.21.15.47:0/3449297982 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fba18066e40 0x7fba18069300 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fba280ad000 tx=0x7fba280a6000 comp rx=0 tx=0).stop 2024-02-19T15:23:25.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.731+0000 7fba3c800640 1 -- 172.21.15.47:0/3449297982 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba340cd7f0 msgr2=0x7fba34157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:25.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.731+0000 7fba3c800640 1 --2- 172.21.15.47:0/3449297982 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba340cd7f0 0x7fba34157a90 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fba2400c750 tx=0x7fba2400cc10 comp rx=0 tx=0).stop 2024-02-19T15:23:25.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.731+0000 7fba3c800640 1 -- 172.21.15.47:0/3449297982 shutdown_connections 2024-02-19T15:23:25.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.731+0000 7fba3c800640 1 --2- 172.21.15.47:0/3449297982 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fba18066e40 0x7fba18069300 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.731+0000 7fba3c800640 1 --2- 172.21.15.47:0/3449297982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba3415f340 0x7fba341524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.731+0000 7fba3c800640 1 --2- 172.21.15.47:0/3449297982 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba340cd7f0 0x7fba34157a90 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.731+0000 7fba3c800640 1 --2- 172.21.15.47:0/3449297982 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fba3406e380 0x7fba34154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.731+0000 7fba3c800640 1 -- 172.21.15.47:0/3449297982 >> 172.21.15.47:0/3449297982 conn(0x7fba340cfc20 msgr2=0x7fba340af950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:25.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.731+0000 7fba3c800640 1 -- 172.21.15.47:0/3449297982 shutdown_connections 2024-02-19T15:23:25.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.731+0000 7fba3c800640 1 -- 172.21.15.47:0/3449297982 wait complete. 2024-02-19T15:23:25.757 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:23:25.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.891+0000 7fc4ca400640 1 Processor -- start 2024-02-19T15:23:25.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.895+0000 7fc4ca400640 1 -- start start 2024-02-19T15:23:25.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.895+0000 7fc4ca400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc4c40c7e80 0x7fc4c40d0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:25.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.895+0000 7fc4ca400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4c40cfbf0 0x7fc4c40d0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:25.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.895+0000 7fc4ca400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c40cd750 con 0x7fc4c40c8260 2024-02-19T15:23:25.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.895+0000 7fc4ca400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4c40ceee0 con 0x7fc4c40c7e80 2024-02-19T15:23:25.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.895+0000 7fc4ca400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c40cdc60 con 0x7fc4c40cfbf0 2024-02-19T15:23:25.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.895+0000 7fc4c3400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc4c40c7e80 0x7fc4c40d0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:25.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.895+0000 7fc4bbe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc4c40c8260 0x7fc4c40c5680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:32982/0 (socket says 172.21.15.47:32982) 2024-02-19T15:23:25.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.895+0000 7fc4bbe00640 1 -- 172.21.15.47:0/4178194723 learned_addr learned my addr 172.21.15.47:0/4178194723 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:25.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.895+0000 7fc4c3e00640 1 --2- 172.21.15.47:0/4178194723 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4c40cfbf0 0x7fc4c40d0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:25.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.895+0000 7fc4c3400640 1 -- 172.21.15.47:0/4178194723 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4c40cfbf0 msgr2=0x7fc4c40d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:25.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.895+0000 7fc4c3400640 1 --2- 172.21.15.47:0/4178194723 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4c40cfbf0 0x7fc4c40d0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.895+0000 7fc4c3400640 1 -- 172.21.15.47:0/4178194723 >> v1:172.21.15.47:6789/0 conn(0x7fc4c40c8260 legacy=0x7fc4c40c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:25.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.895+0000 7fc4c3400640 1 -- 172.21.15.47:0/4178194723 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c40d2a40 con 0x7fc4c40c7e80 2024-02-19T15:23:25.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.895+0000 7fc4c2a00640 1 -- 172.21.15.47:0/4178194723 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478648318 0 0) 0x7fc4c40cd750 con 0x7fc4c40c8260 2024-02-19T15:23:25.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.895+0000 7fc4c3e00640 1 --2- 172.21.15.47:0/4178194723 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4c40cfbf0 0x7fc4c40d0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:23:25.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.899+0000 7fc4c3400640 1 --2- 172.21.15.47:0/4178194723 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4c40c7e80 0x7fc4c40d0110 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc4b40097c0 tx=0x7fc4b401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef2614a71d9734b7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:25.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.899+0000 7fc4c2a00640 1 -- 172.21.15.47:0/4178194723 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b40ab670 con 0x7fc4c40c7e80 2024-02-19T15:23:25.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.899+0000 7fc4c2a00640 1 -- 172.21.15.47:0/4178194723 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b40ab7d0 con 0x7fc4c40c7e80 2024-02-19T15:23:25.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.899+0000 7fc4c2a00640 1 -- 172.21.15.47:0/4178194723 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b40abb60 con 0x7fc4c40c7e80 2024-02-19T15:23:25.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.899+0000 7fc4ca400640 1 -- 172.21.15.47:0/4178194723 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4c40c7e80 msgr2=0x7fc4c40d0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:25.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.899+0000 7fc4ca400640 1 --2- 172.21.15.47:0/4178194723 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4c40c7e80 0x7fc4c40d0110 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc4b40097c0 tx=0x7fc4b401c470 comp rx=0 tx=0).stop 2024-02-19T15:23:25.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.899+0000 7fc4ca400640 1 -- 172.21.15.47:0/4178194723 shutdown_connections 2024-02-19T15:23:25.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.899+0000 7fc4ca400640 1 --2- 172.21.15.47:0/4178194723 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4c40cfbf0 0x7fc4c40d0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.899+0000 7fc4ca400640 1 --2- 172.21.15.47:0/4178194723 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4c40c7e80 0x7fc4c40d0110 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.899+0000 7fc4ca400640 1 -- 172.21.15.47:0/4178194723 >> 172.21.15.47:0/4178194723 conn(0x7fc4c40d1b30 msgr2=0x7fc4c40d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:25.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.899+0000 7fc4ca400640 1 -- 172.21.15.47:0/4178194723 shutdown_connections 2024-02-19T15:23:25.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.899+0000 7fc4ca400640 1 -- 172.21.15.47:0/4178194723 wait complete. 2024-02-19T15:23:25.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.903+0000 7fc4ca400640 1 Processor -- start 2024-02-19T15:23:25.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.903+0000 7fc4ca400640 1 -- start start 2024-02-19T15:23:25.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.903+0000 7fc4ca400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4c40c7e80 0x7fc4c406da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:25.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.903+0000 7fc4ca400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4c40c8260 0x7fc4c41664e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:25.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.903+0000 7fc4ca400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4c40cfbf0 0x7fc4c41576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:25.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.903+0000 7fc4ca400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c40ca150 con 0x7fc4c40c8260 2024-02-19T15:23:25.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.903+0000 7fc4ca400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c415f670 con 0x7fc4c40c7e80 2024-02-19T15:23:25.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.903+0000 7fc4ca400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c415f0b0 con 0x7fc4c40cfbf0 2024-02-19T15:23:25.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.903+0000 7fc4bbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4c40c8260 0x7fc4c41664e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:25.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.903+0000 7fc4bbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4c40c8260 0x7fc4c41664e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60194/0 (socket says 172.21.15.47:60194) 2024-02-19T15:23:25.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.903+0000 7fc4bbe00640 1 -- 172.21.15.47:0/1667129042 learned_addr learned my addr 172.21.15.47:0/1667129042 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:25.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.903+0000 7fc4c3400640 1 --2- 172.21.15.47:0/1667129042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4c40c7e80 0x7fc4c406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:25.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.903+0000 7fc4c3e00640 1 --2- 172.21.15.47:0/1667129042 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4c40cfbf0 0x7fc4c41576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:25.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.903+0000 7fc4bbe00640 1 -- 172.21.15.47:0/1667129042 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4c40cfbf0 msgr2=0x7fc4c41576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:25.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.903+0000 7fc4bbe00640 1 --2- 172.21.15.47:0/1667129042 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4c40cfbf0 0x7fc4c41576d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.903+0000 7fc4bbe00640 1 -- 172.21.15.47:0/1667129042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4c40c7e80 msgr2=0x7fc4c406da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:25.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.903+0000 7fc4bbe00640 1 --2- 172.21.15.47:0/1667129042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4c40c7e80 0x7fc4c406da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:25.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.903+0000 7fc4bbe00640 1 -- 172.21.15.47:0/1667129042 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b4009420 con 0x7fc4c40c8260 2024-02-19T15:23:25.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.907+0000 7fc4bbe00640 1 --2- 172.21.15.47:0/1667129042 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4c40c8260 0x7fc4c41664e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac00c480 tx=0x7fc4ac00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:25.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.907+0000 7fc4bb400640 1 -- 172.21.15.47:0/1667129042 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac0074e0 con 0x7fc4c40c8260 2024-02-19T15:23:25.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.907+0000 7fc4bb400640 1 -- 172.21.15.47:0/1667129042 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4ac007660 con 0x7fc4c40c8260 2024-02-19T15:23:25.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.907+0000 7fc4ca400640 1 -- 172.21.15.47:0/1667129042 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c415f390 con 0x7fc4c40c8260 2024-02-19T15:23:25.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.907+0000 7fc4bb400640 1 -- 172.21.15.47:0/1667129042 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac004a30 con 0x7fc4c40c8260 2024-02-19T15:23:25.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.907+0000 7fc4ca400640 1 -- 172.21.15.47:0/1667129042 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c406df50 con 0x7fc4c40c8260 2024-02-19T15:23:25.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.907+0000 7fc4ca400640 1 -- 172.21.15.47:0/1667129042 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c40d1180 con 0x7fc4c40c8260 2024-02-19T15:23:25.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.907+0000 7fc4bb400640 1 -- 172.21.15.47:0/1667129042 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc4ac004b90 con 0x7fc4c40c8260 2024-02-19T15:23:25.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.907+0000 7fc4bb400640 1 --2- 172.21.15.47:0/1667129042 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc4a0066e40 0x7fc4a0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:25.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.907+0000 7fc4bb400640 1 -- 172.21.15.47:0/1667129042 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4ac083f10 con 0x7fc4c40c8260 2024-02-19T15:23:25.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.911+0000 7fc4c3400640 1 --2- 172.21.15.47:0/1667129042 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc4a0066e40 0x7fc4a0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:25.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.911+0000 7fc4c3400640 1 --2- 172.21.15.47:0/1667129042 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc4a0066e40 0x7fc4a0069300 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc4b40af000 tx=0x7fc4b40a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:25.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:25.911+0000 7fc4bb400640 1 -- 172.21.15.47:0/1667129042 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc4ac050da0 con 0x7fc4c40c8260 2024-02-19T15:23:26.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.075+0000 7fc4ca400640 1 -- 172.21.15.47:0/1667129042 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc4c4075360 con 0x7fc4c40c8260 2024-02-19T15:23:26.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.075+0000 7fc4bb400640 1 -- 172.21.15.47:0/1667129042 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc4ac018020 con 0x7fc4c40c8260 2024-02-19T15:23:26.082 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:23:26.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.079+0000 7fc4ca400640 1 -- 172.21.15.47:0/1667129042 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc4a0066e40 msgr2=0x7fc4a0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:26.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.079+0000 7fc4ca400640 1 --2- 172.21.15.47:0/1667129042 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc4a0066e40 0x7fc4a0069300 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc4b40af000 tx=0x7fc4b40a7000 comp rx=0 tx=0).stop 2024-02-19T15:23:26.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.079+0000 7fc4ca400640 1 -- 172.21.15.47:0/1667129042 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4c40c8260 msgr2=0x7fc4c41664e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:26.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.079+0000 7fc4ca400640 1 --2- 172.21.15.47:0/1667129042 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4c40c8260 0x7fc4c41664e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac00c480 tx=0x7fc4ac00c940 comp rx=0 tx=0).stop 2024-02-19T15:23:26.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.079+0000 7fc4ca400640 1 -- 172.21.15.47:0/1667129042 shutdown_connections 2024-02-19T15:23:26.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.079+0000 7fc4ca400640 1 --2- 172.21.15.47:0/1667129042 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc4a0066e40 0x7fc4a0069300 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:26.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.079+0000 7fc4ca400640 1 --2- 172.21.15.47:0/1667129042 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4c40cfbf0 0x7fc4c41576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:26.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.079+0000 7fc4ca400640 1 --2- 172.21.15.47:0/1667129042 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4c40c8260 0x7fc4c41664e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:26.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.079+0000 7fc4ca400640 1 --2- 172.21.15.47:0/1667129042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4c40c7e80 0x7fc4c406da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:26.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.079+0000 7fc4ca400640 1 -- 172.21.15.47:0/1667129042 >> 172.21.15.47:0/1667129042 conn(0x7fc4c40d1b30 msgr2=0x7fc4c40c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:26.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.079+0000 7fc4ca400640 1 -- 172.21.15.47:0/1667129042 shutdown_connections 2024-02-19T15:23:26.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.079+0000 7fc4ca400640 1 -- 172.21.15.47:0/1667129042 wait complete. 2024-02-19T15:23:26.104 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-19T15:23:26.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.239+0000 7f251aa00640 1 Processor -- start 2024-02-19T15:23:26.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.239+0000 7f251aa00640 1 -- start start 2024-02-19T15:23:26.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.239+0000 7f251aa00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25140ced10 0x7f25141690a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:26.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.239+0000 7f251aa00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25140cce30 con 0x7f25140cded0 2024-02-19T15:23:26.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.239+0000 7f251aa00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25140cd110 con 0x7f251406d650 2024-02-19T15:23:26.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.239+0000 7f251aa00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25140cd3f0 con 0x7f25140ced10 2024-02-19T15:23:26.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.239+0000 7f250b400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f25140cded0 0x7f25140ce600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:32992/0 (socket says 172.21.15.47:32992) 2024-02-19T15:23:26.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.239+0000 7f250b400640 1 -- 172.21.15.47:0/3069407362 learned_addr learned my addr 172.21.15.47:0/3069407362 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:26.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.239+0000 7f2518a00640 1 --2- 172.21.15.47:0/3069407362 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25140ced10 0x7f25141690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:26.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.243+0000 7f2513400640 1 -- 172.21.15.47:0/3069407362 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2264826295 0 0) 0x7f25140cce30 con 0x7f25140cded0 2024-02-19T15:23:26.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.243+0000 7f2513400640 1 -- 172.21.15.47:0/3069407362 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f8003530 con 0x7f25140cded0 2024-02-19T15:23:26.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.243+0000 7f2513400640 1 -- 172.21.15.47:0/3069407362 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653301278 0 0) 0x7f25140cd110 con 0x7f251406d650 2024-02-19T15:23:26.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.243+0000 7f2513400640 1 -- 172.21.15.47:0/3069407362 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25140cce30 con 0x7f251406d650 2024-02-19T15:23:26.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.243+0000 7f2518a00640 1 -- 172.21.15.47:0/3069407362 >> v1:172.21.15.116:6789/0 conn(0x7f251406d650 legacy=0x7f25140d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:26.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.243+0000 7f2518a00640 1 -- 172.21.15.47:0/3069407362 >> v1:172.21.15.47:6789/0 conn(0x7f25140cded0 legacy=0x7f25140ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:26.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.243+0000 7f2518a00640 1 -- 172.21.15.47:0/3069407362 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25140cd6d0 con 0x7f25140ced10 2024-02-19T15:23:26.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.243+0000 7f2518a00640 1 --2- 172.21.15.47:0/3069407362 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25140ced10 0x7f25141690a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f250c0097f0 tx=0x7f250c01c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba94d79a995d7062 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:26.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.243+0000 7f2513400640 1 -- 172.21.15.47:0/3069407362 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c0ab640 con 0x7f25140ced10 2024-02-19T15:23:26.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.243+0000 7f2513400640 1 -- 172.21.15.47:0/3069407362 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250c0a7070 con 0x7f25140ced10 2024-02-19T15:23:26.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.243+0000 7f2513400640 1 -- 172.21.15.47:0/3069407362 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c0ab9c0 con 0x7f25140ced10 2024-02-19T15:23:26.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.247+0000 7f251aa00640 1 -- 172.21.15.47:0/3069407362 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25140ced10 msgr2=0x7f25141690a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:26.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.247+0000 7f251aa00640 1 --2- 172.21.15.47:0/3069407362 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25140ced10 0x7f25141690a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f250c0097f0 tx=0x7f250c01c420 comp rx=0 tx=0).stop 2024-02-19T15:23:26.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.247+0000 7f251aa00640 1 -- 172.21.15.47:0/3069407362 shutdown_connections 2024-02-19T15:23:26.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.247+0000 7f251aa00640 1 --2- 172.21.15.47:0/3069407362 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25140ced10 0x7f25141690a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:26.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.247+0000 7f251aa00640 1 -- 172.21.15.47:0/3069407362 >> 172.21.15.47:0/3069407362 conn(0x7f25140d1bf0 msgr2=0x7f25140d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:26.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.247+0000 7f251aa00640 1 -- 172.21.15.47:0/3069407362 shutdown_connections 2024-02-19T15:23:26.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.247+0000 7f251aa00640 1 -- 172.21.15.47:0/3069407362 wait complete. 2024-02-19T15:23:26.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.247+0000 7f251aa00640 1 Processor -- start 2024-02-19T15:23:26.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.247+0000 7f251aa00640 1 -- start start 2024-02-19T15:23:26.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f251aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f251406d650 0x7f25141557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:26.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f251aa00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25140cded0 0x7f25140cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:26.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f251aa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25140ced10 0x7f25140d03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:26.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f251aa00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251415af90 con 0x7f25140ced10 2024-02-19T15:23:26.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f251aa00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25140c9d80 con 0x7f251406d650 2024-02-19T15:23:26.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f251aa00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25141614c0 con 0x7f25140cded0 2024-02-19T15:23:26.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f2518a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25140ced10 0x7f25140d03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:26.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f2518a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25140ced10 0x7f25140d03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60196/0 (socket says 172.21.15.47:60196) 2024-02-19T15:23:26.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f2518a00640 1 -- 172.21.15.47:0/859333724 learned_addr learned my addr 172.21.15.47:0/859333724 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:26.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f250b400640 1 --2- 172.21.15.47:0/859333724 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25140cded0 0x7f25140cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:26.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f2513e00640 1 --2- 172.21.15.47:0/859333724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f251406d650 0x7f25141557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:26.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f2518a00640 1 -- 172.21.15.47:0/859333724 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25140cded0 msgr2=0x7f25140cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:26.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f2518a00640 1 --2- 172.21.15.47:0/859333724 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25140cded0 0x7f25140cfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:26.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f2518a00640 1 -- 172.21.15.47:0/859333724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f251406d650 msgr2=0x7f25141557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:26.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f2518a00640 1 --2- 172.21.15.47:0/859333724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f251406d650 0x7f25141557e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:26.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f2518a00640 1 -- 172.21.15.47:0/859333724 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250c009420 con 0x7f25140ced10 2024-02-19T15:23:26.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f2513e00640 1 --2- 172.21.15.47:0/859333724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f251406d650 0x7f25141557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:26.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f2518a00640 1 --2- 172.21.15.47:0/859333724 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25140ced10 0x7f25140d03c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f250c009740 tx=0x7f250c0191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:26.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f2510c00640 1 -- 172.21.15.47:0/859333724 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c0ab970 con 0x7f25140ced10 2024-02-19T15:23:26.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f2510c00640 1 -- 172.21.15.47:0/859333724 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250c0abad0 con 0x7f25140ced10 2024-02-19T15:23:26.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f251aa00640 1 -- 172.21.15.47:0/859333724 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25140c9450 con 0x7f25140ced10 2024-02-19T15:23:26.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f2510c00640 1 -- 172.21.15.47:0/859333724 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c0abe60 con 0x7f25140ced10 2024-02-19T15:23:26.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f251aa00640 1 -- 172.21.15.47:0/859333724 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25140c9a10 con 0x7f25140ced10 2024-02-19T15:23:26.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.251+0000 7f251aa00640 1 -- 172.21.15.47:0/859333724 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25140c9f10 con 0x7f25140ced10 2024-02-19T15:23:26.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.255+0000 7f2510c00640 1 -- 172.21.15.47:0/859333724 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f250c0c83f0 con 0x7f25140ced10 2024-02-19T15:23:26.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.255+0000 7f2510c00640 1 --2- 172.21.15.47:0/859333724 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f24f0066e40 0x7f24f0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:26.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.255+0000 7f2510c00640 1 -- 172.21.15.47:0/859333724 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f250c11f1d0 con 0x7f25140ced10 2024-02-19T15:23:26.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.255+0000 7f2513e00640 1 --2- 172.21.15.47:0/859333724 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f24f0066e40 0x7f24f0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:26.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.255+0000 7f2513e00640 1 --2- 172.21.15.47:0/859333724 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f24f0066e40 0x7f24f0069300 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2504007fd0 tx=0x7f2504009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:26.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.259+0000 7f2510c00640 1 -- 172.21.15.47:0/859333724 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f250c0ebfc0 con 0x7f25140ced10 2024-02-19T15:23:26.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.559+0000 7f251aa00640 1 -- 172.21.15.47:0/859333724 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2514075360 con 0x7f25140ced10 2024-02-19T15:23:26.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.559+0000 7f2510c00640 1 -- 172.21.15.47:0/859333724 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v34) v1 ==== 80+0+0 (secure 0 0 0) 0x7f250c0ef680 con 0x7f25140ced10 2024-02-19T15:23:26.567 INFO:teuthology.orchestra.run.smithi047.stderr:listed 0 entries 2024-02-19T15:23:26.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.563+0000 7f251aa00640 1 -- 172.21.15.47:0/859333724 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f24f0066e40 msgr2=0x7f24f0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:26.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.563+0000 7f251aa00640 1 --2- 172.21.15.47:0/859333724 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f24f0066e40 0x7f24f0069300 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2504007fd0 tx=0x7f2504009040 comp rx=0 tx=0).stop 2024-02-19T15:23:26.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.563+0000 7f251aa00640 1 -- 172.21.15.47:0/859333724 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25140ced10 msgr2=0x7f25140d03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:26.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.563+0000 7f251aa00640 1 --2- 172.21.15.47:0/859333724 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25140ced10 0x7f25140d03c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f250c009740 tx=0x7f250c0191c0 comp rx=0 tx=0).stop 2024-02-19T15:23:26.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.563+0000 7f251aa00640 1 -- 172.21.15.47:0/859333724 shutdown_connections 2024-02-19T15:23:26.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.563+0000 7f251aa00640 1 --2- 172.21.15.47:0/859333724 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f24f0066e40 0x7f24f0069300 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:26.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.563+0000 7f251aa00640 1 --2- 172.21.15.47:0/859333724 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25140ced10 0x7f25140d03c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:26.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.563+0000 7f251aa00640 1 --2- 172.21.15.47:0/859333724 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25140cded0 0x7f25140cfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:26.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.563+0000 7f251aa00640 1 --2- 172.21.15.47:0/859333724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f251406d650 0x7f25141557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:26.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.563+0000 7f251aa00640 1 -- 172.21.15.47:0/859333724 >> 172.21.15.47:0/859333724 conn(0x7f25140d1bf0 msgr2=0x7f251415d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:26.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.563+0000 7f251aa00640 1 -- 172.21.15.47:0/859333724 shutdown_connections 2024-02-19T15:23:26.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.563+0000 7f251aa00640 1 -- 172.21.15.47:0/859333724 wait complete. 2024-02-19T15:23:26.588 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-02-19T15:23:26.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:26.588 DEBUG:teuthology.orchestra.run.smithi047:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-02-19T15:23:26.632 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi047.front.sepia.ceph.com 2024-02-19T15:23:26.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:26.632 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-19T15:23:26.683 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:23:26.683 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:23:26.692 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:23:26.693 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:23:26.693 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:23:26.738 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:23:26.738 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:23:26.738 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:23:26.746 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:23:26.746 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:23:26.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.911+0000 7fb6dda00640 1 Processor -- start 2024-02-19T15:23:26.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.911+0000 7fb6dda00640 1 -- start start 2024-02-19T15:23:26.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6dda00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d80d0360 0x7fb6d815d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:26.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6dda00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d80cc700 con 0x7fb6d80cedb0 2024-02-19T15:23:26.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6dda00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d80cf300 con 0x7fb6d815bcd0 2024-02-19T15:23:26.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6dda00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d80cf5e0 con 0x7fb6d80d0360 2024-02-19T15:23:26.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6cfe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb6d80cedb0 0x7fb6d80cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33002/0 (socket says 172.21.15.47:33002) 2024-02-19T15:23:26.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6cfe00640 1 -- 172.21.15.47:0/3353762875 learned_addr learned my addr 172.21.15.47:0/3353762875 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:26.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6d7400640 1 --2- 172.21.15.47:0/3353762875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d80d0360 0x7fb6d815d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:26.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6d6000640 1 -- 172.21.15.47:0/3353762875 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253077998 0 0) 0x7fb6d80cc700 con 0x7fb6d80cedb0 2024-02-19T15:23:26.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6d6000640 1 -- 172.21.15.47:0/3353762875 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6bc003530 con 0x7fb6d80cedb0 2024-02-19T15:23:26.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6d6000640 1 -- 172.21.15.47:0/3353762875 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964888281 0 0) 0x7fb6d80cf300 con 0x7fb6d815bcd0 2024-02-19T15:23:26.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6d6000640 1 -- 172.21.15.47:0/3353762875 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6d80cc700 con 0x7fb6d815bcd0 2024-02-19T15:23:26.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6d6000640 1 -- 172.21.15.47:0/3353762875 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1074275116 0 0) 0x7fb6bc003530 con 0x7fb6d80cedb0 2024-02-19T15:23:26.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6d6000640 1 -- 172.21.15.47:0/3353762875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d80d0360 msgr2=0x7fb6d815d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:26.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6d6000640 1 --2- 172.21.15.47:0/3353762875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d80d0360 0x7fb6d815d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:26.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6d6000640 1 -- 172.21.15.47:0/3353762875 >> v1:172.21.15.116:6789/0 conn(0x7fb6d815bcd0 legacy=0x7fb6d80cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:26.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6d6000640 1 -- 172.21.15.47:0/3353762875 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d80cc9a0 con 0x7fb6d80cedb0 2024-02-19T15:23:26.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6d6000640 1 -- 172.21.15.47:0/3353762875 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7fb6c0001600 con 0x7fb6d80cedb0 2024-02-19T15:23:26.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6d6000640 1 -- 172.21.15.47:0/3353762875 >> v1:172.21.15.47:6789/0 conn(0x7fb6d80cedb0 legacy=0x7fb6d80cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:26.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6d6000640 1 --2- 172.21.15.47:0/3353762875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d80d0360 0x7fb6bc003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:26.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6d6000640 1 --2- 172.21.15.47:0/3353762875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6bc0040f0 0x7fb6bc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:26.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6d6000640 1 --2- 172.21.15.47:0/3353762875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6bc004d10 0x7fb6bc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:26.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6d6000640 1 -- 172.21.15.47:0/3353762875 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d80cf5e0 con 0x7fb6bc0040f0 2024-02-19T15:23:26.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6d6000640 1 -- 172.21.15.47:0/3353762875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc00bad0 con 0x7fb6bc004d10 2024-02-19T15:23:26.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6d6000640 1 -- 172.21.15.47:0/3353762875 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc00bc90 con 0x7fb6d80d0360 2024-02-19T15:23:26.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6cfe00640 1 --2- 172.21.15.47:0/3353762875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6bc0040f0 0x7fb6bc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:26.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6d6a00640 1 --2- 172.21.15.47:0/3353762875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d80d0360 0x7fb6bc003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:26.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6d7400640 1 --2- 172.21.15.47:0/3353762875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6bc004d10 0x7fb6bc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:26.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6cfe00640 1 -- 172.21.15.47:0/3353762875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d80d0360 msgr2=0x7fb6bc003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:26.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6cfe00640 1 --2- 172.21.15.47:0/3353762875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d80d0360 0x7fb6bc003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:26.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6cfe00640 1 -- 172.21.15.47:0/3353762875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6bc004d10 msgr2=0x7fb6bc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:26.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6cfe00640 1 --2- 172.21.15.47:0/3353762875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6bc004d10 0x7fb6bc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:26.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.915+0000 7fb6cfe00640 1 -- 172.21.15.47:0/3353762875 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc003530 con 0x7fb6bc0040f0 2024-02-19T15:23:26.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.919+0000 7fb6cfe00640 1 --2- 172.21.15.47:0/3353762875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6bc0040f0 0x7fb6bc0047d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0007f50 tx=0x7fb6c001c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6715dbc5fbc949a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:26.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.919+0000 7fb6d6000640 1 -- 172.21.15.47:0/3353762875 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c0005ba0 con 0x7fb6bc0040f0 2024-02-19T15:23:26.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.919+0000 7fb6d6000640 1 -- 172.21.15.47:0/3353762875 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6c0005d20 con 0x7fb6bc0040f0 2024-02-19T15:23:26.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.919+0000 7fb6d6000640 1 -- 172.21.15.47:0/3353762875 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c00072b0 con 0x7fb6bc0040f0 2024-02-19T15:23:26.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.919+0000 7fb6dda00640 1 -- 172.21.15.47:0/3353762875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6bc0040f0 msgr2=0x7fb6bc0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:26.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.919+0000 7fb6dda00640 1 --2- 172.21.15.47:0/3353762875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6bc0040f0 0x7fb6bc0047d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0007f50 tx=0x7fb6c001c9e0 comp rx=0 tx=0).stop 2024-02-19T15:23:26.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.919+0000 7fb6d7400640 1 -- 172.21.15.47:0/3353762875 reap_dead start 2024-02-19T15:23:26.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.919+0000 7fb6dda00640 1 -- 172.21.15.47:0/3353762875 shutdown_connections 2024-02-19T15:23:26.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.919+0000 7fb6dda00640 1 -- 172.21.15.47:0/3353762875 >> 172.21.15.47:0/3353762875 conn(0x7fb6d80d1bf0 msgr2=0x7fb6d80d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:26.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.919+0000 7fb6dda00640 1 -- 172.21.15.47:0/3353762875 shutdown_connections 2024-02-19T15:23:26.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.919+0000 7fb6dda00640 1 -- 172.21.15.47:0/3353762875 wait complete. 2024-02-19T15:23:26.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.919+0000 7fb6dda00640 1 Processor -- start 2024-02-19T15:23:26.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.919+0000 7fb6dda00640 1 -- start start 2024-02-19T15:23:26.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6dda00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6bc0040f0 0x7fb6d80cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:26.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6dda00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6d81efc00 0x7fb6d81585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:26.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6dda00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d8157c90 0x7fb6d815c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:26.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6dda00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d80cdb90 con 0x7fb6bc0040f0 2024-02-19T15:23:26.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6dda00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d80b3290 con 0x7fb6d81efc00 2024-02-19T15:23:26.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6dda00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d80c5280 con 0x7fb6d8157c90 2024-02-19T15:23:26.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6d6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6bc0040f0 0x7fb6d80cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:26.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6d6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6bc0040f0 0x7fb6d80cb0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60202/0 (socket says 172.21.15.47:60202) 2024-02-19T15:23:26.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6d6a00640 1 -- 172.21.15.47:0/2123441701 learned_addr learned my addr 172.21.15.47:0/2123441701 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:26.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6d7400640 1 --2- 172.21.15.47:0/2123441701 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d8157c90 0x7fb6d815c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:26.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6cfe00640 1 --2- 172.21.15.47:0/2123441701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6d81efc00 0x7fb6d81585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:26.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6d6a00640 1 -- 172.21.15.47:0/2123441701 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d8157c90 msgr2=0x7fb6d815c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:26.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6d6a00640 1 --2- 172.21.15.47:0/2123441701 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d8157c90 0x7fb6d815c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:26.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6d6a00640 1 -- 172.21.15.47:0/2123441701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6d81efc00 msgr2=0x7fb6d81585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:26.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6d6a00640 1 --2- 172.21.15.47:0/2123441701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6d81efc00 0x7fb6d81585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:26.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6d6a00640 1 -- 172.21.15.47:0/2123441701 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d8158b20 con 0x7fb6bc0040f0 2024-02-19T15:23:26.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6d7400640 1 --2- 172.21.15.47:0/2123441701 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d8157c90 0x7fb6d815c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:26.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6cfe00640 1 --2- 172.21.15.47:0/2123441701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6d81efc00 0x7fb6d81585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:26.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6d6a00640 1 --2- 172.21.15.47:0/2123441701 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6bc0040f0 0x7fb6d80cb0d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb6c80128a0 tx=0x7fb6c8012d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:26.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6cea00640 1 -- 172.21.15.47:0/2123441701 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8010bd0 con 0x7fb6bc0040f0 2024-02-19T15:23:26.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6cea00640 1 -- 172.21.15.47:0/2123441701 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6c8010d50 con 0x7fb6bc0040f0 2024-02-19T15:23:26.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6cea00640 1 -- 172.21.15.47:0/2123441701 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c80166c0 con 0x7fb6bc0040f0 2024-02-19T15:23:26.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6dda00640 1 -- 172.21.15.47:0/2123441701 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6d80c5520 con 0x7fb6bc0040f0 2024-02-19T15:23:26.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6dda00640 1 -- 172.21.15.47:0/2123441701 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d80cb990 con 0x7fb6bc0040f0 2024-02-19T15:23:26.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.923+0000 7fb6dda00640 1 -- 172.21.15.47:0/2123441701 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d8075360 con 0x7fb6bc0040f0 2024-02-19T15:23:26.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.927+0000 7fb6cea00640 1 -- 172.21.15.47:0/2123441701 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb6c8016820 con 0x7fb6bc0040f0 2024-02-19T15:23:26.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.927+0000 7fb6cea00640 1 --2- 172.21.15.47:0/2123441701 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb6b4066e70 0x7fb6b4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:26.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.927+0000 7fb6cea00640 1 -- 172.21.15.47:0/2123441701 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6c80883d0 con 0x7fb6bc0040f0 2024-02-19T15:23:26.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.927+0000 7fb6cfe00640 1 --2- 172.21.15.47:0/2123441701 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb6b4066e70 0x7fb6b4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:26.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.927+0000 7fb6cfe00640 1 --2- 172.21.15.47:0/2123441701 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb6b4066e70 0x7fb6b4069330 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0007f20 tx=0x7fb6c00a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:26.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:26.931+0000 7fb6cea00640 1 -- 172.21.15.47:0/2123441701 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb6c8055130 con 0x7fb6bc0040f0 2024-02-19T15:23:27.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.127+0000 7fb6dda00640 1 -- 172.21.15.47:0/2123441701 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6d815ae90 con 0x7fb6bc0040f0 2024-02-19T15:23:27.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.131+0000 7fb6cea00640 1 -- 172.21.15.47:0/2123441701 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4938 (secure 0 0 0) 0x7fb6c80587f0 con 0x7fb6bc0040f0 2024-02-19T15:23:27.135 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:27.136 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1616905430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1616905430},{"type":"v1","addr":"172.21.15.116:6836","nonce":1616905430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4814,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2845273225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2845273225},{"type":"v1","addr":"172.21.15.116:6837","nonce":2845273225}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5064,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/20043732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":20043732},{"type":"v1","addr":"172.21.15.47:6834","nonce":20043732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:22:58.590074+0000","modified":"2024-02-19T15:23:01.656160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3700339993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3700339993},{"type":"v1","addr":"172.21.15.47:6835","nonce":3700339993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T15:23:27.136 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 16 2024-02-19T15:23:27.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.131+0000 7fb6dda00640 1 -- 172.21.15.47:0/2123441701 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb6b4066e70 msgr2=0x7fb6b4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:27.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.131+0000 7fb6dda00640 1 --2- 172.21.15.47:0/2123441701 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb6b4066e70 0x7fb6b4069330 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0007f20 tx=0x7fb6c00a7070 comp rx=0 tx=0).stop 2024-02-19T15:23:27.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.131+0000 7fb6dda00640 1 -- 172.21.15.47:0/2123441701 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6bc0040f0 msgr2=0x7fb6d80cb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:27.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.131+0000 7fb6dda00640 1 --2- 172.21.15.47:0/2123441701 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6bc0040f0 0x7fb6d80cb0d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb6c80128a0 tx=0x7fb6c8012d60 comp rx=0 tx=0).stop 2024-02-19T15:23:27.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.131+0000 7fb6dda00640 1 -- 172.21.15.47:0/2123441701 shutdown_connections 2024-02-19T15:23:27.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.131+0000 7fb6dda00640 1 --2- 172.21.15.47:0/2123441701 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb6b4066e70 0x7fb6b4069330 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.131+0000 7fb6dda00640 1 --2- 172.21.15.47:0/2123441701 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6d8157c90 0x7fb6d815c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.131+0000 7fb6dda00640 1 --2- 172.21.15.47:0/2123441701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6d81efc00 0x7fb6d81585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.131+0000 7fb6dda00640 1 --2- 172.21.15.47:0/2123441701 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6bc0040f0 0x7fb6d80cb0d0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.131+0000 7fb6dda00640 1 -- 172.21.15.47:0/2123441701 >> 172.21.15.47:0/2123441701 conn(0x7fb6d80d1bf0 msgr2=0x7fb6d806bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:27.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.131+0000 7fb6dda00640 1 -- 172.21.15.47:0/2123441701 shutdown_connections 2024-02-19T15:23:27.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.131+0000 7fb6dda00640 1 -- 172.21.15.47:0/2123441701 wait complete. 2024-02-19T15:23:27.156 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:23:27.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.299+0000 7ff29c200640 1 Processor -- start 2024-02-19T15:23:27.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.299+0000 7ff29c200640 1 -- start start 2024-02-19T15:23:27.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.299+0000 7ff29c200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff29415be90 0x7ff2940d0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:27.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.299+0000 7ff29c200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff2940d26b0 0x7ff2940ce460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:27.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.299+0000 7ff29c200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2940cdae0 con 0x7ff2940ce9a0 2024-02-19T15:23:27.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.299+0000 7ff29c200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2940cee10 con 0x7ff2940d26b0 2024-02-19T15:23:27.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.299+0000 7ff29c200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2940d0af0 con 0x7ff29415be90 2024-02-19T15:23:27.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.303+0000 7ff298e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff2940d26b0 0x7ff2940ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:27.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.303+0000 7ff29a200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff2940ce9a0 0x7ff2940cc730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33016/0 (socket says 172.21.15.47:33016) 2024-02-19T15:23:27.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.303+0000 7ff29a200640 1 -- 172.21.15.47:0/4044361364 learned_addr learned my addr 172.21.15.47:0/4044361364 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:27.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.303+0000 7ff299800640 1 --2- 172.21.15.47:0/4044361364 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff29415be90 0x7ff2940d0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:27.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.303+0000 7ff298e00640 1 -- 172.21.15.47:0/4044361364 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff29415be90 msgr2=0x7ff2940d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:27.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.303+0000 7ff298e00640 1 --2- 172.21.15.47:0/4044361364 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff29415be90 0x7ff2940d0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.303+0000 7ff298e00640 1 -- 172.21.15.47:0/4044361364 >> v1:172.21.15.47:6789/0 conn(0x7ff2940ce9a0 legacy=0x7ff2940cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:27.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.303+0000 7ff298e00640 1 -- 172.21.15.47:0/4044361364 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2940d0810 con 0x7ff2940d26b0 2024-02-19T15:23:27.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.303+0000 7ff299800640 1 --2- 172.21.15.47:0/4044361364 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff29415be90 0x7ff2940d0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:27.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.303+0000 7ff293e00640 1 -- 172.21.15.47:0/4044361364 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4032775046 0 0) 0x7ff2940cdae0 con 0x7ff2940ce9a0 2024-02-19T15:23:27.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.303+0000 7ff298e00640 1 --2- 172.21.15.47:0/4044361364 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2940d26b0 0x7ff2940ce460 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff2800097c0 tx=0x7ff28001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e84e111299512745 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:27.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.303+0000 7ff293e00640 1 -- 172.21.15.47:0/4044361364 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2800ab670 con 0x7ff2940d26b0 2024-02-19T15:23:27.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.303+0000 7ff293e00640 1 -- 172.21.15.47:0/4044361364 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2800ab7d0 con 0x7ff2940d26b0 2024-02-19T15:23:27.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.303+0000 7ff293e00640 1 -- 172.21.15.47:0/4044361364 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2800abb60 con 0x7ff2940d26b0 2024-02-19T15:23:27.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff29c200640 1 -- 172.21.15.47:0/4044361364 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2940d26b0 msgr2=0x7ff2940ce460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:27.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff29c200640 1 --2- 172.21.15.47:0/4044361364 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2940d26b0 0x7ff2940ce460 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff2800097c0 tx=0x7ff28001c470 comp rx=0 tx=0).stop 2024-02-19T15:23:27.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff29c200640 1 -- 172.21.15.47:0/4044361364 shutdown_connections 2024-02-19T15:23:27.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff29c200640 1 --2- 172.21.15.47:0/4044361364 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2940d26b0 0x7ff2940ce460 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff29c200640 1 --2- 172.21.15.47:0/4044361364 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff29415be90 0x7ff2940d0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff29c200640 1 -- 172.21.15.47:0/4044361364 >> 172.21.15.47:0/4044361364 conn(0x7ff2940d1b90 msgr2=0x7ff2940d1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:27.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff29c200640 1 -- 172.21.15.47:0/4044361364 shutdown_connections 2024-02-19T15:23:27.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff29c200640 1 -- 172.21.15.47:0/4044361364 wait complete. 2024-02-19T15:23:27.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff29c200640 1 Processor -- start 2024-02-19T15:23:27.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff29c200640 1 -- start start 2024-02-19T15:23:27.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff29c200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2940ce9a0 0x7ff294152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:27.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff29c200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff2940d26b0 0x7ff29414cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:27.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff29c200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff29415be90 0x7ff294155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:27.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff29c200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff294153100 con 0x7ff2940ce9a0 2024-02-19T15:23:27.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff29c200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff294151b20 con 0x7ff29415be90 2024-02-19T15:23:27.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff29c200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff294151e00 con 0x7ff2940d26b0 2024-02-19T15:23:27.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff299800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2940ce9a0 0x7ff294152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:27.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff299800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2940ce9a0 0x7ff294152bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60222/0 (socket says 172.21.15.47:60222) 2024-02-19T15:23:27.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff299800640 1 -- 172.21.15.47:0/2114696142 learned_addr learned my addr 172.21.15.47:0/2114696142 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:27.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff298e00640 1 --2- 172.21.15.47:0/2114696142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff2940d26b0 0x7ff29414cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:27.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff29a200640 1 --2- 172.21.15.47:0/2114696142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff29415be90 0x7ff294155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:27.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff299800640 1 -- 172.21.15.47:0/2114696142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff2940d26b0 msgr2=0x7ff29414cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:27.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff299800640 1 --2- 172.21.15.47:0/2114696142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff2940d26b0 0x7ff29414cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff299800640 1 -- 172.21.15.47:0/2114696142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff29415be90 msgr2=0x7ff294155400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:27.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff299800640 1 --2- 172.21.15.47:0/2114696142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff29415be90 0x7ff294155400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff299800640 1 -- 172.21.15.47:0/2114696142 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff280009420 con 0x7ff2940ce9a0 2024-02-19T15:23:27.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.307+0000 7ff29a200640 1 --2- 172.21.15.47:0/2114696142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff29415be90 0x7ff294155400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:27.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.311+0000 7ff299800640 1 --2- 172.21.15.47:0/2114696142 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2940ce9a0 0x7ff294152bc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff28800d800 tx=0x7ff28800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:27.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.311+0000 7ff291600640 1 -- 172.21.15.47:0/2114696142 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2880096c0 con 0x7ff2940ce9a0 2024-02-19T15:23:27.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.311+0000 7ff291600640 1 -- 172.21.15.47:0/2114696142 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff288009820 con 0x7ff2940ce9a0 2024-02-19T15:23:27.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.311+0000 7ff291600640 1 -- 172.21.15.47:0/2114696142 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff288009bd0 con 0x7ff2940ce9a0 2024-02-19T15:23:27.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.311+0000 7ff29c200640 1 -- 172.21.15.47:0/2114696142 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2941520e0 con 0x7ff2940ce9a0 2024-02-19T15:23:27.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.311+0000 7ff29c200640 1 -- 172.21.15.47:0/2114696142 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2941526a0 con 0x7ff2940ce9a0 2024-02-19T15:23:27.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.311+0000 7ff29c200640 1 -- 172.21.15.47:0/2114696142 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff294068500 con 0x7ff2940ce9a0 2024-02-19T15:23:27.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.311+0000 7ff291600640 1 -- 172.21.15.47:0/2114696142 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff288009d30 con 0x7ff2940ce9a0 2024-02-19T15:23:27.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.311+0000 7ff291600640 1 --2- 172.21.15.47:0/2114696142 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff274066e40 0x7ff274069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:27.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.315+0000 7ff291600640 1 -- 172.21.15.47:0/2114696142 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff288085120 con 0x7ff2940ce9a0 2024-02-19T15:23:27.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.315+0000 7ff298e00640 1 --2- 172.21.15.47:0/2114696142 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff274066e40 0x7ff274069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:27.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.315+0000 7ff298e00640 1 --2- 172.21.15.47:0/2114696142 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff274066e40 0x7ff274069300 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff2800af000 tx=0x7ff2800a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:27.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.315+0000 7ff291600640 1 -- 172.21.15.47:0/2114696142 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff288051f00 con 0x7ff2940ce9a0 2024-02-19T15:23:27.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.479+0000 7ff29c200640 1 -- 172.21.15.47:0/2114696142 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2940d1170 con 0x7ff2940ce9a0 2024-02-19T15:23:27.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.479+0000 7ff291600640 1 -- 172.21.15.47:0/2114696142 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16681 (secure 0 0 0) 0x7ff2880555c0 con 0x7ff2940ce9a0 2024-02-19T15:23:27.487 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:27.487 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":34,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:23:00.195724+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:56.436273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:57.586982+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:23:27.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.483+0000 7ff29c200640 1 -- 172.21.15.47:0/2114696142 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff274066e40 msgr2=0x7ff274069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:27.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.483+0000 7ff29c200640 1 --2- 172.21.15.47:0/2114696142 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff274066e40 0x7ff274069300 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff2800af000 tx=0x7ff2800a7000 comp rx=0 tx=0).stop 2024-02-19T15:23:27.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.483+0000 7ff29c200640 1 -- 172.21.15.47:0/2114696142 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2940ce9a0 msgr2=0x7ff294152bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:27.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.483+0000 7ff29c200640 1 --2- 172.21.15.47:0/2114696142 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2940ce9a0 0x7ff294152bc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff28800d800 tx=0x7ff28800dcc0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.483+0000 7ff29c200640 1 -- 172.21.15.47:0/2114696142 shutdown_connections 2024-02-19T15:23:27.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.483+0000 7ff29c200640 1 --2- 172.21.15.47:0/2114696142 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff274066e40 0x7ff274069300 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.483+0000 7ff29c200640 1 --2- 172.21.15.47:0/2114696142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff29415be90 0x7ff294155400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.483+0000 7ff29c200640 1 --2- 172.21.15.47:0/2114696142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff2940d26b0 0x7ff29414cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.483+0000 7ff29c200640 1 --2- 172.21.15.47:0/2114696142 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2940ce9a0 0x7ff294152bc0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.483+0000 7ff29c200640 1 -- 172.21.15.47:0/2114696142 >> 172.21.15.47:0/2114696142 conn(0x7ff2940d1b90 msgr2=0x7ff2940b3680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:27.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.483+0000 7ff29c200640 1 -- 172.21.15.47:0/2114696142 shutdown_connections 2024-02-19T15:23:27.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.483+0000 7ff29c200640 1 -- 172.21.15.47:0/2114696142 wait complete. 2024-02-19T15:23:27.509 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:22:56.436273+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:22:57.586982+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:23:27.509 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:23:27.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.647+0000 7f1236200640 1 Processor -- start 2024-02-19T15:23:27.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.647+0000 7f1236200640 1 -- start start 2024-02-19T15:23:27.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.651+0000 7f1236200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f12300b2d70 0x7f12300d0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:27.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.651+0000 7f1236200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f123006c730 con 0x7f123006bfc0 2024-02-19T15:23:27.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.651+0000 7f1236200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f123006c450 con 0x7f12300c4b00 2024-02-19T15:23:27.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.651+0000 7f1236200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12300c9310 con 0x7f12300b2d70 2024-02-19T15:23:27.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.651+0000 7f122fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f12300b2d70 0x7f12300d0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:27.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.651+0000 7f122fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f12300b2d70 0x7f12300d0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60228/0 (socket says 172.21.15.47:60228) 2024-02-19T15:23:27.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.651+0000 7f1226a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f123006bfc0 0x7f12300c8c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33032/0 (socket says 172.21.15.47:33032) 2024-02-19T15:23:27.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.651+0000 7f122fe00640 1 -- 172.21.15.47:0/1477511544 learned_addr learned my addr 172.21.15.47:0/1477511544 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:27.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.651+0000 7f122fe00640 1 -- 172.21.15.47:0/1477511544 >> v1:172.21.15.116:6789/0 conn(0x7f12300c4b00 legacy=0x7f12300c4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:27.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.651+0000 7f122fe00640 1 -- 172.21.15.47:0/1477511544 >> v1:172.21.15.47:6789/0 conn(0x7f123006bfc0 legacy=0x7f12300c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:27.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.651+0000 7f122fe00640 1 -- 172.21.15.47:0/1477511544 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12300d01b0 con 0x7f12300b2d70 2024-02-19T15:23:27.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.655+0000 7f122fe00640 1 --2- 172.21.15.47:0/1477511544 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12300b2d70 0x7f12300d0c90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1228009e50 tx=0x7f122801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdcfb441c74b0308 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:27.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.655+0000 7f122ea00640 1 -- 172.21.15.47:0/1477511544 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12280a5830 con 0x7f12300b2d70 2024-02-19T15:23:27.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.655+0000 7f122ea00640 1 -- 172.21.15.47:0/1477511544 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12280a5990 con 0x7f12300b2d70 2024-02-19T15:23:27.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.655+0000 7f122ea00640 1 -- 172.21.15.47:0/1477511544 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12280a5d20 con 0x7f12300b2d70 2024-02-19T15:23:27.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.655+0000 7f1236200640 1 -- 172.21.15.47:0/1477511544 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12300b2d70 msgr2=0x7f12300d0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:27.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.655+0000 7f1236200640 1 --2- 172.21.15.47:0/1477511544 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12300b2d70 0x7f12300d0c90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1228009e50 tx=0x7f122801c970 comp rx=0 tx=0).stop 2024-02-19T15:23:27.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.655+0000 7f1236200640 1 -- 172.21.15.47:0/1477511544 shutdown_connections 2024-02-19T15:23:27.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.655+0000 7f1236200640 1 --2- 172.21.15.47:0/1477511544 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12300b2d70 0x7f12300d0c90 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.655+0000 7f1236200640 1 -- 172.21.15.47:0/1477511544 >> 172.21.15.47:0/1477511544 conn(0x7f12300d1b30 msgr2=0x7f12300d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:27.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.655+0000 7f1236200640 1 -- 172.21.15.47:0/1477511544 shutdown_connections 2024-02-19T15:23:27.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.655+0000 7f1236200640 1 -- 172.21.15.47:0/1477511544 wait complete. 2024-02-19T15:23:27.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.655+0000 7f1236200640 1 Processor -- start 2024-02-19T15:23:27.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.655+0000 7f1236200640 1 -- start start 2024-02-19T15:23:27.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f1236200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12300b2d70 0x7f123015aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:27.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f1236200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12300c4b00 0x7f12300cdb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:27.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f1236200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f123015d6a0 0x7f12300c7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:27.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f1236200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12300cd6d0 con 0x7f12300b2d70 2024-02-19T15:23:27.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f1236200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12300ccac0 con 0x7f12300c4b00 2024-02-19T15:23:27.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f1236200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12300ccda0 con 0x7f123015d6a0 2024-02-19T15:23:27.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f122f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12300b2d70 0x7f123015aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:27.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f122f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12300b2d70 0x7f123015aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60240/0 (socket says 172.21.15.47:60240) 2024-02-19T15:23:27.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f122f400640 1 -- 172.21.15.47:0/3688022026 learned_addr learned my addr 172.21.15.47:0/3688022026 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:27.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f1226a00640 1 --2- 172.21.15.47:0/3688022026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12300c4b00 0x7f12300cdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:27.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f122fe00640 1 --2- 172.21.15.47:0/3688022026 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f123015d6a0 0x7f12300c7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:27.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f122f400640 1 -- 172.21.15.47:0/3688022026 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f123015d6a0 msgr2=0x7f12300c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:27.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f122f400640 1 --2- 172.21.15.47:0/3688022026 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f123015d6a0 0x7f12300c7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f122f400640 1 -- 172.21.15.47:0/3688022026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12300c4b00 msgr2=0x7f12300cdb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:27.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f122f400640 1 --2- 172.21.15.47:0/3688022026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12300c4b00 0x7f12300cdb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f122f400640 1 -- 172.21.15.47:0/3688022026 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1228009b30 con 0x7f12300b2d70 2024-02-19T15:23:27.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f122f400640 1 --2- 172.21.15.47:0/3688022026 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12300b2d70 0x7f123015aa70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f122000c3c0 tx=0x7f122000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:27.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f1227e00640 1 -- 172.21.15.47:0/3688022026 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12200052d0 con 0x7f12300b2d70 2024-02-19T15:23:27.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f1227e00640 1 -- 172.21.15.47:0/3688022026 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1220005430 con 0x7f12300b2d70 2024-02-19T15:23:27.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f1227e00640 1 -- 172.21.15.47:0/3688022026 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1220003840 con 0x7f12300b2d70 2024-02-19T15:23:27.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f1236200640 1 -- 172.21.15.47:0/3688022026 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12300cd080 con 0x7f12300b2d70 2024-02-19T15:23:27.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f1236200640 1 -- 172.21.15.47:0/3688022026 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12300ce0a0 con 0x7f12300b2d70 2024-02-19T15:23:27.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f1236200640 1 -- 172.21.15.47:0/3688022026 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1230075360 con 0x7f12300b2d70 2024-02-19T15:23:27.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.659+0000 7f1227e00640 1 -- 172.21.15.47:0/3688022026 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f12200039a0 con 0x7f12300b2d70 2024-02-19T15:23:27.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.663+0000 7f1227e00640 1 --2- 172.21.15.47:0/3688022026 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f120c066e70 0x7f120c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:27.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.663+0000 7f1227e00640 1 -- 172.21.15.47:0/3688022026 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1220083fd0 con 0x7f12300b2d70 2024-02-19T15:23:27.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.663+0000 7f1226a00640 1 --2- 172.21.15.47:0/3688022026 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f120c066e70 0x7f120c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:27.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.663+0000 7f1226a00640 1 --2- 172.21.15.47:0/3688022026 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f120c066e70 0x7f120c069330 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f123015db10 tx=0x7f1218007cb0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:27.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.667+0000 7f1227e00640 1 -- 172.21.15.47:0/3688022026 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1220050d20 con 0x7f12300b2d70 2024-02-19T15:23:27.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.827+0000 7f1236200640 1 -- 172.21.15.47:0/3688022026 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f12300cff90 con 0x7f12300b2d70 2024-02-19T15:23:27.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.827+0000 7f1227e00640 1 -- 172.21.15.47:0/3688022026 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7f12200543e0 con 0x7f12300b2d70 2024-02-19T15:23:27.834 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:23:27.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.831+0000 7f1236200640 1 -- 172.21.15.47:0/3688022026 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f120c066e70 msgr2=0x7f120c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:27.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.831+0000 7f1236200640 1 --2- 172.21.15.47:0/3688022026 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f120c066e70 0x7f120c069330 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f123015db10 tx=0x7f1218007cb0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.831+0000 7f1236200640 1 -- 172.21.15.47:0/3688022026 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12300b2d70 msgr2=0x7f123015aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:27.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.831+0000 7f1236200640 1 --2- 172.21.15.47:0/3688022026 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12300b2d70 0x7f123015aa70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f122000c3c0 tx=0x7f122000c880 comp rx=0 tx=0).stop 2024-02-19T15:23:27.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.831+0000 7f1236200640 1 -- 172.21.15.47:0/3688022026 shutdown_connections 2024-02-19T15:23:27.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.831+0000 7f1236200640 1 --2- 172.21.15.47:0/3688022026 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f120c066e70 0x7f120c069330 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.831+0000 7f1236200640 1 --2- 172.21.15.47:0/3688022026 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f123015d6a0 0x7f12300c7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.831+0000 7f1236200640 1 --2- 172.21.15.47:0/3688022026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12300c4b00 0x7f12300cdb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.831+0000 7f1236200640 1 --2- 172.21.15.47:0/3688022026 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12300b2d70 0x7f123015aa70 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:27.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.831+0000 7f1236200640 1 -- 172.21.15.47:0/3688022026 >> 172.21.15.47:0/3688022026 conn(0x7f12300d1b30 msgr2=0x7f123015e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:27.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.831+0000 7f1236200640 1 -- 172.21.15.47:0/3688022026 shutdown_connections 2024-02-19T15:23:27.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.831+0000 7f1236200640 1 -- 172.21.15.47:0/3688022026 wait complete. 2024-02-19T15:23:27.856 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:23:27.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.991+0000 7f301ee00640 1 Processor -- start 2024-02-19T15:23:27.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.991+0000 7f301ee00640 1 -- start start 2024-02-19T15:23:28.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.995+0000 7f301ee00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f30180d1820 0x7f301806f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:28.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.995+0000 7f301ee00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3018071a70 con 0x7f301815e160 2024-02-19T15:23:28.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.995+0000 7f301ee00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30180ced30 con 0x7f301806e1e0 2024-02-19T15:23:28.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.995+0000 7f301ee00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30180d1c90 con 0x7f30180d1820 2024-02-19T15:23:28.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.995+0000 7f3017e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f301815e160 0x7f301806dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33038/0 (socket says 172.21.15.47:33038) 2024-02-19T15:23:28.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.995+0000 7f3017e00640 1 -- 172.21.15.47:0/1853129766 learned_addr learned my addr 172.21.15.47:0/1853129766 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:28.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.995+0000 7f301ce00640 1 --2- 172.21.15.47:0/1853129766 >> v2:172.21.15.116:3300/0 conn(0x7f30180d1820 0x7f301806f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:28.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.995+0000 7f3017400640 1 -- 172.21.15.47:0/1853129766 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781893526 0 0) 0x7f3018071a70 con 0x7f301815e160 2024-02-19T15:23:28.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.995+0000 7f3017400640 1 -- 172.21.15.47:0/1853129766 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ffc003530 con 0x7f301815e160 2024-02-19T15:23:28.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.995+0000 7f3017400640 1 -- 172.21.15.47:0/1853129766 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070053929 0 0) 0x7f30180ced30 con 0x7f301806e1e0 2024-02-19T15:23:28.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.995+0000 7f3017400640 1 -- 172.21.15.47:0/1853129766 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3018071a70 con 0x7f301806e1e0 2024-02-19T15:23:28.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.995+0000 7f301ce00640 1 -- 172.21.15.47:0/1853129766 >> v1:172.21.15.116:6789/0 conn(0x7f301806e1e0 legacy=0x7f301815e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.995+0000 7f301ce00640 1 -- 172.21.15.47:0/1853129766 >> v1:172.21.15.47:6789/0 conn(0x7f301815e160 legacy=0x7f301806dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.995+0000 7f3017400640 1 -- 172.21.15.47:0/1853129766 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 688526472 0 0) 0x7f2ffc003530 con 0x7f301815e160 2024-02-19T15:23:28.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.995+0000 7f301ce00640 1 -- 172.21.15.47:0/1853129766 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3018162a00 con 0x7f30180d1820 2024-02-19T15:23:28.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.999+0000 7f301ce00640 1 --2- 172.21.15.47:0/1853129766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30180d1820 0x7f301806f220 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3010009d90 tx=0x7f301001c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb021849810c64f1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:28.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.999+0000 7f3017400640 1 -- 172.21.15.47:0/1853129766 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3008001680 con 0x7f30180d1820 2024-02-19T15:23:28.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.999+0000 7f3017400640 1 -- 172.21.15.47:0/1853129766 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30100a56a0 con 0x7f30180d1820 2024-02-19T15:23:28.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.999+0000 7f3017400640 1 -- 172.21.15.47:0/1853129766 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30100a5a30 con 0x7f30180d1820 2024-02-19T15:23:28.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.999+0000 7f301ee00640 1 -- 172.21.15.47:0/1853129766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30180d1820 msgr2=0x7f301806f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.999+0000 7f301ee00640 1 --2- 172.21.15.47:0/1853129766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30180d1820 0x7f301806f220 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3010009d90 tx=0x7f301001c860 comp rx=0 tx=0).stop 2024-02-19T15:23:28.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.999+0000 7f301ee00640 1 -- 172.21.15.47:0/1853129766 shutdown_connections 2024-02-19T15:23:28.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.999+0000 7f301ee00640 1 --2- 172.21.15.47:0/1853129766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30180d1820 0x7f301806f220 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.999+0000 7f301ee00640 1 -- 172.21.15.47:0/1853129766 >> 172.21.15.47:0/1853129766 conn(0x7f30180d0d30 msgr2=0x7f30180d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:28.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.999+0000 7f301ee00640 1 -- 172.21.15.47:0/1853129766 shutdown_connections 2024-02-19T15:23:28.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.999+0000 7f301ee00640 1 -- 172.21.15.47:0/1853129766 wait complete. 2024-02-19T15:23:28.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.999+0000 7f301ee00640 1 Processor -- start 2024-02-19T15:23:28.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:27.999+0000 7f301ee00640 1 -- start start 2024-02-19T15:23:28.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f301ee00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f301806e1e0 0x7f3018154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:28.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f301ee00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30180d1820 0x7f301815c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:28.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f301ee00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f301815e160 0x7f301815b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:28.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f301ee00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301815cda0 con 0x7f30180d1820 2024-02-19T15:23:28.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f301ee00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301815d300 con 0x7f301815e160 2024-02-19T15:23:28.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f301ee00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f301815d5e0 con 0x7f301806e1e0 2024-02-19T15:23:28.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f300fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30180d1820 0x7f301815c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:28.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f300fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30180d1820 0x7f301815c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60244/0 (socket says 172.21.15.47:60244) 2024-02-19T15:23:28.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f300fe00640 1 -- 172.21.15.47:0/1439688554 learned_addr learned my addr 172.21.15.47:0/1439688554 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:28.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f3017e00640 1 --2- 172.21.15.47:0/1439688554 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f301806e1e0 0x7f3018154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:28.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f301ce00640 1 --2- 172.21.15.47:0/1439688554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f301815e160 0x7f301815b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:28.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f300fe00640 1 -- 172.21.15.47:0/1439688554 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f301806e1e0 msgr2=0x7f3018154b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f300fe00640 1 --2- 172.21.15.47:0/1439688554 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f301806e1e0 0x7f3018154b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f300fe00640 1 -- 172.21.15.47:0/1439688554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f301815e160 msgr2=0x7f301815b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f300fe00640 1 --2- 172.21.15.47:0/1439688554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f301815e160 0x7f301815b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f300fe00640 1 -- 172.21.15.47:0/1439688554 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3010009a70 con 0x7f30180d1820 2024-02-19T15:23:28.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f3017e00640 1 --2- 172.21.15.47:0/1439688554 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f301806e1e0 0x7f3018154b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:28.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f300fe00640 1 --2- 172.21.15.47:0/1439688554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30180d1820 0x7f301815c860 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f300000c490 tx=0x7f300000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:28.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f3014c00640 1 -- 172.21.15.47:0/1439688554 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30000052d0 con 0x7f30180d1820 2024-02-19T15:23:28.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f3014c00640 1 -- 172.21.15.47:0/1439688554 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3000005430 con 0x7f30180d1820 2024-02-19T15:23:28.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f301ee00640 1 -- 172.21.15.47:0/1439688554 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f301815d8c0 con 0x7f30180d1820 2024-02-19T15:23:28.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f3014c00640 1 -- 172.21.15.47:0/1439688554 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3000012870 con 0x7f30180d1820 2024-02-19T15:23:28.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f301ee00640 1 -- 172.21.15.47:0/1439688554 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f301815bec0 con 0x7f30180d1820 2024-02-19T15:23:28.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f301ee00640 1 -- 172.21.15.47:0/1439688554 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3018075360 con 0x7f30180d1820 2024-02-19T15:23:28.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.003+0000 7f3014c00640 1 -- 172.21.15.47:0/1439688554 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f30000129d0 con 0x7f30180d1820 2024-02-19T15:23:28.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.007+0000 7f3014c00640 1 --2- 172.21.15.47:0/1439688554 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2ff4066e70 0x7f2ff4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:28.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.007+0000 7f3014c00640 1 -- 172.21.15.47:0/1439688554 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30000845e0 con 0x7f30180d1820 2024-02-19T15:23:28.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.007+0000 7f3017e00640 1 --2- 172.21.15.47:0/1439688554 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2ff4066e70 0x7f2ff4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:28.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.007+0000 7f3017e00640 1 --2- 172.21.15.47:0/1439688554 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2ff4066e70 0x7f2ff4069330 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f30080011f0 tx=0x7f3008009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:28.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.011+0000 7f3014c00640 1 -- 172.21.15.47:0/1439688554 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f30000513d0 con 0x7f30180d1820 2024-02-19T15:23:28.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.171+0000 7f301ee00640 1 -- 172.21.15.47:0/1439688554 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3018072cc0 con 0x7f30180d1820 2024-02-19T15:23:28.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.175+0000 7f3014c00640 1 -- 172.21.15.47:0/1439688554 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3000054a90 con 0x7f30180d1820 2024-02-19T15:23:28.179 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:23:28.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.175+0000 7f301ee00640 1 -- 172.21.15.47:0/1439688554 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2ff4066e70 msgr2=0x7f2ff4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.175+0000 7f301ee00640 1 --2- 172.21.15.47:0/1439688554 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2ff4066e70 0x7f2ff4069330 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f30080011f0 tx=0x7f3008009040 comp rx=0 tx=0).stop 2024-02-19T15:23:28.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.175+0000 7f301ee00640 1 -- 172.21.15.47:0/1439688554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30180d1820 msgr2=0x7f301815c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.175+0000 7f301ee00640 1 --2- 172.21.15.47:0/1439688554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30180d1820 0x7f301815c860 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f300000c490 tx=0x7f300000c950 comp rx=0 tx=0).stop 2024-02-19T15:23:28.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.175+0000 7f301ee00640 1 -- 172.21.15.47:0/1439688554 shutdown_connections 2024-02-19T15:23:28.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.175+0000 7f301ee00640 1 --2- 172.21.15.47:0/1439688554 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2ff4066e70 0x7f2ff4069330 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.175+0000 7f301ee00640 1 --2- 172.21.15.47:0/1439688554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f301815e160 0x7f301815b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.175+0000 7f301ee00640 1 --2- 172.21.15.47:0/1439688554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30180d1820 0x7f301815c860 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.175+0000 7f301ee00640 1 --2- 172.21.15.47:0/1439688554 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f301806e1e0 0x7f3018154b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.175+0000 7f301ee00640 1 -- 172.21.15.47:0/1439688554 >> 172.21.15.47:0/1439688554 conn(0x7f30180d0d30 msgr2=0x7f301815ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:28.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.175+0000 7f301ee00640 1 -- 172.21.15.47:0/1439688554 shutdown_connections 2024-02-19T15:23:28.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.175+0000 7f301ee00640 1 -- 172.21.15.47:0/1439688554 wait complete. 2024-02-19T15:23:28.200 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:23:28.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.339+0000 7ff9cbe00640 1 Processor -- start 2024-02-19T15:23:28.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.339+0000 7ff9cbe00640 1 -- start start 2024-02-19T15:23:28.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.339+0000 7ff9cbe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff9c415e280 0x7ff9c406c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:28.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.339+0000 7ff9cbe00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c40cc620 con 0x7ff9c40d2650 2024-02-19T15:23:28.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.339+0000 7ff9cbe00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c40d12e0 con 0x7ff9c40ce3b0 2024-02-19T15:23:28.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.339+0000 7ff9cbe00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c40d1940 con 0x7ff9c415e280 2024-02-19T15:23:28.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.339+0000 7ff9c8a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff9c40d2650 0x7ff9c406c1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33054/0 (socket says 172.21.15.47:33054) 2024-02-19T15:23:28.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.339+0000 7ff9c8a00640 1 -- 172.21.15.47:0/1340553112 learned_addr learned my addr 172.21.15.47:0/1340553112 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:28.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.339+0000 7ff9c9e00640 1 --2- 172.21.15.47:0/1340553112 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff9c415e280 0x7ff9c406c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:28.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.339+0000 7ff9c3e00640 1 -- 172.21.15.47:0/1340553112 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910979085 0 0) 0x7ff9c40cc620 con 0x7ff9c40d2650 2024-02-19T15:23:28.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.339+0000 7ff9c9e00640 1 -- 172.21.15.47:0/1340553112 >> v1:172.21.15.116:6789/0 conn(0x7ff9c40ce3b0 legacy=0x7ff9c40ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.339+0000 7ff9c9e00640 1 -- 172.21.15.47:0/1340553112 >> v1:172.21.15.47:6789/0 conn(0x7ff9c40d2650 legacy=0x7ff9c406c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.339+0000 7ff9c9e00640 1 -- 172.21.15.47:0/1340553112 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c40d1000 con 0x7ff9c415e280 2024-02-19T15:23:28.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.343+0000 7ff9c9e00640 1 --2- 172.21.15.47:0/1340553112 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff9c415e280 0x7ff9c406c8e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc009dc0 tx=0x7ff9bc01c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bedb9d7b8d2a555 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:28.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.343+0000 7ff9c3e00640 1 -- 172.21.15.47:0/1340553112 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc0a4c50 con 0x7ff9c415e280 2024-02-19T15:23:28.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.343+0000 7ff9c3e00640 1 -- 172.21.15.47:0/1340553112 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9bc0a4db0 con 0x7ff9c415e280 2024-02-19T15:23:28.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.343+0000 7ff9c3e00640 1 -- 172.21.15.47:0/1340553112 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc0ac660 con 0x7ff9c415e280 2024-02-19T15:23:28.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.343+0000 7ff9cbe00640 1 -- 172.21.15.47:0/1340553112 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff9c415e280 msgr2=0x7ff9c406c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.343+0000 7ff9cbe00640 1 --2- 172.21.15.47:0/1340553112 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff9c415e280 0x7ff9c406c8e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc009dc0 tx=0x7ff9bc01c490 comp rx=0 tx=0).stop 2024-02-19T15:23:28.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.343+0000 7ff9cbe00640 1 -- 172.21.15.47:0/1340553112 shutdown_connections 2024-02-19T15:23:28.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.343+0000 7ff9cbe00640 1 --2- 172.21.15.47:0/1340553112 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff9c415e280 0x7ff9c406c8e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.343+0000 7ff9cbe00640 1 -- 172.21.15.47:0/1340553112 >> 172.21.15.47:0/1340553112 conn(0x7ff9c40d1b30 msgr2=0x7ff9c40d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:28.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.343+0000 7ff9cbe00640 1 -- 172.21.15.47:0/1340553112 shutdown_connections 2024-02-19T15:23:28.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.343+0000 7ff9cbe00640 1 -- 172.21.15.47:0/1340553112 wait complete. 2024-02-19T15:23:28.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9cbe00640 1 Processor -- start 2024-02-19T15:23:28.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9cbe00640 1 -- start start 2024-02-19T15:23:28.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9cbe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff9c40ce3b0 0x7ff9c4156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:28.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9cbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9c40d2650 0x7ff9c40b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:28.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9cbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9c415e280 0x7ff9c40b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:28.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9cbe00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c40b65f0 con 0x7ff9c415e280 2024-02-19T15:23:28.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9cbe00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c40b68d0 con 0x7ff9c40d2650 2024-02-19T15:23:28.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9cbe00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c40b6bb0 con 0x7ff9c40ce3b0 2024-02-19T15:23:28.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9c9e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9c415e280 0x7ff9c40b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:28.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9c9e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9c415e280 0x7ff9c40b3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60252/0 (socket says 172.21.15.47:60252) 2024-02-19T15:23:28.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9c9e00640 1 -- 172.21.15.47:0/3818729801 learned_addr learned my addr 172.21.15.47:0/3818729801 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:28.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9c9400640 1 --2- 172.21.15.47:0/3818729801 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff9c40ce3b0 0x7ff9c4156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:28.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9c8a00640 1 --2- 172.21.15.47:0/3818729801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9c40d2650 0x7ff9c40b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:28.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9c9e00640 1 -- 172.21.15.47:0/3818729801 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff9c40ce3b0 msgr2=0x7ff9c4156250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9c9e00640 1 --2- 172.21.15.47:0/3818729801 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff9c40ce3b0 0x7ff9c4156250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9c9e00640 1 -- 172.21.15.47:0/3818729801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9c40d2650 msgr2=0x7ff9c40b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9c9e00640 1 --2- 172.21.15.47:0/3818729801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9c40d2650 0x7ff9c40b34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9c9e00640 1 -- 172.21.15.47:0/3818729801 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9bc009a70 con 0x7ff9c415e280 2024-02-19T15:23:28.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9c9e00640 1 --2- 172.21.15.47:0/3818729801 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9c415e280 0x7ff9c40b3a10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc0af040 tx=0x7ff9bc019550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:28.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9c1600640 1 -- 172.21.15.47:0/3818729801 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc0ace80 con 0x7ff9c415e280 2024-02-19T15:23:28.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9c1600640 1 -- 172.21.15.47:0/3818729801 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9bc009450 con 0x7ff9c415e280 2024-02-19T15:23:28.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9c1600640 1 -- 172.21.15.47:0/3818729801 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc0b6610 con 0x7ff9c415e280 2024-02-19T15:23:28.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9cbe00640 1 -- 172.21.15.47:0/3818729801 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c40b6e90 con 0x7ff9c415e280 2024-02-19T15:23:28.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.347+0000 7ff9cbe00640 1 -- 172.21.15.47:0/3818729801 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c4156790 con 0x7ff9c415e280 2024-02-19T15:23:28.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.351+0000 7ff9cbe00640 1 -- 172.21.15.47:0/3818729801 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c4075360 con 0x7ff9c415e280 2024-02-19T15:23:28.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.351+0000 7ff9c1600640 1 -- 172.21.15.47:0/3818729801 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff9bc0b3050 con 0x7ff9c415e280 2024-02-19T15:23:28.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.351+0000 7ff9c1600640 1 --2- 172.21.15.47:0/3818729801 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff9a4066e40 0x7ff9a4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:28.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.351+0000 7ff9c1600640 1 -- 172.21.15.47:0/3818729801 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9bc11ec50 con 0x7ff9c415e280 2024-02-19T15:23:28.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.351+0000 7ff9c9400640 1 --2- 172.21.15.47:0/3818729801 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff9a4066e40 0x7ff9a4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:28.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.351+0000 7ff9c9400640 1 --2- 172.21.15.47:0/3818729801 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff9a4066e40 0x7ff9a4069300 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8007fd0 tx=0x7ff9b8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:28.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.355+0000 7ff9c1600640 1 -- 172.21.15.47:0/3818729801 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff9bc124020 con 0x7ff9c415e280 2024-02-19T15:23:28.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.527+0000 7ff9cbe00640 1 -- 172.21.15.47:0/3818729801 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff9c40cdf80 con 0x7ff9c415e280 2024-02-19T15:23:28.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.527+0000 7ff9c1600640 1 -- 172.21.15.47:0/3818729801 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff9bc0b10a0 con 0x7ff9c415e280 2024-02-19T15:23:28.533 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:23:28.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.527+0000 7ff9cbe00640 1 -- 172.21.15.47:0/3818729801 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff9a4066e40 msgr2=0x7ff9a4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.527+0000 7ff9cbe00640 1 --2- 172.21.15.47:0/3818729801 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff9a4066e40 0x7ff9a4069300 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8007fd0 tx=0x7ff9b8009040 comp rx=0 tx=0).stop 2024-02-19T15:23:28.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.527+0000 7ff9cbe00640 1 -- 172.21.15.47:0/3818729801 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9c415e280 msgr2=0x7ff9c40b3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.527+0000 7ff9cbe00640 1 --2- 172.21.15.47:0/3818729801 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9c415e280 0x7ff9c40b3a10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc0af040 tx=0x7ff9bc019550 comp rx=0 tx=0).stop 2024-02-19T15:23:28.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.527+0000 7ff9cbe00640 1 -- 172.21.15.47:0/3818729801 shutdown_connections 2024-02-19T15:23:28.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.527+0000 7ff9cbe00640 1 --2- 172.21.15.47:0/3818729801 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff9a4066e40 0x7ff9a4069300 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.527+0000 7ff9cbe00640 1 --2- 172.21.15.47:0/3818729801 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9c415e280 0x7ff9c40b3a10 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.527+0000 7ff9cbe00640 1 --2- 172.21.15.47:0/3818729801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9c40d2650 0x7ff9c40b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.527+0000 7ff9cbe00640 1 --2- 172.21.15.47:0/3818729801 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff9c40ce3b0 0x7ff9c4156250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.527+0000 7ff9cbe00640 1 -- 172.21.15.47:0/3818729801 >> 172.21.15.47:0/3818729801 conn(0x7ff9c40d1b30 msgr2=0x7ff9c406fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:28.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.531+0000 7ff9cbe00640 1 -- 172.21.15.47:0/3818729801 shutdown_connections 2024-02-19T15:23:28.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.531+0000 7ff9cbe00640 1 -- 172.21.15.47:0/3818729801 wait complete. 2024-02-19T15:23:28.554 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:23:28.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.687+0000 7f258a400640 1 Processor -- start 2024-02-19T15:23:28.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.687+0000 7f258a400640 1 -- start start 2024-02-19T15:23:28.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.687+0000 7f258a400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25840adbb0 0x7f2584168da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:28.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.687+0000 7f258a400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2584161780 con 0x7f258406fdf0 2024-02-19T15:23:28.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.687+0000 7f258a400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25840b01a0 con 0x7f25841613a0 2024-02-19T15:23:28.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.687+0000 7f258a400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25840ae3c0 con 0x7f25840adbb0 2024-02-19T15:23:28.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.691+0000 7f257be00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f258406fdf0 0x7f258406e7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33066/0 (socket says 172.21.15.47:33066) 2024-02-19T15:23:28.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.691+0000 7f257be00640 1 -- 172.21.15.47:0/2841566181 learned_addr learned my addr 172.21.15.47:0/2841566181 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:28.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.691+0000 7f2583e00640 1 --2- 172.21.15.47:0/2841566181 >> v2:172.21.15.116:3300/0 conn(0x7f25840adbb0 0x7f2584168da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:28.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.691+0000 7f2582a00640 1 -- 172.21.15.47:0/2841566181 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803336072 0 0) 0x7f2584161780 con 0x7f258406fdf0 2024-02-19T15:23:28.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.691+0000 7f2582a00640 1 -- 172.21.15.47:0/2841566181 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2568003530 con 0x7f258406fdf0 2024-02-19T15:23:28.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.691+0000 7f2582a00640 1 -- 172.21.15.47:0/2841566181 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809684179 0 0) 0x7f25840b01a0 con 0x7f25841613a0 2024-02-19T15:23:28.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.691+0000 7f2582a00640 1 -- 172.21.15.47:0/2841566181 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2584161780 con 0x7f25841613a0 2024-02-19T15:23:28.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.691+0000 7f2583e00640 1 -- 172.21.15.47:0/2841566181 >> v1:172.21.15.116:6789/0 conn(0x7f25841613a0 legacy=0x7f258406e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.691+0000 7f2583e00640 1 -- 172.21.15.47:0/2841566181 >> v1:172.21.15.47:6789/0 conn(0x7f258406fdf0 legacy=0x7f258406e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.691+0000 7f2583e00640 1 -- 172.21.15.47:0/2841566181 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25840ae6a0 con 0x7f25840adbb0 2024-02-19T15:23:28.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.695+0000 7f2583e00640 1 --2- 172.21.15.47:0/2841566181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25840adbb0 0x7f2584168da0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f257c009d90 tx=0x7f257c01c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=393a6e45f152703c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:28.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.695+0000 7f2582a00640 1 -- 172.21.15.47:0/2841566181 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c0a58b0 con 0x7f25840adbb0 2024-02-19T15:23:28.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.695+0000 7f2582a00640 1 -- 172.21.15.47:0/2841566181 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257c0a5a10 con 0x7f25840adbb0 2024-02-19T15:23:28.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.695+0000 7f2582a00640 1 -- 172.21.15.47:0/2841566181 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c0a5da0 con 0x7f25840adbb0 2024-02-19T15:23:28.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.695+0000 7f258a400640 1 -- 172.21.15.47:0/2841566181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25840adbb0 msgr2=0x7f2584168da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.695+0000 7f258a400640 1 --2- 172.21.15.47:0/2841566181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25840adbb0 0x7f2584168da0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f257c009d90 tx=0x7f257c01c8e0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.695+0000 7f258a400640 1 -- 172.21.15.47:0/2841566181 shutdown_connections 2024-02-19T15:23:28.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.695+0000 7f258a400640 1 --2- 172.21.15.47:0/2841566181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25840adbb0 0x7f2584168da0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.695+0000 7f258a400640 1 -- 172.21.15.47:0/2841566181 >> 172.21.15.47:0/2841566181 conn(0x7f25840d0ad0 msgr2=0x7f25840d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:28.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.695+0000 7f258a400640 1 -- 172.21.15.47:0/2841566181 shutdown_connections 2024-02-19T15:23:28.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.695+0000 7f258a400640 1 -- 172.21.15.47:0/2841566181 wait complete. 2024-02-19T15:23:28.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.695+0000 7f258a400640 1 Processor -- start 2024-02-19T15:23:28.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.695+0000 7f258a400640 1 -- start start 2024-02-19T15:23:28.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.695+0000 7f258a400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f258406fdf0 0x7f2584152410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:28.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.695+0000 7f258a400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25840adbb0 0x7f25840b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:28.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.695+0000 7f258a400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25841613a0 0x7f2584154be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:28.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.695+0000 7f258a400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2584156f20 con 0x7f258406fdf0 2024-02-19T15:23:28.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.695+0000 7f258a400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2584156c40 con 0x7f25840adbb0 2024-02-19T15:23:28.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.695+0000 7f258a400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2584154100 con 0x7f25841613a0 2024-02-19T15:23:28.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.695+0000 7f2583400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f258406fdf0 0x7f2584152410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:28.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f2583400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f258406fdf0 0x7f2584152410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60260/0 (socket says 172.21.15.47:60260) 2024-02-19T15:23:28.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f2583400640 1 -- 172.21.15.47:0/3052339977 learned_addr learned my addr 172.21.15.47:0/3052339977 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:28.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f257be00640 1 --2- 172.21.15.47:0/3052339977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25840adbb0 0x7f25840b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:28.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f2583e00640 1 --2- 172.21.15.47:0/3052339977 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25841613a0 0x7f2584154be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:28.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f2583400640 1 -- 172.21.15.47:0/3052339977 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25841613a0 msgr2=0x7f2584154be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f2583400640 1 --2- 172.21.15.47:0/3052339977 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25841613a0 0x7f2584154be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f2583400640 1 -- 172.21.15.47:0/3052339977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25840adbb0 msgr2=0x7f25840b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f2583400640 1 --2- 172.21.15.47:0/3052339977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25840adbb0 0x7f25840b0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f2583400640 1 -- 172.21.15.47:0/3052339977 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c009a70 con 0x7f258406fdf0 2024-02-19T15:23:28.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f2583e00640 1 --2- 172.21.15.47:0/3052339977 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25841613a0 0x7f2584154be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:28.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f257be00640 1 --2- 172.21.15.47:0/3052339977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25840adbb0 0x7f25840b0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:28.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f2583400640 1 --2- 172.21.15.47:0/3052339977 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f258406fdf0 0x7f2584152410 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f257400c430 tx=0x7f257400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:28.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f257b400640 1 -- 172.21.15.47:0/3052339977 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25740052d0 con 0x7f258406fdf0 2024-02-19T15:23:28.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f257b400640 1 -- 172.21.15.47:0/3052339977 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2574005450 con 0x7f258406fdf0 2024-02-19T15:23:28.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f257b400640 1 -- 172.21.15.47:0/3052339977 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2574003840 con 0x7f258406fdf0 2024-02-19T15:23:28.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f258a400640 1 -- 172.21.15.47:0/3052339977 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25841543e0 con 0x7f258406fdf0 2024-02-19T15:23:28.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f258a400640 1 -- 172.21.15.47:0/3052339977 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2584152950 con 0x7f258406fdf0 2024-02-19T15:23:28.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f258a400640 1 -- 172.21.15.47:0/3052339977 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2584153f60 con 0x7f258406fdf0 2024-02-19T15:23:28.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f257b400640 1 -- 172.21.15.47:0/3052339977 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f25740039a0 con 0x7f258406fdf0 2024-02-19T15:23:28.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f257b400640 1 --2- 172.21.15.47:0/3052339977 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2560066e40 0x7f2560069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:28.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.699+0000 7f257b400640 1 -- 172.21.15.47:0/3052339977 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2574084130 con 0x7f258406fdf0 2024-02-19T15:23:28.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.703+0000 7f257be00640 1 --2- 172.21.15.47:0/3052339977 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2560066e40 0x7f2560069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:28.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.703+0000 7f257be00640 1 --2- 172.21.15.47:0/3052339977 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2560066e40 0x7f2560069300 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f25840b1490 tx=0x7f256c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:28.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.707+0000 7f257b400640 1 -- 172.21.15.47:0/3052339977 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2584153f60 con 0x7f258406fdf0 2024-02-19T15:23:28.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.899+0000 7f258a400640 1 -- 172.21.15.47:0/3052339977 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2584153f60 con 0x7f258406fdf0 2024-02-19T15:23:28.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.899+0000 7f257b400640 1 -- 172.21.15.47:0/3052339977 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f2584153f60 con 0x7f258406fdf0 2024-02-19T15:23:28.906 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:28.906 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1616905430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1616905430},{"type":"v1","addr":"172.21.15.116:6836","nonce":1616905430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4814,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2845273225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2845273225},{"type":"v1","addr":"172.21.15.116:6837","nonce":2845273225}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5064,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/20043732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":20043732},{"type":"v1","addr":"172.21.15.47:6834","nonce":20043732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:22:58.590074+0000","modified":"2024-02-19T15:23:01.656160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3700339993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3700339993},{"type":"v1","addr":"172.21.15.47:6835","nonce":3700339993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T15:23:28.906 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 16 2024-02-19T15:23:28.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.903+0000 7f258a400640 1 -- 172.21.15.47:0/3052339977 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2560066e40 msgr2=0x7f2560069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.903+0000 7f258a400640 1 --2- 172.21.15.47:0/3052339977 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2560066e40 0x7f2560069300 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f25840b1490 tx=0x7f256c009040 comp rx=0 tx=0).stop 2024-02-19T15:23:28.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.903+0000 7f258a400640 1 -- 172.21.15.47:0/3052339977 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f258406fdf0 msgr2=0x7f2584152410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:28.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.903+0000 7f258a400640 1 --2- 172.21.15.47:0/3052339977 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f258406fdf0 0x7f2584152410 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f257400c430 tx=0x7f257400c8f0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.903+0000 7f258a400640 1 -- 172.21.15.47:0/3052339977 shutdown_connections 2024-02-19T15:23:28.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.903+0000 7f258a400640 1 --2- 172.21.15.47:0/3052339977 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2560066e40 0x7f2560069300 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.903+0000 7f258a400640 1 --2- 172.21.15.47:0/3052339977 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25841613a0 0x7f2584154be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.903+0000 7f258a400640 1 --2- 172.21.15.47:0/3052339977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25840adbb0 0x7f25840b0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.903+0000 7f258a400640 1 --2- 172.21.15.47:0/3052339977 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f258406fdf0 0x7f2584152410 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:28.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.903+0000 7f258a400640 1 -- 172.21.15.47:0/3052339977 >> 172.21.15.47:0/3052339977 conn(0x7f25840d0ad0 msgr2=0x7f2584157700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:28.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.903+0000 7f258a400640 1 -- 172.21.15.47:0/3052339977 shutdown_connections 2024-02-19T15:23:28.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:28.903+0000 7f258a400640 1 -- 172.21.15.47:0/3052339977 wait complete. 2024-02-19T15:23:28.928 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:23:29.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.063+0000 7f54d4600640 1 Processor -- start 2024-02-19T15:23:29.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.063+0000 7f54d4600640 1 -- start start 2024-02-19T15:23:29.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.063+0000 7f54d4600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f54cc1613a0 0x7f54cc0adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:29.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.063+0000 7f54d4600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54cc06fdf0 0x7f54cc06e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:29.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.063+0000 7f54d4600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54cc161780 con 0x7f54cc06e5f0 2024-02-19T15:23:29.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.063+0000 7f54d4600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54cc0b01a0 con 0x7f54cc1613a0 2024-02-19T15:23:29.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.063+0000 7f54d4600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54cc0ae3c0 con 0x7f54cc06fdf0 2024-02-19T15:23:29.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.063+0000 7f54d1c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f54cc1613a0 0x7f54cc0adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:29.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.063+0000 7f54d1200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54cc06fdf0 0x7f54cc06e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:29.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.063+0000 7f54d2600640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f54cc06e5f0 0x7f54cc161a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:45216/0 (socket says 172.21.15.47:45216) 2024-02-19T15:23:29.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.063+0000 7f54d2600640 1 -- 172.21.15.47:0/728698446 learned_addr learned my addr 172.21.15.47:0/728698446 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:29.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.063+0000 7f54cbe00640 1 -- 172.21.15.47:0/728698446 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614078780 0 0) 0x7f54cc161780 con 0x7f54cc06e5f0 2024-02-19T15:23:29.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.063+0000 7f54cbe00640 1 -- 172.21.15.47:0/728698446 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54b4003530 con 0x7f54cc06e5f0 2024-02-19T15:23:29.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.063+0000 7f54d1c00640 1 -- 172.21.15.47:0/728698446 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54cc06fdf0 msgr2=0x7f54cc06e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:29.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.063+0000 7f54d1c00640 1 --2- 172.21.15.47:0/728698446 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54cc06fdf0 0x7f54cc06e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.063+0000 7f54d1c00640 1 -- 172.21.15.47:0/728698446 >> v1:172.21.15.116:6789/0 conn(0x7f54cc06e5f0 legacy=0x7f54cc161a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:29.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.063+0000 7f54d1c00640 1 -- 172.21.15.47:0/728698446 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54cc0ae6a0 con 0x7f54cc1613a0 2024-02-19T15:23:29.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.063+0000 7f54d1200640 1 --2- 172.21.15.47:0/728698446 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54cc06fdf0 0x7f54cc06e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:23:29.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.071+0000 7f54d1c00640 1 --2- 172.21.15.47:0/728698446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54cc1613a0 0x7f54cc0adbb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f54c00097c0 tx=0x7f54c001c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bd9eb7a107ee414 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:29.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.071+0000 7f54cbe00640 1 -- 172.21.15.47:0/728698446 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c00a4ea0 con 0x7f54cc1613a0 2024-02-19T15:23:29.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.071+0000 7f54cbe00640 1 -- 172.21.15.47:0/728698446 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c00a7050 con 0x7f54cc1613a0 2024-02-19T15:23:29.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.071+0000 7f54cbe00640 1 -- 172.21.15.47:0/728698446 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c00ab820 con 0x7f54cc1613a0 2024-02-19T15:23:29.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.071+0000 7f54d4600640 1 -- 172.21.15.47:0/728698446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54cc1613a0 msgr2=0x7f54cc0adbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:29.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.071+0000 7f54d4600640 1 --2- 172.21.15.47:0/728698446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54cc1613a0 0x7f54cc0adbb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f54c00097c0 tx=0x7f54c001c410 comp rx=0 tx=0).stop 2024-02-19T15:23:29.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.071+0000 7f54d4600640 1 -- 172.21.15.47:0/728698446 shutdown_connections 2024-02-19T15:23:29.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.071+0000 7f54d4600640 1 --2- 172.21.15.47:0/728698446 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54cc06fdf0 0x7f54cc06e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.071+0000 7f54d4600640 1 --2- 172.21.15.47:0/728698446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54cc1613a0 0x7f54cc0adbb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.071+0000 7f54d4600640 1 -- 172.21.15.47:0/728698446 >> 172.21.15.47:0/728698446 conn(0x7f54cc0d0ad0 msgr2=0x7f54cc0d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:29.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.071+0000 7f54d4600640 1 -- 172.21.15.47:0/728698446 shutdown_connections 2024-02-19T15:23:29.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.071+0000 7f54d4600640 1 -- 172.21.15.47:0/728698446 wait complete. 2024-02-19T15:23:29.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.071+0000 7f54d4600640 1 Processor -- start 2024-02-19T15:23:29.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.071+0000 7f54d4600640 1 -- start start 2024-02-19T15:23:29.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.071+0000 7f54d4600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54cc06e5f0 0x7f54cc1602d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:29.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.071+0000 7f54d4600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54cc06fdf0 0x7f54cc141660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:29.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.071+0000 7f54d4600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54cc1613a0 0x7f54cc15f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:29.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.071+0000 7f54d4600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54cc15cd80 con 0x7f54cc06e5f0 2024-02-19T15:23:29.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.071+0000 7f54d4600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54cc1545e0 con 0x7f54cc1613a0 2024-02-19T15:23:29.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.071+0000 7f54d4600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54cc0726c0 con 0x7f54cc06fdf0 2024-02-19T15:23:29.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.075+0000 7f54d1c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54cc06e5f0 0x7f54cc1602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:29.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.075+0000 7f54d1c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54cc06e5f0 0x7f54cc1602d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60272/0 (socket says 172.21.15.47:60272) 2024-02-19T15:23:29.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.075+0000 7f54d1c00640 1 -- 172.21.15.47:0/3209146974 learned_addr learned my addr 172.21.15.47:0/3209146974 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:29.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.075+0000 7f54d1200640 1 --2- 172.21.15.47:0/3209146974 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54cc06fdf0 0x7f54cc141660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:29.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.075+0000 7f54d2600640 1 --2- 172.21.15.47:0/3209146974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54cc1613a0 0x7f54cc15f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:29.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.075+0000 7f54d1c00640 1 -- 172.21.15.47:0/3209146974 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54cc06fdf0 msgr2=0x7f54cc141660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:29.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.075+0000 7f54d1c00640 1 --2- 172.21.15.47:0/3209146974 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54cc06fdf0 0x7f54cc141660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.075+0000 7f54d1c00640 1 -- 172.21.15.47:0/3209146974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54cc1613a0 msgr2=0x7f54cc15f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:29.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.075+0000 7f54d1c00640 1 --2- 172.21.15.47:0/3209146974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54cc1613a0 0x7f54cc15f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.075+0000 7f54d1c00640 1 -- 172.21.15.47:0/3209146974 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b8009420 con 0x7f54cc06e5f0 2024-02-19T15:23:29.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.075+0000 7f54d1200640 1 --2- 172.21.15.47:0/3209146974 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54cc06fdf0 0x7f54cc141660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:29.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.075+0000 7f54d1c00640 1 --2- 172.21.15.47:0/3209146974 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54cc06e5f0 0x7f54cc1602d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f54c001a040 tx=0x7f54c001ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:29.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.075+0000 7f54c9600640 1 -- 172.21.15.47:0/3209146974 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c00a4ea0 con 0x7f54cc06e5f0 2024-02-19T15:23:29.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.075+0000 7f54c9600640 1 -- 172.21.15.47:0/3209146974 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c00b7450 con 0x7f54cc06e5f0 2024-02-19T15:23:29.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.075+0000 7f54c9600640 1 -- 172.21.15.47:0/3209146974 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c00b77e0 con 0x7f54cc06e5f0 2024-02-19T15:23:29.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.075+0000 7f54d4600640 1 -- 172.21.15.47:0/3209146974 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c0009420 con 0x7f54cc06e5f0 2024-02-19T15:23:29.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.075+0000 7f54d4600640 1 -- 172.21.15.47:0/3209146974 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54cc154200 con 0x7f54cc06e5f0 2024-02-19T15:23:29.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.075+0000 7f54d4600640 1 -- 172.21.15.47:0/3209146974 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54cc154060 con 0x7f54cc06e5f0 2024-02-19T15:23:29.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.079+0000 7f54c9600640 1 -- 172.21.15.47:0/3209146974 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f54c00b7940 con 0x7f54cc06e5f0 2024-02-19T15:23:29.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.079+0000 7f54c9600640 1 --2- 172.21.15.47:0/3209146974 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f54ac066e40 0x7f54ac069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:29.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.079+0000 7f54c9600640 1 -- 172.21.15.47:0/3209146974 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54c011f110 con 0x7f54cc06e5f0 2024-02-19T15:23:29.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.079+0000 7f54d1200640 1 --2- 172.21.15.47:0/3209146974 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f54ac066e40 0x7f54ac069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:29.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.079+0000 7f54d1200640 1 --2- 172.21.15.47:0/3209146974 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f54ac066e40 0x7f54ac069300 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f54cc15c640 tx=0x7f54b8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:29.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.083+0000 7f54c9600640 1 -- 172.21.15.47:0/3209146974 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f54cc154060 con 0x7f54cc06e5f0 2024-02-19T15:23:29.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.379+0000 7f54d4600640 1 -- 172.21.15.47:0/3209146974 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54cc154060 con 0x7f54cc06e5f0 2024-02-19T15:23:29.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.379+0000 7f54c9600640 1 -- 172.21.15.47:0/3209146974 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16681 (secure 0 0 0) 0x7f54cc154060 con 0x7f54cc06e5f0 2024-02-19T15:23:29.385 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:29.385 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":34,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:23:00.195724+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:56.436273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:57.586982+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:23:29.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.383+0000 7f54d4600640 1 -- 172.21.15.47:0/3209146974 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f54ac066e40 msgr2=0x7f54ac069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:29.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.383+0000 7f54d4600640 1 --2- 172.21.15.47:0/3209146974 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f54ac066e40 0x7f54ac069300 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f54cc15c640 tx=0x7f54b8008040 comp rx=0 tx=0).stop 2024-02-19T15:23:29.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.383+0000 7f54d4600640 1 -- 172.21.15.47:0/3209146974 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54cc06e5f0 msgr2=0x7f54cc1602d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:29.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.383+0000 7f54d4600640 1 --2- 172.21.15.47:0/3209146974 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54cc06e5f0 0x7f54cc1602d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f54c001a040 tx=0x7f54c001ac50 comp rx=0 tx=0).stop 2024-02-19T15:23:29.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.383+0000 7f54d4600640 1 -- 172.21.15.47:0/3209146974 shutdown_connections 2024-02-19T15:23:29.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.383+0000 7f54d4600640 1 --2- 172.21.15.47:0/3209146974 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f54ac066e40 0x7f54ac069300 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.383+0000 7f54d4600640 1 --2- 172.21.15.47:0/3209146974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54cc1613a0 0x7f54cc15f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.383+0000 7f54d4600640 1 --2- 172.21.15.47:0/3209146974 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54cc06fdf0 0x7f54cc141660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.383+0000 7f54d4600640 1 --2- 172.21.15.47:0/3209146974 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54cc06e5f0 0x7f54cc1602d0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.383+0000 7f54d4600640 1 -- 172.21.15.47:0/3209146974 >> 172.21.15.47:0/3209146974 conn(0x7f54cc0d0ad0 msgr2=0x7f54cc0d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:29.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.383+0000 7f54d4600640 1 -- 172.21.15.47:0/3209146974 shutdown_connections 2024-02-19T15:23:29.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.383+0000 7f54d4600640 1 -- 172.21.15.47:0/3209146974 wait complete. 2024-02-19T15:23:29.415 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T15:23:29.415 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:23:29.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.551+0000 7f92b4e00640 1 Processor -- start 2024-02-19T15:23:29.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.551+0000 7f92b4e00640 1 -- start start 2024-02-19T15:23:29.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.551+0000 7f92b4e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f92b0145590 0x7f92b00727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:29.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.551+0000 7f92b4e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f92b00cea80 0x7f92b006c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:29.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.551+0000 7f92b4e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f92b015c390 0x7f92b015c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:29.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.551+0000 7f92b4e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92b015d280 con 0x7f92b0145590 2024-02-19T15:23:29.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.551+0000 7f92b4e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92b0070ff0 con 0x7f92b015c390 2024-02-19T15:23:29.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.551+0000 7f92b4e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b015cd60 con 0x7f92b00cea80 2024-02-19T15:23:29.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.551+0000 7f92ade00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f92b0145590 0x7f92b00727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:29.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.551+0000 7f92ade00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f92b0145590 0x7f92b00727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60282/0 (socket says 172.21.15.47:60282) 2024-02-19T15:23:29.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.551+0000 7f92ade00640 1 -- 172.21.15.47:0/2548455482 learned_addr learned my addr 172.21.15.47:0/2548455482 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:29.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.551+0000 7f92ae800640 1 --2- 172.21.15.47:0/2548455482 >> v2:172.21.15.116:3300/0 conn(0x7f92b015c390 0x7f92b015c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:29.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.551+0000 7f92a5400640 1 --2- 172.21.15.47:0/2548455482 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f92b00cea80 0x7f92b006c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:29.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.551+0000 7f92ade00640 1 -- 172.21.15.47:0/2548455482 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f92b00cea80 msgr2=0x7f92b006c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:29.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.551+0000 7f92ade00640 1 --2- 172.21.15.47:0/2548455482 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f92b00cea80 0x7f92b006c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.551+0000 7f92ade00640 1 -- 172.21.15.47:0/2548455482 >> v2:172.21.15.116:3300/0 conn(0x7f92b015c390 msgr2=0x7f92b015c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:29.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.551+0000 7f92ade00640 1 --2- 172.21.15.47:0/2548455482 >> v2:172.21.15.116:3300/0 conn(0x7f92b015c390 0x7f92b015c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.551+0000 7f92ade00640 1 -- 172.21.15.47:0/2548455482 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b00709e0 con 0x7f92b0145590 2024-02-19T15:23:29.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.555+0000 7f92ade00640 1 --2- 172.21.15.47:0/2548455482 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92b0145590 0x7f92b00727b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f92a0009870 tx=0x7f92a001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b9be444a0e1640c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:29.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.555+0000 7f92ad400640 1 -- 172.21.15.47:0/2548455482 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a00ab5f0 con 0x7f92b0145590 2024-02-19T15:23:29.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.555+0000 7f92ad400640 1 -- 172.21.15.47:0/2548455482 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a00ab750 con 0x7f92b0145590 2024-02-19T15:23:29.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.555+0000 7f92ad400640 1 -- 172.21.15.47:0/2548455482 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a00abae0 con 0x7f92b0145590 2024-02-19T15:23:29.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.555+0000 7f92b4e00640 1 -- 172.21.15.47:0/2548455482 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92b0145590 msgr2=0x7f92b00727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:29.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.555+0000 7f92b4e00640 1 --2- 172.21.15.47:0/2548455482 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92b0145590 0x7f92b00727b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f92a0009870 tx=0x7f92a001c520 comp rx=0 tx=0).stop 2024-02-19T15:23:29.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.555+0000 7f92b4e00640 1 -- 172.21.15.47:0/2548455482 shutdown_connections 2024-02-19T15:23:29.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.555+0000 7f92b4e00640 1 --2- 172.21.15.47:0/2548455482 >> v2:172.21.15.116:3300/0 conn(0x7f92b015c390 0x7f92b015c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.555+0000 7f92b4e00640 1 --2- 172.21.15.47:0/2548455482 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f92b00cea80 0x7f92b006c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.555+0000 7f92b4e00640 1 --2- 172.21.15.47:0/2548455482 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92b0145590 0x7f92b00727b0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.555+0000 7f92b4e00640 1 -- 172.21.15.47:0/2548455482 >> 172.21.15.47:0/2548455482 conn(0x7f92b00b16a0 msgr2=0x7f92b00b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:29.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.555+0000 7f92b4e00640 1 -- 172.21.15.47:0/2548455482 shutdown_connections 2024-02-19T15:23:29.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.555+0000 7f92b4e00640 1 -- 172.21.15.47:0/2548455482 wait complete. 2024-02-19T15:23:29.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92b4e00640 1 Processor -- start 2024-02-19T15:23:29.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92b4e00640 1 -- start start 2024-02-19T15:23:29.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92b4e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92b00cea80 0x7f92b015b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:29.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92b4e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f92b0145590 0x7f92b013c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:29.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92b4e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92b015c390 0x7f92b0140ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:29.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92b4e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b00ab740 con 0x7f92b00cea80 2024-02-19T15:23:29.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92b4e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b00aa870 con 0x7f92b015c390 2024-02-19T15:23:29.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92b4e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b00aaac0 con 0x7f92b0145590 2024-02-19T15:23:29.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92ade00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92b00cea80 0x7f92b015b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:29.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92ade00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92b00cea80 0x7f92b015b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60292/0 (socket says 172.21.15.47:60292) 2024-02-19T15:23:29.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92ade00640 1 -- 172.21.15.47:0/3192499764 learned_addr learned my addr 172.21.15.47:0/3192499764 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:29.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92a5400640 1 --2- 172.21.15.47:0/3192499764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f92b0145590 0x7f92b013c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:29.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92ae800640 1 --2- 172.21.15.47:0/3192499764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92b015c390 0x7f92b0140ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:29.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92ade00640 1 -- 172.21.15.47:0/3192499764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f92b0145590 msgr2=0x7f92b013c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:29.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92ade00640 1 --2- 172.21.15.47:0/3192499764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f92b0145590 0x7f92b013c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92ade00640 1 -- 172.21.15.47:0/3192499764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92b015c390 msgr2=0x7f92b0140ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:29.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92ade00640 1 --2- 172.21.15.47:0/3192499764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92b015c390 0x7f92b0140ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92ade00640 1 -- 172.21.15.47:0/3192499764 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a00094d0 con 0x7f92b00cea80 2024-02-19T15:23:29.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92ade00640 1 --2- 172.21.15.47:0/3192499764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92b00cea80 0x7f92b015b8d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f92a00ad040 tx=0x7f92a001ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:29.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92a6a00640 1 -- 172.21.15.47:0/3192499764 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a00b4800 con 0x7f92b00cea80 2024-02-19T15:23:29.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92a6a00640 1 -- 172.21.15.47:0/3192499764 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a00b4960 con 0x7f92b00cea80 2024-02-19T15:23:29.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92a6a00640 1 -- 172.21.15.47:0/3192499764 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a00b4d10 con 0x7f92b00cea80 2024-02-19T15:23:29.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92b4e00640 1 -- 172.21.15.47:0/3192499764 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b00aada0 con 0x7f92b00cea80 2024-02-19T15:23:29.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.559+0000 7f92b4e00640 1 -- 172.21.15.47:0/3192499764 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b00ab360 con 0x7f92b00cea80 2024-02-19T15:23:29.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.563+0000 7f929f400640 1 -- 172.21.15.47:0/3192499764 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92800052a0 con 0x7f92b00cea80 2024-02-19T15:23:29.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.563+0000 7f92a6a00640 1 -- 172.21.15.47:0/3192499764 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f92a00c7440 con 0x7f92b00cea80 2024-02-19T15:23:29.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.563+0000 7f92a6a00640 1 --2- 172.21.15.47:0/3192499764 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f928c066e40 0x7f928c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:29.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.563+0000 7f92a6a00640 1 -- 172.21.15.47:0/3192499764 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92a00a8050 con 0x7f92b00cea80 2024-02-19T15:23:29.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.563+0000 7f92a5400640 1 --2- 172.21.15.47:0/3192499764 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f928c066e40 0x7f928c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:29.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.563+0000 7f92a5400640 1 --2- 172.21.15.47:0/3192499764 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f928c066e40 0x7f928c069300 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f92b01f8360 tx=0x7f9298008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:29.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.567+0000 7f92a6a00640 1 -- 172.21.15.47:0/3192499764 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f92a00ec430 con 0x7f92b00cea80 2024-02-19T15:23:29.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.763+0000 7f929f400640 1 -- 172.21.15.47:0/3192499764 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9280005ad0 con 0x7f92b00cea80 2024-02-19T15:23:29.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.763+0000 7f92a6a00640 1 -- 172.21.15.47:0/3192499764 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f92a00efaf0 con 0x7f92b00cea80 2024-02-19T15:23:29.770 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:29.770 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1616905430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1616905430},{"type":"v1","addr":"172.21.15.116:6836","nonce":1616905430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4814,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2845273225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2845273225},{"type":"v1","addr":"172.21.15.116:6837","nonce":2845273225}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5064,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/20043732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":20043732},{"type":"v1","addr":"172.21.15.47:6834","nonce":20043732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:22:58.590074+0000","modified":"2024-02-19T15:23:01.656160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3700339993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3700339993},{"type":"v1","addr":"172.21.15.47:6835","nonce":3700339993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T15:23:29.770 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 16 2024-02-19T15:23:29.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.767+0000 7f929f400640 1 -- 172.21.15.47:0/3192499764 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f928c066e40 msgr2=0x7f928c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:29.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.767+0000 7f929f400640 1 --2- 172.21.15.47:0/3192499764 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f928c066e40 0x7f928c069300 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f92b01f8360 tx=0x7f9298008040 comp rx=0 tx=0).stop 2024-02-19T15:23:29.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.767+0000 7f929f400640 1 -- 172.21.15.47:0/3192499764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92b00cea80 msgr2=0x7f92b015b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:29.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.767+0000 7f929f400640 1 --2- 172.21.15.47:0/3192499764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92b00cea80 0x7f92b015b8d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f92a00ad040 tx=0x7f92a001ad60 comp rx=0 tx=0).stop 2024-02-19T15:23:29.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.767+0000 7f929f400640 1 -- 172.21.15.47:0/3192499764 shutdown_connections 2024-02-19T15:23:29.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.767+0000 7f929f400640 1 --2- 172.21.15.47:0/3192499764 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f928c066e40 0x7f928c069300 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.767+0000 7f929f400640 1 --2- 172.21.15.47:0/3192499764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92b015c390 0x7f92b0140ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.767+0000 7f929f400640 1 --2- 172.21.15.47:0/3192499764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f92b0145590 0x7f92b013c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.767+0000 7f929f400640 1 --2- 172.21.15.47:0/3192499764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92b00cea80 0x7f92b015b8d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.767+0000 7f929f400640 1 -- 172.21.15.47:0/3192499764 >> 172.21.15.47:0/3192499764 conn(0x7f92b00b16a0 msgr2=0x7f92b00b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:29.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.767+0000 7f929f400640 1 -- 172.21.15.47:0/3192499764 shutdown_connections 2024-02-19T15:23:29.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.767+0000 7f929f400640 1 -- 172.21.15.47:0/3192499764 wait complete. 2024-02-19T15:23:29.791 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:23:29.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.923+0000 7ff0fb200640 1 Processor -- start 2024-02-19T15:23:29.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.927+0000 7ff0fb200640 1 -- start start 2024-02-19T15:23:29.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.927+0000 7ff0fb200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0fc06b5c0 0x7ff0fc0cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:29.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.927+0000 7ff0fb200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0fc161a10 con 0x7ff0fc15dc90 2024-02-19T15:23:29.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.927+0000 7ff0fb200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0fc161cf0 con 0x7ff0fc15e070 2024-02-19T15:23:29.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.927+0000 7ff0fb200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc161fd0 con 0x7ff0fc06b5c0 2024-02-19T15:23:29.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.927+0000 7ff0f9e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff0fc15dc90 0x7ff0fc06aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33070/0 (socket says 172.21.15.47:33070) 2024-02-19T15:23:29.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.927+0000 7ff0f9e00640 1 -- 172.21.15.47:0/1694008364 learned_addr learned my addr 172.21.15.47:0/1694008364 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:29.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.927+0000 7ff0f3e00640 1 --2- 172.21.15.47:0/1694008364 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0fc06b5c0 0x7ff0fc0cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:29.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.927+0000 7ff0f9400640 1 -- 172.21.15.47:0/1694008364 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487200454 0 0) 0x7ff0fc161a10 con 0x7ff0fc15dc90 2024-02-19T15:23:29.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.927+0000 7ff0f9400640 1 -- 172.21.15.47:0/1694008364 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0e0003530 con 0x7ff0fc15dc90 2024-02-19T15:23:29.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.927+0000 7ff0f9400640 1 -- 172.21.15.47:0/1694008364 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255334848 0 0) 0x7ff0fc161cf0 con 0x7ff0fc15e070 2024-02-19T15:23:29.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.927+0000 7ff0f3e00640 1 -- 172.21.15.47:0/1694008364 >> v1:172.21.15.116:6789/0 conn(0x7ff0fc15e070 legacy=0x7ff0fc071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:29.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.927+0000 7ff0f3e00640 1 -- 172.21.15.47:0/1694008364 >> v1:172.21.15.47:6789/0 conn(0x7ff0fc15dc90 legacy=0x7ff0fc06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:29.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.927+0000 7ff0f3e00640 1 -- 172.21.15.47:0/1694008364 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0fc1622b0 con 0x7ff0fc06b5c0 2024-02-19T15:23:29.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.931+0000 7ff0f3e00640 1 --2- 172.21.15.47:0/1694008364 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0fc06b5c0 0x7ff0fc0cfc10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff0e40097f0 tx=0x7ff0e401eb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e709ae41bcfba28e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:29.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.931+0000 7ff0f9400640 1 -- 172.21.15.47:0/1694008364 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec001680 con 0x7ff0fc06b5c0 2024-02-19T15:23:29.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.931+0000 7ff0f9400640 1 -- 172.21.15.47:0/1694008364 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e40a85d0 con 0x7ff0fc06b5c0 2024-02-19T15:23:29.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.931+0000 7ff0f9400640 1 -- 172.21.15.47:0/1694008364 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e40a8940 con 0x7ff0fc06b5c0 2024-02-19T15:23:29.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.931+0000 7ff0fb200640 1 -- 172.21.15.47:0/1694008364 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0fc06b5c0 msgr2=0x7ff0fc0cfc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:29.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.931+0000 7ff0fb200640 1 --2- 172.21.15.47:0/1694008364 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0fc06b5c0 0x7ff0fc0cfc10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff0e40097f0 tx=0x7ff0e401eb10 comp rx=0 tx=0).stop 2024-02-19T15:23:29.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.931+0000 7ff0fb200640 1 -- 172.21.15.47:0/1694008364 shutdown_connections 2024-02-19T15:23:29.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.931+0000 7ff0fb200640 1 --2- 172.21.15.47:0/1694008364 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0fc06b5c0 0x7ff0fc0cfc10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.931+0000 7ff0fb200640 1 -- 172.21.15.47:0/1694008364 >> 172.21.15.47:0/1694008364 conn(0x7ff0fc0d08d0 msgr2=0x7ff0fc0d0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:29.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.931+0000 7ff0fb200640 1 -- 172.21.15.47:0/1694008364 shutdown_connections 2024-02-19T15:23:29.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.931+0000 7ff0fb200640 1 -- 172.21.15.47:0/1694008364 wait complete. 2024-02-19T15:23:29.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.931+0000 7ff0fb200640 1 Processor -- start 2024-02-19T15:23:29.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0fb200640 1 -- start start 2024-02-19T15:23:29.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0fb200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0fc06b5c0 0x7ff0fc154fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:29.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0fb200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0fc15dc90 0x7ff0fc15c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:29.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0fb200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0fc15e070 0x7ff0fc0b3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:29.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0fb200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc15b430 con 0x7ff0fc15dc90 2024-02-19T15:23:29.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0fb200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc15b710 con 0x7ff0fc15e070 2024-02-19T15:23:29.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0fb200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc15b9f0 con 0x7ff0fc06b5c0 2024-02-19T15:23:29.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0f3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0fc15dc90 0x7ff0fc15c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:29.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0f3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0fc15dc90 0x7ff0fc15c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60304/0 (socket says 172.21.15.47:60304) 2024-02-19T15:23:29.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0f3e00640 1 -- 172.21.15.47:0/2464600835 learned_addr learned my addr 172.21.15.47:0/2464600835 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:29.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0f9e00640 1 --2- 172.21.15.47:0/2464600835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0fc06b5c0 0x7ff0fc154fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:29.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0fa800640 1 --2- 172.21.15.47:0/2464600835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0fc15e070 0x7ff0fc0b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:29.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0f3e00640 1 -- 172.21.15.47:0/2464600835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0fc06b5c0 msgr2=0x7ff0fc154fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:29.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0f3e00640 1 --2- 172.21.15.47:0/2464600835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0fc06b5c0 0x7ff0fc154fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0f3e00640 1 -- 172.21.15.47:0/2464600835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0fc15e070 msgr2=0x7ff0fc0b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:29.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0f3e00640 1 --2- 172.21.15.47:0/2464600835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0fc15e070 0x7ff0fc0b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:29.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0f3e00640 1 -- 172.21.15.47:0/2464600835 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e4009420 con 0x7ff0fc15dc90 2024-02-19T15:23:29.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0f3e00640 1 --2- 172.21.15.47:0/2464600835 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0fc15dc90 0x7ff0fc15c4d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff0e401c740 tx=0x7ff0e401b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:29.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0f2000640 1 -- 172.21.15.47:0/2464600835 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e40c0070 con 0x7ff0fc15dc90 2024-02-19T15:23:29.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0f2000640 1 -- 172.21.15.47:0/2464600835 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e40a9a60 con 0x7ff0fc15dc90 2024-02-19T15:23:29.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0f2000640 1 -- 172.21.15.47:0/2464600835 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e40a9dd0 con 0x7ff0fc15dc90 2024-02-19T15:23:29.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0fb200640 1 -- 172.21.15.47:0/2464600835 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0fc15bcd0 con 0x7ff0fc15dc90 2024-02-19T15:23:29.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0fb200640 1 -- 172.21.15.47:0/2464600835 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0fc155770 con 0x7ff0fc15dc90 2024-02-19T15:23:29.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.935+0000 7ff0fb200640 1 -- 172.21.15.47:0/2464600835 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0fc075360 con 0x7ff0fc15dc90 2024-02-19T15:23:29.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.939+0000 7ff0f2000640 1 -- 172.21.15.47:0/2464600835 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff0e40b7040 con 0x7ff0fc15dc90 2024-02-19T15:23:29.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.939+0000 7ff0f2000640 1 --2- 172.21.15.47:0/2464600835 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff0d8066e90 0x7ff0d8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:29.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.939+0000 7ff0f2000640 1 -- 172.21.15.47:0/2464600835 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0e4121920 con 0x7ff0fc15dc90 2024-02-19T15:23:29.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.939+0000 7ff0f9e00640 1 --2- 172.21.15.47:0/2464600835 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff0d8066e90 0x7ff0d8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:29.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.939+0000 7ff0f9e00640 1 --2- 172.21.15.47:0/2464600835 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff0d8066e90 0x7ff0d8069350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec000f70 tx=0x7ff0ec001940 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:29.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:29.943+0000 7ff0f2000640 1 -- 172.21.15.47:0/2464600835 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff0e40ee780 con 0x7ff0fc15dc90 2024-02-19T15:23:30.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.143+0000 7ff0fb200640 1 -- 172.21.15.47:0/2464600835 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0fc15bb30 con 0x7ff0fc15dc90 2024-02-19T15:23:30.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.143+0000 7ff0f2000640 1 -- 172.21.15.47:0/2464600835 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4938 (secure 0 0 0) 0x7ff0fc15bb30 con 0x7ff0fc15dc90 2024-02-19T15:23:30.150 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:30.150 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1616905430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1616905430},{"type":"v1","addr":"172.21.15.116:6836","nonce":1616905430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4814,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2845273225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2845273225},{"type":"v1","addr":"172.21.15.116:6837","nonce":2845273225}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5064,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/20043732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":20043732},{"type":"v1","addr":"172.21.15.47:6834","nonce":20043732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:22:58.590074+0000","modified":"2024-02-19T15:23:01.656160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3700339993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3700339993},{"type":"v1","addr":"172.21.15.47:6835","nonce":3700339993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T15:23:30.150 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 16 2024-02-19T15:23:30.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.147+0000 7ff0fb200640 1 -- 172.21.15.47:0/2464600835 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff0d8066e90 msgr2=0x7ff0d8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:30.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.147+0000 7ff0fb200640 1 --2- 172.21.15.47:0/2464600835 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff0d8066e90 0x7ff0d8069350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec000f70 tx=0x7ff0ec001940 comp rx=0 tx=0).stop 2024-02-19T15:23:30.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.147+0000 7ff0fb200640 1 -- 172.21.15.47:0/2464600835 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0fc15dc90 msgr2=0x7ff0fc15c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:30.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.147+0000 7ff0fb200640 1 --2- 172.21.15.47:0/2464600835 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0fc15dc90 0x7ff0fc15c4d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff0e401c740 tx=0x7ff0e401b8b0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.147+0000 7ff0fb200640 1 -- 172.21.15.47:0/2464600835 shutdown_connections 2024-02-19T15:23:30.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.147+0000 7ff0fb200640 1 --2- 172.21.15.47:0/2464600835 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff0d8066e90 0x7ff0d8069350 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.147+0000 7ff0fb200640 1 --2- 172.21.15.47:0/2464600835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0fc15e070 0x7ff0fc0b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.147+0000 7ff0fb200640 1 --2- 172.21.15.47:0/2464600835 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0fc15dc90 0x7ff0fc15c4d0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.147+0000 7ff0fb200640 1 --2- 172.21.15.47:0/2464600835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0fc06b5c0 0x7ff0fc154fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.147+0000 7ff0fb200640 1 -- 172.21.15.47:0/2464600835 >> 172.21.15.47:0/2464600835 conn(0x7ff0fc0d08d0 msgr2=0x7ff0fc156290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:30.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.147+0000 7ff0fb200640 1 -- 172.21.15.47:0/2464600835 shutdown_connections 2024-02-19T15:23:30.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.147+0000 7ff0fb200640 1 -- 172.21.15.47:0/2464600835 wait complete. 2024-02-19T15:23:30.171 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:23:30.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.307+0000 7fda65600640 1 Processor -- start 2024-02-19T15:23:30.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.307+0000 7fda65600640 1 -- start start 2024-02-19T15:23:30.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.311+0000 7fda65600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fda60159ae0 0x7fda600acf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:30.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.311+0000 7fda65600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fda6015d060 0x7fda6006d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:30.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.311+0000 7fda65600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda600b1c20 0x7fda600b20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:30.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.311+0000 7fda65600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda6006c700 con 0x7fda60159ae0 2024-02-19T15:23:30.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.311+0000 7fda65600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda600b2610 con 0x7fda6015d060 2024-02-19T15:23:30.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.311+0000 7fda65600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda60066cd0 con 0x7fda600b1c20 2024-02-19T15:23:30.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.311+0000 7fda5ea00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fda60159ae0 0x7fda600acf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:30.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.311+0000 7fda5ea00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fda60159ae0 0x7fda600acf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60308/0 (socket says 172.21.15.47:60308) 2024-02-19T15:23:30.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.311+0000 7fda5ea00640 1 -- 172.21.15.47:0/820345960 learned_addr learned my addr 172.21.15.47:0/820345960 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:30.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.311+0000 7fda5e000640 1 --2- 172.21.15.47:0/820345960 >> v2:172.21.15.116:3300/0 conn(0x7fda6015d060 0x7fda6006d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:30.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.311+0000 7fda5f400640 1 --2- 172.21.15.47:0/820345960 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda600b1c20 0x7fda600b20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:30.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.311+0000 7fda5ea00640 1 -- 172.21.15.47:0/820345960 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda600b1c20 msgr2=0x7fda600b20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:30.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.311+0000 7fda5ea00640 1 --2- 172.21.15.47:0/820345960 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda600b1c20 0x7fda600b20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.311+0000 7fda5ea00640 1 -- 172.21.15.47:0/820345960 >> v2:172.21.15.116:3300/0 conn(0x7fda6015d060 msgr2=0x7fda6006d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:30.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.311+0000 7fda5ea00640 1 --2- 172.21.15.47:0/820345960 >> v2:172.21.15.116:3300/0 conn(0x7fda6015d060 0x7fda6006d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.311+0000 7fda5ea00640 1 -- 172.21.15.47:0/820345960 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6006c0d0 con 0x7fda60159ae0 2024-02-19T15:23:30.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.311+0000 7fda5f400640 1 --2- 172.21.15.47:0/820345960 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda600b1c20 0x7fda600b20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:30.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.311+0000 7fda5ea00640 1 --2- 172.21.15.47:0/820345960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda60159ae0 0x7fda600acf30 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fda600ca7a0 tx=0x7fda5001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc90762eae3fa952 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:30.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.311+0000 7fda5d600640 1 -- 172.21.15.47:0/820345960 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda500ad640 con 0x7fda60159ae0 2024-02-19T15:23:30.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda5d600640 1 -- 172.21.15.47:0/820345960 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda500ad7a0 con 0x7fda60159ae0 2024-02-19T15:23:30.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda5d600640 1 -- 172.21.15.47:0/820345960 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda500adb30 con 0x7fda60159ae0 2024-02-19T15:23:30.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda65600640 1 -- 172.21.15.47:0/820345960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda60159ae0 msgr2=0x7fda600acf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:30.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda65600640 1 --2- 172.21.15.47:0/820345960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda60159ae0 0x7fda600acf30 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fda600ca7a0 tx=0x7fda5001c3d0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda65600640 1 -- 172.21.15.47:0/820345960 shutdown_connections 2024-02-19T15:23:30.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda65600640 1 --2- 172.21.15.47:0/820345960 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda600b1c20 0x7fda600b20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda65600640 1 --2- 172.21.15.47:0/820345960 >> v2:172.21.15.116:3300/0 conn(0x7fda6015d060 0x7fda6006d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda65600640 1 --2- 172.21.15.47:0/820345960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda60159ae0 0x7fda600acf30 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda65600640 1 -- 172.21.15.47:0/820345960 >> 172.21.15.47:0/820345960 conn(0x7fda600ccc20 msgr2=0x7fda600cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:30.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda65600640 1 -- 172.21.15.47:0/820345960 shutdown_connections 2024-02-19T15:23:30.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda65600640 1 -- 172.21.15.47:0/820345960 wait complete. 2024-02-19T15:23:30.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda65600640 1 Processor -- start 2024-02-19T15:23:30.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda65600640 1 -- start start 2024-02-19T15:23:30.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda65600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda600b1c20 0x7fda600720f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:30.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda65600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda60159ae0 0x7fda600729c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:30.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda65600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda6015d060 0x7fda601586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:30.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda65600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda600c8190 con 0x7fda600b1c20 2024-02-19T15:23:30.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda65600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda60072630 con 0x7fda6015d060 2024-02-19T15:23:30.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda65600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda60071330 con 0x7fda60159ae0 2024-02-19T15:23:30.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda5ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda600b1c20 0x7fda600720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:30.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda5ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda600b1c20 0x7fda600720f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60320/0 (socket says 172.21.15.47:60320) 2024-02-19T15:23:30.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda5ea00640 1 -- 172.21.15.47:0/1849285397 learned_addr learned my addr 172.21.15.47:0/1849285397 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:30.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda5e000640 1 --2- 172.21.15.47:0/1849285397 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda60159ae0 0x7fda600729c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:30.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.315+0000 7fda5f400640 1 --2- 172.21.15.47:0/1849285397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda6015d060 0x7fda601586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:30.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.319+0000 7fda5ea00640 1 -- 172.21.15.47:0/1849285397 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda60159ae0 msgr2=0x7fda600729c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:30.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.319+0000 7fda5ea00640 1 --2- 172.21.15.47:0/1849285397 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda60159ae0 0x7fda600729c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.319+0000 7fda5ea00640 1 -- 172.21.15.47:0/1849285397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda6015d060 msgr2=0x7fda601586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:30.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.319+0000 7fda5ea00640 1 --2- 172.21.15.47:0/1849285397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda6015d060 0x7fda601586b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.319+0000 7fda5ea00640 1 -- 172.21.15.47:0/1849285397 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda500094d0 con 0x7fda600b1c20 2024-02-19T15:23:30.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.319+0000 7fda5f400640 1 --2- 172.21.15.47:0/1849285397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda6015d060 0x7fda601586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:30.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.319+0000 7fda5ea00640 1 --2- 172.21.15.47:0/1849285397 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda600b1c20 0x7fda600720f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fda500b0040 tx=0x7fda50019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:30.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.319+0000 7fda56a00640 1 -- 172.21.15.47:0/1849285397 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda500b8680 con 0x7fda600b1c20 2024-02-19T15:23:30.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.319+0000 7fda56a00640 1 -- 172.21.15.47:0/1849285397 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda500a6040 con 0x7fda600b1c20 2024-02-19T15:23:30.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.319+0000 7fda56a00640 1 -- 172.21.15.47:0/1849285397 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda500b8a20 con 0x7fda600b1c20 2024-02-19T15:23:30.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.319+0000 7fda65600640 1 -- 172.21.15.47:0/1849285397 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda60071610 con 0x7fda600b1c20 2024-02-19T15:23:30.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.319+0000 7fda65600640 1 -- 172.21.15.47:0/1849285397 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda60071bd0 con 0x7fda600b1c20 2024-02-19T15:23:30.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.319+0000 7fda65600640 1 -- 172.21.15.47:0/1849285397 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda600b5020 con 0x7fda600b1c20 2024-02-19T15:23:30.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.319+0000 7fda56a00640 1 -- 172.21.15.47:0/1849285397 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fda500c2460 con 0x7fda600b1c20 2024-02-19T15:23:30.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.323+0000 7fda56a00640 1 --2- 172.21.15.47:0/1849285397 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fda40066e40 0x7fda40069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:30.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.323+0000 7fda56a00640 1 -- 172.21.15.47:0/1849285397 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda5011f8a0 con 0x7fda600b1c20 2024-02-19T15:23:30.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.323+0000 7fda5e000640 1 --2- 172.21.15.47:0/1849285397 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fda40066e40 0x7fda40069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:30.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.323+0000 7fda5e000640 1 --2- 172.21.15.47:0/1849285397 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fda40066e40 0x7fda40069300 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fda6006d530 tx=0x7fda4c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:30.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.327+0000 7fda56a00640 1 -- 172.21.15.47:0/1849285397 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fda500ec5f0 con 0x7fda600b1c20 2024-02-19T15:23:30.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.491+0000 7fda65600640 1 -- 172.21.15.47:0/1849285397 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda60075360 con 0x7fda600b1c20 2024-02-19T15:23:30.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.491+0000 7fda56a00640 1 -- 172.21.15.47:0/1849285397 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16681 (secure 0 0 0) 0x7fda500b2090 con 0x7fda600b1c20 2024-02-19T15:23:30.497 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:30.497 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":34,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:23:00.195724+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:22:56.436273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-02-19T15:22:57.586982+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:23:30.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.495+0000 7fda65600640 1 -- 172.21.15.47:0/1849285397 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fda40066e40 msgr2=0x7fda40069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:30.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.495+0000 7fda65600640 1 --2- 172.21.15.47:0/1849285397 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fda40066e40 0x7fda40069300 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fda6006d530 tx=0x7fda4c009190 comp rx=0 tx=0).stop 2024-02-19T15:23:30.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.495+0000 7fda65600640 1 -- 172.21.15.47:0/1849285397 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda600b1c20 msgr2=0x7fda600720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:30.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.495+0000 7fda65600640 1 --2- 172.21.15.47:0/1849285397 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda600b1c20 0x7fda600720f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fda500b0040 tx=0x7fda50019170 comp rx=0 tx=0).stop 2024-02-19T15:23:30.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.495+0000 7fda65600640 1 -- 172.21.15.47:0/1849285397 shutdown_connections 2024-02-19T15:23:30.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.495+0000 7fda65600640 1 --2- 172.21.15.47:0/1849285397 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fda40066e40 0x7fda40069300 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.495+0000 7fda65600640 1 --2- 172.21.15.47:0/1849285397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda6015d060 0x7fda601586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.495+0000 7fda65600640 1 --2- 172.21.15.47:0/1849285397 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda60159ae0 0x7fda600729c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.495+0000 7fda65600640 1 --2- 172.21.15.47:0/1849285397 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda600b1c20 0x7fda600720f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.495+0000 7fda65600640 1 -- 172.21.15.47:0/1849285397 >> 172.21.15.47:0/1849285397 conn(0x7fda600ccc20 msgr2=0x7fda600b03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:30.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.495+0000 7fda65600640 1 -- 172.21.15.47:0/1849285397 shutdown_connections 2024-02-19T15:23:30.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.495+0000 7fda65600640 1 -- 172.21.15.47:0/1849285397 wait complete. 2024-02-19T15:23:30.518 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T15:23:30.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.651+0000 7fc204600640 1 Processor -- start 2024-02-19T15:23:30.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.651+0000 7fc204600640 1 -- start start 2024-02-19T15:23:30.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.655+0000 7fc204600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc1fc0cf670 0x7fc1fc06aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:30.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.655+0000 7fc204600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc1fc06b3f0 0x7fc1fc06dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:30.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.655+0000 7fc204600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1fc145a80 con 0x7fc1fc1456a0 2024-02-19T15:23:30.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.655+0000 7fc204600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1fc0d03c0 con 0x7fc1fc06b3f0 2024-02-19T15:23:30.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.655+0000 7fc204600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1fc06b860 con 0x7fc1fc0cf670 2024-02-19T15:23:30.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.655+0000 7fc202600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc1fc06b3f0 0x7fc1fc06dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:30.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.655+0000 7fc202600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc1fc06b3f0 0x7fc1fc06dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60328/0 (socket says 172.21.15.47:60328) 2024-02-19T15:23:30.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.655+0000 7fc202600640 1 -- 172.21.15.47:0/908276797 learned_addr learned my addr 172.21.15.47:0/908276797 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:30.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.655+0000 7fc201200640 1 --2- 172.21.15.47:0/908276797 >> v2:172.21.15.116:3300/0 conn(0x7fc1fc0cf670 0x7fc1fc06aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:30.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.655+0000 7fc202600640 1 -- 172.21.15.47:0/908276797 >> v2:172.21.15.116:3300/0 conn(0x7fc1fc0cf670 msgr2=0x7fc1fc06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:30.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.655+0000 7fc202600640 1 --2- 172.21.15.47:0/908276797 >> v2:172.21.15.116:3300/0 conn(0x7fc1fc0cf670 0x7fc1fc06aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.655+0000 7fc202600640 1 -- 172.21.15.47:0/908276797 >> v1:172.21.15.116:6789/0 conn(0x7fc1fc1456a0 legacy=0x7fc1fc0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:30.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.655+0000 7fc202600640 1 -- 172.21.15.47:0/908276797 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1fc0d1880 con 0x7fc1fc06b3f0 2024-02-19T15:23:30.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.655+0000 7fc201200640 1 --2- 172.21.15.47:0/908276797 >> v2:172.21.15.116:3300/0 conn(0x7fc1fc0cf670 0x7fc1fc06aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:23:30.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc202600640 1 --2- 172.21.15.47:0/908276797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1fc06b3f0 0x7fc1fc06dd60 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc0748f0 tx=0x7fc1f401c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4863b19bf0dd7d2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:30.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc1fbe00640 1 -- 172.21.15.47:0/908276797 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f40a5760 con 0x7fc1fc06b3f0 2024-02-19T15:23:30.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc1fbe00640 1 -- 172.21.15.47:0/908276797 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f40a58c0 con 0x7fc1fc06b3f0 2024-02-19T15:23:30.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc1fbe00640 1 -- 172.21.15.47:0/908276797 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f40a5c50 con 0x7fc1fc06b3f0 2024-02-19T15:23:30.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc204600640 1 -- 172.21.15.47:0/908276797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1fc06b3f0 msgr2=0x7fc1fc06dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:30.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc204600640 1 --2- 172.21.15.47:0/908276797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1fc06b3f0 0x7fc1fc06dd60 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc0748f0 tx=0x7fc1f401c9b0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc204600640 1 -- 172.21.15.47:0/908276797 shutdown_connections 2024-02-19T15:23:30.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc204600640 1 --2- 172.21.15.47:0/908276797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1fc06b3f0 0x7fc1fc06dd60 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc204600640 1 --2- 172.21.15.47:0/908276797 >> v2:172.21.15.116:3300/0 conn(0x7fc1fc0cf670 0x7fc1fc06aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc204600640 1 -- 172.21.15.47:0/908276797 >> 172.21.15.47:0/908276797 conn(0x7fc1fc0d0ad0 msgr2=0x7fc1fc0d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:30.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc204600640 1 -- 172.21.15.47:0/908276797 shutdown_connections 2024-02-19T15:23:30.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc204600640 1 -- 172.21.15.47:0/908276797 wait complete. 2024-02-19T15:23:30.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc204600640 1 Processor -- start 2024-02-19T15:23:30.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc204600640 1 -- start start 2024-02-19T15:23:30.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc204600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1fc06b3f0 0x7fc1fc1383f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:30.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc204600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1fc0cf670 0x7fc1fc0b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:30.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc204600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1fc1456a0 0x7fc1fc13ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:30.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc204600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1fc0b5a30 con 0x7fc1fc0cf670 2024-02-19T15:23:30.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc204600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1fc0b5470 con 0x7fc1fc1456a0 2024-02-19T15:23:30.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc204600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1fc0b5750 con 0x7fc1fc06b3f0 2024-02-19T15:23:30.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc201200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1fc0cf670 0x7fc1fc0b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:30.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc201200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1fc0cf670 0x7fc1fc0b4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60344/0 (socket says 172.21.15.47:60344) 2024-02-19T15:23:30.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.659+0000 7fc201200640 1 -- 172.21.15.47:0/2804239674 learned_addr learned my addr 172.21.15.47:0/2804239674 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:30.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.663+0000 7fc201200640 1 -- 172.21.15.47:0/2804239674 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1fc06b3f0 msgr2=0x7fc1fc1383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:30.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.663+0000 7fc201c00640 1 --2- 172.21.15.47:0/2804239674 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1fc06b3f0 0x7fc1fc1383f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:30.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.663+0000 7fc202600640 1 --2- 172.21.15.47:0/2804239674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1fc1456a0 0x7fc1fc13ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:30.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.663+0000 7fc201200640 1 --2- 172.21.15.47:0/2804239674 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1fc06b3f0 0x7fc1fc1383f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.663+0000 7fc201200640 1 -- 172.21.15.47:0/2804239674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1fc1456a0 msgr2=0x7fc1fc13ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:30.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.663+0000 7fc201200640 1 --2- 172.21.15.47:0/2804239674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1fc1456a0 0x7fc1fc13ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.663+0000 7fc201200640 1 -- 172.21.15.47:0/2804239674 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f4009b30 con 0x7fc1fc0cf670 2024-02-19T15:23:30.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.663+0000 7fc202600640 1 --2- 172.21.15.47:0/2804239674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1fc1456a0 0x7fc1fc13ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:23:30.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.663+0000 7fc201c00640 1 --2- 172.21.15.47:0/2804239674 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1fc06b3f0 0x7fc1fc1383f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:23:30.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.663+0000 7fc201200640 1 --2- 172.21.15.47:0/2804239674 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1fc0cf670 0x7fc1fc0b4b00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc1e800c7f0 tx=0x7fc1e800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:30.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.663+0000 7fc1f9600640 1 -- 172.21.15.47:0/2804239674 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e800cea0 con 0x7fc1fc0cf670 2024-02-19T15:23:30.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.663+0000 7fc1f9600640 1 -- 172.21.15.47:0/2804239674 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e8007a30 con 0x7fc1fc0cf670 2024-02-19T15:23:30.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.663+0000 7fc1f9600640 1 -- 172.21.15.47:0/2804239674 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e8007dc0 con 0x7fc1fc0cf670 2024-02-19T15:23:30.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.663+0000 7fc204600640 1 -- 172.21.15.47:0/2804239674 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1fc13c240 con 0x7fc1fc0cf670 2024-02-19T15:23:30.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.663+0000 7fc204600640 1 -- 172.21.15.47:0/2804239674 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1fc138930 con 0x7fc1fc0cf670 2024-02-19T15:23:30.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.663+0000 7fc204600640 1 -- 172.21.15.47:0/2804239674 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1fc0b11d0 con 0x7fc1fc0cf670 2024-02-19T15:23:30.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.663+0000 7fc1f9600640 1 -- 172.21.15.47:0/2804239674 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc1e80113f0 con 0x7fc1fc0cf670 2024-02-19T15:23:30.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.663+0000 7fc1f9600640 1 --2- 172.21.15.47:0/2804239674 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc1dc066e40 0x7fc1dc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:30.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.663+0000 7fc1f9600640 1 -- 172.21.15.47:0/2804239674 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1e8084c60 con 0x7fc1fc0cf670 2024-02-19T15:23:30.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.667+0000 7fc201c00640 1 --2- 172.21.15.47:0/2804239674 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc1dc066e40 0x7fc1dc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:30.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.667+0000 7fc201c00640 1 --2- 172.21.15.47:0/2804239674 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc1dc066e40 0x7fc1dc069300 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc1f000a850 tx=0x7fc1f0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:30.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.675+0000 7fc1f9600640 1 -- 172.21.15.47:0/2804239674 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc1e80519b0 con 0x7fc1fc0cf670 2024-02-19T15:23:30.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.871+0000 7fc204600640 1 -- 172.21.15.47:0/2804239674 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc1fc075360 con 0x7fc1fc0cf670 2024-02-19T15:23:30.877 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-02-19T15:23:30.871+0000 7fc100600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T15:23:30.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.875+0000 7fc1f9600640 1 -- 172.21.15.47:0/2804239674 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc1e801f090 con 0x7fc1fc0cf670 2024-02-19T15:23:30.879 INFO:teuthology.orchestra.run.smithi047.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T15:23:30.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.875+0000 7fc204600640 1 -- 172.21.15.47:0/2804239674 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc1dc066e40 msgr2=0x7fc1dc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:30.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.875+0000 7fc204600640 1 --2- 172.21.15.47:0/2804239674 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc1dc066e40 0x7fc1dc069300 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc1f000a850 tx=0x7fc1f0008040 comp rx=0 tx=0).stop 2024-02-19T15:23:30.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.875+0000 7fc204600640 1 -- 172.21.15.47:0/2804239674 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1fc0cf670 msgr2=0x7fc1fc0b4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:30.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.875+0000 7fc204600640 1 --2- 172.21.15.47:0/2804239674 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1fc0cf670 0x7fc1fc0b4b00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc1e800c7f0 tx=0x7fc1e800ccb0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.875+0000 7fc204600640 1 -- 172.21.15.47:0/2804239674 shutdown_connections 2024-02-19T15:23:30.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.875+0000 7fc204600640 1 --2- 172.21.15.47:0/2804239674 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc1dc066e40 0x7fc1dc069300 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.875+0000 7fc204600640 1 --2- 172.21.15.47:0/2804239674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1fc1456a0 0x7fc1fc13ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.875+0000 7fc204600640 1 --2- 172.21.15.47:0/2804239674 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1fc0cf670 0x7fc1fc0b4b00 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.875+0000 7fc204600640 1 --2- 172.21.15.47:0/2804239674 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1fc06b3f0 0x7fc1fc1383f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:30.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.875+0000 7fc204600640 1 -- 172.21.15.47:0/2804239674 >> 172.21.15.47:0/2804239674 conn(0x7fc1fc0d0ad0 msgr2=0x7fc1fc143060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:30.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.879+0000 7fc204600640 1 -- 172.21.15.47:0/2804239674 shutdown_connections 2024-02-19T15:23:30.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:30.879+0000 7fc204600640 1 -- 172.21.15.47:0/2804239674 wait complete. 2024-02-19T15:23:30.905 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T15:23:30.950 INFO:tasks.ceph.mds.c.smithi047.stdout:starting mds.c at 2024-02-19T15:23:31.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.043+0000 7fb882e00640 1 Processor -- start 2024-02-19T15:23:31.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.043+0000 7fb882e00640 1 -- start start 2024-02-19T15:23:31.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.043+0000 7fb882e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb87c15e210 0x7fb87c0ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:31.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.043+0000 7fb882e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb87c0cc180 0x7fb87c0ce8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:31.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.043+0000 7fb882e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87c0b2fd0 con 0x7fb87c0d2650 2024-02-19T15:23:31.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.043+0000 7fb882e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb87c0cfe00 con 0x7fb87c0cc180 2024-02-19T15:23:31.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.043+0000 7fb882e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0cff90 con 0x7fb87c15e210 2024-02-19T15:23:31.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.043+0000 7fb873e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb87c0cc180 0x7fb87c0ce8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:31.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.043+0000 7fb87be00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb87c15e210 0x7fb87c0ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:31.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.043+0000 7fb880e00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb87c0d2650 0x7fb87c0d03e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:45264/0 (socket says 172.21.15.47:45264) 2024-02-19T15:23:31.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.043+0000 7fb880e00640 1 -- 172.21.15.47:0/3046932211 learned_addr learned my addr 172.21.15.47:0/3046932211 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:31.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.047+0000 7fb87be00640 1 -- 172.21.15.47:0/3046932211 >> v2:172.21.15.116:3300/0 conn(0x7fb87c0cc180 msgr2=0x7fb87c0ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:31.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.047+0000 7fb87be00640 1 --2- 172.21.15.47:0/3046932211 >> v2:172.21.15.116:3300/0 conn(0x7fb87c0cc180 0x7fb87c0ce8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.047+0000 7fb87be00640 1 -- 172.21.15.47:0/3046932211 >> v1:172.21.15.116:6789/0 conn(0x7fb87c0d2650 legacy=0x7fb87c0d03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.047+0000 7fb87be00640 1 -- 172.21.15.47:0/3046932211 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c0b2cf0 con 0x7fb87c15e210 2024-02-19T15:23:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.047+0000 7fb873e00640 1 --2- 172.21.15.47:0/3046932211 >> v2:172.21.15.116:3300/0 conn(0x7fb87c0cc180 0x7fb87c0ce8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:23:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.047+0000 7fb87b400640 1 -- 172.21.15.47:0/3046932211 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773833975 0 0) 0x7fb87c0b2fd0 con 0x7fb87c0d2650 2024-02-19T15:23:31.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.047+0000 7fb87be00640 1 --2- 172.21.15.47:0/3046932211 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb87c15e210 0x7fb87c0ce3b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb86c0097c0 tx=0x7fb86c01c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8787ac439a8b3f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:31.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.047+0000 7fb87b400640 1 -- 172.21.15.47:0/3046932211 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c0ab670 con 0x7fb87c15e210 2024-02-19T15:23:31.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.047+0000 7fb87b400640 1 -- 172.21.15.47:0/3046932211 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86c0ab7d0 con 0x7fb87c15e210 2024-02-19T15:23:31.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.047+0000 7fb87b400640 1 -- 172.21.15.47:0/3046932211 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c0abb60 con 0x7fb87c15e210 2024-02-19T15:23:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb882e00640 1 -- 172.21.15.47:0/3046932211 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb87c15e210 msgr2=0x7fb87c0ce3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb882e00640 1 --2- 172.21.15.47:0/3046932211 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb87c15e210 0x7fb87c0ce3b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb86c0097c0 tx=0x7fb86c01c470 comp rx=0 tx=0).stop 2024-02-19T15:23:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb882e00640 1 -- 172.21.15.47:0/3046932211 shutdown_connections 2024-02-19T15:23:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb882e00640 1 --2- 172.21.15.47:0/3046932211 >> v2:172.21.15.116:3300/0 conn(0x7fb87c0cc180 0x7fb87c0ce8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb882e00640 1 --2- 172.21.15.47:0/3046932211 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb87c15e210 0x7fb87c0ce3b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:31.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb882e00640 1 -- 172.21.15.47:0/3046932211 >> 172.21.15.47:0/3046932211 conn(0x7fb87c0d1b30 msgr2=0x7fb87c0d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:31.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb882e00640 1 -- 172.21.15.47:0/3046932211 shutdown_connections 2024-02-19T15:23:31.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb882e00640 1 -- 172.21.15.47:0/3046932211 wait complete. 2024-02-19T15:23:31.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb882e00640 1 Processor -- start 2024-02-19T15:23:31.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb882e00640 1 -- start start 2024-02-19T15:23:31.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb882e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb87c0cc180 0x7fb87c159e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:31.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb882e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb87c0d2650 0x7fb87c15a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:31.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb882e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb87c15e210 0x7fb87c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:31.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb882e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0b7460 con 0x7fb87c0cc180 2024-02-19T15:23:31.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb882e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0b4750 con 0x7fb87c0d2650 2024-02-19T15:23:31.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb882e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c15a8c0 con 0x7fb87c15e210 2024-02-19T15:23:31.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb87be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb87c0cc180 0x7fb87c159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:31.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb87be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb87c0cc180 0x7fb87c159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60354/0 (socket says 172.21.15.47:60354) 2024-02-19T15:23:31.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb87be00640 1 -- 172.21.15.47:0/1572524967 learned_addr learned my addr 172.21.15.47:0/1572524967 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:31.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb873e00640 1 --2- 172.21.15.47:0/1572524967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb87c0d2650 0x7fb87c15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:31.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb87be00640 1 -- 172.21.15.47:0/1572524967 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb87c15e210 msgr2=0x7fb87c142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:31.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb87be00640 1 --2- 172.21.15.47:0/1572524967 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb87c15e210 0x7fb87c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:31.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb87be00640 1 -- 172.21.15.47:0/1572524967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb87c0d2650 msgr2=0x7fb87c15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:31.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb87be00640 1 --2- 172.21.15.47:0/1572524967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb87c0d2650 0x7fb87c15a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:31.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb87be00640 1 -- 172.21.15.47:0/1572524967 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb864009420 con 0x7fb87c0cc180 2024-02-19T15:23:31.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.051+0000 7fb873e00640 1 --2- 172.21.15.47:0/1572524967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb87c0d2650 0x7fb87c15a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:23:31.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.055+0000 7fb87be00640 1 --2- 172.21.15.47:0/1572524967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb87c0cc180 0x7fb87c159e40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb86c0ad040 tx=0x7fb86c01acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:31.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.055+0000 7fb878c00640 1 -- 172.21.15.47:0/1572524967 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c0bf610 con 0x7fb87c0cc180 2024-02-19T15:23:31.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.055+0000 7fb878c00640 1 -- 172.21.15.47:0/1572524967 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86c0bf790 con 0x7fb87c0cc180 2024-02-19T15:23:31.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.055+0000 7fb878c00640 1 -- 172.21.15.47:0/1572524967 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c0bfb40 con 0x7fb87c0cc180 2024-02-19T15:23:31.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.055+0000 7fb882e00640 1 -- 172.21.15.47:0/1572524967 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb86c009420 con 0x7fb87c0cc180 2024-02-19T15:23:31.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.055+0000 7fb882e00640 1 -- 172.21.15.47:0/1572524967 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87c0b49a0 con 0x7fb87c0cc180 2024-02-19T15:23:31.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.055+0000 7fb882e00640 1 -- 172.21.15.47:0/1572524967 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb87c0b75a0 con 0x7fb87c0cc180 2024-02-19T15:23:31.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.055+0000 7fb878c00640 1 -- 172.21.15.47:0/1572524967 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb86c0b5400 con 0x7fb87c0cc180 2024-02-19T15:23:31.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.055+0000 7fb878c00640 1 --2- 172.21.15.47:0/1572524967 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb858066e40 0x7fb858069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:31.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.055+0000 7fb878c00640 1 -- 172.21.15.47:0/1572524967 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb86c11f8c0 con 0x7fb87c0cc180 2024-02-19T15:23:31.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.055+0000 7fb873e00640 1 --2- 172.21.15.47:0/1572524967 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb858066e40 0x7fb858069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:31.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.059+0000 7fb873e00640 1 --2- 172.21.15.47:0/1572524967 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb858066e40 0x7fb858069300 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb87c0cd5d0 tx=0x7fb864008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:31.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.059+0000 7fb878c00640 1 -- 172.21.15.47:0/1572524967 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb86c0ec640 con 0x7fb87c0cc180 2024-02-19T15:23:31.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.263+0000 7fb882e00640 1 -- 172.21.15.47:0/1572524967 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb87c1558f0 con 0x7fb87c0cc180 2024-02-19T15:23:31.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.879+0000 7fb878c00640 1 -- 172.21.15.47:0/1572524967 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb86c0efd00 con 0x7fb87c0cc180 2024-02-19T15:23:31.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.883+0000 7fb882e00640 1 -- 172.21.15.47:0/1572524967 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb858066e40 msgr2=0x7fb858069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:31.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.883+0000 7fb882e00640 1 --2- 172.21.15.47:0/1572524967 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb858066e40 0x7fb858069300 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb87c0cd5d0 tx=0x7fb864008040 comp rx=0 tx=0).stop 2024-02-19T15:23:31.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.883+0000 7fb882e00640 1 -- 172.21.15.47:0/1572524967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb87c0cc180 msgr2=0x7fb87c159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:31.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.883+0000 7fb882e00640 1 --2- 172.21.15.47:0/1572524967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb87c0cc180 0x7fb87c159e40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb86c0ad040 tx=0x7fb86c01acb0 comp rx=0 tx=0).stop 2024-02-19T15:23:31.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.883+0000 7fb882e00640 1 -- 172.21.15.47:0/1572524967 shutdown_connections 2024-02-19T15:23:31.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.883+0000 7fb882e00640 1 --2- 172.21.15.47:0/1572524967 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb858066e40 0x7fb858069300 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:31.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.883+0000 7fb882e00640 1 --2- 172.21.15.47:0/1572524967 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb87c15e210 0x7fb87c142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:31.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.883+0000 7fb882e00640 1 --2- 172.21.15.47:0/1572524967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb87c0d2650 0x7fb87c15a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:31.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.883+0000 7fb882e00640 1 --2- 172.21.15.47:0/1572524967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb87c0cc180 0x7fb87c159e40 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:31.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.883+0000 7fb882e00640 1 -- 172.21.15.47:0/1572524967 >> 172.21.15.47:0/1572524967 conn(0x7fb87c0d1b30 msgr2=0x7fb87c15d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:31.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.883+0000 7fb882e00640 1 -- 172.21.15.47:0/1572524967 shutdown_connections 2024-02-19T15:23:31.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:31.883+0000 7fb882e00640 1 -- 172.21.15.47:0/1572524967 wait complete. 2024-02-19T15:23:31.912 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T15:23:32.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58ffe00640 1 Processor -- start 2024-02-19T15:23:32.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58ffe00640 1 -- start start 2024-02-19T15:23:32.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58ffe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5900071520 0x7f5900071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:32.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58ffe00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5900147080 con 0x7f59000acd00 2024-02-19T15:23:32.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58ffe00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5900070d60 con 0x7f5900146ca0 2024-02-19T15:23:32.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58ffe00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59000b50d0 con 0x7f5900071520 2024-02-19T15:23:32.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58ff400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f59000acd00 0x7f59000cfa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33088/0 (socket says 172.21.15.47:33088) 2024-02-19T15:23:32.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58ff400640 1 -- 172.21.15.47:0/4237462105 learned_addr learned my addr 172.21.15.47:0/4237462105 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:32.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58f6000640 1 --2- 172.21.15.47:0/4237462105 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5900071520 0x7f5900071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:32.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58fe000640 1 -- 172.21.15.47:0/4237462105 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760773236 0 0) 0x7f5900147080 con 0x7f59000acd00 2024-02-19T15:23:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58fe000640 1 -- 172.21.15.47:0/4237462105 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58e4003530 con 0x7f59000acd00 2024-02-19T15:23:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58fe000640 1 -- 172.21.15.47:0/4237462105 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325673130 0 0) 0x7f5900070d60 con 0x7f5900146ca0 2024-02-19T15:23:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58fe000640 1 -- 172.21.15.47:0/4237462105 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5900147080 con 0x7f5900146ca0 2024-02-19T15:23:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58fe000640 1 -- 172.21.15.47:0/4237462105 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 550769794 0 0) 0x7f58e4003530 con 0x7f59000acd00 2024-02-19T15:23:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58fe000640 1 -- 172.21.15.47:0/4237462105 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5900071520 msgr2=0x7f5900071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58fe000640 1 --2- 172.21.15.47:0/4237462105 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5900071520 0x7f5900071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58fe000640 1 -- 172.21.15.47:0/4237462105 >> v1:172.21.15.116:6789/0 conn(0x7f5900146ca0 legacy=0x7f590006e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58fe000640 1 -- 172.21.15.47:0/4237462105 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59000b4b10 con 0x7f59000acd00 2024-02-19T15:23:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58fe000640 1 -- 172.21.15.47:0/4237462105 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f58f8001a00 con 0x7f59000acd00 2024-02-19T15:23:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58fe000640 1 -- 172.21.15.47:0/4237462105 >> v1:172.21.15.47:6789/0 conn(0x7f59000acd00 legacy=0x7f59000cfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.103+0000 7f58fe000640 1 --2- 172.21.15.47:0/4237462105 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58e4003b40 0x7f58e4003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.107+0000 7f58fe000640 1 --2- 172.21.15.47:0/4237462105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58e40047d0 0x7f58e4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.107+0000 7f58fe000640 1 --2- 172.21.15.47:0/4237462105 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58e4005190 0x7f58e400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.107+0000 7f58fe000640 1 -- 172.21.15.47:0/4237462105 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59000b50d0 con 0x7f58e4003b40 2024-02-19T15:23:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.107+0000 7f58fe000640 1 -- 172.21.15.47:0/4237462105 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e400bf80 con 0x7f58e40047d0 2024-02-19T15:23:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.107+0000 7f58fe000640 1 -- 172.21.15.47:0/4237462105 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e400c140 con 0x7f58e4005190 2024-02-19T15:23:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.107+0000 7f58f6000640 1 --2- 172.21.15.47:0/4237462105 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5900071520 0x7f5900071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:23:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.107+0000 7f58fea00640 1 --2- 172.21.15.47:0/4237462105 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58e4003b40 0x7f58e4003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.107+0000 7f58ff400640 1 --2- 172.21.15.47:0/4237462105 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58e4005190 0x7f58e400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.107+0000 7f58f6000640 1 --2- 172.21.15.47:0/4237462105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58e40047d0 0x7f58e4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.107+0000 7f58fea00640 1 -- 172.21.15.47:0/4237462105 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58e4005190 msgr2=0x7f58e400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.107+0000 7f58fea00640 1 --2- 172.21.15.47:0/4237462105 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58e4005190 0x7f58e400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.107+0000 7f58fea00640 1 -- 172.21.15.47:0/4237462105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58e40047d0 msgr2=0x7f58e4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.107+0000 7f58fea00640 1 --2- 172.21.15.47:0/4237462105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58e40047d0 0x7f58e4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.107+0000 7f58fea00640 1 -- 172.21.15.47:0/4237462105 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e4003530 con 0x7f58e4003b40 2024-02-19T15:23:32.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.107+0000 7f58fea00640 1 --2- 172.21.15.47:0/4237462105 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58e4003b40 0x7f58e4003f40 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f58f00018c0 tx=0x7f58f001c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2e3ad84379ec115 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:32.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.107+0000 7f58fe000640 1 -- 172.21.15.47:0/4237462105 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f00052d0 con 0x7f58e4003b40 2024-02-19T15:23:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.107+0000 7f58fe000640 1 -- 172.21.15.47:0/4237462105 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f0005430 con 0x7f58e4003b40 2024-02-19T15:23:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.107+0000 7f58fe000640 1 -- 172.21.15.47:0/4237462105 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0003820 con 0x7f58e4003b40 2024-02-19T15:23:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58ffe00640 1 -- 172.21.15.47:0/4237462105 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58e4003b40 msgr2=0x7f58e4003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58ffe00640 1 --2- 172.21.15.47:0/4237462105 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58e4003b40 0x7f58e4003f40 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f58f00018c0 tx=0x7f58f001c550 comp rx=0 tx=0).stop 2024-02-19T15:23:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58ff400640 1 -- 172.21.15.47:0/4237462105 reap_dead start 2024-02-19T15:23:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58ffe00640 1 -- 172.21.15.47:0/4237462105 shutdown_connections 2024-02-19T15:23:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58ffe00640 1 -- 172.21.15.47:0/4237462105 >> 172.21.15.47:0/4237462105 conn(0x7f590006aeb0 msgr2=0x7f590006b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58ffe00640 1 -- 172.21.15.47:0/4237462105 shutdown_connections 2024-02-19T15:23:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58ffe00640 1 -- 172.21.15.47:0/4237462105 wait complete. 2024-02-19T15:23:32.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58ffe00640 1 Processor -- start 2024-02-19T15:23:32.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58ffe00640 1 -- start start 2024-02-19T15:23:32.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58ffe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58e4003b40 0x7f590015e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:32.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58ffe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5900143f20 0x7f59001443a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:32.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58ffe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f590015bf70 0x7f590015c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:32.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58ffe00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5900144da0 con 0x7f5900143f20 2024-02-19T15:23:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58ffe00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59001448e0 con 0x7f58e4003b40 2024-02-19T15:23:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58ffe00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5900141f80 con 0x7f590015bf70 2024-02-19T15:23:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58f6000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5900143f20 0x7f59001443a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58f6000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5900143f20 0x7f59001443a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60374/0 (socket says 172.21.15.47:60374) 2024-02-19T15:23:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58f6000640 1 -- 172.21.15.47:0/3921541742 learned_addr learned my addr 172.21.15.47:0/3921541742 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58fea00640 1 --2- 172.21.15.47:0/3921541742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58e4003b40 0x7f590015e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58ff400640 1 --2- 172.21.15.47:0/3921541742 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f590015bf70 0x7f590015c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58f6000640 1 -- 172.21.15.47:0/3921541742 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f590015bf70 msgr2=0x7f590015c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58f6000640 1 --2- 172.21.15.47:0/3921541742 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f590015bf70 0x7f590015c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58f6000640 1 -- 172.21.15.47:0/3921541742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58e4003b40 msgr2=0x7f590015e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58f6000640 1 --2- 172.21.15.47:0/3921541742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58e4003b40 0x7f590015e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.111+0000 7f58f6000640 1 -- 172.21.15.47:0/3921541742 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5900142260 con 0x7f5900143f20 2024-02-19T15:23:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.115+0000 7f58f6000640 1 --2- 172.21.15.47:0/3921541742 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5900143f20 0x7f59001443a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f58e8011c00 tx=0x7f58e800f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.115+0000 7f58f7e00640 1 -- 172.21.15.47:0/3921541742 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e8011cf0 con 0x7f5900143f20 2024-02-19T15:23:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.115+0000 7f58f7e00640 1 -- 172.21.15.47:0/3921541742 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e8011e50 con 0x7f5900143f20 2024-02-19T15:23:32.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.115+0000 7f58f7e00640 1 -- 172.21.15.47:0/3921541742 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e8014640 con 0x7f5900143f20 2024-02-19T15:23:32.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.115+0000 7f58ffe00640 1 -- 172.21.15.47:0/3921541742 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5900160970 con 0x7f5900143f20 2024-02-19T15:23:32.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.115+0000 7f58ffe00640 1 -- 172.21.15.47:0/3921541742 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5900160650 con 0x7f5900143f20 2024-02-19T15:23:32.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.115+0000 7f58ffe00640 1 -- 172.21.15.47:0/3921541742 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f590004f260 con 0x7f5900143f20 2024-02-19T15:23:32.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.115+0000 7f58f7e00640 1 -- 172.21.15.47:0/3921541742 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f58e80147a0 con 0x7f5900143f20 2024-02-19T15:23:32.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.115+0000 7f58f7e00640 1 --2- 172.21.15.47:0/3921541742 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f58dc066e40 0x7f58dc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:32.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.115+0000 7f58f7e00640 1 -- 172.21.15.47:0/3921541742 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f58e8085410 con 0x7f5900143f20 2024-02-19T15:23:32.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.115+0000 7f58fea00640 1 --2- 172.21.15.47:0/3921541742 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f58dc066e40 0x7f58dc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:32.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.119+0000 7f58fea00640 1 --2- 172.21.15.47:0/3921541742 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f58dc066e40 0x7f58dc069300 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f58f00ad000 tx=0x7f58f0001350 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:32.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.119+0000 7f58f7e00640 1 -- 172.21.15.47:0/3921541742 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f58e8052270 con 0x7f5900143f20 2024-02-19T15:23:32.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.283+0000 7f58ffe00640 1 -- 172.21.15.47:0/3921541742 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5900167280 con 0x7f5900143f20 2024-02-19T15:23:32.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.879+0000 7f58f7e00640 1 -- 172.21.15.47:0/3921541742 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v36) v1 ==== 181+0+0 (secure 0 0 0) 0x7f58e8055930 con 0x7f5900143f20 2024-02-19T15:23:32.887 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_metadata' removed 2024-02-19T15:23:32.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.887+0000 7f58ffe00640 1 -- 172.21.15.47:0/3921541742 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f58dc066e40 msgr2=0x7f58dc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:32.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.887+0000 7f58ffe00640 1 --2- 172.21.15.47:0/3921541742 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f58dc066e40 0x7f58dc069300 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f58f00ad000 tx=0x7f58f0001350 comp rx=0 tx=0).stop 2024-02-19T15:23:32.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.887+0000 7f58ffe00640 1 -- 172.21.15.47:0/3921541742 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5900143f20 msgr2=0x7f59001443a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:32.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.887+0000 7f58ffe00640 1 --2- 172.21.15.47:0/3921541742 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5900143f20 0x7f59001443a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f58e8011c00 tx=0x7f58e800f610 comp rx=0 tx=0).stop 2024-02-19T15:23:32.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.887+0000 7f58ffe00640 1 -- 172.21.15.47:0/3921541742 shutdown_connections 2024-02-19T15:23:32.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.887+0000 7f58ffe00640 1 --2- 172.21.15.47:0/3921541742 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f58dc066e40 0x7f58dc069300 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:32.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.887+0000 7f58ffe00640 1 --2- 172.21.15.47:0/3921541742 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f590015bf70 0x7f590015c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:32.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.887+0000 7f58ffe00640 1 --2- 172.21.15.47:0/3921541742 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5900143f20 0x7f59001443a0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:32.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.887+0000 7f58ffe00640 1 --2- 172.21.15.47:0/3921541742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58e4003b40 0x7f590015e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:32.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.887+0000 7f58ffe00640 1 -- 172.21.15.47:0/3921541742 >> 172.21.15.47:0/3921541742 conn(0x7f590006aeb0 msgr2=0x7f59001452b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:32.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.887+0000 7f58ffe00640 1 -- 172.21.15.47:0/3921541742 shutdown_connections 2024-02-19T15:23:32.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:32.887+0000 7f58ffe00640 1 -- 172.21.15.47:0/3921541742 wait complete. 2024-02-19T15:23:32.919 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T15:23:33.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.059+0000 7fc0b1800640 1 Processor -- start 2024-02-19T15:23:33.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.059+0000 7fc0b1800640 1 -- start start 2024-02-19T15:23:33.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.063+0000 7fc0b1800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc0ac06e570 0x7fc0ac071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:33.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.063+0000 7fc0b1800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0ac0ca520 con 0x7fc0ac15abf0 2024-02-19T15:23:33.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.063+0000 7fc0b1800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0ac0ca800 con 0x7fc0ac071050 2024-02-19T15:23:33.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.063+0000 7fc0b1800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0ac0caae0 con 0x7fc0ac06e570 2024-02-19T15:23:33.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.063+0000 7fc0aaa00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc0ac15abf0 0x7fc0ac06de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33096/0 (socket says 172.21.15.47:33096) 2024-02-19T15:23:33.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.063+0000 7fc0aaa00640 1 -- 172.21.15.47:0/293820392 learned_addr learned my addr 172.21.15.47:0/293820392 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:33.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.063+0000 7fc0a3e00640 1 --2- 172.21.15.47:0/293820392 >> v2:172.21.15.116:3300/0 conn(0x7fc0ac06e570 0x7fc0ac071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:33.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.063+0000 7fc0aa000640 1 -- 172.21.15.47:0/293820392 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296314441 0 0) 0x7fc0ac0ca520 con 0x7fc0ac15abf0 2024-02-19T15:23:33.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.063+0000 7fc0aa000640 1 -- 172.21.15.47:0/293820392 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc090003530 con 0x7fc0ac15abf0 2024-02-19T15:23:33.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.063+0000 7fc0aa000640 1 -- 172.21.15.47:0/293820392 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285080401 0 0) 0x7fc0ac0ca800 con 0x7fc0ac071050 2024-02-19T15:23:33.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.063+0000 7fc0aa000640 1 -- 172.21.15.47:0/293820392 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0ac0ca520 con 0x7fc0ac071050 2024-02-19T15:23:33.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.063+0000 7fc0a3e00640 1 -- 172.21.15.47:0/293820392 >> v1:172.21.15.116:6789/0 conn(0x7fc0ac071050 legacy=0x7fc0ac0ae490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:33.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.063+0000 7fc0a3e00640 1 -- 172.21.15.47:0/293820392 >> v1:172.21.15.47:6789/0 conn(0x7fc0ac15abf0 legacy=0x7fc0ac06de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:33.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.063+0000 7fc0a3e00640 1 -- 172.21.15.47:0/293820392 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ac0cadc0 con 0x7fc0ac06e570 2024-02-19T15:23:33.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.067+0000 7fc0a3e00640 1 --2- 172.21.15.47:0/293820392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0ac06e570 0x7fc0ac071f00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc0940097c0 tx=0x7fc09401c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6f15de8a4d6cd52 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:33.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.067+0000 7fc0aa000640 1 -- 172.21.15.47:0/293820392 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0940aa5f0 con 0x7fc0ac06e570 2024-02-19T15:23:33.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.067+0000 7fc0aa000640 1 -- 172.21.15.47:0/293820392 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0940aa750 con 0x7fc0ac06e570 2024-02-19T15:23:33.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.067+0000 7fc0aa000640 1 -- 172.21.15.47:0/293820392 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0940aaae0 con 0x7fc0ac06e570 2024-02-19T15:23:33.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.067+0000 7fc0b1800640 1 -- 172.21.15.47:0/293820392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0ac06e570 msgr2=0x7fc0ac071f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:33.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.067+0000 7fc0b1800640 1 --2- 172.21.15.47:0/293820392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0ac06e570 0x7fc0ac071f00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc0940097c0 tx=0x7fc09401c470 comp rx=0 tx=0).stop 2024-02-19T15:23:33.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.067+0000 7fc0b1800640 1 -- 172.21.15.47:0/293820392 shutdown_connections 2024-02-19T15:23:33.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.067+0000 7fc0b1800640 1 --2- 172.21.15.47:0/293820392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0ac06e570 0x7fc0ac071f00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:33.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.067+0000 7fc0b1800640 1 -- 172.21.15.47:0/293820392 >> 172.21.15.47:0/293820392 conn(0x7fc0ac0cc3f0 msgr2=0x7fc0ac0cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:33.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.067+0000 7fc0b1800640 1 -- 172.21.15.47:0/293820392 shutdown_connections 2024-02-19T15:23:33.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.067+0000 7fc0b1800640 1 -- 172.21.15.47:0/293820392 wait complete. 2024-02-19T15:23:33.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.067+0000 7fc0b1800640 1 Processor -- start 2024-02-19T15:23:33.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.067+0000 7fc0b1800640 1 -- start start 2024-02-19T15:23:33.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0b1800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0ac06e570 0x7fc0ac15c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:33.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0b1800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0ac071050 0x7fc0ac0b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0b1800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc0ac15abf0 0x7fc0ac0ad850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0b1800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ac0b46a0 con 0x7fc0ac071050 2024-02-19T15:23:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0b1800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ac1f6600 con 0x7fc0ac06e570 2024-02-19T15:23:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0b1800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ac0b3b20 con 0x7fc0ac15abf0 2024-02-19T15:23:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0a3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0ac071050 0x7fc0ac0b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0a3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0ac071050 0x7fc0ac0b30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60378/0 (socket says 172.21.15.47:60378) 2024-02-19T15:23:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0a3e00640 1 -- 172.21.15.47:0/973712729 learned_addr learned my addr 172.21.15.47:0/973712729 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0aaa00640 1 --2- 172.21.15.47:0/973712729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0ac06e570 0x7fc0ac15c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0ab400640 1 --2- 172.21.15.47:0/973712729 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc0ac15abf0 0x7fc0ac0ad850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0a3e00640 1 -- 172.21.15.47:0/973712729 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc0ac15abf0 msgr2=0x7fc0ac0ad850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0a3e00640 1 --2- 172.21.15.47:0/973712729 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc0ac15abf0 0x7fc0ac0ad850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0a3e00640 1 -- 172.21.15.47:0/973712729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0ac06e570 msgr2=0x7fc0ac15c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0a3e00640 1 --2- 172.21.15.47:0/973712729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0ac06e570 0x7fc0ac15c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0a3e00640 1 -- 172.21.15.47:0/973712729 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc094009420 con 0x7fc0ac071050 2024-02-19T15:23:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0aaa00640 1 --2- 172.21.15.47:0/973712729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0ac06e570 0x7fc0ac15c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0a3e00640 1 --2- 172.21.15.47:0/973712729 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0ac071050 0x7fc0ac0b30d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc09401c950 tx=0x7fc09401acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0a2a00640 1 -- 172.21.15.47:0/973712729 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0940be610 con 0x7fc0ac071050 2024-02-19T15:23:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0a2a00640 1 -- 172.21.15.47:0/973712729 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0940be7b0 con 0x7fc0ac071050 2024-02-19T15:23:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0a2a00640 1 -- 172.21.15.47:0/973712729 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0940beb60 con 0x7fc0ac071050 2024-02-19T15:23:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0b1800640 1 -- 172.21.15.47:0/973712729 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0ac0b3e00 con 0x7fc0ac071050 2024-02-19T15:23:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0b1800640 1 -- 172.21.15.47:0/973712729 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0ac0b43c0 con 0x7fc0ac071050 2024-02-19T15:23:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0b1800640 1 -- 172.21.15.47:0/973712729 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0ac075360 con 0x7fc0ac071050 2024-02-19T15:23:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.071+0000 7fc0a2a00640 1 -- 172.21.15.47:0/973712729 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc0940b4400 con 0x7fc0ac071050 2024-02-19T15:23:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.075+0000 7fc0a2a00640 1 --2- 172.21.15.47:0/973712729 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc088066e40 0x7fc088069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.075+0000 7fc0a2a00640 1 -- 172.21.15.47:0/973712729 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fc0940a7070 con 0x7fc0ac071050 2024-02-19T15:23:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.075+0000 7fc0aaa00640 1 --2- 172.21.15.47:0/973712729 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc088066e40 0x7fc088069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:33.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.075+0000 7fc0aaa00640 1 --2- 172.21.15.47:0/973712729 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc088066e40 0x7fc088069300 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc09c000f70 tx=0x7fc09c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:33.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.079+0000 7fc0a2a00640 1 -- 172.21.15.47:0/973712729 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc0940ec4a0 con 0x7fc0ac071050 2024-02-19T15:23:33.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.239+0000 7fc0b1800640 1 -- 172.21.15.47:0/973712729 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc0ac0cd300 con 0x7fc0ac071050 2024-02-19T15:23:33.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.887+0000 7fc0a2a00640 1 -- 172.21.15.47:0/973712729 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v37) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc0940efb60 con 0x7fc0ac071050 2024-02-19T15:23:33.892 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_data' removed 2024-02-19T15:23:33.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.887+0000 7fc0b1800640 1 -- 172.21.15.47:0/973712729 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc088066e40 msgr2=0x7fc088069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:33.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.887+0000 7fc0b1800640 1 --2- 172.21.15.47:0/973712729 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc088066e40 0x7fc088069300 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc09c000f70 tx=0x7fc09c009040 comp rx=0 tx=0).stop 2024-02-19T15:23:33.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.887+0000 7fc0b1800640 1 -- 172.21.15.47:0/973712729 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0ac071050 msgr2=0x7fc0ac0b30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:33.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.887+0000 7fc0b1800640 1 --2- 172.21.15.47:0/973712729 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0ac071050 0x7fc0ac0b30d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc09401c950 tx=0x7fc09401acb0 comp rx=0 tx=0).stop 2024-02-19T15:23:33.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.891+0000 7fc0b1800640 1 -- 172.21.15.47:0/973712729 shutdown_connections 2024-02-19T15:23:33.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.891+0000 7fc0b1800640 1 --2- 172.21.15.47:0/973712729 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc088066e40 0x7fc088069300 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:33.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.891+0000 7fc0b1800640 1 --2- 172.21.15.47:0/973712729 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc0ac15abf0 0x7fc0ac0ad850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:33.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.891+0000 7fc0b1800640 1 --2- 172.21.15.47:0/973712729 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0ac071050 0x7fc0ac0b30d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:33.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.891+0000 7fc0b1800640 1 --2- 172.21.15.47:0/973712729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0ac06e570 0x7fc0ac15c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:33.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.891+0000 7fc0b1800640 1 -- 172.21.15.47:0/973712729 >> 172.21.15.47:0/973712729 conn(0x7fc0ac0cc3f0 msgr2=0x7fc0ac0af450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:33.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.891+0000 7fc0b1800640 1 -- 172.21.15.47:0/973712729 shutdown_connections 2024-02-19T15:23:33.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:33.891+0000 7fc0b1800640 1 -- 172.21.15.47:0/973712729 wait complete. 2024-02-19T15:23:33.926 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-19T15:23:34.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.067+0000 7f3dfd800640 1 Processor -- start 2024-02-19T15:23:34.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.067+0000 7f3dfd800640 1 -- start start 2024-02-19T15:23:34.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.071+0000 7f3dfd800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3df815bd90 0x7f3df8070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:34.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.071+0000 7f3dfd800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3df80cb490 0x7f3df8072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:34.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.071+0000 7f3dfd800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df815fc90 con 0x7f3df8072680 2024-02-19T15:23:34.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.071+0000 7f3dfd800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3df815ff70 con 0x7f3df815bd90 2024-02-19T15:23:34.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.071+0000 7f3dfd800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df8160250 con 0x7f3df80cb490 2024-02-19T15:23:34.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.071+0000 7f3df6a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3df815bd90 0x7f3df8070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:34.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.071+0000 7f3df6a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3df815bd90 0x7f3df8070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52234/0 (socket says 172.21.15.47:52234) 2024-02-19T15:23:34.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.071+0000 7f3df6a00640 1 -- 172.21.15.47:0/2470747761 learned_addr learned my addr 172.21.15.47:0/2470747761 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:34.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.071+0000 7f3dee000640 1 --2- 172.21.15.47:0/2470747761 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3df80cb490 0x7f3df8072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:34.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.071+0000 7f3df6a00640 1 -- 172.21.15.47:0/2470747761 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3df80cb490 msgr2=0x7f3df8072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:34.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.071+0000 7f3df6a00640 1 --2- 172.21.15.47:0/2470747761 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3df80cb490 0x7f3df8072140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:34.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.071+0000 7f3df6a00640 1 -- 172.21.15.47:0/2470747761 >> v1:172.21.15.116:6789/0 conn(0x7f3df8072680 legacy=0x7f3df80ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:34.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.071+0000 7f3df6a00640 1 -- 172.21.15.47:0/2470747761 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df8160530 con 0x7f3df815bd90 2024-02-19T15:23:34.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.071+0000 7f3dee000640 1 --2- 172.21.15.47:0/2470747761 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3df80cb490 0x7f3df8072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:23:34.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.071+0000 7f3df6a00640 1 --2- 172.21.15.47:0/2470747761 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df815bd90 0x7f3df8070960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009870 tx=0x7f3de801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae71264e5354d08 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:34.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3df6000640 1 -- 172.21.15.47:0/2470747761 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de8004230 con 0x7f3df815bd90 2024-02-19T15:23:34.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3df6000640 1 -- 172.21.15.47:0/2470747761 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de80a7050 con 0x7f3df815bd90 2024-02-19T15:23:34.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3df6000640 1 -- 172.21.15.47:0/2470747761 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de8003620 con 0x7f3df815bd90 2024-02-19T15:23:34.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3dfd800640 1 -- 172.21.15.47:0/2470747761 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df815bd90 msgr2=0x7f3df8070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:34.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3dfd800640 1 --2- 172.21.15.47:0/2470747761 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df815bd90 0x7f3df8070960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009870 tx=0x7f3de801c400 comp rx=0 tx=0).stop 2024-02-19T15:23:34.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3dfd800640 1 -- 172.21.15.47:0/2470747761 shutdown_connections 2024-02-19T15:23:34.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3dfd800640 1 --2- 172.21.15.47:0/2470747761 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3df80cb490 0x7f3df8072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:34.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3dfd800640 1 --2- 172.21.15.47:0/2470747761 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df815bd90 0x7f3df8070960 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:34.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3dfd800640 1 -- 172.21.15.47:0/2470747761 >> 172.21.15.47:0/2470747761 conn(0x7f3df80cfc20 msgr2=0x7f3df80d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:34.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3dfd800640 1 -- 172.21.15.47:0/2470747761 shutdown_connections 2024-02-19T15:23:34.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3dfd800640 1 -- 172.21.15.47:0/2470747761 wait complete. 2024-02-19T15:23:34.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3dfd800640 1 Processor -- start 2024-02-19T15:23:34.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3dfd800640 1 -- start start 2024-02-19T15:23:34.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3dfd800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df8072680 0x7f3df8152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:34.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3dfd800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3df80cb490 0x7f3df81654e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:34.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3dfd800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3df815bd90 0x7f3df80c9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:34.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3dfd800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df8155370 con 0x7f3df8072680 2024-02-19T15:23:34.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3dfd800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df8155650 con 0x7f3df80cb490 2024-02-19T15:23:34.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3dfd800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df8155930 con 0x7f3df815bd90 2024-02-19T15:23:34.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3df6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df8072680 0x7f3df8152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:34.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3df6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df8072680 0x7f3df8152ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52240/0 (socket says 172.21.15.47:52240) 2024-02-19T15:23:34.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3df6a00640 1 -- 172.21.15.47:0/2101879636 learned_addr learned my addr 172.21.15.47:0/2101879636 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:34.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3df6a00640 1 -- 172.21.15.47:0/2101879636 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3df815bd90 msgr2=0x7f3df80c9510 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-19T15:23:34.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3df6a00640 1 --2- 172.21.15.47:0/2101879636 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3df815bd90 0x7f3df80c9510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:34.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3df6a00640 1 -- 172.21.15.47:0/2101879636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3df80cb490 msgr2=0x7f3df81654e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:34.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3df6a00640 1 --2- 172.21.15.47:0/2101879636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3df80cb490 0x7f3df81654e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:34.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.075+0000 7f3df6a00640 1 -- 172.21.15.47:0/2101879636 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de80094d0 con 0x7f3df8072680 2024-02-19T15:23:34.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.079+0000 7f3df6a00640 1 --2- 172.21.15.47:0/2101879636 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df8072680 0x7f3df8152ce0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3de801a030 tx=0x7f3de80191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:34.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.079+0000 7f3def400640 1 -- 172.21.15.47:0/2101879636 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de8003de0 con 0x7f3df8072680 2024-02-19T15:23:34.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.079+0000 7f3def400640 1 -- 172.21.15.47:0/2101879636 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de80b6440 con 0x7f3df8072680 2024-02-19T15:23:34.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.079+0000 7f3dfd800640 1 -- 172.21.15.47:0/2101879636 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df8155c10 con 0x7f3df8072680 2024-02-19T15:23:34.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.079+0000 7f3def400640 1 -- 172.21.15.47:0/2101879636 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de80b67d0 con 0x7f3df8072680 2024-02-19T15:23:34.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.079+0000 7f3dfd800640 1 -- 172.21.15.47:0/2101879636 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df8165a20 con 0x7f3df8072680 2024-02-19T15:23:34.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.079+0000 7f3dfd800640 1 -- 172.21.15.47:0/2101879636 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df8075360 con 0x7f3df8072680 2024-02-19T15:23:34.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.079+0000 7f3def400640 1 -- 172.21.15.47:0/2101879636 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3de80b2040 con 0x7f3df8072680 2024-02-19T15:23:34.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.079+0000 7f3def400640 1 --2- 172.21.15.47:0/2101879636 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3dd4066e40 0x7f3dd4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:34.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.079+0000 7f3def400640 1 -- 172.21.15.47:0/2101879636 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3de811ec80 con 0x7f3df8072680 2024-02-19T15:23:34.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.083+0000 7f3dee000640 1 --2- 172.21.15.47:0/2101879636 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3dd4066e40 0x7f3dd4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:34.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.083+0000 7f3dee000640 1 --2- 172.21.15.47:0/2101879636 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3dd4066e40 0x7f3dd4069300 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3df80ca0a0 tx=0x7f3de0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:34.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.083+0000 7f3def400640 1 -- 172.21.15.47:0/2101879636 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f3de80ebe80 con 0x7f3df8072680 2024-02-19T15:23:34.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.255+0000 7f3dfd800640 1 -- 172.21.15.47:0/2101879636 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3df806e800 con 0x7f3df8072680 2024-02-19T15:23:34.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.259+0000 7f3def400640 1 -- 172.21.15.47:0/2101879636 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3de80ef540 con 0x7f3df8072680 2024-02-19T15:23:34.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.259+0000 7f3dfd800640 1 -- 172.21.15.47:0/2101879636 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3dd4066e40 msgr2=0x7f3dd4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:34.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.259+0000 7f3dfd800640 1 --2- 172.21.15.47:0/2101879636 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3dd4066e40 0x7f3dd4069300 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3df80ca0a0 tx=0x7f3de0008040 comp rx=0 tx=0).stop 2024-02-19T15:23:34.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.259+0000 7f3dfd800640 1 -- 172.21.15.47:0/2101879636 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df8072680 msgr2=0x7f3df8152ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:34.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.259+0000 7f3dfd800640 1 --2- 172.21.15.47:0/2101879636 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df8072680 0x7f3df8152ce0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3de801a030 tx=0x7f3de80191a0 comp rx=0 tx=0).stop 2024-02-19T15:23:34.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.259+0000 7f3dfd800640 1 -- 172.21.15.47:0/2101879636 shutdown_connections 2024-02-19T15:23:34.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.259+0000 7f3dfd800640 1 --2- 172.21.15.47:0/2101879636 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3dd4066e40 0x7f3dd4069300 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:34.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.259+0000 7f3dfd800640 1 --2- 172.21.15.47:0/2101879636 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3df815bd90 0x7f3df80c9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:34.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.259+0000 7f3dfd800640 1 --2- 172.21.15.47:0/2101879636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3df80cb490 0x7f3df81654e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:34.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.259+0000 7f3dfd800640 1 --2- 172.21.15.47:0/2101879636 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df8072680 0x7f3df8152ce0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:34.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.259+0000 7f3dfd800640 1 -- 172.21.15.47:0/2101879636 >> 172.21.15.47:0/2101879636 conn(0x7f3df80cfc20 msgr2=0x7f3df80c6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:34.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.263+0000 7f3dfd800640 1 -- 172.21.15.47:0/2101879636 shutdown_connections 2024-02-19T15:23:34.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.263+0000 7f3dfd800640 1 -- 172.21.15.47:0/2101879636 wait complete. 2024-02-19T15:23:34.286 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-02-19T15:23:34.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.419+0000 7f1672800640 1 Processor -- start 2024-02-19T15:23:34.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.423+0000 7f1672800640 1 -- start start 2024-02-19T15:23:34.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.423+0000 7f1672800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f166c15f620 0x7f166c0b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:34.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.423+0000 7f1672800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f166c0b2330 0x7f166c06ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:34.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.423+0000 7f1672800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166c06b5d0 con 0x7f166c1614e0 2024-02-19T15:23:34.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.423+0000 7f1672800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f166c0b1c00 con 0x7f166c15f620 2024-02-19T15:23:34.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.423+0000 7f1672800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166c0b27e0 con 0x7f166c0b2330 2024-02-19T15:23:34.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.423+0000 7f1662a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f166c15f620 0x7f166c0b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:34.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.423+0000 7f1662a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f166c15f620 0x7f166c0b1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52248/0 (socket says 172.21.15.47:52248) 2024-02-19T15:23:34.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.423+0000 7f1662a00640 1 -- 172.21.15.47:0/2354925255 learned_addr learned my addr 172.21.15.47:0/2354925255 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:34.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.423+0000 7f166be00640 1 --2- 172.21.15.47:0/2354925255 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f166c0b2330 0x7f166c06ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:34.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.423+0000 7f1662a00640 1 -- 172.21.15.47:0/2354925255 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f166c0b2330 msgr2=0x7f166c06ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:34.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.423+0000 7f1662a00640 1 --2- 172.21.15.47:0/2354925255 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f166c0b2330 0x7f166c06ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:34.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.423+0000 7f1662a00640 1 -- 172.21.15.47:0/2354925255 >> v1:172.21.15.116:6789/0 conn(0x7f166c1614e0 legacy=0x7f166c15eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:34.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.423+0000 7f1662a00640 1 -- 172.21.15.47:0/2354925255 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166c0b1640 con 0x7f166c15f620 2024-02-19T15:23:34.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.423+0000 7f166aa00640 1 -- 172.21.15.47:0/2354925255 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42030537 0 0) 0x7f166c06b5d0 con 0x7f166c1614e0 2024-02-19T15:23:34.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.427+0000 7f1662a00640 1 --2- 172.21.15.47:0/2354925255 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f166c15f620 0x7f166c0b1df0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1654009870 tx=0x7f165401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cf3b67af4d50616 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:34.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.427+0000 7f166aa00640 1 -- 172.21.15.47:0/2354925255 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16540a9620 con 0x7f166c15f620 2024-02-19T15:23:34.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.427+0000 7f166aa00640 1 -- 172.21.15.47:0/2354925255 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16540a9780 con 0x7f166c15f620 2024-02-19T15:23:34.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.427+0000 7f166aa00640 1 -- 172.21.15.47:0/2354925255 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16540a9b10 con 0x7f166c15f620 2024-02-19T15:23:34.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.427+0000 7f1672800640 1 -- 172.21.15.47:0/2354925255 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f166c15f620 msgr2=0x7f166c0b1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:34.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.427+0000 7f1672800640 1 --2- 172.21.15.47:0/2354925255 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f166c15f620 0x7f166c0b1df0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1654009870 tx=0x7f165401c560 comp rx=0 tx=0).stop 2024-02-19T15:23:34.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.427+0000 7f1672800640 1 -- 172.21.15.47:0/2354925255 shutdown_connections 2024-02-19T15:23:34.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.427+0000 7f1672800640 1 --2- 172.21.15.47:0/2354925255 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f166c0b2330 0x7f166c06ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:34.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.427+0000 7f1672800640 1 --2- 172.21.15.47:0/2354925255 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f166c15f620 0x7f166c0b1df0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:34.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.427+0000 7f1672800640 1 -- 172.21.15.47:0/2354925255 >> 172.21.15.47:0/2354925255 conn(0x7f166c0b4fc0 msgr2=0x7f166c0b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:34.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.427+0000 7f1672800640 1 -- 172.21.15.47:0/2354925255 shutdown_connections 2024-02-19T15:23:34.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.427+0000 7f1672800640 1 -- 172.21.15.47:0/2354925255 wait complete. 2024-02-19T15:23:34.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.427+0000 7f1672800640 1 Processor -- start 2024-02-19T15:23:34.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.427+0000 7f1672800640 1 -- start start 2024-02-19T15:23:34.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f1672800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f166c0b2330 0x7f166c0af960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:34.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f1672800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f166c15f620 0x7f166c1619e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:34.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f1672800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f166c1614e0 0x7f166c161f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:34.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f1672800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c0ca930 con 0x7f166c1614e0 2024-02-19T15:23:34.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f1672800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c0c9f90 con 0x7f166c15f620 2024-02-19T15:23:34.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f1672800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166c0ca270 con 0x7f166c0b2330 2024-02-19T15:23:34.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f166be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f166c1614e0 0x7f166c161f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:34.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f166be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f166c1614e0 0x7f166c161f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52262/0 (socket says 172.21.15.47:52262) 2024-02-19T15:23:34.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f166be00640 1 -- 172.21.15.47:0/1059659830 learned_addr learned my addr 172.21.15.47:0/1059659830 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:34.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f166b400640 1 --2- 172.21.15.47:0/1059659830 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f166c0b2330 0x7f166c0af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:34.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f1662a00640 1 --2- 172.21.15.47:0/1059659830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f166c15f620 0x7f166c1619e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:34.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f166be00640 1 -- 172.21.15.47:0/1059659830 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f166c0b2330 msgr2=0x7f166c0af960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:34.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f166be00640 1 --2- 172.21.15.47:0/1059659830 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f166c0b2330 0x7f166c0af960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:34.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f166be00640 1 -- 172.21.15.47:0/1059659830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f166c15f620 msgr2=0x7f166c1619e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:34.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f166be00640 1 --2- 172.21.15.47:0/1059659830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f166c15f620 0x7f166c1619e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:34.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f166be00640 1 -- 172.21.15.47:0/1059659830 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16540094d0 con 0x7f166c1614e0 2024-02-19T15:23:34.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f166be00640 1 --2- 172.21.15.47:0/1059659830 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f166c1614e0 0x7f166c161f20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f166400efc0 tx=0x7f166400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:34.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f1663e00640 1 -- 172.21.15.47:0/1059659830 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1664014480 con 0x7f166c1614e0 2024-02-19T15:23:34.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f1663e00640 1 -- 172.21.15.47:0/1059659830 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f166400f040 con 0x7f166c1614e0 2024-02-19T15:23:34.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f1663e00640 1 -- 172.21.15.47:0/1059659830 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1664014820 con 0x7f166c1614e0 2024-02-19T15:23:34.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f1672800640 1 -- 172.21.15.47:0/1059659830 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166c0ca550 con 0x7f166c1614e0 2024-02-19T15:23:34.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f1672800640 1 -- 172.21.15.47:0/1059659830 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166c0aff40 con 0x7f166c1614e0 2024-02-19T15:23:34.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f1672800640 1 -- 172.21.15.47:0/1059659830 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166c075360 con 0x7f166c1614e0 2024-02-19T15:23:34.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.431+0000 7f1663e00640 1 -- 172.21.15.47:0/1059659830 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1664014980 con 0x7f166c1614e0 2024-02-19T15:23:34.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.435+0000 7f1663e00640 1 --2- 172.21.15.47:0/1059659830 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1648066e40 0x7f1648069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:34.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.435+0000 7f1663e00640 1 -- 172.21.15.47:0/1059659830 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f16640852b0 con 0x7f166c1614e0 2024-02-19T15:23:34.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.435+0000 7f166b400640 1 --2- 172.21.15.47:0/1059659830 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1648066e40 0x7f1648069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:34.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.435+0000 7f166b400640 1 --2- 172.21.15.47:0/1059659830 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1648066e40 0x7f1648069300 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f166c154720 tx=0x7f165c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:34.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.439+0000 7f1663e00640 1 -- 172.21.15.47:0/1059659830 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1664052390 con 0x7f166c1614e0 2024-02-19T15:23:34.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.659+0000 7f1672800640 1 -- 172.21.15.47:0/1059659830 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7f166c0d0d70 con 0x7f166c1614e0 2024-02-19T15:23:34.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.891+0000 7f1663e00640 1 -- 172.21.15.47:0/1059659830 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v105) v1 ==== 132+0+0 (secure 0 0 0) 0x7f1664083e60 con 0x7f166c1614e0 2024-02-19T15:23:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.891+0000 7f1672800640 1 -- 172.21.15.47:0/1059659830 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1648066e40 msgr2=0x7f1648069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.891+0000 7f1672800640 1 --2- 172.21.15.47:0/1059659830 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1648066e40 0x7f1648069300 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f166c154720 tx=0x7f165c009040 comp rx=0 tx=0).stop 2024-02-19T15:23:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.891+0000 7f1672800640 1 -- 172.21.15.47:0/1059659830 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f166c1614e0 msgr2=0x7f166c161f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.891+0000 7f1672800640 1 --2- 172.21.15.47:0/1059659830 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f166c1614e0 0x7f166c161f20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f166400efc0 tx=0x7f166400c490 comp rx=0 tx=0).stop 2024-02-19T15:23:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.891+0000 7f1672800640 1 -- 172.21.15.47:0/1059659830 shutdown_connections 2024-02-19T15:23:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.891+0000 7f1672800640 1 --2- 172.21.15.47:0/1059659830 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1648066e40 0x7f1648069300 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.891+0000 7f1672800640 1 --2- 172.21.15.47:0/1059659830 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f166c1614e0 0x7f166c161f20 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.891+0000 7f1672800640 1 --2- 172.21.15.47:0/1059659830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f166c15f620 0x7f166c1619e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.891+0000 7f1672800640 1 --2- 172.21.15.47:0/1059659830 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f166c0b2330 0x7f166c0af960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.891+0000 7f1672800640 1 -- 172.21.15.47:0/1059659830 >> 172.21.15.47:0/1059659830 conn(0x7f166c0b4fc0 msgr2=0x7f166c168440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:34.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.891+0000 7f1672800640 1 -- 172.21.15.47:0/1059659830 shutdown_connections 2024-02-19T15:23:34.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:34.891+0000 7f1672800640 1 -- 172.21.15.47:0/1059659830 wait complete. 2024-02-19T15:23:34.917 INFO:tasks.cephfs_test_runner:That when deleting a bunch of dentries and the containing ... ok 2024-02-19T15:23:34.918 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). 2024-02-19T15:23:34.918 INFO:tasks.cephfs_test_runner:test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) 2024-02-19T15:23:34.918 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:23:35.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.055+0000 7f231b000640 1 Processor -- start 2024-02-19T15:23:35.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.055+0000 7f231b000640 1 -- start start 2024-02-19T15:23:35.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.055+0000 7f231b000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f23140ce510 0x7f23140cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:35.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.055+0000 7f231b000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f23140b2b70 0x7f23140ce8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:35.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.055+0000 7f231b000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23140b32c0 con 0x7f23140d0420 2024-02-19T15:23:35.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.055+0000 7f231b000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23140cf6f0 con 0x7f23140b2b70 2024-02-19T15:23:35.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.055+0000 7f231b000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23140cff90 con 0x7f23140ce510 2024-02-19T15:23:35.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.055+0000 7f2313400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f23140b2b70 0x7f23140ce8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:35.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.055+0000 7f2313400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f23140b2b70 0x7f23140ce8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52264/0 (socket says 172.21.15.47:52264) 2024-02-19T15:23:35.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.055+0000 7f2313400640 1 -- 172.21.15.47:0/116919137 learned_addr learned my addr 172.21.15.47:0/116919137 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:35.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.055+0000 7f2313e00640 1 --2- 172.21.15.47:0/116919137 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f23140ce510 0x7f23140cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:35.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.055+0000 7f2313400640 1 -- 172.21.15.47:0/116919137 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f23140ce510 msgr2=0x7f23140cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:35.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.055+0000 7f2313400640 1 --2- 172.21.15.47:0/116919137 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f23140ce510 0x7f23140cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.055+0000 7f2313400640 1 -- 172.21.15.47:0/116919137 >> v1:172.21.15.116:6789/0 conn(0x7f23140d0420 legacy=0x7f231406bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:35.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.055+0000 7f2313400640 1 -- 172.21.15.47:0/116919137 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23140cee30 con 0x7f23140b2b70 2024-02-19T15:23:35.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.059+0000 7f2313400640 1 --2- 172.21.15.47:0/116919137 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23140b2b70 0x7f23140ce8f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f2300009870 tx=0x7f230001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fc16d9c89021d25 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:35.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.059+0000 7f2312a00640 1 -- 172.21.15.47:0/116919137 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23000aa430 con 0x7f23140b2b70 2024-02-19T15:23:35.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.059+0000 7f2312a00640 1 -- 172.21.15.47:0/116919137 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23000aa590 con 0x7f23140b2b70 2024-02-19T15:23:35.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.059+0000 7f2312a00640 1 -- 172.21.15.47:0/116919137 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23000aa920 con 0x7f23140b2b70 2024-02-19T15:23:35.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.059+0000 7f231b000640 1 -- 172.21.15.47:0/116919137 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23140b2b70 msgr2=0x7f23140ce8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:35.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.059+0000 7f231b000640 1 --2- 172.21.15.47:0/116919137 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23140b2b70 0x7f23140ce8f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f2300009870 tx=0x7f230001c4d0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.059+0000 7f231b000640 1 -- 172.21.15.47:0/116919137 shutdown_connections 2024-02-19T15:23:35.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.059+0000 7f231b000640 1 --2- 172.21.15.47:0/116919137 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23140b2b70 0x7f23140ce8f0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.059+0000 7f231b000640 1 --2- 172.21.15.47:0/116919137 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f23140ce510 0x7f23140cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.059+0000 7f231b000640 1 -- 172.21.15.47:0/116919137 >> 172.21.15.47:0/116919137 conn(0x7f23140d1b20 msgr2=0x7f23140d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:35.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.059+0000 7f231b000640 1 -- 172.21.15.47:0/116919137 shutdown_connections 2024-02-19T15:23:35.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.059+0000 7f231b000640 1 -- 172.21.15.47:0/116919137 wait complete. 2024-02-19T15:23:35.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.059+0000 7f231b000640 1 Processor -- start 2024-02-19T15:23:35.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.059+0000 7f231b000640 1 -- start start 2024-02-19T15:23:35.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f231b000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f23140b2b70 0x7f2314152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:35.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f231b000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23140ce510 0x7f23140c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:35.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f231b000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23140d0420 0x7f23141600a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:35.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f231b000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2314153940 con 0x7f23140d0420 2024-02-19T15:23:35.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f231b000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2314153c20 con 0x7f23140ce510 2024-02-19T15:23:35.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f231b000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2314153f00 con 0x7f23140b2b70 2024-02-19T15:23:35.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f2319000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23140d0420 0x7f23141600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:35.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f2319000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23140d0420 0x7f23141600a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52270/0 (socket says 172.21.15.47:52270) 2024-02-19T15:23:35.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f2319000640 1 -- 172.21.15.47:0/774771399 learned_addr learned my addr 172.21.15.47:0/774771399 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:35.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f2313400640 1 --2- 172.21.15.47:0/774771399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23140ce510 0x7f23140c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:35.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f2313e00640 1 --2- 172.21.15.47:0/774771399 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f23140b2b70 0x7f2314152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:35.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f2319000640 1 -- 172.21.15.47:0/774771399 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f23140b2b70 msgr2=0x7f2314152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:35.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f2319000640 1 --2- 172.21.15.47:0/774771399 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f23140b2b70 0x7f2314152600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f2319000640 1 -- 172.21.15.47:0/774771399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23140ce510 msgr2=0x7f23140c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:35.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f2319000640 1 --2- 172.21.15.47:0/774771399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23140ce510 0x7f23140c5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f2319000640 1 -- 172.21.15.47:0/774771399 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23000094d0 con 0x7f23140d0420 2024-02-19T15:23:35.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f2319000640 1 --2- 172.21.15.47:0/774771399 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23140d0420 0x7f23141600a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f230c00c770 tx=0x7f230c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:35.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f230be00640 1 -- 172.21.15.47:0/774771399 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c003b80 con 0x7f23140d0420 2024-02-19T15:23:35.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f230be00640 1 -- 172.21.15.47:0/774771399 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f230c003ce0 con 0x7f23140d0420 2024-02-19T15:23:35.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f230be00640 1 -- 172.21.15.47:0/774771399 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c00aa90 con 0x7f23140d0420 2024-02-19T15:23:35.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f231b000640 1 -- 172.21.15.47:0/774771399 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23140c71b0 con 0x7f23140d0420 2024-02-19T15:23:35.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f231b000640 1 -- 172.21.15.47:0/774771399 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2314152b40 con 0x7f23140d0420 2024-02-19T15:23:35.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.063+0000 7f231b000640 1 -- 172.21.15.47:0/774771399 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2314075360 con 0x7f23140d0420 2024-02-19T15:23:35.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.067+0000 7f230be00640 1 -- 172.21.15.47:0/774771399 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f230c00abf0 con 0x7f23140d0420 2024-02-19T15:23:35.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.067+0000 7f230be00640 1 --2- 172.21.15.47:0/774771399 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f22f4066e40 0x7f22f4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:35.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.067+0000 7f230be00640 1 -- 172.21.15.47:0/774771399 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f230c084500 con 0x7f23140d0420 2024-02-19T15:23:35.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.067+0000 7f2313e00640 1 --2- 172.21.15.47:0/774771399 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f22f4066e40 0x7f22f4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:35.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.067+0000 7f2313e00640 1 --2- 172.21.15.47:0/774771399 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f22f4066e40 0x7f22f4069300 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f23040079c0 tx=0x7f2304008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:35.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.071+0000 7f230be00640 1 -- 172.21.15.47:0/774771399 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f230c051600 con 0x7f23140d0420 2024-02-19T15:23:35.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.239+0000 7f231b000640 1 -- 172.21.15.47:0/774771399 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23140c7660 con 0x7f23140d0420 2024-02-19T15:23:35.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.239+0000 7f230be00640 1 -- 172.21.15.47:0/774771399 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+13032 (secure 0 0 0) 0x7f230c054cc0 con 0x7f23140d0420 2024-02-19T15:23:35.246 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:35.246 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":37,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:23:33.887515+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:6835/3700339993":"2024-02-20T15:23:30.876972+0000","172.21.15.47:6833/3700339993":"2024-02-20T15:23:30.876972+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:23:35.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.243+0000 7f231b000640 1 -- 172.21.15.47:0/774771399 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f22f4066e40 msgr2=0x7f22f4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:35.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.243+0000 7f231b000640 1 --2- 172.21.15.47:0/774771399 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f22f4066e40 0x7f22f4069300 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f23040079c0 tx=0x7f2304008040 comp rx=0 tx=0).stop 2024-02-19T15:23:35.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.243+0000 7f231b000640 1 -- 172.21.15.47:0/774771399 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23140d0420 msgr2=0x7f23141600a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:35.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.243+0000 7f231b000640 1 --2- 172.21.15.47:0/774771399 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23140d0420 0x7f23141600a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f230c00c770 tx=0x7f230c00cc30 comp rx=0 tx=0).stop 2024-02-19T15:23:35.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.243+0000 7f231b000640 1 -- 172.21.15.47:0/774771399 shutdown_connections 2024-02-19T15:23:35.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.243+0000 7f231b000640 1 --2- 172.21.15.47:0/774771399 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f22f4066e40 0x7f22f4069300 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.243+0000 7f231b000640 1 --2- 172.21.15.47:0/774771399 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23140d0420 0x7f23141600a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.243+0000 7f231b000640 1 --2- 172.21.15.47:0/774771399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23140ce510 0x7f23140c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.243+0000 7f231b000640 1 --2- 172.21.15.47:0/774771399 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f23140b2b70 0x7f2314152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.243+0000 7f231b000640 1 -- 172.21.15.47:0/774771399 >> 172.21.15.47:0/774771399 conn(0x7f23140d1b20 msgr2=0x7f2314157030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:35.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.243+0000 7f231b000640 1 -- 172.21.15.47:0/774771399 shutdown_connections 2024-02-19T15:23:35.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.243+0000 7f231b000640 1 -- 172.21.15.47:0/774771399 wait complete. 2024-02-19T15:23:35.268 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:23:35.268 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:23:35.305 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:23:35.317 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:23:35.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.407+0000 7f3050800640 1 Processor -- start 2024-02-19T15:23:35.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.407+0000 7f3050800640 1 -- start start 2024-02-19T15:23:35.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.407+0000 7f3050800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f304815af10 0x7f304806e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:35.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.407+0000 7f3050800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f30480ca8f0 0x7f30480cad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:35.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.407+0000 7f3050800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3048071c70 con 0x7f30480cb290 2024-02-19T15:23:35.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.407+0000 7f3050800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f304806e8a0 con 0x7f304815af10 2024-02-19T15:23:35.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.407+0000 7f3050800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f304806cef0 con 0x7f30480ca8f0 2024-02-19T15:23:35.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.407+0000 7f304de00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f304815af10 0x7f304806e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:35.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.407+0000 7f304de00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f304815af10 0x7f304806e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52280/0 (socket says 172.21.15.47:52280) 2024-02-19T15:23:35.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.407+0000 7f304de00640 1 -- 172.21.15.47:0/460876553 learned_addr learned my addr 172.21.15.47:0/460876553 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:35.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.407+0000 7f304d400640 1 --2- 172.21.15.47:0/460876553 >> v2:172.21.15.116:3300/0 conn(0x7f30480ca8f0 0x7f30480cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:35.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.407+0000 7f304de00640 1 -- 172.21.15.47:0/460876553 >> v2:172.21.15.116:3300/0 conn(0x7f30480ca8f0 msgr2=0x7f30480cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:35.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.407+0000 7f304de00640 1 --2- 172.21.15.47:0/460876553 >> v2:172.21.15.116:3300/0 conn(0x7f30480ca8f0 0x7f30480cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.407+0000 7f304de00640 1 -- 172.21.15.47:0/460876553 >> v1:172.21.15.116:6789/0 conn(0x7f30480cb290 legacy=0x7f304815eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:35.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.407+0000 7f304de00640 1 -- 172.21.15.47:0/460876553 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30480ca610 con 0x7f304815af10 2024-02-19T15:23:35.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.407+0000 7f304d400640 1 --2- 172.21.15.47:0/460876553 >> v2:172.21.15.116:3300/0 conn(0x7f30480ca8f0 0x7f30480cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:23:35.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.411+0000 7f304de00640 1 --2- 172.21.15.47:0/460876553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f304815af10 0x7f304806e360 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f30480b27b0 tx=0x7f303c01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fc889f9ae94eda6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:35.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.411+0000 7f304ca00640 1 -- 172.21.15.47:0/460876553 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303c0ab620 con 0x7f304815af10 2024-02-19T15:23:35.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.411+0000 7f304ca00640 1 -- 172.21.15.47:0/460876553 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f303c0ab780 con 0x7f304815af10 2024-02-19T15:23:35.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.411+0000 7f304ca00640 1 -- 172.21.15.47:0/460876553 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303c0abb10 con 0x7f304815af10 2024-02-19T15:23:35.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.411+0000 7f3050800640 1 -- 172.21.15.47:0/460876553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f304815af10 msgr2=0x7f304806e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:35.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.411+0000 7f3050800640 1 --2- 172.21.15.47:0/460876553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f304815af10 0x7f304806e360 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f30480b27b0 tx=0x7f303c01c420 comp rx=0 tx=0).stop 2024-02-19T15:23:35.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.411+0000 7f3050800640 1 -- 172.21.15.47:0/460876553 shutdown_connections 2024-02-19T15:23:35.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.411+0000 7f3050800640 1 --2- 172.21.15.47:0/460876553 >> v2:172.21.15.116:3300/0 conn(0x7f30480ca8f0 0x7f30480cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.411+0000 7f3050800640 1 --2- 172.21.15.47:0/460876553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f304815af10 0x7f304806e360 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.411+0000 7f3050800640 1 -- 172.21.15.47:0/460876553 >> 172.21.15.47:0/460876553 conn(0x7f30480cfc20 msgr2=0x7f30480d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:35.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.411+0000 7f3050800640 1 -- 172.21.15.47:0/460876553 shutdown_connections 2024-02-19T15:23:35.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.411+0000 7f3050800640 1 -- 172.21.15.47:0/460876553 wait complete. 2024-02-19T15:23:35.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f3050800640 1 Processor -- start 2024-02-19T15:23:35.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f3050800640 1 -- start start 2024-02-19T15:23:35.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f3050800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30480ca8f0 0x7f3048154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:35.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f3050800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30480cb290 0x7f30480c8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:35.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f3050800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f304815af10 0x7f3048160000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:35.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f3050800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30480c7700 con 0x7f30480ca8f0 2024-02-19T15:23:35.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f3050800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30480c7950 con 0x7f30480cb290 2024-02-19T15:23:35.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f3050800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30480c7c30 con 0x7f304815af10 2024-02-19T15:23:35.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f304de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30480ca8f0 0x7f3048154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:35.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f304de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30480ca8f0 0x7f3048154380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52286/0 (socket says 172.21.15.47:52286) 2024-02-19T15:23:35.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f304de00640 1 -- 172.21.15.47:0/1167532533 learned_addr learned my addr 172.21.15.47:0/1167532533 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:35.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f304d400640 1 --2- 172.21.15.47:0/1167532533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30480cb290 0x7f30480c8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:35.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f304e800640 1 --2- 172.21.15.47:0/1167532533 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f304815af10 0x7f3048160000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:35.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f304de00640 1 -- 172.21.15.47:0/1167532533 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f304815af10 msgr2=0x7f3048160000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:35.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f304de00640 1 --2- 172.21.15.47:0/1167532533 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f304815af10 0x7f3048160000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f304de00640 1 -- 172.21.15.47:0/1167532533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30480cb290 msgr2=0x7f30480c8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:35.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f304de00640 1 --2- 172.21.15.47:0/1167532533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30480cb290 0x7f30480c8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f304de00640 1 -- 172.21.15.47:0/1167532533 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f303c0094d0 con 0x7f30480ca8f0 2024-02-19T15:23:35.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f304e800640 1 --2- 172.21.15.47:0/1167532533 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f304815af10 0x7f3048160000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:23:35.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f304de00640 1 --2- 172.21.15.47:0/1167532533 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30480ca8f0 0x7f3048154380 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f303c01a050 tx=0x7f303c01ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:35.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f3042000640 1 -- 172.21.15.47:0/1167532533 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303c0be3f0 con 0x7f30480ca8f0 2024-02-19T15:23:35.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f3042000640 1 -- 172.21.15.47:0/1167532533 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f303c0be550 con 0x7f30480ca8f0 2024-02-19T15:23:35.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f3042000640 1 -- 172.21.15.47:0/1167532533 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303c0be8e0 con 0x7f30480ca8f0 2024-02-19T15:23:35.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f3050800640 1 -- 172.21.15.47:0/1167532533 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30480c7f10 con 0x7f30480ca8f0 2024-02-19T15:23:35.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f3050800640 1 -- 172.21.15.47:0/1167532533 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30481548c0 con 0x7f30480ca8f0 2024-02-19T15:23:35.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.415+0000 7f3050800640 1 -- 172.21.15.47:0/1167532533 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3048075360 con 0x7f30480ca8f0 2024-02-19T15:23:35.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.419+0000 7f3042000640 1 -- 172.21.15.47:0/1167532533 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f303c0c7440 con 0x7f30480ca8f0 2024-02-19T15:23:35.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.419+0000 7f3042000640 1 --2- 172.21.15.47:0/1167532533 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f302c066e70 0x7f302c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:35.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.419+0000 7f3042000640 1 -- 172.21.15.47:0/1167532533 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f303c11f050 con 0x7f30480ca8f0 2024-02-19T15:23:35.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.419+0000 7f304d400640 1 --2- 172.21.15.47:0/1167532533 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f302c066e70 0x7f302c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:35.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.419+0000 7f304d400640 1 --2- 172.21.15.47:0/1167532533 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f302c066e70 0x7f302c069330 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3038006fd0 tx=0x7f3038008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:35.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.423+0000 7f3042000640 1 -- 172.21.15.47:0/1167532533 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f303c0ec130 con 0x7f30480ca8f0 2024-02-19T15:23:35.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.583+0000 7f3050800640 1 -- 172.21.15.47:0/1167532533 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f30480c7d70 con 0x7f30480ca8f0 2024-02-19T15:23:35.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.587+0000 7f3042000640 1 -- 172.21.15.47:0/1167532533 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7f30480c7d70 con 0x7f30480ca8f0 2024-02-19T15:23:35.591 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:23:35.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.587+0000 7f3050800640 1 -- 172.21.15.47:0/1167532533 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f302c066e70 msgr2=0x7f302c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:35.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.587+0000 7f3050800640 1 --2- 172.21.15.47:0/1167532533 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f302c066e70 0x7f302c069330 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3038006fd0 tx=0x7f3038008040 comp rx=0 tx=0).stop 2024-02-19T15:23:35.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.587+0000 7f3050800640 1 -- 172.21.15.47:0/1167532533 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30480ca8f0 msgr2=0x7f3048154380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:35.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.587+0000 7f3050800640 1 --2- 172.21.15.47:0/1167532533 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30480ca8f0 0x7f3048154380 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f303c01a050 tx=0x7f303c01ac60 comp rx=0 tx=0).stop 2024-02-19T15:23:35.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.587+0000 7f3050800640 1 -- 172.21.15.47:0/1167532533 shutdown_connections 2024-02-19T15:23:35.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.587+0000 7f3050800640 1 --2- 172.21.15.47:0/1167532533 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f302c066e70 0x7f302c069330 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.587+0000 7f3050800640 1 --2- 172.21.15.47:0/1167532533 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f304815af10 0x7f3048160000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.587+0000 7f3050800640 1 --2- 172.21.15.47:0/1167532533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30480cb290 0x7f30480c8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.587+0000 7f3050800640 1 --2- 172.21.15.47:0/1167532533 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30480ca8f0 0x7f3048154380 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.587+0000 7f3050800640 1 -- 172.21.15.47:0/1167532533 >> 172.21.15.47:0/1167532533 conn(0x7f30480cfc20 msgr2=0x7f30480b3480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:35.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.587+0000 7f3050800640 1 -- 172.21.15.47:0/1167532533 shutdown_connections 2024-02-19T15:23:35.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.587+0000 7f3050800640 1 -- 172.21.15.47:0/1167532533 wait complete. 2024-02-19T15:23:35.612 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-02-19T15:23:35.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.747+0000 7f390b600640 1 Processor -- start 2024-02-19T15:23:35.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.747+0000 7f390b600640 1 -- start start 2024-02-19T15:23:35.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.747+0000 7f390b600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f390406d650 0x7f39040d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:35.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.747+0000 7f390b600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39040ced10 0x7f390406bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:35.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.747+0000 7f390b600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39040cce30 con 0x7f39040cded0 2024-02-19T15:23:35.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.747+0000 7f390b600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39040cd110 con 0x7f390406d650 2024-02-19T15:23:35.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.747+0000 7f390b600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39040cd3f0 con 0x7f39040ced10 2024-02-19T15:23:35.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.747+0000 7f3903e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f39040cded0 0x7f39040ce600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52480/0 (socket says 172.21.15.47:52480) 2024-02-19T15:23:35.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.747+0000 7f3903e00640 1 -- 172.21.15.47:0/3486114291 learned_addr learned my addr 172.21.15.47:0/3486114291 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:35.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.747+0000 7f3909600640 1 --2- 172.21.15.47:0/3486114291 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39040ced10 0x7f390406bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:35.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.747+0000 7f3908c00640 1 --2- 172.21.15.47:0/3486114291 >> v2:172.21.15.116:3300/0 conn(0x7f390406d650 0x7f39040d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:35.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.747+0000 7f3903400640 1 -- 172.21.15.47:0/3486114291 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015331280 0 0) 0x7f39040cce30 con 0x7f39040cded0 2024-02-19T15:23:35.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.747+0000 7f3903400640 1 -- 172.21.15.47:0/3486114291 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38ec003530 con 0x7f39040cded0 2024-02-19T15:23:35.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.747+0000 7f3908c00640 1 -- 172.21.15.47:0/3486114291 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39040ced10 msgr2=0x7f390406bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:35.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.747+0000 7f3908c00640 1 --2- 172.21.15.47:0/3486114291 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39040ced10 0x7f390406bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.747+0000 7f3908c00640 1 -- 172.21.15.47:0/3486114291 >> v1:172.21.15.47:6789/0 conn(0x7f39040cded0 legacy=0x7f39040ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:35.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.747+0000 7f3908c00640 1 -- 172.21.15.47:0/3486114291 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39040cd6d0 con 0x7f390406d650 2024-02-19T15:23:35.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.747+0000 7f3909600640 1 --2- 172.21.15.47:0/3486114291 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39040ced10 0x7f390406bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:23:35.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.751+0000 7f3908c00640 1 --2- 172.21.15.47:0/3486114291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f390406d650 0x7f39040d08b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f38f80097c0 tx=0x7f38f801c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c7ce86fe70e4850 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:35.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.751+0000 7f3903400640 1 -- 172.21.15.47:0/3486114291 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f80ab600 con 0x7f390406d650 2024-02-19T15:23:35.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.751+0000 7f3903400640 1 -- 172.21.15.47:0/3486114291 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f80ab760 con 0x7f390406d650 2024-02-19T15:23:35.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.751+0000 7f3903400640 1 -- 172.21.15.47:0/3486114291 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f80abad0 con 0x7f390406d650 2024-02-19T15:23:35.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.751+0000 7f390b600640 1 -- 172.21.15.47:0/3486114291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f390406d650 msgr2=0x7f39040d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:35.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.751+0000 7f390b600640 1 --2- 172.21.15.47:0/3486114291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f390406d650 0x7f39040d08b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f38f80097c0 tx=0x7f38f801c3e0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.751+0000 7f390b600640 1 -- 172.21.15.47:0/3486114291 shutdown_connections 2024-02-19T15:23:35.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.751+0000 7f390b600640 1 --2- 172.21.15.47:0/3486114291 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39040ced10 0x7f390406bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.751+0000 7f390b600640 1 --2- 172.21.15.47:0/3486114291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f390406d650 0x7f39040d08b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.751+0000 7f390b600640 1 -- 172.21.15.47:0/3486114291 >> 172.21.15.47:0/3486114291 conn(0x7f39040d1bf0 msgr2=0x7f39040d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:35.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.751+0000 7f390b600640 1 -- 172.21.15.47:0/3486114291 shutdown_connections 2024-02-19T15:23:35.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.751+0000 7f390b600640 1 -- 172.21.15.47:0/3486114291 wait complete. 2024-02-19T15:23:35.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f390b600640 1 Processor -- start 2024-02-19T15:23:35.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f390b600640 1 -- start start 2024-02-19T15:23:35.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f390b600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f390406d650 0x7f390415d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:35.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f390b600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39040cded0 0x7f39040cb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:35.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f390b600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39040ced10 0x7f39040c6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:35.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f390b600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39040ca5a0 con 0x7f39040ced10 2024-02-19T15:23:35.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f390b600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39040c96d0 con 0x7f390406d650 2024-02-19T15:23:35.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f390b600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39040c9920 con 0x7f39040cded0 2024-02-19T15:23:35.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f3909600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39040ced10 0x7f39040c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:35.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f3908c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f390406d650 0x7f390415d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:35.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f3909600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39040ced10 0x7f39040c6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52290/0 (socket says 172.21.15.47:52290) 2024-02-19T15:23:35.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f3909600640 1 -- 172.21.15.47:0/4269377950 learned_addr learned my addr 172.21.15.47:0/4269377950 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:35.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f3903e00640 1 --2- 172.21.15.47:0/4269377950 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39040cded0 0x7f39040cb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:35.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f3909600640 1 -- 172.21.15.47:0/4269377950 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39040cded0 msgr2=0x7f39040cb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:35.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f3909600640 1 --2- 172.21.15.47:0/4269377950 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39040cded0 0x7f39040cb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f3909600640 1 -- 172.21.15.47:0/4269377950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f390406d650 msgr2=0x7f390415d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:35.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f3909600640 1 --2- 172.21.15.47:0/4269377950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f390406d650 0x7f390415d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:35.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f3909600640 1 -- 172.21.15.47:0/4269377950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38fc009a70 con 0x7f39040ced10 2024-02-19T15:23:35.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f3909600640 1 --2- 172.21.15.47:0/4269377950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39040ced10 0x7f39040c6980 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00efc0 tx=0x7f38fc009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:35.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f3900c00640 1 -- 172.21.15.47:0/4269377950 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc009d70 con 0x7f39040ced10 2024-02-19T15:23:35.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f3900c00640 1 -- 172.21.15.47:0/4269377950 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38fc00f040 con 0x7f39040ced10 2024-02-19T15:23:35.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f3900c00640 1 -- 172.21.15.47:0/4269377950 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc0039a0 con 0x7f39040ced10 2024-02-19T15:23:35.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f390b600640 1 -- 172.21.15.47:0/4269377950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f8009420 con 0x7f39040ced10 2024-02-19T15:23:35.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.755+0000 7f390b600640 1 -- 172.21.15.47:0/4269377950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39040c9ee0 con 0x7f39040ced10 2024-02-19T15:23:35.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.759+0000 7f390b600640 1 -- 172.21.15.47:0/4269377950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39040c66a0 con 0x7f39040ced10 2024-02-19T15:23:35.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.759+0000 7f3900c00640 1 -- 172.21.15.47:0/4269377950 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f38fc0144a0 con 0x7f39040ced10 2024-02-19T15:23:35.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.759+0000 7f3900c00640 1 --2- 172.21.15.47:0/4269377950 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f38e4066e40 0x7f38e4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:35.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.759+0000 7f3900c00640 1 -- 172.21.15.47:0/4269377950 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f38fc0852a0 con 0x7f39040ced10 2024-02-19T15:23:35.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.759+0000 7f3908c00640 1 --2- 172.21.15.47:0/4269377950 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f38e4066e40 0x7f38e4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:35.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.759+0000 7f3908c00640 1 --2- 172.21.15.47:0/4269377950 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f38e4066e40 0x7f38e4069300 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f38f80af000 tx=0x7f38f80a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:35.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.763+0000 7f3900c00640 1 -- 172.21.15.47:0/4269377950 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f39040c66a0 con 0x7f39040ced10 2024-02-19T15:23:35.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:35.983+0000 7f390b600640 1 -- 172.21.15.47:0/4269377950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f39040c66a0 con 0x7f39040ced10 2024-02-19T15:23:36.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:36.895+0000 7f3900c00640 1 -- 172.21.15.47:0/4269377950 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v107) v1 ==== 136+0+0 (secure 0 0 0) 0x7f39040c66a0 con 0x7f39040ced10 2024-02-19T15:23:36.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:36.899+0000 7f390b600640 1 -- 172.21.15.47:0/4269377950 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f38e4066e40 msgr2=0x7f38e4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:36.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:36.899+0000 7f390b600640 1 --2- 172.21.15.47:0/4269377950 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f38e4066e40 0x7f38e4069300 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f38f80af000 tx=0x7f38f80a7000 comp rx=0 tx=0).stop 2024-02-19T15:23:36.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:36.899+0000 7f390b600640 1 -- 172.21.15.47:0/4269377950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39040ced10 msgr2=0x7f39040c6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:36.899+0000 7f390b600640 1 --2- 172.21.15.47:0/4269377950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39040ced10 0x7f39040c6980 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00efc0 tx=0x7f38fc009f90 comp rx=0 tx=0).stop 2024-02-19T15:23:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:36.899+0000 7f390b600640 1 -- 172.21.15.47:0/4269377950 shutdown_connections 2024-02-19T15:23:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:36.899+0000 7f390b600640 1 --2- 172.21.15.47:0/4269377950 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f38e4066e40 0x7f38e4069300 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:36.899+0000 7f390b600640 1 --2- 172.21.15.47:0/4269377950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39040ced10 0x7f39040c6980 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:36.899+0000 7f390b600640 1 --2- 172.21.15.47:0/4269377950 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39040cded0 0x7f39040cb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:36.899+0000 7f390b600640 1 --2- 172.21.15.47:0/4269377950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f390406d650 0x7f390415d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:36.899+0000 7f390b600640 1 -- 172.21.15.47:0/4269377950 >> 172.21.15.47:0/4269377950 conn(0x7f39040d1bf0 msgr2=0x7f39040d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:36.899+0000 7f390b600640 1 -- 172.21.15.47:0/4269377950 shutdown_connections 2024-02-19T15:23:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:36.899+0000 7f390b600640 1 -- 172.21.15.47:0/4269377950 wait complete. 2024-02-19T15:23:36.927 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-19T15:23:37.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.067+0000 7fa66de00640 1 Processor -- start 2024-02-19T15:23:37.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.067+0000 7fa66de00640 1 -- start start 2024-02-19T15:23:37.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.071+0000 7fa66de00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa6680cb000 0x7fa6680cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:37.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.071+0000 7fa66de00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa668071c70 con 0x7fa66806e360 2024-02-19T15:23:37.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.071+0000 7fa66de00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66806cef0 con 0x7fa66815af10 2024-02-19T15:23:37.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.071+0000 7fa66de00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6680ca610 con 0x7fa6680cb000 2024-02-19T15:23:37.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.071+0000 7fa65fe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa66806e360 0x7fa6680cd880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52492/0 (socket says 172.21.15.47:52492) 2024-02-19T15:23:37.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.071+0000 7fa65fe00640 1 -- 172.21.15.47:0/1009485951 learned_addr learned my addr 172.21.15.47:0/1009485951 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:37.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.071+0000 7fa667400640 1 --2- 172.21.15.47:0/1009485951 >> v2:172.21.15.116:3300/0 conn(0x7fa6680cb000 0x7fa6680cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:37.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.071+0000 7fa666000640 1 -- 172.21.15.47:0/1009485951 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463728670 0 0) 0x7fa668071c70 con 0x7fa66806e360 2024-02-19T15:23:37.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.071+0000 7fa666000640 1 -- 172.21.15.47:0/1009485951 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa64c003530 con 0x7fa66806e360 2024-02-19T15:23:37.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.071+0000 7fa667400640 1 -- 172.21.15.47:0/1009485951 >> v1:172.21.15.116:6789/0 conn(0x7fa66815af10 legacy=0x7fa6680ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:37.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.071+0000 7fa666000640 1 -- 172.21.15.47:0/1009485951 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976599463 0 0) 0x7fa66806cef0 con 0x7fa66815af10 2024-02-19T15:23:37.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.071+0000 7fa667400640 1 -- 172.21.15.47:0/1009485951 >> v1:172.21.15.47:6789/0 conn(0x7fa66806e360 legacy=0x7fa6680cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:37.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.071+0000 7fa667400640 1 -- 172.21.15.47:0/1009485951 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa668071ee0 con 0x7fa6680cb000 2024-02-19T15:23:37.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.071+0000 7fa667400640 1 --2- 172.21.15.47:0/1009485951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6680cb000 0x7fa6680cdf90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa660009d90 tx=0x7fa66001c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0536fce0a576d31 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:37.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa666000640 1 -- 172.21.15.47:0/1009485951 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6600a58d0 con 0x7fa6680cb000 2024-02-19T15:23:37.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa666000640 1 -- 172.21.15.47:0/1009485951 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6600a5a30 con 0x7fa6680cb000 2024-02-19T15:23:37.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa666000640 1 -- 172.21.15.47:0/1009485951 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6600a5dc0 con 0x7fa6680cb000 2024-02-19T15:23:37.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa66de00640 1 -- 172.21.15.47:0/1009485951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6680cb000 msgr2=0x7fa6680cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:37.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa66de00640 1 --2- 172.21.15.47:0/1009485951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6680cb000 0x7fa6680cdf90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa660009d90 tx=0x7fa66001c880 comp rx=0 tx=0).stop 2024-02-19T15:23:37.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa66de00640 1 -- 172.21.15.47:0/1009485951 shutdown_connections 2024-02-19T15:23:37.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa66de00640 1 --2- 172.21.15.47:0/1009485951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6680cb000 0x7fa6680cdf90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:37.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa66de00640 1 -- 172.21.15.47:0/1009485951 >> 172.21.15.47:0/1009485951 conn(0x7fa6680cfc20 msgr2=0x7fa6680d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:37.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa66de00640 1 -- 172.21.15.47:0/1009485951 shutdown_connections 2024-02-19T15:23:37.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa66de00640 1 -- 172.21.15.47:0/1009485951 wait complete. 2024-02-19T15:23:37.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa66de00640 1 Processor -- start 2024-02-19T15:23:37.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa66de00640 1 -- start start 2024-02-19T15:23:37.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa66de00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa66806e360 0x7fa6680c4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:37.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa66de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6680cb000 0x7fa6681f8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:37.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa66de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa66815af10 0x7fa668159380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:37.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa66de00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66815c870 con 0x7fa6680cb000 2024-02-19T15:23:37.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa66de00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66815cae0 con 0x7fa66815af10 2024-02-19T15:23:37.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa66de00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa66815cdc0 con 0x7fa66806e360 2024-02-19T15:23:37.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa65fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6680cb000 0x7fa6681f8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:37.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa65fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6680cb000 0x7fa6681f8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52296/0 (socket says 172.21.15.47:52296) 2024-02-19T15:23:37.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa65fe00640 1 -- 172.21.15.47:0/3700291694 learned_addr learned my addr 172.21.15.47:0/3700291694 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:37.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa667400640 1 --2- 172.21.15.47:0/3700291694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa66815af10 0x7fa668159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:37.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.075+0000 7fa666a00640 1 --2- 172.21.15.47:0/3700291694 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa66806e360 0x7fa6680c4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:37.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.079+0000 7fa65fe00640 1 -- 172.21.15.47:0/3700291694 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa66806e360 msgr2=0x7fa6680c4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:37.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.079+0000 7fa65fe00640 1 --2- 172.21.15.47:0/3700291694 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa66806e360 0x7fa6680c4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:37.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.079+0000 7fa65fe00640 1 -- 172.21.15.47:0/3700291694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa66815af10 msgr2=0x7fa668159380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:37.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.079+0000 7fa65fe00640 1 --2- 172.21.15.47:0/3700291694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa66815af10 0x7fa668159380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:37.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.079+0000 7fa65fe00640 1 -- 172.21.15.47:0/3700291694 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa660009a70 con 0x7fa6680cb000 2024-02-19T15:23:37.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.079+0000 7fa65fe00640 1 --2- 172.21.15.47:0/3700291694 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6680cb000 0x7fa6681f8f70 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa65000c450 tx=0x7fa65000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:37.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.079+0000 7fa65f400640 1 -- 172.21.15.47:0/3700291694 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa650005cf0 con 0x7fa6680cb000 2024-02-19T15:23:37.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.079+0000 7fa65f400640 1 -- 172.21.15.47:0/3700291694 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa650005e50 con 0x7fa6680cb000 2024-02-19T15:23:37.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.079+0000 7fa65f400640 1 -- 172.21.15.47:0/3700291694 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa650007290 con 0x7fa6680cb000 2024-02-19T15:23:37.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.079+0000 7fa66de00640 1 -- 172.21.15.47:0/3700291694 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa66815d0a0 con 0x7fa6680cb000 2024-02-19T15:23:37.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.079+0000 7fa66de00640 1 -- 172.21.15.47:0/3700291694 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6680c5500 con 0x7fa6680cb000 2024-02-19T15:23:37.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.079+0000 7fa66de00640 1 -- 172.21.15.47:0/3700291694 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa668075360 con 0x7fa6680cb000 2024-02-19T15:23:37.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.079+0000 7fa65f400640 1 -- 172.21.15.47:0/3700291694 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa6500073f0 con 0x7fa6680cb000 2024-02-19T15:23:37.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.079+0000 7fa65f400640 1 --2- 172.21.15.47:0/3700291694 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa644066e40 0x7fa644069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:37.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.079+0000 7fa65f400640 1 -- 172.21.15.47:0/3700291694 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa650083c50 con 0x7fa6680cb000 2024-02-19T15:23:37.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.083+0000 7fa666a00640 1 --2- 172.21.15.47:0/3700291694 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa644066e40 0x7fa644069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:37.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.083+0000 7fa666a00640 1 --2- 172.21.15.47:0/3700291694 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa644066e40 0x7fa644069300 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa658007fd0 tx=0x7fa658009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:37.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.083+0000 7fa65f400640 1 -- 172.21.15.47:0/3700291694 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa650050de0 con 0x7fa6680cb000 2024-02-19T15:23:37.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.247+0000 7fa66de00640 1 -- 172.21.15.47:0/3700291694 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa66815d550 con 0x7fa6680cb000 2024-02-19T15:23:37.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.251+0000 7fa65f400640 1 -- 172.21.15.47:0/3700291694 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa650017070 con 0x7fa6680cb000 2024-02-19T15:23:37.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.251+0000 7fa66de00640 1 -- 172.21.15.47:0/3700291694 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa644066e40 msgr2=0x7fa644069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:37.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.251+0000 7fa66de00640 1 --2- 172.21.15.47:0/3700291694 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa644066e40 0x7fa644069300 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa658007fd0 tx=0x7fa658009040 comp rx=0 tx=0).stop 2024-02-19T15:23:37.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.251+0000 7fa66de00640 1 -- 172.21.15.47:0/3700291694 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6680cb000 msgr2=0x7fa6681f8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:37.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.251+0000 7fa66de00640 1 --2- 172.21.15.47:0/3700291694 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6680cb000 0x7fa6681f8f70 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa65000c450 tx=0x7fa65000c910 comp rx=0 tx=0).stop 2024-02-19T15:23:37.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.251+0000 7fa66de00640 1 -- 172.21.15.47:0/3700291694 shutdown_connections 2024-02-19T15:23:37.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.251+0000 7fa66de00640 1 --2- 172.21.15.47:0/3700291694 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa644066e40 0x7fa644069300 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:37.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.251+0000 7fa66de00640 1 --2- 172.21.15.47:0/3700291694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa66815af10 0x7fa668159380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:37.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.251+0000 7fa66de00640 1 --2- 172.21.15.47:0/3700291694 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6680cb000 0x7fa6681f8f70 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:37.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.251+0000 7fa66de00640 1 --2- 172.21.15.47:0/3700291694 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa66806e360 0x7fa6680c4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:37.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.251+0000 7fa66de00640 1 -- 172.21.15.47:0/3700291694 >> 172.21.15.47:0/3700291694 conn(0x7fa6680cfc20 msgr2=0x7fa668071080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:37.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.251+0000 7fa66de00640 1 -- 172.21.15.47:0/3700291694 shutdown_connections 2024-02-19T15:23:37.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.251+0000 7fa66de00640 1 -- 172.21.15.47:0/3700291694 wait complete. 2024-02-19T15:23:37.279 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T15:23:37.279 DEBUG:teuthology.orchestra.run.smithi047:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:23:37.369 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:23:37.445 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T15:23:37.445 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:23:37.445 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:23:37.454 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:23:37.454 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:23:37.454 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:23:37.463 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:23:37.463 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:23:37.463 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:23:37.502 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:23:37.502 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:23:37.502 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:23:37.514 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:23:37.514 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-02-19T15:23:37.514 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:23:37.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.679+0000 7fd4eda00640 1 Processor -- start 2024-02-19T15:23:37.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.679+0000 7fd4eda00640 1 -- start start 2024-02-19T15:23:37.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.679+0000 7fd4eda00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd4e8145590 0x7fd4e80727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:37.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.679+0000 7fd4eda00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e815d280 con 0x7fd4e80cea80 2024-02-19T15:23:37.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.679+0000 7fd4eda00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e8070ff0 con 0x7fd4e815c390 2024-02-19T15:23:37.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.679+0000 7fd4eda00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e80709e0 con 0x7fd4e8145590 2024-02-19T15:23:37.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.679+0000 7fd4e6a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd4e8145590 0x7fd4e80727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:37.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.679+0000 7fd4e6a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd4e8145590 0x7fd4e80727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52300/0 (socket says 172.21.15.47:52300) 2024-02-19T15:23:37.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.679+0000 7fd4e6a00640 1 -- 172.21.15.47:0/2603839378 learned_addr learned my addr 172.21.15.47:0/2603839378 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:37.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.683+0000 7fd4e6a00640 1 -- 172.21.15.47:0/2603839378 >> v1:172.21.15.116:6789/0 conn(0x7fd4e815c390 legacy=0x7fd4e815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:37.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.683+0000 7fd4e6a00640 1 -- 172.21.15.47:0/2603839378 >> v1:172.21.15.47:6789/0 conn(0x7fd4e80cea80 legacy=0x7fd4e806c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:37.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.683+0000 7fd4e6a00640 1 -- 172.21.15.47:0/2603839378 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e80cf020 con 0x7fd4e8145590 2024-02-19T15:23:37.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.683+0000 7fd4e6a00640 1 --2- 172.21.15.47:0/2603839378 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e8145590 0x7fd4e80727b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8009870 tx=0x7fd4d801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74a06f084f3331d5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:37.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.683+0000 7fd4e6000640 1 -- 172.21.15.47:0/2603839378 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d80a7070 con 0x7fd4e8145590 2024-02-19T15:23:37.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.683+0000 7fd4e6000640 1 -- 172.21.15.47:0/2603839378 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d80ab430 con 0x7fd4e8145590 2024-02-19T15:23:37.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.683+0000 7fd4e6000640 1 -- 172.21.15.47:0/2603839378 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d80ab7c0 con 0x7fd4e8145590 2024-02-19T15:23:37.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.683+0000 7fd4eda00640 1 -- 172.21.15.47:0/2603839378 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e8145590 msgr2=0x7fd4e80727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:37.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.683+0000 7fd4eda00640 1 --2- 172.21.15.47:0/2603839378 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e8145590 0x7fd4e80727b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8009870 tx=0x7fd4d801c520 comp rx=0 tx=0).stop 2024-02-19T15:23:37.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4eda00640 1 -- 172.21.15.47:0/2603839378 shutdown_connections 2024-02-19T15:23:37.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4eda00640 1 --2- 172.21.15.47:0/2603839378 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e8145590 0x7fd4e80727b0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:37.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4eda00640 1 -- 172.21.15.47:0/2603839378 >> 172.21.15.47:0/2603839378 conn(0x7fd4e80b16a0 msgr2=0x7fd4e80b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:37.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4eda00640 1 -- 172.21.15.47:0/2603839378 shutdown_connections 2024-02-19T15:23:37.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4eda00640 1 -- 172.21.15.47:0/2603839378 wait complete. 2024-02-19T15:23:37.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4eda00640 1 Processor -- start 2024-02-19T15:23:37.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4eda00640 1 -- start start 2024-02-19T15:23:37.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4eda00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4e80cea80 0x7fd4e8143770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:37.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4eda00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4e8145590 0x7fd4e815b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:37.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4eda00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e815c390 0x7fd4e80c6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:37.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4eda00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e80c7780 con 0x7fd4e815c390 2024-02-19T15:23:37.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4eda00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e8144440 con 0x7fd4e80cea80 2024-02-19T15:23:37.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4eda00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e8144160 con 0x7fd4e8145590 2024-02-19T15:23:37.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4e7400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e815c390 0x7fd4e80c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:37.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4e7400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e815c390 0x7fd4e80c6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52304/0 (socket says 172.21.15.47:52304) 2024-02-19T15:23:37.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4e7400640 1 -- 172.21.15.47:0/4012718495 learned_addr learned my addr 172.21.15.47:0/4012718495 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:37.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4e6a00640 1 --2- 172.21.15.47:0/4012718495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4e80cea80 0x7fd4e8143770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:37.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4de000640 1 --2- 172.21.15.47:0/4012718495 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4e8145590 0x7fd4e815b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:37.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4e7400640 1 -- 172.21.15.47:0/4012718495 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4e8145590 msgr2=0x7fd4e815b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:37.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4e7400640 1 --2- 172.21.15.47:0/4012718495 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4e8145590 0x7fd4e815b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:37.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4e7400640 1 -- 172.21.15.47:0/4012718495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4e80cea80 msgr2=0x7fd4e8143770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:37.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4e7400640 1 --2- 172.21.15.47:0/4012718495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4e80cea80 0x7fd4e8143770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:37.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4e7400640 1 -- 172.21.15.47:0/4012718495 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d80094d0 con 0x7fd4e815c390 2024-02-19T15:23:37.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4e7400640 1 --2- 172.21.15.47:0/4012718495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e815c390 0x7fd4e80c6950 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd4e000b9c0 tx=0x7fd4e000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:37.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4df400640 1 -- 172.21.15.47:0/4012718495 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e000cba0 con 0x7fd4e815c390 2024-02-19T15:23:37.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4df400640 1 -- 172.21.15.47:0/4012718495 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e000cd00 con 0x7fd4e815c390 2024-02-19T15:23:37.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4df400640 1 -- 172.21.15.47:0/4012718495 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e0014680 con 0x7fd4e815c390 2024-02-19T15:23:37.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.687+0000 7fd4eda00640 1 -- 172.21.15.47:0/4012718495 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e8142c90 con 0x7fd4e815c390 2024-02-19T15:23:37.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.691+0000 7fd4eda00640 1 -- 172.21.15.47:0/4012718495 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e8143250 con 0x7fd4e815c390 2024-02-19T15:23:37.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.691+0000 7fd4eda00640 1 -- 172.21.15.47:0/4012718495 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e8075360 con 0x7fd4e815c390 2024-02-19T15:23:37.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.691+0000 7fd4df400640 1 -- 172.21.15.47:0/4012718495 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd4e001d3b0 con 0x7fd4e815c390 2024-02-19T15:23:37.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.691+0000 7fd4df400640 1 --2- 172.21.15.47:0/4012718495 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd4c4066e40 0x7fd4c4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:37.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.691+0000 7fd4df400640 1 -- 172.21.15.47:0/4012718495 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd4e00846d0 con 0x7fd4e815c390 2024-02-19T15:23:37.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.691+0000 7fd4e6a00640 1 --2- 172.21.15.47:0/4012718495 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd4c4066e40 0x7fd4c4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:37.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.691+0000 7fd4e6a00640 1 --2- 172.21.15.47:0/4012718495 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd4c4066e40 0x7fd4c4069300 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd4d80af000 tx=0x7fd4d8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:37.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.695+0000 7fd4df400640 1 -- 172.21.15.47:0/4012718495 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd4e00517d0 con 0x7fd4e815c390 2024-02-19T15:23:37.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.891+0000 7fd4eda00640 1 -- 172.21.15.47:0/4012718495 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4e806ca40 con 0x7fd4e815c390 2024-02-19T15:23:37.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.891+0000 7fd4df400640 1 -- 172.21.15.47:0/4012718495 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+3699 (secure 0 0 0) 0x7fd4e0019090 con 0x7fd4e815c390 2024-02-19T15:23:37.898 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:37.898 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":18,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1616905430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1616905430},{"type":"v1","addr":"172.21.15.116:6836","nonce":1616905430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4814,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2845273225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2845273225},{"type":"v1","addr":"172.21.15.116:6837","nonce":2845273225}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5064,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/20043732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":20043732},{"type":"v1","addr":"172.21.15.47:6834","nonce":20043732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5433,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/1729126100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":1729126100},{"type":"v1","addr":"172.21.15.47:6835","nonce":1729126100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[]} 2024-02-19T15:23:37.899 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 18 2024-02-19T15:23:37.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.895+0000 7fd4eda00640 1 -- 172.21.15.47:0/4012718495 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd4c4066e40 msgr2=0x7fd4c4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:37.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.895+0000 7fd4eda00640 1 --2- 172.21.15.47:0/4012718495 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd4c4066e40 0x7fd4c4069300 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd4d80af000 tx=0x7fd4d8008040 comp rx=0 tx=0).stop 2024-02-19T15:23:37.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.895+0000 7fd4eda00640 1 -- 172.21.15.47:0/4012718495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e815c390 msgr2=0x7fd4e80c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:37.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.895+0000 7fd4eda00640 1 --2- 172.21.15.47:0/4012718495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e815c390 0x7fd4e80c6950 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd4e000b9c0 tx=0x7fd4e000be80 comp rx=0 tx=0).stop 2024-02-19T15:23:37.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.895+0000 7fd4eda00640 1 -- 172.21.15.47:0/4012718495 shutdown_connections 2024-02-19T15:23:37.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.895+0000 7fd4eda00640 1 --2- 172.21.15.47:0/4012718495 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd4c4066e40 0x7fd4c4069300 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:37.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.895+0000 7fd4eda00640 1 --2- 172.21.15.47:0/4012718495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e815c390 0x7fd4e80c6950 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:37.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.895+0000 7fd4eda00640 1 --2- 172.21.15.47:0/4012718495 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4e8145590 0x7fd4e815b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:37.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.895+0000 7fd4eda00640 1 --2- 172.21.15.47:0/4012718495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4e80cea80 0x7fd4e8143770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:37.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.895+0000 7fd4eda00640 1 -- 172.21.15.47:0/4012718495 >> 172.21.15.47:0/4012718495 conn(0x7fd4e80b16a0 msgr2=0x7fd4e80aeb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:37.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.895+0000 7fd4eda00640 1 -- 172.21.15.47:0/4012718495 shutdown_connections 2024-02-19T15:23:37.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:37.895+0000 7fd4eda00640 1 -- 172.21.15.47:0/4012718495 wait complete. 2024-02-19T15:23:37.921 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T15:23:37.921 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-19T15:23:37.921 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T15:23:37.921 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:23:37.922 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T15:23:37.922 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-19T15:23:37.922 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T15:23:37.922 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:23:37.922 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T15:23:37.922 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-19T15:23:37.922 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T15:23:37.923 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:23:37.923 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T15:23:37.923 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-19T15:23:37.923 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T15:23:37.923 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:23:37.933 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T15:23:37.934 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T15:23:37.935 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T15:23:37.935 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T15:23:37.938 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T15:23:37.939 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T15:23:37.939 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T15:23:37.939 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T15:23:37.981 INFO:tasks.ceph.mds.a:Started 2024-02-19T15:23:37.981 DEBUG:teuthology.parallel:result is None 2024-02-19T15:23:37.983 INFO:tasks.ceph.mds.c:Started 2024-02-19T15:23:37.983 DEBUG:teuthology.parallel:result is None 2024-02-19T15:23:37.989 INFO:tasks.ceph.mds.b:Started 2024-02-19T15:23:37.989 DEBUG:teuthology.parallel:result is None 2024-02-19T15:23:37.991 INFO:tasks.ceph.mds.d:Started 2024-02-19T15:23:37.991 DEBUG:teuthology.parallel:result is None 2024-02-19T15:23:37.991 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-19T15:23:38.069 INFO:tasks.ceph.mds.c.smithi047.stdout:starting mds.c at 2024-02-19T15:23:38.069 INFO:tasks.ceph.mds.a.smithi047.stdout:starting mds.a at 2024-02-19T15:23:38.079 INFO:tasks.ceph.mds.b.smithi116.stdout:starting mds.b at 2024-02-19T15:23:38.080 INFO:tasks.ceph.mds.d.smithi116.stdout:starting mds.d at 2024-02-19T15:23:38.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.131+0000 7f14b4600640 1 Processor -- start 2024-02-19T15:23:38.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.135+0000 7f14b4600640 1 -- start start 2024-02-19T15:23:38.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.135+0000 7f14b4600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14ac15d6e0 0x7f14ac162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.135+0000 7f14b4600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f14ac071350 0x7f14ac164c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.135+0000 7f14b4600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ac0718c0 con 0x7f14ac15b1a0 2024-02-19T15:23:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.135+0000 7f14b4600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14ac071ba0 con 0x7f14ac071350 2024-02-19T15:23:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.135+0000 7f14b4600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac071e80 con 0x7f14ac15d6e0 2024-02-19T15:23:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.135+0000 7f14b2600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f14ac071350 0x7f14ac164c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.135+0000 7f14b2600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f14ac071350 0x7f14ac164c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52344/0 (socket says 172.21.15.47:52344) 2024-02-19T15:23:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.135+0000 7f14b2600640 1 -- 172.21.15.47:0/2437254182 learned_addr learned my addr 172.21.15.47:0/2437254182 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.135+0000 7f14b1c00640 1 --1- 172.21.15.47:0/2437254182 >> v1:172.21.15.116:6789/0 conn(0x7f14ac15b1a0 0x7f14ac15b580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:52644/0 (socket says 172.21.15.47:52644) 2024-02-19T15:23:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.135+0000 7f14b1200640 1 --2- 172.21.15.47:0/2437254182 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14ac15d6e0 0x7f14ac162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.135+0000 7f14b2600640 1 -- 172.21.15.47:0/2437254182 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14ac15d6e0 msgr2=0x7f14ac162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.135+0000 7f14b2600640 1 --2- 172.21.15.47:0/2437254182 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14ac15d6e0 0x7f14ac162740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.135+0000 7f14b2600640 1 -- 172.21.15.47:0/2437254182 >> v1:172.21.15.116:6789/0 conn(0x7f14ac15b1a0 legacy=0x7f14ac15b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.135+0000 7f14b2600640 1 -- 172.21.15.47:0/2437254182 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ac072160 con 0x7f14ac071350 2024-02-19T15:23:38.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.135+0000 7f14b1200640 1 --2- 172.21.15.47:0/2437254182 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14ac15d6e0 0x7f14ac162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:38.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.139+0000 7f14b2600640 1 --2- 172.21.15.47:0/2437254182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14ac071350 0x7f14ac164c90 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009e50 tx=0x7f14a401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd0154f0f0e9929 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:38.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.139+0000 7f14abe00640 1 -- 172.21.15.47:0/2437254182 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a40a5920 con 0x7f14ac071350 2024-02-19T15:23:38.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.139+0000 7f14abe00640 1 -- 172.21.15.47:0/2437254182 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a40a5a80 con 0x7f14ac071350 2024-02-19T15:23:38.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.139+0000 7f14abe00640 1 -- 172.21.15.47:0/2437254182 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a40a5e10 con 0x7f14ac071350 2024-02-19T15:23:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.139+0000 7f14b4600640 1 -- 172.21.15.47:0/2437254182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14ac071350 msgr2=0x7f14ac164c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.139+0000 7f14b4600640 1 --2- 172.21.15.47:0/2437254182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14ac071350 0x7f14ac164c90 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009e50 tx=0x7f14a401ca30 comp rx=0 tx=0).stop 2024-02-19T15:23:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.139+0000 7f14b4600640 1 -- 172.21.15.47:0/2437254182 shutdown_connections 2024-02-19T15:23:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.139+0000 7f14b4600640 1 --2- 172.21.15.47:0/2437254182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14ac071350 0x7f14ac164c90 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.139+0000 7f14b4600640 1 --2- 172.21.15.47:0/2437254182 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14ac15d6e0 0x7f14ac162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.139+0000 7f14b4600640 1 -- 172.21.15.47:0/2437254182 >> 172.21.15.47:0/2437254182 conn(0x7f14ac0b16a0 msgr2=0x7f14ac0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.139+0000 7f14b4600640 1 -- 172.21.15.47:0/2437254182 shutdown_connections 2024-02-19T15:23:38.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.139+0000 7f14b4600640 1 -- 172.21.15.47:0/2437254182 wait complete. 2024-02-19T15:23:38.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.139+0000 7f14b4600640 1 Processor -- start 2024-02-19T15:23:38.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.139+0000 7f14b4600640 1 -- start start 2024-02-19T15:23:38.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14b4600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14ac071350 0x7f14ac15ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:38.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14b4600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14ac15b1a0 0x7f14ac1553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:38.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14b4600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14ac15d6e0 0x7f14ac155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:38.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14b4600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac1f72e0 con 0x7f14ac071350 2024-02-19T15:23:38.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14b4600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac1f75c0 con 0x7f14ac15d6e0 2024-02-19T15:23:38.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14b4600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac1f78a0 con 0x7f14ac15b1a0 2024-02-19T15:23:38.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14b1c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14ac071350 0x7f14ac15ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:38.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14b1c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14ac071350 0x7f14ac15ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52358/0 (socket says 172.21.15.47:52358) 2024-02-19T15:23:38.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14b1c00640 1 -- 172.21.15.47:0/2054400903 learned_addr learned my addr 172.21.15.47:0/2054400903 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:38.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14b1200640 1 --2- 172.21.15.47:0/2054400903 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14ac15b1a0 0x7f14ac1553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:38.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14b2600640 1 --2- 172.21.15.47:0/2054400903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14ac15d6e0 0x7f14ac155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:38.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14b1c00640 1 -- 172.21.15.47:0/2054400903 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14ac15b1a0 msgr2=0x7f14ac1553c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:38.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14b1c00640 1 --2- 172.21.15.47:0/2054400903 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14ac15b1a0 0x7f14ac1553c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:38.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14b1c00640 1 -- 172.21.15.47:0/2054400903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14ac15d6e0 msgr2=0x7f14ac155c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:38.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14b1c00640 1 --2- 172.21.15.47:0/2054400903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14ac15d6e0 0x7f14ac155c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:38.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14b1c00640 1 -- 172.21.15.47:0/2054400903 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a4009b30 con 0x7f14ac071350 2024-02-19T15:23:38.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14b1c00640 1 --2- 172.21.15.47:0/2054400903 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14ac071350 0x7f14ac15ec40 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f14a000c490 tx=0x7f14a000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:38.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14a9600640 1 -- 172.21.15.47:0/2054400903 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a0007480 con 0x7f14ac071350 2024-02-19T15:23:38.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14a9600640 1 -- 172.21.15.47:0/2054400903 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a00075e0 con 0x7f14ac071350 2024-02-19T15:23:38.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14a9600640 1 -- 172.21.15.47:0/2054400903 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a0004a10 con 0x7f14ac071350 2024-02-19T15:23:38.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14b4600640 1 -- 172.21.15.47:0/2054400903 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ac1f7b80 con 0x7f14ac071350 2024-02-19T15:23:38.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14b4600640 1 -- 172.21.15.47:0/2054400903 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ac15f180 con 0x7f14ac071350 2024-02-19T15:23:38.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14b4600640 1 -- 172.21.15.47:0/2054400903 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14ac1fa580 con 0x7f14ac071350 2024-02-19T15:23:38.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.143+0000 7f14a9600640 1 -- 172.21.15.47:0/2054400903 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f14a0004b70 con 0x7f14ac071350 2024-02-19T15:23:38.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.147+0000 7f14a9600640 1 --2- 172.21.15.47:0/2054400903 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f148c066e40 0x7f148c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:38.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.147+0000 7f14a9600640 1 -- 172.21.15.47:0/2054400903 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f14a0083ae0 con 0x7f14ac071350 2024-02-19T15:23:38.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.147+0000 7f14b1200640 1 --2- 172.21.15.47:0/2054400903 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f148c066e40 0x7f148c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:38.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.147+0000 7f14b1200640 1 --2- 172.21.15.47:0/2054400903 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f148c066e40 0x7f148c069300 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f14ac0cf890 tx=0x7f1498008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:38.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.151+0000 7f14a9600640 1 -- 172.21.15.47:0/2054400903 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f14a0050c60 con 0x7f14ac071350 2024-02-19T15:23:38.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.327+0000 7f14b4600640 1 -- 172.21.15.47:0/2054400903 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f14ac075360 con 0x7f14ac071350 2024-02-19T15:23:38.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.915+0000 7f14a9600640 1 -- 172.21.15.47:0/2054400903 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v39) v1 ==== 97+0+0 (secure 0 0 0) 0x7f14a0018080 con 0x7f14ac071350 2024-02-19T15:23:38.920 INFO:teuthology.orchestra.run.smithi047.stderr: removed all blocklist entries 2024-02-19T15:23:38.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.919+0000 7f14b4600640 1 -- 172.21.15.47:0/2054400903 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f148c066e40 msgr2=0x7f148c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:38.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.919+0000 7f14b4600640 1 --2- 172.21.15.47:0/2054400903 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f148c066e40 0x7f148c069300 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f14ac0cf890 tx=0x7f1498008040 comp rx=0 tx=0).stop 2024-02-19T15:23:38.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.919+0000 7f14b4600640 1 -- 172.21.15.47:0/2054400903 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14ac071350 msgr2=0x7f14ac15ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:38.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.919+0000 7f14b4600640 1 --2- 172.21.15.47:0/2054400903 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14ac071350 0x7f14ac15ec40 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f14a000c490 tx=0x7f14a000c950 comp rx=0 tx=0).stop 2024-02-19T15:23:38.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.919+0000 7f14b4600640 1 -- 172.21.15.47:0/2054400903 shutdown_connections 2024-02-19T15:23:38.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.919+0000 7f14b4600640 1 --2- 172.21.15.47:0/2054400903 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f148c066e40 0x7f148c069300 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:38.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.919+0000 7f14b4600640 1 --2- 172.21.15.47:0/2054400903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14ac15d6e0 0x7f14ac155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:38.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.919+0000 7f14b4600640 1 --2- 172.21.15.47:0/2054400903 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14ac15b1a0 0x7f14ac1553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:38.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.919+0000 7f14b4600640 1 --2- 172.21.15.47:0/2054400903 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14ac071350 0x7f14ac15ec40 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:38.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.919+0000 7f14b4600640 1 -- 172.21.15.47:0/2054400903 >> 172.21.15.47:0/2054400903 conn(0x7f14ac0b16a0 msgr2=0x7f14ac159830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:38.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.919+0000 7f14b4600640 1 -- 172.21.15.47:0/2054400903 shutdown_connections 2024-02-19T15:23:38.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:38.919+0000 7f14b4600640 1 -- 172.21.15.47:0/2054400903 wait complete. 2024-02-19T15:23:38.950 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-19T15:23:39.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.087+0000 7f0aa2600640 1 Processor -- start 2024-02-19T15:23:39.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.087+0000 7f0aa2600640 1 -- start start 2024-02-19T15:23:39.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0aa2600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0a9c15b1a0 0x7f0a9c15b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:39.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0aa2600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9c0718c0 con 0x7f0a9c071350 2024-02-19T15:23:39.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0aa2600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9c071ba0 con 0x7f0a9c06c9a0 2024-02-19T15:23:39.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0aa2600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a9c071e80 con 0x7f0a9c15b1a0 2024-02-19T15:23:39.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a93e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0a9c071350 0x7f0a9c15c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52530/0 (socket says 172.21.15.47:52530) 2024-02-19T15:23:39.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a93e00640 1 -- 172.21.15.47:0/1512211892 learned_addr learned my addr 172.21.15.47:0/1512211892 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:39.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9b400640 1 --2- 172.21.15.47:0/1512211892 >> v2:172.21.15.116:3300/0 conn(0x7f0a9c15b1a0 0x7f0a9c15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:39.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9aa00640 1 -- 172.21.15.47:0/1512211892 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753243894 0 0) 0x7f0a9c0718c0 con 0x7f0a9c071350 2024-02-19T15:23:39.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9aa00640 1 -- 172.21.15.47:0/1512211892 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a80005740 con 0x7f0a9c071350 2024-02-19T15:23:39.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9aa00640 1 -- 172.21.15.47:0/1512211892 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522828857 0 0) 0x7f0a9c071ba0 con 0x7f0a9c06c9a0 2024-02-19T15:23:39.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9aa00640 1 -- 172.21.15.47:0/1512211892 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a9c0718c0 con 0x7f0a9c06c9a0 2024-02-19T15:23:39.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9aa00640 1 -- 172.21.15.47:0/1512211892 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 565685468 0 0) 0x7f0a80005740 con 0x7f0a9c071350 2024-02-19T15:23:39.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9aa00640 1 -- 172.21.15.47:0/1512211892 >> v2:172.21.15.116:3300/0 conn(0x7f0a9c15b1a0 msgr2=0x7f0a9c15b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:39.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9aa00640 1 --2- 172.21.15.47:0/1512211892 >> v2:172.21.15.116:3300/0 conn(0x7f0a9c15b1a0 0x7f0a9c15b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9aa00640 1 -- 172.21.15.47:0/1512211892 >> v1:172.21.15.116:6789/0 conn(0x7f0a9c06c9a0 legacy=0x7f0a9c15cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9aa00640 1 -- 172.21.15.47:0/1512211892 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9c072160 con 0x7f0a9c071350 2024-02-19T15:23:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9aa00640 1 -- 172.21.15.47:0/1512211892 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f0a84001680 con 0x7f0a9c071350 2024-02-19T15:23:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9b400640 1 --2- 172.21.15.47:0/1512211892 >> v2:172.21.15.116:3300/0 conn(0x7f0a9c15b1a0 0x7f0a9c15b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:23:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9aa00640 1 -- 172.21.15.47:0/1512211892 >> v1:172.21.15.47:6789/0 conn(0x7f0a9c071350 legacy=0x7f0a9c15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9aa00640 1 --2- 172.21.15.47:0/1512211892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a80005d20 0x7f0a80006120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9aa00640 1 --2- 172.21.15.47:0/1512211892 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a800069e0 0x7f0a80006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9aa00640 1 --2- 172.21.15.47:0/1512211892 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a800073a0 0x7f0a8000dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9aa00640 1 -- 172.21.15.47:0/1512211892 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c071e80 con 0x7f0a800073a0 2024-02-19T15:23:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9aa00640 1 -- 172.21.15.47:0/1512211892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8000e220 con 0x7f0a80005d20 2024-02-19T15:23:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9aa00640 1 -- 172.21.15.47:0/1512211892 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8000e3e0 con 0x7f0a800069e0 2024-02-19T15:23:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9b400640 1 --2- 172.21.15.47:0/1512211892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a80005d20 0x7f0a80006120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a93e00640 1 --2- 172.21.15.47:0/1512211892 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a800069e0 0x7f0a80006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9be00640 1 --2- 172.21.15.47:0/1512211892 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a800073a0 0x7f0a8000dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9b400640 1 -- 172.21.15.47:0/1512211892 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a800069e0 msgr2=0x7f0a80006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9b400640 1 --2- 172.21.15.47:0/1512211892 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a800069e0 0x7f0a80006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9b400640 1 -- 172.21.15.47:0/1512211892 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a800073a0 msgr2=0x7f0a8000dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:39.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9b400640 1 --2- 172.21.15.47:0/1512211892 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a800073a0 0x7f0a8000dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9b400640 1 -- 172.21.15.47:0/1512211892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a80005740 con 0x7f0a80005d20 2024-02-19T15:23:39.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9be00640 1 --2- 172.21.15.47:0/1512211892 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a800073a0 0x7f0a8000dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:39.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.091+0000 7f0a9be00640 1 -- 172.21.15.47:0/1512211892 reap_dead start 2024-02-19T15:23:39.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.095+0000 7f0a9b400640 1 --2- 172.21.15.47:0/1512211892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a80005d20 0x7f0a80006120 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009cb0 tx=0x7f0a8c01cc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c6ad41261682a3e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:39.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.095+0000 7f0a9aa00640 1 -- 172.21.15.47:0/1512211892 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c0a57c0 con 0x7f0a80005d20 2024-02-19T15:23:39.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.095+0000 7f0a9aa00640 1 -- 172.21.15.47:0/1512211892 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8c0a5920 con 0x7f0a80005d20 2024-02-19T15:23:39.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.095+0000 7f0a9aa00640 1 -- 172.21.15.47:0/1512211892 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c0a5d10 con 0x7f0a80005d20 2024-02-19T15:23:39.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.095+0000 7f0aa2600640 1 -- 172.21.15.47:0/1512211892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a80005d20 msgr2=0x7f0a80006120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:39.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.095+0000 7f0aa2600640 1 --2- 172.21.15.47:0/1512211892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a80005d20 0x7f0a80006120 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009cb0 tx=0x7f0a8c01cc10 comp rx=0 tx=0).stop 2024-02-19T15:23:39.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.095+0000 7f0aa2600640 1 -- 172.21.15.47:0/1512211892 shutdown_connections 2024-02-19T15:23:39.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.095+0000 7f0aa2600640 1 --2- 172.21.15.47:0/1512211892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a80005d20 0x7f0a80006120 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.095+0000 7f0aa2600640 1 -- 172.21.15.47:0/1512211892 >> 172.21.15.47:0/1512211892 conn(0x7f0a9c0b16a0 msgr2=0x7f0a9c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:39.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.095+0000 7f0aa2600640 1 -- 172.21.15.47:0/1512211892 shutdown_connections 2024-02-19T15:23:39.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.095+0000 7f0aa2600640 1 -- 172.21.15.47:0/1512211892 wait complete. 2024-02-19T15:23:39.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0aa2600640 1 Processor -- start 2024-02-19T15:23:39.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0aa2600640 1 -- start start 2024-02-19T15:23:39.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0aa2600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a80005d20 0x7f0a9c0b4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:39.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0aa2600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a9c0b2790 0x7f0a9c0ce170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:39.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0aa2600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a9c155240 0x7f0a9c1556c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:39.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0aa2600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c0b2d50 con 0x7f0a9c0b2790 2024-02-19T15:23:39.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0aa2600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c0b3030 con 0x7f0a80005d20 2024-02-19T15:23:39.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0aa2600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c0b3310 con 0x7f0a9c155240 2024-02-19T15:23:39.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0a93e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a9c0b2790 0x7f0a9c0ce170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:39.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0a93e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a9c0b2790 0x7f0a9c0ce170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52376/0 (socket says 172.21.15.47:52376) 2024-02-19T15:23:39.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0a93e00640 1 -- 172.21.15.47:0/1224414112 learned_addr learned my addr 172.21.15.47:0/1224414112 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:39.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0a9b400640 1 --2- 172.21.15.47:0/1224414112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a80005d20 0x7f0a9c0b4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:39.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0a9be00640 1 --2- 172.21.15.47:0/1224414112 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a9c155240 0x7f0a9c1556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:39.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0a93e00640 1 -- 172.21.15.47:0/1224414112 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a9c155240 msgr2=0x7f0a9c1556c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:39.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0a93e00640 1 --2- 172.21.15.47:0/1224414112 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a9c155240 0x7f0a9c1556c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0a93e00640 1 -- 172.21.15.47:0/1224414112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a80005d20 msgr2=0x7f0a9c0b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:39.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0a93e00640 1 --2- 172.21.15.47:0/1224414112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a80005d20 0x7f0a9c0b4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0a93e00640 1 -- 172.21.15.47:0/1224414112 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9c0b35f0 con 0x7f0a9c0b2790 2024-02-19T15:23:39.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0a9be00640 1 --2- 172.21.15.47:0/1224414112 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a9c155240 0x7f0a9c1556c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:39.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0a93e00640 1 --2- 172.21.15.47:0/1224414112 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a9c0b2790 0x7f0a9c0ce170 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0a84012540 tx=0x7f0a84012a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:39.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0a93400640 1 -- 172.21.15.47:0/1224414112 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a84014810 con 0x7f0a9c0b2790 2024-02-19T15:23:39.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0a93400640 1 -- 172.21.15.47:0/1224414112 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a84014970 con 0x7f0a9c0b2790 2024-02-19T15:23:39.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0a93400640 1 -- 172.21.15.47:0/1224414112 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a84014d20 con 0x7f0a9c0b2790 2024-02-19T15:23:39.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0aa2600640 1 -- 172.21.15.47:0/1224414112 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a9c0b49c0 con 0x7f0a9c0b2790 2024-02-19T15:23:39.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0aa2600640 1 -- 172.21.15.47:0/1224414112 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a9c0ce880 con 0x7f0a9c0b2790 2024-02-19T15:23:39.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.099+0000 7f0aa2600640 1 -- 172.21.15.47:0/1224414112 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a9c075360 con 0x7f0a9c0b2790 2024-02-19T15:23:39.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.103+0000 7f0a93400640 1 -- 172.21.15.47:0/1224414112 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0a84016480 con 0x7f0a9c0b2790 2024-02-19T15:23:39.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.103+0000 7f0a93400640 1 --2- 172.21.15.47:0/1224414112 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a78066e40 0x7f0a78069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:39.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.103+0000 7f0a93400640 1 -- 172.21.15.47:0/1224414112 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0a84087b00 con 0x7f0a9c0b2790 2024-02-19T15:23:39.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.103+0000 7f0a9b400640 1 --2- 172.21.15.47:0/1224414112 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a78066e40 0x7f0a78069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:39.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.103+0000 7f0a9b400640 1 --2- 172.21.15.47:0/1224414112 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a78066e40 0x7f0a78069300 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0af000 tx=0x7f0a8c0046d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:39.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.107+0000 7f0a93400640 1 -- 172.21.15.47:0/1224414112 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0a84089e60 con 0x7f0a9c0b2790 2024-02-19T15:23:39.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.323+0000 7f0aa2600640 1 -- 172.21.15.47:0/1224414112 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0a9c0b3780 con 0x7f0a9c0b2790 2024-02-19T15:23:39.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.323+0000 7f0a93400640 1 -- 172.21.15.47:0/1224414112 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0a84054c30 con 0x7f0a9c0b2790 2024-02-19T15:23:39.331 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:39.331 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:23:39.331 INFO:teuthology.orchestra.run.smithi047.stdout: "auth_dump": [ 2024-02-19T15:23:39.331 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:39.331 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.a", 2024-02-19T15:23:39.331 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNl8BP+HRAAlJ3Jk+1k2XcZAnICzy++Ow==", 2024-02-19T15:23:39.331 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:23:39.331 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:23:39.331 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:23:39.332 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:23:39.332 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:23:39.332 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:39.332 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:39.332 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:39.332 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.b", 2024-02-19T15:23:39.332 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlIOh3KhAAlnU2ZFY4iv/XEw0WqxyZpw==", 2024-02-19T15:23:39.332 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:23:39.332 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:23:39.332 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:23:39.332 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:23:39.332 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:23:39.332 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:39.333 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:39.333 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:39.333 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.c", 2024-02-19T15:23:39.333 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNl48SHJRAAn4GCT5KnP+2PD8IXbGClCg==", 2024-02-19T15:23:39.333 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:23:39.333 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:23:39.333 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:23:39.333 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:23:39.333 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:23:39.333 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:39.333 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:39.333 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:39.333 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.d", 2024-02-19T15:23:39.333 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNljcekMRAA414655SzH43fBA0AtjpxZw==", 2024-02-19T15:23:39.333 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:23:39.334 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:23:39.334 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:23:39.334 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:23:39.334 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:23:39.334 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:39.334 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:39.334 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:39.334 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.0", 2024-02-19T15:23:39.334 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDlcdNl/bycFhAAmzYi/OxEPUW3tcRJC0DNEQ==", 2024-02-19T15:23:39.334 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:23:39.334 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:23:39.334 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:23:39.334 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:23:39.334 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:39.334 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:39.335 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:39.335 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.1", 2024-02-19T15:23:39.335 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDocdNluquhHBAA88aLHCvVol7JoZB0MtoutA==", 2024-02-19T15:23:39.335 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:23:39.335 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:23:39.335 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:23:39.335 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:23:39.335 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:39.335 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:39.335 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:39.335 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.2", 2024-02-19T15:23:39.335 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDrcdNlGAnWIhAAyELvCPKjtfgF0+kX6NQFIA==", 2024-02-19T15:23:39.335 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:23:39.335 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:23:39.335 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:23:39.336 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:23:39.336 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:39.336 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:39.336 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:39.336 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.3", 2024-02-19T15:23:39.336 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDucdNlROZ6KBAAxTCDuYsw5SRIZBcclzwxhg==", 2024-02-19T15:23:39.336 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:23:39.336 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:23:39.336 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:23:39.336 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:23:39.336 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:39.336 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:39.336 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:39.336 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.4", 2024-02-19T15:23:39.337 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD0cdNluOGILBAA66R9RVXD2vdyIw+oZ+iM1g==", 2024-02-19T15:23:39.337 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:23:39.337 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:23:39.337 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:23:39.337 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:23:39.337 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:39.337 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:39.337 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:39.337 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.5", 2024-02-19T15:23:39.337 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD3cdNlHg8dMhAA2LUzfpvvHR4i8e5RBCeGpQ==", 2024-02-19T15:23:39.337 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:23:39.337 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:23:39.337 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:23:39.337 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:23:39.337 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:39.338 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:39.338 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:39.338 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.6", 2024-02-19T15:23:39.338 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD6cdNlXD1EORAAhKGIt0wFeCyoE6QkkZ3caA==", 2024-02-19T15:23:39.338 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:23:39.338 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:23:39.338 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:23:39.338 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:23:39.338 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:39.338 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:39.338 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:39.338 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.7", 2024-02-19T15:23:39.338 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD+cdNlfW65AhAAVWj3ptC8uQHPh/UXcb/uDg==", 2024-02-19T15:23:39.338 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:23:39.339 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:23:39.339 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:23:39.339 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:23:39.339 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:39.339 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:39.339 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:39.339 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.0", 2024-02-19T15:23:39.339 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlb7eRNRAAk6ZLGfjhcdlR9GoXFCd9eA==", 2024-02-19T15:23:39.339 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:23:39.339 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:23:39.339 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow r", 2024-02-19T15:23:39.339 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-19T15:23:39.339 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:39.339 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:39.339 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:39.340 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.1", 2024-02-19T15:23:39.340 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlKBu+AxAAlLRU70SAs3ZJo5wFwRWa4w==", 2024-02-19T15:23:39.340 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:23:39.340 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:23:39.340 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:23:39.340 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:23:39.340 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:23:39.340 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:39.340 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:39.340 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:39.340 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.2", 2024-02-19T15:23:39.340 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlyDk9ABAAIAjKSd4qJO5oHhVnDc1Rgw==", 2024-02-19T15:23:39.340 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:23:39.340 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:23:39.340 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:23:39.341 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:23:39.341 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:23:39.341 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:39.341 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:39.341 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:39.341 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.3", 2024-02-19T15:23:39.341 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlUqK3CRAABmSIzoJvl0ymME6swrvnSQ==", 2024-02-19T15:23:39.341 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:23:39.341 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:23:39.341 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:23:39.341 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:23:39.341 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:23:39.341 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:39.341 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:39.341 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:39.342 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.admin", 2024-02-19T15:23:39.342 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNls3vPAhAA4x2yQTUCz4Dv9vqEWnSTyQ==", 2024-02-19T15:23:39.342 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:23:39.342 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:23:39.342 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:23:39.342 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:23:39.342 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:23:39.342 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:39.342 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:39.342 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:39.342 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mgr.x", 2024-02-19T15:23:39.342 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNliM78GRAABWBg4hMcKjV2pK5WVBjLMQ==", 2024-02-19T15:23:39.342 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:23:39.342 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:23:39.342 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile mgr", 2024-02-19T15:23:39.343 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:23:39.343 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:39.343 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:23:39.343 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:39.343 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mgr.y", 2024-02-19T15:23:39.343 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlwxs9FhAAWqP7YoKio/bbdx7SGiCJ9Q==", 2024-02-19T15:23:39.343 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:23:39.343 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:23:39.343 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile mgr", 2024-02-19T15:23:39.343 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:23:39.343 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:39.343 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:39.343 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-02-19T15:23:39.343 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:23:39.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.327+0000 7f0aa2600640 1 -- 172.21.15.47:0/1224414112 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a78066e40 msgr2=0x7f0a78069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:39.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.327+0000 7f0aa2600640 1 --2- 172.21.15.47:0/1224414112 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a78066e40 0x7f0a78069300 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0af000 tx=0x7f0a8c0046d0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.327+0000 7f0aa2600640 1 -- 172.21.15.47:0/1224414112 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a9c0b2790 msgr2=0x7f0a9c0ce170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:39.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.327+0000 7f0aa2600640 1 --2- 172.21.15.47:0/1224414112 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a9c0b2790 0x7f0a9c0ce170 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0a84012540 tx=0x7f0a84012a00 comp rx=0 tx=0).stop 2024-02-19T15:23:39.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.327+0000 7f0aa2600640 1 -- 172.21.15.47:0/1224414112 shutdown_connections 2024-02-19T15:23:39.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.327+0000 7f0aa2600640 1 --2- 172.21.15.47:0/1224414112 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a78066e40 0x7f0a78069300 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.327+0000 7f0aa2600640 1 --2- 172.21.15.47:0/1224414112 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a9c155240 0x7f0a9c1556c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.327+0000 7f0aa2600640 1 --2- 172.21.15.47:0/1224414112 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a9c0b2790 0x7f0a9c0ce170 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.327+0000 7f0aa2600640 1 --2- 172.21.15.47:0/1224414112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a80005d20 0x7f0a9c0b4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.327+0000 7f0aa2600640 1 -- 172.21.15.47:0/1224414112 >> 172.21.15.47:0/1224414112 conn(0x7f0a9c0b16a0 msgr2=0x7f0a9c158d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:39.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.327+0000 7f0aa2600640 1 -- 172.21.15.47:0/1224414112 shutdown_connections 2024-02-19T15:23:39.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.327+0000 7f0aa2600640 1 -- 172.21.15.47:0/1224414112 wait complete. 2024-02-19T15:23:39.353 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:23:39.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.487+0000 7fd4f0e00640 1 Processor -- start 2024-02-19T15:23:39.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.487+0000 7fd4f0e00640 1 -- start start 2024-02-19T15:23:39.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.487+0000 7fd4f0e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4ec145590 0x7fd4ec0727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:39.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.487+0000 7fd4f0e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd4ec0cea80 0x7fd4ec06c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:39.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.487+0000 7fd4f0e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ec15d280 con 0x7fd4ec15c390 2024-02-19T15:23:39.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.487+0000 7fd4f0e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ec070ff0 con 0x7fd4ec0cea80 2024-02-19T15:23:39.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.487+0000 7fd4f0e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec0709e0 con 0x7fd4ec145590 2024-02-19T15:23:39.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.487+0000 7fd4e9400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd4ec0cea80 0x7fd4ec06c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:39.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.487+0000 7fd4e9e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4ec145590 0x7fd4ec0727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:39.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.487+0000 7fd4e9e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4ec145590 0x7fd4ec0727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:37802/0 (socket says 172.21.15.47:37802) 2024-02-19T15:23:39.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.487+0000 7fd4e9400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd4ec0cea80 0x7fd4ec06c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:41350/0 (socket says 172.21.15.47:41350) 2024-02-19T15:23:39.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.487+0000 7fd4e9e00640 1 -- 172.21.15.47:0/279274503 learned_addr learned my addr 172.21.15.47:0/279274503 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:39.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.487+0000 7fd4e9e00640 1 -- 172.21.15.47:0/279274503 >> v2:172.21.15.116:3300/0 conn(0x7fd4ec0cea80 msgr2=0x7fd4ec06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:39.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.487+0000 7fd4e9e00640 1 --2- 172.21.15.47:0/279274503 >> v2:172.21.15.116:3300/0 conn(0x7fd4ec0cea80 0x7fd4ec06c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.487+0000 7fd4e9e00640 1 -- 172.21.15.47:0/279274503 >> v1:172.21.15.116:6789/0 conn(0x7fd4ec15c390 legacy=0x7fd4ec15c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:39.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.487+0000 7fd4e9e00640 1 -- 172.21.15.47:0/279274503 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ec06cee0 con 0x7fd4ec145590 2024-02-19T15:23:39.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.491+0000 7fd4e9e00640 1 --2- 172.21.15.47:0/279274503 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4ec145590 0x7fd4ec0727b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc0098a0 tx=0x7fd4dc01c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5062386f7f8277e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:39.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.491+0000 7fd4e8a00640 1 -- 172.21.15.47:0/279274503 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc0ab650 con 0x7fd4ec145590 2024-02-19T15:23:39.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.491+0000 7fd4e8a00640 1 -- 172.21.15.47:0/279274503 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4dc0ab7b0 con 0x7fd4ec145590 2024-02-19T15:23:39.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.491+0000 7fd4e8a00640 1 -- 172.21.15.47:0/279274503 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc0abb40 con 0x7fd4ec145590 2024-02-19T15:23:39.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.491+0000 7fd4f0e00640 1 -- 172.21.15.47:0/279274503 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4ec145590 msgr2=0x7fd4ec0727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:39.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.491+0000 7fd4f0e00640 1 --2- 172.21.15.47:0/279274503 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4ec145590 0x7fd4ec0727b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc0098a0 tx=0x7fd4dc01c540 comp rx=0 tx=0).stop 2024-02-19T15:23:39.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.491+0000 7fd4f0e00640 1 -- 172.21.15.47:0/279274503 shutdown_connections 2024-02-19T15:23:39.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.491+0000 7fd4f0e00640 1 --2- 172.21.15.47:0/279274503 >> v2:172.21.15.116:3300/0 conn(0x7fd4ec0cea80 0x7fd4ec06c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.491+0000 7fd4f0e00640 1 --2- 172.21.15.47:0/279274503 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4ec145590 0x7fd4ec0727b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.491+0000 7fd4f0e00640 1 -- 172.21.15.47:0/279274503 >> 172.21.15.47:0/279274503 conn(0x7fd4ec0b16a0 msgr2=0x7fd4ec0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:39.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.491+0000 7fd4f0e00640 1 -- 172.21.15.47:0/279274503 shutdown_connections 2024-02-19T15:23:39.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.491+0000 7fd4f0e00640 1 -- 172.21.15.47:0/279274503 wait complete. 2024-02-19T15:23:39.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4f0e00640 1 Processor -- start 2024-02-19T15:23:39.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4f0e00640 1 -- start start 2024-02-19T15:23:39.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4f0e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4ec0cea80 0x7fd4ec139eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:39.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4f0e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4ec145590 0x7fd4ec158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:39.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4f0e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4ec15c390 0x7fd4ec13b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:39.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4f0e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec13a3f0 con 0x7fd4ec145590 2024-02-19T15:23:39.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4f0e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec159510 con 0x7fd4ec0cea80 2024-02-19T15:23:39.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4f0e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec15a9f0 con 0x7fd4ec15c390 2024-02-19T15:23:39.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4e9400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4ec145590 0x7fd4ec158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:39.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4e9400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4ec145590 0x7fd4ec158fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52384/0 (socket says 172.21.15.47:52384) 2024-02-19T15:23:39.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4e9400640 1 -- 172.21.15.47:0/2762825423 learned_addr learned my addr 172.21.15.47:0/2762825423 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:39.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4e9e00640 1 --2- 172.21.15.47:0/2762825423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4ec0cea80 0x7fd4ec139eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:39.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4ea800640 1 --2- 172.21.15.47:0/2762825423 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4ec15c390 0x7fd4ec13b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:39.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4e9400640 1 -- 172.21.15.47:0/2762825423 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4ec15c390 msgr2=0x7fd4ec13b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:39.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4e9400640 1 --2- 172.21.15.47:0/2762825423 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4ec15c390 0x7fd4ec13b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4e9400640 1 -- 172.21.15.47:0/2762825423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4ec0cea80 msgr2=0x7fd4ec139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:39.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4e9400640 1 --2- 172.21.15.47:0/2762825423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4ec0cea80 0x7fd4ec139eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4e9400640 1 -- 172.21.15.47:0/2762825423 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4dc0094d0 con 0x7fd4ec145590 2024-02-19T15:23:39.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4e9400640 1 --2- 172.21.15.47:0/2762825423 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4ec145590 0x7fd4ec158fd0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd4d800e930 tx=0x7fd4d800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:39.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4e2000640 1 -- 172.21.15.47:0/2762825423 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8009690 con 0x7fd4ec145590 2024-02-19T15:23:39.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4e2000640 1 -- 172.21.15.47:0/2762825423 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d800ca00 con 0x7fd4ec145590 2024-02-19T15:23:39.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4e2000640 1 -- 172.21.15.47:0/2762825423 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d800cd70 con 0x7fd4ec145590 2024-02-19T15:23:39.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4f0e00640 1 -- 172.21.15.47:0/2762825423 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4ec1386b0 con 0x7fd4ec145590 2024-02-19T15:23:39.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4f0e00640 1 -- 172.21.15.47:0/2762825423 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4ec138c70 con 0x7fd4ec145590 2024-02-19T15:23:39.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.495+0000 7fd4f0e00640 1 -- 172.21.15.47:0/2762825423 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4ec075360 con 0x7fd4ec145590 2024-02-19T15:23:39.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.499+0000 7fd4e2000640 1 -- 172.21.15.47:0/2762825423 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd4d8013470 con 0x7fd4ec145590 2024-02-19T15:23:39.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.499+0000 7fd4e2000640 1 --2- 172.21.15.47:0/2762825423 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd4cc066e70 0x7fd4cc069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:39.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.499+0000 7fd4e2000640 1 -- 172.21.15.47:0/2762825423 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd4d8084840 con 0x7fd4ec145590 2024-02-19T15:23:39.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.499+0000 7fd4e9e00640 1 --2- 172.21.15.47:0/2762825423 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd4cc066e70 0x7fd4cc069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:39.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.499+0000 7fd4e9e00640 1 --2- 172.21.15.47:0/2762825423 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd4cc066e70 0x7fd4cc069330 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc0af000 tx=0x7fd4dc0b6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:39.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.503+0000 7fd4e2000640 1 -- 172.21.15.47:0/2762825423 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd4d8051af0 con 0x7fd4ec145590 2024-02-19T15:23:39.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.663+0000 7fd4f0e00640 1 -- 172.21.15.47:0/2762825423 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4ec069020 con 0x7fd4ec145590 2024-02-19T15:23:39.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.667+0000 7fd4e2000640 1 -- 172.21.15.47:0/2762825423 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+12903 (secure 0 0 0) 0x7fd4d8017090 con 0x7fd4ec145590 2024-02-19T15:23:39.672 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:39.672 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":41,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:23:38.926653+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:23:39.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.667+0000 7fd4f0e00640 1 -- 172.21.15.47:0/2762825423 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd4cc066e70 msgr2=0x7fd4cc069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:39.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.667+0000 7fd4f0e00640 1 --2- 172.21.15.47:0/2762825423 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd4cc066e70 0x7fd4cc069330 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc0af000 tx=0x7fd4dc0b6040 comp rx=0 tx=0).stop 2024-02-19T15:23:39.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.667+0000 7fd4f0e00640 1 -- 172.21.15.47:0/2762825423 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4ec145590 msgr2=0x7fd4ec158fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:39.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.667+0000 7fd4f0e00640 1 --2- 172.21.15.47:0/2762825423 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4ec145590 0x7fd4ec158fd0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd4d800e930 tx=0x7fd4d800edf0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.667+0000 7fd4f0e00640 1 -- 172.21.15.47:0/2762825423 shutdown_connections 2024-02-19T15:23:39.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.667+0000 7fd4f0e00640 1 --2- 172.21.15.47:0/2762825423 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd4cc066e70 0x7fd4cc069330 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.667+0000 7fd4f0e00640 1 --2- 172.21.15.47:0/2762825423 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4ec15c390 0x7fd4ec13b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.667+0000 7fd4f0e00640 1 --2- 172.21.15.47:0/2762825423 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4ec145590 0x7fd4ec158fd0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.667+0000 7fd4f0e00640 1 --2- 172.21.15.47:0/2762825423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4ec0cea80 0x7fd4ec139eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.667+0000 7fd4f0e00640 1 -- 172.21.15.47:0/2762825423 >> 172.21.15.47:0/2762825423 conn(0x7fd4ec0b16a0 msgr2=0x7fd4ec0afef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:39.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.667+0000 7fd4f0e00640 1 -- 172.21.15.47:0/2762825423 shutdown_connections 2024-02-19T15:23:39.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.667+0000 7fd4f0e00640 1 -- 172.21.15.47:0/2762825423 wait complete. 2024-02-19T15:23:39.693 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:23:39.693 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:23:39.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.831+0000 7fd80a200640 1 Processor -- start 2024-02-19T15:23:39.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.831+0000 7fd80a200640 1 -- start start 2024-02-19T15:23:39.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.831+0000 7fd80a200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd8040ce3b0 0x7fd8040ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:39.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.831+0000 7fd80a200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8040d2650 0x7fd8040ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:39.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.831+0000 7fd80a200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8040cc620 con 0x7fd80415e280 2024-02-19T15:23:39.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.831+0000 7fd80a200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8040d12e0 con 0x7fd8040ce3b0 2024-02-19T15:23:39.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.831+0000 7fd80a200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8040d1940 con 0x7fd8040d2650 2024-02-19T15:23:39.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.831+0000 7fd802a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8040d2650 0x7fd8040ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:39.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.831+0000 7fd803400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd8040ce3b0 0x7fd8040ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:39.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.831+0000 7fd802a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8040d2650 0x7fd8040ccab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:37812/0 (socket says 172.21.15.47:37812) 2024-02-19T15:23:39.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.831+0000 7fd802a00640 1 -- 172.21.15.47:0/2346160233 learned_addr learned my addr 172.21.15.47:0/2346160233 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:39.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.831+0000 7fd803400640 1 -- 172.21.15.47:0/2346160233 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8040d2650 msgr2=0x7fd8040ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:39.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.831+0000 7fd803400640 1 --2- 172.21.15.47:0/2346160233 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8040d2650 0x7fd8040ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.831+0000 7fd803400640 1 -- 172.21.15.47:0/2346160233 >> v1:172.21.15.116:6789/0 conn(0x7fd80415e280 legacy=0x7fd8040ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:39.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.831+0000 7fd803400640 1 -- 172.21.15.47:0/2346160233 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8040cecd0 con 0x7fd8040ce3b0 2024-02-19T15:23:39.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.835+0000 7fd803400640 1 --2- 172.21.15.47:0/2346160233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8040ce3b0 0x7fd8040ce790 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd7f40097c0 tx=0x7fd7f401c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a67dfb9e982f8f49 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:39.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.835+0000 7fd802000640 1 -- 172.21.15.47:0/2346160233 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f40ab600 con 0x7fd8040ce3b0 2024-02-19T15:23:39.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.835+0000 7fd802000640 1 -- 172.21.15.47:0/2346160233 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7f40a7050 con 0x7fd8040ce3b0 2024-02-19T15:23:39.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.835+0000 7fd802000640 1 -- 172.21.15.47:0/2346160233 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f40ab980 con 0x7fd8040ce3b0 2024-02-19T15:23:39.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.835+0000 7fd80a200640 1 -- 172.21.15.47:0/2346160233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8040ce3b0 msgr2=0x7fd8040ce790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:39.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.835+0000 7fd80a200640 1 --2- 172.21.15.47:0/2346160233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8040ce3b0 0x7fd8040ce790 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd7f40097c0 tx=0x7fd7f401c380 comp rx=0 tx=0).stop 2024-02-19T15:23:39.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.835+0000 7fd80a200640 1 -- 172.21.15.47:0/2346160233 shutdown_connections 2024-02-19T15:23:39.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.835+0000 7fd80a200640 1 --2- 172.21.15.47:0/2346160233 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8040d2650 0x7fd8040ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.835+0000 7fd80a200640 1 --2- 172.21.15.47:0/2346160233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8040ce3b0 0x7fd8040ce790 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.835+0000 7fd80a200640 1 -- 172.21.15.47:0/2346160233 >> 172.21.15.47:0/2346160233 conn(0x7fd8040d1b30 msgr2=0x7fd8040d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:39.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.835+0000 7fd80a200640 1 -- 172.21.15.47:0/2346160233 shutdown_connections 2024-02-19T15:23:39.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.835+0000 7fd80a200640 1 -- 172.21.15.47:0/2346160233 wait complete. 2024-02-19T15:23:39.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd80a200640 1 Processor -- start 2024-02-19T15:23:39.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd80a200640 1 -- start start 2024-02-19T15:23:39.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd80a200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8040ce3b0 0x7fd8041609d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:39.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd80a200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8040d2650 0x7fd804149030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:39.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd80a200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80415e280 0x7fd80415ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:39.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd80a200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd804160f10 con 0x7fd80415e280 2024-02-19T15:23:39.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd80a200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd804150640 con 0x7fd8040ce3b0 2024-02-19T15:23:39.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd80a200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd804150920 con 0x7fd8040d2650 2024-02-19T15:23:39.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd803e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80415e280 0x7fd80415ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:39.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd803e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80415e280 0x7fd80415ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52390/0 (socket says 172.21.15.47:52390) 2024-02-19T15:23:39.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd803e00640 1 -- 172.21.15.47:0/951462808 learned_addr learned my addr 172.21.15.47:0/951462808 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:39.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd802a00640 1 --2- 172.21.15.47:0/951462808 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8040d2650 0x7fd804149030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:39.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd803400640 1 --2- 172.21.15.47:0/951462808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8040ce3b0 0x7fd8041609d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:39.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd803e00640 1 -- 172.21.15.47:0/951462808 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8040d2650 msgr2=0x7fd804149030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:39.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd803e00640 1 --2- 172.21.15.47:0/951462808 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8040d2650 0x7fd804149030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd803e00640 1 -- 172.21.15.47:0/951462808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8040ce3b0 msgr2=0x7fd8041609d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:39.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd803e00640 1 --2- 172.21.15.47:0/951462808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8040ce3b0 0x7fd8041609d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:39.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd803e00640 1 -- 172.21.15.47:0/951462808 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f4009420 con 0x7fd80415e280 2024-02-19T15:23:39.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd802a00640 1 --2- 172.21.15.47:0/951462808 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8040d2650 0x7fd804149030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:23:39.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd803e00640 1 --2- 172.21.15.47:0/951462808 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80415e280 0x7fd80415ca00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc00c730 tx=0x7fd7fc00cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:39.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd7fb400640 1 -- 172.21.15.47:0/951462808 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc006d60 con 0x7fd80415e280 2024-02-19T15:23:39.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd7fb400640 1 -- 172.21.15.47:0/951462808 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7fc003980 con 0x7fd80415e280 2024-02-19T15:23:39.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd80a200640 1 -- 172.21.15.47:0/951462808 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd804150c00 con 0x7fd80415e280 2024-02-19T15:23:39.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd7fb400640 1 -- 172.21.15.47:0/951462808 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc003d30 con 0x7fd80415e280 2024-02-19T15:23:39.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.839+0000 7fd80a200640 1 -- 172.21.15.47:0/951462808 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8041511c0 con 0x7fd80415e280 2024-02-19T15:23:39.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.843+0000 7fd80a200640 1 -- 172.21.15.47:0/951462808 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8041513e0 con 0x7fd80415e280 2024-02-19T15:23:39.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.843+0000 7fd7fb400640 1 -- 172.21.15.47:0/951462808 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd7fc0050d0 con 0x7fd80415e280 2024-02-19T15:23:39.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.843+0000 7fd7fb400640 1 --2- 172.21.15.47:0/951462808 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd7e4066e40 0x7fd7e4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:39.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.843+0000 7fd7fb400640 1 -- 172.21.15.47:0/951462808 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd7fc084180 con 0x7fd80415e280 2024-02-19T15:23:39.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.843+0000 7fd803400640 1 --2- 172.21.15.47:0/951462808 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd7e4066e40 0x7fd7e4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:39.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.843+0000 7fd803400640 1 --2- 172.21.15.47:0/951462808 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd7e4066e40 0x7fd7e4069300 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd7f40ad000 tx=0x7fd7f4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:39.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:39.847+0000 7fd7fb400640 1 -- 172.21.15.47:0/951462808 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd7fc054e10 con 0x7fd80415e280 2024-02-19T15:23:40.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.015+0000 7fd80a200640 1 -- 172.21.15.47:0/951462808 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd804075360 con 0x7fd80415e280 2024-02-19T15:23:40.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.015+0000 7fd7fb400640 1 -- 172.21.15.47:0/951462808 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd7fc0549a0 con 0x7fd80415e280 2024-02-19T15:23:40.022 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:23:40.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.019+0000 7fd80a200640 1 -- 172.21.15.47:0/951462808 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd7e4066e40 msgr2=0x7fd7e4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:40.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.019+0000 7fd80a200640 1 --2- 172.21.15.47:0/951462808 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd7e4066e40 0x7fd7e4069300 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd7f40ad000 tx=0x7fd7f4008040 comp rx=0 tx=0).stop 2024-02-19T15:23:40.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.019+0000 7fd80a200640 1 -- 172.21.15.47:0/951462808 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80415e280 msgr2=0x7fd80415ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:40.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.019+0000 7fd80a200640 1 --2- 172.21.15.47:0/951462808 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80415e280 0x7fd80415ca00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc00c730 tx=0x7fd7fc00cb30 comp rx=0 tx=0).stop 2024-02-19T15:23:40.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.019+0000 7fd80a200640 1 -- 172.21.15.47:0/951462808 shutdown_connections 2024-02-19T15:23:40.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.019+0000 7fd80a200640 1 --2- 172.21.15.47:0/951462808 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd7e4066e40 0x7fd7e4069300 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.019+0000 7fd80a200640 1 --2- 172.21.15.47:0/951462808 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80415e280 0x7fd80415ca00 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.019+0000 7fd80a200640 1 --2- 172.21.15.47:0/951462808 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8040d2650 0x7fd804149030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.019+0000 7fd80a200640 1 --2- 172.21.15.47:0/951462808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8040ce3b0 0x7fd8041609d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.019+0000 7fd80a200640 1 -- 172.21.15.47:0/951462808 >> 172.21.15.47:0/951462808 conn(0x7fd8040d1b30 msgr2=0x7fd8040ccab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:40.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.019+0000 7fd80a200640 1 -- 172.21.15.47:0/951462808 shutdown_connections 2024-02-19T15:23:40.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.019+0000 7fd80a200640 1 -- 172.21.15.47:0/951462808 wait complete. 2024-02-19T15:23:40.043 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T15:23:40.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.175+0000 7f65e7a00640 1 Processor -- start 2024-02-19T15:23:40.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.175+0000 7f65e7a00640 1 -- start start 2024-02-19T15:23:40.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.179+0000 7f65e7a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f65e015f510 0x7f65e0070700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:40.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.179+0000 7f65e7a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f65e015fa90 0x7f65e015ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:40.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.179+0000 7f65e7a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e00ae970 con 0x7f65e0160450 2024-02-19T15:23:40.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.179+0000 7f65e7a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e00cffc0 con 0x7f65e015fa90 2024-02-19T15:23:40.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.179+0000 7f65e7a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e01608c0 con 0x7f65e015f510 2024-02-19T15:23:40.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.179+0000 7f65dfe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f65e015fa90 0x7f65e015ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:40.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.179+0000 7f65e5a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f65e0160450 0x7f65e006ebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52534/0 (socket says 172.21.15.47:52534) 2024-02-19T15:23:40.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.179+0000 7f65dfe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f65e015fa90 0x7f65e015ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52400/0 (socket says 172.21.15.47:52400) 2024-02-19T15:23:40.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.179+0000 7f65e5a00640 1 -- 172.21.15.47:0/2782946245 learned_addr learned my addr 172.21.15.47:0/2782946245 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:40.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.179+0000 7f65e5000640 1 --2- 172.21.15.47:0/2782946245 >> v2:172.21.15.116:3300/0 conn(0x7f65e015f510 0x7f65e0070700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:40.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.179+0000 7f65dfe00640 1 -- 172.21.15.47:0/2782946245 >> v2:172.21.15.116:3300/0 conn(0x7f65e015f510 msgr2=0x7f65e0070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:40.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.179+0000 7f65dfe00640 1 --2- 172.21.15.47:0/2782946245 >> v2:172.21.15.116:3300/0 conn(0x7f65e015f510 0x7f65e0070700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.179+0000 7f65dfe00640 1 -- 172.21.15.47:0/2782946245 >> v1:172.21.15.47:6789/0 conn(0x7f65e0160450 legacy=0x7f65e006ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:40.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.179+0000 7f65dfe00640 1 -- 172.21.15.47:0/2782946245 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e0070c40 con 0x7f65e015fa90 2024-02-19T15:23:40.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.183+0000 7f65dfe00640 1 --2- 172.21.15.47:0/2782946245 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65e015fa90 0x7f65e015ff10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f65e0072ff0 tx=0x7f65cc01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c762781405ec4eca server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:40.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.183+0000 7f65df400640 1 -- 172.21.15.47:0/2782946245 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc0ab600 con 0x7f65e015fa90 2024-02-19T15:23:40.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.183+0000 7f65df400640 1 -- 172.21.15.47:0/2782946245 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65cc0a7050 con 0x7f65e015fa90 2024-02-19T15:23:40.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.183+0000 7f65df400640 1 -- 172.21.15.47:0/2782946245 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc0ab980 con 0x7f65e015fa90 2024-02-19T15:23:40.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.183+0000 7f65e7a00640 1 -- 172.21.15.47:0/2782946245 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65e015fa90 msgr2=0x7f65e015ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:40.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.183+0000 7f65e7a00640 1 --2- 172.21.15.47:0/2782946245 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65e015fa90 0x7f65e015ff10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f65e0072ff0 tx=0x7f65cc01c3f0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.183+0000 7f65e7a00640 1 -- 172.21.15.47:0/2782946245 shutdown_connections 2024-02-19T15:23:40.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.183+0000 7f65e7a00640 1 --2- 172.21.15.47:0/2782946245 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65e015fa90 0x7f65e015ff10 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.183+0000 7f65e7a00640 1 --2- 172.21.15.47:0/2782946245 >> v2:172.21.15.116:3300/0 conn(0x7f65e015f510 0x7f65e0070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.183+0000 7f65e7a00640 1 -- 172.21.15.47:0/2782946245 >> 172.21.15.47:0/2782946245 conn(0x7f65e00b3690 msgr2=0x7f65e00b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:40.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.183+0000 7f65e7a00640 1 -- 172.21.15.47:0/2782946245 shutdown_connections 2024-02-19T15:23:40.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.183+0000 7f65e7a00640 1 -- 172.21.15.47:0/2782946245 wait complete. 2024-02-19T15:23:40.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.183+0000 7f65e7a00640 1 Processor -- start 2024-02-19T15:23:40.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.183+0000 7f65e7a00640 1 -- start start 2024-02-19T15:23:40.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.183+0000 7f65e7a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65e015f510 0x7f65e006bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:40.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.183+0000 7f65e7a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65e015fa90 0x7f65e00b6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:40.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.183+0000 7f65e7a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e0160450 0x7f65e00b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:40.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.183+0000 7f65e7a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e00b3010 con 0x7f65e015fa90 2024-02-19T15:23:40.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.183+0000 7f65e7a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e00b7320 con 0x7f65e0160450 2024-02-19T15:23:40.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.183+0000 7f65e7a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e00b3430 con 0x7f65e015f510 2024-02-19T15:23:40.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.187+0000 7f65dfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65e015fa90 0x7f65e00b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:40.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.187+0000 7f65dfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65e015fa90 0x7f65e00b6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52414/0 (socket says 172.21.15.47:52414) 2024-02-19T15:23:40.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.187+0000 7f65dfe00640 1 -- 172.21.15.47:0/765531811 learned_addr learned my addr 172.21.15.47:0/765531811 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:40.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.187+0000 7f65e5000640 1 --2- 172.21.15.47:0/765531811 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65e015f510 0x7f65e006bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:40.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.187+0000 7f65e5a00640 1 --2- 172.21.15.47:0/765531811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e0160450 0x7f65e00b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:40.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.187+0000 7f65dfe00640 1 -- 172.21.15.47:0/765531811 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65e015f510 msgr2=0x7f65e006bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:40.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.187+0000 7f65dfe00640 1 --2- 172.21.15.47:0/765531811 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65e015f510 0x7f65e006bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.187+0000 7f65dfe00640 1 -- 172.21.15.47:0/765531811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e0160450 msgr2=0x7f65e00b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:40.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.187+0000 7f65dfe00640 1 --2- 172.21.15.47:0/765531811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e0160450 0x7f65e00b6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.187+0000 7f65dfe00640 1 -- 172.21.15.47:0/765531811 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc009420 con 0x7f65e015fa90 2024-02-19T15:23:40.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.187+0000 7f65dfe00640 1 --2- 172.21.15.47:0/765531811 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65e015fa90 0x7f65e00b6650 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f65cc0ae040 tx=0x7f65cc019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:40.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.187+0000 7f65dcc00640 1 -- 172.21.15.47:0/765531811 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc0b4430 con 0x7f65e015fa90 2024-02-19T15:23:40.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.187+0000 7f65dcc00640 1 -- 172.21.15.47:0/765531811 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65cc0b45b0 con 0x7f65e015fa90 2024-02-19T15:23:40.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.187+0000 7f65dcc00640 1 -- 172.21.15.47:0/765531811 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc0b4960 con 0x7f65e015fa90 2024-02-19T15:23:40.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.187+0000 7f65e7a00640 1 -- 172.21.15.47:0/765531811 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e00b2ca0 con 0x7f65e015fa90 2024-02-19T15:23:40.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.187+0000 7f65e7a00640 1 -- 172.21.15.47:0/765531811 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e00b7810 con 0x7f65e015fa90 2024-02-19T15:23:40.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.187+0000 7f65e7a00640 1 -- 172.21.15.47:0/765531811 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e00b3150 con 0x7f65e015fa90 2024-02-19T15:23:40.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.187+0000 7f65dcc00640 1 -- 172.21.15.47:0/765531811 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f65cc0b2040 con 0x7f65e015fa90 2024-02-19T15:23:40.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.191+0000 7f65dcc00640 1 --2- 172.21.15.47:0/765531811 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f65c0066e40 0x7f65c0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:40.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.191+0000 7f65dcc00640 1 -- 172.21.15.47:0/765531811 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f65cc11e600 con 0x7f65e015fa90 2024-02-19T15:23:40.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.191+0000 7f65e5000640 1 --2- 172.21.15.47:0/765531811 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f65c0066e40 0x7f65c0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:40.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.191+0000 7f65e5000640 1 --2- 172.21.15.47:0/765531811 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f65c0066e40 0x7f65c0069300 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f65d4009a50 tx=0x7f65d4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:40.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.195+0000 7f65dcc00640 1 -- 172.21.15.47:0/765531811 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f65e00b3150 con 0x7f65e015fa90 2024-02-19T15:23:40.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.355+0000 7f65e7a00640 1 -- 172.21.15.47:0/765531811 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f65e00b3150 con 0x7f65e015fa90 2024-02-19T15:23:40.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.359+0000 7f65dcc00640 1 -- 172.21.15.47:0/765531811 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v41) v1 ==== 84+0+65 (secure 0 0 0) 0x7f65e00b3150 con 0x7f65e015fa90 2024-02-19T15:23:40.363 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:40.363 INFO:teuthology.orchestra.run.smithi047.stdout:[ 2024-02-19T15:23:40.364 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:23:40.364 INFO:teuthology.orchestra.run.smithi047.stdout: "poolnum": 1, 2024-02-19T15:23:40.364 INFO:teuthology.orchestra.run.smithi047.stdout: "poolname": ".mgr" 2024-02-19T15:23:40.364 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:23:40.364 INFO:teuthology.orchestra.run.smithi047.stdout:] 2024-02-19T15:23:40.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.359+0000 7f65e7a00640 1 -- 172.21.15.47:0/765531811 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f65c0066e40 msgr2=0x7f65c0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:40.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.359+0000 7f65e7a00640 1 --2- 172.21.15.47:0/765531811 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f65c0066e40 0x7f65c0069300 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f65d4009a50 tx=0x7f65d4008040 comp rx=0 tx=0).stop 2024-02-19T15:23:40.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.359+0000 7f65e7a00640 1 -- 172.21.15.47:0/765531811 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65e015fa90 msgr2=0x7f65e00b6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:40.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.359+0000 7f65e7a00640 1 --2- 172.21.15.47:0/765531811 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65e015fa90 0x7f65e00b6650 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f65cc0ae040 tx=0x7f65cc019190 comp rx=0 tx=0).stop 2024-02-19T15:23:40.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.359+0000 7f65e7a00640 1 -- 172.21.15.47:0/765531811 shutdown_connections 2024-02-19T15:23:40.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.359+0000 7f65e7a00640 1 --2- 172.21.15.47:0/765531811 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f65c0066e40 0x7f65c0069300 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.359+0000 7f65e7a00640 1 --2- 172.21.15.47:0/765531811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e0160450 0x7f65e00b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.359+0000 7f65e7a00640 1 --2- 172.21.15.47:0/765531811 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65e015fa90 0x7f65e00b6650 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.359+0000 7f65e7a00640 1 --2- 172.21.15.47:0/765531811 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65e015f510 0x7f65e006bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.359+0000 7f65e7a00640 1 -- 172.21.15.47:0/765531811 >> 172.21.15.47:0/765531811 conn(0x7f65e00b3690 msgr2=0x7f65e0159590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:40.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.359+0000 7f65e7a00640 1 -- 172.21.15.47:0/765531811 shutdown_connections 2024-02-19T15:23:40.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.359+0000 7f65e7a00640 1 -- 172.21.15.47:0/765531811 wait complete. 2024-02-19T15:23:40.383 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T15:23:40.384 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T15:23:40.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.523+0000 7f7ed5c00640 1 Processor -- start 2024-02-19T15:23:40.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.523+0000 7f7ed5c00640 1 -- start start 2024-02-19T15:23:40.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.523+0000 7f7ed5c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ed00b33b0 0x7f7ed00b3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:40.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.523+0000 7f7ed5c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7ed00cf5f0 0x7f7ed00cfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:40.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.523+0000 7f7ed5c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed015f180 con 0x7f7ed00718c0 2024-02-19T15:23:40.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.523+0000 7f7ed5c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ed015e5b0 con 0x7f7ed00cf5f0 2024-02-19T15:23:40.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.523+0000 7f7ed5c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed00b3cd0 con 0x7f7ed00b33b0 2024-02-19T15:23:40.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.523+0000 7f7ec7e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7ed00cf5f0 0x7f7ed00cfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:40.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.523+0000 7f7ec7e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7ed00cf5f0 0x7f7ed00cfa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52426/0 (socket says 172.21.15.47:52426) 2024-02-19T15:23:40.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.523+0000 7f7ec7e00640 1 -- 172.21.15.47:0/1001817120 learned_addr learned my addr 172.21.15.47:0/1001817120 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:40.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.523+0000 7f7ecfe00640 1 --2- 172.21.15.47:0/1001817120 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ed00b33b0 0x7f7ed00b3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:40.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.523+0000 7f7ec7e00640 1 -- 172.21.15.47:0/1001817120 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ed00b33b0 msgr2=0x7f7ed00b3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:40.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.523+0000 7f7ec7e00640 1 --2- 172.21.15.47:0/1001817120 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ed00b33b0 0x7f7ed00b3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.523+0000 7f7ec7e00640 1 -- 172.21.15.47:0/1001817120 >> v1:172.21.15.116:6789/0 conn(0x7f7ed00718c0 legacy=0x7f7ed0071d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:40.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.523+0000 7f7ec7e00640 1 -- 172.21.15.47:0/1001817120 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed015e850 con 0x7f7ed00cf5f0 2024-02-19T15:23:40.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.523+0000 7f7ecfe00640 1 --2- 172.21.15.47:0/1001817120 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ed00b33b0 0x7f7ed00b3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:23:40.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.527+0000 7f7ec7e00640 1 --2- 172.21.15.47:0/1001817120 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ed00cf5f0 0x7f7ed00cfa30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009870 tx=0x7f7eb801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=889603e8bf5f725a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:40.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.527+0000 7f7ecf400640 1 -- 172.21.15.47:0/1001817120 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb80aa610 con 0x7f7ed00cf5f0 2024-02-19T15:23:40.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.527+0000 7f7ecf400640 1 -- 172.21.15.47:0/1001817120 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb80a6040 con 0x7f7ed00cf5f0 2024-02-19T15:23:40.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.527+0000 7f7ecf400640 1 -- 172.21.15.47:0/1001817120 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb80aa990 con 0x7f7ed00cf5f0 2024-02-19T15:23:40.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.527+0000 7f7ed5c00640 1 -- 172.21.15.47:0/1001817120 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ed00cf5f0 msgr2=0x7f7ed00cfa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:40.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.527+0000 7f7ed5c00640 1 --2- 172.21.15.47:0/1001817120 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ed00cf5f0 0x7f7ed00cfa30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009870 tx=0x7f7eb801c4f0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.527+0000 7f7ed5c00640 1 -- 172.21.15.47:0/1001817120 shutdown_connections 2024-02-19T15:23:40.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.527+0000 7f7ed5c00640 1 --2- 172.21.15.47:0/1001817120 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ed00cf5f0 0x7f7ed00cfa30 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.527+0000 7f7ed5c00640 1 --2- 172.21.15.47:0/1001817120 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ed00b33b0 0x7f7ed00b3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.527+0000 7f7ed5c00640 1 -- 172.21.15.47:0/1001817120 >> 172.21.15.47:0/1001817120 conn(0x7f7ed006aeb0 msgr2=0x7f7ed006b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:40.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.527+0000 7f7ed5c00640 1 -- 172.21.15.47:0/1001817120 shutdown_connections 2024-02-19T15:23:40.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.527+0000 7f7ed5c00640 1 -- 172.21.15.47:0/1001817120 wait complete. 2024-02-19T15:23:40.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.527+0000 7f7ed5c00640 1 Processor -- start 2024-02-19T15:23:40.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.527+0000 7f7ed5c00640 1 -- start start 2024-02-19T15:23:40.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7ed5c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ed00718c0 0x7f7ed014e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:40.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7ed5c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ed00b33b0 0x7f7ed0157230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:40.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7ed5c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ed00cf5f0 0x7f7ed00b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:40.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7ed5c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed014e8b0 con 0x7f7ed00718c0 2024-02-19T15:23:40.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7ed5c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed0157b30 con 0x7f7ed00cf5f0 2024-02-19T15:23:40.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7ed5c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed0157e10 con 0x7f7ed00b33b0 2024-02-19T15:23:40.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7ecfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ed00718c0 0x7f7ed014e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:40.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7ecfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ed00718c0 0x7f7ed014e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52434/0 (socket says 172.21.15.47:52434) 2024-02-19T15:23:40.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7ecfe00640 1 -- 172.21.15.47:0/2897129711 learned_addr learned my addr 172.21.15.47:0/2897129711 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:40.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7ec7e00640 1 --2- 172.21.15.47:0/2897129711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ed00b33b0 0x7f7ed0157230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:40.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7ed5200640 1 --2- 172.21.15.47:0/2897129711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ed00cf5f0 0x7f7ed00b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:40.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7ecfe00640 1 -- 172.21.15.47:0/2897129711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ed00b33b0 msgr2=0x7f7ed0157230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:40.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7ecfe00640 1 --2- 172.21.15.47:0/2897129711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ed00b33b0 0x7f7ed0157230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7ecfe00640 1 -- 172.21.15.47:0/2897129711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ed00cf5f0 msgr2=0x7f7ed00b06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:40.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7ecfe00640 1 --2- 172.21.15.47:0/2897129711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ed00cf5f0 0x7f7ed00b06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7ecfe00640 1 -- 172.21.15.47:0/2897129711 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb80094d0 con 0x7f7ed00718c0 2024-02-19T15:23:40.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7ecfe00640 1 --2- 172.21.15.47:0/2897129711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ed00718c0 0x7f7ed014e370 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7ec000c7c0 tx=0x7f7ec000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:40.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7eccc00640 1 -- 172.21.15.47:0/2897129711 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0007a30 con 0x7f7ed00718c0 2024-02-19T15:23:40.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7eccc00640 1 -- 172.21.15.47:0/2897129711 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec0007b90 con 0x7f7ed00718c0 2024-02-19T15:23:40.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7eccc00640 1 -- 172.21.15.47:0/2897129711 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec000ce70 con 0x7f7ed00718c0 2024-02-19T15:23:40.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7ed5c00640 1 -- 172.21.15.47:0/2897129711 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed01580f0 con 0x7f7ed00718c0 2024-02-19T15:23:40.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7ed5c00640 1 -- 172.21.15.47:0/2897129711 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed014f2d0 con 0x7f7ed00718c0 2024-02-19T15:23:40.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.531+0000 7f7ed5c00640 1 -- 172.21.15.47:0/2897129711 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ed004f260 con 0x7f7ed00718c0 2024-02-19T15:23:40.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.535+0000 7f7eccc00640 1 -- 172.21.15.47:0/2897129711 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7ec00113f0 con 0x7f7ed00718c0 2024-02-19T15:23:40.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.535+0000 7f7eccc00640 1 --2- 172.21.15.47:0/2897129711 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7eac066e40 0x7f7eac069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:40.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.535+0000 7f7eccc00640 1 -- 172.21.15.47:0/2897129711 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7ec0084b80 con 0x7f7ed00718c0 2024-02-19T15:23:40.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.535+0000 7f7ec7e00640 1 --2- 172.21.15.47:0/2897129711 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7eac066e40 0x7f7eac069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:40.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.535+0000 7f7ec7e00640 1 --2- 172.21.15.47:0/2897129711 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7eac066e40 0x7f7eac069300 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7eb80ae000 tx=0x7f7eb80a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:40.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.539+0000 7f7eccc00640 1 -- 172.21.15.47:0/2897129711 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7ec0051cb0 con 0x7f7ed00718c0 2024-02-19T15:23:40.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.699+0000 7f7ed5c00640 1 -- 172.21.15.47:0/2897129711 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f7ed015b2e0 con 0x7f7ed00718c0 2024-02-19T15:23:40.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.927+0000 7f7eccc00640 1 -- 172.21.15.47:0/2897129711 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v42) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7ec001e0b0 con 0x7f7ed00718c0 2024-02-19T15:23:40.932 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_metadata' created 2024-02-19T15:23:40.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.931+0000 7f7ed5c00640 1 -- 172.21.15.47:0/2897129711 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7eac066e40 msgr2=0x7f7eac069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:40.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.931+0000 7f7ed5c00640 1 --2- 172.21.15.47:0/2897129711 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7eac066e40 0x7f7eac069300 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7eb80ae000 tx=0x7f7eb80a7000 comp rx=0 tx=0).stop 2024-02-19T15:23:40.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.931+0000 7f7ed5c00640 1 -- 172.21.15.47:0/2897129711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ed00718c0 msgr2=0x7f7ed014e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:40.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.931+0000 7f7ed5c00640 1 --2- 172.21.15.47:0/2897129711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ed00718c0 0x7f7ed014e370 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7ec000c7c0 tx=0x7f7ec000cc80 comp rx=0 tx=0).stop 2024-02-19T15:23:40.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.931+0000 7f7ed5c00640 1 -- 172.21.15.47:0/2897129711 shutdown_connections 2024-02-19T15:23:40.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.931+0000 7f7ed5c00640 1 --2- 172.21.15.47:0/2897129711 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7eac066e40 0x7f7eac069300 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.931+0000 7f7ed5c00640 1 --2- 172.21.15.47:0/2897129711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ed00cf5f0 0x7f7ed00b06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.931+0000 7f7ed5c00640 1 --2- 172.21.15.47:0/2897129711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ed00b33b0 0x7f7ed0157230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.931+0000 7f7ed5c00640 1 --2- 172.21.15.47:0/2897129711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ed00718c0 0x7f7ed014e370 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:40.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.931+0000 7f7ed5c00640 1 -- 172.21.15.47:0/2897129711 >> 172.21.15.47:0/2897129711 conn(0x7f7ed006aeb0 msgr2=0x7f7ed00ae3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:40.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.931+0000 7f7ed5c00640 1 -- 172.21.15.47:0/2897129711 shutdown_connections 2024-02-19T15:23:40.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:40.931+0000 7f7ed5c00640 1 -- 172.21.15.47:0/2897129711 wait complete. 2024-02-19T15:23:40.959 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T15:23:41.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79e7200640 1 Processor -- start 2024-02-19T15:23:41.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79e7200640 1 -- start start 2024-02-19T15:23:41.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79e7200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e00ae140 0x7f79e00b30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:41.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79e7200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e00ace50 con 0x7f79e00b1c80 2024-02-19T15:23:41.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79e7200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e00ad0a0 con 0x7f79e00b3620 2024-02-19T15:23:41.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79e7200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e00ad380 con 0x7f79e00ae140 2024-02-19T15:23:41.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79dfe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f79e00b1c80 0x7f79e00b2060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52540/0 (socket says 172.21.15.47:52540) 2024-02-19T15:23:41.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79dfe00640 1 -- 172.21.15.47:0/1634614431 learned_addr learned my addr 172.21.15.47:0/1634614431 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:41.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79d7e00640 1 --2- 172.21.15.47:0/1634614431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e00ae140 0x7f79e00b30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:41.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79df400640 1 -- 172.21.15.47:0/1634614431 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2378702245 0 0) 0x7f79e00ace50 con 0x7f79e00b1c80 2024-02-19T15:23:41.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79df400640 1 -- 172.21.15.47:0/1634614431 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79c4003530 con 0x7f79e00b1c80 2024-02-19T15:23:41.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79df400640 1 -- 172.21.15.47:0/1634614431 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459227320 0 0) 0x7f79e00ad0a0 con 0x7f79e00b3620 2024-02-19T15:23:41.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79df400640 1 -- 172.21.15.47:0/1634614431 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79e00ace50 con 0x7f79e00b3620 2024-02-19T15:23:41.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79df400640 1 -- 172.21.15.47:0/1634614431 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2987332225 0 0) 0x7f79c4003530 con 0x7f79e00b1c80 2024-02-19T15:23:41.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79df400640 1 -- 172.21.15.47:0/1634614431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e00ae140 msgr2=0x7f79e00b30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:41.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79df400640 1 --2- 172.21.15.47:0/1634614431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e00ae140 0x7f79e00b30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:41.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79df400640 1 -- 172.21.15.47:0/1634614431 >> v1:172.21.15.116:6789/0 conn(0x7f79e00b3620 legacy=0x7f79e006aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:41.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79df400640 1 -- 172.21.15.47:0/1634614431 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e00ad660 con 0x7f79e00b1c80 2024-02-19T15:23:41.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79df400640 1 -- 172.21.15.47:0/1634614431 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f79d0001680 con 0x7f79e00b1c80 2024-02-19T15:23:41.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79df400640 1 -- 172.21.15.47:0/1634614431 >> v1:172.21.15.47:6789/0 conn(0x7f79e00b1c80 legacy=0x7f79e00b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:41.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79df400640 1 --2- 172.21.15.47:0/1634614431 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79c4003b40 0x7f79c4003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:41.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79df400640 1 --2- 172.21.15.47:0/1634614431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c40047d0 0x7f79c4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:41.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.103+0000 7f79df400640 1 --2- 172.21.15.47:0/1634614431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e00ae140 0x7f79c4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:41.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.107+0000 7f79df400640 1 -- 172.21.15.47:0/1634614431 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e00ad380 con 0x7f79c4003b40 2024-02-19T15:23:41.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.107+0000 7f79df400640 1 -- 172.21.15.47:0/1634614431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c400ba40 con 0x7f79c40047d0 2024-02-19T15:23:41.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.107+0000 7f79df400640 1 -- 172.21.15.47:0/1634614431 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c400bbd0 con 0x7f79e00ae140 2024-02-19T15:23:41.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.107+0000 7f79dfe00640 1 --2- 172.21.15.47:0/1634614431 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79c4003b40 0x7f79c4003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:41.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.107+0000 7f79d7e00640 1 --2- 172.21.15.47:0/1634614431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c40047d0 0x7f79c4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:41.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.107+0000 7f79e5200640 1 --2- 172.21.15.47:0/1634614431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e00ae140 0x7f79c4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:41.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.107+0000 7f79dfe00640 1 -- 172.21.15.47:0/1634614431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e00ae140 msgr2=0x7f79c4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:41.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.107+0000 7f79dfe00640 1 --2- 172.21.15.47:0/1634614431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e00ae140 0x7f79c4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:41.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.107+0000 7f79dfe00640 1 -- 172.21.15.47:0/1634614431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c40047d0 msgr2=0x7f79c4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:41.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.107+0000 7f79dfe00640 1 --2- 172.21.15.47:0/1634614431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c40047d0 0x7f79c4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:41.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.107+0000 7f79dfe00640 1 -- 172.21.15.47:0/1634614431 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c4003530 con 0x7f79c4003b40 2024-02-19T15:23:41.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.107+0000 7f79e5200640 1 --2- 172.21.15.47:0/1634614431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e00ae140 0x7f79c4005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:41.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.107+0000 7f79dfe00640 1 --2- 172.21.15.47:0/1634614431 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79c4003b40 0x7f79c4003f40 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f79d000c190 tx=0x7f79e006bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a7c09cf60e0c3c5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:41.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.107+0000 7f79df400640 1 -- 172.21.15.47:0/1634614431 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d00058e0 con 0x7f79c4003b40 2024-02-19T15:23:41.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.107+0000 7f79df400640 1 -- 172.21.15.47:0/1634614431 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d0005a40 con 0x7f79c4003b40 2024-02-19T15:23:41.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.107+0000 7f79df400640 1 -- 172.21.15.47:0/1634614431 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d0005df0 con 0x7f79c4003b40 2024-02-19T15:23:41.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e7200640 1 -- 172.21.15.47:0/1634614431 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79c4003b40 msgr2=0x7f79c4003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:41.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e7200640 1 --2- 172.21.15.47:0/1634614431 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79c4003b40 0x7f79c4003f40 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f79d000c190 tx=0x7f79e006bdd0 comp rx=0 tx=0).stop 2024-02-19T15:23:41.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e5200640 1 -- 172.21.15.47:0/1634614431 reap_dead start 2024-02-19T15:23:41.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e7200640 1 -- 172.21.15.47:0/1634614431 shutdown_connections 2024-02-19T15:23:41.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e7200640 1 -- 172.21.15.47:0/1634614431 >> 172.21.15.47:0/1634614431 conn(0x7f79e00ccc80 msgr2=0x7f79e00cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:41.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e7200640 1 -- 172.21.15.47:0/1634614431 shutdown_connections 2024-02-19T15:23:41.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e7200640 1 -- 172.21.15.47:0/1634614431 wait complete. 2024-02-19T15:23:41.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e7200640 1 Processor -- start 2024-02-19T15:23:41.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e7200640 1 -- start start 2024-02-19T15:23:41.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e7200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c4003b40 0x7f79e00ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:41.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e7200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e0154460 0x7f79e0150620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:41.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e7200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79e00cf970 0x7f79e00cfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:41.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e7200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e00d0330 con 0x7f79e00cf970 2024-02-19T15:23:41.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e7200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e00cb080 con 0x7f79c4003b40 2024-02-19T15:23:41.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e7200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e00cb360 con 0x7f79e0154460 2024-02-19T15:23:41.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e5200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79e00cf970 0x7f79e00cfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:41.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e5200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79e00cf970 0x7f79e00cfdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52454/0 (socket says 172.21.15.47:52454) 2024-02-19T15:23:41.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e5200640 1 -- 172.21.15.47:0/1951126857 learned_addr learned my addr 172.21.15.47:0/1951126857 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:41.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79dfe00640 1 --2- 172.21.15.47:0/1951126857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c4003b40 0x7f79e00ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:41.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79d7e00640 1 --2- 172.21.15.47:0/1951126857 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e0154460 0x7f79e0150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:41.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e5200640 1 -- 172.21.15.47:0/1951126857 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e0154460 msgr2=0x7f79e0150620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:41.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e5200640 1 --2- 172.21.15.47:0/1951126857 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e0154460 0x7f79e0150620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:41.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e5200640 1 -- 172.21.15.47:0/1951126857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c4003b40 msgr2=0x7f79e00ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:41.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e5200640 1 --2- 172.21.15.47:0/1951126857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c4003b40 0x7f79e00ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:41.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e5200640 1 -- 172.21.15.47:0/1951126857 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e00cb640 con 0x7f79e00cf970 2024-02-19T15:23:41.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.111+0000 7f79e5200640 1 --2- 172.21.15.47:0/1951126857 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79e00cf970 0x7f79e00cfdf0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f79d8001340 tx=0x7f79d800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:41.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.115+0000 7f79dcc00640 1 -- 172.21.15.47:0/1951126857 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d8006b60 con 0x7f79e00cf970 2024-02-19T15:23:41.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.115+0000 7f79dcc00640 1 -- 172.21.15.47:0/1951126857 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d8006cc0 con 0x7f79e00cf970 2024-02-19T15:23:41.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.115+0000 7f79e7200640 1 -- 172.21.15.47:0/1951126857 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e00cb920 con 0x7f79e00cf970 2024-02-19T15:23:41.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.115+0000 7f79dcc00640 1 -- 172.21.15.47:0/1951126857 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d8003bc0 con 0x7f79e00cf970 2024-02-19T15:23:41.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.115+0000 7f79e7200640 1 -- 172.21.15.47:0/1951126857 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e00cad70 con 0x7f79e00cf970 2024-02-19T15:23:41.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.115+0000 7f79e7200640 1 -- 172.21.15.47:0/1951126857 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e0075360 con 0x7f79e00cf970 2024-02-19T15:23:41.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.115+0000 7f79dcc00640 1 -- 172.21.15.47:0/1951126857 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f79d8003d20 con 0x7f79e00cf970 2024-02-19T15:23:41.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.115+0000 7f79dcc00640 1 --2- 172.21.15.47:0/1951126857 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f79bc066e40 0x7f79bc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:41.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.115+0000 7f79dcc00640 1 -- 172.21.15.47:0/1951126857 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f79d8083ae0 con 0x7f79e00cf970 2024-02-19T15:23:41.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.119+0000 7f79dfe00640 1 --2- 172.21.15.47:0/1951126857 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f79bc066e40 0x7f79bc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:41.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.119+0000 7f79dfe00640 1 --2- 172.21.15.47:0/1951126857 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f79bc066e40 0x7f79bc069300 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f79d00aa000 tx=0x7f79d0001900 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:41.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.123+0000 7f79dcc00640 1 -- 172.21.15.47:0/1951126857 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f79d8050ac0 con 0x7f79e00cf970 2024-02-19T15:23:41.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.287+0000 7f79e7200640 1 -- 172.21.15.47:0/1951126857 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f79e00cbdd0 con 0x7f79e00cf970 2024-02-19T15:23:41.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.931+0000 7f79dcc00640 1 -- 172.21.15.47:0/1951126857 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v43) v1 ==== 170+0+0 (secure 0 0 0) 0x7f79d80190d0 con 0x7f79e00cf970 2024-02-19T15:23:41.939 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_data' created 2024-02-19T15:23:41.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.935+0000 7f79e7200640 1 -- 172.21.15.47:0/1951126857 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f79bc066e40 msgr2=0x7f79bc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:41.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.935+0000 7f79e7200640 1 --2- 172.21.15.47:0/1951126857 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f79bc066e40 0x7f79bc069300 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f79d00aa000 tx=0x7f79d0001900 comp rx=0 tx=0).stop 2024-02-19T15:23:41.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.935+0000 7f79e7200640 1 -- 172.21.15.47:0/1951126857 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79e00cf970 msgr2=0x7f79e00cfdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:41.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.935+0000 7f79e7200640 1 --2- 172.21.15.47:0/1951126857 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79e00cf970 0x7f79e00cfdf0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f79d8001340 tx=0x7f79d800eb20 comp rx=0 tx=0).stop 2024-02-19T15:23:41.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.935+0000 7f79e7200640 1 -- 172.21.15.47:0/1951126857 shutdown_connections 2024-02-19T15:23:41.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.935+0000 7f79e7200640 1 --2- 172.21.15.47:0/1951126857 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f79bc066e40 0x7f79bc069300 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:41.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.935+0000 7f79e7200640 1 --2- 172.21.15.47:0/1951126857 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79e00cf970 0x7f79e00cfdf0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:41.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.935+0000 7f79e7200640 1 --2- 172.21.15.47:0/1951126857 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e0154460 0x7f79e0150620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:41.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.935+0000 7f79e7200640 1 --2- 172.21.15.47:0/1951126857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c4003b40 0x7f79e00ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:41.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.935+0000 7f79e7200640 1 -- 172.21.15.47:0/1951126857 >> 172.21.15.47:0/1951126857 conn(0x7f79e00ccc80 msgr2=0x7f79e006da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:41.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.939+0000 7f79e7200640 1 -- 172.21.15.47:0/1951126857 shutdown_connections 2024-02-19T15:23:41.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:41.939+0000 7f79e7200640 1 -- 172.21.15.47:0/1951126857 wait complete. 2024-02-19T15:23:41.980 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T15:23:42.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.123+0000 7f1d8ea00640 1 Processor -- start 2024-02-19T15:23:42.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.123+0000 7f1d8ea00640 1 -- start start 2024-02-19T15:23:42.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.123+0000 7f1d8ea00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1d8815bc60 0x7f1d880ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:42.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.123+0000 7f1d8ea00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d880b0db0 con 0x7f1d880d2480 2024-02-19T15:23:42.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.123+0000 7f1d8ea00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d880cf9e0 con 0x7f1d880d0650 2024-02-19T15:23:42.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.123+0000 7f1d8ea00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d880b0fe0 con 0x7f1d8815bc60 2024-02-19T15:23:42.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.123+0000 7f1d8d600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1d8815bc60 0x7f1d880ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:42.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.123+0000 7f1d8cc00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1d880d2480 0x7f1d880cff40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52548/0 (socket says 172.21.15.47:52548) 2024-02-19T15:23:42.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.123+0000 7f1d8cc00640 1 -- 172.21.15.47:0/2316096075 learned_addr learned my addr 172.21.15.47:0/2316096075 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:42.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.123+0000 7f1d8d600640 1 -- 172.21.15.47:0/2316096075 >> v1:172.21.15.116:6789/0 conn(0x7f1d880d0650 legacy=0x7f1d880b1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:42.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.123+0000 7f1d8d600640 1 -- 172.21.15.47:0/2316096075 >> v1:172.21.15.47:6789/0 conn(0x7f1d880d2480 legacy=0x7f1d880cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:42.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.123+0000 7f1d8d600640 1 -- 172.21.15.47:0/2316096075 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d880b12c0 con 0x7f1d8815bc60 2024-02-19T15:23:42.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.123+0000 7f1d87e00640 1 -- 172.21.15.47:0/2316096075 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526980672 0 0) 0x7f1d880b0db0 con 0x7f1d880d2480 2024-02-19T15:23:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.127+0000 7f1d8d600640 1 --2- 172.21.15.47:0/2316096075 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d8815bc60 0x7f1d880ce9a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0097c0 tx=0x7f1d7c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5258debb47acf81 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.127+0000 7f1d87e00640 1 -- 172.21.15.47:0/2316096075 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c0ac640 con 0x7f1d8815bc60 2024-02-19T15:23:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.127+0000 7f1d87e00640 1 -- 172.21.15.47:0/2316096075 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d7c0ac7a0 con 0x7f1d8815bc60 2024-02-19T15:23:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.127+0000 7f1d87e00640 1 -- 172.21.15.47:0/2316096075 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c0acb30 con 0x7f1d8815bc60 2024-02-19T15:23:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.127+0000 7f1d8ea00640 1 -- 172.21.15.47:0/2316096075 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d8815bc60 msgr2=0x7f1d880ce9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.127+0000 7f1d8ea00640 1 --2- 172.21.15.47:0/2316096075 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d8815bc60 0x7f1d880ce9a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0097c0 tx=0x7f1d7c01c3e0 comp rx=0 tx=0).stop 2024-02-19T15:23:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.127+0000 7f1d8ea00640 1 -- 172.21.15.47:0/2316096075 shutdown_connections 2024-02-19T15:23:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.127+0000 7f1d8ea00640 1 --2- 172.21.15.47:0/2316096075 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d8815bc60 0x7f1d880ce9a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.127+0000 7f1d8ea00640 1 -- 172.21.15.47:0/2316096075 >> 172.21.15.47:0/2316096075 conn(0x7f1d880d1960 msgr2=0x7f1d880d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.127+0000 7f1d8ea00640 1 -- 172.21.15.47:0/2316096075 shutdown_connections 2024-02-19T15:23:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.127+0000 7f1d8ea00640 1 -- 172.21.15.47:0/2316096075 wait complete. 2024-02-19T15:23:42.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.127+0000 7f1d8ea00640 1 Processor -- start 2024-02-19T15:23:42.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.127+0000 7f1d8ea00640 1 -- start start 2024-02-19T15:23:42.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d8ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d880d0650 0x7f1d880ca860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:42.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d8ea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d880d2480 0x7f1d880c7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:42.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d8ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d8815bc60 0x7f1d8815a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:42.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d8ea00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d880cb120 con 0x7f1d880d0650 2024-02-19T15:23:42.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d8ea00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d880c55c0 con 0x7f1d8815bc60 2024-02-19T15:23:42.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d8ea00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d880c58a0 con 0x7f1d880d2480 2024-02-19T15:23:42.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d8d600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d880d0650 0x7f1d880ca860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:42.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d8d600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d880d0650 0x7f1d880ca860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52470/0 (socket says 172.21.15.47:52470) 2024-02-19T15:23:42.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d8d600640 1 -- 172.21.15.47:0/3641680418 learned_addr learned my addr 172.21.15.47:0/3641680418 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:42.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d8cc00640 1 --2- 172.21.15.47:0/3641680418 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d880d2480 0x7f1d880c7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:42.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d8e000640 1 --2- 172.21.15.47:0/3641680418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d8815bc60 0x7f1d8815a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:42.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d8d600640 1 -- 172.21.15.47:0/3641680418 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d880d2480 msgr2=0x7f1d880c7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:42.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d8d600640 1 --2- 172.21.15.47:0/3641680418 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d880d2480 0x7f1d880c7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:42.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d8d600640 1 -- 172.21.15.47:0/3641680418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d8815bc60 msgr2=0x7f1d8815a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:42.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d8d600640 1 --2- 172.21.15.47:0/3641680418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d8815bc60 0x7f1d8815a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:42.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d8d600640 1 -- 172.21.15.47:0/3641680418 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c009420 con 0x7f1d880d0650 2024-02-19T15:23:42.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d8d600640 1 --2- 172.21.15.47:0/3641680418 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d880d0650 0x7f1d880ca860 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c01a010 tx=0x7f1d7c01ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:42.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d85600640 1 -- 172.21.15.47:0/3641680418 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c0b7830 con 0x7f1d880d0650 2024-02-19T15:23:42.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d85600640 1 -- 172.21.15.47:0/3641680418 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d7c0b7990 con 0x7f1d880d0650 2024-02-19T15:23:42.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d8ea00640 1 -- 172.21.15.47:0/3641680418 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d880c5b80 con 0x7f1d880d0650 2024-02-19T15:23:42.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d85600640 1 -- 172.21.15.47:0/3641680418 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c0b7d40 con 0x7f1d880d0650 2024-02-19T15:23:42.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d8ea00640 1 -- 172.21.15.47:0/3641680418 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d880cada0 con 0x7f1d880d0650 2024-02-19T15:23:42.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.131+0000 7f1d8ea00640 1 -- 172.21.15.47:0/3641680418 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d880b77e0 con 0x7f1d880d0650 2024-02-19T15:23:42.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.135+0000 7f1d85600640 1 -- 172.21.15.47:0/3641680418 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1d7c0b3040 con 0x7f1d880d0650 2024-02-19T15:23:42.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.135+0000 7f1d85600640 1 --2- 172.21.15.47:0/3641680418 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d68066e40 0x7f1d68069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:42.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.135+0000 7f1d85600640 1 -- 172.21.15.47:0/3641680418 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f1d7c11f340 con 0x7f1d880d0650 2024-02-19T15:23:42.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.135+0000 7f1d8cc00640 1 --2- 172.21.15.47:0/3641680418 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d68066e40 0x7f1d68069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:42.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.135+0000 7f1d8cc00640 1 --2- 172.21.15.47:0/3641680418 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d68066e40 0x7f1d68069300 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1d881f9a90 tx=0x7f1d74007ba0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:42.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.139+0000 7f1d85600640 1 -- 172.21.15.47:0/3641680418 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1d7c0ec180 con 0x7f1d880d0650 2024-02-19T15:23:42.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.335+0000 7f1d8ea00640 1 -- 172.21.15.47:0/3641680418 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1d8804f260 con 0x7f1d880d0650 2024-02-19T15:23:42.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.935+0000 7f1d85600640 1 -- 172.21.15.47:0/3641680418 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v21) v1 ==== 172+0+0 (secure 0 0 0) 0x7f1d7c0ef840 con 0x7f1d880d0650 2024-02-19T15:23:42.941 INFO:teuthology.orchestra.run.smithi047.stderr:new fs with metadata pool 6 and data pool 7 2024-02-19T15:23:42.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.939+0000 7f1d8ea00640 1 -- 172.21.15.47:0/3641680418 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d68066e40 msgr2=0x7f1d68069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:42.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.939+0000 7f1d8ea00640 1 --2- 172.21.15.47:0/3641680418 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d68066e40 0x7f1d68069300 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1d881f9a90 tx=0x7f1d74007ba0 comp rx=0 tx=0).stop 2024-02-19T15:23:42.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.939+0000 7f1d8ea00640 1 -- 172.21.15.47:0/3641680418 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d880d0650 msgr2=0x7f1d880ca860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:42.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.939+0000 7f1d8ea00640 1 --2- 172.21.15.47:0/3641680418 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d880d0650 0x7f1d880ca860 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c01a010 tx=0x7f1d7c01ac20 comp rx=0 tx=0).stop 2024-02-19T15:23:42.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.939+0000 7f1d8ea00640 1 -- 172.21.15.47:0/3641680418 shutdown_connections 2024-02-19T15:23:42.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.939+0000 7f1d8ea00640 1 --2- 172.21.15.47:0/3641680418 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d68066e40 0x7f1d68069300 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:42.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.939+0000 7f1d8ea00640 1 --2- 172.21.15.47:0/3641680418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d8815bc60 0x7f1d8815a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:42.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.939+0000 7f1d8ea00640 1 --2- 172.21.15.47:0/3641680418 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d880d2480 0x7f1d880c7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:42.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.939+0000 7f1d8ea00640 1 --2- 172.21.15.47:0/3641680418 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d880d0650 0x7f1d880ca860 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:42.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.939+0000 7f1d8ea00640 1 -- 172.21.15.47:0/3641680418 >> 172.21.15.47:0/3641680418 conn(0x7f1d880d1960 msgr2=0x7f1d880b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:42.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.939+0000 7f1d8ea00640 1 -- 172.21.15.47:0/3641680418 shutdown_connections 2024-02-19T15:23:42.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:42.939+0000 7f1d8ea00640 1 -- 172.21.15.47:0/3641680418 wait complete. 2024-02-19T15:23:42.999 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:23:43.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.135+0000 7f0d61800640 1 Processor -- start 2024-02-19T15:23:43.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.135+0000 7f0d61800640 1 -- start start 2024-02-19T15:23:43.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.135+0000 7f0d61800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0d5c0cedb0 0x7f0d5c0cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:43.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.135+0000 7f0d61800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0d5c0d2710 0x7f0d5c0d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:43.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.135+0000 7f0d61800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d5c0cc700 con 0x7f0d5c15bcd0 2024-02-19T15:23:43.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.135+0000 7f0d61800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d5c0cf5e0 con 0x7f0d5c0d2710 2024-02-19T15:23:43.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.135+0000 7f0d61800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d5c0cdb90 con 0x7f0d5c0cedb0 2024-02-19T15:23:43.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.135+0000 7f0d5b400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0d5c0d2710 0x7f0d5c0d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:43.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.135+0000 7f0d5b400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0d5c0d2710 0x7f0d5c0d0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52478/0 (socket says 172.21.15.47:52478) 2024-02-19T15:23:43.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.135+0000 7f0d5b400640 1 -- 172.21.15.47:0/2064053644 learned_addr learned my addr 172.21.15.47:0/2064053644 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:43.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.135+0000 7f0d5aa00640 1 --1- 172.21.15.47:0/2064053644 >> v1:172.21.15.47:6789/0 conn(0x7f0d5c15bcd0 0x7f0d5c0cd480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52554/0 (socket says 172.21.15.47:52554) 2024-02-19T15:23:43.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.135+0000 7f0d52000640 1 --2- 172.21.15.47:0/2064053644 >> v2:172.21.15.116:3300/0 conn(0x7f0d5c0cedb0 0x7f0d5c0cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:43.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.135+0000 7f0d5b400640 1 -- 172.21.15.47:0/2064053644 >> v2:172.21.15.116:3300/0 conn(0x7f0d5c0cedb0 msgr2=0x7f0d5c0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:43.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.135+0000 7f0d5b400640 1 --2- 172.21.15.47:0/2064053644 >> v2:172.21.15.116:3300/0 conn(0x7f0d5c0cedb0 0x7f0d5c0cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.135+0000 7f0d5b400640 1 -- 172.21.15.47:0/2064053644 >> v1:172.21.15.47:6789/0 conn(0x7f0d5c15bcd0 legacy=0x7f0d5c0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:43.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.135+0000 7f0d5b400640 1 -- 172.21.15.47:0/2064053644 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5c0cf300 con 0x7f0d5c0d2710 2024-02-19T15:23:43.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.139+0000 7f0d5b400640 1 --2- 172.21.15.47:0/2064053644 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d5c0d2710 0x7f0d5c0d0190 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c0ce100 tx=0x7f0d5401c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b043c0e61f4e367a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:43.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.139+0000 7f0d5a000640 1 -- 172.21.15.47:0/2064053644 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d540a5880 con 0x7f0d5c0d2710 2024-02-19T15:23:43.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.139+0000 7f0d5a000640 1 -- 172.21.15.47:0/2064053644 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d540a59e0 con 0x7f0d5c0d2710 2024-02-19T15:23:43.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.139+0000 7f0d5a000640 1 -- 172.21.15.47:0/2064053644 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d540a5d70 con 0x7f0d5c0d2710 2024-02-19T15:23:43.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.139+0000 7f0d61800640 1 -- 172.21.15.47:0/2064053644 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d5c0d2710 msgr2=0x7f0d5c0d0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:43.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.139+0000 7f0d61800640 1 --2- 172.21.15.47:0/2064053644 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d5c0d2710 0x7f0d5c0d0190 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c0ce100 tx=0x7f0d5401c9c0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.139+0000 7f0d61800640 1 -- 172.21.15.47:0/2064053644 shutdown_connections 2024-02-19T15:23:43.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.139+0000 7f0d61800640 1 --2- 172.21.15.47:0/2064053644 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d5c0d2710 0x7f0d5c0d0190 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.139+0000 7f0d61800640 1 --2- 172.21.15.47:0/2064053644 >> v2:172.21.15.116:3300/0 conn(0x7f0d5c0cedb0 0x7f0d5c0cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.139+0000 7f0d61800640 1 -- 172.21.15.47:0/2064053644 >> 172.21.15.47:0/2064053644 conn(0x7f0d5c0d1bf0 msgr2=0x7f0d5c0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:43.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.139+0000 7f0d61800640 1 -- 172.21.15.47:0/2064053644 shutdown_connections 2024-02-19T15:23:43.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.139+0000 7f0d61800640 1 -- 172.21.15.47:0/2064053644 wait complete. 2024-02-19T15:23:43.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d61800640 1 Processor -- start 2024-02-19T15:23:43.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d61800640 1 -- start start 2024-02-19T15:23:43.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d61800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d5c0cedb0 0x7f0d5c157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:43.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d61800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d5c0d2710 0x7f0d5c06d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:43.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d61800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d5c15bcd0 0x7f0d5c06fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:43.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d61800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c151bf0 con 0x7f0d5c0d2710 2024-02-19T15:23:43.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d61800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c157710 con 0x7f0d5c0cedb0 2024-02-19T15:23:43.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d61800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c06dd00 con 0x7f0d5c15bcd0 2024-02-19T15:23:43.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d52000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d5c0d2710 0x7f0d5c06d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:43.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d52000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d5c0d2710 0x7f0d5c06d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52486/0 (socket says 172.21.15.47:52486) 2024-02-19T15:23:43.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d52000640 1 -- 172.21.15.47:0/22765044 learned_addr learned my addr 172.21.15.47:0/22765044 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:43.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d5aa00640 1 --2- 172.21.15.47:0/22765044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d5c0cedb0 0x7f0d5c157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:43.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d5b400640 1 --2- 172.21.15.47:0/22765044 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d5c15bcd0 0x7f0d5c06fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:43.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d52000640 1 -- 172.21.15.47:0/22765044 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d5c15bcd0 msgr2=0x7f0d5c06fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:43.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d52000640 1 --2- 172.21.15.47:0/22765044 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d5c15bcd0 0x7f0d5c06fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d52000640 1 -- 172.21.15.47:0/22765044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d5c0cedb0 msgr2=0x7f0d5c157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:43.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d52000640 1 --2- 172.21.15.47:0/22765044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d5c0cedb0 0x7f0d5c157ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d52000640 1 -- 172.21.15.47:0/22765044 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d54009b30 con 0x7f0d5c0d2710 2024-02-19T15:23:43.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d5b400640 1 --2- 172.21.15.47:0/22765044 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d5c15bcd0 0x7f0d5c06fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:43.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d5aa00640 1 --2- 172.21.15.47:0/22765044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d5c0cedb0 0x7f0d5c157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:43.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d52000640 1 --2- 172.21.15.47:0/22765044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d5c0d2710 0x7f0d5c06d7c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400d5f0 tx=0x7f0d4400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:43.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d53400640 1 -- 172.21.15.47:0/22765044 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d440125f0 con 0x7f0d5c0d2710 2024-02-19T15:23:43.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d53400640 1 -- 172.21.15.47:0/22765044 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d44012750 con 0x7f0d5c0d2710 2024-02-19T15:23:43.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d61800640 1 -- 172.21.15.47:0/22765044 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d5c15d240 con 0x7f0d5c0d2710 2024-02-19T15:23:43.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d53400640 1 -- 172.21.15.47:0/22765044 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44012ae0 con 0x7f0d5c0d2710 2024-02-19T15:23:43.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d61800640 1 -- 172.21.15.47:0/22765044 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5c1570c0 con 0x7f0d5c0d2710 2024-02-19T15:23:43.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.143+0000 7f0d61800640 1 -- 172.21.15.47:0/22765044 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d5c075360 con 0x7f0d5c0d2710 2024-02-19T15:23:43.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.147+0000 7f0d53400640 1 -- 172.21.15.47:0/22765044 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0d44012c40 con 0x7f0d5c0d2710 2024-02-19T15:23:43.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.147+0000 7f0d53400640 1 --2- 172.21.15.47:0/22765044 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d38066e40 0x7f0d38069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:43.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.147+0000 7f0d53400640 1 -- 172.21.15.47:0/22765044 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0d44084fe0 con 0x7f0d5c0d2710 2024-02-19T15:23:43.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.147+0000 7f0d5aa00640 1 --2- 172.21.15.47:0/22765044 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d38066e40 0x7f0d38069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:43.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.147+0000 7f0d5aa00640 1 --2- 172.21.15.47:0/22765044 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d38066e40 0x7f0d38069300 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c000f70 tx=0x7f0d4c007d70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:43.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.151+0000 7f0d53400640 1 -- 172.21.15.47:0/22765044 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0d44051d30 con 0x7f0d5c0d2710 2024-02-19T15:23:43.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.315+0000 7f0d61800640 1 -- 172.21.15.47:0/22765044 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d5c0b55e0 con 0x7f0d5c0d2710 2024-02-19T15:23:43.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.315+0000 7f0d53400640 1 -- 172.21.15.47:0/22765044 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16638 (secure 0 0 0) 0x7f0d440553f0 con 0x7f0d5c0d2710 2024-02-19T15:23:43.321 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:43.321 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":44,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:23:42.342304+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:23:40.751741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-02-19T15:23:41.338930+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:23:43.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.319+0000 7f0d61800640 1 -- 172.21.15.47:0/22765044 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d38066e40 msgr2=0x7f0d38069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:43.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.319+0000 7f0d61800640 1 --2- 172.21.15.47:0/22765044 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d38066e40 0x7f0d38069300 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c000f70 tx=0x7f0d4c007d70 comp rx=0 tx=0).stop 2024-02-19T15:23:43.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.319+0000 7f0d61800640 1 -- 172.21.15.47:0/22765044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d5c0d2710 msgr2=0x7f0d5c06d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:43.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.319+0000 7f0d61800640 1 --2- 172.21.15.47:0/22765044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d5c0d2710 0x7f0d5c06d7c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400d5f0 tx=0x7f0d4400dab0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.319+0000 7f0d61800640 1 -- 172.21.15.47:0/22765044 shutdown_connections 2024-02-19T15:23:43.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.319+0000 7f0d61800640 1 --2- 172.21.15.47:0/22765044 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d38066e40 0x7f0d38069300 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.319+0000 7f0d61800640 1 --2- 172.21.15.47:0/22765044 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d5c15bcd0 0x7f0d5c06fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.319+0000 7f0d61800640 1 --2- 172.21.15.47:0/22765044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d5c0d2710 0x7f0d5c06d7c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.319+0000 7f0d61800640 1 --2- 172.21.15.47:0/22765044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d5c0cedb0 0x7f0d5c157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.319+0000 7f0d61800640 1 -- 172.21.15.47:0/22765044 >> 172.21.15.47:0/22765044 conn(0x7f0d5c0d1bf0 msgr2=0x7f0d5c0b4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:43.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.319+0000 7f0d61800640 1 -- 172.21.15.47:0/22765044 shutdown_connections 2024-02-19T15:23:43.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.319+0000 7f0d61800640 1 -- 172.21.15.47:0/22765044 wait complete. 2024-02-19T15:23:43.344 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:23:43.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.479+0000 7f1abc200640 1 Processor -- start 2024-02-19T15:23:43.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.479+0000 7f1abc200640 1 -- start start 2024-02-19T15:23:43.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.479+0000 7f1abc200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1ab415b1a0 0x7f1ab415b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:43.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.479+0000 7f1abc200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1ab4071350 0x7f1ab415bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:43.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.479+0000 7f1abc200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab415c000 con 0x7f1ab406c9a0 2024-02-19T15:23:43.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.479+0000 7f1abc200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab40718c0 con 0x7f1ab415b1a0 2024-02-19T15:23:43.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.479+0000 7f1abc200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab4071ba0 con 0x7f1ab4071350 2024-02-19T15:23:43.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.479+0000 7f1ab9800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1ab415b1a0 0x7f1ab415b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:43.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.479+0000 7f1aba200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1ab406c9a0 0x7f1ab415c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49106/0 (socket says 172.21.15.47:49106) 2024-02-19T15:23:43.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.479+0000 7f1aba200640 1 -- 172.21.15.47:0/1123169396 learned_addr learned my addr 172.21.15.47:0/1123169396 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:43.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.479+0000 7f1ab8e00640 1 --2- 172.21.15.47:0/1123169396 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1ab4071350 0x7f1ab415bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:43.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.479+0000 7f1ab9800640 1 -- 172.21.15.47:0/1123169396 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1ab4071350 msgr2=0x7f1ab415bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:43.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.479+0000 7f1ab9800640 1 --2- 172.21.15.47:0/1123169396 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1ab4071350 0x7f1ab415bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.479+0000 7f1ab9800640 1 -- 172.21.15.47:0/1123169396 >> v1:172.21.15.47:6789/0 conn(0x7f1ab406c9a0 legacy=0x7f1ab415c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:43.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.479+0000 7f1ab9800640 1 -- 172.21.15.47:0/1123169396 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab4071e80 con 0x7f1ab415b1a0 2024-02-19T15:23:43.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.479+0000 7f1ab3e00640 1 -- 172.21.15.47:0/1123169396 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1077492162 0 0) 0x7f1ab415c000 con 0x7f1ab406c9a0 2024-02-19T15:23:43.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.483+0000 7f1ab9800640 1 --2- 172.21.15.47:0/1123169396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ab415b1a0 0x7f1ab415b580 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1aa80097c0 tx=0x7f1aa801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=293b652dcff2353e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:43.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.483+0000 7f1ab3e00640 1 -- 172.21.15.47:0/1123169396 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa80aa5f0 con 0x7f1ab415b1a0 2024-02-19T15:23:43.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.483+0000 7f1ab3e00640 1 -- 172.21.15.47:0/1123169396 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa80aa750 con 0x7f1ab415b1a0 2024-02-19T15:23:43.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.483+0000 7f1ab3e00640 1 -- 172.21.15.47:0/1123169396 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa80aaae0 con 0x7f1ab415b1a0 2024-02-19T15:23:43.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.483+0000 7f1abc200640 1 -- 172.21.15.47:0/1123169396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ab415b1a0 msgr2=0x7f1ab415b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:43.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.483+0000 7f1abc200640 1 --2- 172.21.15.47:0/1123169396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ab415b1a0 0x7f1ab415b580 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1aa80097c0 tx=0x7f1aa801c470 comp rx=0 tx=0).stop 2024-02-19T15:23:43.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.483+0000 7f1abc200640 1 -- 172.21.15.47:0/1123169396 shutdown_connections 2024-02-19T15:23:43.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.483+0000 7f1abc200640 1 --2- 172.21.15.47:0/1123169396 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1ab4071350 0x7f1ab415bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.483+0000 7f1abc200640 1 --2- 172.21.15.47:0/1123169396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ab415b1a0 0x7f1ab415b580 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.483+0000 7f1abc200640 1 -- 172.21.15.47:0/1123169396 >> 172.21.15.47:0/1123169396 conn(0x7f1ab40b16a0 msgr2=0x7f1ab40b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:43.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.483+0000 7f1abc200640 1 -- 172.21.15.47:0/1123169396 shutdown_connections 2024-02-19T15:23:43.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.483+0000 7f1abc200640 1 -- 172.21.15.47:0/1123169396 wait complete. 2024-02-19T15:23:43.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.483+0000 7f1abc200640 1 Processor -- start 2024-02-19T15:23:43.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.483+0000 7f1abc200640 1 -- start start 2024-02-19T15:23:43.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1abc200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ab406c9a0 0x7f1ab413db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:43.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1abc200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1ab4071350 0x7f1ab40cf000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:43.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1abc200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ab415b1a0 0x7f1ab4158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:43.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1abc200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab4158640 con 0x7f1ab415b1a0 2024-02-19T15:23:43.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1abc200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab40b2d00 con 0x7f1ab406c9a0 2024-02-19T15:23:43.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1abc200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab413e120 con 0x7f1ab4071350 2024-02-19T15:23:43.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1aba200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ab415b1a0 0x7f1ab4158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:43.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1aba200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ab415b1a0 0x7f1ab4158100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33282/0 (socket says 172.21.15.47:33282) 2024-02-19T15:23:43.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1aba200640 1 -- 172.21.15.47:0/2268196466 learned_addr learned my addr 172.21.15.47:0/2268196466 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:43.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1ab8e00640 1 --2- 172.21.15.47:0/2268196466 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1ab4071350 0x7f1ab40cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:43.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1ab9800640 1 --2- 172.21.15.47:0/2268196466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ab406c9a0 0x7f1ab413db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:43.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1aba200640 1 -- 172.21.15.47:0/2268196466 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1ab4071350 msgr2=0x7f1ab40cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:43.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1aba200640 1 --2- 172.21.15.47:0/2268196466 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1ab4071350 0x7f1ab40cf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1aba200640 1 -- 172.21.15.47:0/2268196466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ab406c9a0 msgr2=0x7f1ab413db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:43.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1aba200640 1 --2- 172.21.15.47:0/2268196466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ab406c9a0 0x7f1ab413db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1aba200640 1 -- 172.21.15.47:0/2268196466 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa8009420 con 0x7f1ab415b1a0 2024-02-19T15:23:43.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1ab8e00640 1 --2- 172.21.15.47:0/2268196466 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1ab4071350 0x7f1ab40cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:43.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1aba200640 1 --2- 172.21.15.47:0/2268196466 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ab415b1a0 0x7f1ab4158100 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1aac00c730 tx=0x7f1aac00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:43.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1ab1600640 1 -- 172.21.15.47:0/2268196466 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac00ce40 con 0x7f1ab415b1a0 2024-02-19T15:23:43.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1ab1600640 1 -- 172.21.15.47:0/2268196466 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aac00a640 con 0x7f1ab415b1a0 2024-02-19T15:23:43.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1ab1600640 1 -- 172.21.15.47:0/2268196466 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac00aa30 con 0x7f1ab415b1a0 2024-02-19T15:23:43.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1abc200640 1 -- 172.21.15.47:0/2268196466 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab413e400 con 0x7f1ab415b1a0 2024-02-19T15:23:43.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1abc200640 1 -- 172.21.15.47:0/2268196466 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab413e9c0 con 0x7f1ab415b1a0 2024-02-19T15:23:43.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.487+0000 7f1abc200640 1 -- 172.21.15.47:0/2268196466 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab40b2ed0 con 0x7f1ab415b1a0 2024-02-19T15:23:43.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.491+0000 7f1ab1600640 1 -- 172.21.15.47:0/2268196466 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1aac00ab90 con 0x7f1ab415b1a0 2024-02-19T15:23:43.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.491+0000 7f1ab1600640 1 --2- 172.21.15.47:0/2268196466 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1a94066e40 0x7f1a94069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:43.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.491+0000 7f1ab1600640 1 -- 172.21.15.47:0/2268196466 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1aac084150 con 0x7f1ab415b1a0 2024-02-19T15:23:43.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.491+0000 7f1ab9800640 1 --2- 172.21.15.47:0/2268196466 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1a94066e40 0x7f1a94069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:43.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.491+0000 7f1ab9800640 1 --2- 172.21.15.47:0/2268196466 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1a94066e40 0x7f1a94069300 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1aa80ae000 tx=0x7f1aa80a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:43.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.495+0000 7f1ab1600640 1 -- 172.21.15.47:0/2268196466 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1aac050f50 con 0x7f1ab415b1a0 2024-02-19T15:23:43.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.659+0000 7f1abc200640 1 -- 172.21.15.47:0/2268196466 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ab4075360 con 0x7f1ab415b1a0 2024-02-19T15:23:43.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.659+0000 7f1ab1600640 1 -- 172.21.15.47:0/2268196466 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16638 (secure 0 0 0) 0x7f1aac054610 con 0x7f1ab415b1a0 2024-02-19T15:23:43.665 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:43.665 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":44,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:23:42.342304+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:23:40.751741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-02-19T15:23:41.338930+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:23:43.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.659+0000 7f1abc200640 1 -- 172.21.15.47:0/2268196466 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1a94066e40 msgr2=0x7f1a94069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:43.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.659+0000 7f1abc200640 1 --2- 172.21.15.47:0/2268196466 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1a94066e40 0x7f1a94069300 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1aa80ae000 tx=0x7f1aa80a7000 comp rx=0 tx=0).stop 2024-02-19T15:23:43.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.659+0000 7f1abc200640 1 -- 172.21.15.47:0/2268196466 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ab415b1a0 msgr2=0x7f1ab4158100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:43.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.659+0000 7f1abc200640 1 --2- 172.21.15.47:0/2268196466 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ab415b1a0 0x7f1ab4158100 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1aac00c730 tx=0x7f1aac00cbf0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.663+0000 7f1abc200640 1 -- 172.21.15.47:0/2268196466 shutdown_connections 2024-02-19T15:23:43.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.663+0000 7f1abc200640 1 --2- 172.21.15.47:0/2268196466 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1a94066e40 0x7f1a94069300 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.663+0000 7f1abc200640 1 --2- 172.21.15.47:0/2268196466 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ab415b1a0 0x7f1ab4158100 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.663+0000 7f1abc200640 1 --2- 172.21.15.47:0/2268196466 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1ab4071350 0x7f1ab40cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.663+0000 7f1abc200640 1 --2- 172.21.15.47:0/2268196466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ab406c9a0 0x7f1ab413db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.663+0000 7f1abc200640 1 -- 172.21.15.47:0/2268196466 >> 172.21.15.47:0/2268196466 conn(0x7f1ab40b16a0 msgr2=0x7f1ab415db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:43.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.663+0000 7f1abc200640 1 -- 172.21.15.47:0/2268196466 shutdown_connections 2024-02-19T15:23:43.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.663+0000 7f1abc200640 1 -- 172.21.15.47:0/2268196466 wait complete. 2024-02-19T15:23:43.686 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T15:23:43.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.823+0000 7fb971000640 1 Processor -- start 2024-02-19T15:23:43.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.823+0000 7fb971000640 1 -- start start 2024-02-19T15:23:43.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.827+0000 7fb971000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb96c0b12f0 0x7fb96c0b1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:43.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.827+0000 7fb971000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb96c0b1c90 con 0x7fb96c15f510 2024-02-19T15:23:43.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.827+0000 7fb971000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb96c0af850 con 0x7fb96c0b1f20 2024-02-19T15:23:43.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.827+0000 7fb971000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb96c0afb30 con 0x7fb96c0b12f0 2024-02-19T15:23:43.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.827+0000 7fb96b400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb96c15f510 0x7fb96c0b0850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49112/0 (socket says 172.21.15.47:49112) 2024-02-19T15:23:43.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.827+0000 7fb96b400640 1 -- 172.21.15.47:0/655581521 learned_addr learned my addr 172.21.15.47:0/655581521 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:43.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.827+0000 7fb962a00640 1 --2- 172.21.15.47:0/655581521 >> v2:172.21.15.116:3300/0 conn(0x7fb96c0b12f0 0x7fb96c0b1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:43.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.827+0000 7fb96aa00640 1 -- 172.21.15.47:0/655581521 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429718813 0 0) 0x7fb96c0b1c90 con 0x7fb96c15f510 2024-02-19T15:23:43.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.827+0000 7fb96aa00640 1 -- 172.21.15.47:0/655581521 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb950003530 con 0x7fb96c15f510 2024-02-19T15:23:43.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.827+0000 7fb96aa00640 1 -- 172.21.15.47:0/655581521 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1956065811 0 0) 0x7fb950003530 con 0x7fb96c15f510 2024-02-19T15:23:43.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.827+0000 7fb962a00640 1 -- 172.21.15.47:0/655581521 >> v1:172.21.15.116:6789/0 conn(0x7fb96c0b1f20 legacy=0x7fb96c0b2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:43.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.827+0000 7fb962a00640 1 -- 172.21.15.47:0/655581521 >> v1:172.21.15.47:6789/0 conn(0x7fb96c15f510 legacy=0x7fb96c0b0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:43.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.827+0000 7fb962a00640 1 -- 172.21.15.47:0/655581521 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb96c0afe10 con 0x7fb96c0b12f0 2024-02-19T15:23:43.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.831+0000 7fb962a00640 1 --2- 172.21.15.47:0/655581521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb96c0b12f0 0x7fb96c0b1750 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb9540097c0 tx=0x7fb95401c270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d54d6cc6470e702a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:43.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.831+0000 7fb96aa00640 1 -- 172.21.15.47:0/655581521 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95c001680 con 0x7fb96c0b12f0 2024-02-19T15:23:43.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.831+0000 7fb96aa00640 1 -- 172.21.15.47:0/655581521 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9540a7050 con 0x7fb96c0b12f0 2024-02-19T15:23:43.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.831+0000 7fb96aa00640 1 -- 172.21.15.47:0/655581521 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9540ab620 con 0x7fb96c0b12f0 2024-02-19T15:23:43.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.831+0000 7fb971000640 1 -- 172.21.15.47:0/655581521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb96c0b12f0 msgr2=0x7fb96c0b1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:43.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.831+0000 7fb971000640 1 --2- 172.21.15.47:0/655581521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb96c0b12f0 0x7fb96c0b1750 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb9540097c0 tx=0x7fb95401c270 comp rx=0 tx=0).stop 2024-02-19T15:23:43.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.831+0000 7fb971000640 1 -- 172.21.15.47:0/655581521 shutdown_connections 2024-02-19T15:23:43.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.831+0000 7fb971000640 1 --2- 172.21.15.47:0/655581521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb96c0b12f0 0x7fb96c0b1750 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.831+0000 7fb971000640 1 -- 172.21.15.47:0/655581521 >> 172.21.15.47:0/655581521 conn(0x7fb96c0d19f0 msgr2=0x7fb96c0d1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:43.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.831+0000 7fb971000640 1 -- 172.21.15.47:0/655581521 shutdown_connections 2024-02-19T15:23:43.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.831+0000 7fb971000640 1 -- 172.21.15.47:0/655581521 wait complete. 2024-02-19T15:23:43.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.831+0000 7fb971000640 1 Processor -- start 2024-02-19T15:23:43.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.831+0000 7fb971000640 1 -- start start 2024-02-19T15:23:43.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb971000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb96c0b12f0 0x7fb96c1533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:43.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb971000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb96c0b1f20 0x7fb96c167200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:43.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb971000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb96c15f510 0x7fb96c06cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:43.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb971000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c0c8ac0 con 0x7fb96c0b12f0 2024-02-19T15:23:43.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb971000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c06db30 con 0x7fb96c15f510 2024-02-19T15:23:43.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb971000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c06de10 con 0x7fb96c0b1f20 2024-02-19T15:23:43.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb96b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb96c0b12f0 0x7fb96c1533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:43.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb96b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb96c0b12f0 0x7fb96c1533e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33290/0 (socket says 172.21.15.47:33290) 2024-02-19T15:23:43.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb96b400640 1 -- 172.21.15.47:0/1909336945 learned_addr learned my addr 172.21.15.47:0/1909336945 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:43.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb962a00640 1 --2- 172.21.15.47:0/1909336945 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb96c0b1f20 0x7fb96c167200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:43.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb96be00640 1 --2- 172.21.15.47:0/1909336945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb96c15f510 0x7fb96c06cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:43.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb96b400640 1 -- 172.21.15.47:0/1909336945 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb96c0b1f20 msgr2=0x7fb96c167200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:43.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb96b400640 1 --2- 172.21.15.47:0/1909336945 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb96c0b1f20 0x7fb96c167200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb96b400640 1 -- 172.21.15.47:0/1909336945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb96c15f510 msgr2=0x7fb96c06cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:43.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb96b400640 1 --2- 172.21.15.47:0/1909336945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb96c15f510 0x7fb96c06cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:43.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb96b400640 1 -- 172.21.15.47:0/1909336945 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb954009420 con 0x7fb96c0b12f0 2024-02-19T15:23:43.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb962a00640 1 --2- 172.21.15.47:0/1909336945 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb96c0b1f20 0x7fb96c167200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:23:43.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb96be00640 1 --2- 172.21.15.47:0/1909336945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb96c15f510 0x7fb96c06cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:23:43.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb96b400640 1 --2- 172.21.15.47:0/1909336945 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb96c0b12f0 0x7fb96c1533e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb95c007fd0 tx=0x7fb95c00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:43.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb963e00640 1 -- 172.21.15.47:0/1909336945 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95c0074e0 con 0x7fb96c0b12f0 2024-02-19T15:23:43.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb963e00640 1 -- 172.21.15.47:0/1909336945 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb95c007660 con 0x7fb96c0b12f0 2024-02-19T15:23:43.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb963e00640 1 -- 172.21.15.47:0/1909336945 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95c004a30 con 0x7fb96c0b12f0 2024-02-19T15:23:43.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb971000640 1 -- 172.21.15.47:0/1909336945 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb96c0c8d10 con 0x7fb96c0b12f0 2024-02-19T15:23:43.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb971000640 1 -- 172.21.15.47:0/1909336945 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb96c153920 con 0x7fb96c0b12f0 2024-02-19T15:23:43.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb971000640 1 -- 172.21.15.47:0/1909336945 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb96c04f260 con 0x7fb96c0b12f0 2024-02-19T15:23:43.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.835+0000 7fb963e00640 1 -- 172.21.15.47:0/1909336945 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb95c004b90 con 0x7fb96c0b12f0 2024-02-19T15:23:43.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.839+0000 7fb963e00640 1 --2- 172.21.15.47:0/1909336945 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb948066e40 0x7fb948069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:43.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.839+0000 7fb963e00640 1 -- 172.21.15.47:0/1909336945 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb95c083e00 con 0x7fb96c0b12f0 2024-02-19T15:23:43.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.839+0000 7fb962a00640 1 --2- 172.21.15.47:0/1909336945 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb948066e40 0x7fb948069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:43.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.839+0000 7fb962a00640 1 --2- 172.21.15.47:0/1909336945 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb948066e40 0x7fb948069300 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb9540ad000 tx=0x7fb954008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:43.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:43.843+0000 7fb963e00640 1 -- 172.21.15.47:0/1909336945 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb95c050be0 con 0x7fb96c0b12f0 2024-02-19T15:23:44.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:44.047+0000 7fb971000640 1 -- 172.21.15.47:0/1909336945 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb96c072b60 con 0x7fb96c0b12f0 2024-02-19T15:23:44.921 INFO:tasks.ceph.mds.a.smithi047.stderr:2024-02-19T15:23:44.915+0000 7feccd200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T15:23:44.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:44.967+0000 7fb963e00640 1 -- 172.21.15.47:0/1909336945 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb95c018070 con 0x7fb96c0b12f0 2024-02-19T15:23:44.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:44.971+0000 7fb971000640 1 -- 172.21.15.47:0/1909336945 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb948066e40 msgr2=0x7fb948069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:44.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:44.971+0000 7fb971000640 1 --2- 172.21.15.47:0/1909336945 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb948066e40 0x7fb948069300 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb9540ad000 tx=0x7fb954008040 comp rx=0 tx=0).stop 2024-02-19T15:23:44.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:44.971+0000 7fb971000640 1 -- 172.21.15.47:0/1909336945 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb96c0b12f0 msgr2=0x7fb96c1533e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:44.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:44.971+0000 7fb971000640 1 --2- 172.21.15.47:0/1909336945 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb96c0b12f0 0x7fb96c1533e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb95c007fd0 tx=0x7fb95c00c9b0 comp rx=0 tx=0).stop 2024-02-19T15:23:44.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:44.971+0000 7fb971000640 1 -- 172.21.15.47:0/1909336945 shutdown_connections 2024-02-19T15:23:44.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:44.971+0000 7fb971000640 1 --2- 172.21.15.47:0/1909336945 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb948066e40 0x7fb948069300 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:44.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:44.971+0000 7fb971000640 1 --2- 172.21.15.47:0/1909336945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb96c15f510 0x7fb96c06cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:44.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:44.971+0000 7fb971000640 1 --2- 172.21.15.47:0/1909336945 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb96c0b1f20 0x7fb96c167200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:44.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:44.971+0000 7fb971000640 1 --2- 172.21.15.47:0/1909336945 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb96c0b12f0 0x7fb96c1533e0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:44.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:44.971+0000 7fb971000640 1 -- 172.21.15.47:0/1909336945 >> 172.21.15.47:0/1909336945 conn(0x7fb96c0d19f0 msgr2=0x7fb96c074000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:44.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:44.971+0000 7fb971000640 1 -- 172.21.15.47:0/1909336945 shutdown_connections 2024-02-19T15:23:44.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:44.971+0000 7fb971000640 1 -- 172.21.15.47:0/1909336945 wait complete. 2024-02-19T15:23:44.994 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T15:23:44.994 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T15:23:45.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.131+0000 7fa960200640 1 Processor -- start 2024-02-19T15:23:45.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.131+0000 7fa960200640 1 -- start start 2024-02-19T15:23:45.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.131+0000 7fa960200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa9580c7e80 0x7fa9580d0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:45.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.131+0000 7fa960200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9580c8260 0x7fa9580d0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:45.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.131+0000 7fa960200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9580cd750 con 0x7fa9580cfbf0 2024-02-19T15:23:45.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.131+0000 7fa960200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9580ceee0 con 0x7fa9580c7e80 2024-02-19T15:23:45.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.131+0000 7fa960200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9580cdc60 con 0x7fa9580c8260 2024-02-19T15:23:45.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.131+0000 7fa95d800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa9580c7e80 0x7fa9580d0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:45.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.131+0000 7fa95e200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa9580cfbf0 0x7fa9580c5680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49114/0 (socket says 172.21.15.47:49114) 2024-02-19T15:23:45.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.131+0000 7fa95e200640 1 -- 172.21.15.47:0/3676654629 learned_addr learned my addr 172.21.15.47:0/3676654629 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:45.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.131+0000 7fa95ce00640 1 --2- 172.21.15.47:0/3676654629 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9580c8260 0x7fa9580d0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:45.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.131+0000 7fa95d800640 1 -- 172.21.15.47:0/3676654629 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9580c8260 msgr2=0x7fa9580d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:45.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.131+0000 7fa95d800640 1 --2- 172.21.15.47:0/3676654629 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9580c8260 0x7fa9580d0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:45.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.131+0000 7fa95d800640 1 -- 172.21.15.47:0/3676654629 >> v1:172.21.15.47:6789/0 conn(0x7fa9580cfbf0 legacy=0x7fa9580c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:45.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.135+0000 7fa957e00640 1 -- 172.21.15.47:0/3676654629 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307699102 0 0) 0x7fa9580cd750 con 0x7fa9580cfbf0 2024-02-19T15:23:45.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.135+0000 7fa95d800640 1 -- 172.21.15.47:0/3676654629 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9580d2a40 con 0x7fa9580c7e80 2024-02-19T15:23:45.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.135+0000 7fa95ce00640 1 --2- 172.21.15.47:0/3676654629 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9580c8260 0x7fa9580d0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:45.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.135+0000 7fa95d800640 1 --2- 172.21.15.47:0/3676654629 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9580c7e80 0x7fa9580d0110 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa94c0097c0 tx=0x7fa94c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=476ff9707303fbe5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:45.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.135+0000 7fa957e00640 1 -- 172.21.15.47:0/3676654629 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94c0ab670 con 0x7fa9580c7e80 2024-02-19T15:23:45.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.135+0000 7fa957e00640 1 -- 172.21.15.47:0/3676654629 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa94c0ab7d0 con 0x7fa9580c7e80 2024-02-19T15:23:45.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.135+0000 7fa957e00640 1 -- 172.21.15.47:0/3676654629 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94c0abb60 con 0x7fa9580c7e80 2024-02-19T15:23:45.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.135+0000 7fa960200640 1 -- 172.21.15.47:0/3676654629 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9580c7e80 msgr2=0x7fa9580d0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:45.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.135+0000 7fa960200640 1 --2- 172.21.15.47:0/3676654629 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9580c7e80 0x7fa9580d0110 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa94c0097c0 tx=0x7fa94c01c470 comp rx=0 tx=0).stop 2024-02-19T15:23:45.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.135+0000 7fa960200640 1 -- 172.21.15.47:0/3676654629 shutdown_connections 2024-02-19T15:23:45.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.135+0000 7fa960200640 1 --2- 172.21.15.47:0/3676654629 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9580c8260 0x7fa9580d0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:45.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.135+0000 7fa960200640 1 --2- 172.21.15.47:0/3676654629 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9580c7e80 0x7fa9580d0110 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:45.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.135+0000 7fa960200640 1 -- 172.21.15.47:0/3676654629 >> 172.21.15.47:0/3676654629 conn(0x7fa9580d1b30 msgr2=0x7fa9580d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:45.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.135+0000 7fa960200640 1 -- 172.21.15.47:0/3676654629 shutdown_connections 2024-02-19T15:23:45.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.135+0000 7fa960200640 1 -- 172.21.15.47:0/3676654629 wait complete. 2024-02-19T15:23:45.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa960200640 1 Processor -- start 2024-02-19T15:23:45.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa960200640 1 -- start start 2024-02-19T15:23:45.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa960200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9580c7e80 0x7fa9580b1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:45.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa960200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9580c8260 0x7fa9580ae760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:45.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa960200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9580cfbf0 0x7fa9580ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:45.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa960200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9580aefb0 con 0x7fa9580c8260 2024-02-19T15:23:45.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa960200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9580b1a20 con 0x7fa9580c7e80 2024-02-19T15:23:45.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa960200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9580b10f0 con 0x7fa9580cfbf0 2024-02-19T15:23:45.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa95ce00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9580c8260 0x7fa9580ae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:45.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa95ce00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9580c8260 0x7fa9580ae760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33304/0 (socket says 172.21.15.47:33304) 2024-02-19T15:23:45.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa95ce00640 1 -- 172.21.15.47:0/3270583380 learned_addr learned my addr 172.21.15.47:0/3270583380 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:45.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa95d800640 1 --2- 172.21.15.47:0/3270583380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9580c7e80 0x7fa9580b1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:45.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa95e200640 1 --2- 172.21.15.47:0/3270583380 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9580cfbf0 0x7fa9580ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:45.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa95ce00640 1 -- 172.21.15.47:0/3270583380 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9580cfbf0 msgr2=0x7fa9580ccf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:45.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa95ce00640 1 --2- 172.21.15.47:0/3270583380 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9580cfbf0 0x7fa9580ccf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:45.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa95ce00640 1 -- 172.21.15.47:0/3270583380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9580c7e80 msgr2=0x7fa9580b1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:45.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa95ce00640 1 --2- 172.21.15.47:0/3270583380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9580c7e80 0x7fa9580b1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:45.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa95ce00640 1 -- 172.21.15.47:0/3270583380 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94c009420 con 0x7fa9580c8260 2024-02-19T15:23:45.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa95ce00640 1 --2- 172.21.15.47:0/3270583380 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9580c8260 0x7fa9580ae760 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa94400d800 tx=0x7fa94400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:45.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa955600640 1 -- 172.21.15.47:0/3270583380 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9440096a0 con 0x7fa9580c8260 2024-02-19T15:23:45.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa955600640 1 -- 172.21.15.47:0/3270583380 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa944009800 con 0x7fa9580c8260 2024-02-19T15:23:45.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.139+0000 7fa960200640 1 -- 172.21.15.47:0/3270583380 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9580b13d0 con 0x7fa9580c8260 2024-02-19T15:23:45.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.143+0000 7fa955600640 1 -- 172.21.15.47:0/3270583380 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa944009bb0 con 0x7fa9580c8260 2024-02-19T15:23:45.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.143+0000 7fa960200640 1 -- 172.21.15.47:0/3270583380 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9580aeca0 con 0x7fa9580c8260 2024-02-19T15:23:45.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.143+0000 7fa960200640 1 -- 172.21.15.47:0/3270583380 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9580d1180 con 0x7fa9580c8260 2024-02-19T15:23:45.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.143+0000 7fa955600640 1 -- 172.21.15.47:0/3270583380 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa944009d10 con 0x7fa9580c8260 2024-02-19T15:23:45.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.143+0000 7fa955600640 1 --2- 172.21.15.47:0/3270583380 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa938066e40 0x7fa938069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:45.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.143+0000 7fa955600640 1 -- 172.21.15.47:0/3270583380 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa944084e90 con 0x7fa9580c8260 2024-02-19T15:23:45.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.143+0000 7fa95d800640 1 --2- 172.21.15.47:0/3270583380 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa938066e40 0x7fa938069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:45.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.147+0000 7fa95d800640 1 --2- 172.21.15.47:0/3270583380 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa938066e40 0x7fa938069300 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa94c0af000 tx=0x7fa94c0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:45.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.147+0000 7fa955600640 1 -- 172.21.15.47:0/3270583380 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa944051c80 con 0x7fa9580c8260 2024-02-19T15:23:45.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.347+0000 7fa960200640 1 -- 172.21.15.47:0/3270583380 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa9580b1b90 con 0x7fa9580c8260 2024-02-19T15:23:45.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.971+0000 7fa955600640 1 -- 172.21.15.47:0/3270583380 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa944055340 con 0x7fa9580c8260 2024-02-19T15:23:45.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.975+0000 7fa960200640 1 -- 172.21.15.47:0/3270583380 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa938066e40 msgr2=0x7fa938069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:45.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.975+0000 7fa960200640 1 --2- 172.21.15.47:0/3270583380 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa938066e40 0x7fa938069300 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa94c0af000 tx=0x7fa94c0a7000 comp rx=0 tx=0).stop 2024-02-19T15:23:45.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.975+0000 7fa960200640 1 -- 172.21.15.47:0/3270583380 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9580c8260 msgr2=0x7fa9580ae760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:45.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.975+0000 7fa960200640 1 --2- 172.21.15.47:0/3270583380 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9580c8260 0x7fa9580ae760 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa94400d800 tx=0x7fa94400dcc0 comp rx=0 tx=0).stop 2024-02-19T15:23:45.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.975+0000 7fa960200640 1 -- 172.21.15.47:0/3270583380 shutdown_connections 2024-02-19T15:23:45.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.975+0000 7fa960200640 1 --2- 172.21.15.47:0/3270583380 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa938066e40 0x7fa938069300 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:45.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.975+0000 7fa960200640 1 --2- 172.21.15.47:0/3270583380 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9580cfbf0 0x7fa9580ccf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:45.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.975+0000 7fa960200640 1 --2- 172.21.15.47:0/3270583380 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9580c8260 0x7fa9580ae760 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:45.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.975+0000 7fa960200640 1 --2- 172.21.15.47:0/3270583380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9580c7e80 0x7fa9580b1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:45.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.975+0000 7fa960200640 1 -- 172.21.15.47:0/3270583380 >> 172.21.15.47:0/3270583380 conn(0x7fa9580d1b30 msgr2=0x7fa95806c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:45.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.975+0000 7fa960200640 1 -- 172.21.15.47:0/3270583380 shutdown_connections 2024-02-19T15:23:45.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:45.975+0000 7fa960200640 1 -- 172.21.15.47:0/3270583380 wait complete. 2024-02-19T15:23:46.000 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T15:23:46.001 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:23:46.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.135+0000 7f46a1400640 1 Processor -- start 2024-02-19T15:23:46.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.135+0000 7f46a1400640 1 -- start start 2024-02-19T15:23:46.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.135+0000 7f46a1400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f469c0cded0 0x7f469c0ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:46.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.135+0000 7f46a1400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c0cce30 con 0x7f469c0ceb40 2024-02-19T15:23:46.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.135+0000 7f46a1400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c0cd110 con 0x7f469c06d650 2024-02-19T15:23:46.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.135+0000 7f46a1400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f469c0cefd0 con 0x7f469c0cded0 2024-02-19T15:23:46.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.139+0000 7f469b200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f469c0ceb40 0x7f469c06bd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49124/0 (socket says 172.21.15.47:49124) 2024-02-19T15:23:46.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.139+0000 7f469b200640 1 -- 172.21.15.47:0/1495374704 learned_addr learned my addr 172.21.15.47:0/1495374704 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:46.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.139+0000 7f4691e00640 1 --2- 172.21.15.47:0/1495374704 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f469c0cded0 0x7f469c0ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:46.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.139+0000 7f4699e00640 1 -- 172.21.15.47:0/1495374704 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121256050 0 0) 0x7f469c0cd110 con 0x7f469c06d650 2024-02-19T15:23:46.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.139+0000 7f4699e00640 1 -- 172.21.15.47:0/1495374704 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4680003530 con 0x7f469c06d650 2024-02-19T15:23:46.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.139+0000 7f4699e00640 1 -- 172.21.15.47:0/1495374704 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867852670 0 0) 0x7f469c0cce30 con 0x7f469c0ceb40 2024-02-19T15:23:46.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.139+0000 7f4699e00640 1 -- 172.21.15.47:0/1495374704 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f469c0cd110 con 0x7f469c0ceb40 2024-02-19T15:23:46.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.139+0000 7f4691e00640 1 -- 172.21.15.47:0/1495374704 >> v1:172.21.15.116:6789/0 conn(0x7f469c06d650 legacy=0x7f469c0d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:46.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.139+0000 7f4691e00640 1 -- 172.21.15.47:0/1495374704 >> v1:172.21.15.47:6789/0 conn(0x7f469c0ceb40 legacy=0x7f469c06bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:46.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.139+0000 7f4691e00640 1 -- 172.21.15.47:0/1495374704 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f469c0cd3f0 con 0x7f469c0cded0 2024-02-19T15:23:46.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.139+0000 7f4691e00640 1 --2- 172.21.15.47:0/1495374704 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f469c0cded0 0x7f469c0ce600 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f46840097f0 tx=0x7f468401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2204860825e57c81 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:46.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.139+0000 7f4699e00640 1 -- 172.21.15.47:0/1495374704 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46840ab620 con 0x7f469c0cded0 2024-02-19T15:23:46.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.139+0000 7f4699e00640 1 -- 172.21.15.47:0/1495374704 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46840ab780 con 0x7f469c0cded0 2024-02-19T15:23:46.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.139+0000 7f4699e00640 1 -- 172.21.15.47:0/1495374704 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46840abb10 con 0x7f469c0cded0 2024-02-19T15:23:46.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.143+0000 7f46a1400640 1 -- 172.21.15.47:0/1495374704 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f469c0cded0 msgr2=0x7f469c0ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:46.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.143+0000 7f46a1400640 1 --2- 172.21.15.47:0/1495374704 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f469c0cded0 0x7f469c0ce600 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f46840097f0 tx=0x7f468401c420 comp rx=0 tx=0).stop 2024-02-19T15:23:46.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.143+0000 7f46a1400640 1 -- 172.21.15.47:0/1495374704 shutdown_connections 2024-02-19T15:23:46.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.143+0000 7f46a1400640 1 --2- 172.21.15.47:0/1495374704 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f469c0cded0 0x7f469c0ce600 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:46.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.143+0000 7f46a1400640 1 -- 172.21.15.47:0/1495374704 >> 172.21.15.47:0/1495374704 conn(0x7f469c0d1bf0 msgr2=0x7f469c0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:46.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.143+0000 7f46a1400640 1 -- 172.21.15.47:0/1495374704 shutdown_connections 2024-02-19T15:23:46.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.143+0000 7f46a1400640 1 -- 172.21.15.47:0/1495374704 wait complete. 2024-02-19T15:23:46.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.143+0000 7f46a1400640 1 Processor -- start 2024-02-19T15:23:46.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.143+0000 7f46a1400640 1 -- start start 2024-02-19T15:23:46.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.143+0000 7f46a1400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f469c06d650 0x7f469c0c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:46.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.143+0000 7f46a1400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f469c0cded0 0x7f469c148200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:46.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.143+0000 7f46a1400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f469c0ceb40 0x7f469c0c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:46.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.143+0000 7f46a1400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c0c6b20 con 0x7f469c0cded0 2024-02-19T15:23:46.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.143+0000 7f46a1400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c148740 con 0x7f469c06d650 2024-02-19T15:23:46.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.143+0000 7f46a1400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f469c0c5ea0 con 0x7f469c0ceb40 2024-02-19T15:23:46.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.143+0000 7f4691e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f469c0cded0 0x7f469c148200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:46.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.143+0000 7f4691e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f469c0cded0 0x7f469c148200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33318/0 (socket says 172.21.15.47:33318) 2024-02-19T15:23:46.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.143+0000 7f4691e00640 1 -- 172.21.15.47:0/2711449624 learned_addr learned my addr 172.21.15.47:0/2711449624 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:46.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.143+0000 7f469a800640 1 --2- 172.21.15.47:0/2711449624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f469c06d650 0x7f469c0c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:46.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.143+0000 7f469b200640 1 --2- 172.21.15.47:0/2711449624 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f469c0ceb40 0x7f469c0c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:46.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.147+0000 7f4691e00640 1 -- 172.21.15.47:0/2711449624 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f469c0ceb40 msgr2=0x7f469c0c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:46.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.147+0000 7f4691e00640 1 --2- 172.21.15.47:0/2711449624 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f469c0ceb40 0x7f469c0c5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:46.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.147+0000 7f4691e00640 1 -- 172.21.15.47:0/2711449624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f469c06d650 msgr2=0x7f469c0c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:46.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.147+0000 7f4691e00640 1 --2- 172.21.15.47:0/2711449624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f469c06d650 0x7f469c0c6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:46.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.147+0000 7f4691e00640 1 -- 172.21.15.47:0/2711449624 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4684009420 con 0x7f469c0cded0 2024-02-19T15:23:46.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.147+0000 7f4691e00640 1 --2- 172.21.15.47:0/2711449624 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f469c0cded0 0x7f469c148200 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f46840ad040 tx=0x7f468401ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:46.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.147+0000 7f4693400640 1 -- 172.21.15.47:0/2711449624 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46840abdc0 con 0x7f469c0cded0 2024-02-19T15:23:46.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.147+0000 7f4693400640 1 -- 172.21.15.47:0/2711449624 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46840bf410 con 0x7f469c0cded0 2024-02-19T15:23:46.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.147+0000 7f46a1400640 1 -- 172.21.15.47:0/2711449624 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f469c0c6180 con 0x7f469c0cded0 2024-02-19T15:23:46.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.147+0000 7f4693400640 1 -- 172.21.15.47:0/2711449624 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46840bf7c0 con 0x7f469c0cded0 2024-02-19T15:23:46.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.147+0000 7f46a1400640 1 -- 172.21.15.47:0/2711449624 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f469c0c6740 con 0x7f469c0cded0 2024-02-19T15:23:46.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.147+0000 7f46a1400640 1 -- 172.21.15.47:0/2711449624 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f469c0c6c60 con 0x7f469c0cded0 2024-02-19T15:23:46.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.147+0000 7f4693400640 1 -- 172.21.15.47:0/2711449624 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f46840b5400 con 0x7f469c0cded0 2024-02-19T15:23:46.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.151+0000 7f4693400640 1 --2- 172.21.15.47:0/2711449624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4678066e70 0x7f4678069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:46.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.151+0000 7f4693400640 1 -- 172.21.15.47:0/2711449624 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f468411f200 con 0x7f469c0cded0 2024-02-19T15:23:46.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.151+0000 7f469a800640 1 --2- 172.21.15.47:0/2711449624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4678066e70 0x7f4678069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:46.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.151+0000 7f469a800640 1 --2- 172.21.15.47:0/2711449624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4678066e70 0x7f4678069330 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f468c0011f0 tx=0x7f468c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:46.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.151+0000 7f4693400640 1 -- 172.21.15.47:0/2711449624 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f469c0c6c60 con 0x7f469c0cded0 2024-02-19T15:23:46.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.347+0000 7f46a1400640 1 -- 172.21.15.47:0/2711449624 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f469c0c6c60 con 0x7f469c0cded0 2024-02-19T15:23:46.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.351+0000 7f4693400640 1 -- 172.21.15.47:0/2711449624 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f469c0c6c60 con 0x7f469c0cded0 2024-02-19T15:23:46.355 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:46.355 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5132,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/548836925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":548836925},{"type":"v1","addr":"172.21.15.116:6836","nonce":548836925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5135,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3530214438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3530214438},{"type":"v1","addr":"172.21.15.116:6837","nonce":3530214438}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/3433978846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3433978846},{"type":"v1","addr":"172.21.15.47:6834","nonce":3433978846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:23:42.342661+0000","modified":"2024-02-19T15:23:45.352372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5514":{"gid":5514,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3137466738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3137466738},{"type":"v1","addr":"172.21.15.47:6835","nonce":3137466738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-02-19T15:23:46.355 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 25 2024-02-19T15:23:46.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.351+0000 7f46a1400640 1 -- 172.21.15.47:0/2711449624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4678066e70 msgr2=0x7f4678069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:46.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.351+0000 7f46a1400640 1 --2- 172.21.15.47:0/2711449624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4678066e70 0x7f4678069330 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f468c0011f0 tx=0x7f468c009040 comp rx=0 tx=0).stop 2024-02-19T15:23:46.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.351+0000 7f46a1400640 1 -- 172.21.15.47:0/2711449624 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f469c0cded0 msgr2=0x7f469c148200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:46.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.351+0000 7f46a1400640 1 --2- 172.21.15.47:0/2711449624 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f469c0cded0 0x7f469c148200 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f46840ad040 tx=0x7f468401ac60 comp rx=0 tx=0).stop 2024-02-19T15:23:46.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.351+0000 7f46a1400640 1 -- 172.21.15.47:0/2711449624 shutdown_connections 2024-02-19T15:23:46.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.351+0000 7f46a1400640 1 --2- 172.21.15.47:0/2711449624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4678066e70 0x7f4678069330 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:46.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.351+0000 7f46a1400640 1 --2- 172.21.15.47:0/2711449624 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f469c0ceb40 0x7f469c0c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:46.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.351+0000 7f46a1400640 1 --2- 172.21.15.47:0/2711449624 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f469c0cded0 0x7f469c148200 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:46.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.351+0000 7f46a1400640 1 --2- 172.21.15.47:0/2711449624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f469c06d650 0x7f469c0c6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:46.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.351+0000 7f46a1400640 1 -- 172.21.15.47:0/2711449624 >> 172.21.15.47:0/2711449624 conn(0x7f469c0d1bf0 msgr2=0x7f469c06bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:46.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.351+0000 7f46a1400640 1 -- 172.21.15.47:0/2711449624 shutdown_connections 2024-02-19T15:23:46.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.351+0000 7f46a1400640 1 -- 172.21.15.47:0/2711449624 wait complete. 2024-02-19T15:23:46.377 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:23:46.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.511+0000 7f9451800640 1 Processor -- start 2024-02-19T15:23:46.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.511+0000 7f9451800640 1 -- start start 2024-02-19T15:23:46.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.511+0000 7f9451800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f944c145590 0x7f944c0727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:46.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.511+0000 7f9451800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f944c15d280 con 0x7f944c0cea80 2024-02-19T15:23:46.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.511+0000 7f9451800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f944c070ff0 con 0x7f944c15c390 2024-02-19T15:23:46.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.511+0000 7f9451800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f944c0709e0 con 0x7f944c145590 2024-02-19T15:23:46.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.511+0000 7f9443e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f944c0cea80 0x7f944c06c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49140/0 (socket says 172.21.15.47:49140) 2024-02-19T15:23:46.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.511+0000 7f944aa00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f944c145590 0x7f944c0727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:46.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.511+0000 7f9443e00640 1 -- 172.21.15.47:0/1574664412 learned_addr learned my addr 172.21.15.47:0/1574664412 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:46.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.511+0000 7f944aa00640 1 -- 172.21.15.47:0/1574664412 >> v1:172.21.15.116:6789/0 conn(0x7f944c15c390 legacy=0x7f944c15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:46.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.511+0000 7f944aa00640 1 -- 172.21.15.47:0/1574664412 >> v1:172.21.15.47:6789/0 conn(0x7f944c0cea80 legacy=0x7f944c06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:46.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.511+0000 7f944aa00640 1 -- 172.21.15.47:0/1574664412 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f944c0cf020 con 0x7f944c145590 2024-02-19T15:23:46.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.519+0000 7f944aa00640 1 --2- 172.21.15.47:0/1574664412 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f944c145590 0x7f944c0727b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f943c0097c0 tx=0x7f943c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfe46c82db9138e4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:46.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.519+0000 7f944a000640 1 -- 172.21.15.47:0/1574664412 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943c0ab600 con 0x7f944c145590 2024-02-19T15:23:46.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.519+0000 7f944a000640 1 -- 172.21.15.47:0/1574664412 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f943c0a7050 con 0x7f944c145590 2024-02-19T15:23:46.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.519+0000 7f944a000640 1 -- 172.21.15.47:0/1574664412 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943c0ab980 con 0x7f944c145590 2024-02-19T15:23:46.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.519+0000 7f9451800640 1 -- 172.21.15.47:0/1574664412 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f944c145590 msgr2=0x7f944c0727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:46.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.519+0000 7f9451800640 1 --2- 172.21.15.47:0/1574664412 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f944c145590 0x7f944c0727b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f943c0097c0 tx=0x7f943c01c460 comp rx=0 tx=0).stop 2024-02-19T15:23:46.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.519+0000 7f9451800640 1 -- 172.21.15.47:0/1574664412 shutdown_connections 2024-02-19T15:23:46.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.519+0000 7f9451800640 1 --2- 172.21.15.47:0/1574664412 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f944c145590 0x7f944c0727b0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:46.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.519+0000 7f9451800640 1 -- 172.21.15.47:0/1574664412 >> 172.21.15.47:0/1574664412 conn(0x7f944c0b16a0 msgr2=0x7f944c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:46.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.519+0000 7f9451800640 1 -- 172.21.15.47:0/1574664412 shutdown_connections 2024-02-19T15:23:46.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.519+0000 7f9451800640 1 -- 172.21.15.47:0/1574664412 wait complete. 2024-02-19T15:23:46.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9451800640 1 Processor -- start 2024-02-19T15:23:46.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9451800640 1 -- start start 2024-02-19T15:23:46.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9451800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f944c0cea80 0x7f944c0b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:46.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9451800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f944c145590 0x7f944c0b4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:46.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9451800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f944c15c390 0x7f944c13c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:46.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9451800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944c0b3190 con 0x7f944c145590 2024-02-19T15:23:46.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9451800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944c0b33c0 con 0x7f944c15c390 2024-02-19T15:23:46.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9451800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f944c0b36a0 con 0x7f944c0cea80 2024-02-19T15:23:46.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9443e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f944c145590 0x7f944c0b4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:46.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9443e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f944c145590 0x7f944c0b4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33334/0 (socket says 172.21.15.47:33334) 2024-02-19T15:23:46.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9443e00640 1 -- 172.21.15.47:0/4009570807 learned_addr learned my addr 172.21.15.47:0/4009570807 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:46.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f944aa00640 1 --2- 172.21.15.47:0/4009570807 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f944c0cea80 0x7f944c0b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:46.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f944b400640 1 --2- 172.21.15.47:0/4009570807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f944c15c390 0x7f944c13c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:46.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9443e00640 1 -- 172.21.15.47:0/4009570807 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f944c0cea80 msgr2=0x7f944c0b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:46.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9443e00640 1 --2- 172.21.15.47:0/4009570807 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f944c0cea80 0x7f944c0b2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:46.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9443e00640 1 -- 172.21.15.47:0/4009570807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f944c15c390 msgr2=0x7f944c13c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:46.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9443e00640 1 --2- 172.21.15.47:0/4009570807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f944c15c390 0x7f944c13c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:46.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9443e00640 1 -- 172.21.15.47:0/4009570807 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f943c009420 con 0x7f944c145590 2024-02-19T15:23:46.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9443e00640 1 --2- 172.21.15.47:0/4009570807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f944c145590 0x7f944c0b4460 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f943400c420 tx=0x7f943400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:46.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9442a00640 1 -- 172.21.15.47:0/4009570807 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9434005cf0 con 0x7f944c145590 2024-02-19T15:23:46.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9442a00640 1 -- 172.21.15.47:0/4009570807 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9434005e50 con 0x7f944c145590 2024-02-19T15:23:46.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9442a00640 1 -- 172.21.15.47:0/4009570807 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94340072f0 con 0x7f944c145590 2024-02-19T15:23:46.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9451800640 1 -- 172.21.15.47:0/4009570807 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f944c0b3980 con 0x7f944c145590 2024-02-19T15:23:46.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.523+0000 7f9451800640 1 -- 172.21.15.47:0/4009570807 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f944c0b3f40 con 0x7f944c145590 2024-02-19T15:23:46.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.527+0000 7f9451800640 1 -- 172.21.15.47:0/4009570807 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f944c0b4160 con 0x7f944c145590 2024-02-19T15:23:46.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.527+0000 7f9442a00640 1 -- 172.21.15.47:0/4009570807 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9434007450 con 0x7f944c145590 2024-02-19T15:23:46.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.527+0000 7f9442a00640 1 --2- 172.21.15.47:0/4009570807 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9428066e40 0x7f9428069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:46.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.527+0000 7f9442a00640 1 -- 172.21.15.47:0/4009570807 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9434084100 con 0x7f944c145590 2024-02-19T15:23:46.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.527+0000 7f944aa00640 1 --2- 172.21.15.47:0/4009570807 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9428066e40 0x7f9428069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:46.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.531+0000 7f944aa00640 1 --2- 172.21.15.47:0/4009570807 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9428066e40 0x7f9428069300 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f943c0ad000 tx=0x7f943c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:46.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.531+0000 7f9442a00640 1 -- 172.21.15.47:0/4009570807 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f9434050e50 con 0x7f944c145590 2024-02-19T15:23:46.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.695+0000 7f9451800640 1 -- 172.21.15.47:0/4009570807 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f944c075360 con 0x7f944c145590 2024-02-19T15:23:46.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.695+0000 7f9442a00640 1 -- 172.21.15.47:0/4009570807 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16625 (secure 0 0 0) 0x7f9434054510 con 0x7f944c145590 2024-02-19T15:23:46.702 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:46.702 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":45,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:23:43.941118+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:23:40.751741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-02-19T15:23:41.338930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:23:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.699+0000 7f9451800640 1 -- 172.21.15.47:0/4009570807 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9428066e40 msgr2=0x7f9428069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.699+0000 7f9451800640 1 --2- 172.21.15.47:0/4009570807 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9428066e40 0x7f9428069300 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f943c0ad000 tx=0x7f943c008040 comp rx=0 tx=0).stop 2024-02-19T15:23:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.699+0000 7f9451800640 1 -- 172.21.15.47:0/4009570807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f944c145590 msgr2=0x7f944c0b4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.699+0000 7f9451800640 1 --2- 172.21.15.47:0/4009570807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f944c145590 0x7f944c0b4460 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f943400c420 tx=0x7f943400c8e0 comp rx=0 tx=0).stop 2024-02-19T15:23:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.699+0000 7f9451800640 1 -- 172.21.15.47:0/4009570807 shutdown_connections 2024-02-19T15:23:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.699+0000 7f9451800640 1 --2- 172.21.15.47:0/4009570807 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9428066e40 0x7f9428069300 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.699+0000 7f9451800640 1 --2- 172.21.15.47:0/4009570807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f944c15c390 0x7f944c13c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.699+0000 7f9451800640 1 --2- 172.21.15.47:0/4009570807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f944c145590 0x7f944c0b4460 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.699+0000 7f9451800640 1 --2- 172.21.15.47:0/4009570807 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f944c0cea80 0x7f944c0b2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.699+0000 7f9451800640 1 -- 172.21.15.47:0/4009570807 >> 172.21.15.47:0/4009570807 conn(0x7f944c0b16a0 msgr2=0x7f944c0b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.699+0000 7f9451800640 1 -- 172.21.15.47:0/4009570807 shutdown_connections 2024-02-19T15:23:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.699+0000 7f9451800640 1 -- 172.21.15.47:0/4009570807 wait complete. 2024-02-19T15:23:46.733 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T15:23:46.733 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:23:46.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.867+0000 7fcf98000640 1 Processor -- start 2024-02-19T15:23:46.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.867+0000 7fcf98000640 1 -- start start 2024-02-19T15:23:46.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.867+0000 7fcf98000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcf9015f590 0x7fcf900d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:46.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.867+0000 7fcf98000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf900b0280 con 0x7fcf900b0a80 2024-02-19T15:23:46.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.867+0000 7fcf98000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf900b04d0 con 0x7fcf900b1be0 2024-02-19T15:23:46.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.867+0000 7fcf98000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf900cc530 con 0x7fcf9015f590 2024-02-19T15:23:46.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.871+0000 7fcf94c00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fcf900b0a80 0x7fcf900b14d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49150/0 (socket says 172.21.15.47:49150) 2024-02-19T15:23:46.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.871+0000 7fcf94c00640 1 -- 172.21.15.47:0/2669755290 learned_addr learned my addr 172.21.15.47:0/2669755290 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:46.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.871+0000 7fcf95600640 1 --2- 172.21.15.47:0/2669755290 >> v2:172.21.15.116:3300/0 conn(0x7fcf9015f590 0x7fcf900d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:46.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.871+0000 7fcf8fe00640 1 -- 172.21.15.47:0/2669755290 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862690726 0 0) 0x7fcf900b0280 con 0x7fcf900b0a80 2024-02-19T15:23:46.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.871+0000 7fcf8fe00640 1 -- 172.21.15.47:0/2669755290 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf78003530 con 0x7fcf900b0a80 2024-02-19T15:23:46.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.871+0000 7fcf95600640 1 -- 172.21.15.47:0/2669755290 >> v1:172.21.15.116:6789/0 conn(0x7fcf900b1be0 legacy=0x7fcf900b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:46.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.871+0000 7fcf95600640 1 -- 172.21.15.47:0/2669755290 >> v1:172.21.15.47:6789/0 conn(0x7fcf900b0a80 legacy=0x7fcf900b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:46.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.871+0000 7fcf95600640 1 -- 172.21.15.47:0/2669755290 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf900b07b0 con 0x7fcf9015f590 2024-02-19T15:23:46.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.871+0000 7fcf95600640 1 --2- 172.21.15.47:0/2669755290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf9015f590 0x7fcf900d2690 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcf840097c0 tx=0x7fcf8401c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdad265c6eb13bfd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:46.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf8fe00640 1 -- 172.21.15.47:0/2669755290 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf840ab670 con 0x7fcf9015f590 2024-02-19T15:23:46.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf8fe00640 1 -- 172.21.15.47:0/2669755290 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf840ab7d0 con 0x7fcf9015f590 2024-02-19T15:23:46.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf8fe00640 1 -- 172.21.15.47:0/2669755290 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf840abb60 con 0x7fcf9015f590 2024-02-19T15:23:46.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf98000640 1 -- 172.21.15.47:0/2669755290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf9015f590 msgr2=0x7fcf900d2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:46.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf98000640 1 --2- 172.21.15.47:0/2669755290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf9015f590 0x7fcf900d2690 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcf840097c0 tx=0x7fcf8401c470 comp rx=0 tx=0).stop 2024-02-19T15:23:46.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf98000640 1 -- 172.21.15.47:0/2669755290 shutdown_connections 2024-02-19T15:23:46.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf98000640 1 --2- 172.21.15.47:0/2669755290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf9015f590 0x7fcf900d2690 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:46.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf98000640 1 -- 172.21.15.47:0/2669755290 >> 172.21.15.47:0/2669755290 conn(0x7fcf900d1b70 msgr2=0x7fcf900d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:46.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf98000640 1 -- 172.21.15.47:0/2669755290 shutdown_connections 2024-02-19T15:23:46.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf98000640 1 -- 172.21.15.47:0/2669755290 wait complete. 2024-02-19T15:23:46.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf98000640 1 Processor -- start 2024-02-19T15:23:46.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf98000640 1 -- start start 2024-02-19T15:23:46.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf98000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf900b0a80 0x7fcf90158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:46.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf98000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcf900b1be0 0x7fcf900c5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:46.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf98000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf9015f590 0x7fcf900c5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:46.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf98000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf900c63c0 con 0x7fcf9015f590 2024-02-19T15:23:46.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf98000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf900ca140 con 0x7fcf900b0a80 2024-02-19T15:23:46.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf98000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf900c9850 con 0x7fcf900b1be0 2024-02-19T15:23:46.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf96000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf9015f590 0x7fcf900c5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:46.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf96000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf9015f590 0x7fcf900c5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33348/0 (socket says 172.21.15.47:33348) 2024-02-19T15:23:46.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.875+0000 7fcf96000640 1 -- 172.21.15.47:0/3898817568 learned_addr learned my addr 172.21.15.47:0/3898817568 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:46.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.879+0000 7fcf94c00640 1 --2- 172.21.15.47:0/3898817568 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcf900b1be0 0x7fcf900c5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:46.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.879+0000 7fcf95600640 1 --2- 172.21.15.47:0/3898817568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf900b0a80 0x7fcf90158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:46.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.879+0000 7fcf96000640 1 -- 172.21.15.47:0/3898817568 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcf900b1be0 msgr2=0x7fcf900c5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:46.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.879+0000 7fcf96000640 1 --2- 172.21.15.47:0/3898817568 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcf900b1be0 0x7fcf900c5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:46.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.879+0000 7fcf96000640 1 -- 172.21.15.47:0/3898817568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf900b0a80 msgr2=0x7fcf90158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:46.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.879+0000 7fcf96000640 1 --2- 172.21.15.47:0/3898817568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf900b0a80 0x7fcf90158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:46.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.879+0000 7fcf96000640 1 -- 172.21.15.47:0/3898817568 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf84009420 con 0x7fcf9015f590 2024-02-19T15:23:46.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.879+0000 7fcf94c00640 1 --2- 172.21.15.47:0/3898817568 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcf900b1be0 0x7fcf900c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:46.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.879+0000 7fcf96000640 1 --2- 172.21.15.47:0/3898817568 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf9015f590 0x7fcf900c5e80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fcf8800c770 tx=0x7fcf8800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:46.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.879+0000 7fcf8d600640 1 -- 172.21.15.47:0/3898817568 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf88003b80 con 0x7fcf9015f590 2024-02-19T15:23:46.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.879+0000 7fcf8d600640 1 -- 172.21.15.47:0/3898817568 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf88003ce0 con 0x7fcf9015f590 2024-02-19T15:23:46.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.879+0000 7fcf98000640 1 -- 172.21.15.47:0/3898817568 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf900c9af0 con 0x7fcf9015f590 2024-02-19T15:23:46.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.879+0000 7fcf8d600640 1 -- 172.21.15.47:0/3898817568 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8800aa90 con 0x7fcf9015f590 2024-02-19T15:23:46.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.879+0000 7fcf98000640 1 -- 172.21.15.47:0/3898817568 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf900c65e0 con 0x7fcf9015f590 2024-02-19T15:23:46.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.879+0000 7fcf98000640 1 -- 172.21.15.47:0/3898817568 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf900ca2d0 con 0x7fcf9015f590 2024-02-19T15:23:46.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.879+0000 7fcf8d600640 1 -- 172.21.15.47:0/3898817568 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fcf8800abf0 con 0x7fcf9015f590 2024-02-19T15:23:46.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.883+0000 7fcf8d600640 1 --2- 172.21.15.47:0/3898817568 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcf70066e40 0x7fcf70069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:46.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.883+0000 7fcf8d600640 1 -- 172.21.15.47:0/3898817568 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf88085220 con 0x7fcf9015f590 2024-02-19T15:23:46.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.883+0000 7fcf95600640 1 --2- 172.21.15.47:0/3898817568 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcf70066e40 0x7fcf70069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:46.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.883+0000 7fcf95600640 1 --2- 172.21.15.47:0/3898817568 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcf70066e40 0x7fcf70069300 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcf840af000 tx=0x7fcf840a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:46.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:46.887+0000 7fcf8d600640 1 -- 172.21.15.47:0/3898817568 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fcf880520f0 con 0x7fcf9015f590 2024-02-19T15:23:47.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.043+0000 7fcf98000640 1 -- 172.21.15.47:0/3898817568 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcf90075360 con 0x7fcf70066e40 2024-02-19T15:23:47.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.055+0000 7fcf8d600640 1 -- 172.21.15.47:0/3898817568 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278172 (secure 0 0 0) 0x7fcf90075360 con 0x7fcf70066e40 2024-02-19T15:23:47.063 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:47.064 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:23:47.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.059+0000 7fcf98000640 1 -- 172.21.15.47:0/3898817568 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcf70066e40 msgr2=0x7fcf70069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:47.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.059+0000 7fcf98000640 1 --2- 172.21.15.47:0/3898817568 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcf70066e40 0x7fcf70069300 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcf840af000 tx=0x7fcf840a7000 comp rx=0 tx=0).stop 2024-02-19T15:23:47.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.059+0000 7fcf98000640 1 -- 172.21.15.47:0/3898817568 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf9015f590 msgr2=0x7fcf900c5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:47.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.059+0000 7fcf98000640 1 --2- 172.21.15.47:0/3898817568 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf9015f590 0x7fcf900c5e80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fcf8800c770 tx=0x7fcf8800cc30 comp rx=0 tx=0).stop 2024-02-19T15:23:47.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.059+0000 7fcf98000640 1 -- 172.21.15.47:0/3898817568 shutdown_connections 2024-02-19T15:23:47.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.059+0000 7fcf98000640 1 --2- 172.21.15.47:0/3898817568 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcf70066e40 0x7fcf70069300 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.059+0000 7fcf98000640 1 --2- 172.21.15.47:0/3898817568 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf9015f590 0x7fcf900c5e80 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.059+0000 7fcf98000640 1 --2- 172.21.15.47:0/3898817568 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcf900b1be0 0x7fcf900c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.059+0000 7fcf98000640 1 --2- 172.21.15.47:0/3898817568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf900b0a80 0x7fcf90158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.059+0000 7fcf98000640 1 -- 172.21.15.47:0/3898817568 >> 172.21.15.47:0/3898817568 conn(0x7fcf900d1b70 msgr2=0x7fcf9015e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:47.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.059+0000 7fcf98000640 1 -- 172.21.15.47:0/3898817568 shutdown_connections 2024-02-19T15:23:47.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.059+0000 7fcf98000640 1 -- 172.21.15.47:0/3898817568 wait complete. 2024-02-19T15:23:47.084 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-02-19T15:23:45.170227+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":246056,"kb_used_data":4908,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754728664,"statfs":{"total":773094113280,"available":772842151936,"internally_reserved":0,"allocated":5025792,"data_stored":2700331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":246795826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.237686"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.945858+0000","last_change":"2024-02-19T15:23:41.945858+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.945858+0000","last_undegraded":"2024-02-19T15:23:41.945858+0000","last_fullsized":"2024-02-19T15:23:41.945858+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.942749+0000","last_change":"2024-02-19T15:23:41.942749+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.942749+0000","last_undegraded":"2024-02-19T15:23:41.942749+0000","last_fullsized":"2024-02-19T15:23:41.942749+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.942606+0000","last_change":"2024-02-19T15:23:41.942606+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.942606+0000","last_undegraded":"2024-02-19T15:23:41.942606+0000","last_fullsized":"2024-02-19T15:23:41.942606+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.945345+0000","last_change":"2024-02-19T15:23:41.945345+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.945345+0000","last_undegraded":"2024-02-19T15:23:41.945345+0000","last_fullsized":"2024-02-19T15:23:41.945345+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.945592+0000","last_change":"2024-02-19T15:23:41.945592+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.945592+0000","last_undegraded":"2024-02-19T15:23:41.945592+0000","last_fullsized":"2024-02-19T15:23:41.945592+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.944819+0000","last_change":"2024-02-19T15:23:41.944819+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.944819+0000","last_undegraded":"2024-02-19T15:23:41.944819+0000","last_fullsized":"2024-02-19T15:23:41.944819+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.937574+0000","last_change":"2024-02-19T15:23:40.937574+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.937574+0000","last_undegraded":"2024-02-19T15:23:40.937574+0000","last_fullsized":"2024-02-19T15:23:40.937574+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.944998+0000","last_change":"2024-02-19T15:23:41.944998+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.944998+0000","last_undegraded":"2024-02-19T15:23:41.944998+0000","last_fullsized":"2024-02-19T15:23:41.944998+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.943046+0000","last_change":"2024-02-19T15:23:40.943046+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.943046+0000","last_undegraded":"2024-02-19T15:23:40.943046+0000","last_fullsized":"2024-02-19T15:23:40.943046+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.934477+0000","last_change":"2024-02-19T15:23:40.934477+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.934477+0000","last_undegraded":"2024-02-19T15:23:40.934477+0000","last_fullsized":"2024-02-19T15:23:40.934477+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.934677+0000","last_change":"2024-02-19T15:23:40.934677+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.934677+0000","last_undegraded":"2024-02-19T15:23:40.934677+0000","last_fullsized":"2024-02-19T15:23:40.934677+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.938804+0000","last_change":"2024-02-19T15:23:40.938804+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.938804+0000","last_undegraded":"2024-02-19T15:23:40.938804+0000","last_fullsized":"2024-02-19T15:23:40.938804+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.943209+0000","last_change":"2024-02-19T15:23:40.943209+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.943209+0000","last_undegraded":"2024-02-19T15:23:40.943209+0000","last_fullsized":"2024-02-19T15:23:40.943209+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.937411+0000","last_change":"2024-02-19T15:23:40.937411+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.937411+0000","last_undegraded":"2024-02-19T15:23:40.937411+0000","last_fullsized":"2024-02-19T15:23:40.937411+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.943222+0000","last_change":"2024-02-19T15:23:41.943222+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.943222+0000","last_undegraded":"2024-02-19T15:23:41.943222+0000","last_fullsized":"2024-02-19T15:23:41.943222+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.941613+0000","last_change":"2024-02-19T15:23:40.941613+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.941613+0000","last_undegraded":"2024-02-19T15:23:40.941613+0000","last_fullsized":"2024-02-19T15:23:40.941613+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.942576+0000","last_change":"2024-02-19T15:23:40.942576+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.942576+0000","last_undegraded":"2024-02-19T15:23:40.942576+0000","last_fullsized":"2024-02-19T15:23:40.942576+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.941677+0000","last_change":"2024-02-19T15:23:40.941677+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.941677+0000","last_undegraded":"2024-02-19T15:23:40.941677+0000","last_fullsized":"2024-02-19T15:23:40.941677+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.935161+0000","last_change":"2024-02-19T15:23:40.935161+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.935161+0000","last_undegraded":"2024-02-19T15:23:40.935161+0000","last_fullsized":"2024-02-19T15:23:40.935161+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.935537+0000","last_change":"2024-02-19T15:23:40.935537+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.935537+0000","last_undegraded":"2024-02-19T15:23:40.935537+0000","last_fullsized":"2024-02-19T15:23:40.935537+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.944967+0000","last_change":"2024-02-19T15:23:41.944967+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.944967+0000","last_undegraded":"2024-02-19T15:23:41.944967+0000","last_fullsized":"2024-02-19T15:23:41.944967+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.938422+0000","last_change":"2024-02-19T15:23:40.938422+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.938422+0000","last_undegraded":"2024-02-19T15:23:40.938422+0000","last_fullsized":"2024-02-19T15:23:40.938422+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.935212+0000","last_change":"2024-02-19T15:23:40.935212+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.935212+0000","last_undegraded":"2024-02-19T15:23:40.935212+0000","last_fullsized":"2024-02-19T15:23:40.935212+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.937519+0000","last_change":"2024-02-19T15:23:40.937519+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.937519+0000","last_undegraded":"2024-02-19T15:23:40.937519+0000","last_fullsized":"2024-02-19T15:23:40.937519+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.941118+0000","last_change":"2024-02-19T15:23:40.941118+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.941118+0000","last_undegraded":"2024-02-19T15:23:40.941118+0000","last_fullsized":"2024-02-19T15:23:40.941118+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.935503+0000","last_change":"2024-02-19T15:23:40.935503+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.935503+0000","last_undegraded":"2024-02-19T15:23:40.935503+0000","last_fullsized":"2024-02-19T15:23:40.935503+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.942526+0000","last_change":"2024-02-19T15:23:40.942526+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.942526+0000","last_undegraded":"2024-02-19T15:23:40.942526+0000","last_fullsized":"2024-02-19T15:23:40.942526+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.944685+0000","last_change":"2024-02-19T15:23:41.944685+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.944685+0000","last_undegraded":"2024-02-19T15:23:41.944685+0000","last_fullsized":"2024-02-19T15:23:41.944685+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.937549+0000","last_change":"2024-02-19T15:23:40.937549+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.937549+0000","last_undegraded":"2024-02-19T15:23:40.937549+0000","last_fullsized":"2024-02-19T15:23:40.937549+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.944426+0000","last_change":"2024-02-19T15:23:41.944426+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.944426+0000","last_undegraded":"2024-02-19T15:23:41.944426+0000","last_fullsized":"2024-02-19T15:23:41.944426+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.937581+0000","last_change":"2024-02-19T15:23:40.937581+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.937581+0000","last_undegraded":"2024-02-19T15:23:40.937581+0000","last_fullsized":"2024-02-19T15:23:40.937581+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.937424+0000","last_change":"2024-02-19T15:23:40.937424+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.937424+0000","last_undegraded":"2024-02-19T15:23:40.937424+0000","last_fullsized":"2024-02-19T15:23:40.937424+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.937555+0000","last_change":"2024-02-19T15:23:40.937555+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.937555+0000","last_undegraded":"2024-02-19T15:23:40.937555+0000","last_fullsized":"2024-02-19T15:23:40.937555+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.941598+0000","last_change":"2024-02-19T15:23:40.941598+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.941598+0000","last_undegraded":"2024-02-19T15:23:40.941598+0000","last_fullsized":"2024-02-19T15:23:40.941598+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.934869+0000","last_change":"2024-02-19T15:23:40.934869+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.934869+0000","last_undegraded":"2024-02-19T15:23:40.934869+0000","last_fullsized":"2024-02-19T15:23:40.934869+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.943060+0000","last_change":"2024-02-19T15:23:41.943060+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.943060+0000","last_undegraded":"2024-02-19T15:23:41.943060+0000","last_fullsized":"2024-02-19T15:23:41.943060+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.937552+0000","last_change":"2024-02-19T15:23:40.937552+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.937552+0000","last_undegraded":"2024-02-19T15:23:40.937552+0000","last_fullsized":"2024-02-19T15:23:40.937552+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.934581+0000","last_change":"2024-02-19T15:23:40.934581+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.934581+0000","last_undegraded":"2024-02-19T15:23:40.934581+0000","last_fullsized":"2024-02-19T15:23:40.934581+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.934462+0000","last_change":"2024-02-19T15:23:40.934462+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.934462+0000","last_undegraded":"2024-02-19T15:23:40.934462+0000","last_fullsized":"2024-02-19T15:23:40.934462+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.934824+0000","last_change":"2024-02-19T15:23:40.934824+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.934824+0000","last_undegraded":"2024-02-19T15:23:40.934824+0000","last_fullsized":"2024-02-19T15:23:40.934824+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.942518+0000","last_change":"2024-02-19T15:23:40.942518+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.942518+0000","last_undegraded":"2024-02-19T15:23:40.942518+0000","last_fullsized":"2024-02-19T15:23:40.942518+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.934471+0000","last_change":"2024-02-19T15:23:40.934471+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.934471+0000","last_undegraded":"2024-02-19T15:23:40.934471+0000","last_fullsized":"2024-02-19T15:23:40.934471+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.940412+0000","last_change":"2024-02-19T15:23:40.940412+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.940412+0000","last_undegraded":"2024-02-19T15:23:40.940412+0000","last_fullsized":"2024-02-19T15:23:40.940412+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:23:41.932392+0000","last_change":"2024-02-19T15:23:41.932392+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.932392+0000","last_undegraded":"2024-02-19T15:23:41.932392+0000","last_fullsized":"2024-02-19T15:23:41.932392+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-02-19T15:23:40.937608+0000","last_change":"2024-02-19T15:23:40.937608+0000","last_active":"2024-02-19T15:23:40.929810+0000","last_peered":"2024-02-19T15:23:40.929810+0000","last_clean":"2024-02-19T15:23:40.929810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:40.937608+0000","last_undegraded":"2024-02-19T15:23:40.937608+0000","last_fullsized":"2024-02-19T15:23:40.937608+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":206,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-02-19T15:23:40.935685+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:23:40.935685+0000","last_peered":"2024-02-19T15:23:40.935685+0000","last_clean":"2024-02-19T15:23:40.935685+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:23:40.935685+0000","last_undegraded":"2024-02-19T15:23:40.935685+0000","last_fullsized":"2024-02-19T15:23:40.935685+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":1},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509465,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":487424,"data_stored":193585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":4.6029999999999998,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.79400000000000004}]},{"osd":1,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.625},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.71699999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.73999999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.75600000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.64900000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.81000000000000005}]}]},{"osd":7,"up_from":14,"seq":60129542169,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28064,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343776,"statfs":{"total":96636764160,"available":96608026624,"internally_reserved":0,"allocated":1069056,"data_stored":778136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.498},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":1.0580000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":2.968,"5min":2.968,"15min":2.968},"last":0.88400000000000001}]},{"osd":2,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":0.63500000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":3.1720000000000002,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":0.84099999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.62}]},{"osd":5,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":1.002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.73999999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":1.103}]}]},{"osd":5,"up_from":14,"seq":60129542170,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27496,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344344,"statfs":{"total":96636764160,"available":96608608256,"internally_reserved":0,"allocated":487424,"data_stored":193585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.56999999999999995}]},{"osd":1,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.85499999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.66300000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.63900000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":0.67600000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.89400000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.59899999999999998}]}]},{"osd":4,"up_from":14,"seq":60129542168,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":475136,"data_stored":187768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.65000000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.85299999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.72399999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.81599999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.88100000000000001}]},{"osd":6,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.94099999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.96999999999999997}]}]},{"osd":3,"up_from":14,"seq":60129542168,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31644,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340196,"statfs":{"total":96636764160,"available":96604360704,"internally_reserved":0,"allocated":475136,"data_stored":187768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.2450000000000001,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.32500000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":4.0220000000000002,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.435}]},{"osd":2,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.47599999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":4.2039999999999997,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.67600000000000005}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":4.1589999999999998,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.85099999999999998}]},{"osd":6,"last update":"Mon Feb 19 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":4.0940000000000003,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.80100000000000005}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":3.9660000000000002,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.88600000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542169,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31656,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340184,"statfs":{"total":96636764160,"available":96604348416,"internally_reserved":0,"allocated":487424,"data_stored":193585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.7160000000000002,"5min":4.7160000000000002,"15min":4.7160000000000002},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":8.2129999999999992,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.59999999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":8.1620000000000008,"5min":8.1620000000000008,"15min":8.1620000000000008},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.53500000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":4.7679999999999998,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":4.9909999999999997,"5min":4.9909999999999997,"15min":4.9909999999999997},"last":0.56000000000000005}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":4.3810000000000002,"5min":4.3810000000000002,"15min":4.3810000000000002},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":4.3440000000000003,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.90300000000000002}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":4.4039999999999999,"5min":4.4039999999999999,"15min":4.4039999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":4.3259999999999996,"5min":4.3259999999999996,"15min":4.3259999999999996},"last":0.88400000000000001}]},{"osd":6,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":4.5220000000000002,"5min":4.5220000000000002,"15min":4.5220000000000002},"last":0.80000000000000004},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":4.4320000000000004,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.78300000000000003}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":4.4829999999999997,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":4.4989999999999997,"5min":4.4989999999999997,"15min":4.4989999999999997},"last":0.73599999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542170,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31656,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340184,"statfs":{"total":96636764160,"available":96604348416,"internally_reserved":0,"allocated":475136,"data_stored":187768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.38400000000000001}]},{"osd":2,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.40200000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.54800000000000004}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.75700000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.73399999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.82499999999999996}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.877}]}]},{"osd":0,"up_from":14,"seq":60129542168,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36384,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335456,"statfs":{"total":96636764160,"available":96599506944,"internally_reserved":0,"allocated":1069056,"data_stored":778136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.0199999999999996,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.503},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.73899999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":5.6669999999999998,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.4649999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.54200000000000004}]},{"osd":3,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.9539999999999997,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.345,"5min":1.345,"15min":1.345},"last":0.55600000000000005}]},{"osd":4,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":4.7880000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.80800000000000005}]},{"osd":5,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.71699999999999997}]},{"osd":6,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":4.8460000000000001,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.873}]},{"osd":7,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":4.9809999999999999,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":4.915,"5min":4.915,"15min":4.915},"last":0.85299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:23:47.089 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:23:47.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.259+0000 7f500b200640 1 Processor -- start 2024-02-19T15:23:47.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.259+0000 7f500b200640 1 -- start start 2024-02-19T15:23:47.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.263+0000 7f500b200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5004145590 0x7f50040727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:47.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.263+0000 7f500b200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f500415c390 0x7f500415c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:47.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.263+0000 7f500b200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500415d280 con 0x7f50040cea80 2024-02-19T15:23:47.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.263+0000 7f500b200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5004070ff0 con 0x7f500415c390 2024-02-19T15:23:47.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.263+0000 7f500b200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f500415cd60 con 0x7f5004145590 2024-02-19T15:23:47.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.263+0000 7f4ffbe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f50040cea80 0x7f500406c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49156/0 (socket says 172.21.15.47:49156) 2024-02-19T15:23:47.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.263+0000 7f4ffbe00640 1 -- 172.21.15.47:0/2514450059 learned_addr learned my addr 172.21.15.47:0/2514450059 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:47.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.263+0000 7f5009200640 1 --2- 172.21.15.47:0/2514450059 >> v2:172.21.15.116:3300/0 conn(0x7f500415c390 0x7f500415c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:47.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.263+0000 7f5003e00640 1 --2- 172.21.15.47:0/2514450059 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5004145590 0x7f50040727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:47.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.263+0000 7f5003400640 1 -- 172.21.15.47:0/2514450059 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615466815 0 0) 0x7f500415d280 con 0x7f50040cea80 2024-02-19T15:23:47.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.263+0000 7f5003400640 1 -- 172.21.15.47:0/2514450059 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe8003530 con 0x7f50040cea80 2024-02-19T15:23:47.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.263+0000 7f5003e00640 1 -- 172.21.15.47:0/2514450059 >> v2:172.21.15.116:3300/0 conn(0x7f500415c390 msgr2=0x7f500415c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:47.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.263+0000 7f5003e00640 1 --2- 172.21.15.47:0/2514450059 >> v2:172.21.15.116:3300/0 conn(0x7f500415c390 0x7f500415c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.263+0000 7f5003e00640 1 -- 172.21.15.47:0/2514450059 >> v1:172.21.15.47:6789/0 conn(0x7f50040cea80 legacy=0x7f500406c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:47.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.263+0000 7f5003e00640 1 -- 172.21.15.47:0/2514450059 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50040709e0 con 0x7f5004145590 2024-02-19T15:23:47.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.263+0000 7f5009200640 1 --2- 172.21.15.47:0/2514450059 >> v2:172.21.15.116:3300/0 conn(0x7f500415c390 0x7f500415c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:23:47.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.267+0000 7f5003e00640 1 --2- 172.21.15.47:0/2514450059 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5004145590 0x7f50040727b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40097c0 tx=0x7f4ff401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7bd7f6314f94547 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:47.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.267+0000 7f5003400640 1 -- 172.21.15.47:0/2514450059 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff40ac410 con 0x7f5004145590 2024-02-19T15:23:47.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.267+0000 7f5003400640 1 -- 172.21.15.47:0/2514450059 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff40ac570 con 0x7f5004145590 2024-02-19T15:23:47.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.267+0000 7f5003400640 1 -- 172.21.15.47:0/2514450059 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff40ac900 con 0x7f5004145590 2024-02-19T15:23:47.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.267+0000 7f500b200640 1 -- 172.21.15.47:0/2514450059 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5004145590 msgr2=0x7f50040727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:47.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.267+0000 7f500b200640 1 --2- 172.21.15.47:0/2514450059 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5004145590 0x7f50040727b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40097c0 tx=0x7f4ff401c4a0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.267+0000 7f500b200640 1 -- 172.21.15.47:0/2514450059 shutdown_connections 2024-02-19T15:23:47.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.267+0000 7f500b200640 1 --2- 172.21.15.47:0/2514450059 >> v2:172.21.15.116:3300/0 conn(0x7f500415c390 0x7f500415c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.267+0000 7f500b200640 1 --2- 172.21.15.47:0/2514450059 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5004145590 0x7f50040727b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.267+0000 7f500b200640 1 -- 172.21.15.47:0/2514450059 >> 172.21.15.47:0/2514450059 conn(0x7f50040b16a0 msgr2=0x7f50040b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:47.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.267+0000 7f500b200640 1 -- 172.21.15.47:0/2514450059 shutdown_connections 2024-02-19T15:23:47.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.267+0000 7f500b200640 1 -- 172.21.15.47:0/2514450059 wait complete. 2024-02-19T15:23:47.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.267+0000 7f500b200640 1 Processor -- start 2024-02-19T15:23:47.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.267+0000 7f500b200640 1 -- start start 2024-02-19T15:23:47.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f500b200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50040cea80 0x7f500415b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:47.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f500b200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5004145590 0x7f500412fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:47.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f500b200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f500415c390 0x7f5004130370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:47.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f500b200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500415b7c0 con 0x7f5004145590 2024-02-19T15:23:47.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f500b200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5004130c20 con 0x7f50040cea80 2024-02-19T15:23:47.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f500b200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5004130f00 con 0x7f500415c390 2024-02-19T15:23:47.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f4ffbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5004145590 0x7f500412fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:47.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f4ffbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5004145590 0x7f500412fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33352/0 (socket says 172.21.15.47:33352) 2024-02-19T15:23:47.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f4ffbe00640 1 -- 172.21.15.47:0/1899084543 learned_addr learned my addr 172.21.15.47:0/1899084543 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:47.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f5009200640 1 --2- 172.21.15.47:0/1899084543 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f500415c390 0x7f5004130370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:47.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f5003e00640 1 --2- 172.21.15.47:0/1899084543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50040cea80 0x7f500415b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:47.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f4ffbe00640 1 -- 172.21.15.47:0/1899084543 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f500415c390 msgr2=0x7f5004130370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:47.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f4ffbe00640 1 --2- 172.21.15.47:0/1899084543 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f500415c390 0x7f5004130370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f4ffbe00640 1 -- 172.21.15.47:0/1899084543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50040cea80 msgr2=0x7f500415b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:47.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f4ffbe00640 1 --2- 172.21.15.47:0/1899084543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50040cea80 0x7f500415b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f4ffbe00640 1 -- 172.21.15.47:0/1899084543 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc009a70 con 0x7f5004145590 2024-02-19T15:23:47.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f4ffbe00640 1 --2- 172.21.15.47:0/1899084543 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5004145590 0x7f500412fe30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00c450 tx=0x7f4fec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:47.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f5000c00640 1 -- 172.21.15.47:0/1899084543 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec005cf0 con 0x7f5004145590 2024-02-19T15:23:47.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f5000c00640 1 -- 172.21.15.47:0/1899084543 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fec005e50 con 0x7f5004145590 2024-02-19T15:23:47.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f5000c00640 1 -- 172.21.15.47:0/1899084543 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec007290 con 0x7f5004145590 2024-02-19T15:23:47.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f500b200640 1 -- 172.21.15.47:0/1899084543 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff4009420 con 0x7f5004145590 2024-02-19T15:23:47.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f500b200640 1 -- 172.21.15.47:0/1899084543 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50041314c0 con 0x7f5004145590 2024-02-19T15:23:47.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.271+0000 7f500b200640 1 -- 172.21.15.47:0/1899084543 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5004075360 con 0x7f5004145590 2024-02-19T15:23:47.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.275+0000 7f5000c00640 1 -- 172.21.15.47:0/1899084543 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4fec0073f0 con 0x7f5004145590 2024-02-19T15:23:47.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.275+0000 7f5000c00640 1 --2- 172.21.15.47:0/1899084543 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4fe0066e40 0x7f4fe0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:47.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.275+0000 7f5000c00640 1 -- 172.21.15.47:0/1899084543 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fec084090 con 0x7f5004145590 2024-02-19T15:23:47.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.275+0000 7f5003e00640 1 --2- 172.21.15.47:0/1899084543 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4fe0066e40 0x7f4fe0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:47.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.275+0000 7f5003e00640 1 --2- 172.21.15.47:0/1899084543 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4fe0066e40 0x7f4fe0069300 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40ae000 tx=0x7f4ff4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:47.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.279+0000 7f5000c00640 1 -- 172.21.15.47:0/1899084543 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f4fec050de0 con 0x7f5004145590 2024-02-19T15:23:47.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.439+0000 7f500b200640 1 -- 172.21.15.47:0/1899084543 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5004071c90 con 0x7f4fe0066e40 2024-02-19T15:23:47.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.447+0000 7f5000c00640 1 -- 172.21.15.47:0/1899084543 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288362 (secure 0 0 0) 0x7f4ff40b8430 con 0x7f4fe0066e40 2024-02-19T15:23:47.454 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:47.455 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:23:47.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.447+0000 7f500b200640 1 -- 172.21.15.47:0/1899084543 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4fe0066e40 msgr2=0x7f4fe0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:47.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.447+0000 7f500b200640 1 --2- 172.21.15.47:0/1899084543 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4fe0066e40 0x7f4fe0069300 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40ae000 tx=0x7f4ff4008040 comp rx=0 tx=0).stop 2024-02-19T15:23:47.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.447+0000 7f500b200640 1 -- 172.21.15.47:0/1899084543 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5004145590 msgr2=0x7f500412fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:47.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.447+0000 7f500b200640 1 --2- 172.21.15.47:0/1899084543 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5004145590 0x7f500412fe30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00c450 tx=0x7f4fec00c910 comp rx=0 tx=0).stop 2024-02-19T15:23:47.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.447+0000 7f500b200640 1 -- 172.21.15.47:0/1899084543 shutdown_connections 2024-02-19T15:23:47.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.447+0000 7f500b200640 1 --2- 172.21.15.47:0/1899084543 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4fe0066e40 0x7f4fe0069300 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.447+0000 7f500b200640 1 --2- 172.21.15.47:0/1899084543 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f500415c390 0x7f5004130370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.447+0000 7f500b200640 1 --2- 172.21.15.47:0/1899084543 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5004145590 0x7f500412fe30 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.447+0000 7f500b200640 1 --2- 172.21.15.47:0/1899084543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50040cea80 0x7f500415b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.447+0000 7f500b200640 1 -- 172.21.15.47:0/1899084543 >> 172.21.15.47:0/1899084543 conn(0x7f50040b16a0 msgr2=0x7f50040aff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:47.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.451+0000 7f500b200640 1 -- 172.21.15.47:0/1899084543 shutdown_connections 2024-02-19T15:23:47.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.451+0000 7f500b200640 1 -- 172.21.15.47:0/1899084543 wait complete. 2024-02-19T15:23:47.474 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-02-19T15:23:47.170538+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":246420,"kb_used_data":5276,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754728300,"statfs":{"total":773094113280,"available":772841779200,"internally_reserved":0,"allocated":5402624,"data_stored":2868958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":246795826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000943"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194769+0000","last_change":"2024-02-19T15:23:42.971601+0000","last_active":"2024-02-19T15:23:44.194769+0000","last_peered":"2024-02-19T15:23:44.194769+0000","last_clean":"2024-02-19T15:23:44.194769+0000","last_became_active":"2024-02-19T15:23:42.971304+0000","last_became_peered":"2024-02-19T15:23:42.971304+0000","last_unstale":"2024-02-19T15:23:44.194769+0000","last_undegraded":"2024-02-19T15:23:44.194769+0000","last_fullsized":"2024-02-19T15:23:44.194769+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:42:40.033469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734972+0000","last_change":"2024-02-19T15:23:42.971058+0000","last_active":"2024-02-19T15:23:44.734972+0000","last_peered":"2024-02-19T15:23:44.734972+0000","last_clean":"2024-02-19T15:23:44.734972+0000","last_became_active":"2024-02-19T15:23:42.970038+0000","last_became_peered":"2024-02-19T15:23:42.970038+0000","last_unstale":"2024-02-19T15:23:44.734972+0000","last_undegraded":"2024-02-19T15:23:44.734972+0000","last_fullsized":"2024-02-19T15:23:44.734972+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:18:23.687527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735368+0000","last_change":"2024-02-19T15:23:42.969350+0000","last_active":"2024-02-19T15:23:44.735368+0000","last_peered":"2024-02-19T15:23:44.735368+0000","last_clean":"2024-02-19T15:23:44.735368+0000","last_became_active":"2024-02-19T15:23:42.969021+0000","last_became_peered":"2024-02-19T15:23:42.969021+0000","last_unstale":"2024-02-19T15:23:44.735368+0000","last_undegraded":"2024-02-19T15:23:44.735368+0000","last_fullsized":"2024-02-19T15:23:44.735368+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:57:21.424216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946334+0000","last_change":"2024-02-19T15:23:42.962243+0000","last_active":"2024-02-19T15:23:43.946334+0000","last_peered":"2024-02-19T15:23:43.946334+0000","last_clean":"2024-02-19T15:23:43.946334+0000","last_became_active":"2024-02-19T15:23:42.961999+0000","last_became_peered":"2024-02-19T15:23:42.961999+0000","last_unstale":"2024-02-19T15:23:43.946334+0000","last_undegraded":"2024-02-19T15:23:43.946334+0000","last_fullsized":"2024-02-19T15:23:43.946334+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:25:42.171965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735126+0000","last_change":"2024-02-19T15:23:42.970850+0000","last_active":"2024-02-19T15:23:44.735126+0000","last_peered":"2024-02-19T15:23:44.735126+0000","last_clean":"2024-02-19T15:23:44.735126+0000","last_became_active":"2024-02-19T15:23:42.969695+0000","last_became_peered":"2024-02-19T15:23:42.969695+0000","last_unstale":"2024-02-19T15:23:44.735126+0000","last_undegraded":"2024-02-19T15:23:44.735126+0000","last_fullsized":"2024-02-19T15:23:44.735126+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:51:57.116389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734445+0000","last_change":"2024-02-19T15:23:42.972292+0000","last_active":"2024-02-19T15:23:44.734445+0000","last_peered":"2024-02-19T15:23:44.734445+0000","last_clean":"2024-02-19T15:23:44.734445+0000","last_became_active":"2024-02-19T15:23:42.971723+0000","last_became_peered":"2024-02-19T15:23:42.971723+0000","last_unstale":"2024-02-19T15:23:44.734445+0000","last_undegraded":"2024-02-19T15:23:44.734445+0000","last_fullsized":"2024-02-19T15:23:44.734445+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:30:47.422419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194498+0000","last_change":"2024-02-19T15:23:42.971748+0000","last_active":"2024-02-19T15:23:44.194498+0000","last_peered":"2024-02-19T15:23:44.194498+0000","last_clean":"2024-02-19T15:23:44.194498+0000","last_became_active":"2024-02-19T15:23:42.971435+0000","last_became_peered":"2024-02-19T15:23:42.971435+0000","last_unstale":"2024-02-19T15:23:44.194498+0000","last_undegraded":"2024-02-19T15:23:44.194498+0000","last_fullsized":"2024-02-19T15:23:44.194498+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:09:04.319698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945616+0000","last_change":"2024-02-19T15:23:42.967933+0000","last_active":"2024-02-19T15:23:43.945616+0000","last_peered":"2024-02-19T15:23:43.945616+0000","last_clean":"2024-02-19T15:23:43.945616+0000","last_became_active":"2024-02-19T15:23:42.967628+0000","last_became_peered":"2024-02-19T15:23:42.967628+0000","last_unstale":"2024-02-19T15:23:43.945616+0000","last_undegraded":"2024-02-19T15:23:43.945616+0000","last_fullsized":"2024-02-19T15:23:43.945616+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:23.119868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946834+0000","last_change":"2024-02-19T15:23:42.968450+0000","last_active":"2024-02-19T15:23:43.946834+0000","last_peered":"2024-02-19T15:23:43.946834+0000","last_clean":"2024-02-19T15:23:43.946834+0000","last_became_active":"2024-02-19T15:23:42.968127+0000","last_became_peered":"2024-02-19T15:23:42.968127+0000","last_unstale":"2024-02-19T15:23:43.946834+0000","last_undegraded":"2024-02-19T15:23:43.946834+0000","last_fullsized":"2024-02-19T15:23:43.946834+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:08:50.431318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946763+0000","last_change":"2024-02-19T15:23:42.967870+0000","last_active":"2024-02-19T15:23:43.946763+0000","last_peered":"2024-02-19T15:23:43.946763+0000","last_clean":"2024-02-19T15:23:43.946763+0000","last_became_active":"2024-02-19T15:23:42.967493+0000","last_became_peered":"2024-02-19T15:23:42.967493+0000","last_unstale":"2024-02-19T15:23:43.946763+0000","last_undegraded":"2024-02-19T15:23:43.946763+0000","last_fullsized":"2024-02-19T15:23:43.946763+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:41:06.816999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946394+0000","last_change":"2024-02-19T15:23:42.981481+0000","last_active":"2024-02-19T15:23:43.946394+0000","last_peered":"2024-02-19T15:23:43.946394+0000","last_clean":"2024-02-19T15:23:43.946394+0000","last_became_active":"2024-02-19T15:23:42.981177+0000","last_became_peered":"2024-02-19T15:23:42.981177+0000","last_unstale":"2024-02-19T15:23:43.946394+0000","last_undegraded":"2024-02-19T15:23:43.946394+0000","last_fullsized":"2024-02-19T15:23:43.946394+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:56:04.086784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734943+0000","last_change":"2024-02-19T15:23:42.970720+0000","last_active":"2024-02-19T15:23:44.734943+0000","last_peered":"2024-02-19T15:23:44.734943+0000","last_clean":"2024-02-19T15:23:44.734943+0000","last_became_active":"2024-02-19T15:23:42.970354+0000","last_became_peered":"2024-02-19T15:23:42.970354+0000","last_unstale":"2024-02-19T15:23:44.734943+0000","last_undegraded":"2024-02-19T15:23:44.734943+0000","last_fullsized":"2024-02-19T15:23:44.734943+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:08:01.896505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061189+0000","last_change":"2024-02-19T15:23:42.981071+0000","last_active":"2024-02-19T15:23:44.061189+0000","last_peered":"2024-02-19T15:23:44.061189+0000","last_clean":"2024-02-19T15:23:44.061189+0000","last_became_active":"2024-02-19T15:23:42.980763+0000","last_became_peered":"2024-02-19T15:23:42.980763+0000","last_unstale":"2024-02-19T15:23:44.061189+0000","last_undegraded":"2024-02-19T15:23:44.061189+0000","last_fullsized":"2024-02-19T15:23:44.061189+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:07:37.152277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946242+0000","last_change":"2024-02-19T15:23:42.979970+0000","last_active":"2024-02-19T15:23:43.946242+0000","last_peered":"2024-02-19T15:23:43.946242+0000","last_clean":"2024-02-19T15:23:43.946242+0000","last_became_active":"2024-02-19T15:23:42.968768+0000","last_became_peered":"2024-02-19T15:23:42.968768+0000","last_unstale":"2024-02-19T15:23:43.946242+0000","last_undegraded":"2024-02-19T15:23:43.946242+0000","last_fullsized":"2024-02-19T15:23:43.946242+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:18:27.465577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734344+0000","last_change":"2024-02-19T15:23:42.974598+0000","last_active":"2024-02-19T15:23:44.734344+0000","last_peered":"2024-02-19T15:23:44.734344+0000","last_clean":"2024-02-19T15:23:44.734344+0000","last_became_active":"2024-02-19T15:23:42.974335+0000","last_became_peered":"2024-02-19T15:23:42.974335+0000","last_unstale":"2024-02-19T15:23:44.734344+0000","last_undegraded":"2024-02-19T15:23:44.734344+0000","last_fullsized":"2024-02-19T15:23:44.734344+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:50:40.431301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.945858+0000","last_change":"2024-02-19T15:23:41.945858+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.945858+0000","last_undegraded":"2024-02-19T15:23:41.945858+0000","last_fullsized":"2024-02-19T15:23:41.945858+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735637+0000","last_change":"2024-02-19T15:23:42.964895+0000","last_active":"2024-02-19T15:23:44.735637+0000","last_peered":"2024-02-19T15:23:44.735637+0000","last_clean":"2024-02-19T15:23:44.735637+0000","last_became_active":"2024-02-19T15:23:42.963832+0000","last_became_peered":"2024-02-19T15:23:42.963832+0000","last_unstale":"2024-02-19T15:23:44.735637+0000","last_undegraded":"2024-02-19T15:23:44.735637+0000","last_fullsized":"2024-02-19T15:23:44.735637+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:01:27.305444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735533+0000","last_change":"2024-02-19T15:23:42.969998+0000","last_active":"2024-02-19T15:23:44.735533+0000","last_peered":"2024-02-19T15:23:44.735533+0000","last_clean":"2024-02-19T15:23:44.735533+0000","last_became_active":"2024-02-19T15:23:42.969741+0000","last_became_peered":"2024-02-19T15:23:42.969741+0000","last_unstale":"2024-02-19T15:23:44.735533+0000","last_undegraded":"2024-02-19T15:23:44.735533+0000","last_fullsized":"2024-02-19T15:23:44.735533+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:18:07.650507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.942749+0000","last_change":"2024-02-19T15:23:41.942749+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.942749+0000","last_undegraded":"2024-02-19T15:23:41.942749+0000","last_fullsized":"2024-02-19T15:23:41.942749+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061366+0000","last_change":"2024-02-19T15:23:42.986099+0000","last_active":"2024-02-19T15:23:44.061366+0000","last_peered":"2024-02-19T15:23:44.061366+0000","last_clean":"2024-02-19T15:23:44.061366+0000","last_became_active":"2024-02-19T15:23:42.985685+0000","last_became_peered":"2024-02-19T15:23:42.985685+0000","last_unstale":"2024-02-19T15:23:44.061366+0000","last_undegraded":"2024-02-19T15:23:44.061366+0000","last_fullsized":"2024-02-19T15:23:44.061366+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:13:36.682515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734637+0000","last_change":"2024-02-19T15:23:42.973184+0000","last_active":"2024-02-19T15:23:44.734637+0000","last_peered":"2024-02-19T15:23:44.734637+0000","last_clean":"2024-02-19T15:23:44.734637+0000","last_became_active":"2024-02-19T15:23:42.972228+0000","last_became_peered":"2024-02-19T15:23:42.972228+0000","last_unstale":"2024-02-19T15:23:44.734637+0000","last_undegraded":"2024-02-19T15:23:44.734637+0000","last_fullsized":"2024-02-19T15:23:44.734637+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:03.977088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946364+0000","last_change":"2024-02-19T15:23:42.979749+0000","last_active":"2024-02-19T15:23:43.946364+0000","last_peered":"2024-02-19T15:23:43.946364+0000","last_clean":"2024-02-19T15:23:43.946364+0000","last_became_active":"2024-02-19T15:23:42.968380+0000","last_became_peered":"2024-02-19T15:23:42.968380+0000","last_unstale":"2024-02-19T15:23:43.946364+0000","last_undegraded":"2024-02-19T15:23:43.946364+0000","last_fullsized":"2024-02-19T15:23:43.946364+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:04:27.961177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.942606+0000","last_change":"2024-02-19T15:23:41.942606+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.942606+0000","last_undegraded":"2024-02-19T15:23:41.942606+0000","last_fullsized":"2024-02-19T15:23:41.942606+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.945345+0000","last_change":"2024-02-19T15:23:41.945345+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.945345+0000","last_undegraded":"2024-02-19T15:23:41.945345+0000","last_fullsized":"2024-02-19T15:23:41.945345+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.945592+0000","last_change":"2024-02-19T15:23:41.945592+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.945592+0000","last_undegraded":"2024-02-19T15:23:41.945592+0000","last_fullsized":"2024-02-19T15:23:41.945592+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061361+0000","last_change":"2024-02-19T15:23:42.980908+0000","last_active":"2024-02-19T15:23:44.061361+0000","last_peered":"2024-02-19T15:23:44.061361+0000","last_clean":"2024-02-19T15:23:44.061361+0000","last_became_active":"2024-02-19T15:23:42.980592+0000","last_became_peered":"2024-02-19T15:23:42.980592+0000","last_unstale":"2024-02-19T15:23:44.061361+0000","last_undegraded":"2024-02-19T15:23:44.061361+0000","last_fullsized":"2024-02-19T15:23:44.061361+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:39:51.560827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735567+0000","last_change":"2024-02-19T15:23:42.961547+0000","last_active":"2024-02-19T15:23:44.735567+0000","last_peered":"2024-02-19T15:23:44.735567+0000","last_clean":"2024-02-19T15:23:44.735567+0000","last_became_active":"2024-02-19T15:23:42.961215+0000","last_became_peered":"2024-02-19T15:23:42.961215+0000","last_unstale":"2024-02-19T15:23:44.735567+0000","last_undegraded":"2024-02-19T15:23:44.735567+0000","last_fullsized":"2024-02-19T15:23:44.735567+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:23:12.865791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735161+0000","last_change":"2024-02-19T15:23:42.971532+0000","last_active":"2024-02-19T15:23:44.735161+0000","last_peered":"2024-02-19T15:23:44.735161+0000","last_clean":"2024-02-19T15:23:44.735161+0000","last_became_active":"2024-02-19T15:23:42.971181+0000","last_became_peered":"2024-02-19T15:23:42.971181+0000","last_unstale":"2024-02-19T15:23:44.735161+0000","last_undegraded":"2024-02-19T15:23:44.735161+0000","last_fullsized":"2024-02-19T15:23:44.735161+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:34:33.315961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.944819+0000","last_change":"2024-02-19T15:23:41.944819+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.944819+0000","last_undegraded":"2024-02-19T15:23:41.944819+0000","last_fullsized":"2024-02-19T15:23:41.944819+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734745+0000","last_change":"2024-02-19T15:23:42.961009+0000","last_active":"2024-02-19T15:23:44.734745+0000","last_peered":"2024-02-19T15:23:44.734745+0000","last_clean":"2024-02-19T15:23:44.734745+0000","last_became_active":"2024-02-19T15:23:42.960723+0000","last_became_peered":"2024-02-19T15:23:42.960723+0000","last_unstale":"2024-02-19T15:23:44.734745+0000","last_undegraded":"2024-02-19T15:23:44.734745+0000","last_fullsized":"2024-02-19T15:23:44.734745+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:57:53.597108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734459+0000","last_change":"2024-02-19T15:23:42.973247+0000","last_active":"2024-02-19T15:23:44.734459+0000","last_peered":"2024-02-19T15:23:44.734459+0000","last_clean":"2024-02-19T15:23:44.734459+0000","last_became_active":"2024-02-19T15:23:42.972957+0000","last_became_peered":"2024-02-19T15:23:42.972957+0000","last_unstale":"2024-02-19T15:23:44.734459+0000","last_undegraded":"2024-02-19T15:23:44.734459+0000","last_fullsized":"2024-02-19T15:23:44.734459+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:36:54.676262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061144+0000","last_change":"2024-02-19T15:23:42.962393+0000","last_active":"2024-02-19T15:23:44.061144+0000","last_peered":"2024-02-19T15:23:44.061144+0000","last_clean":"2024-02-19T15:23:44.061144+0000","last_became_active":"2024-02-19T15:23:42.962112+0000","last_became_peered":"2024-02-19T15:23:42.962112+0000","last_unstale":"2024-02-19T15:23:44.061144+0000","last_undegraded":"2024-02-19T15:23:44.061144+0000","last_fullsized":"2024-02-19T15:23:44.061144+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:28:48.394306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947613+0000","last_change":"2024-02-19T15:23:42.974733+0000","last_active":"2024-02-19T15:23:43.947613+0000","last_peered":"2024-02-19T15:23:43.947613+0000","last_clean":"2024-02-19T15:23:43.947613+0000","last_became_active":"2024-02-19T15:23:42.974439+0000","last_became_peered":"2024-02-19T15:23:42.974439+0000","last_unstale":"2024-02-19T15:23:43.947613+0000","last_undegraded":"2024-02-19T15:23:43.947613+0000","last_fullsized":"2024-02-19T15:23:43.947613+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:39:12.299839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734807+0000","last_change":"2024-02-19T15:23:41.960560+0000","last_active":"2024-02-19T15:23:44.734807+0000","last_peered":"2024-02-19T15:23:44.734807+0000","last_clean":"2024-02-19T15:23:44.734807+0000","last_became_active":"2024-02-19T15:23:41.960237+0000","last_became_peered":"2024-02-19T15:23:41.960237+0000","last_unstale":"2024-02-19T15:23:44.734807+0000","last_undegraded":"2024-02-19T15:23:44.734807+0000","last_fullsized":"2024-02-19T15:23:44.734807+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:14:27.569498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.944998+0000","last_change":"2024-02-19T15:23:41.944998+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.944998+0000","last_undegraded":"2024-02-19T15:23:41.944998+0000","last_fullsized":"2024-02-19T15:23:41.944998+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735605+0000","last_change":"2024-02-19T15:23:41.960962+0000","last_active":"2024-02-19T15:23:44.735605+0000","last_peered":"2024-02-19T15:23:44.735605+0000","last_clean":"2024-02-19T15:23:44.735605+0000","last_became_active":"2024-02-19T15:23:41.959425+0000","last_became_peered":"2024-02-19T15:23:41.959425+0000","last_unstale":"2024-02-19T15:23:44.735605+0000","last_undegraded":"2024-02-19T15:23:44.735605+0000","last_fullsized":"2024-02-19T15:23:44.735605+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:54.136668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735315+0000","last_change":"2024-02-19T15:23:42.972962+0000","last_active":"2024-02-19T15:23:44.735315+0000","last_peered":"2024-02-19T15:23:44.735315+0000","last_clean":"2024-02-19T15:23:44.735315+0000","last_became_active":"2024-02-19T15:23:42.972625+0000","last_became_peered":"2024-02-19T15:23:42.972625+0000","last_unstale":"2024-02-19T15:23:44.735315+0000","last_undegraded":"2024-02-19T15:23:44.735315+0000","last_fullsized":"2024-02-19T15:23:44.735315+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:40:40.055788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947602+0000","last_change":"2024-02-19T15:23:41.948520+0000","last_active":"2024-02-19T15:23:43.947602+0000","last_peered":"2024-02-19T15:23:43.947602+0000","last_clean":"2024-02-19T15:23:43.947602+0000","last_became_active":"2024-02-19T15:23:41.948233+0000","last_became_peered":"2024-02-19T15:23:41.948233+0000","last_unstale":"2024-02-19T15:23:43.947602+0000","last_undegraded":"2024-02-19T15:23:43.947602+0000","last_fullsized":"2024-02-19T15:23:43.947602+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:50:30.218103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947636+0000","last_change":"2024-02-19T15:23:42.961975+0000","last_active":"2024-02-19T15:23:43.947636+0000","last_peered":"2024-02-19T15:23:43.947636+0000","last_clean":"2024-02-19T15:23:43.947636+0000","last_became_active":"2024-02-19T15:23:42.961615+0000","last_became_peered":"2024-02-19T15:23:42.961615+0000","last_unstale":"2024-02-19T15:23:43.947636+0000","last_undegraded":"2024-02-19T15:23:43.947636+0000","last_fullsized":"2024-02-19T15:23:43.947636+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:57:21.424216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194190+0000","last_change":"2024-02-19T15:23:41.961942+0000","last_active":"2024-02-19T15:23:44.194190+0000","last_peered":"2024-02-19T15:23:44.194190+0000","last_clean":"2024-02-19T15:23:44.194190+0000","last_became_active":"2024-02-19T15:23:41.961659+0000","last_became_peered":"2024-02-19T15:23:41.961659+0000","last_unstale":"2024-02-19T15:23:44.194190+0000","last_undegraded":"2024-02-19T15:23:44.194190+0000","last_fullsized":"2024-02-19T15:23:44.194190+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:14:27.569498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734930+0000","last_change":"2024-02-19T15:23:42.961945+0000","last_active":"2024-02-19T15:23:44.734930+0000","last_peered":"2024-02-19T15:23:44.734930+0000","last_clean":"2024-02-19T15:23:44.734930+0000","last_became_active":"2024-02-19T15:23:42.961269+0000","last_became_peered":"2024-02-19T15:23:42.961269+0000","last_unstale":"2024-02-19T15:23:44.734930+0000","last_undegraded":"2024-02-19T15:23:44.734930+0000","last_fullsized":"2024-02-19T15:23:44.734930+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:39:12.299839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734672+0000","last_change":"2024-02-19T15:23:41.966788+0000","last_active":"2024-02-19T15:23:44.734672+0000","last_peered":"2024-02-19T15:23:44.734672+0000","last_clean":"2024-02-19T15:23:44.734672+0000","last_became_active":"2024-02-19T15:23:41.966346+0000","last_became_peered":"2024-02-19T15:23:41.966346+0000","last_unstale":"2024-02-19T15:23:44.734672+0000","last_undegraded":"2024-02-19T15:23:44.734672+0000","last_fullsized":"2024-02-19T15:23:44.734672+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:01:26.302862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946392+0000","last_change":"2024-02-19T15:23:42.968247+0000","last_active":"2024-02-19T15:23:43.946392+0000","last_peered":"2024-02-19T15:23:43.946392+0000","last_clean":"2024-02-19T15:23:43.946392+0000","last_became_active":"2024-02-19T15:23:42.967240+0000","last_became_peered":"2024-02-19T15:23:42.967240+0000","last_unstale":"2024-02-19T15:23:43.946392+0000","last_undegraded":"2024-02-19T15:23:43.946392+0000","last_fullsized":"2024-02-19T15:23:43.946392+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:41:08.031616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735141+0000","last_change":"2024-02-19T15:23:41.955181+0000","last_active":"2024-02-19T15:23:44.735141+0000","last_peered":"2024-02-19T15:23:44.735141+0000","last_clean":"2024-02-19T15:23:44.735141+0000","last_became_active":"2024-02-19T15:23:41.954904+0000","last_became_peered":"2024-02-19T15:23:41.954904+0000","last_unstale":"2024-02-19T15:23:44.735141+0000","last_undegraded":"2024-02-19T15:23:44.735141+0000","last_fullsized":"2024-02-19T15:23:44.735141+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:13:35.679933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945686+0000","last_change":"2024-02-19T15:23:42.980803+0000","last_active":"2024-02-19T15:23:43.945686+0000","last_peered":"2024-02-19T15:23:43.945686+0000","last_clean":"2024-02-19T15:23:43.945686+0000","last_became_active":"2024-02-19T15:23:42.980507+0000","last_became_peered":"2024-02-19T15:23:42.980507+0000","last_unstale":"2024-02-19T15:23:43.945686+0000","last_undegraded":"2024-02-19T15:23:43.945686+0000","last_fullsized":"2024-02-19T15:23:43.945686+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:08:33.495333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734933+0000","last_change":"2024-02-19T15:23:41.951702+0000","last_active":"2024-02-19T15:23:44.734933+0000","last_peered":"2024-02-19T15:23:44.734933+0000","last_clean":"2024-02-19T15:23:44.734933+0000","last_became_active":"2024-02-19T15:23:41.951409+0000","last_became_peered":"2024-02-19T15:23:41.951409+0000","last_unstale":"2024-02-19T15:23:44.734933+0000","last_undegraded":"2024-02-19T15:23:44.734933+0000","last_fullsized":"2024-02-19T15:23:44.734933+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:57:20.421633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.943222+0000","last_change":"2024-02-19T15:23:41.943222+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.943222+0000","last_undegraded":"2024-02-19T15:23:41.943222+0000","last_fullsized":"2024-02-19T15:23:41.943222+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945665+0000","last_change":"2024-02-19T15:23:41.965224+0000","last_active":"2024-02-19T15:23:43.945665+0000","last_peered":"2024-02-19T15:23:43.945665+0000","last_clean":"2024-02-19T15:23:43.945665+0000","last_became_active":"2024-02-19T15:23:41.964654+0000","last_became_peered":"2024-02-19T15:23:41.964654+0000","last_unstale":"2024-02-19T15:23:43.945665+0000","last_undegraded":"2024-02-19T15:23:43.945665+0000","last_fullsized":"2024-02-19T15:23:43.945665+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:49:19.641960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735665+0000","last_change":"2024-02-19T15:23:42.969995+0000","last_active":"2024-02-19T15:23:44.735665+0000","last_peered":"2024-02-19T15:23:44.735665+0000","last_clean":"2024-02-19T15:23:44.735665+0000","last_became_active":"2024-02-19T15:23:42.969651+0000","last_became_peered":"2024-02-19T15:23:42.969651+0000","last_unstale":"2024-02-19T15:23:44.735665+0000","last_undegraded":"2024-02-19T15:23:44.735665+0000","last_fullsized":"2024-02-19T15:23:44.735665+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:45.239277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735692+0000","last_change":"2024-02-19T15:23:41.961017+0000","last_active":"2024-02-19T15:23:44.735692+0000","last_peered":"2024-02-19T15:23:44.735692+0000","last_clean":"2024-02-19T15:23:44.735692+0000","last_became_active":"2024-02-19T15:23:41.960715+0000","last_became_peered":"2024-02-19T15:23:41.960715+0000","last_unstale":"2024-02-19T15:23:44.735692+0000","last_undegraded":"2024-02-19T15:23:44.735692+0000","last_fullsized":"2024-02-19T15:23:44.735692+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:05.009739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194763+0000","last_change":"2024-02-19T15:23:42.966631+0000","last_active":"2024-02-19T15:23:44.194763+0000","last_peered":"2024-02-19T15:23:44.194763+0000","last_clean":"2024-02-19T15:23:44.194763+0000","last_became_active":"2024-02-19T15:23:42.965314+0000","last_became_peered":"2024-02-19T15:23:42.965314+0000","last_unstale":"2024-02-19T15:23:44.194763+0000","last_undegraded":"2024-02-19T15:23:44.194763+0000","last_fullsized":"2024-02-19T15:23:44.194763+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:36:54.676262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946805+0000","last_change":"2024-02-19T15:23:41.958566+0000","last_active":"2024-02-19T15:23:43.946805+0000","last_peered":"2024-02-19T15:23:43.946805+0000","last_clean":"2024-02-19T15:23:43.946805+0000","last_became_active":"2024-02-19T15:23:41.958244+0000","last_became_peered":"2024-02-19T15:23:41.958244+0000","last_unstale":"2024-02-19T15:23:43.946805+0000","last_undegraded":"2024-02-19T15:23:43.946805+0000","last_fullsized":"2024-02-19T15:23:43.946805+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:20:44.307845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734660+0000","last_change":"2024-02-19T15:23:42.972332+0000","last_active":"2024-02-19T15:23:44.734660+0000","last_peered":"2024-02-19T15:23:44.734660+0000","last_clean":"2024-02-19T15:23:44.734660+0000","last_became_active":"2024-02-19T15:23:42.971907+0000","last_became_peered":"2024-02-19T15:23:42.971907+0000","last_unstale":"2024-02-19T15:23:44.734660+0000","last_undegraded":"2024-02-19T15:23:44.734660+0000","last_fullsized":"2024-02-19T15:23:44.734660+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:59:43.637759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194426+0000","last_change":"2024-02-19T15:23:41.964052+0000","last_active":"2024-02-19T15:23:44.194426+0000","last_peered":"2024-02-19T15:23:44.194426+0000","last_clean":"2024-02-19T15:23:44.194426+0000","last_became_active":"2024-02-19T15:23:41.963729+0000","last_became_peered":"2024-02-19T15:23:41.963729+0000","last_unstale":"2024-02-19T15:23:44.194426+0000","last_undegraded":"2024-02-19T15:23:44.194426+0000","last_fullsized":"2024-02-19T15:23:44.194426+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:02:12.172289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194355+0000","last_change":"2024-02-19T15:23:42.969755+0000","last_active":"2024-02-19T15:23:44.194355+0000","last_peered":"2024-02-19T15:23:44.194355+0000","last_clean":"2024-02-19T15:23:44.194355+0000","last_became_active":"2024-02-19T15:23:42.969477+0000","last_became_peered":"2024-02-19T15:23:42.969477+0000","last_unstale":"2024-02-19T15:23:44.194355+0000","last_undegraded":"2024-02-19T15:23:44.194355+0000","last_fullsized":"2024-02-19T15:23:44.194355+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:47:09.498437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":20,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061276+0000","last_change":"2024-02-19T15:23:41.959735+0000","last_active":"2024-02-19T15:23:44.061276+0000","last_peered":"2024-02-19T15:23:44.061276+0000","last_clean":"2024-02-19T15:23:44.061276+0000","last_became_active":"2024-02-19T15:23:41.959171+0000","last_became_peered":"2024-02-19T15:23:41.959171+0000","last_unstale":"2024-02-19T15:23:44.061276+0000","last_undegraded":"2024-02-19T15:23:44.061276+0000","last_fullsized":"2024-02-19T15:23:44.061276+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:07:41.495783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.944967+0000","last_change":"2024-02-19T15:23:41.944967+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.944967+0000","last_undegraded":"2024-02-19T15:23:41.944967+0000","last_fullsized":"2024-02-19T15:23:41.944967+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734694+0000","last_change":"2024-02-19T15:23:41.964586+0000","last_active":"2024-02-19T15:23:44.734694+0000","last_peered":"2024-02-19T15:23:44.734694+0000","last_clean":"2024-02-19T15:23:44.734694+0000","last_became_active":"2024-02-19T15:23:41.964176+0000","last_became_peered":"2024-02-19T15:23:41.964176+0000","last_unstale":"2024-02-19T15:23:44.734694+0000","last_undegraded":"2024-02-19T15:23:44.734694+0000","last_fullsized":"2024-02-19T15:23:44.734694+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:44.236695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946493+0000","last_change":"2024-02-19T15:23:42.965254+0000","last_active":"2024-02-19T15:23:43.946493+0000","last_peered":"2024-02-19T15:23:43.946493+0000","last_clean":"2024-02-19T15:23:43.946493+0000","last_became_active":"2024-02-19T15:23:42.964990+0000","last_became_peered":"2024-02-19T15:23:42.964990+0000","last_unstale":"2024-02-19T15:23:43.946493+0000","last_undegraded":"2024-02-19T15:23:43.946493+0000","last_fullsized":"2024-02-19T15:23:43.946493+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:37:19.468760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194207+0000","last_change":"2024-02-19T15:23:41.965760+0000","last_active":"2024-02-19T15:23:44.194207+0000","last_peered":"2024-02-19T15:23:44.194207+0000","last_clean":"2024-02-19T15:23:44.194207+0000","last_became_active":"2024-02-19T15:23:41.965423+0000","last_became_peered":"2024-02-19T15:23:41.965423+0000","last_unstale":"2024-02-19T15:23:44.194207+0000","last_undegraded":"2024-02-19T15:23:44.194207+0000","last_fullsized":"2024-02-19T15:23:44.194207+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:01:16.303652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734331+0000","last_change":"2024-02-19T15:23:42.970653+0000","last_active":"2024-02-19T15:23:44.734331+0000","last_peered":"2024-02-19T15:23:44.734331+0000","last_clean":"2024-02-19T15:23:44.734331+0000","last_became_active":"2024-02-19T15:23:42.970394+0000","last_became_peered":"2024-02-19T15:23:42.970394+0000","last_unstale":"2024-02-19T15:23:44.734331+0000","last_undegraded":"2024-02-19T15:23:44.734331+0000","last_fullsized":"2024-02-19T15:23:44.734331+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:09:04.319698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734401+0000","last_change":"2024-02-19T15:23:41.964786+0000","last_active":"2024-02-19T15:23:44.734401+0000","last_peered":"2024-02-19T15:23:44.734401+0000","last_clean":"2024-02-19T15:23:44.734401+0000","last_became_active":"2024-02-19T15:23:41.964085+0000","last_became_peered":"2024-02-19T15:23:41.964085+0000","last_unstale":"2024-02-19T15:23:44.734401+0000","last_undegraded":"2024-02-19T15:23:44.734401+0000","last_fullsized":"2024-02-19T15:23:44.734401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:01:16.303652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194624+0000","last_change":"2024-02-19T15:23:42.974233+0000","last_active":"2024-02-19T15:23:44.194624+0000","last_peered":"2024-02-19T15:23:44.194624+0000","last_clean":"2024-02-19T15:23:44.194624+0000","last_became_active":"2024-02-19T15:23:42.973975+0000","last_became_peered":"2024-02-19T15:23:42.973975+0000","last_unstale":"2024-02-19T15:23:44.194624+0000","last_undegraded":"2024-02-19T15:23:44.194624+0000","last_fullsized":"2024-02-19T15:23:44.194624+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:30:47.422419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":22,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945742+0000","last_change":"2024-02-19T15:23:41.957848+0000","last_active":"2024-02-19T15:23:43.945742+0000","last_peered":"2024-02-19T15:23:43.945742+0000","last_clean":"2024-02-19T15:23:43.945742+0000","last_became_active":"2024-02-19T15:23:41.957529+0000","last_became_peered":"2024-02-19T15:23:41.957529+0000","last_unstale":"2024-02-19T15:23:43.945742+0000","last_undegraded":"2024-02-19T15:23:43.945742+0000","last_fullsized":"2024-02-19T15:23:43.945742+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:41:14.948978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061427+0000","last_change":"2024-02-19T15:23:42.971155+0000","last_active":"2024-02-19T15:23:44.061427+0000","last_peered":"2024-02-19T15:23:44.061427+0000","last_clean":"2024-02-19T15:23:44.061427+0000","last_became_active":"2024-02-19T15:23:42.970879+0000","last_became_peered":"2024-02-19T15:23:42.970879+0000","last_unstale":"2024-02-19T15:23:44.061427+0000","last_undegraded":"2024-02-19T15:23:44.061427+0000","last_fullsized":"2024-02-19T15:23:44.061427+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:25:52.362277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061334+0000","last_change":"2024-02-19T15:23:41.966015+0000","last_active":"2024-02-19T15:23:44.061334+0000","last_peered":"2024-02-19T15:23:44.061334+0000","last_clean":"2024-02-19T15:23:44.061334+0000","last_became_active":"2024-02-19T15:23:41.965678+0000","last_became_peered":"2024-02-19T15:23:41.965678+0000","last_unstale":"2024-02-19T15:23:44.061334+0000","last_undegraded":"2024-02-19T15:23:44.061334+0000","last_fullsized":"2024-02-19T15:23:44.061334+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:27:46.811039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946825+0000","last_change":"2024-02-19T15:23:42.983074+0000","last_active":"2024-02-19T15:23:43.946825+0000","last_peered":"2024-02-19T15:23:43.946825+0000","last_clean":"2024-02-19T15:23:43.946825+0000","last_became_active":"2024-02-19T15:23:42.982771+0000","last_became_peered":"2024-02-19T15:23:42.982771+0000","last_unstale":"2024-02-19T15:23:43.946825+0000","last_undegraded":"2024-02-19T15:23:43.946825+0000","last_fullsized":"2024-02-19T15:23:43.946825+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:04:26.404860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735691+0000","last_change":"2024-02-19T15:23:41.963486+0000","last_active":"2024-02-19T15:23:44.735691+0000","last_peered":"2024-02-19T15:23:44.735691+0000","last_clean":"2024-02-19T15:23:44.735691+0000","last_became_active":"2024-02-19T15:23:41.962837+0000","last_became_peered":"2024-02-19T15:23:41.962837+0000","last_unstale":"2024-02-19T15:23:44.735691+0000","last_undegraded":"2024-02-19T15:23:44.735691+0000","last_fullsized":"2024-02-19T15:23:44.735691+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:16:42.881994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.944685+0000","last_change":"2024-02-19T15:23:41.944685+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.944685+0000","last_undegraded":"2024-02-19T15:23:41.944685+0000","last_fullsized":"2024-02-19T15:23:41.944685+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735151+0000","last_change":"2024-02-19T15:23:41.952606+0000","last_active":"2024-02-19T15:23:44.735151+0000","last_peered":"2024-02-19T15:23:44.735151+0000","last_clean":"2024-02-19T15:23:44.735151+0000","last_became_active":"2024-02-19T15:23:41.952291+0000","last_became_peered":"2024-02-19T15:23:41.952291+0000","last_unstale":"2024-02-19T15:23:44.735151+0000","last_undegraded":"2024-02-19T15:23:44.735151+0000","last_fullsized":"2024-02-19T15:23:44.735151+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:01:26.302862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.944426+0000","last_change":"2024-02-19T15:23:41.944426+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.944426+0000","last_undegraded":"2024-02-19T15:23:41.944426+0000","last_fullsized":"2024-02-19T15:23:41.944426+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734712+0000","last_change":"2024-02-19T15:23:41.962283+0000","last_active":"2024-02-19T15:23:44.734712+0000","last_peered":"2024-02-19T15:23:44.734712+0000","last_clean":"2024-02-19T15:23:44.734712+0000","last_became_active":"2024-02-19T15:23:41.961397+0000","last_became_peered":"2024-02-19T15:23:41.961397+0000","last_unstale":"2024-02-19T15:23:44.734712+0000","last_undegraded":"2024-02-19T15:23:44.734712+0000","last_fullsized":"2024-02-19T15:23:44.734712+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:45:37.841170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734509+0000","last_change":"2024-02-19T15:23:42.972039+0000","last_active":"2024-02-19T15:23:44.734509+0000","last_peered":"2024-02-19T15:23:44.734509+0000","last_clean":"2024-02-19T15:23:44.734509+0000","last_became_active":"2024-02-19T15:23:42.971311+0000","last_became_peered":"2024-02-19T15:23:42.971311+0000","last_unstale":"2024-02-19T15:23:44.734509+0000","last_undegraded":"2024-02-19T15:23:44.734509+0000","last_fullsized":"2024-02-19T15:23:44.734509+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:42:40.033469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734827+0000","last_change":"2024-02-19T15:23:41.960894+0000","last_active":"2024-02-19T15:23:44.734827+0000","last_peered":"2024-02-19T15:23:44.734827+0000","last_clean":"2024-02-19T15:23:44.734827+0000","last_became_active":"2024-02-19T15:23:41.959201+0000","last_became_peered":"2024-02-19T15:23:41.959201+0000","last_unstale":"2024-02-19T15:23:44.734827+0000","last_undegraded":"2024-02-19T15:23:44.734827+0000","last_fullsized":"2024-02-19T15:23:44.734827+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:32:59.830263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947496+0000","last_change":"2024-02-19T15:23:42.958733+0000","last_active":"2024-02-19T15:23:43.947496+0000","last_peered":"2024-02-19T15:23:43.947496+0000","last_clean":"2024-02-19T15:23:43.947496+0000","last_became_active":"2024-02-19T15:23:42.958345+0000","last_became_peered":"2024-02-19T15:23:42.958345+0000","last_unstale":"2024-02-19T15:23:43.947496+0000","last_undegraded":"2024-02-19T15:23:43.947496+0000","last_fullsized":"2024-02-19T15:23:43.947496+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:01.020038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734643+0000","last_change":"2024-02-19T15:23:41.961678+0000","last_active":"2024-02-19T15:23:44.734643+0000","last_peered":"2024-02-19T15:23:44.734643+0000","last_clean":"2024-02-19T15:23:44.734643+0000","last_became_active":"2024-02-19T15:23:41.960130+0000","last_became_peered":"2024-02-19T15:23:41.960130+0000","last_unstale":"2024-02-19T15:23:44.734643+0000","last_undegraded":"2024-02-19T15:23:44.734643+0000","last_fullsized":"2024-02-19T15:23:44.734643+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:18:06.647924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061083+0000","last_change":"2024-02-19T15:23:42.965223+0000","last_active":"2024-02-19T15:23:44.061083+0000","last_peered":"2024-02-19T15:23:44.061083+0000","last_clean":"2024-02-19T15:23:44.061083+0000","last_became_active":"2024-02-19T15:23:42.964860+0000","last_became_peered":"2024-02-19T15:23:42.964860+0000","last_unstale":"2024-02-19T15:23:44.061083+0000","last_undegraded":"2024-02-19T15:23:44.061083+0000","last_fullsized":"2024-02-19T15:23:44.061083+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:52.445068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945910+0000","last_change":"2024-02-19T15:23:41.966499+0000","last_active":"2024-02-19T15:23:43.945910+0000","last_peered":"2024-02-19T15:23:43.945910+0000","last_clean":"2024-02-19T15:23:43.945910+0000","last_became_active":"2024-02-19T15:23:41.966206+0000","last_became_peered":"2024-02-19T15:23:41.966206+0000","last_unstale":"2024-02-19T15:23:43.945910+0000","last_undegraded":"2024-02-19T15:23:43.945910+0000","last_fullsized":"2024-02-19T15:23:43.945910+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:50:47.865930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945811+0000","last_change":"2024-02-19T15:23:42.982135+0000","last_active":"2024-02-19T15:23:43.945811+0000","last_peered":"2024-02-19T15:23:43.945811+0000","last_clean":"2024-02-19T15:23:43.945811+0000","last_became_active":"2024-02-19T15:23:42.981890+0000","last_became_peered":"2024-02-19T15:23:42.981890+0000","last_unstale":"2024-02-19T15:23:43.945811+0000","last_undegraded":"2024-02-19T15:23:43.945811+0000","last_fullsized":"2024-02-19T15:23:43.945811+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:28.558554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194347+0000","last_change":"2024-02-19T15:23:41.964986+0000","last_active":"2024-02-19T15:23:44.194347+0000","last_peered":"2024-02-19T15:23:44.194347+0000","last_clean":"2024-02-19T15:23:44.194347+0000","last_became_active":"2024-02-19T15:23:41.964713+0000","last_became_peered":"2024-02-19T15:23:41.964713+0000","last_unstale":"2024-02-19T15:23:44.194347+0000","last_undegraded":"2024-02-19T15:23:44.194347+0000","last_fullsized":"2024-02-19T15:23:44.194347+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:32:59.830263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.943060+0000","last_change":"2024-02-19T15:23:41.943060+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.943060+0000","last_undegraded":"2024-02-19T15:23:41.943060+0000","last_fullsized":"2024-02-19T15:23:41.943060+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"45'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734750+0000","last_change":"2024-02-19T15:23:41.968055+0000","last_active":"2024-02-19T15:23:44.734750+0000","last_peered":"2024-02-19T15:23:44.734750+0000","last_clean":"2024-02-19T15:23:44.734750+0000","last_became_active":"2024-02-19T15:23:41.967728+0000","last_became_peered":"2024-02-19T15:23:41.967728+0000","last_unstale":"2024-02-19T15:23:44.734750+0000","last_undegraded":"2024-02-19T15:23:44.734750+0000","last_fullsized":"2024-02-19T15:23:44.734750+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:02:12.172289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946492+0000","last_change":"2024-02-19T15:23:42.966169+0000","last_active":"2024-02-19T15:23:43.946492+0000","last_peered":"2024-02-19T15:23:43.946492+0000","last_clean":"2024-02-19T15:23:43.946492+0000","last_became_active":"2024-02-19T15:23:42.965898+0000","last_became_peered":"2024-02-19T15:23:42.965898+0000","last_unstale":"2024-02-19T15:23:43.946492+0000","last_undegraded":"2024-02-19T15:23:43.946492+0000","last_fullsized":"2024-02-19T15:23:43.946492+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:41:45.823238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194546+0000","last_change":"2024-02-19T15:23:41.968087+0000","last_active":"2024-02-19T15:23:44.194546+0000","last_peered":"2024-02-19T15:23:44.194546+0000","last_clean":"2024-02-19T15:23:44.194546+0000","last_became_active":"2024-02-19T15:23:41.967798+0000","last_became_peered":"2024-02-19T15:23:41.967798+0000","last_unstale":"2024-02-19T15:23:44.194546+0000","last_undegraded":"2024-02-19T15:23:44.194546+0000","last_fullsized":"2024-02-19T15:23:44.194546+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:08:00.893923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194536+0000","last_change":"2024-02-19T15:23:42.971627+0000","last_active":"2024-02-19T15:23:44.194536+0000","last_peered":"2024-02-19T15:23:44.194536+0000","last_clean":"2024-02-19T15:23:44.194536+0000","last_became_active":"2024-02-19T15:23:42.971344+0000","last_became_peered":"2024-02-19T15:23:42.971344+0000","last_unstale":"2024-02-19T15:23:44.194536+0000","last_undegraded":"2024-02-19T15:23:44.194536+0000","last_fullsized":"2024-02-19T15:23:44.194536+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:50:40.431301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947382+0000","last_change":"2024-02-19T15:23:41.948963+0000","last_active":"2024-02-19T15:23:43.947382+0000","last_peered":"2024-02-19T15:23:43.947382+0000","last_clean":"2024-02-19T15:23:43.947382+0000","last_became_active":"2024-02-19T15:23:41.947243+0000","last_became_peered":"2024-02-19T15:23:41.947243+0000","last_unstale":"2024-02-19T15:23:43.947382+0000","last_undegraded":"2024-02-19T15:23:43.947382+0000","last_fullsized":"2024-02-19T15:23:43.947382+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:51:31.286375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734698+0000","last_change":"2024-02-19T15:23:42.974990+0000","last_active":"2024-02-19T15:23:44.734698+0000","last_peered":"2024-02-19T15:23:44.734698+0000","last_clean":"2024-02-19T15:23:44.734698+0000","last_became_active":"2024-02-19T15:23:42.974011+0000","last_became_peered":"2024-02-19T15:23:42.974011+0000","last_unstale":"2024-02-19T15:23:44.734698+0000","last_undegraded":"2024-02-19T15:23:44.734698+0000","last_fullsized":"2024-02-19T15:23:44.734698+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:48.375387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194296+0000","last_change":"2024-02-19T15:23:41.964025+0000","last_active":"2024-02-19T15:23:44.194296+0000","last_peered":"2024-02-19T15:23:44.194296+0000","last_clean":"2024-02-19T15:23:44.194296+0000","last_became_active":"2024-02-19T15:23:41.963717+0000","last_became_peered":"2024-02-19T15:23:41.963717+0000","last_unstale":"2024-02-19T15:23:44.194296+0000","last_undegraded":"2024-02-19T15:23:44.194296+0000","last_fullsized":"2024-02-19T15:23:44.194296+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:45:37.841170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734618+0000","last_change":"2024-02-19T15:23:42.974421+0000","last_active":"2024-02-19T15:23:44.734618+0000","last_peered":"2024-02-19T15:23:44.734618+0000","last_clean":"2024-02-19T15:23:44.734618+0000","last_became_active":"2024-02-19T15:23:42.974009+0000","last_became_peered":"2024-02-19T15:23:42.974009+0000","last_unstale":"2024-02-19T15:23:44.734618+0000","last_undegraded":"2024-02-19T15:23:44.734618+0000","last_fullsized":"2024-02-19T15:23:44.734618+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:36.710691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735244+0000","last_change":"2024-02-19T15:23:41.969004+0000","last_active":"2024-02-19T15:23:44.735244+0000","last_peered":"2024-02-19T15:23:44.735244+0000","last_clean":"2024-02-19T15:23:44.735244+0000","last_became_active":"2024-02-19T15:23:41.968796+0000","last_became_peered":"2024-02-19T15:23:41.968796+0000","last_unstale":"2024-02-19T15:23:44.735244+0000","last_undegraded":"2024-02-19T15:23:44.735244+0000","last_fullsized":"2024-02-19T15:23:44.735244+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:25:51.359695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735100+0000","last_change":"2024-02-19T15:23:42.956889+0000","last_active":"2024-02-19T15:23:44.735100+0000","last_peered":"2024-02-19T15:23:44.735100+0000","last_clean":"2024-02-19T15:23:44.735100+0000","last_became_active":"2024-02-19T15:23:42.953555+0000","last_became_peered":"2024-02-19T15:23:42.953555+0000","last_unstale":"2024-02-19T15:23:44.735100+0000","last_undegraded":"2024-02-19T15:23:44.735100+0000","last_fullsized":"2024-02-19T15:23:44.735100+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:50:45.305646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947245+0000","last_change":"2024-02-19T15:23:41.956700+0000","last_active":"2024-02-19T15:23:43.947245+0000","last_peered":"2024-02-19T15:23:43.947245+0000","last_clean":"2024-02-19T15:23:43.947245+0000","last_became_active":"2024-02-19T15:23:41.956388+0000","last_became_peered":"2024-02-19T15:23:41.956388+0000","last_unstale":"2024-02-19T15:23:43.947245+0000","last_undegraded":"2024-02-19T15:23:43.947245+0000","last_fullsized":"2024-02-19T15:23:43.947245+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:15:40.531978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734808+0000","last_change":"2024-02-19T15:23:42.976054+0000","last_active":"2024-02-19T15:23:44.734808+0000","last_peered":"2024-02-19T15:23:44.734808+0000","last_clean":"2024-02-19T15:23:44.734808+0000","last_became_active":"2024-02-19T15:23:42.975795+0000","last_became_peered":"2024-02-19T15:23:42.975795+0000","last_unstale":"2024-02-19T15:23:44.734808+0000","last_undegraded":"2024-02-19T15:23:44.734808+0000","last_fullsized":"2024-02-19T15:23:44.734808+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:01.020038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945761+0000","last_change":"2024-02-19T15:23:41.962391+0000","last_active":"2024-02-19T15:23:43.945761+0000","last_peered":"2024-02-19T15:23:43.945761+0000","last_clean":"2024-02-19T15:23:43.945761+0000","last_became_active":"2024-02-19T15:23:41.961437+0000","last_became_peered":"2024-02-19T15:23:41.961437+0000","last_unstale":"2024-02-19T15:23:43.945761+0000","last_undegraded":"2024-02-19T15:23:43.945761+0000","last_fullsized":"2024-02-19T15:23:43.945761+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:16:05.431843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946756+0000","last_change":"2024-02-19T15:23:42.964218+0000","last_active":"2024-02-19T15:23:43.946756+0000","last_peered":"2024-02-19T15:23:43.946756+0000","last_clean":"2024-02-19T15:23:43.946756+0000","last_became_active":"2024-02-19T15:23:42.963959+0000","last_became_peered":"2024-02-19T15:23:42.963959+0000","last_unstale":"2024-02-19T15:23:43.946756+0000","last_undegraded":"2024-02-19T15:23:43.946756+0000","last_fullsized":"2024-02-19T15:23:43.946756+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:43:11.719148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735163+0000","last_change":"2024-02-19T15:23:41.960473+0000","last_active":"2024-02-19T15:23:44.735163+0000","last_peered":"2024-02-19T15:23:44.735163+0000","last_clean":"2024-02-19T15:23:44.735163+0000","last_became_active":"2024-02-19T15:23:41.960208+0000","last_became_peered":"2024-02-19T15:23:41.960208+0000","last_unstale":"2024-02-19T15:23:44.735163+0000","last_undegraded":"2024-02-19T15:23:44.735163+0000","last_fullsized":"2024-02-19T15:23:44.735163+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:23:11.863209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":212,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194375+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:23:44.194375+0000","last_peered":"2024-02-19T15:23:44.194375+0000","last_clean":"2024-02-19T15:23:44.194375+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:23:44.194375+0000","last_undegraded":"2024-02-19T15:23:44.194375+0000","last_fullsized":"2024-02-19T15:23:44.194375+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509465,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":487424,"data_stored":193585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":4.6029999999999998,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.79400000000000004}]},{"osd":1,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.625},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.71699999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.73999999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.75600000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.64900000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.81000000000000005}]}]},{"osd":7,"up_from":14,"seq":60129542170,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28124,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343716,"statfs":{"total":96636764160,"available":96607965184,"internally_reserved":0,"allocated":1122304,"data_stored":803385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.627},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.87}]},{"osd":1,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":2.968,"5min":2.968,"15min":2.968},"last":0.754}]},{"osd":2,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":0.71099999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":3.1720000000000002,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":0.77300000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.73699999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.83199999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.93799999999999994}]}]},{"osd":5,"up_from":14,"seq":60129542171,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27536,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344304,"statfs":{"total":96636764160,"available":96608567296,"internally_reserved":0,"allocated":528384,"data_stored":212997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.56999999999999995}]},{"osd":1,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.85499999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.66300000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.63900000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":0.67600000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.89400000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.59899999999999998}]}]},{"osd":4,"up_from":14,"seq":60129542169,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27544,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344296,"statfs":{"total":96636764160,"available":96608559104,"internally_reserved":0,"allocated":536576,"data_stored":214539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.72899999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.623}]},{"osd":2,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.755}]},{"osd":3,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.49199999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.80200000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.78800000000000003}]},{"osd":7,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.82199999999999995}]}]},{"osd":3,"up_from":14,"seq":60129542169,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":536576,"data_stored":213565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.2450000000000001,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.33500000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":4.0220000000000002,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.52900000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.72899999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":4.2039999999999997,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.78400000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":4.1589999999999998,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.627}]},{"osd":6,"last update":"Mon Feb 19 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":4.0940000000000003,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.69499999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":3.9660000000000002,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.67500000000000004}]}]},{"osd":2,"up_from":14,"seq":60129542170,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":540672,"data_stored":214561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.7160000000000002,"5min":4.7160000000000002,"15min":4.7160000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":8.2129999999999992,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.26000000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":8.1620000000000008,"5min":8.1620000000000008,"15min":8.1620000000000008},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.46100000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":4.7679999999999998,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.371},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":4.9909999999999997,"5min":4.9909999999999997,"15min":4.9909999999999997},"last":0.443}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":4.3810000000000002,"5min":4.3810000000000002,"15min":4.3810000000000002},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":4.3440000000000003,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.83199999999999996}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":4.4039999999999999,"5min":4.4039999999999999,"15min":4.4039999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":4.3259999999999996,"5min":4.3259999999999996,"15min":4.3259999999999996},"last":0.75}]},{"osd":6,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":4.5220000000000002,"5min":4.5220000000000002,"15min":4.5220000000000002},"last":0.71699999999999997},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":4.4320000000000004,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.73899999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":4.4829999999999997,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":4.4989999999999997,"5min":4.4989999999999997,"15min":4.4989999999999997},"last":0.84599999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542171,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":528384,"data_stored":212997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.25}]},{"osd":2,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.41599999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.438},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.38300000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.65700000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.746},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.77500000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.72699999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.70499999999999996}]}]},{"osd":0,"up_from":14,"seq":60129542169,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36436,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335404,"statfs":{"total":96636764160,"available":96599453696,"internally_reserved":0,"allocated":1122304,"data_stored":803329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.0199999999999996,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.31}]},{"osd":2,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":5.6669999999999998,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.4649999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.45700000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.9539999999999997,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.345,"5min":1.345,"15min":1.345},"last":0.44800000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":4.7880000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.61799999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.53200000000000003}]},{"osd":6,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":4.8460000000000001,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.65300000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":4.9809999999999999,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":4.915,"5min":4.915,"15min":4.915},"last":0.56200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:23:47.479 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:23:47.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.655+0000 7f4189800640 1 Processor -- start 2024-02-19T15:23:47.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.655+0000 7f4189800640 1 -- start start 2024-02-19T15:23:47.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.655+0000 7f4189800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4184145590 0x7f41840727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:47.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.655+0000 7f4189800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f418415c390 0x7f418415c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:47.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.655+0000 7f4189800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418415d280 con 0x7f41840cea80 2024-02-19T15:23:47.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.655+0000 7f4189800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4184070ff0 con 0x7f418415c390 2024-02-19T15:23:47.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.655+0000 7f4189800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f418415cd40 con 0x7f4184145590 2024-02-19T15:23:47.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.655+0000 7f4183400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f418415c390 0x7f418415c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:47.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.659+0000 7f4183400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f418415c390 0x7f418415c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33364/0 (socket says 172.21.15.47:33364) 2024-02-19T15:23:47.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.659+0000 7f4183400640 1 -- 172.21.15.47:0/2783712893 learned_addr learned my addr 172.21.15.47:0/2783712893 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:47.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.659+0000 7f4182a00640 1 --2- 172.21.15.47:0/2783712893 >> v2:172.21.15.116:3300/0 conn(0x7f4184145590 0x7f41840727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:47.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.659+0000 7f4183400640 1 -- 172.21.15.47:0/2783712893 >> v2:172.21.15.116:3300/0 conn(0x7f4184145590 msgr2=0x7f41840727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:47.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.659+0000 7f4183400640 1 --2- 172.21.15.47:0/2783712893 >> v2:172.21.15.116:3300/0 conn(0x7f4184145590 0x7f41840727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.659+0000 7f4183400640 1 -- 172.21.15.47:0/2783712893 >> v1:172.21.15.116:6789/0 conn(0x7f41840cea80 legacy=0x7f418406c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:47.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.659+0000 7f4183400640 1 -- 172.21.15.47:0/2783712893 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41840709e0 con 0x7f418415c390 2024-02-19T15:23:47.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.659+0000 7f4183400640 1 --2- 172.21.15.47:0/2783712893 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f418415c390 0x7f418415c800 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4184070560 tx=0x7f417c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28ff285aeb68e8a8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:47.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.659+0000 7f4182000640 1 -- 172.21.15.47:0/2783712893 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c0a57a0 con 0x7f418415c390 2024-02-19T15:23:47.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.659+0000 7f4182000640 1 -- 172.21.15.47:0/2783712893 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c0a5900 con 0x7f418415c390 2024-02-19T15:23:47.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.659+0000 7f4182000640 1 -- 172.21.15.47:0/2783712893 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c0a5c90 con 0x7f418415c390 2024-02-19T15:23:47.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4189800640 1 -- 172.21.15.47:0/2783712893 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f418415c390 msgr2=0x7f418415c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:47.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4189800640 1 --2- 172.21.15.47:0/2783712893 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f418415c390 0x7f418415c800 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4184070560 tx=0x7f417c01c9f0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4189800640 1 -- 172.21.15.47:0/2783712893 shutdown_connections 2024-02-19T15:23:47.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4189800640 1 --2- 172.21.15.47:0/2783712893 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f418415c390 0x7f418415c800 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4189800640 1 --2- 172.21.15.47:0/2783712893 >> v2:172.21.15.116:3300/0 conn(0x7f4184145590 0x7f41840727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4189800640 1 -- 172.21.15.47:0/2783712893 >> 172.21.15.47:0/2783712893 conn(0x7f41840b16a0 msgr2=0x7f41840b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:47.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4189800640 1 -- 172.21.15.47:0/2783712893 shutdown_connections 2024-02-19T15:23:47.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4189800640 1 -- 172.21.15.47:0/2783712893 wait complete. 2024-02-19T15:23:47.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4189800640 1 Processor -- start 2024-02-19T15:23:47.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4189800640 1 -- start start 2024-02-19T15:23:47.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4189800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41840cea80 0x7f418413c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:47.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4189800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4184145590 0x7f41840b2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:47.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4189800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f418415c390 0x7f4184142790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:47.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4189800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41840b4080 con 0x7f41840cea80 2024-02-19T15:23:47.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4189800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41840b44a0 con 0x7f418415c390 2024-02-19T15:23:47.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4189800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41840b3400 con 0x7f4184145590 2024-02-19T15:23:47.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4182a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41840cea80 0x7f418413c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:47.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4182a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41840cea80 0x7f418413c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33380/0 (socket says 172.21.15.47:33380) 2024-02-19T15:23:47.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4182a00640 1 -- 172.21.15.47:0/2366686997 learned_addr learned my addr 172.21.15.47:0/2366686997 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:47.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4183400640 1 --2- 172.21.15.47:0/2366686997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f418415c390 0x7f4184142790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:47.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f417be00640 1 --2- 172.21.15.47:0/2366686997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4184145590 0x7f41840b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:47.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4182a00640 1 -- 172.21.15.47:0/2366686997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4184145590 msgr2=0x7f41840b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:47.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4182a00640 1 --2- 172.21.15.47:0/2366686997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4184145590 0x7f41840b2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4182a00640 1 -- 172.21.15.47:0/2366686997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f418415c390 msgr2=0x7f4184142790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:47.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4182a00640 1 --2- 172.21.15.47:0/2366686997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f418415c390 0x7f4184142790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.663+0000 7f4182a00640 1 -- 172.21.15.47:0/2366686997 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417c009b30 con 0x7f41840cea80 2024-02-19T15:23:47.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.667+0000 7f4182a00640 1 --2- 172.21.15.47:0/2366686997 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41840cea80 0x7f418413c3e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f417400e9e0 tx=0x7f417400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:47.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.667+0000 7f417aa00640 1 -- 172.21.15.47:0/2366686997 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41740096b0 con 0x7f41840cea80 2024-02-19T15:23:47.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.667+0000 7f417aa00640 1 -- 172.21.15.47:0/2366686997 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417400c9d0 con 0x7f41840cea80 2024-02-19T15:23:47.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.667+0000 7f417aa00640 1 -- 172.21.15.47:0/2366686997 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417400cd60 con 0x7f41840cea80 2024-02-19T15:23:47.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.667+0000 7f4189800640 1 -- 172.21.15.47:0/2366686997 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41840b36e0 con 0x7f41840cea80 2024-02-19T15:23:47.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.667+0000 7f4189800640 1 -- 172.21.15.47:0/2366686997 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41840b3ca0 con 0x7f41840cea80 2024-02-19T15:23:47.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.667+0000 7f4189800640 1 -- 172.21.15.47:0/2366686997 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41840b42f0 con 0x7f41840cea80 2024-02-19T15:23:47.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.667+0000 7f417aa00640 1 -- 172.21.15.47:0/2366686997 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4174013470 con 0x7f41840cea80 2024-02-19T15:23:47.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.667+0000 7f417aa00640 1 --2- 172.21.15.47:0/2366686997 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4160066e40 0x7f4160069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:47.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.667+0000 7f417aa00640 1 -- 172.21.15.47:0/2366686997 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4174084c30 con 0x7f41840cea80 2024-02-19T15:23:47.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.667+0000 7f417be00640 1 --2- 172.21.15.47:0/2366686997 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4160066e40 0x7f4160069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:47.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.671+0000 7f417be00640 1 --2- 172.21.15.47:0/2366686997 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4160066e40 0x7f4160069300 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f418413de40 tx=0x7f416c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:47.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.671+0000 7f417aa00640 1 -- 172.21.15.47:0/2366686997 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f4174051ad0 con 0x7f41840cea80 2024-02-19T15:23:47.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.831+0000 7f4189800640 1 -- 172.21.15.47:0/2366686997 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4184071c70 con 0x7f4160066e40 2024-02-19T15:23:47.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.839+0000 7f417aa00640 1 -- 172.21.15.47:0/2366686997 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288362 (secure 0 0 0) 0x7f416c001450 con 0x7f4160066e40 2024-02-19T15:23:47.848 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:47.850 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:23:47.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.843+0000 7f4189800640 1 -- 172.21.15.47:0/2366686997 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4160066e40 msgr2=0x7f4160069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:47.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.843+0000 7f4189800640 1 --2- 172.21.15.47:0/2366686997 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4160066e40 0x7f4160069300 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f418413de40 tx=0x7f416c009040 comp rx=0 tx=0).stop 2024-02-19T15:23:47.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.843+0000 7f4189800640 1 -- 172.21.15.47:0/2366686997 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41840cea80 msgr2=0x7f418413c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:47.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.843+0000 7f4189800640 1 --2- 172.21.15.47:0/2366686997 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41840cea80 0x7f418413c3e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f417400e9e0 tx=0x7f417400eea0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.843+0000 7f4189800640 1 -- 172.21.15.47:0/2366686997 shutdown_connections 2024-02-19T15:23:47.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.843+0000 7f4189800640 1 --2- 172.21.15.47:0/2366686997 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4160066e40 0x7f4160069300 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.843+0000 7f4189800640 1 --2- 172.21.15.47:0/2366686997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f418415c390 0x7f4184142790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.843+0000 7f4189800640 1 --2- 172.21.15.47:0/2366686997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4184145590 0x7f41840b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.843+0000 7f4189800640 1 --2- 172.21.15.47:0/2366686997 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41840cea80 0x7f418413c3e0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:47.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.843+0000 7f4189800640 1 -- 172.21.15.47:0/2366686997 >> 172.21.15.47:0/2366686997 conn(0x7f41840b16a0 msgr2=0x7f41840b0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:47.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.843+0000 7f4189800640 1 -- 172.21.15.47:0/2366686997 shutdown_connections 2024-02-19T15:23:47.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:47.843+0000 7f4189800640 1 -- 172.21.15.47:0/2366686997 wait complete. 2024-02-19T15:23:47.870 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-02-19T15:23:47.170538+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":246420,"kb_used_data":5276,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754728300,"statfs":{"total":773094113280,"available":772841779200,"internally_reserved":0,"allocated":5402624,"data_stored":2868958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":246795826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000943"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194769+0000","last_change":"2024-02-19T15:23:42.971601+0000","last_active":"2024-02-19T15:23:44.194769+0000","last_peered":"2024-02-19T15:23:44.194769+0000","last_clean":"2024-02-19T15:23:44.194769+0000","last_became_active":"2024-02-19T15:23:42.971304+0000","last_became_peered":"2024-02-19T15:23:42.971304+0000","last_unstale":"2024-02-19T15:23:44.194769+0000","last_undegraded":"2024-02-19T15:23:44.194769+0000","last_fullsized":"2024-02-19T15:23:44.194769+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:42:40.033469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734972+0000","last_change":"2024-02-19T15:23:42.971058+0000","last_active":"2024-02-19T15:23:44.734972+0000","last_peered":"2024-02-19T15:23:44.734972+0000","last_clean":"2024-02-19T15:23:44.734972+0000","last_became_active":"2024-02-19T15:23:42.970038+0000","last_became_peered":"2024-02-19T15:23:42.970038+0000","last_unstale":"2024-02-19T15:23:44.734972+0000","last_undegraded":"2024-02-19T15:23:44.734972+0000","last_fullsized":"2024-02-19T15:23:44.734972+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:18:23.687527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735368+0000","last_change":"2024-02-19T15:23:42.969350+0000","last_active":"2024-02-19T15:23:44.735368+0000","last_peered":"2024-02-19T15:23:44.735368+0000","last_clean":"2024-02-19T15:23:44.735368+0000","last_became_active":"2024-02-19T15:23:42.969021+0000","last_became_peered":"2024-02-19T15:23:42.969021+0000","last_unstale":"2024-02-19T15:23:44.735368+0000","last_undegraded":"2024-02-19T15:23:44.735368+0000","last_fullsized":"2024-02-19T15:23:44.735368+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:57:21.424216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946334+0000","last_change":"2024-02-19T15:23:42.962243+0000","last_active":"2024-02-19T15:23:43.946334+0000","last_peered":"2024-02-19T15:23:43.946334+0000","last_clean":"2024-02-19T15:23:43.946334+0000","last_became_active":"2024-02-19T15:23:42.961999+0000","last_became_peered":"2024-02-19T15:23:42.961999+0000","last_unstale":"2024-02-19T15:23:43.946334+0000","last_undegraded":"2024-02-19T15:23:43.946334+0000","last_fullsized":"2024-02-19T15:23:43.946334+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:25:42.171965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735126+0000","last_change":"2024-02-19T15:23:42.970850+0000","last_active":"2024-02-19T15:23:44.735126+0000","last_peered":"2024-02-19T15:23:44.735126+0000","last_clean":"2024-02-19T15:23:44.735126+0000","last_became_active":"2024-02-19T15:23:42.969695+0000","last_became_peered":"2024-02-19T15:23:42.969695+0000","last_unstale":"2024-02-19T15:23:44.735126+0000","last_undegraded":"2024-02-19T15:23:44.735126+0000","last_fullsized":"2024-02-19T15:23:44.735126+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:51:57.116389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734445+0000","last_change":"2024-02-19T15:23:42.972292+0000","last_active":"2024-02-19T15:23:44.734445+0000","last_peered":"2024-02-19T15:23:44.734445+0000","last_clean":"2024-02-19T15:23:44.734445+0000","last_became_active":"2024-02-19T15:23:42.971723+0000","last_became_peered":"2024-02-19T15:23:42.971723+0000","last_unstale":"2024-02-19T15:23:44.734445+0000","last_undegraded":"2024-02-19T15:23:44.734445+0000","last_fullsized":"2024-02-19T15:23:44.734445+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:30:47.422419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194498+0000","last_change":"2024-02-19T15:23:42.971748+0000","last_active":"2024-02-19T15:23:44.194498+0000","last_peered":"2024-02-19T15:23:44.194498+0000","last_clean":"2024-02-19T15:23:44.194498+0000","last_became_active":"2024-02-19T15:23:42.971435+0000","last_became_peered":"2024-02-19T15:23:42.971435+0000","last_unstale":"2024-02-19T15:23:44.194498+0000","last_undegraded":"2024-02-19T15:23:44.194498+0000","last_fullsized":"2024-02-19T15:23:44.194498+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:09:04.319698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945616+0000","last_change":"2024-02-19T15:23:42.967933+0000","last_active":"2024-02-19T15:23:43.945616+0000","last_peered":"2024-02-19T15:23:43.945616+0000","last_clean":"2024-02-19T15:23:43.945616+0000","last_became_active":"2024-02-19T15:23:42.967628+0000","last_became_peered":"2024-02-19T15:23:42.967628+0000","last_unstale":"2024-02-19T15:23:43.945616+0000","last_undegraded":"2024-02-19T15:23:43.945616+0000","last_fullsized":"2024-02-19T15:23:43.945616+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:23.119868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946834+0000","last_change":"2024-02-19T15:23:42.968450+0000","last_active":"2024-02-19T15:23:43.946834+0000","last_peered":"2024-02-19T15:23:43.946834+0000","last_clean":"2024-02-19T15:23:43.946834+0000","last_became_active":"2024-02-19T15:23:42.968127+0000","last_became_peered":"2024-02-19T15:23:42.968127+0000","last_unstale":"2024-02-19T15:23:43.946834+0000","last_undegraded":"2024-02-19T15:23:43.946834+0000","last_fullsized":"2024-02-19T15:23:43.946834+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:08:50.431318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946763+0000","last_change":"2024-02-19T15:23:42.967870+0000","last_active":"2024-02-19T15:23:43.946763+0000","last_peered":"2024-02-19T15:23:43.946763+0000","last_clean":"2024-02-19T15:23:43.946763+0000","last_became_active":"2024-02-19T15:23:42.967493+0000","last_became_peered":"2024-02-19T15:23:42.967493+0000","last_unstale":"2024-02-19T15:23:43.946763+0000","last_undegraded":"2024-02-19T15:23:43.946763+0000","last_fullsized":"2024-02-19T15:23:43.946763+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:41:06.816999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946394+0000","last_change":"2024-02-19T15:23:42.981481+0000","last_active":"2024-02-19T15:23:43.946394+0000","last_peered":"2024-02-19T15:23:43.946394+0000","last_clean":"2024-02-19T15:23:43.946394+0000","last_became_active":"2024-02-19T15:23:42.981177+0000","last_became_peered":"2024-02-19T15:23:42.981177+0000","last_unstale":"2024-02-19T15:23:43.946394+0000","last_undegraded":"2024-02-19T15:23:43.946394+0000","last_fullsized":"2024-02-19T15:23:43.946394+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:56:04.086784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734943+0000","last_change":"2024-02-19T15:23:42.970720+0000","last_active":"2024-02-19T15:23:44.734943+0000","last_peered":"2024-02-19T15:23:44.734943+0000","last_clean":"2024-02-19T15:23:44.734943+0000","last_became_active":"2024-02-19T15:23:42.970354+0000","last_became_peered":"2024-02-19T15:23:42.970354+0000","last_unstale":"2024-02-19T15:23:44.734943+0000","last_undegraded":"2024-02-19T15:23:44.734943+0000","last_fullsized":"2024-02-19T15:23:44.734943+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:08:01.896505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061189+0000","last_change":"2024-02-19T15:23:42.981071+0000","last_active":"2024-02-19T15:23:44.061189+0000","last_peered":"2024-02-19T15:23:44.061189+0000","last_clean":"2024-02-19T15:23:44.061189+0000","last_became_active":"2024-02-19T15:23:42.980763+0000","last_became_peered":"2024-02-19T15:23:42.980763+0000","last_unstale":"2024-02-19T15:23:44.061189+0000","last_undegraded":"2024-02-19T15:23:44.061189+0000","last_fullsized":"2024-02-19T15:23:44.061189+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:07:37.152277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946242+0000","last_change":"2024-02-19T15:23:42.979970+0000","last_active":"2024-02-19T15:23:43.946242+0000","last_peered":"2024-02-19T15:23:43.946242+0000","last_clean":"2024-02-19T15:23:43.946242+0000","last_became_active":"2024-02-19T15:23:42.968768+0000","last_became_peered":"2024-02-19T15:23:42.968768+0000","last_unstale":"2024-02-19T15:23:43.946242+0000","last_undegraded":"2024-02-19T15:23:43.946242+0000","last_fullsized":"2024-02-19T15:23:43.946242+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:18:27.465577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734344+0000","last_change":"2024-02-19T15:23:42.974598+0000","last_active":"2024-02-19T15:23:44.734344+0000","last_peered":"2024-02-19T15:23:44.734344+0000","last_clean":"2024-02-19T15:23:44.734344+0000","last_became_active":"2024-02-19T15:23:42.974335+0000","last_became_peered":"2024-02-19T15:23:42.974335+0000","last_unstale":"2024-02-19T15:23:44.734344+0000","last_undegraded":"2024-02-19T15:23:44.734344+0000","last_fullsized":"2024-02-19T15:23:44.734344+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:50:40.431301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.945858+0000","last_change":"2024-02-19T15:23:41.945858+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.945858+0000","last_undegraded":"2024-02-19T15:23:41.945858+0000","last_fullsized":"2024-02-19T15:23:41.945858+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735637+0000","last_change":"2024-02-19T15:23:42.964895+0000","last_active":"2024-02-19T15:23:44.735637+0000","last_peered":"2024-02-19T15:23:44.735637+0000","last_clean":"2024-02-19T15:23:44.735637+0000","last_became_active":"2024-02-19T15:23:42.963832+0000","last_became_peered":"2024-02-19T15:23:42.963832+0000","last_unstale":"2024-02-19T15:23:44.735637+0000","last_undegraded":"2024-02-19T15:23:44.735637+0000","last_fullsized":"2024-02-19T15:23:44.735637+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:01:27.305444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735533+0000","last_change":"2024-02-19T15:23:42.969998+0000","last_active":"2024-02-19T15:23:44.735533+0000","last_peered":"2024-02-19T15:23:44.735533+0000","last_clean":"2024-02-19T15:23:44.735533+0000","last_became_active":"2024-02-19T15:23:42.969741+0000","last_became_peered":"2024-02-19T15:23:42.969741+0000","last_unstale":"2024-02-19T15:23:44.735533+0000","last_undegraded":"2024-02-19T15:23:44.735533+0000","last_fullsized":"2024-02-19T15:23:44.735533+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:18:07.650507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.942749+0000","last_change":"2024-02-19T15:23:41.942749+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.942749+0000","last_undegraded":"2024-02-19T15:23:41.942749+0000","last_fullsized":"2024-02-19T15:23:41.942749+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061366+0000","last_change":"2024-02-19T15:23:42.986099+0000","last_active":"2024-02-19T15:23:44.061366+0000","last_peered":"2024-02-19T15:23:44.061366+0000","last_clean":"2024-02-19T15:23:44.061366+0000","last_became_active":"2024-02-19T15:23:42.985685+0000","last_became_peered":"2024-02-19T15:23:42.985685+0000","last_unstale":"2024-02-19T15:23:44.061366+0000","last_undegraded":"2024-02-19T15:23:44.061366+0000","last_fullsized":"2024-02-19T15:23:44.061366+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:13:36.682515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734637+0000","last_change":"2024-02-19T15:23:42.973184+0000","last_active":"2024-02-19T15:23:44.734637+0000","last_peered":"2024-02-19T15:23:44.734637+0000","last_clean":"2024-02-19T15:23:44.734637+0000","last_became_active":"2024-02-19T15:23:42.972228+0000","last_became_peered":"2024-02-19T15:23:42.972228+0000","last_unstale":"2024-02-19T15:23:44.734637+0000","last_undegraded":"2024-02-19T15:23:44.734637+0000","last_fullsized":"2024-02-19T15:23:44.734637+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:03.977088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946364+0000","last_change":"2024-02-19T15:23:42.979749+0000","last_active":"2024-02-19T15:23:43.946364+0000","last_peered":"2024-02-19T15:23:43.946364+0000","last_clean":"2024-02-19T15:23:43.946364+0000","last_became_active":"2024-02-19T15:23:42.968380+0000","last_became_peered":"2024-02-19T15:23:42.968380+0000","last_unstale":"2024-02-19T15:23:43.946364+0000","last_undegraded":"2024-02-19T15:23:43.946364+0000","last_fullsized":"2024-02-19T15:23:43.946364+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:04:27.961177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.942606+0000","last_change":"2024-02-19T15:23:41.942606+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.942606+0000","last_undegraded":"2024-02-19T15:23:41.942606+0000","last_fullsized":"2024-02-19T15:23:41.942606+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.945345+0000","last_change":"2024-02-19T15:23:41.945345+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.945345+0000","last_undegraded":"2024-02-19T15:23:41.945345+0000","last_fullsized":"2024-02-19T15:23:41.945345+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.945592+0000","last_change":"2024-02-19T15:23:41.945592+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.945592+0000","last_undegraded":"2024-02-19T15:23:41.945592+0000","last_fullsized":"2024-02-19T15:23:41.945592+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061361+0000","last_change":"2024-02-19T15:23:42.980908+0000","last_active":"2024-02-19T15:23:44.061361+0000","last_peered":"2024-02-19T15:23:44.061361+0000","last_clean":"2024-02-19T15:23:44.061361+0000","last_became_active":"2024-02-19T15:23:42.980592+0000","last_became_peered":"2024-02-19T15:23:42.980592+0000","last_unstale":"2024-02-19T15:23:44.061361+0000","last_undegraded":"2024-02-19T15:23:44.061361+0000","last_fullsized":"2024-02-19T15:23:44.061361+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:39:51.560827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735567+0000","last_change":"2024-02-19T15:23:42.961547+0000","last_active":"2024-02-19T15:23:44.735567+0000","last_peered":"2024-02-19T15:23:44.735567+0000","last_clean":"2024-02-19T15:23:44.735567+0000","last_became_active":"2024-02-19T15:23:42.961215+0000","last_became_peered":"2024-02-19T15:23:42.961215+0000","last_unstale":"2024-02-19T15:23:44.735567+0000","last_undegraded":"2024-02-19T15:23:44.735567+0000","last_fullsized":"2024-02-19T15:23:44.735567+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:23:12.865791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735161+0000","last_change":"2024-02-19T15:23:42.971532+0000","last_active":"2024-02-19T15:23:44.735161+0000","last_peered":"2024-02-19T15:23:44.735161+0000","last_clean":"2024-02-19T15:23:44.735161+0000","last_became_active":"2024-02-19T15:23:42.971181+0000","last_became_peered":"2024-02-19T15:23:42.971181+0000","last_unstale":"2024-02-19T15:23:44.735161+0000","last_undegraded":"2024-02-19T15:23:44.735161+0000","last_fullsized":"2024-02-19T15:23:44.735161+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:34:33.315961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.944819+0000","last_change":"2024-02-19T15:23:41.944819+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.944819+0000","last_undegraded":"2024-02-19T15:23:41.944819+0000","last_fullsized":"2024-02-19T15:23:41.944819+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734745+0000","last_change":"2024-02-19T15:23:42.961009+0000","last_active":"2024-02-19T15:23:44.734745+0000","last_peered":"2024-02-19T15:23:44.734745+0000","last_clean":"2024-02-19T15:23:44.734745+0000","last_became_active":"2024-02-19T15:23:42.960723+0000","last_became_peered":"2024-02-19T15:23:42.960723+0000","last_unstale":"2024-02-19T15:23:44.734745+0000","last_undegraded":"2024-02-19T15:23:44.734745+0000","last_fullsized":"2024-02-19T15:23:44.734745+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:57:53.597108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734459+0000","last_change":"2024-02-19T15:23:42.973247+0000","last_active":"2024-02-19T15:23:44.734459+0000","last_peered":"2024-02-19T15:23:44.734459+0000","last_clean":"2024-02-19T15:23:44.734459+0000","last_became_active":"2024-02-19T15:23:42.972957+0000","last_became_peered":"2024-02-19T15:23:42.972957+0000","last_unstale":"2024-02-19T15:23:44.734459+0000","last_undegraded":"2024-02-19T15:23:44.734459+0000","last_fullsized":"2024-02-19T15:23:44.734459+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:36:54.676262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061144+0000","last_change":"2024-02-19T15:23:42.962393+0000","last_active":"2024-02-19T15:23:44.061144+0000","last_peered":"2024-02-19T15:23:44.061144+0000","last_clean":"2024-02-19T15:23:44.061144+0000","last_became_active":"2024-02-19T15:23:42.962112+0000","last_became_peered":"2024-02-19T15:23:42.962112+0000","last_unstale":"2024-02-19T15:23:44.061144+0000","last_undegraded":"2024-02-19T15:23:44.061144+0000","last_fullsized":"2024-02-19T15:23:44.061144+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:28:48.394306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947613+0000","last_change":"2024-02-19T15:23:42.974733+0000","last_active":"2024-02-19T15:23:43.947613+0000","last_peered":"2024-02-19T15:23:43.947613+0000","last_clean":"2024-02-19T15:23:43.947613+0000","last_became_active":"2024-02-19T15:23:42.974439+0000","last_became_peered":"2024-02-19T15:23:42.974439+0000","last_unstale":"2024-02-19T15:23:43.947613+0000","last_undegraded":"2024-02-19T15:23:43.947613+0000","last_fullsized":"2024-02-19T15:23:43.947613+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:39:12.299839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734807+0000","last_change":"2024-02-19T15:23:41.960560+0000","last_active":"2024-02-19T15:23:44.734807+0000","last_peered":"2024-02-19T15:23:44.734807+0000","last_clean":"2024-02-19T15:23:44.734807+0000","last_became_active":"2024-02-19T15:23:41.960237+0000","last_became_peered":"2024-02-19T15:23:41.960237+0000","last_unstale":"2024-02-19T15:23:44.734807+0000","last_undegraded":"2024-02-19T15:23:44.734807+0000","last_fullsized":"2024-02-19T15:23:44.734807+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:14:27.569498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.944998+0000","last_change":"2024-02-19T15:23:41.944998+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.944998+0000","last_undegraded":"2024-02-19T15:23:41.944998+0000","last_fullsized":"2024-02-19T15:23:41.944998+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735605+0000","last_change":"2024-02-19T15:23:41.960962+0000","last_active":"2024-02-19T15:23:44.735605+0000","last_peered":"2024-02-19T15:23:44.735605+0000","last_clean":"2024-02-19T15:23:44.735605+0000","last_became_active":"2024-02-19T15:23:41.959425+0000","last_became_peered":"2024-02-19T15:23:41.959425+0000","last_unstale":"2024-02-19T15:23:44.735605+0000","last_undegraded":"2024-02-19T15:23:44.735605+0000","last_fullsized":"2024-02-19T15:23:44.735605+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:54.136668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735315+0000","last_change":"2024-02-19T15:23:42.972962+0000","last_active":"2024-02-19T15:23:44.735315+0000","last_peered":"2024-02-19T15:23:44.735315+0000","last_clean":"2024-02-19T15:23:44.735315+0000","last_became_active":"2024-02-19T15:23:42.972625+0000","last_became_peered":"2024-02-19T15:23:42.972625+0000","last_unstale":"2024-02-19T15:23:44.735315+0000","last_undegraded":"2024-02-19T15:23:44.735315+0000","last_fullsized":"2024-02-19T15:23:44.735315+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:40:40.055788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947602+0000","last_change":"2024-02-19T15:23:41.948520+0000","last_active":"2024-02-19T15:23:43.947602+0000","last_peered":"2024-02-19T15:23:43.947602+0000","last_clean":"2024-02-19T15:23:43.947602+0000","last_became_active":"2024-02-19T15:23:41.948233+0000","last_became_peered":"2024-02-19T15:23:41.948233+0000","last_unstale":"2024-02-19T15:23:43.947602+0000","last_undegraded":"2024-02-19T15:23:43.947602+0000","last_fullsized":"2024-02-19T15:23:43.947602+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:50:30.218103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947636+0000","last_change":"2024-02-19T15:23:42.961975+0000","last_active":"2024-02-19T15:23:43.947636+0000","last_peered":"2024-02-19T15:23:43.947636+0000","last_clean":"2024-02-19T15:23:43.947636+0000","last_became_active":"2024-02-19T15:23:42.961615+0000","last_became_peered":"2024-02-19T15:23:42.961615+0000","last_unstale":"2024-02-19T15:23:43.947636+0000","last_undegraded":"2024-02-19T15:23:43.947636+0000","last_fullsized":"2024-02-19T15:23:43.947636+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:57:21.424216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194190+0000","last_change":"2024-02-19T15:23:41.961942+0000","last_active":"2024-02-19T15:23:44.194190+0000","last_peered":"2024-02-19T15:23:44.194190+0000","last_clean":"2024-02-19T15:23:44.194190+0000","last_became_active":"2024-02-19T15:23:41.961659+0000","last_became_peered":"2024-02-19T15:23:41.961659+0000","last_unstale":"2024-02-19T15:23:44.194190+0000","last_undegraded":"2024-02-19T15:23:44.194190+0000","last_fullsized":"2024-02-19T15:23:44.194190+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:14:27.569498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734930+0000","last_change":"2024-02-19T15:23:42.961945+0000","last_active":"2024-02-19T15:23:44.734930+0000","last_peered":"2024-02-19T15:23:44.734930+0000","last_clean":"2024-02-19T15:23:44.734930+0000","last_became_active":"2024-02-19T15:23:42.961269+0000","last_became_peered":"2024-02-19T15:23:42.961269+0000","last_unstale":"2024-02-19T15:23:44.734930+0000","last_undegraded":"2024-02-19T15:23:44.734930+0000","last_fullsized":"2024-02-19T15:23:44.734930+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:39:12.299839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734672+0000","last_change":"2024-02-19T15:23:41.966788+0000","last_active":"2024-02-19T15:23:44.734672+0000","last_peered":"2024-02-19T15:23:44.734672+0000","last_clean":"2024-02-19T15:23:44.734672+0000","last_became_active":"2024-02-19T15:23:41.966346+0000","last_became_peered":"2024-02-19T15:23:41.966346+0000","last_unstale":"2024-02-19T15:23:44.734672+0000","last_undegraded":"2024-02-19T15:23:44.734672+0000","last_fullsized":"2024-02-19T15:23:44.734672+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:01:26.302862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946392+0000","last_change":"2024-02-19T15:23:42.968247+0000","last_active":"2024-02-19T15:23:43.946392+0000","last_peered":"2024-02-19T15:23:43.946392+0000","last_clean":"2024-02-19T15:23:43.946392+0000","last_became_active":"2024-02-19T15:23:42.967240+0000","last_became_peered":"2024-02-19T15:23:42.967240+0000","last_unstale":"2024-02-19T15:23:43.946392+0000","last_undegraded":"2024-02-19T15:23:43.946392+0000","last_fullsized":"2024-02-19T15:23:43.946392+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:41:08.031616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735141+0000","last_change":"2024-02-19T15:23:41.955181+0000","last_active":"2024-02-19T15:23:44.735141+0000","last_peered":"2024-02-19T15:23:44.735141+0000","last_clean":"2024-02-19T15:23:44.735141+0000","last_became_active":"2024-02-19T15:23:41.954904+0000","last_became_peered":"2024-02-19T15:23:41.954904+0000","last_unstale":"2024-02-19T15:23:44.735141+0000","last_undegraded":"2024-02-19T15:23:44.735141+0000","last_fullsized":"2024-02-19T15:23:44.735141+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:13:35.679933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945686+0000","last_change":"2024-02-19T15:23:42.980803+0000","last_active":"2024-02-19T15:23:43.945686+0000","last_peered":"2024-02-19T15:23:43.945686+0000","last_clean":"2024-02-19T15:23:43.945686+0000","last_became_active":"2024-02-19T15:23:42.980507+0000","last_became_peered":"2024-02-19T15:23:42.980507+0000","last_unstale":"2024-02-19T15:23:43.945686+0000","last_undegraded":"2024-02-19T15:23:43.945686+0000","last_fullsized":"2024-02-19T15:23:43.945686+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:08:33.495333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734933+0000","last_change":"2024-02-19T15:23:41.951702+0000","last_active":"2024-02-19T15:23:44.734933+0000","last_peered":"2024-02-19T15:23:44.734933+0000","last_clean":"2024-02-19T15:23:44.734933+0000","last_became_active":"2024-02-19T15:23:41.951409+0000","last_became_peered":"2024-02-19T15:23:41.951409+0000","last_unstale":"2024-02-19T15:23:44.734933+0000","last_undegraded":"2024-02-19T15:23:44.734933+0000","last_fullsized":"2024-02-19T15:23:44.734933+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:57:20.421633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.943222+0000","last_change":"2024-02-19T15:23:41.943222+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.943222+0000","last_undegraded":"2024-02-19T15:23:41.943222+0000","last_fullsized":"2024-02-19T15:23:41.943222+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945665+0000","last_change":"2024-02-19T15:23:41.965224+0000","last_active":"2024-02-19T15:23:43.945665+0000","last_peered":"2024-02-19T15:23:43.945665+0000","last_clean":"2024-02-19T15:23:43.945665+0000","last_became_active":"2024-02-19T15:23:41.964654+0000","last_became_peered":"2024-02-19T15:23:41.964654+0000","last_unstale":"2024-02-19T15:23:43.945665+0000","last_undegraded":"2024-02-19T15:23:43.945665+0000","last_fullsized":"2024-02-19T15:23:43.945665+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:49:19.641960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735665+0000","last_change":"2024-02-19T15:23:42.969995+0000","last_active":"2024-02-19T15:23:44.735665+0000","last_peered":"2024-02-19T15:23:44.735665+0000","last_clean":"2024-02-19T15:23:44.735665+0000","last_became_active":"2024-02-19T15:23:42.969651+0000","last_became_peered":"2024-02-19T15:23:42.969651+0000","last_unstale":"2024-02-19T15:23:44.735665+0000","last_undegraded":"2024-02-19T15:23:44.735665+0000","last_fullsized":"2024-02-19T15:23:44.735665+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:45.239277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735692+0000","last_change":"2024-02-19T15:23:41.961017+0000","last_active":"2024-02-19T15:23:44.735692+0000","last_peered":"2024-02-19T15:23:44.735692+0000","last_clean":"2024-02-19T15:23:44.735692+0000","last_became_active":"2024-02-19T15:23:41.960715+0000","last_became_peered":"2024-02-19T15:23:41.960715+0000","last_unstale":"2024-02-19T15:23:44.735692+0000","last_undegraded":"2024-02-19T15:23:44.735692+0000","last_fullsized":"2024-02-19T15:23:44.735692+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:05.009739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194763+0000","last_change":"2024-02-19T15:23:42.966631+0000","last_active":"2024-02-19T15:23:44.194763+0000","last_peered":"2024-02-19T15:23:44.194763+0000","last_clean":"2024-02-19T15:23:44.194763+0000","last_became_active":"2024-02-19T15:23:42.965314+0000","last_became_peered":"2024-02-19T15:23:42.965314+0000","last_unstale":"2024-02-19T15:23:44.194763+0000","last_undegraded":"2024-02-19T15:23:44.194763+0000","last_fullsized":"2024-02-19T15:23:44.194763+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:36:54.676262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946805+0000","last_change":"2024-02-19T15:23:41.958566+0000","last_active":"2024-02-19T15:23:43.946805+0000","last_peered":"2024-02-19T15:23:43.946805+0000","last_clean":"2024-02-19T15:23:43.946805+0000","last_became_active":"2024-02-19T15:23:41.958244+0000","last_became_peered":"2024-02-19T15:23:41.958244+0000","last_unstale":"2024-02-19T15:23:43.946805+0000","last_undegraded":"2024-02-19T15:23:43.946805+0000","last_fullsized":"2024-02-19T15:23:43.946805+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:20:44.307845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734660+0000","last_change":"2024-02-19T15:23:42.972332+0000","last_active":"2024-02-19T15:23:44.734660+0000","last_peered":"2024-02-19T15:23:44.734660+0000","last_clean":"2024-02-19T15:23:44.734660+0000","last_became_active":"2024-02-19T15:23:42.971907+0000","last_became_peered":"2024-02-19T15:23:42.971907+0000","last_unstale":"2024-02-19T15:23:44.734660+0000","last_undegraded":"2024-02-19T15:23:44.734660+0000","last_fullsized":"2024-02-19T15:23:44.734660+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:59:43.637759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194426+0000","last_change":"2024-02-19T15:23:41.964052+0000","last_active":"2024-02-19T15:23:44.194426+0000","last_peered":"2024-02-19T15:23:44.194426+0000","last_clean":"2024-02-19T15:23:44.194426+0000","last_became_active":"2024-02-19T15:23:41.963729+0000","last_became_peered":"2024-02-19T15:23:41.963729+0000","last_unstale":"2024-02-19T15:23:44.194426+0000","last_undegraded":"2024-02-19T15:23:44.194426+0000","last_fullsized":"2024-02-19T15:23:44.194426+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:02:12.172289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194355+0000","last_change":"2024-02-19T15:23:42.969755+0000","last_active":"2024-02-19T15:23:44.194355+0000","last_peered":"2024-02-19T15:23:44.194355+0000","last_clean":"2024-02-19T15:23:44.194355+0000","last_became_active":"2024-02-19T15:23:42.969477+0000","last_became_peered":"2024-02-19T15:23:42.969477+0000","last_unstale":"2024-02-19T15:23:44.194355+0000","last_undegraded":"2024-02-19T15:23:44.194355+0000","last_fullsized":"2024-02-19T15:23:44.194355+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:47:09.498437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":20,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061276+0000","last_change":"2024-02-19T15:23:41.959735+0000","last_active":"2024-02-19T15:23:44.061276+0000","last_peered":"2024-02-19T15:23:44.061276+0000","last_clean":"2024-02-19T15:23:44.061276+0000","last_became_active":"2024-02-19T15:23:41.959171+0000","last_became_peered":"2024-02-19T15:23:41.959171+0000","last_unstale":"2024-02-19T15:23:44.061276+0000","last_undegraded":"2024-02-19T15:23:44.061276+0000","last_fullsized":"2024-02-19T15:23:44.061276+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:07:41.495783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.944967+0000","last_change":"2024-02-19T15:23:41.944967+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.944967+0000","last_undegraded":"2024-02-19T15:23:41.944967+0000","last_fullsized":"2024-02-19T15:23:41.944967+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734694+0000","last_change":"2024-02-19T15:23:41.964586+0000","last_active":"2024-02-19T15:23:44.734694+0000","last_peered":"2024-02-19T15:23:44.734694+0000","last_clean":"2024-02-19T15:23:44.734694+0000","last_became_active":"2024-02-19T15:23:41.964176+0000","last_became_peered":"2024-02-19T15:23:41.964176+0000","last_unstale":"2024-02-19T15:23:44.734694+0000","last_undegraded":"2024-02-19T15:23:44.734694+0000","last_fullsized":"2024-02-19T15:23:44.734694+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:44.236695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946493+0000","last_change":"2024-02-19T15:23:42.965254+0000","last_active":"2024-02-19T15:23:43.946493+0000","last_peered":"2024-02-19T15:23:43.946493+0000","last_clean":"2024-02-19T15:23:43.946493+0000","last_became_active":"2024-02-19T15:23:42.964990+0000","last_became_peered":"2024-02-19T15:23:42.964990+0000","last_unstale":"2024-02-19T15:23:43.946493+0000","last_undegraded":"2024-02-19T15:23:43.946493+0000","last_fullsized":"2024-02-19T15:23:43.946493+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:37:19.468760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194207+0000","last_change":"2024-02-19T15:23:41.965760+0000","last_active":"2024-02-19T15:23:44.194207+0000","last_peered":"2024-02-19T15:23:44.194207+0000","last_clean":"2024-02-19T15:23:44.194207+0000","last_became_active":"2024-02-19T15:23:41.965423+0000","last_became_peered":"2024-02-19T15:23:41.965423+0000","last_unstale":"2024-02-19T15:23:44.194207+0000","last_undegraded":"2024-02-19T15:23:44.194207+0000","last_fullsized":"2024-02-19T15:23:44.194207+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:01:16.303652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734331+0000","last_change":"2024-02-19T15:23:42.970653+0000","last_active":"2024-02-19T15:23:44.734331+0000","last_peered":"2024-02-19T15:23:44.734331+0000","last_clean":"2024-02-19T15:23:44.734331+0000","last_became_active":"2024-02-19T15:23:42.970394+0000","last_became_peered":"2024-02-19T15:23:42.970394+0000","last_unstale":"2024-02-19T15:23:44.734331+0000","last_undegraded":"2024-02-19T15:23:44.734331+0000","last_fullsized":"2024-02-19T15:23:44.734331+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:09:04.319698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734401+0000","last_change":"2024-02-19T15:23:41.964786+0000","last_active":"2024-02-19T15:23:44.734401+0000","last_peered":"2024-02-19T15:23:44.734401+0000","last_clean":"2024-02-19T15:23:44.734401+0000","last_became_active":"2024-02-19T15:23:41.964085+0000","last_became_peered":"2024-02-19T15:23:41.964085+0000","last_unstale":"2024-02-19T15:23:44.734401+0000","last_undegraded":"2024-02-19T15:23:44.734401+0000","last_fullsized":"2024-02-19T15:23:44.734401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:01:16.303652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194624+0000","last_change":"2024-02-19T15:23:42.974233+0000","last_active":"2024-02-19T15:23:44.194624+0000","last_peered":"2024-02-19T15:23:44.194624+0000","last_clean":"2024-02-19T15:23:44.194624+0000","last_became_active":"2024-02-19T15:23:42.973975+0000","last_became_peered":"2024-02-19T15:23:42.973975+0000","last_unstale":"2024-02-19T15:23:44.194624+0000","last_undegraded":"2024-02-19T15:23:44.194624+0000","last_fullsized":"2024-02-19T15:23:44.194624+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:30:47.422419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":22,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945742+0000","last_change":"2024-02-19T15:23:41.957848+0000","last_active":"2024-02-19T15:23:43.945742+0000","last_peered":"2024-02-19T15:23:43.945742+0000","last_clean":"2024-02-19T15:23:43.945742+0000","last_became_active":"2024-02-19T15:23:41.957529+0000","last_became_peered":"2024-02-19T15:23:41.957529+0000","last_unstale":"2024-02-19T15:23:43.945742+0000","last_undegraded":"2024-02-19T15:23:43.945742+0000","last_fullsized":"2024-02-19T15:23:43.945742+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:41:14.948978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061427+0000","last_change":"2024-02-19T15:23:42.971155+0000","last_active":"2024-02-19T15:23:44.061427+0000","last_peered":"2024-02-19T15:23:44.061427+0000","last_clean":"2024-02-19T15:23:44.061427+0000","last_became_active":"2024-02-19T15:23:42.970879+0000","last_became_peered":"2024-02-19T15:23:42.970879+0000","last_unstale":"2024-02-19T15:23:44.061427+0000","last_undegraded":"2024-02-19T15:23:44.061427+0000","last_fullsized":"2024-02-19T15:23:44.061427+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:25:52.362277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061334+0000","last_change":"2024-02-19T15:23:41.966015+0000","last_active":"2024-02-19T15:23:44.061334+0000","last_peered":"2024-02-19T15:23:44.061334+0000","last_clean":"2024-02-19T15:23:44.061334+0000","last_became_active":"2024-02-19T15:23:41.965678+0000","last_became_peered":"2024-02-19T15:23:41.965678+0000","last_unstale":"2024-02-19T15:23:44.061334+0000","last_undegraded":"2024-02-19T15:23:44.061334+0000","last_fullsized":"2024-02-19T15:23:44.061334+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:27:46.811039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946825+0000","last_change":"2024-02-19T15:23:42.983074+0000","last_active":"2024-02-19T15:23:43.946825+0000","last_peered":"2024-02-19T15:23:43.946825+0000","last_clean":"2024-02-19T15:23:43.946825+0000","last_became_active":"2024-02-19T15:23:42.982771+0000","last_became_peered":"2024-02-19T15:23:42.982771+0000","last_unstale":"2024-02-19T15:23:43.946825+0000","last_undegraded":"2024-02-19T15:23:43.946825+0000","last_fullsized":"2024-02-19T15:23:43.946825+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:04:26.404860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735691+0000","last_change":"2024-02-19T15:23:41.963486+0000","last_active":"2024-02-19T15:23:44.735691+0000","last_peered":"2024-02-19T15:23:44.735691+0000","last_clean":"2024-02-19T15:23:44.735691+0000","last_became_active":"2024-02-19T15:23:41.962837+0000","last_became_peered":"2024-02-19T15:23:41.962837+0000","last_unstale":"2024-02-19T15:23:44.735691+0000","last_undegraded":"2024-02-19T15:23:44.735691+0000","last_fullsized":"2024-02-19T15:23:44.735691+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:16:42.881994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.944685+0000","last_change":"2024-02-19T15:23:41.944685+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.944685+0000","last_undegraded":"2024-02-19T15:23:41.944685+0000","last_fullsized":"2024-02-19T15:23:41.944685+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735151+0000","last_change":"2024-02-19T15:23:41.952606+0000","last_active":"2024-02-19T15:23:44.735151+0000","last_peered":"2024-02-19T15:23:44.735151+0000","last_clean":"2024-02-19T15:23:44.735151+0000","last_became_active":"2024-02-19T15:23:41.952291+0000","last_became_peered":"2024-02-19T15:23:41.952291+0000","last_unstale":"2024-02-19T15:23:44.735151+0000","last_undegraded":"2024-02-19T15:23:44.735151+0000","last_fullsized":"2024-02-19T15:23:44.735151+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:01:26.302862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.944426+0000","last_change":"2024-02-19T15:23:41.944426+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.944426+0000","last_undegraded":"2024-02-19T15:23:41.944426+0000","last_fullsized":"2024-02-19T15:23:41.944426+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734712+0000","last_change":"2024-02-19T15:23:41.962283+0000","last_active":"2024-02-19T15:23:44.734712+0000","last_peered":"2024-02-19T15:23:44.734712+0000","last_clean":"2024-02-19T15:23:44.734712+0000","last_became_active":"2024-02-19T15:23:41.961397+0000","last_became_peered":"2024-02-19T15:23:41.961397+0000","last_unstale":"2024-02-19T15:23:44.734712+0000","last_undegraded":"2024-02-19T15:23:44.734712+0000","last_fullsized":"2024-02-19T15:23:44.734712+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:45:37.841170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734509+0000","last_change":"2024-02-19T15:23:42.972039+0000","last_active":"2024-02-19T15:23:44.734509+0000","last_peered":"2024-02-19T15:23:44.734509+0000","last_clean":"2024-02-19T15:23:44.734509+0000","last_became_active":"2024-02-19T15:23:42.971311+0000","last_became_peered":"2024-02-19T15:23:42.971311+0000","last_unstale":"2024-02-19T15:23:44.734509+0000","last_undegraded":"2024-02-19T15:23:44.734509+0000","last_fullsized":"2024-02-19T15:23:44.734509+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:42:40.033469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734827+0000","last_change":"2024-02-19T15:23:41.960894+0000","last_active":"2024-02-19T15:23:44.734827+0000","last_peered":"2024-02-19T15:23:44.734827+0000","last_clean":"2024-02-19T15:23:44.734827+0000","last_became_active":"2024-02-19T15:23:41.959201+0000","last_became_peered":"2024-02-19T15:23:41.959201+0000","last_unstale":"2024-02-19T15:23:44.734827+0000","last_undegraded":"2024-02-19T15:23:44.734827+0000","last_fullsized":"2024-02-19T15:23:44.734827+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:32:59.830263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947496+0000","last_change":"2024-02-19T15:23:42.958733+0000","last_active":"2024-02-19T15:23:43.947496+0000","last_peered":"2024-02-19T15:23:43.947496+0000","last_clean":"2024-02-19T15:23:43.947496+0000","last_became_active":"2024-02-19T15:23:42.958345+0000","last_became_peered":"2024-02-19T15:23:42.958345+0000","last_unstale":"2024-02-19T15:23:43.947496+0000","last_undegraded":"2024-02-19T15:23:43.947496+0000","last_fullsized":"2024-02-19T15:23:43.947496+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:01.020038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734643+0000","last_change":"2024-02-19T15:23:41.961678+0000","last_active":"2024-02-19T15:23:44.734643+0000","last_peered":"2024-02-19T15:23:44.734643+0000","last_clean":"2024-02-19T15:23:44.734643+0000","last_became_active":"2024-02-19T15:23:41.960130+0000","last_became_peered":"2024-02-19T15:23:41.960130+0000","last_unstale":"2024-02-19T15:23:44.734643+0000","last_undegraded":"2024-02-19T15:23:44.734643+0000","last_fullsized":"2024-02-19T15:23:44.734643+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:18:06.647924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061083+0000","last_change":"2024-02-19T15:23:42.965223+0000","last_active":"2024-02-19T15:23:44.061083+0000","last_peered":"2024-02-19T15:23:44.061083+0000","last_clean":"2024-02-19T15:23:44.061083+0000","last_became_active":"2024-02-19T15:23:42.964860+0000","last_became_peered":"2024-02-19T15:23:42.964860+0000","last_unstale":"2024-02-19T15:23:44.061083+0000","last_undegraded":"2024-02-19T15:23:44.061083+0000","last_fullsized":"2024-02-19T15:23:44.061083+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:52.445068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945910+0000","last_change":"2024-02-19T15:23:41.966499+0000","last_active":"2024-02-19T15:23:43.945910+0000","last_peered":"2024-02-19T15:23:43.945910+0000","last_clean":"2024-02-19T15:23:43.945910+0000","last_became_active":"2024-02-19T15:23:41.966206+0000","last_became_peered":"2024-02-19T15:23:41.966206+0000","last_unstale":"2024-02-19T15:23:43.945910+0000","last_undegraded":"2024-02-19T15:23:43.945910+0000","last_fullsized":"2024-02-19T15:23:43.945910+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:50:47.865930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945811+0000","last_change":"2024-02-19T15:23:42.982135+0000","last_active":"2024-02-19T15:23:43.945811+0000","last_peered":"2024-02-19T15:23:43.945811+0000","last_clean":"2024-02-19T15:23:43.945811+0000","last_became_active":"2024-02-19T15:23:42.981890+0000","last_became_peered":"2024-02-19T15:23:42.981890+0000","last_unstale":"2024-02-19T15:23:43.945811+0000","last_undegraded":"2024-02-19T15:23:43.945811+0000","last_fullsized":"2024-02-19T15:23:43.945811+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:28.558554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194347+0000","last_change":"2024-02-19T15:23:41.964986+0000","last_active":"2024-02-19T15:23:44.194347+0000","last_peered":"2024-02-19T15:23:44.194347+0000","last_clean":"2024-02-19T15:23:44.194347+0000","last_became_active":"2024-02-19T15:23:41.964713+0000","last_became_peered":"2024-02-19T15:23:41.964713+0000","last_unstale":"2024-02-19T15:23:44.194347+0000","last_undegraded":"2024-02-19T15:23:44.194347+0000","last_fullsized":"2024-02-19T15:23:44.194347+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:32:59.830263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.943060+0000","last_change":"2024-02-19T15:23:41.943060+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.943060+0000","last_undegraded":"2024-02-19T15:23:41.943060+0000","last_fullsized":"2024-02-19T15:23:41.943060+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"45'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734750+0000","last_change":"2024-02-19T15:23:41.968055+0000","last_active":"2024-02-19T15:23:44.734750+0000","last_peered":"2024-02-19T15:23:44.734750+0000","last_clean":"2024-02-19T15:23:44.734750+0000","last_became_active":"2024-02-19T15:23:41.967728+0000","last_became_peered":"2024-02-19T15:23:41.967728+0000","last_unstale":"2024-02-19T15:23:44.734750+0000","last_undegraded":"2024-02-19T15:23:44.734750+0000","last_fullsized":"2024-02-19T15:23:44.734750+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:02:12.172289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946492+0000","last_change":"2024-02-19T15:23:42.966169+0000","last_active":"2024-02-19T15:23:43.946492+0000","last_peered":"2024-02-19T15:23:43.946492+0000","last_clean":"2024-02-19T15:23:43.946492+0000","last_became_active":"2024-02-19T15:23:42.965898+0000","last_became_peered":"2024-02-19T15:23:42.965898+0000","last_unstale":"2024-02-19T15:23:43.946492+0000","last_undegraded":"2024-02-19T15:23:43.946492+0000","last_fullsized":"2024-02-19T15:23:43.946492+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:41:45.823238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194546+0000","last_change":"2024-02-19T15:23:41.968087+0000","last_active":"2024-02-19T15:23:44.194546+0000","last_peered":"2024-02-19T15:23:44.194546+0000","last_clean":"2024-02-19T15:23:44.194546+0000","last_became_active":"2024-02-19T15:23:41.967798+0000","last_became_peered":"2024-02-19T15:23:41.967798+0000","last_unstale":"2024-02-19T15:23:44.194546+0000","last_undegraded":"2024-02-19T15:23:44.194546+0000","last_fullsized":"2024-02-19T15:23:44.194546+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:08:00.893923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194536+0000","last_change":"2024-02-19T15:23:42.971627+0000","last_active":"2024-02-19T15:23:44.194536+0000","last_peered":"2024-02-19T15:23:44.194536+0000","last_clean":"2024-02-19T15:23:44.194536+0000","last_became_active":"2024-02-19T15:23:42.971344+0000","last_became_peered":"2024-02-19T15:23:42.971344+0000","last_unstale":"2024-02-19T15:23:44.194536+0000","last_undegraded":"2024-02-19T15:23:44.194536+0000","last_fullsized":"2024-02-19T15:23:44.194536+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:50:40.431301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947382+0000","last_change":"2024-02-19T15:23:41.948963+0000","last_active":"2024-02-19T15:23:43.947382+0000","last_peered":"2024-02-19T15:23:43.947382+0000","last_clean":"2024-02-19T15:23:43.947382+0000","last_became_active":"2024-02-19T15:23:41.947243+0000","last_became_peered":"2024-02-19T15:23:41.947243+0000","last_unstale":"2024-02-19T15:23:43.947382+0000","last_undegraded":"2024-02-19T15:23:43.947382+0000","last_fullsized":"2024-02-19T15:23:43.947382+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:51:31.286375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734698+0000","last_change":"2024-02-19T15:23:42.974990+0000","last_active":"2024-02-19T15:23:44.734698+0000","last_peered":"2024-02-19T15:23:44.734698+0000","last_clean":"2024-02-19T15:23:44.734698+0000","last_became_active":"2024-02-19T15:23:42.974011+0000","last_became_peered":"2024-02-19T15:23:42.974011+0000","last_unstale":"2024-02-19T15:23:44.734698+0000","last_undegraded":"2024-02-19T15:23:44.734698+0000","last_fullsized":"2024-02-19T15:23:44.734698+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:48.375387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194296+0000","last_change":"2024-02-19T15:23:41.964025+0000","last_active":"2024-02-19T15:23:44.194296+0000","last_peered":"2024-02-19T15:23:44.194296+0000","last_clean":"2024-02-19T15:23:44.194296+0000","last_became_active":"2024-02-19T15:23:41.963717+0000","last_became_peered":"2024-02-19T15:23:41.963717+0000","last_unstale":"2024-02-19T15:23:44.194296+0000","last_undegraded":"2024-02-19T15:23:44.194296+0000","last_fullsized":"2024-02-19T15:23:44.194296+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:45:37.841170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734618+0000","last_change":"2024-02-19T15:23:42.974421+0000","last_active":"2024-02-19T15:23:44.734618+0000","last_peered":"2024-02-19T15:23:44.734618+0000","last_clean":"2024-02-19T15:23:44.734618+0000","last_became_active":"2024-02-19T15:23:42.974009+0000","last_became_peered":"2024-02-19T15:23:42.974009+0000","last_unstale":"2024-02-19T15:23:44.734618+0000","last_undegraded":"2024-02-19T15:23:44.734618+0000","last_fullsized":"2024-02-19T15:23:44.734618+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:36.710691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735244+0000","last_change":"2024-02-19T15:23:41.969004+0000","last_active":"2024-02-19T15:23:44.735244+0000","last_peered":"2024-02-19T15:23:44.735244+0000","last_clean":"2024-02-19T15:23:44.735244+0000","last_became_active":"2024-02-19T15:23:41.968796+0000","last_became_peered":"2024-02-19T15:23:41.968796+0000","last_unstale":"2024-02-19T15:23:44.735244+0000","last_undegraded":"2024-02-19T15:23:44.735244+0000","last_fullsized":"2024-02-19T15:23:44.735244+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:25:51.359695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735100+0000","last_change":"2024-02-19T15:23:42.956889+0000","last_active":"2024-02-19T15:23:44.735100+0000","last_peered":"2024-02-19T15:23:44.735100+0000","last_clean":"2024-02-19T15:23:44.735100+0000","last_became_active":"2024-02-19T15:23:42.953555+0000","last_became_peered":"2024-02-19T15:23:42.953555+0000","last_unstale":"2024-02-19T15:23:44.735100+0000","last_undegraded":"2024-02-19T15:23:44.735100+0000","last_fullsized":"2024-02-19T15:23:44.735100+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:50:45.305646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947245+0000","last_change":"2024-02-19T15:23:41.956700+0000","last_active":"2024-02-19T15:23:43.947245+0000","last_peered":"2024-02-19T15:23:43.947245+0000","last_clean":"2024-02-19T15:23:43.947245+0000","last_became_active":"2024-02-19T15:23:41.956388+0000","last_became_peered":"2024-02-19T15:23:41.956388+0000","last_unstale":"2024-02-19T15:23:43.947245+0000","last_undegraded":"2024-02-19T15:23:43.947245+0000","last_fullsized":"2024-02-19T15:23:43.947245+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:15:40.531978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734808+0000","last_change":"2024-02-19T15:23:42.976054+0000","last_active":"2024-02-19T15:23:44.734808+0000","last_peered":"2024-02-19T15:23:44.734808+0000","last_clean":"2024-02-19T15:23:44.734808+0000","last_became_active":"2024-02-19T15:23:42.975795+0000","last_became_peered":"2024-02-19T15:23:42.975795+0000","last_unstale":"2024-02-19T15:23:44.734808+0000","last_undegraded":"2024-02-19T15:23:44.734808+0000","last_fullsized":"2024-02-19T15:23:44.734808+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:01.020038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945761+0000","last_change":"2024-02-19T15:23:41.962391+0000","last_active":"2024-02-19T15:23:43.945761+0000","last_peered":"2024-02-19T15:23:43.945761+0000","last_clean":"2024-02-19T15:23:43.945761+0000","last_became_active":"2024-02-19T15:23:41.961437+0000","last_became_peered":"2024-02-19T15:23:41.961437+0000","last_unstale":"2024-02-19T15:23:43.945761+0000","last_undegraded":"2024-02-19T15:23:43.945761+0000","last_fullsized":"2024-02-19T15:23:43.945761+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:16:05.431843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946756+0000","last_change":"2024-02-19T15:23:42.964218+0000","last_active":"2024-02-19T15:23:43.946756+0000","last_peered":"2024-02-19T15:23:43.946756+0000","last_clean":"2024-02-19T15:23:43.946756+0000","last_became_active":"2024-02-19T15:23:42.963959+0000","last_became_peered":"2024-02-19T15:23:42.963959+0000","last_unstale":"2024-02-19T15:23:43.946756+0000","last_undegraded":"2024-02-19T15:23:43.946756+0000","last_fullsized":"2024-02-19T15:23:43.946756+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:43:11.719148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735163+0000","last_change":"2024-02-19T15:23:41.960473+0000","last_active":"2024-02-19T15:23:44.735163+0000","last_peered":"2024-02-19T15:23:44.735163+0000","last_clean":"2024-02-19T15:23:44.735163+0000","last_became_active":"2024-02-19T15:23:41.960208+0000","last_became_peered":"2024-02-19T15:23:41.960208+0000","last_unstale":"2024-02-19T15:23:44.735163+0000","last_undegraded":"2024-02-19T15:23:44.735163+0000","last_fullsized":"2024-02-19T15:23:44.735163+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:23:11.863209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":212,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194375+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:23:44.194375+0000","last_peered":"2024-02-19T15:23:44.194375+0000","last_clean":"2024-02-19T15:23:44.194375+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:23:44.194375+0000","last_undegraded":"2024-02-19T15:23:44.194375+0000","last_fullsized":"2024-02-19T15:23:44.194375+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509465,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":487424,"data_stored":193585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":4.6029999999999998,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.79400000000000004}]},{"osd":1,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.625},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.71699999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.73999999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.75600000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.64900000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.81000000000000005}]}]},{"osd":7,"up_from":14,"seq":60129542170,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28124,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343716,"statfs":{"total":96636764160,"available":96607965184,"internally_reserved":0,"allocated":1122304,"data_stored":803385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.627},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.87}]},{"osd":1,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":2.968,"5min":2.968,"15min":2.968},"last":0.754}]},{"osd":2,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":0.71099999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":3.1720000000000002,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":0.77300000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.73699999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.83199999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.93799999999999994}]}]},{"osd":5,"up_from":14,"seq":60129542171,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27536,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344304,"statfs":{"total":96636764160,"available":96608567296,"internally_reserved":0,"allocated":528384,"data_stored":212997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.56999999999999995}]},{"osd":1,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.85499999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.66300000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.63900000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":0.67600000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.89400000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.59899999999999998}]}]},{"osd":4,"up_from":14,"seq":60129542169,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27544,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344296,"statfs":{"total":96636764160,"available":96608559104,"internally_reserved":0,"allocated":536576,"data_stored":214539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.72899999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.623}]},{"osd":2,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.755}]},{"osd":3,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.49199999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.80200000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.78800000000000003}]},{"osd":7,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.82199999999999995}]}]},{"osd":3,"up_from":14,"seq":60129542169,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":536576,"data_stored":213565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.2450000000000001,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.33500000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":4.0220000000000002,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.52900000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.72899999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":4.2039999999999997,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.78400000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":4.1589999999999998,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.627}]},{"osd":6,"last update":"Mon Feb 19 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":4.0940000000000003,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.69499999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":3.9660000000000002,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.67500000000000004}]}]},{"osd":2,"up_from":14,"seq":60129542170,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":540672,"data_stored":214561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.7160000000000002,"5min":4.7160000000000002,"15min":4.7160000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":8.2129999999999992,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.26000000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":8.1620000000000008,"5min":8.1620000000000008,"15min":8.1620000000000008},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.46100000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":4.7679999999999998,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.371},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":4.9909999999999997,"5min":4.9909999999999997,"15min":4.9909999999999997},"last":0.443}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":4.3810000000000002,"5min":4.3810000000000002,"15min":4.3810000000000002},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":4.3440000000000003,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.83199999999999996}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":4.4039999999999999,"5min":4.4039999999999999,"15min":4.4039999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":4.3259999999999996,"5min":4.3259999999999996,"15min":4.3259999999999996},"last":0.75}]},{"osd":6,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":4.5220000000000002,"5min":4.5220000000000002,"15min":4.5220000000000002},"last":0.71699999999999997},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":4.4320000000000004,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.73899999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":4.4829999999999997,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":4.4989999999999997,"5min":4.4989999999999997,"15min":4.4989999999999997},"last":0.84599999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542171,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":528384,"data_stored":212997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.25}]},{"osd":2,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.41599999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.438},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.38300000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.65700000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.746},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.77500000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.72699999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.70499999999999996}]}]},{"osd":0,"up_from":14,"seq":60129542169,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36436,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335404,"statfs":{"total":96636764160,"available":96599453696,"internally_reserved":0,"allocated":1122304,"data_stored":803329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.0199999999999996,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.31}]},{"osd":2,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":5.6669999999999998,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.4649999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.45700000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.9539999999999997,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.345,"5min":1.345,"15min":1.345},"last":0.44800000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":4.7880000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.61799999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.53200000000000003}]},{"osd":6,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":4.8460000000000001,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.65300000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":4.9809999999999999,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":4.915,"5min":4.915,"15min":4.915},"last":0.56200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:23:47.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.30 is not active+clean 2024-02-19T15:23:47.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:23:41.945858+0000', 'last_change': '2024-02-19T15:23:41.945858+0000', 'last_active': '2024-02-19T15:23:41.932392+0000', 'last_peered': '2024-02-19T15:23:41.932392+0000', 'last_clean': '2024-02-19T15:23:41.932392+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:23:41.945858+0000', 'last_undegraded': '2024-02-19T15:23:41.945858+0000', 'last_fullsized': '2024-02-19T15:23:41.945858+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_clean_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:23:47.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2d is not active+clean 2024-02-19T15:23:47.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:23:41.942749+0000', 'last_change': '2024-02-19T15:23:41.942749+0000', 'last_active': '2024-02-19T15:23:41.932392+0000', 'last_peered': '2024-02-19T15:23:41.932392+0000', 'last_clean': '2024-02-19T15:23:41.932392+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:23:41.942749+0000', 'last_undegraded': '2024-02-19T15:23:41.942749+0000', 'last_fullsized': '2024-02-19T15:23:41.942749+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_clean_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:23:47.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.29 is not active+clean 2024-02-19T15:23:47.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:23:41.942606+0000', 'last_change': '2024-02-19T15:23:41.942606+0000', 'last_active': '2024-02-19T15:23:41.932392+0000', 'last_peered': '2024-02-19T15:23:41.932392+0000', 'last_clean': '2024-02-19T15:23:41.932392+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:23:41.942606+0000', 'last_undegraded': '2024-02-19T15:23:41.942606+0000', 'last_fullsized': '2024-02-19T15:23:41.942606+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_clean_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:23:47.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.28 is not active+clean 2024-02-19T15:23:47.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:23:41.945345+0000', 'last_change': '2024-02-19T15:23:41.945345+0000', 'last_active': '2024-02-19T15:23:41.932392+0000', 'last_peered': '2024-02-19T15:23:41.932392+0000', 'last_clean': '2024-02-19T15:23:41.932392+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:23:41.945345+0000', 'last_undegraded': '2024-02-19T15:23:41.945345+0000', 'last_fullsized': '2024-02-19T15:23:41.945345+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_clean_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:23:47.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.27 is not active+clean 2024-02-19T15:23:47.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:23:41.945592+0000', 'last_change': '2024-02-19T15:23:41.945592+0000', 'last_active': '2024-02-19T15:23:41.932392+0000', 'last_peered': '2024-02-19T15:23:41.932392+0000', 'last_clean': '2024-02-19T15:23:41.932392+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:23:41.945592+0000', 'last_undegraded': '2024-02-19T15:23:41.945592+0000', 'last_fullsized': '2024-02-19T15:23:41.945592+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_clean_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:23:47.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.23 is not active+clean 2024-02-19T15:23:47.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:23:41.944819+0000', 'last_change': '2024-02-19T15:23:41.944819+0000', 'last_active': '2024-02-19T15:23:41.932392+0000', 'last_peered': '2024-02-19T15:23:41.932392+0000', 'last_clean': '2024-02-19T15:23:41.932392+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:23:41.944819+0000', 'last_undegraded': '2024-02-19T15:23:41.944819+0000', 'last_fullsized': '2024-02-19T15:23:41.944819+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_clean_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:23:47.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1f is not active+clean 2024-02-19T15:23:47.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:23:41.944998+0000', 'last_change': '2024-02-19T15:23:41.944998+0000', 'last_active': '2024-02-19T15:23:41.932392+0000', 'last_peered': '2024-02-19T15:23:41.932392+0000', 'last_clean': '2024-02-19T15:23:41.932392+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:23:41.944998+0000', 'last_undegraded': '2024-02-19T15:23:41.944998+0000', 'last_fullsized': '2024-02-19T15:23:41.944998+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_clean_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:23:47.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.19 is not active+clean 2024-02-19T15:23:47.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:23:41.943222+0000', 'last_change': '2024-02-19T15:23:41.943222+0000', 'last_active': '2024-02-19T15:23:41.932392+0000', 'last_peered': '2024-02-19T15:23:41.932392+0000', 'last_clean': '2024-02-19T15:23:41.932392+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:23:41.943222+0000', 'last_undegraded': '2024-02-19T15:23:41.943222+0000', 'last_fullsized': '2024-02-19T15:23:41.943222+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_clean_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:23:47.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.12 is not active+clean 2024-02-19T15:23:47.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:23:41.944967+0000', 'last_change': '2024-02-19T15:23:41.944967+0000', 'last_active': '2024-02-19T15:23:41.932392+0000', 'last_peered': '2024-02-19T15:23:41.932392+0000', 'last_clean': '2024-02-19T15:23:41.932392+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:23:41.944967+0000', 'last_undegraded': '2024-02-19T15:23:41.944967+0000', 'last_fullsized': '2024-02-19T15:23:41.944967+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_clean_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:23:47.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.c is not active+clean 2024-02-19T15:23:47.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:23:41.944685+0000', 'last_change': '2024-02-19T15:23:41.944685+0000', 'last_active': '2024-02-19T15:23:41.932392+0000', 'last_peered': '2024-02-19T15:23:41.932392+0000', 'last_clean': '2024-02-19T15:23:41.932392+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:23:41.944685+0000', 'last_undegraded': '2024-02-19T15:23:41.944685+0000', 'last_fullsized': '2024-02-19T15:23:41.944685+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_clean_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:23:47.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.d is not active+clean 2024-02-19T15:23:47.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:23:41.944426+0000', 'last_change': '2024-02-19T15:23:41.944426+0000', 'last_active': '2024-02-19T15:23:41.932392+0000', 'last_peered': '2024-02-19T15:23:41.932392+0000', 'last_clean': '2024-02-19T15:23:41.932392+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:23:41.944426+0000', 'last_undegraded': '2024-02-19T15:23:41.944426+0000', 'last_fullsized': '2024-02-19T15:23:41.944426+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_clean_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:23:47.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.7 is not active+clean 2024-02-19T15:23:47.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:23:41.943060+0000', 'last_change': '2024-02-19T15:23:41.943060+0000', 'last_active': '2024-02-19T15:23:41.932392+0000', 'last_peered': '2024-02-19T15:23:41.932392+0000', 'last_clean': '2024-02-19T15:23:41.932392+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:23:41.943060+0000', 'last_undegraded': '2024-02-19T15:23:41.943060+0000', 'last_fullsized': '2024-02-19T15:23:41.943060+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'last_clean_scrub_stamp': '2024-02-19T15:23:41.932392+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:23:47.876 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-02-19T15:23:48.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.047+0000 7f23e4400640 1 Processor -- start 2024-02-19T15:23:48.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.047+0000 7f23e4400640 1 -- start start 2024-02-19T15:23:48.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.047+0000 7f23e4400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f23dc0cedb0 0x7f23dc0cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:48.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.047+0000 7f23e4400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f23dc0d2710 0x7f23dc0d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:48.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.047+0000 7f23e4400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23dc0cc700 con 0x7f23dc15bcd0 2024-02-19T15:23:48.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.047+0000 7f23e4400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23dc0cf5e0 con 0x7f23dc0d2710 2024-02-19T15:23:48.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.047+0000 7f23e4400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc0cdb90 con 0x7f23dc0cedb0 2024-02-19T15:23:48.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.047+0000 7f23e1a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f23dc15bcd0 0x7f23dc0cd480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49170/0 (socket says 172.21.15.47:49170) 2024-02-19T15:23:48.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.047+0000 7f23e1a00640 1 -- 172.21.15.47:0/3993304023 learned_addr learned my addr 172.21.15.47:0/3993304023 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:48.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.047+0000 7f23e2400640 1 --2- 172.21.15.47:0/3993304023 >> v2:172.21.15.47:3300/0 conn(0x7f23dc0d2710 0x7f23dc0d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:48.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.047+0000 7f23e1000640 1 --2- 172.21.15.47:0/3993304023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f23dc0cedb0 0x7f23dc0cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:48.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.051+0000 7f23dbe00640 1 -- 172.21.15.47:0/3993304023 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111350928 0 0) 0x7f23dc0cc700 con 0x7f23dc15bcd0 2024-02-19T15:23:48.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.051+0000 7f23dbe00640 1 -- 172.21.15.47:0/3993304023 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c4003530 con 0x7f23dc15bcd0 2024-02-19T15:23:48.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.051+0000 7f23e2400640 1 -- 172.21.15.47:0/3993304023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f23dc0cedb0 msgr2=0x7f23dc0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:48.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.051+0000 7f23e2400640 1 --2- 172.21.15.47:0/3993304023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f23dc0cedb0 0x7f23dc0cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:48.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.051+0000 7f23e2400640 1 -- 172.21.15.47:0/3993304023 >> v1:172.21.15.47:6789/0 conn(0x7f23dc15bcd0 legacy=0x7f23dc0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:48.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.051+0000 7f23e2400640 1 -- 172.21.15.47:0/3993304023 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23dc0cf300 con 0x7f23dc0d2710 2024-02-19T15:23:48.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.051+0000 7f23e2400640 1 --2- 172.21.15.47:0/3993304023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23dc0d2710 0x7f23dc0d0190 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f23d4009d90 tx=0x7f23d401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2333bb05a5a94c8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:48.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.051+0000 7f23dbe00640 1 -- 172.21.15.47:0/3993304023 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d40a5a00 con 0x7f23dc0d2710 2024-02-19T15:23:48.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.051+0000 7f23dbe00640 1 -- 172.21.15.47:0/3993304023 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d40a5b60 con 0x7f23dc0d2710 2024-02-19T15:23:48.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.051+0000 7f23dbe00640 1 -- 172.21.15.47:0/3993304023 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d40ad440 con 0x7f23dc0d2710 2024-02-19T15:23:48.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e4400640 1 -- 172.21.15.47:0/3993304023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23dc0d2710 msgr2=0x7f23dc0d0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:48.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e4400640 1 --2- 172.21.15.47:0/3993304023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23dc0d2710 0x7f23dc0d0190 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f23d4009d90 tx=0x7f23d401c940 comp rx=0 tx=0).stop 2024-02-19T15:23:48.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e4400640 1 -- 172.21.15.47:0/3993304023 shutdown_connections 2024-02-19T15:23:48.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e4400640 1 --2- 172.21.15.47:0/3993304023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23dc0d2710 0x7f23dc0d0190 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:48.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e4400640 1 --2- 172.21.15.47:0/3993304023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f23dc0cedb0 0x7f23dc0cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:48.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e4400640 1 -- 172.21.15.47:0/3993304023 >> 172.21.15.47:0/3993304023 conn(0x7f23dc0d1bf0 msgr2=0x7f23dc0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:48.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e4400640 1 -- 172.21.15.47:0/3993304023 shutdown_connections 2024-02-19T15:23:48.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e4400640 1 -- 172.21.15.47:0/3993304023 wait complete. 2024-02-19T15:23:48.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e4400640 1 Processor -- start 2024-02-19T15:23:48.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e4400640 1 -- start start 2024-02-19T15:23:48.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e4400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23dc0cedb0 0x7f23dc0c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:48.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e4400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23dc0d2710 0x7f23dc158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:48.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e4400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f23dc15bcd0 0x7f23dc1586a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:48.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e4400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc0cb6f0 con 0x7f23dc0cedb0 2024-02-19T15:23:48.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e4400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc148b80 con 0x7f23dc0d2710 2024-02-19T15:23:48.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e4400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc0caa70 con 0x7f23dc15bcd0 2024-02-19T15:23:48.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e1a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23dc0cedb0 0x7f23dc0c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:48.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e1a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23dc0cedb0 0x7f23dc0c79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33400/0 (socket says 172.21.15.47:33400) 2024-02-19T15:23:48.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e1a00640 1 -- 172.21.15.47:0/1931051858 learned_addr learned my addr 172.21.15.47:0/1931051858 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:48.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e1000640 1 --2- 172.21.15.47:0/1931051858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23dc0d2710 0x7f23dc158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:48.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e2400640 1 --2- 172.21.15.47:0/1931051858 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f23dc15bcd0 0x7f23dc1586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:48.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e1a00640 1 -- 172.21.15.47:0/1931051858 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f23dc15bcd0 msgr2=0x7f23dc1586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:48.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e1a00640 1 --2- 172.21.15.47:0/1931051858 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f23dc15bcd0 0x7f23dc1586a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:48.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e1a00640 1 -- 172.21.15.47:0/1931051858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23dc0d2710 msgr2=0x7f23dc158160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:48.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e1a00640 1 --2- 172.21.15.47:0/1931051858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23dc0d2710 0x7f23dc158160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:48.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e1a00640 1 -- 172.21.15.47:0/1931051858 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d4009a70 con 0x7f23dc0cedb0 2024-02-19T15:23:48.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.055+0000 7f23e1000640 1 --2- 172.21.15.47:0/1931051858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23dc0d2710 0x7f23dc158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:48.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.059+0000 7f23e1a00640 1 --2- 172.21.15.47:0/1931051858 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23dc0cedb0 0x7f23dc0c79c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f23d000c3c0 tx=0x7f23d000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:48.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.059+0000 7f23d9600640 1 -- 172.21.15.47:0/1931051858 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d0005cf0 con 0x7f23dc0cedb0 2024-02-19T15:23:48.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.059+0000 7f23d9600640 1 -- 172.21.15.47:0/1931051858 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d0005e50 con 0x7f23dc0cedb0 2024-02-19T15:23:48.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.059+0000 7f23d9600640 1 -- 172.21.15.47:0/1931051858 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d00072f0 con 0x7f23dc0cedb0 2024-02-19T15:23:48.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.059+0000 7f23e4400640 1 -- 172.21.15.47:0/1931051858 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23dc15d240 con 0x7f23dc0cedb0 2024-02-19T15:23:48.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.059+0000 7f23e4400640 1 -- 172.21.15.47:0/1931051858 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23dc0cb030 con 0x7f23dc0cedb0 2024-02-19T15:23:48.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.059+0000 7f23e4400640 1 -- 172.21.15.47:0/1931051858 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23dc075360 con 0x7f23dc0cedb0 2024-02-19T15:23:48.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.059+0000 7f23d9600640 1 -- 172.21.15.47:0/1931051858 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f23d0007450 con 0x7f23dc0cedb0 2024-02-19T15:23:48.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.059+0000 7f23d9600640 1 --2- 172.21.15.47:0/1931051858 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f23bc066e40 0x7f23bc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:48.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.059+0000 7f23d9600640 1 -- 172.21.15.47:0/1931051858 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23d00840b0 con 0x7f23dc0cedb0 2024-02-19T15:23:48.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.059+0000 7f23e1000640 1 --2- 172.21.15.47:0/1931051858 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f23bc066e40 0x7f23bc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:48.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.063+0000 7f23e1000640 1 --2- 172.21.15.47:0/1931051858 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f23bc066e40 0x7f23bc069300 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f23dc09fbb0 tx=0x7f23c8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:48.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.063+0000 7f23d9600640 1 -- 172.21.15.47:0/1931051858 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f23d0050e00 con 0x7f23dc0cedb0 2024-02-19T15:23:48.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.287+0000 7f23e4400640 1 -- 172.21.15.47:0/1931051858 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f23dc0b2c50 con 0x7f23dc0cedb0 2024-02-19T15:23:48.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.287+0000 7f23d9600640 1 -- 172.21.15.47:0/1931051858 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1518 (secure 0 0 0) 0x7f23d00544c0 con 0x7f23dc0cedb0 2024-02-19T15:23:48.294 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:48.294 INFO:teuthology.orchestra.run.smithi047.stdout:{"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":128,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":8,"num_up_osds":8,"osd_up_since":1708356111,"num_in_osds":8,"osd_in_since":1708356102,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":85},{"state_name":"creating+peering","count":12}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":252334080,"bytes_avail":772841779200,"bytes_total":773094113280,"inactive_pgs_ratio":0.12371134012937546,"write_bytes_sec":2388,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":25,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"a","status":"up:active","gid":5514}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-02-19T15:23:31.167900+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-02-19T15:23:48.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.291+0000 7f23e4400640 1 -- 172.21.15.47:0/1931051858 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f23bc066e40 msgr2=0x7f23bc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.291+0000 7f23e4400640 1 --2- 172.21.15.47:0/1931051858 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f23bc066e40 0x7f23bc069300 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f23dc09fbb0 tx=0x7f23c8008040 comp rx=0 tx=0).stop 2024-02-19T15:23:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.291+0000 7f23e4400640 1 -- 172.21.15.47:0/1931051858 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23dc0cedb0 msgr2=0x7f23dc0c79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.291+0000 7f23e4400640 1 --2- 172.21.15.47:0/1931051858 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23dc0cedb0 0x7f23dc0c79c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f23d000c3c0 tx=0x7f23d000c880 comp rx=0 tx=0).stop 2024-02-19T15:23:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.291+0000 7f23e4400640 1 -- 172.21.15.47:0/1931051858 shutdown_connections 2024-02-19T15:23:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.291+0000 7f23e4400640 1 --2- 172.21.15.47:0/1931051858 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f23bc066e40 0x7f23bc069300 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.291+0000 7f23e4400640 1 --2- 172.21.15.47:0/1931051858 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f23dc15bcd0 0x7f23dc1586a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.291+0000 7f23e4400640 1 --2- 172.21.15.47:0/1931051858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23dc0d2710 0x7f23dc158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.291+0000 7f23e4400640 1 --2- 172.21.15.47:0/1931051858 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23dc0cedb0 0x7f23dc0c79c0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.291+0000 7f23e4400640 1 -- 172.21.15.47:0/1931051858 >> 172.21.15.47:0/1931051858 conn(0x7f23dc0d1bf0 msgr2=0x7f23dc0b4870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.291+0000 7f23e4400640 1 -- 172.21.15.47:0/1931051858 shutdown_connections 2024-02-19T15:23:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.291+0000 7f23e4400640 1 -- 172.21.15.47:0/1931051858 wait complete. 2024-02-19T15:23:48.315 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-02-19T15:23:48.315 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:23:48.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.451+0000 7ffa19800640 1 Processor -- start 2024-02-19T15:23:48.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.451+0000 7ffa19800640 1 -- start start 2024-02-19T15:23:48.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.451+0000 7ffa19800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffa140d0360 0x7ffa1415d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:48.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.451+0000 7ffa19800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa140cc700 con 0x7ffa140cedb0 2024-02-19T15:23:48.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.451+0000 7ffa19800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa140cf300 con 0x7ffa1415bcd0 2024-02-19T15:23:48.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.451+0000 7ffa19800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa140cf5e0 con 0x7ffa140d0360 2024-02-19T15:23:48.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.455+0000 7ffa0a000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ffa140cedb0 0x7ffa140cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49176/0 (socket says 172.21.15.47:49176) 2024-02-19T15:23:48.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.455+0000 7ffa0a000640 1 -- 172.21.15.47:0/3736887416 learned_addr learned my addr 172.21.15.47:0/3736887416 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:48.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.455+0000 7ffa13400640 1 --2- 172.21.15.47:0/3736887416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffa140d0360 0x7ffa1415d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:48.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.455+0000 7ffa12000640 1 -- 172.21.15.47:0/3736887416 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281947309 0 0) 0x7ffa140cc700 con 0x7ffa140cedb0 2024-02-19T15:23:48.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.455+0000 7ffa12000640 1 -- 172.21.15.47:0/3736887416 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9f8003530 con 0x7ffa140cedb0 2024-02-19T15:23:48.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.455+0000 7ffa12000640 1 -- 172.21.15.47:0/3736887416 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779190512 0 0) 0x7ffa140cf300 con 0x7ffa1415bcd0 2024-02-19T15:23:48.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.455+0000 7ffa12000640 1 -- 172.21.15.47:0/3736887416 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa140cc700 con 0x7ffa1415bcd0 2024-02-19T15:23:48.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.455+0000 7ffa13400640 1 -- 172.21.15.47:0/3736887416 >> v1:172.21.15.116:6789/0 conn(0x7ffa1415bcd0 legacy=0x7ffa140cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:48.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.455+0000 7ffa13400640 1 -- 172.21.15.47:0/3736887416 >> v1:172.21.15.47:6789/0 conn(0x7ffa140cedb0 legacy=0x7ffa140cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:48.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.455+0000 7ffa13400640 1 -- 172.21.15.47:0/3736887416 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa140cc9a0 con 0x7ffa140d0360 2024-02-19T15:23:48.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.455+0000 7ffa13400640 1 --2- 172.21.15.47:0/3736887416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffa140d0360 0x7ffa1415d4f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c009dc0 tx=0x7ffa0c01c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a7ff3c642bb7820 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:48.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa12000640 1 -- 172.21.15.47:0/3736887416 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c0a5a20 con 0x7ffa140d0360 2024-02-19T15:23:48.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa12000640 1 -- 172.21.15.47:0/3736887416 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0c0a5b80 con 0x7ffa140d0360 2024-02-19T15:23:48.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa12000640 1 -- 172.21.15.47:0/3736887416 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c0ad440 con 0x7ffa140d0360 2024-02-19T15:23:48.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa19800640 1 -- 172.21.15.47:0/3736887416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffa140d0360 msgr2=0x7ffa1415d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:48.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa19800640 1 --2- 172.21.15.47:0/3736887416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffa140d0360 0x7ffa1415d4f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c009dc0 tx=0x7ffa0c01c960 comp rx=0 tx=0).stop 2024-02-19T15:23:48.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa19800640 1 -- 172.21.15.47:0/3736887416 shutdown_connections 2024-02-19T15:23:48.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa19800640 1 --2- 172.21.15.47:0/3736887416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffa140d0360 0x7ffa1415d4f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:48.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa19800640 1 -- 172.21.15.47:0/3736887416 >> 172.21.15.47:0/3736887416 conn(0x7ffa140d1bf0 msgr2=0x7ffa140d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:48.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa19800640 1 -- 172.21.15.47:0/3736887416 shutdown_connections 2024-02-19T15:23:48.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa19800640 1 -- 172.21.15.47:0/3736887416 wait complete. 2024-02-19T15:23:48.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa19800640 1 Processor -- start 2024-02-19T15:23:48.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa19800640 1 -- start start 2024-02-19T15:23:48.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa19800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa140cedb0 0x7ffa14159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:48.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa19800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffa140d0360 0x7ffa140c7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:48.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa19800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffa1415bcd0 0x7ffa140c7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:48.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa19800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa140cdb90 con 0x7ffa140d0360 2024-02-19T15:23:48.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa19800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa14159a50 con 0x7ffa140cedb0 2024-02-19T15:23:48.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa19800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa140c6720 con 0x7ffa1415bcd0 2024-02-19T15:23:48.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa0a000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffa140d0360 0x7ffa140c7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:48.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa0a000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffa140d0360 0x7ffa140c7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33406/0 (socket says 172.21.15.47:33406) 2024-02-19T15:23:48.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.459+0000 7ffa0a000640 1 -- 172.21.15.47:0/1372597099 learned_addr learned my addr 172.21.15.47:0/1372597099 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:48.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.463+0000 7ffa13400640 1 --2- 172.21.15.47:0/1372597099 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffa1415bcd0 0x7ffa140c7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:48.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.463+0000 7ffa12a00640 1 --2- 172.21.15.47:0/1372597099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa140cedb0 0x7ffa14159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:48.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.463+0000 7ffa0a000640 1 -- 172.21.15.47:0/1372597099 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffa1415bcd0 msgr2=0x7ffa140c7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:48.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.463+0000 7ffa0a000640 1 --2- 172.21.15.47:0/1372597099 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffa1415bcd0 0x7ffa140c7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:48.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.463+0000 7ffa0a000640 1 -- 172.21.15.47:0/1372597099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa140cedb0 msgr2=0x7ffa14159510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:48.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.463+0000 7ffa0a000640 1 --2- 172.21.15.47:0/1372597099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa140cedb0 0x7ffa14159510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:48.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.463+0000 7ffa0a000640 1 -- 172.21.15.47:0/1372597099 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0c009a70 con 0x7ffa140d0360 2024-02-19T15:23:48.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.463+0000 7ffa0a000640 1 --2- 172.21.15.47:0/1372597099 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffa140d0360 0x7ffa140c7500 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc00c420 tx=0x7ff9fc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:48.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.463+0000 7ffa0b400640 1 -- 172.21.15.47:0/1372597099 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc005cf0 con 0x7ffa140d0360 2024-02-19T15:23:48.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.463+0000 7ffa0b400640 1 -- 172.21.15.47:0/1372597099 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9fc005e50 con 0x7ffa140d0360 2024-02-19T15:23:48.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.463+0000 7ffa19800640 1 -- 172.21.15.47:0/1372597099 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa1415d240 con 0x7ffa140d0360 2024-02-19T15:23:48.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.463+0000 7ffa0b400640 1 -- 172.21.15.47:0/1372597099 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc007310 con 0x7ffa140d0360 2024-02-19T15:23:48.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.463+0000 7ffa19800640 1 -- 172.21.15.47:0/1372597099 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa140c9b50 con 0x7ffa140d0360 2024-02-19T15:23:48.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.463+0000 7ffa19800640 1 -- 172.21.15.47:0/1372597099 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa140cec20 con 0x7ffa140d0360 2024-02-19T15:23:48.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.463+0000 7ffa0b400640 1 -- 172.21.15.47:0/1372597099 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff9fc007470 con 0x7ffa140d0360 2024-02-19T15:23:48.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.467+0000 7ffa0b400640 1 --2- 172.21.15.47:0/1372597099 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff9f0066e40 0x7ff9f0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:48.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.467+0000 7ffa0b400640 1 -- 172.21.15.47:0/1372597099 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9fc083ff0 con 0x7ffa140d0360 2024-02-19T15:23:48.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.467+0000 7ffa12a00640 1 --2- 172.21.15.47:0/1372597099 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff9f0066e40 0x7ff9f0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:48.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.467+0000 7ffa12a00640 1 --2- 172.21.15.47:0/1372597099 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff9f0066e40 0x7ff9f0069300 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ffa040011f0 tx=0x7ffa04009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:48.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.471+0000 7ffa0b400640 1 -- 172.21.15.47:0/1372597099 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff9fc050e50 con 0x7ffa140d0360 2024-02-19T15:23:48.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.679+0000 7ffa19800640 1 -- 172.21.15.47:0/1372597099 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffa1415de40 con 0x7ff9f0066e40 2024-02-19T15:23:48.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.687+0000 7ffa0b400640 1 -- 172.21.15.47:0/1372597099 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288362 (secure 0 0 0) 0x7ffa040014e0 con 0x7ff9f0066e40 2024-02-19T15:23:48.695 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:48.697 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:23:48.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.691+0000 7ffa19800640 1 -- 172.21.15.47:0/1372597099 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff9f0066e40 msgr2=0x7ff9f0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:48.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.691+0000 7ffa19800640 1 --2- 172.21.15.47:0/1372597099 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff9f0066e40 0x7ff9f0069300 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ffa040011f0 tx=0x7ffa04009040 comp rx=0 tx=0).stop 2024-02-19T15:23:48.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.691+0000 7ffa19800640 1 -- 172.21.15.47:0/1372597099 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffa140d0360 msgr2=0x7ffa140c7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:48.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.691+0000 7ffa19800640 1 --2- 172.21.15.47:0/1372597099 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffa140d0360 0x7ffa140c7500 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc00c420 tx=0x7ff9fc00c8e0 comp rx=0 tx=0).stop 2024-02-19T15:23:48.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.691+0000 7ffa19800640 1 -- 172.21.15.47:0/1372597099 shutdown_connections 2024-02-19T15:23:48.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.691+0000 7ffa19800640 1 --2- 172.21.15.47:0/1372597099 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff9f0066e40 0x7ff9f0069300 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:48.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.691+0000 7ffa19800640 1 --2- 172.21.15.47:0/1372597099 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffa1415bcd0 0x7ffa140c7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:48.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.691+0000 7ffa19800640 1 --2- 172.21.15.47:0/1372597099 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffa140d0360 0x7ffa140c7500 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:48.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.691+0000 7ffa19800640 1 --2- 172.21.15.47:0/1372597099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa140cedb0 0x7ffa14159510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:48.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.691+0000 7ffa19800640 1 -- 172.21.15.47:0/1372597099 >> 172.21.15.47:0/1372597099 conn(0x7ffa140d1bf0 msgr2=0x7ffa140cd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:48.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.691+0000 7ffa19800640 1 -- 172.21.15.47:0/1372597099 shutdown_connections 2024-02-19T15:23:48.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:48.691+0000 7ffa19800640 1 -- 172.21.15.47:0/1372597099 wait complete. 2024-02-19T15:23:48.725 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-02-19T15:23:47.170538+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":246420,"kb_used_data":5276,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754728300,"statfs":{"total":773094113280,"available":772841779200,"internally_reserved":0,"allocated":5402624,"data_stored":2868958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":246795826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000943"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194769+0000","last_change":"2024-02-19T15:23:42.971601+0000","last_active":"2024-02-19T15:23:44.194769+0000","last_peered":"2024-02-19T15:23:44.194769+0000","last_clean":"2024-02-19T15:23:44.194769+0000","last_became_active":"2024-02-19T15:23:42.971304+0000","last_became_peered":"2024-02-19T15:23:42.971304+0000","last_unstale":"2024-02-19T15:23:44.194769+0000","last_undegraded":"2024-02-19T15:23:44.194769+0000","last_fullsized":"2024-02-19T15:23:44.194769+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:42:40.033469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734972+0000","last_change":"2024-02-19T15:23:42.971058+0000","last_active":"2024-02-19T15:23:44.734972+0000","last_peered":"2024-02-19T15:23:44.734972+0000","last_clean":"2024-02-19T15:23:44.734972+0000","last_became_active":"2024-02-19T15:23:42.970038+0000","last_became_peered":"2024-02-19T15:23:42.970038+0000","last_unstale":"2024-02-19T15:23:44.734972+0000","last_undegraded":"2024-02-19T15:23:44.734972+0000","last_fullsized":"2024-02-19T15:23:44.734972+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:18:23.687527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735368+0000","last_change":"2024-02-19T15:23:42.969350+0000","last_active":"2024-02-19T15:23:44.735368+0000","last_peered":"2024-02-19T15:23:44.735368+0000","last_clean":"2024-02-19T15:23:44.735368+0000","last_became_active":"2024-02-19T15:23:42.969021+0000","last_became_peered":"2024-02-19T15:23:42.969021+0000","last_unstale":"2024-02-19T15:23:44.735368+0000","last_undegraded":"2024-02-19T15:23:44.735368+0000","last_fullsized":"2024-02-19T15:23:44.735368+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:57:21.424216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946334+0000","last_change":"2024-02-19T15:23:42.962243+0000","last_active":"2024-02-19T15:23:43.946334+0000","last_peered":"2024-02-19T15:23:43.946334+0000","last_clean":"2024-02-19T15:23:43.946334+0000","last_became_active":"2024-02-19T15:23:42.961999+0000","last_became_peered":"2024-02-19T15:23:42.961999+0000","last_unstale":"2024-02-19T15:23:43.946334+0000","last_undegraded":"2024-02-19T15:23:43.946334+0000","last_fullsized":"2024-02-19T15:23:43.946334+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:25:42.171965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735126+0000","last_change":"2024-02-19T15:23:42.970850+0000","last_active":"2024-02-19T15:23:44.735126+0000","last_peered":"2024-02-19T15:23:44.735126+0000","last_clean":"2024-02-19T15:23:44.735126+0000","last_became_active":"2024-02-19T15:23:42.969695+0000","last_became_peered":"2024-02-19T15:23:42.969695+0000","last_unstale":"2024-02-19T15:23:44.735126+0000","last_undegraded":"2024-02-19T15:23:44.735126+0000","last_fullsized":"2024-02-19T15:23:44.735126+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:51:57.116389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734445+0000","last_change":"2024-02-19T15:23:42.972292+0000","last_active":"2024-02-19T15:23:44.734445+0000","last_peered":"2024-02-19T15:23:44.734445+0000","last_clean":"2024-02-19T15:23:44.734445+0000","last_became_active":"2024-02-19T15:23:42.971723+0000","last_became_peered":"2024-02-19T15:23:42.971723+0000","last_unstale":"2024-02-19T15:23:44.734445+0000","last_undegraded":"2024-02-19T15:23:44.734445+0000","last_fullsized":"2024-02-19T15:23:44.734445+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:30:47.422419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194498+0000","last_change":"2024-02-19T15:23:42.971748+0000","last_active":"2024-02-19T15:23:44.194498+0000","last_peered":"2024-02-19T15:23:44.194498+0000","last_clean":"2024-02-19T15:23:44.194498+0000","last_became_active":"2024-02-19T15:23:42.971435+0000","last_became_peered":"2024-02-19T15:23:42.971435+0000","last_unstale":"2024-02-19T15:23:44.194498+0000","last_undegraded":"2024-02-19T15:23:44.194498+0000","last_fullsized":"2024-02-19T15:23:44.194498+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:09:04.319698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945616+0000","last_change":"2024-02-19T15:23:42.967933+0000","last_active":"2024-02-19T15:23:43.945616+0000","last_peered":"2024-02-19T15:23:43.945616+0000","last_clean":"2024-02-19T15:23:43.945616+0000","last_became_active":"2024-02-19T15:23:42.967628+0000","last_became_peered":"2024-02-19T15:23:42.967628+0000","last_unstale":"2024-02-19T15:23:43.945616+0000","last_undegraded":"2024-02-19T15:23:43.945616+0000","last_fullsized":"2024-02-19T15:23:43.945616+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:23.119868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946834+0000","last_change":"2024-02-19T15:23:42.968450+0000","last_active":"2024-02-19T15:23:43.946834+0000","last_peered":"2024-02-19T15:23:43.946834+0000","last_clean":"2024-02-19T15:23:43.946834+0000","last_became_active":"2024-02-19T15:23:42.968127+0000","last_became_peered":"2024-02-19T15:23:42.968127+0000","last_unstale":"2024-02-19T15:23:43.946834+0000","last_undegraded":"2024-02-19T15:23:43.946834+0000","last_fullsized":"2024-02-19T15:23:43.946834+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:08:50.431318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946763+0000","last_change":"2024-02-19T15:23:42.967870+0000","last_active":"2024-02-19T15:23:43.946763+0000","last_peered":"2024-02-19T15:23:43.946763+0000","last_clean":"2024-02-19T15:23:43.946763+0000","last_became_active":"2024-02-19T15:23:42.967493+0000","last_became_peered":"2024-02-19T15:23:42.967493+0000","last_unstale":"2024-02-19T15:23:43.946763+0000","last_undegraded":"2024-02-19T15:23:43.946763+0000","last_fullsized":"2024-02-19T15:23:43.946763+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:41:06.816999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946394+0000","last_change":"2024-02-19T15:23:42.981481+0000","last_active":"2024-02-19T15:23:43.946394+0000","last_peered":"2024-02-19T15:23:43.946394+0000","last_clean":"2024-02-19T15:23:43.946394+0000","last_became_active":"2024-02-19T15:23:42.981177+0000","last_became_peered":"2024-02-19T15:23:42.981177+0000","last_unstale":"2024-02-19T15:23:43.946394+0000","last_undegraded":"2024-02-19T15:23:43.946394+0000","last_fullsized":"2024-02-19T15:23:43.946394+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:56:04.086784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734943+0000","last_change":"2024-02-19T15:23:42.970720+0000","last_active":"2024-02-19T15:23:44.734943+0000","last_peered":"2024-02-19T15:23:44.734943+0000","last_clean":"2024-02-19T15:23:44.734943+0000","last_became_active":"2024-02-19T15:23:42.970354+0000","last_became_peered":"2024-02-19T15:23:42.970354+0000","last_unstale":"2024-02-19T15:23:44.734943+0000","last_undegraded":"2024-02-19T15:23:44.734943+0000","last_fullsized":"2024-02-19T15:23:44.734943+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:08:01.896505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061189+0000","last_change":"2024-02-19T15:23:42.981071+0000","last_active":"2024-02-19T15:23:44.061189+0000","last_peered":"2024-02-19T15:23:44.061189+0000","last_clean":"2024-02-19T15:23:44.061189+0000","last_became_active":"2024-02-19T15:23:42.980763+0000","last_became_peered":"2024-02-19T15:23:42.980763+0000","last_unstale":"2024-02-19T15:23:44.061189+0000","last_undegraded":"2024-02-19T15:23:44.061189+0000","last_fullsized":"2024-02-19T15:23:44.061189+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:07:37.152277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946242+0000","last_change":"2024-02-19T15:23:42.979970+0000","last_active":"2024-02-19T15:23:43.946242+0000","last_peered":"2024-02-19T15:23:43.946242+0000","last_clean":"2024-02-19T15:23:43.946242+0000","last_became_active":"2024-02-19T15:23:42.968768+0000","last_became_peered":"2024-02-19T15:23:42.968768+0000","last_unstale":"2024-02-19T15:23:43.946242+0000","last_undegraded":"2024-02-19T15:23:43.946242+0000","last_fullsized":"2024-02-19T15:23:43.946242+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:18:27.465577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734344+0000","last_change":"2024-02-19T15:23:42.974598+0000","last_active":"2024-02-19T15:23:44.734344+0000","last_peered":"2024-02-19T15:23:44.734344+0000","last_clean":"2024-02-19T15:23:44.734344+0000","last_became_active":"2024-02-19T15:23:42.974335+0000","last_became_peered":"2024-02-19T15:23:42.974335+0000","last_unstale":"2024-02-19T15:23:44.734344+0000","last_undegraded":"2024-02-19T15:23:44.734344+0000","last_fullsized":"2024-02-19T15:23:44.734344+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:50:40.431301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.945858+0000","last_change":"2024-02-19T15:23:41.945858+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.945858+0000","last_undegraded":"2024-02-19T15:23:41.945858+0000","last_fullsized":"2024-02-19T15:23:41.945858+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735637+0000","last_change":"2024-02-19T15:23:42.964895+0000","last_active":"2024-02-19T15:23:44.735637+0000","last_peered":"2024-02-19T15:23:44.735637+0000","last_clean":"2024-02-19T15:23:44.735637+0000","last_became_active":"2024-02-19T15:23:42.963832+0000","last_became_peered":"2024-02-19T15:23:42.963832+0000","last_unstale":"2024-02-19T15:23:44.735637+0000","last_undegraded":"2024-02-19T15:23:44.735637+0000","last_fullsized":"2024-02-19T15:23:44.735637+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:01:27.305444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735533+0000","last_change":"2024-02-19T15:23:42.969998+0000","last_active":"2024-02-19T15:23:44.735533+0000","last_peered":"2024-02-19T15:23:44.735533+0000","last_clean":"2024-02-19T15:23:44.735533+0000","last_became_active":"2024-02-19T15:23:42.969741+0000","last_became_peered":"2024-02-19T15:23:42.969741+0000","last_unstale":"2024-02-19T15:23:44.735533+0000","last_undegraded":"2024-02-19T15:23:44.735533+0000","last_fullsized":"2024-02-19T15:23:44.735533+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:18:07.650507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.942749+0000","last_change":"2024-02-19T15:23:41.942749+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.942749+0000","last_undegraded":"2024-02-19T15:23:41.942749+0000","last_fullsized":"2024-02-19T15:23:41.942749+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061366+0000","last_change":"2024-02-19T15:23:42.986099+0000","last_active":"2024-02-19T15:23:44.061366+0000","last_peered":"2024-02-19T15:23:44.061366+0000","last_clean":"2024-02-19T15:23:44.061366+0000","last_became_active":"2024-02-19T15:23:42.985685+0000","last_became_peered":"2024-02-19T15:23:42.985685+0000","last_unstale":"2024-02-19T15:23:44.061366+0000","last_undegraded":"2024-02-19T15:23:44.061366+0000","last_fullsized":"2024-02-19T15:23:44.061366+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:13:36.682515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734637+0000","last_change":"2024-02-19T15:23:42.973184+0000","last_active":"2024-02-19T15:23:44.734637+0000","last_peered":"2024-02-19T15:23:44.734637+0000","last_clean":"2024-02-19T15:23:44.734637+0000","last_became_active":"2024-02-19T15:23:42.972228+0000","last_became_peered":"2024-02-19T15:23:42.972228+0000","last_unstale":"2024-02-19T15:23:44.734637+0000","last_undegraded":"2024-02-19T15:23:44.734637+0000","last_fullsized":"2024-02-19T15:23:44.734637+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:03.977088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946364+0000","last_change":"2024-02-19T15:23:42.979749+0000","last_active":"2024-02-19T15:23:43.946364+0000","last_peered":"2024-02-19T15:23:43.946364+0000","last_clean":"2024-02-19T15:23:43.946364+0000","last_became_active":"2024-02-19T15:23:42.968380+0000","last_became_peered":"2024-02-19T15:23:42.968380+0000","last_unstale":"2024-02-19T15:23:43.946364+0000","last_undegraded":"2024-02-19T15:23:43.946364+0000","last_fullsized":"2024-02-19T15:23:43.946364+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:04:27.961177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.942606+0000","last_change":"2024-02-19T15:23:41.942606+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.942606+0000","last_undegraded":"2024-02-19T15:23:41.942606+0000","last_fullsized":"2024-02-19T15:23:41.942606+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.945345+0000","last_change":"2024-02-19T15:23:41.945345+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.945345+0000","last_undegraded":"2024-02-19T15:23:41.945345+0000","last_fullsized":"2024-02-19T15:23:41.945345+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.945592+0000","last_change":"2024-02-19T15:23:41.945592+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.945592+0000","last_undegraded":"2024-02-19T15:23:41.945592+0000","last_fullsized":"2024-02-19T15:23:41.945592+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061361+0000","last_change":"2024-02-19T15:23:42.980908+0000","last_active":"2024-02-19T15:23:44.061361+0000","last_peered":"2024-02-19T15:23:44.061361+0000","last_clean":"2024-02-19T15:23:44.061361+0000","last_became_active":"2024-02-19T15:23:42.980592+0000","last_became_peered":"2024-02-19T15:23:42.980592+0000","last_unstale":"2024-02-19T15:23:44.061361+0000","last_undegraded":"2024-02-19T15:23:44.061361+0000","last_fullsized":"2024-02-19T15:23:44.061361+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:39:51.560827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735567+0000","last_change":"2024-02-19T15:23:42.961547+0000","last_active":"2024-02-19T15:23:44.735567+0000","last_peered":"2024-02-19T15:23:44.735567+0000","last_clean":"2024-02-19T15:23:44.735567+0000","last_became_active":"2024-02-19T15:23:42.961215+0000","last_became_peered":"2024-02-19T15:23:42.961215+0000","last_unstale":"2024-02-19T15:23:44.735567+0000","last_undegraded":"2024-02-19T15:23:44.735567+0000","last_fullsized":"2024-02-19T15:23:44.735567+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:23:12.865791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735161+0000","last_change":"2024-02-19T15:23:42.971532+0000","last_active":"2024-02-19T15:23:44.735161+0000","last_peered":"2024-02-19T15:23:44.735161+0000","last_clean":"2024-02-19T15:23:44.735161+0000","last_became_active":"2024-02-19T15:23:42.971181+0000","last_became_peered":"2024-02-19T15:23:42.971181+0000","last_unstale":"2024-02-19T15:23:44.735161+0000","last_undegraded":"2024-02-19T15:23:44.735161+0000","last_fullsized":"2024-02-19T15:23:44.735161+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:34:33.315961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.944819+0000","last_change":"2024-02-19T15:23:41.944819+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.944819+0000","last_undegraded":"2024-02-19T15:23:41.944819+0000","last_fullsized":"2024-02-19T15:23:41.944819+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734745+0000","last_change":"2024-02-19T15:23:42.961009+0000","last_active":"2024-02-19T15:23:44.734745+0000","last_peered":"2024-02-19T15:23:44.734745+0000","last_clean":"2024-02-19T15:23:44.734745+0000","last_became_active":"2024-02-19T15:23:42.960723+0000","last_became_peered":"2024-02-19T15:23:42.960723+0000","last_unstale":"2024-02-19T15:23:44.734745+0000","last_undegraded":"2024-02-19T15:23:44.734745+0000","last_fullsized":"2024-02-19T15:23:44.734745+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:57:53.597108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734459+0000","last_change":"2024-02-19T15:23:42.973247+0000","last_active":"2024-02-19T15:23:44.734459+0000","last_peered":"2024-02-19T15:23:44.734459+0000","last_clean":"2024-02-19T15:23:44.734459+0000","last_became_active":"2024-02-19T15:23:42.972957+0000","last_became_peered":"2024-02-19T15:23:42.972957+0000","last_unstale":"2024-02-19T15:23:44.734459+0000","last_undegraded":"2024-02-19T15:23:44.734459+0000","last_fullsized":"2024-02-19T15:23:44.734459+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:36:54.676262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061144+0000","last_change":"2024-02-19T15:23:42.962393+0000","last_active":"2024-02-19T15:23:44.061144+0000","last_peered":"2024-02-19T15:23:44.061144+0000","last_clean":"2024-02-19T15:23:44.061144+0000","last_became_active":"2024-02-19T15:23:42.962112+0000","last_became_peered":"2024-02-19T15:23:42.962112+0000","last_unstale":"2024-02-19T15:23:44.061144+0000","last_undegraded":"2024-02-19T15:23:44.061144+0000","last_fullsized":"2024-02-19T15:23:44.061144+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:28:48.394306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947613+0000","last_change":"2024-02-19T15:23:42.974733+0000","last_active":"2024-02-19T15:23:43.947613+0000","last_peered":"2024-02-19T15:23:43.947613+0000","last_clean":"2024-02-19T15:23:43.947613+0000","last_became_active":"2024-02-19T15:23:42.974439+0000","last_became_peered":"2024-02-19T15:23:42.974439+0000","last_unstale":"2024-02-19T15:23:43.947613+0000","last_undegraded":"2024-02-19T15:23:43.947613+0000","last_fullsized":"2024-02-19T15:23:43.947613+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:39:12.299839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734807+0000","last_change":"2024-02-19T15:23:41.960560+0000","last_active":"2024-02-19T15:23:44.734807+0000","last_peered":"2024-02-19T15:23:44.734807+0000","last_clean":"2024-02-19T15:23:44.734807+0000","last_became_active":"2024-02-19T15:23:41.960237+0000","last_became_peered":"2024-02-19T15:23:41.960237+0000","last_unstale":"2024-02-19T15:23:44.734807+0000","last_undegraded":"2024-02-19T15:23:44.734807+0000","last_fullsized":"2024-02-19T15:23:44.734807+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:14:27.569498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.944998+0000","last_change":"2024-02-19T15:23:41.944998+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.944998+0000","last_undegraded":"2024-02-19T15:23:41.944998+0000","last_fullsized":"2024-02-19T15:23:41.944998+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735605+0000","last_change":"2024-02-19T15:23:41.960962+0000","last_active":"2024-02-19T15:23:44.735605+0000","last_peered":"2024-02-19T15:23:44.735605+0000","last_clean":"2024-02-19T15:23:44.735605+0000","last_became_active":"2024-02-19T15:23:41.959425+0000","last_became_peered":"2024-02-19T15:23:41.959425+0000","last_unstale":"2024-02-19T15:23:44.735605+0000","last_undegraded":"2024-02-19T15:23:44.735605+0000","last_fullsized":"2024-02-19T15:23:44.735605+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:54.136668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735315+0000","last_change":"2024-02-19T15:23:42.972962+0000","last_active":"2024-02-19T15:23:44.735315+0000","last_peered":"2024-02-19T15:23:44.735315+0000","last_clean":"2024-02-19T15:23:44.735315+0000","last_became_active":"2024-02-19T15:23:42.972625+0000","last_became_peered":"2024-02-19T15:23:42.972625+0000","last_unstale":"2024-02-19T15:23:44.735315+0000","last_undegraded":"2024-02-19T15:23:44.735315+0000","last_fullsized":"2024-02-19T15:23:44.735315+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:40:40.055788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947602+0000","last_change":"2024-02-19T15:23:41.948520+0000","last_active":"2024-02-19T15:23:43.947602+0000","last_peered":"2024-02-19T15:23:43.947602+0000","last_clean":"2024-02-19T15:23:43.947602+0000","last_became_active":"2024-02-19T15:23:41.948233+0000","last_became_peered":"2024-02-19T15:23:41.948233+0000","last_unstale":"2024-02-19T15:23:43.947602+0000","last_undegraded":"2024-02-19T15:23:43.947602+0000","last_fullsized":"2024-02-19T15:23:43.947602+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:50:30.218103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947636+0000","last_change":"2024-02-19T15:23:42.961975+0000","last_active":"2024-02-19T15:23:43.947636+0000","last_peered":"2024-02-19T15:23:43.947636+0000","last_clean":"2024-02-19T15:23:43.947636+0000","last_became_active":"2024-02-19T15:23:42.961615+0000","last_became_peered":"2024-02-19T15:23:42.961615+0000","last_unstale":"2024-02-19T15:23:43.947636+0000","last_undegraded":"2024-02-19T15:23:43.947636+0000","last_fullsized":"2024-02-19T15:23:43.947636+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:57:21.424216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194190+0000","last_change":"2024-02-19T15:23:41.961942+0000","last_active":"2024-02-19T15:23:44.194190+0000","last_peered":"2024-02-19T15:23:44.194190+0000","last_clean":"2024-02-19T15:23:44.194190+0000","last_became_active":"2024-02-19T15:23:41.961659+0000","last_became_peered":"2024-02-19T15:23:41.961659+0000","last_unstale":"2024-02-19T15:23:44.194190+0000","last_undegraded":"2024-02-19T15:23:44.194190+0000","last_fullsized":"2024-02-19T15:23:44.194190+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:14:27.569498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734930+0000","last_change":"2024-02-19T15:23:42.961945+0000","last_active":"2024-02-19T15:23:44.734930+0000","last_peered":"2024-02-19T15:23:44.734930+0000","last_clean":"2024-02-19T15:23:44.734930+0000","last_became_active":"2024-02-19T15:23:42.961269+0000","last_became_peered":"2024-02-19T15:23:42.961269+0000","last_unstale":"2024-02-19T15:23:44.734930+0000","last_undegraded":"2024-02-19T15:23:44.734930+0000","last_fullsized":"2024-02-19T15:23:44.734930+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:39:12.299839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734672+0000","last_change":"2024-02-19T15:23:41.966788+0000","last_active":"2024-02-19T15:23:44.734672+0000","last_peered":"2024-02-19T15:23:44.734672+0000","last_clean":"2024-02-19T15:23:44.734672+0000","last_became_active":"2024-02-19T15:23:41.966346+0000","last_became_peered":"2024-02-19T15:23:41.966346+0000","last_unstale":"2024-02-19T15:23:44.734672+0000","last_undegraded":"2024-02-19T15:23:44.734672+0000","last_fullsized":"2024-02-19T15:23:44.734672+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:01:26.302862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946392+0000","last_change":"2024-02-19T15:23:42.968247+0000","last_active":"2024-02-19T15:23:43.946392+0000","last_peered":"2024-02-19T15:23:43.946392+0000","last_clean":"2024-02-19T15:23:43.946392+0000","last_became_active":"2024-02-19T15:23:42.967240+0000","last_became_peered":"2024-02-19T15:23:42.967240+0000","last_unstale":"2024-02-19T15:23:43.946392+0000","last_undegraded":"2024-02-19T15:23:43.946392+0000","last_fullsized":"2024-02-19T15:23:43.946392+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:41:08.031616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735141+0000","last_change":"2024-02-19T15:23:41.955181+0000","last_active":"2024-02-19T15:23:44.735141+0000","last_peered":"2024-02-19T15:23:44.735141+0000","last_clean":"2024-02-19T15:23:44.735141+0000","last_became_active":"2024-02-19T15:23:41.954904+0000","last_became_peered":"2024-02-19T15:23:41.954904+0000","last_unstale":"2024-02-19T15:23:44.735141+0000","last_undegraded":"2024-02-19T15:23:44.735141+0000","last_fullsized":"2024-02-19T15:23:44.735141+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:13:35.679933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945686+0000","last_change":"2024-02-19T15:23:42.980803+0000","last_active":"2024-02-19T15:23:43.945686+0000","last_peered":"2024-02-19T15:23:43.945686+0000","last_clean":"2024-02-19T15:23:43.945686+0000","last_became_active":"2024-02-19T15:23:42.980507+0000","last_became_peered":"2024-02-19T15:23:42.980507+0000","last_unstale":"2024-02-19T15:23:43.945686+0000","last_undegraded":"2024-02-19T15:23:43.945686+0000","last_fullsized":"2024-02-19T15:23:43.945686+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:08:33.495333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734933+0000","last_change":"2024-02-19T15:23:41.951702+0000","last_active":"2024-02-19T15:23:44.734933+0000","last_peered":"2024-02-19T15:23:44.734933+0000","last_clean":"2024-02-19T15:23:44.734933+0000","last_became_active":"2024-02-19T15:23:41.951409+0000","last_became_peered":"2024-02-19T15:23:41.951409+0000","last_unstale":"2024-02-19T15:23:44.734933+0000","last_undegraded":"2024-02-19T15:23:44.734933+0000","last_fullsized":"2024-02-19T15:23:44.734933+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:57:20.421633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.943222+0000","last_change":"2024-02-19T15:23:41.943222+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.943222+0000","last_undegraded":"2024-02-19T15:23:41.943222+0000","last_fullsized":"2024-02-19T15:23:41.943222+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945665+0000","last_change":"2024-02-19T15:23:41.965224+0000","last_active":"2024-02-19T15:23:43.945665+0000","last_peered":"2024-02-19T15:23:43.945665+0000","last_clean":"2024-02-19T15:23:43.945665+0000","last_became_active":"2024-02-19T15:23:41.964654+0000","last_became_peered":"2024-02-19T15:23:41.964654+0000","last_unstale":"2024-02-19T15:23:43.945665+0000","last_undegraded":"2024-02-19T15:23:43.945665+0000","last_fullsized":"2024-02-19T15:23:43.945665+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:49:19.641960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735665+0000","last_change":"2024-02-19T15:23:42.969995+0000","last_active":"2024-02-19T15:23:44.735665+0000","last_peered":"2024-02-19T15:23:44.735665+0000","last_clean":"2024-02-19T15:23:44.735665+0000","last_became_active":"2024-02-19T15:23:42.969651+0000","last_became_peered":"2024-02-19T15:23:42.969651+0000","last_unstale":"2024-02-19T15:23:44.735665+0000","last_undegraded":"2024-02-19T15:23:44.735665+0000","last_fullsized":"2024-02-19T15:23:44.735665+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:45.239277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735692+0000","last_change":"2024-02-19T15:23:41.961017+0000","last_active":"2024-02-19T15:23:44.735692+0000","last_peered":"2024-02-19T15:23:44.735692+0000","last_clean":"2024-02-19T15:23:44.735692+0000","last_became_active":"2024-02-19T15:23:41.960715+0000","last_became_peered":"2024-02-19T15:23:41.960715+0000","last_unstale":"2024-02-19T15:23:44.735692+0000","last_undegraded":"2024-02-19T15:23:44.735692+0000","last_fullsized":"2024-02-19T15:23:44.735692+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:05.009739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194763+0000","last_change":"2024-02-19T15:23:42.966631+0000","last_active":"2024-02-19T15:23:44.194763+0000","last_peered":"2024-02-19T15:23:44.194763+0000","last_clean":"2024-02-19T15:23:44.194763+0000","last_became_active":"2024-02-19T15:23:42.965314+0000","last_became_peered":"2024-02-19T15:23:42.965314+0000","last_unstale":"2024-02-19T15:23:44.194763+0000","last_undegraded":"2024-02-19T15:23:44.194763+0000","last_fullsized":"2024-02-19T15:23:44.194763+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:36:54.676262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946805+0000","last_change":"2024-02-19T15:23:41.958566+0000","last_active":"2024-02-19T15:23:43.946805+0000","last_peered":"2024-02-19T15:23:43.946805+0000","last_clean":"2024-02-19T15:23:43.946805+0000","last_became_active":"2024-02-19T15:23:41.958244+0000","last_became_peered":"2024-02-19T15:23:41.958244+0000","last_unstale":"2024-02-19T15:23:43.946805+0000","last_undegraded":"2024-02-19T15:23:43.946805+0000","last_fullsized":"2024-02-19T15:23:43.946805+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:20:44.307845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734660+0000","last_change":"2024-02-19T15:23:42.972332+0000","last_active":"2024-02-19T15:23:44.734660+0000","last_peered":"2024-02-19T15:23:44.734660+0000","last_clean":"2024-02-19T15:23:44.734660+0000","last_became_active":"2024-02-19T15:23:42.971907+0000","last_became_peered":"2024-02-19T15:23:42.971907+0000","last_unstale":"2024-02-19T15:23:44.734660+0000","last_undegraded":"2024-02-19T15:23:44.734660+0000","last_fullsized":"2024-02-19T15:23:44.734660+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:59:43.637759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194426+0000","last_change":"2024-02-19T15:23:41.964052+0000","last_active":"2024-02-19T15:23:44.194426+0000","last_peered":"2024-02-19T15:23:44.194426+0000","last_clean":"2024-02-19T15:23:44.194426+0000","last_became_active":"2024-02-19T15:23:41.963729+0000","last_became_peered":"2024-02-19T15:23:41.963729+0000","last_unstale":"2024-02-19T15:23:44.194426+0000","last_undegraded":"2024-02-19T15:23:44.194426+0000","last_fullsized":"2024-02-19T15:23:44.194426+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:02:12.172289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194355+0000","last_change":"2024-02-19T15:23:42.969755+0000","last_active":"2024-02-19T15:23:44.194355+0000","last_peered":"2024-02-19T15:23:44.194355+0000","last_clean":"2024-02-19T15:23:44.194355+0000","last_became_active":"2024-02-19T15:23:42.969477+0000","last_became_peered":"2024-02-19T15:23:42.969477+0000","last_unstale":"2024-02-19T15:23:44.194355+0000","last_undegraded":"2024-02-19T15:23:44.194355+0000","last_fullsized":"2024-02-19T15:23:44.194355+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:47:09.498437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":20,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061276+0000","last_change":"2024-02-19T15:23:41.959735+0000","last_active":"2024-02-19T15:23:44.061276+0000","last_peered":"2024-02-19T15:23:44.061276+0000","last_clean":"2024-02-19T15:23:44.061276+0000","last_became_active":"2024-02-19T15:23:41.959171+0000","last_became_peered":"2024-02-19T15:23:41.959171+0000","last_unstale":"2024-02-19T15:23:44.061276+0000","last_undegraded":"2024-02-19T15:23:44.061276+0000","last_fullsized":"2024-02-19T15:23:44.061276+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:07:41.495783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.944967+0000","last_change":"2024-02-19T15:23:41.944967+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.944967+0000","last_undegraded":"2024-02-19T15:23:41.944967+0000","last_fullsized":"2024-02-19T15:23:41.944967+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734694+0000","last_change":"2024-02-19T15:23:41.964586+0000","last_active":"2024-02-19T15:23:44.734694+0000","last_peered":"2024-02-19T15:23:44.734694+0000","last_clean":"2024-02-19T15:23:44.734694+0000","last_became_active":"2024-02-19T15:23:41.964176+0000","last_became_peered":"2024-02-19T15:23:41.964176+0000","last_unstale":"2024-02-19T15:23:44.734694+0000","last_undegraded":"2024-02-19T15:23:44.734694+0000","last_fullsized":"2024-02-19T15:23:44.734694+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:44.236695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946493+0000","last_change":"2024-02-19T15:23:42.965254+0000","last_active":"2024-02-19T15:23:43.946493+0000","last_peered":"2024-02-19T15:23:43.946493+0000","last_clean":"2024-02-19T15:23:43.946493+0000","last_became_active":"2024-02-19T15:23:42.964990+0000","last_became_peered":"2024-02-19T15:23:42.964990+0000","last_unstale":"2024-02-19T15:23:43.946493+0000","last_undegraded":"2024-02-19T15:23:43.946493+0000","last_fullsized":"2024-02-19T15:23:43.946493+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:37:19.468760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194207+0000","last_change":"2024-02-19T15:23:41.965760+0000","last_active":"2024-02-19T15:23:44.194207+0000","last_peered":"2024-02-19T15:23:44.194207+0000","last_clean":"2024-02-19T15:23:44.194207+0000","last_became_active":"2024-02-19T15:23:41.965423+0000","last_became_peered":"2024-02-19T15:23:41.965423+0000","last_unstale":"2024-02-19T15:23:44.194207+0000","last_undegraded":"2024-02-19T15:23:44.194207+0000","last_fullsized":"2024-02-19T15:23:44.194207+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:01:16.303652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734331+0000","last_change":"2024-02-19T15:23:42.970653+0000","last_active":"2024-02-19T15:23:44.734331+0000","last_peered":"2024-02-19T15:23:44.734331+0000","last_clean":"2024-02-19T15:23:44.734331+0000","last_became_active":"2024-02-19T15:23:42.970394+0000","last_became_peered":"2024-02-19T15:23:42.970394+0000","last_unstale":"2024-02-19T15:23:44.734331+0000","last_undegraded":"2024-02-19T15:23:44.734331+0000","last_fullsized":"2024-02-19T15:23:44.734331+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:09:04.319698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734401+0000","last_change":"2024-02-19T15:23:41.964786+0000","last_active":"2024-02-19T15:23:44.734401+0000","last_peered":"2024-02-19T15:23:44.734401+0000","last_clean":"2024-02-19T15:23:44.734401+0000","last_became_active":"2024-02-19T15:23:41.964085+0000","last_became_peered":"2024-02-19T15:23:41.964085+0000","last_unstale":"2024-02-19T15:23:44.734401+0000","last_undegraded":"2024-02-19T15:23:44.734401+0000","last_fullsized":"2024-02-19T15:23:44.734401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:01:16.303652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194624+0000","last_change":"2024-02-19T15:23:42.974233+0000","last_active":"2024-02-19T15:23:44.194624+0000","last_peered":"2024-02-19T15:23:44.194624+0000","last_clean":"2024-02-19T15:23:44.194624+0000","last_became_active":"2024-02-19T15:23:42.973975+0000","last_became_peered":"2024-02-19T15:23:42.973975+0000","last_unstale":"2024-02-19T15:23:44.194624+0000","last_undegraded":"2024-02-19T15:23:44.194624+0000","last_fullsized":"2024-02-19T15:23:44.194624+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:30:47.422419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":22,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945742+0000","last_change":"2024-02-19T15:23:41.957848+0000","last_active":"2024-02-19T15:23:43.945742+0000","last_peered":"2024-02-19T15:23:43.945742+0000","last_clean":"2024-02-19T15:23:43.945742+0000","last_became_active":"2024-02-19T15:23:41.957529+0000","last_became_peered":"2024-02-19T15:23:41.957529+0000","last_unstale":"2024-02-19T15:23:43.945742+0000","last_undegraded":"2024-02-19T15:23:43.945742+0000","last_fullsized":"2024-02-19T15:23:43.945742+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:41:14.948978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061427+0000","last_change":"2024-02-19T15:23:42.971155+0000","last_active":"2024-02-19T15:23:44.061427+0000","last_peered":"2024-02-19T15:23:44.061427+0000","last_clean":"2024-02-19T15:23:44.061427+0000","last_became_active":"2024-02-19T15:23:42.970879+0000","last_became_peered":"2024-02-19T15:23:42.970879+0000","last_unstale":"2024-02-19T15:23:44.061427+0000","last_undegraded":"2024-02-19T15:23:44.061427+0000","last_fullsized":"2024-02-19T15:23:44.061427+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:25:52.362277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061334+0000","last_change":"2024-02-19T15:23:41.966015+0000","last_active":"2024-02-19T15:23:44.061334+0000","last_peered":"2024-02-19T15:23:44.061334+0000","last_clean":"2024-02-19T15:23:44.061334+0000","last_became_active":"2024-02-19T15:23:41.965678+0000","last_became_peered":"2024-02-19T15:23:41.965678+0000","last_unstale":"2024-02-19T15:23:44.061334+0000","last_undegraded":"2024-02-19T15:23:44.061334+0000","last_fullsized":"2024-02-19T15:23:44.061334+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:27:46.811039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946825+0000","last_change":"2024-02-19T15:23:42.983074+0000","last_active":"2024-02-19T15:23:43.946825+0000","last_peered":"2024-02-19T15:23:43.946825+0000","last_clean":"2024-02-19T15:23:43.946825+0000","last_became_active":"2024-02-19T15:23:42.982771+0000","last_became_peered":"2024-02-19T15:23:42.982771+0000","last_unstale":"2024-02-19T15:23:43.946825+0000","last_undegraded":"2024-02-19T15:23:43.946825+0000","last_fullsized":"2024-02-19T15:23:43.946825+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:04:26.404860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735691+0000","last_change":"2024-02-19T15:23:41.963486+0000","last_active":"2024-02-19T15:23:44.735691+0000","last_peered":"2024-02-19T15:23:44.735691+0000","last_clean":"2024-02-19T15:23:44.735691+0000","last_became_active":"2024-02-19T15:23:41.962837+0000","last_became_peered":"2024-02-19T15:23:41.962837+0000","last_unstale":"2024-02-19T15:23:44.735691+0000","last_undegraded":"2024-02-19T15:23:44.735691+0000","last_fullsized":"2024-02-19T15:23:44.735691+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:16:42.881994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.944685+0000","last_change":"2024-02-19T15:23:41.944685+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.944685+0000","last_undegraded":"2024-02-19T15:23:41.944685+0000","last_fullsized":"2024-02-19T15:23:41.944685+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735151+0000","last_change":"2024-02-19T15:23:41.952606+0000","last_active":"2024-02-19T15:23:44.735151+0000","last_peered":"2024-02-19T15:23:44.735151+0000","last_clean":"2024-02-19T15:23:44.735151+0000","last_became_active":"2024-02-19T15:23:41.952291+0000","last_became_peered":"2024-02-19T15:23:41.952291+0000","last_unstale":"2024-02-19T15:23:44.735151+0000","last_undegraded":"2024-02-19T15:23:44.735151+0000","last_fullsized":"2024-02-19T15:23:44.735151+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:01:26.302862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.944426+0000","last_change":"2024-02-19T15:23:41.944426+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.944426+0000","last_undegraded":"2024-02-19T15:23:41.944426+0000","last_fullsized":"2024-02-19T15:23:41.944426+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734712+0000","last_change":"2024-02-19T15:23:41.962283+0000","last_active":"2024-02-19T15:23:44.734712+0000","last_peered":"2024-02-19T15:23:44.734712+0000","last_clean":"2024-02-19T15:23:44.734712+0000","last_became_active":"2024-02-19T15:23:41.961397+0000","last_became_peered":"2024-02-19T15:23:41.961397+0000","last_unstale":"2024-02-19T15:23:44.734712+0000","last_undegraded":"2024-02-19T15:23:44.734712+0000","last_fullsized":"2024-02-19T15:23:44.734712+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:45:37.841170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734509+0000","last_change":"2024-02-19T15:23:42.972039+0000","last_active":"2024-02-19T15:23:44.734509+0000","last_peered":"2024-02-19T15:23:44.734509+0000","last_clean":"2024-02-19T15:23:44.734509+0000","last_became_active":"2024-02-19T15:23:42.971311+0000","last_became_peered":"2024-02-19T15:23:42.971311+0000","last_unstale":"2024-02-19T15:23:44.734509+0000","last_undegraded":"2024-02-19T15:23:44.734509+0000","last_fullsized":"2024-02-19T15:23:44.734509+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:42:40.033469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734827+0000","last_change":"2024-02-19T15:23:41.960894+0000","last_active":"2024-02-19T15:23:44.734827+0000","last_peered":"2024-02-19T15:23:44.734827+0000","last_clean":"2024-02-19T15:23:44.734827+0000","last_became_active":"2024-02-19T15:23:41.959201+0000","last_became_peered":"2024-02-19T15:23:41.959201+0000","last_unstale":"2024-02-19T15:23:44.734827+0000","last_undegraded":"2024-02-19T15:23:44.734827+0000","last_fullsized":"2024-02-19T15:23:44.734827+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:32:59.830263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947496+0000","last_change":"2024-02-19T15:23:42.958733+0000","last_active":"2024-02-19T15:23:43.947496+0000","last_peered":"2024-02-19T15:23:43.947496+0000","last_clean":"2024-02-19T15:23:43.947496+0000","last_became_active":"2024-02-19T15:23:42.958345+0000","last_became_peered":"2024-02-19T15:23:42.958345+0000","last_unstale":"2024-02-19T15:23:43.947496+0000","last_undegraded":"2024-02-19T15:23:43.947496+0000","last_fullsized":"2024-02-19T15:23:43.947496+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:01.020038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734643+0000","last_change":"2024-02-19T15:23:41.961678+0000","last_active":"2024-02-19T15:23:44.734643+0000","last_peered":"2024-02-19T15:23:44.734643+0000","last_clean":"2024-02-19T15:23:44.734643+0000","last_became_active":"2024-02-19T15:23:41.960130+0000","last_became_peered":"2024-02-19T15:23:41.960130+0000","last_unstale":"2024-02-19T15:23:44.734643+0000","last_undegraded":"2024-02-19T15:23:44.734643+0000","last_fullsized":"2024-02-19T15:23:44.734643+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:18:06.647924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061083+0000","last_change":"2024-02-19T15:23:42.965223+0000","last_active":"2024-02-19T15:23:44.061083+0000","last_peered":"2024-02-19T15:23:44.061083+0000","last_clean":"2024-02-19T15:23:44.061083+0000","last_became_active":"2024-02-19T15:23:42.964860+0000","last_became_peered":"2024-02-19T15:23:42.964860+0000","last_unstale":"2024-02-19T15:23:44.061083+0000","last_undegraded":"2024-02-19T15:23:44.061083+0000","last_fullsized":"2024-02-19T15:23:44.061083+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:52.445068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945910+0000","last_change":"2024-02-19T15:23:41.966499+0000","last_active":"2024-02-19T15:23:43.945910+0000","last_peered":"2024-02-19T15:23:43.945910+0000","last_clean":"2024-02-19T15:23:43.945910+0000","last_became_active":"2024-02-19T15:23:41.966206+0000","last_became_peered":"2024-02-19T15:23:41.966206+0000","last_unstale":"2024-02-19T15:23:43.945910+0000","last_undegraded":"2024-02-19T15:23:43.945910+0000","last_fullsized":"2024-02-19T15:23:43.945910+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:50:47.865930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945811+0000","last_change":"2024-02-19T15:23:42.982135+0000","last_active":"2024-02-19T15:23:43.945811+0000","last_peered":"2024-02-19T15:23:43.945811+0000","last_clean":"2024-02-19T15:23:43.945811+0000","last_became_active":"2024-02-19T15:23:42.981890+0000","last_became_peered":"2024-02-19T15:23:42.981890+0000","last_unstale":"2024-02-19T15:23:43.945811+0000","last_undegraded":"2024-02-19T15:23:43.945811+0000","last_fullsized":"2024-02-19T15:23:43.945811+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:28.558554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194347+0000","last_change":"2024-02-19T15:23:41.964986+0000","last_active":"2024-02-19T15:23:44.194347+0000","last_peered":"2024-02-19T15:23:44.194347+0000","last_clean":"2024-02-19T15:23:44.194347+0000","last_became_active":"2024-02-19T15:23:41.964713+0000","last_became_peered":"2024-02-19T15:23:41.964713+0000","last_unstale":"2024-02-19T15:23:44.194347+0000","last_undegraded":"2024-02-19T15:23:44.194347+0000","last_fullsized":"2024-02-19T15:23:44.194347+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:32:59.830263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-02-19T15:23:41.943060+0000","last_change":"2024-02-19T15:23:41.943060+0000","last_active":"2024-02-19T15:23:41.932392+0000","last_peered":"2024-02-19T15:23:41.932392+0000","last_clean":"2024-02-19T15:23:41.932392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:23:41.943060+0000","last_undegraded":"2024-02-19T15:23:41.943060+0000","last_fullsized":"2024-02-19T15:23:41.943060+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"45'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734750+0000","last_change":"2024-02-19T15:23:41.968055+0000","last_active":"2024-02-19T15:23:44.734750+0000","last_peered":"2024-02-19T15:23:44.734750+0000","last_clean":"2024-02-19T15:23:44.734750+0000","last_became_active":"2024-02-19T15:23:41.967728+0000","last_became_peered":"2024-02-19T15:23:41.967728+0000","last_unstale":"2024-02-19T15:23:44.734750+0000","last_undegraded":"2024-02-19T15:23:44.734750+0000","last_fullsized":"2024-02-19T15:23:44.734750+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:02:12.172289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946492+0000","last_change":"2024-02-19T15:23:42.966169+0000","last_active":"2024-02-19T15:23:43.946492+0000","last_peered":"2024-02-19T15:23:43.946492+0000","last_clean":"2024-02-19T15:23:43.946492+0000","last_became_active":"2024-02-19T15:23:42.965898+0000","last_became_peered":"2024-02-19T15:23:42.965898+0000","last_unstale":"2024-02-19T15:23:43.946492+0000","last_undegraded":"2024-02-19T15:23:43.946492+0000","last_fullsized":"2024-02-19T15:23:43.946492+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:41:45.823238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194546+0000","last_change":"2024-02-19T15:23:41.968087+0000","last_active":"2024-02-19T15:23:44.194546+0000","last_peered":"2024-02-19T15:23:44.194546+0000","last_clean":"2024-02-19T15:23:44.194546+0000","last_became_active":"2024-02-19T15:23:41.967798+0000","last_became_peered":"2024-02-19T15:23:41.967798+0000","last_unstale":"2024-02-19T15:23:44.194546+0000","last_undegraded":"2024-02-19T15:23:44.194546+0000","last_fullsized":"2024-02-19T15:23:44.194546+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:08:00.893923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194536+0000","last_change":"2024-02-19T15:23:42.971627+0000","last_active":"2024-02-19T15:23:44.194536+0000","last_peered":"2024-02-19T15:23:44.194536+0000","last_clean":"2024-02-19T15:23:44.194536+0000","last_became_active":"2024-02-19T15:23:42.971344+0000","last_became_peered":"2024-02-19T15:23:42.971344+0000","last_unstale":"2024-02-19T15:23:44.194536+0000","last_undegraded":"2024-02-19T15:23:44.194536+0000","last_fullsized":"2024-02-19T15:23:44.194536+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:50:40.431301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947382+0000","last_change":"2024-02-19T15:23:41.948963+0000","last_active":"2024-02-19T15:23:43.947382+0000","last_peered":"2024-02-19T15:23:43.947382+0000","last_clean":"2024-02-19T15:23:43.947382+0000","last_became_active":"2024-02-19T15:23:41.947243+0000","last_became_peered":"2024-02-19T15:23:41.947243+0000","last_unstale":"2024-02-19T15:23:43.947382+0000","last_undegraded":"2024-02-19T15:23:43.947382+0000","last_fullsized":"2024-02-19T15:23:43.947382+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:51:31.286375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734698+0000","last_change":"2024-02-19T15:23:42.974990+0000","last_active":"2024-02-19T15:23:44.734698+0000","last_peered":"2024-02-19T15:23:44.734698+0000","last_clean":"2024-02-19T15:23:44.734698+0000","last_became_active":"2024-02-19T15:23:42.974011+0000","last_became_peered":"2024-02-19T15:23:42.974011+0000","last_unstale":"2024-02-19T15:23:44.734698+0000","last_undegraded":"2024-02-19T15:23:44.734698+0000","last_fullsized":"2024-02-19T15:23:44.734698+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:48.375387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194296+0000","last_change":"2024-02-19T15:23:41.964025+0000","last_active":"2024-02-19T15:23:44.194296+0000","last_peered":"2024-02-19T15:23:44.194296+0000","last_clean":"2024-02-19T15:23:44.194296+0000","last_became_active":"2024-02-19T15:23:41.963717+0000","last_became_peered":"2024-02-19T15:23:41.963717+0000","last_unstale":"2024-02-19T15:23:44.194296+0000","last_undegraded":"2024-02-19T15:23:44.194296+0000","last_fullsized":"2024-02-19T15:23:44.194296+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:45:37.841170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734618+0000","last_change":"2024-02-19T15:23:42.974421+0000","last_active":"2024-02-19T15:23:44.734618+0000","last_peered":"2024-02-19T15:23:44.734618+0000","last_clean":"2024-02-19T15:23:44.734618+0000","last_became_active":"2024-02-19T15:23:42.974009+0000","last_became_peered":"2024-02-19T15:23:42.974009+0000","last_unstale":"2024-02-19T15:23:44.734618+0000","last_undegraded":"2024-02-19T15:23:44.734618+0000","last_fullsized":"2024-02-19T15:23:44.734618+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:36.710691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735244+0000","last_change":"2024-02-19T15:23:41.969004+0000","last_active":"2024-02-19T15:23:44.735244+0000","last_peered":"2024-02-19T15:23:44.735244+0000","last_clean":"2024-02-19T15:23:44.735244+0000","last_became_active":"2024-02-19T15:23:41.968796+0000","last_became_peered":"2024-02-19T15:23:41.968796+0000","last_unstale":"2024-02-19T15:23:44.735244+0000","last_undegraded":"2024-02-19T15:23:44.735244+0000","last_fullsized":"2024-02-19T15:23:44.735244+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:25:51.359695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735100+0000","last_change":"2024-02-19T15:23:42.956889+0000","last_active":"2024-02-19T15:23:44.735100+0000","last_peered":"2024-02-19T15:23:44.735100+0000","last_clean":"2024-02-19T15:23:44.735100+0000","last_became_active":"2024-02-19T15:23:42.953555+0000","last_became_peered":"2024-02-19T15:23:42.953555+0000","last_unstale":"2024-02-19T15:23:44.735100+0000","last_undegraded":"2024-02-19T15:23:44.735100+0000","last_fullsized":"2024-02-19T15:23:44.735100+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:50:45.305646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947245+0000","last_change":"2024-02-19T15:23:41.956700+0000","last_active":"2024-02-19T15:23:43.947245+0000","last_peered":"2024-02-19T15:23:43.947245+0000","last_clean":"2024-02-19T15:23:43.947245+0000","last_became_active":"2024-02-19T15:23:41.956388+0000","last_became_peered":"2024-02-19T15:23:41.956388+0000","last_unstale":"2024-02-19T15:23:43.947245+0000","last_undegraded":"2024-02-19T15:23:43.947245+0000","last_fullsized":"2024-02-19T15:23:43.947245+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:15:40.531978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734808+0000","last_change":"2024-02-19T15:23:42.976054+0000","last_active":"2024-02-19T15:23:44.734808+0000","last_peered":"2024-02-19T15:23:44.734808+0000","last_clean":"2024-02-19T15:23:44.734808+0000","last_became_active":"2024-02-19T15:23:42.975795+0000","last_became_peered":"2024-02-19T15:23:42.975795+0000","last_unstale":"2024-02-19T15:23:44.734808+0000","last_undegraded":"2024-02-19T15:23:44.734808+0000","last_fullsized":"2024-02-19T15:23:44.734808+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:01.020038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945761+0000","last_change":"2024-02-19T15:23:41.962391+0000","last_active":"2024-02-19T15:23:43.945761+0000","last_peered":"2024-02-19T15:23:43.945761+0000","last_clean":"2024-02-19T15:23:43.945761+0000","last_became_active":"2024-02-19T15:23:41.961437+0000","last_became_peered":"2024-02-19T15:23:41.961437+0000","last_unstale":"2024-02-19T15:23:43.945761+0000","last_undegraded":"2024-02-19T15:23:43.945761+0000","last_fullsized":"2024-02-19T15:23:43.945761+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:16:05.431843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946756+0000","last_change":"2024-02-19T15:23:42.964218+0000","last_active":"2024-02-19T15:23:43.946756+0000","last_peered":"2024-02-19T15:23:43.946756+0000","last_clean":"2024-02-19T15:23:43.946756+0000","last_became_active":"2024-02-19T15:23:42.963959+0000","last_became_peered":"2024-02-19T15:23:42.963959+0000","last_unstale":"2024-02-19T15:23:43.946756+0000","last_undegraded":"2024-02-19T15:23:43.946756+0000","last_fullsized":"2024-02-19T15:23:43.946756+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:43:11.719148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735163+0000","last_change":"2024-02-19T15:23:41.960473+0000","last_active":"2024-02-19T15:23:44.735163+0000","last_peered":"2024-02-19T15:23:44.735163+0000","last_clean":"2024-02-19T15:23:44.735163+0000","last_became_active":"2024-02-19T15:23:41.960208+0000","last_became_peered":"2024-02-19T15:23:41.960208+0000","last_unstale":"2024-02-19T15:23:44.735163+0000","last_undegraded":"2024-02-19T15:23:44.735163+0000","last_fullsized":"2024-02-19T15:23:44.735163+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:23:11.863209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":212,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194375+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:23:44.194375+0000","last_peered":"2024-02-19T15:23:44.194375+0000","last_clean":"2024-02-19T15:23:44.194375+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:23:44.194375+0000","last_undegraded":"2024-02-19T15:23:44.194375+0000","last_fullsized":"2024-02-19T15:23:44.194375+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509465,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":487424,"data_stored":193585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":4.6029999999999998,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.79400000000000004}]},{"osd":1,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.625},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.71699999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.73999999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.75600000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.64900000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.81000000000000005}]}]},{"osd":7,"up_from":14,"seq":60129542170,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28124,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343716,"statfs":{"total":96636764160,"available":96607965184,"internally_reserved":0,"allocated":1122304,"data_stored":803385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.627},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.87}]},{"osd":1,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":2.968,"5min":2.968,"15min":2.968},"last":0.754}]},{"osd":2,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":0.71099999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":3.1720000000000002,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":0.77300000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.73699999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.83199999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.93799999999999994}]}]},{"osd":5,"up_from":14,"seq":60129542171,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27536,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344304,"statfs":{"total":96636764160,"available":96608567296,"internally_reserved":0,"allocated":528384,"data_stored":212997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.56999999999999995}]},{"osd":1,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.85499999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.66300000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.63900000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":0.67600000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.89400000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.59899999999999998}]}]},{"osd":4,"up_from":14,"seq":60129542169,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27544,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344296,"statfs":{"total":96636764160,"available":96608559104,"internally_reserved":0,"allocated":536576,"data_stored":214539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.72899999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.623}]},{"osd":2,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.755}]},{"osd":3,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.49199999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.80200000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.78800000000000003}]},{"osd":7,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.82199999999999995}]}]},{"osd":3,"up_from":14,"seq":60129542169,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":536576,"data_stored":213565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.2450000000000001,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.33500000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":4.0220000000000002,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.52900000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.72899999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":4.2039999999999997,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.78400000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":4.1589999999999998,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.627}]},{"osd":6,"last update":"Mon Feb 19 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":4.0940000000000003,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.69499999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":3.9660000000000002,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.67500000000000004}]}]},{"osd":2,"up_from":14,"seq":60129542170,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":540672,"data_stored":214561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.7160000000000002,"5min":4.7160000000000002,"15min":4.7160000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":8.2129999999999992,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.26000000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":8.1620000000000008,"5min":8.1620000000000008,"15min":8.1620000000000008},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.46100000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":4.7679999999999998,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.371},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":4.9909999999999997,"5min":4.9909999999999997,"15min":4.9909999999999997},"last":0.443}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":4.3810000000000002,"5min":4.3810000000000002,"15min":4.3810000000000002},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":4.3440000000000003,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.83199999999999996}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":4.4039999999999999,"5min":4.4039999999999999,"15min":4.4039999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":4.3259999999999996,"5min":4.3259999999999996,"15min":4.3259999999999996},"last":0.75}]},{"osd":6,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":4.5220000000000002,"5min":4.5220000000000002,"15min":4.5220000000000002},"last":0.71699999999999997},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":4.4320000000000004,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.73899999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":4.4829999999999997,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":4.4989999999999997,"5min":4.4989999999999997,"15min":4.4989999999999997},"last":0.84599999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542171,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":528384,"data_stored":212997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.25}]},{"osd":2,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.41599999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.438},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.38300000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.65700000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.746},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.77500000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.72699999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.70499999999999996}]}]},{"osd":0,"up_from":14,"seq":60129542169,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36436,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335404,"statfs":{"total":96636764160,"available":96599453696,"internally_reserved":0,"allocated":1122304,"data_stored":803329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.0199999999999996,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.31}]},{"osd":2,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":5.6669999999999998,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.4649999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.45700000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.9539999999999997,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.345,"5min":1.345,"15min":1.345},"last":0.44800000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":4.7880000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.61799999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.53200000000000003}]},{"osd":6,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":4.8460000000000001,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.65300000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":4.9809999999999999,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":4.915,"5min":4.915,"15min":4.915},"last":0.56200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:23:51.730 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:23:51.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.863+0000 7f996a000640 1 Processor -- start 2024-02-19T15:23:51.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.863+0000 7f996a000640 1 -- start start 2024-02-19T15:23:51.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.867+0000 7f996a000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f99640cf670 0x7f996406aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:51.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.867+0000 7f996a000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f996406b3f0 0x7f996406dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:51.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.867+0000 7f996a000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9964145a80 con 0x7f99641456a0 2024-02-19T15:23:51.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.867+0000 7f996a000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99640d03c0 con 0x7f99640cf670 2024-02-19T15:23:51.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.867+0000 7f996a000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f996406b860 con 0x7f996406b3f0 2024-02-19T15:23:51.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.867+0000 7f995be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f99640cf670 0x7f996406aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:51.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.867+0000 7f995be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f99640cf670 0x7f996406aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33416/0 (socket says 172.21.15.47:33416) 2024-02-19T15:23:51.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.867+0000 7f995be00640 1 -- 172.21.15.47:0/2098252875 learned_addr learned my addr 172.21.15.47:0/2098252875 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:51.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.867+0000 7f9963e00640 1 --2- 172.21.15.47:0/2098252875 >> v2:172.21.15.116:3300/0 conn(0x7f996406b3f0 0x7f996406dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:51.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.867+0000 7f995be00640 1 -- 172.21.15.47:0/2098252875 >> v2:172.21.15.116:3300/0 conn(0x7f996406b3f0 msgr2=0x7f996406dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:51.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.867+0000 7f995be00640 1 --2- 172.21.15.47:0/2098252875 >> v2:172.21.15.116:3300/0 conn(0x7f996406b3f0 0x7f996406dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:51.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.867+0000 7f995be00640 1 -- 172.21.15.47:0/2098252875 >> v1:172.21.15.116:6789/0 conn(0x7f99641456a0 legacy=0x7f99640cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:51.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.867+0000 7f995be00640 1 -- 172.21.15.47:0/2098252875 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99640d1880 con 0x7f99640cf670 2024-02-19T15:23:51.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.871+0000 7f995be00640 1 --2- 172.21.15.47:0/2098252875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99640cf670 0x7f996406aeb0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9964068400 tx=0x7f994c01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40bf73172f3d52a9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:51.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.871+0000 7f9962a00640 1 -- 172.21.15.47:0/2098252875 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994c0ac630 con 0x7f99640cf670 2024-02-19T15:23:51.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.871+0000 7f9962a00640 1 -- 172.21.15.47:0/2098252875 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994c0ac790 con 0x7f99640cf670 2024-02-19T15:23:51.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.871+0000 7f9962a00640 1 -- 172.21.15.47:0/2098252875 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994c0acb20 con 0x7f99640cf670 2024-02-19T15:23:51.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.871+0000 7f996a000640 1 -- 172.21.15.47:0/2098252875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99640cf670 msgr2=0x7f996406aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:51.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.871+0000 7f996a000640 1 --2- 172.21.15.47:0/2098252875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99640cf670 0x7f996406aeb0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9964068400 tx=0x7f994c01c4e0 comp rx=0 tx=0).stop 2024-02-19T15:23:51.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.871+0000 7f996a000640 1 -- 172.21.15.47:0/2098252875 shutdown_connections 2024-02-19T15:23:51.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.871+0000 7f996a000640 1 --2- 172.21.15.47:0/2098252875 >> v2:172.21.15.116:3300/0 conn(0x7f996406b3f0 0x7f996406dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:51.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.871+0000 7f996a000640 1 --2- 172.21.15.47:0/2098252875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99640cf670 0x7f996406aeb0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:51.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.871+0000 7f996a000640 1 -- 172.21.15.47:0/2098252875 >> 172.21.15.47:0/2098252875 conn(0x7f99640d0ad0 msgr2=0x7f99640d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:51.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.871+0000 7f996a000640 1 -- 172.21.15.47:0/2098252875 shutdown_connections 2024-02-19T15:23:51.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.871+0000 7f996a000640 1 -- 172.21.15.47:0/2098252875 wait complete. 2024-02-19T15:23:51.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.875+0000 7f996a000640 1 Processor -- start 2024-02-19T15:23:51.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.875+0000 7f996a000640 1 -- start start 2024-02-19T15:23:51.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.875+0000 7f996a000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f996406b3f0 0x7f99641550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:51.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.875+0000 7f996a000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f99640cf670 0x7f996415e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:51.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.875+0000 7f996a000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99641456a0 0x7f996415e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:51.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.875+0000 7f996a000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9964155ba0 con 0x7f996406b3f0 2024-02-19T15:23:51.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.875+0000 7f996a000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9964156020 con 0x7f99641456a0 2024-02-19T15:23:51.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.875+0000 7f996a000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99640b4bf0 con 0x7f99640cf670 2024-02-19T15:23:51.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.875+0000 7f9963400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f996406b3f0 0x7f99641550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:51.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.875+0000 7f9963400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f996406b3f0 0x7f99641550e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33424/0 (socket says 172.21.15.47:33424) 2024-02-19T15:23:51.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.875+0000 7f9963400640 1 -- 172.21.15.47:0/321734148 learned_addr learned my addr 172.21.15.47:0/321734148 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:51.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.875+0000 7f9963e00640 1 --2- 172.21.15.47:0/321734148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99641456a0 0x7f996415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:51.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.875+0000 7f995be00640 1 --2- 172.21.15.47:0/321734148 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f99640cf670 0x7f996415e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:51.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.875+0000 7f9963400640 1 -- 172.21.15.47:0/321734148 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f99640cf670 msgr2=0x7f996415e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:51.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.875+0000 7f9963400640 1 --2- 172.21.15.47:0/321734148 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f99640cf670 0x7f996415e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:51.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.875+0000 7f9963400640 1 -- 172.21.15.47:0/321734148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99641456a0 msgr2=0x7f996415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:51.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.875+0000 7f9963400640 1 --2- 172.21.15.47:0/321734148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99641456a0 0x7f996415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:51.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.875+0000 7f9963400640 1 -- 172.21.15.47:0/321734148 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c0094d0 con 0x7f996406b3f0 2024-02-19T15:23:51.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.875+0000 7f995be00640 1 --2- 172.21.15.47:0/321734148 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f99640cf670 0x7f996415e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:51.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.875+0000 7f9963e00640 1 --2- 172.21.15.47:0/321734148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99641456a0 0x7f996415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:51.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.879+0000 7f9963400640 1 --2- 172.21.15.47:0/321734148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f996406b3f0 0x7f99641550e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f995400b410 tx=0x7f995400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:51.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.879+0000 7f995b400640 1 -- 172.21.15.47:0/321734148 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9954004020 con 0x7f996406b3f0 2024-02-19T15:23:51.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.879+0000 7f995b400640 1 -- 172.21.15.47:0/321734148 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9954004180 con 0x7f996406b3f0 2024-02-19T15:23:51.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.879+0000 7f995b400640 1 -- 172.21.15.47:0/321734148 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9954011630 con 0x7f996406b3f0 2024-02-19T15:23:51.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.879+0000 7f996a000640 1 -- 172.21.15.47:0/321734148 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99640b4ed0 con 0x7f996406b3f0 2024-02-19T15:23:51.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.879+0000 7f996a000640 1 -- 172.21.15.47:0/321734148 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99640b59b0 con 0x7f996406b3f0 2024-02-19T15:23:51.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.879+0000 7f996a000640 1 -- 172.21.15.47:0/321734148 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99641652f0 con 0x7f996406b3f0 2024-02-19T15:23:51.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.879+0000 7f995b400640 1 -- 172.21.15.47:0/321734148 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9954011790 con 0x7f996406b3f0 2024-02-19T15:23:51.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.879+0000 7f995b400640 1 --2- 172.21.15.47:0/321734148 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9940066e40 0x7f9940069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:51.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.879+0000 7f995b400640 1 -- 172.21.15.47:0/321734148 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9954084790 con 0x7f996406b3f0 2024-02-19T15:23:51.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.879+0000 7f995be00640 1 --2- 172.21.15.47:0/321734148 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9940066e40 0x7f9940069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:51.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.883+0000 7f995be00640 1 --2- 172.21.15.47:0/321734148 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9940066e40 0x7f9940069300 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f994c0ae000 tx=0x7f994c0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:51.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:51.883+0000 7f995b400640 1 -- 172.21.15.47:0/321734148 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f9954051570 con 0x7f996406b3f0 2024-02-19T15:23:52.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.215+0000 7f996a000640 1 -- 172.21.15.47:0/321734148 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f996406d840 con 0x7f9940066e40 2024-02-19T15:23:52.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.223+0000 7f995b400640 1 -- 172.21.15.47:0/321734148 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289325 (secure 0 0 0) 0x7f994c0aca00 con 0x7f9940066e40 2024-02-19T15:23:52.233 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:52.234 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:23:52.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.227+0000 7f996a000640 1 -- 172.21.15.47:0/321734148 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9940066e40 msgr2=0x7f9940069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:52.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.227+0000 7f996a000640 1 --2- 172.21.15.47:0/321734148 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9940066e40 0x7f9940069300 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f994c0ae000 tx=0x7f994c0a7000 comp rx=0 tx=0).stop 2024-02-19T15:23:52.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.227+0000 7f996a000640 1 -- 172.21.15.47:0/321734148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f996406b3f0 msgr2=0x7f99641550e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:52.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.227+0000 7f996a000640 1 --2- 172.21.15.47:0/321734148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f996406b3f0 0x7f99641550e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f995400b410 tx=0x7f995400b8d0 comp rx=0 tx=0).stop 2024-02-19T15:23:52.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.227+0000 7f996a000640 1 -- 172.21.15.47:0/321734148 shutdown_connections 2024-02-19T15:23:52.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.227+0000 7f996a000640 1 --2- 172.21.15.47:0/321734148 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9940066e40 0x7f9940069300 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:52.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.227+0000 7f996a000640 1 --2- 172.21.15.47:0/321734148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99641456a0 0x7f996415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:52.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.227+0000 7f996a000640 1 --2- 172.21.15.47:0/321734148 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f99640cf670 0x7f996415e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:52.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.227+0000 7f996a000640 1 --2- 172.21.15.47:0/321734148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f996406b3f0 0x7f99641550e0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:52.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.227+0000 7f996a000640 1 -- 172.21.15.47:0/321734148 >> 172.21.15.47:0/321734148 conn(0x7f99640d0ad0 msgr2=0x7f996406f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:52.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.227+0000 7f996a000640 1 -- 172.21.15.47:0/321734148 shutdown_connections 2024-02-19T15:23:52.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.227+0000 7f996a000640 1 -- 172.21.15.47:0/321734148 wait complete. 2024-02-19T15:23:52.254 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-02-19T15:23:51.171362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":173,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":246456,"kb_used_data":5320,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754728264,"statfs":{"total":773094113280,"available":772841742336,"internally_reserved":0,"allocated":5447680,"data_stored":2888972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":246795826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.226817"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194769+0000","last_change":"2024-02-19T15:23:42.971601+0000","last_active":"2024-02-19T15:23:44.194769+0000","last_peered":"2024-02-19T15:23:44.194769+0000","last_clean":"2024-02-19T15:23:44.194769+0000","last_became_active":"2024-02-19T15:23:42.971304+0000","last_became_peered":"2024-02-19T15:23:42.971304+0000","last_unstale":"2024-02-19T15:23:44.194769+0000","last_undegraded":"2024-02-19T15:23:44.194769+0000","last_fullsized":"2024-02-19T15:23:44.194769+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:42:40.033469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734972+0000","last_change":"2024-02-19T15:23:42.971058+0000","last_active":"2024-02-19T15:23:44.734972+0000","last_peered":"2024-02-19T15:23:44.734972+0000","last_clean":"2024-02-19T15:23:44.734972+0000","last_became_active":"2024-02-19T15:23:42.970038+0000","last_became_peered":"2024-02-19T15:23:42.970038+0000","last_unstale":"2024-02-19T15:23:44.734972+0000","last_undegraded":"2024-02-19T15:23:44.734972+0000","last_fullsized":"2024-02-19T15:23:44.734972+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:18:23.687527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735368+0000","last_change":"2024-02-19T15:23:42.969350+0000","last_active":"2024-02-19T15:23:44.735368+0000","last_peered":"2024-02-19T15:23:44.735368+0000","last_clean":"2024-02-19T15:23:44.735368+0000","last_became_active":"2024-02-19T15:23:42.969021+0000","last_became_peered":"2024-02-19T15:23:42.969021+0000","last_unstale":"2024-02-19T15:23:44.735368+0000","last_undegraded":"2024-02-19T15:23:44.735368+0000","last_fullsized":"2024-02-19T15:23:44.735368+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:57:21.424216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946334+0000","last_change":"2024-02-19T15:23:42.962243+0000","last_active":"2024-02-19T15:23:43.946334+0000","last_peered":"2024-02-19T15:23:43.946334+0000","last_clean":"2024-02-19T15:23:43.946334+0000","last_became_active":"2024-02-19T15:23:42.961999+0000","last_became_peered":"2024-02-19T15:23:42.961999+0000","last_unstale":"2024-02-19T15:23:43.946334+0000","last_undegraded":"2024-02-19T15:23:43.946334+0000","last_fullsized":"2024-02-19T15:23:43.946334+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:25:42.171965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735126+0000","last_change":"2024-02-19T15:23:42.970850+0000","last_active":"2024-02-19T15:23:44.735126+0000","last_peered":"2024-02-19T15:23:44.735126+0000","last_clean":"2024-02-19T15:23:44.735126+0000","last_became_active":"2024-02-19T15:23:42.969695+0000","last_became_peered":"2024-02-19T15:23:42.969695+0000","last_unstale":"2024-02-19T15:23:44.735126+0000","last_undegraded":"2024-02-19T15:23:44.735126+0000","last_fullsized":"2024-02-19T15:23:44.735126+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:51:57.116389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734445+0000","last_change":"2024-02-19T15:23:42.972292+0000","last_active":"2024-02-19T15:23:44.734445+0000","last_peered":"2024-02-19T15:23:44.734445+0000","last_clean":"2024-02-19T15:23:44.734445+0000","last_became_active":"2024-02-19T15:23:42.971723+0000","last_became_peered":"2024-02-19T15:23:42.971723+0000","last_unstale":"2024-02-19T15:23:44.734445+0000","last_undegraded":"2024-02-19T15:23:44.734445+0000","last_fullsized":"2024-02-19T15:23:44.734445+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:30:47.422419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194498+0000","last_change":"2024-02-19T15:23:42.971748+0000","last_active":"2024-02-19T15:23:44.194498+0000","last_peered":"2024-02-19T15:23:44.194498+0000","last_clean":"2024-02-19T15:23:44.194498+0000","last_became_active":"2024-02-19T15:23:42.971435+0000","last_became_peered":"2024-02-19T15:23:42.971435+0000","last_unstale":"2024-02-19T15:23:44.194498+0000","last_undegraded":"2024-02-19T15:23:44.194498+0000","last_fullsized":"2024-02-19T15:23:44.194498+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:09:04.319698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945616+0000","last_change":"2024-02-19T15:23:42.967933+0000","last_active":"2024-02-19T15:23:43.945616+0000","last_peered":"2024-02-19T15:23:43.945616+0000","last_clean":"2024-02-19T15:23:43.945616+0000","last_became_active":"2024-02-19T15:23:42.967628+0000","last_became_peered":"2024-02-19T15:23:42.967628+0000","last_unstale":"2024-02-19T15:23:43.945616+0000","last_undegraded":"2024-02-19T15:23:43.945616+0000","last_fullsized":"2024-02-19T15:23:43.945616+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:23.119868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946834+0000","last_change":"2024-02-19T15:23:42.968450+0000","last_active":"2024-02-19T15:23:43.946834+0000","last_peered":"2024-02-19T15:23:43.946834+0000","last_clean":"2024-02-19T15:23:43.946834+0000","last_became_active":"2024-02-19T15:23:42.968127+0000","last_became_peered":"2024-02-19T15:23:42.968127+0000","last_unstale":"2024-02-19T15:23:43.946834+0000","last_undegraded":"2024-02-19T15:23:43.946834+0000","last_fullsized":"2024-02-19T15:23:43.946834+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:08:50.431318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946763+0000","last_change":"2024-02-19T15:23:42.967870+0000","last_active":"2024-02-19T15:23:43.946763+0000","last_peered":"2024-02-19T15:23:43.946763+0000","last_clean":"2024-02-19T15:23:43.946763+0000","last_became_active":"2024-02-19T15:23:42.967493+0000","last_became_peered":"2024-02-19T15:23:42.967493+0000","last_unstale":"2024-02-19T15:23:43.946763+0000","last_undegraded":"2024-02-19T15:23:43.946763+0000","last_fullsized":"2024-02-19T15:23:43.946763+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:41:06.816999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946394+0000","last_change":"2024-02-19T15:23:42.981481+0000","last_active":"2024-02-19T15:23:43.946394+0000","last_peered":"2024-02-19T15:23:43.946394+0000","last_clean":"2024-02-19T15:23:43.946394+0000","last_became_active":"2024-02-19T15:23:42.981177+0000","last_became_peered":"2024-02-19T15:23:42.981177+0000","last_unstale":"2024-02-19T15:23:43.946394+0000","last_undegraded":"2024-02-19T15:23:43.946394+0000","last_fullsized":"2024-02-19T15:23:43.946394+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:56:04.086784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734943+0000","last_change":"2024-02-19T15:23:42.970720+0000","last_active":"2024-02-19T15:23:44.734943+0000","last_peered":"2024-02-19T15:23:44.734943+0000","last_clean":"2024-02-19T15:23:44.734943+0000","last_became_active":"2024-02-19T15:23:42.970354+0000","last_became_peered":"2024-02-19T15:23:42.970354+0000","last_unstale":"2024-02-19T15:23:44.734943+0000","last_undegraded":"2024-02-19T15:23:44.734943+0000","last_fullsized":"2024-02-19T15:23:44.734943+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:08:01.896505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061189+0000","last_change":"2024-02-19T15:23:42.981071+0000","last_active":"2024-02-19T15:23:44.061189+0000","last_peered":"2024-02-19T15:23:44.061189+0000","last_clean":"2024-02-19T15:23:44.061189+0000","last_became_active":"2024-02-19T15:23:42.980763+0000","last_became_peered":"2024-02-19T15:23:42.980763+0000","last_unstale":"2024-02-19T15:23:44.061189+0000","last_undegraded":"2024-02-19T15:23:44.061189+0000","last_fullsized":"2024-02-19T15:23:44.061189+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:07:37.152277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946242+0000","last_change":"2024-02-19T15:23:42.979970+0000","last_active":"2024-02-19T15:23:43.946242+0000","last_peered":"2024-02-19T15:23:43.946242+0000","last_clean":"2024-02-19T15:23:43.946242+0000","last_became_active":"2024-02-19T15:23:42.968768+0000","last_became_peered":"2024-02-19T15:23:42.968768+0000","last_unstale":"2024-02-19T15:23:43.946242+0000","last_undegraded":"2024-02-19T15:23:43.946242+0000","last_fullsized":"2024-02-19T15:23:43.946242+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:18:27.465577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734344+0000","last_change":"2024-02-19T15:23:42.974598+0000","last_active":"2024-02-19T15:23:44.734344+0000","last_peered":"2024-02-19T15:23:44.734344+0000","last_clean":"2024-02-19T15:23:44.734344+0000","last_became_active":"2024-02-19T15:23:42.974335+0000","last_became_peered":"2024-02-19T15:23:42.974335+0000","last_unstale":"2024-02-19T15:23:44.734344+0000","last_undegraded":"2024-02-19T15:23:44.734344+0000","last_fullsized":"2024-02-19T15:23:44.734344+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:50:40.431301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947044+0000","last_change":"2024-02-19T15:23:42.969229+0000","last_active":"2024-02-19T15:23:43.947044+0000","last_peered":"2024-02-19T15:23:43.947044+0000","last_clean":"2024-02-19T15:23:43.947044+0000","last_became_active":"2024-02-19T15:23:42.968961+0000","last_became_peered":"2024-02-19T15:23:42.968961+0000","last_unstale":"2024-02-19T15:23:43.947044+0000","last_undegraded":"2024-02-19T15:23:43.947044+0000","last_fullsized":"2024-02-19T15:23:43.947044+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:42.048210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735637+0000","last_change":"2024-02-19T15:23:42.964895+0000","last_active":"2024-02-19T15:23:44.735637+0000","last_peered":"2024-02-19T15:23:44.735637+0000","last_clean":"2024-02-19T15:23:44.735637+0000","last_became_active":"2024-02-19T15:23:42.963832+0000","last_became_peered":"2024-02-19T15:23:42.963832+0000","last_unstale":"2024-02-19T15:23:44.735637+0000","last_undegraded":"2024-02-19T15:23:44.735637+0000","last_fullsized":"2024-02-19T15:23:44.735637+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:01:27.305444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735533+0000","last_change":"2024-02-19T15:23:42.969998+0000","last_active":"2024-02-19T15:23:44.735533+0000","last_peered":"2024-02-19T15:23:44.735533+0000","last_clean":"2024-02-19T15:23:44.735533+0000","last_became_active":"2024-02-19T15:23:42.969741+0000","last_became_peered":"2024-02-19T15:23:42.969741+0000","last_unstale":"2024-02-19T15:23:44.735533+0000","last_undegraded":"2024-02-19T15:23:44.735533+0000","last_fullsized":"2024-02-19T15:23:44.735533+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:18:07.650507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947861+0000","last_change":"2024-02-19T15:23:42.984421+0000","last_active":"2024-02-19T15:23:43.947861+0000","last_peered":"2024-02-19T15:23:43.947861+0000","last_clean":"2024-02-19T15:23:43.947861+0000","last_became_active":"2024-02-19T15:23:42.984138+0000","last_became_peered":"2024-02-19T15:23:42.984138+0000","last_unstale":"2024-02-19T15:23:43.947861+0000","last_undegraded":"2024-02-19T15:23:43.947861+0000","last_fullsized":"2024-02-19T15:23:43.947861+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:09:39.073967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061366+0000","last_change":"2024-02-19T15:23:42.986099+0000","last_active":"2024-02-19T15:23:44.061366+0000","last_peered":"2024-02-19T15:23:44.061366+0000","last_clean":"2024-02-19T15:23:44.061366+0000","last_became_active":"2024-02-19T15:23:42.985685+0000","last_became_peered":"2024-02-19T15:23:42.985685+0000","last_unstale":"2024-02-19T15:23:44.061366+0000","last_undegraded":"2024-02-19T15:23:44.061366+0000","last_fullsized":"2024-02-19T15:23:44.061366+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:13:36.682515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734637+0000","last_change":"2024-02-19T15:23:42.973184+0000","last_active":"2024-02-19T15:23:44.734637+0000","last_peered":"2024-02-19T15:23:44.734637+0000","last_clean":"2024-02-19T15:23:44.734637+0000","last_became_active":"2024-02-19T15:23:42.972228+0000","last_became_peered":"2024-02-19T15:23:42.972228+0000","last_unstale":"2024-02-19T15:23:44.734637+0000","last_undegraded":"2024-02-19T15:23:44.734637+0000","last_fullsized":"2024-02-19T15:23:44.734637+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:03.977088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946364+0000","last_change":"2024-02-19T15:23:42.979749+0000","last_active":"2024-02-19T15:23:43.946364+0000","last_peered":"2024-02-19T15:23:43.946364+0000","last_clean":"2024-02-19T15:23:43.946364+0000","last_became_active":"2024-02-19T15:23:42.968380+0000","last_became_peered":"2024-02-19T15:23:42.968380+0000","last_unstale":"2024-02-19T15:23:43.946364+0000","last_undegraded":"2024-02-19T15:23:43.946364+0000","last_fullsized":"2024-02-19T15:23:43.946364+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:04:27.961177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947399+0000","last_change":"2024-02-19T15:23:42.961367+0000","last_active":"2024-02-19T15:23:43.947399+0000","last_peered":"2024-02-19T15:23:43.947399+0000","last_clean":"2024-02-19T15:23:43.947399+0000","last_became_active":"2024-02-19T15:23:42.960768+0000","last_became_peered":"2024-02-19T15:23:42.960768+0000","last_unstale":"2024-02-19T15:23:43.947399+0000","last_undegraded":"2024-02-19T15:23:43.947399+0000","last_fullsized":"2024-02-19T15:23:43.947399+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:31:14.899563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947059+0000","last_change":"2024-02-19T15:23:42.964280+0000","last_active":"2024-02-19T15:23:43.947059+0000","last_peered":"2024-02-19T15:23:43.947059+0000","last_clean":"2024-02-19T15:23:43.947059+0000","last_became_active":"2024-02-19T15:23:42.964011+0000","last_became_peered":"2024-02-19T15:23:42.964011+0000","last_unstale":"2024-02-19T15:23:43.947059+0000","last_undegraded":"2024-02-19T15:23:43.947059+0000","last_fullsized":"2024-02-19T15:23:43.947059+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:12:36.706598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947405+0000","last_change":"2024-02-19T15:23:42.968507+0000","last_active":"2024-02-19T15:23:43.947405+0000","last_peered":"2024-02-19T15:23:43.947405+0000","last_clean":"2024-02-19T15:23:43.947405+0000","last_became_active":"2024-02-19T15:23:42.967794+0000","last_became_peered":"2024-02-19T15:23:42.967794+0000","last_unstale":"2024-02-19T15:23:43.947405+0000","last_undegraded":"2024-02-19T15:23:43.947405+0000","last_fullsized":"2024-02-19T15:23:43.947405+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:41:27.771626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061361+0000","last_change":"2024-02-19T15:23:42.980908+0000","last_active":"2024-02-19T15:23:44.061361+0000","last_peered":"2024-02-19T15:23:44.061361+0000","last_clean":"2024-02-19T15:23:44.061361+0000","last_became_active":"2024-02-19T15:23:42.980592+0000","last_became_peered":"2024-02-19T15:23:42.980592+0000","last_unstale":"2024-02-19T15:23:44.061361+0000","last_undegraded":"2024-02-19T15:23:44.061361+0000","last_fullsized":"2024-02-19T15:23:44.061361+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:39:51.560827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735567+0000","last_change":"2024-02-19T15:23:42.961547+0000","last_active":"2024-02-19T15:23:44.735567+0000","last_peered":"2024-02-19T15:23:44.735567+0000","last_clean":"2024-02-19T15:23:44.735567+0000","last_became_active":"2024-02-19T15:23:42.961215+0000","last_became_peered":"2024-02-19T15:23:42.961215+0000","last_unstale":"2024-02-19T15:23:44.735567+0000","last_undegraded":"2024-02-19T15:23:44.735567+0000","last_fullsized":"2024-02-19T15:23:44.735567+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:23:12.865791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735161+0000","last_change":"2024-02-19T15:23:42.971532+0000","last_active":"2024-02-19T15:23:44.735161+0000","last_peered":"2024-02-19T15:23:44.735161+0000","last_clean":"2024-02-19T15:23:44.735161+0000","last_became_active":"2024-02-19T15:23:42.971181+0000","last_became_peered":"2024-02-19T15:23:42.971181+0000","last_unstale":"2024-02-19T15:23:44.735161+0000","last_undegraded":"2024-02-19T15:23:44.735161+0000","last_fullsized":"2024-02-19T15:23:44.735161+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:34:33.315961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947830+0000","last_change":"2024-02-19T15:23:42.966285+0000","last_active":"2024-02-19T15:23:43.947830+0000","last_peered":"2024-02-19T15:23:43.947830+0000","last_clean":"2024-02-19T15:23:43.947830+0000","last_became_active":"2024-02-19T15:23:42.964131+0000","last_became_peered":"2024-02-19T15:23:42.964131+0000","last_unstale":"2024-02-19T15:23:43.947830+0000","last_undegraded":"2024-02-19T15:23:43.947830+0000","last_fullsized":"2024-02-19T15:23:43.947830+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:52.966296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734745+0000","last_change":"2024-02-19T15:23:42.961009+0000","last_active":"2024-02-19T15:23:44.734745+0000","last_peered":"2024-02-19T15:23:44.734745+0000","last_clean":"2024-02-19T15:23:44.734745+0000","last_became_active":"2024-02-19T15:23:42.960723+0000","last_became_peered":"2024-02-19T15:23:42.960723+0000","last_unstale":"2024-02-19T15:23:44.734745+0000","last_undegraded":"2024-02-19T15:23:44.734745+0000","last_fullsized":"2024-02-19T15:23:44.734745+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:57:53.597108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734459+0000","last_change":"2024-02-19T15:23:42.973247+0000","last_active":"2024-02-19T15:23:44.734459+0000","last_peered":"2024-02-19T15:23:44.734459+0000","last_clean":"2024-02-19T15:23:44.734459+0000","last_became_active":"2024-02-19T15:23:42.972957+0000","last_became_peered":"2024-02-19T15:23:42.972957+0000","last_unstale":"2024-02-19T15:23:44.734459+0000","last_undegraded":"2024-02-19T15:23:44.734459+0000","last_fullsized":"2024-02-19T15:23:44.734459+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:36:54.676262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061144+0000","last_change":"2024-02-19T15:23:42.962393+0000","last_active":"2024-02-19T15:23:44.061144+0000","last_peered":"2024-02-19T15:23:44.061144+0000","last_clean":"2024-02-19T15:23:44.061144+0000","last_became_active":"2024-02-19T15:23:42.962112+0000","last_became_peered":"2024-02-19T15:23:42.962112+0000","last_unstale":"2024-02-19T15:23:44.061144+0000","last_undegraded":"2024-02-19T15:23:44.061144+0000","last_fullsized":"2024-02-19T15:23:44.061144+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:28:48.394306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947613+0000","last_change":"2024-02-19T15:23:42.974733+0000","last_active":"2024-02-19T15:23:43.947613+0000","last_peered":"2024-02-19T15:23:43.947613+0000","last_clean":"2024-02-19T15:23:43.947613+0000","last_became_active":"2024-02-19T15:23:42.974439+0000","last_became_peered":"2024-02-19T15:23:42.974439+0000","last_unstale":"2024-02-19T15:23:43.947613+0000","last_undegraded":"2024-02-19T15:23:43.947613+0000","last_fullsized":"2024-02-19T15:23:43.947613+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:39:12.299839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734807+0000","last_change":"2024-02-19T15:23:41.960560+0000","last_active":"2024-02-19T15:23:44.734807+0000","last_peered":"2024-02-19T15:23:44.734807+0000","last_clean":"2024-02-19T15:23:44.734807+0000","last_became_active":"2024-02-19T15:23:41.960237+0000","last_became_peered":"2024-02-19T15:23:41.960237+0000","last_unstale":"2024-02-19T15:23:44.734807+0000","last_undegraded":"2024-02-19T15:23:44.734807+0000","last_fullsized":"2024-02-19T15:23:44.734807+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:14:27.569498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947650+0000","last_change":"2024-02-19T15:23:42.968652+0000","last_active":"2024-02-19T15:23:43.947650+0000","last_peered":"2024-02-19T15:23:43.947650+0000","last_clean":"2024-02-19T15:23:43.947650+0000","last_became_active":"2024-02-19T15:23:42.968388+0000","last_became_peered":"2024-02-19T15:23:42.968388+0000","last_unstale":"2024-02-19T15:23:43.947650+0000","last_undegraded":"2024-02-19T15:23:43.947650+0000","last_fullsized":"2024-02-19T15:23:43.947650+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:15:02.280830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735605+0000","last_change":"2024-02-19T15:23:41.960962+0000","last_active":"2024-02-19T15:23:44.735605+0000","last_peered":"2024-02-19T15:23:44.735605+0000","last_clean":"2024-02-19T15:23:44.735605+0000","last_became_active":"2024-02-19T15:23:41.959425+0000","last_became_peered":"2024-02-19T15:23:41.959425+0000","last_unstale":"2024-02-19T15:23:44.735605+0000","last_undegraded":"2024-02-19T15:23:44.735605+0000","last_fullsized":"2024-02-19T15:23:44.735605+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:54.136668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735315+0000","last_change":"2024-02-19T15:23:42.972962+0000","last_active":"2024-02-19T15:23:44.735315+0000","last_peered":"2024-02-19T15:23:44.735315+0000","last_clean":"2024-02-19T15:23:44.735315+0000","last_became_active":"2024-02-19T15:23:42.972625+0000","last_became_peered":"2024-02-19T15:23:42.972625+0000","last_unstale":"2024-02-19T15:23:44.735315+0000","last_undegraded":"2024-02-19T15:23:44.735315+0000","last_fullsized":"2024-02-19T15:23:44.735315+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:40:40.055788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947602+0000","last_change":"2024-02-19T15:23:41.948520+0000","last_active":"2024-02-19T15:23:43.947602+0000","last_peered":"2024-02-19T15:23:43.947602+0000","last_clean":"2024-02-19T15:23:43.947602+0000","last_became_active":"2024-02-19T15:23:41.948233+0000","last_became_peered":"2024-02-19T15:23:41.948233+0000","last_unstale":"2024-02-19T15:23:43.947602+0000","last_undegraded":"2024-02-19T15:23:43.947602+0000","last_fullsized":"2024-02-19T15:23:43.947602+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:50:30.218103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947636+0000","last_change":"2024-02-19T15:23:42.961975+0000","last_active":"2024-02-19T15:23:43.947636+0000","last_peered":"2024-02-19T15:23:43.947636+0000","last_clean":"2024-02-19T15:23:43.947636+0000","last_became_active":"2024-02-19T15:23:42.961615+0000","last_became_peered":"2024-02-19T15:23:42.961615+0000","last_unstale":"2024-02-19T15:23:43.947636+0000","last_undegraded":"2024-02-19T15:23:43.947636+0000","last_fullsized":"2024-02-19T15:23:43.947636+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:57:21.424216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194190+0000","last_change":"2024-02-19T15:23:41.961942+0000","last_active":"2024-02-19T15:23:44.194190+0000","last_peered":"2024-02-19T15:23:44.194190+0000","last_clean":"2024-02-19T15:23:44.194190+0000","last_became_active":"2024-02-19T15:23:41.961659+0000","last_became_peered":"2024-02-19T15:23:41.961659+0000","last_unstale":"2024-02-19T15:23:44.194190+0000","last_undegraded":"2024-02-19T15:23:44.194190+0000","last_fullsized":"2024-02-19T15:23:44.194190+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:14:27.569498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734930+0000","last_change":"2024-02-19T15:23:42.961945+0000","last_active":"2024-02-19T15:23:44.734930+0000","last_peered":"2024-02-19T15:23:44.734930+0000","last_clean":"2024-02-19T15:23:44.734930+0000","last_became_active":"2024-02-19T15:23:42.961269+0000","last_became_peered":"2024-02-19T15:23:42.961269+0000","last_unstale":"2024-02-19T15:23:44.734930+0000","last_undegraded":"2024-02-19T15:23:44.734930+0000","last_fullsized":"2024-02-19T15:23:44.734930+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:39:12.299839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734672+0000","last_change":"2024-02-19T15:23:41.966788+0000","last_active":"2024-02-19T15:23:44.734672+0000","last_peered":"2024-02-19T15:23:44.734672+0000","last_clean":"2024-02-19T15:23:44.734672+0000","last_became_active":"2024-02-19T15:23:41.966346+0000","last_became_peered":"2024-02-19T15:23:41.966346+0000","last_unstale":"2024-02-19T15:23:44.734672+0000","last_undegraded":"2024-02-19T15:23:44.734672+0000","last_fullsized":"2024-02-19T15:23:44.734672+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:01:26.302862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946392+0000","last_change":"2024-02-19T15:23:42.968247+0000","last_active":"2024-02-19T15:23:43.946392+0000","last_peered":"2024-02-19T15:23:43.946392+0000","last_clean":"2024-02-19T15:23:43.946392+0000","last_became_active":"2024-02-19T15:23:42.967240+0000","last_became_peered":"2024-02-19T15:23:42.967240+0000","last_unstale":"2024-02-19T15:23:43.946392+0000","last_undegraded":"2024-02-19T15:23:43.946392+0000","last_fullsized":"2024-02-19T15:23:43.946392+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:41:08.031616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735141+0000","last_change":"2024-02-19T15:23:41.955181+0000","last_active":"2024-02-19T15:23:44.735141+0000","last_peered":"2024-02-19T15:23:44.735141+0000","last_clean":"2024-02-19T15:23:44.735141+0000","last_became_active":"2024-02-19T15:23:41.954904+0000","last_became_peered":"2024-02-19T15:23:41.954904+0000","last_unstale":"2024-02-19T15:23:44.735141+0000","last_undegraded":"2024-02-19T15:23:44.735141+0000","last_fullsized":"2024-02-19T15:23:44.735141+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:13:35.679933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945686+0000","last_change":"2024-02-19T15:23:42.980803+0000","last_active":"2024-02-19T15:23:43.945686+0000","last_peered":"2024-02-19T15:23:43.945686+0000","last_clean":"2024-02-19T15:23:43.945686+0000","last_became_active":"2024-02-19T15:23:42.980507+0000","last_became_peered":"2024-02-19T15:23:42.980507+0000","last_unstale":"2024-02-19T15:23:43.945686+0000","last_undegraded":"2024-02-19T15:23:43.945686+0000","last_fullsized":"2024-02-19T15:23:43.945686+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:08:33.495333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734933+0000","last_change":"2024-02-19T15:23:41.951702+0000","last_active":"2024-02-19T15:23:44.734933+0000","last_peered":"2024-02-19T15:23:44.734933+0000","last_clean":"2024-02-19T15:23:44.734933+0000","last_became_active":"2024-02-19T15:23:41.951409+0000","last_became_peered":"2024-02-19T15:23:41.951409+0000","last_unstale":"2024-02-19T15:23:44.734933+0000","last_undegraded":"2024-02-19T15:23:44.734933+0000","last_fullsized":"2024-02-19T15:23:44.734933+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:57:20.421633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947109+0000","last_change":"2024-02-19T15:23:42.961588+0000","last_active":"2024-02-19T15:23:43.947109+0000","last_peered":"2024-02-19T15:23:43.947109+0000","last_clean":"2024-02-19T15:23:43.947109+0000","last_became_active":"2024-02-19T15:23:42.961106+0000","last_became_peered":"2024-02-19T15:23:42.961106+0000","last_unstale":"2024-02-19T15:23:43.947109+0000","last_undegraded":"2024-02-19T15:23:43.947109+0000","last_fullsized":"2024-02-19T15:23:43.947109+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:48:13.256599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945665+0000","last_change":"2024-02-19T15:23:41.965224+0000","last_active":"2024-02-19T15:23:43.945665+0000","last_peered":"2024-02-19T15:23:43.945665+0000","last_clean":"2024-02-19T15:23:43.945665+0000","last_became_active":"2024-02-19T15:23:41.964654+0000","last_became_peered":"2024-02-19T15:23:41.964654+0000","last_unstale":"2024-02-19T15:23:43.945665+0000","last_undegraded":"2024-02-19T15:23:43.945665+0000","last_fullsized":"2024-02-19T15:23:43.945665+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:49:19.641960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735665+0000","last_change":"2024-02-19T15:23:42.969995+0000","last_active":"2024-02-19T15:23:44.735665+0000","last_peered":"2024-02-19T15:23:44.735665+0000","last_clean":"2024-02-19T15:23:44.735665+0000","last_became_active":"2024-02-19T15:23:42.969651+0000","last_became_peered":"2024-02-19T15:23:42.969651+0000","last_unstale":"2024-02-19T15:23:44.735665+0000","last_undegraded":"2024-02-19T15:23:44.735665+0000","last_fullsized":"2024-02-19T15:23:44.735665+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:45.239277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735692+0000","last_change":"2024-02-19T15:23:41.961017+0000","last_active":"2024-02-19T15:23:44.735692+0000","last_peered":"2024-02-19T15:23:44.735692+0000","last_clean":"2024-02-19T15:23:44.735692+0000","last_became_active":"2024-02-19T15:23:41.960715+0000","last_became_peered":"2024-02-19T15:23:41.960715+0000","last_unstale":"2024-02-19T15:23:44.735692+0000","last_undegraded":"2024-02-19T15:23:44.735692+0000","last_fullsized":"2024-02-19T15:23:44.735692+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:05.009739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194763+0000","last_change":"2024-02-19T15:23:42.966631+0000","last_active":"2024-02-19T15:23:44.194763+0000","last_peered":"2024-02-19T15:23:44.194763+0000","last_clean":"2024-02-19T15:23:44.194763+0000","last_became_active":"2024-02-19T15:23:42.965314+0000","last_became_peered":"2024-02-19T15:23:42.965314+0000","last_unstale":"2024-02-19T15:23:44.194763+0000","last_undegraded":"2024-02-19T15:23:44.194763+0000","last_fullsized":"2024-02-19T15:23:44.194763+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:36:54.676262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946805+0000","last_change":"2024-02-19T15:23:41.958566+0000","last_active":"2024-02-19T15:23:43.946805+0000","last_peered":"2024-02-19T15:23:43.946805+0000","last_clean":"2024-02-19T15:23:43.946805+0000","last_became_active":"2024-02-19T15:23:41.958244+0000","last_became_peered":"2024-02-19T15:23:41.958244+0000","last_unstale":"2024-02-19T15:23:43.946805+0000","last_undegraded":"2024-02-19T15:23:43.946805+0000","last_fullsized":"2024-02-19T15:23:43.946805+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:20:44.307845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734660+0000","last_change":"2024-02-19T15:23:42.972332+0000","last_active":"2024-02-19T15:23:44.734660+0000","last_peered":"2024-02-19T15:23:44.734660+0000","last_clean":"2024-02-19T15:23:44.734660+0000","last_became_active":"2024-02-19T15:23:42.971907+0000","last_became_peered":"2024-02-19T15:23:42.971907+0000","last_unstale":"2024-02-19T15:23:44.734660+0000","last_undegraded":"2024-02-19T15:23:44.734660+0000","last_fullsized":"2024-02-19T15:23:44.734660+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:59:43.637759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194426+0000","last_change":"2024-02-19T15:23:41.964052+0000","last_active":"2024-02-19T15:23:44.194426+0000","last_peered":"2024-02-19T15:23:44.194426+0000","last_clean":"2024-02-19T15:23:44.194426+0000","last_became_active":"2024-02-19T15:23:41.963729+0000","last_became_peered":"2024-02-19T15:23:41.963729+0000","last_unstale":"2024-02-19T15:23:44.194426+0000","last_undegraded":"2024-02-19T15:23:44.194426+0000","last_fullsized":"2024-02-19T15:23:44.194426+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:02:12.172289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194355+0000","last_change":"2024-02-19T15:23:42.969755+0000","last_active":"2024-02-19T15:23:44.194355+0000","last_peered":"2024-02-19T15:23:44.194355+0000","last_clean":"2024-02-19T15:23:44.194355+0000","last_became_active":"2024-02-19T15:23:42.969477+0000","last_became_peered":"2024-02-19T15:23:42.969477+0000","last_unstale":"2024-02-19T15:23:44.194355+0000","last_undegraded":"2024-02-19T15:23:44.194355+0000","last_fullsized":"2024-02-19T15:23:44.194355+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:47:09.498437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":20,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061276+0000","last_change":"2024-02-19T15:23:41.959735+0000","last_active":"2024-02-19T15:23:44.061276+0000","last_peered":"2024-02-19T15:23:44.061276+0000","last_clean":"2024-02-19T15:23:44.061276+0000","last_became_active":"2024-02-19T15:23:41.959171+0000","last_became_peered":"2024-02-19T15:23:41.959171+0000","last_unstale":"2024-02-19T15:23:44.061276+0000","last_undegraded":"2024-02-19T15:23:44.061276+0000","last_fullsized":"2024-02-19T15:23:44.061276+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:07:41.495783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947482+0000","last_change":"2024-02-19T15:23:42.968052+0000","last_active":"2024-02-19T15:23:43.947482+0000","last_peered":"2024-02-19T15:23:43.947482+0000","last_clean":"2024-02-19T15:23:43.947482+0000","last_became_active":"2024-02-19T15:23:42.967726+0000","last_became_peered":"2024-02-19T15:23:42.967726+0000","last_unstale":"2024-02-19T15:23:43.947482+0000","last_undegraded":"2024-02-19T15:23:43.947482+0000","last_fullsized":"2024-02-19T15:23:43.947482+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:12:42.805608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734694+0000","last_change":"2024-02-19T15:23:41.964586+0000","last_active":"2024-02-19T15:23:44.734694+0000","last_peered":"2024-02-19T15:23:44.734694+0000","last_clean":"2024-02-19T15:23:44.734694+0000","last_became_active":"2024-02-19T15:23:41.964176+0000","last_became_peered":"2024-02-19T15:23:41.964176+0000","last_unstale":"2024-02-19T15:23:44.734694+0000","last_undegraded":"2024-02-19T15:23:44.734694+0000","last_fullsized":"2024-02-19T15:23:44.734694+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:44.236695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946493+0000","last_change":"2024-02-19T15:23:42.965254+0000","last_active":"2024-02-19T15:23:43.946493+0000","last_peered":"2024-02-19T15:23:43.946493+0000","last_clean":"2024-02-19T15:23:43.946493+0000","last_became_active":"2024-02-19T15:23:42.964990+0000","last_became_peered":"2024-02-19T15:23:42.964990+0000","last_unstale":"2024-02-19T15:23:43.946493+0000","last_undegraded":"2024-02-19T15:23:43.946493+0000","last_fullsized":"2024-02-19T15:23:43.946493+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:37:19.468760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194207+0000","last_change":"2024-02-19T15:23:41.965760+0000","last_active":"2024-02-19T15:23:44.194207+0000","last_peered":"2024-02-19T15:23:44.194207+0000","last_clean":"2024-02-19T15:23:44.194207+0000","last_became_active":"2024-02-19T15:23:41.965423+0000","last_became_peered":"2024-02-19T15:23:41.965423+0000","last_unstale":"2024-02-19T15:23:44.194207+0000","last_undegraded":"2024-02-19T15:23:44.194207+0000","last_fullsized":"2024-02-19T15:23:44.194207+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:01:16.303652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734331+0000","last_change":"2024-02-19T15:23:42.970653+0000","last_active":"2024-02-19T15:23:44.734331+0000","last_peered":"2024-02-19T15:23:44.734331+0000","last_clean":"2024-02-19T15:23:44.734331+0000","last_became_active":"2024-02-19T15:23:42.970394+0000","last_became_peered":"2024-02-19T15:23:42.970394+0000","last_unstale":"2024-02-19T15:23:44.734331+0000","last_undegraded":"2024-02-19T15:23:44.734331+0000","last_fullsized":"2024-02-19T15:23:44.734331+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:09:04.319698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734401+0000","last_change":"2024-02-19T15:23:41.964786+0000","last_active":"2024-02-19T15:23:44.734401+0000","last_peered":"2024-02-19T15:23:44.734401+0000","last_clean":"2024-02-19T15:23:44.734401+0000","last_became_active":"2024-02-19T15:23:41.964085+0000","last_became_peered":"2024-02-19T15:23:41.964085+0000","last_unstale":"2024-02-19T15:23:44.734401+0000","last_undegraded":"2024-02-19T15:23:44.734401+0000","last_fullsized":"2024-02-19T15:23:44.734401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:01:16.303652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194624+0000","last_change":"2024-02-19T15:23:42.974233+0000","last_active":"2024-02-19T15:23:44.194624+0000","last_peered":"2024-02-19T15:23:44.194624+0000","last_clean":"2024-02-19T15:23:44.194624+0000","last_became_active":"2024-02-19T15:23:42.973975+0000","last_became_peered":"2024-02-19T15:23:42.973975+0000","last_unstale":"2024-02-19T15:23:44.194624+0000","last_undegraded":"2024-02-19T15:23:44.194624+0000","last_fullsized":"2024-02-19T15:23:44.194624+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:30:47.422419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":22,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945742+0000","last_change":"2024-02-19T15:23:41.957848+0000","last_active":"2024-02-19T15:23:43.945742+0000","last_peered":"2024-02-19T15:23:43.945742+0000","last_clean":"2024-02-19T15:23:43.945742+0000","last_became_active":"2024-02-19T15:23:41.957529+0000","last_became_peered":"2024-02-19T15:23:41.957529+0000","last_unstale":"2024-02-19T15:23:43.945742+0000","last_undegraded":"2024-02-19T15:23:43.945742+0000","last_fullsized":"2024-02-19T15:23:43.945742+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:41:14.948978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061427+0000","last_change":"2024-02-19T15:23:42.971155+0000","last_active":"2024-02-19T15:23:44.061427+0000","last_peered":"2024-02-19T15:23:44.061427+0000","last_clean":"2024-02-19T15:23:44.061427+0000","last_became_active":"2024-02-19T15:23:42.970879+0000","last_became_peered":"2024-02-19T15:23:42.970879+0000","last_unstale":"2024-02-19T15:23:44.061427+0000","last_undegraded":"2024-02-19T15:23:44.061427+0000","last_fullsized":"2024-02-19T15:23:44.061427+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:25:52.362277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061334+0000","last_change":"2024-02-19T15:23:41.966015+0000","last_active":"2024-02-19T15:23:44.061334+0000","last_peered":"2024-02-19T15:23:44.061334+0000","last_clean":"2024-02-19T15:23:44.061334+0000","last_became_active":"2024-02-19T15:23:41.965678+0000","last_became_peered":"2024-02-19T15:23:41.965678+0000","last_unstale":"2024-02-19T15:23:44.061334+0000","last_undegraded":"2024-02-19T15:23:44.061334+0000","last_fullsized":"2024-02-19T15:23:44.061334+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:27:46.811039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946825+0000","last_change":"2024-02-19T15:23:42.983074+0000","last_active":"2024-02-19T15:23:43.946825+0000","last_peered":"2024-02-19T15:23:43.946825+0000","last_clean":"2024-02-19T15:23:43.946825+0000","last_became_active":"2024-02-19T15:23:42.982771+0000","last_became_peered":"2024-02-19T15:23:42.982771+0000","last_unstale":"2024-02-19T15:23:43.946825+0000","last_undegraded":"2024-02-19T15:23:43.946825+0000","last_fullsized":"2024-02-19T15:23:43.946825+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:04:26.404860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735691+0000","last_change":"2024-02-19T15:23:41.963486+0000","last_active":"2024-02-19T15:23:44.735691+0000","last_peered":"2024-02-19T15:23:44.735691+0000","last_clean":"2024-02-19T15:23:44.735691+0000","last_became_active":"2024-02-19T15:23:41.962837+0000","last_became_peered":"2024-02-19T15:23:41.962837+0000","last_unstale":"2024-02-19T15:23:44.735691+0000","last_undegraded":"2024-02-19T15:23:44.735691+0000","last_fullsized":"2024-02-19T15:23:44.735691+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:16:42.881994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947502+0000","last_change":"2024-02-19T15:23:42.973864+0000","last_active":"2024-02-19T15:23:43.947502+0000","last_peered":"2024-02-19T15:23:43.947502+0000","last_clean":"2024-02-19T15:23:43.947502+0000","last_became_active":"2024-02-19T15:23:42.973453+0000","last_became_peered":"2024-02-19T15:23:42.973453+0000","last_unstale":"2024-02-19T15:23:43.947502+0000","last_undegraded":"2024-02-19T15:23:43.947502+0000","last_fullsized":"2024-02-19T15:23:43.947502+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:07:52.527478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735151+0000","last_change":"2024-02-19T15:23:41.952606+0000","last_active":"2024-02-19T15:23:44.735151+0000","last_peered":"2024-02-19T15:23:44.735151+0000","last_clean":"2024-02-19T15:23:44.735151+0000","last_became_active":"2024-02-19T15:23:41.952291+0000","last_became_peered":"2024-02-19T15:23:41.952291+0000","last_unstale":"2024-02-19T15:23:44.735151+0000","last_undegraded":"2024-02-19T15:23:44.735151+0000","last_fullsized":"2024-02-19T15:23:44.735151+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:01:26.302862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947650+0000","last_change":"2024-02-19T15:23:42.966136+0000","last_active":"2024-02-19T15:23:43.947650+0000","last_peered":"2024-02-19T15:23:43.947650+0000","last_clean":"2024-02-19T15:23:43.947650+0000","last_became_active":"2024-02-19T15:23:42.964485+0000","last_became_peered":"2024-02-19T15:23:42.964485+0000","last_unstale":"2024-02-19T15:23:43.947650+0000","last_undegraded":"2024-02-19T15:23:43.947650+0000","last_fullsized":"2024-02-19T15:23:43.947650+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:08:45.814754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734712+0000","last_change":"2024-02-19T15:23:41.962283+0000","last_active":"2024-02-19T15:23:44.734712+0000","last_peered":"2024-02-19T15:23:44.734712+0000","last_clean":"2024-02-19T15:23:44.734712+0000","last_became_active":"2024-02-19T15:23:41.961397+0000","last_became_peered":"2024-02-19T15:23:41.961397+0000","last_unstale":"2024-02-19T15:23:44.734712+0000","last_undegraded":"2024-02-19T15:23:44.734712+0000","last_fullsized":"2024-02-19T15:23:44.734712+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:45:37.841170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734509+0000","last_change":"2024-02-19T15:23:42.972039+0000","last_active":"2024-02-19T15:23:44.734509+0000","last_peered":"2024-02-19T15:23:44.734509+0000","last_clean":"2024-02-19T15:23:44.734509+0000","last_became_active":"2024-02-19T15:23:42.971311+0000","last_became_peered":"2024-02-19T15:23:42.971311+0000","last_unstale":"2024-02-19T15:23:44.734509+0000","last_undegraded":"2024-02-19T15:23:44.734509+0000","last_fullsized":"2024-02-19T15:23:44.734509+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:42:40.033469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734827+0000","last_change":"2024-02-19T15:23:41.960894+0000","last_active":"2024-02-19T15:23:44.734827+0000","last_peered":"2024-02-19T15:23:44.734827+0000","last_clean":"2024-02-19T15:23:44.734827+0000","last_became_active":"2024-02-19T15:23:41.959201+0000","last_became_peered":"2024-02-19T15:23:41.959201+0000","last_unstale":"2024-02-19T15:23:44.734827+0000","last_undegraded":"2024-02-19T15:23:44.734827+0000","last_fullsized":"2024-02-19T15:23:44.734827+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:32:59.830263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947496+0000","last_change":"2024-02-19T15:23:42.958733+0000","last_active":"2024-02-19T15:23:43.947496+0000","last_peered":"2024-02-19T15:23:43.947496+0000","last_clean":"2024-02-19T15:23:43.947496+0000","last_became_active":"2024-02-19T15:23:42.958345+0000","last_became_peered":"2024-02-19T15:23:42.958345+0000","last_unstale":"2024-02-19T15:23:43.947496+0000","last_undegraded":"2024-02-19T15:23:43.947496+0000","last_fullsized":"2024-02-19T15:23:43.947496+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:01.020038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734643+0000","last_change":"2024-02-19T15:23:41.961678+0000","last_active":"2024-02-19T15:23:44.734643+0000","last_peered":"2024-02-19T15:23:44.734643+0000","last_clean":"2024-02-19T15:23:44.734643+0000","last_became_active":"2024-02-19T15:23:41.960130+0000","last_became_peered":"2024-02-19T15:23:41.960130+0000","last_unstale":"2024-02-19T15:23:44.734643+0000","last_undegraded":"2024-02-19T15:23:44.734643+0000","last_fullsized":"2024-02-19T15:23:44.734643+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:18:06.647924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.061083+0000","last_change":"2024-02-19T15:23:42.965223+0000","last_active":"2024-02-19T15:23:44.061083+0000","last_peered":"2024-02-19T15:23:44.061083+0000","last_clean":"2024-02-19T15:23:44.061083+0000","last_became_active":"2024-02-19T15:23:42.964860+0000","last_became_peered":"2024-02-19T15:23:42.964860+0000","last_unstale":"2024-02-19T15:23:44.061083+0000","last_undegraded":"2024-02-19T15:23:44.061083+0000","last_fullsized":"2024-02-19T15:23:44.061083+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:52.445068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945910+0000","last_change":"2024-02-19T15:23:41.966499+0000","last_active":"2024-02-19T15:23:43.945910+0000","last_peered":"2024-02-19T15:23:43.945910+0000","last_clean":"2024-02-19T15:23:43.945910+0000","last_became_active":"2024-02-19T15:23:41.966206+0000","last_became_peered":"2024-02-19T15:23:41.966206+0000","last_unstale":"2024-02-19T15:23:43.945910+0000","last_undegraded":"2024-02-19T15:23:43.945910+0000","last_fullsized":"2024-02-19T15:23:43.945910+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:50:47.865930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945811+0000","last_change":"2024-02-19T15:23:42.982135+0000","last_active":"2024-02-19T15:23:43.945811+0000","last_peered":"2024-02-19T15:23:43.945811+0000","last_clean":"2024-02-19T15:23:43.945811+0000","last_became_active":"2024-02-19T15:23:42.981890+0000","last_became_peered":"2024-02-19T15:23:42.981890+0000","last_unstale":"2024-02-19T15:23:43.945811+0000","last_undegraded":"2024-02-19T15:23:43.945811+0000","last_fullsized":"2024-02-19T15:23:43.945811+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:28.558554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194347+0000","last_change":"2024-02-19T15:23:41.964986+0000","last_active":"2024-02-19T15:23:44.194347+0000","last_peered":"2024-02-19T15:23:44.194347+0000","last_clean":"2024-02-19T15:23:44.194347+0000","last_became_active":"2024-02-19T15:23:41.964713+0000","last_became_peered":"2024-02-19T15:23:41.964713+0000","last_unstale":"2024-02-19T15:23:44.194347+0000","last_undegraded":"2024-02-19T15:23:44.194347+0000","last_fullsized":"2024-02-19T15:23:44.194347+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:32:59.830263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947741+0000","last_change":"2024-02-19T15:23:42.984392+0000","last_active":"2024-02-19T15:23:43.947741+0000","last_peered":"2024-02-19T15:23:43.947741+0000","last_clean":"2024-02-19T15:23:43.947741+0000","last_became_active":"2024-02-19T15:23:42.984111+0000","last_became_peered":"2024-02-19T15:23:42.984111+0000","last_unstale":"2024-02-19T15:23:43.947741+0000","last_undegraded":"2024-02-19T15:23:43.947741+0000","last_fullsized":"2024-02-19T15:23:43.947741+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:35:54.831655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"45'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734750+0000","last_change":"2024-02-19T15:23:41.968055+0000","last_active":"2024-02-19T15:23:44.734750+0000","last_peered":"2024-02-19T15:23:44.734750+0000","last_clean":"2024-02-19T15:23:44.734750+0000","last_became_active":"2024-02-19T15:23:41.967728+0000","last_became_peered":"2024-02-19T15:23:41.967728+0000","last_unstale":"2024-02-19T15:23:44.734750+0000","last_undegraded":"2024-02-19T15:23:44.734750+0000","last_fullsized":"2024-02-19T15:23:44.734750+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:02:12.172289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946492+0000","last_change":"2024-02-19T15:23:42.966169+0000","last_active":"2024-02-19T15:23:43.946492+0000","last_peered":"2024-02-19T15:23:43.946492+0000","last_clean":"2024-02-19T15:23:43.946492+0000","last_became_active":"2024-02-19T15:23:42.965898+0000","last_became_peered":"2024-02-19T15:23:42.965898+0000","last_unstale":"2024-02-19T15:23:43.946492+0000","last_undegraded":"2024-02-19T15:23:43.946492+0000","last_fullsized":"2024-02-19T15:23:43.946492+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:41:45.823238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194546+0000","last_change":"2024-02-19T15:23:41.968087+0000","last_active":"2024-02-19T15:23:44.194546+0000","last_peered":"2024-02-19T15:23:44.194546+0000","last_clean":"2024-02-19T15:23:44.194546+0000","last_became_active":"2024-02-19T15:23:41.967798+0000","last_became_peered":"2024-02-19T15:23:41.967798+0000","last_unstale":"2024-02-19T15:23:44.194546+0000","last_undegraded":"2024-02-19T15:23:44.194546+0000","last_fullsized":"2024-02-19T15:23:44.194546+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:08:00.893923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194536+0000","last_change":"2024-02-19T15:23:42.971627+0000","last_active":"2024-02-19T15:23:44.194536+0000","last_peered":"2024-02-19T15:23:44.194536+0000","last_clean":"2024-02-19T15:23:44.194536+0000","last_became_active":"2024-02-19T15:23:42.971344+0000","last_became_peered":"2024-02-19T15:23:42.971344+0000","last_unstale":"2024-02-19T15:23:44.194536+0000","last_undegraded":"2024-02-19T15:23:44.194536+0000","last_fullsized":"2024-02-19T15:23:44.194536+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:50:40.431301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947382+0000","last_change":"2024-02-19T15:23:41.948963+0000","last_active":"2024-02-19T15:23:43.947382+0000","last_peered":"2024-02-19T15:23:43.947382+0000","last_clean":"2024-02-19T15:23:43.947382+0000","last_became_active":"2024-02-19T15:23:41.947243+0000","last_became_peered":"2024-02-19T15:23:41.947243+0000","last_unstale":"2024-02-19T15:23:43.947382+0000","last_undegraded":"2024-02-19T15:23:43.947382+0000","last_fullsized":"2024-02-19T15:23:43.947382+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:51:31.286375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734698+0000","last_change":"2024-02-19T15:23:42.974990+0000","last_active":"2024-02-19T15:23:44.734698+0000","last_peered":"2024-02-19T15:23:44.734698+0000","last_clean":"2024-02-19T15:23:44.734698+0000","last_became_active":"2024-02-19T15:23:42.974011+0000","last_became_peered":"2024-02-19T15:23:42.974011+0000","last_unstale":"2024-02-19T15:23:44.734698+0000","last_undegraded":"2024-02-19T15:23:44.734698+0000","last_fullsized":"2024-02-19T15:23:44.734698+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:48.375387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194296+0000","last_change":"2024-02-19T15:23:41.964025+0000","last_active":"2024-02-19T15:23:44.194296+0000","last_peered":"2024-02-19T15:23:44.194296+0000","last_clean":"2024-02-19T15:23:44.194296+0000","last_became_active":"2024-02-19T15:23:41.963717+0000","last_became_peered":"2024-02-19T15:23:41.963717+0000","last_unstale":"2024-02-19T15:23:44.194296+0000","last_undegraded":"2024-02-19T15:23:44.194296+0000","last_fullsized":"2024-02-19T15:23:44.194296+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:45:37.841170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734618+0000","last_change":"2024-02-19T15:23:42.974421+0000","last_active":"2024-02-19T15:23:44.734618+0000","last_peered":"2024-02-19T15:23:44.734618+0000","last_clean":"2024-02-19T15:23:44.734618+0000","last_became_active":"2024-02-19T15:23:42.974009+0000","last_became_peered":"2024-02-19T15:23:42.974009+0000","last_unstale":"2024-02-19T15:23:44.734618+0000","last_undegraded":"2024-02-19T15:23:44.734618+0000","last_fullsized":"2024-02-19T15:23:44.734618+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:36.710691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735244+0000","last_change":"2024-02-19T15:23:41.969004+0000","last_active":"2024-02-19T15:23:44.735244+0000","last_peered":"2024-02-19T15:23:44.735244+0000","last_clean":"2024-02-19T15:23:44.735244+0000","last_became_active":"2024-02-19T15:23:41.968796+0000","last_became_peered":"2024-02-19T15:23:41.968796+0000","last_unstale":"2024-02-19T15:23:44.735244+0000","last_undegraded":"2024-02-19T15:23:44.735244+0000","last_fullsized":"2024-02-19T15:23:44.735244+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:25:51.359695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735100+0000","last_change":"2024-02-19T15:23:42.956889+0000","last_active":"2024-02-19T15:23:44.735100+0000","last_peered":"2024-02-19T15:23:44.735100+0000","last_clean":"2024-02-19T15:23:44.735100+0000","last_became_active":"2024-02-19T15:23:42.953555+0000","last_became_peered":"2024-02-19T15:23:42.953555+0000","last_unstale":"2024-02-19T15:23:44.735100+0000","last_undegraded":"2024-02-19T15:23:44.735100+0000","last_fullsized":"2024-02-19T15:23:44.735100+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:50:45.305646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.947245+0000","last_change":"2024-02-19T15:23:41.956700+0000","last_active":"2024-02-19T15:23:43.947245+0000","last_peered":"2024-02-19T15:23:43.947245+0000","last_clean":"2024-02-19T15:23:43.947245+0000","last_became_active":"2024-02-19T15:23:41.956388+0000","last_became_peered":"2024-02-19T15:23:41.956388+0000","last_unstale":"2024-02-19T15:23:43.947245+0000","last_undegraded":"2024-02-19T15:23:43.947245+0000","last_fullsized":"2024-02-19T15:23:43.947245+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:15:40.531978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.734808+0000","last_change":"2024-02-19T15:23:42.976054+0000","last_active":"2024-02-19T15:23:44.734808+0000","last_peered":"2024-02-19T15:23:44.734808+0000","last_clean":"2024-02-19T15:23:44.734808+0000","last_became_active":"2024-02-19T15:23:42.975795+0000","last_became_peered":"2024-02-19T15:23:42.975795+0000","last_unstale":"2024-02-19T15:23:44.734808+0000","last_undegraded":"2024-02-19T15:23:44.734808+0000","last_fullsized":"2024-02-19T15:23:44.734808+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:01.020038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.945761+0000","last_change":"2024-02-19T15:23:41.962391+0000","last_active":"2024-02-19T15:23:43.945761+0000","last_peered":"2024-02-19T15:23:43.945761+0000","last_clean":"2024-02-19T15:23:43.945761+0000","last_became_active":"2024-02-19T15:23:41.961437+0000","last_became_peered":"2024-02-19T15:23:41.961437+0000","last_unstale":"2024-02-19T15:23:43.945761+0000","last_undegraded":"2024-02-19T15:23:43.945761+0000","last_fullsized":"2024-02-19T15:23:43.945761+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:16:05.431843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:43.946756+0000","last_change":"2024-02-19T15:23:42.964218+0000","last_active":"2024-02-19T15:23:43.946756+0000","last_peered":"2024-02-19T15:23:43.946756+0000","last_clean":"2024-02-19T15:23:43.946756+0000","last_became_active":"2024-02-19T15:23:42.963959+0000","last_became_peered":"2024-02-19T15:23:42.963959+0000","last_unstale":"2024-02-19T15:23:43.946756+0000","last_undegraded":"2024-02-19T15:23:43.946756+0000","last_fullsized":"2024-02-19T15:23:43.946756+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:41.932392+0000","last_clean_scrub_stamp":"2024-02-19T15:23:41.932392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:43:11.719148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.735163+0000","last_change":"2024-02-19T15:23:41.960473+0000","last_active":"2024-02-19T15:23:44.735163+0000","last_peered":"2024-02-19T15:23:44.735163+0000","last_clean":"2024-02-19T15:23:44.735163+0000","last_became_active":"2024-02-19T15:23:41.960208+0000","last_became_peered":"2024-02-19T15:23:41.960208+0000","last_unstale":"2024-02-19T15:23:44.735163+0000","last_undegraded":"2024-02-19T15:23:44.735163+0000","last_fullsized":"2024-02-19T15:23:44.735163+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:23:40.929810+0000","last_clean_scrub_stamp":"2024-02-19T15:23:40.929810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:23:11.863209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":212,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-02-19T15:23:44.194375+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:23:44.194375+0000","last_peered":"2024-02-19T15:23:44.194375+0000","last_clean":"2024-02-19T15:23:44.194375+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:23:44.194375+0000","last_undegraded":"2024-02-19T15:23:44.194375+0000","last_fullsized":"2024-02-19T15:23:44.194375+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509466,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":532480,"data_stored":213599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":4.6029999999999998,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.54800000000000004}]},{"osd":1,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.746}]},{"osd":2,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.57299999999999995}]},{"osd":3,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.50700000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.62}]},{"osd":5,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.67600000000000005}]},{"osd":7,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.69999999999999996}]}]},{"osd":7,"up_from":14,"seq":60129542170,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28124,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343716,"statfs":{"total":96636764160,"available":96607965184,"internally_reserved":0,"allocated":1122304,"data_stored":803385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.627},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.87}]},{"osd":1,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":2.968,"5min":2.968,"15min":2.968},"last":0.754}]},{"osd":2,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":0.71099999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":3.1720000000000002,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":0.77300000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.73699999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.83199999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.93799999999999994}]}]},{"osd":5,"up_from":14,"seq":60129542171,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27536,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344304,"statfs":{"total":96636764160,"available":96608567296,"internally_reserved":0,"allocated":528384,"data_stored":212997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.56999999999999995}]},{"osd":1,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.85499999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.66300000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.63900000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":0.67600000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.89400000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:22:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.59899999999999998}]}]},{"osd":4,"up_from":14,"seq":60129542169,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27544,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344296,"statfs":{"total":96636764160,"available":96608559104,"internally_reserved":0,"allocated":536576,"data_stored":214539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.72899999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.623}]},{"osd":2,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.755}]},{"osd":3,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.49199999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.80200000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.78800000000000003}]},{"osd":7,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.82199999999999995}]}]},{"osd":3,"up_from":14,"seq":60129542169,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":536576,"data_stored":213565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.2450000000000001,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.33500000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":4.0220000000000002,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.52900000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.72899999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":4.2039999999999997,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.78400000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":4.1589999999999998,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.627}]},{"osd":6,"last update":"Mon Feb 19 15:22:55 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":4.0940000000000003,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.69499999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":3.9660000000000002,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.67500000000000004}]}]},{"osd":2,"up_from":14,"seq":60129542170,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":540672,"data_stored":214561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.7160000000000002,"5min":4.7160000000000002,"15min":4.7160000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":8.2129999999999992,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.26000000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":8.1620000000000008,"5min":8.1620000000000008,"15min":8.1620000000000008},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.46100000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":4.7679999999999998,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.371},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":4.9909999999999997,"5min":4.9909999999999997,"15min":4.9909999999999997},"last":0.443}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":4.3810000000000002,"5min":4.3810000000000002,"15min":4.3810000000000002},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":4.3440000000000003,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.83199999999999996}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":4.4039999999999999,"5min":4.4039999999999999,"15min":4.4039999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":4.3259999999999996,"5min":4.3259999999999996,"15min":4.3259999999999996},"last":0.75}]},{"osd":6,"last update":"Mon Feb 19 15:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":4.5220000000000002,"5min":4.5220000000000002,"15min":4.5220000000000002},"last":0.71699999999999997},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":4.4320000000000004,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.73899999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":4.4829999999999997,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":4.4989999999999997,"5min":4.4989999999999997,"15min":4.4989999999999997},"last":0.84599999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542171,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":528384,"data_stored":212997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.25}]},{"osd":2,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.41599999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.438},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.38300000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.65700000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.746},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.77500000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:22:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.72699999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.70499999999999996}]}]},{"osd":0,"up_from":14,"seq":60129542169,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36436,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335404,"statfs":{"total":96636764160,"available":96599453696,"internally_reserved":0,"allocated":1122304,"data_stored":803329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.0199999999999996,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.31}]},{"osd":2,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":5.6669999999999998,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.4649999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.45700000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.9539999999999997,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.345,"5min":1.345,"15min":1.345},"last":0.44800000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":4.7880000000000003,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.61799999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.53200000000000003}]},{"osd":6,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":4.8460000000000001,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.65300000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:22:52 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":4.9809999999999999,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":4.915,"5min":4.915,"15min":4.915},"last":0.56200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:23:52.259 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T15:23:52.259 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-19T15:23:52.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.435+0000 7fe56da00640 1 Processor -- start 2024-02-19T15:23:52.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.435+0000 7fe56da00640 1 -- start start 2024-02-19T15:23:52.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.435+0000 7fe56da00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5680b1c80 0x7fe5680b2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:52.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.435+0000 7fe56da00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe5680b37f0 0x7fe56806aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:52.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.435+0000 7fe56da00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5680ace50 con 0x7fe5680ae140 2024-02-19T15:23:52.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.435+0000 7fe56da00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5680b25a0 con 0x7fe5680b37f0 2024-02-19T15:23:52.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.435+0000 7fe56da00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5680ad0a0 con 0x7fe5680b1c80 2024-02-19T15:23:52.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.439+0000 7fe55e000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fe5680ae140 0x7fe5680b30e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49186/0 (socket says 172.21.15.47:49186) 2024-02-19T15:23:52.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.439+0000 7fe567400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe5680b37f0 0x7fe56806aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:52.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.439+0000 7fe55e000640 1 -- 172.21.15.47:0/195234430 learned_addr learned my addr 172.21.15.47:0/195234430 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:52.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.439+0000 7fe566a00640 1 --2- 172.21.15.47:0/195234430 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5680b1c80 0x7fe5680b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:52.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.439+0000 7fe567400640 1 -- 172.21.15.47:0/195234430 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5680b1c80 msgr2=0x7fe5680b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:52.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.439+0000 7fe567400640 1 --2- 172.21.15.47:0/195234430 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5680b1c80 0x7fe5680b2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:52.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.439+0000 7fe567400640 1 -- 172.21.15.47:0/195234430 >> v1:172.21.15.47:6789/0 conn(0x7fe5680ae140 legacy=0x7fe5680b30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:52.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.439+0000 7fe567400640 1 -- 172.21.15.47:0/195234430 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5680ad380 con 0x7fe5680b37f0 2024-02-19T15:23:52.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.439+0000 7fe566000640 1 -- 172.21.15.47:0/195234430 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653435285 0 0) 0x7fe5680ace50 con 0x7fe5680ae140 2024-02-19T15:23:52.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.439+0000 7fe567400640 1 --2- 172.21.15.47:0/195234430 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5680b37f0 0x7fe56806aeb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe560009d90 tx=0x7fe56001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df7b26849e55bb0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:52.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.439+0000 7fe566000640 1 -- 172.21.15.47:0/195234430 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5600a5a00 con 0x7fe5680b37f0 2024-02-19T15:23:52.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe566000640 1 -- 172.21.15.47:0/195234430 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5600a5b60 con 0x7fe5680b37f0 2024-02-19T15:23:52.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe566000640 1 -- 172.21.15.47:0/195234430 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5600ad440 con 0x7fe5680b37f0 2024-02-19T15:23:52.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe56da00640 1 -- 172.21.15.47:0/195234430 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5680b37f0 msgr2=0x7fe56806aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:52.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe56da00640 1 --2- 172.21.15.47:0/195234430 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5680b37f0 0x7fe56806aeb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe560009d90 tx=0x7fe56001c940 comp rx=0 tx=0).stop 2024-02-19T15:23:52.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe56da00640 1 -- 172.21.15.47:0/195234430 shutdown_connections 2024-02-19T15:23:52.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe56da00640 1 --2- 172.21.15.47:0/195234430 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5680b37f0 0x7fe56806aeb0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:52.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe56da00640 1 --2- 172.21.15.47:0/195234430 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5680b1c80 0x7fe5680b2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:52.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe56da00640 1 -- 172.21.15.47:0/195234430 >> 172.21.15.47:0/195234430 conn(0x7fe5680ccc80 msgr2=0x7fe5680cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:52.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe56da00640 1 -- 172.21.15.47:0/195234430 shutdown_connections 2024-02-19T15:23:52.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe56da00640 1 -- 172.21.15.47:0/195234430 wait complete. 2024-02-19T15:23:52.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe56da00640 1 Processor -- start 2024-02-19T15:23:52.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe56da00640 1 -- start start 2024-02-19T15:23:52.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe56da00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5680ae140 0x7fe56815d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe56da00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5680b1c80 0x7fe56814d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe56da00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5680b37f0 0x7fe56814d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe56da00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5680b1ad0 con 0x7fe5680ae140 2024-02-19T15:23:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe56da00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5680cbb30 con 0x7fe5680b1c80 2024-02-19T15:23:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe56da00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5680cb800 con 0x7fe5680b37f0 2024-02-19T15:23:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe566a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5680ae140 0x7fe56815d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe566a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5680ae140 0x7fe56815d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33430/0 (socket says 172.21.15.47:33430) 2024-02-19T15:23:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe566a00640 1 -- 172.21.15.47:0/3351887988 learned_addr learned my addr 172.21.15.47:0/3351887988 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe567400640 1 --2- 172.21.15.47:0/3351887988 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5680b37f0 0x7fe56814d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.443+0000 7fe55e000640 1 --2- 172.21.15.47:0/3351887988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5680b1c80 0x7fe56814d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.447+0000 7fe566a00640 1 -- 172.21.15.47:0/3351887988 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5680b37f0 msgr2=0x7fe56814d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.447+0000 7fe566a00640 1 --2- 172.21.15.47:0/3351887988 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5680b37f0 0x7fe56814d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.447+0000 7fe566a00640 1 -- 172.21.15.47:0/3351887988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5680b1c80 msgr2=0x7fe56814d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:52.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.447+0000 7fe566a00640 1 --2- 172.21.15.47:0/3351887988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5680b1c80 0x7fe56814d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:52.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.447+0000 7fe566a00640 1 -- 172.21.15.47:0/3351887988 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe560009a70 con 0x7fe5680ae140 2024-02-19T15:23:52.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.447+0000 7fe55e000640 1 --2- 172.21.15.47:0/3351887988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5680b1c80 0x7fe56814d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:52.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.447+0000 7fe566a00640 1 --2- 172.21.15.47:0/3351887988 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5680ae140 0x7fe56815d0e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe55800d830 tx=0x7fe55800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:52.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.447+0000 7fe55f400640 1 -- 172.21.15.47:0/3351887988 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55800b6c0 con 0x7fe5680ae140 2024-02-19T15:23:52.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.447+0000 7fe55f400640 1 -- 172.21.15.47:0/3351887988 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55800b820 con 0x7fe5680ae140 2024-02-19T15:23:52.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.447+0000 7fe55f400640 1 -- 172.21.15.47:0/3351887988 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55800bb90 con 0x7fe5680ae140 2024-02-19T15:23:52.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.447+0000 7fe56da00640 1 -- 172.21.15.47:0/3351887988 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5680b0dc0 con 0x7fe5680ae140 2024-02-19T15:23:52.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.447+0000 7fe56da00640 1 -- 172.21.15.47:0/3351887988 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5680cbfc0 con 0x7fe5680ae140 2024-02-19T15:23:52.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.447+0000 7fe56da00640 1 -- 172.21.15.47:0/3351887988 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe56809e000 con 0x7fe5680ae140 2024-02-19T15:23:52.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.447+0000 7fe55f400640 1 -- 172.21.15.47:0/3351887988 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe55801e460 con 0x7fe5680ae140 2024-02-19T15:23:52.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.451+0000 7fe55f400640 1 --2- 172.21.15.47:0/3351887988 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe544066e40 0x7fe544069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:52.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.451+0000 7fe55f400640 1 -- 172.21.15.47:0/3351887988 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5580850f0 con 0x7fe5680ae140 2024-02-19T15:23:52.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.451+0000 7fe55e000640 1 --2- 172.21.15.47:0/3351887988 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe544066e40 0x7fe544069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:52.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.451+0000 7fe55e000640 1 --2- 172.21.15.47:0/3351887988 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe544066e40 0x7fe544069300 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe550007fd0 tx=0x7fe550009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.455+0000 7fe55f400640 1 -- 172.21.15.47:0/3351887988 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe558051f70 con 0x7fe5680ae140 2024-02-19T15:23:52.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.671+0000 7fe56da00640 1 -- 172.21.15.47:0/3351887988 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fe56806b220 con 0x7fe5680ae140 2024-02-19T15:23:52.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.675+0000 7fe55f400640 1 -- 172.21.15.47:0/3351887988 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7fe558055630 con 0x7fe5680ae140 2024-02-19T15:23:52.680 INFO:teuthology.orchestra.run.smithi047.stdout:[client.0] 2024-02-19T15:23:52.680 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDjcdNlb7eRNRAAk6ZLGfjhcdlR9GoXFCd9eA== 2024-02-19T15:23:52.680 INFO:teuthology.orchestra.run.smithi047.stdout: caps mds = "allow" 2024-02-19T15:23:52.680 INFO:teuthology.orchestra.run.smithi047.stdout: caps mon = "allow r" 2024-02-19T15:23:52.680 INFO:teuthology.orchestra.run.smithi047.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-19T15:23:52.681 INFO:teuthology.orchestra.run.smithi047.stderr:updated caps for client.0 2024-02-19T15:23:52.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.675+0000 7fe56da00640 1 -- 172.21.15.47:0/3351887988 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe544066e40 msgr2=0x7fe544069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:52.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.675+0000 7fe56da00640 1 --2- 172.21.15.47:0/3351887988 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe544066e40 0x7fe544069300 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe550007fd0 tx=0x7fe550009040 comp rx=0 tx=0).stop 2024-02-19T15:23:52.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.675+0000 7fe56da00640 1 -- 172.21.15.47:0/3351887988 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5680ae140 msgr2=0x7fe56815d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:52.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.675+0000 7fe56da00640 1 --2- 172.21.15.47:0/3351887988 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5680ae140 0x7fe56815d0e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe55800d830 tx=0x7fe55800dcf0 comp rx=0 tx=0).stop 2024-02-19T15:23:52.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.675+0000 7fe56da00640 1 -- 172.21.15.47:0/3351887988 shutdown_connections 2024-02-19T15:23:52.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.675+0000 7fe56da00640 1 --2- 172.21.15.47:0/3351887988 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe544066e40 0x7fe544069300 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:52.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.675+0000 7fe56da00640 1 --2- 172.21.15.47:0/3351887988 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5680b37f0 0x7fe56814d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:52.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.675+0000 7fe56da00640 1 --2- 172.21.15.47:0/3351887988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5680b1c80 0x7fe56814d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:52.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.675+0000 7fe56da00640 1 --2- 172.21.15.47:0/3351887988 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5680ae140 0x7fe56815d0e0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:52.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.675+0000 7fe56da00640 1 -- 172.21.15.47:0/3351887988 >> 172.21.15.47:0/3351887988 conn(0x7fe5680ccc80 msgr2=0x7fe56814e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:52.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.675+0000 7fe56da00640 1 -- 172.21.15.47:0/3351887988 shutdown_connections 2024-02-19T15:23:52.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.675+0000 7fe56da00640 1 -- 172.21.15.47:0/3351887988 wait complete. 2024-02-19T15:23:52.703 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:23:52.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.835+0000 7f3e7ce00640 1 Processor -- start 2024-02-19T15:23:52.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.839+0000 7f3e7ce00640 1 -- start start 2024-02-19T15:23:52.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.839+0000 7f3e7ce00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3e7815bef0 0x7f3e780b2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:52.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.839+0000 7f3e7ce00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e780b5d90 0x7f3e780d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:52.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.839+0000 7f3e7ce00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e780b2d40 con 0x7f3e780d0df0 2024-02-19T15:23:52.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.839+0000 7f3e7ce00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e780d00f0 con 0x7f3e7815bef0 2024-02-19T15:23:52.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.839+0000 7f3e7ce00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e780af690 con 0x7f3e780b5d90 2024-02-19T15:23:52.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.839+0000 7f3e75e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3e7815bef0 0x7f3e780b2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:52.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.839+0000 7f3e75e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3e7815bef0 0x7f3e780b2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33446/0 (socket says 172.21.15.47:33446) 2024-02-19T15:23:52.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.839+0000 7f3e75e00640 1 -- 172.21.15.47:0/100655169 learned_addr learned my addr 172.21.15.47:0/100655169 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:52.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.839+0000 7f3e6fe00640 1 --2- 172.21.15.47:0/100655169 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e780b5d90 0x7f3e780d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:52.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.839+0000 7f3e75e00640 1 -- 172.21.15.47:0/100655169 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e780b5d90 msgr2=0x7f3e780d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:52.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.839+0000 7f3e75e00640 1 --2- 172.21.15.47:0/100655169 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e780b5d90 0x7f3e780d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:52.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.839+0000 7f3e75e00640 1 -- 172.21.15.47:0/100655169 >> v1:172.21.15.47:6789/0 conn(0x7f3e780d0df0 legacy=0x7f3e7806b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:52.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.839+0000 7f3e75e00640 1 -- 172.21.15.47:0/100655169 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e780af970 con 0x7f3e7815bef0 2024-02-19T15:23:52.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.839+0000 7f3e75400640 1 -- 172.21.15.47:0/100655169 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515433077 0 0) 0x7f3e780b2d40 con 0x7f3e780d0df0 2024-02-19T15:23:52.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.843+0000 7f3e75e00640 1 --2- 172.21.15.47:0/100655169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e7815bef0 0x7f3e780b2800 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3e680098a0 tx=0x7f3e6801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80cb51f9c5baae46 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:52.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.843+0000 7f3e75400640 1 -- 172.21.15.47:0/100655169 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e680ac410 con 0x7f3e7815bef0 2024-02-19T15:23:52.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.843+0000 7f3e75400640 1 -- 172.21.15.47:0/100655169 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e680ac570 con 0x7f3e7815bef0 2024-02-19T15:23:52.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.843+0000 7f3e75400640 1 -- 172.21.15.47:0/100655169 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e680ac900 con 0x7f3e7815bef0 2024-02-19T15:23:52.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.843+0000 7f3e7ce00640 1 -- 172.21.15.47:0/100655169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e7815bef0 msgr2=0x7f3e780b2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:52.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.843+0000 7f3e7ce00640 1 --2- 172.21.15.47:0/100655169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e7815bef0 0x7f3e780b2800 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3e680098a0 tx=0x7f3e6801c480 comp rx=0 tx=0).stop 2024-02-19T15:23:52.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.843+0000 7f3e7ce00640 1 -- 172.21.15.47:0/100655169 shutdown_connections 2024-02-19T15:23:52.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.843+0000 7f3e7ce00640 1 --2- 172.21.15.47:0/100655169 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e780b5d90 0x7f3e780d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:52.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.843+0000 7f3e7ce00640 1 --2- 172.21.15.47:0/100655169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e7815bef0 0x7f3e780b2800 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:52.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.843+0000 7f3e7ce00640 1 -- 172.21.15.47:0/100655169 >> 172.21.15.47:0/100655169 conn(0x7f3e780b4fc0 msgr2=0x7f3e780b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:52.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.847+0000 7f3e7ce00640 1 -- 172.21.15.47:0/100655169 shutdown_connections 2024-02-19T15:23:52.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.847+0000 7f3e7ce00640 1 -- 172.21.15.47:0/100655169 wait complete. 2024-02-19T15:23:52.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.847+0000 7f3e7ce00640 1 Processor -- start 2024-02-19T15:23:52.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.847+0000 7f3e7ce00640 1 -- start start 2024-02-19T15:23:52.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e7ce00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e780b5d90 0x7f3e78149820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:52.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e7ce00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e780d0df0 0x7f3e780ae080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:52.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e7ce00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e7815bef0 0x7f3e78165b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:52.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e7ce00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7806e9f0 con 0x7f3e780b5d90 2024-02-19T15:23:52.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e7ce00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e78151f10 con 0x7f3e780d0df0 2024-02-19T15:23:52.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e7ce00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e781521f0 con 0x7f3e7815bef0 2024-02-19T15:23:52.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e75e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e780b5d90 0x7f3e78149820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:52.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e75e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e780b5d90 0x7f3e78149820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33458/0 (socket says 172.21.15.47:33458) 2024-02-19T15:23:52.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e75e00640 1 -- 172.21.15.47:0/3441111952 learned_addr learned my addr 172.21.15.47:0/3441111952 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:52.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e6fe00640 1 --2- 172.21.15.47:0/3441111952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e780d0df0 0x7f3e780ae080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:52.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e76800640 1 --2- 172.21.15.47:0/3441111952 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e7815bef0 0x7f3e78165b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:52.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e75e00640 1 -- 172.21.15.47:0/3441111952 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e7815bef0 msgr2=0x7f3e78165b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:52.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e75e00640 1 --2- 172.21.15.47:0/3441111952 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e7815bef0 0x7f3e78165b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:52.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e75e00640 1 -- 172.21.15.47:0/3441111952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e780d0df0 msgr2=0x7f3e780ae080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:52.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e75e00640 1 --2- 172.21.15.47:0/3441111952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e780d0df0 0x7f3e780ae080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:52.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e75e00640 1 -- 172.21.15.47:0/3441111952 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e680094d0 con 0x7f3e780b5d90 2024-02-19T15:23:52.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e76800640 1 --2- 172.21.15.47:0/3441111952 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e7815bef0 0x7f3e78165b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:52.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e75e00640 1 --2- 172.21.15.47:0/3441111952 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e780b5d90 0x7f3e78149820 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3e680af040 tx=0x7f3e68019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:52.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e6e000640 1 -- 172.21.15.47:0/3441111952 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e680ac410 con 0x7f3e780b5d90 2024-02-19T15:23:52.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e6e000640 1 -- 172.21.15.47:0/3441111952 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e680ace30 con 0x7f3e780b5d90 2024-02-19T15:23:52.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e7ce00640 1 -- 172.21.15.47:0/3441111952 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e781524d0 con 0x7f3e780b5d90 2024-02-19T15:23:52.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e6e000640 1 -- 172.21.15.47:0/3441111952 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e680b5870 con 0x7f3e780b5d90 2024-02-19T15:23:52.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e7ce00640 1 -- 172.21.15.47:0/3441111952 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e78152a90 con 0x7f3e780b5d90 2024-02-19T15:23:52.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.851+0000 7f3e7ce00640 1 -- 172.21.15.47:0/3441111952 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e78152c80 con 0x7f3e780b5d90 2024-02-19T15:23:52.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.855+0000 7f3e6e000640 1 -- 172.21.15.47:0/3441111952 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3e680b3070 con 0x7f3e780b5d90 2024-02-19T15:23:52.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.855+0000 7f3e6e000640 1 --2- 172.21.15.47:0/3441111952 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3e54066e40 0x7f3e54069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:52.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.855+0000 7f3e6e000640 1 -- 172.21.15.47:0/3441111952 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e6811ee30 con 0x7f3e780b5d90 2024-02-19T15:23:52.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.855+0000 7f3e6fe00640 1 --2- 172.21.15.47:0/3441111952 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3e54066e40 0x7f3e54069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:52.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.855+0000 7f3e6fe00640 1 --2- 172.21.15.47:0/3441111952 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3e54066e40 0x7f3e54069300 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3e7809e420 tx=0x7f3e60009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:52.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:52.859+0000 7f3e6e000640 1 -- 172.21.15.47:0/3441111952 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f3e680ebcb0 con 0x7f3e780b5d90 2024-02-19T15:23:53.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.055+0000 7f3e7ce00640 1 -- 172.21.15.47:0/3441111952 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e78075360 con 0x7f3e780b5d90 2024-02-19T15:23:53.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.055+0000 7f3e6e000640 1 -- 172.21.15.47:0/3441111952 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f3e680b1020 con 0x7f3e780b5d90 2024-02-19T15:23:53.062 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:23:53.062 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5132,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/548836925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":548836925},{"type":"v1","addr":"172.21.15.116:6836","nonce":548836925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5135,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3530214438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3530214438},{"type":"v1","addr":"172.21.15.116:6837","nonce":3530214438}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/3433978846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3433978846},{"type":"v1","addr":"172.21.15.47:6834","nonce":3433978846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:23:42.342661+0000","modified":"2024-02-19T15:23:45.352372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5514":{"gid":5514,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3137466738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3137466738},{"type":"v1","addr":"172.21.15.47:6835","nonce":3137466738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-02-19T15:23:53.062 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 25 2024-02-19T15:23:53.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.059+0000 7f3e7ce00640 1 -- 172.21.15.47:0/3441111952 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3e54066e40 msgr2=0x7f3e54069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:53.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.059+0000 7f3e7ce00640 1 --2- 172.21.15.47:0/3441111952 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3e54066e40 0x7f3e54069300 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3e7809e420 tx=0x7f3e60009190 comp rx=0 tx=0).stop 2024-02-19T15:23:53.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.059+0000 7f3e7ce00640 1 -- 172.21.15.47:0/3441111952 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e780b5d90 msgr2=0x7f3e78149820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:53.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.059+0000 7f3e7ce00640 1 --2- 172.21.15.47:0/3441111952 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e780b5d90 0x7f3e78149820 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3e680af040 tx=0x7f3e68019220 comp rx=0 tx=0).stop 2024-02-19T15:23:53.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.059+0000 7f3e7ce00640 1 -- 172.21.15.47:0/3441111952 shutdown_connections 2024-02-19T15:23:53.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.059+0000 7f3e7ce00640 1 --2- 172.21.15.47:0/3441111952 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3e54066e40 0x7f3e54069300 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.059+0000 7f3e7ce00640 1 --2- 172.21.15.47:0/3441111952 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e7815bef0 0x7f3e78165b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.059+0000 7f3e7ce00640 1 --2- 172.21.15.47:0/3441111952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e780d0df0 0x7f3e780ae080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.059+0000 7f3e7ce00640 1 --2- 172.21.15.47:0/3441111952 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e780b5d90 0x7f3e78149820 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.059+0000 7f3e7ce00640 1 -- 172.21.15.47:0/3441111952 >> 172.21.15.47:0/3441111952 conn(0x7f3e780b4fc0 msgr2=0x7f3e78153510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:53.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.059+0000 7f3e7ce00640 1 -- 172.21.15.47:0/3441111952 shutdown_connections 2024-02-19T15:23:53.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.059+0000 7f3e7ce00640 1 -- 172.21.15.47:0/3441111952 wait complete. 2024-02-19T15:23:53.486 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:23:42.342661+0000', 'modified': '2024-02-19T15:23:45.352372+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5514}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5514': {'gid': 5514, 'name': 'a', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/3137466738', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 3137466738}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 3137466738}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:23:53.486 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T15:23:53.486 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-19T15:23:53.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.623+0000 7fa268800640 1 Processor -- start 2024-02-19T15:23:53.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.623+0000 7fa268800640 1 -- start start 2024-02-19T15:23:53.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.623+0000 7fa268800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa26006d650 0x7fa2600d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.623+0000 7fa268800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2600cce30 con 0x7fa2600ced10 2024-02-19T15:23:53.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.623+0000 7fa268800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2600cd110 con 0x7fa2600cded0 2024-02-19T15:23:53.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.623+0000 7fa268800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2600cd3f0 con 0x7fa26006d650 2024-02-19T15:23:53.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.623+0000 7fa266800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa2600ced10 0x7fa26006bd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:46506/0 (socket says 172.21.15.47:46506) 2024-02-19T15:23:53.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.623+0000 7fa266800640 1 -- 172.21.15.47:0/3770973588 learned_addr learned my addr 172.21.15.47:0/3770973588 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:23:53.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.623+0000 7fa265e00640 1 --2- 172.21.15.47:0/3770973588 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa26006d650 0x7fa2600d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.623+0000 7fa264a00640 1 -- 172.21.15.47:0/3770973588 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514841186 0 0) 0x7fa2600cce30 con 0x7fa2600ced10 2024-02-19T15:23:53.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.623+0000 7fa264a00640 1 -- 172.21.15.47:0/3770973588 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa248003530 con 0x7fa2600ced10 2024-02-19T15:23:53.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.623+0000 7fa264a00640 1 -- 172.21.15.47:0/3770973588 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544440279 0 0) 0x7fa2600cd110 con 0x7fa2600cded0 2024-02-19T15:23:53.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.623+0000 7fa264a00640 1 -- 172.21.15.47:0/3770973588 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2600cce30 con 0x7fa2600cded0 2024-02-19T15:23:53.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.623+0000 7fa265e00640 1 -- 172.21.15.47:0/3770973588 >> v1:172.21.15.116:6789/0 conn(0x7fa2600cded0 legacy=0x7fa2600ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:53.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.623+0000 7fa264a00640 1 -- 172.21.15.47:0/3770973588 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3017239549 0 0) 0x7fa248003530 con 0x7fa2600ced10 2024-02-19T15:23:53.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.623+0000 7fa265e00640 1 -- 172.21.15.47:0/3770973588 >> v1:172.21.15.47:6789/0 conn(0x7fa2600ced10 legacy=0x7fa26006bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:53.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.623+0000 7fa265e00640 1 -- 172.21.15.47:0/3770973588 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2600cd6d0 con 0x7fa26006d650 2024-02-19T15:23:53.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.627+0000 7fa265e00640 1 --2- 172.21.15.47:0/3770973588 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa26006d650 0x7fa2600d08b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa2540097f0 tx=0x7fa25401c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=585e6d25dd27603a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:53.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.627+0000 7fa264a00640 1 -- 172.21.15.47:0/3770973588 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25c001a00 con 0x7fa26006d650 2024-02-19T15:23:53.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.627+0000 7fa264a00640 1 -- 172.21.15.47:0/3770973588 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2540a7050 con 0x7fa26006d650 2024-02-19T15:23:53.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.627+0000 7fa264a00640 1 -- 172.21.15.47:0/3770973588 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2540ab600 con 0x7fa26006d650 2024-02-19T15:23:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.627+0000 7fa268800640 1 -- 172.21.15.47:0/3770973588 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa26006d650 msgr2=0x7fa2600d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.627+0000 7fa268800640 1 --2- 172.21.15.47:0/3770973588 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa26006d650 0x7fa2600d08b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa2540097f0 tx=0x7fa25401c340 comp rx=0 tx=0).stop 2024-02-19T15:23:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.627+0000 7fa268800640 1 -- 172.21.15.47:0/3770973588 shutdown_connections 2024-02-19T15:23:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.627+0000 7fa268800640 1 --2- 172.21.15.47:0/3770973588 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa26006d650 0x7fa2600d08b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.627+0000 7fa268800640 1 -- 172.21.15.47:0/3770973588 >> 172.21.15.47:0/3770973588 conn(0x7fa2600d1bf0 msgr2=0x7fa2600d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.627+0000 7fa268800640 1 -- 172.21.15.47:0/3770973588 shutdown_connections 2024-02-19T15:23:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.627+0000 7fa268800640 1 -- 172.21.15.47:0/3770973588 wait complete. 2024-02-19T15:23:53.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa268800640 1 Processor -- start 2024-02-19T15:23:53.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa268800640 1 -- start start 2024-02-19T15:23:53.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa268800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa26006d650 0x7fa26015b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa268800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2600cded0 0x7fa26015e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa268800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2600ced10 0x7fa2600c5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa268800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2600c62b0 con 0x7fa2600cded0 2024-02-19T15:23:53.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa268800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26015b7f0 con 0x7fa2600ced10 2024-02-19T15:23:53.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa268800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa26015bad0 con 0x7fa26006d650 2024-02-19T15:23:53.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa265400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2600cded0 0x7fa26015e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa265400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2600cded0 0x7fa26015e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39914/0 (socket says 172.21.15.47:39914) 2024-02-19T15:23:53.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa265400640 1 -- 172.21.15.47:0/3517668284 learned_addr learned my addr 172.21.15.47:0/3517668284 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:53.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa266800640 1 --2- 172.21.15.47:0/3517668284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2600ced10 0x7fa2600c5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa265e00640 1 --2- 172.21.15.47:0/3517668284 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa26006d650 0x7fa26015b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa265400640 1 -- 172.21.15.47:0/3517668284 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa26006d650 msgr2=0x7fa26015b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:53.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa265400640 1 --2- 172.21.15.47:0/3517668284 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa26006d650 0x7fa26015b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa265400640 1 -- 172.21.15.47:0/3517668284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2600ced10 msgr2=0x7fa2600c5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:53.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa265400640 1 --2- 172.21.15.47:0/3517668284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2600ced10 0x7fa2600c5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa265400640 1 -- 172.21.15.47:0/3517668284 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa254009420 con 0x7fa2600cded0 2024-02-19T15:23:53.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa265400640 1 --2- 172.21.15.47:0/3517668284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2600cded0 0x7fa26015e3b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa25000c430 tx=0x7fa25000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:53.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa25a000640 1 -- 172.21.15.47:0/3517668284 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2500052d0 con 0x7fa2600cded0 2024-02-19T15:23:53.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa25a000640 1 -- 172.21.15.47:0/3517668284 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa250005450 con 0x7fa2600cded0 2024-02-19T15:23:53.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa25a000640 1 -- 172.21.15.47:0/3517668284 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa250003840 con 0x7fa2600cded0 2024-02-19T15:23:53.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa268800640 1 -- 172.21.15.47:0/3517668284 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa26015bdb0 con 0x7fa2600cded0 2024-02-19T15:23:53.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.631+0000 7fa268800640 1 -- 172.21.15.47:0/3517668284 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26015ec50 con 0x7fa2600cded0 2024-02-19T15:23:53.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.635+0000 7fa25a000640 1 -- 172.21.15.47:0/3517668284 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa2500039a0 con 0x7fa2600cded0 2024-02-19T15:23:53.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.635+0000 7fa25a000640 1 --2- 172.21.15.47:0/3517668284 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa244066e40 0x7fa244069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.635+0000 7fa25a000640 1 -- 172.21.15.47:0/3517668284 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa250084100 con 0x7fa2600cded0 2024-02-19T15:23:53.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.635+0000 7fa265e00640 1 --2- 172.21.15.47:0/3517668284 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa244066e40 0x7fa244069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.635+0000 7fa265e00640 1 --2- 172.21.15.47:0/3517668284 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa244066e40 0x7fa244069300 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa2540ad000 tx=0x7fa2540a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:53.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.635+0000 7fa26bc35000 1 Processor -- start 2024-02-19T15:23:53.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa26bc35000 1 -- start start 2024-02-19T15:23:53.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa26bc35000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55bb366a0920 0x55bb366a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa26bc35000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb366a1240 0x55bb366569f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa26bc35000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb36656f30 0x55bb36657310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa26bc35000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb3651ed00 con 0x55bb366a1240 2024-02-19T15:23:53.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa26bc35000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb364b0430 con 0x55bb36656f30 2024-02-19T15:23:53.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa26bc35000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb364ad960 con 0x55bb366a0920 2024-02-19T15:23:53.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa265400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb366a1240 0x55bb366569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa265400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb366a1240 0x55bb366569f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39930/0 (socket says 172.21.15.47:39930) 2024-02-19T15:23:53.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa265400640 1 -- 172.21.15.47:0/3368085506 learned_addr learned my addr 172.21.15.47:0/3368085506 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:53.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa265400640 1 -- 172.21.15.47:0/3368085506 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55bb366a0920 msgr2=0x55bb366a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:53.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa265400640 1 --2- 172.21.15.47:0/3368085506 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55bb366a0920 0x55bb366a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa265400640 1 -- 172.21.15.47:0/3368085506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb36656f30 msgr2=0x55bb36657310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:53.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa266800640 1 --2- 172.21.15.47:0/3368085506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb36656f30 0x55bb36657310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa265400640 1 --2- 172.21.15.47:0/3368085506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb36656f30 0x55bb36657310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa265400640 1 -- 172.21.15.47:0/3368085506 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb3619f890 con 0x55bb366a1240 2024-02-19T15:23:53.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa266800640 1 --2- 172.21.15.47:0/3368085506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb36656f30 0x55bb36657310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:23:53.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa265400640 1 --2- 172.21.15.47:0/3368085506 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb366a1240 0x55bb366569f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa25005cd90 tx=0x7fa2500552b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4788ab739f09bf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:53.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa24ea00640 1 -- 172.21.15.47:0/3368085506 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa250058d00 con 0x55bb366a1240 2024-02-19T15:23:53.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa24ea00640 1 -- 172.21.15.47:0/3368085506 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa25005d2c0 con 0x55bb366a1240 2024-02-19T15:23:53.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa24ea00640 1 -- 172.21.15.47:0/3368085506 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa250051220 con 0x55bb366a1240 2024-02-19T15:23:53.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa26bc35000 1 -- 172.21.15.47:0/3368085506 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb366a1240 msgr2=0x55bb366569f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:53.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa26bc35000 1 --2- 172.21.15.47:0/3368085506 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb366a1240 0x55bb366569f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa25005cd90 tx=0x7fa2500552b0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa26bc35000 1 -- 172.21.15.47:0/3368085506 shutdown_connections 2024-02-19T15:23:53.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa26bc35000 1 --2- 172.21.15.47:0/3368085506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb36656f30 0x55bb36657310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa26bc35000 1 --2- 172.21.15.47:0/3368085506 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb366a1240 0x55bb366569f0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa26bc35000 1 --2- 172.21.15.47:0/3368085506 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55bb366a0920 0x55bb366a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa26bc35000 1 -- 172.21.15.47:0/3368085506 >> 172.21.15.47:0/3368085506 conn(0x55bb3665d2a0 msgr2=0x55bb366552d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:53.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa26bc35000 1 -- 172.21.15.47:0/3368085506 shutdown_connections 2024-02-19T15:23:53.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.639+0000 7fa26bc35000 1 -- 172.21.15.47:0/3368085506 wait complete. 2024-02-19T15:23:53.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa26bc35000 1 -- start start 2024-02-19T15:23:53.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa26bc35000 1 Processor -- start 2024-02-19T15:23:53.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa26bc35000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55bb36656f30 0x55bb366895e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa26bc35000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb366a0920 0x55bb366a1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa26bc35000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb366a1240 0x55bb366a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa26bc35000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb364a9140 con 0x55bb366a0920 2024-02-19T15:23:53.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa26bc35000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb36443630 con 0x55bb366a1240 2024-02-19T15:23:53.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa26bc35000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb364432d0 con 0x55bb36656f30 2024-02-19T15:23:53.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa26bc35000 10 client.0 _ll_register_callbacks cb 0x55bb365591c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:23:53.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa265400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb366a0920 0x55bb366a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa265400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb366a0920 0x55bb366a1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39934/0 (socket says 172.21.15.47:39934) 2024-02-19T15:23:53.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa265400640 1 -- 172.21.15.47:0/1352684870 learned_addr learned my addr 172.21.15.47:0/1352684870 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:53.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa265e00640 1 --2- 172.21.15.47:0/1352684870 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55bb36656f30 0x55bb366895e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa266800640 1 --2- 172.21.15.47:0/1352684870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb366a1240 0x55bb366a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa265400640 1 -- 172.21.15.47:0/1352684870 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55bb36656f30 msgr2=0x55bb366895e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:53.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa265400640 1 --2- 172.21.15.47:0/1352684870 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55bb36656f30 0x55bb366895e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa265400640 1 -- 172.21.15.47:0/1352684870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb366a1240 msgr2=0x55bb366a1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:53.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa265400640 1 --2- 172.21.15.47:0/1352684870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb366a1240 0x55bb366a1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa265400640 1 -- 172.21.15.47:0/1352684870 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26015ec50 con 0x55bb366a0920 2024-02-19T15:23:53.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa265400640 1 --2- 172.21.15.47:0/1352684870 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb366a0920 0x55bb366a1830 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa2500548b0 tx=0x7fa25001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8921262d1a206836 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:53.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.643+0000 7fa243400640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:23:53.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa243400640 1 -- 172.21.15.47:0/1352684870 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25007d070 con 0x55bb366a0920 2024-02-19T15:23:53.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa243400640 1 -- 172.21.15.47:0/1352684870 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa250051220 con 0x55bb366a0920 2024-02-19T15:23:53.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 1 -- 172.21.15.47:0/1352684870 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bb3645c850 con 0x55bb366a0920 2024-02-19T15:23:53.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa243400640 1 -- 172.21.15.47:0/1352684870 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2500513e0 con 0x55bb366a0920 2024-02-19T15:23:53.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 10 client.5685 fetch_fsmap learned FSMap version 25 2024-02-19T15:23:53.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 1 -- 172.21.15.47:0/1352684870 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x55bb36468e70 con 0x55bb366a0920 2024-02-19T15:23:53.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa243400640 1 -- 172.21.15.47:0/1352684870 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa250078080 con 0x55bb366a0920 2024-02-19T15:23:53.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa243400640 1 -- 172.21.15.47:0/1352684870 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 25) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa250068b70 con 0x55bb366a0920 2024-02-19T15:23:53.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 10 client.5685 fetch_fsmap finished waiting for FSMap version 25 2024-02-19T15:23:53.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 10 client.5685 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-19T15:23:53.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 20 client.5685 populate_metadata read hostname 'smithi047' 2024-02-19T15:23:53.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 1 --2- 172.21.15.47:0/1352684870 >> [v2:172.21.15.47:6833/3137466738,v1:172.21.15.47:6835/3137466738] conn(0x55bb366a2780 0x55bb366a2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 4 client.5685 mds_command: new command op to 5514 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T15:23:53.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 1 -- 172.21.15.47:0/1352684870 --> [v2:172.21.15.47:6833/3137466738,v1:172.21.15.47:6835/3137466738] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bb364a5f20 con 0x55bb366a2780 2024-02-19T15:23:53.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa265e00640 1 --2- 172.21.15.47:0/1352684870 >> [v2:172.21.15.47:6833/3137466738,v1:172.21.15.47:6835/3137466738] conn(0x55bb366a2780 0x55bb366a2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa243400640 10 client.5685 ms_handle_connect on v2:172.21.15.47:6833/3137466738 2024-02-19T15:23:53.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa265e00640 1 --2- 172.21.15.47:0/1352684870 >> [v2:172.21.15.47:6833/3137466738,v1:172.21.15.47:6835/3137466738] conn(0x55bb366a2780 0x55bb366a2b60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7502ba9538462ffc server_cookie=fdfadb4e48c066d9 in_seq=0 out_seq=0 2024-02-19T15:23:53.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa243400640 1 -- 172.21.15.47:0/1352684870 <== mds.0 v2:172.21.15.47:6833/3137466738 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bb364a5f20 con 0x55bb366a2780 2024-02-19T15:23:53.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa243400640 10 client.5685 handle_command_reply: tid=0 2024-02-19T15:23:53.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 1 client.5685 shutdown 2024-02-19T15:23:53.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 1 -- 172.21.15.47:0/1352684870 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb366a0920 msgr2=0x55bb366a1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:53.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 1 --2- 172.21.15.47:0/1352684870 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb366a0920 0x55bb366a1830 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa2500548b0 tx=0x7fa25001d000 comp rx=0 tx=0).stop 2024-02-19T15:23:53.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 1 -- 172.21.15.47:0/1352684870 shutdown_connections 2024-02-19T15:23:53.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 1 --2- 172.21.15.47:0/1352684870 >> [v2:172.21.15.47:6833/3137466738,v1:172.21.15.47:6835/3137466738] conn(0x55bb366a2780 0x55bb366a2b60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 1 --2- 172.21.15.47:0/1352684870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb366a1240 0x55bb366a1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 1 --2- 172.21.15.47:0/1352684870 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb366a0920 0x55bb366a1830 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 1 --2- 172.21.15.47:0/1352684870 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55bb36656f30 0x55bb366895e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa243400640 0 client.5685 ms_handle_reset on v2:172.21.15.47:6833/3137466738 2024-02-19T15:23:53.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 1 -- 172.21.15.47:0/1352684870 >> 172.21.15.47:0/1352684870 conn(0x55bb3665d2a0 msgr2=0x55bb366559e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:53.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 1 -- 172.21.15.47:0/1352684870 shutdown_connections 2024-02-19T15:23:53.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 1 -- 172.21.15.47:0/1352684870 wait complete. 2024-02-19T15:23:53.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.647+0000 7fa26bc35000 20 client.5685 trim_cache size 0 max 16384 2024-02-19T15:23:53.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.663+0000 7fa26bc35000 1 Processor -- start 2024-02-19T15:23:53.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.663+0000 7fa26bc35000 1 -- start start 2024-02-19T15:23:53.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa26bc35000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb36656f30 0x55bb3665c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa26bc35000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55bb366a0920 0x55bb3665c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa26bc35000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb366a1240 0x55bb3665a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa26bc35000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb36160940 con 0x55bb36656f30 2024-02-19T15:23:53.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa26bc35000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb366ad6b0 con 0x55bb366a1240 2024-02-19T15:23:53.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa26bc35000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb366acde0 con 0x55bb366a0920 2024-02-19T15:23:53.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa265e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb36656f30 0x55bb3665c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa265e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb36656f30 0x55bb3665c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39950/0 (socket says 172.21.15.47:39950) 2024-02-19T15:23:53.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa265e00640 1 -- 172.21.15.47:0/1006404082 learned_addr learned my addr 172.21.15.47:0/1006404082 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:53.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa265400640 1 --2- 172.21.15.47:0/1006404082 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55bb366a0920 0x55bb3665c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa266800640 1 --2- 172.21.15.47:0/1006404082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb366a1240 0x55bb3665a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa265e00640 1 -- 172.21.15.47:0/1006404082 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55bb366a0920 msgr2=0x55bb3665c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:53.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa265e00640 1 --2- 172.21.15.47:0/1006404082 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55bb366a0920 0x55bb3665c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa265e00640 1 -- 172.21.15.47:0/1006404082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb366a1240 msgr2=0x55bb3665a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:53.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa265e00640 1 --2- 172.21.15.47:0/1006404082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb366a1240 0x55bb3665a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa265e00640 1 -- 172.21.15.47:0/1006404082 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb3644ee10 con 0x55bb36656f30 2024-02-19T15:23:53.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa265e00640 1 --2- 172.21.15.47:0/1006404082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb36656f30 0x55bb3665c090 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fa254000c10 tx=0x7fa2540046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b7af89adec598d5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:53.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa24e000640 1 -- 172.21.15.47:0/1006404082 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2540c3de0 con 0x55bb36656f30 2024-02-19T15:23:53.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa24e000640 1 -- 172.21.15.47:0/1006404082 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2540bb040 con 0x55bb36656f30 2024-02-19T15:23:53.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa24e000640 1 -- 172.21.15.47:0/1006404082 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa254003820 con 0x55bb36656f30 2024-02-19T15:23:53.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa26bc35000 1 -- 172.21.15.47:0/1006404082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb36656f30 msgr2=0x55bb3665c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:53.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa26bc35000 1 --2- 172.21.15.47:0/1006404082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb36656f30 0x55bb3665c090 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fa254000c10 tx=0x7fa2540046d0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa26bc35000 1 -- 172.21.15.47:0/1006404082 shutdown_connections 2024-02-19T15:23:53.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa26bc35000 1 --2- 172.21.15.47:0/1006404082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb366a1240 0x55bb3665a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa26bc35000 1 --2- 172.21.15.47:0/1006404082 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55bb366a0920 0x55bb3665c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa26bc35000 1 --2- 172.21.15.47:0/1006404082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb36656f30 0x55bb3665c090 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa26bc35000 1 -- 172.21.15.47:0/1006404082 >> 172.21.15.47:0/1006404082 conn(0x55bb3665d2a0 msgr2=0x55bb366561f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:53.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa26bc35000 1 -- 172.21.15.47:0/1006404082 shutdown_connections 2024-02-19T15:23:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa26bc35000 1 -- 172.21.15.47:0/1006404082 wait complete. 2024-02-19T15:23:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa26bc35000 1 -- start start 2024-02-19T15:23:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.667+0000 7fa26bc35000 1 Processor -- start 2024-02-19T15:23:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb36656f30 0x55bb366e95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55bb366a0920 0x55bb366e9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb366a1240 0x55bb366ea420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb364a7fd0 con 0x55bb36656f30 2024-02-19T15:23:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb364e91a0 con 0x55bb366a1240 2024-02-19T15:23:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb3664e750 con 0x55bb366a0920 2024-02-19T15:23:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 10 client.0 _ll_register_callbacks cb 0x7fa250054780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:23:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa265e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb36656f30 0x55bb366e95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa265e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb36656f30 0x55bb366e95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39964/0 (socket says 172.21.15.47:39964) 2024-02-19T15:23:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa265e00640 1 -- 172.21.15.47:0/2800161510 learned_addr learned my addr 172.21.15.47:0/2800161510 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa265400640 1 --2- 172.21.15.47:0/2800161510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55bb366a0920 0x55bb366e9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa266800640 1 --2- 172.21.15.47:0/2800161510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb366a1240 0x55bb366ea420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa265e00640 1 -- 172.21.15.47:0/2800161510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55bb366a0920 msgr2=0x55bb366e9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:53.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa265e00640 1 --2- 172.21.15.47:0/2800161510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55bb366a0920 0x55bb366e9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa265e00640 1 -- 172.21.15.47:0/2800161510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb366a1240 msgr2=0x55bb366ea420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:53.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa265e00640 1 --2- 172.21.15.47:0/2800161510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb366a1240 0x55bb366ea420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa265e00640 1 -- 172.21.15.47:0/2800161510 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2600cd6d0 con 0x55bb36656f30 2024-02-19T15:23:53.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa266800640 1 --2- 172.21.15.47:0/2800161510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb366a1240 0x55bb366ea420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:53.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa265e00640 1 --2- 172.21.15.47:0/2800161510 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb36656f30 0x55bb366e95b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa2540097f0 tx=0x7fa25401ab80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb8b6eb58ff7ae87 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:53.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa243400640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:23:53.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa243400640 1 -- 172.21.15.47:0/2800161510 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2540c3de0 con 0x55bb36656f30 2024-02-19T15:23:53.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa243400640 1 -- 172.21.15.47:0/2800161510 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2540ceb30 con 0x55bb36656f30 2024-02-19T15:23:53.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 -- 172.21.15.47:0/2800161510 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bb366b69e0 con 0x55bb36656f30 2024-02-19T15:23:53.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa243400640 1 -- 172.21.15.47:0/2800161510 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2540cecf0 con 0x55bb36656f30 2024-02-19T15:23:53.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 10 client.5691 fetch_fsmap learned FSMap version 25 2024-02-19T15:23:53.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 -- 172.21.15.47:0/2800161510 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x55bb3619fcc0 con 0x55bb36656f30 2024-02-19T15:23:53.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa243400640 1 -- 172.21.15.47:0/2800161510 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bb366b69e0 con 0x55bb36656f30 2024-02-19T15:23:53.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa243400640 1 -- 172.21.15.47:0/2800161510 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 25) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa2540d4020 con 0x55bb36656f30 2024-02-19T15:23:53.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 10 client.5691 fetch_fsmap finished waiting for FSMap version 25 2024-02-19T15:23:53.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 10 client.5691 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-19T15:23:53.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 20 client.5691 populate_metadata read hostname 'smithi047' 2024-02-19T15:23:53.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 --2- 172.21.15.47:0/2800161510 >> [v2:172.21.15.47:6833/3137466738,v1:172.21.15.47:6835/3137466738] conn(0x55bb366ea960 0x55bb366ead40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 4 client.5691 mds_command: new command op to 5514 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T15:23:53.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 -- 172.21.15.47:0/2800161510 --> [v2:172.21.15.47:6833/3137466738,v1:172.21.15.47:6835/3137466738] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bb3642ee80 con 0x55bb366ea960 2024-02-19T15:23:53.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa265400640 1 --2- 172.21.15.47:0/2800161510 >> [v2:172.21.15.47:6833/3137466738,v1:172.21.15.47:6835/3137466738] conn(0x55bb366ea960 0x55bb366ead40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa243400640 10 client.5691 ms_handle_connect on v2:172.21.15.47:6833/3137466738 2024-02-19T15:23:53.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa265400640 1 --2- 172.21.15.47:0/2800161510 >> [v2:172.21.15.47:6833/3137466738,v1:172.21.15.47:6835/3137466738] conn(0x55bb366ea960 0x55bb366ead40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d94bb2c48c7575d server_cookie=ddf4f24b00cdded7 in_seq=0 out_seq=0 2024-02-19T15:23:53.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa243400640 1 -- 172.21.15.47:0/2800161510 <== mds.0 v2:172.21.15.47:6833/3137466738 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x55bb3642ee80 con 0x55bb366ea960 2024-02-19T15:23:53.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa243400640 10 client.5691 handle_command_reply: tid=0 2024-02-19T15:23:53.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 client.5691 shutdown 2024-02-19T15:23:53.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 -- 172.21.15.47:0/2800161510 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb36656f30 msgr2=0x55bb366e95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:53.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 --2- 172.21.15.47:0/2800161510 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb36656f30 0x55bb366e95b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa2540097f0 tx=0x7fa25401ab80 comp rx=0 tx=0).stop 2024-02-19T15:23:53.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 -- 172.21.15.47:0/2800161510 shutdown_connections 2024-02-19T15:23:53.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 --2- 172.21.15.47:0/2800161510 >> [v2:172.21.15.47:6833/3137466738,v1:172.21.15.47:6835/3137466738] conn(0x55bb366ea960 0x55bb366ead40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 --2- 172.21.15.47:0/2800161510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb366a1240 0x55bb366ea420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 --2- 172.21.15.47:0/2800161510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55bb366a0920 0x55bb366e9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa243400640 0 client.5691 ms_handle_reset on v2:172.21.15.47:6833/3137466738 2024-02-19T15:23:53.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 --2- 172.21.15.47:0/2800161510 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb36656f30 0x55bb366e95b0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 -- 172.21.15.47:0/2800161510 >> 172.21.15.47:0/2800161510 conn(0x55bb3665d2a0 msgr2=0x55bb36655960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:53.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 -- 172.21.15.47:0/2800161510 shutdown_connections 2024-02-19T15:23:53.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 1 -- 172.21.15.47:0/2800161510 wait complete. 2024-02-19T15:23:53.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.671+0000 7fa26bc35000 20 client.5691 trim_cache size 0 max 16384 2024-02-19T15:23:53.683 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:23:53.683 INFO:teuthology.orchestra.run.smithi047.stdout: "cluster_fsid": "ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e", 2024-02-19T15:23:53.683 INFO:teuthology.orchestra.run.smithi047.stdout: "whoami": 0, 2024-02-19T15:23:53.683 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 5514, 2024-02-19T15:23:53.683 INFO:teuthology.orchestra.run.smithi047.stdout: "want_state": "up:active", 2024-02-19T15:23:53.683 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "up:active", 2024-02-19T15:23:53.683 INFO:teuthology.orchestra.run.smithi047.stdout: "fs_name": "cephfs", 2024-02-19T15:23:53.683 INFO:teuthology.orchestra.run.smithi047.stdout: "rank_uptime": 10.736129369, 2024-02-19T15:23:53.683 INFO:teuthology.orchestra.run.smithi047.stdout: "mdsmap_epoch": 25, 2024-02-19T15:23:53.683 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch": 45, 2024-02-19T15:23:53.683 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch_barrier": 45, 2024-02-19T15:23:53.683 INFO:teuthology.orchestra.run.smithi047.stdout: "uptime": 15.603515413 2024-02-19T15:23:53.683 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:23:53.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.675+0000 7fa268800640 1 -- 172.21.15.47:0/3517668284 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa244066e40 msgr2=0x7fa244069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:53.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.675+0000 7fa268800640 1 --2- 172.21.15.47:0/3517668284 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa244066e40 0x7fa244069300 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa2540ad000 tx=0x7fa2540a6000 comp rx=0 tx=0).stop 2024-02-19T15:23:53.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.675+0000 7fa268800640 1 -- 172.21.15.47:0/3517668284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2600cded0 msgr2=0x7fa26015e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:53.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.675+0000 7fa268800640 1 --2- 172.21.15.47:0/3517668284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2600cded0 0x7fa26015e3b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa25000c430 tx=0x7fa25000c8f0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.675+0000 7fa268800640 1 -- 172.21.15.47:0/3517668284 shutdown_connections 2024-02-19T15:23:53.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.675+0000 7fa268800640 1 --2- 172.21.15.47:0/3517668284 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa244066e40 0x7fa244069300 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.675+0000 7fa268800640 1 --2- 172.21.15.47:0/3517668284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2600ced10 0x7fa2600c5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.675+0000 7fa268800640 1 --2- 172.21.15.47:0/3517668284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2600cded0 0x7fa26015e3b0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.675+0000 7fa268800640 1 --2- 172.21.15.47:0/3517668284 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa26006d650 0x7fa26015b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.675+0000 7fa268800640 1 -- 172.21.15.47:0/3517668284 >> 172.21.15.47:0/3517668284 conn(0x7fa2600d1bf0 msgr2=0x7fa2600b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:53.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.675+0000 7fa268800640 1 -- 172.21.15.47:0/3517668284 shutdown_connections 2024-02-19T15:23:53.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.675+0000 7fa268800640 1 -- 172.21.15.47:0/3517668284 wait complete. 2024-02-19T15:23:53.698 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-02-19T15:23:53.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.831+0000 7f63c3e00640 1 Processor -- start 2024-02-19T15:23:53.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.831+0000 7f63c3e00640 1 -- start start 2024-02-19T15:23:53.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.835+0000 7f63c3e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f63bc15df50 0x7f63bc0d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.835+0000 7f63c3e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63bc0d4a70 0x7f63bc0d0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.835+0000 7f63c3e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63bc0d2da0 con 0x7f63bc15e330 2024-02-19T15:23:53.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.835+0000 7f63c3e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63bc0b6df0 con 0x7f63bc15df50 2024-02-19T15:23:53.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.835+0000 7f63c3e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc0b70d0 con 0x7f63bc0d4a70 2024-02-19T15:23:53.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.835+0000 7f63c1400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f63bc15df50 0x7f63bc0d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.835+0000 7f63c0a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63bc0d4a70 0x7f63bc0d0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.835+0000 7f63c1400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f63bc15df50 0x7f63bc0d2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:57562/0 (socket says 172.21.15.47:57562) 2024-02-19T15:23:53.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.835+0000 7f63c1400640 1 -- 172.21.15.47:0/870460430 learned_addr learned my addr 172.21.15.47:0/870460430 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:53.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.835+0000 7f63c0a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63bc0d4a70 0x7f63bc0d0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:47442/0 (socket says 172.21.15.47:47442) 2024-02-19T15:23:53.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.835+0000 7f63c1e00640 1 --1- 172.21.15.47:0/870460430 >> v1:172.21.15.116:6789/0 conn(0x7f63bc15e330 0x7f63bc0ce980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:39172/0 (socket says 172.21.15.47:39172) 2024-02-19T15:23:53.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.835+0000 7f63c1400640 1 -- 172.21.15.47:0/870460430 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63bc0d4a70 msgr2=0x7f63bc0d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:53.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.835+0000 7f63c1400640 1 --2- 172.21.15.47:0/870460430 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63bc0d4a70 0x7f63bc0d0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.835+0000 7f63c1400640 1 -- 172.21.15.47:0/870460430 >> v1:172.21.15.116:6789/0 conn(0x7f63bc15e330 legacy=0x7f63bc0ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:53.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.835+0000 7f63c1400640 1 -- 172.21.15.47:0/870460430 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc0b73b0 con 0x7f63bc15df50 2024-02-19T15:23:53.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.835+0000 7f63bbe00640 1 -- 172.21.15.47:0/870460430 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575934775 0 0) 0x7f63bc0d2da0 con 0x7f63bc15e330 2024-02-19T15:23:53.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.839+0000 7f63c1400640 1 --2- 172.21.15.47:0/870460430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63bc15df50 0x7f63bc0d2450 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f63b0009870 tx=0x7f63b001c530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eec1ab511ea9bf74 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:53.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.839+0000 7f63bbe00640 1 -- 172.21.15.47:0/870460430 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b00ab640 con 0x7f63bc15df50 2024-02-19T15:23:53.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.839+0000 7f63bbe00640 1 -- 172.21.15.47:0/870460430 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b00a7070 con 0x7f63bc15df50 2024-02-19T15:23:53.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.839+0000 7f63bbe00640 1 -- 172.21.15.47:0/870460430 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b00ab9c0 con 0x7f63bc15df50 2024-02-19T15:23:53.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.839+0000 7f63c3e00640 1 -- 172.21.15.47:0/870460430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63bc15df50 msgr2=0x7f63bc0d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:53.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.839+0000 7f63c3e00640 1 --2- 172.21.15.47:0/870460430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63bc15df50 0x7f63bc0d2450 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f63b0009870 tx=0x7f63b001c530 comp rx=0 tx=0).stop 2024-02-19T15:23:53.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.839+0000 7f63c3e00640 1 -- 172.21.15.47:0/870460430 shutdown_connections 2024-02-19T15:23:53.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.839+0000 7f63c3e00640 1 --2- 172.21.15.47:0/870460430 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63bc0d4a70 0x7f63bc0d0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.839+0000 7f63c3e00640 1 --2- 172.21.15.47:0/870460430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63bc15df50 0x7f63bc0d2450 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.839+0000 7f63c3e00640 1 -- 172.21.15.47:0/870460430 >> 172.21.15.47:0/870460430 conn(0x7f63bc0d3e70 msgr2=0x7f63bc0d4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:53.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.839+0000 7f63c3e00640 1 -- 172.21.15.47:0/870460430 shutdown_connections 2024-02-19T15:23:53.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.839+0000 7f63c3e00640 1 -- 172.21.15.47:0/870460430 wait complete. 2024-02-19T15:23:53.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.839+0000 7f63c3e00640 1 Processor -- start 2024-02-19T15:23:53.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c3e00640 1 -- start start 2024-02-19T15:23:53.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63bc0d4a70 0x7f63bc0cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c3e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63bc15df50 0x7f63bc151ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63bc15e330 0x7f63bc14b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c3e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc15dba0 con 0x7f63bc0d4a70 2024-02-19T15:23:53.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c3e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc14c540 con 0x7f63bc15e330 2024-02-19T15:23:53.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c3e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc15b5d0 con 0x7f63bc15df50 2024-02-19T15:23:53.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c1400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63bc0d4a70 0x7f63bc0cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c1400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63bc0d4a70 0x7f63bc0cce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39970/0 (socket says 172.21.15.47:39970) 2024-02-19T15:23:53.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c1400640 1 -- 172.21.15.47:0/2484566693 learned_addr learned my addr 172.21.15.47:0/2484566693 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:53.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c0a00640 1 --2- 172.21.15.47:0/2484566693 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63bc15df50 0x7f63bc151ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c1400640 1 -- 172.21.15.47:0/2484566693 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63bc15df50 msgr2=0x7f63bc151ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:53.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c1400640 1 --2- 172.21.15.47:0/2484566693 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63bc15df50 0x7f63bc151ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c1400640 1 -- 172.21.15.47:0/2484566693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63bc15e330 msgr2=0x7f63bc14b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:53.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c1e00640 1 --2- 172.21.15.47:0/2484566693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63bc15e330 0x7f63bc14b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c1400640 1 --2- 172.21.15.47:0/2484566693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63bc15e330 0x7f63bc14b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:53.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c1400640 1 -- 172.21.15.47:0/2484566693 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b00094d0 con 0x7f63bc0d4a70 2024-02-19T15:23:53.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c0a00640 1 --2- 172.21.15.47:0/2484566693 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63bc15df50 0x7f63bc151ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:23:53.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c1e00640 1 --2- 172.21.15.47:0/2484566693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63bc15e330 0x7f63bc14b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:23:53.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c1400640 1 --2- 172.21.15.47:0/2484566693 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63bc0d4a70 0x7f63bc0cce90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f63b00ad040 tx=0x7f63b001ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:53.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63b9600640 1 -- 172.21.15.47:0/2484566693 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b00abe80 con 0x7f63bc0d4a70 2024-02-19T15:23:53.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63b9600640 1 -- 172.21.15.47:0/2484566693 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b00be3f0 con 0x7f63bc0d4a70 2024-02-19T15:23:53.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63b9600640 1 -- 172.21.15.47:0/2484566693 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b00be7a0 con 0x7f63bc0d4a70 2024-02-19T15:23:53.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c3e00640 1 -- 172.21.15.47:0/2484566693 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63bc0cd470 con 0x7f63bc0d4a70 2024-02-19T15:23:53.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c3e00640 1 -- 172.21.15.47:0/2484566693 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63bc0cda30 con 0x7f63bc0d4a70 2024-02-19T15:23:53.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.843+0000 7f63c3e00640 1 -- 172.21.15.47:0/2484566693 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63bc075360 con 0x7f63bc0d4a70 2024-02-19T15:23:53.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.847+0000 7f63b9600640 1 -- 172.21.15.47:0/2484566693 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f63b00b43f0 con 0x7f63bc0d4a70 2024-02-19T15:23:53.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.847+0000 7f63b9600640 1 --2- 172.21.15.47:0/2484566693 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f639c066e70 0x7f639c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:53.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.847+0000 7f63b9600640 1 -- 172.21.15.47:0/2484566693 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63b011ed90 con 0x7f63bc0d4a70 2024-02-19T15:23:53.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.847+0000 7f63c0a00640 1 --2- 172.21.15.47:0/2484566693 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f639c066e70 0x7f639c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:53.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.847+0000 7f63c0a00640 1 --2- 172.21.15.47:0/2484566693 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f639c066e70 0x7f639c069330 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f63bc15b270 tx=0x7f63a8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:53.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:53.855+0000 7f63b9600640 1 -- 172.21.15.47:0/2484566693 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f63b00ebcc0 con 0x7f63bc0d4a70 2024-02-19T15:23:54.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.055+0000 7f63c3e00640 1 -- 172.21.15.47:0/2484566693 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f63bc0b5020 con 0x7f63bc0d4a70 2024-02-19T15:23:54.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.055+0000 7f63b9600640 1 -- 172.21.15.47:0/2484566693 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7f63b00c4070 con 0x7f63bc0d4a70 2024-02-19T15:23:54.061 INFO:teuthology.orchestra.run.smithi047.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-02-19T15:23:54.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.059+0000 7f63c3e00640 1 -- 172.21.15.47:0/2484566693 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f639c066e70 msgr2=0x7f639c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:54.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.059+0000 7f63c3e00640 1 --2- 172.21.15.47:0/2484566693 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f639c066e70 0x7f639c069330 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f63bc15b270 tx=0x7f63a8008040 comp rx=0 tx=0).stop 2024-02-19T15:23:54.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.059+0000 7f63c3e00640 1 -- 172.21.15.47:0/2484566693 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63bc0d4a70 msgr2=0x7f63bc0cce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:54.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.059+0000 7f63c3e00640 1 --2- 172.21.15.47:0/2484566693 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63bc0d4a70 0x7f63bc0cce90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f63b00ad040 tx=0x7f63b001ad70 comp rx=0 tx=0).stop 2024-02-19T15:23:54.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.059+0000 7f63c3e00640 1 -- 172.21.15.47:0/2484566693 shutdown_connections 2024-02-19T15:23:54.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.059+0000 7f63c3e00640 1 --2- 172.21.15.47:0/2484566693 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f639c066e70 0x7f639c069330 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:54.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.059+0000 7f63c3e00640 1 --2- 172.21.15.47:0/2484566693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63bc15e330 0x7f63bc14b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:54.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.059+0000 7f63c3e00640 1 --2- 172.21.15.47:0/2484566693 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63bc15df50 0x7f63bc151ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:54.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.059+0000 7f63c3e00640 1 --2- 172.21.15.47:0/2484566693 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63bc0d4a70 0x7f63bc0cce90 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:54.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.059+0000 7f63c3e00640 1 -- 172.21.15.47:0/2484566693 >> 172.21.15.47:0/2484566693 conn(0x7f63bc0d3e70 msgr2=0x7f63bc0d4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:54.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.059+0000 7f63c3e00640 1 -- 172.21.15.47:0/2484566693 shutdown_connections 2024-02-19T15:23:54.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.059+0000 7f63c3e00640 1 -- 172.21.15.47:0/2484566693 wait complete. 2024-02-19T15:23:54.086 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-02-19T15:23:54.086 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T15:23:54.087 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi047.front.sepia.ceph.com 2024-02-19T15:23:54.087 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-19T15:23:54.087 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-19T15:23:54.087 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-02-19T15:23:54.087 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-02-19T15:23:54.087 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T15:23:54.087 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:23:54.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:54.087 DEBUG:teuthology.orchestra.run.smithi047:> ip addr 2024-02-19T15:23:54.111 INFO:teuthology.orchestra.run.smithi047.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T15:23:54.111 INFO:teuthology.orchestra.run.smithi047.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T15:23:54.112 INFO:teuthology.orchestra.run.smithi047.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T15:23:54.112 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:23:54.112 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 ::1/128 scope host 2024-02-19T15:23:54.112 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:23:54.112 INFO:teuthology.orchestra.run.smithi047.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:23:54.112 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:78 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:23:54.112 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f0 2024-02-19T15:23:54.112 INFO:teuthology.orchestra.run.smithi047.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T15:23:54.112 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8e brd ff:ff:ff:ff:ff:ff 2024-02-19T15:23:54.112 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f0 2024-02-19T15:23:54.112 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.21.15.47/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-02-19T15:23:54.112 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft 42381sec preferred_lft 42381sec 2024-02-19T15:23:54.112 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::ec4:7aff:febd:148e/64 scope link 2024-02-19T15:23:54.113 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:23:54.113 INFO:teuthology.orchestra.run.smithi047.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:23:54.113 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:79 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:23:54.113 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f1 2024-02-19T15:23:54.113 INFO:teuthology.orchestra.run.smithi047.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:23:54.113 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8f brd ff:ff:ff:ff:ff:ff 2024-02-19T15:23:54.113 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f1 2024-02-19T15:23:54.113 INFO:teuthology.orchestra.run.smithi047.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-02-19T15:23:54.113 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 02:42:47:cf:b1:62 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:23:54.113 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-02-19T15:23:54.113 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:23:54.113 INFO:teuthology.orchestra.run.smithi047.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-02-19T15:23:54.113 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-02-19T15:23:54.113 INFO:teuthology.orchestra.run.smithi047.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-02-19T15:23:54.113 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:23:54.114 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-02-19T15:23:54.114 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:23:54.114 INFO:teuthology.orchestra.run.smithi047.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:23:54.114 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-19T15:23:54.114 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-02-19T15:23:54.114 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:23:54.114 INFO:teuthology.orchestra.run.smithi047.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:23:54.114 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-02-19T15:23:54.114 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-02-19T15:23:54.114 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:23:54.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:54.115 DEBUG:teuthology.orchestra.run.smithi047:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T15:23:54.172 INFO:teuthology.orchestra.run.smithi047.stdout:1 2024-02-19T15:23:54.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:54.175 DEBUG:teuthology.orchestra.run.smithi047:> route 2024-02-19T15:23:54.229 INFO:teuthology.orchestra.run.smithi047.stdout:Kernel IP routing table 2024-02-19T15:23:54.229 INFO:teuthology.orchestra.run.smithi047.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T15:23:54.229 INFO:teuthology.orchestra.run.smithi047.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-02-19T15:23:54.230 INFO:teuthology.orchestra.run.smithi047.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-02-19T15:23:54.230 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-02-19T15:23:54.230 INFO:teuthology.orchestra.run.smithi047.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T15:23:54.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:54.230 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2024-02-19T15:23:54.230 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2024-02-19T15:23:54.230 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-02-19T15:23:54.230 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-02-19T15:23:54.231 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-02-19T15:23:54.231 DEBUG:teuthology.orchestra.run.smithi047:> ') 2024-02-19T15:23:54.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:54.327 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list 2024-02-19T15:23:54.384 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-02-19T15:23:54.384 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-02-19T15:23:54.385 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:23:54.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:54.385 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set brx.0 up 2024-02-19T15:23:54.450 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:23:54.450 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-19T15:23:54.503 INFO:teuthology.orchestra.run.smithi047.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:23:54.503 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:23:54.503 DEBUG:teuthology.orchestra.run.smithi047:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:23:54.557 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-19T15:23:54.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:54.557 DEBUG:teuthology.orchestra.run.smithi047:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:23:54.648 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:23:54.649 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:23:54.649 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:23:54.664 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:23:54.664 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:23:54.664 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:23:54.664 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:23:54.664 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:23:54.673 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:23:54.673 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:23:54.673 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:23:54.673 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:23:54.673 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:23:54.673 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790" 2024-02-19T15:23:54.674 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790:/ 2024-02-19T15:23:54.674 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e" will pass to kernel 2024-02-19T15:23:54.677 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T15:23:54.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:54.677 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:23:54.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:23:54.697 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:23:54.759 INFO:teuthology.orchestra.run.smithi047.stdout:client5255 2024-02-19T15:23:54.759 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:23:54.760 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:23:54.815 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-02-19T15:23:55.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.999+0000 7f22cd600640 1 Processor -- start 2024-02-19T15:23:55.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.999+0000 7f22cd600640 1 -- start start 2024-02-19T15:23:55.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.999+0000 7f22cd600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f22c80ce3b0 0x7f22c80ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:55.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.999+0000 7f22cd600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22c815e280 0x7f22c80cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:55.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.999+0000 7f22cd600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c80cc620 con 0x7f22c80d2650 2024-02-19T15:23:55.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.999+0000 7f22cd600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22c80d12e0 con 0x7f22c80ce3b0 2024-02-19T15:23:55.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.999+0000 7f22cd600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22c80d1940 con 0x7f22c815e280 2024-02-19T15:23:55.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.999+0000 7f22c6a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f22c80ce3b0 0x7f22c80ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:55.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.999+0000 7f22c6a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f22c80ce3b0 0x7f22c80ce790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39978/0 (socket says 172.21.15.47:39978) 2024-02-19T15:23:55.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.999+0000 7f22c6a00640 1 -- 172.21.15.47:0/3660193737 learned_addr learned my addr 172.21.15.47:0/3660193737 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:55.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.999+0000 7f22c7400640 1 --2- 172.21.15.47:0/3660193737 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22c815e280 0x7f22c80cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:55.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.999+0000 7f22c6a00640 1 -- 172.21.15.47:0/3660193737 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22c815e280 msgr2=0x7f22c80cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:55.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.999+0000 7f22c6a00640 1 --2- 172.21.15.47:0/3660193737 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22c815e280 0x7f22c80cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:55.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.999+0000 7f22c6a00640 1 -- 172.21.15.47:0/3660193737 >> v1:172.21.15.116:6789/0 conn(0x7f22c80d2650 legacy=0x7f22c80ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:23:55.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.999+0000 7f22c6a00640 1 -- 172.21.15.47:0/3660193737 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c80cecd0 con 0x7f22c80ce3b0 2024-02-19T15:23:55.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:54.999+0000 7f22c7400640 1 --2- 172.21.15.47:0/3660193737 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22c815e280 0x7f22c80cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:23:55.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.003+0000 7f22c6a00640 1 --2- 172.21.15.47:0/3660193737 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f22c80ce3b0 0x7f22c80ce790 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f22b8009870 tx=0x7f22b801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=100bc68daa021f83 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:55.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.003+0000 7f22c6000640 1 -- 172.21.15.47:0/3660193737 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b80ab600 con 0x7f22c80ce3b0 2024-02-19T15:23:55.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.003+0000 7f22c6000640 1 -- 172.21.15.47:0/3660193737 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22b80a7050 con 0x7f22c80ce3b0 2024-02-19T15:23:55.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.003+0000 7f22c6000640 1 -- 172.21.15.47:0/3660193737 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b80ab980 con 0x7f22c80ce3b0 2024-02-19T15:23:55.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.003+0000 7f22cd600640 1 -- 172.21.15.47:0/3660193737 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f22c80ce3b0 msgr2=0x7f22c80ce790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:55.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.003+0000 7f22cd600640 1 --2- 172.21.15.47:0/3660193737 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f22c80ce3b0 0x7f22c80ce790 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f22b8009870 tx=0x7f22b801c400 comp rx=0 tx=0).stop 2024-02-19T15:23:55.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.003+0000 7f22cd600640 1 -- 172.21.15.47:0/3660193737 shutdown_connections 2024-02-19T15:23:55.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.003+0000 7f22cd600640 1 --2- 172.21.15.47:0/3660193737 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22c815e280 0x7f22c80cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:55.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.003+0000 7f22cd600640 1 --2- 172.21.15.47:0/3660193737 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f22c80ce3b0 0x7f22c80ce790 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:55.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.003+0000 7f22cd600640 1 -- 172.21.15.47:0/3660193737 >> 172.21.15.47:0/3660193737 conn(0x7f22c80d1b30 msgr2=0x7f22c80d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:55.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.003+0000 7f22cd600640 1 -- 172.21.15.47:0/3660193737 shutdown_connections 2024-02-19T15:23:55.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.003+0000 7f22cd600640 1 -- 172.21.15.47:0/3660193737 wait complete. 2024-02-19T15:23:55.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22cd600640 1 Processor -- start 2024-02-19T15:23:55.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22cd600640 1 -- start start 2024-02-19T15:23:55.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22cd600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f22c80ce3b0 0x7f22c8150f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:55.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22cd600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22c80d2650 0x7f22c8159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:55.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22cd600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22c815e280 0x7f22c81560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:55.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22cd600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c8159120 con 0x7f22c80ce3b0 2024-02-19T15:23:55.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22cd600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c81584a0 con 0x7f22c80d2650 2024-02-19T15:23:55.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22cd600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22c8158780 con 0x7f22c815e280 2024-02-19T15:23:55.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22c6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f22c80ce3b0 0x7f22c8150f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:55.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22c6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f22c80ce3b0 0x7f22c8150f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39994/0 (socket says 172.21.15.47:39994) 2024-02-19T15:23:55.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22c6a00640 1 -- 172.21.15.47:0/925387364 learned_addr learned my addr 172.21.15.47:0/925387364 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:23:55.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22bfe00640 1 --2- 172.21.15.47:0/925387364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22c80d2650 0x7f22c8159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:55.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22c7400640 1 --2- 172.21.15.47:0/925387364 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22c815e280 0x7f22c81560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:55.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22c6a00640 1 -- 172.21.15.47:0/925387364 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22c815e280 msgr2=0x7f22c81560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:55.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22c6a00640 1 --2- 172.21.15.47:0/925387364 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22c815e280 0x7f22c81560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:55.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22c6a00640 1 -- 172.21.15.47:0/925387364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22c80d2650 msgr2=0x7f22c8159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:55.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22c6a00640 1 --2- 172.21.15.47:0/925387364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22c80d2650 0x7f22c8159540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:55.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22c6a00640 1 -- 172.21.15.47:0/925387364 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b80094d0 con 0x7f22c80ce3b0 2024-02-19T15:23:55.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22c7400640 1 --2- 172.21.15.47:0/925387364 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22c815e280 0x7f22c81560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:23:55.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22c6a00640 1 --2- 172.21.15.47:0/925387364 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f22c80ce3b0 0x7f22c8150f20 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f22b80ae040 tx=0x7f22b80191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:55.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22bea00640 1 -- 172.21.15.47:0/925387364 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b80b8070 con 0x7f22c80ce3b0 2024-02-19T15:23:55.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22bea00640 1 -- 172.21.15.47:0/925387364 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22b80b3690 con 0x7f22c80ce3b0 2024-02-19T15:23:55.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22cd600640 1 -- 172.21.15.47:0/925387364 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22c8158a60 con 0x7f22c80ce3b0 2024-02-19T15:23:55.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22bea00640 1 -- 172.21.15.47:0/925387364 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b80b3a60 con 0x7f22c80ce3b0 2024-02-19T15:23:55.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.007+0000 7f22cd600640 1 -- 172.21.15.47:0/925387364 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22c8151460 con 0x7f22c80ce3b0 2024-02-19T15:23:55.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.011+0000 7f22cd600640 1 -- 172.21.15.47:0/925387364 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22c8075360 con 0x7f22c80ce3b0 2024-02-19T15:23:55.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.011+0000 7f22bea00640 1 -- 172.21.15.47:0/925387364 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f22b80b2070 con 0x7f22c80ce3b0 2024-02-19T15:23:55.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.011+0000 7f22bea00640 1 --2- 172.21.15.47:0/925387364 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f22a4066e40 0x7f22a4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:23:55.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.011+0000 7f22bea00640 1 -- 172.21.15.47:0/925387364 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f22b811e4b0 con 0x7f22c80ce3b0 2024-02-19T15:23:55.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.011+0000 7f22bfe00640 1 --2- 172.21.15.47:0/925387364 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f22a4066e40 0x7f22a4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:23:55.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.011+0000 7f22bfe00640 1 --2- 172.21.15.47:0/925387364 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f22a4066e40 0x7f22a4069300 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f22c809fab0 tx=0x7f22b0009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:23:55.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.019+0000 7f22bea00640 1 -- 172.21.15.47:0/925387364 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f22b80eb200 con 0x7f22c80ce3b0 2024-02-19T15:23:55.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.187+0000 7f22cd600640 1 -- 172.21.15.47:0/925387364 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f22c80cdf80 con 0x7f22c80ce3b0 2024-02-19T15:23:55.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.191+0000 7f22bea00640 1 -- 172.21.15.47:0/925387364 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v13)=0 v13) v1 ==== 124+0+0 (secure 0 0 0) 0x7f22b80ee8c0 con 0x7f22c80ce3b0 2024-02-19T15:23:55.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.191+0000 7f22cd600640 1 -- 172.21.15.47:0/925387364 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f22a4066e40 msgr2=0x7f22a4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:55.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.191+0000 7f22cd600640 1 --2- 172.21.15.47:0/925387364 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f22a4066e40 0x7f22a4069300 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f22c809fab0 tx=0x7f22b0009060 comp rx=0 tx=0).stop 2024-02-19T15:23:55.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.195+0000 7f22cd600640 1 -- 172.21.15.47:0/925387364 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f22c80ce3b0 msgr2=0x7f22c8150f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:23:55.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.195+0000 7f22cd600640 1 --2- 172.21.15.47:0/925387364 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f22c80ce3b0 0x7f22c8150f20 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f22b80ae040 tx=0x7f22b80191a0 comp rx=0 tx=0).stop 2024-02-19T15:23:55.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.195+0000 7f22cd600640 1 -- 172.21.15.47:0/925387364 shutdown_connections 2024-02-19T15:23:55.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.195+0000 7f22cd600640 1 --2- 172.21.15.47:0/925387364 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f22a4066e40 0x7f22a4069300 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:55.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.195+0000 7f22cd600640 1 --2- 172.21.15.47:0/925387364 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22c815e280 0x7f22c81560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:55.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.195+0000 7f22cd600640 1 --2- 172.21.15.47:0/925387364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22c80d2650 0x7f22c8159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:55.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.195+0000 7f22cd600640 1 --2- 172.21.15.47:0/925387364 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f22c80ce3b0 0x7f22c8150f20 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:23:55.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.195+0000 7f22cd600640 1 -- 172.21.15.47:0/925387364 >> 172.21.15.47:0/925387364 conn(0x7f22c80d1b30 msgr2=0x7f22c80ccab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:23:55.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.195+0000 7f22cd600640 1 -- 172.21.15.47:0/925387364 shutdown_connections 2024-02-19T15:23:55.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:23:55.195+0000 7f22cd600640 1 -- 172.21.15.47:0/925387364 wait complete. 2024-02-19T15:24:05.229 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:24:05.229 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:05.238 DEBUG:teuthology.orchestra.run.smithi047:> stdin-killer -- python3 -c ' 2024-02-19T15:24:05.238 DEBUG:teuthology.orchestra.run.smithi047:> import os 2024-02-19T15:24:05.238 DEBUG:teuthology.orchestra.run.smithi047:> import uuid 2024-02-19T15:24:05.238 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:24:05.238 DEBUG:teuthology.orchestra.run.smithi047:> n = 51 2024-02-19T15:24:05.238 DEBUG:teuthology.orchestra.run.smithi047:> create_hard_links = False 2024-02-19T15:24:05.238 DEBUG:teuthology.orchestra.run.smithi047:> if 0 > 0: 2024-02-19T15:24:05.238 DEBUG:teuthology.orchestra.run.smithi047:> create_hard_links = True 2024-02-19T15:24:05.238 DEBUG:teuthology.orchestra.run.smithi047:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-02-19T15:24:05.238 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:24:05.238 DEBUG:teuthology.orchestra.run.smithi047:> dpath = os.path.dirname(path) 2024-02-19T15:24:05.239 DEBUG:teuthology.orchestra.run.smithi047:> fnameprefix = os.path.basename(path) 2024-02-19T15:24:05.239 DEBUG:teuthology.orchestra.run.smithi047:> os.makedirs(dpath, exist_ok=True) 2024-02-19T15:24:05.239 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:24:05.239 DEBUG:teuthology.orchestra.run.smithi047:> try: 2024-02-19T15:24:05.239 DEBUG:teuthology.orchestra.run.smithi047:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-02-19T15:24:05.239 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:24:05.239 DEBUG:teuthology.orchestra.run.smithi047:> for i in range(n): 2024-02-19T15:24:05.239 DEBUG:teuthology.orchestra.run.smithi047:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-02-19T15:24:05.239 DEBUG:teuthology.orchestra.run.smithi047:> with open(fpath, '"'"'w'"'"') as f: 2024-02-19T15:24:05.239 DEBUG:teuthology.orchestra.run.smithi047:> f.write(f"{i}") 2024-02-19T15:24:05.239 DEBUG:teuthology.orchestra.run.smithi047:> if False: 2024-02-19T15:24:05.239 DEBUG:teuthology.orchestra.run.smithi047:> f.flush() 2024-02-19T15:24:05.239 DEBUG:teuthology.orchestra.run.smithi047:> os.fsync(f.fileno()) 2024-02-19T15:24:05.239 DEBUG:teuthology.orchestra.run.smithi047:> if False: 2024-02-19T15:24:05.239 DEBUG:teuthology.orchestra.run.smithi047:> os.unlink(fpath) 2024-02-19T15:24:05.239 DEBUG:teuthology.orchestra.run.smithi047:> if False: 2024-02-19T15:24:05.239 DEBUG:teuthology.orchestra.run.smithi047:> os.fsync(dirfd) 2024-02-19T15:24:05.240 DEBUG:teuthology.orchestra.run.smithi047:> if create_hard_links: 2024-02-19T15:24:05.240 DEBUG:teuthology.orchestra.run.smithi047:> for j in range(0): 2024-02-19T15:24:05.240 DEBUG:teuthology.orchestra.run.smithi047:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-02-19T15:24:05.240 DEBUG:teuthology.orchestra.run.smithi047:> if True: 2024-02-19T15:24:05.240 DEBUG:teuthology.orchestra.run.smithi047:> os.fsync(dirfd) 2024-02-19T15:24:05.240 DEBUG:teuthology.orchestra.run.smithi047:> finally: 2024-02-19T15:24:05.240 DEBUG:teuthology.orchestra.run.smithi047:> os.close(dirfd) 2024-02-19T15:24:05.240 DEBUG:teuthology.orchestra.run.smithi047:> ' 2024-02-19T15:24:05.370 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:24:05.373 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:24:05.395 INFO:teuthology.orchestra.run.smithi047.stderr:Traceback (most recent call last): 2024-02-19T15:24:05.395 INFO:teuthology.orchestra.run.smithi047.stderr: File "", line 33, in 2024-02-19T15:24:05.395 INFO:teuthology.orchestra.run.smithi047.stderr:OSError: [Errno 28] No space left on device 2024-02-19T15:24:05.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-02-19T15:24:05.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:24:05.407 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T15:24:05.408 DEBUG:teuthology.orchestra.run.smithi047:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:24:05.549 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:24:05.645 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T15:24:05.645 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:24:05.646 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:05.654 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:24:05.654 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:05.702 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:24:05.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.878+0000 7ff15c200640 1 Processor -- start 2024-02-19T15:24:05.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.878+0000 7ff15c200640 1 -- start start 2024-02-19T15:24:05.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.878+0000 7ff15c200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1540ce3c0 0x7ff1540ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:05.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.878+0000 7ff15c200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1540d2ad0 con 0x7ff1540cd630 2024-02-19T15:24:05.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.878+0000 7ff15c200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1540cc190 con 0x7ff15415e570 2024-02-19T15:24:05.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.878+0000 7ff15c200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff15406d110 con 0x7ff1540ce3c0 2024-02-19T15:24:05.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.878+0000 7ff15a200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff1540cd630 0x7ff15406ca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51966/0 (socket says 172.21.15.47:51966) 2024-02-19T15:24:05.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.878+0000 7ff15a200640 1 -- 172.21.15.47:0/3626520925 learned_addr learned my addr 172.21.15.47:0/3626520925 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:05.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.878+0000 7ff159800640 1 --2- 172.21.15.47:0/3626520925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1540ce3c0 0x7ff1540ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:05.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.878+0000 7ff153e00640 1 -- 172.21.15.47:0/3626520925 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4179892620 0 0) 0x7ff14c0014a0 con 0x7ff1540cd630 2024-02-19T15:24:05.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.878+0000 7ff153e00640 1 -- 172.21.15.47:0/3626520925 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff13c003530 con 0x7ff1540cd630 2024-02-19T15:24:05.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff153e00640 1 -- 172.21.15.47:0/3626520925 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2279712259 0 0) 0x7ff13c003530 con 0x7ff1540cd630 2024-02-19T15:24:05.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff153e00640 1 -- 172.21.15.47:0/3626520925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1540ce3c0 msgr2=0x7ff1540ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:05.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff153e00640 1 --2- 172.21.15.47:0/3626520925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1540ce3c0 0x7ff1540ce7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:05.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff153e00640 1 -- 172.21.15.47:0/3626520925 >> v1:172.21.15.116:6789/0 conn(0x7ff15415e570 legacy=0x7ff15406c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:05.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff153e00640 1 -- 172.21.15.47:0/3626520925 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1540cece0 con 0x7ff1540cd630 2024-02-19T15:24:05.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff153e00640 1 -- 172.21.15.47:0/3626520925 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7ff14c001b90 con 0x7ff1540cd630 2024-02-19T15:24:05.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff153e00640 1 -- 172.21.15.47:0/3626520925 >> v1:172.21.15.47:6789/0 conn(0x7ff1540cd630 legacy=0x7ff15406ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:05.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff153e00640 1 --2- 172.21.15.47:0/3626520925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1540ce3c0 0x7ff13c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:05.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff153e00640 1 --2- 172.21.15.47:0/3626520925 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c003f30 0x7ff13c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:05.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff153e00640 1 --2- 172.21.15.47:0/3626520925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c004d10 0x7ff13c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:05.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff153e00640 1 -- 172.21.15.47:0/3626520925 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15406d110 con 0x7ff13c003f30 2024-02-19T15:24:05.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff153e00640 1 -- 172.21.15.47:0/3626520925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c00ba70 con 0x7ff13c004d10 2024-02-19T15:24:05.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff153e00640 1 -- 172.21.15.47:0/3626520925 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c00bc30 con 0x7ff1540ce3c0 2024-02-19T15:24:05.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff158e00640 1 --2- 172.21.15.47:0/3626520925 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c003f30 0x7ff13c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:05.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff15a200640 1 --2- 172.21.15.47:0/3626520925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c004d10 0x7ff13c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:05.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff159800640 1 --2- 172.21.15.47:0/3626520925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1540ce3c0 0x7ff13c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:05.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff158e00640 1 -- 172.21.15.47:0/3626520925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1540ce3c0 msgr2=0x7ff13c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:05.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff158e00640 1 --2- 172.21.15.47:0/3626520925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1540ce3c0 0x7ff13c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:05.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff158e00640 1 -- 172.21.15.47:0/3626520925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c004d10 msgr2=0x7ff13c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:05.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff158e00640 1 --2- 172.21.15.47:0/3626520925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c004d10 0x7ff13c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:05.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.882+0000 7ff158e00640 1 -- 172.21.15.47:0/3626520925 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13c003530 con 0x7ff13c003f30 2024-02-19T15:24:05.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.886+0000 7ff158e00640 1 --2- 172.21.15.47:0/3626520925 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c003f30 0x7ff13c0047d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff140001750 tx=0x7ff14001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e19f27b1a9e319 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:05.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.886+0000 7ff153e00640 1 -- 172.21.15.47:0/3626520925 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1400a4ea0 con 0x7ff13c003f30 2024-02-19T15:24:05.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.886+0000 7ff153e00640 1 -- 172.21.15.47:0/3626520925 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1400a7050 con 0x7ff13c003f30 2024-02-19T15:24:05.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.886+0000 7ff153e00640 1 -- 172.21.15.47:0/3626520925 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1400052d0 con 0x7ff13c003f30 2024-02-19T15:24:05.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.886+0000 7ff15c200640 1 -- 172.21.15.47:0/3626520925 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c003f30 msgr2=0x7ff13c0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:05.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.886+0000 7ff15c200640 1 --2- 172.21.15.47:0/3626520925 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c003f30 0x7ff13c0047d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff140001750 tx=0x7ff14001c3e0 comp rx=0 tx=0).stop 2024-02-19T15:24:05.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.886+0000 7ff15a200640 1 -- 172.21.15.47:0/3626520925 reap_dead start 2024-02-19T15:24:05.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.886+0000 7ff15c200640 1 -- 172.21.15.47:0/3626520925 shutdown_connections 2024-02-19T15:24:05.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.886+0000 7ff15c200640 1 -- 172.21.15.47:0/3626520925 >> 172.21.15.47:0/3626520925 conn(0x7ff1540d1b40 msgr2=0x7ff1540d1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:05.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.886+0000 7ff15c200640 1 -- 172.21.15.47:0/3626520925 shutdown_connections 2024-02-19T15:24:05.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.886+0000 7ff15c200640 1 -- 172.21.15.47:0/3626520925 wait complete. 2024-02-19T15:24:05.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.886+0000 7ff15c200640 1 Processor -- start 2024-02-19T15:24:05.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.886+0000 7ff15c200640 1 -- start start 2024-02-19T15:24:05.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.886+0000 7ff15c200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c003f30 0x7ff1541f2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:05.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.886+0000 7ff15c200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff15409e440 0x7ff15409e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:05.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.886+0000 7ff15c200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1540c9c60 0x7ff1540ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:05.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.886+0000 7ff15c200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1541f3bd0 con 0x7ff13c003f30 2024-02-19T15:24:05.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.886+0000 7ff15c200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15409ee00 con 0x7ff1540c9c60 2024-02-19T15:24:05.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.886+0000 7ff15c200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540ca620 con 0x7ff15409e440 2024-02-19T15:24:05.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.886+0000 7ff159800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c003f30 0x7ff1541f2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:05.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.890+0000 7ff159800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c003f30 0x7ff1541f2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43254/0 (socket says 172.21.15.47:43254) 2024-02-19T15:24:05.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.890+0000 7ff159800640 1 -- 172.21.15.47:0/367076270 learned_addr learned my addr 172.21.15.47:0/367076270 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:05.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.890+0000 7ff158e00640 1 --2- 172.21.15.47:0/367076270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff15409e440 0x7ff15409e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:05.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.890+0000 7ff15a200640 1 --2- 172.21.15.47:0/367076270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1540c9c60 0x7ff1540ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:05.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.890+0000 7ff159800640 1 -- 172.21.15.47:0/367076270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff15409e440 msgr2=0x7ff15409e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:05.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.890+0000 7ff159800640 1 --2- 172.21.15.47:0/367076270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff15409e440 0x7ff15409e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:05.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.890+0000 7ff159800640 1 -- 172.21.15.47:0/367076270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1540c9c60 msgr2=0x7ff1540ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:05.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.890+0000 7ff159800640 1 --2- 172.21.15.47:0/367076270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1540c9c60 0x7ff1540ca0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:05.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.890+0000 7ff159800640 1 -- 172.21.15.47:0/367076270 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1540ca8e0 con 0x7ff13c003f30 2024-02-19T15:24:05.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.890+0000 7ff15a200640 1 --2- 172.21.15.47:0/367076270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1540c9c60 0x7ff1540ca0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:05.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.890+0000 7ff158e00640 1 --2- 172.21.15.47:0/367076270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff15409e440 0x7ff15409e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:05.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.890+0000 7ff159800640 1 --2- 172.21.15.47:0/367076270 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c003f30 0x7ff1541f2ff0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff1480119d0 tx=0x7ff148011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:05.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.890+0000 7ff151600640 1 -- 172.21.15.47:0/367076270 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14800fb50 con 0x7ff13c003f30 2024-02-19T15:24:05.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.890+0000 7ff151600640 1 -- 172.21.15.47:0/367076270 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14800fcb0 con 0x7ff13c003f30 2024-02-19T15:24:05.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.890+0000 7ff15c200640 1 -- 172.21.15.47:0/367076270 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1540cabc0 con 0x7ff13c003f30 2024-02-19T15:24:05.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.890+0000 7ff151600640 1 -- 172.21.15.47:0/367076270 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff148014440 con 0x7ff13c003f30 2024-02-19T15:24:05.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.890+0000 7ff15c200640 1 -- 172.21.15.47:0/367076270 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1540cb180 con 0x7ff13c003f30 2024-02-19T15:24:05.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.890+0000 7ff15c200640 1 -- 172.21.15.47:0/367076270 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff154075360 con 0x7ff13c003f30 2024-02-19T15:24:05.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.890+0000 7ff151600640 1 -- 172.21.15.47:0/367076270 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff1480145a0 con 0x7ff13c003f30 2024-02-19T15:24:05.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.894+0000 7ff151600640 1 --2- 172.21.15.47:0/367076270 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff134066e70 0x7ff134069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:05.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.894+0000 7ff151600640 1 -- 172.21.15.47:0/367076270 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1480853f0 con 0x7ff13c003f30 2024-02-19T15:24:05.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.894+0000 7ff158e00640 1 --2- 172.21.15.47:0/367076270 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff134066e70 0x7ff134069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:05.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.894+0000 7ff158e00640 1 --2- 172.21.15.47:0/367076270 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff134066e70 0x7ff134069330 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff15415e970 tx=0x7ff1400011a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:05.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:05.898+0000 7ff151600640 1 -- 172.21.15.47:0/367076270 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff148052200 con 0x7ff13c003f30 2024-02-19T15:24:06.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.066+0000 7ff15c200640 1 -- 172.21.15.47:0/367076270 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff154074940 con 0x7ff13c003f30 2024-02-19T15:24:06.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.066+0000 7ff151600640 1 -- 172.21.15.47:0/367076270 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16625 (secure 0 0 0) 0x7ff1480558c0 con 0x7ff13c003f30 2024-02-19T15:24:06.070 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:06.070 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":45,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:23:43.941118+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:23:40.751741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-02-19T15:23:41.338930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:24:06.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.070+0000 7ff15c200640 1 -- 172.21.15.47:0/367076270 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff134066e70 msgr2=0x7ff134069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:06.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.070+0000 7ff15c200640 1 --2- 172.21.15.47:0/367076270 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff134066e70 0x7ff134069330 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff15415e970 tx=0x7ff1400011a0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.070+0000 7ff15c200640 1 -- 172.21.15.47:0/367076270 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c003f30 msgr2=0x7ff1541f2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:06.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.070+0000 7ff15c200640 1 --2- 172.21.15.47:0/367076270 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c003f30 0x7ff1541f2ff0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff1480119d0 tx=0x7ff148011e90 comp rx=0 tx=0).stop 2024-02-19T15:24:06.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.070+0000 7ff15c200640 1 -- 172.21.15.47:0/367076270 shutdown_connections 2024-02-19T15:24:06.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.070+0000 7ff15c200640 1 --2- 172.21.15.47:0/367076270 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff134066e70 0x7ff134069330 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.070+0000 7ff15c200640 1 --2- 172.21.15.47:0/367076270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1540c9c60 0x7ff1540ca0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.070+0000 7ff15c200640 1 --2- 172.21.15.47:0/367076270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff15409e440 0x7ff15409e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.070+0000 7ff15c200640 1 --2- 172.21.15.47:0/367076270 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c003f30 0x7ff1541f2ff0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.070+0000 7ff15c200640 1 -- 172.21.15.47:0/367076270 >> 172.21.15.47:0/367076270 conn(0x7ff1540d1b40 msgr2=0x7ff15415c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:06.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.070+0000 7ff15c200640 1 -- 172.21.15.47:0/367076270 shutdown_connections 2024-02-19T15:24:06.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.070+0000 7ff15c200640 1 -- 172.21.15.47:0/367076270 wait complete. 2024-02-19T15:24:06.092 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:23:40.751741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:23:41.338930+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:24:06.092 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:24:06.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.234+0000 7fd45e000640 1 Processor -- start 2024-02-19T15:24:06.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.234+0000 7fd45e000640 1 -- start start 2024-02-19T15:24:06.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.234+0000 7fd45e000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd45815e160 0x7fd4580d1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:06.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.234+0000 7fd45e000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd45806f220 0x7fd45806dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:06.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.234+0000 7fd45e000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd458071a70 con 0x7fd45806e010 2024-02-19T15:24:06.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.234+0000 7fd45e000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45806e440 con 0x7fd45806f220 2024-02-19T15:24:06.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.234+0000 7fd45e000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4580ced30 con 0x7fd45815e160 2024-02-19T15:24:06.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.234+0000 7fd456a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd45806f220 0x7fd45806dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:06.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.234+0000 7fd456a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd45806f220 0x7fd45806dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43264/0 (socket says 172.21.15.47:43264) 2024-02-19T15:24:06.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.234+0000 7fd456a00640 1 -- 172.21.15.47:0/816891449 learned_addr learned my addr 172.21.15.47:0/816891449 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:06.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.234+0000 7fd457400640 1 --2- 172.21.15.47:0/816891449 >> v2:172.21.15.116:3300/0 conn(0x7fd45815e160 0x7fd4580d1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:06.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.234+0000 7fd456a00640 1 -- 172.21.15.47:0/816891449 >> v2:172.21.15.116:3300/0 conn(0x7fd45815e160 msgr2=0x7fd4580d1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:06.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.234+0000 7fd456a00640 1 --2- 172.21.15.47:0/816891449 >> v2:172.21.15.116:3300/0 conn(0x7fd45815e160 0x7fd4580d1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.234+0000 7fd456a00640 1 -- 172.21.15.47:0/816891449 >> v1:172.21.15.116:6789/0 conn(0x7fd45806e010 legacy=0x7fd45806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:06.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.234+0000 7fd456a00640 1 -- 172.21.15.47:0/816891449 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45806f6b0 con 0x7fd45806f220 2024-02-19T15:24:06.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.238+0000 7fd456a00640 1 --2- 172.21.15.47:0/816891449 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd45806f220 0x7fd45806dad0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd458074a90 tx=0x7fd44401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22b276d7da6747ae server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:06.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.238+0000 7fd456000640 1 -- 172.21.15.47:0/816891449 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4440aa660 con 0x7fd45806f220 2024-02-19T15:24:06.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.238+0000 7fd456000640 1 -- 172.21.15.47:0/816891449 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4440aa7c0 con 0x7fd45806f220 2024-02-19T15:24:06.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.238+0000 7fd456000640 1 -- 172.21.15.47:0/816891449 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4440aab50 con 0x7fd45806f220 2024-02-19T15:24:06.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.238+0000 7fd45e000640 1 -- 172.21.15.47:0/816891449 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd45806f220 msgr2=0x7fd45806dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:06.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.238+0000 7fd45e000640 1 --2- 172.21.15.47:0/816891449 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd45806f220 0x7fd45806dad0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd458074a90 tx=0x7fd44401c500 comp rx=0 tx=0).stop 2024-02-19T15:24:06.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.238+0000 7fd45e000640 1 -- 172.21.15.47:0/816891449 shutdown_connections 2024-02-19T15:24:06.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.238+0000 7fd45e000640 1 --2- 172.21.15.47:0/816891449 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd45806f220 0x7fd45806dad0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.238+0000 7fd45e000640 1 --2- 172.21.15.47:0/816891449 >> v2:172.21.15.116:3300/0 conn(0x7fd45815e160 0x7fd4580d1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.238+0000 7fd45e000640 1 -- 172.21.15.47:0/816891449 >> 172.21.15.47:0/816891449 conn(0x7fd4580d0d30 msgr2=0x7fd4580d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:06.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.238+0000 7fd45e000640 1 -- 172.21.15.47:0/816891449 shutdown_connections 2024-02-19T15:24:06.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.238+0000 7fd45e000640 1 -- 172.21.15.47:0/816891449 wait complete. 2024-02-19T15:24:06.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd45e000640 1 Processor -- start 2024-02-19T15:24:06.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd45e000640 1 -- start start 2024-02-19T15:24:06.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd45e000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd45806e010 0x7fd4580b0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:06.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd45e000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd45806f220 0x7fd458157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:06.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd45e000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd45815e160 0x7fd458155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:06.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd45e000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd458155970 con 0x7fd45815e160 2024-02-19T15:24:06.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd45e000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd458155340 con 0x7fd45806f220 2024-02-19T15:24:06.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd45e000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4580b3290 con 0x7fd45806e010 2024-02-19T15:24:06.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd457e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd45815e160 0x7fd458155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:06.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd457400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd45806e010 0x7fd4580b0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:06.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd457e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd45815e160 0x7fd458155e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43268/0 (socket says 172.21.15.47:43268) 2024-02-19T15:24:06.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd457e00640 1 -- 172.21.15.47:0/1593402681 learned_addr learned my addr 172.21.15.47:0/1593402681 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:06.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd456a00640 1 --2- 172.21.15.47:0/1593402681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd45806f220 0x7fd458157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:06.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd457e00640 1 -- 172.21.15.47:0/1593402681 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd45806e010 msgr2=0x7fd4580b0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:06.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd457e00640 1 --2- 172.21.15.47:0/1593402681 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd45806e010 0x7fd4580b0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd457e00640 1 -- 172.21.15.47:0/1593402681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd45806f220 msgr2=0x7fd458157c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:06.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd457e00640 1 --2- 172.21.15.47:0/1593402681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd45806f220 0x7fd458157c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd457e00640 1 -- 172.21.15.47:0/1593402681 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4440094d0 con 0x7fd45815e160 2024-02-19T15:24:06.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd457e00640 1 --2- 172.21.15.47:0/1593402681 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd45815e160 0x7fd458155e00 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd45000c770 tx=0x7fd45000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:06.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd44f400640 1 -- 172.21.15.47:0/1593402681 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd450003b80 con 0x7fd45815e160 2024-02-19T15:24:06.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd44f400640 1 -- 172.21.15.47:0/1593402681 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd450003ce0 con 0x7fd45815e160 2024-02-19T15:24:06.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd45e000640 1 -- 172.21.15.47:0/1593402681 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4580b3570 con 0x7fd45815e160 2024-02-19T15:24:06.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd44f400640 1 -- 172.21.15.47:0/1593402681 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45000aa90 con 0x7fd45815e160 2024-02-19T15:24:06.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.242+0000 7fd45e000640 1 -- 172.21.15.47:0/1593402681 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4580b0c70 con 0x7fd45815e160 2024-02-19T15:24:06.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.246+0000 7fd45e000640 1 -- 172.21.15.47:0/1593402681 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd458155ae0 con 0x7fd45815e160 2024-02-19T15:24:06.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.246+0000 7fd44f400640 1 -- 172.21.15.47:0/1593402681 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd45000abf0 con 0x7fd45815e160 2024-02-19T15:24:06.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.246+0000 7fd44f400640 1 --2- 172.21.15.47:0/1593402681 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd438066e40 0x7fd438069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:06.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.246+0000 7fd44f400640 1 -- 172.21.15.47:0/1593402681 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd450084800 con 0x7fd45815e160 2024-02-19T15:24:06.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.246+0000 7fd457400640 1 --2- 172.21.15.47:0/1593402681 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd438066e40 0x7fd438069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:06.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.250+0000 7fd457400640 1 --2- 172.21.15.47:0/1593402681 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd438066e40 0x7fd438069300 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd448009ac0 tx=0x7fd448009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:06.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.250+0000 7fd44f400640 1 -- 172.21.15.47:0/1593402681 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd450051600 con 0x7fd45815e160 2024-02-19T15:24:06.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.414+0000 7fd45e000640 1 -- 172.21.15.47:0/1593402681 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd458075360 con 0x7fd45815e160 2024-02-19T15:24:06.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.418+0000 7fd44f400640 1 -- 172.21.15.47:0/1593402681 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd450054cc0 con 0x7fd45815e160 2024-02-19T15:24:06.419 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:24:06.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.418+0000 7fd45e000640 1 -- 172.21.15.47:0/1593402681 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd438066e40 msgr2=0x7fd438069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:06.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.418+0000 7fd45e000640 1 --2- 172.21.15.47:0/1593402681 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd438066e40 0x7fd438069300 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd448009ac0 tx=0x7fd448009190 comp rx=0 tx=0).stop 2024-02-19T15:24:06.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.418+0000 7fd45e000640 1 -- 172.21.15.47:0/1593402681 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd45815e160 msgr2=0x7fd458155e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:06.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.418+0000 7fd45e000640 1 --2- 172.21.15.47:0/1593402681 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd45815e160 0x7fd458155e00 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd45000c770 tx=0x7fd45000cc30 comp rx=0 tx=0).stop 2024-02-19T15:24:06.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.418+0000 7fd45e000640 1 -- 172.21.15.47:0/1593402681 shutdown_connections 2024-02-19T15:24:06.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.418+0000 7fd45e000640 1 --2- 172.21.15.47:0/1593402681 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd438066e40 0x7fd438069300 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.418+0000 7fd45e000640 1 --2- 172.21.15.47:0/1593402681 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd45815e160 0x7fd458155e00 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.418+0000 7fd45e000640 1 --2- 172.21.15.47:0/1593402681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd45806f220 0x7fd458157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.418+0000 7fd45e000640 1 --2- 172.21.15.47:0/1593402681 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd45806e010 0x7fd4580b0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.418+0000 7fd45e000640 1 -- 172.21.15.47:0/1593402681 >> 172.21.15.47:0/1593402681 conn(0x7fd4580d0d30 msgr2=0x7fd45806aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:06.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.418+0000 7fd45e000640 1 -- 172.21.15.47:0/1593402681 shutdown_connections 2024-02-19T15:24:06.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.418+0000 7fd45e000640 1 -- 172.21.15.47:0/1593402681 wait complete. 2024-02-19T15:24:06.441 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:24:06.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.582+0000 7fdb03a00640 1 Processor -- start 2024-02-19T15:24:06.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.582+0000 7fdb03a00640 1 -- start start 2024-02-19T15:24:06.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.582+0000 7fdb03a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdafc0c8260 0x7fdafc0cce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:06.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.582+0000 7fdb03a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdafc0ceee0 con 0x7fdafc0d0820 2024-02-19T15:24:06.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.582+0000 7fdb03a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdafc0cdc60 con 0x7fdafc0c7e80 2024-02-19T15:24:06.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.582+0000 7fdb03a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdafc0d2a40 con 0x7fdafc0c8260 2024-02-19T15:24:06.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.582+0000 7fdb01a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdafc0c8260 0x7fdafc0cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:06.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.582+0000 7fdafbe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fdafc0d0820 0x7fdafc0cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51970/0 (socket says 172.21.15.47:51970) 2024-02-19T15:24:06.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.582+0000 7fdb01a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdafc0c8260 0x7fdafc0cce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43270/0 (socket says 172.21.15.47:43270) 2024-02-19T15:24:06.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.582+0000 7fdafbe00640 1 -- 172.21.15.47:0/3020655663 learned_addr learned my addr 172.21.15.47:0/3020655663 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:06.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.582+0000 7fdb01a00640 1 -- 172.21.15.47:0/3020655663 >> v1:172.21.15.116:6789/0 conn(0x7fdafc0c7e80 legacy=0x7fdafc0d0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:06.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.586+0000 7fdb01a00640 1 -- 172.21.15.47:0/3020655663 >> v1:172.21.15.47:6789/0 conn(0x7fdafc0d0820 legacy=0x7fdafc0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:06.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.586+0000 7fdb01a00640 1 -- 172.21.15.47:0/3020655663 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdafc0cd980 con 0x7fdafc0c8260 2024-02-19T15:24:06.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.586+0000 7fdb01a00640 1 --2- 172.21.15.47:0/3020655663 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdafc0c8260 0x7fdafc0cce50 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4009d90 tx=0x7fdaf401c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d3b47abc90638 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:06.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.586+0000 7fdafb400640 1 -- 172.21.15.47:0/3020655663 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf40a5920 con 0x7fdafc0c8260 2024-02-19T15:24:06.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.586+0000 7fdafb400640 1 -- 172.21.15.47:0/3020655663 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaf40a5a80 con 0x7fdafc0c8260 2024-02-19T15:24:06.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.586+0000 7fdafb400640 1 -- 172.21.15.47:0/3020655663 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf40a5e10 con 0x7fdafc0c8260 2024-02-19T15:24:06.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.586+0000 7fdb03a00640 1 -- 172.21.15.47:0/3020655663 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdafc0c8260 msgr2=0x7fdafc0cce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:06.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.586+0000 7fdb03a00640 1 --2- 172.21.15.47:0/3020655663 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdafc0c8260 0x7fdafc0cce50 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4009d90 tx=0x7fdaf401c900 comp rx=0 tx=0).stop 2024-02-19T15:24:06.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.586+0000 7fdb03a00640 1 -- 172.21.15.47:0/3020655663 shutdown_connections 2024-02-19T15:24:06.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.586+0000 7fdb03a00640 1 --2- 172.21.15.47:0/3020655663 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdafc0c8260 0x7fdafc0cce50 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.586+0000 7fdb03a00640 1 -- 172.21.15.47:0/3020655663 >> 172.21.15.47:0/3020655663 conn(0x7fdafc0d1b30 msgr2=0x7fdafc0d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:06.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.586+0000 7fdb03a00640 1 -- 172.21.15.47:0/3020655663 shutdown_connections 2024-02-19T15:24:06.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.586+0000 7fdb03a00640 1 -- 172.21.15.47:0/3020655663 wait complete. 2024-02-19T15:24:06.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdb03a00640 1 Processor -- start 2024-02-19T15:24:06.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdb03a00640 1 -- start start 2024-02-19T15:24:06.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdb03a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdafc0c7e80 0x7fdafc14db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:06.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdb03a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdafc0c8260 0x7fdafc15b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:06.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdb03a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdafc0d0820 0x7fdafc159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:06.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdb03a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc15eb00 con 0x7fdafc0c7e80 2024-02-19T15:24:06.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdb03a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc15a5d0 con 0x7fdafc0d0820 2024-02-19T15:24:06.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdb03a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc15a8b0 con 0x7fdafc0c8260 2024-02-19T15:24:06.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdb01000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdafc0c7e80 0x7fdafc14db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:06.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdb01000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdafc0c7e80 0x7fdafc14db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43274/0 (socket says 172.21.15.47:43274) 2024-02-19T15:24:06.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdb01000640 1 -- 172.21.15.47:0/4057659208 learned_addr learned my addr 172.21.15.47:0/4057659208 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:06.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdb01a00640 1 --2- 172.21.15.47:0/4057659208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdafc0d0820 0x7fdafc159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:06.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdafbe00640 1 --2- 172.21.15.47:0/4057659208 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdafc0c8260 0x7fdafc15b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:06.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdb01000640 1 -- 172.21.15.47:0/4057659208 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdafc0c8260 msgr2=0x7fdafc15b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:06.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdb01000640 1 --2- 172.21.15.47:0/4057659208 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdafc0c8260 0x7fdafc15b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdb01000640 1 -- 172.21.15.47:0/4057659208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdafc0d0820 msgr2=0x7fdafc159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:06.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdb01000640 1 --2- 172.21.15.47:0/4057659208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdafc0d0820 0x7fdafc159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdb01000640 1 -- 172.21.15.47:0/4057659208 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf4009a70 con 0x7fdafc0c7e80 2024-02-19T15:24:06.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdb01000640 1 --2- 172.21.15.47:0/4057659208 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdafc0c7e80 0x7fdafc14db30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fdaf000c450 tx=0x7fdaf000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:06.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdaf8c00640 1 -- 172.21.15.47:0/4057659208 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf0012830 con 0x7fdafc0c7e80 2024-02-19T15:24:06.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdaf8c00640 1 -- 172.21.15.47:0/4057659208 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaf00129b0 con 0x7fdafc0c7e80 2024-02-19T15:24:06.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdb03a00640 1 -- 172.21.15.47:0/4057659208 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdafc15ab90 con 0x7fdafc0c7e80 2024-02-19T15:24:06.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdaf8c00640 1 -- 172.21.15.47:0/4057659208 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf0012d60 con 0x7fdafc0c7e80 2024-02-19T15:24:06.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.590+0000 7fdb03a00640 1 -- 172.21.15.47:0/4057659208 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdafc15b150 con 0x7fdafc0c7e80 2024-02-19T15:24:06.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.594+0000 7fdb03a00640 1 -- 172.21.15.47:0/4057659208 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdafc075360 con 0x7fdafc0c7e80 2024-02-19T15:24:06.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.594+0000 7fdaf8c00640 1 -- 172.21.15.47:0/4057659208 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fdaf001c460 con 0x7fdafc0c7e80 2024-02-19T15:24:06.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.594+0000 7fdaf8c00640 1 --2- 172.21.15.47:0/4057659208 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdadc066e40 0x7fdadc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:06.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.594+0000 7fdaf8c00640 1 -- 172.21.15.47:0/4057659208 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdaf00852a0 con 0x7fdafc0c7e80 2024-02-19T15:24:06.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.594+0000 7fdafbe00640 1 --2- 172.21.15.47:0/4057659208 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdadc066e40 0x7fdadc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:06.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.594+0000 7fdafbe00640 1 --2- 172.21.15.47:0/4057659208 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdadc066e40 0x7fdadc069300 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdafc0c99d0 tx=0x7fdae8007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:06.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.598+0000 7fdaf8c00640 1 -- 172.21.15.47:0/4057659208 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fdaf0052120 con 0x7fdafc0c7e80 2024-02-19T15:24:06.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.762+0000 7fdb03a00640 1 -- 172.21.15.47:0/4057659208 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdafc0b23d0 con 0x7fdafc0c7e80 2024-02-19T15:24:06.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.762+0000 7fdaf8c00640 1 -- 172.21.15.47:0/4057659208 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v45) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdaf00557e0 con 0x7fdafc0c7e80 2024-02-19T15:24:06.766 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:24:06.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.766+0000 7fdb03a00640 1 -- 172.21.15.47:0/4057659208 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdadc066e40 msgr2=0x7fdadc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:06.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.766+0000 7fdb03a00640 1 --2- 172.21.15.47:0/4057659208 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdadc066e40 0x7fdadc069300 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdafc0c99d0 tx=0x7fdae8007e90 comp rx=0 tx=0).stop 2024-02-19T15:24:06.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.766+0000 7fdb03a00640 1 -- 172.21.15.47:0/4057659208 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdafc0c7e80 msgr2=0x7fdafc14db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:06.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.766+0000 7fdb03a00640 1 --2- 172.21.15.47:0/4057659208 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdafc0c7e80 0x7fdafc14db30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fdaf000c450 tx=0x7fdaf000c910 comp rx=0 tx=0).stop 2024-02-19T15:24:06.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.766+0000 7fdb03a00640 1 -- 172.21.15.47:0/4057659208 shutdown_connections 2024-02-19T15:24:06.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.766+0000 7fdb03a00640 1 --2- 172.21.15.47:0/4057659208 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdadc066e40 0x7fdadc069300 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.766+0000 7fdb03a00640 1 --2- 172.21.15.47:0/4057659208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdafc0d0820 0x7fdafc159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.766+0000 7fdb03a00640 1 --2- 172.21.15.47:0/4057659208 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdafc0c8260 0x7fdafc15b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.766+0000 7fdb03a00640 1 --2- 172.21.15.47:0/4057659208 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdafc0c7e80 0x7fdafc14db30 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.766+0000 7fdb03a00640 1 -- 172.21.15.47:0/4057659208 >> 172.21.15.47:0/4057659208 conn(0x7fdafc0d1b30 msgr2=0x7fdafc0c8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:06.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.766+0000 7fdb03a00640 1 -- 172.21.15.47:0/4057659208 shutdown_connections 2024-02-19T15:24:06.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.766+0000 7fdb03a00640 1 -- 172.21.15.47:0/4057659208 wait complete. 2024-02-19T15:24:06.786 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:24:06.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.926+0000 7f5227600640 1 Processor -- start 2024-02-19T15:24:06.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.926+0000 7f5227600640 1 -- start start 2024-02-19T15:24:06.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.926+0000 7f5227600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f52200ce3b0 0x7f52200ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:06.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.926+0000 7f5227600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f522015e280 0x7f52200cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:06.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.926+0000 7f5227600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52200cc620 con 0x7f52200d2650 2024-02-19T15:24:06.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.926+0000 7f5227600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52200d12e0 con 0x7f52200ce3b0 2024-02-19T15:24:06.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.926+0000 7f5227600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52200d1940 con 0x7f522015e280 2024-02-19T15:24:06.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.930+0000 7f5224c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f52200ce3b0 0x7f52200ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:06.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.930+0000 7f5224c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f52200ce3b0 0x7f52200ce790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43284/0 (socket says 172.21.15.47:43284) 2024-02-19T15:24:06.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.930+0000 7f5224c00640 1 -- 172.21.15.47:0/1554026030 learned_addr learned my addr 172.21.15.47:0/1554026030 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:06.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.930+0000 7f5225600640 1 --2- 172.21.15.47:0/1554026030 >> v2:172.21.15.116:3300/0 conn(0x7f522015e280 0x7f52200cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:06.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.930+0000 7f5224c00640 1 -- 172.21.15.47:0/1554026030 >> v2:172.21.15.116:3300/0 conn(0x7f522015e280 msgr2=0x7f52200cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:06.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.930+0000 7f5224c00640 1 --2- 172.21.15.47:0/1554026030 >> v2:172.21.15.116:3300/0 conn(0x7f522015e280 0x7f52200cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.930+0000 7f5224c00640 1 -- 172.21.15.47:0/1554026030 >> v1:172.21.15.116:6789/0 conn(0x7f52200d2650 legacy=0x7f52200ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:06.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.930+0000 7f5224c00640 1 -- 172.21.15.47:0/1554026030 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52200cecd0 con 0x7f52200ce3b0 2024-02-19T15:24:06.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.934+0000 7f5224c00640 1 --2- 172.21.15.47:0/1554026030 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52200ce3b0 0x7f52200ce790 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f5220067b60 tx=0x7f521401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a42306c95eb3a1a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:06.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.934+0000 7f521f400640 1 -- 172.21.15.47:0/1554026030 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52140ab600 con 0x7f52200ce3b0 2024-02-19T15:24:06.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.934+0000 7f521f400640 1 -- 172.21.15.47:0/1554026030 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52140a7050 con 0x7f52200ce3b0 2024-02-19T15:24:06.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.934+0000 7f521f400640 1 -- 172.21.15.47:0/1554026030 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52140ab980 con 0x7f52200ce3b0 2024-02-19T15:24:06.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.934+0000 7f5227600640 1 -- 172.21.15.47:0/1554026030 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52200ce3b0 msgr2=0x7f52200ce790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:06.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.934+0000 7f5227600640 1 --2- 172.21.15.47:0/1554026030 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52200ce3b0 0x7f52200ce790 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f5220067b60 tx=0x7f521401c400 comp rx=0 tx=0).stop 2024-02-19T15:24:06.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.934+0000 7f5227600640 1 -- 172.21.15.47:0/1554026030 shutdown_connections 2024-02-19T15:24:06.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.934+0000 7f5227600640 1 --2- 172.21.15.47:0/1554026030 >> v2:172.21.15.116:3300/0 conn(0x7f522015e280 0x7f52200cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.934+0000 7f5227600640 1 --2- 172.21.15.47:0/1554026030 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52200ce3b0 0x7f52200ce790 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.934+0000 7f5227600640 1 -- 172.21.15.47:0/1554026030 >> 172.21.15.47:0/1554026030 conn(0x7f52200d1b30 msgr2=0x7f52200d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:06.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.934+0000 7f5227600640 1 -- 172.21.15.47:0/1554026030 shutdown_connections 2024-02-19T15:24:06.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.934+0000 7f5227600640 1 -- 172.21.15.47:0/1554026030 wait complete. 2024-02-19T15:24:06.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.934+0000 7f5227600640 1 Processor -- start 2024-02-19T15:24:06.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f5227600640 1 -- start start 2024-02-19T15:24:06.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f5227600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52200ce3b0 0x7f5220154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:06.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f5227600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52200d2650 0x7f522014f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:06.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f5227600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f522015e280 0x7f52200cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:06.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f5227600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522014f760 con 0x7f52200d2650 2024-02-19T15:24:06.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f5227600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52200cbb70 con 0x7f522015e280 2024-02-19T15:24:06.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f5227600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5220154c70 con 0x7f52200ce3b0 2024-02-19T15:24:06.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f521fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52200d2650 0x7f522014f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:06.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f521fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52200d2650 0x7f522014f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43296/0 (socket says 172.21.15.47:43296) 2024-02-19T15:24:06.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f521fe00640 1 -- 172.21.15.47:0/1163002313 learned_addr learned my addr 172.21.15.47:0/1163002313 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:06.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f5224c00640 1 --2- 172.21.15.47:0/1163002313 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52200ce3b0 0x7f5220154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:06.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f5225600640 1 --2- 172.21.15.47:0/1163002313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f522015e280 0x7f52200cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:06.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f521fe00640 1 -- 172.21.15.47:0/1163002313 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52200ce3b0 msgr2=0x7f5220154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:06.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f521fe00640 1 --2- 172.21.15.47:0/1163002313 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52200ce3b0 0x7f5220154590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f521fe00640 1 -- 172.21.15.47:0/1163002313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f522015e280 msgr2=0x7f52200cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:06.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f521fe00640 1 --2- 172.21.15.47:0/1163002313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f522015e280 0x7f52200cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:06.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f521fe00640 1 -- 172.21.15.47:0/1163002313 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52140094d0 con 0x7f52200d2650 2024-02-19T15:24:06.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f5225600640 1 --2- 172.21.15.47:0/1163002313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f522015e280 0x7f52200cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:06.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f521fe00640 1 --2- 172.21.15.47:0/1163002313 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52200d2650 0x7f522014f220 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f520c00c370 tx=0x7f520c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:06.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f521cc00640 1 -- 172.21.15.47:0/1163002313 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c0052d0 con 0x7f52200d2650 2024-02-19T15:24:06.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f521cc00640 1 -- 172.21.15.47:0/1163002313 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520c005430 con 0x7f52200d2650 2024-02-19T15:24:06.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f521cc00640 1 -- 172.21.15.47:0/1163002313 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c007260 con 0x7f52200d2650 2024-02-19T15:24:06.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f5227600640 1 -- 172.21.15.47:0/1163002313 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5220154f50 con 0x7f52200d2650 2024-02-19T15:24:06.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f5227600640 1 -- 172.21.15.47:0/1163002313 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5220155510 con 0x7f52200d2650 2024-02-19T15:24:06.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.938+0000 7f5227600640 1 -- 172.21.15.47:0/1163002313 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52200c8ad0 con 0x7f52200d2650 2024-02-19T15:24:06.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.942+0000 7f521cc00640 1 -- 172.21.15.47:0/1163002313 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f520c0073c0 con 0x7f52200d2650 2024-02-19T15:24:06.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.942+0000 7f521cc00640 1 --2- 172.21.15.47:0/1163002313 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5200066e40 0x7f5200069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:06.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.942+0000 7f521cc00640 1 -- 172.21.15.47:0/1163002313 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f520c083f30 con 0x7f52200d2650 2024-02-19T15:24:06.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.942+0000 7f5224c00640 1 --2- 172.21.15.47:0/1163002313 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5200066e40 0x7f5200069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:06.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.942+0000 7f5224c00640 1 --2- 172.21.15.47:0/1163002313 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5200066e40 0x7f5200069300 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f52140ad000 tx=0x7f5214008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:06.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:06.946+0000 7f521cc00640 1 -- 172.21.15.47:0/1163002313 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f52200c8ad0 con 0x7f52200d2650 2024-02-19T15:24:07.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.138+0000 7f5227600640 1 -- 172.21.15.47:0/1163002313 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f52200c8aa0 con 0x7f52200d2650 2024-02-19T15:24:07.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.138+0000 7f521cc00640 1 -- 172.21.15.47:0/1163002313 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v45) v1 ==== 100+0+11 (secure 0 0 0) 0x7f520c050bf0 con 0x7f52200d2650 2024-02-19T15:24:07.140 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:24:07.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.138+0000 7f5227600640 1 -- 172.21.15.47:0/1163002313 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5200066e40 msgr2=0x7f5200069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:07.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.138+0000 7f5227600640 1 --2- 172.21.15.47:0/1163002313 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5200066e40 0x7f5200069300 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f52140ad000 tx=0x7f5214008040 comp rx=0 tx=0).stop 2024-02-19T15:24:07.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.138+0000 7f5227600640 1 -- 172.21.15.47:0/1163002313 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52200d2650 msgr2=0x7f522014f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:07.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.138+0000 7f5227600640 1 --2- 172.21.15.47:0/1163002313 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52200d2650 0x7f522014f220 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f520c00c370 tx=0x7f520c00c830 comp rx=0 tx=0).stop 2024-02-19T15:24:07.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.138+0000 7f5227600640 1 -- 172.21.15.47:0/1163002313 shutdown_connections 2024-02-19T15:24:07.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.138+0000 7f5227600640 1 --2- 172.21.15.47:0/1163002313 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5200066e40 0x7f5200069300 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:07.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.138+0000 7f5227600640 1 --2- 172.21.15.47:0/1163002313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f522015e280 0x7f52200cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:07.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.138+0000 7f5227600640 1 --2- 172.21.15.47:0/1163002313 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52200d2650 0x7f522014f220 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:07.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.138+0000 7f5227600640 1 --2- 172.21.15.47:0/1163002313 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52200ce3b0 0x7f5220154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:07.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.138+0000 7f5227600640 1 -- 172.21.15.47:0/1163002313 >> 172.21.15.47:0/1163002313 conn(0x7f52200d1b30 msgr2=0x7f52200c5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:07.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.138+0000 7f5227600640 1 -- 172.21.15.47:0/1163002313 shutdown_connections 2024-02-19T15:24:07.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.138+0000 7f5227600640 1 -- 172.21.15.47:0/1163002313 wait complete. 2024-02-19T15:24:07.167 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-19T15:24:07.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.306+0000 7ff6aca00640 1 Processor -- start 2024-02-19T15:24:07.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.306+0000 7ff6aca00640 1 -- start start 2024-02-19T15:24:07.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.310+0000 7ff6aca00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff6a815e150 0x7ff6a80d07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:07.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.310+0000 7ff6aca00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6a80c7f60 con 0x7ff6a80c9400 2024-02-19T15:24:07.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.310+0000 7ff6aca00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6a815e530 con 0x7ff6a80d23c0 2024-02-19T15:24:07.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.310+0000 7ff6aca00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6a80c7930 con 0x7ff6a815e150 2024-02-19T15:24:07.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.310+0000 7ff6a6800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff6a80c9400 0x7ff6a80cf000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51972/0 (socket says 172.21.15.47:51972) 2024-02-19T15:24:07.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.310+0000 7ff6a6800640 1 -- 172.21.15.47:0/3634167123 learned_addr learned my addr 172.21.15.47:0/3634167123 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:07.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.310+0000 7ff6a5e00640 1 --2- 172.21.15.47:0/3634167123 >> v2:172.21.15.116:3300/0 conn(0x7ff6a815e150 0x7ff6a80d07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:07.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.310+0000 7ff6a5400640 1 -- 172.21.15.47:0/3634167123 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719616261 0 0) 0x7ff6a80c7f60 con 0x7ff6a80c9400 2024-02-19T15:24:07.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.310+0000 7ff6a5400640 1 -- 172.21.15.47:0/3634167123 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff68c003530 con 0x7ff6a80c9400 2024-02-19T15:24:07.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.310+0000 7ff6a5400640 1 -- 172.21.15.47:0/3634167123 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504186359 0 0) 0x7ff6a815e530 con 0x7ff6a80d23c0 2024-02-19T15:24:07.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.310+0000 7ff6a5400640 1 -- 172.21.15.47:0/3634167123 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6a80c7f60 con 0x7ff6a80d23c0 2024-02-19T15:24:07.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.310+0000 7ff6a5e00640 1 -- 172.21.15.47:0/3634167123 >> v1:172.21.15.116:6789/0 conn(0x7ff6a80d23c0 legacy=0x7ff6a80ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:07.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.310+0000 7ff6a5e00640 1 -- 172.21.15.47:0/3634167123 >> v1:172.21.15.47:6789/0 conn(0x7ff6a80c9400 legacy=0x7ff6a80cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:07.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.310+0000 7ff6a5400640 1 -- 172.21.15.47:0/3634167123 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 73210827 0 0) 0x7ff68c003530 con 0x7ff6a80c9400 2024-02-19T15:24:07.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.310+0000 7ff6a5e00640 1 -- 172.21.15.47:0/3634167123 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a80d2930 con 0x7ff6a815e150 2024-02-19T15:24:07.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.314+0000 7ff6a5e00640 1 --2- 172.21.15.47:0/3634167123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a815e150 0x7ff6a80d07d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff6980097c0 tx=0x7ff69801c390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a19b14b2126dc469 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:07.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.314+0000 7ff6a5400640 1 -- 172.21.15.47:0/3634167123 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0001a00 con 0x7ff6a815e150 2024-02-19T15:24:07.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.314+0000 7ff6a5400640 1 -- 172.21.15.47:0/3634167123 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6980ac440 con 0x7ff6a815e150 2024-02-19T15:24:07.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.314+0000 7ff6a5400640 1 -- 172.21.15.47:0/3634167123 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6980ac7d0 con 0x7ff6a815e150 2024-02-19T15:24:07.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.314+0000 7ff6aca00640 1 -- 172.21.15.47:0/3634167123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a815e150 msgr2=0x7ff6a80d07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:07.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.314+0000 7ff6aca00640 1 --2- 172.21.15.47:0/3634167123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a815e150 0x7ff6a80d07d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff6980097c0 tx=0x7ff69801c390 comp rx=0 tx=0).stop 2024-02-19T15:24:07.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.314+0000 7ff6aca00640 1 -- 172.21.15.47:0/3634167123 shutdown_connections 2024-02-19T15:24:07.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.314+0000 7ff6aca00640 1 --2- 172.21.15.47:0/3634167123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a815e150 0x7ff6a80d07d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:07.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.314+0000 7ff6aca00640 1 -- 172.21.15.47:0/3634167123 >> 172.21.15.47:0/3634167123 conn(0x7ff6a80d3de0 msgr2=0x7ff6a80d41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:07.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.314+0000 7ff6aca00640 1 -- 172.21.15.47:0/3634167123 shutdown_connections 2024-02-19T15:24:07.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.314+0000 7ff6aca00640 1 -- 172.21.15.47:0/3634167123 wait complete. 2024-02-19T15:24:07.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.314+0000 7ff6aca00640 1 Processor -- start 2024-02-19T15:24:07.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.314+0000 7ff6aca00640 1 -- start start 2024-02-19T15:24:07.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff6aca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6a80c9400 0x7ff6a80cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:07.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff6aca00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6a80d23c0 0x7ff6a80b8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:07.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff6aca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a815e150 0x7ff6a8156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:07.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff6aca00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a80cc600 con 0x7ff6a80c9400 2024-02-19T15:24:07.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff6aca00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a81564b0 con 0x7ff6a815e150 2024-02-19T15:24:07.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff6aca00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a815b600 con 0x7ff6a80d23c0 2024-02-19T15:24:07.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff6a5e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6a80c9400 0x7ff6a80cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:07.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff6a5e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6a80c9400 0x7ff6a80cd9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43308/0 (socket says 172.21.15.47:43308) 2024-02-19T15:24:07.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff6a5e00640 1 -- 172.21.15.47:0/3903584291 learned_addr learned my addr 172.21.15.47:0/3903584291 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:07.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff6a6800640 1 --2- 172.21.15.47:0/3903584291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a815e150 0x7ff6a8156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:07.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff69fe00640 1 --2- 172.21.15.47:0/3903584291 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6a80d23c0 0x7ff6a80b8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:07.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff6a5e00640 1 -- 172.21.15.47:0/3903584291 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6a80d23c0 msgr2=0x7ff6a80b8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:07.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff6a5e00640 1 --2- 172.21.15.47:0/3903584291 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6a80d23c0 0x7ff6a80b8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:07.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff6a5e00640 1 -- 172.21.15.47:0/3903584291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a815e150 msgr2=0x7ff6a8156ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:07.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff6a5e00640 1 --2- 172.21.15.47:0/3903584291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a815e150 0x7ff6a8156ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:07.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff6a5e00640 1 -- 172.21.15.47:0/3903584291 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff698009420 con 0x7ff6a80c9400 2024-02-19T15:24:07.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff6a6800640 1 --2- 172.21.15.47:0/3903584291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a815e150 0x7ff6a8156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:07.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff69fe00640 1 --2- 172.21.15.47:0/3903584291 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6a80d23c0 0x7ff6a80b8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:07.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff6a5e00640 1 --2- 172.21.15.47:0/3903584291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6a80c9400 0x7ff6a80cd9d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff6980af040 tx=0x7ff69801abd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:07.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff69e000640 1 -- 172.21.15.47:0/3903584291 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6980acea0 con 0x7ff6a80c9400 2024-02-19T15:24:07.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff69e000640 1 -- 172.21.15.47:0/3903584291 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff698004220 con 0x7ff6a80c9400 2024-02-19T15:24:07.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff6aca00640 1 -- 172.21.15.47:0/3903584291 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6a815b7e0 con 0x7ff6a80c9400 2024-02-19T15:24:07.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff69e000640 1 -- 172.21.15.47:0/3903584291 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff698003620 con 0x7ff6a80c9400 2024-02-19T15:24:07.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff6aca00640 1 -- 172.21.15.47:0/3903584291 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6a815bda0 con 0x7ff6a80c9400 2024-02-19T15:24:07.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff6aca00640 1 -- 172.21.15.47:0/3903584291 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6a80cc3a0 con 0x7ff6a80c9400 2024-02-19T15:24:07.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.318+0000 7ff69e000640 1 -- 172.21.15.47:0/3903584291 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff698003780 con 0x7ff6a80c9400 2024-02-19T15:24:07.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.322+0000 7ff69e000640 1 --2- 172.21.15.47:0/3903584291 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff684066e40 0x7ff684069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:07.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.322+0000 7ff69e000640 1 -- 172.21.15.47:0/3903584291 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff69811e5e0 con 0x7ff6a80c9400 2024-02-19T15:24:07.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.322+0000 7ff69fe00640 1 --2- 172.21.15.47:0/3903584291 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff684066e40 0x7ff684069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:07.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.322+0000 7ff69fe00640 1 --2- 172.21.15.47:0/3903584291 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff684066e40 0x7ff684069300 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff6a80ce270 tx=0x7ff690009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:07.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.326+0000 7ff69e000640 1 -- 172.21.15.47:0/3903584291 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff6980eb3d0 con 0x7ff6a80c9400 2024-02-19T15:24:07.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.498+0000 7ff6aca00640 1 -- 172.21.15.47:0/3903584291 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff6a8075360 con 0x7ff6a80c9400 2024-02-19T15:24:07.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.498+0000 7ff69e000640 1 -- 172.21.15.47:0/3903584291 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v45) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff6980eea90 con 0x7ff6a80c9400 2024-02-19T15:24:07.503 INFO:teuthology.orchestra.run.smithi047.stderr:listed 0 entries 2024-02-19T15:24:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.502+0000 7ff6aca00640 1 -- 172.21.15.47:0/3903584291 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff684066e40 msgr2=0x7ff684069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.502+0000 7ff6aca00640 1 --2- 172.21.15.47:0/3903584291 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff684066e40 0x7ff684069300 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff6a80ce270 tx=0x7ff690009040 comp rx=0 tx=0).stop 2024-02-19T15:24:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.502+0000 7ff6aca00640 1 -- 172.21.15.47:0/3903584291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6a80c9400 msgr2=0x7ff6a80cd9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.502+0000 7ff6aca00640 1 --2- 172.21.15.47:0/3903584291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6a80c9400 0x7ff6a80cd9d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff6980af040 tx=0x7ff69801abd0 comp rx=0 tx=0).stop 2024-02-19T15:24:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.502+0000 7ff6aca00640 1 -- 172.21.15.47:0/3903584291 shutdown_connections 2024-02-19T15:24:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.502+0000 7ff6aca00640 1 --2- 172.21.15.47:0/3903584291 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff684066e40 0x7ff684069300 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.502+0000 7ff6aca00640 1 --2- 172.21.15.47:0/3903584291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a815e150 0x7ff6a8156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.502+0000 7ff6aca00640 1 --2- 172.21.15.47:0/3903584291 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6a80d23c0 0x7ff6a80b8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.502+0000 7ff6aca00640 1 --2- 172.21.15.47:0/3903584291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6a80c9400 0x7ff6a80cd9d0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:07.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.502+0000 7ff6aca00640 1 -- 172.21.15.47:0/3903584291 >> 172.21.15.47:0/3903584291 conn(0x7ff6a80d3de0 msgr2=0x7ff6a80cfc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:07.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.502+0000 7ff6aca00640 1 -- 172.21.15.47:0/3903584291 shutdown_connections 2024-02-19T15:24:07.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.502+0000 7ff6aca00640 1 -- 172.21.15.47:0/3903584291 wait complete. 2024-02-19T15:24:07.524 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-02-19T15:24:07.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:07.524 DEBUG:teuthology.orchestra.run.smithi047:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-02-19T15:24:07.631 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi047.front.sepia.ceph.com 2024-02-19T15:24:07.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:07.631 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-19T15:24:07.641 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:24:07.641 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:07.650 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:24:07.650 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:24:07.650 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:07.694 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:24:07.695 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:24:07.695 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:07.703 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:24:07.703 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:24:07.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.878+0000 7f759b600640 1 Processor -- start 2024-02-19T15:24:07.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.878+0000 7f759b600640 1 -- start start 2024-02-19T15:24:07.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.878+0000 7f759b600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f759406d650 0x7f75940d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.878+0000 7f759b600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f75940cded0 0x7f75940ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.878+0000 7f759b600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75940cce30 con 0x7f75940ceb40 2024-02-19T15:24:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.878+0000 7f759b600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75940cefd0 con 0x7f759406d650 2024-02-19T15:24:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.878+0000 7f759b600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75940cd110 con 0x7f75940cded0 2024-02-19T15:24:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.882+0000 7f7598c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f759406d650 0x7f75940d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.882+0000 7f7598c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f759406d650 0x7f75940d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43320/0 (socket says 172.21.15.47:43320) 2024-02-19T15:24:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.882+0000 7f7598c00640 1 -- 172.21.15.47:0/455415542 learned_addr learned my addr 172.21.15.47:0/455415542 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.882+0000 7f758be00640 1 --2- 172.21.15.47:0/455415542 >> v2:172.21.15.116:3300/0 conn(0x7f75940cded0 0x7f75940ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.882+0000 7f7598c00640 1 -- 172.21.15.47:0/455415542 >> v2:172.21.15.116:3300/0 conn(0x7f75940cded0 msgr2=0x7f75940ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.882+0000 7f7598c00640 1 --2- 172.21.15.47:0/455415542 >> v2:172.21.15.116:3300/0 conn(0x7f75940cded0 0x7f75940ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.882+0000 7f7598c00640 1 -- 172.21.15.47:0/455415542 >> v1:172.21.15.116:6789/0 conn(0x7f75940ceb40 legacy=0x7f7594161d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:07.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.882+0000 7f7598c00640 1 -- 172.21.15.47:0/455415542 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75940cd3f0 con 0x7f759406d650 2024-02-19T15:24:07.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.882+0000 7f758be00640 1 --2- 172.21.15.47:0/455415542 >> v2:172.21.15.116:3300/0 conn(0x7f75940cded0 0x7f75940ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:07.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.882+0000 7f7598c00640 1 --2- 172.21.15.47:0/455415542 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f759406d650 0x7f75940d08b0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f759406b950 tx=0x7f758401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a79e9330993e525 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:07.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.882+0000 7f7593e00640 1 -- 172.21.15.47:0/455415542 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75840ab600 con 0x7f759406d650 2024-02-19T15:24:07.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.882+0000 7f7593e00640 1 -- 172.21.15.47:0/455415542 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75840a7050 con 0x7f759406d650 2024-02-19T15:24:07.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.882+0000 7f7593e00640 1 -- 172.21.15.47:0/455415542 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75840ab980 con 0x7f759406d650 2024-02-19T15:24:07.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f759b600640 1 -- 172.21.15.47:0/455415542 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f759406d650 msgr2=0x7f75940d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:07.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f759b600640 1 --2- 172.21.15.47:0/455415542 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f759406d650 0x7f75940d08b0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f759406b950 tx=0x7f758401c420 comp rx=0 tx=0).stop 2024-02-19T15:24:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f759b600640 1 -- 172.21.15.47:0/455415542 shutdown_connections 2024-02-19T15:24:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f759b600640 1 --2- 172.21.15.47:0/455415542 >> v2:172.21.15.116:3300/0 conn(0x7f75940cded0 0x7f75940ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f759b600640 1 --2- 172.21.15.47:0/455415542 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f759406d650 0x7f75940d08b0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f759b600640 1 -- 172.21.15.47:0/455415542 >> 172.21.15.47:0/455415542 conn(0x7f75940d1bf0 msgr2=0x7f75940d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f759b600640 1 -- 172.21.15.47:0/455415542 shutdown_connections 2024-02-19T15:24:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f759b600640 1 -- 172.21.15.47:0/455415542 wait complete. 2024-02-19T15:24:07.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f759b600640 1 Processor -- start 2024-02-19T15:24:07.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f759b600640 1 -- start start 2024-02-19T15:24:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f759b600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f759406d650 0x7f7594152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f759b600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75940cded0 0x7f7594160f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f759b600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75940ceb40 0x7f759406eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f759b600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759415fc90 con 0x7f75940ceb40 2024-02-19T15:24:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f759b600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7594160120 con 0x7f759406d650 2024-02-19T15:24:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f759b600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759415eda0 con 0x7f75940cded0 2024-02-19T15:24:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f7599600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75940ceb40 0x7f759406eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f7599600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75940ceb40 0x7f759406eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43324/0 (socket says 172.21.15.47:43324) 2024-02-19T15:24:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f7599600640 1 -- 172.21.15.47:0/1791699396 learned_addr learned my addr 172.21.15.47:0/1791699396 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f7598c00640 1 --2- 172.21.15.47:0/1791699396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f759406d650 0x7f7594152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f758be00640 1 --2- 172.21.15.47:0/1791699396 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75940cded0 0x7f7594160f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f7599600640 1 -- 172.21.15.47:0/1791699396 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75940cded0 msgr2=0x7f7594160f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f7599600640 1 --2- 172.21.15.47:0/1791699396 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75940cded0 0x7f7594160f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f7599600640 1 -- 172.21.15.47:0/1791699396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f759406d650 msgr2=0x7f7594152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f7599600640 1 --2- 172.21.15.47:0/1791699396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f759406d650 0x7f7594152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.886+0000 7f7599600640 1 -- 172.21.15.47:0/1791699396 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75840094d0 con 0x7f75940ceb40 2024-02-19T15:24:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.890+0000 7f7599600640 1 --2- 172.21.15.47:0/1791699396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75940ceb40 0x7f759406eaf0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f758c00b970 tx=0x7f758c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:07.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.890+0000 7f7591600640 1 -- 172.21.15.47:0/1791699396 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758c00cb40 con 0x7f75940ceb40 2024-02-19T15:24:07.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.890+0000 7f7591600640 1 -- 172.21.15.47:0/1791699396 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758c00cca0 con 0x7f75940ceb40 2024-02-19T15:24:07.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.890+0000 7f759b600640 1 -- 172.21.15.47:0/1791699396 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f759415f080 con 0x7f75940ceb40 2024-02-19T15:24:07.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.890+0000 7f7591600640 1 -- 172.21.15.47:0/1791699396 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758c014680 con 0x7f75940ceb40 2024-02-19T15:24:07.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.890+0000 7f759b600640 1 -- 172.21.15.47:0/1791699396 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f759415f640 con 0x7f75940ceb40 2024-02-19T15:24:07.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.890+0000 7f759b600640 1 -- 172.21.15.47:0/1791699396 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f759415fe40 con 0x7f75940ceb40 2024-02-19T15:24:07.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.890+0000 7f7591600640 1 -- 172.21.15.47:0/1791699396 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f758c01d3b0 con 0x7f75940ceb40 2024-02-19T15:24:07.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.890+0000 7f7591600640 1 --2- 172.21.15.47:0/1791699396 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7570066e40 0x7f7570069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.890+0000 7f7591600640 1 -- 172.21.15.47:0/1791699396 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f758c084830 con 0x7f75940ceb40 2024-02-19T15:24:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.894+0000 7f7598c00640 1 --2- 172.21.15.47:0/1791699396 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7570066e40 0x7f7570069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:07.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.894+0000 7f7598c00640 1 --2- 172.21.15.47:0/1791699396 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7570066e40 0x7f7570069300 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f75940ce590 tx=0x7f7584008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:07.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:07.894+0000 7f7591600640 1 -- 172.21.15.47:0/1791699396 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f759415fe40 con 0x7f75940ceb40 2024-02-19T15:24:08.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.094+0000 7f759b600640 1 -- 172.21.15.47:0/1791699396 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7594075360 con 0x7f75940ceb40 2024-02-19T15:24:08.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.094+0000 7f7591600640 1 -- 172.21.15.47:0/1791699396 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f758c0516c0 con 0x7f75940ceb40 2024-02-19T15:24:08.097 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:08.098 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5132,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/548836925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":548836925},{"type":"v1","addr":"172.21.15.116:6836","nonce":548836925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5135,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3530214438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3530214438},{"type":"v1","addr":"172.21.15.116:6837","nonce":3530214438}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/3433978846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3433978846},{"type":"v1","addr":"172.21.15.47:6834","nonce":3433978846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:23:42.342661+0000","modified":"2024-02-19T15:23:45.352372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5514":{"gid":5514,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3137466738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3137466738},{"type":"v1","addr":"172.21.15.47:6835","nonce":3137466738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-02-19T15:24:08.098 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 25 2024-02-19T15:24:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.098+0000 7f759b600640 1 -- 172.21.15.47:0/1791699396 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7570066e40 msgr2=0x7f7570069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.098+0000 7f759b600640 1 --2- 172.21.15.47:0/1791699396 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7570066e40 0x7f7570069300 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f75940ce590 tx=0x7f7584008040 comp rx=0 tx=0).stop 2024-02-19T15:24:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.098+0000 7f759b600640 1 -- 172.21.15.47:0/1791699396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75940ceb40 msgr2=0x7f759406eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.098+0000 7f759b600640 1 --2- 172.21.15.47:0/1791699396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75940ceb40 0x7f759406eaf0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f758c00b970 tx=0x7f758c00be30 comp rx=0 tx=0).stop 2024-02-19T15:24:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.098+0000 7f759b600640 1 -- 172.21.15.47:0/1791699396 shutdown_connections 2024-02-19T15:24:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.098+0000 7f759b600640 1 --2- 172.21.15.47:0/1791699396 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7570066e40 0x7f7570069300 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.098+0000 7f759b600640 1 --2- 172.21.15.47:0/1791699396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75940ceb40 0x7f759406eaf0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.098+0000 7f759b600640 1 --2- 172.21.15.47:0/1791699396 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75940cded0 0x7f7594160f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.098+0000 7f759b600640 1 --2- 172.21.15.47:0/1791699396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f759406d650 0x7f7594152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.098+0000 7f759b600640 1 -- 172.21.15.47:0/1791699396 >> 172.21.15.47:0/1791699396 conn(0x7f75940d1bf0 msgr2=0x7f75940b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.098+0000 7f759b600640 1 -- 172.21.15.47:0/1791699396 shutdown_connections 2024-02-19T15:24:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.098+0000 7f759b600640 1 -- 172.21.15.47:0/1791699396 wait complete. 2024-02-19T15:24:08.120 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:24:08.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.258+0000 7f8878c00640 1 Processor -- start 2024-02-19T15:24:08.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.258+0000 7f8878c00640 1 -- start start 2024-02-19T15:24:08.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.258+0000 7f8878c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8874145590 0x7f88740727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:08.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.258+0000 7f8878c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f88740cea80 0x7f887406c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:08.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.258+0000 7f8878c00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f887415d280 con 0x7f887415c390 2024-02-19T15:24:08.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.258+0000 7f8878c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8874070ff0 con 0x7f88740cea80 2024-02-19T15:24:08.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.258+0000 7f8878c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88740709e0 con 0x7f8874145590 2024-02-19T15:24:08.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.258+0000 7f8869400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f88740cea80 0x7f887406c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:08.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.258+0000 7f8869400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f88740cea80 0x7f887406c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43326/0 (socket says 172.21.15.47:43326) 2024-02-19T15:24:08.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.258+0000 7f8869400640 1 -- 172.21.15.47:0/4025144444 learned_addr learned my addr 172.21.15.47:0/4025144444 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:08.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.258+0000 7f8872800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f887415c390 0x7f887415c820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51974/0 (socket says 172.21.15.47:51974) 2024-02-19T15:24:08.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.258+0000 7f8871e00640 1 --2- 172.21.15.47:0/4025144444 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8874145590 0x7f88740727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:08.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.258+0000 7f8869400640 1 -- 172.21.15.47:0/4025144444 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8874145590 msgr2=0x7f88740727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:08.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.258+0000 7f8869400640 1 --2- 172.21.15.47:0/4025144444 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8874145590 0x7f88740727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.258+0000 7f8869400640 1 -- 172.21.15.47:0/4025144444 >> v1:172.21.15.47:6789/0 conn(0x7f887415c390 legacy=0x7f887415c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:08.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.258+0000 7f8869400640 1 -- 172.21.15.47:0/4025144444 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887406cee0 con 0x7f88740cea80 2024-02-19T15:24:08.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.258+0000 7f8871e00640 1 --2- 172.21.15.47:0/4025144444 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8874145590 0x7f88740727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:08.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.266+0000 7f8869400640 1 --2- 172.21.15.47:0/4025144444 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88740cea80 0x7f887406c9a0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f885c009e80 tx=0x7f885c01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e40fc6f57f62df89 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:08.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.266+0000 7f8871400640 1 -- 172.21.15.47:0/4025144444 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c0a57d0 con 0x7f88740cea80 2024-02-19T15:24:08.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.266+0000 7f8871400640 1 -- 172.21.15.47:0/4025144444 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f885c0a5930 con 0x7f88740cea80 2024-02-19T15:24:08.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.266+0000 7f8871400640 1 -- 172.21.15.47:0/4025144444 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c0a5cc0 con 0x7f88740cea80 2024-02-19T15:24:08.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.266+0000 7f8878c00640 1 -- 172.21.15.47:0/4025144444 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88740cea80 msgr2=0x7f887406c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:08.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.266+0000 7f8878c00640 1 --2- 172.21.15.47:0/4025144444 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88740cea80 0x7f887406c9a0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f885c009e80 tx=0x7f885c01ca10 comp rx=0 tx=0).stop 2024-02-19T15:24:08.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.266+0000 7f8878c00640 1 -- 172.21.15.47:0/4025144444 shutdown_connections 2024-02-19T15:24:08.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.266+0000 7f8878c00640 1 --2- 172.21.15.47:0/4025144444 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88740cea80 0x7f887406c9a0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.266+0000 7f8878c00640 1 --2- 172.21.15.47:0/4025144444 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8874145590 0x7f88740727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.266+0000 7f8878c00640 1 -- 172.21.15.47:0/4025144444 >> 172.21.15.47:0/4025144444 conn(0x7f88740b16a0 msgr2=0x7f88740b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:08.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.266+0000 7f8878c00640 1 -- 172.21.15.47:0/4025144444 shutdown_connections 2024-02-19T15:24:08.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.266+0000 7f8878c00640 1 -- 172.21.15.47:0/4025144444 wait complete. 2024-02-19T15:24:08.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.266+0000 7f8878c00640 1 Processor -- start 2024-02-19T15:24:08.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.266+0000 7f8878c00640 1 -- start start 2024-02-19T15:24:08.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f8878c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88740cea80 0x7f88740b4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:08.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f8878c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8874145590 0x7f887413f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:08.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f8878c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f887415c390 0x7f887415add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:08.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f8878c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88741439b0 con 0x7f88740cea80 2024-02-19T15:24:08.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f8878c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8874142da0 con 0x7f887415c390 2024-02-19T15:24:08.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f8878c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8874143080 con 0x7f8874145590 2024-02-19T15:24:08.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f8871e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88740cea80 0x7f88740b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:08.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f8871e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88740cea80 0x7f88740b4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43336/0 (socket says 172.21.15.47:43336) 2024-02-19T15:24:08.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f8871e00640 1 -- 172.21.15.47:0/479644592 learned_addr learned my addr 172.21.15.47:0/479644592 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:08.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f8871e00640 1 -- 172.21.15.47:0/479644592 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8874145590 msgr2=0x7f887413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:08.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f8871e00640 1 --2- 172.21.15.47:0/479644592 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8874145590 0x7f887413f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f8871e00640 1 -- 172.21.15.47:0/479644592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f887415c390 msgr2=0x7f887415add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:08.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f8871e00640 1 --2- 172.21.15.47:0/479644592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f887415c390 0x7f887415add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f8871e00640 1 -- 172.21.15.47:0/479644592 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f885c009b30 con 0x7f88740cea80 2024-02-19T15:24:08.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f8871e00640 1 --2- 172.21.15.47:0/479644592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88740cea80 0x7f88740b4570 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f886400e810 tx=0x7f886400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:08.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f886aa00640 1 -- 172.21.15.47:0/479644592 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886400c8d0 con 0x7f88740cea80 2024-02-19T15:24:08.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f886aa00640 1 -- 172.21.15.47:0/479644592 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886400ca30 con 0x7f88740cea80 2024-02-19T15:24:08.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f886aa00640 1 -- 172.21.15.47:0/479644592 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886400cdc0 con 0x7f88740cea80 2024-02-19T15:24:08.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f8878c00640 1 -- 172.21.15.47:0/479644592 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8874143360 con 0x7f88740cea80 2024-02-19T15:24:08.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f8878c00640 1 -- 172.21.15.47:0/479644592 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f887415bcd0 con 0x7f88740cea80 2024-02-19T15:24:08.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f8878c00640 1 -- 172.21.15.47:0/479644592 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8874075360 con 0x7f88740cea80 2024-02-19T15:24:08.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.270+0000 7f886aa00640 1 -- 172.21.15.47:0/479644592 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8864013410 con 0x7f88740cea80 2024-02-19T15:24:08.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.274+0000 7f886aa00640 1 --2- 172.21.15.47:0/479644592 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8850066e70 0x7f8850069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:08.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.274+0000 7f886aa00640 1 -- 172.21.15.47:0/479644592 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8864084900 con 0x7f88740cea80 2024-02-19T15:24:08.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.274+0000 7f8869400640 1 --2- 172.21.15.47:0/479644592 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8850066e70 0x7f8850069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:08.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.274+0000 7f8869400640 1 --2- 172.21.15.47:0/479644592 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8850066e70 0x7f8850069330 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f885c0ae000 tx=0x7f885c004ac0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:08.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.282+0000 7f886aa00640 1 -- 172.21.15.47:0/479644592 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8864051650 con 0x7f88740cea80 2024-02-19T15:24:08.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.446+0000 7f8878c00640 1 -- 172.21.15.47:0/479644592 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88740ada50 con 0x7f88740cea80 2024-02-19T15:24:08.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.446+0000 7f886aa00640 1 -- 172.21.15.47:0/479644592 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16625 (secure 0 0 0) 0x7f8864009480 con 0x7f88740cea80 2024-02-19T15:24:08.449 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:08.449 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":45,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:23:43.941118+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:23:40.751741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-02-19T15:23:41.338930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:24:08.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.446+0000 7f8878c00640 1 -- 172.21.15.47:0/479644592 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8850066e70 msgr2=0x7f8850069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:08.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.446+0000 7f8878c00640 1 --2- 172.21.15.47:0/479644592 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8850066e70 0x7f8850069330 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f885c0ae000 tx=0x7f885c004ac0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.446+0000 7f8878c00640 1 -- 172.21.15.47:0/479644592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88740cea80 msgr2=0x7f88740b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:08.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.446+0000 7f8878c00640 1 --2- 172.21.15.47:0/479644592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88740cea80 0x7f88740b4570 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f886400e810 tx=0x7f886400ecd0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.446+0000 7f8878c00640 1 -- 172.21.15.47:0/479644592 shutdown_connections 2024-02-19T15:24:08.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.446+0000 7f8878c00640 1 --2- 172.21.15.47:0/479644592 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8850066e70 0x7f8850069330 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.446+0000 7f8878c00640 1 --2- 172.21.15.47:0/479644592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f887415c390 0x7f887415add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.446+0000 7f8878c00640 1 --2- 172.21.15.47:0/479644592 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8874145590 0x7f887413f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.446+0000 7f8878c00640 1 --2- 172.21.15.47:0/479644592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88740cea80 0x7f88740b4570 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.446+0000 7f8878c00640 1 -- 172.21.15.47:0/479644592 >> 172.21.15.47:0/479644592 conn(0x7f88740b16a0 msgr2=0x7f88740ae250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:08.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.450+0000 7f8878c00640 1 -- 172.21.15.47:0/479644592 shutdown_connections 2024-02-19T15:24:08.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.450+0000 7f8878c00640 1 -- 172.21.15.47:0/479644592 wait complete. 2024-02-19T15:24:08.476 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:23:40.751741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:23:41.338930+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:24:08.477 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:24:08.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.618+0000 7efdf7a00640 1 Processor -- start 2024-02-19T15:24:08.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.618+0000 7efdf7a00640 1 -- start start 2024-02-19T15:24:08.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.618+0000 7efdf7a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efdf00d2650 0x7efdf006c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:08.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.618+0000 7efdf7a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdf015e280 0x7efdf006c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:08.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.618+0000 7efdf7a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf00cc620 con 0x7efdf00ce3b0 2024-02-19T15:24:08.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.618+0000 7efdf7a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf00d12e0 con 0x7efdf00d2650 2024-02-19T15:24:08.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.618+0000 7efdf7a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf00d1940 con 0x7efdf015e280 2024-02-19T15:24:08.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.618+0000 7efdf5a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdf015e280 0x7efdf006c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:08.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.618+0000 7efdefe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efdf00d2650 0x7efdf006c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:08.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.618+0000 7efdf5a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdf015e280 0x7efdf006c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:38684/0 (socket says 172.21.15.47:38684) 2024-02-19T15:24:08.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.618+0000 7efdf5a00640 1 -- 172.21.15.47:0/3165114797 learned_addr learned my addr 172.21.15.47:0/3165114797 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:08.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.618+0000 7efdf5000640 1 --1- 172.21.15.47:0/3165114797 >> v1:172.21.15.116:6789/0 conn(0x7efdf00ce3b0 0x7efdf00ce790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:50920/0 (socket says 172.21.15.47:50920) 2024-02-19T15:24:08.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.618+0000 7efdf5a00640 1 -- 172.21.15.47:0/3165114797 >> v2:172.21.15.116:3300/0 conn(0x7efdf00d2650 msgr2=0x7efdf006c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:08.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.618+0000 7efdf5a00640 1 --2- 172.21.15.47:0/3165114797 >> v2:172.21.15.116:3300/0 conn(0x7efdf00d2650 0x7efdf006c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.618+0000 7efdf5a00640 1 -- 172.21.15.47:0/3165114797 >> v1:172.21.15.116:6789/0 conn(0x7efdf00ce3b0 legacy=0x7efdf00ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:08.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.618+0000 7efdef400640 1 -- 172.21.15.47:0/3165114797 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647453686 0 0) 0x7efdf00cc620 con 0x7efdf00ce3b0 2024-02-19T15:24:08.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.618+0000 7efdf5a00640 1 -- 172.21.15.47:0/3165114797 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf00d1000 con 0x7efdf015e280 2024-02-19T15:24:08.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.622+0000 7efdf5a00640 1 --2- 172.21.15.47:0/3165114797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdf015e280 0x7efdf006c710 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efde8009e50 tx=0x7efde801c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=badec25f81dddafa server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:08.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.622+0000 7efdef400640 1 -- 172.21.15.47:0/3165114797 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde80a56e0 con 0x7efdf015e280 2024-02-19T15:24:08.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.622+0000 7efdef400640 1 -- 172.21.15.47:0/3165114797 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde80a5840 con 0x7efdf015e280 2024-02-19T15:24:08.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.622+0000 7efdef400640 1 -- 172.21.15.47:0/3165114797 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde80a5bd0 con 0x7efdf015e280 2024-02-19T15:24:08.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.622+0000 7efdf7a00640 1 -- 172.21.15.47:0/3165114797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdf015e280 msgr2=0x7efdf006c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:08.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.622+0000 7efdf7a00640 1 --2- 172.21.15.47:0/3165114797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdf015e280 0x7efdf006c710 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efde8009e50 tx=0x7efde801c910 comp rx=0 tx=0).stop 2024-02-19T15:24:08.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.622+0000 7efdf7a00640 1 -- 172.21.15.47:0/3165114797 shutdown_connections 2024-02-19T15:24:08.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.622+0000 7efdf7a00640 1 --2- 172.21.15.47:0/3165114797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdf015e280 0x7efdf006c710 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.622+0000 7efdf7a00640 1 --2- 172.21.15.47:0/3165114797 >> v2:172.21.15.116:3300/0 conn(0x7efdf00d2650 0x7efdf006c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.622+0000 7efdf7a00640 1 -- 172.21.15.47:0/3165114797 >> 172.21.15.47:0/3165114797 conn(0x7efdf00d1b30 msgr2=0x7efdf00d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:08.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.622+0000 7efdf7a00640 1 -- 172.21.15.47:0/3165114797 shutdown_connections 2024-02-19T15:24:08.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.622+0000 7efdf7a00640 1 -- 172.21.15.47:0/3165114797 wait complete. 2024-02-19T15:24:08.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdf7a00640 1 Processor -- start 2024-02-19T15:24:08.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdf7a00640 1 -- start start 2024-02-19T15:24:08.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdf7a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdf00ce3b0 0x7efdf00ca070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:08.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdf7a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdf00d2650 0x7efdf0156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:08.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdf7a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf015e280 0x7efdf0152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:08.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdf7a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf00cb3a0 con 0x7efdf00ce3b0 2024-02-19T15:24:08.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdf7a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf0156f70 con 0x7efdf015e280 2024-02-19T15:24:08.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdf7a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf00ca720 con 0x7efdf00d2650 2024-02-19T15:24:08.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdf5000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdf00ce3b0 0x7efdf00ca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:08.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdf5000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdf00ce3b0 0x7efdf00ca070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43340/0 (socket says 172.21.15.47:43340) 2024-02-19T15:24:08.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdf5000640 1 -- 172.21.15.47:0/3612529068 learned_addr learned my addr 172.21.15.47:0/3612529068 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:08.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdf5a00640 1 --2- 172.21.15.47:0/3612529068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf015e280 0x7efdf0152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:08.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdefe00640 1 --2- 172.21.15.47:0/3612529068 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdf00d2650 0x7efdf0156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:08.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdf5000640 1 -- 172.21.15.47:0/3612529068 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdf00d2650 msgr2=0x7efdf0156a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:08.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdf5000640 1 --2- 172.21.15.47:0/3612529068 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdf00d2650 0x7efdf0156a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdf5000640 1 -- 172.21.15.47:0/3612529068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf015e280 msgr2=0x7efdf0152150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:08.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdf5000640 1 --2- 172.21.15.47:0/3612529068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf015e280 0x7efdf0152150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdf5000640 1 -- 172.21.15.47:0/3612529068 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde8009b30 con 0x7efdf00ce3b0 2024-02-19T15:24:08.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdf5000640 1 --2- 172.21.15.47:0/3612529068 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdf00ce3b0 0x7efdf00ca070 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7efde400c410 tx=0x7efde400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:08.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdecc00640 1 -- 172.21.15.47:0/3612529068 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde40074e0 con 0x7efdf00ce3b0 2024-02-19T15:24:08.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdecc00640 1 -- 172.21.15.47:0/3612529068 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde4007640 con 0x7efdf00ce3b0 2024-02-19T15:24:08.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdecc00640 1 -- 172.21.15.47:0/3612529068 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde4004a30 con 0x7efdf00ce3b0 2024-02-19T15:24:08.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.626+0000 7efdf7a00640 1 -- 172.21.15.47:0/3612529068 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf00caa00 con 0x7efdf00ce3b0 2024-02-19T15:24:08.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.630+0000 7efdf7a00640 1 -- 172.21.15.47:0/3612529068 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf00cafc0 con 0x7efdf00ce3b0 2024-02-19T15:24:08.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.630+0000 7efdf7a00640 1 -- 172.21.15.47:0/3612529068 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf00cb4e0 con 0x7efdf00ce3b0 2024-02-19T15:24:08.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.630+0000 7efdecc00640 1 -- 172.21.15.47:0/3612529068 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efde4004b90 con 0x7efdf00ce3b0 2024-02-19T15:24:08.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.630+0000 7efdecc00640 1 --2- 172.21.15.47:0/3612529068 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efdd0066e40 0x7efdd0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:08.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.630+0000 7efdecc00640 1 -- 172.21.15.47:0/3612529068 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efde40846f0 con 0x7efdf00ce3b0 2024-02-19T15:24:08.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.630+0000 7efdefe00640 1 --2- 172.21.15.47:0/3612529068 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efdd0066e40 0x7efdd0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:08.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.630+0000 7efdefe00640 1 --2- 172.21.15.47:0/3612529068 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efdd0066e40 0x7efdd0069300 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7efdf00b2ca0 tx=0x7efddc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:08.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.634+0000 7efdecc00640 1 -- 172.21.15.47:0/3612529068 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7efdf00cb4e0 con 0x7efdf00ce3b0 2024-02-19T15:24:08.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.798+0000 7efdf7a00640 1 -- 172.21.15.47:0/3612529068 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efdf00cb4e0 con 0x7efdf00ce3b0 2024-02-19T15:24:08.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.798+0000 7efdecc00640 1 -- 172.21.15.47:0/3612529068 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) v1 ==== 93+0+10 (secure 0 0 0) 0x7efdf00cb4e0 con 0x7efdf00ce3b0 2024-02-19T15:24:08.800 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:24:08.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.798+0000 7efdf7a00640 1 -- 172.21.15.47:0/3612529068 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efdd0066e40 msgr2=0x7efdd0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:08.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.798+0000 7efdf7a00640 1 --2- 172.21.15.47:0/3612529068 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efdd0066e40 0x7efdd0069300 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7efdf00b2ca0 tx=0x7efddc009190 comp rx=0 tx=0).stop 2024-02-19T15:24:08.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.798+0000 7efdf7a00640 1 -- 172.21.15.47:0/3612529068 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdf00ce3b0 msgr2=0x7efdf00ca070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:08.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.798+0000 7efdf7a00640 1 --2- 172.21.15.47:0/3612529068 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdf00ce3b0 0x7efdf00ca070 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7efde400c410 tx=0x7efde400c8d0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.798+0000 7efdf7a00640 1 -- 172.21.15.47:0/3612529068 shutdown_connections 2024-02-19T15:24:08.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.798+0000 7efdf7a00640 1 --2- 172.21.15.47:0/3612529068 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efdd0066e40 0x7efdd0069300 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.798+0000 7efdf7a00640 1 --2- 172.21.15.47:0/3612529068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf015e280 0x7efdf0152150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.798+0000 7efdf7a00640 1 --2- 172.21.15.47:0/3612529068 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdf00d2650 0x7efdf0156a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.798+0000 7efdf7a00640 1 --2- 172.21.15.47:0/3612529068 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdf00ce3b0 0x7efdf00ca070 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.798+0000 7efdf7a00640 1 -- 172.21.15.47:0/3612529068 >> 172.21.15.47:0/3612529068 conn(0x7efdf00d1b30 msgr2=0x7efdf00c6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:08.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.798+0000 7efdf7a00640 1 -- 172.21.15.47:0/3612529068 shutdown_connections 2024-02-19T15:24:08.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.798+0000 7efdf7a00640 1 -- 172.21.15.47:0/3612529068 wait complete. 2024-02-19T15:24:08.821 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:24:08.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.962+0000 7efc92000640 1 Processor -- start 2024-02-19T15:24:08.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.962+0000 7efc92000640 1 -- start start 2024-02-19T15:24:08.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.962+0000 7efc92000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc8c0c7e80 0x7efc8c0d0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:08.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.962+0000 7efc92000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efc8c0c8260 0x7efc8c0d0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:08.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.962+0000 7efc92000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efc8c0cfbf0 0x7efc8c0c5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:08.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.962+0000 7efc92000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc8c0ceee0 con 0x7efc8c0cfbf0 2024-02-19T15:24:08.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.962+0000 7efc92000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc8c0cdc60 con 0x7efc8c0c8260 2024-02-19T15:24:08.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.962+0000 7efc92000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c0d2a40 con 0x7efc8c0c7e80 2024-02-19T15:24:08.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.962+0000 7efc8be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efc8c0cfbf0 0x7efc8c0c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:08.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.962+0000 7efc8be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efc8c0cfbf0 0x7efc8c0c5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43352/0 (socket says 172.21.15.47:43352) 2024-02-19T15:24:08.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.962+0000 7efc8be00640 1 -- 172.21.15.47:0/2741636360 learned_addr learned my addr 172.21.15.47:0/2741636360 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:08.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.962+0000 7efc8b400640 1 --2- 172.21.15.47:0/2741636360 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc8c0c7e80 0x7efc8c0d0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:08.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.962+0000 7efc82a00640 1 --2- 172.21.15.47:0/2741636360 >> v2:172.21.15.116:3300/0 conn(0x7efc8c0c8260 0x7efc8c0d0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:08.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.966+0000 7efc8be00640 1 -- 172.21.15.47:0/2741636360 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc8c0c7e80 msgr2=0x7efc8c0d0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:08.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.966+0000 7efc8be00640 1 --2- 172.21.15.47:0/2741636360 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc8c0c7e80 0x7efc8c0d0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.966+0000 7efc8be00640 1 -- 172.21.15.47:0/2741636360 >> v2:172.21.15.116:3300/0 conn(0x7efc8c0c8260 msgr2=0x7efc8c0d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:08.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.966+0000 7efc8be00640 1 --2- 172.21.15.47:0/2741636360 >> v2:172.21.15.116:3300/0 conn(0x7efc8c0c8260 0x7efc8c0d0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.966+0000 7efc8be00640 1 -- 172.21.15.47:0/2741636360 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c0cd980 con 0x7efc8c0cfbf0 2024-02-19T15:24:08.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.966+0000 7efc82a00640 1 --2- 172.21.15.47:0/2741636360 >> v2:172.21.15.116:3300/0 conn(0x7efc8c0c8260 0x7efc8c0d0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:08.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.966+0000 7efc8be00640 1 --2- 172.21.15.47:0/2741636360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc8c0cfbf0 0x7efc8c0c5680 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7efc8c06bc80 tx=0x7efc8401c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=356719b331f50277 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:08.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.966+0000 7efc8aa00640 1 -- 172.21.15.47:0/2741636360 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc840a57a0 con 0x7efc8c0cfbf0 2024-02-19T15:24:08.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.966+0000 7efc8aa00640 1 -- 172.21.15.47:0/2741636360 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc840a5900 con 0x7efc8c0cfbf0 2024-02-19T15:24:08.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.966+0000 7efc8aa00640 1 -- 172.21.15.47:0/2741636360 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc840a5c90 con 0x7efc8c0cfbf0 2024-02-19T15:24:08.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.966+0000 7efc92000640 1 -- 172.21.15.47:0/2741636360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc8c0cfbf0 msgr2=0x7efc8c0c5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:08.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.966+0000 7efc92000640 1 --2- 172.21.15.47:0/2741636360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc8c0cfbf0 0x7efc8c0c5680 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7efc8c06bc80 tx=0x7efc8401c9c0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.966+0000 7efc92000640 1 -- 172.21.15.47:0/2741636360 shutdown_connections 2024-02-19T15:24:08.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.966+0000 7efc92000640 1 --2- 172.21.15.47:0/2741636360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc8c0cfbf0 0x7efc8c0c5680 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.966+0000 7efc92000640 1 --2- 172.21.15.47:0/2741636360 >> v2:172.21.15.116:3300/0 conn(0x7efc8c0c8260 0x7efc8c0d0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.966+0000 7efc92000640 1 --2- 172.21.15.47:0/2741636360 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc8c0c7e80 0x7efc8c0d0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.966+0000 7efc92000640 1 -- 172.21.15.47:0/2741636360 >> 172.21.15.47:0/2741636360 conn(0x7efc8c0d1b30 msgr2=0x7efc8c0d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:08.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc92000640 1 -- 172.21.15.47:0/2741636360 shutdown_connections 2024-02-19T15:24:08.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc92000640 1 -- 172.21.15.47:0/2741636360 wait complete. 2024-02-19T15:24:08.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc92000640 1 Processor -- start 2024-02-19T15:24:08.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc92000640 1 -- start start 2024-02-19T15:24:08.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc92000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc8c0c7e80 0x7efc8c157090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:08.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc92000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc8c0c8260 0x7efc8c153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:08.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc92000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc8c0cfbf0 0x7efc8c0cb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:08.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc92000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c154520 con 0x7efc8c0c8260 2024-02-19T15:24:08.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc92000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c155ff0 con 0x7efc8c0c7e80 2024-02-19T15:24:08.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc92000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c1562d0 con 0x7efc8c0cfbf0 2024-02-19T15:24:08.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc82a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc8c0c8260 0x7efc8c153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:08.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc82a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc8c0c8260 0x7efc8c153fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43358/0 (socket says 172.21.15.47:43358) 2024-02-19T15:24:08.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc8b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc8c0c7e80 0x7efc8c157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:08.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc82a00640 1 -- 172.21.15.47:0/3973233127 learned_addr learned my addr 172.21.15.47:0/3973233127 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:08.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc8be00640 1 --2- 172.21.15.47:0/3973233127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc8c0cfbf0 0x7efc8c0cb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:08.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc82a00640 1 -- 172.21.15.47:0/3973233127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc8c0cfbf0 msgr2=0x7efc8c0cb590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:08.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc82a00640 1 --2- 172.21.15.47:0/3973233127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc8c0cfbf0 0x7efc8c0cb590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc82a00640 1 -- 172.21.15.47:0/3973233127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc8c0c7e80 msgr2=0x7efc8c157090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:08.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc82a00640 1 --2- 172.21.15.47:0/3973233127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc8c0c7e80 0x7efc8c157090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:08.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc82a00640 1 -- 172.21.15.47:0/3973233127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc84009b30 con 0x7efc8c0c8260 2024-02-19T15:24:08.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc82a00640 1 --2- 172.21.15.47:0/3973233127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc8c0c8260 0x7efc8c153fe0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7efc7400e810 tx=0x7efc7400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:08.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc83e00640 1 -- 172.21.15.47:0/3973233127 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7400c6d0 con 0x7efc8c0c8260 2024-02-19T15:24:08.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc83e00640 1 -- 172.21.15.47:0/3973233127 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc7400c830 con 0x7efc8c0c8260 2024-02-19T15:24:08.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.970+0000 7efc83e00640 1 -- 172.21.15.47:0/3973233127 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7400cbe0 con 0x7efc8c0c8260 2024-02-19T15:24:08.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.974+0000 7efc92000640 1 -- 172.21.15.47:0/3973233127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc8c1565b0 con 0x7efc8c0c8260 2024-02-19T15:24:08.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.974+0000 7efc92000640 1 -- 172.21.15.47:0/3973233127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc8c156b70 con 0x7efc8c0c8260 2024-02-19T15:24:08.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.974+0000 7efc92000640 1 -- 172.21.15.47:0/3973233127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc8c0d1180 con 0x7efc8c0c8260 2024-02-19T15:24:08.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.974+0000 7efc83e00640 1 -- 172.21.15.47:0/3973233127 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efc7400cd40 con 0x7efc8c0c8260 2024-02-19T15:24:08.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.974+0000 7efc83e00640 1 --2- 172.21.15.47:0/3973233127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc68066e70 0x7efc68069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:08.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.974+0000 7efc83e00640 1 -- 172.21.15.47:0/3973233127 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc740847e0 con 0x7efc8c0c8260 2024-02-19T15:24:08.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.974+0000 7efc8b400640 1 --2- 172.21.15.47:0/3973233127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc68066e70 0x7efc68069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:08.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.978+0000 7efc8b400640 1 --2- 172.21.15.47:0/3973233127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc68066e70 0x7efc68069330 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efc7c0045f0 tx=0x7efc7c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:08.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:08.978+0000 7efc83e00640 1 -- 172.21.15.47:0/3973233127 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7efc740515e0 con 0x7efc8c0c8260 2024-02-19T15:24:09.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.138+0000 7efc92000640 1 -- 172.21.15.47:0/3973233127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efc8c075360 con 0x7efc8c0c8260 2024-02-19T15:24:09.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.142+0000 7efc83e00640 1 -- 172.21.15.47:0/3973233127 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v45) v1 ==== 104+0+11 (secure 0 0 0) 0x7efc74054ca0 con 0x7efc8c0c8260 2024-02-19T15:24:09.143 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:24:09.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.142+0000 7efc92000640 1 -- 172.21.15.47:0/3973233127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc68066e70 msgr2=0x7efc68069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:09.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.142+0000 7efc92000640 1 --2- 172.21.15.47:0/3973233127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc68066e70 0x7efc68069330 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efc7c0045f0 tx=0x7efc7c008040 comp rx=0 tx=0).stop 2024-02-19T15:24:09.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.142+0000 7efc92000640 1 -- 172.21.15.47:0/3973233127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc8c0c8260 msgr2=0x7efc8c153fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:09.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.142+0000 7efc92000640 1 --2- 172.21.15.47:0/3973233127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc8c0c8260 0x7efc8c153fe0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7efc7400e810 tx=0x7efc7400ecd0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.142+0000 7efc92000640 1 -- 172.21.15.47:0/3973233127 shutdown_connections 2024-02-19T15:24:09.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.142+0000 7efc92000640 1 --2- 172.21.15.47:0/3973233127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc68066e70 0x7efc68069330 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.142+0000 7efc92000640 1 --2- 172.21.15.47:0/3973233127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc8c0cfbf0 0x7efc8c0cb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.142+0000 7efc92000640 1 --2- 172.21.15.47:0/3973233127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc8c0c8260 0x7efc8c153fe0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.142+0000 7efc92000640 1 --2- 172.21.15.47:0/3973233127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc8c0c7e80 0x7efc8c157090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.142+0000 7efc92000640 1 -- 172.21.15.47:0/3973233127 >> 172.21.15.47:0/3973233127 conn(0x7efc8c0d1b30 msgr2=0x7efc8c06eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:09.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.142+0000 7efc92000640 1 -- 172.21.15.47:0/3973233127 shutdown_connections 2024-02-19T15:24:09.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.142+0000 7efc92000640 1 -- 172.21.15.47:0/3973233127 wait complete. 2024-02-19T15:24:09.165 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:24:09.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.306+0000 7f31d7400640 1 Processor -- start 2024-02-19T15:24:09.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.306+0000 7f31d7400640 1 -- start start 2024-02-19T15:24:09.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.306+0000 7f31d7400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f31d00c7e80 0x7f31d00d0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:09.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.306+0000 7f31d7400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d00cd750 con 0x7f31d00cfbf0 2024-02-19T15:24:09.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.306+0000 7f31d7400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d00ceee0 con 0x7f31d00c8260 2024-02-19T15:24:09.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.306+0000 7f31d7400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d00cdc60 con 0x7f31d00c7e80 2024-02-19T15:24:09.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.306+0000 7f31d4a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f31d00c7e80 0x7f31d00d0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:09.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.306+0000 7f31d5400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f31d00cfbf0 0x7f31d00c5d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51986/0 (socket says 172.21.15.47:51986) 2024-02-19T15:24:09.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.306+0000 7f31d5400640 1 -- 172.21.15.47:0/2625479651 learned_addr learned my addr 172.21.15.47:0/2625479651 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:09.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.306+0000 7f31d4a00640 1 -- 172.21.15.47:0/2625479651 >> v1:172.21.15.116:6789/0 conn(0x7f31d00c8260 legacy=0x7f31d00c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:09.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.306+0000 7f31d4a00640 1 -- 172.21.15.47:0/2625479651 >> v1:172.21.15.47:6789/0 conn(0x7f31d00cfbf0 legacy=0x7f31d00c5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:09.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.306+0000 7f31d4a00640 1 -- 172.21.15.47:0/2625479651 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d00d2a40 con 0x7f31d00c7e80 2024-02-19T15:24:09.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.306+0000 7f31cf400640 1 -- 172.21.15.47:0/2625479651 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087335427 0 0) 0x7f31d00cd750 con 0x7f31d00cfbf0 2024-02-19T15:24:09.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.310+0000 7f31d4a00640 1 --2- 172.21.15.47:0/2625479651 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f31d00c7e80 0x7f31d00d0110 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f31c40097c0 tx=0x7f31c401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9d48112757c92a7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:09.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.310+0000 7f31cf400640 1 -- 172.21.15.47:0/2625479651 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c40a6040 con 0x7f31d00c7e80 2024-02-19T15:24:09.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.310+0000 7f31cf400640 1 -- 172.21.15.47:0/2625479651 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c40aa600 con 0x7f31d00c7e80 2024-02-19T15:24:09.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.310+0000 7f31cf400640 1 -- 172.21.15.47:0/2625479651 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c40aa990 con 0x7f31d00c7e80 2024-02-19T15:24:09.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.310+0000 7f31d7400640 1 -- 172.21.15.47:0/2625479651 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f31d00c7e80 msgr2=0x7f31d00d0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:09.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.310+0000 7f31d7400640 1 --2- 172.21.15.47:0/2625479651 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f31d00c7e80 0x7f31d00d0110 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f31c40097c0 tx=0x7f31c401c4a0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.310+0000 7f31d7400640 1 -- 172.21.15.47:0/2625479651 shutdown_connections 2024-02-19T15:24:09.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.310+0000 7f31d7400640 1 --2- 172.21.15.47:0/2625479651 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f31d00c7e80 0x7f31d00d0110 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.310+0000 7f31d7400640 1 -- 172.21.15.47:0/2625479651 >> 172.21.15.47:0/2625479651 conn(0x7f31d00d1b30 msgr2=0x7f31d00d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:09.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.310+0000 7f31d7400640 1 -- 172.21.15.47:0/2625479651 shutdown_connections 2024-02-19T15:24:09.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.310+0000 7f31d7400640 1 -- 172.21.15.47:0/2625479651 wait complete. 2024-02-19T15:24:09.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31d7400640 1 Processor -- start 2024-02-19T15:24:09.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31d7400640 1 -- start start 2024-02-19T15:24:09.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31d7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31d00c7e80 0x7f31d0159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:09.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31d7400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f31d00c8260 0x7f31d00b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:09.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31d7400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31d00cfbf0 0x7f31d006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:09.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31d7400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d00c9810 con 0x7f31d00c8260 2024-02-19T15:24:09.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31d7400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d00c8c00 con 0x7f31d00c7e80 2024-02-19T15:24:09.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31d7400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d00c8ee0 con 0x7f31d00cfbf0 2024-02-19T15:24:09.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31cfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f31d00c8260 0x7f31d00b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:09.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31cfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f31d00c8260 0x7f31d00b0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43384/0 (socket says 172.21.15.47:43384) 2024-02-19T15:24:09.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31cfe00640 1 -- 172.21.15.47:0/2264103116 learned_addr learned my addr 172.21.15.47:0/2264103116 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:09.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31d4a00640 1 --2- 172.21.15.47:0/2264103116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31d00c7e80 0x7f31d0159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:09.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31d5400640 1 --2- 172.21.15.47:0/2264103116 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31d00cfbf0 0x7f31d006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:09.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31cfe00640 1 -- 172.21.15.47:0/2264103116 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31d00cfbf0 msgr2=0x7f31d006d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:09.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31cfe00640 1 --2- 172.21.15.47:0/2264103116 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31d00cfbf0 0x7f31d006d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31cfe00640 1 -- 172.21.15.47:0/2264103116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31d00c7e80 msgr2=0x7f31d0159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:09.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31cfe00640 1 --2- 172.21.15.47:0/2264103116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31d00c7e80 0x7f31d0159660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31cfe00640 1 -- 172.21.15.47:0/2264103116 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c4009420 con 0x7f31d00c8260 2024-02-19T15:24:09.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31cfe00640 1 --2- 172.21.15.47:0/2264103116 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f31d00c8260 0x7f31d00b0f90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00c500 tx=0x7f31bc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:09.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31ccc00640 1 -- 172.21.15.47:0/2264103116 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31bc005220 con 0x7f31d00c8260 2024-02-19T15:24:09.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31ccc00640 1 -- 172.21.15.47:0/2264103116 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31bc005380 con 0x7f31d00c8260 2024-02-19T15:24:09.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31d7400640 1 -- 172.21.15.47:0/2264103116 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d00c91c0 con 0x7f31d00c8260 2024-02-19T15:24:09.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31ccc00640 1 -- 172.21.15.47:0/2264103116 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31bc007270 con 0x7f31d00c8260 2024-02-19T15:24:09.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31d7400640 1 -- 172.21.15.47:0/2264103116 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d0159ba0 con 0x7f31d00c8260 2024-02-19T15:24:09.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.314+0000 7f31d7400640 1 -- 172.21.15.47:0/2264103116 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d00c99c0 con 0x7f31d00c8260 2024-02-19T15:24:09.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.318+0000 7f31ccc00640 1 -- 172.21.15.47:0/2264103116 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f31bc0073d0 con 0x7f31d00c8260 2024-02-19T15:24:09.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.318+0000 7f31ccc00640 1 --2- 172.21.15.47:0/2264103116 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f31b0066e70 0x7f31b0069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:09.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.318+0000 7f31ccc00640 1 -- 172.21.15.47:0/2264103116 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31bc084390 con 0x7f31d00c8260 2024-02-19T15:24:09.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.318+0000 7f31d4a00640 1 --2- 172.21.15.47:0/2264103116 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f31b0066e70 0x7f31b0069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:09.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.318+0000 7f31d4a00640 1 --2- 172.21.15.47:0/2264103116 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f31b0066e70 0x7f31b0069330 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f31c40ae000 tx=0x7f31c40b5040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:09.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.326+0000 7f31ccc00640 1 -- 172.21.15.47:0/2264103116 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f31d00c99c0 con 0x7f31d00c8260 2024-02-19T15:24:09.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.486+0000 7f31d7400640 1 -- 172.21.15.47:0/2264103116 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f31d00ceaf0 con 0x7f31d00c8260 2024-02-19T15:24:09.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.490+0000 7f31ccc00640 1 -- 172.21.15.47:0/2264103116 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v45) v1 ==== 100+0+11 (secure 0 0 0) 0x7f31bc0510f0 con 0x7f31d00c8260 2024-02-19T15:24:09.491 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:24:09.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.490+0000 7f31d7400640 1 -- 172.21.15.47:0/2264103116 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f31b0066e70 msgr2=0x7f31b0069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:09.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.490+0000 7f31d7400640 1 --2- 172.21.15.47:0/2264103116 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f31b0066e70 0x7f31b0069330 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f31c40ae000 tx=0x7f31c40b5040 comp rx=0 tx=0).stop 2024-02-19T15:24:09.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.490+0000 7f31d7400640 1 -- 172.21.15.47:0/2264103116 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f31d00c8260 msgr2=0x7f31d00b0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:09.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.490+0000 7f31d7400640 1 --2- 172.21.15.47:0/2264103116 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f31d00c8260 0x7f31d00b0f90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00c500 tx=0x7f31bc00c9c0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.490+0000 7f31d7400640 1 -- 172.21.15.47:0/2264103116 shutdown_connections 2024-02-19T15:24:09.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.490+0000 7f31d7400640 1 --2- 172.21.15.47:0/2264103116 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f31b0066e70 0x7f31b0069330 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.490+0000 7f31d7400640 1 --2- 172.21.15.47:0/2264103116 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31d00cfbf0 0x7f31d006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.490+0000 7f31d7400640 1 --2- 172.21.15.47:0/2264103116 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f31d00c8260 0x7f31d00b0f90 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.490+0000 7f31d7400640 1 --2- 172.21.15.47:0/2264103116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31d00c7e80 0x7f31d0159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.490+0000 7f31d7400640 1 -- 172.21.15.47:0/2264103116 >> 172.21.15.47:0/2264103116 conn(0x7f31d00d1b30 msgr2=0x7f31d006c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:09.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.490+0000 7f31d7400640 1 -- 172.21.15.47:0/2264103116 shutdown_connections 2024-02-19T15:24:09.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.490+0000 7f31d7400640 1 -- 172.21.15.47:0/2264103116 wait complete. 2024-02-19T15:24:09.513 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:24:09.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.650+0000 7f20a6e00640 1 Processor -- start 2024-02-19T15:24:09.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.650+0000 7f20a6e00640 1 -- start start 2024-02-19T15:24:09.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.654+0000 7f20a6e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f20a006d650 0x7f20a00d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:09.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.654+0000 7f20a6e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a00cce30 con 0x7f20a00cded0 2024-02-19T15:24:09.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.654+0000 7f20a6e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a00cd110 con 0x7f20a00ced10 2024-02-19T15:24:09.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.654+0000 7f20a6e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a00cd3f0 con 0x7f20a006d650 2024-02-19T15:24:09.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.654+0000 7f209fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f20a006d650 0x7f20a00d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:09.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.654+0000 7f2097e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f20a00cded0 0x7f20a00ce600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51990/0 (socket says 172.21.15.47:51990) 2024-02-19T15:24:09.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.654+0000 7f2097e00640 1 -- 172.21.15.47:0/1779116047 learned_addr learned my addr 172.21.15.47:0/1779116047 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:09.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.654+0000 7f209fe00640 1 -- 172.21.15.47:0/1779116047 >> v1:172.21.15.116:6789/0 conn(0x7f20a00ced10 legacy=0x7f20a006bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:09.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.654+0000 7f209fe00640 1 -- 172.21.15.47:0/1779116047 >> v1:172.21.15.47:6789/0 conn(0x7f20a00cded0 legacy=0x7f20a00ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:09.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.654+0000 7f209fe00640 1 -- 172.21.15.47:0/1779116047 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a00cd6d0 con 0x7f20a006d650 2024-02-19T15:24:09.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.654+0000 7f209fe00640 1 --2- 172.21.15.47:0/1779116047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20a006d650 0x7f20a00d08b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f20900097f0 tx=0x7f209001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b34dcaeca35672f9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:09.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f209f400640 1 -- 172.21.15.47:0/1779116047 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20900ab600 con 0x7f20a006d650 2024-02-19T15:24:09.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f209f400640 1 -- 172.21.15.47:0/1779116047 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20900ab760 con 0x7f20a006d650 2024-02-19T15:24:09.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f209f400640 1 -- 172.21.15.47:0/1779116047 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20900abaf0 con 0x7f20a006d650 2024-02-19T15:24:09.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f20a6e00640 1 -- 172.21.15.47:0/1779116047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20a006d650 msgr2=0x7f20a00d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:09.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f20a6e00640 1 --2- 172.21.15.47:0/1779116047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20a006d650 0x7f20a00d08b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f20900097f0 tx=0x7f209001c3e0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f20a6e00640 1 -- 172.21.15.47:0/1779116047 shutdown_connections 2024-02-19T15:24:09.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f20a6e00640 1 --2- 172.21.15.47:0/1779116047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20a006d650 0x7f20a00d08b0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f20a6e00640 1 -- 172.21.15.47:0/1779116047 >> 172.21.15.47:0/1779116047 conn(0x7f20a00d1bf0 msgr2=0x7f20a00d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:09.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f20a6e00640 1 -- 172.21.15.47:0/1779116047 shutdown_connections 2024-02-19T15:24:09.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f20a6e00640 1 -- 172.21.15.47:0/1779116047 wait complete. 2024-02-19T15:24:09.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f20a6e00640 1 Processor -- start 2024-02-19T15:24:09.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f20a6e00640 1 -- start start 2024-02-19T15:24:09.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f20a6e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20a006d650 0x7f20a0159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:09.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f20a6e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20a00cded0 0x7f20a00c5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:09.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f20a6e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f20a00ced10 0x7f20a00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:09.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f20a6e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a00c9790 con 0x7f20a00cded0 2024-02-19T15:24:09.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f20a6e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a015c450 con 0x7f20a006d650 2024-02-19T15:24:09.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f20a6e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a00c8df0 con 0x7f20a00ced10 2024-02-19T15:24:09.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f2097e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20a00cded0 0x7f20a00c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:09.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f2097e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20a00cded0 0x7f20a00c5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43404/0 (socket says 172.21.15.47:43404) 2024-02-19T15:24:09.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f2097e00640 1 -- 172.21.15.47:0/724377099 learned_addr learned my addr 172.21.15.47:0/724377099 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:09.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f209fe00640 1 --2- 172.21.15.47:0/724377099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20a006d650 0x7f20a0159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:09.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.658+0000 7f20a4e00640 1 --2- 172.21.15.47:0/724377099 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f20a00ced10 0x7f20a00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:09.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.662+0000 7f2097e00640 1 -- 172.21.15.47:0/724377099 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f20a00ced10 msgr2=0x7f20a00c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:09.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.662+0000 7f2097e00640 1 --2- 172.21.15.47:0/724377099 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f20a00ced10 0x7f20a00c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.662+0000 7f2097e00640 1 -- 172.21.15.47:0/724377099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20a006d650 msgr2=0x7f20a0159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:09.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.662+0000 7f2097e00640 1 --2- 172.21.15.47:0/724377099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20a006d650 0x7f20a0159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.662+0000 7f2097e00640 1 -- 172.21.15.47:0/724377099 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2090009420 con 0x7f20a00cded0 2024-02-19T15:24:09.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.662+0000 7f20a4e00640 1 --2- 172.21.15.47:0/724377099 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f20a00ced10 0x7f20a00c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:09.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.662+0000 7f2097e00640 1 --2- 172.21.15.47:0/724377099 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20a00cded0 0x7f20a00c5720 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f208800c500 tx=0x7f208800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:09.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.662+0000 7f209cc00640 1 -- 172.21.15.47:0/724377099 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2088005cf0 con 0x7f20a00cded0 2024-02-19T15:24:09.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.662+0000 7f209cc00640 1 -- 172.21.15.47:0/724377099 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2088005e50 con 0x7f20a00cded0 2024-02-19T15:24:09.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.662+0000 7f209cc00640 1 -- 172.21.15.47:0/724377099 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20880072f0 con 0x7f20a00cded0 2024-02-19T15:24:09.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.662+0000 7f20a6e00640 1 -- 172.21.15.47:0/724377099 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a00c90d0 con 0x7f20a00cded0 2024-02-19T15:24:09.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.662+0000 7f20a6e00640 1 -- 172.21.15.47:0/724377099 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a0159bb0 con 0x7f20a00cded0 2024-02-19T15:24:09.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.662+0000 7f20a6e00640 1 -- 172.21.15.47:0/724377099 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a00c98d0 con 0x7f20a00cded0 2024-02-19T15:24:09.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.662+0000 7f209cc00640 1 -- 172.21.15.47:0/724377099 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2088007450 con 0x7f20a00cded0 2024-02-19T15:24:09.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.662+0000 7f209cc00640 1 --2- 172.21.15.47:0/724377099 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f207c066e40 0x7f207c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:09.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.662+0000 7f209cc00640 1 -- 172.21.15.47:0/724377099 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2088083f60 con 0x7f20a00cded0 2024-02-19T15:24:09.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.666+0000 7f209fe00640 1 --2- 172.21.15.47:0/724377099 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f207c066e40 0x7f207c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:09.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.666+0000 7f209fe00640 1 --2- 172.21.15.47:0/724377099 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f207c066e40 0x7f207c069300 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f20900af000 tx=0x7f20900a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:09.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.666+0000 7f209cc00640 1 -- 172.21.15.47:0/724377099 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f20a00c98d0 con 0x7f20a00cded0 2024-02-19T15:24:09.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.862+0000 7f20a6e00640 1 -- 172.21.15.47:0/724377099 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20a00c98d0 con 0x7f20a00cded0 2024-02-19T15:24:09.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.862+0000 7f209cc00640 1 -- 172.21.15.47:0/724377099 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f20a00c98d0 con 0x7f20a00cded0 2024-02-19T15:24:09.866 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:09.867 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5132,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/548836925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":548836925},{"type":"v1","addr":"172.21.15.116:6836","nonce":548836925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5135,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3530214438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3530214438},{"type":"v1","addr":"172.21.15.116:6837","nonce":3530214438}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/3433978846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3433978846},{"type":"v1","addr":"172.21.15.47:6834","nonce":3433978846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:23:42.342661+0000","modified":"2024-02-19T15:23:45.352372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5514":{"gid":5514,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3137466738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3137466738},{"type":"v1","addr":"172.21.15.47:6835","nonce":3137466738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-02-19T15:24:09.867 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 25 2024-02-19T15:24:09.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.866+0000 7f20a6e00640 1 -- 172.21.15.47:0/724377099 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f207c066e40 msgr2=0x7f207c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:09.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.866+0000 7f20a6e00640 1 --2- 172.21.15.47:0/724377099 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f207c066e40 0x7f207c069300 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f20900af000 tx=0x7f20900a7000 comp rx=0 tx=0).stop 2024-02-19T15:24:09.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.866+0000 7f20a6e00640 1 -- 172.21.15.47:0/724377099 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20a00cded0 msgr2=0x7f20a00c5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:09.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.866+0000 7f20a6e00640 1 --2- 172.21.15.47:0/724377099 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20a00cded0 0x7f20a00c5720 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f208800c500 tx=0x7f208800c9c0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.866+0000 7f20a6e00640 1 -- 172.21.15.47:0/724377099 shutdown_connections 2024-02-19T15:24:09.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.866+0000 7f20a6e00640 1 --2- 172.21.15.47:0/724377099 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f207c066e40 0x7f207c069300 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.866+0000 7f20a6e00640 1 --2- 172.21.15.47:0/724377099 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f20a00ced10 0x7f20a00c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.866+0000 7f20a6e00640 1 --2- 172.21.15.47:0/724377099 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20a00cded0 0x7f20a00c5720 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.866+0000 7f20a6e00640 1 --2- 172.21.15.47:0/724377099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20a006d650 0x7f20a0159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:09.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.866+0000 7f20a6e00640 1 -- 172.21.15.47:0/724377099 >> 172.21.15.47:0/724377099 conn(0x7f20a00d1bf0 msgr2=0x7f20a00d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:09.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.866+0000 7f20a6e00640 1 -- 172.21.15.47:0/724377099 shutdown_connections 2024-02-19T15:24:09.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:09.866+0000 7f20a6e00640 1 -- 172.21.15.47:0/724377099 wait complete. 2024-02-19T15:24:09.887 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:24:10.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.026+0000 7fd27ce00640 1 Processor -- start 2024-02-19T15:24:10.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.026+0000 7fd27ce00640 1 -- start start 2024-02-19T15:24:10.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.026+0000 7fd27ce00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd2780cd020 0x7fd2780ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:10.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.026+0000 7fd27ce00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd27815e100 0x7fd27806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:10.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.026+0000 7fd27ce00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2780ce6e0 con 0x7fd27815dd20 2024-02-19T15:24:10.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.026+0000 7fd27ce00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2780acec0 con 0x7fd2780cd020 2024-02-19T15:24:10.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.026+0000 7fd27ce00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd27806c300 con 0x7fd27815e100 2024-02-19T15:24:10.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.026+0000 7fd277400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd27815dd20 0x7fd2780cc910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51994/0 (socket says 172.21.15.47:51994) 2024-02-19T15:24:10.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.026+0000 7fd277400640 1 -- 172.21.15.47:0/2470556027 learned_addr learned my addr 172.21.15.47:0/2470556027 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:10.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.030+0000 7fd26fe00640 1 --2- 172.21.15.47:0/2470556027 >> v2:172.21.15.116:3300/0 conn(0x7fd2780cd020 0x7fd2780ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:10.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.030+0000 7fd277e00640 1 --2- 172.21.15.47:0/2470556027 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd27815e100 0x7fd27806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:10.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.030+0000 7fd276a00640 1 -- 172.21.15.47:0/2470556027 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706700102 0 0) 0x7fd2780ce6e0 con 0x7fd27815dd20 2024-02-19T15:24:10.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.030+0000 7fd276a00640 1 -- 172.21.15.47:0/2470556027 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd25c003530 con 0x7fd27815dd20 2024-02-19T15:24:10.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.030+0000 7fd276a00640 1 -- 172.21.15.47:0/2470556027 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1572731010 0 0) 0x7fd25c003530 con 0x7fd27815dd20 2024-02-19T15:24:10.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.030+0000 7fd26fe00640 1 -- 172.21.15.47:0/2470556027 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd27815e100 msgr2=0x7fd27806b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:10.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.030+0000 7fd26fe00640 1 --2- 172.21.15.47:0/2470556027 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd27815e100 0x7fd27806b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.030+0000 7fd26fe00640 1 -- 172.21.15.47:0/2470556027 >> v1:172.21.15.47:6789/0 conn(0x7fd27815dd20 legacy=0x7fd2780cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:10.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.030+0000 7fd26fe00640 1 -- 172.21.15.47:0/2470556027 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27806c5e0 con 0x7fd2780cd020 2024-02-19T15:24:10.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.030+0000 7fd277e00640 1 --2- 172.21.15.47:0/2470556027 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd27815e100 0x7fd27806b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:24:10.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.034+0000 7fd26fe00640 1 --2- 172.21.15.47:0/2470556027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2780cd020 0x7fd2780ca6e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd2600097c0 tx=0x7fd26001c270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16748988a010f3a4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:10.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.034+0000 7fd276a00640 1 -- 172.21.15.47:0/2470556027 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd268001680 con 0x7fd2780cd020 2024-02-19T15:24:10.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.034+0000 7fd276a00640 1 -- 172.21.15.47:0/2470556027 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2600a7050 con 0x7fd2780cd020 2024-02-19T15:24:10.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.034+0000 7fd276a00640 1 -- 172.21.15.47:0/2470556027 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2600ab620 con 0x7fd2780cd020 2024-02-19T15:24:10.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.034+0000 7fd27ce00640 1 -- 172.21.15.47:0/2470556027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2780cd020 msgr2=0x7fd2780ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:10.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.034+0000 7fd27ce00640 1 --2- 172.21.15.47:0/2470556027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2780cd020 0x7fd2780ca6e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd2600097c0 tx=0x7fd26001c270 comp rx=0 tx=0).stop 2024-02-19T15:24:10.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.034+0000 7fd27ce00640 1 -- 172.21.15.47:0/2470556027 shutdown_connections 2024-02-19T15:24:10.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.034+0000 7fd27ce00640 1 --2- 172.21.15.47:0/2470556027 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd27815e100 0x7fd27806b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.034+0000 7fd27ce00640 1 --2- 172.21.15.47:0/2470556027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2780cd020 0x7fd2780ca6e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.034+0000 7fd27ce00640 1 -- 172.21.15.47:0/2470556027 >> 172.21.15.47:0/2470556027 conn(0x7fd2780b1be0 msgr2=0x7fd2780b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:10.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.034+0000 7fd27ce00640 1 -- 172.21.15.47:0/2470556027 shutdown_connections 2024-02-19T15:24:10.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.034+0000 7fd27ce00640 1 -- 172.21.15.47:0/2470556027 wait complete. 2024-02-19T15:24:10.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd27ce00640 1 Processor -- start 2024-02-19T15:24:10.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd27ce00640 1 -- start start 2024-02-19T15:24:10.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd27ce00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2780cd020 0x7fd278154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:10.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd27ce00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd27815dd20 0x7fd2780b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:10.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd27ce00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd27815e100 0x7fd2780b4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:10.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd27ce00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd278071b30 con 0x7fd27815e100 2024-02-19T15:24:10.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd27ce00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2781554b0 con 0x7fd2780cd020 2024-02-19T15:24:10.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd27ce00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd278070ef0 con 0x7fd27815dd20 2024-02-19T15:24:10.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd277e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd27815e100 0x7fd2780b4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:10.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd277e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd27815e100 0x7fd2780b4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43418/0 (socket says 172.21.15.47:43418) 2024-02-19T15:24:10.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd277e00640 1 -- 172.21.15.47:0/2974836070 learned_addr learned my addr 172.21.15.47:0/2974836070 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:10.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd26fe00640 1 --2- 172.21.15.47:0/2974836070 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd27815dd20 0x7fd2780b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:10.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd277400640 1 --2- 172.21.15.47:0/2974836070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2780cd020 0x7fd278154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:10.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd277e00640 1 -- 172.21.15.47:0/2974836070 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd27815dd20 msgr2=0x7fd2780b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:10.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd277e00640 1 --2- 172.21.15.47:0/2974836070 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd27815dd20 0x7fd2780b42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd277e00640 1 -- 172.21.15.47:0/2974836070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2780cd020 msgr2=0x7fd278154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:10.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd277e00640 1 --2- 172.21.15.47:0/2974836070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2780cd020 0x7fd278154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd277e00640 1 -- 172.21.15.47:0/2974836070 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd270009a70 con 0x7fd27815e100 2024-02-19T15:24:10.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd26fe00640 1 --2- 172.21.15.47:0/2974836070 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd27815dd20 0x7fd2780b42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:10.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd277e00640 1 --2- 172.21.15.47:0/2974836070 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd27815e100 0x7fd2780b4800 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd27000efc0 tx=0x7fd270009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:10.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd26f400640 1 -- 172.21.15.47:0/2974836070 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd270009d70 con 0x7fd27815e100 2024-02-19T15:24:10.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd26f400640 1 -- 172.21.15.47:0/2974836070 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27000f040 con 0x7fd27815e100 2024-02-19T15:24:10.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd27ce00640 1 -- 172.21.15.47:0/2974836070 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd260009420 con 0x7fd27815e100 2024-02-19T15:24:10.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd26f400640 1 -- 172.21.15.47:0/2974836070 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2700039a0 con 0x7fd27815e100 2024-02-19T15:24:10.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd27ce00640 1 -- 172.21.15.47:0/2974836070 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd278071470 con 0x7fd27815e100 2024-02-19T15:24:10.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.038+0000 7fd27ce00640 1 -- 172.21.15.47:0/2974836070 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd278071c70 con 0x7fd27815e100 2024-02-19T15:24:10.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.042+0000 7fd26f400640 1 -- 172.21.15.47:0/2974836070 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd2700144a0 con 0x7fd27815e100 2024-02-19T15:24:10.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.042+0000 7fd26f400640 1 --2- 172.21.15.47:0/2974836070 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd254066e40 0x7fd254069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:10.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.042+0000 7fd26f400640 1 -- 172.21.15.47:0/2974836070 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2700856e0 con 0x7fd27815e100 2024-02-19T15:24:10.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.042+0000 7fd277400640 1 --2- 172.21.15.47:0/2974836070 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd254066e40 0x7fd254069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:10.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.042+0000 7fd277400640 1 --2- 172.21.15.47:0/2974836070 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd254066e40 0x7fd254069300 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd2680011f0 tx=0x7fd2680016e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:10.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.046+0000 7fd26f400640 1 -- 172.21.15.47:0/2974836070 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd278071c70 con 0x7fd27815e100 2024-02-19T15:24:10.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.210+0000 7fd27ce00640 1 -- 172.21.15.47:0/2974836070 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd278075360 con 0x7fd27815e100 2024-02-19T15:24:10.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.210+0000 7fd26f400640 1 -- 172.21.15.47:0/2974836070 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16625 (secure 0 0 0) 0x7fd270052430 con 0x7fd27815e100 2024-02-19T15:24:10.214 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:10.215 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":45,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:23:43.941118+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:23:40.751741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-02-19T15:23:41.338930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:24:10.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.214+0000 7fd27ce00640 1 -- 172.21.15.47:0/2974836070 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd254066e40 msgr2=0x7fd254069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:10.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.214+0000 7fd27ce00640 1 --2- 172.21.15.47:0/2974836070 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd254066e40 0x7fd254069300 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd2680011f0 tx=0x7fd2680016e0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.214+0000 7fd27ce00640 1 -- 172.21.15.47:0/2974836070 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd27815e100 msgr2=0x7fd2780b4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:10.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.214+0000 7fd27ce00640 1 --2- 172.21.15.47:0/2974836070 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd27815e100 0x7fd2780b4800 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd27000efc0 tx=0x7fd270009f90 comp rx=0 tx=0).stop 2024-02-19T15:24:10.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.214+0000 7fd27ce00640 1 -- 172.21.15.47:0/2974836070 shutdown_connections 2024-02-19T15:24:10.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.214+0000 7fd27ce00640 1 --2- 172.21.15.47:0/2974836070 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd254066e40 0x7fd254069300 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.214+0000 7fd27ce00640 1 --2- 172.21.15.47:0/2974836070 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd27815e100 0x7fd2780b4800 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.214+0000 7fd27ce00640 1 --2- 172.21.15.47:0/2974836070 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd27815dd20 0x7fd2780b42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.214+0000 7fd27ce00640 1 --2- 172.21.15.47:0/2974836070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2780cd020 0x7fd278154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.214+0000 7fd27ce00640 1 -- 172.21.15.47:0/2974836070 >> 172.21.15.47:0/2974836070 conn(0x7fd2780b1be0 msgr2=0x7fd278155d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:10.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.214+0000 7fd27ce00640 1 -- 172.21.15.47:0/2974836070 shutdown_connections 2024-02-19T15:24:10.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.214+0000 7fd27ce00640 1 -- 172.21.15.47:0/2974836070 wait complete. 2024-02-19T15:24:10.237 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T15:24:10.237 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:24:10.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.374+0000 7fa6f9e00640 1 Processor -- start 2024-02-19T15:24:10.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.374+0000 7fa6f9e00640 1 -- start start 2024-02-19T15:24:10.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.374+0000 7fa6f9e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6f40b2500 0x7fa6f406ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:10.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.374+0000 7fa6f9e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f406b5d0 con 0x7fa6f415f620 2024-02-19T15:24:10.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.374+0000 7fa6f9e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f40b1640 con 0x7fa6f41614e0 2024-02-19T15:24:10.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.374+0000 7fa6f9e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f40b1c00 con 0x7fa6f40b2500 2024-02-19T15:24:10.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.374+0000 7fa6ebe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa6f415f620 0x7fa6f40b1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52010/0 (socket says 172.21.15.47:52010) 2024-02-19T15:24:10.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.374+0000 7fa6ebe00640 1 -- 172.21.15.47:0/695757248 learned_addr learned my addr 172.21.15.47:0/695757248 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:10.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.374+0000 7fa6f3400640 1 --2- 172.21.15.47:0/695757248 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6f40b2500 0x7fa6f406ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:10.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.374+0000 7fa6f2000640 1 -- 172.21.15.47:0/695757248 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982387231 0 0) 0x7fa6f406b5d0 con 0x7fa6f415f620 2024-02-19T15:24:10.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.374+0000 7fa6f2000640 1 -- 172.21.15.47:0/695757248 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d8003530 con 0x7fa6f415f620 2024-02-19T15:24:10.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.374+0000 7fa6f2000640 1 -- 172.21.15.47:0/695757248 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862396298 0 0) 0x7fa6f40b1640 con 0x7fa6f41614e0 2024-02-19T15:24:10.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.374+0000 7fa6f2000640 1 -- 172.21.15.47:0/695757248 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6f406b5d0 con 0x7fa6f41614e0 2024-02-19T15:24:10.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.374+0000 7fa6f3400640 1 -- 172.21.15.47:0/695757248 >> v1:172.21.15.116:6789/0 conn(0x7fa6f41614e0 legacy=0x7fa6f415eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:10.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.374+0000 7fa6f3400640 1 -- 172.21.15.47:0/695757248 >> v1:172.21.15.47:6789/0 conn(0x7fa6f415f620 legacy=0x7fa6f40b1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:10.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.374+0000 7fa6f3400640 1 -- 172.21.15.47:0/695757248 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f40b1920 con 0x7fa6f40b2500 2024-02-19T15:24:10.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.378+0000 7fa6f3400640 1 --2- 172.21.15.47:0/695757248 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6f40b2500 0x7fa6f406ba60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec009dc0 tx=0x7fa6ec01c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14f5ee37e801b42 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:10.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.378+0000 7fa6f2000640 1 -- 172.21.15.47:0/695757248 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec0a5960 con 0x7fa6f40b2500 2024-02-19T15:24:10.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.378+0000 7fa6f2000640 1 -- 172.21.15.47:0/695757248 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ec0a5ac0 con 0x7fa6f40b2500 2024-02-19T15:24:10.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.378+0000 7fa6f2000640 1 -- 172.21.15.47:0/695757248 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec0a5e50 con 0x7fa6f40b2500 2024-02-19T15:24:10.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.378+0000 7fa6f9e00640 1 -- 172.21.15.47:0/695757248 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6f40b2500 msgr2=0x7fa6f406ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:10.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.378+0000 7fa6f9e00640 1 --2- 172.21.15.47:0/695757248 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6f40b2500 0x7fa6f406ba60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec009dc0 tx=0x7fa6ec01c990 comp rx=0 tx=0).stop 2024-02-19T15:24:10.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.378+0000 7fa6f9e00640 1 -- 172.21.15.47:0/695757248 shutdown_connections 2024-02-19T15:24:10.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.378+0000 7fa6f9e00640 1 --2- 172.21.15.47:0/695757248 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6f40b2500 0x7fa6f406ba60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.378+0000 7fa6f9e00640 1 -- 172.21.15.47:0/695757248 >> 172.21.15.47:0/695757248 conn(0x7fa6f40b4fc0 msgr2=0x7fa6f40b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:10.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.378+0000 7fa6f9e00640 1 -- 172.21.15.47:0/695757248 shutdown_connections 2024-02-19T15:24:10.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.378+0000 7fa6f9e00640 1 -- 172.21.15.47:0/695757248 wait complete. 2024-02-19T15:24:10.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6f9e00640 1 Processor -- start 2024-02-19T15:24:10.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6f9e00640 1 -- start start 2024-02-19T15:24:10.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6f9e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6f40b2500 0x7fa6f40cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:10.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6f9e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6f415f620 0x7fa6f40cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:10.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6f9e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6f41614e0 0x7fa6f4154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:10.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6f9e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f40cf880 con 0x7fa6f415f620 2024-02-19T15:24:10.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6f9e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f40cff90 con 0x7fa6f40b2500 2024-02-19T15:24:10.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6f9e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f40d01c0 con 0x7fa6f41614e0 2024-02-19T15:24:10.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6ebe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6f415f620 0x7fa6f40cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:10.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6ebe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6f415f620 0x7fa6f40cb910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43430/0 (socket says 172.21.15.47:43430) 2024-02-19T15:24:10.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6ebe00640 1 -- 172.21.15.47:0/2170972199 learned_addr learned my addr 172.21.15.47:0/2170972199 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:10.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6f2a00640 1 --2- 172.21.15.47:0/2170972199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6f40b2500 0x7fa6f40cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:10.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6f3400640 1 --2- 172.21.15.47:0/2170972199 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6f41614e0 0x7fa6f4154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:10.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6ebe00640 1 -- 172.21.15.47:0/2170972199 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6f41614e0 msgr2=0x7fa6f4154450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:10.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6ebe00640 1 --2- 172.21.15.47:0/2170972199 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6f41614e0 0x7fa6f4154450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6ebe00640 1 -- 172.21.15.47:0/2170972199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6f40b2500 msgr2=0x7fa6f40cfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:10.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6ebe00640 1 --2- 172.21.15.47:0/2170972199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6f40b2500 0x7fa6f40cfa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6ebe00640 1 -- 172.21.15.47:0/2170972199 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ec009a70 con 0x7fa6f415f620 2024-02-19T15:24:10.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6f2a00640 1 --2- 172.21.15.47:0/2170972199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6f40b2500 0x7fa6f40cfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:10.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6f3400640 1 --2- 172.21.15.47:0/2170972199 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6f41614e0 0x7fa6f4154450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:10.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6ebe00640 1 --2- 172.21.15.47:0/2170972199 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6f415f620 0x7fa6f40cb910 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc005890 tx=0x7fa6dc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:10.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6eb400640 1 -- 172.21.15.47:0/2170972199 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc005cf0 con 0x7fa6f415f620 2024-02-19T15:24:10.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6eb400640 1 -- 172.21.15.47:0/2170972199 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6dc005e50 con 0x7fa6f415f620 2024-02-19T15:24:10.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6f9e00640 1 -- 172.21.15.47:0/2170972199 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f40cdbf0 con 0x7fa6f415f620 2024-02-19T15:24:10.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6eb400640 1 -- 172.21.15.47:0/2170972199 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc0072b0 con 0x7fa6f415f620 2024-02-19T15:24:10.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.382+0000 7fa6f9e00640 1 -- 172.21.15.47:0/2170972199 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f40ce1b0 con 0x7fa6f415f620 2024-02-19T15:24:10.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.386+0000 7fa6f9e00640 1 -- 172.21.15.47:0/2170972199 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6f40ce3d0 con 0x7fa6f415f620 2024-02-19T15:24:10.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.386+0000 7fa6eb400640 1 -- 172.21.15.47:0/2170972199 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa6dc007410 con 0x7fa6f415f620 2024-02-19T15:24:10.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.386+0000 7fa6eb400640 1 --2- 172.21.15.47:0/2170972199 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa6d0066e40 0x7fa6d0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:10.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.386+0000 7fa6eb400640 1 -- 172.21.15.47:0/2170972199 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6dc084180 con 0x7fa6f415f620 2024-02-19T15:24:10.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.386+0000 7fa6f2a00640 1 --2- 172.21.15.47:0/2170972199 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa6d0066e40 0x7fa6d0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:10.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.386+0000 7fa6f2a00640 1 --2- 172.21.15.47:0/2170972199 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa6d0066e40 0x7fa6d0069300 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40011f0 tx=0x7fa6e4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:10.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.390+0000 7fa6eb400640 1 -- 172.21.15.47:0/2170972199 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa6dc050e50 con 0x7fa6f415f620 2024-02-19T15:24:10.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.586+0000 7fa6f9e00640 1 -- 172.21.15.47:0/2170972199 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6f4075360 con 0x7fa6f415f620 2024-02-19T15:24:10.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.586+0000 7fa6eb400640 1 -- 172.21.15.47:0/2170972199 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4941 (secure 0 0 0) 0x7fa6dc054510 con 0x7fa6f415f620 2024-02-19T15:24:10.589 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:10.589 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5132,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/548836925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":548836925},{"type":"v1","addr":"172.21.15.116:6836","nonce":548836925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5135,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3530214438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3530214438},{"type":"v1","addr":"172.21.15.116:6837","nonce":3530214438}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/3433978846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3433978846},{"type":"v1","addr":"172.21.15.47:6834","nonce":3433978846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:23:42.342661+0000","modified":"2024-02-19T15:23:45.352372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5514":{"gid":5514,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3137466738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3137466738},{"type":"v1","addr":"172.21.15.47:6835","nonce":3137466738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-02-19T15:24:10.589 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 25 2024-02-19T15:24:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.586+0000 7fa6f9e00640 1 -- 172.21.15.47:0/2170972199 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa6d0066e40 msgr2=0x7fa6d0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.586+0000 7fa6f9e00640 1 --2- 172.21.15.47:0/2170972199 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa6d0066e40 0x7fa6d0069300 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40011f0 tx=0x7fa6e4009040 comp rx=0 tx=0).stop 2024-02-19T15:24:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.586+0000 7fa6f9e00640 1 -- 172.21.15.47:0/2170972199 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6f415f620 msgr2=0x7fa6f40cb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.586+0000 7fa6f9e00640 1 --2- 172.21.15.47:0/2170972199 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6f415f620 0x7fa6f40cb910 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc005890 tx=0x7fa6dc00c960 comp rx=0 tx=0).stop 2024-02-19T15:24:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.586+0000 7fa6f9e00640 1 -- 172.21.15.47:0/2170972199 shutdown_connections 2024-02-19T15:24:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.586+0000 7fa6f9e00640 1 --2- 172.21.15.47:0/2170972199 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa6d0066e40 0x7fa6d0069300 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.586+0000 7fa6f9e00640 1 --2- 172.21.15.47:0/2170972199 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6f41614e0 0x7fa6f4154450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.586+0000 7fa6f9e00640 1 --2- 172.21.15.47:0/2170972199 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6f415f620 0x7fa6f40cb910 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.586+0000 7fa6f9e00640 1 --2- 172.21.15.47:0/2170972199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6f40b2500 0x7fa6f40cfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.586+0000 7fa6f9e00640 1 -- 172.21.15.47:0/2170972199 >> 172.21.15.47:0/2170972199 conn(0x7fa6f40b4fc0 msgr2=0x7fa6f40b1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.586+0000 7fa6f9e00640 1 -- 172.21.15.47:0/2170972199 shutdown_connections 2024-02-19T15:24:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.586+0000 7fa6f9e00640 1 -- 172.21.15.47:0/2170972199 wait complete. 2024-02-19T15:24:10.614 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:24:10.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.750+0000 7f376ea00640 1 Processor -- start 2024-02-19T15:24:10.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.750+0000 7f376ea00640 1 -- start start 2024-02-19T15:24:10.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.750+0000 7f376ea00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f376806d650 0x7f37680d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:10.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.750+0000 7f376ea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37680cded0 0x7f37680ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:10.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.750+0000 7f376ea00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f37680ceb40 0x7f3768161d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:10.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.750+0000 7f376ea00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37680cefd0 con 0x7f376806d650 2024-02-19T15:24:10.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.750+0000 7f376ea00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37680cce30 con 0x7f37680ceb40 2024-02-19T15:24:10.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.750+0000 7f376ea00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37680cd110 con 0x7f37680cded0 2024-02-19T15:24:10.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.750+0000 7f3767e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f376806d650 0x7f37680d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:10.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.754+0000 7f3767e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f376806d650 0x7f37680d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43440/0 (socket says 172.21.15.47:43440) 2024-02-19T15:24:10.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.754+0000 7f3767e00640 1 -- 172.21.15.47:0/582559731 learned_addr learned my addr 172.21.15.47:0/582559731 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:10.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.754+0000 7f376ca00640 1 --2- 172.21.15.47:0/582559731 >> v2:172.21.15.116:3300/0 conn(0x7f37680ceb40 0x7f3768161d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:10.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.754+0000 7f375f400640 1 --2- 172.21.15.47:0/582559731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37680cded0 0x7f37680ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:10.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.754+0000 7f3767e00640 1 -- 172.21.15.47:0/582559731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37680cded0 msgr2=0x7f37680ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:10.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.754+0000 7f3767e00640 1 --2- 172.21.15.47:0/582559731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37680cded0 0x7f37680ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.754+0000 7f3767e00640 1 -- 172.21.15.47:0/582559731 >> v2:172.21.15.116:3300/0 conn(0x7f37680ceb40 msgr2=0x7f3768161d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:10.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.754+0000 7f3767e00640 1 --2- 172.21.15.47:0/582559731 >> v2:172.21.15.116:3300/0 conn(0x7f37680ceb40 0x7f3768161d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.754+0000 7f3767e00640 1 -- 172.21.15.47:0/582559731 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37680cd3f0 con 0x7f376806d650 2024-02-19T15:24:10.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.754+0000 7f3767e00640 1 --2- 172.21.15.47:0/582559731 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f376806d650 0x7f37680d08b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f376806b8e0 tx=0x7f375801c1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4803d7874867b27d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:10.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.754+0000 7f3767400640 1 -- 172.21.15.47:0/582559731 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37580ab600 con 0x7f376806d650 2024-02-19T15:24:10.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.754+0000 7f3767400640 1 -- 172.21.15.47:0/582559731 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37580a7050 con 0x7f376806d650 2024-02-19T15:24:10.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.754+0000 7f3767400640 1 -- 172.21.15.47:0/582559731 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37580ab980 con 0x7f376806d650 2024-02-19T15:24:10.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ea00640 1 -- 172.21.15.47:0/582559731 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f376806d650 msgr2=0x7f37680d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:10.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ea00640 1 --2- 172.21.15.47:0/582559731 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f376806d650 0x7f37680d08b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f376806b8e0 tx=0x7f375801c1a0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ea00640 1 -- 172.21.15.47:0/582559731 shutdown_connections 2024-02-19T15:24:10.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ea00640 1 --2- 172.21.15.47:0/582559731 >> v2:172.21.15.116:3300/0 conn(0x7f37680ceb40 0x7f3768161d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ea00640 1 --2- 172.21.15.47:0/582559731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37680cded0 0x7f37680ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ea00640 1 --2- 172.21.15.47:0/582559731 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f376806d650 0x7f37680d08b0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ea00640 1 -- 172.21.15.47:0/582559731 >> 172.21.15.47:0/582559731 conn(0x7f37680d1bf0 msgr2=0x7f37680d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:10.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ea00640 1 -- 172.21.15.47:0/582559731 shutdown_connections 2024-02-19T15:24:10.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ea00640 1 -- 172.21.15.47:0/582559731 wait complete. 2024-02-19T15:24:10.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ea00640 1 Processor -- start 2024-02-19T15:24:10.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ea00640 1 -- start start 2024-02-19T15:24:10.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f376806d650 0x7f37681599d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:10.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37680cded0 0x7f3768159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:10.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f37680ceb40 0x7f376815a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:10.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ea00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37680ca170 con 0x7f37680ceb40 2024-02-19T15:24:10.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ea00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376815ca80 con 0x7f37680cded0 2024-02-19T15:24:10.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ea00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f376815cd60 con 0x7f376806d650 2024-02-19T15:24:10.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f37680ceb40 0x7f376815a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:10.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f37680ceb40 0x7f376815a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43444/0 (socket says 172.21.15.47:43444) 2024-02-19T15:24:10.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ca00640 1 -- 172.21.15.47:0/1420844717 learned_addr learned my addr 172.21.15.47:0/1420844717 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:10.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f3767e00640 1 --2- 172.21.15.47:0/1420844717 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f376806d650 0x7f37681599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f375f400640 1 --2- 172.21.15.47:0/1420844717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37680cded0 0x7f3768159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ca00640 1 -- 172.21.15.47:0/1420844717 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f376806d650 msgr2=0x7f37681599d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ca00640 1 --2- 172.21.15.47:0/1420844717 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f376806d650 0x7f37681599d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ca00640 1 -- 172.21.15.47:0/1420844717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37680cded0 msgr2=0x7f3768159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ca00640 1 --2- 172.21.15.47:0/1420844717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37680cded0 0x7f3768159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.758+0000 7f376ca00640 1 -- 172.21.15.47:0/1420844717 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37580094d0 con 0x7f37680ceb40 2024-02-19T15:24:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.762+0000 7f376ca00640 1 --2- 172.21.15.47:0/1420844717 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f37680ceb40 0x7f376815a450 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3760007e60 tx=0x7f376000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.762+0000 7f3764c00640 1 -- 172.21.15.47:0/1420844717 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3760014420 con 0x7f37680ceb40 2024-02-19T15:24:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.762+0000 7f3764c00640 1 -- 172.21.15.47:0/1420844717 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37600145a0 con 0x7f37680ceb40 2024-02-19T15:24:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.762+0000 7f376ea00640 1 -- 172.21.15.47:0/1420844717 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f376815d040 con 0x7f37680ceb40 2024-02-19T15:24:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.762+0000 7f3764c00640 1 -- 172.21.15.47:0/1420844717 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3760014950 con 0x7f37680ceb40 2024-02-19T15:24:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.762+0000 7f376ea00640 1 -- 172.21.15.47:0/1420844717 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f376815abd0 con 0x7f37680ceb40 2024-02-19T15:24:10.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.762+0000 7f376ea00640 1 -- 172.21.15.47:0/1420844717 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f376815ad60 con 0x7f37680ceb40 2024-02-19T15:24:10.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.762+0000 7f3764c00640 1 -- 172.21.15.47:0/1420844717 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3760014ab0 con 0x7f37680ceb40 2024-02-19T15:24:10.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.762+0000 7f3764c00640 1 --2- 172.21.15.47:0/1420844717 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3744066e70 0x7f3744069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:10.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.762+0000 7f3764c00640 1 -- 172.21.15.47:0/1420844717 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37600851d0 con 0x7f37680ceb40 2024-02-19T15:24:10.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.766+0000 7f3767e00640 1 --2- 172.21.15.47:0/1420844717 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3744066e70 0x7f3744069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:10.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.766+0000 7f3767e00640 1 --2- 172.21.15.47:0/1420844717 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3744066e70 0x7f3744069330 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f37580ad000 tx=0x7f3758008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:10.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.766+0000 7f3764c00640 1 -- 172.21.15.47:0/1420844717 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f3760052060 con 0x7f37680ceb40 2024-02-19T15:24:10.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.966+0000 7f376ea00640 1 -- 172.21.15.47:0/1420844717 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3768075360 con 0x7f37680ceb40 2024-02-19T15:24:10.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.966+0000 7f3764c00640 1 -- 172.21.15.47:0/1420844717 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f3760055720 con 0x7f37680ceb40 2024-02-19T15:24:10.968 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:10.968 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5132,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/548836925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":548836925},{"type":"v1","addr":"172.21.15.116:6836","nonce":548836925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5135,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3530214438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3530214438},{"type":"v1","addr":"172.21.15.116:6837","nonce":3530214438}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/3433978846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3433978846},{"type":"v1","addr":"172.21.15.47:6834","nonce":3433978846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:23:42.342661+0000","modified":"2024-02-19T15:23:45.352372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5514":{"gid":5514,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3137466738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3137466738},{"type":"v1","addr":"172.21.15.47:6835","nonce":3137466738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-02-19T15:24:10.969 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 25 2024-02-19T15:24:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.966+0000 7f376ea00640 1 -- 172.21.15.47:0/1420844717 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3744066e70 msgr2=0x7f3744069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.966+0000 7f376ea00640 1 --2- 172.21.15.47:0/1420844717 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3744066e70 0x7f3744069330 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f37580ad000 tx=0x7f3758008040 comp rx=0 tx=0).stop 2024-02-19T15:24:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.966+0000 7f376ea00640 1 -- 172.21.15.47:0/1420844717 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f37680ceb40 msgr2=0x7f376815a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.966+0000 7f376ea00640 1 --2- 172.21.15.47:0/1420844717 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f37680ceb40 0x7f376815a450 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3760007e60 tx=0x7f376000e4b0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.966+0000 7f376ea00640 1 -- 172.21.15.47:0/1420844717 shutdown_connections 2024-02-19T15:24:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.966+0000 7f376ea00640 1 --2- 172.21.15.47:0/1420844717 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3744066e70 0x7f3744069330 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.966+0000 7f376ea00640 1 --2- 172.21.15.47:0/1420844717 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f37680ceb40 0x7f376815a450 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.966+0000 7f376ea00640 1 --2- 172.21.15.47:0/1420844717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37680cded0 0x7f3768159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.966+0000 7f376ea00640 1 --2- 172.21.15.47:0/1420844717 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f376806d650 0x7f37681599d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.966+0000 7f376ea00640 1 -- 172.21.15.47:0/1420844717 >> 172.21.15.47:0/1420844717 conn(0x7f37680d1bf0 msgr2=0x7f37680d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.966+0000 7f376ea00640 1 -- 172.21.15.47:0/1420844717 shutdown_connections 2024-02-19T15:24:10.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:10.966+0000 7f376ea00640 1 -- 172.21.15.47:0/1420844717 wait complete. 2024-02-19T15:24:10.989 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:24:11.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.126+0000 7efc92200640 1 Processor -- start 2024-02-19T15:24:11.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.126+0000 7efc92200640 1 -- start start 2024-02-19T15:24:11.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.126+0000 7efc92200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efc8c0b1be0 0x7efc8c0d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:11.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.126+0000 7efc92200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8c0b0280 con 0x7efc8c0b0a80 2024-02-19T15:24:11.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.126+0000 7efc92200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8c0b04d0 con 0x7efc8c15f590 2024-02-19T15:24:11.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.126+0000 7efc92200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc8c0cc530 con 0x7efc8c0b1be0 2024-02-19T15:24:11.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.130+0000 7efc8be00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7efc8c0b0a80 0x7efc8c0b20e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52020/0 (socket says 172.21.15.47:52020) 2024-02-19T15:24:11.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.130+0000 7efc8be00640 1 -- 172.21.15.47:0/1248173125 learned_addr learned my addr 172.21.15.47:0/1248173125 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:11.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.130+0000 7efc82a00640 1 --2- 172.21.15.47:0/1248173125 >> v2:172.21.15.116:3300/0 conn(0x7efc8c0b1be0 0x7efc8c0d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:11.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.130+0000 7efc8aa00640 1 -- 172.21.15.47:0/1248173125 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926326039 0 0) 0x7efc8c0b0280 con 0x7efc8c0b0a80 2024-02-19T15:24:11.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.130+0000 7efc8aa00640 1 -- 172.21.15.47:0/1248173125 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc70003530 con 0x7efc8c0b0a80 2024-02-19T15:24:11.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.130+0000 7efc82a00640 1 -- 172.21.15.47:0/1248173125 >> v1:172.21.15.116:6789/0 conn(0x7efc8c15f590 legacy=0x7efc8c0b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:11.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.130+0000 7efc82a00640 1 -- 172.21.15.47:0/1248173125 >> v1:172.21.15.47:6789/0 conn(0x7efc8c0b0a80 legacy=0x7efc8c0b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:11.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.130+0000 7efc82a00640 1 -- 172.21.15.47:0/1248173125 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c0b07b0 con 0x7efc8c0b1be0 2024-02-19T15:24:11.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.130+0000 7efc82a00640 1 --2- 172.21.15.47:0/1248173125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc8c0b1be0 0x7efc8c0d2690 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7efc740097c0 tx=0x7efc7401c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca6df1eec6985a92 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:11.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.130+0000 7efc8aa00640 1 -- 172.21.15.47:0/1248173125 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc740ab670 con 0x7efc8c0b1be0 2024-02-19T15:24:11.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.130+0000 7efc8aa00640 1 -- 172.21.15.47:0/1248173125 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc740ab7d0 con 0x7efc8c0b1be0 2024-02-19T15:24:11.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.130+0000 7efc8aa00640 1 -- 172.21.15.47:0/1248173125 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc740abb60 con 0x7efc8c0b1be0 2024-02-19T15:24:11.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.134+0000 7efc92200640 1 -- 172.21.15.47:0/1248173125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc8c0b1be0 msgr2=0x7efc8c0d2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:11.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.134+0000 7efc92200640 1 --2- 172.21.15.47:0/1248173125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc8c0b1be0 0x7efc8c0d2690 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7efc740097c0 tx=0x7efc7401c470 comp rx=0 tx=0).stop 2024-02-19T15:24:11.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.134+0000 7efc92200640 1 -- 172.21.15.47:0/1248173125 shutdown_connections 2024-02-19T15:24:11.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.134+0000 7efc92200640 1 --2- 172.21.15.47:0/1248173125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc8c0b1be0 0x7efc8c0d2690 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.134+0000 7efc92200640 1 -- 172.21.15.47:0/1248173125 >> 172.21.15.47:0/1248173125 conn(0x7efc8c0d1b70 msgr2=0x7efc8c0d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:11.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.134+0000 7efc92200640 1 -- 172.21.15.47:0/1248173125 shutdown_connections 2024-02-19T15:24:11.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.134+0000 7efc92200640 1 -- 172.21.15.47:0/1248173125 wait complete. 2024-02-19T15:24:11.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.134+0000 7efc92200640 1 Processor -- start 2024-02-19T15:24:11.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.134+0000 7efc92200640 1 -- start start 2024-02-19T15:24:11.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.134+0000 7efc92200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc8c0b0a80 0x7efc8c0c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:11.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.134+0000 7efc92200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc8c0b1be0 0x7efc8c14f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:11.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.134+0000 7efc92200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc8c15f590 0x7efc8c0c7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:11.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.134+0000 7efc92200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c0c6600 con 0x7efc8c0b1be0 2024-02-19T15:24:11.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.134+0000 7efc92200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c150360 con 0x7efc8c0b0a80 2024-02-19T15:24:11.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.134+0000 7efc92200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c0c5d60 con 0x7efc8c15f590 2024-02-19T15:24:11.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.134+0000 7efc82a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc8c0b1be0 0x7efc8c14f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:11.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.134+0000 7efc82a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc8c0b1be0 0x7efc8c14f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43454/0 (socket says 172.21.15.47:43454) 2024-02-19T15:24:11.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.134+0000 7efc82a00640 1 -- 172.21.15.47:0/3334676381 learned_addr learned my addr 172.21.15.47:0/3334676381 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:11.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.134+0000 7efc8b400640 1 --2- 172.21.15.47:0/3334676381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc8c0b0a80 0x7efc8c0c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:11.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.134+0000 7efc8be00640 1 --2- 172.21.15.47:0/3334676381 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc8c15f590 0x7efc8c0c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:11.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.138+0000 7efc82a00640 1 -- 172.21.15.47:0/3334676381 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc8c15f590 msgr2=0x7efc8c0c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:11.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.138+0000 7efc82a00640 1 --2- 172.21.15.47:0/3334676381 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc8c15f590 0x7efc8c0c7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.138+0000 7efc82a00640 1 -- 172.21.15.47:0/3334676381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc8c0b0a80 msgr2=0x7efc8c0c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:11.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.138+0000 7efc82a00640 1 --2- 172.21.15.47:0/3334676381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc8c0b0a80 0x7efc8c0c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.138+0000 7efc82a00640 1 -- 172.21.15.47:0/3334676381 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc74009420 con 0x7efc8c0b1be0 2024-02-19T15:24:11.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.138+0000 7efc82a00640 1 --2- 172.21.15.47:0/3334676381 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc8c0b1be0 0x7efc8c14f2f0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7efc7401a0a0 tx=0x7efc7401acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:11.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.138+0000 7efc83e00640 1 -- 172.21.15.47:0/3334676381 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc740bf610 con 0x7efc8c0b1be0 2024-02-19T15:24:11.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.138+0000 7efc83e00640 1 -- 172.21.15.47:0/3334676381 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc740bf770 con 0x7efc8c0b1be0 2024-02-19T15:24:11.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.138+0000 7efc83e00640 1 -- 172.21.15.47:0/3334676381 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc740bfb20 con 0x7efc8c0b1be0 2024-02-19T15:24:11.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.138+0000 7efc92200640 1 -- 172.21.15.47:0/3334676381 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc8c0c6040 con 0x7efc8c0b1be0 2024-02-19T15:24:11.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.138+0000 7efc92200640 1 -- 172.21.15.47:0/3334676381 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc8c0c7040 con 0x7efc8c0b1be0 2024-02-19T15:24:11.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.138+0000 7efc92200640 1 -- 172.21.15.47:0/3334676381 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc8c0c6ea0 con 0x7efc8c0b1be0 2024-02-19T15:24:11.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.138+0000 7efc83e00640 1 -- 172.21.15.47:0/3334676381 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efc740b5400 con 0x7efc8c0b1be0 2024-02-19T15:24:11.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.138+0000 7efc83e00640 1 --2- 172.21.15.47:0/3334676381 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc68066e40 0x7efc68069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:11.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.138+0000 7efc83e00640 1 -- 172.21.15.47:0/3334676381 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc7411f460 con 0x7efc8c0b1be0 2024-02-19T15:24:11.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.142+0000 7efc8b400640 1 --2- 172.21.15.47:0/3334676381 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc68066e40 0x7efc68069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:11.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.142+0000 7efc8b400640 1 --2- 172.21.15.47:0/3334676381 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc68066e40 0x7efc68069300 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7efc7c0012f0 tx=0x7efc7c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:11.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.142+0000 7efc83e00640 1 -- 172.21.15.47:0/3334676381 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7efc8c0c6ea0 con 0x7efc8c0b1be0 2024-02-19T15:24:11.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.314+0000 7efc92200640 1 -- 172.21.15.47:0/3334676381 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc8c0c6ea0 con 0x7efc8c0b1be0 2024-02-19T15:24:11.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.314+0000 7efc83e00640 1 -- 172.21.15.47:0/3334676381 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16625 (secure 0 0 0) 0x7efc8c0c6ea0 con 0x7efc8c0b1be0 2024-02-19T15:24:11.318 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:11.318 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":45,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:23:43.941118+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:23:40.751741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-02-19T15:23:41.338930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:24:11.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.318+0000 7efc92200640 1 -- 172.21.15.47:0/3334676381 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc68066e40 msgr2=0x7efc68069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:11.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.318+0000 7efc92200640 1 --2- 172.21.15.47:0/3334676381 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc68066e40 0x7efc68069300 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7efc7c0012f0 tx=0x7efc7c009040 comp rx=0 tx=0).stop 2024-02-19T15:24:11.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.318+0000 7efc92200640 1 -- 172.21.15.47:0/3334676381 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc8c0b1be0 msgr2=0x7efc8c14f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:11.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.318+0000 7efc92200640 1 --2- 172.21.15.47:0/3334676381 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc8c0b1be0 0x7efc8c14f2f0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7efc7401a0a0 tx=0x7efc7401acb0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.318+0000 7efc92200640 1 -- 172.21.15.47:0/3334676381 shutdown_connections 2024-02-19T15:24:11.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.318+0000 7efc92200640 1 --2- 172.21.15.47:0/3334676381 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc68066e40 0x7efc68069300 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.318+0000 7efc92200640 1 --2- 172.21.15.47:0/3334676381 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc8c15f590 0x7efc8c0c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.318+0000 7efc92200640 1 --2- 172.21.15.47:0/3334676381 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc8c0b1be0 0x7efc8c14f2f0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.318+0000 7efc92200640 1 --2- 172.21.15.47:0/3334676381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc8c0b0a80 0x7efc8c0c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.318+0000 7efc92200640 1 -- 172.21.15.47:0/3334676381 >> 172.21.15.47:0/3334676381 conn(0x7efc8c0d1b70 msgr2=0x7efc8c0cd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:11.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.318+0000 7efc92200640 1 -- 172.21.15.47:0/3334676381 shutdown_connections 2024-02-19T15:24:11.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.318+0000 7efc92200640 1 -- 172.21.15.47:0/3334676381 wait complete. 2024-02-19T15:24:11.341 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T15:24:11.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.482+0000 7f25ca400640 1 Processor -- start 2024-02-19T15:24:11.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.482+0000 7f25ca400640 1 -- start start 2024-02-19T15:24:11.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.486+0000 7f25ca400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25c415b1a0 0x7f25c415b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:11.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.486+0000 7f25ca400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25c4071350 0x7f25c415bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:11.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.486+0000 7f25ca400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c415c000 con 0x7f25c406c9a0 2024-02-19T15:24:11.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.486+0000 7f25ca400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c40718c0 con 0x7f25c4071350 2024-02-19T15:24:11.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.486+0000 7f25ca400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c4071ba0 con 0x7f25c415b1a0 2024-02-19T15:24:11.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.486+0000 7f25c3400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25c415b1a0 0x7f25c415b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:11.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.486+0000 7f25baa00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25c4071350 0x7f25c415bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:11.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.486+0000 7f25c3400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25c415b1a0 0x7f25c415b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:38788/0 (socket says 172.21.15.47:38788) 2024-02-19T15:24:11.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.486+0000 7f25c3400640 1 -- 172.21.15.47:0/1148379695 learned_addr learned my addr 172.21.15.47:0/1148379695 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:11.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.486+0000 7f25c3400640 1 -- 172.21.15.47:0/1148379695 >> v2:172.21.15.116:3300/0 conn(0x7f25c4071350 msgr2=0x7f25c415bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:11.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.486+0000 7f25c3400640 1 --2- 172.21.15.47:0/1148379695 >> v2:172.21.15.116:3300/0 conn(0x7f25c4071350 0x7f25c415bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.486+0000 7f25c3400640 1 -- 172.21.15.47:0/1148379695 >> v1:172.21.15.116:6789/0 conn(0x7f25c406c9a0 legacy=0x7f25c415c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:11.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.486+0000 7f25c3400640 1 -- 172.21.15.47:0/1148379695 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c4071e80 con 0x7f25c415b1a0 2024-02-19T15:24:11.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.490+0000 7f25c3400640 1 --2- 172.21.15.47:0/1148379695 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25c415b1a0 0x7f25c415b580 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f25b4009e50 tx=0x7f25b401ca00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1ffcc587985b291 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:11.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.490+0000 7f25c2a00640 1 -- 172.21.15.47:0/1148379695 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b40a58f0 con 0x7f25c415b1a0 2024-02-19T15:24:11.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.490+0000 7f25c2a00640 1 -- 172.21.15.47:0/1148379695 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b40a5a50 con 0x7f25c415b1a0 2024-02-19T15:24:11.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.490+0000 7f25c2a00640 1 -- 172.21.15.47:0/1148379695 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b40a5de0 con 0x7f25c415b1a0 2024-02-19T15:24:11.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.490+0000 7f25ca400640 1 -- 172.21.15.47:0/1148379695 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25c415b1a0 msgr2=0x7f25c415b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:11.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.490+0000 7f25ca400640 1 --2- 172.21.15.47:0/1148379695 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25c415b1a0 0x7f25c415b580 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f25b4009e50 tx=0x7f25b401ca00 comp rx=0 tx=0).stop 2024-02-19T15:24:11.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.490+0000 7f25ca400640 1 -- 172.21.15.47:0/1148379695 shutdown_connections 2024-02-19T15:24:11.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.490+0000 7f25ca400640 1 --2- 172.21.15.47:0/1148379695 >> v2:172.21.15.116:3300/0 conn(0x7f25c4071350 0x7f25c415bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.490+0000 7f25ca400640 1 --2- 172.21.15.47:0/1148379695 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25c415b1a0 0x7f25c415b580 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.490+0000 7f25ca400640 1 -- 172.21.15.47:0/1148379695 >> 172.21.15.47:0/1148379695 conn(0x7f25c40b16a0 msgr2=0x7f25c40b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:11.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.490+0000 7f25ca400640 1 -- 172.21.15.47:0/1148379695 shutdown_connections 2024-02-19T15:24:11.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.490+0000 7f25ca400640 1 -- 172.21.15.47:0/1148379695 wait complete. 2024-02-19T15:24:11.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.490+0000 7f25ca400640 1 Processor -- start 2024-02-19T15:24:11.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.490+0000 7f25ca400640 1 -- start start 2024-02-19T15:24:11.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.490+0000 7f25ca400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25c406c9a0 0x7f25c414c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:11.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.490+0000 7f25ca400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25c4071350 0x7f25c41542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:11.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.490+0000 7f25ca400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25c415b1a0 0x7f25c40ce6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:11.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.494+0000 7f25ca400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c4154820 con 0x7f25c4071350 2024-02-19T15:24:11.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.494+0000 7f25ca400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c40ce260 con 0x7f25c415b1a0 2024-02-19T15:24:11.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.494+0000 7f25ca400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c40cd970 con 0x7f25c406c9a0 2024-02-19T15:24:11.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.494+0000 7f25baa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25c4071350 0x7f25c41542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:11.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.494+0000 7f25baa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25c4071350 0x7f25c41542e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43468/0 (socket says 172.21.15.47:43468) 2024-02-19T15:24:11.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.494+0000 7f25baa00640 1 -- 172.21.15.47:0/4169432473 learned_addr learned my addr 172.21.15.47:0/4169432473 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:11.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.494+0000 7f25c3400640 1 --2- 172.21.15.47:0/4169432473 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25c406c9a0 0x7f25c414c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:11.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.494+0000 7f25c3e00640 1 --2- 172.21.15.47:0/4169432473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25c415b1a0 0x7f25c40ce6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:11.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.494+0000 7f25baa00640 1 -- 172.21.15.47:0/4169432473 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25c406c9a0 msgr2=0x7f25c414c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:11.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.494+0000 7f25baa00640 1 --2- 172.21.15.47:0/4169432473 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25c406c9a0 0x7f25c414c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.494+0000 7f25baa00640 1 -- 172.21.15.47:0/4169432473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25c415b1a0 msgr2=0x7f25c40ce6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:11.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.494+0000 7f25baa00640 1 --2- 172.21.15.47:0/4169432473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25c415b1a0 0x7f25c40ce6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.494+0000 7f25baa00640 1 -- 172.21.15.47:0/4169432473 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b4009b30 con 0x7f25c4071350 2024-02-19T15:24:11.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.494+0000 7f25baa00640 1 --2- 172.21.15.47:0/4169432473 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25c4071350 0x7f25c41542e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f25ac00e930 tx=0x7f25ac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:11.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.494+0000 7f25bbe00640 1 -- 172.21.15.47:0/4169432473 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac009690 con 0x7f25c4071350 2024-02-19T15:24:11.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.494+0000 7f25bbe00640 1 -- 172.21.15.47:0/4169432473 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ac00ca00 con 0x7f25c4071350 2024-02-19T15:24:11.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.494+0000 7f25ca400640 1 -- 172.21.15.47:0/4169432473 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c40cdc10 con 0x7f25c4071350 2024-02-19T15:24:11.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.494+0000 7f25bbe00640 1 -- 172.21.15.47:0/4169432473 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac00cd70 con 0x7f25c4071350 2024-02-19T15:24:11.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.494+0000 7f25ca400640 1 -- 172.21.15.47:0/4169432473 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c40cec30 con 0x7f25c4071350 2024-02-19T15:24:11.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.494+0000 7f25ca400640 1 -- 172.21.15.47:0/4169432473 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c4075360 con 0x7f25c4071350 2024-02-19T15:24:11.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.498+0000 7f25bbe00640 1 -- 172.21.15.47:0/4169432473 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f25ac013470 con 0x7f25c4071350 2024-02-19T15:24:11.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.498+0000 7f25bbe00640 1 --2- 172.21.15.47:0/4169432473 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f25a0066e40 0x7f25a0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:11.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.498+0000 7f25bbe00640 1 -- 172.21.15.47:0/4169432473 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25ac084c70 con 0x7f25c4071350 2024-02-19T15:24:11.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.498+0000 7f25c3400640 1 --2- 172.21.15.47:0/4169432473 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f25a0066e40 0x7f25a0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:11.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.498+0000 7f25c3400640 1 --2- 172.21.15.47:0/4169432473 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f25a0066e40 0x7f25a0069300 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f25b40ae000 tx=0x7f25b40a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:11.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.502+0000 7f25bbe00640 1 -- 172.21.15.47:0/4169432473 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f25ac051b30 con 0x7f25c4071350 2024-02-19T15:24:11.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.698+0000 7f25ca400640 1 -- 172.21.15.47:0/4169432473 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f25c40ce3d0 con 0x7f25c4071350 2024-02-19T15:24:11.702 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-02-19T15:24:11.698+0000 7fc100600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T15:24:11.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.702+0000 7f25bbe00640 1 -- 172.21.15.47:0/4169432473 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) v1 ==== 161+0+0 (secure 0 0 0) 0x7f25ac017090 con 0x7f25c4071350 2024-02-19T15:24:11.704 INFO:teuthology.orchestra.run.smithi047.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T15:24:11.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.706+0000 7f25ca400640 1 -- 172.21.15.47:0/4169432473 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f25a0066e40 msgr2=0x7f25a0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:11.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.706+0000 7f25ca400640 1 --2- 172.21.15.47:0/4169432473 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f25a0066e40 0x7f25a0069300 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f25b40ae000 tx=0x7f25b40a7070 comp rx=0 tx=0).stop 2024-02-19T15:24:11.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.706+0000 7f25ca400640 1 -- 172.21.15.47:0/4169432473 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25c4071350 msgr2=0x7f25c41542e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:11.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.706+0000 7f25ca400640 1 --2- 172.21.15.47:0/4169432473 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25c4071350 0x7f25c41542e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f25ac00e930 tx=0x7f25ac00edf0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.706+0000 7f25ca400640 1 -- 172.21.15.47:0/4169432473 shutdown_connections 2024-02-19T15:24:11.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.706+0000 7f25ca400640 1 --2- 172.21.15.47:0/4169432473 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f25a0066e40 0x7f25a0069300 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.706+0000 7f25ca400640 1 --2- 172.21.15.47:0/4169432473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25c415b1a0 0x7f25c40ce6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.706+0000 7f25ca400640 1 --2- 172.21.15.47:0/4169432473 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25c4071350 0x7f25c41542e0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.706+0000 7f25ca400640 1 --2- 172.21.15.47:0/4169432473 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25c406c9a0 0x7f25c414c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.706+0000 7f25ca400640 1 -- 172.21.15.47:0/4169432473 >> 172.21.15.47:0/4169432473 conn(0x7f25c40b16a0 msgr2=0x7f25c415b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:11.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.706+0000 7f25ca400640 1 -- 172.21.15.47:0/4169432473 shutdown_connections 2024-02-19T15:24:11.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.706+0000 7f25ca400640 1 -- 172.21.15.47:0/4169432473 wait complete. 2024-02-19T15:24:11.740 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T15:24:11.785 INFO:tasks.ceph.mds.a.smithi047.stdout:starting mds.a at 2024-02-19T15:24:11.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.882+0000 7fef63a00640 1 Processor -- start 2024-02-19T15:24:11.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.882+0000 7fef63a00640 1 -- start start 2024-02-19T15:24:11.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.882+0000 7fef63a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fef5c0ceac0 0x7fef5c0d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:11.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.882+0000 7fef63a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c0cc180 0x7fef5c0d0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:11.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.882+0000 7fef63a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5c0b2fd0 con 0x7fef5c15e210 2024-02-19T15:24:11.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.882+0000 7fef63a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef5c0cfe00 con 0x7fef5c0ceac0 2024-02-19T15:24:11.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.882+0000 7fef63a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c0cff90 con 0x7fef5c0cc180 2024-02-19T15:24:11.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.882+0000 7fef5be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fef5c0ceac0 0x7fef5c0d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:11.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.882+0000 7fef5be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fef5c0ceac0 0x7fef5c0d03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43486/0 (socket says 172.21.15.47:43486) 2024-02-19T15:24:11.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.882+0000 7fef5be00640 1 -- 172.21.15.47:0/3146801462 learned_addr learned my addr 172.21.15.47:0/3146801462 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:11.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.882+0000 7fef61a00640 1 --2- 172.21.15.47:0/3146801462 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c0cc180 0x7fef5c0d0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:11.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.882+0000 7fef5be00640 1 -- 172.21.15.47:0/3146801462 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c0cc180 msgr2=0x7fef5c0d0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:11.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.882+0000 7fef5be00640 1 --2- 172.21.15.47:0/3146801462 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c0cc180 0x7fef5c0d0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.882+0000 7fef5be00640 1 -- 172.21.15.47:0/3146801462 >> v1:172.21.15.116:6789/0 conn(0x7fef5c15e210 legacy=0x7fef5c0ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:11.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.882+0000 7fef5be00640 1 -- 172.21.15.47:0/3146801462 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5c0b2cf0 con 0x7fef5c0ceac0 2024-02-19T15:24:11.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.882+0000 7fef61a00640 1 --2- 172.21.15.47:0/3146801462 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c0cc180 0x7fef5c0d0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:11.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.886+0000 7fef5be00640 1 --2- 172.21.15.47:0/3146801462 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c0ceac0 0x7fef5c0d03e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fef48009870 tx=0x7fef4801c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff7e82dd97b7966 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:11.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.886+0000 7fef5b400640 1 -- 172.21.15.47:0/3146801462 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef480a95f0 con 0x7fef5c0ceac0 2024-02-19T15:24:11.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.886+0000 7fef5b400640 1 -- 172.21.15.47:0/3146801462 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef480a9750 con 0x7fef5c0ceac0 2024-02-19T15:24:11.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.886+0000 7fef5b400640 1 -- 172.21.15.47:0/3146801462 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef480a9ae0 con 0x7fef5c0ceac0 2024-02-19T15:24:11.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.886+0000 7fef63a00640 1 -- 172.21.15.47:0/3146801462 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c0ceac0 msgr2=0x7fef5c0d03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:11.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.886+0000 7fef63a00640 1 --2- 172.21.15.47:0/3146801462 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c0ceac0 0x7fef5c0d03e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fef48009870 tx=0x7fef4801c530 comp rx=0 tx=0).stop 2024-02-19T15:24:11.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.886+0000 7fef63a00640 1 -- 172.21.15.47:0/3146801462 shutdown_connections 2024-02-19T15:24:11.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.886+0000 7fef63a00640 1 --2- 172.21.15.47:0/3146801462 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c0cc180 0x7fef5c0d0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.886+0000 7fef63a00640 1 --2- 172.21.15.47:0/3146801462 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c0ceac0 0x7fef5c0d03e0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.886+0000 7fef63a00640 1 -- 172.21.15.47:0/3146801462 >> 172.21.15.47:0/3146801462 conn(0x7fef5c0d1b30 msgr2=0x7fef5c0d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:11.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.886+0000 7fef63a00640 1 -- 172.21.15.47:0/3146801462 shutdown_connections 2024-02-19T15:24:11.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.886+0000 7fef63a00640 1 -- 172.21.15.47:0/3146801462 wait complete. 2024-02-19T15:24:11.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef63a00640 1 Processor -- start 2024-02-19T15:24:11.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef63a00640 1 -- start start 2024-02-19T15:24:11.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef63a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c0cc180 0x7fef5c0b4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:11.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef63a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c0ceac0 0x7fef5c156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:11.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef63a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c15e210 0x7fef5c15be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:11.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef63a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c143670 con 0x7fef5c0ceac0 2024-02-19T15:24:11.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef63a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c0b4d30 con 0x7fef5c0cc180 2024-02-19T15:24:11.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef63a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c143850 con 0x7fef5c15e210 2024-02-19T15:24:11.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef5be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c0ceac0 0x7fef5c156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:11.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef5be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c0ceac0 0x7fef5c156380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43494/0 (socket says 172.21.15.47:43494) 2024-02-19T15:24:11.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef5be00640 1 -- 172.21.15.47:0/620782455 learned_addr learned my addr 172.21.15.47:0/620782455 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:11.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef61a00640 1 --2- 172.21.15.47:0/620782455 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c15e210 0x7fef5c15be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:11.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef61000640 1 --2- 172.21.15.47:0/620782455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c0cc180 0x7fef5c0b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:11.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef5be00640 1 -- 172.21.15.47:0/620782455 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c15e210 msgr2=0x7fef5c15be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:11.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef5be00640 1 --2- 172.21.15.47:0/620782455 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c15e210 0x7fef5c15be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef5be00640 1 -- 172.21.15.47:0/620782455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c0cc180 msgr2=0x7fef5c0b4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:11.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef5be00640 1 --2- 172.21.15.47:0/620782455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c0cc180 0x7fef5c0b4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:11.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef5be00640 1 -- 172.21.15.47:0/620782455 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef480094d0 con 0x7fef5c0ceac0 2024-02-19T15:24:11.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef5be00640 1 --2- 172.21.15.47:0/620782455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c0ceac0 0x7fef5c156380 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fef480ab040 tx=0x7fef4801ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:11.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef58c00640 1 -- 172.21.15.47:0/620782455 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef480a95f0 con 0x7fef5c0ceac0 2024-02-19T15:24:11.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef58c00640 1 -- 172.21.15.47:0/620782455 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef480b1820 con 0x7fef5c0ceac0 2024-02-19T15:24:11.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef58c00640 1 -- 172.21.15.47:0/620782455 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef480b1bd0 con 0x7fef5c0ceac0 2024-02-19T15:24:11.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef63a00640 1 -- 172.21.15.47:0/620782455 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef5c143b30 con 0x7fef5c0ceac0 2024-02-19T15:24:11.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.890+0000 7fef63a00640 1 -- 172.21.15.47:0/620782455 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5c0b7520 con 0x7fef5c0ceac0 2024-02-19T15:24:11.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.894+0000 7fef63a00640 1 -- 172.21.15.47:0/620782455 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef5c075360 con 0x7fef5c0ceac0 2024-02-19T15:24:11.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.894+0000 7fef58c00640 1 -- 172.21.15.47:0/620782455 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fef480b1d30 con 0x7fef5c0ceac0 2024-02-19T15:24:11.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.894+0000 7fef58c00640 1 --2- 172.21.15.47:0/620782455 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef3c066e40 0x7fef3c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:11.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.894+0000 7fef58c00640 1 -- 172.21.15.47:0/620782455 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fef4811f730 con 0x7fef5c0ceac0 2024-02-19T15:24:11.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.894+0000 7fef61000640 1 --2- 172.21.15.47:0/620782455 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef3c066e40 0x7fef3c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:11.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.894+0000 7fef61000640 1 --2- 172.21.15.47:0/620782455 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef3c066e40 0x7fef3c069300 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fef50001770 tx=0x7fef50009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:11.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:11.898+0000 7fef58c00640 1 -- 172.21.15.47:0/620782455 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fef480ec4d0 con 0x7fef5c0ceac0 2024-02-19T15:24:12.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.094+0000 7fef63a00640 1 -- 172.21.15.47:0/620782455 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fef5c0d2290 con 0x7fef5c0ceac0 2024-02-19T15:24:12.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.702+0000 7fef58c00640 1 -- 172.21.15.47:0/620782455 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7fef480efb90 con 0x7fef5c0ceac0 2024-02-19T15:24:12.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.706+0000 7fef63a00640 1 -- 172.21.15.47:0/620782455 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef3c066e40 msgr2=0x7fef3c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:12.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.706+0000 7fef63a00640 1 --2- 172.21.15.47:0/620782455 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef3c066e40 0x7fef3c069300 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fef50001770 tx=0x7fef50009040 comp rx=0 tx=0).stop 2024-02-19T15:24:12.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.706+0000 7fef63a00640 1 -- 172.21.15.47:0/620782455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c0ceac0 msgr2=0x7fef5c156380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:12.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.706+0000 7fef63a00640 1 --2- 172.21.15.47:0/620782455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c0ceac0 0x7fef5c156380 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fef480ab040 tx=0x7fef4801ad70 comp rx=0 tx=0).stop 2024-02-19T15:24:12.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.706+0000 7fef63a00640 1 -- 172.21.15.47:0/620782455 shutdown_connections 2024-02-19T15:24:12.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.706+0000 7fef63a00640 1 --2- 172.21.15.47:0/620782455 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef3c066e40 0x7fef3c069300 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:12.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.706+0000 7fef63a00640 1 --2- 172.21.15.47:0/620782455 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef5c15e210 0x7fef5c15be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:12.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.706+0000 7fef63a00640 1 --2- 172.21.15.47:0/620782455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef5c0ceac0 0x7fef5c156380 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:12.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.706+0000 7fef63a00640 1 --2- 172.21.15.47:0/620782455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c0cc180 0x7fef5c0b4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:12.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.706+0000 7fef63a00640 1 -- 172.21.15.47:0/620782455 >> 172.21.15.47:0/620782455 conn(0x7fef5c0d1b30 msgr2=0x7fef5c15a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:12.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.706+0000 7fef63a00640 1 -- 172.21.15.47:0/620782455 shutdown_connections 2024-02-19T15:24:12.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.706+0000 7fef63a00640 1 -- 172.21.15.47:0/620782455 wait complete. 2024-02-19T15:24:12.730 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T15:24:12.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b35a00640 1 Processor -- start 2024-02-19T15:24:12.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b35a00640 1 -- start start 2024-02-19T15:24:12.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b35a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5b300ceb20 0x7f5b300d2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:12.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b35a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b300cc690 con 0x7f5b3015d860 2024-02-19T15:24:12.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b35a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b300cc950 con 0x7f5b300d0800 2024-02-19T15:24:12.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b35a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b300cfca0 con 0x7f5b300ceb20 2024-02-19T15:24:12.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b26000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5b3015d860 0x7f5b300d00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52028/0 (socket says 172.21.15.47:52028) 2024-02-19T15:24:12.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b26000640 1 -- 172.21.15.47:0/625085505 learned_addr learned my addr 172.21.15.47:0/625085505 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:12.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b2ea00640 1 --2- 172.21.15.47:0/625085505 >> v2:172.21.15.116:3300/0 conn(0x7f5b300ceb20 0x7f5b300d2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:12.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b2e000640 1 -- 172.21.15.47:0/625085505 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498936492 0 0) 0x7f5b300cc690 con 0x7f5b3015d860 2024-02-19T15:24:12.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b2e000640 1 -- 172.21.15.47:0/625085505 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b14003530 con 0x7f5b3015d860 2024-02-19T15:24:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b2e000640 1 -- 172.21.15.47:0/625085505 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 748360775 0 0) 0x7f5b14003530 con 0x7f5b3015d860 2024-02-19T15:24:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b2e000640 1 -- 172.21.15.47:0/625085505 >> v2:172.21.15.116:3300/0 conn(0x7f5b300ceb20 msgr2=0x7f5b300d2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b2e000640 1 --2- 172.21.15.47:0/625085505 >> v2:172.21.15.116:3300/0 conn(0x7f5b300ceb20 0x7f5b300d2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b2e000640 1 -- 172.21.15.47:0/625085505 >> v1:172.21.15.116:6789/0 conn(0x7f5b300d0800 legacy=0x7f5b300ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b2e000640 1 -- 172.21.15.47:0/625085505 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b300cef00 con 0x7f5b3015d860 2024-02-19T15:24:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b2e000640 1 -- 172.21.15.47:0/625085505 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f5b18001680 con 0x7f5b3015d860 2024-02-19T15:24:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b2e000640 1 -- 172.21.15.47:0/625085505 >> v1:172.21.15.47:6789/0 conn(0x7f5b3015d860 legacy=0x7f5b300d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b2e000640 1 --2- 172.21.15.47:0/625085505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b140039d0 0x7f5b14003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b2e000640 1 --2- 172.21.15.47:0/625085505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b140047d0 0x7f5b14004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b2e000640 1 --2- 172.21.15.47:0/625085505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b140051e0 0x7f5b1400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b2e000640 1 -- 172.21.15.47:0/625085505 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b300cfca0 con 0x7f5b140039d0 2024-02-19T15:24:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b2e000640 1 -- 172.21.15.47:0/625085505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1400c130 con 0x7f5b140047d0 2024-02-19T15:24:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b2e000640 1 -- 172.21.15.47:0/625085505 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1400c2f0 con 0x7f5b140051e0 2024-02-19T15:24:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.914+0000 7f5b2ea00640 1 --2- 172.21.15.47:0/625085505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b140039d0 0x7f5b14003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.918+0000 7f5b26000640 1 --2- 172.21.15.47:0/625085505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b140047d0 0x7f5b14004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:12.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.918+0000 7f5b2f400640 1 --2- 172.21.15.47:0/625085505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b140051e0 0x7f5b1400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:12.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.918+0000 7f5b2ea00640 1 -- 172.21.15.47:0/625085505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b140051e0 msgr2=0x7f5b1400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:12.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.918+0000 7f5b2ea00640 1 --2- 172.21.15.47:0/625085505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b140051e0 0x7f5b1400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:12.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.918+0000 7f5b2ea00640 1 -- 172.21.15.47:0/625085505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b140047d0 msgr2=0x7f5b14004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:12.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.918+0000 7f5b2ea00640 1 --2- 172.21.15.47:0/625085505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b140047d0 0x7f5b14004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:12.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.918+0000 7f5b2ea00640 1 -- 172.21.15.47:0/625085505 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b14003530 con 0x7f5b140039d0 2024-02-19T15:24:12.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.918+0000 7f5b2f400640 1 -- 172.21.15.47:0/625085505 reap_dead start 2024-02-19T15:24:12.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.918+0000 7f5b2ea00640 1 --2- 172.21.15.47:0/625085505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b140039d0 0x7f5b14003df0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f5b20009ab0 tx=0x7f5b2001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eeffbffdf54920a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:12.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.918+0000 7f5b2e000640 1 -- 172.21.15.47:0/625085505 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b200a6430 con 0x7f5b140039d0 2024-02-19T15:24:12.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.918+0000 7f5b2e000640 1 -- 172.21.15.47:0/625085505 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b200a6590 con 0x7f5b140039d0 2024-02-19T15:24:12.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.918+0000 7f5b2e000640 1 -- 172.21.15.47:0/625085505 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b200a6940 con 0x7f5b140039d0 2024-02-19T15:24:12.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.918+0000 7f5b35a00640 1 -- 172.21.15.47:0/625085505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b140039d0 msgr2=0x7f5b14003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:12.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.918+0000 7f5b35a00640 1 --2- 172.21.15.47:0/625085505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b140039d0 0x7f5b14003df0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f5b20009ab0 tx=0x7f5b2001ca50 comp rx=0 tx=0).stop 2024-02-19T15:24:12.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.918+0000 7f5b35a00640 1 -- 172.21.15.47:0/625085505 shutdown_connections 2024-02-19T15:24:12.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.918+0000 7f5b35a00640 1 --2- 172.21.15.47:0/625085505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b140039d0 0x7f5b14003df0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:12.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.918+0000 7f5b35a00640 1 -- 172.21.15.47:0/625085505 >> 172.21.15.47:0/625085505 conn(0x7f5b3009aed0 msgr2=0x7f5b300d1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:12.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.918+0000 7f5b35a00640 1 -- 172.21.15.47:0/625085505 shutdown_connections 2024-02-19T15:24:12.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.918+0000 7f5b35a00640 1 -- 172.21.15.47:0/625085505 wait complete. 2024-02-19T15:24:12.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b35a00640 1 Processor -- start 2024-02-19T15:24:12.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b35a00640 1 -- start start 2024-02-19T15:24:12.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b35a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b140039d0 0x7f5b301518f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:12.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b35a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b30151e30 0x7f5b3015a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:12.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b35a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b3015ae00 0x7f5b300b4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:12.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b35a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b30159820 con 0x7f5b140039d0 2024-02-19T15:24:12.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b35a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b30159b00 con 0x7f5b3015ae00 2024-02-19T15:24:12.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b35a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b30159de0 con 0x7f5b30151e30 2024-02-19T15:24:12.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b2ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b140039d0 0x7f5b301518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:12.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b2ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b140039d0 0x7f5b301518f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43508/0 (socket says 172.21.15.47:43508) 2024-02-19T15:24:12.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b2ea00640 1 -- 172.21.15.47:0/3397535745 learned_addr learned my addr 172.21.15.47:0/3397535745 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:12.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b26000640 1 --2- 172.21.15.47:0/3397535745 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b30151e30 0x7f5b3015a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:12.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b2f400640 1 --2- 172.21.15.47:0/3397535745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b3015ae00 0x7f5b300b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:12.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b2ea00640 1 -- 172.21.15.47:0/3397535745 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b30151e30 msgr2=0x7f5b3015a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:12.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b2ea00640 1 --2- 172.21.15.47:0/3397535745 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b30151e30 0x7f5b3015a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:12.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b2ea00640 1 -- 172.21.15.47:0/3397535745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b3015ae00 msgr2=0x7f5b300b4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:12.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b2ea00640 1 --2- 172.21.15.47:0/3397535745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b3015ae00 0x7f5b300b4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:12.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b2ea00640 1 -- 172.21.15.47:0/3397535745 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3015a0c0 con 0x7f5b140039d0 2024-02-19T15:24:12.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b2ea00640 1 --2- 172.21.15.47:0/3397535745 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b140039d0 0x7f5b301518f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f5b20009ab0 tx=0x7f5b200197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:12.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b27400640 1 -- 172.21.15.47:0/3397535745 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2000f000 con 0x7f5b140039d0 2024-02-19T15:24:12.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b27400640 1 -- 172.21.15.47:0/3397535745 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b200b9440 con 0x7f5b140039d0 2024-02-19T15:24:12.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b27400640 1 -- 172.21.15.47:0/3397535745 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b200b97d0 con 0x7f5b140039d0 2024-02-19T15:24:12.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b35a00640 1 -- 172.21.15.47:0/3397535745 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b3015a3a0 con 0x7f5b140039d0 2024-02-19T15:24:12.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.922+0000 7f5b35a00640 1 -- 172.21.15.47:0/3397535745 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b300b4e10 con 0x7f5b140039d0 2024-02-19T15:24:12.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.926+0000 7f5b35a00640 1 -- 172.21.15.47:0/3397535745 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b300b5000 con 0x7f5b140039d0 2024-02-19T15:24:12.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.926+0000 7f5b27400640 1 -- 172.21.15.47:0/3397535745 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5b200c2440 con 0x7f5b140039d0 2024-02-19T15:24:12.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.926+0000 7f5b27400640 1 --2- 172.21.15.47:0/3397535745 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5b0c066e70 0x7f5b0c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:12.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.926+0000 7f5b27400640 1 -- 172.21.15.47:0/3397535745 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5b2011eb00 con 0x7f5b140039d0 2024-02-19T15:24:12.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.926+0000 7f5b26000640 1 --2- 172.21.15.47:0/3397535745 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5b0c066e70 0x7f5b0c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:12.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.926+0000 7f5b26000640 1 --2- 172.21.15.47:0/3397535745 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5b0c066e70 0x7f5b0c069330 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5b30152250 tx=0x7f5b18007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:12.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:12.930+0000 7f5b27400640 1 -- 172.21.15.47:0/3397535745 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f5b20120e60 con 0x7f5b140039d0 2024-02-19T15:24:13.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.094+0000 7f5b35a00640 1 -- 172.21.15.47:0/3397535745 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5b30075360 con 0x7f5b140039d0 2024-02-19T15:24:13.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.710+0000 7f5b27400640 1 -- 172.21.15.47:0/3397535745 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v47) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5b200ef460 con 0x7f5b140039d0 2024-02-19T15:24:13.712 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_metadata' removed 2024-02-19T15:24:13.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.710+0000 7f5b35a00640 1 -- 172.21.15.47:0/3397535745 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5b0c066e70 msgr2=0x7f5b0c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:13.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.710+0000 7f5b35a00640 1 --2- 172.21.15.47:0/3397535745 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5b0c066e70 0x7f5b0c069330 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5b30152250 tx=0x7f5b18007040 comp rx=0 tx=0).stop 2024-02-19T15:24:13.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.710+0000 7f5b35a00640 1 -- 172.21.15.47:0/3397535745 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b140039d0 msgr2=0x7f5b301518f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.710+0000 7f5b35a00640 1 --2- 172.21.15.47:0/3397535745 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b140039d0 0x7f5b301518f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f5b20009ab0 tx=0x7f5b200197f0 comp rx=0 tx=0).stop 2024-02-19T15:24:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.710+0000 7f5b35a00640 1 -- 172.21.15.47:0/3397535745 shutdown_connections 2024-02-19T15:24:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.710+0000 7f5b35a00640 1 --2- 172.21.15.47:0/3397535745 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5b0c066e70 0x7f5b0c069330 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.710+0000 7f5b35a00640 1 --2- 172.21.15.47:0/3397535745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b3015ae00 0x7f5b300b4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.710+0000 7f5b35a00640 1 --2- 172.21.15.47:0/3397535745 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b30151e30 0x7f5b3015a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.710+0000 7f5b35a00640 1 --2- 172.21.15.47:0/3397535745 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b140039d0 0x7f5b301518f0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.710+0000 7f5b35a00640 1 -- 172.21.15.47:0/3397535745 >> 172.21.15.47:0/3397535745 conn(0x7f5b3009aed0 msgr2=0x7f5b3015e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.710+0000 7f5b35a00640 1 -- 172.21.15.47:0/3397535745 shutdown_connections 2024-02-19T15:24:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.710+0000 7f5b35a00640 1 -- 172.21.15.47:0/3397535745 wait complete. 2024-02-19T15:24:13.737 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T15:24:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.878+0000 7f5a1b800640 1 Processor -- start 2024-02-19T15:24:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.878+0000 7f5a1b800640 1 -- start start 2024-02-19T15:24:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.878+0000 7f5a1b800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5a14072540 0x7f5a140cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.878+0000 7f5a1b800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5a140ce190 0x7f5a140c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.878+0000 7f5a1b800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a140cf780 con 0x7f5a140cbb50 2024-02-19T15:24:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.878+0000 7f5a1b800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a140cbf30 con 0x7f5a140ce190 2024-02-19T15:24:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.878+0000 7f5a1b800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a140729d0 con 0x7f5a14072540 2024-02-19T15:24:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.878+0000 7f5a19800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5a140ce190 0x7f5a140c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.878+0000 7f5a18e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5a140cbb50 0x7f5a140cd540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36446/0 (socket says 172.21.15.47:36446) 2024-02-19T15:24:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.878+0000 7f5a18e00640 1 -- 172.21.15.47:0/1219439727 learned_addr learned my addr 172.21.15.47:0/1219439727 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.882+0000 7f5a13e00640 1 --2- 172.21.15.47:0/1219439727 >> v2:172.21.15.116:3300/0 conn(0x7f5a14072540 0x7f5a140cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.882+0000 7f5a19800640 1 -- 172.21.15.47:0/1219439727 >> v2:172.21.15.116:3300/0 conn(0x7f5a14072540 msgr2=0x7f5a140cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.882+0000 7f5a19800640 1 --2- 172.21.15.47:0/1219439727 >> v2:172.21.15.116:3300/0 conn(0x7f5a14072540 0x7f5a140cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.882+0000 7f5a19800640 1 -- 172.21.15.47:0/1219439727 >> v1:172.21.15.47:6789/0 conn(0x7f5a140cbb50 legacy=0x7f5a140cd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.882+0000 7f5a19800640 1 -- 172.21.15.47:0/1219439727 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a140ce680 con 0x7f5a140ce190 2024-02-19T15:24:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.882+0000 7f5a13400640 1 -- 172.21.15.47:0/1219439727 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4187146319 0 0) 0x7f5a140cf780 con 0x7f5a140cbb50 2024-02-19T15:24:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.882+0000 7f5a13e00640 1 --2- 172.21.15.47:0/1219439727 >> v2:172.21.15.116:3300/0 conn(0x7f5a14072540 0x7f5a140cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:13.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.882+0000 7f5a19800640 1 --2- 172.21.15.47:0/1219439727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a140ce190 0x7f5a140c9fc0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f5a140b2500 tx=0x7f5a0c01c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1815d57222eeaf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:13.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.882+0000 7f5a13400640 1 -- 172.21.15.47:0/1219439727 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c0a5850 con 0x7f5a140ce190 2024-02-19T15:24:13.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.882+0000 7f5a13400640 1 -- 172.21.15.47:0/1219439727 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0c0a59b0 con 0x7f5a140ce190 2024-02-19T15:24:13.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.882+0000 7f5a13400640 1 -- 172.21.15.47:0/1219439727 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c0a5d40 con 0x7f5a140ce190 2024-02-19T15:24:13.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.882+0000 7f5a1b800640 1 -- 172.21.15.47:0/1219439727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a140ce190 msgr2=0x7f5a140c9fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:13.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.882+0000 7f5a1b800640 1 --2- 172.21.15.47:0/1219439727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a140ce190 0x7f5a140c9fc0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f5a140b2500 tx=0x7f5a0c01c8a0 comp rx=0 tx=0).stop 2024-02-19T15:24:13.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a1b800640 1 -- 172.21.15.47:0/1219439727 shutdown_connections 2024-02-19T15:24:13.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a1b800640 1 --2- 172.21.15.47:0/1219439727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a140ce190 0x7f5a140c9fc0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:13.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a1b800640 1 --2- 172.21.15.47:0/1219439727 >> v2:172.21.15.116:3300/0 conn(0x7f5a14072540 0x7f5a140cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:13.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a1b800640 1 -- 172.21.15.47:0/1219439727 >> 172.21.15.47:0/1219439727 conn(0x7f5a140cc3f0 msgr2=0x7f5a140cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:13.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a1b800640 1 -- 172.21.15.47:0/1219439727 shutdown_connections 2024-02-19T15:24:13.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a1b800640 1 -- 172.21.15.47:0/1219439727 wait complete. 2024-02-19T15:24:13.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a1b800640 1 Processor -- start 2024-02-19T15:24:13.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a1b800640 1 -- start start 2024-02-19T15:24:13.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a1b800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a14072540 0x7f5a140b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:13.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a1b800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a140cbb50 0x7f5a14154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:13.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a1b800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a140ce190 0x7f5a14155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:13.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a1b800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1415a300 con 0x7f5a140cbb50 2024-02-19T15:24:13.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a1b800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1415a5e0 con 0x7f5a140ce190 2024-02-19T15:24:13.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a1b800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1415a8c0 con 0x7f5a14072540 2024-02-19T15:24:13.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a13e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a140cbb50 0x7f5a14154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:13.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a13e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a140cbb50 0x7f5a14154df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36124/0 (socket says 172.21.15.47:36124) 2024-02-19T15:24:13.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a13e00640 1 -- 172.21.15.47:0/2172550339 learned_addr learned my addr 172.21.15.47:0/2172550339 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:13.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a19800640 1 --2- 172.21.15.47:0/2172550339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a140ce190 0x7f5a14155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:13.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a18e00640 1 --2- 172.21.15.47:0/2172550339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a14072540 0x7f5a140b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:13.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a13e00640 1 -- 172.21.15.47:0/2172550339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a14072540 msgr2=0x7f5a140b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:13.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a13e00640 1 --2- 172.21.15.47:0/2172550339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a14072540 0x7f5a140b0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:13.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a13e00640 1 -- 172.21.15.47:0/2172550339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a140ce190 msgr2=0x7f5a14155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:13.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a13e00640 1 --2- 172.21.15.47:0/2172550339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a140ce190 0x7f5a14155330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:13.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a13e00640 1 -- 172.21.15.47:0/2172550339 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c009a70 con 0x7f5a140cbb50 2024-02-19T15:24:13.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a19800640 1 --2- 172.21.15.47:0/2172550339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a140ce190 0x7f5a14155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:13.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a18e00640 1 --2- 172.21.15.47:0/2172550339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a14072540 0x7f5a140b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:13.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a13e00640 1 --2- 172.21.15.47:0/2172550339 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a140cbb50 0x7f5a14154df0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5a0000e810 tx=0x7f5a0000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:13.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a10c00640 1 -- 172.21.15.47:0/2172550339 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0000c8d0 con 0x7f5a140cbb50 2024-02-19T15:24:13.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a10c00640 1 -- 172.21.15.47:0/2172550339 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0000ca30 con 0x7f5a140cbb50 2024-02-19T15:24:13.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a10c00640 1 -- 172.21.15.47:0/2172550339 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0000cdc0 con 0x7f5a140cbb50 2024-02-19T15:24:13.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.886+0000 7f5a1b800640 1 -- 172.21.15.47:0/2172550339 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a1415aba0 con 0x7f5a140cbb50 2024-02-19T15:24:13.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.890+0000 7f5a1b800640 1 -- 172.21.15.47:0/2172550339 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a140b0f40 con 0x7f5a140cbb50 2024-02-19T15:24:13.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.890+0000 7f5a1b800640 1 -- 172.21.15.47:0/2172550339 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a14075360 con 0x7f5a140cbb50 2024-02-19T15:24:13.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.890+0000 7f5a10c00640 1 -- 172.21.15.47:0/2172550339 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5a00013410 con 0x7f5a140cbb50 2024-02-19T15:24:13.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.890+0000 7f5a10c00640 1 --2- 172.21.15.47:0/2172550339 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f59f4066e40 0x7f59f4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:13.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.890+0000 7f5a10c00640 1 -- 172.21.15.47:0/2172550339 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f5a000846d0 con 0x7f5a140cbb50 2024-02-19T15:24:13.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.890+0000 7f5a18e00640 1 --2- 172.21.15.47:0/2172550339 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f59f4066e40 0x7f59f4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:13.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.890+0000 7f5a18e00640 1 --2- 172.21.15.47:0/2172550339 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f59f4066e40 0x7f59f4069300 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5a08000f70 tx=0x7f5a08009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:13.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:13.894+0000 7f5a10c00640 1 -- 172.21.15.47:0/2172550339 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f5a00051610 con 0x7f5a140cbb50 2024-02-19T15:24:14.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.058+0000 7f5a1b800640 1 -- 172.21.15.47:0/2172550339 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5a140cf150 con 0x7f5a140cbb50 2024-02-19T15:24:14.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.710+0000 7f5a10c00640 1 -- 172.21.15.47:0/2172550339 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v48) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5a00009480 con 0x7f5a140cbb50 2024-02-19T15:24:14.715 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_data' removed 2024-02-19T15:24:14.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.722+0000 7f5a1b800640 1 -- 172.21.15.47:0/2172550339 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f59f4066e40 msgr2=0x7f59f4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:14.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.722+0000 7f5a1b800640 1 --2- 172.21.15.47:0/2172550339 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f59f4066e40 0x7f59f4069300 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5a08000f70 tx=0x7f5a08009040 comp rx=0 tx=0).stop 2024-02-19T15:24:14.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.722+0000 7f5a1b800640 1 -- 172.21.15.47:0/2172550339 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a140cbb50 msgr2=0x7f5a14154df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:14.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.722+0000 7f5a1b800640 1 --2- 172.21.15.47:0/2172550339 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a140cbb50 0x7f5a14154df0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5a0000e810 tx=0x7f5a0000ecd0 comp rx=0 tx=0).stop 2024-02-19T15:24:14.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.722+0000 7f5a1b800640 1 -- 172.21.15.47:0/2172550339 shutdown_connections 2024-02-19T15:24:14.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.722+0000 7f5a1b800640 1 --2- 172.21.15.47:0/2172550339 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f59f4066e40 0x7f59f4069300 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:14.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.722+0000 7f5a1b800640 1 --2- 172.21.15.47:0/2172550339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a140ce190 0x7f5a14155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:14.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.722+0000 7f5a1b800640 1 --2- 172.21.15.47:0/2172550339 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a140cbb50 0x7f5a14154df0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:14.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.722+0000 7f5a1b800640 1 --2- 172.21.15.47:0/2172550339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a14072540 0x7f5a140b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:14.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.722+0000 7f5a1b800640 1 -- 172.21.15.47:0/2172550339 >> 172.21.15.47:0/2172550339 conn(0x7f5a140cc3f0 msgr2=0x7f5a140cd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:14.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.722+0000 7f5a1b800640 1 -- 172.21.15.47:0/2172550339 shutdown_connections 2024-02-19T15:24:14.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.722+0000 7f5a1b800640 1 -- 172.21.15.47:0/2172550339 wait complete. 2024-02-19T15:24:14.755 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-19T15:24:14.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.898+0000 7f52e1200640 1 Processor -- start 2024-02-19T15:24:14.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.898+0000 7f52e1200640 1 -- start start 2024-02-19T15:24:14.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.898+0000 7f52e1200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52dc0ce780 0x7f52dc160080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:14.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.898+0000 7f52e1200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f52dc0c6720 0x7f52dc0cf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:14.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.898+0000 7f52e1200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52dc0d2840 con 0x7f52dc0ceb60 2024-02-19T15:24:14.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.898+0000 7f52e1200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52dc0cda20 con 0x7f52dc0c6720 2024-02-19T15:24:14.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.898+0000 7f52e1200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc0d2650 con 0x7f52dc0ce780 2024-02-19T15:24:14.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.898+0000 7f52da800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f52dc0c6720 0x7f52dc0cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:14.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.898+0000 7f52da800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f52dc0c6720 0x7f52dc0cf700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36130/0 (socket says 172.21.15.47:36130) 2024-02-19T15:24:14.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.898+0000 7f52da800640 1 -- 172.21.15.47:0/4053960603 learned_addr learned my addr 172.21.15.47:0/4053960603 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:14.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.898+0000 7f52da800640 1 -- 172.21.15.47:0/4053960603 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52dc0ce780 msgr2=0x7f52dc160080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:14.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.898+0000 7f52da800640 1 --2- 172.21.15.47:0/4053960603 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52dc0ce780 0x7f52dc160080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:14.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.898+0000 7f52da800640 1 -- 172.21.15.47:0/4053960603 >> v1:172.21.15.116:6789/0 conn(0x7f52dc0ceb60 legacy=0x7f52dc1605c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:14.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.898+0000 7f52da800640 1 -- 172.21.15.47:0/4053960603 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52dc0cd360 con 0x7f52dc0c6720 2024-02-19T15:24:14.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.902+0000 7f52da800640 1 --2- 172.21.15.47:0/4053960603 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52dc0c6720 0x7f52dc0cf700 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f52d4009870 tx=0x7f52d401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1180ba79eea57b1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:14.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.902+0000 7f52d9400640 1 -- 172.21.15.47:0/4053960603 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d40ac640 con 0x7f52dc0c6720 2024-02-19T15:24:14.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.902+0000 7f52d9400640 1 -- 172.21.15.47:0/4053960603 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d40ac7a0 con 0x7f52dc0c6720 2024-02-19T15:24:14.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.902+0000 7f52d9400640 1 -- 172.21.15.47:0/4053960603 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d40acb30 con 0x7f52dc0c6720 2024-02-19T15:24:14.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.902+0000 7f52e1200640 1 -- 172.21.15.47:0/4053960603 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52dc0c6720 msgr2=0x7f52dc0cf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:14.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.902+0000 7f52e1200640 1 --2- 172.21.15.47:0/4053960603 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52dc0c6720 0x7f52dc0cf700 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f52d4009870 tx=0x7f52d401c520 comp rx=0 tx=0).stop 2024-02-19T15:24:14.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.902+0000 7f52e1200640 1 -- 172.21.15.47:0/4053960603 shutdown_connections 2024-02-19T15:24:14.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.902+0000 7f52e1200640 1 --2- 172.21.15.47:0/4053960603 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52dc0c6720 0x7f52dc0cf700 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:14.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.902+0000 7f52e1200640 1 --2- 172.21.15.47:0/4053960603 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52dc0ce780 0x7f52dc160080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:14.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.902+0000 7f52e1200640 1 -- 172.21.15.47:0/4053960603 >> 172.21.15.47:0/4053960603 conn(0x7f52dc0d1b30 msgr2=0x7f52dc0d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:14.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.902+0000 7f52e1200640 1 -- 172.21.15.47:0/4053960603 shutdown_connections 2024-02-19T15:24:14.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.902+0000 7f52e1200640 1 -- 172.21.15.47:0/4053960603 wait complete. 2024-02-19T15:24:14.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52e1200640 1 Processor -- start 2024-02-19T15:24:14.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52e1200640 1 -- start start 2024-02-19T15:24:14.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52e1200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52dc0c6720 0x7f52dc155060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:14.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52e1200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52dc0ce780 0x7f52dc15f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:14.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52e1200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52dc0ceb60 0x7f52dc15b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:14.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52e1200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc06c380 con 0x7f52dc0c6720 2024-02-19T15:24:14.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52e1200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc06c660 con 0x7f52dc0ce780 2024-02-19T15:24:14.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52e1200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc06c940 con 0x7f52dc0ceb60 2024-02-19T15:24:14.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52d9e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52dc0c6720 0x7f52dc155060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:14.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52d9e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52dc0c6720 0x7f52dc155060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36146/0 (socket says 172.21.15.47:36146) 2024-02-19T15:24:14.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52d9e00640 1 -- 172.21.15.47:0/706458170 learned_addr learned my addr 172.21.15.47:0/706458170 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:14.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52d1400640 1 --2- 172.21.15.47:0/706458170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52dc0ce780 0x7f52dc15f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:14.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52da800640 1 --2- 172.21.15.47:0/706458170 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52dc0ceb60 0x7f52dc15b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:14.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52d9e00640 1 -- 172.21.15.47:0/706458170 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52dc0ceb60 msgr2=0x7f52dc15b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:14.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52d9e00640 1 --2- 172.21.15.47:0/706458170 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52dc0ceb60 0x7f52dc15b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:14.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52d9e00640 1 -- 172.21.15.47:0/706458170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52dc0ce780 msgr2=0x7f52dc15f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:14.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52d9e00640 1 --2- 172.21.15.47:0/706458170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52dc0ce780 0x7f52dc15f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:14.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52d9e00640 1 -- 172.21.15.47:0/706458170 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d40094d0 con 0x7f52dc0c6720 2024-02-19T15:24:14.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52d9e00640 1 --2- 172.21.15.47:0/706458170 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52dc0c6720 0x7f52dc155060 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f52cc00ba50 tx=0x7f52cc00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:14.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52d2a00640 1 -- 172.21.15.47:0/706458170 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc00cbe0 con 0x7f52dc0c6720 2024-02-19T15:24:14.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52d2a00640 1 -- 172.21.15.47:0/706458170 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52cc00cd60 con 0x7f52dc0c6720 2024-02-19T15:24:14.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52d2a00640 1 -- 172.21.15.47:0/706458170 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc014680 con 0x7f52dc0c6720 2024-02-19T15:24:14.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52e1200640 1 -- 172.21.15.47:0/706458170 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52dc06cc20 con 0x7f52dc0c6720 2024-02-19T15:24:14.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52e1200640 1 -- 172.21.15.47:0/706458170 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52dc1555a0 con 0x7f52dc0c6720 2024-02-19T15:24:14.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.906+0000 7f52e1200640 1 -- 172.21.15.47:0/706458170 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52dc075360 con 0x7f52dc0c6720 2024-02-19T15:24:14.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.910+0000 7f52d2a00640 1 -- 172.21.15.47:0/706458170 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f52cc01d3b0 con 0x7f52dc0c6720 2024-02-19T15:24:14.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.910+0000 7f52d2a00640 1 --2- 172.21.15.47:0/706458170 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52b8066e40 0x7f52b8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:14.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.910+0000 7f52d2a00640 1 -- 172.21.15.47:0/706458170 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f52cc084b20 con 0x7f52dc0c6720 2024-02-19T15:24:14.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.910+0000 7f52d1400640 1 --2- 172.21.15.47:0/706458170 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52b8066e40 0x7f52b8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:14.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.910+0000 7f52d1400640 1 --2- 172.21.15.47:0/706458170 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52b8066e40 0x7f52b8069300 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f52dc15c150 tx=0x7f52c4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:14.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:14.914+0000 7f52d2a00640 1 -- 172.21.15.47:0/706458170 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f52cc051ca0 con 0x7f52dc0c6720 2024-02-19T15:24:15.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.078+0000 7f52e1200640 1 -- 172.21.15.47:0/706458170 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f52dc0c6ec0 con 0x7f52dc0c6720 2024-02-19T15:24:15.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.082+0000 7f52d2a00640 1 -- 172.21.15.47:0/706458170 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7f52cc0190a0 con 0x7f52dc0c6720 2024-02-19T15:24:15.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.086+0000 7f52e1200640 1 -- 172.21.15.47:0/706458170 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52b8066e40 msgr2=0x7f52b8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:15.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.086+0000 7f52e1200640 1 --2- 172.21.15.47:0/706458170 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52b8066e40 0x7f52b8069300 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f52dc15c150 tx=0x7f52c4009040 comp rx=0 tx=0).stop 2024-02-19T15:24:15.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.086+0000 7f52e1200640 1 -- 172.21.15.47:0/706458170 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52dc0c6720 msgr2=0x7f52dc155060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:15.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.086+0000 7f52e1200640 1 --2- 172.21.15.47:0/706458170 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52dc0c6720 0x7f52dc155060 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f52cc00ba50 tx=0x7f52cc00bf10 comp rx=0 tx=0).stop 2024-02-19T15:24:15.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.086+0000 7f52e1200640 1 -- 172.21.15.47:0/706458170 shutdown_connections 2024-02-19T15:24:15.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.086+0000 7f52e1200640 1 --2- 172.21.15.47:0/706458170 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52b8066e40 0x7f52b8069300 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:15.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.086+0000 7f52e1200640 1 --2- 172.21.15.47:0/706458170 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52dc0ceb60 0x7f52dc15b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:15.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.086+0000 7f52e1200640 1 --2- 172.21.15.47:0/706458170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52dc0ce780 0x7f52dc15f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:15.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.086+0000 7f52e1200640 1 --2- 172.21.15.47:0/706458170 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52dc0c6720 0x7f52dc155060 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:15.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.086+0000 7f52e1200640 1 -- 172.21.15.47:0/706458170 >> 172.21.15.47:0/706458170 conn(0x7f52dc0d1b30 msgr2=0x7f52dc15c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:15.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.086+0000 7f52e1200640 1 -- 172.21.15.47:0/706458170 shutdown_connections 2024-02-19T15:24:15.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.086+0000 7f52e1200640 1 -- 172.21.15.47:0/706458170 wait complete. 2024-02-19T15:24:15.107 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-02-19T15:24:15.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.246+0000 7f1f96400640 1 Processor -- start 2024-02-19T15:24:15.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.246+0000 7f1f96400640 1 -- start start 2024-02-19T15:24:15.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.246+0000 7f1f96400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f9015bb60 0x7f1f900cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:15.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.246+0000 7f1f96400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1f900aec00 0x7f1f900cc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:15.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.246+0000 7f1f96400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f900ce320 con 0x7f1f900b5010 2024-02-19T15:24:15.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.246+0000 7f1f96400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f900cf060 con 0x7f1f900aec00 2024-02-19T15:24:15.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.246+0000 7f1f96400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f900cf590 con 0x7f1f9015bb60 2024-02-19T15:24:15.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.246+0000 7f1f8fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1f900aec00 0x7f1f900cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:15.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.246+0000 7f1f95a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1f900b5010 0x7f1f9015f640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36458/0 (socket says 172.21.15.47:36458) 2024-02-19T15:24:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.246+0000 7f1f95a00640 1 -- 172.21.15.47:0/791856321 learned_addr learned my addr 172.21.15.47:0/791856321 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.246+0000 7f1f95000640 1 --2- 172.21.15.47:0/791856321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f9015bb60 0x7f1f900cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.250+0000 7f1f8fe00640 1 -- 172.21.15.47:0/791856321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f9015bb60 msgr2=0x7f1f900cc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.250+0000 7f1f8fe00640 1 --2- 172.21.15.47:0/791856321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f9015bb60 0x7f1f900cc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.250+0000 7f1f8fe00640 1 -- 172.21.15.47:0/791856321 >> v1:172.21.15.47:6789/0 conn(0x7f1f900b5010 legacy=0x7f1f9015f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.250+0000 7f1f8fe00640 1 -- 172.21.15.47:0/791856321 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9015bf40 con 0x7f1f900aec00 2024-02-19T15:24:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.250+0000 7f1f8f400640 1 -- 172.21.15.47:0/791856321 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90157547 0 0) 0x7f1f900ce320 con 0x7f1f900b5010 2024-02-19T15:24:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.250+0000 7f1f95000640 1 --2- 172.21.15.47:0/791856321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f9015bb60 0x7f1f900cc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:15.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.250+0000 7f1f8fe00640 1 --2- 172.21.15.47:0/791856321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f900aec00 0x7f1f900cc620 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0097c0 tx=0x7f1f7c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ad072f1cb2dc68 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:15.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.250+0000 7f1f8f400640 1 -- 172.21.15.47:0/791856321 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c0ab600 con 0x7f1f900aec00 2024-02-19T15:24:15.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.250+0000 7f1f8f400640 1 -- 172.21.15.47:0/791856321 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7c0a7050 con 0x7f1f900aec00 2024-02-19T15:24:15.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.250+0000 7f1f8f400640 1 -- 172.21.15.47:0/791856321 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c0ab980 con 0x7f1f900aec00 2024-02-19T15:24:15.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.250+0000 7f1f96400640 1 -- 172.21.15.47:0/791856321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f900aec00 msgr2=0x7f1f900cc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:15.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.250+0000 7f1f96400640 1 --2- 172.21.15.47:0/791856321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f900aec00 0x7f1f900cc620 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0097c0 tx=0x7f1f7c01c3e0 comp rx=0 tx=0).stop 2024-02-19T15:24:15.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f96400640 1 -- 172.21.15.47:0/791856321 shutdown_connections 2024-02-19T15:24:15.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f96400640 1 --2- 172.21.15.47:0/791856321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f900aec00 0x7f1f900cc620 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:15.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f96400640 1 --2- 172.21.15.47:0/791856321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f9015bb60 0x7f1f900cc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:15.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f96400640 1 -- 172.21.15.47:0/791856321 >> 172.21.15.47:0/791856321 conn(0x7f1f900b12d0 msgr2=0x7f1f900b16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:15.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f96400640 1 -- 172.21.15.47:0/791856321 shutdown_connections 2024-02-19T15:24:15.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f96400640 1 -- 172.21.15.47:0/791856321 wait complete. 2024-02-19T15:24:15.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f96400640 1 Processor -- start 2024-02-19T15:24:15.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f96400640 1 -- start start 2024-02-19T15:24:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f96400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f900aec00 0x7f1f90151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f96400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f900b5010 0x7f1f900b3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f96400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f9015bb60 0x7f1f90159c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f96400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f901526b0 con 0x7f1f9015bb60 2024-02-19T15:24:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f96400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f90152900 con 0x7f1f900aec00 2024-02-19T15:24:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f96400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f90152be0 con 0x7f1f900b5010 2024-02-19T15:24:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f95a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f9015bb60 0x7f1f90159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f95a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f9015bb60 0x7f1f90159c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36166/0 (socket says 172.21.15.47:36166) 2024-02-19T15:24:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f95a00640 1 -- 172.21.15.47:0/3733420654 learned_addr learned my addr 172.21.15.47:0/3733420654 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f95000640 1 --2- 172.21.15.47:0/3733420654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f900aec00 0x7f1f90151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f8fe00640 1 --2- 172.21.15.47:0/3733420654 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f900b5010 0x7f1f900b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f95a00640 1 -- 172.21.15.47:0/3733420654 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f900b5010 msgr2=0x7f1f900b3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f95a00640 1 --2- 172.21.15.47:0/3733420654 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f900b5010 0x7f1f900b3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f95a00640 1 -- 172.21.15.47:0/3733420654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f900aec00 msgr2=0x7f1f90151e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f95a00640 1 --2- 172.21.15.47:0/3733420654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f900aec00 0x7f1f90151e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f95a00640 1 -- 172.21.15.47:0/3733420654 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7c009420 con 0x7f1f9015bb60 2024-02-19T15:24:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f95000640 1 --2- 172.21.15.47:0/3733420654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f900aec00 0x7f1f90151e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f95a00640 1 --2- 172.21.15.47:0/3733420654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f9015bb60 0x7f1f90159c40 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1f8800c7a0 tx=0x7f1f8800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.254+0000 7f1f8cc00640 1 -- 172.21.15.47:0/3733420654 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8800a640 con 0x7f1f9015bb60 2024-02-19T15:24:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.258+0000 7f1f8cc00640 1 -- 172.21.15.47:0/3733420654 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8800a7a0 con 0x7f1f9015bb60 2024-02-19T15:24:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.258+0000 7f1f96400640 1 -- 172.21.15.47:0/3733420654 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f90152ec0 con 0x7f1f9015bb60 2024-02-19T15:24:15.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.258+0000 7f1f8cc00640 1 -- 172.21.15.47:0/3733420654 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8800abd0 con 0x7f1f9015bb60 2024-02-19T15:24:15.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.258+0000 7f1f96400640 1 -- 172.21.15.47:0/3733420654 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f901523c0 con 0x7f1f9015bb60 2024-02-19T15:24:15.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.258+0000 7f1f96400640 1 -- 172.21.15.47:0/3733420654 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f901533e0 con 0x7f1f9015bb60 2024-02-19T15:24:15.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.258+0000 7f1f8cc00640 1 -- 172.21.15.47:0/3733420654 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1f8800ad30 con 0x7f1f9015bb60 2024-02-19T15:24:15.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.258+0000 7f1f8cc00640 1 --2- 172.21.15.47:0/3733420654 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1f70066e40 0x7f1f70069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:15.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.258+0000 7f1f8cc00640 1 -- 172.21.15.47:0/3733420654 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1f88083f60 con 0x7f1f9015bb60 2024-02-19T15:24:15.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.258+0000 7f1f95000640 1 --2- 172.21.15.47:0/3733420654 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1f70066e40 0x7f1f70069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:15.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.262+0000 7f1f95000640 1 --2- 172.21.15.47:0/3733420654 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1f70066e40 0x7f1f70069300 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1f8400b990 tx=0x7f1f84008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:15.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.262+0000 7f1f8cc00640 1 -- 172.21.15.47:0/3733420654 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1f901533e0 con 0x7f1f9015bb60 2024-02-19T15:24:15.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.426+0000 7f1f96400640 1 -- 172.21.15.47:0/3733420654 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f1f901533e0 con 0x7f1f9015bb60 2024-02-19T15:24:15.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.430+0000 7f1f8cc00640 1 -- 172.21.15.47:0/3733420654 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v15) v1 ==== 108+0+0 (secure 0 0 0) 0x7f1f901533e0 con 0x7f1f9015bb60 2024-02-19T15:24:15.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.430+0000 7f1f96400640 1 -- 172.21.15.47:0/3733420654 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1f70066e40 msgr2=0x7f1f70069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:15.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.430+0000 7f1f96400640 1 --2- 172.21.15.47:0/3733420654 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1f70066e40 0x7f1f70069300 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1f8400b990 tx=0x7f1f84008040 comp rx=0 tx=0).stop 2024-02-19T15:24:15.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.430+0000 7f1f96400640 1 -- 172.21.15.47:0/3733420654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f9015bb60 msgr2=0x7f1f90159c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:15.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.430+0000 7f1f96400640 1 --2- 172.21.15.47:0/3733420654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f9015bb60 0x7f1f90159c40 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1f8800c7a0 tx=0x7f1f8800cc60 comp rx=0 tx=0).stop 2024-02-19T15:24:15.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.430+0000 7f1f96400640 1 -- 172.21.15.47:0/3733420654 shutdown_connections 2024-02-19T15:24:15.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.430+0000 7f1f96400640 1 --2- 172.21.15.47:0/3733420654 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1f70066e40 0x7f1f70069300 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:15.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.430+0000 7f1f96400640 1 --2- 172.21.15.47:0/3733420654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f9015bb60 0x7f1f90159c40 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:15.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.430+0000 7f1f96400640 1 --2- 172.21.15.47:0/3733420654 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f900b5010 0x7f1f900b3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:15.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.430+0000 7f1f96400640 1 --2- 172.21.15.47:0/3733420654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f900aec00 0x7f1f90151e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:15.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.430+0000 7f1f96400640 1 -- 172.21.15.47:0/3733420654 >> 172.21.15.47:0/3733420654 conn(0x7f1f900b12d0 msgr2=0x7f1f90154a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:15.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.430+0000 7f1f96400640 1 -- 172.21.15.47:0/3733420654 shutdown_connections 2024-02-19T15:24:15.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.430+0000 7f1f96400640 1 -- 172.21.15.47:0/3733420654 wait complete. 2024-02-19T15:24:15.454 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-02-19T15:24:15.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.594+0000 7f2385200640 1 Processor -- start 2024-02-19T15:24:15.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.598+0000 7f2385200640 1 -- start start 2024-02-19T15:24:15.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.598+0000 7f2385200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f238015f620 0x7f23800b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:15.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.598+0000 7f2385200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f23800b2330 0x7f238006ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:15.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.598+0000 7f2385200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238006b5d0 con 0x7f23801614e0 2024-02-19T15:24:15.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.598+0000 7f2385200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23800b1c00 con 0x7f238015f620 2024-02-19T15:24:15.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.598+0000 7f2385200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23800b27e0 con 0x7f23800b2330 2024-02-19T15:24:15.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.598+0000 7f2377e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f238015f620 0x7f23800b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:15.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.598+0000 7f2377e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f238015f620 0x7f23800b1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36170/0 (socket says 172.21.15.47:36170) 2024-02-19T15:24:15.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.598+0000 7f2377e00640 1 -- 172.21.15.47:0/4057050831 learned_addr learned my addr 172.21.15.47:0/4057050831 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:15.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.598+0000 7f237e800640 1 --2- 172.21.15.47:0/4057050831 >> v2:172.21.15.116:3300/0 conn(0x7f23800b2330 0x7f238006ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:15.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.598+0000 7f2377e00640 1 -- 172.21.15.47:0/4057050831 >> v2:172.21.15.116:3300/0 conn(0x7f23800b2330 msgr2=0x7f238006ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:15.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.598+0000 7f2377e00640 1 --2- 172.21.15.47:0/4057050831 >> v2:172.21.15.116:3300/0 conn(0x7f23800b2330 0x7f238006ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:15.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.598+0000 7f2377e00640 1 -- 172.21.15.47:0/4057050831 >> v1:172.21.15.116:6789/0 conn(0x7f23801614e0 legacy=0x7f238015eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:15.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.598+0000 7f2377e00640 1 -- 172.21.15.47:0/4057050831 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23800b1640 con 0x7f238015f620 2024-02-19T15:24:15.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.602+0000 7f2377e00640 1 --2- 172.21.15.47:0/4057050831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f238015f620 0x7f23800b1df0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f23800ccaf0 tx=0x7f236801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e60c6fe7dcdf8d86 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:15.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.602+0000 7f237d400640 1 -- 172.21.15.47:0/4057050831 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23680aa680 con 0x7f238015f620 2024-02-19T15:24:15.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.602+0000 7f237d400640 1 -- 172.21.15.47:0/4057050831 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23680aa7e0 con 0x7f238015f620 2024-02-19T15:24:15.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.602+0000 7f237d400640 1 -- 172.21.15.47:0/4057050831 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23680aab70 con 0x7f238015f620 2024-02-19T15:24:15.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.602+0000 7f2385200640 1 -- 172.21.15.47:0/4057050831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f238015f620 msgr2=0x7f23800b1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:15.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.602+0000 7f2385200640 1 --2- 172.21.15.47:0/4057050831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f238015f620 0x7f23800b1df0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f23800ccaf0 tx=0x7f236801c520 comp rx=0 tx=0).stop 2024-02-19T15:24:15.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.602+0000 7f2385200640 1 -- 172.21.15.47:0/4057050831 shutdown_connections 2024-02-19T15:24:15.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.602+0000 7f2385200640 1 --2- 172.21.15.47:0/4057050831 >> v2:172.21.15.116:3300/0 conn(0x7f23800b2330 0x7f238006ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:15.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.602+0000 7f2385200640 1 --2- 172.21.15.47:0/4057050831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f238015f620 0x7f23800b1df0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:15.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.602+0000 7f2385200640 1 -- 172.21.15.47:0/4057050831 >> 172.21.15.47:0/4057050831 conn(0x7f23800b4fc0 msgr2=0x7f23800b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:15.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.602+0000 7f2385200640 1 -- 172.21.15.47:0/4057050831 shutdown_connections 2024-02-19T15:24:15.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.602+0000 7f2385200640 1 -- 172.21.15.47:0/4057050831 wait complete. 2024-02-19T15:24:15.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.602+0000 7f2385200640 1 Processor -- start 2024-02-19T15:24:15.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.602+0000 7f2385200640 1 -- start start 2024-02-19T15:24:15.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f2385200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23800b2330 0x7f23800ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:15.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f2385200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f238015f620 0x7f2380156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:15.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f2385200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23801614e0 0x7f23800c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:15.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f2385200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2380159500 con 0x7f23801614e0 2024-02-19T15:24:15.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f2385200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23801568c0 con 0x7f23800b2330 2024-02-19T15:24:15.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f2385200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2380158880 con 0x7f238015f620 2024-02-19T15:24:15.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f237e800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23801614e0 0x7f23800c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:15.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f237e800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23801614e0 0x7f23800c9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36176/0 (socket says 172.21.15.47:36176) 2024-02-19T15:24:15.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f237e800640 1 -- 172.21.15.47:0/4263404586 learned_addr learned my addr 172.21.15.47:0/4263404586 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:15.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f237de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23800b2330 0x7f23800ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:15.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f2377e00640 1 --2- 172.21.15.47:0/4263404586 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f238015f620 0x7f2380156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:15.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f237e800640 1 -- 172.21.15.47:0/4263404586 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f238015f620 msgr2=0x7f2380156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:15.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f237e800640 1 --2- 172.21.15.47:0/4263404586 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f238015f620 0x7f2380156380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:15.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f237e800640 1 -- 172.21.15.47:0/4263404586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23800b2330 msgr2=0x7f23800ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:15.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f237e800640 1 --2- 172.21.15.47:0/4263404586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23800b2330 0x7f23800ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:15.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f237e800640 1 -- 172.21.15.47:0/4263404586 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23680094d0 con 0x7f23801614e0 2024-02-19T15:24:15.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f237e800640 1 --2- 172.21.15.47:0/4263404586 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23801614e0 0x7f23800c9ed0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f237800efc0 tx=0x7f237800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:15.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f2376000640 1 -- 172.21.15.47:0/4263404586 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2378014480 con 0x7f23801614e0 2024-02-19T15:24:15.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f2376000640 1 -- 172.21.15.47:0/4263404586 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237800f040 con 0x7f23801614e0 2024-02-19T15:24:15.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f2376000640 1 -- 172.21.15.47:0/4263404586 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2378014820 con 0x7f23801614e0 2024-02-19T15:24:15.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f2385200640 1 -- 172.21.15.47:0/4263404586 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2380158b60 con 0x7f23801614e0 2024-02-19T15:24:15.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f2385200640 1 -- 172.21.15.47:0/4263404586 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2380159120 con 0x7f23801614e0 2024-02-19T15:24:15.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f2385200640 1 -- 172.21.15.47:0/4263404586 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2380159640 con 0x7f23801614e0 2024-02-19T15:24:15.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.606+0000 7f2376000640 1 -- 172.21.15.47:0/4263404586 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2378014980 con 0x7f23801614e0 2024-02-19T15:24:15.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.610+0000 7f2376000640 1 --2- 172.21.15.47:0/4263404586 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f235c066e40 0x7f235c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:15.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.610+0000 7f2376000640 1 -- 172.21.15.47:0/4263404586 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f23780851d0 con 0x7f23801614e0 2024-02-19T15:24:15.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.610+0000 7f237de00640 1 --2- 172.21.15.47:0/4263404586 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f235c066e40 0x7f235c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:15.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.610+0000 7f237de00640 1 --2- 172.21.15.47:0/4263404586 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f235c066e40 0x7f235c069300 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2370001770 tx=0x7f2370009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:15.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.614+0000 7f2376000640 1 -- 172.21.15.47:0/4263404586 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2380159640 con 0x7f23801614e0 2024-02-19T15:24:15.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:15.838+0000 7f2385200640 1 -- 172.21.15.47:0/4263404586 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f2380159640 con 0x7f23801614e0 2024-02-19T15:24:16.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.722+0000 7f2376000640 1 -- 172.21.15.47:0/4263404586 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v140) v1 ==== 133+0+0 (secure 0 0 0) 0x7f2380159640 con 0x7f23801614e0 2024-02-19T15:24:16.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.726+0000 7f2385200640 1 -- 172.21.15.47:0/4263404586 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f235c066e40 msgr2=0x7f235c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:16.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.726+0000 7f2385200640 1 --2- 172.21.15.47:0/4263404586 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f235c066e40 0x7f235c069300 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2370001770 tx=0x7f2370009040 comp rx=0 tx=0).stop 2024-02-19T15:24:16.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.726+0000 7f2385200640 1 -- 172.21.15.47:0/4263404586 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23801614e0 msgr2=0x7f23800c9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:16.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.726+0000 7f2385200640 1 --2- 172.21.15.47:0/4263404586 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23801614e0 0x7f23800c9ed0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f237800efc0 tx=0x7f237800c490 comp rx=0 tx=0).stop 2024-02-19T15:24:16.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.726+0000 7f2385200640 1 -- 172.21.15.47:0/4263404586 shutdown_connections 2024-02-19T15:24:16.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.726+0000 7f2385200640 1 --2- 172.21.15.47:0/4263404586 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f235c066e40 0x7f235c069300 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:16.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.726+0000 7f2385200640 1 --2- 172.21.15.47:0/4263404586 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23801614e0 0x7f23800c9ed0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:16.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.726+0000 7f2385200640 1 --2- 172.21.15.47:0/4263404586 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f238015f620 0x7f2380156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:16.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.726+0000 7f2385200640 1 --2- 172.21.15.47:0/4263404586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23800b2330 0x7f23800ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:16.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.726+0000 7f2385200640 1 -- 172.21.15.47:0/4263404586 >> 172.21.15.47:0/4263404586 conn(0x7f23800b4fc0 msgr2=0x7f2380168440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:16.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.726+0000 7f2385200640 1 -- 172.21.15.47:0/4263404586 shutdown_connections 2024-02-19T15:24:16.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.726+0000 7f2385200640 1 -- 172.21.15.47:0/4263404586 wait complete. 2024-02-19T15:24:16.752 INFO:tasks.cephfs_test_runner:That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). ... ok 2024-02-19T15:24:16.753 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) That fragmentation (forced) will allow more entries to be created. 2024-02-19T15:24:16.753 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) 2024-02-19T15:24:16.753 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:24:16.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.894+0000 7fe6d4400640 1 Processor -- start 2024-02-19T15:24:16.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.894+0000 7fe6d4400640 1 -- start start 2024-02-19T15:24:16.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.894+0000 7fe6d4400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe6cc15b1a0 0x7fe6cc15b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:16.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.894+0000 7fe6d4400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe6cc071350 0x7fe6cc15bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:16.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.894+0000 7fe6d4400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6cc15c000 con 0x7fe6cc06c9a0 2024-02-19T15:24:16.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.894+0000 7fe6d4400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6cc0718c0 con 0x7fe6cc15b1a0 2024-02-19T15:24:16.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.894+0000 7fe6d4400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6cc071ba0 con 0x7fe6cc071350 2024-02-19T15:24:16.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.894+0000 7fe6d1a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe6cc15b1a0 0x7fe6cc15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:16.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.894+0000 7fe6d1a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe6cc15b1a0 0x7fe6cc15b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36182/0 (socket says 172.21.15.47:36182) 2024-02-19T15:24:16.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.894+0000 7fe6d1a00640 1 -- 172.21.15.47:0/616617217 learned_addr learned my addr 172.21.15.47:0/616617217 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:16.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.894+0000 7fe6d2400640 1 --1- 172.21.15.47:0/616617217 >> v1:172.21.15.47:6789/0 conn(0x7fe6cc06c9a0 0x7fe6cc15c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36470/0 (socket says 172.21.15.47:36470) 2024-02-19T15:24:16.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.894+0000 7fe6d1000640 1 --2- 172.21.15.47:0/616617217 >> v2:172.21.15.116:3300/0 conn(0x7fe6cc071350 0x7fe6cc15bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:16.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.894+0000 7fe6d1a00640 1 -- 172.21.15.47:0/616617217 >> v2:172.21.15.116:3300/0 conn(0x7fe6cc071350 msgr2=0x7fe6cc15bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:16.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.894+0000 7fe6d1a00640 1 --2- 172.21.15.47:0/616617217 >> v2:172.21.15.116:3300/0 conn(0x7fe6cc071350 0x7fe6cc15bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:16.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.894+0000 7fe6d1a00640 1 -- 172.21.15.47:0/616617217 >> v1:172.21.15.47:6789/0 conn(0x7fe6cc06c9a0 legacy=0x7fe6cc15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:16.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.894+0000 7fe6d1a00640 1 -- 172.21.15.47:0/616617217 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6cc071e80 con 0x7fe6cc15b1a0 2024-02-19T15:24:16.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.898+0000 7fe6d1a00640 1 --2- 172.21.15.47:0/616617217 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6cc15b1a0 0x7fe6cc15b580 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0009870 tx=0x7fe6c001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79c9b3906b329e70 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:16.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.898+0000 7fe6cbe00640 1 -- 172.21.15.47:0/616617217 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c00aa670 con 0x7fe6cc15b1a0 2024-02-19T15:24:16.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.898+0000 7fe6cbe00640 1 -- 172.21.15.47:0/616617217 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c00aa7d0 con 0x7fe6cc15b1a0 2024-02-19T15:24:16.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.898+0000 7fe6cbe00640 1 -- 172.21.15.47:0/616617217 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c00aab60 con 0x7fe6cc15b1a0 2024-02-19T15:24:16.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.898+0000 7fe6d4400640 1 -- 172.21.15.47:0/616617217 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6cc15b1a0 msgr2=0x7fe6cc15b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:16.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.898+0000 7fe6d4400640 1 --2- 172.21.15.47:0/616617217 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6cc15b1a0 0x7fe6cc15b580 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0009870 tx=0x7fe6c001c4f0 comp rx=0 tx=0).stop 2024-02-19T15:24:16.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.898+0000 7fe6d4400640 1 -- 172.21.15.47:0/616617217 shutdown_connections 2024-02-19T15:24:16.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.898+0000 7fe6d4400640 1 --2- 172.21.15.47:0/616617217 >> v2:172.21.15.116:3300/0 conn(0x7fe6cc071350 0x7fe6cc15bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:16.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.898+0000 7fe6d4400640 1 --2- 172.21.15.47:0/616617217 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6cc15b1a0 0x7fe6cc15b580 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:16.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.898+0000 7fe6d4400640 1 -- 172.21.15.47:0/616617217 >> 172.21.15.47:0/616617217 conn(0x7fe6cc0b16a0 msgr2=0x7fe6cc0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:16.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.898+0000 7fe6d4400640 1 -- 172.21.15.47:0/616617217 shutdown_connections 2024-02-19T15:24:16.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.898+0000 7fe6d4400640 1 -- 172.21.15.47:0/616617217 wait complete. 2024-02-19T15:24:16.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d4400640 1 Processor -- start 2024-02-19T15:24:16.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d4400640 1 -- start start 2024-02-19T15:24:16.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d4400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe6cc06c9a0 0x7fe6cc144480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:16.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d4400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6cc071350 0x7fe6cc0c4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:16.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d4400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6cc15b1a0 0x7fe6cc0c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:16.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d4400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc1449c0 con 0x7fe6cc071350 2024-02-19T15:24:16.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d4400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc144c40 con 0x7fe6cc15b1a0 2024-02-19T15:24:16.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d4400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc144e90 con 0x7fe6cc06c9a0 2024-02-19T15:24:16.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d1000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6cc071350 0x7fe6cc0c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:16.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d1000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6cc071350 0x7fe6cc0c4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36192/0 (socket says 172.21.15.47:36192) 2024-02-19T15:24:16.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d1000640 1 -- 172.21.15.47:0/1100584093 learned_addr learned my addr 172.21.15.47:0/1100584093 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:16.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d2400640 1 --2- 172.21.15.47:0/1100584093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6cc15b1a0 0x7fe6cc0c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:16.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d1a00640 1 --2- 172.21.15.47:0/1100584093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe6cc06c9a0 0x7fe6cc144480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:16.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d1000640 1 -- 172.21.15.47:0/1100584093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe6cc06c9a0 msgr2=0x7fe6cc144480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:16.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d1000640 1 --2- 172.21.15.47:0/1100584093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe6cc06c9a0 0x7fe6cc144480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:16.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d1000640 1 -- 172.21.15.47:0/1100584093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6cc15b1a0 msgr2=0x7fe6cc0c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:16.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d1000640 1 --2- 172.21.15.47:0/1100584093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6cc15b1a0 0x7fe6cc0c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:16.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d1000640 1 -- 172.21.15.47:0/1100584093 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c00094d0 con 0x7fe6cc071350 2024-02-19T15:24:16.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d2400640 1 --2- 172.21.15.47:0/1100584093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6cc15b1a0 0x7fe6cc0c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:16.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d1a00640 1 --2- 172.21.15.47:0/1100584093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe6cc06c9a0 0x7fe6cc144480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:16.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d1000640 1 --2- 172.21.15.47:0/1100584093 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6cc071350 0x7fe6cc0c4910 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe6b800e960 tx=0x7fe6b800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:16.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6c9600640 1 -- 172.21.15.47:0/1100584093 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8009690 con 0x7fe6cc071350 2024-02-19T15:24:16.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6c9600640 1 -- 172.21.15.47:0/1100584093 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b800ca00 con 0x7fe6cc071350 2024-02-19T15:24:16.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d4400640 1 -- 172.21.15.47:0/1100584093 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6cc145170 con 0x7fe6cc071350 2024-02-19T15:24:16.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6c9600640 1 -- 172.21.15.47:0/1100584093 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b800cd70 con 0x7fe6cc071350 2024-02-19T15:24:16.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.902+0000 7fe6d4400640 1 -- 172.21.15.47:0/1100584093 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6cc145730 con 0x7fe6cc071350 2024-02-19T15:24:16.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.906+0000 7fe6d4400640 1 -- 172.21.15.47:0/1100584093 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6cc075360 con 0x7fe6cc071350 2024-02-19T15:24:16.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.906+0000 7fe6c9600640 1 -- 172.21.15.47:0/1100584093 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe6b8013470 con 0x7fe6cc071350 2024-02-19T15:24:16.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.906+0000 7fe6c9600640 1 --2- 172.21.15.47:0/1100584093 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe6ac066e40 0x7fe6ac069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:16.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.906+0000 7fe6c9600640 1 -- 172.21.15.47:0/1100584093 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe6b8084960 con 0x7fe6cc071350 2024-02-19T15:24:16.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.906+0000 7fe6d1a00640 1 --2- 172.21.15.47:0/1100584093 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe6ac066e40 0x7fe6ac069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:16.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.906+0000 7fe6d1a00640 1 --2- 172.21.15.47:0/1100584093 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe6ac066e40 0x7fe6ac069300 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe6c00ae000 tx=0x7fe6c00a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:16.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:16.910+0000 7fe6c9600640 1 -- 172.21.15.47:0/1100584093 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe6b8051ae0 con 0x7fe6cc071350 2024-02-19T15:24:17.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.074+0000 7fe6d4400640 1 -- 172.21.15.47:0/1100584093 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6cc0cf340 con 0x7fe6cc071350 2024-02-19T15:24:17.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.078+0000 7fe6c9600640 1 -- 172.21.15.47:0/1100584093 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+13032 (secure 0 0 0) 0x7fe6b8017090 con 0x7fe6cc071350 2024-02-19T15:24:17.079 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:17.079 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":48,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:24:14.711969+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:6835/3137466738":"2024-02-20T15:24:11.702252+0000","172.21.15.47:6833/3137466738":"2024-02-20T15:24:11.702252+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:24:17.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.078+0000 7fe6d4400640 1 -- 172.21.15.47:0/1100584093 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe6ac066e40 msgr2=0x7fe6ac069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:17.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.078+0000 7fe6d4400640 1 --2- 172.21.15.47:0/1100584093 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe6ac066e40 0x7fe6ac069300 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe6c00ae000 tx=0x7fe6c00a7000 comp rx=0 tx=0).stop 2024-02-19T15:24:17.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.078+0000 7fe6d4400640 1 -- 172.21.15.47:0/1100584093 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6cc071350 msgr2=0x7fe6cc0c4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:17.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.078+0000 7fe6d4400640 1 --2- 172.21.15.47:0/1100584093 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6cc071350 0x7fe6cc0c4910 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe6b800e960 tx=0x7fe6b800ee20 comp rx=0 tx=0).stop 2024-02-19T15:24:17.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.078+0000 7fe6d4400640 1 -- 172.21.15.47:0/1100584093 shutdown_connections 2024-02-19T15:24:17.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.078+0000 7fe6d4400640 1 --2- 172.21.15.47:0/1100584093 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe6ac066e40 0x7fe6ac069300 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:17.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.078+0000 7fe6d4400640 1 --2- 172.21.15.47:0/1100584093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6cc15b1a0 0x7fe6cc0c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:17.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.078+0000 7fe6d4400640 1 --2- 172.21.15.47:0/1100584093 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6cc071350 0x7fe6cc0c4910 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:17.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.078+0000 7fe6d4400640 1 --2- 172.21.15.47:0/1100584093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe6cc06c9a0 0x7fe6cc144480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:17.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.078+0000 7fe6d4400640 1 -- 172.21.15.47:0/1100584093 >> 172.21.15.47:0/1100584093 conn(0x7fe6cc0b16a0 msgr2=0x7fe6cc06d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:17.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.078+0000 7fe6d4400640 1 -- 172.21.15.47:0/1100584093 shutdown_connections 2024-02-19T15:24:17.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.078+0000 7fe6d4400640 1 -- 172.21.15.47:0/1100584093 wait complete. 2024-02-19T15:24:17.100 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:24:17.100 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:24:17.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.242+0000 7fb665000640 1 Processor -- start 2024-02-19T15:24:17.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.242+0000 7fb665000640 1 -- start start 2024-02-19T15:24:17.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.242+0000 7fb665000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb66015ded0 0x7fb660066cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:17.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.242+0000 7fb665000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb66006b5c0 0x7fb660167c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:17.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.242+0000 7fb665000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb660067210 con 0x7fb6600b1280 2024-02-19T15:24:17.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.242+0000 7fb665000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6600b04c0 con 0x7fb66015ded0 2024-02-19T15:24:17.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.242+0000 7fb665000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6600b07a0 con 0x7fb66006b5c0 2024-02-19T15:24:17.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.242+0000 7fb65de00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb66015ded0 0x7fb660066cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:17.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.242+0000 7fb65de00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb66015ded0 0x7fb660066cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36196/0 (socket says 172.21.15.47:36196) 2024-02-19T15:24:17.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.242+0000 7fb65de00640 1 -- 172.21.15.47:0/3353942363 learned_addr learned my addr 172.21.15.47:0/3353942363 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:17.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.242+0000 7fb655400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb6600b1280 0x7fb66006aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36478/0 (socket says 172.21.15.47:36478) 2024-02-19T15:24:17.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.242+0000 7fb65e800640 1 --2- 172.21.15.47:0/3353942363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb66006b5c0 0x7fb660167c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:17.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.242+0000 7fb65de00640 1 -- 172.21.15.47:0/3353942363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb66006b5c0 msgr2=0x7fb660167c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:17.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.242+0000 7fb65de00640 1 --2- 172.21.15.47:0/3353942363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb66006b5c0 0x7fb660167c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:17.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.242+0000 7fb65de00640 1 -- 172.21.15.47:0/3353942363 >> v1:172.21.15.47:6789/0 conn(0x7fb6600b1280 legacy=0x7fb66006aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:17.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.242+0000 7fb65de00640 1 -- 172.21.15.47:0/3353942363 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6600b0a80 con 0x7fb66015ded0 2024-02-19T15:24:17.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.242+0000 7fb65e800640 1 --2- 172.21.15.47:0/3353942363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb66006b5c0 0x7fb660167c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:17.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.246+0000 7fb65de00640 1 --2- 172.21.15.47:0/3353942363 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb66015ded0 0x7fb660066cd0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb650009870 tx=0x7fb65001c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21cf4822d7d52773 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:17.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.246+0000 7fb65d400640 1 -- 172.21.15.47:0/3353942363 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6500a9610 con 0x7fb66015ded0 2024-02-19T15:24:17.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.246+0000 7fb65d400640 1 -- 172.21.15.47:0/3353942363 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6500a9770 con 0x7fb66015ded0 2024-02-19T15:24:17.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.246+0000 7fb65d400640 1 -- 172.21.15.47:0/3353942363 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6500a9b00 con 0x7fb66015ded0 2024-02-19T15:24:17.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.246+0000 7fb665000640 1 -- 172.21.15.47:0/3353942363 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb66015ded0 msgr2=0x7fb660066cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:17.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.246+0000 7fb665000640 1 --2- 172.21.15.47:0/3353942363 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb66015ded0 0x7fb660066cd0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb650009870 tx=0x7fb65001c530 comp rx=0 tx=0).stop 2024-02-19T15:24:17.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.246+0000 7fb665000640 1 -- 172.21.15.47:0/3353942363 shutdown_connections 2024-02-19T15:24:17.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.246+0000 7fb665000640 1 --2- 172.21.15.47:0/3353942363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb66006b5c0 0x7fb660167c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:17.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.246+0000 7fb665000640 1 --2- 172.21.15.47:0/3353942363 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb66015ded0 0x7fb660066cd0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:17.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.246+0000 7fb665000640 1 -- 172.21.15.47:0/3353942363 >> 172.21.15.47:0/3353942363 conn(0x7fb6600b32f0 msgr2=0x7fb6600b36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:17.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.246+0000 7fb665000640 1 -- 172.21.15.47:0/3353942363 shutdown_connections 2024-02-19T15:24:17.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.246+0000 7fb665000640 1 -- 172.21.15.47:0/3353942363 wait complete. 2024-02-19T15:24:17.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.246+0000 7fb665000640 1 Processor -- start 2024-02-19T15:24:17.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.246+0000 7fb665000640 1 -- start start 2024-02-19T15:24:17.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb665000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb66006b5c0 0x7fb660099030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:17.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb665000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6600b1280 0x7fb6600c7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:17.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb665000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb66015ded0 0x7fb6600cab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:17.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb665000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb660099570 con 0x7fb6600b1280 2024-02-19T15:24:17.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb665000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6600ad030 con 0x7fb66006b5c0 2024-02-19T15:24:17.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb665000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6600ad2f0 con 0x7fb66015ded0 2024-02-19T15:24:17.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb655400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6600b1280 0x7fb6600c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:17.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb655400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6600b1280 0x7fb6600c7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36212/0 (socket says 172.21.15.47:36212) 2024-02-19T15:24:17.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb655400640 1 -- 172.21.15.47:0/1039610277 learned_addr learned my addr 172.21.15.47:0/1039610277 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:17.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb65de00640 1 --2- 172.21.15.47:0/1039610277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb66006b5c0 0x7fb660099030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:17.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb65e800640 1 --2- 172.21.15.47:0/1039610277 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb66015ded0 0x7fb6600cab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:17.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb655400640 1 -- 172.21.15.47:0/1039610277 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb66015ded0 msgr2=0x7fb6600cab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:17.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb655400640 1 --2- 172.21.15.47:0/1039610277 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb66015ded0 0x7fb6600cab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:17.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb655400640 1 -- 172.21.15.47:0/1039610277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb66006b5c0 msgr2=0x7fb660099030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:17.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb655400640 1 --2- 172.21.15.47:0/1039610277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb66006b5c0 0x7fb660099030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:17.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb655400640 1 -- 172.21.15.47:0/1039610277 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6500094d0 con 0x7fb6600b1280 2024-02-19T15:24:17.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb65e800640 1 --2- 172.21.15.47:0/1039610277 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb66015ded0 0x7fb6600cab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:17.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb655400640 1 --2- 172.21.15.47:0/1039610277 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6600b1280 0x7fb6600c7470 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb64800c420 tx=0x7fb64800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:17.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb656a00640 1 -- 172.21.15.47:0/1039610277 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6480074e0 con 0x7fb6600b1280 2024-02-19T15:24:17.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb656a00640 1 -- 172.21.15.47:0/1039610277 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb648007640 con 0x7fb6600b1280 2024-02-19T15:24:17.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb665000640 1 -- 172.21.15.47:0/1039610277 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6600ad5d0 con 0x7fb6600b1280 2024-02-19T15:24:17.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb656a00640 1 -- 172.21.15.47:0/1039610277 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb648004a30 con 0x7fb6600b1280 2024-02-19T15:24:17.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb665000640 1 -- 172.21.15.47:0/1039610277 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6600adb90 con 0x7fb6600b1280 2024-02-19T15:24:17.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb665000640 1 -- 172.21.15.47:0/1039610277 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6600addb0 con 0x7fb6600b1280 2024-02-19T15:24:17.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.250+0000 7fb656a00640 1 -- 172.21.15.47:0/1039610277 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb648004b90 con 0x7fb6600b1280 2024-02-19T15:24:17.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.254+0000 7fb656a00640 1 --2- 172.21.15.47:0/1039610277 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb63c066e70 0x7fb63c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:17.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.254+0000 7fb656a00640 1 -- 172.21.15.47:0/1039610277 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb6480841c0 con 0x7fb6600b1280 2024-02-19T15:24:17.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.254+0000 7fb65de00640 1 --2- 172.21.15.47:0/1039610277 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb63c066e70 0x7fb63c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:17.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.254+0000 7fb65de00640 1 --2- 172.21.15.47:0/1039610277 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb63c066e70 0x7fb63c069330 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb6500ae000 tx=0x7fb650009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:17.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.258+0000 7fb656a00640 1 -- 172.21.15.47:0/1039610277 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb648051180 con 0x7fb6600b1280 2024-02-19T15:24:17.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.422+0000 7fb665000640 1 -- 172.21.15.47:0/1039610277 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb660075360 con 0x7fb6600b1280 2024-02-19T15:24:17.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.426+0000 7fb656a00640 1 -- 172.21.15.47:0/1039610277 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb648054840 con 0x7fb6600b1280 2024-02-19T15:24:17.427 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:24:17.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.426+0000 7fb665000640 1 -- 172.21.15.47:0/1039610277 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb63c066e70 msgr2=0x7fb63c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:17.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.426+0000 7fb665000640 1 --2- 172.21.15.47:0/1039610277 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb63c066e70 0x7fb63c069330 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb6500ae000 tx=0x7fb650009040 comp rx=0 tx=0).stop 2024-02-19T15:24:17.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.426+0000 7fb665000640 1 -- 172.21.15.47:0/1039610277 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6600b1280 msgr2=0x7fb6600c7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:17.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.426+0000 7fb665000640 1 --2- 172.21.15.47:0/1039610277 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6600b1280 0x7fb6600c7470 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb64800c420 tx=0x7fb64800c8e0 comp rx=0 tx=0).stop 2024-02-19T15:24:17.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.426+0000 7fb665000640 1 -- 172.21.15.47:0/1039610277 shutdown_connections 2024-02-19T15:24:17.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.426+0000 7fb665000640 1 --2- 172.21.15.47:0/1039610277 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb63c066e70 0x7fb63c069330 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:17.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.426+0000 7fb665000640 1 --2- 172.21.15.47:0/1039610277 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb66015ded0 0x7fb6600cab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:17.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.426+0000 7fb665000640 1 --2- 172.21.15.47:0/1039610277 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6600b1280 0x7fb6600c7470 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:17.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.426+0000 7fb665000640 1 --2- 172.21.15.47:0/1039610277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb66006b5c0 0x7fb660099030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:17.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.426+0000 7fb665000640 1 -- 172.21.15.47:0/1039610277 >> 172.21.15.47:0/1039610277 conn(0x7fb6600b32f0 msgr2=0x7fb66006d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:17.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.426+0000 7fb665000640 1 -- 172.21.15.47:0/1039610277 shutdown_connections 2024-02-19T15:24:17.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.426+0000 7fb665000640 1 -- 172.21.15.47:0/1039610277 wait complete. 2024-02-19T15:24:17.448 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-02-19T15:24:17.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.586+0000 7fbc63400640 1 Processor -- start 2024-02-19T15:24:17.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.586+0000 7fbc63400640 1 -- start start 2024-02-19T15:24:17.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.586+0000 7fbc63400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc5c0b2500 0x7fbc5c06ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:17.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.586+0000 7fbc63400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc5c06b5d0 con 0x7fbc5c1614e0 2024-02-19T15:24:17.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.586+0000 7fbc63400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc5c0b1640 con 0x7fbc5c15f620 2024-02-19T15:24:17.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.586+0000 7fbc63400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c0b1c00 con 0x7fbc5c0b2500 2024-02-19T15:24:17.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.586+0000 7fbc60a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbc5c1614e0 0x7fbc5c15eee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36494/0 (socket says 172.21.15.47:36494) 2024-02-19T15:24:17.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.586+0000 7fbc60a00640 1 -- 172.21.15.47:0/623456023 learned_addr learned my addr 172.21.15.47:0/623456023 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:17.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.586+0000 7fbc61400640 1 --2- 172.21.15.47:0/623456023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc5c0b2500 0x7fbc5c06ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:17.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.586+0000 7fbc5be00640 1 -- 172.21.15.47:0/623456023 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421201678 0 0) 0x7fbc5c06b5d0 con 0x7fbc5c1614e0 2024-02-19T15:24:17.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.586+0000 7fbc5be00640 1 -- 172.21.15.47:0/623456023 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc40003530 con 0x7fbc5c1614e0 2024-02-19T15:24:17.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc5be00640 1 -- 172.21.15.47:0/623456023 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2188972523 0 0) 0x7fbc40003530 con 0x7fbc5c1614e0 2024-02-19T15:24:17.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc5be00640 1 -- 172.21.15.47:0/623456023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc5c0b2500 msgr2=0x7fbc5c06ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:17.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc5be00640 1 --2- 172.21.15.47:0/623456023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc5c0b2500 0x7fbc5c06ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:17.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc5be00640 1 -- 172.21.15.47:0/623456023 >> v1:172.21.15.116:6789/0 conn(0x7fbc5c15f620 legacy=0x7fbc5c0b1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:17.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc5be00640 1 -- 172.21.15.47:0/623456023 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5c0b1920 con 0x7fbc5c1614e0 2024-02-19T15:24:17.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc5be00640 1 -- 172.21.15.47:0/623456023 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7fbc4c001680 con 0x7fbc5c1614e0 2024-02-19T15:24:17.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc5be00640 1 -- 172.21.15.47:0/623456023 >> v1:172.21.15.47:6789/0 conn(0x7fbc5c1614e0 legacy=0x7fbc5c15eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:17.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc5be00640 1 --2- 172.21.15.47:0/623456023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc5c0b2500 0x7fbc400039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:17.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc5be00640 1 --2- 172.21.15.47:0/623456023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc40003f80 0x7fbc400047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:17.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc5be00640 1 --2- 172.21.15.47:0/623456023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc40004d10 0x7fbc40005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:17.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc5be00640 1 -- 172.21.15.47:0/623456023 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c0b1c00 con 0x7fbc40003f80 2024-02-19T15:24:17.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc5be00640 1 -- 172.21.15.47:0/623456023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4000bad0 con 0x7fbc40004d10 2024-02-19T15:24:17.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc5be00640 1 -- 172.21.15.47:0/623456023 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4000bc90 con 0x7fbc5c0b2500 2024-02-19T15:24:17.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc53e00640 1 --2- 172.21.15.47:0/623456023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc40003f80 0x7fbc400047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:17.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc60a00640 1 --2- 172.21.15.47:0/623456023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc5c0b2500 0x7fbc400039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:17.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc61400640 1 --2- 172.21.15.47:0/623456023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc40004d10 0x7fbc40005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:17.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc53e00640 1 -- 172.21.15.47:0/623456023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc5c0b2500 msgr2=0x7fbc400039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:17.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc53e00640 1 --2- 172.21.15.47:0/623456023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc5c0b2500 0x7fbc400039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:17.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc53e00640 1 -- 172.21.15.47:0/623456023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc40004d10 msgr2=0x7fbc40005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:17.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc53e00640 1 --2- 172.21.15.47:0/623456023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc40004d10 0x7fbc40005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:17.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc53e00640 1 -- 172.21.15.47:0/623456023 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc40003530 con 0x7fbc40003f80 2024-02-19T15:24:17.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.590+0000 7fbc53e00640 1 --2- 172.21.15.47:0/623456023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc40003f80 0x7fbc400047d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fbc44001750 tx=0x7fbc4401c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32f81991b0f8637 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:17.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc5be00640 1 -- 172.21.15.47:0/623456023 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc440052d0 con 0x7fbc40003f80 2024-02-19T15:24:17.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc5be00640 1 -- 172.21.15.47:0/623456023 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc440a7050 con 0x7fbc40003f80 2024-02-19T15:24:17.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc5be00640 1 -- 172.21.15.47:0/623456023 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc44003600 con 0x7fbc40003f80 2024-02-19T15:24:17.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc63400640 1 -- 172.21.15.47:0/623456023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc40003f80 msgr2=0x7fbc400047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:17.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc63400640 1 --2- 172.21.15.47:0/623456023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc40003f80 0x7fbc400047d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fbc44001750 tx=0x7fbc4401c320 comp rx=0 tx=0).stop 2024-02-19T15:24:17.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc61400640 1 -- 172.21.15.47:0/623456023 reap_dead start 2024-02-19T15:24:17.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc63400640 1 -- 172.21.15.47:0/623456023 shutdown_connections 2024-02-19T15:24:17.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc63400640 1 -- 172.21.15.47:0/623456023 >> 172.21.15.47:0/623456023 conn(0x7fbc5c0b4fc0 msgr2=0x7fbc5c0b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:17.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc63400640 1 -- 172.21.15.47:0/623456023 shutdown_connections 2024-02-19T15:24:17.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc63400640 1 -- 172.21.15.47:0/623456023 wait complete. 2024-02-19T15:24:17.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc63400640 1 Processor -- start 2024-02-19T15:24:17.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc63400640 1 -- start start 2024-02-19T15:24:17.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc63400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc40003f80 0x7fbc5c157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:17.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc63400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc5c0c94e0 0x7fbc5c0ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:17.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc63400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc5c15ba00 0x7fbc5c15be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:17.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc63400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c0ad3e0 con 0x7fbc40003f80 2024-02-19T15:24:17.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc63400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c15c3c0 con 0x7fbc5c15ba00 2024-02-19T15:24:17.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc63400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c0cb4f0 con 0x7fbc5c0c94e0 2024-02-19T15:24:17.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc60a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc40003f80 0x7fbc5c157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:17.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc60a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc40003f80 0x7fbc5c157440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36232/0 (socket says 172.21.15.47:36232) 2024-02-19T15:24:17.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc60a00640 1 -- 172.21.15.47:0/2958682132 learned_addr learned my addr 172.21.15.47:0/2958682132 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:17.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc61400640 1 --2- 172.21.15.47:0/2958682132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc5c15ba00 0x7fbc5c15be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:17.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc53e00640 1 --2- 172.21.15.47:0/2958682132 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc5c0c94e0 0x7fbc5c0ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:17.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.594+0000 7fbc60a00640 1 -- 172.21.15.47:0/2958682132 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc5c0c94e0 msgr2=0x7fbc5c0ca3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:17.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.598+0000 7fbc60a00640 1 --2- 172.21.15.47:0/2958682132 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc5c0c94e0 0x7fbc5c0ca3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:17.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.598+0000 7fbc60a00640 1 -- 172.21.15.47:0/2958682132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc5c15ba00 msgr2=0x7fbc5c15be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:17.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.598+0000 7fbc60a00640 1 --2- 172.21.15.47:0/2958682132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc5c15ba00 0x7fbc5c15be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:17.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.598+0000 7fbc60a00640 1 -- 172.21.15.47:0/2958682132 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5c0cac50 con 0x7fbc40003f80 2024-02-19T15:24:17.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.598+0000 7fbc60a00640 1 --2- 172.21.15.47:0/2958682132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc40003f80 0x7fbc5c157440 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c001210 tx=0x7fbc4c012920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:17.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.598+0000 7fbc59600640 1 -- 172.21.15.47:0/2958682132 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c014810 con 0x7fbc40003f80 2024-02-19T15:24:17.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.598+0000 7fbc63400640 1 -- 172.21.15.47:0/2958682132 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc5c0caf30 con 0x7fbc40003f80 2024-02-19T15:24:17.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.598+0000 7fbc59600640 1 -- 172.21.15.47:0/2958682132 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc4c014970 con 0x7fbc40003f80 2024-02-19T15:24:17.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.598+0000 7fbc59600640 1 -- 172.21.15.47:0/2958682132 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c014d20 con 0x7fbc40003f80 2024-02-19T15:24:17.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.598+0000 7fbc63400640 1 -- 172.21.15.47:0/2958682132 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc5c157980 con 0x7fbc40003f80 2024-02-19T15:24:17.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.598+0000 7fbc63400640 1 -- 172.21.15.47:0/2958682132 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc5c0cc940 con 0x7fbc40003f80 2024-02-19T15:24:17.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.598+0000 7fbc59600640 1 -- 172.21.15.47:0/2958682132 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fbc4c0153e0 con 0x7fbc40003f80 2024-02-19T15:24:17.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.598+0000 7fbc59600640 1 --2- 172.21.15.47:0/2958682132 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbc38066e70 0x7fbc38069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:17.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.598+0000 7fbc59600640 1 -- 172.21.15.47:0/2958682132 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbc4c087210 con 0x7fbc40003f80 2024-02-19T15:24:17.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.598+0000 7fbc53e00640 1 --2- 172.21.15.47:0/2958682132 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbc38066e70 0x7fbc38069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:17.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.602+0000 7fbc53e00640 1 --2- 172.21.15.47:0/2958682132 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbc38066e70 0x7fbc38069330 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c0705c0 tx=0x7fbc440011a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:17.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.602+0000 7fbc59600640 1 -- 172.21.15.47:0/2958682132 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fbc4c054320 con 0x7fbc40003f80 2024-02-19T15:24:17.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:17.822+0000 7fbc63400640 1 -- 172.21.15.47:0/2958682132 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7fbc5c075360 con 0x7fbc40003f80 2024-02-19T15:24:18.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.730+0000 7fbc59600640 1 -- 172.21.15.47:0/2958682132 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v142) v1 ==== 147+0+0 (secure 0 0 0) 0x7fbc4c0579e0 con 0x7fbc40003f80 2024-02-19T15:24:18.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.730+0000 7fbc63400640 1 -- 172.21.15.47:0/2958682132 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbc38066e70 msgr2=0x7fbc38069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:18.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.730+0000 7fbc63400640 1 --2- 172.21.15.47:0/2958682132 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbc38066e70 0x7fbc38069330 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c0705c0 tx=0x7fbc440011a0 comp rx=0 tx=0).stop 2024-02-19T15:24:18.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.730+0000 7fbc63400640 1 -- 172.21.15.47:0/2958682132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc40003f80 msgr2=0x7fbc5c157440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:18.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.730+0000 7fbc63400640 1 --2- 172.21.15.47:0/2958682132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc40003f80 0x7fbc5c157440 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c001210 tx=0x7fbc4c012920 comp rx=0 tx=0).stop 2024-02-19T15:24:18.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.734+0000 7fbc63400640 1 -- 172.21.15.47:0/2958682132 shutdown_connections 2024-02-19T15:24:18.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.734+0000 7fbc63400640 1 --2- 172.21.15.47:0/2958682132 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbc38066e70 0x7fbc38069330 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:18.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.734+0000 7fbc63400640 1 --2- 172.21.15.47:0/2958682132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc5c15ba00 0x7fbc5c15be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:18.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.734+0000 7fbc63400640 1 --2- 172.21.15.47:0/2958682132 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc5c0c94e0 0x7fbc5c0ca3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:18.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.734+0000 7fbc63400640 1 --2- 172.21.15.47:0/2958682132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc40003f80 0x7fbc5c157440 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:18.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.734+0000 7fbc63400640 1 -- 172.21.15.47:0/2958682132 >> 172.21.15.47:0/2958682132 conn(0x7fbc5c0b4fc0 msgr2=0x7fbc5c15a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:18.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.734+0000 7fbc63400640 1 -- 172.21.15.47:0/2958682132 shutdown_connections 2024-02-19T15:24:18.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.734+0000 7fbc63400640 1 -- 172.21.15.47:0/2958682132 wait complete. 2024-02-19T15:24:18.761 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-19T15:24:18.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.906+0000 7fc0a1400640 1 Processor -- start 2024-02-19T15:24:18.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.906+0000 7fc0a1400640 1 -- start start 2024-02-19T15:24:18.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.910+0000 7fc0a1400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc09c15e160 0x7fc09c0d1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:18.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.910+0000 7fc0a1400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc09c06f220 0x7fc09c06dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:18.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.910+0000 7fc0a1400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09c071a70 con 0x7fc09c06e010 2024-02-19T15:24:18.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.910+0000 7fc0a1400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc09c06e440 con 0x7fc09c15e160 2024-02-19T15:24:18.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.910+0000 7fc0a1400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc09c0ced30 con 0x7fc09c06f220 2024-02-19T15:24:18.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.910+0000 7fc09a800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc09c15e160 0x7fc09c0d1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:18.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.910+0000 7fc09b200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc09c06e010 0x7fc09c06aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36502/0 (socket says 172.21.15.47:36502) 2024-02-19T15:24:18.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.910+0000 7fc09b200640 1 -- 172.21.15.47:0/1208852325 learned_addr learned my addr 172.21.15.47:0/1208852325 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:18.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.910+0000 7fc091e00640 1 --2- 172.21.15.47:0/1208852325 >> v2:172.21.15.116:3300/0 conn(0x7fc09c06f220 0x7fc09c06dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:18.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.910+0000 7fc09a800640 1 -- 172.21.15.47:0/1208852325 >> v2:172.21.15.116:3300/0 conn(0x7fc09c06f220 msgr2=0x7fc09c06dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:18.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.910+0000 7fc09a800640 1 --2- 172.21.15.47:0/1208852325 >> v2:172.21.15.116:3300/0 conn(0x7fc09c06f220 0x7fc09c06dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:18.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.910+0000 7fc09a800640 1 -- 172.21.15.47:0/1208852325 >> v1:172.21.15.47:6789/0 conn(0x7fc09c06e010 legacy=0x7fc09c06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:18.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.910+0000 7fc09a800640 1 -- 172.21.15.47:0/1208852325 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09c06f6b0 con 0x7fc09c15e160 2024-02-19T15:24:18.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.910+0000 7fc099e00640 1 -- 172.21.15.47:0/1208852325 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109915995 0 0) 0x7fc09c071a70 con 0x7fc09c06e010 2024-02-19T15:24:18.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.914+0000 7fc09a800640 1 --2- 172.21.15.47:0/1208852325 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc09c15e160 0x7fc09c0d1820 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc09c0cd760 tx=0x7fc08c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1666dd2a1ce8c4c5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:18.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.914+0000 7fc099e00640 1 -- 172.21.15.47:0/1208852325 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c0ab660 con 0x7fc09c15e160 2024-02-19T15:24:18.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.914+0000 7fc099e00640 1 -- 172.21.15.47:0/1208852325 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08c0ab7c0 con 0x7fc09c15e160 2024-02-19T15:24:18.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.914+0000 7fc099e00640 1 -- 172.21.15.47:0/1208852325 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c0abb50 con 0x7fc09c15e160 2024-02-19T15:24:18.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.914+0000 7fc0a1400640 1 -- 172.21.15.47:0/1208852325 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc09c15e160 msgr2=0x7fc09c0d1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:18.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.914+0000 7fc0a1400640 1 --2- 172.21.15.47:0/1208852325 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc09c15e160 0x7fc09c0d1820 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc09c0cd760 tx=0x7fc08c01c460 comp rx=0 tx=0).stop 2024-02-19T15:24:18.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.914+0000 7fc0a1400640 1 -- 172.21.15.47:0/1208852325 shutdown_connections 2024-02-19T15:24:18.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.914+0000 7fc0a1400640 1 --2- 172.21.15.47:0/1208852325 >> v2:172.21.15.116:3300/0 conn(0x7fc09c06f220 0x7fc09c06dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:18.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.914+0000 7fc0a1400640 1 --2- 172.21.15.47:0/1208852325 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc09c15e160 0x7fc09c0d1820 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:18.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.914+0000 7fc0a1400640 1 -- 172.21.15.47:0/1208852325 >> 172.21.15.47:0/1208852325 conn(0x7fc09c0d0d30 msgr2=0x7fc09c0d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:18.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.914+0000 7fc0a1400640 1 -- 172.21.15.47:0/1208852325 shutdown_connections 2024-02-19T15:24:18.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.914+0000 7fc0a1400640 1 -- 172.21.15.47:0/1208852325 wait complete. 2024-02-19T15:24:18.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.914+0000 7fc0a1400640 1 Processor -- start 2024-02-19T15:24:18.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.914+0000 7fc0a1400640 1 -- start start 2024-02-19T15:24:18.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.914+0000 7fc0a1400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc09c06e010 0x7fc09c0b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:18.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.914+0000 7fc0a1400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc09c06f220 0x7fc09c0b1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:18.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.914+0000 7fc0a1400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc09c15e160 0x7fc09c14d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:18.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.914+0000 7fc0a1400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c0b1660 con 0x7fc09c15e160 2024-02-19T15:24:18.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.914+0000 7fc0a1400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c0b6b30 con 0x7fc09c06e010 2024-02-19T15:24:18.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.914+0000 7fc0a1400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c0b3780 con 0x7fc09c06f220 2024-02-19T15:24:18.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.918+0000 7fc09b200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc09c15e160 0x7fc09c14d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:18.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.918+0000 7fc09b200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc09c15e160 0x7fc09c14d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36246/0 (socket says 172.21.15.47:36246) 2024-02-19T15:24:18.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.918+0000 7fc09b200640 1 -- 172.21.15.47:0/4164755728 learned_addr learned my addr 172.21.15.47:0/4164755728 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:18.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.918+0000 7fc091e00640 1 --2- 172.21.15.47:0/4164755728 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc09c06f220 0x7fc09c0b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:18.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.918+0000 7fc09a800640 1 --2- 172.21.15.47:0/4164755728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc09c06e010 0x7fc09c0b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:18.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.918+0000 7fc09b200640 1 -- 172.21.15.47:0/4164755728 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc09c06f220 msgr2=0x7fc09c0b1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:18.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.918+0000 7fc09b200640 1 --2- 172.21.15.47:0/4164755728 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc09c06f220 0x7fc09c0b1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:18.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.918+0000 7fc09b200640 1 -- 172.21.15.47:0/4164755728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc09c06e010 msgr2=0x7fc09c0b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:18.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.918+0000 7fc09b200640 1 --2- 172.21.15.47:0/4164755728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc09c06e010 0x7fc09c0b2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:18.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.918+0000 7fc09b200640 1 -- 172.21.15.47:0/4164755728 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c009420 con 0x7fc09c15e160 2024-02-19T15:24:18.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.918+0000 7fc091e00640 1 --2- 172.21.15.47:0/4164755728 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc09c06f220 0x7fc09c0b1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:18.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.918+0000 7fc09b200640 1 --2- 172.21.15.47:0/4164755728 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc09c15e160 0x7fc09c14d210 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc09400c7d0 tx=0x7fc09400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:18.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.918+0000 7fc093400640 1 -- 172.21.15.47:0/4164755728 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09400a5e0 con 0x7fc09c15e160 2024-02-19T15:24:18.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.918+0000 7fc093400640 1 -- 172.21.15.47:0/4164755728 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09400a740 con 0x7fc09c15e160 2024-02-19T15:24:18.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.918+0000 7fc0a1400640 1 -- 172.21.15.47:0/4164755728 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc09c14d750 con 0x7fc09c15e160 2024-02-19T15:24:18.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.918+0000 7fc093400640 1 -- 172.21.15.47:0/4164755728 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09400ab30 con 0x7fc09c15e160 2024-02-19T15:24:18.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.918+0000 7fc0a1400640 1 -- 172.21.15.47:0/4164755728 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc09c0b34a0 con 0x7fc09c15e160 2024-02-19T15:24:18.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.918+0000 7fc0a1400640 1 -- 172.21.15.47:0/4164755728 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09c06dd40 con 0x7fc09c15e160 2024-02-19T15:24:18.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.918+0000 7fc093400640 1 -- 172.21.15.47:0/4164755728 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc09400ac90 con 0x7fc09c15e160 2024-02-19T15:24:18.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.922+0000 7fc093400640 1 --2- 172.21.15.47:0/4164755728 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc078066e40 0x7fc078069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:18.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.922+0000 7fc093400640 1 -- 172.21.15.47:0/4164755728 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc094083e80 con 0x7fc09c15e160 2024-02-19T15:24:18.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.922+0000 7fc09a800640 1 --2- 172.21.15.47:0/4164755728 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc078066e40 0x7fc078069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:18.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.922+0000 7fc09a800640 1 --2- 172.21.15.47:0/4164755728 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc078066e40 0x7fc078069300 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc08c0af000 tx=0x7fc08c0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:18.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:18.922+0000 7fc093400640 1 -- 172.21.15.47:0/4164755728 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc094051010 con 0x7fc09c15e160 2024-02-19T15:24:19.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.086+0000 7fc0a1400640 1 -- 172.21.15.47:0/4164755728 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc09c075360 con 0x7fc09c15e160 2024-02-19T15:24:19.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.090+0000 7fc093400640 1 -- 172.21.15.47:0/4164755728 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc0940180c0 con 0x7fc09c15e160 2024-02-19T15:24:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.090+0000 7fc0a1400640 1 -- 172.21.15.47:0/4164755728 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc078066e40 msgr2=0x7fc078069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.090+0000 7fc0a1400640 1 --2- 172.21.15.47:0/4164755728 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc078066e40 0x7fc078069300 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc08c0af000 tx=0x7fc08c0a7000 comp rx=0 tx=0).stop 2024-02-19T15:24:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.090+0000 7fc0a1400640 1 -- 172.21.15.47:0/4164755728 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc09c15e160 msgr2=0x7fc09c14d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.090+0000 7fc0a1400640 1 --2- 172.21.15.47:0/4164755728 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc09c15e160 0x7fc09c14d210 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc09400c7d0 tx=0x7fc09400cc90 comp rx=0 tx=0).stop 2024-02-19T15:24:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.090+0000 7fc0a1400640 1 -- 172.21.15.47:0/4164755728 shutdown_connections 2024-02-19T15:24:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.090+0000 7fc0a1400640 1 --2- 172.21.15.47:0/4164755728 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc078066e40 0x7fc078069300 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.090+0000 7fc0a1400640 1 --2- 172.21.15.47:0/4164755728 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc09c15e160 0x7fc09c14d210 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.090+0000 7fc0a1400640 1 --2- 172.21.15.47:0/4164755728 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc09c06f220 0x7fc09c0b1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.090+0000 7fc0a1400640 1 --2- 172.21.15.47:0/4164755728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc09c06e010 0x7fc09c0b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.090+0000 7fc0a1400640 1 -- 172.21.15.47:0/4164755728 >> 172.21.15.47:0/4164755728 conn(0x7fc09c0d0d30 msgr2=0x7fc09c0d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.090+0000 7fc0a1400640 1 -- 172.21.15.47:0/4164755728 shutdown_connections 2024-02-19T15:24:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.090+0000 7fc0a1400640 1 -- 172.21.15.47:0/4164755728 wait complete. 2024-02-19T15:24:19.114 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T15:24:19.114 DEBUG:teuthology.orchestra.run.smithi047:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:24:19.162 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:24:19.230 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T15:24:19.230 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:24:19.231 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:19.239 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:24:19.239 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:24:19.239 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:19.248 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:24:19.248 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:24:19.248 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:19.290 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:24:19.291 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:24:19.291 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:19.299 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:24:19.299 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-02-19T15:24:19.299 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:24:19.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.470+0000 7f2bcc800640 1 Processor -- start 2024-02-19T15:24:19.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.470+0000 7f2bcc800640 1 -- start start 2024-02-19T15:24:19.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.474+0000 7f2bcc800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2bc415d6e0 0x7f2bc4162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:19.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.474+0000 7f2bcc800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc40718c0 con 0x7f2bc4071350 2024-02-19T15:24:19.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.474+0000 7f2bcc800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc4071ba0 con 0x7f2bc415b1a0 2024-02-19T15:24:19.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.474+0000 7f2bcc800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bc4071e80 con 0x7f2bc415d6e0 2024-02-19T15:24:19.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.474+0000 7f2bc9400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2bc415d6e0 0x7f2bc4162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:19.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.474+0000 7f2bca800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f2bc4071350 0x7f2bc4164c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36508/0 (socket says 172.21.15.47:36508) 2024-02-19T15:24:19.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.474+0000 7f2bc9400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2bc415d6e0 0x7f2bc4162740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36248/0 (socket says 172.21.15.47:36248) 2024-02-19T15:24:19.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.474+0000 7f2bca800640 1 -- 172.21.15.47:0/433956087 learned_addr learned my addr 172.21.15.47:0/433956087 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:19.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.474+0000 7f2bc9400640 1 -- 172.21.15.47:0/433956087 >> v1:172.21.15.116:6789/0 conn(0x7f2bc415b1a0 legacy=0x7f2bc415b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:19.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.474+0000 7f2bc9400640 1 -- 172.21.15.47:0/433956087 >> v1:172.21.15.47:6789/0 conn(0x7f2bc4071350 legacy=0x7f2bc4164c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:19.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.474+0000 7f2bc9400640 1 -- 172.21.15.47:0/433956087 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc4072160 con 0x7f2bc415d6e0 2024-02-19T15:24:19.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.474+0000 7f2bc8a00640 1 -- 172.21.15.47:0/433956087 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928812333 0 0) 0x7f2bc40718c0 con 0x7f2bc4071350 2024-02-19T15:24:19.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.478+0000 7f2bc9400640 1 --2- 172.21.15.47:0/433956087 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bc415d6e0 0x7f2bc4162740 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f2bb40097f0 tx=0x7f2bb401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f15f4e0a0a7f769d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:19.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.478+0000 7f2bc8a00640 1 -- 172.21.15.47:0/433956087 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb40ab680 con 0x7f2bc415d6e0 2024-02-19T15:24:19.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.478+0000 7f2bc8a00640 1 -- 172.21.15.47:0/433956087 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb40ab7e0 con 0x7f2bc415d6e0 2024-02-19T15:24:19.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.478+0000 7f2bc8a00640 1 -- 172.21.15.47:0/433956087 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb40abb70 con 0x7f2bc415d6e0 2024-02-19T15:24:19.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.478+0000 7f2bcc800640 1 -- 172.21.15.47:0/433956087 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bc415d6e0 msgr2=0x7f2bc4162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:19.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.478+0000 7f2bcc800640 1 --2- 172.21.15.47:0/433956087 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bc415d6e0 0x7f2bc4162740 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f2bb40097f0 tx=0x7f2bb401c460 comp rx=0 tx=0).stop 2024-02-19T15:24:19.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.478+0000 7f2bcc800640 1 -- 172.21.15.47:0/433956087 shutdown_connections 2024-02-19T15:24:19.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.478+0000 7f2bcc800640 1 --2- 172.21.15.47:0/433956087 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bc415d6e0 0x7f2bc4162740 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:19.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.478+0000 7f2bcc800640 1 -- 172.21.15.47:0/433956087 >> 172.21.15.47:0/433956087 conn(0x7f2bc40b16a0 msgr2=0x7f2bc40b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:19.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.478+0000 7f2bcc800640 1 -- 172.21.15.47:0/433956087 shutdown_connections 2024-02-19T15:24:19.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.478+0000 7f2bcc800640 1 -- 172.21.15.47:0/433956087 wait complete. 2024-02-19T15:24:19.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.478+0000 7f2bcc800640 1 Processor -- start 2024-02-19T15:24:19.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.478+0000 7f2bcc800640 1 -- start start 2024-02-19T15:24:19.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.478+0000 7f2bcc800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc4071350 0x7f2bc4165110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:19.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.478+0000 7f2bcc800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2bc415b1a0 0x7f2bc40b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:19.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.478+0000 7f2bcc800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bc415d6e0 0x7f2bc41572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:19.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bcc800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc4140cb0 con 0x7f2bc415d6e0 2024-02-19T15:24:19.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bcc800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc4165650 con 0x7f2bc4071350 2024-02-19T15:24:19.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bcc800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc40b4140 con 0x7f2bc415b1a0 2024-02-19T15:24:19.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bca800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bc415d6e0 0x7f2bc41572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:19.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bca800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bc415d6e0 0x7f2bc41572b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36250/0 (socket says 172.21.15.47:36250) 2024-02-19T15:24:19.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bca800640 1 -- 172.21.15.47:0/912509251 learned_addr learned my addr 172.21.15.47:0/912509251 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:19.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bc9e00640 1 --2- 172.21.15.47:0/912509251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc4071350 0x7f2bc4165110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:19.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bc9400640 1 --2- 172.21.15.47:0/912509251 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2bc415b1a0 0x7f2bc40b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:19.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bca800640 1 -- 172.21.15.47:0/912509251 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2bc415b1a0 msgr2=0x7f2bc40b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:19.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bca800640 1 --2- 172.21.15.47:0/912509251 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2bc415b1a0 0x7f2bc40b31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:19.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bca800640 1 -- 172.21.15.47:0/912509251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc4071350 msgr2=0x7f2bc4165110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:19.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bca800640 1 --2- 172.21.15.47:0/912509251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc4071350 0x7f2bc4165110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:19.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bca800640 1 -- 172.21.15.47:0/912509251 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb4009420 con 0x7f2bc415d6e0 2024-02-19T15:24:19.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bc9400640 1 --2- 172.21.15.47:0/912509251 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2bc415b1a0 0x7f2bc40b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:19.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bca800640 1 --2- 172.21.15.47:0/912509251 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bc415d6e0 0x7f2bc41572b0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f2bc000c700 tx=0x7f2bc000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:19.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bbe000640 1 -- 172.21.15.47:0/912509251 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc000ce10 con 0x7f2bc415d6e0 2024-02-19T15:24:19.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bbe000640 1 -- 172.21.15.47:0/912509251 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc000a5e0 con 0x7f2bc415d6e0 2024-02-19T15:24:19.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bbe000640 1 -- 172.21.15.47:0/912509251 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc000aa10 con 0x7f2bc415d6e0 2024-02-19T15:24:19.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bcc800640 1 -- 172.21.15.47:0/912509251 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc40b3b80 con 0x7f2bc415d6e0 2024-02-19T15:24:19.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bcc800640 1 -- 172.21.15.47:0/912509251 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc4141080 con 0x7f2bc415d6e0 2024-02-19T15:24:19.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bcc800640 1 -- 172.21.15.47:0/912509251 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc4075360 con 0x7f2bc415d6e0 2024-02-19T15:24:19.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.482+0000 7f2bbe000640 1 -- 172.21.15.47:0/912509251 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2bc000ab70 con 0x7f2bc415d6e0 2024-02-19T15:24:19.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.486+0000 7f2bbe000640 1 --2- 172.21.15.47:0/912509251 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2ba8066e40 0x7f2ba8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:19.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.486+0000 7f2bbe000640 1 -- 172.21.15.47:0/912509251 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2bc0083d30 con 0x7f2bc415d6e0 2024-02-19T15:24:19.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.486+0000 7f2bc9e00640 1 --2- 172.21.15.47:0/912509251 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2ba8066e40 0x7f2ba8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:19.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.486+0000 7f2bc9e00640 1 --2- 172.21.15.47:0/912509251 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2ba8066e40 0x7f2ba8069300 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8007fd0 tx=0x7f2bb8007e40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:19.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.490+0000 7f2bbe000640 1 -- 172.21.15.47:0/912509251 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2bc0050f60 con 0x7f2bc415d6e0 2024-02-19T15:24:19.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.686+0000 7f2bcc800640 1 -- 172.21.15.47:0/912509251 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bc415a8e0 con 0x7f2bc415d6e0 2024-02-19T15:24:19.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.686+0000 7f2bbe000640 1 -- 172.21.15.47:0/912509251 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f2bc0018090 con 0x7f2bc415d6e0 2024-02-19T15:24:19.688 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:19.688 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5132,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/548836925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":548836925},{"type":"v1","addr":"172.21.15.116:6836","nonce":548836925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5135,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3530214438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3530214438},{"type":"v1","addr":"172.21.15.116:6837","nonce":3530214438}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/3433978846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3433978846},{"type":"v1","addr":"172.21.15.47:6834","nonce":3433978846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3362905657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3362905657},{"type":"v1","addr":"172.21.15.47:6835","nonce":3362905657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[]} 2024-02-19T15:24:19.689 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 27 2024-02-19T15:24:19.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.686+0000 7f2bcc800640 1 -- 172.21.15.47:0/912509251 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2ba8066e40 msgr2=0x7f2ba8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:19.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.686+0000 7f2bcc800640 1 --2- 172.21.15.47:0/912509251 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2ba8066e40 0x7f2ba8069300 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8007fd0 tx=0x7f2bb8007e40 comp rx=0 tx=0).stop 2024-02-19T15:24:19.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.686+0000 7f2bcc800640 1 -- 172.21.15.47:0/912509251 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bc415d6e0 msgr2=0x7f2bc41572b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:19.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.686+0000 7f2bcc800640 1 --2- 172.21.15.47:0/912509251 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bc415d6e0 0x7f2bc41572b0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f2bc000c700 tx=0x7f2bc000cbc0 comp rx=0 tx=0).stop 2024-02-19T15:24:19.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.686+0000 7f2bcc800640 1 -- 172.21.15.47:0/912509251 shutdown_connections 2024-02-19T15:24:19.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.686+0000 7f2bcc800640 1 --2- 172.21.15.47:0/912509251 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2ba8066e40 0x7f2ba8069300 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:19.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.686+0000 7f2bcc800640 1 --2- 172.21.15.47:0/912509251 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bc415d6e0 0x7f2bc41572b0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:19.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.686+0000 7f2bcc800640 1 --2- 172.21.15.47:0/912509251 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2bc415b1a0 0x7f2bc40b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:19.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.686+0000 7f2bcc800640 1 --2- 172.21.15.47:0/912509251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc4071350 0x7f2bc4165110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:19.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.686+0000 7f2bcc800640 1 -- 172.21.15.47:0/912509251 >> 172.21.15.47:0/912509251 conn(0x7f2bc40b16a0 msgr2=0x7f2bc415db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:19.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.686+0000 7f2bcc800640 1 -- 172.21.15.47:0/912509251 shutdown_connections 2024-02-19T15:24:19.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.686+0000 7f2bcc800640 1 -- 172.21.15.47:0/912509251 wait complete. 2024-02-19T15:24:19.709 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T15:24:19.710 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-19T15:24:19.710 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T15:24:19.710 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:24:19.710 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T15:24:19.710 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-19T15:24:19.710 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T15:24:19.710 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:24:19.711 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T15:24:19.711 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-19T15:24:19.711 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T15:24:19.711 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:24:19.711 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T15:24:19.711 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-19T15:24:19.711 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T15:24:19.712 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:24:19.722 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T15:24:19.723 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T15:24:19.723 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T15:24:19.723 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T15:24:19.725 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T15:24:19.725 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T15:24:19.727 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T15:24:19.728 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T15:24:19.769 INFO:tasks.ceph.mds.c:Started 2024-02-19T15:24:19.769 DEBUG:teuthology.parallel:result is None 2024-02-19T15:24:19.771 INFO:tasks.ceph.mds.a:Started 2024-02-19T15:24:19.771 DEBUG:teuthology.parallel:result is None 2024-02-19T15:24:19.773 INFO:tasks.ceph.mds.b:Started 2024-02-19T15:24:19.773 DEBUG:teuthology.parallel:result is None 2024-02-19T15:24:19.775 INFO:tasks.ceph.mds.d:Started 2024-02-19T15:24:19.775 DEBUG:teuthology.parallel:result is None 2024-02-19T15:24:19.776 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-19T15:24:19.859 INFO:tasks.ceph.mds.a.smithi047.stdout:starting mds.a at 2024-02-19T15:24:19.859 INFO:tasks.ceph.mds.c.smithi047.stdout:starting mds.c at 2024-02-19T15:24:19.862 INFO:tasks.ceph.mds.b.smithi116.stdout:starting mds.b at 2024-02-19T15:24:19.862 INFO:tasks.ceph.mds.d.smithi116.stdout:starting mds.d at 2024-02-19T15:24:19.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.918+0000 7f1c09800640 1 Processor -- start 2024-02-19T15:24:19.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.918+0000 7f1c09800640 1 -- start start 2024-02-19T15:24:19.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.922+0000 7f1c09800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1c04161b20 0x7f1c040cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:19.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.922+0000 7f1c09800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1c040cf460 0x7f1c040cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:19.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.922+0000 7f1c09800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0406d180 con 0x7f1c0415df20 2024-02-19T15:24:19.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.922+0000 7f1c09800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c040cbd80 con 0x7f1c04161b20 2024-02-19T15:24:19.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.922+0000 7f1c09800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c0406cf90 con 0x7f1c040cf460 2024-02-19T15:24:19.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.922+0000 7f1bfb400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1c04161b20 0x7f1c040cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:19.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.922+0000 7f1bfb400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1c04161b20 0x7f1c040cef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36296/0 (socket says 172.21.15.47:36296) 2024-02-19T15:24:19.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.922+0000 7f1bfb400640 1 -- 172.21.15.47:0/3519127084 learned_addr learned my addr 172.21.15.47:0/3519127084 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:19.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.922+0000 7f1c08e00640 1 --2- 172.21.15.47:0/3519127084 >> v2:172.21.15.116:3300/0 conn(0x7f1c040cf460 0x7f1c040cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:19.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.922+0000 7f1bfb400640 1 -- 172.21.15.47:0/3519127084 >> v2:172.21.15.116:3300/0 conn(0x7f1c040cf460 msgr2=0x7f1c040cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:19.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.922+0000 7f1bfb400640 1 --2- 172.21.15.47:0/3519127084 >> v2:172.21.15.116:3300/0 conn(0x7f1c040cf460 0x7f1c040cc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:19.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.922+0000 7f1bfb400640 1 -- 172.21.15.47:0/3519127084 >> v1:172.21.15.47:6789/0 conn(0x7f1c0415df20 legacy=0x7f1c0406c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:19.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.922+0000 7f1bfb400640 1 -- 172.21.15.47:0/3519127084 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0406d460 con 0x7f1c04161b20 2024-02-19T15:24:19.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.922+0000 7f1c08e00640 1 --2- 172.21.15.47:0/3519127084 >> v2:172.21.15.116:3300/0 conn(0x7f1c040cf460 0x7f1c040cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:19.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.922+0000 7f1c03400640 1 -- 172.21.15.47:0/3519127084 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 545623097 0 0) 0x7f1c0406d180 con 0x7f1c0415df20 2024-02-19T15:24:19.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.922+0000 7f1bfb400640 1 --2- 172.21.15.47:0/3519127084 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c04161b20 0x7f1c040cef20 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f1c040ca250 tx=0x7f1bf401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74816afbe761c9a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:19.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.922+0000 7f1c03400640 1 -- 172.21.15.47:0/3519127084 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf40ab600 con 0x7f1c04161b20 2024-02-19T15:24:19.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.922+0000 7f1c03400640 1 -- 172.21.15.47:0/3519127084 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf40a7050 con 0x7f1c04161b20 2024-02-19T15:24:19.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.922+0000 7f1c03400640 1 -- 172.21.15.47:0/3519127084 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf40ab980 con 0x7f1c04161b20 2024-02-19T15:24:19.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1c09800640 1 -- 172.21.15.47:0/3519127084 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c04161b20 msgr2=0x7f1c040cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:19.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1c09800640 1 --2- 172.21.15.47:0/3519127084 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c04161b20 0x7f1c040cef20 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f1c040ca250 tx=0x7f1bf401c490 comp rx=0 tx=0).stop 2024-02-19T15:24:19.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1c09800640 1 -- 172.21.15.47:0/3519127084 shutdown_connections 2024-02-19T15:24:19.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1c09800640 1 --2- 172.21.15.47:0/3519127084 >> v2:172.21.15.116:3300/0 conn(0x7f1c040cf460 0x7f1c040cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:19.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1c09800640 1 --2- 172.21.15.47:0/3519127084 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c04161b20 0x7f1c040cef20 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:19.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1c09800640 1 -- 172.21.15.47:0/3519127084 >> 172.21.15.47:0/3519127084 conn(0x7f1c040b1840 msgr2=0x7f1c040b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:19.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1c09800640 1 -- 172.21.15.47:0/3519127084 shutdown_connections 2024-02-19T15:24:19.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1c09800640 1 -- 172.21.15.47:0/3519127084 wait complete. 2024-02-19T15:24:19.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1c09800640 1 Processor -- start 2024-02-19T15:24:19.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1c09800640 1 -- start start 2024-02-19T15:24:19.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1c09800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c040cf460 0x7f1c04151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:19.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1c09800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c0415df20 0x7f1c041592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:19.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1c09800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c04161b20 0x7f1c041597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:19.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1c09800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c04152190 con 0x7f1c0415df20 2024-02-19T15:24:19.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1c09800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0415b7b0 con 0x7f1c04161b20 2024-02-19T15:24:19.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1c09800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0415ba90 con 0x7f1c040cf460 2024-02-19T15:24:19.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1bfb400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c0415df20 0x7f1c041592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:19.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1bfb400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c0415df20 0x7f1c041592b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36304/0 (socket says 172.21.15.47:36304) 2024-02-19T15:24:19.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1bfb400640 1 -- 172.21.15.47:0/1802660516 learned_addr learned my addr 172.21.15.47:0/1802660516 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:19.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1c08e00640 1 --2- 172.21.15.47:0/1802660516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c04161b20 0x7f1c041597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:19.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1c03e00640 1 --2- 172.21.15.47:0/1802660516 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c040cf460 0x7f1c04151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:19.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1bfb400640 1 -- 172.21.15.47:0/1802660516 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c040cf460 msgr2=0x7f1c04151c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:19.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1bfb400640 1 --2- 172.21.15.47:0/1802660516 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c040cf460 0x7f1c04151c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:19.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1bfb400640 1 -- 172.21.15.47:0/1802660516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c04161b20 msgr2=0x7f1c041597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:19.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1bfb400640 1 --2- 172.21.15.47:0/1802660516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c04161b20 0x7f1c041597f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:19.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.926+0000 7f1bfb400640 1 -- 172.21.15.47:0/1802660516 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf40094d0 con 0x7f1c0415df20 2024-02-19T15:24:19.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.930+0000 7f1bfb400640 1 --2- 172.21.15.47:0/1802660516 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c0415df20 0x7f1c041592b0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1bf401ab40 tx=0x7f1bf4019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:19.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.930+0000 7f1c00c00640 1 -- 172.21.15.47:0/1802660516 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf40b6670 con 0x7f1c0415df20 2024-02-19T15:24:19.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.930+0000 7f1c00c00640 1 -- 172.21.15.47:0/1802660516 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf40b67d0 con 0x7f1c0415df20 2024-02-19T15:24:19.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.930+0000 7f1c00c00640 1 -- 172.21.15.47:0/1802660516 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf40b6b60 con 0x7f1c0415df20 2024-02-19T15:24:19.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.930+0000 7f1c09800640 1 -- 172.21.15.47:0/1802660516 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c0415bd70 con 0x7f1c0415df20 2024-02-19T15:24:19.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.930+0000 7f1c09800640 1 -- 172.21.15.47:0/1802660516 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c0415c330 con 0x7f1c0415df20 2024-02-19T15:24:19.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.930+0000 7f1c09800640 1 -- 172.21.15.47:0/1802660516 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c0409def0 con 0x7f1c0415df20 2024-02-19T15:24:19.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.930+0000 7f1c00c00640 1 -- 172.21.15.47:0/1802660516 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1bf40b2070 con 0x7f1c0415df20 2024-02-19T15:24:19.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.930+0000 7f1c00c00640 1 --2- 172.21.15.47:0/1802660516 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1be0066e90 0x7f1be0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:19.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.930+0000 7f1c00c00640 1 -- 172.21.15.47:0/1802660516 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1bf411eda0 con 0x7f1c0415df20 2024-02-19T15:24:19.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.930+0000 7f1c03e00640 1 --2- 172.21.15.47:0/1802660516 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1be0066e90 0x7f1be0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:19.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.934+0000 7f1c03e00640 1 --2- 172.21.15.47:0/1802660516 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1be0066e90 0x7f1be0069350 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1bec000f70 tx=0x7f1bec009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:19.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:19.934+0000 7f1c00c00640 1 -- 172.21.15.47:0/1802660516 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1bf40ebe70 con 0x7f1c0415df20 2024-02-19T15:24:20.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:20.114+0000 7f1c09800640 1 -- 172.21.15.47:0/1802660516 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f1c0415c520 con 0x7f1c0415df20 2024-02-19T15:24:21.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.106+0000 7f1c00c00640 1 -- 172.21.15.47:0/1802660516 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v53) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1bf40b0080 con 0x7f1c0415df20 2024-02-19T15:24:21.108 INFO:teuthology.orchestra.run.smithi047.stderr: removed all blocklist entries 2024-02-19T15:24:21.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.106+0000 7f1c09800640 1 -- 172.21.15.47:0/1802660516 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1be0066e90 msgr2=0x7f1be0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:21.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.106+0000 7f1c09800640 1 --2- 172.21.15.47:0/1802660516 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1be0066e90 0x7f1be0069350 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1bec000f70 tx=0x7f1bec009040 comp rx=0 tx=0).stop 2024-02-19T15:24:21.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.106+0000 7f1c09800640 1 -- 172.21.15.47:0/1802660516 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c0415df20 msgr2=0x7f1c041592b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:21.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.106+0000 7f1c09800640 1 --2- 172.21.15.47:0/1802660516 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c0415df20 0x7f1c041592b0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1bf401ab40 tx=0x7f1bf4019230 comp rx=0 tx=0).stop 2024-02-19T15:24:21.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.106+0000 7f1c09800640 1 -- 172.21.15.47:0/1802660516 shutdown_connections 2024-02-19T15:24:21.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.106+0000 7f1c09800640 1 --2- 172.21.15.47:0/1802660516 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1be0066e90 0x7f1be0069350 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.106+0000 7f1c09800640 1 --2- 172.21.15.47:0/1802660516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c04161b20 0x7f1c041597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.106+0000 7f1c09800640 1 --2- 172.21.15.47:0/1802660516 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c0415df20 0x7f1c041592b0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.106+0000 7f1c09800640 1 --2- 172.21.15.47:0/1802660516 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c040cf460 0x7f1c04151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.106+0000 7f1c09800640 1 -- 172.21.15.47:0/1802660516 >> 172.21.15.47:0/1802660516 conn(0x7f1c040b1840 msgr2=0x7f1c040b1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:21.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.106+0000 7f1c09800640 1 -- 172.21.15.47:0/1802660516 shutdown_connections 2024-02-19T15:24:21.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.106+0000 7f1c09800640 1 -- 172.21.15.47:0/1802660516 wait complete. 2024-02-19T15:24:21.136 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-19T15:24:21.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.278+0000 7f9b98e00640 1 Processor -- start 2024-02-19T15:24:21.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.278+0000 7f9b98e00640 1 -- start start 2024-02-19T15:24:21.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.278+0000 7f9b98e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9b940cded0 0x7f9b940ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:21.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.278+0000 7f9b98e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b940ceb40 0x7f9b9406bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:21.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.278+0000 7f9b98e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b940cce30 con 0x7f9b9406d650 2024-02-19T15:24:21.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.278+0000 7f9b98e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b940cefd0 con 0x7f9b940cded0 2024-02-19T15:24:21.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.278+0000 7f9b98e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b940cd110 con 0x7f9b940ceb40 2024-02-19T15:24:21.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.278+0000 7f9b92800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b940ceb40 0x7f9b9406bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:21.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.278+0000 7f9b91e00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9b9406d650 0x7f9b940d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:37722/0 (socket says 172.21.15.47:37722) 2024-02-19T15:24:21.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.278+0000 7f9b92800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b940ceb40 0x7f9b9406bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:58628/0 (socket says 172.21.15.47:58628) 2024-02-19T15:24:21.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.278+0000 7f9b91e00640 1 -- 172.21.15.47:0/2889669496 learned_addr learned my addr 172.21.15.47:0/2889669496 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:21.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.278+0000 7f9b89400640 1 --2- 172.21.15.47:0/2889669496 >> v2:172.21.15.116:3300/0 conn(0x7f9b940cded0 0x7f9b940ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:21.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.278+0000 7f9b92800640 1 -- 172.21.15.47:0/2889669496 >> v2:172.21.15.116:3300/0 conn(0x7f9b940cded0 msgr2=0x7f9b940ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:21.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.278+0000 7f9b92800640 1 --2- 172.21.15.47:0/2889669496 >> v2:172.21.15.116:3300/0 conn(0x7f9b940cded0 0x7f9b940ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.278+0000 7f9b92800640 1 -- 172.21.15.47:0/2889669496 >> v1:172.21.15.116:6789/0 conn(0x7f9b9406d650 legacy=0x7f9b940d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:21.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.278+0000 7f9b92800640 1 -- 172.21.15.47:0/2889669496 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b940cd3f0 con 0x7f9b940ceb40 2024-02-19T15:24:21.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.282+0000 7f9b92800640 1 --2- 172.21.15.47:0/2889669496 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b940ceb40 0x7f9b9406bd60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c0097c0 tx=0x7f9b8c01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69a0cd8051a59b97 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:21.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.282+0000 7f9b91400640 1 -- 172.21.15.47:0/2889669496 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c0ac640 con 0x7f9b940ceb40 2024-02-19T15:24:21.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.282+0000 7f9b91400640 1 -- 172.21.15.47:0/2889669496 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8c0ac7a0 con 0x7f9b940ceb40 2024-02-19T15:24:21.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.282+0000 7f9b91400640 1 -- 172.21.15.47:0/2889669496 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c0acb30 con 0x7f9b940ceb40 2024-02-19T15:24:21.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.282+0000 7f9b98e00640 1 -- 172.21.15.47:0/2889669496 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b940ceb40 msgr2=0x7f9b9406bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:21.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.282+0000 7f9b98e00640 1 --2- 172.21.15.47:0/2889669496 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b940ceb40 0x7f9b9406bd60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c0097c0 tx=0x7f9b8c01c3e0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.282+0000 7f9b98e00640 1 -- 172.21.15.47:0/2889669496 shutdown_connections 2024-02-19T15:24:21.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.282+0000 7f9b98e00640 1 --2- 172.21.15.47:0/2889669496 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b940ceb40 0x7f9b9406bd60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.282+0000 7f9b98e00640 1 --2- 172.21.15.47:0/2889669496 >> v2:172.21.15.116:3300/0 conn(0x7f9b940cded0 0x7f9b940ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.282+0000 7f9b98e00640 1 -- 172.21.15.47:0/2889669496 >> 172.21.15.47:0/2889669496 conn(0x7f9b940d1bf0 msgr2=0x7f9b940d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:21.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.282+0000 7f9b98e00640 1 -- 172.21.15.47:0/2889669496 shutdown_connections 2024-02-19T15:24:21.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.282+0000 7f9b98e00640 1 -- 172.21.15.47:0/2889669496 wait complete. 2024-02-19T15:24:21.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.286+0000 7f9b98e00640 1 Processor -- start 2024-02-19T15:24:21.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.286+0000 7f9b98e00640 1 -- start start 2024-02-19T15:24:21.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.286+0000 7f9b98e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b9406d650 0x7f9b940cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:21.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.286+0000 7f9b98e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b940cded0 0x7f9b94160730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:21.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.286+0000 7f9b98e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b940ceb40 0x7f9b940c5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:21.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.286+0000 7f9b98e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b940c6f80 con 0x7f9b940cded0 2024-02-19T15:24:21.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.286+0000 7f9b98e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b940c9a40 con 0x7f9b9406d650 2024-02-19T15:24:21.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.286+0000 7f9b98e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b940c91a0 con 0x7f9b940ceb40 2024-02-19T15:24:21.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.286+0000 7f9b91e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b9406d650 0x7f9b940cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:21.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.286+0000 7f9b89400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b940cded0 0x7f9b94160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:21.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.286+0000 7f9b91e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b9406d650 0x7f9b940cba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:45972/0 (socket says 172.21.15.47:45972) 2024-02-19T15:24:21.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.286+0000 7f9b91e00640 1 -- 172.21.15.47:0/2733751041 learned_addr learned my addr 172.21.15.47:0/2733751041 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:21.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.286+0000 7f9b92800640 1 --2- 172.21.15.47:0/2733751041 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b940ceb40 0x7f9b940c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:21.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.286+0000 7f9b91e00640 1 -- 172.21.15.47:0/2733751041 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b940ceb40 msgr2=0x7f9b940c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:21.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.286+0000 7f9b91e00640 1 --2- 172.21.15.47:0/2733751041 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b940ceb40 0x7f9b940c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.286+0000 7f9b91e00640 1 -- 172.21.15.47:0/2733751041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b940cded0 msgr2=0x7f9b94160730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:21.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.286+0000 7f9b91e00640 1 --2- 172.21.15.47:0/2733751041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b940cded0 0x7f9b94160730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.286+0000 7f9b91e00640 1 -- 172.21.15.47:0/2733751041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c009420 con 0x7f9b9406d650 2024-02-19T15:24:21.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.286+0000 7f9b89400640 1 --2- 172.21.15.47:0/2733751041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b940cded0 0x7f9b94160730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:24:21.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.286+0000 7f9b91e00640 1 --2- 172.21.15.47:0/2733751041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b9406d650 0x7f9b940cba80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9b8400ca70 tx=0x7f9b8400cf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:21.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.290+0000 7f9b8aa00640 1 -- 172.21.15.47:0/2733751041 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8400abd0 con 0x7f9b9406d650 2024-02-19T15:24:21.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.290+0000 7f9b8aa00640 1 -- 172.21.15.47:0/2733751041 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8400ad30 con 0x7f9b9406d650 2024-02-19T15:24:21.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.290+0000 7f9b8aa00640 1 -- 172.21.15.47:0/2733751041 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b84012620 con 0x7f9b9406d650 2024-02-19T15:24:21.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.290+0000 7f9b98e00640 1 -- 172.21.15.47:0/2733751041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b940c9480 con 0x7f9b9406d650 2024-02-19T15:24:21.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.290+0000 7f9b98e00640 1 -- 172.21.15.47:0/2733751041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b940cbfc0 con 0x7f9b9406d650 2024-02-19T15:24:21.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.290+0000 7f9b98e00640 1 -- 172.21.15.47:0/2733751041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b940c6de0 con 0x7f9b9406d650 2024-02-19T15:24:21.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.290+0000 7f9b8aa00640 1 -- 172.21.15.47:0/2733751041 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9b84007070 con 0x7f9b9406d650 2024-02-19T15:24:21.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.290+0000 7f9b8aa00640 1 --2- 172.21.15.47:0/2733751041 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9b70066e40 0x7f9b70069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:21.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.290+0000 7f9b8aa00640 1 -- 172.21.15.47:0/2733751041 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9b84084810 con 0x7f9b9406d650 2024-02-19T15:24:21.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.290+0000 7f9b89400640 1 --2- 172.21.15.47:0/2733751041 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9b70066e40 0x7f9b70069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:21.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.294+0000 7f9b89400640 1 --2- 172.21.15.47:0/2733751041 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9b70066e40 0x7f9b70069300 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9b940b3420 tx=0x7f9b7c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:21.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.294+0000 7f9b8aa00640 1 -- 172.21.15.47:0/2733751041 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f9b84051a70 con 0x7f9b9406d650 2024-02-19T15:24:21.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.510+0000 7f9b98e00640 1 -- 172.21.15.47:0/2733751041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9b940c6de0 con 0x7f9b9406d650 2024-02-19T15:24:21.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.510+0000 7f9b8aa00640 1 -- 172.21.15.47:0/2733751041 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f9b940c6de0 con 0x7f9b9406d650 2024-02-19T15:24:21.515 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:21.515 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:24:21.515 INFO:teuthology.orchestra.run.smithi047.stdout: "auth_dump": [ 2024-02-19T15:24:21.515 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:24:21.515 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.a", 2024-02-19T15:24:21.515 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNl8BP+HRAAlJ3Jk+1k2XcZAnICzy++Ow==", 2024-02-19T15:24:21.515 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:24:21.515 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:24:21.515 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:24:21.515 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:24:21.515 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:24:21.515 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:21.516 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:24:21.516 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:24:21.516 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.b", 2024-02-19T15:24:21.516 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlIOh3KhAAlnU2ZFY4iv/XEw0WqxyZpw==", 2024-02-19T15:24:21.516 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:24:21.516 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:24:21.516 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:24:21.516 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:24:21.516 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:24:21.516 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:21.516 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:24:21.516 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:24:21.516 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.c", 2024-02-19T15:24:21.516 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNl48SHJRAAn4GCT5KnP+2PD8IXbGClCg==", 2024-02-19T15:24:21.517 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:24:21.517 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:24:21.517 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:24:21.517 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:24:21.517 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:24:21.517 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:21.517 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:24:21.517 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:24:21.517 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.d", 2024-02-19T15:24:21.517 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNljcekMRAA414655SzH43fBA0AtjpxZw==", 2024-02-19T15:24:21.517 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:24:21.517 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:24:21.517 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:24:21.517 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:24:21.517 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:24:21.518 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:21.518 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:24:21.518 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:24:21.518 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.0", 2024-02-19T15:24:21.518 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDlcdNl/bycFhAAmzYi/OxEPUW3tcRJC0DNEQ==", 2024-02-19T15:24:21.518 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:24:21.518 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:24:21.518 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:24:21.518 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:24:21.518 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:21.518 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:24:21.518 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:24:21.518 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.1", 2024-02-19T15:24:21.518 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDocdNluquhHBAA88aLHCvVol7JoZB0MtoutA==", 2024-02-19T15:24:21.518 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:24:21.518 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:24:21.519 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:24:21.519 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:24:21.519 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:21.519 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:24:21.519 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:24:21.519 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.2", 2024-02-19T15:24:21.519 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDrcdNlGAnWIhAAyELvCPKjtfgF0+kX6NQFIA==", 2024-02-19T15:24:21.519 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:24:21.519 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:24:21.519 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:24:21.519 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:24:21.519 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:21.519 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:24:21.519 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:24:21.519 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.3", 2024-02-19T15:24:21.519 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDucdNlROZ6KBAAxTCDuYsw5SRIZBcclzwxhg==", 2024-02-19T15:24:21.520 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:24:21.520 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:24:21.520 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:24:21.520 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:24:21.520 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:21.520 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:24:21.520 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:24:21.520 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.4", 2024-02-19T15:24:21.520 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD0cdNluOGILBAA66R9RVXD2vdyIw+oZ+iM1g==", 2024-02-19T15:24:21.520 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:24:21.520 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:24:21.520 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:24:21.520 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:24:21.520 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:21.520 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:24:21.521 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:24:21.521 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.5", 2024-02-19T15:24:21.521 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD3cdNlHg8dMhAA2LUzfpvvHR4i8e5RBCeGpQ==", 2024-02-19T15:24:21.521 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:24:21.521 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:24:21.521 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:24:21.521 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:24:21.521 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:21.521 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:24:21.521 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:24:21.521 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.6", 2024-02-19T15:24:21.521 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD6cdNlXD1EORAAhKGIt0wFeCyoE6QkkZ3caA==", 2024-02-19T15:24:21.521 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:24:21.521 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:24:21.521 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:24:21.521 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:24:21.522 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:21.522 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:24:21.522 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:24:21.522 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.7", 2024-02-19T15:24:21.522 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD+cdNlfW65AhAAVWj3ptC8uQHPh/UXcb/uDg==", 2024-02-19T15:24:21.522 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:24:21.522 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:24:21.522 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:24:21.522 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:24:21.522 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:21.522 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:24:21.522 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:24:21.522 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.0", 2024-02-19T15:24:21.522 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlb7eRNRAAk6ZLGfjhcdlR9GoXFCd9eA==", 2024-02-19T15:24:21.522 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:24:21.523 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:24:21.523 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow r", 2024-02-19T15:24:21.523 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-19T15:24:21.523 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:21.523 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:24:21.523 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:24:21.523 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.1", 2024-02-19T15:24:21.523 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlKBu+AxAAlLRU70SAs3ZJo5wFwRWa4w==", 2024-02-19T15:24:21.523 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:24:21.523 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:24:21.523 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:24:21.523 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:24:21.523 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:24:21.523 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:21.523 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:24:21.523 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:24:21.523 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.2", 2024-02-19T15:24:21.524 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlyDk9ABAAIAjKSd4qJO5oHhVnDc1Rgw==", 2024-02-19T15:24:21.524 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:24:21.524 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:24:21.524 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:24:21.524 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:24:21.524 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:24:21.524 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:21.524 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:24:21.524 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:24:21.524 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.3", 2024-02-19T15:24:21.524 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlUqK3CRAABmSIzoJvl0ymME6swrvnSQ==", 2024-02-19T15:24:21.524 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:24:21.524 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:24:21.524 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:24:21.525 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:24:21.525 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:24:21.525 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:21.525 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:24:21.525 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:24:21.525 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.admin", 2024-02-19T15:24:21.525 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNls3vPAhAA4x2yQTUCz4Dv9vqEWnSTyQ==", 2024-02-19T15:24:21.525 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:24:21.525 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:24:21.525 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:24:21.525 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:24:21.525 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:24:21.525 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:21.525 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:24:21.525 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:24:21.525 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mgr.x", 2024-02-19T15:24:21.526 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNliM78GRAABWBg4hMcKjV2pK5WVBjLMQ==", 2024-02-19T15:24:21.526 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:24:21.526 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:24:21.526 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile mgr", 2024-02-19T15:24:21.526 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:24:21.526 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:21.526 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:24:21.526 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:24:21.526 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mgr.y", 2024-02-19T15:24:21.526 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlwxs9FhAAWqP7YoKio/bbdx7SGiCJ9Q==", 2024-02-19T15:24:21.526 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:24:21.526 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:24:21.526 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile mgr", 2024-02-19T15:24:21.526 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:24:21.526 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:21.526 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:21.526 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-02-19T15:24:21.526 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:24:21.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.514+0000 7f9b98e00640 1 -- 172.21.15.47:0/2733751041 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9b70066e40 msgr2=0x7f9b70069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:21.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.514+0000 7f9b98e00640 1 --2- 172.21.15.47:0/2733751041 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9b70066e40 0x7f9b70069300 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9b940b3420 tx=0x7f9b7c009190 comp rx=0 tx=0).stop 2024-02-19T15:24:21.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.514+0000 7f9b98e00640 1 -- 172.21.15.47:0/2733751041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b9406d650 msgr2=0x7f9b940cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:21.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.514+0000 7f9b98e00640 1 --2- 172.21.15.47:0/2733751041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b9406d650 0x7f9b940cba80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9b8400ca70 tx=0x7f9b8400cf30 comp rx=0 tx=0).stop 2024-02-19T15:24:21.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.514+0000 7f9b98e00640 1 -- 172.21.15.47:0/2733751041 shutdown_connections 2024-02-19T15:24:21.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.514+0000 7f9b98e00640 1 --2- 172.21.15.47:0/2733751041 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9b70066e40 0x7f9b70069300 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.514+0000 7f9b98e00640 1 --2- 172.21.15.47:0/2733751041 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b940ceb40 0x7f9b940c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.514+0000 7f9b98e00640 1 --2- 172.21.15.47:0/2733751041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b940cded0 0x7f9b94160730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.514+0000 7f9b98e00640 1 --2- 172.21.15.47:0/2733751041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b9406d650 0x7f9b940cba80 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.514+0000 7f9b98e00640 1 -- 172.21.15.47:0/2733751041 >> 172.21.15.47:0/2733751041 conn(0x7f9b940d1bf0 msgr2=0x7f9b940d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:21.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.514+0000 7f9b98e00640 1 -- 172.21.15.47:0/2733751041 shutdown_connections 2024-02-19T15:24:21.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.514+0000 7f9b98e00640 1 -- 172.21.15.47:0/2733751041 wait complete. 2024-02-19T15:24:21.536 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:24:21.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.674+0000 7f9191000640 1 Processor -- start 2024-02-19T15:24:21.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.674+0000 7f9191000640 1 -- start start 2024-02-19T15:24:21.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.674+0000 7f9191000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f918c15acf0 0x7f918c0b5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:21.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.674+0000 7f9191000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f918c0b09a0 0x7f918c06aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:21.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.674+0000 7f9191000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f918c0afa40 con 0x7f918c06c440 2024-02-19T15:24:21.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.674+0000 7f9191000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f918c0b2de0 con 0x7f918c0b09a0 2024-02-19T15:24:21.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.674+0000 7f9191000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f918c0b62d0 con 0x7f918c15acf0 2024-02-19T15:24:21.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.674+0000 7f918a800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f918c0b09a0 0x7f918c06aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:21.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.674+0000 7f9183e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f918c06c440 0x7f918c0b0290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36538/0 (socket says 172.21.15.47:36538) 2024-02-19T15:24:21.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.674+0000 7f9183e00640 1 -- 172.21.15.47:0/3340841143 learned_addr learned my addr 172.21.15.47:0/3340841143 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:21.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.674+0000 7f9189e00640 1 --2- 172.21.15.47:0/3340841143 >> v2:172.21.15.116:3300/0 conn(0x7f918c15acf0 0x7f918c0b5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:21.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.674+0000 7f918a800640 1 -- 172.21.15.47:0/3340841143 >> v2:172.21.15.116:3300/0 conn(0x7f918c15acf0 msgr2=0x7f918c0b5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:21.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.674+0000 7f918a800640 1 --2- 172.21.15.47:0/3340841143 >> v2:172.21.15.116:3300/0 conn(0x7f918c15acf0 0x7f918c0b5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.674+0000 7f918a800640 1 -- 172.21.15.47:0/3340841143 >> v1:172.21.15.47:6789/0 conn(0x7f918c06c440 legacy=0x7f918c0b0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:21.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.674+0000 7f918a800640 1 -- 172.21.15.47:0/3340841143 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918c0b1d40 con 0x7f918c0b09a0 2024-02-19T15:24:21.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.674+0000 7f9189400640 1 -- 172.21.15.47:0/3340841143 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122076129 0 0) 0x7f918c0afa40 con 0x7f918c06c440 2024-02-19T15:24:21.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.678+0000 7f918a800640 1 --2- 172.21.15.47:0/3340841143 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f918c0b09a0 0x7f918c06aeb0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f918c068a60 tx=0x7f918401c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=828336715ecbe114 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:21.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.678+0000 7f9189400640 1 -- 172.21.15.47:0/3340841143 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91840a59c0 con 0x7f918c0b09a0 2024-02-19T15:24:21.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.678+0000 7f9189400640 1 -- 172.21.15.47:0/3340841143 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91840a5b20 con 0x7f918c0b09a0 2024-02-19T15:24:21.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.678+0000 7f9189400640 1 -- 172.21.15.47:0/3340841143 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91840ad440 con 0x7f918c0b09a0 2024-02-19T15:24:21.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.678+0000 7f9191000640 1 -- 172.21.15.47:0/3340841143 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f918c0b09a0 msgr2=0x7f918c06aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:21.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.678+0000 7f9191000640 1 --2- 172.21.15.47:0/3340841143 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f918c0b09a0 0x7f918c06aeb0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f918c068a60 tx=0x7f918401c900 comp rx=0 tx=0).stop 2024-02-19T15:24:21.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.678+0000 7f9191000640 1 -- 172.21.15.47:0/3340841143 shutdown_connections 2024-02-19T15:24:21.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.678+0000 7f9191000640 1 --2- 172.21.15.47:0/3340841143 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f918c0b09a0 0x7f918c06aeb0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.678+0000 7f9191000640 1 --2- 172.21.15.47:0/3340841143 >> v2:172.21.15.116:3300/0 conn(0x7f918c15acf0 0x7f918c0b5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.678+0000 7f9191000640 1 -- 172.21.15.47:0/3340841143 >> 172.21.15.47:0/3340841143 conn(0x7f918c0b4fc0 msgr2=0x7f918c0b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:21.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.678+0000 7f9191000640 1 -- 172.21.15.47:0/3340841143 shutdown_connections 2024-02-19T15:24:21.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.678+0000 7f9191000640 1 -- 172.21.15.47:0/3340841143 wait complete. 2024-02-19T15:24:21.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9191000640 1 Processor -- start 2024-02-19T15:24:21.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9191000640 1 -- start start 2024-02-19T15:24:21.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9191000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f918c06c440 0x7f918c1f6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:21.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9191000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f918c0b09a0 0x7f918c15d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:21.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9191000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f918c15acf0 0x7f918c163740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:21.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9191000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c15d8a0 con 0x7f918c06c440 2024-02-19T15:24:21.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9191000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c15db80 con 0x7f918c15acf0 2024-02-19T15:24:21.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9191000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f918c15de60 con 0x7f918c0b09a0 2024-02-19T15:24:21.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9189e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f918c06c440 0x7f918c1f6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:21.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9189e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f918c06c440 0x7f918c1f6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36310/0 (socket says 172.21.15.47:36310) 2024-02-19T15:24:21.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9189e00640 1 -- 172.21.15.47:0/3845546374 learned_addr learned my addr 172.21.15.47:0/3845546374 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:21.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f918a800640 1 --2- 172.21.15.47:0/3845546374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f918c15acf0 0x7f918c163740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9183e00640 1 --2- 172.21.15.47:0/3845546374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f918c0b09a0 0x7f918c15d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9189e00640 1 -- 172.21.15.47:0/3845546374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f918c0b09a0 msgr2=0x7f918c15d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9189e00640 1 --2- 172.21.15.47:0/3845546374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f918c0b09a0 0x7f918c15d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9189e00640 1 -- 172.21.15.47:0/3845546374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f918c15acf0 msgr2=0x7f918c163740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9189e00640 1 --2- 172.21.15.47:0/3845546374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f918c15acf0 0x7f918c163740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9189e00640 1 -- 172.21.15.47:0/3845546374 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9184009a70 con 0x7f918c06c440 2024-02-19T15:24:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9183e00640 1 --2- 172.21.15.47:0/3845546374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f918c0b09a0 0x7f918c15d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9189e00640 1 --2- 172.21.15.47:0/3845546374 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f918c06c440 0x7f918c1f6390 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f917c00e870 tx=0x7f917c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9182000640 1 -- 172.21.15.47:0/3845546374 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c009510 con 0x7f918c06c440 2024-02-19T15:24:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9182000640 1 -- 172.21.15.47:0/3845546374 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917c00eec0 con 0x7f918c06c440 2024-02-19T15:24:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9182000640 1 -- 172.21.15.47:0/3845546374 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c00cb40 con 0x7f918c06c440 2024-02-19T15:24:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9191000640 1 -- 172.21.15.47:0/3845546374 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f918c1f9fa0 con 0x7f918c06c440 2024-02-19T15:24:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.682+0000 7f9191000640 1 -- 172.21.15.47:0/3845546374 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f918c1fa4a0 con 0x7f918c06c440 2024-02-19T15:24:21.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.686+0000 7f9191000640 1 -- 172.21.15.47:0/3845546374 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f918c1fa130 con 0x7f918c06c440 2024-02-19T15:24:21.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.686+0000 7f9182000640 1 -- 172.21.15.47:0/3845546374 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f917c00ccc0 con 0x7f918c06c440 2024-02-19T15:24:21.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.686+0000 7f9182000640 1 --2- 172.21.15.47:0/3845546374 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9168066e40 0x7f9168069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:21.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.686+0000 7f9182000640 1 -- 172.21.15.47:0/3845546374 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f917c0848d0 con 0x7f918c06c440 2024-02-19T15:24:21.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.686+0000 7f9183e00640 1 --2- 172.21.15.47:0/3845546374 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9168066e40 0x7f9168069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:21.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.686+0000 7f9183e00640 1 --2- 172.21.15.47:0/3845546374 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9168066e40 0x7f9168069300 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f918c1646a0 tx=0x7f9174009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:21.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.690+0000 7f9182000640 1 -- 172.21.15.47:0/3845546374 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f917c051970 con 0x7f918c06c440 2024-02-19T15:24:21.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.854+0000 7f9191000640 1 -- 172.21.15.47:0/3845546374 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f918c067310 con 0x7f918c06c440 2024-02-19T15:24:21.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.854+0000 7f9182000640 1 -- 172.21.15.47:0/3845546374 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+12903 (secure 0 0 0) 0x7f917c017090 con 0x7f918c06c440 2024-02-19T15:24:21.857 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:21.857 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":53,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:24:21.105049+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:24:21.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.854+0000 7f9191000640 1 -- 172.21.15.47:0/3845546374 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9168066e40 msgr2=0x7f9168069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:21.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.854+0000 7f9191000640 1 --2- 172.21.15.47:0/3845546374 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9168066e40 0x7f9168069300 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f918c1646a0 tx=0x7f9174009040 comp rx=0 tx=0).stop 2024-02-19T15:24:21.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.854+0000 7f9191000640 1 -- 172.21.15.47:0/3845546374 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f918c06c440 msgr2=0x7f918c1f6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:21.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.854+0000 7f9191000640 1 --2- 172.21.15.47:0/3845546374 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f918c06c440 0x7f918c1f6390 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f917c00e870 tx=0x7f917c00ec70 comp rx=0 tx=0).stop 2024-02-19T15:24:21.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.854+0000 7f9191000640 1 -- 172.21.15.47:0/3845546374 shutdown_connections 2024-02-19T15:24:21.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.854+0000 7f9191000640 1 --2- 172.21.15.47:0/3845546374 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9168066e40 0x7f9168069300 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.854+0000 7f9191000640 1 --2- 172.21.15.47:0/3845546374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f918c15acf0 0x7f918c163740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.854+0000 7f9191000640 1 --2- 172.21.15.47:0/3845546374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f918c0b09a0 0x7f918c15d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.854+0000 7f9191000640 1 --2- 172.21.15.47:0/3845546374 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f918c06c440 0x7f918c1f6390 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:21.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.854+0000 7f9191000640 1 -- 172.21.15.47:0/3845546374 >> 172.21.15.47:0/3845546374 conn(0x7f918c0b4fc0 msgr2=0x7f918c0b5660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:21.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.854+0000 7f9191000640 1 -- 172.21.15.47:0/3845546374 shutdown_connections 2024-02-19T15:24:21.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:21.854+0000 7f9191000640 1 -- 172.21.15.47:0/3845546374 wait complete. 2024-02-19T15:24:21.878 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:24:21.878 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:24:22.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.018+0000 7f43d0400640 1 Processor -- start 2024-02-19T15:24:22.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.018+0000 7f43d0400640 1 -- start start 2024-02-19T15:24:22.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.018+0000 7f43d0400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43c815af10 0x7f43c806e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:22.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.018+0000 7f43d0400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f43c80cd880 0x7f43c80cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:22.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.018+0000 7f43d0400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43c8071c70 con 0x7f43c80ca8f0 2024-02-19T15:24:22.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.018+0000 7f43d0400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43c806e8a0 con 0x7f43c80cd880 2024-02-19T15:24:22.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.018+0000 7f43d0400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43c806cef0 con 0x7f43c815af10 2024-02-19T15:24:22.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.022+0000 7f43cd000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f43c80ca8f0 0x7f43c80cad50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36554/0 (socket says 172.21.15.47:36554) 2024-02-19T15:24:22.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.022+0000 7f43cd000640 1 -- 172.21.15.47:0/1512423007 learned_addr learned my addr 172.21.15.47:0/1512423007 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:22.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.022+0000 7f43ce400640 1 --2- 172.21.15.47:0/1512423007 >> v2:172.21.15.47:3300/0 conn(0x7f43c80cd880 0x7f43c80cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:22.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.022+0000 7f43cda00640 1 --2- 172.21.15.47:0/1512423007 >> v2:172.21.15.116:3300/0 conn(0x7f43c815af10 0x7f43c806e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:22.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.022+0000 7f43c7e00640 1 -- 172.21.15.47:0/1512423007 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194719146 0 0) 0x7f43c8071c70 con 0x7f43c80ca8f0 2024-02-19T15:24:22.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.022+0000 7f43c7e00640 1 -- 172.21.15.47:0/1512423007 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43b0003530 con 0x7f43c80ca8f0 2024-02-19T15:24:22.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.022+0000 7f43ce400640 1 -- 172.21.15.47:0/1512423007 >> v2:172.21.15.116:3300/0 conn(0x7f43c815af10 msgr2=0x7f43c806e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:22.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.022+0000 7f43ce400640 1 --2- 172.21.15.47:0/1512423007 >> v2:172.21.15.116:3300/0 conn(0x7f43c815af10 0x7f43c806e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.022+0000 7f43ce400640 1 -- 172.21.15.47:0/1512423007 >> v1:172.21.15.47:6789/0 conn(0x7f43c80ca8f0 legacy=0x7f43c80cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:22.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.022+0000 7f43ce400640 1 -- 172.21.15.47:0/1512423007 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c80ca610 con 0x7f43c80cd880 2024-02-19T15:24:22.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.022+0000 7f43ce400640 1 --2- 172.21.15.47:0/1512423007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c80cd880 0x7f43c80cdcf0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f43c8073660 tx=0x7f43c001c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b8965ee04e1012a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:22.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.022+0000 7f43c7e00640 1 -- 172.21.15.47:0/1512423007 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c00a5930 con 0x7f43c80cd880 2024-02-19T15:24:22.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.022+0000 7f43c7e00640 1 -- 172.21.15.47:0/1512423007 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c00a5a90 con 0x7f43c80cd880 2024-02-19T15:24:22.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.022+0000 7f43c7e00640 1 -- 172.21.15.47:0/1512423007 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c00a5e20 con 0x7f43c80cd880 2024-02-19T15:24:22.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43d0400640 1 -- 172.21.15.47:0/1512423007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c80cd880 msgr2=0x7f43c80cdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:22.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43d0400640 1 --2- 172.21.15.47:0/1512423007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c80cd880 0x7f43c80cdcf0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f43c8073660 tx=0x7f43c001c870 comp rx=0 tx=0).stop 2024-02-19T15:24:22.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43d0400640 1 -- 172.21.15.47:0/1512423007 shutdown_connections 2024-02-19T15:24:22.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43d0400640 1 --2- 172.21.15.47:0/1512423007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c80cd880 0x7f43c80cdcf0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43d0400640 1 --2- 172.21.15.47:0/1512423007 >> v2:172.21.15.116:3300/0 conn(0x7f43c815af10 0x7f43c806e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43d0400640 1 -- 172.21.15.47:0/1512423007 >> 172.21.15.47:0/1512423007 conn(0x7f43c80cfc20 msgr2=0x7f43c80d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:22.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43d0400640 1 -- 172.21.15.47:0/1512423007 shutdown_connections 2024-02-19T15:24:22.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43d0400640 1 -- 172.21.15.47:0/1512423007 wait complete. 2024-02-19T15:24:22.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43d0400640 1 Processor -- start 2024-02-19T15:24:22.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43d0400640 1 -- start start 2024-02-19T15:24:22.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43d0400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43c80ca8f0 0x7f43c80c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:22.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43d0400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c80cd880 0x7f43c81471b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:22.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43d0400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c815af10 0x7f43c81f9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:22.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43d0400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c81fa420 con 0x7f43c80cd880 2024-02-19T15:24:22.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43d0400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c81fa700 con 0x7f43c80ca8f0 2024-02-19T15:24:22.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43d0400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c81fa9e0 con 0x7f43c815af10 2024-02-19T15:24:22.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43cd000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c80cd880 0x7f43c81471b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:22.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43cd000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c80cd880 0x7f43c81471b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36332/0 (socket says 172.21.15.47:36332) 2024-02-19T15:24:22.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43cd000640 1 -- 172.21.15.47:0/2550689703 learned_addr learned my addr 172.21.15.47:0/2550689703 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:22.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43cda00640 1 --2- 172.21.15.47:0/2550689703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43c80ca8f0 0x7f43c80c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:22.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43ce400640 1 --2- 172.21.15.47:0/2550689703 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c815af10 0x7f43c81f9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:22.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43cd000640 1 -- 172.21.15.47:0/2550689703 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c815af10 msgr2=0x7f43c81f9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:22.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43cd000640 1 --2- 172.21.15.47:0/2550689703 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c815af10 0x7f43c81f9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43cd000640 1 -- 172.21.15.47:0/2550689703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43c80ca8f0 msgr2=0x7f43c80c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:22.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43cd000640 1 --2- 172.21.15.47:0/2550689703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43c80ca8f0 0x7f43c80c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43cd000640 1 -- 172.21.15.47:0/2550689703 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c0009a70 con 0x7f43c80cd880 2024-02-19T15:24:22.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43ce400640 1 --2- 172.21.15.47:0/2550689703 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c815af10 0x7f43c81f9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:22.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.026+0000 7f43cda00640 1 --2- 172.21.15.47:0/2550689703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43c80ca8f0 0x7f43c80c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:22.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.030+0000 7f43cd000640 1 --2- 172.21.15.47:0/2550689703 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c80cd880 0x7f43c81471b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f43b400c3c0 tx=0x7f43b400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:22.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.030+0000 7f43c5600640 1 -- 172.21.15.47:0/2550689703 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b4005cf0 con 0x7f43c80cd880 2024-02-19T15:24:22.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.030+0000 7f43c5600640 1 -- 172.21.15.47:0/2550689703 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b4005e50 con 0x7f43c80cd880 2024-02-19T15:24:22.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.030+0000 7f43c5600640 1 -- 172.21.15.47:0/2550689703 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b40072f0 con 0x7f43c80cd880 2024-02-19T15:24:22.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.030+0000 7f43d0400640 1 -- 172.21.15.47:0/2550689703 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43c81644b0 con 0x7f43c80cd880 2024-02-19T15:24:22.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.030+0000 7f43d0400640 1 -- 172.21.15.47:0/2550689703 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43c81fa060 con 0x7f43c80cd880 2024-02-19T15:24:22.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.030+0000 7f43d0400640 1 -- 172.21.15.47:0/2550689703 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43c8075360 con 0x7f43c80cd880 2024-02-19T15:24:22.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.030+0000 7f43c5600640 1 -- 172.21.15.47:0/2550689703 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f43b4007450 con 0x7f43c80cd880 2024-02-19T15:24:22.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.030+0000 7f43c5600640 1 --2- 172.21.15.47:0/2550689703 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f43a8066e40 0x7f43a8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:22.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.030+0000 7f43c5600640 1 -- 172.21.15.47:0/2550689703 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f43b4083db0 con 0x7f43c80cd880 2024-02-19T15:24:22.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.030+0000 7f43cda00640 1 --2- 172.21.15.47:0/2550689703 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f43a8066e40 0x7f43a8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:22.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.034+0000 7f43cda00640 1 --2- 172.21.15.47:0/2550689703 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f43a8066e40 0x7f43a8069300 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f43c81574b0 tx=0x7f43bc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:22.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.034+0000 7f43c5600640 1 -- 172.21.15.47:0/2550689703 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f43b4088070 con 0x7f43c80cd880 2024-02-19T15:24:22.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.198+0000 7f43d0400640 1 -- 172.21.15.47:0/2550689703 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f43c8070cc0 con 0x7f43c80cd880 2024-02-19T15:24:22.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.198+0000 7f43c5600640 1 -- 172.21.15.47:0/2550689703 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7f43b4050e50 con 0x7f43c80cd880 2024-02-19T15:24:22.203 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:24:22.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.202+0000 7f43d0400640 1 -- 172.21.15.47:0/2550689703 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f43a8066e40 msgr2=0x7f43a8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:22.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.202+0000 7f43d0400640 1 --2- 172.21.15.47:0/2550689703 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f43a8066e40 0x7f43a8069300 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f43c81574b0 tx=0x7f43bc008040 comp rx=0 tx=0).stop 2024-02-19T15:24:22.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.202+0000 7f43d0400640 1 -- 172.21.15.47:0/2550689703 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c80cd880 msgr2=0x7f43c81471b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:22.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.202+0000 7f43d0400640 1 --2- 172.21.15.47:0/2550689703 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c80cd880 0x7f43c81471b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f43b400c3c0 tx=0x7f43b400c880 comp rx=0 tx=0).stop 2024-02-19T15:24:22.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.202+0000 7f43d0400640 1 -- 172.21.15.47:0/2550689703 shutdown_connections 2024-02-19T15:24:22.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.202+0000 7f43d0400640 1 --2- 172.21.15.47:0/2550689703 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f43a8066e40 0x7f43a8069300 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.202+0000 7f43d0400640 1 --2- 172.21.15.47:0/2550689703 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c815af10 0x7f43c81f9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.202+0000 7f43d0400640 1 --2- 172.21.15.47:0/2550689703 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c80cd880 0x7f43c81471b0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.202+0000 7f43d0400640 1 --2- 172.21.15.47:0/2550689703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43c80ca8f0 0x7f43c80c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.202+0000 7f43d0400640 1 -- 172.21.15.47:0/2550689703 >> 172.21.15.47:0/2550689703 conn(0x7f43c80cfc20 msgr2=0x7f43c815a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:22.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.202+0000 7f43d0400640 1 -- 172.21.15.47:0/2550689703 shutdown_connections 2024-02-19T15:24:22.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.202+0000 7f43d0400640 1 -- 172.21.15.47:0/2550689703 wait complete. 2024-02-19T15:24:22.223 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T15:24:22.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.358+0000 7fad04e00640 1 Processor -- start 2024-02-19T15:24:22.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.358+0000 7fad04e00640 1 -- start start 2024-02-19T15:24:22.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.362+0000 7fad04e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fad0015d6e0 0x7fad00162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:22.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.362+0000 7fad04e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad000718c0 con 0x7fad00071350 2024-02-19T15:24:22.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.362+0000 7fad04e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad00071ba0 con 0x7fad0015b1a0 2024-02-19T15:24:22.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.362+0000 7fad04e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad00071e80 con 0x7fad0015d6e0 2024-02-19T15:24:22.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.362+0000 7facfe800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fad00071350 0x7fad00164c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36562/0 (socket says 172.21.15.47:36562) 2024-02-19T15:24:22.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.362+0000 7facfe800640 1 -- 172.21.15.47:0/2538624968 learned_addr learned my addr 172.21.15.47:0/2538624968 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:22.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.362+0000 7facf7e00640 1 --2- 172.21.15.47:0/2538624968 >> v2:172.21.15.116:3300/0 conn(0x7fad0015d6e0 0x7fad00162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:22.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.362+0000 7facfd400640 1 -- 172.21.15.47:0/2538624968 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263111090 0 0) 0x7fad000718c0 con 0x7fad00071350 2024-02-19T15:24:22.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.362+0000 7facfd400640 1 -- 172.21.15.47:0/2538624968 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7face4005740 con 0x7fad00071350 2024-02-19T15:24:22.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.362+0000 7facfd400640 1 -- 172.21.15.47:0/2538624968 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585886882 0 0) 0x7fad00071ba0 con 0x7fad0015b1a0 2024-02-19T15:24:22.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.362+0000 7facfd400640 1 -- 172.21.15.47:0/2538624968 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad000718c0 con 0x7fad0015b1a0 2024-02-19T15:24:22.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.362+0000 7facfd400640 1 -- 172.21.15.47:0/2538624968 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 127234408 0 0) 0x7face4005740 con 0x7fad00071350 2024-02-19T15:24:22.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.362+0000 7facf7e00640 1 -- 172.21.15.47:0/2538624968 >> v1:172.21.15.116:6789/0 conn(0x7fad0015b1a0 legacy=0x7fad0015b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:22.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.362+0000 7facf7e00640 1 -- 172.21.15.47:0/2538624968 >> v1:172.21.15.47:6789/0 conn(0x7fad00071350 legacy=0x7fad00164c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:22.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.362+0000 7facf7e00640 1 -- 172.21.15.47:0/2538624968 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad00072160 con 0x7fad0015d6e0 2024-02-19T15:24:22.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.366+0000 7facf7e00640 1 --2- 172.21.15.47:0/2538624968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad0015d6e0 0x7fad00162740 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7face80097f0 tx=0x7face801c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88035c150919b421 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:22.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.366+0000 7facfd400640 1 -- 172.21.15.47:0/2538624968 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf8001a00 con 0x7fad0015d6e0 2024-02-19T15:24:22.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.366+0000 7facfd400640 1 -- 172.21.15.47:0/2538624968 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face80a7050 con 0x7fad0015d6e0 2024-02-19T15:24:22.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.366+0000 7facfd400640 1 -- 172.21.15.47:0/2538624968 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face80ab620 con 0x7fad0015d6e0 2024-02-19T15:24:22.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.366+0000 7fad04e00640 1 -- 172.21.15.47:0/2538624968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad0015d6e0 msgr2=0x7fad00162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:22.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.366+0000 7fad04e00640 1 --2- 172.21.15.47:0/2538624968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad0015d6e0 0x7fad00162740 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7face80097f0 tx=0x7face801c3c0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.366+0000 7fad04e00640 1 -- 172.21.15.47:0/2538624968 shutdown_connections 2024-02-19T15:24:22.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.366+0000 7fad04e00640 1 --2- 172.21.15.47:0/2538624968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad0015d6e0 0x7fad00162740 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.366+0000 7fad04e00640 1 -- 172.21.15.47:0/2538624968 >> 172.21.15.47:0/2538624968 conn(0x7fad000b16a0 msgr2=0x7fad000b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:22.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.366+0000 7fad04e00640 1 -- 172.21.15.47:0/2538624968 shutdown_connections 2024-02-19T15:24:22.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.366+0000 7fad04e00640 1 -- 172.21.15.47:0/2538624968 wait complete. 2024-02-19T15:24:22.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.366+0000 7fad04e00640 1 Processor -- start 2024-02-19T15:24:22.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.366+0000 7fad04e00640 1 -- start start 2024-02-19T15:24:22.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7fad04e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad00071350 0x7fad00158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:22.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7fad04e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad0015b1a0 0x7fad00159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:22.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7fad04e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad0015d6e0 0x7fad00147fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:22.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7fad04e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad00148ff0 con 0x7fad0015b1a0 2024-02-19T15:24:22.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7fad04e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad001492d0 con 0x7fad0015d6e0 2024-02-19T15:24:22.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7fad04e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad001495b0 con 0x7fad00071350 2024-02-19T15:24:22.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7facf7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad0015b1a0 0x7fad00159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:22.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7facf7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad0015b1a0 0x7fad00159290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36338/0 (socket says 172.21.15.47:36338) 2024-02-19T15:24:22.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7facf7e00640 1 -- 172.21.15.47:0/2935149027 learned_addr learned my addr 172.21.15.47:0/2935149027 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:22.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7facfde00640 1 --2- 172.21.15.47:0/2935149027 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad00071350 0x7fad00158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:22.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7facfe800640 1 --2- 172.21.15.47:0/2935149027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad0015d6e0 0x7fad00147fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:22.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7facf7e00640 1 -- 172.21.15.47:0/2935149027 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad00071350 msgr2=0x7fad00158d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:22.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7facf7e00640 1 --2- 172.21.15.47:0/2935149027 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad00071350 0x7fad00158d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7facf7e00640 1 -- 172.21.15.47:0/2935149027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad0015d6e0 msgr2=0x7fad00147fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:22.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7facf7e00640 1 --2- 172.21.15.47:0/2935149027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad0015d6e0 0x7fad00147fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7facf7e00640 1 -- 172.21.15.47:0/2935149027 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face8009420 con 0x7fad0015b1a0 2024-02-19T15:24:22.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7facfde00640 1 --2- 172.21.15.47:0/2935149027 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad00071350 0x7fad00158d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:22.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7facf7e00640 1 --2- 172.21.15.47:0/2935149027 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad0015b1a0 0x7fad00159290 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7face80097f0 tx=0x7face8019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:22.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7facf6000640 1 -- 172.21.15.47:0/2935149027 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face80b5410 con 0x7fad0015b1a0 2024-02-19T15:24:22.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7facf6000640 1 -- 172.21.15.47:0/2935149027 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face80b5590 con 0x7fad0015b1a0 2024-02-19T15:24:22.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7facf6000640 1 -- 172.21.15.47:0/2935149027 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face80b5940 con 0x7fad0015b1a0 2024-02-19T15:24:22.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7fad04e00640 1 -- 172.21.15.47:0/2935149027 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad00149890 con 0x7fad0015b1a0 2024-02-19T15:24:22.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7fad04e00640 1 -- 172.21.15.47:0/2935149027 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad000ae230 con 0x7fad0015b1a0 2024-02-19T15:24:22.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.370+0000 7fad04e00640 1 -- 172.21.15.47:0/2935149027 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad000af380 con 0x7fad0015b1a0 2024-02-19T15:24:22.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.374+0000 7facf6000640 1 -- 172.21.15.47:0/2935149027 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7face80b2050 con 0x7fad0015b1a0 2024-02-19T15:24:22.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.374+0000 7facf6000640 1 --2- 172.21.15.47:0/2935149027 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7facdc066e70 0x7facdc069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:22.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.374+0000 7facf6000640 1 -- 172.21.15.47:0/2935149027 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4857+0+0 (secure 0 0 0) 0x7face811e680 con 0x7fad0015b1a0 2024-02-19T15:24:22.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.374+0000 7facfde00640 1 --2- 172.21.15.47:0/2935149027 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7facdc066e70 0x7facdc069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:22.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.374+0000 7facfde00640 1 --2- 172.21.15.47:0/2935149027 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7facdc066e70 0x7facdc069330 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7facf00011f0 tx=0x7facf0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:22.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.378+0000 7facf6000640 1 -- 172.21.15.47:0/2935149027 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7face80eb720 con 0x7fad0015b1a0 2024-02-19T15:24:22.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.538+0000 7fad04e00640 1 -- 172.21.15.47:0/2935149027 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fad00075360 con 0x7fad0015b1a0 2024-02-19T15:24:22.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.538+0000 7facf6000640 1 -- 172.21.15.47:0/2935149027 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v53) v1 ==== 84+0+65 (secure 0 0 0) 0x7face80b0020 con 0x7fad0015b1a0 2024-02-19T15:24:22.542 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:22.542 INFO:teuthology.orchestra.run.smithi047.stdout:[ 2024-02-19T15:24:22.542 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:24:22.542 INFO:teuthology.orchestra.run.smithi047.stdout: "poolnum": 1, 2024-02-19T15:24:22.542 INFO:teuthology.orchestra.run.smithi047.stdout: "poolname": ".mgr" 2024-02-19T15:24:22.542 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:24:22.542 INFO:teuthology.orchestra.run.smithi047.stdout:] 2024-02-19T15:24:22.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.542+0000 7fad04e00640 1 -- 172.21.15.47:0/2935149027 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7facdc066e70 msgr2=0x7facdc069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:22.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.542+0000 7fad04e00640 1 --2- 172.21.15.47:0/2935149027 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7facdc066e70 0x7facdc069330 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7facf00011f0 tx=0x7facf0009040 comp rx=0 tx=0).stop 2024-02-19T15:24:22.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.542+0000 7fad04e00640 1 -- 172.21.15.47:0/2935149027 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad0015b1a0 msgr2=0x7fad00159290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:22.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.542+0000 7fad04e00640 1 --2- 172.21.15.47:0/2935149027 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad0015b1a0 0x7fad00159290 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7face80097f0 tx=0x7face8019160 comp rx=0 tx=0).stop 2024-02-19T15:24:22.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.542+0000 7fad04e00640 1 -- 172.21.15.47:0/2935149027 shutdown_connections 2024-02-19T15:24:22.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.542+0000 7fad04e00640 1 --2- 172.21.15.47:0/2935149027 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7facdc066e70 0x7facdc069330 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.542+0000 7fad04e00640 1 --2- 172.21.15.47:0/2935149027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad0015d6e0 0x7fad00147fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.542+0000 7fad04e00640 1 --2- 172.21.15.47:0/2935149027 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad0015b1a0 0x7fad00159290 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.542+0000 7fad04e00640 1 --2- 172.21.15.47:0/2935149027 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad00071350 0x7fad00158d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.542+0000 7fad04e00640 1 -- 172.21.15.47:0/2935149027 >> 172.21.15.47:0/2935149027 conn(0x7fad000b16a0 msgr2=0x7fad0015a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:22.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.542+0000 7fad04e00640 1 -- 172.21.15.47:0/2935149027 shutdown_connections 2024-02-19T15:24:22.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.542+0000 7fad04e00640 1 -- 172.21.15.47:0/2935149027 wait complete. 2024-02-19T15:24:22.562 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T15:24:22.562 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T15:24:22.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.702+0000 7fee34a00640 1 Processor -- start 2024-02-19T15:24:22.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.706+0000 7fee34a00640 1 -- start start 2024-02-19T15:24:22.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.706+0000 7fee34a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fee3015f420 0x7fee3006fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:22.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.706+0000 7fee34a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fee3006b5c0 0x7fee300cb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:22.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.706+0000 7fee34a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee3015fbe0 con 0x7fee30070040 2024-02-19T15:24:22.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.706+0000 7fee34a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee3015fe50 con 0x7fee3015f420 2024-02-19T15:24:22.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.706+0000 7fee34a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee30160130 con 0x7fee3006b5c0 2024-02-19T15:24:22.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.706+0000 7fee2de00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fee3015f420 0x7fee3006fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:22.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.706+0000 7fee2de00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fee3015f420 0x7fee3006fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36342/0 (socket says 172.21.15.47:36342) 2024-02-19T15:24:22.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.706+0000 7fee2de00640 1 -- 172.21.15.47:0/1784676554 learned_addr learned my addr 172.21.15.47:0/1784676554 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:22.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.706+0000 7fee2de00640 1 -- 172.21.15.47:0/1784676554 >> v2:172.21.15.116:3300/0 conn(0x7fee3006b5c0 msgr2=0x7fee300cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:22.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.706+0000 7fee2de00640 1 --2- 172.21.15.47:0/1784676554 >> v2:172.21.15.116:3300/0 conn(0x7fee3006b5c0 0x7fee300cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.706+0000 7fee2de00640 1 -- 172.21.15.47:0/1784676554 >> v1:172.21.15.116:6789/0 conn(0x7fee30070040 legacy=0x7fee3006aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:22.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.706+0000 7fee2de00640 1 -- 172.21.15.47:0/1784676554 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee30160410 con 0x7fee3015f420 2024-02-19T15:24:22.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.710+0000 7fee2de00640 1 --2- 172.21.15.47:0/1784676554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee3015f420 0x7fee3006fb00 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fee3006ea80 tx=0x7fee2001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5490499ac760f525 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:22.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.710+0000 7fee2d400640 1 -- 172.21.15.47:0/1784676554 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee200ab620 con 0x7fee3015f420 2024-02-19T15:24:22.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.710+0000 7fee2d400640 1 -- 172.21.15.47:0/1784676554 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee200ab780 con 0x7fee3015f420 2024-02-19T15:24:22.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.710+0000 7fee2d400640 1 -- 172.21.15.47:0/1784676554 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee200abb10 con 0x7fee3015f420 2024-02-19T15:24:22.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.710+0000 7fee34a00640 1 -- 172.21.15.47:0/1784676554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee3015f420 msgr2=0x7fee3006fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:22.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.710+0000 7fee34a00640 1 --2- 172.21.15.47:0/1784676554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee3015f420 0x7fee3006fb00 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fee3006ea80 tx=0x7fee2001c420 comp rx=0 tx=0).stop 2024-02-19T15:24:22.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.710+0000 7fee34a00640 1 -- 172.21.15.47:0/1784676554 shutdown_connections 2024-02-19T15:24:22.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.710+0000 7fee34a00640 1 --2- 172.21.15.47:0/1784676554 >> v2:172.21.15.116:3300/0 conn(0x7fee3006b5c0 0x7fee300cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.710+0000 7fee34a00640 1 --2- 172.21.15.47:0/1784676554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee3015f420 0x7fee3006fb00 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.710+0000 7fee34a00640 1 -- 172.21.15.47:0/1784676554 >> 172.21.15.47:0/1784676554 conn(0x7fee300b2580 msgr2=0x7fee300b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:22.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.710+0000 7fee34a00640 1 -- 172.21.15.47:0/1784676554 shutdown_connections 2024-02-19T15:24:22.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.710+0000 7fee34a00640 1 -- 172.21.15.47:0/1784676554 wait complete. 2024-02-19T15:24:22.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.710+0000 7fee34a00640 1 Processor -- start 2024-02-19T15:24:22.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.710+0000 7fee34a00640 1 -- start start 2024-02-19T15:24:22.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee34a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee3006b5c0 0x7fee300aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:22.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee34a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee3015f420 0x7fee300b5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:22.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee34a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee30152740 0x7fee30152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:22.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee34a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee300ce660 con 0x7fee3006b5c0 2024-02-19T15:24:22.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee34a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee300b6340 con 0x7fee3015f420 2024-02-19T15:24:22.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee34a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee300b3560 con 0x7fee30152740 2024-02-19T15:24:22.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee2de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee3006b5c0 0x7fee300aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:22.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee2de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee3006b5c0 0x7fee300aed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36356/0 (socket says 172.21.15.47:36356) 2024-02-19T15:24:22.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee2de00640 1 -- 172.21.15.47:0/1666924975 learned_addr learned my addr 172.21.15.47:0/1666924975 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:22.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee2e800640 1 --2- 172.21.15.47:0/1666924975 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee30152740 0x7fee30152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:22.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee25400640 1 --2- 172.21.15.47:0/1666924975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee3015f420 0x7fee300b5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:22.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee2de00640 1 -- 172.21.15.47:0/1666924975 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee30152740 msgr2=0x7fee30152bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:22.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee2de00640 1 --2- 172.21.15.47:0/1666924975 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee30152740 0x7fee30152bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee2de00640 1 -- 172.21.15.47:0/1666924975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee3015f420 msgr2=0x7fee300b5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:22.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee2de00640 1 --2- 172.21.15.47:0/1666924975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee3015f420 0x7fee300b5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:22.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee2de00640 1 -- 172.21.15.47:0/1666924975 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee200094d0 con 0x7fee3006b5c0 2024-02-19T15:24:22.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee2de00640 1 --2- 172.21.15.47:0/1666924975 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee3006b5c0 0x7fee300aed70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fee2001a050 tx=0x7fee2001ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:22.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee26a00640 1 -- 172.21.15.47:0/1666924975 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee200be3f0 con 0x7fee3006b5c0 2024-02-19T15:24:22.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee26a00640 1 -- 172.21.15.47:0/1666924975 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee200be570 con 0x7fee3006b5c0 2024-02-19T15:24:22.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee26a00640 1 -- 172.21.15.47:0/1666924975 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee200be920 con 0x7fee3006b5c0 2024-02-19T15:24:22.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee34a00640 1 -- 172.21.15.47:0/1666924975 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee300b3840 con 0x7fee3006b5c0 2024-02-19T15:24:22.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee34a00640 1 -- 172.21.15.47:0/1666924975 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee300af400 con 0x7fee3006b5c0 2024-02-19T15:24:22.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee34a00640 1 -- 172.21.15.47:0/1666924975 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee300ce7a0 con 0x7fee3006b5c0 2024-02-19T15:24:22.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.714+0000 7fee26a00640 1 -- 172.21.15.47:0/1666924975 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fee200bea80 con 0x7fee3006b5c0 2024-02-19T15:24:22.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.718+0000 7fee26a00640 1 --2- 172.21.15.47:0/1666924975 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fee0c066e40 0x7fee0c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:22.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.718+0000 7fee26a00640 1 -- 172.21.15.47:0/1666924975 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fee2011f110 con 0x7fee3006b5c0 2024-02-19T15:24:22.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.718+0000 7fee25400640 1 --2- 172.21.15.47:0/1666924975 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fee0c066e40 0x7fee0c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:22.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.718+0000 7fee25400640 1 --2- 172.21.15.47:0/1666924975 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fee0c066e40 0x7fee0c069300 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fee300b2310 tx=0x7fee18009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:22.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.722+0000 7fee26a00640 1 -- 172.21.15.47:0/1666924975 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fee300ce7a0 con 0x7fee3006b5c0 2024-02-19T15:24:22.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:22.882+0000 7fee34a00640 1 -- 172.21.15.47:0/1666924975 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fee300ce7a0 con 0x7fee3006b5c0 2024-02-19T15:24:23.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.750+0000 7fee26a00640 1 -- 172.21.15.47:0/1666924975 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v54) v1 ==== 138+0+0 (secure 0 0 0) 0x7fee300ce7a0 con 0x7fee3006b5c0 2024-02-19T15:24:23.753 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_metadata' created 2024-02-19T15:24:23.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.754+0000 7fee34a00640 1 -- 172.21.15.47:0/1666924975 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fee0c066e40 msgr2=0x7fee0c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:23.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.754+0000 7fee34a00640 1 --2- 172.21.15.47:0/1666924975 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fee0c066e40 0x7fee0c069300 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fee300b2310 tx=0x7fee18009040 comp rx=0 tx=0).stop 2024-02-19T15:24:23.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.754+0000 7fee34a00640 1 -- 172.21.15.47:0/1666924975 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee3006b5c0 msgr2=0x7fee300aed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:23.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.754+0000 7fee34a00640 1 --2- 172.21.15.47:0/1666924975 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee3006b5c0 0x7fee300aed70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fee2001a050 tx=0x7fee2001ac60 comp rx=0 tx=0).stop 2024-02-19T15:24:23.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.754+0000 7fee34a00640 1 -- 172.21.15.47:0/1666924975 shutdown_connections 2024-02-19T15:24:23.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.754+0000 7fee34a00640 1 --2- 172.21.15.47:0/1666924975 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fee0c066e40 0x7fee0c069300 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:23.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.754+0000 7fee34a00640 1 --2- 172.21.15.47:0/1666924975 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee30152740 0x7fee30152bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:23.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.754+0000 7fee34a00640 1 --2- 172.21.15.47:0/1666924975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee3015f420 0x7fee300b5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:23.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.754+0000 7fee34a00640 1 --2- 172.21.15.47:0/1666924975 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee3006b5c0 0x7fee300aed70 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:23.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.754+0000 7fee34a00640 1 -- 172.21.15.47:0/1666924975 >> 172.21.15.47:0/1666924975 conn(0x7fee300b2580 msgr2=0x7fee3015cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:23.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.754+0000 7fee34a00640 1 -- 172.21.15.47:0/1666924975 shutdown_connections 2024-02-19T15:24:23.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.754+0000 7fee34a00640 1 -- 172.21.15.47:0/1666924975 wait complete. 2024-02-19T15:24:23.785 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T15:24:23.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.934+0000 7efedc800640 1 Processor -- start 2024-02-19T15:24:23.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.934+0000 7efedc800640 1 -- start start 2024-02-19T15:24:23.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.934+0000 7efedc800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efed415f190 0x7efed40d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:23.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.934+0000 7efedc800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efed415f8f0 0x7efed415fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:23.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.934+0000 7efedc800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed40d0a50 con 0x7efed40cec80 2024-02-19T15:24:23.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.934+0000 7efedc800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed406d470 con 0x7efed415f8f0 2024-02-19T15:24:23.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.934+0000 7efedc800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed40cfe80 con 0x7efed415f190 2024-02-19T15:24:23.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.934+0000 7efed9e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7efed40cec80 0x7efed406cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35924/0 (socket says 172.21.15.47:35924) 2024-02-19T15:24:23.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.934+0000 7efed9e00640 1 -- 172.21.15.47:0/4255634904 learned_addr learned my addr 172.21.15.47:0/4255634904 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:23.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.934+0000 7efeda800640 1 --2- 172.21.15.47:0/4255634904 >> v2:172.21.15.47:3300/0 conn(0x7efed415f8f0 0x7efed415fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:23.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.934+0000 7efed9400640 1 --2- 172.21.15.47:0/4255634904 >> v2:172.21.15.116:3300/0 conn(0x7efed415f190 0x7efed40d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:23.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.934+0000 7efed8a00640 1 -- 172.21.15.47:0/4255634904 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243201178 0 0) 0x7efed40d0a50 con 0x7efed40cec80 2024-02-19T15:24:23.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.934+0000 7efed8a00640 1 -- 172.21.15.47:0/4255634904 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efebc003530 con 0x7efed40cec80 2024-02-19T15:24:23.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.934+0000 7efeda800640 1 -- 172.21.15.47:0/4255634904 >> v2:172.21.15.116:3300/0 conn(0x7efed415f190 msgr2=0x7efed40d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:23.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.934+0000 7efeda800640 1 --2- 172.21.15.47:0/4255634904 >> v2:172.21.15.116:3300/0 conn(0x7efed415f190 0x7efed40d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:23.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.934+0000 7efeda800640 1 -- 172.21.15.47:0/4255634904 >> v1:172.21.15.47:6789/0 conn(0x7efed40cec80 legacy=0x7efed406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:23.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.934+0000 7efeda800640 1 -- 172.21.15.47:0/4255634904 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed40d0120 con 0x7efed415f8f0 2024-02-19T15:24:23.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.938+0000 7efeda800640 1 --2- 172.21.15.47:0/4255634904 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415f8f0 0x7efed415fd80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7efed406e1a0 tx=0x7efed001c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbd6d6f3bc1bcc66 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:23.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.938+0000 7efed8a00640 1 -- 172.21.15.47:0/4255634904 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed00a58d0 con 0x7efed415f8f0 2024-02-19T15:24:23.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.938+0000 7efed8a00640 1 -- 172.21.15.47:0/4255634904 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed00a5a30 con 0x7efed415f8f0 2024-02-19T15:24:23.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.938+0000 7efed8a00640 1 -- 172.21.15.47:0/4255634904 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed00a5dc0 con 0x7efed415f8f0 2024-02-19T15:24:23.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.938+0000 7efedc800640 1 -- 172.21.15.47:0/4255634904 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415f8f0 msgr2=0x7efed415fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:23.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.938+0000 7efedc800640 1 --2- 172.21.15.47:0/4255634904 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415f8f0 0x7efed415fd80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7efed406e1a0 tx=0x7efed001c900 comp rx=0 tx=0).stop 2024-02-19T15:24:23.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.938+0000 7efedc800640 1 -- 172.21.15.47:0/4255634904 shutdown_connections 2024-02-19T15:24:23.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.938+0000 7efedc800640 1 --2- 172.21.15.47:0/4255634904 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415f8f0 0x7efed415fd80 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:23.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.938+0000 7efedc800640 1 --2- 172.21.15.47:0/4255634904 >> v2:172.21.15.116:3300/0 conn(0x7efed415f190 0x7efed40d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:23.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.938+0000 7efedc800640 1 -- 172.21.15.47:0/4255634904 >> 172.21.15.47:0/4255634904 conn(0x7efed409aed0 msgr2=0x7efed40d1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:23.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.938+0000 7efedc800640 1 -- 172.21.15.47:0/4255634904 shutdown_connections 2024-02-19T15:24:23.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.938+0000 7efedc800640 1 -- 172.21.15.47:0/4255634904 wait complete. 2024-02-19T15:24:23.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efedc800640 1 Processor -- start 2024-02-19T15:24:23.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efedc800640 1 -- start start 2024-02-19T15:24:23.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efedc800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efed40cec80 0x7efed41516c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:23.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efedc800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415f190 0x7efed4156390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:23.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efedc800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efed415f8f0 0x7efed4151ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:23.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efedc800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed4155f00 con 0x7efed415f190 2024-02-19T15:24:23.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efedc800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed40b4bf0 con 0x7efed415f8f0 2024-02-19T15:24:23.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efedc800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed41552f0 con 0x7efed40cec80 2024-02-19T15:24:23.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efed9400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415f190 0x7efed4156390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:23.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efed9400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415f190 0x7efed4156390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50894/0 (socket says 172.21.15.47:50894) 2024-02-19T15:24:23.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efed9400640 1 -- 172.21.15.47:0/1008653217 learned_addr learned my addr 172.21.15.47:0/1008653217 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:23.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efed9e00640 1 --2- 172.21.15.47:0/1008653217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efed40cec80 0x7efed41516c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:23.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efeda800640 1 --2- 172.21.15.47:0/1008653217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efed415f8f0 0x7efed4151ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:23.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efed9400640 1 -- 172.21.15.47:0/1008653217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efed40cec80 msgr2=0x7efed41516c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:23.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efed9400640 1 --2- 172.21.15.47:0/1008653217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efed40cec80 0x7efed41516c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:23.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efed9400640 1 -- 172.21.15.47:0/1008653217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efed415f8f0 msgr2=0x7efed4151ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:23.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efed9400640 1 --2- 172.21.15.47:0/1008653217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efed415f8f0 0x7efed4151ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:23.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efed9400640 1 -- 172.21.15.47:0/1008653217 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed0009a70 con 0x7efed415f190 2024-02-19T15:24:23.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efed9400640 1 --2- 172.21.15.47:0/1008653217 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415f190 0x7efed4156390 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7efec400e930 tx=0x7efec400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:23.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efece000640 1 -- 172.21.15.47:0/1008653217 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec4009690 con 0x7efed415f190 2024-02-19T15:24:23.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efece000640 1 -- 172.21.15.47:0/1008653217 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec400ca00 con 0x7efed415f190 2024-02-19T15:24:23.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efece000640 1 -- 172.21.15.47:0/1008653217 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec400cd70 con 0x7efed415f190 2024-02-19T15:24:23.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efedc800640 1 -- 172.21.15.47:0/1008653217 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed41555d0 con 0x7efed415f190 2024-02-19T15:24:23.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.942+0000 7efedc800640 1 -- 172.21.15.47:0/1008653217 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed4155b90 con 0x7efed415f190 2024-02-19T15:24:23.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.946+0000 7efedc800640 1 -- 172.21.15.47:0/1008653217 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed41560b0 con 0x7efed415f190 2024-02-19T15:24:23.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.946+0000 7efece000640 1 -- 172.21.15.47:0/1008653217 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efec4013470 con 0x7efed415f190 2024-02-19T15:24:23.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.946+0000 7efece000640 1 --2- 172.21.15.47:0/1008653217 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efeb8066e40 0x7efeb8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:23.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.946+0000 7efece000640 1 -- 172.21.15.47:0/1008653217 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5227+0+0 (secure 0 0 0) 0x7efec4084b50 con 0x7efed415f190 2024-02-19T15:24:23.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.946+0000 7efed9e00640 1 --2- 172.21.15.47:0/1008653217 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efeb8066e40 0x7efeb8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:23.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.946+0000 7efed9e00640 1 --2- 172.21.15.47:0/1008653217 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efeb8066e40 0x7efeb8069300 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7efec8005890 tx=0x7efec8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:23.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:23.950+0000 7efece000640 1 -- 172.21.15.47:0/1008653217 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7efed41560b0 con 0x7efed415f190 2024-02-19T15:24:24.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.114+0000 7efedc800640 1 -- 172.21.15.47:0/1008653217 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7efed4156040 con 0x7efed415f190 2024-02-19T15:24:24.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.754+0000 7efece000640 1 -- 172.21.15.47:0/1008653217 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v55) v1 ==== 170+0+0 (secure 0 0 0) 0x7efec40170d0 con 0x7efed415f190 2024-02-19T15:24:24.756 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_data' created 2024-02-19T15:24:24.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.758+0000 7efedc800640 1 -- 172.21.15.47:0/1008653217 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efeb8066e40 msgr2=0x7efeb8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:24.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.758+0000 7efedc800640 1 --2- 172.21.15.47:0/1008653217 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efeb8066e40 0x7efeb8069300 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7efec8005890 tx=0x7efec8009040 comp rx=0 tx=0).stop 2024-02-19T15:24:24.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.758+0000 7efedc800640 1 -- 172.21.15.47:0/1008653217 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415f190 msgr2=0x7efed4156390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:24.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.758+0000 7efedc800640 1 --2- 172.21.15.47:0/1008653217 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415f190 0x7efed4156390 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7efec400e930 tx=0x7efec400edf0 comp rx=0 tx=0).stop 2024-02-19T15:24:24.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.758+0000 7efedc800640 1 -- 172.21.15.47:0/1008653217 shutdown_connections 2024-02-19T15:24:24.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.758+0000 7efedc800640 1 --2- 172.21.15.47:0/1008653217 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efeb8066e40 0x7efeb8069300 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:24.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.758+0000 7efedc800640 1 --2- 172.21.15.47:0/1008653217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efed415f8f0 0x7efed4151ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:24.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.758+0000 7efedc800640 1 --2- 172.21.15.47:0/1008653217 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415f190 0x7efed4156390 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:24.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.758+0000 7efedc800640 1 --2- 172.21.15.47:0/1008653217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efed40cec80 0x7efed41516c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:24.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.758+0000 7efedc800640 1 -- 172.21.15.47:0/1008653217 >> 172.21.15.47:0/1008653217 conn(0x7efed409aed0 msgr2=0x7efed4070280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:24.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.762+0000 7efedc800640 1 -- 172.21.15.47:0/1008653217 shutdown_connections 2024-02-19T15:24:24.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.762+0000 7efedc800640 1 -- 172.21.15.47:0/1008653217 wait complete. 2024-02-19T15:24:24.801 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T15:24:24.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.946+0000 7f1469e00640 1 Processor -- start 2024-02-19T15:24:24.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.946+0000 7f1469e00640 1 -- start start 2024-02-19T15:24:24.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.946+0000 7f1469e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14640ce3b0 0x7f14640ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:24.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.946+0000 7f1469e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f146415e280 0x7f14640cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:24.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.946+0000 7f1469e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14640cc620 con 0x7f14640d2650 2024-02-19T15:24:24.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.946+0000 7f1469e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14640d12e0 con 0x7f146415e280 2024-02-19T15:24:24.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.946+0000 7f1469e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14640d1940 con 0x7f14640ce3b0 2024-02-19T15:24:24.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.946+0000 7f1462a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14640ce3b0 0x7f14640ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:24.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.946+0000 7f145a000640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f14640d2650 0x7f14640ccab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:36368/0 (socket says 172.21.15.47:36368) 2024-02-19T15:24:24.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.946+0000 7f145a000640 1 -- 172.21.15.47:0/425910405 learned_addr learned my addr 172.21.15.47:0/425910405 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:24.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.946+0000 7f1463400640 1 --2- 172.21.15.47:0/425910405 >> v2:172.21.15.116:3300/0 conn(0x7f146415e280 0x7f14640cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:24.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.946+0000 7f1462000640 1 -- 172.21.15.47:0/425910405 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581833627 0 0) 0x7f14640cc620 con 0x7f14640d2650 2024-02-19T15:24:24.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.946+0000 7f1462000640 1 -- 172.21.15.47:0/425910405 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1448003530 con 0x7f14640d2650 2024-02-19T15:24:24.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.946+0000 7f1462a00640 1 -- 172.21.15.47:0/425910405 >> v2:172.21.15.116:3300/0 conn(0x7f146415e280 msgr2=0x7f14640cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:24.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.946+0000 7f1462a00640 1 --2- 172.21.15.47:0/425910405 >> v2:172.21.15.116:3300/0 conn(0x7f146415e280 0x7f14640cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:24.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.946+0000 7f1462a00640 1 -- 172.21.15.47:0/425910405 >> v1:172.21.15.116:6789/0 conn(0x7f14640d2650 legacy=0x7f14640ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:24.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.946+0000 7f1462a00640 1 -- 172.21.15.47:0/425910405 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14640cecd0 con 0x7f14640ce3b0 2024-02-19T15:24:24.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.946+0000 7f1463400640 1 --2- 172.21.15.47:0/425910405 >> v2:172.21.15.116:3300/0 conn(0x7f146415e280 0x7f14640cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:24:24.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.950+0000 7f1462a00640 1 --2- 172.21.15.47:0/425910405 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14640ce3b0 0x7f14640ce790 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f14540097c0 tx=0x7f145401c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22d5f82b8fd8a61b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:24.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.950+0000 7f1462000640 1 -- 172.21.15.47:0/425910405 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14540ab600 con 0x7f14640ce3b0 2024-02-19T15:24:24.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.950+0000 7f1462000640 1 -- 172.21.15.47:0/425910405 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14540a7050 con 0x7f14640ce3b0 2024-02-19T15:24:24.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.950+0000 7f1462000640 1 -- 172.21.15.47:0/425910405 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14540ab980 con 0x7f14640ce3b0 2024-02-19T15:24:24.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.950+0000 7f1469e00640 1 -- 172.21.15.47:0/425910405 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14640ce3b0 msgr2=0x7f14640ce790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:24.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.950+0000 7f1469e00640 1 --2- 172.21.15.47:0/425910405 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14640ce3b0 0x7f14640ce790 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f14540097c0 tx=0x7f145401c380 comp rx=0 tx=0).stop 2024-02-19T15:24:24.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.950+0000 7f1469e00640 1 -- 172.21.15.47:0/425910405 shutdown_connections 2024-02-19T15:24:24.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.950+0000 7f1469e00640 1 --2- 172.21.15.47:0/425910405 >> v2:172.21.15.116:3300/0 conn(0x7f146415e280 0x7f14640cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:24.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.950+0000 7f1469e00640 1 --2- 172.21.15.47:0/425910405 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14640ce3b0 0x7f14640ce790 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:24.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.950+0000 7f1469e00640 1 -- 172.21.15.47:0/425910405 >> 172.21.15.47:0/425910405 conn(0x7f14640d1b30 msgr2=0x7f14640d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:24.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.950+0000 7f1469e00640 1 -- 172.21.15.47:0/425910405 shutdown_connections 2024-02-19T15:24:24.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.950+0000 7f1469e00640 1 -- 172.21.15.47:0/425910405 wait complete. 2024-02-19T15:24:24.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1469e00640 1 Processor -- start 2024-02-19T15:24:24.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1469e00640 1 -- start start 2024-02-19T15:24:24.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1469e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14640ce3b0 0x7f146415c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:24.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1469e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14640d2650 0x7f14640ae760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:24.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1469e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f146415e280 0x7f146414be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:24.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1469e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14640aefb0 con 0x7f14640ce3b0 2024-02-19T15:24:24.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1469e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14640af290 con 0x7f146415e280 2024-02-19T15:24:24.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1469e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14640af570 con 0x7f14640d2650 2024-02-19T15:24:24.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1462a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14640ce3b0 0x7f146415c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:24.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1462a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14640ce3b0 0x7f146415c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50904/0 (socket says 172.21.15.47:50904) 2024-02-19T15:24:24.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1462a00640 1 -- 172.21.15.47:0/1418079502 learned_addr learned my addr 172.21.15.47:0/1418079502 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:24.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1463400640 1 --2- 172.21.15.47:0/1418079502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f146415e280 0x7f146414be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:24.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f145a000640 1 --2- 172.21.15.47:0/1418079502 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14640d2650 0x7f14640ae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:24.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1462a00640 1 -- 172.21.15.47:0/1418079502 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14640d2650 msgr2=0x7f14640ae760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:24.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1462a00640 1 --2- 172.21.15.47:0/1418079502 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14640d2650 0x7f14640ae760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:24.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1462a00640 1 -- 172.21.15.47:0/1418079502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f146415e280 msgr2=0x7f146414be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:24.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1462a00640 1 --2- 172.21.15.47:0/1418079502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f146415e280 0x7f146414be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:24.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1462a00640 1 -- 172.21.15.47:0/1418079502 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f145c009a70 con 0x7f14640ce3b0 2024-02-19T15:24:24.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1462a00640 1 --2- 172.21.15.47:0/1418079502 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14640ce3b0 0x7f146415c1a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f14540ae040 tx=0x7f1454019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:24.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f145be00640 1 -- 172.21.15.47:0/1418079502 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1454006e20 con 0x7f14640ce3b0 2024-02-19T15:24:24.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f145be00640 1 -- 172.21.15.47:0/1418079502 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14540b4430 con 0x7f14640ce3b0 2024-02-19T15:24:24.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1469e00640 1 -- 172.21.15.47:0/1418079502 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1454009420 con 0x7f14640ce3b0 2024-02-19T15:24:24.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f145be00640 1 -- 172.21.15.47:0/1418079502 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14540b4800 con 0x7f14640ce3b0 2024-02-19T15:24:24.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1469e00640 1 -- 172.21.15.47:0/1418079502 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14640afb30 con 0x7f14640ce3b0 2024-02-19T15:24:24.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.954+0000 7f1469e00640 1 -- 172.21.15.47:0/1418079502 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1464068080 con 0x7f14640ce3b0 2024-02-19T15:24:24.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.958+0000 7f145be00640 1 -- 172.21.15.47:0/1418079502 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f14540b2040 con 0x7f14640ce3b0 2024-02-19T15:24:24.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.958+0000 7f145be00640 1 --2- 172.21.15.47:0/1418079502 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1440066e40 0x7f1440069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:24.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.958+0000 7f145be00640 1 -- 172.21.15.47:0/1418079502 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f145411f1c0 con 0x7f14640ce3b0 2024-02-19T15:24:24.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.958+0000 7f145a000640 1 --2- 172.21.15.47:0/1418079502 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1440066e40 0x7f1440069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:24.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.958+0000 7f145a000640 1 --2- 172.21.15.47:0/1418079502 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1440066e40 0x7f1440069300 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f144c007fd0 tx=0x7f144c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:24.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:24.962+0000 7f145be00640 1 -- 172.21.15.47:0/1418079502 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f14540ec130 con 0x7f14640ce3b0 2024-02-19T15:24:25.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.158+0000 7f1469e00640 1 -- 172.21.15.47:0/1418079502 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f14640b7d40 con 0x7f14640ce3b0 2024-02-19T15:24:25.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.782+0000 7f145be00640 1 -- 172.21.15.47:0/1418079502 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v29) v1 ==== 172+0+0 (secure 0 0 0) 0x7f14640b7d40 con 0x7f14640ce3b0 2024-02-19T15:24:25.791 INFO:teuthology.orchestra.run.smithi047.stderr:new fs with metadata pool 8 and data pool 9 2024-02-19T15:24:25.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.794+0000 7f1469e00640 1 -- 172.21.15.47:0/1418079502 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1440066e40 msgr2=0x7f1440069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:25.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.794+0000 7f1469e00640 1 --2- 172.21.15.47:0/1418079502 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1440066e40 0x7f1440069300 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f144c007fd0 tx=0x7f144c009040 comp rx=0 tx=0).stop 2024-02-19T15:24:25.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.794+0000 7f1469e00640 1 -- 172.21.15.47:0/1418079502 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14640ce3b0 msgr2=0x7f146415c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:25.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.794+0000 7f1469e00640 1 --2- 172.21.15.47:0/1418079502 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14640ce3b0 0x7f146415c1a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f14540ae040 tx=0x7f1454019120 comp rx=0 tx=0).stop 2024-02-19T15:24:25.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.794+0000 7f1469e00640 1 -- 172.21.15.47:0/1418079502 shutdown_connections 2024-02-19T15:24:25.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.794+0000 7f1469e00640 1 --2- 172.21.15.47:0/1418079502 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1440066e40 0x7f1440069300 secure :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f144c007fd0 tx=0x7f144c009040 comp rx=0 tx=0).stop 2024-02-19T15:24:25.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.794+0000 7f1469e00640 1 --2- 172.21.15.47:0/1418079502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f146415e280 0x7f146414be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:25.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.794+0000 7f1469e00640 1 --2- 172.21.15.47:0/1418079502 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14640d2650 0x7f14640ae760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:25.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.794+0000 7f1469e00640 1 --2- 172.21.15.47:0/1418079502 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14640ce3b0 0x7f146415c1a0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:25.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.794+0000 7f1469e00640 1 -- 172.21.15.47:0/1418079502 >> 172.21.15.47:0/1418079502 conn(0x7f14640d1b30 msgr2=0x7f14640c6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:25.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.798+0000 7f1469e00640 1 -- 172.21.15.47:0/1418079502 shutdown_connections 2024-02-19T15:24:25.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.798+0000 7f1469e00640 1 -- 172.21.15.47:0/1418079502 wait complete. 2024-02-19T15:24:25.832 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:24:25.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.970+0000 7f0f56c00640 1 Processor -- start 2024-02-19T15:24:25.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.970+0000 7f0f56c00640 1 -- start start 2024-02-19T15:24:25.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.970+0000 7f0f56c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f50143a90 0x7f0f5015da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:25.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.970+0000 7f0f56c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0f5015dfb0 0x7f0f500b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:25.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.970+0000 7f0f56c00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f5015e410 con 0x7f0f500b34e0 2024-02-19T15:24:25.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.970+0000 7f0f56c00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f500b3970 con 0x7f0f5015dfb0 2024-02-19T15:24:25.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.970+0000 7f0f56c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f50070f50 con 0x7f0f50143a90 2024-02-19T15:24:25.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f54c00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0f500b34e0 0x7f0f5015a430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35930/0 (socket says 172.21.15.47:35930) 2024-02-19T15:24:25.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f54c00640 1 -- 172.21.15.47:0/214126144 learned_addr learned my addr 172.21.15.47:0/214126144 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:25.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f47e00640 1 --2- 172.21.15.47:0/214126144 >> v2:172.21.15.116:3300/0 conn(0x7f0f5015dfb0 0x7f0f500b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:25.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f4fe00640 1 --2- 172.21.15.47:0/214126144 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f50143a90 0x7f0f5015da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:25.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f4f400640 1 -- 172.21.15.47:0/214126144 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085050398 0 0) 0x7f0f5015e410 con 0x7f0f500b34e0 2024-02-19T15:24:25.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f4f400640 1 -- 172.21.15.47:0/214126144 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f34003530 con 0x7f0f500b34e0 2024-02-19T15:24:25.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f4f400640 1 -- 172.21.15.47:0/214126144 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4022840045 0 0) 0x7f0f34003530 con 0x7f0f500b34e0 2024-02-19T15:24:25.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f4f400640 1 -- 172.21.15.47:0/214126144 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f50143a90 msgr2=0x7f0f5015da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:25.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f4f400640 1 --2- 172.21.15.47:0/214126144 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f50143a90 0x7f0f5015da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:25.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f4f400640 1 -- 172.21.15.47:0/214126144 >> v2:172.21.15.116:3300/0 conn(0x7f0f5015dfb0 msgr2=0x7f0f500b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:25.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f4f400640 1 --2- 172.21.15.47:0/214126144 >> v2:172.21.15.116:3300/0 conn(0x7f0f5015dfb0 0x7f0f500b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:25.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f4f400640 1 -- 172.21.15.47:0/214126144 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5015e8c0 con 0x7f0f500b34e0 2024-02-19T15:24:25.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f4f400640 1 -- 172.21.15.47:0/214126144 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f0f48001a00 con 0x7f0f500b34e0 2024-02-19T15:24:25.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f4f400640 1 -- 172.21.15.47:0/214126144 >> v1:172.21.15.47:6789/0 conn(0x7f0f500b34e0 legacy=0x7f0f5015a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:25.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f4f400640 1 --2- 172.21.15.47:0/214126144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f34003980 0x7f0f34003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:25.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f4f400640 1 --2- 172.21.15.47:0/214126144 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f50143a90 0x7f0f340047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:25.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f4f400640 1 --2- 172.21.15.47:0/214126144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f34004d10 0x7f0f34005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:25.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f4f400640 1 -- 172.21.15.47:0/214126144 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f500b3970 con 0x7f0f34004d10 2024-02-19T15:24:25.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f4f400640 1 -- 172.21.15.47:0/214126144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f50070f50 con 0x7f0f34003980 2024-02-19T15:24:25.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f4f400640 1 -- 172.21.15.47:0/214126144 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3400dba0 con 0x7f0f50143a90 2024-02-19T15:24:25.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f47e00640 1 --2- 172.21.15.47:0/214126144 >> v2:172.21.15.116:3300/0 conn(0x7f0f5015dfb0 0x7f0f500b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:24:25.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f54c00640 1 --2- 172.21.15.47:0/214126144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f34004d10 0x7f0f34005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:25.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f4fe00640 1 --2- 172.21.15.47:0/214126144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f34003980 0x7f0f34003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:25.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f47e00640 1 --2- 172.21.15.47:0/214126144 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f50143a90 0x7f0f340047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:25.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f54c00640 1 -- 172.21.15.47:0/214126144 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f50143a90 msgr2=0x7f0f340047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:25.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f54c00640 1 --2- 172.21.15.47:0/214126144 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f50143a90 0x7f0f340047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:25.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f54c00640 1 -- 172.21.15.47:0/214126144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f34003980 msgr2=0x7f0f34003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:25.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f54c00640 1 --2- 172.21.15.47:0/214126144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f34003980 0x7f0f34003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:25.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f54c00640 1 -- 172.21.15.47:0/214126144 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f34003530 con 0x7f0f34004d10 2024-02-19T15:24:25.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.974+0000 7f0f4fe00640 1 --2- 172.21.15.47:0/214126144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f34003980 0x7f0f34003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:25.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.978+0000 7f0f54c00640 1 --2- 172.21.15.47:0/214126144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f34004d10 0x7f0f34005190 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0f4800a860 tx=0x7f0f4801fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f88b973746beeda0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:25.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.978+0000 7f0f4f400640 1 -- 172.21.15.47:0/214126144 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f480aa710 con 0x7f0f34004d10 2024-02-19T15:24:25.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.978+0000 7f0f4f400640 1 -- 172.21.15.47:0/214126144 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f480aa870 con 0x7f0f34004d10 2024-02-19T15:24:25.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.978+0000 7f0f4f400640 1 -- 172.21.15.47:0/214126144 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f480aac40 con 0x7f0f34004d10 2024-02-19T15:24:25.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.978+0000 7f0f56c00640 1 -- 172.21.15.47:0/214126144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f34004d10 msgr2=0x7f0f34005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:25.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.978+0000 7f0f56c00640 1 --2- 172.21.15.47:0/214126144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f34004d10 0x7f0f34005190 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0f4800a860 tx=0x7f0f4801fae0 comp rx=0 tx=0).stop 2024-02-19T15:24:25.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.978+0000 7f0f54c00640 1 -- 172.21.15.47:0/214126144 reap_dead start 2024-02-19T15:24:25.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.978+0000 7f0f56c00640 1 -- 172.21.15.47:0/214126144 shutdown_connections 2024-02-19T15:24:25.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.978+0000 7f0f56c00640 1 -- 172.21.15.47:0/214126144 >> 172.21.15.47:0/214126144 conn(0x7f0f500b1ca0 msgr2=0x7f0f500b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:25.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.978+0000 7f0f56c00640 1 -- 172.21.15.47:0/214126144 shutdown_connections 2024-02-19T15:24:25.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.978+0000 7f0f56c00640 1 -- 172.21.15.47:0/214126144 wait complete. 2024-02-19T15:24:25.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.978+0000 7f0f56c00640 1 Processor -- start 2024-02-19T15:24:25.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.978+0000 7f0f56c00640 1 -- start start 2024-02-19T15:24:25.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.978+0000 7f0f56c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f34004d10 0x7f0f50156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:25.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.978+0000 7f0f56c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f500cad70 0x7f0f5015c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:25.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f56c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f500cb1f0 0x7f0f5015ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:25.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f56c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f50157b40 con 0x7f0f34004d10 2024-02-19T15:24:25.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f56c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f50156560 con 0x7f0f500cb1f0 2024-02-19T15:24:25.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f56c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5013af20 con 0x7f0f500cad70 2024-02-19T15:24:25.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f4fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f34004d10 0x7f0f50156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:25.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f4fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f34004d10 0x7f0f50156020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50908/0 (socket says 172.21.15.47:50908) 2024-02-19T15:24:25.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f4fe00640 1 -- 172.21.15.47:0/2731373210 learned_addr learned my addr 172.21.15.47:0/2731373210 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:25.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f47e00640 1 --2- 172.21.15.47:0/2731373210 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f500cad70 0x7f0f5015c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:25.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f54c00640 1 --2- 172.21.15.47:0/2731373210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f500cb1f0 0x7f0f5015ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:25.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f4fe00640 1 -- 172.21.15.47:0/2731373210 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f500cad70 msgr2=0x7f0f5015c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:25.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f4fe00640 1 --2- 172.21.15.47:0/2731373210 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f500cad70 0x7f0f5015c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:25.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f4fe00640 1 -- 172.21.15.47:0/2731373210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f500cb1f0 msgr2=0x7f0f5015ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:25.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f4fe00640 1 --2- 172.21.15.47:0/2731373210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f500cb1f0 0x7f0f5015ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:25.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f4fe00640 1 -- 172.21.15.47:0/2731373210 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f50157580 con 0x7f0f34004d10 2024-02-19T15:24:25.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f47e00640 1 --2- 172.21.15.47:0/2731373210 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f500cad70 0x7f0f5015c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:25.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f54c00640 1 --2- 172.21.15.47:0/2731373210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f500cb1f0 0x7f0f5015ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:25.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f4fe00640 1 --2- 172.21.15.47:0/2731373210 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f34004d10 0x7f0f50156020 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0f40009160 tx=0x7f0f40011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:25.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f4cc00640 1 -- 172.21.15.47:0/2731373210 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f40004220 con 0x7f0f34004d10 2024-02-19T15:24:25.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f4cc00640 1 -- 172.21.15.47:0/2731373210 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f40004380 con 0x7f0f34004d10 2024-02-19T15:24:25.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f4cc00640 1 -- 172.21.15.47:0/2731373210 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f40015890 con 0x7f0f34004d10 2024-02-19T15:24:25.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f56c00640 1 -- 172.21.15.47:0/2731373210 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f50157860 con 0x7f0f34004d10 2024-02-19T15:24:25.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f56c00640 1 -- 172.21.15.47:0/2731373210 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f50139750 con 0x7f0f34004d10 2024-02-19T15:24:25.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f56c00640 1 -- 172.21.15.47:0/2731373210 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f501392d0 con 0x7f0f34004d10 2024-02-19T15:24:25.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f4cc00640 1 -- 172.21.15.47:0/2731373210 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0f400159f0 con 0x7f0f34004d10 2024-02-19T15:24:25.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.982+0000 7f0f4cc00640 1 --2- 172.21.15.47:0/2731373210 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0f2c066e70 0x7f0f2c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:25.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.986+0000 7f0f4cc00640 1 -- 172.21.15.47:0/2731373210 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0f40084f90 con 0x7f0f34004d10 2024-02-19T15:24:25.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.986+0000 7f0f47e00640 1 --2- 172.21.15.47:0/2731373210 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0f2c066e70 0x7f0f2c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:25.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.986+0000 7f0f47e00640 1 --2- 172.21.15.47:0/2731373210 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0f2c066e70 0x7f0f2c069330 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0f500b2160 tx=0x7f0f38008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:25.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:25.990+0000 7f0f4cc00640 1 -- 172.21.15.47:0/2731373210 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0f501392d0 con 0x7f0f34004d10 2024-02-19T15:24:26.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.150+0000 7f0f56c00640 1 -- 172.21.15.47:0/2731373210 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f501392d0 con 0x7f0f34004d10 2024-02-19T15:24:26.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.150+0000 7f0f4cc00640 1 -- 172.21.15.47:0/2731373210 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+16690 (secure 0 0 0) 0x7f0f501392d0 con 0x7f0f34004d10 2024-02-19T15:24:26.155 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:26.155 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":56,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:24:25.161051+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:24:22.931622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-02-19T15:24:24.163465+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:24:26.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.154+0000 7f0f56c00640 1 -- 172.21.15.47:0/2731373210 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0f2c066e70 msgr2=0x7f0f2c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:26.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.154+0000 7f0f56c00640 1 --2- 172.21.15.47:0/2731373210 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0f2c066e70 0x7f0f2c069330 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0f500b2160 tx=0x7f0f38008040 comp rx=0 tx=0).stop 2024-02-19T15:24:26.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.154+0000 7f0f56c00640 1 -- 172.21.15.47:0/2731373210 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f34004d10 msgr2=0x7f0f50156020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:26.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.154+0000 7f0f56c00640 1 --2- 172.21.15.47:0/2731373210 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f34004d10 0x7f0f50156020 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0f40009160 tx=0x7f0f40011a60 comp rx=0 tx=0).stop 2024-02-19T15:24:26.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.154+0000 7f0f56c00640 1 -- 172.21.15.47:0/2731373210 shutdown_connections 2024-02-19T15:24:26.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.154+0000 7f0f56c00640 1 --2- 172.21.15.47:0/2731373210 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0f2c066e70 0x7f0f2c069330 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:26.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.154+0000 7f0f56c00640 1 --2- 172.21.15.47:0/2731373210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f500cb1f0 0x7f0f5015ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:26.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.154+0000 7f0f56c00640 1 --2- 172.21.15.47:0/2731373210 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f500cad70 0x7f0f5015c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:26.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.154+0000 7f0f56c00640 1 --2- 172.21.15.47:0/2731373210 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f34004d10 0x7f0f50156020 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:26.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.154+0000 7f0f56c00640 1 -- 172.21.15.47:0/2731373210 >> 172.21.15.47:0/2731373210 conn(0x7f0f500b1ca0 msgr2=0x7f0f501580a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:26.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.154+0000 7f0f56c00640 1 -- 172.21.15.47:0/2731373210 shutdown_connections 2024-02-19T15:24:26.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.154+0000 7f0f56c00640 1 -- 172.21.15.47:0/2731373210 wait complete. 2024-02-19T15:24:26.176 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:24:26.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.314+0000 7ff6de400640 1 Processor -- start 2024-02-19T15:24:26.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.314+0000 7ff6de400640 1 -- start start 2024-02-19T15:24:26.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.314+0000 7ff6de400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff6d8145590 0x7ff6d80727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:26.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.314+0000 7ff6de400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d815c390 0x7ff6d815c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:26.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.314+0000 7ff6de400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d815d280 con 0x7ff6d80cea80 2024-02-19T15:24:26.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.314+0000 7ff6de400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6d8070ff0 con 0x7ff6d8145590 2024-02-19T15:24:26.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.314+0000 7ff6de400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d815cd60 con 0x7ff6d815c390 2024-02-19T15:24:26.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.314+0000 7ff6d7e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d815c390 0x7ff6d815c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:26.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.314+0000 7ff6d7e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d815c390 0x7ff6d815c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:58844/0 (socket says 172.21.15.47:58844) 2024-02-19T15:24:26.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.314+0000 7ff6cfe00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff6d80cea80 0x7ff6d806c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:36374/0 (socket says 172.21.15.47:36374) 2024-02-19T15:24:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.314+0000 7ff6d7e00640 1 -- 172.21.15.47:0/284570099 learned_addr learned my addr 172.21.15.47:0/284570099 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.314+0000 7ff6d7400640 1 --2- 172.21.15.47:0/284570099 >> v2:172.21.15.116:3300/0 conn(0x7ff6d8145590 0x7ff6d80727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.314+0000 7ff6d7e00640 1 -- 172.21.15.47:0/284570099 >> v2:172.21.15.116:3300/0 conn(0x7ff6d8145590 msgr2=0x7ff6d80727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.314+0000 7ff6d7e00640 1 --2- 172.21.15.47:0/284570099 >> v2:172.21.15.116:3300/0 conn(0x7ff6d8145590 0x7ff6d80727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.314+0000 7ff6d7e00640 1 -- 172.21.15.47:0/284570099 >> v1:172.21.15.116:6789/0 conn(0x7ff6d80cea80 legacy=0x7ff6d806c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.314+0000 7ff6d6a00640 1 -- 172.21.15.47:0/284570099 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715677134 0 0) 0x7ff6d815d280 con 0x7ff6d80cea80 2024-02-19T15:24:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.314+0000 7ff6d7e00640 1 -- 172.21.15.47:0/284570099 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d80709e0 con 0x7ff6d815c390 2024-02-19T15:24:26.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.318+0000 7ff6d7e00640 1 --2- 172.21.15.47:0/284570099 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d815c390 0x7ff6d815c820 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009e50 tx=0x7ff6d001ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd6df0e2a40eec2e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:26.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.318+0000 7ff6d6a00640 1 -- 172.21.15.47:0/284570099 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d00a57f0 con 0x7ff6d815c390 2024-02-19T15:24:26.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.318+0000 7ff6d6a00640 1 -- 172.21.15.47:0/284570099 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d00a5950 con 0x7ff6d815c390 2024-02-19T15:24:26.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.318+0000 7ff6d6a00640 1 -- 172.21.15.47:0/284570099 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d00a5ce0 con 0x7ff6d815c390 2024-02-19T15:24:26.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.318+0000 7ff6de400640 1 -- 172.21.15.47:0/284570099 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d815c390 msgr2=0x7ff6d815c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:26.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.318+0000 7ff6de400640 1 --2- 172.21.15.47:0/284570099 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d815c390 0x7ff6d815c820 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009e50 tx=0x7ff6d001ca30 comp rx=0 tx=0).stop 2024-02-19T15:24:26.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.318+0000 7ff6de400640 1 -- 172.21.15.47:0/284570099 shutdown_connections 2024-02-19T15:24:26.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.318+0000 7ff6de400640 1 --2- 172.21.15.47:0/284570099 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d815c390 0x7ff6d815c820 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:26.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.318+0000 7ff6de400640 1 --2- 172.21.15.47:0/284570099 >> v2:172.21.15.116:3300/0 conn(0x7ff6d8145590 0x7ff6d80727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:26.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.318+0000 7ff6de400640 1 -- 172.21.15.47:0/284570099 >> 172.21.15.47:0/284570099 conn(0x7ff6d80b16a0 msgr2=0x7ff6d80b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:26.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.318+0000 7ff6de400640 1 -- 172.21.15.47:0/284570099 shutdown_connections 2024-02-19T15:24:26.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.318+0000 7ff6de400640 1 -- 172.21.15.47:0/284570099 wait complete. 2024-02-19T15:24:26.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6de400640 1 Processor -- start 2024-02-19T15:24:26.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6de400640 1 -- start start 2024-02-19T15:24:26.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6de400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6d80cea80 0x7ff6d80c67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:26.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6de400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d8145590 0x7ff6d80cc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:26.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6de400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d815c390 0x7ff6d80aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:26.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6de400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d80c5700 con 0x7ff6d80cea80 2024-02-19T15:24:26.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6de400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d80c59e0 con 0x7ff6d815c390 2024-02-19T15:24:26.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6de400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d80c5cc0 con 0x7ff6d8145590 2024-02-19T15:24:26.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6d7400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6d80cea80 0x7ff6d80c67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:26.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6d7400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6d80cea80 0x7ff6d80c67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50920/0 (socket says 172.21.15.47:50920) 2024-02-19T15:24:26.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6d7400640 1 -- 172.21.15.47:0/3368034797 learned_addr learned my addr 172.21.15.47:0/3368034797 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:26.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6cfe00640 1 --2- 172.21.15.47:0/3368034797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d8145590 0x7ff6d80cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:26.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6d7e00640 1 --2- 172.21.15.47:0/3368034797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d815c390 0x7ff6d80aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:26.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6d7400640 1 -- 172.21.15.47:0/3368034797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d8145590 msgr2=0x7ff6d80cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:26.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6d7400640 1 --2- 172.21.15.47:0/3368034797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d8145590 0x7ff6d80cc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:26.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6d7400640 1 -- 172.21.15.47:0/3368034797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d815c390 msgr2=0x7ff6d80aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:26.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6d7400640 1 --2- 172.21.15.47:0/3368034797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d815c390 0x7ff6d80aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:26.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6d7400640 1 -- 172.21.15.47:0/3368034797 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d0009b30 con 0x7ff6d80cea80 2024-02-19T15:24:26.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6d7e00640 1 --2- 172.21.15.47:0/3368034797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d815c390 0x7ff6d80aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:26.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6d7400640 1 --2- 172.21.15.47:0/3368034797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6d80cea80 0x7ff6d80c67a0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800e930 tx=0x7ff6c800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:26.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6cf400640 1 -- 172.21.15.47:0/3368034797 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8009690 con 0x7ff6d80cea80 2024-02-19T15:24:26.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6cf400640 1 -- 172.21.15.47:0/3368034797 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c800ca00 con 0x7ff6d80cea80 2024-02-19T15:24:26.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6de400640 1 -- 172.21.15.47:0/3368034797 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d80c5fa0 con 0x7ff6d80cea80 2024-02-19T15:24:26.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6cf400640 1 -- 172.21.15.47:0/3368034797 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c800cd70 con 0x7ff6d80cea80 2024-02-19T15:24:26.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.322+0000 7ff6de400640 1 -- 172.21.15.47:0/3368034797 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d80cc9b0 con 0x7ff6d80cea80 2024-02-19T15:24:26.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.326+0000 7ff6de400640 1 -- 172.21.15.47:0/3368034797 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d8075360 con 0x7ff6d80cea80 2024-02-19T15:24:26.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.326+0000 7ff6cf400640 1 -- 172.21.15.47:0/3368034797 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff6c8013470 con 0x7ff6d80cea80 2024-02-19T15:24:26.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.326+0000 7ff6cf400640 1 --2- 172.21.15.47:0/3368034797 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff6b4066e40 0x7ff6b4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:26.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.326+0000 7ff6cf400640 1 -- 172.21.15.47:0/3368034797 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6c8084cd0 con 0x7ff6d80cea80 2024-02-19T15:24:26.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.326+0000 7ff6cfe00640 1 --2- 172.21.15.47:0/3368034797 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff6b4066e40 0x7ff6b4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:26.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.326+0000 7ff6cfe00640 1 --2- 172.21.15.47:0/3368034797 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff6b4066e40 0x7ff6b4069300 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80b3810 tx=0x7ff6c0007b60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:26.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.330+0000 7ff6cf400640 1 -- 172.21.15.47:0/3368034797 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff6c8051b70 con 0x7ff6d80cea80 2024-02-19T15:24:26.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.494+0000 7ff6de400640 1 -- 172.21.15.47:0/3368034797 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6d80ae050 con 0x7ff6d80cea80 2024-02-19T15:24:26.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.494+0000 7ff6cf400640 1 -- 172.21.15.47:0/3368034797 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+16690 (secure 0 0 0) 0x7ff6c8017090 con 0x7ff6d80cea80 2024-02-19T15:24:26.497 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:26.497 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":56,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:24:25.161051+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:24:22.931622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-02-19T15:24:24.163465+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:24:26.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.494+0000 7ff6de400640 1 -- 172.21.15.47:0/3368034797 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff6b4066e40 msgr2=0x7ff6b4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:26.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.494+0000 7ff6de400640 1 --2- 172.21.15.47:0/3368034797 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff6b4066e40 0x7ff6b4069300 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80b3810 tx=0x7ff6c0007b60 comp rx=0 tx=0).stop 2024-02-19T15:24:26.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.494+0000 7ff6de400640 1 -- 172.21.15.47:0/3368034797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6d80cea80 msgr2=0x7ff6d80c67a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:26.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.494+0000 7ff6de400640 1 --2- 172.21.15.47:0/3368034797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6d80cea80 0x7ff6d80c67a0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800e930 tx=0x7ff6c800edf0 comp rx=0 tx=0).stop 2024-02-19T15:24:26.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.494+0000 7ff6de400640 1 -- 172.21.15.47:0/3368034797 shutdown_connections 2024-02-19T15:24:26.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.494+0000 7ff6de400640 1 --2- 172.21.15.47:0/3368034797 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff6b4066e40 0x7ff6b4069300 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:26.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.494+0000 7ff6de400640 1 --2- 172.21.15.47:0/3368034797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d815c390 0x7ff6d80aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:26.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.494+0000 7ff6de400640 1 --2- 172.21.15.47:0/3368034797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d8145590 0x7ff6d80cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:26.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.494+0000 7ff6de400640 1 --2- 172.21.15.47:0/3368034797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6d80cea80 0x7ff6d80c67a0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:26.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.494+0000 7ff6de400640 1 -- 172.21.15.47:0/3368034797 >> 172.21.15.47:0/3368034797 conn(0x7ff6d80b16a0 msgr2=0x7ff6d80b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:26.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.498+0000 7ff6de400640 1 -- 172.21.15.47:0/3368034797 shutdown_connections 2024-02-19T15:24:26.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.498+0000 7ff6de400640 1 -- 172.21.15.47:0/3368034797 wait complete. 2024-02-19T15:24:26.518 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T15:24:26.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.658+0000 7f82dee00640 1 Processor -- start 2024-02-19T15:24:26.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.658+0000 7f82dee00640 1 -- start start 2024-02-19T15:24:26.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.658+0000 7f82dee00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82d8158bc0 0x7f82d815be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:26.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.658+0000 7f82dee00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f82d80cf310 0x7f82d80cf770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:26.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.658+0000 7f82dee00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d80cfcb0 con 0x7f82d8071a70 2024-02-19T15:24:26.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.658+0000 7f82dee00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82d815c380 con 0x7f82d80cf310 2024-02-19T15:24:26.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.658+0000 7f82dee00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d815f8f0 con 0x7f82d8158bc0 2024-02-19T15:24:26.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.658+0000 7f82cf400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f82d80cf310 0x7f82d80cf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:26.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.658+0000 7f82cf400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f82d80cf310 0x7f82d80cf770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50936/0 (socket says 172.21.15.47:50936) 2024-02-19T15:24:26.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.658+0000 7f82cf400640 1 -- 172.21.15.47:0/997586763 learned_addr learned my addr 172.21.15.47:0/997586763 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:26.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.658+0000 7f82d7e00640 1 --2- 172.21.15.47:0/997586763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82d8158bc0 0x7f82d815be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:26.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.658+0000 7f82cf400640 1 -- 172.21.15.47:0/997586763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82d8158bc0 msgr2=0x7f82d815be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:26.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.658+0000 7f82cf400640 1 --2- 172.21.15.47:0/997586763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82d8158bc0 0x7f82d815be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:26.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.658+0000 7f82cf400640 1 -- 172.21.15.47:0/997586763 >> v1:172.21.15.116:6789/0 conn(0x7f82d8071a70 legacy=0x7f82d8071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:26.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.658+0000 7f82cf400640 1 -- 172.21.15.47:0/997586763 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d806dad0 con 0x7f82d80cf310 2024-02-19T15:24:26.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.662+0000 7f82cf400640 1 --2- 172.21.15.47:0/997586763 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82d80cf310 0x7f82d80cf770 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f82c0009870 tx=0x7f82c001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=707f3c55a0a8ecda server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:26.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.662+0000 7f82d7400640 1 -- 172.21.15.47:0/997586763 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c00a6040 con 0x7f82d80cf310 2024-02-19T15:24:26.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.662+0000 7f82d7400640 1 -- 172.21.15.47:0/997586763 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82c00aa600 con 0x7f82d80cf310 2024-02-19T15:24:26.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.662+0000 7f82d7400640 1 -- 172.21.15.47:0/997586763 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c00aa990 con 0x7f82d80cf310 2024-02-19T15:24:26.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.662+0000 7f82dee00640 1 -- 172.21.15.47:0/997586763 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82d80cf310 msgr2=0x7f82d80cf770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:26.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.662+0000 7f82dee00640 1 --2- 172.21.15.47:0/997586763 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82d80cf310 0x7f82d80cf770 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f82c0009870 tx=0x7f82c001c4a0 comp rx=0 tx=0).stop 2024-02-19T15:24:26.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.662+0000 7f82dee00640 1 -- 172.21.15.47:0/997586763 shutdown_connections 2024-02-19T15:24:26.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.662+0000 7f82dee00640 1 --2- 172.21.15.47:0/997586763 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82d80cf310 0x7f82d80cf770 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:26.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.662+0000 7f82dee00640 1 --2- 172.21.15.47:0/997586763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82d8158bc0 0x7f82d815be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:26.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.662+0000 7f82dee00640 1 -- 172.21.15.47:0/997586763 >> 172.21.15.47:0/997586763 conn(0x7f82d80d0d30 msgr2=0x7f82d80d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:26.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.662+0000 7f82dee00640 1 -- 172.21.15.47:0/997586763 shutdown_connections 2024-02-19T15:24:26.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.662+0000 7f82dee00640 1 -- 172.21.15.47:0/997586763 wait complete. 2024-02-19T15:24:26.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82dee00640 1 Processor -- start 2024-02-19T15:24:26.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82dee00640 1 -- start start 2024-02-19T15:24:26.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82dee00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82d8071a70 0x7f82d80c5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82dee00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d80cf310 0x7f82d814f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82dee00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82d8158bc0 0x7f82d8150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82dee00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d80c75d0 con 0x7f82d8071a70 2024-02-19T15:24:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82dee00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d80c5c50 con 0x7f82d80cf310 2024-02-19T15:24:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82dee00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d81599e0 con 0x7f82d8158bc0 2024-02-19T15:24:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82d7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82d8071a70 0x7f82d80c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82d7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82d8071a70 0x7f82d80c5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50948/0 (socket says 172.21.15.47:50948) 2024-02-19T15:24:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82d7e00640 1 -- 172.21.15.47:0/1542988326 learned_addr learned my addr 172.21.15.47:0/1542988326 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82cf400640 1 --2- 172.21.15.47:0/1542988326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d80cf310 0x7f82d814f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82dce00640 1 --2- 172.21.15.47:0/1542988326 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82d8158bc0 0x7f82d8150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82d7e00640 1 -- 172.21.15.47:0/1542988326 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82d8158bc0 msgr2=0x7f82d8150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82d7e00640 1 --2- 172.21.15.47:0/1542988326 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82d8158bc0 0x7f82d8150d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:26.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82d7e00640 1 -- 172.21.15.47:0/1542988326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d80cf310 msgr2=0x7f82d814f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:26.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82d7e00640 1 --2- 172.21.15.47:0/1542988326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d80cf310 0x7f82d814f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:26.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82d7e00640 1 -- 172.21.15.47:0/1542988326 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c00094d0 con 0x7f82d8071a70 2024-02-19T15:24:26.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82cf400640 1 --2- 172.21.15.47:0/1542988326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d80cf310 0x7f82d814f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:26.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82d7e00640 1 --2- 172.21.15.47:0/1542988326 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82d8071a70 0x7f82d80c5710 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f82c8007e60 tx=0x7f82c800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:26.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82d4c00640 1 -- 172.21.15.47:0/1542988326 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c8014620 con 0x7f82d8071a70 2024-02-19T15:24:26.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82d4c00640 1 -- 172.21.15.47:0/1542988326 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82c800ee90 con 0x7f82d8071a70 2024-02-19T15:24:26.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82d4c00640 1 -- 172.21.15.47:0/1542988326 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c80149e0 con 0x7f82d8071a70 2024-02-19T15:24:26.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82dee00640 1 -- 172.21.15.47:0/1542988326 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d815a280 con 0x7f82d8071a70 2024-02-19T15:24:26.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.666+0000 7f82dee00640 1 -- 172.21.15.47:0/1542988326 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d8159c30 con 0x7f82d8071a70 2024-02-19T15:24:26.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.670+0000 7f82dee00640 1 -- 172.21.15.47:0/1542988326 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d8067a10 con 0x7f82d8071a70 2024-02-19T15:24:26.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.670+0000 7f82d4c00640 1 -- 172.21.15.47:0/1542988326 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f82c8014b40 con 0x7f82d8071a70 2024-02-19T15:24:26.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.670+0000 7f82d4c00640 1 --2- 172.21.15.47:0/1542988326 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f82b4066e40 0x7f82b4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:26.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.670+0000 7f82d4c00640 1 -- 172.21.15.47:0/1542988326 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82c8085360 con 0x7f82d8071a70 2024-02-19T15:24:26.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.670+0000 7f82cf400640 1 --2- 172.21.15.47:0/1542988326 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f82b4066e40 0x7f82b4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:26.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.674+0000 7f82cf400640 1 --2- 172.21.15.47:0/1542988326 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f82b4066e40 0x7f82b4069300 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f82c00ae000 tx=0x7f82c00a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:26.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.674+0000 7f82d4c00640 1 -- 172.21.15.47:0/1542988326 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f82c80521e0 con 0x7f82d8071a70 2024-02-19T15:24:26.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:26.874+0000 7f82dee00640 1 -- 172.21.15.47:0/1542988326 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f82d80cd5b0 con 0x7f82d8071a70 2024-02-19T15:24:27.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.798+0000 7f82d4c00640 1 -- 172.21.15.47:0/1542988326 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v32) v1 ==== 118+0+0 (secure 0 0 0) 0x7f82c80558a0 con 0x7f82d8071a70 2024-02-19T15:24:27.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.802+0000 7f82dee00640 1 -- 172.21.15.47:0/1542988326 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f82b4066e40 msgr2=0x7f82b4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:27.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.802+0000 7f82dee00640 1 --2- 172.21.15.47:0/1542988326 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f82b4066e40 0x7f82b4069300 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f82c00ae000 tx=0x7f82c00a7000 comp rx=0 tx=0).stop 2024-02-19T15:24:27.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.802+0000 7f82dee00640 1 -- 172.21.15.47:0/1542988326 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82d8071a70 msgr2=0x7f82d80c5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:27.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.802+0000 7f82dee00640 1 --2- 172.21.15.47:0/1542988326 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82d8071a70 0x7f82d80c5710 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f82c8007e60 tx=0x7f82c800e4b0 comp rx=0 tx=0).stop 2024-02-19T15:24:27.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.802+0000 7f82dee00640 1 -- 172.21.15.47:0/1542988326 shutdown_connections 2024-02-19T15:24:27.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.802+0000 7f82dee00640 1 --2- 172.21.15.47:0/1542988326 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f82b4066e40 0x7f82b4069300 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:27.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.802+0000 7f82dee00640 1 --2- 172.21.15.47:0/1542988326 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82d8158bc0 0x7f82d8150d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:27.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.802+0000 7f82dee00640 1 --2- 172.21.15.47:0/1542988326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d80cf310 0x7f82d814f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:27.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.802+0000 7f82dee00640 1 --2- 172.21.15.47:0/1542988326 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82d8071a70 0x7f82d80c5710 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:27.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.802+0000 7f82dee00640 1 -- 172.21.15.47:0/1542988326 >> 172.21.15.47:0/1542988326 conn(0x7f82d80d0d30 msgr2=0x7f82d806aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:27.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.802+0000 7f82dee00640 1 -- 172.21.15.47:0/1542988326 shutdown_connections 2024-02-19T15:24:27.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.802+0000 7f82dee00640 1 -- 172.21.15.47:0/1542988326 wait complete. 2024-02-19T15:24:27.824 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T15:24:27.824 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T15:24:27.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.966+0000 7f115f600640 1 Processor -- start 2024-02-19T15:24:27.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.966+0000 7f115f600640 1 -- start start 2024-02-19T15:24:27.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.966+0000 7f115f600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f11580d2650 0x7f115806c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:27.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.966+0000 7f115f600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f115815e280 0x7f115806c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:27.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.966+0000 7f115f600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11580cc620 con 0x7f11580ce3b0 2024-02-19T15:24:27.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.966+0000 7f115f600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11580d12e0 con 0x7f11580d2650 2024-02-19T15:24:27.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.966+0000 7f115f600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11580d1940 con 0x7f115815e280 2024-02-19T15:24:27.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.966+0000 7f115cc00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f11580ce3b0 0x7f11580ce790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35944/0 (socket says 172.21.15.47:35944) 2024-02-19T15:24:27.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.966+0000 7f115cc00640 1 -- 172.21.15.47:0/3442878377 learned_addr learned my addr 172.21.15.47:0/3442878377 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:27.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.966+0000 7f115d600640 1 --2- 172.21.15.47:0/3442878377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f115815e280 0x7f115806c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:27.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.966+0000 7f1157e00640 1 --2- 172.21.15.47:0/3442878377 >> v2:172.21.15.116:3300/0 conn(0x7f11580d2650 0x7f115806c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:27.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.966+0000 7f1157400640 1 -- 172.21.15.47:0/3442878377 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2005526701 0 0) 0x7f11580cc620 con 0x7f11580ce3b0 2024-02-19T15:24:27.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.966+0000 7f1157400640 1 -- 172.21.15.47:0/3442878377 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1140003530 con 0x7f11580ce3b0 2024-02-19T15:24:27.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.966+0000 7f115d600640 1 -- 172.21.15.47:0/3442878377 >> v2:172.21.15.116:3300/0 conn(0x7f11580d2650 msgr2=0x7f115806c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:27.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.966+0000 7f115d600640 1 --2- 172.21.15.47:0/3442878377 >> v2:172.21.15.116:3300/0 conn(0x7f11580d2650 0x7f115806c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:27.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.966+0000 7f115d600640 1 -- 172.21.15.47:0/3442878377 >> v1:172.21.15.47:6789/0 conn(0x7f11580ce3b0 legacy=0x7f11580ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:27.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.966+0000 7f115d600640 1 -- 172.21.15.47:0/3442878377 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11580d1000 con 0x7f115815e280 2024-02-19T15:24:27.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.970+0000 7f115d600640 1 --2- 172.21.15.47:0/3442878377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f115815e280 0x7f115806c710 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1150009d90 tx=0x7f115001c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcf8a1b35aa16e02 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:27.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.970+0000 7f1157400640 1 -- 172.21.15.47:0/3442878377 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11500a5820 con 0x7f115815e280 2024-02-19T15:24:27.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.970+0000 7f1157400640 1 -- 172.21.15.47:0/3442878377 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11500a5980 con 0x7f115815e280 2024-02-19T15:24:27.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.970+0000 7f1157400640 1 -- 172.21.15.47:0/3442878377 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11500a5cf0 con 0x7f115815e280 2024-02-19T15:24:27.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.970+0000 7f115f600640 1 -- 172.21.15.47:0/3442878377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f115815e280 msgr2=0x7f115806c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:27.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.970+0000 7f115f600640 1 --2- 172.21.15.47:0/3442878377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f115815e280 0x7f115806c710 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1150009d90 tx=0x7f115001c850 comp rx=0 tx=0).stop 2024-02-19T15:24:27.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.970+0000 7f115f600640 1 -- 172.21.15.47:0/3442878377 shutdown_connections 2024-02-19T15:24:27.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.970+0000 7f115f600640 1 --2- 172.21.15.47:0/3442878377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f115815e280 0x7f115806c710 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:27.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.970+0000 7f115f600640 1 --2- 172.21.15.47:0/3442878377 >> v2:172.21.15.116:3300/0 conn(0x7f11580d2650 0x7f115806c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:27.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.970+0000 7f115f600640 1 -- 172.21.15.47:0/3442878377 >> 172.21.15.47:0/3442878377 conn(0x7f11580d1b30 msgr2=0x7f11580d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:27.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.970+0000 7f115f600640 1 -- 172.21.15.47:0/3442878377 shutdown_connections 2024-02-19T15:24:27.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.970+0000 7f115f600640 1 -- 172.21.15.47:0/3442878377 wait complete. 2024-02-19T15:24:27.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.970+0000 7f115f600640 1 Processor -- start 2024-02-19T15:24:27.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f115f600640 1 -- start start 2024-02-19T15:24:27.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f115f600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11580ce3b0 0x7f115814bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:27.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f115f600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11580d2650 0x7f11580c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:27.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f115f600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f115815e280 0x7f11580c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:27.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f115f600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11580af1a0 con 0x7f115815e280 2024-02-19T15:24:27.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f115f600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115814c430 con 0x7f11580ce3b0 2024-02-19T15:24:27.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f115f600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11580cbe10 con 0x7f11580d2650 2024-02-19T15:24:27.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f115d600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f115815e280 0x7f11580c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:27.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f115d600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f115815e280 0x7f11580c7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50956/0 (socket says 172.21.15.47:50956) 2024-02-19T15:24:27.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f115d600640 1 -- 172.21.15.47:0/519074556 learned_addr learned my addr 172.21.15.47:0/519074556 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:27.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f1157e00640 1 --2- 172.21.15.47:0/519074556 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11580d2650 0x7f11580c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:27.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f115cc00640 1 --2- 172.21.15.47:0/519074556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11580ce3b0 0x7f115814bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:27.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f115d600640 1 -- 172.21.15.47:0/519074556 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11580d2650 msgr2=0x7f11580c7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:27.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f115d600640 1 --2- 172.21.15.47:0/519074556 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11580d2650 0x7f11580c7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:27.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f115d600640 1 -- 172.21.15.47:0/519074556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11580ce3b0 msgr2=0x7f115814bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:27.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f115d600640 1 --2- 172.21.15.47:0/519074556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11580ce3b0 0x7f115814bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:27.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f115d600640 1 -- 172.21.15.47:0/519074556 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1150009a70 con 0x7f115815e280 2024-02-19T15:24:27.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f115d600640 1 --2- 172.21.15.47:0/519074556 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f115815e280 0x7f11580c7f70 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f115001af00 tx=0x7f11500195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:27.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f1154c00640 1 -- 172.21.15.47:0/519074556 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11500b9650 con 0x7f115815e280 2024-02-19T15:24:27.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f1154c00640 1 -- 172.21.15.47:0/519074556 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11500b97b0 con 0x7f115815e280 2024-02-19T15:24:27.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f115f600640 1 -- 172.21.15.47:0/519074556 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11580ae800 con 0x7f115815e280 2024-02-19T15:24:27.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f1154c00640 1 -- 172.21.15.47:0/519074556 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11500b9b20 con 0x7f115815e280 2024-02-19T15:24:27.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f115f600640 1 -- 172.21.15.47:0/519074556 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11580aedc0 con 0x7f115815e280 2024-02-19T15:24:27.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.974+0000 7f115f600640 1 -- 172.21.15.47:0/519074556 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11580af2e0 con 0x7f115815e280 2024-02-19T15:24:27.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.978+0000 7f1154c00640 1 -- 172.21.15.47:0/519074556 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f11580cdcd0 con 0x7f115815e280 2024-02-19T15:24:27.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.978+0000 7f1154c00640 1 --2- 172.21.15.47:0/519074556 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1138066e40 0x7f1138069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:27.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.978+0000 7f1154c00640 1 -- 172.21.15.47:0/519074556 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f115011fd20 con 0x7f115815e280 2024-02-19T15:24:27.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.978+0000 7f115cc00640 1 --2- 172.21.15.47:0/519074556 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1138066e40 0x7f1138069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:27.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.978+0000 7f115cc00640 1 --2- 172.21.15.47:0/519074556 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1138066e40 0x7f1138069300 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f114c005890 tx=0x7f114c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:27.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:27.982+0000 7f1154c00640 1 -- 172.21.15.47:0/519074556 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f11500ecaf0 con 0x7f115815e280 2024-02-19T15:24:28.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:28.182+0000 7f115f600640 1 -- 172.21.15.47:0/519074556 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1158075360 con 0x7f115815e280 2024-02-19T15:24:28.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:28.798+0000 7f1154c00640 1 -- 172.21.15.47:0/519074556 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v33) v1 ==== 122+0+0 (secure 0 0 0) 0x7f11500f01b0 con 0x7f115815e280 2024-02-19T15:24:28.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:28.802+0000 7f115f600640 1 -- 172.21.15.47:0/519074556 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1138066e40 msgr2=0x7f1138069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:28.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:28.802+0000 7f115f600640 1 --2- 172.21.15.47:0/519074556 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1138066e40 0x7f1138069300 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f114c005890 tx=0x7f114c009040 comp rx=0 tx=0).stop 2024-02-19T15:24:28.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:28.802+0000 7f115f600640 1 -- 172.21.15.47:0/519074556 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f115815e280 msgr2=0x7f11580c7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:28.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:28.802+0000 7f115f600640 1 --2- 172.21.15.47:0/519074556 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f115815e280 0x7f11580c7f70 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f115001af00 tx=0x7f11500195f0 comp rx=0 tx=0).stop 2024-02-19T15:24:28.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:28.802+0000 7f115f600640 1 -- 172.21.15.47:0/519074556 shutdown_connections 2024-02-19T15:24:28.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:28.802+0000 7f115f600640 1 --2- 172.21.15.47:0/519074556 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1138066e40 0x7f1138069300 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:28.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:28.802+0000 7f115f600640 1 --2- 172.21.15.47:0/519074556 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f115815e280 0x7f11580c7f70 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:28.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:28.802+0000 7f115f600640 1 --2- 172.21.15.47:0/519074556 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11580d2650 0x7f11580c7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:28.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:28.802+0000 7f115f600640 1 --2- 172.21.15.47:0/519074556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11580ce3b0 0x7f115814bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:28.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:28.802+0000 7f115f600640 1 -- 172.21.15.47:0/519074556 >> 172.21.15.47:0/519074556 conn(0x7f11580d1b30 msgr2=0x7f11580c6610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:28.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:28.802+0000 7f115f600640 1 -- 172.21.15.47:0/519074556 shutdown_connections 2024-02-19T15:24:28.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:28.802+0000 7f115f600640 1 -- 172.21.15.47:0/519074556 wait complete. 2024-02-19T15:24:28.824 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T15:24:28.824 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:24:29.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.010+0000 7fedf4c00640 1 Processor -- start 2024-02-19T15:24:29.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.010+0000 7fedf4c00640 1 -- start start 2024-02-19T15:24:29.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.010+0000 7fedf4c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fedf00cedb0 0x7fedf00cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:29.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.010+0000 7fedf4c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedf00d2710 0x7fedf00d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:29.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.010+0000 7fedf4c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf00cc700 con 0x7fedf015bcd0 2024-02-19T15:24:29.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.010+0000 7fedf4c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf00cf5e0 con 0x7fedf00cedb0 2024-02-19T15:24:29.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.010+0000 7fedf4c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf00cdb90 con 0x7fedf00d2710 2024-02-19T15:24:29.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.014+0000 7fede7e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fedf00cedb0 0x7fedf00cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:29.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.014+0000 7fede7e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fedf00cedb0 0x7fedf00cfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50958/0 (socket says 172.21.15.47:50958) 2024-02-19T15:24:29.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.014+0000 7fede7e00640 1 -- 172.21.15.47:0/486354071 learned_addr learned my addr 172.21.15.47:0/486354071 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:29.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.014+0000 7fedee800640 1 --2- 172.21.15.47:0/486354071 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedf00d2710 0x7fedf00d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:29.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.014+0000 7fede7e00640 1 -- 172.21.15.47:0/486354071 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedf00d2710 msgr2=0x7fedf00d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:29.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.014+0000 7fede7e00640 1 --2- 172.21.15.47:0/486354071 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedf00d2710 0x7fedf00d0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.014+0000 7fede7e00640 1 -- 172.21.15.47:0/486354071 >> v1:172.21.15.116:6789/0 conn(0x7fedf015bcd0 legacy=0x7fedf00cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:29.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.014+0000 7fede7e00640 1 -- 172.21.15.47:0/486354071 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf00cf300 con 0x7fedf00cedb0 2024-02-19T15:24:29.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.014+0000 7fede7e00640 1 --2- 172.21.15.47:0/486354071 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedf00cedb0 0x7fedf00cfc50 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fedd80098a0 tx=0x7fedd801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e414c0b90ca21c91 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:29.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.014+0000 7feded400640 1 -- 172.21.15.47:0/486354071 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd80ab680 con 0x7fedf00cedb0 2024-02-19T15:24:29.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.014+0000 7feded400640 1 -- 172.21.15.47:0/486354071 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd80ab7e0 con 0x7fedf00cedb0 2024-02-19T15:24:29.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.014+0000 7feded400640 1 -- 172.21.15.47:0/486354071 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd80abb70 con 0x7fedf00cedb0 2024-02-19T15:24:29.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fedf4c00640 1 -- 172.21.15.47:0/486354071 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedf00cedb0 msgr2=0x7fedf00cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:29.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fedf4c00640 1 --2- 172.21.15.47:0/486354071 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedf00cedb0 0x7fedf00cfc50 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fedd80098a0 tx=0x7fedd801c480 comp rx=0 tx=0).stop 2024-02-19T15:24:29.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fedf4c00640 1 -- 172.21.15.47:0/486354071 shutdown_connections 2024-02-19T15:24:29.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fedf4c00640 1 --2- 172.21.15.47:0/486354071 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedf00d2710 0x7fedf00d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fedf4c00640 1 --2- 172.21.15.47:0/486354071 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedf00cedb0 0x7fedf00cfc50 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fedf4c00640 1 -- 172.21.15.47:0/486354071 >> 172.21.15.47:0/486354071 conn(0x7fedf00d1bf0 msgr2=0x7fedf00d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:29.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fedf4c00640 1 -- 172.21.15.47:0/486354071 shutdown_connections 2024-02-19T15:24:29.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fedf4c00640 1 -- 172.21.15.47:0/486354071 wait complete. 2024-02-19T15:24:29.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fedf4c00640 1 Processor -- start 2024-02-19T15:24:29.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fedf4c00640 1 -- start start 2024-02-19T15:24:29.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fedf4c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedf00cedb0 0x7fedf00c5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:29.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fedf4c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedf00d2710 0x7fedf01538e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:29.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fedf4c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedf015bcd0 0x7fedf00c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:29.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fedf4c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf00c57c0 con 0x7fedf00d2710 2024-02-19T15:24:29.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fedf4c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf00c7bd0 con 0x7fedf015bcd0 2024-02-19T15:24:29.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fedf4c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf00c7eb0 con 0x7fedf00cedb0 2024-02-19T15:24:29.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fede7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedf00d2710 0x7fedf01538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:29.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fede7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedf00d2710 0x7fedf01538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50966/0 (socket says 172.21.15.47:50966) 2024-02-19T15:24:29.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fede7e00640 1 -- 172.21.15.47:0/4166862509 learned_addr learned my addr 172.21.15.47:0/4166862509 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:29.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fedee800640 1 --2- 172.21.15.47:0/4166862509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedf015bcd0 0x7fedf00c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:29.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.018+0000 7fedede00640 1 --2- 172.21.15.47:0/4166862509 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedf00cedb0 0x7fedf00c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:29.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.022+0000 7fede7e00640 1 -- 172.21.15.47:0/4166862509 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedf00cedb0 msgr2=0x7fedf00c5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:29.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.022+0000 7fede7e00640 1 --2- 172.21.15.47:0/4166862509 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedf00cedb0 0x7fedf00c5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.022+0000 7fede7e00640 1 -- 172.21.15.47:0/4166862509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedf015bcd0 msgr2=0x7fedf00c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:29.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.022+0000 7fede7e00640 1 --2- 172.21.15.47:0/4166862509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedf015bcd0 0x7fedf00c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.022+0000 7fede7e00640 1 -- 172.21.15.47:0/4166862509 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd80094d0 con 0x7fedf00d2710 2024-02-19T15:24:29.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.022+0000 7fedee800640 1 --2- 172.21.15.47:0/4166862509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedf015bcd0 0x7fedf00c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:29.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.022+0000 7fede7e00640 1 --2- 172.21.15.47:0/4166862509 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedf00d2710 0x7fedf01538e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fedd80ad040 tx=0x7fedd801acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:29.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.022+0000 7fede6000640 1 -- 172.21.15.47:0/4166862509 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd80b4800 con 0x7fedf00d2710 2024-02-19T15:24:29.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.022+0000 7fede6000640 1 -- 172.21.15.47:0/4166862509 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd80b49a0 con 0x7fedf00d2710 2024-02-19T15:24:29.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.022+0000 7fedf4c00640 1 -- 172.21.15.47:0/4166862509 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf015d240 con 0x7fedf00d2710 2024-02-19T15:24:29.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.022+0000 7fede6000640 1 -- 172.21.15.47:0/4166862509 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd80b4d50 con 0x7fedf00d2710 2024-02-19T15:24:29.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.022+0000 7fedf4c00640 1 -- 172.21.15.47:0/4166862509 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf00c8470 con 0x7fedf00d2710 2024-02-19T15:24:29.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.022+0000 7fedf4c00640 1 -- 172.21.15.47:0/4166862509 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf00c7ff0 con 0x7fedf00d2710 2024-02-19T15:24:29.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.022+0000 7fede6000640 1 -- 172.21.15.47:0/4166862509 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fedd80c7390 con 0x7fedf00d2710 2024-02-19T15:24:29.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.022+0000 7fede6000640 1 --2- 172.21.15.47:0/4166862509 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fedcc066e40 0x7fedcc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:29.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.022+0000 7fede6000640 1 -- 172.21.15.47:0/4166862509 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fedd811f740 con 0x7fedf00d2710 2024-02-19T15:24:29.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.022+0000 7fedede00640 1 --2- 172.21.15.47:0/4166862509 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fedcc066e40 0x7fedcc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:29.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.026+0000 7fedede00640 1 --2- 172.21.15.47:0/4166862509 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fedcc066e40 0x7fedcc069300 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fedf00ce810 tx=0x7fede0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:29.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.026+0000 7fede6000640 1 -- 172.21.15.47:0/4166862509 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fedf00c7ff0 con 0x7fedf00d2710 2024-02-19T15:24:29.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.226+0000 7fedf4c00640 1 -- 172.21.15.47:0/4166862509 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedf00c7ff0 con 0x7fedf00d2710 2024-02-19T15:24:29.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.226+0000 7fede6000640 1 -- 172.21.15.47:0/4166862509 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4938 (secure 0 0 0) 0x7fedf00c7ff0 con 0x7fedf00d2710 2024-02-19T15:24:29.228 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:29.228 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5384,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2825837364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2825837364},{"type":"v1","addr":"172.21.15.47:6835","nonce":2825837364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2251385237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2251385237},{"type":"v1","addr":"172.21.15.116:6837","nonce":2251385237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/2438199468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2438199468},{"type":"v1","addr":"172.21.15.116:6836","nonce":2438199468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:24:25.161496+0000","modified":"2024-02-19T15:24:28.184021+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5910":{"gid":5910,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6834/61582103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":61582103},{"type":"v1","addr":"172.21.15.47:6834","nonce":61582103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-19T15:24:29.228 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 33 2024-02-19T15:24:29.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.226+0000 7fedf4c00640 1 -- 172.21.15.47:0/4166862509 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fedcc066e40 msgr2=0x7fedcc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:29.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.226+0000 7fedf4c00640 1 --2- 172.21.15.47:0/4166862509 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fedcc066e40 0x7fedcc069300 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fedf00ce810 tx=0x7fede0009040 comp rx=0 tx=0).stop 2024-02-19T15:24:29.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.226+0000 7fedf4c00640 1 -- 172.21.15.47:0/4166862509 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedf00d2710 msgr2=0x7fedf01538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:29.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.226+0000 7fedf4c00640 1 --2- 172.21.15.47:0/4166862509 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedf00d2710 0x7fedf01538e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fedd80ad040 tx=0x7fedd801acc0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.226+0000 7fedf4c00640 1 -- 172.21.15.47:0/4166862509 shutdown_connections 2024-02-19T15:24:29.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.226+0000 7fedf4c00640 1 --2- 172.21.15.47:0/4166862509 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fedcc066e40 0x7fedcc069300 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.226+0000 7fedf4c00640 1 --2- 172.21.15.47:0/4166862509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedf015bcd0 0x7fedf00c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.226+0000 7fedf4c00640 1 --2- 172.21.15.47:0/4166862509 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedf00d2710 0x7fedf01538e0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.226+0000 7fedf4c00640 1 --2- 172.21.15.47:0/4166862509 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedf00cedb0 0x7fedf00c5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.226+0000 7fedf4c00640 1 -- 172.21.15.47:0/4166862509 >> 172.21.15.47:0/4166862509 conn(0x7fedf00d1bf0 msgr2=0x7fedf00b4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:29.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.226+0000 7fedf4c00640 1 -- 172.21.15.47:0/4166862509 shutdown_connections 2024-02-19T15:24:29.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.226+0000 7fedf4c00640 1 -- 172.21.15.47:0/4166862509 wait complete. 2024-02-19T15:24:29.249 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:24:29.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.386+0000 7fdc27600640 1 Processor -- start 2024-02-19T15:24:29.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.386+0000 7fdc27600640 1 -- start start 2024-02-19T15:24:29.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.390+0000 7fdc27600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdc2015d6e0 0x7fdc20162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:29.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.390+0000 7fdc27600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc20071350 0x7fdc20164c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:29.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.390+0000 7fdc27600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc200718c0 con 0x7fdc2015b1a0 2024-02-19T15:24:29.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.390+0000 7fdc27600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc20071ba0 con 0x7fdc2015d6e0 2024-02-19T15:24:29.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.390+0000 7fdc27600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc20071e80 con 0x7fdc20071350 2024-02-19T15:24:29.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.390+0000 7fdc1fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdc2015d6e0 0x7fdc20162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:29.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.390+0000 7fdc1fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdc2015d6e0 0x7fdc20162740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50968/0 (socket says 172.21.15.47:50968) 2024-02-19T15:24:29.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.390+0000 7fdc1fe00640 1 -- 172.21.15.47:0/3589795905 learned_addr learned my addr 172.21.15.47:0/3589795905 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:29.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.390+0000 7fdc25600640 1 --2- 172.21.15.47:0/3589795905 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc20071350 0x7fdc20164c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:29.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.390+0000 7fdc1fe00640 1 -- 172.21.15.47:0/3589795905 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc20071350 msgr2=0x7fdc20164c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:29.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.390+0000 7fdc1fe00640 1 --2- 172.21.15.47:0/3589795905 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc20071350 0x7fdc20164c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.390+0000 7fdc1fe00640 1 -- 172.21.15.47:0/3589795905 >> v1:172.21.15.47:6789/0 conn(0x7fdc2015b1a0 legacy=0x7fdc2015b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:29.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.390+0000 7fdc1fe00640 1 -- 172.21.15.47:0/3589795905 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc20072160 con 0x7fdc2015d6e0 2024-02-19T15:24:29.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.390+0000 7fdc25600640 1 --2- 172.21.15.47:0/3589795905 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc20071350 0x7fdc20164c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:29.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.390+0000 7fdc1f400640 1 -- 172.21.15.47:0/3589795905 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382436812 0 0) 0x7fdc200718c0 con 0x7fdc2015b1a0 2024-02-19T15:24:29.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc1fe00640 1 --2- 172.21.15.47:0/3589795905 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc2015d6e0 0x7fdc20162740 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0098a0 tx=0x7fdc0c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=379b712c9dda810 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:29.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc1f400640 1 -- 172.21.15.47:0/3589795905 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c0a9620 con 0x7fdc2015d6e0 2024-02-19T15:24:29.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc1f400640 1 -- 172.21.15.47:0/3589795905 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0c0a9780 con 0x7fdc2015d6e0 2024-02-19T15:24:29.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc1f400640 1 -- 172.21.15.47:0/3589795905 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c0a9b10 con 0x7fdc2015d6e0 2024-02-19T15:24:29.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc27600640 1 -- 172.21.15.47:0/3589795905 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc2015d6e0 msgr2=0x7fdc20162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:29.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc27600640 1 --2- 172.21.15.47:0/3589795905 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc2015d6e0 0x7fdc20162740 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0098a0 tx=0x7fdc0c01c540 comp rx=0 tx=0).stop 2024-02-19T15:24:29.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc27600640 1 -- 172.21.15.47:0/3589795905 shutdown_connections 2024-02-19T15:24:29.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc27600640 1 --2- 172.21.15.47:0/3589795905 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc20071350 0x7fdc20164c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc27600640 1 --2- 172.21.15.47:0/3589795905 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc2015d6e0 0x7fdc20162740 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc27600640 1 -- 172.21.15.47:0/3589795905 >> 172.21.15.47:0/3589795905 conn(0x7fdc200b16a0 msgr2=0x7fdc200b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:29.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc27600640 1 -- 172.21.15.47:0/3589795905 shutdown_connections 2024-02-19T15:24:29.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc27600640 1 -- 172.21.15.47:0/3589795905 wait complete. 2024-02-19T15:24:29.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc27600640 1 Processor -- start 2024-02-19T15:24:29.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc27600640 1 -- start start 2024-02-19T15:24:29.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc27600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc20071350 0x7fdc20154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:29.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc27600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc2015b1a0 0x7fdc200b37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:29.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc27600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc2015d6e0 0x7fdc200b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:29.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc27600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc201562c0 con 0x7fdc2015d6e0 2024-02-19T15:24:29.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc27600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc20155640 con 0x7fdc20071350 2024-02-19T15:24:29.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc27600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc20155920 con 0x7fdc2015b1a0 2024-02-19T15:24:29.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.394+0000 7fdc25600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc2015d6e0 0x7fdc200b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:29.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.398+0000 7fdc25600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc2015d6e0 0x7fdc200b3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50982/0 (socket says 172.21.15.47:50982) 2024-02-19T15:24:29.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.398+0000 7fdc25600640 1 -- 172.21.15.47:0/687040695 learned_addr learned my addr 172.21.15.47:0/687040695 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:29.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.398+0000 7fdc1fe00640 1 --2- 172.21.15.47:0/687040695 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc2015b1a0 0x7fdc200b37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:29.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.398+0000 7fdc24c00640 1 --2- 172.21.15.47:0/687040695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc20071350 0x7fdc20154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:29.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.398+0000 7fdc25600640 1 -- 172.21.15.47:0/687040695 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc2015b1a0 msgr2=0x7fdc200b37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:29.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.398+0000 7fdc25600640 1 --2- 172.21.15.47:0/687040695 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc2015b1a0 0x7fdc200b37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.398+0000 7fdc25600640 1 -- 172.21.15.47:0/687040695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc20071350 msgr2=0x7fdc20154e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:29.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.398+0000 7fdc25600640 1 --2- 172.21.15.47:0/687040695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc20071350 0x7fdc20154e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.398+0000 7fdc25600640 1 -- 172.21.15.47:0/687040695 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0c0094d0 con 0x7fdc2015d6e0 2024-02-19T15:24:29.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.398+0000 7fdc24c00640 1 --2- 172.21.15.47:0/687040695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc20071350 0x7fdc20154e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:29.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.398+0000 7fdc25600640 1 --2- 172.21.15.47:0/687040695 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc2015d6e0 0x7fdc200b3d20 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fdc18007df0 tx=0x7fdc1800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:29.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.398+0000 7fdc1cc00640 1 -- 172.21.15.47:0/687040695 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc180039a0 con 0x7fdc2015d6e0 2024-02-19T15:24:29.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.398+0000 7fdc1cc00640 1 -- 172.21.15.47:0/687040695 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1800f040 con 0x7fdc2015d6e0 2024-02-19T15:24:29.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.398+0000 7fdc1cc00640 1 -- 172.21.15.47:0/687040695 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc18003d40 con 0x7fdc2015d6e0 2024-02-19T15:24:29.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.398+0000 7fdc27600640 1 -- 172.21.15.47:0/687040695 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc20155c00 con 0x7fdc2015d6e0 2024-02-19T15:24:29.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.398+0000 7fdc27600640 1 -- 172.21.15.47:0/687040695 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc20155350 con 0x7fdc2015d6e0 2024-02-19T15:24:29.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.398+0000 7fdc27600640 1 -- 172.21.15.47:0/687040695 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc20156400 con 0x7fdc2015d6e0 2024-02-19T15:24:29.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.398+0000 7fdc1cc00640 1 -- 172.21.15.47:0/687040695 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fdc18004030 con 0x7fdc2015d6e0 2024-02-19T15:24:29.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.402+0000 7fdc1cc00640 1 --2- 172.21.15.47:0/687040695 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdc00066e40 0x7fdc00069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:29.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.402+0000 7fdc1cc00640 1 -- 172.21.15.47:0/687040695 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc18084c80 con 0x7fdc2015d6e0 2024-02-19T15:24:29.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.402+0000 7fdc24c00640 1 --2- 172.21.15.47:0/687040695 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdc00066e40 0x7fdc00069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:29.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.402+0000 7fdc24c00640 1 --2- 172.21.15.47:0/687040695 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdc00066e40 0x7fdc00069300 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdc14000f70 tx=0x7fdc14009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:29.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.406+0000 7fdc1cc00640 1 -- 172.21.15.47:0/687040695 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fdc20156400 con 0x7fdc2015d6e0 2024-02-19T15:24:29.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.694+0000 7fdc27600640 1 -- 172.21.15.47:0/687040695 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc20156400 con 0x7fdc2015d6e0 2024-02-19T15:24:29.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.694+0000 7fdc1cc00640 1 -- 172.21.15.47:0/687040695 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16677 (secure 0 0 0) 0x7fdc20156400 con 0x7fdc2015d6e0 2024-02-19T15:24:29.699 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:29.699 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":57,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:24:26.785375+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:24:22.931622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-02-19T15:24:24.163465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:24:29.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.698+0000 7fdc27600640 1 -- 172.21.15.47:0/687040695 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdc00066e40 msgr2=0x7fdc00069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:29.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.698+0000 7fdc27600640 1 --2- 172.21.15.47:0/687040695 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdc00066e40 0x7fdc00069300 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdc14000f70 tx=0x7fdc14009040 comp rx=0 tx=0).stop 2024-02-19T15:24:29.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.698+0000 7fdc27600640 1 -- 172.21.15.47:0/687040695 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc2015d6e0 msgr2=0x7fdc200b3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:29.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.698+0000 7fdc27600640 1 --2- 172.21.15.47:0/687040695 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc2015d6e0 0x7fdc200b3d20 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fdc18007df0 tx=0x7fdc1800d4a0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.698+0000 7fdc27600640 1 -- 172.21.15.47:0/687040695 shutdown_connections 2024-02-19T15:24:29.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.698+0000 7fdc27600640 1 --2- 172.21.15.47:0/687040695 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdc00066e40 0x7fdc00069300 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.698+0000 7fdc27600640 1 --2- 172.21.15.47:0/687040695 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc2015d6e0 0x7fdc200b3d20 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.698+0000 7fdc27600640 1 --2- 172.21.15.47:0/687040695 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc2015b1a0 0x7fdc200b37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.698+0000 7fdc27600640 1 --2- 172.21.15.47:0/687040695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc20071350 0x7fdc20154e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.698+0000 7fdc27600640 1 -- 172.21.15.47:0/687040695 >> 172.21.15.47:0/687040695 conn(0x7fdc200b16a0 msgr2=0x7fdc200aed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:29.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.698+0000 7fdc27600640 1 -- 172.21.15.47:0/687040695 shutdown_connections 2024-02-19T15:24:29.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.698+0000 7fdc27600640 1 -- 172.21.15.47:0/687040695 wait complete. 2024-02-19T15:24:29.720 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T15:24:29.720 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:24:29.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.858+0000 7f2f0a600640 1 Processor -- start 2024-02-19T15:24:29.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.858+0000 7f2f0a600640 1 -- start start 2024-02-19T15:24:29.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.858+0000 7f2f0a600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2f040ceb00 0x7f2f04168e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:29.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.858+0000 7f2f0a600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f040d10a0 con 0x7f2f040b2fc0 2024-02-19T15:24:29.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.858+0000 7f2f0a600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f040b33a0 con 0x7f2f040cef90 2024-02-19T15:24:29.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.858+0000 7f2f0a600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f040d0e80 con 0x7f2f040ceb00 2024-02-19T15:24:29.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.858+0000 7f2f03400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f2f040b2fc0 0x7f2f040d0770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35966/0 (socket says 172.21.15.47:35966) 2024-02-19T15:24:29.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.858+0000 7f2f03400640 1 -- 172.21.15.47:0/2942533038 learned_addr learned my addr 172.21.15.47:0/2942533038 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:29.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f03e00640 1 --2- 172.21.15.47:0/2942533038 >> v2:172.21.15.116:3300/0 conn(0x7f2f040ceb00 0x7f2f04168e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:29.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f02a00640 1 -- 172.21.15.47:0/2942533038 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926407225 0 0) 0x7f2f040d10a0 con 0x7f2f040b2fc0 2024-02-19T15:24:29.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f02a00640 1 -- 172.21.15.47:0/2942533038 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee8003530 con 0x7f2f040b2fc0 2024-02-19T15:24:29.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f02a00640 1 -- 172.21.15.47:0/2942533038 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040842164 0 0) 0x7f2f040b33a0 con 0x7f2f040cef90 2024-02-19T15:24:29.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f02a00640 1 -- 172.21.15.47:0/2942533038 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f040d10a0 con 0x7f2f040cef90 2024-02-19T15:24:29.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f02a00640 1 -- 172.21.15.47:0/2942533038 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2125059719 0 0) 0x7f2ee8003530 con 0x7f2f040b2fc0 2024-02-19T15:24:29.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f02a00640 1 -- 172.21.15.47:0/2942533038 >> v2:172.21.15.116:3300/0 conn(0x7f2f040ceb00 msgr2=0x7f2f04168e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:29.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f02a00640 1 --2- 172.21.15.47:0/2942533038 >> v2:172.21.15.116:3300/0 conn(0x7f2f040ceb00 0x7f2f04168e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f02a00640 1 -- 172.21.15.47:0/2942533038 >> v1:172.21.15.116:6789/0 conn(0x7f2f040cef90 legacy=0x7f2f040ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:29.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f02a00640 1 -- 172.21.15.47:0/2942533038 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f040cc2f0 con 0x7f2f040b2fc0 2024-02-19T15:24:29.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f02a00640 1 -- 172.21.15.47:0/2942533038 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f2ef4001680 con 0x7f2f040b2fc0 2024-02-19T15:24:29.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f02a00640 1 -- 172.21.15.47:0/2942533038 >> v1:172.21.15.47:6789/0 conn(0x7f2f040b2fc0 legacy=0x7f2f040d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:29.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f02a00640 1 --2- 172.21.15.47:0/2942533038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2ee8003b60 0x7f2ee8003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:29.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f02a00640 1 --2- 172.21.15.47:0/2942533038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ee80047d0 0x7f2ee8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:29.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f02a00640 1 --2- 172.21.15.47:0/2942533038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ee80051e0 0x7f2ee800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:29.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f02a00640 1 -- 172.21.15.47:0/2942533038 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f040d0e80 con 0x7f2ee8003b60 2024-02-19T15:24:29.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f02a00640 1 -- 172.21.15.47:0/2942533038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee800c110 con 0x7f2ee80051e0 2024-02-19T15:24:29.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f02a00640 1 -- 172.21.15.47:0/2942533038 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee800c2d0 con 0x7f2ee80047d0 2024-02-19T15:24:29.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f03400640 1 --2- 172.21.15.47:0/2942533038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2ee8003b60 0x7f2ee8003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:29.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f03e00640 1 --2- 172.21.15.47:0/2942533038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ee80051e0 0x7f2ee800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:29.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2efaa00640 1 --2- 172.21.15.47:0/2942533038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ee80047d0 0x7f2ee8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:29.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f03400640 1 -- 172.21.15.47:0/2942533038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ee80047d0 msgr2=0x7f2ee8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:29.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f03400640 1 --2- 172.21.15.47:0/2942533038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ee80047d0 0x7f2ee8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f03400640 1 -- 172.21.15.47:0/2942533038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ee80051e0 msgr2=0x7f2ee800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:29.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f03400640 1 --2- 172.21.15.47:0/2942533038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ee80051e0 0x7f2ee800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f03400640 1 -- 172.21.15.47:0/2942533038 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee8003530 con 0x7f2ee8003b60 2024-02-19T15:24:29.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2f03e00640 1 -- 172.21.15.47:0/2942533038 reap_dead start 2024-02-19T15:24:29.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.862+0000 7f2efaa00640 1 --2- 172.21.15.47:0/2942533038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ee80047d0 0x7f2ee8004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:29.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.866+0000 7f2f03400640 1 --2- 172.21.15.47:0/2942533038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2ee8003b60 0x7f2ee8003f60 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4007fa0 tx=0x7f2ef401caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e2f6b23dd89bb34 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:29.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.866+0000 7f2f02a00640 1 -- 172.21.15.47:0/2942533038 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4005ae0 con 0x7f2ee8003b60 2024-02-19T15:24:29.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.866+0000 7f2f02a00640 1 -- 172.21.15.47:0/2942533038 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef4005c40 con 0x7f2ee8003b60 2024-02-19T15:24:29.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.866+0000 7f2f02a00640 1 -- 172.21.15.47:0/2942533038 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4007290 con 0x7f2ee8003b60 2024-02-19T15:24:29.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.866+0000 7f2f0a600640 1 -- 172.21.15.47:0/2942533038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2ee8003b60 msgr2=0x7f2ee8003f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:29.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.866+0000 7f2f0a600640 1 --2- 172.21.15.47:0/2942533038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2ee8003b60 0x7f2ee8003f60 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4007fa0 tx=0x7f2ef401caf0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.866+0000 7f2f0a600640 1 -- 172.21.15.47:0/2942533038 shutdown_connections 2024-02-19T15:24:29.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.866+0000 7f2f0a600640 1 --2- 172.21.15.47:0/2942533038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2ee8003b60 0x7f2ee8003f60 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.866+0000 7f2f0a600640 1 -- 172.21.15.47:0/2942533038 >> 172.21.15.47:0/2942533038 conn(0x7f2f040d1b70 msgr2=0x7f2f040d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:29.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.866+0000 7f2f0a600640 1 -- 172.21.15.47:0/2942533038 shutdown_connections 2024-02-19T15:24:29.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.866+0000 7f2f0a600640 1 -- 172.21.15.47:0/2942533038 wait complete. 2024-02-19T15:24:29.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.870+0000 7f2f0a600640 1 Processor -- start 2024-02-19T15:24:29.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.870+0000 7f2f0a600640 1 -- start start 2024-02-19T15:24:29.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.870+0000 7f2f0a600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2ee8003b60 0x7f2f041f8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:29.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2f0a600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f040c8fc0 0x7f2f040c7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:29.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2f0a600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f040cb090 0x7f2f040cb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:29.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2f0a600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f041f7600 con 0x7f2ee8003b60 2024-02-19T15:24:29.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2f0a600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f040cba50 con 0x7f2f040cb090 2024-02-19T15:24:29.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2f0a600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f041f86a0 con 0x7f2f040c8fc0 2024-02-19T15:24:29.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2f03400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2ee8003b60 0x7f2f041f8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:29.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2f03400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2ee8003b60 0x7f2f041f8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51002/0 (socket says 172.21.15.47:51002) 2024-02-19T15:24:29.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2f03400640 1 -- 172.21.15.47:0/1694059544 learned_addr learned my addr 172.21.15.47:0/1694059544 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:29.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2efaa00640 1 --2- 172.21.15.47:0/1694059544 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f040c8fc0 0x7f2f040c7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:29.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2f03e00640 1 --2- 172.21.15.47:0/1694059544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f040cb090 0x7f2f040cb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:29.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2f03400640 1 -- 172.21.15.47:0/1694059544 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f040c8fc0 msgr2=0x7f2f040c7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:29.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2f03400640 1 --2- 172.21.15.47:0/1694059544 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f040c8fc0 0x7f2f040c7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2f03400640 1 -- 172.21.15.47:0/1694059544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f040cb090 msgr2=0x7f2f040cb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:29.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2f03400640 1 --2- 172.21.15.47:0/1694059544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f040cb090 0x7f2f040cb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:29.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2f03400640 1 -- 172.21.15.47:0/1694059544 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f040c9540 con 0x7f2ee8003b60 2024-02-19T15:24:29.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2efaa00640 1 --2- 172.21.15.47:0/1694059544 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f040c8fc0 0x7f2f040c7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:29.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2f03400640 1 --2- 172.21.15.47:0/1694059544 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2ee8003b60 0x7f2f041f8160 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4007fa0 tx=0x7f2ee8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:29.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2efbe00640 1 -- 172.21.15.47:0/1694059544 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef40b4070 con 0x7f2ee8003b60 2024-02-19T15:24:29.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2efbe00640 1 -- 172.21.15.47:0/1694059544 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef4001540 con 0x7f2ee8003b60 2024-02-19T15:24:29.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2f0a600640 1 -- 172.21.15.47:0/1694059544 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f040c9820 con 0x7f2ee8003b60 2024-02-19T15:24:29.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2efbe00640 1 -- 172.21.15.47:0/1694059544 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4007270 con 0x7f2ee8003b60 2024-02-19T15:24:29.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2f0a600640 1 -- 172.21.15.47:0/1694059544 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f040c9de0 con 0x7f2ee8003b60 2024-02-19T15:24:29.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2f0a600640 1 -- 172.21.15.47:0/1694059544 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f04075360 con 0x7f2ee8003b60 2024-02-19T15:24:29.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.874+0000 7f2efbe00640 1 -- 172.21.15.47:0/1694059544 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2ef40073d0 con 0x7f2ee8003b60 2024-02-19T15:24:29.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.878+0000 7f2efbe00640 1 --2- 172.21.15.47:0/1694059544 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2ee0066e70 0x7f2ee0069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:29.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.878+0000 7f2efbe00640 1 -- 172.21.15.47:0/1694059544 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ef411e5c0 con 0x7f2ee8003b60 2024-02-19T15:24:29.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.878+0000 7f2efaa00640 1 --2- 172.21.15.47:0/1694059544 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2ee0066e70 0x7f2ee0069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:29.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.878+0000 7f2efaa00640 1 --2- 172.21.15.47:0/1694059544 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2ee0066e70 0x7f2ee0069330 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2f040c79a0 tx=0x7f2eec009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:29.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:29.882+0000 7f2efbe00640 1 -- 172.21.15.47:0/1694059544 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2ef40eb310 con 0x7f2ee8003b60 2024-02-19T15:24:30.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.042+0000 7f2f0a600640 1 -- 172.21.15.47:0/1694059544 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2f040703c0 con 0x7f2ee0066e70 2024-02-19T15:24:30.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.050+0000 7f2efbe00640 1 -- 172.21.15.47:0/1694059544 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288881 (secure 0 0 0) 0x7f2f040703c0 con 0x7f2ee0066e70 2024-02-19T15:24:30.056 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:30.058 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:24:30.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.054+0000 7f2f0a600640 1 -- 172.21.15.47:0/1694059544 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2ee0066e70 msgr2=0x7f2ee0069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:30.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.054+0000 7f2f0a600640 1 --2- 172.21.15.47:0/1694059544 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2ee0066e70 0x7f2ee0069330 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2f040c79a0 tx=0x7f2eec009040 comp rx=0 tx=0).stop 2024-02-19T15:24:30.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.054+0000 7f2f0a600640 1 -- 172.21.15.47:0/1694059544 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2ee8003b60 msgr2=0x7f2f041f8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:30.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.054+0000 7f2f0a600640 1 --2- 172.21.15.47:0/1694059544 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2ee8003b60 0x7f2f041f8160 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4007fa0 tx=0x7f2ee8005800 comp rx=0 tx=0).stop 2024-02-19T15:24:30.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.054+0000 7f2f0a600640 1 -- 172.21.15.47:0/1694059544 shutdown_connections 2024-02-19T15:24:30.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.054+0000 7f2f0a600640 1 --2- 172.21.15.47:0/1694059544 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2ee0066e70 0x7f2ee0069330 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.054+0000 7f2f0a600640 1 --2- 172.21.15.47:0/1694059544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f040cb090 0x7f2f040cb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.054+0000 7f2f0a600640 1 --2- 172.21.15.47:0/1694059544 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f040c8fc0 0x7f2f040c7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.054+0000 7f2f0a600640 1 --2- 172.21.15.47:0/1694059544 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2ee8003b60 0x7f2f041f8160 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.054+0000 7f2f0a600640 1 -- 172.21.15.47:0/1694059544 >> 172.21.15.47:0/1694059544 conn(0x7f2f040d1b70 msgr2=0x7f2f0406f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:30.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.054+0000 7f2f0a600640 1 -- 172.21.15.47:0/1694059544 shutdown_connections 2024-02-19T15:24:30.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.054+0000 7f2f0a600640 1 -- 172.21.15.47:0/1694059544 wait complete. 2024-02-19T15:24:30.085 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-02-19T15:24:29.180006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":103,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255800,"kb_used_data":6332,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718920,"statfs":{"total":773094113280,"available":772832174080,"internally_reserved":0,"allocated":6483968,"data_stored":3348641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.422556"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.797928+0000","last_change":"2024-02-19T15:24:25.793151+0000","last_active":"2024-02-19T15:24:25.797928+0000","last_peered":"2024-02-19T15:24:25.797928+0000","last_clean":"2024-02-19T15:24:25.797928+0000","last_became_active":"2024-02-19T15:24:25.791062+0000","last_became_peered":"2024-02-19T15:24:25.791062+0000","last_unstale":"2024-02-19T15:24:25.797928+0000","last_undegraded":"2024-02-19T15:24:25.797928+0000","last_fullsized":"2024-02-19T15:24:25.797928+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:39:55.119850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.813902+0000","last_change":"2024-02-19T15:24:25.811654+0000","last_active":"2024-02-19T15:24:25.813902+0000","last_peered":"2024-02-19T15:24:25.813902+0000","last_clean":"2024-02-19T15:24:25.813902+0000","last_became_active":"2024-02-19T15:24:25.807942+0000","last_became_peered":"2024-02-19T15:24:25.807942+0000","last_unstale":"2024-02-19T15:24:25.813902+0000","last_undegraded":"2024-02-19T15:24:25.813902+0000","last_fullsized":"2024-02-19T15:24:25.813902+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:58:20.513106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.798429+0000","last_change":"2024-02-19T15:24:25.792593+0000","last_active":"2024-02-19T15:24:25.798429+0000","last_peered":"2024-02-19T15:24:25.798429+0000","last_clean":"2024-02-19T15:24:25.798429+0000","last_became_active":"2024-02-19T15:24:25.789965+0000","last_became_peered":"2024-02-19T15:24:25.789965+0000","last_unstale":"2024-02-19T15:24:25.798429+0000","last_undegraded":"2024-02-19T15:24:25.798429+0000","last_fullsized":"2024-02-19T15:24:25.798429+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:47:28.890973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.800641+0000","last_change":"2024-02-19T15:24:25.795963+0000","last_active":"2024-02-19T15:24:25.800641+0000","last_peered":"2024-02-19T15:24:25.800641+0000","last_clean":"2024-02-19T15:24:25.800641+0000","last_became_active":"2024-02-19T15:24:25.795628+0000","last_became_peered":"2024-02-19T15:24:25.795628+0000","last_unstale":"2024-02-19T15:24:25.800641+0000","last_undegraded":"2024-02-19T15:24:25.800641+0000","last_fullsized":"2024-02-19T15:24:25.800641+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:08:51.040190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.797819+0000","last_change":"2024-02-19T15:24:25.790397+0000","last_active":"2024-02-19T15:24:25.797819+0000","last_peered":"2024-02-19T15:24:25.797819+0000","last_clean":"2024-02-19T15:24:25.797819+0000","last_became_active":"2024-02-19T15:24:25.790119+0000","last_became_peered":"2024-02-19T15:24:25.790119+0000","last_unstale":"2024-02-19T15:24:25.797819+0000","last_undegraded":"2024-02-19T15:24:25.797819+0000","last_fullsized":"2024-02-19T15:24:25.797819+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:58:36.417120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.813688+0000","last_change":"2024-02-19T15:24:25.804365+0000","last_active":"2024-02-19T15:24:25.813688+0000","last_peered":"2024-02-19T15:24:25.813688+0000","last_clean":"2024-02-19T15:24:25.813688+0000","last_became_active":"2024-02-19T15:24:25.804127+0000","last_became_peered":"2024-02-19T15:24:25.804127+0000","last_unstale":"2024-02-19T15:24:25.813688+0000","last_undegraded":"2024-02-19T15:24:25.813688+0000","last_fullsized":"2024-02-19T15:24:25.813688+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:59:58.667597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.796206+0000","last_change":"2024-02-19T15:24:25.791344+0000","last_active":"2024-02-19T15:24:25.796206+0000","last_peered":"2024-02-19T15:24:25.796206+0000","last_clean":"2024-02-19T15:24:25.796206+0000","last_became_active":"2024-02-19T15:24:25.790033+0000","last_became_peered":"2024-02-19T15:24:25.790033+0000","last_unstale":"2024-02-19T15:24:25.796206+0000","last_undegraded":"2024-02-19T15:24:25.796206+0000","last_fullsized":"2024-02-19T15:24:25.796206+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:00:26.457771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.801267+0000","last_change":"2024-02-19T15:24:25.792062+0000","last_active":"2024-02-19T15:24:25.801267+0000","last_peered":"2024-02-19T15:24:25.801267+0000","last_clean":"2024-02-19T15:24:25.801267+0000","last_became_active":"2024-02-19T15:24:25.791620+0000","last_became_peered":"2024-02-19T15:24:25.791620+0000","last_unstale":"2024-02-19T15:24:25.801267+0000","last_undegraded":"2024-02-19T15:24:25.801267+0000","last_fullsized":"2024-02-19T15:24:25.801267+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:58:44.811899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.794654+0000","last_change":"2024-02-19T15:24:25.790905+0000","last_active":"2024-02-19T15:24:25.794654+0000","last_peered":"2024-02-19T15:24:25.794654+0000","last_clean":"2024-02-19T15:24:25.794654+0000","last_became_active":"2024-02-19T15:24:25.790620+0000","last_became_peered":"2024-02-19T15:24:25.790620+0000","last_unstale":"2024-02-19T15:24:25.794654+0000","last_undegraded":"2024-02-19T15:24:25.794654+0000","last_fullsized":"2024-02-19T15:24:25.794654+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:46.797100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.808563+0000","last_change":"2024-02-19T15:24:25.800569+0000","last_active":"2024-02-19T15:24:25.808563+0000","last_peered":"2024-02-19T15:24:25.808563+0000","last_clean":"2024-02-19T15:24:25.808563+0000","last_became_active":"2024-02-19T15:24:25.800265+0000","last_became_peered":"2024-02-19T15:24:25.800265+0000","last_unstale":"2024-02-19T15:24:25.808563+0000","last_undegraded":"2024-02-19T15:24:25.808563+0000","last_fullsized":"2024-02-19T15:24:25.808563+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:00:26.457771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.804749+0000","last_change":"2024-02-19T15:24:25.797257+0000","last_active":"2024-02-19T15:24:25.804749+0000","last_peered":"2024-02-19T15:24:25.804749+0000","last_clean":"2024-02-19T15:24:25.804749+0000","last_became_active":"2024-02-19T15:24:25.796935+0000","last_became_peered":"2024-02-19T15:24:25.796935+0000","last_unstale":"2024-02-19T15:24:25.804749+0000","last_undegraded":"2024-02-19T15:24:25.804749+0000","last_fullsized":"2024-02-19T15:24:25.804749+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:31:30.242430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.795280+0000","last_change":"2024-02-19T15:24:25.787718+0000","last_active":"2024-02-19T15:24:25.795280+0000","last_peered":"2024-02-19T15:24:25.795280+0000","last_clean":"2024-02-19T15:24:25.795280+0000","last_became_active":"2024-02-19T15:24:25.787400+0000","last_became_peered":"2024-02-19T15:24:25.787400+0000","last_unstale":"2024-02-19T15:24:25.795280+0000","last_undegraded":"2024-02-19T15:24:25.795280+0000","last_fullsized":"2024-02-19T15:24:25.795280+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:23:42.984554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T15:24:26.794754+0000","last_change":"2024-02-19T15:24:25.805528+0000","last_active":"2024-02-19T15:24:26.794754+0000","last_peered":"2024-02-19T15:24:26.794754+0000","last_clean":"2024-02-19T15:24:26.794754+0000","last_became_active":"2024-02-19T15:24:25.805029+0000","last_became_peered":"2024-02-19T15:24:25.805029+0000","last_unstale":"2024-02-19T15:24:26.794754+0000","last_undegraded":"2024-02-19T15:24:26.794754+0000","last_fullsized":"2024-02-19T15:24:26.794754+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:15:18.548657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.796496+0000","last_change":"2024-02-19T15:24:25.793791+0000","last_active":"2024-02-19T15:24:25.796496+0000","last_peered":"2024-02-19T15:24:25.796496+0000","last_clean":"2024-02-19T15:24:25.796496+0000","last_became_active":"2024-02-19T15:24:25.793409+0000","last_became_peered":"2024-02-19T15:24:25.793409+0000","last_unstale":"2024-02-19T15:24:25.796496+0000","last_undegraded":"2024-02-19T15:24:25.796496+0000","last_fullsized":"2024-02-19T15:24:25.796496+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:32:31.195399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.800599+0000","last_change":"2024-02-19T15:24:25.797765+0000","last_active":"2024-02-19T15:24:25.800599+0000","last_peered":"2024-02-19T15:24:25.800599+0000","last_clean":"2024-02-19T15:24:25.800599+0000","last_became_active":"2024-02-19T15:24:25.797156+0000","last_became_peered":"2024-02-19T15:24:25.797156+0000","last_unstale":"2024-02-19T15:24:25.800599+0000","last_undegraded":"2024-02-19T15:24:25.800599+0000","last_fullsized":"2024-02-19T15:24:25.800599+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:24:05.884765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.801049+0000","last_change":"2024-02-19T15:24:25.789116+0000","last_active":"2024-02-19T15:24:25.801049+0000","last_peered":"2024-02-19T15:24:25.801049+0000","last_clean":"2024-02-19T15:24:25.801049+0000","last_became_active":"2024-02-19T15:24:25.788273+0000","last_became_peered":"2024-02-19T15:24:25.788273+0000","last_unstale":"2024-02-19T15:24:25.801049+0000","last_undegraded":"2024-02-19T15:24:25.801049+0000","last_fullsized":"2024-02-19T15:24:25.801049+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:22:58.227504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.801322+0000","last_change":"2024-02-19T15:24:25.791269+0000","last_active":"2024-02-19T15:24:25.801322+0000","last_peered":"2024-02-19T15:24:25.801322+0000","last_clean":"2024-02-19T15:24:25.801322+0000","last_became_active":"2024-02-19T15:24:25.790948+0000","last_became_peered":"2024-02-19T15:24:25.790948+0000","last_unstale":"2024-02-19T15:24:25.801322+0000","last_undegraded":"2024-02-19T15:24:25.801322+0000","last_fullsized":"2024-02-19T15:24:25.801322+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:15:50.743494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.799984+0000","last_change":"2024-02-19T15:24:25.793869+0000","last_active":"2024-02-19T15:24:25.799984+0000","last_peered":"2024-02-19T15:24:25.799984+0000","last_clean":"2024-02-19T15:24:25.799984+0000","last_became_active":"2024-02-19T15:24:25.793458+0000","last_became_peered":"2024-02-19T15:24:25.793458+0000","last_unstale":"2024-02-19T15:24:25.799984+0000","last_undegraded":"2024-02-19T15:24:25.799984+0000","last_fullsized":"2024-02-19T15:24:25.799984+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:04:44.101587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.798658+0000","last_change":"2024-02-19T15:24:25.795041+0000","last_active":"2024-02-19T15:24:25.798658+0000","last_peered":"2024-02-19T15:24:25.798658+0000","last_clean":"2024-02-19T15:24:25.798658+0000","last_became_active":"2024-02-19T15:24:25.794820+0000","last_became_peered":"2024-02-19T15:24:25.794820+0000","last_unstale":"2024-02-19T15:24:25.798658+0000","last_undegraded":"2024-02-19T15:24:25.798658+0000","last_fullsized":"2024-02-19T15:24:25.798658+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:58:04.244227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.804873+0000","last_change":"2024-02-19T15:24:25.802212+0000","last_active":"2024-02-19T15:24:25.804873+0000","last_peered":"2024-02-19T15:24:25.804873+0000","last_clean":"2024-02-19T15:24:25.804873+0000","last_became_active":"2024-02-19T15:24:25.801956+0000","last_became_peered":"2024-02-19T15:24:25.801956+0000","last_unstale":"2024-02-19T15:24:25.804873+0000","last_undegraded":"2024-02-19T15:24:25.804873+0000","last_fullsized":"2024-02-19T15:24:25.804873+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:48:02.565569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.789657+0000","last_change":"2024-02-19T15:24:25.786159+0000","last_active":"2024-02-19T15:24:25.789657+0000","last_peered":"2024-02-19T15:24:25.789657+0000","last_clean":"2024-02-19T15:24:25.789657+0000","last_became_active":"2024-02-19T15:24:25.785806+0000","last_became_peered":"2024-02-19T15:24:25.785806+0000","last_unstale":"2024-02-19T15:24:25.789657+0000","last_undegraded":"2024-02-19T15:24:25.789657+0000","last_fullsized":"2024-02-19T15:24:25.789657+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:40:41.107930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.801353+0000","last_change":"2024-02-19T15:24:25.794415+0000","last_active":"2024-02-19T15:24:25.801353+0000","last_peered":"2024-02-19T15:24:25.801353+0000","last_clean":"2024-02-19T15:24:25.801353+0000","last_became_active":"2024-02-19T15:24:25.794092+0000","last_became_peered":"2024-02-19T15:24:25.794092+0000","last_unstale":"2024-02-19T15:24:25.801353+0000","last_undegraded":"2024-02-19T15:24:25.801353+0000","last_fullsized":"2024-02-19T15:24:25.801353+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:07:22.816266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.800029+0000","last_change":"2024-02-19T15:24:25.792140+0000","last_active":"2024-02-19T15:24:25.800029+0000","last_peered":"2024-02-19T15:24:25.800029+0000","last_clean":"2024-02-19T15:24:25.800029+0000","last_became_active":"2024-02-19T15:24:25.789775+0000","last_became_peered":"2024-02-19T15:24:25.789775+0000","last_unstale":"2024-02-19T15:24:25.800029+0000","last_undegraded":"2024-02-19T15:24:25.800029+0000","last_fullsized":"2024-02-19T15:24:25.800029+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:57:38.611541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T15:24:26.794669+0000","last_change":"2024-02-19T15:24:25.793174+0000","last_active":"2024-02-19T15:24:26.794669+0000","last_peered":"2024-02-19T15:24:26.794669+0000","last_clean":"2024-02-19T15:24:26.794669+0000","last_became_active":"2024-02-19T15:24:25.792851+0000","last_became_peered":"2024-02-19T15:24:25.792851+0000","last_unstale":"2024-02-19T15:24:26.794669+0000","last_undegraded":"2024-02-19T15:24:26.794669+0000","last_fullsized":"2024-02-19T15:24:26.794669+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:04:14.340364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.802614+0000","last_change":"2024-02-19T15:24:25.799850+0000","last_active":"2024-02-19T15:24:25.802614+0000","last_peered":"2024-02-19T15:24:25.802614+0000","last_clean":"2024-02-19T15:24:25.802614+0000","last_became_active":"2024-02-19T15:24:25.799557+0000","last_became_peered":"2024-02-19T15:24:25.799557+0000","last_unstale":"2024-02-19T15:24:25.802614+0000","last_undegraded":"2024-02-19T15:24:25.802614+0000","last_fullsized":"2024-02-19T15:24:25.802614+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:18:50.470518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.794458+0000","last_change":"2024-02-19T15:24:25.792532+0000","last_active":"2024-02-19T15:24:25.794458+0000","last_peered":"2024-02-19T15:24:25.794458+0000","last_clean":"2024-02-19T15:24:25.794458+0000","last_became_active":"2024-02-19T15:24:25.789010+0000","last_became_peered":"2024-02-19T15:24:25.789010+0000","last_unstale":"2024-02-19T15:24:25.794458+0000","last_undegraded":"2024-02-19T15:24:25.794458+0000","last_fullsized":"2024-02-19T15:24:25.794458+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:03:45.431655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.799743+0000","last_change":"2024-02-19T15:24:25.793770+0000","last_active":"2024-02-19T15:24:25.799743+0000","last_peered":"2024-02-19T15:24:25.799743+0000","last_clean":"2024-02-19T15:24:25.799743+0000","last_became_active":"2024-02-19T15:24:25.792785+0000","last_became_peered":"2024-02-19T15:24:25.792785+0000","last_unstale":"2024-02-19T15:24:25.799743+0000","last_undegraded":"2024-02-19T15:24:25.799743+0000","last_fullsized":"2024-02-19T15:24:25.799743+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:36:43.119696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.797963+0000","last_change":"2024-02-19T15:24:25.791810+0000","last_active":"2024-02-19T15:24:25.797963+0000","last_peered":"2024-02-19T15:24:25.797963+0000","last_clean":"2024-02-19T15:24:25.797963+0000","last_became_active":"2024-02-19T15:24:25.791504+0000","last_became_peered":"2024-02-19T15:24:25.791504+0000","last_unstale":"2024-02-19T15:24:25.797963+0000","last_undegraded":"2024-02-19T15:24:25.797963+0000","last_fullsized":"2024-02-19T15:24:25.797963+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:43.840049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.803858+0000","last_change":"2024-02-19T15:24:25.798676+0000","last_active":"2024-02-19T15:24:25.803858+0000","last_peered":"2024-02-19T15:24:25.803858+0000","last_clean":"2024-02-19T15:24:25.803858+0000","last_became_active":"2024-02-19T15:24:25.798316+0000","last_became_peered":"2024-02-19T15:24:25.798316+0000","last_unstale":"2024-02-19T15:24:25.803858+0000","last_undegraded":"2024-02-19T15:24:25.803858+0000","last_fullsized":"2024-02-19T15:24:25.803858+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:28:31.257699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.797954+0000","last_change":"2024-02-19T15:24:25.791923+0000","last_active":"2024-02-19T15:24:25.797954+0000","last_peered":"2024-02-19T15:24:25.797954+0000","last_clean":"2024-02-19T15:24:25.797954+0000","last_became_active":"2024-02-19T15:24:25.790030+0000","last_became_peered":"2024-02-19T15:24:25.790030+0000","last_unstale":"2024-02-19T15:24:25.797954+0000","last_undegraded":"2024-02-19T15:24:25.797954+0000","last_fullsized":"2024-02-19T15:24:25.797954+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:53:52.438163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.808232+0000","last_change":"2024-02-19T15:24:25.802237+0000","last_active":"2024-02-19T15:24:25.808232+0000","last_peered":"2024-02-19T15:24:25.808232+0000","last_clean":"2024-02-19T15:24:25.808232+0000","last_became_active":"2024-02-19T15:24:25.801961+0000","last_became_peered":"2024-02-19T15:24:25.801961+0000","last_unstale":"2024-02-19T15:24:25.808232+0000","last_undegraded":"2024-02-19T15:24:25.808232+0000","last_fullsized":"2024-02-19T15:24:25.808232+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:23:55.685803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.804920+0000","last_change":"2024-02-19T15:24:25.799780+0000","last_active":"2024-02-19T15:24:25.804920+0000","last_peered":"2024-02-19T15:24:25.804920+0000","last_clean":"2024-02-19T15:24:25.804920+0000","last_became_active":"2024-02-19T15:24:25.792312+0000","last_became_peered":"2024-02-19T15:24:25.792312+0000","last_unstale":"2024-02-19T15:24:25.804920+0000","last_undegraded":"2024-02-19T15:24:25.804920+0000","last_fullsized":"2024-02-19T15:24:25.804920+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:54:08.004283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.800924+0000","last_change":"2024-02-19T15:24:25.792493+0000","last_active":"2024-02-19T15:24:25.800924+0000","last_peered":"2024-02-19T15:24:25.800924+0000","last_clean":"2024-02-19T15:24:25.800924+0000","last_became_active":"2024-02-19T15:24:25.792127+0000","last_became_peered":"2024-02-19T15:24:25.792127+0000","last_unstale":"2024-02-19T15:24:25.800924+0000","last_undegraded":"2024-02-19T15:24:25.800924+0000","last_fullsized":"2024-02-19T15:24:25.800924+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:55:20.381916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"56'4","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.816146+0000","last_change":"2024-02-19T15:24:24.780943+0000","last_active":"2024-02-19T15:24:25.816146+0000","last_peered":"2024-02-19T15:24:25.816146+0000","last_clean":"2024-02-19T15:24:25.816146+0000","last_became_active":"2024-02-19T15:24:24.780445+0000","last_became_peered":"2024-02-19T15:24:24.780445+0000","last_unstale":"2024-02-19T15:24:25.816146+0000","last_undegraded":"2024-02-19T15:24:25.816146+0000","last_fullsized":"2024-02-19T15:24:25.816146+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:20:56.982139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.808115+0000","last_change":"2024-02-19T15:24:25.800140+0000","last_active":"2024-02-19T15:24:25.808115+0000","last_peered":"2024-02-19T15:24:25.808115+0000","last_clean":"2024-02-19T15:24:25.808115+0000","last_became_active":"2024-02-19T15:24:25.799884+0000","last_became_peered":"2024-02-19T15:24:25.799884+0000","last_unstale":"2024-02-19T15:24:25.808115+0000","last_undegraded":"2024-02-19T15:24:25.808115+0000","last_fullsized":"2024-02-19T15:24:25.808115+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:46.797100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"56'1","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.809211+0000","last_change":"2024-02-19T15:24:24.782062+0000","last_active":"2024-02-19T15:24:25.809211+0000","last_peered":"2024-02-19T15:24:25.809211+0000","last_clean":"2024-02-19T15:24:25.809211+0000","last_became_active":"2024-02-19T15:24:24.781187+0000","last_became_peered":"2024-02-19T15:24:24.781187+0000","last_unstale":"2024-02-19T15:24:25.809211+0000","last_undegraded":"2024-02-19T15:24:25.809211+0000","last_fullsized":"2024-02-19T15:24:25.809211+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:20:49.931259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.813833+0000","last_change":"2024-02-19T15:24:25.805592+0000","last_active":"2024-02-19T15:24:25.813833+0000","last_peered":"2024-02-19T15:24:25.813833+0000","last_clean":"2024-02-19T15:24:25.813833+0000","last_became_active":"2024-02-19T15:24:25.805307+0000","last_became_peered":"2024-02-19T15:24:25.805307+0000","last_unstale":"2024-02-19T15:24:25.813833+0000","last_undegraded":"2024-02-19T15:24:25.813833+0000","last_fullsized":"2024-02-19T15:24:25.813833+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:29:06.908077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"56'2","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.817927+0000","last_change":"2024-02-19T15:24:24.778780+0000","last_active":"2024-02-19T15:24:25.817927+0000","last_peered":"2024-02-19T15:24:25.817927+0000","last_clean":"2024-02-19T15:24:25.817927+0000","last_became_active":"2024-02-19T15:24:24.775721+0000","last_became_peered":"2024-02-19T15:24:24.775721+0000","last_unstale":"2024-02-19T15:24:25.817927+0000","last_undegraded":"2024-02-19T15:24:25.817927+0000","last_fullsized":"2024-02-19T15:24:25.817927+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:51:59.754257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.810953+0000","last_change":"2024-02-19T15:24:25.794253+0000","last_active":"2024-02-19T15:24:25.810953+0000","last_peered":"2024-02-19T15:24:25.810953+0000","last_clean":"2024-02-19T15:24:25.810953+0000","last_became_active":"2024-02-19T15:24:25.793501+0000","last_became_peered":"2024-02-19T15:24:25.793501+0000","last_unstale":"2024-02-19T15:24:25.810953+0000","last_undegraded":"2024-02-19T15:24:25.810953+0000","last_fullsized":"2024-02-19T15:24:25.810953+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:14:50.168672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"56'2","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.807453+0000","last_change":"2024-02-19T15:24:24.778907+0000","last_active":"2024-02-19T15:24:25.807453+0000","last_peered":"2024-02-19T15:24:25.807453+0000","last_clean":"2024-02-19T15:24:25.807453+0000","last_became_active":"2024-02-19T15:24:24.778611+0000","last_became_peered":"2024-02-19T15:24:24.778611+0000","last_unstale":"2024-02-19T15:24:25.807453+0000","last_undegraded":"2024-02-19T15:24:25.807453+0000","last_fullsized":"2024-02-19T15:24:25.807453+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:03:16.895694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.803971+0000","last_change":"2024-02-19T15:24:25.799210+0000","last_active":"2024-02-19T15:24:25.803971+0000","last_peered":"2024-02-19T15:24:25.803971+0000","last_clean":"2024-02-19T15:24:25.803971+0000","last_became_active":"2024-02-19T15:24:25.798887+0000","last_became_peered":"2024-02-19T15:24:25.798887+0000","last_unstale":"2024-02-19T15:24:25.803971+0000","last_undegraded":"2024-02-19T15:24:25.803971+0000","last_fullsized":"2024-02-19T15:24:25.803971+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:52:51.662334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"56'1","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.808084+0000","last_change":"2024-02-19T15:24:24.786129+0000","last_active":"2024-02-19T15:24:25.808084+0000","last_peered":"2024-02-19T15:24:25.808084+0000","last_clean":"2024-02-19T15:24:25.808084+0000","last_became_active":"2024-02-19T15:24:24.785862+0000","last_became_peered":"2024-02-19T15:24:24.785862+0000","last_unstale":"2024-02-19T15:24:25.808084+0000","last_undegraded":"2024-02-19T15:24:25.808084+0000","last_fullsized":"2024-02-19T15:24:25.808084+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:37:36.490461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.812766+0000","last_change":"2024-02-19T15:24:25.803601+0000","last_active":"2024-02-19T15:24:25.812766+0000","last_peered":"2024-02-19T15:24:25.812766+0000","last_clean":"2024-02-19T15:24:25.812766+0000","last_became_active":"2024-02-19T15:24:25.803335+0000","last_became_peered":"2024-02-19T15:24:25.803335+0000","last_unstale":"2024-02-19T15:24:25.812766+0000","last_undegraded":"2024-02-19T15:24:25.812766+0000","last_fullsized":"2024-02-19T15:24:25.812766+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:42:01.819654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"56'1","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.816054+0000","last_change":"2024-02-19T15:24:24.772468+0000","last_active":"2024-02-19T15:24:25.816054+0000","last_peered":"2024-02-19T15:24:25.816054+0000","last_clean":"2024-02-19T15:24:25.816054+0000","last_became_active":"2024-02-19T15:24:24.770991+0000","last_became_peered":"2024-02-19T15:24:24.770991+0000","last_unstale":"2024-02-19T15:24:25.816054+0000","last_undegraded":"2024-02-19T15:24:25.816054+0000","last_fullsized":"2024-02-19T15:24:25.816054+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:33:05.568133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T15:24:26.794581+0000","last_change":"2024-02-19T15:24:25.791109+0000","last_active":"2024-02-19T15:24:26.794581+0000","last_peered":"2024-02-19T15:24:26.794581+0000","last_clean":"2024-02-19T15:24:26.794581+0000","last_became_active":"2024-02-19T15:24:25.790828+0000","last_became_peered":"2024-02-19T15:24:25.790828+0000","last_unstale":"2024-02-19T15:24:26.794581+0000","last_undegraded":"2024-02-19T15:24:26.794581+0000","last_fullsized":"2024-02-19T15:24:26.794581+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:42:56.739146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.774036+0000","last_change":"2024-02-19T15:24:24.779166+0000","last_active":"2024-02-19T15:24:25.774036+0000","last_peered":"2024-02-19T15:24:25.774036+0000","last_clean":"2024-02-19T15:24:25.774036+0000","last_became_active":"2024-02-19T15:24:24.778855+0000","last_became_peered":"2024-02-19T15:24:24.778855+0000","last_unstale":"2024-02-19T15:24:25.774036+0000","last_undegraded":"2024-02-19T15:24:25.774036+0000","last_fullsized":"2024-02-19T15:24:25.774036+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:24:45.653235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.800680+0000","last_change":"2024-02-19T15:24:25.788623+0000","last_active":"2024-02-19T15:24:25.800680+0000","last_peered":"2024-02-19T15:24:25.800680+0000","last_clean":"2024-02-19T15:24:25.800680+0000","last_became_active":"2024-02-19T15:24:25.788031+0000","last_became_peered":"2024-02-19T15:24:25.788031+0000","last_unstale":"2024-02-19T15:24:25.800680+0000","last_undegraded":"2024-02-19T15:24:25.800680+0000","last_fullsized":"2024-02-19T15:24:25.800680+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:59:03.679558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.784202+0000","last_change":"2024-02-19T15:24:24.781184+0000","last_active":"2024-02-19T15:24:25.784202+0000","last_peered":"2024-02-19T15:24:25.784202+0000","last_clean":"2024-02-19T15:24:25.784202+0000","last_became_active":"2024-02-19T15:24:24.780208+0000","last_became_peered":"2024-02-19T15:24:24.780208+0000","last_unstale":"2024-02-19T15:24:25.784202+0000","last_undegraded":"2024-02-19T15:24:25.784202+0000","last_fullsized":"2024-02-19T15:24:25.784202+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:31:29.236618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.816820+0000","last_change":"2024-02-19T15:24:25.815143+0000","last_active":"2024-02-19T15:24:25.816820+0000","last_peered":"2024-02-19T15:24:25.816820+0000","last_clean":"2024-02-19T15:24:25.816820+0000","last_became_active":"2024-02-19T15:24:25.814626+0000","last_became_peered":"2024-02-19T15:24:25.814626+0000","last_unstale":"2024-02-19T15:24:25.816820+0000","last_undegraded":"2024-02-19T15:24:25.816820+0000","last_fullsized":"2024-02-19T15:24:25.816820+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:15:56.825197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.776004+0000","last_change":"2024-02-19T15:24:24.782764+0000","last_active":"2024-02-19T15:24:25.776004+0000","last_peered":"2024-02-19T15:24:25.776004+0000","last_clean":"2024-02-19T15:24:25.776004+0000","last_became_active":"2024-02-19T15:24:24.782458+0000","last_became_peered":"2024-02-19T15:24:24.782458+0000","last_unstale":"2024-02-19T15:24:25.776004+0000","last_undegraded":"2024-02-19T15:24:25.776004+0000","last_fullsized":"2024-02-19T15:24:25.776004+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:23:41.978742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.801343+0000","last_change":"2024-02-19T15:24:25.797502+0000","last_active":"2024-02-19T15:24:25.801343+0000","last_peered":"2024-02-19T15:24:25.801343+0000","last_clean":"2024-02-19T15:24:25.801343+0000","last_became_active":"2024-02-19T15:24:25.797201+0000","last_became_peered":"2024-02-19T15:24:25.797201+0000","last_unstale":"2024-02-19T15:24:25.801343+0000","last_undegraded":"2024-02-19T15:24:25.801343+0000","last_fullsized":"2024-02-19T15:24:25.801343+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:39:49.038147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.786652+0000","last_change":"2024-02-19T15:24:24.788388+0000","last_active":"2024-02-19T15:24:25.786652+0000","last_peered":"2024-02-19T15:24:25.786652+0000","last_clean":"2024-02-19T15:24:25.786652+0000","last_became_active":"2024-02-19T15:24:24.788079+0000","last_became_peered":"2024-02-19T15:24:24.788079+0000","last_unstale":"2024-02-19T15:24:25.786652+0000","last_undegraded":"2024-02-19T15:24:25.786652+0000","last_fullsized":"2024-02-19T15:24:25.786652+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:04:43.095775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.807288+0000","last_change":"2024-02-19T15:24:25.798423+0000","last_active":"2024-02-19T15:24:25.807288+0000","last_peered":"2024-02-19T15:24:25.807288+0000","last_clean":"2024-02-19T15:24:25.807288+0000","last_became_active":"2024-02-19T15:24:25.797928+0000","last_became_peered":"2024-02-19T15:24:25.797928+0000","last_unstale":"2024-02-19T15:24:25.807288+0000","last_undegraded":"2024-02-19T15:24:25.807288+0000","last_fullsized":"2024-02-19T15:24:25.807288+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:58:19.530703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"56'3","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.815423+0000","last_change":"2024-02-19T15:24:24.778146+0000","last_active":"2024-02-19T15:24:25.815423+0000","last_peered":"2024-02-19T15:24:25.815423+0000","last_clean":"2024-02-19T15:24:25.815423+0000","last_became_active":"2024-02-19T15:24:24.777832+0000","last_became_peered":"2024-02-19T15:24:24.777832+0000","last_unstale":"2024-02-19T15:24:25.815423+0000","last_undegraded":"2024-02-19T15:24:25.815423+0000","last_fullsized":"2024-02-19T15:24:25.815423+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:08:50.034377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.803486+0000","last_change":"2024-02-19T15:24:25.797982+0000","last_active":"2024-02-19T15:24:25.803486+0000","last_peered":"2024-02-19T15:24:25.803486+0000","last_clean":"2024-02-19T15:24:25.803486+0000","last_became_active":"2024-02-19T15:24:25.797264+0000","last_became_peered":"2024-02-19T15:24:25.797264+0000","last_unstale":"2024-02-19T15:24:25.803486+0000","last_undegraded":"2024-02-19T15:24:25.803486+0000","last_fullsized":"2024-02-19T15:24:25.803486+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:38:38.879866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"56'6","reported_seq":18,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.817988+0000","last_change":"2024-02-19T15:24:24.776974+0000","last_active":"2024-02-19T15:24:25.817988+0000","last_peered":"2024-02-19T15:24:25.817988+0000","last_clean":"2024-02-19T15:24:25.817988+0000","last_became_active":"2024-02-19T15:24:24.776702+0000","last_became_peered":"2024-02-19T15:24:24.776702+0000","last_unstale":"2024-02-19T15:24:25.817988+0000","last_undegraded":"2024-02-19T15:24:25.817988+0000","last_fullsized":"2024-02-19T15:24:25.817988+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:43:21.847668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.800396+0000","last_change":"2024-02-19T15:24:25.798573+0000","last_active":"2024-02-19T15:24:25.800396+0000","last_peered":"2024-02-19T15:24:25.800396+0000","last_clean":"2024-02-19T15:24:25.800396+0000","last_became_active":"2024-02-19T15:24:25.798311+0000","last_became_peered":"2024-02-19T15:24:25.798311+0000","last_unstale":"2024-02-19T15:24:25.800396+0000","last_undegraded":"2024-02-19T15:24:25.800396+0000","last_fullsized":"2024-02-19T15:24:25.800396+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:40:41.107930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"56'2","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.814889+0000","last_change":"2024-02-19T15:24:24.788679+0000","last_active":"2024-02-19T15:24:25.814889+0000","last_peered":"2024-02-19T15:24:25.814889+0000","last_clean":"2024-02-19T15:24:25.814889+0000","last_became_active":"2024-02-19T15:24:24.786821+0000","last_became_peered":"2024-02-19T15:24:24.786821+0000","last_unstale":"2024-02-19T15:24:25.814889+0000","last_undegraded":"2024-02-19T15:24:25.814889+0000","last_fullsized":"2024-02-19T15:24:25.814889+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:47:27.885160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.789679+0000","last_change":"2024-02-19T15:24:25.786651+0000","last_active":"2024-02-19T15:24:25.789679+0000","last_peered":"2024-02-19T15:24:25.789679+0000","last_clean":"2024-02-19T15:24:25.789679+0000","last_became_active":"2024-02-19T15:24:25.786212+0000","last_became_peered":"2024-02-19T15:24:25.786212+0000","last_unstale":"2024-02-19T15:24:25.789679+0000","last_undegraded":"2024-02-19T15:24:25.789679+0000","last_fullsized":"2024-02-19T15:24:25.789679+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:34:11.045516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T15:24:26.794589+0000","last_change":"2024-02-19T15:24:24.775478+0000","last_active":"2024-02-19T15:24:26.794589+0000","last_peered":"2024-02-19T15:24:26.794589+0000","last_clean":"2024-02-19T15:24:26.794589+0000","last_became_active":"2024-02-19T15:24:24.775221+0000","last_became_peered":"2024-02-19T15:24:24.775221+0000","last_unstale":"2024-02-19T15:24:26.794589+0000","last_undegraded":"2024-02-19T15:24:26.794589+0000","last_fullsized":"2024-02-19T15:24:26.794589+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:57:32.721092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.794132+0000","last_change":"2024-02-19T15:24:25.789486+0000","last_active":"2024-02-19T15:24:25.794132+0000","last_peered":"2024-02-19T15:24:25.794132+0000","last_clean":"2024-02-19T15:24:25.794132+0000","last_became_active":"2024-02-19T15:24:25.789200+0000","last_became_peered":"2024-02-19T15:24:25.789200+0000","last_unstale":"2024-02-19T15:24:25.794132+0000","last_undegraded":"2024-02-19T15:24:25.794132+0000","last_fullsized":"2024-02-19T15:24:25.794132+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:58:19.530703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.774251+0000","last_change":"2024-02-19T15:24:24.780605+0000","last_active":"2024-02-19T15:24:25.774251+0000","last_peered":"2024-02-19T15:24:25.774251+0000","last_clean":"2024-02-19T15:24:25.774251+0000","last_became_active":"2024-02-19T15:24:24.778657+0000","last_became_peered":"2024-02-19T15:24:24.778657+0000","last_unstale":"2024-02-19T15:24:25.774251+0000","last_undegraded":"2024-02-19T15:24:25.774251+0000","last_fullsized":"2024-02-19T15:24:25.774251+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:34:41.694128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.799996+0000","last_change":"2024-02-19T15:24:25.793802+0000","last_active":"2024-02-19T15:24:25.799996+0000","last_peered":"2024-02-19T15:24:25.799996+0000","last_clean":"2024-02-19T15:24:25.799996+0000","last_became_active":"2024-02-19T15:24:25.790134+0000","last_became_peered":"2024-02-19T15:24:25.790134+0000","last_unstale":"2024-02-19T15:24:25.799996+0000","last_undegraded":"2024-02-19T15:24:25.799996+0000","last_fullsized":"2024-02-19T15:24:25.799996+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:20:50.937071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"56'8","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.807293+0000","last_change":"2024-02-19T15:24:24.777310+0000","last_active":"2024-02-19T15:24:25.807293+0000","last_peered":"2024-02-19T15:24:25.807293+0000","last_clean":"2024-02-19T15:24:25.807293+0000","last_became_active":"2024-02-19T15:24:24.777040+0000","last_became_peered":"2024-02-19T15:24:24.777040+0000","last_unstale":"2024-02-19T15:24:25.807293+0000","last_undegraded":"2024-02-19T15:24:25.807293+0000","last_fullsized":"2024-02-19T15:24:25.807293+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:03:44.425842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.797905+0000","last_change":"2024-02-19T15:24:25.789501+0000","last_active":"2024-02-19T15:24:25.797905+0000","last_peered":"2024-02-19T15:24:25.797905+0000","last_clean":"2024-02-19T15:24:25.797905+0000","last_became_active":"2024-02-19T15:24:25.787235+0000","last_became_peered":"2024-02-19T15:24:25.787235+0000","last_unstale":"2024-02-19T15:24:25.797905+0000","last_undegraded":"2024-02-19T15:24:25.797905+0000","last_fullsized":"2024-02-19T15:24:25.797905+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:02:06.503687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"56'1","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.813830+0000","last_change":"2024-02-19T15:24:24.788842+0000","last_active":"2024-02-19T15:24:25.813830+0000","last_peered":"2024-02-19T15:24:25.813830+0000","last_clean":"2024-02-19T15:24:25.813830+0000","last_became_active":"2024-02-19T15:24:24.788228+0000","last_became_peered":"2024-02-19T15:24:24.788228+0000","last_unstale":"2024-02-19T15:24:25.813830+0000","last_undegraded":"2024-02-19T15:24:25.813830+0000","last_fullsized":"2024-02-19T15:24:25.813830+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:39:48.032335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.800672+0000","last_change":"2024-02-19T15:24:25.798800+0000","last_active":"2024-02-19T15:24:25.800672+0000","last_peered":"2024-02-19T15:24:25.800672+0000","last_clean":"2024-02-19T15:24:25.800672+0000","last_became_active":"2024-02-19T15:24:25.798432+0000","last_became_peered":"2024-02-19T15:24:25.798432+0000","last_unstale":"2024-02-19T15:24:25.800672+0000","last_undegraded":"2024-02-19T15:24:25.800672+0000","last_fullsized":"2024-02-19T15:24:25.800672+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:34:11.045516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"56'2","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.816961+0000","last_change":"2024-02-19T15:24:24.789536+0000","last_active":"2024-02-19T15:24:25.816961+0000","last_peered":"2024-02-19T15:24:25.816961+0000","last_clean":"2024-02-19T15:24:25.816961+0000","last_became_active":"2024-02-19T15:24:24.788777+0000","last_became_peered":"2024-02-19T15:24:24.788777+0000","last_unstale":"2024-02-19T15:24:25.816961+0000","last_undegraded":"2024-02-19T15:24:25.816961+0000","last_fullsized":"2024-02-19T15:24:25.816961+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:18:38.223058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.801228+0000","last_change":"2024-02-19T15:24:25.790592+0000","last_active":"2024-02-19T15:24:25.801228+0000","last_peered":"2024-02-19T15:24:25.801228+0000","last_clean":"2024-02-19T15:24:25.801228+0000","last_became_active":"2024-02-19T15:24:25.790334+0000","last_became_peered":"2024-02-19T15:24:25.790334+0000","last_unstale":"2024-02-19T15:24:25.801228+0000","last_undegraded":"2024-02-19T15:24:25.801228+0000","last_fullsized":"2024-02-19T15:24:25.801228+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:46:45.322881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"56'1","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.807477+0000","last_change":"2024-02-19T15:24:24.778871+0000","last_active":"2024-02-19T15:24:25.807477+0000","last_peered":"2024-02-19T15:24:25.807477+0000","last_clean":"2024-02-19T15:24:25.807477+0000","last_became_active":"2024-02-19T15:24:24.778576+0000","last_became_peered":"2024-02-19T15:24:24.778576+0000","last_unstale":"2024-02-19T15:24:25.807477+0000","last_undegraded":"2024-02-19T15:24:25.807477+0000","last_fullsized":"2024-02-19T15:24:25.807477+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:51:09.277753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.803512+0000","last_change":"2024-02-19T15:24:25.798235+0000","last_active":"2024-02-19T15:24:25.803512+0000","last_peered":"2024-02-19T15:24:25.803512+0000","last_clean":"2024-02-19T15:24:25.803512+0000","last_became_active":"2024-02-19T15:24:25.797669+0000","last_became_peered":"2024-02-19T15:24:25.797669+0000","last_unstale":"2024-02-19T15:24:25.803512+0000","last_undegraded":"2024-02-19T15:24:25.803512+0000","last_fullsized":"2024-02-19T15:24:25.803512+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:59:03.679558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.786047+0000","last_change":"2024-02-19T15:24:24.782022+0000","last_active":"2024-02-19T15:24:25.786047+0000","last_peered":"2024-02-19T15:24:25.786047+0000","last_clean":"2024-02-19T15:24:25.786047+0000","last_became_active":"2024-02-19T15:24:24.781731+0000","last_became_peered":"2024-02-19T15:24:24.781731+0000","last_unstale":"2024-02-19T15:24:25.786047+0000","last_undegraded":"2024-02-19T15:24:25.786047+0000","last_fullsized":"2024-02-19T15:24:25.786047+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:37:52.194274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.800636+0000","last_change":"2024-02-19T15:24:25.798992+0000","last_active":"2024-02-19T15:24:25.800636+0000","last_peered":"2024-02-19T15:24:25.800636+0000","last_clean":"2024-02-19T15:24:25.800636+0000","last_became_active":"2024-02-19T15:24:25.797625+0000","last_became_peered":"2024-02-19T15:24:25.797625+0000","last_unstale":"2024-02-19T15:24:25.800636+0000","last_undegraded":"2024-02-19T15:24:25.800636+0000","last_fullsized":"2024-02-19T15:24:25.800636+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:02:06.503687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"56'2","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.814827+0000","last_change":"2024-02-19T15:24:24.772661+0000","last_active":"2024-02-19T15:24:25.814827+0000","last_peered":"2024-02-19T15:24:25.814827+0000","last_clean":"2024-02-19T15:24:25.814827+0000","last_became_active":"2024-02-19T15:24:24.769218+0000","last_became_peered":"2024-02-19T15:24:24.769218+0000","last_unstale":"2024-02-19T15:24:25.814827+0000","last_undegraded":"2024-02-19T15:24:25.814827+0000","last_fullsized":"2024-02-19T15:24:25.814827+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:00:40.629030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.812212+0000","last_change":"2024-02-19T15:24:25.802788+0000","last_active":"2024-02-19T15:24:25.812212+0000","last_peered":"2024-02-19T15:24:25.812212+0000","last_clean":"2024-02-19T15:24:25.812212+0000","last_became_active":"2024-02-19T15:24:25.798924+0000","last_became_peered":"2024-02-19T15:24:25.798924+0000","last_unstale":"2024-02-19T15:24:25.812212+0000","last_undegraded":"2024-02-19T15:24:25.812212+0000","last_fullsized":"2024-02-19T15:24:25.812212+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:58:34.401332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.784096+0000","last_change":"2024-02-19T15:24:24.785330+0000","last_active":"2024-02-19T15:24:25.784096+0000","last_peered":"2024-02-19T15:24:25.784096+0000","last_clean":"2024-02-19T15:24:25.784096+0000","last_became_active":"2024-02-19T15:24:24.784968+0000","last_became_peered":"2024-02-19T15:24:24.784968+0000","last_unstale":"2024-02-19T15:24:25.784096+0000","last_undegraded":"2024-02-19T15:24:25.784096+0000","last_fullsized":"2024-02-19T15:24:25.784096+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:51:22.245500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.802987+0000","last_change":"2024-02-19T15:24:25.792382+0000","last_active":"2024-02-19T15:24:25.802987+0000","last_peered":"2024-02-19T15:24:25.802987+0000","last_clean":"2024-02-19T15:24:25.802987+0000","last_became_active":"2024-02-19T15:24:25.792064+0000","last_became_peered":"2024-02-19T15:24:25.792064+0000","last_unstale":"2024-02-19T15:24:25.802987+0000","last_undegraded":"2024-02-19T15:24:25.802987+0000","last_fullsized":"2024-02-19T15:24:25.802987+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:04:12.392497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.774009+0000","last_change":"2024-02-19T15:24:24.783263+0000","last_active":"2024-02-19T15:24:25.774009+0000","last_peered":"2024-02-19T15:24:25.774009+0000","last_clean":"2024-02-19T15:24:25.774009+0000","last_became_active":"2024-02-19T15:24:24.783015+0000","last_became_peered":"2024-02-19T15:24:24.783015+0000","last_unstale":"2024-02-19T15:24:25.774009+0000","last_undegraded":"2024-02-19T15:24:25.774009+0000","last_fullsized":"2024-02-19T15:24:25.774009+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:13:33.903931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.792741+0000","last_change":"2024-02-19T15:24:25.784498+0000","last_active":"2024-02-19T15:24:25.792741+0000","last_peered":"2024-02-19T15:24:25.792741+0000","last_clean":"2024-02-19T15:24:25.792741+0000","last_became_active":"2024-02-19T15:24:25.784174+0000","last_became_peered":"2024-02-19T15:24:25.784174+0000","last_unstale":"2024-02-19T15:24:25.792741+0000","last_undegraded":"2024-02-19T15:24:25.792741+0000","last_fullsized":"2024-02-19T15:24:25.792741+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:35:30.247678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"56'2","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.815198+0000","last_change":"2024-02-19T15:24:24.767265+0000","last_active":"2024-02-19T15:24:25.815198+0000","last_peered":"2024-02-19T15:24:25.815198+0000","last_clean":"2024-02-19T15:24:25.815198+0000","last_became_active":"2024-02-19T15:24:24.766916+0000","last_became_peered":"2024-02-19T15:24:24.766916+0000","last_unstale":"2024-02-19T15:24:25.815198+0000","last_undegraded":"2024-02-19T15:24:25.815198+0000","last_fullsized":"2024-02-19T15:24:25.815198+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:09:46.133897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.792497+0000","last_change":"2024-02-19T15:24:25.787065+0000","last_active":"2024-02-19T15:24:25.792497+0000","last_peered":"2024-02-19T15:24:25.792497+0000","last_clean":"2024-02-19T15:24:25.792497+0000","last_became_active":"2024-02-19T15:24:25.786562+0000","last_became_peered":"2024-02-19T15:24:25.786562+0000","last_unstale":"2024-02-19T15:24:25.792497+0000","last_undegraded":"2024-02-19T15:24:25.792497+0000","last_fullsized":"2024-02-19T15:24:25.792497+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:59.656620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.787306+0000","last_change":"2024-02-19T15:24:24.778112+0000","last_active":"2024-02-19T15:24:25.787306+0000","last_peered":"2024-02-19T15:24:25.787306+0000","last_clean":"2024-02-19T15:24:25.787306+0000","last_became_active":"2024-02-19T15:24:24.777664+0000","last_became_peered":"2024-02-19T15:24:24.777664+0000","last_unstale":"2024-02-19T15:24:25.787306+0000","last_undegraded":"2024-02-19T15:24:25.787306+0000","last_fullsized":"2024-02-19T15:24:25.787306+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:46:45.287461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.807231+0000","last_change":"2024-02-19T15:24:25.798196+0000","last_active":"2024-02-19T15:24:25.807231+0000","last_peered":"2024-02-19T15:24:25.807231+0000","last_clean":"2024-02-19T15:24:25.807231+0000","last_became_active":"2024-02-19T15:24:25.797511+0000","last_became_peered":"2024-02-19T15:24:25.797511+0000","last_unstale":"2024-02-19T15:24:25.807231+0000","last_undegraded":"2024-02-19T15:24:25.807231+0000","last_fullsized":"2024-02-19T15:24:25.807231+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:43:22.853480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T15:24:26.794862+0000","last_change":"2024-02-19T15:24:24.769353+0000","last_active":"2024-02-19T15:24:26.794862+0000","last_peered":"2024-02-19T15:24:26.794862+0000","last_clean":"2024-02-19T15:24:26.794862+0000","last_became_active":"2024-02-19T15:24:24.768970+0000","last_became_peered":"2024-02-19T15:24:24.768970+0000","last_unstale":"2024-02-19T15:24:26.794862+0000","last_undegraded":"2024-02-19T15:24:26.794862+0000","last_fullsized":"2024-02-19T15:24:26.794862+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:06:35.623552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.799769+0000","last_change":"2024-02-19T15:24:25.794049+0000","last_active":"2024-02-19T15:24:25.799769+0000","last_peered":"2024-02-19T15:24:25.799769+0000","last_clean":"2024-02-19T15:24:25.799769+0000","last_became_active":"2024-02-19T15:24:25.793167+0000","last_became_peered":"2024-02-19T15:24:25.793167+0000","last_unstale":"2024-02-19T15:24:25.799769+0000","last_undegraded":"2024-02-19T15:24:25.799769+0000","last_fullsized":"2024-02-19T15:24:25.799769+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:17:49.928845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.775778+0000","last_change":"2024-02-19T15:24:24.780196+0000","last_active":"2024-02-19T15:24:25.775778+0000","last_peered":"2024-02-19T15:24:25.775778+0000","last_clean":"2024-02-19T15:24:25.775778+0000","last_became_active":"2024-02-19T15:24:24.778936+0000","last_became_peered":"2024-02-19T15:24:24.778936+0000","last_unstale":"2024-02-19T15:24:25.775778+0000","last_undegraded":"2024-02-19T15:24:25.775778+0000","last_fullsized":"2024-02-19T15:24:25.775778+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:56:38.605488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.804053+0000","last_change":"2024-02-19T15:24:25.800386+0000","last_active":"2024-02-19T15:24:25.804053+0000","last_peered":"2024-02-19T15:24:25.804053+0000","last_clean":"2024-02-19T15:24:25.804053+0000","last_became_active":"2024-02-19T15:24:25.800150+0000","last_became_peered":"2024-02-19T15:24:25.800150+0000","last_unstale":"2024-02-19T15:24:25.804053+0000","last_undegraded":"2024-02-19T15:24:25.804053+0000","last_fullsized":"2024-02-19T15:24:25.804053+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:58:25.822438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.779173+0000","last_change":"2024-02-19T15:24:24.776772+0000","last_active":"2024-02-19T15:24:25.779173+0000","last_peered":"2024-02-19T15:24:25.779173+0000","last_clean":"2024-02-19T15:24:25.779173+0000","last_became_active":"2024-02-19T15:24:24.776479+0000","last_became_peered":"2024-02-19T15:24:24.776479+0000","last_unstale":"2024-02-19T15:24:25.779173+0000","last_undegraded":"2024-02-19T15:24:25.779173+0000","last_fullsized":"2024-02-19T15:24:25.779173+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:47:51.312636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.800888+0000","last_change":"2024-02-19T15:24:25.795341+0000","last_active":"2024-02-19T15:24:25.800888+0000","last_peered":"2024-02-19T15:24:25.800888+0000","last_clean":"2024-02-19T15:24:25.800888+0000","last_became_active":"2024-02-19T15:24:25.792938+0000","last_became_peered":"2024-02-19T15:24:25.792938+0000","last_unstale":"2024-02-19T15:24:25.800888+0000","last_undegraded":"2024-02-19T15:24:25.800888+0000","last_fullsized":"2024-02-19T15:24:25.800888+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:00:41.634842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.783554+0000","last_change":"2024-02-19T15:24:24.777682+0000","last_active":"2024-02-19T15:24:25.783554+0000","last_peered":"2024-02-19T15:24:25.783554+0000","last_clean":"2024-02-19T15:24:25.783554+0000","last_became_active":"2024-02-19T15:24:24.777391+0000","last_became_peered":"2024-02-19T15:24:24.777391+0000","last_unstale":"2024-02-19T15:24:25.783554+0000","last_undegraded":"2024-02-19T15:24:25.783554+0000","last_fullsized":"2024-02-19T15:24:25.783554+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:51:22.245500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.805976+0000","last_change":"2024-02-19T15:24:25.797547+0000","last_active":"2024-02-19T15:24:25.805976+0000","last_peered":"2024-02-19T15:24:25.805976+0000","last_clean":"2024-02-19T15:24:25.805976+0000","last_became_active":"2024-02-19T15:24:25.797250+0000","last_became_peered":"2024-02-19T15:24:25.797250+0000","last_unstale":"2024-02-19T15:24:25.805976+0000","last_undegraded":"2024-02-19T15:24:25.805976+0000","last_fullsized":"2024-02-19T15:24:25.805976+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:37:37.496274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.778961+0000","last_change":"2024-02-19T15:24:24.770314+0000","last_active":"2024-02-19T15:24:25.778961+0000","last_peered":"2024-02-19T15:24:25.778961+0000","last_clean":"2024-02-19T15:24:25.778961+0000","last_became_active":"2024-02-19T15:24:24.770005+0000","last_became_peered":"2024-02-19T15:24:24.770005+0000","last_unstale":"2024-02-19T15:24:25.778961+0000","last_undegraded":"2024-02-19T15:24:25.778961+0000","last_fullsized":"2024-02-19T15:24:25.778961+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:37:35.076825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.800654+0000","last_change":"2024-02-19T15:24:25.788284+0000","last_active":"2024-02-19T15:24:25.800654+0000","last_peered":"2024-02-19T15:24:25.800654+0000","last_clean":"2024-02-19T15:24:25.800654+0000","last_became_active":"2024-02-19T15:24:25.787700+0000","last_became_peered":"2024-02-19T15:24:25.787700+0000","last_unstale":"2024-02-19T15:24:25.800654+0000","last_undegraded":"2024-02-19T15:24:25.800654+0000","last_fullsized":"2024-02-19T15:24:25.800654+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:38:38.879866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.782402+0000","last_change":"2024-02-19T15:24:24.779487+0000","last_active":"2024-02-19T15:24:25.782402+0000","last_peered":"2024-02-19T15:24:25.782402+0000","last_clean":"2024-02-19T15:24:25.782402+0000","last_became_active":"2024-02-19T15:24:24.779229+0000","last_became_peered":"2024-02-19T15:24:24.779229+0000","last_unstale":"2024-02-19T15:24:25.782402+0000","last_undegraded":"2024-02-19T15:24:25.782402+0000","last_fullsized":"2024-02-19T15:24:25.782402+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:57:37.605728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.813337+0000","last_change":"2024-02-19T15:24:25.810469+0000","last_active":"2024-02-19T15:24:25.813337+0000","last_peered":"2024-02-19T15:24:25.813337+0000","last_clean":"2024-02-19T15:24:25.813337+0000","last_became_active":"2024-02-19T15:24:25.810203+0000","last_became_peered":"2024-02-19T15:24:25.810203+0000","last_unstale":"2024-02-19T15:24:25.813337+0000","last_undegraded":"2024-02-19T15:24:25.813337+0000","last_fullsized":"2024-02-19T15:24:25.813337+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:58:47.729061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T15:24:26.794482+0000","last_change":"2024-02-19T15:24:24.774529+0000","last_active":"2024-02-19T15:24:26.794482+0000","last_peered":"2024-02-19T15:24:26.794482+0000","last_clean":"2024-02-19T15:24:26.794482+0000","last_became_active":"2024-02-19T15:24:24.774254+0000","last_became_peered":"2024-02-19T15:24:24.774254+0000","last_unstale":"2024-02-19T15:24:26.794482+0000","last_undegraded":"2024-02-19T15:24:26.794482+0000","last_fullsized":"2024-02-19T15:24:26.794482+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:59:22.650366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":232,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.776343+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:24:25.776343+0000","last_peered":"2024-02-19T15:24:25.776343+0000","last_clean":"2024-02-19T15:24:25.776343+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:24:25.776343+0000","last_undegraded":"2024-02-19T15:24:25.776343+0000","last_fullsized":"2024-02-19T15:24:25.776343+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509474,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":667648,"data_stored":275134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.39700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82199999999999995,"5min":1.022,"15min":1.022},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.748,"15min":0.77500000000000002},"min":{"1min":0.45800000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.85499999999999998,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.83699999999999997}]},{"osd":1,"last update":"Mon Feb 19 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.64600000000000002,"15min":0.65200000000000002},"min":{"1min":0.442,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.82799999999999996,"5min":1.171,"15min":1.171},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.64300000000000002,"15min":0.64800000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.755,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.68600000000000005}]},{"osd":2,"last update":"Mon Feb 19 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.626,"15min":0.63},"min":{"1min":0.45400000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.75,"5min":1.216,"15min":1.216},"last":0.753},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.67800000000000005,"15min":0.68200000000000005},"min":{"1min":0.53100000000000003,"5min":0.495,"15min":0.495},"max":{"1min":0.80900000000000005,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.70199999999999996}]},{"osd":3,"last update":"Mon Feb 19 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.67900000000000005,"15min":0.67500000000000004},"min":{"1min":0.47999999999999998,"5min":0.438,"15min":0.438},"max":{"1min":0.91300000000000003,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.65400000000000003},"min":{"1min":0.46700000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88300000000000001,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.72099999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.60799999999999998,"15min":0.60099999999999998},"min":{"1min":0.40899999999999997,"5min":0.309,"15min":0.309},"max":{"1min":0.86699999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.61299999999999999,"15min":0.59899999999999998},"min":{"1min":0.435,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.58799999999999997}]},{"osd":5,"last update":"Mon Feb 19 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.38400000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.629},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66900000000000004,"15min":0.66500000000000004},"min":{"1min":0.46700000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.879,"5min":1.02,"15min":1.02},"last":0.61499999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.66000000000000003,"15min":0.64800000000000002},"min":{"1min":0.54800000000000004,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.95699999999999996,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.67400000000000004,"15min":0.66600000000000004},"min":{"1min":0.54800000000000004,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.82099999999999995}]}]},{"osd":7,"up_from":14,"seq":60129542178,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32404,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339436,"statfs":{"total":96636764160,"available":96603582464,"internally_reserved":0,"allocated":1249280,"data_stored":858727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.42399999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.90500000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67900000000000005,"15min":0.68100000000000005},"min":{"1min":0.52400000000000002,"5min":0.433,"15min":0.433},"max":{"1min":1.0580000000000001,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.79100000000000004}]},{"osd":1,"last update":"Mon Feb 19 15:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.60699999999999998,"15min":0.59199999999999997},"min":{"1min":0.53600000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.72099999999999997,"15min":0.72699999999999998},"min":{"1min":0.47199999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.90000000000000002,"5min":2.968,"15min":2.968},"last":0.67300000000000004}]},{"osd":2,"last update":"Mon Feb 19 15:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.63500000000000001,"15min":0.63200000000000001},"min":{"1min":0.46200000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.69199999999999995,"15min":0.69299999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.91000000000000003,"5min":2.508,"15min":2.508},"last":0.77300000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.65400000000000003,"15min":0.65600000000000003},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.88300000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.55100000000000005,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.99099999999999999,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":0.73499999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.54000000000000004,"15min":0.52000000000000002},"min":{"1min":0.373,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.95799999999999996,"5min":1.03,"15min":1.03},"last":0.621},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.57199999999999995,"15min":0.56599999999999995},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78400000000000003,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.56799999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.58699999999999997,"15min":0.56100000000000005},"min":{"1min":0.45700000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.754},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.504,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.85099999999999998,"5min":1.046,"15min":1.046},"last":0.66200000000000003}]},{"osd":6,"last update":"Mon Feb 19 15:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.60999999999999999,"15min":0.59299999999999997},"min":{"1min":0.41999999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.67500000000000004,"15min":0.65800000000000003},"min":{"1min":0.499,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.81799999999999995}]}]},{"osd":5,"up_from":14,"seq":60129542179,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":675840,"data_stored":275786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.62,"15min":0.627},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.89000000000000001,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.59499999999999997,"15min":0.59799999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.77600000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.75900000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.63800000000000001,"15min":0.63600000000000001},"min":{"1min":0.47499999999999998,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.85799999999999998,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67700000000000005,"15min":0.67500000000000004},"min":{"1min":0.47199999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.874,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.79200000000000004}]},{"osd":2,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.67700000000000005,"15min":0.68100000000000005},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.89200000000000002,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59199999999999997,"15min":0.59499999999999997},"min":{"1min":0.46800000000000003,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.80200000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.82399999999999995}]},{"osd":3,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.54000000000000004,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82499999999999996,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65200000000000002,"15min":0.65400000000000003},"min":{"1min":0.48299999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.73599999999999999,"5min":1.103,"15min":1.103},"last":0.77400000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56399999999999995,"15min":0.55500000000000005},"min":{"1min":0.35499999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.89200000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64200000000000002,"15min":0.64800000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.876,"5min":1.458,"15min":1.458},"last":0.56399999999999995}]},{"osd":6,"last update":"Mon Feb 19 15:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.64100000000000001,"15min":0.63500000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93899999999999995,"5min":1.004,"15min":1.004},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.72299999999999998,"15min":0.71699999999999997},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.97799999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.67000000000000004}]},{"osd":7,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.82499999999999996,"5min":1.038,"15min":1.038},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.68799999999999994,"15min":0.68500000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88600000000000001,"5min":1.131,"15min":1.131},"last":0.58099999999999996}]}]},{"osd":4,"up_from":14,"seq":60129542177,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":655360,"data_stored":268359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.34399999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76300000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.437},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.60799999999999998,"15min":0.59999999999999998},"min":{"1min":0.47299999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.84499999999999997,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.54900000000000004}]},{"osd":1,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.622,"15min":0.61299999999999999},"min":{"1min":0.495,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86899999999999999,"5min":1.081,"15min":1.081},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.63600000000000001,"15min":0.63100000000000001},"min":{"1min":0.441,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86199999999999999,"5min":1.008,"15min":1.008},"last":0.52200000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.66200000000000003,"15min":0.66300000000000003},"min":{"1min":0.45800000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.91100000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.67800000000000005,"15min":0.67500000000000004},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.96299999999999997,"5min":1.002,"15min":1.002},"last":0.63700000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.68000000000000005,"15min":0.67300000000000004},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65300000000000002,"15min":0.65100000000000002},"min":{"1min":0.47199999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.90000000000000002,"5min":1.153,"15min":1.153},"last":0.59799999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.38200000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.98199999999999998,"5min":1.042,"15min":1.042},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.68899999999999995,"15min":0.67700000000000005},"min":{"1min":0.54300000000000004,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.079,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.57399999999999995}]},{"osd":6,"last update":"Mon Feb 19 15:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.41399999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.65800000000000003,"15min":0.64600000000000002},"min":{"1min":0.45400000000000001,"5min":0.438,"15min":0.438},"max":{"1min":1.0249999999999999,"5min":1.054,"15min":1.054},"last":0.61499999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.67500000000000004,"15min":0.66200000000000003},"min":{"1min":0.57099999999999995,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.96399999999999997,"5min":1.103,"15min":1.103},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.73699999999999999,"15min":0.72699999999999998},"min":{"1min":0.56000000000000005,"5min":0.443,"15min":0.443},"max":{"1min":1.048,"5min":1.24,"15min":1.24},"last":0.747}]}]},{"osd":3,"up_from":14,"seq":60129542177,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":675840,"data_stored":275190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.45000000000000001,"15min":0.45400000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.80400000000000005,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.64300000000000002,"15min":0.68000000000000005},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.56100000000000005,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.34999999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.438,"15min":0.44400000000000001},"min":{"1min":0.313,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.58999999999999997,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.312},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.70899999999999996,"15min":0.747},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.79600000000000004,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.46700000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.61299999999999999,"15min":0.627},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.88100000000000001,"5min":1.008,"15min":1.008},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.58399999999999996,"15min":0.59999999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72899999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.38400000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.68300000000000005,"15min":0.67200000000000004},"min":{"1min":0.57099999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.85899999999999999,"15min":0.88},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.96099999999999997,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.85699999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.85699999999999998,"15min":0.873},"min":{"1min":0.54500000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.95299999999999996,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.748,"15min":0.75},"min":{"1min":0.53500000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.95299999999999996,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.89300000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.89200000000000002,"15min":0.91200000000000003},"min":{"1min":0.58699999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.99299999999999999,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.80700000000000005,"15min":0.82099999999999995},"min":{"1min":0.52800000000000002,"5min":0.502,"15min":0.502},"max":{"1min":0.94299999999999995,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.873}]},{"osd":7,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78200000000000003,"15min":0.78600000000000003},"min":{"1min":0.56299999999999994,"5min":0.504,"15min":0.504},"max":{"1min":1.026,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.92000000000000004,"15min":0.94599999999999995},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.93000000000000005,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.82399999999999995}]}]},{"osd":2,"up_from":14,"seq":60129542178,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31824,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":655360,"data_stored":268359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":0.80100000000000005,"15min":0.74099999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":15.42,"5min":15.42,"15min":15.42},"last":0.44},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.92800000000000005,"15min":0.95599999999999996},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":6.7000000000000002,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.35399999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.96299999999999997,"15min":0.96599999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":9.5269999999999992,"5min":9.5269999999999992,"15min":9.5269999999999992},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.1559999999999999,"15min":1.0780000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":22.954999999999998,"5min":22.954999999999998,"15min":22.954999999999998},"last":0.47399999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.80500000000000005,"15min":0.82799999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":3.7799999999999998,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.873,"15min":0.86599999999999999},"min":{"1min":0.443,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":7.734,"5min":7.734,"15min":7.734},"last":0.48999999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.0069999999999999,"15min":0.93400000000000005},"min":{"1min":0.54900000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":15.369,"5min":15.369,"15min":15.369},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.92300000000000004,"15min":0.92500000000000004},"min":{"1min":0.61799999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":3.6819999999999999,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.95199999999999996}]},{"osd":5,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":0.94199999999999995,"15min":0.91200000000000003},"min":{"1min":0.58399999999999996,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":7.6520000000000001,"5min":7.6520000000000001,"15min":7.6520000000000001},"last":1.135},{"interface":"front","average":{"1min":1.526,"5min":1.0800000000000001,"15min":1.006},"min":{"1min":0.57999999999999996,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":15.391,"5min":15.391,"15min":15.391},"last":0.71999999999999997}]},{"osd":6,"last update":"Mon Feb 19 15:23:57 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.0449999999999999,"15min":0.97299999999999998},"min":{"1min":0.57799999999999996,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":15.337,"5min":15.337,"15min":15.337},"last":1.002},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.98099999999999998,"15min":0.99299999999999999},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":3.7450000000000001,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":1.1539999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.006,"15min":1.014},"min":{"1min":0.65000000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":3.71,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":1.073},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.006,"15min":0.98299999999999998},"min":{"1min":0.57499999999999996,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":7.6929999999999996,"5min":7.6929999999999996,"15min":7.6929999999999996},"last":1.0309999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31824,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":655360,"data_stored":268359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.42699999999999999,"15min":0.435},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.60499999999999998,"5min":0.747,"15min":0.747},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.46800000000000003,"15min":0.47599999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.442,"15min":0.45000000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.55000000000000004,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.52300000000000002,"15min":0.53800000000000003},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.55800000000000005,"5min":1.071,"15min":1.071},"last":0.623}]},{"osd":3,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.63200000000000001,"15min":0.64800000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.76000000000000001,"5min":1.365,"15min":1.365},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.63,"15min":0.64900000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76200000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.57599999999999996}]},{"osd":4,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.61599999999999999,"15min":0.59999999999999998},"min":{"1min":0.48199999999999998,"5min":0.377,"15min":0.377},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.68100000000000005,"15min":0.67600000000000005},"min":{"1min":0.505,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.0109999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.98899999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.69799999999999995,"15min":0.69399999999999995},"min":{"1min":0.56499999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0509999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.68200000000000005,"15min":0.67800000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.92000000000000004,"5min":1.107,"15min":1.107},"last":1.0860000000000001}]},{"osd":6,"last update":"Mon Feb 19 15:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.73099999999999998,"15min":0.72899999999999998},"min":{"1min":0.54900000000000004,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0649999999999999,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.58599999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0840000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":1.0669999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.78900000000000003,"15min":0.79300000000000004},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.1120000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":1.101},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.748,"15min":0.74199999999999999},"min":{"1min":0.59799999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.032,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":1.032}]}]},{"osd":0,"up_from":14,"seq":60129542177,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36564,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335276,"statfs":{"total":96636764160,"available":96599322624,"internally_reserved":0,"allocated":1249280,"data_stored":858727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.626,"15min":0.66800000000000004},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57399999999999995,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.48399999999999999,"15min":0.48999999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73899999999999999,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.40500000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.78200000000000003,"15min":0.82599999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.72399999999999998,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.74399999999999999,"15min":0.78800000000000003},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.70499999999999996,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.53000000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.67700000000000005,"15min":0.71499999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.752,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.60799999999999998,"15min":0.63},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.77100000000000002,"5min":1.345,"15min":1.345},"last":0.46999999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.745,"15min":0.74299999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.94699999999999995,"5min":1.095,"15min":1.095},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.92100000000000004,"15min":0.95099999999999996},"min":{"1min":0.59599999999999997,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.89500000000000002,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.81599999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.751,"15min":0.755},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.85199999999999998,"15min":0.88},"min":{"1min":0.53200000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.96099999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.86299999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.91200000000000003,"15min":0.94599999999999995},"min":{"1min":0.53600000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.93100000000000005,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.75800000000000001,"15min":0.76400000000000001},"min":{"1min":0.54600000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.873,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.76900000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.94299999999999995,"15min":0.97299999999999998},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.2450000000000001,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.93799999999999994,"15min":0.97599999999999998},"min":{"1min":0.56200000000000006,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.91500000000000004,"5min":4.915,"15min":4.915},"last":0.89400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:24:30.090 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:24:30.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.262+0000 7fac43000640 1 Processor -- start 2024-02-19T15:24:30.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.262+0000 7fac43000640 1 -- start start 2024-02-19T15:24:30.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.262+0000 7fac43000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fac3c0b1be0 0x7fac3c0d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:30.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.262+0000 7fac43000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3c0b0280 con 0x7fac3c15f590 2024-02-19T15:24:30.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.266+0000 7fac43000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3c0b04d0 con 0x7fac3c0b0a80 2024-02-19T15:24:30.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.266+0000 7fac43000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac3c0cc530 con 0x7fac3c0b1be0 2024-02-19T15:24:30.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.266+0000 7fac3b400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fac3c0b1be0 0x7fac3c0d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:30.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.266+0000 7fac3be00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fac3c15f590 0x7fac3c0b14d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35970/0 (socket says 172.21.15.47:35970) 2024-02-19T15:24:30.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.266+0000 7fac3be00640 1 -- 172.21.15.47:0/295707341 learned_addr learned my addr 172.21.15.47:0/295707341 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:30.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.266+0000 7fac3b400640 1 -- 172.21.15.47:0/295707341 >> v1:172.21.15.116:6789/0 conn(0x7fac3c0b0a80 legacy=0x7fac3c0b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:30.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.266+0000 7fac3b400640 1 -- 172.21.15.47:0/295707341 >> v1:172.21.15.47:6789/0 conn(0x7fac3c15f590 legacy=0x7fac3c0b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:30.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.266+0000 7fac3b400640 1 -- 172.21.15.47:0/295707341 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c0b07b0 con 0x7fac3c0b1be0 2024-02-19T15:24:30.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.266+0000 7fac3aa00640 1 -- 172.21.15.47:0/295707341 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171825611 0 0) 0x7fac3c0b0280 con 0x7fac3c15f590 2024-02-19T15:24:30.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.266+0000 7fac3b400640 1 --2- 172.21.15.47:0/295707341 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac3c0b1be0 0x7fac3c0d2690 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fac280097c0 tx=0x7fac2801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c38f0af3b9f71df8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:30.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.266+0000 7fac3aa00640 1 -- 172.21.15.47:0/295707341 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac280ab670 con 0x7fac3c0b1be0 2024-02-19T15:24:30.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac3aa00640 1 -- 172.21.15.47:0/295707341 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac280ab7d0 con 0x7fac3c0b1be0 2024-02-19T15:24:30.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac3aa00640 1 -- 172.21.15.47:0/295707341 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac280abb60 con 0x7fac3c0b1be0 2024-02-19T15:24:30.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac43000640 1 -- 172.21.15.47:0/295707341 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac3c0b1be0 msgr2=0x7fac3c0d2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:30.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac43000640 1 --2- 172.21.15.47:0/295707341 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac3c0b1be0 0x7fac3c0d2690 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fac280097c0 tx=0x7fac2801c470 comp rx=0 tx=0).stop 2024-02-19T15:24:30.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac43000640 1 -- 172.21.15.47:0/295707341 shutdown_connections 2024-02-19T15:24:30.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac43000640 1 --2- 172.21.15.47:0/295707341 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac3c0b1be0 0x7fac3c0d2690 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac43000640 1 -- 172.21.15.47:0/295707341 >> 172.21.15.47:0/295707341 conn(0x7fac3c0d1b70 msgr2=0x7fac3c0d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:30.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac43000640 1 -- 172.21.15.47:0/295707341 shutdown_connections 2024-02-19T15:24:30.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac43000640 1 -- 172.21.15.47:0/295707341 wait complete. 2024-02-19T15:24:30.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac43000640 1 Processor -- start 2024-02-19T15:24:30.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac43000640 1 -- start start 2024-02-19T15:24:30.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac43000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac3c0b0a80 0x7fac3c15ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:30.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac43000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac3c0b1be0 0x7fac3c1539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:30.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac43000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac3c15f590 0x7fac3c153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:30.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac43000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3c15abb0 con 0x7fac3c0b0a80 2024-02-19T15:24:30.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac43000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3c156880 con 0x7fac3c0b1be0 2024-02-19T15:24:30.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac43000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac3c154460 con 0x7fac3c15f590 2024-02-19T15:24:30.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac3be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac3c0b0a80 0x7fac3c15ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:30.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac3be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac3c0b0a80 0x7fac3c15ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51016/0 (socket says 172.21.15.47:51016) 2024-02-19T15:24:30.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac3be00640 1 -- 172.21.15.47:0/3970423189 learned_addr learned my addr 172.21.15.47:0/3970423189 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:30.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac3b400640 1 --2- 172.21.15.47:0/3970423189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac3c0b1be0 0x7fac3c1539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:30.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.270+0000 7fac41000640 1 --2- 172.21.15.47:0/3970423189 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac3c15f590 0x7fac3c153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:30.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.274+0000 7fac3be00640 1 -- 172.21.15.47:0/3970423189 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac3c15f590 msgr2=0x7fac3c153f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:30.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.274+0000 7fac3be00640 1 --2- 172.21.15.47:0/3970423189 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac3c15f590 0x7fac3c153f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.274+0000 7fac3be00640 1 -- 172.21.15.47:0/3970423189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac3c0b1be0 msgr2=0x7fac3c1539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:30.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.274+0000 7fac3be00640 1 --2- 172.21.15.47:0/3970423189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac3c0b1be0 0x7fac3c1539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.274+0000 7fac3be00640 1 -- 172.21.15.47:0/3970423189 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac28009420 con 0x7fac3c0b0a80 2024-02-19T15:24:30.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.274+0000 7fac41000640 1 --2- 172.21.15.47:0/3970423189 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac3c15f590 0x7fac3c153f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:30.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.274+0000 7fac3be00640 1 --2- 172.21.15.47:0/3970423189 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac3c0b0a80 0x7fac3c15ba50 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fac2c001ec0 tx=0x7fac2c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:30.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.274+0000 7fac33e00640 1 -- 172.21.15.47:0/3970423189 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c005cf0 con 0x7fac3c0b0a80 2024-02-19T15:24:30.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.274+0000 7fac33e00640 1 -- 172.21.15.47:0/3970423189 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2c005e70 con 0x7fac3c0b0a80 2024-02-19T15:24:30.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.274+0000 7fac43000640 1 -- 172.21.15.47:0/3970423189 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac3c15b070 con 0x7fac3c0b0a80 2024-02-19T15:24:30.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.274+0000 7fac33e00640 1 -- 172.21.15.47:0/3970423189 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0072d0 con 0x7fac3c0b0a80 2024-02-19T15:24:30.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.274+0000 7fac43000640 1 -- 172.21.15.47:0/3970423189 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac3c15ad90 con 0x7fac3c0b0a80 2024-02-19T15:24:30.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.274+0000 7fac43000640 1 -- 172.21.15.47:0/3970423189 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac3c1593c0 con 0x7fac3c0b0a80 2024-02-19T15:24:30.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.274+0000 7fac33e00640 1 -- 172.21.15.47:0/3970423189 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fac2c007430 con 0x7fac3c0b0a80 2024-02-19T15:24:30.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.274+0000 7fac33e00640 1 --2- 172.21.15.47:0/3970423189 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fac1c066e40 0x7fac1c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:30.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.274+0000 7fac33e00640 1 -- 172.21.15.47:0/3970423189 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac2c084050 con 0x7fac3c0b0a80 2024-02-19T15:24:30.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.274+0000 7fac3b400640 1 --2- 172.21.15.47:0/3970423189 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fac1c066e40 0x7fac1c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:30.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.278+0000 7fac3b400640 1 --2- 172.21.15.47:0/3970423189 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fac1c066e40 0x7fac1c069300 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fac280af000 tx=0x7fac280a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:30.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.278+0000 7fac33e00640 1 -- 172.21.15.47:0/3970423189 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fac2c0b61c0 con 0x7fac3c0b0a80 2024-02-19T15:24:30.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.442+0000 7fac43000640 1 -- 172.21.15.47:0/3970423189 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fac3c15f970 con 0x7fac1c066e40 2024-02-19T15:24:30.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.450+0000 7fac33e00640 1 -- 172.21.15.47:0/3970423189 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288881 (secure 0 0 0) 0x7fac280aba40 con 0x7fac1c066e40 2024-02-19T15:24:30.455 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:30.456 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:24:30.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.454+0000 7fac43000640 1 -- 172.21.15.47:0/3970423189 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fac1c066e40 msgr2=0x7fac1c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:30.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.454+0000 7fac43000640 1 --2- 172.21.15.47:0/3970423189 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fac1c066e40 0x7fac1c069300 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fac280af000 tx=0x7fac280a7000 comp rx=0 tx=0).stop 2024-02-19T15:24:30.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.454+0000 7fac43000640 1 -- 172.21.15.47:0/3970423189 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac3c0b0a80 msgr2=0x7fac3c15ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:30.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.454+0000 7fac43000640 1 --2- 172.21.15.47:0/3970423189 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac3c0b0a80 0x7fac3c15ba50 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fac2c001ec0 tx=0x7fac2c00c8e0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.454+0000 7fac43000640 1 -- 172.21.15.47:0/3970423189 shutdown_connections 2024-02-19T15:24:30.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.454+0000 7fac43000640 1 --2- 172.21.15.47:0/3970423189 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fac1c066e40 0x7fac1c069300 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.454+0000 7fac43000640 1 --2- 172.21.15.47:0/3970423189 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac3c15f590 0x7fac3c153f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.454+0000 7fac43000640 1 --2- 172.21.15.47:0/3970423189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac3c0b1be0 0x7fac3c1539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.454+0000 7fac43000640 1 --2- 172.21.15.47:0/3970423189 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac3c0b0a80 0x7fac3c15ba50 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.454+0000 7fac43000640 1 -- 172.21.15.47:0/3970423189 >> 172.21.15.47:0/3970423189 conn(0x7fac3c0d1b70 msgr2=0x7fac3c0d1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:30.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.454+0000 7fac43000640 1 -- 172.21.15.47:0/3970423189 shutdown_connections 2024-02-19T15:24:30.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.454+0000 7fac43000640 1 -- 172.21.15.47:0/3970423189 wait complete. 2024-02-19T15:24:30.484 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-02-19T15:24:29.180006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":103,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255800,"kb_used_data":6332,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718920,"statfs":{"total":773094113280,"available":772832174080,"internally_reserved":0,"allocated":6483968,"data_stored":3348641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.422556"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.797928+0000","last_change":"2024-02-19T15:24:25.793151+0000","last_active":"2024-02-19T15:24:25.797928+0000","last_peered":"2024-02-19T15:24:25.797928+0000","last_clean":"2024-02-19T15:24:25.797928+0000","last_became_active":"2024-02-19T15:24:25.791062+0000","last_became_peered":"2024-02-19T15:24:25.791062+0000","last_unstale":"2024-02-19T15:24:25.797928+0000","last_undegraded":"2024-02-19T15:24:25.797928+0000","last_fullsized":"2024-02-19T15:24:25.797928+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:39:55.119850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.813902+0000","last_change":"2024-02-19T15:24:25.811654+0000","last_active":"2024-02-19T15:24:25.813902+0000","last_peered":"2024-02-19T15:24:25.813902+0000","last_clean":"2024-02-19T15:24:25.813902+0000","last_became_active":"2024-02-19T15:24:25.807942+0000","last_became_peered":"2024-02-19T15:24:25.807942+0000","last_unstale":"2024-02-19T15:24:25.813902+0000","last_undegraded":"2024-02-19T15:24:25.813902+0000","last_fullsized":"2024-02-19T15:24:25.813902+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:58:20.513106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.798429+0000","last_change":"2024-02-19T15:24:25.792593+0000","last_active":"2024-02-19T15:24:25.798429+0000","last_peered":"2024-02-19T15:24:25.798429+0000","last_clean":"2024-02-19T15:24:25.798429+0000","last_became_active":"2024-02-19T15:24:25.789965+0000","last_became_peered":"2024-02-19T15:24:25.789965+0000","last_unstale":"2024-02-19T15:24:25.798429+0000","last_undegraded":"2024-02-19T15:24:25.798429+0000","last_fullsized":"2024-02-19T15:24:25.798429+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:47:28.890973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.800641+0000","last_change":"2024-02-19T15:24:25.795963+0000","last_active":"2024-02-19T15:24:25.800641+0000","last_peered":"2024-02-19T15:24:25.800641+0000","last_clean":"2024-02-19T15:24:25.800641+0000","last_became_active":"2024-02-19T15:24:25.795628+0000","last_became_peered":"2024-02-19T15:24:25.795628+0000","last_unstale":"2024-02-19T15:24:25.800641+0000","last_undegraded":"2024-02-19T15:24:25.800641+0000","last_fullsized":"2024-02-19T15:24:25.800641+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:08:51.040190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.797819+0000","last_change":"2024-02-19T15:24:25.790397+0000","last_active":"2024-02-19T15:24:25.797819+0000","last_peered":"2024-02-19T15:24:25.797819+0000","last_clean":"2024-02-19T15:24:25.797819+0000","last_became_active":"2024-02-19T15:24:25.790119+0000","last_became_peered":"2024-02-19T15:24:25.790119+0000","last_unstale":"2024-02-19T15:24:25.797819+0000","last_undegraded":"2024-02-19T15:24:25.797819+0000","last_fullsized":"2024-02-19T15:24:25.797819+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:58:36.417120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.813688+0000","last_change":"2024-02-19T15:24:25.804365+0000","last_active":"2024-02-19T15:24:25.813688+0000","last_peered":"2024-02-19T15:24:25.813688+0000","last_clean":"2024-02-19T15:24:25.813688+0000","last_became_active":"2024-02-19T15:24:25.804127+0000","last_became_peered":"2024-02-19T15:24:25.804127+0000","last_unstale":"2024-02-19T15:24:25.813688+0000","last_undegraded":"2024-02-19T15:24:25.813688+0000","last_fullsized":"2024-02-19T15:24:25.813688+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:59:58.667597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.796206+0000","last_change":"2024-02-19T15:24:25.791344+0000","last_active":"2024-02-19T15:24:25.796206+0000","last_peered":"2024-02-19T15:24:25.796206+0000","last_clean":"2024-02-19T15:24:25.796206+0000","last_became_active":"2024-02-19T15:24:25.790033+0000","last_became_peered":"2024-02-19T15:24:25.790033+0000","last_unstale":"2024-02-19T15:24:25.796206+0000","last_undegraded":"2024-02-19T15:24:25.796206+0000","last_fullsized":"2024-02-19T15:24:25.796206+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:00:26.457771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.801267+0000","last_change":"2024-02-19T15:24:25.792062+0000","last_active":"2024-02-19T15:24:25.801267+0000","last_peered":"2024-02-19T15:24:25.801267+0000","last_clean":"2024-02-19T15:24:25.801267+0000","last_became_active":"2024-02-19T15:24:25.791620+0000","last_became_peered":"2024-02-19T15:24:25.791620+0000","last_unstale":"2024-02-19T15:24:25.801267+0000","last_undegraded":"2024-02-19T15:24:25.801267+0000","last_fullsized":"2024-02-19T15:24:25.801267+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:58:44.811899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.794654+0000","last_change":"2024-02-19T15:24:25.790905+0000","last_active":"2024-02-19T15:24:25.794654+0000","last_peered":"2024-02-19T15:24:25.794654+0000","last_clean":"2024-02-19T15:24:25.794654+0000","last_became_active":"2024-02-19T15:24:25.790620+0000","last_became_peered":"2024-02-19T15:24:25.790620+0000","last_unstale":"2024-02-19T15:24:25.794654+0000","last_undegraded":"2024-02-19T15:24:25.794654+0000","last_fullsized":"2024-02-19T15:24:25.794654+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:46.797100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.808563+0000","last_change":"2024-02-19T15:24:25.800569+0000","last_active":"2024-02-19T15:24:25.808563+0000","last_peered":"2024-02-19T15:24:25.808563+0000","last_clean":"2024-02-19T15:24:25.808563+0000","last_became_active":"2024-02-19T15:24:25.800265+0000","last_became_peered":"2024-02-19T15:24:25.800265+0000","last_unstale":"2024-02-19T15:24:25.808563+0000","last_undegraded":"2024-02-19T15:24:25.808563+0000","last_fullsized":"2024-02-19T15:24:25.808563+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:00:26.457771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.804749+0000","last_change":"2024-02-19T15:24:25.797257+0000","last_active":"2024-02-19T15:24:25.804749+0000","last_peered":"2024-02-19T15:24:25.804749+0000","last_clean":"2024-02-19T15:24:25.804749+0000","last_became_active":"2024-02-19T15:24:25.796935+0000","last_became_peered":"2024-02-19T15:24:25.796935+0000","last_unstale":"2024-02-19T15:24:25.804749+0000","last_undegraded":"2024-02-19T15:24:25.804749+0000","last_fullsized":"2024-02-19T15:24:25.804749+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:31:30.242430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.795280+0000","last_change":"2024-02-19T15:24:25.787718+0000","last_active":"2024-02-19T15:24:25.795280+0000","last_peered":"2024-02-19T15:24:25.795280+0000","last_clean":"2024-02-19T15:24:25.795280+0000","last_became_active":"2024-02-19T15:24:25.787400+0000","last_became_peered":"2024-02-19T15:24:25.787400+0000","last_unstale":"2024-02-19T15:24:25.795280+0000","last_undegraded":"2024-02-19T15:24:25.795280+0000","last_fullsized":"2024-02-19T15:24:25.795280+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:23:42.984554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T15:24:26.794754+0000","last_change":"2024-02-19T15:24:25.805528+0000","last_active":"2024-02-19T15:24:26.794754+0000","last_peered":"2024-02-19T15:24:26.794754+0000","last_clean":"2024-02-19T15:24:26.794754+0000","last_became_active":"2024-02-19T15:24:25.805029+0000","last_became_peered":"2024-02-19T15:24:25.805029+0000","last_unstale":"2024-02-19T15:24:26.794754+0000","last_undegraded":"2024-02-19T15:24:26.794754+0000","last_fullsized":"2024-02-19T15:24:26.794754+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:15:18.548657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.796496+0000","last_change":"2024-02-19T15:24:25.793791+0000","last_active":"2024-02-19T15:24:25.796496+0000","last_peered":"2024-02-19T15:24:25.796496+0000","last_clean":"2024-02-19T15:24:25.796496+0000","last_became_active":"2024-02-19T15:24:25.793409+0000","last_became_peered":"2024-02-19T15:24:25.793409+0000","last_unstale":"2024-02-19T15:24:25.796496+0000","last_undegraded":"2024-02-19T15:24:25.796496+0000","last_fullsized":"2024-02-19T15:24:25.796496+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:32:31.195399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.800599+0000","last_change":"2024-02-19T15:24:25.797765+0000","last_active":"2024-02-19T15:24:25.800599+0000","last_peered":"2024-02-19T15:24:25.800599+0000","last_clean":"2024-02-19T15:24:25.800599+0000","last_became_active":"2024-02-19T15:24:25.797156+0000","last_became_peered":"2024-02-19T15:24:25.797156+0000","last_unstale":"2024-02-19T15:24:25.800599+0000","last_undegraded":"2024-02-19T15:24:25.800599+0000","last_fullsized":"2024-02-19T15:24:25.800599+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:24:05.884765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.801049+0000","last_change":"2024-02-19T15:24:25.789116+0000","last_active":"2024-02-19T15:24:25.801049+0000","last_peered":"2024-02-19T15:24:25.801049+0000","last_clean":"2024-02-19T15:24:25.801049+0000","last_became_active":"2024-02-19T15:24:25.788273+0000","last_became_peered":"2024-02-19T15:24:25.788273+0000","last_unstale":"2024-02-19T15:24:25.801049+0000","last_undegraded":"2024-02-19T15:24:25.801049+0000","last_fullsized":"2024-02-19T15:24:25.801049+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:22:58.227504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.801322+0000","last_change":"2024-02-19T15:24:25.791269+0000","last_active":"2024-02-19T15:24:25.801322+0000","last_peered":"2024-02-19T15:24:25.801322+0000","last_clean":"2024-02-19T15:24:25.801322+0000","last_became_active":"2024-02-19T15:24:25.790948+0000","last_became_peered":"2024-02-19T15:24:25.790948+0000","last_unstale":"2024-02-19T15:24:25.801322+0000","last_undegraded":"2024-02-19T15:24:25.801322+0000","last_fullsized":"2024-02-19T15:24:25.801322+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:15:50.743494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.799984+0000","last_change":"2024-02-19T15:24:25.793869+0000","last_active":"2024-02-19T15:24:25.799984+0000","last_peered":"2024-02-19T15:24:25.799984+0000","last_clean":"2024-02-19T15:24:25.799984+0000","last_became_active":"2024-02-19T15:24:25.793458+0000","last_became_peered":"2024-02-19T15:24:25.793458+0000","last_unstale":"2024-02-19T15:24:25.799984+0000","last_undegraded":"2024-02-19T15:24:25.799984+0000","last_fullsized":"2024-02-19T15:24:25.799984+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:04:44.101587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.798658+0000","last_change":"2024-02-19T15:24:25.795041+0000","last_active":"2024-02-19T15:24:25.798658+0000","last_peered":"2024-02-19T15:24:25.798658+0000","last_clean":"2024-02-19T15:24:25.798658+0000","last_became_active":"2024-02-19T15:24:25.794820+0000","last_became_peered":"2024-02-19T15:24:25.794820+0000","last_unstale":"2024-02-19T15:24:25.798658+0000","last_undegraded":"2024-02-19T15:24:25.798658+0000","last_fullsized":"2024-02-19T15:24:25.798658+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:58:04.244227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.804873+0000","last_change":"2024-02-19T15:24:25.802212+0000","last_active":"2024-02-19T15:24:25.804873+0000","last_peered":"2024-02-19T15:24:25.804873+0000","last_clean":"2024-02-19T15:24:25.804873+0000","last_became_active":"2024-02-19T15:24:25.801956+0000","last_became_peered":"2024-02-19T15:24:25.801956+0000","last_unstale":"2024-02-19T15:24:25.804873+0000","last_undegraded":"2024-02-19T15:24:25.804873+0000","last_fullsized":"2024-02-19T15:24:25.804873+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:48:02.565569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.789657+0000","last_change":"2024-02-19T15:24:25.786159+0000","last_active":"2024-02-19T15:24:25.789657+0000","last_peered":"2024-02-19T15:24:25.789657+0000","last_clean":"2024-02-19T15:24:25.789657+0000","last_became_active":"2024-02-19T15:24:25.785806+0000","last_became_peered":"2024-02-19T15:24:25.785806+0000","last_unstale":"2024-02-19T15:24:25.789657+0000","last_undegraded":"2024-02-19T15:24:25.789657+0000","last_fullsized":"2024-02-19T15:24:25.789657+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:40:41.107930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.801353+0000","last_change":"2024-02-19T15:24:25.794415+0000","last_active":"2024-02-19T15:24:25.801353+0000","last_peered":"2024-02-19T15:24:25.801353+0000","last_clean":"2024-02-19T15:24:25.801353+0000","last_became_active":"2024-02-19T15:24:25.794092+0000","last_became_peered":"2024-02-19T15:24:25.794092+0000","last_unstale":"2024-02-19T15:24:25.801353+0000","last_undegraded":"2024-02-19T15:24:25.801353+0000","last_fullsized":"2024-02-19T15:24:25.801353+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:07:22.816266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.800029+0000","last_change":"2024-02-19T15:24:25.792140+0000","last_active":"2024-02-19T15:24:25.800029+0000","last_peered":"2024-02-19T15:24:25.800029+0000","last_clean":"2024-02-19T15:24:25.800029+0000","last_became_active":"2024-02-19T15:24:25.789775+0000","last_became_peered":"2024-02-19T15:24:25.789775+0000","last_unstale":"2024-02-19T15:24:25.800029+0000","last_undegraded":"2024-02-19T15:24:25.800029+0000","last_fullsized":"2024-02-19T15:24:25.800029+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:57:38.611541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T15:24:26.794669+0000","last_change":"2024-02-19T15:24:25.793174+0000","last_active":"2024-02-19T15:24:26.794669+0000","last_peered":"2024-02-19T15:24:26.794669+0000","last_clean":"2024-02-19T15:24:26.794669+0000","last_became_active":"2024-02-19T15:24:25.792851+0000","last_became_peered":"2024-02-19T15:24:25.792851+0000","last_unstale":"2024-02-19T15:24:26.794669+0000","last_undegraded":"2024-02-19T15:24:26.794669+0000","last_fullsized":"2024-02-19T15:24:26.794669+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:04:14.340364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.802614+0000","last_change":"2024-02-19T15:24:25.799850+0000","last_active":"2024-02-19T15:24:25.802614+0000","last_peered":"2024-02-19T15:24:25.802614+0000","last_clean":"2024-02-19T15:24:25.802614+0000","last_became_active":"2024-02-19T15:24:25.799557+0000","last_became_peered":"2024-02-19T15:24:25.799557+0000","last_unstale":"2024-02-19T15:24:25.802614+0000","last_undegraded":"2024-02-19T15:24:25.802614+0000","last_fullsized":"2024-02-19T15:24:25.802614+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:18:50.470518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.794458+0000","last_change":"2024-02-19T15:24:25.792532+0000","last_active":"2024-02-19T15:24:25.794458+0000","last_peered":"2024-02-19T15:24:25.794458+0000","last_clean":"2024-02-19T15:24:25.794458+0000","last_became_active":"2024-02-19T15:24:25.789010+0000","last_became_peered":"2024-02-19T15:24:25.789010+0000","last_unstale":"2024-02-19T15:24:25.794458+0000","last_undegraded":"2024-02-19T15:24:25.794458+0000","last_fullsized":"2024-02-19T15:24:25.794458+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:03:45.431655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.799743+0000","last_change":"2024-02-19T15:24:25.793770+0000","last_active":"2024-02-19T15:24:25.799743+0000","last_peered":"2024-02-19T15:24:25.799743+0000","last_clean":"2024-02-19T15:24:25.799743+0000","last_became_active":"2024-02-19T15:24:25.792785+0000","last_became_peered":"2024-02-19T15:24:25.792785+0000","last_unstale":"2024-02-19T15:24:25.799743+0000","last_undegraded":"2024-02-19T15:24:25.799743+0000","last_fullsized":"2024-02-19T15:24:25.799743+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:36:43.119696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.797963+0000","last_change":"2024-02-19T15:24:25.791810+0000","last_active":"2024-02-19T15:24:25.797963+0000","last_peered":"2024-02-19T15:24:25.797963+0000","last_clean":"2024-02-19T15:24:25.797963+0000","last_became_active":"2024-02-19T15:24:25.791504+0000","last_became_peered":"2024-02-19T15:24:25.791504+0000","last_unstale":"2024-02-19T15:24:25.797963+0000","last_undegraded":"2024-02-19T15:24:25.797963+0000","last_fullsized":"2024-02-19T15:24:25.797963+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:43.840049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.803858+0000","last_change":"2024-02-19T15:24:25.798676+0000","last_active":"2024-02-19T15:24:25.803858+0000","last_peered":"2024-02-19T15:24:25.803858+0000","last_clean":"2024-02-19T15:24:25.803858+0000","last_became_active":"2024-02-19T15:24:25.798316+0000","last_became_peered":"2024-02-19T15:24:25.798316+0000","last_unstale":"2024-02-19T15:24:25.803858+0000","last_undegraded":"2024-02-19T15:24:25.803858+0000","last_fullsized":"2024-02-19T15:24:25.803858+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:28:31.257699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.797954+0000","last_change":"2024-02-19T15:24:25.791923+0000","last_active":"2024-02-19T15:24:25.797954+0000","last_peered":"2024-02-19T15:24:25.797954+0000","last_clean":"2024-02-19T15:24:25.797954+0000","last_became_active":"2024-02-19T15:24:25.790030+0000","last_became_peered":"2024-02-19T15:24:25.790030+0000","last_unstale":"2024-02-19T15:24:25.797954+0000","last_undegraded":"2024-02-19T15:24:25.797954+0000","last_fullsized":"2024-02-19T15:24:25.797954+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:53:52.438163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.808232+0000","last_change":"2024-02-19T15:24:25.802237+0000","last_active":"2024-02-19T15:24:25.808232+0000","last_peered":"2024-02-19T15:24:25.808232+0000","last_clean":"2024-02-19T15:24:25.808232+0000","last_became_active":"2024-02-19T15:24:25.801961+0000","last_became_peered":"2024-02-19T15:24:25.801961+0000","last_unstale":"2024-02-19T15:24:25.808232+0000","last_undegraded":"2024-02-19T15:24:25.808232+0000","last_fullsized":"2024-02-19T15:24:25.808232+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:23:55.685803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.804920+0000","last_change":"2024-02-19T15:24:25.799780+0000","last_active":"2024-02-19T15:24:25.804920+0000","last_peered":"2024-02-19T15:24:25.804920+0000","last_clean":"2024-02-19T15:24:25.804920+0000","last_became_active":"2024-02-19T15:24:25.792312+0000","last_became_peered":"2024-02-19T15:24:25.792312+0000","last_unstale":"2024-02-19T15:24:25.804920+0000","last_undegraded":"2024-02-19T15:24:25.804920+0000","last_fullsized":"2024-02-19T15:24:25.804920+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:54:08.004283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.800924+0000","last_change":"2024-02-19T15:24:25.792493+0000","last_active":"2024-02-19T15:24:25.800924+0000","last_peered":"2024-02-19T15:24:25.800924+0000","last_clean":"2024-02-19T15:24:25.800924+0000","last_became_active":"2024-02-19T15:24:25.792127+0000","last_became_peered":"2024-02-19T15:24:25.792127+0000","last_unstale":"2024-02-19T15:24:25.800924+0000","last_undegraded":"2024-02-19T15:24:25.800924+0000","last_fullsized":"2024-02-19T15:24:25.800924+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:55:20.381916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"56'4","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.816146+0000","last_change":"2024-02-19T15:24:24.780943+0000","last_active":"2024-02-19T15:24:25.816146+0000","last_peered":"2024-02-19T15:24:25.816146+0000","last_clean":"2024-02-19T15:24:25.816146+0000","last_became_active":"2024-02-19T15:24:24.780445+0000","last_became_peered":"2024-02-19T15:24:24.780445+0000","last_unstale":"2024-02-19T15:24:25.816146+0000","last_undegraded":"2024-02-19T15:24:25.816146+0000","last_fullsized":"2024-02-19T15:24:25.816146+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:20:56.982139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.808115+0000","last_change":"2024-02-19T15:24:25.800140+0000","last_active":"2024-02-19T15:24:25.808115+0000","last_peered":"2024-02-19T15:24:25.808115+0000","last_clean":"2024-02-19T15:24:25.808115+0000","last_became_active":"2024-02-19T15:24:25.799884+0000","last_became_peered":"2024-02-19T15:24:25.799884+0000","last_unstale":"2024-02-19T15:24:25.808115+0000","last_undegraded":"2024-02-19T15:24:25.808115+0000","last_fullsized":"2024-02-19T15:24:25.808115+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:46.797100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"56'1","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.809211+0000","last_change":"2024-02-19T15:24:24.782062+0000","last_active":"2024-02-19T15:24:25.809211+0000","last_peered":"2024-02-19T15:24:25.809211+0000","last_clean":"2024-02-19T15:24:25.809211+0000","last_became_active":"2024-02-19T15:24:24.781187+0000","last_became_peered":"2024-02-19T15:24:24.781187+0000","last_unstale":"2024-02-19T15:24:25.809211+0000","last_undegraded":"2024-02-19T15:24:25.809211+0000","last_fullsized":"2024-02-19T15:24:25.809211+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:20:49.931259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.813833+0000","last_change":"2024-02-19T15:24:25.805592+0000","last_active":"2024-02-19T15:24:25.813833+0000","last_peered":"2024-02-19T15:24:25.813833+0000","last_clean":"2024-02-19T15:24:25.813833+0000","last_became_active":"2024-02-19T15:24:25.805307+0000","last_became_peered":"2024-02-19T15:24:25.805307+0000","last_unstale":"2024-02-19T15:24:25.813833+0000","last_undegraded":"2024-02-19T15:24:25.813833+0000","last_fullsized":"2024-02-19T15:24:25.813833+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:29:06.908077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"56'2","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.817927+0000","last_change":"2024-02-19T15:24:24.778780+0000","last_active":"2024-02-19T15:24:25.817927+0000","last_peered":"2024-02-19T15:24:25.817927+0000","last_clean":"2024-02-19T15:24:25.817927+0000","last_became_active":"2024-02-19T15:24:24.775721+0000","last_became_peered":"2024-02-19T15:24:24.775721+0000","last_unstale":"2024-02-19T15:24:25.817927+0000","last_undegraded":"2024-02-19T15:24:25.817927+0000","last_fullsized":"2024-02-19T15:24:25.817927+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:51:59.754257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.810953+0000","last_change":"2024-02-19T15:24:25.794253+0000","last_active":"2024-02-19T15:24:25.810953+0000","last_peered":"2024-02-19T15:24:25.810953+0000","last_clean":"2024-02-19T15:24:25.810953+0000","last_became_active":"2024-02-19T15:24:25.793501+0000","last_became_peered":"2024-02-19T15:24:25.793501+0000","last_unstale":"2024-02-19T15:24:25.810953+0000","last_undegraded":"2024-02-19T15:24:25.810953+0000","last_fullsized":"2024-02-19T15:24:25.810953+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:14:50.168672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"56'2","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.807453+0000","last_change":"2024-02-19T15:24:24.778907+0000","last_active":"2024-02-19T15:24:25.807453+0000","last_peered":"2024-02-19T15:24:25.807453+0000","last_clean":"2024-02-19T15:24:25.807453+0000","last_became_active":"2024-02-19T15:24:24.778611+0000","last_became_peered":"2024-02-19T15:24:24.778611+0000","last_unstale":"2024-02-19T15:24:25.807453+0000","last_undegraded":"2024-02-19T15:24:25.807453+0000","last_fullsized":"2024-02-19T15:24:25.807453+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:03:16.895694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.803971+0000","last_change":"2024-02-19T15:24:25.799210+0000","last_active":"2024-02-19T15:24:25.803971+0000","last_peered":"2024-02-19T15:24:25.803971+0000","last_clean":"2024-02-19T15:24:25.803971+0000","last_became_active":"2024-02-19T15:24:25.798887+0000","last_became_peered":"2024-02-19T15:24:25.798887+0000","last_unstale":"2024-02-19T15:24:25.803971+0000","last_undegraded":"2024-02-19T15:24:25.803971+0000","last_fullsized":"2024-02-19T15:24:25.803971+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:52:51.662334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"56'1","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.808084+0000","last_change":"2024-02-19T15:24:24.786129+0000","last_active":"2024-02-19T15:24:25.808084+0000","last_peered":"2024-02-19T15:24:25.808084+0000","last_clean":"2024-02-19T15:24:25.808084+0000","last_became_active":"2024-02-19T15:24:24.785862+0000","last_became_peered":"2024-02-19T15:24:24.785862+0000","last_unstale":"2024-02-19T15:24:25.808084+0000","last_undegraded":"2024-02-19T15:24:25.808084+0000","last_fullsized":"2024-02-19T15:24:25.808084+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:37:36.490461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.812766+0000","last_change":"2024-02-19T15:24:25.803601+0000","last_active":"2024-02-19T15:24:25.812766+0000","last_peered":"2024-02-19T15:24:25.812766+0000","last_clean":"2024-02-19T15:24:25.812766+0000","last_became_active":"2024-02-19T15:24:25.803335+0000","last_became_peered":"2024-02-19T15:24:25.803335+0000","last_unstale":"2024-02-19T15:24:25.812766+0000","last_undegraded":"2024-02-19T15:24:25.812766+0000","last_fullsized":"2024-02-19T15:24:25.812766+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:42:01.819654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"56'1","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.816054+0000","last_change":"2024-02-19T15:24:24.772468+0000","last_active":"2024-02-19T15:24:25.816054+0000","last_peered":"2024-02-19T15:24:25.816054+0000","last_clean":"2024-02-19T15:24:25.816054+0000","last_became_active":"2024-02-19T15:24:24.770991+0000","last_became_peered":"2024-02-19T15:24:24.770991+0000","last_unstale":"2024-02-19T15:24:25.816054+0000","last_undegraded":"2024-02-19T15:24:25.816054+0000","last_fullsized":"2024-02-19T15:24:25.816054+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:33:05.568133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T15:24:26.794581+0000","last_change":"2024-02-19T15:24:25.791109+0000","last_active":"2024-02-19T15:24:26.794581+0000","last_peered":"2024-02-19T15:24:26.794581+0000","last_clean":"2024-02-19T15:24:26.794581+0000","last_became_active":"2024-02-19T15:24:25.790828+0000","last_became_peered":"2024-02-19T15:24:25.790828+0000","last_unstale":"2024-02-19T15:24:26.794581+0000","last_undegraded":"2024-02-19T15:24:26.794581+0000","last_fullsized":"2024-02-19T15:24:26.794581+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:42:56.739146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.774036+0000","last_change":"2024-02-19T15:24:24.779166+0000","last_active":"2024-02-19T15:24:25.774036+0000","last_peered":"2024-02-19T15:24:25.774036+0000","last_clean":"2024-02-19T15:24:25.774036+0000","last_became_active":"2024-02-19T15:24:24.778855+0000","last_became_peered":"2024-02-19T15:24:24.778855+0000","last_unstale":"2024-02-19T15:24:25.774036+0000","last_undegraded":"2024-02-19T15:24:25.774036+0000","last_fullsized":"2024-02-19T15:24:25.774036+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:24:45.653235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.800680+0000","last_change":"2024-02-19T15:24:25.788623+0000","last_active":"2024-02-19T15:24:25.800680+0000","last_peered":"2024-02-19T15:24:25.800680+0000","last_clean":"2024-02-19T15:24:25.800680+0000","last_became_active":"2024-02-19T15:24:25.788031+0000","last_became_peered":"2024-02-19T15:24:25.788031+0000","last_unstale":"2024-02-19T15:24:25.800680+0000","last_undegraded":"2024-02-19T15:24:25.800680+0000","last_fullsized":"2024-02-19T15:24:25.800680+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:59:03.679558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.784202+0000","last_change":"2024-02-19T15:24:24.781184+0000","last_active":"2024-02-19T15:24:25.784202+0000","last_peered":"2024-02-19T15:24:25.784202+0000","last_clean":"2024-02-19T15:24:25.784202+0000","last_became_active":"2024-02-19T15:24:24.780208+0000","last_became_peered":"2024-02-19T15:24:24.780208+0000","last_unstale":"2024-02-19T15:24:25.784202+0000","last_undegraded":"2024-02-19T15:24:25.784202+0000","last_fullsized":"2024-02-19T15:24:25.784202+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:31:29.236618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.816820+0000","last_change":"2024-02-19T15:24:25.815143+0000","last_active":"2024-02-19T15:24:25.816820+0000","last_peered":"2024-02-19T15:24:25.816820+0000","last_clean":"2024-02-19T15:24:25.816820+0000","last_became_active":"2024-02-19T15:24:25.814626+0000","last_became_peered":"2024-02-19T15:24:25.814626+0000","last_unstale":"2024-02-19T15:24:25.816820+0000","last_undegraded":"2024-02-19T15:24:25.816820+0000","last_fullsized":"2024-02-19T15:24:25.816820+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:15:56.825197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.776004+0000","last_change":"2024-02-19T15:24:24.782764+0000","last_active":"2024-02-19T15:24:25.776004+0000","last_peered":"2024-02-19T15:24:25.776004+0000","last_clean":"2024-02-19T15:24:25.776004+0000","last_became_active":"2024-02-19T15:24:24.782458+0000","last_became_peered":"2024-02-19T15:24:24.782458+0000","last_unstale":"2024-02-19T15:24:25.776004+0000","last_undegraded":"2024-02-19T15:24:25.776004+0000","last_fullsized":"2024-02-19T15:24:25.776004+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:23:41.978742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.801343+0000","last_change":"2024-02-19T15:24:25.797502+0000","last_active":"2024-02-19T15:24:25.801343+0000","last_peered":"2024-02-19T15:24:25.801343+0000","last_clean":"2024-02-19T15:24:25.801343+0000","last_became_active":"2024-02-19T15:24:25.797201+0000","last_became_peered":"2024-02-19T15:24:25.797201+0000","last_unstale":"2024-02-19T15:24:25.801343+0000","last_undegraded":"2024-02-19T15:24:25.801343+0000","last_fullsized":"2024-02-19T15:24:25.801343+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:39:49.038147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.786652+0000","last_change":"2024-02-19T15:24:24.788388+0000","last_active":"2024-02-19T15:24:25.786652+0000","last_peered":"2024-02-19T15:24:25.786652+0000","last_clean":"2024-02-19T15:24:25.786652+0000","last_became_active":"2024-02-19T15:24:24.788079+0000","last_became_peered":"2024-02-19T15:24:24.788079+0000","last_unstale":"2024-02-19T15:24:25.786652+0000","last_undegraded":"2024-02-19T15:24:25.786652+0000","last_fullsized":"2024-02-19T15:24:25.786652+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:04:43.095775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.807288+0000","last_change":"2024-02-19T15:24:25.798423+0000","last_active":"2024-02-19T15:24:25.807288+0000","last_peered":"2024-02-19T15:24:25.807288+0000","last_clean":"2024-02-19T15:24:25.807288+0000","last_became_active":"2024-02-19T15:24:25.797928+0000","last_became_peered":"2024-02-19T15:24:25.797928+0000","last_unstale":"2024-02-19T15:24:25.807288+0000","last_undegraded":"2024-02-19T15:24:25.807288+0000","last_fullsized":"2024-02-19T15:24:25.807288+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:58:19.530703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"56'3","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.815423+0000","last_change":"2024-02-19T15:24:24.778146+0000","last_active":"2024-02-19T15:24:25.815423+0000","last_peered":"2024-02-19T15:24:25.815423+0000","last_clean":"2024-02-19T15:24:25.815423+0000","last_became_active":"2024-02-19T15:24:24.777832+0000","last_became_peered":"2024-02-19T15:24:24.777832+0000","last_unstale":"2024-02-19T15:24:25.815423+0000","last_undegraded":"2024-02-19T15:24:25.815423+0000","last_fullsized":"2024-02-19T15:24:25.815423+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:08:50.034377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.803486+0000","last_change":"2024-02-19T15:24:25.797982+0000","last_active":"2024-02-19T15:24:25.803486+0000","last_peered":"2024-02-19T15:24:25.803486+0000","last_clean":"2024-02-19T15:24:25.803486+0000","last_became_active":"2024-02-19T15:24:25.797264+0000","last_became_peered":"2024-02-19T15:24:25.797264+0000","last_unstale":"2024-02-19T15:24:25.803486+0000","last_undegraded":"2024-02-19T15:24:25.803486+0000","last_fullsized":"2024-02-19T15:24:25.803486+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:38:38.879866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"56'6","reported_seq":18,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.817988+0000","last_change":"2024-02-19T15:24:24.776974+0000","last_active":"2024-02-19T15:24:25.817988+0000","last_peered":"2024-02-19T15:24:25.817988+0000","last_clean":"2024-02-19T15:24:25.817988+0000","last_became_active":"2024-02-19T15:24:24.776702+0000","last_became_peered":"2024-02-19T15:24:24.776702+0000","last_unstale":"2024-02-19T15:24:25.817988+0000","last_undegraded":"2024-02-19T15:24:25.817988+0000","last_fullsized":"2024-02-19T15:24:25.817988+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:43:21.847668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.800396+0000","last_change":"2024-02-19T15:24:25.798573+0000","last_active":"2024-02-19T15:24:25.800396+0000","last_peered":"2024-02-19T15:24:25.800396+0000","last_clean":"2024-02-19T15:24:25.800396+0000","last_became_active":"2024-02-19T15:24:25.798311+0000","last_became_peered":"2024-02-19T15:24:25.798311+0000","last_unstale":"2024-02-19T15:24:25.800396+0000","last_undegraded":"2024-02-19T15:24:25.800396+0000","last_fullsized":"2024-02-19T15:24:25.800396+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:40:41.107930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"56'2","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.814889+0000","last_change":"2024-02-19T15:24:24.788679+0000","last_active":"2024-02-19T15:24:25.814889+0000","last_peered":"2024-02-19T15:24:25.814889+0000","last_clean":"2024-02-19T15:24:25.814889+0000","last_became_active":"2024-02-19T15:24:24.786821+0000","last_became_peered":"2024-02-19T15:24:24.786821+0000","last_unstale":"2024-02-19T15:24:25.814889+0000","last_undegraded":"2024-02-19T15:24:25.814889+0000","last_fullsized":"2024-02-19T15:24:25.814889+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:47:27.885160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.789679+0000","last_change":"2024-02-19T15:24:25.786651+0000","last_active":"2024-02-19T15:24:25.789679+0000","last_peered":"2024-02-19T15:24:25.789679+0000","last_clean":"2024-02-19T15:24:25.789679+0000","last_became_active":"2024-02-19T15:24:25.786212+0000","last_became_peered":"2024-02-19T15:24:25.786212+0000","last_unstale":"2024-02-19T15:24:25.789679+0000","last_undegraded":"2024-02-19T15:24:25.789679+0000","last_fullsized":"2024-02-19T15:24:25.789679+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:34:11.045516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T15:24:26.794589+0000","last_change":"2024-02-19T15:24:24.775478+0000","last_active":"2024-02-19T15:24:26.794589+0000","last_peered":"2024-02-19T15:24:26.794589+0000","last_clean":"2024-02-19T15:24:26.794589+0000","last_became_active":"2024-02-19T15:24:24.775221+0000","last_became_peered":"2024-02-19T15:24:24.775221+0000","last_unstale":"2024-02-19T15:24:26.794589+0000","last_undegraded":"2024-02-19T15:24:26.794589+0000","last_fullsized":"2024-02-19T15:24:26.794589+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:57:32.721092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.794132+0000","last_change":"2024-02-19T15:24:25.789486+0000","last_active":"2024-02-19T15:24:25.794132+0000","last_peered":"2024-02-19T15:24:25.794132+0000","last_clean":"2024-02-19T15:24:25.794132+0000","last_became_active":"2024-02-19T15:24:25.789200+0000","last_became_peered":"2024-02-19T15:24:25.789200+0000","last_unstale":"2024-02-19T15:24:25.794132+0000","last_undegraded":"2024-02-19T15:24:25.794132+0000","last_fullsized":"2024-02-19T15:24:25.794132+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:58:19.530703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.774251+0000","last_change":"2024-02-19T15:24:24.780605+0000","last_active":"2024-02-19T15:24:25.774251+0000","last_peered":"2024-02-19T15:24:25.774251+0000","last_clean":"2024-02-19T15:24:25.774251+0000","last_became_active":"2024-02-19T15:24:24.778657+0000","last_became_peered":"2024-02-19T15:24:24.778657+0000","last_unstale":"2024-02-19T15:24:25.774251+0000","last_undegraded":"2024-02-19T15:24:25.774251+0000","last_fullsized":"2024-02-19T15:24:25.774251+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:34:41.694128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.799996+0000","last_change":"2024-02-19T15:24:25.793802+0000","last_active":"2024-02-19T15:24:25.799996+0000","last_peered":"2024-02-19T15:24:25.799996+0000","last_clean":"2024-02-19T15:24:25.799996+0000","last_became_active":"2024-02-19T15:24:25.790134+0000","last_became_peered":"2024-02-19T15:24:25.790134+0000","last_unstale":"2024-02-19T15:24:25.799996+0000","last_undegraded":"2024-02-19T15:24:25.799996+0000","last_fullsized":"2024-02-19T15:24:25.799996+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:20:50.937071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"56'8","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.807293+0000","last_change":"2024-02-19T15:24:24.777310+0000","last_active":"2024-02-19T15:24:25.807293+0000","last_peered":"2024-02-19T15:24:25.807293+0000","last_clean":"2024-02-19T15:24:25.807293+0000","last_became_active":"2024-02-19T15:24:24.777040+0000","last_became_peered":"2024-02-19T15:24:24.777040+0000","last_unstale":"2024-02-19T15:24:25.807293+0000","last_undegraded":"2024-02-19T15:24:25.807293+0000","last_fullsized":"2024-02-19T15:24:25.807293+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:03:44.425842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.797905+0000","last_change":"2024-02-19T15:24:25.789501+0000","last_active":"2024-02-19T15:24:25.797905+0000","last_peered":"2024-02-19T15:24:25.797905+0000","last_clean":"2024-02-19T15:24:25.797905+0000","last_became_active":"2024-02-19T15:24:25.787235+0000","last_became_peered":"2024-02-19T15:24:25.787235+0000","last_unstale":"2024-02-19T15:24:25.797905+0000","last_undegraded":"2024-02-19T15:24:25.797905+0000","last_fullsized":"2024-02-19T15:24:25.797905+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:02:06.503687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"56'1","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.813830+0000","last_change":"2024-02-19T15:24:24.788842+0000","last_active":"2024-02-19T15:24:25.813830+0000","last_peered":"2024-02-19T15:24:25.813830+0000","last_clean":"2024-02-19T15:24:25.813830+0000","last_became_active":"2024-02-19T15:24:24.788228+0000","last_became_peered":"2024-02-19T15:24:24.788228+0000","last_unstale":"2024-02-19T15:24:25.813830+0000","last_undegraded":"2024-02-19T15:24:25.813830+0000","last_fullsized":"2024-02-19T15:24:25.813830+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:39:48.032335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.800672+0000","last_change":"2024-02-19T15:24:25.798800+0000","last_active":"2024-02-19T15:24:25.800672+0000","last_peered":"2024-02-19T15:24:25.800672+0000","last_clean":"2024-02-19T15:24:25.800672+0000","last_became_active":"2024-02-19T15:24:25.798432+0000","last_became_peered":"2024-02-19T15:24:25.798432+0000","last_unstale":"2024-02-19T15:24:25.800672+0000","last_undegraded":"2024-02-19T15:24:25.800672+0000","last_fullsized":"2024-02-19T15:24:25.800672+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:34:11.045516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"56'2","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.816961+0000","last_change":"2024-02-19T15:24:24.789536+0000","last_active":"2024-02-19T15:24:25.816961+0000","last_peered":"2024-02-19T15:24:25.816961+0000","last_clean":"2024-02-19T15:24:25.816961+0000","last_became_active":"2024-02-19T15:24:24.788777+0000","last_became_peered":"2024-02-19T15:24:24.788777+0000","last_unstale":"2024-02-19T15:24:25.816961+0000","last_undegraded":"2024-02-19T15:24:25.816961+0000","last_fullsized":"2024-02-19T15:24:25.816961+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:18:38.223058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.801228+0000","last_change":"2024-02-19T15:24:25.790592+0000","last_active":"2024-02-19T15:24:25.801228+0000","last_peered":"2024-02-19T15:24:25.801228+0000","last_clean":"2024-02-19T15:24:25.801228+0000","last_became_active":"2024-02-19T15:24:25.790334+0000","last_became_peered":"2024-02-19T15:24:25.790334+0000","last_unstale":"2024-02-19T15:24:25.801228+0000","last_undegraded":"2024-02-19T15:24:25.801228+0000","last_fullsized":"2024-02-19T15:24:25.801228+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:46:45.322881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"56'1","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.807477+0000","last_change":"2024-02-19T15:24:24.778871+0000","last_active":"2024-02-19T15:24:25.807477+0000","last_peered":"2024-02-19T15:24:25.807477+0000","last_clean":"2024-02-19T15:24:25.807477+0000","last_became_active":"2024-02-19T15:24:24.778576+0000","last_became_peered":"2024-02-19T15:24:24.778576+0000","last_unstale":"2024-02-19T15:24:25.807477+0000","last_undegraded":"2024-02-19T15:24:25.807477+0000","last_fullsized":"2024-02-19T15:24:25.807477+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:51:09.277753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.803512+0000","last_change":"2024-02-19T15:24:25.798235+0000","last_active":"2024-02-19T15:24:25.803512+0000","last_peered":"2024-02-19T15:24:25.803512+0000","last_clean":"2024-02-19T15:24:25.803512+0000","last_became_active":"2024-02-19T15:24:25.797669+0000","last_became_peered":"2024-02-19T15:24:25.797669+0000","last_unstale":"2024-02-19T15:24:25.803512+0000","last_undegraded":"2024-02-19T15:24:25.803512+0000","last_fullsized":"2024-02-19T15:24:25.803512+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:59:03.679558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.786047+0000","last_change":"2024-02-19T15:24:24.782022+0000","last_active":"2024-02-19T15:24:25.786047+0000","last_peered":"2024-02-19T15:24:25.786047+0000","last_clean":"2024-02-19T15:24:25.786047+0000","last_became_active":"2024-02-19T15:24:24.781731+0000","last_became_peered":"2024-02-19T15:24:24.781731+0000","last_unstale":"2024-02-19T15:24:25.786047+0000","last_undegraded":"2024-02-19T15:24:25.786047+0000","last_fullsized":"2024-02-19T15:24:25.786047+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:37:52.194274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.800636+0000","last_change":"2024-02-19T15:24:25.798992+0000","last_active":"2024-02-19T15:24:25.800636+0000","last_peered":"2024-02-19T15:24:25.800636+0000","last_clean":"2024-02-19T15:24:25.800636+0000","last_became_active":"2024-02-19T15:24:25.797625+0000","last_became_peered":"2024-02-19T15:24:25.797625+0000","last_unstale":"2024-02-19T15:24:25.800636+0000","last_undegraded":"2024-02-19T15:24:25.800636+0000","last_fullsized":"2024-02-19T15:24:25.800636+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:02:06.503687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"56'2","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.814827+0000","last_change":"2024-02-19T15:24:24.772661+0000","last_active":"2024-02-19T15:24:25.814827+0000","last_peered":"2024-02-19T15:24:25.814827+0000","last_clean":"2024-02-19T15:24:25.814827+0000","last_became_active":"2024-02-19T15:24:24.769218+0000","last_became_peered":"2024-02-19T15:24:24.769218+0000","last_unstale":"2024-02-19T15:24:25.814827+0000","last_undegraded":"2024-02-19T15:24:25.814827+0000","last_fullsized":"2024-02-19T15:24:25.814827+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:00:40.629030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.812212+0000","last_change":"2024-02-19T15:24:25.802788+0000","last_active":"2024-02-19T15:24:25.812212+0000","last_peered":"2024-02-19T15:24:25.812212+0000","last_clean":"2024-02-19T15:24:25.812212+0000","last_became_active":"2024-02-19T15:24:25.798924+0000","last_became_peered":"2024-02-19T15:24:25.798924+0000","last_unstale":"2024-02-19T15:24:25.812212+0000","last_undegraded":"2024-02-19T15:24:25.812212+0000","last_fullsized":"2024-02-19T15:24:25.812212+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:58:34.401332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.784096+0000","last_change":"2024-02-19T15:24:24.785330+0000","last_active":"2024-02-19T15:24:25.784096+0000","last_peered":"2024-02-19T15:24:25.784096+0000","last_clean":"2024-02-19T15:24:25.784096+0000","last_became_active":"2024-02-19T15:24:24.784968+0000","last_became_peered":"2024-02-19T15:24:24.784968+0000","last_unstale":"2024-02-19T15:24:25.784096+0000","last_undegraded":"2024-02-19T15:24:25.784096+0000","last_fullsized":"2024-02-19T15:24:25.784096+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:51:22.245500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.802987+0000","last_change":"2024-02-19T15:24:25.792382+0000","last_active":"2024-02-19T15:24:25.802987+0000","last_peered":"2024-02-19T15:24:25.802987+0000","last_clean":"2024-02-19T15:24:25.802987+0000","last_became_active":"2024-02-19T15:24:25.792064+0000","last_became_peered":"2024-02-19T15:24:25.792064+0000","last_unstale":"2024-02-19T15:24:25.802987+0000","last_undegraded":"2024-02-19T15:24:25.802987+0000","last_fullsized":"2024-02-19T15:24:25.802987+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:04:12.392497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.774009+0000","last_change":"2024-02-19T15:24:24.783263+0000","last_active":"2024-02-19T15:24:25.774009+0000","last_peered":"2024-02-19T15:24:25.774009+0000","last_clean":"2024-02-19T15:24:25.774009+0000","last_became_active":"2024-02-19T15:24:24.783015+0000","last_became_peered":"2024-02-19T15:24:24.783015+0000","last_unstale":"2024-02-19T15:24:25.774009+0000","last_undegraded":"2024-02-19T15:24:25.774009+0000","last_fullsized":"2024-02-19T15:24:25.774009+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:13:33.903931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.792741+0000","last_change":"2024-02-19T15:24:25.784498+0000","last_active":"2024-02-19T15:24:25.792741+0000","last_peered":"2024-02-19T15:24:25.792741+0000","last_clean":"2024-02-19T15:24:25.792741+0000","last_became_active":"2024-02-19T15:24:25.784174+0000","last_became_peered":"2024-02-19T15:24:25.784174+0000","last_unstale":"2024-02-19T15:24:25.792741+0000","last_undegraded":"2024-02-19T15:24:25.792741+0000","last_fullsized":"2024-02-19T15:24:25.792741+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:35:30.247678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"56'2","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.815198+0000","last_change":"2024-02-19T15:24:24.767265+0000","last_active":"2024-02-19T15:24:25.815198+0000","last_peered":"2024-02-19T15:24:25.815198+0000","last_clean":"2024-02-19T15:24:25.815198+0000","last_became_active":"2024-02-19T15:24:24.766916+0000","last_became_peered":"2024-02-19T15:24:24.766916+0000","last_unstale":"2024-02-19T15:24:25.815198+0000","last_undegraded":"2024-02-19T15:24:25.815198+0000","last_fullsized":"2024-02-19T15:24:25.815198+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:09:46.133897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.792497+0000","last_change":"2024-02-19T15:24:25.787065+0000","last_active":"2024-02-19T15:24:25.792497+0000","last_peered":"2024-02-19T15:24:25.792497+0000","last_clean":"2024-02-19T15:24:25.792497+0000","last_became_active":"2024-02-19T15:24:25.786562+0000","last_became_peered":"2024-02-19T15:24:25.786562+0000","last_unstale":"2024-02-19T15:24:25.792497+0000","last_undegraded":"2024-02-19T15:24:25.792497+0000","last_fullsized":"2024-02-19T15:24:25.792497+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:59.656620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.787306+0000","last_change":"2024-02-19T15:24:24.778112+0000","last_active":"2024-02-19T15:24:25.787306+0000","last_peered":"2024-02-19T15:24:25.787306+0000","last_clean":"2024-02-19T15:24:25.787306+0000","last_became_active":"2024-02-19T15:24:24.777664+0000","last_became_peered":"2024-02-19T15:24:24.777664+0000","last_unstale":"2024-02-19T15:24:25.787306+0000","last_undegraded":"2024-02-19T15:24:25.787306+0000","last_fullsized":"2024-02-19T15:24:25.787306+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:46:45.287461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.807231+0000","last_change":"2024-02-19T15:24:25.798196+0000","last_active":"2024-02-19T15:24:25.807231+0000","last_peered":"2024-02-19T15:24:25.807231+0000","last_clean":"2024-02-19T15:24:25.807231+0000","last_became_active":"2024-02-19T15:24:25.797511+0000","last_became_peered":"2024-02-19T15:24:25.797511+0000","last_unstale":"2024-02-19T15:24:25.807231+0000","last_undegraded":"2024-02-19T15:24:25.807231+0000","last_fullsized":"2024-02-19T15:24:25.807231+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:43:22.853480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T15:24:26.794862+0000","last_change":"2024-02-19T15:24:24.769353+0000","last_active":"2024-02-19T15:24:26.794862+0000","last_peered":"2024-02-19T15:24:26.794862+0000","last_clean":"2024-02-19T15:24:26.794862+0000","last_became_active":"2024-02-19T15:24:24.768970+0000","last_became_peered":"2024-02-19T15:24:24.768970+0000","last_unstale":"2024-02-19T15:24:26.794862+0000","last_undegraded":"2024-02-19T15:24:26.794862+0000","last_fullsized":"2024-02-19T15:24:26.794862+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:06:35.623552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.799769+0000","last_change":"2024-02-19T15:24:25.794049+0000","last_active":"2024-02-19T15:24:25.799769+0000","last_peered":"2024-02-19T15:24:25.799769+0000","last_clean":"2024-02-19T15:24:25.799769+0000","last_became_active":"2024-02-19T15:24:25.793167+0000","last_became_peered":"2024-02-19T15:24:25.793167+0000","last_unstale":"2024-02-19T15:24:25.799769+0000","last_undegraded":"2024-02-19T15:24:25.799769+0000","last_fullsized":"2024-02-19T15:24:25.799769+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:17:49.928845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.775778+0000","last_change":"2024-02-19T15:24:24.780196+0000","last_active":"2024-02-19T15:24:25.775778+0000","last_peered":"2024-02-19T15:24:25.775778+0000","last_clean":"2024-02-19T15:24:25.775778+0000","last_became_active":"2024-02-19T15:24:24.778936+0000","last_became_peered":"2024-02-19T15:24:24.778936+0000","last_unstale":"2024-02-19T15:24:25.775778+0000","last_undegraded":"2024-02-19T15:24:25.775778+0000","last_fullsized":"2024-02-19T15:24:25.775778+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:56:38.605488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.804053+0000","last_change":"2024-02-19T15:24:25.800386+0000","last_active":"2024-02-19T15:24:25.804053+0000","last_peered":"2024-02-19T15:24:25.804053+0000","last_clean":"2024-02-19T15:24:25.804053+0000","last_became_active":"2024-02-19T15:24:25.800150+0000","last_became_peered":"2024-02-19T15:24:25.800150+0000","last_unstale":"2024-02-19T15:24:25.804053+0000","last_undegraded":"2024-02-19T15:24:25.804053+0000","last_fullsized":"2024-02-19T15:24:25.804053+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:58:25.822438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.779173+0000","last_change":"2024-02-19T15:24:24.776772+0000","last_active":"2024-02-19T15:24:25.779173+0000","last_peered":"2024-02-19T15:24:25.779173+0000","last_clean":"2024-02-19T15:24:25.779173+0000","last_became_active":"2024-02-19T15:24:24.776479+0000","last_became_peered":"2024-02-19T15:24:24.776479+0000","last_unstale":"2024-02-19T15:24:25.779173+0000","last_undegraded":"2024-02-19T15:24:25.779173+0000","last_fullsized":"2024-02-19T15:24:25.779173+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:47:51.312636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.800888+0000","last_change":"2024-02-19T15:24:25.795341+0000","last_active":"2024-02-19T15:24:25.800888+0000","last_peered":"2024-02-19T15:24:25.800888+0000","last_clean":"2024-02-19T15:24:25.800888+0000","last_became_active":"2024-02-19T15:24:25.792938+0000","last_became_peered":"2024-02-19T15:24:25.792938+0000","last_unstale":"2024-02-19T15:24:25.800888+0000","last_undegraded":"2024-02-19T15:24:25.800888+0000","last_fullsized":"2024-02-19T15:24:25.800888+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:00:41.634842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.783554+0000","last_change":"2024-02-19T15:24:24.777682+0000","last_active":"2024-02-19T15:24:25.783554+0000","last_peered":"2024-02-19T15:24:25.783554+0000","last_clean":"2024-02-19T15:24:25.783554+0000","last_became_active":"2024-02-19T15:24:24.777391+0000","last_became_peered":"2024-02-19T15:24:24.777391+0000","last_unstale":"2024-02-19T15:24:25.783554+0000","last_undegraded":"2024-02-19T15:24:25.783554+0000","last_fullsized":"2024-02-19T15:24:25.783554+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:51:22.245500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.805976+0000","last_change":"2024-02-19T15:24:25.797547+0000","last_active":"2024-02-19T15:24:25.805976+0000","last_peered":"2024-02-19T15:24:25.805976+0000","last_clean":"2024-02-19T15:24:25.805976+0000","last_became_active":"2024-02-19T15:24:25.797250+0000","last_became_peered":"2024-02-19T15:24:25.797250+0000","last_unstale":"2024-02-19T15:24:25.805976+0000","last_undegraded":"2024-02-19T15:24:25.805976+0000","last_fullsized":"2024-02-19T15:24:25.805976+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:37:37.496274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.778961+0000","last_change":"2024-02-19T15:24:24.770314+0000","last_active":"2024-02-19T15:24:25.778961+0000","last_peered":"2024-02-19T15:24:25.778961+0000","last_clean":"2024-02-19T15:24:25.778961+0000","last_became_active":"2024-02-19T15:24:24.770005+0000","last_became_peered":"2024-02-19T15:24:24.770005+0000","last_unstale":"2024-02-19T15:24:25.778961+0000","last_undegraded":"2024-02-19T15:24:25.778961+0000","last_fullsized":"2024-02-19T15:24:25.778961+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:37:35.076825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.800654+0000","last_change":"2024-02-19T15:24:25.788284+0000","last_active":"2024-02-19T15:24:25.800654+0000","last_peered":"2024-02-19T15:24:25.800654+0000","last_clean":"2024-02-19T15:24:25.800654+0000","last_became_active":"2024-02-19T15:24:25.787700+0000","last_became_peered":"2024-02-19T15:24:25.787700+0000","last_unstale":"2024-02-19T15:24:25.800654+0000","last_undegraded":"2024-02-19T15:24:25.800654+0000","last_fullsized":"2024-02-19T15:24:25.800654+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:38:38.879866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.782402+0000","last_change":"2024-02-19T15:24:24.779487+0000","last_active":"2024-02-19T15:24:25.782402+0000","last_peered":"2024-02-19T15:24:25.782402+0000","last_clean":"2024-02-19T15:24:25.782402+0000","last_became_active":"2024-02-19T15:24:24.779229+0000","last_became_peered":"2024-02-19T15:24:24.779229+0000","last_unstale":"2024-02-19T15:24:25.782402+0000","last_undegraded":"2024-02-19T15:24:25.782402+0000","last_fullsized":"2024-02-19T15:24:25.782402+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:57:37.605728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.813337+0000","last_change":"2024-02-19T15:24:25.810469+0000","last_active":"2024-02-19T15:24:25.813337+0000","last_peered":"2024-02-19T15:24:25.813337+0000","last_clean":"2024-02-19T15:24:25.813337+0000","last_became_active":"2024-02-19T15:24:25.810203+0000","last_became_peered":"2024-02-19T15:24:25.810203+0000","last_unstale":"2024-02-19T15:24:25.813337+0000","last_undegraded":"2024-02-19T15:24:25.813337+0000","last_fullsized":"2024-02-19T15:24:25.813337+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:24.752404+0000","last_clean_scrub_stamp":"2024-02-19T15:24:24.752404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:58:47.729061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T15:24:26.794482+0000","last_change":"2024-02-19T15:24:24.774529+0000","last_active":"2024-02-19T15:24:26.794482+0000","last_peered":"2024-02-19T15:24:26.794482+0000","last_clean":"2024-02-19T15:24:26.794482+0000","last_became_active":"2024-02-19T15:24:24.774254+0000","last_became_peered":"2024-02-19T15:24:24.774254+0000","last_unstale":"2024-02-19T15:24:26.794482+0000","last_undegraded":"2024-02-19T15:24:26.794482+0000","last_fullsized":"2024-02-19T15:24:26.794482+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:24:23.746592+0000","last_clean_scrub_stamp":"2024-02-19T15:24:23.746592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:59:22.650366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":232,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-02-19T15:24:25.776343+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:24:25.776343+0000","last_peered":"2024-02-19T15:24:25.776343+0000","last_clean":"2024-02-19T15:24:25.776343+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:24:25.776343+0000","last_undegraded":"2024-02-19T15:24:25.776343+0000","last_fullsized":"2024-02-19T15:24:25.776343+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509474,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":667648,"data_stored":275134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.39700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82199999999999995,"5min":1.022,"15min":1.022},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.748,"15min":0.77500000000000002},"min":{"1min":0.45800000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.85499999999999998,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.83699999999999997}]},{"osd":1,"last update":"Mon Feb 19 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.64600000000000002,"15min":0.65200000000000002},"min":{"1min":0.442,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.82799999999999996,"5min":1.171,"15min":1.171},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.64300000000000002,"15min":0.64800000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.755,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.68600000000000005}]},{"osd":2,"last update":"Mon Feb 19 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.626,"15min":0.63},"min":{"1min":0.45400000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.75,"5min":1.216,"15min":1.216},"last":0.753},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.67800000000000005,"15min":0.68200000000000005},"min":{"1min":0.53100000000000003,"5min":0.495,"15min":0.495},"max":{"1min":0.80900000000000005,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.70199999999999996}]},{"osd":3,"last update":"Mon Feb 19 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.67900000000000005,"15min":0.67500000000000004},"min":{"1min":0.47999999999999998,"5min":0.438,"15min":0.438},"max":{"1min":0.91300000000000003,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.65400000000000003},"min":{"1min":0.46700000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88300000000000001,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.72099999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.60799999999999998,"15min":0.60099999999999998},"min":{"1min":0.40899999999999997,"5min":0.309,"15min":0.309},"max":{"1min":0.86699999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.61299999999999999,"15min":0.59899999999999998},"min":{"1min":0.435,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.58799999999999997}]},{"osd":5,"last update":"Mon Feb 19 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.38400000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.629},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66900000000000004,"15min":0.66500000000000004},"min":{"1min":0.46700000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.879,"5min":1.02,"15min":1.02},"last":0.61499999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:24:00 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.66000000000000003,"15min":0.64800000000000002},"min":{"1min":0.54800000000000004,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.95699999999999996,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.67400000000000004,"15min":0.66600000000000004},"min":{"1min":0.54800000000000004,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.82099999999999995}]}]},{"osd":7,"up_from":14,"seq":60129542178,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32404,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339436,"statfs":{"total":96636764160,"available":96603582464,"internally_reserved":0,"allocated":1249280,"data_stored":858727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.42399999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.90500000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67900000000000005,"15min":0.68100000000000005},"min":{"1min":0.52400000000000002,"5min":0.433,"15min":0.433},"max":{"1min":1.0580000000000001,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.79100000000000004}]},{"osd":1,"last update":"Mon Feb 19 15:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.60699999999999998,"15min":0.59199999999999997},"min":{"1min":0.53600000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.72099999999999997,"15min":0.72699999999999998},"min":{"1min":0.47199999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.90000000000000002,"5min":2.968,"15min":2.968},"last":0.67300000000000004}]},{"osd":2,"last update":"Mon Feb 19 15:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.63500000000000001,"15min":0.63200000000000001},"min":{"1min":0.46200000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.69199999999999995,"15min":0.69299999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.91000000000000003,"5min":2.508,"15min":2.508},"last":0.77300000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.65400000000000003,"15min":0.65600000000000003},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.88300000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.55100000000000005,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.99099999999999999,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":0.73499999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.54000000000000004,"15min":0.52000000000000002},"min":{"1min":0.373,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.95799999999999996,"5min":1.03,"15min":1.03},"last":0.621},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.57199999999999995,"15min":0.56599999999999995},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78400000000000003,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.56799999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.58699999999999997,"15min":0.56100000000000005},"min":{"1min":0.45700000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.754},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.504,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.85099999999999998,"5min":1.046,"15min":1.046},"last":0.66200000000000003}]},{"osd":6,"last update":"Mon Feb 19 15:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.60999999999999999,"15min":0.59299999999999997},"min":{"1min":0.41999999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.67500000000000004,"15min":0.65800000000000003},"min":{"1min":0.499,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.81799999999999995}]}]},{"osd":5,"up_from":14,"seq":60129542179,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":675840,"data_stored":275786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.62,"15min":0.627},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.89000000000000001,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.59499999999999997,"15min":0.59799999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.77600000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.75900000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.63800000000000001,"15min":0.63600000000000001},"min":{"1min":0.47499999999999998,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.85799999999999998,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67700000000000005,"15min":0.67500000000000004},"min":{"1min":0.47199999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.874,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.79200000000000004}]},{"osd":2,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.67700000000000005,"15min":0.68100000000000005},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.89200000000000002,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59199999999999997,"15min":0.59499999999999997},"min":{"1min":0.46800000000000003,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.80200000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.82399999999999995}]},{"osd":3,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.54000000000000004,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82499999999999996,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65200000000000002,"15min":0.65400000000000003},"min":{"1min":0.48299999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.73599999999999999,"5min":1.103,"15min":1.103},"last":0.77400000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56399999999999995,"15min":0.55500000000000005},"min":{"1min":0.35499999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.89200000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64200000000000002,"15min":0.64800000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.876,"5min":1.458,"15min":1.458},"last":0.56399999999999995}]},{"osd":6,"last update":"Mon Feb 19 15:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.64100000000000001,"15min":0.63500000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93899999999999995,"5min":1.004,"15min":1.004},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.72299999999999998,"15min":0.71699999999999997},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.97799999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.67000000000000004}]},{"osd":7,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.82499999999999996,"5min":1.038,"15min":1.038},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.68799999999999994,"15min":0.68500000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88600000000000001,"5min":1.131,"15min":1.131},"last":0.58099999999999996}]}]},{"osd":4,"up_from":14,"seq":60129542177,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":655360,"data_stored":268359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.34399999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76300000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.437},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.60799999999999998,"15min":0.59999999999999998},"min":{"1min":0.47299999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.84499999999999997,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.54900000000000004}]},{"osd":1,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.622,"15min":0.61299999999999999},"min":{"1min":0.495,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86899999999999999,"5min":1.081,"15min":1.081},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.63600000000000001,"15min":0.63100000000000001},"min":{"1min":0.441,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86199999999999999,"5min":1.008,"15min":1.008},"last":0.52200000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.66200000000000003,"15min":0.66300000000000003},"min":{"1min":0.45800000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.91100000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.67800000000000005,"15min":0.67500000000000004},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.96299999999999997,"5min":1.002,"15min":1.002},"last":0.63700000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.68000000000000005,"15min":0.67300000000000004},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65300000000000002,"15min":0.65100000000000002},"min":{"1min":0.47199999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.90000000000000002,"5min":1.153,"15min":1.153},"last":0.59799999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.38200000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.98199999999999998,"5min":1.042,"15min":1.042},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.68899999999999995,"15min":0.67700000000000005},"min":{"1min":0.54300000000000004,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.079,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.57399999999999995}]},{"osd":6,"last update":"Mon Feb 19 15:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.41399999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.65800000000000003,"15min":0.64600000000000002},"min":{"1min":0.45400000000000001,"5min":0.438,"15min":0.438},"max":{"1min":1.0249999999999999,"5min":1.054,"15min":1.054},"last":0.61499999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.67500000000000004,"15min":0.66200000000000003},"min":{"1min":0.57099999999999995,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.96399999999999997,"5min":1.103,"15min":1.103},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.73699999999999999,"15min":0.72699999999999998},"min":{"1min":0.56000000000000005,"5min":0.443,"15min":0.443},"max":{"1min":1.048,"5min":1.24,"15min":1.24},"last":0.747}]}]},{"osd":3,"up_from":14,"seq":60129542177,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":675840,"data_stored":275190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.45000000000000001,"15min":0.45400000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.80400000000000005,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.64300000000000002,"15min":0.68000000000000005},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.56100000000000005,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.34999999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.438,"15min":0.44400000000000001},"min":{"1min":0.313,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.58999999999999997,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.312},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.70899999999999996,"15min":0.747},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.79600000000000004,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.46700000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.61299999999999999,"15min":0.627},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.88100000000000001,"5min":1.008,"15min":1.008},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.58399999999999996,"15min":0.59999999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72899999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.38400000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.68300000000000005,"15min":0.67200000000000004},"min":{"1min":0.57099999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.85899999999999999,"15min":0.88},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.96099999999999997,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.85699999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.85699999999999998,"15min":0.873},"min":{"1min":0.54500000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.95299999999999996,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.748,"15min":0.75},"min":{"1min":0.53500000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.95299999999999996,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.89300000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.89200000000000002,"15min":0.91200000000000003},"min":{"1min":0.58699999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.99299999999999999,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.80700000000000005,"15min":0.82099999999999995},"min":{"1min":0.52800000000000002,"5min":0.502,"15min":0.502},"max":{"1min":0.94299999999999995,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.873}]},{"osd":7,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78200000000000003,"15min":0.78600000000000003},"min":{"1min":0.56299999999999994,"5min":0.504,"15min":0.504},"max":{"1min":1.026,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.92000000000000004,"15min":0.94599999999999995},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.93000000000000005,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.82399999999999995}]}]},{"osd":2,"up_from":14,"seq":60129542178,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31824,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":655360,"data_stored":268359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":0.80100000000000005,"15min":0.74099999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":15.42,"5min":15.42,"15min":15.42},"last":0.44},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.92800000000000005,"15min":0.95599999999999996},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":6.7000000000000002,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.35399999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.96299999999999997,"15min":0.96599999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":9.5269999999999992,"5min":9.5269999999999992,"15min":9.5269999999999992},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.1559999999999999,"15min":1.0780000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":22.954999999999998,"5min":22.954999999999998,"15min":22.954999999999998},"last":0.47399999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.80500000000000005,"15min":0.82799999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":3.7799999999999998,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.873,"15min":0.86599999999999999},"min":{"1min":0.443,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":7.734,"5min":7.734,"15min":7.734},"last":0.48999999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.0069999999999999,"15min":0.93400000000000005},"min":{"1min":0.54900000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":15.369,"5min":15.369,"15min":15.369},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.92300000000000004,"15min":0.92500000000000004},"min":{"1min":0.61799999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":3.6819999999999999,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.95199999999999996}]},{"osd":5,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":0.94199999999999995,"15min":0.91200000000000003},"min":{"1min":0.58399999999999996,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":7.6520000000000001,"5min":7.6520000000000001,"15min":7.6520000000000001},"last":1.135},{"interface":"front","average":{"1min":1.526,"5min":1.0800000000000001,"15min":1.006},"min":{"1min":0.57999999999999996,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":15.391,"5min":15.391,"15min":15.391},"last":0.71999999999999997}]},{"osd":6,"last update":"Mon Feb 19 15:23:57 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.0449999999999999,"15min":0.97299999999999998},"min":{"1min":0.57799999999999996,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":15.337,"5min":15.337,"15min":15.337},"last":1.002},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.98099999999999998,"15min":0.99299999999999999},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":3.7450000000000001,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":1.1539999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.006,"15min":1.014},"min":{"1min":0.65000000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":3.71,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":1.073},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.006,"15min":0.98299999999999998},"min":{"1min":0.57499999999999996,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":7.6929999999999996,"5min":7.6929999999999996,"15min":7.6929999999999996},"last":1.0309999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31824,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":655360,"data_stored":268359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.42699999999999999,"15min":0.435},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.60499999999999998,"5min":0.747,"15min":0.747},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.46800000000000003,"15min":0.47599999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.442,"15min":0.45000000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.55000000000000004,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.52300000000000002,"15min":0.53800000000000003},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.55800000000000005,"5min":1.071,"15min":1.071},"last":0.623}]},{"osd":3,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.63200000000000001,"15min":0.64800000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.76000000000000001,"5min":1.365,"15min":1.365},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.63,"15min":0.64900000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76200000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.57599999999999996}]},{"osd":4,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.61599999999999999,"15min":0.59999999999999998},"min":{"1min":0.48199999999999998,"5min":0.377,"15min":0.377},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.68100000000000005,"15min":0.67600000000000005},"min":{"1min":0.505,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.0109999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.98899999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.69799999999999995,"15min":0.69399999999999995},"min":{"1min":0.56499999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0509999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.68200000000000005,"15min":0.67800000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.92000000000000004,"5min":1.107,"15min":1.107},"last":1.0860000000000001}]},{"osd":6,"last update":"Mon Feb 19 15:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.73099999999999998,"15min":0.72899999999999998},"min":{"1min":0.54900000000000004,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0649999999999999,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.58599999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0840000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":1.0669999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.78900000000000003,"15min":0.79300000000000004},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.1120000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":1.101},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.748,"15min":0.74199999999999999},"min":{"1min":0.59799999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.032,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":1.032}]}]},{"osd":0,"up_from":14,"seq":60129542177,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36564,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335276,"statfs":{"total":96636764160,"available":96599322624,"internally_reserved":0,"allocated":1249280,"data_stored":858727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.626,"15min":0.66800000000000004},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57399999999999995,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.48399999999999999,"15min":0.48999999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73899999999999999,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.40500000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.78200000000000003,"15min":0.82599999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.72399999999999998,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.74399999999999999,"15min":0.78800000000000003},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.70499999999999996,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.53000000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.67700000000000005,"15min":0.71499999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.752,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.60799999999999998,"15min":0.63},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.77100000000000002,"5min":1.345,"15min":1.345},"last":0.46999999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.745,"15min":0.74299999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.94699999999999995,"5min":1.095,"15min":1.095},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.92100000000000004,"15min":0.95099999999999996},"min":{"1min":0.59599999999999997,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.89500000000000002,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.81599999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.751,"15min":0.755},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.85199999999999998,"15min":0.88},"min":{"1min":0.53200000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.96099999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.86299999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.91200000000000003,"15min":0.94599999999999995},"min":{"1min":0.53600000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.93100000000000005,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.75800000000000001,"15min":0.76400000000000001},"min":{"1min":0.54600000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.873,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.76900000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.94299999999999995,"15min":0.97299999999999998},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.2450000000000001,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.93799999999999994,"15min":0.97599999999999998},"min":{"1min":0.56200000000000006,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.91500000000000004,"5min":4.915,"15min":4.915},"last":0.89400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:24:30.489 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T15:24:30.489 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-19T15:24:30.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.670+0000 7f35d8c00640 1 Processor -- start 2024-02-19T15:24:30.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.670+0000 7f35d8c00640 1 -- start start 2024-02-19T15:24:30.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.670+0000 7f35d8c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f35d406aeb0 0x7f35d406b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:30.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.674+0000 7f35d8c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35d40cb320 0x7f35d40cb790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:30.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.674+0000 7f35d8c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d415fbe0 con 0x7f35d415f420 2024-02-19T15:24:30.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.674+0000 7f35d8c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35d406b880 con 0x7f35d406aeb0 2024-02-19T15:24:30.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.674+0000 7f35d8c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d4070210 con 0x7f35d40cb320 2024-02-19T15:24:30.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.674+0000 7f35d1400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f35d406aeb0 0x7f35d406b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:30.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.674+0000 7f35d1e00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f35d415f420 0x7f35d406fb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:36418/0 (socket says 172.21.15.47:36418) 2024-02-19T15:24:30.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.674+0000 7f35d1e00640 1 -- 172.21.15.47:0/1389662 learned_addr learned my addr 172.21.15.47:0/1389662 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:30.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.674+0000 7f35d2800640 1 --2- 172.21.15.47:0/1389662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35d40cb320 0x7f35d40cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:30.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.674+0000 7f35d0a00640 1 -- 172.21.15.47:0/1389662 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414602648 0 0) 0x7f35d415fbe0 con 0x7f35d415f420 2024-02-19T15:24:30.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.674+0000 7f35d0a00640 1 -- 172.21.15.47:0/1389662 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35b8003530 con 0x7f35d415f420 2024-02-19T15:24:30.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.674+0000 7f35d1400640 1 -- 172.21.15.47:0/1389662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35d40cb320 msgr2=0x7f35d40cb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:30.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.674+0000 7f35d1400640 1 --2- 172.21.15.47:0/1389662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35d40cb320 0x7f35d40cb790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.674+0000 7f35d1400640 1 -- 172.21.15.47:0/1389662 >> v1:172.21.15.116:6789/0 conn(0x7f35d415f420 legacy=0x7f35d406fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:30.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.674+0000 7f35d1400640 1 -- 172.21.15.47:0/1389662 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d415fe50 con 0x7f35d406aeb0 2024-02-19T15:24:30.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.674+0000 7f35d1400640 1 --2- 172.21.15.47:0/1389662 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35d406aeb0 0x7f35d406b340 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f35c00097c0 tx=0x7f35c001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=836be0be31328345 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:30.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d0a00640 1 -- 172.21.15.47:0/1389662 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c00ab670 con 0x7f35d406aeb0 2024-02-19T15:24:30.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d0a00640 1 -- 172.21.15.47:0/1389662 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c00ab7d0 con 0x7f35d406aeb0 2024-02-19T15:24:30.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d0a00640 1 -- 172.21.15.47:0/1389662 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c00abb40 con 0x7f35d406aeb0 2024-02-19T15:24:30.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d8c00640 1 -- 172.21.15.47:0/1389662 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35d406aeb0 msgr2=0x7f35d406b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:30.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d8c00640 1 --2- 172.21.15.47:0/1389662 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35d406aeb0 0x7f35d406b340 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f35c00097c0 tx=0x7f35c001c470 comp rx=0 tx=0).stop 2024-02-19T15:24:30.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d8c00640 1 -- 172.21.15.47:0/1389662 shutdown_connections 2024-02-19T15:24:30.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d8c00640 1 --2- 172.21.15.47:0/1389662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35d40cb320 0x7f35d40cb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d8c00640 1 --2- 172.21.15.47:0/1389662 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35d406aeb0 0x7f35d406b340 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d8c00640 1 -- 172.21.15.47:0/1389662 >> 172.21.15.47:0/1389662 conn(0x7f35d40b2580 msgr2=0x7f35d40b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:30.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d8c00640 1 -- 172.21.15.47:0/1389662 shutdown_connections 2024-02-19T15:24:30.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d8c00640 1 -- 172.21.15.47:0/1389662 wait complete. 2024-02-19T15:24:30.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d8c00640 1 Processor -- start 2024-02-19T15:24:30.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d8c00640 1 -- start start 2024-02-19T15:24:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d8c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35d406aeb0 0x7f35d41582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d8c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35d40cb320 0x7f35d4151ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d8c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d415f420 0x7f35d4152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d8c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d40b5160 con 0x7f35d406aeb0 2024-02-19T15:24:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d8c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d40b5440 con 0x7f35d415f420 2024-02-19T15:24:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d8c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d40b5720 con 0x7f35d40cb320 2024-02-19T15:24:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d1e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35d406aeb0 0x7f35d41582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d1e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35d406aeb0 0x7f35d41582b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51040/0 (socket says 172.21.15.47:51040) 2024-02-19T15:24:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.678+0000 7f35d1e00640 1 -- 172.21.15.47:0/435518907 learned_addr learned my addr 172.21.15.47:0/435518907 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.682+0000 7f35d1400640 1 --2- 172.21.15.47:0/435518907 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35d40cb320 0x7f35d4151ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.682+0000 7f35d1e00640 1 -- 172.21.15.47:0/435518907 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35d40cb320 msgr2=0x7f35d4151ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.682+0000 7f35d1e00640 1 --2- 172.21.15.47:0/435518907 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35d40cb320 0x7f35d4151ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.682+0000 7f35d2800640 1 --2- 172.21.15.47:0/435518907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d415f420 0x7f35d4152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.682+0000 7f35d1e00640 1 -- 172.21.15.47:0/435518907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d415f420 msgr2=0x7f35d4152530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:30.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.682+0000 7f35d1e00640 1 --2- 172.21.15.47:0/435518907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d415f420 0x7f35d4152530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.682+0000 7f35d1e00640 1 -- 172.21.15.47:0/435518907 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c0009420 con 0x7f35d406aeb0 2024-02-19T15:24:30.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.682+0000 7f35d1400640 1 --2- 172.21.15.47:0/435518907 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35d40cb320 0x7f35d4151ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:30.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.682+0000 7f35d2800640 1 --2- 172.21.15.47:0/435518907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d415f420 0x7f35d4152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:30.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.682+0000 7f35d1e00640 1 --2- 172.21.15.47:0/435518907 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35d406aeb0 0x7f35d41582b0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f35c400c3c0 tx=0x7f35c400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:30.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.682+0000 7f35ca000640 1 -- 172.21.15.47:0/435518907 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c4005cf0 con 0x7f35d406aeb0 2024-02-19T15:24:30.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.682+0000 7f35ca000640 1 -- 172.21.15.47:0/435518907 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c4005e50 con 0x7f35d406aeb0 2024-02-19T15:24:30.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.682+0000 7f35ca000640 1 -- 172.21.15.47:0/435518907 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c40072f0 con 0x7f35d406aeb0 2024-02-19T15:24:30.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.682+0000 7f35d8c00640 1 -- 172.21.15.47:0/435518907 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d40b5a00 con 0x7f35d406aeb0 2024-02-19T15:24:30.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.682+0000 7f35d8c00640 1 -- 172.21.15.47:0/435518907 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d40b6200 con 0x7f35d406aeb0 2024-02-19T15:24:30.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.682+0000 7f35d8c00640 1 -- 172.21.15.47:0/435518907 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d40b5f20 con 0x7f35d406aeb0 2024-02-19T15:24:30.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.682+0000 7f35ca000640 1 -- 172.21.15.47:0/435518907 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f35c4007450 con 0x7f35d406aeb0 2024-02-19T15:24:30.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.682+0000 7f35ca000640 1 --2- 172.21.15.47:0/435518907 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f35b4066e40 0x7f35b4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:30.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.682+0000 7f35ca000640 1 -- 172.21.15.47:0/435518907 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35c4084060 con 0x7f35d406aeb0 2024-02-19T15:24:30.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.686+0000 7f35d1400640 1 --2- 172.21.15.47:0/435518907 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f35b4066e40 0x7f35b4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:30.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.686+0000 7f35d1400640 1 --2- 172.21.15.47:0/435518907 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f35b4066e40 0x7f35b4069300 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f35c00af000 tx=0x7f35c0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:30.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.686+0000 7f35ca000640 1 -- 172.21.15.47:0/435518907 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f35d40b5f20 con 0x7f35d406aeb0 2024-02-19T15:24:30.750 INFO:tasks.ceph.mds.a.smithi047.stderr:2024-02-19T15:24:30.746+0000 7f45d4600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T15:24:30.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.906+0000 7f35d8c00640 1 -- 172.21.15.47:0/435518907 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f35d40b5f20 con 0x7f35d406aeb0 2024-02-19T15:24:30.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.910+0000 7f35ca000640 1 -- 172.21.15.47:0/435518907 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7f35c4054980 con 0x7f35d406aeb0 2024-02-19T15:24:30.911 INFO:teuthology.orchestra.run.smithi047.stdout:[client.0] 2024-02-19T15:24:30.911 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDjcdNlb7eRNRAAk6ZLGfjhcdlR9GoXFCd9eA== 2024-02-19T15:24:30.911 INFO:teuthology.orchestra.run.smithi047.stdout: caps mds = "allow" 2024-02-19T15:24:30.911 INFO:teuthology.orchestra.run.smithi047.stdout: caps mon = "allow r" 2024-02-19T15:24:30.911 INFO:teuthology.orchestra.run.smithi047.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-19T15:24:30.912 INFO:teuthology.orchestra.run.smithi047.stderr:updated caps for client.0 2024-02-19T15:24:30.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.910+0000 7f35d8c00640 1 -- 172.21.15.47:0/435518907 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f35b4066e40 msgr2=0x7f35b4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:30.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.910+0000 7f35d8c00640 1 --2- 172.21.15.47:0/435518907 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f35b4066e40 0x7f35b4069300 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f35c00af000 tx=0x7f35c0008040 comp rx=0 tx=0).stop 2024-02-19T15:24:30.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.910+0000 7f35d8c00640 1 -- 172.21.15.47:0/435518907 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35d406aeb0 msgr2=0x7f35d41582b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:30.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.910+0000 7f35d8c00640 1 --2- 172.21.15.47:0/435518907 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35d406aeb0 0x7f35d41582b0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f35c400c3c0 tx=0x7f35c400c880 comp rx=0 tx=0).stop 2024-02-19T15:24:30.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.910+0000 7f35d8c00640 1 -- 172.21.15.47:0/435518907 shutdown_connections 2024-02-19T15:24:30.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.910+0000 7f35d8c00640 1 --2- 172.21.15.47:0/435518907 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f35b4066e40 0x7f35b4069300 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.910+0000 7f35d8c00640 1 --2- 172.21.15.47:0/435518907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d415f420 0x7f35d4152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.910+0000 7f35d8c00640 1 --2- 172.21.15.47:0/435518907 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35d40cb320 0x7f35d4151ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.910+0000 7f35d8c00640 1 --2- 172.21.15.47:0/435518907 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35d406aeb0 0x7f35d41582b0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:30.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.910+0000 7f35d8c00640 1 -- 172.21.15.47:0/435518907 >> 172.21.15.47:0/435518907 conn(0x7f35d40b2580 msgr2=0x7f35d4154920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:30.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.910+0000 7f35d8c00640 1 -- 172.21.15.47:0/435518907 shutdown_connections 2024-02-19T15:24:30.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:30.910+0000 7f35d8c00640 1 -- 172.21.15.47:0/435518907 wait complete. 2024-02-19T15:24:30.933 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:24:31.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.070+0000 7f0e96200640 1 Processor -- start 2024-02-19T15:24:31.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.074+0000 7f0e96200640 1 -- start start 2024-02-19T15:24:31.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.074+0000 7f0e96200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0e90145590 0x7f0e900727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.074+0000 7f0e96200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0e900cea80 0x7f0e9006c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.074+0000 7f0e96200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e9015c390 0x7f0e9015c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.074+0000 7f0e96200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e9015d280 con 0x7f0e90145590 2024-02-19T15:24:31.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.074+0000 7f0e96200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e90070ff0 con 0x7f0e900cea80 2024-02-19T15:24:31.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.074+0000 7f0e96200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9015cd60 con 0x7f0e9015c390 2024-02-19T15:24:31.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.074+0000 7f0e8f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0e90145590 0x7f0e900727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.074+0000 7f0e8f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0e90145590 0x7f0e900727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51052/0 (socket says 172.21.15.47:51052) 2024-02-19T15:24:31.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.074+0000 7f0e8f400640 1 -- 172.21.15.47:0/3484514596 learned_addr learned my addr 172.21.15.47:0/3484514596 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:31.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.074+0000 7f0e8ea00640 1 --2- 172.21.15.47:0/3484514596 >> v2:172.21.15.116:3300/0 conn(0x7f0e900cea80 0x7f0e9006c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.074+0000 7f0e8fe00640 1 --2- 172.21.15.47:0/3484514596 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e9015c390 0x7f0e9015c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.074+0000 7f0e8f400640 1 -- 172.21.15.47:0/3484514596 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e9015c390 msgr2=0x7f0e9015c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:31.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.074+0000 7f0e8f400640 1 --2- 172.21.15.47:0/3484514596 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e9015c390 0x7f0e9015c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.074+0000 7f0e8f400640 1 -- 172.21.15.47:0/3484514596 >> v2:172.21.15.116:3300/0 conn(0x7f0e900cea80 msgr2=0x7f0e9006c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:31.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.074+0000 7f0e8f400640 1 --2- 172.21.15.47:0/3484514596 >> v2:172.21.15.116:3300/0 conn(0x7f0e900cea80 0x7f0e9006c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.074+0000 7f0e8f400640 1 -- 172.21.15.47:0/3484514596 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e900709e0 con 0x7f0e90145590 2024-02-19T15:24:31.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.074+0000 7f0e8ea00640 1 --2- 172.21.15.47:0/3484514596 >> v2:172.21.15.116:3300/0 conn(0x7f0e900cea80 0x7f0e9006c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:31.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.078+0000 7f0e8f400640 1 --2- 172.21.15.47:0/3484514596 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e90145590 0x7f0e900727b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f0e9006c770 tx=0x7f0e8001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3087958368b43797 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:31.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.078+0000 7f0e8e000640 1 -- 172.21.15.47:0/3484514596 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e800ac630 con 0x7f0e90145590 2024-02-19T15:24:31.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.078+0000 7f0e8e000640 1 -- 172.21.15.47:0/3484514596 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e800ac790 con 0x7f0e90145590 2024-02-19T15:24:31.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.078+0000 7f0e8e000640 1 -- 172.21.15.47:0/3484514596 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e800acb20 con 0x7f0e90145590 2024-02-19T15:24:31.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.078+0000 7f0e96200640 1 -- 172.21.15.47:0/3484514596 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e90145590 msgr2=0x7f0e900727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:31.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.078+0000 7f0e96200640 1 --2- 172.21.15.47:0/3484514596 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e90145590 0x7f0e900727b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f0e9006c770 tx=0x7f0e8001c520 comp rx=0 tx=0).stop 2024-02-19T15:24:31.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.078+0000 7f0e96200640 1 -- 172.21.15.47:0/3484514596 shutdown_connections 2024-02-19T15:24:31.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.078+0000 7f0e96200640 1 --2- 172.21.15.47:0/3484514596 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e9015c390 0x7f0e9015c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.078+0000 7f0e96200640 1 --2- 172.21.15.47:0/3484514596 >> v2:172.21.15.116:3300/0 conn(0x7f0e900cea80 0x7f0e9006c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.078+0000 7f0e96200640 1 --2- 172.21.15.47:0/3484514596 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e90145590 0x7f0e900727b0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.078+0000 7f0e96200640 1 -- 172.21.15.47:0/3484514596 >> 172.21.15.47:0/3484514596 conn(0x7f0e900b16a0 msgr2=0x7f0e900b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:31.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.078+0000 7f0e96200640 1 -- 172.21.15.47:0/3484514596 shutdown_connections 2024-02-19T15:24:31.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.078+0000 7f0e96200640 1 -- 172.21.15.47:0/3484514596 wait complete. 2024-02-19T15:24:31.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e96200640 1 Processor -- start 2024-02-19T15:24:31.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e96200640 1 -- start start 2024-02-19T15:24:31.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e96200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e900cea80 0x7f0e900ab130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e96200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e90145590 0x7f0e9015b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e96200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e9015c390 0x7f0e9013cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e96200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e900a9e40 con 0x7f0e90145590 2024-02-19T15:24:31.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e96200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e900ab670 con 0x7f0e900cea80 2024-02-19T15:24:31.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e96200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e900aa090 con 0x7f0e9015c390 2024-02-19T15:24:31.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e8ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e90145590 0x7f0e9015b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e8ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e90145590 0x7f0e9015b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51060/0 (socket says 172.21.15.47:51060) 2024-02-19T15:24:31.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e8ea00640 1 -- 172.21.15.47:0/4110693624 learned_addr learned my addr 172.21.15.47:0/4110693624 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:31.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e8f400640 1 --2- 172.21.15.47:0/4110693624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e900cea80 0x7f0e900ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e8fe00640 1 --2- 172.21.15.47:0/4110693624 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e9015c390 0x7f0e9013cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e8ea00640 1 -- 172.21.15.47:0/4110693624 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e9015c390 msgr2=0x7f0e9013cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:31.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e8ea00640 1 --2- 172.21.15.47:0/4110693624 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e9015c390 0x7f0e9013cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e8ea00640 1 -- 172.21.15.47:0/4110693624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e900cea80 msgr2=0x7f0e900ab130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:31.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e8ea00640 1 --2- 172.21.15.47:0/4110693624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e900cea80 0x7f0e900ab130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e8ea00640 1 -- 172.21.15.47:0/4110693624 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e800094d0 con 0x7f0e90145590 2024-02-19T15:24:31.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e8ea00640 1 --2- 172.21.15.47:0/4110693624 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e90145590 0x7f0e9015b680 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c00e810 tx=0x7f0e7c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:31.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e87400640 1 -- 172.21.15.47:0/4110693624 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c004030 con 0x7f0e90145590 2024-02-19T15:24:31.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e87400640 1 -- 172.21.15.47:0/4110693624 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e7c004190 con 0x7f0e90145590 2024-02-19T15:24:31.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e96200640 1 -- 172.21.15.47:0/4110693624 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e900aa370 con 0x7f0e90145590 2024-02-19T15:24:31.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e87400640 1 -- 172.21.15.47:0/4110693624 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c004520 con 0x7f0e90145590 2024-02-19T15:24:31.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.082+0000 7f0e96200640 1 -- 172.21.15.47:0/4110693624 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e900aa930 con 0x7f0e90145590 2024-02-19T15:24:31.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.086+0000 7f0e96200640 1 -- 172.21.15.47:0/4110693624 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e900cf2e0 con 0x7f0e90145590 2024-02-19T15:24:31.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.086+0000 7f0e87400640 1 -- 172.21.15.47:0/4110693624 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0e7c00c6d0 con 0x7f0e90145590 2024-02-19T15:24:31.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.086+0000 7f0e87400640 1 --2- 172.21.15.47:0/4110693624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0e70066e40 0x7f0e70069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.086+0000 7f0e87400640 1 -- 172.21.15.47:0/4110693624 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e7c084690 con 0x7f0e90145590 2024-02-19T15:24:31.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.086+0000 7f0e8f400640 1 --2- 172.21.15.47:0/4110693624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0e70066e40 0x7f0e70069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.090+0000 7f0e8f400640 1 --2- 172.21.15.47:0/4110693624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0e70066e40 0x7f0e70069300 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0e800ae000 tx=0x7f0e80008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:31.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.094+0000 7f0e87400640 1 -- 172.21.15.47:0/4110693624 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0e900cf2e0 con 0x7f0e90145590 2024-02-19T15:24:31.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.290+0000 7f0e96200640 1 -- 172.21.15.47:0/4110693624 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e900cf2e0 con 0x7f0e90145590 2024-02-19T15:24:31.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.294+0000 7f0e87400640 1 -- 172.21.15.47:0/4110693624 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f0e900cf2e0 con 0x7f0e90145590 2024-02-19T15:24:31.296 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:31.296 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5384,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2825837364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2825837364},{"type":"v1","addr":"172.21.15.47:6835","nonce":2825837364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2251385237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2251385237},{"type":"v1","addr":"172.21.15.116:6837","nonce":2251385237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/2438199468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2438199468},{"type":"v1","addr":"172.21.15.116:6836","nonce":2438199468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:24:25.161496+0000","modified":"2024-02-19T15:24:28.184021+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5910":{"gid":5910,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6834/61582103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":61582103},{"type":"v1","addr":"172.21.15.47:6834","nonce":61582103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-19T15:24:31.296 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 33 2024-02-19T15:24:31.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.294+0000 7f0e96200640 1 -- 172.21.15.47:0/4110693624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0e70066e40 msgr2=0x7f0e70069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:31.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.294+0000 7f0e96200640 1 --2- 172.21.15.47:0/4110693624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0e70066e40 0x7f0e70069300 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0e800ae000 tx=0x7f0e80008040 comp rx=0 tx=0).stop 2024-02-19T15:24:31.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.294+0000 7f0e96200640 1 -- 172.21.15.47:0/4110693624 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e90145590 msgr2=0x7f0e9015b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:31.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.294+0000 7f0e96200640 1 --2- 172.21.15.47:0/4110693624 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e90145590 0x7f0e9015b680 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c00e810 tx=0x7f0e7c00ecd0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.294+0000 7f0e96200640 1 -- 172.21.15.47:0/4110693624 shutdown_connections 2024-02-19T15:24:31.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.294+0000 7f0e96200640 1 --2- 172.21.15.47:0/4110693624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0e70066e40 0x7f0e70069300 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.294+0000 7f0e96200640 1 --2- 172.21.15.47:0/4110693624 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e9015c390 0x7f0e9013cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.294+0000 7f0e96200640 1 --2- 172.21.15.47:0/4110693624 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e90145590 0x7f0e9015b680 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.294+0000 7f0e96200640 1 --2- 172.21.15.47:0/4110693624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e900cea80 0x7f0e900ab130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.294+0000 7f0e96200640 1 -- 172.21.15.47:0/4110693624 >> 172.21.15.47:0/4110693624 conn(0x7f0e900b16a0 msgr2=0x7f0e90141710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:31.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.294+0000 7f0e96200640 1 -- 172.21.15.47:0/4110693624 shutdown_connections 2024-02-19T15:24:31.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.294+0000 7f0e96200640 1 -- 172.21.15.47:0/4110693624 wait complete. 2024-02-19T15:24:31.720 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 33, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:24:25.161496+0000', 'modified': '2024-02-19T15:24:28.184021+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5910}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5910': {'gid': 5910, 'name': 'a', 'rank': 0, 'incarnation': 29, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6834/61582103', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6832', 'nonce': 61582103}, {'type': 'v1', 'addr': '172.21.15.47:6834', 'nonce': 61582103}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:24:31.721 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T15:24:31.721 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-19T15:24:31.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.878+0000 7f438dc00640 1 Processor -- start 2024-02-19T15:24:31.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.878+0000 7f438dc00640 1 -- start start 2024-02-19T15:24:31.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.878+0000 7f438dc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43880cd020 0x7f43880ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.878+0000 7f438dc00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f438815e100 0x7f438806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.878+0000 7f438dc00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43880ce6e0 con 0x7f438815dd20 2024-02-19T15:24:31.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.878+0000 7f438dc00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43880acec0 con 0x7f438815e100 2024-02-19T15:24:31.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.878+0000 7f438dc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438806c300 con 0x7f43880cd020 2024-02-19T15:24:31.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.878+0000 7f438d200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f438815e100 0x7f438806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.878+0000 7f4387e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f438815dd20 0x7f43880cc910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35986/0 (socket says 172.21.15.47:35986) 2024-02-19T15:24:31.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.878+0000 7f4387e00640 1 -- 172.21.15.47:0/2999536365 learned_addr learned my addr 172.21.15.47:0/2999536365 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:31.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.878+0000 7f4387400640 1 --2- 172.21.15.47:0/2999536365 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43880cd020 0x7f43880ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.878+0000 7f4386a00640 1 -- 172.21.15.47:0/2999536365 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205409797 0 0) 0x7f43880ce6e0 con 0x7f438815dd20 2024-02-19T15:24:31.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.878+0000 7f4386a00640 1 -- 172.21.15.47:0/2999536365 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f436c003530 con 0x7f438815dd20 2024-02-19T15:24:31.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.878+0000 7f438d200640 1 -- 172.21.15.47:0/2999536365 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43880cd020 msgr2=0x7f43880ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:31.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.878+0000 7f438d200640 1 --2- 172.21.15.47:0/2999536365 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43880cd020 0x7f43880ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.878+0000 7f438d200640 1 -- 172.21.15.47:0/2999536365 >> v1:172.21.15.47:6789/0 conn(0x7f438815dd20 legacy=0x7f43880cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:31.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.878+0000 7f438d200640 1 -- 172.21.15.47:0/2999536365 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438806c5e0 con 0x7f438815e100 2024-02-19T15:24:31.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.882+0000 7f438d200640 1 --2- 172.21.15.47:0/2999536365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f438815e100 0x7f438806b9a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f4380009d90 tx=0x7f438001c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c47e5d2576581c9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:31.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.882+0000 7f4386a00640 1 -- 172.21.15.47:0/2999536365 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43800a5870 con 0x7f438815e100 2024-02-19T15:24:31.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.882+0000 7f4386a00640 1 -- 172.21.15.47:0/2999536365 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43800a59d0 con 0x7f438815e100 2024-02-19T15:24:31.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.882+0000 7f4386a00640 1 -- 172.21.15.47:0/2999536365 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43800a5d60 con 0x7f438815e100 2024-02-19T15:24:31.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.882+0000 7f438dc00640 1 -- 172.21.15.47:0/2999536365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f438815e100 msgr2=0x7f438806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:31.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.882+0000 7f438dc00640 1 --2- 172.21.15.47:0/2999536365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f438815e100 0x7f438806b9a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f4380009d90 tx=0x7f438001c820 comp rx=0 tx=0).stop 2024-02-19T15:24:31.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.882+0000 7f438dc00640 1 -- 172.21.15.47:0/2999536365 shutdown_connections 2024-02-19T15:24:31.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.882+0000 7f438dc00640 1 --2- 172.21.15.47:0/2999536365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f438815e100 0x7f438806b9a0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.882+0000 7f438dc00640 1 --2- 172.21.15.47:0/2999536365 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43880cd020 0x7f43880ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.882+0000 7f438dc00640 1 -- 172.21.15.47:0/2999536365 >> 172.21.15.47:0/2999536365 conn(0x7f43880b1be0 msgr2=0x7f43880b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:31.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.882+0000 7f438dc00640 1 -- 172.21.15.47:0/2999536365 shutdown_connections 2024-02-19T15:24:31.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.882+0000 7f438dc00640 1 -- 172.21.15.47:0/2999536365 wait complete. 2024-02-19T15:24:31.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.882+0000 7f438dc00640 1 Processor -- start 2024-02-19T15:24:31.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.882+0000 7f438dc00640 1 -- start start 2024-02-19T15:24:31.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f438dc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43880cd020 0x7f43880b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f438dc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f438815dd20 0x7f4388159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f438dc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f438815e100 0x7f43880b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f438dc00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4388150e70 con 0x7f438815dd20 2024-02-19T15:24:31.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f438dc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438806ee80 con 0x7f43880cd020 2024-02-19T15:24:31.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f438dc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438815a190 con 0x7f438815e100 2024-02-19T15:24:31.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f4387400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f438815dd20 0x7f4388159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f4387400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f438815dd20 0x7f4388159c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51064/0 (socket says 172.21.15.47:51064) 2024-02-19T15:24:31.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f4387400640 1 -- 172.21.15.47:0/4199384018 learned_addr learned my addr 172.21.15.47:0/4199384018 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:31.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f4387e00640 1 --2- 172.21.15.47:0/4199384018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43880cd020 0x7f43880b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f4387400640 1 -- 172.21.15.47:0/4199384018 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f438815e100 msgr2=0x7f43880b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:31.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f438d200640 1 --2- 172.21.15.47:0/4199384018 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f438815e100 0x7f43880b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f4387400640 1 --2- 172.21.15.47:0/4199384018 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f438815e100 0x7f43880b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f4387400640 1 -- 172.21.15.47:0/4199384018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43880cd020 msgr2=0x7f43880b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:31.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f4387400640 1 --2- 172.21.15.47:0/4199384018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43880cd020 0x7f43880b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f4387400640 1 -- 172.21.15.47:0/4199384018 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4380009a70 con 0x7f438815dd20 2024-02-19T15:24:31.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f4387e00640 1 --2- 172.21.15.47:0/4199384018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43880cd020 0x7f43880b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:31.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f438d200640 1 --2- 172.21.15.47:0/4199384018 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f438815e100 0x7f43880b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:31.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f4387400640 1 --2- 172.21.15.47:0/4199384018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f438815dd20 0x7f4388159c50 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f437400d5f0 tx=0x7f437400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:31.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f437fe00640 1 -- 172.21.15.47:0/4199384018 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43740125f0 con 0x7f438815dd20 2024-02-19T15:24:31.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f437fe00640 1 -- 172.21.15.47:0/4199384018 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4374012750 con 0x7f438815dd20 2024-02-19T15:24:31.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f437fe00640 1 -- 172.21.15.47:0/4199384018 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4374012ae0 con 0x7f438815dd20 2024-02-19T15:24:31.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f438dc00640 1 -- 172.21.15.47:0/4199384018 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4388072a80 con 0x7f438815dd20 2024-02-19T15:24:31.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.886+0000 7f438dc00640 1 -- 172.21.15.47:0/4199384018 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43880b30a0 con 0x7f438815dd20 2024-02-19T15:24:31.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f437fe00640 1 -- 172.21.15.47:0/4199384018 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4374012c40 con 0x7f438815dd20 2024-02-19T15:24:31.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f439273d000 1 Processor -- start 2024-02-19T15:24:31.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f439273d000 1 -- start start 2024-02-19T15:24:31.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f437fe00640 1 --2- 172.21.15.47:0/4199384018 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4368066e90 0x7f4368069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f437fe00640 1 -- 172.21.15.47:0/4199384018 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4374084e60 con 0x7f438815dd20 2024-02-19T15:24:31.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f438d200640 1 --2- 172.21.15.47:0/4199384018 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4368066e90 0x7f4368069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f439273d000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d4a9920 0x55c95d4a9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f439273d000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d4aa240 0x55c95d45f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f439273d000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d45ff30 0x55c95d460310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f439273d000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c95d327d00 con 0x55c95d45ff30 2024-02-19T15:24:31.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f439273d000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c95d2b9430 con 0x55c95d4aa240 2024-02-19T15:24:31.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f439273d000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c95d2b6960 con 0x55c95d4a9920 2024-02-19T15:24:31.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f4387e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d4a9920 0x55c95d4a9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f4387400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d4aa240 0x55c95d45f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f4387e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d4a9920 0x55c95d4a9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:59038/0 (socket says 172.21.15.47:59038) 2024-02-19T15:24:31.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f4387e00640 1 -- 172.21.15.47:0/4114655578 learned_addr learned my addr 172.21.15.47:0/4114655578 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:31.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f438d200640 1 --2- 172.21.15.47:0/4114655578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d45ff30 0x55c95d460310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f438d200640 1 --2- 172.21.15.47:0/4199384018 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4368066e90 0x7f4368069350 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f43800af000 tx=0x7f4380004ac0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:31.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f4387400640 1 -- 172.21.15.47:0/4114655578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d4a9920 msgr2=0x55c95d4a9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:31.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f4387400640 1 --2- 172.21.15.47:0/4114655578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d4a9920 0x55c95d4a9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f4387400640 1 -- 172.21.15.47:0/4114655578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d45ff30 msgr2=0x55c95d460310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:31.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f4387400640 1 --2- 172.21.15.47:0/4114655578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d45ff30 0x55c95d460310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f4387400640 1 -- 172.21.15.47:0/4114655578 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c95cfa8890 con 0x55c95d4aa240 2024-02-19T15:24:31.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f438d200640 1 --2- 172.21.15.47:0/4114655578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d45ff30 0x55c95d460310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:24:31.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f4387e00640 1 --2- 172.21.15.47:0/4114655578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d4a9920 0x55c95d4a9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:24:31.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.890+0000 7f4387400640 1 --2- 172.21.15.47:0/4114655578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d4aa240 0x55c95d45f9f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4374083400 tx=0x7f4374007380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d55afa553a6d615 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:31.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.894+0000 7f437cc00640 1 -- 172.21.15.47:0/4114655578 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4374059b70 con 0x55c95d4aa240 2024-02-19T15:24:31.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.894+0000 7f437cc00640 1 -- 172.21.15.47:0/4114655578 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437405e130 con 0x55c95d4aa240 2024-02-19T15:24:31.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.894+0000 7f437cc00640 1 -- 172.21.15.47:0/4114655578 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4374052090 con 0x55c95d4aa240 2024-02-19T15:24:31.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.894+0000 7f439273d000 1 -- 172.21.15.47:0/4114655578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d4aa240 msgr2=0x55c95d45f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:31.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.894+0000 7f439273d000 1 --2- 172.21.15.47:0/4114655578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d4aa240 0x55c95d45f9f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4374083400 tx=0x7f4374007380 comp rx=0 tx=0).stop 2024-02-19T15:24:31.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.894+0000 7f439273d000 1 -- 172.21.15.47:0/4114655578 shutdown_connections 2024-02-19T15:24:31.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.894+0000 7f439273d000 1 --2- 172.21.15.47:0/4114655578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d45ff30 0x55c95d460310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.894+0000 7f439273d000 1 --2- 172.21.15.47:0/4114655578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d4aa240 0x55c95d45f9f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.894+0000 7f439273d000 1 --2- 172.21.15.47:0/4114655578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d4a9920 0x55c95d4a9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.894+0000 7f439273d000 1 -- 172.21.15.47:0/4114655578 >> 172.21.15.47:0/4114655578 conn(0x55c95d4662a0 msgr2=0x55c95d45e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:31.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.894+0000 7f439273d000 1 -- 172.21.15.47:0/4114655578 shutdown_connections 2024-02-19T15:24:31.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.894+0000 7f439273d000 1 -- 172.21.15.47:0/4114655578 wait complete. 2024-02-19T15:24:31.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.894+0000 7f439273d000 1 -- start start 2024-02-19T15:24:31.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.894+0000 7f439273d000 1 Processor -- start 2024-02-19T15:24:31.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f439273d000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d45ff30 0x55c95d4926f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f439273d000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d4a9920 0x55c95d463aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f439273d000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4aa240 0x55c95d463fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f439273d000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c95d2b2140 con 0x55c95d4aa240 2024-02-19T15:24:31.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f439273d000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c95d24c630 con 0x55c95d4a9920 2024-02-19T15:24:31.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f439273d000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c95d24c2d0 con 0x55c95d45ff30 2024-02-19T15:24:31.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f439273d000 10 client.0 _ll_register_callbacks cb 0x55c95d3621c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:24:31.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f438d200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4aa240 0x55c95d463fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f438d200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4aa240 0x55c95d463fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51080/0 (socket says 172.21.15.47:51080) 2024-02-19T15:24:31.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f438d200640 1 -- 172.21.15.47:0/2905714653 learned_addr learned my addr 172.21.15.47:0/2905714653 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:31.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f4387400640 1 --2- 172.21.15.47:0/2905714653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d4a9920 0x55c95d463aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f4387e00640 1 --2- 172.21.15.47:0/2905714653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d45ff30 0x55c95d4926f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f438d200640 1 -- 172.21.15.47:0/2905714653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d45ff30 msgr2=0x55c95d4926f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:31.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f438d200640 1 --2- 172.21.15.47:0/2905714653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d45ff30 0x55c95d4926f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f438d200640 1 -- 172.21.15.47:0/2905714653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d4a9920 msgr2=0x55c95d463aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:31.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f438d200640 1 --2- 172.21.15.47:0/2905714653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d4a9920 0x55c95d463aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f438d200640 1 -- 172.21.15.47:0/2905714653 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438806c5e0 con 0x55c95d4aa240 2024-02-19T15:24:31.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f438d200640 1 --2- 172.21.15.47:0/2905714653 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4aa240 0x55c95d463fe0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f43800b0040 tx=0x7f438001b060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=577043b5b91b14e6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:31.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f4371600640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:24:31.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f4371600640 1 -- 172.21.15.47:0/2905714653 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43800b9630 con 0x55c95d4aa240 2024-02-19T15:24:31.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f4371600640 1 -- 172.21.15.47:0/2905714653 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43800b9790 con 0x55c95d4aa240 2024-02-19T15:24:31.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f439273d000 1 -- 172.21.15.47:0/2905714653 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c95d265850 con 0x55c95d4aa240 2024-02-19T15:24:31.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f4371600640 1 -- 172.21.15.47:0/2905714653 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43800b9950 con 0x55c95d4aa240 2024-02-19T15:24:31.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f439273d000 10 client.6051 fetch_fsmap learned FSMap version 33 2024-02-19T15:24:31.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f439273d000 1 -- 172.21.15.47:0/2905714653 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x7f436c003530 con 0x55c95d4aa240 2024-02-19T15:24:31.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f4371600640 1 -- 172.21.15.47:0/2905714653 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c95d265850 con 0x55c95d4aa240 2024-02-19T15:24:31.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f4371600640 1 -- 172.21.15.47:0/2905714653 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 33) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f43800b9ab0 con 0x55c95d4aa240 2024-02-19T15:24:31.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f439273d000 10 client.6051 fetch_fsmap finished waiting for FSMap version 33 2024-02-19T15:24:31.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f439273d000 10 client.6051 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-19T15:24:31.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f439273d000 20 client.6051 populate_metadata read hostname 'smithi047' 2024-02-19T15:24:31.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f439273d000 1 --2- 172.21.15.47:0/2905714653 >> [v2:172.21.15.47:6832/61582103,v1:172.21.15.47:6834/61582103] conn(0x55c95d4649f0 0x55c95d464dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f439273d000 4 client.6051 mds_command: new command op to 5910 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T15:24:31.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f439273d000 1 -- 172.21.15.47:0/2905714653 --> [v2:172.21.15.47:6832/61582103,v1:172.21.15.47:6834/61582103] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c95d2aef20 con 0x55c95d4649f0 2024-02-19T15:24:31.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f4387e00640 1 --2- 172.21.15.47:0/2905714653 >> [v2:172.21.15.47:6832/61582103,v1:172.21.15.47:6834/61582103] conn(0x55c95d4649f0 0x55c95d464dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f4387e00640 1 --2- 172.21.15.47:0/2905714653 >> [v2:172.21.15.47:6832/61582103,v1:172.21.15.47:6834/61582103] conn(0x55c95d4649f0 0x55c95d464dd0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d6c0f80ed2d36c6 server_cookie=2cb222489309b39a in_seq=0 out_seq=0 2024-02-19T15:24:31.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.898+0000 7f4371600640 10 client.6051 ms_handle_connect on v2:172.21.15.47:6832/61582103 2024-02-19T15:24:31.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.902+0000 7f4371600640 1 -- 172.21.15.47:0/2905714653 <== mds.0 v2:172.21.15.47:6832/61582103 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c95d2aef20 con 0x55c95d4649f0 2024-02-19T15:24:31.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.902+0000 7f4371600640 10 client.6051 handle_command_reply: tid=0 2024-02-19T15:24:31.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.902+0000 7f439273d000 1 client.6051 shutdown 2024-02-19T15:24:31.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.902+0000 7f439273d000 1 -- 172.21.15.47:0/2905714653 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4aa240 msgr2=0x55c95d463fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:31.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.902+0000 7f439273d000 1 --2- 172.21.15.47:0/2905714653 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4aa240 0x55c95d463fe0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f43800b0040 tx=0x7f438001b060 comp rx=0 tx=0).stop 2024-02-19T15:24:31.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.902+0000 7f439273d000 1 -- 172.21.15.47:0/2905714653 shutdown_connections 2024-02-19T15:24:31.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.902+0000 7f439273d000 1 --2- 172.21.15.47:0/2905714653 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4aa240 0x55c95d463fe0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.902+0000 7f439273d000 1 --2- 172.21.15.47:0/2905714653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d4a9920 0x55c95d463aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.902+0000 7f439273d000 1 --2- 172.21.15.47:0/2905714653 >> [v2:172.21.15.47:6832/61582103,v1:172.21.15.47:6834/61582103] conn(0x55c95d4649f0 0x55c95d464dd0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.902+0000 7f439273d000 1 --2- 172.21.15.47:0/2905714653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d45ff30 0x55c95d4926f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.902+0000 7f439273d000 1 -- 172.21.15.47:0/2905714653 >> 172.21.15.47:0/2905714653 conn(0x55c95d4662a0 msgr2=0x55c95d45e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:31.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.902+0000 7f4371600640 0 client.6051 ms_handle_reset on v2:172.21.15.47:6832/61582103 2024-02-19T15:24:31.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.902+0000 7f439273d000 1 -- 172.21.15.47:0/2905714653 shutdown_connections 2024-02-19T15:24:31.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.902+0000 7f439273d000 1 -- 172.21.15.47:0/2905714653 wait complete. 2024-02-19T15:24:31.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.902+0000 7f439273d000 20 client.6051 trim_cache size 0 max 16384 2024-02-19T15:24:31.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.918+0000 7f439273d000 1 Processor -- start 2024-02-19T15:24:31.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.918+0000 7f439273d000 1 -- start start 2024-02-19T15:24:31.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.918+0000 7f439273d000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d45ff30 0x55c95d509030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.918+0000 7f439273d000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4a9920 0x55c95d509570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.918+0000 7f439273d000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d4aa240 0x55c95d509ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.918+0000 7f439273d000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c95cf69940 con 0x55c95d4a9920 2024-02-19T15:24:31.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.918+0000 7f439273d000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c95d2b0fd0 con 0x55c95d45ff30 2024-02-19T15:24:31.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.918+0000 7f439273d000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c95d2f21a0 con 0x55c95d4aa240 2024-02-19T15:24:31.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.918+0000 7f4387400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4a9920 0x55c95d509570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.918+0000 7f438d200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d4aa240 0x55c95d509ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.918+0000 7f4387400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4a9920 0x55c95d509570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51096/0 (socket says 172.21.15.47:51096) 2024-02-19T15:24:31.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.918+0000 7f4387400640 1 -- 172.21.15.47:0/162529139 learned_addr learned my addr 172.21.15.47:0/162529139 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:31.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.918+0000 7f4387e00640 1 --2- 172.21.15.47:0/162529139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d45ff30 0x55c95d509030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.918+0000 7f4387400640 1 -- 172.21.15.47:0/162529139 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d4aa240 msgr2=0x55c95d509ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:31.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.918+0000 7f4387400640 1 --2- 172.21.15.47:0/162529139 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d4aa240 0x55c95d509ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.918+0000 7f4387400640 1 -- 172.21.15.47:0/162529139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d45ff30 msgr2=0x55c95d509030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:31.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.918+0000 7f4387400640 1 --2- 172.21.15.47:0/162529139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d45ff30 0x55c95d509030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.918+0000 7f4387400640 1 -- 172.21.15.47:0/162529139 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43880b30a0 con 0x55c95d4a9920 2024-02-19T15:24:31.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.918+0000 7f4387400640 1 --2- 172.21.15.47:0/162529139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4a9920 0x55c95d509570 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f43740074f0 tx=0x7f4374020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b2bbff27de05a2f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:31.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.918+0000 7f4373e00640 1 -- 172.21.15.47:0/162529139 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4374012ef0 con 0x55c95d4a9920 2024-02-19T15:24:31.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f4373e00640 1 -- 172.21.15.47:0/162529139 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437405e130 con 0x55c95d4a9920 2024-02-19T15:24:31.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f4373e00640 1 -- 172.21.15.47:0/162529139 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4374059b70 con 0x55c95d4a9920 2024-02-19T15:24:31.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f439273d000 1 -- 172.21.15.47:0/162529139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4a9920 msgr2=0x55c95d509570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:31.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f439273d000 1 --2- 172.21.15.47:0/162529139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4a9920 0x55c95d509570 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f43740074f0 tx=0x7f4374020000 comp rx=0 tx=0).stop 2024-02-19T15:24:31.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f439273d000 1 -- 172.21.15.47:0/162529139 shutdown_connections 2024-02-19T15:24:31.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f439273d000 1 --2- 172.21.15.47:0/162529139 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d4aa240 0x55c95d509ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f439273d000 1 --2- 172.21.15.47:0/162529139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4a9920 0x55c95d509570 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f439273d000 1 --2- 172.21.15.47:0/162529139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d45ff30 0x55c95d509030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f439273d000 1 -- 172.21.15.47:0/162529139 >> 172.21.15.47:0/162529139 conn(0x55c95d4662a0 msgr2=0x55c95d4bd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:31.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f439273d000 1 -- 172.21.15.47:0/162529139 shutdown_connections 2024-02-19T15:24:31.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f439273d000 1 -- 172.21.15.47:0/162529139 wait complete. 2024-02-19T15:24:31.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f439273d000 1 -- start start 2024-02-19T15:24:31.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f439273d000 1 Processor -- start 2024-02-19T15:24:31.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f439273d000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d45ff30 0x55c95d50ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f439273d000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4a9920 0x55c95d50f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f439273d000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d4aa240 0x55c95d4c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f439273d000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c95d2af950 con 0x55c95d4a9920 2024-02-19T15:24:31.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f439273d000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c95d2ac2c0 con 0x55c95d4aa240 2024-02-19T15:24:31.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f439273d000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c95d2718c0 con 0x55c95d45ff30 2024-02-19T15:24:31.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f439273d000 10 client.0 _ll_register_callbacks cb 0x7f436c0036c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:24:31.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f4387400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4a9920 0x55c95d50f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f4387400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4a9920 0x55c95d50f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51104/0 (socket says 172.21.15.47:51104) 2024-02-19T15:24:31.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f4387400640 1 -- 172.21.15.47:0/1828026601 learned_addr learned my addr 172.21.15.47:0/1828026601 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:31.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f4387e00640 1 --2- 172.21.15.47:0/1828026601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d45ff30 0x55c95d50ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f438d200640 1 --2- 172.21.15.47:0/1828026601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d4aa240 0x55c95d4c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f4387400640 1 -- 172.21.15.47:0/1828026601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d45ff30 msgr2=0x55c95d50ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:31.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f4387400640 1 --2- 172.21.15.47:0/1828026601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d45ff30 0x55c95d50ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f4387400640 1 -- 172.21.15.47:0/1828026601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d4aa240 msgr2=0x55c95d4c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:31.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f4387400640 1 --2- 172.21.15.47:0/1828026601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d4aa240 0x55c95d4c7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f4387400640 1 -- 172.21.15.47:0/1828026601 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c95cfa8890 con 0x55c95d4a9920 2024-02-19T15:24:31.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f4387e00640 1 --2- 172.21.15.47:0/1828026601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d45ff30 0x55c95d50ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:31.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f438d200640 1 --2- 172.21.15.47:0/1828026601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d4aa240 0x55c95d4c7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:31.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f4387400640 1 --2- 172.21.15.47:0/1828026601 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4a9920 0x55c95d50f220 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f4374083370 tx=0x7f4374000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a3d1154b5879944 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:31.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f4371600640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:24:31.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f4371600640 1 -- 172.21.15.47:0/1828026601 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4374059b70 con 0x55c95d4a9920 2024-02-19T15:24:31.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f4371600640 1 -- 172.21.15.47:0/1828026601 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f437405dca0 con 0x55c95d4a9920 2024-02-19T15:24:31.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.922+0000 7f439273d000 1 -- 172.21.15.47:0/1828026601 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c95d4ba370 con 0x55c95d4a9920 2024-02-19T15:24:31.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f4371600640 1 -- 172.21.15.47:0/1828026601 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f437405de60 con 0x55c95d4a9920 2024-02-19T15:24:31.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f439273d000 10 client.6057 fetch_fsmap learned FSMap version 33 2024-02-19T15:24:31.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f439273d000 1 -- 172.21.15.47:0/1828026601 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x55c95d271e70 con 0x55c95d4a9920 2024-02-19T15:24:31.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f4371600640 1 -- 172.21.15.47:0/1828026601 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c95d4ba370 con 0x55c95d4a9920 2024-02-19T15:24:31.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f4371600640 1 -- 172.21.15.47:0/1828026601 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 33) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4374054e70 con 0x55c95d4a9920 2024-02-19T15:24:31.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f439273d000 10 client.6057 fetch_fsmap finished waiting for FSMap version 33 2024-02-19T15:24:31.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f439273d000 10 client.6057 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-19T15:24:31.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f439273d000 20 client.6057 populate_metadata read hostname 'smithi047' 2024-02-19T15:24:31.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f439273d000 1 --2- 172.21.15.47:0/1828026601 >> [v2:172.21.15.47:6832/61582103,v1:172.21.15.47:6834/61582103] conn(0x55c95d4c80d0 0x55c95d4c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:31.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f439273d000 4 client.6057 mds_command: new command op to 5910 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T15:24:31.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f439273d000 1 -- 172.21.15.47:0/1828026601 --> [v2:172.21.15.47:6832/61582103,v1:172.21.15.47:6834/61582103] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c95d237e80 con 0x55c95d4c80d0 2024-02-19T15:24:31.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f4387e00640 1 --2- 172.21.15.47:0/1828026601 >> [v2:172.21.15.47:6832/61582103,v1:172.21.15.47:6834/61582103] conn(0x55c95d4c80d0 0x55c95d4c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:31.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f4371600640 10 client.6057 ms_handle_connect on v2:172.21.15.47:6832/61582103 2024-02-19T15:24:31.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f4387e00640 1 --2- 172.21.15.47:0/1828026601 >> [v2:172.21.15.47:6832/61582103,v1:172.21.15.47:6834/61582103] conn(0x55c95d4c80d0 0x55c95d4c84b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0c36514d5385db3 server_cookie=f6d972482059b027 in_seq=0 out_seq=0 2024-02-19T15:24:31.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f4371600640 1 -- 172.21.15.47:0/1828026601 <== mds.0 v2:172.21.15.47:6832/61582103 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55c95d237e80 con 0x55c95d4c80d0 2024-02-19T15:24:31.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f4371600640 10 client.6057 handle_command_reply: tid=0 2024-02-19T15:24:31.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f439273d000 1 client.6057 shutdown 2024-02-19T15:24:31.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f439273d000 1 -- 172.21.15.47:0/1828026601 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4a9920 msgr2=0x55c95d50f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:31.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f439273d000 1 --2- 172.21.15.47:0/1828026601 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4a9920 0x55c95d50f220 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f4374083370 tx=0x7f4374000bd0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f439273d000 1 -- 172.21.15.47:0/1828026601 shutdown_connections 2024-02-19T15:24:31.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f439273d000 1 --2- 172.21.15.47:0/1828026601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95d4aa240 0x55c95d4c7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f439273d000 1 --2- 172.21.15.47:0/1828026601 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c95d4a9920 0x55c95d50f220 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f439273d000 1 --2- 172.21.15.47:0/1828026601 >> [v2:172.21.15.47:6832/61582103,v1:172.21.15.47:6834/61582103] conn(0x55c95d4c80d0 0x55c95d4c84b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f439273d000 1 --2- 172.21.15.47:0/1828026601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c95d45ff30 0x55c95d50ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f4371600640 0 client.6057 ms_handle_reset on v2:172.21.15.47:6832/61582103 2024-02-19T15:24:31.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f439273d000 1 -- 172.21.15.47:0/1828026601 >> 172.21.15.47:0/1828026601 conn(0x55c95d4662a0 msgr2=0x55c95d4bdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:31.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f439273d000 1 -- 172.21.15.47:0/1828026601 shutdown_connections 2024-02-19T15:24:31.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f439273d000 1 -- 172.21.15.47:0/1828026601 wait complete. 2024-02-19T15:24:31.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f439273d000 20 client.6057 trim_cache size 0 max 16384 2024-02-19T15:24:31.933 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:24:31.933 INFO:teuthology.orchestra.run.smithi047.stdout: "cluster_fsid": "ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e", 2024-02-19T15:24:31.933 INFO:teuthology.orchestra.run.smithi047.stdout: "whoami": 0, 2024-02-19T15:24:31.933 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 5910, 2024-02-19T15:24:31.933 INFO:teuthology.orchestra.run.smithi047.stdout: "want_state": "up:active", 2024-02-19T15:24:31.933 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "up:active", 2024-02-19T15:24:31.933 INFO:teuthology.orchestra.run.smithi047.stdout: "fs_name": "cephfs", 2024-02-19T15:24:31.933 INFO:teuthology.orchestra.run.smithi047.stdout: "rank_uptime": 6.1415743540000003, 2024-02-19T15:24:31.933 INFO:teuthology.orchestra.run.smithi047.stdout: "mdsmap_epoch": 33, 2024-02-19T15:24:31.933 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch": 57, 2024-02-19T15:24:31.933 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch_barrier": 57, 2024-02-19T15:24:31.933 INFO:teuthology.orchestra.run.smithi047.stdout: "uptime": 12.06509896 2024-02-19T15:24:31.934 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:24:31.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f438dc00640 1 -- 172.21.15.47:0/4199384018 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4368066e90 msgr2=0x7f4368069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:31.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f438dc00640 1 --2- 172.21.15.47:0/4199384018 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4368066e90 0x7f4368069350 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f43800af000 tx=0x7f4380004ac0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f438dc00640 1 -- 172.21.15.47:0/4199384018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f438815dd20 msgr2=0x7f4388159c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:31.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f438dc00640 1 --2- 172.21.15.47:0/4199384018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f438815dd20 0x7f4388159c50 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f437400d5f0 tx=0x7f437400dab0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f438dc00640 1 -- 172.21.15.47:0/4199384018 shutdown_connections 2024-02-19T15:24:31.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f438dc00640 1 --2- 172.21.15.47:0/4199384018 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4368066e90 0x7f4368069350 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f438dc00640 1 --2- 172.21.15.47:0/4199384018 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f438815e100 0x7f43880b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f438dc00640 1 --2- 172.21.15.47:0/4199384018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f438815dd20 0x7f4388159c50 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f438dc00640 1 --2- 172.21.15.47:0/4199384018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43880cd020 0x7f43880b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:31.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f438dc00640 1 -- 172.21.15.47:0/4199384018 >> 172.21.15.47:0/4199384018 conn(0x7f43880b1be0 msgr2=0x7f43880b0880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:31.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f438dc00640 1 -- 172.21.15.47:0/4199384018 shutdown_connections 2024-02-19T15:24:31.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:31.926+0000 7f438dc00640 1 -- 172.21.15.47:0/4199384018 wait complete. 2024-02-19T15:24:31.949 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-02-19T15:24:32.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.086+0000 7f8a01000640 1 Processor -- start 2024-02-19T15:24:32.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.086+0000 7f8a01000640 1 -- start start 2024-02-19T15:24:32.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.086+0000 7f8a01000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f89fc0cef90 0x7f89fc0ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:32.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.086+0000 7f8a01000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89fc0ce930 0x7f89fc0cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:32.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.086+0000 7f8a01000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89fc0d10a0 con 0x7f89fc0b2fc0 2024-02-19T15:24:32.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.086+0000 7f8a01000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89fc0cedc0 con 0x7f89fc0cef90 2024-02-19T15:24:32.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.086+0000 7f8a01000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89fc0d0e80 con 0x7f89fc0ce930 2024-02-19T15:24:32.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.086+0000 7f89f3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f89fc0cef90 0x7f89fc0ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:32.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.086+0000 7f89f3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f89fc0cef90 0x7f89fc0ce3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51116/0 (socket says 172.21.15.47:51116) 2024-02-19T15:24:32.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.086+0000 7f89f3e00640 1 -- 172.21.15.47:0/4046547650 learned_addr learned my addr 172.21.15.47:0/4046547650 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:32.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.090+0000 7f89fa800640 1 --2- 172.21.15.47:0/4046547650 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89fc0ce930 0x7f89fc0cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:32.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.090+0000 7f89f3e00640 1 -- 172.21.15.47:0/4046547650 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89fc0ce930 msgr2=0x7f89fc0cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:32.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.090+0000 7f89f3e00640 1 --2- 172.21.15.47:0/4046547650 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89fc0ce930 0x7f89fc0cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:32.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.090+0000 7f89f3e00640 1 -- 172.21.15.47:0/4046547650 >> v1:172.21.15.47:6789/0 conn(0x7f89fc0b2fc0 legacy=0x7f89fc0d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:32.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.090+0000 7f89f3e00640 1 -- 172.21.15.47:0/4046547650 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89fc0b33a0 con 0x7f89fc0cef90 2024-02-19T15:24:32.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.090+0000 7f89f3e00640 1 --2- 172.21.15.47:0/4046547650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89fc0cef90 0x7f89fc0ce3f0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009870 tx=0x7f89e401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=576306ca6f1af645 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:32.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.090+0000 7f89f9400640 1 -- 172.21.15.47:0/4046547650 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e40a6040 con 0x7f89fc0cef90 2024-02-19T15:24:32.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.090+0000 7f89f9400640 1 -- 172.21.15.47:0/4046547650 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e40aa600 con 0x7f89fc0cef90 2024-02-19T15:24:32.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.090+0000 7f89f9400640 1 -- 172.21.15.47:0/4046547650 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e40aa990 con 0x7f89fc0cef90 2024-02-19T15:24:32.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.090+0000 7f8a01000640 1 -- 172.21.15.47:0/4046547650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89fc0cef90 msgr2=0x7f89fc0ce3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:32.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.090+0000 7f8a01000640 1 --2- 172.21.15.47:0/4046547650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89fc0cef90 0x7f89fc0ce3f0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009870 tx=0x7f89e401c4a0 comp rx=0 tx=0).stop 2024-02-19T15:24:32.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f8a01000640 1 -- 172.21.15.47:0/4046547650 shutdown_connections 2024-02-19T15:24:32.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f8a01000640 1 --2- 172.21.15.47:0/4046547650 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89fc0ce930 0x7f89fc0cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:32.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f8a01000640 1 --2- 172.21.15.47:0/4046547650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89fc0cef90 0x7f89fc0ce3f0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:32.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f8a01000640 1 -- 172.21.15.47:0/4046547650 >> 172.21.15.47:0/4046547650 conn(0x7f89fc0d1b70 msgr2=0x7f89fc0d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:32.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f8a01000640 1 -- 172.21.15.47:0/4046547650 shutdown_connections 2024-02-19T15:24:32.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f8a01000640 1 -- 172.21.15.47:0/4046547650 wait complete. 2024-02-19T15:24:32.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f8a01000640 1 Processor -- start 2024-02-19T15:24:32.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f8a01000640 1 -- start start 2024-02-19T15:24:32.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f8a01000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89fc0b2fc0 0x7f89fc0c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:32.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f8a01000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89fc0ce930 0x7f89fc15f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:32.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f8a01000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89fc0cef90 0x7f89fc1515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:32.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f8a01000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89fc15f710 con 0x7f89fc0b2fc0 2024-02-19T15:24:32.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f8a01000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89fc0c9ef0 con 0x7f89fc0cef90 2024-02-19T15:24:32.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f8a01000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89fc151af0 con 0x7f89fc0ce930 2024-02-19T15:24:32.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f89f9e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89fc0b2fc0 0x7f89fc0c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:32.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f89f9e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89fc0b2fc0 0x7f89fc0c8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51132/0 (socket says 172.21.15.47:51132) 2024-02-19T15:24:32.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f89f9e00640 1 -- 172.21.15.47:0/1264843351 learned_addr learned my addr 172.21.15.47:0/1264843351 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:32.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f89f3e00640 1 --2- 172.21.15.47:0/1264843351 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89fc0ce930 0x7f89fc15f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:32.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f89fa800640 1 --2- 172.21.15.47:0/1264843351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89fc0cef90 0x7f89fc1515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:32.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f89f9e00640 1 -- 172.21.15.47:0/1264843351 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89fc0ce930 msgr2=0x7f89fc15f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:32.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f89f9e00640 1 --2- 172.21.15.47:0/1264843351 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89fc0ce930 0x7f89fc15f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:32.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f89f9e00640 1 -- 172.21.15.47:0/1264843351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89fc0cef90 msgr2=0x7f89fc1515b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:32.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f89f9e00640 1 --2- 172.21.15.47:0/1264843351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89fc0cef90 0x7f89fc1515b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:32.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f89f9e00640 1 -- 172.21.15.47:0/1264843351 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e40094d0 con 0x7f89fc0b2fc0 2024-02-19T15:24:32.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f89f9e00640 1 --2- 172.21.15.47:0/1264843351 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89fc0b2fc0 0x7f89fc0c8c40 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f89ec00c370 tx=0x7f89ec00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:32.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f89f2000640 1 -- 172.21.15.47:0/1264843351 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec0052d0 con 0x7f89fc0b2fc0 2024-02-19T15:24:32.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f89f2000640 1 -- 172.21.15.47:0/1264843351 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89ec005430 con 0x7f89fc0b2fc0 2024-02-19T15:24:32.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f8a01000640 1 -- 172.21.15.47:0/1264843351 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89fc0c92e0 con 0x7f89fc0b2fc0 2024-02-19T15:24:32.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.094+0000 7f89f2000640 1 -- 172.21.15.47:0/1264843351 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec007260 con 0x7f89fc0b2fc0 2024-02-19T15:24:32.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.098+0000 7f8a01000640 1 -- 172.21.15.47:0/1264843351 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89fc0c98a0 con 0x7f89fc0b2fc0 2024-02-19T15:24:32.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.098+0000 7f8a01000640 1 -- 172.21.15.47:0/1264843351 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89fc075360 con 0x7f89fc0b2fc0 2024-02-19T15:24:32.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.098+0000 7f89f2000640 1 -- 172.21.15.47:0/1264843351 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f89ec0073c0 con 0x7f89fc0b2fc0 2024-02-19T15:24:32.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.098+0000 7f89f2000640 1 --2- 172.21.15.47:0/1264843351 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f89d8066e40 0x7f89d8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:32.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.098+0000 7f89f2000640 1 -- 172.21.15.47:0/1264843351 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89ec083dd0 con 0x7f89fc0b2fc0 2024-02-19T15:24:32.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.098+0000 7f89f3e00640 1 --2- 172.21.15.47:0/1264843351 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f89d8066e40 0x7f89d8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:32.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.102+0000 7f89f3e00640 1 --2- 172.21.15.47:0/1264843351 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f89d8066e40 0x7f89d8069300 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f89e40ae000 tx=0x7f89e40a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:32.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.102+0000 7f89f2000640 1 -- 172.21.15.47:0/1264843351 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f89ec050bc0 con 0x7f89fc0b2fc0 2024-02-19T15:24:32.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.302+0000 7f8a01000640 1 -- 172.21.15.47:0/1264843351 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f89fc0b0600 con 0x7f89fc0b2fc0 2024-02-19T15:24:32.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.302+0000 7f89f2000640 1 -- 172.21.15.47:0/1264843351 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v33) v1 ==== 53+0+73 (secure 0 0 0) 0x7f89ec085070 con 0x7f89fc0b2fc0 2024-02-19T15:24:32.305 INFO:teuthology.orchestra.run.smithi047.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-02-19T15:24:32.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.306+0000 7f8a01000640 1 -- 172.21.15.47:0/1264843351 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f89d8066e40 msgr2=0x7f89d8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:32.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.306+0000 7f8a01000640 1 --2- 172.21.15.47:0/1264843351 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f89d8066e40 0x7f89d8069300 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f89e40ae000 tx=0x7f89e40a7000 comp rx=0 tx=0).stop 2024-02-19T15:24:32.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.306+0000 7f8a01000640 1 -- 172.21.15.47:0/1264843351 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89fc0b2fc0 msgr2=0x7f89fc0c8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:32.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.306+0000 7f8a01000640 1 --2- 172.21.15.47:0/1264843351 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89fc0b2fc0 0x7f89fc0c8c40 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f89ec00c370 tx=0x7f89ec00c830 comp rx=0 tx=0).stop 2024-02-19T15:24:32.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.306+0000 7f8a01000640 1 -- 172.21.15.47:0/1264843351 shutdown_connections 2024-02-19T15:24:32.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.306+0000 7f8a01000640 1 --2- 172.21.15.47:0/1264843351 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f89d8066e40 0x7f89d8069300 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:32.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.306+0000 7f8a01000640 1 --2- 172.21.15.47:0/1264843351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89fc0cef90 0x7f89fc1515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:32.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.306+0000 7f8a01000640 1 --2- 172.21.15.47:0/1264843351 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89fc0ce930 0x7f89fc15f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:32.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.306+0000 7f8a01000640 1 --2- 172.21.15.47:0/1264843351 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89fc0b2fc0 0x7f89fc0c8c40 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:32.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.306+0000 7f8a01000640 1 -- 172.21.15.47:0/1264843351 >> 172.21.15.47:0/1264843351 conn(0x7f89fc0d1b70 msgr2=0x7f89fc06bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:32.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.306+0000 7f8a01000640 1 -- 172.21.15.47:0/1264843351 shutdown_connections 2024-02-19T15:24:32.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:32.306+0000 7f8a01000640 1 -- 172.21.15.47:0/1264843351 wait complete. 2024-02-19T15:24:32.326 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-02-19T15:24:32.326 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T15:24:32.326 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi047.front.sepia.ceph.com 2024-02-19T15:24:32.326 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-19T15:24:32.327 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-19T15:24:32.327 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-02-19T15:24:32.327 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-02-19T15:24:32.327 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T15:24:32.327 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:24:32.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:32.327 DEBUG:teuthology.orchestra.run.smithi047:> ip addr 2024-02-19T15:24:32.336 INFO:teuthology.orchestra.run.smithi047.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T15:24:32.336 INFO:teuthology.orchestra.run.smithi047.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T15:24:32.336 INFO:teuthology.orchestra.run.smithi047.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T15:24:32.337 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:24:32.337 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 ::1/128 scope host 2024-02-19T15:24:32.337 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:24:32.337 INFO:teuthology.orchestra.run.smithi047.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:24:32.337 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:78 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:24:32.337 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f0 2024-02-19T15:24:32.337 INFO:teuthology.orchestra.run.smithi047.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T15:24:32.337 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8e brd ff:ff:ff:ff:ff:ff 2024-02-19T15:24:32.337 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f0 2024-02-19T15:24:32.337 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.21.15.47/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-02-19T15:24:32.337 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft 42343sec preferred_lft 42343sec 2024-02-19T15:24:32.337 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::ec4:7aff:febd:148e/64 scope link 2024-02-19T15:24:32.337 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:24:32.337 INFO:teuthology.orchestra.run.smithi047.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:24:32.337 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:79 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:24:32.338 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f1 2024-02-19T15:24:32.338 INFO:teuthology.orchestra.run.smithi047.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:24:32.338 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8f brd ff:ff:ff:ff:ff:ff 2024-02-19T15:24:32.338 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f1 2024-02-19T15:24:32.338 INFO:teuthology.orchestra.run.smithi047.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-02-19T15:24:32.338 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 02:42:47:cf:b1:62 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:24:32.338 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-02-19T15:24:32.338 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:24:32.338 INFO:teuthology.orchestra.run.smithi047.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-02-19T15:24:32.338 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-02-19T15:24:32.338 INFO:teuthology.orchestra.run.smithi047.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-02-19T15:24:32.338 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:24:32.338 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-02-19T15:24:32.338 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:24:32.338 INFO:teuthology.orchestra.run.smithi047.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:24:32.338 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-19T15:24:32.338 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-02-19T15:24:32.339 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:24:32.339 INFO:teuthology.orchestra.run.smithi047.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:24:32.339 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-02-19T15:24:32.339 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-02-19T15:24:32.339 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:24:32.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:32.339 DEBUG:teuthology.orchestra.run.smithi047:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T15:24:32.396 INFO:teuthology.orchestra.run.smithi047.stdout:1 2024-02-19T15:24:32.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:32.398 DEBUG:teuthology.orchestra.run.smithi047:> route 2024-02-19T15:24:32.447 INFO:teuthology.orchestra.run.smithi047.stdout:Kernel IP routing table 2024-02-19T15:24:32.447 INFO:teuthology.orchestra.run.smithi047.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T15:24:32.447 INFO:teuthology.orchestra.run.smithi047.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-02-19T15:24:32.447 INFO:teuthology.orchestra.run.smithi047.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-02-19T15:24:32.448 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-02-19T15:24:32.448 INFO:teuthology.orchestra.run.smithi047.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T15:24:32.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:32.448 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2024-02-19T15:24:32.448 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2024-02-19T15:24:32.448 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-02-19T15:24:32.448 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-02-19T15:24:32.448 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-02-19T15:24:32.448 DEBUG:teuthology.orchestra.run.smithi047:> ') 2024-02-19T15:24:32.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:32.540 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list 2024-02-19T15:24:32.549 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-02-19T15:24:32.549 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-02-19T15:24:32.549 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:24:32.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:32.549 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set brx.0 up 2024-02-19T15:24:32.610 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:24:32.610 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-19T15:24:32.658 INFO:teuthology.orchestra.run.smithi047.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:24:32.658 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:24:32.658 DEBUG:teuthology.orchestra.run.smithi047:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:24:32.706 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-19T15:24:32.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:32.706 DEBUG:teuthology.orchestra.run.smithi047:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:24:32.797 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:24:32.797 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:24:32.797 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:24:32.812 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:24:32.812 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:24:32.813 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:24:32.813 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:24:32.813 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:24:32.822 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:24:32.822 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:24:32.822 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:24:32.822 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:24:32.822 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:24:32.822 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790" 2024-02-19T15:24:32.822 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790:/ 2024-02-19T15:24:32.822 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e" will pass to kernel 2024-02-19T15:24:32.827 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T15:24:32.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:32.827 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:24:32.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:32.847 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:24:32.906 INFO:teuthology.orchestra.run.smithi047.stdout:client5944 2024-02-19T15:24:32.906 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:24:32.906 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:32.955 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-02-19T15:24:33.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.138+0000 7f3ebf200640 1 Processor -- start 2024-02-19T15:24:33.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.138+0000 7f3ebf200640 1 -- start start 2024-02-19T15:24:33.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.138+0000 7f3ebf200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3eb815e000 0x7f3eb80cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:33.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.138+0000 7f3ebf200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3eb815fe90 0x7f3eb80d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:33.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.138+0000 7f3ebf200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb80d31e0 con 0x7f3eb80d0240 2024-02-19T15:24:33.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.138+0000 7f3ebf200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb80ceca0 con 0x7f3eb815e000 2024-02-19T15:24:33.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.138+0000 7f3ebf200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb80cef40 con 0x7f3eb815fe90 2024-02-19T15:24:33.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.138+0000 7f3eb7e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3eb815e000 0x7f3eb80cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.138+0000 7f3eafe00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3eb80d0240 0x7f3eb815f780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:36428/0 (socket says 172.21.15.47:36428) 2024-02-19T15:24:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.138+0000 7f3eafe00640 1 -- 172.21.15.47:0/340811397 learned_addr learned my addr 172.21.15.47:0/340811397 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.138+0000 7f3ebd200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3eb815fe90 0x7f3eb80d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.138+0000 7f3eb7e00640 1 -- 172.21.15.47:0/340811397 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3eb815fe90 msgr2=0x7f3eb80d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.138+0000 7f3eb7e00640 1 --2- 172.21.15.47:0/340811397 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3eb815fe90 0x7f3eb80d24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.138+0000 7f3eb7e00640 1 -- 172.21.15.47:0/340811397 >> v1:172.21.15.116:6789/0 conn(0x7f3eb80d0240 legacy=0x7f3eb815f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.138+0000 7f3eb7e00640 1 -- 172.21.15.47:0/340811397 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb80cf220 con 0x7f3eb815e000 2024-02-19T15:24:33.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.142+0000 7f3eb7e00640 1 --2- 172.21.15.47:0/340811397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3eb815e000 0x7f3eb80cfd00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80097c0 tx=0x7f3ea801c350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e62be7ab004136aa server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:33.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.142+0000 7f3eb7400640 1 -- 172.21.15.47:0/340811397 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8004230 con 0x7f3eb815e000 2024-02-19T15:24:33.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.142+0000 7f3eb7400640 1 -- 172.21.15.47:0/340811397 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea80a7050 con 0x7f3eb815e000 2024-02-19T15:24:33.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.142+0000 7f3eb7400640 1 -- 172.21.15.47:0/340811397 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8003620 con 0x7f3eb815e000 2024-02-19T15:24:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.142+0000 7f3ebf200640 1 -- 172.21.15.47:0/340811397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3eb815e000 msgr2=0x7f3eb80cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.142+0000 7f3ebf200640 1 --2- 172.21.15.47:0/340811397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3eb815e000 0x7f3eb80cfd00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80097c0 tx=0x7f3ea801c350 comp rx=0 tx=0).stop 2024-02-19T15:24:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.142+0000 7f3ebf200640 1 -- 172.21.15.47:0/340811397 shutdown_connections 2024-02-19T15:24:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.142+0000 7f3ebf200640 1 --2- 172.21.15.47:0/340811397 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3eb815fe90 0x7f3eb80d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.142+0000 7f3ebf200640 1 --2- 172.21.15.47:0/340811397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3eb815e000 0x7f3eb80cfd00 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.142+0000 7f3ebf200640 1 -- 172.21.15.47:0/340811397 >> 172.21.15.47:0/340811397 conn(0x7f3eb80d3d90 msgr2=0x7f3eb80d41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.142+0000 7f3ebf200640 1 -- 172.21.15.47:0/340811397 shutdown_connections 2024-02-19T15:24:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.142+0000 7f3ebf200640 1 -- 172.21.15.47:0/340811397 wait complete. 2024-02-19T15:24:33.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.146+0000 7f3ebf200640 1 Processor -- start 2024-02-19T15:24:33.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.146+0000 7f3ebf200640 1 -- start start 2024-02-19T15:24:33.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.146+0000 7f3ebf200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3eb815e000 0x7f3eb8155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:33.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.146+0000 7f3ebf200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3eb815fe90 0x7f3eb815c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:33.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.146+0000 7f3ebf200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3eb815c9b0 0x7f3eb81614a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:33.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.146+0000 7f3ebf200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb81560e0 con 0x7f3eb815e000 2024-02-19T15:24:33.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.146+0000 7f3ebf200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb815be20 con 0x7f3eb815c9b0 2024-02-19T15:24:33.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.146+0000 7f3ebf200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb815b250 con 0x7f3eb815fe90 2024-02-19T15:24:33.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.146+0000 7f3eb7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3eb815e000 0x7f3eb8155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:33.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.146+0000 7f3eb7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3eb815e000 0x7f3eb8155ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51148/0 (socket says 172.21.15.47:51148) 2024-02-19T15:24:33.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.146+0000 7f3eb7e00640 1 -- 172.21.15.47:0/1893803902 learned_addr learned my addr 172.21.15.47:0/1893803902 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:33.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.146+0000 7f3eafe00640 1 --2- 172.21.15.47:0/1893803902 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3eb815fe90 0x7f3eb815c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:33.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.146+0000 7f3ebd200640 1 --2- 172.21.15.47:0/1893803902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3eb815c9b0 0x7f3eb81614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:33.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.146+0000 7f3eb7e00640 1 -- 172.21.15.47:0/1893803902 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3eb815fe90 msgr2=0x7f3eb815c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:33.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.146+0000 7f3eb7e00640 1 --2- 172.21.15.47:0/1893803902 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3eb815fe90 0x7f3eb815c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:33.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.146+0000 7f3eb7e00640 1 -- 172.21.15.47:0/1893803902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3eb815c9b0 msgr2=0x7f3eb81614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:33.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.146+0000 7f3eb7e00640 1 --2- 172.21.15.47:0/1893803902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3eb815c9b0 0x7f3eb81614a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:33.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.146+0000 7f3eb7e00640 1 -- 172.21.15.47:0/1893803902 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea8009420 con 0x7f3eb815e000 2024-02-19T15:24:33.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.146+0000 7f3eb7e00640 1 --2- 172.21.15.47:0/1893803902 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3eb815e000 0x7f3eb8155ba0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80097c0 tx=0x7f3ea80190f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:33.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.146+0000 7f3eb4c00640 1 -- 172.21.15.47:0/1893803902 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8003b70 con 0x7f3eb815e000 2024-02-19T15:24:33.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.150+0000 7f3eb4c00640 1 -- 172.21.15.47:0/1893803902 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea8003cd0 con 0x7f3eb815e000 2024-02-19T15:24:33.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.150+0000 7f3ebf200640 1 -- 172.21.15.47:0/1893803902 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb815b4f0 con 0x7f3eb815e000 2024-02-19T15:24:33.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.150+0000 7f3eb4c00640 1 -- 172.21.15.47:0/1893803902 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea80ab660 con 0x7f3eb815e000 2024-02-19T15:24:33.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.150+0000 7f3ebf200640 1 -- 172.21.15.47:0/1893803902 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb815bab0 con 0x7f3eb815e000 2024-02-19T15:24:33.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.150+0000 7f3ebf200640 1 -- 172.21.15.47:0/1893803902 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb80d3750 con 0x7f3eb815e000 2024-02-19T15:24:33.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.150+0000 7f3eb4c00640 1 -- 172.21.15.47:0/1893803902 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3ea80b2040 con 0x7f3eb815e000 2024-02-19T15:24:33.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.150+0000 7f3eb4c00640 1 --2- 172.21.15.47:0/1893803902 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3e94066e70 0x7f3e94069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:33.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.150+0000 7f3eb4c00640 1 -- 172.21.15.47:0/1893803902 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ea811e810 con 0x7f3eb815e000 2024-02-19T15:24:33.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.154+0000 7f3eafe00640 1 --2- 172.21.15.47:0/1893803902 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3e94066e70 0x7f3e94069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:33.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.154+0000 7f3eafe00640 1 --2- 172.21.15.47:0/1893803902 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3e94066e70 0x7f3e94069330 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3eb815d8d0 tx=0x7f3ea0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:33.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.158+0000 7f3eb4c00640 1 -- 172.21.15.47:0/1893803902 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f3ea80eb6c0 con 0x7f3eb815e000 2024-02-19T15:24:33.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.326+0000 7f3ebf200640 1 -- 172.21.15.47:0/1893803902 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f3eb8075360 con 0x7f3eb815e000 2024-02-19T15:24:33.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.330+0000 7f3eb4c00640 1 -- 172.21.15.47:0/1893803902 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v17)=0 v17) v1 ==== 124+0+0 (secure 0 0 0) 0x7f3ea80b00c0 con 0x7f3eb815e000 2024-02-19T15:24:33.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.330+0000 7f3ebf200640 1 -- 172.21.15.47:0/1893803902 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3e94066e70 msgr2=0x7f3e94069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:33.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.330+0000 7f3ebf200640 1 --2- 172.21.15.47:0/1893803902 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3e94066e70 0x7f3e94069330 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3eb815d8d0 tx=0x7f3ea0009040 comp rx=0 tx=0).stop 2024-02-19T15:24:33.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.330+0000 7f3ebf200640 1 -- 172.21.15.47:0/1893803902 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3eb815e000 msgr2=0x7f3eb8155ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:33.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.330+0000 7f3ebf200640 1 --2- 172.21.15.47:0/1893803902 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3eb815e000 0x7f3eb8155ba0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80097c0 tx=0x7f3ea80190f0 comp rx=0 tx=0).stop 2024-02-19T15:24:33.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.330+0000 7f3ebf200640 1 -- 172.21.15.47:0/1893803902 shutdown_connections 2024-02-19T15:24:33.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.330+0000 7f3ebf200640 1 --2- 172.21.15.47:0/1893803902 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3e94066e70 0x7f3e94069330 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:33.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.330+0000 7f3ebf200640 1 --2- 172.21.15.47:0/1893803902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3eb815c9b0 0x7f3eb81614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:33.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.330+0000 7f3ebf200640 1 --2- 172.21.15.47:0/1893803902 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3eb815fe90 0x7f3eb815c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:33.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.330+0000 7f3ebf200640 1 --2- 172.21.15.47:0/1893803902 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3eb815e000 0x7f3eb8155ba0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:33.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.330+0000 7f3ebf200640 1 -- 172.21.15.47:0/1893803902 >> 172.21.15.47:0/1893803902 conn(0x7f3eb80d3d90 msgr2=0x7f3eb80d4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:33.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.330+0000 7f3ebf200640 1 -- 172.21.15.47:0/1893803902 shutdown_connections 2024-02-19T15:24:33.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.330+0000 7f3ebf200640 1 -- 172.21.15.47:0/1893803902 wait complete. 2024-02-19T15:24:33.355 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_merge_size 1 2024-02-19T15:24:33.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.494+0000 7f796b200640 1 Processor -- start 2024-02-19T15:24:33.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.494+0000 7f796b200640 1 -- start start 2024-02-19T15:24:33.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.498+0000 7f796b200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f796415e000 0x7f79640cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:33.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.498+0000 7f796b200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f796415fe90 0x7f79640d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:33.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.498+0000 7f796b200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79640d31e0 con 0x7f79640d0240 2024-02-19T15:24:33.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.498+0000 7f796b200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79640ceca0 con 0x7f796415e000 2024-02-19T15:24:33.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.498+0000 7f796b200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79640cef40 con 0x7f796415fe90 2024-02-19T15:24:33.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.498+0000 7f7963e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f796415e000 0x7f79640cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:33.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.498+0000 7f7963e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f796415e000 0x7f79640cfd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:35372/0 (socket says 172.21.15.47:35372) 2024-02-19T15:24:33.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.498+0000 7f7963e00640 1 -- 172.21.15.47:0/524555199 learned_addr learned my addr 172.21.15.47:0/524555199 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:33.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.498+0000 7f7969200640 1 --2- 172.21.15.47:0/524555199 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f796415fe90 0x7f79640d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:33.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.498+0000 7f7963e00640 1 -- 172.21.15.47:0/524555199 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f796415fe90 msgr2=0x7f79640d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:33.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.498+0000 7f7963e00640 1 --2- 172.21.15.47:0/524555199 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f796415fe90 0x7f79640d24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:33.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.498+0000 7f7963e00640 1 -- 172.21.15.47:0/524555199 >> v1:172.21.15.47:6789/0 conn(0x7f79640d0240 legacy=0x7f796415f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:33.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.498+0000 7f7963400640 1 -- 172.21.15.47:0/524555199 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3388758310 0 0) 0x7f79640d31e0 con 0x7f79640d0240 2024-02-19T15:24:33.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.498+0000 7f7963e00640 1 -- 172.21.15.47:0/524555199 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79640cf220 con 0x7f796415e000 2024-02-19T15:24:33.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.502+0000 7f7963e00640 1 --2- 172.21.15.47:0/524555199 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f796415e000 0x7f79640cfd00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7954009870 tx=0x7f795401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea39e82156b077e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:33.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.502+0000 7f7963400640 1 -- 172.21.15.47:0/524555199 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79540ab610 con 0x7f796415e000 2024-02-19T15:24:33.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.502+0000 7f7963400640 1 -- 172.21.15.47:0/524555199 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79540ab770 con 0x7f796415e000 2024-02-19T15:24:33.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.502+0000 7f7963400640 1 -- 172.21.15.47:0/524555199 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79540abb00 con 0x7f796415e000 2024-02-19T15:24:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.502+0000 7f796b200640 1 -- 172.21.15.47:0/524555199 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f796415e000 msgr2=0x7f79640cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.502+0000 7f796b200640 1 --2- 172.21.15.47:0/524555199 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f796415e000 0x7f79640cfd00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7954009870 tx=0x7f795401c410 comp rx=0 tx=0).stop 2024-02-19T15:24:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.502+0000 7f796b200640 1 -- 172.21.15.47:0/524555199 shutdown_connections 2024-02-19T15:24:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.502+0000 7f796b200640 1 --2- 172.21.15.47:0/524555199 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f796415fe90 0x7f79640d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.502+0000 7f796b200640 1 --2- 172.21.15.47:0/524555199 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f796415e000 0x7f79640cfd00 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.502+0000 7f796b200640 1 -- 172.21.15.47:0/524555199 >> 172.21.15.47:0/524555199 conn(0x7f79640d3d90 msgr2=0x7f79640d41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.502+0000 7f796b200640 1 -- 172.21.15.47:0/524555199 shutdown_connections 2024-02-19T15:24:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.502+0000 7f796b200640 1 -- 172.21.15.47:0/524555199 wait complete. 2024-02-19T15:24:33.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f796b200640 1 Processor -- start 2024-02-19T15:24:33.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f796b200640 1 -- start start 2024-02-19T15:24:33.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f796b200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f796415e000 0x7f796415bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:33.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f796b200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f796415fe90 0x7f79641615d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:33.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f796b200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79640c8d70 0x7f79640c91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:33.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f796b200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796415b7c0 con 0x7f796415e000 2024-02-19T15:24:33.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f796b200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796415b190 con 0x7f79640c8d70 2024-02-19T15:24:33.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f796b200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796415c120 con 0x7f796415fe90 2024-02-19T15:24:33.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f7963e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f796415e000 0x7f796415bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:33.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f7963e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f796415e000 0x7f796415bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:35374/0 (socket says 172.21.15.47:35374) 2024-02-19T15:24:33.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f7963e00640 1 -- 172.21.15.47:0/3421846731 learned_addr learned my addr 172.21.15.47:0/3421846731 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:33.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f7963e00640 1 -- 172.21.15.47:0/3421846731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f796415fe90 msgr2=0x7f79641615d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:33.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f7963e00640 1 --2- 172.21.15.47:0/3421846731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f796415fe90 0x7f79641615d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:33.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f7963e00640 1 -- 172.21.15.47:0/3421846731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79640c8d70 msgr2=0x7f79640c91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:33.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f7963e00640 1 --2- 172.21.15.47:0/3421846731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79640c8d70 0x7f79640c91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:33.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f7963e00640 1 -- 172.21.15.47:0/3421846731 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79540094d0 con 0x7f796415e000 2024-02-19T15:24:33.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f7963e00640 1 --2- 172.21.15.47:0/3421846731 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f796415e000 0x7f796415bbe0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f795401a040 tx=0x7f795401ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:33.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f7960c00640 1 -- 172.21.15.47:0/3421846731 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79540abd70 con 0x7f796415e000 2024-02-19T15:24:33.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f7960c00640 1 -- 172.21.15.47:0/3421846731 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79540b8070 con 0x7f796415e000 2024-02-19T15:24:33.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f7960c00640 1 -- 172.21.15.47:0/3421846731 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7954004220 con 0x7f796415e000 2024-02-19T15:24:33.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f796b200640 1 -- 172.21.15.47:0/3421846731 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79640c9830 con 0x7f796415e000 2024-02-19T15:24:33.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f796b200640 1 -- 172.21.15.47:0/3421846731 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796415b3e0 con 0x7f796415e000 2024-02-19T15:24:33.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.506+0000 7f796b200640 1 -- 172.21.15.47:0/3421846731 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79640d3750 con 0x7f796415e000 2024-02-19T15:24:33.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.510+0000 7f7960c00640 1 -- 172.21.15.47:0/3421846731 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7954004380 con 0x7f796415e000 2024-02-19T15:24:33.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.510+0000 7f7960c00640 1 --2- 172.21.15.47:0/3421846731 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7940066e40 0x7f7940069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:33.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.510+0000 7f7960c00640 1 -- 172.21.15.47:0/3421846731 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f795411ee20 con 0x7f796415e000 2024-02-19T15:24:33.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.510+0000 7f795b400640 1 --2- 172.21.15.47:0/3421846731 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7940066e40 0x7f7940069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:33.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.510+0000 7f795b400640 1 --2- 172.21.15.47:0/3421846731 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7940066e40 0x7f7940069300 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f79640c7290 tx=0x7f794c007cc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:33.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.514+0000 7f7960c00640 1 -- 172.21.15.47:0/3421846731 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f79540ebb70 con 0x7f796415e000 2024-02-19T15:24:33.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.686+0000 7f796b200640 1 -- 172.21.15.47:0/3421846731 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) v1 -- 0x7f7964075360 con 0x7f796415e000 2024-02-19T15:24:33.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.690+0000 7f7960c00640 1 -- 172.21.15.47:0/3421846731 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v18)=0 v18) v1 ==== 116+0+0 (secure 0 0 0) 0x7f79540ba070 con 0x7f796415e000 2024-02-19T15:24:33.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.690+0000 7f796b200640 1 -- 172.21.15.47:0/3421846731 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7940066e40 msgr2=0x7f7940069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:33.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.690+0000 7f796b200640 1 --2- 172.21.15.47:0/3421846731 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7940066e40 0x7f7940069300 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f79640c7290 tx=0x7f794c007cc0 comp rx=0 tx=0).stop 2024-02-19T15:24:33.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.690+0000 7f796b200640 1 -- 172.21.15.47:0/3421846731 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f796415e000 msgr2=0x7f796415bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:33.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.690+0000 7f796b200640 1 --2- 172.21.15.47:0/3421846731 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f796415e000 0x7f796415bbe0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f795401a040 tx=0x7f795401ac50 comp rx=0 tx=0).stop 2024-02-19T15:24:33.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.690+0000 7f796b200640 1 -- 172.21.15.47:0/3421846731 shutdown_connections 2024-02-19T15:24:33.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.690+0000 7f796b200640 1 --2- 172.21.15.47:0/3421846731 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7940066e40 0x7f7940069300 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:33.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.690+0000 7f796b200640 1 --2- 172.21.15.47:0/3421846731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79640c8d70 0x7f79640c91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:33.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.690+0000 7f796b200640 1 --2- 172.21.15.47:0/3421846731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f796415fe90 0x7f79641615d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:33.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.690+0000 7f796b200640 1 --2- 172.21.15.47:0/3421846731 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f796415e000 0x7f796415bbe0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:33.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.690+0000 7f796b200640 1 -- 172.21.15.47:0/3421846731 >> 172.21.15.47:0/3421846731 conn(0x7f79640d3d90 msgr2=0x7f796406c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:33.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.690+0000 7f796b200640 1 -- 172.21.15.47:0/3421846731 shutdown_connections 2024-02-19T15:24:33.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:33.690+0000 7f796b200640 1 -- 172.21.15.47:0/3421846731 wait complete. 2024-02-19T15:24:35.426 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:24:35.429 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:24:43.715 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:24:43.715 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:43.724 DEBUG:teuthology.orchestra.run.smithi047:> stdin-killer -- python3 -c ' 2024-02-19T15:24:43.724 DEBUG:teuthology.orchestra.run.smithi047:> import os 2024-02-19T15:24:43.724 DEBUG:teuthology.orchestra.run.smithi047:> import uuid 2024-02-19T15:24:43.724 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:24:43.724 DEBUG:teuthology.orchestra.run.smithi047:> n = 50 2024-02-19T15:24:43.724 DEBUG:teuthology.orchestra.run.smithi047:> create_hard_links = False 2024-02-19T15:24:43.724 DEBUG:teuthology.orchestra.run.smithi047:> if 0 > 0: 2024-02-19T15:24:43.724 DEBUG:teuthology.orchestra.run.smithi047:> create_hard_links = True 2024-02-19T15:24:43.725 DEBUG:teuthology.orchestra.run.smithi047:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-02-19T15:24:43.725 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:24:43.725 DEBUG:teuthology.orchestra.run.smithi047:> dpath = os.path.dirname(path) 2024-02-19T15:24:43.725 DEBUG:teuthology.orchestra.run.smithi047:> fnameprefix = os.path.basename(path) 2024-02-19T15:24:43.725 DEBUG:teuthology.orchestra.run.smithi047:> os.makedirs(dpath, exist_ok=True) 2024-02-19T15:24:43.725 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:24:43.725 DEBUG:teuthology.orchestra.run.smithi047:> try: 2024-02-19T15:24:43.725 DEBUG:teuthology.orchestra.run.smithi047:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-02-19T15:24:43.725 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:24:43.725 DEBUG:teuthology.orchestra.run.smithi047:> for i in range(n): 2024-02-19T15:24:43.725 DEBUG:teuthology.orchestra.run.smithi047:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-02-19T15:24:43.725 DEBUG:teuthology.orchestra.run.smithi047:> with open(fpath, '"'"'w'"'"') as f: 2024-02-19T15:24:43.725 DEBUG:teuthology.orchestra.run.smithi047:> f.write(f"{i}") 2024-02-19T15:24:43.725 DEBUG:teuthology.orchestra.run.smithi047:> if False: 2024-02-19T15:24:43.725 DEBUG:teuthology.orchestra.run.smithi047:> f.flush() 2024-02-19T15:24:43.725 DEBUG:teuthology.orchestra.run.smithi047:> os.fsync(f.fileno()) 2024-02-19T15:24:43.725 DEBUG:teuthology.orchestra.run.smithi047:> if False: 2024-02-19T15:24:43.725 DEBUG:teuthology.orchestra.run.smithi047:> os.unlink(fpath) 2024-02-19T15:24:43.726 DEBUG:teuthology.orchestra.run.smithi047:> if False: 2024-02-19T15:24:43.726 DEBUG:teuthology.orchestra.run.smithi047:> os.fsync(dirfd) 2024-02-19T15:24:43.726 DEBUG:teuthology.orchestra.run.smithi047:> if create_hard_links: 2024-02-19T15:24:43.726 DEBUG:teuthology.orchestra.run.smithi047:> for j in range(0): 2024-02-19T15:24:43.726 DEBUG:teuthology.orchestra.run.smithi047:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-02-19T15:24:43.726 DEBUG:teuthology.orchestra.run.smithi047:> if True: 2024-02-19T15:24:43.726 DEBUG:teuthology.orchestra.run.smithi047:> os.fsync(dirfd) 2024-02-19T15:24:43.726 DEBUG:teuthology.orchestra.run.smithi047:> finally: 2024-02-19T15:24:43.726 DEBUG:teuthology.orchestra.run.smithi047:> os.close(dirfd) 2024-02-19T15:24:43.726 DEBUG:teuthology.orchestra.run.smithi047:> ' 2024-02-19T15:24:43.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-02-19T15:24:43.943 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:24:43.943 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:43.995 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:24:43.995 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:44.046 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:24:44.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.225+0000 7f6d70800640 1 Processor -- start 2024-02-19T15:24:44.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.225+0000 7f6d70800640 1 -- start start 2024-02-19T15:24:44.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.225+0000 7f6d70800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6d680cddc0 0x7f6d6806c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:44.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.225+0000 7f6d70800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d680cfe00 0x7f6d680ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:44.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.225+0000 7f6d70800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6d680ce6e0 0x7f6d6815d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:44.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.225+0000 7f6d70800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d6806c780 con 0x7f6d680cddc0 2024-02-19T15:24:44.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.225+0000 7f6d70800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d6806d200 con 0x7f6d680ce6e0 2024-02-19T15:24:44.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.225+0000 7f6d70800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d680d1180 con 0x7f6d680cfe00 2024-02-19T15:24:44.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.225+0000 7f6d6de00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6d680cddc0 0x7f6d6806c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:44.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.225+0000 7f6d6de00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6d680cddc0 0x7f6d6806c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34270/0 (socket says 172.21.15.47:34270) 2024-02-19T15:24:44.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.225+0000 7f6d6de00640 1 -- 172.21.15.47:0/128772653 learned_addr learned my addr 172.21.15.47:0/128772653 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:44.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.225+0000 7f6d6e800640 1 --2- 172.21.15.47:0/128772653 >> v2:172.21.15.116:3300/0 conn(0x7f6d680ce6e0 0x7f6d6815d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:44.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.225+0000 7f6d6d400640 1 --2- 172.21.15.47:0/128772653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d680cfe00 0x7f6d680ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:44.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.225+0000 7f6d6de00640 1 -- 172.21.15.47:0/128772653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d680cfe00 msgr2=0x7f6d680ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:44.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.225+0000 7f6d6de00640 1 --2- 172.21.15.47:0/128772653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d680cfe00 0x7f6d680ce1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.225+0000 7f6d6de00640 1 -- 172.21.15.47:0/128772653 >> v2:172.21.15.116:3300/0 conn(0x7f6d680ce6e0 msgr2=0x7f6d6815d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:44.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.225+0000 7f6d6de00640 1 --2- 172.21.15.47:0/128772653 >> v2:172.21.15.116:3300/0 conn(0x7f6d680ce6e0 0x7f6d6815d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.225+0000 7f6d6de00640 1 -- 172.21.15.47:0/128772653 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d680d0600 con 0x7f6d680cddc0 2024-02-19T15:24:44.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.225+0000 7f6d6e800640 1 --2- 172.21.15.47:0/128772653 >> v2:172.21.15.116:3300/0 conn(0x7f6d680ce6e0 0x7f6d6815d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:44.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.225+0000 7f6d6d400640 1 --2- 172.21.15.47:0/128772653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d680cfe00 0x7f6d680ce1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:44.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.233+0000 7f6d6de00640 1 --2- 172.21.15.47:0/128772653 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d680cddc0 0x7f6d6806c240 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f6d680b7be0 tx=0x7f6d5c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da7fcfed905c13c4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:44.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.233+0000 7f6d6ca00640 1 -- 172.21.15.47:0/128772653 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c0ac420 con 0x7f6d680cddc0 2024-02-19T15:24:44.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.233+0000 7f6d6ca00640 1 -- 172.21.15.47:0/128772653 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5c0ac580 con 0x7f6d680cddc0 2024-02-19T15:24:44.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.233+0000 7f6d6ca00640 1 -- 172.21.15.47:0/128772653 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c0ac910 con 0x7f6d680cddc0 2024-02-19T15:24:44.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.233+0000 7f6d70800640 1 -- 172.21.15.47:0/128772653 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d680cddc0 msgr2=0x7f6d6806c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:44.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.233+0000 7f6d70800640 1 --2- 172.21.15.47:0/128772653 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d680cddc0 0x7f6d6806c240 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f6d680b7be0 tx=0x7f6d5c01c4f0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.233+0000 7f6d70800640 1 -- 172.21.15.47:0/128772653 shutdown_connections 2024-02-19T15:24:44.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.233+0000 7f6d70800640 1 --2- 172.21.15.47:0/128772653 >> v2:172.21.15.116:3300/0 conn(0x7f6d680ce6e0 0x7f6d6815d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.233+0000 7f6d70800640 1 --2- 172.21.15.47:0/128772653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d680cfe00 0x7f6d680ce1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.233+0000 7f6d70800640 1 --2- 172.21.15.47:0/128772653 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d680cddc0 0x7f6d6806c240 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.233+0000 7f6d70800640 1 -- 172.21.15.47:0/128772653 >> 172.21.15.47:0/128772653 conn(0x7f6d680d1b30 msgr2=0x7f6d680d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:44.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.233+0000 7f6d70800640 1 -- 172.21.15.47:0/128772653 shutdown_connections 2024-02-19T15:24:44.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.233+0000 7f6d70800640 1 -- 172.21.15.47:0/128772653 wait complete. 2024-02-19T15:24:44.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d70800640 1 Processor -- start 2024-02-19T15:24:44.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d70800640 1 -- start start 2024-02-19T15:24:44.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d70800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d680cddc0 0x7f6d68152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:44.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d70800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d680ce6e0 0x7f6d68153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:44.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d70800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d680cfe00 0x7f6d680c76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:44.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d70800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d68155ca0 con 0x7f6d680cddc0 2024-02-19T15:24:44.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d70800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d681526b0 con 0x7f6d680ce6e0 2024-02-19T15:24:44.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d70800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d68155090 con 0x7f6d680cfe00 2024-02-19T15:24:44.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d6de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d680cddc0 0x7f6d68152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:44.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d6de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d680cddc0 0x7f6d68152170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34282/0 (socket says 172.21.15.47:34282) 2024-02-19T15:24:44.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d6de00640 1 -- 172.21.15.47:0/294430802 learned_addr learned my addr 172.21.15.47:0/294430802 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:44.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d6e800640 1 --2- 172.21.15.47:0/294430802 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d680cfe00 0x7f6d680c76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:44.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d6d400640 1 --2- 172.21.15.47:0/294430802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d680ce6e0 0x7f6d68153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:44.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d6de00640 1 -- 172.21.15.47:0/294430802 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d680cfe00 msgr2=0x7f6d680c76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:44.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d6de00640 1 --2- 172.21.15.47:0/294430802 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d680cfe00 0x7f6d680c76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d6de00640 1 -- 172.21.15.47:0/294430802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d680ce6e0 msgr2=0x7f6d68153ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:44.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d6de00640 1 --2- 172.21.15.47:0/294430802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d680ce6e0 0x7f6d68153ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d6de00640 1 -- 172.21.15.47:0/294430802 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d5c0094d0 con 0x7f6d680cddc0 2024-02-19T15:24:44.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d6de00640 1 --2- 172.21.15.47:0/294430802 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d680cddc0 0x7f6d68152170 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c01cd90 tx=0x7f6d5c01ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:44.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d62000640 1 -- 172.21.15.47:0/294430802 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c0ac420 con 0x7f6d680cddc0 2024-02-19T15:24:44.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d62000640 1 -- 172.21.15.47:0/294430802 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5c0b7640 con 0x7f6d680cddc0 2024-02-19T15:24:44.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d62000640 1 -- 172.21.15.47:0/294430802 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c0b79f0 con 0x7f6d680cddc0 2024-02-19T15:24:44.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.237+0000 7f6d70800640 1 -- 172.21.15.47:0/294430802 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d68155370 con 0x7f6d680cddc0 2024-02-19T15:24:44.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.241+0000 7f6d70800640 1 -- 172.21.15.47:0/294430802 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d68155930 con 0x7f6d680cddc0 2024-02-19T15:24:44.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.241+0000 7f6d70800640 1 -- 172.21.15.47:0/294430802 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d68075360 con 0x7f6d680cddc0 2024-02-19T15:24:44.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.241+0000 7f6d62000640 1 -- 172.21.15.47:0/294430802 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6d5c0b3040 con 0x7f6d680cddc0 2024-02-19T15:24:44.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.241+0000 7f6d62000640 1 --2- 172.21.15.47:0/294430802 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6d4c066e40 0x7f6d4c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:44.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.241+0000 7f6d62000640 1 -- 172.21.15.47:0/294430802 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d5c11edb0 con 0x7f6d680cddc0 2024-02-19T15:24:44.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.241+0000 7f6d6d400640 1 --2- 172.21.15.47:0/294430802 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6d4c066e40 0x7f6d4c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:44.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.245+0000 7f6d6d400640 1 --2- 172.21.15.47:0/294430802 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6d4c066e40 0x7f6d4c069300 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6d680c7470 tx=0x7f6d58008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:44.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.245+0000 7f6d62000640 1 -- 172.21.15.47:0/294430802 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6d5c123070 con 0x7f6d680cddc0 2024-02-19T15:24:44.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.409+0000 7f6d70800640 1 -- 172.21.15.47:0/294430802 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d680ce4c0 con 0x7f6d680cddc0 2024-02-19T15:24:44.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.413+0000 7f6d62000640 1 -- 172.21.15.47:0/294430802 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16677 (secure 0 0 0) 0x7f6d5c0ebbb0 con 0x7f6d680cddc0 2024-02-19T15:24:44.415 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:44.415 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":57,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:24:26.785375+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:24:22.931622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-02-19T15:24:24.163465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:24:44.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.413+0000 7f6d70800640 1 -- 172.21.15.47:0/294430802 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6d4c066e40 msgr2=0x7f6d4c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:44.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.413+0000 7f6d70800640 1 --2- 172.21.15.47:0/294430802 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6d4c066e40 0x7f6d4c069300 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6d680c7470 tx=0x7f6d58008040 comp rx=0 tx=0).stop 2024-02-19T15:24:44.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.413+0000 7f6d70800640 1 -- 172.21.15.47:0/294430802 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d680cddc0 msgr2=0x7f6d68152170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:44.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.413+0000 7f6d70800640 1 --2- 172.21.15.47:0/294430802 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d680cddc0 0x7f6d68152170 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c01cd90 tx=0x7f6d5c01ad30 comp rx=0 tx=0).stop 2024-02-19T15:24:44.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.413+0000 7f6d70800640 1 -- 172.21.15.47:0/294430802 shutdown_connections 2024-02-19T15:24:44.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.413+0000 7f6d70800640 1 --2- 172.21.15.47:0/294430802 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6d4c066e40 0x7f6d4c069300 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.413+0000 7f6d70800640 1 --2- 172.21.15.47:0/294430802 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d680cfe00 0x7f6d680c76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.413+0000 7f6d70800640 1 --2- 172.21.15.47:0/294430802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d680ce6e0 0x7f6d68153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.413+0000 7f6d70800640 1 --2- 172.21.15.47:0/294430802 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d680cddc0 0x7f6d68152170 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.413+0000 7f6d70800640 1 -- 172.21.15.47:0/294430802 >> 172.21.15.47:0/294430802 conn(0x7f6d680d1b30 msgr2=0x7f6d680d1f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:44.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.413+0000 7f6d70800640 1 -- 172.21.15.47:0/294430802 shutdown_connections 2024-02-19T15:24:44.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.413+0000 7f6d70800640 1 -- 172.21.15.47:0/294430802 wait complete. 2024-02-19T15:24:44.440 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:24:22.931622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:24:24.163465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:24:44.440 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:24:44.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.581+0000 7f08de600640 1 Processor -- start 2024-02-19T15:24:44.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.581+0000 7f08de600640 1 -- start start 2024-02-19T15:24:44.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.581+0000 7f08de600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f08d80ce3b0 0x7f08d80ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:44.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.581+0000 7f08de600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f08d815e280 0x7f08d80cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:44.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.581+0000 7f08de600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d80cc620 con 0x7f08d80d2650 2024-02-19T15:24:44.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.581+0000 7f08de600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08d80d12e0 con 0x7f08d80ce3b0 2024-02-19T15:24:44.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.581+0000 7f08de600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d80d1940 con 0x7f08d815e280 2024-02-19T15:24:44.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.581+0000 7f08d7400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f08d80ce3b0 0x7f08d80ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:44.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.581+0000 7f08d7400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f08d80ce3b0 0x7f08d80ce790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34290/0 (socket says 172.21.15.47:34290) 2024-02-19T15:24:44.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.581+0000 7f08d7400640 1 -- 172.21.15.47:0/975799475 learned_addr learned my addr 172.21.15.47:0/975799475 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:44.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.581+0000 7f08d7e00640 1 --2- 172.21.15.47:0/975799475 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f08d815e280 0x7f08d80cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:44.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.581+0000 7f08d7400640 1 -- 172.21.15.47:0/975799475 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f08d815e280 msgr2=0x7f08d80cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:44.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.581+0000 7f08d7400640 1 --2- 172.21.15.47:0/975799475 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f08d815e280 0x7f08d80cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.581+0000 7f08d7400640 1 -- 172.21.15.47:0/975799475 >> v1:172.21.15.116:6789/0 conn(0x7f08d80d2650 legacy=0x7f08d80ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:44.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.581+0000 7f08d7400640 1 -- 172.21.15.47:0/975799475 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d80cecd0 con 0x7f08d80ce3b0 2024-02-19T15:24:44.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.585+0000 7f08d7400640 1 --2- 172.21.15.47:0/975799475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08d80ce3b0 0x7f08d80ce790 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f08c8009e50 tx=0x7f08c801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439b6b4b92d5f245 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:44.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.585+0000 7f08d6a00640 1 -- 172.21.15.47:0/975799475 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c80a56c0 con 0x7f08d80ce3b0 2024-02-19T15:24:44.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.585+0000 7f08d6a00640 1 -- 172.21.15.47:0/975799475 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c80a5820 con 0x7f08d80ce3b0 2024-02-19T15:24:44.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.585+0000 7f08d6a00640 1 -- 172.21.15.47:0/975799475 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c80a5bb0 con 0x7f08d80ce3b0 2024-02-19T15:24:44.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.585+0000 7f08de600640 1 -- 172.21.15.47:0/975799475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08d80ce3b0 msgr2=0x7f08d80ce790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:44.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.585+0000 7f08de600640 1 --2- 172.21.15.47:0/975799475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08d80ce3b0 0x7f08d80ce790 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f08c8009e50 tx=0x7f08c801c910 comp rx=0 tx=0).stop 2024-02-19T15:24:44.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.585+0000 7f08de600640 1 -- 172.21.15.47:0/975799475 shutdown_connections 2024-02-19T15:24:44.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.585+0000 7f08de600640 1 --2- 172.21.15.47:0/975799475 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f08d815e280 0x7f08d80cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.585+0000 7f08de600640 1 --2- 172.21.15.47:0/975799475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08d80ce3b0 0x7f08d80ce790 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.585+0000 7f08de600640 1 -- 172.21.15.47:0/975799475 >> 172.21.15.47:0/975799475 conn(0x7f08d80d1b30 msgr2=0x7f08d80d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:44.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.585+0000 7f08de600640 1 -- 172.21.15.47:0/975799475 shutdown_connections 2024-02-19T15:24:44.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.585+0000 7f08de600640 1 -- 172.21.15.47:0/975799475 wait complete. 2024-02-19T15:24:44.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08de600640 1 Processor -- start 2024-02-19T15:24:44.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08de600640 1 -- start start 2024-02-19T15:24:44.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08de600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08d80ce3b0 0x7f08d8156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:44.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08de600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f08d80d2650 0x7f08d8156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:44.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08de600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08d815e280 0x7f08d815cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:44.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08de600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d81583e0 con 0x7f08d80ce3b0 2024-02-19T15:24:44.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08de600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d8157260 con 0x7f08d815e280 2024-02-19T15:24:44.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08de600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d8157540 con 0x7f08d80d2650 2024-02-19T15:24:44.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08d7400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08d80ce3b0 0x7f08d8156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:44.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08d7400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08d80ce3b0 0x7f08d8156100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34294/0 (socket says 172.21.15.47:34294) 2024-02-19T15:24:44.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08d7400640 1 -- 172.21.15.47:0/941249184 learned_addr learned my addr 172.21.15.47:0/941249184 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:44.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08cea00640 1 --2- 172.21.15.47:0/941249184 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f08d80d2650 0x7f08d8156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:44.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08d7e00640 1 --2- 172.21.15.47:0/941249184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08d815e280 0x7f08d815cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:44.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08d7400640 1 -- 172.21.15.47:0/941249184 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f08d80d2650 msgr2=0x7f08d8156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:44.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08d7400640 1 --2- 172.21.15.47:0/941249184 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f08d80d2650 0x7f08d8156930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08d7400640 1 -- 172.21.15.47:0/941249184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08d815e280 msgr2=0x7f08d815cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:44.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08d7400640 1 --2- 172.21.15.47:0/941249184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08d815e280 0x7f08d815cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08d7400640 1 -- 172.21.15.47:0/941249184 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c8009b30 con 0x7f08d80ce3b0 2024-02-19T15:24:44.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08d7400640 1 --2- 172.21.15.47:0/941249184 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08d80ce3b0 0x7f08d8156100 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f08c801a540 tx=0x7f08c80196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:44.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08cfe00640 1 -- 172.21.15.47:0/941249184 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c80b8670 con 0x7f08d80ce3b0 2024-02-19T15:24:44.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08cfe00640 1 -- 172.21.15.47:0/941249184 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c80b87d0 con 0x7f08d80ce3b0 2024-02-19T15:24:44.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08cfe00640 1 -- 172.21.15.47:0/941249184 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c80b8b80 con 0x7f08d80ce3b0 2024-02-19T15:24:44.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08de600640 1 -- 172.21.15.47:0/941249184 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d8157820 con 0x7f08d80ce3b0 2024-02-19T15:24:44.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.589+0000 7f08de600640 1 -- 172.21.15.47:0/941249184 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d8156640 con 0x7f08d80ce3b0 2024-02-19T15:24:44.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.593+0000 7f08de600640 1 -- 172.21.15.47:0/941249184 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d8075360 con 0x7f08d80ce3b0 2024-02-19T15:24:44.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.593+0000 7f08cfe00640 1 -- 172.21.15.47:0/941249184 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f08c80b8ce0 con 0x7f08d80ce3b0 2024-02-19T15:24:44.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.593+0000 7f08cfe00640 1 --2- 172.21.15.47:0/941249184 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f08b4066e40 0x7f08b4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:44.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.593+0000 7f08cfe00640 1 -- 172.21.15.47:0/941249184 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08c811ffc0 con 0x7f08d80ce3b0 2024-02-19T15:24:44.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.593+0000 7f08cea00640 1 --2- 172.21.15.47:0/941249184 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f08b4066e40 0x7f08b4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:44.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.593+0000 7f08cea00640 1 --2- 172.21.15.47:0/941249184 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f08b4066e40 0x7f08b4069300 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f08d80cb680 tx=0x7f08c0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:44.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.601+0000 7f08cfe00640 1 -- 172.21.15.47:0/941249184 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f08c80ecec0 con 0x7f08d80ce3b0 2024-02-19T15:24:44.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.765+0000 7f08de600640 1 -- 172.21.15.47:0/941249184 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f08d80cdf80 con 0x7f08d80ce3b0 2024-02-19T15:24:44.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.765+0000 7f08cfe00640 1 -- 172.21.15.47:0/941249184 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7f08c80f0580 con 0x7f08d80ce3b0 2024-02-19T15:24:44.769 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:24:44.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.765+0000 7f08de600640 1 -- 172.21.15.47:0/941249184 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f08b4066e40 msgr2=0x7f08b4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:44.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.765+0000 7f08de600640 1 --2- 172.21.15.47:0/941249184 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f08b4066e40 0x7f08b4069300 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f08d80cb680 tx=0x7f08c0009040 comp rx=0 tx=0).stop 2024-02-19T15:24:44.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.765+0000 7f08de600640 1 -- 172.21.15.47:0/941249184 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08d80ce3b0 msgr2=0x7f08d8156100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:44.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.765+0000 7f08de600640 1 --2- 172.21.15.47:0/941249184 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08d80ce3b0 0x7f08d8156100 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f08c801a540 tx=0x7f08c80196b0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.765+0000 7f08de600640 1 -- 172.21.15.47:0/941249184 shutdown_connections 2024-02-19T15:24:44.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.765+0000 7f08de600640 1 --2- 172.21.15.47:0/941249184 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f08b4066e40 0x7f08b4069300 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.765+0000 7f08de600640 1 --2- 172.21.15.47:0/941249184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08d815e280 0x7f08d815cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.765+0000 7f08de600640 1 --2- 172.21.15.47:0/941249184 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f08d80d2650 0x7f08d8156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.765+0000 7f08de600640 1 --2- 172.21.15.47:0/941249184 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08d80ce3b0 0x7f08d8156100 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.769+0000 7f08de600640 1 -- 172.21.15.47:0/941249184 >> 172.21.15.47:0/941249184 conn(0x7f08d80d1b30 msgr2=0x7f08d80c7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:44.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.769+0000 7f08de600640 1 -- 172.21.15.47:0/941249184 shutdown_connections 2024-02-19T15:24:44.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.769+0000 7f08de600640 1 -- 172.21.15.47:0/941249184 wait complete. 2024-02-19T15:24:44.790 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:24:44.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.937+0000 7fa385600640 1 Processor -- start 2024-02-19T15:24:44.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.937+0000 7fa385600640 1 -- start start 2024-02-19T15:24:44.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.937+0000 7fa385600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa3800d0820 0x7fa3800cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:44.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.937+0000 7fa385600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3800c8260 0x7fa3800ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:44.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.937+0000 7fa385600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3800ceee0 con 0x7fa3800c7e80 2024-02-19T15:24:44.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.937+0000 7fa385600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3800cdc60 con 0x7fa3800d0820 2024-02-19T15:24:44.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.937+0000 7fa385600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3800d2a40 con 0x7fa3800c8260 2024-02-19T15:24:44.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.937+0000 7fa377e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa3800d0820 0x7fa3800cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:44.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.937+0000 7fa377e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa3800d0820 0x7fa3800cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34304/0 (socket says 172.21.15.47:34304) 2024-02-19T15:24:44.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.937+0000 7fa377e00640 1 -- 172.21.15.47:0/240166089 learned_addr learned my addr 172.21.15.47:0/240166089 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:44.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.937+0000 7fa37f400640 1 --2- 172.21.15.47:0/240166089 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3800c8260 0x7fa3800ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:44.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.937+0000 7fa377e00640 1 -- 172.21.15.47:0/240166089 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3800c8260 msgr2=0x7fa3800ccc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:44.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.937+0000 7fa377e00640 1 --2- 172.21.15.47:0/240166089 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3800c8260 0x7fa3800ccc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.937+0000 7fa377e00640 1 -- 172.21.15.47:0/240166089 >> v1:172.21.15.116:6789/0 conn(0x7fa3800c7e80 legacy=0x7fa3800d0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:44.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.937+0000 7fa377e00640 1 -- 172.21.15.47:0/240166089 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3800cd980 con 0x7fa3800d0820 2024-02-19T15:24:44.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.941+0000 7fa377e00640 1 --2- 172.21.15.47:0/240166089 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3800d0820 0x7fa3800cc740 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa368009870 tx=0x7fa36801c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a27d672c240e0a95 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:44.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.941+0000 7fa37e000640 1 -- 172.21.15.47:0/240166089 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3680aa5f0 con 0x7fa3800d0820 2024-02-19T15:24:44.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.941+0000 7fa37e000640 1 -- 172.21.15.47:0/240166089 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3680aa750 con 0x7fa3800d0820 2024-02-19T15:24:44.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.941+0000 7fa37e000640 1 -- 172.21.15.47:0/240166089 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3680aaae0 con 0x7fa3800d0820 2024-02-19T15:24:44.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.941+0000 7fa385600640 1 -- 172.21.15.47:0/240166089 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3800d0820 msgr2=0x7fa3800cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:44.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.941+0000 7fa385600640 1 --2- 172.21.15.47:0/240166089 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3800d0820 0x7fa3800cc740 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa368009870 tx=0x7fa36801c530 comp rx=0 tx=0).stop 2024-02-19T15:24:44.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.941+0000 7fa385600640 1 -- 172.21.15.47:0/240166089 shutdown_connections 2024-02-19T15:24:44.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.941+0000 7fa385600640 1 --2- 172.21.15.47:0/240166089 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3800c8260 0x7fa3800ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.941+0000 7fa385600640 1 --2- 172.21.15.47:0/240166089 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3800d0820 0x7fa3800cc740 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.941+0000 7fa385600640 1 -- 172.21.15.47:0/240166089 >> 172.21.15.47:0/240166089 conn(0x7fa3800d1b30 msgr2=0x7fa3800d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:44.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.941+0000 7fa385600640 1 -- 172.21.15.47:0/240166089 shutdown_connections 2024-02-19T15:24:44.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.941+0000 7fa385600640 1 -- 172.21.15.47:0/240166089 wait complete. 2024-02-19T15:24:44.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa385600640 1 Processor -- start 2024-02-19T15:24:44.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa385600640 1 -- start start 2024-02-19T15:24:44.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa385600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3800c7e80 0x7fa380157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:44.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa385600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3800c8260 0x7fa380158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:44.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa385600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3800d0820 0x7fa38006fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:44.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa385600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38006d0f0 con 0x7fa3800c7e80 2024-02-19T15:24:44.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa385600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa380158180 con 0x7fa3800d0820 2024-02-19T15:24:44.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa385600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38006cb70 con 0x7fa3800c8260 2024-02-19T15:24:44.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa37ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3800c7e80 0x7fa380157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:44.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa37ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3800c7e80 0x7fa380157c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34316/0 (socket says 172.21.15.47:34316) 2024-02-19T15:24:44.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa37ea00640 1 -- 172.21.15.47:0/651891652 learned_addr learned my addr 172.21.15.47:0/651891652 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:44.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa37f400640 1 --2- 172.21.15.47:0/651891652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3800d0820 0x7fa38006fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:44.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa377e00640 1 --2- 172.21.15.47:0/651891652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3800c8260 0x7fa380158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:44.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa37ea00640 1 -- 172.21.15.47:0/651891652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3800c8260 msgr2=0x7fa380158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:44.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa37ea00640 1 --2- 172.21.15.47:0/651891652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3800c8260 0x7fa380158850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa37ea00640 1 -- 172.21.15.47:0/651891652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3800d0820 msgr2=0x7fa38006fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:44.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa37ea00640 1 --2- 172.21.15.47:0/651891652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3800d0820 0x7fa38006fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:44.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa37ea00640 1 -- 172.21.15.47:0/651891652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3680094d0 con 0x7fa3800c7e80 2024-02-19T15:24:44.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa37ea00640 1 --2- 172.21.15.47:0/651891652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3800c7e80 0x7fa380157c40 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa37000c420 tx=0x7fa37000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:44.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa376a00640 1 -- 172.21.15.47:0/651891652 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3700052d0 con 0x7fa3800c7e80 2024-02-19T15:24:44.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa376a00640 1 -- 172.21.15.47:0/651891652 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa370005430 con 0x7fa3800c7e80 2024-02-19T15:24:44.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa376a00640 1 -- 172.21.15.47:0/651891652 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa370007260 con 0x7fa3800c7e80 2024-02-19T15:24:44.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa385600640 1 -- 172.21.15.47:0/651891652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa38006ce10 con 0x7fa3800c7e80 2024-02-19T15:24:44.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.945+0000 7fa385600640 1 -- 172.21.15.47:0/651891652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3800b0f30 con 0x7fa3800c7e80 2024-02-19T15:24:44.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.949+0000 7fa385600640 1 -- 172.21.15.47:0/651891652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3800d1180 con 0x7fa3800c7e80 2024-02-19T15:24:44.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.949+0000 7fa376a00640 1 -- 172.21.15.47:0/651891652 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa3700073c0 con 0x7fa3800c7e80 2024-02-19T15:24:44.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.949+0000 7fa376a00640 1 --2- 172.21.15.47:0/651891652 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa35c066e40 0x7fa35c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:44.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.949+0000 7fa376a00640 1 -- 172.21.15.47:0/651891652 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa370083dc0 con 0x7fa3800c7e80 2024-02-19T15:24:44.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.949+0000 7fa377e00640 1 --2- 172.21.15.47:0/651891652 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa35c066e40 0x7fa35c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:44.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.949+0000 7fa377e00640 1 --2- 172.21.15.47:0/651891652 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa35c066e40 0x7fa35c069300 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa3680ae000 tx=0x7fa3680b5040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:44.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:44.953+0000 7fa376a00640 1 -- 172.21.15.47:0/651891652 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa370050c50 con 0x7fa3800c7e80 2024-02-19T15:24:45.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.113+0000 7fa385600640 1 -- 172.21.15.47:0/651891652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa3800b0ab0 con 0x7fa3800c7e80 2024-02-19T15:24:45.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.113+0000 7fa376a00640 1 -- 172.21.15.47:0/651891652 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa3800b0ab0 con 0x7fa3800c7e80 2024-02-19T15:24:45.118 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:24:45.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.117+0000 7fa385600640 1 -- 172.21.15.47:0/651891652 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa35c066e40 msgr2=0x7fa35c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:45.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.117+0000 7fa385600640 1 --2- 172.21.15.47:0/651891652 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa35c066e40 0x7fa35c069300 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa3680ae000 tx=0x7fa3680b5040 comp rx=0 tx=0).stop 2024-02-19T15:24:45.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.117+0000 7fa385600640 1 -- 172.21.15.47:0/651891652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3800c7e80 msgr2=0x7fa380157c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:45.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.117+0000 7fa385600640 1 --2- 172.21.15.47:0/651891652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3800c7e80 0x7fa380157c40 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa37000c420 tx=0x7fa37000c8e0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.117+0000 7fa385600640 1 -- 172.21.15.47:0/651891652 shutdown_connections 2024-02-19T15:24:45.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.117+0000 7fa385600640 1 --2- 172.21.15.47:0/651891652 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa35c066e40 0x7fa35c069300 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.117+0000 7fa385600640 1 --2- 172.21.15.47:0/651891652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3800d0820 0x7fa38006fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.117+0000 7fa385600640 1 --2- 172.21.15.47:0/651891652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3800c8260 0x7fa380158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.117+0000 7fa385600640 1 --2- 172.21.15.47:0/651891652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3800c7e80 0x7fa380157c40 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.117+0000 7fa385600640 1 -- 172.21.15.47:0/651891652 >> 172.21.15.47:0/651891652 conn(0x7fa3800d1b30 msgr2=0x7fa3800c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:45.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.117+0000 7fa385600640 1 -- 172.21.15.47:0/651891652 shutdown_connections 2024-02-19T15:24:45.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.117+0000 7fa385600640 1 -- 172.21.15.47:0/651891652 wait complete. 2024-02-19T15:24:45.141 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:24:45.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.281+0000 7f694ec00640 1 Processor -- start 2024-02-19T15:24:45.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.281+0000 7f694ec00640 1 -- start start 2024-02-19T15:24:45.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.281+0000 7f694ec00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f69480ce3b0 0x7f69480ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:45.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.281+0000 7f694ec00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f69480d2650 0x7f69480ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:45.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.281+0000 7f694ec00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69480cc620 con 0x7f694815e280 2024-02-19T15:24:45.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.281+0000 7f694ec00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69480d12e0 con 0x7f69480d2650 2024-02-19T15:24:45.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.281+0000 7f694ec00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69480d1940 con 0x7f69480ce3b0 2024-02-19T15:24:45.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.281+0000 7f693fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f69480d2650 0x7f69480ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:45.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.281+0000 7f693fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f69480d2650 0x7f69480ccab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34326/0 (socket says 172.21.15.47:34326) 2024-02-19T15:24:45.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.281+0000 7f693fe00640 1 -- 172.21.15.47:0/4149842321 learned_addr learned my addr 172.21.15.47:0/4149842321 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:45.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.281+0000 7f6947e00640 1 --2- 172.21.15.47:0/4149842321 >> v2:172.21.15.116:3300/0 conn(0x7f69480ce3b0 0x7f69480ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:45.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.281+0000 7f693fe00640 1 -- 172.21.15.47:0/4149842321 >> v2:172.21.15.116:3300/0 conn(0x7f69480ce3b0 msgr2=0x7f69480ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:45.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.281+0000 7f693fe00640 1 --2- 172.21.15.47:0/4149842321 >> v2:172.21.15.116:3300/0 conn(0x7f69480ce3b0 0x7f69480ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.281+0000 7f693fe00640 1 -- 172.21.15.47:0/4149842321 >> v1:172.21.15.116:6789/0 conn(0x7f694815e280 legacy=0x7f69480ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:45.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.281+0000 7f693fe00640 1 -- 172.21.15.47:0/4149842321 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69480cecd0 con 0x7f69480d2650 2024-02-19T15:24:45.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.285+0000 7f693fe00640 1 --2- 172.21.15.47:0/4149842321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69480d2650 0x7f69480ccab0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6948071220 tx=0x7f693001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d00d396be838431e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:45.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.285+0000 7f6947400640 1 -- 172.21.15.47:0/4149842321 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69300ab600 con 0x7f69480d2650 2024-02-19T15:24:45.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.285+0000 7f6947400640 1 -- 172.21.15.47:0/4149842321 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69300a7050 con 0x7f69480d2650 2024-02-19T15:24:45.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.285+0000 7f6947400640 1 -- 172.21.15.47:0/4149842321 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69300ab980 con 0x7f69480d2650 2024-02-19T15:24:45.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.285+0000 7f694ec00640 1 -- 172.21.15.47:0/4149842321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69480d2650 msgr2=0x7f69480ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:45.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.285+0000 7f694ec00640 1 --2- 172.21.15.47:0/4149842321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69480d2650 0x7f69480ccab0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6948071220 tx=0x7f693001c3c0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.285+0000 7f694ec00640 1 -- 172.21.15.47:0/4149842321 shutdown_connections 2024-02-19T15:24:45.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.285+0000 7f694ec00640 1 --2- 172.21.15.47:0/4149842321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69480d2650 0x7f69480ccab0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.285+0000 7f694ec00640 1 --2- 172.21.15.47:0/4149842321 >> v2:172.21.15.116:3300/0 conn(0x7f69480ce3b0 0x7f69480ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.285+0000 7f694ec00640 1 -- 172.21.15.47:0/4149842321 >> 172.21.15.47:0/4149842321 conn(0x7f69480d1b30 msgr2=0x7f69480d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:45.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.285+0000 7f694ec00640 1 -- 172.21.15.47:0/4149842321 shutdown_connections 2024-02-19T15:24:45.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.285+0000 7f694ec00640 1 -- 172.21.15.47:0/4149842321 wait complete. 2024-02-19T15:24:45.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f694ec00640 1 Processor -- start 2024-02-19T15:24:45.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f694ec00640 1 -- start start 2024-02-19T15:24:45.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f694ec00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69480ce3b0 0x7f694815ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:45.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f694ec00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69480d2650 0x7f6948158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:45.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f694ec00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f694815e280 0x7f6948159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:45.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f694ec00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6948155d50 con 0x7f694815e280 2024-02-19T15:24:45.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f694ec00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6948159c90 con 0x7f69480ce3b0 2024-02-19T15:24:45.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f694ec00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6948155700 con 0x7f69480d2650 2024-02-19T15:24:45.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f694cc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f694815e280 0x7f6948159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:45.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f6947e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69480ce3b0 0x7f694815ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:45.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f694cc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f694815e280 0x7f6948159350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34334/0 (socket says 172.21.15.47:34334) 2024-02-19T15:24:45.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f693fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69480d2650 0x7f6948158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:45.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f694cc00640 1 -- 172.21.15.47:0/3760187050 learned_addr learned my addr 172.21.15.47:0/3760187050 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:45.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f6947e00640 1 -- 172.21.15.47:0/3760187050 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69480d2650 msgr2=0x7f6948158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:45.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f6947e00640 1 --2- 172.21.15.47:0/3760187050 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69480d2650 0x7f6948158e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f6947e00640 1 -- 172.21.15.47:0/3760187050 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f694815e280 msgr2=0x7f6948159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:45.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f6947e00640 1 --2- 172.21.15.47:0/3760187050 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f694815e280 0x7f6948159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f6947e00640 1 -- 172.21.15.47:0/3760187050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69300094d0 con 0x7f69480ce3b0 2024-02-19T15:24:45.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f693fe00640 1 --2- 172.21.15.47:0/3760187050 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69480d2650 0x7f6948158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:24:45.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f6947e00640 1 --2- 172.21.15.47:0/3760187050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69480ce3b0 0x7f694815ad60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f693800d5d0 tx=0x7f693800da90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:45.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f6944c00640 1 -- 172.21.15.47:0/3760187050 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69380125f0 con 0x7f69480ce3b0 2024-02-19T15:24:45.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f6944c00640 1 -- 172.21.15.47:0/3760187050 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6938012750 con 0x7f69480ce3b0 2024-02-19T15:24:45.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f694ec00640 1 -- 172.21.15.47:0/3760187050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69481559e0 con 0x7f69480ce3b0 2024-02-19T15:24:45.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f6944c00640 1 -- 172.21.15.47:0/3760187050 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6938012ae0 con 0x7f69480ce3b0 2024-02-19T15:24:45.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.289+0000 7f694ec00640 1 -- 172.21.15.47:0/3760187050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6948159890 con 0x7f69480ce3b0 2024-02-19T15:24:45.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.293+0000 7f694ec00640 1 -- 172.21.15.47:0/3760187050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69481554f0 con 0x7f69480ce3b0 2024-02-19T15:24:45.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.293+0000 7f6944c00640 1 -- 172.21.15.47:0/3760187050 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6938012c40 con 0x7f69480ce3b0 2024-02-19T15:24:45.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.293+0000 7f6944c00640 1 --2- 172.21.15.47:0/3760187050 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6924066e40 0x7f6924069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:45.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.293+0000 7f6944c00640 1 -- 172.21.15.47:0/3760187050 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6938085890 con 0x7f69480ce3b0 2024-02-19T15:24:45.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.297+0000 7f693fe00640 1 --2- 172.21.15.47:0/3760187050 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6924066e40 0x7f6924069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:45.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.297+0000 7f693fe00640 1 --2- 172.21.15.47:0/3760187050 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6924066e40 0x7f6924069300 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f69300ad000 tx=0x7f69300a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:45.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.297+0000 7f6944c00640 1 -- 172.21.15.47:0/3760187050 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6938052720 con 0x7f69480ce3b0 2024-02-19T15:24:45.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.461+0000 7f694ec00640 1 -- 172.21.15.47:0/3760187050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6948075360 con 0x7f69480ce3b0 2024-02-19T15:24:45.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.461+0000 7f6944c00640 1 -- 172.21.15.47:0/3760187050 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6938055de0 con 0x7f69480ce3b0 2024-02-19T15:24:45.465 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:24:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.461+0000 7f694ec00640 1 -- 172.21.15.47:0/3760187050 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6924066e40 msgr2=0x7f6924069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.461+0000 7f694ec00640 1 --2- 172.21.15.47:0/3760187050 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6924066e40 0x7f6924069300 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f69300ad000 tx=0x7f69300a6000 comp rx=0 tx=0).stop 2024-02-19T15:24:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.461+0000 7f694ec00640 1 -- 172.21.15.47:0/3760187050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69480ce3b0 msgr2=0x7f694815ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.461+0000 7f694ec00640 1 --2- 172.21.15.47:0/3760187050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69480ce3b0 0x7f694815ad60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f693800d5d0 tx=0x7f693800da90 comp rx=0 tx=0).stop 2024-02-19T15:24:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.461+0000 7f694ec00640 1 -- 172.21.15.47:0/3760187050 shutdown_connections 2024-02-19T15:24:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.461+0000 7f694ec00640 1 --2- 172.21.15.47:0/3760187050 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6924066e40 0x7f6924069300 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.461+0000 7f694ec00640 1 --2- 172.21.15.47:0/3760187050 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f694815e280 0x7f6948159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.461+0000 7f694ec00640 1 --2- 172.21.15.47:0/3760187050 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69480d2650 0x7f6948158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.461+0000 7f694ec00640 1 --2- 172.21.15.47:0/3760187050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69480ce3b0 0x7f694815ad60 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.461+0000 7f694ec00640 1 -- 172.21.15.47:0/3760187050 >> 172.21.15.47:0/3760187050 conn(0x7f69480d1b30 msgr2=0x7f69480cc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.461+0000 7f694ec00640 1 -- 172.21.15.47:0/3760187050 shutdown_connections 2024-02-19T15:24:45.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.461+0000 7f694ec00640 1 -- 172.21.15.47:0/3760187050 wait complete. 2024-02-19T15:24:45.487 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-19T15:24:45.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde37200640 1 Processor -- start 2024-02-19T15:24:45.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde37200640 1 -- start start 2024-02-19T15:24:45.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde37200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fde3015dd20 0x7fde300ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:45.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde37200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde300ce6e0 con 0x7fde3015e100 2024-02-19T15:24:45.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde37200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde300acec0 con 0x7fde300cd020 2024-02-19T15:24:45.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde37200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde3006c300 con 0x7fde3015dd20 2024-02-19T15:24:45.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde35400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fde3015e100 0x7fde300cc910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43504/0 (socket says 172.21.15.47:43504) 2024-02-19T15:24:45.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde35400640 1 -- 172.21.15.47:0/3375947386 learned_addr learned my addr 172.21.15.47:0/3375947386 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:45.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde35e00640 1 --2- 172.21.15.47:0/3375947386 >> v2:172.21.15.116:3300/0 conn(0x7fde3015dd20 0x7fde300ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:45.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde34a00640 1 -- 172.21.15.47:0/3375947386 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4026317622 0 0) 0x7fde300ce6e0 con 0x7fde3015e100 2024-02-19T15:24:45.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde34a00640 1 -- 172.21.15.47:0/3375947386 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde18003530 con 0x7fde3015e100 2024-02-19T15:24:45.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde34a00640 1 -- 172.21.15.47:0/3375947386 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 994448433 0 0) 0x7fde18003530 con 0x7fde3015e100 2024-02-19T15:24:45.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde34a00640 1 -- 172.21.15.47:0/3375947386 >> v2:172.21.15.116:3300/0 conn(0x7fde3015dd20 msgr2=0x7fde300ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:45.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde34a00640 1 --2- 172.21.15.47:0/3375947386 >> v2:172.21.15.116:3300/0 conn(0x7fde3015dd20 0x7fde300ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde34a00640 1 -- 172.21.15.47:0/3375947386 >> v1:172.21.15.116:6789/0 conn(0x7fde300cd020 legacy=0x7fde300ad3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:45.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde34a00640 1 -- 172.21.15.47:0/3375947386 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde3006c5e0 con 0x7fde3015e100 2024-02-19T15:24:45.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde34a00640 1 -- 172.21.15.47:0/3375947386 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7fde20001680 con 0x7fde3015e100 2024-02-19T15:24:45.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde34a00640 1 -- 172.21.15.47:0/3375947386 >> v1:172.21.15.47:6789/0 conn(0x7fde3015e100 legacy=0x7fde300cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:45.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde34a00640 1 --2- 172.21.15.47:0/3375947386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde18003980 0x7fde18003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:45.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde34a00640 1 --2- 172.21.15.47:0/3375947386 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde180047d0 0x7fde18004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:45.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde34a00640 1 --2- 172.21.15.47:0/3375947386 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde18005190 0x7fde1800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:45.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde34a00640 1 -- 172.21.15.47:0/3375947386 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3006c300 con 0x7fde180047d0 2024-02-19T15:24:45.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde34a00640 1 -- 172.21.15.47:0/3375947386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1800c0e0 con 0x7fde18003980 2024-02-19T15:24:45.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde34a00640 1 -- 172.21.15.47:0/3375947386 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1800c2a0 con 0x7fde18005190 2024-02-19T15:24:45.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde35e00640 1 --2- 172.21.15.47:0/3375947386 >> v2:172.21.15.116:3300/0 conn(0x7fde3015dd20 0x7fde300ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:24:45.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.625+0000 7fde35400640 1 --2- 172.21.15.47:0/3375947386 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde180047d0 0x7fde18004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:45.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.629+0000 7fde36800640 1 --2- 172.21.15.47:0/3375947386 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde18005190 0x7fde1800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:45.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.629+0000 7fde35e00640 1 --2- 172.21.15.47:0/3375947386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde18003980 0x7fde18003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:45.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.629+0000 7fde35400640 1 -- 172.21.15.47:0/3375947386 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde18005190 msgr2=0x7fde1800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:45.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.629+0000 7fde35400640 1 --2- 172.21.15.47:0/3375947386 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde18005190 0x7fde1800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.629+0000 7fde35400640 1 -- 172.21.15.47:0/3375947386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde18003980 msgr2=0x7fde18003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:45.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.629+0000 7fde35400640 1 --2- 172.21.15.47:0/3375947386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde18003980 0x7fde18003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.629+0000 7fde35400640 1 -- 172.21.15.47:0/3375947386 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde18003530 con 0x7fde180047d0 2024-02-19T15:24:45.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.629+0000 7fde35e00640 1 --2- 172.21.15.47:0/3375947386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde18003980 0x7fde18003d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:45.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.629+0000 7fde36800640 1 -- 172.21.15.47:0/3375947386 reap_dead start 2024-02-19T15:24:45.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.633+0000 7fde35400640 1 --2- 172.21.15.47:0/3375947386 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde180047d0 0x7fde18004c50 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fde2000c1c0 tx=0x7fde2001cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3e7e4252acaae04 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:45.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.633+0000 7fde34a00640 1 -- 172.21.15.47:0/3375947386 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde20005ae0 con 0x7fde180047d0 2024-02-19T15:24:45.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.633+0000 7fde34a00640 1 -- 172.21.15.47:0/3375947386 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde20005c40 con 0x7fde180047d0 2024-02-19T15:24:45.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.633+0000 7fde34a00640 1 -- 172.21.15.47:0/3375947386 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde200a5790 con 0x7fde180047d0 2024-02-19T15:24:45.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.633+0000 7fde37200640 1 -- 172.21.15.47:0/3375947386 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde180047d0 msgr2=0x7fde18004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:45.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.633+0000 7fde37200640 1 --2- 172.21.15.47:0/3375947386 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde180047d0 0x7fde18004c50 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fde2000c1c0 tx=0x7fde2001cbd0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.633+0000 7fde37200640 1 -- 172.21.15.47:0/3375947386 shutdown_connections 2024-02-19T15:24:45.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.633+0000 7fde37200640 1 --2- 172.21.15.47:0/3375947386 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde180047d0 0x7fde18004c50 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.633+0000 7fde37200640 1 -- 172.21.15.47:0/3375947386 >> 172.21.15.47:0/3375947386 conn(0x7fde300b1be0 msgr2=0x7fde300b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:45.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.633+0000 7fde37200640 1 -- 172.21.15.47:0/3375947386 shutdown_connections 2024-02-19T15:24:45.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.633+0000 7fde37200640 1 -- 172.21.15.47:0/3375947386 wait complete. 2024-02-19T15:24:45.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.633+0000 7fde37200640 1 Processor -- start 2024-02-19T15:24:45.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde37200640 1 -- start start 2024-02-19T15:24:45.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde37200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde180047d0 0x7fde3015e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:45.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde37200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde3009d250 0x7fde30153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:45.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde37200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde3009b1e0 0x7fde3009b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:45.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde37200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3009bba0 con 0x7fde3009b1e0 2024-02-19T15:24:45.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde37200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3015ec00 con 0x7fde180047d0 2024-02-19T15:24:45.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde37200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde3015eee0 con 0x7fde3009d250 2024-02-19T15:24:45.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde36800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde3009b1e0 0x7fde3009b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:45.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde36800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde3009b1e0 0x7fde3009b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34358/0 (socket says 172.21.15.47:34358) 2024-02-19T15:24:45.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde36800640 1 -- 172.21.15.47:0/755197 learned_addr learned my addr 172.21.15.47:0/755197 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:45.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde35400640 1 --2- 172.21.15.47:0/755197 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde3009d250 0x7fde30153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:45.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde35e00640 1 --2- 172.21.15.47:0/755197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde180047d0 0x7fde3015e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:45.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde36800640 1 -- 172.21.15.47:0/755197 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde3009d250 msgr2=0x7fde30153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:45.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde36800640 1 --2- 172.21.15.47:0/755197 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde3009d250 0x7fde30153400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde36800640 1 -- 172.21.15.47:0/755197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde180047d0 msgr2=0x7fde3015e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:45.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde36800640 1 --2- 172.21.15.47:0/755197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde180047d0 0x7fde3015e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde36800640 1 -- 172.21.15.47:0/755197 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde3015f1c0 con 0x7fde3009b1e0 2024-02-19T15:24:45.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde35e00640 1 --2- 172.21.15.47:0/755197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde180047d0 0x7fde3015e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:45.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde36800640 1 --2- 172.21.15.47:0/755197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde3009b1e0 0x7fde3009b660 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fde300674f0 tx=0x7fde2c00ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:45.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde2a000640 1 -- 172.21.15.47:0/755197 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2c006e30 con 0x7fde3009b1e0 2024-02-19T15:24:45.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde2a000640 1 -- 172.21.15.47:0/755197 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde2c003980 con 0x7fde3009b1e0 2024-02-19T15:24:45.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde2a000640 1 -- 172.21.15.47:0/755197 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2c003d30 con 0x7fde3009b1e0 2024-02-19T15:24:45.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde37200640 1 -- 172.21.15.47:0/755197 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde3015f4a0 con 0x7fde3009b1e0 2024-02-19T15:24:45.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde37200640 1 -- 172.21.15.47:0/755197 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde301617c0 con 0x7fde3009b1e0 2024-02-19T15:24:45.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.637+0000 7fde37200640 1 -- 172.21.15.47:0/755197 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde3015f9c0 con 0x7fde3009b1e0 2024-02-19T15:24:45.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.641+0000 7fde2a000640 1 -- 172.21.15.47:0/755197 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fde2c005110 con 0x7fde3009b1e0 2024-02-19T15:24:45.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.641+0000 7fde2a000640 1 --2- 172.21.15.47:0/755197 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fde14066e40 0x7fde14069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:45.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.641+0000 7fde2a000640 1 -- 172.21.15.47:0/755197 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde2c0841d0 con 0x7fde3009b1e0 2024-02-19T15:24:45.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.641+0000 7fde35e00640 1 --2- 172.21.15.47:0/755197 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fde14066e40 0x7fde14069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:45.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.641+0000 7fde35e00640 1 --2- 172.21.15.47:0/755197 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fde14066e40 0x7fde14069300 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fde30050570 tx=0x7fde24008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:45.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.645+0000 7fde2a000640 1 -- 172.21.15.47:0/755197 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fde3015f9c0 con 0x7fde3009b1e0 2024-02-19T15:24:45.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.817+0000 7fde37200640 1 -- 172.21.15.47:0/755197 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fde3015f9c0 con 0x7fde3009b1e0 2024-02-19T15:24:45.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.817+0000 7fde2a000640 1 -- 172.21.15.47:0/755197 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v57) v1 ==== 80+0+0 (secure 0 0 0) 0x7fde3015f9c0 con 0x7fde3009b1e0 2024-02-19T15:24:45.823 INFO:teuthology.orchestra.run.smithi047.stderr:listed 0 entries 2024-02-19T15:24:45.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.821+0000 7fde37200640 1 -- 172.21.15.47:0/755197 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fde14066e40 msgr2=0x7fde14069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:45.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.821+0000 7fde37200640 1 --2- 172.21.15.47:0/755197 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fde14066e40 0x7fde14069300 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fde30050570 tx=0x7fde24008040 comp rx=0 tx=0).stop 2024-02-19T15:24:45.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.821+0000 7fde37200640 1 -- 172.21.15.47:0/755197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde3009b1e0 msgr2=0x7fde3009b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:45.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.821+0000 7fde37200640 1 --2- 172.21.15.47:0/755197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde3009b1e0 0x7fde3009b660 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fde300674f0 tx=0x7fde2c00ff70 comp rx=0 tx=0).stop 2024-02-19T15:24:45.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.821+0000 7fde37200640 1 -- 172.21.15.47:0/755197 shutdown_connections 2024-02-19T15:24:45.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.821+0000 7fde37200640 1 --2- 172.21.15.47:0/755197 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fde14066e40 0x7fde14069300 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.821+0000 7fde37200640 1 --2- 172.21.15.47:0/755197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde3009b1e0 0x7fde3009b660 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.821+0000 7fde37200640 1 --2- 172.21.15.47:0/755197 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde3009d250 0x7fde30153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.821+0000 7fde37200640 1 --2- 172.21.15.47:0/755197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde180047d0 0x7fde3015e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:45.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.821+0000 7fde37200640 1 -- 172.21.15.47:0/755197 >> 172.21.15.47:0/755197 conn(0x7fde300b1be0 msgr2=0x7fde30074000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:45.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.821+0000 7fde37200640 1 -- 172.21.15.47:0/755197 shutdown_connections 2024-02-19T15:24:45.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:45.821+0000 7fde37200640 1 -- 172.21.15.47:0/755197 wait complete. 2024-02-19T15:24:45.854 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-02-19T15:24:45.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:45.854 DEBUG:teuthology.orchestra.run.smithi047:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-02-19T15:24:45.938 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi047.front.sepia.ceph.com 2024-02-19T15:24:45.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:45.938 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-19T15:24:45.952 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:24:46.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.129+0000 7f40df400640 1 Processor -- start 2024-02-19T15:24:46.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.129+0000 7f40df400640 1 -- start start 2024-02-19T15:24:46.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.133+0000 7f40df400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f40d806d650 0x7f40d80d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:46.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.133+0000 7f40df400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d80ced10 0x7f40d806bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:46.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.133+0000 7f40df400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40d80cce30 con 0x7f40d80cded0 2024-02-19T15:24:46.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.133+0000 7f40df400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40d80cd110 con 0x7f40d806d650 2024-02-19T15:24:46.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.133+0000 7f40df400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d80cd3f0 con 0x7f40d80ced10 2024-02-19T15:24:46.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.133+0000 7f40dca00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f40d806d650 0x7f40d80d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:46.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.133+0000 7f40d7e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f40d80cded0 0x7f40d80ce600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43508/0 (socket says 172.21.15.47:43508) 2024-02-19T15:24:46.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.133+0000 7f40d7e00640 1 -- 172.21.15.47:0/3453465737 learned_addr learned my addr 172.21.15.47:0/3453465737 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:46.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.133+0000 7f40dd400640 1 --2- 172.21.15.47:0/3453465737 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d80ced10 0x7f40d806bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:46.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.133+0000 7f40dca00640 1 -- 172.21.15.47:0/3453465737 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d80ced10 msgr2=0x7f40d806bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:46.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.133+0000 7f40dca00640 1 --2- 172.21.15.47:0/3453465737 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d80ced10 0x7f40d806bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:46.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.133+0000 7f40dca00640 1 -- 172.21.15.47:0/3453465737 >> v1:172.21.15.47:6789/0 conn(0x7f40d80cded0 legacy=0x7f40d80ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:46.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.133+0000 7f40dca00640 1 -- 172.21.15.47:0/3453465737 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d80cd6d0 con 0x7f40d806d650 2024-02-19T15:24:46.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.133+0000 7f40dd400640 1 --2- 172.21.15.47:0/3453465737 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d80ced10 0x7f40d806bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:24:46.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.133+0000 7f40d7400640 1 -- 172.21.15.47:0/3453465737 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540723997 0 0) 0x7f40d80cce30 con 0x7f40d80cded0 2024-02-19T15:24:46.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.133+0000 7f40dca00640 1 --2- 172.21.15.47:0/3453465737 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40d806d650 0x7f40d80d08b0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f40cc0097c0 tx=0x7f40cc01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c49998836d747904 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:46.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40d7400640 1 -- 172.21.15.47:0/3453465737 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40cc0ab600 con 0x7f40d806d650 2024-02-19T15:24:46.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40d7400640 1 -- 172.21.15.47:0/3453465737 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40cc0ab760 con 0x7f40d806d650 2024-02-19T15:24:46.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40d7400640 1 -- 172.21.15.47:0/3453465737 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40cc0abaf0 con 0x7f40d806d650 2024-02-19T15:24:46.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40df400640 1 -- 172.21.15.47:0/3453465737 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40d806d650 msgr2=0x7f40d80d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:46.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40df400640 1 --2- 172.21.15.47:0/3453465737 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40d806d650 0x7f40d80d08b0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f40cc0097c0 tx=0x7f40cc01c3e0 comp rx=0 tx=0).stop 2024-02-19T15:24:46.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40df400640 1 -- 172.21.15.47:0/3453465737 shutdown_connections 2024-02-19T15:24:46.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40df400640 1 --2- 172.21.15.47:0/3453465737 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d80ced10 0x7f40d806bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:46.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40df400640 1 --2- 172.21.15.47:0/3453465737 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40d806d650 0x7f40d80d08b0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:46.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40df400640 1 -- 172.21.15.47:0/3453465737 >> 172.21.15.47:0/3453465737 conn(0x7f40d80d1bf0 msgr2=0x7f40d80d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:46.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40df400640 1 -- 172.21.15.47:0/3453465737 shutdown_connections 2024-02-19T15:24:46.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40df400640 1 -- 172.21.15.47:0/3453465737 wait complete. 2024-02-19T15:24:46.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40df400640 1 Processor -- start 2024-02-19T15:24:46.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40df400640 1 -- start start 2024-02-19T15:24:46.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40df400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d806d650 0x7f40d814aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:46.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40df400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40d80cded0 0x7f40d814d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:46.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40df400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d80ced10 0x7f40d814d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:46.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40df400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d814af50 con 0x7f40d80cded0 2024-02-19T15:24:46.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40df400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d814bf90 con 0x7f40d806d650 2024-02-19T15:24:46.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40df400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d814c270 con 0x7f40d80ced10 2024-02-19T15:24:46.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40d7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40d80cded0 0x7f40d814d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:46.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40d7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40d80cded0 0x7f40d814d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34376/0 (socket says 172.21.15.47:34376) 2024-02-19T15:24:46.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.137+0000 7f40d7e00640 1 -- 172.21.15.47:0/3925877180 learned_addr learned my addr 172.21.15.47:0/3925877180 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:46.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.141+0000 7f40dd400640 1 --2- 172.21.15.47:0/3925877180 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d80ced10 0x7f40d814d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:46.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.141+0000 7f40dca00640 1 --2- 172.21.15.47:0/3925877180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d806d650 0x7f40d814aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:46.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.141+0000 7f40d7e00640 1 -- 172.21.15.47:0/3925877180 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d80ced10 msgr2=0x7f40d814d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:46.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.141+0000 7f40d7e00640 1 --2- 172.21.15.47:0/3925877180 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d80ced10 0x7f40d814d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:46.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.141+0000 7f40d7e00640 1 -- 172.21.15.47:0/3925877180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d806d650 msgr2=0x7f40d814aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:46.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.141+0000 7f40d7e00640 1 --2- 172.21.15.47:0/3925877180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d806d650 0x7f40d814aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:46.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.141+0000 7f40d7e00640 1 -- 172.21.15.47:0/3925877180 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d0009a70 con 0x7f40d80cded0 2024-02-19T15:24:46.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.141+0000 7f40d7e00640 1 --2- 172.21.15.47:0/3925877180 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40d80cded0 0x7f40d814d310 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f40c400c480 tx=0x7f40c400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:46.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.141+0000 7f40d4c00640 1 -- 172.21.15.47:0/3925877180 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c40074e0 con 0x7f40d80cded0 2024-02-19T15:24:46.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.141+0000 7f40d4c00640 1 -- 172.21.15.47:0/3925877180 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c4007660 con 0x7f40d80cded0 2024-02-19T15:24:46.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.141+0000 7f40d4c00640 1 -- 172.21.15.47:0/3925877180 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c4004a30 con 0x7f40d80cded0 2024-02-19T15:24:46.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.141+0000 7f40df400640 1 -- 172.21.15.47:0/3925877180 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40cc009420 con 0x7f40d80cded0 2024-02-19T15:24:46.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.141+0000 7f40df400640 1 -- 172.21.15.47:0/3925877180 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40d814c830 con 0x7f40d80cded0 2024-02-19T15:24:46.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.141+0000 7f40df400640 1 -- 172.21.15.47:0/3925877180 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d8075360 con 0x7f40d80cded0 2024-02-19T15:24:46.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.141+0000 7f40d4c00640 1 -- 172.21.15.47:0/3925877180 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f40c4004b90 con 0x7f40d80cded0 2024-02-19T15:24:46.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.141+0000 7f40d4c00640 1 --2- 172.21.15.47:0/3925877180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f40b8066e40 0x7f40b8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:46.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.145+0000 7f40d4c00640 1 -- 172.21.15.47:0/3925877180 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40c4084790 con 0x7f40d80cded0 2024-02-19T15:24:46.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.145+0000 7f40dca00640 1 --2- 172.21.15.47:0/3925877180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f40b8066e40 0x7f40b8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:46.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.145+0000 7f40dca00640 1 --2- 172.21.15.47:0/3925877180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f40b8066e40 0x7f40b8069300 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f40cc0af000 tx=0x7f40cc0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:46.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.153+0000 7f40d4c00640 1 -- 172.21.15.47:0/3925877180 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f40c40514e0 con 0x7f40d80cded0 2024-02-19T15:24:46.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.465+0000 7f40df400640 1 -- 172.21.15.47:0/3925877180 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40d80cf580 con 0x7f40d80cded0 2024-02-19T15:24:46.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.465+0000 7f40d4c00640 1 -- 172.21.15.47:0/3925877180 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f40c4054ba0 con 0x7f40d80cded0 2024-02-19T15:24:46.468 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:46.468 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5384,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2825837364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2825837364},{"type":"v1","addr":"172.21.15.47:6835","nonce":2825837364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2251385237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2251385237},{"type":"v1","addr":"172.21.15.116:6837","nonce":2251385237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/2438199468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2438199468},{"type":"v1","addr":"172.21.15.116:6836","nonce":2438199468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:24:25.161496+0000","modified":"2024-02-19T15:24:28.184021+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5910":{"gid":5910,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6834/61582103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":61582103},{"type":"v1","addr":"172.21.15.47:6834","nonce":61582103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-19T15:24:46.468 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 33 2024-02-19T15:24:46.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.465+0000 7f40df400640 1 -- 172.21.15.47:0/3925877180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f40b8066e40 msgr2=0x7f40b8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:46.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.465+0000 7f40df400640 1 --2- 172.21.15.47:0/3925877180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f40b8066e40 0x7f40b8069300 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f40cc0af000 tx=0x7f40cc0a7000 comp rx=0 tx=0).stop 2024-02-19T15:24:46.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.465+0000 7f40df400640 1 -- 172.21.15.47:0/3925877180 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40d80cded0 msgr2=0x7f40d814d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:46.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.465+0000 7f40df400640 1 --2- 172.21.15.47:0/3925877180 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40d80cded0 0x7f40d814d310 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f40c400c480 tx=0x7f40c400c880 comp rx=0 tx=0).stop 2024-02-19T15:24:46.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.465+0000 7f40df400640 1 -- 172.21.15.47:0/3925877180 shutdown_connections 2024-02-19T15:24:46.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.465+0000 7f40df400640 1 --2- 172.21.15.47:0/3925877180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f40b8066e40 0x7f40b8069300 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:46.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.465+0000 7f40df400640 1 --2- 172.21.15.47:0/3925877180 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d80ced10 0x7f40d814d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:46.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.465+0000 7f40df400640 1 --2- 172.21.15.47:0/3925877180 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40d80cded0 0x7f40d814d310 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:46.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.465+0000 7f40df400640 1 --2- 172.21.15.47:0/3925877180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d806d650 0x7f40d814aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:46.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.465+0000 7f40df400640 1 -- 172.21.15.47:0/3925877180 >> 172.21.15.47:0/3925877180 conn(0x7f40d80d1bf0 msgr2=0x7f40d80c7660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:46.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.469+0000 7f40df400640 1 -- 172.21.15.47:0/3925877180 shutdown_connections 2024-02-19T15:24:46.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.469+0000 7f40df400640 1 -- 172.21.15.47:0/3925877180 wait complete. 2024-02-19T15:24:46.499 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dirfrag split /subdir 0/0 1 2024-02-19T15:24:46.657 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2024-02-19T15:24:46.657 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:24:46.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.793+0000 7f9402a00640 1 Processor -- start 2024-02-19T15:24:46.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.793+0000 7f9402a00640 1 -- start start 2024-02-19T15:24:46.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.793+0000 7f9402a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f93fc0cd020 0x7f93fc0ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:46.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.793+0000 7f9402a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93fc15e100 0x7f93fc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:46.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.793+0000 7f9402a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93fc0ce6e0 con 0x7f93fc15dd20 2024-02-19T15:24:46.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.793+0000 7f9402a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93fc0acec0 con 0x7f93fc0cd020 2024-02-19T15:24:46.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.793+0000 7f9402a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc06c300 con 0x7f93fc15e100 2024-02-19T15:24:46.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.797+0000 7f9401600640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f93fc15dd20 0x7f93fc0cc910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:43558/0 (socket says 172.21.15.47:43558) 2024-02-19T15:24:46.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.797+0000 7f9401600640 1 -- 172.21.15.47:0/4189628270 learned_addr learned my addr 172.21.15.47:0/4189628270 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:46.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.797+0000 7f9400c00640 1 --2- 172.21.15.47:0/4189628270 >> v2:172.21.15.116:3300/0 conn(0x7f93fc0cd020 0x7f93fc0ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:46.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.797+0000 7f9402000640 1 --2- 172.21.15.47:0/4189628270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93fc15e100 0x7f93fc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:46.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.797+0000 7f93fbe00640 1 -- 172.21.15.47:0/4189628270 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889200994 0 0) 0x7f93fc0ce6e0 con 0x7f93fc15dd20 2024-02-19T15:24:46.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.797+0000 7f93fbe00640 1 -- 172.21.15.47:0/4189628270 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93e4003530 con 0x7f93fc15dd20 2024-02-19T15:24:46.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.797+0000 7f9400c00640 1 -- 172.21.15.47:0/4189628270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93fc15e100 msgr2=0x7f93fc06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:46.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.797+0000 7f9400c00640 1 --2- 172.21.15.47:0/4189628270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93fc15e100 0x7f93fc06b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:46.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.797+0000 7f9400c00640 1 -- 172.21.15.47:0/4189628270 >> v1:172.21.15.116:6789/0 conn(0x7f93fc15dd20 legacy=0x7f93fc0cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:46.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.797+0000 7f9400c00640 1 -- 172.21.15.47:0/4189628270 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93fc06c5e0 con 0x7f93fc0cd020 2024-02-19T15:24:46.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.797+0000 7f9402000640 1 --2- 172.21.15.47:0/4189628270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93fc15e100 0x7f93fc06b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:24:46.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.797+0000 7f9400c00640 1 --2- 172.21.15.47:0/4189628270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93fc0cd020 0x7f93fc0ca6e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f93e80097c0 tx=0x7f93e801c350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbf16be31e020c5c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:46.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f93fbe00640 1 -- 172.21.15.47:0/4189628270 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e80ab600 con 0x7f93fc0cd020 2024-02-19T15:24:46.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f93fbe00640 1 -- 172.21.15.47:0/4189628270 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e80a7050 con 0x7f93fc0cd020 2024-02-19T15:24:46.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f93fbe00640 1 -- 172.21.15.47:0/4189628270 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e80ab980 con 0x7f93fc0cd020 2024-02-19T15:24:46.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f9402a00640 1 -- 172.21.15.47:0/4189628270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93fc0cd020 msgr2=0x7f93fc0ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:46.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f9402a00640 1 --2- 172.21.15.47:0/4189628270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93fc0cd020 0x7f93fc0ca6e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f93e80097c0 tx=0x7f93e801c350 comp rx=0 tx=0).stop 2024-02-19T15:24:46.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f9402a00640 1 -- 172.21.15.47:0/4189628270 shutdown_connections 2024-02-19T15:24:46.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f9402a00640 1 --2- 172.21.15.47:0/4189628270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93fc15e100 0x7f93fc06b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:46.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f9402a00640 1 --2- 172.21.15.47:0/4189628270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93fc0cd020 0x7f93fc0ca6e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:46.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f9402a00640 1 -- 172.21.15.47:0/4189628270 >> 172.21.15.47:0/4189628270 conn(0x7f93fc0b1be0 msgr2=0x7f93fc0b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:46.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f9402a00640 1 -- 172.21.15.47:0/4189628270 shutdown_connections 2024-02-19T15:24:46.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f9402a00640 1 -- 172.21.15.47:0/4189628270 wait complete. 2024-02-19T15:24:46.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f9402a00640 1 Processor -- start 2024-02-19T15:24:46.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f9402a00640 1 -- start start 2024-02-19T15:24:46.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f9402a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93fc0cd020 0x7f93fc15b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:46.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f9402a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93fc15dd20 0x7f93fc0729c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:46.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f9402a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93fc15e100 0x7f93fc154540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:46.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f9402a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc072f00 con 0x7f93fc0cd020 2024-02-19T15:24:46.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f9402a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc159a20 con 0x7f93fc15dd20 2024-02-19T15:24:46.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f9402a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc15bd70 con 0x7f93fc15e100 2024-02-19T15:24:46.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f9401600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93fc0cd020 0x7f93fc15b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:46.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f9401600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93fc0cd020 0x7f93fc15b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34378/0 (socket says 172.21.15.47:34378) 2024-02-19T15:24:46.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.801+0000 7f9401600640 1 -- 172.21.15.47:0/3214116940 learned_addr learned my addr 172.21.15.47:0/3214116940 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:46.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.805+0000 7f9400c00640 1 --2- 172.21.15.47:0/3214116940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93fc15dd20 0x7f93fc0729c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:46.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.805+0000 7f9402000640 1 --2- 172.21.15.47:0/3214116940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93fc15e100 0x7f93fc154540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:46.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.805+0000 7f9401600640 1 -- 172.21.15.47:0/3214116940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93fc15e100 msgr2=0x7f93fc154540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:46.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.805+0000 7f9401600640 1 --2- 172.21.15.47:0/3214116940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93fc15e100 0x7f93fc154540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:46.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.805+0000 7f9401600640 1 -- 172.21.15.47:0/3214116940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93fc15dd20 msgr2=0x7f93fc0729c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:46.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.805+0000 7f9401600640 1 --2- 172.21.15.47:0/3214116940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93fc15dd20 0x7f93fc0729c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:46.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.805+0000 7f9401600640 1 -- 172.21.15.47:0/3214116940 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f4009a70 con 0x7f93fc0cd020 2024-02-19T15:24:46.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.805+0000 7f9401600640 1 --2- 172.21.15.47:0/3214116940 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93fc0cd020 0x7f93fc15b720 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f93f000c3c0 tx=0x7f93f000c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:46.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.805+0000 7f93f9600640 1 -- 172.21.15.47:0/3214116940 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f0005db0 con 0x7f93fc0cd020 2024-02-19T15:24:46.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.805+0000 7f93f9600640 1 -- 172.21.15.47:0/3214116940 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f00070d0 con 0x7f93fc0cd020 2024-02-19T15:24:46.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.805+0000 7f9402a00640 1 -- 172.21.15.47:0/3214116940 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e8009420 con 0x7f93fc0cd020 2024-02-19T15:24:46.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.805+0000 7f93f9600640 1 -- 172.21.15.47:0/3214116940 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f00074a0 con 0x7f93fc0cd020 2024-02-19T15:24:46.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.805+0000 7f9402a00640 1 -- 172.21.15.47:0/3214116940 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93fc15c330 con 0x7f93fc0cd020 2024-02-19T15:24:46.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.805+0000 7f9402a00640 1 -- 172.21.15.47:0/3214116940 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93fc075360 con 0x7f93fc0cd020 2024-02-19T15:24:46.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.805+0000 7f93f9600640 1 -- 172.21.15.47:0/3214116940 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f93f0007600 con 0x7f93fc0cd020 2024-02-19T15:24:46.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.805+0000 7f93f9600640 1 --2- 172.21.15.47:0/3214116940 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f93dc066e40 0x7f93dc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:46.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.805+0000 7f93f9600640 1 -- 172.21.15.47:0/3214116940 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f93f0084010 con 0x7f93fc0cd020 2024-02-19T15:24:46.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.809+0000 7f9400c00640 1 --2- 172.21.15.47:0/3214116940 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f93dc066e40 0x7f93dc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:46.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.809+0000 7f9400c00640 1 --2- 172.21.15.47:0/3214116940 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f93dc066e40 0x7f93dc069300 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f93e80ad000 tx=0x7f93e8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:46.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:46.813+0000 7f93f9600640 1 -- 172.21.15.47:0/3214116940 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f93f0050eb0 con 0x7f93fc0cd020 2024-02-19T15:24:47.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.177+0000 7f9402a00640 1 -- 172.21.15.47:0/3214116940 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93fc066e40 con 0x7f93fc0cd020 2024-02-19T15:24:47.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.177+0000 7f93f9600640 1 -- 172.21.15.47:0/3214116940 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f93f0054570 con 0x7f93fc0cd020 2024-02-19T15:24:47.182 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:47.182 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5384,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2825837364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2825837364},{"type":"v1","addr":"172.21.15.47:6835","nonce":2825837364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2251385237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2251385237},{"type":"v1","addr":"172.21.15.116:6837","nonce":2251385237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/2438199468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2438199468},{"type":"v1","addr":"172.21.15.116:6836","nonce":2438199468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:24:25.161496+0000","modified":"2024-02-19T15:24:28.184021+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5910":{"gid":5910,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6834/61582103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":61582103},{"type":"v1","addr":"172.21.15.47:6834","nonce":61582103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-19T15:24:47.182 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 33 2024-02-19T15:24:47.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.181+0000 7f9402a00640 1 -- 172.21.15.47:0/3214116940 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f93dc066e40 msgr2=0x7f93dc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:47.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.181+0000 7f9402a00640 1 --2- 172.21.15.47:0/3214116940 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f93dc066e40 0x7f93dc069300 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f93e80ad000 tx=0x7f93e8008040 comp rx=0 tx=0).stop 2024-02-19T15:24:47.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.181+0000 7f9402a00640 1 -- 172.21.15.47:0/3214116940 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93fc0cd020 msgr2=0x7f93fc15b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:47.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.181+0000 7f9402a00640 1 --2- 172.21.15.47:0/3214116940 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93fc0cd020 0x7f93fc15b720 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f93f000c3c0 tx=0x7f93f000c7c0 comp rx=0 tx=0).stop 2024-02-19T15:24:47.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.181+0000 7f9402a00640 1 -- 172.21.15.47:0/3214116940 shutdown_connections 2024-02-19T15:24:47.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.181+0000 7f9402a00640 1 --2- 172.21.15.47:0/3214116940 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f93dc066e40 0x7f93dc069300 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:47.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.181+0000 7f9402a00640 1 --2- 172.21.15.47:0/3214116940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f93fc15e100 0x7f93fc154540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:47.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.181+0000 7f9402a00640 1 --2- 172.21.15.47:0/3214116940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93fc15dd20 0x7f93fc0729c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:47.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.181+0000 7f9402a00640 1 --2- 172.21.15.47:0/3214116940 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93fc0cd020 0x7f93fc15b720 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:47.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.181+0000 7f9402a00640 1 -- 172.21.15.47:0/3214116940 >> 172.21.15.47:0/3214116940 conn(0x7f93fc0b1be0 msgr2=0x7f93fc0cc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:47.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.181+0000 7f9402a00640 1 -- 172.21.15.47:0/3214116940 shutdown_connections 2024-02-19T15:24:47.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.181+0000 7f9402a00640 1 -- 172.21.15.47:0/3214116940 wait complete. 2024-02-19T15:24:47.204 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-02-19T15:24:47.387 INFO:teuthology.orchestra.run.smithi047.stdout:{"message":"","return_code":0} 2024-02-19T15:24:47.388 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-02-19T15:24:47.388 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-02-19T15:24:47.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.525+0000 7f11da200640 1 Processor -- start 2024-02-19T15:24:47.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.525+0000 7f11da200640 1 -- start start 2024-02-19T15:24:47.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.525+0000 7f11da200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f11d4145590 0x7f11d40727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:47.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.525+0000 7f11da200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11d40cea80 0x7f11d406c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:47.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.525+0000 7f11da200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d415d280 con 0x7f11d415c390 2024-02-19T15:24:47.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.525+0000 7f11da200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11d4070ff0 con 0x7f11d4145590 2024-02-19T15:24:47.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.525+0000 7f11da200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d40709e0 con 0x7f11d40cea80 2024-02-19T15:24:47.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.525+0000 7f11d3400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f11d4145590 0x7f11d40727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:47.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.525+0000 7f11d3400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f11d4145590 0x7f11d40727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34380/0 (socket says 172.21.15.47:34380) 2024-02-19T15:24:47.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.525+0000 7f11d3400640 1 -- 172.21.15.47:0/4027444778 learned_addr learned my addr 172.21.15.47:0/4027444778 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:47.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.525+0000 7f11cbe00640 1 --2- 172.21.15.47:0/4027444778 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11d40cea80 0x7f11d406c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:47.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.525+0000 7f11d3400640 1 -- 172.21.15.47:0/4027444778 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11d40cea80 msgr2=0x7f11d406c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:47.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.525+0000 7f11d3400640 1 --2- 172.21.15.47:0/4027444778 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11d40cea80 0x7f11d406c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:47.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.525+0000 7f11d3400640 1 -- 172.21.15.47:0/4027444778 >> v1:172.21.15.116:6789/0 conn(0x7f11d415c390 legacy=0x7f11d415c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:47.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.525+0000 7f11d3400640 1 -- 172.21.15.47:0/4027444778 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d406cee0 con 0x7f11d4145590 2024-02-19T15:24:47.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.529+0000 7f11d3400640 1 --2- 172.21.15.47:0/4027444778 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11d4145590 0x7f11d40727b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f11c40098a0 tx=0x7f11c401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc7074caa44e26e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:47.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.529+0000 7f11d2a00640 1 -- 172.21.15.47:0/4027444778 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c40ab650 con 0x7f11d4145590 2024-02-19T15:24:47.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.529+0000 7f11d2a00640 1 -- 172.21.15.47:0/4027444778 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c40ab7b0 con 0x7f11d4145590 2024-02-19T15:24:47.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.529+0000 7f11d2a00640 1 -- 172.21.15.47:0/4027444778 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c40abb40 con 0x7f11d4145590 2024-02-19T15:24:47.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.529+0000 7f11da200640 1 -- 172.21.15.47:0/4027444778 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11d4145590 msgr2=0x7f11d40727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:47.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.529+0000 7f11da200640 1 --2- 172.21.15.47:0/4027444778 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11d4145590 0x7f11d40727b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f11c40098a0 tx=0x7f11c401c540 comp rx=0 tx=0).stop 2024-02-19T15:24:47.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.529+0000 7f11da200640 1 -- 172.21.15.47:0/4027444778 shutdown_connections 2024-02-19T15:24:47.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.529+0000 7f11da200640 1 --2- 172.21.15.47:0/4027444778 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11d40cea80 0x7f11d406c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:47.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.529+0000 7f11da200640 1 --2- 172.21.15.47:0/4027444778 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11d4145590 0x7f11d40727b0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:47.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.529+0000 7f11da200640 1 -- 172.21.15.47:0/4027444778 >> 172.21.15.47:0/4027444778 conn(0x7f11d40b16a0 msgr2=0x7f11d40b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:47.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.529+0000 7f11da200640 1 -- 172.21.15.47:0/4027444778 shutdown_connections 2024-02-19T15:24:47.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.529+0000 7f11da200640 1 -- 172.21.15.47:0/4027444778 wait complete. 2024-02-19T15:24:47.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11da200640 1 Processor -- start 2024-02-19T15:24:47.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11da200640 1 -- start start 2024-02-19T15:24:47.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11da200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11d40cea80 0x7f11d415adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:47.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11da200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11d4145590 0x7f11d4134140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:47.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11da200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d415c390 0x7f11d40b2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:47.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11da200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d415b910 con 0x7f11d40cea80 2024-02-19T15:24:47.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11da200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d40b3230 con 0x7f11d415c390 2024-02-19T15:24:47.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11da200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d40b3510 con 0x7f11d4145590 2024-02-19T15:24:47.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11d3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11d40cea80 0x7f11d415adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:47.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11d3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11d40cea80 0x7f11d415adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34386/0 (socket says 172.21.15.47:34386) 2024-02-19T15:24:47.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11d3400640 1 -- 172.21.15.47:0/2887694539 learned_addr learned my addr 172.21.15.47:0/2887694539 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:47.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11cbe00640 1 --2- 172.21.15.47:0/2887694539 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11d4145590 0x7f11d4134140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:47.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11d3e00640 1 --2- 172.21.15.47:0/2887694539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d415c390 0x7f11d40b2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:47.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11d3400640 1 -- 172.21.15.47:0/2887694539 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11d4145590 msgr2=0x7f11d4134140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:47.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11d3400640 1 --2- 172.21.15.47:0/2887694539 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11d4145590 0x7f11d4134140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:47.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11d3400640 1 -- 172.21.15.47:0/2887694539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d415c390 msgr2=0x7f11d40b2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:47.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11d3400640 1 --2- 172.21.15.47:0/2887694539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d415c390 0x7f11d40b2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:47.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11d3400640 1 -- 172.21.15.47:0/2887694539 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c40094d0 con 0x7f11d40cea80 2024-02-19T15:24:47.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11d3e00640 1 --2- 172.21.15.47:0/2887694539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d415c390 0x7f11d40b2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:47.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11d3400640 1 --2- 172.21.15.47:0/2887694539 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11d40cea80 0x7f11d415adc0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f11c40ad040 tx=0x7f11c401ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:47.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11cb400640 1 -- 172.21.15.47:0/2887694539 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c40b4800 con 0x7f11d40cea80 2024-02-19T15:24:47.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11cb400640 1 -- 172.21.15.47:0/2887694539 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c40b4980 con 0x7f11d40cea80 2024-02-19T15:24:47.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11cb400640 1 -- 172.21.15.47:0/2887694539 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c40b4d30 con 0x7f11d40cea80 2024-02-19T15:24:47.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11da200640 1 -- 172.21.15.47:0/2887694539 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d4138490 con 0x7f11d40cea80 2024-02-19T15:24:47.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11da200640 1 -- 172.21.15.47:0/2887694539 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d40b37f0 con 0x7f11d40cea80 2024-02-19T15:24:47.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.533+0000 7f11da200640 1 -- 172.21.15.47:0/2887694539 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d4075360 con 0x7f11d40cea80 2024-02-19T15:24:47.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.537+0000 7f11cb400640 1 -- 172.21.15.47:0/2887694539 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f11c40b3380 con 0x7f11d40cea80 2024-02-19T15:24:47.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.537+0000 7f11cb400640 1 --2- 172.21.15.47:0/2887694539 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f11b0066e70 0x7f11b0069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:47.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.537+0000 7f11cb400640 1 -- 172.21.15.47:0/2887694539 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11c411f140 con 0x7f11d40cea80 2024-02-19T15:24:47.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.537+0000 7f11cbe00640 1 --2- 172.21.15.47:0/2887694539 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f11b0066e70 0x7f11b0069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:47.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.537+0000 7f11cbe00640 1 --2- 172.21.15.47:0/2887694539 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f11b0066e70 0x7f11b0069330 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f11d41f9d40 tx=0x7f11bc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:47.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.545+0000 7f11cb400640 1 -- 172.21.15.47:0/2887694539 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f11c40ec100 con 0x7f11d40cea80 2024-02-19T15:24:47.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.741+0000 7f11da200640 1 -- 172.21.15.47:0/2887694539 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f11d413d170 con 0x7f11d40cea80 2024-02-19T15:24:47.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.741+0000 7f11cb400640 1 -- 172.21.15.47:0/2887694539 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v33) v1 ==== 53+0+73 (secure 0 0 0) 0x7f11c40ef7c0 con 0x7f11d40cea80 2024-02-19T15:24:47.746 INFO:teuthology.orchestra.run.smithi047.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-02-19T15:24:47.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.745+0000 7f11da200640 1 -- 172.21.15.47:0/2887694539 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f11b0066e70 msgr2=0x7f11b0069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:47.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.745+0000 7f11da200640 1 --2- 172.21.15.47:0/2887694539 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f11b0066e70 0x7f11b0069330 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f11d41f9d40 tx=0x7f11bc008040 comp rx=0 tx=0).stop 2024-02-19T15:24:47.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.745+0000 7f11da200640 1 -- 172.21.15.47:0/2887694539 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11d40cea80 msgr2=0x7f11d415adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:47.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.745+0000 7f11da200640 1 --2- 172.21.15.47:0/2887694539 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11d40cea80 0x7f11d415adc0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f11c40ad040 tx=0x7f11c401ad80 comp rx=0 tx=0).stop 2024-02-19T15:24:47.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.745+0000 7f11da200640 1 -- 172.21.15.47:0/2887694539 shutdown_connections 2024-02-19T15:24:47.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.745+0000 7f11da200640 1 --2- 172.21.15.47:0/2887694539 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f11b0066e70 0x7f11b0069330 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:47.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.745+0000 7f11da200640 1 --2- 172.21.15.47:0/2887694539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d415c390 0x7f11d40b2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:47.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.745+0000 7f11da200640 1 --2- 172.21.15.47:0/2887694539 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11d4145590 0x7f11d4134140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:47.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.745+0000 7f11da200640 1 --2- 172.21.15.47:0/2887694539 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11d40cea80 0x7f11d415adc0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:47.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.745+0000 7f11da200640 1 -- 172.21.15.47:0/2887694539 >> 172.21.15.47:0/2887694539 conn(0x7f11d40b16a0 msgr2=0x7f11d40b0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:47.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.745+0000 7f11da200640 1 -- 172.21.15.47:0/2887694539 shutdown_connections 2024-02-19T15:24:47.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:47.745+0000 7f11da200640 1 -- 172.21.15.47:0/2887694539 wait complete. 2024-02-19T15:24:47.767 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-02-19T15:24:47.767 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T15:24:47.767 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi047.front.sepia.ceph.com 2024-02-19T15:24:47.767 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-19T15:24:47.768 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-19T15:24:47.768 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-02-19T15:24:47.768 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-02-19T15:24:47.768 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T15:24:47.768 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:24:47.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:47.768 DEBUG:teuthology.orchestra.run.smithi047:> ip addr 2024-02-19T15:24:47.777 INFO:teuthology.orchestra.run.smithi047.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T15:24:47.777 INFO:teuthology.orchestra.run.smithi047.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T15:24:47.777 INFO:teuthology.orchestra.run.smithi047.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T15:24:47.777 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:24:47.778 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 ::1/128 scope host 2024-02-19T15:24:47.778 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:24:47.778 INFO:teuthology.orchestra.run.smithi047.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:24:47.778 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:78 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:24:47.778 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f0 2024-02-19T15:24:47.778 INFO:teuthology.orchestra.run.smithi047.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T15:24:47.778 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8e brd ff:ff:ff:ff:ff:ff 2024-02-19T15:24:47.778 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f0 2024-02-19T15:24:47.778 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.21.15.47/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-02-19T15:24:47.778 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft 42327sec preferred_lft 42327sec 2024-02-19T15:24:47.778 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::ec4:7aff:febd:148e/64 scope link 2024-02-19T15:24:47.778 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:24:47.778 INFO:teuthology.orchestra.run.smithi047.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:24:47.778 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:79 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:24:47.778 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f1 2024-02-19T15:24:47.778 INFO:teuthology.orchestra.run.smithi047.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:24:47.779 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8f brd ff:ff:ff:ff:ff:ff 2024-02-19T15:24:47.779 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f1 2024-02-19T15:24:47.779 INFO:teuthology.orchestra.run.smithi047.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-02-19T15:24:47.779 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 02:42:47:cf:b1:62 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:24:47.779 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-02-19T15:24:47.779 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:24:47.779 INFO:teuthology.orchestra.run.smithi047.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-02-19T15:24:47.779 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-02-19T15:24:47.779 INFO:teuthology.orchestra.run.smithi047.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-02-19T15:24:47.779 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:24:47.779 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-02-19T15:24:47.779 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:24:47.779 INFO:teuthology.orchestra.run.smithi047.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:24:47.779 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-19T15:24:47.779 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-02-19T15:24:47.780 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:24:47.780 INFO:teuthology.orchestra.run.smithi047.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:24:47.780 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-02-19T15:24:47.780 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-02-19T15:24:47.780 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:24:47.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:47.780 DEBUG:teuthology.orchestra.run.smithi047:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T15:24:47.836 INFO:teuthology.orchestra.run.smithi047.stdout:1 2024-02-19T15:24:47.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:47.839 DEBUG:teuthology.orchestra.run.smithi047:> route 2024-02-19T15:24:47.891 INFO:teuthology.orchestra.run.smithi047.stdout:Kernel IP routing table 2024-02-19T15:24:47.891 INFO:teuthology.orchestra.run.smithi047.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T15:24:47.892 INFO:teuthology.orchestra.run.smithi047.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-02-19T15:24:47.892 INFO:teuthology.orchestra.run.smithi047.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-02-19T15:24:47.892 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-02-19T15:24:47.892 INFO:teuthology.orchestra.run.smithi047.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T15:24:47.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:47.892 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2024-02-19T15:24:47.892 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2024-02-19T15:24:47.892 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-02-19T15:24:47.892 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-02-19T15:24:47.892 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-02-19T15:24:47.893 DEBUG:teuthology.orchestra.run.smithi047:> ') 2024-02-19T15:24:47.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:47.983 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list 2024-02-19T15:24:47.992 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-02-19T15:24:47.992 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-02-19T15:24:47.993 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:24:47.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:47.993 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set brx.0 up 2024-02-19T15:24:48.054 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:24:48.054 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-19T15:24:48.102 INFO:teuthology.orchestra.run.smithi047.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:24:48.103 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:24:48.103 DEBUG:teuthology.orchestra.run.smithi047:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:24:48.153 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-19T15:24:48.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:48.154 DEBUG:teuthology.orchestra.run.smithi047:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:24:48.248 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:24:48.248 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:24:48.248 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:24:48.264 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:24:48.265 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:24:48.265 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:24:48.265 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:24:48.265 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:24:48.273 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:24:48.273 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:24:48.273 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:24:48.273 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:24:48.273 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:24:48.273 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790" 2024-02-19T15:24:48.274 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790:/ 2024-02-19T15:24:48.274 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e" will pass to kernel 2024-02-19T15:24:48.277 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T15:24:48.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:48.277 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:24:48.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:48.298 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:24:48.349 INFO:teuthology.orchestra.run.smithi047.stdout:client6004 2024-02-19T15:24:48.349 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:24:48.349 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:48.402 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:24:48.402 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:48.454 DEBUG:teuthology.orchestra.run.smithi047:> stdin-killer -- python3 -c ' 2024-02-19T15:24:48.454 DEBUG:teuthology.orchestra.run.smithi047:> import os 2024-02-19T15:24:48.454 DEBUG:teuthology.orchestra.run.smithi047:> import uuid 2024-02-19T15:24:48.454 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:24:48.454 DEBUG:teuthology.orchestra.run.smithi047:> n = 75 2024-02-19T15:24:48.454 DEBUG:teuthology.orchestra.run.smithi047:> create_hard_links = False 2024-02-19T15:24:48.454 DEBUG:teuthology.orchestra.run.smithi047:> if 0 > 0: 2024-02-19T15:24:48.454 DEBUG:teuthology.orchestra.run.smithi047:> create_hard_links = True 2024-02-19T15:24:48.455 DEBUG:teuthology.orchestra.run.smithi047:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-02-19T15:24:48.455 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:24:48.455 DEBUG:teuthology.orchestra.run.smithi047:> dpath = os.path.dirname(path) 2024-02-19T15:24:48.455 DEBUG:teuthology.orchestra.run.smithi047:> fnameprefix = os.path.basename(path) 2024-02-19T15:24:48.455 DEBUG:teuthology.orchestra.run.smithi047:> os.makedirs(dpath, exist_ok=True) 2024-02-19T15:24:48.455 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:24:48.455 DEBUG:teuthology.orchestra.run.smithi047:> try: 2024-02-19T15:24:48.455 DEBUG:teuthology.orchestra.run.smithi047:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-02-19T15:24:48.455 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:24:48.455 DEBUG:teuthology.orchestra.run.smithi047:> for i in range(n): 2024-02-19T15:24:48.455 DEBUG:teuthology.orchestra.run.smithi047:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-02-19T15:24:48.455 DEBUG:teuthology.orchestra.run.smithi047:> with open(fpath, '"'"'w'"'"') as f: 2024-02-19T15:24:48.455 DEBUG:teuthology.orchestra.run.smithi047:> f.write(f"{i}") 2024-02-19T15:24:48.455 DEBUG:teuthology.orchestra.run.smithi047:> if False: 2024-02-19T15:24:48.455 DEBUG:teuthology.orchestra.run.smithi047:> f.flush() 2024-02-19T15:24:48.455 DEBUG:teuthology.orchestra.run.smithi047:> os.fsync(f.fileno()) 2024-02-19T15:24:48.455 DEBUG:teuthology.orchestra.run.smithi047:> if False: 2024-02-19T15:24:48.455 DEBUG:teuthology.orchestra.run.smithi047:> os.unlink(fpath) 2024-02-19T15:24:48.456 DEBUG:teuthology.orchestra.run.smithi047:> if False: 2024-02-19T15:24:48.456 DEBUG:teuthology.orchestra.run.smithi047:> os.fsync(dirfd) 2024-02-19T15:24:48.456 DEBUG:teuthology.orchestra.run.smithi047:> if create_hard_links: 2024-02-19T15:24:48.456 DEBUG:teuthology.orchestra.run.smithi047:> for j in range(0): 2024-02-19T15:24:48.456 DEBUG:teuthology.orchestra.run.smithi047:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-02-19T15:24:48.456 DEBUG:teuthology.orchestra.run.smithi047:> if True: 2024-02-19T15:24:48.456 DEBUG:teuthology.orchestra.run.smithi047:> os.fsync(dirfd) 2024-02-19T15:24:48.456 DEBUG:teuthology.orchestra.run.smithi047:> finally: 2024-02-19T15:24:48.456 DEBUG:teuthology.orchestra.run.smithi047:> os.close(dirfd) 2024-02-19T15:24:48.456 DEBUG:teuthology.orchestra.run.smithi047:> ' 2024-02-19T15:24:48.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-02-19T15:24:48.962 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T15:24:48.963 DEBUG:teuthology.orchestra.run.smithi047:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:24:49.093 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:24:49.133 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T15:24:49.133 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:24:49.134 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:49.142 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:24:49.142 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:49.190 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:24:49.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.365+0000 7f190b200640 1 Processor -- start 2024-02-19T15:24:49.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.365+0000 7f190b200640 1 -- start start 2024-02-19T15:24:49.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.365+0000 7f190b200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f190406d650 0x7f19040d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:49.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.365+0000 7f190b200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f19040cded0 0x7f19040ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:49.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.365+0000 7f190b200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19040cce30 con 0x7f19040ceb40 2024-02-19T15:24:49.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.365+0000 7f190b200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19040cefd0 con 0x7f19040cded0 2024-02-19T15:24:49.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.365+0000 7f190b200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19040cd110 con 0x7f190406d650 2024-02-19T15:24:49.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.365+0000 7f18fbe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f19040cded0 0x7f19040ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:49.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.365+0000 7f18fbe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f19040cded0 0x7f19040ce600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34394/0 (socket says 172.21.15.47:34394) 2024-02-19T15:24:49.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.365+0000 7f18fbe00640 1 -- 172.21.15.47:0/2918420304 learned_addr learned my addr 172.21.15.47:0/2918420304 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:49.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.369+0000 7f1903e00640 1 --2- 172.21.15.47:0/2918420304 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f190406d650 0x7f19040d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:49.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.369+0000 7f18fbe00640 1 -- 172.21.15.47:0/2918420304 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f190406d650 msgr2=0x7f19040d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:49.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.369+0000 7f18fbe00640 1 --2- 172.21.15.47:0/2918420304 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f190406d650 0x7f19040d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.369+0000 7f18fbe00640 1 -- 172.21.15.47:0/2918420304 >> v1:172.21.15.116:6789/0 conn(0x7f19040ceb40 legacy=0x7f1904161d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:49.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.369+0000 7f18fbe00640 1 -- 172.21.15.47:0/2918420304 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19040cd3f0 con 0x7f19040cded0 2024-02-19T15:24:49.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.369+0000 7f1903e00640 1 --2- 172.21.15.47:0/2918420304 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f190406d650 0x7f19040d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:49.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.369+0000 7f18fbe00640 1 --2- 172.21.15.47:0/2918420304 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19040cded0 0x7f19040ce600 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f18ec009870 tx=0x7f18ec01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4708eace18edd59 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:49.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.369+0000 7f1903400640 1 -- 172.21.15.47:0/2918420304 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec0ac620 con 0x7f19040cded0 2024-02-19T15:24:49.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.369+0000 7f1903400640 1 -- 172.21.15.47:0/2918420304 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18ec0ac780 con 0x7f19040cded0 2024-02-19T15:24:49.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.369+0000 7f1903400640 1 -- 172.21.15.47:0/2918420304 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec0acb10 con 0x7f19040cded0 2024-02-19T15:24:49.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f190b200640 1 -- 172.21.15.47:0/2918420304 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19040cded0 msgr2=0x7f19040ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:49.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f190b200640 1 --2- 172.21.15.47:0/2918420304 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19040cded0 0x7f19040ce600 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f18ec009870 tx=0x7f18ec01c4a0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f190b200640 1 -- 172.21.15.47:0/2918420304 shutdown_connections 2024-02-19T15:24:49.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f190b200640 1 --2- 172.21.15.47:0/2918420304 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19040cded0 0x7f19040ce600 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f190b200640 1 --2- 172.21.15.47:0/2918420304 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f190406d650 0x7f19040d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f190b200640 1 -- 172.21.15.47:0/2918420304 >> 172.21.15.47:0/2918420304 conn(0x7f19040d1bf0 msgr2=0x7f19040d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:49.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f190b200640 1 -- 172.21.15.47:0/2918420304 shutdown_connections 2024-02-19T15:24:49.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f190b200640 1 -- 172.21.15.47:0/2918420304 wait complete. 2024-02-19T15:24:49.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f190b200640 1 Processor -- start 2024-02-19T15:24:49.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f190b200640 1 -- start start 2024-02-19T15:24:49.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f190b200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f190406d650 0x7f19040c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:49.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f190b200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19040cded0 0x7f19040b7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:49.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f190b200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19040ceb40 0x7f19040cb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:49.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f190b200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19040c8ae0 con 0x7f190406d650 2024-02-19T15:24:49.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f190b200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19040ca0b0 con 0x7f19040ceb40 2024-02-19T15:24:49.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f190b200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19040c9dd0 con 0x7f19040cded0 2024-02-19T15:24:49.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f1903e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f190406d650 0x7f19040c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:49.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f1903e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f190406d650 0x7f19040c96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34402/0 (socket says 172.21.15.47:34402) 2024-02-19T15:24:49.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f1903e00640 1 -- 172.21.15.47:0/2179184084 learned_addr learned my addr 172.21.15.47:0/2179184084 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:49.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f18fbe00640 1 --2- 172.21.15.47:0/2179184084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19040cded0 0x7f19040b7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:49.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f1909200640 1 --2- 172.21.15.47:0/2179184084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19040ceb40 0x7f19040cb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:49.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f1903e00640 1 -- 172.21.15.47:0/2179184084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19040cded0 msgr2=0x7f19040b7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:49.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f1903e00640 1 --2- 172.21.15.47:0/2179184084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19040cded0 0x7f19040b7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f1903e00640 1 -- 172.21.15.47:0/2179184084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19040ceb40 msgr2=0x7f19040cb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:49.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.373+0000 7f1903e00640 1 --2- 172.21.15.47:0/2179184084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19040ceb40 0x7f19040cb460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.377+0000 7f1903e00640 1 -- 172.21.15.47:0/2179184084 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ec0094d0 con 0x7f190406d650 2024-02-19T15:24:49.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.377+0000 7f1903e00640 1 --2- 172.21.15.47:0/2179184084 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f190406d650 0x7f19040c96d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f18f400d800 tx=0x7f18f400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:49.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.377+0000 7f1900c00640 1 -- 172.21.15.47:0/2179184084 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f40096c0 con 0x7f190406d650 2024-02-19T15:24:49.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.377+0000 7f1900c00640 1 -- 172.21.15.47:0/2179184084 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f4009820 con 0x7f190406d650 2024-02-19T15:24:49.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.377+0000 7f190b200640 1 -- 172.21.15.47:0/2179184084 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19040ca980 con 0x7f190406d650 2024-02-19T15:24:49.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.377+0000 7f1900c00640 1 -- 172.21.15.47:0/2179184084 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f4009bd0 con 0x7f190406d650 2024-02-19T15:24:49.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.377+0000 7f190b200640 1 -- 172.21.15.47:0/2179184084 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19040caf40 con 0x7f190406d650 2024-02-19T15:24:49.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.377+0000 7f190b200640 1 -- 172.21.15.47:0/2179184084 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1904075360 con 0x7f190406d650 2024-02-19T15:24:49.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.377+0000 7f1900c00640 1 -- 172.21.15.47:0/2179184084 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f18f4009d30 con 0x7f190406d650 2024-02-19T15:24:49.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.381+0000 7f1900c00640 1 --2- 172.21.15.47:0/2179184084 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f18e0066e70 0x7f18e0069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:49.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.381+0000 7f1900c00640 1 -- 172.21.15.47:0/2179184084 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18f4084fc0 con 0x7f190406d650 2024-02-19T15:24:49.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.381+0000 7f18fbe00640 1 --2- 172.21.15.47:0/2179184084 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f18e0066e70 0x7f18e0069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:49.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.381+0000 7f18fbe00640 1 --2- 172.21.15.47:0/2179184084 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f18e0066e70 0x7f18e0069330 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f18ec0ae000 tx=0x7f18ec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:49.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.381+0000 7f1900c00640 1 -- 172.21.15.47:0/2179184084 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f18f4051e50 con 0x7f190406d650 2024-02-19T15:24:49.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.557+0000 7f190b200640 1 -- 172.21.15.47:0/2179184084 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19040cf580 con 0x7f190406d650 2024-02-19T15:24:49.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.557+0000 7f1900c00640 1 -- 172.21.15.47:0/2179184084 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16677 (secure 0 0 0) 0x7f18f4055510 con 0x7f190406d650 2024-02-19T15:24:49.561 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:49.561 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":57,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:24:26.785375+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:24:22.931622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-02-19T15:24:24.163465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:24:49.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.561+0000 7f190b200640 1 -- 172.21.15.47:0/2179184084 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f18e0066e70 msgr2=0x7f18e0069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:49.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.561+0000 7f190b200640 1 --2- 172.21.15.47:0/2179184084 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f18e0066e70 0x7f18e0069330 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f18ec0ae000 tx=0x7f18ec008040 comp rx=0 tx=0).stop 2024-02-19T15:24:49.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.561+0000 7f190b200640 1 -- 172.21.15.47:0/2179184084 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f190406d650 msgr2=0x7f19040c96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:49.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.561+0000 7f190b200640 1 --2- 172.21.15.47:0/2179184084 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f190406d650 0x7f19040c96d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f18f400d800 tx=0x7f18f400dcc0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.561+0000 7f190b200640 1 -- 172.21.15.47:0/2179184084 shutdown_connections 2024-02-19T15:24:49.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.561+0000 7f190b200640 1 --2- 172.21.15.47:0/2179184084 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f18e0066e70 0x7f18e0069330 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.561+0000 7f190b200640 1 --2- 172.21.15.47:0/2179184084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19040ceb40 0x7f19040cb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.561+0000 7f190b200640 1 --2- 172.21.15.47:0/2179184084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19040cded0 0x7f19040b7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.561+0000 7f190b200640 1 --2- 172.21.15.47:0/2179184084 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f190406d650 0x7f19040c96d0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.561+0000 7f190b200640 1 -- 172.21.15.47:0/2179184084 >> 172.21.15.47:0/2179184084 conn(0x7f19040d1bf0 msgr2=0x7f19040cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:49.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.561+0000 7f190b200640 1 -- 172.21.15.47:0/2179184084 shutdown_connections 2024-02-19T15:24:49.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.561+0000 7f190b200640 1 -- 172.21.15.47:0/2179184084 wait complete. 2024-02-19T15:24:49.583 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:24:22.931622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:24:24.163465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:24:49.583 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:24:49.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.725+0000 7f69a3000640 1 Processor -- start 2024-02-19T15:24:49.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.725+0000 7f69a3000640 1 -- start start 2024-02-19T15:24:49.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.725+0000 7f69a3000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f699c15e160 0x7f699c0d1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:49.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.725+0000 7f69a3000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f699c06e1e0 0x7f699c15e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:49.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.725+0000 7f69a3000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f699c071a70 con 0x7f699c06f220 2024-02-19T15:24:49.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.725+0000 7f69a3000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f699c0ced30 con 0x7f699c15e160 2024-02-19T15:24:49.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.725+0000 7f69a3000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699c162a00 con 0x7f699c06e1e0 2024-02-19T15:24:49.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.725+0000 7f699b400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f699c06f220 0x7f699c06dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43516/0 (socket says 172.21.15.47:43516) 2024-02-19T15:24:49.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.725+0000 7f699b400640 1 -- 172.21.15.47:0/873149947 learned_addr learned my addr 172.21.15.47:0/873149947 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:49.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.725+0000 7f69a1000640 1 --2- 172.21.15.47:0/873149947 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f699c06e1e0 0x7f699c15e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:49.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.725+0000 7f699be00640 1 --2- 172.21.15.47:0/873149947 >> v2:172.21.15.116:3300/0 conn(0x7f699c15e160 0x7f699c0d1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:49.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.725+0000 7f699aa00640 1 -- 172.21.15.47:0/873149947 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040224054 0 0) 0x7f699c071a70 con 0x7f699c06f220 2024-02-19T15:24:49.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.725+0000 7f699aa00640 1 -- 172.21.15.47:0/873149947 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6980003530 con 0x7f699c06f220 2024-02-19T15:24:49.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.725+0000 7f69a1000640 1 -- 172.21.15.47:0/873149947 >> v2:172.21.15.116:3300/0 conn(0x7f699c15e160 msgr2=0x7f699c0d1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:49.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.725+0000 7f69a1000640 1 --2- 172.21.15.47:0/873149947 >> v2:172.21.15.116:3300/0 conn(0x7f699c15e160 0x7f699c0d1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.725+0000 7f69a1000640 1 -- 172.21.15.47:0/873149947 >> v1:172.21.15.47:6789/0 conn(0x7f699c06f220 legacy=0x7f699c06dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:49.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.725+0000 7f69a1000640 1 -- 172.21.15.47:0/873149947 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699c06f6b0 con 0x7f699c06e1e0 2024-02-19T15:24:49.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.729+0000 7f69a1000640 1 --2- 172.21.15.47:0/873149947 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f699c06e1e0 0x7f699c15e540 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6994009d90 tx=0x7f699401c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=528b91d247f8399e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:49.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.729+0000 7f699aa00640 1 -- 172.21.15.47:0/873149947 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69940a5a00 con 0x7f699c06e1e0 2024-02-19T15:24:49.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.729+0000 7f699aa00640 1 -- 172.21.15.47:0/873149947 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69940a5b60 con 0x7f699c06e1e0 2024-02-19T15:24:49.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.729+0000 7f699aa00640 1 -- 172.21.15.47:0/873149947 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69940ad440 con 0x7f699c06e1e0 2024-02-19T15:24:49.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.729+0000 7f69a3000640 1 -- 172.21.15.47:0/873149947 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f699c06e1e0 msgr2=0x7f699c15e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:49.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.729+0000 7f69a3000640 1 --2- 172.21.15.47:0/873149947 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f699c06e1e0 0x7f699c15e540 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6994009d90 tx=0x7f699401c940 comp rx=0 tx=0).stop 2024-02-19T15:24:49.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.729+0000 7f69a3000640 1 -- 172.21.15.47:0/873149947 shutdown_connections 2024-02-19T15:24:49.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.729+0000 7f69a3000640 1 --2- 172.21.15.47:0/873149947 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f699c06e1e0 0x7f699c15e540 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.729+0000 7f69a3000640 1 --2- 172.21.15.47:0/873149947 >> v2:172.21.15.116:3300/0 conn(0x7f699c15e160 0x7f699c0d1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.729+0000 7f69a3000640 1 -- 172.21.15.47:0/873149947 >> 172.21.15.47:0/873149947 conn(0x7f699c0d0d30 msgr2=0x7f699c0d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:49.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.729+0000 7f69a3000640 1 -- 172.21.15.47:0/873149947 shutdown_connections 2024-02-19T15:24:49.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.729+0000 7f69a3000640 1 -- 172.21.15.47:0/873149947 wait complete. 2024-02-19T15:24:49.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f69a3000640 1 Processor -- start 2024-02-19T15:24:49.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f69a3000640 1 -- start start 2024-02-19T15:24:49.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f69a3000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f699c06e1e0 0x7f699c0b6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:49.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f69a3000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f699c06f220 0x7f699c06e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:49.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f69a3000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f699c15e160 0x7f699c155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:49.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f69a3000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c15af50 con 0x7f699c06e1e0 2024-02-19T15:24:49.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f69a3000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c15a3d0 con 0x7f699c06f220 2024-02-19T15:24:49.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f69a3000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699c15a6b0 con 0x7f699c15e160 2024-02-19T15:24:49.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f699be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f699c06e1e0 0x7f699c0b6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:49.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f699be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f699c06e1e0 0x7f699c0b6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34404/0 (socket says 172.21.15.47:34404) 2024-02-19T15:24:49.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f699be00640 1 -- 172.21.15.47:0/1585431196 learned_addr learned my addr 172.21.15.47:0/1585431196 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:49.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f699b400640 1 --2- 172.21.15.47:0/1585431196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f699c06f220 0x7f699c06e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:49.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f69a1000640 1 --2- 172.21.15.47:0/1585431196 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f699c15e160 0x7f699c155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:49.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f699be00640 1 -- 172.21.15.47:0/1585431196 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f699c15e160 msgr2=0x7f699c155910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:49.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f699be00640 1 --2- 172.21.15.47:0/1585431196 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f699c15e160 0x7f699c155910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f699be00640 1 -- 172.21.15.47:0/1585431196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f699c06f220 msgr2=0x7f699c06e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:49.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f699be00640 1 --2- 172.21.15.47:0/1585431196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f699c06f220 0x7f699c06e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f699be00640 1 -- 172.21.15.47:0/1585431196 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6994009a70 con 0x7f699c06e1e0 2024-02-19T15:24:49.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f699be00640 1 --2- 172.21.15.47:0/1585431196 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f699c06e1e0 0x7f699c0b6440 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f698c00e930 tx=0x7f698c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:49.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f6993e00640 1 -- 172.21.15.47:0/1585431196 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c009690 con 0x7f699c06e1e0 2024-02-19T15:24:49.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f6993e00640 1 -- 172.21.15.47:0/1585431196 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698c00ca00 con 0x7f699c06e1e0 2024-02-19T15:24:49.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f6993e00640 1 -- 172.21.15.47:0/1585431196 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c00cd70 con 0x7f699c06e1e0 2024-02-19T15:24:49.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f69a3000640 1 -- 172.21.15.47:0/1585431196 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f699c15a990 con 0x7f699c06e1e0 2024-02-19T15:24:49.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f69a3000640 1 -- 172.21.15.47:0/1585431196 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f699c0b6980 con 0x7f699c06e1e0 2024-02-19T15:24:49.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.733+0000 7f69a3000640 1 -- 172.21.15.47:0/1585431196 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f699c075360 con 0x7f699c06e1e0 2024-02-19T15:24:49.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.737+0000 7f6993e00640 1 -- 172.21.15.47:0/1585431196 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f698c013470 con 0x7f699c06e1e0 2024-02-19T15:24:49.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.737+0000 7f6993e00640 1 --2- 172.21.15.47:0/1585431196 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f697c066e40 0x7f697c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:49.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.737+0000 7f6993e00640 1 -- 172.21.15.47:0/1585431196 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f698c084c70 con 0x7f699c06e1e0 2024-02-19T15:24:49.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.737+0000 7f699b400640 1 --2- 172.21.15.47:0/1585431196 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f697c066e40 0x7f697c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:49.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.737+0000 7f699b400640 1 --2- 172.21.15.47:0/1585431196 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f697c066e40 0x7f697c069300 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6988007fd0 tx=0x7f6988009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:49.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.741+0000 7f6993e00640 1 -- 172.21.15.47:0/1585431196 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f698c051b30 con 0x7f699c06e1e0 2024-02-19T15:24:49.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.909+0000 7f69a3000640 1 -- 172.21.15.47:0/1585431196 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f699c0b3a70 con 0x7f699c06e1e0 2024-02-19T15:24:49.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.913+0000 7f6993e00640 1 -- 172.21.15.47:0/1585431196 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7f698c0170a0 con 0x7f699c06e1e0 2024-02-19T15:24:49.915 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:24:49.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.913+0000 7f69a3000640 1 -- 172.21.15.47:0/1585431196 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f697c066e40 msgr2=0x7f697c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:49.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.913+0000 7f69a3000640 1 --2- 172.21.15.47:0/1585431196 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f697c066e40 0x7f697c069300 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6988007fd0 tx=0x7f6988009040 comp rx=0 tx=0).stop 2024-02-19T15:24:49.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.913+0000 7f69a3000640 1 -- 172.21.15.47:0/1585431196 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f699c06e1e0 msgr2=0x7f699c0b6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:49.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.913+0000 7f69a3000640 1 --2- 172.21.15.47:0/1585431196 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f699c06e1e0 0x7f699c0b6440 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f698c00e930 tx=0x7f698c00edf0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.913+0000 7f69a3000640 1 -- 172.21.15.47:0/1585431196 shutdown_connections 2024-02-19T15:24:49.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.913+0000 7f69a3000640 1 --2- 172.21.15.47:0/1585431196 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f697c066e40 0x7f697c069300 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.913+0000 7f69a3000640 1 --2- 172.21.15.47:0/1585431196 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f699c15e160 0x7f699c155910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.913+0000 7f69a3000640 1 --2- 172.21.15.47:0/1585431196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f699c06f220 0x7f699c06e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.913+0000 7f69a3000640 1 --2- 172.21.15.47:0/1585431196 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f699c06e1e0 0x7f699c0b6440 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:49.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.913+0000 7f69a3000640 1 -- 172.21.15.47:0/1585431196 >> 172.21.15.47:0/1585431196 conn(0x7f699c0d0d30 msgr2=0x7f699c15f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:49.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.913+0000 7f69a3000640 1 -- 172.21.15.47:0/1585431196 shutdown_connections 2024-02-19T15:24:49.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:49.913+0000 7f69a3000640 1 -- 172.21.15.47:0/1585431196 wait complete. 2024-02-19T15:24:49.936 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:24:50.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.077+0000 7f96fe000640 1 Processor -- start 2024-02-19T15:24:50.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.077+0000 7f96fe000640 1 -- start start 2024-02-19T15:24:50.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.077+0000 7f96fe000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96f80c7e80 0x7f96f80d0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:50.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.077+0000 7f96fe000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f80cd750 con 0x7f96f80c8260 2024-02-19T15:24:50.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.077+0000 7f96fe000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f80ceee0 con 0x7f96f80cfbf0 2024-02-19T15:24:50.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.077+0000 7f96fe000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96f80cdc60 con 0x7f96f80c7e80 2024-02-19T15:24:50.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.077+0000 7f96eea00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f96f80c8260 0x7f96f80c5680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43532/0 (socket says 172.21.15.47:43532) 2024-02-19T15:24:50.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.077+0000 7f96eea00640 1 -- 172.21.15.47:0/924442565 learned_addr learned my addr 172.21.15.47:0/924442565 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:50.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.077+0000 7f96f7400640 1 --2- 172.21.15.47:0/924442565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96f80c7e80 0x7f96f80d0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:50.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.077+0000 7f96f6a00640 1 -- 172.21.15.47:0/924442565 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310108254 0 0) 0x7f96f80cd750 con 0x7f96f80c8260 2024-02-19T15:24:50.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.077+0000 7f96f6a00640 1 -- 172.21.15.47:0/924442565 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96dc003530 con 0x7f96f80c8260 2024-02-19T15:24:50.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.077+0000 7f96f6a00640 1 -- 172.21.15.47:0/924442565 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908853202 0 0) 0x7f96f80ceee0 con 0x7f96f80cfbf0 2024-02-19T15:24:50.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.077+0000 7f96f6a00640 1 -- 172.21.15.47:0/924442565 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f80cd750 con 0x7f96f80cfbf0 2024-02-19T15:24:50.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.077+0000 7f96f7400640 1 -- 172.21.15.47:0/924442565 >> v1:172.21.15.116:6789/0 conn(0x7f96f80cfbf0 legacy=0x7f96f80c5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:50.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.077+0000 7f96f7400640 1 -- 172.21.15.47:0/924442565 >> v1:172.21.15.47:6789/0 conn(0x7f96f80c8260 legacy=0x7f96f80c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:50.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.077+0000 7f96f7400640 1 -- 172.21.15.47:0/924442565 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f80d2a40 con 0x7f96f80c7e80 2024-02-19T15:24:50.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.081+0000 7f96f7400640 1 --2- 172.21.15.47:0/924442565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96f80c7e80 0x7f96f80d0110 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f96e80097f0 tx=0x7f96e801c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=442bac4c0dcbfe7e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:50.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.081+0000 7f96f6a00640 1 -- 172.21.15.47:0/924442565 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e80aa640 con 0x7f96f80c7e80 2024-02-19T15:24:50.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.081+0000 7f96f6a00640 1 -- 172.21.15.47:0/924442565 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e80aa7a0 con 0x7f96f80c7e80 2024-02-19T15:24:50.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.081+0000 7f96f6a00640 1 -- 172.21.15.47:0/924442565 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e80aab30 con 0x7f96f80c7e80 2024-02-19T15:24:50.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.081+0000 7f96fe000640 1 -- 172.21.15.47:0/924442565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96f80c7e80 msgr2=0x7f96f80d0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:50.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.081+0000 7f96fe000640 1 --2- 172.21.15.47:0/924442565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96f80c7e80 0x7f96f80d0110 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f96e80097f0 tx=0x7f96e801c4e0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.081+0000 7f96fe000640 1 -- 172.21.15.47:0/924442565 shutdown_connections 2024-02-19T15:24:50.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.081+0000 7f96fe000640 1 --2- 172.21.15.47:0/924442565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96f80c7e80 0x7f96f80d0110 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.081+0000 7f96fe000640 1 -- 172.21.15.47:0/924442565 >> 172.21.15.47:0/924442565 conn(0x7f96f80d1b30 msgr2=0x7f96f80d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:50.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.081+0000 7f96fe000640 1 -- 172.21.15.47:0/924442565 shutdown_connections 2024-02-19T15:24:50.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.081+0000 7f96fe000640 1 -- 172.21.15.47:0/924442565 wait complete. 2024-02-19T15:24:50.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96fe000640 1 Processor -- start 2024-02-19T15:24:50.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96fe000640 1 -- start start 2024-02-19T15:24:50.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96fe000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96f80c7e80 0x7f96f8156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:50.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96fe000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96f80c8260 0x7f96f8153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:50.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96fe000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96f80cfbf0 0x7f96f814db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:50.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96fe000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f8153d10 con 0x7f96f80c8260 2024-02-19T15:24:50.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96fe000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f8153ff0 con 0x7f96f80cfbf0 2024-02-19T15:24:50.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96fe000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96f81542d0 con 0x7f96f80c7e80 2024-02-19T15:24:50.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96eea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96f80c8260 0x7f96f8153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:50.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96eea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96f80c8260 0x7f96f8153310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34406/0 (socket says 172.21.15.47:34406) 2024-02-19T15:24:50.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96eea00640 1 -- 172.21.15.47:0/3530454559 learned_addr learned my addr 172.21.15.47:0/3530454559 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:50.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96f7400640 1 --2- 172.21.15.47:0/3530454559 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96f80c7e80 0x7f96f8156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:50.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96f7e00640 1 --2- 172.21.15.47:0/3530454559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96f80cfbf0 0x7f96f814db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:50.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96eea00640 1 -- 172.21.15.47:0/3530454559 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96f80c7e80 msgr2=0x7f96f8156c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:50.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96eea00640 1 --2- 172.21.15.47:0/3530454559 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96f80c7e80 0x7f96f8156c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96eea00640 1 -- 172.21.15.47:0/3530454559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96f80cfbf0 msgr2=0x7f96f814db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:50.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96eea00640 1 --2- 172.21.15.47:0/3530454559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96f80cfbf0 0x7f96f814db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96eea00640 1 -- 172.21.15.47:0/3530454559 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e8009420 con 0x7f96f80c8260 2024-02-19T15:24:50.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96eea00640 1 --2- 172.21.15.47:0/3530454559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96f80c8260 0x7f96f8153310 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f96e0005890 tx=0x7f96e000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:50.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96efe00640 1 -- 172.21.15.47:0/3530454559 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e0005cf0 con 0x7f96f80c8260 2024-02-19T15:24:50.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96efe00640 1 -- 172.21.15.47:0/3530454559 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e0005e50 con 0x7f96f80c8260 2024-02-19T15:24:50.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96fe000640 1 -- 172.21.15.47:0/3530454559 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96f81545b0 con 0x7f96f80c8260 2024-02-19T15:24:50.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96efe00640 1 -- 172.21.15.47:0/3530454559 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e00072b0 con 0x7f96f80c8260 2024-02-19T15:24:50.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.085+0000 7f96fe000640 1 -- 172.21.15.47:0/3530454559 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96f8157190 con 0x7f96f80c8260 2024-02-19T15:24:50.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.089+0000 7f96fe000640 1 -- 172.21.15.47:0/3530454559 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f80d1100 con 0x7f96f80c8260 2024-02-19T15:24:50.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.089+0000 7f96efe00640 1 -- 172.21.15.47:0/3530454559 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f96e0007410 con 0x7f96f80c8260 2024-02-19T15:24:50.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.089+0000 7f96efe00640 1 --2- 172.21.15.47:0/3530454559 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f96d4066e40 0x7f96d4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:50.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.089+0000 7f96efe00640 1 -- 172.21.15.47:0/3530454559 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f96e0083ff0 con 0x7f96f80c8260 2024-02-19T15:24:50.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.089+0000 7f96f7400640 1 --2- 172.21.15.47:0/3530454559 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f96d4066e40 0x7f96d4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:50.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.093+0000 7f96f7400640 1 --2- 172.21.15.47:0/3530454559 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f96d4066e40 0x7f96d4069300 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f96e80ae000 tx=0x7f96e80a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:50.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.093+0000 7f96efe00640 1 -- 172.21.15.47:0/3530454559 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f96e0050e50 con 0x7f96f80c8260 2024-02-19T15:24:50.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.257+0000 7f96fe000640 1 -- 172.21.15.47:0/3530454559 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f96f8154a60 con 0x7f96f80c8260 2024-02-19T15:24:50.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.257+0000 7f96efe00640 1 -- 172.21.15.47:0/3530454559 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) v1 ==== 104+0+11 (secure 0 0 0) 0x7f96f8154a60 con 0x7f96f80c8260 2024-02-19T15:24:50.262 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:24:50.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.261+0000 7f96fe000640 1 -- 172.21.15.47:0/3530454559 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f96d4066e40 msgr2=0x7f96d4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:50.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.261+0000 7f96fe000640 1 --2- 172.21.15.47:0/3530454559 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f96d4066e40 0x7f96d4069300 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f96e80ae000 tx=0x7f96e80a7000 comp rx=0 tx=0).stop 2024-02-19T15:24:50.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.261+0000 7f96fe000640 1 -- 172.21.15.47:0/3530454559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96f80c8260 msgr2=0x7f96f8153310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:50.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.261+0000 7f96fe000640 1 --2- 172.21.15.47:0/3530454559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96f80c8260 0x7f96f8153310 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f96e0005890 tx=0x7f96e000c960 comp rx=0 tx=0).stop 2024-02-19T15:24:50.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.261+0000 7f96fe000640 1 -- 172.21.15.47:0/3530454559 shutdown_connections 2024-02-19T15:24:50.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.261+0000 7f96fe000640 1 --2- 172.21.15.47:0/3530454559 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f96d4066e40 0x7f96d4069300 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.261+0000 7f96fe000640 1 --2- 172.21.15.47:0/3530454559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96f80cfbf0 0x7f96f814db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.261+0000 7f96fe000640 1 --2- 172.21.15.47:0/3530454559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96f80c8260 0x7f96f8153310 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.261+0000 7f96fe000640 1 --2- 172.21.15.47:0/3530454559 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96f80c7e80 0x7f96f8156c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.261+0000 7f96fe000640 1 -- 172.21.15.47:0/3530454559 >> 172.21.15.47:0/3530454559 conn(0x7f96f80d1b30 msgr2=0x7f96f815f130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:50.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.261+0000 7f96fe000640 1 -- 172.21.15.47:0/3530454559 shutdown_connections 2024-02-19T15:24:50.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.261+0000 7f96fe000640 1 -- 172.21.15.47:0/3530454559 wait complete. 2024-02-19T15:24:50.283 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:24:50.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.433+0000 7f48b3200640 1 Processor -- start 2024-02-19T15:24:50.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.433+0000 7f48b3200640 1 -- start start 2024-02-19T15:24:50.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.433+0000 7f48b3200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f48ac0d0820 0x7f48ac0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:50.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.433+0000 7f48b3200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f48ac0c8260 0x7f48ac0ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:50.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.433+0000 7f48b3200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ac0ceee0 con 0x7f48ac0c7e80 2024-02-19T15:24:50.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.433+0000 7f48b3200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48ac0cdc60 con 0x7f48ac0d0820 2024-02-19T15:24:50.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.433+0000 7f48b3200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48ac0d2a40 con 0x7f48ac0c8260 2024-02-19T15:24:50.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.437+0000 7f48a3400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f48ac0d0820 0x7f48ac0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:50.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.437+0000 7f48a3400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f48ac0d0820 0x7f48ac0cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34408/0 (socket says 172.21.15.47:34408) 2024-02-19T15:24:50.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.437+0000 7f48a3400640 1 -- 172.21.15.47:0/888740024 learned_addr learned my addr 172.21.15.47:0/888740024 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:50.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.437+0000 7f48abe00640 1 --1- 172.21.15.47:0/888740024 >> v1:172.21.15.47:6789/0 conn(0x7f48ac0c7e80 0x7f48ac0d0110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43544/0 (socket says 172.21.15.47:43544) 2024-02-19T15:24:50.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.437+0000 7f48b1200640 1 --2- 172.21.15.47:0/888740024 >> v2:172.21.15.116:3300/0 conn(0x7f48ac0c8260 0x7f48ac0ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:50.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.437+0000 7f48a3400640 1 -- 172.21.15.47:0/888740024 >> v2:172.21.15.116:3300/0 conn(0x7f48ac0c8260 msgr2=0x7f48ac0ccc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:50.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.437+0000 7f48a3400640 1 --2- 172.21.15.47:0/888740024 >> v2:172.21.15.116:3300/0 conn(0x7f48ac0c8260 0x7f48ac0ccc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.437+0000 7f48a3400640 1 -- 172.21.15.47:0/888740024 >> v1:172.21.15.47:6789/0 conn(0x7f48ac0c7e80 legacy=0x7f48ac0d0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:50.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.437+0000 7f48a3400640 1 -- 172.21.15.47:0/888740024 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ac0cd980 con 0x7f48ac0d0820 2024-02-19T15:24:50.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.437+0000 7f48ab400640 1 -- 172.21.15.47:0/888740024 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593522546 0 0) 0x7f48ac0ceee0 con 0x7f48ac0c7e80 2024-02-19T15:24:50.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.441+0000 7f48a3400640 1 --2- 172.21.15.47:0/888740024 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48ac0d0820 0x7f48ac0cc740 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f48ac0b7be0 tx=0x7f489401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28bb349f120b400a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:50.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.441+0000 7f48ab400640 1 -- 172.21.15.47:0/888740024 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48940ab650 con 0x7f48ac0d0820 2024-02-19T15:24:50.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.441+0000 7f48ab400640 1 -- 172.21.15.47:0/888740024 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48940ab7b0 con 0x7f48ac0d0820 2024-02-19T15:24:50.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.441+0000 7f48ab400640 1 -- 172.21.15.47:0/888740024 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48940abb40 con 0x7f48ac0d0820 2024-02-19T15:24:50.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.441+0000 7f48b3200640 1 -- 172.21.15.47:0/888740024 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48ac0d0820 msgr2=0x7f48ac0cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:50.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.441+0000 7f48b3200640 1 --2- 172.21.15.47:0/888740024 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48ac0d0820 0x7f48ac0cc740 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f48ac0b7be0 tx=0x7f489401c4f0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.441+0000 7f48b3200640 1 -- 172.21.15.47:0/888740024 shutdown_connections 2024-02-19T15:24:50.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.441+0000 7f48b3200640 1 --2- 172.21.15.47:0/888740024 >> v2:172.21.15.116:3300/0 conn(0x7f48ac0c8260 0x7f48ac0ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.441+0000 7f48b3200640 1 --2- 172.21.15.47:0/888740024 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48ac0d0820 0x7f48ac0cc740 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.441+0000 7f48b3200640 1 -- 172.21.15.47:0/888740024 >> 172.21.15.47:0/888740024 conn(0x7f48ac0d1b30 msgr2=0x7f48ac0d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:50.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.441+0000 7f48b3200640 1 -- 172.21.15.47:0/888740024 shutdown_connections 2024-02-19T15:24:50.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.441+0000 7f48b3200640 1 -- 172.21.15.47:0/888740024 wait complete. 2024-02-19T15:24:50.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.445+0000 7f48b3200640 1 Processor -- start 2024-02-19T15:24:50.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.445+0000 7f48b3200640 1 -- start start 2024-02-19T15:24:50.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.445+0000 7f48b3200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48ac0c7e80 0x7f48ac148280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:50.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.445+0000 7f48b3200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48ac0c8260 0x7f48ac156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:50.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.445+0000 7f48b3200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48ac0d0820 0x7f48ac156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:50.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.445+0000 7f48b3200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ac145950 con 0x7f48ac0d0820 2024-02-19T15:24:50.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.445+0000 7f48b3200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ac145ba0 con 0x7f48ac0c8260 2024-02-19T15:24:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.445+0000 7f48b3200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48ac145e80 con 0x7f48ac0c7e80 2024-02-19T15:24:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.445+0000 7f48b1200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48ac0d0820 0x7f48ac156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.445+0000 7f48b1200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48ac0d0820 0x7f48ac156ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34416/0 (socket says 172.21.15.47:34416) 2024-02-19T15:24:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.445+0000 7f48b1200640 1 -- 172.21.15.47:0/3588296827 learned_addr learned my addr 172.21.15.47:0/3588296827 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.445+0000 7f48abe00640 1 --2- 172.21.15.47:0/3588296827 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48ac0c7e80 0x7f48ac148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.445+0000 7f48b1200640 1 -- 172.21.15.47:0/3588296827 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48ac0c7e80 msgr2=0x7f48ac148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.445+0000 7f48b1200640 1 --2- 172.21.15.47:0/3588296827 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48ac0c7e80 0x7f48ac148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.445+0000 7f48a3400640 1 --2- 172.21.15.47:0/3588296827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48ac0c8260 0x7f48ac156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.445+0000 7f48b1200640 1 -- 172.21.15.47:0/3588296827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48ac0c8260 msgr2=0x7f48ac156570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.445+0000 7f48b1200640 1 --2- 172.21.15.47:0/3588296827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48ac0c8260 0x7f48ac156570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.445+0000 7f48b1200640 1 -- 172.21.15.47:0/3588296827 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48940094d0 con 0x7f48ac0d0820 2024-02-19T15:24:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.445+0000 7f48abe00640 1 --2- 172.21.15.47:0/3588296827 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48ac0c7e80 0x7f48ac148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.445+0000 7f48a3400640 1 --2- 172.21.15.47:0/3588296827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48ac0c8260 0x7f48ac156570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.449+0000 7f48b1200640 1 --2- 172.21.15.47:0/3588296827 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48ac0d0820 0x7f48ac156ab0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f48a400efc0 tx=0x7f48a400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.449+0000 7f48a8c00640 1 -- 172.21.15.47:0/3588296827 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a4014480 con 0x7f48ac0d0820 2024-02-19T15:24:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.449+0000 7f48a8c00640 1 -- 172.21.15.47:0/3588296827 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a400f040 con 0x7f48ac0d0820 2024-02-19T15:24:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.449+0000 7f48a8c00640 1 -- 172.21.15.47:0/3588296827 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a4014820 con 0x7f48ac0d0820 2024-02-19T15:24:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.449+0000 7f48b3200640 1 -- 172.21.15.47:0/3588296827 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48ac146160 con 0x7f48ac0d0820 2024-02-19T15:24:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.449+0000 7f48b3200640 1 -- 172.21.15.47:0/3588296827 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48ac157c80 con 0x7f48ac0d0820 2024-02-19T15:24:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.449+0000 7f48b3200640 1 -- 172.21.15.47:0/3588296827 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48ac0d1180 con 0x7f48ac0d0820 2024-02-19T15:24:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.449+0000 7f48a8c00640 1 -- 172.21.15.47:0/3588296827 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f48a4014980 con 0x7f48ac0d0820 2024-02-19T15:24:50.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.453+0000 7f48a8c00640 1 --2- 172.21.15.47:0/3588296827 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4888066e70 0x7f4888069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:50.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.453+0000 7f48a8c00640 1 -- 172.21.15.47:0/3588296827 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f48a4085d50 con 0x7f48ac0d0820 2024-02-19T15:24:50.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.453+0000 7f48abe00640 1 --2- 172.21.15.47:0/3588296827 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4888066e70 0x7f4888069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:50.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.453+0000 7f48abe00640 1 --2- 172.21.15.47:0/3588296827 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4888066e70 0x7f4888069330 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f489c000f70 tx=0x7f489c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:50.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.453+0000 7f48a8c00640 1 -- 172.21.15.47:0/3588296827 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f48a4052bb0 con 0x7f48ac0d0820 2024-02-19T15:24:50.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.613+0000 7f48b3200640 1 -- 172.21.15.47:0/3588296827 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f48ac075360 con 0x7f48ac0d0820 2024-02-19T15:24:50.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.613+0000 7f48a8c00640 1 -- 172.21.15.47:0/3588296827 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) v1 ==== 100+0+11 (secure 0 0 0) 0x7f48a4018020 con 0x7f48ac0d0820 2024-02-19T15:24:50.618 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:24:50.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.617+0000 7f48b3200640 1 -- 172.21.15.47:0/3588296827 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4888066e70 msgr2=0x7f4888069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:50.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.617+0000 7f48b3200640 1 --2- 172.21.15.47:0/3588296827 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4888066e70 0x7f4888069330 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f489c000f70 tx=0x7f489c009040 comp rx=0 tx=0).stop 2024-02-19T15:24:50.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.617+0000 7f48b3200640 1 -- 172.21.15.47:0/3588296827 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48ac0d0820 msgr2=0x7f48ac156ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:50.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.617+0000 7f48b3200640 1 --2- 172.21.15.47:0/3588296827 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48ac0d0820 0x7f48ac156ab0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f48a400efc0 tx=0x7f48a400c490 comp rx=0 tx=0).stop 2024-02-19T15:24:50.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.617+0000 7f48b3200640 1 -- 172.21.15.47:0/3588296827 shutdown_connections 2024-02-19T15:24:50.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.617+0000 7f48b3200640 1 --2- 172.21.15.47:0/3588296827 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4888066e70 0x7f4888069330 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.617+0000 7f48b3200640 1 --2- 172.21.15.47:0/3588296827 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48ac0d0820 0x7f48ac156ab0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.617+0000 7f48b3200640 1 --2- 172.21.15.47:0/3588296827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48ac0c8260 0x7f48ac156570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.617+0000 7f48b3200640 1 --2- 172.21.15.47:0/3588296827 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48ac0c7e80 0x7f48ac148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.617+0000 7f48b3200640 1 -- 172.21.15.47:0/3588296827 >> 172.21.15.47:0/3588296827 conn(0x7f48ac0d1b30 msgr2=0x7f48ac0cbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:50.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.617+0000 7f48b3200640 1 -- 172.21.15.47:0/3588296827 shutdown_connections 2024-02-19T15:24:50.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.617+0000 7f48b3200640 1 -- 172.21.15.47:0/3588296827 wait complete. 2024-02-19T15:24:50.639 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-19T15:24:50.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.789+0000 7fc744200640 1 Processor -- start 2024-02-19T15:24:50.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.789+0000 7fc744200640 1 -- start start 2024-02-19T15:24:50.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.789+0000 7fc744200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc73c1614e0 0x7fc73c15f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:50.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.789+0000 7fc744200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc73c0b2500 0x7fc73c06ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:50.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.789+0000 7fc744200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73c06b5d0 con 0x7fc73c15eee0 2024-02-19T15:24:50.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.789+0000 7fc744200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc73c0b1c00 con 0x7fc73c1614e0 2024-02-19T15:24:50.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.789+0000 7fc744200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c0b1640 con 0x7fc73c0b2500 2024-02-19T15:24:50.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.789+0000 7fc740e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc73c15eee0 0x7fc73c0b1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43548/0 (socket says 172.21.15.47:43548) 2024-02-19T15:24:50.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.789+0000 7fc740e00640 1 -- 172.21.15.47:0/1735706761 learned_addr learned my addr 172.21.15.47:0/1735706761 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:50.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.789+0000 7fc742200640 1 --2- 172.21.15.47:0/1735706761 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc73c0b2500 0x7fc73c06ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:50.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.789+0000 7fc741800640 1 --2- 172.21.15.47:0/1735706761 >> v2:172.21.15.116:3300/0 conn(0x7fc73c1614e0 0x7fc73c15f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:50.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.789+0000 7fc742200640 1 -- 172.21.15.47:0/1735706761 >> v2:172.21.15.116:3300/0 conn(0x7fc73c1614e0 msgr2=0x7fc73c15f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:50.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.789+0000 7fc742200640 1 --2- 172.21.15.47:0/1735706761 >> v2:172.21.15.116:3300/0 conn(0x7fc73c1614e0 0x7fc73c15f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.789+0000 7fc742200640 1 -- 172.21.15.47:0/1735706761 >> v1:172.21.15.47:6789/0 conn(0x7fc73c15eee0 legacy=0x7fc73c0b1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:50.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.789+0000 7fc742200640 1 -- 172.21.15.47:0/1735706761 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73c0b1920 con 0x7fc73c0b2500 2024-02-19T15:24:50.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.793+0000 7fc742200640 1 --2- 172.21.15.47:0/1735706761 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc73c0b2500 0x7fc73c06ba60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc734009d90 tx=0x7fc73401c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d956465ff8f977b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:50.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.793+0000 7fc73be00640 1 -- 172.21.15.47:0/1735706761 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7340a5960 con 0x7fc73c0b2500 2024-02-19T15:24:50.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.793+0000 7fc73be00640 1 -- 172.21.15.47:0/1735706761 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7340a5ac0 con 0x7fc73c0b2500 2024-02-19T15:24:50.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.793+0000 7fc73be00640 1 -- 172.21.15.47:0/1735706761 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7340a5e50 con 0x7fc73c0b2500 2024-02-19T15:24:50.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.793+0000 7fc744200640 1 -- 172.21.15.47:0/1735706761 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc73c0b2500 msgr2=0x7fc73c06ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:50.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.793+0000 7fc744200640 1 --2- 172.21.15.47:0/1735706761 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc73c0b2500 0x7fc73c06ba60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc734009d90 tx=0x7fc73401c970 comp rx=0 tx=0).stop 2024-02-19T15:24:50.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.793+0000 7fc744200640 1 -- 172.21.15.47:0/1735706761 shutdown_connections 2024-02-19T15:24:50.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.793+0000 7fc744200640 1 --2- 172.21.15.47:0/1735706761 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc73c0b2500 0x7fc73c06ba60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.793+0000 7fc744200640 1 --2- 172.21.15.47:0/1735706761 >> v2:172.21.15.116:3300/0 conn(0x7fc73c1614e0 0x7fc73c15f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.793+0000 7fc744200640 1 -- 172.21.15.47:0/1735706761 >> 172.21.15.47:0/1735706761 conn(0x7fc73c0b4fc0 msgr2=0x7fc73c0b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:50.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.793+0000 7fc744200640 1 -- 172.21.15.47:0/1735706761 shutdown_connections 2024-02-19T15:24:50.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.793+0000 7fc744200640 1 -- 172.21.15.47:0/1735706761 wait complete. 2024-02-19T15:24:50.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.797+0000 7fc744200640 1 Processor -- start 2024-02-19T15:24:50.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.797+0000 7fc744200640 1 -- start start 2024-02-19T15:24:50.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.797+0000 7fc744200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc73c0b2500 0x7fc73c0cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:50.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.797+0000 7fc744200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc73c15eee0 0x7fc73c158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:50.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.797+0000 7fc744200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc73c1614e0 0x7fc73c09bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:50.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.797+0000 7fc744200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c09daa0 con 0x7fc73c0b2500 2024-02-19T15:24:50.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.797+0000 7fc744200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c09dd80 con 0x7fc73c15eee0 2024-02-19T15:24:50.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.797+0000 7fc744200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c09b300 con 0x7fc73c1614e0 2024-02-19T15:24:50.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.797+0000 7fc741800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc73c0b2500 0x7fc73c0cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:50.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.797+0000 7fc741800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc73c0b2500 0x7fc73c0cbc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34428/0 (socket says 172.21.15.47:34428) 2024-02-19T15:24:50.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.797+0000 7fc741800640 1 -- 172.21.15.47:0/1868937600 learned_addr learned my addr 172.21.15.47:0/1868937600 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:50.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.801+0000 7fc742200640 1 --2- 172.21.15.47:0/1868937600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc73c1614e0 0x7fc73c09bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:50.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.801+0000 7fc740e00640 1 --2- 172.21.15.47:0/1868937600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc73c15eee0 0x7fc73c158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:50.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.801+0000 7fc741800640 1 -- 172.21.15.47:0/1868937600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc73c1614e0 msgr2=0x7fc73c09bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:50.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.801+0000 7fc741800640 1 --2- 172.21.15.47:0/1868937600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc73c1614e0 0x7fc73c09bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.801+0000 7fc741800640 1 -- 172.21.15.47:0/1868937600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc73c15eee0 msgr2=0x7fc73c158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:50.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.801+0000 7fc741800640 1 --2- 172.21.15.47:0/1868937600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc73c15eee0 0x7fc73c158430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.801+0000 7fc741800640 1 -- 172.21.15.47:0/1868937600 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc734009a70 con 0x7fc73c0b2500 2024-02-19T15:24:50.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.801+0000 7fc741800640 1 --2- 172.21.15.47:0/1868937600 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc73c0b2500 0x7fc73c0cbc70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc73000e870 tx=0x7fc73000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:50.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.801+0000 7fc739600640 1 -- 172.21.15.47:0/1868937600 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc730009510 con 0x7fc73c0b2500 2024-02-19T15:24:50.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.801+0000 7fc739600640 1 -- 172.21.15.47:0/1868937600 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73000eec0 con 0x7fc73c0b2500 2024-02-19T15:24:50.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.801+0000 7fc739600640 1 -- 172.21.15.47:0/1868937600 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73000cb40 con 0x7fc73c0b2500 2024-02-19T15:24:50.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.801+0000 7fc744200640 1 -- 172.21.15.47:0/1868937600 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc73c09b5e0 con 0x7fc73c0b2500 2024-02-19T15:24:50.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.801+0000 7fc744200640 1 -- 172.21.15.47:0/1868937600 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc73c09e060 con 0x7fc73c0b2500 2024-02-19T15:24:50.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.801+0000 7fc744200640 1 -- 172.21.15.47:0/1868937600 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc73c075360 con 0x7fc73c0b2500 2024-02-19T15:24:50.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.801+0000 7fc739600640 1 -- 172.21.15.47:0/1868937600 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc73000ccc0 con 0x7fc73c0b2500 2024-02-19T15:24:50.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.801+0000 7fc739600640 1 --2- 172.21.15.47:0/1868937600 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc71c066e40 0x7fc71c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:50.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.801+0000 7fc739600640 1 -- 172.21.15.47:0/1868937600 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc730084ae0 con 0x7fc73c0b2500 2024-02-19T15:24:50.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.805+0000 7fc740e00640 1 --2- 172.21.15.47:0/1868937600 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc71c066e40 0x7fc71c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:50.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.805+0000 7fc740e00640 1 --2- 172.21.15.47:0/1868937600 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc71c066e40 0x7fc71c069300 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc73c168140 tx=0x7fc728007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:50.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.813+0000 7fc739600640 1 -- 172.21.15.47:0/1868937600 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc730051970 con 0x7fc73c0b2500 2024-02-19T15:24:50.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.985+0000 7fc744200640 1 -- 172.21.15.47:0/1868937600 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc73c09b770 con 0x7fc73c0b2500 2024-02-19T15:24:50.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.985+0000 7fc739600640 1 -- 172.21.15.47:0/1868937600 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v57) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc730017080 con 0x7fc73c0b2500 2024-02-19T15:24:50.989 INFO:teuthology.orchestra.run.smithi047.stderr:listed 0 entries 2024-02-19T15:24:50.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.989+0000 7fc744200640 1 -- 172.21.15.47:0/1868937600 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc71c066e40 msgr2=0x7fc71c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:50.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.989+0000 7fc744200640 1 --2- 172.21.15.47:0/1868937600 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc71c066e40 0x7fc71c069300 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc73c168140 tx=0x7fc728007e90 comp rx=0 tx=0).stop 2024-02-19T15:24:50.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.989+0000 7fc744200640 1 -- 172.21.15.47:0/1868937600 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc73c0b2500 msgr2=0x7fc73c0cbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:50.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.989+0000 7fc744200640 1 --2- 172.21.15.47:0/1868937600 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc73c0b2500 0x7fc73c0cbc70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc73000e870 tx=0x7fc73000ec70 comp rx=0 tx=0).stop 2024-02-19T15:24:50.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.989+0000 7fc744200640 1 -- 172.21.15.47:0/1868937600 shutdown_connections 2024-02-19T15:24:50.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.989+0000 7fc744200640 1 --2- 172.21.15.47:0/1868937600 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc71c066e40 0x7fc71c069300 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.989+0000 7fc744200640 1 --2- 172.21.15.47:0/1868937600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc73c1614e0 0x7fc73c09bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.989+0000 7fc744200640 1 --2- 172.21.15.47:0/1868937600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc73c15eee0 0x7fc73c158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.989+0000 7fc744200640 1 --2- 172.21.15.47:0/1868937600 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc73c0b2500 0x7fc73c0cbc70 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:50.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.989+0000 7fc744200640 1 -- 172.21.15.47:0/1868937600 >> 172.21.15.47:0/1868937600 conn(0x7fc73c0b4fc0 msgr2=0x7fc73c15e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:50.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.989+0000 7fc744200640 1 -- 172.21.15.47:0/1868937600 shutdown_connections 2024-02-19T15:24:50.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:50.989+0000 7fc744200640 1 -- 172.21.15.47:0/1868937600 wait complete. 2024-02-19T15:24:51.010 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-02-19T15:24:51.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:51.010 DEBUG:teuthology.orchestra.run.smithi047:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-02-19T15:24:51.139 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi047.front.sepia.ceph.com 2024-02-19T15:24:51.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:24:51.139 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-19T15:24:51.150 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:24:51.150 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:51.158 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:24:51.158 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:24:51.158 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:51.199 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:24:51.199 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:24:51.199 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:24:51.210 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:24:51.210 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:24:51.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.381+0000 7f9752000640 1 Processor -- start 2024-02-19T15:24:51.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.381+0000 7f9752000640 1 -- start start 2024-02-19T15:24:51.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.381+0000 7f9752000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f974c15b1a0 0x7f974c15b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:51.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.381+0000 7f9752000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f974c071350 0x7f974c15bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:51.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.381+0000 7f9752000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f974c06c9a0 0x7f974c15c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:51.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.381+0000 7f9752000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f974c15c000 con 0x7f974c071350 2024-02-19T15:24:51.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.381+0000 7f9752000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f974c0718c0 con 0x7f974c15b1a0 2024-02-19T15:24:51.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.381+0000 7f9752000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974c071ba0 con 0x7f974c06c9a0 2024-02-19T15:24:51.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.381+0000 7f974aa00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f974c071350 0x7f974c15bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:51.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.381+0000 7f974b400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f974c15b1a0 0x7f974c15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:51.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.381+0000 7f974aa00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f974c071350 0x7f974c15bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34438/0 (socket says 172.21.15.47:34438) 2024-02-19T15:24:51.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.381+0000 7f974aa00640 1 -- 172.21.15.47:0/3280694101 learned_addr learned my addr 172.21.15.47:0/3280694101 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:51.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.381+0000 7f974be00640 1 --2- 172.21.15.47:0/3280694101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f974c06c9a0 0x7f974c15c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:51.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.381+0000 7f974aa00640 1 -- 172.21.15.47:0/3280694101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f974c06c9a0 msgr2=0x7f974c15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:51.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.381+0000 7f974aa00640 1 --2- 172.21.15.47:0/3280694101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f974c06c9a0 0x7f974c15c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.381+0000 7f974aa00640 1 -- 172.21.15.47:0/3280694101 >> v2:172.21.15.116:3300/0 conn(0x7f974c15b1a0 msgr2=0x7f974c15b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:51.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.381+0000 7f974aa00640 1 --2- 172.21.15.47:0/3280694101 >> v2:172.21.15.116:3300/0 conn(0x7f974c15b1a0 0x7f974c15b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.381+0000 7f974aa00640 1 -- 172.21.15.47:0/3280694101 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974c071e80 con 0x7f974c071350 2024-02-19T15:24:51.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.385+0000 7f974aa00640 1 --2- 172.21.15.47:0/3280694101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f974c071350 0x7f974c15bac0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f9738009870 tx=0x7f973801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e575b98911fc7a1b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:51.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.385+0000 7f974a000640 1 -- 172.21.15.47:0/3280694101 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97380aa430 con 0x7f974c071350 2024-02-19T15:24:51.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.385+0000 7f974a000640 1 -- 172.21.15.47:0/3280694101 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97380aa590 con 0x7f974c071350 2024-02-19T15:24:51.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.385+0000 7f974a000640 1 -- 172.21.15.47:0/3280694101 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97380aa920 con 0x7f974c071350 2024-02-19T15:24:51.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.385+0000 7f9752000640 1 -- 172.21.15.47:0/3280694101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f974c071350 msgr2=0x7f974c15bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:51.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.385+0000 7f9752000640 1 --2- 172.21.15.47:0/3280694101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f974c071350 0x7f974c15bac0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f9738009870 tx=0x7f973801c4f0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.385+0000 7f9752000640 1 -- 172.21.15.47:0/3280694101 shutdown_connections 2024-02-19T15:24:51.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.385+0000 7f9752000640 1 --2- 172.21.15.47:0/3280694101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f974c06c9a0 0x7f974c15c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.385+0000 7f9752000640 1 --2- 172.21.15.47:0/3280694101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f974c071350 0x7f974c15bac0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.385+0000 7f9752000640 1 --2- 172.21.15.47:0/3280694101 >> v2:172.21.15.116:3300/0 conn(0x7f974c15b1a0 0x7f974c15b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.385+0000 7f9752000640 1 -- 172.21.15.47:0/3280694101 >> 172.21.15.47:0/3280694101 conn(0x7f974c0b16a0 msgr2=0x7f974c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:51.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.385+0000 7f9752000640 1 -- 172.21.15.47:0/3280694101 shutdown_connections 2024-02-19T15:24:51.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.385+0000 7f9752000640 1 -- 172.21.15.47:0/3280694101 wait complete. 2024-02-19T15:24:51.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.385+0000 7f9752000640 1 Processor -- start 2024-02-19T15:24:51.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.385+0000 7f9752000640 1 -- start start 2024-02-19T15:24:51.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f9752000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f974c06c9a0 0x7f974c14c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:51.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f9752000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f974c071350 0x7f974c153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:51.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f9752000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f974c15b1a0 0x7f974c150600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:51.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f9752000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c1551e0 con 0x7f974c06c9a0 2024-02-19T15:24:51.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f9752000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c1545d0 con 0x7f974c15b1a0 2024-02-19T15:24:51.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f9752000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974c1548b0 con 0x7f974c071350 2024-02-19T15:24:51.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f974b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f974c06c9a0 0x7f974c14c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:51.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f974b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f974c06c9a0 0x7f974c14c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34444/0 (socket says 172.21.15.47:34444) 2024-02-19T15:24:51.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f974b400640 1 -- 172.21.15.47:0/1879842867 learned_addr learned my addr 172.21.15.47:0/1879842867 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:51.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f974aa00640 1 --2- 172.21.15.47:0/1879842867 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f974c071350 0x7f974c153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:51.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f974be00640 1 --2- 172.21.15.47:0/1879842867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f974c15b1a0 0x7f974c150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:51.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f974b400640 1 -- 172.21.15.47:0/1879842867 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f974c071350 msgr2=0x7f974c153b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:51.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f974b400640 1 --2- 172.21.15.47:0/1879842867 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f974c071350 0x7f974c153b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f974b400640 1 -- 172.21.15.47:0/1879842867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f974c15b1a0 msgr2=0x7f974c150600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:51.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f974b400640 1 --2- 172.21.15.47:0/1879842867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f974c15b1a0 0x7f974c150600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f974b400640 1 -- 172.21.15.47:0/1879842867 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97380094d0 con 0x7f974c06c9a0 2024-02-19T15:24:51.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f974b400640 1 --2- 172.21.15.47:0/1879842867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f974c06c9a0 0x7f974c14c820 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f973c010df0 tx=0x7f973c00e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:51.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f9743400640 1 -- 172.21.15.47:0/1879842867 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c00b930 con 0x7f974c06c9a0 2024-02-19T15:24:51.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f9743400640 1 -- 172.21.15.47:0/1879842867 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973c011860 con 0x7f974c06c9a0 2024-02-19T15:24:51.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f9752000640 1 -- 172.21.15.47:0/1879842867 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974c154b90 con 0x7f974c06c9a0 2024-02-19T15:24:51.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f9743400640 1 -- 172.21.15.47:0/1879842867 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c011bd0 con 0x7f974c06c9a0 2024-02-19T15:24:51.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f9752000640 1 -- 172.21.15.47:0/1879842867 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974c14cd60 con 0x7f974c06c9a0 2024-02-19T15:24:51.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.389+0000 7f9752000640 1 -- 172.21.15.47:0/1879842867 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974c075360 con 0x7f974c06c9a0 2024-02-19T15:24:51.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.393+0000 7f9743400640 1 -- 172.21.15.47:0/1879842867 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f973c011d30 con 0x7f974c06c9a0 2024-02-19T15:24:51.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.393+0000 7f9743400640 1 --2- 172.21.15.47:0/1879842867 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f972c066e40 0x7f972c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:51.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.393+0000 7f9743400640 1 -- 172.21.15.47:0/1879842867 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f973c0878e0 con 0x7f974c06c9a0 2024-02-19T15:24:51.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.393+0000 7f974aa00640 1 --2- 172.21.15.47:0/1879842867 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f972c066e40 0x7f972c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:51.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.393+0000 7f974aa00640 1 --2- 172.21.15.47:0/1879842867 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f972c066e40 0x7f972c069300 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f97380ae000 tx=0x7f97380b5040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:51.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.397+0000 7f9743400640 1 -- 172.21.15.47:0/1879842867 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f973c0546e0 con 0x7f974c06c9a0 2024-02-19T15:24:51.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.593+0000 7f9752000640 1 -- 172.21.15.47:0/1879842867 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f974c0cefd0 con 0x7f974c06c9a0 2024-02-19T15:24:51.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.593+0000 7f9743400640 1 -- 172.21.15.47:0/1879842867 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f973c01e020 con 0x7f974c06c9a0 2024-02-19T15:24:51.597 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:51.597 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5384,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2825837364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2825837364},{"type":"v1","addr":"172.21.15.47:6835","nonce":2825837364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2251385237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2251385237},{"type":"v1","addr":"172.21.15.116:6837","nonce":2251385237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/2438199468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2438199468},{"type":"v1","addr":"172.21.15.116:6836","nonce":2438199468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:24:25.161496+0000","modified":"2024-02-19T15:24:28.184021+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5910":{"gid":5910,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6834/61582103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":61582103},{"type":"v1","addr":"172.21.15.47:6834","nonce":61582103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-19T15:24:51.597 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 33 2024-02-19T15:24:51.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.597+0000 7f9752000640 1 -- 172.21.15.47:0/1879842867 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f972c066e40 msgr2=0x7f972c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:51.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.597+0000 7f9752000640 1 --2- 172.21.15.47:0/1879842867 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f972c066e40 0x7f972c069300 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f97380ae000 tx=0x7f97380b5040 comp rx=0 tx=0).stop 2024-02-19T15:24:51.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.597+0000 7f9752000640 1 -- 172.21.15.47:0/1879842867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f974c06c9a0 msgr2=0x7f974c14c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:51.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.597+0000 7f9752000640 1 --2- 172.21.15.47:0/1879842867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f974c06c9a0 0x7f974c14c820 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f973c010df0 tx=0x7f973c00e730 comp rx=0 tx=0).stop 2024-02-19T15:24:51.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.597+0000 7f9752000640 1 -- 172.21.15.47:0/1879842867 shutdown_connections 2024-02-19T15:24:51.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.597+0000 7f9752000640 1 --2- 172.21.15.47:0/1879842867 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f972c066e40 0x7f972c069300 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.597+0000 7f9752000640 1 --2- 172.21.15.47:0/1879842867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f974c15b1a0 0x7f974c150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.597+0000 7f9752000640 1 --2- 172.21.15.47:0/1879842867 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f974c071350 0x7f974c153b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.597+0000 7f9752000640 1 --2- 172.21.15.47:0/1879842867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f974c06c9a0 0x7f974c14c820 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.597+0000 7f9752000640 1 -- 172.21.15.47:0/1879842867 >> 172.21.15.47:0/1879842867 conn(0x7f974c0b16a0 msgr2=0x7f974c070af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:51.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.597+0000 7f9752000640 1 -- 172.21.15.47:0/1879842867 shutdown_connections 2024-02-19T15:24:51.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.597+0000 7f9752000640 1 -- 172.21.15.47:0/1879842867 wait complete. 2024-02-19T15:24:51.618 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:24:51.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.777+0000 7f1d00000640 1 Processor -- start 2024-02-19T15:24:51.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.777+0000 7f1d00000640 1 -- start start 2024-02-19T15:24:51.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.781+0000 7f1d00000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1cf80b5d90 0x7f1cf80d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:51.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.781+0000 7f1d00000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cf80d0df0 0x7f1cf806b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:51.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.781+0000 7f1d00000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf80d00f0 con 0x7f1cf815bef0 2024-02-19T15:24:51.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.781+0000 7f1d00000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf80af690 con 0x7f1cf80b5d90 2024-02-19T15:24:51.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.781+0000 7f1d00000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf80af970 con 0x7f1cf80d0df0 2024-02-19T15:24:51.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.781+0000 7f1cfd600640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1cf815bef0 0x7f1cf80b2800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43564/0 (socket says 172.21.15.47:43564) 2024-02-19T15:24:51.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.781+0000 7f1cfd600640 1 -- 172.21.15.47:0/2666243254 learned_addr learned my addr 172.21.15.47:0/2666243254 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:51.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.781+0000 7f1cfe000640 1 --2- 172.21.15.47:0/2666243254 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cf80d0df0 0x7f1cf806b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:51.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.781+0000 7f1cfcc00640 1 --2- 172.21.15.47:0/2666243254 >> v2:172.21.15.116:3300/0 conn(0x7f1cf80b5d90 0x7f1cf80d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:51.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.781+0000 7f1cf7e00640 1 -- 172.21.15.47:0/2666243254 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887095589 0 0) 0x7f1cf80d00f0 con 0x7f1cf815bef0 2024-02-19T15:24:51.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.781+0000 7f1cf7e00640 1 -- 172.21.15.47:0/2666243254 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce0003530 con 0x7f1cf815bef0 2024-02-19T15:24:51.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.781+0000 7f1cfcc00640 1 -- 172.21.15.47:0/2666243254 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cf80d0df0 msgr2=0x7f1cf806b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:51.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.781+0000 7f1cfcc00640 1 --2- 172.21.15.47:0/2666243254 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cf80d0df0 0x7f1cf806b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.781+0000 7f1cfcc00640 1 -- 172.21.15.47:0/2666243254 >> v1:172.21.15.47:6789/0 conn(0x7f1cf815bef0 legacy=0x7f1cf80b2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:51.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.781+0000 7f1cfcc00640 1 -- 172.21.15.47:0/2666243254 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf80d0390 con 0x7f1cf80b5d90 2024-02-19T15:24:51.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.781+0000 7f1cfe000640 1 --2- 172.21.15.47:0/2666243254 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cf80d0df0 0x7f1cf806b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:24:51.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.785+0000 7f1cfcc00640 1 --2- 172.21.15.47:0/2666243254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cf80b5d90 0x7f1cf80d08b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40097c0 tx=0x7f1ce401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=614d562e33761f59 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:51.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.785+0000 7f1cf7e00640 1 -- 172.21.15.47:0/2666243254 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce40ab600 con 0x7f1cf80b5d90 2024-02-19T15:24:51.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.785+0000 7f1cf7e00640 1 -- 172.21.15.47:0/2666243254 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce40a7050 con 0x7f1cf80b5d90 2024-02-19T15:24:51.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.785+0000 7f1cf7e00640 1 -- 172.21.15.47:0/2666243254 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce40ab980 con 0x7f1cf80b5d90 2024-02-19T15:24:51.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.785+0000 7f1d00000640 1 -- 172.21.15.47:0/2666243254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cf80b5d90 msgr2=0x7f1cf80d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:51.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.785+0000 7f1d00000640 1 --2- 172.21.15.47:0/2666243254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cf80b5d90 0x7f1cf80d08b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40097c0 tx=0x7f1ce401c410 comp rx=0 tx=0).stop 2024-02-19T15:24:51.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.785+0000 7f1d00000640 1 -- 172.21.15.47:0/2666243254 shutdown_connections 2024-02-19T15:24:51.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.785+0000 7f1d00000640 1 --2- 172.21.15.47:0/2666243254 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cf80d0df0 0x7f1cf806b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.785+0000 7f1d00000640 1 --2- 172.21.15.47:0/2666243254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cf80b5d90 0x7f1cf80d08b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.785+0000 7f1d00000640 1 -- 172.21.15.47:0/2666243254 >> 172.21.15.47:0/2666243254 conn(0x7f1cf80b4fc0 msgr2=0x7f1cf80b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:51.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.785+0000 7f1d00000640 1 -- 172.21.15.47:0/2666243254 shutdown_connections 2024-02-19T15:24:51.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.785+0000 7f1d00000640 1 -- 172.21.15.47:0/2666243254 wait complete. 2024-02-19T15:24:51.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.785+0000 7f1d00000640 1 Processor -- start 2024-02-19T15:24:51.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1d00000640 1 -- start start 2024-02-19T15:24:51.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1d00000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cf80b5d90 0x7f1cf8147190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:51.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1d00000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cf80d0df0 0x7f1cf81520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:51.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1d00000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1cf815bef0 0x7f1cf8146980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:51.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1d00000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf815ea40 con 0x7f1cf815bef0 2024-02-19T15:24:51.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1d00000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf814fd20 con 0x7f1cf80d0df0 2024-02-19T15:24:51.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1d00000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf815f1f0 con 0x7f1cf80b5d90 2024-02-19T15:24:51.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1cfe000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1cf815bef0 0x7f1cf8146980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:51.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1cfe000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1cf815bef0 0x7f1cf8146980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34450/0 (socket says 172.21.15.47:34450) 2024-02-19T15:24:51.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1cfe000640 1 -- 172.21.15.47:0/3125387321 learned_addr learned my addr 172.21.15.47:0/3125387321 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:51.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1cfcc00640 1 --2- 172.21.15.47:0/3125387321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cf80d0df0 0x7f1cf81520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:51.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1cfd600640 1 --2- 172.21.15.47:0/3125387321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cf80b5d90 0x7f1cf8147190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:51.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1cfe000640 1 -- 172.21.15.47:0/3125387321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cf80b5d90 msgr2=0x7f1cf8147190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:51.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1cfe000640 1 --2- 172.21.15.47:0/3125387321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cf80b5d90 0x7f1cf8147190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1cfe000640 1 -- 172.21.15.47:0/3125387321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cf80d0df0 msgr2=0x7f1cf81520d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:51.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1cfe000640 1 --2- 172.21.15.47:0/3125387321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cf80d0df0 0x7f1cf81520d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1cfe000640 1 -- 172.21.15.47:0/3125387321 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf0009a70 con 0x7f1cf815bef0 2024-02-19T15:24:51.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1cfe000640 1 --2- 172.21.15.47:0/3125387321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1cf815bef0 0x7f1cf8146980 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f1cf000ef90 tx=0x7f1cf0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:51.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1cf5600640 1 -- 172.21.15.47:0/3125387321 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf0009d70 con 0x7f1cf815bef0 2024-02-19T15:24:51.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1cf5600640 1 -- 172.21.15.47:0/3125387321 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf000f040 con 0x7f1cf815bef0 2024-02-19T15:24:51.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1cf5600640 1 -- 172.21.15.47:0/3125387321 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf00146c0 con 0x7f1cf815bef0 2024-02-19T15:24:51.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1d00000640 1 -- 172.21.15.47:0/3125387321 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce4009420 con 0x7f1cf815bef0 2024-02-19T15:24:51.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1d00000640 1 -- 172.21.15.47:0/3125387321 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf815ec70 con 0x7f1cf815bef0 2024-02-19T15:24:51.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.789+0000 7f1d00000640 1 -- 172.21.15.47:0/3125387321 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf814fe90 con 0x7f1cf815bef0 2024-02-19T15:24:51.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.793+0000 7f1cf5600640 1 -- 172.21.15.47:0/3125387321 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1cf0014820 con 0x7f1cf815bef0 2024-02-19T15:24:51.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.793+0000 7f1cf5600640 1 --2- 172.21.15.47:0/3125387321 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1cd8066e70 0x7f1cd8069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:51.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.793+0000 7f1cf5600640 1 -- 172.21.15.47:0/3125387321 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1cf00857e0 con 0x7f1cf815bef0 2024-02-19T15:24:51.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.793+0000 7f1cfd600640 1 --2- 172.21.15.47:0/3125387321 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1cd8066e70 0x7f1cd8069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:51.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.793+0000 7f1cfd600640 1 --2- 172.21.15.47:0/3125387321 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1cd8066e70 0x7f1cd8069330 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1cec007fd0 tx=0x7f1cec009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:51.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.797+0000 7f1cf5600640 1 -- 172.21.15.47:0/3125387321 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1cf0052710 con 0x7f1cf815bef0 2024-02-19T15:24:51.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.961+0000 7f1d00000640 1 -- 172.21.15.47:0/3125387321 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1cf8152610 con 0x7f1cf815bef0 2024-02-19T15:24:51.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.961+0000 7f1cf5600640 1 -- 172.21.15.47:0/3125387321 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16677 (secure 0 0 0) 0x7f1cf0018020 con 0x7f1cf815bef0 2024-02-19T15:24:51.965 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:51.965 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":57,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:24:26.785375+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:24:22.931622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-02-19T15:24:24.163465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:24:51.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.965+0000 7f1d00000640 1 -- 172.21.15.47:0/3125387321 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1cd8066e70 msgr2=0x7f1cd8069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:51.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.965+0000 7f1d00000640 1 --2- 172.21.15.47:0/3125387321 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1cd8066e70 0x7f1cd8069330 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1cec007fd0 tx=0x7f1cec009040 comp rx=0 tx=0).stop 2024-02-19T15:24:51.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.965+0000 7f1d00000640 1 -- 172.21.15.47:0/3125387321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1cf815bef0 msgr2=0x7f1cf8146980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:51.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.965+0000 7f1d00000640 1 --2- 172.21.15.47:0/3125387321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1cf815bef0 0x7f1cf8146980 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f1cf000ef90 tx=0x7f1cf0009f90 comp rx=0 tx=0).stop 2024-02-19T15:24:51.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.965+0000 7f1d00000640 1 -- 172.21.15.47:0/3125387321 shutdown_connections 2024-02-19T15:24:51.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.965+0000 7f1d00000640 1 --2- 172.21.15.47:0/3125387321 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1cd8066e70 0x7f1cd8069330 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.965+0000 7f1d00000640 1 --2- 172.21.15.47:0/3125387321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1cf815bef0 0x7f1cf8146980 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.965+0000 7f1d00000640 1 --2- 172.21.15.47:0/3125387321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cf80d0df0 0x7f1cf81520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.965+0000 7f1d00000640 1 --2- 172.21.15.47:0/3125387321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cf80b5d90 0x7f1cf8147190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:51.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.965+0000 7f1d00000640 1 -- 172.21.15.47:0/3125387321 >> 172.21.15.47:0/3125387321 conn(0x7f1cf80b4fc0 msgr2=0x7f1cf8165e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:51.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.965+0000 7f1d00000640 1 -- 172.21.15.47:0/3125387321 shutdown_connections 2024-02-19T15:24:51.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:51.965+0000 7f1d00000640 1 -- 172.21.15.47:0/3125387321 wait complete. 2024-02-19T15:24:51.987 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:24:22.931622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:24:24.163465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:24:51.988 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:24:52.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.125+0000 7fa007800640 1 Processor -- start 2024-02-19T15:24:52.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.125+0000 7fa007800640 1 -- start start 2024-02-19T15:24:52.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.129+0000 7fa007800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa0000ce3b0 0x7fa0000ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:52.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.129+0000 7fa007800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0000cc620 con 0x7fa0000d2650 2024-02-19T15:24:52.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.129+0000 7fa007800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0000d12e0 con 0x7fa00015e280 2024-02-19T15:24:52.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.129+0000 7fa007800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0000d1940 con 0x7fa0000ce3b0 2024-02-19T15:24:52.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.129+0000 7fa004e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa0000ce3b0 0x7fa0000ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:52.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.129+0000 7f9ff7e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa0000d2650 0x7fa0000ccab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43578/0 (socket says 172.21.15.47:43578) 2024-02-19T15:24:52.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.129+0000 7fa004e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa0000ce3b0 0x7fa0000ce790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34456/0 (socket says 172.21.15.47:34456) 2024-02-19T15:24:52.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.129+0000 7f9ff7e00640 1 -- 172.21.15.47:0/2954751213 learned_addr learned my addr 172.21.15.47:0/2954751213 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:52.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.129+0000 7fa004e00640 1 -- 172.21.15.47:0/2954751213 >> v1:172.21.15.116:6789/0 conn(0x7fa00015e280 legacy=0x7fa0000cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:52.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.129+0000 7fa004e00640 1 -- 172.21.15.47:0/2954751213 >> v1:172.21.15.47:6789/0 conn(0x7fa0000d2650 legacy=0x7fa0000ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:52.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.129+0000 7f9fffe00640 1 -- 172.21.15.47:0/2954751213 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094743558 0 0) 0x7fa0000cc620 con 0x7fa0000d2650 2024-02-19T15:24:52.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.129+0000 7fa004e00640 1 -- 172.21.15.47:0/2954751213 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0000cecd0 con 0x7fa0000ce3b0 2024-02-19T15:24:52.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7fa004e00640 1 --2- 172.21.15.47:0/2954751213 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0000ce3b0 0x7fa0000ce790 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f9ff00097c0 tx=0x7f9ff001c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ffe53d2e46ba3a6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:52.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7f9fffe00640 1 -- 172.21.15.47:0/2954751213 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff00ab600 con 0x7fa0000ce3b0 2024-02-19T15:24:52.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7f9fffe00640 1 -- 172.21.15.47:0/2954751213 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff00a7050 con 0x7fa0000ce3b0 2024-02-19T15:24:52.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7f9fffe00640 1 -- 172.21.15.47:0/2954751213 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff00ab980 con 0x7fa0000ce3b0 2024-02-19T15:24:52.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7fa007800640 1 -- 172.21.15.47:0/2954751213 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0000ce3b0 msgr2=0x7fa0000ce790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:52.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7fa007800640 1 --2- 172.21.15.47:0/2954751213 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0000ce3b0 0x7fa0000ce790 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f9ff00097c0 tx=0x7f9ff001c380 comp rx=0 tx=0).stop 2024-02-19T15:24:52.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7fa007800640 1 -- 172.21.15.47:0/2954751213 shutdown_connections 2024-02-19T15:24:52.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7fa007800640 1 --2- 172.21.15.47:0/2954751213 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0000ce3b0 0x7fa0000ce790 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7fa007800640 1 -- 172.21.15.47:0/2954751213 >> 172.21.15.47:0/2954751213 conn(0x7fa0000d1b30 msgr2=0x7fa0000d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:52.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7fa007800640 1 -- 172.21.15.47:0/2954751213 shutdown_connections 2024-02-19T15:24:52.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7fa007800640 1 -- 172.21.15.47:0/2954751213 wait complete. 2024-02-19T15:24:52.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7fa007800640 1 Processor -- start 2024-02-19T15:24:52.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7fa007800640 1 -- start start 2024-02-19T15:24:52.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7fa007800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0000ce3b0 0x7fa00014ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:52.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7fa007800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0000d2650 0x7fa000154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:52.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7fa007800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa00015e280 0x7fa000155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:52.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7fa007800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00015a370 con 0x7fa0000ce3b0 2024-02-19T15:24:52.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7fa007800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0000c9660 con 0x7fa0000d2650 2024-02-19T15:24:52.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7fa007800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa000157c20 con 0x7fa00015e280 2024-02-19T15:24:52.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7fa004e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0000ce3b0 0x7fa00014ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:52.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.133+0000 7fa004e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0000ce3b0 0x7fa00014ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34462/0 (socket says 172.21.15.47:34462) 2024-02-19T15:24:52.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.137+0000 7fa004e00640 1 -- 172.21.15.47:0/4232583807 learned_addr learned my addr 172.21.15.47:0/4232583807 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:52.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.137+0000 7f9ff7e00640 1 --2- 172.21.15.47:0/4232583807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0000d2650 0x7fa000154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:52.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.137+0000 7fa005800640 1 --2- 172.21.15.47:0/4232583807 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa00015e280 0x7fa000155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:52.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.137+0000 7fa004e00640 1 -- 172.21.15.47:0/4232583807 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa00015e280 msgr2=0x7fa000155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:52.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.137+0000 7fa004e00640 1 --2- 172.21.15.47:0/4232583807 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa00015e280 0x7fa000155180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.137+0000 7fa004e00640 1 -- 172.21.15.47:0/4232583807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0000d2650 msgr2=0x7fa000154c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:52.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.137+0000 7fa004e00640 1 --2- 172.21.15.47:0/4232583807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0000d2650 0x7fa000154c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.137+0000 7fa004e00640 1 -- 172.21.15.47:0/4232583807 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff0009420 con 0x7fa0000ce3b0 2024-02-19T15:24:52.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.137+0000 7fa005800640 1 --2- 172.21.15.47:0/4232583807 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa00015e280 0x7fa000155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:52.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.137+0000 7fa004e00640 1 --2- 172.21.15.47:0/4232583807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0000ce3b0 0x7fa00014ff60 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f9ff00ae040 tx=0x7f9ff0019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:52.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.137+0000 7f9ffd600640 1 -- 172.21.15.47:0/4232583807 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff00b4900 con 0x7fa0000ce3b0 2024-02-19T15:24:52.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.137+0000 7f9ffd600640 1 -- 172.21.15.47:0/4232583807 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff00b4a80 con 0x7fa0000ce3b0 2024-02-19T15:24:52.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.137+0000 7f9ffd600640 1 -- 172.21.15.47:0/4232583807 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff00b4e30 con 0x7fa0000ce3b0 2024-02-19T15:24:52.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.137+0000 7fa007800640 1 -- 172.21.15.47:0/4232583807 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0000c9900 con 0x7fa0000ce3b0 2024-02-19T15:24:52.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.137+0000 7fa007800640 1 -- 172.21.15.47:0/4232583807 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0000c9ec0 con 0x7fa0000ce3b0 2024-02-19T15:24:52.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.137+0000 7fa007800640 1 -- 172.21.15.47:0/4232583807 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0000ca0b0 con 0x7fa0000ce3b0 2024-02-19T15:24:52.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.137+0000 7f9ffd600640 1 -- 172.21.15.47:0/4232583807 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9ff00b2040 con 0x7fa0000ce3b0 2024-02-19T15:24:52.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.141+0000 7f9ffd600640 1 --2- 172.21.15.47:0/4232583807 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9fdc066e40 0x7f9fdc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:52.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.141+0000 7f9ffd600640 1 -- 172.21.15.47:0/4232583807 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ff011edb0 con 0x7fa0000ce3b0 2024-02-19T15:24:52.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.141+0000 7f9ff7e00640 1 --2- 172.21.15.47:0/4232583807 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9fdc066e40 0x7f9fdc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:52.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.141+0000 7f9ff7e00640 1 --2- 172.21.15.47:0/4232583807 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9fdc066e40 0x7f9fdc069300 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa0001564f0 tx=0x7f9fe8007240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:52.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.145+0000 7f9ffd600640 1 -- 172.21.15.47:0/4232583807 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f9ff0123070 con 0x7fa0000ce3b0 2024-02-19T15:24:52.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.309+0000 7fa007800640 1 -- 172.21.15.47:0/4232583807 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa000075360 con 0x7fa0000ce3b0 2024-02-19T15:24:52.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.309+0000 7f9ffd600640 1 -- 172.21.15.47:0/4232583807 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9ff00b8020 con 0x7fa0000ce3b0 2024-02-19T15:24:52.311 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:24:52.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.309+0000 7fa007800640 1 -- 172.21.15.47:0/4232583807 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9fdc066e40 msgr2=0x7f9fdc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:52.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.309+0000 7fa007800640 1 --2- 172.21.15.47:0/4232583807 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9fdc066e40 0x7f9fdc069300 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa0001564f0 tx=0x7f9fe8007240 comp rx=0 tx=0).stop 2024-02-19T15:24:52.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.309+0000 7fa007800640 1 -- 172.21.15.47:0/4232583807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0000ce3b0 msgr2=0x7fa00014ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:52.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.309+0000 7fa007800640 1 --2- 172.21.15.47:0/4232583807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0000ce3b0 0x7fa00014ff60 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f9ff00ae040 tx=0x7f9ff0019120 comp rx=0 tx=0).stop 2024-02-19T15:24:52.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.309+0000 7fa007800640 1 -- 172.21.15.47:0/4232583807 shutdown_connections 2024-02-19T15:24:52.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.309+0000 7fa007800640 1 --2- 172.21.15.47:0/4232583807 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9fdc066e40 0x7f9fdc069300 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.309+0000 7fa007800640 1 --2- 172.21.15.47:0/4232583807 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa00015e280 0x7fa000155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.309+0000 7fa007800640 1 --2- 172.21.15.47:0/4232583807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0000d2650 0x7fa000154c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.309+0000 7fa007800640 1 --2- 172.21.15.47:0/4232583807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0000ce3b0 0x7fa00014ff60 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.309+0000 7fa007800640 1 -- 172.21.15.47:0/4232583807 >> 172.21.15.47:0/4232583807 conn(0x7fa0000d1b30 msgr2=0x7fa000156680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:52.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.309+0000 7fa007800640 1 -- 172.21.15.47:0/4232583807 shutdown_connections 2024-02-19T15:24:52.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.309+0000 7fa007800640 1 -- 172.21.15.47:0/4232583807 wait complete. 2024-02-19T15:24:52.332 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:24:52.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.473+0000 7f2660a00640 1 Processor -- start 2024-02-19T15:24:52.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.473+0000 7f2660a00640 1 -- start start 2024-02-19T15:24:52.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.473+0000 7f2660a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f265c09dc10 0x7f265c06c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:52.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.473+0000 7f2660a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f265c0acec0 0x7f265c0ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:52.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.473+0000 7f2660a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f265c06bf40 con 0x7f265c0ae290 2024-02-19T15:24:52.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.473+0000 7f2660a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f265c0ae670 con 0x7f265c0acec0 2024-02-19T15:24:52.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.473+0000 7f2660a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f265c15c200 con 0x7f265c09dc10 2024-02-19T15:24:52.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.473+0000 7f265a800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f265c0acec0 0x7f265c0ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:52.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.473+0000 7f265a800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f265c0acec0 0x7f265c0ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34464/0 (socket says 172.21.15.47:34464) 2024-02-19T15:24:52.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.473+0000 7f265a800640 1 -- 172.21.15.47:0/3931746564 learned_addr learned my addr 172.21.15.47:0/3931746564 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:52.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.473+0000 7f2659e00640 1 --1- 172.21.15.47:0/3931746564 >> v1:172.21.15.47:6789/0 conn(0x7f265c0ae290 0x7f265c06b830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43586/0 (socket says 172.21.15.47:43586) 2024-02-19T15:24:52.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.473+0000 7f2653e00640 1 --2- 172.21.15.47:0/3931746564 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f265c09dc10 0x7f265c06c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:52.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.473+0000 7f265a800640 1 -- 172.21.15.47:0/3931746564 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f265c09dc10 msgr2=0x7f265c06c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:52.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.473+0000 7f265a800640 1 --2- 172.21.15.47:0/3931746564 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f265c09dc10 0x7f265c06c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.473+0000 7f265a800640 1 -- 172.21.15.47:0/3931746564 >> v1:172.21.15.47:6789/0 conn(0x7f265c0ae290 legacy=0x7f265c06b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:52.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.473+0000 7f265a800640 1 -- 172.21.15.47:0/3931746564 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265c066e70 con 0x7f265c0acec0 2024-02-19T15:24:52.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.477+0000 7f265a800640 1 --2- 172.21.15.47:0/3931746564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f265c0acec0 0x7f265c0ad350 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f2654009e50 tx=0x7f265401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c5af4011d286035 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:52.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.477+0000 7f2659400640 1 -- 172.21.15.47:0/3931746564 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26540a57f0 con 0x7f265c0acec0 2024-02-19T15:24:52.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.477+0000 7f2659400640 1 -- 172.21.15.47:0/3931746564 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26540a5950 con 0x7f265c0acec0 2024-02-19T15:24:52.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.477+0000 7f2659400640 1 -- 172.21.15.47:0/3931746564 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26540a5ce0 con 0x7f265c0acec0 2024-02-19T15:24:52.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.477+0000 7f2660a00640 1 -- 172.21.15.47:0/3931746564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f265c0acec0 msgr2=0x7f265c0ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:52.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.477+0000 7f2660a00640 1 --2- 172.21.15.47:0/3931746564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f265c0acec0 0x7f265c0ad350 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f2654009e50 tx=0x7f265401ca30 comp rx=0 tx=0).stop 2024-02-19T15:24:52.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.477+0000 7f2660a00640 1 -- 172.21.15.47:0/3931746564 shutdown_connections 2024-02-19T15:24:52.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.477+0000 7f2660a00640 1 --2- 172.21.15.47:0/3931746564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f265c0acec0 0x7f265c0ad350 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.477+0000 7f2660a00640 1 --2- 172.21.15.47:0/3931746564 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f265c09dc10 0x7f265c06c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.477+0000 7f2660a00640 1 -- 172.21.15.47:0/3931746564 >> 172.21.15.47:0/3931746564 conn(0x7f265c0ccc50 msgr2=0x7f265c0cd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.477+0000 7f2660a00640 1 -- 172.21.15.47:0/3931746564 shutdown_connections 2024-02-19T15:24:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.477+0000 7f2660a00640 1 -- 172.21.15.47:0/3931746564 wait complete. 2024-02-19T15:24:52.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2660a00640 1 Processor -- start 2024-02-19T15:24:52.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2660a00640 1 -- start start 2024-02-19T15:24:52.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2660a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f265c09dc10 0x7f265c1549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:52.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2660a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f265c0acec0 0x7f265c0c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:52.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2660a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f265c0ae290 0x7f265c0c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:52.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2660a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c144080 con 0x7f265c0acec0 2024-02-19T15:24:52.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2660a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c1418b0 con 0x7f265c0ae290 2024-02-19T15:24:52.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2660a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f265c144540 con 0x7f265c09dc10 2024-02-19T15:24:52.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2653e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f265c0acec0 0x7f265c0c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:52.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2653e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f265c0acec0 0x7f265c0c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34474/0 (socket says 172.21.15.47:34474) 2024-02-19T15:24:52.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2653e00640 1 -- 172.21.15.47:0/554079864 learned_addr learned my addr 172.21.15.47:0/554079864 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:52.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2659e00640 1 --2- 172.21.15.47:0/554079864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f265c09dc10 0x7f265c1549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:52.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f265a800640 1 --2- 172.21.15.47:0/554079864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f265c0ae290 0x7f265c0c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:52.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2653e00640 1 -- 172.21.15.47:0/554079864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f265c09dc10 msgr2=0x7f265c1549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:52.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2653e00640 1 --2- 172.21.15.47:0/554079864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f265c09dc10 0x7f265c1549b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2653e00640 1 -- 172.21.15.47:0/554079864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f265c0ae290 msgr2=0x7f265c0c9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:52.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2653e00640 1 --2- 172.21.15.47:0/554079864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f265c0ae290 0x7f265c0c9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2653e00640 1 -- 172.21.15.47:0/554079864 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2654009b30 con 0x7f265c0acec0 2024-02-19T15:24:52.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2653e00640 1 --2- 172.21.15.47:0/554079864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f265c0acec0 0x7f265c0c95e0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f264400d620 tx=0x7f264400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:52.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2652000640 1 -- 172.21.15.47:0/554079864 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264400b620 con 0x7f265c0acec0 2024-02-19T15:24:52.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2652000640 1 -- 172.21.15.47:0/554079864 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f264400b780 con 0x7f265c0acec0 2024-02-19T15:24:52.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2660a00640 1 -- 172.21.15.47:0/554079864 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f265c0ca060 con 0x7f265c0acec0 2024-02-19T15:24:52.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2652000640 1 -- 172.21.15.47:0/554079864 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264400bb10 con 0x7f265c0acec0 2024-02-19T15:24:52.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2660a00640 1 -- 172.21.15.47:0/554079864 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f265c1411f0 con 0x7f265c0acec0 2024-02-19T15:24:52.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.481+0000 7f2660a00640 1 -- 172.21.15.47:0/554079864 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f265c141050 con 0x7f265c0acec0 2024-02-19T15:24:52.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.485+0000 7f2652000640 1 -- 172.21.15.47:0/554079864 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f264400bc70 con 0x7f265c0acec0 2024-02-19T15:24:52.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.485+0000 7f2652000640 1 --2- 172.21.15.47:0/554079864 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2638066e40 0x7f2638069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:52.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.485+0000 7f2652000640 1 -- 172.21.15.47:0/554079864 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2644084e20 con 0x7f265c0acec0 2024-02-19T15:24:52.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.485+0000 7f2659e00640 1 --2- 172.21.15.47:0/554079864 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2638066e40 0x7f2638069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:52.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.485+0000 7f2659e00640 1 --2- 172.21.15.47:0/554079864 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2638066e40 0x7f2638069300 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f264c000f70 tx=0x7f264c007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:52.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.493+0000 7f2652000640 1 -- 172.21.15.47:0/554079864 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f265c141050 con 0x7f265c0acec0 2024-02-19T15:24:52.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.653+0000 7f2660a00640 1 -- 172.21.15.47:0/554079864 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f265c141050 con 0x7f265c0acec0 2024-02-19T15:24:52.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.653+0000 7f2652000640 1 -- 172.21.15.47:0/554079864 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) v1 ==== 104+0+11 (secure 0 0 0) 0x7f265c141050 con 0x7f265c0acec0 2024-02-19T15:24:52.658 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:24:52.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.657+0000 7f2660a00640 1 -- 172.21.15.47:0/554079864 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2638066e40 msgr2=0x7f2638069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:52.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.657+0000 7f2660a00640 1 --2- 172.21.15.47:0/554079864 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2638066e40 0x7f2638069300 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f264c000f70 tx=0x7f264c007e90 comp rx=0 tx=0).stop 2024-02-19T15:24:52.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.657+0000 7f2660a00640 1 -- 172.21.15.47:0/554079864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f265c0acec0 msgr2=0x7f265c0c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:52.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.657+0000 7f2660a00640 1 --2- 172.21.15.47:0/554079864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f265c0acec0 0x7f265c0c95e0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f264400d620 tx=0x7f264400dae0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.657+0000 7f2660a00640 1 -- 172.21.15.47:0/554079864 shutdown_connections 2024-02-19T15:24:52.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.657+0000 7f2660a00640 1 --2- 172.21.15.47:0/554079864 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2638066e40 0x7f2638069300 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.657+0000 7f2660a00640 1 --2- 172.21.15.47:0/554079864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f265c0ae290 0x7f265c0c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.657+0000 7f2660a00640 1 --2- 172.21.15.47:0/554079864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f265c0acec0 0x7f265c0c95e0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.657+0000 7f2660a00640 1 --2- 172.21.15.47:0/554079864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f265c09dc10 0x7f265c1549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.657+0000 7f2660a00640 1 -- 172.21.15.47:0/554079864 >> 172.21.15.47:0/554079864 conn(0x7f265c0ccc50 msgr2=0x7f265c0cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:52.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.657+0000 7f2660a00640 1 -- 172.21.15.47:0/554079864 shutdown_connections 2024-02-19T15:24:52.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.657+0000 7f2660a00640 1 -- 172.21.15.47:0/554079864 wait complete. 2024-02-19T15:24:52.679 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:24:52.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.817+0000 7f6bc5400640 1 Processor -- start 2024-02-19T15:24:52.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.817+0000 7f6bc5400640 1 -- start start 2024-02-19T15:24:52.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.821+0000 7f6bc5400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6bc00d0550 0x7f6bc0161ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:52.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.821+0000 7f6bc5400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6bc00b1a80 0x7f6bc00cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:52.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.821+0000 7f6bc5400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bc0162090 con 0x7f6bc015df60 2024-02-19T15:24:52.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.821+0000 7f6bc5400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bc0162370 con 0x7f6bc00b1a80 2024-02-19T15:24:52.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.821+0000 7f6bc5400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc0162650 con 0x7f6bc00d0550 2024-02-19T15:24:52.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.821+0000 7f6bc4a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6bc00b1a80 0x7f6bc00cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:52.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.821+0000 7f6bc4a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6bc00b1a80 0x7f6bc00cf8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34490/0 (socket says 172.21.15.47:34490) 2024-02-19T15:24:52.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.821+0000 7f6bc4a00640 1 -- 172.21.15.47:0/2815555156 learned_addr learned my addr 172.21.15.47:0/2815555156 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:52.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.821+0000 7f6bb7400640 1 --2- 172.21.15.47:0/2815555156 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6bc00d0550 0x7f6bc0161ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:52.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.821+0000 7f6bc4a00640 1 -- 172.21.15.47:0/2815555156 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6bc00d0550 msgr2=0x7f6bc0161ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:52.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.821+0000 7f6bc4a00640 1 --2- 172.21.15.47:0/2815555156 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6bc00d0550 0x7f6bc0161ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.821+0000 7f6bc4a00640 1 -- 172.21.15.47:0/2815555156 >> v1:172.21.15.116:6789/0 conn(0x7f6bc015df60 legacy=0x7f6bc00cfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:52.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.821+0000 7f6bc4a00640 1 -- 172.21.15.47:0/2815555156 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc0162930 con 0x7f6bc00b1a80 2024-02-19T15:24:52.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bc4a00640 1 --2- 172.21.15.47:0/2815555156 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bc00b1a80 0x7f6bc00cf8a0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8009e50 tx=0x7f6bb801c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15e7527d341468e6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:52.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bbf400640 1 -- 172.21.15.47:0/2815555156 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb80a5890 con 0x7f6bc00b1a80 2024-02-19T15:24:52.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bbf400640 1 -- 172.21.15.47:0/2815555156 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb80a59f0 con 0x7f6bc00b1a80 2024-02-19T15:24:52.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bbf400640 1 -- 172.21.15.47:0/2815555156 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb80a5d80 con 0x7f6bc00b1a80 2024-02-19T15:24:52.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bc5400640 1 -- 172.21.15.47:0/2815555156 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bc00b1a80 msgr2=0x7f6bc00cf8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:52.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bc5400640 1 --2- 172.21.15.47:0/2815555156 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bc00b1a80 0x7f6bc00cf8a0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8009e50 tx=0x7f6bb801c9e0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bc5400640 1 -- 172.21.15.47:0/2815555156 shutdown_connections 2024-02-19T15:24:52.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bc5400640 1 --2- 172.21.15.47:0/2815555156 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bc00b1a80 0x7f6bc00cf8a0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bc5400640 1 --2- 172.21.15.47:0/2815555156 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6bc00d0550 0x7f6bc0161ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bc5400640 1 -- 172.21.15.47:0/2815555156 >> 172.21.15.47:0/2815555156 conn(0x7f6bc009a9d0 msgr2=0x7f6bc009ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:52.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bc5400640 1 -- 172.21.15.47:0/2815555156 shutdown_connections 2024-02-19T15:24:52.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bc5400640 1 -- 172.21.15.47:0/2815555156 wait complete. 2024-02-19T15:24:52.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bc5400640 1 Processor -- start 2024-02-19T15:24:52.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bc5400640 1 -- start start 2024-02-19T15:24:52.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bc5400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bc00b1a80 0x7f6bc015fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:52.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bc5400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6bc00d0550 0x7f6bc006fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:52.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bc5400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bc015df60 0x7f6bc0153900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:52.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bc5400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc01600d0 con 0x7f6bc00b1a80 2024-02-19T15:24:52.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bc5400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc0160320 con 0x7f6bc015df60 2024-02-19T15:24:52.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bc5400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc0160600 con 0x7f6bc00d0550 2024-02-19T15:24:52.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.825+0000 7f6bbfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bc00b1a80 0x7f6bc015fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:52.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.829+0000 7f6bbfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bc00b1a80 0x7f6bc015fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34500/0 (socket says 172.21.15.47:34500) 2024-02-19T15:24:52.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.829+0000 7f6bbfe00640 1 -- 172.21.15.47:0/241892993 learned_addr learned my addr 172.21.15.47:0/241892993 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:52.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.829+0000 7f6bb7400640 1 --2- 172.21.15.47:0/241892993 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6bc00d0550 0x7f6bc006fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:52.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.829+0000 7f6bc4a00640 1 --2- 172.21.15.47:0/241892993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bc015df60 0x7f6bc0153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:52.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.829+0000 7f6bbfe00640 1 -- 172.21.15.47:0/241892993 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6bc00d0550 msgr2=0x7f6bc006fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:52.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.829+0000 7f6bbfe00640 1 --2- 172.21.15.47:0/241892993 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6bc00d0550 0x7f6bc006fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.829+0000 7f6bbfe00640 1 -- 172.21.15.47:0/241892993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bc015df60 msgr2=0x7f6bc0153900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:52.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.829+0000 7f6bbfe00640 1 --2- 172.21.15.47:0/241892993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bc015df60 0x7f6bc0153900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:52.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.829+0000 7f6bbfe00640 1 -- 172.21.15.47:0/241892993 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb8009b30 con 0x7f6bc00b1a80 2024-02-19T15:24:52.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.829+0000 7f6bbfe00640 1 --2- 172.21.15.47:0/241892993 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bc00b1a80 0x7f6bc015fb90 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6bb000c370 tx=0x7f6bb000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:52.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.829+0000 7f6bbcc00640 1 -- 172.21.15.47:0/241892993 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb00052d0 con 0x7f6bc00b1a80 2024-02-19T15:24:52.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.829+0000 7f6bbcc00640 1 -- 172.21.15.47:0/241892993 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb0005430 con 0x7f6bc00b1a80 2024-02-19T15:24:52.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.829+0000 7f6bc5400640 1 -- 172.21.15.47:0/241892993 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc01608e0 con 0x7f6bc00b1a80 2024-02-19T15:24:52.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.829+0000 7f6bbcc00640 1 -- 172.21.15.47:0/241892993 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0007260 con 0x7f6bc00b1a80 2024-02-19T15:24:52.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.829+0000 7f6bc5400640 1 -- 172.21.15.47:0/241892993 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc0160ea0 con 0x7f6bc00b1a80 2024-02-19T15:24:52.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.829+0000 7f6bc5400640 1 -- 172.21.15.47:0/241892993 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc01544b0 con 0x7f6bc00b1a80 2024-02-19T15:24:52.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.829+0000 7f6bbcc00640 1 -- 172.21.15.47:0/241892993 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6bb00073c0 con 0x7f6bc00b1a80 2024-02-19T15:24:52.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.829+0000 7f6bbcc00640 1 --2- 172.21.15.47:0/241892993 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6b9c066e40 0x7f6b9c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:52.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.829+0000 7f6bbcc00640 1 -- 172.21.15.47:0/241892993 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6bb0083d50 con 0x7f6bc00b1a80 2024-02-19T15:24:52.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.833+0000 7f6bb7400640 1 --2- 172.21.15.47:0/241892993 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6b9c066e40 0x7f6b9c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:52.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.833+0000 7f6bb7400640 1 --2- 172.21.15.47:0/241892993 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6b9c066e40 0x7f6b9c069300 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0070780 tx=0x7f6ba8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:52.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.833+0000 7f6bbcc00640 1 -- 172.21.15.47:0/241892993 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6bc01544b0 con 0x7f6bc00b1a80 2024-02-19T15:24:53.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.997+0000 7f6bc5400640 1 -- 172.21.15.47:0/241892993 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6bc01544b0 con 0x7f6bc00b1a80 2024-02-19T15:24:53.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:52.997+0000 7f6bbcc00640 1 -- 172.21.15.47:0/241892993 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6bc01544b0 con 0x7f6bc00b1a80 2024-02-19T15:24:53.001 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:24:53.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.001+0000 7f6bc5400640 1 -- 172.21.15.47:0/241892993 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6b9c066e40 msgr2=0x7f6b9c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:53.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.001+0000 7f6bc5400640 1 --2- 172.21.15.47:0/241892993 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6b9c066e40 0x7f6b9c069300 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0070780 tx=0x7f6ba8009190 comp rx=0 tx=0).stop 2024-02-19T15:24:53.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.001+0000 7f6bc5400640 1 -- 172.21.15.47:0/241892993 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bc00b1a80 msgr2=0x7f6bc015fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:53.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.001+0000 7f6bc5400640 1 --2- 172.21.15.47:0/241892993 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bc00b1a80 0x7f6bc015fb90 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6bb000c370 tx=0x7f6bb000c830 comp rx=0 tx=0).stop 2024-02-19T15:24:53.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.001+0000 7f6bc5400640 1 -- 172.21.15.47:0/241892993 shutdown_connections 2024-02-19T15:24:53.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.001+0000 7f6bc5400640 1 --2- 172.21.15.47:0/241892993 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6b9c066e40 0x7f6b9c069300 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.001+0000 7f6bc5400640 1 --2- 172.21.15.47:0/241892993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bc015df60 0x7f6bc0153900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.001+0000 7f6bc5400640 1 --2- 172.21.15.47:0/241892993 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6bc00d0550 0x7f6bc006fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.001+0000 7f6bc5400640 1 --2- 172.21.15.47:0/241892993 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bc00b1a80 0x7f6bc015fb90 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.001+0000 7f6bc5400640 1 -- 172.21.15.47:0/241892993 >> 172.21.15.47:0/241892993 conn(0x7f6bc009a9d0 msgr2=0x7f6bc006d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:53.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.001+0000 7f6bc5400640 1 -- 172.21.15.47:0/241892993 shutdown_connections 2024-02-19T15:24:53.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.001+0000 7f6bc5400640 1 -- 172.21.15.47:0/241892993 wait complete. 2024-02-19T15:24:53.022 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:24:53.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.157+0000 7f7cd1a00640 1 Processor -- start 2024-02-19T15:24:53.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.157+0000 7f7cd1a00640 1 -- start start 2024-02-19T15:24:53.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.161+0000 7f7cd1a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7ccc0cedb0 0x7f7ccc0cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:53.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.161+0000 7f7cd1a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ccc0d2710 0x7f7ccc0d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:53.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.161+0000 7f7cd1a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ccc0cc700 con 0x7f7ccc15bcd0 2024-02-19T15:24:53.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.161+0000 7f7cd1a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ccc0cf5e0 con 0x7f7ccc0cedb0 2024-02-19T15:24:53.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.161+0000 7f7cd1a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc0cdb90 con 0x7f7ccc0d2710 2024-02-19T15:24:53.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.161+0000 7f7ccaa00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f7ccc15bcd0 0x7f7ccc0cd480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43594/0 (socket says 172.21.15.47:43594) 2024-02-19T15:24:53.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.161+0000 7f7ccaa00640 1 -- 172.21.15.47:0/3117768333 learned_addr learned my addr 172.21.15.47:0/3117768333 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:53.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.161+0000 7f7cc2000640 1 --2- 172.21.15.47:0/3117768333 >> v2:172.21.15.47:3300/0 conn(0x7f7ccc0cedb0 0x7f7ccc0cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:53.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.161+0000 7f7ccb400640 1 --2- 172.21.15.47:0/3117768333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ccc0d2710 0x7f7ccc0d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:53.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.161+0000 7f7cca000640 1 -- 172.21.15.47:0/3117768333 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368164761 0 0) 0x7f7ccc0cc700 con 0x7f7ccc15bcd0 2024-02-19T15:24:53.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.161+0000 7f7cca000640 1 -- 172.21.15.47:0/3117768333 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cb0003530 con 0x7f7ccc15bcd0 2024-02-19T15:24:53.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.161+0000 7f7cc2000640 1 -- 172.21.15.47:0/3117768333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ccc0d2710 msgr2=0x7f7ccc0d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:53.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.161+0000 7f7cc2000640 1 --2- 172.21.15.47:0/3117768333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ccc0d2710 0x7f7ccc0d0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.161+0000 7f7cc2000640 1 -- 172.21.15.47:0/3117768333 >> v1:172.21.15.47:6789/0 conn(0x7f7ccc15bcd0 legacy=0x7f7ccc0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:53.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.161+0000 7f7cc2000640 1 -- 172.21.15.47:0/3117768333 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc0cf300 con 0x7f7ccc0cedb0 2024-02-19T15:24:53.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.165+0000 7f7cc2000640 1 --2- 172.21.15.47:0/3117768333 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ccc0cedb0 0x7f7ccc0cfc50 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f7cb40097f0 tx=0x7f7cb401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc9b9970248ab46 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:53.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.165+0000 7f7cca000640 1 -- 172.21.15.47:0/3117768333 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb40ac640 con 0x7f7ccc0cedb0 2024-02-19T15:24:53.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.165+0000 7f7cca000640 1 -- 172.21.15.47:0/3117768333 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb40ac7a0 con 0x7f7ccc0cedb0 2024-02-19T15:24:53.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.165+0000 7f7cca000640 1 -- 172.21.15.47:0/3117768333 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb40acb30 con 0x7f7ccc0cedb0 2024-02-19T15:24:53.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.165+0000 7f7cd1a00640 1 -- 172.21.15.47:0/3117768333 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ccc0cedb0 msgr2=0x7f7ccc0cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:53.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.165+0000 7f7cd1a00640 1 --2- 172.21.15.47:0/3117768333 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ccc0cedb0 0x7f7ccc0cfc50 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f7cb40097f0 tx=0x7f7cb401c3e0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.165+0000 7f7cd1a00640 1 -- 172.21.15.47:0/3117768333 shutdown_connections 2024-02-19T15:24:53.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.165+0000 7f7cd1a00640 1 --2- 172.21.15.47:0/3117768333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ccc0d2710 0x7f7ccc0d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.165+0000 7f7cd1a00640 1 --2- 172.21.15.47:0/3117768333 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ccc0cedb0 0x7f7ccc0cfc50 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.165+0000 7f7cd1a00640 1 -- 172.21.15.47:0/3117768333 >> 172.21.15.47:0/3117768333 conn(0x7f7ccc0d1bf0 msgr2=0x7f7ccc0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:53.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.165+0000 7f7cd1a00640 1 -- 172.21.15.47:0/3117768333 shutdown_connections 2024-02-19T15:24:53.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.165+0000 7f7cd1a00640 1 -- 172.21.15.47:0/3117768333 wait complete. 2024-02-19T15:24:53.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7cd1a00640 1 Processor -- start 2024-02-19T15:24:53.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7cd1a00640 1 -- start start 2024-02-19T15:24:53.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7cd1a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ccc0cedb0 0x7f7ccc1fa040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:53.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7cd1a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ccc0d2710 0x7f7ccc15c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:53.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7cd1a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ccc15bcd0 0x7f7ccc158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:53.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7cd1a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc159980 con 0x7f7ccc15bcd0 2024-02-19T15:24:53.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7cd1a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc157c80 con 0x7f7ccc0cedb0 2024-02-19T15:24:53.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7cd1a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc1f6770 con 0x7f7ccc0d2710 2024-02-19T15:24:53.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7ccb400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ccc15bcd0 0x7f7ccc158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:53.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7ccb400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ccc15bcd0 0x7f7ccc158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34516/0 (socket says 172.21.15.47:34516) 2024-02-19T15:24:53.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7ccb400640 1 -- 172.21.15.47:0/1066653831 learned_addr learned my addr 172.21.15.47:0/1066653831 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:53.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7cc2000640 1 --2- 172.21.15.47:0/1066653831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ccc0d2710 0x7f7ccc15c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:53.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7ccaa00640 1 --2- 172.21.15.47:0/1066653831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ccc0cedb0 0x7f7ccc1fa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:53.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7ccb400640 1 -- 172.21.15.47:0/1066653831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ccc0d2710 msgr2=0x7f7ccc15c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:53.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7ccb400640 1 --2- 172.21.15.47:0/1066653831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ccc0d2710 0x7f7ccc15c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7ccb400640 1 -- 172.21.15.47:0/1066653831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ccc0cedb0 msgr2=0x7f7ccc1fa040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:53.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7ccb400640 1 --2- 172.21.15.47:0/1066653831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ccc0cedb0 0x7f7ccc1fa040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7ccb400640 1 -- 172.21.15.47:0/1066653831 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb4009420 con 0x7f7ccc15bcd0 2024-02-19T15:24:53.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7ccaa00640 1 --2- 172.21.15.47:0/1066653831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ccc0cedb0 0x7f7ccc1fa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:53.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7ccb400640 1 --2- 172.21.15.47:0/1066653831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ccc15bcd0 0x7f7ccc158110 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4007e40 tx=0x7f7cc400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:53.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7cc3400640 1 -- 172.21.15.47:0/1066653831 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4014620 con 0x7f7ccc15bcd0 2024-02-19T15:24:53.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7cc3400640 1 -- 172.21.15.47:0/1066653831 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc400ee90 con 0x7f7ccc15bcd0 2024-02-19T15:24:53.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7cc3400640 1 -- 172.21.15.47:0/1066653831 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc40149e0 con 0x7f7ccc15bcd0 2024-02-19T15:24:53.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7cd1a00640 1 -- 172.21.15.47:0/1066653831 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ccc15d240 con 0x7f7ccc15bcd0 2024-02-19T15:24:53.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7cd1a00640 1 -- 172.21.15.47:0/1066653831 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ccc1f6d30 con 0x7f7ccc15bcd0 2024-02-19T15:24:53.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.169+0000 7f7cd1a00640 1 -- 172.21.15.47:0/1066653831 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ccc157df0 con 0x7f7ccc15bcd0 2024-02-19T15:24:53.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.173+0000 7f7cc3400640 1 -- 172.21.15.47:0/1066653831 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7cc4014b40 con 0x7f7ccc15bcd0 2024-02-19T15:24:53.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.173+0000 7f7cc3400640 1 --2- 172.21.15.47:0/1066653831 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ca8066e40 0x7f7ca8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:53.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.173+0000 7f7cc3400640 1 -- 172.21.15.47:0/1066653831 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7cc4085380 con 0x7f7ccc15bcd0 2024-02-19T15:24:53.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.173+0000 7f7ccaa00640 1 --2- 172.21.15.47:0/1066653831 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ca8066e40 0x7f7ca8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:53.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.173+0000 7f7ccaa00640 1 --2- 172.21.15.47:0/1066653831 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ca8066e40 0x7f7ca8069300 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc005890 tx=0x7f7cbc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:53.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.181+0000 7f7cc3400640 1 -- 172.21.15.47:0/1066653831 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7cc4052160 con 0x7f7ccc15bcd0 2024-02-19T15:24:53.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.429+0000 7f7cd1a00640 1 -- 172.21.15.47:0/1066653831 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ccc075360 con 0x7f7ccc15bcd0 2024-02-19T15:24:53.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.433+0000 7f7cc3400640 1 -- 172.21.15.47:0/1066653831 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f7cc4055820 con 0x7f7ccc15bcd0 2024-02-19T15:24:53.435 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:53.436 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5384,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2825837364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2825837364},{"type":"v1","addr":"172.21.15.47:6835","nonce":2825837364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2251385237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2251385237},{"type":"v1","addr":"172.21.15.116:6837","nonce":2251385237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/2438199468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2438199468},{"type":"v1","addr":"172.21.15.116:6836","nonce":2438199468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:24:25.161496+0000","modified":"2024-02-19T15:24:28.184021+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5910":{"gid":5910,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6834/61582103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":61582103},{"type":"v1","addr":"172.21.15.47:6834","nonce":61582103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-19T15:24:53.436 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 33 2024-02-19T15:24:53.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.433+0000 7f7cd1a00640 1 -- 172.21.15.47:0/1066653831 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ca8066e40 msgr2=0x7f7ca8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:53.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.433+0000 7f7cd1a00640 1 --2- 172.21.15.47:0/1066653831 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ca8066e40 0x7f7ca8069300 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc005890 tx=0x7f7cbc009040 comp rx=0 tx=0).stop 2024-02-19T15:24:53.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.433+0000 7f7cd1a00640 1 -- 172.21.15.47:0/1066653831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ccc15bcd0 msgr2=0x7f7ccc158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:53.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.433+0000 7f7cd1a00640 1 --2- 172.21.15.47:0/1066653831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ccc15bcd0 0x7f7ccc158110 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4007e40 tx=0x7f7cc400e4b0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.433+0000 7f7cd1a00640 1 -- 172.21.15.47:0/1066653831 shutdown_connections 2024-02-19T15:24:53.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.433+0000 7f7cd1a00640 1 --2- 172.21.15.47:0/1066653831 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ca8066e40 0x7f7ca8069300 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.433+0000 7f7cd1a00640 1 --2- 172.21.15.47:0/1066653831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ccc15bcd0 0x7f7ccc158110 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.433+0000 7f7cd1a00640 1 --2- 172.21.15.47:0/1066653831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ccc0d2710 0x7f7ccc15c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.433+0000 7f7cd1a00640 1 --2- 172.21.15.47:0/1066653831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ccc0cedb0 0x7f7ccc1fa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.433+0000 7f7cd1a00640 1 -- 172.21.15.47:0/1066653831 >> 172.21.15.47:0/1066653831 conn(0x7f7ccc0d1bf0 msgr2=0x7f7ccc06fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:53.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.433+0000 7f7cd1a00640 1 -- 172.21.15.47:0/1066653831 shutdown_connections 2024-02-19T15:24:53.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.433+0000 7f7cd1a00640 1 -- 172.21.15.47:0/1066653831 wait complete. 2024-02-19T15:24:53.458 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:24:53.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.593+0000 7feed0800640 1 Processor -- start 2024-02-19T15:24:53.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.593+0000 7feed0800640 1 -- start start 2024-02-19T15:24:53.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.593+0000 7feed0800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feec806d650 0x7feec80d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:53.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.593+0000 7feed0800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feec80cded0 0x7feec80ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:53.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.593+0000 7feed0800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec80cce30 con 0x7feec80ceb40 2024-02-19T15:24:53.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.593+0000 7feed0800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feec80cefd0 con 0x7feec80cded0 2024-02-19T15:24:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.593+0000 7feed0800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feec80cd110 con 0x7feec806d650 2024-02-19T15:24:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.593+0000 7feece800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7feec80ceb40 0x7feec8161d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54692/0 (socket says 172.21.15.47:54692) 2024-02-19T15:24:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.593+0000 7feece800640 1 -- 172.21.15.47:0/179478177 learned_addr learned my addr 172.21.15.47:0/179478177 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.593+0000 7feecd400640 1 --2- 172.21.15.47:0/179478177 >> v2:172.21.15.116:3300/0 conn(0x7feec80cded0 0x7feec80ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.593+0000 7feecde00640 1 --2- 172.21.15.47:0/179478177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feec806d650 0x7feec80d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.597+0000 7feecca00640 1 -- 172.21.15.47:0/179478177 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324597714 0 0) 0x7feec80cce30 con 0x7feec80ceb40 2024-02-19T15:24:53.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.597+0000 7feecd400640 1 -- 172.21.15.47:0/179478177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feec806d650 msgr2=0x7feec80d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:53.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.597+0000 7feecd400640 1 --2- 172.21.15.47:0/179478177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feec806d650 0x7feec80d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.597+0000 7feecd400640 1 -- 172.21.15.47:0/179478177 >> v1:172.21.15.47:6789/0 conn(0x7feec80ceb40 legacy=0x7feec8161d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:53.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.597+0000 7feecd400640 1 -- 172.21.15.47:0/179478177 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec80cd3f0 con 0x7feec80cded0 2024-02-19T15:24:53.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.601+0000 7feecd400640 1 --2- 172.21.15.47:0/179478177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec80cded0 0x7feec80ce600 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7feeb80097c0 tx=0x7feeb801c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f757360e1fe299e4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.601+0000 7feecca00640 1 -- 172.21.15.47:0/179478177 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb80ab600 con 0x7feec80cded0 2024-02-19T15:24:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.601+0000 7feecca00640 1 -- 172.21.15.47:0/179478177 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb80a7050 con 0x7feec80cded0 2024-02-19T15:24:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.601+0000 7feecca00640 1 -- 172.21.15.47:0/179478177 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb80ab980 con 0x7feec80cded0 2024-02-19T15:24:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.601+0000 7feed0800640 1 -- 172.21.15.47:0/179478177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec80cded0 msgr2=0x7feec80ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.601+0000 7feed0800640 1 --2- 172.21.15.47:0/179478177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec80cded0 0x7feec80ce600 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7feeb80097c0 tx=0x7feeb801c3e0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.601+0000 7feed0800640 1 -- 172.21.15.47:0/179478177 shutdown_connections 2024-02-19T15:24:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.601+0000 7feed0800640 1 --2- 172.21.15.47:0/179478177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec80cded0 0x7feec80ce600 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.601+0000 7feed0800640 1 --2- 172.21.15.47:0/179478177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feec806d650 0x7feec80d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.601+0000 7feed0800640 1 -- 172.21.15.47:0/179478177 >> 172.21.15.47:0/179478177 conn(0x7feec80d1bf0 msgr2=0x7feec80d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:53.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.601+0000 7feed0800640 1 -- 172.21.15.47:0/179478177 shutdown_connections 2024-02-19T15:24:53.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.601+0000 7feed0800640 1 -- 172.21.15.47:0/179478177 wait complete. 2024-02-19T15:24:53.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feed0800640 1 Processor -- start 2024-02-19T15:24:53.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feed0800640 1 -- start start 2024-02-19T15:24:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feed0800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feec806d650 0x7feec80b5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feed0800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feec80cded0 0x7feec8154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feed0800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec80ceb40 0x7feec815d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feed0800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec80b67c0 con 0x7feec806d650 2024-02-19T15:24:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feed0800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec80b5d30 con 0x7feec80ceb40 2024-02-19T15:24:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feed0800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feec80b4f70 con 0x7feec80cded0 2024-02-19T15:24:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feecde00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feec806d650 0x7feec80b5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feecde00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feec806d650 0x7feec80b5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51828/0 (socket says 172.21.15.47:51828) 2024-02-19T15:24:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feecde00640 1 -- 172.21.15.47:0/2154907242 learned_addr learned my addr 172.21.15.47:0/2154907242 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:53.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feecde00640 1 -- 172.21.15.47:0/2154907242 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feec80cded0 msgr2=0x7feec8154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:53.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feecde00640 1 --2- 172.21.15.47:0/2154907242 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feec80cded0 0x7feec8154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feecde00640 1 -- 172.21.15.47:0/2154907242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec80ceb40 msgr2=0x7feec815d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:53.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feece800640 1 --2- 172.21.15.47:0/2154907242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec80ceb40 0x7feec815d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:53.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feecde00640 1 --2- 172.21.15.47:0/2154907242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec80ceb40 0x7feec815d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feecde00640 1 -- 172.21.15.47:0/2154907242 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb8009420 con 0x7feec806d650 2024-02-19T15:24:53.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feece800640 1 --2- 172.21.15.47:0/2154907242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec80ceb40 0x7feec815d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:53.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feecde00640 1 --2- 172.21.15.47:0/2154907242 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feec806d650 0x7feec80b5f90 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7feebc00e870 tx=0x7feebc00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:53.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feec2000640 1 -- 172.21.15.47:0/2154907242 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feebc009510 con 0x7feec806d650 2024-02-19T15:24:53.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feec2000640 1 -- 172.21.15.47:0/2154907242 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feebc00eec0 con 0x7feec806d650 2024-02-19T15:24:53.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feec2000640 1 -- 172.21.15.47:0/2154907242 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feebc00cb40 con 0x7feec806d650 2024-02-19T15:24:53.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feed0800640 1 -- 172.21.15.47:0/2154907242 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec80b5250 con 0x7feec806d650 2024-02-19T15:24:53.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feed0800640 1 -- 172.21.15.47:0/2154907242 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec80b5810 con 0x7feec806d650 2024-02-19T15:24:53.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.605+0000 7feed0800640 1 -- 172.21.15.47:0/2154907242 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec8075360 con 0x7feec806d650 2024-02-19T15:24:53.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.609+0000 7feec2000640 1 -- 172.21.15.47:0/2154907242 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7feebc00cca0 con 0x7feec806d650 2024-02-19T15:24:53.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.609+0000 7feec2000640 1 --2- 172.21.15.47:0/2154907242 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feeac066e40 0x7feeac069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:53.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.609+0000 7feec2000640 1 -- 172.21.15.47:0/2154907242 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feebc084b60 con 0x7feec806d650 2024-02-19T15:24:53.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.609+0000 7feecd400640 1 --2- 172.21.15.47:0/2154907242 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feeac066e40 0x7feeac069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:53.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.609+0000 7feecd400640 1 --2- 172.21.15.47:0/2154907242 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feeac066e40 0x7feeac069300 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7feeb80ad000 tx=0x7feeb8004740 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:53.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.613+0000 7feec2000640 1 -- 172.21.15.47:0/2154907242 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7feebc051940 con 0x7feec806d650 2024-02-19T15:24:53.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.789+0000 7feed0800640 1 -- 172.21.15.47:0/2154907242 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feec80cf9b0 con 0x7feec806d650 2024-02-19T15:24:53.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.793+0000 7feec2000640 1 -- 172.21.15.47:0/2154907242 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16677 (secure 0 0 0) 0x7feebc017090 con 0x7feec806d650 2024-02-19T15:24:53.795 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:53.796 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":57,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:24:26.785375+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:24:22.931622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-02-19T15:24:24.163465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:24:53.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.793+0000 7feed0800640 1 -- 172.21.15.47:0/2154907242 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feeac066e40 msgr2=0x7feeac069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:53.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.793+0000 7feed0800640 1 --2- 172.21.15.47:0/2154907242 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feeac066e40 0x7feeac069300 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7feeb80ad000 tx=0x7feeb8004740 comp rx=0 tx=0).stop 2024-02-19T15:24:53.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.793+0000 7feed0800640 1 -- 172.21.15.47:0/2154907242 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feec806d650 msgr2=0x7feec80b5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:53.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.793+0000 7feed0800640 1 --2- 172.21.15.47:0/2154907242 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feec806d650 0x7feec80b5f90 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7feebc00e870 tx=0x7feebc00ec70 comp rx=0 tx=0).stop 2024-02-19T15:24:53.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.793+0000 7feed0800640 1 -- 172.21.15.47:0/2154907242 shutdown_connections 2024-02-19T15:24:53.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.793+0000 7feed0800640 1 --2- 172.21.15.47:0/2154907242 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feeac066e40 0x7feeac069300 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.793+0000 7feed0800640 1 --2- 172.21.15.47:0/2154907242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec80ceb40 0x7feec815d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.793+0000 7feed0800640 1 --2- 172.21.15.47:0/2154907242 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feec80cded0 0x7feec8154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.793+0000 7feed0800640 1 --2- 172.21.15.47:0/2154907242 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feec806d650 0x7feec80b5f90 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.793+0000 7feed0800640 1 -- 172.21.15.47:0/2154907242 >> 172.21.15.47:0/2154907242 conn(0x7feec80d1bf0 msgr2=0x7feec8070370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:53.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.793+0000 7feed0800640 1 -- 172.21.15.47:0/2154907242 shutdown_connections 2024-02-19T15:24:53.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.793+0000 7feed0800640 1 -- 172.21.15.47:0/2154907242 wait complete. 2024-02-19T15:24:53.818 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T15:24:53.818 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:24:53.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.953+0000 7fc42dc00640 1 Processor -- start 2024-02-19T15:24:53.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.953+0000 7fc42dc00640 1 -- start start 2024-02-19T15:24:53.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.957+0000 7fc42dc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc42815bcd0 0x7fc4280cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:53.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.957+0000 7fc42dc00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc4280d0360 0x7fc42815d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:53.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.957+0000 7fc42dc00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4280cc700 con 0x7fc4280cedb0 2024-02-19T15:24:53.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.957+0000 7fc42dc00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4280cf5e0 con 0x7fc4280d0360 2024-02-19T15:24:53.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.957+0000 7fc42dc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4280cf300 con 0x7fc42815bcd0 2024-02-19T15:24:53.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.957+0000 7fc427400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc4280d0360 0x7fc42815d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:53.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.957+0000 7fc41e000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc4280cedb0 0x7fc4280cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54704/0 (socket says 172.21.15.47:54704) 2024-02-19T15:24:53.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.957+0000 7fc41e000640 1 -- 172.21.15.47:0/2632080941 learned_addr learned my addr 172.21.15.47:0/2632080941 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:53.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.957+0000 7fc426a00640 1 --2- 172.21.15.47:0/2632080941 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc42815bcd0 0x7fc4280cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:53.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.957+0000 7fc427400640 1 -- 172.21.15.47:0/2632080941 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc42815bcd0 msgr2=0x7fc4280cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:53.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.957+0000 7fc427400640 1 --2- 172.21.15.47:0/2632080941 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc42815bcd0 0x7fc4280cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.957+0000 7fc427400640 1 -- 172.21.15.47:0/2632080941 >> v1:172.21.15.47:6789/0 conn(0x7fc4280cedb0 legacy=0x7fc4280cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:53.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.957+0000 7fc427400640 1 -- 172.21.15.47:0/2632080941 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4280cc9a0 con 0x7fc4280d0360 2024-02-19T15:24:53.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.957+0000 7fc426000640 1 -- 172.21.15.47:0/2632080941 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178815580 0 0) 0x7fc4280cc700 con 0x7fc4280cedb0 2024-02-19T15:24:53.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc427400640 1 --2- 172.21.15.47:0/2632080941 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4280d0360 0x7fc42815d4f0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fc420009d90 tx=0x7fc42001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15c89e29f4ed8e23 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:53.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc426000640 1 -- 172.21.15.47:0/2632080941 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4200a5a00 con 0x7fc4280d0360 2024-02-19T15:24:53.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc426000640 1 -- 172.21.15.47:0/2632080941 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4200a5b60 con 0x7fc4280d0360 2024-02-19T15:24:53.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc426000640 1 -- 172.21.15.47:0/2632080941 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4200ad440 con 0x7fc4280d0360 2024-02-19T15:24:53.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc42dc00640 1 -- 172.21.15.47:0/2632080941 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4280d0360 msgr2=0x7fc42815d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:53.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc42dc00640 1 --2- 172.21.15.47:0/2632080941 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4280d0360 0x7fc42815d4f0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fc420009d90 tx=0x7fc42001c940 comp rx=0 tx=0).stop 2024-02-19T15:24:53.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc42dc00640 1 -- 172.21.15.47:0/2632080941 shutdown_connections 2024-02-19T15:24:53.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc42dc00640 1 --2- 172.21.15.47:0/2632080941 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4280d0360 0x7fc42815d4f0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc42dc00640 1 --2- 172.21.15.47:0/2632080941 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc42815bcd0 0x7fc4280cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc42dc00640 1 -- 172.21.15.47:0/2632080941 >> 172.21.15.47:0/2632080941 conn(0x7fc4280d1bf0 msgr2=0x7fc4280d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:53.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc42dc00640 1 -- 172.21.15.47:0/2632080941 shutdown_connections 2024-02-19T15:24:53.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc42dc00640 1 -- 172.21.15.47:0/2632080941 wait complete. 2024-02-19T15:24:53.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc42dc00640 1 Processor -- start 2024-02-19T15:24:53.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc42dc00640 1 -- start start 2024-02-19T15:24:53.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc42dc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4280cedb0 0x7fc42815aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:53.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc42dc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4280d0360 0x7fc428154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:53.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc42dc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc42815bcd0 0x7fc428157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:53.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc42dc00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc428155010 con 0x7fc42815bcd0 2024-02-19T15:24:53.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc42dc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42815a570 con 0x7fc4280d0360 2024-02-19T15:24:53.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc42dc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4280b6390 con 0x7fc4280cedb0 2024-02-19T15:24:53.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.961+0000 7fc427400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc42815bcd0 0x7fc428157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:53.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.965+0000 7fc427400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc42815bcd0 0x7fc428157360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51860/0 (socket says 172.21.15.47:51860) 2024-02-19T15:24:53.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.965+0000 7fc427400640 1 -- 172.21.15.47:0/1909621341 learned_addr learned my addr 172.21.15.47:0/1909621341 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:53.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.965+0000 7fc426a00640 1 --2- 172.21.15.47:0/1909621341 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4280cedb0 0x7fc42815aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:53.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.965+0000 7fc41e000640 1 --2- 172.21.15.47:0/1909621341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4280d0360 0x7fc428154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:53.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.965+0000 7fc427400640 1 -- 172.21.15.47:0/1909621341 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4280cedb0 msgr2=0x7fc42815aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:53.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.965+0000 7fc427400640 1 --2- 172.21.15.47:0/1909621341 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4280cedb0 0x7fc42815aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.965+0000 7fc427400640 1 -- 172.21.15.47:0/1909621341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4280d0360 msgr2=0x7fc428154ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:53.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.965+0000 7fc427400640 1 --2- 172.21.15.47:0/1909621341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4280d0360 0x7fc428154ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:53.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.965+0000 7fc427400640 1 -- 172.21.15.47:0/1909621341 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc420009a70 con 0x7fc42815bcd0 2024-02-19T15:24:53.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.965+0000 7fc427400640 1 --2- 172.21.15.47:0/1909621341 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc42815bcd0 0x7fc428157360 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fc4200b0040 tx=0x7fc4200196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:53.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.965+0000 7fc41f400640 1 -- 172.21.15.47:0/1909621341 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4200be070 con 0x7fc42815bcd0 2024-02-19T15:24:53.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.965+0000 7fc41f400640 1 -- 172.21.15.47:0/1909621341 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc420009450 con 0x7fc42815bcd0 2024-02-19T15:24:53.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.965+0000 7fc41f400640 1 -- 172.21.15.47:0/1909621341 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4200b9450 con 0x7fc42815bcd0 2024-02-19T15:24:53.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.965+0000 7fc42dc00640 1 -- 172.21.15.47:0/1909621341 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4280cdb40 con 0x7fc42815bcd0 2024-02-19T15:24:53.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.965+0000 7fc42dc00640 1 -- 172.21.15.47:0/1909621341 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4280b6950 con 0x7fc42815bcd0 2024-02-19T15:24:53.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.965+0000 7fc42dc00640 1 -- 172.21.15.47:0/1909621341 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc42815a700 con 0x7fc42815bcd0 2024-02-19T15:24:53.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.965+0000 7fc41f400640 1 -- 172.21.15.47:0/1909621341 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc4200b9670 con 0x7fc42815bcd0 2024-02-19T15:24:53.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.969+0000 7fc41f400640 1 --2- 172.21.15.47:0/1909621341 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc404066e40 0x7fc404069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:53.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.969+0000 7fc41f400640 1 -- 172.21.15.47:0/1909621341 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc42011fb30 con 0x7fc42815bcd0 2024-02-19T15:24:53.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.969+0000 7fc426a00640 1 --2- 172.21.15.47:0/1909621341 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc404066e40 0x7fc404069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:53.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.969+0000 7fc426a00640 1 --2- 172.21.15.47:0/1909621341 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc404066e40 0x7fc404069300 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc418007a60 tx=0x7fc418008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:53.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:53.973+0000 7fc41f400640 1 -- 172.21.15.47:0/1909621341 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc4200eca50 con 0x7fc42815bcd0 2024-02-19T15:24:54.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.169+0000 7fc42dc00640 1 -- 172.21.15.47:0/1909621341 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc428075360 con 0x7fc42815bcd0 2024-02-19T15:24:54.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.169+0000 7fc41f400640 1 -- 172.21.15.47:0/1909621341 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4938 (secure 0 0 0) 0x7fc4200f0110 con 0x7fc42815bcd0 2024-02-19T15:24:54.171 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:54.172 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5384,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2825837364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2825837364},{"type":"v1","addr":"172.21.15.47:6835","nonce":2825837364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2251385237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2251385237},{"type":"v1","addr":"172.21.15.116:6837","nonce":2251385237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/2438199468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2438199468},{"type":"v1","addr":"172.21.15.116:6836","nonce":2438199468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:24:25.161496+0000","modified":"2024-02-19T15:24:28.184021+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5910":{"gid":5910,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6834/61582103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":61582103},{"type":"v1","addr":"172.21.15.47:6834","nonce":61582103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-19T15:24:54.172 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 33 2024-02-19T15:24:54.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.169+0000 7fc42dc00640 1 -- 172.21.15.47:0/1909621341 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc404066e40 msgr2=0x7fc404069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:54.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.169+0000 7fc42dc00640 1 --2- 172.21.15.47:0/1909621341 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc404066e40 0x7fc404069300 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc418007a60 tx=0x7fc418008040 comp rx=0 tx=0).stop 2024-02-19T15:24:54.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.169+0000 7fc42dc00640 1 -- 172.21.15.47:0/1909621341 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc42815bcd0 msgr2=0x7fc428157360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:54.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.169+0000 7fc42dc00640 1 --2- 172.21.15.47:0/1909621341 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc42815bcd0 0x7fc428157360 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fc4200b0040 tx=0x7fc4200196e0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.169+0000 7fc42dc00640 1 -- 172.21.15.47:0/1909621341 shutdown_connections 2024-02-19T15:24:54.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.169+0000 7fc42dc00640 1 --2- 172.21.15.47:0/1909621341 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc404066e40 0x7fc404069300 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.169+0000 7fc42dc00640 1 --2- 172.21.15.47:0/1909621341 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc42815bcd0 0x7fc428157360 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.169+0000 7fc42dc00640 1 --2- 172.21.15.47:0/1909621341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4280d0360 0x7fc428154ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.169+0000 7fc42dc00640 1 --2- 172.21.15.47:0/1909621341 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4280cedb0 0x7fc42815aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.169+0000 7fc42dc00640 1 -- 172.21.15.47:0/1909621341 >> 172.21.15.47:0/1909621341 conn(0x7fc4280d1bf0 msgr2=0x7fc4280b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:54.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.169+0000 7fc42dc00640 1 -- 172.21.15.47:0/1909621341 shutdown_connections 2024-02-19T15:24:54.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.169+0000 7fc42dc00640 1 -- 172.21.15.47:0/1909621341 wait complete. 2024-02-19T15:24:54.202 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:24:54.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.337+0000 7fb435e00640 1 Processor -- start 2024-02-19T15:24:54.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.337+0000 7fb435e00640 1 -- start start 2024-02-19T15:24:54.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.341+0000 7fb435e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb43006d650 0x7fb4300d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:54.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.341+0000 7fb435e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb4300ced10 0x7fb43006bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:54.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.341+0000 7fb435e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4300cce30 con 0x7fb4300cded0 2024-02-19T15:24:54.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.341+0000 7fb435e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4300cd110 con 0x7fb4300ced10 2024-02-19T15:24:54.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.341+0000 7fb435e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4300cd3f0 con 0x7fb43006d650 2024-02-19T15:24:54.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.341+0000 7fb42f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb4300ced10 0x7fb43006bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:54.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.341+0000 7fb42f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb4300ced10 0x7fb43006bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51866/0 (socket says 172.21.15.47:51866) 2024-02-19T15:24:54.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.341+0000 7fb42f400640 1 -- 172.21.15.47:0/1758482616 learned_addr learned my addr 172.21.15.47:0/1758482616 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:54.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.341+0000 7fb42ea00640 1 --2- 172.21.15.47:0/1758482616 >> v2:172.21.15.116:3300/0 conn(0x7fb43006d650 0x7fb4300d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:54.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.341+0000 7fb42f400640 1 -- 172.21.15.47:0/1758482616 >> v2:172.21.15.116:3300/0 conn(0x7fb43006d650 msgr2=0x7fb4300d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:54.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.341+0000 7fb42f400640 1 --2- 172.21.15.47:0/1758482616 >> v2:172.21.15.116:3300/0 conn(0x7fb43006d650 0x7fb4300d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.341+0000 7fb42f400640 1 -- 172.21.15.47:0/1758482616 >> v1:172.21.15.47:6789/0 conn(0x7fb4300cded0 legacy=0x7fb4300ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:54.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.341+0000 7fb42f400640 1 -- 172.21.15.47:0/1758482616 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4300cd6d0 con 0x7fb4300ced10 2024-02-19T15:24:54.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.345+0000 7fb42f400640 1 --2- 172.21.15.47:0/1758482616 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4300ced10 0x7fb43006bd60 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fb428009e50 tx=0x7fb42801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23733156318c6fe server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:54.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.345+0000 7fb42e000640 1 -- 172.21.15.47:0/1758482616 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4280a5810 con 0x7fb4300ced10 2024-02-19T15:24:54.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.345+0000 7fb42e000640 1 -- 172.21.15.47:0/1758482616 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4280a5970 con 0x7fb4300ced10 2024-02-19T15:24:54.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.345+0000 7fb42e000640 1 -- 172.21.15.47:0/1758482616 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4280a5d00 con 0x7fb4300ced10 2024-02-19T15:24:54.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.345+0000 7fb435e00640 1 -- 172.21.15.47:0/1758482616 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4300ced10 msgr2=0x7fb43006bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:54.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.345+0000 7fb435e00640 1 --2- 172.21.15.47:0/1758482616 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4300ced10 0x7fb43006bd60 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fb428009e50 tx=0x7fb42801c930 comp rx=0 tx=0).stop 2024-02-19T15:24:54.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.345+0000 7fb435e00640 1 -- 172.21.15.47:0/1758482616 shutdown_connections 2024-02-19T15:24:54.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.345+0000 7fb435e00640 1 --2- 172.21.15.47:0/1758482616 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4300ced10 0x7fb43006bd60 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.345+0000 7fb435e00640 1 --2- 172.21.15.47:0/1758482616 >> v2:172.21.15.116:3300/0 conn(0x7fb43006d650 0x7fb4300d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.345+0000 7fb435e00640 1 -- 172.21.15.47:0/1758482616 >> 172.21.15.47:0/1758482616 conn(0x7fb4300d1bf0 msgr2=0x7fb4300d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:54.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.345+0000 7fb435e00640 1 -- 172.21.15.47:0/1758482616 shutdown_connections 2024-02-19T15:24:54.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.345+0000 7fb435e00640 1 -- 172.21.15.47:0/1758482616 wait complete. 2024-02-19T15:24:54.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.345+0000 7fb435e00640 1 Processor -- start 2024-02-19T15:24:54.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.345+0000 7fb435e00640 1 -- start start 2024-02-19T15:24:54.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb435e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43006d650 0x7fb4300ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:54.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb435e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4300cded0 0x7fb430159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:54.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb435e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4300ced10 0x7fb4300b6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:54.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb435e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4300cbe50 con 0x7fb4300ced10 2024-02-19T15:24:54.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb435e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4300cb240 con 0x7fb4300cded0 2024-02-19T15:24:54.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb435e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4300cb520 con 0x7fb43006d650 2024-02-19T15:24:54.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb42f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4300ced10 0x7fb4300b6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:54.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb42f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4300ced10 0x7fb4300b6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51880/0 (socket says 172.21.15.47:51880) 2024-02-19T15:24:54.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb42f400640 1 -- 172.21.15.47:0/173928288 learned_addr learned my addr 172.21.15.47:0/173928288 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:54.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb42ea00640 1 --2- 172.21.15.47:0/173928288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43006d650 0x7fb4300ca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:54.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb427e00640 1 --2- 172.21.15.47:0/173928288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4300cded0 0x7fb430159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:54.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb42f400640 1 -- 172.21.15.47:0/173928288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43006d650 msgr2=0x7fb4300ca8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:54.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb42f400640 1 --2- 172.21.15.47:0/173928288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43006d650 0x7fb4300ca8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb42f400640 1 -- 172.21.15.47:0/173928288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4300cded0 msgr2=0x7fb430159370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:54.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb42f400640 1 --2- 172.21.15.47:0/173928288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4300cded0 0x7fb430159370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb42f400640 1 -- 172.21.15.47:0/173928288 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb428009b30 con 0x7fb4300ced10 2024-02-19T15:24:54.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb427e00640 1 --2- 172.21.15.47:0/173928288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4300cded0 0x7fb430159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:54.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb42f400640 1 --2- 172.21.15.47:0/173928288 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4300ced10 0x7fb4300b6bd0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fb428019540 tx=0x7fb4280196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:54.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb427400640 1 -- 172.21.15.47:0/173928288 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4280b8670 con 0x7fb4300ced10 2024-02-19T15:24:54.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb427400640 1 -- 172.21.15.47:0/173928288 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4280b87f0 con 0x7fb4300ced10 2024-02-19T15:24:54.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb435e00640 1 -- 172.21.15.47:0/173928288 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4300cb800 con 0x7fb4300ced10 2024-02-19T15:24:54.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb427400640 1 -- 172.21.15.47:0/173928288 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4280b8ba0 con 0x7fb4300ced10 2024-02-19T15:24:54.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb435e00640 1 -- 172.21.15.47:0/173928288 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4300cae10 con 0x7fb4300ced10 2024-02-19T15:24:54.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb435e00640 1 -- 172.21.15.47:0/173928288 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4300cb000 con 0x7fb4300ced10 2024-02-19T15:24:54.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.349+0000 7fb427400640 1 -- 172.21.15.47:0/173928288 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb4280b8d00 con 0x7fb4300ced10 2024-02-19T15:24:54.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.353+0000 7fb427400640 1 --2- 172.21.15.47:0/173928288 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb40c066e90 0x7fb40c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:54.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.353+0000 7fb427400640 1 -- 172.21.15.47:0/173928288 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb42811ff70 con 0x7fb4300ced10 2024-02-19T15:24:54.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.353+0000 7fb42ea00640 1 --2- 172.21.15.47:0/173928288 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb40c066e90 0x7fb40c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:54.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.353+0000 7fb42ea00640 1 --2- 172.21.15.47:0/173928288 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb40c066e90 0x7fb40c069350 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb4300b78c0 tx=0x7fb420008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:54.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.357+0000 7fb427400640 1 -- 172.21.15.47:0/173928288 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb4300cb000 con 0x7fb4300ced10 2024-02-19T15:24:54.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.553+0000 7fb435e00640 1 -- 172.21.15.47:0/173928288 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4300cb000 con 0x7fb4300ced10 2024-02-19T15:24:54.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.553+0000 7fb427400640 1 -- 172.21.15.47:0/173928288 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4938 (secure 0 0 0) 0x7fb4300cb000 con 0x7fb4300ced10 2024-02-19T15:24:54.558 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:54.558 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5384,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2825837364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2825837364},{"type":"v1","addr":"172.21.15.47:6835","nonce":2825837364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2251385237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2251385237},{"type":"v1","addr":"172.21.15.116:6837","nonce":2251385237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/2438199468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2438199468},{"type":"v1","addr":"172.21.15.116:6836","nonce":2438199468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:24:25.161496+0000","modified":"2024-02-19T15:24:28.184021+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5910":{"gid":5910,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6834/61582103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":61582103},{"type":"v1","addr":"172.21.15.47:6834","nonce":61582103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-19T15:24:54.559 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 33 2024-02-19T15:24:54.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.557+0000 7fb435e00640 1 -- 172.21.15.47:0/173928288 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb40c066e90 msgr2=0x7fb40c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:54.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.557+0000 7fb435e00640 1 --2- 172.21.15.47:0/173928288 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb40c066e90 0x7fb40c069350 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb4300b78c0 tx=0x7fb420008040 comp rx=0 tx=0).stop 2024-02-19T15:24:54.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.557+0000 7fb435e00640 1 -- 172.21.15.47:0/173928288 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4300ced10 msgr2=0x7fb4300b6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:54.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.557+0000 7fb435e00640 1 --2- 172.21.15.47:0/173928288 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4300ced10 0x7fb4300b6bd0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fb428019540 tx=0x7fb4280196d0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.557+0000 7fb435e00640 1 -- 172.21.15.47:0/173928288 shutdown_connections 2024-02-19T15:24:54.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.557+0000 7fb435e00640 1 --2- 172.21.15.47:0/173928288 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb40c066e90 0x7fb40c069350 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.557+0000 7fb435e00640 1 --2- 172.21.15.47:0/173928288 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4300ced10 0x7fb4300b6bd0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.557+0000 7fb435e00640 1 --2- 172.21.15.47:0/173928288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4300cded0 0x7fb430159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.557+0000 7fb435e00640 1 --2- 172.21.15.47:0/173928288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43006d650 0x7fb4300ca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.557+0000 7fb435e00640 1 -- 172.21.15.47:0/173928288 >> 172.21.15.47:0/173928288 conn(0x7fb4300d1bf0 msgr2=0x7fb4300ce5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:54.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.557+0000 7fb435e00640 1 -- 172.21.15.47:0/173928288 shutdown_connections 2024-02-19T15:24:54.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.557+0000 7fb435e00640 1 -- 172.21.15.47:0/173928288 wait complete. 2024-02-19T15:24:54.579 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:24:54.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.717+0000 7f7b83e00640 1 Processor -- start 2024-02-19T15:24:54.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.717+0000 7f7b83e00640 1 -- start start 2024-02-19T15:24:54.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.717+0000 7f7b83e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b7c0cedb0 0x7f7b7c0cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:54.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.717+0000 7f7b83e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7b7c0d2710 0x7f7b7c0d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:54.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.717+0000 7f7b83e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c0cc700 con 0x7f7b7c15bcd0 2024-02-19T15:24:54.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.717+0000 7f7b83e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b7c0cf5e0 con 0x7f7b7c0d2710 2024-02-19T15:24:54.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.717+0000 7f7b83e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c0cdb90 con 0x7f7b7c0cedb0 2024-02-19T15:24:54.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.717+0000 7f7b81e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7b7c0d2710 0x7f7b7c0d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:54.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.717+0000 7f7b81400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f7b7c15bcd0 0x7f7b7c0cd480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54716/0 (socket says 172.21.15.47:54716) 2024-02-19T15:24:54.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.717+0000 7f7b81e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7b7c0d2710 0x7f7b7c0d0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51884/0 (socket says 172.21.15.47:51884) 2024-02-19T15:24:54.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.717+0000 7f7b81400640 1 -- 172.21.15.47:0/657200553 learned_addr learned my addr 172.21.15.47:0/657200553 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:54.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.717+0000 7f7b80a00640 1 --2- 172.21.15.47:0/657200553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b7c0cedb0 0x7f7b7c0cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:54.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.717+0000 7f7b81e00640 1 -- 172.21.15.47:0/657200553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b7c0cedb0 msgr2=0x7f7b7c0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:54.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.717+0000 7f7b81e00640 1 --2- 172.21.15.47:0/657200553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b7c0cedb0 0x7f7b7c0cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.717+0000 7f7b81e00640 1 -- 172.21.15.47:0/657200553 >> v1:172.21.15.47:6789/0 conn(0x7f7b7c15bcd0 legacy=0x7f7b7c0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:54.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.717+0000 7f7b81e00640 1 -- 172.21.15.47:0/657200553 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b7c0cf300 con 0x7f7b7c0d2710 2024-02-19T15:24:54.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.717+0000 7f7b7be00640 1 -- 172.21.15.47:0/657200553 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894628443 0 0) 0x7f7b7c0cc700 con 0x7f7b7c15bcd0 2024-02-19T15:24:54.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.721+0000 7f7b81e00640 1 --2- 172.21.15.47:0/657200553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0d2710 0x7f7b7c0d0190 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009d90 tx=0x7f7b7401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca22ff0056c7a0e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:54.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.721+0000 7f7b7be00640 1 -- 172.21.15.47:0/657200553 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b740a5a00 con 0x7f7b7c0d2710 2024-02-19T15:24:54.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.721+0000 7f7b7be00640 1 -- 172.21.15.47:0/657200553 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b740a5b60 con 0x7f7b7c0d2710 2024-02-19T15:24:54.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.721+0000 7f7b7be00640 1 -- 172.21.15.47:0/657200553 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b740ad440 con 0x7f7b7c0d2710 2024-02-19T15:24:54.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.721+0000 7f7b83e00640 1 -- 172.21.15.47:0/657200553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0d2710 msgr2=0x7f7b7c0d0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:54.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.721+0000 7f7b83e00640 1 --2- 172.21.15.47:0/657200553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0d2710 0x7f7b7c0d0190 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009d90 tx=0x7f7b7401c940 comp rx=0 tx=0).stop 2024-02-19T15:24:54.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.721+0000 7f7b83e00640 1 -- 172.21.15.47:0/657200553 shutdown_connections 2024-02-19T15:24:54.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.721+0000 7f7b83e00640 1 --2- 172.21.15.47:0/657200553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0d2710 0x7f7b7c0d0190 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.721+0000 7f7b83e00640 1 --2- 172.21.15.47:0/657200553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b7c0cedb0 0x7f7b7c0cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.721+0000 7f7b83e00640 1 -- 172.21.15.47:0/657200553 >> 172.21.15.47:0/657200553 conn(0x7f7b7c0d1bf0 msgr2=0x7f7b7c0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:54.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.721+0000 7f7b83e00640 1 -- 172.21.15.47:0/657200553 shutdown_connections 2024-02-19T15:24:54.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.721+0000 7f7b83e00640 1 -- 172.21.15.47:0/657200553 wait complete. 2024-02-19T15:24:54.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.721+0000 7f7b83e00640 1 Processor -- start 2024-02-19T15:24:54.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.721+0000 7f7b83e00640 1 -- start start 2024-02-19T15:24:54.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b83e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b7c0cedb0 0x7f7b7c156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:54.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b83e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0d2710 0x7f7b7c0c7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:54.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b83e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b7c15bcd0 0x7f7b7c0c7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:54.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b83e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c0ca7c0 con 0x7f7b7c0d2710 2024-02-19T15:24:54.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b83e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c155a00 con 0x7f7b7c0cedb0 2024-02-19T15:24:54.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b83e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c0c9bb0 con 0x7f7b7c15bcd0 2024-02-19T15:24:54.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b80a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0d2710 0x7f7b7c0c7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:54.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b80a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0d2710 0x7f7b7c0c7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51890/0 (socket says 172.21.15.47:51890) 2024-02-19T15:24:54.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b80a00640 1 -- 172.21.15.47:0/2185804434 learned_addr learned my addr 172.21.15.47:0/2185804434 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:54.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b81e00640 1 --2- 172.21.15.47:0/2185804434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b7c15bcd0 0x7f7b7c0c7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:54.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b81400640 1 --2- 172.21.15.47:0/2185804434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b7c0cedb0 0x7f7b7c156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:54.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b80a00640 1 -- 172.21.15.47:0/2185804434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b7c15bcd0 msgr2=0x7f7b7c0c7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:54.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b80a00640 1 --2- 172.21.15.47:0/2185804434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b7c15bcd0 0x7f7b7c0c7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b80a00640 1 -- 172.21.15.47:0/2185804434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b7c0cedb0 msgr2=0x7f7b7c156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:54.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b80a00640 1 --2- 172.21.15.47:0/2185804434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b7c0cedb0 0x7f7b7c156d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b80a00640 1 -- 172.21.15.47:0/2185804434 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b74009a70 con 0x7f7b7c0d2710 2024-02-19T15:24:54.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b81e00640 1 --2- 172.21.15.47:0/2185804434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b7c15bcd0 0x7f7b7c0c7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:54.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b80a00640 1 --2- 172.21.15.47:0/2185804434 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0d2710 0x7f7b7c0c7a40 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7b6800d800 tx=0x7f7b6800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:54.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b79600640 1 -- 172.21.15.47:0/2185804434 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6800b6c0 con 0x7f7b7c0d2710 2024-02-19T15:24:54.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b79600640 1 -- 172.21.15.47:0/2185804434 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6800b820 con 0x7f7b7c0d2710 2024-02-19T15:24:54.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b79600640 1 -- 172.21.15.47:0/2185804434 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6800bb90 con 0x7f7b7c0d2710 2024-02-19T15:24:54.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b83e00640 1 -- 172.21.15.47:0/2185804434 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7c15d240 con 0x7f7b7c0d2710 2024-02-19T15:24:54.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b83e00640 1 -- 172.21.15.47:0/2185804434 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7c0ca170 con 0x7f7b7c0d2710 2024-02-19T15:24:54.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b83e00640 1 -- 172.21.15.47:0/2185804434 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7c075360 con 0x7f7b7c0d2710 2024-02-19T15:24:54.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.725+0000 7f7b79600640 1 -- 172.21.15.47:0/2185804434 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7b6800bcf0 con 0x7f7b7c0d2710 2024-02-19T15:24:54.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.729+0000 7f7b79600640 1 --2- 172.21.15.47:0/2185804434 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7b5c066e40 0x7f7b5c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:54.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.729+0000 7f7b79600640 1 -- 172.21.15.47:0/2185804434 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b68084f80 con 0x7f7b7c0d2710 2024-02-19T15:24:54.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.729+0000 7f7b81400640 1 --2- 172.21.15.47:0/2185804434 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7b5c066e40 0x7f7b5c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:54.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.729+0000 7f7b81400640 1 --2- 172.21.15.47:0/2185804434 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7b5c066e40 0x7f7b5c069300 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7b70000f70 tx=0x7f7b70007c10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:54.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.733+0000 7f7b79600640 1 -- 172.21.15.47:0/2185804434 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7b68051d60 con 0x7f7b7c0d2710 2024-02-19T15:24:54.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.893+0000 7f7b83e00640 1 -- 172.21.15.47:0/2185804434 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b7c0cbcf0 con 0x7f7b7c0d2710 2024-02-19T15:24:54.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.897+0000 7f7b79600640 1 -- 172.21.15.47:0/2185804434 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16677 (secure 0 0 0) 0x7f7b68011340 con 0x7f7b7c0d2710 2024-02-19T15:24:54.900 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:24:54.900 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":57,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:24:26.785375+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:24:22.931622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-02-19T15:24:24.163465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:24:54.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.897+0000 7f7b83e00640 1 -- 172.21.15.47:0/2185804434 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7b5c066e40 msgr2=0x7f7b5c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:54.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.897+0000 7f7b83e00640 1 --2- 172.21.15.47:0/2185804434 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7b5c066e40 0x7f7b5c069300 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7b70000f70 tx=0x7f7b70007c10 comp rx=0 tx=0).stop 2024-02-19T15:24:54.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.897+0000 7f7b83e00640 1 -- 172.21.15.47:0/2185804434 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0d2710 msgr2=0x7f7b7c0c7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:54.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.897+0000 7f7b83e00640 1 --2- 172.21.15.47:0/2185804434 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0d2710 0x7f7b7c0c7a40 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7b6800d800 tx=0x7f7b6800dcc0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.897+0000 7f7b83e00640 1 -- 172.21.15.47:0/2185804434 shutdown_connections 2024-02-19T15:24:54.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.897+0000 7f7b83e00640 1 --2- 172.21.15.47:0/2185804434 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7b5c066e40 0x7f7b5c069300 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.897+0000 7f7b83e00640 1 --2- 172.21.15.47:0/2185804434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b7c15bcd0 0x7f7b7c0c7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.897+0000 7f7b83e00640 1 --2- 172.21.15.47:0/2185804434 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b7c0d2710 0x7f7b7c0c7a40 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.897+0000 7f7b83e00640 1 --2- 172.21.15.47:0/2185804434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b7c0cedb0 0x7f7b7c156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:54.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.897+0000 7f7b83e00640 1 -- 172.21.15.47:0/2185804434 >> 172.21.15.47:0/2185804434 conn(0x7f7b7c0d1bf0 msgr2=0x7f7b7c06eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:54.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.897+0000 7f7b83e00640 1 -- 172.21.15.47:0/2185804434 shutdown_connections 2024-02-19T15:24:54.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:54.897+0000 7f7b83e00640 1 -- 172.21.15.47:0/2185804434 wait complete. 2024-02-19T15:24:54.921 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T15:24:55.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.057+0000 7f168c200640 1 Processor -- start 2024-02-19T15:24:55.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.057+0000 7f168c200640 1 -- start start 2024-02-19T15:24:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.061+0000 7f168c200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1684145590 0x7f16840727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.061+0000 7f168c200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f16840cea80 0x7f168406c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.061+0000 7f168c200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f168415c390 0x7f168415c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.061+0000 7f168c200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f168415d280 con 0x7f168415c390 2024-02-19T15:24:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.061+0000 7f168c200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1684070ff0 con 0x7f16840cea80 2024-02-19T15:24:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.061+0000 7f168c200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f168415cd60 con 0x7f1684145590 2024-02-19T15:24:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.061+0000 7f168a200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f168415c390 0x7f168415c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.061+0000 7f168a200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f168415c390 0x7f168415c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51900/0 (socket says 172.21.15.47:51900) 2024-02-19T15:24:55.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.061+0000 7f168a200640 1 -- 172.21.15.47:0/1468716798 learned_addr learned my addr 172.21.15.47:0/1468716798 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:55.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.061+0000 7f1689800640 1 --2- 172.21.15.47:0/1468716798 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1684145590 0x7f16840727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:55.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.061+0000 7f1688e00640 1 --2- 172.21.15.47:0/1468716798 >> v2:172.21.15.116:3300/0 conn(0x7f16840cea80 0x7f168406c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:55.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.061+0000 7f168a200640 1 -- 172.21.15.47:0/1468716798 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1684145590 msgr2=0x7f16840727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:55.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.061+0000 7f168a200640 1 --2- 172.21.15.47:0/1468716798 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1684145590 0x7f16840727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:55.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.061+0000 7f168a200640 1 -- 172.21.15.47:0/1468716798 >> v2:172.21.15.116:3300/0 conn(0x7f16840cea80 msgr2=0x7f168406c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:55.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.061+0000 7f168a200640 1 --2- 172.21.15.47:0/1468716798 >> v2:172.21.15.116:3300/0 conn(0x7f16840cea80 0x7f168406c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:55.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.061+0000 7f168a200640 1 -- 172.21.15.47:0/1468716798 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16840709e0 con 0x7f168415c390 2024-02-19T15:24:55.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.061+0000 7f1688e00640 1 --2- 172.21.15.47:0/1468716798 >> v2:172.21.15.116:3300/0 conn(0x7f16840cea80 0x7f168406c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:55.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f168a200640 1 --2- 172.21.15.47:0/1468716798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f168415c390 0x7f168415c820 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f167c009e50 tx=0x7f167c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bffc11297bbfddf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:55.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f1683e00640 1 -- 172.21.15.47:0/1468716798 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167c0a5780 con 0x7f168415c390 2024-02-19T15:24:55.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f1683e00640 1 -- 172.21.15.47:0/1468716798 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167c0a58e0 con 0x7f168415c390 2024-02-19T15:24:55.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f1683e00640 1 -- 172.21.15.47:0/1468716798 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167c0a5c70 con 0x7f168415c390 2024-02-19T15:24:55.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f168c200640 1 -- 172.21.15.47:0/1468716798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f168415c390 msgr2=0x7f168415c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:55.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f168c200640 1 --2- 172.21.15.47:0/1468716798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f168415c390 0x7f168415c820 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f167c009e50 tx=0x7f167c01c9f0 comp rx=0 tx=0).stop 2024-02-19T15:24:55.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f168c200640 1 -- 172.21.15.47:0/1468716798 shutdown_connections 2024-02-19T15:24:55.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f168c200640 1 --2- 172.21.15.47:0/1468716798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f168415c390 0x7f168415c820 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:55.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f168c200640 1 --2- 172.21.15.47:0/1468716798 >> v2:172.21.15.116:3300/0 conn(0x7f16840cea80 0x7f168406c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:55.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f168c200640 1 --2- 172.21.15.47:0/1468716798 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1684145590 0x7f16840727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:55.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f168c200640 1 -- 172.21.15.47:0/1468716798 >> 172.21.15.47:0/1468716798 conn(0x7f16840b16a0 msgr2=0x7f16840b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:55.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f168c200640 1 -- 172.21.15.47:0/1468716798 shutdown_connections 2024-02-19T15:24:55.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f168c200640 1 -- 172.21.15.47:0/1468716798 wait complete. 2024-02-19T15:24:55.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f168c200640 1 Processor -- start 2024-02-19T15:24:55.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f168c200640 1 -- start start 2024-02-19T15:24:55.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f168c200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16840cea80 0x7f168415b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:55.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f168c200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1684145590 0x7f16841f6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:55.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f168c200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f168415c390 0x7f16840b36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:55.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f168c200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168415bcc0 con 0x7f16840cea80 2024-02-19T15:24:55.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f168c200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16840ae2f0 con 0x7f1684145590 2024-02-19T15:24:55.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.065+0000 7f168c200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16840ad9c0 con 0x7f168415c390 2024-02-19T15:24:55.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.069+0000 7f1689800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16840cea80 0x7f168415b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:55.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.069+0000 7f1689800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16840cea80 0x7f168415b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51916/0 (socket says 172.21.15.47:51916) 2024-02-19T15:24:55.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.069+0000 7f1689800640 1 -- 172.21.15.47:0/3838783836 learned_addr learned my addr 172.21.15.47:0/3838783836 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:55.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.069+0000 7f1689800640 1 -- 172.21.15.47:0/3838783836 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f168415c390 msgr2=0x7f16840b36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:55.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.069+0000 7f1689800640 1 --2- 172.21.15.47:0/3838783836 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f168415c390 0x7f16840b36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:55.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.069+0000 7f1689800640 1 -- 172.21.15.47:0/3838783836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1684145590 msgr2=0x7f16841f6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:55.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.069+0000 7f1688e00640 1 --2- 172.21.15.47:0/3838783836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1684145590 0x7f16841f6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:55.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.069+0000 7f1689800640 1 --2- 172.21.15.47:0/3838783836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1684145590 0x7f16841f6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.069+0000 7f1689800640 1 -- 172.21.15.47:0/3838783836 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c009b30 con 0x7f16840cea80 2024-02-19T15:24:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.069+0000 7f1689800640 1 --2- 172.21.15.47:0/3838783836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16840cea80 0x7f168415b780 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f167800e930 tx=0x7f167800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.069+0000 7f1681600640 1 -- 172.21.15.47:0/3838783836 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1678009690 con 0x7f16840cea80 2024-02-19T15:24:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.069+0000 7f1681600640 1 -- 172.21.15.47:0/3838783836 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167800ca00 con 0x7f16840cea80 2024-02-19T15:24:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.069+0000 7f1681600640 1 -- 172.21.15.47:0/3838783836 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167800cd70 con 0x7f16840cea80 2024-02-19T15:24:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.069+0000 7f168c200640 1 -- 172.21.15.47:0/3838783836 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16840adca0 con 0x7f16840cea80 2024-02-19T15:24:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.069+0000 7f168c200640 1 -- 172.21.15.47:0/3838783836 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16840b0d80 con 0x7f16840cea80 2024-02-19T15:24:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.069+0000 7f168c200640 1 -- 172.21.15.47:0/3838783836 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1684075360 con 0x7f16840cea80 2024-02-19T15:24:55.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.069+0000 7f1681600640 1 -- 172.21.15.47:0/3838783836 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1678013470 con 0x7f16840cea80 2024-02-19T15:24:55.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.069+0000 7f1681600640 1 --2- 172.21.15.47:0/3838783836 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1664066e40 0x7f1664069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:55.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.073+0000 7f1681600640 1 -- 172.21.15.47:0/3838783836 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1678084bf0 con 0x7f16840cea80 2024-02-19T15:24:55.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.073+0000 7f1688e00640 1 --2- 172.21.15.47:0/3838783836 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1664066e40 0x7f1664069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:55.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.073+0000 7f1688e00640 1 --2- 172.21.15.47:0/3838783836 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1664066e40 0x7f1664069300 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f167000d890 tx=0x7f1670008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:55.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.077+0000 7f1681600640 1 -- 172.21.15.47:0/3838783836 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1678051b30 con 0x7f16840cea80 2024-02-19T15:24:55.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.489+0000 7f168c200640 1 -- 172.21.15.47:0/3838783836 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f168406cc70 con 0x7f16840cea80 2024-02-19T15:24:55.495 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-02-19T15:24:55.493+0000 7fc100600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T15:24:55.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.493+0000 7f1681600640 1 -- 172.21.15.47:0/3838783836 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v34) v1 ==== 161+0+0 (secure 0 0 0) 0x7f1678017090 con 0x7f16840cea80 2024-02-19T15:24:55.497 INFO:teuthology.orchestra.run.smithi047.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T15:24:55.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.497+0000 7f168c200640 1 -- 172.21.15.47:0/3838783836 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1664066e40 msgr2=0x7f1664069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:55.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.497+0000 7f168c200640 1 --2- 172.21.15.47:0/3838783836 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1664066e40 0x7f1664069300 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f167000d890 tx=0x7f1670008040 comp rx=0 tx=0).stop 2024-02-19T15:24:55.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.497+0000 7f168c200640 1 -- 172.21.15.47:0/3838783836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16840cea80 msgr2=0x7f168415b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:55.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.497+0000 7f168c200640 1 --2- 172.21.15.47:0/3838783836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16840cea80 0x7f168415b780 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f167800e930 tx=0x7f167800edf0 comp rx=0 tx=0).stop 2024-02-19T15:24:55.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.497+0000 7f168c200640 1 -- 172.21.15.47:0/3838783836 shutdown_connections 2024-02-19T15:24:55.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.497+0000 7f168c200640 1 --2- 172.21.15.47:0/3838783836 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1664066e40 0x7f1664069300 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:55.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.497+0000 7f168c200640 1 --2- 172.21.15.47:0/3838783836 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f168415c390 0x7f16840b36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:55.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.497+0000 7f168c200640 1 --2- 172.21.15.47:0/3838783836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1684145590 0x7f16841f6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:55.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.497+0000 7f168c200640 1 --2- 172.21.15.47:0/3838783836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16840cea80 0x7f168415b780 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:55.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.497+0000 7f168c200640 1 -- 172.21.15.47:0/3838783836 >> 172.21.15.47:0/3838783836 conn(0x7f16840b16a0 msgr2=0x7f16840af510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:55.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.497+0000 7f168c200640 1 -- 172.21.15.47:0/3838783836 shutdown_connections 2024-02-19T15:24:55.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.497+0000 7f168c200640 1 -- 172.21.15.47:0/3838783836 wait complete. 2024-02-19T15:24:55.521 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T15:24:55.569 INFO:tasks.ceph.mds.a.smithi047.stdout:starting mds.a at 2024-02-19T15:24:55.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.661+0000 7fe1acc00640 1 Processor -- start 2024-02-19T15:24:55.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.661+0000 7fe1acc00640 1 -- start start 2024-02-19T15:24:55.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.661+0000 7fe1acc00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe1a80d0f40 0x7fe1a8072f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:55.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.661+0000 7fe1acc00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a8073570 con 0x7fe1a806b110 2024-02-19T15:24:55.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.661+0000 7fe1acc00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a80b0210 con 0x7fe1a815e690 2024-02-19T15:24:55.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.661+0000 7fe1acc00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a80b0490 con 0x7fe1a80d0f40 2024-02-19T15:24:55.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.661+0000 7fe1a6800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe1a80d0f40 0x7fe1a8072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:55.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.661+0000 7fe1a6800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe1a80d0f40 0x7fe1a8072f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51932/0 (socket says 172.21.15.47:51932) 2024-02-19T15:24:55.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.661+0000 7fe1a5e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fe1a806b110 0x7fe1a815df80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54738/0 (socket says 172.21.15.47:54738) 2024-02-19T15:24:55.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.661+0000 7fe1a6800640 1 -- 172.21.15.47:0/2208064594 learned_addr learned my addr 172.21.15.47:0/2208064594 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:55.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.661+0000 7fe1a6800640 1 -- 172.21.15.47:0/2208064594 >> v1:172.21.15.116:6789/0 conn(0x7fe1a815e690 legacy=0x7fe1a80b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:55.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.661+0000 7fe1a6800640 1 -- 172.21.15.47:0/2208064594 >> v1:172.21.15.47:6789/0 conn(0x7fe1a806b110 legacy=0x7fe1a815df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:55.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.661+0000 7fe1a6800640 1 -- 172.21.15.47:0/2208064594 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a8066e00 con 0x7fe1a80d0f40 2024-02-19T15:24:55.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.665+0000 7fe1a6800640 1 --2- 172.21.15.47:0/2208064594 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1a80d0f40 0x7fe1a8072f40 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0009870 tx=0x7fe1a001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d936be55cd4f703 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:55.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.665+0000 7fe1a5400640 1 -- 172.21.15.47:0/2208064594 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a00ab600 con 0x7fe1a80d0f40 2024-02-19T15:24:55.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.665+0000 7fe1a5400640 1 -- 172.21.15.47:0/2208064594 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a00ab760 con 0x7fe1a80d0f40 2024-02-19T15:24:55.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.665+0000 7fe1a5400640 1 -- 172.21.15.47:0/2208064594 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a00abaf0 con 0x7fe1a80d0f40 2024-02-19T15:24:55.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.665+0000 7fe1acc00640 1 -- 172.21.15.47:0/2208064594 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1a80d0f40 msgr2=0x7fe1a8072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:55.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.665+0000 7fe1acc00640 1 --2- 172.21.15.47:0/2208064594 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1a80d0f40 0x7fe1a8072f40 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0009870 tx=0x7fe1a001c440 comp rx=0 tx=0).stop 2024-02-19T15:24:55.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.665+0000 7fe1acc00640 1 -- 172.21.15.47:0/2208064594 shutdown_connections 2024-02-19T15:24:55.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.665+0000 7fe1acc00640 1 --2- 172.21.15.47:0/2208064594 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1a80d0f40 0x7fe1a8072f40 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:55.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.665+0000 7fe1acc00640 1 -- 172.21.15.47:0/2208064594 >> 172.21.15.47:0/2208064594 conn(0x7fe1a80d0420 msgr2=0x7fe1a80d0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:55.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.665+0000 7fe1acc00640 1 -- 172.21.15.47:0/2208064594 shutdown_connections 2024-02-19T15:24:55.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.665+0000 7fe1acc00640 1 -- 172.21.15.47:0/2208064594 wait complete. 2024-02-19T15:24:55.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.669+0000 7fe1acc00640 1 Processor -- start 2024-02-19T15:24:55.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.669+0000 7fe1acc00640 1 -- start start 2024-02-19T15:24:55.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.669+0000 7fe1acc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1a806b110 0x7fe1a80af650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:55.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.669+0000 7fe1acc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1a80d0f40 0x7fe1a81672c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:55.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.669+0000 7fe1acc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a815e690 0x7fe1a8168950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:55.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.669+0000 7fe1acc00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a8169ae0 con 0x7fe1a80d0f40 2024-02-19T15:24:55.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.669+0000 7fe1acc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a80afd50 con 0x7fe1a815e690 2024-02-19T15:24:55.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.669+0000 7fe1acc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a8167b90 con 0x7fe1a806b110 2024-02-19T15:24:55.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.669+0000 7fe19d400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1a80d0f40 0x7fe1a81672c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:55.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.669+0000 7fe19d400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1a80d0f40 0x7fe1a81672c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51936/0 (socket says 172.21.15.47:51936) 2024-02-19T15:24:55.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.669+0000 7fe19d400640 1 -- 172.21.15.47:0/3742850329 learned_addr learned my addr 172.21.15.47:0/3742850329 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:55.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.669+0000 7fe1a5e00640 1 --2- 172.21.15.47:0/3742850329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1a806b110 0x7fe1a80af650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:55.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.673+0000 7fe1a6800640 1 --2- 172.21.15.47:0/3742850329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a815e690 0x7fe1a8168950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:55.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.673+0000 7fe19d400640 1 -- 172.21.15.47:0/3742850329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1a806b110 msgr2=0x7fe1a80af650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:55.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.673+0000 7fe19d400640 1 --2- 172.21.15.47:0/3742850329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1a806b110 0x7fe1a80af650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:55.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.673+0000 7fe19d400640 1 -- 172.21.15.47:0/3742850329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a815e690 msgr2=0x7fe1a8168950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:55.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.673+0000 7fe19d400640 1 --2- 172.21.15.47:0/3742850329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a815e690 0x7fe1a8168950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:55.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.673+0000 7fe19d400640 1 -- 172.21.15.47:0/3742850329 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a00094d0 con 0x7fe1a80d0f40 2024-02-19T15:24:55.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.673+0000 7fe19d400640 1 --2- 172.21.15.47:0/3742850329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1a80d0f40 0x7fe1a81672c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe19000c4d0 tx=0x7fe19000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:55.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.673+0000 7fe19ea00640 1 -- 172.21.15.47:0/3742850329 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1900052d0 con 0x7fe1a80d0f40 2024-02-19T15:24:55.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.673+0000 7fe19ea00640 1 -- 172.21.15.47:0/3742850329 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe190005430 con 0x7fe1a80d0f40 2024-02-19T15:24:55.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.673+0000 7fe19ea00640 1 -- 172.21.15.47:0/3742850329 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe190012830 con 0x7fe1a80d0f40 2024-02-19T15:24:55.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.673+0000 7fe1acc00640 1 -- 172.21.15.47:0/3742850329 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a8167e70 con 0x7fe1a80d0f40 2024-02-19T15:24:55.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.673+0000 7fe1acc00640 1 -- 172.21.15.47:0/3742850329 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a8168430 con 0x7fe1a80d0f40 2024-02-19T15:24:55.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.673+0000 7fe1acc00640 1 -- 172.21.15.47:0/3742850329 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a8168620 con 0x7fe1a80d0f40 2024-02-19T15:24:55.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.673+0000 7fe19ea00640 1 -- 172.21.15.47:0/3742850329 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe190012990 con 0x7fe1a80d0f40 2024-02-19T15:24:55.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.677+0000 7fe19ea00640 1 --2- 172.21.15.47:0/3742850329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe184066e40 0x7fe184069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:55.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.677+0000 7fe19ea00640 1 -- 172.21.15.47:0/3742850329 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe190084500 con 0x7fe1a80d0f40 2024-02-19T15:24:55.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.677+0000 7fe1a5e00640 1 --2- 172.21.15.47:0/3742850329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe184066e40 0x7fe184069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:55.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.677+0000 7fe1a5e00640 1 --2- 172.21.15.47:0/3742850329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe184066e40 0x7fe184069300 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe198004fd0 tx=0x7fe198001660 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:55.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.681+0000 7fe19ea00640 1 -- 172.21.15.47:0/3742850329 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe1900512a0 con 0x7fe1a80d0f40 2024-02-19T15:24:55.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:55.877+0000 7fe1acc00640 1 -- 172.21.15.47:0/3742850329 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe1a8075360 con 0x7fe1a80d0f40 2024-02-19T15:24:56.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.497+0000 7fe19ea00640 1 -- 172.21.15.47:0/3742850329 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe190054960 con 0x7fe1a80d0f40 2024-02-19T15:24:56.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.497+0000 7fe1acc00640 1 -- 172.21.15.47:0/3742850329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe184066e40 msgr2=0x7fe184069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:56.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.497+0000 7fe1acc00640 1 --2- 172.21.15.47:0/3742850329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe184066e40 0x7fe184069300 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe198004fd0 tx=0x7fe198001660 comp rx=0 tx=0).stop 2024-02-19T15:24:56.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.497+0000 7fe1acc00640 1 -- 172.21.15.47:0/3742850329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1a80d0f40 msgr2=0x7fe1a81672c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:56.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.497+0000 7fe1acc00640 1 --2- 172.21.15.47:0/3742850329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1a80d0f40 0x7fe1a81672c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe19000c4d0 tx=0x7fe19000c990 comp rx=0 tx=0).stop 2024-02-19T15:24:56.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.497+0000 7fe1acc00640 1 -- 172.21.15.47:0/3742850329 shutdown_connections 2024-02-19T15:24:56.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.497+0000 7fe1acc00640 1 --2- 172.21.15.47:0/3742850329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe184066e40 0x7fe184069300 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:56.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.497+0000 7fe1acc00640 1 --2- 172.21.15.47:0/3742850329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a815e690 0x7fe1a8168950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:56.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.497+0000 7fe1acc00640 1 --2- 172.21.15.47:0/3742850329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1a80d0f40 0x7fe1a81672c0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:56.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.497+0000 7fe1acc00640 1 --2- 172.21.15.47:0/3742850329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1a806b110 0x7fe1a80af650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:56.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.501+0000 7fe1acc00640 1 -- 172.21.15.47:0/3742850329 >> 172.21.15.47:0/3742850329 conn(0x7fe1a80d0420 msgr2=0x7fe1a80cee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:56.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.501+0000 7fe1acc00640 1 -- 172.21.15.47:0/3742850329 shutdown_connections 2024-02-19T15:24:56.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.501+0000 7fe1acc00640 1 -- 172.21.15.47:0/3742850329 wait complete. 2024-02-19T15:24:56.526 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T15:24:56.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.669+0000 7fdbdc600640 1 Processor -- start 2024-02-19T15:24:56.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.669+0000 7fdbdc600640 1 -- start start 2024-02-19T15:24:56.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.673+0000 7fdbdc600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd4071050 0x7fdbd4071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:56.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.673+0000 7fdbdc600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd40ca520 con 0x7fdbd415abf0 2024-02-19T15:24:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.673+0000 7fdbdc600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd40ca800 con 0x7fdbd406e570 2024-02-19T15:24:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.673+0000 7fdbdc600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd40caae0 con 0x7fdbd4071050 2024-02-19T15:24:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.673+0000 7fdbd9c00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fdbd415abf0 0x7fdbd406de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54742/0 (socket says 172.21.15.47:54742) 2024-02-19T15:24:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.673+0000 7fdbd9c00640 1 -- 172.21.15.47:0/2905979522 learned_addr learned my addr 172.21.15.47:0/2905979522 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.673+0000 7fdbda600640 1 --2- 172.21.15.47:0/2905979522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd4071050 0x7fdbd4071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.673+0000 7fdbd3e00640 1 -- 172.21.15.47:0/2905979522 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156012997 0 0) 0x7fdbd40ca520 con 0x7fdbd415abf0 2024-02-19T15:24:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.673+0000 7fdbd3e00640 1 -- 172.21.15.47:0/2905979522 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbbc003530 con 0x7fdbd415abf0 2024-02-19T15:24:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.673+0000 7fdbd3e00640 1 -- 172.21.15.47:0/2905979522 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983571545 0 0) 0x7fdbd40ca800 con 0x7fdbd406e570 2024-02-19T15:24:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.673+0000 7fdbd3e00640 1 -- 172.21.15.47:0/2905979522 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbd40ca520 con 0x7fdbd406e570 2024-02-19T15:24:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.673+0000 7fdbda600640 1 -- 172.21.15.47:0/2905979522 >> v1:172.21.15.116:6789/0 conn(0x7fdbd406e570 legacy=0x7fdbd40ae490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.673+0000 7fdbda600640 1 -- 172.21.15.47:0/2905979522 >> v1:172.21.15.47:6789/0 conn(0x7fdbd415abf0 legacy=0x7fdbd406de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.673+0000 7fdbd3e00640 1 -- 172.21.15.47:0/2905979522 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1546308590 0 0) 0x7fdbbc003530 con 0x7fdbd415abf0 2024-02-19T15:24:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.673+0000 7fdbda600640 1 -- 172.21.15.47:0/2905979522 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd40cadc0 con 0x7fdbd4071050 2024-02-19T15:24:56.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.677+0000 7fdbda600640 1 --2- 172.21.15.47:0/2905979522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd4071050 0x7fdbd4071f00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc009dc0 tx=0x7fdbcc01c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78efd9f1b1dd9a40 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:56.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.677+0000 7fdbd3e00640 1 -- 172.21.15.47:0/2905979522 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc0a5a40 con 0x7fdbd4071050 2024-02-19T15:24:56.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.677+0000 7fdbd3e00640 1 -- 172.21.15.47:0/2905979522 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbcc0a5ba0 con 0x7fdbd4071050 2024-02-19T15:24:56.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.677+0000 7fdbd3e00640 1 -- 172.21.15.47:0/2905979522 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc0ad440 con 0x7fdbd4071050 2024-02-19T15:24:56.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.677+0000 7fdbdc600640 1 -- 172.21.15.47:0/2905979522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd4071050 msgr2=0x7fdbd4071f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:56.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.677+0000 7fdbdc600640 1 --2- 172.21.15.47:0/2905979522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd4071050 0x7fdbd4071f00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc009dc0 tx=0x7fdbcc01c960 comp rx=0 tx=0).stop 2024-02-19T15:24:56.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.677+0000 7fdbdc600640 1 -- 172.21.15.47:0/2905979522 shutdown_connections 2024-02-19T15:24:56.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.677+0000 7fdbdc600640 1 --2- 172.21.15.47:0/2905979522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd4071050 0x7fdbd4071f00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:56.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.677+0000 7fdbdc600640 1 -- 172.21.15.47:0/2905979522 >> 172.21.15.47:0/2905979522 conn(0x7fdbd40cc3f0 msgr2=0x7fdbd40cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:56.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.677+0000 7fdbdc600640 1 -- 172.21.15.47:0/2905979522 shutdown_connections 2024-02-19T15:24:56.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.677+0000 7fdbdc600640 1 -- 172.21.15.47:0/2905979522 wait complete. 2024-02-19T15:24:56.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.677+0000 7fdbdc600640 1 Processor -- start 2024-02-19T15:24:56.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.677+0000 7fdbdc600640 1 -- start start 2024-02-19T15:24:56.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbdc600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd406e570 0x7fdbd41540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:56.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbdc600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd4071050 0x7fdbd4153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:56.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbdc600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd415abf0 0x7fdbd40b4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:56.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbdc600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd40c33e0 con 0x7fdbd406e570 2024-02-19T15:24:56.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbdc600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd40b3060 con 0x7fdbd4071050 2024-02-19T15:24:56.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbdc600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd41539c0 con 0x7fdbd415abf0 2024-02-19T15:24:56.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbd9c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd406e570 0x7fdbd41540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:56.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbd9c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd406e570 0x7fdbd41540f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51946/0 (socket says 172.21.15.47:51946) 2024-02-19T15:24:56.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbd9c00640 1 -- 172.21.15.47:0/3749855820 learned_addr learned my addr 172.21.15.47:0/3749855820 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:56.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbda600640 1 --2- 172.21.15.47:0/3749855820 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd415abf0 0x7fdbd40b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:56.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbd9200640 1 --2- 172.21.15.47:0/3749855820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd4071050 0x7fdbd4153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:56.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbd9c00640 1 -- 172.21.15.47:0/3749855820 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd415abf0 msgr2=0x7fdbd40b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:56.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbd9c00640 1 --2- 172.21.15.47:0/3749855820 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd415abf0 0x7fdbd40b4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:56.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbd9c00640 1 -- 172.21.15.47:0/3749855820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd4071050 msgr2=0x7fdbd4153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:56.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbd9c00640 1 --2- 172.21.15.47:0/3749855820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd4071050 0x7fdbd4153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:56.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbd9c00640 1 -- 172.21.15.47:0/3749855820 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbcc009a70 con 0x7fdbd406e570 2024-02-19T15:24:56.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbda600640 1 --2- 172.21.15.47:0/3749855820 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd415abf0 0x7fdbd40b4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:24:56.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbd9c00640 1 --2- 172.21.15.47:0/3749855820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd406e570 0x7fdbd41540f0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fdbc800c450 tx=0x7fdbc800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:56.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbd1600640 1 -- 172.21.15.47:0/3749855820 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8005cf0 con 0x7fdbd406e570 2024-02-19T15:24:56.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbd1600640 1 -- 172.21.15.47:0/3749855820 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc8005e50 con 0x7fdbd406e570 2024-02-19T15:24:56.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbd1600640 1 -- 172.21.15.47:0/3749855820 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc80072b0 con 0x7fdbd406e570 2024-02-19T15:24:56.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbdc600640 1 -- 172.21.15.47:0/3749855820 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd40b32b0 con 0x7fdbd406e570 2024-02-19T15:24:56.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbdc600640 1 -- 172.21.15.47:0/3749855820 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd40b3870 con 0x7fdbd406e570 2024-02-19T15:24:56.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbdc600640 1 -- 172.21.15.47:0/3749855820 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbd4075360 con 0x7fdbd406e570 2024-02-19T15:24:56.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.681+0000 7fdbd1600640 1 -- 172.21.15.47:0/3749855820 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fdbc8007410 con 0x7fdbd406e570 2024-02-19T15:24:56.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.685+0000 7fdbd1600640 1 --2- 172.21.15.47:0/3749855820 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbb4066e40 0x7fdbb4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:56.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.685+0000 7fdbd1600640 1 -- 172.21.15.47:0/3749855820 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdbc8083fb0 con 0x7fdbd406e570 2024-02-19T15:24:56.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.685+0000 7fdbd9200640 1 --2- 172.21.15.47:0/3749855820 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbb4066e40 0x7fdbb4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:56.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.685+0000 7fdbd9200640 1 --2- 172.21.15.47:0/3749855820 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbb4066e40 0x7fdbb4069300 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40b2ff0 tx=0x7fdbc0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:56.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.689+0000 7fdbd1600640 1 -- 172.21.15.47:0/3749855820 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fdbc8050de0 con 0x7fdbd406e570 2024-02-19T15:24:56.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:56.853+0000 7fdbdc600640 1 -- 172.21.15.47:0/3749855820 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdbd40cd300 con 0x7fdbd406e570 2024-02-19T15:24:57.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.501+0000 7fdbd1600640 1 -- 172.21.15.47:0/3749855820 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v59) v1 ==== 181+0+0 (secure 0 0 0) 0x7fdbc8017020 con 0x7fdbd406e570 2024-02-19T15:24:57.504 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_metadata' removed 2024-02-19T15:24:57.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.501+0000 7fdbdc600640 1 -- 172.21.15.47:0/3749855820 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbb4066e40 msgr2=0x7fdbb4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:57.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.501+0000 7fdbdc600640 1 --2- 172.21.15.47:0/3749855820 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbb4066e40 0x7fdbb4069300 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40b2ff0 tx=0x7fdbc0009040 comp rx=0 tx=0).stop 2024-02-19T15:24:57.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.501+0000 7fdbdc600640 1 -- 172.21.15.47:0/3749855820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd406e570 msgr2=0x7fdbd41540f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:57.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.501+0000 7fdbdc600640 1 --2- 172.21.15.47:0/3749855820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd406e570 0x7fdbd41540f0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fdbc800c450 tx=0x7fdbc800c910 comp rx=0 tx=0).stop 2024-02-19T15:24:57.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.505+0000 7fdbdc600640 1 -- 172.21.15.47:0/3749855820 shutdown_connections 2024-02-19T15:24:57.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.505+0000 7fdbdc600640 1 --2- 172.21.15.47:0/3749855820 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbb4066e40 0x7fdbb4069300 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:57.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.505+0000 7fdbdc600640 1 --2- 172.21.15.47:0/3749855820 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd415abf0 0x7fdbd40b4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:57.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.505+0000 7fdbdc600640 1 --2- 172.21.15.47:0/3749855820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd4071050 0x7fdbd4153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:57.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.505+0000 7fdbdc600640 1 --2- 172.21.15.47:0/3749855820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd406e570 0x7fdbd41540f0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:57.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.505+0000 7fdbdc600640 1 -- 172.21.15.47:0/3749855820 >> 172.21.15.47:0/3749855820 conn(0x7fdbd40cc3f0 msgr2=0x7fdbd40b5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:57.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.505+0000 7fdbdc600640 1 -- 172.21.15.47:0/3749855820 shutdown_connections 2024-02-19T15:24:57.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.505+0000 7fdbdc600640 1 -- 172.21.15.47:0/3749855820 wait complete. 2024-02-19T15:24:57.534 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T15:24:57.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.685+0000 7f4e63400640 1 Processor -- start 2024-02-19T15:24:57.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.685+0000 7f4e63400640 1 -- start start 2024-02-19T15:24:57.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.685+0000 7f4e63400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4e5c15f190 0x7f4e5c0d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:57.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.685+0000 7f4e63400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c0d0a50 con 0x7f4e5c15f8f0 2024-02-19T15:24:57.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.685+0000 7f4e63400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c06d470 con 0x7f4e5c0cec80 2024-02-19T15:24:57.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.685+0000 7f4e63400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e5c0cfe80 con 0x7f4e5c15f190 2024-02-19T15:24:57.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.685+0000 7f4e53e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4e5c15f190 0x7f4e5c0d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:57.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.685+0000 7f4e53e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4e5c15f190 0x7f4e5c0d0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51952/0 (socket says 172.21.15.47:51952) 2024-02-19T15:24:57.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.685+0000 7f4e53e00640 1 -- 172.21.15.47:0/2134651358 learned_addr learned my addr 172.21.15.47:0/2134651358 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:57.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.685+0000 7f4e61400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f4e5c15f8f0 0x7f4e5c15fd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54752/0 (socket says 172.21.15.47:54752) 2024-02-19T15:24:57.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.685+0000 7f4e53e00640 1 -- 172.21.15.47:0/2134651358 >> v1:172.21.15.116:6789/0 conn(0x7f4e5c0cec80 legacy=0x7f4e5c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:57.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.685+0000 7f4e53e00640 1 -- 172.21.15.47:0/2134651358 >> v1:172.21.15.47:6789/0 conn(0x7f4e5c15f8f0 legacy=0x7f4e5c15fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:57.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.685+0000 7f4e53e00640 1 -- 172.21.15.47:0/2134651358 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c0d0120 con 0x7f4e5c15f190 2024-02-19T15:24:57.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.689+0000 7f4e53e00640 1 --2- 172.21.15.47:0/2134651358 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e5c15f190 0x7f4e5c0d0ee0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f4e44009870 tx=0x7f4e4401c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37689aa48b5f9d2f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:57.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.689+0000 7f4e5be00640 1 -- 172.21.15.47:0/2134651358 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e440a95f0 con 0x7f4e5c15f190 2024-02-19T15:24:57.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.689+0000 7f4e5be00640 1 -- 172.21.15.47:0/2134651358 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e440a9750 con 0x7f4e5c15f190 2024-02-19T15:24:57.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.689+0000 7f4e5be00640 1 -- 172.21.15.47:0/2134651358 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e440a9ae0 con 0x7f4e5c15f190 2024-02-19T15:24:57.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.689+0000 7f4e63400640 1 -- 172.21.15.47:0/2134651358 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e5c15f190 msgr2=0x7f4e5c0d0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:57.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.689+0000 7f4e63400640 1 --2- 172.21.15.47:0/2134651358 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e5c15f190 0x7f4e5c0d0ee0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f4e44009870 tx=0x7f4e4401c530 comp rx=0 tx=0).stop 2024-02-19T15:24:57.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.689+0000 7f4e63400640 1 -- 172.21.15.47:0/2134651358 shutdown_connections 2024-02-19T15:24:57.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.689+0000 7f4e63400640 1 --2- 172.21.15.47:0/2134651358 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e5c15f190 0x7f4e5c0d0ee0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:57.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.689+0000 7f4e63400640 1 -- 172.21.15.47:0/2134651358 >> 172.21.15.47:0/2134651358 conn(0x7f4e5c09aed0 msgr2=0x7f4e5c0d1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:57.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.689+0000 7f4e63400640 1 -- 172.21.15.47:0/2134651358 shutdown_connections 2024-02-19T15:24:57.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.689+0000 7f4e63400640 1 -- 172.21.15.47:0/2134651358 wait complete. 2024-02-19T15:24:57.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.689+0000 7f4e63400640 1 Processor -- start 2024-02-19T15:24:57.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.689+0000 7f4e63400640 1 -- start start 2024-02-19T15:24:57.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e63400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e5c0cec80 0x7f4e5c0b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:57.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e63400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e5c15f190 0x7f4e5c09c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:57.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e63400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e5c15f8f0 0x7f4e5c09d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:57.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e63400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5c09c990 con 0x7f4e5c15f190 2024-02-19T15:24:57.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e63400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5c09dc80 con 0x7f4e5c0cec80 2024-02-19T15:24:57.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e63400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5c09df20 con 0x7f4e5c15f8f0 2024-02-19T15:24:57.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e53e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e5c15f190 0x7f4e5c09c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:57.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e53e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e5c15f190 0x7f4e5c09c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51968/0 (socket says 172.21.15.47:51968) 2024-02-19T15:24:57.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e53e00640 1 -- 172.21.15.47:0/1769519391 learned_addr learned my addr 172.21.15.47:0/1769519391 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:57.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e60a00640 1 --2- 172.21.15.47:0/1769519391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e5c0cec80 0x7f4e5c0b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:57.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e61400640 1 --2- 172.21.15.47:0/1769519391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e5c15f8f0 0x7f4e5c09d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:57.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e53e00640 1 -- 172.21.15.47:0/1769519391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e5c15f8f0 msgr2=0x7f4e5c09d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:57.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e53e00640 1 --2- 172.21.15.47:0/1769519391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e5c15f8f0 0x7f4e5c09d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:57.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e53e00640 1 -- 172.21.15.47:0/1769519391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e5c0cec80 msgr2=0x7f4e5c0b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:57.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e53e00640 1 --2- 172.21.15.47:0/1769519391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e5c0cec80 0x7f4e5c0b23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:57.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e53e00640 1 -- 172.21.15.47:0/1769519391 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e440094d0 con 0x7f4e5c15f190 2024-02-19T15:24:57.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e53e00640 1 --2- 172.21.15.47:0/1769519391 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e5c15f190 0x7f4e5c09c450 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f4e4401a160 tx=0x7f4e440192d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:57.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e59600640 1 -- 172.21.15.47:0/1769519391 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e440b1800 con 0x7f4e5c15f190 2024-02-19T15:24:57.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e59600640 1 -- 172.21.15.47:0/1769519391 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e440b1980 con 0x7f4e5c15f190 2024-02-19T15:24:57.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e59600640 1 -- 172.21.15.47:0/1769519391 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e440b1d30 con 0x7f4e5c15f190 2024-02-19T15:24:57.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e63400640 1 -- 172.21.15.47:0/1769519391 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e5c09b970 con 0x7f4e5c15f190 2024-02-19T15:24:57.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e63400640 1 -- 172.21.15.47:0/1769519391 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e5c09bf30 con 0x7f4e5c15f190 2024-02-19T15:24:57.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e63400640 1 -- 172.21.15.47:0/1769519391 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e5c075360 con 0x7f4e5c15f190 2024-02-19T15:24:57.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.693+0000 7f4e59600640 1 -- 172.21.15.47:0/1769519391 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4e440c43f0 con 0x7f4e5c15f190 2024-02-19T15:24:57.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.697+0000 7f4e59600640 1 --2- 172.21.15.47:0/1769519391 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4e38066e40 0x7f4e38069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:57.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.697+0000 7f4e59600640 1 -- 172.21.15.47:0/1769519391 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f4e4411f410 con 0x7f4e5c15f190 2024-02-19T15:24:57.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.697+0000 7f4e60a00640 1 --2- 172.21.15.47:0/1769519391 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4e38066e40 0x7f4e38069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:57.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.697+0000 7f4e60a00640 1 --2- 172.21.15.47:0/1769519391 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4e38066e40 0x7f4e38069300 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c000f70 tx=0x7f4e4c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:57.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.701+0000 7f4e59600640 1 -- 172.21.15.47:0/1769519391 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f4e440ec350 con 0x7f4e5c15f190 2024-02-19T15:24:57.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:57.869+0000 7f4e63400640 1 -- 172.21.15.47:0/1769519391 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4e5c09d520 con 0x7f4e5c15f190 2024-02-19T15:24:58.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.501+0000 7f4e59600640 1 -- 172.21.15.47:0/1769519391 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v60) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4e440efa10 con 0x7f4e5c15f190 2024-02-19T15:24:58.506 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_data' removed 2024-02-19T15:24:58.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.505+0000 7f4e63400640 1 -- 172.21.15.47:0/1769519391 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4e38066e40 msgr2=0x7f4e38069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:58.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.505+0000 7f4e63400640 1 --2- 172.21.15.47:0/1769519391 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4e38066e40 0x7f4e38069300 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c000f70 tx=0x7f4e4c009040 comp rx=0 tx=0).stop 2024-02-19T15:24:58.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.505+0000 7f4e63400640 1 -- 172.21.15.47:0/1769519391 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e5c15f190 msgr2=0x7f4e5c09c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:58.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.505+0000 7f4e63400640 1 --2- 172.21.15.47:0/1769519391 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e5c15f190 0x7f4e5c09c450 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f4e4401a160 tx=0x7f4e440192d0 comp rx=0 tx=0).stop 2024-02-19T15:24:58.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.509+0000 7f4e63400640 1 -- 172.21.15.47:0/1769519391 shutdown_connections 2024-02-19T15:24:58.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.509+0000 7f4e63400640 1 --2- 172.21.15.47:0/1769519391 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4e38066e40 0x7f4e38069300 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:58.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.509+0000 7f4e63400640 1 --2- 172.21.15.47:0/1769519391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e5c15f8f0 0x7f4e5c09d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:58.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.509+0000 7f4e63400640 1 --2- 172.21.15.47:0/1769519391 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e5c15f190 0x7f4e5c09c450 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:58.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.509+0000 7f4e63400640 1 --2- 172.21.15.47:0/1769519391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e5c0cec80 0x7f4e5c0b23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:58.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.509+0000 7f4e63400640 1 -- 172.21.15.47:0/1769519391 >> 172.21.15.47:0/1769519391 conn(0x7f4e5c09aed0 msgr2=0x7f4e5c06ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:58.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.509+0000 7f4e63400640 1 -- 172.21.15.47:0/1769519391 shutdown_connections 2024-02-19T15:24:58.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.509+0000 7f4e63400640 1 -- 172.21.15.47:0/1769519391 wait complete. 2024-02-19T15:24:58.544 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-19T15:24:58.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.693+0000 7f3ae1000640 1 Processor -- start 2024-02-19T15:24:58.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.693+0000 7f3ae1000640 1 -- start start 2024-02-19T15:24:58.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.693+0000 7f3ae1000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3adc15dfb0 0x7f3adc15e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:58.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.693+0000 7f3ae1000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3adc0cbfb0 con 0x7f3adc0d2480 2024-02-19T15:24:58.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.693+0000 7f3ae1000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3adc0d0980 con 0x7f3adc0ce8f0 2024-02-19T15:24:58.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.693+0000 7f3ae1000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3adc0d0c60 con 0x7f3adc15dfb0 2024-02-19T15:24:58.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.693+0000 7f3ad2a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f3adc0d2480 0x7f3adc0ce1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54758/0 (socket says 172.21.15.47:54758) 2024-02-19T15:24:58.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.693+0000 7f3ad2a00640 1 -- 172.21.15.47:0/1774495152 learned_addr learned my addr 172.21.15.47:0/1774495152 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:58.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.693+0000 7f3adb400640 1 --2- 172.21.15.47:0/1774495152 >> v2:172.21.15.116:3300/0 conn(0x7f3adc15dfb0 0x7f3adc15e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:58.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.693+0000 7f3adaa00640 1 -- 172.21.15.47:0/1774495152 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994102500 0 0) 0x7f3adc0cbfb0 con 0x7f3adc0d2480 2024-02-19T15:24:58.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.697+0000 7f3adaa00640 1 -- 172.21.15.47:0/1774495152 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ac0003530 con 0x7f3adc0d2480 2024-02-19T15:24:58.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.697+0000 7f3adaa00640 1 -- 172.21.15.47:0/1774495152 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1243897000 0 0) 0x7f3adc0d0980 con 0x7f3adc0ce8f0 2024-02-19T15:24:58.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.697+0000 7f3adaa00640 1 -- 172.21.15.47:0/1774495152 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3adc0cbfb0 con 0x7f3adc0ce8f0 2024-02-19T15:24:58.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.697+0000 7f3adb400640 1 -- 172.21.15.47:0/1774495152 >> v1:172.21.15.116:6789/0 conn(0x7f3adc0ce8f0 legacy=0x7f3adc0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:58.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.697+0000 7f3adb400640 1 -- 172.21.15.47:0/1774495152 >> v1:172.21.15.47:6789/0 conn(0x7f3adc0d2480 legacy=0x7f3adc0ce1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:58.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.697+0000 7f3adb400640 1 -- 172.21.15.47:0/1774495152 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3adc0d0f40 con 0x7f3adc15dfb0 2024-02-19T15:24:58.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.697+0000 7f3adb400640 1 --2- 172.21.15.47:0/1774495152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3adc15dfb0 0x7f3adc15e390 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3acc0097c0 tx=0x7f3acc01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=103dccc451a6109d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:58.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.697+0000 7f3adaa00640 1 -- 172.21.15.47:0/1774495152 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3acc0aa620 con 0x7f3adc15dfb0 2024-02-19T15:24:58.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.697+0000 7f3adaa00640 1 -- 172.21.15.47:0/1774495152 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3acc0aa780 con 0x7f3adc15dfb0 2024-02-19T15:24:58.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.697+0000 7f3adaa00640 1 -- 172.21.15.47:0/1774495152 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3acc0aaaf0 con 0x7f3adc15dfb0 2024-02-19T15:24:58.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.697+0000 7f3ae1000640 1 -- 172.21.15.47:0/1774495152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3adc15dfb0 msgr2=0x7f3adc15e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:58.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.697+0000 7f3ae1000640 1 --2- 172.21.15.47:0/1774495152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3adc15dfb0 0x7f3adc15e390 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3acc0097c0 tx=0x7f3acc01c4a0 comp rx=0 tx=0).stop 2024-02-19T15:24:58.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.697+0000 7f3ae1000640 1 -- 172.21.15.47:0/1774495152 shutdown_connections 2024-02-19T15:24:58.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.697+0000 7f3ae1000640 1 --2- 172.21.15.47:0/1774495152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3adc15dfb0 0x7f3adc15e390 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:58.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.697+0000 7f3ae1000640 1 -- 172.21.15.47:0/1774495152 >> 172.21.15.47:0/1774495152 conn(0x7f3adc0d1960 msgr2=0x7f3adc0d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:58.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.697+0000 7f3ae1000640 1 -- 172.21.15.47:0/1774495152 shutdown_connections 2024-02-19T15:24:58.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.697+0000 7f3ae1000640 1 -- 172.21.15.47:0/1774495152 wait complete. 2024-02-19T15:24:58.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ae1000640 1 Processor -- start 2024-02-19T15:24:58.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ae1000640 1 -- start start 2024-02-19T15:24:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ae1000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3adc0ce8f0 0x7f3adc142e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ae1000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3adc15dfb0 0x7f3adc0c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ae1000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3adc1445a0 0x7f3adc145160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ae1000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc0c8620 con 0x7f3adc15dfb0 2024-02-19T15:24:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ae1000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc143500 con 0x7f3adc1445a0 2024-02-19T15:24:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ae1000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc1437e0 con 0x7f3adc0ce8f0 2024-02-19T15:24:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ad2a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3adc15dfb0 0x7f3adc0c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3adb400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3adc0ce8f0 0x7f3adc142e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ad2a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3adc15dfb0 0x7f3adc0c80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51984/0 (socket says 172.21.15.47:51984) 2024-02-19T15:24:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ad2a00640 1 -- 172.21.15.47:0/3707425709 learned_addr learned my addr 172.21.15.47:0/3707425709 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3adbe00640 1 --2- 172.21.15.47:0/3707425709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3adc1445a0 0x7f3adc145160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ad2a00640 1 -- 172.21.15.47:0/3707425709 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3adc0ce8f0 msgr2=0x7f3adc142e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ad2a00640 1 --2- 172.21.15.47:0/3707425709 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3adc0ce8f0 0x7f3adc142e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ad2a00640 1 -- 172.21.15.47:0/3707425709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3adc1445a0 msgr2=0x7f3adc145160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ad2a00640 1 --2- 172.21.15.47:0/3707425709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3adc1445a0 0x7f3adc145160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ad2a00640 1 -- 172.21.15.47:0/3707425709 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3acc009420 con 0x7f3adc15dfb0 2024-02-19T15:24:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ad2a00640 1 --2- 172.21.15.47:0/3707425709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3adc15dfb0 0x7f3adc0c80e0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400c4d0 tx=0x7f3ac400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ad3e00640 1 -- 172.21.15.47:0/3707425709 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4005cf0 con 0x7f3adc15dfb0 2024-02-19T15:24:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ad3e00640 1 -- 172.21.15.47:0/3707425709 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac4005e50 con 0x7f3adc15dfb0 2024-02-19T15:24:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ad3e00640 1 -- 172.21.15.47:0/3707425709 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4007270 con 0x7f3adc15dfb0 2024-02-19T15:24:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ae1000640 1 -- 172.21.15.47:0/3707425709 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3adc143ac0 con 0x7f3adc15dfb0 2024-02-19T15:24:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.701+0000 7f3ae1000640 1 -- 172.21.15.47:0/3707425709 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3adc144080 con 0x7f3adc15dfb0 2024-02-19T15:24:58.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.705+0000 7f3ae1000640 1 -- 172.21.15.47:0/3707425709 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3adc1442a0 con 0x7f3adc15dfb0 2024-02-19T15:24:58.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.705+0000 7f3ad3e00640 1 -- 172.21.15.47:0/3707425709 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3ac40073d0 con 0x7f3adc15dfb0 2024-02-19T15:24:58.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.705+0000 7f3ad3e00640 1 --2- 172.21.15.47:0/3707425709 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ab8066e40 0x7f3ab8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:58.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.705+0000 7f3ad3e00640 1 -- 172.21.15.47:0/3707425709 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3ac4083c70 con 0x7f3adc15dfb0 2024-02-19T15:24:58.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.705+0000 7f3adb400640 1 --2- 172.21.15.47:0/3707425709 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ab8066e40 0x7f3ab8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:58.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.709+0000 7f3adb400640 1 --2- 172.21.15.47:0/3707425709 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ab8066e40 0x7f3ab8069300 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3acc0ae000 tx=0x7f3acc0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:58.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.709+0000 7f3ad3e00640 1 -- 172.21.15.47:0/3707425709 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f3ac4050d70 con 0x7f3adc15dfb0 2024-02-19T15:24:58.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.873+0000 7f3ae1000640 1 -- 172.21.15.47:0/3707425709 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3adc04f260 con 0x7f3adc15dfb0 2024-02-19T15:24:58.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.877+0000 7f3ad3e00640 1 -- 172.21.15.47:0/3707425709 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3ac40170a0 con 0x7f3adc15dfb0 2024-02-19T15:24:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.877+0000 7f3ae1000640 1 -- 172.21.15.47:0/3707425709 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ab8066e40 msgr2=0x7f3ab8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.877+0000 7f3ae1000640 1 --2- 172.21.15.47:0/3707425709 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ab8066e40 0x7f3ab8069300 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3acc0ae000 tx=0x7f3acc0a7000 comp rx=0 tx=0).stop 2024-02-19T15:24:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.877+0000 7f3ae1000640 1 -- 172.21.15.47:0/3707425709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3adc15dfb0 msgr2=0x7f3adc0c80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.877+0000 7f3ae1000640 1 --2- 172.21.15.47:0/3707425709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3adc15dfb0 0x7f3adc0c80e0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400c4d0 tx=0x7f3ac400c990 comp rx=0 tx=0).stop 2024-02-19T15:24:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.877+0000 7f3ae1000640 1 -- 172.21.15.47:0/3707425709 shutdown_connections 2024-02-19T15:24:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.877+0000 7f3ae1000640 1 --2- 172.21.15.47:0/3707425709 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ab8066e40 0x7f3ab8069300 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.877+0000 7f3ae1000640 1 --2- 172.21.15.47:0/3707425709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3adc1445a0 0x7f3adc145160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.877+0000 7f3ae1000640 1 --2- 172.21.15.47:0/3707425709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3adc15dfb0 0x7f3adc0c80e0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.877+0000 7f3ae1000640 1 --2- 172.21.15.47:0/3707425709 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3adc0ce8f0 0x7f3adc142e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:58.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.877+0000 7f3ae1000640 1 -- 172.21.15.47:0/3707425709 >> 172.21.15.47:0/3707425709 conn(0x7f3adc0d1960 msgr2=0x7f3adc06cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:58.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.877+0000 7f3ae1000640 1 -- 172.21.15.47:0/3707425709 shutdown_connections 2024-02-19T15:24:58.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:58.877+0000 7f3ae1000640 1 -- 172.21.15.47:0/3707425709 wait complete. 2024-02-19T15:24:58.901 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-02-19T15:24:59.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.037+0000 7f74edc00640 1 Processor -- start 2024-02-19T15:24:59.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.041+0000 7f74edc00640 1 -- start start 2024-02-19T15:24:59.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.041+0000 7f74edc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74e815bd90 0x7f74e8070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:59.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.041+0000 7f74edc00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f74e8072850 0x7f74e806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:59.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.041+0000 7f74edc00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e815fc90 con 0x7f74e80cb490 2024-02-19T15:24:59.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.041+0000 7f74edc00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74e815ff70 con 0x7f74e8072850 2024-02-19T15:24:59.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.041+0000 7f74edc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e8160250 con 0x7f74e815bd90 2024-02-19T15:24:59.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.041+0000 7f74e6a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74e815bd90 0x7f74e8070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:59.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.041+0000 7f74de000640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f74e80cb490 0x7f74e8072140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:43724/0 (socket says 172.21.15.47:43724) 2024-02-19T15:24:59.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.041+0000 7f74de000640 1 -- 172.21.15.47:0/359958608 learned_addr learned my addr 172.21.15.47:0/359958608 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:59.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.041+0000 7f74e7400640 1 --2- 172.21.15.47:0/359958608 >> v2:172.21.15.116:3300/0 conn(0x7f74e8072850 0x7f74e806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:59.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.041+0000 7f74e6000640 1 -- 172.21.15.47:0/359958608 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403779148 0 0) 0x7f74e815fc90 con 0x7f74e80cb490 2024-02-19T15:24:59.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.041+0000 7f74e6000640 1 -- 172.21.15.47:0/359958608 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74cc003530 con 0x7f74e80cb490 2024-02-19T15:24:59.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.041+0000 7f74e6a00640 1 -- 172.21.15.47:0/359958608 >> v2:172.21.15.116:3300/0 conn(0x7f74e8072850 msgr2=0x7f74e806cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:59.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.041+0000 7f74e6a00640 1 --2- 172.21.15.47:0/359958608 >> v2:172.21.15.116:3300/0 conn(0x7f74e8072850 0x7f74e806cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.041+0000 7f74e6a00640 1 -- 172.21.15.47:0/359958608 >> v1:172.21.15.116:6789/0 conn(0x7f74e80cb490 legacy=0x7f74e8072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:59.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.041+0000 7f74e6a00640 1 -- 172.21.15.47:0/359958608 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e8160530 con 0x7f74e815bd90 2024-02-19T15:24:59.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.041+0000 7f74e7400640 1 --2- 172.21.15.47:0/359958608 >> v2:172.21.15.116:3300/0 conn(0x7f74e8072850 0x7f74e806cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:24:59.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.045+0000 7f74e6a00640 1 --2- 172.21.15.47:0/359958608 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74e815bd90 0x7f74e8070960 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f74d8009d90 tx=0x7f74d801c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5b4ef8a5ce4893c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:59.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.045+0000 7f74e6000640 1 -- 172.21.15.47:0/359958608 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d80a5820 con 0x7f74e815bd90 2024-02-19T15:24:59.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.045+0000 7f74e6000640 1 -- 172.21.15.47:0/359958608 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d80a5980 con 0x7f74e815bd90 2024-02-19T15:24:59.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.045+0000 7f74e6000640 1 -- 172.21.15.47:0/359958608 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d80a5d10 con 0x7f74e815bd90 2024-02-19T15:24:59.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.045+0000 7f74edc00640 1 -- 172.21.15.47:0/359958608 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74e815bd90 msgr2=0x7f74e8070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:59.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.045+0000 7f74edc00640 1 --2- 172.21.15.47:0/359958608 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74e815bd90 0x7f74e8070960 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f74d8009d90 tx=0x7f74d801c810 comp rx=0 tx=0).stop 2024-02-19T15:24:59.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.045+0000 7f74edc00640 1 -- 172.21.15.47:0/359958608 shutdown_connections 2024-02-19T15:24:59.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.045+0000 7f74edc00640 1 --2- 172.21.15.47:0/359958608 >> v2:172.21.15.116:3300/0 conn(0x7f74e8072850 0x7f74e806cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.045+0000 7f74edc00640 1 --2- 172.21.15.47:0/359958608 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74e815bd90 0x7f74e8070960 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.045+0000 7f74edc00640 1 -- 172.21.15.47:0/359958608 >> 172.21.15.47:0/359958608 conn(0x7f74e80cfc20 msgr2=0x7f74e80d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:59.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.045+0000 7f74edc00640 1 -- 172.21.15.47:0/359958608 shutdown_connections 2024-02-19T15:24:59.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.045+0000 7f74edc00640 1 -- 172.21.15.47:0/359958608 wait complete. 2024-02-19T15:24:59.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.045+0000 7f74edc00640 1 Processor -- start 2024-02-19T15:24:59.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74edc00640 1 -- start start 2024-02-19T15:24:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74edc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74e8072850 0x7f74e814e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74edc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e80cb490 0x7f74e8157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74edc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74e815bd90 0x7f74e8157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74edc00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e8155aa0 con 0x7f74e8072850 2024-02-19T15:24:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74edc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e8156900 con 0x7f74e80cb490 2024-02-19T15:24:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74edc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e8155cf0 con 0x7f74e815bd90 2024-02-19T15:24:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74e6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74e8072850 0x7f74e814e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74e6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74e8072850 0x7f74e814e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51988/0 (socket says 172.21.15.47:51988) 2024-02-19T15:24:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74e6a00640 1 -- 172.21.15.47:0/3227783401 learned_addr learned my addr 172.21.15.47:0/3227783401 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74e7400640 1 --2- 172.21.15.47:0/3227783401 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74e815bd90 0x7f74e8157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74de000640 1 --2- 172.21.15.47:0/3227783401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e80cb490 0x7f74e8157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74e6a00640 1 -- 172.21.15.47:0/3227783401 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74e815bd90 msgr2=0x7f74e8157680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74e6a00640 1 --2- 172.21.15.47:0/3227783401 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74e815bd90 0x7f74e8157680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74e6a00640 1 -- 172.21.15.47:0/3227783401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e80cb490 msgr2=0x7f74e8157140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74e6a00640 1 --2- 172.21.15.47:0/3227783401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e80cb490 0x7f74e8157140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74e6a00640 1 -- 172.21.15.47:0/3227783401 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e0009420 con 0x7f74e8072850 2024-02-19T15:24:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74de000640 1 --2- 172.21.15.47:0/3227783401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e80cb490 0x7f74e8157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74e6a00640 1 --2- 172.21.15.47:0/3227783401 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74e8072850 0x7f74e814e700 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f74d80b0040 tx=0x7f74d80195b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74df400640 1 -- 172.21.15.47:0/3227783401 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d80b9650 con 0x7f74e8072850 2024-02-19T15:24:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74df400640 1 -- 172.21.15.47:0/3227783401 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d80b97b0 con 0x7f74e8072850 2024-02-19T15:24:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74df400640 1 -- 172.21.15.47:0/3227783401 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d80b9b60 con 0x7f74e8072850 2024-02-19T15:24:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74edc00640 1 -- 172.21.15.47:0/3227783401 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d8009a70 con 0x7f74e8072850 2024-02-19T15:24:59.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74edc00640 1 -- 172.21.15.47:0/3227783401 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e81562b0 con 0x7f74e8072850 2024-02-19T15:24:59.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.049+0000 7f74edc00640 1 -- 172.21.15.47:0/3227783401 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74e8075360 con 0x7f74e8072850 2024-02-19T15:24:59.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.053+0000 7f74df400640 1 -- 172.21.15.47:0/3227783401 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f74d80b9cc0 con 0x7f74e8072850 2024-02-19T15:24:59.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.053+0000 7f74df400640 1 --2- 172.21.15.47:0/3227783401 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f74c4066e40 0x7f74c4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:59.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.053+0000 7f74df400640 1 -- 172.21.15.47:0/3227783401 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f74d811fe90 con 0x7f74e8072850 2024-02-19T15:24:59.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.053+0000 7f74de000640 1 --2- 172.21.15.47:0/3227783401 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f74c4066e40 0x7f74c4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:59.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.053+0000 7f74de000640 1 --2- 172.21.15.47:0/3227783401 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f74c4066e40 0x7f74c4069300 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f74e80ca200 tx=0x7f74d0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:59.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.057+0000 7f74df400640 1 -- 172.21.15.47:0/3227783401 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f74d80ecf60 con 0x7f74e8072850 2024-02-19T15:24:59.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.217+0000 7f74edc00640 1 -- 172.21.15.47:0/3227783401 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f74e806f2a0 con 0x7f74e8072850 2024-02-19T15:24:59.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.221+0000 7f74df400640 1 -- 172.21.15.47:0/3227783401 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v20) v1 ==== 108+0+0 (secure 0 0 0) 0x7f74d80f0620 con 0x7f74e8072850 2024-02-19T15:24:59.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.221+0000 7f74edc00640 1 -- 172.21.15.47:0/3227783401 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f74c4066e40 msgr2=0x7f74c4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:59.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.221+0000 7f74edc00640 1 --2- 172.21.15.47:0/3227783401 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f74c4066e40 0x7f74c4069300 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f74e80ca200 tx=0x7f74d0009040 comp rx=0 tx=0).stop 2024-02-19T15:24:59.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.221+0000 7f74edc00640 1 -- 172.21.15.47:0/3227783401 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74e8072850 msgr2=0x7f74e814e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:59.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.221+0000 7f74edc00640 1 --2- 172.21.15.47:0/3227783401 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74e8072850 0x7f74e814e700 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f74d80b0040 tx=0x7f74d80195b0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.221+0000 7f74edc00640 1 -- 172.21.15.47:0/3227783401 shutdown_connections 2024-02-19T15:24:59.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.221+0000 7f74edc00640 1 --2- 172.21.15.47:0/3227783401 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f74c4066e40 0x7f74c4069300 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.221+0000 7f74edc00640 1 --2- 172.21.15.47:0/3227783401 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74e815bd90 0x7f74e8157680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.221+0000 7f74edc00640 1 --2- 172.21.15.47:0/3227783401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e80cb490 0x7f74e8157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.221+0000 7f74edc00640 1 --2- 172.21.15.47:0/3227783401 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74e8072850 0x7f74e814e700 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.221+0000 7f74edc00640 1 -- 172.21.15.47:0/3227783401 >> 172.21.15.47:0/3227783401 conn(0x7f74e80cfc20 msgr2=0x7f74e80d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:59.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.221+0000 7f74edc00640 1 -- 172.21.15.47:0/3227783401 shutdown_connections 2024-02-19T15:24:59.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.225+0000 7f74edc00640 1 -- 172.21.15.47:0/3227783401 wait complete. 2024-02-19T15:24:59.246 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_merge_size 2024-02-19T15:24:59.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.385+0000 7f7a46e00640 1 Processor -- start 2024-02-19T15:24:59.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.385+0000 7f7a46e00640 1 -- start start 2024-02-19T15:24:59.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.385+0000 7f7a46e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a4015e210 0x7f7a400ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:59.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.385+0000 7f7a46e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7a400cc180 0x7f7a400ce8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:59.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.385+0000 7f7a46e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7a400d2650 0x7f7a400d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:59.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.385+0000 7f7a46e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a400cfe00 con 0x7f7a400cc180 2024-02-19T15:24:59.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.385+0000 7f7a46e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a400cff90 con 0x7f7a400d2650 2024-02-19T15:24:59.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.385+0000 7f7a46e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a400b2fd0 con 0x7f7a4015e210 2024-02-19T15:24:59.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.385+0000 7f7a37400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7a400cc180 0x7f7a400ce8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:59.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.385+0000 7f7a37400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7a400cc180 0x7f7a400ce8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51994/0 (socket says 172.21.15.47:51994) 2024-02-19T15:24:59.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.385+0000 7f7a37400640 1 -- 172.21.15.47:0/3960288649 learned_addr learned my addr 172.21.15.47:0/3960288649 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:59.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.385+0000 7f7a3fe00640 1 --2- 172.21.15.47:0/3960288649 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a4015e210 0x7f7a400ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:59.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.385+0000 7f7a44e00640 1 --2- 172.21.15.47:0/3960288649 >> v2:172.21.15.116:3300/0 conn(0x7f7a400d2650 0x7f7a400d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:59.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.385+0000 7f7a37400640 1 -- 172.21.15.47:0/3960288649 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a4015e210 msgr2=0x7f7a400ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:59.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.385+0000 7f7a37400640 1 --2- 172.21.15.47:0/3960288649 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a4015e210 0x7f7a400ce3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.385+0000 7f7a37400640 1 -- 172.21.15.47:0/3960288649 >> v2:172.21.15.116:3300/0 conn(0x7f7a400d2650 msgr2=0x7f7a400d03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:24:59.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.385+0000 7f7a37400640 1 --2- 172.21.15.47:0/3960288649 >> v2:172.21.15.116:3300/0 conn(0x7f7a400d2650 0x7f7a400d03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.385+0000 7f7a37400640 1 -- 172.21.15.47:0/3960288649 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a400b2cf0 con 0x7f7a400cc180 2024-02-19T15:24:59.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.389+0000 7f7a37400640 1 --2- 172.21.15.47:0/3960288649 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a400cc180 0x7f7a400ce8f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f7a400725c0 tx=0x7f7a2801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58849c8872707a5a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:59.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.389+0000 7f7a3f400640 1 -- 172.21.15.47:0/3960288649 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a280aa610 con 0x7f7a400cc180 2024-02-19T15:24:59.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.389+0000 7f7a3f400640 1 -- 172.21.15.47:0/3960288649 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a280a6040 con 0x7f7a400cc180 2024-02-19T15:24:59.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.389+0000 7f7a3f400640 1 -- 172.21.15.47:0/3960288649 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a280aa990 con 0x7f7a400cc180 2024-02-19T15:24:59.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.389+0000 7f7a46e00640 1 -- 172.21.15.47:0/3960288649 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a400cc180 msgr2=0x7f7a400ce8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:59.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.389+0000 7f7a46e00640 1 --2- 172.21.15.47:0/3960288649 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a400cc180 0x7f7a400ce8f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f7a400725c0 tx=0x7f7a2801c4f0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.389+0000 7f7a46e00640 1 -- 172.21.15.47:0/3960288649 shutdown_connections 2024-02-19T15:24:59.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.389+0000 7f7a46e00640 1 --2- 172.21.15.47:0/3960288649 >> v2:172.21.15.116:3300/0 conn(0x7f7a400d2650 0x7f7a400d03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.389+0000 7f7a46e00640 1 --2- 172.21.15.47:0/3960288649 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a400cc180 0x7f7a400ce8f0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.389+0000 7f7a46e00640 1 --2- 172.21.15.47:0/3960288649 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a4015e210 0x7f7a400ce3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.389+0000 7f7a46e00640 1 -- 172.21.15.47:0/3960288649 >> 172.21.15.47:0/3960288649 conn(0x7f7a400d1b30 msgr2=0x7f7a400d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:59.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.389+0000 7f7a46e00640 1 -- 172.21.15.47:0/3960288649 shutdown_connections 2024-02-19T15:24:59.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.389+0000 7f7a46e00640 1 -- 172.21.15.47:0/3960288649 wait complete. 2024-02-19T15:24:59.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a46e00640 1 Processor -- start 2024-02-19T15:24:59.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a46e00640 1 -- start start 2024-02-19T15:24:59.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a46e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a400cc180 0x7f7a4014e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:59.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a46e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a400d2650 0x7f7a401554d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:59.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a46e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a4015e210 0x7f7a40155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:59.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a46e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a401505a0 con 0x7f7a4015e210 2024-02-19T15:24:59.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a46e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a400ca380 con 0x7f7a400d2650 2024-02-19T15:24:59.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a46e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a401507f0 con 0x7f7a400cc180 2024-02-19T15:24:59.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a44e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a4015e210 0x7f7a40155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:59.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a44e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a4015e210 0x7f7a40155a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52006/0 (socket says 172.21.15.47:52006) 2024-02-19T15:24:59.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a44e00640 1 -- 172.21.15.47:0/1268210572 learned_addr learned my addr 172.21.15.47:0/1268210572 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:59.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a3fe00640 1 --2- 172.21.15.47:0/1268210572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a400cc180 0x7f7a4014e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:59.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a37400640 1 --2- 172.21.15.47:0/1268210572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a400d2650 0x7f7a401554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:59.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a44e00640 1 -- 172.21.15.47:0/1268210572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a400cc180 msgr2=0x7f7a4014e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:59.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a44e00640 1 --2- 172.21.15.47:0/1268210572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a400cc180 0x7f7a4014e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a44e00640 1 -- 172.21.15.47:0/1268210572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a400d2650 msgr2=0x7f7a401554d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:59.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a44e00640 1 --2- 172.21.15.47:0/1268210572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a400d2650 0x7f7a401554d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a44e00640 1 -- 172.21.15.47:0/1268210572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a280094d0 con 0x7f7a4015e210 2024-02-19T15:24:59.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a3fe00640 1 --2- 172.21.15.47:0/1268210572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a400cc180 0x7f7a4014e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:59.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a37400640 1 --2- 172.21.15.47:0/1268210572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a400d2650 0x7f7a401554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:24:59.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a44e00640 1 --2- 172.21.15.47:0/1268210572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a4015e210 0x7f7a40155a10 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f7a38007e40 tx=0x7f7a3800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:59.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a3cc00640 1 -- 172.21.15.47:0/1268210572 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a38014620 con 0x7f7a4015e210 2024-02-19T15:24:59.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a3cc00640 1 -- 172.21.15.47:0/1268210572 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a3800ee90 con 0x7f7a4015e210 2024-02-19T15:24:59.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a3cc00640 1 -- 172.21.15.47:0/1268210572 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a380149e0 con 0x7f7a4015e210 2024-02-19T15:24:59.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a46e00640 1 -- 172.21.15.47:0/1268210572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a4015b110 con 0x7f7a4015e210 2024-02-19T15:24:59.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a46e00640 1 -- 172.21.15.47:0/1268210572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a4015b6d0 con 0x7f7a4015e210 2024-02-19T15:24:59.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.393+0000 7f7a46e00640 1 -- 172.21.15.47:0/1268210572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a40075360 con 0x7f7a4015e210 2024-02-19T15:24:59.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.397+0000 7f7a3cc00640 1 -- 172.21.15.47:0/1268210572 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7a38014b40 con 0x7f7a4015e210 2024-02-19T15:24:59.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.397+0000 7f7a3cc00640 1 --2- 172.21.15.47:0/1268210572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a1c066e40 0x7f7a1c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:59.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.397+0000 7f7a3cc00640 1 -- 172.21.15.47:0/1268210572 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7a380850d0 con 0x7f7a4015e210 2024-02-19T15:24:59.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.397+0000 7f7a3fe00640 1 --2- 172.21.15.47:0/1268210572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a1c066e40 0x7f7a1c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:59.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.397+0000 7f7a3fe00640 1 --2- 172.21.15.47:0/1268210572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a1c066e40 0x7f7a1c069300 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7a300079f0 tx=0x7f7a30008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:59.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.401+0000 7f7a3cc00640 1 -- 172.21.15.47:0/1268210572 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7a38052230 con 0x7f7a4015e210 2024-02-19T15:24:59.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.565+0000 7f7a46e00640 1 -- 172.21.15.47:0/1268210572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"} v 0) v1 -- 0x7f7a400b4450 con 0x7f7a4015e210 2024-02-19T15:24:59.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.569+0000 7f7a3cc00640 1 -- 172.21.15.47:0/1268210572 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"}]=0 v21) v1 ==== 101+0+0 (secure 0 0 0) 0x7f7a380558f0 con 0x7f7a4015e210 2024-02-19T15:24:59.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.569+0000 7f7a46e00640 1 -- 172.21.15.47:0/1268210572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a1c066e40 msgr2=0x7f7a1c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:59.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.569+0000 7f7a46e00640 1 --2- 172.21.15.47:0/1268210572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a1c066e40 0x7f7a1c069300 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7a300079f0 tx=0x7f7a30008040 comp rx=0 tx=0).stop 2024-02-19T15:24:59.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.573+0000 7f7a46e00640 1 -- 172.21.15.47:0/1268210572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a4015e210 msgr2=0x7f7a40155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:59.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.573+0000 7f7a46e00640 1 --2- 172.21.15.47:0/1268210572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a4015e210 0x7f7a40155a10 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f7a38007e40 tx=0x7f7a3800e4b0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.573+0000 7f7a46e00640 1 -- 172.21.15.47:0/1268210572 shutdown_connections 2024-02-19T15:24:59.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.573+0000 7f7a46e00640 1 --2- 172.21.15.47:0/1268210572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a1c066e40 0x7f7a1c069300 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.573+0000 7f7a46e00640 1 --2- 172.21.15.47:0/1268210572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a4015e210 0x7f7a40155a10 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.573+0000 7f7a46e00640 1 --2- 172.21.15.47:0/1268210572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a400d2650 0x7f7a401554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.573+0000 7f7a46e00640 1 --2- 172.21.15.47:0/1268210572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a400cc180 0x7f7a4014e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.573+0000 7f7a46e00640 1 -- 172.21.15.47:0/1268210572 >> 172.21.15.47:0/1268210572 conn(0x7f7a400d1b30 msgr2=0x7f7a400cd300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:59.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.573+0000 7f7a46e00640 1 -- 172.21.15.47:0/1268210572 shutdown_connections 2024-02-19T15:24:59.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.573+0000 7f7a46e00640 1 -- 172.21.15.47:0/1268210572 wait complete. 2024-02-19T15:24:59.594 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-02-19T15:24:59.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.729+0000 7f25a3800640 1 Processor -- start 2024-02-19T15:24:59.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.729+0000 7f25a3800640 1 -- start start 2024-02-19T15:24:59.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.733+0000 7f25a3800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f259c15f620 0x7f259c0b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:59.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.733+0000 7f25a3800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259c06b5d0 con 0x7f259c0b2330 2024-02-19T15:24:59.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.733+0000 7f25a3800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259c0b1640 con 0x7f259c1614e0 2024-02-19T15:24:59.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.733+0000 7f25a3800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259c0b1c00 con 0x7f259c15f620 2024-02-19T15:24:59.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.733+0000 7f25a1800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f259c0b2330 0x7f259c06ba60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54760/0 (socket says 172.21.15.47:54760) 2024-02-19T15:24:59.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.733+0000 7f25a1800640 1 -- 172.21.15.47:0/299491736 learned_addr learned my addr 172.21.15.47:0/299491736 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:24:59.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.733+0000 7f2593e00640 1 --2- 172.21.15.47:0/299491736 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f259c15f620 0x7f259c0b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:59.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.733+0000 7f259be00640 1 -- 172.21.15.47:0/299491736 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 61311102 0 0) 0x7f259c06b5d0 con 0x7f259c0b2330 2024-02-19T15:24:59.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.733+0000 7f259be00640 1 -- 172.21.15.47:0/299491736 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2580003530 con 0x7f259c0b2330 2024-02-19T15:24:59.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.733+0000 7f259be00640 1 -- 172.21.15.47:0/299491736 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437782037 0 0) 0x7f259c0b1640 con 0x7f259c1614e0 2024-02-19T15:24:59.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.733+0000 7f259be00640 1 -- 172.21.15.47:0/299491736 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f259c06b5d0 con 0x7f259c1614e0 2024-02-19T15:24:59.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.733+0000 7f2593e00640 1 -- 172.21.15.47:0/299491736 >> v1:172.21.15.116:6789/0 conn(0x7f259c1614e0 legacy=0x7f259c15eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:59.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.733+0000 7f2593e00640 1 -- 172.21.15.47:0/299491736 >> v1:172.21.15.47:6789/0 conn(0x7f259c0b2330 legacy=0x7f259c06ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:59.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.733+0000 7f259be00640 1 -- 172.21.15.47:0/299491736 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2407296249 0 0) 0x7f2580003530 con 0x7f259c0b2330 2024-02-19T15:24:59.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.733+0000 7f2593e00640 1 -- 172.21.15.47:0/299491736 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259c0b1920 con 0x7f259c15f620 2024-02-19T15:24:59.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.737+0000 7f2593e00640 1 --2- 172.21.15.47:0/299491736 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f259c15f620 0x7f259c0b1df0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f25840097f0 tx=0x7f258401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=751ba50dff75a06a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:59.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.737+0000 7f259be00640 1 -- 172.21.15.47:0/299491736 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2594001a00 con 0x7f259c15f620 2024-02-19T15:24:59.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.737+0000 7f259be00640 1 -- 172.21.15.47:0/299491736 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25840a7050 con 0x7f259c15f620 2024-02-19T15:24:59.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.737+0000 7f259be00640 1 -- 172.21.15.47:0/299491736 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25840ab620 con 0x7f259c15f620 2024-02-19T15:24:59.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.737+0000 7f25a3800640 1 -- 172.21.15.47:0/299491736 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f259c15f620 msgr2=0x7f259c0b1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:59.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.737+0000 7f25a3800640 1 --2- 172.21.15.47:0/299491736 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f259c15f620 0x7f259c0b1df0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f25840097f0 tx=0x7f258401c400 comp rx=0 tx=0).stop 2024-02-19T15:24:59.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.737+0000 7f25a3800640 1 -- 172.21.15.47:0/299491736 shutdown_connections 2024-02-19T15:24:59.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.737+0000 7f25a3800640 1 --2- 172.21.15.47:0/299491736 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f259c15f620 0x7f259c0b1df0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.737+0000 7f25a3800640 1 -- 172.21.15.47:0/299491736 >> 172.21.15.47:0/299491736 conn(0x7f259c0b4fc0 msgr2=0x7f259c0b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:24:59.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.737+0000 7f25a3800640 1 -- 172.21.15.47:0/299491736 shutdown_connections 2024-02-19T15:24:59.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.737+0000 7f25a3800640 1 -- 172.21.15.47:0/299491736 wait complete. 2024-02-19T15:24:59.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.737+0000 7f25a3800640 1 Processor -- start 2024-02-19T15:24:59.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.737+0000 7f25a3800640 1 -- start start 2024-02-19T15:24:59.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f25a3800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f259c0b2330 0x7f259c0ae350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:59.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f25a3800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f259c15f620 0x7f259c156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:59.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f25a3800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f259c1614e0 0x7f259c156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:59.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f25a3800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259c156bb0 con 0x7f259c0b2330 2024-02-19T15:24:59.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f25a3800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259c0ca740 con 0x7f259c15f620 2024-02-19T15:24:59.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f25a3800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259c0caa20 con 0x7f259c1614e0 2024-02-19T15:24:59.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f25a0e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f259c0b2330 0x7f259c0ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:59.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f25a0e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f259c0b2330 0x7f259c0ae350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52016/0 (socket says 172.21.15.47:52016) 2024-02-19T15:24:59.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f25a0e00640 1 -- 172.21.15.47:0/2499282675 learned_addr learned my addr 172.21.15.47:0/2499282675 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:24:59.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f2593e00640 1 --2- 172.21.15.47:0/2499282675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f259c15f620 0x7f259c156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:59.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f25a0e00640 1 -- 172.21.15.47:0/2499282675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f259c1614e0 msgr2=0x7f259c156670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:59.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f25a1800640 1 --2- 172.21.15.47:0/2499282675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f259c1614e0 0x7f259c156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:59.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f25a0e00640 1 --2- 172.21.15.47:0/2499282675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f259c1614e0 0x7f259c156670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f25a0e00640 1 -- 172.21.15.47:0/2499282675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f259c15f620 msgr2=0x7f259c156130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:24:59.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f25a0e00640 1 --2- 172.21.15.47:0/2499282675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f259c15f620 0x7f259c156130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:24:59.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f25a0e00640 1 -- 172.21.15.47:0/2499282675 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2584009420 con 0x7f259c0b2330 2024-02-19T15:24:59.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f2593e00640 1 --2- 172.21.15.47:0/2499282675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f259c15f620 0x7f259c156130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:59.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f25a1800640 1 --2- 172.21.15.47:0/2499282675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f259c1614e0 0x7f259c156670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:24:59.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f25a0e00640 1 --2- 172.21.15.47:0/2499282675 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f259c0b2330 0x7f259c0ae350 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f258c00c490 tx=0x7f258c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:59.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f2599600640 1 -- 172.21.15.47:0/2499282675 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c0052d0 con 0x7f259c0b2330 2024-02-19T15:24:59.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f2599600640 1 -- 172.21.15.47:0/2499282675 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258c005430 con 0x7f259c0b2330 2024-02-19T15:24:59.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f2599600640 1 -- 172.21.15.47:0/2499282675 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c012870 con 0x7f259c0b2330 2024-02-19T15:24:59.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f25a3800640 1 -- 172.21.15.47:0/2499282675 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f259c0adb50 con 0x7f259c0b2330 2024-02-19T15:24:59.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f25a3800640 1 -- 172.21.15.47:0/2499282675 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f259c150250 con 0x7f259c0b2330 2024-02-19T15:24:59.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.741+0000 7f25a3800640 1 -- 172.21.15.47:0/2499282675 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f259c075360 con 0x7f259c0b2330 2024-02-19T15:24:59.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.745+0000 7f2599600640 1 -- 172.21.15.47:0/2499282675 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f258c0129d0 con 0x7f259c0b2330 2024-02-19T15:24:59.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.745+0000 7f2599600640 1 --2- 172.21.15.47:0/2499282675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2578066e40 0x7f2578069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:24:59.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.745+0000 7f2599600640 1 -- 172.21.15.47:0/2499282675 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f258c084320 con 0x7f259c0b2330 2024-02-19T15:24:59.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.745+0000 7f2593e00640 1 --2- 172.21.15.47:0/2499282675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2578066e40 0x7f2578069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:24:59.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.745+0000 7f2593e00640 1 --2- 172.21.15.47:0/2499282675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2578066e40 0x7f2578069300 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f25840ad000 tx=0x7f2584008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:24:59.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.749+0000 7f2599600640 1 -- 172.21.15.47:0/2499282675 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f258c051400 con 0x7f259c0b2330 2024-02-19T15:24:59.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:24:59.965+0000 7f25a3800640 1 -- 172.21.15.47:0/2499282675 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7f259c1699a0 con 0x7f259c0b2330 2024-02-19T15:25:00.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.513+0000 7f2599600640 1 -- 172.21.15.47:0/2499282675 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v177) v1 ==== 144+0+0 (secure 0 0 0) 0x7f258c054ac0 con 0x7f259c0b2330 2024-02-19T15:25:00.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.517+0000 7f25a3800640 1 -- 172.21.15.47:0/2499282675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2578066e40 msgr2=0x7f2578069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:00.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.517+0000 7f25a3800640 1 --2- 172.21.15.47:0/2499282675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2578066e40 0x7f2578069300 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f25840ad000 tx=0x7f2584008040 comp rx=0 tx=0).stop 2024-02-19T15:25:00.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.517+0000 7f25a3800640 1 -- 172.21.15.47:0/2499282675 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f259c0b2330 msgr2=0x7f259c0ae350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:00.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.517+0000 7f25a3800640 1 --2- 172.21.15.47:0/2499282675 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f259c0b2330 0x7f259c0ae350 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f258c00c490 tx=0x7f258c00c950 comp rx=0 tx=0).stop 2024-02-19T15:25:00.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.517+0000 7f25a3800640 1 -- 172.21.15.47:0/2499282675 shutdown_connections 2024-02-19T15:25:00.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.517+0000 7f25a3800640 1 --2- 172.21.15.47:0/2499282675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2578066e40 0x7f2578069300 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:00.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.517+0000 7f25a3800640 1 --2- 172.21.15.47:0/2499282675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f259c1614e0 0x7f259c156670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:00.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.517+0000 7f25a3800640 1 --2- 172.21.15.47:0/2499282675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f259c15f620 0x7f259c156130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:00.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.517+0000 7f25a3800640 1 --2- 172.21.15.47:0/2499282675 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f259c0b2330 0x7f259c0ae350 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:00.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.517+0000 7f25a3800640 1 -- 172.21.15.47:0/2499282675 >> 172.21.15.47:0/2499282675 conn(0x7f259c0b4fc0 msgr2=0x7f259c15e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:00.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.517+0000 7f25a3800640 1 -- 172.21.15.47:0/2499282675 shutdown_connections 2024-02-19T15:25:00.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.517+0000 7f25a3800640 1 -- 172.21.15.47:0/2499282675 wait complete. 2024-02-19T15:25:00.540 INFO:tasks.cephfs_test_runner:That fragmentation (forced) will allow more entries to be created. ... ok 2024-02-19T15:25:00.540 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) That unlinking fails when the stray directory fragment becomes too 2024-02-19T15:25:00.540 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) 2024-02-19T15:25:00.541 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:25:00.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.677+0000 7f1a9b400640 1 Processor -- start 2024-02-19T15:25:00.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.677+0000 7f1a9b400640 1 -- start start 2024-02-19T15:25:00.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.677+0000 7f1a9b400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1a94071350 0x7f1a94165e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:00.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.677+0000 7f1a9b400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a940718c0 con 0x7f1a9415d6e0 2024-02-19T15:25:00.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.677+0000 7f1a9b400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a94071ba0 con 0x7f1a9415b1a0 2024-02-19T15:25:00.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.677+0000 7f1a9b400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a94071e80 con 0x7f1a94071350 2024-02-19T15:25:00.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.677+0000 7f1a99400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1a94071350 0x7f1a94165e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:00.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.677+0000 7f1a93e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1a9415d6e0 0x7f1a94162740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54776/0 (socket says 172.21.15.47:54776) 2024-02-19T15:25:00.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.677+0000 7f1a93e00640 1 -- 172.21.15.47:0/849102689 learned_addr learned my addr 172.21.15.47:0/849102689 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:00.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.681+0000 7f1a99400640 1 -- 172.21.15.47:0/849102689 >> v1:172.21.15.116:6789/0 conn(0x7f1a9415b1a0 legacy=0x7f1a9415b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:00.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.681+0000 7f1a93400640 1 -- 172.21.15.47:0/849102689 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2127041090 0 0) 0x7f1a940718c0 con 0x7f1a9415d6e0 2024-02-19T15:25:00.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.681+0000 7f1a99400640 1 -- 172.21.15.47:0/849102689 >> v1:172.21.15.47:6789/0 conn(0x7f1a9415d6e0 legacy=0x7f1a94162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:00.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.681+0000 7f1a99400640 1 -- 172.21.15.47:0/849102689 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a94072160 con 0x7f1a94071350 2024-02-19T15:25:00.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.681+0000 7f1a99400640 1 --2- 172.21.15.47:0/849102689 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a94071350 0x7f1a94165e70 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c009dc0 tx=0x7f1a8c01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f765306c4b0b54c4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:00.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.681+0000 7f1a93400640 1 -- 172.21.15.47:0/849102689 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c0a5a30 con 0x7f1a94071350 2024-02-19T15:25:00.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.681+0000 7f1a93400640 1 -- 172.21.15.47:0/849102689 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a8c0a5b90 con 0x7f1a94071350 2024-02-19T15:25:00.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.681+0000 7f1a93400640 1 -- 172.21.15.47:0/849102689 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c0ad440 con 0x7f1a94071350 2024-02-19T15:25:00.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.681+0000 7f1a9b400640 1 -- 172.21.15.47:0/849102689 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a94071350 msgr2=0x7f1a94165e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:00.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.681+0000 7f1a9b400640 1 --2- 172.21.15.47:0/849102689 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a94071350 0x7f1a94165e70 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c009dc0 tx=0x7f1a8c01c950 comp rx=0 tx=0).stop 2024-02-19T15:25:00.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.681+0000 7f1a9b400640 1 -- 172.21.15.47:0/849102689 shutdown_connections 2024-02-19T15:25:00.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.681+0000 7f1a9b400640 1 --2- 172.21.15.47:0/849102689 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a94071350 0x7f1a94165e70 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:00.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.681+0000 7f1a9b400640 1 -- 172.21.15.47:0/849102689 >> 172.21.15.47:0/849102689 conn(0x7f1a940b16a0 msgr2=0x7f1a940b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:00.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a9b400640 1 -- 172.21.15.47:0/849102689 shutdown_connections 2024-02-19T15:25:00.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a9b400640 1 -- 172.21.15.47:0/849102689 wait complete. 2024-02-19T15:25:00.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a9b400640 1 Processor -- start 2024-02-19T15:25:00.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a9b400640 1 -- start start 2024-02-19T15:25:00.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a9b400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a94071350 0x7f1a9414aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:00.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a9b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a9415b1a0 0x7f1a94148d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:00.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a9b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a9415d6e0 0x7f1a940b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:00.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a9b400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a940b2830 con 0x7f1a9415d6e0 2024-02-19T15:25:00.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a9b400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a940b2b10 con 0x7f1a9415b1a0 2024-02-19T15:25:00.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a9b400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a940b2df0 con 0x7f1a94071350 2024-02-19T15:25:00.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a99400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a9415d6e0 0x7f1a940b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:00.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a99400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a9415d6e0 0x7f1a940b38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52040/0 (socket says 172.21.15.47:52040) 2024-02-19T15:25:00.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a99400640 1 -- 172.21.15.47:0/306894261 learned_addr learned my addr 172.21.15.47:0/306894261 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:00.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a93e00640 1 --2- 172.21.15.47:0/306894261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a9415b1a0 0x7f1a94148d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:00.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a98a00640 1 --2- 172.21.15.47:0/306894261 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a94071350 0x7f1a9414aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:00.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a99400640 1 -- 172.21.15.47:0/306894261 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a94071350 msgr2=0x7f1a9414aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:00.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a99400640 1 --2- 172.21.15.47:0/306894261 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a94071350 0x7f1a9414aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:00.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a99400640 1 -- 172.21.15.47:0/306894261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a9415b1a0 msgr2=0x7f1a94148d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:00.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a99400640 1 --2- 172.21.15.47:0/306894261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a9415b1a0 0x7f1a94148d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:00.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a99400640 1 -- 172.21.15.47:0/306894261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8c009a70 con 0x7f1a9415d6e0 2024-02-19T15:25:00.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a99400640 1 --2- 172.21.15.47:0/306894261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a9415d6e0 0x7f1a940b38d0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c01a580 tx=0x7f1a8c0196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:00.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a90c00640 1 -- 172.21.15.47:0/306894261 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c0addc0 con 0x7f1a9415d6e0 2024-02-19T15:25:00.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a90c00640 1 -- 172.21.15.47:0/306894261 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a8c009450 con 0x7f1a9415d6e0 2024-02-19T15:25:00.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a90c00640 1 -- 172.21.15.47:0/306894261 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c0b9650 con 0x7f1a9415d6e0 2024-02-19T15:25:00.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.685+0000 7f1a9b400640 1 -- 172.21.15.47:0/306894261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a940b30d0 con 0x7f1a9415d6e0 2024-02-19T15:25:00.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.689+0000 7f1a9b400640 1 -- 172.21.15.47:0/306894261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a94149270 con 0x7f1a9415d6e0 2024-02-19T15:25:00.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.689+0000 7f1a9b400640 1 -- 172.21.15.47:0/306894261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a940b2f30 con 0x7f1a9415d6e0 2024-02-19T15:25:00.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.689+0000 7f1a90c00640 1 -- 172.21.15.47:0/306894261 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1a8c0b97b0 con 0x7f1a9415d6e0 2024-02-19T15:25:00.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.689+0000 7f1a90c00640 1 --2- 172.21.15.47:0/306894261 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1a74066e40 0x7f1a74069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:00.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.689+0000 7f1a90c00640 1 -- 172.21.15.47:0/306894261 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1a8c120090 con 0x7f1a9415d6e0 2024-02-19T15:25:00.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.689+0000 7f1a98a00640 1 --2- 172.21.15.47:0/306894261 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1a74066e40 0x7f1a74069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:00.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.689+0000 7f1a98a00640 1 --2- 172.21.15.47:0/306894261 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1a74066e40 0x7f1a74069300 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1a88000f70 tx=0x7f1a88009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:00.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.693+0000 7f1a90c00640 1 -- 172.21.15.47:0/306894261 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1a940b2f30 con 0x7f1a9415d6e0 2024-02-19T15:25:00.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.861+0000 7f1a9b400640 1 -- 172.21.15.47:0/306894261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a940b2f30 con 0x7f1a9415d6e0 2024-02-19T15:25:00.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.861+0000 7f1a90c00640 1 -- 172.21.15.47:0/306894261 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+13028 (secure 0 0 0) 0x7f1a940b2f30 con 0x7f1a9415d6e0 2024-02-19T15:25:00.865 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:00.865 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":60,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:24:58.503662+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:6834/61582103":"2024-02-20T15:24:55.495406+0000","172.21.15.47:6832/61582103":"2024-02-20T15:24:55.495406+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:25:00.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.865+0000 7f1a9b400640 1 -- 172.21.15.47:0/306894261 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1a74066e40 msgr2=0x7f1a74069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:00.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.865+0000 7f1a9b400640 1 --2- 172.21.15.47:0/306894261 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1a74066e40 0x7f1a74069300 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1a88000f70 tx=0x7f1a88009040 comp rx=0 tx=0).stop 2024-02-19T15:25:00.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.865+0000 7f1a9b400640 1 -- 172.21.15.47:0/306894261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a9415d6e0 msgr2=0x7f1a940b38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:00.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.865+0000 7f1a9b400640 1 --2- 172.21.15.47:0/306894261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a9415d6e0 0x7f1a940b38d0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c01a580 tx=0x7f1a8c0196f0 comp rx=0 tx=0).stop 2024-02-19T15:25:00.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.865+0000 7f1a9b400640 1 -- 172.21.15.47:0/306894261 shutdown_connections 2024-02-19T15:25:00.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.865+0000 7f1a9b400640 1 --2- 172.21.15.47:0/306894261 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1a74066e40 0x7f1a74069300 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:00.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.865+0000 7f1a9b400640 1 --2- 172.21.15.47:0/306894261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a9415d6e0 0x7f1a940b38d0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:00.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.865+0000 7f1a9b400640 1 --2- 172.21.15.47:0/306894261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a9415b1a0 0x7f1a94148d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:00.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.865+0000 7f1a9b400640 1 --2- 172.21.15.47:0/306894261 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a94071350 0x7f1a9414aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:00.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.865+0000 7f1a9b400640 1 -- 172.21.15.47:0/306894261 >> 172.21.15.47:0/306894261 conn(0x7f1a940b16a0 msgr2=0x7f1a940b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:00.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.865+0000 7f1a9b400640 1 -- 172.21.15.47:0/306894261 shutdown_connections 2024-02-19T15:25:00.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:00.865+0000 7f1a9b400640 1 -- 172.21.15.47:0/306894261 wait complete. 2024-02-19T15:25:00.887 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:25:00.887 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:25:01.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.029+0000 7f1c58600640 1 Processor -- start 2024-02-19T15:25:01.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.029+0000 7f1c58600640 1 -- start start 2024-02-19T15:25:01.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.029+0000 7f1c58600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c500d2650 0x7f1c5006c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:01.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.029+0000 7f1c58600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c500cc620 con 0x7f1c5015e280 2024-02-19T15:25:01.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.029+0000 7f1c58600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c500d12e0 con 0x7f1c500ce3b0 2024-02-19T15:25:01.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.029+0000 7f1c58600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c500d1940 con 0x7f1c500d2650 2024-02-19T15:25:01.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.029+0000 7f1c56600640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1c5015e280 0x7f1c5006c710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54784/0 (socket says 172.21.15.47:54784) 2024-02-19T15:25:01.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.029+0000 7f1c56600640 1 -- 172.21.15.47:0/3175728245 learned_addr learned my addr 172.21.15.47:0/3175728245 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:01.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.029+0000 7f1c55200640 1 --2- 172.21.15.47:0/3175728245 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c500d2650 0x7f1c5006c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:01.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.029+0000 7f1c4fe00640 1 -- 172.21.15.47:0/3175728245 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078711594 0 0) 0x7f1c500cc620 con 0x7f1c5015e280 2024-02-19T15:25:01.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.029+0000 7f1c4fe00640 1 -- 172.21.15.47:0/3175728245 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c38003530 con 0x7f1c5015e280 2024-02-19T15:25:01.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.029+0000 7f1c4fe00640 1 -- 172.21.15.47:0/3175728245 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644790731 0 0) 0x7f1c500d12e0 con 0x7f1c500ce3b0 2024-02-19T15:25:01.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.029+0000 7f1c4fe00640 1 -- 172.21.15.47:0/3175728245 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c500cc620 con 0x7f1c500ce3b0 2024-02-19T15:25:01.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.029+0000 7f1c55200640 1 -- 172.21.15.47:0/3175728245 >> v1:172.21.15.116:6789/0 conn(0x7f1c500ce3b0 legacy=0x7f1c500ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:01.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.029+0000 7f1c55200640 1 -- 172.21.15.47:0/3175728245 >> v1:172.21.15.47:6789/0 conn(0x7f1c5015e280 legacy=0x7f1c5006c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:01.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.029+0000 7f1c55200640 1 -- 172.21.15.47:0/3175728245 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c500d1000 con 0x7f1c500d2650 2024-02-19T15:25:01.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.033+0000 7f1c55200640 1 --2- 172.21.15.47:0/3175728245 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c500d2650 0x7f1c5006c1d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c0097f0 tx=0x7f1c3c01c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=500d34e465e39043 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:01.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.033+0000 7f1c4fe00640 1 -- 172.21.15.47:0/3175728245 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c0ab600 con 0x7f1c500d2650 2024-02-19T15:25:01.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.033+0000 7f1c4fe00640 1 -- 172.21.15.47:0/3175728245 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c3c0a7050 con 0x7f1c500d2650 2024-02-19T15:25:01.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.033+0000 7f1c4fe00640 1 -- 172.21.15.47:0/3175728245 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c0ab980 con 0x7f1c500d2650 2024-02-19T15:25:01.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.033+0000 7f1c58600640 1 -- 172.21.15.47:0/3175728245 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c500d2650 msgr2=0x7f1c5006c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:01.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.033+0000 7f1c58600640 1 --2- 172.21.15.47:0/3175728245 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c500d2650 0x7f1c5006c1d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c0097f0 tx=0x7f1c3c01c3c0 comp rx=0 tx=0).stop 2024-02-19T15:25:01.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.033+0000 7f1c58600640 1 -- 172.21.15.47:0/3175728245 shutdown_connections 2024-02-19T15:25:01.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.033+0000 7f1c58600640 1 --2- 172.21.15.47:0/3175728245 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c500d2650 0x7f1c5006c1d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:01.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.033+0000 7f1c58600640 1 -- 172.21.15.47:0/3175728245 >> 172.21.15.47:0/3175728245 conn(0x7f1c500d1b30 msgr2=0x7f1c500d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:01.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.033+0000 7f1c58600640 1 -- 172.21.15.47:0/3175728245 shutdown_connections 2024-02-19T15:25:01.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.033+0000 7f1c58600640 1 -- 172.21.15.47:0/3175728245 wait complete. 2024-02-19T15:25:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c58600640 1 Processor -- start 2024-02-19T15:25:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c58600640 1 -- start start 2024-02-19T15:25:01.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c58600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c500ce3b0 0x7f1c5015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:01.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c58600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c500d2650 0x7f1c50159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:01.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c58600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c5015e280 0x7f1c5015e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:01.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c58600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c50159860 con 0x7f1c5015e280 2024-02-19T15:25:01.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c58600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5015eba0 con 0x7f1c500ce3b0 2024-02-19T15:25:01.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c58600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5015a650 con 0x7f1c500d2650 2024-02-19T15:25:01.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c56600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c5015e280 0x7f1c5015e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:01.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c56600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c5015e280 0x7f1c5015e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52056/0 (socket says 172.21.15.47:52056) 2024-02-19T15:25:01.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c56600640 1 -- 172.21.15.47:0/501352588 learned_addr learned my addr 172.21.15.47:0/501352588 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:01.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c55200640 1 --2- 172.21.15.47:0/501352588 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c500d2650 0x7f1c50159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:01.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c55c00640 1 --2- 172.21.15.47:0/501352588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c500ce3b0 0x7f1c5015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:01.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c56600640 1 -- 172.21.15.47:0/501352588 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c500d2650 msgr2=0x7f1c50159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:01.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c56600640 1 --2- 172.21.15.47:0/501352588 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c500d2650 0x7f1c50159320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:01.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c56600640 1 -- 172.21.15.47:0/501352588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c500ce3b0 msgr2=0x7f1c5015fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:01.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c56600640 1 --2- 172.21.15.47:0/501352588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c500ce3b0 0x7f1c5015fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:01.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c56600640 1 -- 172.21.15.47:0/501352588 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3c009420 con 0x7f1c5015e280 2024-02-19T15:25:01.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c55c00640 1 --2- 172.21.15.47:0/501352588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c500ce3b0 0x7f1c5015fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:01.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c56600640 1 --2- 172.21.15.47:0/501352588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c5015e280 0x7f1c5015e660 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f1c4800cb70 tx=0x7f1c4800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:01.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c4d600640 1 -- 172.21.15.47:0/501352588 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4800ace0 con 0x7f1c5015e280 2024-02-19T15:25:01.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c4d600640 1 -- 172.21.15.47:0/501352588 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4800ae40 con 0x7f1c5015e280 2024-02-19T15:25:01.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c4d600640 1 -- 172.21.15.47:0/501352588 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c48012620 con 0x7f1c5015e280 2024-02-19T15:25:01.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c58600640 1 -- 172.21.15.47:0/501352588 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c5015ed90 con 0x7f1c5015e280 2024-02-19T15:25:01.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c58600640 1 -- 172.21.15.47:0/501352588 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c5015f350 con 0x7f1c5015e280 2024-02-19T15:25:01.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.037+0000 7f1c58600640 1 -- 172.21.15.47:0/501352588 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c50075360 con 0x7f1c5015e280 2024-02-19T15:25:01.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.041+0000 7f1c4d600640 1 -- 172.21.15.47:0/501352588 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1c48007070 con 0x7f1c5015e280 2024-02-19T15:25:01.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.041+0000 7f1c4d600640 1 --2- 172.21.15.47:0/501352588 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1c30066e70 0x7f1c30069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:01.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.041+0000 7f1c4d600640 1 -- 172.21.15.47:0/501352588 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1c480846f0 con 0x7f1c5015e280 2024-02-19T15:25:01.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.041+0000 7f1c55c00640 1 --2- 172.21.15.47:0/501352588 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1c30066e70 0x7f1c30069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:01.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.041+0000 7f1c55c00640 1 --2- 172.21.15.47:0/501352588 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1c30066e70 0x7f1c30069330 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1c440011f0 tx=0x7f1c44009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:01.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.045+0000 7f1c4d600640 1 -- 172.21.15.47:0/501352588 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1c48055320 con 0x7f1c5015e280 2024-02-19T15:25:01.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.209+0000 7f1c58600640 1 -- 172.21.15.47:0/501352588 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1c500cdf80 con 0x7f1c5015e280 2024-02-19T15:25:01.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.209+0000 7f1c4d600640 1 -- 172.21.15.47:0/501352588 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v60) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1c480170a0 con 0x7f1c5015e280 2024-02-19T15:25:01.212 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:25:01.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.209+0000 7f1c58600640 1 -- 172.21.15.47:0/501352588 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1c30066e70 msgr2=0x7f1c30069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:01.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.209+0000 7f1c58600640 1 --2- 172.21.15.47:0/501352588 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1c30066e70 0x7f1c30069330 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1c440011f0 tx=0x7f1c44009040 comp rx=0 tx=0).stop 2024-02-19T15:25:01.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.209+0000 7f1c58600640 1 -- 172.21.15.47:0/501352588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c5015e280 msgr2=0x7f1c5015e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:01.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.209+0000 7f1c58600640 1 --2- 172.21.15.47:0/501352588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c5015e280 0x7f1c5015e660 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f1c4800cb70 tx=0x7f1c4800a430 comp rx=0 tx=0).stop 2024-02-19T15:25:01.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.209+0000 7f1c58600640 1 -- 172.21.15.47:0/501352588 shutdown_connections 2024-02-19T15:25:01.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.209+0000 7f1c58600640 1 --2- 172.21.15.47:0/501352588 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1c30066e70 0x7f1c30069330 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:01.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.209+0000 7f1c58600640 1 --2- 172.21.15.47:0/501352588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c5015e280 0x7f1c5015e660 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:01.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.209+0000 7f1c58600640 1 --2- 172.21.15.47:0/501352588 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c500d2650 0x7f1c50159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:01.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.209+0000 7f1c58600640 1 --2- 172.21.15.47:0/501352588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c500ce3b0 0x7f1c5015fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:01.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.209+0000 7f1c58600640 1 -- 172.21.15.47:0/501352588 >> 172.21.15.47:0/501352588 conn(0x7f1c500d1b30 msgr2=0x7f1c5006bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:01.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.209+0000 7f1c58600640 1 -- 172.21.15.47:0/501352588 shutdown_connections 2024-02-19T15:25:01.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.209+0000 7f1c58600640 1 -- 172.21.15.47:0/501352588 wait complete. 2024-02-19T15:25:01.237 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-02-19T15:25:01.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.373+0000 7f78c9400640 1 Processor -- start 2024-02-19T15:25:01.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.373+0000 7f78c9400640 1 -- start start 2024-02-19T15:25:01.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.373+0000 7f78c9400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f78c4166b60 0x7f78c4168f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:01.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.373+0000 7f78c9400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c40ad840 con 0x7f78c40cb520 2024-02-19T15:25:01.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.373+0000 7f78c9400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c40c62f0 con 0x7f78c406aeb0 2024-02-19T15:25:01.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.373+0000 7f78c9400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c40cc810 con 0x7f78c4166b60 2024-02-19T15:25:01.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.377+0000 7f78c3200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f78c4166b60 0x7f78c4168f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:01.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.377+0000 7f78c3200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f78c4166b60 0x7f78c4168f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52068/0 (socket says 172.21.15.47:52068) 2024-02-19T15:25:01.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.377+0000 7f78c3200640 1 -- 172.21.15.47:0/4231950867 learned_addr learned my addr 172.21.15.47:0/4231950867 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:01.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.377+0000 7f78c3200640 1 -- 172.21.15.47:0/4231950867 >> v1:172.21.15.116:6789/0 conn(0x7f78c406aeb0 legacy=0x7f78c406b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:01.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.377+0000 7f78c3200640 1 -- 172.21.15.47:0/4231950867 >> v1:172.21.15.47:6789/0 conn(0x7f78c40cb520 legacy=0x7f78c40cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:01.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.377+0000 7f78c1e00640 1 -- 172.21.15.47:0/4231950867 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651141601 0 0) 0x7f78c40ad840 con 0x7f78c40cb520 2024-02-19T15:25:01.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.377+0000 7f78c3200640 1 -- 172.21.15.47:0/4231950867 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c40cc250 con 0x7f78c4166b60 2024-02-19T15:25:01.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.377+0000 7f78c3200640 1 --2- 172.21.15.47:0/4231950867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78c4166b60 0x7f78c4168f50 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f78bc009e50 tx=0x7f78bc01ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5856b9bdea1f4c22 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:01.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.377+0000 7f78c1e00640 1 -- 172.21.15.47:0/4231950867 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc0a58d0 con 0x7f78c4166b60 2024-02-19T15:25:01.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.377+0000 7f78c1e00640 1 -- 172.21.15.47:0/4231950867 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78bc0a5a30 con 0x7f78c4166b60 2024-02-19T15:25:01.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.377+0000 7f78c1e00640 1 -- 172.21.15.47:0/4231950867 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc0a5dc0 con 0x7f78c4166b60 2024-02-19T15:25:01.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c9400640 1 -- 172.21.15.47:0/4231950867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78c4166b60 msgr2=0x7f78c4168f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:01.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c9400640 1 --2- 172.21.15.47:0/4231950867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78c4166b60 0x7f78c4168f50 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f78bc009e50 tx=0x7f78bc01ca00 comp rx=0 tx=0).stop 2024-02-19T15:25:01.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c9400640 1 -- 172.21.15.47:0/4231950867 shutdown_connections 2024-02-19T15:25:01.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c9400640 1 --2- 172.21.15.47:0/4231950867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78c4166b60 0x7f78c4168f50 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:01.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c9400640 1 -- 172.21.15.47:0/4231950867 >> 172.21.15.47:0/4231950867 conn(0x7f78c40d0ad0 msgr2=0x7f78c40d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:01.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c9400640 1 -- 172.21.15.47:0/4231950867 shutdown_connections 2024-02-19T15:25:01.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c9400640 1 -- 172.21.15.47:0/4231950867 wait complete. 2024-02-19T15:25:01.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c9400640 1 Processor -- start 2024-02-19T15:25:01.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c9400640 1 -- start start 2024-02-19T15:25:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c9400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c406aeb0 0x7f78c40ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c9400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f78c40cb520 0x7f78c4167f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c9400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78c4166b60 0x7f78c41684d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c9400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c4145990 con 0x7f78c4166b60 2024-02-19T15:25:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c9400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c4144d90 con 0x7f78c406aeb0 2024-02-19T15:25:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c9400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78c4145060 con 0x7f78c40cb520 2024-02-19T15:25:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c3200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78c4166b60 0x7f78c41684d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c3200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78c4166b60 0x7f78c41684d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52070/0 (socket says 172.21.15.47:52070) 2024-02-19T15:25:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c3200640 1 -- 172.21.15.47:0/180062119 learned_addr learned my addr 172.21.15.47:0/180062119 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c2800640 1 --2- 172.21.15.47:0/180062119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c406aeb0 0x7f78c40ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78b9e00640 1 --2- 172.21.15.47:0/180062119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f78c40cb520 0x7f78c4167f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c3200640 1 -- 172.21.15.47:0/180062119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f78c40cb520 msgr2=0x7f78c4167f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c3200640 1 --2- 172.21.15.47:0/180062119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f78c40cb520 0x7f78c4167f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c3200640 1 -- 172.21.15.47:0/180062119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c406aeb0 msgr2=0x7f78c40ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c3200640 1 --2- 172.21.15.47:0/180062119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c406aeb0 0x7f78c40ca4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c3200640 1 -- 172.21.15.47:0/180062119 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78bc009b30 con 0x7f78c4166b60 2024-02-19T15:25:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78b9e00640 1 --2- 172.21.15.47:0/180062119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f78c40cb520 0x7f78c4167f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:25:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.381+0000 7f78c3200640 1 --2- 172.21.15.47:0/180062119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78c4166b60 0x7f78c41684d0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f78bc0af040 tx=0x7f78bc0197a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.385+0000 7f78bb400640 1 -- 172.21.15.47:0/180062119 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc0b8880 con 0x7f78c4166b60 2024-02-19T15:25:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.385+0000 7f78bb400640 1 -- 172.21.15.47:0/180062119 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78bc0b8a20 con 0x7f78c4166b60 2024-02-19T15:25:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.385+0000 7f78bb400640 1 -- 172.21.15.47:0/180062119 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc0b8dd0 con 0x7f78c4166b60 2024-02-19T15:25:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.385+0000 7f78c9400640 1 -- 172.21.15.47:0/180062119 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78c4145340 con 0x7f78c4166b60 2024-02-19T15:25:01.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.385+0000 7f78c9400640 1 -- 172.21.15.47:0/180062119 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c4159a00 con 0x7f78c4166b60 2024-02-19T15:25:01.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.385+0000 7f78c9400640 1 -- 172.21.15.47:0/180062119 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78c4145b00 con 0x7f78c4166b60 2024-02-19T15:25:01.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.385+0000 7f78bb400640 1 -- 172.21.15.47:0/180062119 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f78bc0c3460 con 0x7f78c4166b60 2024-02-19T15:25:01.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.385+0000 7f78bb400640 1 --2- 172.21.15.47:0/180062119 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f78a0066e40 0x7f78a0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:01.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.385+0000 7f78bb400640 1 -- 172.21.15.47:0/180062119 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f78bc11ff50 con 0x7f78c4166b60 2024-02-19T15:25:01.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.385+0000 7f78c2800640 1 --2- 172.21.15.47:0/180062119 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f78a0066e40 0x7f78a0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:01.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.385+0000 7f78c2800640 1 --2- 172.21.15.47:0/180062119 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f78a0066e40 0x7f78a0069300 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f78c40cbc50 tx=0x7f78b4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:01.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.389+0000 7f78bb400640 1 -- 172.21.15.47:0/180062119 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f78bc0ecfa0 con 0x7f78c4166b60 2024-02-19T15:25:01.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:01.609+0000 7f78c9400640 1 -- 172.21.15.47:0/180062119 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7f78c4075360 con 0x7f78c4166b60 2024-02-19T15:25:02.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.521+0000 7f78bb400640 1 -- 172.21.15.47:0/180062119 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v179) v1 ==== 143+0+0 (secure 0 0 0) 0x7f78bc0f0660 con 0x7f78c4166b60 2024-02-19T15:25:02.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.525+0000 7f78c9400640 1 -- 172.21.15.47:0/180062119 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f78a0066e40 msgr2=0x7f78a0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:02.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.525+0000 7f78c9400640 1 --2- 172.21.15.47:0/180062119 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f78a0066e40 0x7f78a0069300 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f78c40cbc50 tx=0x7f78b4009040 comp rx=0 tx=0).stop 2024-02-19T15:25:02.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.525+0000 7f78c9400640 1 -- 172.21.15.47:0/180062119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78c4166b60 msgr2=0x7f78c41684d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:02.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.525+0000 7f78c9400640 1 --2- 172.21.15.47:0/180062119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78c4166b60 0x7f78c41684d0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f78bc0af040 tx=0x7f78bc0197a0 comp rx=0 tx=0).stop 2024-02-19T15:25:02.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.525+0000 7f78c9400640 1 -- 172.21.15.47:0/180062119 shutdown_connections 2024-02-19T15:25:02.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.525+0000 7f78c9400640 1 --2- 172.21.15.47:0/180062119 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f78a0066e40 0x7f78a0069300 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:02.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.525+0000 7f78c9400640 1 --2- 172.21.15.47:0/180062119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78c4166b60 0x7f78c41684d0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:02.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.525+0000 7f78c9400640 1 --2- 172.21.15.47:0/180062119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f78c40cb520 0x7f78c4167f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:02.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.525+0000 7f78c9400640 1 --2- 172.21.15.47:0/180062119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c406aeb0 0x7f78c40ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:02.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.525+0000 7f78c9400640 1 -- 172.21.15.47:0/180062119 >> 172.21.15.47:0/180062119 conn(0x7f78c40d0ad0 msgr2=0x7f78c40b0590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:02.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.525+0000 7f78c9400640 1 -- 172.21.15.47:0/180062119 shutdown_connections 2024-02-19T15:25:02.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.525+0000 7f78c9400640 1 -- 172.21.15.47:0/180062119 wait complete. 2024-02-19T15:25:02.551 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-19T15:25:02.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.693+0000 7f0f5d200640 1 Processor -- start 2024-02-19T15:25:02.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.697+0000 7f0f5d200640 1 -- start start 2024-02-19T15:25:02.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.697+0000 7f0f5d200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0f5815e160 0x7f0f580d1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:02.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.697+0000 7f0f5d200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0f5806f220 0x7f0f5806dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:02.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.697+0000 7f0f5d200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f5806e010 0x7f0f5806aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:02.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.697+0000 7f0f5d200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f58071a70 con 0x7f0f5815e160 2024-02-19T15:25:02.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.697+0000 7f0f5d200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f5806f6b0 con 0x7f0f5806f220 2024-02-19T15:25:02.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.697+0000 7f0f5d200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5806e440 con 0x7f0f5806e010 2024-02-19T15:25:02.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.697+0000 7f0f55e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0f5815e160 0x7f0f580d1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:02.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.697+0000 7f0f55e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0f5815e160 0x7f0f580d1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52072/0 (socket says 172.21.15.47:52072) 2024-02-19T15:25:02.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.697+0000 7f0f55e00640 1 -- 172.21.15.47:0/3637891897 learned_addr learned my addr 172.21.15.47:0/3637891897 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:02.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.697+0000 7f0f4d400640 1 --2- 172.21.15.47:0/3637891897 >> v2:172.21.15.116:3300/0 conn(0x7f0f5806f220 0x7f0f5806dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:02.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.697+0000 7f0f56800640 1 --2- 172.21.15.47:0/3637891897 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f5806e010 0x7f0f5806aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:02.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.697+0000 7f0f55e00640 1 -- 172.21.15.47:0/3637891897 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f5806e010 msgr2=0x7f0f5806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:02.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.697+0000 7f0f55e00640 1 --2- 172.21.15.47:0/3637891897 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f5806e010 0x7f0f5806aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:02.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.697+0000 7f0f55e00640 1 -- 172.21.15.47:0/3637891897 >> v2:172.21.15.116:3300/0 conn(0x7f0f5806f220 msgr2=0x7f0f5806dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:02.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.697+0000 7f0f55e00640 1 --2- 172.21.15.47:0/3637891897 >> v2:172.21.15.116:3300/0 conn(0x7f0f5806f220 0x7f0f5806dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:02.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.697+0000 7f0f55e00640 1 -- 172.21.15.47:0/3637891897 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f580ced30 con 0x7f0f5815e160 2024-02-19T15:25:02.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.701+0000 7f0f55e00640 1 --2- 172.21.15.47:0/3637891897 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f5815e160 0x7f0f580d1820 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0f580cd760 tx=0x7f0f4801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=263a525264fc6c36 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:02.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.701+0000 7f0f55400640 1 -- 172.21.15.47:0/3637891897 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f480aa620 con 0x7f0f5815e160 2024-02-19T15:25:02.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.701+0000 7f0f55400640 1 -- 172.21.15.47:0/3637891897 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f480aa780 con 0x7f0f5815e160 2024-02-19T15:25:02.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.701+0000 7f0f55400640 1 -- 172.21.15.47:0/3637891897 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f480aab10 con 0x7f0f5815e160 2024-02-19T15:25:02.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.701+0000 7f0f5d200640 1 -- 172.21.15.47:0/3637891897 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f5815e160 msgr2=0x7f0f580d1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:02.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.701+0000 7f0f5d200640 1 --2- 172.21.15.47:0/3637891897 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f5815e160 0x7f0f580d1820 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0f580cd760 tx=0x7f0f4801c500 comp rx=0 tx=0).stop 2024-02-19T15:25:02.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.701+0000 7f0f5d200640 1 -- 172.21.15.47:0/3637891897 shutdown_connections 2024-02-19T15:25:02.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.701+0000 7f0f5d200640 1 --2- 172.21.15.47:0/3637891897 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f5806e010 0x7f0f5806aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:02.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.701+0000 7f0f5d200640 1 --2- 172.21.15.47:0/3637891897 >> v2:172.21.15.116:3300/0 conn(0x7f0f5806f220 0x7f0f5806dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:02.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.701+0000 7f0f5d200640 1 --2- 172.21.15.47:0/3637891897 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f5815e160 0x7f0f580d1820 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:02.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.701+0000 7f0f5d200640 1 -- 172.21.15.47:0/3637891897 >> 172.21.15.47:0/3637891897 conn(0x7f0f580d0d30 msgr2=0x7f0f580d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:02.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.701+0000 7f0f5d200640 1 -- 172.21.15.47:0/3637891897 shutdown_connections 2024-02-19T15:25:02.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.701+0000 7f0f5d200640 1 -- 172.21.15.47:0/3637891897 wait complete. 2024-02-19T15:25:02.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.701+0000 7f0f5d200640 1 Processor -- start 2024-02-19T15:25:02.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.701+0000 7f0f5d200640 1 -- start start 2024-02-19T15:25:02.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f5d200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f5806e010 0x7f0f580b3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:02.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f5d200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f5806f220 0x7f0f5814cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:02.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f5d200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f5815e160 0x7f0f5814d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:02.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f5d200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5815fd10 con 0x7f0f5806e010 2024-02-19T15:25:02.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f5d200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5815f090 con 0x7f0f5815e160 2024-02-19T15:25:02.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f5d200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5815f370 con 0x7f0f5806f220 2024-02-19T15:25:02.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f55e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f5806e010 0x7f0f580b3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:02.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f55e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f5806e010 0x7f0f580b3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52076/0 (socket says 172.21.15.47:52076) 2024-02-19T15:25:02.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f55e00640 1 -- 172.21.15.47:0/3504107243 learned_addr learned my addr 172.21.15.47:0/3504107243 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:02.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f4d400640 1 --2- 172.21.15.47:0/3504107243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f5806f220 0x7f0f5814cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:02.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f56800640 1 --2- 172.21.15.47:0/3504107243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f5815e160 0x7f0f5814d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:02.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f55e00640 1 -- 172.21.15.47:0/3504107243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f5806f220 msgr2=0x7f0f5814cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:02.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f55e00640 1 --2- 172.21.15.47:0/3504107243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f5806f220 0x7f0f5814cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:02.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f55e00640 1 -- 172.21.15.47:0/3504107243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f5815e160 msgr2=0x7f0f5814d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:02.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f55e00640 1 --2- 172.21.15.47:0/3504107243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f5815e160 0x7f0f5814d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:02.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f55e00640 1 -- 172.21.15.47:0/3504107243 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f480094d0 con 0x7f0f5806e010 2024-02-19T15:25:02.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f4d400640 1 --2- 172.21.15.47:0/3504107243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f5806f220 0x7f0f5814cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:02.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f55e00640 1 --2- 172.21.15.47:0/3504107243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f5806e010 0x7f0f580b3df0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f0f480ac040 tx=0x7f0f4801ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:02.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f4ea00640 1 -- 172.21.15.47:0/3504107243 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f480bd3f0 con 0x7f0f5806e010 2024-02-19T15:25:02.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f4ea00640 1 -- 172.21.15.47:0/3504107243 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f480bd570 con 0x7f0f5806e010 2024-02-19T15:25:02.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f5d200640 1 -- 172.21.15.47:0/3504107243 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5815f650 con 0x7f0f5806e010 2024-02-19T15:25:02.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f4ea00640 1 -- 172.21.15.47:0/3504107243 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f480bd920 con 0x7f0f5806e010 2024-02-19T15:25:02.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f5d200640 1 -- 172.21.15.47:0/3504107243 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f580b4330 con 0x7f0f5806e010 2024-02-19T15:25:02.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f5d200640 1 -- 172.21.15.47:0/3504107243 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f5815fe50 con 0x7f0f5806e010 2024-02-19T15:25:02.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.705+0000 7f0f4ea00640 1 -- 172.21.15.47:0/3504107243 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0f480c6440 con 0x7f0f5806e010 2024-02-19T15:25:02.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.709+0000 7f0f4ea00640 1 --2- 172.21.15.47:0/3504107243 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0f34066e40 0x7f0f34069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:02.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.709+0000 7f0f4ea00640 1 -- 172.21.15.47:0/3504107243 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0f4811f050 con 0x7f0f5806e010 2024-02-19T15:25:02.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.709+0000 7f0f4d400640 1 --2- 172.21.15.47:0/3504107243 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0f34066e40 0x7f0f34069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:02.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.709+0000 7f0f4d400640 1 --2- 172.21.15.47:0/3504107243 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0f34066e40 0x7f0f34069300 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0f580b0440 tx=0x7f0f40008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:02.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.709+0000 7f0f4ea00640 1 -- 172.21.15.47:0/3504107243 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0f480ec130 con 0x7f0f5806e010 2024-02-19T15:25:02.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.873+0000 7f0f5d200640 1 -- 172.21.15.47:0/3504107243 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0f580b1f40 con 0x7f0f5806e010 2024-02-19T15:25:02.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.877+0000 7f0f4ea00640 1 -- 172.21.15.47:0/3504107243 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0f480ef7f0 con 0x7f0f5806e010 2024-02-19T15:25:02.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.877+0000 7f0f5d200640 1 -- 172.21.15.47:0/3504107243 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0f34066e40 msgr2=0x7f0f34069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:02.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.877+0000 7f0f5d200640 1 --2- 172.21.15.47:0/3504107243 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0f34066e40 0x7f0f34069300 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0f580b0440 tx=0x7f0f40008040 comp rx=0 tx=0).stop 2024-02-19T15:25:02.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.877+0000 7f0f5d200640 1 -- 172.21.15.47:0/3504107243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f5806e010 msgr2=0x7f0f580b3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:02.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.877+0000 7f0f5d200640 1 --2- 172.21.15.47:0/3504107243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f5806e010 0x7f0f580b3df0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f0f480ac040 tx=0x7f0f4801ad40 comp rx=0 tx=0).stop 2024-02-19T15:25:02.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.877+0000 7f0f5d200640 1 -- 172.21.15.47:0/3504107243 shutdown_connections 2024-02-19T15:25:02.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.877+0000 7f0f5d200640 1 --2- 172.21.15.47:0/3504107243 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0f34066e40 0x7f0f34069300 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:02.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.877+0000 7f0f5d200640 1 --2- 172.21.15.47:0/3504107243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f5815e160 0x7f0f5814d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:02.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.877+0000 7f0f5d200640 1 --2- 172.21.15.47:0/3504107243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f5806f220 0x7f0f5814cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:02.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.877+0000 7f0f5d200640 1 --2- 172.21.15.47:0/3504107243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f5806e010 0x7f0f580b3df0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:02.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.877+0000 7f0f5d200640 1 -- 172.21.15.47:0/3504107243 >> 172.21.15.47:0/3504107243 conn(0x7f0f580d0d30 msgr2=0x7f0f580d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:02.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.877+0000 7f0f5d200640 1 -- 172.21.15.47:0/3504107243 shutdown_connections 2024-02-19T15:25:02.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:02.877+0000 7f0f5d200640 1 -- 172.21.15.47:0/3504107243 wait complete. 2024-02-19T15:25:02.901 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T15:25:02.901 DEBUG:teuthology.orchestra.run.smithi047:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:25:02.973 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:25:03.018 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T15:25:03.018 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:25:03.018 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:25:03.027 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:25:03.027 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:25:03.027 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:25:03.067 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:25:03.067 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:25:03.067 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:25:03.078 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:25:03.078 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:25:03.078 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:25:03.119 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:25:03.119 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-02-19T15:25:03.119 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:25:03.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.261+0000 7fa1d2200640 1 Processor -- start 2024-02-19T15:25:03.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.261+0000 7fa1d2200640 1 -- start start 2024-02-19T15:25:03.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.261+0000 7fa1d2200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa1cc06d650 0x7fa1cc0d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:03.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.261+0000 7fa1d2200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa1cc0cded0 0x7fa1cc0ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:03.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.261+0000 7fa1d2200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1cc0cce30 con 0x7fa1cc0ceb40 2024-02-19T15:25:03.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.261+0000 7fa1d2200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1cc0cefd0 con 0x7fa1cc0cded0 2024-02-19T15:25:03.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.261+0000 7fa1d2200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1cc0cd110 con 0x7fa1cc06d650 2024-02-19T15:25:03.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.261+0000 7fa1c2a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa1cc0cded0 0x7fa1cc0ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:03.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.261+0000 7fa1cbe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa1cc0ceb40 0x7fa1cc161d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54816/0 (socket says 172.21.15.47:54816) 2024-02-19T15:25:03.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.261+0000 7fa1c2a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa1cc0cded0 0x7fa1cc0ce600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52086/0 (socket says 172.21.15.47:52086) 2024-02-19T15:25:03.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.261+0000 7fa1cbe00640 1 -- 172.21.15.47:0/4279555904 learned_addr learned my addr 172.21.15.47:0/4279555904 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:03.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.261+0000 7fa1cb400640 1 --2- 172.21.15.47:0/4279555904 >> v2:172.21.15.116:3300/0 conn(0x7fa1cc06d650 0x7fa1cc0d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:03.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.261+0000 7fa1c2a00640 1 -- 172.21.15.47:0/4279555904 >> v2:172.21.15.116:3300/0 conn(0x7fa1cc06d650 msgr2=0x7fa1cc0d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:03.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.261+0000 7fa1c2a00640 1 --2- 172.21.15.47:0/4279555904 >> v2:172.21.15.116:3300/0 conn(0x7fa1cc06d650 0x7fa1cc0d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:03.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.261+0000 7fa1c2a00640 1 -- 172.21.15.47:0/4279555904 >> v1:172.21.15.47:6789/0 conn(0x7fa1cc0ceb40 legacy=0x7fa1cc161d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:03.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.261+0000 7fa1caa00640 1 -- 172.21.15.47:0/4279555904 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1110107626 0 0) 0x7fa1cc0cce30 con 0x7fa1cc0ceb40 2024-02-19T15:25:03.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.261+0000 7fa1c2a00640 1 -- 172.21.15.47:0/4279555904 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1cc0cd3f0 con 0x7fa1cc0cded0 2024-02-19T15:25:03.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.261+0000 7fa1cb400640 1 --2- 172.21.15.47:0/4279555904 >> v2:172.21.15.116:3300/0 conn(0x7fa1cc06d650 0x7fa1cc0d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:25:03.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.265+0000 7fa1c2a00640 1 --2- 172.21.15.47:0/4279555904 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1cc0cded0 0x7fa1cc0ce600 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc072160 tx=0x7fa1b401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1665a932fef85a5e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:03.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.265+0000 7fa1caa00640 1 -- 172.21.15.47:0/4279555904 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b40ab600 con 0x7fa1cc0cded0 2024-02-19T15:25:03.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.265+0000 7fa1caa00640 1 -- 172.21.15.47:0/4279555904 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b40a7050 con 0x7fa1cc0cded0 2024-02-19T15:25:03.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.265+0000 7fa1caa00640 1 -- 172.21.15.47:0/4279555904 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b40ab980 con 0x7fa1cc0cded0 2024-02-19T15:25:03.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.265+0000 7fa1d2200640 1 -- 172.21.15.47:0/4279555904 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1cc0cded0 msgr2=0x7fa1cc0ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:03.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.265+0000 7fa1d2200640 1 --2- 172.21.15.47:0/4279555904 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1cc0cded0 0x7fa1cc0ce600 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc072160 tx=0x7fa1b401c3c0 comp rx=0 tx=0).stop 2024-02-19T15:25:03.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.265+0000 7fa1d2200640 1 -- 172.21.15.47:0/4279555904 shutdown_connections 2024-02-19T15:25:03.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.265+0000 7fa1d2200640 1 --2- 172.21.15.47:0/4279555904 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1cc0cded0 0x7fa1cc0ce600 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:03.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.265+0000 7fa1d2200640 1 --2- 172.21.15.47:0/4279555904 >> v2:172.21.15.116:3300/0 conn(0x7fa1cc06d650 0x7fa1cc0d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:03.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.265+0000 7fa1d2200640 1 -- 172.21.15.47:0/4279555904 >> 172.21.15.47:0/4279555904 conn(0x7fa1cc0d1bf0 msgr2=0x7fa1cc0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:03.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.265+0000 7fa1d2200640 1 -- 172.21.15.47:0/4279555904 shutdown_connections 2024-02-19T15:25:03.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.265+0000 7fa1d2200640 1 -- 172.21.15.47:0/4279555904 wait complete. 2024-02-19T15:25:03.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.265+0000 7fa1d2200640 1 Processor -- start 2024-02-19T15:25:03.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.265+0000 7fa1d2200640 1 -- start start 2024-02-19T15:25:03.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1d2200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1cc06d650 0x7fa1cc0b6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:03.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1d2200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1cc0cded0 0x7fa1cc153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:03.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1d2200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1cc0ceb40 0x7fa1cc15c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:03.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1d2200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1cc0b6a90 con 0x7fa1cc0cded0 2024-02-19T15:25:03.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1d2200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1cc0c6c30 con 0x7fa1cc0ceb40 2024-02-19T15:25:03.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1d2200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1cc0c6f10 con 0x7fa1cc06d650 2024-02-19T15:25:03.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1c2a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1cc0cded0 0x7fa1cc153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:03.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1c2a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1cc0cded0 0x7fa1cc153c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52098/0 (socket says 172.21.15.47:52098) 2024-02-19T15:25:03.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1c2a00640 1 -- 172.21.15.47:0/3063945329 learned_addr learned my addr 172.21.15.47:0/3063945329 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:03.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1cb400640 1 --2- 172.21.15.47:0/3063945329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1cc06d650 0x7fa1cc0b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:03.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1cbe00640 1 --2- 172.21.15.47:0/3063945329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1cc0ceb40 0x7fa1cc15c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:03.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1c2a00640 1 -- 172.21.15.47:0/3063945329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1cc06d650 msgr2=0x7fa1cc0b6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:03.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1c2a00640 1 --2- 172.21.15.47:0/3063945329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1cc06d650 0x7fa1cc0b6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:03.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1c2a00640 1 -- 172.21.15.47:0/3063945329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1cc0ceb40 msgr2=0x7fa1cc15c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:03.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1c2a00640 1 --2- 172.21.15.47:0/3063945329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1cc0ceb40 0x7fa1cc15c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:03.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1c2a00640 1 -- 172.21.15.47:0/3063945329 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b4009420 con 0x7fa1cc0cded0 2024-02-19T15:25:03.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1c2a00640 1 --2- 172.21.15.47:0/3063945329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1cc0cded0 0x7fa1cc153c30 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4019ff0 tx=0x7fa1b401ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:03.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1c3e00640 1 -- 172.21.15.47:0/3063945329 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b4004020 con 0x7fa1cc0cded0 2024-02-19T15:25:03.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1c3e00640 1 -- 172.21.15.47:0/3063945329 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b4004180 con 0x7fa1cc0cded0 2024-02-19T15:25:03.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1c3e00640 1 -- 172.21.15.47:0/3063945329 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b40044f0 con 0x7fa1cc0cded0 2024-02-19T15:25:03.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1d2200640 1 -- 172.21.15.47:0/3063945329 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1cc0c71f0 con 0x7fa1cc0cded0 2024-02-19T15:25:03.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1d2200640 1 -- 172.21.15.47:0/3063945329 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1cc0c77b0 con 0x7fa1cc0cded0 2024-02-19T15:25:03.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.269+0000 7fa1d2200640 1 -- 172.21.15.47:0/3063945329 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1cc0c79a0 con 0x7fa1cc0cded0 2024-02-19T15:25:03.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.273+0000 7fa1c3e00640 1 -- 172.21.15.47:0/3063945329 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa1b4003620 con 0x7fa1cc0cded0 2024-02-19T15:25:03.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.273+0000 7fa1c3e00640 1 --2- 172.21.15.47:0/3063945329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa1a8066e40 0x7fa1a8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:03.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.273+0000 7fa1c3e00640 1 -- 172.21.15.47:0/3063945329 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa1b411ec90 con 0x7fa1cc0cded0 2024-02-19T15:25:03.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.273+0000 7fa1cb400640 1 --2- 172.21.15.47:0/3063945329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa1a8066e40 0x7fa1a8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:03.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.273+0000 7fa1cb400640 1 --2- 172.21.15.47:0/3063945329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa1a8066e40 0x7fa1a8069300 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc006fd0 tx=0x7fa1bc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:03.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.277+0000 7fa1c3e00640 1 -- 172.21.15.47:0/3063945329 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa1b40ebf30 con 0x7fa1cc0cded0 2024-02-19T15:25:03.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.481+0000 7fa1d2200640 1 -- 172.21.15.47:0/3063945329 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1cc075360 con 0x7fa1cc0cded0 2024-02-19T15:25:03.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.481+0000 7fa1c3e00640 1 -- 172.21.15.47:0/3063945329 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+3705 (secure 0 0 0) 0x7fa1b40bb090 con 0x7fa1cc0cded0 2024-02-19T15:25:03.486 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:03.486 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":35,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5384,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2825837364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2825837364},{"type":"v1","addr":"172.21.15.47:6835","nonce":2825837364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2251385237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2251385237},{"type":"v1","addr":"172.21.15.116:6837","nonce":2251385237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5818,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/2438199468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2438199468},{"type":"v1","addr":"172.21.15.116:6836","nonce":2438199468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":6249,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/1103163153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1103163153},{"type":"v1","addr":"172.21.15.47:6834","nonce":1103163153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[]} 2024-02-19T15:25:03.486 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 35 2024-02-19T15:25:03.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.485+0000 7fa1d2200640 1 -- 172.21.15.47:0/3063945329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa1a8066e40 msgr2=0x7fa1a8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:03.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.485+0000 7fa1d2200640 1 --2- 172.21.15.47:0/3063945329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa1a8066e40 0x7fa1a8069300 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc006fd0 tx=0x7fa1bc008040 comp rx=0 tx=0).stop 2024-02-19T15:25:03.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.485+0000 7fa1d2200640 1 -- 172.21.15.47:0/3063945329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1cc0cded0 msgr2=0x7fa1cc153c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:03.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.485+0000 7fa1d2200640 1 --2- 172.21.15.47:0/3063945329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1cc0cded0 0x7fa1cc153c30 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4019ff0 tx=0x7fa1b401ac00 comp rx=0 tx=0).stop 2024-02-19T15:25:03.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.485+0000 7fa1d2200640 1 -- 172.21.15.47:0/3063945329 shutdown_connections 2024-02-19T15:25:03.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.485+0000 7fa1d2200640 1 --2- 172.21.15.47:0/3063945329 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa1a8066e40 0x7fa1a8069300 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:03.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.485+0000 7fa1d2200640 1 --2- 172.21.15.47:0/3063945329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1cc0ceb40 0x7fa1cc15c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:03.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.485+0000 7fa1d2200640 1 --2- 172.21.15.47:0/3063945329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1cc0cded0 0x7fa1cc153c30 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:03.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.485+0000 7fa1d2200640 1 --2- 172.21.15.47:0/3063945329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1cc06d650 0x7fa1cc0b6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:03.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.485+0000 7fa1d2200640 1 -- 172.21.15.47:0/3063945329 >> 172.21.15.47:0/3063945329 conn(0x7fa1cc0d1bf0 msgr2=0x7fa1cc161110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:03.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.485+0000 7fa1d2200640 1 -- 172.21.15.47:0/3063945329 shutdown_connections 2024-02-19T15:25:03.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.485+0000 7fa1d2200640 1 -- 172.21.15.47:0/3063945329 wait complete. 2024-02-19T15:25:03.507 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T15:25:03.507 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-19T15:25:03.507 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T15:25:03.507 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:25:03.507 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T15:25:03.507 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-19T15:25:03.508 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T15:25:03.508 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:25:03.508 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T15:25:03.508 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-19T15:25:03.508 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T15:25:03.508 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:25:03.508 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T15:25:03.508 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-19T15:25:03.509 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T15:25:03.509 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:25:03.520 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T15:25:03.520 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T15:25:03.521 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T15:25:03.521 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T15:25:03.523 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T15:25:03.523 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T15:25:03.525 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T15:25:03.525 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T15:25:03.569 INFO:tasks.ceph.mds.a:Started 2024-02-19T15:25:03.569 DEBUG:teuthology.parallel:result is None 2024-02-19T15:25:03.570 INFO:tasks.ceph.mds.c:Started 2024-02-19T15:25:03.570 DEBUG:teuthology.parallel:result is None 2024-02-19T15:25:03.572 INFO:tasks.ceph.mds.b:Started 2024-02-19T15:25:03.573 DEBUG:teuthology.parallel:result is None 2024-02-19T15:25:03.575 INFO:tasks.ceph.mds.d:Started 2024-02-19T15:25:03.575 DEBUG:teuthology.parallel:result is None 2024-02-19T15:25:03.575 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-19T15:25:03.657 INFO:tasks.ceph.mds.c.smithi047.stdout:starting mds.c at 2024-02-19T15:25:03.658 INFO:tasks.ceph.mds.a.smithi047.stdout:starting mds.a at 2024-02-19T15:25:03.662 INFO:tasks.ceph.mds.b.smithi116.stdout:starting mds.b at 2024-02-19T15:25:03.662 INFO:tasks.ceph.mds.d.smithi116.stdout:starting mds.d at 2024-02-19T15:25:03.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.717+0000 7fc151000640 1 Processor -- start 2024-02-19T15:25:03.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.717+0000 7fc151000640 1 -- start start 2024-02-19T15:25:03.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.721+0000 7fc151000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc14c15b1a0 0x7fc14c15b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:03.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.721+0000 7fc151000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc14c06c9a0 0x7fc14c15bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:03.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.721+0000 7fc151000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc14c15c000 con 0x7fc14c071350 2024-02-19T15:25:03.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.721+0000 7fc151000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc14c0718c0 con 0x7fc14c15b1a0 2024-02-19T15:25:03.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.721+0000 7fc151000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c071ba0 con 0x7fc14c06c9a0 2024-02-19T15:25:03.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.721+0000 7fc143e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc14c071350 0x7fc14c15c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38240/0 (socket says 172.21.15.47:38240) 2024-02-19T15:25:03.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.721+0000 7fc143e00640 1 -- 172.21.15.47:0/3499764333 learned_addr learned my addr 172.21.15.47:0/3499764333 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:03.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.721+0000 7fc149e00640 1 --2- 172.21.15.47:0/3499764333 >> v2:172.21.15.116:3300/0 conn(0x7fc14c15b1a0 0x7fc14c15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:03.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.721+0000 7fc14a800640 1 --2- 172.21.15.47:0/3499764333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc14c06c9a0 0x7fc14c15bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:03.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.721+0000 7fc149e00640 1 -- 172.21.15.47:0/3499764333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc14c06c9a0 msgr2=0x7fc14c15bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:03.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.721+0000 7fc149e00640 1 --2- 172.21.15.47:0/3499764333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc14c06c9a0 0x7fc14c15bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:03.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.721+0000 7fc149e00640 1 -- 172.21.15.47:0/3499764333 >> v1:172.21.15.47:6789/0 conn(0x7fc14c071350 legacy=0x7fc14c15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:03.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.721+0000 7fc149e00640 1 -- 172.21.15.47:0/3499764333 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14c071e80 con 0x7fc14c15b1a0 2024-02-19T15:25:03.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.721+0000 7fc149400640 1 -- 172.21.15.47:0/3499764333 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176104691 0 0) 0x7fc14c15c000 con 0x7fc14c071350 2024-02-19T15:25:03.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.725+0000 7fc149e00640 1 --2- 172.21.15.47:0/3499764333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc14c15b1a0 0x7fc14c15b580 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc13c0097f0 tx=0x7fc13c01c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a97b38330b73ffa server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:03.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.725+0000 7fc149400640 1 -- 172.21.15.47:0/3499764333 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c0aa5f0 con 0x7fc14c15b1a0 2024-02-19T15:25:03.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.725+0000 7fc149400640 1 -- 172.21.15.47:0/3499764333 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13c0aa750 con 0x7fc14c15b1a0 2024-02-19T15:25:03.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.725+0000 7fc149400640 1 -- 172.21.15.47:0/3499764333 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c0aaae0 con 0x7fc14c15b1a0 2024-02-19T15:25:03.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.725+0000 7fc151000640 1 -- 172.21.15.47:0/3499764333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc14c15b1a0 msgr2=0x7fc14c15b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:03.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.725+0000 7fc151000640 1 --2- 172.21.15.47:0/3499764333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc14c15b1a0 0x7fc14c15b580 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc13c0097f0 tx=0x7fc13c01c470 comp rx=0 tx=0).stop 2024-02-19T15:25:03.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.725+0000 7fc151000640 1 -- 172.21.15.47:0/3499764333 shutdown_connections 2024-02-19T15:25:03.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.725+0000 7fc151000640 1 --2- 172.21.15.47:0/3499764333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc14c06c9a0 0x7fc14c15bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:03.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.725+0000 7fc151000640 1 --2- 172.21.15.47:0/3499764333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc14c15b1a0 0x7fc14c15b580 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:03.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.725+0000 7fc151000640 1 -- 172.21.15.47:0/3499764333 >> 172.21.15.47:0/3499764333 conn(0x7fc14c0b16a0 msgr2=0x7fc14c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:03.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.725+0000 7fc151000640 1 -- 172.21.15.47:0/3499764333 shutdown_connections 2024-02-19T15:25:03.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.725+0000 7fc151000640 1 -- 172.21.15.47:0/3499764333 wait complete. 2024-02-19T15:25:03.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.725+0000 7fc151000640 1 Processor -- start 2024-02-19T15:25:03.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.725+0000 7fc151000640 1 -- start start 2024-02-19T15:25:03.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.725+0000 7fc151000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc14c06c9a0 0x7fc14c1559c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:03.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc151000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc14c071350 0x7fc14c0ae770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:03.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc151000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc14c15b1a0 0x7fc14c14bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:03.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc151000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c14cdb0 con 0x7fc14c06c9a0 2024-02-19T15:25:03.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc151000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c0aecb0 con 0x7fc14c071350 2024-02-19T15:25:03.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc151000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c155530 con 0x7fc14c15b1a0 2024-02-19T15:25:03.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc149e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc14c06c9a0 0x7fc14c1559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:03.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc149e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc14c06c9a0 0x7fc14c1559c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39732/0 (socket says 172.21.15.47:39732) 2024-02-19T15:25:03.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc149e00640 1 -- 172.21.15.47:0/2001325740 learned_addr learned my addr 172.21.15.47:0/2001325740 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:03.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc143e00640 1 --2- 172.21.15.47:0/2001325740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc14c071350 0x7fc14c0ae770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:03.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc14a800640 1 --2- 172.21.15.47:0/2001325740 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc14c15b1a0 0x7fc14c14bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:03.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc149e00640 1 -- 172.21.15.47:0/2001325740 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc14c15b1a0 msgr2=0x7fc14c14bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:03.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc149e00640 1 --2- 172.21.15.47:0/2001325740 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc14c15b1a0 0x7fc14c14bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:03.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc149e00640 1 -- 172.21.15.47:0/2001325740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc14c071350 msgr2=0x7fc14c0ae770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:03.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc149e00640 1 --2- 172.21.15.47:0/2001325740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc14c071350 0x7fc14c0ae770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:03.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc149e00640 1 -- 172.21.15.47:0/2001325740 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c009420 con 0x7fc14c06c9a0 2024-02-19T15:25:03.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc14a800640 1 --2- 172.21.15.47:0/2001325740 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc14c15b1a0 0x7fc14c14bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:25:03.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc149e00640 1 --2- 172.21.15.47:0/2001325740 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc14c06c9a0 0x7fc14c1559c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc13c0ac040 tx=0x7fc13c01acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:03.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc142000640 1 -- 172.21.15.47:0/2001325740 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c0be610 con 0x7fc14c06c9a0 2024-02-19T15:25:03.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc142000640 1 -- 172.21.15.47:0/2001325740 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13c0be770 con 0x7fc14c06c9a0 2024-02-19T15:25:03.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc142000640 1 -- 172.21.15.47:0/2001325740 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c0beb20 con 0x7fc14c06c9a0 2024-02-19T15:25:03.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc151000640 1 -- 172.21.15.47:0/2001325740 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc14c0aeee0 con 0x7fc14c06c9a0 2024-02-19T15:25:03.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc151000640 1 -- 172.21.15.47:0/2001325740 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc14c1551c0 con 0x7fc14c06c9a0 2024-02-19T15:25:03.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc151000640 1 -- 172.21.15.47:0/2001325740 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc14c075360 con 0x7fc14c06c9a0 2024-02-19T15:25:03.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.729+0000 7fc142000640 1 -- 172.21.15.47:0/2001325740 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc13c0bec80 con 0x7fc14c06c9a0 2024-02-19T15:25:03.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.733+0000 7fc142000640 1 --2- 172.21.15.47:0/2001325740 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc128066e40 0x7fc128069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:03.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.733+0000 7fc142000640 1 -- 172.21.15.47:0/2001325740 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc13c0a7070 con 0x7fc14c06c9a0 2024-02-19T15:25:03.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.733+0000 7fc143e00640 1 --2- 172.21.15.47:0/2001325740 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc128066e40 0x7fc128069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:03.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.733+0000 7fc143e00640 1 --2- 172.21.15.47:0/2001325740 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc128066e40 0x7fc128069300 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc14c1586b0 tx=0x7fc134007d60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:03.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.737+0000 7fc142000640 1 -- 172.21.15.47:0/2001325740 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc13c0ec5d0 con 0x7fc14c06c9a0 2024-02-19T15:25:03.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:03.909+0000 7fc151000640 1 -- 172.21.15.47:0/2001325740 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc14c15a120 con 0x7fc14c06c9a0 2024-02-19T15:25:04.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:04.889+0000 7fc142000640 1 -- 172.21.15.47:0/2001325740 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v65) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc13c0efc90 con 0x7fc14c06c9a0 2024-02-19T15:25:04.893 INFO:teuthology.orchestra.run.smithi047.stderr: removed all blocklist entries 2024-02-19T15:25:04.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:04.893+0000 7fc151000640 1 -- 172.21.15.47:0/2001325740 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc128066e40 msgr2=0x7fc128069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:04.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:04.893+0000 7fc151000640 1 --2- 172.21.15.47:0/2001325740 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc128066e40 0x7fc128069300 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc14c1586b0 tx=0x7fc134007d60 comp rx=0 tx=0).stop 2024-02-19T15:25:04.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:04.893+0000 7fc151000640 1 -- 172.21.15.47:0/2001325740 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc14c06c9a0 msgr2=0x7fc14c1559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:04.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:04.893+0000 7fc151000640 1 --2- 172.21.15.47:0/2001325740 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc14c06c9a0 0x7fc14c1559c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc13c0ac040 tx=0x7fc13c01acb0 comp rx=0 tx=0).stop 2024-02-19T15:25:04.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:04.893+0000 7fc151000640 1 -- 172.21.15.47:0/2001325740 shutdown_connections 2024-02-19T15:25:04.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:04.893+0000 7fc151000640 1 --2- 172.21.15.47:0/2001325740 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc128066e40 0x7fc128069300 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:04.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:04.893+0000 7fc151000640 1 --2- 172.21.15.47:0/2001325740 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc14c15b1a0 0x7fc14c14bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:04.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:04.893+0000 7fc151000640 1 --2- 172.21.15.47:0/2001325740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc14c071350 0x7fc14c0ae770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:04.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:04.893+0000 7fc151000640 1 --2- 172.21.15.47:0/2001325740 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc14c06c9a0 0x7fc14c1559c0 secure :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc13c0ac040 tx=0x7fc13c01acb0 comp rx=0 tx=0).stop 2024-02-19T15:25:04.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:04.893+0000 7fc151000640 1 -- 172.21.15.47:0/2001325740 >> 172.21.15.47:0/2001325740 conn(0x7fc14c0b16a0 msgr2=0x7fc14c0af5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:04.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:04.893+0000 7fc151000640 1 -- 172.21.15.47:0/2001325740 shutdown_connections 2024-02-19T15:25:04.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:04.893+0000 7fc151000640 1 -- 172.21.15.47:0/2001325740 wait complete. 2024-02-19T15:25:04.919 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-19T15:25:05.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.061+0000 7fa27f000640 1 Processor -- start 2024-02-19T15:25:05.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.061+0000 7fa27f000640 1 -- start start 2024-02-19T15:25:05.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.065+0000 7fa27f000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa27806aeb0 0x7fa27806b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:05.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.065+0000 7fa27f000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa2780cb520 0x7fa2780cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:05.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.065+0000 7fa27f000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2780ad840 con 0x7fa2780719e0 2024-02-19T15:25:05.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.065+0000 7fa27f000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2780c62f0 con 0x7fa2780cb520 2024-02-19T15:25:05.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.065+0000 7fa27f000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa27806b7d0 con 0x7fa27806aeb0 2024-02-19T15:25:05.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.065+0000 7fa27d000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa2780719e0 0x7fa278071e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38242/0 (socket says 172.21.15.47:38242) 2024-02-19T15:25:05.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.065+0000 7fa27d000640 1 -- 172.21.15.47:0/1712324662 learned_addr learned my addr 172.21.15.47:0/1712324662 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:05.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.065+0000 7fa277e00640 1 --2- 172.21.15.47:0/1712324662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa27806aeb0 0x7fa27806b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:05.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.065+0000 7fa277400640 1 --2- 172.21.15.47:0/1712324662 >> v2:172.21.15.116:3300/0 conn(0x7fa2780cb520 0x7fa2780cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:05.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.065+0000 7fa276a00640 1 -- 172.21.15.47:0/1712324662 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013023000 0 0) 0x7fa2780ad840 con 0x7fa2780719e0 2024-02-19T15:25:05.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.065+0000 7fa276a00640 1 -- 172.21.15.47:0/1712324662 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa25c003530 con 0x7fa2780719e0 2024-02-19T15:25:05.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.065+0000 7fa277e00640 1 -- 172.21.15.47:0/1712324662 >> v2:172.21.15.116:3300/0 conn(0x7fa2780cb520 msgr2=0x7fa2780cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:05.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.065+0000 7fa277e00640 1 --2- 172.21.15.47:0/1712324662 >> v2:172.21.15.116:3300/0 conn(0x7fa2780cb520 0x7fa2780cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.065+0000 7fa277e00640 1 -- 172.21.15.47:0/1712324662 >> v1:172.21.15.47:6789/0 conn(0x7fa2780719e0 legacy=0x7fa278071e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:05.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.065+0000 7fa277e00640 1 -- 172.21.15.47:0/1712324662 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2780cc810 con 0x7fa27806aeb0 2024-02-19T15:25:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.069+0000 7fa277e00640 1 --2- 172.21.15.47:0/1712324662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa27806aeb0 0x7fa27806b290 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa2680097c0 tx=0x7fa26801c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46ee6fb00414a83b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.069+0000 7fa276a00640 1 -- 172.21.15.47:0/1712324662 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2680ab670 con 0x7fa27806aeb0 2024-02-19T15:25:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.069+0000 7fa276a00640 1 -- 172.21.15.47:0/1712324662 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2680ab7d0 con 0x7fa27806aeb0 2024-02-19T15:25:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.069+0000 7fa276a00640 1 -- 172.21.15.47:0/1712324662 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2680abb60 con 0x7fa27806aeb0 2024-02-19T15:25:05.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.069+0000 7fa27f000640 1 -- 172.21.15.47:0/1712324662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa27806aeb0 msgr2=0x7fa27806b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:05.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.069+0000 7fa27f000640 1 --2- 172.21.15.47:0/1712324662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa27806aeb0 0x7fa27806b290 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa2680097c0 tx=0x7fa26801c470 comp rx=0 tx=0).stop 2024-02-19T15:25:05.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.069+0000 7fa27f000640 1 -- 172.21.15.47:0/1712324662 shutdown_connections 2024-02-19T15:25:05.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.069+0000 7fa27f000640 1 --2- 172.21.15.47:0/1712324662 >> v2:172.21.15.116:3300/0 conn(0x7fa2780cb520 0x7fa2780cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.069+0000 7fa27f000640 1 --2- 172.21.15.47:0/1712324662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa27806aeb0 0x7fa27806b290 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.069+0000 7fa27f000640 1 -- 172.21.15.47:0/1712324662 >> 172.21.15.47:0/1712324662 conn(0x7fa2780d0ad0 msgr2=0x7fa2780d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:05.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.069+0000 7fa27f000640 1 -- 172.21.15.47:0/1712324662 shutdown_connections 2024-02-19T15:25:05.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.069+0000 7fa27f000640 1 -- 172.21.15.47:0/1712324662 wait complete. 2024-02-19T15:25:05.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.069+0000 7fa27f000640 1 Processor -- start 2024-02-19T15:25:05.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.069+0000 7fa27f000640 1 -- start start 2024-02-19T15:25:05.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.069+0000 7fa27f000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa27806aeb0 0x7fa2781474c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:05.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa27f000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2780719e0 0x7fa2780c92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:05.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa27f000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2780cb520 0x7fa278147d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:05.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa27f000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27815f190 con 0x7fa27806aeb0 2024-02-19T15:25:05.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa27f000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27815a1a0 con 0x7fa2780cb520 2024-02-19T15:25:05.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa27f000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa27815f3a0 con 0x7fa2780719e0 2024-02-19T15:25:05.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa277e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa27806aeb0 0x7fa2781474c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:05.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa277e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa27806aeb0 0x7fa2781474c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39734/0 (socket says 172.21.15.47:39734) 2024-02-19T15:25:05.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa277e00640 1 -- 172.21.15.47:0/857660810 learned_addr learned my addr 172.21.15.47:0/857660810 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:05.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa277400640 1 --2- 172.21.15.47:0/857660810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2780719e0 0x7fa2780c92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:05.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa27d000640 1 --2- 172.21.15.47:0/857660810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2780cb520 0x7fa278147d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:05.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa277e00640 1 -- 172.21.15.47:0/857660810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2780719e0 msgr2=0x7fa2780c92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:05.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa277e00640 1 --2- 172.21.15.47:0/857660810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2780719e0 0x7fa2780c92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa277e00640 1 -- 172.21.15.47:0/857660810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2780cb520 msgr2=0x7fa278147d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:05.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa277e00640 1 --2- 172.21.15.47:0/857660810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2780cb520 0x7fa278147d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa277e00640 1 -- 172.21.15.47:0/857660810 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa268009420 con 0x7fa27806aeb0 2024-02-19T15:25:05.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa277e00640 1 --2- 172.21.15.47:0/857660810 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa27806aeb0 0x7fa2781474c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fa2680ad040 tx=0x7fa26801acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:05.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa26fe00640 1 -- 172.21.15.47:0/857660810 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2680bf610 con 0x7fa27806aeb0 2024-02-19T15:25:05.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa26fe00640 1 -- 172.21.15.47:0/857660810 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2680bf7b0 con 0x7fa27806aeb0 2024-02-19T15:25:05.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa26fe00640 1 -- 172.21.15.47:0/857660810 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2680bfb60 con 0x7fa27806aeb0 2024-02-19T15:25:05.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa27f000640 1 -- 172.21.15.47:0/857660810 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa27815f680 con 0x7fa27806aeb0 2024-02-19T15:25:05.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa27f000640 1 -- 172.21.15.47:0/857660810 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2780c9cf0 con 0x7fa27806aeb0 2024-02-19T15:25:05.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa27f000640 1 -- 172.21.15.47:0/857660810 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2780c5c60 con 0x7fa27806aeb0 2024-02-19T15:25:05.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.073+0000 7fa26fe00640 1 -- 172.21.15.47:0/857660810 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa2680bfcc0 con 0x7fa27806aeb0 2024-02-19T15:25:05.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.077+0000 7fa26fe00640 1 --2- 172.21.15.47:0/857660810 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa258066e40 0x7fa258069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:05.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.077+0000 7fa26fe00640 1 -- 172.21.15.47:0/857660810 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa26811f220 con 0x7fa27806aeb0 2024-02-19T15:25:05.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.077+0000 7fa277400640 1 --2- 172.21.15.47:0/857660810 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa258066e40 0x7fa258069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:05.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.077+0000 7fa277400640 1 --2- 172.21.15.47:0/857660810 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa258066e40 0x7fa258069300 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa27815a850 tx=0x7fa264009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:05.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.081+0000 7fa26fe00640 1 -- 172.21.15.47:0/857660810 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa2680ec490 con 0x7fa27806aeb0 2024-02-19T15:25:05.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.309+0000 7fa27f000640 1 -- 172.21.15.47:0/857660810 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa2780c9ee0 con 0x7fa27806aeb0 2024-02-19T15:25:05.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.309+0000 7fa26fe00640 1 -- 172.21.15.47:0/857660810 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fa2680efb50 con 0x7fa27806aeb0 2024-02-19T15:25:05.312 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:05.312 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:25:05.312 INFO:teuthology.orchestra.run.smithi047.stdout: "auth_dump": [ 2024-02-19T15:25:05.312 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:05.312 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.a", 2024-02-19T15:25:05.312 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNl8BP+HRAAlJ3Jk+1k2XcZAnICzy++Ow==", 2024-02-19T15:25:05.313 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:05.313 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:25:05.313 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:25:05.313 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:25:05.313 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:05.313 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:05.313 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:05.313 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:05.313 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.b", 2024-02-19T15:25:05.313 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlIOh3KhAAlnU2ZFY4iv/XEw0WqxyZpw==", 2024-02-19T15:25:05.313 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:05.313 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:25:05.313 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:25:05.313 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:25:05.313 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:05.313 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:05.314 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:05.314 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:05.314 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.c", 2024-02-19T15:25:05.314 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNl48SHJRAAn4GCT5KnP+2PD8IXbGClCg==", 2024-02-19T15:25:05.314 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:05.314 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:25:05.314 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:25:05.314 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:25:05.314 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:05.314 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:05.314 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:05.314 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:05.314 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.d", 2024-02-19T15:25:05.314 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNljcekMRAA414655SzH43fBA0AtjpxZw==", 2024-02-19T15:25:05.314 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:05.314 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:25:05.315 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:25:05.315 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:25:05.315 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:05.315 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:05.315 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:05.315 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:05.315 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.0", 2024-02-19T15:25:05.315 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDlcdNl/bycFhAAmzYi/OxEPUW3tcRJC0DNEQ==", 2024-02-19T15:25:05.315 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:05.315 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:25:05.315 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:25:05.315 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:05.315 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:05.315 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:05.315 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:05.316 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.1", 2024-02-19T15:25:05.316 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDocdNluquhHBAA88aLHCvVol7JoZB0MtoutA==", 2024-02-19T15:25:05.316 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:05.316 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:25:05.316 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:25:05.316 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:05.316 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:05.316 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:05.316 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:05.316 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.2", 2024-02-19T15:25:05.316 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDrcdNlGAnWIhAAyELvCPKjtfgF0+kX6NQFIA==", 2024-02-19T15:25:05.316 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:05.316 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:25:05.316 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:25:05.316 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:05.316 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:05.317 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:05.317 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:05.317 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.3", 2024-02-19T15:25:05.317 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDucdNlROZ6KBAAxTCDuYsw5SRIZBcclzwxhg==", 2024-02-19T15:25:05.317 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:05.317 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:25:05.317 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:25:05.317 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:05.317 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:05.317 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:05.317 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:05.317 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.4", 2024-02-19T15:25:05.317 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD0cdNluOGILBAA66R9RVXD2vdyIw+oZ+iM1g==", 2024-02-19T15:25:05.317 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:05.317 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:25:05.317 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:25:05.318 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:05.318 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:05.318 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:05.318 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:05.318 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.5", 2024-02-19T15:25:05.318 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD3cdNlHg8dMhAA2LUzfpvvHR4i8e5RBCeGpQ==", 2024-02-19T15:25:05.318 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:05.318 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:25:05.318 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:25:05.318 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:05.318 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:05.318 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:05.318 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:05.318 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.6", 2024-02-19T15:25:05.318 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD6cdNlXD1EORAAhKGIt0wFeCyoE6QkkZ3caA==", 2024-02-19T15:25:05.318 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:05.318 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:25:05.319 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:25:05.319 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:05.319 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:05.319 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:05.319 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:05.319 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.7", 2024-02-19T15:25:05.319 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD+cdNlfW65AhAAVWj3ptC8uQHPh/UXcb/uDg==", 2024-02-19T15:25:05.319 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:05.319 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:25:05.319 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:25:05.319 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:05.319 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:05.319 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:05.319 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:05.319 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.0", 2024-02-19T15:25:05.319 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlb7eRNRAAk6ZLGfjhcdlR9GoXFCd9eA==", 2024-02-19T15:25:05.319 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:05.320 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:25:05.320 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow r", 2024-02-19T15:25:05.320 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-19T15:25:05.320 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:05.320 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:05.320 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:05.320 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.1", 2024-02-19T15:25:05.320 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlKBu+AxAAlLRU70SAs3ZJo5wFwRWa4w==", 2024-02-19T15:25:05.320 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:05.320 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:25:05.320 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:25:05.320 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:25:05.320 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:25:05.320 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:05.320 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:05.320 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:05.321 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.2", 2024-02-19T15:25:05.321 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlyDk9ABAAIAjKSd4qJO5oHhVnDc1Rgw==", 2024-02-19T15:25:05.321 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:05.321 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:25:05.321 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:25:05.321 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:25:05.321 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:25:05.321 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:05.321 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:05.321 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:05.321 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.3", 2024-02-19T15:25:05.321 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlUqK3CRAABmSIzoJvl0ymME6swrvnSQ==", 2024-02-19T15:25:05.321 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:05.321 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:25:05.321 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:25:05.321 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:25:05.322 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:25:05.322 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:05.322 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:05.322 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:05.322 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.admin", 2024-02-19T15:25:05.322 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNls3vPAhAA4x2yQTUCz4Dv9vqEWnSTyQ==", 2024-02-19T15:25:05.322 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:05.322 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:25:05.322 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:25:05.322 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:25:05.322 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:05.322 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:05.322 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:05.322 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:05.322 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mgr.x", 2024-02-19T15:25:05.322 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNliM78GRAABWBg4hMcKjV2pK5WVBjLMQ==", 2024-02-19T15:25:05.322 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:05.323 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:25:05.323 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile mgr", 2024-02-19T15:25:05.323 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:05.323 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:05.323 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:05.323 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:05.323 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mgr.y", 2024-02-19T15:25:05.323 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlwxs9FhAAWqP7YoKio/bbdx7SGiCJ9Q==", 2024-02-19T15:25:05.323 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:05.323 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:25:05.323 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile mgr", 2024-02-19T15:25:05.323 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:05.323 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:05.323 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:05.324 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-02-19T15:25:05.324 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:25:05.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.309+0000 7fa27f000640 1 -- 172.21.15.47:0/857660810 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa258066e40 msgr2=0x7fa258069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:05.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.309+0000 7fa27f000640 1 --2- 172.21.15.47:0/857660810 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa258066e40 0x7fa258069300 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa27815a850 tx=0x7fa264009190 comp rx=0 tx=0).stop 2024-02-19T15:25:05.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.309+0000 7fa27f000640 1 -- 172.21.15.47:0/857660810 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa27806aeb0 msgr2=0x7fa2781474c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:05.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.309+0000 7fa27f000640 1 --2- 172.21.15.47:0/857660810 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa27806aeb0 0x7fa2781474c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fa2680ad040 tx=0x7fa26801acb0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.309+0000 7fa27f000640 1 -- 172.21.15.47:0/857660810 shutdown_connections 2024-02-19T15:25:05.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.309+0000 7fa27f000640 1 --2- 172.21.15.47:0/857660810 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa258066e40 0x7fa258069300 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.309+0000 7fa27f000640 1 --2- 172.21.15.47:0/857660810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2780cb520 0x7fa278147d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.309+0000 7fa27f000640 1 --2- 172.21.15.47:0/857660810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2780719e0 0x7fa2780c92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.309+0000 7fa27f000640 1 --2- 172.21.15.47:0/857660810 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa27806aeb0 0x7fa2781474c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.309+0000 7fa27f000640 1 -- 172.21.15.47:0/857660810 >> 172.21.15.47:0/857660810 conn(0x7fa2780d0ad0 msgr2=0x7fa278071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:05.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.309+0000 7fa27f000640 1 -- 172.21.15.47:0/857660810 shutdown_connections 2024-02-19T15:25:05.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.309+0000 7fa27f000640 1 -- 172.21.15.47:0/857660810 wait complete. 2024-02-19T15:25:05.334 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:25:05.473 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:25:05.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.469+0000 7fb555800640 1 Processor -- start 2024-02-19T15:25:05.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.469+0000 7fb555800640 1 -- start start 2024-02-19T15:25:05.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.473+0000 7fb555800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb5500d19e0 0x7fb5500d1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:05.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.473+0000 7fb555800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5500cec90 0x7fb5500cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:05.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.473+0000 7fb555800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5500d2300 con 0x7fb5500cf0f0 2024-02-19T15:25:05.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.473+0000 7fb555800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5500ce280 con 0x7fb5500d19e0 2024-02-19T15:25:05.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.473+0000 7fb555800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55015cd40 con 0x7fb5500cec90 2024-02-19T15:25:05.477 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:25:05.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.473+0000 7fb554e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb5500cf0f0 0x7fb5500cca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38248/0 (socket says 172.21.15.47:38248) 2024-02-19T15:25:05.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.473+0000 7fb554e00640 1 -- 172.21.15.47:0/960473804 learned_addr learned my addr 172.21.15.47:0/960473804 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:05.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.473+0000 7fb54fe00640 1 --2- 172.21.15.47:0/960473804 >> v2:172.21.15.116:3300/0 conn(0x7fb5500d19e0 0x7fb5500d1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:05.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.473+0000 7fb547400640 1 --2- 172.21.15.47:0/960473804 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5500cec90 0x7fb5500cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:05.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.473+0000 7fb54fe00640 1 -- 172.21.15.47:0/960473804 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5500cec90 msgr2=0x7fb5500cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:05.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.473+0000 7fb54fe00640 1 --2- 172.21.15.47:0/960473804 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5500cec90 0x7fb5500cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.473+0000 7fb54f400640 1 -- 172.21.15.47:0/960473804 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1458845210 0 0) 0x7fb5500d2300 con 0x7fb5500cf0f0 2024-02-19T15:25:05.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.473+0000 7fb54fe00640 1 -- 172.21.15.47:0/960473804 >> v1:172.21.15.47:6789/0 conn(0x7fb5500cf0f0 legacy=0x7fb5500cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:05.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.473+0000 7fb54fe00640 1 -- 172.21.15.47:0/960473804 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55015d020 con 0x7fb5500d19e0 2024-02-19T15:25:05.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.477+0000 7fb54fe00640 1 --2- 172.21.15.47:0/960473804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5500d19e0 0x7fb5500d1dc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb5400097c0 tx=0x7fb54001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2142dbf6d414dc6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:05.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.477+0000 7fb54f400640 1 -- 172.21.15.47:0/960473804 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5400ad640 con 0x7fb5500d19e0 2024-02-19T15:25:05.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.477+0000 7fb54f400640 1 -- 172.21.15.47:0/960473804 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5400ad7a0 con 0x7fb5500d19e0 2024-02-19T15:25:05.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.477+0000 7fb54f400640 1 -- 172.21.15.47:0/960473804 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5400adb30 con 0x7fb5500d19e0 2024-02-19T15:25:05.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.477+0000 7fb555800640 1 -- 172.21.15.47:0/960473804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5500d19e0 msgr2=0x7fb5500d1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:05.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.477+0000 7fb555800640 1 --2- 172.21.15.47:0/960473804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5500d19e0 0x7fb5500d1dc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb5400097c0 tx=0x7fb54001c4a0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.477+0000 7fb555800640 1 -- 172.21.15.47:0/960473804 shutdown_connections 2024-02-19T15:25:05.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.477+0000 7fb555800640 1 --2- 172.21.15.47:0/960473804 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5500cec90 0x7fb5500cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.477+0000 7fb555800640 1 --2- 172.21.15.47:0/960473804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5500d19e0 0x7fb5500d1dc0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.477+0000 7fb555800640 1 -- 172.21.15.47:0/960473804 >> 172.21.15.47:0/960473804 conn(0x7fb55009a9d0 msgr2=0x7fb55009ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:05.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.477+0000 7fb555800640 1 -- 172.21.15.47:0/960473804 shutdown_connections 2024-02-19T15:25:05.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.477+0000 7fb555800640 1 -- 172.21.15.47:0/960473804 wait complete. 2024-02-19T15:25:05.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.477+0000 7fb555800640 1 Processor -- start 2024-02-19T15:25:05.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.477+0000 7fb555800640 1 -- start start 2024-02-19T15:25:05.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.477+0000 7fb555800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5500cec90 0x7fb550159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:05.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.477+0000 7fb555800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5500d19e0 0x7fb5500b6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:05.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.477+0000 7fb555800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5500b4670 0x7fb550152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:05.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb555800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55006ccc0 con 0x7fb5500cec90 2024-02-19T15:25:05.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb555800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55006c0f0 con 0x7fb5500d19e0 2024-02-19T15:25:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb555800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55006c390 con 0x7fb5500b4670 2024-02-19T15:25:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb54fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5500cec90 0x7fb550159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb54fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5500cec90 0x7fb550159af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39740/0 (socket says 172.21.15.47:39740) 2024-02-19T15:25:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb54fe00640 1 -- 172.21.15.47:0/453075839 learned_addr learned my addr 172.21.15.47:0/453075839 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb547400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5500d19e0 0x7fb5500b6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb554e00640 1 --2- 172.21.15.47:0/453075839 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5500b4670 0x7fb550152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb54fe00640 1 -- 172.21.15.47:0/453075839 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5500b4670 msgr2=0x7fb550152810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb54fe00640 1 --2- 172.21.15.47:0/453075839 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5500b4670 0x7fb550152810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb54fe00640 1 -- 172.21.15.47:0/453075839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5500d19e0 msgr2=0x7fb5500b6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb54fe00640 1 --2- 172.21.15.47:0/453075839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5500d19e0 0x7fb5500b6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb54fe00640 1 -- 172.21.15.47:0/453075839 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb540009420 con 0x7fb5500cec90 2024-02-19T15:25:05.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb54fe00640 1 --2- 172.21.15.47:0/453075839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5500cec90 0x7fb550159af0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fb54001a0d0 tx=0x7fb540019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:05.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb54cc00640 1 -- 172.21.15.47:0/453075839 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5400b6610 con 0x7fb5500cec90 2024-02-19T15:25:05.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb54cc00640 1 -- 172.21.15.47:0/453075839 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5400a6040 con 0x7fb5500cec90 2024-02-19T15:25:05.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb54cc00640 1 -- 172.21.15.47:0/453075839 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5400b6990 con 0x7fb5500cec90 2024-02-19T15:25:05.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb555800640 1 -- 172.21.15.47:0/453075839 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb55006c670 con 0x7fb5500cec90 2024-02-19T15:25:05.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb555800640 1 -- 172.21.15.47:0/453075839 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55015a030 con 0x7fb5500cec90 2024-02-19T15:25:05.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb555800640 1 -- 172.21.15.47:0/453075839 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb550075360 con 0x7fb5500cec90 2024-02-19T15:25:05.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.481+0000 7fb54cc00640 1 -- 172.21.15.47:0/453075839 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb5400b4040 con 0x7fb5500cec90 2024-02-19T15:25:05.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.485+0000 7fb54cc00640 1 --2- 172.21.15.47:0/453075839 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb52c066e70 0x7fb52c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:05.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.485+0000 7fb54cc00640 1 -- 172.21.15.47:0/453075839 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb54011e330 con 0x7fb5500cec90 2024-02-19T15:25:05.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.485+0000 7fb547400640 1 --2- 172.21.15.47:0/453075839 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb52c066e70 0x7fb52c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:05.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.485+0000 7fb547400640 1 --2- 172.21.15.47:0/453075839 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb52c066e70 0x7fb52c069330 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb5500cc880 tx=0x7fb538009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:05.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.489+0000 7fb54cc00640 1 -- 172.21.15.47:0/453075839 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb5400eb630 con 0x7fb5500cec90 2024-02-19T15:25:05.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.653+0000 7fb555800640 1 -- 172.21.15.47:0/453075839 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5500d0890 con 0x7fb5500cec90 2024-02-19T15:25:05.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.653+0000 7fb54cc00640 1 -- 172.21.15.47:0/453075839 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+12903 (secure 0 0 0) 0x7fb5400eecf0 con 0x7fb5500cec90 2024-02-19T15:25:05.656 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:05.656 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":65,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:25:04.890509+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:25:05.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.653+0000 7fb555800640 1 -- 172.21.15.47:0/453075839 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb52c066e70 msgr2=0x7fb52c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:05.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.653+0000 7fb555800640 1 --2- 172.21.15.47:0/453075839 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb52c066e70 0x7fb52c069330 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb5500cc880 tx=0x7fb538009190 comp rx=0 tx=0).stop 2024-02-19T15:25:05.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.653+0000 7fb555800640 1 -- 172.21.15.47:0/453075839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5500cec90 msgr2=0x7fb550159af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:05.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.653+0000 7fb555800640 1 --2- 172.21.15.47:0/453075839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5500cec90 0x7fb550159af0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fb54001a0d0 tx=0x7fb540019240 comp rx=0 tx=0).stop 2024-02-19T15:25:05.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.653+0000 7fb555800640 1 -- 172.21.15.47:0/453075839 shutdown_connections 2024-02-19T15:25:05.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.653+0000 7fb555800640 1 --2- 172.21.15.47:0/453075839 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb52c066e70 0x7fb52c069330 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.653+0000 7fb555800640 1 --2- 172.21.15.47:0/453075839 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5500b4670 0x7fb550152810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.653+0000 7fb555800640 1 --2- 172.21.15.47:0/453075839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5500d19e0 0x7fb5500b6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.653+0000 7fb555800640 1 --2- 172.21.15.47:0/453075839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5500cec90 0x7fb550159af0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.653+0000 7fb555800640 1 -- 172.21.15.47:0/453075839 >> 172.21.15.47:0/453075839 conn(0x7fb55009a9d0 msgr2=0x7fb55009adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:05.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.653+0000 7fb555800640 1 -- 172.21.15.47:0/453075839 shutdown_connections 2024-02-19T15:25:05.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.653+0000 7fb555800640 1 -- 172.21.15.47:0/453075839 wait complete. 2024-02-19T15:25:05.677 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:25:05.677 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:25:05.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.817+0000 7fab8ae00640 1 Processor -- start 2024-02-19T15:25:05.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.817+0000 7fab8ae00640 1 -- start start 2024-02-19T15:25:05.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.817+0000 7fab8ae00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab840cb490 0x7fab84070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:05.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.817+0000 7fab8ae00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fab84072850 0x7fab8406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:05.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.817+0000 7fab8ae00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8415fc90 con 0x7fab8415bd90 2024-02-19T15:25:05.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.817+0000 7fab8ae00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab8415ff70 con 0x7fab84072850 2024-02-19T15:25:05.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.817+0000 7fab8ae00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab84160250 con 0x7fab840cb490 2024-02-19T15:25:05.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.817+0000 7fab88e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fab84072850 0x7fab8406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:05.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.817+0000 7fab88e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fab84072850 0x7fab8406cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39756/0 (socket says 172.21.15.47:39756) 2024-02-19T15:25:05.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.817+0000 7fab88e00640 1 -- 172.21.15.47:0/4205722810 learned_addr learned my addr 172.21.15.47:0/4205722810 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:05.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.821+0000 7fab88e00640 1 -- 172.21.15.47:0/4205722810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab840cb490 msgr2=0x7fab84070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:05.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.821+0000 7fab7b400640 1 --2- 172.21.15.47:0/4205722810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab840cb490 0x7fab84070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:05.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.821+0000 7fab88e00640 1 --2- 172.21.15.47:0/4205722810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab840cb490 0x7fab84070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.821+0000 7fab88e00640 1 -- 172.21.15.47:0/4205722810 >> v1:172.21.15.116:6789/0 conn(0x7fab8415bd90 legacy=0x7fab84072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:05.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.821+0000 7fab88e00640 1 -- 172.21.15.47:0/4205722810 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab84160530 con 0x7fab84072850 2024-02-19T15:25:05.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.821+0000 7fab7b400640 1 --2- 172.21.15.47:0/4205722810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab840cb490 0x7fab84070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:25:05.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.821+0000 7fab88e00640 1 --2- 172.21.15.47:0/4205722810 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab84072850 0x7fab8406cc60 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fab7c009e50 tx=0x7fab7c01c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=babe27654926761 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:05.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.821+0000 7fab83400640 1 -- 172.21.15.47:0/4205722810 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c0a5660 con 0x7fab84072850 2024-02-19T15:25:05.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.821+0000 7fab83400640 1 -- 172.21.15.47:0/4205722810 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7c0a57c0 con 0x7fab84072850 2024-02-19T15:25:05.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.821+0000 7fab83400640 1 -- 172.21.15.47:0/4205722810 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c0a5b50 con 0x7fab84072850 2024-02-19T15:25:05.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.821+0000 7fab8ae00640 1 -- 172.21.15.47:0/4205722810 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab84072850 msgr2=0x7fab8406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:05.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.821+0000 7fab8ae00640 1 --2- 172.21.15.47:0/4205722810 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab84072850 0x7fab8406cc60 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fab7c009e50 tx=0x7fab7c01c890 comp rx=0 tx=0).stop 2024-02-19T15:25:05.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.821+0000 7fab8ae00640 1 -- 172.21.15.47:0/4205722810 shutdown_connections 2024-02-19T15:25:05.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.821+0000 7fab8ae00640 1 --2- 172.21.15.47:0/4205722810 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab84072850 0x7fab8406cc60 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.821+0000 7fab8ae00640 1 --2- 172.21.15.47:0/4205722810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab840cb490 0x7fab84070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.821+0000 7fab8ae00640 1 -- 172.21.15.47:0/4205722810 >> 172.21.15.47:0/4205722810 conn(0x7fab840cfc20 msgr2=0x7fab840d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:05.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab8ae00640 1 -- 172.21.15.47:0/4205722810 shutdown_connections 2024-02-19T15:25:05.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab8ae00640 1 -- 172.21.15.47:0/4205722810 wait complete. 2024-02-19T15:25:05.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab8ae00640 1 Processor -- start 2024-02-19T15:25:05.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab8ae00640 1 -- start start 2024-02-19T15:25:05.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab8ae00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab84072850 0x7fab840c5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:05.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab8ae00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab840cb490 0x7fab841f9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab8ae00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab8415bd90 0x7fab841fa010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab8ae00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab840c7de0 con 0x7fab8415bd90 2024-02-19T15:25:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab8ae00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab840c5c60 con 0x7fab840cb490 2024-02-19T15:25:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab8ae00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8415aca0 con 0x7fab84072850 2024-02-19T15:25:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab88e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab8415bd90 0x7fab841fa010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab88e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab8415bd90 0x7fab841fa010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39760/0 (socket says 172.21.15.47:39760) 2024-02-19T15:25:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab88e00640 1 -- 172.21.15.47:0/1816312993 learned_addr learned my addr 172.21.15.47:0/1816312993 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab7b400640 1 --2- 172.21.15.47:0/1816312993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab840cb490 0x7fab841f9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab83e00640 1 --2- 172.21.15.47:0/1816312993 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab84072850 0x7fab840c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab88e00640 1 -- 172.21.15.47:0/1816312993 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab84072850 msgr2=0x7fab840c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab88e00640 1 --2- 172.21.15.47:0/1816312993 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab84072850 0x7fab840c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab88e00640 1 -- 172.21.15.47:0/1816312993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab840cb490 msgr2=0x7fab841f9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:05.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab88e00640 1 --2- 172.21.15.47:0/1816312993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab840cb490 0x7fab841f9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:05.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab88e00640 1 -- 172.21.15.47:0/1816312993 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7c009b30 con 0x7fab8415bd90 2024-02-19T15:25:05.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab83e00640 1 --2- 172.21.15.47:0/1816312993 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab84072850 0x7fab840c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:05.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab88e00640 1 --2- 172.21.15.47:0/1816312993 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab8415bd90 0x7fab841fa010 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fab7c0af040 tx=0x7fab7c019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:05.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab80c00640 1 -- 172.21.15.47:0/1816312993 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c0b8670 con 0x7fab8415bd90 2024-02-19T15:25:05.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab80c00640 1 -- 172.21.15.47:0/1816312993 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7c0b87d0 con 0x7fab8415bd90 2024-02-19T15:25:05.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab80c00640 1 -- 172.21.15.47:0/1816312993 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c0b8b80 con 0x7fab8415bd90 2024-02-19T15:25:05.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.825+0000 7fab8ae00640 1 -- 172.21.15.47:0/1816312993 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab840c8080 con 0x7fab8415bd90 2024-02-19T15:25:05.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.829+0000 7fab8ae00640 1 -- 172.21.15.47:0/1816312993 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8415af80 con 0x7fab8415bd90 2024-02-19T15:25:05.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.829+0000 7fab8ae00640 1 -- 172.21.15.47:0/1816312993 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab84075360 con 0x7fab8415bd90 2024-02-19T15:25:05.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.829+0000 7fab80c00640 1 -- 172.21.15.47:0/1816312993 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fab7c0b8ce0 con 0x7fab8415bd90 2024-02-19T15:25:05.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.829+0000 7fab80c00640 1 --2- 172.21.15.47:0/1816312993 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fab60066e70 0x7fab60069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:05.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.829+0000 7fab80c00640 1 -- 172.21.15.47:0/1816312993 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fab7c11fcc0 con 0x7fab8415bd90 2024-02-19T15:25:05.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.829+0000 7fab83e00640 1 --2- 172.21.15.47:0/1816312993 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fab60066e70 0x7fab60069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:05.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.829+0000 7fab83e00640 1 --2- 172.21.15.47:0/1816312993 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fab60066e70 0x7fab60069330 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fab84143910 tx=0x7fab74008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:05.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.833+0000 7fab80c00640 1 -- 172.21.15.47:0/1816312993 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fab7c126080 con 0x7fab8415bd90 2024-02-19T15:25:06.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.997+0000 7fab8ae00640 1 -- 172.21.15.47:0/1816312993 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fab840c8530 con 0x7fab8415bd90 2024-02-19T15:25:06.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.997+0000 7fab80c00640 1 -- 172.21.15.47:0/1816312993 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v65) v1 ==== 93+0+10 (secure 0 0 0) 0x7fab7c0ecf10 con 0x7fab8415bd90 2024-02-19T15:25:06.045 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:25:06.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:05.997+0000 7fab8ae00640 1 -- 172.21.15.47:0/1816312993 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fab60066e70 msgr2=0x7fab60069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:06.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.001+0000 7fab8ae00640 1 --2- 172.21.15.47:0/1816312993 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fab60066e70 0x7fab60069330 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fab84143910 tx=0x7fab74008040 comp rx=0 tx=0).stop 2024-02-19T15:25:06.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.001+0000 7fab8ae00640 1 -- 172.21.15.47:0/1816312993 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab8415bd90 msgr2=0x7fab841fa010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:06.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.001+0000 7fab8ae00640 1 --2- 172.21.15.47:0/1816312993 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab8415bd90 0x7fab841fa010 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fab7c0af040 tx=0x7fab7c019630 comp rx=0 tx=0).stop 2024-02-19T15:25:06.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.001+0000 7fab8ae00640 1 -- 172.21.15.47:0/1816312993 shutdown_connections 2024-02-19T15:25:06.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.001+0000 7fab8ae00640 1 --2- 172.21.15.47:0/1816312993 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fab60066e70 0x7fab60069330 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:06.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.001+0000 7fab8ae00640 1 --2- 172.21.15.47:0/1816312993 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab8415bd90 0x7fab841fa010 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:06.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.001+0000 7fab8ae00640 1 --2- 172.21.15.47:0/1816312993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab840cb490 0x7fab841f9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:06.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.001+0000 7fab8ae00640 1 --2- 172.21.15.47:0/1816312993 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab84072850 0x7fab840c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:06.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.001+0000 7fab8ae00640 1 -- 172.21.15.47:0/1816312993 >> 172.21.15.47:0/1816312993 conn(0x7fab840cfc20 msgr2=0x7fab84072100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:06.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.001+0000 7fab8ae00640 1 -- 172.21.15.47:0/1816312993 shutdown_connections 2024-02-19T15:25:06.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.001+0000 7fab8ae00640 1 -- 172.21.15.47:0/1816312993 wait complete. 2024-02-19T15:25:06.048 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T15:25:06.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.185+0000 7f7a52a00640 1 Processor -- start 2024-02-19T15:25:06.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.185+0000 7f7a52a00640 1 -- start start 2024-02-19T15:25:06.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.185+0000 7f7a52a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a4c0b4240 0x7f7a4c0d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:06.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.185+0000 7f7a52a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7a4c0ce400 0x7f7a4c0ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:06.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.185+0000 7f7a52a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4c0d1070 con 0x7f7a4c15dfa0 2024-02-19T15:25:06.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.185+0000 7f7a52a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a4c0cedd0 con 0x7f7a4c0ce400 2024-02-19T15:25:06.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.185+0000 7f7a52a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c06ec60 con 0x7f7a4c0b4240 2024-02-19T15:25:06.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.185+0000 7f7a50a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7a4c0ce400 0x7f7a4c0ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:06.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.185+0000 7f7a4be00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f7a4c15dfa0 0x7f7a4c06fa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38264/0 (socket says 172.21.15.47:38264) 2024-02-19T15:25:06.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.185+0000 7f7a4be00640 1 -- 172.21.15.47:0/3835507711 learned_addr learned my addr 172.21.15.47:0/3835507711 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:06.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.185+0000 7f7a43e00640 1 --2- 172.21.15.47:0/3835507711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a4c0b4240 0x7f7a4c0d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:06.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.185+0000 7f7a50a00640 1 -- 172.21.15.47:0/3835507711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a4c0b4240 msgr2=0x7f7a4c0d0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:06.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.185+0000 7f7a50a00640 1 --2- 172.21.15.47:0/3835507711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a4c0b4240 0x7f7a4c0d0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:06.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.185+0000 7f7a50a00640 1 -- 172.21.15.47:0/3835507711 >> v1:172.21.15.47:6789/0 conn(0x7f7a4c15dfa0 legacy=0x7f7a4c06fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:06.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.185+0000 7f7a50a00640 1 -- 172.21.15.47:0/3835507711 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c06ef40 con 0x7f7a4c0ce400 2024-02-19T15:25:06.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.185+0000 7f7a4b400640 1 -- 172.21.15.47:0/3835507711 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640518530 0 0) 0x7f7a4c0d1070 con 0x7f7a4c15dfa0 2024-02-19T15:25:06.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.189+0000 7f7a50a00640 1 --2- 172.21.15.47:0/3835507711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a4c0ce400 0x7f7a4c0ce890 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f7a44009d90 tx=0x7f7a4401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f28e05af49fa3e2b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:06.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.189+0000 7f7a4b400640 1 -- 172.21.15.47:0/3835507711 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a440a5910 con 0x7f7a4c0ce400 2024-02-19T15:25:06.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.189+0000 7f7a4b400640 1 -- 172.21.15.47:0/3835507711 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a440a5a70 con 0x7f7a4c0ce400 2024-02-19T15:25:06.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.189+0000 7f7a4b400640 1 -- 172.21.15.47:0/3835507711 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a440a5e00 con 0x7f7a4c0ce400 2024-02-19T15:25:06.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.189+0000 7f7a52a00640 1 -- 172.21.15.47:0/3835507711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a4c0ce400 msgr2=0x7f7a4c0ce890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:06.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.189+0000 7f7a52a00640 1 --2- 172.21.15.47:0/3835507711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a4c0ce400 0x7f7a4c0ce890 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f7a44009d90 tx=0x7f7a4401c940 comp rx=0 tx=0).stop 2024-02-19T15:25:06.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.189+0000 7f7a52a00640 1 -- 172.21.15.47:0/3835507711 shutdown_connections 2024-02-19T15:25:06.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.189+0000 7f7a52a00640 1 --2- 172.21.15.47:0/3835507711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a4c0ce400 0x7f7a4c0ce890 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:06.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.189+0000 7f7a52a00640 1 --2- 172.21.15.47:0/3835507711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a4c0b4240 0x7f7a4c0d0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:06.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.189+0000 7f7a52a00640 1 -- 172.21.15.47:0/3835507711 >> 172.21.15.47:0/3835507711 conn(0x7f7a4c0b3690 msgr2=0x7f7a4c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:06.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.189+0000 7f7a52a00640 1 -- 172.21.15.47:0/3835507711 shutdown_connections 2024-02-19T15:25:06.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.189+0000 7f7a52a00640 1 -- 172.21.15.47:0/3835507711 wait complete. 2024-02-19T15:25:06.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a52a00640 1 Processor -- start 2024-02-19T15:25:06.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a52a00640 1 -- start start 2024-02-19T15:25:06.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a52a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a4c0b4240 0x7f7a4c15c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:06.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a52a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a4c0ce400 0x7f7a4c1540a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:06.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a52a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a4c15dfa0 0x7f7a4c0af1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:06.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a52a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c15ca30 con 0x7f7a4c0ce400 2024-02-19T15:25:06.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a52a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c0aea10 con 0x7f7a4c15dfa0 2024-02-19T15:25:06.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a52a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c15ccb0 con 0x7f7a4c0b4240 2024-02-19T15:25:06.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a43e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a4c0ce400 0x7f7a4c1540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:06.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a43e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a4c0ce400 0x7f7a4c1540a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39786/0 (socket says 172.21.15.47:39786) 2024-02-19T15:25:06.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a43e00640 1 -- 172.21.15.47:0/1041257747 learned_addr learned my addr 172.21.15.47:0/1041257747 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:06.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a4be00640 1 --2- 172.21.15.47:0/1041257747 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a4c0b4240 0x7f7a4c15c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:06.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a50a00640 1 --2- 172.21.15.47:0/1041257747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a4c15dfa0 0x7f7a4c0af1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:06.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a43e00640 1 -- 172.21.15.47:0/1041257747 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a4c0b4240 msgr2=0x7f7a4c15c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:06.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a43e00640 1 --2- 172.21.15.47:0/1041257747 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a4c0b4240 0x7f7a4c15c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:06.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a43e00640 1 -- 172.21.15.47:0/1041257747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a4c15dfa0 msgr2=0x7f7a4c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:06.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a43e00640 1 --2- 172.21.15.47:0/1041257747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a4c15dfa0 0x7f7a4c0af1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:06.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a43e00640 1 -- 172.21.15.47:0/1041257747 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a44009a70 con 0x7f7a4c0ce400 2024-02-19T15:25:06.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a43e00640 1 --2- 172.21.15.47:0/1041257747 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a4c0ce400 0x7f7a4c1540a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f7a3400d800 tx=0x7f7a3400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:06.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a48c00640 1 -- 172.21.15.47:0/1041257747 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3400b6c0 con 0x7f7a4c0ce400 2024-02-19T15:25:06.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a48c00640 1 -- 172.21.15.47:0/1041257747 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a3400b820 con 0x7f7a4c0ce400 2024-02-19T15:25:06.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a52a00640 1 -- 172.21.15.47:0/1041257747 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a4c15cf90 con 0x7f7a4c0ce400 2024-02-19T15:25:06.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a48c00640 1 -- 172.21.15.47:0/1041257747 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3400bb90 con 0x7f7a4c0ce400 2024-02-19T15:25:06.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.193+0000 7f7a52a00640 1 -- 172.21.15.47:0/1041257747 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a4c1545e0 con 0x7f7a4c0ce400 2024-02-19T15:25:06.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.197+0000 7f7a52a00640 1 -- 172.21.15.47:0/1041257747 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a4c075360 con 0x7f7a4c0ce400 2024-02-19T15:25:06.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.197+0000 7f7a48c00640 1 -- 172.21.15.47:0/1041257747 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7a3400bcf0 con 0x7f7a4c0ce400 2024-02-19T15:25:06.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.197+0000 7f7a48c00640 1 --2- 172.21.15.47:0/1041257747 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a28066e70 0x7f7a28069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:06.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.197+0000 7f7a48c00640 1 -- 172.21.15.47:0/1041257747 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7a34084c40 con 0x7f7a4c0ce400 2024-02-19T15:25:06.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.197+0000 7f7a4be00640 1 --2- 172.21.15.47:0/1041257747 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a28066e70 0x7f7a28069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:06.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.197+0000 7f7a4be00640 1 --2- 172.21.15.47:0/1041257747 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a28066e70 0x7f7a28069330 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c000f70 tx=0x7f7a3c007df0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:06.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.201+0000 7f7a48c00640 1 -- 172.21.15.47:0/1041257747 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7a34051d90 con 0x7f7a4c0ce400 2024-02-19T15:25:06.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.365+0000 7f7a52a00640 1 -- 172.21.15.47:0/1041257747 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7a4c0d1610 con 0x7f7a4c0ce400 2024-02-19T15:25:06.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.365+0000 7f7a48c00640 1 -- 172.21.15.47:0/1041257747 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v65) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7a34055450 con 0x7f7a4c0ce400 2024-02-19T15:25:06.370 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:06.370 INFO:teuthology.orchestra.run.smithi047.stdout:[ 2024-02-19T15:25:06.370 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:06.370 INFO:teuthology.orchestra.run.smithi047.stdout: "poolnum": 1, 2024-02-19T15:25:06.370 INFO:teuthology.orchestra.run.smithi047.stdout: "poolname": ".mgr" 2024-02-19T15:25:06.370 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:06.370 INFO:teuthology.orchestra.run.smithi047.stdout:] 2024-02-19T15:25:06.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.369+0000 7f7a52a00640 1 -- 172.21.15.47:0/1041257747 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a28066e70 msgr2=0x7f7a28069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:06.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.369+0000 7f7a52a00640 1 --2- 172.21.15.47:0/1041257747 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a28066e70 0x7f7a28069330 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c000f70 tx=0x7f7a3c007df0 comp rx=0 tx=0).stop 2024-02-19T15:25:06.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.369+0000 7f7a52a00640 1 -- 172.21.15.47:0/1041257747 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a4c0ce400 msgr2=0x7f7a4c1540a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:06.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.369+0000 7f7a52a00640 1 --2- 172.21.15.47:0/1041257747 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a4c0ce400 0x7f7a4c1540a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f7a3400d800 tx=0x7f7a3400dcc0 comp rx=0 tx=0).stop 2024-02-19T15:25:06.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.369+0000 7f7a52a00640 1 -- 172.21.15.47:0/1041257747 shutdown_connections 2024-02-19T15:25:06.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.369+0000 7f7a52a00640 1 --2- 172.21.15.47:0/1041257747 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a28066e70 0x7f7a28069330 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:06.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.369+0000 7f7a52a00640 1 --2- 172.21.15.47:0/1041257747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a4c15dfa0 0x7f7a4c0af1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:06.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.369+0000 7f7a52a00640 1 --2- 172.21.15.47:0/1041257747 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a4c0ce400 0x7f7a4c1540a0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:06.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.369+0000 7f7a52a00640 1 --2- 172.21.15.47:0/1041257747 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a4c0b4240 0x7f7a4c15c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:06.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.369+0000 7f7a52a00640 1 -- 172.21.15.47:0/1041257747 >> 172.21.15.47:0/1041257747 conn(0x7f7a4c0b3690 msgr2=0x7f7a4c0b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:06.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.369+0000 7f7a52a00640 1 -- 172.21.15.47:0/1041257747 shutdown_connections 2024-02-19T15:25:06.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.369+0000 7f7a52a00640 1 -- 172.21.15.47:0/1041257747 wait complete. 2024-02-19T15:25:06.391 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T15:25:06.391 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T15:25:06.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.533+0000 7f664e400640 1 Processor -- start 2024-02-19T15:25:06.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.533+0000 7f664e400640 1 -- start start 2024-02-19T15:25:06.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.533+0000 7f664e400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f66480c3e50 0x7f66480ce680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:06.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.533+0000 7f664e400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6648165e50 con 0x7f66480caee0 2024-02-19T15:25:06.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.533+0000 7f664e400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6648162780 con 0x7f66480c3920 2024-02-19T15:25:06.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.533+0000 7f664e400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66480cf750 con 0x7f66480c3e50 2024-02-19T15:25:06.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.533+0000 7f664d000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f66480caee0 0x7f66480cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38274/0 (socket says 172.21.15.47:38274) 2024-02-19T15:25:06.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.533+0000 7f664d000640 1 -- 172.21.15.47:0/3516870327 learned_addr learned my addr 172.21.15.47:0/3516870327 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:06.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.533+0000 7f664da00640 1 --2- 172.21.15.47:0/3516870327 >> v2:172.21.15.47:3300/0 conn(0x7f66480c3e50 0x7f66480ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:06.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.533+0000 7f664da00640 1 -- 172.21.15.47:0/3516870327 >> v1:172.21.15.116:6789/0 conn(0x7f66480c3920 legacy=0x7f6648162b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:06.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.533+0000 7f664da00640 1 -- 172.21.15.47:0/3516870327 >> v1:172.21.15.47:6789/0 conn(0x7f66480caee0 legacy=0x7f66480cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:06.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.533+0000 7f664da00640 1 -- 172.21.15.47:0/3516870327 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66481660a0 con 0x7f66480c3e50 2024-02-19T15:25:06.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.537+0000 7f664da00640 1 --2- 172.21.15.47:0/3516870327 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66480c3e50 0x7f66480ce680 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f6640009d90 tx=0x7f664001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=373146772b4e40f5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:06.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.537+0000 7f6647400640 1 -- 172.21.15.47:0/3516870327 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66400a5a30 con 0x7f66480c3e50 2024-02-19T15:25:06.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.537+0000 7f6647400640 1 -- 172.21.15.47:0/3516870327 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66400a5b90 con 0x7f66480c3e50 2024-02-19T15:25:06.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.537+0000 7f6647400640 1 -- 172.21.15.47:0/3516870327 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66400ad440 con 0x7f66480c3e50 2024-02-19T15:25:06.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.537+0000 7f664e400640 1 -- 172.21.15.47:0/3516870327 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66480c3e50 msgr2=0x7f66480ce680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:06.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.537+0000 7f664e400640 1 --2- 172.21.15.47:0/3516870327 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66480c3e50 0x7f66480ce680 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f6640009d90 tx=0x7f664001c970 comp rx=0 tx=0).stop 2024-02-19T15:25:06.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.537+0000 7f664e400640 1 -- 172.21.15.47:0/3516870327 shutdown_connections 2024-02-19T15:25:06.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.537+0000 7f664e400640 1 --2- 172.21.15.47:0/3516870327 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66480c3e50 0x7f66480ce680 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:06.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.537+0000 7f664e400640 1 -- 172.21.15.47:0/3516870327 >> 172.21.15.47:0/3516870327 conn(0x7f66480d0890 msgr2=0x7f66480d0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:06.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.537+0000 7f664e400640 1 -- 172.21.15.47:0/3516870327 shutdown_connections 2024-02-19T15:25:06.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.537+0000 7f664e400640 1 -- 172.21.15.47:0/3516870327 wait complete. 2024-02-19T15:25:06.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664e400640 1 Processor -- start 2024-02-19T15:25:06.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664e400640 1 -- start start 2024-02-19T15:25:06.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664e400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66480c3920 0x7f66481fb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:06.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664e400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66480c3e50 0x7f664815a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:06.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664e400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66480caee0 0x7f66480c9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:06.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664e400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66480c4420 con 0x7f66480caee0 2024-02-19T15:25:06.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664e400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66480c46c0 con 0x7f66480c3e50 2024-02-19T15:25:06.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664e400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66480c49a0 con 0x7f66480c3920 2024-02-19T15:25:06.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664da00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66480caee0 0x7f66480c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:06.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664da00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66480caee0 0x7f66480c9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39796/0 (socket says 172.21.15.47:39796) 2024-02-19T15:25:06.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664da00640 1 -- 172.21.15.47:0/2472193276 learned_addr learned my addr 172.21.15.47:0/2472193276 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:06.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f6647e00640 1 --2- 172.21.15.47:0/2472193276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66480c3e50 0x7f664815a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:06.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664d000640 1 --2- 172.21.15.47:0/2472193276 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66480c3920 0x7f66481fb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:06.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664da00640 1 -- 172.21.15.47:0/2472193276 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66480c3920 msgr2=0x7f66481fb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:06.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664da00640 1 --2- 172.21.15.47:0/2472193276 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66480c3920 0x7f66481fb610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:06.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664da00640 1 -- 172.21.15.47:0/2472193276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66480c3e50 msgr2=0x7f664815a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:06.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664da00640 1 --2- 172.21.15.47:0/2472193276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66480c3e50 0x7f664815a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:06.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664da00640 1 -- 172.21.15.47:0/2472193276 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6640009a70 con 0x7f66480caee0 2024-02-19T15:25:06.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664d000640 1 --2- 172.21.15.47:0/2472193276 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66480c3920 0x7f66481fb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:06.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664da00640 1 --2- 172.21.15.47:0/2472193276 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66480caee0 0x7f66480c9330 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f66400b0040 tx=0x7f6640019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:06.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f6644c00640 1 -- 172.21.15.47:0/2472193276 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66400ade40 con 0x7f66480caee0 2024-02-19T15:25:06.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f6644c00640 1 -- 172.21.15.47:0/2472193276 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66400b9450 con 0x7f66480caee0 2024-02-19T15:25:06.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f6644c00640 1 -- 172.21.15.47:0/2472193276 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66400b9800 con 0x7f66480caee0 2024-02-19T15:25:06.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664e400640 1 -- 172.21.15.47:0/2472193276 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66480c4c80 con 0x7f66480caee0 2024-02-19T15:25:06.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664e400640 1 -- 172.21.15.47:0/2472193276 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664815ac80 con 0x7f66480caee0 2024-02-19T15:25:06.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.541+0000 7f664e400640 1 -- 172.21.15.47:0/2472193276 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66480c4e70 con 0x7f66480caee0 2024-02-19T15:25:06.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.545+0000 7f6644c00640 1 -- 172.21.15.47:0/2472193276 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f66400b9960 con 0x7f66480caee0 2024-02-19T15:25:06.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.545+0000 7f6644c00640 1 --2- 172.21.15.47:0/2472193276 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6628066e40 0x7f6628069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:06.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.545+0000 7f6644c00640 1 -- 172.21.15.47:0/2472193276 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f664011fe00 con 0x7f66480caee0 2024-02-19T15:25:06.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.545+0000 7f664d000640 1 --2- 172.21.15.47:0/2472193276 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6628066e40 0x7f6628069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:06.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.545+0000 7f664d000640 1 --2- 172.21.15.47:0/2472193276 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6628066e40 0x7f6628069300 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f663c007fd0 tx=0x7f663c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:06.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.549+0000 7f6644c00640 1 -- 172.21.15.47:0/2472193276 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f66400ecff0 con 0x7f66480caee0 2024-02-19T15:25:06.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:06.709+0000 7f664e400640 1 -- 172.21.15.47:0/2472193276 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f66480c4ae0 con 0x7f66480caee0 2024-02-19T15:25:07.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.537+0000 7f6644c00640 1 -- 172.21.15.47:0/2472193276 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v66) v1 ==== 138+0+0 (secure 0 0 0) 0x7f66480c4ae0 con 0x7f66480caee0 2024-02-19T15:25:07.542 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_metadata' created 2024-02-19T15:25:07.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.541+0000 7f664e400640 1 -- 172.21.15.47:0/2472193276 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6628066e40 msgr2=0x7f6628069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:07.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.541+0000 7f664e400640 1 --2- 172.21.15.47:0/2472193276 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6628066e40 0x7f6628069300 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f663c007fd0 tx=0x7f663c009040 comp rx=0 tx=0).stop 2024-02-19T15:25:07.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.541+0000 7f664e400640 1 -- 172.21.15.47:0/2472193276 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66480caee0 msgr2=0x7f66480c9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:07.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.541+0000 7f664e400640 1 --2- 172.21.15.47:0/2472193276 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66480caee0 0x7f66480c9330 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f66400b0040 tx=0x7f6640019710 comp rx=0 tx=0).stop 2024-02-19T15:25:07.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.541+0000 7f664e400640 1 -- 172.21.15.47:0/2472193276 shutdown_connections 2024-02-19T15:25:07.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.541+0000 7f664e400640 1 --2- 172.21.15.47:0/2472193276 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6628066e40 0x7f6628069300 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:07.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.541+0000 7f664e400640 1 --2- 172.21.15.47:0/2472193276 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66480caee0 0x7f66480c9330 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:07.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.541+0000 7f664e400640 1 --2- 172.21.15.47:0/2472193276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66480c3e50 0x7f664815a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:07.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.541+0000 7f664e400640 1 --2- 172.21.15.47:0/2472193276 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66480c3920 0x7f66481fb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:07.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.541+0000 7f664e400640 1 -- 172.21.15.47:0/2472193276 >> 172.21.15.47:0/2472193276 conn(0x7f66480d0890 msgr2=0x7f6648074000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:07.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.541+0000 7f664e400640 1 -- 172.21.15.47:0/2472193276 shutdown_connections 2024-02-19T15:25:07.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.541+0000 7f664e400640 1 -- 172.21.15.47:0/2472193276 wait complete. 2024-02-19T15:25:07.574 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T15:25:07.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.721+0000 7f74c3800640 1 Processor -- start 2024-02-19T15:25:07.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.721+0000 7f74c3800640 1 -- start start 2024-02-19T15:25:07.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.721+0000 7f74c3800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f74bc06c920 0x7f74bc0c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:07.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.721+0000 7f74c3800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74bc0b5650 con 0x7f74bc0b5270 2024-02-19T15:25:07.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.721+0000 7f74c3800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74bc071560 con 0x7f74bc06c4c0 2024-02-19T15:25:07.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.721+0000 7f74c3800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74bc15f7b0 con 0x7f74bc06c920 2024-02-19T15:25:07.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.725+0000 7f74c1800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f74bc06c920 0x7f74bc0c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:07.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.725+0000 7f74c1800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f74bc06c920 0x7f74bc0c9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39810/0 (socket says 172.21.15.47:39810) 2024-02-19T15:25:07.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.725+0000 7f74c1800640 1 -- 172.21.15.47:0/1175658568 learned_addr learned my addr 172.21.15.47:0/1175658568 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:07.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.725+0000 7f74c0e00640 1 --1- 172.21.15.47:0/1175658568 >> v1:172.21.15.47:6789/0 conn(0x7f74bc0b5270 0x7f74bc0719f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38288/0 (socket says 172.21.15.47:38288) 2024-02-19T15:25:07.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.725+0000 7f74c1800640 1 -- 172.21.15.47:0/1175658568 >> v1:172.21.15.116:6789/0 conn(0x7f74bc06c4c0 legacy=0x7f74bc06e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:07.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.725+0000 7f74c1800640 1 -- 172.21.15.47:0/1175658568 >> v1:172.21.15.47:6789/0 conn(0x7f74bc0b5270 legacy=0x7f74bc0719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:07.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.725+0000 7f74c1800640 1 -- 172.21.15.47:0/1175658568 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74bc072320 con 0x7f74bc06c920 2024-02-19T15:25:07.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.725+0000 7f74c1800640 1 --2- 172.21.15.47:0/1175658568 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74bc06c920 0x7f74bc0c9fc0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f74b4009e50 tx=0x7f74b401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e16e8850b9bf805 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:07.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.725+0000 7f74bb400640 1 -- 172.21.15.47:0/1175658568 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b40a56c0 con 0x7f74bc06c920 2024-02-19T15:25:07.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.725+0000 7f74bb400640 1 -- 172.21.15.47:0/1175658568 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b40a5820 con 0x7f74bc06c920 2024-02-19T15:25:07.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.725+0000 7f74bb400640 1 -- 172.21.15.47:0/1175658568 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b40a5bb0 con 0x7f74bc06c920 2024-02-19T15:25:07.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c3800640 1 -- 172.21.15.47:0/1175658568 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74bc06c920 msgr2=0x7f74bc0c9fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:07.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c3800640 1 --2- 172.21.15.47:0/1175658568 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74bc06c920 0x7f74bc0c9fc0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f74b4009e50 tx=0x7f74b401c9f0 comp rx=0 tx=0).stop 2024-02-19T15:25:07.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c3800640 1 -- 172.21.15.47:0/1175658568 shutdown_connections 2024-02-19T15:25:07.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c3800640 1 --2- 172.21.15.47:0/1175658568 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74bc06c920 0x7f74bc0c9fc0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:07.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c3800640 1 -- 172.21.15.47:0/1175658568 >> 172.21.15.47:0/1175658568 conn(0x7f74bc0cc3f0 msgr2=0x7f74bc0cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:07.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c3800640 1 -- 172.21.15.47:0/1175658568 shutdown_connections 2024-02-19T15:25:07.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c3800640 1 -- 172.21.15.47:0/1175658568 wait complete. 2024-02-19T15:25:07.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c3800640 1 Processor -- start 2024-02-19T15:25:07.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c3800640 1 -- start start 2024-02-19T15:25:07.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c3800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74bc06c4c0 0x7f74bc14c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:07.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c3800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74bc06c920 0x7f74bc156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:07.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c3800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74bc0b5270 0x7f74bc0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:07.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c3800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc1584f0 con 0x7f74bc06c4c0 2024-02-19T15:25:07.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c3800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc14c590 con 0x7f74bc06c920 2024-02-19T15:25:07.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c3800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc1578e0 con 0x7f74bc0b5270 2024-02-19T15:25:07.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c0e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74bc06c4c0 0x7f74bc14c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:07.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c0e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74bc06c4c0 0x7f74bc14c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39812/0 (socket says 172.21.15.47:39812) 2024-02-19T15:25:07.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c0e00640 1 -- 172.21.15.47:0/3861552311 learned_addr learned my addr 172.21.15.47:0/3861552311 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:07.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74bbe00640 1 --2- 172.21.15.47:0/3861552311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74bc06c920 0x7f74bc156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:07.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c1800640 1 --2- 172.21.15.47:0/3861552311 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74bc0b5270 0x7f74bc0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:07.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c0e00640 1 -- 172.21.15.47:0/3861552311 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74bc0b5270 msgr2=0x7f74bc0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:07.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c0e00640 1 --2- 172.21.15.47:0/3861552311 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74bc0b5270 0x7f74bc0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:07.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c0e00640 1 -- 172.21.15.47:0/3861552311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74bc06c920 msgr2=0x7f74bc156eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:07.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c0e00640 1 --2- 172.21.15.47:0/3861552311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74bc06c920 0x7f74bc156eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:07.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.729+0000 7f74c0e00640 1 -- 172.21.15.47:0/3861552311 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b4009b30 con 0x7f74bc06c4c0 2024-02-19T15:25:07.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.733+0000 7f74c0e00640 1 --2- 172.21.15.47:0/3861552311 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74bc06c4c0 0x7f74bc14c050 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f74b000c450 tx=0x7f74b000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:07.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.733+0000 7f74b8c00640 1 -- 172.21.15.47:0/3861552311 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b0007480 con 0x7f74bc06c4c0 2024-02-19T15:25:07.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.733+0000 7f74b8c00640 1 -- 172.21.15.47:0/3861552311 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b00075e0 con 0x7f74bc06c4c0 2024-02-19T15:25:07.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.733+0000 7f74b8c00640 1 -- 172.21.15.47:0/3861552311 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b0004a30 con 0x7f74bc06c4c0 2024-02-19T15:25:07.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.733+0000 7f74c3800640 1 -- 172.21.15.47:0/3861552311 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74bc157bc0 con 0x7f74bc06c4c0 2024-02-19T15:25:07.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.733+0000 7f74c3800640 1 -- 172.21.15.47:0/3861552311 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74bc158180 con 0x7f74bc06c4c0 2024-02-19T15:25:07.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.733+0000 7f74c3800640 1 -- 172.21.15.47:0/3861552311 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74bc0c3ba0 con 0x7f74bc06c4c0 2024-02-19T15:25:07.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.733+0000 7f74b8c00640 1 -- 172.21.15.47:0/3861552311 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f74b0004b90 con 0x7f74bc06c4c0 2024-02-19T15:25:07.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.733+0000 7f74b8c00640 1 --2- 172.21.15.47:0/3861552311 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f749c066e40 0x7f749c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:07.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.733+0000 7f74b8c00640 1 -- 172.21.15.47:0/3861552311 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f74b0083be0 con 0x7f74bc06c4c0 2024-02-19T15:25:07.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.733+0000 7f74bbe00640 1 --2- 172.21.15.47:0/3861552311 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f749c066e40 0x7f749c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:07.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.737+0000 7f74bbe00640 1 --2- 172.21.15.47:0/3861552311 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f749c066e40 0x7f749c069300 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f74bc14fc60 tx=0x7f74a8009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:07.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.737+0000 7f74b8c00640 1 -- 172.21.15.47:0/3861552311 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f74bc0c3ba0 con 0x7f74bc06c4c0 2024-02-19T15:25:07.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:07.901+0000 7f74c3800640 1 -- 172.21.15.47:0/3861552311 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f74bc0c3ba0 con 0x7f74bc06c4c0 2024-02-19T15:25:08.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.545+0000 7f74b8c00640 1 -- 172.21.15.47:0/3861552311 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v67) v1 ==== 170+0+0 (secure 0 0 0) 0x7f74bc0c3ba0 con 0x7f74bc06c4c0 2024-02-19T15:25:08.550 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_data' created 2024-02-19T15:25:08.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.549+0000 7f74c3800640 1 -- 172.21.15.47:0/3861552311 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f749c066e40 msgr2=0x7f749c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:08.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.549+0000 7f74c3800640 1 --2- 172.21.15.47:0/3861552311 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f749c066e40 0x7f749c069300 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f74bc14fc60 tx=0x7f74a8009060 comp rx=0 tx=0).stop 2024-02-19T15:25:08.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.549+0000 7f74c3800640 1 -- 172.21.15.47:0/3861552311 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74bc06c4c0 msgr2=0x7f74bc14c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:08.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.549+0000 7f74c3800640 1 --2- 172.21.15.47:0/3861552311 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74bc06c4c0 0x7f74bc14c050 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f74b000c450 tx=0x7f74b000c910 comp rx=0 tx=0).stop 2024-02-19T15:25:08.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.549+0000 7f74c3800640 1 -- 172.21.15.47:0/3861552311 shutdown_connections 2024-02-19T15:25:08.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.549+0000 7f74c3800640 1 --2- 172.21.15.47:0/3861552311 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f749c066e40 0x7f749c069300 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:08.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.549+0000 7f74c3800640 1 --2- 172.21.15.47:0/3861552311 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74bc0b5270 0x7f74bc0ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:08.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.549+0000 7f74c3800640 1 --2- 172.21.15.47:0/3861552311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74bc06c920 0x7f74bc156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:08.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.549+0000 7f74c3800640 1 --2- 172.21.15.47:0/3861552311 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74bc06c4c0 0x7f74bc14c050 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:08.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.549+0000 7f74c3800640 1 -- 172.21.15.47:0/3861552311 >> 172.21.15.47:0/3861552311 conn(0x7f74bc0cc3f0 msgr2=0x7f74bc070640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:08.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.549+0000 7f74c3800640 1 -- 172.21.15.47:0/3861552311 shutdown_connections 2024-02-19T15:25:08.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.553+0000 7f74c3800640 1 -- 172.21.15.47:0/3861552311 wait complete. 2024-02-19T15:25:08.588 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T15:25:08.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.729+0000 7fc406a00640 1 Processor -- start 2024-02-19T15:25:08.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.729+0000 7fc406a00640 1 -- start start 2024-02-19T15:25:08.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.729+0000 7fc406a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4000cb490 0x7fc400070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:08.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.729+0000 7fc406a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc400072850 0x7fc40006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:08.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.729+0000 7fc406a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40015fc90 con 0x7fc40015bd90 2024-02-19T15:25:08.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.729+0000 7fc406a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc40015ff70 con 0x7fc400072850 2024-02-19T15:25:08.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.729+0000 7fc406a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc400160250 con 0x7fc4000cb490 2024-02-19T15:25:08.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.729+0000 7fc404a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc400072850 0x7fc40006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:08.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.729+0000 7fc3f7e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4000cb490 0x7fc400070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:08.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.729+0000 7fc3ffe00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc40015bd90 0x7fc400072140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:43188/0 (socket says 172.21.15.47:43188) 2024-02-19T15:25:08.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.729+0000 7fc3ffe00640 1 -- 172.21.15.47:0/2733898665 learned_addr learned my addr 172.21.15.47:0/2733898665 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:08.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.729+0000 7fc3ff400640 1 -- 172.21.15.47:0/2733898665 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887503172 0 0) 0x7fc40015fc90 con 0x7fc40015bd90 2024-02-19T15:25:08.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.729+0000 7fc3ff400640 1 -- 172.21.15.47:0/2733898665 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e4003530 con 0x7fc40015bd90 2024-02-19T15:25:08.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.729+0000 7fc404a00640 1 -- 172.21.15.47:0/2733898665 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4000cb490 msgr2=0x7fc400070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:08.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.729+0000 7fc404a00640 1 --2- 172.21.15.47:0/2733898665 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4000cb490 0x7fc400070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:08.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.729+0000 7fc404a00640 1 -- 172.21.15.47:0/2733898665 >> v1:172.21.15.116:6789/0 conn(0x7fc40015bd90 legacy=0x7fc400072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:08.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.729+0000 7fc404a00640 1 -- 172.21.15.47:0/2733898665 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc400160530 con 0x7fc400072850 2024-02-19T15:25:08.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.729+0000 7fc3f7e00640 1 --2- 172.21.15.47:0/2733898665 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4000cb490 0x7fc400070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:25:08.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.733+0000 7fc404a00640 1 --2- 172.21.15.47:0/2733898665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc400072850 0x7fc40006cc60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8009d90 tx=0x7fc3f801c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=283f935caac99ad2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:08.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.733+0000 7fc3ff400640 1 -- 172.21.15.47:0/2733898665 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f80a5820 con 0x7fc400072850 2024-02-19T15:25:08.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.733+0000 7fc3ff400640 1 -- 172.21.15.47:0/2733898665 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f80a5980 con 0x7fc400072850 2024-02-19T15:25:08.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.733+0000 7fc3ff400640 1 -- 172.21.15.47:0/2733898665 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f80a5d10 con 0x7fc400072850 2024-02-19T15:25:08.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.733+0000 7fc406a00640 1 -- 172.21.15.47:0/2733898665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc400072850 msgr2=0x7fc40006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:08.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.733+0000 7fc406a00640 1 --2- 172.21.15.47:0/2733898665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc400072850 0x7fc40006cc60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8009d90 tx=0x7fc3f801c810 comp rx=0 tx=0).stop 2024-02-19T15:25:08.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.733+0000 7fc406a00640 1 -- 172.21.15.47:0/2733898665 shutdown_connections 2024-02-19T15:25:08.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.733+0000 7fc406a00640 1 --2- 172.21.15.47:0/2733898665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc400072850 0x7fc40006cc60 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:08.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.733+0000 7fc406a00640 1 --2- 172.21.15.47:0/2733898665 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4000cb490 0x7fc400070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:08.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.733+0000 7fc406a00640 1 -- 172.21.15.47:0/2733898665 >> 172.21.15.47:0/2733898665 conn(0x7fc4000cfc20 msgr2=0x7fc4000d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:08.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.733+0000 7fc406a00640 1 -- 172.21.15.47:0/2733898665 shutdown_connections 2024-02-19T15:25:08.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.733+0000 7fc406a00640 1 -- 172.21.15.47:0/2733898665 wait complete. 2024-02-19T15:25:08.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc406a00640 1 Processor -- start 2024-02-19T15:25:08.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc406a00640 1 -- start start 2024-02-19T15:25:08.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc406a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc400072850 0x7fc40014c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:08.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc406a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4000cb490 0x7fc400164e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:08.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc406a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc40015bd90 0x7fc4001653c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:08.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc406a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4000cf520 con 0x7fc4000cb490 2024-02-19T15:25:08.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc406a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4000ce910 con 0x7fc400072850 2024-02-19T15:25:08.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc406a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4000cebf0 con 0x7fc40015bd90 2024-02-19T15:25:08.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc3f7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4000cb490 0x7fc400164e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:08.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc3f7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4000cb490 0x7fc400164e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39814/0 (socket says 172.21.15.47:39814) 2024-02-19T15:25:08.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc3f7e00640 1 -- 172.21.15.47:0/701494764 learned_addr learned my addr 172.21.15.47:0/701494764 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:08.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc404a00640 1 --2- 172.21.15.47:0/701494764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc40015bd90 0x7fc4001653c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc3ffe00640 1 --2- 172.21.15.47:0/701494764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc400072850 0x7fc40014c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc3f7e00640 1 -- 172.21.15.47:0/701494764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc40015bd90 msgr2=0x7fc4001653c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc3f7e00640 1 --2- 172.21.15.47:0/701494764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc40015bd90 0x7fc4001653c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc3f7e00640 1 -- 172.21.15.47:0/701494764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc400072850 msgr2=0x7fc40014c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc3f7e00640 1 --2- 172.21.15.47:0/701494764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc400072850 0x7fc40014c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc3f7e00640 1 -- 172.21.15.47:0/701494764 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e8009420 con 0x7fc4000cb490 2024-02-19T15:25:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc3f7e00640 1 --2- 172.21.15.47:0/701494764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4000cb490 0x7fc400164e80 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800ebe0 tx=0x7fc3e800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc3fcc00640 1 -- 172.21.15.47:0/701494764 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e800eee0 con 0x7fc4000cb490 2024-02-19T15:25:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc3fcc00640 1 -- 172.21.15.47:0/701494764 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e800cc30 con 0x7fc4000cb490 2024-02-19T15:25:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc3fcc00640 1 -- 172.21.15.47:0/701494764 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8013470 con 0x7fc4000cb490 2024-02-19T15:25:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc406a00640 1 -- 172.21.15.47:0/701494764 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f8009a70 con 0x7fc4000cb490 2024-02-19T15:25:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc406a00640 1 -- 172.21.15.47:0/701494764 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4000cf1b0 con 0x7fc4000cb490 2024-02-19T15:25:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.737+0000 7fc406a00640 1 -- 172.21.15.47:0/701494764 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc400075360 con 0x7fc4000cb490 2024-02-19T15:25:08.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.741+0000 7fc3fcc00640 1 -- 172.21.15.47:0/701494764 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc3e80135d0 con 0x7fc4000cb490 2024-02-19T15:25:08.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.741+0000 7fc3fcc00640 1 --2- 172.21.15.47:0/701494764 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc3dc066e70 0x7fc3dc069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:08.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.741+0000 7fc3fcc00640 1 -- 172.21.15.47:0/701494764 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fc3e8085410 con 0x7fc4000cb490 2024-02-19T15:25:08.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.741+0000 7fc3ffe00640 1 --2- 172.21.15.47:0/701494764 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc3dc066e70 0x7fc3dc069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:08.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.741+0000 7fc3ffe00640 1 --2- 172.21.15.47:0/701494764 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc3dc066e70 0x7fc3dc069330 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc3f00011f0 tx=0x7fc3f0007d40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:08.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.745+0000 7fc3fcc00640 1 -- 172.21.15.47:0/701494764 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc3e8052380 con 0x7fc4000cb490 2024-02-19T15:25:08.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:08.941+0000 7fc406a00640 1 -- 172.21.15.47:0/701494764 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc40006e800 con 0x7fc4000cb490 2024-02-19T15:25:09.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.565+0000 7fc3fcc00640 1 -- 172.21.15.47:0/701494764 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v37) v1 ==== 174+0+0 (secure 0 0 0) 0x7fc3e8055a40 con 0x7fc4000cb490 2024-02-19T15:25:09.574 INFO:teuthology.orchestra.run.smithi047.stderr:new fs with metadata pool 10 and data pool 11 2024-02-19T15:25:09.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.573+0000 7fc406a00640 1 -- 172.21.15.47:0/701494764 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc3dc066e70 msgr2=0x7fc3dc069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:09.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.573+0000 7fc406a00640 1 --2- 172.21.15.47:0/701494764 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc3dc066e70 0x7fc3dc069330 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc3f00011f0 tx=0x7fc3f0007d40 comp rx=0 tx=0).stop 2024-02-19T15:25:09.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.573+0000 7fc406a00640 1 -- 172.21.15.47:0/701494764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4000cb490 msgr2=0x7fc400164e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:09.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.573+0000 7fc406a00640 1 --2- 172.21.15.47:0/701494764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4000cb490 0x7fc400164e80 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800ebe0 tx=0x7fc3e800c520 comp rx=0 tx=0).stop 2024-02-19T15:25:09.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.577+0000 7fc406a00640 1 -- 172.21.15.47:0/701494764 shutdown_connections 2024-02-19T15:25:09.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.577+0000 7fc406a00640 1 --2- 172.21.15.47:0/701494764 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc3dc066e70 0x7fc3dc069330 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:09.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.577+0000 7fc406a00640 1 --2- 172.21.15.47:0/701494764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc40015bd90 0x7fc4001653c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:09.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.577+0000 7fc406a00640 1 --2- 172.21.15.47:0/701494764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4000cb490 0x7fc400164e80 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:09.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.577+0000 7fc406a00640 1 --2- 172.21.15.47:0/701494764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc400072850 0x7fc40014c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:09.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.577+0000 7fc406a00640 1 -- 172.21.15.47:0/701494764 >> 172.21.15.47:0/701494764 conn(0x7fc4000cfc20 msgr2=0x7fc400072100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:09.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.581+0000 7fc406a00640 1 -- 172.21.15.47:0/701494764 shutdown_connections 2024-02-19T15:25:09.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.581+0000 7fc406a00640 1 -- 172.21.15.47:0/701494764 wait complete. 2024-02-19T15:25:09.614 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:25:09.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.753+0000 7f2b77a00640 1 Processor -- start 2024-02-19T15:25:09.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.753+0000 7f2b77a00640 1 -- start start 2024-02-19T15:25:09.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.757+0000 7f2b77a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b701614e0 0x7f2b7015f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:09.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.757+0000 7f2b77a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2b7015eee0 0x7f2b700b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:09.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.757+0000 7f2b77a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b7006b5d0 con 0x7f2b700b2330 2024-02-19T15:25:09.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.757+0000 7f2b77a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b700b1c00 con 0x7f2b7015eee0 2024-02-19T15:25:09.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.757+0000 7f2b77a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b700b1640 con 0x7f2b701614e0 2024-02-19T15:25:09.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.757+0000 7f2b67e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2b7015eee0 0x7f2b700b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:09.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.757+0000 7f2b75a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f2b700b2330 0x7f2b7006ba60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38296/0 (socket says 172.21.15.47:38296) 2024-02-19T15:25:09.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.757+0000 7f2b75a00640 1 -- 172.21.15.47:0/2966726437 learned_addr learned my addr 172.21.15.47:0/2966726437 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:09.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.757+0000 7f2b67e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2b7015eee0 0x7f2b700b1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39816/0 (socket says 172.21.15.47:39816) 2024-02-19T15:25:09.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.757+0000 7f2b75000640 1 --2- 172.21.15.47:0/2966726437 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b701614e0 0x7f2b7015f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:09.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.757+0000 7f2b67e00640 1 -- 172.21.15.47:0/2966726437 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b701614e0 msgr2=0x7f2b7015f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:09.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.757+0000 7f2b67e00640 1 --2- 172.21.15.47:0/2966726437 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b701614e0 0x7f2b7015f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:09.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.757+0000 7f2b67e00640 1 -- 172.21.15.47:0/2966726437 >> v1:172.21.15.47:6789/0 conn(0x7f2b700b2330 legacy=0x7f2b7006ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:09.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.757+0000 7f2b67e00640 1 -- 172.21.15.47:0/2966726437 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b700b2790 con 0x7f2b7015eee0 2024-02-19T15:25:09.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.757+0000 7f2b6fe00640 1 -- 172.21.15.47:0/2966726437 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1997953326 0 0) 0x7f2b7006b5d0 con 0x7f2b700b2330 2024-02-19T15:25:09.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.757+0000 7f2b67e00640 1 --2- 172.21.15.47:0/2966726437 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b7015eee0 0x7f2b700b1df0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f2b580097c0 tx=0x7f2b5801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826fdb32f182856f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:09.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b6fe00640 1 -- 172.21.15.47:0/2966726437 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b580a6040 con 0x7f2b7015eee0 2024-02-19T15:25:09.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b6fe00640 1 -- 172.21.15.47:0/2966726437 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b580aa600 con 0x7f2b7015eee0 2024-02-19T15:25:09.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b6fe00640 1 -- 172.21.15.47:0/2966726437 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b580aa990 con 0x7f2b7015eee0 2024-02-19T15:25:09.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b77a00640 1 -- 172.21.15.47:0/2966726437 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b7015eee0 msgr2=0x7f2b700b1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:09.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b77a00640 1 --2- 172.21.15.47:0/2966726437 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b7015eee0 0x7f2b700b1df0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f2b580097c0 tx=0x7f2b5801c4a0 comp rx=0 tx=0).stop 2024-02-19T15:25:09.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b77a00640 1 -- 172.21.15.47:0/2966726437 shutdown_connections 2024-02-19T15:25:09.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b77a00640 1 --2- 172.21.15.47:0/2966726437 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b7015eee0 0x7f2b700b1df0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:09.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b77a00640 1 --2- 172.21.15.47:0/2966726437 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b701614e0 0x7f2b7015f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:09.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b77a00640 1 -- 172.21.15.47:0/2966726437 >> 172.21.15.47:0/2966726437 conn(0x7f2b700b4fc0 msgr2=0x7f2b700b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:09.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b77a00640 1 -- 172.21.15.47:0/2966726437 shutdown_connections 2024-02-19T15:25:09.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b77a00640 1 -- 172.21.15.47:0/2966726437 wait complete. 2024-02-19T15:25:09.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b77a00640 1 Processor -- start 2024-02-19T15:25:09.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b77a00640 1 -- start start 2024-02-19T15:25:09.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b77a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b700b2330 0x7f2b7015f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:09.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b77a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b7015eee0 0x7f2b700ad870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:09.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b77a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b701614e0 0x7f2b70153fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:09.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b77a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b700ad140 con 0x7f2b701614e0 2024-02-19T15:25:09.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b77a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b700ad3e0 con 0x7f2b700b2330 2024-02-19T15:25:09.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b77a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b70158e50 con 0x7f2b7015eee0 2024-02-19T15:25:09.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b75a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b701614e0 0x7f2b70153fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:09.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b75000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b700b2330 0x7f2b7015f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:09.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b75a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b701614e0 0x7f2b70153fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39820/0 (socket says 172.21.15.47:39820) 2024-02-19T15:25:09.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b75a00640 1 -- 172.21.15.47:0/2920077732 learned_addr learned my addr 172.21.15.47:0/2920077732 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:09.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.761+0000 7f2b67e00640 1 --2- 172.21.15.47:0/2920077732 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b7015eee0 0x7f2b700ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:09.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.765+0000 7f2b75a00640 1 -- 172.21.15.47:0/2920077732 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b7015eee0 msgr2=0x7f2b700ad870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:09.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.765+0000 7f2b75a00640 1 --2- 172.21.15.47:0/2920077732 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b7015eee0 0x7f2b700ad870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:09.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.765+0000 7f2b75a00640 1 -- 172.21.15.47:0/2920077732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b700b2330 msgr2=0x7f2b7015f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:09.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.765+0000 7f2b75a00640 1 --2- 172.21.15.47:0/2920077732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b700b2330 0x7f2b7015f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:09.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.765+0000 7f2b75a00640 1 -- 172.21.15.47:0/2920077732 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b58009420 con 0x7f2b701614e0 2024-02-19T15:25:09.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.765+0000 7f2b75a00640 1 --2- 172.21.15.47:0/2920077732 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b701614e0 0x7f2b70153fd0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f2b6800c7d0 tx=0x7f2b6800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:09.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.765+0000 7f2b6d600640 1 -- 172.21.15.47:0/2920077732 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6800a640 con 0x7f2b701614e0 2024-02-19T15:25:09.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.765+0000 7f2b6d600640 1 -- 172.21.15.47:0/2920077732 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b6800a7a0 con 0x7f2b701614e0 2024-02-19T15:25:09.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.765+0000 7f2b6d600640 1 -- 172.21.15.47:0/2920077732 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6800ab90 con 0x7f2b701614e0 2024-02-19T15:25:09.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.765+0000 7f2b77a00640 1 -- 172.21.15.47:0/2920077732 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b70159130 con 0x7f2b701614e0 2024-02-19T15:25:09.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.765+0000 7f2b77a00640 1 -- 172.21.15.47:0/2920077732 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7015fcf0 con 0x7f2b701614e0 2024-02-19T15:25:09.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.765+0000 7f2b77a00640 1 -- 172.21.15.47:0/2920077732 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b700ad570 con 0x7f2b701614e0 2024-02-19T15:25:09.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.765+0000 7f2b6d600640 1 -- 172.21.15.47:0/2920077732 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2b6800acf0 con 0x7f2b701614e0 2024-02-19T15:25:09.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.765+0000 7f2b6d600640 1 --2- 172.21.15.47:0/2920077732 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2b4c066e40 0x7f2b4c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:09.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.765+0000 7f2b6d600640 1 -- 172.21.15.47:0/2920077732 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b68084260 con 0x7f2b701614e0 2024-02-19T15:25:09.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.769+0000 7f2b75000640 1 --2- 172.21.15.47:0/2920077732 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2b4c066e40 0x7f2b4c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:09.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.769+0000 7f2b75000640 1 --2- 172.21.15.47:0/2920077732 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2b4c066e40 0x7f2b4c069300 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2b60009ac0 tx=0x7f2b60009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:09.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.773+0000 7f2b6d600640 1 -- 172.21.15.47:0/2920077732 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2b68051060 con 0x7f2b701614e0 2024-02-19T15:25:09.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.937+0000 7f2b77a00640 1 -- 172.21.15.47:0/2920077732 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b70075360 con 0x7f2b701614e0 2024-02-19T15:25:09.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.937+0000 7f2b6d600640 1 -- 172.21.15.47:0/2920077732 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16693 (secure 0 0 0) 0x7f2b68054720 con 0x7f2b701614e0 2024-02-19T15:25:09.942 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:09.942 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":68,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:25:08.947383+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:25:06.759758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-02-19T15:25:07.949435+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:25:09.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.941+0000 7f2b77a00640 1 -- 172.21.15.47:0/2920077732 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2b4c066e40 msgr2=0x7f2b4c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:09.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.941+0000 7f2b77a00640 1 --2- 172.21.15.47:0/2920077732 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2b4c066e40 0x7f2b4c069300 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2b60009ac0 tx=0x7f2b60009190 comp rx=0 tx=0).stop 2024-02-19T15:25:09.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.941+0000 7f2b77a00640 1 -- 172.21.15.47:0/2920077732 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b701614e0 msgr2=0x7f2b70153fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:09.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.941+0000 7f2b77a00640 1 --2- 172.21.15.47:0/2920077732 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b701614e0 0x7f2b70153fd0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f2b6800c7d0 tx=0x7f2b6800cc90 comp rx=0 tx=0).stop 2024-02-19T15:25:09.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.941+0000 7f2b77a00640 1 -- 172.21.15.47:0/2920077732 shutdown_connections 2024-02-19T15:25:09.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.941+0000 7f2b77a00640 1 --2- 172.21.15.47:0/2920077732 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2b4c066e40 0x7f2b4c069300 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:09.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.941+0000 7f2b77a00640 1 --2- 172.21.15.47:0/2920077732 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b701614e0 0x7f2b70153fd0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:09.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.941+0000 7f2b77a00640 1 --2- 172.21.15.47:0/2920077732 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b7015eee0 0x7f2b700ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:09.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.941+0000 7f2b77a00640 1 --2- 172.21.15.47:0/2920077732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b700b2330 0x7f2b7015f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:09.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.941+0000 7f2b77a00640 1 -- 172.21.15.47:0/2920077732 >> 172.21.15.47:0/2920077732 conn(0x7f2b700b4fc0 msgr2=0x7f2b7006ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:09.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.941+0000 7f2b77a00640 1 -- 172.21.15.47:0/2920077732 shutdown_connections 2024-02-19T15:25:09.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:09.941+0000 7f2b77a00640 1 -- 172.21.15.47:0/2920077732 wait complete. 2024-02-19T15:25:09.963 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:25:10.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.101+0000 7fc79a200640 1 Processor -- start 2024-02-19T15:25:10.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.101+0000 7fc79a200640 1 -- start start 2024-02-19T15:25:10.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.101+0000 7fc79a200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc7940d0e70 0x7fc7940b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:10.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.101+0000 7fc79a200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7940d4970 con 0x7fc79415e250 2024-02-19T15:25:10.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.101+0000 7fc79a200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7940b3240 con 0x7fc7940cfb30 2024-02-19T15:25:10.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.101+0000 7fc79a200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc79415e630 con 0x7fc7940d0e70 2024-02-19T15:25:10.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.101+0000 7fc793e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc7940d0e70 0x7fc7940b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:10.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.101+0000 7fc793400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc79415e250 0x7fc7940d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38308/0 (socket says 172.21.15.47:38308) 2024-02-19T15:25:10.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.101+0000 7fc793400640 1 -- 172.21.15.47:0/3680487532 learned_addr learned my addr 172.21.15.47:0/3680487532 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:10.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.101+0000 7fc793e00640 1 -- 172.21.15.47:0/3680487532 >> v1:172.21.15.116:6789/0 conn(0x7fc7940cfb30 legacy=0x7fc7940b3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:10.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.101+0000 7fc793e00640 1 -- 172.21.15.47:0/3680487532 >> v1:172.21.15.47:6789/0 conn(0x7fc79415e250 legacy=0x7fc7940d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:10.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.105+0000 7fc793e00640 1 -- 172.21.15.47:0/3680487532 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7940d4bc0 con 0x7fc7940d0e70 2024-02-19T15:25:10.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.105+0000 7fc793e00640 1 --2- 172.21.15.47:0/3680487532 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7940d0e70 0x7fc7940b43a0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc78c0097c0 tx=0x7fc78c01c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3195c8db7b15181c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:10.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.105+0000 7fc792a00640 1 -- 172.21.15.47:0/3680487532 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c004230 con 0x7fc7940d0e70 2024-02-19T15:25:10.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.105+0000 7fc792a00640 1 -- 172.21.15.47:0/3680487532 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78c0a7050 con 0x7fc7940d0e70 2024-02-19T15:25:10.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.105+0000 7fc792a00640 1 -- 172.21.15.47:0/3680487532 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c003620 con 0x7fc7940d0e70 2024-02-19T15:25:10.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.105+0000 7fc79a200640 1 -- 172.21.15.47:0/3680487532 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7940d0e70 msgr2=0x7fc7940b43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:10.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.105+0000 7fc79a200640 1 --2- 172.21.15.47:0/3680487532 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7940d0e70 0x7fc7940b43a0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc78c0097c0 tx=0x7fc78c01c350 comp rx=0 tx=0).stop 2024-02-19T15:25:10.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.105+0000 7fc79a200640 1 -- 172.21.15.47:0/3680487532 shutdown_connections 2024-02-19T15:25:10.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.105+0000 7fc79a200640 1 --2- 172.21.15.47:0/3680487532 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7940d0e70 0x7fc7940b43a0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:10.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.105+0000 7fc79a200640 1 -- 172.21.15.47:0/3680487532 >> 172.21.15.47:0/3680487532 conn(0x7fc7940d3e50 msgr2=0x7fc7940d4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:10.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.105+0000 7fc79a200640 1 -- 172.21.15.47:0/3680487532 shutdown_connections 2024-02-19T15:25:10.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.105+0000 7fc79a200640 1 -- 172.21.15.47:0/3680487532 wait complete. 2024-02-19T15:25:10.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.109+0000 7fc79a200640 1 Processor -- start 2024-02-19T15:25:10.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.109+0000 7fc79a200640 1 -- start start 2024-02-19T15:25:10.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc79a200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7940cfb30 0x7fc7940c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:10.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc79a200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7940d0e70 0x7fc7940b9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:10.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc79a200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc79415e250 0x7fc7940b6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:10.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc79a200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7940c9ad0 con 0x7fc7940cfb30 2024-02-19T15:25:10.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc79a200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7940c93b0 con 0x7fc79415e250 2024-02-19T15:25:10.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc79a200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79415e930 con 0x7fc7940d0e70 2024-02-19T15:25:10.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc793400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7940cfb30 0x7fc7940c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:10.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc793400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7940cfb30 0x7fc7940c9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39848/0 (socket says 172.21.15.47:39848) 2024-02-19T15:25:10.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc793400640 1 -- 172.21.15.47:0/2448140221 learned_addr learned my addr 172.21.15.47:0/2448140221 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:10.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc78aa00640 1 --2- 172.21.15.47:0/2448140221 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7940d0e70 0x7fc7940b9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:10.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc793e00640 1 --2- 172.21.15.47:0/2448140221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc79415e250 0x7fc7940b6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:10.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc793400640 1 -- 172.21.15.47:0/2448140221 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7940d0e70 msgr2=0x7fc7940b9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:10.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc793400640 1 --2- 172.21.15.47:0/2448140221 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7940d0e70 0x7fc7940b9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:10.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc793400640 1 -- 172.21.15.47:0/2448140221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc79415e250 msgr2=0x7fc7940b6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:10.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc793400640 1 --2- 172.21.15.47:0/2448140221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc79415e250 0x7fc7940b6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:10.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc793400640 1 -- 172.21.15.47:0/2448140221 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c009420 con 0x7fc7940cfb30 2024-02-19T15:25:10.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc793400640 1 --2- 172.21.15.47:0/2448140221 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7940cfb30 0x7fc7940c9e70 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc78400c400 tx=0x7fc78400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:10.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc78be00640 1 -- 172.21.15.47:0/2448140221 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc784005cf0 con 0x7fc7940cfb30 2024-02-19T15:25:10.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc78be00640 1 -- 172.21.15.47:0/2448140221 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc784005e50 con 0x7fc7940cfb30 2024-02-19T15:25:10.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc78be00640 1 -- 172.21.15.47:0/2448140221 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7840072d0 con 0x7fc7940cfb30 2024-02-19T15:25:10.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc79a200640 1 -- 172.21.15.47:0/2448140221 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7940c90d0 con 0x7fc7940cfb30 2024-02-19T15:25:10.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc79a200640 1 -- 172.21.15.47:0/2448140221 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc79414c580 con 0x7fc7940cfb30 2024-02-19T15:25:10.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc79a200640 1 -- 172.21.15.47:0/2448140221 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc794068500 con 0x7fc7940cfb30 2024-02-19T15:25:10.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.113+0000 7fc78be00640 1 -- 172.21.15.47:0/2448140221 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc784007430 con 0x7fc7940cfb30 2024-02-19T15:25:10.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.117+0000 7fc78be00640 1 --2- 172.21.15.47:0/2448140221 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc770066e40 0x7fc770069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:10.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.117+0000 7fc78be00640 1 -- 172.21.15.47:0/2448140221 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc784083ee0 con 0x7fc7940cfb30 2024-02-19T15:25:10.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.117+0000 7fc78aa00640 1 --2- 172.21.15.47:0/2448140221 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc770066e40 0x7fc770069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:10.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.117+0000 7fc78aa00640 1 --2- 172.21.15.47:0/2448140221 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc770066e40 0x7fc770069300 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc77c0012f0 tx=0x7fc77c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:10.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.125+0000 7fc78be00640 1 -- 172.21.15.47:0/2448140221 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc784050d70 con 0x7fc7940cfb30 2024-02-19T15:25:10.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.293+0000 7fc79a200640 1 -- 172.21.15.47:0/2448140221 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc794075360 con 0x7fc7940cfb30 2024-02-19T15:25:10.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.293+0000 7fc78be00640 1 -- 172.21.15.47:0/2448140221 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16693 (secure 0 0 0) 0x7fc784017090 con 0x7fc7940cfb30 2024-02-19T15:25:10.299 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:10.299 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":68,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:25:08.947383+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:25:06.759758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-02-19T15:25:07.949435+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:25:10.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.297+0000 7fc79a200640 1 -- 172.21.15.47:0/2448140221 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc770066e40 msgr2=0x7fc770069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:10.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.297+0000 7fc79a200640 1 --2- 172.21.15.47:0/2448140221 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc770066e40 0x7fc770069300 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc77c0012f0 tx=0x7fc77c009040 comp rx=0 tx=0).stop 2024-02-19T15:25:10.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.297+0000 7fc79a200640 1 -- 172.21.15.47:0/2448140221 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7940cfb30 msgr2=0x7fc7940c9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:10.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.297+0000 7fc79a200640 1 --2- 172.21.15.47:0/2448140221 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7940cfb30 0x7fc7940c9e70 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc78400c400 tx=0x7fc78400c8c0 comp rx=0 tx=0).stop 2024-02-19T15:25:10.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.297+0000 7fc79a200640 1 -- 172.21.15.47:0/2448140221 shutdown_connections 2024-02-19T15:25:10.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.297+0000 7fc79a200640 1 --2- 172.21.15.47:0/2448140221 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc770066e40 0x7fc770069300 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:10.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.297+0000 7fc79a200640 1 --2- 172.21.15.47:0/2448140221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc79415e250 0x7fc7940b6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:10.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.297+0000 7fc79a200640 1 --2- 172.21.15.47:0/2448140221 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7940d0e70 0x7fc7940b9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:10.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.297+0000 7fc79a200640 1 --2- 172.21.15.47:0/2448140221 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7940cfb30 0x7fc7940c9e70 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:10.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.297+0000 7fc79a200640 1 -- 172.21.15.47:0/2448140221 >> 172.21.15.47:0/2448140221 conn(0x7fc7940d3e50 msgr2=0x7fc7940b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:10.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.297+0000 7fc79a200640 1 -- 172.21.15.47:0/2448140221 shutdown_connections 2024-02-19T15:25:10.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.297+0000 7fc79a200640 1 -- 172.21.15.47:0/2448140221 wait complete. 2024-02-19T15:25:10.320 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T15:25:10.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.457+0000 7f01ab400640 1 Processor -- start 2024-02-19T15:25:10.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01ab400640 1 -- start start 2024-02-19T15:25:10.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01ab400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01a40cb490 0x7f01a4070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:10.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01ab400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a415fc90 con 0x7f01a415bd90 2024-02-19T15:25:10.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01ab400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a415ff70 con 0x7f01a4072850 2024-02-19T15:25:10.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01ab400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a4160250 con 0x7f01a40cb490 2024-02-19T15:25:10.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a8a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f01a415bd90 0x7f01a4072140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38310/0 (socket says 172.21.15.47:38310) 2024-02-19T15:25:10.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a8a00640 1 -- 172.21.15.47:0/2377794577 learned_addr learned my addr 172.21.15.47:0/2377794577 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:10.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f019be00640 1 --2- 172.21.15.47:0/2377794577 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01a40cb490 0x7f01a4070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:10.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a3e00640 1 -- 172.21.15.47:0/2377794577 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588985074 0 0) 0x7f01a415fc90 con 0x7f01a415bd90 2024-02-19T15:25:10.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a3e00640 1 -- 172.21.15.47:0/2377794577 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0188003530 con 0x7f01a415bd90 2024-02-19T15:25:10.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a3e00640 1 -- 172.21.15.47:0/2377794577 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3518605503 0 0) 0x7f0188003530 con 0x7f01a415bd90 2024-02-19T15:25:10.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a3e00640 1 -- 172.21.15.47:0/2377794577 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01a40cb490 msgr2=0x7f01a4070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:10.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a3e00640 1 --2- 172.21.15.47:0/2377794577 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01a40cb490 0x7f01a4070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:10.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a3e00640 1 -- 172.21.15.47:0/2377794577 >> v1:172.21.15.116:6789/0 conn(0x7f01a4072850 legacy=0x7f01a406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:10.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a3e00640 1 -- 172.21.15.47:0/2377794577 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a4160530 con 0x7f01a415bd90 2024-02-19T15:25:10.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a3e00640 1 -- 172.21.15.47:0/2377794577 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f0194001680 con 0x7f01a415bd90 2024-02-19T15:25:10.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a3e00640 1 -- 172.21.15.47:0/2377794577 >> v1:172.21.15.47:6789/0 conn(0x7f01a415bd90 legacy=0x7f01a4072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:10.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a3e00640 1 --2- 172.21.15.47:0/2377794577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01880039b0 0x7f0188003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:10.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a3e00640 1 --2- 172.21.15.47:0/2377794577 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01a40cb490 0x7f01880047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:10.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a3e00640 1 --2- 172.21.15.47:0/2377794577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0188004d10 0x7f0188005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:10.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a3e00640 1 -- 172.21.15.47:0/2377794577 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a4160250 con 0x7f0188004d10 2024-02-19T15:25:10.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a3e00640 1 -- 172.21.15.47:0/2377794577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018800bb20 con 0x7f01880039b0 2024-02-19T15:25:10.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a3e00640 1 -- 172.21.15.47:0/2377794577 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018800bce0 con 0x7f01a40cb490 2024-02-19T15:25:10.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a9400640 1 --2- 172.21.15.47:0/2377794577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0188004d10 0x7f0188005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:10.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f019be00640 1 --2- 172.21.15.47:0/2377794577 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01a40cb490 0x7f01880047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:10.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a8a00640 1 --2- 172.21.15.47:0/2377794577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01880039b0 0x7f0188003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:10.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a9400640 1 -- 172.21.15.47:0/2377794577 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01a40cb490 msgr2=0x7f01880047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:10.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a9400640 1 --2- 172.21.15.47:0/2377794577 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01a40cb490 0x7f01880047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:10.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a9400640 1 -- 172.21.15.47:0/2377794577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01880039b0 msgr2=0x7f0188003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:10.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a9400640 1 --2- 172.21.15.47:0/2377794577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01880039b0 0x7f0188003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:10.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.461+0000 7f01a9400640 1 -- 172.21.15.47:0/2377794577 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0188003530 con 0x7f0188004d10 2024-02-19T15:25:10.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.465+0000 7f01a9400640 1 --2- 172.21.15.47:0/2377794577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0188004d10 0x7f0188005190 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f019c001f60 tx=0x7f019c01eff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4accee0a7d5408 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:10.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.465+0000 7f01a3e00640 1 -- 172.21.15.47:0/2377794577 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c0a8c10 con 0x7f0188004d10 2024-02-19T15:25:10.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.465+0000 7f01a3e00640 1 -- 172.21.15.47:0/2377794577 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019c0a8d70 con 0x7f0188004d10 2024-02-19T15:25:10.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.465+0000 7f01a3e00640 1 -- 172.21.15.47:0/2377794577 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c0b0680 con 0x7f0188004d10 2024-02-19T15:25:10.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.465+0000 7f01ab400640 1 -- 172.21.15.47:0/2377794577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0188004d10 msgr2=0x7f0188005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:10.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.465+0000 7f01ab400640 1 --2- 172.21.15.47:0/2377794577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0188004d10 0x7f0188005190 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f019c001f60 tx=0x7f019c01eff0 comp rx=0 tx=0).stop 2024-02-19T15:25:10.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.465+0000 7f01a9400640 1 -- 172.21.15.47:0/2377794577 reap_dead start 2024-02-19T15:25:10.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.465+0000 7f01ab400640 1 -- 172.21.15.47:0/2377794577 shutdown_connections 2024-02-19T15:25:10.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.465+0000 7f01ab400640 1 -- 172.21.15.47:0/2377794577 >> 172.21.15.47:0/2377794577 conn(0x7f01a40cfc20 msgr2=0x7f01a40d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:10.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.465+0000 7f01ab400640 1 -- 172.21.15.47:0/2377794577 shutdown_connections 2024-02-19T15:25:10.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.465+0000 7f01ab400640 1 -- 172.21.15.47:0/2377794577 wait complete. 2024-02-19T15:25:10.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01ab400640 1 Processor -- start 2024-02-19T15:25:10.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01ab400640 1 -- start start 2024-02-19T15:25:10.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01ab400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0188004d10 0x7f01a40a9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:10.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01ab400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01a40c7d40 0x7f01a40ab900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:10.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01ab400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01a415ece0 0x7f01a40c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:10.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01ab400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a40abe40 con 0x7f0188004d10 2024-02-19T15:25:10.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01ab400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a40c84e0 con 0x7f01a40c7d40 2024-02-19T15:25:10.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01ab400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a40c87c0 con 0x7f01a415ece0 2024-02-19T15:25:10.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01a8a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0188004d10 0x7f01a40a9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:10.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01a8a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0188004d10 0x7f01a40a9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39864/0 (socket says 172.21.15.47:39864) 2024-02-19T15:25:10.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01a8a00640 1 -- 172.21.15.47:0/4057054531 learned_addr learned my addr 172.21.15.47:0/4057054531 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:10.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f019be00640 1 --2- 172.21.15.47:0/4057054531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01a40c7d40 0x7f01a40ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:10.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01a9400640 1 --2- 172.21.15.47:0/4057054531 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01a415ece0 0x7f01a40c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:10.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01a8a00640 1 -- 172.21.15.47:0/4057054531 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01a415ece0 msgr2=0x7f01a40c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:10.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01a8a00640 1 --2- 172.21.15.47:0/4057054531 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01a415ece0 0x7f01a40c9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:10.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01a8a00640 1 -- 172.21.15.47:0/4057054531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01a40c7d40 msgr2=0x7f01a40ab900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:10.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01a8a00640 1 --2- 172.21.15.47:0/4057054531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01a40c7d40 0x7f01a40ab900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:10.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01a8a00640 1 -- 172.21.15.47:0/4057054531 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a40c8aa0 con 0x7f0188004d10 2024-02-19T15:25:10.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f019be00640 1 --2- 172.21.15.47:0/4057054531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01a40c7d40 0x7f01a40ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:10.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01a8a00640 1 --2- 172.21.15.47:0/4057054531 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0188004d10 0x7f01a40a9f30 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f0194007290 tx=0x7f0194007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:10.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01a1600640 1 -- 172.21.15.47:0/4057054531 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019400f940 con 0x7f0188004d10 2024-02-19T15:25:10.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01a1600640 1 -- 172.21.15.47:0/4057054531 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019400fac0 con 0x7f0188004d10 2024-02-19T15:25:10.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01a1600640 1 -- 172.21.15.47:0/4057054531 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019400fe70 con 0x7f0188004d10 2024-02-19T15:25:10.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01ab400640 1 -- 172.21.15.47:0/4057054531 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a40c8d80 con 0x7f0188004d10 2024-02-19T15:25:10.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01ab400640 1 -- 172.21.15.47:0/4057054531 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a40c81c0 con 0x7f0188004d10 2024-02-19T15:25:10.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.469+0000 7f01ab400640 1 -- 172.21.15.47:0/4057054531 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a4075360 con 0x7f0188004d10 2024-02-19T15:25:10.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.473+0000 7f01a1600640 1 -- 172.21.15.47:0/4057054531 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f019401d460 con 0x7f0188004d10 2024-02-19T15:25:10.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.473+0000 7f01a1600640 1 --2- 172.21.15.47:0/4057054531 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0180066e40 0x7f0180069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:10.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.473+0000 7f01a1600640 1 -- 172.21.15.47:0/4057054531 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0194084b40 con 0x7f0188004d10 2024-02-19T15:25:10.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.473+0000 7f019be00640 1 --2- 172.21.15.47:0/4057054531 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0180066e40 0x7f0180069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:10.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.473+0000 7f019be00640 1 --2- 172.21.15.47:0/4057054531 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0180066e40 0x7f0180069300 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f01a415f710 tx=0x7f018c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:10.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.477+0000 7f01a1600640 1 -- 172.21.15.47:0/4057054531 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f01940519c0 con 0x7f0188004d10 2024-02-19T15:25:10.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:10.677+0000 7f01ab400640 1 -- 172.21.15.47:0/4057054531 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f01a40c9230 con 0x7f0188004d10 2024-02-19T15:25:11.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.569+0000 7f01a1600640 1 -- 172.21.15.47:0/4057054531 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v39) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0194055080 con 0x7f0188004d10 2024-02-19T15:25:11.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.569+0000 7f01ab400640 1 -- 172.21.15.47:0/4057054531 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0180066e40 msgr2=0x7f0180069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:11.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.569+0000 7f01ab400640 1 --2- 172.21.15.47:0/4057054531 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0180066e40 0x7f0180069300 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f01a415f710 tx=0x7f018c008040 comp rx=0 tx=0).stop 2024-02-19T15:25:11.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.569+0000 7f01ab400640 1 -- 172.21.15.47:0/4057054531 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0188004d10 msgr2=0x7f01a40a9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:11.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.569+0000 7f01ab400640 1 --2- 172.21.15.47:0/4057054531 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0188004d10 0x7f01a40a9f30 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f0194007290 tx=0x7f0194007750 comp rx=0 tx=0).stop 2024-02-19T15:25:11.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.569+0000 7f01ab400640 1 -- 172.21.15.47:0/4057054531 shutdown_connections 2024-02-19T15:25:11.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.569+0000 7f01ab400640 1 --2- 172.21.15.47:0/4057054531 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0180066e40 0x7f0180069300 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:11.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.569+0000 7f01ab400640 1 --2- 172.21.15.47:0/4057054531 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01a415ece0 0x7f01a40c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:11.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.569+0000 7f01ab400640 1 --2- 172.21.15.47:0/4057054531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01a40c7d40 0x7f01a40ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:11.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.569+0000 7f01ab400640 1 --2- 172.21.15.47:0/4057054531 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0188004d10 0x7f01a40a9f30 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:11.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.569+0000 7f01ab400640 1 -- 172.21.15.47:0/4057054531 >> 172.21.15.47:0/4057054531 conn(0x7f01a40cfc20 msgr2=0x7f01a40b19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:11.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.569+0000 7f01ab400640 1 -- 172.21.15.47:0/4057054531 shutdown_connections 2024-02-19T15:25:11.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.569+0000 7f01ab400640 1 -- 172.21.15.47:0/4057054531 wait complete. 2024-02-19T15:25:11.593 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T15:25:11.594 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T15:25:11.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.733+0000 7f4625000640 1 Processor -- start 2024-02-19T15:25:11.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.733+0000 7f4625000640 1 -- start start 2024-02-19T15:25:11.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.733+0000 7f4625000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f46200cb490 0x7f4620070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:11.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.737+0000 7f4625000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f462015fc90 con 0x7f4620072850 2024-02-19T15:25:11.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.737+0000 7f4625000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f462015ff70 con 0x7f462015bd90 2024-02-19T15:25:11.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.737+0000 7f4625000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4620160250 con 0x7f46200cb490 2024-02-19T15:25:11.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.737+0000 7f4617e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f46200cb490 0x7f4620070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:11.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.737+0000 7f461e800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f4620072850 0x7f462006cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38316/0 (socket says 172.21.15.47:38316) 2024-02-19T15:25:11.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.737+0000 7f461e800640 1 -- 172.21.15.47:0/2735250225 learned_addr learned my addr 172.21.15.47:0/2735250225 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:11.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.737+0000 7f4617e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f46200cb490 0x7f4620070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39876/0 (socket says 172.21.15.47:39876) 2024-02-19T15:25:11.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.737+0000 7f4617e00640 1 -- 172.21.15.47:0/2735250225 >> v1:172.21.15.116:6789/0 conn(0x7f462015bd90 legacy=0x7f4620072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:11.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.737+0000 7f4617e00640 1 -- 172.21.15.47:0/2735250225 >> v1:172.21.15.47:6789/0 conn(0x7f4620072850 legacy=0x7f462006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:11.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.737+0000 7f4617e00640 1 -- 172.21.15.47:0/2735250225 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4620160530 con 0x7f46200cb490 2024-02-19T15:25:11.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.737+0000 7f461d400640 1 -- 172.21.15.47:0/2735250225 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846646606 0 0) 0x7f462015fc90 con 0x7f4620072850 2024-02-19T15:25:11.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.737+0000 7f4617e00640 1 --2- 172.21.15.47:0/2735250225 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46200cb490 0x7f4620070960 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f46080097c0 tx=0x7f460801c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12edbde220e3de65 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:11.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f461d400640 1 -- 172.21.15.47:0/2735250225 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46080ab600 con 0x7f46200cb490 2024-02-19T15:25:11.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f461d400640 1 -- 172.21.15.47:0/2735250225 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46080a7050 con 0x7f46200cb490 2024-02-19T15:25:11.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f461d400640 1 -- 172.21.15.47:0/2735250225 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46080ab980 con 0x7f46200cb490 2024-02-19T15:25:11.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f4625000640 1 -- 172.21.15.47:0/2735250225 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46200cb490 msgr2=0x7f4620070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:11.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f4625000640 1 --2- 172.21.15.47:0/2735250225 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46200cb490 0x7f4620070960 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f46080097c0 tx=0x7f460801c380 comp rx=0 tx=0).stop 2024-02-19T15:25:11.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f4625000640 1 -- 172.21.15.47:0/2735250225 shutdown_connections 2024-02-19T15:25:11.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f4625000640 1 --2- 172.21.15.47:0/2735250225 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46200cb490 0x7f4620070960 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:11.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f4625000640 1 -- 172.21.15.47:0/2735250225 >> 172.21.15.47:0/2735250225 conn(0x7f46200cfc20 msgr2=0x7f46200d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:11.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f4625000640 1 -- 172.21.15.47:0/2735250225 shutdown_connections 2024-02-19T15:25:11.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f4625000640 1 -- 172.21.15.47:0/2735250225 wait complete. 2024-02-19T15:25:11.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f4625000640 1 Processor -- start 2024-02-19T15:25:11.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f4625000640 1 -- start start 2024-02-19T15:25:11.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f4625000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4620072850 0x7f4620166b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:11.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f4625000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46200cb490 0x7f4620152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:11.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f4625000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f462015bd90 0x7f46200ce920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:11.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f4625000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4620153040 con 0x7f4620072850 2024-02-19T15:25:11.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f4625000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4620165810 con 0x7f46200cb490 2024-02-19T15:25:11.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f4625000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4620165a90 con 0x7f462015bd90 2024-02-19T15:25:11.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f461de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4620072850 0x7f4620166b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:11.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f461de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4620072850 0x7f4620166b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39878/0 (socket says 172.21.15.47:39878) 2024-02-19T15:25:11.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f461de00640 1 -- 172.21.15.47:0/2075945854 learned_addr learned my addr 172.21.15.47:0/2075945854 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:11.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f4617e00640 1 --2- 172.21.15.47:0/2075945854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46200cb490 0x7f4620152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:11.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.741+0000 7f461e800640 1 --2- 172.21.15.47:0/2075945854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f462015bd90 0x7f46200ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:11.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.745+0000 7f461de00640 1 -- 172.21.15.47:0/2075945854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f462015bd90 msgr2=0x7f46200ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:11.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.745+0000 7f461de00640 1 --2- 172.21.15.47:0/2075945854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f462015bd90 0x7f46200ce920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:11.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.745+0000 7f461de00640 1 -- 172.21.15.47:0/2075945854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46200cb490 msgr2=0x7f4620152b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:11.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.745+0000 7f461de00640 1 --2- 172.21.15.47:0/2075945854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46200cb490 0x7f4620152b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:11.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.745+0000 7f461de00640 1 -- 172.21.15.47:0/2075945854 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4608009420 con 0x7f4620072850 2024-02-19T15:25:11.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.745+0000 7f461de00640 1 --2- 172.21.15.47:0/2075945854 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4620072850 0x7f4620166b30 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f461000c3c0 tx=0x7f461000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:11.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.745+0000 7f4616000640 1 -- 172.21.15.47:0/2075945854 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46100052d0 con 0x7f4620072850 2024-02-19T15:25:11.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.745+0000 7f4616000640 1 -- 172.21.15.47:0/2075945854 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4610005430 con 0x7f4620072850 2024-02-19T15:25:11.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.745+0000 7f4616000640 1 -- 172.21.15.47:0/2075945854 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4610003840 con 0x7f4620072850 2024-02-19T15:25:11.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.745+0000 7f4625000640 1 -- 172.21.15.47:0/2075945854 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4620165d70 con 0x7f4620072850 2024-02-19T15:25:11.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.745+0000 7f4625000640 1 -- 172.21.15.47:0/2075945854 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4620166330 con 0x7f4620072850 2024-02-19T15:25:11.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.745+0000 7f4625000640 1 -- 172.21.15.47:0/2075945854 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4620075360 con 0x7f4620072850 2024-02-19T15:25:11.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.745+0000 7f4616000640 1 -- 172.21.15.47:0/2075945854 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f46100039a0 con 0x7f4620072850 2024-02-19T15:25:11.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.745+0000 7f4616000640 1 --2- 172.21.15.47:0/2075945854 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f45fc066e40 0x7f45fc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:11.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.745+0000 7f4616000640 1 -- 172.21.15.47:0/2075945854 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4610083e40 con 0x7f4620072850 2024-02-19T15:25:11.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.745+0000 7f4617e00640 1 --2- 172.21.15.47:0/2075945854 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f45fc066e40 0x7f45fc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:11.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.749+0000 7f4617e00640 1 --2- 172.21.15.47:0/2075945854 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f45fc066e40 0x7f45fc069300 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f46080ad000 tx=0x7f4608008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:11.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.749+0000 7f4616000640 1 -- 172.21.15.47:0/2075945854 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f4610050d70 con 0x7f4620072850 2024-02-19T15:25:11.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:11.949+0000 7f4625000640 1 -- 172.21.15.47:0/2075945854 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f462006e800 con 0x7f4620072850 2024-02-19T15:25:12.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.569+0000 7f4616000640 1 -- 172.21.15.47:0/2075945854 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4610054430 con 0x7f4620072850 2024-02-19T15:25:12.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.573+0000 7f4625000640 1 -- 172.21.15.47:0/2075945854 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f45fc066e40 msgr2=0x7f45fc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:12.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.573+0000 7f4625000640 1 --2- 172.21.15.47:0/2075945854 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f45fc066e40 0x7f45fc069300 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f46080ad000 tx=0x7f4608008040 comp rx=0 tx=0).stop 2024-02-19T15:25:12.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.573+0000 7f4625000640 1 -- 172.21.15.47:0/2075945854 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4620072850 msgr2=0x7f4620166b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:12.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.573+0000 7f4625000640 1 --2- 172.21.15.47:0/2075945854 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4620072850 0x7f4620166b30 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f461000c3c0 tx=0x7f461000c880 comp rx=0 tx=0).stop 2024-02-19T15:25:12.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.573+0000 7f4625000640 1 -- 172.21.15.47:0/2075945854 shutdown_connections 2024-02-19T15:25:12.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.573+0000 7f4625000640 1 --2- 172.21.15.47:0/2075945854 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f45fc066e40 0x7f45fc069300 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:12.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.573+0000 7f4625000640 1 --2- 172.21.15.47:0/2075945854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f462015bd90 0x7f46200ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:12.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.573+0000 7f4625000640 1 --2- 172.21.15.47:0/2075945854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46200cb490 0x7f4620152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:12.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.573+0000 7f4625000640 1 --2- 172.21.15.47:0/2075945854 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4620072850 0x7f4620166b30 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:12.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.573+0000 7f4625000640 1 -- 172.21.15.47:0/2075945854 >> 172.21.15.47:0/2075945854 conn(0x7f46200cfc20 msgr2=0x7f46200ca230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:12.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.573+0000 7f4625000640 1 -- 172.21.15.47:0/2075945854 shutdown_connections 2024-02-19T15:25:12.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.573+0000 7f4625000640 1 -- 172.21.15.47:0/2075945854 wait complete. 2024-02-19T15:25:12.597 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T15:25:12.597 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:25:12.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.733+0000 7f4497800640 1 Processor -- start 2024-02-19T15:25:12.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.733+0000 7f4497800640 1 -- start start 2024-02-19T15:25:12.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.733+0000 7f4497800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f449015d660 0x7f449006d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:12.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.733+0000 7f4497800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f449015aef0 0x7f44901646f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:12.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.733+0000 7f4497800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449006ddf0 con 0x7f449006ef50 2024-02-19T15:25:12.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.733+0000 7f4497800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f449015c350 con 0x7f449015d660 2024-02-19T15:25:12.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.733+0000 7f4497800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f449006dbe0 con 0x7f449015aef0 2024-02-19T15:25:12.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.733+0000 7f4487e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f449015d660 0x7f449006d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:12.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.733+0000 7f4487e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f449015d660 0x7f449006d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39892/0 (socket says 172.21.15.47:39892) 2024-02-19T15:25:12.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.733+0000 7f4487e00640 1 -- 172.21.15.47:0/2815203105 learned_addr learned my addr 172.21.15.47:0/2815203105 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:12.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.733+0000 7f4495800640 1 --2- 172.21.15.47:0/2815203105 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f449015aef0 0x7f44901646f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:12.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.737+0000 7f4487e00640 1 -- 172.21.15.47:0/2815203105 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f449015aef0 msgr2=0x7f44901646f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:12.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.737+0000 7f4487e00640 1 --2- 172.21.15.47:0/2815203105 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f449015aef0 0x7f44901646f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:12.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.737+0000 7f4487e00640 1 -- 172.21.15.47:0/2815203105 >> v1:172.21.15.116:6789/0 conn(0x7f449006ef50 legacy=0x7f449006f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:12.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.737+0000 7f4487e00640 1 -- 172.21.15.47:0/2815203105 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449006e0d0 con 0x7f449015d660 2024-02-19T15:25:12.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.741+0000 7f4487e00640 1 --2- 172.21.15.47:0/2815203105 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f449015d660 0x7f449006d6a0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4478009870 tx=0x7f447801c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2ff13cbe4e033d4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:12.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.741+0000 7f448fe00640 1 -- 172.21.15.47:0/2815203105 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44780a95f0 con 0x7f449015d660 2024-02-19T15:25:12.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.741+0000 7f448fe00640 1 -- 172.21.15.47:0/2815203105 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44780a9750 con 0x7f449015d660 2024-02-19T15:25:12.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.741+0000 7f448fe00640 1 -- 172.21.15.47:0/2815203105 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44780a9ae0 con 0x7f449015d660 2024-02-19T15:25:12.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.741+0000 7f4497800640 1 -- 172.21.15.47:0/2815203105 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f449015d660 msgr2=0x7f449006d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:12.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.741+0000 7f4497800640 1 --2- 172.21.15.47:0/2815203105 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f449015d660 0x7f449006d6a0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4478009870 tx=0x7f447801c530 comp rx=0 tx=0).stop 2024-02-19T15:25:12.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.741+0000 7f4497800640 1 -- 172.21.15.47:0/2815203105 shutdown_connections 2024-02-19T15:25:12.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.741+0000 7f4497800640 1 --2- 172.21.15.47:0/2815203105 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f449015aef0 0x7f44901646f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:12.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.741+0000 7f4497800640 1 --2- 172.21.15.47:0/2815203105 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f449015d660 0x7f449006d6a0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:12.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.741+0000 7f4497800640 1 -- 172.21.15.47:0/2815203105 >> 172.21.15.47:0/2815203105 conn(0x7f44900d0ad0 msgr2=0x7f44900d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:12.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.741+0000 7f4497800640 1 -- 172.21.15.47:0/2815203105 shutdown_connections 2024-02-19T15:25:12.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.741+0000 7f4497800640 1 -- 172.21.15.47:0/2815203105 wait complete. 2024-02-19T15:25:12.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.745+0000 7f4497800640 1 Processor -- start 2024-02-19T15:25:12.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.745+0000 7f4497800640 1 -- start start 2024-02-19T15:25:12.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.745+0000 7f4497800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f449006ef50 0x7f44900b3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:12.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.745+0000 7f4497800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f449015aef0 0x7f449014a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:12.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.745+0000 7f4497800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f449015d660 0x7f44900cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:12.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.745+0000 7f4497800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44900b4120 con 0x7f449015aef0 2024-02-19T15:25:12.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.745+0000 7f4497800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44900b3e40 con 0x7f449006ef50 2024-02-19T15:25:12.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.745+0000 7f4497800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4490135500 con 0x7f449015d660 2024-02-19T15:25:12.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.745+0000 7f4487e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f449015aef0 0x7f449014a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:12.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.745+0000 7f4487e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f449015aef0 0x7f449014a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39908/0 (socket says 172.21.15.47:39908) 2024-02-19T15:25:12.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.745+0000 7f4487e00640 1 -- 172.21.15.47:0/2524499470 learned_addr learned my addr 172.21.15.47:0/2524499470 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:12.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.745+0000 7f4495800640 1 --2- 172.21.15.47:0/2524499470 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f449015d660 0x7f44900cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:12.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.745+0000 7f4494e00640 1 --2- 172.21.15.47:0/2524499470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f449006ef50 0x7f44900b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:12.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.745+0000 7f4487e00640 1 -- 172.21.15.47:0/2524499470 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f449015d660 msgr2=0x7f44900cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:12.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.745+0000 7f4487e00640 1 --2- 172.21.15.47:0/2524499470 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f449015d660 0x7f44900cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:12.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.745+0000 7f4487e00640 1 -- 172.21.15.47:0/2524499470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f449006ef50 msgr2=0x7f44900b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:12.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.745+0000 7f4487e00640 1 --2- 172.21.15.47:0/2524499470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f449006ef50 0x7f44900b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:12.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.745+0000 7f4487e00640 1 -- 172.21.15.47:0/2524499470 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44780094d0 con 0x7f449015aef0 2024-02-19T15:25:12.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.745+0000 7f4487e00640 1 --2- 172.21.15.47:0/2524499470 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f449015aef0 0x7f449014a0c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f44780ab040 tx=0x7f44780192d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:12.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.749+0000 7f448d600640 1 -- 172.21.15.47:0/2524499470 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44780a95f0 con 0x7f449015aef0 2024-02-19T15:25:12.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.749+0000 7f448d600640 1 -- 172.21.15.47:0/2524499470 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44780b1600 con 0x7f449015aef0 2024-02-19T15:25:12.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.749+0000 7f4497800640 1 -- 172.21.15.47:0/2524499470 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44901357e0 con 0x7f449015aef0 2024-02-19T15:25:12.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.749+0000 7f448d600640 1 -- 172.21.15.47:0/2524499470 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44780b19d0 con 0x7f449015aef0 2024-02-19T15:25:12.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.749+0000 7f4497800640 1 -- 172.21.15.47:0/2524499470 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44900b3ac0 con 0x7f449015aef0 2024-02-19T15:25:12.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.749+0000 7f4497800640 1 -- 172.21.15.47:0/2524499470 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4490075360 con 0x7f449015aef0 2024-02-19T15:25:12.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.749+0000 7f448d600640 1 -- 172.21.15.47:0/2524499470 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f44780b1b30 con 0x7f449015aef0 2024-02-19T15:25:12.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.749+0000 7f448d600640 1 --2- 172.21.15.47:0/2524499470 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f446c066e40 0x7f446c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:12.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.753+0000 7f448d600640 1 -- 172.21.15.47:0/2524499470 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f447811ee20 con 0x7f449015aef0 2024-02-19T15:25:12.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.753+0000 7f4494e00640 1 --2- 172.21.15.47:0/2524499470 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f446c066e40 0x7f446c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:12.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.753+0000 7f4494e00640 1 --2- 172.21.15.47:0/2524499470 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f446c066e40 0x7f446c069300 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f44901f6e60 tx=0x7f4480009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:12.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.753+0000 7f448d600640 1 -- 172.21.15.47:0/2524499470 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f44780ebd50 con 0x7f449015aef0 2024-02-19T15:25:12.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.961+0000 7f4497800640 1 -- 172.21.15.47:0/2524499470 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f449015c110 con 0x7f449015aef0 2024-02-19T15:25:12.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.961+0000 7f448d600640 1 -- 172.21.15.47:0/2524499470 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4946 (secure 0 0 0) 0x7f44780ef410 con 0x7f449015aef0 2024-02-19T15:25:12.964 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:12.964 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5684,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3206627105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3206627105},{"type":"v1","addr":"172.21.15.116:6836","nonce":3206627105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5687,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3532185042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3532185042},{"type":"v1","addr":"172.21.15.116:6837","nonce":3532185042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/1482478568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1482478568},{"type":"v1","addr":"172.21.15.47:6834","nonce":1482478568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:25:08.947740+0000","modified":"2024-02-19T15:25:11.954116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/2490197058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2490197058},{"type":"v1","addr":"172.21.15.47:6835","nonce":2490197058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-02-19T15:25:12.964 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 40 2024-02-19T15:25:12.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.961+0000 7f4497800640 1 -- 172.21.15.47:0/2524499470 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f446c066e40 msgr2=0x7f446c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:12.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.961+0000 7f4497800640 1 --2- 172.21.15.47:0/2524499470 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f446c066e40 0x7f446c069300 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f44901f6e60 tx=0x7f4480009040 comp rx=0 tx=0).stop 2024-02-19T15:25:12.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.961+0000 7f4497800640 1 -- 172.21.15.47:0/2524499470 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f449015aef0 msgr2=0x7f449014a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:12.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.961+0000 7f4497800640 1 --2- 172.21.15.47:0/2524499470 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f449015aef0 0x7f449014a0c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f44780ab040 tx=0x7f44780192d0 comp rx=0 tx=0).stop 2024-02-19T15:25:12.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.961+0000 7f4497800640 1 -- 172.21.15.47:0/2524499470 shutdown_connections 2024-02-19T15:25:12.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.961+0000 7f4497800640 1 --2- 172.21.15.47:0/2524499470 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f446c066e40 0x7f446c069300 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:12.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.961+0000 7f4497800640 1 --2- 172.21.15.47:0/2524499470 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f449015d660 0x7f44900cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:12.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.961+0000 7f4497800640 1 --2- 172.21.15.47:0/2524499470 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f449015aef0 0x7f449014a0c0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:12.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.961+0000 7f4497800640 1 --2- 172.21.15.47:0/2524499470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f449006ef50 0x7f44900b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:12.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.961+0000 7f4497800640 1 -- 172.21.15.47:0/2524499470 >> 172.21.15.47:0/2524499470 conn(0x7f44900d0ad0 msgr2=0x7f44900b14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:12.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.961+0000 7f4497800640 1 -- 172.21.15.47:0/2524499470 shutdown_connections 2024-02-19T15:25:12.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:12.961+0000 7f4497800640 1 -- 172.21.15.47:0/2524499470 wait complete. 2024-02-19T15:25:12.984 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:25:13.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.121+0000 7f8259c00640 1 Processor -- start 2024-02-19T15:25:13.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.121+0000 7f8259c00640 1 -- start start 2024-02-19T15:25:13.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.121+0000 7f8259c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f825415bcd0 0x7f82540cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:13.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.121+0000 7f8259c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f82540cedb0 0x7f82540cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:13.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.125+0000 7f8259c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82540cc700 con 0x7f82540d2710 2024-02-19T15:25:13.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.125+0000 7f8259c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82540cf5e0 con 0x7f82540cedb0 2024-02-19T15:25:13.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.125+0000 7f8259c00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82540cf300 con 0x7f825415bcd0 2024-02-19T15:25:13.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.125+0000 7f824be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f82540cedb0 0x7f82540cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:13.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.125+0000 7f824be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f82540cedb0 0x7f82540cfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39920/0 (socket says 172.21.15.47:39920) 2024-02-19T15:25:13.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.125+0000 7f824be00640 1 -- 172.21.15.47:0/223900889 learned_addr learned my addr 172.21.15.47:0/223900889 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:13.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.125+0000 7f8252a00640 1 --2- 172.21.15.47:0/223900889 >> v2:172.21.15.116:3300/0 conn(0x7f825415bcd0 0x7f82540cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:13.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.125+0000 7f824be00640 1 -- 172.21.15.47:0/223900889 >> v2:172.21.15.116:3300/0 conn(0x7f825415bcd0 msgr2=0x7f82540cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:13.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.125+0000 7f824be00640 1 --2- 172.21.15.47:0/223900889 >> v2:172.21.15.116:3300/0 conn(0x7f825415bcd0 0x7f82540cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.125+0000 7f824be00640 1 -- 172.21.15.47:0/223900889 >> v1:172.21.15.116:6789/0 conn(0x7f82540d2710 legacy=0x7f825415d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:13.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.125+0000 7f824be00640 1 -- 172.21.15.47:0/223900889 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82540cc9a0 con 0x7f82540cedb0 2024-02-19T15:25:13.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.125+0000 7f824be00640 1 --2- 172.21.15.47:0/223900889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82540cedb0 0x7f82540cfc50 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f8254071a60 tx=0x7f823c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afc0efbd887606bf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:13.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.125+0000 7f8252000640 1 -- 172.21.15.47:0/223900889 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c0ab600 con 0x7f82540cedb0 2024-02-19T15:25:13.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8252000640 1 -- 172.21.15.47:0/223900889 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823c0ab760 con 0x7f82540cedb0 2024-02-19T15:25:13.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8252000640 1 -- 172.21.15.47:0/223900889 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c0abaf0 con 0x7f82540cedb0 2024-02-19T15:25:13.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8259c00640 1 -- 172.21.15.47:0/223900889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82540cedb0 msgr2=0x7f82540cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:13.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8259c00640 1 --2- 172.21.15.47:0/223900889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82540cedb0 0x7f82540cfc50 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f8254071a60 tx=0x7f823c01c4f0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8259c00640 1 -- 172.21.15.47:0/223900889 shutdown_connections 2024-02-19T15:25:13.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8259c00640 1 --2- 172.21.15.47:0/223900889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82540cedb0 0x7f82540cfc50 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8259c00640 1 --2- 172.21.15.47:0/223900889 >> v2:172.21.15.116:3300/0 conn(0x7f825415bcd0 0x7f82540cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8259c00640 1 -- 172.21.15.47:0/223900889 >> 172.21.15.47:0/223900889 conn(0x7f82540d1bf0 msgr2=0x7f82540d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:13.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8259c00640 1 -- 172.21.15.47:0/223900889 shutdown_connections 2024-02-19T15:25:13.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8259c00640 1 -- 172.21.15.47:0/223900889 wait complete. 2024-02-19T15:25:13.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8259c00640 1 Processor -- start 2024-02-19T15:25:13.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8259c00640 1 -- start start 2024-02-19T15:25:13.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8259c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82540cedb0 0x7f82540ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:13.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8259c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82540d2710 0x7f82540b6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:13.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8259c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f825415bcd0 0x7f82541f6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:13.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8259c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82540c9770 con 0x7f825415bcd0 2024-02-19T15:25:13.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8259c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82540c9a50 con 0x7f82540cedb0 2024-02-19T15:25:13.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8259c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82540c9d30 con 0x7f82540d2710 2024-02-19T15:25:13.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8253400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f825415bcd0 0x7f82541f6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:13.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8252a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82540cedb0 0x7f82540ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:13.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8253400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f825415bcd0 0x7f82541f6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39922/0 (socket says 172.21.15.47:39922) 2024-02-19T15:25:13.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8252a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82540cedb0 0x7f82540ca9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:58892/0 (socket says 172.21.15.47:58892) 2024-02-19T15:25:13.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f8253400640 1 -- 172.21.15.47:0/1801668855 learned_addr learned my addr 172.21.15.47:0/1801668855 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:13.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.129+0000 7f824be00640 1 --2- 172.21.15.47:0/1801668855 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82540d2710 0x7f82540b6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:13.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.133+0000 7f8253400640 1 -- 172.21.15.47:0/1801668855 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82540d2710 msgr2=0x7f82540b6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:13.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.133+0000 7f8253400640 1 --2- 172.21.15.47:0/1801668855 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82540d2710 0x7f82540b6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.133+0000 7f8253400640 1 -- 172.21.15.47:0/1801668855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82540cedb0 msgr2=0x7f82540ca9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:13.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.133+0000 7f8253400640 1 --2- 172.21.15.47:0/1801668855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82540cedb0 0x7f82540ca9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.133+0000 7f8253400640 1 -- 172.21.15.47:0/1801668855 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823c0094d0 con 0x7f825415bcd0 2024-02-19T15:25:13.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.133+0000 7f8253400640 1 --2- 172.21.15.47:0/1801668855 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f825415bcd0 0x7f82541f6ad0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f824c00c770 tx=0x7f824c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:13.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.133+0000 7f824aa00640 1 -- 172.21.15.47:0/1801668855 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824c003b80 con 0x7f825415bcd0 2024-02-19T15:25:13.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.133+0000 7f824aa00640 1 -- 172.21.15.47:0/1801668855 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f824c003ce0 con 0x7f825415bcd0 2024-02-19T15:25:13.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.133+0000 7f8259c00640 1 -- 172.21.15.47:0/1801668855 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82540cdb40 con 0x7f825415bcd0 2024-02-19T15:25:13.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.133+0000 7f824aa00640 1 -- 172.21.15.47:0/1801668855 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824c00aa90 con 0x7f825415bcd0 2024-02-19T15:25:13.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.133+0000 7f8259c00640 1 -- 172.21.15.47:0/1801668855 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82540ca2f0 con 0x7f825415bcd0 2024-02-19T15:25:13.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.133+0000 7f8259c00640 1 -- 172.21.15.47:0/1801668855 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82540ca4e0 con 0x7f825415bcd0 2024-02-19T15:25:13.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.133+0000 7f824aa00640 1 -- 172.21.15.47:0/1801668855 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f824c00abf0 con 0x7f825415bcd0 2024-02-19T15:25:13.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.137+0000 7f824aa00640 1 --2- 172.21.15.47:0/1801668855 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8230066e40 0x7f8230069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:13.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.137+0000 7f824aa00640 1 -- 172.21.15.47:0/1801668855 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f824c084770 con 0x7f825415bcd0 2024-02-19T15:25:13.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.137+0000 7f8252a00640 1 --2- 172.21.15.47:0/1801668855 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8230066e40 0x7f8230069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:13.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.137+0000 7f8252a00640 1 --2- 172.21.15.47:0/1801668855 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8230066e40 0x7f8230069300 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8244009b60 tx=0x7f8244009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:13.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.141+0000 7f824aa00640 1 -- 172.21.15.47:0/1801668855 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f824c051600 con 0x7f825415bcd0 2024-02-19T15:25:13.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.305+0000 7f8259c00640 1 -- 172.21.15.47:0/1801668855 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8254075360 con 0x7f825415bcd0 2024-02-19T15:25:13.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.305+0000 7f824aa00640 1 -- 172.21.15.47:0/1801668855 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16680 (secure 0 0 0) 0x7f824c054cc0 con 0x7f825415bcd0 2024-02-19T15:25:13.309 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:13.310 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":69,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:25:10.552537+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:25:06.759758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-02-19T15:25:07.949435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:25:13.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.309+0000 7f8259c00640 1 -- 172.21.15.47:0/1801668855 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8230066e40 msgr2=0x7f8230069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:13.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.309+0000 7f8259c00640 1 --2- 172.21.15.47:0/1801668855 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8230066e40 0x7f8230069300 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8244009b60 tx=0x7f8244009190 comp rx=0 tx=0).stop 2024-02-19T15:25:13.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.309+0000 7f8259c00640 1 -- 172.21.15.47:0/1801668855 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f825415bcd0 msgr2=0x7f82541f6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:13.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.309+0000 7f8259c00640 1 --2- 172.21.15.47:0/1801668855 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f825415bcd0 0x7f82541f6ad0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f824c00c770 tx=0x7f824c00cc30 comp rx=0 tx=0).stop 2024-02-19T15:25:13.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.309+0000 7f8259c00640 1 -- 172.21.15.47:0/1801668855 shutdown_connections 2024-02-19T15:25:13.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.309+0000 7f8259c00640 1 --2- 172.21.15.47:0/1801668855 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8230066e40 0x7f8230069300 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.309+0000 7f8259c00640 1 --2- 172.21.15.47:0/1801668855 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f825415bcd0 0x7f82541f6ad0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.309+0000 7f8259c00640 1 --2- 172.21.15.47:0/1801668855 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82540d2710 0x7f82540b6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.309+0000 7f8259c00640 1 --2- 172.21.15.47:0/1801668855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82540cedb0 0x7f82540ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.309+0000 7f8259c00640 1 -- 172.21.15.47:0/1801668855 >> 172.21.15.47:0/1801668855 conn(0x7f82540d1bf0 msgr2=0x7f82540d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:13.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.309+0000 7f8259c00640 1 -- 172.21.15.47:0/1801668855 shutdown_connections 2024-02-19T15:25:13.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.309+0000 7f8259c00640 1 -- 172.21.15.47:0/1801668855 wait complete. 2024-02-19T15:25:13.332 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T15:25:13.332 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:25:13.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.469+0000 7fc673a00640 1 Processor -- start 2024-02-19T15:25:13.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.469+0000 7fc673a00640 1 -- start start 2024-02-19T15:25:13.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.469+0000 7fc673a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc66c1456a0 0x7fc66c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:13.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.469+0000 7fc673a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc66c0cf4a0 0x7fc66c06aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:13.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.469+0000 7fc673a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc66c06b3f0 0x7fc66c15f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:13.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.469+0000 7fc673a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc66c0d03c0 con 0x7fc66c1456a0 2024-02-19T15:25:13.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.469+0000 7fc673a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc66c06b880 con 0x7fc66c0cf4a0 2024-02-19T15:25:13.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.469+0000 7fc673a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c0cf900 con 0x7fc66c06b3f0 2024-02-19T15:25:13.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.469+0000 7fc671000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc66c1456a0 0x7fc66c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.469+0000 7fc671000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc66c1456a0 0x7fc66c0cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55004/0 (socket says 172.21.15.47:55004) 2024-02-19T15:25:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.469+0000 7fc671000640 1 -- 172.21.15.47:0/3100882761 learned_addr learned my addr 172.21.15.47:0/3100882761 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.469+0000 7fc671a00640 1 --2- 172.21.15.47:0/3100882761 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc66c06b3f0 0x7fc66c15f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.469+0000 7fc66be00640 1 --2- 172.21.15.47:0/3100882761 >> v2:172.21.15.116:3300/0 conn(0x7fc66c0cf4a0 0x7fc66c06aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.469+0000 7fc671000640 1 -- 172.21.15.47:0/3100882761 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc66c06b3f0 msgr2=0x7fc66c15f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.469+0000 7fc671000640 1 --2- 172.21.15.47:0/3100882761 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc66c06b3f0 0x7fc66c15f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.469+0000 7fc671000640 1 -- 172.21.15.47:0/3100882761 >> v2:172.21.15.116:3300/0 conn(0x7fc66c0cf4a0 msgr2=0x7fc66c06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.469+0000 7fc671000640 1 --2- 172.21.15.47:0/3100882761 >> v2:172.21.15.116:3300/0 conn(0x7fc66c0cf4a0 0x7fc66c06aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.469+0000 7fc671000640 1 -- 172.21.15.47:0/3100882761 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66c145a80 con 0x7fc66c1456a0 2024-02-19T15:25:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.469+0000 7fc66be00640 1 --2- 172.21.15.47:0/3100882761 >> v2:172.21.15.116:3300/0 conn(0x7fc66c0cf4a0 0x7fc66c06aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:13.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.473+0000 7fc671000640 1 --2- 172.21.15.47:0/3100882761 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc66c1456a0 0x7fc66c0cef60 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc66c0748f0 tx=0x7fc66001c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=606f14bab7b91892 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:13.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.473+0000 7fc66b400640 1 -- 172.21.15.47:0/3100882761 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6600ab600 con 0x7fc66c1456a0 2024-02-19T15:25:13.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.473+0000 7fc66b400640 1 -- 172.21.15.47:0/3100882761 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6600a7050 con 0x7fc66c1456a0 2024-02-19T15:25:13.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.473+0000 7fc66b400640 1 -- 172.21.15.47:0/3100882761 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6600ab980 con 0x7fc66c1456a0 2024-02-19T15:25:13.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.473+0000 7fc673a00640 1 -- 172.21.15.47:0/3100882761 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc66c1456a0 msgr2=0x7fc66c0cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:13.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.473+0000 7fc673a00640 1 --2- 172.21.15.47:0/3100882761 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc66c1456a0 0x7fc66c0cef60 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc66c0748f0 tx=0x7fc66001c260 comp rx=0 tx=0).stop 2024-02-19T15:25:13.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.473+0000 7fc673a00640 1 -- 172.21.15.47:0/3100882761 shutdown_connections 2024-02-19T15:25:13.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.473+0000 7fc673a00640 1 --2- 172.21.15.47:0/3100882761 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc66c06b3f0 0x7fc66c15f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.473+0000 7fc673a00640 1 --2- 172.21.15.47:0/3100882761 >> v2:172.21.15.116:3300/0 conn(0x7fc66c0cf4a0 0x7fc66c06aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.473+0000 7fc673a00640 1 --2- 172.21.15.47:0/3100882761 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc66c1456a0 0x7fc66c0cef60 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.473+0000 7fc673a00640 1 -- 172.21.15.47:0/3100882761 >> 172.21.15.47:0/3100882761 conn(0x7fc66c0d0ad0 msgr2=0x7fc66c0d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:13.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.473+0000 7fc673a00640 1 -- 172.21.15.47:0/3100882761 shutdown_connections 2024-02-19T15:25:13.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.473+0000 7fc673a00640 1 -- 172.21.15.47:0/3100882761 wait complete. 2024-02-19T15:25:13.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc673a00640 1 Processor -- start 2024-02-19T15:25:13.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc673a00640 1 -- start start 2024-02-19T15:25:13.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc673a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc66c06b3f0 0x7fc66c0b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:13.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc673a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc66c0cf4a0 0x7fc66c15c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:13.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc673a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc66c1456a0 0x7fc66c1618e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:13.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc673a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c165c70 con 0x7fc66c1456a0 2024-02-19T15:25:13.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc673a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c0b1cf0 con 0x7fc66c0cf4a0 2024-02-19T15:25:13.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc673a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c1657e0 con 0x7fc66c06b3f0 2024-02-19T15:25:13.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc671a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc66c1456a0 0x7fc66c1618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:13.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc671a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc66c1456a0 0x7fc66c1618e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55016/0 (socket says 172.21.15.47:55016) 2024-02-19T15:25:13.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc671a00640 1 -- 172.21.15.47:0/3060576439 learned_addr learned my addr 172.21.15.47:0/3060576439 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:13.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc671000640 1 --2- 172.21.15.47:0/3060576439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc66c06b3f0 0x7fc66c0b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:13.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc66be00640 1 --2- 172.21.15.47:0/3060576439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc66c0cf4a0 0x7fc66c15c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:13.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc671a00640 1 -- 172.21.15.47:0/3060576439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc66c06b3f0 msgr2=0x7fc66c0b17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:13.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc671a00640 1 --2- 172.21.15.47:0/3060576439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc66c06b3f0 0x7fc66c0b17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc671a00640 1 -- 172.21.15.47:0/3060576439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc66c0cf4a0 msgr2=0x7fc66c15c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:13.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc671a00640 1 --2- 172.21.15.47:0/3060576439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc66c0cf4a0 0x7fc66c15c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc671a00640 1 -- 172.21.15.47:0/3060576439 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6600094d0 con 0x7fc66c1456a0 2024-02-19T15:25:13.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc671a00640 1 --2- 172.21.15.47:0/3060576439 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc66c1456a0 0x7fc66c1618e0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fc664007e40 tx=0x7fc66400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:13.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc668c00640 1 -- 172.21.15.47:0/3060576439 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc664014620 con 0x7fc66c1456a0 2024-02-19T15:25:13.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc668c00640 1 -- 172.21.15.47:0/3060576439 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66400ee90 con 0x7fc66c1456a0 2024-02-19T15:25:13.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc668c00640 1 -- 172.21.15.47:0/3060576439 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6640149e0 con 0x7fc66c1456a0 2024-02-19T15:25:13.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.477+0000 7fc673a00640 1 -- 172.21.15.47:0/3060576439 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc66c0b5a40 con 0x7fc66c1456a0 2024-02-19T15:25:13.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.481+0000 7fc673a00640 1 -- 172.21.15.47:0/3060576439 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc66c164eb0 con 0x7fc66c1456a0 2024-02-19T15:25:13.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.481+0000 7fc673a00640 1 -- 172.21.15.47:0/3060576439 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc66c075360 con 0x7fc66c1456a0 2024-02-19T15:25:13.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.481+0000 7fc668c00640 1 -- 172.21.15.47:0/3060576439 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc664014b40 con 0x7fc66c1456a0 2024-02-19T15:25:13.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.481+0000 7fc668c00640 1 --2- 172.21.15.47:0/3060576439 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc64c066e40 0x7fc64c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:13.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.481+0000 7fc668c00640 1 -- 172.21.15.47:0/3060576439 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6640852e0 con 0x7fc66c1456a0 2024-02-19T15:25:13.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.481+0000 7fc671000640 1 --2- 172.21.15.47:0/3060576439 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc64c066e40 0x7fc64c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:13.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.485+0000 7fc671000640 1 --2- 172.21.15.47:0/3060576439 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc64c066e40 0x7fc64c069300 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc6600ad000 tx=0x7fc6600a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:13.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.485+0000 7fc668c00640 1 -- 172.21.15.47:0/3060576439 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc664052170 con 0x7fc66c1456a0 2024-02-19T15:25:13.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.649+0000 7fc673a00640 1 -- 172.21.15.47:0/3060576439 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc66c165920 con 0x7fc64c066e40 2024-02-19T15:25:13.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.657+0000 7fc668c00640 1 -- 172.21.15.47:0/3060576439 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289238 (secure 0 0 0) 0x7fc66c165920 con 0x7fc64c066e40 2024-02-19T15:25:13.663 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:13.665 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:25:13.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.661+0000 7fc673a00640 1 -- 172.21.15.47:0/3060576439 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc64c066e40 msgr2=0x7fc64c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:13.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.661+0000 7fc673a00640 1 --2- 172.21.15.47:0/3060576439 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc64c066e40 0x7fc64c069300 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc6600ad000 tx=0x7fc6600a6000 comp rx=0 tx=0).stop 2024-02-19T15:25:13.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.661+0000 7fc673a00640 1 -- 172.21.15.47:0/3060576439 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc66c1456a0 msgr2=0x7fc66c1618e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:13.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.661+0000 7fc673a00640 1 --2- 172.21.15.47:0/3060576439 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc66c1456a0 0x7fc66c1618e0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fc664007e40 tx=0x7fc66400e4b0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.661+0000 7fc673a00640 1 -- 172.21.15.47:0/3060576439 shutdown_connections 2024-02-19T15:25:13.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.661+0000 7fc673a00640 1 --2- 172.21.15.47:0/3060576439 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc64c066e40 0x7fc64c069300 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.661+0000 7fc673a00640 1 --2- 172.21.15.47:0/3060576439 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc66c1456a0 0x7fc66c1618e0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.661+0000 7fc673a00640 1 --2- 172.21.15.47:0/3060576439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc66c0cf4a0 0x7fc66c15c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.661+0000 7fc673a00640 1 --2- 172.21.15.47:0/3060576439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc66c06b3f0 0x7fc66c0b17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.661+0000 7fc673a00640 1 -- 172.21.15.47:0/3060576439 >> 172.21.15.47:0/3060576439 conn(0x7fc66c0d0ad0 msgr2=0x7fc66c0b0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:13.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.661+0000 7fc673a00640 1 -- 172.21.15.47:0/3060576439 shutdown_connections 2024-02-19T15:25:13.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.661+0000 7fc673a00640 1 -- 172.21.15.47:0/3060576439 wait complete. 2024-02-19T15:25:13.683 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-02-19T15:25:13.191005+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":91,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":261308,"kb_used_data":7680,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754713412,"statfs":{"total":773094113280,"available":772826533888,"internally_reserved":0,"allocated":7864320,"data_stored":3947348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":259575346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.648219"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.588426+0000","last_change":"2024-02-19T15:25:09.582871+0000","last_active":"2024-02-19T15:25:09.588426+0000","last_peered":"2024-02-19T15:25:09.588426+0000","last_clean":"2024-02-19T15:25:09.588426+0000","last_became_active":"2024-02-19T15:25:09.581537+0000","last_became_peered":"2024-02-19T15:25:09.581537+0000","last_unstale":"2024-02-19T15:25:09.588426+0000","last_undegraded":"2024-02-19T15:25:09.588426+0000","last_fullsized":"2024-02-19T15:25:09.588426+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:37.897102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558530+0000","last_change":"2024-02-19T15:25:09.574801+0000","last_active":"2024-02-19T15:25:10.558530+0000","last_peered":"2024-02-19T15:25:10.558530+0000","last_clean":"2024-02-19T15:25:10.558530+0000","last_became_active":"2024-02-19T15:25:09.573339+0000","last_became_peered":"2024-02-19T15:25:09.573339+0000","last_unstale":"2024-02-19T15:25:10.558530+0000","last_undegraded":"2024-02-19T15:25:10.558530+0000","last_fullsized":"2024-02-19T15:25:10.558530+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:08:09.019535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.570326+0000","last_change":"2024-02-19T15:25:09.568275+0000","last_active":"2024-02-19T15:25:09.570326+0000","last_peered":"2024-02-19T15:25:09.570326+0000","last_clean":"2024-02-19T15:25:09.570326+0000","last_became_active":"2024-02-19T15:25:09.567978+0000","last_became_peered":"2024-02-19T15:25:09.567978+0000","last_unstale":"2024-02-19T15:25:09.570326+0000","last_undegraded":"2024-02-19T15:25:09.570326+0000","last_fullsized":"2024-02-19T15:25:09.570326+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:07:42.213615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558458+0000","last_change":"2024-02-19T15:25:09.573255+0000","last_active":"2024-02-19T15:25:10.558458+0000","last_peered":"2024-02-19T15:25:10.558458+0000","last_clean":"2024-02-19T15:25:10.558458+0000","last_became_active":"2024-02-19T15:25:09.570700+0000","last_became_peered":"2024-02-19T15:25:09.570700+0000","last_unstale":"2024-02-19T15:25:10.558458+0000","last_undegraded":"2024-02-19T15:25:10.558458+0000","last_fullsized":"2024-02-19T15:25:10.558458+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:53:35.452518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558098+0000","last_change":"2024-02-19T15:25:09.578467+0000","last_active":"2024-02-19T15:25:10.558098+0000","last_peered":"2024-02-19T15:25:10.558098+0000","last_clean":"2024-02-19T15:25:10.558098+0000","last_became_active":"2024-02-19T15:25:09.578192+0000","last_became_peered":"2024-02-19T15:25:09.578192+0000","last_unstale":"2024-02-19T15:25:10.558098+0000","last_undegraded":"2024-02-19T15:25:10.558098+0000","last_fullsized":"2024-02-19T15:25:10.558098+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:04:29.221838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558337+0000","last_change":"2024-02-19T15:25:09.573124+0000","last_active":"2024-02-19T15:25:10.558337+0000","last_peered":"2024-02-19T15:25:10.558337+0000","last_clean":"2024-02-19T15:25:10.558337+0000","last_became_active":"2024-02-19T15:25:09.570133+0000","last_became_peered":"2024-02-19T15:25:09.570133+0000","last_unstale":"2024-02-19T15:25:10.558337+0000","last_undegraded":"2024-02-19T15:25:10.558337+0000","last_fullsized":"2024-02-19T15:25:10.558337+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:28:21.379442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.583168+0000","last_change":"2024-02-19T15:25:09.579686+0000","last_active":"2024-02-19T15:25:10.583168+0000","last_peered":"2024-02-19T15:25:10.583168+0000","last_clean":"2024-02-19T15:25:10.583168+0000","last_became_active":"2024-02-19T15:25:09.579320+0000","last_became_peered":"2024-02-19T15:25:09.579320+0000","last_unstale":"2024-02-19T15:25:10.583168+0000","last_undegraded":"2024-02-19T15:25:10.583168+0000","last_fullsized":"2024-02-19T15:25:10.583168+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:12:47.620288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.557953+0000","last_change":"2024-02-19T15:25:09.582015+0000","last_active":"2024-02-19T15:25:10.557953+0000","last_peered":"2024-02-19T15:25:10.557953+0000","last_clean":"2024-02-19T15:25:10.557953+0000","last_became_active":"2024-02-19T15:25:09.581703+0000","last_became_peered":"2024-02-19T15:25:09.581703+0000","last_unstale":"2024-02-19T15:25:10.557953+0000","last_undegraded":"2024-02-19T15:25:10.557953+0000","last_fullsized":"2024-02-19T15:25:10.557953+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:35:01.678546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558744+0000","last_change":"2024-02-19T15:25:09.576831+0000","last_active":"2024-02-19T15:25:10.558744+0000","last_peered":"2024-02-19T15:25:10.558744+0000","last_clean":"2024-02-19T15:25:10.558744+0000","last_became_active":"2024-02-19T15:25:09.576297+0000","last_became_peered":"2024-02-19T15:25:09.576297+0000","last_unstale":"2024-02-19T15:25:10.558744+0000","last_undegraded":"2024-02-19T15:25:10.558744+0000","last_fullsized":"2024-02-19T15:25:10.558744+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:02:50.293871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.583712+0000","last_change":"2024-02-19T15:25:09.578978+0000","last_active":"2024-02-19T15:25:09.583712+0000","last_peered":"2024-02-19T15:25:09.583712+0000","last_clean":"2024-02-19T15:25:09.583712+0000","last_became_active":"2024-02-19T15:25:09.576539+0000","last_became_peered":"2024-02-19T15:25:09.576539+0000","last_unstale":"2024-02-19T15:25:09.583712+0000","last_undegraded":"2024-02-19T15:25:09.583712+0000","last_fullsized":"2024-02-19T15:25:09.583712+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:09.612621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.590360+0000","last_change":"2024-02-19T15:25:09.585017+0000","last_active":"2024-02-19T15:25:09.590360+0000","last_peered":"2024-02-19T15:25:09.590360+0000","last_clean":"2024-02-19T15:25:09.590360+0000","last_became_active":"2024-02-19T15:25:09.581865+0000","last_became_peered":"2024-02-19T15:25:09.581865+0000","last_unstale":"2024-02-19T15:25:09.590360+0000","last_undegraded":"2024-02-19T15:25:09.590360+0000","last_fullsized":"2024-02-19T15:25:09.590360+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:35.960944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558594+0000","last_change":"2024-02-19T15:25:09.588372+0000","last_active":"2024-02-19T15:25:10.558594+0000","last_peered":"2024-02-19T15:25:10.558594+0000","last_clean":"2024-02-19T15:25:10.558594+0000","last_became_active":"2024-02-19T15:25:09.587556+0000","last_became_peered":"2024-02-19T15:25:09.587556+0000","last_unstale":"2024-02-19T15:25:10.558594+0000","last_undegraded":"2024-02-19T15:25:10.558594+0000","last_fullsized":"2024-02-19T15:25:10.558594+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:09.612621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.585955+0000","last_change":"2024-02-19T15:25:09.573743+0000","last_active":"2024-02-19T15:25:10.585955+0000","last_peered":"2024-02-19T15:25:10.585955+0000","last_clean":"2024-02-19T15:25:10.585955+0000","last_became_active":"2024-02-19T15:25:09.573386+0000","last_became_peered":"2024-02-19T15:25:09.573386+0000","last_unstale":"2024-02-19T15:25:10.585955+0000","last_undegraded":"2024-02-19T15:25:10.585955+0000","last_fullsized":"2024-02-19T15:25:10.585955+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:49:12.070117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.557366+0000","last_change":"2024-02-19T15:25:09.570867+0000","last_active":"2024-02-19T15:25:10.557366+0000","last_peered":"2024-02-19T15:25:10.557366+0000","last_clean":"2024-02-19T15:25:10.557366+0000","last_became_active":"2024-02-19T15:25:09.569829+0000","last_became_peered":"2024-02-19T15:25:09.569829+0000","last_unstale":"2024-02-19T15:25:10.557366+0000","last_undegraded":"2024-02-19T15:25:10.557366+0000","last_fullsized":"2024-02-19T15:25:10.557366+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:58:43.446804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.585886+0000","last_change":"2024-02-19T15:25:09.577004+0000","last_active":"2024-02-19T15:25:10.585886+0000","last_peered":"2024-02-19T15:25:10.585886+0000","last_clean":"2024-02-19T15:25:10.585886+0000","last_became_active":"2024-02-19T15:25:09.576695+0000","last_became_peered":"2024-02-19T15:25:09.576695+0000","last_unstale":"2024-02-19T15:25:10.585886+0000","last_undegraded":"2024-02-19T15:25:10.585886+0000","last_fullsized":"2024-02-19T15:25:10.585886+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:01:58.697058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.579031+0000","last_change":"2024-02-19T15:25:09.569337+0000","last_active":"2024-02-19T15:25:09.579031+0000","last_peered":"2024-02-19T15:25:09.579031+0000","last_clean":"2024-02-19T15:25:09.579031+0000","last_became_active":"2024-02-19T15:25:09.568940+0000","last_became_peered":"2024-02-19T15:25:09.568940+0000","last_unstale":"2024-02-19T15:25:09.579031+0000","last_undegraded":"2024-02-19T15:25:09.579031+0000","last_fullsized":"2024-02-19T15:25:09.579031+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:59:28.602083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558520+0000","last_change":"2024-02-19T15:25:09.576597+0000","last_active":"2024-02-19T15:25:10.558520+0000","last_peered":"2024-02-19T15:25:10.558520+0000","last_clean":"2024-02-19T15:25:10.558520+0000","last_became_active":"2024-02-19T15:25:09.576089+0000","last_became_peered":"2024-02-19T15:25:09.576089+0000","last_unstale":"2024-02-19T15:25:10.558520+0000","last_undegraded":"2024-02-19T15:25:10.558520+0000","last_fullsized":"2024-02-19T15:25:10.558520+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:34:54.835700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558807+0000","last_change":"2024-02-19T15:25:09.569269+0000","last_active":"2024-02-19T15:25:10.558807+0000","last_peered":"2024-02-19T15:25:10.558807+0000","last_clean":"2024-02-19T15:25:10.558807+0000","last_became_active":"2024-02-19T15:25:09.569007+0000","last_became_peered":"2024-02-19T15:25:09.569007+0000","last_unstale":"2024-02-19T15:25:10.558807+0000","last_undegraded":"2024-02-19T15:25:10.558807+0000","last_fullsized":"2024-02-19T15:25:10.558807+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:42:45.609838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558182+0000","last_change":"2024-02-19T15:25:09.582843+0000","last_active":"2024-02-19T15:25:10.558182+0000","last_peered":"2024-02-19T15:25:10.558182+0000","last_clean":"2024-02-19T15:25:10.558182+0000","last_became_active":"2024-02-19T15:25:09.582508+0000","last_became_peered":"2024-02-19T15:25:09.582508+0000","last_unstale":"2024-02-19T15:25:10.558182+0000","last_undegraded":"2024-02-19T15:25:10.558182+0000","last_fullsized":"2024-02-19T15:25:10.558182+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:33:28.706967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.585960+0000","last_change":"2024-02-19T15:25:09.567465+0000","last_active":"2024-02-19T15:25:10.585960+0000","last_peered":"2024-02-19T15:25:10.585960+0000","last_clean":"2024-02-19T15:25:10.585960+0000","last_became_active":"2024-02-19T15:25:09.567175+0000","last_became_peered":"2024-02-19T15:25:09.567175+0000","last_unstale":"2024-02-19T15:25:10.585960+0000","last_undegraded":"2024-02-19T15:25:10.585960+0000","last_fullsized":"2024-02-19T15:25:10.585960+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:49:41.563813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.579964+0000","last_change":"2024-02-19T15:25:09.569487+0000","last_active":"2024-02-19T15:25:09.579964+0000","last_peered":"2024-02-19T15:25:09.579964+0000","last_clean":"2024-02-19T15:25:09.579964+0000","last_became_active":"2024-02-19T15:25:09.569089+0000","last_became_peered":"2024-02-19T15:25:09.569089+0000","last_unstale":"2024-02-19T15:25:09.579964+0000","last_undegraded":"2024-02-19T15:25:09.579964+0000","last_fullsized":"2024-02-19T15:25:09.579964+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:08:06.606450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.581013+0000","last_change":"2024-02-19T15:25:09.575891+0000","last_active":"2024-02-19T15:25:09.581013+0000","last_peered":"2024-02-19T15:25:09.581013+0000","last_clean":"2024-02-19T15:25:09.581013+0000","last_became_active":"2024-02-19T15:25:09.575560+0000","last_became_peered":"2024-02-19T15:25:09.575560+0000","last_unstale":"2024-02-19T15:25:09.581013+0000","last_undegraded":"2024-02-19T15:25:09.581013+0000","last_fullsized":"2024-02-19T15:25:09.581013+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:29:15.047882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.588109+0000","last_change":"2024-02-19T15:25:09.582056+0000","last_active":"2024-02-19T15:25:09.588109+0000","last_peered":"2024-02-19T15:25:09.588109+0000","last_clean":"2024-02-19T15:25:09.588109+0000","last_became_active":"2024-02-19T15:25:09.581708+0000","last_became_peered":"2024-02-19T15:25:09.581708+0000","last_unstale":"2024-02-19T15:25:09.588109+0000","last_undegraded":"2024-02-19T15:25:09.588109+0000","last_fullsized":"2024-02-19T15:25:09.588109+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:37:30.944398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.591602+0000","last_change":"2024-02-19T15:25:09.585362+0000","last_active":"2024-02-19T15:25:09.591602+0000","last_peered":"2024-02-19T15:25:09.591602+0000","last_clean":"2024-02-19T15:25:09.591602+0000","last_became_active":"2024-02-19T15:25:09.584519+0000","last_became_peered":"2024-02-19T15:25:09.584519+0000","last_unstale":"2024-02-19T15:25:09.591602+0000","last_undegraded":"2024-02-19T15:25:09.591602+0000","last_fullsized":"2024-02-19T15:25:09.591602+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:03:58.020170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.581140+0000","last_change":"2024-02-19T15:25:09.577318+0000","last_active":"2024-02-19T15:25:09.581140+0000","last_peered":"2024-02-19T15:25:09.581140+0000","last_clean":"2024-02-19T15:25:09.581140+0000","last_became_active":"2024-02-19T15:25:09.577044+0000","last_became_peered":"2024-02-19T15:25:09.577044+0000","last_unstale":"2024-02-19T15:25:09.581140+0000","last_undegraded":"2024-02-19T15:25:09.581140+0000","last_fullsized":"2024-02-19T15:25:09.581140+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:53:35.452518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.583392+0000","last_change":"2024-02-19T15:25:09.567406+0000","last_active":"2024-02-19T15:25:10.583392+0000","last_peered":"2024-02-19T15:25:10.583392+0000","last_clean":"2024-02-19T15:25:10.583392+0000","last_became_active":"2024-02-19T15:25:09.567151+0000","last_became_peered":"2024-02-19T15:25:09.567151+0000","last_unstale":"2024-02-19T15:25:10.583392+0000","last_undegraded":"2024-02-19T15:25:10.583392+0000","last_fullsized":"2024-02-19T15:25:10.583392+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:28:11.906012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.585043+0000","last_change":"2024-02-19T15:25:09.580152+0000","last_active":"2024-02-19T15:25:09.585043+0000","last_peered":"2024-02-19T15:25:09.585043+0000","last_clean":"2024-02-19T15:25:09.585043+0000","last_became_active":"2024-02-19T15:25:09.579878+0000","last_became_peered":"2024-02-19T15:25:09.579878+0000","last_unstale":"2024-02-19T15:25:09.585043+0000","last_undegraded":"2024-02-19T15:25:09.585043+0000","last_fullsized":"2024-02-19T15:25:09.585043+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:13:11.065363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.593681+0000","last_change":"2024-02-19T15:25:09.589035+0000","last_active":"2024-02-19T15:25:09.593681+0000","last_peered":"2024-02-19T15:25:09.593681+0000","last_clean":"2024-02-19T15:25:09.593681+0000","last_became_active":"2024-02-19T15:25:09.586288+0000","last_became_peered":"2024-02-19T15:25:09.586288+0000","last_unstale":"2024-02-19T15:25:09.593681+0000","last_undegraded":"2024-02-19T15:25:09.593681+0000","last_fullsized":"2024-02-19T15:25:09.593681+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:59:31.519245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.579288+0000","last_change":"2024-02-19T15:25:09.573556+0000","last_active":"2024-02-19T15:25:09.579288+0000","last_peered":"2024-02-19T15:25:09.579288+0000","last_clean":"2024-02-19T15:25:09.579288+0000","last_became_active":"2024-02-19T15:25:09.573245+0000","last_became_peered":"2024-02-19T15:25:09.573245+0000","last_unstale":"2024-02-19T15:25:09.579288+0000","last_undegraded":"2024-02-19T15:25:09.579288+0000","last_fullsized":"2024-02-19T15:25:09.579288+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:08:22.288508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.583220+0000","last_change":"2024-02-19T15:25:09.567324+0000","last_active":"2024-02-19T15:25:10.583220+0000","last_peered":"2024-02-19T15:25:10.583220+0000","last_clean":"2024-02-19T15:25:10.583220+0000","last_became_active":"2024-02-19T15:25:09.567049+0000","last_became_peered":"2024-02-19T15:25:09.567049+0000","last_unstale":"2024-02-19T15:25:10.583220+0000","last_undegraded":"2024-02-19T15:25:10.583220+0000","last_fullsized":"2024-02-19T15:25:10.583220+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:23:16.816936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.583357+0000","last_change":"2024-02-19T15:25:09.573494+0000","last_active":"2024-02-19T15:25:10.583357+0000","last_peered":"2024-02-19T15:25:10.583357+0000","last_clean":"2024-02-19T15:25:10.583357+0000","last_became_active":"2024-02-19T15:25:09.572506+0000","last_became_peered":"2024-02-19T15:25:09.572506+0000","last_unstale":"2024-02-19T15:25:10.583357+0000","last_undegraded":"2024-02-19T15:25:10.583357+0000","last_fullsized":"2024-02-19T15:25:10.583357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:33:28.632640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558529+0000","last_change":"2024-02-19T15:25:09.575585+0000","last_active":"2024-02-19T15:25:10.558529+0000","last_peered":"2024-02-19T15:25:10.558529+0000","last_clean":"2024-02-19T15:25:10.558529+0000","last_became_active":"2024-02-19T15:25:09.575312+0000","last_became_peered":"2024-02-19T15:25:09.575312+0000","last_unstale":"2024-02-19T15:25:10.558529+0000","last_undegraded":"2024-02-19T15:25:10.558529+0000","last_fullsized":"2024-02-19T15:25:10.558529+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:24.336459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.591381+0000","last_change":"2024-02-19T15:25:09.587937+0000","last_active":"2024-02-19T15:25:09.591381+0000","last_peered":"2024-02-19T15:25:09.591381+0000","last_clean":"2024-02-19T15:25:09.591381+0000","last_became_active":"2024-02-19T15:25:09.586236+0000","last_became_peered":"2024-02-19T15:25:09.586236+0000","last_unstale":"2024-02-19T15:25:09.591381+0000","last_undegraded":"2024-02-19T15:25:09.591381+0000","last_fullsized":"2024-02-19T15:25:09.591381+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:36:37.030998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"68'4","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.593881+0000","last_change":"2024-02-19T15:25:08.580513+0000","last_active":"2024-02-19T15:25:09.593881+0000","last_peered":"2024-02-19T15:25:09.593881+0000","last_clean":"2024-02-19T15:25:09.593881+0000","last_became_active":"2024-02-19T15:25:08.580206+0000","last_became_peered":"2024-02-19T15:25:08.580206+0000","last_unstale":"2024-02-19T15:25:09.593881+0000","last_undegraded":"2024-02-19T15:25:09.593881+0000","last_fullsized":"2024-02-19T15:25:09.593881+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:12:46.613959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558913+0000","last_change":"2024-02-19T15:25:09.577238+0000","last_active":"2024-02-19T15:25:10.558913+0000","last_peered":"2024-02-19T15:25:10.558913+0000","last_clean":"2024-02-19T15:25:10.558913+0000","last_became_active":"2024-02-19T15:25:09.575907+0000","last_became_peered":"2024-02-19T15:25:09.575907+0000","last_unstale":"2024-02-19T15:25:10.558913+0000","last_undegraded":"2024-02-19T15:25:10.558913+0000","last_fullsized":"2024-02-19T15:25:10.558913+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:54:36.228347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.586249+0000","last_change":"2024-02-19T15:25:08.572351+0000","last_active":"2024-02-19T15:25:10.586249+0000","last_peered":"2024-02-19T15:25:10.586249+0000","last_clean":"2024-02-19T15:25:10.586249+0000","last_became_active":"2024-02-19T15:25:08.571637+0000","last_became_peered":"2024-02-19T15:25:08.571637+0000","last_unstale":"2024-02-19T15:25:10.586249+0000","last_undegraded":"2024-02-19T15:25:10.586249+0000","last_fullsized":"2024-02-19T15:25:10.586249+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:48:08.453725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558565+0000","last_change":"2024-02-19T15:25:09.578182+0000","last_active":"2024-02-19T15:25:10.558565+0000","last_peered":"2024-02-19T15:25:10.558565+0000","last_clean":"2024-02-19T15:25:10.558565+0000","last_became_active":"2024-02-19T15:25:09.577876+0000","last_became_peered":"2024-02-19T15:25:09.577876+0000","last_unstale":"2024-02-19T15:25:10.558565+0000","last_undegraded":"2024-02-19T15:25:10.558565+0000","last_fullsized":"2024-02-19T15:25:10.558565+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:16.154993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"68'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.594368+0000","last_change":"2024-02-19T15:25:08.566970+0000","last_active":"2024-02-19T15:25:09.594368+0000","last_peered":"2024-02-19T15:25:09.594368+0000","last_clean":"2024-02-19T15:25:09.594368+0000","last_became_active":"2024-02-19T15:25:08.566757+0000","last_became_peered":"2024-02-19T15:25:08.566757+0000","last_unstale":"2024-02-19T15:25:09.594368+0000","last_undegraded":"2024-02-19T15:25:09.594368+0000","last_fullsized":"2024-02-19T15:25:09.594368+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:39:21.663720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.588151+0000","last_change":"2024-02-19T15:25:09.579826+0000","last_active":"2024-02-19T15:25:09.588151+0000","last_peered":"2024-02-19T15:25:09.588151+0000","last_clean":"2024-02-19T15:25:09.588151+0000","last_became_active":"2024-02-19T15:25:09.578258+0000","last_became_peered":"2024-02-19T15:25:09.578258+0000","last_unstale":"2024-02-19T15:25:09.588151+0000","last_undegraded":"2024-02-19T15:25:09.588151+0000","last_fullsized":"2024-02-19T15:25:09.588151+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:33:28.632640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558675+0000","last_change":"2024-02-19T15:25:08.571511+0000","last_active":"2024-02-19T15:25:10.558675+0000","last_peered":"2024-02-19T15:25:10.558675+0000","last_clean":"2024-02-19T15:25:10.558675+0000","last_became_active":"2024-02-19T15:25:08.571146+0000","last_became_peered":"2024-02-19T15:25:08.571146+0000","last_unstale":"2024-02-19T15:25:10.558675+0000","last_undegraded":"2024-02-19T15:25:10.558675+0000","last_fullsized":"2024-02-19T15:25:10.558675+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:40:48.711503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.583425+0000","last_change":"2024-02-19T15:25:09.576499+0000","last_active":"2024-02-19T15:25:10.583425+0000","last_peered":"2024-02-19T15:25:10.583425+0000","last_clean":"2024-02-19T15:25:10.583425+0000","last_became_active":"2024-02-19T15:25:09.576162+0000","last_became_peered":"2024-02-19T15:25:09.576162+0000","last_unstale":"2024-02-19T15:25:10.583425+0000","last_undegraded":"2024-02-19T15:25:10.583425+0000","last_fullsized":"2024-02-19T15:25:10.583425+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:08:07.559774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"68'1","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.595048+0000","last_change":"2024-02-19T15:25:08.567701+0000","last_active":"2024-02-19T15:25:09.595048+0000","last_peered":"2024-02-19T15:25:09.595048+0000","last_clean":"2024-02-19T15:25:09.595048+0000","last_became_active":"2024-02-19T15:25:08.566265+0000","last_became_peered":"2024-02-19T15:25:08.566265+0000","last_unstale":"2024-02-19T15:25:09.595048+0000","last_undegraded":"2024-02-19T15:25:09.595048+0000","last_fullsized":"2024-02-19T15:25:09.595048+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:28:30.590100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.584767+0000","last_change":"2024-02-19T15:25:09.579816+0000","last_active":"2024-02-19T15:25:09.584767+0000","last_peered":"2024-02-19T15:25:09.584767+0000","last_clean":"2024-02-19T15:25:09.584767+0000","last_became_active":"2024-02-19T15:25:09.578785+0000","last_became_peered":"2024-02-19T15:25:09.578785+0000","last_unstale":"2024-02-19T15:25:09.584767+0000","last_undegraded":"2024-02-19T15:25:09.584767+0000","last_fullsized":"2024-02-19T15:25:09.584767+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:42:45.609838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558280+0000","last_change":"2024-02-19T15:25:08.575448+0000","last_active":"2024-02-19T15:25:10.558280+0000","last_peered":"2024-02-19T15:25:10.558280+0000","last_clean":"2024-02-19T15:25:10.558280+0000","last_became_active":"2024-02-19T15:25:08.573670+0000","last_became_peered":"2024-02-19T15:25:08.573670+0000","last_unstale":"2024-02-19T15:25:10.558280+0000","last_undegraded":"2024-02-19T15:25:10.558280+0000","last_fullsized":"2024-02-19T15:25:10.558280+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:29:49.691931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.557154+0000","last_change":"2024-02-19T15:25:09.572296+0000","last_active":"2024-02-19T15:25:10.557154+0000","last_peered":"2024-02-19T15:25:10.557154+0000","last_clean":"2024-02-19T15:25:10.557154+0000","last_became_active":"2024-02-19T15:25:09.571786+0000","last_became_peered":"2024-02-19T15:25:09.571786+0000","last_unstale":"2024-02-19T15:25:10.557154+0000","last_undegraded":"2024-02-19T15:25:10.557154+0000","last_fullsized":"2024-02-19T15:25:10.557154+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:57:23.401485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558375+0000","last_change":"2024-02-19T15:25:08.569630+0000","last_active":"2024-02-19T15:25:10.558375+0000","last_peered":"2024-02-19T15:25:10.558375+0000","last_clean":"2024-02-19T15:25:10.558375+0000","last_became_active":"2024-02-19T15:25:08.569354+0000","last_became_peered":"2024-02-19T15:25:08.569354+0000","last_unstale":"2024-02-19T15:25:10.558375+0000","last_undegraded":"2024-02-19T15:25:10.558375+0000","last_fullsized":"2024-02-19T15:25:10.558375+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:05:26.885441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.585908+0000","last_change":"2024-02-19T15:25:09.578689+0000","last_active":"2024-02-19T15:25:10.585908+0000","last_peered":"2024-02-19T15:25:10.585908+0000","last_clean":"2024-02-19T15:25:10.585908+0000","last_became_active":"2024-02-19T15:25:09.578400+0000","last_became_peered":"2024-02-19T15:25:09.578400+0000","last_unstale":"2024-02-19T15:25:10.585908+0000","last_undegraded":"2024-02-19T15:25:10.585908+0000","last_fullsized":"2024-02-19T15:25:10.585908+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:19:21.711082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.557148+0000","last_change":"2024-02-19T15:25:08.564608+0000","last_active":"2024-02-19T15:25:10.557148+0000","last_peered":"2024-02-19T15:25:10.557148+0000","last_clean":"2024-02-19T15:25:10.557148+0000","last_became_active":"2024-02-19T15:25:08.564341+0000","last_became_peered":"2024-02-19T15:25:08.564341+0000","last_unstale":"2024-02-19T15:25:10.557148+0000","last_undegraded":"2024-02-19T15:25:10.557148+0000","last_fullsized":"2024-02-19T15:25:10.557148+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:33:13.979253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.583432+0000","last_change":"2024-02-19T15:25:09.577318+0000","last_active":"2024-02-19T15:25:10.583432+0000","last_peered":"2024-02-19T15:25:10.583432+0000","last_clean":"2024-02-19T15:25:10.583432+0000","last_became_active":"2024-02-19T15:25:09.577023+0000","last_became_peered":"2024-02-19T15:25:09.577023+0000","last_unstale":"2024-02-19T15:25:10.583432+0000","last_undegraded":"2024-02-19T15:25:10.583432+0000","last_fullsized":"2024-02-19T15:25:10.583432+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:37.897102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.559049+0000","last_change":"2024-02-19T15:25:08.568933+0000","last_active":"2024-02-19T15:25:09.559049+0000","last_peered":"2024-02-19T15:25:09.559049+0000","last_clean":"2024-02-19T15:25:09.559049+0000","last_became_active":"2024-02-19T15:25:08.568435+0000","last_became_peered":"2024-02-19T15:25:08.568435+0000","last_unstale":"2024-02-19T15:25:09.559049+0000","last_undegraded":"2024-02-19T15:25:09.559049+0000","last_fullsized":"2024-02-19T15:25:09.559049+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:59:46.463412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.588443+0000","last_change":"2024-02-19T15:25:09.584180+0000","last_active":"2024-02-19T15:25:09.588443+0000","last_peered":"2024-02-19T15:25:09.588443+0000","last_clean":"2024-02-19T15:25:09.588443+0000","last_became_active":"2024-02-19T15:25:09.583866+0000","last_became_peered":"2024-02-19T15:25:09.583866+0000","last_unstale":"2024-02-19T15:25:09.588443+0000","last_undegraded":"2024-02-19T15:25:09.588443+0000","last_fullsized":"2024-02-19T15:25:09.588443+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:51.514855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558670+0000","last_change":"2024-02-19T15:25:08.575700+0000","last_active":"2024-02-19T15:25:10.558670+0000","last_peered":"2024-02-19T15:25:10.558670+0000","last_clean":"2024-02-19T15:25:10.558670+0000","last_became_active":"2024-02-19T15:25:08.575341+0000","last_became_peered":"2024-02-19T15:25:08.575341+0000","last_unstale":"2024-02-19T15:25:10.558670+0000","last_undegraded":"2024-02-19T15:25:10.558670+0000","last_fullsized":"2024-02-19T15:25:10.558670+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:49:36.529580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558533+0000","last_change":"2024-02-19T15:25:09.573713+0000","last_active":"2024-02-19T15:25:10.558533+0000","last_peered":"2024-02-19T15:25:10.558533+0000","last_clean":"2024-02-19T15:25:10.558533+0000","last_became_active":"2024-02-19T15:25:09.570159+0000","last_became_peered":"2024-02-19T15:25:09.570159+0000","last_unstale":"2024-02-19T15:25:10.558533+0000","last_undegraded":"2024-02-19T15:25:10.558533+0000","last_fullsized":"2024-02-19T15:25:10.558533+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:41:34.522285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"68'3","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.595036+0000","last_change":"2024-02-19T15:25:08.568892+0000","last_active":"2024-02-19T15:25:09.595036+0000","last_peered":"2024-02-19T15:25:09.595036+0000","last_clean":"2024-02-19T15:25:09.595036+0000","last_became_active":"2024-02-19T15:25:08.567779+0000","last_became_peered":"2024-02-19T15:25:08.567779+0000","last_unstale":"2024-02-19T15:25:09.595036+0000","last_undegraded":"2024-02-19T15:25:09.595036+0000","last_fullsized":"2024-02-19T15:25:09.595036+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:59.001326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558172+0000","last_change":"2024-02-19T15:25:09.584828+0000","last_active":"2024-02-19T15:25:10.558172+0000","last_peered":"2024-02-19T15:25:10.558172+0000","last_clean":"2024-02-19T15:25:10.558172+0000","last_became_active":"2024-02-19T15:25:09.584518+0000","last_became_peered":"2024-02-19T15:25:09.584518+0000","last_unstale":"2024-02-19T15:25:10.558172+0000","last_undegraded":"2024-02-19T15:25:10.558172+0000","last_fullsized":"2024-02-19T15:25:10.558172+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:09:34.830373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"68'6","reported_seq":20,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558698+0000","last_change":"2024-02-19T15:25:08.568960+0000","last_active":"2024-02-19T15:25:10.558698+0000","last_peered":"2024-02-19T15:25:10.558698+0000","last_clean":"2024-02-19T15:25:10.558698+0000","last_became_active":"2024-02-19T15:25:08.568696+0000","last_became_peered":"2024-02-19T15:25:08.568696+0000","last_unstale":"2024-02-19T15:25:10.558698+0000","last_undegraded":"2024-02-19T15:25:10.558698+0000","last_fullsized":"2024-02-19T15:25:10.558698+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:40:31.822001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.589446+0000","last_change":"2024-02-19T15:25:09.582489+0000","last_active":"2024-02-19T15:25:09.589446+0000","last_peered":"2024-02-19T15:25:09.589446+0000","last_clean":"2024-02-19T15:25:09.589446+0000","last_became_active":"2024-02-19T15:25:09.582227+0000","last_became_peered":"2024-02-19T15:25:09.582227+0000","last_unstale":"2024-02-19T15:25:09.589446+0000","last_undegraded":"2024-02-19T15:25:09.589446+0000","last_fullsized":"2024-02-19T15:25:09.589446+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:08:07.559774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558486+0000","last_change":"2024-02-19T15:25:08.575611+0000","last_active":"2024-02-19T15:25:10.558486+0000","last_peered":"2024-02-19T15:25:10.558486+0000","last_clean":"2024-02-19T15:25:10.558486+0000","last_became_active":"2024-02-19T15:25:08.575301+0000","last_became_peered":"2024-02-19T15:25:08.575301+0000","last_unstale":"2024-02-19T15:25:10.558486+0000","last_undegraded":"2024-02-19T15:25:10.558486+0000","last_fullsized":"2024-02-19T15:25:10.558486+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:39:21.663720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.583490+0000","last_change":"2024-02-19T15:25:09.573517+0000","last_active":"2024-02-19T15:25:10.583490+0000","last_peered":"2024-02-19T15:25:10.583490+0000","last_clean":"2024-02-19T15:25:10.583490+0000","last_became_active":"2024-02-19T15:25:09.573215+0000","last_became_peered":"2024-02-19T15:25:09.573215+0000","last_unstale":"2024-02-19T15:25:10.583490+0000","last_undegraded":"2024-02-19T15:25:10.583490+0000","last_fullsized":"2024-02-19T15:25:10.583490+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:51.514855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558497+0000","last_change":"2024-02-19T15:25:08.574007+0000","last_active":"2024-02-19T15:25:10.558497+0000","last_peered":"2024-02-19T15:25:10.558497+0000","last_clean":"2024-02-19T15:25:10.558497+0000","last_became_active":"2024-02-19T15:25:08.572500+0000","last_became_peered":"2024-02-19T15:25:08.572500+0000","last_unstale":"2024-02-19T15:25:10.558497+0000","last_undegraded":"2024-02-19T15:25:10.558497+0000","last_fullsized":"2024-02-19T15:25:10.558497+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:32:26.159687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.585830+0000","last_change":"2024-02-19T15:25:09.570069+0000","last_active":"2024-02-19T15:25:10.585830+0000","last_peered":"2024-02-19T15:25:10.585830+0000","last_clean":"2024-02-19T15:25:10.585830+0000","last_became_active":"2024-02-19T15:25:09.569741+0000","last_became_peered":"2024-02-19T15:25:09.569741+0000","last_unstale":"2024-02-19T15:25:10.585830+0000","last_undegraded":"2024-02-19T15:25:10.585830+0000","last_fullsized":"2024-02-19T15:25:10.585830+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:47:40.288953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.583115+0000","last_change":"2024-02-19T15:25:08.573220+0000","last_active":"2024-02-19T15:25:10.583115+0000","last_peered":"2024-02-19T15:25:10.583115+0000","last_clean":"2024-02-19T15:25:10.583115+0000","last_became_active":"2024-02-19T15:25:08.572950+0000","last_became_peered":"2024-02-19T15:25:08.572950+0000","last_unstale":"2024-02-19T15:25:10.583115+0000","last_undegraded":"2024-02-19T15:25:10.583115+0000","last_fullsized":"2024-02-19T15:25:10.583115+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:20:35.629924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.557306+0000","last_change":"2024-02-19T15:25:09.576673+0000","last_active":"2024-02-19T15:25:10.557306+0000","last_peered":"2024-02-19T15:25:10.557306+0000","last_clean":"2024-02-19T15:25:10.557306+0000","last_became_active":"2024-02-19T15:25:09.576167+0000","last_became_peered":"2024-02-19T15:25:09.576167+0000","last_unstale":"2024-02-19T15:25:10.557306+0000","last_undegraded":"2024-02-19T15:25:10.557306+0000","last_fullsized":"2024-02-19T15:25:10.557306+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:36:14.037862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"68'8","reported_seq":20,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.597105+0000","last_change":"2024-02-19T15:25:08.576263+0000","last_active":"2024-02-19T15:25:09.597105+0000","last_peered":"2024-02-19T15:25:09.597105+0000","last_clean":"2024-02-19T15:25:09.597105+0000","last_became_active":"2024-02-19T15:25:08.575964+0000","last_became_peered":"2024-02-19T15:25:08.575964+0000","last_unstale":"2024-02-19T15:25:09.597105+0000","last_undegraded":"2024-02-19T15:25:09.597105+0000","last_fullsized":"2024-02-19T15:25:09.597105+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:16:33.527348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.584731+0000","last_change":"2024-02-19T15:25:09.579619+0000","last_active":"2024-02-19T15:25:09.584731+0000","last_peered":"2024-02-19T15:25:09.584731+0000","last_clean":"2024-02-19T15:25:09.584731+0000","last_became_active":"2024-02-19T15:25:09.579345+0000","last_became_peered":"2024-02-19T15:25:09.579345+0000","last_unstale":"2024-02-19T15:25:09.584731+0000","last_undegraded":"2024-02-19T15:25:09.584731+0000","last_fullsized":"2024-02-19T15:25:09.584731+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:48:46.355752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558345+0000","last_change":"2024-02-19T15:25:08.575432+0000","last_active":"2024-02-19T15:25:10.558345+0000","last_peered":"2024-02-19T15:25:10.558345+0000","last_clean":"2024-02-19T15:25:10.558345+0000","last_became_active":"2024-02-19T15:25:08.575033+0000","last_became_peered":"2024-02-19T15:25:08.575033+0000","last_unstale":"2024-02-19T15:25:10.558345+0000","last_undegraded":"2024-02-19T15:25:10.558345+0000","last_fullsized":"2024-02-19T15:25:10.558345+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:59:17.185186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.588405+0000","last_change":"2024-02-19T15:25:09.581219+0000","last_active":"2024-02-19T15:25:09.588405+0000","last_peered":"2024-02-19T15:25:09.588405+0000","last_clean":"2024-02-19T15:25:09.588405+0000","last_became_active":"2024-02-19T15:25:09.580926+0000","last_became_peered":"2024-02-19T15:25:09.580926+0000","last_unstale":"2024-02-19T15:25:09.588405+0000","last_undegraded":"2024-02-19T15:25:09.588405+0000","last_fullsized":"2024-02-19T15:25:09.588405+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:28:11.906012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"68'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.594090+0000","last_change":"2024-02-19T15:25:08.568049+0000","last_active":"2024-02-19T15:25:09.594090+0000","last_peered":"2024-02-19T15:25:09.594090+0000","last_clean":"2024-02-19T15:25:09.594090+0000","last_became_active":"2024-02-19T15:25:08.567486+0000","last_became_peered":"2024-02-19T15:25:08.567486+0000","last_unstale":"2024-02-19T15:25:09.594090+0000","last_undegraded":"2024-02-19T15:25:09.594090+0000","last_fullsized":"2024-02-19T15:25:09.594090+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:47:28.106735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558628+0000","last_change":"2024-02-19T15:25:09.588109+0000","last_active":"2024-02-19T15:25:10.558628+0000","last_peered":"2024-02-19T15:25:10.558628+0000","last_clean":"2024-02-19T15:25:10.558628+0000","last_became_active":"2024-02-19T15:25:09.587513+0000","last_became_peered":"2024-02-19T15:25:09.587513+0000","last_unstale":"2024-02-19T15:25:10.558628+0000","last_undegraded":"2024-02-19T15:25:10.558628+0000","last_fullsized":"2024-02-19T15:25:10.558628+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:48:46.355752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558618+0000","last_change":"2024-02-19T15:25:08.576669+0000","last_active":"2024-02-19T15:25:10.558618+0000","last_peered":"2024-02-19T15:25:10.558618+0000","last_clean":"2024-02-19T15:25:10.558618+0000","last_became_active":"2024-02-19T15:25:08.576376+0000","last_became_peered":"2024-02-19T15:25:08.576376+0000","last_unstale":"2024-02-19T15:25:10.558618+0000","last_undegraded":"2024-02-19T15:25:10.558618+0000","last_fullsized":"2024-02-19T15:25:10.558618+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:42:53.328412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558316+0000","last_change":"2024-02-19T15:25:09.577601+0000","last_active":"2024-02-19T15:25:10.558316+0000","last_peered":"2024-02-19T15:25:10.558316+0000","last_clean":"2024-02-19T15:25:10.558316+0000","last_became_active":"2024-02-19T15:25:09.577339+0000","last_became_peered":"2024-02-19T15:25:09.577339+0000","last_unstale":"2024-02-19T15:25:10.558316+0000","last_undegraded":"2024-02-19T15:25:10.558316+0000","last_fullsized":"2024-02-19T15:25:10.558316+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:41:24.898113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.571722+0000","last_change":"2024-02-19T15:25:08.580632+0000","last_active":"2024-02-19T15:25:09.571722+0000","last_peered":"2024-02-19T15:25:09.571722+0000","last_clean":"2024-02-19T15:25:09.571722+0000","last_became_active":"2024-02-19T15:25:08.580293+0000","last_became_peered":"2024-02-19T15:25:08.580293+0000","last_unstale":"2024-02-19T15:25:09.571722+0000","last_undegraded":"2024-02-19T15:25:09.571722+0000","last_fullsized":"2024-02-19T15:25:09.571722+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:23:15.810607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.579311+0000","last_change":"2024-02-19T15:25:09.575733+0000","last_active":"2024-02-19T15:25:09.579311+0000","last_peered":"2024-02-19T15:25:09.579311+0000","last_clean":"2024-02-19T15:25:09.579311+0000","last_became_active":"2024-02-19T15:25:09.575443+0000","last_became_peered":"2024-02-19T15:25:09.575443+0000","last_unstale":"2024-02-19T15:25:09.579311+0000","last_undegraded":"2024-02-19T15:25:09.579311+0000","last_fullsized":"2024-02-19T15:25:09.579311+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:11:07.163392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558543+0000","last_change":"2024-02-19T15:25:08.575599+0000","last_active":"2024-02-19T15:25:10.558543+0000","last_peered":"2024-02-19T15:25:10.558543+0000","last_clean":"2024-02-19T15:25:10.558543+0000","last_became_active":"2024-02-19T15:25:08.573017+0000","last_became_peered":"2024-02-19T15:25:08.573017+0000","last_unstale":"2024-02-19T15:25:10.558543+0000","last_undegraded":"2024-02-19T15:25:10.558543+0000","last_fullsized":"2024-02-19T15:25:10.558543+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:54:29.908393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.569204+0000","last_change":"2024-02-19T15:25:09.564299+0000","last_active":"2024-02-19T15:25:09.569204+0000","last_peered":"2024-02-19T15:25:09.569204+0000","last_clean":"2024-02-19T15:25:09.569204+0000","last_became_active":"2024-02-19T15:25:09.564006+0000","last_became_peered":"2024-02-19T15:25:09.564006+0000","last_unstale":"2024-02-19T15:25:09.569204+0000","last_undegraded":"2024-02-19T15:25:09.569204+0000","last_fullsized":"2024-02-19T15:25:09.569204+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:01:31.594016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.583319+0000","last_change":"2024-02-19T15:25:08.569639+0000","last_active":"2024-02-19T15:25:10.583319+0000","last_peered":"2024-02-19T15:25:10.583319+0000","last_clean":"2024-02-19T15:25:10.583319+0000","last_became_active":"2024-02-19T15:25:08.569345+0000","last_became_peered":"2024-02-19T15:25:08.569345+0000","last_unstale":"2024-02-19T15:25:10.583319+0000","last_undegraded":"2024-02-19T15:25:10.583319+0000","last_fullsized":"2024-02-19T15:25:10.583319+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:11:58.566234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.573755+0000","last_change":"2024-02-19T15:25:09.564896+0000","last_active":"2024-02-19T15:25:09.573755+0000","last_peered":"2024-02-19T15:25:09.573755+0000","last_clean":"2024-02-19T15:25:09.573755+0000","last_became_active":"2024-02-19T15:25:09.563357+0000","last_became_peered":"2024-02-19T15:25:09.563357+0000","last_unstale":"2024-02-19T15:25:09.573755+0000","last_undegraded":"2024-02-19T15:25:09.573755+0000","last_fullsized":"2024-02-19T15:25:09.573755+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:24:16.133415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.585761+0000","last_change":"2024-02-19T15:25:08.572136+0000","last_active":"2024-02-19T15:25:10.585761+0000","last_peered":"2024-02-19T15:25:10.585761+0000","last_clean":"2024-02-19T15:25:10.585761+0000","last_became_active":"2024-02-19T15:25:08.571755+0000","last_became_peered":"2024-02-19T15:25:08.571755+0000","last_unstale":"2024-02-19T15:25:10.585761+0000","last_undegraded":"2024-02-19T15:25:10.585761+0000","last_fullsized":"2024-02-19T15:25:10.585761+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:15:58.923487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.586206+0000","last_change":"2024-02-19T15:25:09.579080+0000","last_active":"2024-02-19T15:25:10.586206+0000","last_peered":"2024-02-19T15:25:10.586206+0000","last_clean":"2024-02-19T15:25:10.586206+0000","last_became_active":"2024-02-19T15:25:09.578565+0000","last_became_peered":"2024-02-19T15:25:09.578565+0000","last_unstale":"2024-02-19T15:25:10.586206+0000","last_undegraded":"2024-02-19T15:25:10.586206+0000","last_fullsized":"2024-02-19T15:25:10.586206+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:32:36.999401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558297+0000","last_change":"2024-02-19T15:25:08.573789+0000","last_active":"2024-02-19T15:25:10.558297+0000","last_peered":"2024-02-19T15:25:10.558297+0000","last_clean":"2024-02-19T15:25:10.558297+0000","last_became_active":"2024-02-19T15:25:08.571824+0000","last_became_peered":"2024-02-19T15:25:08.571824+0000","last_unstale":"2024-02-19T15:25:10.558297+0000","last_undegraded":"2024-02-19T15:25:10.558297+0000","last_fullsized":"2024-02-19T15:25:10.558297+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:00:41.451451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.583428+0000","last_change":"2024-02-19T15:25:09.579622+0000","last_active":"2024-02-19T15:25:10.583428+0000","last_peered":"2024-02-19T15:25:10.583428+0000","last_clean":"2024-02-19T15:25:10.583428+0000","last_became_active":"2024-02-19T15:25:09.579310+0000","last_became_peered":"2024-02-19T15:25:09.579310+0000","last_unstale":"2024-02-19T15:25:10.583428+0000","last_undegraded":"2024-02-19T15:25:10.583428+0000","last_fullsized":"2024-02-19T15:25:10.583428+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:37:30.944398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"69'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.585029+0000","last_change":"2024-02-19T15:25:08.566886+0000","last_active":"2024-02-19T15:25:10.585029+0000","last_peered":"2024-02-19T15:25:10.585029+0000","last_clean":"2024-02-19T15:25:10.585029+0000","last_became_active":"2024-02-19T15:25:08.566587+0000","last_became_peered":"2024-02-19T15:25:08.566587+0000","last_unstale":"2024-02-19T15:25:10.585029+0000","last_undegraded":"2024-02-19T15:25:10.585029+0000","last_fullsized":"2024-02-19T15:25:10.585029+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:30:36.388006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.590560+0000","last_change":"2024-02-19T15:25:09.587172+0000","last_active":"2024-02-19T15:25:09.590560+0000","last_peered":"2024-02-19T15:25:09.590560+0000","last_clean":"2024-02-19T15:25:09.590560+0000","last_became_active":"2024-02-19T15:25:09.585035+0000","last_became_peered":"2024-02-19T15:25:09.585035+0000","last_unstale":"2024-02-19T15:25:09.590560+0000","last_undegraded":"2024-02-19T15:25:09.590560+0000","last_fullsized":"2024-02-19T15:25:09.590560+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:10:54.529180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558404+0000","last_change":"2024-02-19T15:25:08.572671+0000","last_active":"2024-02-19T15:25:10.558404+0000","last_peered":"2024-02-19T15:25:10.558404+0000","last_clean":"2024-02-19T15:25:10.558404+0000","last_became_active":"2024-02-19T15:25:08.572368+0000","last_became_peered":"2024-02-19T15:25:08.572368+0000","last_unstale":"2024-02-19T15:25:10.558404+0000","last_undegraded":"2024-02-19T15:25:10.558404+0000","last_fullsized":"2024-02-19T15:25:10.558404+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:24:48.668619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.590292+0000","last_change":"2024-02-19T15:25:09.584770+0000","last_active":"2024-02-19T15:25:09.590292+0000","last_peered":"2024-02-19T15:25:09.590292+0000","last_clean":"2024-02-19T15:25:09.590292+0000","last_became_active":"2024-02-19T15:25:09.581776+0000","last_became_peered":"2024-02-19T15:25:09.581776+0000","last_unstale":"2024-02-19T15:25:09.590292+0000","last_undegraded":"2024-02-19T15:25:09.590292+0000","last_fullsized":"2024-02-19T15:25:09.590292+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:42:27.619249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.586093+0000","last_change":"2024-02-19T15:25:08.573089+0000","last_active":"2024-02-19T15:25:10.586093+0000","last_peered":"2024-02-19T15:25:10.586093+0000","last_clean":"2024-02-19T15:25:10.586093+0000","last_became_active":"2024-02-19T15:25:08.571307+0000","last_became_peered":"2024-02-19T15:25:08.571307+0000","last_unstale":"2024-02-19T15:25:10.586093+0000","last_undegraded":"2024-02-19T15:25:10.586093+0000","last_fullsized":"2024-02-19T15:25:10.586093+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:43:01.462003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558068+0000","last_change":"2024-02-19T15:25:09.574232+0000","last_active":"2024-02-19T15:25:10.558068+0000","last_peered":"2024-02-19T15:25:10.558068+0000","last_clean":"2024-02-19T15:25:10.558068+0000","last_became_active":"2024-02-19T15:25:09.573079+0000","last_became_peered":"2024-02-19T15:25:09.573079+0000","last_unstale":"2024-02-19T15:25:10.558068+0000","last_undegraded":"2024-02-19T15:25:10.558068+0000","last_fullsized":"2024-02-19T15:25:10.558068+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:24:26.774738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558126+0000","last_change":"2024-02-19T15:25:08.565030+0000","last_active":"2024-02-19T15:25:10.558126+0000","last_peered":"2024-02-19T15:25:10.558126+0000","last_clean":"2024-02-19T15:25:10.558126+0000","last_became_active":"2024-02-19T15:25:08.564757+0000","last_became_peered":"2024-02-19T15:25:08.564757+0000","last_unstale":"2024-02-19T15:25:10.558126+0000","last_undegraded":"2024-02-19T15:25:10.558126+0000","last_fullsized":"2024-02-19T15:25:10.558126+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:01:09.241624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558620+0000","last_change":"2024-02-19T15:25:09.578218+0000","last_active":"2024-02-19T15:25:10.558620+0000","last_peered":"2024-02-19T15:25:10.558620+0000","last_clean":"2024-02-19T15:25:10.558620+0000","last_became_active":"2024-02-19T15:25:09.577914+0000","last_became_peered":"2024-02-19T15:25:09.577914+0000","last_unstale":"2024-02-19T15:25:10.558620+0000","last_undegraded":"2024-02-19T15:25:10.558620+0000","last_fullsized":"2024-02-19T15:25:10.558620+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:01:34.195995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558363+0000","last_change":"2024-02-19T15:25:08.569740+0000","last_active":"2024-02-19T15:25:10.558363+0000","last_peered":"2024-02-19T15:25:10.558363+0000","last_clean":"2024-02-19T15:25:10.558363+0000","last_became_active":"2024-02-19T15:25:08.569463+0000","last_became_peered":"2024-02-19T15:25:08.569463+0000","last_unstale":"2024-02-19T15:25:10.558363+0000","last_undegraded":"2024-02-19T15:25:10.558363+0000","last_fullsized":"2024-02-19T15:25:10.558363+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:01:24.418696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558243+0000","last_change":"2024-02-19T15:25:09.581476+0000","last_active":"2024-02-19T15:25:10.558243+0000","last_peered":"2024-02-19T15:25:10.558243+0000","last_clean":"2024-02-19T15:25:10.558243+0000","last_became_active":"2024-02-19T15:25:09.578600+0000","last_became_peered":"2024-02-19T15:25:09.578600+0000","last_unstale":"2024-02-19T15:25:10.558243+0000","last_undegraded":"2024-02-19T15:25:10.558243+0000","last_fullsized":"2024-02-19T15:25:10.558243+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:29:15.047882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.558373+0000","last_change":"2024-02-19T15:25:08.573705+0000","last_active":"2024-02-19T15:25:09.558373+0000","last_peered":"2024-02-19T15:25:09.558373+0000","last_clean":"2024-02-19T15:25:09.558373+0000","last_became_active":"2024-02-19T15:25:08.572748+0000","last_became_peered":"2024-02-19T15:25:08.572748+0000","last_unstale":"2024-02-19T15:25:09.558373+0000","last_undegraded":"2024-02-19T15:25:09.558373+0000","last_fullsized":"2024-02-19T15:25:09.558373+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:40:31.822001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.557886+0000","last_change":"2024-02-19T15:25:09.568735+0000","last_active":"2024-02-19T15:25:10.557886+0000","last_peered":"2024-02-19T15:25:10.557886+0000","last_clean":"2024-02-19T15:25:10.557886+0000","last_became_active":"2024-02-19T15:25:09.568412+0000","last_became_peered":"2024-02-19T15:25:09.568412+0000","last_unstale":"2024-02-19T15:25:10.557886+0000","last_undegraded":"2024-02-19T15:25:10.557886+0000","last_fullsized":"2024-02-19T15:25:10.557886+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:34:55.231867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558130+0000","last_change":"2024-02-19T15:25:08.572975+0000","last_active":"2024-02-19T15:25:10.558130+0000","last_peered":"2024-02-19T15:25:10.558130+0000","last_clean":"2024-02-19T15:25:10.558130+0000","last_became_active":"2024-02-19T15:25:08.572475+0000","last_became_peered":"2024-02-19T15:25:08.572475+0000","last_unstale":"2024-02-19T15:25:10.558130+0000","last_undegraded":"2024-02-19T15:25:10.558130+0000","last_fullsized":"2024-02-19T15:25:10.558130+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:59:20.698245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.574151+0000","last_change":"2024-02-19T15:25:09.564302+0000","last_active":"2024-02-19T15:25:09.574151+0000","last_peered":"2024-02-19T15:25:09.574151+0000","last_clean":"2024-02-19T15:25:09.574151+0000","last_became_active":"2024-02-19T15:25:09.563400+0000","last_became_peered":"2024-02-19T15:25:09.563400+0000","last_unstale":"2024-02-19T15:25:09.574151+0000","last_undegraded":"2024-02-19T15:25:09.574151+0000","last_fullsized":"2024-02-19T15:25:09.574151+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:12:54.194862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558670+0000","last_change":"2024-02-19T15:25:08.571581+0000","last_active":"2024-02-19T15:25:10.558670+0000","last_peered":"2024-02-19T15:25:10.558670+0000","last_clean":"2024-02-19T15:25:10.558670+0000","last_became_active":"2024-02-19T15:25:08.571291+0000","last_became_peered":"2024-02-19T15:25:08.571291+0000","last_unstale":"2024-02-19T15:25:10.558670+0000","last_undegraded":"2024-02-19T15:25:10.558670+0000","last_fullsized":"2024-02-19T15:25:10.558670+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:34:19.496374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":254,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.570725+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:25:09.570725+0000","last_peered":"2024-02-19T15:25:09.570725+0000","last_clean":"2024-02-19T15:25:09.570725+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:25:09.570725+0000","last_undegraded":"2024-02-19T15:25:09.570725+0000","last_fullsized":"2024-02-19T15:25:09.570725+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509483,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":843776,"data_stored":350041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.61699999999999999,"15min":0.61799999999999999},"min":{"1min":0.42699999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.90100000000000002,"5min":1.022,"15min":1.022},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.71599999999999997,"15min":0.76400000000000001},"min":{"1min":0.47499999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83699999999999997,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.47999999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64400000000000002,"15min":0.65200000000000002},"min":{"1min":0.48499999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85099999999999998,"5min":1.171,"15min":1.171},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.65500000000000003,"15min":0.65200000000000002},"min":{"1min":0.50700000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.92300000000000004,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.628,"15min":0.63100000000000001},"min":{"1min":0.44600000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.85299999999999998,"5min":1.216,"15min":1.216},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.69699999999999995,"15min":0.68799999999999994},"min":{"1min":0.55800000000000005,"5min":0.495,"15min":0.495},"max":{"1min":1.01,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.73399999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.68999999999999995,"15min":0.67900000000000005},"min":{"1min":0.54800000000000004,"5min":0.438,"15min":0.438},"max":{"1min":0.91600000000000004,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.752,"5min":0.67100000000000004,"15min":0.66000000000000003},"min":{"1min":0.55600000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.64400000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.626,"15min":0.60699999999999998},"min":{"1min":0.50800000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.90400000000000003,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63,"15min":0.60399999999999998},"min":{"1min":0.48499999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92600000000000005,"5min":0.998,"15min":0.998},"last":0.66600000000000004}]},{"osd":5,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.63200000000000001,"15min":0.60499999999999998},"min":{"1min":0.58699999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.749},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.69499999999999995,"15min":0.67300000000000004},"min":{"1min":0.61499999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.95299999999999996,"5min":1.02,"15min":1.02},"last":0.78900000000000003}]},{"osd":7,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.68799999999999994,"15min":0.65700000000000003},"min":{"1min":0.61899999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.626},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.70899999999999996,"15min":0.67700000000000005},"min":{"1min":0.69799999999999995,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.77800000000000002}]}]},{"osd":7,"up_from":14,"seq":60129542187,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1425408,"data_stored":934633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58699999999999997,"15min":0.58599999999999997},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.82499999999999996,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67300000000000004,"15min":0.67900000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.97399999999999998,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.64200000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.621,"15min":0.59699999999999998},"min":{"1min":0.51800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86099999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.70699999999999996,"15min":0.72199999999999998},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.92800000000000005,"5min":2.968,"15min":2.968},"last":0.68000000000000005}]},{"osd":2,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.64100000000000001,"15min":0.63400000000000001},"min":{"1min":0.47499999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.83599999999999997,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69099999999999995,"15min":0.69299999999999995},"min":{"1min":0.45300000000000001,"5min":0.438,"15min":0.438},"max":{"1min":0.91700000000000004,"5min":2.508,"15min":2.508},"last":0.69699999999999995}]},{"osd":3,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.66200000000000003,"15min":0.65900000000000003},"min":{"1min":0.49299999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.747},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69999999999999996,"15min":0.69199999999999995},"min":{"1min":0.48099999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89000000000000001,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":0.624}]},{"osd":4,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.56499999999999995,"15min":0.52900000000000003},"min":{"1min":0.47399999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81699999999999995,"5min":1.03,"15min":1.03},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.374,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.879,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.45400000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.61599999999999999,"15min":0.56999999999999995},"min":{"1min":0.504,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94499999999999995,"5min":1.002,"15min":1.002},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.66400000000000003,"15min":0.63400000000000001},"min":{"1min":0.44600000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.94599999999999995,"5min":1.046,"15min":1.046},"last":0.58399999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.64000000000000001,"15min":0.60299999999999998},"min":{"1min":0.443,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.69999999999999996,"15min":0.66600000000000004},"min":{"1min":0.61399999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0349999999999999,"5min":1.103,"15min":1.103},"last":0.71499999999999997}]}]},{"osd":5,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":827392,"data_stored":343739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.67200000000000004,"15min":0.64400000000000002},"min":{"1min":0.41999999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":5.7789999999999999,"5min":5.7789999999999999,"15min":5.7789999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.155,"5min":0.70599999999999996,"15min":0.63500000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":12.359999999999999,"5min":12.359999999999999,"15min":12.359999999999999},"last":0.67300000000000004}]},{"osd":1,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":0.74099999999999999,"15min":0.67100000000000004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":10.938000000000001,"5min":10.938000000000001,"15min":10.938000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.2150000000000001,"5min":0.78500000000000003,"15min":0.71099999999999997},"min":{"1min":0.53300000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":10.531000000000001,"5min":10.531000000000001,"15min":10.531000000000001},"last":0.66500000000000004}]},{"osd":2,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.79900000000000004,"15min":0.72099999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":12.925000000000001,"5min":12.925000000000001,"15min":12.925000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.1919999999999999,"5min":0.71099999999999997,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":12.401999999999999,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.60399999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.9179999999999999,"5min":0.90100000000000002,"15min":0.73099999999999998},"min":{"1min":0.51300000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":26.161999999999999,"5min":26.161999999999999,"15min":26.161999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":1.952,"5min":0.91100000000000003,"15min":0.74099999999999999},"min":{"1min":0.44500000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":27.934999999999999,"5min":27.934999999999999,"15min":27.934999999999999},"last":0.64000000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.66200000000000003,"15min":0.58799999999999997},"min":{"1min":0.40100000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":9.0549999999999997,"5min":9.0549999999999997,"15min":9.0549999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.70699999999999996,"15min":0.67000000000000004},"min":{"1min":0.46200000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":6.2549999999999999,"5min":6.2549999999999999,"15min":6.2549999999999999},"last":0.628}]},{"osd":6,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":0.72099999999999997,"15min":0.66100000000000003},"min":{"1min":0.46999999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":6.5229999999999997,"5min":6.5229999999999997,"15min":6.5229999999999997},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.0920000000000001,"5min":0.79900000000000004,"15min":0.74199999999999999},"min":{"1min":0.58399999999999996,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":6.5640000000000001,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":0.77200000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":0.76200000000000001,"15min":0.68999999999999995},"min":{"1min":0.52800000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":11.986000000000001,"5min":11.986000000000001,"15min":11.986000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.1160000000000001,"5min":0.77400000000000002,"15min":0.71399999999999997},"min":{"1min":0.48899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":8.9770000000000003,"5min":8.9770000000000003,"15min":8.9770000000000003},"last":0.70099999999999996}]}]},{"osd":4,"up_from":14,"seq":60129542186,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":827392,"data_stored":343719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.35199999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.82999999999999996,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60999999999999999,"15min":0.59999999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78300000000000003,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.68899999999999995}]},{"osd":1,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.61499999999999999},"min":{"1min":0.437,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.871,"5min":1.081,"15min":1.081},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64300000000000002,"15min":0.63300000000000001},"min":{"1min":0.52200000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86199999999999999,"5min":1.008,"15min":1.008},"last":0.46800000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66100000000000003,"15min":0.66300000000000003},"min":{"1min":0.46300000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.88600000000000001,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67900000000000005,"15min":0.67500000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.89800000000000002,"5min":1.002,"15min":1.002},"last":0.74399999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.69599999999999995,"15min":0.67800000000000005},"min":{"1min":0.53200000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0049999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66000000000000003,"15min":0.65400000000000003},"min":{"1min":0.51100000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.91200000000000003,"5min":1.153,"15min":1.153},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.625,"15min":0.60299999999999998},"min":{"1min":0.42499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.97099999999999997,"5min":1.042,"15min":1.042},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.69999999999999996,"15min":0.68000000000000005},"min":{"1min":0.40000000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0489999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.57299999999999995}]},{"osd":6,"last update":"Mon Feb 19 15:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.60799999999999998,"15min":0.57799999999999996},"min":{"1min":0.36699999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.96899999999999997,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.67500000000000004,"15min":0.65100000000000002},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0309999999999999,"5min":1.054,"15min":1.054},"last":0.71799999999999997}]},{"osd":7,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.69699999999999995,"15min":0.66900000000000004},"min":{"1min":0.41299999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.755,"15min":0.73299999999999998},"min":{"1min":0.50600000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.0600000000000001,"5min":1.24,"15min":1.24},"last":0.78900000000000003}]}]},{"osd":3,"up_from":14,"seq":60129542186,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":831488,"data_stored":343761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.435,"15min":0.44900000000000001},"min":{"1min":0.252,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.54800000000000004,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.58699999999999997,"15min":0.66100000000000003},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.59399999999999997,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.39500000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.42899999999999999,"15min":0.441},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.58099999999999996,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.65300000000000002,"15min":0.72799999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.629,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.54500000000000004}]},{"osd":2,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.58799999999999997,"15min":0.61799999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.747,"5min":1.008,"15min":1.008},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.56000000000000005,"15min":0.59199999999999997},"min":{"1min":0.36399999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61499999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.52100000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.68899999999999995,"15min":0.67400000000000004},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.89500000000000002,"5min":1.004,"15min":1.004},"last":0.745},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.82199999999999995,"15min":0.86799999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90300000000000002,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.78300000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.82099999999999995,"15min":0.86099999999999999},"min":{"1min":0.50800000000000001,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.82099999999999995,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.747,"15min":0.749},"min":{"1min":0.53500000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.97499999999999998,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.68899999999999995}]},{"osd":6,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.85499999999999998,"15min":0.90000000000000002},"min":{"1min":0.55300000000000005,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.90800000000000003,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.79600000000000004,"15min":0.81699999999999995},"min":{"1min":0.51800000000000002,"5min":0.502,"15min":0.502},"max":{"1min":1.0209999999999999,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.71299999999999997}]},{"osd":7,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.77100000000000002,"15min":0.78200000000000003},"min":{"1min":0.57599999999999996,"5min":0.504,"15min":0.504},"max":{"1min":0.88700000000000001,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.878,"15min":0.93200000000000005},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.95899999999999996,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.70399999999999996}]}]},{"osd":2,"up_from":14,"seq":60129542187,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32000,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339840,"statfs":{"total":96636764160,"available":96603996160,"internally_reserved":0,"allocated":835584,"data_stored":345261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.73199999999999998,"15min":0.71799999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.63600000000000001,"5min":15.42,"15min":15.42},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.81799999999999995,"15min":0.92000000000000004},"min":{"1min":0.27600000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.79700000000000004,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.49099999999999999}]},{"osd":1,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.86299999999999999,"15min":0.93300000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.97299999999999998,"5min":9.5269999999999992,"15min":9.5269999999999992},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":1.05,"15min":1.042},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83399999999999996,"5min":22.954999999999998,"15min":22.954999999999998},"last":0.46100000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.73799999999999999,"15min":0.80600000000000005},"min":{"1min":0.37,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76200000000000001,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.80900000000000005,"15min":0.84499999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.92000000000000004,"5min":7.734,"15min":7.734},"last":0.44400000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.95599999999999996,"15min":0.91700000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.90100000000000002,"5min":15.369,"15min":15.369},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.871,"15min":0.90700000000000003},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.95199999999999996,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.53800000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.90500000000000003,"15min":0.90000000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.135,"5min":7.6520000000000001,"15min":7.6520000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":1.0269999999999999,"15min":0.98799999999999999},"min":{"1min":0.59599999999999997,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.871,"5min":15.391,"15min":15.391},"last":0.60199999999999998}]},{"osd":6,"last update":"Mon Feb 19 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":1.0049999999999999,"15min":0.95999999999999996},"min":{"1min":0.55200000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.226,"5min":15.337,"15min":15.337},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.93600000000000005,"15min":0.97799999999999998},"min":{"1min":0.53800000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.1539999999999999,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.58499999999999996}]},{"osd":7,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.95499999999999996,"15min":0.997},"min":{"1min":0.58099999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.081,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.626},{"interface":"front","average":{"1min":0.752,"5min":0.96199999999999997,"15min":0.96799999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.171,"5min":7.6929999999999996,"15min":7.6929999999999996},"last":0.51400000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":856064,"data_stored":352177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.40500000000000003,"15min":0.42799999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54400000000000004,"5min":0.747,"15min":0.747},"last":0.443},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.45400000000000001,"15min":0.47199999999999998},"min":{"1min":0.26700000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.66200000000000003,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.41399999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.434,"15min":0.44700000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.59599999999999997,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.50700000000000001,"15min":0.53300000000000003},"min":{"1min":0.30599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.75,"5min":1.071,"15min":1.071},"last":0.5}]},{"osd":3,"last update":"Mon Feb 19 15:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.60999999999999999,"15min":0.64000000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.69999999999999996,"5min":1.365,"15min":1.365},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.60299999999999998,"15min":0.64000000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.75600000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.54100000000000004}]},{"osd":4,"last update":"Mon Feb 19 15:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63200000000000001,"15min":0.60599999999999998},"min":{"1min":0.48699999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.96799999999999997,"5min":1.127,"15min":1.127},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68400000000000005,"15min":0.67700000000000005},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.98899999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.88100000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70099999999999996,"15min":0.69499999999999995},"min":{"1min":0.503,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.94599999999999995,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.69199999999999995,"15min":0.68100000000000005},"min":{"1min":0.45600000000000002,"5min":0.443,"15min":0.443},"max":{"1min":1.0860000000000001,"5min":1.107,"15min":1.107},"last":0.81599999999999995}]},{"osd":6,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73199999999999998,"15min":0.72899999999999998},"min":{"1min":0.57799999999999996,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0449999999999999,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.74399999999999999,"15min":0.746},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0669999999999999,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.97199999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.77700000000000002,"15min":0.78900000000000003},"min":{"1min":0.51500000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.101,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.751,"15min":0.74299999999999999},"min":{"1min":0.52400000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.032,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.92400000000000004}]}]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36728,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335112,"statfs":{"total":96636764160,"available":96599154688,"internally_reserved":0,"allocated":1417216,"data_stored":934017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.56499999999999995,"15min":0.64700000000000002},"min":{"1min":0.23699999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54000000000000004,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.442,"5min":0.47299999999999998,"15min":0.48699999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.746,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.40799999999999997}]},{"osd":2,"last update":"Mon Feb 19 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.71599999999999997,"15min":0.80400000000000005},"min":{"1min":0.41699999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.69699999999999995,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.67900000000000005,"15min":0.76600000000000001},"min":{"1min":0.40100000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.60799999999999998,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.45600000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.626,"15min":0.69799999999999995},"min":{"1min":0.33200000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.70199999999999996,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.57699999999999996,"15min":0.62},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.66600000000000004,"5min":1.345,"15min":1.345},"last":0.56599999999999995}]},{"osd":4,"last update":"Mon Feb 19 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.73699999999999999,"15min":0.73999999999999999},"min":{"1min":0.55800000000000005,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.88800000000000001,"5min":1.095,"15min":1.095},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.875,"15min":0.93500000000000005},"min":{"1min":0.61599999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.93100000000000005,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.64300000000000002}]},{"osd":5,"last update":"Mon Feb 19 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.73899999999999999,"15min":0.751},"min":{"1min":0.51700000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.877,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.626},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.80600000000000005,"15min":0.86399999999999999},"min":{"1min":0.52000000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.86299999999999999,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.52900000000000003}]},{"osd":6,"last update":"Mon Feb 19 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.85499999999999998,"15min":0.92700000000000005},"min":{"1min":0.54400000000000004,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.89800000000000002,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.745,"15min":0.75900000000000001},"min":{"1min":0.53500000000000003,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.92200000000000004,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.61099999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.88900000000000001,"15min":0.95499999999999996},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.94999999999999996,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.88100000000000001,"15min":0.95699999999999996},"min":{"1min":0.53300000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.89400000000000002,"5min":4.915,"15min":4.915},"last":0.55900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:25:13.688 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:25:13.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.869+0000 7f274be00640 1 Processor -- start 2024-02-19T15:25:13.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.869+0000 7f274be00640 1 -- start start 2024-02-19T15:25:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.869+0000 7f274be00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f274c1585d0 0x7f274c0adfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.869+0000 7f274be00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274c161aa0 con 0x7f274c0b57b0 2024-02-19T15:25:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.869+0000 7f274be00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274c15f310 con 0x7f274c1581f0 2024-02-19T15:25:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.869+0000 7f274be00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f274c162400 con 0x7f274c1585d0 2024-02-19T15:25:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.869+0000 7f2742000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f274c0b57b0 0x7f274c0ad8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:56140/0 (socket says 172.21.15.47:56140) 2024-02-19T15:25:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.869+0000 7f2742000640 1 -- 172.21.15.47:0/1127381264 learned_addr learned my addr 172.21.15.47:0/1127381264 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.869+0000 7f274b400640 1 --2- 172.21.15.47:0/1127381264 >> v2:172.21.15.116:3300/0 conn(0x7f274c1585d0 0x7f274c0adfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.869+0000 7f274a000640 1 -- 172.21.15.47:0/1127381264 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2221717653 0 0) 0x7f274c161aa0 con 0x7f274c0b57b0 2024-02-19T15:25:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.869+0000 7f274a000640 1 -- 172.21.15.47:0/1127381264 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2730003530 con 0x7f274c0b57b0 2024-02-19T15:25:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.873+0000 7f274b400640 1 -- 172.21.15.47:0/1127381264 >> v1:172.21.15.116:6789/0 conn(0x7f274c1581f0 legacy=0x7f274c0b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.873+0000 7f274a000640 1 -- 172.21.15.47:0/1127381264 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3629655112 0 0) 0x7f2730003530 con 0x7f274c0b57b0 2024-02-19T15:25:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.873+0000 7f274b400640 1 -- 172.21.15.47:0/1127381264 >> v1:172.21.15.47:6789/0 conn(0x7f274c0b57b0 legacy=0x7f274c0ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.873+0000 7f274b400640 1 -- 172.21.15.47:0/1127381264 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274c162820 con 0x7f274c1585d0 2024-02-19T15:25:13.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.873+0000 7f274b400640 1 --2- 172.21.15.47:0/1127381264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f274c1585d0 0x7f274c0adfc0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2744009d90 tx=0x7f274401c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=538582857663a91 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:13.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.873+0000 7f274a000640 1 -- 172.21.15.47:0/1127381264 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2734001680 con 0x7f274c1585d0 2024-02-19T15:25:13.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.873+0000 7f274a000640 1 -- 172.21.15.47:0/1127381264 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27440a5530 con 0x7f274c1585d0 2024-02-19T15:25:13.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.873+0000 7f274a000640 1 -- 172.21.15.47:0/1127381264 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27440a58c0 con 0x7f274c1585d0 2024-02-19T15:25:13.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.873+0000 7f274be00640 1 -- 172.21.15.47:0/1127381264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f274c1585d0 msgr2=0x7f274c0adfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:13.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.873+0000 7f274be00640 1 --2- 172.21.15.47:0/1127381264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f274c1585d0 0x7f274c0adfc0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2744009d90 tx=0x7f274401c800 comp rx=0 tx=0).stop 2024-02-19T15:25:13.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f274be00640 1 -- 172.21.15.47:0/1127381264 shutdown_connections 2024-02-19T15:25:13.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f274be00640 1 --2- 172.21.15.47:0/1127381264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f274c1585d0 0x7f274c0adfc0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f274be00640 1 -- 172.21.15.47:0/1127381264 >> 172.21.15.47:0/1127381264 conn(0x7f274c0b24f0 msgr2=0x7f274c0b2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:13.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f274be00640 1 -- 172.21.15.47:0/1127381264 shutdown_connections 2024-02-19T15:25:13.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f274be00640 1 -- 172.21.15.47:0/1127381264 wait complete. 2024-02-19T15:25:13.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f274be00640 1 Processor -- start 2024-02-19T15:25:13.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f274be00640 1 -- start start 2024-02-19T15:25:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f274be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f274c0b57b0 0x7f274c0cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f274be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f274c1581f0 0x7f274c154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f274be00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f274c1585d0 0x7f274c157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f274be00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274c154fc0 con 0x7f274c1581f0 2024-02-19T15:25:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f274be00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274c157ab0 con 0x7f274c0b57b0 2024-02-19T15:25:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f274be00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f274c14fc40 con 0x7f274c1585d0 2024-02-19T15:25:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f2742000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f274c1581f0 0x7f274c154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f2742000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f274c1581f0 0x7f274c154a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55030/0 (socket says 172.21.15.47:55030) 2024-02-19T15:25:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f2742000640 1 -- 172.21.15.47:0/1428743619 learned_addr learned my addr 172.21.15.47:0/1428743619 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f274aa00640 1 --2- 172.21.15.47:0/1428743619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f274c0b57b0 0x7f274c0cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f274b400640 1 --2- 172.21.15.47:0/1428743619 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f274c1585d0 0x7f274c157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f2742000640 1 -- 172.21.15.47:0/1428743619 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f274c1585d0 msgr2=0x7f274c157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f2742000640 1 --2- 172.21.15.47:0/1428743619 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f274c1585d0 0x7f274c157570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f2742000640 1 -- 172.21.15.47:0/1428743619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f274c0b57b0 msgr2=0x7f274c0cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f2742000640 1 --2- 172.21.15.47:0/1428743619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f274c0b57b0 0x7f274c0cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f2742000640 1 -- 172.21.15.47:0/1428743619 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2744009a70 con 0x7f274c1581f0 2024-02-19T15:25:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f274b400640 1 --2- 172.21.15.47:0/1428743619 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f274c1585d0 0x7f274c157570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:25:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f2742000640 1 --2- 172.21.15.47:0/1428743619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f274c1581f0 0x7f274c154a80 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f273400c7b0 tx=0x7f273400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f2743400640 1 -- 172.21.15.47:0/1428743619 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2734005ae0 con 0x7f274c1581f0 2024-02-19T15:25:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f2743400640 1 -- 172.21.15.47:0/1428743619 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2734005c40 con 0x7f274c1581f0 2024-02-19T15:25:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.877+0000 7f2743400640 1 -- 172.21.15.47:0/1428743619 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2734007070 con 0x7f274c1581f0 2024-02-19T15:25:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.881+0000 7f274be00640 1 -- 172.21.15.47:0/1428743619 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f274c14feb0 con 0x7f274c1581f0 2024-02-19T15:25:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.881+0000 7f274be00640 1 -- 172.21.15.47:0/1428743619 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f274c150470 con 0x7f274c1581f0 2024-02-19T15:25:13.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.881+0000 7f274be00640 1 -- 172.21.15.47:0/1428743619 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f274c150660 con 0x7f274c1581f0 2024-02-19T15:25:13.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.881+0000 7f2743400640 1 -- 172.21.15.47:0/1428743619 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f27340071d0 con 0x7f274c1581f0 2024-02-19T15:25:13.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.881+0000 7f2743400640 1 --2- 172.21.15.47:0/1428743619 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2728066e40 0x7f2728069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:13.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.881+0000 7f2743400640 1 -- 172.21.15.47:0/1428743619 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2734084780 con 0x7f274c1581f0 2024-02-19T15:25:13.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.881+0000 7f274aa00640 1 --2- 172.21.15.47:0/1428743619 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2728066e40 0x7f2728069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:13.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.885+0000 7f274aa00640 1 --2- 172.21.15.47:0/1428743619 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2728066e40 0x7f2728069300 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f274c0ce890 tx=0x7f273c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:13.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:13.885+0000 7f2743400640 1 -- 172.21.15.47:0/1428743619 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2734051620 con 0x7f274c1581f0 2024-02-19T15:25:14.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.045+0000 7f274be00640 1 -- 172.21.15.47:0/1428743619 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f274c160b50 con 0x7f2728066e40 2024-02-19T15:25:14.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.053+0000 7f2743400640 1 -- 172.21.15.47:0/1428743619 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289238 (secure 0 0 0) 0x7f274c160b50 con 0x7f2728066e40 2024-02-19T15:25:14.060 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:14.061 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:25:14.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.057+0000 7f274be00640 1 -- 172.21.15.47:0/1428743619 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2728066e40 msgr2=0x7f2728069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:14.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.057+0000 7f274be00640 1 --2- 172.21.15.47:0/1428743619 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2728066e40 0x7f2728069300 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f274c0ce890 tx=0x7f273c009040 comp rx=0 tx=0).stop 2024-02-19T15:25:14.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.057+0000 7f274be00640 1 -- 172.21.15.47:0/1428743619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f274c1581f0 msgr2=0x7f274c154a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:14.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.057+0000 7f274be00640 1 --2- 172.21.15.47:0/1428743619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f274c1581f0 0x7f274c154a80 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f273400c7b0 tx=0x7f273400cc70 comp rx=0 tx=0).stop 2024-02-19T15:25:14.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.057+0000 7f274be00640 1 -- 172.21.15.47:0/1428743619 shutdown_connections 2024-02-19T15:25:14.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.057+0000 7f274be00640 1 --2- 172.21.15.47:0/1428743619 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2728066e40 0x7f2728069300 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.057+0000 7f274be00640 1 --2- 172.21.15.47:0/1428743619 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f274c1585d0 0x7f274c157570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.057+0000 7f274be00640 1 --2- 172.21.15.47:0/1428743619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f274c1581f0 0x7f274c154a80 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.057+0000 7f274be00640 1 --2- 172.21.15.47:0/1428743619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f274c0b57b0 0x7f274c0cdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.057+0000 7f274be00640 1 -- 172.21.15.47:0/1428743619 >> 172.21.15.47:0/1428743619 conn(0x7f274c0b24f0 msgr2=0x7f274c0b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:14.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.057+0000 7f274be00640 1 -- 172.21.15.47:0/1428743619 shutdown_connections 2024-02-19T15:25:14.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.057+0000 7f274be00640 1 -- 172.21.15.47:0/1428743619 wait complete. 2024-02-19T15:25:14.080 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-02-19T15:25:13.191005+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":91,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":261308,"kb_used_data":7680,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754713412,"statfs":{"total":773094113280,"available":772826533888,"internally_reserved":0,"allocated":7864320,"data_stored":3947348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":259575346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.648219"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.588426+0000","last_change":"2024-02-19T15:25:09.582871+0000","last_active":"2024-02-19T15:25:09.588426+0000","last_peered":"2024-02-19T15:25:09.588426+0000","last_clean":"2024-02-19T15:25:09.588426+0000","last_became_active":"2024-02-19T15:25:09.581537+0000","last_became_peered":"2024-02-19T15:25:09.581537+0000","last_unstale":"2024-02-19T15:25:09.588426+0000","last_undegraded":"2024-02-19T15:25:09.588426+0000","last_fullsized":"2024-02-19T15:25:09.588426+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:37.897102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558530+0000","last_change":"2024-02-19T15:25:09.574801+0000","last_active":"2024-02-19T15:25:10.558530+0000","last_peered":"2024-02-19T15:25:10.558530+0000","last_clean":"2024-02-19T15:25:10.558530+0000","last_became_active":"2024-02-19T15:25:09.573339+0000","last_became_peered":"2024-02-19T15:25:09.573339+0000","last_unstale":"2024-02-19T15:25:10.558530+0000","last_undegraded":"2024-02-19T15:25:10.558530+0000","last_fullsized":"2024-02-19T15:25:10.558530+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:08:09.019535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.570326+0000","last_change":"2024-02-19T15:25:09.568275+0000","last_active":"2024-02-19T15:25:09.570326+0000","last_peered":"2024-02-19T15:25:09.570326+0000","last_clean":"2024-02-19T15:25:09.570326+0000","last_became_active":"2024-02-19T15:25:09.567978+0000","last_became_peered":"2024-02-19T15:25:09.567978+0000","last_unstale":"2024-02-19T15:25:09.570326+0000","last_undegraded":"2024-02-19T15:25:09.570326+0000","last_fullsized":"2024-02-19T15:25:09.570326+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:07:42.213615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558458+0000","last_change":"2024-02-19T15:25:09.573255+0000","last_active":"2024-02-19T15:25:10.558458+0000","last_peered":"2024-02-19T15:25:10.558458+0000","last_clean":"2024-02-19T15:25:10.558458+0000","last_became_active":"2024-02-19T15:25:09.570700+0000","last_became_peered":"2024-02-19T15:25:09.570700+0000","last_unstale":"2024-02-19T15:25:10.558458+0000","last_undegraded":"2024-02-19T15:25:10.558458+0000","last_fullsized":"2024-02-19T15:25:10.558458+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:53:35.452518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558098+0000","last_change":"2024-02-19T15:25:09.578467+0000","last_active":"2024-02-19T15:25:10.558098+0000","last_peered":"2024-02-19T15:25:10.558098+0000","last_clean":"2024-02-19T15:25:10.558098+0000","last_became_active":"2024-02-19T15:25:09.578192+0000","last_became_peered":"2024-02-19T15:25:09.578192+0000","last_unstale":"2024-02-19T15:25:10.558098+0000","last_undegraded":"2024-02-19T15:25:10.558098+0000","last_fullsized":"2024-02-19T15:25:10.558098+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:04:29.221838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558337+0000","last_change":"2024-02-19T15:25:09.573124+0000","last_active":"2024-02-19T15:25:10.558337+0000","last_peered":"2024-02-19T15:25:10.558337+0000","last_clean":"2024-02-19T15:25:10.558337+0000","last_became_active":"2024-02-19T15:25:09.570133+0000","last_became_peered":"2024-02-19T15:25:09.570133+0000","last_unstale":"2024-02-19T15:25:10.558337+0000","last_undegraded":"2024-02-19T15:25:10.558337+0000","last_fullsized":"2024-02-19T15:25:10.558337+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:28:21.379442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.583168+0000","last_change":"2024-02-19T15:25:09.579686+0000","last_active":"2024-02-19T15:25:10.583168+0000","last_peered":"2024-02-19T15:25:10.583168+0000","last_clean":"2024-02-19T15:25:10.583168+0000","last_became_active":"2024-02-19T15:25:09.579320+0000","last_became_peered":"2024-02-19T15:25:09.579320+0000","last_unstale":"2024-02-19T15:25:10.583168+0000","last_undegraded":"2024-02-19T15:25:10.583168+0000","last_fullsized":"2024-02-19T15:25:10.583168+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:12:47.620288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.557953+0000","last_change":"2024-02-19T15:25:09.582015+0000","last_active":"2024-02-19T15:25:10.557953+0000","last_peered":"2024-02-19T15:25:10.557953+0000","last_clean":"2024-02-19T15:25:10.557953+0000","last_became_active":"2024-02-19T15:25:09.581703+0000","last_became_peered":"2024-02-19T15:25:09.581703+0000","last_unstale":"2024-02-19T15:25:10.557953+0000","last_undegraded":"2024-02-19T15:25:10.557953+0000","last_fullsized":"2024-02-19T15:25:10.557953+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:35:01.678546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558744+0000","last_change":"2024-02-19T15:25:09.576831+0000","last_active":"2024-02-19T15:25:10.558744+0000","last_peered":"2024-02-19T15:25:10.558744+0000","last_clean":"2024-02-19T15:25:10.558744+0000","last_became_active":"2024-02-19T15:25:09.576297+0000","last_became_peered":"2024-02-19T15:25:09.576297+0000","last_unstale":"2024-02-19T15:25:10.558744+0000","last_undegraded":"2024-02-19T15:25:10.558744+0000","last_fullsized":"2024-02-19T15:25:10.558744+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:02:50.293871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.583712+0000","last_change":"2024-02-19T15:25:09.578978+0000","last_active":"2024-02-19T15:25:09.583712+0000","last_peered":"2024-02-19T15:25:09.583712+0000","last_clean":"2024-02-19T15:25:09.583712+0000","last_became_active":"2024-02-19T15:25:09.576539+0000","last_became_peered":"2024-02-19T15:25:09.576539+0000","last_unstale":"2024-02-19T15:25:09.583712+0000","last_undegraded":"2024-02-19T15:25:09.583712+0000","last_fullsized":"2024-02-19T15:25:09.583712+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:09.612621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.590360+0000","last_change":"2024-02-19T15:25:09.585017+0000","last_active":"2024-02-19T15:25:09.590360+0000","last_peered":"2024-02-19T15:25:09.590360+0000","last_clean":"2024-02-19T15:25:09.590360+0000","last_became_active":"2024-02-19T15:25:09.581865+0000","last_became_peered":"2024-02-19T15:25:09.581865+0000","last_unstale":"2024-02-19T15:25:09.590360+0000","last_undegraded":"2024-02-19T15:25:09.590360+0000","last_fullsized":"2024-02-19T15:25:09.590360+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:35.960944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558594+0000","last_change":"2024-02-19T15:25:09.588372+0000","last_active":"2024-02-19T15:25:10.558594+0000","last_peered":"2024-02-19T15:25:10.558594+0000","last_clean":"2024-02-19T15:25:10.558594+0000","last_became_active":"2024-02-19T15:25:09.587556+0000","last_became_peered":"2024-02-19T15:25:09.587556+0000","last_unstale":"2024-02-19T15:25:10.558594+0000","last_undegraded":"2024-02-19T15:25:10.558594+0000","last_fullsized":"2024-02-19T15:25:10.558594+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:09.612621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.585955+0000","last_change":"2024-02-19T15:25:09.573743+0000","last_active":"2024-02-19T15:25:10.585955+0000","last_peered":"2024-02-19T15:25:10.585955+0000","last_clean":"2024-02-19T15:25:10.585955+0000","last_became_active":"2024-02-19T15:25:09.573386+0000","last_became_peered":"2024-02-19T15:25:09.573386+0000","last_unstale":"2024-02-19T15:25:10.585955+0000","last_undegraded":"2024-02-19T15:25:10.585955+0000","last_fullsized":"2024-02-19T15:25:10.585955+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:49:12.070117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.557366+0000","last_change":"2024-02-19T15:25:09.570867+0000","last_active":"2024-02-19T15:25:10.557366+0000","last_peered":"2024-02-19T15:25:10.557366+0000","last_clean":"2024-02-19T15:25:10.557366+0000","last_became_active":"2024-02-19T15:25:09.569829+0000","last_became_peered":"2024-02-19T15:25:09.569829+0000","last_unstale":"2024-02-19T15:25:10.557366+0000","last_undegraded":"2024-02-19T15:25:10.557366+0000","last_fullsized":"2024-02-19T15:25:10.557366+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:58:43.446804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.585886+0000","last_change":"2024-02-19T15:25:09.577004+0000","last_active":"2024-02-19T15:25:10.585886+0000","last_peered":"2024-02-19T15:25:10.585886+0000","last_clean":"2024-02-19T15:25:10.585886+0000","last_became_active":"2024-02-19T15:25:09.576695+0000","last_became_peered":"2024-02-19T15:25:09.576695+0000","last_unstale":"2024-02-19T15:25:10.585886+0000","last_undegraded":"2024-02-19T15:25:10.585886+0000","last_fullsized":"2024-02-19T15:25:10.585886+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:01:58.697058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.579031+0000","last_change":"2024-02-19T15:25:09.569337+0000","last_active":"2024-02-19T15:25:09.579031+0000","last_peered":"2024-02-19T15:25:09.579031+0000","last_clean":"2024-02-19T15:25:09.579031+0000","last_became_active":"2024-02-19T15:25:09.568940+0000","last_became_peered":"2024-02-19T15:25:09.568940+0000","last_unstale":"2024-02-19T15:25:09.579031+0000","last_undegraded":"2024-02-19T15:25:09.579031+0000","last_fullsized":"2024-02-19T15:25:09.579031+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:59:28.602083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558520+0000","last_change":"2024-02-19T15:25:09.576597+0000","last_active":"2024-02-19T15:25:10.558520+0000","last_peered":"2024-02-19T15:25:10.558520+0000","last_clean":"2024-02-19T15:25:10.558520+0000","last_became_active":"2024-02-19T15:25:09.576089+0000","last_became_peered":"2024-02-19T15:25:09.576089+0000","last_unstale":"2024-02-19T15:25:10.558520+0000","last_undegraded":"2024-02-19T15:25:10.558520+0000","last_fullsized":"2024-02-19T15:25:10.558520+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:34:54.835700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558807+0000","last_change":"2024-02-19T15:25:09.569269+0000","last_active":"2024-02-19T15:25:10.558807+0000","last_peered":"2024-02-19T15:25:10.558807+0000","last_clean":"2024-02-19T15:25:10.558807+0000","last_became_active":"2024-02-19T15:25:09.569007+0000","last_became_peered":"2024-02-19T15:25:09.569007+0000","last_unstale":"2024-02-19T15:25:10.558807+0000","last_undegraded":"2024-02-19T15:25:10.558807+0000","last_fullsized":"2024-02-19T15:25:10.558807+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:42:45.609838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558182+0000","last_change":"2024-02-19T15:25:09.582843+0000","last_active":"2024-02-19T15:25:10.558182+0000","last_peered":"2024-02-19T15:25:10.558182+0000","last_clean":"2024-02-19T15:25:10.558182+0000","last_became_active":"2024-02-19T15:25:09.582508+0000","last_became_peered":"2024-02-19T15:25:09.582508+0000","last_unstale":"2024-02-19T15:25:10.558182+0000","last_undegraded":"2024-02-19T15:25:10.558182+0000","last_fullsized":"2024-02-19T15:25:10.558182+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:33:28.706967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.585960+0000","last_change":"2024-02-19T15:25:09.567465+0000","last_active":"2024-02-19T15:25:10.585960+0000","last_peered":"2024-02-19T15:25:10.585960+0000","last_clean":"2024-02-19T15:25:10.585960+0000","last_became_active":"2024-02-19T15:25:09.567175+0000","last_became_peered":"2024-02-19T15:25:09.567175+0000","last_unstale":"2024-02-19T15:25:10.585960+0000","last_undegraded":"2024-02-19T15:25:10.585960+0000","last_fullsized":"2024-02-19T15:25:10.585960+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:49:41.563813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.579964+0000","last_change":"2024-02-19T15:25:09.569487+0000","last_active":"2024-02-19T15:25:09.579964+0000","last_peered":"2024-02-19T15:25:09.579964+0000","last_clean":"2024-02-19T15:25:09.579964+0000","last_became_active":"2024-02-19T15:25:09.569089+0000","last_became_peered":"2024-02-19T15:25:09.569089+0000","last_unstale":"2024-02-19T15:25:09.579964+0000","last_undegraded":"2024-02-19T15:25:09.579964+0000","last_fullsized":"2024-02-19T15:25:09.579964+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:08:06.606450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.581013+0000","last_change":"2024-02-19T15:25:09.575891+0000","last_active":"2024-02-19T15:25:09.581013+0000","last_peered":"2024-02-19T15:25:09.581013+0000","last_clean":"2024-02-19T15:25:09.581013+0000","last_became_active":"2024-02-19T15:25:09.575560+0000","last_became_peered":"2024-02-19T15:25:09.575560+0000","last_unstale":"2024-02-19T15:25:09.581013+0000","last_undegraded":"2024-02-19T15:25:09.581013+0000","last_fullsized":"2024-02-19T15:25:09.581013+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:29:15.047882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.588109+0000","last_change":"2024-02-19T15:25:09.582056+0000","last_active":"2024-02-19T15:25:09.588109+0000","last_peered":"2024-02-19T15:25:09.588109+0000","last_clean":"2024-02-19T15:25:09.588109+0000","last_became_active":"2024-02-19T15:25:09.581708+0000","last_became_peered":"2024-02-19T15:25:09.581708+0000","last_unstale":"2024-02-19T15:25:09.588109+0000","last_undegraded":"2024-02-19T15:25:09.588109+0000","last_fullsized":"2024-02-19T15:25:09.588109+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:37:30.944398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.591602+0000","last_change":"2024-02-19T15:25:09.585362+0000","last_active":"2024-02-19T15:25:09.591602+0000","last_peered":"2024-02-19T15:25:09.591602+0000","last_clean":"2024-02-19T15:25:09.591602+0000","last_became_active":"2024-02-19T15:25:09.584519+0000","last_became_peered":"2024-02-19T15:25:09.584519+0000","last_unstale":"2024-02-19T15:25:09.591602+0000","last_undegraded":"2024-02-19T15:25:09.591602+0000","last_fullsized":"2024-02-19T15:25:09.591602+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:03:58.020170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.581140+0000","last_change":"2024-02-19T15:25:09.577318+0000","last_active":"2024-02-19T15:25:09.581140+0000","last_peered":"2024-02-19T15:25:09.581140+0000","last_clean":"2024-02-19T15:25:09.581140+0000","last_became_active":"2024-02-19T15:25:09.577044+0000","last_became_peered":"2024-02-19T15:25:09.577044+0000","last_unstale":"2024-02-19T15:25:09.581140+0000","last_undegraded":"2024-02-19T15:25:09.581140+0000","last_fullsized":"2024-02-19T15:25:09.581140+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:53:35.452518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.583392+0000","last_change":"2024-02-19T15:25:09.567406+0000","last_active":"2024-02-19T15:25:10.583392+0000","last_peered":"2024-02-19T15:25:10.583392+0000","last_clean":"2024-02-19T15:25:10.583392+0000","last_became_active":"2024-02-19T15:25:09.567151+0000","last_became_peered":"2024-02-19T15:25:09.567151+0000","last_unstale":"2024-02-19T15:25:10.583392+0000","last_undegraded":"2024-02-19T15:25:10.583392+0000","last_fullsized":"2024-02-19T15:25:10.583392+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:28:11.906012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.585043+0000","last_change":"2024-02-19T15:25:09.580152+0000","last_active":"2024-02-19T15:25:09.585043+0000","last_peered":"2024-02-19T15:25:09.585043+0000","last_clean":"2024-02-19T15:25:09.585043+0000","last_became_active":"2024-02-19T15:25:09.579878+0000","last_became_peered":"2024-02-19T15:25:09.579878+0000","last_unstale":"2024-02-19T15:25:09.585043+0000","last_undegraded":"2024-02-19T15:25:09.585043+0000","last_fullsized":"2024-02-19T15:25:09.585043+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:13:11.065363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.593681+0000","last_change":"2024-02-19T15:25:09.589035+0000","last_active":"2024-02-19T15:25:09.593681+0000","last_peered":"2024-02-19T15:25:09.593681+0000","last_clean":"2024-02-19T15:25:09.593681+0000","last_became_active":"2024-02-19T15:25:09.586288+0000","last_became_peered":"2024-02-19T15:25:09.586288+0000","last_unstale":"2024-02-19T15:25:09.593681+0000","last_undegraded":"2024-02-19T15:25:09.593681+0000","last_fullsized":"2024-02-19T15:25:09.593681+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:59:31.519245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.579288+0000","last_change":"2024-02-19T15:25:09.573556+0000","last_active":"2024-02-19T15:25:09.579288+0000","last_peered":"2024-02-19T15:25:09.579288+0000","last_clean":"2024-02-19T15:25:09.579288+0000","last_became_active":"2024-02-19T15:25:09.573245+0000","last_became_peered":"2024-02-19T15:25:09.573245+0000","last_unstale":"2024-02-19T15:25:09.579288+0000","last_undegraded":"2024-02-19T15:25:09.579288+0000","last_fullsized":"2024-02-19T15:25:09.579288+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:08:22.288508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.583220+0000","last_change":"2024-02-19T15:25:09.567324+0000","last_active":"2024-02-19T15:25:10.583220+0000","last_peered":"2024-02-19T15:25:10.583220+0000","last_clean":"2024-02-19T15:25:10.583220+0000","last_became_active":"2024-02-19T15:25:09.567049+0000","last_became_peered":"2024-02-19T15:25:09.567049+0000","last_unstale":"2024-02-19T15:25:10.583220+0000","last_undegraded":"2024-02-19T15:25:10.583220+0000","last_fullsized":"2024-02-19T15:25:10.583220+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:23:16.816936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.583357+0000","last_change":"2024-02-19T15:25:09.573494+0000","last_active":"2024-02-19T15:25:10.583357+0000","last_peered":"2024-02-19T15:25:10.583357+0000","last_clean":"2024-02-19T15:25:10.583357+0000","last_became_active":"2024-02-19T15:25:09.572506+0000","last_became_peered":"2024-02-19T15:25:09.572506+0000","last_unstale":"2024-02-19T15:25:10.583357+0000","last_undegraded":"2024-02-19T15:25:10.583357+0000","last_fullsized":"2024-02-19T15:25:10.583357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:33:28.632640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558529+0000","last_change":"2024-02-19T15:25:09.575585+0000","last_active":"2024-02-19T15:25:10.558529+0000","last_peered":"2024-02-19T15:25:10.558529+0000","last_clean":"2024-02-19T15:25:10.558529+0000","last_became_active":"2024-02-19T15:25:09.575312+0000","last_became_peered":"2024-02-19T15:25:09.575312+0000","last_unstale":"2024-02-19T15:25:10.558529+0000","last_undegraded":"2024-02-19T15:25:10.558529+0000","last_fullsized":"2024-02-19T15:25:10.558529+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:24.336459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.591381+0000","last_change":"2024-02-19T15:25:09.587937+0000","last_active":"2024-02-19T15:25:09.591381+0000","last_peered":"2024-02-19T15:25:09.591381+0000","last_clean":"2024-02-19T15:25:09.591381+0000","last_became_active":"2024-02-19T15:25:09.586236+0000","last_became_peered":"2024-02-19T15:25:09.586236+0000","last_unstale":"2024-02-19T15:25:09.591381+0000","last_undegraded":"2024-02-19T15:25:09.591381+0000","last_fullsized":"2024-02-19T15:25:09.591381+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:36:37.030998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"68'4","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.593881+0000","last_change":"2024-02-19T15:25:08.580513+0000","last_active":"2024-02-19T15:25:09.593881+0000","last_peered":"2024-02-19T15:25:09.593881+0000","last_clean":"2024-02-19T15:25:09.593881+0000","last_became_active":"2024-02-19T15:25:08.580206+0000","last_became_peered":"2024-02-19T15:25:08.580206+0000","last_unstale":"2024-02-19T15:25:09.593881+0000","last_undegraded":"2024-02-19T15:25:09.593881+0000","last_fullsized":"2024-02-19T15:25:09.593881+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:12:46.613959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558913+0000","last_change":"2024-02-19T15:25:09.577238+0000","last_active":"2024-02-19T15:25:10.558913+0000","last_peered":"2024-02-19T15:25:10.558913+0000","last_clean":"2024-02-19T15:25:10.558913+0000","last_became_active":"2024-02-19T15:25:09.575907+0000","last_became_peered":"2024-02-19T15:25:09.575907+0000","last_unstale":"2024-02-19T15:25:10.558913+0000","last_undegraded":"2024-02-19T15:25:10.558913+0000","last_fullsized":"2024-02-19T15:25:10.558913+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:54:36.228347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.586249+0000","last_change":"2024-02-19T15:25:08.572351+0000","last_active":"2024-02-19T15:25:10.586249+0000","last_peered":"2024-02-19T15:25:10.586249+0000","last_clean":"2024-02-19T15:25:10.586249+0000","last_became_active":"2024-02-19T15:25:08.571637+0000","last_became_peered":"2024-02-19T15:25:08.571637+0000","last_unstale":"2024-02-19T15:25:10.586249+0000","last_undegraded":"2024-02-19T15:25:10.586249+0000","last_fullsized":"2024-02-19T15:25:10.586249+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:48:08.453725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558565+0000","last_change":"2024-02-19T15:25:09.578182+0000","last_active":"2024-02-19T15:25:10.558565+0000","last_peered":"2024-02-19T15:25:10.558565+0000","last_clean":"2024-02-19T15:25:10.558565+0000","last_became_active":"2024-02-19T15:25:09.577876+0000","last_became_peered":"2024-02-19T15:25:09.577876+0000","last_unstale":"2024-02-19T15:25:10.558565+0000","last_undegraded":"2024-02-19T15:25:10.558565+0000","last_fullsized":"2024-02-19T15:25:10.558565+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:16.154993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"68'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.594368+0000","last_change":"2024-02-19T15:25:08.566970+0000","last_active":"2024-02-19T15:25:09.594368+0000","last_peered":"2024-02-19T15:25:09.594368+0000","last_clean":"2024-02-19T15:25:09.594368+0000","last_became_active":"2024-02-19T15:25:08.566757+0000","last_became_peered":"2024-02-19T15:25:08.566757+0000","last_unstale":"2024-02-19T15:25:09.594368+0000","last_undegraded":"2024-02-19T15:25:09.594368+0000","last_fullsized":"2024-02-19T15:25:09.594368+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:39:21.663720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.588151+0000","last_change":"2024-02-19T15:25:09.579826+0000","last_active":"2024-02-19T15:25:09.588151+0000","last_peered":"2024-02-19T15:25:09.588151+0000","last_clean":"2024-02-19T15:25:09.588151+0000","last_became_active":"2024-02-19T15:25:09.578258+0000","last_became_peered":"2024-02-19T15:25:09.578258+0000","last_unstale":"2024-02-19T15:25:09.588151+0000","last_undegraded":"2024-02-19T15:25:09.588151+0000","last_fullsized":"2024-02-19T15:25:09.588151+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:33:28.632640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558675+0000","last_change":"2024-02-19T15:25:08.571511+0000","last_active":"2024-02-19T15:25:10.558675+0000","last_peered":"2024-02-19T15:25:10.558675+0000","last_clean":"2024-02-19T15:25:10.558675+0000","last_became_active":"2024-02-19T15:25:08.571146+0000","last_became_peered":"2024-02-19T15:25:08.571146+0000","last_unstale":"2024-02-19T15:25:10.558675+0000","last_undegraded":"2024-02-19T15:25:10.558675+0000","last_fullsized":"2024-02-19T15:25:10.558675+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:40:48.711503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.583425+0000","last_change":"2024-02-19T15:25:09.576499+0000","last_active":"2024-02-19T15:25:10.583425+0000","last_peered":"2024-02-19T15:25:10.583425+0000","last_clean":"2024-02-19T15:25:10.583425+0000","last_became_active":"2024-02-19T15:25:09.576162+0000","last_became_peered":"2024-02-19T15:25:09.576162+0000","last_unstale":"2024-02-19T15:25:10.583425+0000","last_undegraded":"2024-02-19T15:25:10.583425+0000","last_fullsized":"2024-02-19T15:25:10.583425+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:08:07.559774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"68'1","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.595048+0000","last_change":"2024-02-19T15:25:08.567701+0000","last_active":"2024-02-19T15:25:09.595048+0000","last_peered":"2024-02-19T15:25:09.595048+0000","last_clean":"2024-02-19T15:25:09.595048+0000","last_became_active":"2024-02-19T15:25:08.566265+0000","last_became_peered":"2024-02-19T15:25:08.566265+0000","last_unstale":"2024-02-19T15:25:09.595048+0000","last_undegraded":"2024-02-19T15:25:09.595048+0000","last_fullsized":"2024-02-19T15:25:09.595048+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:28:30.590100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.584767+0000","last_change":"2024-02-19T15:25:09.579816+0000","last_active":"2024-02-19T15:25:09.584767+0000","last_peered":"2024-02-19T15:25:09.584767+0000","last_clean":"2024-02-19T15:25:09.584767+0000","last_became_active":"2024-02-19T15:25:09.578785+0000","last_became_peered":"2024-02-19T15:25:09.578785+0000","last_unstale":"2024-02-19T15:25:09.584767+0000","last_undegraded":"2024-02-19T15:25:09.584767+0000","last_fullsized":"2024-02-19T15:25:09.584767+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:42:45.609838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558280+0000","last_change":"2024-02-19T15:25:08.575448+0000","last_active":"2024-02-19T15:25:10.558280+0000","last_peered":"2024-02-19T15:25:10.558280+0000","last_clean":"2024-02-19T15:25:10.558280+0000","last_became_active":"2024-02-19T15:25:08.573670+0000","last_became_peered":"2024-02-19T15:25:08.573670+0000","last_unstale":"2024-02-19T15:25:10.558280+0000","last_undegraded":"2024-02-19T15:25:10.558280+0000","last_fullsized":"2024-02-19T15:25:10.558280+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:29:49.691931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.557154+0000","last_change":"2024-02-19T15:25:09.572296+0000","last_active":"2024-02-19T15:25:10.557154+0000","last_peered":"2024-02-19T15:25:10.557154+0000","last_clean":"2024-02-19T15:25:10.557154+0000","last_became_active":"2024-02-19T15:25:09.571786+0000","last_became_peered":"2024-02-19T15:25:09.571786+0000","last_unstale":"2024-02-19T15:25:10.557154+0000","last_undegraded":"2024-02-19T15:25:10.557154+0000","last_fullsized":"2024-02-19T15:25:10.557154+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:57:23.401485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558375+0000","last_change":"2024-02-19T15:25:08.569630+0000","last_active":"2024-02-19T15:25:10.558375+0000","last_peered":"2024-02-19T15:25:10.558375+0000","last_clean":"2024-02-19T15:25:10.558375+0000","last_became_active":"2024-02-19T15:25:08.569354+0000","last_became_peered":"2024-02-19T15:25:08.569354+0000","last_unstale":"2024-02-19T15:25:10.558375+0000","last_undegraded":"2024-02-19T15:25:10.558375+0000","last_fullsized":"2024-02-19T15:25:10.558375+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:05:26.885441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.585908+0000","last_change":"2024-02-19T15:25:09.578689+0000","last_active":"2024-02-19T15:25:10.585908+0000","last_peered":"2024-02-19T15:25:10.585908+0000","last_clean":"2024-02-19T15:25:10.585908+0000","last_became_active":"2024-02-19T15:25:09.578400+0000","last_became_peered":"2024-02-19T15:25:09.578400+0000","last_unstale":"2024-02-19T15:25:10.585908+0000","last_undegraded":"2024-02-19T15:25:10.585908+0000","last_fullsized":"2024-02-19T15:25:10.585908+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:19:21.711082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.557148+0000","last_change":"2024-02-19T15:25:08.564608+0000","last_active":"2024-02-19T15:25:10.557148+0000","last_peered":"2024-02-19T15:25:10.557148+0000","last_clean":"2024-02-19T15:25:10.557148+0000","last_became_active":"2024-02-19T15:25:08.564341+0000","last_became_peered":"2024-02-19T15:25:08.564341+0000","last_unstale":"2024-02-19T15:25:10.557148+0000","last_undegraded":"2024-02-19T15:25:10.557148+0000","last_fullsized":"2024-02-19T15:25:10.557148+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:33:13.979253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.583432+0000","last_change":"2024-02-19T15:25:09.577318+0000","last_active":"2024-02-19T15:25:10.583432+0000","last_peered":"2024-02-19T15:25:10.583432+0000","last_clean":"2024-02-19T15:25:10.583432+0000","last_became_active":"2024-02-19T15:25:09.577023+0000","last_became_peered":"2024-02-19T15:25:09.577023+0000","last_unstale":"2024-02-19T15:25:10.583432+0000","last_undegraded":"2024-02-19T15:25:10.583432+0000","last_fullsized":"2024-02-19T15:25:10.583432+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:37.897102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.559049+0000","last_change":"2024-02-19T15:25:08.568933+0000","last_active":"2024-02-19T15:25:09.559049+0000","last_peered":"2024-02-19T15:25:09.559049+0000","last_clean":"2024-02-19T15:25:09.559049+0000","last_became_active":"2024-02-19T15:25:08.568435+0000","last_became_peered":"2024-02-19T15:25:08.568435+0000","last_unstale":"2024-02-19T15:25:09.559049+0000","last_undegraded":"2024-02-19T15:25:09.559049+0000","last_fullsized":"2024-02-19T15:25:09.559049+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:59:46.463412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.588443+0000","last_change":"2024-02-19T15:25:09.584180+0000","last_active":"2024-02-19T15:25:09.588443+0000","last_peered":"2024-02-19T15:25:09.588443+0000","last_clean":"2024-02-19T15:25:09.588443+0000","last_became_active":"2024-02-19T15:25:09.583866+0000","last_became_peered":"2024-02-19T15:25:09.583866+0000","last_unstale":"2024-02-19T15:25:09.588443+0000","last_undegraded":"2024-02-19T15:25:09.588443+0000","last_fullsized":"2024-02-19T15:25:09.588443+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:51.514855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558670+0000","last_change":"2024-02-19T15:25:08.575700+0000","last_active":"2024-02-19T15:25:10.558670+0000","last_peered":"2024-02-19T15:25:10.558670+0000","last_clean":"2024-02-19T15:25:10.558670+0000","last_became_active":"2024-02-19T15:25:08.575341+0000","last_became_peered":"2024-02-19T15:25:08.575341+0000","last_unstale":"2024-02-19T15:25:10.558670+0000","last_undegraded":"2024-02-19T15:25:10.558670+0000","last_fullsized":"2024-02-19T15:25:10.558670+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:49:36.529580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558533+0000","last_change":"2024-02-19T15:25:09.573713+0000","last_active":"2024-02-19T15:25:10.558533+0000","last_peered":"2024-02-19T15:25:10.558533+0000","last_clean":"2024-02-19T15:25:10.558533+0000","last_became_active":"2024-02-19T15:25:09.570159+0000","last_became_peered":"2024-02-19T15:25:09.570159+0000","last_unstale":"2024-02-19T15:25:10.558533+0000","last_undegraded":"2024-02-19T15:25:10.558533+0000","last_fullsized":"2024-02-19T15:25:10.558533+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:41:34.522285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"68'3","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.595036+0000","last_change":"2024-02-19T15:25:08.568892+0000","last_active":"2024-02-19T15:25:09.595036+0000","last_peered":"2024-02-19T15:25:09.595036+0000","last_clean":"2024-02-19T15:25:09.595036+0000","last_became_active":"2024-02-19T15:25:08.567779+0000","last_became_peered":"2024-02-19T15:25:08.567779+0000","last_unstale":"2024-02-19T15:25:09.595036+0000","last_undegraded":"2024-02-19T15:25:09.595036+0000","last_fullsized":"2024-02-19T15:25:09.595036+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:59.001326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558172+0000","last_change":"2024-02-19T15:25:09.584828+0000","last_active":"2024-02-19T15:25:10.558172+0000","last_peered":"2024-02-19T15:25:10.558172+0000","last_clean":"2024-02-19T15:25:10.558172+0000","last_became_active":"2024-02-19T15:25:09.584518+0000","last_became_peered":"2024-02-19T15:25:09.584518+0000","last_unstale":"2024-02-19T15:25:10.558172+0000","last_undegraded":"2024-02-19T15:25:10.558172+0000","last_fullsized":"2024-02-19T15:25:10.558172+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:09:34.830373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"68'6","reported_seq":20,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558698+0000","last_change":"2024-02-19T15:25:08.568960+0000","last_active":"2024-02-19T15:25:10.558698+0000","last_peered":"2024-02-19T15:25:10.558698+0000","last_clean":"2024-02-19T15:25:10.558698+0000","last_became_active":"2024-02-19T15:25:08.568696+0000","last_became_peered":"2024-02-19T15:25:08.568696+0000","last_unstale":"2024-02-19T15:25:10.558698+0000","last_undegraded":"2024-02-19T15:25:10.558698+0000","last_fullsized":"2024-02-19T15:25:10.558698+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:40:31.822001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.589446+0000","last_change":"2024-02-19T15:25:09.582489+0000","last_active":"2024-02-19T15:25:09.589446+0000","last_peered":"2024-02-19T15:25:09.589446+0000","last_clean":"2024-02-19T15:25:09.589446+0000","last_became_active":"2024-02-19T15:25:09.582227+0000","last_became_peered":"2024-02-19T15:25:09.582227+0000","last_unstale":"2024-02-19T15:25:09.589446+0000","last_undegraded":"2024-02-19T15:25:09.589446+0000","last_fullsized":"2024-02-19T15:25:09.589446+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:08:07.559774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558486+0000","last_change":"2024-02-19T15:25:08.575611+0000","last_active":"2024-02-19T15:25:10.558486+0000","last_peered":"2024-02-19T15:25:10.558486+0000","last_clean":"2024-02-19T15:25:10.558486+0000","last_became_active":"2024-02-19T15:25:08.575301+0000","last_became_peered":"2024-02-19T15:25:08.575301+0000","last_unstale":"2024-02-19T15:25:10.558486+0000","last_undegraded":"2024-02-19T15:25:10.558486+0000","last_fullsized":"2024-02-19T15:25:10.558486+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:39:21.663720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.583490+0000","last_change":"2024-02-19T15:25:09.573517+0000","last_active":"2024-02-19T15:25:10.583490+0000","last_peered":"2024-02-19T15:25:10.583490+0000","last_clean":"2024-02-19T15:25:10.583490+0000","last_became_active":"2024-02-19T15:25:09.573215+0000","last_became_peered":"2024-02-19T15:25:09.573215+0000","last_unstale":"2024-02-19T15:25:10.583490+0000","last_undegraded":"2024-02-19T15:25:10.583490+0000","last_fullsized":"2024-02-19T15:25:10.583490+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:51.514855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558497+0000","last_change":"2024-02-19T15:25:08.574007+0000","last_active":"2024-02-19T15:25:10.558497+0000","last_peered":"2024-02-19T15:25:10.558497+0000","last_clean":"2024-02-19T15:25:10.558497+0000","last_became_active":"2024-02-19T15:25:08.572500+0000","last_became_peered":"2024-02-19T15:25:08.572500+0000","last_unstale":"2024-02-19T15:25:10.558497+0000","last_undegraded":"2024-02-19T15:25:10.558497+0000","last_fullsized":"2024-02-19T15:25:10.558497+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:32:26.159687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.585830+0000","last_change":"2024-02-19T15:25:09.570069+0000","last_active":"2024-02-19T15:25:10.585830+0000","last_peered":"2024-02-19T15:25:10.585830+0000","last_clean":"2024-02-19T15:25:10.585830+0000","last_became_active":"2024-02-19T15:25:09.569741+0000","last_became_peered":"2024-02-19T15:25:09.569741+0000","last_unstale":"2024-02-19T15:25:10.585830+0000","last_undegraded":"2024-02-19T15:25:10.585830+0000","last_fullsized":"2024-02-19T15:25:10.585830+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:47:40.288953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.583115+0000","last_change":"2024-02-19T15:25:08.573220+0000","last_active":"2024-02-19T15:25:10.583115+0000","last_peered":"2024-02-19T15:25:10.583115+0000","last_clean":"2024-02-19T15:25:10.583115+0000","last_became_active":"2024-02-19T15:25:08.572950+0000","last_became_peered":"2024-02-19T15:25:08.572950+0000","last_unstale":"2024-02-19T15:25:10.583115+0000","last_undegraded":"2024-02-19T15:25:10.583115+0000","last_fullsized":"2024-02-19T15:25:10.583115+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:20:35.629924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.557306+0000","last_change":"2024-02-19T15:25:09.576673+0000","last_active":"2024-02-19T15:25:10.557306+0000","last_peered":"2024-02-19T15:25:10.557306+0000","last_clean":"2024-02-19T15:25:10.557306+0000","last_became_active":"2024-02-19T15:25:09.576167+0000","last_became_peered":"2024-02-19T15:25:09.576167+0000","last_unstale":"2024-02-19T15:25:10.557306+0000","last_undegraded":"2024-02-19T15:25:10.557306+0000","last_fullsized":"2024-02-19T15:25:10.557306+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:36:14.037862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"68'8","reported_seq":20,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.597105+0000","last_change":"2024-02-19T15:25:08.576263+0000","last_active":"2024-02-19T15:25:09.597105+0000","last_peered":"2024-02-19T15:25:09.597105+0000","last_clean":"2024-02-19T15:25:09.597105+0000","last_became_active":"2024-02-19T15:25:08.575964+0000","last_became_peered":"2024-02-19T15:25:08.575964+0000","last_unstale":"2024-02-19T15:25:09.597105+0000","last_undegraded":"2024-02-19T15:25:09.597105+0000","last_fullsized":"2024-02-19T15:25:09.597105+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:16:33.527348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.584731+0000","last_change":"2024-02-19T15:25:09.579619+0000","last_active":"2024-02-19T15:25:09.584731+0000","last_peered":"2024-02-19T15:25:09.584731+0000","last_clean":"2024-02-19T15:25:09.584731+0000","last_became_active":"2024-02-19T15:25:09.579345+0000","last_became_peered":"2024-02-19T15:25:09.579345+0000","last_unstale":"2024-02-19T15:25:09.584731+0000","last_undegraded":"2024-02-19T15:25:09.584731+0000","last_fullsized":"2024-02-19T15:25:09.584731+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:48:46.355752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558345+0000","last_change":"2024-02-19T15:25:08.575432+0000","last_active":"2024-02-19T15:25:10.558345+0000","last_peered":"2024-02-19T15:25:10.558345+0000","last_clean":"2024-02-19T15:25:10.558345+0000","last_became_active":"2024-02-19T15:25:08.575033+0000","last_became_peered":"2024-02-19T15:25:08.575033+0000","last_unstale":"2024-02-19T15:25:10.558345+0000","last_undegraded":"2024-02-19T15:25:10.558345+0000","last_fullsized":"2024-02-19T15:25:10.558345+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:59:17.185186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.588405+0000","last_change":"2024-02-19T15:25:09.581219+0000","last_active":"2024-02-19T15:25:09.588405+0000","last_peered":"2024-02-19T15:25:09.588405+0000","last_clean":"2024-02-19T15:25:09.588405+0000","last_became_active":"2024-02-19T15:25:09.580926+0000","last_became_peered":"2024-02-19T15:25:09.580926+0000","last_unstale":"2024-02-19T15:25:09.588405+0000","last_undegraded":"2024-02-19T15:25:09.588405+0000","last_fullsized":"2024-02-19T15:25:09.588405+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:28:11.906012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"68'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.594090+0000","last_change":"2024-02-19T15:25:08.568049+0000","last_active":"2024-02-19T15:25:09.594090+0000","last_peered":"2024-02-19T15:25:09.594090+0000","last_clean":"2024-02-19T15:25:09.594090+0000","last_became_active":"2024-02-19T15:25:08.567486+0000","last_became_peered":"2024-02-19T15:25:08.567486+0000","last_unstale":"2024-02-19T15:25:09.594090+0000","last_undegraded":"2024-02-19T15:25:09.594090+0000","last_fullsized":"2024-02-19T15:25:09.594090+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:47:28.106735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558628+0000","last_change":"2024-02-19T15:25:09.588109+0000","last_active":"2024-02-19T15:25:10.558628+0000","last_peered":"2024-02-19T15:25:10.558628+0000","last_clean":"2024-02-19T15:25:10.558628+0000","last_became_active":"2024-02-19T15:25:09.587513+0000","last_became_peered":"2024-02-19T15:25:09.587513+0000","last_unstale":"2024-02-19T15:25:10.558628+0000","last_undegraded":"2024-02-19T15:25:10.558628+0000","last_fullsized":"2024-02-19T15:25:10.558628+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:48:46.355752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558618+0000","last_change":"2024-02-19T15:25:08.576669+0000","last_active":"2024-02-19T15:25:10.558618+0000","last_peered":"2024-02-19T15:25:10.558618+0000","last_clean":"2024-02-19T15:25:10.558618+0000","last_became_active":"2024-02-19T15:25:08.576376+0000","last_became_peered":"2024-02-19T15:25:08.576376+0000","last_unstale":"2024-02-19T15:25:10.558618+0000","last_undegraded":"2024-02-19T15:25:10.558618+0000","last_fullsized":"2024-02-19T15:25:10.558618+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:42:53.328412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558316+0000","last_change":"2024-02-19T15:25:09.577601+0000","last_active":"2024-02-19T15:25:10.558316+0000","last_peered":"2024-02-19T15:25:10.558316+0000","last_clean":"2024-02-19T15:25:10.558316+0000","last_became_active":"2024-02-19T15:25:09.577339+0000","last_became_peered":"2024-02-19T15:25:09.577339+0000","last_unstale":"2024-02-19T15:25:10.558316+0000","last_undegraded":"2024-02-19T15:25:10.558316+0000","last_fullsized":"2024-02-19T15:25:10.558316+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:41:24.898113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.571722+0000","last_change":"2024-02-19T15:25:08.580632+0000","last_active":"2024-02-19T15:25:09.571722+0000","last_peered":"2024-02-19T15:25:09.571722+0000","last_clean":"2024-02-19T15:25:09.571722+0000","last_became_active":"2024-02-19T15:25:08.580293+0000","last_became_peered":"2024-02-19T15:25:08.580293+0000","last_unstale":"2024-02-19T15:25:09.571722+0000","last_undegraded":"2024-02-19T15:25:09.571722+0000","last_fullsized":"2024-02-19T15:25:09.571722+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:23:15.810607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.579311+0000","last_change":"2024-02-19T15:25:09.575733+0000","last_active":"2024-02-19T15:25:09.579311+0000","last_peered":"2024-02-19T15:25:09.579311+0000","last_clean":"2024-02-19T15:25:09.579311+0000","last_became_active":"2024-02-19T15:25:09.575443+0000","last_became_peered":"2024-02-19T15:25:09.575443+0000","last_unstale":"2024-02-19T15:25:09.579311+0000","last_undegraded":"2024-02-19T15:25:09.579311+0000","last_fullsized":"2024-02-19T15:25:09.579311+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:11:07.163392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558543+0000","last_change":"2024-02-19T15:25:08.575599+0000","last_active":"2024-02-19T15:25:10.558543+0000","last_peered":"2024-02-19T15:25:10.558543+0000","last_clean":"2024-02-19T15:25:10.558543+0000","last_became_active":"2024-02-19T15:25:08.573017+0000","last_became_peered":"2024-02-19T15:25:08.573017+0000","last_unstale":"2024-02-19T15:25:10.558543+0000","last_undegraded":"2024-02-19T15:25:10.558543+0000","last_fullsized":"2024-02-19T15:25:10.558543+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:54:29.908393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.569204+0000","last_change":"2024-02-19T15:25:09.564299+0000","last_active":"2024-02-19T15:25:09.569204+0000","last_peered":"2024-02-19T15:25:09.569204+0000","last_clean":"2024-02-19T15:25:09.569204+0000","last_became_active":"2024-02-19T15:25:09.564006+0000","last_became_peered":"2024-02-19T15:25:09.564006+0000","last_unstale":"2024-02-19T15:25:09.569204+0000","last_undegraded":"2024-02-19T15:25:09.569204+0000","last_fullsized":"2024-02-19T15:25:09.569204+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:01:31.594016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.583319+0000","last_change":"2024-02-19T15:25:08.569639+0000","last_active":"2024-02-19T15:25:10.583319+0000","last_peered":"2024-02-19T15:25:10.583319+0000","last_clean":"2024-02-19T15:25:10.583319+0000","last_became_active":"2024-02-19T15:25:08.569345+0000","last_became_peered":"2024-02-19T15:25:08.569345+0000","last_unstale":"2024-02-19T15:25:10.583319+0000","last_undegraded":"2024-02-19T15:25:10.583319+0000","last_fullsized":"2024-02-19T15:25:10.583319+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:11:58.566234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.573755+0000","last_change":"2024-02-19T15:25:09.564896+0000","last_active":"2024-02-19T15:25:09.573755+0000","last_peered":"2024-02-19T15:25:09.573755+0000","last_clean":"2024-02-19T15:25:09.573755+0000","last_became_active":"2024-02-19T15:25:09.563357+0000","last_became_peered":"2024-02-19T15:25:09.563357+0000","last_unstale":"2024-02-19T15:25:09.573755+0000","last_undegraded":"2024-02-19T15:25:09.573755+0000","last_fullsized":"2024-02-19T15:25:09.573755+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:24:16.133415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.585761+0000","last_change":"2024-02-19T15:25:08.572136+0000","last_active":"2024-02-19T15:25:10.585761+0000","last_peered":"2024-02-19T15:25:10.585761+0000","last_clean":"2024-02-19T15:25:10.585761+0000","last_became_active":"2024-02-19T15:25:08.571755+0000","last_became_peered":"2024-02-19T15:25:08.571755+0000","last_unstale":"2024-02-19T15:25:10.585761+0000","last_undegraded":"2024-02-19T15:25:10.585761+0000","last_fullsized":"2024-02-19T15:25:10.585761+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:15:58.923487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.586206+0000","last_change":"2024-02-19T15:25:09.579080+0000","last_active":"2024-02-19T15:25:10.586206+0000","last_peered":"2024-02-19T15:25:10.586206+0000","last_clean":"2024-02-19T15:25:10.586206+0000","last_became_active":"2024-02-19T15:25:09.578565+0000","last_became_peered":"2024-02-19T15:25:09.578565+0000","last_unstale":"2024-02-19T15:25:10.586206+0000","last_undegraded":"2024-02-19T15:25:10.586206+0000","last_fullsized":"2024-02-19T15:25:10.586206+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:32:36.999401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558297+0000","last_change":"2024-02-19T15:25:08.573789+0000","last_active":"2024-02-19T15:25:10.558297+0000","last_peered":"2024-02-19T15:25:10.558297+0000","last_clean":"2024-02-19T15:25:10.558297+0000","last_became_active":"2024-02-19T15:25:08.571824+0000","last_became_peered":"2024-02-19T15:25:08.571824+0000","last_unstale":"2024-02-19T15:25:10.558297+0000","last_undegraded":"2024-02-19T15:25:10.558297+0000","last_fullsized":"2024-02-19T15:25:10.558297+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:00:41.451451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.583428+0000","last_change":"2024-02-19T15:25:09.579622+0000","last_active":"2024-02-19T15:25:10.583428+0000","last_peered":"2024-02-19T15:25:10.583428+0000","last_clean":"2024-02-19T15:25:10.583428+0000","last_became_active":"2024-02-19T15:25:09.579310+0000","last_became_peered":"2024-02-19T15:25:09.579310+0000","last_unstale":"2024-02-19T15:25:10.583428+0000","last_undegraded":"2024-02-19T15:25:10.583428+0000","last_fullsized":"2024-02-19T15:25:10.583428+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:37:30.944398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"69'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.585029+0000","last_change":"2024-02-19T15:25:08.566886+0000","last_active":"2024-02-19T15:25:10.585029+0000","last_peered":"2024-02-19T15:25:10.585029+0000","last_clean":"2024-02-19T15:25:10.585029+0000","last_became_active":"2024-02-19T15:25:08.566587+0000","last_became_peered":"2024-02-19T15:25:08.566587+0000","last_unstale":"2024-02-19T15:25:10.585029+0000","last_undegraded":"2024-02-19T15:25:10.585029+0000","last_fullsized":"2024-02-19T15:25:10.585029+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:30:36.388006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.590560+0000","last_change":"2024-02-19T15:25:09.587172+0000","last_active":"2024-02-19T15:25:09.590560+0000","last_peered":"2024-02-19T15:25:09.590560+0000","last_clean":"2024-02-19T15:25:09.590560+0000","last_became_active":"2024-02-19T15:25:09.585035+0000","last_became_peered":"2024-02-19T15:25:09.585035+0000","last_unstale":"2024-02-19T15:25:09.590560+0000","last_undegraded":"2024-02-19T15:25:09.590560+0000","last_fullsized":"2024-02-19T15:25:09.590560+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:10:54.529180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558404+0000","last_change":"2024-02-19T15:25:08.572671+0000","last_active":"2024-02-19T15:25:10.558404+0000","last_peered":"2024-02-19T15:25:10.558404+0000","last_clean":"2024-02-19T15:25:10.558404+0000","last_became_active":"2024-02-19T15:25:08.572368+0000","last_became_peered":"2024-02-19T15:25:08.572368+0000","last_unstale":"2024-02-19T15:25:10.558404+0000","last_undegraded":"2024-02-19T15:25:10.558404+0000","last_fullsized":"2024-02-19T15:25:10.558404+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:24:48.668619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.590292+0000","last_change":"2024-02-19T15:25:09.584770+0000","last_active":"2024-02-19T15:25:09.590292+0000","last_peered":"2024-02-19T15:25:09.590292+0000","last_clean":"2024-02-19T15:25:09.590292+0000","last_became_active":"2024-02-19T15:25:09.581776+0000","last_became_peered":"2024-02-19T15:25:09.581776+0000","last_unstale":"2024-02-19T15:25:09.590292+0000","last_undegraded":"2024-02-19T15:25:09.590292+0000","last_fullsized":"2024-02-19T15:25:09.590292+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:42:27.619249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.586093+0000","last_change":"2024-02-19T15:25:08.573089+0000","last_active":"2024-02-19T15:25:10.586093+0000","last_peered":"2024-02-19T15:25:10.586093+0000","last_clean":"2024-02-19T15:25:10.586093+0000","last_became_active":"2024-02-19T15:25:08.571307+0000","last_became_peered":"2024-02-19T15:25:08.571307+0000","last_unstale":"2024-02-19T15:25:10.586093+0000","last_undegraded":"2024-02-19T15:25:10.586093+0000","last_fullsized":"2024-02-19T15:25:10.586093+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:43:01.462003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558068+0000","last_change":"2024-02-19T15:25:09.574232+0000","last_active":"2024-02-19T15:25:10.558068+0000","last_peered":"2024-02-19T15:25:10.558068+0000","last_clean":"2024-02-19T15:25:10.558068+0000","last_became_active":"2024-02-19T15:25:09.573079+0000","last_became_peered":"2024-02-19T15:25:09.573079+0000","last_unstale":"2024-02-19T15:25:10.558068+0000","last_undegraded":"2024-02-19T15:25:10.558068+0000","last_fullsized":"2024-02-19T15:25:10.558068+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:24:26.774738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558126+0000","last_change":"2024-02-19T15:25:08.565030+0000","last_active":"2024-02-19T15:25:10.558126+0000","last_peered":"2024-02-19T15:25:10.558126+0000","last_clean":"2024-02-19T15:25:10.558126+0000","last_became_active":"2024-02-19T15:25:08.564757+0000","last_became_peered":"2024-02-19T15:25:08.564757+0000","last_unstale":"2024-02-19T15:25:10.558126+0000","last_undegraded":"2024-02-19T15:25:10.558126+0000","last_fullsized":"2024-02-19T15:25:10.558126+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:01:09.241624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558620+0000","last_change":"2024-02-19T15:25:09.578218+0000","last_active":"2024-02-19T15:25:10.558620+0000","last_peered":"2024-02-19T15:25:10.558620+0000","last_clean":"2024-02-19T15:25:10.558620+0000","last_became_active":"2024-02-19T15:25:09.577914+0000","last_became_peered":"2024-02-19T15:25:09.577914+0000","last_unstale":"2024-02-19T15:25:10.558620+0000","last_undegraded":"2024-02-19T15:25:10.558620+0000","last_fullsized":"2024-02-19T15:25:10.558620+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:01:34.195995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558363+0000","last_change":"2024-02-19T15:25:08.569740+0000","last_active":"2024-02-19T15:25:10.558363+0000","last_peered":"2024-02-19T15:25:10.558363+0000","last_clean":"2024-02-19T15:25:10.558363+0000","last_became_active":"2024-02-19T15:25:08.569463+0000","last_became_peered":"2024-02-19T15:25:08.569463+0000","last_unstale":"2024-02-19T15:25:10.558363+0000","last_undegraded":"2024-02-19T15:25:10.558363+0000","last_fullsized":"2024-02-19T15:25:10.558363+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:01:24.418696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558243+0000","last_change":"2024-02-19T15:25:09.581476+0000","last_active":"2024-02-19T15:25:10.558243+0000","last_peered":"2024-02-19T15:25:10.558243+0000","last_clean":"2024-02-19T15:25:10.558243+0000","last_became_active":"2024-02-19T15:25:09.578600+0000","last_became_peered":"2024-02-19T15:25:09.578600+0000","last_unstale":"2024-02-19T15:25:10.558243+0000","last_undegraded":"2024-02-19T15:25:10.558243+0000","last_fullsized":"2024-02-19T15:25:10.558243+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:29:15.047882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.558373+0000","last_change":"2024-02-19T15:25:08.573705+0000","last_active":"2024-02-19T15:25:09.558373+0000","last_peered":"2024-02-19T15:25:09.558373+0000","last_clean":"2024-02-19T15:25:09.558373+0000","last_became_active":"2024-02-19T15:25:08.572748+0000","last_became_peered":"2024-02-19T15:25:08.572748+0000","last_unstale":"2024-02-19T15:25:09.558373+0000","last_undegraded":"2024-02-19T15:25:09.558373+0000","last_fullsized":"2024-02-19T15:25:09.558373+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:40:31.822001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.557886+0000","last_change":"2024-02-19T15:25:09.568735+0000","last_active":"2024-02-19T15:25:10.557886+0000","last_peered":"2024-02-19T15:25:10.557886+0000","last_clean":"2024-02-19T15:25:10.557886+0000","last_became_active":"2024-02-19T15:25:09.568412+0000","last_became_peered":"2024-02-19T15:25:09.568412+0000","last_unstale":"2024-02-19T15:25:10.557886+0000","last_undegraded":"2024-02-19T15:25:10.557886+0000","last_fullsized":"2024-02-19T15:25:10.557886+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:34:55.231867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558130+0000","last_change":"2024-02-19T15:25:08.572975+0000","last_active":"2024-02-19T15:25:10.558130+0000","last_peered":"2024-02-19T15:25:10.558130+0000","last_clean":"2024-02-19T15:25:10.558130+0000","last_became_active":"2024-02-19T15:25:08.572475+0000","last_became_peered":"2024-02-19T15:25:08.572475+0000","last_unstale":"2024-02-19T15:25:10.558130+0000","last_undegraded":"2024-02-19T15:25:10.558130+0000","last_fullsized":"2024-02-19T15:25:10.558130+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:59:20.698245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.574151+0000","last_change":"2024-02-19T15:25:09.564302+0000","last_active":"2024-02-19T15:25:09.574151+0000","last_peered":"2024-02-19T15:25:09.574151+0000","last_clean":"2024-02-19T15:25:09.574151+0000","last_became_active":"2024-02-19T15:25:09.563400+0000","last_became_peered":"2024-02-19T15:25:09.563400+0000","last_unstale":"2024-02-19T15:25:09.574151+0000","last_undegraded":"2024-02-19T15:25:09.574151+0000","last_fullsized":"2024-02-19T15:25:09.574151+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:08.542588+0000","last_clean_scrub_stamp":"2024-02-19T15:25:08.542588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:12:54.194862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-02-19T15:25:10.558670+0000","last_change":"2024-02-19T15:25:08.571581+0000","last_active":"2024-02-19T15:25:10.558670+0000","last_peered":"2024-02-19T15:25:10.558670+0000","last_clean":"2024-02-19T15:25:10.558670+0000","last_became_active":"2024-02-19T15:25:08.571291+0000","last_became_peered":"2024-02-19T15:25:08.571291+0000","last_unstale":"2024-02-19T15:25:10.558670+0000","last_undegraded":"2024-02-19T15:25:10.558670+0000","last_fullsized":"2024-02-19T15:25:10.558670+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:07.536258+0000","last_clean_scrub_stamp":"2024-02-19T15:25:07.536258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:34:19.496374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":254,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-02-19T15:25:09.570725+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:25:09.570725+0000","last_peered":"2024-02-19T15:25:09.570725+0000","last_clean":"2024-02-19T15:25:09.570725+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:25:09.570725+0000","last_undegraded":"2024-02-19T15:25:09.570725+0000","last_fullsized":"2024-02-19T15:25:09.570725+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509483,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":843776,"data_stored":350041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.61699999999999999,"15min":0.61799999999999999},"min":{"1min":0.42699999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.90100000000000002,"5min":1.022,"15min":1.022},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.71599999999999997,"15min":0.76400000000000001},"min":{"1min":0.47499999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83699999999999997,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.47999999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64400000000000002,"15min":0.65200000000000002},"min":{"1min":0.48499999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85099999999999998,"5min":1.171,"15min":1.171},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.65500000000000003,"15min":0.65200000000000002},"min":{"1min":0.50700000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.92300000000000004,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.628,"15min":0.63100000000000001},"min":{"1min":0.44600000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.85299999999999998,"5min":1.216,"15min":1.216},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.69699999999999995,"15min":0.68799999999999994},"min":{"1min":0.55800000000000005,"5min":0.495,"15min":0.495},"max":{"1min":1.01,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.73399999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.68999999999999995,"15min":0.67900000000000005},"min":{"1min":0.54800000000000004,"5min":0.438,"15min":0.438},"max":{"1min":0.91600000000000004,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.752,"5min":0.67100000000000004,"15min":0.66000000000000003},"min":{"1min":0.55600000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.64400000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.626,"15min":0.60699999999999998},"min":{"1min":0.50800000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.90400000000000003,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63,"15min":0.60399999999999998},"min":{"1min":0.48499999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92600000000000005,"5min":0.998,"15min":0.998},"last":0.66600000000000004}]},{"osd":5,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.63200000000000001,"15min":0.60499999999999998},"min":{"1min":0.58699999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.749},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.69499999999999995,"15min":0.67300000000000004},"min":{"1min":0.61499999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.95299999999999996,"5min":1.02,"15min":1.02},"last":0.78900000000000003}]},{"osd":7,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.68799999999999994,"15min":0.65700000000000003},"min":{"1min":0.61899999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.626},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.70899999999999996,"15min":0.67700000000000005},"min":{"1min":0.69799999999999995,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.77800000000000002}]}]},{"osd":7,"up_from":14,"seq":60129542187,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1425408,"data_stored":934633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58699999999999997,"15min":0.58599999999999997},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.82499999999999996,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67300000000000004,"15min":0.67900000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.97399999999999998,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.64200000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.621,"15min":0.59699999999999998},"min":{"1min":0.51800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86099999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.70699999999999996,"15min":0.72199999999999998},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.92800000000000005,"5min":2.968,"15min":2.968},"last":0.68000000000000005}]},{"osd":2,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.64100000000000001,"15min":0.63400000000000001},"min":{"1min":0.47499999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.83599999999999997,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69099999999999995,"15min":0.69299999999999995},"min":{"1min":0.45300000000000001,"5min":0.438,"15min":0.438},"max":{"1min":0.91700000000000004,"5min":2.508,"15min":2.508},"last":0.69699999999999995}]},{"osd":3,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.66200000000000003,"15min":0.65900000000000003},"min":{"1min":0.49299999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.747},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69999999999999996,"15min":0.69199999999999995},"min":{"1min":0.48099999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89000000000000001,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":0.624}]},{"osd":4,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.56499999999999995,"15min":0.52900000000000003},"min":{"1min":0.47399999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81699999999999995,"5min":1.03,"15min":1.03},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.374,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.879,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.45400000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.61599999999999999,"15min":0.56999999999999995},"min":{"1min":0.504,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94499999999999995,"5min":1.002,"15min":1.002},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.66400000000000003,"15min":0.63400000000000001},"min":{"1min":0.44600000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.94599999999999995,"5min":1.046,"15min":1.046},"last":0.58399999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.64000000000000001,"15min":0.60299999999999998},"min":{"1min":0.443,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.69999999999999996,"15min":0.66600000000000004},"min":{"1min":0.61399999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0349999999999999,"5min":1.103,"15min":1.103},"last":0.71499999999999997}]}]},{"osd":5,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":827392,"data_stored":343739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.67200000000000004,"15min":0.64400000000000002},"min":{"1min":0.41999999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":5.7789999999999999,"5min":5.7789999999999999,"15min":5.7789999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.155,"5min":0.70599999999999996,"15min":0.63500000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":12.359999999999999,"5min":12.359999999999999,"15min":12.359999999999999},"last":0.67300000000000004}]},{"osd":1,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":0.74099999999999999,"15min":0.67100000000000004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":10.938000000000001,"5min":10.938000000000001,"15min":10.938000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.2150000000000001,"5min":0.78500000000000003,"15min":0.71099999999999997},"min":{"1min":0.53300000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":10.531000000000001,"5min":10.531000000000001,"15min":10.531000000000001},"last":0.66500000000000004}]},{"osd":2,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.79900000000000004,"15min":0.72099999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":12.925000000000001,"5min":12.925000000000001,"15min":12.925000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.1919999999999999,"5min":0.71099999999999997,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":12.401999999999999,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.60399999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.9179999999999999,"5min":0.90100000000000002,"15min":0.73099999999999998},"min":{"1min":0.51300000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":26.161999999999999,"5min":26.161999999999999,"15min":26.161999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":1.952,"5min":0.91100000000000003,"15min":0.74099999999999999},"min":{"1min":0.44500000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":27.934999999999999,"5min":27.934999999999999,"15min":27.934999999999999},"last":0.64000000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.66200000000000003,"15min":0.58799999999999997},"min":{"1min":0.40100000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":9.0549999999999997,"5min":9.0549999999999997,"15min":9.0549999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.70699999999999996,"15min":0.67000000000000004},"min":{"1min":0.46200000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":6.2549999999999999,"5min":6.2549999999999999,"15min":6.2549999999999999},"last":0.628}]},{"osd":6,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":0.72099999999999997,"15min":0.66100000000000003},"min":{"1min":0.46999999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":6.5229999999999997,"5min":6.5229999999999997,"15min":6.5229999999999997},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.0920000000000001,"5min":0.79900000000000004,"15min":0.74199999999999999},"min":{"1min":0.58399999999999996,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":6.5640000000000001,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":0.77200000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":0.76200000000000001,"15min":0.68999999999999995},"min":{"1min":0.52800000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":11.986000000000001,"5min":11.986000000000001,"15min":11.986000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.1160000000000001,"5min":0.77400000000000002,"15min":0.71399999999999997},"min":{"1min":0.48899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":8.9770000000000003,"5min":8.9770000000000003,"15min":8.9770000000000003},"last":0.70099999999999996}]}]},{"osd":4,"up_from":14,"seq":60129542186,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":827392,"data_stored":343719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.35199999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.82999999999999996,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60999999999999999,"15min":0.59999999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78300000000000003,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.68899999999999995}]},{"osd":1,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.61499999999999999},"min":{"1min":0.437,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.871,"5min":1.081,"15min":1.081},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64300000000000002,"15min":0.63300000000000001},"min":{"1min":0.52200000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86199999999999999,"5min":1.008,"15min":1.008},"last":0.46800000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66100000000000003,"15min":0.66300000000000003},"min":{"1min":0.46300000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.88600000000000001,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67900000000000005,"15min":0.67500000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.89800000000000002,"5min":1.002,"15min":1.002},"last":0.74399999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.69599999999999995,"15min":0.67800000000000005},"min":{"1min":0.53200000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0049999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66000000000000003,"15min":0.65400000000000003},"min":{"1min":0.51100000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.91200000000000003,"5min":1.153,"15min":1.153},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.625,"15min":0.60299999999999998},"min":{"1min":0.42499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.97099999999999997,"5min":1.042,"15min":1.042},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.69999999999999996,"15min":0.68000000000000005},"min":{"1min":0.40000000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0489999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.57299999999999995}]},{"osd":6,"last update":"Mon Feb 19 15:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.60799999999999998,"15min":0.57799999999999996},"min":{"1min":0.36699999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.96899999999999997,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.67500000000000004,"15min":0.65100000000000002},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0309999999999999,"5min":1.054,"15min":1.054},"last":0.71799999999999997}]},{"osd":7,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.69699999999999995,"15min":0.66900000000000004},"min":{"1min":0.41299999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.755,"15min":0.73299999999999998},"min":{"1min":0.50600000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.0600000000000001,"5min":1.24,"15min":1.24},"last":0.78900000000000003}]}]},{"osd":3,"up_from":14,"seq":60129542186,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":831488,"data_stored":343761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.435,"15min":0.44900000000000001},"min":{"1min":0.252,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.54800000000000004,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.58699999999999997,"15min":0.66100000000000003},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.59399999999999997,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.39500000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.42899999999999999,"15min":0.441},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.58099999999999996,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.65300000000000002,"15min":0.72799999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.629,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.54500000000000004}]},{"osd":2,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.58799999999999997,"15min":0.61799999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.747,"5min":1.008,"15min":1.008},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.56000000000000005,"15min":0.59199999999999997},"min":{"1min":0.36399999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61499999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.52100000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.68899999999999995,"15min":0.67400000000000004},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.89500000000000002,"5min":1.004,"15min":1.004},"last":0.745},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.82199999999999995,"15min":0.86799999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90300000000000002,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.78300000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.82099999999999995,"15min":0.86099999999999999},"min":{"1min":0.50800000000000001,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.82099999999999995,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.747,"15min":0.749},"min":{"1min":0.53500000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.97499999999999998,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.68899999999999995}]},{"osd":6,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.85499999999999998,"15min":0.90000000000000002},"min":{"1min":0.55300000000000005,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.90800000000000003,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.79600000000000004,"15min":0.81699999999999995},"min":{"1min":0.51800000000000002,"5min":0.502,"15min":0.502},"max":{"1min":1.0209999999999999,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.71299999999999997}]},{"osd":7,"last update":"Mon Feb 19 15:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.77100000000000002,"15min":0.78200000000000003},"min":{"1min":0.57599999999999996,"5min":0.504,"15min":0.504},"max":{"1min":0.88700000000000001,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.878,"15min":0.93200000000000005},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.95899999999999996,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.70399999999999996}]}]},{"osd":2,"up_from":14,"seq":60129542187,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32000,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339840,"statfs":{"total":96636764160,"available":96603996160,"internally_reserved":0,"allocated":835584,"data_stored":345261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.73199999999999998,"15min":0.71799999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.63600000000000001,"5min":15.42,"15min":15.42},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.81799999999999995,"15min":0.92000000000000004},"min":{"1min":0.27600000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.79700000000000004,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.49099999999999999}]},{"osd":1,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.86299999999999999,"15min":0.93300000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.97299999999999998,"5min":9.5269999999999992,"15min":9.5269999999999992},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":1.05,"15min":1.042},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83399999999999996,"5min":22.954999999999998,"15min":22.954999999999998},"last":0.46100000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.73799999999999999,"15min":0.80600000000000005},"min":{"1min":0.37,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76200000000000001,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.80900000000000005,"15min":0.84499999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.92000000000000004,"5min":7.734,"15min":7.734},"last":0.44400000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.95599999999999996,"15min":0.91700000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.90100000000000002,"5min":15.369,"15min":15.369},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.871,"15min":0.90700000000000003},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.95199999999999996,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.53800000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.90500000000000003,"15min":0.90000000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.135,"5min":7.6520000000000001,"15min":7.6520000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":1.0269999999999999,"15min":0.98799999999999999},"min":{"1min":0.59599999999999997,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.871,"5min":15.391,"15min":15.391},"last":0.60199999999999998}]},{"osd":6,"last update":"Mon Feb 19 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":1.0049999999999999,"15min":0.95999999999999996},"min":{"1min":0.55200000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.226,"5min":15.337,"15min":15.337},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.93600000000000005,"15min":0.97799999999999998},"min":{"1min":0.53800000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.1539999999999999,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.58499999999999996}]},{"osd":7,"last update":"Mon Feb 19 15:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.95499999999999996,"15min":0.997},"min":{"1min":0.58099999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.081,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.626},{"interface":"front","average":{"1min":0.752,"5min":0.96199999999999997,"15min":0.96799999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.171,"5min":7.6929999999999996,"15min":7.6929999999999996},"last":0.51400000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":856064,"data_stored":352177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.40500000000000003,"15min":0.42799999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54400000000000004,"5min":0.747,"15min":0.747},"last":0.443},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.45400000000000001,"15min":0.47199999999999998},"min":{"1min":0.26700000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.66200000000000003,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.41399999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.434,"15min":0.44700000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.59599999999999997,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.50700000000000001,"15min":0.53300000000000003},"min":{"1min":0.30599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.75,"5min":1.071,"15min":1.071},"last":0.5}]},{"osd":3,"last update":"Mon Feb 19 15:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.60999999999999999,"15min":0.64000000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.69999999999999996,"5min":1.365,"15min":1.365},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.60299999999999998,"15min":0.64000000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.75600000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.54100000000000004}]},{"osd":4,"last update":"Mon Feb 19 15:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63200000000000001,"15min":0.60599999999999998},"min":{"1min":0.48699999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.96799999999999997,"5min":1.127,"15min":1.127},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68400000000000005,"15min":0.67700000000000005},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.98899999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.88100000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70099999999999996,"15min":0.69499999999999995},"min":{"1min":0.503,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.94599999999999995,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.69199999999999995,"15min":0.68100000000000005},"min":{"1min":0.45600000000000002,"5min":0.443,"15min":0.443},"max":{"1min":1.0860000000000001,"5min":1.107,"15min":1.107},"last":0.81599999999999995}]},{"osd":6,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73199999999999998,"15min":0.72899999999999998},"min":{"1min":0.57799999999999996,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0449999999999999,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.74399999999999999,"15min":0.746},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0669999999999999,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.97199999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.77700000000000002,"15min":0.78900000000000003},"min":{"1min":0.51500000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.101,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.751,"15min":0.74299999999999999},"min":{"1min":0.52400000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.032,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.92400000000000004}]}]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36728,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335112,"statfs":{"total":96636764160,"available":96599154688,"internally_reserved":0,"allocated":1417216,"data_stored":934017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.56499999999999995,"15min":0.64700000000000002},"min":{"1min":0.23699999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54000000000000004,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.442,"5min":0.47299999999999998,"15min":0.48699999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.746,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.40799999999999997}]},{"osd":2,"last update":"Mon Feb 19 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.71599999999999997,"15min":0.80400000000000005},"min":{"1min":0.41699999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.69699999999999995,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.67900000000000005,"15min":0.76600000000000001},"min":{"1min":0.40100000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.60799999999999998,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.45600000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.626,"15min":0.69799999999999995},"min":{"1min":0.33200000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.70199999999999996,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.57699999999999996,"15min":0.62},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.66600000000000004,"5min":1.345,"15min":1.345},"last":0.56599999999999995}]},{"osd":4,"last update":"Mon Feb 19 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.73699999999999999,"15min":0.73999999999999999},"min":{"1min":0.55800000000000005,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.88800000000000001,"5min":1.095,"15min":1.095},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.875,"15min":0.93500000000000005},"min":{"1min":0.61599999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.93100000000000005,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.64300000000000002}]},{"osd":5,"last update":"Mon Feb 19 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.73899999999999999,"15min":0.751},"min":{"1min":0.51700000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.877,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.626},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.80600000000000005,"15min":0.86399999999999999},"min":{"1min":0.52000000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.86299999999999999,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.52900000000000003}]},{"osd":6,"last update":"Mon Feb 19 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.85499999999999998,"15min":0.92700000000000005},"min":{"1min":0.54400000000000004,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.89800000000000002,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.745,"15min":0.75900000000000001},"min":{"1min":0.53500000000000003,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.92200000000000004,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.61099999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.88900000000000001,"15min":0.95499999999999996},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.94999999999999996,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.88100000000000001,"15min":0.95699999999999996},"min":{"1min":0.53300000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.89400000000000002,"5min":4.915,"15min":4.915},"last":0.55900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:25:14.084 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T15:25:14.085 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-19T15:25:14.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.269+0000 7f4347c00640 1 Processor -- start 2024-02-19T15:25:14.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.269+0000 7f4347c00640 1 -- start start 2024-02-19T15:25:14.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.269+0000 7f4347c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f434015e250 0x7f43400cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:14.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.269+0000 7f4347c00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43400cff70 con 0x7f43400ca500 2024-02-19T15:25:14.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.269+0000 7f4347c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43400cfc90 con 0x7f43400d0d00 2024-02-19T15:25:14.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.269+0000 7f4347c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43400c7a10 con 0x7f434015e250 2024-02-19T15:25:14.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.269+0000 7f4345200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f434015e250 0x7f43400cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:14.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.269+0000 7f4345200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f434015e250 0x7f43400cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55044/0 (socket says 172.21.15.47:55044) 2024-02-19T15:25:14.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.269+0000 7f4345200640 1 -- 172.21.15.47:0/3583373888 learned_addr learned my addr 172.21.15.47:0/3583373888 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:14.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.269+0000 7f4345200640 1 -- 172.21.15.47:0/3583373888 >> v1:172.21.15.116:6789/0 conn(0x7f43400d0d00 legacy=0x7f4340167e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:14.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.269+0000 7f4345200640 1 -- 172.21.15.47:0/3583373888 >> v1:172.21.15.47:6789/0 conn(0x7f43400ca500 legacy=0x7f43400d05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:14.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.269+0000 7f4345200640 1 -- 172.21.15.47:0/3583373888 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43400c7cf0 con 0x7f434015e250 2024-02-19T15:25:14.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.273+0000 7f4345200640 1 --2- 172.21.15.47:0/3583373888 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f434015e250 0x7f43400cf3f0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f4330009870 tx=0x7f433001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a43269bcc2b0bb9a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:14.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.273+0000 7f433fe00640 1 -- 172.21.15.47:0/3583373888 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43300aa620 con 0x7f434015e250 2024-02-19T15:25:14.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.273+0000 7f433fe00640 1 -- 172.21.15.47:0/3583373888 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43300aa780 con 0x7f434015e250 2024-02-19T15:25:14.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.273+0000 7f433fe00640 1 -- 172.21.15.47:0/3583373888 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43300aab10 con 0x7f434015e250 2024-02-19T15:25:14.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.273+0000 7f4347c00640 1 -- 172.21.15.47:0/3583373888 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f434015e250 msgr2=0x7f43400cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:14.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.273+0000 7f4347c00640 1 --2- 172.21.15.47:0/3583373888 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f434015e250 0x7f43400cf3f0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f4330009870 tx=0x7f433001c4a0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.273+0000 7f4347c00640 1 -- 172.21.15.47:0/3583373888 shutdown_connections 2024-02-19T15:25:14.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.273+0000 7f4347c00640 1 --2- 172.21.15.47:0/3583373888 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f434015e250 0x7f43400cf3f0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.273+0000 7f4347c00640 1 -- 172.21.15.47:0/3583373888 >> 172.21.15.47:0/3583373888 conn(0x7f43400d3d90 msgr2=0x7f43400d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:14.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.273+0000 7f4347c00640 1 -- 172.21.15.47:0/3583373888 shutdown_connections 2024-02-19T15:25:14.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.273+0000 7f4347c00640 1 -- 172.21.15.47:0/3583373888 wait complete. 2024-02-19T15:25:14.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f4347c00640 1 Processor -- start 2024-02-19T15:25:14.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f4347c00640 1 -- start start 2024-02-19T15:25:14.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f4347c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43400ca500 0x7f43401545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:14.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f4347c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43400d0d00 0x7f43400b69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:14.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f4347c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f434015e250 0x7f434006b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:14.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f4347c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434015d860 con 0x7f43400ca500 2024-02-19T15:25:14.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f4347c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43401f73f0 con 0x7f434015e250 2024-02-19T15:25:14.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f4347c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43401f76d0 con 0x7f43400d0d00 2024-02-19T15:25:14.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f4345200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43400ca500 0x7f43401545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:14.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f4345200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43400ca500 0x7f43401545d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55054/0 (socket says 172.21.15.47:55054) 2024-02-19T15:25:14.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f4345200640 1 -- 172.21.15.47:0/1422763717 learned_addr learned my addr 172.21.15.47:0/1422763717 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:14.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f4345200640 1 -- 172.21.15.47:0/1422763717 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43400d0d00 msgr2=0x7f43400b69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:14.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f4345200640 1 --2- 172.21.15.47:0/1422763717 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43400d0d00 0x7f43400b69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f4345200640 1 -- 172.21.15.47:0/1422763717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f434015e250 msgr2=0x7f434006b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:14.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f4345200640 1 --2- 172.21.15.47:0/1422763717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f434015e250 0x7f434006b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f4345200640 1 -- 172.21.15.47:0/1422763717 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43300094d0 con 0x7f43400ca500 2024-02-19T15:25:14.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f4345200640 1 --2- 172.21.15.47:0/1422763717 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43400ca500 0x7f43401545d0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f43300ac040 tx=0x7f433001ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:14.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f433d600640 1 -- 172.21.15.47:0/1422763717 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43300bd3f0 con 0x7f43400ca500 2024-02-19T15:25:14.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f433d600640 1 -- 172.21.15.47:0/1422763717 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43300bd590 con 0x7f43400ca500 2024-02-19T15:25:14.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f433d600640 1 -- 172.21.15.47:0/1422763717 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43300bd940 con 0x7f43400ca500 2024-02-19T15:25:14.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f4347c00640 1 -- 172.21.15.47:0/1422763717 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43401f79b0 con 0x7f43400ca500 2024-02-19T15:25:14.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.277+0000 7f4347c00640 1 -- 172.21.15.47:0/1422763717 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43401f7f70 con 0x7f43400ca500 2024-02-19T15:25:14.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.281+0000 7f4347c00640 1 -- 172.21.15.47:0/1422763717 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4340075360 con 0x7f43400ca500 2024-02-19T15:25:14.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.281+0000 7f433d600640 1 -- 172.21.15.47:0/1422763717 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f43300b33f0 con 0x7f43400ca500 2024-02-19T15:25:14.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.281+0000 7f433d600640 1 --2- 172.21.15.47:0/1422763717 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f431c066e40 0x7f431c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:14.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.281+0000 7f433d600640 1 -- 172.21.15.47:0/1422763717 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f433011f2a0 con 0x7f43400ca500 2024-02-19T15:25:14.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.281+0000 7f4337e00640 1 --2- 172.21.15.47:0/1422763717 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f431c066e40 0x7f431c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:14.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.281+0000 7f4337e00640 1 --2- 172.21.15.47:0/1422763717 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f431c066e40 0x7f431c069300 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4328000f70 tx=0x7f43280079c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:14.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.285+0000 7f433d600640 1 -- 172.21.15.47:0/1422763717 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f43300ebff0 con 0x7f43400ca500 2024-02-19T15:25:14.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.513+0000 7f4347c00640 1 -- 172.21.15.47:0/1422763717 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f43401691e0 con 0x7f43400ca500 2024-02-19T15:25:14.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.517+0000 7f433d600640 1 -- 172.21.15.47:0/1422763717 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7f43300ef6b0 con 0x7f43400ca500 2024-02-19T15:25:14.519 INFO:teuthology.orchestra.run.smithi047.stdout:[client.0] 2024-02-19T15:25:14.519 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDjcdNlb7eRNRAAk6ZLGfjhcdlR9GoXFCd9eA== 2024-02-19T15:25:14.519 INFO:teuthology.orchestra.run.smithi047.stdout: caps mds = "allow" 2024-02-19T15:25:14.519 INFO:teuthology.orchestra.run.smithi047.stdout: caps mon = "allow r" 2024-02-19T15:25:14.519 INFO:teuthology.orchestra.run.smithi047.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-19T15:25:14.520 INFO:teuthology.orchestra.run.smithi047.stderr:updated caps for client.0 2024-02-19T15:25:14.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.517+0000 7f4347c00640 1 -- 172.21.15.47:0/1422763717 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f431c066e40 msgr2=0x7f431c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:14.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.517+0000 7f4347c00640 1 --2- 172.21.15.47:0/1422763717 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f431c066e40 0x7f431c069300 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4328000f70 tx=0x7f43280079c0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.517+0000 7f4347c00640 1 -- 172.21.15.47:0/1422763717 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43400ca500 msgr2=0x7f43401545d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:14.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.517+0000 7f4347c00640 1 --2- 172.21.15.47:0/1422763717 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43400ca500 0x7f43401545d0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f43300ac040 tx=0x7f433001ace0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.517+0000 7f4347c00640 1 -- 172.21.15.47:0/1422763717 shutdown_connections 2024-02-19T15:25:14.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.517+0000 7f4347c00640 1 --2- 172.21.15.47:0/1422763717 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f431c066e40 0x7f431c069300 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.517+0000 7f4347c00640 1 --2- 172.21.15.47:0/1422763717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f434015e250 0x7f434006b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.517+0000 7f4347c00640 1 --2- 172.21.15.47:0/1422763717 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43400d0d00 0x7f43400b69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.517+0000 7f4347c00640 1 --2- 172.21.15.47:0/1422763717 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43400ca500 0x7f43401545d0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.517+0000 7f4347c00640 1 -- 172.21.15.47:0/1422763717 >> 172.21.15.47:0/1422763717 conn(0x7f43400d3d90 msgr2=0x7f43400cf0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:14.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.517+0000 7f4347c00640 1 -- 172.21.15.47:0/1422763717 shutdown_connections 2024-02-19T15:25:14.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.517+0000 7f4347c00640 1 -- 172.21.15.47:0/1422763717 wait complete. 2024-02-19T15:25:14.540 INFO:tasks.ceph.mds.c.smithi047.stderr:2024-02-19T15:25:14.537+0000 7f1079600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T15:25:14.542 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:25:14.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.677+0000 7fb12de00640 1 Processor -- start 2024-02-19T15:25:14.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.677+0000 7fb12de00640 1 -- start start 2024-02-19T15:25:14.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.681+0000 7fb12de00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb1280ced10 0x7fb1281690a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:14.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.681+0000 7fb12de00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1280cce30 con 0x7fb1280cded0 2024-02-19T15:25:14.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.681+0000 7fb12de00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1280cd110 con 0x7fb12806d650 2024-02-19T15:25:14.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.681+0000 7fb12de00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1280cd3f0 con 0x7fb1280ced10 2024-02-19T15:25:14.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.681+0000 7fb11fe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb1280cded0 0x7fb1280ce600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:56148/0 (socket says 172.21.15.47:56148) 2024-02-19T15:25:14.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.681+0000 7fb11fe00640 1 -- 172.21.15.47:0/2906868404 learned_addr learned my addr 172.21.15.47:0/2906868404 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:14.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.681+0000 7fb127400640 1 --2- 172.21.15.47:0/2906868404 >> v2:172.21.15.116:3300/0 conn(0x7fb1280ced10 0x7fb1281690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:14.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.681+0000 7fb126000640 1 -- 172.21.15.47:0/2906868404 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865018285 0 0) 0x7fb1280cce30 con 0x7fb1280cded0 2024-02-19T15:25:14.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.681+0000 7fb126000640 1 -- 172.21.15.47:0/2906868404 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb10c003530 con 0x7fb1280cded0 2024-02-19T15:25:14.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.681+0000 7fb126000640 1 -- 172.21.15.47:0/2906868404 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2160302820 0 0) 0x7fb1280cd110 con 0x7fb12806d650 2024-02-19T15:25:14.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.681+0000 7fb126000640 1 -- 172.21.15.47:0/2906868404 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1280cce30 con 0x7fb12806d650 2024-02-19T15:25:14.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.681+0000 7fb127400640 1 -- 172.21.15.47:0/2906868404 >> v1:172.21.15.116:6789/0 conn(0x7fb12806d650 legacy=0x7fb1280d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:14.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.681+0000 7fb127400640 1 -- 172.21.15.47:0/2906868404 >> v1:172.21.15.47:6789/0 conn(0x7fb1280cded0 legacy=0x7fb1280ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:14.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.681+0000 7fb127400640 1 -- 172.21.15.47:0/2906868404 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1280cd6d0 con 0x7fb1280ced10 2024-02-19T15:25:14.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.685+0000 7fb127400640 1 --2- 172.21.15.47:0/2906868404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1280ced10 0x7fb1281690a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb120009dc0 tx=0x7fb12001c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc73156dfd3ede00 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:14.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.685+0000 7fb126000640 1 -- 172.21.15.47:0/2906868404 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1200a5970 con 0x7fb1280ced10 2024-02-19T15:25:14.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.685+0000 7fb126000640 1 -- 172.21.15.47:0/2906868404 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1200a5ad0 con 0x7fb1280ced10 2024-02-19T15:25:14.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.685+0000 7fb126000640 1 -- 172.21.15.47:0/2906868404 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1200a5e60 con 0x7fb1280ced10 2024-02-19T15:25:14.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.685+0000 7fb12de00640 1 -- 172.21.15.47:0/2906868404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1280ced10 msgr2=0x7fb1281690a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:14.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.685+0000 7fb12de00640 1 --2- 172.21.15.47:0/2906868404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1280ced10 0x7fb1281690a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb120009dc0 tx=0x7fb12001c890 comp rx=0 tx=0).stop 2024-02-19T15:25:14.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.685+0000 7fb12de00640 1 -- 172.21.15.47:0/2906868404 shutdown_connections 2024-02-19T15:25:14.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.685+0000 7fb12de00640 1 --2- 172.21.15.47:0/2906868404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1280ced10 0x7fb1281690a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.685+0000 7fb12de00640 1 -- 172.21.15.47:0/2906868404 >> 172.21.15.47:0/2906868404 conn(0x7fb1280d1bf0 msgr2=0x7fb1280d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:14.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.685+0000 7fb12de00640 1 -- 172.21.15.47:0/2906868404 shutdown_connections 2024-02-19T15:25:14.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.685+0000 7fb12de00640 1 -- 172.21.15.47:0/2906868404 wait complete. 2024-02-19T15:25:14.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.685+0000 7fb12de00640 1 Processor -- start 2024-02-19T15:25:14.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.685+0000 7fb12de00640 1 -- start start 2024-02-19T15:25:14.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb12de00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb12806d650 0x7fb1280c8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:14.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb12de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1280cded0 0x7fb12815b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:14.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb12de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1280ced10 0x7fb1280b6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:14.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb12de00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12815c9b0 con 0x7fb1280cded0 2024-02-19T15:25:14.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb12de00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1280c94a0 con 0x7fb1280ced10 2024-02-19T15:25:14.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb12de00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1280c96d0 con 0x7fb12806d650 2024-02-19T15:25:14.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb11fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1280cded0 0x7fb12815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:14.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb11fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1280cded0 0x7fb12815b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55068/0 (socket says 172.21.15.47:55068) 2024-02-19T15:25:14.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb126a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb12806d650 0x7fb1280c8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:14.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb11fe00640 1 -- 172.21.15.47:0/3064306249 learned_addr learned my addr 172.21.15.47:0/3064306249 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:14.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb127400640 1 --2- 172.21.15.47:0/3064306249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1280ced10 0x7fb1280b6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:14.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb11fe00640 1 -- 172.21.15.47:0/3064306249 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb12806d650 msgr2=0x7fb1280c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:14.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb11fe00640 1 --2- 172.21.15.47:0/3064306249 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb12806d650 0x7fb1280c8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb11fe00640 1 -- 172.21.15.47:0/3064306249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1280ced10 msgr2=0x7fb1280b6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:14.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb11fe00640 1 --2- 172.21.15.47:0/3064306249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1280ced10 0x7fb1280b6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb11fe00640 1 -- 172.21.15.47:0/3064306249 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb120009a70 con 0x7fb1280cded0 2024-02-19T15:25:14.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb11fe00640 1 --2- 172.21.15.47:0/3064306249 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1280cded0 0x7fb12815b430 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fb11000c4d0 tx=0x7fb11000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:14.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb11f400640 1 -- 172.21.15.47:0/3064306249 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb110005cf0 con 0x7fb1280cded0 2024-02-19T15:25:14.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb11f400640 1 -- 172.21.15.47:0/3064306249 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb110005e50 con 0x7fb1280cded0 2024-02-19T15:25:14.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb11f400640 1 -- 172.21.15.47:0/3064306249 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb110007270 con 0x7fb1280cded0 2024-02-19T15:25:14.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb12de00640 1 -- 172.21.15.47:0/3064306249 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1280c99b0 con 0x7fb1280cded0 2024-02-19T15:25:14.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb12de00640 1 -- 172.21.15.47:0/3064306249 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1280c6ad0 con 0x7fb1280cded0 2024-02-19T15:25:14.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.689+0000 7fb12de00640 1 -- 172.21.15.47:0/3064306249 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1280c6cf0 con 0x7fb1280cded0 2024-02-19T15:25:14.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.693+0000 7fb11f400640 1 -- 172.21.15.47:0/3064306249 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb1100073d0 con 0x7fb1280cded0 2024-02-19T15:25:14.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.693+0000 7fb11f400640 1 --2- 172.21.15.47:0/3064306249 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb104066e40 0x7fb104069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:14.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.693+0000 7fb11f400640 1 -- 172.21.15.47:0/3064306249 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb110083eb0 con 0x7fb1280cded0 2024-02-19T15:25:14.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.693+0000 7fb126a00640 1 --2- 172.21.15.47:0/3064306249 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb104066e40 0x7fb104069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:14.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.693+0000 7fb126a00640 1 --2- 172.21.15.47:0/3064306249 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb104066e40 0x7fb104069300 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb118001720 tx=0x7fb118009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:14.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.697+0000 7fb11f400640 1 -- 172.21.15.47:0/3064306249 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb110050d50 con 0x7fb1280cded0 2024-02-19T15:25:14.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.893+0000 7fb12de00640 1 -- 172.21.15.47:0/3064306249 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb128075360 con 0x7fb1280cded0 2024-02-19T15:25:14.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.893+0000 7fb11f400640 1 -- 172.21.15.47:0/3064306249 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4946 (secure 0 0 0) 0x7fb110017020 con 0x7fb1280cded0 2024-02-19T15:25:14.898 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:14.898 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5684,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3206627105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3206627105},{"type":"v1","addr":"172.21.15.116:6836","nonce":3206627105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5687,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3532185042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3532185042},{"type":"v1","addr":"172.21.15.116:6837","nonce":3532185042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/1482478568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1482478568},{"type":"v1","addr":"172.21.15.47:6834","nonce":1482478568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:25:08.947740+0000","modified":"2024-02-19T15:25:11.954116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/2490197058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2490197058},{"type":"v1","addr":"172.21.15.47:6835","nonce":2490197058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-02-19T15:25:14.898 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 40 2024-02-19T15:25:14.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.897+0000 7fb12de00640 1 -- 172.21.15.47:0/3064306249 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb104066e40 msgr2=0x7fb104069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:14.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.897+0000 7fb12de00640 1 --2- 172.21.15.47:0/3064306249 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb104066e40 0x7fb104069300 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb118001720 tx=0x7fb118009040 comp rx=0 tx=0).stop 2024-02-19T15:25:14.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.897+0000 7fb12de00640 1 -- 172.21.15.47:0/3064306249 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1280cded0 msgr2=0x7fb12815b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:14.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.897+0000 7fb12de00640 1 --2- 172.21.15.47:0/3064306249 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1280cded0 0x7fb12815b430 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fb11000c4d0 tx=0x7fb11000c990 comp rx=0 tx=0).stop 2024-02-19T15:25:14.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.897+0000 7fb12de00640 1 -- 172.21.15.47:0/3064306249 shutdown_connections 2024-02-19T15:25:14.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.897+0000 7fb12de00640 1 --2- 172.21.15.47:0/3064306249 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb104066e40 0x7fb104069300 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.897+0000 7fb12de00640 1 --2- 172.21.15.47:0/3064306249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1280ced10 0x7fb1280b6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.897+0000 7fb12de00640 1 --2- 172.21.15.47:0/3064306249 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1280cded0 0x7fb12815b430 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.897+0000 7fb12de00640 1 --2- 172.21.15.47:0/3064306249 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb12806d650 0x7fb1280c8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:14.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.897+0000 7fb12de00640 1 -- 172.21.15.47:0/3064306249 >> 172.21.15.47:0/3064306249 conn(0x7fb1280d1bf0 msgr2=0x7fb1280b5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:14.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.897+0000 7fb12de00640 1 -- 172.21.15.47:0/3064306249 shutdown_connections 2024-02-19T15:25:14.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:14.897+0000 7fb12de00640 1 -- 172.21.15.47:0/3064306249 wait complete. 2024-02-19T15:25:15.321 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 40, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:25:08.947740+0000', 'modified': '2024-02-19T15:25:11.954116+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6354}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6354': {'gid': 6354, 'name': 'c', 'rank': 0, 'incarnation': 37, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/2490197058', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 2490197058}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 2490197058}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:25:15.322 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T15:25:15.322 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T15:25:15.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.457+0000 7f1335800640 1 Processor -- start 2024-02-19T15:25:15.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.457+0000 7f1335800640 1 -- start start 2024-02-19T15:25:15.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.461+0000 7f1335800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f133006d650 0x7f13300d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.461+0000 7f1335800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13300cded0 0x7f13300ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.461+0000 7f1335800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13300cce30 con 0x7f13300ceb40 2024-02-19T15:25:15.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.461+0000 7f1335800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13300cefd0 con 0x7f133006d650 2024-02-19T15:25:15.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.461+0000 7f1335800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13300cd110 con 0x7f13300cded0 2024-02-19T15:25:15.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.461+0000 7f132f400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f13300ceb40 0x7f1330161d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:56152/0 (socket says 172.21.15.47:56152) 2024-02-19T15:25:15.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.461+0000 7f132f400640 1 -- 172.21.15.47:0/429887210 learned_addr learned my addr 172.21.15.47:0/429887210 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:15.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.461+0000 7f132ea00640 1 --2- 172.21.15.47:0/429887210 >> v2:172.21.15.116:3300/0 conn(0x7f133006d650 0x7f13300d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.461+0000 7f1327e00640 1 --2- 172.21.15.47:0/429887210 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13300cded0 0x7f13300ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.461+0000 7f132e000640 1 -- 172.21.15.47:0/429887210 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849412697 0 0) 0x7f13300cce30 con 0x7f13300ceb40 2024-02-19T15:25:15.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.461+0000 7f132e000640 1 -- 172.21.15.47:0/429887210 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1314003530 con 0x7f13300ceb40 2024-02-19T15:25:15.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.461+0000 7f132ea00640 1 -- 172.21.15.47:0/429887210 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13300cded0 msgr2=0x7f13300ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:15.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.461+0000 7f132ea00640 1 --2- 172.21.15.47:0/429887210 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13300cded0 0x7f13300ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.461+0000 7f132ea00640 1 -- 172.21.15.47:0/429887210 >> v1:172.21.15.47:6789/0 conn(0x7f13300ceb40 legacy=0x7f1330161d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:15.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.461+0000 7f132ea00640 1 -- 172.21.15.47:0/429887210 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13300cd3f0 con 0x7f133006d650 2024-02-19T15:25:15.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.461+0000 7f1327e00640 1 --2- 172.21.15.47:0/429887210 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13300cded0 0x7f13300ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:25:15.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.465+0000 7f132ea00640 1 --2- 172.21.15.47:0/429887210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f133006d650 0x7f13300d08b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f13200097c0 tx=0x7f132001c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e73d6df54709e726 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:15.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.465+0000 7f132e000640 1 -- 172.21.15.47:0/429887210 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13200ab600 con 0x7f133006d650 2024-02-19T15:25:15.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.465+0000 7f132e000640 1 -- 172.21.15.47:0/429887210 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13200a7050 con 0x7f133006d650 2024-02-19T15:25:15.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.465+0000 7f132e000640 1 -- 172.21.15.47:0/429887210 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13200ab980 con 0x7f133006d650 2024-02-19T15:25:15.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.465+0000 7f1335800640 1 -- 172.21.15.47:0/429887210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f133006d650 msgr2=0x7f13300d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:15.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.465+0000 7f1335800640 1 --2- 172.21.15.47:0/429887210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f133006d650 0x7f13300d08b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f13200097c0 tx=0x7f132001c3e0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.465+0000 7f1335800640 1 -- 172.21.15.47:0/429887210 shutdown_connections 2024-02-19T15:25:15.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.465+0000 7f1335800640 1 --2- 172.21.15.47:0/429887210 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13300cded0 0x7f13300ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.465+0000 7f1335800640 1 --2- 172.21.15.47:0/429887210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f133006d650 0x7f13300d08b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.465+0000 7f1335800640 1 -- 172.21.15.47:0/429887210 >> 172.21.15.47:0/429887210 conn(0x7f13300d1bf0 msgr2=0x7f13300d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:15.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.465+0000 7f1335800640 1 -- 172.21.15.47:0/429887210 shutdown_connections 2024-02-19T15:25:15.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.465+0000 7f1335800640 1 -- 172.21.15.47:0/429887210 wait complete. 2024-02-19T15:25:15.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.465+0000 7f1335800640 1 Processor -- start 2024-02-19T15:25:15.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.465+0000 7f1335800640 1 -- start start 2024-02-19T15:25:15.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1335800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f133006d650 0x7f1330157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1335800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f13300cded0 0x7f13300c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1335800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13300ceb40 0x7f1330158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1335800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13300c8870 con 0x7f13300cded0 2024-02-19T15:25:15.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1335800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1330159490 con 0x7f133006d650 2024-02-19T15:25:15.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1335800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f133015a1d0 con 0x7f13300ceb40 2024-02-19T15:25:15.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1327e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f13300cded0 0x7f13300c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1327e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f13300cded0 0x7f13300c5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55082/0 (socket says 172.21.15.47:55082) 2024-02-19T15:25:15.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1327e00640 1 -- 172.21.15.47:0/41072796 learned_addr learned my addr 172.21.15.47:0/41072796 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:15.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f132ea00640 1 --2- 172.21.15.47:0/41072796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f133006d650 0x7f1330157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f132f400640 1 --2- 172.21.15.47:0/41072796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13300ceb40 0x7f1330158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1327e00640 1 -- 172.21.15.47:0/41072796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13300ceb40 msgr2=0x7f1330158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:15.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1327e00640 1 --2- 172.21.15.47:0/41072796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13300ceb40 0x7f1330158850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1327e00640 1 -- 172.21.15.47:0/41072796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f133006d650 msgr2=0x7f1330157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:15.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1327e00640 1 --2- 172.21.15.47:0/41072796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f133006d650 0x7f1330157590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1327e00640 1 -- 172.21.15.47:0/41072796 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1318009420 con 0x7f13300cded0 2024-02-19T15:25:15.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f132ea00640 1 --2- 172.21.15.47:0/41072796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f133006d650 0x7f1330157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:15.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f132f400640 1 --2- 172.21.15.47:0/41072796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13300ceb40 0x7f1330158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:15.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1327e00640 1 --2- 172.21.15.47:0/41072796 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f13300cded0 0x7f13300c5650 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f131800e980 tx=0x7f131800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:15.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1326a00640 1 -- 172.21.15.47:0/41072796 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131800ca00 con 0x7f13300cded0 2024-02-19T15:25:15.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1326a00640 1 -- 172.21.15.47:0/41072796 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131800cb60 con 0x7f13300cded0 2024-02-19T15:25:15.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1335800640 1 -- 172.21.15.47:0/41072796 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1320009420 con 0x7f13300cded0 2024-02-19T15:25:15.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1326a00640 1 -- 172.21.15.47:0/41072796 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1318013470 con 0x7f13300cded0 2024-02-19T15:25:15.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1335800640 1 -- 172.21.15.47:0/41072796 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13300c81b0 con 0x7f13300cded0 2024-02-19T15:25:15.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.469+0000 7f1326a00640 1 -- 172.21.15.47:0/41072796 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f13180135d0 con 0x7f13300cded0 2024-02-19T15:25:15.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f1326a00640 1 --2- 172.21.15.47:0/41072796 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f130c066e40 0x7f130c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f1326a00640 1 -- 172.21.15.47:0/41072796 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1318084d70 con 0x7f13300cded0 2024-02-19T15:25:15.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f1338c9e000 1 Processor -- start 2024-02-19T15:25:15.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f132ea00640 1 --2- 172.21.15.47:0/41072796 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f130c066e40 0x7f130c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f1338c9e000 1 -- start start 2024-02-19T15:25:15.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f132ea00640 1 --2- 172.21.15.47:0/41072796 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f130c066e40 0x7f130c069300 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f13200ad000 tx=0x7f1320008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:15.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f1338c9e000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x558e7701d920 0x558e7701dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f1338c9e000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e7701e240 0x558e76fd39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f1338c9e000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 0x558e76fd4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f1338c9e000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e76e9bd00 con 0x558e76fd3f30 2024-02-19T15:25:15.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f1338c9e000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e76e2d430 con 0x558e7701e240 2024-02-19T15:25:15.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f1338c9e000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e76e2a960 con 0x558e7701d920 2024-02-19T15:25:15.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f132f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 0x558e76fd4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f132f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 0x558e76fd4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55096/0 (socket says 172.21.15.47:55096) 2024-02-19T15:25:15.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f132f400640 1 -- 172.21.15.47:0/695987972 learned_addr learned my addr 172.21.15.47:0/695987972 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:15.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f132ea00640 1 --2- 172.21.15.47:0/695987972 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x558e7701d920 0x558e7701dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f1327e00640 1 --2- 172.21.15.47:0/695987972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e7701e240 0x558e76fd39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f132f400640 1 -- 172.21.15.47:0/695987972 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x558e7701d920 msgr2=0x558e7701dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:15.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f132f400640 1 --2- 172.21.15.47:0/695987972 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x558e7701d920 0x558e7701dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f132f400640 1 -- 172.21.15.47:0/695987972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e7701e240 msgr2=0x558e76fd39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:15.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f132f400640 1 --2- 172.21.15.47:0/695987972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e7701e240 0x558e76fd39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f132f400640 1 -- 172.21.15.47:0/695987972 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e76b1c890 con 0x558e76fd3f30 2024-02-19T15:25:15.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f132f400640 1 --2- 172.21.15.47:0/695987972 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 0x558e76fd4310 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f13280020b0 tx=0x7f132800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30742497708912b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:15.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f131f400640 1 -- 172.21.15.47:0/695987972 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328014480 con 0x558e76fd3f30 2024-02-19T15:25:15.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f131f400640 1 -- 172.21.15.47:0/695987972 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13280145e0 con 0x558e76fd3f30 2024-02-19T15:25:15.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f131f400640 1 -- 172.21.15.47:0/695987972 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328014950 con 0x558e76fd3f30 2024-02-19T15:25:15.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f1338c9e000 1 -- 172.21.15.47:0/695987972 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 msgr2=0x558e76fd4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:15.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f1338c9e000 1 --2- 172.21.15.47:0/695987972 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 0x558e76fd4310 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f13280020b0 tx=0x7f132800e530 comp rx=0 tx=0).stop 2024-02-19T15:25:15.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f1338c9e000 1 -- 172.21.15.47:0/695987972 shutdown_connections 2024-02-19T15:25:15.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f1338c9e000 1 --2- 172.21.15.47:0/695987972 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 0x558e76fd4310 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f1338c9e000 1 --2- 172.21.15.47:0/695987972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e7701e240 0x558e76fd39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f1338c9e000 1 --2- 172.21.15.47:0/695987972 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x558e7701d920 0x558e7701dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f1338c9e000 1 -- 172.21.15.47:0/695987972 >> 172.21.15.47:0/695987972 conn(0x558e76fda2a0 msgr2=0x558e76fd22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:15.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.473+0000 7f1338c9e000 1 -- 172.21.15.47:0/695987972 shutdown_connections 2024-02-19T15:25:15.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.477+0000 7f1338c9e000 1 -- 172.21.15.47:0/695987972 wait complete. 2024-02-19T15:25:15.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.477+0000 7f1338c9e000 1 -- start start 2024-02-19T15:25:15.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.477+0000 7f1338c9e000 1 Processor -- start 2024-02-19T15:25:15.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.477+0000 7f1338c9e000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e76fd3f30 0x558e7701d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.477+0000 7f1338c9e000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x558e7701d920 0x558e76fd8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.477+0000 7f1338c9e000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e7701e240 0x558e76fd8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.477+0000 7f1338c9e000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e76e26140 con 0x558e7701e240 2024-02-19T15:25:15.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.477+0000 7f1338c9e000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e76dc0630 con 0x558e76fd3f30 2024-02-19T15:25:15.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.477+0000 7f1338c9e000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e76dc02d0 con 0x558e7701d920 2024-02-19T15:25:15.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.477+0000 7f1338c9e000 10 client.0 _ll_register_callbacks cb 0x558e76ed61c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:25:15.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.477+0000 7f132f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e7701e240 0x558e76fd8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.477+0000 7f132f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e7701e240 0x558e76fd8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55102/0 (socket says 172.21.15.47:55102) 2024-02-19T15:25:15.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.477+0000 7f132f400640 1 -- 172.21.15.47:0/1130692803 learned_addr learned my addr 172.21.15.47:0/1130692803 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:15.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.477+0000 7f132ea00640 1 --2- 172.21.15.47:0/1130692803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e76fd3f30 0x558e7701d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.477+0000 7f1327e00640 1 --2- 172.21.15.47:0/1130692803 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x558e7701d920 0x558e76fd8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.477+0000 7f132f400640 1 -- 172.21.15.47:0/1130692803 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x558e7701d920 msgr2=0x558e76fd8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:15.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.477+0000 7f132f400640 1 --2- 172.21.15.47:0/1130692803 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x558e7701d920 0x558e76fd8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.477+0000 7f132f400640 1 -- 172.21.15.47:0/1130692803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e76fd3f30 msgr2=0x558e7701d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.477+0000 7f132f400640 1 --2- 172.21.15.47:0/1130692803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e76fd3f30 0x558e7701d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.477+0000 7f132f400640 1 -- 172.21.15.47:0/1130692803 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f132800f040 con 0x558e7701e240 2024-02-19T15:25:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f132f400640 1 --2- 172.21.15.47:0/1130692803 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e7701e240 0x558e76fd8a60 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f1328009010 tx=0x7f1328001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3333adf33ef49ab5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1313e00640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:25:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1313e00640 1 -- 172.21.15.47:0/1130692803 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328001d20 con 0x558e7701e240 2024-02-19T15:25:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 1 -- 172.21.15.47:0/1130692803 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558e76dd9850 con 0x558e7701e240 2024-02-19T15:25:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1313e00640 1 -- 172.21.15.47:0/1130692803 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1328014a40 con 0x558e7701e240 2024-02-19T15:25:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1313e00640 1 -- 172.21.15.47:0/1130692803 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1328014c00 con 0x558e7701e240 2024-02-19T15:25:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 10 client.6495 fetch_fsmap learned FSMap version 40 2024-02-19T15:25:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 1 -- 172.21.15.47:0/1130692803 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) v3 -- 0x558e76de5e70 con 0x558e7701e240 2024-02-19T15:25:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1313e00640 1 -- 172.21.15.47:0/1130692803 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558e76dd9850 con 0x558e7701e240 2024-02-19T15:25:15.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1313e00640 1 -- 172.21.15.47:0/1130692803 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 40) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1328028440 con 0x558e7701e240 2024-02-19T15:25:15.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 10 client.6495 fetch_fsmap finished waiting for FSMap version 40 2024-02-19T15:25:15.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 10 client.6495 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T15:25:15.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 20 client.6495 populate_metadata read hostname 'smithi047' 2024-02-19T15:25:15.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 1 --2- 172.21.15.47:0/1130692803 >> [v2:172.21.15.47:6833/2490197058,v1:172.21.15.47:6835/2490197058] conn(0x558e7701ed00 0x558e7701f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 4 client.6495 mds_command: new command op to 6354 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T15:25:15.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 1 -- 172.21.15.47:0/1130692803 --> [v2:172.21.15.47:6833/2490197058,v1:172.21.15.47:6835/2490197058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558e76e22f20 con 0x558e7701ed00 2024-02-19T15:25:15.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f132ea00640 1 --2- 172.21.15.47:0/1130692803 >> [v2:172.21.15.47:6833/2490197058,v1:172.21.15.47:6835/2490197058] conn(0x558e7701ed00 0x558e7701f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f132ea00640 1 --2- 172.21.15.47:0/1130692803 >> [v2:172.21.15.47:6833/2490197058,v1:172.21.15.47:6835/2490197058] conn(0x558e7701ed00 0x558e7701f0e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47b5f10ed4f3284 server_cookie=ef3c7a1319f6ca06 in_seq=0 out_seq=0 2024-02-19T15:25:15.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1313e00640 10 client.6495 ms_handle_connect on v2:172.21.15.47:6833/2490197058 2024-02-19T15:25:15.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1313e00640 1 -- 172.21.15.47:0/1130692803 <== mds.0 v2:172.21.15.47:6833/2490197058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558e76e22f20 con 0x558e7701ed00 2024-02-19T15:25:15.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1313e00640 10 client.6495 handle_command_reply: tid=0 2024-02-19T15:25:15.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 1 client.6495 shutdown 2024-02-19T15:25:15.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 1 -- 172.21.15.47:0/1130692803 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e7701e240 msgr2=0x558e76fd8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:15.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 1 --2- 172.21.15.47:0/1130692803 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e7701e240 0x558e76fd8a60 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f1328009010 tx=0x7f1328001260 comp rx=0 tx=0).stop 2024-02-19T15:25:15.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 1 -- 172.21.15.47:0/1130692803 shutdown_connections 2024-02-19T15:25:15.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 1 --2- 172.21.15.47:0/1130692803 >> [v2:172.21.15.47:6833/2490197058,v1:172.21.15.47:6835/2490197058] conn(0x558e7701ed00 0x558e7701f0e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 1 --2- 172.21.15.47:0/1130692803 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e7701e240 0x558e76fd8a60 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 1 --2- 172.21.15.47:0/1130692803 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x558e7701d920 0x558e76fd8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 1 --2- 172.21.15.47:0/1130692803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e76fd3f30 0x558e7701d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 1 -- 172.21.15.47:0/1130692803 >> 172.21.15.47:0/1130692803 conn(0x558e76fda2a0 msgr2=0x558e76fd29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:15.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1313e00640 0 client.6495 ms_handle_reset on v2:172.21.15.47:6833/2490197058 2024-02-19T15:25:15.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 1 -- 172.21.15.47:0/1130692803 shutdown_connections 2024-02-19T15:25:15.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 1 -- 172.21.15.47:0/1130692803 wait complete. 2024-02-19T15:25:15.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.481+0000 7f1338c9e000 20 client.6495 trim_cache size 0 max 16384 2024-02-19T15:25:15.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f1338c9e000 1 Processor -- start 2024-02-19T15:25:15.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f1338c9e000 1 -- start start 2024-02-19T15:25:15.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f1338c9e000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e76fda2a0 0x558e7707e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f1338c9e000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 0x558e7707e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f1338c9e000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x558e7701d920 0x558e7707ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f1338c9e000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e76add940 con 0x558e76fd3f30 2024-02-19T15:25:15.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f1338c9e000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e76fcb750 con 0x558e76fda2a0 2024-02-19T15:25:15.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f1338c9e000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e76e202c0 con 0x558e7701d920 2024-02-19T15:25:15.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f1327e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 0x558e7707e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f1327e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 0x558e7707e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55112/0 (socket says 172.21.15.47:55112) 2024-02-19T15:25:15.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f1327e00640 1 -- 172.21.15.47:0/816774797 learned_addr learned my addr 172.21.15.47:0/816774797 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:15.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f132ea00640 1 --2- 172.21.15.47:0/816774797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e76fda2a0 0x558e7707e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f132f400640 1 --2- 172.21.15.47:0/816774797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x558e7701d920 0x558e7707ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f1327e00640 1 -- 172.21.15.47:0/816774797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x558e7701d920 msgr2=0x558e7707ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:15.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f1327e00640 1 --2- 172.21.15.47:0/816774797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x558e7701d920 0x558e7707ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f1327e00640 1 -- 172.21.15.47:0/816774797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e76fda2a0 msgr2=0x558e7707e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:15.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f1327e00640 1 --2- 172.21.15.47:0/816774797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e76fda2a0 0x558e7707e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f1327e00640 1 -- 172.21.15.47:0/816774797 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e76dcbe10 con 0x558e76fd3f30 2024-02-19T15:25:15.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f1327e00640 1 --2- 172.21.15.47:0/816774797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 0x558e7707e680 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f1318009fd0 tx=0x7f1318051ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32fc9a9e00244345 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:15.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f131ea00640 1 -- 172.21.15.47:0/816774797 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1318080070 con 0x558e76fd3f30 2024-02-19T15:25:15.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f131ea00640 1 -- 172.21.15.47:0/816774797 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131805df30 con 0x558e76fd3f30 2024-02-19T15:25:15.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.509+0000 7f131ea00640 1 -- 172.21.15.47:0/816774797 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1318053b20 con 0x558e76fd3f30 2024-02-19T15:25:15.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f1338c9e000 1 -- 172.21.15.47:0/816774797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 msgr2=0x558e7707e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:15.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f1338c9e000 1 --2- 172.21.15.47:0/816774797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 0x558e7707e680 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f1318009fd0 tx=0x7f1318051ac0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f1338c9e000 1 -- 172.21.15.47:0/816774797 shutdown_connections 2024-02-19T15:25:15.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f1338c9e000 1 --2- 172.21.15.47:0/816774797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x558e7701d920 0x558e7707ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f1338c9e000 1 --2- 172.21.15.47:0/816774797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 0x558e7707e680 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f1338c9e000 1 --2- 172.21.15.47:0/816774797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e76fda2a0 0x558e7707e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f1338c9e000 1 -- 172.21.15.47:0/816774797 >> 172.21.15.47:0/816774797 conn(0x558e7702a510 msgr2=0x558e77030290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:15.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f1338c9e000 1 -- 172.21.15.47:0/816774797 shutdown_connections 2024-02-19T15:25:15.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f1338c9e000 1 -- 172.21.15.47:0/816774797 wait complete. 2024-02-19T15:25:15.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f1338c9e000 1 -- start start 2024-02-19T15:25:15.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f1338c9e000 1 Processor -- start 2024-02-19T15:25:15.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f1338c9e000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 0x558e77066930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f1338c9e000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x558e76fda2a0 0x558e770673b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f1338c9e000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e7701d920 0x558e770678f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f1338c9e000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e76e28ca0 con 0x558e76fd3f30 2024-02-19T15:25:15.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f1338c9e000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e76dbe660 con 0x558e7701d920 2024-02-19T15:25:15.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f1338c9e000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e76e21f30 con 0x558e76fda2a0 2024-02-19T15:25:15.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f1338c9e000 10 client.0 _ll_register_callbacks cb 0x7f1328017070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:25:15.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f132ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 0x558e77066930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f132ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 0x558e77066930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55120/0 (socket says 172.21.15.47:55120) 2024-02-19T15:25:15.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f132ea00640 1 -- 172.21.15.47:0/2986441493 learned_addr learned my addr 172.21.15.47:0/2986441493 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:15.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.513+0000 7f1327e00640 1 --2- 172.21.15.47:0/2986441493 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x558e76fda2a0 0x558e770673b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f132f400640 1 --2- 172.21.15.47:0/2986441493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e7701d920 0x558e770678f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f132ea00640 1 -- 172.21.15.47:0/2986441493 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x558e76fda2a0 msgr2=0x558e770673b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:15.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f132ea00640 1 --2- 172.21.15.47:0/2986441493 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x558e76fda2a0 0x558e770673b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f132ea00640 1 -- 172.21.15.47:0/2986441493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e7701d920 msgr2=0x558e770678f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:15.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f132ea00640 1 --2- 172.21.15.47:0/2986441493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e7701d920 0x558e770678f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f132ea00640 1 -- 172.21.15.47:0/2986441493 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13300c81b0 con 0x558e76fd3f30 2024-02-19T15:25:15.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f132ea00640 1 --2- 172.21.15.47:0/2986441493 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 0x558e77066930 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f132001a010 tx=0x7f132001ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18ac4cb64d0feb37 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:15.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1313e00640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:25:15.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1313e00640 1 -- 172.21.15.47:0/2986441493 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13200c8350 con 0x558e76fd3f30 2024-02-19T15:25:15.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1313e00640 1 -- 172.21.15.47:0/2986441493 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13200c8a70 con 0x558e76fd3f30 2024-02-19T15:25:15.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 1 -- 172.21.15.47:0/2986441493 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558e76dabe80 con 0x558e76fd3f30 2024-02-19T15:25:15.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1313e00640 1 -- 172.21.15.47:0/2986441493 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13200c8c30 con 0x558e76fd3f30 2024-02-19T15:25:15.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 10 client.6501 fetch_fsmap learned FSMap version 40 2024-02-19T15:25:15.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 1 -- 172.21.15.47:0/2986441493 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) v3 -- 0x558e76b1ccc0 con 0x558e76fd3f30 2024-02-19T15:25:15.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1313e00640 1 -- 172.21.15.47:0/2986441493 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558e76dabe80 con 0x558e76fd3f30 2024-02-19T15:25:15.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1313e00640 1 -- 172.21.15.47:0/2986441493 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 40) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f13200d0020 con 0x558e76fd3f30 2024-02-19T15:25:15.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 10 client.6501 fetch_fsmap finished waiting for FSMap version 40 2024-02-19T15:25:15.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 10 client.6501 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T15:25:15.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 20 client.6501 populate_metadata read hostname 'smithi047' 2024-02-19T15:25:15.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 1 --2- 172.21.15.47:0/2986441493 >> [v2:172.21.15.47:6833/2490197058,v1:172.21.15.47:6835/2490197058] conn(0x558e7701e240 0x558e77067e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 4 client.6501 mds_command: new command op to 6354 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T15:25:15.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 1 -- 172.21.15.47:0/2986441493 --> [v2:172.21.15.47:6833/2490197058,v1:172.21.15.47:6835/2490197058] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558e76deefa0 con 0x558e7701e240 2024-02-19T15:25:15.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1327e00640 1 --2- 172.21.15.47:0/2986441493 >> [v2:172.21.15.47:6833/2490197058,v1:172.21.15.47:6835/2490197058] conn(0x558e7701e240 0x558e77067e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1313e00640 10 client.6501 ms_handle_connect on v2:172.21.15.47:6833/2490197058 2024-02-19T15:25:15.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1327e00640 1 --2- 172.21.15.47:0/2986441493 >> [v2:172.21.15.47:6833/2490197058,v1:172.21.15.47:6835/2490197058] conn(0x558e7701e240 0x558e77067e30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe3182b5d036c17c server_cookie=98941bd8e5127fc7 in_seq=0 out_seq=0 2024-02-19T15:25:15.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1313e00640 1 -- 172.21.15.47:0/2986441493 <== mds.0 v2:172.21.15.47:6833/2490197058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x558e76deefa0 con 0x558e7701e240 2024-02-19T15:25:15.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1313e00640 10 client.6501 handle_command_reply: tid=0 2024-02-19T15:25:15.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 1 client.6501 shutdown 2024-02-19T15:25:15.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 1 -- 172.21.15.47:0/2986441493 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 msgr2=0x558e77066930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:15.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 1 --2- 172.21.15.47:0/2986441493 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 0x558e77066930 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f132001a010 tx=0x7f132001ac20 comp rx=0 tx=0).stop 2024-02-19T15:25:15.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 1 -- 172.21.15.47:0/2986441493 shutdown_connections 2024-02-19T15:25:15.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 1 --2- 172.21.15.47:0/2986441493 >> [v2:172.21.15.47:6833/2490197058,v1:172.21.15.47:6835/2490197058] conn(0x558e7701e240 0x558e77067e30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 1 --2- 172.21.15.47:0/2986441493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e7701d920 0x558e770678f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 1 --2- 172.21.15.47:0/2986441493 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x558e76fda2a0 0x558e770673b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 1 --2- 172.21.15.47:0/2986441493 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558e76fd3f30 0x558e77066930 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 1 -- 172.21.15.47:0/2986441493 >> 172.21.15.47:0/2986441493 conn(0x558e7702a510 msgr2=0x558e770309a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:15.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1313e00640 0 client.6501 ms_handle_reset on v2:172.21.15.47:6833/2490197058 2024-02-19T15:25:15.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 1 -- 172.21.15.47:0/2986441493 shutdown_connections 2024-02-19T15:25:15.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 1 -- 172.21.15.47:0/2986441493 wait complete. 2024-02-19T15:25:15.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1338c9e000 20 client.6501 trim_cache size 0 max 16384 2024-02-19T15:25:15.525 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:25:15.526 INFO:teuthology.orchestra.run.smithi047.stdout: "cluster_fsid": "ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e", 2024-02-19T15:25:15.526 INFO:teuthology.orchestra.run.smithi047.stdout: "whoami": 0, 2024-02-19T15:25:15.526 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 6354, 2024-02-19T15:25:15.526 INFO:teuthology.orchestra.run.smithi047.stdout: "want_state": "up:active", 2024-02-19T15:25:15.526 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "up:active", 2024-02-19T15:25:15.526 INFO:teuthology.orchestra.run.smithi047.stdout: "fs_name": "cephfs", 2024-02-19T15:25:15.526 INFO:teuthology.orchestra.run.smithi047.stdout: "rank_uptime": 5.9543612650000002, 2024-02-19T15:25:15.526 INFO:teuthology.orchestra.run.smithi047.stdout: "mdsmap_epoch": 40, 2024-02-19T15:25:15.526 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch": 69, 2024-02-19T15:25:15.526 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch_barrier": 69, 2024-02-19T15:25:15.526 INFO:teuthology.orchestra.run.smithi047.stdout: "uptime": 11.859794040000001 2024-02-19T15:25:15.526 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:25:15.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1335800640 1 -- 172.21.15.47:0/41072796 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f130c066e40 msgr2=0x7f130c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:15.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1335800640 1 --2- 172.21.15.47:0/41072796 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f130c066e40 0x7f130c069300 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f13200ad000 tx=0x7f1320008040 comp rx=0 tx=0).stop 2024-02-19T15:25:15.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1335800640 1 -- 172.21.15.47:0/41072796 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f13300cded0 msgr2=0x7f13300c5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:15.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1335800640 1 --2- 172.21.15.47:0/41072796 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f13300cded0 0x7f13300c5650 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f131800e980 tx=0x7f131800ee40 comp rx=0 tx=0).stop 2024-02-19T15:25:15.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1335800640 1 -- 172.21.15.47:0/41072796 shutdown_connections 2024-02-19T15:25:15.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1335800640 1 --2- 172.21.15.47:0/41072796 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f130c066e40 0x7f130c069300 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1335800640 1 --2- 172.21.15.47:0/41072796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13300ceb40 0x7f1330158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1335800640 1 --2- 172.21.15.47:0/41072796 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f13300cded0 0x7f13300c5650 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1335800640 1 --2- 172.21.15.47:0/41072796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f133006d650 0x7f1330157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1335800640 1 -- 172.21.15.47:0/41072796 >> 172.21.15.47:0/41072796 conn(0x7f13300d1bf0 msgr2=0x7f133006fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:15.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.517+0000 7f1335800640 1 -- 172.21.15.47:0/41072796 shutdown_connections 2024-02-19T15:25:15.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.521+0000 7f1335800640 1 -- 172.21.15.47:0/41072796 wait complete. 2024-02-19T15:25:15.551 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-02-19T15:25:15.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.689+0000 7f8b72a00640 1 Processor -- start 2024-02-19T15:25:15.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.689+0000 7f8b72a00640 1 -- start start 2024-02-19T15:25:15.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.689+0000 7f8b72a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8b6c0cedb0 0x7f8b6c0cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.689+0000 7f8b72a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8b6c0d2710 0x7f8b6c0d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.689+0000 7f8b72a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6c0cc700 con 0x7f8b6c15bcd0 2024-02-19T15:25:15.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.689+0000 7f8b72a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b6c0cf5e0 con 0x7f8b6c0d2710 2024-02-19T15:25:15.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.689+0000 7f8b72a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b6c0cdb90 con 0x7f8b6c0cedb0 2024-02-19T15:25:15.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.693+0000 7f8b70a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8b6c0d2710 0x7f8b6c0d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.693+0000 7f8b70a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8b6c0d2710 0x7f8b6c0d0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55136/0 (socket says 172.21.15.47:55136) 2024-02-19T15:25:15.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.693+0000 7f8b70a00640 1 -- 172.21.15.47:0/2901072392 learned_addr learned my addr 172.21.15.47:0/2901072392 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:15.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.693+0000 7f8b63400640 1 --2- 172.21.15.47:0/2901072392 >> v2:172.21.15.116:3300/0 conn(0x7f8b6c0cedb0 0x7f8b6c0cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.693+0000 7f8b70a00640 1 -- 172.21.15.47:0/2901072392 >> v2:172.21.15.116:3300/0 conn(0x7f8b6c0cedb0 msgr2=0x7f8b6c0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:15.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.693+0000 7f8b70a00640 1 --2- 172.21.15.47:0/2901072392 >> v2:172.21.15.116:3300/0 conn(0x7f8b6c0cedb0 0x7f8b6c0cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.693+0000 7f8b70a00640 1 -- 172.21.15.47:0/2901072392 >> v1:172.21.15.116:6789/0 conn(0x7f8b6c15bcd0 legacy=0x7f8b6c0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:15.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.693+0000 7f8b70a00640 1 -- 172.21.15.47:0/2901072392 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6c0cf300 con 0x7f8b6c0d2710 2024-02-19T15:25:15.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.693+0000 7f8b70a00640 1 --2- 172.21.15.47:0/2901072392 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b6c0d2710 0x7f8b6c0d0190 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c0ce100 tx=0x7f8b6401c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ae701647c9aaf6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:15.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.693+0000 7f8b6b400640 1 -- 172.21.15.47:0/2901072392 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b640a5880 con 0x7f8b6c0d2710 2024-02-19T15:25:15.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.693+0000 7f8b6b400640 1 -- 172.21.15.47:0/2901072392 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b640a59e0 con 0x7f8b6c0d2710 2024-02-19T15:25:15.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.693+0000 7f8b6b400640 1 -- 172.21.15.47:0/2901072392 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b640a5d70 con 0x7f8b6c0d2710 2024-02-19T15:25:15.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b72a00640 1 -- 172.21.15.47:0/2901072392 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b6c0d2710 msgr2=0x7f8b6c0d0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:15.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b72a00640 1 --2- 172.21.15.47:0/2901072392 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b6c0d2710 0x7f8b6c0d0190 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c0ce100 tx=0x7f8b6401c9c0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b72a00640 1 -- 172.21.15.47:0/2901072392 shutdown_connections 2024-02-19T15:25:15.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b72a00640 1 --2- 172.21.15.47:0/2901072392 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b6c0d2710 0x7f8b6c0d0190 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b72a00640 1 --2- 172.21.15.47:0/2901072392 >> v2:172.21.15.116:3300/0 conn(0x7f8b6c0cedb0 0x7f8b6c0cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b72a00640 1 -- 172.21.15.47:0/2901072392 >> 172.21.15.47:0/2901072392 conn(0x7f8b6c0d1bf0 msgr2=0x7f8b6c0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:15.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b72a00640 1 -- 172.21.15.47:0/2901072392 shutdown_connections 2024-02-19T15:25:15.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b72a00640 1 -- 172.21.15.47:0/2901072392 wait complete. 2024-02-19T15:25:15.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b72a00640 1 Processor -- start 2024-02-19T15:25:15.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b72a00640 1 -- start start 2024-02-19T15:25:15.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b72a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b6c0cedb0 0x7f8b6c1514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b72a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b6c0d2710 0x7f8b6c0ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b72a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b6c15bcd0 0x7f8b6c149720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b72a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c0ca900 con 0x7f8b6c0d2710 2024-02-19T15:25:15.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b72a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c1f9aa0 con 0x7f8b6c15bcd0 2024-02-19T15:25:15.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b72a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c0caaf0 con 0x7f8b6c0cedb0 2024-02-19T15:25:15.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b63400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b6c0d2710 0x7f8b6c0ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b63400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b6c0d2710 0x7f8b6c0ca3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55146/0 (socket says 172.21.15.47:55146) 2024-02-19T15:25:15.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b63400640 1 -- 172.21.15.47:0/2483816510 learned_addr learned my addr 172.21.15.47:0/2483816510 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:15.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b6be00640 1 --2- 172.21.15.47:0/2483816510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b6c0cedb0 0x7f8b6c1514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.697+0000 7f8b70a00640 1 --2- 172.21.15.47:0/2483816510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b6c15bcd0 0x7f8b6c149720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.701+0000 7f8b63400640 1 -- 172.21.15.47:0/2483816510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b6c0cedb0 msgr2=0x7f8b6c1514d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:15.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.701+0000 7f8b63400640 1 --2- 172.21.15.47:0/2483816510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b6c0cedb0 0x7f8b6c1514d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.701+0000 7f8b63400640 1 -- 172.21.15.47:0/2483816510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b6c15bcd0 msgr2=0x7f8b6c149720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:15.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.701+0000 7f8b63400640 1 --2- 172.21.15.47:0/2483816510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b6c15bcd0 0x7f8b6c149720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.701+0000 7f8b63400640 1 -- 172.21.15.47:0/2483816510 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b64009b30 con 0x7f8b6c0d2710 2024-02-19T15:25:15.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.701+0000 7f8b63400640 1 --2- 172.21.15.47:0/2483816510 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b6c0d2710 0x7f8b6c0ca3c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400e870 tx=0x7f8b5400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:15.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.701+0000 7f8b68c00640 1 -- 172.21.15.47:0/2483816510 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b54009510 con 0x7f8b6c0d2710 2024-02-19T15:25:15.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.701+0000 7f8b68c00640 1 -- 172.21.15.47:0/2483816510 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5400eec0 con 0x7f8b6c0d2710 2024-02-19T15:25:15.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.701+0000 7f8b72a00640 1 -- 172.21.15.47:0/2483816510 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b6c15d240 con 0x7f8b6c0d2710 2024-02-19T15:25:15.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.701+0000 7f8b68c00640 1 -- 172.21.15.47:0/2483816510 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5400cb40 con 0x7f8b6c0d2710 2024-02-19T15:25:15.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.701+0000 7f8b72a00640 1 -- 172.21.15.47:0/2483816510 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b6c0cb0b0 con 0x7f8b6c0d2710 2024-02-19T15:25:15.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.701+0000 7f8b72a00640 1 -- 172.21.15.47:0/2483816510 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b6c075360 con 0x7f8b6c0d2710 2024-02-19T15:25:15.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.701+0000 7f8b68c00640 1 -- 172.21.15.47:0/2483816510 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8b5400cca0 con 0x7f8b6c0d2710 2024-02-19T15:25:15.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.701+0000 7f8b68c00640 1 --2- 172.21.15.47:0/2483816510 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8b48066e40 0x7f8b48069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:15.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.701+0000 7f8b68c00640 1 -- 172.21.15.47:0/2483816510 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8b54084bf0 con 0x7f8b6c0d2710 2024-02-19T15:25:15.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.705+0000 7f8b6be00640 1 --2- 172.21.15.47:0/2483816510 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8b48066e40 0x7f8b48069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:15.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.705+0000 7f8b6be00640 1 --2- 172.21.15.47:0/2483816510 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8b48066e40 0x7f8b48069300 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c001650 tx=0x7f8b5c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:15.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.709+0000 7f8b68c00640 1 -- 172.21.15.47:0/2483816510 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8b540519d0 con 0x7f8b6c0d2710 2024-02-19T15:25:15.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.917+0000 7f8b72a00640 1 -- 172.21.15.47:0/2483816510 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8b6c15cda0 con 0x7f8b6c0d2710 2024-02-19T15:25:15.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.921+0000 7f8b68c00640 1 -- 172.21.15.47:0/2483816510 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v40) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8b54017070 con 0x7f8b6c0d2710 2024-02-19T15:25:15.923 INFO:teuthology.orchestra.run.smithi047.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-02-19T15:25:15.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.921+0000 7f8b72a00640 1 -- 172.21.15.47:0/2483816510 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8b48066e40 msgr2=0x7f8b48069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:15.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.921+0000 7f8b72a00640 1 --2- 172.21.15.47:0/2483816510 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8b48066e40 0x7f8b48069300 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c001650 tx=0x7f8b5c009040 comp rx=0 tx=0).stop 2024-02-19T15:25:15.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.921+0000 7f8b72a00640 1 -- 172.21.15.47:0/2483816510 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b6c0d2710 msgr2=0x7f8b6c0ca3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:15.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.921+0000 7f8b72a00640 1 --2- 172.21.15.47:0/2483816510 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b6c0d2710 0x7f8b6c0ca3c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400e870 tx=0x7f8b5400ec70 comp rx=0 tx=0).stop 2024-02-19T15:25:15.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.921+0000 7f8b72a00640 1 -- 172.21.15.47:0/2483816510 shutdown_connections 2024-02-19T15:25:15.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.921+0000 7f8b72a00640 1 --2- 172.21.15.47:0/2483816510 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8b48066e40 0x7f8b48069300 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.921+0000 7f8b72a00640 1 --2- 172.21.15.47:0/2483816510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b6c15bcd0 0x7f8b6c149720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.921+0000 7f8b72a00640 1 --2- 172.21.15.47:0/2483816510 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8b6c0d2710 0x7f8b6c0ca3c0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.921+0000 7f8b72a00640 1 --2- 172.21.15.47:0/2483816510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b6c0cedb0 0x7f8b6c1514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:15.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.921+0000 7f8b72a00640 1 -- 172.21.15.47:0/2483816510 >> 172.21.15.47:0/2483816510 conn(0x7f8b6c0d1bf0 msgr2=0x7f8b6c06bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:15.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.921+0000 7f8b72a00640 1 -- 172.21.15.47:0/2483816510 shutdown_connections 2024-02-19T15:25:15.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:15.921+0000 7f8b72a00640 1 -- 172.21.15.47:0/2483816510 wait complete. 2024-02-19T15:25:15.944 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-02-19T15:25:15.944 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T15:25:15.945 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi047.front.sepia.ceph.com 2024-02-19T15:25:15.945 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-19T15:25:15.945 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-19T15:25:15.945 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-02-19T15:25:15.945 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-02-19T15:25:15.945 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T15:25:15.945 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:25:15.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:25:15.945 DEBUG:teuthology.orchestra.run.smithi047:> ip addr 2024-02-19T15:25:15.954 INFO:teuthology.orchestra.run.smithi047.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T15:25:15.954 INFO:teuthology.orchestra.run.smithi047.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T15:25:15.954 INFO:teuthology.orchestra.run.smithi047.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T15:25:15.954 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:25:15.954 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 ::1/128 scope host 2024-02-19T15:25:15.954 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:25:15.955 INFO:teuthology.orchestra.run.smithi047.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:25:15.955 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:78 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:25:15.955 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f0 2024-02-19T15:25:15.955 INFO:teuthology.orchestra.run.smithi047.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T15:25:15.955 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8e brd ff:ff:ff:ff:ff:ff 2024-02-19T15:25:15.955 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f0 2024-02-19T15:25:15.955 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.21.15.47/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-02-19T15:25:15.955 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft 42299sec preferred_lft 42299sec 2024-02-19T15:25:15.955 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::ec4:7aff:febd:148e/64 scope link 2024-02-19T15:25:15.955 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:25:15.955 INFO:teuthology.orchestra.run.smithi047.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:25:15.955 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:79 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:25:15.955 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f1 2024-02-19T15:25:15.955 INFO:teuthology.orchestra.run.smithi047.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:25:15.955 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8f brd ff:ff:ff:ff:ff:ff 2024-02-19T15:25:15.956 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f1 2024-02-19T15:25:15.956 INFO:teuthology.orchestra.run.smithi047.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-02-19T15:25:15.956 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 02:42:47:cf:b1:62 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:25:15.956 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-02-19T15:25:15.956 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:25:15.956 INFO:teuthology.orchestra.run.smithi047.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-02-19T15:25:15.956 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-02-19T15:25:15.956 INFO:teuthology.orchestra.run.smithi047.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-02-19T15:25:15.956 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:25:15.956 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-02-19T15:25:15.956 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:25:15.956 INFO:teuthology.orchestra.run.smithi047.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:25:15.956 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-19T15:25:15.957 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-02-19T15:25:15.957 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:25:15.957 INFO:teuthology.orchestra.run.smithi047.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:25:15.957 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-02-19T15:25:15.957 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-02-19T15:25:15.957 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:25:15.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:25:15.958 DEBUG:teuthology.orchestra.run.smithi047:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T15:25:16.019 INFO:teuthology.orchestra.run.smithi047.stdout:1 2024-02-19T15:25:16.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:25:16.021 DEBUG:teuthology.orchestra.run.smithi047:> route 2024-02-19T15:25:16.071 INFO:teuthology.orchestra.run.smithi047.stdout:Kernel IP routing table 2024-02-19T15:25:16.071 INFO:teuthology.orchestra.run.smithi047.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T15:25:16.071 INFO:teuthology.orchestra.run.smithi047.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-02-19T15:25:16.071 INFO:teuthology.orchestra.run.smithi047.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-02-19T15:25:16.071 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-02-19T15:25:16.071 INFO:teuthology.orchestra.run.smithi047.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T15:25:16.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:25:16.072 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2024-02-19T15:25:16.072 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2024-02-19T15:25:16.072 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-02-19T15:25:16.072 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-02-19T15:25:16.072 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-02-19T15:25:16.072 DEBUG:teuthology.orchestra.run.smithi047:> ') 2024-02-19T15:25:16.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:25:16.163 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list 2024-02-19T15:25:16.172 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-02-19T15:25:16.172 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-02-19T15:25:16.172 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:25:16.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:25:16.173 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set brx.0 up 2024-02-19T15:25:16.234 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:25:16.234 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-19T15:25:16.287 INFO:teuthology.orchestra.run.smithi047.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:25:16.288 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:25:16.288 DEBUG:teuthology.orchestra.run.smithi047:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:25:16.337 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-19T15:25:16.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:25:16.337 DEBUG:teuthology.orchestra.run.smithi047:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:25:16.426 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:25:16.426 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:25:16.426 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:25:16.441 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:25:16.441 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:25:16.441 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:25:16.441 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:25:16.441 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:25:16.450 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:25:16.450 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:25:16.450 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:25:16.450 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:25:16.450 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:25:16.450 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790" 2024-02-19T15:25:16.450 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790:/ 2024-02-19T15:25:16.451 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e" will pass to kernel 2024-02-19T15:25:16.454 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T15:25:16.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:25:16.455 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:25:16.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:25:16.476 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:25:16.535 INFO:teuthology.orchestra.run.smithi047.stdout:client5786 2024-02-19T15:25:16.535 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:25:16.535 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:25:16.592 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 10 2024-02-19T15:25:16.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.773+0000 7f50cb400640 1 Processor -- start 2024-02-19T15:25:16.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.773+0000 7f50cb400640 1 -- start start 2024-02-19T15:25:16.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.773+0000 7f50cb400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f50c415f780 0x7f50c415fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:16.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.773+0000 7f50cb400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f50c40d24f0 0x7f50c40d2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:16.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.773+0000 7f50cb400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c4160150 con 0x7f50c415e000 2024-02-19T15:25:16.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.773+0000 7f50cb400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50c40d31e0 con 0x7f50c40d24f0 2024-02-19T15:25:16.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.773+0000 7f50cb400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50c40ceca0 con 0x7f50c415f780 2024-02-19T15:25:16.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.773+0000 7f50c9400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f50c40d24f0 0x7f50c40d2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:16.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.773+0000 7f50c9400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f50c40d24f0 0x7f50c40d2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55150/0 (socket says 172.21.15.47:55150) 2024-02-19T15:25:16.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.773+0000 7f50c9400640 1 -- 172.21.15.47:0/804885527 learned_addr learned my addr 172.21.15.47:0/804885527 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:16.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.773+0000 7f50c3e00640 1 --2- 172.21.15.47:0/804885527 >> v2:172.21.15.116:3300/0 conn(0x7f50c415f780 0x7f50c415fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:16.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.777+0000 7f50c9400640 1 -- 172.21.15.47:0/804885527 >> v2:172.21.15.116:3300/0 conn(0x7f50c415f780 msgr2=0x7f50c415fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:16.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.777+0000 7f50c9400640 1 --2- 172.21.15.47:0/804885527 >> v2:172.21.15.116:3300/0 conn(0x7f50c415f780 0x7f50c415fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:16.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.777+0000 7f50c9400640 1 -- 172.21.15.47:0/804885527 >> v1:172.21.15.47:6789/0 conn(0x7f50c415e000 legacy=0x7f50c40cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:16.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.777+0000 7f50c9400640 1 -- 172.21.15.47:0/804885527 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c40d0410 con 0x7f50c40d24f0 2024-02-19T15:25:16.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.777+0000 7f50c3400640 1 -- 172.21.15.47:0/804885527 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811284311 0 0) 0x7f50c4160150 con 0x7f50c415e000 2024-02-19T15:25:16.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.777+0000 7f50c9400640 1 --2- 172.21.15.47:0/804885527 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50c40d24f0 0x7f50c40d2920 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f50c4068780 tx=0x7f50bc01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccbcf55ce030e662 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:16.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.777+0000 7f50c3400640 1 -- 172.21.15.47:0/804885527 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50bc0a57f0 con 0x7f50c40d24f0 2024-02-19T15:25:16.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.777+0000 7f50c3400640 1 -- 172.21.15.47:0/804885527 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50bc0a5950 con 0x7f50c40d24f0 2024-02-19T15:25:16.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.777+0000 7f50c3400640 1 -- 172.21.15.47:0/804885527 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50bc0a5ce0 con 0x7f50c40d24f0 2024-02-19T15:25:16.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.777+0000 7f50cb400640 1 -- 172.21.15.47:0/804885527 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50c40d24f0 msgr2=0x7f50c40d2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:16.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.777+0000 7f50cb400640 1 --2- 172.21.15.47:0/804885527 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50c40d24f0 0x7f50c40d2920 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f50c4068780 tx=0x7f50bc01c930 comp rx=0 tx=0).stop 2024-02-19T15:25:16.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.781+0000 7f50cb400640 1 -- 172.21.15.47:0/804885527 shutdown_connections 2024-02-19T15:25:16.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.781+0000 7f50cb400640 1 --2- 172.21.15.47:0/804885527 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50c40d24f0 0x7f50c40d2920 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:16.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.781+0000 7f50cb400640 1 --2- 172.21.15.47:0/804885527 >> v2:172.21.15.116:3300/0 conn(0x7f50c415f780 0x7f50c415fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:16.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.781+0000 7f50cb400640 1 -- 172.21.15.47:0/804885527 >> 172.21.15.47:0/804885527 conn(0x7f50c40d3d90 msgr2=0x7f50c40d41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:16.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.781+0000 7f50cb400640 1 -- 172.21.15.47:0/804885527 shutdown_connections 2024-02-19T15:25:16.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.781+0000 7f50cb400640 1 -- 172.21.15.47:0/804885527 wait complete. 2024-02-19T15:25:16.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.781+0000 7f50cb400640 1 Processor -- start 2024-02-19T15:25:16.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.781+0000 7f50cb400640 1 -- start start 2024-02-19T15:25:16.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50cb400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f50c40d24f0 0x7f50c4159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:16.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50cb400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50c415e000 0x7f50c4150a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:16.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50cb400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c415f780 0x7f50c4154410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:16.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50cb400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c4150f90 con 0x7f50c415e000 2024-02-19T15:25:16.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50cb400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c415ab30 con 0x7f50c415f780 2024-02-19T15:25:16.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50cb400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c4159a90 con 0x7f50c40d24f0 2024-02-19T15:25:16.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50c3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50c415e000 0x7f50c4150a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:16.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50c3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50c415e000 0x7f50c4150a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55154/0 (socket says 172.21.15.47:55154) 2024-02-19T15:25:16.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50c3e00640 1 -- 172.21.15.47:0/1261203113 learned_addr learned my addr 172.21.15.47:0/1261203113 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:16.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50c8a00640 1 --2- 172.21.15.47:0/1261203113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f50c40d24f0 0x7f50c4159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:16.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50c9400640 1 --2- 172.21.15.47:0/1261203113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c415f780 0x7f50c4154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:16.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50c3e00640 1 -- 172.21.15.47:0/1261203113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f50c40d24f0 msgr2=0x7f50c4159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:16.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50c3e00640 1 --2- 172.21.15.47:0/1261203113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f50c40d24f0 0x7f50c4159490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:16.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50c3e00640 1 -- 172.21.15.47:0/1261203113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c415f780 msgr2=0x7f50c4154410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:16.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50c3e00640 1 --2- 172.21.15.47:0/1261203113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c415f780 0x7f50c4154410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:16.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50c3e00640 1 -- 172.21.15.47:0/1261203113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50bc009b30 con 0x7f50c415e000 2024-02-19T15:25:16.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50c3e00640 1 --2- 172.21.15.47:0/1261203113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50c415e000 0x7f50c4150a50 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f50b000e870 tx=0x7f50b000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:16.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50c0c00640 1 -- 172.21.15.47:0/1261203113 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0009510 con 0x7f50c415e000 2024-02-19T15:25:16.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50c0c00640 1 -- 172.21.15.47:0/1261203113 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b000eec0 con 0x7f50c415e000 2024-02-19T15:25:16.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50c0c00640 1 -- 172.21.15.47:0/1261203113 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b000cb40 con 0x7f50c415e000 2024-02-19T15:25:16.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50cb400640 1 -- 172.21.15.47:0/1261203113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c4159d70 con 0x7f50c415e000 2024-02-19T15:25:16.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50cb400640 1 -- 172.21.15.47:0/1261203113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c415a330 con 0x7f50c415e000 2024-02-19T15:25:16.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.785+0000 7f50cb400640 1 -- 172.21.15.47:0/1261203113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c40d3750 con 0x7f50c415e000 2024-02-19T15:25:16.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.789+0000 7f50c0c00640 1 -- 172.21.15.47:0/1261203113 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f50b000ccc0 con 0x7f50c415e000 2024-02-19T15:25:16.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.789+0000 7f50c0c00640 1 --2- 172.21.15.47:0/1261203113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f50a4066e40 0x7f50a4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:16.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.789+0000 7f50c0c00640 1 -- 172.21.15.47:0/1261203113 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50b0084b70 con 0x7f50c415e000 2024-02-19T15:25:16.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.789+0000 7f50c8a00640 1 --2- 172.21.15.47:0/1261203113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f50a4066e40 0x7f50a4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:16.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.789+0000 7f50c8a00640 1 --2- 172.21.15.47:0/1261203113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f50a4066e40 0x7f50a4069300 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f50b8000f70 tx=0x7f50b8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:16.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.793+0000 7f50c0c00640 1 -- 172.21.15.47:0/1261203113 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f50b0051970 con 0x7f50c415e000 2024-02-19T15:25:16.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.953+0000 7f50cb400640 1 -- 172.21.15.47:0/1261203113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f50c4075360 con 0x7f50c415e000 2024-02-19T15:25:16.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.957+0000 7f50c0c00640 1 -- 172.21.15.47:0/1261203113 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v23)=0 v23) v1 ==== 124+0+0 (secure 0 0 0) 0x7f50b00170c0 con 0x7f50c415e000 2024-02-19T15:25:16.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.961+0000 7f50cb400640 1 -- 172.21.15.47:0/1261203113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f50a4066e40 msgr2=0x7f50a4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:16.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.961+0000 7f50cb400640 1 --2- 172.21.15.47:0/1261203113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f50a4066e40 0x7f50a4069300 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f50b8000f70 tx=0x7f50b8009040 comp rx=0 tx=0).stop 2024-02-19T15:25:16.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.961+0000 7f50cb400640 1 -- 172.21.15.47:0/1261203113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50c415e000 msgr2=0x7f50c4150a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:16.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.961+0000 7f50cb400640 1 --2- 172.21.15.47:0/1261203113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50c415e000 0x7f50c4150a50 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f50b000e870 tx=0x7f50b000ec70 comp rx=0 tx=0).stop 2024-02-19T15:25:16.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.961+0000 7f50cb400640 1 -- 172.21.15.47:0/1261203113 shutdown_connections 2024-02-19T15:25:16.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.961+0000 7f50cb400640 1 --2- 172.21.15.47:0/1261203113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f50a4066e40 0x7f50a4069300 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:16.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.961+0000 7f50cb400640 1 --2- 172.21.15.47:0/1261203113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c415f780 0x7f50c4154410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:16.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.961+0000 7f50cb400640 1 --2- 172.21.15.47:0/1261203113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50c415e000 0x7f50c4150a50 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:16.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.961+0000 7f50cb400640 1 --2- 172.21.15.47:0/1261203113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f50c40d24f0 0x7f50c4159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:16.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.961+0000 7f50cb400640 1 -- 172.21.15.47:0/1261203113 >> 172.21.15.47:0/1261203113 conn(0x7f50c40d3d90 msgr2=0x7f50c406b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:16.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.961+0000 7f50cb400640 1 -- 172.21.15.47:0/1261203113 shutdown_connections 2024-02-19T15:25:16.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:16.961+0000 7f50cb400640 1 -- 172.21.15.47:0/1261203113 wait complete. 2024-02-19T15:25:26.985 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:25:27.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.125+0000 7f8efe200640 1 Processor -- start 2024-02-19T15:25:27.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.125+0000 7f8efe200640 1 -- start start 2024-02-19T15:25:27.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.125+0000 7f8efe200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8ef80cc0d0 0x7f8ef806d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:27.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.125+0000 7f8efe200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8ef806dff0 0x7f8ef8071fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:27.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.125+0000 7f8efe200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef806c880 con 0x7f8ef80cc680 2024-02-19T15:25:27.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.125+0000 7f8efe200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ef8159c80 con 0x7f8ef806dff0 2024-02-19T15:25:27.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.125+0000 7f8efe200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ef80cc4b0 con 0x7f8ef80cc0d0 2024-02-19T15:25:27.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.125+0000 7f8ef7e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8ef806dff0 0x7f8ef8071fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:27.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.125+0000 7f8efce00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8ef80cc0d0 0x7f8ef806d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:27.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.125+0000 7f8efd800640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8ef80cc680 0x7f8ef815c170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:42442/0 (socket says 172.21.15.47:42442) 2024-02-19T15:25:27.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.125+0000 7f8ef7e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8ef806dff0 0x7f8ef8071fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60664/0 (socket says 172.21.15.47:60664) 2024-02-19T15:25:27.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.125+0000 7f8efd800640 1 -- 172.21.15.47:0/2073584483 learned_addr learned my addr 172.21.15.47:0/2073584483 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:27.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.125+0000 7f8ef7e00640 1 -- 172.21.15.47:0/2073584483 >> v2:172.21.15.116:3300/0 conn(0x7f8ef80cc0d0 msgr2=0x7f8ef806d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:27.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.125+0000 7f8ef7e00640 1 --2- 172.21.15.47:0/2073584483 >> v2:172.21.15.116:3300/0 conn(0x7f8ef80cc0d0 0x7f8ef806d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:27.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.125+0000 7f8ef7e00640 1 -- 172.21.15.47:0/2073584483 >> v1:172.21.15.116:6789/0 conn(0x7f8ef80cc680 legacy=0x7f8ef815c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:27.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.125+0000 7f8ef7e00640 1 -- 172.21.15.47:0/2073584483 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef8072900 con 0x7f8ef806dff0 2024-02-19T15:25:27.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.133+0000 7f8ef7e00640 1 --2- 172.21.15.47:0/2073584483 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ef806dff0 0x7f8ef8071fb0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8072fb0 tx=0x7f8ee401c280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd74db0b8ff6cf8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:27.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.133+0000 7f8ef7400640 1 -- 172.21.15.47:0/2073584483 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee40ab600 con 0x7f8ef806dff0 2024-02-19T15:25:27.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.133+0000 7f8ef7400640 1 -- 172.21.15.47:0/2073584483 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee40a7050 con 0x7f8ef806dff0 2024-02-19T15:25:27.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.133+0000 7f8ef7400640 1 -- 172.21.15.47:0/2073584483 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee40ab980 con 0x7f8ef806dff0 2024-02-19T15:25:27.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.133+0000 7f8efe200640 1 -- 172.21.15.47:0/2073584483 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ef806dff0 msgr2=0x7f8ef8071fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:27.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.133+0000 7f8efe200640 1 --2- 172.21.15.47:0/2073584483 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ef806dff0 0x7f8ef8071fb0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8072fb0 tx=0x7f8ee401c280 comp rx=0 tx=0).stop 2024-02-19T15:25:27.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.133+0000 7f8efe200640 1 -- 172.21.15.47:0/2073584483 shutdown_connections 2024-02-19T15:25:27.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.133+0000 7f8efe200640 1 --2- 172.21.15.47:0/2073584483 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ef806dff0 0x7f8ef8071fb0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:27.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.133+0000 7f8efe200640 1 --2- 172.21.15.47:0/2073584483 >> v2:172.21.15.116:3300/0 conn(0x7f8ef80cc0d0 0x7f8ef806d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:27.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.133+0000 7f8efe200640 1 -- 172.21.15.47:0/2073584483 >> 172.21.15.47:0/2073584483 conn(0x7f8ef80b1840 msgr2=0x7f8ef80b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:27.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.133+0000 7f8efe200640 1 -- 172.21.15.47:0/2073584483 shutdown_connections 2024-02-19T15:25:27.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.133+0000 7f8efe200640 1 -- 172.21.15.47:0/2073584483 wait complete. 2024-02-19T15:25:27.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.133+0000 7f8efe200640 1 Processor -- start 2024-02-19T15:25:27.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.133+0000 7f8efe200640 1 -- start start 2024-02-19T15:25:27.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.133+0000 7f8efe200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ef806dff0 0x7f8ef81f79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:27.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.133+0000 7f8efe200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ef80cc0d0 0x7f8ef80adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:27.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.133+0000 7f8efe200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ef80cc680 0x7f8ef8150c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:27.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.133+0000 7f8efe200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef80c3540 con 0x7f8ef806dff0 2024-02-19T15:25:27.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.133+0000 7f8efe200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef815a260 con 0x7f8ef80cc680 2024-02-19T15:25:27.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.133+0000 7f8efe200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef815a540 con 0x7f8ef80cc0d0 2024-02-19T15:25:27.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.137+0000 7f8efce00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ef806dff0 0x7f8ef81f79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:27.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.137+0000 7f8efce00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ef806dff0 0x7f8ef81f79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60678/0 (socket says 172.21.15.47:60678) 2024-02-19T15:25:27.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.137+0000 7f8efce00640 1 -- 172.21.15.47:0/234119610 learned_addr learned my addr 172.21.15.47:0/234119610 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:27.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.137+0000 7f8efd800640 1 --2- 172.21.15.47:0/234119610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ef80cc680 0x7f8ef8150c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:27.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.137+0000 7f8ef7e00640 1 --2- 172.21.15.47:0/234119610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ef80cc0d0 0x7f8ef80adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:27.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.137+0000 7f8efce00640 1 -- 172.21.15.47:0/234119610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ef80cc0d0 msgr2=0x7f8ef80adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:27.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.137+0000 7f8efce00640 1 --2- 172.21.15.47:0/234119610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ef80cc0d0 0x7f8ef80adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:27.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.137+0000 7f8efce00640 1 -- 172.21.15.47:0/234119610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ef80cc680 msgr2=0x7f8ef8150c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:27.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.137+0000 7f8efce00640 1 --2- 172.21.15.47:0/234119610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ef80cc680 0x7f8ef8150c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:27.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.137+0000 7f8efce00640 1 -- 172.21.15.47:0/234119610 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee4009420 con 0x7f8ef806dff0 2024-02-19T15:25:27.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.137+0000 7f8efce00640 1 --2- 172.21.15.47:0/234119610 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ef806dff0 0x7f8ef81f79e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f8eec00e870 tx=0x7f8eec00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:27.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.137+0000 7f8ef4c00640 1 -- 172.21.15.47:0/234119610 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec009510 con 0x7f8ef806dff0 2024-02-19T15:25:27.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.137+0000 7f8ef4c00640 1 -- 172.21.15.47:0/234119610 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eec00eec0 con 0x7f8ef806dff0 2024-02-19T15:25:27.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.137+0000 7f8ef4c00640 1 -- 172.21.15.47:0/234119610 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec00cb40 con 0x7f8ef806dff0 2024-02-19T15:25:27.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.137+0000 7f8efe200640 1 -- 172.21.15.47:0/234119610 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef815a820 con 0x7f8ef806dff0 2024-02-19T15:25:27.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.137+0000 7f8efe200640 1 -- 172.21.15.47:0/234119610 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef81593d0 con 0x7f8ef806dff0 2024-02-19T15:25:27.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.137+0000 7f8efe200640 1 -- 172.21.15.47:0/234119610 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ef8075360 con 0x7f8ef806dff0 2024-02-19T15:25:27.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.137+0000 7f8ef4c00640 1 -- 172.21.15.47:0/234119610 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8eec00cca0 con 0x7f8ef806dff0 2024-02-19T15:25:27.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.141+0000 7f8ef4c00640 1 --2- 172.21.15.47:0/234119610 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ed8066e40 0x7f8ed8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:27.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.141+0000 7f8ef4c00640 1 -- 172.21.15.47:0/234119610 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8eec084b40 con 0x7f8ef806dff0 2024-02-19T15:25:27.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.141+0000 7f8ef7e00640 1 --2- 172.21.15.47:0/234119610 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ed8066e40 0x7f8ed8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:27.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.141+0000 7f8ef7e00640 1 --2- 172.21.15.47:0/234119610 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ed8066e40 0x7f8ed8069300 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40ad000 tx=0x7f8ee40a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:27.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.149+0000 7f8ef4c00640 1 -- 172.21.15.47:0/234119610 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8eec051940 con 0x7f8ef806dff0 2024-02-19T15:25:27.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.345+0000 7f8efe200640 1 -- 172.21.15.47:0/234119610 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ef815acd0 con 0x7f8ef806dff0 2024-02-19T15:25:27.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.345+0000 7f8ef4c00640 1 -- 172.21.15.47:0/234119610 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4946 (secure 0 0 0) 0x7f8eec017090 con 0x7f8ef806dff0 2024-02-19T15:25:27.348 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:27.348 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5684,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3206627105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3206627105},{"type":"v1","addr":"172.21.15.116:6836","nonce":3206627105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5687,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3532185042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3532185042},{"type":"v1","addr":"172.21.15.116:6837","nonce":3532185042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/1482478568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1482478568},{"type":"v1","addr":"172.21.15.47:6834","nonce":1482478568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:25:08.947740+0000","modified":"2024-02-19T15:25:11.954116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/2490197058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2490197058},{"type":"v1","addr":"172.21.15.47:6835","nonce":2490197058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-02-19T15:25:27.348 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 40 2024-02-19T15:25:27.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.345+0000 7f8efe200640 1 -- 172.21.15.47:0/234119610 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ed8066e40 msgr2=0x7f8ed8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:27.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.345+0000 7f8efe200640 1 --2- 172.21.15.47:0/234119610 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ed8066e40 0x7f8ed8069300 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40ad000 tx=0x7f8ee40a6000 comp rx=0 tx=0).stop 2024-02-19T15:25:27.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.345+0000 7f8efe200640 1 -- 172.21.15.47:0/234119610 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ef806dff0 msgr2=0x7f8ef81f79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:27.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.345+0000 7f8efe200640 1 --2- 172.21.15.47:0/234119610 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ef806dff0 0x7f8ef81f79e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f8eec00e870 tx=0x7f8eec00ec70 comp rx=0 tx=0).stop 2024-02-19T15:25:27.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.345+0000 7f8efe200640 1 -- 172.21.15.47:0/234119610 shutdown_connections 2024-02-19T15:25:27.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.345+0000 7f8efe200640 1 --2- 172.21.15.47:0/234119610 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ed8066e40 0x7f8ed8069300 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:27.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.345+0000 7f8efe200640 1 --2- 172.21.15.47:0/234119610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ef80cc680 0x7f8ef8150c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:27.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.345+0000 7f8efe200640 1 --2- 172.21.15.47:0/234119610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8ef80cc0d0 0x7f8ef80adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:27.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.345+0000 7f8efe200640 1 --2- 172.21.15.47:0/234119610 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ef806dff0 0x7f8ef81f79e0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:27.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.345+0000 7f8efe200640 1 -- 172.21.15.47:0/234119610 >> 172.21.15.47:0/234119610 conn(0x7f8ef80b1840 msgr2=0x7f8ef80cd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:27.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.345+0000 7f8efe200640 1 -- 172.21.15.47:0/234119610 shutdown_connections 2024-02-19T15:25:27.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.345+0000 7f8efe200640 1 -- 172.21.15.47:0/234119610 wait complete. 2024-02-19T15:25:27.370 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-02-19T15:25:27.530 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"strays_created":0}} 2024-02-19T15:25:27.530 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 0 } } 2024-02-19T15:25:27.531 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:25:27.531 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:25:27.539 DEBUG:teuthology.orchestra.run.smithi047:> stdin-killer -- python3 -c ' 2024-02-19T15:25:27.539 DEBUG:teuthology.orchestra.run.smithi047:> import os 2024-02-19T15:25:27.540 DEBUG:teuthology.orchestra.run.smithi047:> import uuid 2024-02-19T15:25:27.540 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:25:27.540 DEBUG:teuthology.orchestra.run.smithi047:> n = 100 2024-02-19T15:25:27.540 DEBUG:teuthology.orchestra.run.smithi047:> create_hard_links = False 2024-02-19T15:25:27.540 DEBUG:teuthology.orchestra.run.smithi047:> if 0 > 0: 2024-02-19T15:25:27.540 DEBUG:teuthology.orchestra.run.smithi047:> create_hard_links = True 2024-02-19T15:25:27.540 DEBUG:teuthology.orchestra.run.smithi047:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-02-19T15:25:27.540 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:25:27.540 DEBUG:teuthology.orchestra.run.smithi047:> dpath = os.path.dirname(path) 2024-02-19T15:25:27.540 DEBUG:teuthology.orchestra.run.smithi047:> fnameprefix = os.path.basename(path) 2024-02-19T15:25:27.540 DEBUG:teuthology.orchestra.run.smithi047:> os.makedirs(dpath, exist_ok=True) 2024-02-19T15:25:27.540 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:25:27.540 DEBUG:teuthology.orchestra.run.smithi047:> try: 2024-02-19T15:25:27.540 DEBUG:teuthology.orchestra.run.smithi047:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-02-19T15:25:27.540 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:25:27.540 DEBUG:teuthology.orchestra.run.smithi047:> for i in range(n): 2024-02-19T15:25:27.540 DEBUG:teuthology.orchestra.run.smithi047:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-02-19T15:25:27.541 DEBUG:teuthology.orchestra.run.smithi047:> with open(fpath, '"'"'w'"'"') as f: 2024-02-19T15:25:27.541 DEBUG:teuthology.orchestra.run.smithi047:> f.write(f"{i}") 2024-02-19T15:25:27.541 DEBUG:teuthology.orchestra.run.smithi047:> if False: 2024-02-19T15:25:27.541 DEBUG:teuthology.orchestra.run.smithi047:> f.flush() 2024-02-19T15:25:27.541 DEBUG:teuthology.orchestra.run.smithi047:> os.fsync(f.fileno()) 2024-02-19T15:25:27.541 DEBUG:teuthology.orchestra.run.smithi047:> if True: 2024-02-19T15:25:27.541 DEBUG:teuthology.orchestra.run.smithi047:> os.unlink(fpath) 2024-02-19T15:25:27.541 DEBUG:teuthology.orchestra.run.smithi047:> if False: 2024-02-19T15:25:27.541 DEBUG:teuthology.orchestra.run.smithi047:> os.fsync(dirfd) 2024-02-19T15:25:27.541 DEBUG:teuthology.orchestra.run.smithi047:> if create_hard_links: 2024-02-19T15:25:27.541 DEBUG:teuthology.orchestra.run.smithi047:> for j in range(0): 2024-02-19T15:25:27.541 DEBUG:teuthology.orchestra.run.smithi047:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-02-19T15:25:27.541 DEBUG:teuthology.orchestra.run.smithi047:> if True: 2024-02-19T15:25:27.541 DEBUG:teuthology.orchestra.run.smithi047:> os.fsync(dirfd) 2024-02-19T15:25:27.541 DEBUG:teuthology.orchestra.run.smithi047:> finally: 2024-02-19T15:25:27.541 DEBUG:teuthology.orchestra.run.smithi047:> os.close(dirfd) 2024-02-19T15:25:27.541 DEBUG:teuthology.orchestra.run.smithi047:> ' 2024-02-19T15:25:27.697 INFO:teuthology.orchestra.run.smithi047.stderr:Traceback (most recent call last): 2024-02-19T15:25:27.697 INFO:teuthology.orchestra.run.smithi047.stderr: File "", line 26, in 2024-02-19T15:25:27.697 INFO:teuthology.orchestra.run.smithi047.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/file_10' 2024-02-19T15:25:27.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-02-19T15:25:27.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:25:27.708 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:25:27.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.889+0000 7fb728a00640 1 Processor -- start 2024-02-19T15:25:27.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.889+0000 7fb728a00640 1 -- start start 2024-02-19T15:25:27.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.889+0000 7fb728a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb7241456a0 0x7fb7240cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:27.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.889+0000 7fb728a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb72406b5c0 0x7fb72406dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:27.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.889+0000 7fb728a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb724145a80 con 0x7fb7240cf4a0 2024-02-19T15:25:27.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.889+0000 7fb728a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7240d03c0 con 0x7fb7241456a0 2024-02-19T15:25:27.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.889+0000 7fb728a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7240cf900 con 0x7fb72406b5c0 2024-02-19T15:25:27.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.893+0000 7fb719400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb7240cf4a0 0x7fb72406aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52150/0 (socket says 172.21.15.47:52150) 2024-02-19T15:25:27.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.893+0000 7fb719400640 1 -- 172.21.15.47:0/3707812461 learned_addr learned my addr 172.21.15.47:0/3707812461 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:27.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.893+0000 7fb722800640 1 --2- 172.21.15.47:0/3707812461 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb72406b5c0 0x7fb72406dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:27.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.893+0000 7fb721e00640 1 --2- 172.21.15.47:0/3707812461 >> v2:172.21.15.116:3300/0 conn(0x7fb7241456a0 0x7fb7240cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:27.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.893+0000 7fb721400640 1 -- 172.21.15.47:0/3707812461 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338314312 0 0) 0x7fb724145a80 con 0x7fb7240cf4a0 2024-02-19T15:25:27.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.893+0000 7fb721400640 1 -- 172.21.15.47:0/3707812461 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb708003530 con 0x7fb7240cf4a0 2024-02-19T15:25:27.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.893+0000 7fb722800640 1 -- 172.21.15.47:0/3707812461 >> v2:172.21.15.116:3300/0 conn(0x7fb7241456a0 msgr2=0x7fb7240cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:27.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.893+0000 7fb722800640 1 --2- 172.21.15.47:0/3707812461 >> v2:172.21.15.116:3300/0 conn(0x7fb7241456a0 0x7fb7240cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:27.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.893+0000 7fb722800640 1 -- 172.21.15.47:0/3707812461 >> v1:172.21.15.47:6789/0 conn(0x7fb7240cf4a0 legacy=0x7fb72406aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:27.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.893+0000 7fb722800640 1 -- 172.21.15.47:0/3707812461 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7240d1880 con 0x7fb72406b5c0 2024-02-19T15:25:27.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.893+0000 7fb722800640 1 --2- 172.21.15.47:0/3707812461 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb72406b5c0 0x7fb72406dd60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb71c009d90 tx=0x7fb71c01c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ceadeca8be76bf1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:27.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.897+0000 7fb721400640 1 -- 172.21.15.47:0/3707812461 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71c0a5920 con 0x7fb72406b5c0 2024-02-19T15:25:27.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.897+0000 7fb721400640 1 -- 172.21.15.47:0/3707812461 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb71c0a5a80 con 0x7fb72406b5c0 2024-02-19T15:25:27.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.897+0000 7fb721400640 1 -- 172.21.15.47:0/3707812461 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71c0a5df0 con 0x7fb72406b5c0 2024-02-19T15:25:27.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.897+0000 7fb728a00640 1 -- 172.21.15.47:0/3707812461 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb72406b5c0 msgr2=0x7fb72406dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:27.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.897+0000 7fb728a00640 1 --2- 172.21.15.47:0/3707812461 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb72406b5c0 0x7fb72406dd60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb71c009d90 tx=0x7fb71c01c970 comp rx=0 tx=0).stop 2024-02-19T15:25:27.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.897+0000 7fb728a00640 1 -- 172.21.15.47:0/3707812461 shutdown_connections 2024-02-19T15:25:27.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.897+0000 7fb728a00640 1 --2- 172.21.15.47:0/3707812461 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb72406b5c0 0x7fb72406dd60 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:27.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.897+0000 7fb728a00640 1 --2- 172.21.15.47:0/3707812461 >> v2:172.21.15.116:3300/0 conn(0x7fb7241456a0 0x7fb7240cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:27.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.897+0000 7fb728a00640 1 -- 172.21.15.47:0/3707812461 >> 172.21.15.47:0/3707812461 conn(0x7fb7240d0ad0 msgr2=0x7fb7240d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:27.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.897+0000 7fb728a00640 1 -- 172.21.15.47:0/3707812461 shutdown_connections 2024-02-19T15:25:27.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.897+0000 7fb728a00640 1 -- 172.21.15.47:0/3707812461 wait complete. 2024-02-19T15:25:27.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.897+0000 7fb728a00640 1 Processor -- start 2024-02-19T15:25:27.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb728a00640 1 -- start start 2024-02-19T15:25:27.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb728a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb72406b5c0 0x7fb724165580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:27.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb728a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb7240cf4a0 0x7fb724162580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:27.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb728a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7241456a0 0x7fb72413ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:27.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb728a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb724165ac0 con 0x7fb72406b5c0 2024-02-19T15:25:27.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb728a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7241644e0 con 0x7fb7241456a0 2024-02-19T15:25:27.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb728a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7241647c0 con 0x7fb7240cf4a0 2024-02-19T15:25:27.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb721e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb72406b5c0 0x7fb724165580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:27.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb721e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb72406b5c0 0x7fb724165580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60686/0 (socket says 172.21.15.47:60686) 2024-02-19T15:25:27.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb721e00640 1 -- 172.21.15.47:0/451492941 learned_addr learned my addr 172.21.15.47:0/451492941 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:27.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb719400640 1 --2- 172.21.15.47:0/451492941 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb7240cf4a0 0x7fb724162580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:27.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb722800640 1 --2- 172.21.15.47:0/451492941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7241456a0 0x7fb72413ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:27.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb721e00640 1 -- 172.21.15.47:0/451492941 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb7240cf4a0 msgr2=0x7fb724162580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:27.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb721e00640 1 --2- 172.21.15.47:0/451492941 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb7240cf4a0 0x7fb724162580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:27.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb721e00640 1 -- 172.21.15.47:0/451492941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7241456a0 msgr2=0x7fb72413ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:27.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb721e00640 1 --2- 172.21.15.47:0/451492941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7241456a0 0x7fb72413ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:27.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb721e00640 1 -- 172.21.15.47:0/451492941 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71c009a70 con 0x7fb72406b5c0 2024-02-19T15:25:27.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb722800640 1 --2- 172.21.15.47:0/451492941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7241456a0 0x7fb72413ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:25:27.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb719400640 1 --2- 172.21.15.47:0/451492941 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb7240cf4a0 0x7fb724162580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:25:27.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb721e00640 1 --2- 172.21.15.47:0/451492941 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb72406b5c0 0x7fb724165580 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb71400e870 tx=0x7fb71400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:27.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb71aa00640 1 -- 172.21.15.47:0/451492941 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb714009510 con 0x7fb72406b5c0 2024-02-19T15:25:27.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb71aa00640 1 -- 172.21.15.47:0/451492941 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb71400eec0 con 0x7fb72406b5c0 2024-02-19T15:25:27.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb728a00640 1 -- 172.21.15.47:0/451492941 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb724164aa0 con 0x7fb72406b5c0 2024-02-19T15:25:27.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb71aa00640 1 -- 172.21.15.47:0/451492941 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71400cb40 con 0x7fb72406b5c0 2024-02-19T15:25:27.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.901+0000 7fb728a00640 1 -- 172.21.15.47:0/451492941 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb724165060 con 0x7fb72406b5c0 2024-02-19T15:25:27.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.905+0000 7fb728a00640 1 -- 172.21.15.47:0/451492941 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb724075360 con 0x7fb72406b5c0 2024-02-19T15:25:27.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.905+0000 7fb71aa00640 1 -- 172.21.15.47:0/451492941 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb71400cca0 con 0x7fb72406b5c0 2024-02-19T15:25:27.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.905+0000 7fb71aa00640 1 --2- 172.21.15.47:0/451492941 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb700066e40 0x7fb700069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:27.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.905+0000 7fb71aa00640 1 -- 172.21.15.47:0/451492941 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb714085220 con 0x7fb72406b5c0 2024-02-19T15:25:27.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.905+0000 7fb719400640 1 --2- 172.21.15.47:0/451492941 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb700066e40 0x7fb700069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:27.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.905+0000 7fb719400640 1 --2- 172.21.15.47:0/451492941 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb700066e40 0x7fb700069300 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb724160b10 tx=0x7fb70c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:27.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:27.909+0000 7fb71aa00640 1 -- 172.21.15.47:0/451492941 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb714052180 con 0x7fb72406b5c0 2024-02-19T15:25:28.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.105+0000 7fb728a00640 1 -- 172.21.15.47:0/451492941 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb724072b90 con 0x7fb72406b5c0 2024-02-19T15:25:28.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.105+0000 7fb71aa00640 1 -- 172.21.15.47:0/451492941 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4946 (secure 0 0 0) 0x7fb714055840 con 0x7fb72406b5c0 2024-02-19T15:25:28.110 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:28.110 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5684,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3206627105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3206627105},{"type":"v1","addr":"172.21.15.116:6836","nonce":3206627105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5687,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3532185042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3532185042},{"type":"v1","addr":"172.21.15.116:6837","nonce":3532185042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/1482478568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1482478568},{"type":"v1","addr":"172.21.15.47:6834","nonce":1482478568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:25:08.947740+0000","modified":"2024-02-19T15:25:11.954116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/2490197058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2490197058},{"type":"v1","addr":"172.21.15.47:6835","nonce":2490197058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-02-19T15:25:28.110 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 40 2024-02-19T15:25:28.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.109+0000 7fb728a00640 1 -- 172.21.15.47:0/451492941 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb700066e40 msgr2=0x7fb700069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:28.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.109+0000 7fb728a00640 1 --2- 172.21.15.47:0/451492941 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb700066e40 0x7fb700069300 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb724160b10 tx=0x7fb70c009040 comp rx=0 tx=0).stop 2024-02-19T15:25:28.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.109+0000 7fb728a00640 1 -- 172.21.15.47:0/451492941 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb72406b5c0 msgr2=0x7fb724165580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:28.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.109+0000 7fb728a00640 1 --2- 172.21.15.47:0/451492941 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb72406b5c0 0x7fb724165580 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb71400e870 tx=0x7fb71400ec70 comp rx=0 tx=0).stop 2024-02-19T15:25:28.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.109+0000 7fb728a00640 1 -- 172.21.15.47:0/451492941 shutdown_connections 2024-02-19T15:25:28.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.109+0000 7fb728a00640 1 --2- 172.21.15.47:0/451492941 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb700066e40 0x7fb700069300 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:28.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.109+0000 7fb728a00640 1 --2- 172.21.15.47:0/451492941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7241456a0 0x7fb72413ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:28.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.109+0000 7fb728a00640 1 --2- 172.21.15.47:0/451492941 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb7240cf4a0 0x7fb724162580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:28.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.109+0000 7fb728a00640 1 --2- 172.21.15.47:0/451492941 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb72406b5c0 0x7fb724165580 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:28.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.109+0000 7fb728a00640 1 -- 172.21.15.47:0/451492941 >> 172.21.15.47:0/451492941 conn(0x7fb7240d0ad0 msgr2=0x7fb72406fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:28.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.109+0000 7fb728a00640 1 -- 172.21.15.47:0/451492941 shutdown_connections 2024-02-19T15:25:28.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.109+0000 7fb728a00640 1 -- 172.21.15.47:0/451492941 wait complete. 2024-02-19T15:25:28.134 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-02-19T15:25:28.293 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"strays_created":10}} 2024-02-19T15:25:28.294 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 10 } } 2024-02-19T15:25:28.294 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:25:28.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.429+0000 7fb60a200640 1 Processor -- start 2024-02-19T15:25:28.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.429+0000 7fb60a200640 1 -- start start 2024-02-19T15:25:28.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.433+0000 7fb60a200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb6040cedb0 0x7fb6040cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:28.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.433+0000 7fb60a200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6040cc700 con 0x7fb6040d2710 2024-02-19T15:25:28.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.433+0000 7fb60a200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6040cf300 con 0x7fb60415bcd0 2024-02-19T15:25:28.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.433+0000 7fb60a200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6040cf5e0 con 0x7fb6040cedb0 2024-02-19T15:25:28.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.433+0000 7fb602a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb6040cedb0 0x7fb6040cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:28.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.433+0000 7fb602a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb6040cedb0 0x7fb6040cfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60702/0 (socket says 172.21.15.47:60702) 2024-02-19T15:25:28.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.433+0000 7fb603e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb6040d2710 0x7fb60415d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52164/0 (socket says 172.21.15.47:52164) 2024-02-19T15:25:28.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.433+0000 7fb602a00640 1 -- 172.21.15.47:0/788456181 learned_addr learned my addr 172.21.15.47:0/788456181 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:28.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.433+0000 7fb602a00640 1 -- 172.21.15.47:0/788456181 >> v1:172.21.15.116:6789/0 conn(0x7fb60415bcd0 legacy=0x7fb6040cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:28.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.433+0000 7fb602a00640 1 -- 172.21.15.47:0/788456181 >> v1:172.21.15.47:6789/0 conn(0x7fb6040d2710 legacy=0x7fb60415d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:28.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.433+0000 7fb602a00640 1 -- 172.21.15.47:0/788456181 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6040cc9a0 con 0x7fb6040cedb0 2024-02-19T15:25:28.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.437+0000 7fb602a00640 1 --2- 172.21.15.47:0/788456181 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6040cedb0 0x7fb6040cfc50 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0009870 tx=0x7fb5f001c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85ce7281d247b2ba server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:28.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.437+0000 7fb602000640 1 -- 172.21.15.47:0/788456181 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f00ab640 con 0x7fb6040cedb0 2024-02-19T15:25:28.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.437+0000 7fb602000640 1 -- 172.21.15.47:0/788456181 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f00a7070 con 0x7fb6040cedb0 2024-02-19T15:25:28.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.437+0000 7fb602000640 1 -- 172.21.15.47:0/788456181 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f00ab9c0 con 0x7fb6040cedb0 2024-02-19T15:25:28.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.437+0000 7fb60a200640 1 -- 172.21.15.47:0/788456181 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6040cedb0 msgr2=0x7fb6040cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:28.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.437+0000 7fb60a200640 1 --2- 172.21.15.47:0/788456181 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6040cedb0 0x7fb6040cfc50 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0009870 tx=0x7fb5f001c530 comp rx=0 tx=0).stop 2024-02-19T15:25:28.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.437+0000 7fb60a200640 1 -- 172.21.15.47:0/788456181 shutdown_connections 2024-02-19T15:25:28.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.437+0000 7fb60a200640 1 --2- 172.21.15.47:0/788456181 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6040cedb0 0x7fb6040cfc50 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:28.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.437+0000 7fb60a200640 1 -- 172.21.15.47:0/788456181 >> 172.21.15.47:0/788456181 conn(0x7fb6040d1bf0 msgr2=0x7fb6040d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:28.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.437+0000 7fb60a200640 1 -- 172.21.15.47:0/788456181 shutdown_connections 2024-02-19T15:25:28.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.437+0000 7fb60a200640 1 -- 172.21.15.47:0/788456181 wait complete. 2024-02-19T15:25:28.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.437+0000 7fb60a200640 1 Processor -- start 2024-02-19T15:25:28.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.437+0000 7fb60a200640 1 -- start start 2024-02-19T15:25:28.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.437+0000 7fb60a200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6040cedb0 0x7fb6040c5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:28.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.437+0000 7fb60a200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6040d2710 0x7fb6040c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:28.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.437+0000 7fb60a200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb60415bcd0 0x7fb604142390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:28.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.437+0000 7fb60a200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6040cdb90 con 0x7fb6040d2710 2024-02-19T15:25:28.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.437+0000 7fb60a200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6040c57c0 con 0x7fb60415bcd0 2024-02-19T15:25:28.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.437+0000 7fb60a200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb604142f10 con 0x7fb6040cedb0 2024-02-19T15:25:28.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.441+0000 7fb602a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6040d2710 0x7fb6040c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:28.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.441+0000 7fb602a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6040d2710 0x7fb6040c7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60718/0 (socket says 172.21.15.47:60718) 2024-02-19T15:25:28.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.441+0000 7fb602a00640 1 -- 172.21.15.47:0/4038820894 learned_addr learned my addr 172.21.15.47:0/4038820894 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:28.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.441+0000 7fb603400640 1 --2- 172.21.15.47:0/4038820894 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6040cedb0 0x7fb6040c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:28.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.441+0000 7fb603e00640 1 --2- 172.21.15.47:0/4038820894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb60415bcd0 0x7fb604142390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:28.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.441+0000 7fb602a00640 1 -- 172.21.15.47:0/4038820894 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6040cedb0 msgr2=0x7fb6040c5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:28.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.441+0000 7fb602a00640 1 --2- 172.21.15.47:0/4038820894 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6040cedb0 0x7fb6040c5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:28.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.441+0000 7fb602a00640 1 -- 172.21.15.47:0/4038820894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb60415bcd0 msgr2=0x7fb604142390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:28.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.441+0000 7fb602a00640 1 --2- 172.21.15.47:0/4038820894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb60415bcd0 0x7fb604142390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:28.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.441+0000 7fb602a00640 1 -- 172.21.15.47:0/4038820894 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f00094d0 con 0x7fb6040d2710 2024-02-19T15:25:28.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.441+0000 7fb603e00640 1 --2- 172.21.15.47:0/4038820894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb60415bcd0 0x7fb604142390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:28.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.441+0000 7fb602a00640 1 --2- 172.21.15.47:0/4038820894 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6040d2710 0x7fb6040c7b10 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0019140 tx=0x7fb5f001ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:28.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.441+0000 7fb5fb400640 1 -- 172.21.15.47:0/4038820894 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f00be3f0 con 0x7fb6040d2710 2024-02-19T15:25:28.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.441+0000 7fb5fb400640 1 -- 172.21.15.47:0/4038820894 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f00be570 con 0x7fb6040d2710 2024-02-19T15:25:28.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.441+0000 7fb5fb400640 1 -- 172.21.15.47:0/4038820894 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f00be920 con 0x7fb6040d2710 2024-02-19T15:25:28.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.441+0000 7fb60a200640 1 -- 172.21.15.47:0/4038820894 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60415d240 con 0x7fb6040d2710 2024-02-19T15:25:28.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.441+0000 7fb60a200640 1 -- 172.21.15.47:0/4038820894 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6040c7310 con 0x7fb6040d2710 2024-02-19T15:25:28.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.441+0000 7fb60a200640 1 -- 172.21.15.47:0/4038820894 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb604075360 con 0x7fb6040d2710 2024-02-19T15:25:28.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.441+0000 7fb5fb400640 1 -- 172.21.15.47:0/4038820894 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb5f00b43f0 con 0x7fb6040d2710 2024-02-19T15:25:28.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.445+0000 7fb5fb400640 1 --2- 172.21.15.47:0/4038820894 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb5e4066e40 0x7fb5e4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:28.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.445+0000 7fb5fb400640 1 -- 172.21.15.47:0/4038820894 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5f011ee60 con 0x7fb6040d2710 2024-02-19T15:25:28.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.445+0000 7fb603400640 1 --2- 172.21.15.47:0/4038820894 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb5e4066e40 0x7fb5e4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:28.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.445+0000 7fb603400640 1 --2- 172.21.15.47:0/4038820894 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb5e4066e40 0x7fb5e4069300 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb6041584c0 tx=0x7fb5f4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:28.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.449+0000 7fb5fb400640 1 -- 172.21.15.47:0/4038820894 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb5f00ebc60 con 0x7fb6040d2710 2024-02-19T15:25:28.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.677+0000 7fb60a200640 1 -- 172.21.15.47:0/4038820894 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6040b3380 con 0x7fb6040d2710 2024-02-19T15:25:28.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.681+0000 7fb5fb400640 1 -- 172.21.15.47:0/4038820894 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4946 (secure 0 0 0) 0x7fb5f00c4090 con 0x7fb6040d2710 2024-02-19T15:25:28.683 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:28.683 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5684,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3206627105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3206627105},{"type":"v1","addr":"172.21.15.116:6836","nonce":3206627105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5687,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3532185042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3532185042},{"type":"v1","addr":"172.21.15.116:6837","nonce":3532185042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/1482478568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1482478568},{"type":"v1","addr":"172.21.15.47:6834","nonce":1482478568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:25:08.947740+0000","modified":"2024-02-19T15:25:11.954116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/2490197058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2490197058},{"type":"v1","addr":"172.21.15.47:6835","nonce":2490197058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-02-19T15:25:28.684 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 40 2024-02-19T15:25:28.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.681+0000 7fb60a200640 1 -- 172.21.15.47:0/4038820894 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb5e4066e40 msgr2=0x7fb5e4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:28.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.681+0000 7fb60a200640 1 --2- 172.21.15.47:0/4038820894 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb5e4066e40 0x7fb5e4069300 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb6041584c0 tx=0x7fb5f4009040 comp rx=0 tx=0).stop 2024-02-19T15:25:28.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.681+0000 7fb60a200640 1 -- 172.21.15.47:0/4038820894 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6040d2710 msgr2=0x7fb6040c7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:28.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.681+0000 7fb60a200640 1 --2- 172.21.15.47:0/4038820894 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6040d2710 0x7fb6040c7b10 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0019140 tx=0x7fb5f001ad70 comp rx=0 tx=0).stop 2024-02-19T15:25:28.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.681+0000 7fb60a200640 1 -- 172.21.15.47:0/4038820894 shutdown_connections 2024-02-19T15:25:28.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.681+0000 7fb60a200640 1 --2- 172.21.15.47:0/4038820894 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb5e4066e40 0x7fb5e4069300 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:28.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.681+0000 7fb60a200640 1 --2- 172.21.15.47:0/4038820894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb60415bcd0 0x7fb604142390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:28.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.681+0000 7fb60a200640 1 --2- 172.21.15.47:0/4038820894 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6040d2710 0x7fb6040c7b10 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:28.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.681+0000 7fb60a200640 1 --2- 172.21.15.47:0/4038820894 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6040cedb0 0x7fb6040c5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:28.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.681+0000 7fb60a200640 1 -- 172.21.15.47:0/4038820894 >> 172.21.15.47:0/4038820894 conn(0x7fb6040d1bf0 msgr2=0x7fb60406c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:28.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.681+0000 7fb60a200640 1 -- 172.21.15.47:0/4038820894 shutdown_connections 2024-02-19T15:25:28.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:28.681+0000 7fb60a200640 1 -- 172.21.15.47:0/4038820894 wait complete. 2024-02-19T15:25:28.706 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-02-19T15:25:28.866 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-02-19T15:25:28.866 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-02-19T15:25:28.866 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 10, waiting (timeout=60)... 2024-02-19T15:25:33.868 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:25:34.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.005+0000 7f91e8e00640 1 Processor -- start 2024-02-19T15:25:34.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.005+0000 7f91e8e00640 1 -- start start 2024-02-19T15:25:34.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.005+0000 7f91e8e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f91e40ce3c0 0x7f91e40ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:34.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.009+0000 7f91e8e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f91e415e570 0x7f91e406c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:34.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.009+0000 7f91e8e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e40d2ad0 con 0x7f91e40b2e20 2024-02-19T15:25:34.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.009+0000 7f91e8e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e40cc190 con 0x7f91e40ce3c0 2024-02-19T15:25:34.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.009+0000 7f91e8e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e40cece0 con 0x7f91e415e570 2024-02-19T15:25:34.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.009+0000 7f91e1e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f91e40ce3c0 0x7f91e40ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:34.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.009+0000 7f91e2800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f91e40b2e20 0x7f91e406c830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39514/0 (socket says 172.21.15.47:39514) 2024-02-19T15:25:34.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.009+0000 7f91e2800640 1 -- 172.21.15.47:0/2161338290 learned_addr learned my addr 172.21.15.47:0/2161338290 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:34.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.009+0000 7f91dbe00640 1 --2- 172.21.15.47:0/2161338290 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f91e415e570 0x7f91e406c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:34.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.009+0000 7f91e1e00640 1 -- 172.21.15.47:0/2161338290 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f91e415e570 msgr2=0x7f91e406c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:34.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.009+0000 7f91e1e00640 1 --2- 172.21.15.47:0/2161338290 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f91e415e570 0x7f91e406c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:34.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.009+0000 7f91e1e00640 1 -- 172.21.15.47:0/2161338290 >> v1:172.21.15.47:6789/0 conn(0x7f91e40b2e20 legacy=0x7f91e406c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:34.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.009+0000 7f91e1e00640 1 -- 172.21.15.47:0/2161338290 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e40d0ac0 con 0x7f91e40ce3c0 2024-02-19T15:25:34.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.009+0000 7f91dbe00640 1 --2- 172.21.15.47:0/2161338290 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f91e415e570 0x7f91e406c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:34.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.009+0000 7f91e1e00640 1 --2- 172.21.15.47:0/2161338290 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91e40ce3c0 0x7f91e40ce7a0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f91d40097f0 tx=0x7f91d401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc65cd5c8d91879 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:34.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.009+0000 7f91e1400640 1 -- 172.21.15.47:0/2161338290 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d40ad640 con 0x7f91e40ce3c0 2024-02-19T15:25:34.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.009+0000 7f91e1400640 1 -- 172.21.15.47:0/2161338290 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d40ad7a0 con 0x7f91e40ce3c0 2024-02-19T15:25:34.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.009+0000 7f91e1400640 1 -- 172.21.15.47:0/2161338290 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d40adb30 con 0x7f91e40ce3c0 2024-02-19T15:25:34.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e8e00640 1 -- 172.21.15.47:0/2161338290 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91e40ce3c0 msgr2=0x7f91e40ce7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:34.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e8e00640 1 --2- 172.21.15.47:0/2161338290 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91e40ce3c0 0x7f91e40ce7a0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f91d40097f0 tx=0x7f91d401c4a0 comp rx=0 tx=0).stop 2024-02-19T15:25:34.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e8e00640 1 -- 172.21.15.47:0/2161338290 shutdown_connections 2024-02-19T15:25:34.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e8e00640 1 --2- 172.21.15.47:0/2161338290 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f91e415e570 0x7f91e406c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:34.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e8e00640 1 --2- 172.21.15.47:0/2161338290 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91e40ce3c0 0x7f91e40ce7a0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:34.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e8e00640 1 -- 172.21.15.47:0/2161338290 >> 172.21.15.47:0/2161338290 conn(0x7f91e40d1b40 msgr2=0x7f91e40d1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:34.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e8e00640 1 -- 172.21.15.47:0/2161338290 shutdown_connections 2024-02-19T15:25:34.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e8e00640 1 -- 172.21.15.47:0/2161338290 wait complete. 2024-02-19T15:25:34.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e8e00640 1 Processor -- start 2024-02-19T15:25:34.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e8e00640 1 -- start start 2024-02-19T15:25:34.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e8e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f91e40b2e20 0x7f91e40cba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:34.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e8e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e40ce3c0 0x7f91e4152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:34.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e8e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91e415e570 0x7f91e4159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:34.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e8e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e41531e0 con 0x7f91e415e570 2024-02-19T15:25:34.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e8e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e40ca9d0 con 0x7f91e40ce3c0 2024-02-19T15:25:34.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e8e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e40cacb0 con 0x7f91e40b2e20 2024-02-19T15:25:34.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e2800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91e415e570 0x7f91e4159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:34.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e1e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f91e40b2e20 0x7f91e40cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:34.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e2800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91e415e570 0x7f91e4159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48188/0 (socket says 172.21.15.47:48188) 2024-02-19T15:25:34.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e2800640 1 -- 172.21.15.47:0/3599727244 learned_addr learned my addr 172.21.15.47:0/3599727244 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:34.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91dbe00640 1 --2- 172.21.15.47:0/3599727244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e40ce3c0 0x7f91e4152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:34.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e2800640 1 -- 172.21.15.47:0/3599727244 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f91e40b2e20 msgr2=0x7f91e40cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:34.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e2800640 1 --2- 172.21.15.47:0/3599727244 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f91e40b2e20 0x7f91e40cba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:34.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e2800640 1 -- 172.21.15.47:0/3599727244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e40ce3c0 msgr2=0x7f91e4152ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:34.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e2800640 1 --2- 172.21.15.47:0/3599727244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e40ce3c0 0x7f91e4152ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:34.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.013+0000 7f91e2800640 1 -- 172.21.15.47:0/3599727244 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d4009420 con 0x7f91e415e570 2024-02-19T15:25:34.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.017+0000 7f91e1e00640 1 --2- 172.21.15.47:0/3599727244 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f91e40b2e20 0x7f91e40cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:25:34.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.017+0000 7f91e2800640 1 --2- 172.21.15.47:0/3599727244 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91e415e570 0x7f91e4159b50 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f91dc00cbc0 tx=0x7f91dc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:34.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.017+0000 7f91da000640 1 -- 172.21.15.47:0/3599727244 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc00ad40 con 0x7f91e415e570 2024-02-19T15:25:34.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.017+0000 7f91da000640 1 -- 172.21.15.47:0/3599727244 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91dc012400 con 0x7f91e415e570 2024-02-19T15:25:34.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.017+0000 7f91da000640 1 -- 172.21.15.47:0/3599727244 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc00aea0 con 0x7f91e415e570 2024-02-19T15:25:34.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.017+0000 7f91e8e00640 1 -- 172.21.15.47:0/3599727244 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e40caf90 con 0x7f91e415e570 2024-02-19T15:25:34.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.017+0000 7f91e8e00640 1 -- 172.21.15.47:0/3599727244 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e40cb550 con 0x7f91e415e570 2024-02-19T15:25:34.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.017+0000 7f91e8e00640 1 -- 172.21.15.47:0/3599727244 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e40cb770 con 0x7f91e415e570 2024-02-19T15:25:34.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.017+0000 7f91da000640 1 -- 172.21.15.47:0/3599727244 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f91dc007070 con 0x7f91e415e570 2024-02-19T15:25:34.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.017+0000 7f91da000640 1 --2- 172.21.15.47:0/3599727244 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f91c0066e40 0x7f91c0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:34.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.017+0000 7f91da000640 1 -- 172.21.15.47:0/3599727244 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91dc0847f0 con 0x7f91e415e570 2024-02-19T15:25:34.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.017+0000 7f91e1e00640 1 --2- 172.21.15.47:0/3599727244 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f91c0066e40 0x7f91c0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:34.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.021+0000 7f91e1e00640 1 --2- 172.21.15.47:0/3599727244 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f91c0066e40 0x7f91c0069300 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f91d40af000 tx=0x7f91d4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:34.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.021+0000 7f91da000640 1 -- 172.21.15.47:0/3599727244 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f91dc0515f0 con 0x7f91e415e570 2024-02-19T15:25:34.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.221+0000 7f91e8e00640 1 -- 172.21.15.47:0/3599727244 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91e40d2ad0 con 0x7f91e415e570 2024-02-19T15:25:34.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.221+0000 7f91da000640 1 -- 172.21.15.47:0/3599727244 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4946 (secure 0 0 0) 0x7f91e40d2ad0 con 0x7f91e415e570 2024-02-19T15:25:34.224 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:34.224 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5684,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3206627105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3206627105},{"type":"v1","addr":"172.21.15.116:6836","nonce":3206627105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5687,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3532185042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3532185042},{"type":"v1","addr":"172.21.15.116:6837","nonce":3532185042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/1482478568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1482478568},{"type":"v1","addr":"172.21.15.47:6834","nonce":1482478568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:25:08.947740+0000","modified":"2024-02-19T15:25:11.954116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/2490197058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2490197058},{"type":"v1","addr":"172.21.15.47:6835","nonce":2490197058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-02-19T15:25:34.225 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 40 2024-02-19T15:25:34.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.221+0000 7f91e8e00640 1 -- 172.21.15.47:0/3599727244 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f91c0066e40 msgr2=0x7f91c0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:34.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.221+0000 7f91e8e00640 1 --2- 172.21.15.47:0/3599727244 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f91c0066e40 0x7f91c0069300 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f91d40af000 tx=0x7f91d4008040 comp rx=0 tx=0).stop 2024-02-19T15:25:34.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.221+0000 7f91e8e00640 1 -- 172.21.15.47:0/3599727244 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91e415e570 msgr2=0x7f91e4159b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:34.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.221+0000 7f91e8e00640 1 --2- 172.21.15.47:0/3599727244 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91e415e570 0x7f91e4159b50 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f91dc00cbc0 tx=0x7f91dc00a430 comp rx=0 tx=0).stop 2024-02-19T15:25:34.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.221+0000 7f91e8e00640 1 -- 172.21.15.47:0/3599727244 shutdown_connections 2024-02-19T15:25:34.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.221+0000 7f91e8e00640 1 --2- 172.21.15.47:0/3599727244 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f91c0066e40 0x7f91c0069300 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:34.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.221+0000 7f91e8e00640 1 --2- 172.21.15.47:0/3599727244 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91e415e570 0x7f91e4159b50 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:34.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.221+0000 7f91e8e00640 1 --2- 172.21.15.47:0/3599727244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e40ce3c0 0x7f91e4152ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:34.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.221+0000 7f91e8e00640 1 --2- 172.21.15.47:0/3599727244 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f91e40b2e20 0x7f91e40cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:34.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.221+0000 7f91e8e00640 1 -- 172.21.15.47:0/3599727244 >> 172.21.15.47:0/3599727244 conn(0x7f91e40d1b40 msgr2=0x7f91e40d1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:34.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.221+0000 7f91e8e00640 1 -- 172.21.15.47:0/3599727244 shutdown_connections 2024-02-19T15:25:34.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:34.221+0000 7f91e8e00640 1 -- 172.21.15.47:0/3599727244 wait complete. 2024-02-19T15:25:34.254 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-02-19T15:25:34.414 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-02-19T15:25:34.414 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-02-19T15:25:34.415 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 10, waiting (timeout=60)... 2024-02-19T15:25:35.529 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:25:35.532 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:25:39.415 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:25:39.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.552+0000 7fb4a3400640 1 Processor -- start 2024-02-19T15:25:39.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.552+0000 7fb4a3400640 1 -- start start 2024-02-19T15:25:39.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.552+0000 7fb4a3400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb49c145590 0x7fb49c0727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:39.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.552+0000 7fb4a3400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb49c0cea80 0x7fb49c06c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:39.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.552+0000 7fb4a3400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49c15d280 con 0x7fb49c15c390 2024-02-19T15:25:39.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.552+0000 7fb4a3400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb49c070ff0 con 0x7fb49c145590 2024-02-19T15:25:39.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.552+0000 7fb4a3400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c0709e0 con 0x7fb49c0cea80 2024-02-19T15:25:39.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.552+0000 7fb4a0a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb49c145590 0x7fb49c0727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:39.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.552+0000 7fb4a0a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb49c145590 0x7fb49c0727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48200/0 (socket says 172.21.15.47:48200) 2024-02-19T15:25:39.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.552+0000 7fb4a0a00640 1 -- 172.21.15.47:0/2942692595 learned_addr learned my addr 172.21.15.47:0/2942692595 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:39.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.552+0000 7fb49be00640 1 --2- 172.21.15.47:0/2942692595 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb49c0cea80 0x7fb49c06c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:39.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.556+0000 7fb4a0a00640 1 -- 172.21.15.47:0/2942692595 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb49c0cea80 msgr2=0x7fb49c06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:39.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.556+0000 7fb4a0a00640 1 --2- 172.21.15.47:0/2942692595 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb49c0cea80 0x7fb49c06c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:39.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.556+0000 7fb4a0a00640 1 -- 172.21.15.47:0/2942692595 >> v1:172.21.15.116:6789/0 conn(0x7fb49c15c390 legacy=0x7fb49c15c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:39.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.556+0000 7fb4a0a00640 1 -- 172.21.15.47:0/2942692595 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c06cee0 con 0x7fb49c145590 2024-02-19T15:25:39.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.560+0000 7fb4a0a00640 1 --2- 172.21.15.47:0/2942692595 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb49c145590 0x7fb49c0727b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fb4900098a0 tx=0x7fb49001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a851bbce7d4ad6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:39.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.560+0000 7fb49b400640 1 -- 172.21.15.47:0/2942692595 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4900ab650 con 0x7fb49c145590 2024-02-19T15:25:39.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.560+0000 7fb49b400640 1 -- 172.21.15.47:0/2942692595 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4900ab7b0 con 0x7fb49c145590 2024-02-19T15:25:39.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.560+0000 7fb49b400640 1 -- 172.21.15.47:0/2942692595 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4900abb40 con 0x7fb49c145590 2024-02-19T15:25:39.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.560+0000 7fb4a3400640 1 -- 172.21.15.47:0/2942692595 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb49c145590 msgr2=0x7fb49c0727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:39.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.560+0000 7fb4a3400640 1 --2- 172.21.15.47:0/2942692595 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb49c145590 0x7fb49c0727b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fb4900098a0 tx=0x7fb49001c540 comp rx=0 tx=0).stop 2024-02-19T15:25:39.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.560+0000 7fb4a3400640 1 -- 172.21.15.47:0/2942692595 shutdown_connections 2024-02-19T15:25:39.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.560+0000 7fb4a3400640 1 --2- 172.21.15.47:0/2942692595 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb49c0cea80 0x7fb49c06c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:39.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.560+0000 7fb4a3400640 1 --2- 172.21.15.47:0/2942692595 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb49c145590 0x7fb49c0727b0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:39.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.560+0000 7fb4a3400640 1 -- 172.21.15.47:0/2942692595 >> 172.21.15.47:0/2942692595 conn(0x7fb49c0b16a0 msgr2=0x7fb49c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:39.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.560+0000 7fb4a3400640 1 -- 172.21.15.47:0/2942692595 shutdown_connections 2024-02-19T15:25:39.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.560+0000 7fb4a3400640 1 -- 172.21.15.47:0/2942692595 wait complete. 2024-02-19T15:25:39.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.560+0000 7fb4a3400640 1 Processor -- start 2024-02-19T15:25:39.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb4a3400640 1 -- start start 2024-02-19T15:25:39.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb4a3400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb49c0cea80 0x7fb49c15b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:39.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb4a3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb49c145590 0x7fb49c141560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:39.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb4a3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb49c15c390 0x7fb49c0af510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:39.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb4a3400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c15a870 con 0x7fb49c145590 2024-02-19T15:25:39.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb4a3400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c15bb00 con 0x7fb49c15c390 2024-02-19T15:25:39.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb4a3400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c15aae0 con 0x7fb49c0cea80 2024-02-19T15:25:39.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb49be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb49c145590 0x7fb49c141560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:39.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb49be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb49c145590 0x7fb49c141560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48210/0 (socket says 172.21.15.47:48210) 2024-02-19T15:25:39.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb49be00640 1 -- 172.21.15.47:0/1835460247 learned_addr learned my addr 172.21.15.47:0/1835460247 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:39.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb4a0a00640 1 --2- 172.21.15.47:0/1835460247 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb49c0cea80 0x7fb49c15b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:39.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb4a1400640 1 --2- 172.21.15.47:0/1835460247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb49c15c390 0x7fb49c0af510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:39.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb49be00640 1 -- 172.21.15.47:0/1835460247 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb49c0cea80 msgr2=0x7fb49c15b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:39.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb49be00640 1 --2- 172.21.15.47:0/1835460247 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb49c0cea80 0x7fb49c15b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:39.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb49be00640 1 -- 172.21.15.47:0/1835460247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb49c15c390 msgr2=0x7fb49c0af510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:39.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb49be00640 1 --2- 172.21.15.47:0/1835460247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb49c15c390 0x7fb49c0af510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:39.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb49be00640 1 -- 172.21.15.47:0/1835460247 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4900094d0 con 0x7fb49c145590 2024-02-19T15:25:39.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb49be00640 1 --2- 172.21.15.47:0/1835460247 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb49c145590 0x7fb49c141560 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fb48c00e930 tx=0x7fb48c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:39.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb498c00640 1 -- 172.21.15.47:0/1835460247 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48c009690 con 0x7fb49c145590 2024-02-19T15:25:39.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb498c00640 1 -- 172.21.15.47:0/1835460247 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48c00ca00 con 0x7fb49c145590 2024-02-19T15:25:39.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb498c00640 1 -- 172.21.15.47:0/1835460247 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48c00cd70 con 0x7fb49c145590 2024-02-19T15:25:39.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb4a3400640 1 -- 172.21.15.47:0/1835460247 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb49c15adc0 con 0x7fb49c145590 2024-02-19T15:25:39.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb4a3400640 1 -- 172.21.15.47:0/1835460247 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb49c137e80 con 0x7fb49c145590 2024-02-19T15:25:39.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.564+0000 7fb4a3400640 1 -- 172.21.15.47:0/1835460247 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb49c075360 con 0x7fb49c145590 2024-02-19T15:25:39.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.568+0000 7fb498c00640 1 -- 172.21.15.47:0/1835460247 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb48c013470 con 0x7fb49c145590 2024-02-19T15:25:39.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.568+0000 7fb498c00640 1 --2- 172.21.15.47:0/1835460247 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb480066e40 0x7fb480069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:39.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.568+0000 7fb498c00640 1 -- 172.21.15.47:0/1835460247 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb48c084c30 con 0x7fb49c145590 2024-02-19T15:25:39.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.568+0000 7fb4a0a00640 1 --2- 172.21.15.47:0/1835460247 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb480066e40 0x7fb480069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:39.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.568+0000 7fb4a0a00640 1 --2- 172.21.15.47:0/1835460247 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb480066e40 0x7fb480069300 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb4900af000 tx=0x7fb4900a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:39.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.572+0000 7fb498c00640 1 -- 172.21.15.47:0/1835460247 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb48c051b70 con 0x7fb49c145590 2024-02-19T15:25:39.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.780+0000 7fb4a3400640 1 -- 172.21.15.47:0/1835460247 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb49c13eb60 con 0x7fb49c145590 2024-02-19T15:25:39.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.780+0000 7fb498c00640 1 -- 172.21.15.47:0/1835460247 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4946 (secure 0 0 0) 0x7fb48c017090 con 0x7fb49c145590 2024-02-19T15:25:39.785 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:39.785 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5684,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3206627105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3206627105},{"type":"v1","addr":"172.21.15.116:6836","nonce":3206627105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5687,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3532185042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3532185042},{"type":"v1","addr":"172.21.15.116:6837","nonce":3532185042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/1482478568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1482478568},{"type":"v1","addr":"172.21.15.47:6834","nonce":1482478568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:25:08.947740+0000","modified":"2024-02-19T15:25:11.954116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/2490197058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2490197058},{"type":"v1","addr":"172.21.15.47:6835","nonce":2490197058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-02-19T15:25:39.785 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 40 2024-02-19T15:25:39.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.784+0000 7fb4a3400640 1 -- 172.21.15.47:0/1835460247 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb480066e40 msgr2=0x7fb480069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:39.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.784+0000 7fb4a3400640 1 --2- 172.21.15.47:0/1835460247 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb480066e40 0x7fb480069300 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb4900af000 tx=0x7fb4900a7000 comp rx=0 tx=0).stop 2024-02-19T15:25:39.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.784+0000 7fb4a3400640 1 -- 172.21.15.47:0/1835460247 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb49c145590 msgr2=0x7fb49c141560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:39.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.784+0000 7fb4a3400640 1 --2- 172.21.15.47:0/1835460247 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb49c145590 0x7fb49c141560 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fb48c00e930 tx=0x7fb48c00edf0 comp rx=0 tx=0).stop 2024-02-19T15:25:39.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.784+0000 7fb4a3400640 1 -- 172.21.15.47:0/1835460247 shutdown_connections 2024-02-19T15:25:39.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.784+0000 7fb4a3400640 1 --2- 172.21.15.47:0/1835460247 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb480066e40 0x7fb480069300 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:39.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.784+0000 7fb4a3400640 1 --2- 172.21.15.47:0/1835460247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb49c15c390 0x7fb49c0af510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:39.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.784+0000 7fb4a3400640 1 --2- 172.21.15.47:0/1835460247 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb49c145590 0x7fb49c141560 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:39.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.784+0000 7fb4a3400640 1 --2- 172.21.15.47:0/1835460247 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb49c0cea80 0x7fb49c15b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:39.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.784+0000 7fb4a3400640 1 -- 172.21.15.47:0/1835460247 >> 172.21.15.47:0/1835460247 conn(0x7fb49c0b16a0 msgr2=0x7fb49c0b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:39.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.784+0000 7fb4a3400640 1 -- 172.21.15.47:0/1835460247 shutdown_connections 2024-02-19T15:25:39.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:39.784+0000 7fb4a3400640 1 -- 172.21.15.47:0/1835460247 wait complete. 2024-02-19T15:25:39.806 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-02-19T15:25:39.966 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"strays_enqueued":10}} 2024-02-19T15:25:39.966 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 10 } } 2024-02-19T15:25:39.966 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:25:40.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.112+0000 7f1414a00640 1 Processor -- start 2024-02-19T15:25:40.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.112+0000 7f1414a00640 1 -- start start 2024-02-19T15:25:40.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.112+0000 7f1414a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f141015f590 0x7f14100d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:40.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.112+0000 7f1414a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14100b0280 con 0x7f14100b1be0 2024-02-19T15:25:40.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.112+0000 7f1414a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14100b04d0 con 0x7f14100b0a80 2024-02-19T15:25:40.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.112+0000 7f1414a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14100cc530 con 0x7f141015f590 2024-02-19T15:25:40.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.112+0000 7f140e800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f14100b1be0 0x7f14100b20e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39528/0 (socket says 172.21.15.47:39528) 2024-02-19T15:25:40.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.112+0000 7f140e800640 1 -- 172.21.15.47:0/3547437724 learned_addr learned my addr 172.21.15.47:0/3547437724 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:40.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.112+0000 7f140de00640 1 --2- 172.21.15.47:0/3547437724 >> v2:172.21.15.116:3300/0 conn(0x7f141015f590 0x7f14100d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:40.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.116+0000 7f140d400640 1 -- 172.21.15.47:0/3547437724 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397779204 0 0) 0x7f14100b0280 con 0x7f14100b1be0 2024-02-19T15:25:40.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.116+0000 7f140d400640 1 -- 172.21.15.47:0/3547437724 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13f4003530 con 0x7f14100b1be0 2024-02-19T15:25:40.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.116+0000 7f140de00640 1 -- 172.21.15.47:0/3547437724 >> v1:172.21.15.116:6789/0 conn(0x7f14100b0a80 legacy=0x7f14100b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:40.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.116+0000 7f140de00640 1 -- 172.21.15.47:0/3547437724 >> v1:172.21.15.47:6789/0 conn(0x7f14100b1be0 legacy=0x7f14100b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:40.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.116+0000 7f140de00640 1 -- 172.21.15.47:0/3547437724 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14100b07b0 con 0x7f141015f590 2024-02-19T15:25:40.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.116+0000 7f140de00640 1 --2- 172.21.15.47:0/3547437724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f141015f590 0x7f14100d2690 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f14000097c0 tx=0x7f140001c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b053ddfae2d85d9d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:40.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.116+0000 7f140d400640 1 -- 172.21.15.47:0/3547437724 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14000ab670 con 0x7f141015f590 2024-02-19T15:25:40.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.116+0000 7f140d400640 1 -- 172.21.15.47:0/3547437724 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14000ab7d0 con 0x7f141015f590 2024-02-19T15:25:40.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.116+0000 7f140d400640 1 -- 172.21.15.47:0/3547437724 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14000abb60 con 0x7f141015f590 2024-02-19T15:25:40.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f1414a00640 1 -- 172.21.15.47:0/3547437724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f141015f590 msgr2=0x7f14100d2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:40.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f1414a00640 1 --2- 172.21.15.47:0/3547437724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f141015f590 0x7f14100d2690 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f14000097c0 tx=0x7f140001c470 comp rx=0 tx=0).stop 2024-02-19T15:25:40.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f1414a00640 1 -- 172.21.15.47:0/3547437724 shutdown_connections 2024-02-19T15:25:40.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f1414a00640 1 --2- 172.21.15.47:0/3547437724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f141015f590 0x7f14100d2690 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:40.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f1414a00640 1 -- 172.21.15.47:0/3547437724 >> 172.21.15.47:0/3547437724 conn(0x7f14100d1b70 msgr2=0x7f14100d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:40.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f1414a00640 1 -- 172.21.15.47:0/3547437724 shutdown_connections 2024-02-19T15:25:40.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f1414a00640 1 -- 172.21.15.47:0/3547437724 wait complete. 2024-02-19T15:25:40.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f1414a00640 1 Processor -- start 2024-02-19T15:25:40.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f1414a00640 1 -- start start 2024-02-19T15:25:40.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f1414a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14100b0a80 0x7f141014b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:40.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f1414a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14100b1be0 0x7f14100ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:40.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f1414a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f141015f590 0x7f14101532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:40.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f1414a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141014b5d0 con 0x7f14100b0a80 2024-02-19T15:25:40.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f1414a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14100cad70 con 0x7f14100b1be0 2024-02-19T15:25:40.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f1414a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14100cb050 con 0x7f141015f590 2024-02-19T15:25:40.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f140de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14100b0a80 0x7f141014b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:40.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f140de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14100b0a80 0x7f141014b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48220/0 (socket says 172.21.15.47:48220) 2024-02-19T15:25:40.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f140de00640 1 -- 172.21.15.47:0/831965387 learned_addr learned my addr 172.21.15.47:0/831965387 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:40.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f140e800640 1 --2- 172.21.15.47:0/831965387 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f141015f590 0x7f14101532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:40.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f1407e00640 1 --2- 172.21.15.47:0/831965387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14100b1be0 0x7f14100ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:40.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f140de00640 1 -- 172.21.15.47:0/831965387 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f141015f590 msgr2=0x7f14101532a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:40.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f140de00640 1 --2- 172.21.15.47:0/831965387 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f141015f590 0x7f14101532a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:40.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f140de00640 1 -- 172.21.15.47:0/831965387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14100b1be0 msgr2=0x7f14100ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:40.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f140de00640 1 --2- 172.21.15.47:0/831965387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14100b1be0 0x7f14100ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:40.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.120+0000 7f140de00640 1 -- 172.21.15.47:0/831965387 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1400009420 con 0x7f14100b0a80 2024-02-19T15:25:40.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.124+0000 7f140de00640 1 --2- 172.21.15.47:0/831965387 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14100b0a80 0x7f141014b090 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f14000ad040 tx=0x7f140001acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:40.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.124+0000 7f1406000640 1 -- 172.21.15.47:0/831965387 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14000bf610 con 0x7f14100b0a80 2024-02-19T15:25:40.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.124+0000 7f1406000640 1 -- 172.21.15.47:0/831965387 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14000bf770 con 0x7f14100b0a80 2024-02-19T15:25:40.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.124+0000 7f1406000640 1 -- 172.21.15.47:0/831965387 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14000bfb20 con 0x7f14100b0a80 2024-02-19T15:25:40.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.124+0000 7f1414a00640 1 -- 172.21.15.47:0/831965387 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14100cb330 con 0x7f14100b0a80 2024-02-19T15:25:40.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.124+0000 7f1414a00640 1 -- 172.21.15.47:0/831965387 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14100cb8f0 con 0x7f14100b0a80 2024-02-19T15:25:40.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.124+0000 7f1414a00640 1 -- 172.21.15.47:0/831965387 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14100d19e0 con 0x7f14100b0a80 2024-02-19T15:25:40.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.124+0000 7f1406000640 1 -- 172.21.15.47:0/831965387 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f14000b5400 con 0x7f14100b0a80 2024-02-19T15:25:40.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.124+0000 7f1406000640 1 --2- 172.21.15.47:0/831965387 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f13ec066e40 0x7f13ec069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:40.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.124+0000 7f1406000640 1 -- 172.21.15.47:0/831965387 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f140011f600 con 0x7f14100b0a80 2024-02-19T15:25:40.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.124+0000 7f1407e00640 1 --2- 172.21.15.47:0/831965387 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f13ec066e40 0x7f13ec069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:40.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.128+0000 7f1407e00640 1 --2- 172.21.15.47:0/831965387 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f13ec066e40 0x7f13ec069300 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f14100cbcd0 tx=0x7f13f8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:40.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.132+0000 7f1406000640 1 -- 172.21.15.47:0/831965387 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f14000ec520 con 0x7f14100b0a80 2024-02-19T15:25:40.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.328+0000 7f1414a00640 1 -- 172.21.15.47:0/831965387 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14100cbae0 con 0x7f14100b0a80 2024-02-19T15:25:40.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.328+0000 7f1406000640 1 -- 172.21.15.47:0/831965387 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4946 (secure 0 0 0) 0x7f14000efbe0 con 0x7f14100b0a80 2024-02-19T15:25:40.333 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:40.333 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5684,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3206627105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3206627105},{"type":"v1","addr":"172.21.15.116:6836","nonce":3206627105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5687,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3532185042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3532185042},{"type":"v1","addr":"172.21.15.116:6837","nonce":3532185042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/1482478568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1482478568},{"type":"v1","addr":"172.21.15.47:6834","nonce":1482478568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:25:08.947740+0000","modified":"2024-02-19T15:25:11.954116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/2490197058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2490197058},{"type":"v1","addr":"172.21.15.47:6835","nonce":2490197058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-02-19T15:25:40.333 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 40 2024-02-19T15:25:40.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.332+0000 7f1414a00640 1 -- 172.21.15.47:0/831965387 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f13ec066e40 msgr2=0x7f13ec069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:40.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.332+0000 7f1414a00640 1 --2- 172.21.15.47:0/831965387 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f13ec066e40 0x7f13ec069300 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f14100cbcd0 tx=0x7f13f8009040 comp rx=0 tx=0).stop 2024-02-19T15:25:40.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.332+0000 7f1414a00640 1 -- 172.21.15.47:0/831965387 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14100b0a80 msgr2=0x7f141014b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:40.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.332+0000 7f1414a00640 1 --2- 172.21.15.47:0/831965387 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14100b0a80 0x7f141014b090 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f14000ad040 tx=0x7f140001acb0 comp rx=0 tx=0).stop 2024-02-19T15:25:40.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.332+0000 7f1414a00640 1 -- 172.21.15.47:0/831965387 shutdown_connections 2024-02-19T15:25:40.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.332+0000 7f1414a00640 1 --2- 172.21.15.47:0/831965387 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f13ec066e40 0x7f13ec069300 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:40.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.332+0000 7f1414a00640 1 --2- 172.21.15.47:0/831965387 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f141015f590 0x7f14101532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:40.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.332+0000 7f1414a00640 1 --2- 172.21.15.47:0/831965387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14100b1be0 0x7f14100ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:40.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.332+0000 7f1414a00640 1 --2- 172.21.15.47:0/831965387 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14100b0a80 0x7f141014b090 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:40.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.332+0000 7f1414a00640 1 -- 172.21.15.47:0/831965387 >> 172.21.15.47:0/831965387 conn(0x7f14100d1b70 msgr2=0x7f1410074000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:40.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.332+0000 7f1414a00640 1 -- 172.21.15.47:0/831965387 shutdown_connections 2024-02-19T15:25:40.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40.332+0000 7f1414a00640 1 -- 172.21.15.47:0/831965387 wait complete. 2024-02-19T15:25:40.355 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue pq_executed 2024-02-19T15:25:40.516 INFO:teuthology.orchestra.run.smithi047.stdout:{"purge_queue":{"pq_executed":10}} 2024-02-19T15:25:40.516 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 10 } } 2024-02-19T15:25:40.516 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:25:40.516 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:25:40.525 DEBUG:teuthology.orchestra.run.smithi047:> stdin-killer -- python3 -c ' 2024-02-19T15:25:40.525 DEBUG:teuthology.orchestra.run.smithi047:> import os 2024-02-19T15:25:40.525 DEBUG:teuthology.orchestra.run.smithi047:> import uuid 2024-02-19T15:25:40.525 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:25:40.526 DEBUG:teuthology.orchestra.run.smithi047:> n = 10 2024-02-19T15:25:40.526 DEBUG:teuthology.orchestra.run.smithi047:> create_hard_links = False 2024-02-19T15:25:40.526 DEBUG:teuthology.orchestra.run.smithi047:> if 0 > 0: 2024-02-19T15:25:40.526 DEBUG:teuthology.orchestra.run.smithi047:> create_hard_links = True 2024-02-19T15:25:40.526 DEBUG:teuthology.orchestra.run.smithi047:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-02-19T15:25:40.526 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:25:40.526 DEBUG:teuthology.orchestra.run.smithi047:> dpath = os.path.dirname(path) 2024-02-19T15:25:40.526 DEBUG:teuthology.orchestra.run.smithi047:> fnameprefix = os.path.basename(path) 2024-02-19T15:25:40.526 DEBUG:teuthology.orchestra.run.smithi047:> os.makedirs(dpath, exist_ok=True) 2024-02-19T15:25:40.526 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:25:40.526 DEBUG:teuthology.orchestra.run.smithi047:> try: 2024-02-19T15:25:40.526 DEBUG:teuthology.orchestra.run.smithi047:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-02-19T15:25:40.526 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:25:40.526 DEBUG:teuthology.orchestra.run.smithi047:> for i in range(n): 2024-02-19T15:25:40.526 DEBUG:teuthology.orchestra.run.smithi047:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-02-19T15:25:40.526 DEBUG:teuthology.orchestra.run.smithi047:> with open(fpath, '"'"'w'"'"') as f: 2024-02-19T15:25:40.526 DEBUG:teuthology.orchestra.run.smithi047:> f.write(f"{i}") 2024-02-19T15:25:40.526 DEBUG:teuthology.orchestra.run.smithi047:> if False: 2024-02-19T15:25:40.527 DEBUG:teuthology.orchestra.run.smithi047:> f.flush() 2024-02-19T15:25:40.527 DEBUG:teuthology.orchestra.run.smithi047:> os.fsync(f.fileno()) 2024-02-19T15:25:40.527 DEBUG:teuthology.orchestra.run.smithi047:> if True: 2024-02-19T15:25:40.527 DEBUG:teuthology.orchestra.run.smithi047:> os.unlink(fpath) 2024-02-19T15:25:40.527 DEBUG:teuthology.orchestra.run.smithi047:> if True: 2024-02-19T15:25:40.527 DEBUG:teuthology.orchestra.run.smithi047:> os.fsync(dirfd) 2024-02-19T15:25:40.527 DEBUG:teuthology.orchestra.run.smithi047:> if create_hard_links: 2024-02-19T15:25:40.527 DEBUG:teuthology.orchestra.run.smithi047:> for j in range(0): 2024-02-19T15:25:40.527 DEBUG:teuthology.orchestra.run.smithi047:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-02-19T15:25:40.527 DEBUG:teuthology.orchestra.run.smithi047:> if False: 2024-02-19T15:25:40.527 DEBUG:teuthology.orchestra.run.smithi047:> os.fsync(dirfd) 2024-02-19T15:25:40.527 DEBUG:teuthology.orchestra.run.smithi047:> finally: 2024-02-19T15:25:40.527 DEBUG:teuthology.orchestra.run.smithi047:> os.close(dirfd) 2024-02-19T15:25:40.527 DEBUG:teuthology.orchestra.run.smithi047:> ' 2024-02-19T15:25:40.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-02-19T15:25:40.708 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T15:25:40.709 DEBUG:teuthology.orchestra.run.smithi047:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:25:40.820 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:25:40.905 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T15:25:40.905 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:25:40.905 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:25:40.913 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:25:40.913 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:25:40.962 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:25:41.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.140+0000 7faac9a00640 1 Processor -- start 2024-02-19T15:25:41.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.140+0000 7faac9a00640 1 -- start start 2024-02-19T15:25:41.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.144+0000 7faac9a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faac4145590 0x7faac40727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:41.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.144+0000 7faac9a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faac415d280 con 0x7faac415c390 2024-02-19T15:25:41.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.144+0000 7faac9a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faac4070ff0 con 0x7faac40cea80 2024-02-19T15:25:41.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.144+0000 7faac9a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faac40709e0 con 0x7faac4145590 2024-02-19T15:25:41.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.144+0000 7faac3400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7faac415c390 0x7faac415c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39540/0 (socket says 172.21.15.47:39540) 2024-02-19T15:25:41.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.144+0000 7faac3400640 1 -- 172.21.15.47:0/57483102 learned_addr learned my addr 172.21.15.47:0/57483102 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:41.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.144+0000 7faac2a00640 1 --2- 172.21.15.47:0/57483102 >> v2:172.21.15.116:3300/0 conn(0x7faac4145590 0x7faac40727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:41.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.144+0000 7faac2000640 1 -- 172.21.15.47:0/57483102 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228335056 0 0) 0x7faac415d280 con 0x7faac415c390 2024-02-19T15:25:41.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.144+0000 7faac2000640 1 -- 172.21.15.47:0/57483102 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaa8003530 con 0x7faac415c390 2024-02-19T15:25:41.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.144+0000 7faac2000640 1 -- 172.21.15.47:0/57483102 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743430750 0 0) 0x7faac4070ff0 con 0x7faac40cea80 2024-02-19T15:25:41.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.144+0000 7faac2000640 1 -- 172.21.15.47:0/57483102 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faac415d280 con 0x7faac40cea80 2024-02-19T15:25:41.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.144+0000 7faac2000640 1 -- 172.21.15.47:0/57483102 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3477959932 0 0) 0x7faaa8003530 con 0x7faac415c390 2024-02-19T15:25:41.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.144+0000 7faac2a00640 1 -- 172.21.15.47:0/57483102 >> v1:172.21.15.116:6789/0 conn(0x7faac40cea80 legacy=0x7faac406c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:41.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.144+0000 7faac2a00640 1 -- 172.21.15.47:0/57483102 >> v1:172.21.15.47:6789/0 conn(0x7faac415c390 legacy=0x7faac415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:41.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.144+0000 7faac2a00640 1 -- 172.21.15.47:0/57483102 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac40cf020 con 0x7faac4145590 2024-02-19T15:25:41.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.148+0000 7faac2a00640 1 --2- 172.21.15.47:0/57483102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faac4145590 0x7faac40727b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7faab40097c0 tx=0x7faab401c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31402e5c14473069 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:41.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.148+0000 7faac2000640 1 -- 172.21.15.47:0/57483102 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faabc001a00 con 0x7faac4145590 2024-02-19T15:25:41.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.148+0000 7faac2000640 1 -- 172.21.15.47:0/57483102 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faab40a7050 con 0x7faac4145590 2024-02-19T15:25:41.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.148+0000 7faac2000640 1 -- 172.21.15.47:0/57483102 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab40ab600 con 0x7faac4145590 2024-02-19T15:25:41.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.148+0000 7faac9a00640 1 -- 172.21.15.47:0/57483102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faac4145590 msgr2=0x7faac40727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:41.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.148+0000 7faac9a00640 1 --2- 172.21.15.47:0/57483102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faac4145590 0x7faac40727b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7faab40097c0 tx=0x7faab401c3c0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.148+0000 7faac9a00640 1 -- 172.21.15.47:0/57483102 shutdown_connections 2024-02-19T15:25:41.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.148+0000 7faac9a00640 1 --2- 172.21.15.47:0/57483102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faac4145590 0x7faac40727b0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.148+0000 7faac9a00640 1 -- 172.21.15.47:0/57483102 >> 172.21.15.47:0/57483102 conn(0x7faac40b16a0 msgr2=0x7faac40b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:41.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.148+0000 7faac9a00640 1 -- 172.21.15.47:0/57483102 shutdown_connections 2024-02-19T15:25:41.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.148+0000 7faac9a00640 1 -- 172.21.15.47:0/57483102 wait complete. 2024-02-19T15:25:41.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.148+0000 7faac9a00640 1 Processor -- start 2024-02-19T15:25:41.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.148+0000 7faac9a00640 1 -- start start 2024-02-19T15:25:41.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.148+0000 7faac9a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faac40cea80 0x7faac4158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:41.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.148+0000 7faac9a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faac4145590 0x7faac4133f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:41.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.148+0000 7faac9a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faac415c390 0x7faac413fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:41.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.148+0000 7faac9a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac4159180 con 0x7faac40cea80 2024-02-19T15:25:41.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.148+0000 7faac9a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac4134810 con 0x7faac415c390 2024-02-19T15:25:41.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.148+0000 7faac9a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faac4134af0 con 0x7faac4145590 2024-02-19T15:25:41.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.152+0000 7faac2a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faac40cea80 0x7faac4158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:41.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.152+0000 7faac2a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faac40cea80 0x7faac4158c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48226/0 (socket says 172.21.15.47:48226) 2024-02-19T15:25:41.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.152+0000 7faac2a00640 1 -- 172.21.15.47:0/2639269113 learned_addr learned my addr 172.21.15.47:0/2639269113 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:41.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.152+0000 7faac3400640 1 --2- 172.21.15.47:0/2639269113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faac415c390 0x7faac413fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:41.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.152+0000 7faabbe00640 1 --2- 172.21.15.47:0/2639269113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faac4145590 0x7faac4133f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:41.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.152+0000 7faac2a00640 1 -- 172.21.15.47:0/2639269113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faac4145590 msgr2=0x7faac4133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:41.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.152+0000 7faac2a00640 1 --2- 172.21.15.47:0/2639269113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faac4145590 0x7faac4133f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.152+0000 7faac2a00640 1 -- 172.21.15.47:0/2639269113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faac415c390 msgr2=0x7faac413fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:41.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.152+0000 7faac2a00640 1 --2- 172.21.15.47:0/2639269113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faac415c390 0x7faac413fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.152+0000 7faac2a00640 1 -- 172.21.15.47:0/2639269113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab4009420 con 0x7faac40cea80 2024-02-19T15:25:41.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.152+0000 7faabbe00640 1 --2- 172.21.15.47:0/2639269113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faac4145590 0x7faac4133f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:25:41.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.152+0000 7faac2a00640 1 --2- 172.21.15.47:0/2639269113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faac40cea80 0x7faac4158c40 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7faab4018fd0 tx=0x7faab4019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:41.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.152+0000 7faabaa00640 1 -- 172.21.15.47:0/2639269113 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab40b9070 con 0x7faac40cea80 2024-02-19T15:25:41.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.152+0000 7faabaa00640 1 -- 172.21.15.47:0/2639269113 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faab40038b0 con 0x7faac40cea80 2024-02-19T15:25:41.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.152+0000 7faac9a00640 1 -- 172.21.15.47:0/2639269113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faac4134dd0 con 0x7faac40cea80 2024-02-19T15:25:41.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.152+0000 7faabaa00640 1 -- 172.21.15.47:0/2639269113 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab4003c20 con 0x7faac40cea80 2024-02-19T15:25:41.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.152+0000 7faac9a00640 1 -- 172.21.15.47:0/2639269113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faac4135390 con 0x7faac40cea80 2024-02-19T15:25:41.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.152+0000 7faac9a00640 1 -- 172.21.15.47:0/2639269113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faac4075360 con 0x7faac40cea80 2024-02-19T15:25:41.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.152+0000 7faabaa00640 1 -- 172.21.15.47:0/2639269113 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7faab40b2040 con 0x7faac40cea80 2024-02-19T15:25:41.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.152+0000 7faabaa00640 1 --2- 172.21.15.47:0/2639269113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faaa0066e40 0x7faaa0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:41.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.156+0000 7faabaa00640 1 -- 172.21.15.47:0/2639269113 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faab411dfe0 con 0x7faac40cea80 2024-02-19T15:25:41.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.156+0000 7faabbe00640 1 --2- 172.21.15.47:0/2639269113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faaa0066e40 0x7faaa0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:41.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.156+0000 7faabbe00640 1 --2- 172.21.15.47:0/2639269113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faaa0066e40 0x7faaa0069300 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7faaac0011f0 tx=0x7faaac009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:41.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.160+0000 7faabaa00640 1 -- 172.21.15.47:0/2639269113 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7faab40eade0 con 0x7faac40cea80 2024-02-19T15:25:41.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.324+0000 7faac9a00640 1 -- 172.21.15.47:0/2639269113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faac40af5b0 con 0x7faac40cea80 2024-02-19T15:25:41.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.324+0000 7faabaa00640 1 -- 172.21.15.47:0/2639269113 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16680 (secure 0 0 0) 0x7faab40ee4a0 con 0x7faac40cea80 2024-02-19T15:25:41.329 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:41.329 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":69,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:25:10.552537+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:25:06.759758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-02-19T15:25:07.949435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:25:41.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.328+0000 7faac9a00640 1 -- 172.21.15.47:0/2639269113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faaa0066e40 msgr2=0x7faaa0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:41.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.328+0000 7faac9a00640 1 --2- 172.21.15.47:0/2639269113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faaa0066e40 0x7faaa0069300 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7faaac0011f0 tx=0x7faaac009040 comp rx=0 tx=0).stop 2024-02-19T15:25:41.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.328+0000 7faac9a00640 1 -- 172.21.15.47:0/2639269113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faac40cea80 msgr2=0x7faac4158c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:41.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.328+0000 7faac9a00640 1 --2- 172.21.15.47:0/2639269113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faac40cea80 0x7faac4158c40 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7faab4018fd0 tx=0x7faab4019160 comp rx=0 tx=0).stop 2024-02-19T15:25:41.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.328+0000 7faac9a00640 1 -- 172.21.15.47:0/2639269113 shutdown_connections 2024-02-19T15:25:41.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.328+0000 7faac9a00640 1 --2- 172.21.15.47:0/2639269113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faaa0066e40 0x7faaa0069300 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.328+0000 7faac9a00640 1 --2- 172.21.15.47:0/2639269113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faac415c390 0x7faac413fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.328+0000 7faac9a00640 1 --2- 172.21.15.47:0/2639269113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faac4145590 0x7faac4133f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.328+0000 7faac9a00640 1 --2- 172.21.15.47:0/2639269113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faac40cea80 0x7faac4158c40 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.328+0000 7faac9a00640 1 -- 172.21.15.47:0/2639269113 >> 172.21.15.47:0/2639269113 conn(0x7faac40b16a0 msgr2=0x7faac41398e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:41.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.328+0000 7faac9a00640 1 -- 172.21.15.47:0/2639269113 shutdown_connections 2024-02-19T15:25:41.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.328+0000 7faac9a00640 1 -- 172.21.15.47:0/2639269113 wait complete. 2024-02-19T15:25:41.350 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:25:06.759758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:25:07.949435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:25:41.351 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:25:41.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.488+0000 7fdbe6800640 1 Processor -- start 2024-02-19T15:25:41.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.492+0000 7fdbe6800640 1 -- start start 2024-02-19T15:25:41.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.492+0000 7fdbe6800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdbe0158bc0 0x7fdbe015be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:41.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.492+0000 7fdbe6800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe015c380 con 0x7fdbe00cf310 2024-02-19T15:25:41.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.492+0000 7fdbe6800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe0072890 con 0x7fdbe00cf7a0 2024-02-19T15:25:41.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.492+0000 7fdbe6800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe015f8f0 con 0x7fdbe0158bc0 2024-02-19T15:25:41.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.492+0000 7fdbdf400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdbe0158bc0 0x7fdbe015be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:41.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.492+0000 7fdbd7e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fdbe00cf310 0x7fdbe0071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39556/0 (socket says 172.21.15.47:39556) 2024-02-19T15:25:41.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.492+0000 7fdbdf400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdbe0158bc0 0x7fdbe015be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48236/0 (socket says 172.21.15.47:48236) 2024-02-19T15:25:41.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.492+0000 7fdbd7e00640 1 -- 172.21.15.47:0/2991169079 learned_addr learned my addr 172.21.15.47:0/2991169079 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:41.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.492+0000 7fdbdf400640 1 -- 172.21.15.47:0/2991169079 >> v1:172.21.15.116:6789/0 conn(0x7fdbe00cf7a0 legacy=0x7fdbe0072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:41.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.492+0000 7fdbdf400640 1 -- 172.21.15.47:0/2991169079 >> v1:172.21.15.47:6789/0 conn(0x7fdbe00cf310 legacy=0x7fdbe0071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:41.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.492+0000 7fdbdf400640 1 -- 172.21.15.47:0/2991169079 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe006dad0 con 0x7fdbe0158bc0 2024-02-19T15:25:41.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.492+0000 7fdbdea00640 1 -- 172.21.15.47:0/2991169079 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311823969 0 0) 0x7fdbe015c380 con 0x7fdbe00cf310 2024-02-19T15:25:41.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.496+0000 7fdbdf400640 1 --2- 172.21.15.47:0/2991169079 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbe0158bc0 0x7fdbe015be40 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fdbd00097f0 tx=0x7fdbd001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc9e73171ab519f6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:41.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.496+0000 7fdbdea00640 1 -- 172.21.15.47:0/2991169079 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd00ab5f0 con 0x7fdbe0158bc0 2024-02-19T15:25:41.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.496+0000 7fdbdea00640 1 -- 172.21.15.47:0/2991169079 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd00ab750 con 0x7fdbe0158bc0 2024-02-19T15:25:41.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.496+0000 7fdbdea00640 1 -- 172.21.15.47:0/2991169079 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd00abae0 con 0x7fdbe0158bc0 2024-02-19T15:25:41.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.496+0000 7fdbe6800640 1 -- 172.21.15.47:0/2991169079 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbe0158bc0 msgr2=0x7fdbe015be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:41.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.496+0000 7fdbe6800640 1 --2- 172.21.15.47:0/2991169079 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbe0158bc0 0x7fdbe015be40 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fdbd00097f0 tx=0x7fdbd001c3f0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.496+0000 7fdbe6800640 1 -- 172.21.15.47:0/2991169079 shutdown_connections 2024-02-19T15:25:41.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.496+0000 7fdbe6800640 1 --2- 172.21.15.47:0/2991169079 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbe0158bc0 0x7fdbe015be40 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.496+0000 7fdbe6800640 1 -- 172.21.15.47:0/2991169079 >> 172.21.15.47:0/2991169079 conn(0x7fdbe00d0d30 msgr2=0x7fdbe00d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:41.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.496+0000 7fdbe6800640 1 -- 172.21.15.47:0/2991169079 shutdown_connections 2024-02-19T15:25:41.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.496+0000 7fdbe6800640 1 -- 172.21.15.47:0/2991169079 wait complete. 2024-02-19T15:25:41.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbe6800640 1 Processor -- start 2024-02-19T15:25:41.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbe6800640 1 -- start start 2024-02-19T15:25:41.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbe6800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbe00cf310 0x7fdbe00c6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:41.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbe6800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbe00cf7a0 0x7fdbe00c4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:41.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbe6800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbe0158bc0 0x7fdbe006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:41.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbe6800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe00c5cc0 con 0x7fdbe0158bc0 2024-02-19T15:25:41.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbe6800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe00c50b0 con 0x7fdbe00cf310 2024-02-19T15:25:41.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbe6800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe00c5390 con 0x7fdbe00cf7a0 2024-02-19T15:25:41.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbdfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbe0158bc0 0x7fdbe006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:41.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbdfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbe0158bc0 0x7fdbe006e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48238/0 (socket says 172.21.15.47:48238) 2024-02-19T15:25:41.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbdfe00640 1 -- 172.21.15.47:0/1310720798 learned_addr learned my addr 172.21.15.47:0/1310720798 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:41.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbdf400640 1 --2- 172.21.15.47:0/1310720798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbe00cf310 0x7fdbe00c6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:41.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbd7e00640 1 --2- 172.21.15.47:0/1310720798 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbe00cf7a0 0x7fdbe00c4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:41.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbdfe00640 1 -- 172.21.15.47:0/1310720798 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbe00cf7a0 msgr2=0x7fdbe00c4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:41.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbdfe00640 1 --2- 172.21.15.47:0/1310720798 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbe00cf7a0 0x7fdbe00c4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbdfe00640 1 -- 172.21.15.47:0/1310720798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbe00cf310 msgr2=0x7fdbe00c6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:41.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbdfe00640 1 --2- 172.21.15.47:0/1310720798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbe00cf310 0x7fdbe00c6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbdfe00640 1 -- 172.21.15.47:0/1310720798 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd0009420 con 0x7fdbe0158bc0 2024-02-19T15:25:41.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbdfe00640 1 --2- 172.21.15.47:0/1310720798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbe0158bc0 0x7fdbe006e3e0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fdbd800c770 tx=0x7fdbd800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:41.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbd7400640 1 -- 172.21.15.47:0/1310720798 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd8003b80 con 0x7fdbe0158bc0 2024-02-19T15:25:41.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbd7400640 1 -- 172.21.15.47:0/1310720798 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd8003ce0 con 0x7fdbe0158bc0 2024-02-19T15:25:41.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbd7400640 1 -- 172.21.15.47:0/1310720798 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd800aa90 con 0x7fdbe0158bc0 2024-02-19T15:25:41.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.500+0000 7fdbe6800640 1 -- 172.21.15.47:0/1310720798 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe00c5670 con 0x7fdbe0158bc0 2024-02-19T15:25:41.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.504+0000 7fdbe6800640 1 -- 172.21.15.47:0/1310720798 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe00c4da0 con 0x7fdbe0158bc0 2024-02-19T15:25:41.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.504+0000 7fdbe6800640 1 -- 172.21.15.47:0/1310720798 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe0067a10 con 0x7fdbe0158bc0 2024-02-19T15:25:41.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.504+0000 7fdbd7400640 1 -- 172.21.15.47:0/1310720798 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fdbd800abf0 con 0x7fdbe0158bc0 2024-02-19T15:25:41.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.504+0000 7fdbd7400640 1 --2- 172.21.15.47:0/1310720798 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbbc066e70 0x7fdbbc069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:41.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.504+0000 7fdbd7400640 1 -- 172.21.15.47:0/1310720798 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdbd8084700 con 0x7fdbe0158bc0 2024-02-19T15:25:41.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.504+0000 7fdbdf400640 1 --2- 172.21.15.47:0/1310720798 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbbc066e70 0x7fdbbc069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:41.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.508+0000 7fdbdf400640 1 --2- 172.21.15.47:0/1310720798 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbbc066e70 0x7fdbbc069330 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdbd00af000 tx=0x7fdbd00a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:41.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.508+0000 7fdbd7400640 1 -- 172.21.15.47:0/1310720798 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fdbd8051590 con 0x7fdbe0158bc0 2024-02-19T15:25:41.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.680+0000 7fdbe6800640 1 -- 172.21.15.47:0/1310720798 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdbe0075360 con 0x7fdbe0158bc0 2024-02-19T15:25:41.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.680+0000 7fdbd7400640 1 -- 172.21.15.47:0/1310720798 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdbd8054c50 con 0x7fdbe0158bc0 2024-02-19T15:25:41.684 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:25:41.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.680+0000 7fdbe6800640 1 -- 172.21.15.47:0/1310720798 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbbc066e70 msgr2=0x7fdbbc069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:41.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.680+0000 7fdbe6800640 1 --2- 172.21.15.47:0/1310720798 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbbc066e70 0x7fdbbc069330 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdbd00af000 tx=0x7fdbd00a7000 comp rx=0 tx=0).stop 2024-02-19T15:25:41.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.680+0000 7fdbe6800640 1 -- 172.21.15.47:0/1310720798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbe0158bc0 msgr2=0x7fdbe006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:41.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.680+0000 7fdbe6800640 1 --2- 172.21.15.47:0/1310720798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbe0158bc0 0x7fdbe006e3e0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fdbd800c770 tx=0x7fdbd800cc30 comp rx=0 tx=0).stop 2024-02-19T15:25:41.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.680+0000 7fdbe6800640 1 -- 172.21.15.47:0/1310720798 shutdown_connections 2024-02-19T15:25:41.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.680+0000 7fdbe6800640 1 --2- 172.21.15.47:0/1310720798 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbbc066e70 0x7fdbbc069330 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.680+0000 7fdbe6800640 1 --2- 172.21.15.47:0/1310720798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbe0158bc0 0x7fdbe006e3e0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.680+0000 7fdbe6800640 1 --2- 172.21.15.47:0/1310720798 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbe00cf7a0 0x7fdbe00c4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.680+0000 7fdbe6800640 1 --2- 172.21.15.47:0/1310720798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbe00cf310 0x7fdbe00c6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.684+0000 7fdbe6800640 1 -- 172.21.15.47:0/1310720798 >> 172.21.15.47:0/1310720798 conn(0x7fdbe00d0d30 msgr2=0x7fdbe0157930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:41.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.684+0000 7fdbe6800640 1 -- 172.21.15.47:0/1310720798 shutdown_connections 2024-02-19T15:25:41.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.684+0000 7fdbe6800640 1 -- 172.21.15.47:0/1310720798 wait complete. 2024-02-19T15:25:41.706 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:25:41.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.844+0000 7f4d4c600640 1 Processor -- start 2024-02-19T15:25:41.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.844+0000 7f4d4c600640 1 -- start start 2024-02-19T15:25:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.844+0000 7f4d4c600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4d440d0820 0x7f4d440cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.844+0000 7f4d4c600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4d440c8260 0x7f4d440ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.844+0000 7f4d4c600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d440ceee0 con 0x7f4d440c7e80 2024-02-19T15:25:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.844+0000 7f4d4c600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d440cdc60 con 0x7f4d440c8260 2024-02-19T15:25:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.844+0000 7f4d4c600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d440d2a40 con 0x7f4d440d0820 2024-02-19T15:25:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.844+0000 7f4d4a600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4d440c8260 0x7f4d440ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.844+0000 7f4d4a600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4d440c8260 0x7f4d440ccc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48248/0 (socket says 172.21.15.47:48248) 2024-02-19T15:25:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.844+0000 7f4d4a600640 1 -- 172.21.15.47:0/673133076 learned_addr learned my addr 172.21.15.47:0/673133076 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.844+0000 7f4d49c00640 1 --1- 172.21.15.47:0/673133076 >> v1:172.21.15.47:6789/0 conn(0x7f4d440c7e80 0x7f4d440d0110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39566/0 (socket says 172.21.15.47:39566) 2024-02-19T15:25:41.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.844+0000 7f4d49200640 1 --2- 172.21.15.47:0/673133076 >> v2:172.21.15.116:3300/0 conn(0x7f4d440d0820 0x7f4d440cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:41.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.848+0000 7f4d4a600640 1 -- 172.21.15.47:0/673133076 >> v2:172.21.15.116:3300/0 conn(0x7f4d440d0820 msgr2=0x7f4d440cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:41.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.848+0000 7f4d4a600640 1 --2- 172.21.15.47:0/673133076 >> v2:172.21.15.116:3300/0 conn(0x7f4d440d0820 0x7f4d440cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.848+0000 7f4d4a600640 1 -- 172.21.15.47:0/673133076 >> v1:172.21.15.47:6789/0 conn(0x7f4d440c7e80 legacy=0x7f4d440d0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:41.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.848+0000 7f4d4a600640 1 -- 172.21.15.47:0/673133076 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d440cd980 con 0x7f4d440c8260 2024-02-19T15:25:41.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.848+0000 7f4d4a600640 1 --2- 172.21.15.47:0/673133076 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d440c8260 0x7f4d440ccc80 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f4d4415fbf0 tx=0x7f4d3c01c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a336cd5239661c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:41.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.848+0000 7f4d43e00640 1 -- 172.21.15.47:0/673133076 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c0a57e0 con 0x7f4d440c8260 2024-02-19T15:25:41.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.848+0000 7f4d43e00640 1 -- 172.21.15.47:0/673133076 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3c0a5940 con 0x7f4d440c8260 2024-02-19T15:25:41.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.848+0000 7f4d43e00640 1 -- 172.21.15.47:0/673133076 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c0a5cd0 con 0x7f4d440c8260 2024-02-19T15:25:41.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.848+0000 7f4d4c600640 1 -- 172.21.15.47:0/673133076 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d440c8260 msgr2=0x7f4d440ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:41.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.848+0000 7f4d4c600640 1 --2- 172.21.15.47:0/673133076 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d440c8260 0x7f4d440ccc80 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f4d4415fbf0 tx=0x7f4d3c01c9c0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d4c600640 1 -- 172.21.15.47:0/673133076 shutdown_connections 2024-02-19T15:25:41.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d4c600640 1 --2- 172.21.15.47:0/673133076 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d440c8260 0x7f4d440ccc80 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d4c600640 1 --2- 172.21.15.47:0/673133076 >> v2:172.21.15.116:3300/0 conn(0x7f4d440d0820 0x7f4d440cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d4c600640 1 -- 172.21.15.47:0/673133076 >> 172.21.15.47:0/673133076 conn(0x7f4d440d1b30 msgr2=0x7f4d440d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:41.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d4c600640 1 -- 172.21.15.47:0/673133076 shutdown_connections 2024-02-19T15:25:41.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d4c600640 1 -- 172.21.15.47:0/673133076 wait complete. 2024-02-19T15:25:41.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d4c600640 1 Processor -- start 2024-02-19T15:25:41.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d4c600640 1 -- start start 2024-02-19T15:25:41.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d4c600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d440c7e80 0x7f4d441f7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:41.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d4c600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d440c8260 0x7f4d44151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:41.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d4c600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4d440d0820 0x7f4d44159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:41.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d4c600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d44152860 con 0x7f4d440c8260 2024-02-19T15:25:41.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d4c600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4415c420 con 0x7f4d440c7e80 2024-02-19T15:25:41.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d4c600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4415c700 con 0x7f4d440d0820 2024-02-19T15:25:41.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d49200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d440c8260 0x7f4d44151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:41.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d49200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d440c8260 0x7f4d44151910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48262/0 (socket says 172.21.15.47:48262) 2024-02-19T15:25:41.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d49200640 1 -- 172.21.15.47:0/2132638488 learned_addr learned my addr 172.21.15.47:0/2132638488 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:41.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d4a600640 1 --2- 172.21.15.47:0/2132638488 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4d440d0820 0x7f4d44159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:41.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d49c00640 1 --2- 172.21.15.47:0/2132638488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d440c7e80 0x7f4d441f7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:41.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d49200640 1 -- 172.21.15.47:0/2132638488 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4d440d0820 msgr2=0x7f4d44159fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:41.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d49200640 1 --2- 172.21.15.47:0/2132638488 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4d440d0820 0x7f4d44159fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d49200640 1 -- 172.21.15.47:0/2132638488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d440c7e80 msgr2=0x7f4d441f7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:41.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d49200640 1 --2- 172.21.15.47:0/2132638488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d440c7e80 0x7f4d441f7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:41.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d49200640 1 -- 172.21.15.47:0/2132638488 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d3c009b30 con 0x7f4d440c8260 2024-02-19T15:25:41.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d49c00640 1 --2- 172.21.15.47:0/2132638488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d440c7e80 0x7f4d441f7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:41.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d49200640 1 --2- 172.21.15.47:0/2132638488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d440c8260 0x7f4d44151910 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000d5f0 tx=0x7f4d3000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:41.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.852+0000 7f4d41600640 1 -- 172.21.15.47:0/2132638488 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d300125f0 con 0x7f4d440c8260 2024-02-19T15:25:41.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.856+0000 7f4d41600640 1 -- 172.21.15.47:0/2132638488 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d30012750 con 0x7f4d440c8260 2024-02-19T15:25:41.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.856+0000 7f4d41600640 1 -- 172.21.15.47:0/2132638488 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d30012ae0 con 0x7f4d440c8260 2024-02-19T15:25:41.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.856+0000 7f4d4c600640 1 -- 172.21.15.47:0/2132638488 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4415c0d0 con 0x7f4d440c8260 2024-02-19T15:25:41.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.856+0000 7f4d4c600640 1 -- 172.21.15.47:0/2132638488 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d440cb460 con 0x7f4d440c8260 2024-02-19T15:25:41.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.856+0000 7f4d4c600640 1 -- 172.21.15.47:0/2132638488 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d4415c8c0 con 0x7f4d440c8260 2024-02-19T15:25:41.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.856+0000 7f4d41600640 1 -- 172.21.15.47:0/2132638488 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4d30012c40 con 0x7f4d440c8260 2024-02-19T15:25:41.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.856+0000 7f4d41600640 1 --2- 172.21.15.47:0/2132638488 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4d24066e40 0x7f4d24069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:41.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.856+0000 7f4d41600640 1 -- 172.21.15.47:0/2132638488 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d30084fb0 con 0x7f4d440c8260 2024-02-19T15:25:41.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.856+0000 7f4d49c00640 1 --2- 172.21.15.47:0/2132638488 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4d24066e40 0x7f4d24069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:41.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.860+0000 7f4d49c00640 1 --2- 172.21.15.47:0/2132638488 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4d24066e40 0x7f4d24069300 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4d38000f70 tx=0x7f4d38007d70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:41.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:41.864+0000 7f4d41600640 1 -- 172.21.15.47:0/2132638488 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f4d30051d00 con 0x7f4d440c8260 2024-02-19T15:25:42.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.028+0000 7f4d4c600640 1 -- 172.21.15.47:0/2132638488 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4d440d1180 con 0x7f4d440c8260 2024-02-19T15:25:42.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.028+0000 7f4d41600640 1 -- 172.21.15.47:0/2132638488 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4d30007380 con 0x7f4d440c8260 2024-02-19T15:25:42.033 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:25:42.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.032+0000 7f4d4c600640 1 -- 172.21.15.47:0/2132638488 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4d24066e40 msgr2=0x7f4d24069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:42.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.032+0000 7f4d4c600640 1 --2- 172.21.15.47:0/2132638488 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4d24066e40 0x7f4d24069300 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4d38000f70 tx=0x7f4d38007d70 comp rx=0 tx=0).stop 2024-02-19T15:25:42.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.032+0000 7f4d4c600640 1 -- 172.21.15.47:0/2132638488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d440c8260 msgr2=0x7f4d44151910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:42.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.032+0000 7f4d4c600640 1 --2- 172.21.15.47:0/2132638488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d440c8260 0x7f4d44151910 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000d5f0 tx=0x7f4d3000dab0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.032+0000 7f4d4c600640 1 -- 172.21.15.47:0/2132638488 shutdown_connections 2024-02-19T15:25:42.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.032+0000 7f4d4c600640 1 --2- 172.21.15.47:0/2132638488 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4d24066e40 0x7f4d24069300 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.032+0000 7f4d4c600640 1 --2- 172.21.15.47:0/2132638488 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4d440d0820 0x7f4d44159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.032+0000 7f4d4c600640 1 --2- 172.21.15.47:0/2132638488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d440c8260 0x7f4d44151910 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.032+0000 7f4d4c600640 1 --2- 172.21.15.47:0/2132638488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d440c7e80 0x7f4d441f7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.032+0000 7f4d4c600640 1 -- 172.21.15.47:0/2132638488 >> 172.21.15.47:0/2132638488 conn(0x7f4d440d1b30 msgr2=0x7f4d440c9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:42.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.032+0000 7f4d4c600640 1 -- 172.21.15.47:0/2132638488 shutdown_connections 2024-02-19T15:25:42.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.032+0000 7f4d4c600640 1 -- 172.21.15.47:0/2132638488 wait complete. 2024-02-19T15:25:42.055 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:25:42.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.196+0000 7f43bcc00640 1 Processor -- start 2024-02-19T15:25:42.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.196+0000 7f43bcc00640 1 -- start start 2024-02-19T15:25:42.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.196+0000 7f43bcc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43b80ce3b0 0x7f43b80ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:42.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.196+0000 7f43bcc00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43b815e280 0x7f43b80cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:42.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.196+0000 7f43bcc00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b80cc620 con 0x7f43b80d2650 2024-02-19T15:25:42.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.196+0000 7f43bcc00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b80d12e0 con 0x7f43b815e280 2024-02-19T15:25:42.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.196+0000 7f43bcc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b80d1940 con 0x7f43b80ce3b0 2024-02-19T15:25:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.196+0000 7f43b5e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43b80ce3b0 0x7f43b80ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.196+0000 7f43b6800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43b815e280 0x7f43b80cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.196+0000 7f43b6800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43b815e280 0x7f43b80cd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:36106/0 (socket says 172.21.15.47:36106) 2024-02-19T15:25:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.196+0000 7f43b5400640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f43b80d2650 0x7f43b80ccab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:34628/0 (socket says 172.21.15.47:34628) 2024-02-19T15:25:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.196+0000 7f43b6800640 1 -- 172.21.15.47:0/1450703264 learned_addr learned my addr 172.21.15.47:0/1450703264 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.196+0000 7f43b5e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43b80ce3b0 0x7f43b80ce790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:54746/0 (socket says 172.21.15.47:54746) 2024-02-19T15:25:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.196+0000 7f43b6800640 1 -- 172.21.15.47:0/1450703264 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43b80ce3b0 msgr2=0x7f43b80ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.196+0000 7f43b6800640 1 --2- 172.21.15.47:0/1450703264 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43b80ce3b0 0x7f43b80ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.196+0000 7f43b6800640 1 -- 172.21.15.47:0/1450703264 >> v1:172.21.15.116:6789/0 conn(0x7f43b80d2650 legacy=0x7f43b80ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:42.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.196+0000 7f43b6800640 1 -- 172.21.15.47:0/1450703264 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b80cecd0 con 0x7f43b815e280 2024-02-19T15:25:42.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.200+0000 7f43b6800640 1 --2- 172.21.15.47:0/1450703264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b815e280 0x7f43b80cd1c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f43b0009e50 tx=0x7f43b001c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97e53e56372cafdb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:42.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.200+0000 7f43b4a00640 1 -- 172.21.15.47:0/1450703264 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b00a56c0 con 0x7f43b815e280 2024-02-19T15:25:42.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.200+0000 7f43b4a00640 1 -- 172.21.15.47:0/1450703264 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b00a5820 con 0x7f43b815e280 2024-02-19T15:25:42.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.200+0000 7f43b4a00640 1 -- 172.21.15.47:0/1450703264 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b00a5bb0 con 0x7f43b815e280 2024-02-19T15:25:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.200+0000 7f43bcc00640 1 -- 172.21.15.47:0/1450703264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b815e280 msgr2=0x7f43b80cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.200+0000 7f43bcc00640 1 --2- 172.21.15.47:0/1450703264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b815e280 0x7f43b80cd1c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f43b0009e50 tx=0x7f43b001c910 comp rx=0 tx=0).stop 2024-02-19T15:25:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.200+0000 7f43bcc00640 1 -- 172.21.15.47:0/1450703264 shutdown_connections 2024-02-19T15:25:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.200+0000 7f43bcc00640 1 --2- 172.21.15.47:0/1450703264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b815e280 0x7f43b80cd1c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.200+0000 7f43bcc00640 1 --2- 172.21.15.47:0/1450703264 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43b80ce3b0 0x7f43b80ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.200+0000 7f43bcc00640 1 -- 172.21.15.47:0/1450703264 >> 172.21.15.47:0/1450703264 conn(0x7f43b80d1b30 msgr2=0x7f43b80d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.200+0000 7f43bcc00640 1 -- 172.21.15.47:0/1450703264 shutdown_connections 2024-02-19T15:25:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.200+0000 7f43bcc00640 1 -- 172.21.15.47:0/1450703264 wait complete. 2024-02-19T15:25:42.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.200+0000 7f43bcc00640 1 Processor -- start 2024-02-19T15:25:42.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.200+0000 7f43bcc00640 1 -- start start 2024-02-19T15:25:42.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43bcc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43b80ce3b0 0x7f43b814c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:42.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43bcc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43b80d2650 0x7f43b815cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:42.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43bcc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b815e280 0x7f43b815d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:42.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43bcc00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b815a060 con 0x7f43b80d2650 2024-02-19T15:25:42.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43bcc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b815a2b0 con 0x7f43b815e280 2024-02-19T15:25:42.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43bcc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b815a590 con 0x7f43b80ce3b0 2024-02-19T15:25:42.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43b5400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43b80d2650 0x7f43b815cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:42.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43b5e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43b80ce3b0 0x7f43b814c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:42.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43b6800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b815e280 0x7f43b815d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:42.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43b5400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43b80d2650 0x7f43b815cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48266/0 (socket says 172.21.15.47:48266) 2024-02-19T15:25:42.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43b5400640 1 -- 172.21.15.47:0/2015251126 learned_addr learned my addr 172.21.15.47:0/2015251126 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:42.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43b5400640 1 -- 172.21.15.47:0/2015251126 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43b80ce3b0 msgr2=0x7f43b814c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:42.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43b5400640 1 --2- 172.21.15.47:0/2015251126 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43b80ce3b0 0x7f43b814c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43b5400640 1 -- 172.21.15.47:0/2015251126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b815e280 msgr2=0x7f43b815d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:42.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43b5400640 1 --2- 172.21.15.47:0/2015251126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b815e280 0x7f43b815d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43b5400640 1 -- 172.21.15.47:0/2015251126 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b0009b30 con 0x7f43b80d2650 2024-02-19T15:25:42.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43b5e00640 1 --2- 172.21.15.47:0/2015251126 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43b80ce3b0 0x7f43b814c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:25:42.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43b5400640 1 --2- 172.21.15.47:0/2015251126 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43b80d2650 0x7f43b815cd50 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f43a400c370 tx=0x7f43a400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:42.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43ae000640 1 -- 172.21.15.47:0/2015251126 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a40074e0 con 0x7f43b80d2650 2024-02-19T15:25:42.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43ae000640 1 -- 172.21.15.47:0/2015251126 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a4007640 con 0x7f43b80d2650 2024-02-19T15:25:42.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43ae000640 1 -- 172.21.15.47:0/2015251126 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a4004a30 con 0x7f43b80d2650 2024-02-19T15:25:42.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43bcc00640 1 -- 172.21.15.47:0/2015251126 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b815a870 con 0x7f43b80d2650 2024-02-19T15:25:42.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43bcc00640 1 -- 172.21.15.47:0/2015251126 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b815ae30 con 0x7f43b80d2650 2024-02-19T15:25:42.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.204+0000 7f43bcc00640 1 -- 172.21.15.47:0/2015251126 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b8075360 con 0x7f43b80d2650 2024-02-19T15:25:42.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.208+0000 7f43ae000640 1 -- 172.21.15.47:0/2015251126 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f43a4004b90 con 0x7f43b80d2650 2024-02-19T15:25:42.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.208+0000 7f43ae000640 1 --2- 172.21.15.47:0/2015251126 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4398066e40 0x7f4398069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:42.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.208+0000 7f43ae000640 1 -- 172.21.15.47:0/2015251126 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43a4084280 con 0x7f43b80d2650 2024-02-19T15:25:42.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.208+0000 7f43b5e00640 1 --2- 172.21.15.47:0/2015251126 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4398066e40 0x7f4398069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:42.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.208+0000 7f43b5e00640 1 --2- 172.21.15.47:0/2015251126 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4398066e40 0x7f4398069300 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f43b80700c0 tx=0x7f43a8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:42.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.212+0000 7f43ae000640 1 -- 172.21.15.47:0/2015251126 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f43a4051120 con 0x7f43b80d2650 2024-02-19T15:25:42.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.372+0000 7f43bcc00640 1 -- 172.21.15.47:0/2015251126 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f43b80cdf80 con 0x7f43b80d2650 2024-02-19T15:25:42.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.372+0000 7f43ae000640 1 -- 172.21.15.47:0/2015251126 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7f43a40547e0 con 0x7f43b80d2650 2024-02-19T15:25:42.378 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:25:42.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.376+0000 7f43bcc00640 1 -- 172.21.15.47:0/2015251126 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4398066e40 msgr2=0x7f4398069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:42.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.376+0000 7f43bcc00640 1 --2- 172.21.15.47:0/2015251126 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4398066e40 0x7f4398069300 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f43b80700c0 tx=0x7f43a8009190 comp rx=0 tx=0).stop 2024-02-19T15:25:42.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.376+0000 7f43bcc00640 1 -- 172.21.15.47:0/2015251126 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43b80d2650 msgr2=0x7f43b815cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:42.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.376+0000 7f43bcc00640 1 --2- 172.21.15.47:0/2015251126 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43b80d2650 0x7f43b815cd50 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f43a400c370 tx=0x7f43a400c830 comp rx=0 tx=0).stop 2024-02-19T15:25:42.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.376+0000 7f43bcc00640 1 -- 172.21.15.47:0/2015251126 shutdown_connections 2024-02-19T15:25:42.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.376+0000 7f43bcc00640 1 --2- 172.21.15.47:0/2015251126 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4398066e40 0x7f4398069300 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.376+0000 7f43bcc00640 1 --2- 172.21.15.47:0/2015251126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b815e280 0x7f43b815d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.376+0000 7f43bcc00640 1 --2- 172.21.15.47:0/2015251126 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43b80d2650 0x7f43b815cd50 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.376+0000 7f43bcc00640 1 --2- 172.21.15.47:0/2015251126 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43b80ce3b0 0x7f43b814c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.376+0000 7f43bcc00640 1 -- 172.21.15.47:0/2015251126 >> 172.21.15.47:0/2015251126 conn(0x7f43b80d1b30 msgr2=0x7f43b80c5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:42.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.376+0000 7f43bcc00640 1 -- 172.21.15.47:0/2015251126 shutdown_connections 2024-02-19T15:25:42.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.376+0000 7f43bcc00640 1 -- 172.21.15.47:0/2015251126 wait complete. 2024-02-19T15:25:42.400 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-19T15:25:42.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.536+0000 7fb785600640 1 Processor -- start 2024-02-19T15:25:42.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.536+0000 7fb785600640 1 -- start start 2024-02-19T15:25:42.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.540+0000 7fb785600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb7800cef90 0x7fb7800ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:42.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.540+0000 7fb785600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7800ced80 con 0x7fb7800ce930 2024-02-19T15:25:42.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.540+0000 7fb785600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7800d10a0 con 0x7fb7800b2fc0 2024-02-19T15:25:42.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.540+0000 7fb785600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7800d0e80 con 0x7fb7800cef90 2024-02-19T15:25:42.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.540+0000 7fb77f400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb7800ce930 0x7fb7800cd0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39576/0 (socket says 172.21.15.47:39576) 2024-02-19T15:25:42.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.540+0000 7fb77f400640 1 -- 172.21.15.47:0/1409285055 learned_addr learned my addr 172.21.15.47:0/1409285055 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:42.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.540+0000 7fb776000640 1 --2- 172.21.15.47:0/1409285055 >> v2:172.21.15.116:3300/0 conn(0x7fb7800cef90 0x7fb7800ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:42.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.540+0000 7fb77e000640 1 -- 172.21.15.47:0/1409285055 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029900231 0 0) 0x7fb778001120 con 0x7fb7800ce930 2024-02-19T15:25:42.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.540+0000 7fb77e000640 1 -- 172.21.15.47:0/1409285055 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb764003530 con 0x7fb7800ce930 2024-02-19T15:25:42.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.540+0000 7fb776000640 1 -- 172.21.15.47:0/1409285055 >> v1:172.21.15.116:6789/0 conn(0x7fb7800b2fc0 legacy=0x7fb7800d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:42.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.540+0000 7fb776000640 1 -- 172.21.15.47:0/1409285055 >> v1:172.21.15.47:6789/0 conn(0x7fb7800ce930 legacy=0x7fb7800cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:42.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.540+0000 7fb776000640 1 -- 172.21.15.47:0/1409285055 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7800b33a0 con 0x7fb7800cef90 2024-02-19T15:25:42.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.544+0000 7fb776000640 1 --2- 172.21.15.47:0/1409285055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7800cef90 0x7fb7800ce3f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb768009d90 tx=0x7fb76801c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15f2c8db14f67878 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:42.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.544+0000 7fb77e000640 1 -- 172.21.15.47:0/1409285055 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7680a5970 con 0x7fb7800cef90 2024-02-19T15:25:42.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.544+0000 7fb77e000640 1 -- 172.21.15.47:0/1409285055 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7680a5ad0 con 0x7fb7800cef90 2024-02-19T15:25:42.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.544+0000 7fb77e000640 1 -- 172.21.15.47:0/1409285055 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7680a5e40 con 0x7fb7800cef90 2024-02-19T15:25:42.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.544+0000 7fb785600640 1 -- 172.21.15.47:0/1409285055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7800cef90 msgr2=0x7fb7800ce3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:42.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.544+0000 7fb785600640 1 --2- 172.21.15.47:0/1409285055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7800cef90 0x7fb7800ce3f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb768009d90 tx=0x7fb76801c8b0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.544+0000 7fb785600640 1 -- 172.21.15.47:0/1409285055 shutdown_connections 2024-02-19T15:25:42.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.544+0000 7fb785600640 1 --2- 172.21.15.47:0/1409285055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7800cef90 0x7fb7800ce3f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.544+0000 7fb785600640 1 -- 172.21.15.47:0/1409285055 >> 172.21.15.47:0/1409285055 conn(0x7fb7800d1b70 msgr2=0x7fb7800d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:42.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.544+0000 7fb785600640 1 -- 172.21.15.47:0/1409285055 shutdown_connections 2024-02-19T15:25:42.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.544+0000 7fb785600640 1 -- 172.21.15.47:0/1409285055 wait complete. 2024-02-19T15:25:42.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.544+0000 7fb785600640 1 Processor -- start 2024-02-19T15:25:42.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.544+0000 7fb785600640 1 -- start start 2024-02-19T15:25:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.544+0000 7fb785600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7800b2fc0 0x7fb780153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.544+0000 7fb785600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb7800ce930 0x7fb780141b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.544+0000 7fb785600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7800cef90 0x7fb7801420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.544+0000 7fb785600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78015fa70 con 0x7fb7800b2fc0 2024-02-19T15:25:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.544+0000 7fb785600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7800c84a0 con 0x7fb7800cef90 2024-02-19T15:25:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.544+0000 7fb785600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78013fc30 con 0x7fb7800ce930 2024-02-19T15:25:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.548+0000 7fb77ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7800b2fc0 0x7fb780153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.548+0000 7fb77ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7800b2fc0 0x7fb780153440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48280/0 (socket says 172.21.15.47:48280) 2024-02-19T15:25:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.548+0000 7fb77ea00640 1 -- 172.21.15.47:0/1608972700 learned_addr learned my addr 172.21.15.47:0/1608972700 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.548+0000 7fb776000640 1 --2- 172.21.15.47:0/1608972700 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb7800ce930 0x7fb780141b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.548+0000 7fb77f400640 1 --2- 172.21.15.47:0/1608972700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7800cef90 0x7fb7801420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.548+0000 7fb77ea00640 1 -- 172.21.15.47:0/1608972700 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb7800ce930 msgr2=0x7fb780141b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.548+0000 7fb77ea00640 1 --2- 172.21.15.47:0/1608972700 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb7800ce930 0x7fb780141b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.548+0000 7fb77ea00640 1 -- 172.21.15.47:0/1608972700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7800cef90 msgr2=0x7fb7801420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.548+0000 7fb77ea00640 1 --2- 172.21.15.47:0/1608972700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7800cef90 0x7fb7801420d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.548+0000 7fb77ea00640 1 -- 172.21.15.47:0/1608972700 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb768009a70 con 0x7fb7800b2fc0 2024-02-19T15:25:42.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.548+0000 7fb77ea00640 1 --2- 172.21.15.47:0/1608972700 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7800b2fc0 0x7fb780153440 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fb77000c3c0 tx=0x7fb77000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:42.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.548+0000 7fb777400640 1 -- 172.21.15.47:0/1608972700 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7700052d0 con 0x7fb7800b2fc0 2024-02-19T15:25:42.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.548+0000 7fb777400640 1 -- 172.21.15.47:0/1608972700 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb770005430 con 0x7fb7800b2fc0 2024-02-19T15:25:42.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.548+0000 7fb785600640 1 -- 172.21.15.47:0/1608972700 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7800c81c0 con 0x7fb7800b2fc0 2024-02-19T15:25:42.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.548+0000 7fb777400640 1 -- 172.21.15.47:0/1608972700 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb770003840 con 0x7fb7800b2fc0 2024-02-19T15:25:42.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.548+0000 7fb785600640 1 -- 172.21.15.47:0/1608972700 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7801401b0 con 0x7fb7800b2fc0 2024-02-19T15:25:42.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.548+0000 7fb785600640 1 -- 172.21.15.47:0/1608972700 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb780075360 con 0x7fb7800b2fc0 2024-02-19T15:25:42.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.548+0000 7fb777400640 1 -- 172.21.15.47:0/1608972700 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb7700039a0 con 0x7fb7800b2fc0 2024-02-19T15:25:42.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.552+0000 7fb777400640 1 --2- 172.21.15.47:0/1608972700 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb75c066e40 0x7fb75c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:42.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.552+0000 7fb777400640 1 -- 172.21.15.47:0/1608972700 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb770083e90 con 0x7fb7800b2fc0 2024-02-19T15:25:42.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.552+0000 7fb776000640 1 --2- 172.21.15.47:0/1608972700 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb75c066e40 0x7fb75c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:42.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.552+0000 7fb776000640 1 --2- 172.21.15.47:0/1608972700 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb75c066e40 0x7fb75c069300 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb7680af000 tx=0x7fb7680a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:42.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.556+0000 7fb777400640 1 -- 172.21.15.47:0/1608972700 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb770050dc0 con 0x7fb7800b2fc0 2024-02-19T15:25:42.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.728+0000 7fb785600640 1 -- 172.21.15.47:0/1608972700 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb780153980 con 0x7fb7800b2fc0 2024-02-19T15:25:42.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.728+0000 7fb777400640 1 -- 172.21.15.47:0/1608972700 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v69) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb770054480 con 0x7fb7800b2fc0 2024-02-19T15:25:42.733 INFO:teuthology.orchestra.run.smithi047.stderr:listed 0 entries 2024-02-19T15:25:42.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.732+0000 7fb785600640 1 -- 172.21.15.47:0/1608972700 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb75c066e40 msgr2=0x7fb75c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:42.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.732+0000 7fb785600640 1 --2- 172.21.15.47:0/1608972700 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb75c066e40 0x7fb75c069300 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb7680af000 tx=0x7fb7680a7070 comp rx=0 tx=0).stop 2024-02-19T15:25:42.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.732+0000 7fb785600640 1 -- 172.21.15.47:0/1608972700 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7800b2fc0 msgr2=0x7fb780153440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:42.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.732+0000 7fb785600640 1 --2- 172.21.15.47:0/1608972700 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7800b2fc0 0x7fb780153440 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fb77000c3c0 tx=0x7fb77000c880 comp rx=0 tx=0).stop 2024-02-19T15:25:42.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.732+0000 7fb785600640 1 -- 172.21.15.47:0/1608972700 shutdown_connections 2024-02-19T15:25:42.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.732+0000 7fb785600640 1 --2- 172.21.15.47:0/1608972700 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb75c066e40 0x7fb75c069300 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.732+0000 7fb785600640 1 --2- 172.21.15.47:0/1608972700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7800cef90 0x7fb7801420d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.732+0000 7fb785600640 1 --2- 172.21.15.47:0/1608972700 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb7800ce930 0x7fb780141b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.732+0000 7fb785600640 1 --2- 172.21.15.47:0/1608972700 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7800b2fc0 0x7fb780153440 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:42.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.732+0000 7fb785600640 1 -- 172.21.15.47:0/1608972700 >> 172.21.15.47:0/1608972700 conn(0x7fb7800d1b70 msgr2=0x7fb78006ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:42.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.732+0000 7fb785600640 1 -- 172.21.15.47:0/1608972700 shutdown_connections 2024-02-19T15:25:42.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:42.732+0000 7fb785600640 1 -- 172.21.15.47:0/1608972700 wait complete. 2024-02-19T15:25:42.753 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-02-19T15:25:42.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:25:42.753 DEBUG:teuthology.orchestra.run.smithi047:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-02-19T15:25:42.806 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi047.front.sepia.ceph.com 2024-02-19T15:25:42.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:25:42.807 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-19T15:25:42.833 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:25:42.833 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:25:42.842 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:25:42.842 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:25:42.842 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:25:42.884 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:25:42.884 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:25:42.884 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:25:42.893 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:25:42.893 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:25:43.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.064+0000 7f0bb6400640 1 Processor -- start 2024-02-19T15:25:43.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.064+0000 7f0bb6400640 1 -- start start 2024-02-19T15:25:43.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.064+0000 7f0bb6400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0bb00b2500 0x7f0bb006ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:43.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.064+0000 7f0bb6400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb006b5d0 con 0x7f0bb015f620 2024-02-19T15:25:43.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.064+0000 7f0bb6400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb00b1640 con 0x7f0bb01614e0 2024-02-19T15:25:43.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.064+0000 7f0bb6400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb00b1c00 con 0x7f0bb00b2500 2024-02-19T15:25:43.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.064+0000 7f0bafe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0bb00b2500 0x7f0bb006ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:43.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.064+0000 7f0ba6a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0bb015f620 0x7f0bb00b1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39592/0 (socket says 172.21.15.47:39592) 2024-02-19T15:25:43.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.064+0000 7f0ba6a00640 1 -- 172.21.15.47:0/1484304928 learned_addr learned my addr 172.21.15.47:0/1484304928 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.064+0000 7f0baea00640 1 -- 172.21.15.47:0/1484304928 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265810817 0 0) 0x7f0bb006b5d0 con 0x7f0bb015f620 2024-02-19T15:25:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.064+0000 7f0baea00640 1 -- 172.21.15.47:0/1484304928 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b94003530 con 0x7f0bb015f620 2024-02-19T15:25:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.064+0000 7f0baea00640 1 -- 172.21.15.47:0/1484304928 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 363282205 0 0) 0x7f0bb00b1640 con 0x7f0bb01614e0 2024-02-19T15:25:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.064+0000 7f0bafe00640 1 -- 172.21.15.47:0/1484304928 >> v1:172.21.15.116:6789/0 conn(0x7f0bb01614e0 legacy=0x7f0bb015eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.064+0000 7f0bafe00640 1 -- 172.21.15.47:0/1484304928 >> v1:172.21.15.47:6789/0 conn(0x7f0bb015f620 legacy=0x7f0bb00b1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.064+0000 7f0bafe00640 1 -- 172.21.15.47:0/1484304928 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb00b1920 con 0x7f0bb00b2500 2024-02-19T15:25:43.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.068+0000 7f0bafe00640 1 --2- 172.21.15.47:0/1484304928 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bb00b2500 0x7f0bb006ba60 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8009dc0 tx=0x7f0ba801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9deec2c6d4b5624 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:43.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0baea00640 1 -- 172.21.15.47:0/1484304928 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba80a5920 con 0x7f0bb00b2500 2024-02-19T15:25:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0baea00640 1 -- 172.21.15.47:0/1484304928 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba80a5a80 con 0x7f0bb00b2500 2024-02-19T15:25:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0baea00640 1 -- 172.21.15.47:0/1484304928 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba80a5e10 con 0x7f0bb00b2500 2024-02-19T15:25:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0bb6400640 1 -- 172.21.15.47:0/1484304928 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bb00b2500 msgr2=0x7f0bb006ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0bb6400640 1 --2- 172.21.15.47:0/1484304928 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bb00b2500 0x7f0bb006ba60 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8009dc0 tx=0x7f0ba801c950 comp rx=0 tx=0).stop 2024-02-19T15:25:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0bb6400640 1 -- 172.21.15.47:0/1484304928 shutdown_connections 2024-02-19T15:25:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0bb6400640 1 --2- 172.21.15.47:0/1484304928 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bb00b2500 0x7f0bb006ba60 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0bb6400640 1 -- 172.21.15.47:0/1484304928 >> 172.21.15.47:0/1484304928 conn(0x7f0bb00b4fc0 msgr2=0x7f0bb00b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:43.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0bb6400640 1 -- 172.21.15.47:0/1484304928 shutdown_connections 2024-02-19T15:25:43.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0bb6400640 1 -- 172.21.15.47:0/1484304928 wait complete. 2024-02-19T15:25:43.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0bb6400640 1 Processor -- start 2024-02-19T15:25:43.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0bb6400640 1 -- start start 2024-02-19T15:25:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0bb6400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bb00b2500 0x7f0bb00cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0bb6400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0bb015f620 0x7f0bb00ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0bb6400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bb01614e0 0x7f0bb0156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0bb6400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb01fa3f0 con 0x7f0bb01614e0 2024-02-19T15:25:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0bb6400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb00cd080 con 0x7f0bb00b2500 2024-02-19T15:25:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0bb6400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb0156850 con 0x7f0bb015f620 2024-02-19T15:25:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0bafe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bb01614e0 0x7f0bb0156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0bafe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bb01614e0 0x7f0bb0156310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48290/0 (socket says 172.21.15.47:48290) 2024-02-19T15:25:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.072+0000 7f0bafe00640 1 -- 172.21.15.47:0/3391067779 learned_addr learned my addr 172.21.15.47:0/3391067779 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.076+0000 7f0baf400640 1 --2- 172.21.15.47:0/3391067779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bb00b2500 0x7f0bb00cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.076+0000 7f0ba6a00640 1 --2- 172.21.15.47:0/3391067779 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0bb015f620 0x7f0bb00ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.076+0000 7f0bafe00640 1 -- 172.21.15.47:0/3391067779 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0bb015f620 msgr2=0x7f0bb00ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.076+0000 7f0bafe00640 1 --2- 172.21.15.47:0/3391067779 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0bb015f620 0x7f0bb00ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.076+0000 7f0bafe00640 1 -- 172.21.15.47:0/3391067779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bb00b2500 msgr2=0x7f0bb00cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:43.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.076+0000 7f0bafe00640 1 --2- 172.21.15.47:0/3391067779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bb00b2500 0x7f0bb00cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.076+0000 7f0bafe00640 1 -- 172.21.15.47:0/3391067779 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba8009a70 con 0x7f0bb01614e0 2024-02-19T15:25:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.076+0000 7f0bafe00640 1 --2- 172.21.15.47:0/3391067779 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bb01614e0 0x7f0bb0156310 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f0ba801ce30 tx=0x7f0ba80196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.076+0000 7f0ba7e00640 1 -- 172.21.15.47:0/3391067779 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba80add50 con 0x7f0bb01614e0 2024-02-19T15:25:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.076+0000 7f0ba7e00640 1 -- 172.21.15.47:0/3391067779 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba8009450 con 0x7f0bb01614e0 2024-02-19T15:25:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.076+0000 7f0bb6400640 1 -- 172.21.15.47:0/3391067779 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb00cd2b0 con 0x7f0bb01614e0 2024-02-19T15:25:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.076+0000 7f0ba7e00640 1 -- 172.21.15.47:0/3391067779 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8003be0 con 0x7f0bb01614e0 2024-02-19T15:25:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.076+0000 7f0bb6400640 1 -- 172.21.15.47:0/3391067779 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb00cd870 con 0x7f0bb01614e0 2024-02-19T15:25:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.076+0000 7f0bb6400640 1 -- 172.21.15.47:0/3391067779 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb0075360 con 0x7f0bb01614e0 2024-02-19T15:25:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.076+0000 7f0ba7e00640 1 -- 172.21.15.47:0/3391067779 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0ba80b4050 con 0x7f0bb01614e0 2024-02-19T15:25:43.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.076+0000 7f0ba7e00640 1 --2- 172.21.15.47:0/3391067779 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0b8c066e40 0x7f0b8c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:43.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.076+0000 7f0ba7e00640 1 -- 172.21.15.47:0/3391067779 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ba811f630 con 0x7f0bb01614e0 2024-02-19T15:25:43.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.080+0000 7f0baf400640 1 --2- 172.21.15.47:0/3391067779 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0b8c066e40 0x7f0b8c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:43.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.080+0000 7f0baf400640 1 --2- 172.21.15.47:0/3391067779 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0b8c066e40 0x7f0b8c069300 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0157e60 tx=0x7f0ba0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:43.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.080+0000 7f0ba7e00640 1 -- 172.21.15.47:0/3391067779 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0ba80ec4c0 con 0x7f0bb01614e0 2024-02-19T15:25:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.404+0000 7f0bb6400640 1 -- 172.21.15.47:0/3391067779 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bb006ee70 con 0x7f0bb01614e0 2024-02-19T15:25:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.404+0000 7f0ba7e00640 1 -- 172.21.15.47:0/3391067779 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4946 (secure 0 0 0) 0x7f0ba80efb80 con 0x7f0bb01614e0 2024-02-19T15:25:43.408 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:43.408 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5684,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3206627105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3206627105},{"type":"v1","addr":"172.21.15.116:6836","nonce":3206627105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5687,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3532185042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3532185042},{"type":"v1","addr":"172.21.15.116:6837","nonce":3532185042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/1482478568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1482478568},{"type":"v1","addr":"172.21.15.47:6834","nonce":1482478568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:25:08.947740+0000","modified":"2024-02-19T15:25:11.954116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/2490197058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2490197058},{"type":"v1","addr":"172.21.15.47:6835","nonce":2490197058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-02-19T15:25:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 40 2024-02-19T15:25:43.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.408+0000 7f0bb6400640 1 -- 172.21.15.47:0/3391067779 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0b8c066e40 msgr2=0x7f0b8c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:43.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.408+0000 7f0bb6400640 1 --2- 172.21.15.47:0/3391067779 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0b8c066e40 0x7f0b8c069300 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0157e60 tx=0x7f0ba0009040 comp rx=0 tx=0).stop 2024-02-19T15:25:43.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.408+0000 7f0bb6400640 1 -- 172.21.15.47:0/3391067779 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bb01614e0 msgr2=0x7f0bb0156310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:43.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.408+0000 7f0bb6400640 1 --2- 172.21.15.47:0/3391067779 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bb01614e0 0x7f0bb0156310 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f0ba801ce30 tx=0x7f0ba80196f0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.408+0000 7f0bb6400640 1 -- 172.21.15.47:0/3391067779 shutdown_connections 2024-02-19T15:25:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.408+0000 7f0bb6400640 1 --2- 172.21.15.47:0/3391067779 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0b8c066e40 0x7f0b8c069300 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.408+0000 7f0bb6400640 1 --2- 172.21.15.47:0/3391067779 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bb01614e0 0x7f0bb0156310 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.408+0000 7f0bb6400640 1 --2- 172.21.15.47:0/3391067779 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0bb015f620 0x7f0bb00ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.408+0000 7f0bb6400640 1 --2- 172.21.15.47:0/3391067779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bb00b2500 0x7f0bb00cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.408+0000 7f0bb6400640 1 -- 172.21.15.47:0/3391067779 >> 172.21.15.47:0/3391067779 conn(0x7f0bb00b4fc0 msgr2=0x7f0bb00b1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.408+0000 7f0bb6400640 1 -- 172.21.15.47:0/3391067779 shutdown_connections 2024-02-19T15:25:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.408+0000 7f0bb6400640 1 -- 172.21.15.47:0/3391067779 wait complete. 2024-02-19T15:25:43.438 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:25:43.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.572+0000 7fe6f8600640 1 Processor -- start 2024-02-19T15:25:43.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.572+0000 7fe6f8600640 1 -- start start 2024-02-19T15:25:43.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.576+0000 7fe6f8600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe6f006cf30 0x7fe6f0071960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:43.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.576+0000 7fe6f8600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe6f0071ea0 0x7fe6f00cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:43.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.576+0000 7fe6f8600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f0070c50 con 0x7fe6f0152f00 2024-02-19T15:25:43.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.576+0000 7fe6f8600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f006d7d0 con 0x7fe6f0071ea0 2024-02-19T15:25:43.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.576+0000 7fe6f8600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f006d4f0 con 0x7fe6f006cf30 2024-02-19T15:25:43.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.576+0000 7fe6f6600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe6f0071ea0 0x7fe6f00cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:43.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.576+0000 7fe6f6600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe6f0071ea0 0x7fe6f00cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48230/0 (socket says 172.21.15.47:48230) 2024-02-19T15:25:43.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.576+0000 7fe6f6600640 1 -- 172.21.15.47:0/462230123 learned_addr learned my addr 172.21.15.47:0/462230123 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:43.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.576+0000 7fe6f5c00640 1 --1- 172.21.15.47:0/462230123 >> v1:172.21.15.47:6789/0 conn(0x7fe6f0152f00 0x7fe6f015cd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52844/0 (socket says 172.21.15.47:52844) 2024-02-19T15:25:43.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.576+0000 7fe6f5200640 1 --2- 172.21.15.47:0/462230123 >> v2:172.21.15.116:3300/0 conn(0x7fe6f006cf30 0x7fe6f0071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:43.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.576+0000 7fe6f6600640 1 -- 172.21.15.47:0/462230123 >> v2:172.21.15.116:3300/0 conn(0x7fe6f006cf30 msgr2=0x7fe6f0071960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:43.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.576+0000 7fe6f6600640 1 --2- 172.21.15.47:0/462230123 >> v2:172.21.15.116:3300/0 conn(0x7fe6f006cf30 0x7fe6f0071960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.576+0000 7fe6f6600640 1 -- 172.21.15.47:0/462230123 >> v1:172.21.15.47:6789/0 conn(0x7fe6f0152f00 legacy=0x7fe6f015cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:43.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.576+0000 7fe6f6600640 1 -- 172.21.15.47:0/462230123 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f0070e80 con 0x7fe6f0071ea0 2024-02-19T15:25:43.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.576+0000 7fe6f6600640 1 --2- 172.21.15.47:0/462230123 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6f0071ea0 0x7fe6f00cd9b0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8009e50 tx=0x7fe6e801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76ebc6c386df413d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:43.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6efe00640 1 -- 172.21.15.47:0/462230123 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e80a5700 con 0x7fe6f0071ea0 2024-02-19T15:25:43.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6efe00640 1 -- 172.21.15.47:0/462230123 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e80a5860 con 0x7fe6f0071ea0 2024-02-19T15:25:43.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6efe00640 1 -- 172.21.15.47:0/462230123 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e80a5bf0 con 0x7fe6f0071ea0 2024-02-19T15:25:43.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6f8600640 1 -- 172.21.15.47:0/462230123 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6f0071ea0 msgr2=0x7fe6f00cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:43.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6f8600640 1 --2- 172.21.15.47:0/462230123 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6f0071ea0 0x7fe6f00cd9b0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8009e50 tx=0x7fe6e801c930 comp rx=0 tx=0).stop 2024-02-19T15:25:43.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6f8600640 1 -- 172.21.15.47:0/462230123 shutdown_connections 2024-02-19T15:25:43.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6f8600640 1 --2- 172.21.15.47:0/462230123 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6f0071ea0 0x7fe6f00cd9b0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6f8600640 1 --2- 172.21.15.47:0/462230123 >> v2:172.21.15.116:3300/0 conn(0x7fe6f006cf30 0x7fe6f0071960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6f8600640 1 -- 172.21.15.47:0/462230123 >> 172.21.15.47:0/462230123 conn(0x7fe6f00cfde0 msgr2=0x7fe6f00d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:43.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6f8600640 1 -- 172.21.15.47:0/462230123 shutdown_connections 2024-02-19T15:25:43.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6f8600640 1 -- 172.21.15.47:0/462230123 wait complete. 2024-02-19T15:25:43.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6f8600640 1 Processor -- start 2024-02-19T15:25:43.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6f8600640 1 -- start start 2024-02-19T15:25:43.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6f8600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6f006cf30 0x7fe6f0151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:43.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6f8600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6f0071ea0 0x7fe6f00c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:43.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6f8600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe6f0152f00 0x7fe6f0148550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:43.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6f8600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f00b4c40 con 0x7fe6f0071ea0 2024-02-19T15:25:43.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6f8600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f00b3a10 con 0x7fe6f006cf30 2024-02-19T15:25:43.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6f8600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f00b4680 con 0x7fe6f0152f00 2024-02-19T15:25:43.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6f5200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6f0071ea0 0x7fe6f00c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:43.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6f5200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6f0071ea0 0x7fe6f00c4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48232/0 (socket says 172.21.15.47:48232) 2024-02-19T15:25:43.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6f5200640 1 -- 172.21.15.47:0/2613953482 learned_addr learned my addr 172.21.15.47:0/2613953482 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:43.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.580+0000 7fe6f5c00640 1 --2- 172.21.15.47:0/2613953482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6f006cf30 0x7fe6f0151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:43.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.584+0000 7fe6f6600640 1 --2- 172.21.15.47:0/2613953482 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe6f0152f00 0x7fe6f0148550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:43.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.584+0000 7fe6f5200640 1 -- 172.21.15.47:0/2613953482 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe6f0152f00 msgr2=0x7fe6f0148550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:43.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.584+0000 7fe6f5200640 1 --2- 172.21.15.47:0/2613953482 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe6f0152f00 0x7fe6f0148550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.584+0000 7fe6f5200640 1 -- 172.21.15.47:0/2613953482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6f006cf30 msgr2=0x7fe6f0151800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:43.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.584+0000 7fe6f5200640 1 --2- 172.21.15.47:0/2613953482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6f006cf30 0x7fe6f0151800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.584+0000 7fe6f5200640 1 -- 172.21.15.47:0/2613953482 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e8009b30 con 0x7fe6f0071ea0 2024-02-19T15:25:43.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.584+0000 7fe6f5200640 1 --2- 172.21.15.47:0/2613953482 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6f0071ea0 0x7fe6f00c4ce0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc00e870 tx=0x7fe6dc00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:43.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.584+0000 7fe6ed600640 1 -- 172.21.15.47:0/2613953482 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc009510 con 0x7fe6f0071ea0 2024-02-19T15:25:43.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.584+0000 7fe6ed600640 1 -- 172.21.15.47:0/2613953482 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6dc00eec0 con 0x7fe6f0071ea0 2024-02-19T15:25:43.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.584+0000 7fe6f8600640 1 -- 172.21.15.47:0/2613953482 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f00b4960 con 0x7fe6f0071ea0 2024-02-19T15:25:43.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.584+0000 7fe6ed600640 1 -- 172.21.15.47:0/2613953482 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc00cb40 con 0x7fe6f0071ea0 2024-02-19T15:25:43.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.584+0000 7fe6f8600640 1 -- 172.21.15.47:0/2613953482 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f014a5b0 con 0x7fe6f0071ea0 2024-02-19T15:25:43.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.584+0000 7fe6f8600640 1 -- 172.21.15.47:0/2613953482 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f0147e90 con 0x7fe6f0071ea0 2024-02-19T15:25:43.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.584+0000 7fe6ed600640 1 -- 172.21.15.47:0/2613953482 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe6dc00cca0 con 0x7fe6f0071ea0 2024-02-19T15:25:43.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.588+0000 7fe6ed600640 1 --2- 172.21.15.47:0/2613953482 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe6d0066e40 0x7fe6d0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:43.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.588+0000 7fe6ed600640 1 -- 172.21.15.47:0/2613953482 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe6dc084ab0 con 0x7fe6f0071ea0 2024-02-19T15:25:43.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.588+0000 7fe6f5c00640 1 --2- 172.21.15.47:0/2613953482 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe6d0066e40 0x7fe6d0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:43.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.588+0000 7fe6f5c00640 1 --2- 172.21.15.47:0/2613953482 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe6d0066e40 0x7fe6d0069300 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0152370 tx=0x7fe6e4007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:43.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.588+0000 7fe6ed600640 1 -- 172.21.15.47:0/2613953482 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe6f0147e90 con 0x7fe6f0071ea0 2024-02-19T15:25:43.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.764+0000 7fe6f8600640 1 -- 172.21.15.47:0/2613953482 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6f0147e90 con 0x7fe6f0071ea0 2024-02-19T15:25:43.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.764+0000 7fe6ed600640 1 -- 172.21.15.47:0/2613953482 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16680 (secure 0 0 0) 0x7fe6f0147e90 con 0x7fe6f0071ea0 2024-02-19T15:25:43.769 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:43.769 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":69,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:25:10.552537+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:25:06.759758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-02-19T15:25:07.949435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:25:43.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.764+0000 7fe6f8600640 1 -- 172.21.15.47:0/2613953482 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe6d0066e40 msgr2=0x7fe6d0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:43.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.764+0000 7fe6f8600640 1 --2- 172.21.15.47:0/2613953482 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe6d0066e40 0x7fe6d0069300 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0152370 tx=0x7fe6e4007e90 comp rx=0 tx=0).stop 2024-02-19T15:25:43.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.764+0000 7fe6f8600640 1 -- 172.21.15.47:0/2613953482 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6f0071ea0 msgr2=0x7fe6f00c4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:43.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.764+0000 7fe6f8600640 1 --2- 172.21.15.47:0/2613953482 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6f0071ea0 0x7fe6f00c4ce0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc00e870 tx=0x7fe6dc00ec70 comp rx=0 tx=0).stop 2024-02-19T15:25:43.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.764+0000 7fe6f8600640 1 -- 172.21.15.47:0/2613953482 shutdown_connections 2024-02-19T15:25:43.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.764+0000 7fe6f8600640 1 --2- 172.21.15.47:0/2613953482 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe6d0066e40 0x7fe6d0069300 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.764+0000 7fe6f8600640 1 --2- 172.21.15.47:0/2613953482 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe6f0152f00 0x7fe6f0148550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.764+0000 7fe6f8600640 1 --2- 172.21.15.47:0/2613953482 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6f0071ea0 0x7fe6f00c4ce0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.764+0000 7fe6f8600640 1 --2- 172.21.15.47:0/2613953482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6f006cf30 0x7fe6f0151800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.764+0000 7fe6f8600640 1 -- 172.21.15.47:0/2613953482 >> 172.21.15.47:0/2613953482 conn(0x7fe6f00cfde0 msgr2=0x7fe6f00b02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:43.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.768+0000 7fe6f8600640 1 -- 172.21.15.47:0/2613953482 shutdown_connections 2024-02-19T15:25:43.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.768+0000 7fe6f8600640 1 -- 172.21.15.47:0/2613953482 wait complete. 2024-02-19T15:25:43.790 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:25:06.759758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:25:07.949435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:25:43.790 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:25:43.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.928+0000 7fac0f000640 1 Processor -- start 2024-02-19T15:25:43.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.928+0000 7fac0f000640 1 -- start start 2024-02-19T15:25:43.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.932+0000 7fac0f000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac080ce3b0 0x7fac080ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:43.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.932+0000 7fac0f000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fac080d2650 0x7fac080ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:43.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.932+0000 7fac0f000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac080cc620 con 0x7fac0815e280 2024-02-19T15:25:43.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.932+0000 7fac0f000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac080d12e0 con 0x7fac080d2650 2024-02-19T15:25:43.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.932+0000 7fac0f000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac080d1940 con 0x7fac080ce3b0 2024-02-19T15:25:43.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.932+0000 7fabffe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fac080d2650 0x7fac080ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:43.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.932+0000 7fabffe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fac080d2650 0x7fac080ccab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48244/0 (socket says 172.21.15.47:48244) 2024-02-19T15:25:43.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.932+0000 7fac0d000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fac0815e280 0x7fac080ccff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52860/0 (socket says 172.21.15.47:52860) 2024-02-19T15:25:43.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.932+0000 7fabffe00640 1 -- 172.21.15.47:0/2551852066 learned_addr learned my addr 172.21.15.47:0/2551852066 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:43.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.932+0000 7fac07e00640 1 --2- 172.21.15.47:0/2551852066 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac080ce3b0 0x7fac080ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:43.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.932+0000 7fabffe00640 1 -- 172.21.15.47:0/2551852066 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac080ce3b0 msgr2=0x7fac080ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:43.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.932+0000 7fabffe00640 1 --2- 172.21.15.47:0/2551852066 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac080ce3b0 0x7fac080ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.932+0000 7fabffe00640 1 -- 172.21.15.47:0/2551852066 >> v1:172.21.15.47:6789/0 conn(0x7fac0815e280 legacy=0x7fac080ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:43.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.932+0000 7fabffe00640 1 -- 172.21.15.47:0/2551852066 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac080cecd0 con 0x7fac080d2650 2024-02-19T15:25:43.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.932+0000 7fac07400640 1 -- 172.21.15.47:0/2551852066 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2875003335 0 0) 0x7fac080cc620 con 0x7fac0815e280 2024-02-19T15:25:43.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.936+0000 7fabffe00640 1 --2- 172.21.15.47:0/2551852066 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac080d2650 0x7fac080ccab0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fabf0009870 tx=0x7fabf001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c10016a0d5b6e24 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:43.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.936+0000 7fac07400640 1 -- 172.21.15.47:0/2551852066 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf00ab600 con 0x7fac080d2650 2024-02-19T15:25:43.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.936+0000 7fac07400640 1 -- 172.21.15.47:0/2551852066 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf00a7050 con 0x7fac080d2650 2024-02-19T15:25:43.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.936+0000 7fac07400640 1 -- 172.21.15.47:0/2551852066 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf00ab980 con 0x7fac080d2650 2024-02-19T15:25:43.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.936+0000 7fac0f000640 1 -- 172.21.15.47:0/2551852066 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac080d2650 msgr2=0x7fac080ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:43.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.936+0000 7fac0f000640 1 --2- 172.21.15.47:0/2551852066 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac080d2650 0x7fac080ccab0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fabf0009870 tx=0x7fabf001c440 comp rx=0 tx=0).stop 2024-02-19T15:25:43.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.936+0000 7fac0f000640 1 -- 172.21.15.47:0/2551852066 shutdown_connections 2024-02-19T15:25:43.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.936+0000 7fac0f000640 1 --2- 172.21.15.47:0/2551852066 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac080d2650 0x7fac080ccab0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.936+0000 7fac0f000640 1 --2- 172.21.15.47:0/2551852066 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac080ce3b0 0x7fac080ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.936+0000 7fac0f000640 1 -- 172.21.15.47:0/2551852066 >> 172.21.15.47:0/2551852066 conn(0x7fac080d1b30 msgr2=0x7fac080d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:43.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.936+0000 7fac0f000640 1 -- 172.21.15.47:0/2551852066 shutdown_connections 2024-02-19T15:25:43.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.936+0000 7fac0f000640 1 -- 172.21.15.47:0/2551852066 wait complete. 2024-02-19T15:25:43.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.936+0000 7fac0f000640 1 Processor -- start 2024-02-19T15:25:43.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.936+0000 7fac0f000640 1 -- start start 2024-02-19T15:25:43.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac0f000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac080ce3b0 0x7fac08160040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:43.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac0f000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac080d2650 0x7fac0814e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:43.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac0f000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac0815e280 0x7fac081570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:43.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac0f000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0815f070 con 0x7fac0815e280 2024-02-19T15:25:43.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac0f000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0815fbb0 con 0x7fac080ce3b0 2024-02-19T15:25:43.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac0f000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac08160580 con 0x7fac080d2650 2024-02-19T15:25:43.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac0d000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac0815e280 0x7fac081570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:43.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac0d000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac0815e280 0x7fac081570b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48250/0 (socket says 172.21.15.47:48250) 2024-02-19T15:25:43.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac0d000640 1 -- 172.21.15.47:0/4267102175 learned_addr learned my addr 172.21.15.47:0/4267102175 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:43.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac07e00640 1 --2- 172.21.15.47:0/4267102175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac080ce3b0 0x7fac08160040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:43.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fabffe00640 1 --2- 172.21.15.47:0/4267102175 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac080d2650 0x7fac0814e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:43.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac0d000640 1 -- 172.21.15.47:0/4267102175 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac080d2650 msgr2=0x7fac0814e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:43.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac0d000640 1 --2- 172.21.15.47:0/4267102175 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac080d2650 0x7fac0814e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac0d000640 1 -- 172.21.15.47:0/4267102175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac080ce3b0 msgr2=0x7fac08160040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:43.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac0d000640 1 --2- 172.21.15.47:0/4267102175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac080ce3b0 0x7fac08160040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:43.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac0d000640 1 -- 172.21.15.47:0/4267102175 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf00094d0 con 0x7fac0815e280 2024-02-19T15:25:43.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac0d000640 1 --2- 172.21.15.47:0/4267102175 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac0815e280 0x7fac081570b0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fac0000c700 tx=0x7fac0000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:43.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac04c00640 1 -- 172.21.15.47:0/4267102175 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0000ce10 con 0x7fac0815e280 2024-02-19T15:25:43.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac04c00640 1 -- 172.21.15.47:0/4267102175 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0000a640 con 0x7fac0815e280 2024-02-19T15:25:43.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac04c00640 1 -- 172.21.15.47:0/4267102175 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0000aa70 con 0x7fac0815e280 2024-02-19T15:25:43.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac0f000640 1 -- 172.21.15.47:0/4267102175 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac0815f280 con 0x7fac0815e280 2024-02-19T15:25:43.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac0f000640 1 -- 172.21.15.47:0/4267102175 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac0815f840 con 0x7fac0815e280 2024-02-19T15:25:43.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac0f000640 1 -- 172.21.15.47:0/4267102175 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac08075360 con 0x7fac0815e280 2024-02-19T15:25:43.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.940+0000 7fac04c00640 1 -- 172.21.15.47:0/4267102175 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fac0000abd0 con 0x7fac0815e280 2024-02-19T15:25:43.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.944+0000 7fac04c00640 1 --2- 172.21.15.47:0/4267102175 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fabe4066e40 0x7fabe4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:43.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.944+0000 7fac04c00640 1 -- 172.21.15.47:0/4267102175 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac00084120 con 0x7fac0815e280 2024-02-19T15:25:43.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.944+0000 7fac07e00640 1 --2- 172.21.15.47:0/4267102175 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fabe4066e40 0x7fabe4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:43.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.944+0000 7fac07e00640 1 --2- 172.21.15.47:0/4267102175 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fabe4066e40 0x7fabe4069300 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fabf8009ac0 tx=0x7fabf8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:43.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:43.948+0000 7fac04c00640 1 -- 172.21.15.47:0/4267102175 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fac00051050 con 0x7fac0815e280 2024-02-19T15:25:44.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.112+0000 7fac0f000640 1 -- 172.21.15.47:0/4267102175 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fac0815fcf0 con 0x7fac0815e280 2024-02-19T15:25:44.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.112+0000 7fac04c00640 1 -- 172.21.15.47:0/4267102175 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7fac00054710 con 0x7fac0815e280 2024-02-19T15:25:44.117 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:25:44.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.116+0000 7fac0f000640 1 -- 172.21.15.47:0/4267102175 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fabe4066e40 msgr2=0x7fabe4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:44.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.116+0000 7fac0f000640 1 --2- 172.21.15.47:0/4267102175 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fabe4066e40 0x7fabe4069300 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fabf8009ac0 tx=0x7fabf8009190 comp rx=0 tx=0).stop 2024-02-19T15:25:44.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.116+0000 7fac0f000640 1 -- 172.21.15.47:0/4267102175 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac0815e280 msgr2=0x7fac081570b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:44.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.116+0000 7fac0f000640 1 --2- 172.21.15.47:0/4267102175 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac0815e280 0x7fac081570b0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fac0000c700 tx=0x7fac0000cbc0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.116+0000 7fac0f000640 1 -- 172.21.15.47:0/4267102175 shutdown_connections 2024-02-19T15:25:44.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.116+0000 7fac0f000640 1 --2- 172.21.15.47:0/4267102175 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fabe4066e40 0x7fabe4069300 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.116+0000 7fac0f000640 1 --2- 172.21.15.47:0/4267102175 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac0815e280 0x7fac081570b0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.116+0000 7fac0f000640 1 --2- 172.21.15.47:0/4267102175 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac080d2650 0x7fac0814e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.116+0000 7fac0f000640 1 --2- 172.21.15.47:0/4267102175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac080ce3b0 0x7fac08160040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.116+0000 7fac0f000640 1 -- 172.21.15.47:0/4267102175 >> 172.21.15.47:0/4267102175 conn(0x7fac080d1b30 msgr2=0x7fac080c7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:44.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.116+0000 7fac0f000640 1 -- 172.21.15.47:0/4267102175 shutdown_connections 2024-02-19T15:25:44.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.116+0000 7fac0f000640 1 -- 172.21.15.47:0/4267102175 wait complete. 2024-02-19T15:25:44.139 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:25:44.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.280+0000 7fe9af600640 1 Processor -- start 2024-02-19T15:25:44.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.280+0000 7fe9af600640 1 -- start start 2024-02-19T15:25:44.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.280+0000 7fe9af600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe9a806e1e0 0x7fe9a80d1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:44.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.280+0000 7fe9af600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a8071a70 con 0x7fe9a806f220 2024-02-19T15:25:44.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.280+0000 7fe9af600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a806f690 con 0x7fe9a815e160 2024-02-19T15:25:44.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.280+0000 7fe9af600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9a80ced30 con 0x7fe9a806e1e0 2024-02-19T15:25:44.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.280+0000 7fe9a7e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe9a806e1e0 0x7fe9a80d1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:44.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.280+0000 7fe9a7e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe9a806e1e0 0x7fe9a80d1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48264/0 (socket says 172.21.15.47:48264) 2024-02-19T15:25:44.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.280+0000 7fe9a7e00640 1 -- 172.21.15.47:0/2671700018 learned_addr learned my addr 172.21.15.47:0/2671700018 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:44.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.280+0000 7fe9ad600640 1 --1- 172.21.15.47:0/2671700018 >> v1:172.21.15.47:6789/0 conn(0x7fe9a806f220 0x7fe9a815e540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52862/0 (socket says 172.21.15.47:52862) 2024-02-19T15:25:44.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.280+0000 7fe9a7e00640 1 -- 172.21.15.47:0/2671700018 >> v1:172.21.15.116:6789/0 conn(0x7fe9a815e160 legacy=0x7fe9a806dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:44.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.280+0000 7fe9a7e00640 1 -- 172.21.15.47:0/2671700018 >> v1:172.21.15.47:6789/0 conn(0x7fe9a806f220 legacy=0x7fe9a815e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:44.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.280+0000 7fe9a7e00640 1 -- 172.21.15.47:0/2671700018 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a8162a00 con 0x7fe9a806e1e0 2024-02-19T15:25:44.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.280+0000 7fe9a7400640 1 -- 172.21.15.47:0/2671700018 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2613832416 0 0) 0x7fe9a8071a70 con 0x7fe9a806f220 2024-02-19T15:25:44.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.284+0000 7fe9a7e00640 1 --2- 172.21.15.47:0/2671700018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9a806e1e0 0x7fe9a80d1820 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fe994009870 tx=0x7fe99401c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa8680cb3eccad3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:44.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.284+0000 7fe9a7400640 1 -- 172.21.15.47:0/2671700018 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9940a95f0 con 0x7fe9a806e1e0 2024-02-19T15:25:44.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.284+0000 7fe9a7400640 1 -- 172.21.15.47:0/2671700018 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9940a9750 con 0x7fe9a806e1e0 2024-02-19T15:25:44.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.284+0000 7fe9a7400640 1 -- 172.21.15.47:0/2671700018 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9940a9ae0 con 0x7fe9a806e1e0 2024-02-19T15:25:44.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.284+0000 7fe9af600640 1 -- 172.21.15.47:0/2671700018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9a806e1e0 msgr2=0x7fe9a80d1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:44.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.284+0000 7fe9af600640 1 --2- 172.21.15.47:0/2671700018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9a806e1e0 0x7fe9a80d1820 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fe994009870 tx=0x7fe99401c530 comp rx=0 tx=0).stop 2024-02-19T15:25:44.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.284+0000 7fe9af600640 1 -- 172.21.15.47:0/2671700018 shutdown_connections 2024-02-19T15:25:44.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.284+0000 7fe9af600640 1 --2- 172.21.15.47:0/2671700018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9a806e1e0 0x7fe9a80d1820 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.284+0000 7fe9af600640 1 -- 172.21.15.47:0/2671700018 >> 172.21.15.47:0/2671700018 conn(0x7fe9a80d0d30 msgr2=0x7fe9a80d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:44.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.284+0000 7fe9af600640 1 -- 172.21.15.47:0/2671700018 shutdown_connections 2024-02-19T15:25:44.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.284+0000 7fe9af600640 1 -- 172.21.15.47:0/2671700018 wait complete. 2024-02-19T15:25:44.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.284+0000 7fe9af600640 1 Processor -- start 2024-02-19T15:25:44.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.284+0000 7fe9af600640 1 -- start start 2024-02-19T15:25:44.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9af600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9a806e1e0 0x7fe9a815cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:44.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9af600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9a806f220 0x7fe9a81598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:44.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9af600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9a815e160 0x7fe9a8157ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:44.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9af600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a8158820 con 0x7fe9a806f220 2024-02-19T15:25:44.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9af600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a8158b00 con 0x7fe9a806e1e0 2024-02-19T15:25:44.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9af600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a8158de0 con 0x7fe9a815e160 2024-02-19T15:25:44.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9a7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9a806f220 0x7fe9a81598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:44.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9a7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9a806f220 0x7fe9a81598c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48274/0 (socket says 172.21.15.47:48274) 2024-02-19T15:25:44.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9a7e00640 1 -- 172.21.15.47:0/3431050797 learned_addr learned my addr 172.21.15.47:0/3431050797 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:44.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9acc00640 1 --2- 172.21.15.47:0/3431050797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9a806e1e0 0x7fe9a815cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:44.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9ad600640 1 --2- 172.21.15.47:0/3431050797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9a815e160 0x7fe9a8157ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:44.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9a7e00640 1 -- 172.21.15.47:0/3431050797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9a815e160 msgr2=0x7fe9a8157ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:44.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9a7e00640 1 --2- 172.21.15.47:0/3431050797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9a815e160 0x7fe9a8157ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9a7e00640 1 -- 172.21.15.47:0/3431050797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9a806e1e0 msgr2=0x7fe9a815cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:44.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9a7e00640 1 --2- 172.21.15.47:0/3431050797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9a806e1e0 0x7fe9a815cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9a7e00640 1 -- 172.21.15.47:0/3431050797 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9940094d0 con 0x7fe9a806f220 2024-02-19T15:25:44.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9acc00640 1 --2- 172.21.15.47:0/3431050797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9a806e1e0 0x7fe9a815cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:44.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9a7e00640 1 --2- 172.21.15.47:0/3431050797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9a806f220 0x7fe9a81598c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fe9940ab040 tx=0x7fe99401ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:44.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9a4c00640 1 -- 172.21.15.47:0/3431050797 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9940b1800 con 0x7fe9a806f220 2024-02-19T15:25:44.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9a4c00640 1 -- 172.21.15.47:0/3431050797 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9940b1980 con 0x7fe9a806f220 2024-02-19T15:25:44.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9a4c00640 1 -- 172.21.15.47:0/3431050797 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9940b1d30 con 0x7fe9a806f220 2024-02-19T15:25:44.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9af600640 1 -- 172.21.15.47:0/3431050797 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a81590c0 con 0x7fe9a806f220 2024-02-19T15:25:44.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9af600640 1 -- 172.21.15.47:0/3431050797 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a8158010 con 0x7fe9a806f220 2024-02-19T15:25:44.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.288+0000 7fe9af600640 1 -- 172.21.15.47:0/3431050797 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9a8075360 con 0x7fe9a806f220 2024-02-19T15:25:44.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.292+0000 7fe9a4c00640 1 -- 172.21.15.47:0/3431050797 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe9940a8350 con 0x7fe9a806f220 2024-02-19T15:25:44.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.292+0000 7fe9a4c00640 1 --2- 172.21.15.47:0/3431050797 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe988066e40 0x7fe988069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:44.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.292+0000 7fe9a4c00640 1 -- 172.21.15.47:0/3431050797 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe99411f4c0 con 0x7fe9a806f220 2024-02-19T15:25:44.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.292+0000 7fe9acc00640 1 --2- 172.21.15.47:0/3431050797 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe988066e40 0x7fe988069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:44.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.292+0000 7fe9acc00640 1 --2- 172.21.15.47:0/3431050797 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe988066e40 0x7fe988069300 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe99c000f70 tx=0x7fe99c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:44.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.296+0000 7fe9a4c00640 1 -- 172.21.15.47:0/3431050797 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe9940ec2a0 con 0x7fe9a806f220 2024-02-19T15:25:44.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.456+0000 7fe9af600640 1 -- 172.21.15.47:0/3431050797 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe9a8159570 con 0x7fe9a806f220 2024-02-19T15:25:44.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.456+0000 7fe9a4c00640 1 -- 172.21.15.47:0/3431050797 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe9940ef960 con 0x7fe9a806f220 2024-02-19T15:25:44.461 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:25:44.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.460+0000 7fe9af600640 1 -- 172.21.15.47:0/3431050797 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe988066e40 msgr2=0x7fe988069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:44.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.460+0000 7fe9af600640 1 --2- 172.21.15.47:0/3431050797 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe988066e40 0x7fe988069300 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe99c000f70 tx=0x7fe99c009040 comp rx=0 tx=0).stop 2024-02-19T15:25:44.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.460+0000 7fe9af600640 1 -- 172.21.15.47:0/3431050797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9a806f220 msgr2=0x7fe9a81598c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:44.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.460+0000 7fe9af600640 1 --2- 172.21.15.47:0/3431050797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9a806f220 0x7fe9a81598c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fe9940ab040 tx=0x7fe99401ad70 comp rx=0 tx=0).stop 2024-02-19T15:25:44.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.460+0000 7fe9af600640 1 -- 172.21.15.47:0/3431050797 shutdown_connections 2024-02-19T15:25:44.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.460+0000 7fe9af600640 1 --2- 172.21.15.47:0/3431050797 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe988066e40 0x7fe988069300 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.460+0000 7fe9af600640 1 --2- 172.21.15.47:0/3431050797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9a815e160 0x7fe9a8157ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.460+0000 7fe9af600640 1 --2- 172.21.15.47:0/3431050797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9a806f220 0x7fe9a81598c0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.460+0000 7fe9af600640 1 --2- 172.21.15.47:0/3431050797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9a806e1e0 0x7fe9a815cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.460+0000 7fe9af600640 1 -- 172.21.15.47:0/3431050797 >> 172.21.15.47:0/3431050797 conn(0x7fe9a80d0d30 msgr2=0x7fe9a8161510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:44.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.460+0000 7fe9af600640 1 -- 172.21.15.47:0/3431050797 shutdown_connections 2024-02-19T15:25:44.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.460+0000 7fe9af600640 1 -- 172.21.15.47:0/3431050797 wait complete. 2024-02-19T15:25:44.491 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:25:44.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.632+0000 7f4c8f800640 1 Processor -- start 2024-02-19T15:25:44.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.632+0000 7f4c8f800640 1 -- start start 2024-02-19T15:25:44.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.632+0000 7f4c8f800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4c880c7e80 0x7f4c880d0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:44.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.632+0000 7f4c8f800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c880cfbf0 0x7f4c880d0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:44.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.632+0000 7f4c8f800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c880cd750 con 0x7f4c880c8260 2024-02-19T15:25:44.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.632+0000 7f4c8f800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c880ceee0 con 0x7f4c880c7e80 2024-02-19T15:25:44.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.632+0000 7f4c8f800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c880cdc60 con 0x7f4c880cfbf0 2024-02-19T15:25:44.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.632+0000 7f4c8ce00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4c880c7e80 0x7f4c880d0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:44.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.632+0000 7f4c87e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f4c880c8260 0x7f4c880c5680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52878/0 (socket says 172.21.15.47:52878) 2024-02-19T15:25:44.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.632+0000 7f4c8ce00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4c880c7e80 0x7f4c880d0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48276/0 (socket says 172.21.15.47:48276) 2024-02-19T15:25:44.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.632+0000 7f4c87e00640 1 -- 172.21.15.47:0/829797868 learned_addr learned my addr 172.21.15.47:0/829797868 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:44.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.632+0000 7f4c8d800640 1 --2- 172.21.15.47:0/829797868 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c880cfbf0 0x7f4c880d0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:44.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.632+0000 7f4c8ce00640 1 -- 172.21.15.47:0/829797868 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c880cfbf0 msgr2=0x7f4c880d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:44.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.632+0000 7f4c8ce00640 1 --2- 172.21.15.47:0/829797868 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c880cfbf0 0x7f4c880d0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.632+0000 7f4c8ce00640 1 -- 172.21.15.47:0/829797868 >> v1:172.21.15.47:6789/0 conn(0x7f4c880c8260 legacy=0x7f4c880c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:44.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.632+0000 7f4c8ce00640 1 -- 172.21.15.47:0/829797868 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c880d2a40 con 0x7f4c880c7e80 2024-02-19T15:25:44.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.632+0000 7f4c87400640 1 -- 172.21.15.47:0/829797868 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544633144 0 0) 0x7f4c880cd750 con 0x7f4c880c8260 2024-02-19T15:25:44.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.636+0000 7f4c8ce00640 1 --2- 172.21.15.47:0/829797868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c880c7e80 0x7f4c880d0110 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c0097c0 tx=0x7f4c7c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a5d73aa3135748c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:44.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.636+0000 7f4c87400640 1 -- 172.21.15.47:0/829797868 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c0ab670 con 0x7f4c880c7e80 2024-02-19T15:25:44.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.636+0000 7f4c87400640 1 -- 172.21.15.47:0/829797868 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7c0ab7d0 con 0x7f4c880c7e80 2024-02-19T15:25:44.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.636+0000 7f4c87400640 1 -- 172.21.15.47:0/829797868 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c0abb60 con 0x7f4c880c7e80 2024-02-19T15:25:44.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.640+0000 7f4c8f800640 1 -- 172.21.15.47:0/829797868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c880c7e80 msgr2=0x7f4c880d0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:44.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.640+0000 7f4c8f800640 1 --2- 172.21.15.47:0/829797868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c880c7e80 0x7f4c880d0110 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c0097c0 tx=0x7f4c7c01c470 comp rx=0 tx=0).stop 2024-02-19T15:25:44.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.640+0000 7f4c8f800640 1 -- 172.21.15.47:0/829797868 shutdown_connections 2024-02-19T15:25:44.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.640+0000 7f4c8f800640 1 --2- 172.21.15.47:0/829797868 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c880cfbf0 0x7f4c880d0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.640+0000 7f4c8f800640 1 --2- 172.21.15.47:0/829797868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c880c7e80 0x7f4c880d0110 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.640+0000 7f4c8f800640 1 -- 172.21.15.47:0/829797868 >> 172.21.15.47:0/829797868 conn(0x7f4c880d1b30 msgr2=0x7f4c880d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:44.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.640+0000 7f4c8f800640 1 -- 172.21.15.47:0/829797868 shutdown_connections 2024-02-19T15:25:44.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.640+0000 7f4c8f800640 1 -- 172.21.15.47:0/829797868 wait complete. 2024-02-19T15:25:44.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.640+0000 7f4c8f800640 1 Processor -- start 2024-02-19T15:25:44.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.640+0000 7f4c8f800640 1 -- start start 2024-02-19T15:25:44.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c8f800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c880c7e80 0x7f4c88155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:44.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c8f800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c880c8260 0x7f4c881556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:44.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c8f800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c880cfbf0 0x7f4c88159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:44.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c8f800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c88144660 con 0x7f4c880cfbf0 2024-02-19T15:25:44.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c8f800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c88157c10 con 0x7f4c880c8260 2024-02-19T15:25:44.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c8f800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c88157ef0 con 0x7f4c880c7e80 2024-02-19T15:25:44.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c8d800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c880cfbf0 0x7f4c88159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:44.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c8d800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c880cfbf0 0x7f4c88159530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48292/0 (socket says 172.21.15.47:48292) 2024-02-19T15:25:44.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c8d800640 1 -- 172.21.15.47:0/1744309218 learned_addr learned my addr 172.21.15.47:0/1744309218 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:44.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c87e00640 1 --2- 172.21.15.47:0/1744309218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c880c8260 0x7f4c881556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:44.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c8ce00640 1 --2- 172.21.15.47:0/1744309218 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c880c7e80 0x7f4c88155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:44.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c8d800640 1 -- 172.21.15.47:0/1744309218 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c880c7e80 msgr2=0x7f4c88155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:44.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c8d800640 1 --2- 172.21.15.47:0/1744309218 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c880c7e80 0x7f4c88155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c8d800640 1 -- 172.21.15.47:0/1744309218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c880c8260 msgr2=0x7f4c881556d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:44.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c8d800640 1 --2- 172.21.15.47:0/1744309218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c880c8260 0x7f4c881556d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c8d800640 1 -- 172.21.15.47:0/1744309218 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7c009420 con 0x7f4c880cfbf0 2024-02-19T15:25:44.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c8d800640 1 --2- 172.21.15.47:0/1744309218 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c880cfbf0 0x7f4c88159530 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f4c8000efc0 tx=0x7f4c8000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:44.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c84c00640 1 -- 172.21.15.47:0/1744309218 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c80014480 con 0x7f4c880cfbf0 2024-02-19T15:25:44.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c84c00640 1 -- 172.21.15.47:0/1744309218 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c8000f040 con 0x7f4c880cfbf0 2024-02-19T15:25:44.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c8f800640 1 -- 172.21.15.47:0/1744309218 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c881581d0 con 0x7f4c880cfbf0 2024-02-19T15:25:44.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c84c00640 1 -- 172.21.15.47:0/1744309218 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c80014820 con 0x7f4c880cfbf0 2024-02-19T15:25:44.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c8f800640 1 -- 172.21.15.47:0/1744309218 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c88152830 con 0x7f4c880cfbf0 2024-02-19T15:25:44.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.644+0000 7f4c8f800640 1 -- 172.21.15.47:0/1744309218 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c880d1180 con 0x7f4c880cfbf0 2024-02-19T15:25:44.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.648+0000 7f4c84c00640 1 -- 172.21.15.47:0/1744309218 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4c80014980 con 0x7f4c880cfbf0 2024-02-19T15:25:44.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.648+0000 7f4c84c00640 1 --2- 172.21.15.47:0/1744309218 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4c68066e70 0x7f4c68069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:44.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.648+0000 7f4c84c00640 1 -- 172.21.15.47:0/1744309218 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c80085950 con 0x7f4c880cfbf0 2024-02-19T15:25:44.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.648+0000 7f4c8ce00640 1 --2- 172.21.15.47:0/1744309218 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4c68066e70 0x7f4c68069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:44.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.648+0000 7f4c8ce00640 1 --2- 172.21.15.47:0/1744309218 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4c68066e70 0x7f4c68069330 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c0af000 tx=0x7f4c7c0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:44.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.652+0000 7f4c84c00640 1 -- 172.21.15.47:0/1744309218 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f4c800526a0 con 0x7f4c880cfbf0 2024-02-19T15:25:44.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.812+0000 7f4c8f800640 1 -- 172.21.15.47:0/1744309218 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4c880d09f0 con 0x7f4c880cfbf0 2024-02-19T15:25:44.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.812+0000 7f4c84c00640 1 -- 172.21.15.47:0/1744309218 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4c800180a0 con 0x7f4c880cfbf0 2024-02-19T15:25:44.816 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:25:44.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.816+0000 7f4c8f800640 1 -- 172.21.15.47:0/1744309218 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4c68066e70 msgr2=0x7f4c68069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:44.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.816+0000 7f4c8f800640 1 --2- 172.21.15.47:0/1744309218 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4c68066e70 0x7f4c68069330 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c0af000 tx=0x7f4c7c0a7000 comp rx=0 tx=0).stop 2024-02-19T15:25:44.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.816+0000 7f4c8f800640 1 -- 172.21.15.47:0/1744309218 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c880cfbf0 msgr2=0x7f4c88159530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:44.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.816+0000 7f4c8f800640 1 --2- 172.21.15.47:0/1744309218 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c880cfbf0 0x7f4c88159530 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f4c8000efc0 tx=0x7f4c8000c490 comp rx=0 tx=0).stop 2024-02-19T15:25:44.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.816+0000 7f4c8f800640 1 -- 172.21.15.47:0/1744309218 shutdown_connections 2024-02-19T15:25:44.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.816+0000 7f4c8f800640 1 --2- 172.21.15.47:0/1744309218 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4c68066e70 0x7f4c68069330 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.816+0000 7f4c8f800640 1 --2- 172.21.15.47:0/1744309218 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c880cfbf0 0x7f4c88159530 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.816+0000 7f4c8f800640 1 --2- 172.21.15.47:0/1744309218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c880c8260 0x7f4c881556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.816+0000 7f4c8f800640 1 --2- 172.21.15.47:0/1744309218 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c880c7e80 0x7f4c88155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.816+0000 7f4c8f800640 1 -- 172.21.15.47:0/1744309218 >> 172.21.15.47:0/1744309218 conn(0x7f4c880d1b30 msgr2=0x7f4c880c5640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:44.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.816+0000 7f4c8f800640 1 -- 172.21.15.47:0/1744309218 shutdown_connections 2024-02-19T15:25:44.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.816+0000 7f4c8f800640 1 -- 172.21.15.47:0/1744309218 wait complete. 2024-02-19T15:25:44.847 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:25:44.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.980+0000 7fa937600640 1 Processor -- start 2024-02-19T15:25:44.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.980+0000 7fa937600640 1 -- start start 2024-02-19T15:25:44.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.984+0000 7fa937600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa93015f590 0x7fa9300d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:44.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.984+0000 7fa937600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa9300b0a80 0x7fa9300b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:44.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.984+0000 7fa937600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9300b0280 con 0x7fa9300b1a10 2024-02-19T15:25:44.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.984+0000 7fa937600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9300cc530 con 0x7fa93015f590 2024-02-19T15:25:44.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.984+0000 7fa937600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9300b1e20 con 0x7fa9300b0a80 2024-02-19T15:25:44.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.984+0000 7fa934c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa93015f590 0x7fa9300d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:44.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.984+0000 7fa934c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa93015f590 0x7fa9300d2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48294/0 (socket says 172.21.15.47:48294) 2024-02-19T15:25:44.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.984+0000 7fa934c00640 1 -- 172.21.15.47:0/1875622814 learned_addr learned my addr 172.21.15.47:0/1875622814 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:44.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.984+0000 7fa92fe00640 1 --2- 172.21.15.47:0/1875622814 >> v2:172.21.15.116:3300/0 conn(0x7fa9300b0a80 0x7fa9300b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:44.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.984+0000 7fa934c00640 1 -- 172.21.15.47:0/1875622814 >> v2:172.21.15.116:3300/0 conn(0x7fa9300b0a80 msgr2=0x7fa9300b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:44.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.984+0000 7fa934c00640 1 --2- 172.21.15.47:0/1875622814 >> v2:172.21.15.116:3300/0 conn(0x7fa9300b0a80 0x7fa9300b14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.984+0000 7fa934c00640 1 -- 172.21.15.47:0/1875622814 >> v1:172.21.15.116:6789/0 conn(0x7fa9300b1a10 legacy=0x7fa9300ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:44.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.984+0000 7fa934c00640 1 -- 172.21.15.47:0/1875622814 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9300b04d0 con 0x7fa93015f590 2024-02-19T15:25:44.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.988+0000 7fa934c00640 1 --2- 172.21.15.47:0/1875622814 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa93015f590 0x7fa9300d2690 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fa930072f00 tx=0x7fa92401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd3f47a4b61c752 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:44.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.988+0000 7fa92f400640 1 -- 172.21.15.47:0/1875622814 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9240aa650 con 0x7fa93015f590 2024-02-19T15:25:44.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.988+0000 7fa92f400640 1 -- 172.21.15.47:0/1875622814 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9240aa7b0 con 0x7fa93015f590 2024-02-19T15:25:44.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.988+0000 7fa92f400640 1 -- 172.21.15.47:0/1875622814 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9240aab40 con 0x7fa93015f590 2024-02-19T15:25:44.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.988+0000 7fa937600640 1 -- 172.21.15.47:0/1875622814 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa93015f590 msgr2=0x7fa9300d2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:44.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.988+0000 7fa937600640 1 --2- 172.21.15.47:0/1875622814 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa93015f590 0x7fa9300d2690 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fa930072f00 tx=0x7fa92401c4f0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.988+0000 7fa937600640 1 -- 172.21.15.47:0/1875622814 shutdown_connections 2024-02-19T15:25:44.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.988+0000 7fa937600640 1 --2- 172.21.15.47:0/1875622814 >> v2:172.21.15.116:3300/0 conn(0x7fa9300b0a80 0x7fa9300b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.988+0000 7fa937600640 1 --2- 172.21.15.47:0/1875622814 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa93015f590 0x7fa9300d2690 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.988+0000 7fa937600640 1 -- 172.21.15.47:0/1875622814 >> 172.21.15.47:0/1875622814 conn(0x7fa9300d1b70 msgr2=0x7fa9300d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:44.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.988+0000 7fa937600640 1 -- 172.21.15.47:0/1875622814 shutdown_connections 2024-02-19T15:25:44.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.988+0000 7fa937600640 1 -- 172.21.15.47:0/1875622814 wait complete. 2024-02-19T15:25:44.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.988+0000 7fa937600640 1 Processor -- start 2024-02-19T15:25:44.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.988+0000 7fa937600640 1 -- start start 2024-02-19T15:25:44.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa937600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9300b0a80 0x7fa9300c7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:44.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa937600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9300b1a10 0x7fa93015e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:44.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa937600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa93015f590 0x7fa9300c6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:44.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa937600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9300c79c0 con 0x7fa93015f590 2024-02-19T15:25:44.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa937600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9300c6dd0 con 0x7fa9300b0a80 2024-02-19T15:25:44.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa937600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9300cb9c0 con 0x7fa9300b1a10 2024-02-19T15:25:44.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa935600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa93015f590 0x7fa9300c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:44.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa935600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa93015f590 0x7fa9300c6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48302/0 (socket says 172.21.15.47:48302) 2024-02-19T15:25:44.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa935600640 1 -- 172.21.15.47:0/2909292841 learned_addr learned my addr 172.21.15.47:0/2909292841 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:44.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa92fe00640 1 --2- 172.21.15.47:0/2909292841 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9300b1a10 0x7fa93015e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:44.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa934c00640 1 --2- 172.21.15.47:0/2909292841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9300b0a80 0x7fa9300c7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:44.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa935600640 1 -- 172.21.15.47:0/2909292841 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9300b1a10 msgr2=0x7fa93015e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:44.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa935600640 1 --2- 172.21.15.47:0/2909292841 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9300b1a10 0x7fa93015e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa935600640 1 -- 172.21.15.47:0/2909292841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9300b0a80 msgr2=0x7fa9300c7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:44.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa935600640 1 --2- 172.21.15.47:0/2909292841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9300b0a80 0x7fa9300c7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:44.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa935600640 1 -- 172.21.15.47:0/2909292841 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9240094d0 con 0x7fa93015f590 2024-02-19T15:25:44.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa935600640 1 --2- 172.21.15.47:0/2909292841 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa93015f590 0x7fa9300c6050 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fa92800c7a0 tx=0x7fa92800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:44.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa92cc00640 1 -- 172.21.15.47:0/2909292841 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa928003b80 con 0x7fa93015f590 2024-02-19T15:25:44.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa92cc00640 1 -- 172.21.15.47:0/2909292841 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa928003ce0 con 0x7fa93015f590 2024-02-19T15:25:44.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa937600640 1 -- 172.21.15.47:0/2909292841 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9300c7020 con 0x7fa93015f590 2024-02-19T15:25:44.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa92cc00640 1 -- 172.21.15.47:0/2909292841 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92800aa90 con 0x7fa93015f590 2024-02-19T15:25:44.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa937600640 1 -- 172.21.15.47:0/2909292841 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9300c75e0 con 0x7fa93015f590 2024-02-19T15:25:44.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa937600640 1 -- 172.21.15.47:0/2909292841 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9300c7b00 con 0x7fa93015f590 2024-02-19T15:25:44.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.992+0000 7fa92cc00640 1 -- 172.21.15.47:0/2909292841 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa92800abf0 con 0x7fa93015f590 2024-02-19T15:25:44.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.996+0000 7fa92cc00640 1 --2- 172.21.15.47:0/2909292841 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa910066e40 0x7fa910069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:44.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.996+0000 7fa92cc00640 1 -- 172.21.15.47:0/2909292841 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa928084760 con 0x7fa93015f590 2024-02-19T15:25:44.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.996+0000 7fa934c00640 1 --2- 172.21.15.47:0/2909292841 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa910066e40 0x7fa910069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:45.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:44.996+0000 7fa934c00640 1 --2- 172.21.15.47:0/2909292841 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa910066e40 0x7fa910069300 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa9240ae000 tx=0x7fa9240b5040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:45.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.000+0000 7fa92cc00640 1 -- 172.21.15.47:0/2909292841 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa9300c7b00 con 0x7fa93015f590 2024-02-19T15:25:45.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.196+0000 7fa937600640 1 -- 172.21.15.47:0/2909292841 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9300c7b00 con 0x7fa93015f590 2024-02-19T15:25:45.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.196+0000 7fa92cc00640 1 -- 172.21.15.47:0/2909292841 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4946 (secure 0 0 0) 0x7fa9300c7b00 con 0x7fa93015f590 2024-02-19T15:25:45.199 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:45.199 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5684,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3206627105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3206627105},{"type":"v1","addr":"172.21.15.116:6836","nonce":3206627105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5687,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3532185042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3532185042},{"type":"v1","addr":"172.21.15.116:6837","nonce":3532185042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/1482478568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1482478568},{"type":"v1","addr":"172.21.15.47:6834","nonce":1482478568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:25:08.947740+0000","modified":"2024-02-19T15:25:11.954116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/2490197058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2490197058},{"type":"v1","addr":"172.21.15.47:6835","nonce":2490197058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-02-19T15:25:45.200 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 40 2024-02-19T15:25:45.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.196+0000 7fa937600640 1 -- 172.21.15.47:0/2909292841 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa910066e40 msgr2=0x7fa910069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:45.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.196+0000 7fa937600640 1 --2- 172.21.15.47:0/2909292841 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa910066e40 0x7fa910069300 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa9240ae000 tx=0x7fa9240b5040 comp rx=0 tx=0).stop 2024-02-19T15:25:45.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.196+0000 7fa937600640 1 -- 172.21.15.47:0/2909292841 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa93015f590 msgr2=0x7fa9300c6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:45.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.196+0000 7fa937600640 1 --2- 172.21.15.47:0/2909292841 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa93015f590 0x7fa9300c6050 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fa92800c7a0 tx=0x7fa92800cc60 comp rx=0 tx=0).stop 2024-02-19T15:25:45.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.196+0000 7fa937600640 1 -- 172.21.15.47:0/2909292841 shutdown_connections 2024-02-19T15:25:45.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.196+0000 7fa937600640 1 --2- 172.21.15.47:0/2909292841 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa910066e40 0x7fa910069300 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.196+0000 7fa937600640 1 --2- 172.21.15.47:0/2909292841 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa93015f590 0x7fa9300c6050 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.196+0000 7fa937600640 1 --2- 172.21.15.47:0/2909292841 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9300b1a10 0x7fa93015e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.196+0000 7fa937600640 1 --2- 172.21.15.47:0/2909292841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9300b0a80 0x7fa9300c7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.196+0000 7fa937600640 1 -- 172.21.15.47:0/2909292841 >> 172.21.15.47:0/2909292841 conn(0x7fa9300d1b70 msgr2=0x7fa9300ae860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:45.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.196+0000 7fa937600640 1 -- 172.21.15.47:0/2909292841 shutdown_connections 2024-02-19T15:25:45.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.196+0000 7fa937600640 1 -- 172.21.15.47:0/2909292841 wait complete. 2024-02-19T15:25:45.221 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:25:45.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.356+0000 7febd5c00640 1 Processor -- start 2024-02-19T15:25:45.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.356+0000 7febd5c00640 1 -- start start 2024-02-19T15:25:45.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.356+0000 7febd5c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7febd015bea0 0x7febd00cc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:45.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.356+0000 7febd5c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febd0072610 0x7febd00b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:45.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.356+0000 7febd5c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd006dd60 con 0x7febd006aeb0 2024-02-19T15:25:45.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.356+0000 7febd5c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febd00cd340 con 0x7febd015bea0 2024-02-19T15:25:45.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.356+0000 7febd5c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd015c280 con 0x7febd0072610 2024-02-19T15:25:45.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.356+0000 7febcea00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7febd015bea0 0x7febd00cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:45.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.356+0000 7febcea00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7febd015bea0 0x7febd00cc780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48310/0 (socket says 172.21.15.47:48310) 2024-02-19T15:25:45.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.356+0000 7febcea00640 1 -- 172.21.15.47:0/1225894915 learned_addr learned my addr 172.21.15.47:0/1225894915 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:45.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.356+0000 7febce000640 1 --2- 172.21.15.47:0/1225894915 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febd0072610 0x7febd00b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:45.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.360+0000 7febcea00640 1 -- 172.21.15.47:0/1225894915 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febd0072610 msgr2=0x7febd00b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:45.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.360+0000 7febcea00640 1 --2- 172.21.15.47:0/1225894915 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febd0072610 0x7febd00b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.360+0000 7febcea00640 1 -- 172.21.15.47:0/1225894915 >> v1:172.21.15.116:6789/0 conn(0x7febd006aeb0 legacy=0x7febd006e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:45.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.360+0000 7febcea00640 1 -- 172.21.15.47:0/1225894915 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd00ccd80 con 0x7febd015bea0 2024-02-19T15:25:45.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.360+0000 7febcea00640 1 --2- 172.21.15.47:0/1225894915 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febd015bea0 0x7febd00cc780 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7febc00098a0 tx=0x7febc001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ba3e6b17c9e29 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:45.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.360+0000 7febcd600640 1 -- 172.21.15.47:0/1225894915 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc00ab620 con 0x7febd015bea0 2024-02-19T15:25:45.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.360+0000 7febcd600640 1 -- 172.21.15.47:0/1225894915 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc00ab780 con 0x7febd015bea0 2024-02-19T15:25:45.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.360+0000 7febcd600640 1 -- 172.21.15.47:0/1225894915 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc00abb10 con 0x7febd015bea0 2024-02-19T15:25:45.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febd5c00640 1 -- 172.21.15.47:0/1225894915 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febd015bea0 msgr2=0x7febd00cc780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:45.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febd5c00640 1 --2- 172.21.15.47:0/1225894915 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febd015bea0 0x7febd00cc780 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7febc00098a0 tx=0x7febc001c510 comp rx=0 tx=0).stop 2024-02-19T15:25:45.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febd5c00640 1 -- 172.21.15.47:0/1225894915 shutdown_connections 2024-02-19T15:25:45.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febd5c00640 1 --2- 172.21.15.47:0/1225894915 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febd0072610 0x7febd00b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febd5c00640 1 --2- 172.21.15.47:0/1225894915 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febd015bea0 0x7febd00cc780 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febd5c00640 1 -- 172.21.15.47:0/1225894915 >> 172.21.15.47:0/1225894915 conn(0x7febd00cd510 msgr2=0x7febd00cd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:45.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febd5c00640 1 -- 172.21.15.47:0/1225894915 shutdown_connections 2024-02-19T15:25:45.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febd5c00640 1 -- 172.21.15.47:0/1225894915 wait complete. 2024-02-19T15:25:45.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febd5c00640 1 Processor -- start 2024-02-19T15:25:45.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febd5c00640 1 -- start start 2024-02-19T15:25:45.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febd5c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febd006aeb0 0x7febd0152d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:45.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febd5c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febd0072610 0x7febd00ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:45.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febd5c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febd015bea0 0x7febd00b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:45.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febd5c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd00cab00 con 0x7febd015bea0 2024-02-19T15:25:45.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febd5c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd00c7c00 con 0x7febd0072610 2024-02-19T15:25:45.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febd5c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd00c7ee0 con 0x7febd006aeb0 2024-02-19T15:25:45.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febcf400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febd015bea0 0x7febd00b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:45.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febcea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febd006aeb0 0x7febd0152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:45.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febcea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febd006aeb0 0x7febd0152d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:37822/0 (socket says 172.21.15.47:37822) 2024-02-19T15:25:45.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febcea00640 1 -- 172.21.15.47:0/1009795248 learned_addr learned my addr 172.21.15.47:0/1009795248 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:45.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febce000640 1 --2- 172.21.15.47:0/1009795248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febd0072610 0x7febd00ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:45.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febcea00640 1 -- 172.21.15.47:0/1009795248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febd0072610 msgr2=0x7febd00ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:45.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febcea00640 1 --2- 172.21.15.47:0/1009795248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febd0072610 0x7febd00ca5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febcea00640 1 -- 172.21.15.47:0/1009795248 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febd015bea0 msgr2=0x7febd00b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:45.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febcea00640 1 --2- 172.21.15.47:0/1009795248 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febd015bea0 0x7febd00b40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febcea00640 1 -- 172.21.15.47:0/1009795248 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc00094d0 con 0x7febd006aeb0 2024-02-19T15:25:45.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.364+0000 7febcea00640 1 --2- 172.21.15.47:0/1009795248 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febd006aeb0 0x7febd0152d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7febc00ad040 tx=0x7febc001ad50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:45.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.368+0000 7febc6a00640 1 -- 172.21.15.47:0/1009795248 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc00abe80 con 0x7febd006aeb0 2024-02-19T15:25:45.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.368+0000 7febc6a00640 1 -- 172.21.15.47:0/1009795248 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc00be3f0 con 0x7febd006aeb0 2024-02-19T15:25:45.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.368+0000 7febc6a00640 1 -- 172.21.15.47:0/1009795248 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc00be7a0 con 0x7febd006aeb0 2024-02-19T15:25:45.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.368+0000 7febd5c00640 1 -- 172.21.15.47:0/1009795248 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd006b850 con 0x7febd006aeb0 2024-02-19T15:25:45.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.368+0000 7febd5c00640 1 -- 172.21.15.47:0/1009795248 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd01543a0 con 0x7febd006aeb0 2024-02-19T15:25:45.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.368+0000 7febd5c00640 1 -- 172.21.15.47:0/1009795248 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd0068500 con 0x7febd006aeb0 2024-02-19T15:25:45.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.368+0000 7febc6a00640 1 -- 172.21.15.47:0/1009795248 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7febc00b43f0 con 0x7febd006aeb0 2024-02-19T15:25:45.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.368+0000 7febc6a00640 1 --2- 172.21.15.47:0/1009795248 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7febb0066e40 0x7febb0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:45.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.368+0000 7febc6a00640 1 -- 172.21.15.47:0/1009795248 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7febc011eff0 con 0x7febd006aeb0 2024-02-19T15:25:45.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.368+0000 7febce000640 1 --2- 172.21.15.47:0/1009795248 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7febb0066e40 0x7febb0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:45.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.372+0000 7febce000640 1 --2- 172.21.15.47:0/1009795248 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7febb0066e40 0x7febb0069300 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7febd01f8170 tx=0x7febbc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:45.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.376+0000 7febc6a00640 1 -- 172.21.15.47:0/1009795248 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7febc00ebe10 con 0x7febd006aeb0 2024-02-19T15:25:45.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.540+0000 7febd5c00640 1 -- 172.21.15.47:0/1009795248 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febd0075360 con 0x7febd006aeb0 2024-02-19T15:25:45.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.540+0000 7febc6a00640 1 -- 172.21.15.47:0/1009795248 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16680 (secure 0 0 0) 0x7febc00ef4d0 con 0x7febd006aeb0 2024-02-19T15:25:45.546 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:45.546 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":69,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:25:10.552537+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:25:06.759758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-02-19T15:25:07.949435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:25:45.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.544+0000 7febd5c00640 1 -- 172.21.15.47:0/1009795248 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7febb0066e40 msgr2=0x7febb0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:45.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.544+0000 7febd5c00640 1 --2- 172.21.15.47:0/1009795248 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7febb0066e40 0x7febb0069300 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7febd01f8170 tx=0x7febbc009190 comp rx=0 tx=0).stop 2024-02-19T15:25:45.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.544+0000 7febd5c00640 1 -- 172.21.15.47:0/1009795248 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febd006aeb0 msgr2=0x7febd0152d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:45.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.544+0000 7febd5c00640 1 --2- 172.21.15.47:0/1009795248 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febd006aeb0 0x7febd0152d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7febc00ad040 tx=0x7febc001ad50 comp rx=0 tx=0).stop 2024-02-19T15:25:45.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.544+0000 7febd5c00640 1 -- 172.21.15.47:0/1009795248 shutdown_connections 2024-02-19T15:25:45.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.544+0000 7febd5c00640 1 --2- 172.21.15.47:0/1009795248 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7febb0066e40 0x7febb0069300 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.544+0000 7febd5c00640 1 --2- 172.21.15.47:0/1009795248 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febd015bea0 0x7febd00b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.544+0000 7febd5c00640 1 --2- 172.21.15.47:0/1009795248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febd0072610 0x7febd00ca5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.544+0000 7febd5c00640 1 --2- 172.21.15.47:0/1009795248 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febd006aeb0 0x7febd0152d40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.544+0000 7febd5c00640 1 -- 172.21.15.47:0/1009795248 >> 172.21.15.47:0/1009795248 conn(0x7febd00cd510 msgr2=0x7febd0159360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:45.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.544+0000 7febd5c00640 1 -- 172.21.15.47:0/1009795248 shutdown_connections 2024-02-19T15:25:45.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.544+0000 7febd5c00640 1 -- 172.21.15.47:0/1009795248 wait complete. 2024-02-19T15:25:45.568 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T15:25:45.568 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:25:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.704+0000 7fee28800640 1 Processor -- start 2024-02-19T15:25:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.704+0000 7fee28800640 1 -- start start 2024-02-19T15:25:45.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.704+0000 7fee28800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee2015b1a0 0x7fee2015b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:45.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.704+0000 7fee28800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fee20071350 0x7fee2015bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:45.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.704+0000 7fee28800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2015c000 con 0x7fee2006c9a0 2024-02-19T15:25:45.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.704+0000 7fee28800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee200718c0 con 0x7fee20071350 2024-02-19T15:25:45.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.704+0000 7fee28800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee20071ba0 con 0x7fee2015b1a0 2024-02-19T15:25:45.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.704+0000 7fee26800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fee2006c9a0 0x7fee2015c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52886/0 (socket says 172.21.15.47:52886) 2024-02-19T15:25:45.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.704+0000 7fee26800640 1 -- 172.21.15.47:0/3149593480 learned_addr learned my addr 172.21.15.47:0/3149593480 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:45.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.704+0000 7fee25400640 1 --2- 172.21.15.47:0/3149593480 >> v2:172.21.15.116:3300/0 conn(0x7fee20071350 0x7fee2015bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:45.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.704+0000 7fee25e00640 1 --2- 172.21.15.47:0/3149593480 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee2015b1a0 0x7fee2015b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:45.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.704+0000 7fee24a00640 1 -- 172.21.15.47:0/3149593480 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182355674 0 0) 0x7fee2015c000 con 0x7fee2006c9a0 2024-02-19T15:25:45.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.704+0000 7fee24a00640 1 -- 172.21.15.47:0/3149593480 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee08005740 con 0x7fee2006c9a0 2024-02-19T15:25:45.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.704+0000 7fee24a00640 1 -- 172.21.15.47:0/3149593480 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2655816799 0 0) 0x7fee08005740 con 0x7fee2006c9a0 2024-02-19T15:25:45.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.704+0000 7fee24a00640 1 -- 172.21.15.47:0/3149593480 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee2015b1a0 msgr2=0x7fee2015b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:45.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.704+0000 7fee24a00640 1 --2- 172.21.15.47:0/3149593480 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee2015b1a0 0x7fee2015b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.704+0000 7fee24a00640 1 -- 172.21.15.47:0/3149593480 >> v2:172.21.15.116:3300/0 conn(0x7fee20071350 msgr2=0x7fee2015bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:45.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.704+0000 7fee24a00640 1 --2- 172.21.15.47:0/3149593480 >> v2:172.21.15.116:3300/0 conn(0x7fee20071350 0x7fee2015bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.704+0000 7fee24a00640 1 -- 172.21.15.47:0/3149593480 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee20071e80 con 0x7fee2006c9a0 2024-02-19T15:25:45.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee24a00640 1 -- 172.21.15.47:0/3149593480 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7fee1c001a00 con 0x7fee2006c9a0 2024-02-19T15:25:45.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee25400640 1 --2- 172.21.15.47:0/3149593480 >> v2:172.21.15.116:3300/0 conn(0x7fee20071350 0x7fee2015bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:25:45.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee24a00640 1 -- 172.21.15.47:0/3149593480 >> v1:172.21.15.47:6789/0 conn(0x7fee2006c9a0 legacy=0x7fee2015c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:45.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee24a00640 1 --2- 172.21.15.47:0/3149593480 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee2015b1a0 0x7fee08005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:45.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee24a00640 1 --2- 172.21.15.47:0/3149593480 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee08006110 0x7fee080069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:45.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee24a00640 1 --2- 172.21.15.47:0/3149593480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee08006f20 0x7fee080073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:45.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee24a00640 1 -- 172.21.15.47:0/3149593480 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee200718c0 con 0x7fee08006110 2024-02-19T15:25:45.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee24a00640 1 -- 172.21.15.47:0/3149593480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee20071ba0 con 0x7fee08006f20 2024-02-19T15:25:45.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee24a00640 1 -- 172.21.15.47:0/3149593480 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee0800dba0 con 0x7fee2015b1a0 2024-02-19T15:25:45.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee25400640 1 --2- 172.21.15.47:0/3149593480 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee08006110 0x7fee080069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:45.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee25e00640 1 --2- 172.21.15.47:0/3149593480 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee2015b1a0 0x7fee08005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:45.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee26800640 1 --2- 172.21.15.47:0/3149593480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee08006f20 0x7fee080073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:45.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee25400640 1 -- 172.21.15.47:0/3149593480 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee2015b1a0 msgr2=0x7fee08005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:45.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee25400640 1 --2- 172.21.15.47:0/3149593480 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee2015b1a0 0x7fee08005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee25400640 1 -- 172.21.15.47:0/3149593480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee08006f20 msgr2=0x7fee080073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:45.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee25400640 1 --2- 172.21.15.47:0/3149593480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee08006f20 0x7fee080073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee25400640 1 -- 172.21.15.47:0/3149593480 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee08005740 con 0x7fee08006110 2024-02-19T15:25:45.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee26800640 1 --2- 172.21.15.47:0/3149593480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee08006f20 0x7fee080073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:45.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee25400640 1 --2- 172.21.15.47:0/3149593480 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee08006110 0x7fee080069e0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fee10009cb0 tx=0x7fee1001cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709a8794a00dea2c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:45.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee24a00640 1 -- 172.21.15.47:0/3149593480 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee100a5950 con 0x7fee08006110 2024-02-19T15:25:45.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee24a00640 1 -- 172.21.15.47:0/3149593480 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee100a5ab0 con 0x7fee08006110 2024-02-19T15:25:45.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.708+0000 7fee24a00640 1 -- 172.21.15.47:0/3149593480 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee100a5ea0 con 0x7fee08006110 2024-02-19T15:25:45.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee28800640 1 -- 172.21.15.47:0/3149593480 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee08006110 msgr2=0x7fee080069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:45.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee28800640 1 --2- 172.21.15.47:0/3149593480 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee08006110 0x7fee080069e0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fee10009cb0 tx=0x7fee1001cc30 comp rx=0 tx=0).stop 2024-02-19T15:25:45.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee26800640 1 -- 172.21.15.47:0/3149593480 reap_dead start 2024-02-19T15:25:45.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee28800640 1 -- 172.21.15.47:0/3149593480 shutdown_connections 2024-02-19T15:25:45.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee28800640 1 -- 172.21.15.47:0/3149593480 >> 172.21.15.47:0/3149593480 conn(0x7fee200b16a0 msgr2=0x7fee200b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:45.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee28800640 1 -- 172.21.15.47:0/3149593480 shutdown_connections 2024-02-19T15:25:45.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee28800640 1 -- 172.21.15.47:0/3149593480 wait complete. 2024-02-19T15:25:45.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee28800640 1 Processor -- start 2024-02-19T15:25:45.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee28800640 1 -- start start 2024-02-19T15:25:45.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee28800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee08006110 0x7fee201517e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:45.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee28800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee20150780 0x7fee20155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:45.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee28800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee20157440 0x7fee201578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:45.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee28800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2015c9a0 con 0x7fee20150780 2024-02-19T15:25:45.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee28800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee20152760 con 0x7fee20157440 2024-02-19T15:25:45.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee28800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee20157e30 con 0x7fee08006110 2024-02-19T15:25:45.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee25400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee20150780 0x7fee20155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:45.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee25400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee20150780 0x7fee20155a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48350/0 (socket says 172.21.15.47:48350) 2024-02-19T15:25:45.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee25400640 1 -- 172.21.15.47:0/464618087 learned_addr learned my addr 172.21.15.47:0/464618087 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:45.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee26800640 1 --2- 172.21.15.47:0/464618087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee20157440 0x7fee201578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:45.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee25e00640 1 --2- 172.21.15.47:0/464618087 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee08006110 0x7fee201517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:45.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee25400640 1 -- 172.21.15.47:0/464618087 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee08006110 msgr2=0x7fee201517e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:45.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee25400640 1 --2- 172.21.15.47:0/464618087 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee08006110 0x7fee201517e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee25400640 1 -- 172.21.15.47:0/464618087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee20157440 msgr2=0x7fee201578f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:45.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee25400640 1 --2- 172.21.15.47:0/464618087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee20157440 0x7fee201578f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.712+0000 7fee25400640 1 -- 172.21.15.47:0/464618087 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee20151ec0 con 0x7fee20150780 2024-02-19T15:25:45.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.716+0000 7fee25400640 1 --2- 172.21.15.47:0/464618087 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee20150780 0x7fee20155a20 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fee1001d4d0 tx=0x7fee08007a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:45.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.716+0000 7fee1a000640 1 -- 172.21.15.47:0/464618087 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee100a5950 con 0x7fee20150780 2024-02-19T15:25:45.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.716+0000 7fee1a000640 1 -- 172.21.15.47:0/464618087 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee100ade40 con 0x7fee20150780 2024-02-19T15:25:45.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.716+0000 7fee1a000640 1 -- 172.21.15.47:0/464618087 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee10004260 con 0x7fee20150780 2024-02-19T15:25:45.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.716+0000 7fee28800640 1 -- 172.21.15.47:0/464618087 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee201521a0 con 0x7fee20150780 2024-02-19T15:25:45.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.716+0000 7fee28800640 1 -- 172.21.15.47:0/464618087 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee20155f60 con 0x7fee20150780 2024-02-19T15:25:45.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.716+0000 7fee28800640 1 -- 172.21.15.47:0/464618087 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee20075360 con 0x7fee20150780 2024-02-19T15:25:45.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.716+0000 7fee1a000640 1 -- 172.21.15.47:0/464618087 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fee100b4040 con 0x7fee20150780 2024-02-19T15:25:45.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.716+0000 7fee1a000640 1 --2- 172.21.15.47:0/464618087 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fee04066e70 0x7fee04069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:45.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.716+0000 7fee1a000640 1 -- 172.21.15.47:0/464618087 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee1011f3e0 con 0x7fee20150780 2024-02-19T15:25:45.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.716+0000 7fee25e00640 1 --2- 172.21.15.47:0/464618087 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fee04066e70 0x7fee04069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:45.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.720+0000 7fee25e00640 1 --2- 172.21.15.47:0/464618087 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fee04066e70 0x7fee04069330 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fee14009b00 tx=0x7fee14009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:45.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.728+0000 7fee1a000640 1 -- 172.21.15.47:0/464618087 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fee100ec490 con 0x7fee20150780 2024-02-19T15:25:45.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.924+0000 7fee28800640 1 -- 172.21.15.47:0/464618087 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee2015bca0 con 0x7fee20150780 2024-02-19T15:25:45.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.924+0000 7fee1a000640 1 -- 172.21.15.47:0/464618087 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4946 (secure 0 0 0) 0x7fee100f6020 con 0x7fee20150780 2024-02-19T15:25:45.929 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:45.929 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5684,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3206627105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3206627105},{"type":"v1","addr":"172.21.15.116:6836","nonce":3206627105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5687,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3532185042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3532185042},{"type":"v1","addr":"172.21.15.116:6837","nonce":3532185042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/1482478568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1482478568},{"type":"v1","addr":"172.21.15.47:6834","nonce":1482478568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:25:08.947740+0000","modified":"2024-02-19T15:25:11.954116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/2490197058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2490197058},{"type":"v1","addr":"172.21.15.47:6835","nonce":2490197058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-02-19T15:25:45.929 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 40 2024-02-19T15:25:45.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.928+0000 7fee28800640 1 -- 172.21.15.47:0/464618087 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fee04066e70 msgr2=0x7fee04069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:45.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.928+0000 7fee28800640 1 --2- 172.21.15.47:0/464618087 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fee04066e70 0x7fee04069330 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fee14009b00 tx=0x7fee14009190 comp rx=0 tx=0).stop 2024-02-19T15:25:45.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.928+0000 7fee28800640 1 -- 172.21.15.47:0/464618087 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee20150780 msgr2=0x7fee20155a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:45.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.928+0000 7fee28800640 1 --2- 172.21.15.47:0/464618087 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee20150780 0x7fee20155a20 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fee1001d4d0 tx=0x7fee08007a10 comp rx=0 tx=0).stop 2024-02-19T15:25:45.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.928+0000 7fee28800640 1 -- 172.21.15.47:0/464618087 shutdown_connections 2024-02-19T15:25:45.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.928+0000 7fee28800640 1 --2- 172.21.15.47:0/464618087 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fee04066e70 0x7fee04069330 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.928+0000 7fee28800640 1 --2- 172.21.15.47:0/464618087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee20157440 0x7fee201578f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.928+0000 7fee28800640 1 --2- 172.21.15.47:0/464618087 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fee20150780 0x7fee20155a20 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.928+0000 7fee28800640 1 --2- 172.21.15.47:0/464618087 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fee08006110 0x7fee201517e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:45.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.928+0000 7fee28800640 1 -- 172.21.15.47:0/464618087 >> 172.21.15.47:0/464618087 conn(0x7fee200b16a0 msgr2=0x7fee20159f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:45.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.928+0000 7fee28800640 1 -- 172.21.15.47:0/464618087 shutdown_connections 2024-02-19T15:25:45.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:45.928+0000 7fee28800640 1 -- 172.21.15.47:0/464618087 wait complete. 2024-02-19T15:25:45.958 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:25:46.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.096+0000 7fe89dc00640 1 Processor -- start 2024-02-19T15:25:46.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.096+0000 7fe89dc00640 1 -- start start 2024-02-19T15:25:46.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.100+0000 7fe89dc00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe89806aeb0 0x7fe89806b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:46.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.100+0000 7fe89dc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe8980c6780 0x7fe8980c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:46.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.100+0000 7fe89dc00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8980ad840 con 0x7fe8980cb520 2024-02-19T15:25:46.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.100+0000 7fe89dc00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8980c62f0 con 0x7fe89806aeb0 2024-02-19T15:25:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.100+0000 7fe89dc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89806b7d0 con 0x7fe8980c6780 2024-02-19T15:25:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.100+0000 7fe896a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe89806aeb0 0x7fe89806b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.100+0000 7fe896a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe89806aeb0 0x7fe89806b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48362/0 (socket says 172.21.15.47:48362) 2024-02-19T15:25:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.100+0000 7fe896a00640 1 -- 172.21.15.47:0/52903906 learned_addr learned my addr 172.21.15.47:0/52903906 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.100+0000 7fe88fe00640 1 --1- 172.21.15.47:0/52903906 >> v1:172.21.15.47:6789/0 conn(0x7fe8980cb520 0x7fe8980cb9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52896/0 (socket says 172.21.15.47:52896) 2024-02-19T15:25:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.100+0000 7fe897400640 1 --2- 172.21.15.47:0/52903906 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe8980c6780 0x7fe8980c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.100+0000 7fe896a00640 1 -- 172.21.15.47:0/52903906 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe8980c6780 msgr2=0x7fe8980c6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.100+0000 7fe896a00640 1 --2- 172.21.15.47:0/52903906 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe8980c6780 0x7fe8980c6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.100+0000 7fe896a00640 1 -- 172.21.15.47:0/52903906 >> v1:172.21.15.47:6789/0 conn(0x7fe8980cb520 legacy=0x7fe8980cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.100+0000 7fe896a00640 1 -- 172.21.15.47:0/52903906 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8980cc810 con 0x7fe89806aeb0 2024-02-19T15:25:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.100+0000 7fe896000640 1 -- 172.21.15.47:0/52903906 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298903048 0 0) 0x7fe8980ad840 con 0x7fe8980cb520 2024-02-19T15:25:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.100+0000 7fe896a00640 1 --2- 172.21.15.47:0/52903906 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe89806aeb0 0x7fe89806b290 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fe888009870 tx=0x7fe88801c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f98a30e3d90f659 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe896000640 1 -- 172.21.15.47:0/52903906 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8980c7960 con 0x7fe89806aeb0 2024-02-19T15:25:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe896000640 1 -- 172.21.15.47:0/52903906 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8880a95f0 con 0x7fe89806aeb0 2024-02-19T15:25:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe896000640 1 -- 172.21.15.47:0/52903906 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8880a9980 con 0x7fe89806aeb0 2024-02-19T15:25:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe89dc00640 1 -- 172.21.15.47:0/52903906 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe89806aeb0 msgr2=0x7fe89806b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe89dc00640 1 --2- 172.21.15.47:0/52903906 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe89806aeb0 0x7fe89806b290 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fe888009870 tx=0x7fe88801c530 comp rx=0 tx=0).stop 2024-02-19T15:25:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe89dc00640 1 -- 172.21.15.47:0/52903906 shutdown_connections 2024-02-19T15:25:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe89dc00640 1 --2- 172.21.15.47:0/52903906 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe8980c6780 0x7fe8980c6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe89dc00640 1 --2- 172.21.15.47:0/52903906 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe89806aeb0 0x7fe89806b290 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe89dc00640 1 -- 172.21.15.47:0/52903906 >> 172.21.15.47:0/52903906 conn(0x7fe8980d0ad0 msgr2=0x7fe8980d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe89dc00640 1 -- 172.21.15.47:0/52903906 shutdown_connections 2024-02-19T15:25:46.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe89dc00640 1 -- 172.21.15.47:0/52903906 wait complete. 2024-02-19T15:25:46.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe89dc00640 1 Processor -- start 2024-02-19T15:25:46.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe89dc00640 1 -- start start 2024-02-19T15:25:46.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe89dc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe89806aeb0 0x7fe8980b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:46.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe89dc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8980c6780 0x7fe8981541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:46.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe89dc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8980cb520 0x7fe89815ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:46.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe89dc00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89815f460 con 0x7fe8980c6780 2024-02-19T15:25:46.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe89dc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe898154f20 con 0x7fe8980cb520 2024-02-19T15:25:46.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe89dc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe898155200 con 0x7fe89806aeb0 2024-02-19T15:25:46.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe88fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8980c6780 0x7fe8981541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:46.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe88fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8980c6780 0x7fe8981541d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48368/0 (socket says 172.21.15.47:48368) 2024-02-19T15:25:46.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.104+0000 7fe88fe00640 1 -- 172.21.15.47:0/248327637 learned_addr learned my addr 172.21.15.47:0/248327637 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:46.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.108+0000 7fe896a00640 1 --2- 172.21.15.47:0/248327637 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe89806aeb0 0x7fe8980b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:46.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.108+0000 7fe897400640 1 --2- 172.21.15.47:0/248327637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8980cb520 0x7fe89815ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:46.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.108+0000 7fe88fe00640 1 -- 172.21.15.47:0/248327637 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe89806aeb0 msgr2=0x7fe8980b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:46.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.108+0000 7fe88fe00640 1 --2- 172.21.15.47:0/248327637 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe89806aeb0 0x7fe8980b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.108+0000 7fe88fe00640 1 -- 172.21.15.47:0/248327637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8980cb520 msgr2=0x7fe89815ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:46.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.108+0000 7fe88fe00640 1 --2- 172.21.15.47:0/248327637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8980cb520 0x7fe89815ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.108+0000 7fe88fe00640 1 -- 172.21.15.47:0/248327637 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8880094d0 con 0x7fe8980c6780 2024-02-19T15:25:46.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.108+0000 7fe897400640 1 --2- 172.21.15.47:0/248327637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8980cb520 0x7fe89815ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:46.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.108+0000 7fe88fe00640 1 --2- 172.21.15.47:0/248327637 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8980c6780 0x7fe8981541d0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fe88000c480 tx=0x7fe88000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:46.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.108+0000 7fe88ea00640 1 -- 172.21.15.47:0/248327637 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8800074e0 con 0x7fe8980c6780 2024-02-19T15:25:46.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.108+0000 7fe88ea00640 1 -- 172.21.15.47:0/248327637 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe880007660 con 0x7fe8980c6780 2024-02-19T15:25:46.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.108+0000 7fe88ea00640 1 -- 172.21.15.47:0/248327637 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880004a30 con 0x7fe8980c6780 2024-02-19T15:25:46.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.108+0000 7fe89dc00640 1 -- 172.21.15.47:0/248327637 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8981554e0 con 0x7fe8980c6780 2024-02-19T15:25:46.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.108+0000 7fe89dc00640 1 -- 172.21.15.47:0/248327637 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe898155aa0 con 0x7fe8980c6780 2024-02-19T15:25:46.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.108+0000 7fe89dc00640 1 -- 172.21.15.47:0/248327637 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe898155c80 con 0x7fe8980c6780 2024-02-19T15:25:46.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.108+0000 7fe88ea00640 1 -- 172.21.15.47:0/248327637 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe880004b90 con 0x7fe8980c6780 2024-02-19T15:25:46.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.108+0000 7fe88ea00640 1 --2- 172.21.15.47:0/248327637 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe874066e40 0x7fe874069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:46.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.108+0000 7fe88ea00640 1 -- 172.21.15.47:0/248327637 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe880083c70 con 0x7fe8980c6780 2024-02-19T15:25:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.112+0000 7fe896a00640 1 --2- 172.21.15.47:0/248327637 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe874066e40 0x7fe874069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.112+0000 7fe896a00640 1 --2- 172.21.15.47:0/248327637 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe874066e40 0x7fe874069300 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe8880ae000 tx=0x7fe8880b3040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:46.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.120+0000 7fe88ea00640 1 -- 172.21.15.47:0/248327637 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe880054560 con 0x7fe8980c6780 2024-02-19T15:25:46.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.328+0000 7fe89dc00640 1 -- 172.21.15.47:0/248327637 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe898075360 con 0x7fe8980c6780 2024-02-19T15:25:46.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.328+0000 7fe88ea00640 1 -- 172.21.15.47:0/248327637 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4946 (secure 0 0 0) 0x7fe880018020 con 0x7fe8980c6780 2024-02-19T15:25:46.334 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:46.334 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5684,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3206627105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3206627105},{"type":"v1","addr":"172.21.15.116:6836","nonce":3206627105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5687,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3532185042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3532185042},{"type":"v1","addr":"172.21.15.116:6837","nonce":3532185042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/1482478568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1482478568},{"type":"v1","addr":"172.21.15.47:6834","nonce":1482478568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:25:08.947740+0000","modified":"2024-02-19T15:25:11.954116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/2490197058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":2490197058},{"type":"v1","addr":"172.21.15.47:6835","nonce":2490197058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-02-19T15:25:46.334 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 40 2024-02-19T15:25:46.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.332+0000 7fe89dc00640 1 -- 172.21.15.47:0/248327637 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe874066e40 msgr2=0x7fe874069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:46.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.332+0000 7fe89dc00640 1 --2- 172.21.15.47:0/248327637 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe874066e40 0x7fe874069300 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe8880ae000 tx=0x7fe8880b3040 comp rx=0 tx=0).stop 2024-02-19T15:25:46.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.332+0000 7fe89dc00640 1 -- 172.21.15.47:0/248327637 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8980c6780 msgr2=0x7fe8981541d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:46.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.332+0000 7fe89dc00640 1 --2- 172.21.15.47:0/248327637 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8980c6780 0x7fe8981541d0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fe88000c480 tx=0x7fe88000c940 comp rx=0 tx=0).stop 2024-02-19T15:25:46.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.332+0000 7fe89dc00640 1 -- 172.21.15.47:0/248327637 shutdown_connections 2024-02-19T15:25:46.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.332+0000 7fe89dc00640 1 --2- 172.21.15.47:0/248327637 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe874066e40 0x7fe874069300 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.332+0000 7fe89dc00640 1 --2- 172.21.15.47:0/248327637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8980cb520 0x7fe89815ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.332+0000 7fe89dc00640 1 --2- 172.21.15.47:0/248327637 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8980c6780 0x7fe8981541d0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.332+0000 7fe89dc00640 1 --2- 172.21.15.47:0/248327637 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe89806aeb0 0x7fe8980b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.332+0000 7fe89dc00640 1 -- 172.21.15.47:0/248327637 >> 172.21.15.47:0/248327637 conn(0x7fe8980d0ad0 msgr2=0x7fe8980cca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:46.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.332+0000 7fe89dc00640 1 -- 172.21.15.47:0/248327637 shutdown_connections 2024-02-19T15:25:46.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.332+0000 7fe89dc00640 1 -- 172.21.15.47:0/248327637 wait complete. 2024-02-19T15:25:46.354 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:25:46.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.492+0000 7f0cb2200640 1 Processor -- start 2024-02-19T15:25:46.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.492+0000 7f0cb2200640 1 -- start start 2024-02-19T15:25:46.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.492+0000 7f0cb2200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cac145590 0x7f0cac0727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:46.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.492+0000 7f0cb2200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0cac15c390 0x7f0cac15c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:46.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.492+0000 7f0cb2200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cac15d280 con 0x7f0cac0cea80 2024-02-19T15:25:46.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.492+0000 7f0cb2200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cac070ff0 con 0x7f0cac15c390 2024-02-19T15:25:46.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.492+0000 7f0cb2200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cac15cd60 con 0x7f0cac145590 2024-02-19T15:25:46.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.492+0000 7f0cabe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0cac15c390 0x7f0cac15c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:46.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.492+0000 7f0ca3e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0cac0cea80 0x7f0cac06c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52906/0 (socket says 172.21.15.47:52906) 2024-02-19T15:25:46.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.492+0000 7f0ca3e00640 1 -- 172.21.15.47:0/3209513708 learned_addr learned my addr 172.21.15.47:0/3209513708 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:46.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.492+0000 7f0cab400640 1 --2- 172.21.15.47:0/3209513708 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cac145590 0x7f0cac0727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:46.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.492+0000 7f0cabe00640 1 -- 172.21.15.47:0/3209513708 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cac145590 msgr2=0x7f0cac0727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:46.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.492+0000 7f0cabe00640 1 --2- 172.21.15.47:0/3209513708 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cac145590 0x7f0cac0727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.492+0000 7f0cabe00640 1 -- 172.21.15.47:0/3209513708 >> v1:172.21.15.47:6789/0 conn(0x7f0cac0cea80 legacy=0x7f0cac06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:46.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.492+0000 7f0cabe00640 1 -- 172.21.15.47:0/3209513708 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cac0709e0 con 0x7f0cac15c390 2024-02-19T15:25:46.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.492+0000 7f0caaa00640 1 -- 172.21.15.47:0/3209513708 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675629874 0 0) 0x7f0cac15d280 con 0x7f0cac0cea80 2024-02-19T15:25:46.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.496+0000 7f0cabe00640 1 --2- 172.21.15.47:0/3209513708 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0cac15c390 0x7f0cac15c820 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4009d90 tx=0x7f0ca401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1d7ce33770f00 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:46.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.496+0000 7f0caaa00640 1 -- 172.21.15.47:0/3209513708 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca40a5940 con 0x7f0cac15c390 2024-02-19T15:25:46.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.496+0000 7f0caaa00640 1 -- 172.21.15.47:0/3209513708 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca40a5aa0 con 0x7f0cac15c390 2024-02-19T15:25:46.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.496+0000 7f0caaa00640 1 -- 172.21.15.47:0/3209513708 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca40a5e30 con 0x7f0cac15c390 2024-02-19T15:25:46.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.496+0000 7f0cb2200640 1 -- 172.21.15.47:0/3209513708 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0cac15c390 msgr2=0x7f0cac15c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:46.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.496+0000 7f0cb2200640 1 --2- 172.21.15.47:0/3209513708 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0cac15c390 0x7f0cac15c820 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4009d90 tx=0x7f0ca401c970 comp rx=0 tx=0).stop 2024-02-19T15:25:46.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.496+0000 7f0cb2200640 1 -- 172.21.15.47:0/3209513708 shutdown_connections 2024-02-19T15:25:46.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.496+0000 7f0cb2200640 1 --2- 172.21.15.47:0/3209513708 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0cac15c390 0x7f0cac15c820 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.496+0000 7f0cb2200640 1 --2- 172.21.15.47:0/3209513708 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cac145590 0x7f0cac0727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.496+0000 7f0cb2200640 1 -- 172.21.15.47:0/3209513708 >> 172.21.15.47:0/3209513708 conn(0x7f0cac0b16a0 msgr2=0x7f0cac0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:46.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.496+0000 7f0cb2200640 1 -- 172.21.15.47:0/3209513708 shutdown_connections 2024-02-19T15:25:46.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.496+0000 7f0cb2200640 1 -- 172.21.15.47:0/3209513708 wait complete. 2024-02-19T15:25:46.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0cb2200640 1 Processor -- start 2024-02-19T15:25:46.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0cb2200640 1 -- start start 2024-02-19T15:25:46.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0cb2200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0cac0cea80 0x7f0cac141ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:46.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0cb2200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cac145590 0x7f0cac0c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:46.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0cb2200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cac15c390 0x7f0cac0b2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:46.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0cb2200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cac15b800 con 0x7f0cac0cea80 2024-02-19T15:25:46.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0cb2200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cac141ff0 con 0x7f0cac145590 2024-02-19T15:25:46.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0cb2200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cac142220 con 0x7f0cac15c390 2024-02-19T15:25:46.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0cab400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0cac0cea80 0x7f0cac141ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:46.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0cab400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0cac0cea80 0x7f0cac141ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48388/0 (socket says 172.21.15.47:48388) 2024-02-19T15:25:46.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0cab400640 1 -- 172.21.15.47:0/2553156702 learned_addr learned my addr 172.21.15.47:0/2553156702 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:46.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0cabe00640 1 --2- 172.21.15.47:0/2553156702 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cac15c390 0x7f0cac0b2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:46.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0ca3e00640 1 --2- 172.21.15.47:0/2553156702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cac145590 0x7f0cac0c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:46.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0cab400640 1 -- 172.21.15.47:0/2553156702 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cac15c390 msgr2=0x7f0cac0b2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:46.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0cab400640 1 --2- 172.21.15.47:0/2553156702 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cac15c390 0x7f0cac0b2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0cab400640 1 -- 172.21.15.47:0/2553156702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cac145590 msgr2=0x7f0cac0c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:46.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0cab400640 1 --2- 172.21.15.47:0/2553156702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cac145590 0x7f0cac0c4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0cab400640 1 -- 172.21.15.47:0/2553156702 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca4009a70 con 0x7f0cac0cea80 2024-02-19T15:25:46.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0ca3e00640 1 --2- 172.21.15.47:0/2553156702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cac145590 0x7f0cac0c4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:25:46.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0cab400640 1 --2- 172.21.15.47:0/2553156702 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0cac0cea80 0x7f0cac141ab0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c00d800 tx=0x7f0c9c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:46.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0ca3400640 1 -- 172.21.15.47:0/2553156702 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c00b6c0 con 0x7f0cac0cea80 2024-02-19T15:25:46.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0ca3400640 1 -- 172.21.15.47:0/2553156702 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c9c00b820 con 0x7f0cac0cea80 2024-02-19T15:25:46.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0cb2200640 1 -- 172.21.15.47:0/2553156702 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cac142500 con 0x7f0cac0cea80 2024-02-19T15:25:46.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0ca3400640 1 -- 172.21.15.47:0/2553156702 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c00bb90 con 0x7f0cac0cea80 2024-02-19T15:25:46.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.500+0000 7f0cb2200640 1 -- 172.21.15.47:0/2553156702 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cac142ac0 con 0x7f0cac0cea80 2024-02-19T15:25:46.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.504+0000 7f0cb2200640 1 -- 172.21.15.47:0/2553156702 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cac0af380 con 0x7f0cac0cea80 2024-02-19T15:25:46.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.504+0000 7f0ca3400640 1 -- 172.21.15.47:0/2553156702 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0c9c00bcf0 con 0x7f0cac0cea80 2024-02-19T15:25:46.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.504+0000 7f0ca3400640 1 --2- 172.21.15.47:0/2553156702 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0c88066e40 0x7f0c88069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:46.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.504+0000 7f0ca3400640 1 -- 172.21.15.47:0/2553156702 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0c9c084fb0 con 0x7f0cac0cea80 2024-02-19T15:25:46.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.504+0000 7f0ca3e00640 1 --2- 172.21.15.47:0/2553156702 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0c88066e40 0x7f0c88069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:46.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.508+0000 7f0ca3e00640 1 --2- 172.21.15.47:0/2553156702 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0c88066e40 0x7f0c88069300 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0cac0af330 tx=0x7f0c94009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:46.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.508+0000 7f0ca3400640 1 -- 172.21.15.47:0/2553156702 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0c9c051d50 con 0x7f0cac0cea80 2024-02-19T15:25:46.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.668+0000 7f0cb2200640 1 -- 172.21.15.47:0/2553156702 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cac075360 con 0x7f0cac0cea80 2024-02-19T15:25:46.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.672+0000 7f0ca3400640 1 -- 172.21.15.47:0/2553156702 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16778 (secure 0 0 0) 0x7f0c9c055410 con 0x7f0cac0cea80 2024-02-19T15:25:46.675 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:46.675 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":70,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:25:46.025127+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:25:06.759758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-02-19T15:25:07.949435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.2d","mappings":[{"from":6,"to":0}]},{"pgid":"11.3a","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:25:46.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.672+0000 7f0cb2200640 1 -- 172.21.15.47:0/2553156702 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0c88066e40 msgr2=0x7f0c88069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:46.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.672+0000 7f0cb2200640 1 --2- 172.21.15.47:0/2553156702 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0c88066e40 0x7f0c88069300 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0cac0af330 tx=0x7f0c94009040 comp rx=0 tx=0).stop 2024-02-19T15:25:46.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.672+0000 7f0cb2200640 1 -- 172.21.15.47:0/2553156702 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0cac0cea80 msgr2=0x7f0cac141ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:46.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.672+0000 7f0cb2200640 1 --2- 172.21.15.47:0/2553156702 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0cac0cea80 0x7f0cac141ab0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c00d800 tx=0x7f0c9c00dcc0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.672+0000 7f0cb2200640 1 -- 172.21.15.47:0/2553156702 shutdown_connections 2024-02-19T15:25:46.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.672+0000 7f0cb2200640 1 --2- 172.21.15.47:0/2553156702 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0c88066e40 0x7f0c88069300 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.672+0000 7f0cb2200640 1 --2- 172.21.15.47:0/2553156702 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cac15c390 0x7f0cac0b2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.672+0000 7f0cb2200640 1 --2- 172.21.15.47:0/2553156702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cac145590 0x7f0cac0c4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.672+0000 7f0cb2200640 1 --2- 172.21.15.47:0/2553156702 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0cac0cea80 0x7f0cac141ab0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.672+0000 7f0cb2200640 1 -- 172.21.15.47:0/2553156702 >> 172.21.15.47:0/2553156702 conn(0x7f0cac0b16a0 msgr2=0x7f0cac140040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:46.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.672+0000 7f0cb2200640 1 -- 172.21.15.47:0/2553156702 shutdown_connections 2024-02-19T15:25:46.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.672+0000 7f0cb2200640 1 -- 172.21.15.47:0/2553156702 wait complete. 2024-02-19T15:25:46.697 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T15:25:46.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.832+0000 7fc20c800640 1 Processor -- start 2024-02-19T15:25:46.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.832+0000 7fc20c800640 1 -- start start 2024-02-19T15:25:46.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.832+0000 7fc20c800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc20415bcd0 0x7fc2040cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:46.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.832+0000 7fc20c800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc2040cedb0 0x7fc2040cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:46.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.832+0000 7fc20c800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2040cc700 con 0x7fc2040d2710 2024-02-19T15:25:46.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.832+0000 7fc20c800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2040cf5e0 con 0x7fc20415bcd0 2024-02-19T15:25:46.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.832+0000 7fc20c800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2040cf300 con 0x7fc2040cedb0 2024-02-19T15:25:46.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.832+0000 7fc209e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc20415bcd0 0x7fc2040cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:46.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.832+0000 7fc209e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc20415bcd0 0x7fc2040cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48390/0 (socket says 172.21.15.47:48390) 2024-02-19T15:25:46.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.832+0000 7fc209e00640 1 -- 172.21.15.47:0/1664143303 learned_addr learned my addr 172.21.15.47:0/1664143303 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:46.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.832+0000 7fc209400640 1 --2- 172.21.15.47:0/1664143303 >> v2:172.21.15.116:3300/0 conn(0x7fc2040cedb0 0x7fc2040cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:46.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.836+0000 7fc209e00640 1 -- 172.21.15.47:0/1664143303 >> v2:172.21.15.116:3300/0 conn(0x7fc2040cedb0 msgr2=0x7fc2040cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:46.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.836+0000 7fc209e00640 1 --2- 172.21.15.47:0/1664143303 >> v2:172.21.15.116:3300/0 conn(0x7fc2040cedb0 0x7fc2040cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.836+0000 7fc209e00640 1 -- 172.21.15.47:0/1664143303 >> v1:172.21.15.116:6789/0 conn(0x7fc2040d2710 legacy=0x7fc20415d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:46.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.836+0000 7fc209e00640 1 -- 172.21.15.47:0/1664143303 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2040cc9a0 con 0x7fc20415bcd0 2024-02-19T15:25:46.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.836+0000 7fc209e00640 1 --2- 172.21.15.47:0/1664143303 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc20415bcd0 0x7fc2040cd480 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fc2040ce100 tx=0x7fc1f801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8541d01332ba5bc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:46.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.836+0000 7fc208a00640 1 -- 172.21.15.47:0/1664143303 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f80ab600 con 0x7fc20415bcd0 2024-02-19T15:25:46.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.836+0000 7fc208a00640 1 -- 172.21.15.47:0/1664143303 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f80ab760 con 0x7fc20415bcd0 2024-02-19T15:25:46.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.836+0000 7fc208a00640 1 -- 172.21.15.47:0/1664143303 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f80abaf0 con 0x7fc20415bcd0 2024-02-19T15:25:46.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.836+0000 7fc20c800640 1 -- 172.21.15.47:0/1664143303 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc20415bcd0 msgr2=0x7fc2040cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:46.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.836+0000 7fc20c800640 1 --2- 172.21.15.47:0/1664143303 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc20415bcd0 0x7fc2040cd480 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fc2040ce100 tx=0x7fc1f801c4f0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.836+0000 7fc20c800640 1 -- 172.21.15.47:0/1664143303 shutdown_connections 2024-02-19T15:25:46.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.836+0000 7fc20c800640 1 --2- 172.21.15.47:0/1664143303 >> v2:172.21.15.116:3300/0 conn(0x7fc2040cedb0 0x7fc2040cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.836+0000 7fc20c800640 1 --2- 172.21.15.47:0/1664143303 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc20415bcd0 0x7fc2040cd480 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.836+0000 7fc20c800640 1 -- 172.21.15.47:0/1664143303 >> 172.21.15.47:0/1664143303 conn(0x7fc2040d1bf0 msgr2=0x7fc2040d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:46.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc20c800640 1 -- 172.21.15.47:0/1664143303 shutdown_connections 2024-02-19T15:25:46.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc20c800640 1 -- 172.21.15.47:0/1664143303 wait complete. 2024-02-19T15:25:46.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc20c800640 1 Processor -- start 2024-02-19T15:25:46.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc20c800640 1 -- start start 2024-02-19T15:25:46.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc20c800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2040cedb0 0x7fc2040cab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:46.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc20c800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2040d2710 0x7fc20406bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:46.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc20c800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc20415bcd0 0x7fc2040cb600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:46.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc20c800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20406d3c0 con 0x7fc20415bcd0 2024-02-19T15:25:46.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc20c800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20406c500 con 0x7fc2040d2710 2024-02-19T15:25:46.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc20c800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20406c7b0 con 0x7fc2040cedb0 2024-02-19T15:25:46.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc20a800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc20415bcd0 0x7fc2040cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:46.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc20a800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc20415bcd0 0x7fc2040cb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48396/0 (socket says 172.21.15.47:48396) 2024-02-19T15:25:46.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc209e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2040cedb0 0x7fc2040cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:46.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc20a800640 1 -- 172.21.15.47:0/757599060 learned_addr learned my addr 172.21.15.47:0/757599060 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:46.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc209400640 1 --2- 172.21.15.47:0/757599060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2040d2710 0x7fc20406bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:46.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc20a800640 1 -- 172.21.15.47:0/757599060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2040cedb0 msgr2=0x7fc2040cab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:46.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc20a800640 1 --2- 172.21.15.47:0/757599060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2040cedb0 0x7fc2040cab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc20a800640 1 -- 172.21.15.47:0/757599060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2040d2710 msgr2=0x7fc20406bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:46.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc20a800640 1 --2- 172.21.15.47:0/757599060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2040d2710 0x7fc20406bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:46.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc20a800640 1 -- 172.21.15.47:0/757599060 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f80094d0 con 0x7fc20415bcd0 2024-02-19T15:25:46.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.840+0000 7fc20a800640 1 --2- 172.21.15.47:0/757599060 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc20415bcd0 0x7fc2040cb600 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fc20000c730 tx=0x7fc20000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:46.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.844+0000 7fc1fe000640 1 -- 172.21.15.47:0/757599060 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc200003b80 con 0x7fc20415bcd0 2024-02-19T15:25:46.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.844+0000 7fc1fe000640 1 -- 172.21.15.47:0/757599060 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc200003ce0 con 0x7fc20415bcd0 2024-02-19T15:25:46.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.844+0000 7fc1fe000640 1 -- 172.21.15.47:0/757599060 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20000aa30 con 0x7fc20415bcd0 2024-02-19T15:25:46.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.844+0000 7fc20c800640 1 -- 172.21.15.47:0/757599060 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2040cdb40 con 0x7fc20415bcd0 2024-02-19T15:25:46.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.844+0000 7fc20c800640 1 -- 172.21.15.47:0/757599060 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc20406cd70 con 0x7fc20415bcd0 2024-02-19T15:25:46.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.844+0000 7fc20c800640 1 -- 172.21.15.47:0/757599060 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc20406d530 con 0x7fc20415bcd0 2024-02-19T15:25:46.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.844+0000 7fc1fe000640 1 -- 172.21.15.47:0/757599060 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc20000ab90 con 0x7fc20415bcd0 2024-02-19T15:25:46.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.844+0000 7fc1fe000640 1 --2- 172.21.15.47:0/757599060 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc1e8066e40 0x7fc1e8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:46.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.844+0000 7fc1fe000640 1 -- 172.21.15.47:0/757599060 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc2000847a0 con 0x7fc20415bcd0 2024-02-19T15:25:46.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.844+0000 7fc209e00640 1 --2- 172.21.15.47:0/757599060 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc1e8066e40 0x7fc1e8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:46.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.844+0000 7fc209e00640 1 --2- 172.21.15.47:0/757599060 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc1e8066e40 0x7fc1e8069300 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc1f80af000 tx=0x7fc1f80b6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:46.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:46.848+0000 7fc1fe000640 1 -- 172.21.15.47:0/757599060 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc200051560 con 0x7fc20415bcd0 2024-02-19T15:25:47.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.056+0000 7fc20c800640 1 -- 172.21.15.47:0/757599060 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc204075360 con 0x7fc20415bcd0 2024-02-19T15:25:47.062 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-02-19T15:25:47.056+0000 7fc100600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T15:25:47.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.060+0000 7fc1fe000640 1 -- 172.21.15.47:0/757599060 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v41) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc200054c20 con 0x7fc20415bcd0 2024-02-19T15:25:47.064 INFO:teuthology.orchestra.run.smithi047.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T15:25:47.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.064+0000 7fc20c800640 1 -- 172.21.15.47:0/757599060 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc1e8066e40 msgr2=0x7fc1e8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:47.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.064+0000 7fc20c800640 1 --2- 172.21.15.47:0/757599060 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc1e8066e40 0x7fc1e8069300 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc1f80af000 tx=0x7fc1f80b6040 comp rx=0 tx=0).stop 2024-02-19T15:25:47.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.064+0000 7fc20c800640 1 -- 172.21.15.47:0/757599060 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc20415bcd0 msgr2=0x7fc2040cb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:47.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.064+0000 7fc20c800640 1 --2- 172.21.15.47:0/757599060 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc20415bcd0 0x7fc2040cb600 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fc20000c730 tx=0x7fc20000cbf0 comp rx=0 tx=0).stop 2024-02-19T15:25:47.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.064+0000 7fc20c800640 1 -- 172.21.15.47:0/757599060 shutdown_connections 2024-02-19T15:25:47.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.064+0000 7fc20c800640 1 --2- 172.21.15.47:0/757599060 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc1e8066e40 0x7fc1e8069300 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:47.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.064+0000 7fc20c800640 1 --2- 172.21.15.47:0/757599060 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc20415bcd0 0x7fc2040cb600 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:47.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.064+0000 7fc20c800640 1 --2- 172.21.15.47:0/757599060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2040d2710 0x7fc20406bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:47.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.064+0000 7fc20c800640 1 --2- 172.21.15.47:0/757599060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2040cedb0 0x7fc2040cab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:47.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.064+0000 7fc20c800640 1 -- 172.21.15.47:0/757599060 >> 172.21.15.47:0/757599060 conn(0x7fc2040d1bf0 msgr2=0x7fc204158910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:47.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.064+0000 7fc20c800640 1 -- 172.21.15.47:0/757599060 shutdown_connections 2024-02-19T15:25:47.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.064+0000 7fc20c800640 1 -- 172.21.15.47:0/757599060 wait complete. 2024-02-19T15:25:47.090 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T15:25:47.131 INFO:tasks.ceph.mds.c.smithi047.stdout:starting mds.c at 2024-02-19T15:25:47.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.228+0000 7f0e99800640 1 Processor -- start 2024-02-19T15:25:47.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.228+0000 7f0e99800640 1 -- start start 2024-02-19T15:25:47.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e99800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0e9415e210 0x7f0e940ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:47.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e99800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0e940cc180 0x7f0e940ce8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:47.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e99800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e940d2650 0x7f0e940d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:47.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e99800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e940cfe00 con 0x7f0e940cc180 2024-02-19T15:25:47.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e99800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e940cff90 con 0x7f0e9415e210 2024-02-19T15:25:47.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e99800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e940b2fd0 con 0x7f0e940d2650 2024-02-19T15:25:47.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e8a000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0e940cc180 0x7f0e940ce8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:47.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e8a000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0e940cc180 0x7f0e940ce8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48410/0 (socket says 172.21.15.47:48410) 2024-02-19T15:25:47.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e8a000640 1 -- 172.21.15.47:0/3808206670 learned_addr learned my addr 172.21.15.47:0/3808206670 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:47.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e93400640 1 --2- 172.21.15.47:0/3808206670 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e940d2650 0x7f0e940d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:47.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e92a00640 1 --2- 172.21.15.47:0/3808206670 >> v2:172.21.15.116:3300/0 conn(0x7f0e9415e210 0x7f0e940ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:47.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e8a000640 1 -- 172.21.15.47:0/3808206670 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e940d2650 msgr2=0x7f0e940d03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:47.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e8a000640 1 --2- 172.21.15.47:0/3808206670 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e940d2650 0x7f0e940d03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:47.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e8a000640 1 -- 172.21.15.47:0/3808206670 >> v2:172.21.15.116:3300/0 conn(0x7f0e9415e210 msgr2=0x7f0e940ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:47.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e8a000640 1 --2- 172.21.15.47:0/3808206670 >> v2:172.21.15.116:3300/0 conn(0x7f0e9415e210 0x7f0e940ce3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:47.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e8a000640 1 -- 172.21.15.47:0/3808206670 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e940b2cf0 con 0x7f0e940cc180 2024-02-19T15:25:47.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e93400640 1 --2- 172.21.15.47:0/3808206670 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e940d2650 0x7f0e940d03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:47.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e8a000640 1 --2- 172.21.15.47:0/3808206670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e940cc180 0x7f0e940ce8f0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f0e94071e50 tx=0x7f0e7c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae514563fa3f7ff server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:47.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e92000640 1 -- 172.21.15.47:0/3808206670 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c0aa610 con 0x7f0e940cc180 2024-02-19T15:25:47.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e92000640 1 -- 172.21.15.47:0/3808206670 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e7c0a6040 con 0x7f0e940cc180 2024-02-19T15:25:47.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.232+0000 7f0e92000640 1 -- 172.21.15.47:0/3808206670 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c0aa990 con 0x7f0e940cc180 2024-02-19T15:25:47.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e99800640 1 -- 172.21.15.47:0/3808206670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e940cc180 msgr2=0x7f0e940ce8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:47.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e99800640 1 --2- 172.21.15.47:0/3808206670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e940cc180 0x7f0e940ce8f0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f0e94071e50 tx=0x7f0e7c01c4f0 comp rx=0 tx=0).stop 2024-02-19T15:25:47.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e99800640 1 -- 172.21.15.47:0/3808206670 shutdown_connections 2024-02-19T15:25:47.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e99800640 1 --2- 172.21.15.47:0/3808206670 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e940d2650 0x7f0e940d03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:47.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e99800640 1 --2- 172.21.15.47:0/3808206670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e940cc180 0x7f0e940ce8f0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:47.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e99800640 1 --2- 172.21.15.47:0/3808206670 >> v2:172.21.15.116:3300/0 conn(0x7f0e9415e210 0x7f0e940ce3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:47.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e99800640 1 -- 172.21.15.47:0/3808206670 >> 172.21.15.47:0/3808206670 conn(0x7f0e940d1b30 msgr2=0x7f0e940d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:47.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e99800640 1 -- 172.21.15.47:0/3808206670 shutdown_connections 2024-02-19T15:25:47.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e99800640 1 -- 172.21.15.47:0/3808206670 wait complete. 2024-02-19T15:25:47.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e99800640 1 Processor -- start 2024-02-19T15:25:47.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e99800640 1 -- start start 2024-02-19T15:25:47.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e99800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e940cc180 0x7f0e940c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:47.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e99800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e940d2650 0x7f0e941557f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:47.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e99800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e9415e210 0x7f0e9414df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:47.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e99800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9414fc80 con 0x7f0e9415e210 2024-02-19T15:25:47.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e99800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9414f000 con 0x7f0e940d2650 2024-02-19T15:25:47.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e99800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9414f2e0 con 0x7f0e940cc180 2024-02-19T15:25:47.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e93400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e9415e210 0x7f0e9414df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:47.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e93400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e9415e210 0x7f0e9414df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48422/0 (socket says 172.21.15.47:48422) 2024-02-19T15:25:47.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e93400640 1 -- 172.21.15.47:0/2317108675 learned_addr learned my addr 172.21.15.47:0/2317108675 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:47.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e92a00640 1 --2- 172.21.15.47:0/2317108675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e940cc180 0x7f0e940c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:47.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e8a000640 1 --2- 172.21.15.47:0/2317108675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e940d2650 0x7f0e941557f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:47.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e93400640 1 -- 172.21.15.47:0/2317108675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e940cc180 msgr2=0x7f0e940c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:47.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e93400640 1 --2- 172.21.15.47:0/2317108675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e940cc180 0x7f0e940c67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:47.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e93400640 1 -- 172.21.15.47:0/2317108675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e940d2650 msgr2=0x7f0e941557f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:47.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e93400640 1 --2- 172.21.15.47:0/2317108675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e940d2650 0x7f0e941557f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:47.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e93400640 1 -- 172.21.15.47:0/2317108675 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7c0094d0 con 0x7f0e9415e210 2024-02-19T15:25:47.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.236+0000 7f0e92a00640 1 --2- 172.21.15.47:0/2317108675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e940cc180 0x7f0e940c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:47.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.240+0000 7f0e93400640 1 --2- 172.21.15.47:0/2317108675 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e9415e210 0x7f0e9414df40 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c007ef0 tx=0x7f0e8c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:47.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.240+0000 7f0e8b400640 1 -- 172.21.15.47:0/2317108675 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c003ba0 con 0x7f0e9415e210 2024-02-19T15:25:47.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.240+0000 7f0e8b400640 1 -- 172.21.15.47:0/2317108675 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e8c00f040 con 0x7f0e9415e210 2024-02-19T15:25:47.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.240+0000 7f0e99800640 1 -- 172.21.15.47:0/2317108675 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e9414f5c0 con 0x7f0e9415e210 2024-02-19T15:25:47.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.240+0000 7f0e8b400640 1 -- 172.21.15.47:0/2317108675 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c00de80 con 0x7f0e9415e210 2024-02-19T15:25:47.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.240+0000 7f0e99800640 1 -- 172.21.15.47:0/2317108675 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e941507f0 con 0x7f0e9415e210 2024-02-19T15:25:47.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.240+0000 7f0e99800640 1 -- 172.21.15.47:0/2317108675 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e9414fdc0 con 0x7f0e9415e210 2024-02-19T15:25:47.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.240+0000 7f0e8b400640 1 -- 172.21.15.47:0/2317108675 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0e8c004030 con 0x7f0e9415e210 2024-02-19T15:25:47.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.240+0000 7f0e8b400640 1 --2- 172.21.15.47:0/2317108675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0e70066e70 0x7f0e70069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:47.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.240+0000 7f0e8b400640 1 -- 172.21.15.47:0/2317108675 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f0e8c084e40 con 0x7f0e9415e210 2024-02-19T15:25:47.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.240+0000 7f0e92a00640 1 --2- 172.21.15.47:0/2317108675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0e70066e70 0x7f0e70069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:47.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.244+0000 7f0e92a00640 1 --2- 172.21.15.47:0/2317108675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0e70066e70 0x7f0e70069330 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0e8400b970 tx=0x7f0e84008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:47.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.244+0000 7f0e8b400640 1 -- 172.21.15.47:0/2317108675 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0e9414fdc0 con 0x7f0e9415e210 2024-02-19T15:25:47.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:47.440+0000 7f0e99800640 1 -- 172.21.15.47:0/2317108675 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0e9414fdc0 con 0x7f0e9415e210 2024-02-19T15:25:48.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.068+0000 7f0e8b400640 1 -- 172.21.15.47:0/2317108675 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v42) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0e9414fdc0 con 0x7f0e9415e210 2024-02-19T15:25:48.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.068+0000 7f0e99800640 1 -- 172.21.15.47:0/2317108675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0e70066e70 msgr2=0x7f0e70069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:48.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.068+0000 7f0e99800640 1 --2- 172.21.15.47:0/2317108675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0e70066e70 0x7f0e70069330 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0e8400b970 tx=0x7f0e84008040 comp rx=0 tx=0).stop 2024-02-19T15:25:48.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.068+0000 7f0e99800640 1 -- 172.21.15.47:0/2317108675 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e9415e210 msgr2=0x7f0e9414df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:48.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.068+0000 7f0e99800640 1 --2- 172.21.15.47:0/2317108675 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e9415e210 0x7f0e9414df40 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c007ef0 tx=0x7f0e8c00d4a0 comp rx=0 tx=0).stop 2024-02-19T15:25:48.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.068+0000 7f0e99800640 1 -- 172.21.15.47:0/2317108675 shutdown_connections 2024-02-19T15:25:48.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.068+0000 7f0e99800640 1 --2- 172.21.15.47:0/2317108675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0e70066e70 0x7f0e70069330 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:48.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.068+0000 7f0e99800640 1 --2- 172.21.15.47:0/2317108675 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0e9415e210 0x7f0e9414df40 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:48.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.068+0000 7f0e99800640 1 --2- 172.21.15.47:0/2317108675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e940d2650 0x7f0e941557f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:48.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.068+0000 7f0e99800640 1 --2- 172.21.15.47:0/2317108675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e940cc180 0x7f0e940c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:48.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.068+0000 7f0e99800640 1 -- 172.21.15.47:0/2317108675 >> 172.21.15.47:0/2317108675 conn(0x7f0e940d1b30 msgr2=0x7f0e9406eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:48.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.068+0000 7f0e99800640 1 -- 172.21.15.47:0/2317108675 shutdown_connections 2024-02-19T15:25:48.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.068+0000 7f0e99800640 1 -- 172.21.15.47:0/2317108675 wait complete. 2024-02-19T15:25:48.096 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T15:25:48.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.284+0000 7f87d8400640 1 Processor -- start 2024-02-19T15:25:48.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.284+0000 7f87d8400640 1 -- start start 2024-02-19T15:25:48.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.284+0000 7f87d8400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f87d006c4c0 0x7f87d006e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:48.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.284+0000 7f87d8400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87d006c920 0x7f87d006e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:48.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.284+0000 7f87d8400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d00b5650 con 0x7f87d00b5270 2024-02-19T15:25:48.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.284+0000 7f87d8400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87d015f7b0 con 0x7f87d006c4c0 2024-02-19T15:25:48.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.284+0000 7f87d8400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d0071560 con 0x7f87d006c920 2024-02-19T15:25:48.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.284+0000 7f87d5a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f87d00b5270 0x7f87d00719f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52926/0 (socket says 172.21.15.47:52926) 2024-02-19T15:25:48.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.284+0000 7f87d5a00640 1 -- 172.21.15.47:0/55272553 learned_addr learned my addr 172.21.15.47:0/55272553 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:48.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.284+0000 7f87d6400640 1 --2- 172.21.15.47:0/55272553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87d006c920 0x7f87d006e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:48.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.284+0000 7f87d5000640 1 --2- 172.21.15.47:0/55272553 >> v2:172.21.15.116:3300/0 conn(0x7f87d006c4c0 0x7f87d006e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:48.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.284+0000 7f87cfe00640 1 -- 172.21.15.47:0/55272553 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995434821 0 0) 0x7f87d00b5650 con 0x7f87d00b5270 2024-02-19T15:25:48.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.284+0000 7f87cfe00640 1 -- 172.21.15.47:0/55272553 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b8003530 con 0x7f87d00b5270 2024-02-19T15:25:48.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.284+0000 7f87d6400640 1 -- 172.21.15.47:0/55272553 >> v2:172.21.15.116:3300/0 conn(0x7f87d006c4c0 msgr2=0x7f87d006e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:48.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.284+0000 7f87d6400640 1 --2- 172.21.15.47:0/55272553 >> v2:172.21.15.116:3300/0 conn(0x7f87d006c4c0 0x7f87d006e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:48.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.284+0000 7f87d6400640 1 -- 172.21.15.47:0/55272553 >> v1:172.21.15.47:6789/0 conn(0x7f87d00b5270 legacy=0x7f87d00719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:48.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.284+0000 7f87d6400640 1 -- 172.21.15.47:0/55272553 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d006eaa0 con 0x7f87d006c920 2024-02-19T15:25:48.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.288+0000 7f87d6400640 1 --2- 172.21.15.47:0/55272553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87d006c920 0x7f87d006e560 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f87c8009d90 tx=0x7f87c801c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90677617af5888f5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:48.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.288+0000 7f87cfe00640 1 -- 172.21.15.47:0/55272553 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c80a5920 con 0x7f87d006c920 2024-02-19T15:25:48.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.288+0000 7f87cfe00640 1 -- 172.21.15.47:0/55272553 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c80a5a80 con 0x7f87d006c920 2024-02-19T15:25:48.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.288+0000 7f87cfe00640 1 -- 172.21.15.47:0/55272553 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c80a5e10 con 0x7f87d006c920 2024-02-19T15:25:48.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.288+0000 7f87d8400640 1 -- 172.21.15.47:0/55272553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87d006c920 msgr2=0x7f87d006e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:48.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.288+0000 7f87d8400640 1 --2- 172.21.15.47:0/55272553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87d006c920 0x7f87d006e560 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f87c8009d90 tx=0x7f87c801c970 comp rx=0 tx=0).stop 2024-02-19T15:25:48.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.288+0000 7f87d8400640 1 -- 172.21.15.47:0/55272553 shutdown_connections 2024-02-19T15:25:48.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.288+0000 7f87d8400640 1 --2- 172.21.15.47:0/55272553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87d006c920 0x7f87d006e560 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:48.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.288+0000 7f87d8400640 1 --2- 172.21.15.47:0/55272553 >> v2:172.21.15.116:3300/0 conn(0x7f87d006c4c0 0x7f87d006e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:48.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.288+0000 7f87d8400640 1 -- 172.21.15.47:0/55272553 >> 172.21.15.47:0/55272553 conn(0x7f87d00cc3f0 msgr2=0x7f87d00cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:48.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.288+0000 7f87d8400640 1 -- 172.21.15.47:0/55272553 shutdown_connections 2024-02-19T15:25:48.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.288+0000 7f87d8400640 1 -- 172.21.15.47:0/55272553 wait complete. 2024-02-19T15:25:48.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d8400640 1 Processor -- start 2024-02-19T15:25:48.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d8400640 1 -- start start 2024-02-19T15:25:48.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d8400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87d006c4c0 0x7f87d0143b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d8400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d006c920 0x7f87d015ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d8400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87d00b5270 0x7f87d0158f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d8400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d00b3770 con 0x7f87d006c4c0 2024-02-19T15:25:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d8400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d00b3a50 con 0x7f87d006c920 2024-02-19T15:25:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d8400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d00b3d30 con 0x7f87d00b5270 2024-02-19T15:25:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d5a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87d006c4c0 0x7f87d0143b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d5a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87d006c4c0 0x7f87d0143b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48424/0 (socket says 172.21.15.47:48424) 2024-02-19T15:25:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d5a00640 1 -- 172.21.15.47:0/1421749784 learned_addr learned my addr 172.21.15.47:0/1421749784 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d6400640 1 --2- 172.21.15.47:0/1421749784 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87d00b5270 0x7f87d0158f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d5000640 1 --2- 172.21.15.47:0/1421749784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d006c920 0x7f87d015ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d5a00640 1 -- 172.21.15.47:0/1421749784 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87d00b5270 msgr2=0x7f87d0158f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:48.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d5a00640 1 --2- 172.21.15.47:0/1421749784 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87d00b5270 0x7f87d0158f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:48.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d5a00640 1 -- 172.21.15.47:0/1421749784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d006c920 msgr2=0x7f87d015ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:48.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d5a00640 1 --2- 172.21.15.47:0/1421749784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d006c920 0x7f87d015ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:48.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d5a00640 1 -- 172.21.15.47:0/1421749784 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c8009a70 con 0x7f87d006c4c0 2024-02-19T15:25:48.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d5a00640 1 --2- 172.21.15.47:0/1421749784 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87d006c4c0 0x7f87d0143b10 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f87c400c450 tx=0x7f87c400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:48.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87cd600640 1 -- 172.21.15.47:0/1421749784 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c4005cf0 con 0x7f87d006c4c0 2024-02-19T15:25:48.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87cd600640 1 -- 172.21.15.47:0/1421749784 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c4005e50 con 0x7f87d006c4c0 2024-02-19T15:25:48.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87cd600640 1 -- 172.21.15.47:0/1421749784 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c4007290 con 0x7f87d006c4c0 2024-02-19T15:25:48.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d8400640 1 -- 172.21.15.47:0/1421749784 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d00b4010 con 0x7f87d006c4c0 2024-02-19T15:25:48.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d8400640 1 -- 172.21.15.47:0/1421749784 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d00b33d0 con 0x7f87d006c4c0 2024-02-19T15:25:48.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.292+0000 7f87d8400640 1 -- 172.21.15.47:0/1421749784 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d00b4530 con 0x7f87d006c4c0 2024-02-19T15:25:48.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.296+0000 7f87cd600640 1 -- 172.21.15.47:0/1421749784 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f87c40073f0 con 0x7f87d006c4c0 2024-02-19T15:25:48.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.296+0000 7f87cd600640 1 --2- 172.21.15.47:0/1421749784 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f87b0066e40 0x7f87b0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:48.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.296+0000 7f87cd600640 1 -- 172.21.15.47:0/1421749784 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f87c4084070 con 0x7f87d006c4c0 2024-02-19T15:25:48.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.296+0000 7f87d5000640 1 --2- 172.21.15.47:0/1421749784 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f87b0066e40 0x7f87b0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:48.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.296+0000 7f87d5000640 1 --2- 172.21.15.47:0/1421749784 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f87b0066e40 0x7f87b0069300 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f87d015d560 tx=0x7f87bc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:48.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.300+0000 7f87cd600640 1 -- 172.21.15.47:0/1421749784 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f87c40b61c0 con 0x7f87d006c4c0 2024-02-19T15:25:48.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:48.464+0000 7f87d8400640 1 -- 172.21.15.47:0/1421749784 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f87d0075360 con 0x7f87d006c4c0 2024-02-19T15:25:49.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.068+0000 7f87cd600640 1 -- 172.21.15.47:0/1421749784 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v73) v1 ==== 181+0+0 (secure 0 0 0) 0x7f87c4017070 con 0x7f87d006c4c0 2024-02-19T15:25:49.072 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_metadata' removed 2024-02-19T15:25:49.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.072+0000 7f87d8400640 1 -- 172.21.15.47:0/1421749784 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f87b0066e40 msgr2=0x7f87b0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:49.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.072+0000 7f87d8400640 1 --2- 172.21.15.47:0/1421749784 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f87b0066e40 0x7f87b0069300 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f87d015d560 tx=0x7f87bc009190 comp rx=0 tx=0).stop 2024-02-19T15:25:49.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.072+0000 7f87d8400640 1 -- 172.21.15.47:0/1421749784 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87d006c4c0 msgr2=0x7f87d0143b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:49.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.072+0000 7f87d8400640 1 --2- 172.21.15.47:0/1421749784 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87d006c4c0 0x7f87d0143b10 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f87c400c450 tx=0x7f87c400c910 comp rx=0 tx=0).stop 2024-02-19T15:25:49.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.072+0000 7f87d8400640 1 -- 172.21.15.47:0/1421749784 shutdown_connections 2024-02-19T15:25:49.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.072+0000 7f87d8400640 1 --2- 172.21.15.47:0/1421749784 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f87b0066e40 0x7f87b0069300 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:49.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.072+0000 7f87d8400640 1 --2- 172.21.15.47:0/1421749784 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87d00b5270 0x7f87d0158f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:49.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.072+0000 7f87d8400640 1 --2- 172.21.15.47:0/1421749784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d006c920 0x7f87d015ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:49.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.072+0000 7f87d8400640 1 --2- 172.21.15.47:0/1421749784 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87d006c4c0 0x7f87d0143b10 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:49.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.072+0000 7f87d8400640 1 -- 172.21.15.47:0/1421749784 >> 172.21.15.47:0/1421749784 conn(0x7f87d00cc3f0 msgr2=0x7f87d015ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:49.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.072+0000 7f87d8400640 1 -- 172.21.15.47:0/1421749784 shutdown_connections 2024-02-19T15:25:49.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.072+0000 7f87d8400640 1 -- 172.21.15.47:0/1421749784 wait complete. 2024-02-19T15:25:49.100 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T15:25:49.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1679400640 1 Processor -- start 2024-02-19T15:25:49.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1679400640 1 -- start start 2024-02-19T15:25:49.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1679400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16740cec80 0x7f16740d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:49.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1679400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f167415f190 0x7f167406cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:49.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1679400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16740d0a50 con 0x7f167415f8f0 2024-02-19T15:25:49.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1679400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f167406d470 con 0x7f167415f190 2024-02-19T15:25:49.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1679400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16740cfe80 con 0x7f16740cec80 2024-02-19T15:25:49.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1673200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f167415f8f0 0x7f167415fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52934/0 (socket says 172.21.15.47:52934) 2024-02-19T15:25:49.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1673200640 1 -- 172.21.15.47:0/1841886873 learned_addr learned my addr 172.21.15.47:0/1841886873 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:49.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1672800640 1 --2- 172.21.15.47:0/1841886873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16740cec80 0x7f16740d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:49.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f166be00640 1 --2- 172.21.15.47:0/1841886873 >> v2:172.21.15.116:3300/0 conn(0x7f167415f190 0x7f167406cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:49.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1671e00640 1 -- 172.21.15.47:0/1841886873 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652761910 0 0) 0x7f16740d0a50 con 0x7f167415f8f0 2024-02-19T15:25:49.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1671e00640 1 -- 172.21.15.47:0/1841886873 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1658003530 con 0x7f167415f8f0 2024-02-19T15:25:49.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1671e00640 1 -- 172.21.15.47:0/1841886873 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3868525950 0 0) 0x7f1658003530 con 0x7f167415f8f0 2024-02-19T15:25:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1671e00640 1 -- 172.21.15.47:0/1841886873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16740cec80 msgr2=0x7f16740d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1671e00640 1 --2- 172.21.15.47:0/1841886873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16740cec80 0x7f16740d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1671e00640 1 -- 172.21.15.47:0/1841886873 >> v2:172.21.15.116:3300/0 conn(0x7f167415f190 msgr2=0x7f167406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1671e00640 1 --2- 172.21.15.47:0/1841886873 >> v2:172.21.15.116:3300/0 conn(0x7f167415f190 0x7f167406cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1671e00640 1 -- 172.21.15.47:0/1841886873 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16740d0120 con 0x7f167415f8f0 2024-02-19T15:25:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1671e00640 1 -- 172.21.15.47:0/1841886873 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f166c001a00 con 0x7f167415f8f0 2024-02-19T15:25:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1671e00640 1 -- 172.21.15.47:0/1841886873 >> v1:172.21.15.47:6789/0 conn(0x7f167415f8f0 legacy=0x7f167415fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1671e00640 1 --2- 172.21.15.47:0/1841886873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1658003980 0x7f1658003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1671e00640 1 --2- 172.21.15.47:0/1841886873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16580047d0 0x7f1658004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1671e00640 1 --2- 172.21.15.47:0/1841886873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16740cec80 0x7f1658005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1671e00640 1 -- 172.21.15.47:0/1841886873 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167406d470 con 0x7f1658003980 2024-02-19T15:25:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1671e00640 1 -- 172.21.15.47:0/1841886873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16740cfe80 con 0x7f16580047d0 2024-02-19T15:25:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1671e00640 1 -- 172.21.15.47:0/1841886873 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f165800b990 con 0x7f16740cec80 2024-02-19T15:25:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1672800640 1 --2- 172.21.15.47:0/1841886873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1658003980 0x7f1658003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f166be00640 1 --2- 172.21.15.47:0/1841886873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16580047d0 0x7f1658004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1673200640 1 --2- 172.21.15.47:0/1841886873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16740cec80 0x7f1658005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:49.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1672800640 1 -- 172.21.15.47:0/1841886873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16740cec80 msgr2=0x7f1658005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:49.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1672800640 1 --2- 172.21.15.47:0/1841886873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16740cec80 0x7f1658005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:49.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1672800640 1 -- 172.21.15.47:0/1841886873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16580047d0 msgr2=0x7f1658004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:49.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1672800640 1 --2- 172.21.15.47:0/1841886873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16580047d0 0x7f1658004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:49.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.244+0000 7f1672800640 1 -- 172.21.15.47:0/1841886873 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1658003530 con 0x7f1658003980 2024-02-19T15:25:49.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.248+0000 7f1672800640 1 --2- 172.21.15.47:0/1841886873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1658003980 0x7f1658003da0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f1664009ae0 tx=0x7f166401ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9724540f47b02ca server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:49.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.248+0000 7f1671e00640 1 -- 172.21.15.47:0/1841886873 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16640a6450 con 0x7f1658003980 2024-02-19T15:25:49.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.248+0000 7f1671e00640 1 -- 172.21.15.47:0/1841886873 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16640a65b0 con 0x7f1658003980 2024-02-19T15:25:49.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.248+0000 7f1671e00640 1 -- 172.21.15.47:0/1841886873 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16640a6960 con 0x7f1658003980 2024-02-19T15:25:49.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.248+0000 7f1679400640 1 -- 172.21.15.47:0/1841886873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1658003980 msgr2=0x7f1658003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:49.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.248+0000 7f1679400640 1 --2- 172.21.15.47:0/1841886873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1658003980 0x7f1658003da0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f1664009ae0 tx=0x7f166401ca40 comp rx=0 tx=0).stop 2024-02-19T15:25:49.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.248+0000 7f1673200640 1 -- 172.21.15.47:0/1841886873 reap_dead start 2024-02-19T15:25:49.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.248+0000 7f1679400640 1 -- 172.21.15.47:0/1841886873 shutdown_connections 2024-02-19T15:25:49.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.248+0000 7f1679400640 1 -- 172.21.15.47:0/1841886873 >> 172.21.15.47:0/1841886873 conn(0x7f167409aed0 msgr2=0x7f16740d1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:49.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.248+0000 7f1679400640 1 -- 172.21.15.47:0/1841886873 shutdown_connections 2024-02-19T15:25:49.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.248+0000 7f1679400640 1 -- 172.21.15.47:0/1841886873 wait complete. 2024-02-19T15:25:49.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f1679400640 1 Processor -- start 2024-02-19T15:25:49.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f1679400640 1 -- start start 2024-02-19T15:25:49.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f1679400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1658003980 0x7f16740c5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:49.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f1679400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f167415cf50 0x7f16741ecfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:49.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f1679400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f167415c310 0x7f167415c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:49.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f1679400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16740c9f00 con 0x7f167415cf50 2024-02-19T15:25:49.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f1679400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16740c6860 con 0x7f167415c310 2024-02-19T15:25:49.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f1679400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16740c6050 con 0x7f1658003980 2024-02-19T15:25:49.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f166be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f167415cf50 0x7f16741ecfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:49.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f166be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f167415cf50 0x7f16741ecfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48440/0 (socket says 172.21.15.47:48440) 2024-02-19T15:25:49.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f166be00640 1 -- 172.21.15.47:0/3406629148 learned_addr learned my addr 172.21.15.47:0/3406629148 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:49.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f1672800640 1 --2- 172.21.15.47:0/3406629148 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1658003980 0x7f16740c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:49.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f1673200640 1 --2- 172.21.15.47:0/3406629148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f167415c310 0x7f167415c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:49.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f166be00640 1 -- 172.21.15.47:0/3406629148 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1658003980 msgr2=0x7f16740c5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:49.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f166be00640 1 --2- 172.21.15.47:0/3406629148 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1658003980 0x7f16740c5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:49.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f166be00640 1 -- 172.21.15.47:0/3406629148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f167415c310 msgr2=0x7f167415c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:49.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f166be00640 1 --2- 172.21.15.47:0/3406629148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f167415c310 0x7f167415c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:49.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f166be00640 1 -- 172.21.15.47:0/3406629148 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16740c62a0 con 0x7f167415cf50 2024-02-19T15:25:49.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f1673200640 1 --2- 172.21.15.47:0/3406629148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f167415c310 0x7f167415c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:49.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f166be00640 1 --2- 172.21.15.47:0/3406629148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f167415cf50 0x7f16741ecfb0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f165c0119d0 tx=0x7f165c011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:49.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f166aa00640 1 -- 172.21.15.47:0/3406629148 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c00fb50 con 0x7f167415cf50 2024-02-19T15:25:49.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f166aa00640 1 -- 172.21.15.47:0/3406629148 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165c00fcb0 con 0x7f167415cf50 2024-02-19T15:25:49.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f1679400640 1 -- 172.21.15.47:0/3406629148 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16740c6580 con 0x7f167415cf50 2024-02-19T15:25:49.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f166aa00640 1 -- 172.21.15.47:0/3406629148 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c014440 con 0x7f167415cf50 2024-02-19T15:25:49.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.252+0000 7f1679400640 1 -- 172.21.15.47:0/3406629148 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f167415ccd0 con 0x7f167415cf50 2024-02-19T15:25:49.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.256+0000 7f1679400640 1 -- 172.21.15.47:0/3406629148 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1674075360 con 0x7f167415cf50 2024-02-19T15:25:49.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.256+0000 7f166aa00640 1 -- 172.21.15.47:0/3406629148 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f165c01d400 con 0x7f167415cf50 2024-02-19T15:25:49.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.256+0000 7f166aa00640 1 --2- 172.21.15.47:0/3406629148 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1650066e40 0x7f1650069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:49.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.256+0000 7f166aa00640 1 -- 172.21.15.47:0/3406629148 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f165c085070 con 0x7f167415cf50 2024-02-19T15:25:49.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.256+0000 7f1672800640 1 --2- 172.21.15.47:0/3406629148 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1650066e40 0x7f1650069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:49.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.256+0000 7f1672800640 1 --2- 172.21.15.47:0/3406629148 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1650066e40 0x7f1650069300 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f16640ad000 tx=0x7f16640b3040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:49.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.260+0000 7f166aa00640 1 -- 172.21.15.47:0/3406629148 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f165c0b71c0 con 0x7f167415cf50 2024-02-19T15:25:49.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:49.428+0000 7f1679400640 1 -- 172.21.15.47:0/3406629148 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f16740cf0c0 con 0x7f167415cf50 2024-02-19T15:25:50.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.072+0000 7f166aa00640 1 -- 172.21.15.47:0/3406629148 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v74) v1 ==== 169+0+0 (secure 0 0 0) 0x7f165c052030 con 0x7f167415cf50 2024-02-19T15:25:50.077 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_data' removed 2024-02-19T15:25:50.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.076+0000 7f1679400640 1 -- 172.21.15.47:0/3406629148 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1650066e40 msgr2=0x7f1650069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:50.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.076+0000 7f1679400640 1 --2- 172.21.15.47:0/3406629148 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1650066e40 0x7f1650069300 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f16640ad000 tx=0x7f16640b3040 comp rx=0 tx=0).stop 2024-02-19T15:25:50.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.076+0000 7f1679400640 1 -- 172.21.15.47:0/3406629148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f167415cf50 msgr2=0x7f16741ecfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:50.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.076+0000 7f1679400640 1 --2- 172.21.15.47:0/3406629148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f167415cf50 0x7f16741ecfb0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f165c0119d0 tx=0x7f165c011e90 comp rx=0 tx=0).stop 2024-02-19T15:25:50.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.080+0000 7f1679400640 1 -- 172.21.15.47:0/3406629148 shutdown_connections 2024-02-19T15:25:50.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.080+0000 7f1679400640 1 --2- 172.21.15.47:0/3406629148 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1650066e40 0x7f1650069300 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.080+0000 7f1679400640 1 --2- 172.21.15.47:0/3406629148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f167415c310 0x7f167415c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.080+0000 7f1679400640 1 --2- 172.21.15.47:0/3406629148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f167415cf50 0x7f16741ecfb0 secure :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f165c0119d0 tx=0x7f165c011e90 comp rx=0 tx=0).stop 2024-02-19T15:25:50.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.080+0000 7f1679400640 1 --2- 172.21.15.47:0/3406629148 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1658003980 0x7f16740c5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.080+0000 7f1679400640 1 -- 172.21.15.47:0/3406629148 >> 172.21.15.47:0/3406629148 conn(0x7f167409aed0 msgr2=0x7f1674162390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:50.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.080+0000 7f1679400640 1 -- 172.21.15.47:0/3406629148 shutdown_connections 2024-02-19T15:25:50.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.080+0000 7f1679400640 1 -- 172.21.15.47:0/3406629148 wait complete. 2024-02-19T15:25:50.108 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-19T15:25:50.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.252+0000 7f4353200640 1 Processor -- start 2024-02-19T15:25:50.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.252+0000 7f4353200640 1 -- start start 2024-02-19T15:25:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.252+0000 7f4353200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434c15bd90 0x7f434c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.252+0000 7f4353200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f434c0cb490 0x7f434c072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.252+0000 7f4353200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f434c072680 0x7f434c0ca270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.252+0000 7f4353200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f434c15fc90 con 0x7f434c0cb490 2024-02-19T15:25:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.252+0000 7f4353200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f434c15ff70 con 0x7f434c072680 2024-02-19T15:25:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.252+0000 7f4353200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434c160250 con 0x7f434c15bd90 2024-02-19T15:25:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.252+0000 7f4343e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f434c0cb490 0x7f434c072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.252+0000 7f4343e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f434c0cb490 0x7f434c072140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48442/0 (socket says 172.21.15.47:48442) 2024-02-19T15:25:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.252+0000 7f4343e00640 1 -- 172.21.15.47:0/3251980293 learned_addr learned my addr 172.21.15.47:0/3251980293 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.252+0000 7f4351200640 1 --2- 172.21.15.47:0/3251980293 >> v2:172.21.15.116:3300/0 conn(0x7f434c072680 0x7f434c0ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.252+0000 7f434be00640 1 --2- 172.21.15.47:0/3251980293 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434c15bd90 0x7f434c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.252+0000 7f4343e00640 1 -- 172.21.15.47:0/3251980293 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434c15bd90 msgr2=0x7f434c070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:50.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.252+0000 7f4343e00640 1 --2- 172.21.15.47:0/3251980293 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434c15bd90 0x7f434c070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.252+0000 7f4343e00640 1 -- 172.21.15.47:0/3251980293 >> v2:172.21.15.116:3300/0 conn(0x7f434c072680 msgr2=0x7f434c0ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:50.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.252+0000 7f4343e00640 1 --2- 172.21.15.47:0/3251980293 >> v2:172.21.15.116:3300/0 conn(0x7f434c072680 0x7f434c0ca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.252+0000 7f4343e00640 1 -- 172.21.15.47:0/3251980293 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434c160530 con 0x7f434c0cb490 2024-02-19T15:25:50.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.256+0000 7f4343e00640 1 --2- 172.21.15.47:0/3251980293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f434c0cb490 0x7f434c072140 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f434c0b5620 tx=0x7f433401c100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=896027786a006aca server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:50.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.256+0000 7f434b400640 1 -- 172.21.15.47:0/3251980293 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43340ab600 con 0x7f434c0cb490 2024-02-19T15:25:50.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.256+0000 7f434b400640 1 -- 172.21.15.47:0/3251980293 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43340a7050 con 0x7f434c0cb490 2024-02-19T15:25:50.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.256+0000 7f434b400640 1 -- 172.21.15.47:0/3251980293 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43340ab980 con 0x7f434c0cb490 2024-02-19T15:25:50.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.256+0000 7f4353200640 1 -- 172.21.15.47:0/3251980293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f434c0cb490 msgr2=0x7f434c072140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:50.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.256+0000 7f4353200640 1 --2- 172.21.15.47:0/3251980293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f434c0cb490 0x7f434c072140 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f434c0b5620 tx=0x7f433401c100 comp rx=0 tx=0).stop 2024-02-19T15:25:50.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.256+0000 7f4353200640 1 -- 172.21.15.47:0/3251980293 shutdown_connections 2024-02-19T15:25:50.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.256+0000 7f4353200640 1 --2- 172.21.15.47:0/3251980293 >> v2:172.21.15.116:3300/0 conn(0x7f434c072680 0x7f434c0ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.256+0000 7f4353200640 1 --2- 172.21.15.47:0/3251980293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f434c0cb490 0x7f434c072140 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.256+0000 7f4353200640 1 --2- 172.21.15.47:0/3251980293 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434c15bd90 0x7f434c070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.256+0000 7f4353200640 1 -- 172.21.15.47:0/3251980293 >> 172.21.15.47:0/3251980293 conn(0x7f434c0cfc20 msgr2=0x7f434c0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:50.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.256+0000 7f4353200640 1 -- 172.21.15.47:0/3251980293 shutdown_connections 2024-02-19T15:25:50.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.256+0000 7f4353200640 1 -- 172.21.15.47:0/3251980293 wait complete. 2024-02-19T15:25:50.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f4353200640 1 Processor -- start 2024-02-19T15:25:50.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f4353200640 1 -- start start 2024-02-19T15:25:50.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f4353200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f434c072680 0x7f434c15d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:50.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f4353200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434c0cb490 0x7f434c155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:50.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f4353200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f434c15bd90 0x7f434c15ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:50.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f4353200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c0c6070 con 0x7f434c072680 2024-02-19T15:25:50.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f4353200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c15dd70 con 0x7f434c15bd90 2024-02-19T15:25:50.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f4353200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434c1556d0 con 0x7f434c0cb490 2024-02-19T15:25:50.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f434be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f434c072680 0x7f434c15d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:50.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f434be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f434c072680 0x7f434c15d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48456/0 (socket says 172.21.15.47:48456) 2024-02-19T15:25:50.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f434be00640 1 -- 172.21.15.47:0/3811625216 learned_addr learned my addr 172.21.15.47:0/3811625216 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:50.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f4351200640 1 --2- 172.21.15.47:0/3811625216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f434c15bd90 0x7f434c15ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:50.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f4343e00640 1 --2- 172.21.15.47:0/3811625216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434c0cb490 0x7f434c155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:50.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f434be00640 1 -- 172.21.15.47:0/3811625216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434c0cb490 msgr2=0x7f434c155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:50.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f434be00640 1 --2- 172.21.15.47:0/3811625216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434c0cb490 0x7f434c155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f434be00640 1 -- 172.21.15.47:0/3811625216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f434c15bd90 msgr2=0x7f434c15ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:50.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f434be00640 1 --2- 172.21.15.47:0/3811625216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f434c15bd90 0x7f434c15ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f434be00640 1 -- 172.21.15.47:0/3811625216 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43340094d0 con 0x7f434c072680 2024-02-19T15:25:50.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f4343e00640 1 --2- 172.21.15.47:0/3811625216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434c0cb490 0x7f434c155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:50.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f4351200640 1 --2- 172.21.15.47:0/3811625216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f434c15bd90 0x7f434c15ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:50.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f434be00640 1 --2- 172.21.15.47:0/3811625216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f434c072680 0x7f434c15d830 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f433c00e870 tx=0x7f433c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:50.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f4348c00640 1 -- 172.21.15.47:0/3811625216 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c009510 con 0x7f434c072680 2024-02-19T15:25:50.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f4348c00640 1 -- 172.21.15.47:0/3811625216 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433c00eec0 con 0x7f434c072680 2024-02-19T15:25:50.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f4353200640 1 -- 172.21.15.47:0/3811625216 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434c1f59e0 con 0x7f434c072680 2024-02-19T15:25:50.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f4348c00640 1 -- 172.21.15.47:0/3811625216 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c00cae0 con 0x7f434c072680 2024-02-19T15:25:50.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f4353200640 1 -- 172.21.15.47:0/3811625216 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f434c1546b0 con 0x7f434c072680 2024-02-19T15:25:50.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.260+0000 7f4353200640 1 -- 172.21.15.47:0/3811625216 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f434c075360 con 0x7f434c072680 2024-02-19T15:25:50.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.264+0000 7f4348c00640 1 -- 172.21.15.47:0/3811625216 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f433c00cc40 con 0x7f434c072680 2024-02-19T15:25:50.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.264+0000 7f4348c00640 1 --2- 172.21.15.47:0/3811625216 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4328066e40 0x7f4328069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:50.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.264+0000 7f4348c00640 1 -- 172.21.15.47:0/3811625216 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f433c0848b0 con 0x7f434c072680 2024-02-19T15:25:50.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.264+0000 7f4343e00640 1 --2- 172.21.15.47:0/3811625216 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4328066e40 0x7f4328069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:50.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.264+0000 7f4343e00640 1 --2- 172.21.15.47:0/3811625216 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4328066e40 0x7f4328069300 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f43340ad000 tx=0x7f4334008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:50.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.268+0000 7f4348c00640 1 -- 172.21.15.47:0/3811625216 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f433c051900 con 0x7f434c072680 2024-02-19T15:25:50.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.436+0000 7f4353200640 1 -- 172.21.15.47:0/3811625216 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f434c06e800 con 0x7f434c072680 2024-02-19T15:25:50.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.440+0000 7f4348c00640 1 -- 172.21.15.47:0/3811625216 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v24) v1 ==== 104+0+0 (secure 0 0 0) 0x7f433c054fc0 con 0x7f434c072680 2024-02-19T15:25:50.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.440+0000 7f4353200640 1 -- 172.21.15.47:0/3811625216 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4328066e40 msgr2=0x7f4328069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:50.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.440+0000 7f4353200640 1 --2- 172.21.15.47:0/3811625216 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4328066e40 0x7f4328069300 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f43340ad000 tx=0x7f4334008040 comp rx=0 tx=0).stop 2024-02-19T15:25:50.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.440+0000 7f4353200640 1 -- 172.21.15.47:0/3811625216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f434c072680 msgr2=0x7f434c15d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:50.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.440+0000 7f4353200640 1 --2- 172.21.15.47:0/3811625216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f434c072680 0x7f434c15d830 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f433c00e870 tx=0x7f433c00ec70 comp rx=0 tx=0).stop 2024-02-19T15:25:50.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.440+0000 7f4353200640 1 -- 172.21.15.47:0/3811625216 shutdown_connections 2024-02-19T15:25:50.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.440+0000 7f4353200640 1 --2- 172.21.15.47:0/3811625216 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4328066e40 0x7f4328069300 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.440+0000 7f4353200640 1 --2- 172.21.15.47:0/3811625216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f434c15bd90 0x7f434c15ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.440+0000 7f4353200640 1 --2- 172.21.15.47:0/3811625216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434c0cb490 0x7f434c155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.440+0000 7f4353200640 1 --2- 172.21.15.47:0/3811625216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f434c072680 0x7f434c15d830 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.440+0000 7f4353200640 1 -- 172.21.15.47:0/3811625216 >> 172.21.15.47:0/3811625216 conn(0x7f434c0cfc20 msgr2=0x7f434c0b4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:50.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.440+0000 7f4353200640 1 -- 172.21.15.47:0/3811625216 shutdown_connections 2024-02-19T15:25:50.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.440+0000 7f4353200640 1 -- 172.21.15.47:0/3811625216 wait complete. 2024-02-19T15:25:50.464 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-02-19T15:25:50.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.600+0000 7fa9f8000640 1 Processor -- start 2024-02-19T15:25:50.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.600+0000 7fa9f8000640 1 -- start start 2024-02-19T15:25:50.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.604+0000 7fa9f8000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9f00adb80 0x7fa9f00adf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:50.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.604+0000 7fa9f8000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa9f0147990 0x7fa9f00cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:50.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.604+0000 7fa9f8000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9f00ae540 con 0x7fa9f0147440 2024-02-19T15:25:50.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.604+0000 7fa9f8000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f00ae820 con 0x7fa9f0147990 2024-02-19T15:25:50.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.604+0000 7fa9f8000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f00aeb00 con 0x7fa9f00adb80 2024-02-19T15:25:50.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.604+0000 7fa9f4c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa9f0147990 0x7fa9f00cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:50.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.604+0000 7fa9f4c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa9f0147990 0x7fa9f00cd7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48460/0 (socket says 172.21.15.47:48460) 2024-02-19T15:25:50.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.604+0000 7fa9f4c00640 1 -- 172.21.15.47:0/548946728 learned_addr learned my addr 172.21.15.47:0/548946728 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:50.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.604+0000 7fa9f6000640 1 --1- 172.21.15.47:0/548946728 >> v1:172.21.15.47:6789/0 conn(0x7fa9f0147440 0x7fa9f0161cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52940/0 (socket says 172.21.15.47:52940) 2024-02-19T15:25:50.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.604+0000 7fa9f5600640 1 --2- 172.21.15.47:0/548946728 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9f00adb80 0x7fa9f00adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:50.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.604+0000 7fa9f4c00640 1 -- 172.21.15.47:0/548946728 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9f00adb80 msgr2=0x7fa9f00adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:50.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.604+0000 7fa9f4c00640 1 --2- 172.21.15.47:0/548946728 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9f00adb80 0x7fa9f00adf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.604+0000 7fa9f4c00640 1 -- 172.21.15.47:0/548946728 >> v1:172.21.15.47:6789/0 conn(0x7fa9f0147440 legacy=0x7fa9f0161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:50.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.604+0000 7fa9f4c00640 1 -- 172.21.15.47:0/548946728 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f00aede0 con 0x7fa9f0147990 2024-02-19T15:25:50.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.604+0000 7fa9f5600640 1 --2- 172.21.15.47:0/548946728 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9f00adb80 0x7fa9f00adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:50.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.604+0000 7fa9f4c00640 1 --2- 172.21.15.47:0/548946728 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9f0147990 0x7fa9f00cd7f0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc009870 tx=0x7fa9dc01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac7ee468cba5798 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:50.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9efe00640 1 -- 172.21.15.47:0/548946728 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc0ab660 con 0x7fa9f0147990 2024-02-19T15:25:50.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9efe00640 1 -- 172.21.15.47:0/548946728 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9dc0ab7c0 con 0x7fa9f0147990 2024-02-19T15:25:50.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9efe00640 1 -- 172.21.15.47:0/548946728 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc0abb50 con 0x7fa9f0147990 2024-02-19T15:25:50.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f8000640 1 -- 172.21.15.47:0/548946728 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9f0147990 msgr2=0x7fa9f00cd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:50.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f8000640 1 --2- 172.21.15.47:0/548946728 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9f0147990 0x7fa9f00cd7f0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc009870 tx=0x7fa9dc01c440 comp rx=0 tx=0).stop 2024-02-19T15:25:50.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f8000640 1 -- 172.21.15.47:0/548946728 shutdown_connections 2024-02-19T15:25:50.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f8000640 1 --2- 172.21.15.47:0/548946728 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9f0147990 0x7fa9f00cd7f0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f8000640 1 --2- 172.21.15.47:0/548946728 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9f00adb80 0x7fa9f00adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f8000640 1 -- 172.21.15.47:0/548946728 >> 172.21.15.47:0/548946728 conn(0x7fa9f00cfc20 msgr2=0x7fa9f00d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:50.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f8000640 1 -- 172.21.15.47:0/548946728 shutdown_connections 2024-02-19T15:25:50.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f8000640 1 -- 172.21.15.47:0/548946728 wait complete. 2024-02-19T15:25:50.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f8000640 1 Processor -- start 2024-02-19T15:25:50.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f8000640 1 -- start start 2024-02-19T15:25:50.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f8000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f00adb80 0x7fa9f015b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:50.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f8000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9f0147440 0x7fa9f00afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:50.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f8000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9f0147990 0x7fa9f00b0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:50.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f8000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f015ca70 con 0x7fa9f0147990 2024-02-19T15:25:50.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f8000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f015c1d0 con 0x7fa9f00adb80 2024-02-19T15:25:50.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f8000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f015c4b0 con 0x7fa9f0147440 2024-02-19T15:25:50.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f6000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9f0147990 0x7fa9f00b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:50.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f6000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9f0147990 0x7fa9f00b0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48462/0 (socket says 172.21.15.47:48462) 2024-02-19T15:25:50.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f6000640 1 -- 172.21.15.47:0/100649710 learned_addr learned my addr 172.21.15.47:0/100649710 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:50.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f4c00640 1 --2- 172.21.15.47:0/100649710 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9f0147440 0x7fa9f00afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:50.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.608+0000 7fa9f5600640 1 --2- 172.21.15.47:0/100649710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f00adb80 0x7fa9f015b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:50.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.612+0000 7fa9f6000640 1 -- 172.21.15.47:0/100649710 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9f0147440 msgr2=0x7fa9f00afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:50.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.612+0000 7fa9f6000640 1 --2- 172.21.15.47:0/100649710 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9f0147440 0x7fa9f00afaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.612+0000 7fa9f6000640 1 -- 172.21.15.47:0/100649710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f00adb80 msgr2=0x7fa9f015b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:50.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.612+0000 7fa9f6000640 1 --2- 172.21.15.47:0/100649710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f00adb80 0x7fa9f015b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.612+0000 7fa9f6000640 1 -- 172.21.15.47:0/100649710 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc0094d0 con 0x7fa9f0147990 2024-02-19T15:25:50.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.612+0000 7fa9f6000640 1 --2- 172.21.15.47:0/100649710 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9f0147990 0x7fa9f00b0030 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa9e800c700 tx=0x7fa9e800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:50.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.612+0000 7fa9ed600640 1 -- 172.21.15.47:0/100649710 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e800ce10 con 0x7fa9f0147990 2024-02-19T15:25:50.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.612+0000 7fa9ed600640 1 -- 172.21.15.47:0/100649710 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e800a640 con 0x7fa9f0147990 2024-02-19T15:25:50.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.612+0000 7fa9f8000640 1 -- 172.21.15.47:0/100649710 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f015c790 con 0x7fa9f0147990 2024-02-19T15:25:50.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.612+0000 7fa9ed600640 1 -- 172.21.15.47:0/100649710 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e800a9b0 con 0x7fa9f0147990 2024-02-19T15:25:50.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.612+0000 7fa9f8000640 1 -- 172.21.15.47:0/100649710 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f00b4770 con 0x7fa9f0147990 2024-02-19T15:25:50.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.612+0000 7fa9f8000640 1 -- 172.21.15.47:0/100649710 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f00cc3f0 con 0x7fa9f0147990 2024-02-19T15:25:50.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.612+0000 7fa9ed600640 1 -- 172.21.15.47:0/100649710 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa9e800ab10 con 0x7fa9f0147990 2024-02-19T15:25:50.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.616+0000 7fa9ed600640 1 --2- 172.21.15.47:0/100649710 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa9d0066e40 0x7fa9d0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:50.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.616+0000 7fa9ed600640 1 -- 172.21.15.47:0/100649710 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa9e8083e40 con 0x7fa9f0147990 2024-02-19T15:25:50.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.616+0000 7fa9f5600640 1 --2- 172.21.15.47:0/100649710 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa9d0066e40 0x7fa9d0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:50.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.616+0000 7fa9f5600640 1 --2- 172.21.15.47:0/100649710 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa9d0066e40 0x7fa9d0069300 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa9e40045f0 tx=0x7fa9e4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:50.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.620+0000 7fa9ed600640 1 -- 172.21.15.47:0/100649710 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa9e8050fe0 con 0x7fa9f0147990 2024-02-19T15:25:50.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.780+0000 7fa9f8000640 1 -- 172.21.15.47:0/100649710 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7fa9f0075360 con 0x7fa9f0147990 2024-02-19T15:25:50.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.784+0000 7fa9ed600640 1 -- 172.21.15.47:0/100649710 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v25) v1 ==== 108+0+0 (secure 0 0 0) 0x7fa9e80170b0 con 0x7fa9f0147990 2024-02-19T15:25:50.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.784+0000 7fa9f8000640 1 -- 172.21.15.47:0/100649710 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa9d0066e40 msgr2=0x7fa9d0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:50.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.784+0000 7fa9f8000640 1 --2- 172.21.15.47:0/100649710 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa9d0066e40 0x7fa9d0069300 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa9e40045f0 tx=0x7fa9e4008040 comp rx=0 tx=0).stop 2024-02-19T15:25:50.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.784+0000 7fa9f8000640 1 -- 172.21.15.47:0/100649710 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9f0147990 msgr2=0x7fa9f00b0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:50.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.784+0000 7fa9f8000640 1 --2- 172.21.15.47:0/100649710 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9f0147990 0x7fa9f00b0030 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa9e800c700 tx=0x7fa9e800cbc0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.784+0000 7fa9f8000640 1 -- 172.21.15.47:0/100649710 shutdown_connections 2024-02-19T15:25:50.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.784+0000 7fa9f8000640 1 --2- 172.21.15.47:0/100649710 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa9d0066e40 0x7fa9d0069300 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.784+0000 7fa9f8000640 1 --2- 172.21.15.47:0/100649710 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9f0147990 0x7fa9f00b0030 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.784+0000 7fa9f8000640 1 --2- 172.21.15.47:0/100649710 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9f0147440 0x7fa9f00afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.784+0000 7fa9f8000640 1 --2- 172.21.15.47:0/100649710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f00adb80 0x7fa9f015b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.784+0000 7fa9f8000640 1 -- 172.21.15.47:0/100649710 >> 172.21.15.47:0/100649710 conn(0x7fa9f00cfc20 msgr2=0x7fa9f00cbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:50.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.788+0000 7fa9f8000640 1 -- 172.21.15.47:0/100649710 shutdown_connections 2024-02-19T15:25:50.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.788+0000 7fa9f8000640 1 -- 172.21.15.47:0/100649710 wait complete. 2024-02-19T15:25:50.810 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-02-19T15:25:50.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.944+0000 7f0357400640 1 Processor -- start 2024-02-19T15:25:50.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.944+0000 7f0357400640 1 -- start start 2024-02-19T15:25:50.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.948+0000 7f0357400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f035015b1a0 0x7f035015b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:50.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.948+0000 7f0357400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0350071350 0x7f035015bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:50.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.948+0000 7f0357400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f035015c000 con 0x7f035006c9a0 2024-02-19T15:25:50.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.948+0000 7f0357400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03500718c0 con 0x7f035015b1a0 2024-02-19T15:25:50.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.948+0000 7f0357400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0350071ba0 con 0x7f0350071350 2024-02-19T15:25:50.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.948+0000 7f0355400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f035006c9a0 0x7f035015c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52950/0 (socket says 172.21.15.47:52950) 2024-02-19T15:25:50.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.948+0000 7f0355400640 1 -- 172.21.15.47:0/2084776457 learned_addr learned my addr 172.21.15.47:0/2084776457 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:50.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.948+0000 7f034fe00640 1 --2- 172.21.15.47:0/2084776457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0350071350 0x7f035015bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:50.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.948+0000 7f0354a00640 1 --2- 172.21.15.47:0/2084776457 >> v2:172.21.15.116:3300/0 conn(0x7f035015b1a0 0x7f035015b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:50.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.948+0000 7f034f400640 1 -- 172.21.15.47:0/2084776457 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316986255 0 0) 0x7f035015c000 con 0x7f035006c9a0 2024-02-19T15:25:50.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.948+0000 7f034f400640 1 -- 172.21.15.47:0/2084776457 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0338003530 con 0x7f035006c9a0 2024-02-19T15:25:50.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.948+0000 7f034fe00640 1 -- 172.21.15.47:0/2084776457 >> v2:172.21.15.116:3300/0 conn(0x7f035015b1a0 msgr2=0x7f035015b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:50.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.948+0000 7f034fe00640 1 --2- 172.21.15.47:0/2084776457 >> v2:172.21.15.116:3300/0 conn(0x7f035015b1a0 0x7f035015b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.948+0000 7f034fe00640 1 -- 172.21.15.47:0/2084776457 >> v1:172.21.15.47:6789/0 conn(0x7f035006c9a0 legacy=0x7f035015c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:50.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.948+0000 7f034fe00640 1 -- 172.21.15.47:0/2084776457 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0350071e80 con 0x7f0350071350 2024-02-19T15:25:50.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.952+0000 7f034fe00640 1 --2- 172.21.15.47:0/2084776457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0350071350 0x7f035015bac0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f033c0097c0 tx=0x7f033c01c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b8477bb3d8fec94 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:50.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.952+0000 7f034f400640 1 -- 172.21.15.47:0/2084776457 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033c0aa5f0 con 0x7f0350071350 2024-02-19T15:25:50.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.952+0000 7f034f400640 1 -- 172.21.15.47:0/2084776457 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f033c0aa750 con 0x7f0350071350 2024-02-19T15:25:50.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.952+0000 7f034f400640 1 -- 172.21.15.47:0/2084776457 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033c0aaae0 con 0x7f0350071350 2024-02-19T15:25:50.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.952+0000 7f0357400640 1 -- 172.21.15.47:0/2084776457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0350071350 msgr2=0x7f035015bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:50.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.952+0000 7f0357400640 1 --2- 172.21.15.47:0/2084776457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0350071350 0x7f035015bac0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f033c0097c0 tx=0x7f033c01c470 comp rx=0 tx=0).stop 2024-02-19T15:25:50.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.952+0000 7f0357400640 1 -- 172.21.15.47:0/2084776457 shutdown_connections 2024-02-19T15:25:50.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.952+0000 7f0357400640 1 --2- 172.21.15.47:0/2084776457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0350071350 0x7f035015bac0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.952+0000 7f0357400640 1 --2- 172.21.15.47:0/2084776457 >> v2:172.21.15.116:3300/0 conn(0x7f035015b1a0 0x7f035015b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.952+0000 7f0357400640 1 -- 172.21.15.47:0/2084776457 >> 172.21.15.47:0/2084776457 conn(0x7f03500b16a0 msgr2=0x7f03500b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:50.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.952+0000 7f0357400640 1 -- 172.21.15.47:0/2084776457 shutdown_connections 2024-02-19T15:25:50.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.952+0000 7f0357400640 1 -- 172.21.15.47:0/2084776457 wait complete. 2024-02-19T15:25:50.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.952+0000 7f0357400640 1 Processor -- start 2024-02-19T15:25:50.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.952+0000 7f0357400640 1 -- start start 2024-02-19T15:25:50.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f0357400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f035006c9a0 0x7f035013d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:50.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f0357400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0350071350 0x7f0350156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:50.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f0357400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f035015b1a0 0x7f0350157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:50.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f0357400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035013dae0 con 0x7f0350071350 2024-02-19T15:25:50.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f0357400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0350157930 con 0x7f035015b1a0 2024-02-19T15:25:50.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f0357400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0350157c10 con 0x7f035006c9a0 2024-02-19T15:25:50.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f034fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0350071350 0x7f0350156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:50.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f034fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0350071350 0x7f0350156dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48472/0 (socket says 172.21.15.47:48472) 2024-02-19T15:25:50.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f034fe00640 1 -- 172.21.15.47:0/396095397 learned_addr learned my addr 172.21.15.47:0/396095397 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:50.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f0354a00640 1 --2- 172.21.15.47:0/396095397 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f035006c9a0 0x7f035013d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:50.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f0355400640 1 --2- 172.21.15.47:0/396095397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f035015b1a0 0x7f0350157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:50.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f034fe00640 1 -- 172.21.15.47:0/396095397 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f035006c9a0 msgr2=0x7f035013d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:50.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f034fe00640 1 --2- 172.21.15.47:0/396095397 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f035006c9a0 0x7f035013d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f034fe00640 1 -- 172.21.15.47:0/396095397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f035015b1a0 msgr2=0x7f0350157300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:50.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f034fe00640 1 --2- 172.21.15.47:0/396095397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f035015b1a0 0x7f0350157300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:50.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f034fe00640 1 -- 172.21.15.47:0/396095397 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033c009420 con 0x7f0350071350 2024-02-19T15:25:50.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f034fe00640 1 --2- 172.21.15.47:0/396095397 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0350071350 0x7f0350156dc0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f033c0ac040 tx=0x7f033c01acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:50.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f034cc00640 1 -- 172.21.15.47:0/396095397 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033c0be610 con 0x7f0350071350 2024-02-19T15:25:50.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f034cc00640 1 -- 172.21.15.47:0/396095397 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f033c0be770 con 0x7f0350071350 2024-02-19T15:25:50.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f034cc00640 1 -- 172.21.15.47:0/396095397 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033c0beb20 con 0x7f0350071350 2024-02-19T15:25:50.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f0357400640 1 -- 172.21.15.47:0/396095397 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0350157ef0 con 0x7f0350071350 2024-02-19T15:25:50.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f0357400640 1 -- 172.21.15.47:0/396095397 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03501584b0 con 0x7f0350071350 2024-02-19T15:25:50.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f0357400640 1 -- 172.21.15.47:0/396095397 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03500ccec0 con 0x7f0350071350 2024-02-19T15:25:50.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.956+0000 7f034cc00640 1 -- 172.21.15.47:0/396095397 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f033c0bec80 con 0x7f0350071350 2024-02-19T15:25:50.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.960+0000 7f034cc00640 1 --2- 172.21.15.47:0/396095397 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0330066e40 0x7f0330069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:50.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.960+0000 7f034cc00640 1 -- 172.21.15.47:0/396095397 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f033c0a7070 con 0x7f0350071350 2024-02-19T15:25:50.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.960+0000 7f0354a00640 1 --2- 172.21.15.47:0/396095397 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0330066e40 0x7f0330069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:50.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.960+0000 7f0354a00640 1 --2- 172.21.15.47:0/396095397 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0330066e40 0x7f0330069300 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0344009b60 tx=0x7f0344009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:50.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:50.964+0000 7f034cc00640 1 -- 172.21.15.47:0/396095397 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f03500ccec0 con 0x7f0350071350 2024-02-19T15:25:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:51.180+0000 7f0357400640 1 -- 172.21.15.47:0/396095397 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7f03500ccec0 con 0x7f0350071350 2024-02-19T15:25:52.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.080+0000 7f034cc00640 1 -- 172.21.15.47:0/396095397 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v219) v1 ==== 140+0+0 (secure 0 0 0) 0x7f03500ccec0 con 0x7f0350071350 2024-02-19T15:25:52.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.084+0000 7f0357400640 1 -- 172.21.15.47:0/396095397 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0330066e40 msgr2=0x7f0330069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:52.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.084+0000 7f0357400640 1 --2- 172.21.15.47:0/396095397 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0330066e40 0x7f0330069300 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0344009b60 tx=0x7f0344009190 comp rx=0 tx=0).stop 2024-02-19T15:25:52.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.084+0000 7f0357400640 1 -- 172.21.15.47:0/396095397 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0350071350 msgr2=0x7f0350156dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:52.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.084+0000 7f0357400640 1 --2- 172.21.15.47:0/396095397 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0350071350 0x7f0350156dc0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f033c0ac040 tx=0x7f033c01acb0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.084+0000 7f0357400640 1 -- 172.21.15.47:0/396095397 shutdown_connections 2024-02-19T15:25:52.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.084+0000 7f0357400640 1 --2- 172.21.15.47:0/396095397 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0330066e40 0x7f0330069300 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.084+0000 7f0357400640 1 --2- 172.21.15.47:0/396095397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f035015b1a0 0x7f0350157300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.084+0000 7f0357400640 1 --2- 172.21.15.47:0/396095397 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0350071350 0x7f0350156dc0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.084+0000 7f0357400640 1 --2- 172.21.15.47:0/396095397 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f035006c9a0 0x7f035013d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.084+0000 7f0357400640 1 -- 172.21.15.47:0/396095397 >> 172.21.15.47:0/396095397 conn(0x7f03500b16a0 msgr2=0x7f0350151990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:52.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.084+0000 7f0357400640 1 -- 172.21.15.47:0/396095397 shutdown_connections 2024-02-19T15:25:52.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.084+0000 7f0357400640 1 -- 172.21.15.47:0/396095397 wait complete. 2024-02-19T15:25:52.111 INFO:tasks.cephfs_test_runner:That unlinking fails when the stray directory fragment becomes too ... ok 2024-02-19T15:25:52.111 INFO:tasks.cephfs_test_runner:Starting test: test_fancy_layout (tasks.cephfs.test_strays.TestStrays) purge stray file with fancy layout 2024-02-19T15:25:52.111 INFO:tasks.cephfs_test_runner:test_fancy_layout (tasks.cephfs.test_strays.TestStrays) 2024-02-19T15:25:52.112 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:25:52.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.252+0000 7f1e25e00640 1 Processor -- start 2024-02-19T15:25:52.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.252+0000 7f1e25e00640 1 -- start start 2024-02-19T15:25:52.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.252+0000 7f1e25e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e2015dfa0 0x7f1e2006fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:52.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.252+0000 7f1e25e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1e200b4240 0x7f1e200d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:52.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.252+0000 7f1e25e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2006ff60 con 0x7f1e200ce400 2024-02-19T15:25:52.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.252+0000 7f1e25e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e200d1070 con 0x7f1e200b4240 2024-02-19T15:25:52.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.252+0000 7f1e25e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2006ec60 con 0x7f1e2015dfa0 2024-02-19T15:25:52.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.252+0000 7f1e16000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1e200b4240 0x7f1e200d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:52.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.252+0000 7f1e16000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1e200b4240 0x7f1e200d0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48486/0 (socket says 172.21.15.47:48486) 2024-02-19T15:25:52.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.252+0000 7f1e16000640 1 -- 172.21.15.47:0/1974846774 learned_addr learned my addr 172.21.15.47:0/1974846774 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:52.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.252+0000 7f1e16000640 1 -- 172.21.15.47:0/1974846774 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e2015dfa0 msgr2=0x7f1e2006fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:52.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.252+0000 7f1e16000640 1 --2- 172.21.15.47:0/1974846774 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e2015dfa0 0x7f1e2006fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.252+0000 7f1e16000640 1 -- 172.21.15.47:0/1974846774 >> v1:172.21.15.116:6789/0 conn(0x7f1e200ce400 legacy=0x7f1e200cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:52.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.252+0000 7f1e16000640 1 -- 172.21.15.47:0/1974846774 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2006ef40 con 0x7f1e200b4240 2024-02-19T15:25:52.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.256+0000 7f1e16000640 1 --2- 172.21.15.47:0/1974846774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e200b4240 0x7f1e200d0820 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f1e08009870 tx=0x7f1e0801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b85dd7bfd08a67ce server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:52.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.256+0000 7f1e1e000640 1 -- 172.21.15.47:0/1974846774 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e080aa610 con 0x7f1e200b4240 2024-02-19T15:25:52.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.256+0000 7f1e1e000640 1 -- 172.21.15.47:0/1974846774 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e080a6040 con 0x7f1e200b4240 2024-02-19T15:25:52.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.256+0000 7f1e1e000640 1 -- 172.21.15.47:0/1974846774 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e080aa990 con 0x7f1e200b4240 2024-02-19T15:25:52.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.256+0000 7f1e25e00640 1 -- 172.21.15.47:0/1974846774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e200b4240 msgr2=0x7f1e200d0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:52.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.256+0000 7f1e25e00640 1 --2- 172.21.15.47:0/1974846774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e200b4240 0x7f1e200d0820 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f1e08009870 tx=0x7f1e0801c4f0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.256+0000 7f1e25e00640 1 -- 172.21.15.47:0/1974846774 shutdown_connections 2024-02-19T15:25:52.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.256+0000 7f1e25e00640 1 --2- 172.21.15.47:0/1974846774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e200b4240 0x7f1e200d0820 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.256+0000 7f1e25e00640 1 --2- 172.21.15.47:0/1974846774 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e2015dfa0 0x7f1e2006fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.256+0000 7f1e25e00640 1 -- 172.21.15.47:0/1974846774 >> 172.21.15.47:0/1974846774 conn(0x7f1e200b3690 msgr2=0x7f1e200b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:52.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.256+0000 7f1e25e00640 1 -- 172.21.15.47:0/1974846774 shutdown_connections 2024-02-19T15:25:52.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.256+0000 7f1e25e00640 1 -- 172.21.15.47:0/1974846774 wait complete. 2024-02-19T15:25:52.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e25e00640 1 Processor -- start 2024-02-19T15:25:52.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e25e00640 1 -- start start 2024-02-19T15:25:52.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e25e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e200b4240 0x7f1e2015e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:52.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e25e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e200ce400 0x7f1e2015b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:52.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e25e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e2015dfa0 0x7f1e2015bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:52.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e25e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e200cb3f0 con 0x7f1e2015dfa0 2024-02-19T15:25:52.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e25e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e200cb6d0 con 0x7f1e200b4240 2024-02-19T15:25:52.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e25e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e200cb9b0 con 0x7f1e200ce400 2024-02-19T15:25:52.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e1f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e2015dfa0 0x7f1e2015bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:52.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e1f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e2015dfa0 0x7f1e2015bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48490/0 (socket says 172.21.15.47:48490) 2024-02-19T15:25:52.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e1f400640 1 -- 172.21.15.47:0/1475913983 learned_addr learned my addr 172.21.15.47:0/1475913983 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:52.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e16000640 1 --2- 172.21.15.47:0/1475913983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e200ce400 0x7f1e2015b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:52.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e1ea00640 1 --2- 172.21.15.47:0/1475913983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e200b4240 0x7f1e2015e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:52.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e1f400640 1 -- 172.21.15.47:0/1475913983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e200ce400 msgr2=0x7f1e2015b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:52.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e1f400640 1 --2- 172.21.15.47:0/1475913983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e200ce400 0x7f1e2015b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e1f400640 1 -- 172.21.15.47:0/1475913983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e200b4240 msgr2=0x7f1e2015e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:52.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e1f400640 1 --2- 172.21.15.47:0/1475913983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e200b4240 0x7f1e2015e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e1f400640 1 -- 172.21.15.47:0/1475913983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e080094d0 con 0x7f1e2015dfa0 2024-02-19T15:25:52.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e1ea00640 1 --2- 172.21.15.47:0/1475913983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e200b4240 0x7f1e2015e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:52.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e1f400640 1 --2- 172.21.15.47:0/1475913983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e2015dfa0 0x7f1e2015bed0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f1e1800c410 tx=0x7f1e1800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:52.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e17e00640 1 -- 172.21.15.47:0/1475913983 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e18003600 con 0x7f1e2015dfa0 2024-02-19T15:25:52.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e17e00640 1 -- 172.21.15.47:0/1475913983 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e18003760 con 0x7f1e2015dfa0 2024-02-19T15:25:52.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e17e00640 1 -- 172.21.15.47:0/1475913983 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e18003b50 con 0x7f1e2015dfa0 2024-02-19T15:25:52.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e25e00640 1 -- 172.21.15.47:0/1475913983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2015b190 con 0x7f1e2015dfa0 2024-02-19T15:25:52.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.260+0000 7f1e25e00640 1 -- 172.21.15.47:0/1475913983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e200c8c50 con 0x7f1e2015dfa0 2024-02-19T15:25:52.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.264+0000 7f1e25e00640 1 -- 172.21.15.47:0/1475913983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e2015b6b0 con 0x7f1e2015dfa0 2024-02-19T15:25:52.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.264+0000 7f1e17e00640 1 -- 172.21.15.47:0/1475913983 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1e18003cb0 con 0x7f1e2015dfa0 2024-02-19T15:25:52.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.264+0000 7f1e17e00640 1 --2- 172.21.15.47:0/1475913983 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1dfc066e40 0x7f1dfc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:52.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.264+0000 7f1e17e00640 1 -- 172.21.15.47:0/1475913983 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1e18083c30 con 0x7f1e2015dfa0 2024-02-19T15:25:52.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.264+0000 7f1e1ea00640 1 --2- 172.21.15.47:0/1475913983 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1dfc066e40 0x7f1dfc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:52.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.264+0000 7f1e1ea00640 1 --2- 172.21.15.47:0/1475913983 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1dfc066e40 0x7f1dfc069300 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009fa0 tx=0x7f1e10009570 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:52.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.268+0000 7f1e17e00640 1 -- 172.21.15.47:0/1475913983 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1e2015b6b0 con 0x7f1e2015dfa0 2024-02-19T15:25:52.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.432+0000 7f1e25e00640 1 -- 172.21.15.47:0/1475913983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e2015b6b0 con 0x7f1e2015dfa0 2024-02-19T15:25:52.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.432+0000 7f1e17e00640 1 -- 172.21.15.47:0/1475913983 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+13033 (secure 0 0 0) 0x7f1e2015b6b0 con 0x7f1e2015dfa0 2024-02-19T15:25:52.437 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:52.437 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":74,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:25:50.073025+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:6835/2490197058":"2024-02-20T15:25:47.062600+0000","172.21.15.47:6833/2490197058":"2024-02-20T15:25:47.062600+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:25:52.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.436+0000 7f1e25e00640 1 -- 172.21.15.47:0/1475913983 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1dfc066e40 msgr2=0x7f1dfc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.436+0000 7f1e25e00640 1 --2- 172.21.15.47:0/1475913983 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1dfc066e40 0x7f1dfc069300 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009fa0 tx=0x7f1e10009570 comp rx=0 tx=0).stop 2024-02-19T15:25:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.436+0000 7f1e25e00640 1 -- 172.21.15.47:0/1475913983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e2015dfa0 msgr2=0x7f1e2015bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.436+0000 7f1e25e00640 1 --2- 172.21.15.47:0/1475913983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e2015dfa0 0x7f1e2015bed0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f1e1800c410 tx=0x7f1e1800c8d0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.436+0000 7f1e25e00640 1 -- 172.21.15.47:0/1475913983 shutdown_connections 2024-02-19T15:25:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.436+0000 7f1e25e00640 1 --2- 172.21.15.47:0/1475913983 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1dfc066e40 0x7f1dfc069300 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.436+0000 7f1e25e00640 1 --2- 172.21.15.47:0/1475913983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e2015dfa0 0x7f1e2015bed0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.436+0000 7f1e25e00640 1 --2- 172.21.15.47:0/1475913983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e200ce400 0x7f1e2015b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.436+0000 7f1e25e00640 1 --2- 172.21.15.47:0/1475913983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e200b4240 0x7f1e2015e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.436+0000 7f1e25e00640 1 -- 172.21.15.47:0/1475913983 >> 172.21.15.47:0/1475913983 conn(0x7f1e200b3690 msgr2=0x7f1e2006ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.436+0000 7f1e25e00640 1 -- 172.21.15.47:0/1475913983 shutdown_connections 2024-02-19T15:25:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.436+0000 7f1e25e00640 1 -- 172.21.15.47:0/1475913983 wait complete. 2024-02-19T15:25:52.459 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:25:52.460 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:25:52.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.596+0000 7f344a800640 1 Processor -- start 2024-02-19T15:25:52.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.596+0000 7f344a800640 1 -- start start 2024-02-19T15:25:52.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.600+0000 7f344a800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f344415bd90 0x7f3444070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:52.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.600+0000 7f344a800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34440cb490 0x7f3444072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:52.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.600+0000 7f344a800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f344415fc90 con 0x7f3444072680 2024-02-19T15:25:52.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.600+0000 7f344a800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f344415ff70 con 0x7f344415bd90 2024-02-19T15:25:52.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.600+0000 7f344a800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3444160250 con 0x7f34440cb490 2024-02-19T15:25:52.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.600+0000 7f3443e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f3444072680 0x7f34440ca270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52952/0 (socket says 172.21.15.47:52952) 2024-02-19T15:25:52.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.600+0000 7f3443e00640 1 -- 172.21.15.47:0/3728349352 learned_addr learned my addr 172.21.15.47:0/3728349352 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:52.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.600+0000 7f343be00640 1 --2- 172.21.15.47:0/3728349352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34440cb490 0x7f3444072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:52.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.600+0000 7f3443400640 1 --2- 172.21.15.47:0/3728349352 >> v2:172.21.15.116:3300/0 conn(0x7f344415bd90 0x7f3444070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:52.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.600+0000 7f3442a00640 1 -- 172.21.15.47:0/3728349352 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217194053 0 0) 0x7f344415fc90 con 0x7f3444072680 2024-02-19T15:25:52.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.600+0000 7f3442a00640 1 -- 172.21.15.47:0/3728349352 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3428003530 con 0x7f3444072680 2024-02-19T15:25:52.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.600+0000 7f343be00640 1 -- 172.21.15.47:0/3728349352 >> v2:172.21.15.116:3300/0 conn(0x7f344415bd90 msgr2=0x7f3444070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:52.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.600+0000 7f343be00640 1 --2- 172.21.15.47:0/3728349352 >> v2:172.21.15.116:3300/0 conn(0x7f344415bd90 0x7f3444070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.600+0000 7f343be00640 1 -- 172.21.15.47:0/3728349352 >> v1:172.21.15.47:6789/0 conn(0x7f3444072680 legacy=0x7f34440ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:52.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.600+0000 7f343be00640 1 -- 172.21.15.47:0/3728349352 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3444160530 con 0x7f34440cb490 2024-02-19T15:25:52.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.604+0000 7f343be00640 1 --2- 172.21.15.47:0/3728349352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34440cb490 0x7f3444072140 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f342c0097c0 tx=0x7f342c01c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bff8c705cbc46bd7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:52.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.604+0000 7f3442a00640 1 -- 172.21.15.47:0/3728349352 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c0ac600 con 0x7f34440cb490 2024-02-19T15:25:52.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.604+0000 7f3442a00640 1 -- 172.21.15.47:0/3728349352 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342c0a8050 con 0x7f34440cb490 2024-02-19T15:25:52.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.604+0000 7f3442a00640 1 -- 172.21.15.47:0/3728349352 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c0ac980 con 0x7f34440cb490 2024-02-19T15:25:52.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.604+0000 7f344a800640 1 -- 172.21.15.47:0/3728349352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34440cb490 msgr2=0x7f3444072140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:52.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.604+0000 7f344a800640 1 --2- 172.21.15.47:0/3728349352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34440cb490 0x7f3444072140 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f342c0097c0 tx=0x7f342c01c340 comp rx=0 tx=0).stop 2024-02-19T15:25:52.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.604+0000 7f344a800640 1 -- 172.21.15.47:0/3728349352 shutdown_connections 2024-02-19T15:25:52.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.604+0000 7f344a800640 1 --2- 172.21.15.47:0/3728349352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34440cb490 0x7f3444072140 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.604+0000 7f344a800640 1 --2- 172.21.15.47:0/3728349352 >> v2:172.21.15.116:3300/0 conn(0x7f344415bd90 0x7f3444070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.604+0000 7f344a800640 1 -- 172.21.15.47:0/3728349352 >> 172.21.15.47:0/3728349352 conn(0x7f34440cfc20 msgr2=0x7f34440d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:52.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.604+0000 7f344a800640 1 -- 172.21.15.47:0/3728349352 shutdown_connections 2024-02-19T15:25:52.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.604+0000 7f344a800640 1 -- 172.21.15.47:0/3728349352 wait complete. 2024-02-19T15:25:52.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.604+0000 7f344a800640 1 Processor -- start 2024-02-19T15:25:52.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.604+0000 7f344a800640 1 -- start start 2024-02-19T15:25:52.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f344a800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3444072680 0x7f3444152890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:52.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f344a800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34440cb490 0x7f3444158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:52.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f344a800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f344415bd90 0x7f34440b2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:52.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f344a800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344415d880 con 0x7f3444072680 2024-02-19T15:25:52.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f344a800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344415db60 con 0x7f34440cb490 2024-02-19T15:25:52.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f344a800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f344415de40 con 0x7f344415bd90 2024-02-19T15:25:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f3443400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3444072680 0x7f3444152890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f3443400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3444072680 0x7f3444152890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48506/0 (socket says 172.21.15.47:48506) 2024-02-19T15:25:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f3443400640 1 -- 172.21.15.47:0/3951308537 learned_addr learned my addr 172.21.15.47:0/3951308537 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f3443400640 1 -- 172.21.15.47:0/3951308537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f344415bd90 msgr2=0x7f34440b2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f3443400640 1 --2- 172.21.15.47:0/3951308537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f344415bd90 0x7f34440b2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f3443400640 1 -- 172.21.15.47:0/3951308537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34440cb490 msgr2=0x7f3444158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f3443400640 1 --2- 172.21.15.47:0/3951308537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34440cb490 0x7f3444158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f3443400640 1 -- 172.21.15.47:0/3951308537 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342c009420 con 0x7f3444072680 2024-02-19T15:25:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f3443400640 1 --2- 172.21.15.47:0/3951308537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3444072680 0x7f3444152890 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f343400e930 tx=0x7f343400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f343b400640 1 -- 172.21.15.47:0/3951308537 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3434009690 con 0x7f3444072680 2024-02-19T15:25:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f343b400640 1 -- 172.21.15.47:0/3951308537 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343400ca00 con 0x7f3444072680 2024-02-19T15:25:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f344a800640 1 -- 172.21.15.47:0/3951308537 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f344415e120 con 0x7f3444072680 2024-02-19T15:25:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f343b400640 1 -- 172.21.15.47:0/3951308537 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343400cd70 con 0x7f3444072680 2024-02-19T15:25:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f344a800640 1 -- 172.21.15.47:0/3951308537 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f344415e920 con 0x7f3444072680 2024-02-19T15:25:52.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f344a800640 1 -- 172.21.15.47:0/3951308537 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3444075360 con 0x7f3444072680 2024-02-19T15:25:52.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.608+0000 7f343b400640 1 -- 172.21.15.47:0/3951308537 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3434013470 con 0x7f3444072680 2024-02-19T15:25:52.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.612+0000 7f343b400640 1 --2- 172.21.15.47:0/3951308537 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3420066e40 0x7f3420069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:52.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.612+0000 7f343b400640 1 -- 172.21.15.47:0/3951308537 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f34340849d0 con 0x7f3444072680 2024-02-19T15:25:52.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.612+0000 7f343be00640 1 --2- 172.21.15.47:0/3951308537 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3420066e40 0x7f3420069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:52.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.612+0000 7f343be00640 1 --2- 172.21.15.47:0/3951308537 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3420066e40 0x7f3420069300 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f342c0ae000 tx=0x7f342c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:52.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.616+0000 7f343b400640 1 -- 172.21.15.47:0/3951308537 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f3434051b70 con 0x7f3444072680 2024-02-19T15:25:52.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.780+0000 7f344a800640 1 -- 172.21.15.47:0/3951308537 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f34441538f0 con 0x7f3444072680 2024-02-19T15:25:52.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.780+0000 7f343b400640 1 -- 172.21.15.47:0/3951308537 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) v1 ==== 93+0+10 (secure 0 0 0) 0x7f34340170a0 con 0x7f3444072680 2024-02-19T15:25:52.784 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:25:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.780+0000 7f344a800640 1 -- 172.21.15.47:0/3951308537 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3420066e40 msgr2=0x7f3420069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.780+0000 7f344a800640 1 --2- 172.21.15.47:0/3951308537 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3420066e40 0x7f3420069300 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f342c0ae000 tx=0x7f342c008040 comp rx=0 tx=0).stop 2024-02-19T15:25:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.780+0000 7f344a800640 1 -- 172.21.15.47:0/3951308537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3444072680 msgr2=0x7f3444152890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.780+0000 7f344a800640 1 --2- 172.21.15.47:0/3951308537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3444072680 0x7f3444152890 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f343400e930 tx=0x7f343400edf0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.780+0000 7f344a800640 1 -- 172.21.15.47:0/3951308537 shutdown_connections 2024-02-19T15:25:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.780+0000 7f344a800640 1 --2- 172.21.15.47:0/3951308537 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3420066e40 0x7f3420069300 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.780+0000 7f344a800640 1 --2- 172.21.15.47:0/3951308537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f344415bd90 0x7f34440b2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.780+0000 7f344a800640 1 --2- 172.21.15.47:0/3951308537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34440cb490 0x7f3444158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.780+0000 7f344a800640 1 --2- 172.21.15.47:0/3951308537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3444072680 0x7f3444152890 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.780+0000 7f344a800640 1 -- 172.21.15.47:0/3951308537 >> 172.21.15.47:0/3951308537 conn(0x7f34440cfc20 msgr2=0x7f34440d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:52.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.780+0000 7f344a800640 1 -- 172.21.15.47:0/3951308537 shutdown_connections 2024-02-19T15:25:52.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.780+0000 7f344a800640 1 -- 172.21.15.47:0/3951308537 wait complete. 2024-02-19T15:25:52.805 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-02-19T15:25:52.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.940+0000 7fefa7e00640 1 Processor -- start 2024-02-19T15:25:52.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.940+0000 7fefa7e00640 1 -- start start 2024-02-19T15:25:52.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.940+0000 7fefa7e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fefa0071350 0x7fefa0165e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:52.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.940+0000 7fefa7e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa00718c0 con 0x7fefa015d6e0 2024-02-19T15:25:52.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.940+0000 7fefa7e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa0071ba0 con 0x7fefa015b1a0 2024-02-19T15:25:52.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.940+0000 7fefa7e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefa0071e80 con 0x7fefa0071350 2024-02-19T15:25:52.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.944+0000 7fefa5e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fefa0071350 0x7fefa0165e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:52.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.944+0000 7fefa5e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fefa0071350 0x7fefa0165e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48516/0 (socket says 172.21.15.47:48516) 2024-02-19T15:25:52.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.944+0000 7fefa4a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fefa015d6e0 0x7fefa0162740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52960/0 (socket says 172.21.15.47:52960) 2024-02-19T15:25:52.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.944+0000 7fefa5e00640 1 -- 172.21.15.47:0/1564064378 learned_addr learned my addr 172.21.15.47:0/1564064378 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:52.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.944+0000 7fefa5e00640 1 -- 172.21.15.47:0/1564064378 >> v1:172.21.15.116:6789/0 conn(0x7fefa015b1a0 legacy=0x7fefa015b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:52.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.944+0000 7fefa5e00640 1 -- 172.21.15.47:0/1564064378 >> v1:172.21.15.47:6789/0 conn(0x7fefa015d6e0 legacy=0x7fefa0162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:52.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.944+0000 7fefa5e00640 1 -- 172.21.15.47:0/1564064378 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa0072160 con 0x7fefa0071350 2024-02-19T15:25:52.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.944+0000 7fef9fe00640 1 -- 172.21.15.47:0/1564064378 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625863504 0 0) 0x7fefa00718c0 con 0x7fefa015d6e0 2024-02-19T15:25:52.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.944+0000 7fefa5e00640 1 --2- 172.21.15.47:0/1564064378 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefa0071350 0x7fefa0165e70 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fef9800d180 tx=0x7fef9801ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=725e97ae76f7e180 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:52.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.944+0000 7fef9fe00640 1 -- 172.21.15.47:0/1564064378 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9800b2d0 con 0x7fefa0071350 2024-02-19T15:25:52.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.944+0000 7fef9fe00640 1 -- 172.21.15.47:0/1564064378 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9800b430 con 0x7fefa0071350 2024-02-19T15:25:52.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.944+0000 7fef9fe00640 1 -- 172.21.15.47:0/1564064378 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef980a7c20 con 0x7fefa0071350 2024-02-19T15:25:52.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa7e00640 1 -- 172.21.15.47:0/1564064378 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefa0071350 msgr2=0x7fefa0165e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:52.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa7e00640 1 --2- 172.21.15.47:0/1564064378 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefa0071350 0x7fefa0165e70 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fef9800d180 tx=0x7fef9801ec10 comp rx=0 tx=0).stop 2024-02-19T15:25:52.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa7e00640 1 -- 172.21.15.47:0/1564064378 shutdown_connections 2024-02-19T15:25:52.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa7e00640 1 --2- 172.21.15.47:0/1564064378 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefa0071350 0x7fefa0165e70 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa7e00640 1 -- 172.21.15.47:0/1564064378 >> 172.21.15.47:0/1564064378 conn(0x7fefa00b16a0 msgr2=0x7fefa00b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:52.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa7e00640 1 -- 172.21.15.47:0/1564064378 shutdown_connections 2024-02-19T15:25:52.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa7e00640 1 -- 172.21.15.47:0/1564064378 wait complete. 2024-02-19T15:25:52.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa7e00640 1 Processor -- start 2024-02-19T15:25:52.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa7e00640 1 -- start start 2024-02-19T15:25:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa0071350 0x7fefa0141370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa7e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefa015b1a0 0x7fefa0155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefa015d6e0 0x7fefa0156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa7e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa0157610 con 0x7fefa015d6e0 2024-02-19T15:25:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa7e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa0156990 con 0x7fefa0071350 2024-02-19T15:25:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa7e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefa0156c70 con 0x7fefa015b1a0 2024-02-19T15:25:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa5e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefa015d6e0 0x7fefa0156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa5e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefa015d6e0 0x7fefa0156170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48532/0 (socket says 172.21.15.47:48532) 2024-02-19T15:25:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa5e00640 1 -- 172.21.15.47:0/4258947523 learned_addr learned my addr 172.21.15.47:0/4258947523 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa4a00640 1 --2- 172.21.15.47:0/4258947523 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefa015b1a0 0x7fefa0155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa5400640 1 --2- 172.21.15.47:0/4258947523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa0071350 0x7fefa0141370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa5e00640 1 -- 172.21.15.47:0/4258947523 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefa015b1a0 msgr2=0x7fefa0155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa5e00640 1 --2- 172.21.15.47:0/4258947523 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefa015b1a0 0x7fefa0155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa5e00640 1 -- 172.21.15.47:0/4258947523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa0071350 msgr2=0x7fefa0141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:52.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa5e00640 1 --2- 172.21.15.47:0/4258947523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa0071350 0x7fefa0141370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:52.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa5e00640 1 -- 172.21.15.47:0/4258947523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9800bd40 con 0x7fefa015d6e0 2024-02-19T15:25:52.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.948+0000 7fefa4a00640 1 --2- 172.21.15.47:0/4258947523 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefa015b1a0 0x7fefa0155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:52.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.952+0000 7fefa5e00640 1 --2- 172.21.15.47:0/4258947523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefa015d6e0 0x7fefa0156170 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fef9801b820 tx=0x7fef9801b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:52.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.952+0000 7fef9d600640 1 -- 172.21.15.47:0/4258947523 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef98009800 con 0x7fefa015d6e0 2024-02-19T15:25:52.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.952+0000 7fef9d600640 1 -- 172.21.15.47:0/4258947523 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef98009980 con 0x7fefa015d6e0 2024-02-19T15:25:52.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.952+0000 7fef9d600640 1 -- 172.21.15.47:0/4258947523 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef98009d30 con 0x7fefa015d6e0 2024-02-19T15:25:52.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.952+0000 7fefa7e00640 1 -- 172.21.15.47:0/4258947523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa0156f50 con 0x7fefa015d6e0 2024-02-19T15:25:52.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.952+0000 7fefa7e00640 1 -- 172.21.15.47:0/4258947523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa01566b0 con 0x7fefa015d6e0 2024-02-19T15:25:52.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.952+0000 7fefa7e00640 1 -- 172.21.15.47:0/4258947523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa0157750 con 0x7fefa015d6e0 2024-02-19T15:25:52.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.952+0000 7fef9d600640 1 -- 172.21.15.47:0/4258947523 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fef980c3460 con 0x7fefa015d6e0 2024-02-19T15:25:52.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.952+0000 7fef9d600640 1 --2- 172.21.15.47:0/4258947523 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef80066e40 0x7fef80069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:52.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.952+0000 7fef9d600640 1 -- 172.21.15.47:0/4258947523 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fef9811fea0 con 0x7fefa015d6e0 2024-02-19T15:25:52.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.952+0000 7fefa5400640 1 --2- 172.21.15.47:0/4258947523 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef80066e40 0x7fef80069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:52.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.956+0000 7fefa5400640 1 --2- 172.21.15.47:0/4258947523 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef80066e40 0x7fef80069300 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fef94000f70 tx=0x7fef94009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:52.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:52.956+0000 7fef9d600640 1 -- 172.21.15.47:0/4258947523 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fefa0157750 con 0x7fefa015d6e0 2024-02-19T15:25:53.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:53.176+0000 7fefa7e00640 1 -- 172.21.15.47:0/4258947523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7fefa0157750 con 0x7fefa015d6e0 2024-02-19T15:25:54.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.088+0000 7fef9d600640 1 -- 172.21.15.47:0/4258947523 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v221) v1 ==== 135+0+0 (secure 0 0 0) 0x7fefa0157750 con 0x7fefa015d6e0 2024-02-19T15:25:54.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.092+0000 7fefa7e00640 1 -- 172.21.15.47:0/4258947523 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef80066e40 msgr2=0x7fef80069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:54.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.092+0000 7fefa7e00640 1 --2- 172.21.15.47:0/4258947523 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef80066e40 0x7fef80069300 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fef94000f70 tx=0x7fef94009040 comp rx=0 tx=0).stop 2024-02-19T15:25:54.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.092+0000 7fefa7e00640 1 -- 172.21.15.47:0/4258947523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefa015d6e0 msgr2=0x7fefa0156170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:54.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.092+0000 7fefa7e00640 1 --2- 172.21.15.47:0/4258947523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefa015d6e0 0x7fefa0156170 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fef9801b820 tx=0x7fef9801b9b0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.092+0000 7fefa7e00640 1 -- 172.21.15.47:0/4258947523 shutdown_connections 2024-02-19T15:25:54.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.092+0000 7fefa7e00640 1 --2- 172.21.15.47:0/4258947523 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef80066e40 0x7fef80069300 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.092+0000 7fefa7e00640 1 --2- 172.21.15.47:0/4258947523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fefa015d6e0 0x7fefa0156170 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.092+0000 7fefa7e00640 1 --2- 172.21.15.47:0/4258947523 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefa015b1a0 0x7fefa0155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.092+0000 7fefa7e00640 1 --2- 172.21.15.47:0/4258947523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa0071350 0x7fefa0141370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.092+0000 7fefa7e00640 1 -- 172.21.15.47:0/4258947523 >> 172.21.15.47:0/4258947523 conn(0x7fefa00b16a0 msgr2=0x7fefa00cf0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:54.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.092+0000 7fefa7e00640 1 -- 172.21.15.47:0/4258947523 shutdown_connections 2024-02-19T15:25:54.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.092+0000 7fefa7e00640 1 -- 172.21.15.47:0/4258947523 wait complete. 2024-02-19T15:25:54.120 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-19T15:25:54.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.264+0000 7f222ca00640 1 Processor -- start 2024-02-19T15:25:54.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.264+0000 7f222ca00640 1 -- start start 2024-02-19T15:25:54.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.264+0000 7f222ca00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f22280ce900 0x7f2228070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:54.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.264+0000 7f222ca00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f222806ea50 0x7f222806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:54.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.264+0000 7f222ca00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22280d0b00 con 0x7f222806dd60 2024-02-19T15:25:54.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.264+0000 7f222ca00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f222806ca60 con 0x7f22280ce900 2024-02-19T15:25:54.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.264+0000 7f222ca00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f222806cd40 con 0x7f222806ea50 2024-02-19T15:25:54.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.264+0000 7f2225e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f22280ce900 0x7f2228070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:54.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.264+0000 7f221d400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f222806ea50 0x7f222806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:54.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.264+0000 7f221d400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f222806ea50 0x7f222806d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:34636/0 (socket says 172.21.15.47:34636) 2024-02-19T15:25:54.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.264+0000 7f221d400640 1 -- 172.21.15.47:0/1093093135 learned_addr learned my addr 172.21.15.47:0/1093093135 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:54.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.264+0000 7f2225e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f22280ce900 0x7f2228070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:53702/0 (socket says 172.21.15.47:53702) 2024-02-19T15:25:54.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.264+0000 7f2226800640 1 --1- 172.21.15.47:0/1093093135 >> v1:172.21.15.116:6789/0 conn(0x7f222806dd60 0x7f22280cd7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:56540/0 (socket says 172.21.15.47:56540) 2024-02-19T15:25:54.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.264+0000 7f221d400640 1 -- 172.21.15.47:0/1093093135 >> v2:172.21.15.116:3300/0 conn(0x7f22280ce900 msgr2=0x7f2228070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:54.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.264+0000 7f221d400640 1 --2- 172.21.15.47:0/1093093135 >> v2:172.21.15.116:3300/0 conn(0x7f22280ce900 0x7f2228070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.264+0000 7f221d400640 1 -- 172.21.15.47:0/1093093135 >> v1:172.21.15.116:6789/0 conn(0x7f222806dd60 legacy=0x7f22280cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:54.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.264+0000 7f221d400640 1 -- 172.21.15.47:0/1093093135 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222806d020 con 0x7f222806ea50 2024-02-19T15:25:54.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.268+0000 7f221d400640 1 --2- 172.21.15.47:0/1093093135 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f222806ea50 0x7f222806d820 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2210009870 tx=0x7f221001c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb61d99d1ca801f6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:54.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.268+0000 7f2225400640 1 -- 172.21.15.47:0/1093093135 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22100ab670 con 0x7f222806ea50 2024-02-19T15:25:54.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.268+0000 7f2225400640 1 -- 172.21.15.47:0/1093093135 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22100ab7d0 con 0x7f222806ea50 2024-02-19T15:25:54.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.268+0000 7f2225400640 1 -- 172.21.15.47:0/1093093135 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22100abb60 con 0x7f222806ea50 2024-02-19T15:25:54.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.268+0000 7f222ca00640 1 -- 172.21.15.47:0/1093093135 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f222806ea50 msgr2=0x7f222806d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.268+0000 7f222ca00640 1 --2- 172.21.15.47:0/1093093135 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f222806ea50 0x7f222806d820 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2210009870 tx=0x7f221001c4d0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.268+0000 7f222ca00640 1 -- 172.21.15.47:0/1093093135 shutdown_connections 2024-02-19T15:25:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.268+0000 7f222ca00640 1 --2- 172.21.15.47:0/1093093135 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f222806ea50 0x7f222806d820 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.268+0000 7f222ca00640 1 --2- 172.21.15.47:0/1093093135 >> v2:172.21.15.116:3300/0 conn(0x7f22280ce900 0x7f2228070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.268+0000 7f222ca00640 1 -- 172.21.15.47:0/1093093135 >> 172.21.15.47:0/1093093135 conn(0x7f22280cfc20 msgr2=0x7f22280d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.268+0000 7f222ca00640 1 -- 172.21.15.47:0/1093093135 shutdown_connections 2024-02-19T15:25:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.268+0000 7f222ca00640 1 -- 172.21.15.47:0/1093093135 wait complete. 2024-02-19T15:25:54.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f222ca00640 1 Processor -- start 2024-02-19T15:25:54.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f222ca00640 1 -- start start 2024-02-19T15:25:54.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f222ca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f222806dd60 0x7f2228142d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:54.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f222ca00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f222806ea50 0x7f22281545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:54.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f222ca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22280ce900 0x7f22280c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:54.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f222ca00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22281507a0 con 0x7f222806dd60 2024-02-19T15:25:54.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f222ca00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2228151d90 con 0x7f22280ce900 2024-02-19T15:25:54.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f222ca00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22280c5940 con 0x7f222806ea50 2024-02-19T15:25:54.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f2225e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f222806dd60 0x7f2228142d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:54.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f2225e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f222806dd60 0x7f2228142d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45866/0 (socket says 172.21.15.47:45866) 2024-02-19T15:25:54.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f2225e00640 1 -- 172.21.15.47:0/1418211199 learned_addr learned my addr 172.21.15.47:0/1418211199 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:54.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f221d400640 1 --2- 172.21.15.47:0/1418211199 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f222806ea50 0x7f22281545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:54.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f2226800640 1 --2- 172.21.15.47:0/1418211199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22280ce900 0x7f22280c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:54.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f2225e00640 1 -- 172.21.15.47:0/1418211199 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f222806ea50 msgr2=0x7f22281545d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:54.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f2225e00640 1 --2- 172.21.15.47:0/1418211199 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f222806ea50 0x7f22281545d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f2225e00640 1 -- 172.21.15.47:0/1418211199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22280ce900 msgr2=0x7f22280c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:54.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f2225e00640 1 --2- 172.21.15.47:0/1418211199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22280ce900 0x7f22280c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f2225e00640 1 -- 172.21.15.47:0/1418211199 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22100094d0 con 0x7f222806dd60 2024-02-19T15:25:54.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f2225e00640 1 --2- 172.21.15.47:0/1418211199 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f222806dd60 0x7f2228142d20 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f221800efc0 tx=0x7f221800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:54.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f221ea00640 1 -- 172.21.15.47:0/1418211199 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2218009b80 con 0x7f222806dd60 2024-02-19T15:25:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f221ea00640 1 -- 172.21.15.47:0/1418211199 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f221800f040 con 0x7f222806dd60 2024-02-19T15:25:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f221ea00640 1 -- 172.21.15.47:0/1418211199 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22180146a0 con 0x7f222806dd60 2024-02-19T15:25:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f222ca00640 1 -- 172.21.15.47:0/1418211199 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22280c5c20 con 0x7f222806dd60 2024-02-19T15:25:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.272+0000 7f222ca00640 1 -- 172.21.15.47:0/1418211199 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22280c61e0 con 0x7f222806dd60 2024-02-19T15:25:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.276+0000 7f222ca00640 1 -- 172.21.15.47:0/1418211199 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2228075360 con 0x7f222806dd60 2024-02-19T15:25:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.276+0000 7f221ea00640 1 -- 172.21.15.47:0/1418211199 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2218014800 con 0x7f222806dd60 2024-02-19T15:25:54.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.276+0000 7f221ea00640 1 --2- 172.21.15.47:0/1418211199 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2204066e40 0x7f2204069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:54.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.276+0000 7f221ea00640 1 -- 172.21.15.47:0/1418211199 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2218085180 con 0x7f222806dd60 2024-02-19T15:25:54.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.276+0000 7f221d400640 1 --2- 172.21.15.47:0/1418211199 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2204066e40 0x7f2204069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:54.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.276+0000 7f221d400640 1 --2- 172.21.15.47:0/1418211199 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2204066e40 0x7f2204069300 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f22100af000 tx=0x7f22100b6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:54.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.280+0000 7f221ea00640 1 -- 172.21.15.47:0/1418211199 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2218052280 con 0x7f222806dd60 2024-02-19T15:25:54.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.444+0000 7f222ca00640 1 -- 172.21.15.47:0/1418211199 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f22280c64f0 con 0x7f222806dd60 2024-02-19T15:25:54.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.448+0000 7f221ea00640 1 -- 172.21.15.47:0/1418211199 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2218055940 con 0x7f222806dd60 2024-02-19T15:25:54.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.448+0000 7f222ca00640 1 -- 172.21.15.47:0/1418211199 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2204066e40 msgr2=0x7f2204069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:54.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.448+0000 7f222ca00640 1 --2- 172.21.15.47:0/1418211199 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2204066e40 0x7f2204069300 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f22100af000 tx=0x7f22100b6040 comp rx=0 tx=0).stop 2024-02-19T15:25:54.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.448+0000 7f222ca00640 1 -- 172.21.15.47:0/1418211199 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f222806dd60 msgr2=0x7f2228142d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:54.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.448+0000 7f222ca00640 1 --2- 172.21.15.47:0/1418211199 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f222806dd60 0x7f2228142d20 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f221800efc0 tx=0x7f221800c490 comp rx=0 tx=0).stop 2024-02-19T15:25:54.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.448+0000 7f222ca00640 1 -- 172.21.15.47:0/1418211199 shutdown_connections 2024-02-19T15:25:54.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.448+0000 7f222ca00640 1 --2- 172.21.15.47:0/1418211199 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2204066e40 0x7f2204069300 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.448+0000 7f222ca00640 1 --2- 172.21.15.47:0/1418211199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22280ce900 0x7f22280c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.448+0000 7f222ca00640 1 --2- 172.21.15.47:0/1418211199 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f222806ea50 0x7f22281545d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.448+0000 7f222ca00640 1 --2- 172.21.15.47:0/1418211199 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f222806dd60 0x7f2228142d20 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.448+0000 7f222ca00640 1 -- 172.21.15.47:0/1418211199 >> 172.21.15.47:0/1418211199 conn(0x7f22280cfc20 msgr2=0x7f222815cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:54.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.448+0000 7f222ca00640 1 -- 172.21.15.47:0/1418211199 shutdown_connections 2024-02-19T15:25:54.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.448+0000 7f222ca00640 1 -- 172.21.15.47:0/1418211199 wait complete. 2024-02-19T15:25:54.473 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T15:25:54.473 DEBUG:teuthology.orchestra.run.smithi047:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:25:54.553 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:25:54.641 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T15:25:54.641 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:25:54.641 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:25:54.650 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:25:54.650 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:25:54.650 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:25:54.659 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:25:54.659 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:25:54.660 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:25:54.702 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:25:54.702 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:25:54.702 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:25:54.711 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:25:54.711 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-02-19T15:25:54.711 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:25:54.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbef6a00640 1 Processor -- start 2024-02-19T15:25:54.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbef6a00640 1 -- start start 2024-02-19T15:25:54.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbef6a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbef00cea80 0x7fbef00727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:54.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbef6a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef015d280 con 0x7fbef015c390 2024-02-19T15:25:54.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbef6a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef0070ff0 con 0x7fbef0145590 2024-02-19T15:25:54.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbef6a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef00709e0 con 0x7fbef00cea80 2024-02-19T15:25:54.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbef4a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbef015c390 0x7fbef015c820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36002/0 (socket says 172.21.15.47:36002) 2024-02-19T15:25:54.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbef4a00640 1 -- 172.21.15.47:0/3737501758 learned_addr learned my addr 172.21.15.47:0/3737501758 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:54.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeef400640 1 --2- 172.21.15.47:0/3737501758 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbef00cea80 0x7fbef00727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:54.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeeea00640 1 -- 172.21.15.47:0/3737501758 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3619671375 0 0) 0x7fbef015d280 con 0x7fbef015c390 2024-02-19T15:25:54.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeeea00640 1 -- 172.21.15.47:0/3737501758 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed4003530 con 0x7fbef015c390 2024-02-19T15:25:54.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeeea00640 1 -- 172.21.15.47:0/3737501758 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 248048844 0 0) 0x7fbed4003530 con 0x7fbef015c390 2024-02-19T15:25:54.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeeea00640 1 -- 172.21.15.47:0/3737501758 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbef00cea80 msgr2=0x7fbef00727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:54.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeeea00640 1 --2- 172.21.15.47:0/3737501758 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbef00cea80 0x7fbef00727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeeea00640 1 -- 172.21.15.47:0/3737501758 >> v1:172.21.15.116:6789/0 conn(0x7fbef0145590 legacy=0x7fbef006c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeeea00640 1 -- 172.21.15.47:0/3737501758 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef00cf020 con 0x7fbef015c390 2024-02-19T15:25:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeeea00640 1 -- 172.21.15.47:0/3737501758 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7fbee8001a00 con 0x7fbef015c390 2024-02-19T15:25:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeeea00640 1 -- 172.21.15.47:0/3737501758 >> v1:172.21.15.47:6789/0 conn(0x7fbef015c390 legacy=0x7fbef015c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeeea00640 1 --2- 172.21.15.47:0/3737501758 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbef00cea80 0x7fbed40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeeea00640 1 --2- 172.21.15.47:0/3737501758 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbed4003ef0 0x7fbed40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeeea00640 1 --2- 172.21.15.47:0/3737501758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4004d10 0x7fbed4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeeea00640 1 -- 172.21.15.47:0/3737501758 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef00709e0 con 0x7fbed4003ef0 2024-02-19T15:25:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeeea00640 1 -- 172.21.15.47:0/3737501758 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed400b9f0 con 0x7fbed4004d10 2024-02-19T15:25:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeeea00640 1 -- 172.21.15.47:0/3737501758 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbed400bbb0 con 0x7fbef00cea80 2024-02-19T15:25:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeef400640 1 --2- 172.21.15.47:0/3737501758 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbed4003ef0 0x7fbed40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeefe00640 1 --2- 172.21.15.47:0/3737501758 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbef00cea80 0x7fbed40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbef4a00640 1 --2- 172.21.15.47:0/3737501758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4004d10 0x7fbed4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeef400640 1 -- 172.21.15.47:0/3737501758 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbef00cea80 msgr2=0x7fbed40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:54.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeef400640 1 --2- 172.21.15.47:0/3737501758 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbef00cea80 0x7fbed40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeef400640 1 -- 172.21.15.47:0/3737501758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4004d10 msgr2=0x7fbed4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:54.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeef400640 1 --2- 172.21.15.47:0/3737501758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4004d10 0x7fbed4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbeef400640 1 -- 172.21.15.47:0/3737501758 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed4003530 con 0x7fbed4003ef0 2024-02-19T15:25:54.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.884+0000 7fbef4a00640 1 --2- 172.21.15.47:0/3737501758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4004d10 0x7fbed4005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:54.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.888+0000 7fbeef400640 1 --2- 172.21.15.47:0/3737501758 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbed4003ef0 0x7fbed40047d0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fbedc009ae0 tx=0x7fbedc01c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92ee96008fbbffe7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:54.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.888+0000 7fbeeea00640 1 -- 172.21.15.47:0/3737501758 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc0a55a0 con 0x7fbed4003ef0 2024-02-19T15:25:54.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.888+0000 7fbeeea00640 1 -- 172.21.15.47:0/3737501758 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbedc0a5700 con 0x7fbed4003ef0 2024-02-19T15:25:54.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.888+0000 7fbeeea00640 1 -- 172.21.15.47:0/3737501758 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc0a5ab0 con 0x7fbed4003ef0 2024-02-19T15:25:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.888+0000 7fbef6a00640 1 -- 172.21.15.47:0/3737501758 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbed4003ef0 msgr2=0x7fbed40047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.888+0000 7fbef6a00640 1 --2- 172.21.15.47:0/3737501758 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbed4003ef0 0x7fbed40047d0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fbedc009ae0 tx=0x7fbedc01c9b0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.888+0000 7fbef4a00640 1 -- 172.21.15.47:0/3737501758 reap_dead start 2024-02-19T15:25:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.888+0000 7fbef6a00640 1 -- 172.21.15.47:0/3737501758 shutdown_connections 2024-02-19T15:25:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.888+0000 7fbef6a00640 1 -- 172.21.15.47:0/3737501758 >> 172.21.15.47:0/3737501758 conn(0x7fbef00b16a0 msgr2=0x7fbef00b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.888+0000 7fbef6a00640 1 -- 172.21.15.47:0/3737501758 shutdown_connections 2024-02-19T15:25:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.888+0000 7fbef6a00640 1 -- 172.21.15.47:0/3737501758 wait complete. 2024-02-19T15:25:54.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbef6a00640 1 Processor -- start 2024-02-19T15:25:54.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbef6a00640 1 -- start start 2024-02-19T15:25:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbef6a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4003ef0 0x7fbef014e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbef6a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbef0143360 0x7fbef00b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbef6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbef01437e0 0x7fbef00b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbef6a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef0144a30 con 0x7fbef01437e0 2024-02-19T15:25:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbef6a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef0144e50 con 0x7fbed4003ef0 2024-02-19T15:25:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbef6a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef0143db0 con 0x7fbef0143360 2024-02-19T15:25:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbef4a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbef01437e0 0x7fbef00b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbef4a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbef01437e0 0x7fbef00b3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45880/0 (socket says 172.21.15.47:45880) 2024-02-19T15:25:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbef4a00640 1 -- 172.21.15.47:0/3750439676 learned_addr learned my addr 172.21.15.47:0/3750439676 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbeef400640 1 --2- 172.21.15.47:0/3750439676 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbef0143360 0x7fbef00b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbeefe00640 1 --2- 172.21.15.47:0/3750439676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4003ef0 0x7fbef014e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbef4a00640 1 -- 172.21.15.47:0/3750439676 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbef0143360 msgr2=0x7fbef00b36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:54.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbef4a00640 1 --2- 172.21.15.47:0/3750439676 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbef0143360 0x7fbef00b36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbef4a00640 1 -- 172.21.15.47:0/3750439676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4003ef0 msgr2=0x7fbef014e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:54.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbef4a00640 1 --2- 172.21.15.47:0/3750439676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4003ef0 0x7fbef014e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:54.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbef4a00640 1 -- 172.21.15.47:0/3750439676 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef0144090 con 0x7fbef01437e0 2024-02-19T15:25:54.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbeef400640 1 --2- 172.21.15.47:0/3750439676 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbef0143360 0x7fbef00b36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:25:54.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbeefe00640 1 --2- 172.21.15.47:0/3750439676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4003ef0 0x7fbef014e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:25:54.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbef4a00640 1 --2- 172.21.15.47:0/3750439676 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbef01437e0 0x7fbef00b3c20 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fbef0068880 tx=0x7fbee800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:54.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbee7e00640 1 -- 172.21.15.47:0/3750439676 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee800ed50 con 0x7fbef01437e0 2024-02-19T15:25:54.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbee7e00640 1 -- 172.21.15.47:0/3750439676 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee800eed0 con 0x7fbef01437e0 2024-02-19T15:25:54.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbef6a00640 1 -- 172.21.15.47:0/3750439676 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef0144370 con 0x7fbef01437e0 2024-02-19T15:25:54.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbee7e00640 1 -- 172.21.15.47:0/3750439676 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee8014670 con 0x7fbef01437e0 2024-02-19T15:25:54.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbef6a00640 1 -- 172.21.15.47:0/3750439676 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef014ed30 con 0x7fbef01437e0 2024-02-19T15:25:54.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.892+0000 7fbef6a00640 1 -- 172.21.15.47:0/3750439676 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef0144b70 con 0x7fbef01437e0 2024-02-19T15:25:54.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.896+0000 7fbee7e00640 1 -- 172.21.15.47:0/3750439676 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fbee80147d0 con 0x7fbef01437e0 2024-02-19T15:25:54.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.896+0000 7fbee7e00640 1 --2- 172.21.15.47:0/3750439676 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbed0066e40 0x7fbed0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:54.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.896+0000 7fbee7e00640 1 -- 172.21.15.47:0/3750439676 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbee80845f0 con 0x7fbef01437e0 2024-02-19T15:25:54.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.896+0000 7fbeefe00640 1 --2- 172.21.15.47:0/3750439676 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbed0066e40 0x7fbed0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:54.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.896+0000 7fbeefe00640 1 --2- 172.21.15.47:0/3750439676 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbed0066e40 0x7fbed0069300 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbee0007990 tx=0x7fbee0007920 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:54.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:54.900+0000 7fbee7e00640 1 -- 172.21.15.47:0/3750439676 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fbef0144b70 con 0x7fbef01437e0 2024-02-19T15:25:55.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.096+0000 7fbef6a00640 1 -- 172.21.15.47:0/3750439676 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbef0144b70 con 0x7fbef01437e0 2024-02-19T15:25:55.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.096+0000 7fbee7e00640 1 -- 172.21.15.47:0/3750439676 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+3705 (secure 0 0 0) 0x7fbef0144b70 con 0x7fbef01437e0 2024-02-19T15:25:55.102 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:55.102 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":42,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5684,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3206627105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3206627105},{"type":"v1","addr":"172.21.15.116:6836","nonce":3206627105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5687,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3532185042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":3532185042},{"type":"v1","addr":"172.21.15.116:6837","nonce":3532185042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/1482478568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1482478568},{"type":"v1","addr":"172.21.15.47:6834","nonce":1482478568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6687,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3099846295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3099846295},{"type":"v1","addr":"172.21.15.47:6835","nonce":3099846295}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[]} 2024-02-19T15:25:55.102 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 42 2024-02-19T15:25:55.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.100+0000 7fbef6a00640 1 -- 172.21.15.47:0/3750439676 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbed0066e40 msgr2=0x7fbed0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:55.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.100+0000 7fbef6a00640 1 --2- 172.21.15.47:0/3750439676 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbed0066e40 0x7fbed0069300 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbee0007990 tx=0x7fbee0007920 comp rx=0 tx=0).stop 2024-02-19T15:25:55.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.100+0000 7fbef6a00640 1 -- 172.21.15.47:0/3750439676 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbef01437e0 msgr2=0x7fbef00b3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:55.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.100+0000 7fbef6a00640 1 --2- 172.21.15.47:0/3750439676 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbef01437e0 0x7fbef00b3c20 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fbef0068880 tx=0x7fbee800e530 comp rx=0 tx=0).stop 2024-02-19T15:25:55.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.100+0000 7fbef6a00640 1 -- 172.21.15.47:0/3750439676 shutdown_connections 2024-02-19T15:25:55.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.100+0000 7fbef6a00640 1 --2- 172.21.15.47:0/3750439676 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbed0066e40 0x7fbed0069300 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:55.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.100+0000 7fbef6a00640 1 --2- 172.21.15.47:0/3750439676 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbef01437e0 0x7fbef00b3c20 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:55.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.100+0000 7fbef6a00640 1 --2- 172.21.15.47:0/3750439676 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbef0143360 0x7fbef00b36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:55.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.100+0000 7fbef6a00640 1 --2- 172.21.15.47:0/3750439676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4003ef0 0x7fbef014e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:55.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.100+0000 7fbef6a00640 1 -- 172.21.15.47:0/3750439676 >> 172.21.15.47:0/3750439676 conn(0x7fbef00b16a0 msgr2=0x7fbef00aead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:55.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.100+0000 7fbef6a00640 1 -- 172.21.15.47:0/3750439676 shutdown_connections 2024-02-19T15:25:55.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.100+0000 7fbef6a00640 1 -- 172.21.15.47:0/3750439676 wait complete. 2024-02-19T15:25:55.123 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T15:25:55.123 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-19T15:25:55.123 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T15:25:55.123 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:25:55.123 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T15:25:55.123 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-19T15:25:55.124 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T15:25:55.124 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:25:55.124 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T15:25:55.124 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-19T15:25:55.124 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T15:25:55.124 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:25:55.124 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T15:25:55.124 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-19T15:25:55.125 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T15:25:55.125 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:25:55.136 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T15:25:55.136 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T15:25:55.137 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T15:25:55.137 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T15:25:55.144 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T15:25:55.144 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T15:25:55.145 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T15:25:55.145 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T15:25:55.181 INFO:tasks.ceph.mds.a:Started 2024-02-19T15:25:55.181 DEBUG:teuthology.parallel:result is None 2024-02-19T15:25:55.182 INFO:tasks.ceph.mds.c:Started 2024-02-19T15:25:55.183 DEBUG:teuthology.parallel:result is None 2024-02-19T15:25:55.193 INFO:tasks.ceph.mds.d:Started 2024-02-19T15:25:55.193 DEBUG:teuthology.parallel:result is None 2024-02-19T15:25:55.194 INFO:tasks.ceph.mds.b:Started 2024-02-19T15:25:55.195 DEBUG:teuthology.parallel:result is None 2024-02-19T15:25:55.195 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-19T15:25:55.267 INFO:tasks.ceph.mds.a.smithi047.stdout:starting mds.a at 2024-02-19T15:25:55.269 INFO:tasks.ceph.mds.c.smithi047.stdout:starting mds.c at 2024-02-19T15:25:55.285 INFO:tasks.ceph.mds.b.smithi116.stdout:starting mds.b at 2024-02-19T15:25:55.285 INFO:tasks.ceph.mds.d.smithi116.stdout:starting mds.d at 2024-02-19T15:25:55.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.340+0000 7f1618200640 1 Processor -- start 2024-02-19T15:25:55.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.340+0000 7f1618200640 1 -- start start 2024-02-19T15:25:55.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.340+0000 7f1618200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f161015bcd0 0x7f16100cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:55.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.340+0000 7f1618200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f16100cedb0 0x7f16100cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:55.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.340+0000 7f1618200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f16100d2710 0x7f16100d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:55.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.340+0000 7f1618200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16100cf5e0 con 0x7f16100d2710 2024-02-19T15:25:55.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.340+0000 7f1618200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16100cc700 con 0x7f16100cedb0 2024-02-19T15:25:55.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.340+0000 7f1618200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16100cf300 con 0x7f161015bcd0 2024-02-19T15:25:55.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.340+0000 7f1616200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f16100d2710 0x7f16100d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:55.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.340+0000 7f1616200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f16100d2710 0x7f16100d0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45926/0 (socket says 172.21.15.47:45926) 2024-02-19T15:25:55.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.340+0000 7f1616200640 1 -- 172.21.15.47:0/2849118612 learned_addr learned my addr 172.21.15.47:0/2849118612 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:55.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.340+0000 7f1614e00640 1 --2- 172.21.15.47:0/2849118612 >> v2:172.21.15.116:3300/0 conn(0x7f16100cedb0 0x7f16100cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:55.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.340+0000 7f1615800640 1 --2- 172.21.15.47:0/2849118612 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f161015bcd0 0x7f16100cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:55.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.340+0000 7f1616200640 1 -- 172.21.15.47:0/2849118612 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f161015bcd0 msgr2=0x7f16100cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:55.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.340+0000 7f1616200640 1 --2- 172.21.15.47:0/2849118612 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f161015bcd0 0x7f16100cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:55.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.340+0000 7f1616200640 1 -- 172.21.15.47:0/2849118612 >> v2:172.21.15.116:3300/0 conn(0x7f16100cedb0 msgr2=0x7f16100cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:55.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.340+0000 7f1616200640 1 --2- 172.21.15.47:0/2849118612 >> v2:172.21.15.116:3300/0 conn(0x7f16100cedb0 0x7f16100cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:55.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.340+0000 7f1616200640 1 -- 172.21.15.47:0/2849118612 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16100cc9a0 con 0x7f16100d2710 2024-02-19T15:25:55.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.344+0000 7f1616200640 1 --2- 172.21.15.47:0/2849118612 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16100d2710 0x7f16100d0190 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f161006bc80 tx=0x7f160801c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb36842783663dbc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:55.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.344+0000 7f160fe00640 1 -- 172.21.15.47:0/2849118612 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16080a5840 con 0x7f16100d2710 2024-02-19T15:25:55.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.344+0000 7f160fe00640 1 -- 172.21.15.47:0/2849118612 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16080a59a0 con 0x7f16100d2710 2024-02-19T15:25:55.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.344+0000 7f160fe00640 1 -- 172.21.15.47:0/2849118612 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16080a5d30 con 0x7f16100d2710 2024-02-19T15:25:55.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.344+0000 7f1618200640 1 -- 172.21.15.47:0/2849118612 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16100d2710 msgr2=0x7f16100d0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:55.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.344+0000 7f1618200640 1 --2- 172.21.15.47:0/2849118612 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16100d2710 0x7f16100d0190 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f161006bc80 tx=0x7f160801c9c0 comp rx=0 tx=0).stop 2024-02-19T15:25:55.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.344+0000 7f1618200640 1 -- 172.21.15.47:0/2849118612 shutdown_connections 2024-02-19T15:25:55.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.344+0000 7f1618200640 1 --2- 172.21.15.47:0/2849118612 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16100d2710 0x7f16100d0190 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:55.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.344+0000 7f1618200640 1 --2- 172.21.15.47:0/2849118612 >> v2:172.21.15.116:3300/0 conn(0x7f16100cedb0 0x7f16100cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:55.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.344+0000 7f1618200640 1 --2- 172.21.15.47:0/2849118612 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f161015bcd0 0x7f16100cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:55.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.344+0000 7f1618200640 1 -- 172.21.15.47:0/2849118612 >> 172.21.15.47:0/2849118612 conn(0x7f16100d1bf0 msgr2=0x7f16100d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:55.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.344+0000 7f1618200640 1 -- 172.21.15.47:0/2849118612 shutdown_connections 2024-02-19T15:25:55.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.344+0000 7f1618200640 1 -- 172.21.15.47:0/2849118612 wait complete. 2024-02-19T15:25:55.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1618200640 1 Processor -- start 2024-02-19T15:25:55.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1618200640 1 -- start start 2024-02-19T15:25:55.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1618200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16100cedb0 0x7f161014dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:55.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1618200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16100d2710 0x7f1610157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:55.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1618200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f161015bcd0 0x7f1610157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:55.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1618200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1610159670 con 0x7f16100d2710 2024-02-19T15:25:55.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1618200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1610156820 con 0x7f161015bcd0 2024-02-19T15:25:55.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1618200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1610156b00 con 0x7f16100cedb0 2024-02-19T15:25:55.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1614e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16100d2710 0x7f1610157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:55.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1614e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16100d2710 0x7f1610157050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45934/0 (socket says 172.21.15.47:45934) 2024-02-19T15:25:55.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1614e00640 1 -- 172.21.15.47:0/2702595134 learned_addr learned my addr 172.21.15.47:0/2702595134 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:55.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1615800640 1 --2- 172.21.15.47:0/2702595134 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16100cedb0 0x7f161014dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:55.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1616200640 1 --2- 172.21.15.47:0/2702595134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f161015bcd0 0x7f1610157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:55.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1614e00640 1 -- 172.21.15.47:0/2702595134 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16100cedb0 msgr2=0x7f161014dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:55.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1614e00640 1 --2- 172.21.15.47:0/2702595134 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16100cedb0 0x7f161014dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:55.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1614e00640 1 -- 172.21.15.47:0/2702595134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f161015bcd0 msgr2=0x7f1610157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:55.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1614e00640 1 --2- 172.21.15.47:0/2702595134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f161015bcd0 0x7f1610157590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:55.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1614e00640 1 -- 172.21.15.47:0/2702595134 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1608009b30 con 0x7f16100d2710 2024-02-19T15:25:55.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1614e00640 1 --2- 172.21.15.47:0/2702595134 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16100d2710 0x7f1610157050 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f15fc00d800 tx=0x7f15fc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:55.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f160d600640 1 -- 172.21.15.47:0/2702595134 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15fc00b6c0 con 0x7f16100d2710 2024-02-19T15:25:55.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f160d600640 1 -- 172.21.15.47:0/2702595134 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15fc00b820 con 0x7f16100d2710 2024-02-19T15:25:55.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f160d600640 1 -- 172.21.15.47:0/2702595134 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15fc00bb90 con 0x7f16100d2710 2024-02-19T15:25:55.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1618200640 1 -- 172.21.15.47:0/2702595134 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16100cdb40 con 0x7f16100d2710 2024-02-19T15:25:55.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1618200640 1 -- 172.21.15.47:0/2702595134 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16100c8cc0 con 0x7f16100d2710 2024-02-19T15:25:55.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.348+0000 7f1618200640 1 -- 172.21.15.47:0/2702595134 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16101597b0 con 0x7f16100d2710 2024-02-19T15:25:55.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.352+0000 7f160d600640 1 -- 172.21.15.47:0/2702595134 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f15fc00bcf0 con 0x7f16100d2710 2024-02-19T15:25:55.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.352+0000 7f160d600640 1 --2- 172.21.15.47:0/2702595134 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f15f0066e40 0x7f15f0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:55.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.352+0000 7f160d600640 1 -- 172.21.15.47:0/2702595134 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f15fc084bc0 con 0x7f16100d2710 2024-02-19T15:25:55.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.352+0000 7f1615800640 1 --2- 172.21.15.47:0/2702595134 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f15f0066e40 0x7f15f0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:55.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.352+0000 7f1615800640 1 --2- 172.21.15.47:0/2702595134 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f15f0066e40 0x7f15f0069300 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1604007940 tx=0x7f1604008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:55.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.356+0000 7f160d600640 1 -- 172.21.15.47:0/2702595134 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f16101597b0 con 0x7f16100d2710 2024-02-19T15:25:55.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:55.532+0000 7f1618200640 1 -- 172.21.15.47:0/2702595134 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f16101597b0 con 0x7f16100d2710 2024-02-19T15:25:56.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.104+0000 7f160d600640 1 -- 172.21.15.47:0/2702595134 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v76) v1 ==== 97+0+0 (secure 0 0 0) 0x7f16101597b0 con 0x7f16100d2710 2024-02-19T15:25:56.110 INFO:teuthology.orchestra.run.smithi047.stderr: removed all blocklist entries 2024-02-19T15:25:56.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.108+0000 7f1618200640 1 -- 172.21.15.47:0/2702595134 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f15f0066e40 msgr2=0x7f15f0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:56.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.108+0000 7f1618200640 1 --2- 172.21.15.47:0/2702595134 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f15f0066e40 0x7f15f0069300 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1604007940 tx=0x7f1604008040 comp rx=0 tx=0).stop 2024-02-19T15:25:56.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.108+0000 7f1618200640 1 -- 172.21.15.47:0/2702595134 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16100d2710 msgr2=0x7f1610157050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:56.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.108+0000 7f1618200640 1 --2- 172.21.15.47:0/2702595134 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16100d2710 0x7f1610157050 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f15fc00d800 tx=0x7f15fc00dcc0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.108+0000 7f1618200640 1 -- 172.21.15.47:0/2702595134 shutdown_connections 2024-02-19T15:25:56.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.108+0000 7f1618200640 1 --2- 172.21.15.47:0/2702595134 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f15f0066e40 0x7f15f0069300 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.108+0000 7f1618200640 1 --2- 172.21.15.47:0/2702595134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f161015bcd0 0x7f1610157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.108+0000 7f1618200640 1 --2- 172.21.15.47:0/2702595134 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16100d2710 0x7f1610157050 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.108+0000 7f1618200640 1 --2- 172.21.15.47:0/2702595134 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16100cedb0 0x7f161014dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.108+0000 7f1618200640 1 -- 172.21.15.47:0/2702595134 >> 172.21.15.47:0/2702595134 conn(0x7f16100d1bf0 msgr2=0x7f16100703c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:56.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.108+0000 7f1618200640 1 -- 172.21.15.47:0/2702595134 shutdown_connections 2024-02-19T15:25:56.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.108+0000 7f1618200640 1 -- 172.21.15.47:0/2702595134 wait complete. 2024-02-19T15:25:56.138 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-19T15:25:56.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.276+0000 7f208e600640 1 Processor -- start 2024-02-19T15:25:56.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.276+0000 7f208e600640 1 -- start start 2024-02-19T15:25:56.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.276+0000 7f208e600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f20881456a0 0x7f20880cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:56.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.276+0000 7f208e600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f208806b5c0 0x7f208806dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:56.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.276+0000 7f208e600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2088145a80 con 0x7f20880cf4a0 2024-02-19T15:25:56.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.276+0000 7f208e600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20880d03c0 con 0x7f20881456a0 2024-02-19T15:25:56.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.276+0000 7f208e600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20880cf900 con 0x7f208806b5c0 2024-02-19T15:25:56.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.276+0000 7f2087400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f20881456a0 0x7f20880cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:56.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.276+0000 7f2087e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f208806b5c0 0x7f208806dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:56.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.276+0000 7f207ea00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f20880cf4a0 0x7f208806aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:56558/0 (socket says 172.21.15.47:56558) 2024-02-19T15:25:56.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.276+0000 7f207ea00640 1 -- 172.21.15.47:0/2862410598 learned_addr learned my addr 172.21.15.47:0/2862410598 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:56.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.276+0000 7f2087400640 1 -- 172.21.15.47:0/2862410598 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f208806b5c0 msgr2=0x7f208806dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:56.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.276+0000 7f2087400640 1 --2- 172.21.15.47:0/2862410598 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f208806b5c0 0x7f208806dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.276+0000 7f2087400640 1 -- 172.21.15.47:0/2862410598 >> v1:172.21.15.116:6789/0 conn(0x7f20880cf4a0 legacy=0x7f208806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:56.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.276+0000 7f2087400640 1 -- 172.21.15.47:0/2862410598 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20880d1880 con 0x7f20881456a0 2024-02-19T15:25:56.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.276+0000 7f2086a00640 1 -- 172.21.15.47:0/2862410598 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 83355305 0 0) 0x7f2088145a80 con 0x7f20880cf4a0 2024-02-19T15:25:56.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.280+0000 7f2087400640 1 --2- 172.21.15.47:0/2862410598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20881456a0 0x7f20880cef60 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f20780097c0 tx=0x7f207801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ce7e3da04a1d0e9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:56.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.280+0000 7f2086a00640 1 -- 172.21.15.47:0/2862410598 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20780ad640 con 0x7f20881456a0 2024-02-19T15:25:56.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.280+0000 7f2086a00640 1 -- 172.21.15.47:0/2862410598 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20780ad7a0 con 0x7f20881456a0 2024-02-19T15:25:56.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.280+0000 7f2086a00640 1 -- 172.21.15.47:0/2862410598 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20780adb30 con 0x7f20881456a0 2024-02-19T15:25:56.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.280+0000 7f208e600640 1 -- 172.21.15.47:0/2862410598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20881456a0 msgr2=0x7f20880cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:56.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.280+0000 7f208e600640 1 --2- 172.21.15.47:0/2862410598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20881456a0 0x7f20880cef60 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f20780097c0 tx=0x7f207801c4a0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.280+0000 7f208e600640 1 -- 172.21.15.47:0/2862410598 shutdown_connections 2024-02-19T15:25:56.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.280+0000 7f208e600640 1 --2- 172.21.15.47:0/2862410598 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f208806b5c0 0x7f208806dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.280+0000 7f208e600640 1 --2- 172.21.15.47:0/2862410598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20881456a0 0x7f20880cef60 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.280+0000 7f208e600640 1 -- 172.21.15.47:0/2862410598 >> 172.21.15.47:0/2862410598 conn(0x7f20880d0ad0 msgr2=0x7f20880d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:56.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.280+0000 7f208e600640 1 -- 172.21.15.47:0/2862410598 shutdown_connections 2024-02-19T15:25:56.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.280+0000 7f208e600640 1 -- 172.21.15.47:0/2862410598 wait complete. 2024-02-19T15:25:56.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f208e600640 1 Processor -- start 2024-02-19T15:25:56.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f208e600640 1 -- start start 2024-02-19T15:25:56.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f208e600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f208806b5c0 0x7f2088156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:56.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f208e600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20880cf4a0 0x7f208813b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:56.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f208e600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20881456a0 0x7f20881405b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:56.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f208e600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208813ba40 con 0x7f20880cf4a0 2024-02-19T15:25:56.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f208e600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20880b3ae0 con 0x7f20881456a0 2024-02-19T15:25:56.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f208e600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20880b3d80 con 0x7f208806b5c0 2024-02-19T15:25:56.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f207ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20880cf4a0 0x7f208813b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:56.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f207ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20880cf4a0 0x7f208813b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45948/0 (socket says 172.21.15.47:45948) 2024-02-19T15:25:56.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f207ea00640 1 -- 172.21.15.47:0/1910083128 learned_addr learned my addr 172.21.15.47:0/1910083128 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:56.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f2087e00640 1 --2- 172.21.15.47:0/1910083128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20881456a0 0x7f20881405b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:56.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f2087400640 1 --2- 172.21.15.47:0/1910083128 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f208806b5c0 0x7f2088156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:56.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f207ea00640 1 -- 172.21.15.47:0/1910083128 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f208806b5c0 msgr2=0x7f2088156b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:56.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f207ea00640 1 --2- 172.21.15.47:0/1910083128 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f208806b5c0 0x7f2088156b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f207ea00640 1 -- 172.21.15.47:0/1910083128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20881456a0 msgr2=0x7f20881405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:56.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f207ea00640 1 --2- 172.21.15.47:0/1910083128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20881456a0 0x7f20881405b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f207ea00640 1 -- 172.21.15.47:0/1910083128 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2078009420 con 0x7f20880cf4a0 2024-02-19T15:25:56.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f207ea00640 1 --2- 172.21.15.47:0/1910083128 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20880cf4a0 0x7f208813b500 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f207000c4a0 tx=0x7f207000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:56.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f207fe00640 1 -- 172.21.15.47:0/1910083128 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20700074e0 con 0x7f20880cf4a0 2024-02-19T15:25:56.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f207fe00640 1 -- 172.21.15.47:0/1910083128 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2070007640 con 0x7f20880cf4a0 2024-02-19T15:25:56.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f207fe00640 1 -- 172.21.15.47:0/1910083128 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2070004a30 con 0x7f20880cf4a0 2024-02-19T15:25:56.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f208e600640 1 -- 172.21.15.47:0/1910083128 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20880b4060 con 0x7f20880cf4a0 2024-02-19T15:25:56.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.284+0000 7f208e600640 1 -- 172.21.15.47:0/1910083128 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20880b4620 con 0x7f20880cf4a0 2024-02-19T15:25:56.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.288+0000 7f208e600640 1 -- 172.21.15.47:0/1910083128 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2088075360 con 0x7f20880cf4a0 2024-02-19T15:25:56.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.288+0000 7f207fe00640 1 -- 172.21.15.47:0/1910083128 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2070004b90 con 0x7f20880cf4a0 2024-02-19T15:25:56.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.288+0000 7f207fe00640 1 --2- 172.21.15.47:0/1910083128 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2064066e40 0x7f2064069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:56.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.288+0000 7f207fe00640 1 -- 172.21.15.47:0/1910083128 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2070083960 con 0x7f20880cf4a0 2024-02-19T15:25:56.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.288+0000 7f2087400640 1 --2- 172.21.15.47:0/1910083128 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2064066e40 0x7f2064069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:56.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.288+0000 7f2087400640 1 --2- 172.21.15.47:0/1910083128 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2064066e40 0x7f2064069300 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f20780af000 tx=0x7f20780a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:56.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.292+0000 7f207fe00640 1 -- 172.21.15.47:0/1910083128 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2070050ab0 con 0x7f20880cf4a0 2024-02-19T15:25:56.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.508+0000 7f208e600640 1 -- 172.21.15.47:0/1910083128 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f2088072b90 con 0x7f20880cf4a0 2024-02-19T15:25:56.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.508+0000 7f207fe00640 1 -- 172.21.15.47:0/1910083128 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f2070018070 con 0x7f20880cf4a0 2024-02-19T15:25:56.514 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:56.514 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:25:56.514 INFO:teuthology.orchestra.run.smithi047.stdout: "auth_dump": [ 2024-02-19T15:25:56.514 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:56.514 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.a", 2024-02-19T15:25:56.515 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNl8BP+HRAAlJ3Jk+1k2XcZAnICzy++Ow==", 2024-02-19T15:25:56.515 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:56.515 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:25:56.515 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:25:56.515 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:25:56.515 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:56.515 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:56.515 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:56.515 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:56.515 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.b", 2024-02-19T15:25:56.515 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlIOh3KhAAlnU2ZFY4iv/XEw0WqxyZpw==", 2024-02-19T15:25:56.515 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:56.515 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:25:56.515 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:25:56.515 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:25:56.516 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:56.516 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:56.516 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:56.516 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:56.516 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.c", 2024-02-19T15:25:56.516 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNl48SHJRAAn4GCT5KnP+2PD8IXbGClCg==", 2024-02-19T15:25:56.516 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:56.516 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:25:56.516 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:25:56.516 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:25:56.516 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:56.516 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:56.516 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:56.516 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:56.517 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.d", 2024-02-19T15:25:56.517 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNljcekMRAA414655SzH43fBA0AtjpxZw==", 2024-02-19T15:25:56.517 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:56.517 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:25:56.517 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:25:56.517 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:25:56.517 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:56.517 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:56.517 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:56.517 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:56.517 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.0", 2024-02-19T15:25:56.517 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDlcdNl/bycFhAAmzYi/OxEPUW3tcRJC0DNEQ==", 2024-02-19T15:25:56.518 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:56.518 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:25:56.518 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:25:56.518 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:56.518 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:56.518 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:56.518 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:56.518 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.1", 2024-02-19T15:25:56.518 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDocdNluquhHBAA88aLHCvVol7JoZB0MtoutA==", 2024-02-19T15:25:56.518 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:56.518 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:25:56.518 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:25:56.518 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:56.518 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:56.518 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:56.519 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:56.519 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.2", 2024-02-19T15:25:56.519 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDrcdNlGAnWIhAAyELvCPKjtfgF0+kX6NQFIA==", 2024-02-19T15:25:56.519 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:56.519 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:25:56.519 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:25:56.519 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:56.519 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:56.519 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:56.519 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:56.519 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.3", 2024-02-19T15:25:56.519 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDucdNlROZ6KBAAxTCDuYsw5SRIZBcclzwxhg==", 2024-02-19T15:25:56.519 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:56.519 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:25:56.519 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:25:56.520 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:56.520 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:56.520 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:56.520 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:56.520 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.4", 2024-02-19T15:25:56.520 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD0cdNluOGILBAA66R9RVXD2vdyIw+oZ+iM1g==", 2024-02-19T15:25:56.520 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:56.520 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:25:56.520 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:25:56.520 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:56.520 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:56.520 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:56.520 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:56.520 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.5", 2024-02-19T15:25:56.520 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD3cdNlHg8dMhAA2LUzfpvvHR4i8e5RBCeGpQ==", 2024-02-19T15:25:56.521 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:56.521 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:25:56.521 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:25:56.521 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:56.521 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:56.521 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:56.521 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:56.521 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.6", 2024-02-19T15:25:56.521 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD6cdNlXD1EORAAhKGIt0wFeCyoE6QkkZ3caA==", 2024-02-19T15:25:56.521 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:56.521 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:25:56.521 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:25:56.521 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:56.521 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:56.521 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:56.521 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:56.522 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.7", 2024-02-19T15:25:56.522 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD+cdNlfW65AhAAVWj3ptC8uQHPh/UXcb/uDg==", 2024-02-19T15:25:56.522 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:56.522 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:25:56.522 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:25:56.522 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:56.522 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:56.522 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:56.522 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:56.522 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.0", 2024-02-19T15:25:56.522 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlb7eRNRAAk6ZLGfjhcdlR9GoXFCd9eA==", 2024-02-19T15:25:56.522 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:56.522 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:25:56.522 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow r", 2024-02-19T15:25:56.522 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-19T15:25:56.522 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:56.523 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:56.523 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:56.523 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.1", 2024-02-19T15:25:56.523 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlKBu+AxAAlLRU70SAs3ZJo5wFwRWa4w==", 2024-02-19T15:25:56.523 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:56.523 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:25:56.523 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:25:56.523 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:25:56.523 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:25:56.523 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:56.523 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:56.523 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:56.523 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.2", 2024-02-19T15:25:56.524 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlyDk9ABAAIAjKSd4qJO5oHhVnDc1Rgw==", 2024-02-19T15:25:56.524 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:56.524 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:25:56.524 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:25:56.524 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:25:56.524 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:25:56.524 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:56.524 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:56.524 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:56.524 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.3", 2024-02-19T15:25:56.524 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlUqK3CRAABmSIzoJvl0ymME6swrvnSQ==", 2024-02-19T15:25:56.524 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:56.524 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:25:56.525 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:25:56.525 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:25:56.525 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:25:56.525 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:56.525 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:56.525 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:56.525 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.admin", 2024-02-19T15:25:56.525 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNls3vPAhAA4x2yQTUCz4Dv9vqEWnSTyQ==", 2024-02-19T15:25:56.525 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:56.525 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:25:56.525 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:25:56.525 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:25:56.525 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:56.525 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:56.525 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:56.526 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:56.526 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mgr.x", 2024-02-19T15:25:56.526 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNliM78GRAABWBg4hMcKjV2pK5WVBjLMQ==", 2024-02-19T15:25:56.526 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:56.526 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:25:56.526 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile mgr", 2024-02-19T15:25:56.526 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:56.526 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:56.526 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:25:56.526 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:56.526 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mgr.y", 2024-02-19T15:25:56.526 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlwxs9FhAAWqP7YoKio/bbdx7SGiCJ9Q==", 2024-02-19T15:25:56.526 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:25:56.526 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:25:56.526 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile mgr", 2024-02-19T15:25:56.526 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:25:56.527 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:56.527 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:56.527 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-02-19T15:25:56.527 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:25:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.512+0000 7f208e600640 1 -- 172.21.15.47:0/1910083128 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2064066e40 msgr2=0x7f2064069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.512+0000 7f208e600640 1 --2- 172.21.15.47:0/1910083128 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2064066e40 0x7f2064069300 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f20780af000 tx=0x7f20780a7070 comp rx=0 tx=0).stop 2024-02-19T15:25:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.512+0000 7f208e600640 1 -- 172.21.15.47:0/1910083128 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20880cf4a0 msgr2=0x7f208813b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.512+0000 7f208e600640 1 --2- 172.21.15.47:0/1910083128 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20880cf4a0 0x7f208813b500 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f207000c4a0 tx=0x7f207000c960 comp rx=0 tx=0).stop 2024-02-19T15:25:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.512+0000 7f208e600640 1 -- 172.21.15.47:0/1910083128 shutdown_connections 2024-02-19T15:25:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.512+0000 7f208e600640 1 --2- 172.21.15.47:0/1910083128 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2064066e40 0x7f2064069300 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.512+0000 7f208e600640 1 --2- 172.21.15.47:0/1910083128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20881456a0 0x7f20881405b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.512+0000 7f208e600640 1 --2- 172.21.15.47:0/1910083128 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20880cf4a0 0x7f208813b500 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.512+0000 7f208e600640 1 --2- 172.21.15.47:0/1910083128 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f208806b5c0 0x7f2088156b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.512+0000 7f208e600640 1 -- 172.21.15.47:0/1910083128 >> 172.21.15.47:0/1910083128 conn(0x7f20880d0ad0 msgr2=0x7f2088158730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.512+0000 7f208e600640 1 -- 172.21.15.47:0/1910083128 shutdown_connections 2024-02-19T15:25:56.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.512+0000 7f208e600640 1 -- 172.21.15.47:0/1910083128 wait complete. 2024-02-19T15:25:56.535 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:25:56.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b0ec00640 1 Processor -- start 2024-02-19T15:25:56.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b0ec00640 1 -- start start 2024-02-19T15:25:56.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b0ec00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b080cedb0 0x7f4b080cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:56.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b0ec00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b080cc700 con 0x7f4b0815bcd0 2024-02-19T15:25:56.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b0ec00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b080cf300 con 0x7f4b080d2710 2024-02-19T15:25:56.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b0ec00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b080cf5e0 con 0x7f4b080cedb0 2024-02-19T15:25:56.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b07e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f4b0815bcd0 0x7f4b080cd480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36030/0 (socket says 172.21.15.47:36030) 2024-02-19T15:25:56.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b07e00640 1 -- 172.21.15.47:0/3144551646 learned_addr learned my addr 172.21.15.47:0/3144551646 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:56.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b07400640 1 --2- 172.21.15.47:0/3144551646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b080cedb0 0x7f4b080cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:56.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b06a00640 1 -- 172.21.15.47:0/3144551646 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144196241 0 0) 0x7f4b080cc700 con 0x7f4b0815bcd0 2024-02-19T15:25:56.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b06a00640 1 -- 172.21.15.47:0/3144551646 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aec003530 con 0x7f4b0815bcd0 2024-02-19T15:25:56.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b06a00640 1 -- 172.21.15.47:0/3144551646 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3343667615 0 0) 0x7f4aec003530 con 0x7f4b0815bcd0 2024-02-19T15:25:56.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b06a00640 1 -- 172.21.15.47:0/3144551646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b080cedb0 msgr2=0x7f4b080cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:56.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b06a00640 1 --2- 172.21.15.47:0/3144551646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b080cedb0 0x7f4b080cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b06a00640 1 -- 172.21.15.47:0/3144551646 >> v1:172.21.15.116:6789/0 conn(0x7f4b080d2710 legacy=0x7f4b0815d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:56.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b06a00640 1 -- 172.21.15.47:0/3144551646 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b080cc9a0 con 0x7f4b0815bcd0 2024-02-19T15:25:56.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b06a00640 1 -- 172.21.15.47:0/3144551646 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f4af8001680 con 0x7f4b0815bcd0 2024-02-19T15:25:56.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b06a00640 1 -- 172.21.15.47:0/3144551646 >> v1:172.21.15.47:6789/0 conn(0x7f4b0815bcd0 legacy=0x7f4b080cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:56.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b06a00640 1 --2- 172.21.15.47:0/3144551646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4aec0039b0 0x7f4aec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:56.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b06a00640 1 --2- 172.21.15.47:0/3144551646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aec0047d0 0x7f4aec004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:56.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b06a00640 1 --2- 172.21.15.47:0/3144551646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b080cedb0 0x7f4aec005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:56.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b06a00640 1 -- 172.21.15.47:0/3144551646 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b080cf5e0 con 0x7f4aec0039b0 2024-02-19T15:25:56.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b06a00640 1 -- 172.21.15.47:0/3144551646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec00ba70 con 0x7f4aec0047d0 2024-02-19T15:25:56.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b06a00640 1 -- 172.21.15.47:0/3144551646 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec00bc30 con 0x7f4b080cedb0 2024-02-19T15:25:56.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b07e00640 1 --2- 172.21.15.47:0/3144551646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4aec0039b0 0x7f4aec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:56.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b07400640 1 --2- 172.21.15.47:0/3144551646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aec0047d0 0x7f4aec004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:56.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b0cc00640 1 --2- 172.21.15.47:0/3144551646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b080cedb0 0x7f4aec005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:56.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b07e00640 1 -- 172.21.15.47:0/3144551646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b080cedb0 msgr2=0x7f4aec005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:56.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b07e00640 1 --2- 172.21.15.47:0/3144551646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b080cedb0 0x7f4aec005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b07e00640 1 -- 172.21.15.47:0/3144551646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aec0047d0 msgr2=0x7f4aec004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:56.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b07e00640 1 --2- 172.21.15.47:0/3144551646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aec0047d0 0x7f4aec004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b07e00640 1 -- 172.21.15.47:0/3144551646 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aec003530 con 0x7f4aec0039b0 2024-02-19T15:25:56.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.672+0000 7f4b0cc00640 1 --2- 172.21.15.47:0/3144551646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b080cedb0 0x7f4aec005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:25:56.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.676+0000 7f4b07e00640 1 --2- 172.21.15.47:0/3144551646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4aec0039b0 0x7f4aec003dd0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f4af800c210 tx=0x7f4af801cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e428c385cfd99142 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:56.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.676+0000 7f4b06a00640 1 -- 172.21.15.47:0/3144551646 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af8005ae0 con 0x7f4aec0039b0 2024-02-19T15:25:56.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.676+0000 7f4b06a00640 1 -- 172.21.15.47:0/3144551646 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af8005c40 con 0x7f4aec0039b0 2024-02-19T15:25:56.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.676+0000 7f4b06a00640 1 -- 172.21.15.47:0/3144551646 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af80a57e0 con 0x7f4aec0039b0 2024-02-19T15:25:56.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.676+0000 7f4b0ec00640 1 -- 172.21.15.47:0/3144551646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4aec0039b0 msgr2=0x7f4aec003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:56.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.676+0000 7f4b0ec00640 1 --2- 172.21.15.47:0/3144551646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4aec0039b0 0x7f4aec003dd0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f4af800c210 tx=0x7f4af801cbf0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.676+0000 7f4b0cc00640 1 -- 172.21.15.47:0/3144551646 reap_dead start 2024-02-19T15:25:56.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.676+0000 7f4b0ec00640 1 -- 172.21.15.47:0/3144551646 shutdown_connections 2024-02-19T15:25:56.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.676+0000 7f4b0ec00640 1 -- 172.21.15.47:0/3144551646 >> 172.21.15.47:0/3144551646 conn(0x7f4b080d1bf0 msgr2=0x7f4b080d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:56.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.676+0000 7f4b0ec00640 1 -- 172.21.15.47:0/3144551646 shutdown_connections 2024-02-19T15:25:56.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.676+0000 7f4b0ec00640 1 -- 172.21.15.47:0/3144551646 wait complete. 2024-02-19T15:25:56.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b0ec00640 1 Processor -- start 2024-02-19T15:25:56.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b0ec00640 1 -- start start 2024-02-19T15:25:56.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b0ec00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aec0039b0 0x7f4b080c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:56.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b0ec00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b08158490 0x7f4b081669a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:56.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b0ec00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b08149b00 0x7f4b08149f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:56.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b0ec00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b080cdb90 con 0x7f4b08149b00 2024-02-19T15:25:56.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b0ec00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0814a4c0 con 0x7f4aec0039b0 2024-02-19T15:25:56.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b0ec00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b080cb390 con 0x7f4b08158490 2024-02-19T15:25:56.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b0cc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b08149b00 0x7f4b08149f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:56.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b0cc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b08149b00 0x7f4b08149f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45962/0 (socket says 172.21.15.47:45962) 2024-02-19T15:25:56.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b0cc00640 1 -- 172.21.15.47:0/169004485 learned_addr learned my addr 172.21.15.47:0/169004485 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:56.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b07400640 1 --2- 172.21.15.47:0/169004485 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b08158490 0x7f4b081669a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:56.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b0cc00640 1 -- 172.21.15.47:0/169004485 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b08158490 msgr2=0x7f4b081669a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:56.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b07e00640 1 --2- 172.21.15.47:0/169004485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aec0039b0 0x7f4b080c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:56.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b0cc00640 1 --2- 172.21.15.47:0/169004485 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b08158490 0x7f4b081669a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b0cc00640 1 -- 172.21.15.47:0/169004485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aec0039b0 msgr2=0x7f4b080c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:56.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b0cc00640 1 --2- 172.21.15.47:0/169004485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aec0039b0 0x7f4b080c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b0cc00640 1 -- 172.21.15.47:0/169004485 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b0815d240 con 0x7f4b08149b00 2024-02-19T15:25:56.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b07400640 1 --2- 172.21.15.47:0/169004485 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b08158490 0x7f4b081669a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:25:56.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b07e00640 1 --2- 172.21.15.47:0/169004485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aec0039b0 0x7f4b080c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:25:56.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b0cc00640 1 --2- 172.21.15.47:0/169004485 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b08149b00 0x7f4b08149f80 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f4b08071f70 tx=0x7f4b0000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:56.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4affe00640 1 -- 172.21.15.47:0/169004485 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b00017070 con 0x7f4b08149b00 2024-02-19T15:25:56.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4affe00640 1 -- 172.21.15.47:0/169004485 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b00006b60 con 0x7f4b08149b00 2024-02-19T15:25:56.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b0ec00640 1 -- 172.21.15.47:0/169004485 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b080cb600 con 0x7f4b08149b00 2024-02-19T15:25:56.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4affe00640 1 -- 172.21.15.47:0/169004485 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b00006d10 con 0x7f4b08149b00 2024-02-19T15:25:56.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.680+0000 7f4b0ec00640 1 -- 172.21.15.47:0/169004485 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b080cbbc0 con 0x7f4b08149b00 2024-02-19T15:25:56.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.684+0000 7f4b0ec00640 1 -- 172.21.15.47:0/169004485 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b08075360 con 0x7f4b08149b00 2024-02-19T15:25:56.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.684+0000 7f4affe00640 1 -- 172.21.15.47:0/169004485 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4b00003980 con 0x7f4b08149b00 2024-02-19T15:25:56.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.684+0000 7f4affe00640 1 --2- 172.21.15.47:0/169004485 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4ae8066e90 0x7f4ae8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:56.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.684+0000 7f4affe00640 1 -- 172.21.15.47:0/169004485 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4b00083d20 con 0x7f4b08149b00 2024-02-19T15:25:56.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.684+0000 7f4b07e00640 1 --2- 172.21.15.47:0/169004485 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4ae8066e90 0x7f4ae8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:56.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.684+0000 7f4b07e00640 1 --2- 172.21.15.47:0/169004485 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4ae8066e90 0x7f4ae8069350 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4af80ab000 tx=0x7f4af80a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:56.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.688+0000 7f4affe00640 1 -- 172.21.15.47:0/169004485 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f4b00050fd0 con 0x7f4b08149b00 2024-02-19T15:25:56.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.852+0000 7f4b0ec00640 1 -- 172.21.15.47:0/169004485 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b08164a80 con 0x7f4b08149b00 2024-02-19T15:25:56.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.852+0000 7f4affe00640 1 -- 172.21.15.47:0/169004485 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+12904 (secure 0 0 0) 0x7f4b00054690 con 0x7f4b08149b00 2024-02-19T15:25:56.856 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:56.856 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":78,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:25:56.117772+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:25:56.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.852+0000 7f4b0ec00640 1 -- 172.21.15.47:0/169004485 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4ae8066e90 msgr2=0x7f4ae8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:56.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.852+0000 7f4b0ec00640 1 --2- 172.21.15.47:0/169004485 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4ae8066e90 0x7f4ae8069350 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4af80ab000 tx=0x7f4af80a7070 comp rx=0 tx=0).stop 2024-02-19T15:25:56.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.852+0000 7f4b0ec00640 1 -- 172.21.15.47:0/169004485 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b08149b00 msgr2=0x7f4b08149f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:56.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.852+0000 7f4b0ec00640 1 --2- 172.21.15.47:0/169004485 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b08149b00 0x7f4b08149f80 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f4b08071f70 tx=0x7f4b0000eb90 comp rx=0 tx=0).stop 2024-02-19T15:25:56.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.852+0000 7f4b0ec00640 1 -- 172.21.15.47:0/169004485 shutdown_connections 2024-02-19T15:25:56.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.852+0000 7f4b0ec00640 1 --2- 172.21.15.47:0/169004485 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4ae8066e90 0x7f4ae8069350 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.852+0000 7f4b0ec00640 1 --2- 172.21.15.47:0/169004485 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b08149b00 0x7f4b08149f80 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.852+0000 7f4b0ec00640 1 --2- 172.21.15.47:0/169004485 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b08158490 0x7f4b081669a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.852+0000 7f4b0ec00640 1 --2- 172.21.15.47:0/169004485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aec0039b0 0x7f4b080c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:56.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.852+0000 7f4b0ec00640 1 -- 172.21.15.47:0/169004485 >> 172.21.15.47:0/169004485 conn(0x7f4b080d1bf0 msgr2=0x7f4b080cd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:56.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.856+0000 7f4b0ec00640 1 -- 172.21.15.47:0/169004485 shutdown_connections 2024-02-19T15:25:56.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:56.856+0000 7f4b0ec00640 1 -- 172.21.15.47:0/169004485 wait complete. 2024-02-19T15:25:56.878 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:25:56.879 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:25:57.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f7583e00640 1 Processor -- start 2024-02-19T15:25:57.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f7583e00640 1 -- start start 2024-02-19T15:25:57.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f7583e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f757c15af10 0x7f757c06e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:57.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f7583e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f757c0ca8f0 0x7f757c0cad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:57.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f7583e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757c071c70 con 0x7f757c0cb290 2024-02-19T15:25:57.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f7583e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f757c06e8a0 con 0x7f757c15af10 2024-02-19T15:25:57.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f7583e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f757c06cef0 con 0x7f757c0ca8f0 2024-02-19T15:25:57.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f7581e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f757c0cb290 0x7f757c15eaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36042/0 (socket says 172.21.15.47:36042) 2024-02-19T15:25:57.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f7581e00640 1 -- 172.21.15.47:0/3733215190 learned_addr learned my addr 172.21.15.47:0/3733215190 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:57.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f7580a00640 1 --2- 172.21.15.47:0/3733215190 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f757c0ca8f0 0x7f757c0cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:57.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f7581400640 1 --2- 172.21.15.47:0/3733215190 >> v2:172.21.15.116:3300/0 conn(0x7f757c15af10 0x7f757c06e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:57.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f757be00640 1 -- 172.21.15.47:0/3733215190 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321735530 0 0) 0x7f757c071c70 con 0x7f757c0cb290 2024-02-19T15:25:57.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f757be00640 1 -- 172.21.15.47:0/3733215190 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7564003530 con 0x7f757c0cb290 2024-02-19T15:25:57.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f757be00640 1 -- 172.21.15.47:0/3733215190 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1012221239 0 0) 0x7f7564003530 con 0x7f757c0cb290 2024-02-19T15:25:57.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f757be00640 1 -- 172.21.15.47:0/3733215190 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f757c0ca8f0 msgr2=0x7f757c0cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:57.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f757be00640 1 --2- 172.21.15.47:0/3733215190 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f757c0ca8f0 0x7f757c0cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f757be00640 1 -- 172.21.15.47:0/3733215190 >> v2:172.21.15.116:3300/0 conn(0x7f757c15af10 msgr2=0x7f757c06e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:57.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f757be00640 1 --2- 172.21.15.47:0/3733215190 >> v2:172.21.15.116:3300/0 conn(0x7f757c15af10 0x7f757c06e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f757be00640 1 -- 172.21.15.47:0/3733215190 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c0ca610 con 0x7f757c0cb290 2024-02-19T15:25:57.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f757be00640 1 -- 172.21.15.47:0/3733215190 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f7574001a00 con 0x7f757c0cb290 2024-02-19T15:25:57.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f757be00640 1 -- 172.21.15.47:0/3733215190 >> v1:172.21.15.47:6789/0 conn(0x7f757c0cb290 legacy=0x7f757c15eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:57.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f757be00640 1 --2- 172.21.15.47:0/3733215190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7564003980 0x7f7564003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:57.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f757be00640 1 --2- 172.21.15.47:0/3733215190 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f757c0ca8f0 0x7f75640047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:57.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f757be00640 1 --2- 172.21.15.47:0/3733215190 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7564004d10 0x7f7564005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:57.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f757be00640 1 -- 172.21.15.47:0/3733215190 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c06e8a0 con 0x7f7564004d10 2024-02-19T15:25:57.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f757be00640 1 -- 172.21.15.47:0/3733215190 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c06cef0 con 0x7f7564003980 2024-02-19T15:25:57.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f757be00640 1 -- 172.21.15.47:0/3733215190 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756400dba0 con 0x7f757c0ca8f0 2024-02-19T15:25:57.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f7581e00640 1 --2- 172.21.15.47:0/3733215190 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7564004d10 0x7f7564005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:57.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f7581400640 1 --2- 172.21.15.47:0/3733215190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7564003980 0x7f7564003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:57.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f7580a00640 1 --2- 172.21.15.47:0/3733215190 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f757c0ca8f0 0x7f75640047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:57.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f7581e00640 1 -- 172.21.15.47:0/3733215190 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f757c0ca8f0 msgr2=0x7f75640047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:57.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f7581e00640 1 --2- 172.21.15.47:0/3733215190 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f757c0ca8f0 0x7f75640047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f7581e00640 1 -- 172.21.15.47:0/3733215190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7564003980 msgr2=0x7f7564003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:57.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f7581e00640 1 --2- 172.21.15.47:0/3733215190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7564003980 0x7f7564003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.020+0000 7f7581e00640 1 -- 172.21.15.47:0/3733215190 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7564003530 con 0x7f7564004d10 2024-02-19T15:25:57.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.024+0000 7f7581e00640 1 --2- 172.21.15.47:0/3733215190 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7564004d10 0x7f7564005190 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f757c0cb6a0 tx=0x7f757401fb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=696c08bcc1b4b05e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:57.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.024+0000 7f757be00640 1 -- 172.21.15.47:0/3733215190 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75740aa6c0 con 0x7f7564004d10 2024-02-19T15:25:57.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.024+0000 7f757be00640 1 -- 172.21.15.47:0/3733215190 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75740aa820 con 0x7f7564004d10 2024-02-19T15:25:57.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.024+0000 7f757be00640 1 -- 172.21.15.47:0/3733215190 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75740aabd0 con 0x7f7564004d10 2024-02-19T15:25:57.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.024+0000 7f7583e00640 1 -- 172.21.15.47:0/3733215190 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7564004d10 msgr2=0x7f7564005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:57.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.024+0000 7f7583e00640 1 --2- 172.21.15.47:0/3733215190 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7564004d10 0x7f7564005190 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f757c0cb6a0 tx=0x7f757401fb20 comp rx=0 tx=0).stop 2024-02-19T15:25:57.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.024+0000 7f7581e00640 1 -- 172.21.15.47:0/3733215190 reap_dead start 2024-02-19T15:25:57.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.024+0000 7f7583e00640 1 -- 172.21.15.47:0/3733215190 shutdown_connections 2024-02-19T15:25:57.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.024+0000 7f7583e00640 1 -- 172.21.15.47:0/3733215190 >> 172.21.15.47:0/3733215190 conn(0x7f757c0cfc20 msgr2=0x7f757c0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:57.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.024+0000 7f7583e00640 1 -- 172.21.15.47:0/3733215190 shutdown_connections 2024-02-19T15:25:57.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.024+0000 7f7583e00640 1 -- 172.21.15.47:0/3733215190 wait complete. 2024-02-19T15:25:57.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7583e00640 1 Processor -- start 2024-02-19T15:25:57.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7583e00640 1 -- start start 2024-02-19T15:25:57.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7583e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7564004d10 0x7f757c15df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:57.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7583e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f757c0abec0 0x7f757c14de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:57.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7583e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f757c0ce930 0x7f757c0cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:57.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7583e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c1f3f30 con 0x7f757c0abec0 2024-02-19T15:25:57.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7583e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c0cf2f0 con 0x7f757c0ce930 2024-02-19T15:25:57.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7583e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f757c1f32d0 con 0x7f7564004d10 2024-02-19T15:25:57.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7580a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f757c0abec0 0x7f757c14de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:57.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7580a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f757c0abec0 0x7f757c14de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45982/0 (socket says 172.21.15.47:45982) 2024-02-19T15:25:57.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7580a00640 1 -- 172.21.15.47:0/1996537567 learned_addr learned my addr 172.21.15.47:0/1996537567 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:57.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7581400640 1 --2- 172.21.15.47:0/1996537567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7564004d10 0x7f757c15df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:57.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7581e00640 1 --2- 172.21.15.47:0/1996537567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f757c0ce930 0x7f757c0cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:57.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7580a00640 1 -- 172.21.15.47:0/1996537567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7564004d10 msgr2=0x7f757c15df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:57.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7580a00640 1 --2- 172.21.15.47:0/1996537567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7564004d10 0x7f757c15df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7580a00640 1 -- 172.21.15.47:0/1996537567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f757c0ce930 msgr2=0x7f757c0cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:57.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7580a00640 1 --2- 172.21.15.47:0/1996537567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f757c0ce930 0x7f757c0cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7580a00640 1 -- 172.21.15.47:0/1996537567 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c1f3590 con 0x7f757c0abec0 2024-02-19T15:25:57.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7580a00640 1 --2- 172.21.15.47:0/1996537567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f757c0abec0 0x7f757c14de60 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f75680119d0 tx=0x7f7568011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:57.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7579600640 1 -- 172.21.15.47:0/1996537567 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756800faf0 con 0x7f757c0abec0 2024-02-19T15:25:57.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7579600640 1 -- 172.21.15.47:0/1996537567 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f756800fc50 con 0x7f757c0abec0 2024-02-19T15:25:57.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7579600640 1 -- 172.21.15.47:0/1996537567 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7568014440 con 0x7f757c0abec0 2024-02-19T15:25:57.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7583e00640 1 -- 172.21.15.47:0/1996537567 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f757c1f3870 con 0x7f757c0abec0 2024-02-19T15:25:57.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.028+0000 7f7583e00640 1 -- 172.21.15.47:0/1996537567 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f757c15e470 con 0x7f757c0abec0 2024-02-19T15:25:57.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.032+0000 7f7583e00640 1 -- 172.21.15.47:0/1996537567 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f757c1f4070 con 0x7f757c0abec0 2024-02-19T15:25:57.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.032+0000 7f7579600640 1 -- 172.21.15.47:0/1996537567 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f75680145a0 con 0x7f757c0abec0 2024-02-19T15:25:57.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.032+0000 7f7579600640 1 --2- 172.21.15.47:0/1996537567 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f755c066e40 0x7f755c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:57.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.032+0000 7f7579600640 1 -- 172.21.15.47:0/1996537567 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7568084e50 con 0x7f757c0abec0 2024-02-19T15:25:57.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.032+0000 7f7581400640 1 --2- 172.21.15.47:0/1996537567 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f755c066e40 0x7f755c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:57.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.032+0000 7f7581400640 1 --2- 172.21.15.47:0/1996537567 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f755c066e40 0x7f755c069300 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7570009be0 tx=0x7f7570009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:57.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.036+0000 7f7579600640 1 -- 172.21.15.47:0/1996537567 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7568051fb0 con 0x7f757c0abec0 2024-02-19T15:25:57.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.200+0000 7f7583e00640 1 -- 172.21.15.47:0/1996537567 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f757c075360 con 0x7f757c0abec0 2024-02-19T15:25:57.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.200+0000 7f7579600640 1 -- 172.21.15.47:0/1996537567 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f75680220a0 con 0x7f757c0abec0 2024-02-19T15:25:57.204 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:25:57.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.200+0000 7f7583e00640 1 -- 172.21.15.47:0/1996537567 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f755c066e40 msgr2=0x7f755c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:57.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.200+0000 7f7583e00640 1 --2- 172.21.15.47:0/1996537567 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f755c066e40 0x7f755c069300 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7570009be0 tx=0x7f7570009190 comp rx=0 tx=0).stop 2024-02-19T15:25:57.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.200+0000 7f7583e00640 1 -- 172.21.15.47:0/1996537567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f757c0abec0 msgr2=0x7f757c14de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:57.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.200+0000 7f7583e00640 1 --2- 172.21.15.47:0/1996537567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f757c0abec0 0x7f757c14de60 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f75680119d0 tx=0x7f7568011dd0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.200+0000 7f7583e00640 1 -- 172.21.15.47:0/1996537567 shutdown_connections 2024-02-19T15:25:57.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.200+0000 7f7583e00640 1 --2- 172.21.15.47:0/1996537567 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f755c066e40 0x7f755c069300 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.200+0000 7f7583e00640 1 --2- 172.21.15.47:0/1996537567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f757c0ce930 0x7f757c0cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.200+0000 7f7583e00640 1 --2- 172.21.15.47:0/1996537567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f757c0abec0 0x7f757c14de60 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.200+0000 7f7583e00640 1 --2- 172.21.15.47:0/1996537567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7564004d10 0x7f757c15df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.200+0000 7f7583e00640 1 -- 172.21.15.47:0/1996537567 >> 172.21.15.47:0/1996537567 conn(0x7f757c0cfc20 msgr2=0x7f757c0c49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.200+0000 7f7583e00640 1 -- 172.21.15.47:0/1996537567 shutdown_connections 2024-02-19T15:25:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.200+0000 7f7583e00640 1 -- 172.21.15.47:0/1996537567 wait complete. 2024-02-19T15:25:57.225 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T15:25:57.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.360+0000 7fb667e00640 1 Processor -- start 2024-02-19T15:25:57.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.360+0000 7fb667e00640 1 -- start start 2024-02-19T15:25:57.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.364+0000 7fb667e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb66006d650 0x7fb6600d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:57.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.364+0000 7fb667e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6600cce30 con 0x7fb6600ced10 2024-02-19T15:25:57.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.364+0000 7fb667e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6600cd110 con 0x7fb6600cded0 2024-02-19T15:25:57.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.364+0000 7fb667e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6600cd3f0 con 0x7fb66006d650 2024-02-19T15:25:57.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.364+0000 7fb665e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb6600ced10 0x7fb66006bd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36044/0 (socket says 172.21.15.47:36044) 2024-02-19T15:25:57.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.364+0000 7fb665e00640 1 -- 172.21.15.47:0/503771078 learned_addr learned my addr 172.21.15.47:0/503771078 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:57.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.364+0000 7fb665400640 1 --2- 172.21.15.47:0/503771078 >> v2:172.21.15.116:3300/0 conn(0x7fb66006d650 0x7fb6600d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:57.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.364+0000 7fb65fe00640 1 -- 172.21.15.47:0/503771078 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873025680 0 0) 0x7fb6600cce30 con 0x7fb6600ced10 2024-02-19T15:25:57.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.364+0000 7fb65fe00640 1 -- 172.21.15.47:0/503771078 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb648003530 con 0x7fb6600ced10 2024-02-19T15:25:57.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.364+0000 7fb65fe00640 1 -- 172.21.15.47:0/503771078 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779173710 0 0) 0x7fb6600cd110 con 0x7fb6600cded0 2024-02-19T15:25:57.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.364+0000 7fb65fe00640 1 -- 172.21.15.47:0/503771078 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6600cce30 con 0x7fb6600cded0 2024-02-19T15:25:57.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.364+0000 7fb665400640 1 -- 172.21.15.47:0/503771078 >> v1:172.21.15.116:6789/0 conn(0x7fb6600cded0 legacy=0x7fb6600ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:57.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.364+0000 7fb665400640 1 -- 172.21.15.47:0/503771078 >> v1:172.21.15.47:6789/0 conn(0x7fb6600ced10 legacy=0x7fb66006bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:57.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.364+0000 7fb665400640 1 -- 172.21.15.47:0/503771078 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6600cd6d0 con 0x7fb66006d650 2024-02-19T15:25:57.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.368+0000 7fb665400640 1 --2- 172.21.15.47:0/503771078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb66006d650 0x7fb6600d08b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb6540097f0 tx=0x7fb65401c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df9fc87e67cc25f0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:57.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.368+0000 7fb65fe00640 1 -- 172.21.15.47:0/503771078 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6540ab600 con 0x7fb66006d650 2024-02-19T15:25:57.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.368+0000 7fb65fe00640 1 -- 172.21.15.47:0/503771078 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6540ab760 con 0x7fb66006d650 2024-02-19T15:25:57.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.368+0000 7fb65fe00640 1 -- 172.21.15.47:0/503771078 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6540abaf0 con 0x7fb66006d650 2024-02-19T15:25:57.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.368+0000 7fb667e00640 1 -- 172.21.15.47:0/503771078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb66006d650 msgr2=0x7fb6600d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:57.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.368+0000 7fb667e00640 1 --2- 172.21.15.47:0/503771078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb66006d650 0x7fb6600d08b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb6540097f0 tx=0x7fb65401c3e0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.368+0000 7fb667e00640 1 -- 172.21.15.47:0/503771078 shutdown_connections 2024-02-19T15:25:57.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.368+0000 7fb667e00640 1 --2- 172.21.15.47:0/503771078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb66006d650 0x7fb6600d08b0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.368+0000 7fb667e00640 1 -- 172.21.15.47:0/503771078 >> 172.21.15.47:0/503771078 conn(0x7fb6600d1bf0 msgr2=0x7fb6600d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:57.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.368+0000 7fb667e00640 1 -- 172.21.15.47:0/503771078 shutdown_connections 2024-02-19T15:25:57.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.368+0000 7fb667e00640 1 -- 172.21.15.47:0/503771078 wait complete. 2024-02-19T15:25:57.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.368+0000 7fb667e00640 1 Processor -- start 2024-02-19T15:25:57.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.368+0000 7fb667e00640 1 -- start start 2024-02-19T15:25:57.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.368+0000 7fb667e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb66006d650 0x7fb660161010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:57.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.368+0000 7fb667e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6600cded0 0x7fb660151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:57.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.368+0000 7fb667e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6600ced10 0x7fb6600c7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:57.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.368+0000 7fb667e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb660161550 con 0x7fb6600cded0 2024-02-19T15:25:57.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.368+0000 7fb667e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb660155710 con 0x7fb66006d650 2024-02-19T15:25:57.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.368+0000 7fb667e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6601559f0 con 0x7fb6600ced10 2024-02-19T15:25:57.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.372+0000 7fb664a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6600cded0 0x7fb660151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:57.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.372+0000 7fb664a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6600cded0 0x7fb660151dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45992/0 (socket says 172.21.15.47:45992) 2024-02-19T15:25:57.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.372+0000 7fb664a00640 1 -- 172.21.15.47:0/1893779997 learned_addr learned my addr 172.21.15.47:0/1893779997 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:57.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.372+0000 7fb665400640 1 --2- 172.21.15.47:0/1893779997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb66006d650 0x7fb660161010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:57.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.372+0000 7fb665e00640 1 --2- 172.21.15.47:0/1893779997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6600ced10 0x7fb6600c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:57.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.372+0000 7fb664a00640 1 -- 172.21.15.47:0/1893779997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6600ced10 msgr2=0x7fb6600c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:57.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.372+0000 7fb664a00640 1 --2- 172.21.15.47:0/1893779997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6600ced10 0x7fb6600c7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.372+0000 7fb664a00640 1 -- 172.21.15.47:0/1893779997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb66006d650 msgr2=0x7fb660161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:57.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.372+0000 7fb664a00640 1 --2- 172.21.15.47:0/1893779997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb66006d650 0x7fb660161010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.372+0000 7fb664a00640 1 -- 172.21.15.47:0/1893779997 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb654009420 con 0x7fb6600cded0 2024-02-19T15:25:57.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.372+0000 7fb664a00640 1 --2- 172.21.15.47:0/1893779997 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6600cded0 0x7fb660151dc0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00c490 tx=0x7fb64c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:57.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.372+0000 7fb65d600640 1 -- 172.21.15.47:0/1893779997 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c0052d0 con 0x7fb6600cded0 2024-02-19T15:25:57.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.372+0000 7fb65d600640 1 -- 172.21.15.47:0/1893779997 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb64c005430 con 0x7fb6600cded0 2024-02-19T15:25:57.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.372+0000 7fb65d600640 1 -- 172.21.15.47:0/1893779997 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c012870 con 0x7fb6600cded0 2024-02-19T15:25:57.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.372+0000 7fb667e00640 1 -- 172.21.15.47:0/1893779997 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb660155cd0 con 0x7fb6600cded0 2024-02-19T15:25:57.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.372+0000 7fb667e00640 1 -- 172.21.15.47:0/1893779997 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb660156290 con 0x7fb6600cded0 2024-02-19T15:25:57.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.372+0000 7fb667e00640 1 -- 172.21.15.47:0/1893779997 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb660075360 con 0x7fb6600cded0 2024-02-19T15:25:57.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.372+0000 7fb65d600640 1 -- 172.21.15.47:0/1893779997 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb64c0129d0 con 0x7fb6600cded0 2024-02-19T15:25:57.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.376+0000 7fb65d600640 1 --2- 172.21.15.47:0/1893779997 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb640066ec0 0x7fb640069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:57.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.376+0000 7fb65d600640 1 -- 172.21.15.47:0/1893779997 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb64c0840c0 con 0x7fb6600cded0 2024-02-19T15:25:57.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.376+0000 7fb665400640 1 --2- 172.21.15.47:0/1893779997 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb640066ec0 0x7fb640069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:57.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.376+0000 7fb665400640 1 --2- 172.21.15.47:0/1893779997 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb640066ec0 0x7fb640069380 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fb6540af000 tx=0x7fb6540a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:57.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.380+0000 7fb65d600640 1 -- 172.21.15.47:0/1893779997 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb64c051320 con 0x7fb6600cded0 2024-02-19T15:25:57.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.544+0000 7fb667e00640 1 -- 172.21.15.47:0/1893779997 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb6601616f0 con 0x7fb6600cded0 2024-02-19T15:25:57.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.544+0000 7fb65d600640 1 -- 172.21.15.47:0/1893779997 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v78) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb64c0549e0 con 0x7fb6600cded0 2024-02-19T15:25:57.548 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:25:57.548 INFO:teuthology.orchestra.run.smithi047.stdout:[ 2024-02-19T15:25:57.548 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:25:57.548 INFO:teuthology.orchestra.run.smithi047.stdout: "poolnum": 1, 2024-02-19T15:25:57.548 INFO:teuthology.orchestra.run.smithi047.stdout: "poolname": ".mgr" 2024-02-19T15:25:57.548 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:25:57.548 INFO:teuthology.orchestra.run.smithi047.stdout:] 2024-02-19T15:25:57.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.544+0000 7fb667e00640 1 -- 172.21.15.47:0/1893779997 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb640066ec0 msgr2=0x7fb640069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:57.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.544+0000 7fb667e00640 1 --2- 172.21.15.47:0/1893779997 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb640066ec0 0x7fb640069380 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fb6540af000 tx=0x7fb6540a7000 comp rx=0 tx=0).stop 2024-02-19T15:25:57.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.544+0000 7fb667e00640 1 -- 172.21.15.47:0/1893779997 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6600cded0 msgr2=0x7fb660151dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:57.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.544+0000 7fb667e00640 1 --2- 172.21.15.47:0/1893779997 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6600cded0 0x7fb660151dc0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00c490 tx=0x7fb64c00c950 comp rx=0 tx=0).stop 2024-02-19T15:25:57.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.544+0000 7fb667e00640 1 -- 172.21.15.47:0/1893779997 shutdown_connections 2024-02-19T15:25:57.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.544+0000 7fb667e00640 1 --2- 172.21.15.47:0/1893779997 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb640066ec0 0x7fb640069380 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.544+0000 7fb667e00640 1 --2- 172.21.15.47:0/1893779997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6600ced10 0x7fb6600c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.544+0000 7fb667e00640 1 --2- 172.21.15.47:0/1893779997 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6600cded0 0x7fb660151dc0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.544+0000 7fb667e00640 1 --2- 172.21.15.47:0/1893779997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb66006d650 0x7fb660161010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.544+0000 7fb667e00640 1 -- 172.21.15.47:0/1893779997 >> 172.21.15.47:0/1893779997 conn(0x7fb6600d1bf0 msgr2=0x7fb6600d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:57.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.544+0000 7fb667e00640 1 -- 172.21.15.47:0/1893779997 shutdown_connections 2024-02-19T15:25:57.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.544+0000 7fb667e00640 1 -- 172.21.15.47:0/1893779997 wait complete. 2024-02-19T15:25:57.568 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T15:25:57.569 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T15:25:57.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.708+0000 7fbb71800640 1 Processor -- start 2024-02-19T15:25:57.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.708+0000 7fbb71800640 1 -- start start 2024-02-19T15:25:57.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.708+0000 7fbb71800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbb6c06e570 0x7fbb6c071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:57.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.708+0000 7fbb71800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbb6c071050 0x7fbb6c0ae490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:57.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.708+0000 7fbb71800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6c0ca520 con 0x7fbb6c15abf0 2024-02-19T15:25:57.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.708+0000 7fbb71800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb6c0ca800 con 0x7fbb6c071050 2024-02-19T15:25:57.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.708+0000 7fbb71800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c0caae0 con 0x7fbb6c06e570 2024-02-19T15:25:57.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.708+0000 7fbb6b400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbb6c071050 0x7fbb6c0ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:57.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.708+0000 7fbb6aa00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbb6c15abf0 0x7fbb6c06de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36054/0 (socket says 172.21.15.47:36054) 2024-02-19T15:25:57.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.708+0000 7fbb6aa00640 1 -- 172.21.15.47:0/2888861384 learned_addr learned my addr 172.21.15.47:0/2888861384 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:57.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.712+0000 7fbb63e00640 1 --2- 172.21.15.47:0/2888861384 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbb6c06e570 0x7fbb6c071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:57.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.712+0000 7fbb6b400640 1 -- 172.21.15.47:0/2888861384 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbb6c06e570 msgr2=0x7fbb6c071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:57.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.712+0000 7fbb6b400640 1 --2- 172.21.15.47:0/2888861384 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbb6c06e570 0x7fbb6c071f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.712+0000 7fbb6b400640 1 -- 172.21.15.47:0/2888861384 >> v1:172.21.15.47:6789/0 conn(0x7fbb6c15abf0 legacy=0x7fbb6c06de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:57.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.712+0000 7fbb6b400640 1 -- 172.21.15.47:0/2888861384 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6c0cadc0 con 0x7fbb6c071050 2024-02-19T15:25:57.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.712+0000 7fbb6b400640 1 --2- 172.21.15.47:0/2888861384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb6c071050 0x7fbb6c0ae490 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009d90 tx=0x7fbb6401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79fab7ac0e77e115 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:57.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.712+0000 7fbb6a000640 1 -- 172.21.15.47:0/2888861384 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb640a5a20 con 0x7fbb6c071050 2024-02-19T15:25:57.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.712+0000 7fbb6a000640 1 -- 172.21.15.47:0/2888861384 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb640a5b80 con 0x7fbb6c071050 2024-02-19T15:25:57.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.712+0000 7fbb6a000640 1 -- 172.21.15.47:0/2888861384 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb640ad440 con 0x7fbb6c071050 2024-02-19T15:25:57.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb71800640 1 -- 172.21.15.47:0/2888861384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb6c071050 msgr2=0x7fbb6c0ae490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:57.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb71800640 1 --2- 172.21.15.47:0/2888861384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb6c071050 0x7fbb6c0ae490 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009d90 tx=0x7fbb6401c940 comp rx=0 tx=0).stop 2024-02-19T15:25:57.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb71800640 1 -- 172.21.15.47:0/2888861384 shutdown_connections 2024-02-19T15:25:57.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb71800640 1 --2- 172.21.15.47:0/2888861384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb6c071050 0x7fbb6c0ae490 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb71800640 1 --2- 172.21.15.47:0/2888861384 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbb6c06e570 0x7fbb6c071f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb71800640 1 -- 172.21.15.47:0/2888861384 >> 172.21.15.47:0/2888861384 conn(0x7fbb6c0cc3f0 msgr2=0x7fbb6c0cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:57.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb71800640 1 -- 172.21.15.47:0/2888861384 shutdown_connections 2024-02-19T15:25:57.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb71800640 1 -- 172.21.15.47:0/2888861384 wait complete. 2024-02-19T15:25:57.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb71800640 1 Processor -- start 2024-02-19T15:25:57.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb71800640 1 -- start start 2024-02-19T15:25:57.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb71800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb6c06e570 0x7fbb6c06d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:57.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb71800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c071050 0x7fbb6c0c6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:57.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb71800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbb6c15abf0 0x7fbb6c14c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:57.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb71800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c0c8cb0 con 0x7fbb6c06e570 2024-02-19T15:25:57.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb71800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c0c8f50 con 0x7fbb6c071050 2024-02-19T15:25:57.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb71800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c0c9230 con 0x7fbb6c15abf0 2024-02-19T15:25:57.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb6aa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb6c06e570 0x7fbb6c06d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:57.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb6aa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb6c06e570 0x7fbb6c06d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:46018/0 (socket says 172.21.15.47:46018) 2024-02-19T15:25:57.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb6aa00640 1 -- 172.21.15.47:0/1121955119 learned_addr learned my addr 172.21.15.47:0/1121955119 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:57.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb63e00640 1 --2- 172.21.15.47:0/1121955119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c071050 0x7fbb6c0c6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:57.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb6b400640 1 --2- 172.21.15.47:0/1121955119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbb6c15abf0 0x7fbb6c14c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:57.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb6aa00640 1 -- 172.21.15.47:0/1121955119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbb6c15abf0 msgr2=0x7fbb6c14c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:57.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb6aa00640 1 --2- 172.21.15.47:0/1121955119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbb6c15abf0 0x7fbb6c14c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb6aa00640 1 -- 172.21.15.47:0/1121955119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c071050 msgr2=0x7fbb6c0c6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:57.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb6aa00640 1 --2- 172.21.15.47:0/1121955119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c071050 0x7fbb6c0c6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:57.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb6aa00640 1 -- 172.21.15.47:0/1121955119 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb64009a70 con 0x7fbb6c06e570 2024-02-19T15:25:57.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.716+0000 7fbb6aa00640 1 --2- 172.21.15.47:0/1121955119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb6c06e570 0x7fbb6c06d950 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c00c370 tx=0x7fbb5c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:57.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.720+0000 7fbb62a00640 1 -- 172.21.15.47:0/1121955119 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c005cf0 con 0x7fbb6c06e570 2024-02-19T15:25:57.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.720+0000 7fbb62a00640 1 -- 172.21.15.47:0/1121955119 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5c005e50 con 0x7fbb6c06e570 2024-02-19T15:25:57.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.720+0000 7fbb62a00640 1 -- 172.21.15.47:0/1121955119 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c0072f0 con 0x7fbb6c06e570 2024-02-19T15:25:57.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.720+0000 7fbb71800640 1 -- 172.21.15.47:0/1121955119 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb6c0c9510 con 0x7fbb6c06e570 2024-02-19T15:25:57.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.720+0000 7fbb71800640 1 -- 172.21.15.47:0/1121955119 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb6c0c7590 con 0x7fbb6c06e570 2024-02-19T15:25:57.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.720+0000 7fbb71800640 1 -- 172.21.15.47:0/1121955119 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb6c0c9750 con 0x7fbb6c06e570 2024-02-19T15:25:57.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.720+0000 7fbb62a00640 1 -- 172.21.15.47:0/1121955119 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fbb5c007450 con 0x7fbb6c06e570 2024-02-19T15:25:57.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.720+0000 7fbb62a00640 1 --2- 172.21.15.47:0/1121955119 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbb48066e40 0x7fbb48069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:57.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.720+0000 7fbb62a00640 1 -- 172.21.15.47:0/1121955119 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbb5c083d20 con 0x7fbb6c06e570 2024-02-19T15:25:57.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.720+0000 7fbb63e00640 1 --2- 172.21.15.47:0/1121955119 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbb48066e40 0x7fbb48069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:57.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.724+0000 7fbb63e00640 1 --2- 172.21.15.47:0/1121955119 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbb48066e40 0x7fbb48069300 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c072410 tx=0x7fbb54008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:57.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.724+0000 7fbb62a00640 1 -- 172.21.15.47:0/1121955119 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fbb6c0c9750 con 0x7fbb6c06e570 2024-02-19T15:25:57.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:57.888+0000 7fbb71800640 1 -- 172.21.15.47:0/1121955119 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fbb6c075360 con 0x7fbb6c06e570 2024-02-19T15:25:58.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.112+0000 7fbb62a00640 1 -- 172.21.15.47:0/1121955119 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v79) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbb5c0170b0 con 0x7fbb6c06e570 2024-02-19T15:25:58.115 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_metadata' created 2024-02-19T15:25:58.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.112+0000 7fbb71800640 1 -- 172.21.15.47:0/1121955119 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbb48066e40 msgr2=0x7fbb48069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:58.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.112+0000 7fbb71800640 1 --2- 172.21.15.47:0/1121955119 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbb48066e40 0x7fbb48069300 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c072410 tx=0x7fbb54008040 comp rx=0 tx=0).stop 2024-02-19T15:25:58.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.112+0000 7fbb71800640 1 -- 172.21.15.47:0/1121955119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb6c06e570 msgr2=0x7fbb6c06d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:58.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.112+0000 7fbb71800640 1 --2- 172.21.15.47:0/1121955119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb6c06e570 0x7fbb6c06d950 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c00c370 tx=0x7fbb5c00c830 comp rx=0 tx=0).stop 2024-02-19T15:25:58.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.112+0000 7fbb71800640 1 -- 172.21.15.47:0/1121955119 shutdown_connections 2024-02-19T15:25:58.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.112+0000 7fbb71800640 1 --2- 172.21.15.47:0/1121955119 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbb48066e40 0x7fbb48069300 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:58.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.112+0000 7fbb71800640 1 --2- 172.21.15.47:0/1121955119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbb6c15abf0 0x7fbb6c14c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:58.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.112+0000 7fbb71800640 1 --2- 172.21.15.47:0/1121955119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c071050 0x7fbb6c0c6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:58.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.112+0000 7fbb71800640 1 --2- 172.21.15.47:0/1121955119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb6c06e570 0x7fbb6c06d950 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:58.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.112+0000 7fbb71800640 1 -- 172.21.15.47:0/1121955119 >> 172.21.15.47:0/1121955119 conn(0x7fbb6c0cc3f0 msgr2=0x7fbb6c0afb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:58.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.112+0000 7fbb71800640 1 -- 172.21.15.47:0/1121955119 shutdown_connections 2024-02-19T15:25:58.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.112+0000 7fbb71800640 1 -- 172.21.15.47:0/1121955119 wait complete. 2024-02-19T15:25:58.142 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T15:25:58.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.284+0000 7f981fe00640 1 Processor -- start 2024-02-19T15:25:58.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.284+0000 7f981fe00640 1 -- start start 2024-02-19T15:25:58.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.288+0000 7f981fe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f98180d0150 0x7f981806aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:58.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.288+0000 7f981fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9818162460 0x7f98180ae4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:58.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.288+0000 7f981fe00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98180ce660 con 0x7f981806b3f0 2024-02-19T15:25:58.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.288+0000 7f981fe00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9818162d60 con 0x7f98180d0150 2024-02-19T15:25:58.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.288+0000 7f981fe00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9818162a80 con 0x7f9818162460 2024-02-19T15:25:58.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.288+0000 7f981d400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f98180d0150 0x7f981806aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:58.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.288+0000 7f981de00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9818162460 0x7f98180ae4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:58.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.288+0000 7f981ca00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f981806b3f0 0x7f9818161d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:56596/0 (socket says 172.21.15.47:56596) 2024-02-19T15:25:58.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.288+0000 7f981ca00640 1 -- 172.21.15.47:0/2936284587 learned_addr learned my addr 172.21.15.47:0/2936284587 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:58.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.288+0000 7f981de00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9818162460 0x7f98180ae4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:34830/0 (socket says 172.21.15.47:34830) 2024-02-19T15:25:58.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.288+0000 7f9817e00640 1 -- 172.21.15.47:0/2936284587 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592446433 0 0) 0x7f98180ce660 con 0x7f981806b3f0 2024-02-19T15:25:58.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.288+0000 7f9817e00640 1 -- 172.21.15.47:0/2936284587 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9800003530 con 0x7f981806b3f0 2024-02-19T15:25:58.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.288+0000 7f981de00640 1 -- 172.21.15.47:0/2936284587 >> v2:172.21.15.116:3300/0 conn(0x7f98180d0150 msgr2=0x7f981806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:25:58.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.288+0000 7f981de00640 1 --2- 172.21.15.47:0/2936284587 >> v2:172.21.15.116:3300/0 conn(0x7f98180d0150 0x7f981806aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:58.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.288+0000 7f981de00640 1 -- 172.21.15.47:0/2936284587 >> v1:172.21.15.116:6789/0 conn(0x7f981806b3f0 legacy=0x7f9818161d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:58.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.288+0000 7f981de00640 1 -- 172.21.15.47:0/2936284587 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981806b850 con 0x7f9818162460 2024-02-19T15:25:58.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.292+0000 7f981de00640 1 --2- 172.21.15.47:0/2936284587 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9818162460 0x7f98180ae4d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9810009d90 tx=0x7f981001c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e16fd67d651c82b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:58.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.292+0000 7f9817e00640 1 -- 172.21.15.47:0/2936284587 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98100a5a00 con 0x7f9818162460 2024-02-19T15:25:58.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.292+0000 7f9817e00640 1 -- 172.21.15.47:0/2936284587 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98100a5b60 con 0x7f9818162460 2024-02-19T15:25:58.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.292+0000 7f9817e00640 1 -- 172.21.15.47:0/2936284587 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98100ad440 con 0x7f9818162460 2024-02-19T15:25:58.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.292+0000 7f981fe00640 1 -- 172.21.15.47:0/2936284587 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9818162460 msgr2=0x7f98180ae4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:58.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.292+0000 7f981fe00640 1 --2- 172.21.15.47:0/2936284587 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9818162460 0x7f98180ae4d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9810009d90 tx=0x7f981001c940 comp rx=0 tx=0).stop 2024-02-19T15:25:58.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.292+0000 7f981fe00640 1 -- 172.21.15.47:0/2936284587 shutdown_connections 2024-02-19T15:25:58.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.292+0000 7f981fe00640 1 --2- 172.21.15.47:0/2936284587 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9818162460 0x7f98180ae4d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:58.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.292+0000 7f981fe00640 1 --2- 172.21.15.47:0/2936284587 >> v2:172.21.15.116:3300/0 conn(0x7f98180d0150 0x7f981806aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:58.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.292+0000 7f981fe00640 1 -- 172.21.15.47:0/2936284587 >> 172.21.15.47:0/2936284587 conn(0x7f98180b2580 msgr2=0x7f98180b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:58.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.292+0000 7f981fe00640 1 -- 172.21.15.47:0/2936284587 shutdown_connections 2024-02-19T15:25:58.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.292+0000 7f981fe00640 1 -- 172.21.15.47:0/2936284587 wait complete. 2024-02-19T15:25:58.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.292+0000 7f981fe00640 1 Processor -- start 2024-02-19T15:25:58.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.292+0000 7f981fe00640 1 -- start start 2024-02-19T15:25:58.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f981806b3f0 0x7f98180c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:58.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98180d0150 0x7f98180c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:58.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9818162460 0x7f9818150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:58.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981fe00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981815f9b0 con 0x7f9818162460 2024-02-19T15:25:58.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981fe00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98180c80a0 con 0x7f981806b3f0 2024-02-19T15:25:58.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981fe00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98180c7470 con 0x7f98180d0150 2024-02-19T15:25:58.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9818162460 0x7f9818150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:58.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981ca00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98180d0150 0x7f98180c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:58.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9818162460 0x7f9818150230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:46032/0 (socket says 172.21.15.47:46032) 2024-02-19T15:25:58.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981de00640 1 -- 172.21.15.47:0/749860941 learned_addr learned my addr 172.21.15.47:0/749860941 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:58.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981d400640 1 --2- 172.21.15.47:0/749860941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f981806b3f0 0x7f98180c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:58.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981de00640 1 -- 172.21.15.47:0/749860941 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98180d0150 msgr2=0x7f98180c7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:58.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981de00640 1 --2- 172.21.15.47:0/749860941 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98180d0150 0x7f98180c7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:58.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981de00640 1 -- 172.21.15.47:0/749860941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f981806b3f0 msgr2=0x7f98180c6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:58.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981de00640 1 --2- 172.21.15.47:0/749860941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f981806b3f0 0x7f98180c6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:58.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981de00640 1 -- 172.21.15.47:0/749860941 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9810009a70 con 0x7f9818162460 2024-02-19T15:25:58.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981ca00640 1 --2- 172.21.15.47:0/749860941 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98180d0150 0x7f98180c7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:25:58.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981de00640 1 --2- 172.21.15.47:0/749860941 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9818162460 0x7f9818150230 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f981001aff0 tx=0x7f98100196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:58.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f9815600640 1 -- 172.21.15.47:0/749860941 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98100adc60 con 0x7f9818162460 2024-02-19T15:25:58.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f9815600640 1 -- 172.21.15.47:0/749860941 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98100addc0 con 0x7f9818162460 2024-02-19T15:25:58.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981fe00640 1 -- 172.21.15.47:0/749860941 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f981815f080 con 0x7f9818162460 2024-02-19T15:25:58.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f9815600640 1 -- 172.21.15.47:0/749860941 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98100b9650 con 0x7f9818162460 2024-02-19T15:25:58.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981fe00640 1 -- 172.21.15.47:0/749860941 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f981815f640 con 0x7f9818162460 2024-02-19T15:25:58.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.296+0000 7f981fe00640 1 -- 172.21.15.47:0/749860941 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f981815fb20 con 0x7f9818162460 2024-02-19T15:25:58.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.300+0000 7f9815600640 1 -- 172.21.15.47:0/749860941 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f98100b97b0 con 0x7f9818162460 2024-02-19T15:25:58.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.300+0000 7f9815600640 1 --2- 172.21.15.47:0/749860941 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f97f8066e40 0x7f97f8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:58.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.300+0000 7f9815600640 1 -- 172.21.15.47:0/749860941 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f9810120220 con 0x7f9818162460 2024-02-19T15:25:58.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.300+0000 7f981d400640 1 --2- 172.21.15.47:0/749860941 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f97f8066e40 0x7f97f8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:58.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.300+0000 7f981d400640 1 --2- 172.21.15.47:0/749860941 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f97f8066e40 0x7f97f8069300 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f980c009b60 tx=0x7f980c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:58.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.304+0000 7f9815600640 1 -- 172.21.15.47:0/749860941 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f98100ed310 con 0x7f9818162460 2024-02-19T15:25:58.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:58.464+0000 7f981fe00640 1 -- 172.21.15.47:0/749860941 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9818075360 con 0x7f9818162460 2024-02-19T15:25:59.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.116+0000 7f9815600640 1 -- 172.21.15.47:0/749860941 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v80) v1 ==== 170+0+0 (secure 0 0 0) 0x7f9810009450 con 0x7f9818162460 2024-02-19T15:25:59.121 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_data' created 2024-02-19T15:25:59.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.120+0000 7f981fe00640 1 -- 172.21.15.47:0/749860941 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f97f8066e40 msgr2=0x7f97f8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:59.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.120+0000 7f981fe00640 1 --2- 172.21.15.47:0/749860941 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f97f8066e40 0x7f97f8069300 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f980c009b60 tx=0x7f980c009190 comp rx=0 tx=0).stop 2024-02-19T15:25:59.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.120+0000 7f981fe00640 1 -- 172.21.15.47:0/749860941 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9818162460 msgr2=0x7f9818150230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:59.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.120+0000 7f981fe00640 1 --2- 172.21.15.47:0/749860941 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9818162460 0x7f9818150230 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f981001aff0 tx=0x7f98100196e0 comp rx=0 tx=0).stop 2024-02-19T15:25:59.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.120+0000 7f981fe00640 1 -- 172.21.15.47:0/749860941 shutdown_connections 2024-02-19T15:25:59.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.120+0000 7f981fe00640 1 --2- 172.21.15.47:0/749860941 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f97f8066e40 0x7f97f8069300 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:59.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.120+0000 7f981fe00640 1 --2- 172.21.15.47:0/749860941 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9818162460 0x7f9818150230 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:59.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.120+0000 7f981fe00640 1 --2- 172.21.15.47:0/749860941 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f98180d0150 0x7f98180c7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:59.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.120+0000 7f981fe00640 1 --2- 172.21.15.47:0/749860941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f981806b3f0 0x7f98180c6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:59.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.120+0000 7f981fe00640 1 -- 172.21.15.47:0/749860941 >> 172.21.15.47:0/749860941 conn(0x7f98180b2580 msgr2=0x7f98180af7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:59.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.120+0000 7f981fe00640 1 -- 172.21.15.47:0/749860941 shutdown_connections 2024-02-19T15:25:59.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.120+0000 7f981fe00640 1 -- 172.21.15.47:0/749860941 wait complete. 2024-02-19T15:25:59.160 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T15:25:59.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.300+0000 7f3f13200640 1 Processor -- start 2024-02-19T15:25:59.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.300+0000 7f3f13200640 1 -- start start 2024-02-19T15:25:59.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.304+0000 7f3f13200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f1415bc60 0x7f3f140ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:59.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.304+0000 7f3f13200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f140b0db0 con 0x7f3f140d2480 2024-02-19T15:25:59.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.304+0000 7f3f13200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f140cf9e0 con 0x7f3f140d0650 2024-02-19T15:25:59.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.304+0000 7f3f13200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f140b0fe0 con 0x7f3f1415bc60 2024-02-19T15:25:59.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.304+0000 7f3f09400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f3f140d2480 0x7f3f140cff40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36060/0 (socket says 172.21.15.47:36060) 2024-02-19T15:25:59.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.304+0000 7f3f09400640 1 -- 172.21.15.47:0/3264773083 learned_addr learned my addr 172.21.15.47:0/3264773083 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:25:59.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.304+0000 7f3f11e00640 1 --2- 172.21.15.47:0/3264773083 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f1415bc60 0x7f3f140ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:59.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.304+0000 7f3f11400640 1 -- 172.21.15.47:0/3264773083 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090755718 0 0) 0x7f3f140b0db0 con 0x7f3f140d2480 2024-02-19T15:25:59.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.304+0000 7f3f11400640 1 -- 172.21.15.47:0/3264773083 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ef8003530 con 0x7f3f140d2480 2024-02-19T15:25:59.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.304+0000 7f3f11400640 1 -- 172.21.15.47:0/3264773083 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955503435 0 0) 0x7f3f140cf9e0 con 0x7f3f140d0650 2024-02-19T15:25:59.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.304+0000 7f3f11400640 1 -- 172.21.15.47:0/3264773083 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f140b0db0 con 0x7f3f140d0650 2024-02-19T15:25:59.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.304+0000 7f3f11e00640 1 -- 172.21.15.47:0/3264773083 >> v1:172.21.15.116:6789/0 conn(0x7f3f140d0650 legacy=0x7f3f140b1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:59.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.304+0000 7f3f11e00640 1 -- 172.21.15.47:0/3264773083 >> v1:172.21.15.47:6789/0 conn(0x7f3f140d2480 legacy=0x7f3f140cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:59.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.304+0000 7f3f11e00640 1 -- 172.21.15.47:0/3264773083 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f140b12c0 con 0x7f3f1415bc60 2024-02-19T15:25:59.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.308+0000 7f3f11e00640 1 --2- 172.21.15.47:0/3264773083 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f1415bc60 0x7f3f140ce9a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3f040097f0 tx=0x7f3f0401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f03c2490381c0a2e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:59.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.308+0000 7f3f11400640 1 -- 172.21.15.47:0/3264773083 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f040ab620 con 0x7f3f1415bc60 2024-02-19T15:25:59.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.308+0000 7f3f11400640 1 -- 172.21.15.47:0/3264773083 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f040ab780 con 0x7f3f1415bc60 2024-02-19T15:25:59.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.308+0000 7f3f11400640 1 -- 172.21.15.47:0/3264773083 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f040abb10 con 0x7f3f1415bc60 2024-02-19T15:25:59.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.308+0000 7f3f13200640 1 -- 172.21.15.47:0/3264773083 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f1415bc60 msgr2=0x7f3f140ce9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:59.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.308+0000 7f3f13200640 1 --2- 172.21.15.47:0/3264773083 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f1415bc60 0x7f3f140ce9a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3f040097f0 tx=0x7f3f0401c420 comp rx=0 tx=0).stop 2024-02-19T15:25:59.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.308+0000 7f3f13200640 1 -- 172.21.15.47:0/3264773083 shutdown_connections 2024-02-19T15:25:59.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.308+0000 7f3f13200640 1 --2- 172.21.15.47:0/3264773083 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f1415bc60 0x7f3f140ce9a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:59.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.308+0000 7f3f13200640 1 -- 172.21.15.47:0/3264773083 >> 172.21.15.47:0/3264773083 conn(0x7f3f140d1960 msgr2=0x7f3f140d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:25:59.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.308+0000 7f3f13200640 1 -- 172.21.15.47:0/3264773083 shutdown_connections 2024-02-19T15:25:59.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.308+0000 7f3f13200640 1 -- 172.21.15.47:0/3264773083 wait complete. 2024-02-19T15:25:59.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.308+0000 7f3f13200640 1 Processor -- start 2024-02-19T15:25:59.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.308+0000 7f3f13200640 1 -- start start 2024-02-19T15:25:59.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.308+0000 7f3f13200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f140d0650 0x7f3f1415ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:59.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.308+0000 7f3f13200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f140d2480 0x7f3f14159d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:59.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f13200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f1415bc60 0x7f3f141537e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:59.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f13200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f140c7d90 con 0x7f3f1415bc60 2024-02-19T15:25:59.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f13200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1415c250 con 0x7f3f140d0650 2024-02-19T15:25:59.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f13200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f140c7780 con 0x7f3f140d2480 2024-02-19T15:25:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f12800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f1415bc60 0x7f3f141537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f09400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f140d2480 0x7f3f14159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f11e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f140d0650 0x7f3f1415ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f11e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f140d0650 0x7f3f1415ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:53868/0 (socket says 172.21.15.47:53868) 2024-02-19T15:25:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f12800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f1415bc60 0x7f3f141537e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:46048/0 (socket says 172.21.15.47:46048) 2024-02-19T15:25:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f11e00640 1 -- 172.21.15.47:0/1467257154 learned_addr learned my addr 172.21.15.47:0/1467257154 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:25:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f09400640 1 -- 172.21.15.47:0/1467257154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f140d0650 msgr2=0x7f3f1415ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f09400640 1 --2- 172.21.15.47:0/1467257154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f140d0650 0x7f3f1415ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f09400640 1 -- 172.21.15.47:0/1467257154 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f1415bc60 msgr2=0x7f3f141537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:25:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f09400640 1 --2- 172.21.15.47:0/1467257154 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f1415bc60 0x7f3f141537e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:25:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f09400640 1 -- 172.21.15.47:0/1467257154 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f04009420 con 0x7f3f140d2480 2024-02-19T15:25:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f12800640 1 --2- 172.21.15.47:0/1467257154 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f1415bc60 0x7f3f141537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:25:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f09400640 1 --2- 172.21.15.47:0/1467257154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f140d2480 0x7f3f14159d60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3efc00cb90 tx=0x7f3efc00a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:59.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f0aa00640 1 -- 172.21.15.47:0/1467257154 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3efc00ad40 con 0x7f3f140d2480 2024-02-19T15:25:59.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f0aa00640 1 -- 172.21.15.47:0/1467257154 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3efc012400 con 0x7f3f140d2480 2024-02-19T15:25:59.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f0aa00640 1 -- 172.21.15.47:0/1467257154 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3efc00aea0 con 0x7f3f140d2480 2024-02-19T15:25:59.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f13200640 1 -- 172.21.15.47:0/1467257154 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f1415c4f0 con 0x7f3f140d2480 2024-02-19T15:25:59.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f13200640 1 -- 172.21.15.47:0/1467257154 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f140c7a20 con 0x7f3f140d2480 2024-02-19T15:25:59.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f13200640 1 -- 172.21.15.47:0/1467257154 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f140c7f40 con 0x7f3f140d2480 2024-02-19T15:25:59.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.312+0000 7f3f0aa00640 1 -- 172.21.15.47:0/1467257154 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3efc007070 con 0x7f3f140d2480 2024-02-19T15:25:59.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.316+0000 7f3f0aa00640 1 --2- 172.21.15.47:0/1467257154 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ef0066e40 0x7f3ef0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:25:59.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.316+0000 7f3f0aa00640 1 -- 172.21.15.47:0/1467257154 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f3efc084910 con 0x7f3f140d2480 2024-02-19T15:25:59.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.316+0000 7f3f11e00640 1 --2- 172.21.15.47:0/1467257154 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ef0066e40 0x7f3ef0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:25:59.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.316+0000 7f3f11e00640 1 --2- 172.21.15.47:0/1467257154 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ef0066e40 0x7f3ef0069300 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3f040097f0 tx=0x7f3f040a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:25:59.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.320+0000 7f3f0aa00640 1 -- 172.21.15.47:0/1467257154 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f3f140c7f40 con 0x7f3f140d2480 2024-02-19T15:25:59.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:25:59.516+0000 7f3f13200640 1 -- 172.21.15.47:0/1467257154 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f3f140cfc00 con 0x7f3f140d2480 2024-02-19T15:26:00.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.128+0000 7f3f0aa00640 1 -- 172.21.15.47:0/1467257154 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v45) v1 ==== 174+0+0 (secure 0 0 0) 0x7f3efc017020 con 0x7f3f140d2480 2024-02-19T15:26:00.137 INFO:teuthology.orchestra.run.smithi047.stderr:new fs with metadata pool 12 and data pool 13 2024-02-19T15:26:00.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.144+0000 7f3f13200640 1 -- 172.21.15.47:0/1467257154 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ef0066e40 msgr2=0x7f3ef0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:00.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.144+0000 7f3f13200640 1 --2- 172.21.15.47:0/1467257154 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ef0066e40 0x7f3ef0069300 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3f040097f0 tx=0x7f3f040a7000 comp rx=0 tx=0).stop 2024-02-19T15:26:00.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.144+0000 7f3f13200640 1 -- 172.21.15.47:0/1467257154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f140d2480 msgr2=0x7f3f14159d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:00.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.144+0000 7f3f13200640 1 --2- 172.21.15.47:0/1467257154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f140d2480 0x7f3f14159d60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3efc00cb90 tx=0x7f3efc00a430 comp rx=0 tx=0).stop 2024-02-19T15:26:00.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.148+0000 7f3f13200640 1 -- 172.21.15.47:0/1467257154 shutdown_connections 2024-02-19T15:26:00.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.148+0000 7f3f13200640 1 --2- 172.21.15.47:0/1467257154 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ef0066e40 0x7f3ef0069300 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.148+0000 7f3f13200640 1 --2- 172.21.15.47:0/1467257154 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f1415bc60 0x7f3f141537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.148+0000 7f3f13200640 1 --2- 172.21.15.47:0/1467257154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f140d2480 0x7f3f14159d60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.148+0000 7f3f13200640 1 --2- 172.21.15.47:0/1467257154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f140d0650 0x7f3f1415ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.148+0000 7f3f13200640 1 -- 172.21.15.47:0/1467257154 >> 172.21.15.47:0/1467257154 conn(0x7f3f140d1960 msgr2=0x7f3f1406e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:00.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.152+0000 7f3f13200640 1 -- 172.21.15.47:0/1467257154 shutdown_connections 2024-02-19T15:26:00.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.156+0000 7f3f13200640 1 -- 172.21.15.47:0/1467257154 wait complete. 2024-02-19T15:26:00.187 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:26:00.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.324+0000 7f9ff8e00640 1 Processor -- start 2024-02-19T15:26:00.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.324+0000 7f9ff8e00640 1 -- start start 2024-02-19T15:26:00.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.324+0000 7f9ff8e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9ff40b1be0 0x7f9ff40d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:00.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.324+0000 7f9ff8e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff40b0280 con 0x7f9ff40b0a80 2024-02-19T15:26:00.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.324+0000 7f9ff8e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff40b04d0 con 0x7f9ff415f590 2024-02-19T15:26:00.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.324+0000 7f9ff8e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ff40cc530 con 0x7f9ff40b1be0 2024-02-19T15:26:00.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.324+0000 7f9fe9400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9ff40b1be0 0x7f9ff40d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:00.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.324+0000 7f9ff2800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f9ff40b0a80 0x7f9ff40b20e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36072/0 (socket says 172.21.15.47:36072) 2024-02-19T15:26:00.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.324+0000 7f9ff2800640 1 -- 172.21.15.47:0/783952024 learned_addr learned my addr 172.21.15.47:0/783952024 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:00.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.324+0000 7f9fe9400640 1 -- 172.21.15.47:0/783952024 >> v1:172.21.15.116:6789/0 conn(0x7f9ff415f590 legacy=0x7f9ff40b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:00.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.324+0000 7f9fe9400640 1 -- 172.21.15.47:0/783952024 >> v1:172.21.15.47:6789/0 conn(0x7f9ff40b0a80 legacy=0x7f9ff40b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:00.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.324+0000 7f9ff1400640 1 -- 172.21.15.47:0/783952024 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176022768 0 0) 0x7f9ff40b0280 con 0x7f9ff40b0a80 2024-02-19T15:26:00.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.324+0000 7f9fe9400640 1 -- 172.21.15.47:0/783952024 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff40b07b0 con 0x7f9ff40b1be0 2024-02-19T15:26:00.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.328+0000 7f9fe9400640 1 --2- 172.21.15.47:0/783952024 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ff40b1be0 0x7f9ff40d2690 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc0097c0 tx=0x7f9fdc01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb7cfde0880f0815 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:00.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.328+0000 7f9ff1400640 1 -- 172.21.15.47:0/783952024 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc0ab670 con 0x7f9ff40b1be0 2024-02-19T15:26:00.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.328+0000 7f9ff1400640 1 -- 172.21.15.47:0/783952024 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fdc0ab7d0 con 0x7f9ff40b1be0 2024-02-19T15:26:00.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.328+0000 7f9ff1400640 1 -- 172.21.15.47:0/783952024 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc0abb60 con 0x7f9ff40b1be0 2024-02-19T15:26:00.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.328+0000 7f9ff8e00640 1 -- 172.21.15.47:0/783952024 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ff40b1be0 msgr2=0x7f9ff40d2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:00.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.328+0000 7f9ff8e00640 1 --2- 172.21.15.47:0/783952024 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ff40b1be0 0x7f9ff40d2690 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc0097c0 tx=0x7f9fdc01c470 comp rx=0 tx=0).stop 2024-02-19T15:26:00.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.328+0000 7f9ff8e00640 1 -- 172.21.15.47:0/783952024 shutdown_connections 2024-02-19T15:26:00.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.328+0000 7f9ff8e00640 1 --2- 172.21.15.47:0/783952024 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ff40b1be0 0x7f9ff40d2690 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.328+0000 7f9ff8e00640 1 -- 172.21.15.47:0/783952024 >> 172.21.15.47:0/783952024 conn(0x7f9ff40d1b70 msgr2=0x7f9ff40d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:00.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.328+0000 7f9ff8e00640 1 -- 172.21.15.47:0/783952024 shutdown_connections 2024-02-19T15:26:00.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.328+0000 7f9ff8e00640 1 -- 172.21.15.47:0/783952024 wait complete. 2024-02-19T15:26:00.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9ff8e00640 1 Processor -- start 2024-02-19T15:26:00.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9ff8e00640 1 -- start start 2024-02-19T15:26:00.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9ff8e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ff40b0a80 0x7f9ff4152990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:00.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9ff8e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ff40b1be0 0x7f9ff406d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:00.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9ff8e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ff415f590 0x7f9ff406d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:00.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9ff8e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40c61e0 con 0x7f9ff415f590 2024-02-19T15:26:00.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9ff8e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40cac70 con 0x7f9ff40b1be0 2024-02-19T15:26:00.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9ff8e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40caf50 con 0x7f9ff40b0a80 2024-02-19T15:26:00.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9ff2800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ff415f590 0x7f9ff406d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:00.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9ff2800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ff415f590 0x7f9ff406d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:46060/0 (socket says 172.21.15.47:46060) 2024-02-19T15:26:00.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9ff2800640 1 -- 172.21.15.47:0/3503441892 learned_addr learned my addr 172.21.15.47:0/3503441892 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:00.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9fe9400640 1 --2- 172.21.15.47:0/3503441892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ff40b1be0 0x7f9ff406d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:00.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9ff1e00640 1 --2- 172.21.15.47:0/3503441892 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ff40b0a80 0x7f9ff4152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:00.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9ff2800640 1 -- 172.21.15.47:0/3503441892 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ff40b0a80 msgr2=0x7f9ff4152990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:00.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9ff2800640 1 --2- 172.21.15.47:0/3503441892 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ff40b0a80 0x7f9ff4152990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9ff2800640 1 -- 172.21.15.47:0/3503441892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ff40b1be0 msgr2=0x7f9ff406d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:00.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9ff2800640 1 --2- 172.21.15.47:0/3503441892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ff40b1be0 0x7f9ff406d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9ff2800640 1 -- 172.21.15.47:0/3503441892 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fdc009420 con 0x7f9ff415f590 2024-02-19T15:26:00.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9ff2800640 1 --2- 172.21.15.47:0/3503441892 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ff415f590 0x7f9ff406d550 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9fec00c480 tx=0x7f9fec00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:00.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9feaa00640 1 -- 172.21.15.47:0/3503441892 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec0074e0 con 0x7f9ff415f590 2024-02-19T15:26:00.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9feaa00640 1 -- 172.21.15.47:0/3503441892 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fec007640 con 0x7f9ff415f590 2024-02-19T15:26:00.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9ff8e00640 1 -- 172.21.15.47:0/3503441892 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff40cb230 con 0x7f9ff415f590 2024-02-19T15:26:00.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9feaa00640 1 -- 172.21.15.47:0/3503441892 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec004a10 con 0x7f9ff415f590 2024-02-19T15:26:00.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.332+0000 7f9ff8e00640 1 -- 172.21.15.47:0/3503441892 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff40cb7f0 con 0x7f9ff415f590 2024-02-19T15:26:00.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.336+0000 7f9ff8e00640 1 -- 172.21.15.47:0/3503441892 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff4075360 con 0x7f9ff415f590 2024-02-19T15:26:00.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.336+0000 7f9feaa00640 1 -- 172.21.15.47:0/3503441892 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9fec004b70 con 0x7f9ff415f590 2024-02-19T15:26:00.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.336+0000 7f9feaa00640 1 --2- 172.21.15.47:0/3503441892 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9fd0066e40 0x7f9fd0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:00.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.336+0000 7f9feaa00640 1 -- 172.21.15.47:0/3503441892 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9fec083fb0 con 0x7f9ff415f590 2024-02-19T15:26:00.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.336+0000 7f9ff1e00640 1 --2- 172.21.15.47:0/3503441892 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9fd0066e40 0x7f9fd0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:00.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.336+0000 7f9ff1e00640 1 --2- 172.21.15.47:0/3503441892 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9fd0066e40 0x7f9fd0069300 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f9fe40012f0 tx=0x7f9fe4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:00.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.340+0000 7f9feaa00640 1 -- 172.21.15.47:0/3503441892 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f9fec050db0 con 0x7f9ff415f590 2024-02-19T15:26:00.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.504+0000 7f9ff8e00640 1 -- 172.21.15.47:0/3503441892 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ff40cb9e0 con 0x7f9ff415f590 2024-02-19T15:26:00.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.504+0000 7f9feaa00640 1 -- 172.21.15.47:0/3503441892 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16685 (secure 0 0 0) 0x7f9fec018020 con 0x7f9ff415f590 2024-02-19T15:26:00.509 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:00.509 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":81,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:25:59.521909+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:25:57.938354+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-02-19T15:25:58.514866+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:26:00.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.508+0000 7f9ff8e00640 1 -- 172.21.15.47:0/3503441892 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9fd0066e40 msgr2=0x7f9fd0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:00.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.508+0000 7f9ff8e00640 1 --2- 172.21.15.47:0/3503441892 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9fd0066e40 0x7f9fd0069300 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f9fe40012f0 tx=0x7f9fe4009040 comp rx=0 tx=0).stop 2024-02-19T15:26:00.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.508+0000 7f9ff8e00640 1 -- 172.21.15.47:0/3503441892 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ff415f590 msgr2=0x7f9ff406d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:00.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.508+0000 7f9ff8e00640 1 --2- 172.21.15.47:0/3503441892 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ff415f590 0x7f9ff406d550 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9fec00c480 tx=0x7f9fec00c940 comp rx=0 tx=0).stop 2024-02-19T15:26:00.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.508+0000 7f9ff8e00640 1 -- 172.21.15.47:0/3503441892 shutdown_connections 2024-02-19T15:26:00.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.508+0000 7f9ff8e00640 1 --2- 172.21.15.47:0/3503441892 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9fd0066e40 0x7f9fd0069300 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.508+0000 7f9ff8e00640 1 --2- 172.21.15.47:0/3503441892 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ff415f590 0x7f9ff406d550 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.508+0000 7f9ff8e00640 1 --2- 172.21.15.47:0/3503441892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ff40b1be0 0x7f9ff406d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.508+0000 7f9ff8e00640 1 --2- 172.21.15.47:0/3503441892 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ff40b0a80 0x7f9ff4152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.508+0000 7f9ff8e00640 1 -- 172.21.15.47:0/3503441892 >> 172.21.15.47:0/3503441892 conn(0x7f9ff40d1b70 msgr2=0x7f9ff4074150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:00.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.508+0000 7f9ff8e00640 1 -- 172.21.15.47:0/3503441892 shutdown_connections 2024-02-19T15:26:00.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.508+0000 7f9ff8e00640 1 -- 172.21.15.47:0/3503441892 wait complete. 2024-02-19T15:26:00.530 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:26:00.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.664+0000 7f76c5c00640 1 Processor -- start 2024-02-19T15:26:00.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.664+0000 7f76c5c00640 1 -- start start 2024-02-19T15:26:00.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.668+0000 7f76c5c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f76c0145590 0x7f76c00727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:00.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.668+0000 7f76c5c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76c00cea80 0x7f76c006c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:00.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.668+0000 7f76c5c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f76c015c390 0x7f76c015c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:00.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.668+0000 7f76c5c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76c015d280 con 0x7f76c015c390 2024-02-19T15:26:00.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.668+0000 7f76c5c00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76c0070ff0 con 0x7f76c0145590 2024-02-19T15:26:00.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.668+0000 7f76c5c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76c015cd60 con 0x7f76c00cea80 2024-02-19T15:26:00.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.668+0000 7f76bf400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f76c015c390 0x7f76c015c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:00.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.668+0000 7f76bf400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f76c015c390 0x7f76c015c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:46074/0 (socket says 172.21.15.47:46074) 2024-02-19T15:26:00.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.668+0000 7f76bf400640 1 -- 172.21.15.47:0/505375363 learned_addr learned my addr 172.21.15.47:0/505375363 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:00.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.668+0000 7f76bea00640 1 --2- 172.21.15.47:0/505375363 >> v2:172.21.15.116:3300/0 conn(0x7f76c0145590 0x7f76c00727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:00.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.668+0000 7f76b7e00640 1 --2- 172.21.15.47:0/505375363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76c00cea80 0x7f76c006c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:00.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.668+0000 7f76bf400640 1 -- 172.21.15.47:0/505375363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76c00cea80 msgr2=0x7f76c006c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:00.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.668+0000 7f76bf400640 1 --2- 172.21.15.47:0/505375363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76c00cea80 0x7f76c006c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.668+0000 7f76bf400640 1 -- 172.21.15.47:0/505375363 >> v2:172.21.15.116:3300/0 conn(0x7f76c0145590 msgr2=0x7f76c00727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:00.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.668+0000 7f76bf400640 1 --2- 172.21.15.47:0/505375363 >> v2:172.21.15.116:3300/0 conn(0x7f76c0145590 0x7f76c00727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.668+0000 7f76bf400640 1 -- 172.21.15.47:0/505375363 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c00709e0 con 0x7f76c015c390 2024-02-19T15:26:00.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.668+0000 7f76b7e00640 1 --2- 172.21.15.47:0/505375363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76c00cea80 0x7f76c006c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:00.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76bf400640 1 --2- 172.21.15.47:0/505375363 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76c015c390 0x7f76c015c820 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f76b8009e50 tx=0x7f76b801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba9f2daee6668e34 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:00.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76be000640 1 -- 172.21.15.47:0/505375363 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b80a5780 con 0x7f76c015c390 2024-02-19T15:26:00.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76be000640 1 -- 172.21.15.47:0/505375363 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b80a58e0 con 0x7f76c015c390 2024-02-19T15:26:00.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76be000640 1 -- 172.21.15.47:0/505375363 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b80a5c70 con 0x7f76c015c390 2024-02-19T15:26:00.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76c5c00640 1 -- 172.21.15.47:0/505375363 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76c015c390 msgr2=0x7f76c015c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:00.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76c5c00640 1 --2- 172.21.15.47:0/505375363 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76c015c390 0x7f76c015c820 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f76b8009e50 tx=0x7f76b801c9f0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76c5c00640 1 -- 172.21.15.47:0/505375363 shutdown_connections 2024-02-19T15:26:00.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76c5c00640 1 --2- 172.21.15.47:0/505375363 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76c015c390 0x7f76c015c820 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76c5c00640 1 --2- 172.21.15.47:0/505375363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76c00cea80 0x7f76c006c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76c5c00640 1 --2- 172.21.15.47:0/505375363 >> v2:172.21.15.116:3300/0 conn(0x7f76c0145590 0x7f76c00727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76c5c00640 1 -- 172.21.15.47:0/505375363 >> 172.21.15.47:0/505375363 conn(0x7f76c00b16a0 msgr2=0x7f76c00b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:00.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76c5c00640 1 -- 172.21.15.47:0/505375363 shutdown_connections 2024-02-19T15:26:00.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76c5c00640 1 -- 172.21.15.47:0/505375363 wait complete. 2024-02-19T15:26:00.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76c5c00640 1 Processor -- start 2024-02-19T15:26:00.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76c5c00640 1 -- start start 2024-02-19T15:26:00.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76c5c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c00cea80 0x7f76c006d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:00.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76c5c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76c0145590 0x7f76c015b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:00.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76c5c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76c015c390 0x7f76c015b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:00.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76c5c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c00c53d0 con 0x7f76c015c390 2024-02-19T15:26:00.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76c5c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c00b35c0 con 0x7f76c00cea80 2024-02-19T15:26:00.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.672+0000 7f76c5c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76c00c4a30 con 0x7f76c0145590 2024-02-19T15:26:00.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.676+0000 7f76bf400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76c015c390 0x7f76c015b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:00.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.676+0000 7f76bf400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76c015c390 0x7f76c015b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:46084/0 (socket says 172.21.15.47:46084) 2024-02-19T15:26:00.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.676+0000 7f76bf400640 1 -- 172.21.15.47:0/356863875 learned_addr learned my addr 172.21.15.47:0/356863875 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:00.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.676+0000 7f76b7e00640 1 --2- 172.21.15.47:0/356863875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76c0145590 0x7f76c015b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:00.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.676+0000 7f76bea00640 1 --2- 172.21.15.47:0/356863875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c00cea80 0x7f76c006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:00.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.676+0000 7f76bf400640 1 -- 172.21.15.47:0/356863875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76c0145590 msgr2=0x7f76c015b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:00.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.676+0000 7f76bf400640 1 --2- 172.21.15.47:0/356863875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76c0145590 0x7f76c015b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.676+0000 7f76bf400640 1 -- 172.21.15.47:0/356863875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c00cea80 msgr2=0x7f76c006d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:00.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.676+0000 7f76bf400640 1 --2- 172.21.15.47:0/356863875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c00cea80 0x7f76c006d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.676+0000 7f76bf400640 1 -- 172.21.15.47:0/356863875 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b8009b30 con 0x7f76c015c390 2024-02-19T15:26:00.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.676+0000 7f76bf400640 1 --2- 172.21.15.47:0/356863875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76c015c390 0x7f76c015b7c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f76b801b0a0 tx=0x7f76b8019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:00.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.676+0000 7f76b6a00640 1 -- 172.21.15.47:0/356863875 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b80a5780 con 0x7f76c015c390 2024-02-19T15:26:00.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.676+0000 7f76b6a00640 1 -- 172.21.15.47:0/356863875 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b80b8880 con 0x7f76c015c390 2024-02-19T15:26:00.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.676+0000 7f76b6a00640 1 -- 172.21.15.47:0/356863875 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b80b8c30 con 0x7f76c015c390 2024-02-19T15:26:00.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.676+0000 7f76c5c00640 1 -- 172.21.15.47:0/356863875 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76c00c4d10 con 0x7f76c015c390 2024-02-19T15:26:00.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.676+0000 7f76c5c00640 1 -- 172.21.15.47:0/356863875 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76c006d8a0 con 0x7f76c015c390 2024-02-19T15:26:00.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.676+0000 7f76c5c00640 1 -- 172.21.15.47:0/356863875 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76c00cf2e0 con 0x7f76c015c390 2024-02-19T15:26:00.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.676+0000 7f76b6a00640 1 -- 172.21.15.47:0/356863875 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f76b80c3460 con 0x7f76c015c390 2024-02-19T15:26:00.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.676+0000 7f76b6a00640 1 --2- 172.21.15.47:0/356863875 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f769c066e40 0x7f769c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:00.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.680+0000 7f76b6a00640 1 -- 172.21.15.47:0/356863875 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f76b811fff0 con 0x7f76c015c390 2024-02-19T15:26:00.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.680+0000 7f76bea00640 1 --2- 172.21.15.47:0/356863875 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f769c066e40 0x7f769c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:00.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.680+0000 7f76bea00640 1 --2- 172.21.15.47:0/356863875 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f769c066e40 0x7f769c069300 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f76c00b1170 tx=0x7f76b0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:00.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.684+0000 7f76b6a00640 1 -- 172.21.15.47:0/356863875 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f76c00cf2e0 con 0x7f76c015c390 2024-02-19T15:26:00.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.848+0000 7f76c5c00640 1 -- 172.21.15.47:0/356863875 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76c00cf2e0 con 0x7f76c015c390 2024-02-19T15:26:00.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.852+0000 7f76b6a00640 1 -- 172.21.15.47:0/356863875 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16685 (secure 0 0 0) 0x7f76c00cf2e0 con 0x7f76c015c390 2024-02-19T15:26:00.855 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:00.855 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":81,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:25:59.521909+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:25:57.938354+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-02-19T15:25:58.514866+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:26:00.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.852+0000 7f76c5c00640 1 -- 172.21.15.47:0/356863875 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f769c066e40 msgr2=0x7f769c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:00.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.852+0000 7f76c5c00640 1 --2- 172.21.15.47:0/356863875 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f769c066e40 0x7f769c069300 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f76c00b1170 tx=0x7f76b0008040 comp rx=0 tx=0).stop 2024-02-19T15:26:00.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.852+0000 7f76c5c00640 1 -- 172.21.15.47:0/356863875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76c015c390 msgr2=0x7f76c015b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:00.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.852+0000 7f76c5c00640 1 --2- 172.21.15.47:0/356863875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76c015c390 0x7f76c015b7c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f76b801b0a0 tx=0x7f76b8019790 comp rx=0 tx=0).stop 2024-02-19T15:26:00.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.852+0000 7f76c5c00640 1 -- 172.21.15.47:0/356863875 shutdown_connections 2024-02-19T15:26:00.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.852+0000 7f76c5c00640 1 --2- 172.21.15.47:0/356863875 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f769c066e40 0x7f769c069300 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.852+0000 7f76c5c00640 1 --2- 172.21.15.47:0/356863875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76c015c390 0x7f76c015b7c0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.852+0000 7f76c5c00640 1 --2- 172.21.15.47:0/356863875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76c0145590 0x7f76c015b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.852+0000 7f76c5c00640 1 --2- 172.21.15.47:0/356863875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c00cea80 0x7f76c006d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:00.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.852+0000 7f76c5c00640 1 -- 172.21.15.47:0/356863875 >> 172.21.15.47:0/356863875 conn(0x7f76c00b16a0 msgr2=0x7f76c0144860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:00.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.852+0000 7f76c5c00640 1 -- 172.21.15.47:0/356863875 shutdown_connections 2024-02-19T15:26:00.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:00.852+0000 7f76c5c00640 1 -- 172.21.15.47:0/356863875 wait complete. 2024-02-19T15:26:00.876 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T15:26:01.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.016+0000 7f43dc200640 1 Processor -- start 2024-02-19T15:26:01.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.016+0000 7f43dc200640 1 -- start start 2024-02-19T15:26:01.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.016+0000 7f43dc200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43d40c7e80 0x7f43d40d0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:01.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.016+0000 7f43dc200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f43d40cfbf0 0x7f43d40d0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:01.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.016+0000 7f43dc200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d40cd750 con 0x7f43d40c8260 2024-02-19T15:26:01.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.016+0000 7f43dc200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d40ceee0 con 0x7f43d40cfbf0 2024-02-19T15:26:01.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.016+0000 7f43dc200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d40cdc60 con 0x7f43d40c7e80 2024-02-19T15:26:01.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.016+0000 7f43da200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f43d40cfbf0 0x7f43d40d0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:01.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.016+0000 7f43da200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f43d40cfbf0 0x7f43d40d0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:46092/0 (socket says 172.21.15.47:46092) 2024-02-19T15:26:01.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.016+0000 7f43da200640 1 -- 172.21.15.47:0/2630239119 learned_addr learned my addr 172.21.15.47:0/2630239119 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:01.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.016+0000 7f43d9800640 1 --2- 172.21.15.47:0/2630239119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43d40c7e80 0x7f43d40d0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:01.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.016+0000 7f43da200640 1 -- 172.21.15.47:0/2630239119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43d40c7e80 msgr2=0x7f43d40d0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:01.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.016+0000 7f43da200640 1 --2- 172.21.15.47:0/2630239119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43d40c7e80 0x7f43d40d0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:01.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.016+0000 7f43da200640 1 -- 172.21.15.47:0/2630239119 >> v1:172.21.15.116:6789/0 conn(0x7f43d40c8260 legacy=0x7f43d40c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:01.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.016+0000 7f43da200640 1 -- 172.21.15.47:0/2630239119 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d40d2a40 con 0x7f43d40cfbf0 2024-02-19T15:26:01.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.016+0000 7f43d9800640 1 --2- 172.21.15.47:0/2630239119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43d40c7e80 0x7f43d40d0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:01.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.020+0000 7f43da200640 1 --2- 172.21.15.47:0/2630239119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43d40cfbf0 0x7f43d40d0650 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f43cc009e50 tx=0x7f43cc01ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4430ef3840901abb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:01.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.020+0000 7f43d3e00640 1 -- 172.21.15.47:0/2630239119 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc0a58d0 con 0x7f43d40cfbf0 2024-02-19T15:26:01.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.020+0000 7f43d3e00640 1 -- 172.21.15.47:0/2630239119 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43cc0a5a30 con 0x7f43d40cfbf0 2024-02-19T15:26:01.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.020+0000 7f43d3e00640 1 -- 172.21.15.47:0/2630239119 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc0a5dc0 con 0x7f43d40cfbf0 2024-02-19T15:26:01.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.020+0000 7f43dc200640 1 -- 172.21.15.47:0/2630239119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43d40cfbf0 msgr2=0x7f43d40d0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:01.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.020+0000 7f43dc200640 1 --2- 172.21.15.47:0/2630239119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43d40cfbf0 0x7f43d40d0650 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f43cc009e50 tx=0x7f43cc01ca00 comp rx=0 tx=0).stop 2024-02-19T15:26:01.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.020+0000 7f43dc200640 1 -- 172.21.15.47:0/2630239119 shutdown_connections 2024-02-19T15:26:01.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.020+0000 7f43dc200640 1 --2- 172.21.15.47:0/2630239119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43d40cfbf0 0x7f43d40d0650 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:01.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.020+0000 7f43dc200640 1 --2- 172.21.15.47:0/2630239119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43d40c7e80 0x7f43d40d0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:01.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.020+0000 7f43dc200640 1 -- 172.21.15.47:0/2630239119 >> 172.21.15.47:0/2630239119 conn(0x7f43d40d1b30 msgr2=0x7f43d40d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:01.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.020+0000 7f43dc200640 1 -- 172.21.15.47:0/2630239119 shutdown_connections 2024-02-19T15:26:01.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.020+0000 7f43dc200640 1 -- 172.21.15.47:0/2630239119 wait complete. 2024-02-19T15:26:01.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43dc200640 1 Processor -- start 2024-02-19T15:26:01.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43dc200640 1 -- start start 2024-02-19T15:26:01.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43dc200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43d40c7e80 0x7f43d414e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:01.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43dc200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43d40c8260 0x7f43d40b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43dc200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d40cfbf0 0x7f43d406d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43dc200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d40b1e30 con 0x7f43d40c7e80 2024-02-19T15:26:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43dc200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d414e550 con 0x7f43d40cfbf0 2024-02-19T15:26:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43dc200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d40b0f40 con 0x7f43d40c8260 2024-02-19T15:26:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43d9800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43d40c7e80 0x7f43d414e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43d9800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43d40c7e80 0x7f43d414e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:46102/0 (socket says 172.21.15.47:46102) 2024-02-19T15:26:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43d9800640 1 -- 172.21.15.47:0/1348395339 learned_addr learned my addr 172.21.15.47:0/1348395339 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43da200640 1 --2- 172.21.15.47:0/1348395339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d40cfbf0 0x7f43d406d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43d8e00640 1 --2- 172.21.15.47:0/1348395339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43d40c8260 0x7f43d40b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43d9800640 1 -- 172.21.15.47:0/1348395339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43d40c8260 msgr2=0x7f43d40b01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43d9800640 1 --2- 172.21.15.47:0/1348395339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43d40c8260 0x7f43d40b01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43d9800640 1 -- 172.21.15.47:0/1348395339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d40cfbf0 msgr2=0x7f43d406d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43d9800640 1 --2- 172.21.15.47:0/1348395339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d40cfbf0 0x7f43d406d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43d9800640 1 -- 172.21.15.47:0/1348395339 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43cc009b30 con 0x7f43d40c7e80 2024-02-19T15:26:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43d9800640 1 --2- 172.21.15.47:0/1348395339 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43d40c7e80 0x7f43d414e010 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f43c800c7f0 tx=0x7f43c800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:01.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43d1600640 1 -- 172.21.15.47:0/1348395339 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c8007820 con 0x7f43d40c7e80 2024-02-19T15:26:01.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43d1600640 1 -- 172.21.15.47:0/1348395339 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c8007980 con 0x7f43d40c7e80 2024-02-19T15:26:01.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43d1600640 1 -- 172.21.15.47:0/1348395339 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c8007d30 con 0x7f43d40c7e80 2024-02-19T15:26:01.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43dc200640 1 -- 172.21.15.47:0/1348395339 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d40b1220 con 0x7f43d40c7e80 2024-02-19T15:26:01.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43dc200640 1 -- 172.21.15.47:0/1348395339 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d40b17e0 con 0x7f43d40c7e80 2024-02-19T15:26:01.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.024+0000 7f43dc200640 1 -- 172.21.15.47:0/1348395339 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d40d1180 con 0x7f43d40c7e80 2024-02-19T15:26:01.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.028+0000 7f43d1600640 1 -- 172.21.15.47:0/1348395339 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f43c80113f0 con 0x7f43d40c7e80 2024-02-19T15:26:01.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.028+0000 7f43d1600640 1 --2- 172.21.15.47:0/1348395339 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f43b4066e40 0x7f43b4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:01.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.028+0000 7f43d1600640 1 -- 172.21.15.47:0/1348395339 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43c8084dc0 con 0x7f43d40c7e80 2024-02-19T15:26:01.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.028+0000 7f43d8e00640 1 --2- 172.21.15.47:0/1348395339 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f43b4066e40 0x7f43b4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:01.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.028+0000 7f43d8e00640 1 --2- 172.21.15.47:0/1348395339 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f43b4066e40 0x7f43b4069300 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f43d41faab0 tx=0x7f43c0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:01.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.032+0000 7f43d1600640 1 -- 172.21.15.47:0/1348395339 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f43c8051b30 con 0x7f43d40c7e80 2024-02-19T15:26:01.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:01.232+0000 7f43dc200640 1 -- 172.21.15.47:0/1348395339 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f43d415cc40 con 0x7f43d40c7e80 2024-02-19T15:26:02.114 INFO:tasks.ceph.mds.c.smithi047.stderr:2024-02-19T15:26:02.108+0000 7f366e200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T15:26:02.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.148+0000 7f43d1600640 1 -- 172.21.15.47:0/1348395339 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v48) v1 ==== 118+0+0 (secure 0 0 0) 0x7f43c801f020 con 0x7f43d40c7e80 2024-02-19T15:26:02.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.148+0000 7f43dc200640 1 -- 172.21.15.47:0/1348395339 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f43b4066e40 msgr2=0x7f43b4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:02.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.148+0000 7f43dc200640 1 --2- 172.21.15.47:0/1348395339 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f43b4066e40 0x7f43b4069300 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f43d41faab0 tx=0x7f43c0009040 comp rx=0 tx=0).stop 2024-02-19T15:26:02.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.148+0000 7f43dc200640 1 -- 172.21.15.47:0/1348395339 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43d40c7e80 msgr2=0x7f43d414e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:02.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.148+0000 7f43dc200640 1 --2- 172.21.15.47:0/1348395339 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43d40c7e80 0x7f43d414e010 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f43c800c7f0 tx=0x7f43c800ccb0 comp rx=0 tx=0).stop 2024-02-19T15:26:02.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.148+0000 7f43dc200640 1 -- 172.21.15.47:0/1348395339 shutdown_connections 2024-02-19T15:26:02.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.148+0000 7f43dc200640 1 --2- 172.21.15.47:0/1348395339 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f43b4066e40 0x7f43b4069300 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:02.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.148+0000 7f43dc200640 1 --2- 172.21.15.47:0/1348395339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d40cfbf0 0x7f43d406d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:02.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.148+0000 7f43dc200640 1 --2- 172.21.15.47:0/1348395339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43d40c8260 0x7f43d40b01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:02.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.148+0000 7f43dc200640 1 --2- 172.21.15.47:0/1348395339 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43d40c7e80 0x7f43d414e010 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:02.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.148+0000 7f43dc200640 1 -- 172.21.15.47:0/1348395339 >> 172.21.15.47:0/1348395339 conn(0x7f43d40d1b30 msgr2=0x7f43d40c6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:02.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.148+0000 7f43dc200640 1 -- 172.21.15.47:0/1348395339 shutdown_connections 2024-02-19T15:26:02.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.148+0000 7f43dc200640 1 -- 172.21.15.47:0/1348395339 wait complete. 2024-02-19T15:26:02.173 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T15:26:02.174 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T15:26:02.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.312+0000 7fb11c600640 1 Processor -- start 2024-02-19T15:26:02.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.312+0000 7fb11c600640 1 -- start start 2024-02-19T15:26:02.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.312+0000 7fb11c600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb1140d0820 0x7fb1140cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:02.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.312+0000 7fb11c600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb1140c8260 0x7fb1140ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:02.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.312+0000 7fb11c600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1140ceee0 con 0x7fb1140c7e80 2024-02-19T15:26:02.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.312+0000 7fb11c600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1140cdc60 con 0x7fb1140d0820 2024-02-19T15:26:02.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.312+0000 7fb11c600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1140d2a40 con 0x7fb1140c8260 2024-02-19T15:26:02.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.312+0000 7fb119200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb1140d0820 0x7fb1140cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:02.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.312+0000 7fb119200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb1140d0820 0x7fb1140cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:46116/0 (socket says 172.21.15.47:46116) 2024-02-19T15:26:02.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.312+0000 7fb119200640 1 -- 172.21.15.47:0/2737376827 learned_addr learned my addr 172.21.15.47:0/2737376827 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:02.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.312+0000 7fb11a600640 1 --2- 172.21.15.47:0/2737376827 >> v2:172.21.15.116:3300/0 conn(0x7fb1140c8260 0x7fb1140ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:02.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.312+0000 7fb119200640 1 -- 172.21.15.47:0/2737376827 >> v2:172.21.15.116:3300/0 conn(0x7fb1140c8260 msgr2=0x7fb1140ccc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:02.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.312+0000 7fb119200640 1 --2- 172.21.15.47:0/2737376827 >> v2:172.21.15.116:3300/0 conn(0x7fb1140c8260 0x7fb1140ccc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:02.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.312+0000 7fb119200640 1 -- 172.21.15.47:0/2737376827 >> v1:172.21.15.116:6789/0 conn(0x7fb1140c7e80 legacy=0x7fb1140d0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:02.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.312+0000 7fb119200640 1 -- 172.21.15.47:0/2737376827 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1140cd980 con 0x7fb1140d0820 2024-02-19T15:26:02.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.320+0000 7fb119200640 1 --2- 172.21.15.47:0/2737376827 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1140d0820 0x7fb1140cc740 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fb1140b7be0 tx=0x7fb10001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b20eb9b9814e5ff4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:02.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.320+0000 7fb113e00640 1 -- 172.21.15.47:0/2737376827 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1000ab650 con 0x7fb1140d0820 2024-02-19T15:26:02.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.320+0000 7fb113e00640 1 -- 172.21.15.47:0/2737376827 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1000ab7b0 con 0x7fb1140d0820 2024-02-19T15:26:02.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.320+0000 7fb113e00640 1 -- 172.21.15.47:0/2737376827 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1000abb40 con 0x7fb1140d0820 2024-02-19T15:26:02.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.320+0000 7fb11c600640 1 -- 172.21.15.47:0/2737376827 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1140d0820 msgr2=0x7fb1140cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:02.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.320+0000 7fb11c600640 1 --2- 172.21.15.47:0/2737376827 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1140d0820 0x7fb1140cc740 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fb1140b7be0 tx=0x7fb10001c4f0 comp rx=0 tx=0).stop 2024-02-19T15:26:02.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.320+0000 7fb11c600640 1 -- 172.21.15.47:0/2737376827 shutdown_connections 2024-02-19T15:26:02.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.320+0000 7fb11c600640 1 --2- 172.21.15.47:0/2737376827 >> v2:172.21.15.116:3300/0 conn(0x7fb1140c8260 0x7fb1140ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:02.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.320+0000 7fb11c600640 1 --2- 172.21.15.47:0/2737376827 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1140d0820 0x7fb1140cc740 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:02.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.320+0000 7fb11c600640 1 -- 172.21.15.47:0/2737376827 >> 172.21.15.47:0/2737376827 conn(0x7fb1140d1b30 msgr2=0x7fb1140d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:02.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.320+0000 7fb11c600640 1 -- 172.21.15.47:0/2737376827 shutdown_connections 2024-02-19T15:26:02.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.320+0000 7fb11c600640 1 -- 172.21.15.47:0/2737376827 wait complete. 2024-02-19T15:26:02.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.320+0000 7fb11c600640 1 Processor -- start 2024-02-19T15:26:02.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.320+0000 7fb11c600640 1 -- start start 2024-02-19T15:26:02.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb11c600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1140c7e80 0x7fb11414ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:02.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb11c600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1140c8260 0x7fb11415b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:02.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb11c600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1140d0820 0x7fb114159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:02.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb11c600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1140b1690 con 0x7fb1140c7e80 2024-02-19T15:26:02.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb11c600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11414b360 con 0x7fb1140c8260 2024-02-19T15:26:02.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb11c600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb114158c50 con 0x7fb1140d0820 2024-02-19T15:26:02.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb119c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1140c7e80 0x7fb11414ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:02.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb119c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1140c7e80 0x7fb11414ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:46120/0 (socket says 172.21.15.47:46120) 2024-02-19T15:26:02.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb119c00640 1 -- 172.21.15.47:0/670297180 learned_addr learned my addr 172.21.15.47:0/670297180 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:02.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb119200640 1 --2- 172.21.15.47:0/670297180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1140c8260 0x7fb11415b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:02.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb11a600640 1 --2- 172.21.15.47:0/670297180 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1140d0820 0x7fb114159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:02.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb119c00640 1 -- 172.21.15.47:0/670297180 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1140d0820 msgr2=0x7fb114159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:02.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb119c00640 1 --2- 172.21.15.47:0/670297180 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1140d0820 0x7fb114159440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:02.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb119c00640 1 -- 172.21.15.47:0/670297180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1140c8260 msgr2=0x7fb11415b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:02.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb119c00640 1 --2- 172.21.15.47:0/670297180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1140c8260 0x7fb11415b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:02.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb119c00640 1 -- 172.21.15.47:0/670297180 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1000094d0 con 0x7fb1140c7e80 2024-02-19T15:26:02.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb119c00640 1 --2- 172.21.15.47:0/670297180 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1140c7e80 0x7fb11414ae20 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fb10800c450 tx=0x7fb10800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:02.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb111600640 1 -- 172.21.15.47:0/670297180 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1080052d0 con 0x7fb1140c7e80 2024-02-19T15:26:02.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb111600640 1 -- 172.21.15.47:0/670297180 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb108005430 con 0x7fb1140c7e80 2024-02-19T15:26:02.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb111600640 1 -- 172.21.15.47:0/670297180 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb108007260 con 0x7fb1140c7e80 2024-02-19T15:26:02.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb11c600640 1 -- 172.21.15.47:0/670297180 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb114158f30 con 0x7fb1140c7e80 2024-02-19T15:26:02.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb11c600640 1 -- 172.21.15.47:0/670297180 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb11415b8c0 con 0x7fb1140c7e80 2024-02-19T15:26:02.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb11c600640 1 -- 172.21.15.47:0/670297180 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1140d1180 con 0x7fb1140c7e80 2024-02-19T15:26:02.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.324+0000 7fb111600640 1 -- 172.21.15.47:0/670297180 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb1080073c0 con 0x7fb1140c7e80 2024-02-19T15:26:02.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.328+0000 7fb111600640 1 --2- 172.21.15.47:0/670297180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb0f4066e40 0x7fb0f4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:02.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.328+0000 7fb111600640 1 -- 172.21.15.47:0/670297180 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb108084340 con 0x7fb1140c7e80 2024-02-19T15:26:02.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.328+0000 7fb119200640 1 --2- 172.21.15.47:0/670297180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb0f4066e40 0x7fb0f4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:02.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.328+0000 7fb119200640 1 --2- 172.21.15.47:0/670297180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb0f4066e40 0x7fb0f4069300 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb1000af000 tx=0x7fb1000a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:02.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.332+0000 7fb111600640 1 -- 172.21.15.47:0/670297180 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb108051120 con 0x7fb1140c7e80 2024-02-19T15:26:02.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:02.532+0000 7fb11c600640 1 -- 172.21.15.47:0/670297180 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb114075360 con 0x7fb1140c7e80 2024-02-19T15:26:03.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.148+0000 7fb111600640 1 -- 172.21.15.47:0/670297180 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v49) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb1080547e0 con 0x7fb1140c7e80 2024-02-19T15:26:03.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.152+0000 7fb11c600640 1 -- 172.21.15.47:0/670297180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb0f4066e40 msgr2=0x7fb0f4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:03.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.152+0000 7fb11c600640 1 --2- 172.21.15.47:0/670297180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb0f4066e40 0x7fb0f4069300 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb1000af000 tx=0x7fb1000a7000 comp rx=0 tx=0).stop 2024-02-19T15:26:03.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.152+0000 7fb11c600640 1 -- 172.21.15.47:0/670297180 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1140c7e80 msgr2=0x7fb11414ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:03.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.152+0000 7fb11c600640 1 --2- 172.21.15.47:0/670297180 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1140c7e80 0x7fb11414ae20 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fb10800c450 tx=0x7fb10800c910 comp rx=0 tx=0).stop 2024-02-19T15:26:03.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.152+0000 7fb11c600640 1 -- 172.21.15.47:0/670297180 shutdown_connections 2024-02-19T15:26:03.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.152+0000 7fb11c600640 1 --2- 172.21.15.47:0/670297180 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb0f4066e40 0x7fb0f4069300 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.152+0000 7fb11c600640 1 --2- 172.21.15.47:0/670297180 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1140d0820 0x7fb114159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.152+0000 7fb11c600640 1 --2- 172.21.15.47:0/670297180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1140c8260 0x7fb11415b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.152+0000 7fb11c600640 1 --2- 172.21.15.47:0/670297180 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1140c7e80 0x7fb11414ae20 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.152+0000 7fb11c600640 1 -- 172.21.15.47:0/670297180 >> 172.21.15.47:0/670297180 conn(0x7fb1140d1b30 msgr2=0x7fb1140c8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:03.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.152+0000 7fb11c600640 1 -- 172.21.15.47:0/670297180 shutdown_connections 2024-02-19T15:26:03.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.152+0000 7fb11c600640 1 -- 172.21.15.47:0/670297180 wait complete. 2024-02-19T15:26:03.175 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T15:26:03.175 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:26:03.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.312+0000 7fb5a1c00640 1 Processor -- start 2024-02-19T15:26:03.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.312+0000 7fb5a1c00640 1 -- start start 2024-02-19T15:26:03.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.316+0000 7fb5a1c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb59c0cded0 0x7fb59c0ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:03.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.316+0000 7fb5a1c00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59c0cce30 con 0x7fb59c0ceb40 2024-02-19T15:26:03.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.316+0000 7fb5a1c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59c0cd110 con 0x7fb59c06d650 2024-02-19T15:26:03.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.316+0000 7fb5a1c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb59c0cefd0 con 0x7fb59c0cded0 2024-02-19T15:26:03.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.316+0000 7fb593e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb59c0cded0 0x7fb59c0ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:03.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.316+0000 7fb593e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb59c0cded0 0x7fb59c0ce600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:46122/0 (socket says 172.21.15.47:46122) 2024-02-19T15:26:03.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.316+0000 7fb593e00640 1 -- 172.21.15.47:0/1554556113 learned_addr learned my addr 172.21.15.47:0/1554556113 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:03.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.316+0000 7fb593e00640 1 -- 172.21.15.47:0/1554556113 >> v1:172.21.15.116:6789/0 conn(0x7fb59c06d650 legacy=0x7fb59c0d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:03.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.316+0000 7fb593e00640 1 -- 172.21.15.47:0/1554556113 >> v1:172.21.15.47:6789/0 conn(0x7fb59c0ceb40 legacy=0x7fb59c06bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:03.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.316+0000 7fb593e00640 1 -- 172.21.15.47:0/1554556113 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c0cd3f0 con 0x7fb59c0cded0 2024-02-19T15:26:03.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.316+0000 7fb593e00640 1 --2- 172.21.15.47:0/1554556113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb59c0cded0 0x7fb59c0ce600 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fb584009870 tx=0x7fb58401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73af87acc32c7deb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:03.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb59a000640 1 -- 172.21.15.47:0/1554556113 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5840a6040 con 0x7fb59c0cded0 2024-02-19T15:26:03.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb59a000640 1 -- 172.21.15.47:0/1554556113 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5840aa600 con 0x7fb59c0cded0 2024-02-19T15:26:03.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb59a000640 1 -- 172.21.15.47:0/1554556113 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5840aa990 con 0x7fb59c0cded0 2024-02-19T15:26:03.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb5a1c00640 1 -- 172.21.15.47:0/1554556113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb59c0cded0 msgr2=0x7fb59c0ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:03.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb5a1c00640 1 --2- 172.21.15.47:0/1554556113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb59c0cded0 0x7fb59c0ce600 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fb584009870 tx=0x7fb58401c4a0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb5a1c00640 1 -- 172.21.15.47:0/1554556113 shutdown_connections 2024-02-19T15:26:03.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb5a1c00640 1 --2- 172.21.15.47:0/1554556113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb59c0cded0 0x7fb59c0ce600 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb5a1c00640 1 -- 172.21.15.47:0/1554556113 >> 172.21.15.47:0/1554556113 conn(0x7fb59c0d1bf0 msgr2=0x7fb59c0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:03.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb5a1c00640 1 -- 172.21.15.47:0/1554556113 shutdown_connections 2024-02-19T15:26:03.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb5a1c00640 1 -- 172.21.15.47:0/1554556113 wait complete. 2024-02-19T15:26:03.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb5a1c00640 1 Processor -- start 2024-02-19T15:26:03.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb5a1c00640 1 -- start start 2024-02-19T15:26:03.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb5a1c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb59c06d650 0x7fb59c143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:03.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb5a1c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb59c0cded0 0x7fb59c1575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:03.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb5a1c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb59c0ceb40 0x7fb59c1607e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:03.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb5a1c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c157b20 con 0x7fb59c06d650 2024-02-19T15:26:03.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb5a1c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c0ca640 con 0x7fb59c0ceb40 2024-02-19T15:26:03.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb5a1c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c0ca920 con 0x7fb59c0cded0 2024-02-19T15:26:03.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb59aa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb59c06d650 0x7fb59c143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:03.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb59aa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb59c06d650 0x7fb59c143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:46126/0 (socket says 172.21.15.47:46126) 2024-02-19T15:26:03.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb59aa00640 1 -- 172.21.15.47:0/2935196795 learned_addr learned my addr 172.21.15.47:0/2935196795 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:03.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb59b400640 1 --2- 172.21.15.47:0/2935196795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb59c0ceb40 0x7fb59c1607e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:03.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.320+0000 7fb593e00640 1 --2- 172.21.15.47:0/2935196795 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb59c0cded0 0x7fb59c1575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:03.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.324+0000 7fb59aa00640 1 -- 172.21.15.47:0/2935196795 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb59c0cded0 msgr2=0x7fb59c1575e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:03.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.324+0000 7fb59aa00640 1 --2- 172.21.15.47:0/2935196795 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb59c0cded0 0x7fb59c1575e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.324+0000 7fb59aa00640 1 -- 172.21.15.47:0/2935196795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb59c0ceb40 msgr2=0x7fb59c1607e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:03.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.324+0000 7fb59aa00640 1 --2- 172.21.15.47:0/2935196795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb59c0ceb40 0x7fb59c1607e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.324+0000 7fb59aa00640 1 -- 172.21.15.47:0/2935196795 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5840094d0 con 0x7fb59c06d650 2024-02-19T15:26:03.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.324+0000 7fb59aa00640 1 --2- 172.21.15.47:0/2935196795 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb59c06d650 0x7fb59c143bc0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fb58c00c3c0 tx=0x7fb58c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:03.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.324+0000 7fb592a00640 1 -- 172.21.15.47:0/2935196795 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58c0052d0 con 0x7fb59c06d650 2024-02-19T15:26:03.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.324+0000 7fb592a00640 1 -- 172.21.15.47:0/2935196795 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb58c005430 con 0x7fb59c06d650 2024-02-19T15:26:03.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.324+0000 7fb5a1c00640 1 -- 172.21.15.47:0/2935196795 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb59c0cac00 con 0x7fb59c06d650 2024-02-19T15:26:03.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.324+0000 7fb592a00640 1 -- 172.21.15.47:0/2935196795 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58c003840 con 0x7fb59c06d650 2024-02-19T15:26:03.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.324+0000 7fb5a1c00640 1 -- 172.21.15.47:0/2935196795 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb59c0ca2f0 con 0x7fb59c06d650 2024-02-19T15:26:03.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.324+0000 7fb5a1c00640 1 -- 172.21.15.47:0/2935196795 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb59c075360 con 0x7fb59c06d650 2024-02-19T15:26:03.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.324+0000 7fb592a00640 1 -- 172.21.15.47:0/2935196795 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb58c0039a0 con 0x7fb59c06d650 2024-02-19T15:26:03.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.328+0000 7fb592a00640 1 --2- 172.21.15.47:0/2935196795 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb578066e40 0x7fb578069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:03.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.328+0000 7fb592a00640 1 -- 172.21.15.47:0/2935196795 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb58c083fc0 con 0x7fb59c06d650 2024-02-19T15:26:03.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.328+0000 7fb593e00640 1 --2- 172.21.15.47:0/2935196795 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb578066e40 0x7fb578069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:03.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.328+0000 7fb593e00640 1 --2- 172.21.15.47:0/2935196795 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb578066e40 0x7fb578069300 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb5840ae000 tx=0x7fb5840b5040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:03.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.328+0000 7fb592a00640 1 -- 172.21.15.47:0/2935196795 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb58c050dc0 con 0x7fb59c06d650 2024-02-19T15:26:03.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.528+0000 7fb5a1c00640 1 -- 172.21.15.47:0/2935196795 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb59c0b7850 con 0x7fb59c06d650 2024-02-19T15:26:03.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.528+0000 7fb592a00640 1 -- 172.21.15.47:0/2935196795 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4943 (secure 0 0 0) 0x7fb58c054480 con 0x7fb59c06d650 2024-02-19T15:26:03.534 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:03.534 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5954,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1402677843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1402677843},{"type":"v1","addr":"172.21.15.116:6836","nonce":1402677843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":5957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1507554634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":1507554634},{"type":"v1","addr":"172.21.15.116:6837","nonce":1507554634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6771,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/888455604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":888455604},{"type":"v1","addr":"172.21.15.47:6833","nonce":888455604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:25:59.522224+0000","modified":"2024-02-19T15:26:02.535882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6774":{"gid":6774,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3598443511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3598443511},{"type":"v1","addr":"172.21.15.47:6835","nonce":3598443511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-19T15:26:03.535 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 49 2024-02-19T15:26:03.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.532+0000 7fb5a1c00640 1 -- 172.21.15.47:0/2935196795 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb578066e40 msgr2=0x7fb578069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:03.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.532+0000 7fb5a1c00640 1 --2- 172.21.15.47:0/2935196795 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb578066e40 0x7fb578069300 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb5840ae000 tx=0x7fb5840b5040 comp rx=0 tx=0).stop 2024-02-19T15:26:03.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.532+0000 7fb5a1c00640 1 -- 172.21.15.47:0/2935196795 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb59c06d650 msgr2=0x7fb59c143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:03.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.532+0000 7fb5a1c00640 1 --2- 172.21.15.47:0/2935196795 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb59c06d650 0x7fb59c143bc0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fb58c00c3c0 tx=0x7fb58c00c880 comp rx=0 tx=0).stop 2024-02-19T15:26:03.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.532+0000 7fb5a1c00640 1 -- 172.21.15.47:0/2935196795 shutdown_connections 2024-02-19T15:26:03.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.532+0000 7fb5a1c00640 1 --2- 172.21.15.47:0/2935196795 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb578066e40 0x7fb578069300 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.532+0000 7fb5a1c00640 1 --2- 172.21.15.47:0/2935196795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb59c0ceb40 0x7fb59c1607e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.532+0000 7fb5a1c00640 1 --2- 172.21.15.47:0/2935196795 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb59c0cded0 0x7fb59c1575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.532+0000 7fb5a1c00640 1 --2- 172.21.15.47:0/2935196795 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb59c06d650 0x7fb59c143bc0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.532+0000 7fb5a1c00640 1 -- 172.21.15.47:0/2935196795 >> 172.21.15.47:0/2935196795 conn(0x7fb59c0d1bf0 msgr2=0x7fb59c06bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:03.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.532+0000 7fb5a1c00640 1 -- 172.21.15.47:0/2935196795 shutdown_connections 2024-02-19T15:26:03.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.532+0000 7fb5a1c00640 1 -- 172.21.15.47:0/2935196795 wait complete. 2024-02-19T15:26:03.555 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:26:03.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.692+0000 7f64e9c00640 1 Processor -- start 2024-02-19T15:26:03.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.692+0000 7f64e9c00640 1 -- start start 2024-02-19T15:26:03.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.692+0000 7f64e9c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f64e406aeb0 0x7f64e406b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:03.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.692+0000 7f64e9c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f64e40cb520 0x7f64e40cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:03.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.692+0000 7f64e9c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e40ad840 con 0x7f64e40719e0 2024-02-19T15:26:03.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.692+0000 7f64e9c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e40c62f0 con 0x7f64e406aeb0 2024-02-19T15:26:03.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.692+0000 7f64e9c00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e406b7d0 con 0x7f64e40cb520 2024-02-19T15:26:03.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.692+0000 7f64e2a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f64e406aeb0 0x7f64e406b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:03.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.692+0000 7f64e2a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f64e406aeb0 0x7f64e406b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50566/0 (socket says 172.21.15.47:50566) 2024-02-19T15:26:03.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.692+0000 7f64e2a00640 1 -- 172.21.15.47:0/2088443604 learned_addr learned my addr 172.21.15.47:0/2088443604 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:03.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.692+0000 7f64e2000640 1 --2- 172.21.15.47:0/2088443604 >> v2:172.21.15.116:3300/0 conn(0x7f64e40cb520 0x7f64e40cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:03.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.692+0000 7f64e2a00640 1 -- 172.21.15.47:0/2088443604 >> v2:172.21.15.116:3300/0 conn(0x7f64e40cb520 msgr2=0x7f64e40cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:03.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.692+0000 7f64e2a00640 1 --2- 172.21.15.47:0/2088443604 >> v2:172.21.15.116:3300/0 conn(0x7f64e40cb520 0x7f64e40cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.692+0000 7f64e2a00640 1 -- 172.21.15.47:0/2088443604 >> v1:172.21.15.116:6789/0 conn(0x7f64e40719e0 legacy=0x7f64e4071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:03.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.692+0000 7f64e2a00640 1 -- 172.21.15.47:0/2088443604 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e40cc810 con 0x7f64e406aeb0 2024-02-19T15:26:03.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.696+0000 7f64e2a00640 1 --2- 172.21.15.47:0/2088443604 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64e406aeb0 0x7f64e406b290 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f64e40d0aa0 tx=0x7f64d401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7f8d9540538436 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:03.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.696+0000 7f64e1600640 1 -- 172.21.15.47:0/2088443604 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d40aa650 con 0x7f64e406aeb0 2024-02-19T15:26:03.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.696+0000 7f64e1600640 1 -- 172.21.15.47:0/2088443604 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d40aa7b0 con 0x7f64e406aeb0 2024-02-19T15:26:03.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.696+0000 7f64e1600640 1 -- 172.21.15.47:0/2088443604 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d40aab40 con 0x7f64e406aeb0 2024-02-19T15:26:03.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.700+0000 7f64e9c00640 1 -- 172.21.15.47:0/2088443604 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64e406aeb0 msgr2=0x7f64e406b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:03.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.700+0000 7f64e9c00640 1 --2- 172.21.15.47:0/2088443604 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64e406aeb0 0x7f64e406b290 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f64e40d0aa0 tx=0x7f64d401c4f0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.700+0000 7f64e9c00640 1 -- 172.21.15.47:0/2088443604 shutdown_connections 2024-02-19T15:26:03.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.700+0000 7f64e9c00640 1 --2- 172.21.15.47:0/2088443604 >> v2:172.21.15.116:3300/0 conn(0x7f64e40cb520 0x7f64e40cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.700+0000 7f64e9c00640 1 --2- 172.21.15.47:0/2088443604 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64e406aeb0 0x7f64e406b290 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.700+0000 7f64e9c00640 1 -- 172.21.15.47:0/2088443604 >> 172.21.15.47:0/2088443604 conn(0x7f64e40d0ad0 msgr2=0x7f64e40d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:03.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.700+0000 7f64e9c00640 1 -- 172.21.15.47:0/2088443604 shutdown_connections 2024-02-19T15:26:03.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.700+0000 7f64e9c00640 1 -- 172.21.15.47:0/2088443604 wait complete. 2024-02-19T15:26:03.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.700+0000 7f64e9c00640 1 Processor -- start 2024-02-19T15:26:03.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.700+0000 7f64e9c00640 1 -- start start 2024-02-19T15:26:03.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e9c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64e406aeb0 0x7f64e415e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:03.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e9c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f64e40719e0 0x7f64e4147510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:03.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e9c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64e40cb520 0x7f64e41597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:03.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e9c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e41614b0 con 0x7f64e40cb520 2024-02-19T15:26:03.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e9c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e415e7b0 con 0x7f64e406aeb0 2024-02-19T15:26:03.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e9c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e415e9e0 con 0x7f64e40719e0 2024-02-19T15:26:03.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64e40cb520 0x7f64e41597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:03.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64e40cb520 0x7f64e41597d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50576/0 (socket says 172.21.15.47:50576) 2024-02-19T15:26:03.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e3400640 1 -- 172.21.15.47:0/2189918834 learned_addr learned my addr 172.21.15.47:0/2189918834 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:03.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e2000640 1 --2- 172.21.15.47:0/2189918834 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f64e40719e0 0x7f64e4147510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:03.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e2a00640 1 --2- 172.21.15.47:0/2189918834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64e406aeb0 0x7f64e415e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:03.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e3400640 1 -- 172.21.15.47:0/2189918834 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f64e40719e0 msgr2=0x7f64e4147510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:03.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e3400640 1 --2- 172.21.15.47:0/2189918834 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f64e40719e0 0x7f64e4147510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e3400640 1 -- 172.21.15.47:0/2189918834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64e406aeb0 msgr2=0x7f64e415e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:03.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e3400640 1 --2- 172.21.15.47:0/2189918834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64e406aeb0 0x7f64e415e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e3400640 1 -- 172.21.15.47:0/2189918834 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d40094d0 con 0x7f64e40cb520 2024-02-19T15:26:03.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e2000640 1 --2- 172.21.15.47:0/2189918834 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f64e40719e0 0x7f64e4147510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:03.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e3400640 1 --2- 172.21.15.47:0/2189918834 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64e40cb520 0x7f64e41597d0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f64dc00c770 tx=0x7f64dc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:03.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64daa00640 1 -- 172.21.15.47:0/2189918834 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc003b80 con 0x7f64e40cb520 2024-02-19T15:26:03.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64daa00640 1 -- 172.21.15.47:0/2189918834 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64dc003ce0 con 0x7f64e40cb520 2024-02-19T15:26:03.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64daa00640 1 -- 172.21.15.47:0/2189918834 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc00aa90 con 0x7f64e40cb520 2024-02-19T15:26:03.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e9c00640 1 -- 172.21.15.47:0/2189918834 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e415ecc0 con 0x7f64e40cb520 2024-02-19T15:26:03.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e9c00640 1 -- 172.21.15.47:0/2189918834 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e4158fd0 con 0x7f64e40cb520 2024-02-19T15:26:03.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.704+0000 7f64e9c00640 1 -- 172.21.15.47:0/2189918834 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e4075360 con 0x7f64e40cb520 2024-02-19T15:26:03.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.708+0000 7f64daa00640 1 -- 172.21.15.47:0/2189918834 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f64dc00abf0 con 0x7f64e40cb520 2024-02-19T15:26:03.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.708+0000 7f64daa00640 1 --2- 172.21.15.47:0/2189918834 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f64c4066e70 0x7f64c4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:03.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.708+0000 7f64daa00640 1 -- 172.21.15.47:0/2189918834 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f64dc0847e0 con 0x7f64e40cb520 2024-02-19T15:26:03.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.708+0000 7f64e2a00640 1 --2- 172.21.15.47:0/2189918834 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f64c4066e70 0x7f64c4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:03.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.708+0000 7f64e2a00640 1 --2- 172.21.15.47:0/2189918834 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f64c4066e70 0x7f64c4069330 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f64d40ae000 tx=0x7f64d40b5040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:03.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.716+0000 7f64daa00640 1 -- 172.21.15.47:0/2189918834 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f64dc0515c0 con 0x7f64e40cb520 2024-02-19T15:26:03.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.880+0000 7f64e9c00640 1 -- 172.21.15.47:0/2189918834 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64e40b0b30 con 0x7f64e40cb520 2024-02-19T15:26:03.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.880+0000 7f64daa00640 1 -- 172.21.15.47:0/2189918834 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16672 (secure 0 0 0) 0x7f64dc054c80 con 0x7f64e40cb520 2024-02-19T15:26:03.884 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:03.884 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:26:01.137113+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:25:57.938354+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-02-19T15:25:58.514866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:26:03.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.880+0000 7f64e9c00640 1 -- 172.21.15.47:0/2189918834 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f64c4066e70 msgr2=0x7f64c4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:03.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.880+0000 7f64e9c00640 1 --2- 172.21.15.47:0/2189918834 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f64c4066e70 0x7f64c4069330 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f64d40ae000 tx=0x7f64d40b5040 comp rx=0 tx=0).stop 2024-02-19T15:26:03.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.880+0000 7f64e9c00640 1 -- 172.21.15.47:0/2189918834 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64e40cb520 msgr2=0x7f64e41597d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:03.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.880+0000 7f64e9c00640 1 --2- 172.21.15.47:0/2189918834 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64e40cb520 0x7f64e41597d0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f64dc00c770 tx=0x7f64dc00cc30 comp rx=0 tx=0).stop 2024-02-19T15:26:03.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.880+0000 7f64e9c00640 1 -- 172.21.15.47:0/2189918834 shutdown_connections 2024-02-19T15:26:03.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.880+0000 7f64e9c00640 1 --2- 172.21.15.47:0/2189918834 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f64c4066e70 0x7f64c4069330 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.880+0000 7f64e9c00640 1 --2- 172.21.15.47:0/2189918834 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64e40cb520 0x7f64e41597d0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.880+0000 7f64e9c00640 1 --2- 172.21.15.47:0/2189918834 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f64e40719e0 0x7f64e4147510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.880+0000 7f64e9c00640 1 --2- 172.21.15.47:0/2189918834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64e406aeb0 0x7f64e415e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:03.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.880+0000 7f64e9c00640 1 -- 172.21.15.47:0/2189918834 >> 172.21.15.47:0/2189918834 conn(0x7f64e40d0ad0 msgr2=0x7f64e415d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:03.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.880+0000 7f64e9c00640 1 -- 172.21.15.47:0/2189918834 shutdown_connections 2024-02-19T15:26:03.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:03.880+0000 7f64e9c00640 1 -- 172.21.15.47:0/2189918834 wait complete. 2024-02-19T15:26:03.910 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T15:26:03.910 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:26:04.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.048+0000 7f0d6b600640 1 Processor -- start 2024-02-19T15:26:04.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.048+0000 7f0d6b600640 1 -- start start 2024-02-19T15:26:04.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d6b600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d640ced10 0x7f0d641690a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:04.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d6b600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d640cce30 con 0x7f0d6406d650 2024-02-19T15:26:04.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d6b600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d640cd110 con 0x7f0d640cded0 2024-02-19T15:26:04.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d6b600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d640cd3f0 con 0x7f0d640ced10 2024-02-19T15:26:04.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d68c00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0d6406d650 0x7f0d640d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54838/0 (socket says 172.21.15.47:54838) 2024-02-19T15:26:04.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d68c00640 1 -- 172.21.15.47:0/3839268260 learned_addr learned my addr 172.21.15.47:0/3839268260 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:04.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d69600640 1 --2- 172.21.15.47:0/3839268260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d640ced10 0x7f0d641690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:04.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d63400640 1 -- 172.21.15.47:0/3839268260 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152869854 0 0) 0x7f0d640cce30 con 0x7f0d6406d650 2024-02-19T15:26:04.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d63400640 1 -- 172.21.15.47:0/3839268260 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d4c003530 con 0x7f0d6406d650 2024-02-19T15:26:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d63400640 1 -- 172.21.15.47:0/3839268260 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555223210 0 0) 0x7f0d640cd110 con 0x7f0d640cded0 2024-02-19T15:26:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d63400640 1 -- 172.21.15.47:0/3839268260 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d640cce30 con 0x7f0d640cded0 2024-02-19T15:26:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d63400640 1 -- 172.21.15.47:0/3839268260 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4257767671 0 0) 0x7f0d4c003530 con 0x7f0d6406d650 2024-02-19T15:26:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d63400640 1 -- 172.21.15.47:0/3839268260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d640ced10 msgr2=0x7f0d641690a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d63400640 1 --2- 172.21.15.47:0/3839268260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d640ced10 0x7f0d641690a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d63400640 1 -- 172.21.15.47:0/3839268260 >> v1:172.21.15.116:6789/0 conn(0x7f0d640cded0 legacy=0x7f0d640ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d63400640 1 -- 172.21.15.47:0/3839268260 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d640cd6d0 con 0x7f0d6406d650 2024-02-19T15:26:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d63400640 1 -- 172.21.15.47:0/3839268260 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f0d58001680 con 0x7f0d6406d650 2024-02-19T15:26:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d63400640 1 -- 172.21.15.47:0/3839268260 >> v1:172.21.15.47:6789/0 conn(0x7f0d6406d650 legacy=0x7f0d640d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d63400640 1 --2- 172.21.15.47:0/3839268260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d4c003b40 0x7f0d4c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d63400640 1 --2- 172.21.15.47:0/3839268260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d640ced10 0x7f0d4c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d63400640 1 --2- 172.21.15.47:0/3839268260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d4c004d10 0x7f0d4c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d63400640 1 -- 172.21.15.47:0/3839268260 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d640cd3f0 con 0x7f0d4c004d10 2024-02-19T15:26:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d63400640 1 -- 172.21.15.47:0/3839268260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c00bb50 con 0x7f0d4c003b40 2024-02-19T15:26:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d63400640 1 -- 172.21.15.47:0/3839268260 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c00bd10 con 0x7f0d640ced10 2024-02-19T15:26:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d69600640 1 --2- 172.21.15.47:0/3839268260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d4c004d10 0x7f0d4c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d68c00640 1 --2- 172.21.15.47:0/3839268260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d4c003b40 0x7f0d4c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d63e00640 1 --2- 172.21.15.47:0/3839268260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d640ced10 0x7f0d4c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d69600640 1 -- 172.21.15.47:0/3839268260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d640ced10 msgr2=0x7f0d4c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d69600640 1 --2- 172.21.15.47:0/3839268260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d640ced10 0x7f0d4c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d69600640 1 -- 172.21.15.47:0/3839268260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d4c003b40 msgr2=0x7f0d4c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d69600640 1 --2- 172.21.15.47:0/3839268260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d4c003b40 0x7f0d4c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d69600640 1 -- 172.21.15.47:0/3839268260 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c003530 con 0x7f0d4c004d10 2024-02-19T15:26:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.052+0000 7f0d63e00640 1 --2- 172.21.15.47:0/3839268260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d640ced10 0x7f0d4c0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:04.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.056+0000 7f0d69600640 1 --2- 172.21.15.47:0/3839268260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d4c004d10 0x7f0d4c005190 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c009fc0 tx=0x7f0d5c01cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390f177db7630240 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:04.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.056+0000 7f0d63400640 1 -- 172.21.15.47:0/3839268260 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c0a5c90 con 0x7f0d4c004d10 2024-02-19T15:26:04.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.056+0000 7f0d63400640 1 -- 172.21.15.47:0/3839268260 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5c00f040 con 0x7f0d4c004d10 2024-02-19T15:26:04.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.056+0000 7f0d63400640 1 -- 172.21.15.47:0/3839268260 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c0ad440 con 0x7f0d4c004d10 2024-02-19T15:26:04.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.056+0000 7f0d6b600640 1 -- 172.21.15.47:0/3839268260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d4c004d10 msgr2=0x7f0d4c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:04.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.056+0000 7f0d6b600640 1 --2- 172.21.15.47:0/3839268260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d4c004d10 0x7f0d4c005190 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c009fc0 tx=0x7f0d5c01cc80 comp rx=0 tx=0).stop 2024-02-19T15:26:04.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.056+0000 7f0d69600640 1 -- 172.21.15.47:0/3839268260 reap_dead start 2024-02-19T15:26:04.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.056+0000 7f0d6b600640 1 -- 172.21.15.47:0/3839268260 shutdown_connections 2024-02-19T15:26:04.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.056+0000 7f0d6b600640 1 -- 172.21.15.47:0/3839268260 >> 172.21.15.47:0/3839268260 conn(0x7f0d640d1bf0 msgr2=0x7f0d640d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:04.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.056+0000 7f0d6b600640 1 -- 172.21.15.47:0/3839268260 shutdown_connections 2024-02-19T15:26:04.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.056+0000 7f0d6b600640 1 -- 172.21.15.47:0/3839268260 wait complete. 2024-02-19T15:26:04.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.060+0000 7f0d6b600640 1 Processor -- start 2024-02-19T15:26:04.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.060+0000 7f0d6b600640 1 -- start start 2024-02-19T15:26:04.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.060+0000 7f0d6b600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d4c004d10 0x7f0d640c6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:04.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.060+0000 7f0d6b600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d640c5650 0x7f0d641579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:04.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.060+0000 7f0d6b600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d6415cbd0 0x7f0d640c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:04.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.060+0000 7f0d6b600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d640c8cc0 con 0x7f0d640c5650 2024-02-19T15:26:04.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.060+0000 7f0d6b600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d640c7a00 con 0x7f0d4c004d10 2024-02-19T15:26:04.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.060+0000 7f0d6b600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d640c6a80 con 0x7f0d6415cbd0 2024-02-19T15:26:04.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.060+0000 7f0d63e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d640c5650 0x7f0d641579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:04.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.060+0000 7f0d63e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d640c5650 0x7f0d641579d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50594/0 (socket says 172.21.15.47:50594) 2024-02-19T15:26:04.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.060+0000 7f0d63e00640 1 -- 172.21.15.47:0/2221381522 learned_addr learned my addr 172.21.15.47:0/2221381522 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:04.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.060+0000 7f0d68c00640 1 --2- 172.21.15.47:0/2221381522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d4c004d10 0x7f0d640c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:04.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.060+0000 7f0d69600640 1 --2- 172.21.15.47:0/2221381522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d6415cbd0 0x7f0d640c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:04.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.060+0000 7f0d63e00640 1 -- 172.21.15.47:0/2221381522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d6415cbd0 msgr2=0x7f0d640c7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:04.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.060+0000 7f0d63e00640 1 --2- 172.21.15.47:0/2221381522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d6415cbd0 0x7f0d640c7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.060+0000 7f0d63e00640 1 -- 172.21.15.47:0/2221381522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d4c004d10 msgr2=0x7f0d640c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:04.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.060+0000 7f0d63e00640 1 --2- 172.21.15.47:0/2221381522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d4c004d10 0x7f0d640c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.060+0000 7f0d63e00640 1 -- 172.21.15.47:0/2221381522 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d640c8ea0 con 0x7f0d640c5650 2024-02-19T15:26:04.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.064+0000 7f0d63e00640 1 --2- 172.21.15.47:0/2221381522 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d640c5650 0x7f0d641579d0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f0d50011e60 tx=0x7f0d5000f540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:04.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.064+0000 7f0d60c00640 1 -- 172.21.15.47:0/2221381522 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d50015610 con 0x7f0d640c5650 2024-02-19T15:26:04.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.064+0000 7f0d60c00640 1 -- 172.21.15.47:0/2221381522 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d50015770 con 0x7f0d640c5650 2024-02-19T15:26:04.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.064+0000 7f0d60c00640 1 -- 172.21.15.47:0/2221381522 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d50015b00 con 0x7f0d640c5650 2024-02-19T15:26:04.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.064+0000 7f0d6b600640 1 -- 172.21.15.47:0/2221381522 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d640c9180 con 0x7f0d640c5650 2024-02-19T15:26:04.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.064+0000 7f0d6b600640 1 -- 172.21.15.47:0/2221381522 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d640c9740 con 0x7f0d640c5650 2024-02-19T15:26:04.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.064+0000 7f0d6b600640 1 -- 172.21.15.47:0/2221381522 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d640ccaa0 con 0x7f0d640c5650 2024-02-19T15:26:04.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.064+0000 7f0d60c00640 1 -- 172.21.15.47:0/2221381522 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0d50015c60 con 0x7f0d640c5650 2024-02-19T15:26:04.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.064+0000 7f0d60c00640 1 --2- 172.21.15.47:0/2221381522 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d44066e70 0x7f0d44069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:04.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.064+0000 7f0d60c00640 1 -- 172.21.15.47:0/2221381522 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0d500882a0 con 0x7f0d640c5650 2024-02-19T15:26:04.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.064+0000 7f0d68c00640 1 --2- 172.21.15.47:0/2221381522 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d44066e70 0x7f0d44069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:04.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.068+0000 7f0d68c00640 1 --2- 172.21.15.47:0/2221381522 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d44066e70 0x7f0d44069330 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f0d58001130 tx=0x7f0d58000ee0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:04.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.068+0000 7f0d60c00640 1 -- 172.21.15.47:0/2221381522 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0d50054ff0 con 0x7f0d640c5650 2024-02-19T15:26:04.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.232+0000 7f0d6b600640 1 -- 172.21.15.47:0/2221381522 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0d640cfbd0 con 0x7f0d44066e70 2024-02-19T15:26:04.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.240+0000 7f0d60c00640 1 -- 172.21.15.47:0/2221381522 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289181 (secure 0 0 0) 0x7f0d580076b0 con 0x7f0d44066e70 2024-02-19T15:26:04.247 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:04.248 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:26:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.244+0000 7f0d6b600640 1 -- 172.21.15.47:0/2221381522 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d44066e70 msgr2=0x7f0d44069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.244+0000 7f0d6b600640 1 --2- 172.21.15.47:0/2221381522 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d44066e70 0x7f0d44069330 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f0d58001130 tx=0x7f0d58000ee0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.244+0000 7f0d6b600640 1 -- 172.21.15.47:0/2221381522 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d640c5650 msgr2=0x7f0d641579d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.244+0000 7f0d6b600640 1 --2- 172.21.15.47:0/2221381522 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d640c5650 0x7f0d641579d0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f0d50011e60 tx=0x7f0d5000f540 comp rx=0 tx=0).stop 2024-02-19T15:26:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.244+0000 7f0d6b600640 1 -- 172.21.15.47:0/2221381522 shutdown_connections 2024-02-19T15:26:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.244+0000 7f0d6b600640 1 --2- 172.21.15.47:0/2221381522 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d44066e70 0x7f0d44069330 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.244+0000 7f0d6b600640 1 --2- 172.21.15.47:0/2221381522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d6415cbd0 0x7f0d640c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.244+0000 7f0d6b600640 1 --2- 172.21.15.47:0/2221381522 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d640c5650 0x7f0d641579d0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.244+0000 7f0d6b600640 1 --2- 172.21.15.47:0/2221381522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d4c004d10 0x7f0d640c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.244+0000 7f0d6b600640 1 -- 172.21.15.47:0/2221381522 >> 172.21.15.47:0/2221381522 conn(0x7f0d640d1bf0 msgr2=0x7f0d64161090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.244+0000 7f0d6b600640 1 -- 172.21.15.47:0/2221381522 shutdown_connections 2024-02-19T15:26:04.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.244+0000 7f0d6b600640 1 -- 172.21.15.47:0/2221381522 wait complete. 2024-02-19T15:26:04.269 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-02-19T15:26:03.204094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":131,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262620,"kb_used_data":8992,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754712100,"statfs":{"total":773094113280,"available":772825190400,"internally_reserved":0,"allocated":9207808,"data_stored":4550148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":259575346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.083940"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147297+0000","last_change":"2024-02-19T15:26:00.141172+0000","last_active":"2024-02-19T15:26:01.147297+0000","last_peered":"2024-02-19T15:26:01.147297+0000","last_clean":"2024-02-19T15:26:01.147297+0000","last_became_active":"2024-02-19T15:26:00.139831+0000","last_became_peered":"2024-02-19T15:26:00.139831+0000","last_unstale":"2024-02-19T15:26:01.147297+0000","last_undegraded":"2024-02-19T15:26:01.147297+0000","last_fullsized":"2024-02-19T15:26:01.147297+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:50:34.796433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147043+0000","last_change":"2024-02-19T15:26:00.149926+0000","last_active":"2024-02-19T15:26:01.147043+0000","last_peered":"2024-02-19T15:26:01.147043+0000","last_clean":"2024-02-19T15:26:01.147043+0000","last_became_active":"2024-02-19T15:26:00.149550+0000","last_became_peered":"2024-02-19T15:26:00.149550+0000","last_unstale":"2024-02-19T15:26:01.147043+0000","last_undegraded":"2024-02-19T15:26:01.147043+0000","last_fullsized":"2024-02-19T15:26:01.147043+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:44:01.635671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.149528+0000","last_change":"2024-02-19T15:26:00.144802+0000","last_active":"2024-02-19T15:26:00.149528+0000","last_peered":"2024-02-19T15:26:00.149528+0000","last_clean":"2024-02-19T15:26:00.149528+0000","last_became_active":"2024-02-19T15:26:00.144432+0000","last_became_peered":"2024-02-19T15:26:00.144432+0000","last_unstale":"2024-02-19T15:26:00.149528+0000","last_undegraded":"2024-02-19T15:26:00.149528+0000","last_fullsized":"2024-02-19T15:26:00.149528+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:22:04.556374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.161013+0000","last_change":"2024-02-19T15:26:00.159660+0000","last_active":"2024-02-19T15:26:00.161013+0000","last_peered":"2024-02-19T15:26:00.161013+0000","last_clean":"2024-02-19T15:26:00.161013+0000","last_became_active":"2024-02-19T15:26:00.159391+0000","last_became_peered":"2024-02-19T15:26:00.159391+0000","last_unstale":"2024-02-19T15:26:00.161013+0000","last_undegraded":"2024-02-19T15:26:00.161013+0000","last_fullsized":"2024-02-19T15:26:00.161013+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:21:27.210479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.159223+0000","last_change":"2024-02-19T15:26:00.158174+0000","last_active":"2024-02-19T15:26:00.159223+0000","last_peered":"2024-02-19T15:26:00.159223+0000","last_clean":"2024-02-19T15:26:00.159223+0000","last_became_active":"2024-02-19T15:26:00.157878+0000","last_became_peered":"2024-02-19T15:26:00.157878+0000","last_unstale":"2024-02-19T15:26:00.159223+0000","last_undegraded":"2024-02-19T15:26:00.159223+0000","last_fullsized":"2024-02-19T15:26:00.159223+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:11:57.737618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.145831+0000","last_change":"2024-02-19T15:26:00.145481+0000","last_active":"2024-02-19T15:26:01.145831+0000","last_peered":"2024-02-19T15:26:01.145831+0000","last_clean":"2024-02-19T15:26:01.145831+0000","last_became_active":"2024-02-19T15:26:00.145091+0000","last_became_peered":"2024-02-19T15:26:00.145091+0000","last_unstale":"2024-02-19T15:26:01.145831+0000","last_undegraded":"2024-02-19T15:26:01.145831+0000","last_fullsized":"2024-02-19T15:26:01.145831+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:56:56.303004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.146690+0000","last_change":"2024-02-19T15:26:00.142559+0000","last_active":"2024-02-19T15:26:00.146690+0000","last_peered":"2024-02-19T15:26:00.146690+0000","last_clean":"2024-02-19T15:26:00.146690+0000","last_became_active":"2024-02-19T15:26:00.142213+0000","last_became_peered":"2024-02-19T15:26:00.142213+0000","last_unstale":"2024-02-19T15:26:00.146690+0000","last_undegraded":"2024-02-19T15:26:00.146690+0000","last_fullsized":"2024-02-19T15:26:00.146690+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:59:54.877515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.145800+0000","last_change":"2024-02-19T15:26:00.165638+0000","last_active":"2024-02-19T15:26:01.145800+0000","last_peered":"2024-02-19T15:26:01.145800+0000","last_clean":"2024-02-19T15:26:01.145800+0000","last_became_active":"2024-02-19T15:26:00.165342+0000","last_became_peered":"2024-02-19T15:26:00.165342+0000","last_unstale":"2024-02-19T15:26:01.145800+0000","last_undegraded":"2024-02-19T15:26:01.145800+0000","last_fullsized":"2024-02-19T15:26:01.145800+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:17:42.181002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146321+0000","last_change":"2024-02-19T15:26:00.156900+0000","last_active":"2024-02-19T15:26:01.146321+0000","last_peered":"2024-02-19T15:26:01.146321+0000","last_clean":"2024-02-19T15:26:01.146321+0000","last_became_active":"2024-02-19T15:26:00.156492+0000","last_became_peered":"2024-02-19T15:26:00.156492+0000","last_unstale":"2024-02-19T15:26:01.146321+0000","last_undegraded":"2024-02-19T15:26:01.146321+0000","last_fullsized":"2024-02-19T15:26:01.146321+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:49:38.953872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147763+0000","last_change":"2024-02-19T15:26:00.153441+0000","last_active":"2024-02-19T15:26:01.147763+0000","last_peered":"2024-02-19T15:26:01.147763+0000","last_clean":"2024-02-19T15:26:01.147763+0000","last_became_active":"2024-02-19T15:26:00.153179+0000","last_became_peered":"2024-02-19T15:26:00.153179+0000","last_unstale":"2024-02-19T15:26:01.147763+0000","last_undegraded":"2024-02-19T15:26:01.147763+0000","last_fullsized":"2024-02-19T15:26:01.147763+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:38:59.987721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146927+0000","last_change":"2024-02-19T15:26:00.143083+0000","last_active":"2024-02-19T15:26:01.146927+0000","last_peered":"2024-02-19T15:26:01.146927+0000","last_clean":"2024-02-19T15:26:01.146927+0000","last_became_active":"2024-02-19T15:26:00.142790+0000","last_became_peered":"2024-02-19T15:26:00.142790+0000","last_unstale":"2024-02-19T15:26:01.146927+0000","last_undegraded":"2024-02-19T15:26:01.146927+0000","last_fullsized":"2024-02-19T15:26:01.146927+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:11:57.737618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147111+0000","last_change":"2024-02-19T15:26:00.154101+0000","last_active":"2024-02-19T15:26:01.147111+0000","last_peered":"2024-02-19T15:26:01.147111+0000","last_clean":"2024-02-19T15:26:01.147111+0000","last_became_active":"2024-02-19T15:26:00.153705+0000","last_became_peered":"2024-02-19T15:26:00.153705+0000","last_unstale":"2024-02-19T15:26:01.147111+0000","last_undegraded":"2024-02-19T15:26:01.147111+0000","last_fullsized":"2024-02-19T15:26:01.147111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:04:14.206461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.151614+0000","last_change":"2024-02-19T15:26:00.147347+0000","last_active":"2024-02-19T15:26:00.151614+0000","last_peered":"2024-02-19T15:26:00.151614+0000","last_clean":"2024-02-19T15:26:00.151614+0000","last_became_active":"2024-02-19T15:26:00.147068+0000","last_became_peered":"2024-02-19T15:26:00.147068+0000","last_unstale":"2024-02-19T15:26:00.151614+0000","last_undegraded":"2024-02-19T15:26:00.151614+0000","last_fullsized":"2024-02-19T15:26:00.151614+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:09:50.208629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.163132+0000","last_change":"2024-02-19T15:26:00.161568+0000","last_active":"2024-02-19T15:26:00.163132+0000","last_peered":"2024-02-19T15:26:00.163132+0000","last_clean":"2024-02-19T15:26:00.163132+0000","last_became_active":"2024-02-19T15:26:00.161315+0000","last_became_peered":"2024-02-19T15:26:00.161315+0000","last_unstale":"2024-02-19T15:26:00.163132+0000","last_undegraded":"2024-02-19T15:26:00.163132+0000","last_fullsized":"2024-02-19T15:26:00.163132+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:31:27.968562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.150229+0000","last_change":"2024-02-19T15:26:00.146394+0000","last_active":"2024-02-19T15:26:00.150229+0000","last_peered":"2024-02-19T15:26:00.150229+0000","last_clean":"2024-02-19T15:26:00.150229+0000","last_became_active":"2024-02-19T15:26:00.146064+0000","last_became_peered":"2024-02-19T15:26:00.146064+0000","last_unstale":"2024-02-19T15:26:00.150229+0000","last_undegraded":"2024-02-19T15:26:00.150229+0000","last_fullsized":"2024-02-19T15:26:00.150229+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:31:52.766037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.150344+0000","last_change":"2024-02-19T15:26:00.140780+0000","last_active":"2024-02-19T15:26:00.150344+0000","last_peered":"2024-02-19T15:26:00.150344+0000","last_clean":"2024-02-19T15:26:00.150344+0000","last_became_active":"2024-02-19T15:26:00.140313+0000","last_became_peered":"2024-02-19T15:26:00.140313+0000","last_unstale":"2024-02-19T15:26:00.150344+0000","last_undegraded":"2024-02-19T15:26:00.150344+0000","last_fullsized":"2024-02-19T15:26:00.150344+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:29:22.170656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.149621+0000","last_change":"2024-02-19T15:26:00.146625+0000","last_active":"2024-02-19T15:26:00.149621+0000","last_peered":"2024-02-19T15:26:00.149621+0000","last_clean":"2024-02-19T15:26:00.149621+0000","last_became_active":"2024-02-19T15:26:00.146280+0000","last_became_peered":"2024-02-19T15:26:00.146280+0000","last_unstale":"2024-02-19T15:26:00.149621+0000","last_undegraded":"2024-02-19T15:26:00.149621+0000","last_fullsized":"2024-02-19T15:26:00.149621+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:25:06.707641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.148998+0000","last_change":"2024-02-19T15:26:00.145504+0000","last_active":"2024-02-19T15:26:00.148998+0000","last_peered":"2024-02-19T15:26:00.148998+0000","last_clean":"2024-02-19T15:26:00.148998+0000","last_became_active":"2024-02-19T15:26:00.145240+0000","last_became_peered":"2024-02-19T15:26:00.145240+0000","last_unstale":"2024-02-19T15:26:00.148998+0000","last_undegraded":"2024-02-19T15:26:00.148998+0000","last_fullsized":"2024-02-19T15:26:00.148998+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:39:15.242573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.164463+0000","last_change":"2024-02-19T15:26:00.151957+0000","last_active":"2024-02-19T15:26:00.164463+0000","last_peered":"2024-02-19T15:26:00.164463+0000","last_clean":"2024-02-19T15:26:00.164463+0000","last_became_active":"2024-02-19T15:26:00.151700+0000","last_became_peered":"2024-02-19T15:26:00.151700+0000","last_unstale":"2024-02-19T15:26:00.164463+0000","last_undegraded":"2024-02-19T15:26:00.164463+0000","last_fullsized":"2024-02-19T15:26:00.164463+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:50:40.234904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.164718+0000","last_change":"2024-02-19T15:26:00.163579+0000","last_active":"2024-02-19T15:26:00.164718+0000","last_peered":"2024-02-19T15:26:00.164718+0000","last_clean":"2024-02-19T15:26:00.164718+0000","last_became_active":"2024-02-19T15:26:00.163018+0000","last_became_peered":"2024-02-19T15:26:00.163018+0000","last_unstale":"2024-02-19T15:26:00.164718+0000","last_undegraded":"2024-02-19T15:26:00.164718+0000","last_fullsized":"2024-02-19T15:26:00.164718+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:57:16.434575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.159440+0000","last_change":"2024-02-19T15:26:00.154028+0000","last_active":"2024-02-19T15:26:00.159440+0000","last_peered":"2024-02-19T15:26:00.159440+0000","last_clean":"2024-02-19T15:26:00.159440+0000","last_became_active":"2024-02-19T15:26:00.151254+0000","last_became_peered":"2024-02-19T15:26:00.151254+0000","last_unstale":"2024-02-19T15:26:00.159440+0000","last_undegraded":"2024-02-19T15:26:00.159440+0000","last_fullsized":"2024-02-19T15:26:00.159440+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:32:09.736412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146476+0000","last_change":"2024-02-19T15:26:00.167151+0000","last_active":"2024-02-19T15:26:01.146476+0000","last_peered":"2024-02-19T15:26:01.146476+0000","last_clean":"2024-02-19T15:26:01.146476+0000","last_became_active":"2024-02-19T15:26:00.166846+0000","last_became_peered":"2024-02-19T15:26:00.166846+0000","last_unstale":"2024-02-19T15:26:01.146476+0000","last_undegraded":"2024-02-19T15:26:01.146476+0000","last_fullsized":"2024-02-19T15:26:01.146476+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:50.581881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147359+0000","last_change":"2024-02-19T15:26:00.158679+0000","last_active":"2024-02-19T15:26:01.147359+0000","last_peered":"2024-02-19T15:26:01.147359+0000","last_clean":"2024-02-19T15:26:01.147359+0000","last_became_active":"2024-02-19T15:26:00.158412+0000","last_became_peered":"2024-02-19T15:26:00.158412+0000","last_unstale":"2024-02-19T15:26:01.147359+0000","last_undegraded":"2024-02-19T15:26:01.147359+0000","last_fullsized":"2024-02-19T15:26:01.147359+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:48:58.609617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.161058+0000","last_change":"2024-02-19T15:26:00.160327+0000","last_active":"2024-02-19T15:26:00.161058+0000","last_peered":"2024-02-19T15:26:00.161058+0000","last_clean":"2024-02-19T15:26:00.161058+0000","last_became_active":"2024-02-19T15:26:00.160047+0000","last_became_peered":"2024-02-19T15:26:00.160047+0000","last_unstale":"2024-02-19T15:26:00.161058+0000","last_undegraded":"2024-02-19T15:26:00.161058+0000","last_fullsized":"2024-02-19T15:26:00.161058+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:12:50.146790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146969+0000","last_change":"2024-02-19T15:26:00.156835+0000","last_active":"2024-02-19T15:26:01.146969+0000","last_peered":"2024-02-19T15:26:01.146969+0000","last_clean":"2024-02-19T15:26:01.146969+0000","last_became_active":"2024-02-19T15:26:00.156598+0000","last_became_peered":"2024-02-19T15:26:00.156598+0000","last_unstale":"2024-02-19T15:26:01.146969+0000","last_undegraded":"2024-02-19T15:26:01.146969+0000","last_fullsized":"2024-02-19T15:26:01.146969+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:14:01.639589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.162544+0000","last_change":"2024-02-19T15:26:00.161536+0000","last_active":"2024-02-19T15:26:00.162544+0000","last_peered":"2024-02-19T15:26:00.162544+0000","last_clean":"2024-02-19T15:26:00.162544+0000","last_became_active":"2024-02-19T15:26:00.161224+0000","last_became_peered":"2024-02-19T15:26:00.161224+0000","last_unstale":"2024-02-19T15:26:00.162544+0000","last_undegraded":"2024-02-19T15:26:00.162544+0000","last_fullsized":"2024-02-19T15:26:00.162544+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:31:52.766037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.151205+0000","last_change":"2024-02-19T15:26:00.150179+0000","last_active":"2024-02-19T15:26:00.151205+0000","last_peered":"2024-02-19T15:26:00.151205+0000","last_clean":"2024-02-19T15:26:00.151205+0000","last_became_active":"2024-02-19T15:26:00.149900+0000","last_became_peered":"2024-02-19T15:26:00.149900+0000","last_unstale":"2024-02-19T15:26:00.151205+0000","last_undegraded":"2024-02-19T15:26:00.151205+0000","last_fullsized":"2024-02-19T15:26:00.151205+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:08:32.787841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147269+0000","last_change":"2024-02-19T15:26:00.138814+0000","last_active":"2024-02-19T15:26:01.147269+0000","last_peered":"2024-02-19T15:26:01.147269+0000","last_clean":"2024-02-19T15:26:01.147269+0000","last_became_active":"2024-02-19T15:26:00.138494+0000","last_became_peered":"2024-02-19T15:26:00.138494+0000","last_unstale":"2024-02-19T15:26:01.147269+0000","last_undegraded":"2024-02-19T15:26:01.147269+0000","last_fullsized":"2024-02-19T15:26:01.147269+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:11.778091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146505+0000","last_change":"2024-02-19T15:26:00.157251+0000","last_active":"2024-02-19T15:26:01.146505+0000","last_peered":"2024-02-19T15:26:01.146505+0000","last_clean":"2024-02-19T15:26:01.146505+0000","last_became_active":"2024-02-19T15:26:00.156992+0000","last_became_peered":"2024-02-19T15:26:00.156992+0000","last_unstale":"2024-02-19T15:26:01.146505+0000","last_undegraded":"2024-02-19T15:26:01.146505+0000","last_fullsized":"2024-02-19T15:26:01.146505+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:02:22.168242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146533+0000","last_change":"2024-02-19T15:26:00.147939+0000","last_active":"2024-02-19T15:26:01.146533+0000","last_peered":"2024-02-19T15:26:01.146533+0000","last_clean":"2024-02-19T15:26:01.146533+0000","last_became_active":"2024-02-19T15:26:00.147682+0000","last_became_peered":"2024-02-19T15:26:00.147682+0000","last_unstale":"2024-02-19T15:26:01.146533+0000","last_undegraded":"2024-02-19T15:26:01.146533+0000","last_fullsized":"2024-02-19T15:26:01.146533+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:44.769088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.158014+0000","last_change":"2024-02-19T15:26:00.155419+0000","last_active":"2024-02-19T15:26:00.158014+0000","last_peered":"2024-02-19T15:26:00.158014+0000","last_clean":"2024-02-19T15:26:00.158014+0000","last_became_active":"2024-02-19T15:26:00.155080+0000","last_became_peered":"2024-02-19T15:26:00.155080+0000","last_unstale":"2024-02-19T15:26:00.158014+0000","last_undegraded":"2024-02-19T15:26:00.158014+0000","last_fullsized":"2024-02-19T15:26:00.158014+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:09:12.862734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146893+0000","last_change":"2024-02-19T15:26:00.158776+0000","last_active":"2024-02-19T15:26:01.146893+0000","last_peered":"2024-02-19T15:26:01.146893+0000","last_clean":"2024-02-19T15:26:01.146893+0000","last_became_active":"2024-02-19T15:26:00.157244+0000","last_became_peered":"2024-02-19T15:26:00.157244+0000","last_unstale":"2024-02-19T15:26:01.146893+0000","last_undegraded":"2024-02-19T15:26:01.146893+0000","last_fullsized":"2024-02-19T15:26:01.146893+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:49:11.620738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.149002+0000","last_change":"2024-02-19T15:26:00.143469+0000","last_active":"2024-02-19T15:26:00.149002+0000","last_peered":"2024-02-19T15:26:00.149002+0000","last_clean":"2024-02-19T15:26:00.149002+0000","last_became_active":"2024-02-19T15:26:00.143199+0000","last_became_peered":"2024-02-19T15:26:00.143199+0000","last_unstale":"2024-02-19T15:26:00.149002+0000","last_undegraded":"2024-02-19T15:26:00.149002+0000","last_fullsized":"2024-02-19T15:26:00.149002+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:02:22.168242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"81'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147478+0000","last_change":"2024-02-19T15:25:59.138426+0000","last_active":"2024-02-19T15:26:01.147478+0000","last_peered":"2024-02-19T15:26:01.147478+0000","last_clean":"2024-02-19T15:26:01.147478+0000","last_became_active":"2024-02-19T15:25:59.138137+0000","last_became_peered":"2024-02-19T15:25:59.138137+0000","last_unstale":"2024-02-19T15:26:01.147478+0000","last_undegraded":"2024-02-19T15:26:01.147478+0000","last_fullsized":"2024-02-19T15:26:01.147478+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:00:31.176094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147379+0000","last_change":"2024-02-19T15:26:00.158856+0000","last_active":"2024-02-19T15:26:01.147379+0000","last_peered":"2024-02-19T15:26:01.147379+0000","last_clean":"2024-02-19T15:26:01.147379+0000","last_became_active":"2024-02-19T15:26:00.158374+0000","last_became_peered":"2024-02-19T15:26:00.158374+0000","last_unstale":"2024-02-19T15:26:01.147379+0000","last_undegraded":"2024-02-19T15:26:01.147379+0000","last_fullsized":"2024-02-19T15:26:01.147379+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:50:40.234904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.160658+0000","last_change":"2024-02-19T15:25:59.140196+0000","last_active":"2024-02-19T15:26:00.160658+0000","last_peered":"2024-02-19T15:26:00.160658+0000","last_clean":"2024-02-19T15:26:00.160658+0000","last_became_active":"2024-02-19T15:25:59.139892+0000","last_became_peered":"2024-02-19T15:25:59.139892+0000","last_unstale":"2024-02-19T15:26:00.160658+0000","last_undegraded":"2024-02-19T15:26:00.160658+0000","last_fullsized":"2024-02-19T15:26:00.160658+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:37:26.597887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147289+0000","last_change":"2024-02-19T15:26:00.161341+0000","last_active":"2024-02-19T15:26:01.147289+0000","last_peered":"2024-02-19T15:26:01.147289+0000","last_clean":"2024-02-19T15:26:01.147289+0000","last_became_active":"2024-02-19T15:26:00.160999+0000","last_became_peered":"2024-02-19T15:26:00.160999+0000","last_unstale":"2024-02-19T15:26:01.147289+0000","last_undegraded":"2024-02-19T15:26:01.147289+0000","last_fullsized":"2024-02-19T15:26:01.147289+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:28:17.733970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"81'2","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.165835+0000","last_change":"2024-02-19T15:25:59.150116+0000","last_active":"2024-02-19T15:26:00.165835+0000","last_peered":"2024-02-19T15:26:00.165835+0000","last_clean":"2024-02-19T15:26:00.165835+0000","last_became_active":"2024-02-19T15:25:59.149815+0000","last_became_peered":"2024-02-19T15:25:59.149815+0000","last_unstale":"2024-02-19T15:26:00.165835+0000","last_undegraded":"2024-02-19T15:26:00.165835+0000","last_fullsized":"2024-02-19T15:26:00.165835+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:04:13.199124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.149506+0000","last_change":"2024-02-19T15:26:00.144350+0000","last_active":"2024-02-19T15:26:00.149506+0000","last_peered":"2024-02-19T15:26:00.149506+0000","last_clean":"2024-02-19T15:26:00.149506+0000","last_became_active":"2024-02-19T15:26:00.142102+0000","last_became_peered":"2024-02-19T15:26:00.142102+0000","last_unstale":"2024-02-19T15:26:00.149506+0000","last_undegraded":"2024-02-19T15:26:00.149506+0000","last_fullsized":"2024-02-19T15:26:00.149506+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:12:56.043581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"81'2","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.165584+0000","last_change":"2024-02-19T15:25:59.150705+0000","last_active":"2024-02-19T15:26:00.165584+0000","last_peered":"2024-02-19T15:26:00.165584+0000","last_clean":"2024-02-19T15:26:00.165584+0000","last_became_active":"2024-02-19T15:25:59.150301+0000","last_became_peered":"2024-02-19T15:25:59.150301+0000","last_unstale":"2024-02-19T15:26:00.165584+0000","last_undegraded":"2024-02-19T15:26:00.165584+0000","last_fullsized":"2024-02-19T15:26:00.165584+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:26:00.767871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146631+0000","last_change":"2024-02-19T15:26:00.147938+0000","last_active":"2024-02-19T15:26:01.146631+0000","last_peered":"2024-02-19T15:26:01.146631+0000","last_clean":"2024-02-19T15:26:01.146631+0000","last_became_active":"2024-02-19T15:26:00.147671+0000","last_became_peered":"2024-02-19T15:26:00.147671+0000","last_unstale":"2024-02-19T15:26:01.146631+0000","last_undegraded":"2024-02-19T15:26:01.146631+0000","last_fullsized":"2024-02-19T15:26:01.146631+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:08:32.787841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.165175+0000","last_change":"2024-02-19T15:25:59.139252+0000","last_active":"2024-02-19T15:26:00.165175+0000","last_peered":"2024-02-19T15:26:00.165175+0000","last_clean":"2024-02-19T15:26:00.165175+0000","last_became_active":"2024-02-19T15:25:59.138894+0000","last_became_peered":"2024-02-19T15:25:59.138894+0000","last_unstale":"2024-02-19T15:26:00.165175+0000","last_undegraded":"2024-02-19T15:26:00.165175+0000","last_fullsized":"2024-02-19T15:26:00.165175+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:04:47.587059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.147800+0000","last_change":"2024-02-19T15:26:00.142832+0000","last_active":"2024-02-19T15:26:00.147800+0000","last_peered":"2024-02-19T15:26:00.147800+0000","last_clean":"2024-02-19T15:26:00.147800+0000","last_became_active":"2024-02-19T15:26:00.141435+0000","last_became_peered":"2024-02-19T15:26:00.141435+0000","last_unstale":"2024-02-19T15:26:00.147800+0000","last_undegraded":"2024-02-19T15:26:00.147800+0000","last_fullsized":"2024-02-19T15:26:00.147800+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:50.581881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.160629+0000","last_change":"2024-02-19T15:25:59.141148+0000","last_active":"2024-02-19T15:26:00.160629+0000","last_peered":"2024-02-19T15:26:00.160629+0000","last_clean":"2024-02-19T15:26:00.160629+0000","last_became_active":"2024-02-19T15:25:59.140799+0000","last_became_peered":"2024-02-19T15:25:59.140799+0000","last_unstale":"2024-02-19T15:26:00.160629+0000","last_undegraded":"2024-02-19T15:26:00.160629+0000","last_fullsized":"2024-02-19T15:26:00.160629+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:03:25.527833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.164084+0000","last_change":"2024-02-19T15:26:00.163537+0000","last_active":"2024-02-19T15:26:00.164084+0000","last_peered":"2024-02-19T15:26:00.164084+0000","last_clean":"2024-02-19T15:26:00.164084+0000","last_became_active":"2024-02-19T15:26:00.163198+0000","last_became_peered":"2024-02-19T15:26:00.163198+0000","last_unstale":"2024-02-19T15:26:00.164084+0000","last_undegraded":"2024-02-19T15:26:00.164084+0000","last_fullsized":"2024-02-19T15:26:00.164084+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:05.118962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.134809+0000","last_change":"2024-02-19T15:25:59.145705+0000","last_active":"2024-02-19T15:26:00.134809+0000","last_peered":"2024-02-19T15:26:00.134809+0000","last_clean":"2024-02-19T15:26:00.134809+0000","last_became_active":"2024-02-19T15:25:59.145398+0000","last_became_peered":"2024-02-19T15:25:59.145398+0000","last_unstale":"2024-02-19T15:26:00.134809+0000","last_undegraded":"2024-02-19T15:26:00.134809+0000","last_fullsized":"2024-02-19T15:26:00.134809+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:46:04.901935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.150207+0000","last_change":"2024-02-19T15:26:00.146267+0000","last_active":"2024-02-19T15:26:00.150207+0000","last_peered":"2024-02-19T15:26:00.150207+0000","last_clean":"2024-02-19T15:26:00.150207+0000","last_became_active":"2024-02-19T15:26:00.146042+0000","last_became_peered":"2024-02-19T15:26:00.146042+0000","last_unstale":"2024-02-19T15:26:00.150207+0000","last_undegraded":"2024-02-19T15:26:00.150207+0000","last_fullsized":"2024-02-19T15:26:00.150207+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:06.498178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146639+0000","last_change":"2024-02-19T15:25:59.147976+0000","last_active":"2024-02-19T15:26:01.146639+0000","last_peered":"2024-02-19T15:26:01.146639+0000","last_clean":"2024-02-19T15:26:01.146639+0000","last_became_active":"2024-02-19T15:25:59.145034+0000","last_became_peered":"2024-02-19T15:25:59.145034+0000","last_unstale":"2024-02-19T15:26:01.146639+0000","last_undegraded":"2024-02-19T15:26:01.146639+0000","last_fullsized":"2024-02-19T15:26:01.146639+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:13:37.187177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.163984+0000","last_change":"2024-02-19T15:26:00.161844+0000","last_active":"2024-02-19T15:26:00.163984+0000","last_peered":"2024-02-19T15:26:00.163984+0000","last_clean":"2024-02-19T15:26:00.163984+0000","last_became_active":"2024-02-19T15:26:00.161581+0000","last_became_peered":"2024-02-19T15:26:00.161581+0000","last_unstale":"2024-02-19T15:26:00.163984+0000","last_undegraded":"2024-02-19T15:26:00.163984+0000","last_fullsized":"2024-02-19T15:26:00.163984+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:53:03.954569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147553+0000","last_change":"2024-02-19T15:25:59.144876+0000","last_active":"2024-02-19T15:26:01.147553+0000","last_peered":"2024-02-19T15:26:01.147553+0000","last_clean":"2024-02-19T15:26:01.147553+0000","last_became_active":"2024-02-19T15:25:59.144571+0000","last_became_peered":"2024-02-19T15:25:59.144571+0000","last_unstale":"2024-02-19T15:26:01.147553+0000","last_undegraded":"2024-02-19T15:26:01.147553+0000","last_fullsized":"2024-02-19T15:26:01.147553+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:00:37.738268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146304+0000","last_change":"2024-02-19T15:26:00.152580+0000","last_active":"2024-02-19T15:26:01.146304+0000","last_peered":"2024-02-19T15:26:01.146304+0000","last_clean":"2024-02-19T15:26:01.146304+0000","last_became_active":"2024-02-19T15:26:00.151404+0000","last_became_peered":"2024-02-19T15:26:00.151404+0000","last_unstale":"2024-02-19T15:26:01.146304+0000","last_undegraded":"2024-02-19T15:26:01.146304+0000","last_fullsized":"2024-02-19T15:26:01.146304+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:32:55.515920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147776+0000","last_change":"2024-02-19T15:25:59.138603+0000","last_active":"2024-02-19T15:26:01.147776+0000","last_peered":"2024-02-19T15:26:01.147776+0000","last_clean":"2024-02-19T15:26:01.147776+0000","last_became_active":"2024-02-19T15:25:59.138324+0000","last_became_peered":"2024-02-19T15:25:59.138324+0000","last_unstale":"2024-02-19T15:26:01.147776+0000","last_undegraded":"2024-02-19T15:26:01.147776+0000","last_fullsized":"2024-02-19T15:26:01.147776+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:21:21.944376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.165637+0000","last_change":"2024-02-19T15:26:00.153959+0000","last_active":"2024-02-19T15:26:00.165637+0000","last_peered":"2024-02-19T15:26:00.165637+0000","last_clean":"2024-02-19T15:26:00.165637+0000","last_became_active":"2024-02-19T15:26:00.153667+0000","last_became_peered":"2024-02-19T15:26:00.153667+0000","last_unstale":"2024-02-19T15:26:00.165637+0000","last_undegraded":"2024-02-19T15:26:00.165637+0000","last_fullsized":"2024-02-19T15:26:00.165637+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:05:34.715231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"81'3","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147149+0000","last_change":"2024-02-19T15:25:59.147818+0000","last_active":"2024-02-19T15:26:01.147149+0000","last_peered":"2024-02-19T15:26:01.147149+0000","last_clean":"2024-02-19T15:26:01.147149+0000","last_became_active":"2024-02-19T15:25:59.147481+0000","last_became_peered":"2024-02-19T15:25:59.147481+0000","last_unstale":"2024-02-19T15:26:01.147149+0000","last_undegraded":"2024-02-19T15:26:01.147149+0000","last_fullsized":"2024-02-19T15:26:01.147149+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:48.954957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.163829+0000","last_change":"2024-02-19T15:26:00.163053+0000","last_active":"2024-02-19T15:26:00.163829+0000","last_peered":"2024-02-19T15:26:00.163829+0000","last_clean":"2024-02-19T15:26:00.163829+0000","last_became_active":"2024-02-19T15:26:00.162665+0000","last_became_peered":"2024-02-19T15:26:00.162665+0000","last_unstale":"2024-02-19T15:26:00.163829+0000","last_undegraded":"2024-02-19T15:26:00.163829+0000","last_fullsized":"2024-02-19T15:26:00.163829+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:43:15.055359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"81'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146113+0000","last_change":"2024-02-19T15:25:59.140299+0000","last_active":"2024-02-19T15:26:01.146113+0000","last_peered":"2024-02-19T15:26:01.146113+0000","last_clean":"2024-02-19T15:26:01.146113+0000","last_became_active":"2024-02-19T15:25:59.139999+0000","last_became_peered":"2024-02-19T15:25:59.139999+0000","last_unstale":"2024-02-19T15:26:01.146113+0000","last_undegraded":"2024-02-19T15:26:01.146113+0000","last_fullsized":"2024-02-19T15:26:01.146113+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:35:45.839506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.158852+0000","last_change":"2024-02-19T15:26:00.149867+0000","last_active":"2024-02-19T15:26:00.158852+0000","last_peered":"2024-02-19T15:26:00.158852+0000","last_clean":"2024-02-19T15:26:00.158852+0000","last_became_active":"2024-02-19T15:26:00.149559+0000","last_became_peered":"2024-02-19T15:26:00.149559+0000","last_unstale":"2024-02-19T15:26:00.158852+0000","last_undegraded":"2024-02-19T15:26:00.158852+0000","last_fullsized":"2024-02-19T15:26:00.158852+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:28:17.733970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146550+0000","last_change":"2024-02-19T15:25:59.147891+0000","last_active":"2024-02-19T15:26:01.146550+0000","last_peered":"2024-02-19T15:26:01.146550+0000","last_clean":"2024-02-19T15:26:01.146550+0000","last_became_active":"2024-02-19T15:25:59.147497+0000","last_became_peered":"2024-02-19T15:25:59.147497+0000","last_unstale":"2024-02-19T15:26:01.146550+0000","last_undegraded":"2024-02-19T15:26:01.146550+0000","last_fullsized":"2024-02-19T15:26:01.146550+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:34:18.199529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.153046+0000","last_change":"2024-02-19T15:26:00.151252+0000","last_active":"2024-02-19T15:26:00.153046+0000","last_peered":"2024-02-19T15:26:00.153046+0000","last_clean":"2024-02-19T15:26:00.153046+0000","last_became_active":"2024-02-19T15:26:00.150931+0000","last_became_peered":"2024-02-19T15:26:00.150931+0000","last_unstale":"2024-02-19T15:26:00.153046+0000","last_undegraded":"2024-02-19T15:26:00.153046+0000","last_fullsized":"2024-02-19T15:26:00.153046+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:06.498178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.133589+0000","last_change":"2024-02-19T15:25:59.141154+0000","last_active":"2024-02-19T15:26:00.133589+0000","last_peered":"2024-02-19T15:26:00.133589+0000","last_clean":"2024-02-19T15:26:00.133589+0000","last_became_active":"2024-02-19T15:25:59.140880+0000","last_became_peered":"2024-02-19T15:25:59.140880+0000","last_unstale":"2024-02-19T15:26:00.133589+0000","last_undegraded":"2024-02-19T15:26:00.133589+0000","last_fullsized":"2024-02-19T15:26:00.133589+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:49:18.087693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146542+0000","last_change":"2024-02-19T15:26:00.156369+0000","last_active":"2024-02-19T15:26:01.146542+0000","last_peered":"2024-02-19T15:26:01.146542+0000","last_clean":"2024-02-19T15:26:01.146542+0000","last_became_active":"2024-02-19T15:26:00.156127+0000","last_became_peered":"2024-02-19T15:26:00.156127+0000","last_unstale":"2024-02-19T15:26:01.146542+0000","last_undegraded":"2024-02-19T15:26:01.146542+0000","last_fullsized":"2024-02-19T15:26:01.146542+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:25:06.707641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.145948+0000","last_change":"2024-02-19T15:25:59.141165+0000","last_active":"2024-02-19T15:26:01.145948+0000","last_peered":"2024-02-19T15:26:01.145948+0000","last_clean":"2024-02-19T15:26:01.145948+0000","last_became_active":"2024-02-19T15:25:59.140902+0000","last_became_peered":"2024-02-19T15:25:59.140902+0000","last_unstale":"2024-02-19T15:26:01.145948+0000","last_undegraded":"2024-02-19T15:26:01.145948+0000","last_fullsized":"2024-02-19T15:26:01.145948+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:28:35.694362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146422+0000","last_change":"2024-02-19T15:26:00.142607+0000","last_active":"2024-02-19T15:26:01.146422+0000","last_peered":"2024-02-19T15:26:01.146422+0000","last_clean":"2024-02-19T15:26:01.146422+0000","last_became_active":"2024-02-19T15:26:00.142307+0000","last_became_peered":"2024-02-19T15:26:00.142307+0000","last_unstale":"2024-02-19T15:26:01.146422+0000","last_undegraded":"2024-02-19T15:26:01.146422+0000","last_fullsized":"2024-02-19T15:26:01.146422+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:29:22.170656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"81'8","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.167052+0000","last_change":"2024-02-19T15:25:59.142789+0000","last_active":"2024-02-19T15:26:00.167052+0000","last_peered":"2024-02-19T15:26:00.167052+0000","last_clean":"2024-02-19T15:26:00.167052+0000","last_became_active":"2024-02-19T15:25:59.142534+0000","last_became_peered":"2024-02-19T15:25:59.142534+0000","last_unstale":"2024-02-19T15:26:00.167052+0000","last_undegraded":"2024-02-19T15:26:00.167052+0000","last_fullsized":"2024-02-19T15:26:00.167052+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:48:57.602279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.150840+0000","last_change":"2024-02-19T15:26:00.145781+0000","last_active":"2024-02-19T15:26:00.150840+0000","last_peered":"2024-02-19T15:26:00.150840+0000","last_clean":"2024-02-19T15:26:00.150840+0000","last_became_active":"2024-02-19T15:26:00.145451+0000","last_became_peered":"2024-02-19T15:26:00.145451+0000","last_unstale":"2024-02-19T15:26:00.150840+0000","last_undegraded":"2024-02-19T15:26:00.150840+0000","last_fullsized":"2024-02-19T15:26:00.150840+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:46:58.907351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147605+0000","last_change":"2024-02-19T15:25:59.138247+0000","last_active":"2024-02-19T15:26:01.147605+0000","last_peered":"2024-02-19T15:26:01.147605+0000","last_clean":"2024-02-19T15:26:01.147605+0000","last_became_active":"2024-02-19T15:25:59.137938+0000","last_became_peered":"2024-02-19T15:25:59.137938+0000","last_unstale":"2024-02-19T15:26:01.147605+0000","last_undegraded":"2024-02-19T15:26:01.147605+0000","last_fullsized":"2024-02-19T15:26:01.147605+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:17:22.547059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147449+0000","last_change":"2024-02-19T15:26:00.142334+0000","last_active":"2024-02-19T15:26:01.147449+0000","last_peered":"2024-02-19T15:26:01.147449+0000","last_clean":"2024-02-19T15:26:01.147449+0000","last_became_active":"2024-02-19T15:26:00.142028+0000","last_became_peered":"2024-02-19T15:26:00.142028+0000","last_unstale":"2024-02-19T15:26:01.147449+0000","last_undegraded":"2024-02-19T15:26:01.147449+0000","last_fullsized":"2024-02-19T15:26:01.147449+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:05:34.715231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"81'2","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.162160+0000","last_change":"2024-02-19T15:25:59.145572+0000","last_active":"2024-02-19T15:26:00.162160+0000","last_peered":"2024-02-19T15:26:00.162160+0000","last_clean":"2024-02-19T15:26:00.162160+0000","last_became_active":"2024-02-19T15:25:59.145289+0000","last_became_peered":"2024-02-19T15:25:59.145289+0000","last_unstale":"2024-02-19T15:26:00.162160+0000","last_undegraded":"2024-02-19T15:26:00.162160+0000","last_fullsized":"2024-02-19T15:26:00.162160+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:41:41.452759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.162865+0000","last_change":"2024-02-19T15:26:00.161606+0000","last_active":"2024-02-19T15:26:00.162865+0000","last_peered":"2024-02-19T15:26:00.162865+0000","last_clean":"2024-02-19T15:26:00.162865+0000","last_became_active":"2024-02-19T15:26:00.161324+0000","last_became_peered":"2024-02-19T15:26:00.161324+0000","last_unstale":"2024-02-19T15:26:00.162865+0000","last_undegraded":"2024-02-19T15:26:00.162865+0000","last_fullsized":"2024-02-19T15:26:00.162865+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:14:01.639589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.141483+0000","last_change":"2024-02-19T15:25:59.143481+0000","last_active":"2024-02-19T15:26:00.141483+0000","last_peered":"2024-02-19T15:26:00.141483+0000","last_clean":"2024-02-19T15:26:00.141483+0000","last_became_active":"2024-02-19T15:25:59.143199+0000","last_became_peered":"2024-02-19T15:25:59.143199+0000","last_unstale":"2024-02-19T15:26:00.141483+0000","last_undegraded":"2024-02-19T15:26:00.141483+0000","last_fullsized":"2024-02-19T15:26:00.141483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:10.770753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.163839+0000","last_change":"2024-02-19T15:26:00.161744+0000","last_active":"2024-02-19T15:26:00.163839+0000","last_peered":"2024-02-19T15:26:00.163839+0000","last_clean":"2024-02-19T15:26:00.163839+0000","last_became_active":"2024-02-19T15:26:00.161435+0000","last_became_peered":"2024-02-19T15:26:00.161435+0000","last_unstale":"2024-02-19T15:26:00.163839+0000","last_undegraded":"2024-02-19T15:26:00.163839+0000","last_fullsized":"2024-02-19T15:26:00.163839+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:47.889730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"81'2","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.151623+0000","last_change":"2024-02-19T15:25:59.140721+0000","last_active":"2024-02-19T15:26:00.151623+0000","last_peered":"2024-02-19T15:26:00.151623+0000","last_clean":"2024-02-19T15:26:00.151623+0000","last_became_active":"2024-02-19T15:25:59.140434+0000","last_became_peered":"2024-02-19T15:25:59.140434+0000","last_unstale":"2024-02-19T15:26:00.151623+0000","last_undegraded":"2024-02-19T15:26:00.151623+0000","last_fullsized":"2024-02-19T15:26:00.151623+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:09:49.201292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.145898+0000","last_change":"2024-02-19T15:26:00.147849+0000","last_active":"2024-02-19T15:26:01.145898+0000","last_peered":"2024-02-19T15:26:01.145898+0000","last_clean":"2024-02-19T15:26:01.145898+0000","last_became_active":"2024-02-19T15:26:00.147500+0000","last_became_peered":"2024-02-19T15:26:00.147500+0000","last_unstale":"2024-02-19T15:26:01.145898+0000","last_undegraded":"2024-02-19T15:26:01.145898+0000","last_fullsized":"2024-02-19T15:26:01.145898+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:12:11.535536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.132922+0000","last_change":"2024-02-19T15:25:59.145297+0000","last_active":"2024-02-19T15:26:00.132922+0000","last_peered":"2024-02-19T15:26:00.132922+0000","last_clean":"2024-02-19T15:26:00.132922+0000","last_became_active":"2024-02-19T15:25:59.145047+0000","last_became_peered":"2024-02-19T15:25:59.145047+0000","last_unstale":"2024-02-19T15:26:00.132922+0000","last_undegraded":"2024-02-19T15:26:00.132922+0000","last_fullsized":"2024-02-19T15:26:00.132922+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:21.086134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.148261+0000","last_change":"2024-02-19T15:26:00.144473+0000","last_active":"2024-02-19T15:26:00.148261+0000","last_peered":"2024-02-19T15:26:00.148261+0000","last_clean":"2024-02-19T15:26:00.148261+0000","last_became_active":"2024-02-19T15:26:00.144209+0000","last_became_peered":"2024-02-19T15:26:00.144209+0000","last_unstale":"2024-02-19T15:26:00.148261+0000","last_undegraded":"2024-02-19T15:26:00.148261+0000","last_fullsized":"2024-02-19T15:26:00.148261+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:10:21.715225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.131663+0000","last_change":"2024-02-19T15:25:59.144906+0000","last_active":"2024-02-19T15:26:00.131663+0000","last_peered":"2024-02-19T15:26:00.131663+0000","last_clean":"2024-02-19T15:26:00.131663+0000","last_became_active":"2024-02-19T15:25:59.144601+0000","last_became_peered":"2024-02-19T15:25:59.144601+0000","last_unstale":"2024-02-19T15:26:00.131663+0000","last_undegraded":"2024-02-19T15:26:00.131663+0000","last_fullsized":"2024-02-19T15:26:00.131663+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:43:35.176726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146061+0000","last_change":"2024-02-19T15:26:00.156882+0000","last_active":"2024-02-19T15:26:01.146061+0000","last_peered":"2024-02-19T15:26:01.146061+0000","last_clean":"2024-02-19T15:26:01.146061+0000","last_became_active":"2024-02-19T15:26:00.156490+0000","last_became_peered":"2024-02-19T15:26:00.156490+0000","last_unstale":"2024-02-19T15:26:01.146061+0000","last_undegraded":"2024-02-19T15:26:01.146061+0000","last_fullsized":"2024-02-19T15:26:01.146061+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:31:39.275272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.133355+0000","last_change":"2024-02-19T15:25:59.140834+0000","last_active":"2024-02-19T15:26:00.133355+0000","last_peered":"2024-02-19T15:26:00.133355+0000","last_clean":"2024-02-19T15:26:00.133355+0000","last_became_active":"2024-02-19T15:25:59.140559+0000","last_became_peered":"2024-02-19T15:25:59.140559+0000","last_unstale":"2024-02-19T15:26:00.133355+0000","last_undegraded":"2024-02-19T15:26:00.133355+0000","last_fullsized":"2024-02-19T15:26:00.133355+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:46:57.900013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.149272+0000","last_change":"2024-02-19T15:26:00.146112+0000","last_active":"2024-02-19T15:26:00.149272+0000","last_peered":"2024-02-19T15:26:00.149272+0000","last_clean":"2024-02-19T15:26:00.149272+0000","last_became_active":"2024-02-19T15:26:00.143543+0000","last_became_peered":"2024-02-19T15:26:00.143543+0000","last_unstale":"2024-02-19T15:26:00.149272+0000","last_undegraded":"2024-02-19T15:26:00.149272+0000","last_fullsized":"2024-02-19T15:26:00.149272+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:44.769088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147000+0000","last_change":"2024-02-19T15:25:59.145996+0000","last_active":"2024-02-19T15:26:01.147000+0000","last_peered":"2024-02-19T15:26:01.147000+0000","last_clean":"2024-02-19T15:26:01.147000+0000","last_became_active":"2024-02-19T15:25:59.144656+0000","last_became_peered":"2024-02-19T15:25:59.144656+0000","last_unstale":"2024-02-19T15:26:01.147000+0000","last_undegraded":"2024-02-19T15:26:01.147000+0000","last_fullsized":"2024-02-19T15:26:01.147000+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:24:06.383825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.148414+0000","last_change":"2024-02-19T15:26:00.145811+0000","last_active":"2024-02-19T15:26:00.148414+0000","last_peered":"2024-02-19T15:26:00.148414+0000","last_clean":"2024-02-19T15:26:00.148414+0000","last_became_active":"2024-02-19T15:26:00.145521+0000","last_became_peered":"2024-02-19T15:26:00.145521+0000","last_unstale":"2024-02-19T15:26:00.148414+0000","last_undegraded":"2024-02-19T15:26:00.148414+0000","last_fullsized":"2024-02-19T15:26:00.148414+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:49:19.095030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146991+0000","last_change":"2024-02-19T15:25:59.147788+0000","last_active":"2024-02-19T15:26:01.146991+0000","last_peered":"2024-02-19T15:26:01.146991+0000","last_clean":"2024-02-19T15:26:01.146991+0000","last_became_active":"2024-02-19T15:25:59.147480+0000","last_became_peered":"2024-02-19T15:25:59.147480+0000","last_unstale":"2024-02-19T15:26:01.146991+0000","last_undegraded":"2024-02-19T15:26:01.146991+0000","last_fullsized":"2024-02-19T15:26:01.146991+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:29:01.472900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147512+0000","last_change":"2024-02-19T15:26:00.141946+0000","last_active":"2024-02-19T15:26:01.147512+0000","last_peered":"2024-02-19T15:26:01.147512+0000","last_clean":"2024-02-19T15:26:01.147512+0000","last_became_active":"2024-02-19T15:26:00.141407+0000","last_became_peered":"2024-02-19T15:26:00.141407+0000","last_unstale":"2024-02-19T15:26:01.147512+0000","last_undegraded":"2024-02-19T15:26:01.147512+0000","last_fullsized":"2024-02-19T15:26:01.147512+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:32.505935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147172+0000","last_change":"2024-02-19T15:25:59.139522+0000","last_active":"2024-02-19T15:26:01.147172+0000","last_peered":"2024-02-19T15:26:01.147172+0000","last_clean":"2024-02-19T15:26:01.147172+0000","last_became_active":"2024-02-19T15:25:59.139081+0000","last_became_peered":"2024-02-19T15:25:59.139081+0000","last_unstale":"2024-02-19T15:26:01.147172+0000","last_undegraded":"2024-02-19T15:26:01.147172+0000","last_fullsized":"2024-02-19T15:26:01.147172+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:17:30.699147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.163816+0000","last_change":"2024-02-19T15:26:00.162872+0000","last_active":"2024-02-19T15:26:00.163816+0000","last_peered":"2024-02-19T15:26:00.163816+0000","last_clean":"2024-02-19T15:26:00.163816+0000","last_became_active":"2024-02-19T15:26:00.162658+0000","last_became_peered":"2024-02-19T15:26:00.162658+0000","last_unstale":"2024-02-19T15:26:00.163816+0000","last_undegraded":"2024-02-19T15:26:00.163816+0000","last_fullsized":"2024-02-19T15:26:00.163816+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:44:01.635671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147455+0000","last_change":"2024-02-19T15:25:59.144981+0000","last_active":"2024-02-19T15:26:01.147455+0000","last_peered":"2024-02-19T15:26:01.147455+0000","last_clean":"2024-02-19T15:26:01.147455+0000","last_became_active":"2024-02-19T15:25:59.144592+0000","last_became_peered":"2024-02-19T15:25:59.144592+0000","last_unstale":"2024-02-19T15:26:01.147455+0000","last_undegraded":"2024-02-19T15:26:01.147455+0000","last_fullsized":"2024-02-19T15:26:01.147455+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:56.829482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.145843+0000","last_change":"2024-02-19T15:26:00.151112+0000","last_active":"2024-02-19T15:26:01.145843+0000","last_peered":"2024-02-19T15:26:01.145843+0000","last_clean":"2024-02-19T15:26:01.145843+0000","last_became_active":"2024-02-19T15:26:00.150833+0000","last_became_peered":"2024-02-19T15:26:00.150833+0000","last_unstale":"2024-02-19T15:26:01.145843+0000","last_undegraded":"2024-02-19T15:26:01.145843+0000","last_fullsized":"2024-02-19T15:26:01.145843+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:15:36.439258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.135569+0000","last_change":"2024-02-19T15:25:59.141990+0000","last_active":"2024-02-19T15:26:00.135569+0000","last_peered":"2024-02-19T15:26:00.135569+0000","last_clean":"2024-02-19T15:26:00.135569+0000","last_became_active":"2024-02-19T15:25:59.141648+0000","last_became_peered":"2024-02-19T15:25:59.141648+0000","last_unstale":"2024-02-19T15:26:00.135569+0000","last_undegraded":"2024-02-19T15:26:00.135569+0000","last_fullsized":"2024-02-19T15:26:00.135569+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:49:10.613400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.161993+0000","last_change":"2024-02-19T15:26:00.150519+0000","last_active":"2024-02-19T15:26:00.161993+0000","last_peered":"2024-02-19T15:26:00.161993+0000","last_clean":"2024-02-19T15:26:00.161993+0000","last_became_active":"2024-02-19T15:26:00.150235+0000","last_became_peered":"2024-02-19T15:26:00.150235+0000","last_unstale":"2024-02-19T15:26:00.161993+0000","last_undegraded":"2024-02-19T15:26:00.161993+0000","last_fullsized":"2024-02-19T15:26:00.161993+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:50:34.796433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.130335+0000","last_change":"2024-02-19T15:25:59.140936+0000","last_active":"2024-02-19T15:26:00.130335+0000","last_peered":"2024-02-19T15:26:00.130335+0000","last_clean":"2024-02-19T15:26:00.130335+0000","last_became_active":"2024-02-19T15:25:59.140448+0000","last_became_peered":"2024-02-19T15:25:59.140448+0000","last_unstale":"2024-02-19T15:26:00.130335+0000","last_undegraded":"2024-02-19T15:26:00.130335+0000","last_fullsized":"2024-02-19T15:26:00.130335+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:44.096068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.162599+0000","last_change":"2024-02-19T15:26:00.161556+0000","last_active":"2024-02-19T15:26:00.162599+0000","last_peered":"2024-02-19T15:26:00.162599+0000","last_clean":"2024-02-19T15:26:00.162599+0000","last_became_active":"2024-02-19T15:26:00.161274+0000","last_became_peered":"2024-02-19T15:26:00.161274+0000","last_unstale":"2024-02-19T15:26:00.162599+0000","last_undegraded":"2024-02-19T15:26:00.162599+0000","last_fullsized":"2024-02-19T15:26:00.162599+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:53:03.954569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147265+0000","last_change":"2024-02-19T15:25:59.144110+0000","last_active":"2024-02-19T15:26:01.147265+0000","last_peered":"2024-02-19T15:26:01.147265+0000","last_clean":"2024-02-19T15:26:01.147265+0000","last_became_active":"2024-02-19T15:25:59.143881+0000","last_became_peered":"2024-02-19T15:25:59.143881+0000","last_unstale":"2024-02-19T15:26:01.147265+0000","last_undegraded":"2024-02-19T15:26:01.147265+0000","last_fullsized":"2024-02-19T15:26:01.147265+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:00:56.567362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147125+0000","last_change":"2024-02-19T15:26:00.140998+0000","last_active":"2024-02-19T15:26:01.147125+0000","last_peered":"2024-02-19T15:26:01.147125+0000","last_clean":"2024-02-19T15:26:01.147125+0000","last_became_active":"2024-02-19T15:26:00.139693+0000","last_became_peered":"2024-02-19T15:26:00.139693+0000","last_unstale":"2024-02-19T15:26:01.147125+0000","last_undegraded":"2024-02-19T15:26:01.147125+0000","last_fullsized":"2024-02-19T15:26:01.147125+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:26:01.775208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.132476+0000","last_change":"2024-02-19T15:25:59.142322+0000","last_active":"2024-02-19T15:26:00.132476+0000","last_peered":"2024-02-19T15:26:00.132476+0000","last_clean":"2024-02-19T15:26:00.132476+0000","last_became_active":"2024-02-19T15:25:59.142044+0000","last_became_peered":"2024-02-19T15:25:59.142044+0000","last_unstale":"2024-02-19T15:26:00.132476+0000","last_undegraded":"2024-02-19T15:26:00.132476+0000","last_fullsized":"2024-02-19T15:26:00.132476+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:25.400223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146866+0000","last_change":"2024-02-19T15:26:00.150250+0000","last_active":"2024-02-19T15:26:01.146866+0000","last_peered":"2024-02-19T15:26:01.146866+0000","last_clean":"2024-02-19T15:26:01.146866+0000","last_became_active":"2024-02-19T15:26:00.143077+0000","last_became_peered":"2024-02-19T15:26:00.143077+0000","last_unstale":"2024-02-19T15:26:01.146866+0000","last_undegraded":"2024-02-19T15:26:01.146866+0000","last_fullsized":"2024-02-19T15:26:01.146866+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:09:12.862734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.165097+0000","last_change":"2024-02-19T15:25:59.145365+0000","last_active":"2024-02-19T15:26:00.165097+0000","last_peered":"2024-02-19T15:26:00.165097+0000","last_clean":"2024-02-19T15:26:00.165097+0000","last_became_active":"2024-02-19T15:25:59.145075+0000","last_became_peered":"2024-02-19T15:25:59.145075+0000","last_unstale":"2024-02-19T15:26:00.165097+0000","last_undegraded":"2024-02-19T15:26:00.165097+0000","last_fullsized":"2024-02-19T15:26:00.165097+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:01:06.434946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":277,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.133214+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:26:00.133214+0000","last_peered":"2024-02-19T15:26:00.133214+0000","last_clean":"2024-02-19T15:26:00.133214+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:26:00.133214+0000","last_undegraded":"2024-02-19T15:26:00.133214+0000","last_fullsized":"2024-02-19T15:26:00.133214+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509493,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1015808,"data_stored":425993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.61699999999999999,"15min":0.61799999999999999},"min":{"1min":0.42699999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.90100000000000002,"5min":1.022,"15min":1.022},"last":0.441},{"interface":"front","average":{"1min":0.629,"5min":0.71599999999999997,"15min":0.76400000000000001},"min":{"1min":0.47499999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83699999999999997,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.55200000000000005}]},{"osd":1,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64400000000000002,"15min":0.65200000000000002},"min":{"1min":0.48499999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85099999999999998,"5min":1.171,"15min":1.171},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.65500000000000003,"15min":0.65200000000000002},"min":{"1min":0.50700000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.92300000000000004,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.628,"15min":0.63100000000000001},"min":{"1min":0.44600000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.85299999999999998,"5min":1.216,"15min":1.216},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.69699999999999995,"15min":0.68799999999999994},"min":{"1min":0.55800000000000005,"5min":0.495,"15min":0.495},"max":{"1min":1.01,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.48999999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.68999999999999995,"15min":0.67900000000000005},"min":{"1min":0.54800000000000004,"5min":0.438,"15min":0.438},"max":{"1min":0.91600000000000004,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.752,"5min":0.67100000000000004,"15min":0.66000000000000003},"min":{"1min":0.55600000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.57699999999999996}]},{"osd":4,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.626,"15min":0.60699999999999998},"min":{"1min":0.50800000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.90400000000000003,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63,"15min":0.60399999999999998},"min":{"1min":0.48499999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92600000000000005,"5min":0.998,"15min":0.998},"last":0.41699999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.63200000000000001,"15min":0.60499999999999998},"min":{"1min":0.58699999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.629},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.69499999999999995,"15min":0.67300000000000004},"min":{"1min":0.61499999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.95299999999999996,"5min":1.02,"15min":1.02},"last":0.65600000000000003}]},{"osd":7,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.68799999999999994,"15min":0.65700000000000003},"min":{"1min":0.61899999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.70899999999999996,"15min":0.67700000000000005},"min":{"1min":0.69799999999999995,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.72999999999999998}]}]},{"osd":7,"up_from":14,"seq":60129542197,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32732,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339108,"statfs":{"total":96636764160,"available":96603246592,"internally_reserved":0,"allocated":1589248,"data_stored":1009437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.59699999999999998,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.87,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.67400000000000004},"min":{"1min":0.38800000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.60799999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.63500000000000001,"15min":0.60099999999999998},"min":{"1min":0.498,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86799999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.70399999999999996,"15min":0.72099999999999997},"min":{"1min":0.496,"5min":0.437,"15min":0.437},"max":{"1min":1.008,"5min":2.968,"15min":2.968},"last":0.66100000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.64700000000000002,"15min":0.63600000000000001},"min":{"1min":0.47599999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85399999999999998,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.497,"5min":0.438,"15min":0.438},"max":{"1min":0.97399999999999998,"5min":2.508,"15min":2.508},"last":0.63600000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66300000000000003,"15min":0.65900000000000003},"min":{"1min":0.50900000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.91300000000000003,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.70999999999999996,"15min":0.69499999999999995},"min":{"1min":0.59399999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.94699999999999995,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":0.69999999999999996}]},{"osd":4,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.59499999999999997,"15min":0.53900000000000003},"min":{"1min":0.432,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.61799999999999999,"15min":0.58199999999999996},"min":{"1min":0.40100000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93600000000000005,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.628}]},{"osd":5,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.65700000000000003,"15min":0.58399999999999996},"min":{"1min":0.53500000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.63800000000000001},"min":{"1min":0.54700000000000004,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.97299999999999998,"5min":1.046,"15min":1.046},"last":0.68200000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.67100000000000004,"15min":0.61299999999999999},"min":{"1min":0.46999999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72599999999999998,"15min":0.67500000000000004},"min":{"1min":0.59699999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.76600000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542198,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32184,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339656,"statfs":{"total":96636764160,"available":96603807744,"internally_reserved":0,"allocated":1019904,"data_stored":426933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.67200000000000004,"15min":0.64400000000000002},"min":{"1min":0.38300000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.97699999999999998,"5min":5.7789999999999999,"15min":5.7789999999999999},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.71099999999999997,"15min":0.63700000000000001},"min":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.86499999999999999,"5min":12.359999999999999,"15min":12.359999999999999},"last":0.432}]},{"osd":1,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.73799999999999999,"15min":0.67000000000000004},"min":{"1min":0.432,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80000000000000004,"5min":10.938000000000001,"15min":10.938000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.78900000000000003,"15min":0.71299999999999997},"min":{"1min":0.50600000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85599999999999998,"5min":10.531000000000001,"15min":10.531000000000001},"last":0.66800000000000004}]},{"osd":2,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.79700000000000004,"15min":0.72099999999999997},"min":{"1min":0.42999999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.89900000000000002,"5min":12.925000000000001,"15min":12.925000000000001},"last":0.875},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.72999999999999998,"15min":0.64100000000000001},"min":{"1min":0.48899999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89000000000000001,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.85299999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.91100000000000003,"15min":0.73499999999999999},"min":{"1min":0.49399999999999999,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82299999999999995,"5min":26.161999999999999,"15min":26.161999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.91000000000000003,"15min":0.73999999999999999},"min":{"1min":0.47299999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1120000000000001,"5min":27.934999999999999,"15min":27.934999999999999},"last":1.1120000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.67500000000000004,"15min":0.59199999999999997},"min":{"1min":0.45700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83099999999999996,"5min":9.0549999999999997,"15min":9.0549999999999997},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.70899999999999996,"15min":0.67100000000000004},"min":{"1min":0.44600000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.92800000000000005,"5min":6.2549999999999999,"15min":6.2549999999999999},"last":0.92800000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.746,"15min":0.67000000000000004},"min":{"1min":0.50700000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.6539999999999999,"5min":6.5229999999999997,"15min":6.5229999999999997},"last":1.6539999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81899999999999995,"15min":0.749},"min":{"1min":0.63500000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.2310000000000001,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":1.2310000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.77600000000000002,"15min":0.69399999999999995},"min":{"1min":0.41499999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.95399999999999996,"5min":11.986000000000001,"15min":11.986000000000001},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.77700000000000002,"15min":0.71499999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.89100000000000001,"5min":8.9770000000000003,"15min":8.9770000000000003},"last":0.80500000000000005}]}]},{"osd":4,"up_from":14,"seq":60129542196,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":991232,"data_stored":418999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.378,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.623,"15min":0.60499999999999998},"min":{"1min":0.47099999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85499999999999998,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.51900000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64100000000000001,"15min":0.61899999999999999},"min":{"1min":0.48999999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.879,"5min":1.081,"15min":1.081},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64400000000000002,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83699999999999997,"5min":1.008,"15min":1.008},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.65400000000000003,"15min":0.66000000000000003},"min":{"1min":0.49299999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.90700000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68200000000000005,"15min":0.67600000000000005},"min":{"1min":0.45100000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92000000000000004,"5min":1.002,"15min":1.002},"last":0.65600000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.70799999999999996,"15min":0.68200000000000005},"min":{"1min":0.50600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94899999999999995,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.65900000000000003,"15min":0.65300000000000002},"min":{"1min":0.42799999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89600000000000002,"5min":1.153,"15min":1.153},"last":0.57299999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63600000000000001,"15min":0.60599999999999998},"min":{"1min":0.433,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85399999999999998,"5min":1.042,"15min":1.042},"last":0.495},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70599999999999996,"15min":0.68200000000000005},"min":{"1min":0.54200000000000004,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.94799999999999995,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.81699999999999995}]},{"osd":6,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.622,"15min":0.58299999999999996},"min":{"1min":0.376,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.92200000000000004,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.628},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68600000000000005,"15min":0.65500000000000003},"min":{"1min":0.52900000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90800000000000003,"5min":1.054,"15min":1.054},"last":0.747}]},{"osd":7,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.70199999999999996,"15min":0.67100000000000004},"min":{"1min":0.44800000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.98199999999999998,"5min":1.115,"15min":1.115},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.75600000000000001,"15min":0.73399999999999999},"min":{"1min":0.46999999999999997,"5min":0.443,"15min":0.443},"max":{"1min":1.0169999999999999,"5min":1.24,"15min":1.24},"last":0.70799999999999996}]}]},{"osd":3,"up_from":14,"seq":60129542196,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":1003520,"data_stored":420611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42899999999999999,"15min":0.44700000000000001},"min":{"1min":0.23699999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76600000000000001,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.54100000000000004,"15min":0.64600000000000002},"min":{"1min":0.27500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67600000000000005,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.54900000000000004}]},{"osd":1,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.433,"15min":0.442},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.66000000000000003,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.59699999999999998,"15min":0.70899999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.66400000000000003,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.316}]},{"osd":2,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56799999999999995,"15min":0.61199999999999999},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.66500000000000004,"5min":1.008,"15min":1.008},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55000000000000004,"15min":0.58899999999999997},"min":{"1min":0.39900000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86199999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.44800000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.70299999999999996,"15min":0.67900000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.90600000000000003,"5min":1.004,"15min":1.004},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.80100000000000005,"15min":0.86099999999999999},"min":{"1min":0.58199999999999996,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0269999999999999,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.754}]},{"osd":5,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.80100000000000005,"15min":0.85499999999999998},"min":{"1min":0.54700000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.96699999999999997,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.74199999999999999,"15min":0.748},"min":{"1min":0.53300000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.93899999999999995,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.72699999999999998}]},{"osd":6,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.85499999999999998,"15min":0.90000000000000002},"min":{"1min":0.55300000000000005,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.90800000000000003,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.79600000000000004,"15min":0.81699999999999995},"min":{"1min":0.51800000000000002,"5min":0.502,"15min":0.502},"max":{"1min":1.0209999999999999,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.61899999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76400000000000001,"15min":0.78000000000000003},"min":{"1min":0.52700000000000002,"5min":0.504,"15min":0.504},"max":{"1min":0.96799999999999997,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.84099999999999997,"15min":0.92000000000000004},"min":{"1min":0.57399999999999995,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0009999999999999,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.68100000000000005}]}]},{"osd":2,"up_from":14,"seq":60129542197,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":995328,"data_stored":419581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.66700000000000004,"15min":0.69699999999999995},"min":{"1min":0.27400000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.64400000000000002,"5min":15.42,"15min":15.42},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.71499999999999997,"15min":0.88500000000000001},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.76500000000000001,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.60099999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.75700000000000001,"15min":0.89700000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58999999999999997,"5min":9.5269999999999992,"15min":9.5269999999999992},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.502,"5min":0.94199999999999995,"15min":1.006},"min":{"1min":0.39700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.621,"5min":22.954999999999998,"15min":22.954999999999998},"last":0.621}]},{"osd":3,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.67100000000000004,"15min":0.78300000000000003},"min":{"1min":0.376,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66200000000000003,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.74299999999999999,"15min":0.82299999999999995},"min":{"1min":0.41799999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69399999999999995,"5min":7.734,"15min":7.734},"last":0.64100000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.91400000000000003,"15min":0.90300000000000002},"min":{"1min":0.52000000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.016,"5min":15.369,"15min":15.369},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.82699999999999996,"15min":0.89300000000000002},"min":{"1min":0.53800000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93700000000000006,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.76200000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.871,"15min":0.88900000000000001},"min":{"1min":0.53900000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.95999999999999996,"5min":7.6520000000000001,"15min":7.6520000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.98699999999999999,"15min":0.97499999999999998},"min":{"1min":0.60199999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0660000000000001,"5min":15.391,"15min":15.391},"last":0.751}]},{"osd":6,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.95799999999999996,"15min":0.94399999999999995},"min":{"1min":0.54900000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.90100000000000002,"5min":15.337,"15min":15.337},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.876,"15min":0.95799999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.98499999999999999,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.78900000000000003}]},{"osd":7,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.90600000000000003,"15min":0.98099999999999998},"min":{"1min":0.58399999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.028,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.874},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.91600000000000004,"15min":0.95299999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.98099999999999998,"5min":7.6929999999999996,"15min":7.6929999999999996},"last":0.90200000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542198,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339680,"statfs":{"total":96636764160,"available":96603832320,"internally_reserved":0,"allocated":1003520,"data_stored":419193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.39200000000000002,"15min":0.42399999999999999},"min":{"1min":0.248,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56799999999999995,"5min":0.747,"15min":0.747},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.438,"15min":0.46600000000000003},"min":{"1min":0.28100000000000003,"5min":0.25,"15min":0.25},"max":{"1min":0.69599999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.38}]},{"osd":2,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.42599999999999999,"15min":0.44400000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.56100000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.48799999999999999,"15min":0.52600000000000002},"min":{"1min":0.36299999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.501,"5min":1.071,"15min":1.071},"last":0.36299999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58899999999999997,"15min":0.63300000000000001},"min":{"1min":0.44800000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.64600000000000002,"5min":1.365,"15min":1.365},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.57399999999999995,"15min":0.63},"min":{"1min":0.42999999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.61399999999999999,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.55500000000000005}]},{"osd":4,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.65300000000000002,"15min":0.61299999999999999},"min":{"1min":0.52100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.99099999999999999,"5min":1.127,"15min":1.127},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.69299999999999995,"15min":0.68000000000000005},"min":{"1min":0.54900000000000004,"5min":0.435,"15min":0.435},"max":{"1min":0.98399999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.67900000000000005}]},{"osd":5,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70899999999999996,"15min":0.69699999999999995},"min":{"1min":0.54200000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.95599999999999996,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.70399999999999996,"15min":0.68500000000000005},"min":{"1min":0.503,"5min":0.443,"15min":0.443},"max":{"1min":0.94199999999999995,"5min":1.107,"15min":1.107},"last":0.871}]},{"osd":6,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73199999999999998,"15min":0.72899999999999998},"min":{"1min":0.57799999999999996,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0449999999999999,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.74399999999999999,"15min":0.746},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0669999999999999,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.63}]},{"osd":7,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.76800000000000002,"15min":0.78600000000000003},"min":{"1min":0.59999999999999998,"5min":0.496,"15min":0.496},"max":{"1min":0.95299999999999996,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.76400000000000001,"15min":0.748},"min":{"1min":0.58499999999999996,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0229999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.90900000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542196,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36896,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334944,"statfs":{"total":96636764160,"available":96598982656,"internally_reserved":0,"allocated":1589248,"data_stored":1009401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.50700000000000001,"15min":0.628},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.66200000000000003,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.46300000000000002,"15min":0.48299999999999998},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.76000000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.37}]},{"osd":2,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.65400000000000003,"15min":0.78400000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.67500000000000004,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.61799999999999999,"15min":0.746},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69099999999999995,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.53700000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.56699999999999995,"15min":0.67900000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.61699999999999999,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.5},{"interface":"front","average":{"1min":0.495,"5min":0.54800000000000004,"15min":0.60999999999999999},"min":{"1min":0.38100000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.72499999999999998,"5min":1.345,"15min":1.345},"last":0.441}]},{"osd":4,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.745,"15min":0.74299999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.0700000000000001,"5min":1.095,"15min":1.095},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.84399999999999997,"15min":0.92500000000000004},"min":{"1min":0.64000000000000001,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0049999999999999,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.79200000000000004}]},{"osd":5,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.73799999999999999,"15min":0.751},"min":{"1min":0.55900000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.97999999999999998,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.77300000000000002,"15min":0.85299999999999998},"min":{"1min":0.52900000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.05,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.65100000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.82099999999999995,"15min":0.91500000000000004},"min":{"1min":0.59799999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.95899999999999996,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73799999999999999,"15min":0.75700000000000001},"min":{"1min":0.57999999999999996,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.84999999999999998,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.64200000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.85199999999999998,"15min":0.94199999999999995},"min":{"1min":0.58599999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.0229999999999999,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.83499999999999996,"15min":0.94099999999999995},"min":{"1min":0.55900000000000005,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.96499999999999997,"5min":4.915,"15min":4.915},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:26:04.276 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:26:04.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.444+0000 7fa84ec00640 1 Processor -- start 2024-02-19T15:26:04.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.444+0000 7fa84ec00640 1 -- start start 2024-02-19T15:26:04.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.444+0000 7fa84ec00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa84806b5c0 0x7fa848166ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:04.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.444+0000 7fa84ec00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa848145a80 con 0x7fa8480cf670 2024-02-19T15:26:04.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.444+0000 7fa84ec00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8480d1880 con 0x7fa8481456a0 2024-02-19T15:26:04.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.444+0000 7fa84ec00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8480d03c0 con 0x7fa84806b5c0 2024-02-19T15:26:04.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.444+0000 7fa83f400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa8480cf670 0x7fa84806aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54854/0 (socket says 172.21.15.47:54854) 2024-02-19T15:26:04.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.444+0000 7fa83f400640 1 -- 172.21.15.47:0/2658867540 learned_addr learned my addr 172.21.15.47:0/2658867540 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:04.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.448+0000 7fa84cc00640 1 --2- 172.21.15.47:0/2658867540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa84806b5c0 0x7fa848166ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:04.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.448+0000 7fa847400640 1 -- 172.21.15.47:0/2658867540 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008884091 0 0) 0x7fa848145a80 con 0x7fa8480cf670 2024-02-19T15:26:04.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.448+0000 7fa847400640 1 -- 172.21.15.47:0/2658867540 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa82c003530 con 0x7fa8480cf670 2024-02-19T15:26:04.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.448+0000 7fa847400640 1 -- 172.21.15.47:0/2658867540 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938502876 0 0) 0x7fa8480d1880 con 0x7fa8481456a0 2024-02-19T15:26:04.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.448+0000 7fa847400640 1 -- 172.21.15.47:0/2658867540 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa848145a80 con 0x7fa8481456a0 2024-02-19T15:26:04.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.448+0000 7fa84cc00640 1 -- 172.21.15.47:0/2658867540 >> v1:172.21.15.116:6789/0 conn(0x7fa8481456a0 legacy=0x7fa8480cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:04.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.448+0000 7fa84cc00640 1 -- 172.21.15.47:0/2658867540 >> v1:172.21.15.47:6789/0 conn(0x7fa8480cf670 legacy=0x7fa84806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:04.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.448+0000 7fa84cc00640 1 -- 172.21.15.47:0/2658867540 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8480cfb20 con 0x7fa84806b5c0 2024-02-19T15:26:04.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.448+0000 7fa84cc00640 1 --2- 172.21.15.47:0/2658867540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa84806b5c0 0x7fa848166ba0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa8400097f0 tx=0x7fa84001c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=460b2ac2f61d73d0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:04.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa847400640 1 -- 172.21.15.47:0/2658867540 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8400ac630 con 0x7fa84806b5c0 2024-02-19T15:26:04.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa847400640 1 -- 172.21.15.47:0/2658867540 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8400ac790 con 0x7fa84806b5c0 2024-02-19T15:26:04.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa847400640 1 -- 172.21.15.47:0/2658867540 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8400acb00 con 0x7fa84806b5c0 2024-02-19T15:26:04.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa84ec00640 1 -- 172.21.15.47:0/2658867540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa84806b5c0 msgr2=0x7fa848166ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:04.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa84ec00640 1 --2- 172.21.15.47:0/2658867540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa84806b5c0 0x7fa848166ba0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa8400097f0 tx=0x7fa84001c4e0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa84ec00640 1 -- 172.21.15.47:0/2658867540 shutdown_connections 2024-02-19T15:26:04.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa84ec00640 1 --2- 172.21.15.47:0/2658867540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa84806b5c0 0x7fa848166ba0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa84ec00640 1 -- 172.21.15.47:0/2658867540 >> 172.21.15.47:0/2658867540 conn(0x7fa8480d0ad0 msgr2=0x7fa8480d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:04.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa84ec00640 1 -- 172.21.15.47:0/2658867540 shutdown_connections 2024-02-19T15:26:04.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa84ec00640 1 -- 172.21.15.47:0/2658867540 wait complete. 2024-02-19T15:26:04.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa84ec00640 1 Processor -- start 2024-02-19T15:26:04.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa84ec00640 1 -- start start 2024-02-19T15:26:04.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa84ec00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa84806b5c0 0x7fa848157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:04.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa84ec00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8480cf670 0x7fa848141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:04.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa84ec00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa8481456a0 0x7fa848142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:04.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa84ec00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84815dc80 con 0x7fa8480cf670 2024-02-19T15:26:04.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa84ec00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa848142560 con 0x7fa84806b5c0 2024-02-19T15:26:04.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa84ec00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa84815d070 con 0x7fa8481456a0 2024-02-19T15:26:04.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa83f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8480cf670 0x7fa848141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:04.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa847e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa84806b5c0 0x7fa848157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:04.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa83f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8480cf670 0x7fa848141ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50602/0 (socket says 172.21.15.47:50602) 2024-02-19T15:26:04.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.452+0000 7fa83f400640 1 -- 172.21.15.47:0/2032331590 learned_addr learned my addr 172.21.15.47:0/2032331590 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:04.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.456+0000 7fa84cc00640 1 --2- 172.21.15.47:0/2032331590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa8481456a0 0x7fa848142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:04.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.456+0000 7fa83f400640 1 -- 172.21.15.47:0/2032331590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa8481456a0 msgr2=0x7fa848142020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:04.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.456+0000 7fa83f400640 1 --2- 172.21.15.47:0/2032331590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa8481456a0 0x7fa848142020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.456+0000 7fa83f400640 1 -- 172.21.15.47:0/2032331590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa84806b5c0 msgr2=0x7fa848157f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:04.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.456+0000 7fa83f400640 1 --2- 172.21.15.47:0/2032331590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa84806b5c0 0x7fa848157f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.456+0000 7fa83f400640 1 -- 172.21.15.47:0/2032331590 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa840009420 con 0x7fa8480cf670 2024-02-19T15:26:04.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.456+0000 7fa83f400640 1 --2- 172.21.15.47:0/2032331590 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8480cf670 0x7fa848141ae0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa83000c4d0 tx=0x7fa83000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:04.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.456+0000 7fa844c00640 1 -- 172.21.15.47:0/2032331590 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa830005cf0 con 0x7fa8480cf670 2024-02-19T15:26:04.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.456+0000 7fa844c00640 1 -- 172.21.15.47:0/2032331590 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa830005e50 con 0x7fa8480cf670 2024-02-19T15:26:04.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.456+0000 7fa844c00640 1 -- 172.21.15.47:0/2032331590 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa830007270 con 0x7fa8480cf670 2024-02-19T15:26:04.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.456+0000 7fa84ec00640 1 -- 172.21.15.47:0/2032331590 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84815d350 con 0x7fa8480cf670 2024-02-19T15:26:04.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.456+0000 7fa84ec00640 1 -- 172.21.15.47:0/2032331590 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84815d910 con 0x7fa8480cf670 2024-02-19T15:26:04.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.456+0000 7fa84ec00640 1 -- 172.21.15.47:0/2032331590 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa848075360 con 0x7fa8480cf670 2024-02-19T15:26:04.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.456+0000 7fa844c00640 1 -- 172.21.15.47:0/2032331590 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa8300073d0 con 0x7fa8480cf670 2024-02-19T15:26:04.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.460+0000 7fa844c00640 1 --2- 172.21.15.47:0/2032331590 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa824066e40 0x7fa824069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:04.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.460+0000 7fa844c00640 1 -- 172.21.15.47:0/2032331590 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa830083ea0 con 0x7fa8480cf670 2024-02-19T15:26:04.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.460+0000 7fa847e00640 1 --2- 172.21.15.47:0/2032331590 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa824066e40 0x7fa824069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:04.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.460+0000 7fa847e00640 1 --2- 172.21.15.47:0/2032331590 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa824066e40 0x7fa824069300 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa838007fd0 tx=0x7fa838009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:04.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.464+0000 7fa844c00640 1 -- 172.21.15.47:0/2032331590 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa830050d40 con 0x7fa8480cf670 2024-02-19T15:26:04.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.636+0000 7fa84ec00640 1 -- 172.21.15.47:0/2032331590 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa84806aeb0 con 0x7fa824066e40 2024-02-19T15:26:04.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.644+0000 7fa844c00640 1 -- 172.21.15.47:0/2032331590 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289181 (secure 0 0 0) 0x7fa84806aeb0 con 0x7fa824066e40 2024-02-19T15:26:04.652 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:04.655 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:26:04.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.644+0000 7fa84ec00640 1 -- 172.21.15.47:0/2032331590 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa824066e40 msgr2=0x7fa824069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:04.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.644+0000 7fa84ec00640 1 --2- 172.21.15.47:0/2032331590 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa824066e40 0x7fa824069300 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa838007fd0 tx=0x7fa838009040 comp rx=0 tx=0).stop 2024-02-19T15:26:04.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.644+0000 7fa84ec00640 1 -- 172.21.15.47:0/2032331590 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8480cf670 msgr2=0x7fa848141ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:04.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.644+0000 7fa84ec00640 1 --2- 172.21.15.47:0/2032331590 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8480cf670 0x7fa848141ae0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa83000c4d0 tx=0x7fa83000c990 comp rx=0 tx=0).stop 2024-02-19T15:26:04.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.644+0000 7fa84ec00640 1 -- 172.21.15.47:0/2032331590 shutdown_connections 2024-02-19T15:26:04.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.644+0000 7fa84ec00640 1 --2- 172.21.15.47:0/2032331590 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa824066e40 0x7fa824069300 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.644+0000 7fa84ec00640 1 --2- 172.21.15.47:0/2032331590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa8481456a0 0x7fa848142020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.644+0000 7fa84ec00640 1 --2- 172.21.15.47:0/2032331590 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8480cf670 0x7fa848141ae0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.644+0000 7fa84ec00640 1 --2- 172.21.15.47:0/2032331590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa84806b5c0 0x7fa848157f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.644+0000 7fa84ec00640 1 -- 172.21.15.47:0/2032331590 >> 172.21.15.47:0/2032331590 conn(0x7fa8480d0ad0 msgr2=0x7fa8480b19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:04.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.644+0000 7fa84ec00640 1 -- 172.21.15.47:0/2032331590 shutdown_connections 2024-02-19T15:26:04.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.648+0000 7fa84ec00640 1 -- 172.21.15.47:0/2032331590 wait complete. 2024-02-19T15:26:04.669 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-02-19T15:26:03.204094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":131,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262620,"kb_used_data":8992,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754712100,"statfs":{"total":773094113280,"available":772825190400,"internally_reserved":0,"allocated":9207808,"data_stored":4550148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":259575346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.083940"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147297+0000","last_change":"2024-02-19T15:26:00.141172+0000","last_active":"2024-02-19T15:26:01.147297+0000","last_peered":"2024-02-19T15:26:01.147297+0000","last_clean":"2024-02-19T15:26:01.147297+0000","last_became_active":"2024-02-19T15:26:00.139831+0000","last_became_peered":"2024-02-19T15:26:00.139831+0000","last_unstale":"2024-02-19T15:26:01.147297+0000","last_undegraded":"2024-02-19T15:26:01.147297+0000","last_fullsized":"2024-02-19T15:26:01.147297+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:50:34.796433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147043+0000","last_change":"2024-02-19T15:26:00.149926+0000","last_active":"2024-02-19T15:26:01.147043+0000","last_peered":"2024-02-19T15:26:01.147043+0000","last_clean":"2024-02-19T15:26:01.147043+0000","last_became_active":"2024-02-19T15:26:00.149550+0000","last_became_peered":"2024-02-19T15:26:00.149550+0000","last_unstale":"2024-02-19T15:26:01.147043+0000","last_undegraded":"2024-02-19T15:26:01.147043+0000","last_fullsized":"2024-02-19T15:26:01.147043+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:44:01.635671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.149528+0000","last_change":"2024-02-19T15:26:00.144802+0000","last_active":"2024-02-19T15:26:00.149528+0000","last_peered":"2024-02-19T15:26:00.149528+0000","last_clean":"2024-02-19T15:26:00.149528+0000","last_became_active":"2024-02-19T15:26:00.144432+0000","last_became_peered":"2024-02-19T15:26:00.144432+0000","last_unstale":"2024-02-19T15:26:00.149528+0000","last_undegraded":"2024-02-19T15:26:00.149528+0000","last_fullsized":"2024-02-19T15:26:00.149528+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:22:04.556374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.161013+0000","last_change":"2024-02-19T15:26:00.159660+0000","last_active":"2024-02-19T15:26:00.161013+0000","last_peered":"2024-02-19T15:26:00.161013+0000","last_clean":"2024-02-19T15:26:00.161013+0000","last_became_active":"2024-02-19T15:26:00.159391+0000","last_became_peered":"2024-02-19T15:26:00.159391+0000","last_unstale":"2024-02-19T15:26:00.161013+0000","last_undegraded":"2024-02-19T15:26:00.161013+0000","last_fullsized":"2024-02-19T15:26:00.161013+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:21:27.210479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.159223+0000","last_change":"2024-02-19T15:26:00.158174+0000","last_active":"2024-02-19T15:26:00.159223+0000","last_peered":"2024-02-19T15:26:00.159223+0000","last_clean":"2024-02-19T15:26:00.159223+0000","last_became_active":"2024-02-19T15:26:00.157878+0000","last_became_peered":"2024-02-19T15:26:00.157878+0000","last_unstale":"2024-02-19T15:26:00.159223+0000","last_undegraded":"2024-02-19T15:26:00.159223+0000","last_fullsized":"2024-02-19T15:26:00.159223+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:11:57.737618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.145831+0000","last_change":"2024-02-19T15:26:00.145481+0000","last_active":"2024-02-19T15:26:01.145831+0000","last_peered":"2024-02-19T15:26:01.145831+0000","last_clean":"2024-02-19T15:26:01.145831+0000","last_became_active":"2024-02-19T15:26:00.145091+0000","last_became_peered":"2024-02-19T15:26:00.145091+0000","last_unstale":"2024-02-19T15:26:01.145831+0000","last_undegraded":"2024-02-19T15:26:01.145831+0000","last_fullsized":"2024-02-19T15:26:01.145831+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:56:56.303004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.146690+0000","last_change":"2024-02-19T15:26:00.142559+0000","last_active":"2024-02-19T15:26:00.146690+0000","last_peered":"2024-02-19T15:26:00.146690+0000","last_clean":"2024-02-19T15:26:00.146690+0000","last_became_active":"2024-02-19T15:26:00.142213+0000","last_became_peered":"2024-02-19T15:26:00.142213+0000","last_unstale":"2024-02-19T15:26:00.146690+0000","last_undegraded":"2024-02-19T15:26:00.146690+0000","last_fullsized":"2024-02-19T15:26:00.146690+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:59:54.877515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.145800+0000","last_change":"2024-02-19T15:26:00.165638+0000","last_active":"2024-02-19T15:26:01.145800+0000","last_peered":"2024-02-19T15:26:01.145800+0000","last_clean":"2024-02-19T15:26:01.145800+0000","last_became_active":"2024-02-19T15:26:00.165342+0000","last_became_peered":"2024-02-19T15:26:00.165342+0000","last_unstale":"2024-02-19T15:26:01.145800+0000","last_undegraded":"2024-02-19T15:26:01.145800+0000","last_fullsized":"2024-02-19T15:26:01.145800+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:17:42.181002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146321+0000","last_change":"2024-02-19T15:26:00.156900+0000","last_active":"2024-02-19T15:26:01.146321+0000","last_peered":"2024-02-19T15:26:01.146321+0000","last_clean":"2024-02-19T15:26:01.146321+0000","last_became_active":"2024-02-19T15:26:00.156492+0000","last_became_peered":"2024-02-19T15:26:00.156492+0000","last_unstale":"2024-02-19T15:26:01.146321+0000","last_undegraded":"2024-02-19T15:26:01.146321+0000","last_fullsized":"2024-02-19T15:26:01.146321+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:49:38.953872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147763+0000","last_change":"2024-02-19T15:26:00.153441+0000","last_active":"2024-02-19T15:26:01.147763+0000","last_peered":"2024-02-19T15:26:01.147763+0000","last_clean":"2024-02-19T15:26:01.147763+0000","last_became_active":"2024-02-19T15:26:00.153179+0000","last_became_peered":"2024-02-19T15:26:00.153179+0000","last_unstale":"2024-02-19T15:26:01.147763+0000","last_undegraded":"2024-02-19T15:26:01.147763+0000","last_fullsized":"2024-02-19T15:26:01.147763+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:38:59.987721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146927+0000","last_change":"2024-02-19T15:26:00.143083+0000","last_active":"2024-02-19T15:26:01.146927+0000","last_peered":"2024-02-19T15:26:01.146927+0000","last_clean":"2024-02-19T15:26:01.146927+0000","last_became_active":"2024-02-19T15:26:00.142790+0000","last_became_peered":"2024-02-19T15:26:00.142790+0000","last_unstale":"2024-02-19T15:26:01.146927+0000","last_undegraded":"2024-02-19T15:26:01.146927+0000","last_fullsized":"2024-02-19T15:26:01.146927+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:11:57.737618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147111+0000","last_change":"2024-02-19T15:26:00.154101+0000","last_active":"2024-02-19T15:26:01.147111+0000","last_peered":"2024-02-19T15:26:01.147111+0000","last_clean":"2024-02-19T15:26:01.147111+0000","last_became_active":"2024-02-19T15:26:00.153705+0000","last_became_peered":"2024-02-19T15:26:00.153705+0000","last_unstale":"2024-02-19T15:26:01.147111+0000","last_undegraded":"2024-02-19T15:26:01.147111+0000","last_fullsized":"2024-02-19T15:26:01.147111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:04:14.206461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.151614+0000","last_change":"2024-02-19T15:26:00.147347+0000","last_active":"2024-02-19T15:26:00.151614+0000","last_peered":"2024-02-19T15:26:00.151614+0000","last_clean":"2024-02-19T15:26:00.151614+0000","last_became_active":"2024-02-19T15:26:00.147068+0000","last_became_peered":"2024-02-19T15:26:00.147068+0000","last_unstale":"2024-02-19T15:26:00.151614+0000","last_undegraded":"2024-02-19T15:26:00.151614+0000","last_fullsized":"2024-02-19T15:26:00.151614+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:09:50.208629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.163132+0000","last_change":"2024-02-19T15:26:00.161568+0000","last_active":"2024-02-19T15:26:00.163132+0000","last_peered":"2024-02-19T15:26:00.163132+0000","last_clean":"2024-02-19T15:26:00.163132+0000","last_became_active":"2024-02-19T15:26:00.161315+0000","last_became_peered":"2024-02-19T15:26:00.161315+0000","last_unstale":"2024-02-19T15:26:00.163132+0000","last_undegraded":"2024-02-19T15:26:00.163132+0000","last_fullsized":"2024-02-19T15:26:00.163132+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:31:27.968562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.150229+0000","last_change":"2024-02-19T15:26:00.146394+0000","last_active":"2024-02-19T15:26:00.150229+0000","last_peered":"2024-02-19T15:26:00.150229+0000","last_clean":"2024-02-19T15:26:00.150229+0000","last_became_active":"2024-02-19T15:26:00.146064+0000","last_became_peered":"2024-02-19T15:26:00.146064+0000","last_unstale":"2024-02-19T15:26:00.150229+0000","last_undegraded":"2024-02-19T15:26:00.150229+0000","last_fullsized":"2024-02-19T15:26:00.150229+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:31:52.766037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.150344+0000","last_change":"2024-02-19T15:26:00.140780+0000","last_active":"2024-02-19T15:26:00.150344+0000","last_peered":"2024-02-19T15:26:00.150344+0000","last_clean":"2024-02-19T15:26:00.150344+0000","last_became_active":"2024-02-19T15:26:00.140313+0000","last_became_peered":"2024-02-19T15:26:00.140313+0000","last_unstale":"2024-02-19T15:26:00.150344+0000","last_undegraded":"2024-02-19T15:26:00.150344+0000","last_fullsized":"2024-02-19T15:26:00.150344+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:29:22.170656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.149621+0000","last_change":"2024-02-19T15:26:00.146625+0000","last_active":"2024-02-19T15:26:00.149621+0000","last_peered":"2024-02-19T15:26:00.149621+0000","last_clean":"2024-02-19T15:26:00.149621+0000","last_became_active":"2024-02-19T15:26:00.146280+0000","last_became_peered":"2024-02-19T15:26:00.146280+0000","last_unstale":"2024-02-19T15:26:00.149621+0000","last_undegraded":"2024-02-19T15:26:00.149621+0000","last_fullsized":"2024-02-19T15:26:00.149621+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:25:06.707641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.148998+0000","last_change":"2024-02-19T15:26:00.145504+0000","last_active":"2024-02-19T15:26:00.148998+0000","last_peered":"2024-02-19T15:26:00.148998+0000","last_clean":"2024-02-19T15:26:00.148998+0000","last_became_active":"2024-02-19T15:26:00.145240+0000","last_became_peered":"2024-02-19T15:26:00.145240+0000","last_unstale":"2024-02-19T15:26:00.148998+0000","last_undegraded":"2024-02-19T15:26:00.148998+0000","last_fullsized":"2024-02-19T15:26:00.148998+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:39:15.242573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.164463+0000","last_change":"2024-02-19T15:26:00.151957+0000","last_active":"2024-02-19T15:26:00.164463+0000","last_peered":"2024-02-19T15:26:00.164463+0000","last_clean":"2024-02-19T15:26:00.164463+0000","last_became_active":"2024-02-19T15:26:00.151700+0000","last_became_peered":"2024-02-19T15:26:00.151700+0000","last_unstale":"2024-02-19T15:26:00.164463+0000","last_undegraded":"2024-02-19T15:26:00.164463+0000","last_fullsized":"2024-02-19T15:26:00.164463+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:50:40.234904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.164718+0000","last_change":"2024-02-19T15:26:00.163579+0000","last_active":"2024-02-19T15:26:00.164718+0000","last_peered":"2024-02-19T15:26:00.164718+0000","last_clean":"2024-02-19T15:26:00.164718+0000","last_became_active":"2024-02-19T15:26:00.163018+0000","last_became_peered":"2024-02-19T15:26:00.163018+0000","last_unstale":"2024-02-19T15:26:00.164718+0000","last_undegraded":"2024-02-19T15:26:00.164718+0000","last_fullsized":"2024-02-19T15:26:00.164718+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:57:16.434575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.159440+0000","last_change":"2024-02-19T15:26:00.154028+0000","last_active":"2024-02-19T15:26:00.159440+0000","last_peered":"2024-02-19T15:26:00.159440+0000","last_clean":"2024-02-19T15:26:00.159440+0000","last_became_active":"2024-02-19T15:26:00.151254+0000","last_became_peered":"2024-02-19T15:26:00.151254+0000","last_unstale":"2024-02-19T15:26:00.159440+0000","last_undegraded":"2024-02-19T15:26:00.159440+0000","last_fullsized":"2024-02-19T15:26:00.159440+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:32:09.736412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146476+0000","last_change":"2024-02-19T15:26:00.167151+0000","last_active":"2024-02-19T15:26:01.146476+0000","last_peered":"2024-02-19T15:26:01.146476+0000","last_clean":"2024-02-19T15:26:01.146476+0000","last_became_active":"2024-02-19T15:26:00.166846+0000","last_became_peered":"2024-02-19T15:26:00.166846+0000","last_unstale":"2024-02-19T15:26:01.146476+0000","last_undegraded":"2024-02-19T15:26:01.146476+0000","last_fullsized":"2024-02-19T15:26:01.146476+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:50.581881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147359+0000","last_change":"2024-02-19T15:26:00.158679+0000","last_active":"2024-02-19T15:26:01.147359+0000","last_peered":"2024-02-19T15:26:01.147359+0000","last_clean":"2024-02-19T15:26:01.147359+0000","last_became_active":"2024-02-19T15:26:00.158412+0000","last_became_peered":"2024-02-19T15:26:00.158412+0000","last_unstale":"2024-02-19T15:26:01.147359+0000","last_undegraded":"2024-02-19T15:26:01.147359+0000","last_fullsized":"2024-02-19T15:26:01.147359+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:48:58.609617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.161058+0000","last_change":"2024-02-19T15:26:00.160327+0000","last_active":"2024-02-19T15:26:00.161058+0000","last_peered":"2024-02-19T15:26:00.161058+0000","last_clean":"2024-02-19T15:26:00.161058+0000","last_became_active":"2024-02-19T15:26:00.160047+0000","last_became_peered":"2024-02-19T15:26:00.160047+0000","last_unstale":"2024-02-19T15:26:00.161058+0000","last_undegraded":"2024-02-19T15:26:00.161058+0000","last_fullsized":"2024-02-19T15:26:00.161058+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:12:50.146790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146969+0000","last_change":"2024-02-19T15:26:00.156835+0000","last_active":"2024-02-19T15:26:01.146969+0000","last_peered":"2024-02-19T15:26:01.146969+0000","last_clean":"2024-02-19T15:26:01.146969+0000","last_became_active":"2024-02-19T15:26:00.156598+0000","last_became_peered":"2024-02-19T15:26:00.156598+0000","last_unstale":"2024-02-19T15:26:01.146969+0000","last_undegraded":"2024-02-19T15:26:01.146969+0000","last_fullsized":"2024-02-19T15:26:01.146969+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:14:01.639589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.162544+0000","last_change":"2024-02-19T15:26:00.161536+0000","last_active":"2024-02-19T15:26:00.162544+0000","last_peered":"2024-02-19T15:26:00.162544+0000","last_clean":"2024-02-19T15:26:00.162544+0000","last_became_active":"2024-02-19T15:26:00.161224+0000","last_became_peered":"2024-02-19T15:26:00.161224+0000","last_unstale":"2024-02-19T15:26:00.162544+0000","last_undegraded":"2024-02-19T15:26:00.162544+0000","last_fullsized":"2024-02-19T15:26:00.162544+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:31:52.766037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.151205+0000","last_change":"2024-02-19T15:26:00.150179+0000","last_active":"2024-02-19T15:26:00.151205+0000","last_peered":"2024-02-19T15:26:00.151205+0000","last_clean":"2024-02-19T15:26:00.151205+0000","last_became_active":"2024-02-19T15:26:00.149900+0000","last_became_peered":"2024-02-19T15:26:00.149900+0000","last_unstale":"2024-02-19T15:26:00.151205+0000","last_undegraded":"2024-02-19T15:26:00.151205+0000","last_fullsized":"2024-02-19T15:26:00.151205+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:08:32.787841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147269+0000","last_change":"2024-02-19T15:26:00.138814+0000","last_active":"2024-02-19T15:26:01.147269+0000","last_peered":"2024-02-19T15:26:01.147269+0000","last_clean":"2024-02-19T15:26:01.147269+0000","last_became_active":"2024-02-19T15:26:00.138494+0000","last_became_peered":"2024-02-19T15:26:00.138494+0000","last_unstale":"2024-02-19T15:26:01.147269+0000","last_undegraded":"2024-02-19T15:26:01.147269+0000","last_fullsized":"2024-02-19T15:26:01.147269+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:11.778091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146505+0000","last_change":"2024-02-19T15:26:00.157251+0000","last_active":"2024-02-19T15:26:01.146505+0000","last_peered":"2024-02-19T15:26:01.146505+0000","last_clean":"2024-02-19T15:26:01.146505+0000","last_became_active":"2024-02-19T15:26:00.156992+0000","last_became_peered":"2024-02-19T15:26:00.156992+0000","last_unstale":"2024-02-19T15:26:01.146505+0000","last_undegraded":"2024-02-19T15:26:01.146505+0000","last_fullsized":"2024-02-19T15:26:01.146505+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:02:22.168242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146533+0000","last_change":"2024-02-19T15:26:00.147939+0000","last_active":"2024-02-19T15:26:01.146533+0000","last_peered":"2024-02-19T15:26:01.146533+0000","last_clean":"2024-02-19T15:26:01.146533+0000","last_became_active":"2024-02-19T15:26:00.147682+0000","last_became_peered":"2024-02-19T15:26:00.147682+0000","last_unstale":"2024-02-19T15:26:01.146533+0000","last_undegraded":"2024-02-19T15:26:01.146533+0000","last_fullsized":"2024-02-19T15:26:01.146533+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:44.769088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.158014+0000","last_change":"2024-02-19T15:26:00.155419+0000","last_active":"2024-02-19T15:26:00.158014+0000","last_peered":"2024-02-19T15:26:00.158014+0000","last_clean":"2024-02-19T15:26:00.158014+0000","last_became_active":"2024-02-19T15:26:00.155080+0000","last_became_peered":"2024-02-19T15:26:00.155080+0000","last_unstale":"2024-02-19T15:26:00.158014+0000","last_undegraded":"2024-02-19T15:26:00.158014+0000","last_fullsized":"2024-02-19T15:26:00.158014+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:09:12.862734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146893+0000","last_change":"2024-02-19T15:26:00.158776+0000","last_active":"2024-02-19T15:26:01.146893+0000","last_peered":"2024-02-19T15:26:01.146893+0000","last_clean":"2024-02-19T15:26:01.146893+0000","last_became_active":"2024-02-19T15:26:00.157244+0000","last_became_peered":"2024-02-19T15:26:00.157244+0000","last_unstale":"2024-02-19T15:26:01.146893+0000","last_undegraded":"2024-02-19T15:26:01.146893+0000","last_fullsized":"2024-02-19T15:26:01.146893+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:49:11.620738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.149002+0000","last_change":"2024-02-19T15:26:00.143469+0000","last_active":"2024-02-19T15:26:00.149002+0000","last_peered":"2024-02-19T15:26:00.149002+0000","last_clean":"2024-02-19T15:26:00.149002+0000","last_became_active":"2024-02-19T15:26:00.143199+0000","last_became_peered":"2024-02-19T15:26:00.143199+0000","last_unstale":"2024-02-19T15:26:00.149002+0000","last_undegraded":"2024-02-19T15:26:00.149002+0000","last_fullsized":"2024-02-19T15:26:00.149002+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:02:22.168242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"81'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147478+0000","last_change":"2024-02-19T15:25:59.138426+0000","last_active":"2024-02-19T15:26:01.147478+0000","last_peered":"2024-02-19T15:26:01.147478+0000","last_clean":"2024-02-19T15:26:01.147478+0000","last_became_active":"2024-02-19T15:25:59.138137+0000","last_became_peered":"2024-02-19T15:25:59.138137+0000","last_unstale":"2024-02-19T15:26:01.147478+0000","last_undegraded":"2024-02-19T15:26:01.147478+0000","last_fullsized":"2024-02-19T15:26:01.147478+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:00:31.176094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147379+0000","last_change":"2024-02-19T15:26:00.158856+0000","last_active":"2024-02-19T15:26:01.147379+0000","last_peered":"2024-02-19T15:26:01.147379+0000","last_clean":"2024-02-19T15:26:01.147379+0000","last_became_active":"2024-02-19T15:26:00.158374+0000","last_became_peered":"2024-02-19T15:26:00.158374+0000","last_unstale":"2024-02-19T15:26:01.147379+0000","last_undegraded":"2024-02-19T15:26:01.147379+0000","last_fullsized":"2024-02-19T15:26:01.147379+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:50:40.234904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.160658+0000","last_change":"2024-02-19T15:25:59.140196+0000","last_active":"2024-02-19T15:26:00.160658+0000","last_peered":"2024-02-19T15:26:00.160658+0000","last_clean":"2024-02-19T15:26:00.160658+0000","last_became_active":"2024-02-19T15:25:59.139892+0000","last_became_peered":"2024-02-19T15:25:59.139892+0000","last_unstale":"2024-02-19T15:26:00.160658+0000","last_undegraded":"2024-02-19T15:26:00.160658+0000","last_fullsized":"2024-02-19T15:26:00.160658+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:37:26.597887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147289+0000","last_change":"2024-02-19T15:26:00.161341+0000","last_active":"2024-02-19T15:26:01.147289+0000","last_peered":"2024-02-19T15:26:01.147289+0000","last_clean":"2024-02-19T15:26:01.147289+0000","last_became_active":"2024-02-19T15:26:00.160999+0000","last_became_peered":"2024-02-19T15:26:00.160999+0000","last_unstale":"2024-02-19T15:26:01.147289+0000","last_undegraded":"2024-02-19T15:26:01.147289+0000","last_fullsized":"2024-02-19T15:26:01.147289+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:28:17.733970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"81'2","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.165835+0000","last_change":"2024-02-19T15:25:59.150116+0000","last_active":"2024-02-19T15:26:00.165835+0000","last_peered":"2024-02-19T15:26:00.165835+0000","last_clean":"2024-02-19T15:26:00.165835+0000","last_became_active":"2024-02-19T15:25:59.149815+0000","last_became_peered":"2024-02-19T15:25:59.149815+0000","last_unstale":"2024-02-19T15:26:00.165835+0000","last_undegraded":"2024-02-19T15:26:00.165835+0000","last_fullsized":"2024-02-19T15:26:00.165835+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:04:13.199124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.149506+0000","last_change":"2024-02-19T15:26:00.144350+0000","last_active":"2024-02-19T15:26:00.149506+0000","last_peered":"2024-02-19T15:26:00.149506+0000","last_clean":"2024-02-19T15:26:00.149506+0000","last_became_active":"2024-02-19T15:26:00.142102+0000","last_became_peered":"2024-02-19T15:26:00.142102+0000","last_unstale":"2024-02-19T15:26:00.149506+0000","last_undegraded":"2024-02-19T15:26:00.149506+0000","last_fullsized":"2024-02-19T15:26:00.149506+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:12:56.043581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"81'2","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.165584+0000","last_change":"2024-02-19T15:25:59.150705+0000","last_active":"2024-02-19T15:26:00.165584+0000","last_peered":"2024-02-19T15:26:00.165584+0000","last_clean":"2024-02-19T15:26:00.165584+0000","last_became_active":"2024-02-19T15:25:59.150301+0000","last_became_peered":"2024-02-19T15:25:59.150301+0000","last_unstale":"2024-02-19T15:26:00.165584+0000","last_undegraded":"2024-02-19T15:26:00.165584+0000","last_fullsized":"2024-02-19T15:26:00.165584+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:26:00.767871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146631+0000","last_change":"2024-02-19T15:26:00.147938+0000","last_active":"2024-02-19T15:26:01.146631+0000","last_peered":"2024-02-19T15:26:01.146631+0000","last_clean":"2024-02-19T15:26:01.146631+0000","last_became_active":"2024-02-19T15:26:00.147671+0000","last_became_peered":"2024-02-19T15:26:00.147671+0000","last_unstale":"2024-02-19T15:26:01.146631+0000","last_undegraded":"2024-02-19T15:26:01.146631+0000","last_fullsized":"2024-02-19T15:26:01.146631+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:08:32.787841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.165175+0000","last_change":"2024-02-19T15:25:59.139252+0000","last_active":"2024-02-19T15:26:00.165175+0000","last_peered":"2024-02-19T15:26:00.165175+0000","last_clean":"2024-02-19T15:26:00.165175+0000","last_became_active":"2024-02-19T15:25:59.138894+0000","last_became_peered":"2024-02-19T15:25:59.138894+0000","last_unstale":"2024-02-19T15:26:00.165175+0000","last_undegraded":"2024-02-19T15:26:00.165175+0000","last_fullsized":"2024-02-19T15:26:00.165175+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:04:47.587059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.147800+0000","last_change":"2024-02-19T15:26:00.142832+0000","last_active":"2024-02-19T15:26:00.147800+0000","last_peered":"2024-02-19T15:26:00.147800+0000","last_clean":"2024-02-19T15:26:00.147800+0000","last_became_active":"2024-02-19T15:26:00.141435+0000","last_became_peered":"2024-02-19T15:26:00.141435+0000","last_unstale":"2024-02-19T15:26:00.147800+0000","last_undegraded":"2024-02-19T15:26:00.147800+0000","last_fullsized":"2024-02-19T15:26:00.147800+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:50.581881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.160629+0000","last_change":"2024-02-19T15:25:59.141148+0000","last_active":"2024-02-19T15:26:00.160629+0000","last_peered":"2024-02-19T15:26:00.160629+0000","last_clean":"2024-02-19T15:26:00.160629+0000","last_became_active":"2024-02-19T15:25:59.140799+0000","last_became_peered":"2024-02-19T15:25:59.140799+0000","last_unstale":"2024-02-19T15:26:00.160629+0000","last_undegraded":"2024-02-19T15:26:00.160629+0000","last_fullsized":"2024-02-19T15:26:00.160629+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:03:25.527833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.164084+0000","last_change":"2024-02-19T15:26:00.163537+0000","last_active":"2024-02-19T15:26:00.164084+0000","last_peered":"2024-02-19T15:26:00.164084+0000","last_clean":"2024-02-19T15:26:00.164084+0000","last_became_active":"2024-02-19T15:26:00.163198+0000","last_became_peered":"2024-02-19T15:26:00.163198+0000","last_unstale":"2024-02-19T15:26:00.164084+0000","last_undegraded":"2024-02-19T15:26:00.164084+0000","last_fullsized":"2024-02-19T15:26:00.164084+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:05.118962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.134809+0000","last_change":"2024-02-19T15:25:59.145705+0000","last_active":"2024-02-19T15:26:00.134809+0000","last_peered":"2024-02-19T15:26:00.134809+0000","last_clean":"2024-02-19T15:26:00.134809+0000","last_became_active":"2024-02-19T15:25:59.145398+0000","last_became_peered":"2024-02-19T15:25:59.145398+0000","last_unstale":"2024-02-19T15:26:00.134809+0000","last_undegraded":"2024-02-19T15:26:00.134809+0000","last_fullsized":"2024-02-19T15:26:00.134809+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:46:04.901935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.150207+0000","last_change":"2024-02-19T15:26:00.146267+0000","last_active":"2024-02-19T15:26:00.150207+0000","last_peered":"2024-02-19T15:26:00.150207+0000","last_clean":"2024-02-19T15:26:00.150207+0000","last_became_active":"2024-02-19T15:26:00.146042+0000","last_became_peered":"2024-02-19T15:26:00.146042+0000","last_unstale":"2024-02-19T15:26:00.150207+0000","last_undegraded":"2024-02-19T15:26:00.150207+0000","last_fullsized":"2024-02-19T15:26:00.150207+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:06.498178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146639+0000","last_change":"2024-02-19T15:25:59.147976+0000","last_active":"2024-02-19T15:26:01.146639+0000","last_peered":"2024-02-19T15:26:01.146639+0000","last_clean":"2024-02-19T15:26:01.146639+0000","last_became_active":"2024-02-19T15:25:59.145034+0000","last_became_peered":"2024-02-19T15:25:59.145034+0000","last_unstale":"2024-02-19T15:26:01.146639+0000","last_undegraded":"2024-02-19T15:26:01.146639+0000","last_fullsized":"2024-02-19T15:26:01.146639+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:13:37.187177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.163984+0000","last_change":"2024-02-19T15:26:00.161844+0000","last_active":"2024-02-19T15:26:00.163984+0000","last_peered":"2024-02-19T15:26:00.163984+0000","last_clean":"2024-02-19T15:26:00.163984+0000","last_became_active":"2024-02-19T15:26:00.161581+0000","last_became_peered":"2024-02-19T15:26:00.161581+0000","last_unstale":"2024-02-19T15:26:00.163984+0000","last_undegraded":"2024-02-19T15:26:00.163984+0000","last_fullsized":"2024-02-19T15:26:00.163984+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:53:03.954569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147553+0000","last_change":"2024-02-19T15:25:59.144876+0000","last_active":"2024-02-19T15:26:01.147553+0000","last_peered":"2024-02-19T15:26:01.147553+0000","last_clean":"2024-02-19T15:26:01.147553+0000","last_became_active":"2024-02-19T15:25:59.144571+0000","last_became_peered":"2024-02-19T15:25:59.144571+0000","last_unstale":"2024-02-19T15:26:01.147553+0000","last_undegraded":"2024-02-19T15:26:01.147553+0000","last_fullsized":"2024-02-19T15:26:01.147553+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:00:37.738268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146304+0000","last_change":"2024-02-19T15:26:00.152580+0000","last_active":"2024-02-19T15:26:01.146304+0000","last_peered":"2024-02-19T15:26:01.146304+0000","last_clean":"2024-02-19T15:26:01.146304+0000","last_became_active":"2024-02-19T15:26:00.151404+0000","last_became_peered":"2024-02-19T15:26:00.151404+0000","last_unstale":"2024-02-19T15:26:01.146304+0000","last_undegraded":"2024-02-19T15:26:01.146304+0000","last_fullsized":"2024-02-19T15:26:01.146304+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:32:55.515920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147776+0000","last_change":"2024-02-19T15:25:59.138603+0000","last_active":"2024-02-19T15:26:01.147776+0000","last_peered":"2024-02-19T15:26:01.147776+0000","last_clean":"2024-02-19T15:26:01.147776+0000","last_became_active":"2024-02-19T15:25:59.138324+0000","last_became_peered":"2024-02-19T15:25:59.138324+0000","last_unstale":"2024-02-19T15:26:01.147776+0000","last_undegraded":"2024-02-19T15:26:01.147776+0000","last_fullsized":"2024-02-19T15:26:01.147776+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:21:21.944376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.165637+0000","last_change":"2024-02-19T15:26:00.153959+0000","last_active":"2024-02-19T15:26:00.165637+0000","last_peered":"2024-02-19T15:26:00.165637+0000","last_clean":"2024-02-19T15:26:00.165637+0000","last_became_active":"2024-02-19T15:26:00.153667+0000","last_became_peered":"2024-02-19T15:26:00.153667+0000","last_unstale":"2024-02-19T15:26:00.165637+0000","last_undegraded":"2024-02-19T15:26:00.165637+0000","last_fullsized":"2024-02-19T15:26:00.165637+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:05:34.715231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"81'3","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147149+0000","last_change":"2024-02-19T15:25:59.147818+0000","last_active":"2024-02-19T15:26:01.147149+0000","last_peered":"2024-02-19T15:26:01.147149+0000","last_clean":"2024-02-19T15:26:01.147149+0000","last_became_active":"2024-02-19T15:25:59.147481+0000","last_became_peered":"2024-02-19T15:25:59.147481+0000","last_unstale":"2024-02-19T15:26:01.147149+0000","last_undegraded":"2024-02-19T15:26:01.147149+0000","last_fullsized":"2024-02-19T15:26:01.147149+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:48.954957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.163829+0000","last_change":"2024-02-19T15:26:00.163053+0000","last_active":"2024-02-19T15:26:00.163829+0000","last_peered":"2024-02-19T15:26:00.163829+0000","last_clean":"2024-02-19T15:26:00.163829+0000","last_became_active":"2024-02-19T15:26:00.162665+0000","last_became_peered":"2024-02-19T15:26:00.162665+0000","last_unstale":"2024-02-19T15:26:00.163829+0000","last_undegraded":"2024-02-19T15:26:00.163829+0000","last_fullsized":"2024-02-19T15:26:00.163829+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:43:15.055359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"81'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146113+0000","last_change":"2024-02-19T15:25:59.140299+0000","last_active":"2024-02-19T15:26:01.146113+0000","last_peered":"2024-02-19T15:26:01.146113+0000","last_clean":"2024-02-19T15:26:01.146113+0000","last_became_active":"2024-02-19T15:25:59.139999+0000","last_became_peered":"2024-02-19T15:25:59.139999+0000","last_unstale":"2024-02-19T15:26:01.146113+0000","last_undegraded":"2024-02-19T15:26:01.146113+0000","last_fullsized":"2024-02-19T15:26:01.146113+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:35:45.839506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.158852+0000","last_change":"2024-02-19T15:26:00.149867+0000","last_active":"2024-02-19T15:26:00.158852+0000","last_peered":"2024-02-19T15:26:00.158852+0000","last_clean":"2024-02-19T15:26:00.158852+0000","last_became_active":"2024-02-19T15:26:00.149559+0000","last_became_peered":"2024-02-19T15:26:00.149559+0000","last_unstale":"2024-02-19T15:26:00.158852+0000","last_undegraded":"2024-02-19T15:26:00.158852+0000","last_fullsized":"2024-02-19T15:26:00.158852+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:28:17.733970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146550+0000","last_change":"2024-02-19T15:25:59.147891+0000","last_active":"2024-02-19T15:26:01.146550+0000","last_peered":"2024-02-19T15:26:01.146550+0000","last_clean":"2024-02-19T15:26:01.146550+0000","last_became_active":"2024-02-19T15:25:59.147497+0000","last_became_peered":"2024-02-19T15:25:59.147497+0000","last_unstale":"2024-02-19T15:26:01.146550+0000","last_undegraded":"2024-02-19T15:26:01.146550+0000","last_fullsized":"2024-02-19T15:26:01.146550+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:34:18.199529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.153046+0000","last_change":"2024-02-19T15:26:00.151252+0000","last_active":"2024-02-19T15:26:00.153046+0000","last_peered":"2024-02-19T15:26:00.153046+0000","last_clean":"2024-02-19T15:26:00.153046+0000","last_became_active":"2024-02-19T15:26:00.150931+0000","last_became_peered":"2024-02-19T15:26:00.150931+0000","last_unstale":"2024-02-19T15:26:00.153046+0000","last_undegraded":"2024-02-19T15:26:00.153046+0000","last_fullsized":"2024-02-19T15:26:00.153046+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:06.498178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.133589+0000","last_change":"2024-02-19T15:25:59.141154+0000","last_active":"2024-02-19T15:26:00.133589+0000","last_peered":"2024-02-19T15:26:00.133589+0000","last_clean":"2024-02-19T15:26:00.133589+0000","last_became_active":"2024-02-19T15:25:59.140880+0000","last_became_peered":"2024-02-19T15:25:59.140880+0000","last_unstale":"2024-02-19T15:26:00.133589+0000","last_undegraded":"2024-02-19T15:26:00.133589+0000","last_fullsized":"2024-02-19T15:26:00.133589+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:49:18.087693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146542+0000","last_change":"2024-02-19T15:26:00.156369+0000","last_active":"2024-02-19T15:26:01.146542+0000","last_peered":"2024-02-19T15:26:01.146542+0000","last_clean":"2024-02-19T15:26:01.146542+0000","last_became_active":"2024-02-19T15:26:00.156127+0000","last_became_peered":"2024-02-19T15:26:00.156127+0000","last_unstale":"2024-02-19T15:26:01.146542+0000","last_undegraded":"2024-02-19T15:26:01.146542+0000","last_fullsized":"2024-02-19T15:26:01.146542+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:25:06.707641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.145948+0000","last_change":"2024-02-19T15:25:59.141165+0000","last_active":"2024-02-19T15:26:01.145948+0000","last_peered":"2024-02-19T15:26:01.145948+0000","last_clean":"2024-02-19T15:26:01.145948+0000","last_became_active":"2024-02-19T15:25:59.140902+0000","last_became_peered":"2024-02-19T15:25:59.140902+0000","last_unstale":"2024-02-19T15:26:01.145948+0000","last_undegraded":"2024-02-19T15:26:01.145948+0000","last_fullsized":"2024-02-19T15:26:01.145948+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:28:35.694362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146422+0000","last_change":"2024-02-19T15:26:00.142607+0000","last_active":"2024-02-19T15:26:01.146422+0000","last_peered":"2024-02-19T15:26:01.146422+0000","last_clean":"2024-02-19T15:26:01.146422+0000","last_became_active":"2024-02-19T15:26:00.142307+0000","last_became_peered":"2024-02-19T15:26:00.142307+0000","last_unstale":"2024-02-19T15:26:01.146422+0000","last_undegraded":"2024-02-19T15:26:01.146422+0000","last_fullsized":"2024-02-19T15:26:01.146422+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:29:22.170656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"81'8","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.167052+0000","last_change":"2024-02-19T15:25:59.142789+0000","last_active":"2024-02-19T15:26:00.167052+0000","last_peered":"2024-02-19T15:26:00.167052+0000","last_clean":"2024-02-19T15:26:00.167052+0000","last_became_active":"2024-02-19T15:25:59.142534+0000","last_became_peered":"2024-02-19T15:25:59.142534+0000","last_unstale":"2024-02-19T15:26:00.167052+0000","last_undegraded":"2024-02-19T15:26:00.167052+0000","last_fullsized":"2024-02-19T15:26:00.167052+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:48:57.602279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.150840+0000","last_change":"2024-02-19T15:26:00.145781+0000","last_active":"2024-02-19T15:26:00.150840+0000","last_peered":"2024-02-19T15:26:00.150840+0000","last_clean":"2024-02-19T15:26:00.150840+0000","last_became_active":"2024-02-19T15:26:00.145451+0000","last_became_peered":"2024-02-19T15:26:00.145451+0000","last_unstale":"2024-02-19T15:26:00.150840+0000","last_undegraded":"2024-02-19T15:26:00.150840+0000","last_fullsized":"2024-02-19T15:26:00.150840+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:46:58.907351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147605+0000","last_change":"2024-02-19T15:25:59.138247+0000","last_active":"2024-02-19T15:26:01.147605+0000","last_peered":"2024-02-19T15:26:01.147605+0000","last_clean":"2024-02-19T15:26:01.147605+0000","last_became_active":"2024-02-19T15:25:59.137938+0000","last_became_peered":"2024-02-19T15:25:59.137938+0000","last_unstale":"2024-02-19T15:26:01.147605+0000","last_undegraded":"2024-02-19T15:26:01.147605+0000","last_fullsized":"2024-02-19T15:26:01.147605+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:17:22.547059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147449+0000","last_change":"2024-02-19T15:26:00.142334+0000","last_active":"2024-02-19T15:26:01.147449+0000","last_peered":"2024-02-19T15:26:01.147449+0000","last_clean":"2024-02-19T15:26:01.147449+0000","last_became_active":"2024-02-19T15:26:00.142028+0000","last_became_peered":"2024-02-19T15:26:00.142028+0000","last_unstale":"2024-02-19T15:26:01.147449+0000","last_undegraded":"2024-02-19T15:26:01.147449+0000","last_fullsized":"2024-02-19T15:26:01.147449+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:05:34.715231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"81'2","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.162160+0000","last_change":"2024-02-19T15:25:59.145572+0000","last_active":"2024-02-19T15:26:00.162160+0000","last_peered":"2024-02-19T15:26:00.162160+0000","last_clean":"2024-02-19T15:26:00.162160+0000","last_became_active":"2024-02-19T15:25:59.145289+0000","last_became_peered":"2024-02-19T15:25:59.145289+0000","last_unstale":"2024-02-19T15:26:00.162160+0000","last_undegraded":"2024-02-19T15:26:00.162160+0000","last_fullsized":"2024-02-19T15:26:00.162160+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:41:41.452759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.162865+0000","last_change":"2024-02-19T15:26:00.161606+0000","last_active":"2024-02-19T15:26:00.162865+0000","last_peered":"2024-02-19T15:26:00.162865+0000","last_clean":"2024-02-19T15:26:00.162865+0000","last_became_active":"2024-02-19T15:26:00.161324+0000","last_became_peered":"2024-02-19T15:26:00.161324+0000","last_unstale":"2024-02-19T15:26:00.162865+0000","last_undegraded":"2024-02-19T15:26:00.162865+0000","last_fullsized":"2024-02-19T15:26:00.162865+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:14:01.639589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.141483+0000","last_change":"2024-02-19T15:25:59.143481+0000","last_active":"2024-02-19T15:26:00.141483+0000","last_peered":"2024-02-19T15:26:00.141483+0000","last_clean":"2024-02-19T15:26:00.141483+0000","last_became_active":"2024-02-19T15:25:59.143199+0000","last_became_peered":"2024-02-19T15:25:59.143199+0000","last_unstale":"2024-02-19T15:26:00.141483+0000","last_undegraded":"2024-02-19T15:26:00.141483+0000","last_fullsized":"2024-02-19T15:26:00.141483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:10.770753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.163839+0000","last_change":"2024-02-19T15:26:00.161744+0000","last_active":"2024-02-19T15:26:00.163839+0000","last_peered":"2024-02-19T15:26:00.163839+0000","last_clean":"2024-02-19T15:26:00.163839+0000","last_became_active":"2024-02-19T15:26:00.161435+0000","last_became_peered":"2024-02-19T15:26:00.161435+0000","last_unstale":"2024-02-19T15:26:00.163839+0000","last_undegraded":"2024-02-19T15:26:00.163839+0000","last_fullsized":"2024-02-19T15:26:00.163839+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:47.889730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"81'2","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.151623+0000","last_change":"2024-02-19T15:25:59.140721+0000","last_active":"2024-02-19T15:26:00.151623+0000","last_peered":"2024-02-19T15:26:00.151623+0000","last_clean":"2024-02-19T15:26:00.151623+0000","last_became_active":"2024-02-19T15:25:59.140434+0000","last_became_peered":"2024-02-19T15:25:59.140434+0000","last_unstale":"2024-02-19T15:26:00.151623+0000","last_undegraded":"2024-02-19T15:26:00.151623+0000","last_fullsized":"2024-02-19T15:26:00.151623+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:09:49.201292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.145898+0000","last_change":"2024-02-19T15:26:00.147849+0000","last_active":"2024-02-19T15:26:01.145898+0000","last_peered":"2024-02-19T15:26:01.145898+0000","last_clean":"2024-02-19T15:26:01.145898+0000","last_became_active":"2024-02-19T15:26:00.147500+0000","last_became_peered":"2024-02-19T15:26:00.147500+0000","last_unstale":"2024-02-19T15:26:01.145898+0000","last_undegraded":"2024-02-19T15:26:01.145898+0000","last_fullsized":"2024-02-19T15:26:01.145898+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:12:11.535536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.132922+0000","last_change":"2024-02-19T15:25:59.145297+0000","last_active":"2024-02-19T15:26:00.132922+0000","last_peered":"2024-02-19T15:26:00.132922+0000","last_clean":"2024-02-19T15:26:00.132922+0000","last_became_active":"2024-02-19T15:25:59.145047+0000","last_became_peered":"2024-02-19T15:25:59.145047+0000","last_unstale":"2024-02-19T15:26:00.132922+0000","last_undegraded":"2024-02-19T15:26:00.132922+0000","last_fullsized":"2024-02-19T15:26:00.132922+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:21.086134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.148261+0000","last_change":"2024-02-19T15:26:00.144473+0000","last_active":"2024-02-19T15:26:00.148261+0000","last_peered":"2024-02-19T15:26:00.148261+0000","last_clean":"2024-02-19T15:26:00.148261+0000","last_became_active":"2024-02-19T15:26:00.144209+0000","last_became_peered":"2024-02-19T15:26:00.144209+0000","last_unstale":"2024-02-19T15:26:00.148261+0000","last_undegraded":"2024-02-19T15:26:00.148261+0000","last_fullsized":"2024-02-19T15:26:00.148261+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:10:21.715225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.131663+0000","last_change":"2024-02-19T15:25:59.144906+0000","last_active":"2024-02-19T15:26:00.131663+0000","last_peered":"2024-02-19T15:26:00.131663+0000","last_clean":"2024-02-19T15:26:00.131663+0000","last_became_active":"2024-02-19T15:25:59.144601+0000","last_became_peered":"2024-02-19T15:25:59.144601+0000","last_unstale":"2024-02-19T15:26:00.131663+0000","last_undegraded":"2024-02-19T15:26:00.131663+0000","last_fullsized":"2024-02-19T15:26:00.131663+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:43:35.176726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146061+0000","last_change":"2024-02-19T15:26:00.156882+0000","last_active":"2024-02-19T15:26:01.146061+0000","last_peered":"2024-02-19T15:26:01.146061+0000","last_clean":"2024-02-19T15:26:01.146061+0000","last_became_active":"2024-02-19T15:26:00.156490+0000","last_became_peered":"2024-02-19T15:26:00.156490+0000","last_unstale":"2024-02-19T15:26:01.146061+0000","last_undegraded":"2024-02-19T15:26:01.146061+0000","last_fullsized":"2024-02-19T15:26:01.146061+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:31:39.275272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.133355+0000","last_change":"2024-02-19T15:25:59.140834+0000","last_active":"2024-02-19T15:26:00.133355+0000","last_peered":"2024-02-19T15:26:00.133355+0000","last_clean":"2024-02-19T15:26:00.133355+0000","last_became_active":"2024-02-19T15:25:59.140559+0000","last_became_peered":"2024-02-19T15:25:59.140559+0000","last_unstale":"2024-02-19T15:26:00.133355+0000","last_undegraded":"2024-02-19T15:26:00.133355+0000","last_fullsized":"2024-02-19T15:26:00.133355+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:46:57.900013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.149272+0000","last_change":"2024-02-19T15:26:00.146112+0000","last_active":"2024-02-19T15:26:00.149272+0000","last_peered":"2024-02-19T15:26:00.149272+0000","last_clean":"2024-02-19T15:26:00.149272+0000","last_became_active":"2024-02-19T15:26:00.143543+0000","last_became_peered":"2024-02-19T15:26:00.143543+0000","last_unstale":"2024-02-19T15:26:00.149272+0000","last_undegraded":"2024-02-19T15:26:00.149272+0000","last_fullsized":"2024-02-19T15:26:00.149272+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:44.769088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147000+0000","last_change":"2024-02-19T15:25:59.145996+0000","last_active":"2024-02-19T15:26:01.147000+0000","last_peered":"2024-02-19T15:26:01.147000+0000","last_clean":"2024-02-19T15:26:01.147000+0000","last_became_active":"2024-02-19T15:25:59.144656+0000","last_became_peered":"2024-02-19T15:25:59.144656+0000","last_unstale":"2024-02-19T15:26:01.147000+0000","last_undegraded":"2024-02-19T15:26:01.147000+0000","last_fullsized":"2024-02-19T15:26:01.147000+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:24:06.383825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.148414+0000","last_change":"2024-02-19T15:26:00.145811+0000","last_active":"2024-02-19T15:26:00.148414+0000","last_peered":"2024-02-19T15:26:00.148414+0000","last_clean":"2024-02-19T15:26:00.148414+0000","last_became_active":"2024-02-19T15:26:00.145521+0000","last_became_peered":"2024-02-19T15:26:00.145521+0000","last_unstale":"2024-02-19T15:26:00.148414+0000","last_undegraded":"2024-02-19T15:26:00.148414+0000","last_fullsized":"2024-02-19T15:26:00.148414+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:49:19.095030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146991+0000","last_change":"2024-02-19T15:25:59.147788+0000","last_active":"2024-02-19T15:26:01.146991+0000","last_peered":"2024-02-19T15:26:01.146991+0000","last_clean":"2024-02-19T15:26:01.146991+0000","last_became_active":"2024-02-19T15:25:59.147480+0000","last_became_peered":"2024-02-19T15:25:59.147480+0000","last_unstale":"2024-02-19T15:26:01.146991+0000","last_undegraded":"2024-02-19T15:26:01.146991+0000","last_fullsized":"2024-02-19T15:26:01.146991+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:29:01.472900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147512+0000","last_change":"2024-02-19T15:26:00.141946+0000","last_active":"2024-02-19T15:26:01.147512+0000","last_peered":"2024-02-19T15:26:01.147512+0000","last_clean":"2024-02-19T15:26:01.147512+0000","last_became_active":"2024-02-19T15:26:00.141407+0000","last_became_peered":"2024-02-19T15:26:00.141407+0000","last_unstale":"2024-02-19T15:26:01.147512+0000","last_undegraded":"2024-02-19T15:26:01.147512+0000","last_fullsized":"2024-02-19T15:26:01.147512+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:32.505935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147172+0000","last_change":"2024-02-19T15:25:59.139522+0000","last_active":"2024-02-19T15:26:01.147172+0000","last_peered":"2024-02-19T15:26:01.147172+0000","last_clean":"2024-02-19T15:26:01.147172+0000","last_became_active":"2024-02-19T15:25:59.139081+0000","last_became_peered":"2024-02-19T15:25:59.139081+0000","last_unstale":"2024-02-19T15:26:01.147172+0000","last_undegraded":"2024-02-19T15:26:01.147172+0000","last_fullsized":"2024-02-19T15:26:01.147172+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:17:30.699147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.163816+0000","last_change":"2024-02-19T15:26:00.162872+0000","last_active":"2024-02-19T15:26:00.163816+0000","last_peered":"2024-02-19T15:26:00.163816+0000","last_clean":"2024-02-19T15:26:00.163816+0000","last_became_active":"2024-02-19T15:26:00.162658+0000","last_became_peered":"2024-02-19T15:26:00.162658+0000","last_unstale":"2024-02-19T15:26:00.163816+0000","last_undegraded":"2024-02-19T15:26:00.163816+0000","last_fullsized":"2024-02-19T15:26:00.163816+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:44:01.635671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147455+0000","last_change":"2024-02-19T15:25:59.144981+0000","last_active":"2024-02-19T15:26:01.147455+0000","last_peered":"2024-02-19T15:26:01.147455+0000","last_clean":"2024-02-19T15:26:01.147455+0000","last_became_active":"2024-02-19T15:25:59.144592+0000","last_became_peered":"2024-02-19T15:25:59.144592+0000","last_unstale":"2024-02-19T15:26:01.147455+0000","last_undegraded":"2024-02-19T15:26:01.147455+0000","last_fullsized":"2024-02-19T15:26:01.147455+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:56.829482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.145843+0000","last_change":"2024-02-19T15:26:00.151112+0000","last_active":"2024-02-19T15:26:01.145843+0000","last_peered":"2024-02-19T15:26:01.145843+0000","last_clean":"2024-02-19T15:26:01.145843+0000","last_became_active":"2024-02-19T15:26:00.150833+0000","last_became_peered":"2024-02-19T15:26:00.150833+0000","last_unstale":"2024-02-19T15:26:01.145843+0000","last_undegraded":"2024-02-19T15:26:01.145843+0000","last_fullsized":"2024-02-19T15:26:01.145843+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:15:36.439258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.135569+0000","last_change":"2024-02-19T15:25:59.141990+0000","last_active":"2024-02-19T15:26:00.135569+0000","last_peered":"2024-02-19T15:26:00.135569+0000","last_clean":"2024-02-19T15:26:00.135569+0000","last_became_active":"2024-02-19T15:25:59.141648+0000","last_became_peered":"2024-02-19T15:25:59.141648+0000","last_unstale":"2024-02-19T15:26:00.135569+0000","last_undegraded":"2024-02-19T15:26:00.135569+0000","last_fullsized":"2024-02-19T15:26:00.135569+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:49:10.613400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.161993+0000","last_change":"2024-02-19T15:26:00.150519+0000","last_active":"2024-02-19T15:26:00.161993+0000","last_peered":"2024-02-19T15:26:00.161993+0000","last_clean":"2024-02-19T15:26:00.161993+0000","last_became_active":"2024-02-19T15:26:00.150235+0000","last_became_peered":"2024-02-19T15:26:00.150235+0000","last_unstale":"2024-02-19T15:26:00.161993+0000","last_undegraded":"2024-02-19T15:26:00.161993+0000","last_fullsized":"2024-02-19T15:26:00.161993+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:50:34.796433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.130335+0000","last_change":"2024-02-19T15:25:59.140936+0000","last_active":"2024-02-19T15:26:00.130335+0000","last_peered":"2024-02-19T15:26:00.130335+0000","last_clean":"2024-02-19T15:26:00.130335+0000","last_became_active":"2024-02-19T15:25:59.140448+0000","last_became_peered":"2024-02-19T15:25:59.140448+0000","last_unstale":"2024-02-19T15:26:00.130335+0000","last_undegraded":"2024-02-19T15:26:00.130335+0000","last_fullsized":"2024-02-19T15:26:00.130335+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:44.096068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.162599+0000","last_change":"2024-02-19T15:26:00.161556+0000","last_active":"2024-02-19T15:26:00.162599+0000","last_peered":"2024-02-19T15:26:00.162599+0000","last_clean":"2024-02-19T15:26:00.162599+0000","last_became_active":"2024-02-19T15:26:00.161274+0000","last_became_peered":"2024-02-19T15:26:00.161274+0000","last_unstale":"2024-02-19T15:26:00.162599+0000","last_undegraded":"2024-02-19T15:26:00.162599+0000","last_fullsized":"2024-02-19T15:26:00.162599+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:53:03.954569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147265+0000","last_change":"2024-02-19T15:25:59.144110+0000","last_active":"2024-02-19T15:26:01.147265+0000","last_peered":"2024-02-19T15:26:01.147265+0000","last_clean":"2024-02-19T15:26:01.147265+0000","last_became_active":"2024-02-19T15:25:59.143881+0000","last_became_peered":"2024-02-19T15:25:59.143881+0000","last_unstale":"2024-02-19T15:26:01.147265+0000","last_undegraded":"2024-02-19T15:26:01.147265+0000","last_fullsized":"2024-02-19T15:26:01.147265+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:00:56.567362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.147125+0000","last_change":"2024-02-19T15:26:00.140998+0000","last_active":"2024-02-19T15:26:01.147125+0000","last_peered":"2024-02-19T15:26:01.147125+0000","last_clean":"2024-02-19T15:26:01.147125+0000","last_became_active":"2024-02-19T15:26:00.139693+0000","last_became_peered":"2024-02-19T15:26:00.139693+0000","last_unstale":"2024-02-19T15:26:01.147125+0000","last_undegraded":"2024-02-19T15:26:01.147125+0000","last_fullsized":"2024-02-19T15:26:01.147125+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:26:01.775208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.132476+0000","last_change":"2024-02-19T15:25:59.142322+0000","last_active":"2024-02-19T15:26:00.132476+0000","last_peered":"2024-02-19T15:26:00.132476+0000","last_clean":"2024-02-19T15:26:00.132476+0000","last_became_active":"2024-02-19T15:25:59.142044+0000","last_became_peered":"2024-02-19T15:25:59.142044+0000","last_unstale":"2024-02-19T15:26:00.132476+0000","last_undegraded":"2024-02-19T15:26:00.132476+0000","last_fullsized":"2024-02-19T15:26:00.132476+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:25.400223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T15:26:01.146866+0000","last_change":"2024-02-19T15:26:00.150250+0000","last_active":"2024-02-19T15:26:01.146866+0000","last_peered":"2024-02-19T15:26:01.146866+0000","last_clean":"2024-02-19T15:26:01.146866+0000","last_became_active":"2024-02-19T15:26:00.143077+0000","last_became_peered":"2024-02-19T15:26:00.143077+0000","last_unstale":"2024-02-19T15:26:01.146866+0000","last_undegraded":"2024-02-19T15:26:01.146866+0000","last_fullsized":"2024-02-19T15:26:01.146866+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:59.116813+0000","last_clean_scrub_stamp":"2024-02-19T15:25:59.116813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:09:12.862734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.165097+0000","last_change":"2024-02-19T15:25:59.145365+0000","last_active":"2024-02-19T15:26:00.165097+0000","last_peered":"2024-02-19T15:26:00.165097+0000","last_clean":"2024-02-19T15:26:00.165097+0000","last_became_active":"2024-02-19T15:25:59.145075+0000","last_became_peered":"2024-02-19T15:25:59.145075+0000","last_unstale":"2024-02-19T15:26:00.165097+0000","last_undegraded":"2024-02-19T15:26:00.165097+0000","last_fullsized":"2024-02-19T15:26:00.165097+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:25:58.109476+0000","last_clean_scrub_stamp":"2024-02-19T15:25:58.109476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:01:06.434946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":277,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T15:26:00.133214+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:26:00.133214+0000","last_peered":"2024-02-19T15:26:00.133214+0000","last_clean":"2024-02-19T15:26:00.133214+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:26:00.133214+0000","last_undegraded":"2024-02-19T15:26:00.133214+0000","last_fullsized":"2024-02-19T15:26:00.133214+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509493,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1015808,"data_stored":425993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.61699999999999999,"15min":0.61799999999999999},"min":{"1min":0.42699999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.90100000000000002,"5min":1.022,"15min":1.022},"last":0.441},{"interface":"front","average":{"1min":0.629,"5min":0.71599999999999997,"15min":0.76400000000000001},"min":{"1min":0.47499999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83699999999999997,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.55200000000000005}]},{"osd":1,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64400000000000002,"15min":0.65200000000000002},"min":{"1min":0.48499999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85099999999999998,"5min":1.171,"15min":1.171},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.65500000000000003,"15min":0.65200000000000002},"min":{"1min":0.50700000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.92300000000000004,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.628,"15min":0.63100000000000001},"min":{"1min":0.44600000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.85299999999999998,"5min":1.216,"15min":1.216},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.69699999999999995,"15min":0.68799999999999994},"min":{"1min":0.55800000000000005,"5min":0.495,"15min":0.495},"max":{"1min":1.01,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.48999999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.68999999999999995,"15min":0.67900000000000005},"min":{"1min":0.54800000000000004,"5min":0.438,"15min":0.438},"max":{"1min":0.91600000000000004,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.752,"5min":0.67100000000000004,"15min":0.66000000000000003},"min":{"1min":0.55600000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.57699999999999996}]},{"osd":4,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.626,"15min":0.60699999999999998},"min":{"1min":0.50800000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.90400000000000003,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63,"15min":0.60399999999999998},"min":{"1min":0.48499999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92600000000000005,"5min":0.998,"15min":0.998},"last":0.41699999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.63200000000000001,"15min":0.60499999999999998},"min":{"1min":0.58699999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.629},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.69499999999999995,"15min":0.67300000000000004},"min":{"1min":0.61499999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.95299999999999996,"5min":1.02,"15min":1.02},"last":0.65600000000000003}]},{"osd":7,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.68799999999999994,"15min":0.65700000000000003},"min":{"1min":0.61899999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.70899999999999996,"15min":0.67700000000000005},"min":{"1min":0.69799999999999995,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.72999999999999998}]}]},{"osd":7,"up_from":14,"seq":60129542197,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32732,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339108,"statfs":{"total":96636764160,"available":96603246592,"internally_reserved":0,"allocated":1589248,"data_stored":1009437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.59699999999999998,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.87,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.67400000000000004},"min":{"1min":0.38800000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.60799999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.63500000000000001,"15min":0.60099999999999998},"min":{"1min":0.498,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86799999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.70399999999999996,"15min":0.72099999999999997},"min":{"1min":0.496,"5min":0.437,"15min":0.437},"max":{"1min":1.008,"5min":2.968,"15min":2.968},"last":0.66100000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.64700000000000002,"15min":0.63600000000000001},"min":{"1min":0.47599999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85399999999999998,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.497,"5min":0.438,"15min":0.438},"max":{"1min":0.97399999999999998,"5min":2.508,"15min":2.508},"last":0.63600000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66300000000000003,"15min":0.65900000000000003},"min":{"1min":0.50900000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.91300000000000003,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.70999999999999996,"15min":0.69499999999999995},"min":{"1min":0.59399999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.94699999999999995,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":0.69999999999999996}]},{"osd":4,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.59499999999999997,"15min":0.53900000000000003},"min":{"1min":0.432,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.61799999999999999,"15min":0.58199999999999996},"min":{"1min":0.40100000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93600000000000005,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.628}]},{"osd":5,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.65700000000000003,"15min":0.58399999999999996},"min":{"1min":0.53500000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.63800000000000001},"min":{"1min":0.54700000000000004,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.97299999999999998,"5min":1.046,"15min":1.046},"last":0.68200000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.67100000000000004,"15min":0.61299999999999999},"min":{"1min":0.46999999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72599999999999998,"15min":0.67500000000000004},"min":{"1min":0.59699999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.76600000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542198,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32184,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339656,"statfs":{"total":96636764160,"available":96603807744,"internally_reserved":0,"allocated":1019904,"data_stored":426933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.67200000000000004,"15min":0.64400000000000002},"min":{"1min":0.38300000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.97699999999999998,"5min":5.7789999999999999,"15min":5.7789999999999999},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.71099999999999997,"15min":0.63700000000000001},"min":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.86499999999999999,"5min":12.359999999999999,"15min":12.359999999999999},"last":0.432}]},{"osd":1,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.73799999999999999,"15min":0.67000000000000004},"min":{"1min":0.432,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80000000000000004,"5min":10.938000000000001,"15min":10.938000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.78900000000000003,"15min":0.71299999999999997},"min":{"1min":0.50600000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85599999999999998,"5min":10.531000000000001,"15min":10.531000000000001},"last":0.66800000000000004}]},{"osd":2,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.79700000000000004,"15min":0.72099999999999997},"min":{"1min":0.42999999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.89900000000000002,"5min":12.925000000000001,"15min":12.925000000000001},"last":0.875},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.72999999999999998,"15min":0.64100000000000001},"min":{"1min":0.48899999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89000000000000001,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.85299999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.91100000000000003,"15min":0.73499999999999999},"min":{"1min":0.49399999999999999,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82299999999999995,"5min":26.161999999999999,"15min":26.161999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.91000000000000003,"15min":0.73999999999999999},"min":{"1min":0.47299999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1120000000000001,"5min":27.934999999999999,"15min":27.934999999999999},"last":1.1120000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.67500000000000004,"15min":0.59199999999999997},"min":{"1min":0.45700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83099999999999996,"5min":9.0549999999999997,"15min":9.0549999999999997},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.70899999999999996,"15min":0.67100000000000004},"min":{"1min":0.44600000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.92800000000000005,"5min":6.2549999999999999,"15min":6.2549999999999999},"last":0.92800000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.746,"15min":0.67000000000000004},"min":{"1min":0.50700000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.6539999999999999,"5min":6.5229999999999997,"15min":6.5229999999999997},"last":1.6539999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81899999999999995,"15min":0.749},"min":{"1min":0.63500000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.2310000000000001,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":1.2310000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.77600000000000002,"15min":0.69399999999999995},"min":{"1min":0.41499999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.95399999999999996,"5min":11.986000000000001,"15min":11.986000000000001},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.77700000000000002,"15min":0.71499999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.89100000000000001,"5min":8.9770000000000003,"15min":8.9770000000000003},"last":0.80500000000000005}]}]},{"osd":4,"up_from":14,"seq":60129542196,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":991232,"data_stored":418999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.378,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.623,"15min":0.60499999999999998},"min":{"1min":0.47099999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85499999999999998,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.51900000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64100000000000001,"15min":0.61899999999999999},"min":{"1min":0.48999999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.879,"5min":1.081,"15min":1.081},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64400000000000002,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83699999999999997,"5min":1.008,"15min":1.008},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.65400000000000003,"15min":0.66000000000000003},"min":{"1min":0.49299999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.90700000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68200000000000005,"15min":0.67600000000000005},"min":{"1min":0.45100000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92000000000000004,"5min":1.002,"15min":1.002},"last":0.65600000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.70799999999999996,"15min":0.68200000000000005},"min":{"1min":0.50600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94899999999999995,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.65900000000000003,"15min":0.65300000000000002},"min":{"1min":0.42799999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89600000000000002,"5min":1.153,"15min":1.153},"last":0.57299999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63600000000000001,"15min":0.60599999999999998},"min":{"1min":0.433,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85399999999999998,"5min":1.042,"15min":1.042},"last":0.495},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70599999999999996,"15min":0.68200000000000005},"min":{"1min":0.54200000000000004,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.94799999999999995,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.81699999999999995}]},{"osd":6,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.622,"15min":0.58299999999999996},"min":{"1min":0.376,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.92200000000000004,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.628},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68600000000000005,"15min":0.65500000000000003},"min":{"1min":0.52900000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90800000000000003,"5min":1.054,"15min":1.054},"last":0.747}]},{"osd":7,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.70199999999999996,"15min":0.67100000000000004},"min":{"1min":0.44800000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.98199999999999998,"5min":1.115,"15min":1.115},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.75600000000000001,"15min":0.73399999999999999},"min":{"1min":0.46999999999999997,"5min":0.443,"15min":0.443},"max":{"1min":1.0169999999999999,"5min":1.24,"15min":1.24},"last":0.70799999999999996}]}]},{"osd":3,"up_from":14,"seq":60129542196,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":1003520,"data_stored":420611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42899999999999999,"15min":0.44700000000000001},"min":{"1min":0.23699999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76600000000000001,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.54100000000000004,"15min":0.64600000000000002},"min":{"1min":0.27500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67600000000000005,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.54900000000000004}]},{"osd":1,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.433,"15min":0.442},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.66000000000000003,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.59699999999999998,"15min":0.70899999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.66400000000000003,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.316}]},{"osd":2,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56799999999999995,"15min":0.61199999999999999},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.66500000000000004,"5min":1.008,"15min":1.008},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55000000000000004,"15min":0.58899999999999997},"min":{"1min":0.39900000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86199999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.44800000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.70299999999999996,"15min":0.67900000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.90600000000000003,"5min":1.004,"15min":1.004},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.80100000000000005,"15min":0.86099999999999999},"min":{"1min":0.58199999999999996,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0269999999999999,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.754}]},{"osd":5,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.80100000000000005,"15min":0.85499999999999998},"min":{"1min":0.54700000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.96699999999999997,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.74199999999999999,"15min":0.748},"min":{"1min":0.53300000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.93899999999999995,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.72699999999999998}]},{"osd":6,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.85499999999999998,"15min":0.90000000000000002},"min":{"1min":0.55300000000000005,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.90800000000000003,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.79600000000000004,"15min":0.81699999999999995},"min":{"1min":0.51800000000000002,"5min":0.502,"15min":0.502},"max":{"1min":1.0209999999999999,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.61899999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76400000000000001,"15min":0.78000000000000003},"min":{"1min":0.52700000000000002,"5min":0.504,"15min":0.504},"max":{"1min":0.96799999999999997,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.84099999999999997,"15min":0.92000000000000004},"min":{"1min":0.57399999999999995,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0009999999999999,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.68100000000000005}]}]},{"osd":2,"up_from":14,"seq":60129542197,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":995328,"data_stored":419581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.66700000000000004,"15min":0.69699999999999995},"min":{"1min":0.27400000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.64400000000000002,"5min":15.42,"15min":15.42},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.71499999999999997,"15min":0.88500000000000001},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.76500000000000001,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.60099999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.75700000000000001,"15min":0.89700000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58999999999999997,"5min":9.5269999999999992,"15min":9.5269999999999992},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.502,"5min":0.94199999999999995,"15min":1.006},"min":{"1min":0.39700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.621,"5min":22.954999999999998,"15min":22.954999999999998},"last":0.621}]},{"osd":3,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.67100000000000004,"15min":0.78300000000000003},"min":{"1min":0.376,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66200000000000003,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.74299999999999999,"15min":0.82299999999999995},"min":{"1min":0.41799999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69399999999999995,"5min":7.734,"15min":7.734},"last":0.64100000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.91400000000000003,"15min":0.90300000000000002},"min":{"1min":0.52000000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.016,"5min":15.369,"15min":15.369},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.82699999999999996,"15min":0.89300000000000002},"min":{"1min":0.53800000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93700000000000006,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.76200000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.871,"15min":0.88900000000000001},"min":{"1min":0.53900000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.95999999999999996,"5min":7.6520000000000001,"15min":7.6520000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.98699999999999999,"15min":0.97499999999999998},"min":{"1min":0.60199999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0660000000000001,"5min":15.391,"15min":15.391},"last":0.751}]},{"osd":6,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.95799999999999996,"15min":0.94399999999999995},"min":{"1min":0.54900000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.90100000000000002,"5min":15.337,"15min":15.337},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.876,"15min":0.95799999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.98499999999999999,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.78900000000000003}]},{"osd":7,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.90600000000000003,"15min":0.98099999999999998},"min":{"1min":0.58399999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.028,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.874},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.91600000000000004,"15min":0.95299999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.98099999999999998,"5min":7.6929999999999996,"15min":7.6929999999999996},"last":0.90200000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542198,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339680,"statfs":{"total":96636764160,"available":96603832320,"internally_reserved":0,"allocated":1003520,"data_stored":419193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.39200000000000002,"15min":0.42399999999999999},"min":{"1min":0.248,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56799999999999995,"5min":0.747,"15min":0.747},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.438,"15min":0.46600000000000003},"min":{"1min":0.28100000000000003,"5min":0.25,"15min":0.25},"max":{"1min":0.69599999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.38}]},{"osd":2,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.42599999999999999,"15min":0.44400000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.56100000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.48799999999999999,"15min":0.52600000000000002},"min":{"1min":0.36299999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.501,"5min":1.071,"15min":1.071},"last":0.36299999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58899999999999997,"15min":0.63300000000000001},"min":{"1min":0.44800000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.64600000000000002,"5min":1.365,"15min":1.365},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.57399999999999995,"15min":0.63},"min":{"1min":0.42999999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.61399999999999999,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.55500000000000005}]},{"osd":4,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.65300000000000002,"15min":0.61299999999999999},"min":{"1min":0.52100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.99099999999999999,"5min":1.127,"15min":1.127},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.69299999999999995,"15min":0.68000000000000005},"min":{"1min":0.54900000000000004,"5min":0.435,"15min":0.435},"max":{"1min":0.98399999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.67900000000000005}]},{"osd":5,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70899999999999996,"15min":0.69699999999999995},"min":{"1min":0.54200000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.95599999999999996,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.70399999999999996,"15min":0.68500000000000005},"min":{"1min":0.503,"5min":0.443,"15min":0.443},"max":{"1min":0.94199999999999995,"5min":1.107,"15min":1.107},"last":0.871}]},{"osd":6,"last update":"Mon Feb 19 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73199999999999998,"15min":0.72899999999999998},"min":{"1min":0.57799999999999996,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0449999999999999,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.74399999999999999,"15min":0.746},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0669999999999999,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.63}]},{"osd":7,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.76800000000000002,"15min":0.78600000000000003},"min":{"1min":0.59999999999999998,"5min":0.496,"15min":0.496},"max":{"1min":0.95299999999999996,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.76400000000000001,"15min":0.748},"min":{"1min":0.58499999999999996,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0229999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.90900000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542196,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36896,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334944,"statfs":{"total":96636764160,"available":96598982656,"internally_reserved":0,"allocated":1589248,"data_stored":1009401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.50700000000000001,"15min":0.628},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.66200000000000003,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.46300000000000002,"15min":0.48299999999999998},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.76000000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.37}]},{"osd":2,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.65400000000000003,"15min":0.78400000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.67500000000000004,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.61799999999999999,"15min":0.746},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69099999999999995,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.53700000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.56699999999999995,"15min":0.67900000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.61699999999999999,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.5},{"interface":"front","average":{"1min":0.495,"5min":0.54800000000000004,"15min":0.60999999999999999},"min":{"1min":0.38100000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.72499999999999998,"5min":1.345,"15min":1.345},"last":0.441}]},{"osd":4,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.745,"15min":0.74299999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.0700000000000001,"5min":1.095,"15min":1.095},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.84399999999999997,"15min":0.92500000000000004},"min":{"1min":0.64000000000000001,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0049999999999999,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.79200000000000004}]},{"osd":5,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.73799999999999999,"15min":0.751},"min":{"1min":0.55900000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.97999999999999998,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.77300000000000002,"15min":0.85299999999999998},"min":{"1min":0.52900000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.05,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.65100000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.82099999999999995,"15min":0.91500000000000004},"min":{"1min":0.59799999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.95899999999999996,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73799999999999999,"15min":0.75700000000000001},"min":{"1min":0.57999999999999996,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.84999999999999998,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.64200000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.85199999999999998,"15min":0.94199999999999995},"min":{"1min":0.58599999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.0229999999999999,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.83499999999999996,"15min":0.94099999999999995},"min":{"1min":0.55900000000000005,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.96499999999999997,"5min":4.915,"15min":4.915},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:26:04.674 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T15:26:04.675 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-19T15:26:04.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.868+0000 7f63ee200640 1 Processor -- start 2024-02-19T15:26:04.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.868+0000 7f63ee200640 1 -- start start 2024-02-19T15:26:04.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.868+0000 7f63ee200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63e806b5c0 0x7f63e8161d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:04.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.868+0000 7f63ee200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f63e8162290 0x7f63e80ae4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:04.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.868+0000 7f63ee200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e80ce660 con 0x7f63e80d0150 2024-02-19T15:26:04.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.868+0000 7f63ee200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63e8162d60 con 0x7f63e8162290 2024-02-19T15:26:04.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.868+0000 7f63ee200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e8162a80 con 0x7f63e806b5c0 2024-02-19T15:26:04.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.868+0000 7f63e7400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f63e80d0150 0x7f63e806aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54870/0 (socket says 172.21.15.47:54870) 2024-02-19T15:26:04.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.868+0000 7f63e7400640 1 -- 172.21.15.47:0/3993904995 learned_addr learned my addr 172.21.15.47:0/3993904995 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:04.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.868+0000 7f63e7e00640 1 --2- 172.21.15.47:0/3993904995 >> v2:172.21.15.116:3300/0 conn(0x7f63e8162290 0x7f63e80ae4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:04.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.868+0000 7f63dfe00640 1 --2- 172.21.15.47:0/3993904995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63e806b5c0 0x7f63e8161d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:04.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.868+0000 7f63e6a00640 1 -- 172.21.15.47:0/3993904995 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424085298 0 0) 0x7f63e80ce660 con 0x7f63e80d0150 2024-02-19T15:26:04.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.868+0000 7f63e6a00640 1 -- 172.21.15.47:0/3993904995 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63cc003530 con 0x7f63e80d0150 2024-02-19T15:26:04.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.868+0000 7f63dfe00640 1 -- 172.21.15.47:0/3993904995 >> v2:172.21.15.116:3300/0 conn(0x7f63e8162290 msgr2=0x7f63e80ae4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:04.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.868+0000 7f63dfe00640 1 --2- 172.21.15.47:0/3993904995 >> v2:172.21.15.116:3300/0 conn(0x7f63e8162290 0x7f63e80ae4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.868+0000 7f63dfe00640 1 -- 172.21.15.47:0/3993904995 >> v1:172.21.15.47:6789/0 conn(0x7f63e80d0150 legacy=0x7f63e806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:04.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.868+0000 7f63dfe00640 1 -- 172.21.15.47:0/3993904995 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e80ad9f0 con 0x7f63e806b5c0 2024-02-19T15:26:04.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.872+0000 7f63dfe00640 1 --2- 172.21.15.47:0/3993904995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63e806b5c0 0x7f63e8161d50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f63d00097c0 tx=0x7f63d001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3157f136440cd16 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:04.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.872+0000 7f63e6a00640 1 -- 172.21.15.47:0/3993904995 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d00ab670 con 0x7f63e806b5c0 2024-02-19T15:26:04.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.872+0000 7f63e6a00640 1 -- 172.21.15.47:0/3993904995 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d00ab7d0 con 0x7f63e806b5c0 2024-02-19T15:26:04.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.872+0000 7f63e6a00640 1 -- 172.21.15.47:0/3993904995 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d00abb60 con 0x7f63e806b5c0 2024-02-19T15:26:04.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.872+0000 7f63ee200640 1 -- 172.21.15.47:0/3993904995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63e806b5c0 msgr2=0x7f63e8161d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:04.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.872+0000 7f63ee200640 1 --2- 172.21.15.47:0/3993904995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63e806b5c0 0x7f63e8161d50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f63d00097c0 tx=0x7f63d001c470 comp rx=0 tx=0).stop 2024-02-19T15:26:04.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.872+0000 7f63ee200640 1 -- 172.21.15.47:0/3993904995 shutdown_connections 2024-02-19T15:26:04.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.872+0000 7f63ee200640 1 --2- 172.21.15.47:0/3993904995 >> v2:172.21.15.116:3300/0 conn(0x7f63e8162290 0x7f63e80ae4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.872+0000 7f63ee200640 1 --2- 172.21.15.47:0/3993904995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63e806b5c0 0x7f63e8161d50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.872+0000 7f63ee200640 1 -- 172.21.15.47:0/3993904995 >> 172.21.15.47:0/3993904995 conn(0x7f63e80b2580 msgr2=0x7f63e80b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:04.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.872+0000 7f63ee200640 1 -- 172.21.15.47:0/3993904995 shutdown_connections 2024-02-19T15:26:04.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.872+0000 7f63ee200640 1 -- 172.21.15.47:0/3993904995 wait complete. 2024-02-19T15:26:04.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63ee200640 1 Processor -- start 2024-02-19T15:26:04.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63ee200640 1 -- start start 2024-02-19T15:26:04.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63ee200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63e806b5c0 0x7f63e81611b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:04.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63ee200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63e80d0150 0x7f63e814f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:04.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63ee200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63e8162290 0x7f63e815cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:04.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63ee200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e8160d90 con 0x7f63e80d0150 2024-02-19T15:26:04.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63ee200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e815d4a0 con 0x7f63e8162290 2024-02-19T15:26:04.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63ee200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e8160430 con 0x7f63e806b5c0 2024-02-19T15:26:04.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63dfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63e80d0150 0x7f63e814f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:04.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63dfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63e80d0150 0x7f63e814f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50616/0 (socket says 172.21.15.47:50616) 2024-02-19T15:26:04.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63dfe00640 1 -- 172.21.15.47:0/3310871271 learned_addr learned my addr 172.21.15.47:0/3310871271 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:04.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63e7400640 1 --2- 172.21.15.47:0/3310871271 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63e806b5c0 0x7f63e81611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:04.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63e7e00640 1 --2- 172.21.15.47:0/3310871271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63e8162290 0x7f63e815cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:04.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63dfe00640 1 -- 172.21.15.47:0/3310871271 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63e806b5c0 msgr2=0x7f63e81611b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:04.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63dfe00640 1 --2- 172.21.15.47:0/3310871271 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63e806b5c0 0x7f63e81611b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63dfe00640 1 -- 172.21.15.47:0/3310871271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63e8162290 msgr2=0x7f63e815cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:04.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63dfe00640 1 --2- 172.21.15.47:0/3310871271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63e8162290 0x7f63e815cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:04.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63dfe00640 1 -- 172.21.15.47:0/3310871271 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d0009420 con 0x7f63e80d0150 2024-02-19T15:26:04.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63e7400640 1 --2- 172.21.15.47:0/3310871271 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63e806b5c0 0x7f63e81611b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:04.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63dfe00640 1 --2- 172.21.15.47:0/3310871271 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63e80d0150 0x7f63e814f090 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f63d00ad040 tx=0x7f63d001acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:04.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63df400640 1 -- 172.21.15.47:0/3310871271 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d00bf610 con 0x7f63e80d0150 2024-02-19T15:26:04.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63df400640 1 -- 172.21.15.47:0/3310871271 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d00bf790 con 0x7f63e80d0150 2024-02-19T15:26:04.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63df400640 1 -- 172.21.15.47:0/3310871271 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d00bfb40 con 0x7f63e80d0150 2024-02-19T15:26:04.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63ee200640 1 -- 172.21.15.47:0/3310871271 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e81606d0 con 0x7f63e80d0150 2024-02-19T15:26:04.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.876+0000 7f63ee200640 1 -- 172.21.15.47:0/3310871271 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e814f5d0 con 0x7f63e80d0150 2024-02-19T15:26:04.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.880+0000 7f63ee200640 1 -- 172.21.15.47:0/3310871271 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e8075360 con 0x7f63e80d0150 2024-02-19T15:26:04.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.880+0000 7f63df400640 1 -- 172.21.15.47:0/3310871271 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f63d00b5400 con 0x7f63e80d0150 2024-02-19T15:26:04.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.880+0000 7f63df400640 1 --2- 172.21.15.47:0/3310871271 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f63c4066e40 0x7f63c4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:04.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.880+0000 7f63df400640 1 -- 172.21.15.47:0/3310871271 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63d011f790 con 0x7f63e80d0150 2024-02-19T15:26:04.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.880+0000 7f63e7400640 1 --2- 172.21.15.47:0/3310871271 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f63c4066e40 0x7f63c4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:04.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.880+0000 7f63e7400640 1 --2- 172.21.15.47:0/3310871271 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f63c4066e40 0x7f63c4069300 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f63d8005890 tx=0x7f63d8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:04.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:04.884+0000 7f63df400640 1 -- 172.21.15.47:0/3310871271 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f63d00ec570 con 0x7f63e80d0150 2024-02-19T15:26:05.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.104+0000 7f63ee200640 1 -- 172.21.15.47:0/3310871271 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f63e806b9a0 con 0x7f63e80d0150 2024-02-19T15:26:05.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.108+0000 7f63df400640 1 -- 172.21.15.47:0/3310871271 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f63d00efc30 con 0x7f63e80d0150 2024-02-19T15:26:05.111 INFO:teuthology.orchestra.run.smithi047.stdout:[client.0] 2024-02-19T15:26:05.112 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDjcdNlb7eRNRAAk6ZLGfjhcdlR9GoXFCd9eA== 2024-02-19T15:26:05.112 INFO:teuthology.orchestra.run.smithi047.stdout: caps mds = "allow" 2024-02-19T15:26:05.112 INFO:teuthology.orchestra.run.smithi047.stdout: caps mon = "allow r" 2024-02-19T15:26:05.112 INFO:teuthology.orchestra.run.smithi047.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-19T15:26:05.114 INFO:teuthology.orchestra.run.smithi047.stderr:updated caps for client.0 2024-02-19T15:26:05.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.108+0000 7f63ee200640 1 -- 172.21.15.47:0/3310871271 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f63c4066e40 msgr2=0x7f63c4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:05.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.108+0000 7f63ee200640 1 --2- 172.21.15.47:0/3310871271 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f63c4066e40 0x7f63c4069300 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f63d8005890 tx=0x7f63d8009040 comp rx=0 tx=0).stop 2024-02-19T15:26:05.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.108+0000 7f63ee200640 1 -- 172.21.15.47:0/3310871271 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63e80d0150 msgr2=0x7f63e814f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:05.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.108+0000 7f63ee200640 1 --2- 172.21.15.47:0/3310871271 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63e80d0150 0x7f63e814f090 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f63d00ad040 tx=0x7f63d001acb0 comp rx=0 tx=0).stop 2024-02-19T15:26:05.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.108+0000 7f63ee200640 1 -- 172.21.15.47:0/3310871271 shutdown_connections 2024-02-19T15:26:05.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.108+0000 7f63ee200640 1 --2- 172.21.15.47:0/3310871271 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f63c4066e40 0x7f63c4069300 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:05.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.108+0000 7f63ee200640 1 --2- 172.21.15.47:0/3310871271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63e8162290 0x7f63e815cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:05.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.108+0000 7f63ee200640 1 --2- 172.21.15.47:0/3310871271 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63e80d0150 0x7f63e814f090 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:05.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.108+0000 7f63ee200640 1 --2- 172.21.15.47:0/3310871271 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63e806b5c0 0x7f63e81611b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:05.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.108+0000 7f63ee200640 1 -- 172.21.15.47:0/3310871271 >> 172.21.15.47:0/3310871271 conn(0x7f63e80b2580 msgr2=0x7f63e80b1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:05.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.108+0000 7f63ee200640 1 -- 172.21.15.47:0/3310871271 shutdown_connections 2024-02-19T15:26:05.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.108+0000 7f63ee200640 1 -- 172.21.15.47:0/3310871271 wait complete. 2024-02-19T15:26:05.132 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:26:05.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.268+0000 7f38cf400640 1 Processor -- start 2024-02-19T15:26:05.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.268+0000 7f38cf400640 1 -- start start 2024-02-19T15:26:05.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.272+0000 7f38cf400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f38c806d650 0x7f38c80d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:05.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.272+0000 7f38cf400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f38c80cded0 0x7f38c80ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:05.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.272+0000 7f38cf400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c80cce30 con 0x7f38c80ceb40 2024-02-19T15:26:05.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.272+0000 7f38cf400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c80cefd0 con 0x7f38c80cded0 2024-02-19T15:26:05.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.272+0000 7f38cf400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c80cd110 con 0x7f38c806d650 2024-02-19T15:26:05.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.272+0000 7f38bfe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f38c80cded0 0x7f38c80ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:05.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.272+0000 7f38bfe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f38c80cded0 0x7f38c80ce600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50628/0 (socket says 172.21.15.47:50628) 2024-02-19T15:26:05.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.272+0000 7f38bfe00640 1 -- 172.21.15.47:0/1655897551 learned_addr learned my addr 172.21.15.47:0/1655897551 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:05.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.272+0000 7f38cd400640 1 --1- 172.21.15.47:0/1655897551 >> v1:172.21.15.47:6789/0 conn(0x7f38c80ceb40 0x7f38c8161d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54882/0 (socket says 172.21.15.47:54882) 2024-02-19T15:26:05.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.272+0000 7f38cca00640 1 --2- 172.21.15.47:0/1655897551 >> v2:172.21.15.116:3300/0 conn(0x7f38c806d650 0x7f38c80d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:05.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.272+0000 7f38bfe00640 1 -- 172.21.15.47:0/1655897551 >> v2:172.21.15.116:3300/0 conn(0x7f38c806d650 msgr2=0x7f38c80d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:05.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.272+0000 7f38bfe00640 1 --2- 172.21.15.47:0/1655897551 >> v2:172.21.15.116:3300/0 conn(0x7f38c806d650 0x7f38c80d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:05.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.272+0000 7f38bfe00640 1 -- 172.21.15.47:0/1655897551 >> v1:172.21.15.47:6789/0 conn(0x7f38c80ceb40 legacy=0x7f38c8161d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:05.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.272+0000 7f38bfe00640 1 -- 172.21.15.47:0/1655897551 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c80cd3f0 con 0x7f38c80cded0 2024-02-19T15:26:05.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.276+0000 7f38bfe00640 1 --2- 172.21.15.47:0/1655897551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80cded0 0x7f38c80ce600 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f38c8072160 tx=0x7f38b001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89e7c9cf7947dd7b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:05.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.276+0000 7f38c7e00640 1 -- 172.21.15.47:0/1655897551 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b00ad680 con 0x7f38c80cded0 2024-02-19T15:26:05.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.276+0000 7f38c7e00640 1 -- 172.21.15.47:0/1655897551 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b00ad7e0 con 0x7f38c80cded0 2024-02-19T15:26:05.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.276+0000 7f38c7e00640 1 -- 172.21.15.47:0/1655897551 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b00adb70 con 0x7f38c80cded0 2024-02-19T15:26:05.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.276+0000 7f38cf400640 1 -- 172.21.15.47:0/1655897551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80cded0 msgr2=0x7f38c80ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:05.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.276+0000 7f38cf400640 1 --2- 172.21.15.47:0/1655897551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80cded0 0x7f38c80ce600 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f38c8072160 tx=0x7f38b001c460 comp rx=0 tx=0).stop 2024-02-19T15:26:05.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.276+0000 7f38cf400640 1 -- 172.21.15.47:0/1655897551 shutdown_connections 2024-02-19T15:26:05.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.276+0000 7f38cf400640 1 --2- 172.21.15.47:0/1655897551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80cded0 0x7f38c80ce600 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:05.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.276+0000 7f38cf400640 1 --2- 172.21.15.47:0/1655897551 >> v2:172.21.15.116:3300/0 conn(0x7f38c806d650 0x7f38c80d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:05.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.276+0000 7f38cf400640 1 -- 172.21.15.47:0/1655897551 >> 172.21.15.47:0/1655897551 conn(0x7f38c80d1bf0 msgr2=0x7f38c80d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:05.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.276+0000 7f38cf400640 1 -- 172.21.15.47:0/1655897551 shutdown_connections 2024-02-19T15:26:05.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.276+0000 7f38cf400640 1 -- 172.21.15.47:0/1655897551 wait complete. 2024-02-19T15:26:05.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.276+0000 7f38cf400640 1 Processor -- start 2024-02-19T15:26:05.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.276+0000 7f38cf400640 1 -- start start 2024-02-19T15:26:05.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38cf400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f38c806d650 0x7f38c815ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:05.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38cf400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38c80cded0 0x7f38c80c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:05.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38cf400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80ceb40 0x7f38c815bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:05.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38cf400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c80c7220 con 0x7f38c80ceb40 2024-02-19T15:26:05.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38cf400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c80c6980 con 0x7f38c80cded0 2024-02-19T15:26:05.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38cf400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c80c6c60 con 0x7f38c806d650 2024-02-19T15:26:05.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38cd400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80ceb40 0x7f38c815bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:05.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38cd400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80ceb40 0x7f38c815bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50636/0 (socket says 172.21.15.47:50636) 2024-02-19T15:26:05.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38cd400640 1 -- 172.21.15.47:0/4271921025 learned_addr learned my addr 172.21.15.47:0/4271921025 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:05.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38bfe00640 1 --2- 172.21.15.47:0/4271921025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38c80cded0 0x7f38c80c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:05.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38cca00640 1 --2- 172.21.15.47:0/4271921025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f38c806d650 0x7f38c815ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:05.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38cd400640 1 -- 172.21.15.47:0/4271921025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f38c806d650 msgr2=0x7f38c815ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:05.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38cd400640 1 --2- 172.21.15.47:0/4271921025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f38c806d650 0x7f38c815ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:05.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38cd400640 1 -- 172.21.15.47:0/4271921025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38c80cded0 msgr2=0x7f38c80c86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:05.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38cd400640 1 --2- 172.21.15.47:0/4271921025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38c80cded0 0x7f38c80c86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:05.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38cd400640 1 -- 172.21.15.47:0/4271921025 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b00094d0 con 0x7f38c80ceb40 2024-02-19T15:26:05.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38cd400640 1 --2- 172.21.15.47:0/4271921025 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80ceb40 0x7f38c815bb80 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f38c000c700 tx=0x7f38c000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:05.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38c5600640 1 -- 172.21.15.47:0/4271921025 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c000ce10 con 0x7f38c80ceb40 2024-02-19T15:26:05.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38c5600640 1 -- 172.21.15.47:0/4271921025 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c000a640 con 0x7f38c80ceb40 2024-02-19T15:26:05.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38cf400640 1 -- 172.21.15.47:0/4271921025 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c80c6f40 con 0x7f38c80ceb40 2024-02-19T15:26:05.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38c5600640 1 -- 172.21.15.47:0/4271921025 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c000a9b0 con 0x7f38c80ceb40 2024-02-19T15:26:05.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38cf400640 1 -- 172.21.15.47:0/4271921025 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c815c0c0 con 0x7f38c80ceb40 2024-02-19T15:26:05.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38cf400640 1 -- 172.21.15.47:0/4271921025 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c8075360 con 0x7f38c80ceb40 2024-02-19T15:26:05.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.280+0000 7f38c5600640 1 -- 172.21.15.47:0/4271921025 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f38c000ab10 con 0x7f38c80ceb40 2024-02-19T15:26:05.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.284+0000 7f38c5600640 1 --2- 172.21.15.47:0/4271921025 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f38a4066e70 0x7f38a4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:05.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.284+0000 7f38c5600640 1 -- 172.21.15.47:0/4271921025 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f38c0084130 con 0x7f38c80ceb40 2024-02-19T15:26:05.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.284+0000 7f38cca00640 1 --2- 172.21.15.47:0/4271921025 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f38a4066e70 0x7f38a4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:05.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.284+0000 7f38cca00640 1 --2- 172.21.15.47:0/4271921025 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f38a4066e70 0x7f38a4069330 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f38b8009ac0 tx=0x7f38b8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:05.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.292+0000 7f38c5600640 1 -- 172.21.15.47:0/4271921025 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f38c0050fc0 con 0x7f38c80ceb40 2024-02-19T15:26:05.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.488+0000 7f38cf400640 1 -- 172.21.15.47:0/4271921025 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38c80d14e0 con 0x7f38c80ceb40 2024-02-19T15:26:05.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.488+0000 7f38c5600640 1 -- 172.21.15.47:0/4271921025 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4943 (secure 0 0 0) 0x7f38c0054680 con 0x7f38c80ceb40 2024-02-19T15:26:05.493 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:05.494 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5954,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1402677843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1402677843},{"type":"v1","addr":"172.21.15.116:6836","nonce":1402677843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":5957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1507554634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":1507554634},{"type":"v1","addr":"172.21.15.116:6837","nonce":1507554634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6771,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/888455604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":888455604},{"type":"v1","addr":"172.21.15.47:6833","nonce":888455604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:25:59.522224+0000","modified":"2024-02-19T15:26:02.535882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6774":{"gid":6774,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3598443511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3598443511},{"type":"v1","addr":"172.21.15.47:6835","nonce":3598443511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-19T15:26:05.494 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 49 2024-02-19T15:26:05.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.492+0000 7f38cf400640 1 -- 172.21.15.47:0/4271921025 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f38a4066e70 msgr2=0x7f38a4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:05.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.492+0000 7f38cf400640 1 --2- 172.21.15.47:0/4271921025 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f38a4066e70 0x7f38a4069330 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f38b8009ac0 tx=0x7f38b8009190 comp rx=0 tx=0).stop 2024-02-19T15:26:05.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.492+0000 7f38cf400640 1 -- 172.21.15.47:0/4271921025 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80ceb40 msgr2=0x7f38c815bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:05.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.492+0000 7f38cf400640 1 --2- 172.21.15.47:0/4271921025 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80ceb40 0x7f38c815bb80 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f38c000c700 tx=0x7f38c000cbc0 comp rx=0 tx=0).stop 2024-02-19T15:26:05.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.492+0000 7f38cf400640 1 -- 172.21.15.47:0/4271921025 shutdown_connections 2024-02-19T15:26:05.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.492+0000 7f38cf400640 1 --2- 172.21.15.47:0/4271921025 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f38a4066e70 0x7f38a4069330 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:05.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.492+0000 7f38cf400640 1 --2- 172.21.15.47:0/4271921025 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38c80ceb40 0x7f38c815bb80 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:05.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.492+0000 7f38cf400640 1 --2- 172.21.15.47:0/4271921025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38c80cded0 0x7f38c80c86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:05.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.492+0000 7f38cf400640 1 --2- 172.21.15.47:0/4271921025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f38c806d650 0x7f38c815ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:05.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.492+0000 7f38cf400640 1 -- 172.21.15.47:0/4271921025 >> 172.21.15.47:0/4271921025 conn(0x7f38c80d1bf0 msgr2=0x7f38c80cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:05.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.492+0000 7f38cf400640 1 -- 172.21.15.47:0/4271921025 shutdown_connections 2024-02-19T15:26:05.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:05.492+0000 7f38cf400640 1 -- 172.21.15.47:0/4271921025 wait complete. 2024-02-19T15:26:05.584 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:26:05.588 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:26:05.918 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 49, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:25:59.522224+0000', 'modified': '2024-02-19T15:26:02.535882+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6774}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6774': {'gid': 6774, 'name': 'c', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/3598443511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6834', 'nonce': 3598443511}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 3598443511}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:26:05.918 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T15:26:05.918 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T15:26:06.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.056+0000 7efd58a00640 1 Processor -- start 2024-02-19T15:26:06.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.056+0000 7efd58a00640 1 -- start start 2024-02-19T15:26:06.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.056+0000 7efd58a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efd5415d6e0 0x7efd54162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.056+0000 7efd58a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd54071350 0x7efd54164c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.056+0000 7efd58a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd540718c0 con 0x7efd5415b1a0 2024-02-19T15:26:06.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.056+0000 7efd58a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd54071ba0 con 0x7efd5415d6e0 2024-02-19T15:26:06.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.056+0000 7efd58a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd54071e80 con 0x7efd54071350 2024-02-19T15:26:06.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.056+0000 7efd51400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efd5415d6e0 0x7efd54162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.056+0000 7efd51400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efd5415d6e0 0x7efd54162740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50640/0 (socket says 172.21.15.47:50640) 2024-02-19T15:26:06.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.056+0000 7efd51400640 1 -- 172.21.15.47:0/2934617296 learned_addr learned my addr 172.21.15.47:0/2934617296 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:06.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.056+0000 7efd52800640 1 --2- 172.21.15.47:0/2934617296 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd54071350 0x7efd54164c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.056+0000 7efd51400640 1 -- 172.21.15.47:0/2934617296 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd54071350 msgr2=0x7efd54164c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:06.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.056+0000 7efd51400640 1 --2- 172.21.15.47:0/2934617296 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd54071350 0x7efd54164c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.056+0000 7efd51400640 1 -- 172.21.15.47:0/2934617296 >> v1:172.21.15.116:6789/0 conn(0x7efd5415b1a0 legacy=0x7efd5415b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:06.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.056+0000 7efd51400640 1 -- 172.21.15.47:0/2934617296 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd54072160 con 0x7efd5415d6e0 2024-02-19T15:26:06.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.056+0000 7efd52800640 1 --2- 172.21.15.47:0/2934617296 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd54071350 0x7efd54164c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:06.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.060+0000 7efd51400640 1 --2- 172.21.15.47:0/2934617296 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd5415d6e0 0x7efd54162740 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7efd4000bb40 tx=0x7efd4001ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d42c918f2361aa76 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:06.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.060+0000 7efd50a00640 1 -- 172.21.15.47:0/2934617296 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd400a7c00 con 0x7efd5415d6e0 2024-02-19T15:26:06.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.060+0000 7efd50a00640 1 -- 172.21.15.47:0/2934617296 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd400a7d60 con 0x7efd5415d6e0 2024-02-19T15:26:06.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.060+0000 7efd50a00640 1 -- 172.21.15.47:0/2934617296 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd400af660 con 0x7efd5415d6e0 2024-02-19T15:26:06.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.060+0000 7efd58a00640 1 -- 172.21.15.47:0/2934617296 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd5415d6e0 msgr2=0x7efd54162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:06.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.060+0000 7efd58a00640 1 --2- 172.21.15.47:0/2934617296 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd5415d6e0 0x7efd54162740 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7efd4000bb40 tx=0x7efd4001ec10 comp rx=0 tx=0).stop 2024-02-19T15:26:06.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.060+0000 7efd58a00640 1 -- 172.21.15.47:0/2934617296 shutdown_connections 2024-02-19T15:26:06.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.060+0000 7efd58a00640 1 --2- 172.21.15.47:0/2934617296 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd54071350 0x7efd54164c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.060+0000 7efd58a00640 1 --2- 172.21.15.47:0/2934617296 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd5415d6e0 0x7efd54162740 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.060+0000 7efd58a00640 1 -- 172.21.15.47:0/2934617296 >> 172.21.15.47:0/2934617296 conn(0x7efd540b16a0 msgr2=0x7efd540b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:06.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.060+0000 7efd58a00640 1 -- 172.21.15.47:0/2934617296 shutdown_connections 2024-02-19T15:26:06.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.060+0000 7efd58a00640 1 -- 172.21.15.47:0/2934617296 wait complete. 2024-02-19T15:26:06.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.060+0000 7efd58a00640 1 Processor -- start 2024-02-19T15:26:06.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.060+0000 7efd58a00640 1 -- start start 2024-02-19T15:26:06.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd58a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd54071350 0x7efd540cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd58a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd5415b1a0 0x7efd5414d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd58a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd5415d6e0 0x7efd540cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd58a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5414c360 con 0x7efd54071350 2024-02-19T15:26:06.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd58a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5414c640 con 0x7efd5415d6e0 2024-02-19T15:26:06.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd58a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5414c920 con 0x7efd5415b1a0 2024-02-19T15:26:06.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd51e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd54071350 0x7efd540cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd51e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd54071350 0x7efd540cbf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50642/0 (socket says 172.21.15.47:50642) 2024-02-19T15:26:06.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd51e00640 1 -- 172.21.15.47:0/2044375550 learned_addr learned my addr 172.21.15.47:0/2044375550 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:06.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd51400640 1 --2- 172.21.15.47:0/2044375550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd5415b1a0 0x7efd5414d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd52800640 1 --2- 172.21.15.47:0/2044375550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd5415d6e0 0x7efd540cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd51e00640 1 -- 172.21.15.47:0/2044375550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd5415b1a0 msgr2=0x7efd5414d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:06.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd51e00640 1 --2- 172.21.15.47:0/2044375550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd5415b1a0 0x7efd5414d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd51e00640 1 -- 172.21.15.47:0/2044375550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd5415d6e0 msgr2=0x7efd540cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:06.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd51e00640 1 --2- 172.21.15.47:0/2044375550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd5415d6e0 0x7efd540cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd51e00640 1 -- 172.21.15.47:0/2044375550 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd4000b770 con 0x7efd54071350 2024-02-19T15:26:06.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd51e00640 1 --2- 172.21.15.47:0/2044375550 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd54071350 0x7efd540cbf40 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7efd4400c420 tx=0x7efd4400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:06.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd4a000640 1 -- 172.21.15.47:0/2044375550 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd440052d0 con 0x7efd54071350 2024-02-19T15:26:06.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd4a000640 1 -- 172.21.15.47:0/2044375550 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd44005430 con 0x7efd54071350 2024-02-19T15:26:06.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd58a00640 1 -- 172.21.15.47:0/2044375550 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd5414cc00 con 0x7efd54071350 2024-02-19T15:26:06.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd4a000640 1 -- 172.21.15.47:0/2044375550 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd44007260 con 0x7efd54071350 2024-02-19T15:26:06.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.064+0000 7efd58a00640 1 -- 172.21.15.47:0/2044375550 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd540cc480 con 0x7efd54071350 2024-02-19T15:26:06.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd4a000640 1 -- 172.21.15.47:0/2044375550 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efd440073c0 con 0x7efd54071350 2024-02-19T15:26:06.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd4a000640 1 --2- 172.21.15.47:0/2044375550 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd34066e40 0x7efd34069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd4a000640 1 -- 172.21.15.47:0/2044375550 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd44083e50 con 0x7efd54071350 2024-02-19T15:26:06.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd5be46000 1 Processor -- start 2024-02-19T15:26:06.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd51400640 1 --2- 172.21.15.47:0/2044375550 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd34066e40 0x7efd34069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd5be46000 1 -- start start 2024-02-19T15:26:06.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd5be46000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891bbfd80 0x55f891bc0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd5be46000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891bc06a0 0x55f891b761e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd5be46000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891b76720 0x55f891b76b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd5be46000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f891a39d00 con 0x55f891b76720 2024-02-19T15:26:06.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd5be46000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8919cb430 con 0x55f891bc06a0 2024-02-19T15:26:06.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd5be46000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f8919c8960 con 0x55f891bbfd80 2024-02-19T15:26:06.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd51400640 1 --2- 172.21.15.47:0/2044375550 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd34066e40 0x7efd34069300 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7efd400b1000 tx=0x7efd40008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:06.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd52800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891b76720 0x55f891b76b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd51e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891bbfd80 0x55f891bc0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd52800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891b76720 0x55f891b76b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50658/0 (socket says 172.21.15.47:50658) 2024-02-19T15:26:06.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd52800640 1 -- 172.21.15.47:0/3016308906 learned_addr learned my addr 172.21.15.47:0/3016308906 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:06.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd51400640 1 --2- 172.21.15.47:0/3016308906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891bc06a0 0x55f891b761e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd52800640 1 -- 172.21.15.47:0/3016308906 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891bbfd80 msgr2=0x55f891bc0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:06.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd52800640 1 --2- 172.21.15.47:0/3016308906 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891bbfd80 0x55f891bc0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd52800640 1 -- 172.21.15.47:0/3016308906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891bc06a0 msgr2=0x55f891b761e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:06.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd52800640 1 --2- 172.21.15.47:0/3016308906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891bc06a0 0x55f891b761e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd52800640 1 -- 172.21.15.47:0/3016308906 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8916ba890 con 0x55f891b76720 2024-02-19T15:26:06.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.068+0000 7efd51e00640 1 --2- 172.21.15.47:0/3016308906 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891bbfd80 0x55f891bc0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:06.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd52800640 1 --2- 172.21.15.47:0/3016308906 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891b76720 0x55f891b76b00 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7efd4c0073e0 tx=0x7efd4c0078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd753e9f42f38eba server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:06.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd3ea00640 1 -- 172.21.15.47:0/3016308906 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4c015850 con 0x55f891b76720 2024-02-19T15:26:06.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd3ea00640 1 -- 172.21.15.47:0/3016308906 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd4c00f040 con 0x55f891b76720 2024-02-19T15:26:06.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd3ea00640 1 -- 172.21.15.47:0/3016308906 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4c015bb0 con 0x55f891b76720 2024-02-19T15:26:06.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd5be46000 1 -- 172.21.15.47:0/3016308906 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891b76720 msgr2=0x55f891b76b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:06.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd5be46000 1 --2- 172.21.15.47:0/3016308906 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891b76720 0x55f891b76b00 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7efd4c0073e0 tx=0x7efd4c0078a0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd5be46000 1 -- 172.21.15.47:0/3016308906 shutdown_connections 2024-02-19T15:26:06.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd5be46000 1 --2- 172.21.15.47:0/3016308906 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891b76720 0x55f891b76b00 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd5be46000 1 --2- 172.21.15.47:0/3016308906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891bc06a0 0x55f891b761e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd5be46000 1 --2- 172.21.15.47:0/3016308906 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891bbfd80 0x55f891bc0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd5be46000 1 -- 172.21.15.47:0/3016308906 >> 172.21.15.47:0/3016308906 conn(0x55f891b726d0 msgr2=0x55f891b74ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:06.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd5be46000 1 -- 172.21.15.47:0/3016308906 shutdown_connections 2024-02-19T15:26:06.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd5be46000 1 -- 172.21.15.47:0/3016308906 wait complete. 2024-02-19T15:26:06.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd5be46000 1 -- start start 2024-02-19T15:26:06.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd5be46000 1 Processor -- start 2024-02-19T15:26:06.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd5be46000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891b76720 0x55f891bc0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd5be46000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891bbfd80 0x55f891bc11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd5be46000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 0x55f891bc1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd5be46000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8919c4140 con 0x55f891bc06a0 2024-02-19T15:26:06.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd5be46000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f89195e630 con 0x55f891b76720 2024-02-19T15:26:06.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd5be46000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f89195e2d0 con 0x55f891bbfd80 2024-02-19T15:26:06.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd5be46000 10 client.0 _ll_register_callbacks cb 0x55f891a741c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:26:06.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd52800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 0x55f891bc1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd52800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 0x55f891bc1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50672/0 (socket says 172.21.15.47:50672) 2024-02-19T15:26:06.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd51e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891b76720 0x55f891bc0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd52800640 1 -- 172.21.15.47:0/3105733047 learned_addr learned my addr 172.21.15.47:0/3105733047 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:06.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd51400640 1 --2- 172.21.15.47:0/3105733047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891bbfd80 0x55f891bc11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd52800640 1 -- 172.21.15.47:0/3105733047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891bbfd80 msgr2=0x55f891bc11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:06.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd52800640 1 --2- 172.21.15.47:0/3105733047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891bbfd80 0x55f891bc11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd52800640 1 -- 172.21.15.47:0/3105733047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891b76720 msgr2=0x55f891bc0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:06.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd52800640 1 --2- 172.21.15.47:0/3105733047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891b76720 0x55f891bc0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.072+0000 7efd52800640 1 -- 172.21.15.47:0/3105733047 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd540cc480 con 0x55f891bc06a0 2024-02-19T15:26:06.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd52800640 1 --2- 172.21.15.47:0/3105733047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 0x55f891bc1710 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7efd4c0078e0 tx=0x7efd4c004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2da32961af8ca21 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:06.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd33400640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:26:06.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd33400640 1 -- 172.21.15.47:0/3105733047 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4c016af0 con 0x55f891bc06a0 2024-02-19T15:26:06.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd33400640 1 -- 172.21.15.47:0/3105733047 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd4c024070 con 0x55f891bc06a0 2024-02-19T15:26:06.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 1 -- 172.21.15.47:0/3105733047 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f891977850 con 0x55f891bc06a0 2024-02-19T15:26:06.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd33400640 1 -- 172.21.15.47:0/3105733047 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd4c016c50 con 0x55f891bc06a0 2024-02-19T15:26:06.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 10 client.6912 fetch_fsmap learned FSMap version 49 2024-02-19T15:26:06.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 1 -- 172.21.15.47:0/3105733047 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x7efd4c009ce0 con 0x55f891bc06a0 2024-02-19T15:26:06.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd33400640 1 -- 172.21.15.47:0/3105733047 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f891977850 con 0x55f891bc06a0 2024-02-19T15:26:06.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd33400640 1 -- 172.21.15.47:0/3105733047 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 49) v1 ==== 3294+0+0 (secure 0 0 0) 0x7efd4c004040 con 0x55f891bc06a0 2024-02-19T15:26:06.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 10 client.6912 fetch_fsmap finished waiting for FSMap version 49 2024-02-19T15:26:06.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 10 client.6912 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T15:26:06.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 20 client.6912 populate_metadata read hostname 'smithi047' 2024-02-19T15:26:06.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 1 --2- 172.21.15.47:0/3105733047 >> [v2:172.21.15.47:6834/3598443511,v1:172.21.15.47:6835/3598443511] conn(0x55f891bc1c50 0x55f891bc2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 4 client.6912 mds_command: new command op to 6774 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T15:26:06.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 1 -- 172.21.15.47:0/3105733047 --> [v2:172.21.15.47:6834/3598443511,v1:172.21.15.47:6835/3598443511] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f8919c0f20 con 0x55f891bc1c50 2024-02-19T15:26:06.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd51e00640 1 --2- 172.21.15.47:0/3105733047 >> [v2:172.21.15.47:6834/3598443511,v1:172.21.15.47:6835/3598443511] conn(0x55f891bc1c50 0x55f891bc2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd51e00640 1 --2- 172.21.15.47:0/3105733047 >> [v2:172.21.15.47:6834/3598443511,v1:172.21.15.47:6835/3598443511] conn(0x55f891bc1c50 0x55f891bc2030 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71daae4c506944a5 server_cookie=5abca43742914237 in_seq=0 out_seq=0 2024-02-19T15:26:06.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd33400640 10 client.6912 ms_handle_connect on v2:172.21.15.47:6834/3598443511 2024-02-19T15:26:06.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd33400640 1 -- 172.21.15.47:0/3105733047 <== mds.0 v2:172.21.15.47:6834/3598443511 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f8919c0f20 con 0x55f891bc1c50 2024-02-19T15:26:06.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd33400640 10 client.6912 handle_command_reply: tid=0 2024-02-19T15:26:06.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 1 client.6912 shutdown 2024-02-19T15:26:06.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 1 -- 172.21.15.47:0/3105733047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 msgr2=0x55f891bc1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:06.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 1 --2- 172.21.15.47:0/3105733047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 0x55f891bc1710 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7efd4c0078e0 tx=0x7efd4c004ac0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 1 -- 172.21.15.47:0/3105733047 shutdown_connections 2024-02-19T15:26:06.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 1 --2- 172.21.15.47:0/3105733047 >> [v2:172.21.15.47:6834/3598443511,v1:172.21.15.47:6835/3598443511] conn(0x55f891bc1c50 0x55f891bc2030 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 1 --2- 172.21.15.47:0/3105733047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 0x55f891bc1710 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 1 --2- 172.21.15.47:0/3105733047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891bbfd80 0x55f891bc11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 1 --2- 172.21.15.47:0/3105733047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891b76720 0x55f891bc0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 1 -- 172.21.15.47:0/3105733047 >> 172.21.15.47:0/3105733047 conn(0x55f891b726d0 msgr2=0x55f891b77390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:06.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd33400640 0 client.6912 ms_handle_reset on v2:172.21.15.47:6834/3598443511 2024-02-19T15:26:06.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 1 -- 172.21.15.47:0/3105733047 shutdown_connections 2024-02-19T15:26:06.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 1 -- 172.21.15.47:0/3105733047 wait complete. 2024-02-19T15:26:06.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.076+0000 7efd5be46000 20 client.6912 trim_cache size 0 max 16384 2024-02-19T15:26:06.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd5be46000 1 Processor -- start 2024-02-19T15:26:06.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd5be46000 1 -- start start 2024-02-19T15:26:06.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd5be46000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891b76720 0x55f891b75ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd5be46000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891bbfd80 0x55f891b76010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd5be46000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 0x55f891baa230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd5be46000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f89167b940 con 0x55f891bc06a0 2024-02-19T15:26:06.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd5be46000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f891945b10 con 0x55f891b76720 2024-02-19T15:26:06.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd5be46000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f891963f10 con 0x55f891bbfd80 2024-02-19T15:26:06.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd52800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 0x55f891baa230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd52800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 0x55f891baa230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50684/0 (socket says 172.21.15.47:50684) 2024-02-19T15:26:06.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd52800640 1 -- 172.21.15.47:0/4276857564 learned_addr learned my addr 172.21.15.47:0/4276857564 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:06.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd51e00640 1 --2- 172.21.15.47:0/4276857564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891b76720 0x55f891b75ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd51400640 1 --2- 172.21.15.47:0/4276857564 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891bbfd80 0x55f891b76010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd52800640 1 -- 172.21.15.47:0/4276857564 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891bbfd80 msgr2=0x55f891b76010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:06.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd52800640 1 --2- 172.21.15.47:0/4276857564 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891bbfd80 0x55f891b76010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd52800640 1 -- 172.21.15.47:0/4276857564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891b76720 msgr2=0x55f891b75ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:06.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd52800640 1 --2- 172.21.15.47:0/4276857564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891b76720 0x55f891b75ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd52800640 1 -- 172.21.15.47:0/4276857564 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f891983e70 con 0x55f891bc06a0 2024-02-19T15:26:06.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd51400640 1 --2- 172.21.15.47:0/4276857564 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891bbfd80 0x55f891b76010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:06.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd52800640 1 --2- 172.21.15.47:0/4276857564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 0x55f891baa230 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7efd4c004b40 tx=0x7efd4c020070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b5c8cc6c4cbead6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:06.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd3e000640 1 -- 172.21.15.47:0/4276857564 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4c007ed0 con 0x55f891bc06a0 2024-02-19T15:26:06.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd3e000640 1 -- 172.21.15.47:0/4276857564 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd4c022070 con 0x55f891bc06a0 2024-02-19T15:26:06.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd3e000640 1 -- 172.21.15.47:0/4276857564 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4c004120 con 0x55f891bc06a0 2024-02-19T15:26:06.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd5be46000 1 -- 172.21.15.47:0/4276857564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 msgr2=0x55f891baa230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:06.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd5be46000 1 --2- 172.21.15.47:0/4276857564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 0x55f891baa230 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7efd4c004b40 tx=0x7efd4c020070 comp rx=0 tx=0).stop 2024-02-19T15:26:06.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd5be46000 1 -- 172.21.15.47:0/4276857564 shutdown_connections 2024-02-19T15:26:06.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd5be46000 1 --2- 172.21.15.47:0/4276857564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 0x55f891baa230 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd5be46000 1 --2- 172.21.15.47:0/4276857564 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891bbfd80 0x55f891b76010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd5be46000 1 --2- 172.21.15.47:0/4276857564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891b76720 0x55f891b75ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd5be46000 1 -- 172.21.15.47:0/4276857564 >> 172.21.15.47:0/4276857564 conn(0x55f891b726d0 msgr2=0x55f891bcc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd5be46000 1 -- 172.21.15.47:0/4276857564 shutdown_connections 2024-02-19T15:26:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd5be46000 1 -- 172.21.15.47:0/4276857564 wait complete. 2024-02-19T15:26:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.096+0000 7efd5be46000 1 -- start start 2024-02-19T15:26:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd5be46000 1 Processor -- start 2024-02-19T15:26:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd5be46000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891b76720 0x55f891c249a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd5be46000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891bbfd80 0x55f891bd2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd5be46000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 0x55f891bd2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd5be46000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8919c2fd0 con 0x55f891bc06a0 2024-02-19T15:26:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd5be46000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f891b7cd20 con 0x55f891bbfd80 2024-02-19T15:26:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd5be46000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f8919cbe80 con 0x55f891b76720 2024-02-19T15:26:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd5be46000 10 client.0 _ll_register_callbacks cb 0x7efd4000b990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:26:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd52800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 0x55f891bd2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd52800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 0x55f891bd2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50688/0 (socket says 172.21.15.47:50688) 2024-02-19T15:26:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd52800640 1 -- 172.21.15.47:0/3400846081 learned_addr learned my addr 172.21.15.47:0/3400846081 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd51400640 1 --2- 172.21.15.47:0/3400846081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891bbfd80 0x55f891bd2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd51e00640 1 --2- 172.21.15.47:0/3400846081 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891b76720 0x55f891c249a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd52800640 1 -- 172.21.15.47:0/3400846081 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891b76720 msgr2=0x55f891c249a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd52800640 1 --2- 172.21.15.47:0/3400846081 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891b76720 0x55f891c249a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd52800640 1 -- 172.21.15.47:0/3400846081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891bbfd80 msgr2=0x55f891bd2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd52800640 1 --2- 172.21.15.47:0/3400846081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891bbfd80 0x55f891bd2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd52800640 1 -- 172.21.15.47:0/3400846081 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd4c009ce0 con 0x55f891bc06a0 2024-02-19T15:26:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd51400640 1 --2- 172.21.15.47:0/3400846081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891bbfd80 0x55f891bd2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd52800640 1 --2- 172.21.15.47:0/3400846081 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 0x55f891bd2ca0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7efd4c0078e0 tx=0x7efd4c009730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ce84a50992e0241 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd33400640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:26:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd33400640 1 -- 172.21.15.47:0/3400846081 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4c016240 con 0x55f891bc06a0 2024-02-19T15:26:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd33400640 1 -- 172.21.15.47:0/3400846081 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd4c024070 con 0x55f891bc06a0 2024-02-19T15:26:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd5be46000 1 -- 172.21.15.47:0/3400846081 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f891949e80 con 0x55f891bc06a0 2024-02-19T15:26:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd33400640 1 -- 172.21.15.47:0/3400846081 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd4c004460 con 0x55f891bc06a0 2024-02-19T15:26:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd5be46000 10 client.6918 fetch_fsmap learned FSMap version 49 2024-02-19T15:26:06.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd5be46000 1 -- 172.21.15.47:0/3400846081 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x55f891969e10 con 0x55f891bc06a0 2024-02-19T15:26:06.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd33400640 1 -- 172.21.15.47:0/3400846081 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f891949e80 con 0x55f891bc06a0 2024-02-19T15:26:06.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd33400640 1 -- 172.21.15.47:0/3400846081 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 49) v1 ==== 3294+0+0 (secure 0 0 0) 0x7efd4c0163a0 con 0x55f891bc06a0 2024-02-19T15:26:06.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd5be46000 10 client.6918 fetch_fsmap finished waiting for FSMap version 49 2024-02-19T15:26:06.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd5be46000 10 client.6918 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T15:26:06.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd5be46000 20 client.6918 populate_metadata read hostname 'smithi047' 2024-02-19T15:26:06.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd5be46000 1 --2- 172.21.15.47:0/3400846081 >> [v2:172.21.15.47:6834/3598443511,v1:172.21.15.47:6835/3598443511] conn(0x55f891bd31e0 0x55f891bd35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd5be46000 4 client.6918 mds_command: new command op to 6774 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T15:26:06.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd5be46000 1 -- 172.21.15.47:0/3400846081 --> [v2:172.21.15.47:6834/3598443511,v1:172.21.15.47:6835/3598443511] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f89198cfa0 con 0x55f891bd31e0 2024-02-19T15:26:06.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd51e00640 1 --2- 172.21.15.47:0/3400846081 >> [v2:172.21.15.47:6834/3598443511,v1:172.21.15.47:6835/3598443511] conn(0x55f891bd31e0 0x55f891bd35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd33400640 10 client.6918 ms_handle_connect on v2:172.21.15.47:6834/3598443511 2024-02-19T15:26:06.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd51e00640 1 --2- 172.21.15.47:0/3400846081 >> [v2:172.21.15.47:6834/3598443511,v1:172.21.15.47:6835/3598443511] conn(0x55f891bd31e0 0x55f891bd35c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fffb380f0c51fbb1 server_cookie=61e806200835d1ec in_seq=0 out_seq=0 2024-02-19T15:26:06.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd33400640 1 -- 172.21.15.47:0/3400846081 <== mds.0 v2:172.21.15.47:6834/3598443511 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55f89198cfa0 con 0x55f891bd31e0 2024-02-19T15:26:06.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd33400640 10 client.6918 handle_command_reply: tid=0 2024-02-19T15:26:06.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.100+0000 7efd5be46000 1 client.6918 shutdown 2024-02-19T15:26:06.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd5be46000 1 -- 172.21.15.47:0/3400846081 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 msgr2=0x55f891bd2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:06.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd5be46000 1 --2- 172.21.15.47:0/3400846081 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 0x55f891bd2ca0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7efd4c0078e0 tx=0x7efd4c009730 comp rx=0 tx=0).stop 2024-02-19T15:26:06.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd5be46000 1 -- 172.21.15.47:0/3400846081 shutdown_connections 2024-02-19T15:26:06.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd5be46000 1 --2- 172.21.15.47:0/3400846081 >> [v2:172.21.15.47:6834/3598443511,v1:172.21.15.47:6835/3598443511] conn(0x55f891bd31e0 0x55f891bd35c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd5be46000 1 --2- 172.21.15.47:0/3400846081 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f891bc06a0 0x55f891bd2ca0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd5be46000 1 --2- 172.21.15.47:0/3400846081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f891bbfd80 0x55f891bd2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd5be46000 1 --2- 172.21.15.47:0/3400846081 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f891b76720 0x55f891c249a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd33400640 0 client.6918 ms_handle_reset on v2:172.21.15.47:6834/3598443511 2024-02-19T15:26:06.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd5be46000 1 -- 172.21.15.47:0/3400846081 >> 172.21.15.47:0/3400846081 conn(0x55f891b726d0 msgr2=0x55f891bcc910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:06.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd5be46000 1 -- 172.21.15.47:0/3400846081 shutdown_connections 2024-02-19T15:26:06.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd5be46000 1 -- 172.21.15.47:0/3400846081 wait complete. 2024-02-19T15:26:06.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd5be46000 20 client.6918 trim_cache size 0 max 16384 2024-02-19T15:26:06.111 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:26:06.111 INFO:teuthology.orchestra.run.smithi047.stdout: "cluster_fsid": "ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e", 2024-02-19T15:26:06.111 INFO:teuthology.orchestra.run.smithi047.stdout: "whoami": 0, 2024-02-19T15:26:06.111 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 6774, 2024-02-19T15:26:06.111 INFO:teuthology.orchestra.run.smithi047.stdout: "want_state": "up:active", 2024-02-19T15:26:06.111 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "up:active", 2024-02-19T15:26:06.111 INFO:teuthology.orchestra.run.smithi047.stdout: "fs_name": "cephfs", 2024-02-19T15:26:06.111 INFO:teuthology.orchestra.run.smithi047.stdout: "rank_uptime": 5.9822663110000001, 2024-02-19T15:26:06.111 INFO:teuthology.orchestra.run.smithi047.stdout: "mdsmap_epoch": 49, 2024-02-19T15:26:06.111 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch": 82, 2024-02-19T15:26:06.111 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch_barrier": 82, 2024-02-19T15:26:06.112 INFO:teuthology.orchestra.run.smithi047.stdout: "uptime": 10.832209877 2024-02-19T15:26:06.112 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:26:06.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd58a00640 1 -- 172.21.15.47:0/2044375550 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd34066e40 msgr2=0x7efd34069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:06.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd58a00640 1 --2- 172.21.15.47:0/2044375550 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd34066e40 0x7efd34069300 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7efd400b1000 tx=0x7efd40008040 comp rx=0 tx=0).stop 2024-02-19T15:26:06.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd58a00640 1 -- 172.21.15.47:0/2044375550 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd54071350 msgr2=0x7efd540cbf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:06.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd58a00640 1 --2- 172.21.15.47:0/2044375550 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd54071350 0x7efd540cbf40 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7efd4400c420 tx=0x7efd4400c8e0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd58a00640 1 -- 172.21.15.47:0/2044375550 shutdown_connections 2024-02-19T15:26:06.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd58a00640 1 --2- 172.21.15.47:0/2044375550 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd34066e40 0x7efd34069300 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd58a00640 1 --2- 172.21.15.47:0/2044375550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd5415d6e0 0x7efd540cdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd58a00640 1 --2- 172.21.15.47:0/2044375550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd5415b1a0 0x7efd5414d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd58a00640 1 --2- 172.21.15.47:0/2044375550 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd54071350 0x7efd540cbf40 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd58a00640 1 -- 172.21.15.47:0/2044375550 >> 172.21.15.47:0/2044375550 conn(0x7efd540b16a0 msgr2=0x7efd5414e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:06.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd58a00640 1 -- 172.21.15.47:0/2044375550 shutdown_connections 2024-02-19T15:26:06.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.104+0000 7efd58a00640 1 -- 172.21.15.47:0/2044375550 wait complete. 2024-02-19T15:26:06.127 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-02-19T15:26:06.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.264+0000 7f6da4c00640 1 Processor -- start 2024-02-19T15:26:06.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.264+0000 7f6da4c00640 1 -- start start 2024-02-19T15:26:06.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.264+0000 7f6da4c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6da00d0540 0x7f6da00cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.264+0000 7f6da4c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6da00cd180 0x7f6da00b1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.264+0000 7f6da4c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6da00b0fe0 con 0x7f6da015c0d0 2024-02-19T15:26:06.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.264+0000 7f6da4c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6da015c4b0 con 0x7f6da00d0540 2024-02-19T15:26:06.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.264+0000 7f6da4c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da00b1880 con 0x7f6da00cd180 2024-02-19T15:26:06.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.264+0000 7f6d97e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6da00d0540 0x7f6da00cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.264+0000 7f6d97e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6da00d0540 0x7f6da00cf850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50690/0 (socket says 172.21.15.47:50690) 2024-02-19T15:26:06.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.264+0000 7f6d97e00640 1 -- 172.21.15.47:0/4036136768 learned_addr learned my addr 172.21.15.47:0/4036136768 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:06.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.268+0000 7f6d9e800640 1 --2- 172.21.15.47:0/4036136768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6da00cd180 0x7f6da00b1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.268+0000 7f6d97e00640 1 -- 172.21.15.47:0/4036136768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6da00cd180 msgr2=0x7f6da00b1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:06.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.268+0000 7f6d97e00640 1 --2- 172.21.15.47:0/4036136768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6da00cd180 0x7f6da00b1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.268+0000 7f6d97e00640 1 -- 172.21.15.47:0/4036136768 >> v1:172.21.15.116:6789/0 conn(0x7f6da015c0d0 legacy=0x7f6da00cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:06.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.268+0000 7f6d97e00640 1 -- 172.21.15.47:0/4036136768 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da00cd630 con 0x7f6da00d0540 2024-02-19T15:26:06.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.268+0000 7f6d97e00640 1 --2- 172.21.15.47:0/4036136768 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6da00d0540 0x7f6da00cf850 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f6d88009870 tx=0x7f6d8801c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd81febbecf139a2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:06.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.268+0000 7f6d9d400640 1 -- 172.21.15.47:0/4036136768 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d880a95f0 con 0x7f6da00d0540 2024-02-19T15:26:06.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.268+0000 7f6d9d400640 1 -- 172.21.15.47:0/4036136768 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d880a9750 con 0x7f6da00d0540 2024-02-19T15:26:06.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.268+0000 7f6d9d400640 1 -- 172.21.15.47:0/4036136768 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d880a9ae0 con 0x7f6da00d0540 2024-02-19T15:26:06.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6da4c00640 1 -- 172.21.15.47:0/4036136768 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6da00d0540 msgr2=0x7f6da00cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:06.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6da4c00640 1 --2- 172.21.15.47:0/4036136768 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6da00d0540 0x7f6da00cf850 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f6d88009870 tx=0x7f6d8801c530 comp rx=0 tx=0).stop 2024-02-19T15:26:06.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6da4c00640 1 -- 172.21.15.47:0/4036136768 shutdown_connections 2024-02-19T15:26:06.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6da4c00640 1 --2- 172.21.15.47:0/4036136768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6da00cd180 0x7f6da00b1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6da4c00640 1 --2- 172.21.15.47:0/4036136768 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6da00d0540 0x7f6da00cf850 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6da4c00640 1 -- 172.21.15.47:0/4036136768 >> 172.21.15.47:0/4036136768 conn(0x7f6da00d1ba0 msgr2=0x7f6da00d1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:06.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6da4c00640 1 -- 172.21.15.47:0/4036136768 shutdown_connections 2024-02-19T15:26:06.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6da4c00640 1 -- 172.21.15.47:0/4036136768 wait complete. 2024-02-19T15:26:06.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6da4c00640 1 Processor -- start 2024-02-19T15:26:06.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6da4c00640 1 -- start start 2024-02-19T15:26:06.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6da4c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6da00cd180 0x7f6da015a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6da4c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6da00d0540 0x7f6da00af8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6da4c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da015c0d0 0x7f6da00afe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6da4c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da015ad20 con 0x7f6da00d0540 2024-02-19T15:26:06.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6da4c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da01f98b0 con 0x7f6da015c0d0 2024-02-19T15:26:06.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6da4c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da01f7080 con 0x7f6da00cd180 2024-02-19T15:26:06.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6d97e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6da00d0540 0x7f6da00af8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6d97e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6da00d0540 0x7f6da00af8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50704/0 (socket says 172.21.15.47:50704) 2024-02-19T15:26:06.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6d97e00640 1 -- 172.21.15.47:0/2116213690 learned_addr learned my addr 172.21.15.47:0/2116213690 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:06.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6d9de00640 1 --2- 172.21.15.47:0/2116213690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6da00cd180 0x7f6da015a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6d9e800640 1 --2- 172.21.15.47:0/2116213690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da015c0d0 0x7f6da00afe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6d97e00640 1 -- 172.21.15.47:0/2116213690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6da00cd180 msgr2=0x7f6da015a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:06.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6d97e00640 1 --2- 172.21.15.47:0/2116213690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6da00cd180 0x7f6da015a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6d97e00640 1 -- 172.21.15.47:0/2116213690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da015c0d0 msgr2=0x7f6da00afe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:06.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6d97e00640 1 --2- 172.21.15.47:0/2116213690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da015c0d0 0x7f6da00afe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.272+0000 7f6d97e00640 1 -- 172.21.15.47:0/2116213690 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d880094d0 con 0x7f6da00d0540 2024-02-19T15:26:06.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.276+0000 7f6d97e00640 1 --2- 172.21.15.47:0/2116213690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6da00d0540 0x7f6da00af8c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f6d880ab040 tx=0x7f6d8801ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:06.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.276+0000 7f6d96000640 1 -- 172.21.15.47:0/2116213690 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d880b1800 con 0x7f6da00d0540 2024-02-19T15:26:06.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.276+0000 7f6d96000640 1 -- 172.21.15.47:0/2116213690 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d880b1960 con 0x7f6da00d0540 2024-02-19T15:26:06.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.276+0000 7f6d96000640 1 -- 172.21.15.47:0/2116213690 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d880b1d10 con 0x7f6da00d0540 2024-02-19T15:26:06.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.276+0000 7f6da4c00640 1 -- 172.21.15.47:0/2116213690 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6da01f9b00 con 0x7f6da00d0540 2024-02-19T15:26:06.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.276+0000 7f6da4c00640 1 -- 172.21.15.47:0/2116213690 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da01fa0c0 con 0x7f6da00d0540 2024-02-19T15:26:06.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.276+0000 7f6da4c00640 1 -- 172.21.15.47:0/2116213690 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6da0068500 con 0x7f6da00d0540 2024-02-19T15:26:06.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.276+0000 7f6d96000640 1 -- 172.21.15.47:0/2116213690 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6d880bc450 con 0x7f6da00d0540 2024-02-19T15:26:06.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.276+0000 7f6d96000640 1 --2- 172.21.15.47:0/2116213690 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6d7c066e40 0x7f6d7c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:06.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.276+0000 7f6d96000640 1 -- 172.21.15.47:0/2116213690 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d8811f230 con 0x7f6da00d0540 2024-02-19T15:26:06.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.276+0000 7f6d9de00640 1 --2- 172.21.15.47:0/2116213690 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6d7c066e40 0x7f6d7c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:06.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.280+0000 7f6d9de00640 1 --2- 172.21.15.47:0/2116213690 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6d7c066e40 0x7f6d7c069300 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6da006b4a0 tx=0x7f6d90009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:06.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.280+0000 7f6d96000640 1 -- 172.21.15.47:0/2116213690 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6d880ec160 con 0x7f6da00d0540 2024-02-19T15:26:06.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.480+0000 7f6da4c00640 1 -- 172.21.15.47:0/2116213690 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6da0165c50 con 0x7f6da00d0540 2024-02-19T15:26:06.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.480+0000 7f6d96000640 1 -- 172.21.15.47:0/2116213690 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v49) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6d880ef820 con 0x7f6da00d0540 2024-02-19T15:26:06.484 INFO:teuthology.orchestra.run.smithi047.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-02-19T15:26:06.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.480+0000 7f6da4c00640 1 -- 172.21.15.47:0/2116213690 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6d7c066e40 msgr2=0x7f6d7c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:06.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.480+0000 7f6da4c00640 1 --2- 172.21.15.47:0/2116213690 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6d7c066e40 0x7f6d7c069300 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6da006b4a0 tx=0x7f6d90009040 comp rx=0 tx=0).stop 2024-02-19T15:26:06.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.480+0000 7f6da4c00640 1 -- 172.21.15.47:0/2116213690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6da00d0540 msgr2=0x7f6da00af8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:06.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.480+0000 7f6da4c00640 1 --2- 172.21.15.47:0/2116213690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6da00d0540 0x7f6da00af8c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f6d880ab040 tx=0x7f6d8801ad70 comp rx=0 tx=0).stop 2024-02-19T15:26:06.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.480+0000 7f6da4c00640 1 -- 172.21.15.47:0/2116213690 shutdown_connections 2024-02-19T15:26:06.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.480+0000 7f6da4c00640 1 --2- 172.21.15.47:0/2116213690 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6d7c066e40 0x7f6d7c069300 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.480+0000 7f6da4c00640 1 --2- 172.21.15.47:0/2116213690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da015c0d0 0x7f6da00afe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.480+0000 7f6da4c00640 1 --2- 172.21.15.47:0/2116213690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6da00d0540 0x7f6da00af8c0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.480+0000 7f6da4c00640 1 --2- 172.21.15.47:0/2116213690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6da00cd180 0x7f6da015a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:06.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.480+0000 7f6da4c00640 1 -- 172.21.15.47:0/2116213690 >> 172.21.15.47:0/2116213690 conn(0x7f6da00d1ba0 msgr2=0x7f6da006bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:06.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.480+0000 7f6da4c00640 1 -- 172.21.15.47:0/2116213690 shutdown_connections 2024-02-19T15:26:06.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:06.480+0000 7f6da4c00640 1 -- 172.21.15.47:0/2116213690 wait complete. 2024-02-19T15:26:06.507 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-02-19T15:26:06.507 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T15:26:06.507 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi047.front.sepia.ceph.com 2024-02-19T15:26:06.507 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-19T15:26:06.507 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-19T15:26:06.507 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-02-19T15:26:06.507 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-02-19T15:26:06.507 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T15:26:06.507 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:26:06.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:06.508 DEBUG:teuthology.orchestra.run.smithi047:> ip addr 2024-02-19T15:26:06.517 INFO:teuthology.orchestra.run.smithi047.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T15:26:06.517 INFO:teuthology.orchestra.run.smithi047.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T15:26:06.517 INFO:teuthology.orchestra.run.smithi047.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T15:26:06.517 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:26:06.517 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 ::1/128 scope host 2024-02-19T15:26:06.518 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:26:06.518 INFO:teuthology.orchestra.run.smithi047.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:26:06.518 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:78 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:26:06.518 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f0 2024-02-19T15:26:06.518 INFO:teuthology.orchestra.run.smithi047.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T15:26:06.518 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8e brd ff:ff:ff:ff:ff:ff 2024-02-19T15:26:06.518 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f0 2024-02-19T15:26:06.518 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.21.15.47/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-02-19T15:26:06.518 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft 42248sec preferred_lft 42248sec 2024-02-19T15:26:06.518 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::ec4:7aff:febd:148e/64 scope link 2024-02-19T15:26:06.518 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:26:06.518 INFO:teuthology.orchestra.run.smithi047.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:26:06.518 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:79 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:26:06.518 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f1 2024-02-19T15:26:06.518 INFO:teuthology.orchestra.run.smithi047.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:26:06.519 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8f brd ff:ff:ff:ff:ff:ff 2024-02-19T15:26:06.519 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f1 2024-02-19T15:26:06.519 INFO:teuthology.orchestra.run.smithi047.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-02-19T15:26:06.519 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 02:42:47:cf:b1:62 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:26:06.519 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-02-19T15:26:06.519 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:26:06.519 INFO:teuthology.orchestra.run.smithi047.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-02-19T15:26:06.519 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-02-19T15:26:06.519 INFO:teuthology.orchestra.run.smithi047.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-02-19T15:26:06.519 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:26:06.519 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-02-19T15:26:06.519 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:26:06.519 INFO:teuthology.orchestra.run.smithi047.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:26:06.519 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-19T15:26:06.520 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-02-19T15:26:06.520 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:26:06.520 INFO:teuthology.orchestra.run.smithi047.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:26:06.520 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-02-19T15:26:06.520 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-02-19T15:26:06.520 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:26:06.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:06.520 DEBUG:teuthology.orchestra.run.smithi047:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T15:26:06.575 INFO:teuthology.orchestra.run.smithi047.stdout:1 2024-02-19T15:26:06.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:06.578 DEBUG:teuthology.orchestra.run.smithi047:> route 2024-02-19T15:26:06.627 INFO:teuthology.orchestra.run.smithi047.stdout:Kernel IP routing table 2024-02-19T15:26:06.627 INFO:teuthology.orchestra.run.smithi047.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T15:26:06.627 INFO:teuthology.orchestra.run.smithi047.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-02-19T15:26:06.627 INFO:teuthology.orchestra.run.smithi047.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-02-19T15:26:06.627 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-02-19T15:26:06.628 INFO:teuthology.orchestra.run.smithi047.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T15:26:06.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:06.631 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2024-02-19T15:26:06.631 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2024-02-19T15:26:06.631 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-02-19T15:26:06.631 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-02-19T15:26:06.631 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-02-19T15:26:06.631 DEBUG:teuthology.orchestra.run.smithi047:> ') 2024-02-19T15:26:06.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:06.719 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list 2024-02-19T15:26:06.727 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-02-19T15:26:06.728 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-02-19T15:26:06.728 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:26:06.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:06.729 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set brx.0 up 2024-02-19T15:26:06.790 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:26:06.790 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-19T15:26:06.838 INFO:teuthology.orchestra.run.smithi047.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:26:06.838 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:26:06.838 DEBUG:teuthology.orchestra.run.smithi047:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:26:06.889 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-19T15:26:06.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:06.889 DEBUG:teuthology.orchestra.run.smithi047:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:26:06.978 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:26:06.978 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:26:06.978 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:26:06.993 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:26:06.993 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:26:06.993 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:26:06.993 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:26:06.993 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:26:07.003 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:26:07.004 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:26:07.004 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:26:07.004 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:26:07.004 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:26:07.004 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790" 2024-02-19T15:26:07.004 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790:/ 2024-02-19T15:26:07.004 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e" will pass to kernel 2024-02-19T15:26:07.006 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T15:26:07.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:07.007 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:26:07.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:07.026 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:26:07.089 INFO:teuthology.orchestra.run.smithi047.stdout:client6927 2024-02-19T15:26:07.090 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:26:07.090 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:26:07.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:07.138 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch fancy_layout_file) 2024-02-19T15:26:07.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:07.201 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout -v 'stripe_unit=1048576 stripe_count=4 object_size=8388608' fancy_layout_file) 2024-02-19T15:26:07.262 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:26:07.263 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:26:07.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:07.314 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=fancy_layout_file bs=1M conv=fdatasync count=35 seek=0) 2024-02-19T15:26:07.762 INFO:teuthology.orchestra.run.smithi047.stderr:35+0 records in 2024-02-19T15:26:07.763 INFO:teuthology.orchestra.run.smithi047.stderr:35+0 records out 2024-02-19T15:26:07.763 INFO:teuthology.orchestra.run.smithi047.stderr:36700160 bytes (37 MB, 35 MiB) copied, 0.394083 s, 93.1 MB/s 2024-02-19T15:26:07.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:07.764 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f fancy_layout_file) 2024-02-19T15:26:07.774 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:26:07.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.952+0000 7f6d56800640 1 Processor -- start 2024-02-19T15:26:07.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.952+0000 7f6d56800640 1 -- start start 2024-02-19T15:26:07.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.956+0000 7f6d56800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6d50145590 0x7f6d500727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:07.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.956+0000 7f6d56800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d5015c390 0x7f6d5015c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:07.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.956+0000 7f6d56800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d5015d280 con 0x7f6d500cea80 2024-02-19T15:26:07.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.956+0000 7f6d56800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d50070ff0 con 0x7f6d50145590 2024-02-19T15:26:07.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.956+0000 7f6d56800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5015cd60 con 0x7f6d5015c390 2024-02-19T15:26:07.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.956+0000 7f6d4f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6d50145590 0x7f6d500727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:07.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.956+0000 7f6d4f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6d50145590 0x7f6d500727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50712/0 (socket says 172.21.15.47:50712) 2024-02-19T15:26:07.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.956+0000 7f6d4f400640 1 -- 172.21.15.47:0/3360731412 learned_addr learned my addr 172.21.15.47:0/3360731412 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:07.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.956+0000 7f6d4fe00640 1 --2- 172.21.15.47:0/3360731412 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d5015c390 0x7f6d5015c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:07.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.956+0000 7f6d4f400640 1 -- 172.21.15.47:0/3360731412 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d5015c390 msgr2=0x7f6d5015c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:07.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.956+0000 7f6d4f400640 1 --2- 172.21.15.47:0/3360731412 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d5015c390 0x7f6d5015c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:07.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.956+0000 7f6d4f400640 1 -- 172.21.15.47:0/3360731412 >> v1:172.21.15.116:6789/0 conn(0x7f6d500cea80 legacy=0x7f6d5006c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:07.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.956+0000 7f6d4f400640 1 -- 172.21.15.47:0/3360731412 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d500709e0 con 0x7f6d50145590 2024-02-19T15:26:07.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.960+0000 7f6d4f400640 1 --2- 172.21.15.47:0/3360731412 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d50145590 0x7f6d500727b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f6d40009870 tx=0x7f6d4001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558f32a1024e63e6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:07.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.960+0000 7f6d4ea00640 1 -- 172.21.15.47:0/3360731412 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d400ab650 con 0x7f6d50145590 2024-02-19T15:26:07.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.960+0000 7f6d4ea00640 1 -- 172.21.15.47:0/3360731412 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d400ab7b0 con 0x7f6d50145590 2024-02-19T15:26:07.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.960+0000 7f6d4ea00640 1 -- 172.21.15.47:0/3360731412 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d400abb40 con 0x7f6d50145590 2024-02-19T15:26:07.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.960+0000 7f6d56800640 1 -- 172.21.15.47:0/3360731412 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d50145590 msgr2=0x7f6d500727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:07.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.960+0000 7f6d56800640 1 --2- 172.21.15.47:0/3360731412 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d50145590 0x7f6d500727b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f6d40009870 tx=0x7f6d4001c560 comp rx=0 tx=0).stop 2024-02-19T15:26:07.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.960+0000 7f6d56800640 1 -- 172.21.15.47:0/3360731412 shutdown_connections 2024-02-19T15:26:07.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.960+0000 7f6d56800640 1 --2- 172.21.15.47:0/3360731412 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d5015c390 0x7f6d5015c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:07.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.960+0000 7f6d56800640 1 --2- 172.21.15.47:0/3360731412 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d50145590 0x7f6d500727b0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:07.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.960+0000 7f6d56800640 1 -- 172.21.15.47:0/3360731412 >> 172.21.15.47:0/3360731412 conn(0x7f6d500b16a0 msgr2=0x7f6d500b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:07.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.960+0000 7f6d56800640 1 -- 172.21.15.47:0/3360731412 shutdown_connections 2024-02-19T15:26:07.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.960+0000 7f6d56800640 1 -- 172.21.15.47:0/3360731412 wait complete. 2024-02-19T15:26:07.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.960+0000 7f6d56800640 1 Processor -- start 2024-02-19T15:26:07.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.960+0000 7f6d56800640 1 -- start start 2024-02-19T15:26:07.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d56800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d500cea80 0x7f6d50157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:07.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d56800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d50145590 0x7f6d50140c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:07.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d56800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d5015c390 0x7f6d5013cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:07.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d56800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d50144690 con 0x7f6d5015c390 2024-02-19T15:26:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d56800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d50144e60 con 0x7f6d50145590 2024-02-19T15:26:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d56800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5015bed0 con 0x7f6d500cea80 2024-02-19T15:26:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d4fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d5015c390 0x7f6d5013cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d4fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d5015c390 0x7f6d5013cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50726/0 (socket says 172.21.15.47:50726) 2024-02-19T15:26:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d4fe00640 1 -- 172.21.15.47:0/613200660 learned_addr learned my addr 172.21.15.47:0/613200660 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d4f400640 1 --2- 172.21.15.47:0/613200660 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d500cea80 0x7f6d50157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d46a00640 1 --2- 172.21.15.47:0/613200660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d50145590 0x7f6d50140c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d4fe00640 1 -- 172.21.15.47:0/613200660 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d500cea80 msgr2=0x7f6d50157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d4fe00640 1 --2- 172.21.15.47:0/613200660 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d500cea80 0x7f6d50157b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d4fe00640 1 -- 172.21.15.47:0/613200660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d50145590 msgr2=0x7f6d50140c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d4fe00640 1 --2- 172.21.15.47:0/613200660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d50145590 0x7f6d50140c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d4fe00640 1 -- 172.21.15.47:0/613200660 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d400094d0 con 0x7f6d5015c390 2024-02-19T15:26:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d46a00640 1 --2- 172.21.15.47:0/613200660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d50145590 0x7f6d50140c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d4fe00640 1 --2- 172.21.15.47:0/613200660 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d5015c390 0x7f6d5013cae0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f6d48007e60 tx=0x7f6d4800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:07.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d47e00640 1 -- 172.21.15.47:0/613200660 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d48014420 con 0x7f6d5015c390 2024-02-19T15:26:07.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d47e00640 1 -- 172.21.15.47:0/613200660 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d480145a0 con 0x7f6d5015c390 2024-02-19T15:26:07.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d56800640 1 -- 172.21.15.47:0/613200660 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d501411d0 con 0x7f6d5015c390 2024-02-19T15:26:07.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d47e00640 1 -- 172.21.15.47:0/613200660 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d48014950 con 0x7f6d5015c390 2024-02-19T15:26:07.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d56800640 1 -- 172.21.15.47:0/613200660 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d500b2b60 con 0x7f6d5015c390 2024-02-19T15:26:07.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.964+0000 7f6d56800640 1 -- 172.21.15.47:0/613200660 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d500b2970 con 0x7f6d5015c390 2024-02-19T15:26:07.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.968+0000 7f6d47e00640 1 -- 172.21.15.47:0/613200660 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6d48014ab0 con 0x7f6d5015c390 2024-02-19T15:26:07.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.968+0000 7f6d47e00640 1 --2- 172.21.15.47:0/613200660 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6d2c066e40 0x7f6d2c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:07.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.968+0000 7f6d47e00640 1 -- 172.21.15.47:0/613200660 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d480851e0 con 0x7f6d5015c390 2024-02-19T15:26:07.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.968+0000 7f6d4f400640 1 --2- 172.21.15.47:0/613200660 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6d2c066e40 0x7f6d2c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:07.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.968+0000 7f6d4f400640 1 --2- 172.21.15.47:0/613200660 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6d2c066e40 0x7f6d2c069300 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6d400af000 tx=0x7f6d400b6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:07.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:07.976+0000 7f6d47e00640 1 -- 172.21.15.47:0/613200660 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6d500b2970 con 0x7f6d5015c390 2024-02-19T15:26:08.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.188+0000 7f6d56800640 1 -- 172.21.15.47:0/613200660 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d500b2970 con 0x7f6d5015c390 2024-02-19T15:26:08.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.188+0000 7f6d47e00640 1 -- 172.21.15.47:0/613200660 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4943 (secure 0 0 0) 0x7f6d500b2970 con 0x7f6d5015c390 2024-02-19T15:26:08.194 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:08.194 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5954,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1402677843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1402677843},{"type":"v1","addr":"172.21.15.116:6836","nonce":1402677843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":5957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1507554634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":1507554634},{"type":"v1","addr":"172.21.15.116:6837","nonce":1507554634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6771,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/888455604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":888455604},{"type":"v1","addr":"172.21.15.47:6833","nonce":888455604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:25:59.522224+0000","modified":"2024-02-19T15:26:02.535882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6774":{"gid":6774,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3598443511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3598443511},{"type":"v1","addr":"172.21.15.47:6835","nonce":3598443511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-19T15:26:08.194 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 49 2024-02-19T15:26:08.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.192+0000 7f6d56800640 1 -- 172.21.15.47:0/613200660 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6d2c066e40 msgr2=0x7f6d2c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:08.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.192+0000 7f6d56800640 1 --2- 172.21.15.47:0/613200660 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6d2c066e40 0x7f6d2c069300 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6d400af000 tx=0x7f6d400b6040 comp rx=0 tx=0).stop 2024-02-19T15:26:08.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.192+0000 7f6d56800640 1 -- 172.21.15.47:0/613200660 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d5015c390 msgr2=0x7f6d5013cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:08.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.192+0000 7f6d56800640 1 --2- 172.21.15.47:0/613200660 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d5015c390 0x7f6d5013cae0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f6d48007e60 tx=0x7f6d4800e4b0 comp rx=0 tx=0).stop 2024-02-19T15:26:08.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.192+0000 7f6d56800640 1 -- 172.21.15.47:0/613200660 shutdown_connections 2024-02-19T15:26:08.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.192+0000 7f6d56800640 1 --2- 172.21.15.47:0/613200660 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6d2c066e40 0x7f6d2c069300 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:08.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.192+0000 7f6d56800640 1 --2- 172.21.15.47:0/613200660 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d5015c390 0x7f6d5013cae0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:08.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.192+0000 7f6d56800640 1 --2- 172.21.15.47:0/613200660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d50145590 0x7f6d50140c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:08.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.192+0000 7f6d56800640 1 --2- 172.21.15.47:0/613200660 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d500cea80 0x7f6d50157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:08.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.192+0000 7f6d56800640 1 -- 172.21.15.47:0/613200660 >> 172.21.15.47:0/613200660 conn(0x7f6d500b16a0 msgr2=0x7f6d501419d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:08.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.192+0000 7f6d56800640 1 -- 172.21.15.47:0/613200660 shutdown_connections 2024-02-19T15:26:08.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.192+0000 7f6d56800640 1 -- 172.21.15.47:0/613200660 wait complete. 2024-02-19T15:26:08.216 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-02-19T15:26:08.385 INFO:teuthology.orchestra.run.smithi047.stdout:{"message":"","return_code":0} 2024-02-19T15:26:08.386 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-02-19T15:26:08.386 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-02-19T15:26:08.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.520+0000 7fcb79a00640 1 Processor -- start 2024-02-19T15:26:08.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.520+0000 7fcb79a00640 1 -- start start 2024-02-19T15:26:08.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.520+0000 7fcb79a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcb740d0360 0x7fcb7415d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:08.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.520+0000 7fcb79a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb740cc700 con 0x7fcb7415bcd0 2024-02-19T15:26:08.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.520+0000 7fcb79a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb740cf300 con 0x7fcb740cedb0 2024-02-19T15:26:08.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.520+0000 7fcb79a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb740cf5e0 con 0x7fcb740d0360 2024-02-19T15:26:08.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.524+0000 7fcb72a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fcb7415bcd0 0x7fcb740cd480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54890/0 (socket says 172.21.15.47:54890) 2024-02-19T15:26:08.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.524+0000 7fcb72a00640 1 -- 172.21.15.47:0/660418056 learned_addr learned my addr 172.21.15.47:0/660418056 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:08.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.524+0000 7fcb73400640 1 --2- 172.21.15.47:0/660418056 >> v2:172.21.15.116:3300/0 conn(0x7fcb740d0360 0x7fcb7415d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:08.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.524+0000 7fcb72000640 1 -- 172.21.15.47:0/660418056 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986403686 0 0) 0x7fcb740cc700 con 0x7fcb7415bcd0 2024-02-19T15:26:08.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.524+0000 7fcb72000640 1 -- 172.21.15.47:0/660418056 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb58003530 con 0x7fcb7415bcd0 2024-02-19T15:26:08.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.524+0000 7fcb72000640 1 -- 172.21.15.47:0/660418056 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264985826 0 0) 0x7fcb740cf300 con 0x7fcb740cedb0 2024-02-19T15:26:08.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.524+0000 7fcb72000640 1 -- 172.21.15.47:0/660418056 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb740cc700 con 0x7fcb740cedb0 2024-02-19T15:26:08.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.524+0000 7fcb72000640 1 -- 172.21.15.47:0/660418056 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4197541050 0 0) 0x7fcb58003530 con 0x7fcb7415bcd0 2024-02-19T15:26:08.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.524+0000 7fcb73400640 1 -- 172.21.15.47:0/660418056 >> v1:172.21.15.116:6789/0 conn(0x7fcb740cedb0 legacy=0x7fcb740cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:08.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.524+0000 7fcb73400640 1 -- 172.21.15.47:0/660418056 >> v1:172.21.15.47:6789/0 conn(0x7fcb7415bcd0 legacy=0x7fcb740cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:08.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.524+0000 7fcb73400640 1 -- 172.21.15.47:0/660418056 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb740cc9a0 con 0x7fcb740d0360 2024-02-19T15:26:08.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb73400640 1 --2- 172.21.15.47:0/660418056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb740d0360 0x7fcb7415d4f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c009d90 tx=0x7fcb6c01c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25e19522e2866da0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:08.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb72000640 1 -- 172.21.15.47:0/660418056 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb64001600 con 0x7fcb740d0360 2024-02-19T15:26:08.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb72000640 1 -- 172.21.15.47:0/660418056 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6c0a56a0 con 0x7fcb740d0360 2024-02-19T15:26:08.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb72000640 1 -- 172.21.15.47:0/660418056 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c0a5a30 con 0x7fcb740d0360 2024-02-19T15:26:08.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb79a00640 1 -- 172.21.15.47:0/660418056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb740d0360 msgr2=0x7fcb7415d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:08.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb79a00640 1 --2- 172.21.15.47:0/660418056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb740d0360 0x7fcb7415d4f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c009d90 tx=0x7fcb6c01c860 comp rx=0 tx=0).stop 2024-02-19T15:26:08.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb79a00640 1 -- 172.21.15.47:0/660418056 shutdown_connections 2024-02-19T15:26:08.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb79a00640 1 --2- 172.21.15.47:0/660418056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb740d0360 0x7fcb7415d4f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:08.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb79a00640 1 -- 172.21.15.47:0/660418056 >> 172.21.15.47:0/660418056 conn(0x7fcb740d1bf0 msgr2=0x7fcb740d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:08.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb79a00640 1 -- 172.21.15.47:0/660418056 shutdown_connections 2024-02-19T15:26:08.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb79a00640 1 -- 172.21.15.47:0/660418056 wait complete. 2024-02-19T15:26:08.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb79a00640 1 Processor -- start 2024-02-19T15:26:08.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb79a00640 1 -- start start 2024-02-19T15:26:08.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb79a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb740cedb0 0x7fcb74159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:08.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb79a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb740d0360 0x7fcb740c7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:08.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb79a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb7415bcd0 0x7fcb740c5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:08.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb79a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb740cdb90 con 0x7fcb740d0360 2024-02-19T15:26:08.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb79a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb740cbf90 con 0x7fcb7415bcd0 2024-02-19T15:26:08.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb79a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb740cbcb0 con 0x7fcb740cedb0 2024-02-19T15:26:08.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb6a000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb740d0360 0x7fcb740c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:08.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb6a000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb740d0360 0x7fcb740c7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50736/0 (socket says 172.21.15.47:50736) 2024-02-19T15:26:08.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.528+0000 7fcb6a000640 1 -- 172.21.15.47:0/3508845593 learned_addr learned my addr 172.21.15.47:0/3508845593 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:08.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.532+0000 7fcb73400640 1 --2- 172.21.15.47:0/3508845593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb7415bcd0 0x7fcb740c5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:08.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.532+0000 7fcb72a00640 1 --2- 172.21.15.47:0/3508845593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb740cedb0 0x7fcb74159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:08.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.532+0000 7fcb6a000640 1 -- 172.21.15.47:0/3508845593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb740cedb0 msgr2=0x7fcb74159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:08.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.532+0000 7fcb6a000640 1 --2- 172.21.15.47:0/3508845593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb740cedb0 0x7fcb74159770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:08.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.532+0000 7fcb6a000640 1 -- 172.21.15.47:0/3508845593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb7415bcd0 msgr2=0x7fcb740c5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:08.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.532+0000 7fcb6a000640 1 --2- 172.21.15.47:0/3508845593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb7415bcd0 0x7fcb740c5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:08.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.532+0000 7fcb6a000640 1 -- 172.21.15.47:0/3508845593 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c009a70 con 0x7fcb740d0360 2024-02-19T15:26:08.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.532+0000 7fcb72a00640 1 --2- 172.21.15.47:0/3508845593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb740cedb0 0x7fcb74159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:08.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.532+0000 7fcb6a000640 1 --2- 172.21.15.47:0/3508845593 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb740d0360 0x7fcb740c7990 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00c420 tx=0x7fcb5c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:08.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.532+0000 7fcb6b400640 1 -- 172.21.15.47:0/3508845593 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c0052d0 con 0x7fcb740d0360 2024-02-19T15:26:08.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.532+0000 7fcb6b400640 1 -- 172.21.15.47:0/3508845593 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5c005430 con 0x7fcb740d0360 2024-02-19T15:26:08.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.532+0000 7fcb79a00640 1 -- 172.21.15.47:0/3508845593 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb7415d240 con 0x7fcb740d0360 2024-02-19T15:26:08.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.532+0000 7fcb6b400640 1 -- 172.21.15.47:0/3508845593 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c003840 con 0x7fcb740d0360 2024-02-19T15:26:08.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.532+0000 7fcb79a00640 1 -- 172.21.15.47:0/3508845593 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb74159cb0 con 0x7fcb740d0360 2024-02-19T15:26:08.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.532+0000 7fcb79a00640 1 -- 172.21.15.47:0/3508845593 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb74075360 con 0x7fcb740d0360 2024-02-19T15:26:08.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.532+0000 7fcb6b400640 1 -- 172.21.15.47:0/3508845593 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fcb5c0039a0 con 0x7fcb740d0360 2024-02-19T15:26:08.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.536+0000 7fcb6b400640 1 --2- 172.21.15.47:0/3508845593 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcb50066e40 0x7fcb50069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:08.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.536+0000 7fcb6b400640 1 -- 172.21.15.47:0/3508845593 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb5c084100 con 0x7fcb740d0360 2024-02-19T15:26:08.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.536+0000 7fcb72a00640 1 --2- 172.21.15.47:0/3508845593 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcb50066e40 0x7fcb50069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:08.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.536+0000 7fcb72a00640 1 --2- 172.21.15.47:0/3508845593 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcb50066e40 0x7fcb50069300 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fcb64005890 tx=0x7fcb64009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:08.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.536+0000 7fcb6b400640 1 -- 172.21.15.47:0/3508845593 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fcb5c050e50 con 0x7fcb740d0360 2024-02-19T15:26:08.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.836+0000 7fcb79a00640 1 -- 172.21.15.47:0/3508845593 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7fcb740c9210 con 0x7fcb740d0360 2024-02-19T15:26:08.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.836+0000 7fcb6b400640 1 -- 172.21.15.47:0/3508845593 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1653 (secure 0 0 0) 0x7fcb5c054510 con 0x7fcb740d0360 2024-02-19T15:26:08.842 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:08.842 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:26:08.842 INFO:teuthology.orchestra.run.smithi047.stdout: "stats": { 2024-02-19T15:26:08.842 INFO:teuthology.orchestra.run.smithi047.stdout: "total_bytes": 773094113280, 2024-02-19T15:26:08.842 INFO:teuthology.orchestra.run.smithi047.stdout: "total_avail_bytes": 772825120768, 2024-02-19T15:26:08.842 INFO:teuthology.orchestra.run.smithi047.stdout: "total_used_bytes": 268992512, 2024-02-19T15:26:08.842 INFO:teuthology.orchestra.run.smithi047.stdout: "total_used_raw_bytes": 268992512, 2024-02-19T15:26:08.843 INFO:teuthology.orchestra.run.smithi047.stdout: "total_used_raw_ratio": 0.00034794278326444328, 2024-02-19T15:26:08.843 INFO:teuthology.orchestra.run.smithi047.stdout: "num_osds": 8, 2024-02-19T15:26:08.843 INFO:teuthology.orchestra.run.smithi047.stdout: "num_per_pool_osds": 8, 2024-02-19T15:26:08.843 INFO:teuthology.orchestra.run.smithi047.stdout: "num_per_pool_omap_osds": 8 2024-02-19T15:26:08.843 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:08.843 INFO:teuthology.orchestra.run.smithi047.stdout: "stats_by_class": { 2024-02-19T15:26:08.843 INFO:teuthology.orchestra.run.smithi047.stdout: "ssd": { 2024-02-19T15:26:08.843 INFO:teuthology.orchestra.run.smithi047.stdout: "total_bytes": 773094113280, 2024-02-19T15:26:08.843 INFO:teuthology.orchestra.run.smithi047.stdout: "total_avail_bytes": 772825120768, 2024-02-19T15:26:08.843 INFO:teuthology.orchestra.run.smithi047.stdout: "total_used_bytes": 268992512, 2024-02-19T15:26:08.843 INFO:teuthology.orchestra.run.smithi047.stdout: "total_used_raw_bytes": 268992512, 2024-02-19T15:26:08.843 INFO:teuthology.orchestra.run.smithi047.stdout: "total_used_raw_ratio": 0.00034794278326444328 2024-02-19T15:26:08.843 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:08.843 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:08.843 INFO:teuthology.orchestra.run.smithi047.stdout: "pools": [ 2024-02-19T15:26:08.844 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:08.844 INFO:teuthology.orchestra.run.smithi047.stdout: "name": ".mgr", 2024-02-19T15:26:08.844 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 1, 2024-02-19T15:26:08.844 INFO:teuthology.orchestra.run.smithi047.stdout: "stats": { 2024-02-19T15:26:08.844 INFO:teuthology.orchestra.run.smithi047.stdout: "stored": 590368, 2024-02-19T15:26:08.844 INFO:teuthology.orchestra.run.smithi047.stdout: "objects": 2, 2024-02-19T15:26:08.844 INFO:teuthology.orchestra.run.smithi047.stdout: "kb_used": 1160, 2024-02-19T15:26:08.844 INFO:teuthology.orchestra.run.smithi047.stdout: "bytes_used": 1187840, 2024-02-19T15:26:08.844 INFO:teuthology.orchestra.run.smithi047.stdout: "percent_used": 1.6180058537429431e-06, 2024-02-19T15:26:08.844 INFO:teuthology.orchestra.run.smithi047.stdout: "max_avail": 367068545024 2024-02-19T15:26:08.844 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:08.844 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:08.844 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:08.845 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "cephfs_metadata", 2024-02-19T15:26:08.845 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 12, 2024-02-19T15:26:08.845 INFO:teuthology.orchestra.run.smithi047.stdout: "stats": { 2024-02-19T15:26:08.845 INFO:teuthology.orchestra.run.smithi047.stdout: "stored": 2410, 2024-02-19T15:26:08.845 INFO:teuthology.orchestra.run.smithi047.stdout: "objects": 22, 2024-02-19T15:26:08.845 INFO:teuthology.orchestra.run.smithi047.stdout: "kb_used": 64, 2024-02-19T15:26:08.845 INFO:teuthology.orchestra.run.smithi047.stdout: "bytes_used": 65536, 2024-02-19T15:26:08.845 INFO:teuthology.orchestra.run.smithi047.stdout: "percent_used": 8.9269420300297497e-08, 2024-02-19T15:26:08.845 INFO:teuthology.orchestra.run.smithi047.stdout: "max_avail": 367068545024 2024-02-19T15:26:08.845 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:08.845 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:08.845 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:08.845 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "cephfs_data", 2024-02-19T15:26:08.845 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 13, 2024-02-19T15:26:08.845 INFO:teuthology.orchestra.run.smithi047.stdout: "stats": { 2024-02-19T15:26:08.845 INFO:teuthology.orchestra.run.smithi047.stdout: "stored": 0, 2024-02-19T15:26:08.846 INFO:teuthology.orchestra.run.smithi047.stdout: "objects": 0, 2024-02-19T15:26:08.846 INFO:teuthology.orchestra.run.smithi047.stdout: "kb_used": 0, 2024-02-19T15:26:08.846 INFO:teuthology.orchestra.run.smithi047.stdout: "bytes_used": 0, 2024-02-19T15:26:08.846 INFO:teuthology.orchestra.run.smithi047.stdout: "percent_used": 0, 2024-02-19T15:26:08.846 INFO:teuthology.orchestra.run.smithi047.stdout: "max_avail": 367068545024 2024-02-19T15:26:08.846 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:08.846 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:08.846 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-02-19T15:26:08.846 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:26:08.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.840+0000 7fcb79a00640 1 -- 172.21.15.47:0/3508845593 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcb50066e40 msgr2=0x7fcb50069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:08.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.840+0000 7fcb79a00640 1 --2- 172.21.15.47:0/3508845593 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcb50066e40 0x7fcb50069300 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fcb64005890 tx=0x7fcb64009040 comp rx=0 tx=0).stop 2024-02-19T15:26:08.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.840+0000 7fcb79a00640 1 -- 172.21.15.47:0/3508845593 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb740d0360 msgr2=0x7fcb740c7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:08.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.840+0000 7fcb79a00640 1 --2- 172.21.15.47:0/3508845593 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb740d0360 0x7fcb740c7990 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00c420 tx=0x7fcb5c00c8e0 comp rx=0 tx=0).stop 2024-02-19T15:26:08.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.840+0000 7fcb79a00640 1 -- 172.21.15.47:0/3508845593 shutdown_connections 2024-02-19T15:26:08.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.840+0000 7fcb79a00640 1 --2- 172.21.15.47:0/3508845593 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcb50066e40 0x7fcb50069300 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:08.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.840+0000 7fcb79a00640 1 --2- 172.21.15.47:0/3508845593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb7415bcd0 0x7fcb740c5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:08.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.840+0000 7fcb79a00640 1 --2- 172.21.15.47:0/3508845593 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb740d0360 0x7fcb740c7990 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:08.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.840+0000 7fcb79a00640 1 --2- 172.21.15.47:0/3508845593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb740cedb0 0x7fcb74159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:08.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.840+0000 7fcb79a00640 1 -- 172.21.15.47:0/3508845593 >> 172.21.15.47:0/3508845593 conn(0x7fcb740d1bf0 msgr2=0x7fcb740cd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:08.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.840+0000 7fcb79a00640 1 -- 172.21.15.47:0/3508845593 shutdown_connections 2024-02-19T15:26:08.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:08.840+0000 7fcb79a00640 1 -- 172.21.15.47:0/3508845593 wait complete. 2024-02-19T15:26:08.873 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-02-19T15:26:08.873 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T15:26:08.873 DEBUG:teuthology.orchestra.run.smithi047:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:26:08.965 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:26:09.053 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T15:26:09.053 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:26:09.053 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:26:09.062 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:26:09.062 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:26:09.110 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:26:09.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.284+0000 7f2d2b400640 1 Processor -- start 2024-02-19T15:26:09.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.284+0000 7f2d2b400640 1 -- start start 2024-02-19T15:26:09.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.288+0000 7f2d2b400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2d240cff60 0x7f2d240d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:09.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.288+0000 7f2d2b400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d2415e270 0x7f2d240d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:09.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.288+0000 7f2d2b400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d240b3240 con 0x7f2d240d1760 2024-02-19T15:26:09.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.288+0000 7f2d2b400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d240b3520 con 0x7f2d240cff60 2024-02-19T15:26:09.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.288+0000 7f2d2b400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2406cb50 con 0x7f2d2415e270 2024-02-19T15:26:09.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.288+0000 7f2d23e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d2415e270 0x7f2d240d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:09.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.288+0000 7f2d28a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2d240cff60 0x7f2d240d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:09.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.288+0000 7f2d28a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2d240cff60 0x7f2d240d0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:37266/0 (socket says 172.21.15.47:37266) 2024-02-19T15:26:09.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.288+0000 7f2d28a00640 1 -- 172.21.15.47:0/3579134094 learned_addr learned my addr 172.21.15.47:0/3579134094 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:09.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.288+0000 7f2d23e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d2415e270 0x7f2d240d0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:53092/0 (socket says 172.21.15.47:53092) 2024-02-19T15:26:09.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.288+0000 7f2d28a00640 1 -- 172.21.15.47:0/3579134094 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d2415e270 msgr2=0x7f2d240d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:09.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.288+0000 7f2d28a00640 1 --2- 172.21.15.47:0/3579134094 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d2415e270 0x7f2d240d0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.288+0000 7f2d28a00640 1 -- 172.21.15.47:0/3579134094 >> v1:172.21.15.116:6789/0 conn(0x7f2d240d1760 legacy=0x7f2d240ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:09.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.288+0000 7f2d28a00640 1 -- 172.21.15.47:0/3579134094 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2406ce30 con 0x7f2d240cff60 2024-02-19T15:26:09.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.292+0000 7f2d28a00640 1 --2- 172.21.15.47:0/3579134094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d240cff60 0x7f2d240d0dc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2d18009870 tx=0x7f2d1801c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97a3f75dbcfd0243 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:09.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.292+0000 7f2d23400640 1 -- 172.21.15.47:0/3579134094 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d180ab660 con 0x7f2d240cff60 2024-02-19T15:26:09.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.292+0000 7f2d23400640 1 -- 172.21.15.47:0/3579134094 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d180ab7c0 con 0x7f2d240cff60 2024-02-19T15:26:09.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.292+0000 7f2d23400640 1 -- 172.21.15.47:0/3579134094 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d180abb50 con 0x7f2d240cff60 2024-02-19T15:26:09.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.292+0000 7f2d2b400640 1 -- 172.21.15.47:0/3579134094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d240cff60 msgr2=0x7f2d240d0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:09.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.292+0000 7f2d2b400640 1 --2- 172.21.15.47:0/3579134094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d240cff60 0x7f2d240d0dc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2d18009870 tx=0x7f2d1801c4d0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.292+0000 7f2d2b400640 1 -- 172.21.15.47:0/3579134094 shutdown_connections 2024-02-19T15:26:09.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.292+0000 7f2d2b400640 1 --2- 172.21.15.47:0/3579134094 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d2415e270 0x7f2d240d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.292+0000 7f2d2b400640 1 --2- 172.21.15.47:0/3579134094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d240cff60 0x7f2d240d0dc0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.292+0000 7f2d2b400640 1 -- 172.21.15.47:0/3579134094 >> 172.21.15.47:0/3579134094 conn(0x7f2d240d1b90 msgr2=0x7f2d240d1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:09.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.292+0000 7f2d2b400640 1 -- 172.21.15.47:0/3579134094 shutdown_connections 2024-02-19T15:26:09.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.292+0000 7f2d2b400640 1 -- 172.21.15.47:0/3579134094 wait complete. 2024-02-19T15:26:09.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.292+0000 7f2d2b400640 1 Processor -- start 2024-02-19T15:26:09.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.292+0000 7f2d2b400640 1 -- start start 2024-02-19T15:26:09.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d2b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d240cff60 0x7f2d2414f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:09.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d2b400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d240d1760 0x7f2d2415b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:09.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d2b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d2415e270 0x7f2d2415b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:09.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d2b400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2415f080 con 0x7f2d240cff60 2024-02-19T15:26:09.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d2b400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2415be60 con 0x7f2d2415e270 2024-02-19T15:26:09.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d2b400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2415f2d0 con 0x7f2d240d1760 2024-02-19T15:26:09.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d28a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d240cff60 0x7f2d2414f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:09.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d28a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d240cff60 0x7f2d2414f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50748/0 (socket says 172.21.15.47:50748) 2024-02-19T15:26:09.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d28a00640 1 -- 172.21.15.47:0/400319281 learned_addr learned my addr 172.21.15.47:0/400319281 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:09.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d23e00640 1 --2- 172.21.15.47:0/400319281 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d240d1760 0x7f2d2415b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:09.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d29400640 1 --2- 172.21.15.47:0/400319281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d2415e270 0x7f2d2415b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:09.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d28a00640 1 -- 172.21.15.47:0/400319281 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d240d1760 msgr2=0x7f2d2415b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:09.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d28a00640 1 --2- 172.21.15.47:0/400319281 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d240d1760 0x7f2d2415b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d28a00640 1 -- 172.21.15.47:0/400319281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d2415e270 msgr2=0x7f2d2415b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:09.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d28a00640 1 --2- 172.21.15.47:0/400319281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d2415e270 0x7f2d2415b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d28a00640 1 -- 172.21.15.47:0/400319281 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d180094d0 con 0x7f2d240cff60 2024-02-19T15:26:09.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d23e00640 1 --2- 172.21.15.47:0/400319281 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d240d1760 0x7f2d2415b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:09.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d28a00640 1 --2- 172.21.15.47:0/400319281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d240cff60 0x7f2d2414f970 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f2d1801a100 tx=0x7f2d1801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:09.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d20c00640 1 -- 172.21.15.47:0/400319281 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d180ab660 con 0x7f2d240cff60 2024-02-19T15:26:09.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d20c00640 1 -- 172.21.15.47:0/400319281 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d180b4800 con 0x7f2d240cff60 2024-02-19T15:26:09.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d20c00640 1 -- 172.21.15.47:0/400319281 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d180b4bb0 con 0x7f2d240cff60 2024-02-19T15:26:09.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d2b400640 1 -- 172.21.15.47:0/400319281 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2415f5b0 con 0x7f2d240cff60 2024-02-19T15:26:09.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d2b400640 1 -- 172.21.15.47:0/400319281 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2415fdb0 con 0x7f2d240cff60 2024-02-19T15:26:09.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.296+0000 7f2d2b400640 1 -- 172.21.15.47:0/400319281 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d24068500 con 0x7f2d240cff60 2024-02-19T15:26:09.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.300+0000 7f2d20c00640 1 -- 172.21.15.47:0/400319281 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2d180a9070 con 0x7f2d240cff60 2024-02-19T15:26:09.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.300+0000 7f2d20c00640 1 --2- 172.21.15.47:0/400319281 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2d04066e40 0x7f2d04069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:09.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.300+0000 7f2d20c00640 1 -- 172.21.15.47:0/400319281 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d1811f210 con 0x7f2d240cff60 2024-02-19T15:26:09.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.300+0000 7f2d23e00640 1 --2- 172.21.15.47:0/400319281 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2d04066e40 0x7f2d04069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:09.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.300+0000 7f2d23e00640 1 --2- 172.21.15.47:0/400319281 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2d04066e40 0x7f2d04069300 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2d240b2d60 tx=0x7f2d10008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:09.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.304+0000 7f2d20c00640 1 -- 172.21.15.47:0/400319281 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2d180ec1d0 con 0x7f2d240cff60 2024-02-19T15:26:09.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.464+0000 7f2d2b400640 1 -- 172.21.15.47:0/400319281 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d2406c840 con 0x7f2d240cff60 2024-02-19T15:26:09.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.468+0000 7f2d20c00640 1 -- 172.21.15.47:0/400319281 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16672 (secure 0 0 0) 0x7f2d180ef890 con 0x7f2d240cff60 2024-02-19T15:26:09.471 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:09.471 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:26:01.137113+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:25:57.938354+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-02-19T15:25:58.514866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:26:09.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.468+0000 7f2d2b400640 1 -- 172.21.15.47:0/400319281 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2d04066e40 msgr2=0x7f2d04069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:09.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.468+0000 7f2d2b400640 1 --2- 172.21.15.47:0/400319281 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2d04066e40 0x7f2d04069300 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2d240b2d60 tx=0x7f2d10008040 comp rx=0 tx=0).stop 2024-02-19T15:26:09.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.468+0000 7f2d2b400640 1 -- 172.21.15.47:0/400319281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d240cff60 msgr2=0x7f2d2414f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:09.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.468+0000 7f2d2b400640 1 --2- 172.21.15.47:0/400319281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d240cff60 0x7f2d2414f970 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f2d1801a100 tx=0x7f2d1801ad10 comp rx=0 tx=0).stop 2024-02-19T15:26:09.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.468+0000 7f2d2b400640 1 -- 172.21.15.47:0/400319281 shutdown_connections 2024-02-19T15:26:09.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.468+0000 7f2d2b400640 1 --2- 172.21.15.47:0/400319281 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2d04066e40 0x7f2d04069300 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.468+0000 7f2d2b400640 1 --2- 172.21.15.47:0/400319281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d2415e270 0x7f2d2415b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.468+0000 7f2d2b400640 1 --2- 172.21.15.47:0/400319281 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d240d1760 0x7f2d2415b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.468+0000 7f2d2b400640 1 --2- 172.21.15.47:0/400319281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d240cff60 0x7f2d2414f970 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.468+0000 7f2d2b400640 1 -- 172.21.15.47:0/400319281 >> 172.21.15.47:0/400319281 conn(0x7f2d240d1b90 msgr2=0x7f2d240b0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:09.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.468+0000 7f2d2b400640 1 -- 172.21.15.47:0/400319281 shutdown_connections 2024-02-19T15:26:09.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.468+0000 7f2d2b400640 1 -- 172.21.15.47:0/400319281 wait complete. 2024-02-19T15:26:09.493 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:25:57.938354+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:25:58.514866+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:26:09.493 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:26:09.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.632+0000 7fe344400640 1 Processor -- start 2024-02-19T15:26:09.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.632+0000 7fe344400640 1 -- start start 2024-02-19T15:26:09.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.632+0000 7fe344400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe33c15e000 0x7fe33c0cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:09.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.632+0000 7fe344400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe33c0d0240 0x7fe33c15f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:09.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.632+0000 7fe344400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe33c0d31e0 con 0x7fe33c15fcc0 2024-02-19T15:26:09.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.632+0000 7fe344400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe33c0ceca0 con 0x7fe33c0d0240 2024-02-19T15:26:09.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.632+0000 7fe344400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe33c0cef40 con 0x7fe33c15e000 2024-02-19T15:26:09.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.632+0000 7fe341000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe33c0d0240 0x7fe33c15f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:09.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.632+0000 7fe341000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe33c0d0240 0x7fe33c15f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50762/0 (socket says 172.21.15.47:50762) 2024-02-19T15:26:09.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.632+0000 7fe341000640 1 -- 172.21.15.47:0/4242784422 learned_addr learned my addr 172.21.15.47:0/4242784422 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:09.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.632+0000 7fe341a00640 1 --2- 172.21.15.47:0/4242784422 >> v2:172.21.15.116:3300/0 conn(0x7fe33c15e000 0x7fe33c0cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:09.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.632+0000 7fe341000640 1 -- 172.21.15.47:0/4242784422 >> v2:172.21.15.116:3300/0 conn(0x7fe33c15e000 msgr2=0x7fe33c0cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:09.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.632+0000 7fe341000640 1 --2- 172.21.15.47:0/4242784422 >> v2:172.21.15.116:3300/0 conn(0x7fe33c15e000 0x7fe33c0cfd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.632+0000 7fe341000640 1 -- 172.21.15.47:0/4242784422 >> v1:172.21.15.116:6789/0 conn(0x7fe33c15fcc0 legacy=0x7fe33c0d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:09.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.632+0000 7fe341000640 1 -- 172.21.15.47:0/4242784422 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33c0cf220 con 0x7fe33c0d0240 2024-02-19T15:26:09.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.636+0000 7fe341000640 1 --2- 172.21.15.47:0/4242784422 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe33c0d0240 0x7fe33c15f780 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fe33c0d4e00 tx=0x7fe32801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f558bf64d59bcfff server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:09.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.636+0000 7fe33be00640 1 -- 172.21.15.47:0/4242784422 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3280ab600 con 0x7fe33c0d0240 2024-02-19T15:26:09.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.636+0000 7fe33be00640 1 -- 172.21.15.47:0/4242784422 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3280a7050 con 0x7fe33c0d0240 2024-02-19T15:26:09.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.636+0000 7fe33be00640 1 -- 172.21.15.47:0/4242784422 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3280ab980 con 0x7fe33c0d0240 2024-02-19T15:26:09.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.636+0000 7fe344400640 1 -- 172.21.15.47:0/4242784422 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe33c0d0240 msgr2=0x7fe33c15f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:09.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.636+0000 7fe344400640 1 --2- 172.21.15.47:0/4242784422 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe33c0d0240 0x7fe33c15f780 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fe33c0d4e00 tx=0x7fe32801c420 comp rx=0 tx=0).stop 2024-02-19T15:26:09.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.636+0000 7fe344400640 1 -- 172.21.15.47:0/4242784422 shutdown_connections 2024-02-19T15:26:09.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.636+0000 7fe344400640 1 --2- 172.21.15.47:0/4242784422 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe33c0d0240 0x7fe33c15f780 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.636+0000 7fe344400640 1 --2- 172.21.15.47:0/4242784422 >> v2:172.21.15.116:3300/0 conn(0x7fe33c15e000 0x7fe33c0cfd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.636+0000 7fe344400640 1 -- 172.21.15.47:0/4242784422 >> 172.21.15.47:0/4242784422 conn(0x7fe33c0d3d90 msgr2=0x7fe33c0d41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:09.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.636+0000 7fe344400640 1 -- 172.21.15.47:0/4242784422 shutdown_connections 2024-02-19T15:26:09.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.636+0000 7fe344400640 1 -- 172.21.15.47:0/4242784422 wait complete. 2024-02-19T15:26:09.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.640+0000 7fe344400640 1 Processor -- start 2024-02-19T15:26:09.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.640+0000 7fe344400640 1 -- start start 2024-02-19T15:26:09.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.640+0000 7fe344400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe33c15e000 0x7fe33c1531a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:09.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.640+0000 7fe344400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe33c15fcc0 0x7fe33c158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:09.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.640+0000 7fe344400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe33c1556e0 0x7fe33c15c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:09.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.640+0000 7fe344400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33c156300 con 0x7fe33c15fcc0 2024-02-19T15:26:09.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.640+0000 7fe344400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33c15bc00 con 0x7fe33c1556e0 2024-02-19T15:26:09.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.640+0000 7fe344400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe33c159170 con 0x7fe33c15e000 2024-02-19T15:26:09.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.640+0000 7fe341000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe33c15fcc0 0x7fe33c158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:09.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.640+0000 7fe341000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe33c15fcc0 0x7fe33c158a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50772/0 (socket says 172.21.15.47:50772) 2024-02-19T15:26:09.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.640+0000 7fe341000640 1 -- 172.21.15.47:0/2409721381 learned_addr learned my addr 172.21.15.47:0/2409721381 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:09.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.640+0000 7fe342400640 1 --2- 172.21.15.47:0/2409721381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe33c1556e0 0x7fe33c15c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:09.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.640+0000 7fe341a00640 1 --2- 172.21.15.47:0/2409721381 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe33c15e000 0x7fe33c1531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:09.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.640+0000 7fe341000640 1 -- 172.21.15.47:0/2409721381 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe33c15e000 msgr2=0x7fe33c1531a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:09.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.640+0000 7fe341000640 1 --2- 172.21.15.47:0/2409721381 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe33c15e000 0x7fe33c1531a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.640+0000 7fe341000640 1 -- 172.21.15.47:0/2409721381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe33c1556e0 msgr2=0x7fe33c15c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:09.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.640+0000 7fe341000640 1 --2- 172.21.15.47:0/2409721381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe33c1556e0 0x7fe33c15c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.640+0000 7fe341000640 1 -- 172.21.15.47:0/2409721381 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3280094d0 con 0x7fe33c15fcc0 2024-02-19T15:26:09.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.644+0000 7fe341000640 1 --2- 172.21.15.47:0/2409721381 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe33c15fcc0 0x7fe33c158a90 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe3280ae040 tx=0x7fe3280191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:09.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.644+0000 7fe339600640 1 -- 172.21.15.47:0/2409721381 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3280b4610 con 0x7fe33c15fcc0 2024-02-19T15:26:09.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.644+0000 7fe339600640 1 -- 172.21.15.47:0/2409721381 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3280b4770 con 0x7fe33c15fcc0 2024-02-19T15:26:09.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.644+0000 7fe344400640 1 -- 172.21.15.47:0/2409721381 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe33c159450 con 0x7fe33c15fcc0 2024-02-19T15:26:09.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.644+0000 7fe339600640 1 -- 172.21.15.47:0/2409721381 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3280b4b20 con 0x7fe33c15fcc0 2024-02-19T15:26:09.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.644+0000 7fe344400640 1 -- 172.21.15.47:0/2409721381 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe33c159a10 con 0x7fe33c15fcc0 2024-02-19T15:26:09.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.644+0000 7fe344400640 1 -- 172.21.15.47:0/2409721381 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe33c0d3750 con 0x7fe33c15fcc0 2024-02-19T15:26:09.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.644+0000 7fe339600640 1 -- 172.21.15.47:0/2409721381 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe3280b2040 con 0x7fe33c15fcc0 2024-02-19T15:26:09.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.644+0000 7fe339600640 1 --2- 172.21.15.47:0/2409721381 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe31c066e70 0x7fe31c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:09.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.644+0000 7fe339600640 1 -- 172.21.15.47:0/2409721381 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe32811eb70 con 0x7fe33c15fcc0 2024-02-19T15:26:09.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.644+0000 7fe341a00640 1 --2- 172.21.15.47:0/2409721381 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe31c066e70 0x7fe31c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:09.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.648+0000 7fe341a00640 1 --2- 172.21.15.47:0/2409721381 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe31c066e70 0x7fe31c069330 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe33c0b9bf0 tx=0x7fe330008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:09.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.648+0000 7fe339600640 1 -- 172.21.15.47:0/2409721381 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe3280eb970 con 0x7fe33c15fcc0 2024-02-19T15:26:09.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.812+0000 7fe344400640 1 -- 172.21.15.47:0/2409721381 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe33c075360 con 0x7fe33c15fcc0 2024-02-19T15:26:09.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.812+0000 7fe339600640 1 -- 172.21.15.47:0/2409721381 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe3280ef030 con 0x7fe33c15fcc0 2024-02-19T15:26:09.815 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:26:09.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.812+0000 7fe344400640 1 -- 172.21.15.47:0/2409721381 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe31c066e70 msgr2=0x7fe31c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:09.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.812+0000 7fe344400640 1 --2- 172.21.15.47:0/2409721381 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe31c066e70 0x7fe31c069330 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe33c0b9bf0 tx=0x7fe330008040 comp rx=0 tx=0).stop 2024-02-19T15:26:09.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.812+0000 7fe344400640 1 -- 172.21.15.47:0/2409721381 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe33c15fcc0 msgr2=0x7fe33c158a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:09.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.812+0000 7fe344400640 1 --2- 172.21.15.47:0/2409721381 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe33c15fcc0 0x7fe33c158a90 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe3280ae040 tx=0x7fe3280191c0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.812+0000 7fe344400640 1 -- 172.21.15.47:0/2409721381 shutdown_connections 2024-02-19T15:26:09.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.812+0000 7fe344400640 1 --2- 172.21.15.47:0/2409721381 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe31c066e70 0x7fe31c069330 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.812+0000 7fe344400640 1 --2- 172.21.15.47:0/2409721381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe33c1556e0 0x7fe33c15c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.812+0000 7fe344400640 1 --2- 172.21.15.47:0/2409721381 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe33c15fcc0 0x7fe33c158a90 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.812+0000 7fe344400640 1 --2- 172.21.15.47:0/2409721381 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe33c15e000 0x7fe33c1531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.812+0000 7fe344400640 1 -- 172.21.15.47:0/2409721381 >> 172.21.15.47:0/2409721381 conn(0x7fe33c0d3d90 msgr2=0x7fe33c1602d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:09.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.812+0000 7fe344400640 1 -- 172.21.15.47:0/2409721381 shutdown_connections 2024-02-19T15:26:09.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.812+0000 7fe344400640 1 -- 172.21.15.47:0/2409721381 wait complete. 2024-02-19T15:26:09.837 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:26:09.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.976+0000 7f571b600640 1 Processor -- start 2024-02-19T15:26:09.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.976+0000 7f571b600640 1 -- start start 2024-02-19T15:26:09.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.976+0000 7f571b600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57140ce3b0 0x7f57140ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:09.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.976+0000 7f571b600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f571415e280 0x7f57140cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:09.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.976+0000 7f571b600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57140cc620 con 0x7f57140d2650 2024-02-19T15:26:09.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.976+0000 7f571b600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57140d12e0 con 0x7f571415e280 2024-02-19T15:26:09.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.976+0000 7f571b600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57140d1940 con 0x7f57140ce3b0 2024-02-19T15:26:09.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.976+0000 7f5719600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f571415e280 0x7f57140cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:09.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.976+0000 7f5719600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f571415e280 0x7f57140cd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50788/0 (socket says 172.21.15.47:50788) 2024-02-19T15:26:09.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.976+0000 7f5719600640 1 -- 172.21.15.47:0/574944697 learned_addr learned my addr 172.21.15.47:0/574944697 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:09.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.976+0000 7f5718c00640 1 --2- 172.21.15.47:0/574944697 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57140ce3b0 0x7f57140ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:09.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.976+0000 7f5719600640 1 -- 172.21.15.47:0/574944697 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57140ce3b0 msgr2=0x7f57140ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:09.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.976+0000 7f5719600640 1 --2- 172.21.15.47:0/574944697 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57140ce3b0 0x7f57140ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.976+0000 7f5719600640 1 -- 172.21.15.47:0/574944697 >> v1:172.21.15.116:6789/0 conn(0x7f57140d2650 legacy=0x7f57140ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:09.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.976+0000 7f5719600640 1 -- 172.21.15.47:0/574944697 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57140cecd0 con 0x7f571415e280 2024-02-19T15:26:09.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.980+0000 7f5719600640 1 --2- 172.21.15.47:0/574944697 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f571415e280 0x7f57140cd1c0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f570c009e50 tx=0x7f570c01c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341920dbeb14cc55 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:09.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.980+0000 7f5713400640 1 -- 172.21.15.47:0/574944697 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c0a5640 con 0x7f571415e280 2024-02-19T15:26:09.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.980+0000 7f5713400640 1 -- 172.21.15.47:0/574944697 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570c0a57a0 con 0x7f571415e280 2024-02-19T15:26:09.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.980+0000 7f5713400640 1 -- 172.21.15.47:0/574944697 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c0a5b30 con 0x7f571415e280 2024-02-19T15:26:09.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.980+0000 7f571b600640 1 -- 172.21.15.47:0/574944697 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f571415e280 msgr2=0x7f57140cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:09.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.980+0000 7f571b600640 1 --2- 172.21.15.47:0/574944697 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f571415e280 0x7f57140cd1c0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f570c009e50 tx=0x7f570c01c8d0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.980+0000 7f571b600640 1 -- 172.21.15.47:0/574944697 shutdown_connections 2024-02-19T15:26:09.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.980+0000 7f571b600640 1 --2- 172.21.15.47:0/574944697 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f571415e280 0x7f57140cd1c0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.980+0000 7f571b600640 1 --2- 172.21.15.47:0/574944697 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57140ce3b0 0x7f57140ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.980+0000 7f571b600640 1 -- 172.21.15.47:0/574944697 >> 172.21.15.47:0/574944697 conn(0x7f57140d1b30 msgr2=0x7f57140d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:09.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.980+0000 7f571b600640 1 -- 172.21.15.47:0/574944697 shutdown_connections 2024-02-19T15:26:09.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f571b600640 1 -- 172.21.15.47:0/574944697 wait complete. 2024-02-19T15:26:09.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f571b600640 1 Processor -- start 2024-02-19T15:26:09.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f571b600640 1 -- start start 2024-02-19T15:26:09.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f571b600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57140ce3b0 0x7f5714149af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:09.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f571b600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57140d2650 0x7f5714158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:09.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f571b600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f571415e280 0x7f571409b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:09.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f571b600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571409ec80 con 0x7f571415e280 2024-02-19T15:26:09.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f571b600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571414a030 con 0x7f57140d2650 2024-02-19T15:26:09.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f571b600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571409e000 con 0x7f57140ce3b0 2024-02-19T15:26:09.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f5719600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f571415e280 0x7f571409b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:09.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f5719600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f571415e280 0x7f571409b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50804/0 (socket says 172.21.15.47:50804) 2024-02-19T15:26:09.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f5719600640 1 -- 172.21.15.47:0/4100082673 learned_addr learned my addr 172.21.15.47:0/4100082673 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:09.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f5718c00640 1 --2- 172.21.15.47:0/4100082673 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57140ce3b0 0x7f5714149af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:09.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f5713e00640 1 --2- 172.21.15.47:0/4100082673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57140d2650 0x7f5714158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:09.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f5719600640 1 -- 172.21.15.47:0/4100082673 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57140ce3b0 msgr2=0x7f5714149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:09.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f5719600640 1 --2- 172.21.15.47:0/4100082673 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57140ce3b0 0x7f5714149af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f5719600640 1 -- 172.21.15.47:0/4100082673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57140d2650 msgr2=0x7f5714158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:09.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f5719600640 1 --2- 172.21.15.47:0/4100082673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57140d2650 0x7f5714158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:09.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f5719600640 1 -- 172.21.15.47:0/4100082673 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570c009b30 con 0x7f571415e280 2024-02-19T15:26:09.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f5713e00640 1 --2- 172.21.15.47:0/4100082673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57140d2650 0x7f5714158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:09.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f5719600640 1 --2- 172.21.15.47:0/4100082673 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f571415e280 0x7f571409b8e0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f570c01af80 tx=0x7f570c019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:09.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f5710c00640 1 -- 172.21.15.47:0/4100082673 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c0b8670 con 0x7f571415e280 2024-02-19T15:26:09.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f5710c00640 1 -- 172.21.15.47:0/4100082673 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570c0b87f0 con 0x7f571415e280 2024-02-19T15:26:09.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f5710c00640 1 -- 172.21.15.47:0/4100082673 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c0b8ba0 con 0x7f571415e280 2024-02-19T15:26:09.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.984+0000 7f571b600640 1 -- 172.21.15.47:0/4100082673 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571409e2e0 con 0x7f571415e280 2024-02-19T15:26:09.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.988+0000 7f571b600640 1 -- 172.21.15.47:0/4100082673 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f571409e8a0 con 0x7f571415e280 2024-02-19T15:26:09.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.988+0000 7f5706000640 1 -- 172.21.15.47:0/4100082673 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f571409edc0 con 0x7f571415e280 2024-02-19T15:26:09.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.988+0000 7f5710c00640 1 -- 172.21.15.47:0/4100082673 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f570c0b8d00 con 0x7f571415e280 2024-02-19T15:26:09.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.988+0000 7f5710c00640 1 --2- 172.21.15.47:0/4100082673 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f56f4066e40 0x7f56f4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:09.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.988+0000 7f5710c00640 1 -- 172.21.15.47:0/4100082673 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f570c120210 con 0x7f571415e280 2024-02-19T15:26:09.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.988+0000 7f5718c00640 1 --2- 172.21.15.47:0/4100082673 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f56f4066e40 0x7f56f4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:09.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.992+0000 7f5718c00640 1 --2- 172.21.15.47:0/4100082673 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f56f4066e40 0x7f56f4069300 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f57080079c0 tx=0x7f5708008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:09.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:09.992+0000 7f5710c00640 1 -- 172.21.15.47:0/4100082673 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f571409edc0 con 0x7f571415e280 2024-02-19T15:26:10.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.156+0000 7f5706000640 1 -- 172.21.15.47:0/4100082673 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f571409edc0 con 0x7f571415e280 2024-02-19T15:26:10.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.156+0000 7f5710c00640 1 -- 172.21.15.47:0/4100082673 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f571409edc0 con 0x7f571415e280 2024-02-19T15:26:10.161 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:26:10.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.160+0000 7f5706000640 1 -- 172.21.15.47:0/4100082673 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f56f4066e40 msgr2=0x7f56f4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:10.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.160+0000 7f5706000640 1 --2- 172.21.15.47:0/4100082673 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f56f4066e40 0x7f56f4069300 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f57080079c0 tx=0x7f5708008040 comp rx=0 tx=0).stop 2024-02-19T15:26:10.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.160+0000 7f5706000640 1 -- 172.21.15.47:0/4100082673 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f571415e280 msgr2=0x7f571409b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:10.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.160+0000 7f5706000640 1 --2- 172.21.15.47:0/4100082673 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f571415e280 0x7f571409b8e0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f570c01af80 tx=0x7f570c019670 comp rx=0 tx=0).stop 2024-02-19T15:26:10.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.160+0000 7f5706000640 1 -- 172.21.15.47:0/4100082673 shutdown_connections 2024-02-19T15:26:10.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.160+0000 7f5706000640 1 --2- 172.21.15.47:0/4100082673 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f56f4066e40 0x7f56f4069300 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.160+0000 7f5706000640 1 --2- 172.21.15.47:0/4100082673 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f571415e280 0x7f571409b8e0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.160+0000 7f5706000640 1 --2- 172.21.15.47:0/4100082673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57140d2650 0x7f5714158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.160+0000 7f5706000640 1 --2- 172.21.15.47:0/4100082673 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57140ce3b0 0x7f5714149af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.160+0000 7f5706000640 1 -- 172.21.15.47:0/4100082673 >> 172.21.15.47:0/4100082673 conn(0x7f57140d1b30 msgr2=0x7f57140c7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:10.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.160+0000 7f5706000640 1 -- 172.21.15.47:0/4100082673 shutdown_connections 2024-02-19T15:26:10.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.160+0000 7f5706000640 1 -- 172.21.15.47:0/4100082673 wait complete. 2024-02-19T15:26:10.190 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:26:10.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.328+0000 7f1e16a00640 1 Processor -- start 2024-02-19T15:26:10.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.328+0000 7f1e16a00640 1 -- start start 2024-02-19T15:26:10.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.332+0000 7f1e16a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e100ce3b0 0x7f1e100ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:10.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.332+0000 7f1e16a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1e100d2650 0x7f1e100ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:10.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.332+0000 7f1e16a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1e1015e280 0x7f1e100ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:10.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.332+0000 7f1e16a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e100cc620 con 0x7f1e1015e280 2024-02-19T15:26:10.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.332+0000 7f1e16a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e100d12e0 con 0x7f1e100d2650 2024-02-19T15:26:10.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.332+0000 7f1e16a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e100d1940 con 0x7f1e100ce3b0 2024-02-19T15:26:10.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.332+0000 7f1e14a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1e1015e280 0x7f1e100ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:10.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.332+0000 7f1e14a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1e1015e280 0x7f1e100ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50814/0 (socket says 172.21.15.47:50814) 2024-02-19T15:26:10.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.332+0000 7f1e0fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e100ce3b0 0x7f1e100ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:10.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.332+0000 7f1e14a00640 1 -- 172.21.15.47:0/1253149035 learned_addr learned my addr 172.21.15.47:0/1253149035 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:10.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.332+0000 7f1e07e00640 1 --2- 172.21.15.47:0/1253149035 >> v2:172.21.15.116:3300/0 conn(0x7f1e100d2650 0x7f1e100ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:10.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.332+0000 7f1e14a00640 1 -- 172.21.15.47:0/1253149035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e100ce3b0 msgr2=0x7f1e100ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:10.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.332+0000 7f1e14a00640 1 --2- 172.21.15.47:0/1253149035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e100ce3b0 0x7f1e100ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.332+0000 7f1e14a00640 1 -- 172.21.15.47:0/1253149035 >> v2:172.21.15.116:3300/0 conn(0x7f1e100d2650 msgr2=0x7f1e100ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:10.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.332+0000 7f1e14a00640 1 --2- 172.21.15.47:0/1253149035 >> v2:172.21.15.116:3300/0 conn(0x7f1e100d2650 0x7f1e100ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.332+0000 7f1e14a00640 1 -- 172.21.15.47:0/1253149035 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e100cecd0 con 0x7f1e1015e280 2024-02-19T15:26:10.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.332+0000 7f1e07e00640 1 --2- 172.21.15.47:0/1253149035 >> v2:172.21.15.116:3300/0 conn(0x7f1e100d2650 0x7f1e100ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:10.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.336+0000 7f1e14a00640 1 --2- 172.21.15.47:0/1253149035 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e1015e280 0x7f1e100ccff0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f1e1006bc80 tx=0x7f1e0801c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=333bebaff8ede8ba server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:10.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.336+0000 7f1e0f400640 1 -- 172.21.15.47:0/1253149035 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e080a5640 con 0x7f1e1015e280 2024-02-19T15:26:10.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.336+0000 7f1e0f400640 1 -- 172.21.15.47:0/1253149035 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e080a57a0 con 0x7f1e1015e280 2024-02-19T15:26:10.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.336+0000 7f1e0f400640 1 -- 172.21.15.47:0/1253149035 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e080a5b30 con 0x7f1e1015e280 2024-02-19T15:26:10.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.336+0000 7f1e16a00640 1 -- 172.21.15.47:0/1253149035 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e1015e280 msgr2=0x7f1e100ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:10.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.336+0000 7f1e16a00640 1 --2- 172.21.15.47:0/1253149035 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e1015e280 0x7f1e100ccff0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f1e1006bc80 tx=0x7f1e0801c8d0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.336+0000 7f1e16a00640 1 -- 172.21.15.47:0/1253149035 shutdown_connections 2024-02-19T15:26:10.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.336+0000 7f1e16a00640 1 --2- 172.21.15.47:0/1253149035 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e1015e280 0x7f1e100ccff0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.336+0000 7f1e16a00640 1 --2- 172.21.15.47:0/1253149035 >> v2:172.21.15.116:3300/0 conn(0x7f1e100d2650 0x7f1e100ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.336+0000 7f1e16a00640 1 --2- 172.21.15.47:0/1253149035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e100ce3b0 0x7f1e100ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.336+0000 7f1e16a00640 1 -- 172.21.15.47:0/1253149035 >> 172.21.15.47:0/1253149035 conn(0x7f1e100d1b30 msgr2=0x7f1e100d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:10.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.336+0000 7f1e16a00640 1 -- 172.21.15.47:0/1253149035 shutdown_connections 2024-02-19T15:26:10.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.336+0000 7f1e16a00640 1 -- 172.21.15.47:0/1253149035 wait complete. 2024-02-19T15:26:10.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e16a00640 1 Processor -- start 2024-02-19T15:26:10.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e16a00640 1 -- start start 2024-02-19T15:26:10.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e16a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e100ce3b0 0x7f1e1009bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:10.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e16a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e100d2650 0x7f1e10159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:10.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e16a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e1015e280 0x7f1e1015a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:10.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e16a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e101586c0 con 0x7f1e100ce3b0 2024-02-19T15:26:10.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e16a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e101577f0 con 0x7f1e1015e280 2024-02-19T15:26:10.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e16a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e10157a40 con 0x7f1e100d2650 2024-02-19T15:26:10.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e0fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e100ce3b0 0x7f1e1009bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:10.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e0fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e100ce3b0 0x7f1e1009bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50826/0 (socket says 172.21.15.47:50826) 2024-02-19T15:26:10.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e0fe00640 1 -- 172.21.15.47:0/1800362950 learned_addr learned my addr 172.21.15.47:0/1800362950 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:10.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e07e00640 1 --2- 172.21.15.47:0/1800362950 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e100d2650 0x7f1e10159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:10.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e14a00640 1 --2- 172.21.15.47:0/1800362950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e1015e280 0x7f1e1015a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:10.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e0fe00640 1 -- 172.21.15.47:0/1800362950 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e100d2650 msgr2=0x7f1e10159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:10.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e0fe00640 1 --2- 172.21.15.47:0/1800362950 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e100d2650 0x7f1e10159cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e0fe00640 1 -- 172.21.15.47:0/1800362950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e1015e280 msgr2=0x7f1e1015a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:10.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e0fe00640 1 --2- 172.21.15.47:0/1800362950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e1015e280 0x7f1e1015a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e0fe00640 1 -- 172.21.15.47:0/1800362950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e08009b30 con 0x7f1e100ce3b0 2024-02-19T15:26:10.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e14a00640 1 --2- 172.21.15.47:0/1800362950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e1015e280 0x7f1e1015a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:10.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e0fe00640 1 --2- 172.21.15.47:0/1800362950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e100ce3b0 0x7f1e1009bba0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f1e0000e8b0 tx=0x7f1e0000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:10.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e0cc00640 1 -- 172.21.15.47:0/1800362950 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0000c940 con 0x7f1e100ce3b0 2024-02-19T15:26:10.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e0cc00640 1 -- 172.21.15.47:0/1800362950 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0000caa0 con 0x7f1e100ce3b0 2024-02-19T15:26:10.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e16a00640 1 -- 172.21.15.47:0/1800362950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e10157d20 con 0x7f1e100ce3b0 2024-02-19T15:26:10.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e0cc00640 1 -- 172.21.15.47:0/1800362950 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0000ce30 con 0x7f1e100ce3b0 2024-02-19T15:26:10.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.340+0000 7f1e16a00640 1 -- 172.21.15.47:0/1800362950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e101582e0 con 0x7f1e100ce3b0 2024-02-19T15:26:10.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.344+0000 7f1e16a00640 1 -- 172.21.15.47:0/1800362950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e10158930 con 0x7f1e100ce3b0 2024-02-19T15:26:10.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.344+0000 7f1e0cc00640 1 -- 172.21.15.47:0/1800362950 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1e00013470 con 0x7f1e100ce3b0 2024-02-19T15:26:10.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.344+0000 7f1e0cc00640 1 --2- 172.21.15.47:0/1800362950 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1dec066e40 0x7f1dec069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:10.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.344+0000 7f1e0cc00640 1 -- 172.21.15.47:0/1800362950 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e00084be0 con 0x7f1e100ce3b0 2024-02-19T15:26:10.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.344+0000 7f1e07e00640 1 --2- 172.21.15.47:0/1800362950 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1dec066e40 0x7f1dec069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:10.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.348+0000 7f1e07e00640 1 --2- 172.21.15.47:0/1800362950 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1dec066e40 0x7f1dec069300 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1e100cbe20 tx=0x7f1df8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:10.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.352+0000 7f1e0cc00640 1 -- 172.21.15.47:0/1800362950 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1e00051a60 con 0x7f1e100ce3b0 2024-02-19T15:26:10.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.512+0000 7f1e16a00640 1 -- 172.21.15.47:0/1800362950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1e10075360 con 0x7f1e100ce3b0 2024-02-19T15:26:10.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.512+0000 7f1e0cc00640 1 -- 172.21.15.47:0/1800362950 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1e000210a0 con 0x7f1e100ce3b0 2024-02-19T15:26:10.518 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:26:10.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.516+0000 7f1e16a00640 1 -- 172.21.15.47:0/1800362950 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1dec066e40 msgr2=0x7f1dec069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:10.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.516+0000 7f1e16a00640 1 --2- 172.21.15.47:0/1800362950 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1dec066e40 0x7f1dec069300 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1e100cbe20 tx=0x7f1df8008040 comp rx=0 tx=0).stop 2024-02-19T15:26:10.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.516+0000 7f1e16a00640 1 -- 172.21.15.47:0/1800362950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e100ce3b0 msgr2=0x7f1e1009bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:10.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.516+0000 7f1e16a00640 1 --2- 172.21.15.47:0/1800362950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e100ce3b0 0x7f1e1009bba0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f1e0000e8b0 tx=0x7f1e0000ed70 comp rx=0 tx=0).stop 2024-02-19T15:26:10.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.516+0000 7f1e16a00640 1 -- 172.21.15.47:0/1800362950 shutdown_connections 2024-02-19T15:26:10.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.516+0000 7f1e16a00640 1 --2- 172.21.15.47:0/1800362950 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1dec066e40 0x7f1dec069300 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.516+0000 7f1e16a00640 1 --2- 172.21.15.47:0/1800362950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e1015e280 0x7f1e1015a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.516+0000 7f1e16a00640 1 --2- 172.21.15.47:0/1800362950 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e100d2650 0x7f1e10159cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.516+0000 7f1e16a00640 1 --2- 172.21.15.47:0/1800362950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e100ce3b0 0x7f1e1009bba0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.516+0000 7f1e16a00640 1 -- 172.21.15.47:0/1800362950 >> 172.21.15.47:0/1800362950 conn(0x7f1e100d1b30 msgr2=0x7f1e100d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:10.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.516+0000 7f1e16a00640 1 -- 172.21.15.47:0/1800362950 shutdown_connections 2024-02-19T15:26:10.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.516+0000 7f1e16a00640 1 -- 172.21.15.47:0/1800362950 wait complete. 2024-02-19T15:26:10.539 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-19T15:26:10.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.676+0000 7fa9b4a00640 1 Processor -- start 2024-02-19T15:26:10.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.676+0000 7fa9b4a00640 1 -- start start 2024-02-19T15:26:10.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.676+0000 7fa9b4a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa9b00b2fc0 0x7fa9b00d0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:10.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.676+0000 7fa9b4a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa9b00cef90 0x7fa9b00ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:10.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.676+0000 7fa9b4a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b00ced80 con 0x7fa9b00ce930 2024-02-19T15:26:10.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.676+0000 7fa9b4a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b00d10a0 con 0x7fa9b00b2fc0 2024-02-19T15:26:10.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.676+0000 7fa9b4a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b00b33a0 con 0x7fa9b00cef90 2024-02-19T15:26:10.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.676+0000 7fa9ade00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa9b00b2fc0 0x7fa9b00d0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:10.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.676+0000 7fa9ae800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa9b00ce930 0x7fa9b006bfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54900/0 (socket says 172.21.15.47:54900) 2024-02-19T15:26:10.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.676+0000 7fa9ae800640 1 -- 172.21.15.47:0/287913647 learned_addr learned my addr 172.21.15.47:0/287913647 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:10.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.680+0000 7fa9a7e00640 1 --2- 172.21.15.47:0/287913647 >> v2:172.21.15.116:3300/0 conn(0x7fa9b00cef90 0x7fa9b00ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:10.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.680+0000 7fa9ade00640 1 -- 172.21.15.47:0/287913647 >> v2:172.21.15.116:3300/0 conn(0x7fa9b00cef90 msgr2=0x7fa9b00ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:10.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.680+0000 7fa9ade00640 1 --2- 172.21.15.47:0/287913647 >> v2:172.21.15.116:3300/0 conn(0x7fa9b00cef90 0x7fa9b00ce3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.680+0000 7fa9ade00640 1 -- 172.21.15.47:0/287913647 >> v1:172.21.15.47:6789/0 conn(0x7fa9b00ce930 legacy=0x7fa9b006bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:10.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.680+0000 7fa9ade00640 1 -- 172.21.15.47:0/287913647 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b00cc2f0 con 0x7fa9b00b2fc0 2024-02-19T15:26:10.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.680+0000 7fa9ade00640 1 --2- 172.21.15.47:0/287913647 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00b2fc0 0x7fa9b00d0770 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa9b009fbb0 tx=0x7fa9a001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac13636da9a9da7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:10.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.680+0000 7fa9ad400640 1 -- 172.21.15.47:0/287913647 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a00ac640 con 0x7fa9b00b2fc0 2024-02-19T15:26:10.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.680+0000 7fa9ad400640 1 -- 172.21.15.47:0/287913647 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a00ac7a0 con 0x7fa9b00b2fc0 2024-02-19T15:26:10.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.680+0000 7fa9ad400640 1 -- 172.21.15.47:0/287913647 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a00acb30 con 0x7fa9b00b2fc0 2024-02-19T15:26:10.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.680+0000 7fa9b4a00640 1 -- 172.21.15.47:0/287913647 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00b2fc0 msgr2=0x7fa9b00d0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:10.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9b4a00640 1 --2- 172.21.15.47:0/287913647 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00b2fc0 0x7fa9b00d0770 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa9b009fbb0 tx=0x7fa9a001c3f0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9b4a00640 1 -- 172.21.15.47:0/287913647 shutdown_connections 2024-02-19T15:26:10.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9b4a00640 1 --2- 172.21.15.47:0/287913647 >> v2:172.21.15.116:3300/0 conn(0x7fa9b00cef90 0x7fa9b00ce3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9b4a00640 1 --2- 172.21.15.47:0/287913647 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00b2fc0 0x7fa9b00d0770 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9b4a00640 1 -- 172.21.15.47:0/287913647 >> 172.21.15.47:0/287913647 conn(0x7fa9b00d1b70 msgr2=0x7fa9b00d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:10.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9b4a00640 1 -- 172.21.15.47:0/287913647 shutdown_connections 2024-02-19T15:26:10.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9b4a00640 1 -- 172.21.15.47:0/287913647 wait complete. 2024-02-19T15:26:10.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9b4a00640 1 Processor -- start 2024-02-19T15:26:10.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9b4a00640 1 -- start start 2024-02-19T15:26:10.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9b4a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00b2fc0 0x7fa9b00c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:10.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9b4a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9b00ce930 0x7fa9b015ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:10.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9b4a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b00cef90 0x7fa9b0154ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:10.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9b4a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b0155af0 con 0x7fa9b00b2fc0 2024-02-19T15:26:10.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9b4a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b00ca660 con 0x7fa9b00ce930 2024-02-19T15:26:10.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9b4a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b00ca940 con 0x7fa9b00cef90 2024-02-19T15:26:10.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9ade00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00b2fc0 0x7fa9b00c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:10.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9ade00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00b2fc0 0x7fa9b00c9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50846/0 (socket says 172.21.15.47:50846) 2024-02-19T15:26:10.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9ade00640 1 -- 172.21.15.47:0/3065119628 learned_addr learned my addr 172.21.15.47:0/3065119628 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:10.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9a7e00640 1 --2- 172.21.15.47:0/3065119628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9b00ce930 0x7fa9b015ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:10.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9ade00640 1 -- 172.21.15.47:0/3065119628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b00cef90 msgr2=0x7fa9b0154ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:10.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9ae800640 1 --2- 172.21.15.47:0/3065119628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b00cef90 0x7fa9b0154ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:10.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9ade00640 1 --2- 172.21.15.47:0/3065119628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b00cef90 0x7fa9b0154ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9ade00640 1 -- 172.21.15.47:0/3065119628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9b00ce930 msgr2=0x7fa9b015ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:10.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9ade00640 1 --2- 172.21.15.47:0/3065119628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9b00ce930 0x7fa9b015ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9ade00640 1 -- 172.21.15.47:0/3065119628 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a0009420 con 0x7fa9b00b2fc0 2024-02-19T15:26:10.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9a7e00640 1 --2- 172.21.15.47:0/3065119628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9b00ce930 0x7fa9b015ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:10.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9ae800640 1 --2- 172.21.15.47:0/3065119628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b00cef90 0x7fa9b0154ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:10.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.684+0000 7fa9ade00640 1 --2- 172.21.15.47:0/3065119628 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00b2fc0 0x7fa9b00c9cf0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa9a00af040 tx=0x7fa9a001ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:10.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.688+0000 7fa9a6000640 1 -- 172.21.15.47:0/3065119628 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a00b8650 con 0x7fa9b00b2fc0 2024-02-19T15:26:10.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.688+0000 7fa9a6000640 1 -- 172.21.15.47:0/3065119628 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a00b87d0 con 0x7fa9b00b2fc0 2024-02-19T15:26:10.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.688+0000 7fa9a6000640 1 -- 172.21.15.47:0/3065119628 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a00b8b80 con 0x7fa9b00b2fc0 2024-02-19T15:26:10.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.688+0000 7fa9b4a00640 1 -- 172.21.15.47:0/3065119628 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b00cac20 con 0x7fa9b00b2fc0 2024-02-19T15:26:10.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.688+0000 7fa9b4a00640 1 -- 172.21.15.47:0/3065119628 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b00ca230 con 0x7fa9b00b2fc0 2024-02-19T15:26:10.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.688+0000 7fa9b4a00640 1 -- 172.21.15.47:0/3065119628 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b00ca420 con 0x7fa9b00b2fc0 2024-02-19T15:26:10.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.688+0000 7fa9a6000640 1 -- 172.21.15.47:0/3065119628 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa9a00b6400 con 0x7fa9b00b2fc0 2024-02-19T15:26:10.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.688+0000 7fa9a6000640 1 --2- 172.21.15.47:0/3065119628 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa98c066e70 0x7fa98c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:10.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.688+0000 7fa9a6000640 1 -- 172.21.15.47:0/3065119628 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9a011f6a0 con 0x7fa9b00b2fc0 2024-02-19T15:26:10.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.688+0000 7fa9a7e00640 1 --2- 172.21.15.47:0/3065119628 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa98c066e70 0x7fa98c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:10.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.688+0000 7fa9a7e00640 1 --2- 172.21.15.47:0/3065119628 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa98c066e70 0x7fa98c069330 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fa9b00b0720 tx=0x7fa998008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:10.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.696+0000 7fa9a6000640 1 -- 172.21.15.47:0/3065119628 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa9b00ca420 con 0x7fa9b00b2fc0 2024-02-19T15:26:10.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.988+0000 7fa9b4a00640 1 -- 172.21.15.47:0/3065119628 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa9b00ca420 con 0x7fa9b00b2fc0 2024-02-19T15:26:10.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.988+0000 7fa9a6000640 1 -- 172.21.15.47:0/3065119628 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v82) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa9b00ca420 con 0x7fa9b00b2fc0 2024-02-19T15:26:10.993 INFO:teuthology.orchestra.run.smithi047.stderr:listed 0 entries 2024-02-19T15:26:10.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.992+0000 7fa9b4a00640 1 -- 172.21.15.47:0/3065119628 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa98c066e70 msgr2=0x7fa98c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:10.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.992+0000 7fa9b4a00640 1 --2- 172.21.15.47:0/3065119628 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa98c066e70 0x7fa98c069330 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fa9b00b0720 tx=0x7fa998008040 comp rx=0 tx=0).stop 2024-02-19T15:26:10.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.992+0000 7fa9b4a00640 1 -- 172.21.15.47:0/3065119628 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00b2fc0 msgr2=0x7fa9b00c9cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:10.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.992+0000 7fa9b4a00640 1 --2- 172.21.15.47:0/3065119628 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00b2fc0 0x7fa9b00c9cf0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa9a00af040 tx=0x7fa9a001ac30 comp rx=0 tx=0).stop 2024-02-19T15:26:10.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.992+0000 7fa9b4a00640 1 -- 172.21.15.47:0/3065119628 shutdown_connections 2024-02-19T15:26:10.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.992+0000 7fa9b4a00640 1 --2- 172.21.15.47:0/3065119628 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa98c066e70 0x7fa98c069330 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.992+0000 7fa9b4a00640 1 --2- 172.21.15.47:0/3065119628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b00cef90 0x7fa9b0154ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.992+0000 7fa9b4a00640 1 --2- 172.21.15.47:0/3065119628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9b00ce930 0x7fa9b015ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.992+0000 7fa9b4a00640 1 --2- 172.21.15.47:0/3065119628 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00b2fc0 0x7fa9b00c9cf0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:10.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.992+0000 7fa9b4a00640 1 -- 172.21.15.47:0/3065119628 >> 172.21.15.47:0/3065119628 conn(0x7fa9b00d1b70 msgr2=0x7fa9b00703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:10.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.992+0000 7fa9b4a00640 1 -- 172.21.15.47:0/3065119628 shutdown_connections 2024-02-19T15:26:10.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:10.992+0000 7fa9b4a00640 1 -- 172.21.15.47:0/3065119628 wait complete. 2024-02-19T15:26:11.016 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-02-19T15:26:11.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:11.016 DEBUG:teuthology.orchestra.run.smithi047:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-02-19T15:26:11.043 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi047.front.sepia.ceph.com 2024-02-19T15:26:11.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:11.043 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-19T15:26:11.096 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:26:11.096 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:26:11.105 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:26:11.106 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:26:11.106 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:26:11.147 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:26:11.147 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:26:11.147 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:26:11.156 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:26:11.156 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:26:11.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.328+0000 7f24a9600640 1 Processor -- start 2024-02-19T15:26:11.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.328+0000 7f24a9600640 1 -- start start 2024-02-19T15:26:11.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.328+0000 7f24a9600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f24a40cded0 0x7f24a40ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:11.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.328+0000 7f24a9600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f24a40ceb40 0x7f24a406bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:11.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.328+0000 7f24a9600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a40cce30 con 0x7f24a406d650 2024-02-19T15:26:11.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.328+0000 7f24a9600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a40cefd0 con 0x7f24a40cded0 2024-02-19T15:26:11.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.328+0000 7f24a9600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a40cd110 con 0x7f24a40ceb40 2024-02-19T15:26:11.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.328+0000 7f24a2000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f24a40cded0 0x7f24a40ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:11.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.328+0000 7f24a2000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f24a40cded0 0x7f24a40ce600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50848/0 (socket says 172.21.15.47:50848) 2024-02-19T15:26:11.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.328+0000 7f24a2000640 1 -- 172.21.15.47:0/1115588763 learned_addr learned my addr 172.21.15.47:0/1115588763 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:11.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.328+0000 7f24a3400640 1 --2- 172.21.15.47:0/1115588763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f24a40ceb40 0x7f24a406bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:11.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.328+0000 7f24a2000640 1 -- 172.21.15.47:0/1115588763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f24a40ceb40 msgr2=0x7f24a406bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:11.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.328+0000 7f24a2000640 1 --2- 172.21.15.47:0/1115588763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f24a40ceb40 0x7f24a406bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:11.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.328+0000 7f24a2000640 1 -- 172.21.15.47:0/1115588763 >> v1:172.21.15.116:6789/0 conn(0x7f24a406d650 legacy=0x7f24a40d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:11.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.328+0000 7f24a2000640 1 -- 172.21.15.47:0/1115588763 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a40cd3f0 con 0x7f24a40cded0 2024-02-19T15:26:11.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.328+0000 7f24a3400640 1 --2- 172.21.15.47:0/1115588763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f24a40ceb40 0x7f24a406bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:11.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.332+0000 7f24a2000640 1 --2- 172.21.15.47:0/1115588763 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24a40cded0 0x7f24a40ce600 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f2490009870 tx=0x7f249001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faecf252488216df server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:11.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.332+0000 7f24a1600640 1 -- 172.21.15.47:0/1115588763 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24900ab620 con 0x7f24a40cded0 2024-02-19T15:26:11.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.332+0000 7f24a1600640 1 -- 172.21.15.47:0/1115588763 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24900ab780 con 0x7f24a40cded0 2024-02-19T15:26:11.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.332+0000 7f24a1600640 1 -- 172.21.15.47:0/1115588763 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24900abb10 con 0x7f24a40cded0 2024-02-19T15:26:11.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.332+0000 7f24a9600640 1 -- 172.21.15.47:0/1115588763 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24a40cded0 msgr2=0x7f24a40ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:11.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.332+0000 7f24a9600640 1 --2- 172.21.15.47:0/1115588763 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24a40cded0 0x7f24a40ce600 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f2490009870 tx=0x7f249001c460 comp rx=0 tx=0).stop 2024-02-19T15:26:11.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.332+0000 7f24a9600640 1 -- 172.21.15.47:0/1115588763 shutdown_connections 2024-02-19T15:26:11.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.332+0000 7f24a9600640 1 --2- 172.21.15.47:0/1115588763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f24a40ceb40 0x7f24a406bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:11.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.332+0000 7f24a9600640 1 --2- 172.21.15.47:0/1115588763 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24a40cded0 0x7f24a40ce600 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:11.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.332+0000 7f24a9600640 1 -- 172.21.15.47:0/1115588763 >> 172.21.15.47:0/1115588763 conn(0x7f24a40d1bf0 msgr2=0x7f24a40d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:11.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.332+0000 7f24a9600640 1 -- 172.21.15.47:0/1115588763 shutdown_connections 2024-02-19T15:26:11.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.332+0000 7f24a9600640 1 -- 172.21.15.47:0/1115588763 wait complete. 2024-02-19T15:26:11.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.332+0000 7f24a9600640 1 Processor -- start 2024-02-19T15:26:11.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.332+0000 7f24a9600640 1 -- start start 2024-02-19T15:26:11.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f24a9600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24a406d650 0x7f24a4160d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:11.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f24a9600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f24a40cded0 0x7f24a41598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:11.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f24a9600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24a40ceb40 0x7f24a40c8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:11.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f24a9600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a40cae00 con 0x7f24a40ceb40 2024-02-19T15:26:11.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f24a9600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a40ca230 con 0x7f24a406d650 2024-02-19T15:26:11.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f24a9600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a40ca4d0 con 0x7f24a40cded0 2024-02-19T15:26:11.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f24a3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24a40ceb40 0x7f24a40c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:11.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f24a2a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24a406d650 0x7f24a4160d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:11.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f24a3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24a40ceb40 0x7f24a40c8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50860/0 (socket says 172.21.15.47:50860) 2024-02-19T15:26:11.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f24a3400640 1 -- 172.21.15.47:0/938359298 learned_addr learned my addr 172.21.15.47:0/938359298 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:11.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f24a2000640 1 --2- 172.21.15.47:0/938359298 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f24a40cded0 0x7f24a41598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:11.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f24a3400640 1 -- 172.21.15.47:0/938359298 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f24a40cded0 msgr2=0x7f24a41598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:11.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f24a3400640 1 --2- 172.21.15.47:0/938359298 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f24a40cded0 0x7f24a41598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:11.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f24a3400640 1 -- 172.21.15.47:0/938359298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24a406d650 msgr2=0x7f24a4160d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:11.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f24a3400640 1 --2- 172.21.15.47:0/938359298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24a406d650 0x7f24a4160d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:11.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f24a3400640 1 -- 172.21.15.47:0/938359298 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24900094d0 con 0x7f24a40ceb40 2024-02-19T15:26:11.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f24a3400640 1 --2- 172.21.15.47:0/938359298 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24a40ceb40 0x7f24a40c8340 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f249c00cef0 tx=0x7f249c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:11.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f249aa00640 1 -- 172.21.15.47:0/938359298 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c00d840 con 0x7f24a40ceb40 2024-02-19T15:26:11.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f249aa00640 1 -- 172.21.15.47:0/938359298 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249c00d9c0 con 0x7f24a40ceb40 2024-02-19T15:26:11.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f249aa00640 1 -- 172.21.15.47:0/938359298 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c00dd70 con 0x7f24a40ceb40 2024-02-19T15:26:11.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f24a9600640 1 -- 172.21.15.47:0/938359298 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a40ca7b0 con 0x7f24a40ceb40 2024-02-19T15:26:11.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f24a9600640 1 -- 172.21.15.47:0/938359298 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a4159e20 con 0x7f24a40ceb40 2024-02-19T15:26:11.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.336+0000 7f24a9600640 1 -- 172.21.15.47:0/938359298 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a4075360 con 0x7f24a40ceb40 2024-02-19T15:26:11.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.340+0000 7f249aa00640 1 -- 172.21.15.47:0/938359298 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f249c01e3b0 con 0x7f24a40ceb40 2024-02-19T15:26:11.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.340+0000 7f249aa00640 1 --2- 172.21.15.47:0/938359298 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2484066e40 0x7f2484069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:11.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.340+0000 7f249aa00640 1 -- 172.21.15.47:0/938359298 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f249c085470 con 0x7f24a40ceb40 2024-02-19T15:26:11.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.340+0000 7f24a2a00640 1 --2- 172.21.15.47:0/938359298 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2484066e40 0x7f2484069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:11.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.340+0000 7f24a2a00640 1 --2- 172.21.15.47:0/938359298 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2484066e40 0x7f2484069300 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f24940098e0 tx=0x7f2494009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:11.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.344+0000 7f249aa00640 1 -- 172.21.15.47:0/938359298 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f249c0523c0 con 0x7f24a40ceb40 2024-02-19T15:26:11.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.540+0000 7f24a9600640 1 -- 172.21.15.47:0/938359298 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24a40708f0 con 0x7f24a40ceb40 2024-02-19T15:26:11.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.540+0000 7f249aa00640 1 -- 172.21.15.47:0/938359298 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4943 (secure 0 0 0) 0x7f249c055a80 con 0x7f24a40ceb40 2024-02-19T15:26:11.545 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:11.545 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5954,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1402677843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1402677843},{"type":"v1","addr":"172.21.15.116:6836","nonce":1402677843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":5957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1507554634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":1507554634},{"type":"v1","addr":"172.21.15.116:6837","nonce":1507554634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6771,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/888455604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":888455604},{"type":"v1","addr":"172.21.15.47:6833","nonce":888455604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:25:59.522224+0000","modified":"2024-02-19T15:26:02.535882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6774":{"gid":6774,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3598443511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3598443511},{"type":"v1","addr":"172.21.15.47:6835","nonce":3598443511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-19T15:26:11.546 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 49 2024-02-19T15:26:11.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.544+0000 7f24a9600640 1 -- 172.21.15.47:0/938359298 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2484066e40 msgr2=0x7f2484069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:11.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.544+0000 7f24a9600640 1 --2- 172.21.15.47:0/938359298 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2484066e40 0x7f2484069300 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f24940098e0 tx=0x7f2494009060 comp rx=0 tx=0).stop 2024-02-19T15:26:11.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.544+0000 7f24a9600640 1 -- 172.21.15.47:0/938359298 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24a40ceb40 msgr2=0x7f24a40c8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:11.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.544+0000 7f24a9600640 1 --2- 172.21.15.47:0/938359298 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24a40ceb40 0x7f24a40c8340 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f249c00cef0 tx=0x7f249c00a710 comp rx=0 tx=0).stop 2024-02-19T15:26:11.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.544+0000 7f24a9600640 1 -- 172.21.15.47:0/938359298 shutdown_connections 2024-02-19T15:26:11.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.544+0000 7f24a9600640 1 --2- 172.21.15.47:0/938359298 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2484066e40 0x7f2484069300 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:11.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.544+0000 7f24a9600640 1 --2- 172.21.15.47:0/938359298 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24a40ceb40 0x7f24a40c8340 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:11.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.544+0000 7f24a9600640 1 --2- 172.21.15.47:0/938359298 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f24a40cded0 0x7f24a41598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:11.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.544+0000 7f24a9600640 1 --2- 172.21.15.47:0/938359298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24a406d650 0x7f24a4160d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:11.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.544+0000 7f24a9600640 1 -- 172.21.15.47:0/938359298 >> 172.21.15.47:0/938359298 conn(0x7f24a40d1bf0 msgr2=0x7f24a40d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:11.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.544+0000 7f24a9600640 1 -- 172.21.15.47:0/938359298 shutdown_connections 2024-02-19T15:26:11.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.544+0000 7f24a9600640 1 -- 172.21.15.47:0/938359298 wait complete. 2024-02-19T15:26:11.566 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:26:11.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.700+0000 7f7d1c400640 1 Processor -- start 2024-02-19T15:26:11.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.700+0000 7f7d1c400640 1 -- start start 2024-02-19T15:26:11.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.704+0000 7f7d1c400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d14145590 0x7f7d140727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:11.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.704+0000 7f7d1c400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7d140cea80 0x7f7d1406c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:11.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.704+0000 7f7d1c400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d1415d280 con 0x7f7d1415c390 2024-02-19T15:26:11.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.704+0000 7f7d1c400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d14070ff0 con 0x7f7d140cea80 2024-02-19T15:26:11.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.704+0000 7f7d1c400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d140709e0 con 0x7f7d14145590 2024-02-19T15:26:11.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.704+0000 7f7d1a400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f7d1415c390 0x7f7d1415c820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54914/0 (socket says 172.21.15.47:54914) 2024-02-19T15:26:11.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.704+0000 7f7d1a400640 1 -- 172.21.15.47:0/282346433 learned_addr learned my addr 172.21.15.47:0/282346433 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:11.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.704+0000 7f7d19a00640 1 --2- 172.21.15.47:0/282346433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d14145590 0x7f7d140727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:11.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.704+0000 7f7d19000640 1 --2- 172.21.15.47:0/282346433 >> v2:172.21.15.116:3300/0 conn(0x7f7d140cea80 0x7f7d1406c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:11.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.704+0000 7f7d13e00640 1 -- 172.21.15.47:0/282346433 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331113236 0 0) 0x7f7d1415d280 con 0x7f7d1415c390 2024-02-19T15:26:11.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.704+0000 7f7d13e00640 1 -- 172.21.15.47:0/282346433 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cfc003530 con 0x7f7d1415c390 2024-02-19T15:26:11.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.704+0000 7f7d19a00640 1 -- 172.21.15.47:0/282346433 >> v2:172.21.15.116:3300/0 conn(0x7f7d140cea80 msgr2=0x7f7d1406c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:11.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.704+0000 7f7d19a00640 1 --2- 172.21.15.47:0/282346433 >> v2:172.21.15.116:3300/0 conn(0x7f7d140cea80 0x7f7d1406c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:11.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.704+0000 7f7d19a00640 1 -- 172.21.15.47:0/282346433 >> v1:172.21.15.47:6789/0 conn(0x7f7d1415c390 legacy=0x7f7d1415c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:11.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.704+0000 7f7d19a00640 1 -- 172.21.15.47:0/282346433 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1406cee0 con 0x7f7d14145590 2024-02-19T15:26:11.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.704+0000 7f7d19000640 1 --2- 172.21.15.47:0/282346433 >> v2:172.21.15.116:3300/0 conn(0x7f7d140cea80 0x7f7d1406c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:11.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.708+0000 7f7d19a00640 1 --2- 172.21.15.47:0/282346433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d14145590 0x7f7d140727b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7d080097f0 tx=0x7f7d0801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa2bed09ab011cfd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:11.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.708+0000 7f7d13e00640 1 -- 172.21.15.47:0/282346433 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d080ac410 con 0x7f7d14145590 2024-02-19T15:26:11.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.708+0000 7f7d13e00640 1 -- 172.21.15.47:0/282346433 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d080ac570 con 0x7f7d14145590 2024-02-19T15:26:11.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.708+0000 7f7d13e00640 1 -- 172.21.15.47:0/282346433 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d080ac900 con 0x7f7d14145590 2024-02-19T15:26:11.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.708+0000 7f7d1c400640 1 -- 172.21.15.47:0/282346433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d14145590 msgr2=0x7f7d140727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:11.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.708+0000 7f7d1c400640 1 --2- 172.21.15.47:0/282346433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d14145590 0x7f7d140727b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7d080097f0 tx=0x7f7d0801c4a0 comp rx=0 tx=0).stop 2024-02-19T15:26:11.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.708+0000 7f7d1c400640 1 -- 172.21.15.47:0/282346433 shutdown_connections 2024-02-19T15:26:11.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.708+0000 7f7d1c400640 1 --2- 172.21.15.47:0/282346433 >> v2:172.21.15.116:3300/0 conn(0x7f7d140cea80 0x7f7d1406c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:11.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.708+0000 7f7d1c400640 1 --2- 172.21.15.47:0/282346433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d14145590 0x7f7d140727b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:11.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.708+0000 7f7d1c400640 1 -- 172.21.15.47:0/282346433 >> 172.21.15.47:0/282346433 conn(0x7f7d140b16a0 msgr2=0x7f7d140b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:11.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.712+0000 7f7d1c400640 1 -- 172.21.15.47:0/282346433 shutdown_connections 2024-02-19T15:26:11.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.712+0000 7f7d1c400640 1 -- 172.21.15.47:0/282346433 wait complete. 2024-02-19T15:26:11.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.712+0000 7f7d1c400640 1 Processor -- start 2024-02-19T15:26:11.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.712+0000 7f7d1c400640 1 -- start start 2024-02-19T15:26:11.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d1c400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d140cea80 0x7f7d140c49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:11.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d1c400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d14145590 0x7f7d140c4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:11.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d1c400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d1415c390 0x7f7d14140810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:11.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d1c400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d141426e0 con 0x7f7d14145590 2024-02-19T15:26:11.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d1c400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d141429c0 con 0x7f7d140cea80 2024-02-19T15:26:11.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d1c400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d14142ca0 con 0x7f7d1415c390 2024-02-19T15:26:11.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d19000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d14145590 0x7f7d140c4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:11.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d19000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d14145590 0x7f7d140c4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50868/0 (socket says 172.21.15.47:50868) 2024-02-19T15:26:11.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d19000640 1 -- 172.21.15.47:0/356320846 learned_addr learned my addr 172.21.15.47:0/356320846 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:11.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d19a00640 1 --2- 172.21.15.47:0/356320846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d140cea80 0x7f7d140c49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:11.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d1a400640 1 --2- 172.21.15.47:0/356320846 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d1415c390 0x7f7d14140810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:11.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d19000640 1 -- 172.21.15.47:0/356320846 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d1415c390 msgr2=0x7f7d14140810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:11.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d19000640 1 --2- 172.21.15.47:0/356320846 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d1415c390 0x7f7d14140810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:11.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d19000640 1 -- 172.21.15.47:0/356320846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d140cea80 msgr2=0x7f7d140c49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:11.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d19000640 1 --2- 172.21.15.47:0/356320846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d140cea80 0x7f7d140c49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:11.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d19000640 1 -- 172.21.15.47:0/356320846 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d00009420 con 0x7f7d14145590 2024-02-19T15:26:11.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d1a400640 1 --2- 172.21.15.47:0/356320846 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d1415c390 0x7f7d14140810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:11.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d19000640 1 --2- 172.21.15.47:0/356320846 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d14145590 0x7f7d140c4f00 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f7d0000ebe0 tx=0x7f7d0000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:11.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d11600640 1 -- 172.21.15.47:0/356320846 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0000eee0 con 0x7f7d14145590 2024-02-19T15:26:11.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d11600640 1 -- 172.21.15.47:0/356320846 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0000cc30 con 0x7f7d14145590 2024-02-19T15:26:11.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d1c400640 1 -- 172.21.15.47:0/356320846 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d08009420 con 0x7f7d14145590 2024-02-19T15:26:11.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d11600640 1 -- 172.21.15.47:0/356320846 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d00013470 con 0x7f7d14145590 2024-02-19T15:26:11.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d1c400640 1 -- 172.21.15.47:0/356320846 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d14143260 con 0x7f7d14145590 2024-02-19T15:26:11.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d1c400640 1 -- 172.21.15.47:0/356320846 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d14075360 con 0x7f7d14145590 2024-02-19T15:26:11.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.716+0000 7f7d11600640 1 -- 172.21.15.47:0/356320846 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7d000135d0 con 0x7f7d14145590 2024-02-19T15:26:11.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.720+0000 7f7d11600640 1 --2- 172.21.15.47:0/356320846 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7cf4066e70 0x7f7cf4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:11.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.720+0000 7f7d11600640 1 -- 172.21.15.47:0/356320846 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d00084d70 con 0x7f7d14145590 2024-02-19T15:26:11.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.720+0000 7f7d19a00640 1 --2- 172.21.15.47:0/356320846 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7cf4066e70 0x7f7cf4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:11.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.720+0000 7f7d19a00640 1 --2- 172.21.15.47:0/356320846 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7cf4066e70 0x7f7cf4069330 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7d080ae000 tx=0x7f7d080a6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:11.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.724+0000 7f7d11600640 1 -- 172.21.15.47:0/356320846 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7d00051b50 con 0x7f7d14145590 2024-02-19T15:26:11.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.904+0000 7f7d1c400640 1 -- 172.21.15.47:0/356320846 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d140ccc60 con 0x7f7d14145590 2024-02-19T15:26:11.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.904+0000 7f7d11600640 1 -- 172.21.15.47:0/356320846 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16672 (secure 0 0 0) 0x7f7d00055210 con 0x7f7d14145590 2024-02-19T15:26:11.909 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:11.910 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:26:01.137113+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:25:57.938354+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-02-19T15:25:58.514866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:26:11.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.908+0000 7f7d1c400640 1 -- 172.21.15.47:0/356320846 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7cf4066e70 msgr2=0x7f7cf4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:11.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.908+0000 7f7d1c400640 1 --2- 172.21.15.47:0/356320846 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7cf4066e70 0x7f7cf4069330 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7d080ae000 tx=0x7f7d080a6040 comp rx=0 tx=0).stop 2024-02-19T15:26:11.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.908+0000 7f7d1c400640 1 -- 172.21.15.47:0/356320846 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d14145590 msgr2=0x7f7d140c4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:11.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.908+0000 7f7d1c400640 1 --2- 172.21.15.47:0/356320846 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d14145590 0x7f7d140c4f00 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f7d0000ebe0 tx=0x7f7d0000c520 comp rx=0 tx=0).stop 2024-02-19T15:26:11.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.908+0000 7f7d1c400640 1 -- 172.21.15.47:0/356320846 shutdown_connections 2024-02-19T15:26:11.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.908+0000 7f7d1c400640 1 --2- 172.21.15.47:0/356320846 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7cf4066e70 0x7f7cf4069330 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:11.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.908+0000 7f7d1c400640 1 --2- 172.21.15.47:0/356320846 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d1415c390 0x7f7d14140810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:11.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.908+0000 7f7d1c400640 1 --2- 172.21.15.47:0/356320846 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d14145590 0x7f7d140c4f00 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:11.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.908+0000 7f7d1c400640 1 --2- 172.21.15.47:0/356320846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d140cea80 0x7f7d140c49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:11.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.908+0000 7f7d1c400640 1 -- 172.21.15.47:0/356320846 >> 172.21.15.47:0/356320846 conn(0x7f7d140b16a0 msgr2=0x7f7d140af7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:11.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.908+0000 7f7d1c400640 1 -- 172.21.15.47:0/356320846 shutdown_connections 2024-02-19T15:26:11.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:11.908+0000 7f7d1c400640 1 -- 172.21.15.47:0/356320846 wait complete. 2024-02-19T15:26:11.931 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:25:57.938354+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:25:58.514866+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:26:11.932 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:26:12.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.076+0000 7fa64b600640 1 Processor -- start 2024-02-19T15:26:12.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.076+0000 7fa64b600640 1 -- start start 2024-02-19T15:26:12.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.076+0000 7fa64b600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6440d2650 0x7fa64406c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:12.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.076+0000 7fa64b600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6440cc620 con 0x7fa64415e280 2024-02-19T15:26:12.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.076+0000 7fa64b600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6440d12e0 con 0x7fa6440ce3b0 2024-02-19T15:26:12.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.076+0000 7fa64b600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6440d1940 con 0x7fa6440d2650 2024-02-19T15:26:12.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.080+0000 7fa649600640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa64415e280 0x7fa64406c710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54926/0 (socket says 172.21.15.47:54926) 2024-02-19T15:26:12.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.080+0000 7fa649600640 1 -- 172.21.15.47:0/4204845944 learned_addr learned my addr 172.21.15.47:0/4204845944 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:12.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.080+0000 7fa643e00640 1 --2- 172.21.15.47:0/4204845944 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6440d2650 0x7fa64406c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:12.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.080+0000 7fa643400640 1 -- 172.21.15.47:0/4204845944 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119550652 0 0) 0x7fa6440cc620 con 0x7fa64415e280 2024-02-19T15:26:12.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.080+0000 7fa643400640 1 -- 172.21.15.47:0/4204845944 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa62c003530 con 0x7fa64415e280 2024-02-19T15:26:12.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.080+0000 7fa643400640 1 -- 172.21.15.47:0/4204845944 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251769368 0 0) 0x7fa6440d12e0 con 0x7fa6440ce3b0 2024-02-19T15:26:12.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.080+0000 7fa643400640 1 -- 172.21.15.47:0/4204845944 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6440cc620 con 0x7fa6440ce3b0 2024-02-19T15:26:12.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.080+0000 7fa643e00640 1 -- 172.21.15.47:0/4204845944 >> v1:172.21.15.116:6789/0 conn(0x7fa6440ce3b0 legacy=0x7fa6440ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:12.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.080+0000 7fa643e00640 1 -- 172.21.15.47:0/4204845944 >> v1:172.21.15.47:6789/0 conn(0x7fa64415e280 legacy=0x7fa64406c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:12.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.080+0000 7fa643e00640 1 -- 172.21.15.47:0/4204845944 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6440d1000 con 0x7fa6440d2650 2024-02-19T15:26:12.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.080+0000 7fa643e00640 1 --2- 172.21.15.47:0/4204845944 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6440d2650 0x7fa64406c1d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa6300097f0 tx=0x7fa63001c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66d4dcdca68e5ef6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:12.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.080+0000 7fa643400640 1 -- 172.21.15.47:0/4204845944 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6300ab600 con 0x7fa6440d2650 2024-02-19T15:26:12.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.080+0000 7fa643400640 1 -- 172.21.15.47:0/4204845944 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6300a7050 con 0x7fa6440d2650 2024-02-19T15:26:12.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.080+0000 7fa643400640 1 -- 172.21.15.47:0/4204845944 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6300ab980 con 0x7fa6440d2650 2024-02-19T15:26:12.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa64b600640 1 -- 172.21.15.47:0/4204845944 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6440d2650 msgr2=0x7fa64406c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:12.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa64b600640 1 --2- 172.21.15.47:0/4204845944 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6440d2650 0x7fa64406c1d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa6300097f0 tx=0x7fa63001c3c0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa64b600640 1 -- 172.21.15.47:0/4204845944 shutdown_connections 2024-02-19T15:26:12.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa64b600640 1 --2- 172.21.15.47:0/4204845944 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6440d2650 0x7fa64406c1d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa64b600640 1 -- 172.21.15.47:0/4204845944 >> 172.21.15.47:0/4204845944 conn(0x7fa6440d1b30 msgr2=0x7fa6440d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:12.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa64b600640 1 -- 172.21.15.47:0/4204845944 shutdown_connections 2024-02-19T15:26:12.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa64b600640 1 -- 172.21.15.47:0/4204845944 wait complete. 2024-02-19T15:26:12.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa64b600640 1 Processor -- start 2024-02-19T15:26:12.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa64b600640 1 -- start start 2024-02-19T15:26:12.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa64b600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6440ce3b0 0x7fa64415fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:12.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa64b600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6440d2650 0x7fa644159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:12.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa64b600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64415e280 0x7fa64415e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:12.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa64b600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa644159860 con 0x7fa64415e280 2024-02-19T15:26:12.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa64b600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64415eba0 con 0x7fa6440d2650 2024-02-19T15:26:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa64b600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa64415a650 con 0x7fa6440ce3b0 2024-02-19T15:26:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa649600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64415e280 0x7fa64415e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa649600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64415e280 0x7fa64415e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50872/0 (socket says 172.21.15.47:50872) 2024-02-19T15:26:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa649600640 1 -- 172.21.15.47:0/588791385 learned_addr learned my addr 172.21.15.47:0/588791385 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa648c00640 1 --2- 172.21.15.47:0/588791385 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6440ce3b0 0x7fa64415fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa643e00640 1 --2- 172.21.15.47:0/588791385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6440d2650 0x7fa644159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa649600640 1 -- 172.21.15.47:0/588791385 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6440ce3b0 msgr2=0x7fa64415fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa649600640 1 --2- 172.21.15.47:0/588791385 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6440ce3b0 0x7fa64415fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa649600640 1 -- 172.21.15.47:0/588791385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6440d2650 msgr2=0x7fa644159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa649600640 1 --2- 172.21.15.47:0/588791385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6440d2650 0x7fa644159320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa649600640 1 -- 172.21.15.47:0/588791385 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa630009420 con 0x7fa64415e280 2024-02-19T15:26:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.084+0000 7fa643e00640 1 --2- 172.21.15.47:0/588791385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6440d2650 0x7fa644159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:12.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.088+0000 7fa649600640 1 --2- 172.21.15.47:0/588791385 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64415e280 0x7fa64415e660 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00cb70 tx=0x7fa63c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:12.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.088+0000 7fa640c00640 1 -- 172.21.15.47:0/588791385 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c00ace0 con 0x7fa64415e280 2024-02-19T15:26:12.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.088+0000 7fa640c00640 1 -- 172.21.15.47:0/588791385 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa63c00ae40 con 0x7fa64415e280 2024-02-19T15:26:12.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.088+0000 7fa640c00640 1 -- 172.21.15.47:0/588791385 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c012620 con 0x7fa64415e280 2024-02-19T15:26:12.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.088+0000 7fa64b600640 1 -- 172.21.15.47:0/588791385 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64415ed90 con 0x7fa64415e280 2024-02-19T15:26:12.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.088+0000 7fa64b600640 1 -- 172.21.15.47:0/588791385 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa64415f350 con 0x7fa64415e280 2024-02-19T15:26:12.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.088+0000 7fa64b600640 1 -- 172.21.15.47:0/588791385 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa644075360 con 0x7fa64415e280 2024-02-19T15:26:12.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.088+0000 7fa640c00640 1 -- 172.21.15.47:0/588791385 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa63c007070 con 0x7fa64415e280 2024-02-19T15:26:12.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.088+0000 7fa640c00640 1 --2- 172.21.15.47:0/588791385 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa624066e70 0x7fa624069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:12.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.088+0000 7fa640c00640 1 -- 172.21.15.47:0/588791385 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa63c084aa0 con 0x7fa64415e280 2024-02-19T15:26:12.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.088+0000 7fa648c00640 1 --2- 172.21.15.47:0/588791385 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa624066e70 0x7fa624069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:12.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.092+0000 7fa648c00640 1 --2- 172.21.15.47:0/588791385 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa624066e70 0x7fa624069330 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa6380011f0 tx=0x7fa638009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:12.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.092+0000 7fa640c00640 1 -- 172.21.15.47:0/588791385 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa63c055320 con 0x7fa64415e280 2024-02-19T15:26:12.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.256+0000 7fa64b600640 1 -- 172.21.15.47:0/588791385 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa6440cdf80 con 0x7fa64415e280 2024-02-19T15:26:12.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.256+0000 7fa640c00640 1 -- 172.21.15.47:0/588791385 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa63c017020 con 0x7fa64415e280 2024-02-19T15:26:12.262 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:26:12.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.260+0000 7fa64b600640 1 -- 172.21.15.47:0/588791385 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa624066e70 msgr2=0x7fa624069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:12.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.260+0000 7fa64b600640 1 --2- 172.21.15.47:0/588791385 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa624066e70 0x7fa624069330 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa6380011f0 tx=0x7fa638009040 comp rx=0 tx=0).stop 2024-02-19T15:26:12.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.260+0000 7fa64b600640 1 -- 172.21.15.47:0/588791385 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64415e280 msgr2=0x7fa64415e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:12.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.260+0000 7fa64b600640 1 --2- 172.21.15.47:0/588791385 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64415e280 0x7fa64415e660 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00cb70 tx=0x7fa63c00a430 comp rx=0 tx=0).stop 2024-02-19T15:26:12.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.260+0000 7fa64b600640 1 -- 172.21.15.47:0/588791385 shutdown_connections 2024-02-19T15:26:12.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.260+0000 7fa64b600640 1 --2- 172.21.15.47:0/588791385 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa624066e70 0x7fa624069330 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.260+0000 7fa64b600640 1 --2- 172.21.15.47:0/588791385 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64415e280 0x7fa64415e660 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.260+0000 7fa64b600640 1 --2- 172.21.15.47:0/588791385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6440d2650 0x7fa644159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.260+0000 7fa64b600640 1 --2- 172.21.15.47:0/588791385 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6440ce3b0 0x7fa64415fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.260+0000 7fa64b600640 1 -- 172.21.15.47:0/588791385 >> 172.21.15.47:0/588791385 conn(0x7fa6440d1b30 msgr2=0x7fa64406bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:12.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.260+0000 7fa64b600640 1 -- 172.21.15.47:0/588791385 shutdown_connections 2024-02-19T15:26:12.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.260+0000 7fa64b600640 1 -- 172.21.15.47:0/588791385 wait complete. 2024-02-19T15:26:12.282 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:26:12.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.424+0000 7fb17ba00640 1 Processor -- start 2024-02-19T15:26:12.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.424+0000 7fb17ba00640 1 -- start start 2024-02-19T15:26:12.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.424+0000 7fb17ba00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb1740ce3b0 0x7fb1740ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:12.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.424+0000 7fb17ba00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1740d2650 0x7fb1740ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:12.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.424+0000 7fb17ba00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1740cc620 con 0x7fb17415e280 2024-02-19T15:26:12.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.424+0000 7fb17ba00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1740d12e0 con 0x7fb1740ce3b0 2024-02-19T15:26:12.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.424+0000 7fb17ba00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1740d1940 con 0x7fb1740d2650 2024-02-19T15:26:12.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.424+0000 7fb179a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb17415e280 0x7fb1740ccff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54932/0 (socket says 172.21.15.47:54932) 2024-02-19T15:26:12.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.424+0000 7fb179a00640 1 -- 172.21.15.47:0/3331015326 learned_addr learned my addr 172.21.15.47:0/3331015326 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:12.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.424+0000 7fb16be00640 1 --2- 172.21.15.47:0/3331015326 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1740d2650 0x7fb1740ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:12.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.424+0000 7fb179000640 1 --2- 172.21.15.47:0/3331015326 >> v2:172.21.15.116:3300/0 conn(0x7fb1740ce3b0 0x7fb1740ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:12.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.424+0000 7fb173e00640 1 -- 172.21.15.47:0/3331015326 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774557121 0 0) 0x7fb1740cc620 con 0x7fb17415e280 2024-02-19T15:26:12.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.424+0000 7fb173e00640 1 -- 172.21.15.47:0/3331015326 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb154003530 con 0x7fb17415e280 2024-02-19T15:26:12.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.424+0000 7fb16be00640 1 -- 172.21.15.47:0/3331015326 >> v2:172.21.15.116:3300/0 conn(0x7fb1740ce3b0 msgr2=0x7fb1740ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:12.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.424+0000 7fb16be00640 1 --2- 172.21.15.47:0/3331015326 >> v2:172.21.15.116:3300/0 conn(0x7fb1740ce3b0 0x7fb1740ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.424+0000 7fb16be00640 1 -- 172.21.15.47:0/3331015326 >> v1:172.21.15.47:6789/0 conn(0x7fb17415e280 legacy=0x7fb1740ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:12.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.424+0000 7fb16be00640 1 -- 172.21.15.47:0/3331015326 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1740cecd0 con 0x7fb1740d2650 2024-02-19T15:26:12.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.424+0000 7fb179000640 1 --2- 172.21.15.47:0/3331015326 >> v2:172.21.15.116:3300/0 conn(0x7fb1740ce3b0 0x7fb1740ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:12.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.432+0000 7fb16be00640 1 --2- 172.21.15.47:0/3331015326 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1740d2650 0x7fb1740ccab0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb15c0097c0 tx=0x7fb15c01c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0c448cb09ffae30 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:12.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.432+0000 7fb173e00640 1 -- 172.21.15.47:0/3331015326 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c0ab600 con 0x7fb1740d2650 2024-02-19T15:26:12.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.432+0000 7fb173e00640 1 -- 172.21.15.47:0/3331015326 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15c0a7050 con 0x7fb1740d2650 2024-02-19T15:26:12.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.432+0000 7fb173e00640 1 -- 172.21.15.47:0/3331015326 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c0ab980 con 0x7fb1740d2650 2024-02-19T15:26:12.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.432+0000 7fb17ba00640 1 -- 172.21.15.47:0/3331015326 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1740d2650 msgr2=0x7fb1740ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:12.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.432+0000 7fb17ba00640 1 --2- 172.21.15.47:0/3331015326 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1740d2650 0x7fb1740ccab0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb15c0097c0 tx=0x7fb15c01c380 comp rx=0 tx=0).stop 2024-02-19T15:26:12.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.432+0000 7fb17ba00640 1 -- 172.21.15.47:0/3331015326 shutdown_connections 2024-02-19T15:26:12.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.432+0000 7fb17ba00640 1 --2- 172.21.15.47:0/3331015326 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1740d2650 0x7fb1740ccab0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.432+0000 7fb17ba00640 1 --2- 172.21.15.47:0/3331015326 >> v2:172.21.15.116:3300/0 conn(0x7fb1740ce3b0 0x7fb1740ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.432+0000 7fb17ba00640 1 -- 172.21.15.47:0/3331015326 >> 172.21.15.47:0/3331015326 conn(0x7fb1740d1b30 msgr2=0x7fb1740d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:12.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.432+0000 7fb17ba00640 1 -- 172.21.15.47:0/3331015326 shutdown_connections 2024-02-19T15:26:12.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.432+0000 7fb17ba00640 1 -- 172.21.15.47:0/3331015326 wait complete. 2024-02-19T15:26:12.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.432+0000 7fb17ba00640 1 Processor -- start 2024-02-19T15:26:12.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.432+0000 7fb17ba00640 1 -- start start 2024-02-19T15:26:12.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb17ba00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1740ce3b0 0x7fb17415b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:12.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb17ba00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1740d2650 0x7fb1741f70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:12.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb17ba00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb17415e280 0x7fb17409c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:12.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb17ba00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17409be60 con 0x7fb1740d2650 2024-02-19T15:26:12.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb17ba00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17415b770 con 0x7fb1740ce3b0 2024-02-19T15:26:12.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb17ba00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb17409da60 con 0x7fb17415e280 2024-02-19T15:26:12.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb16be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1740d2650 0x7fb1741f70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:12.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb16be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1740d2650 0x7fb1741f70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50882/0 (socket says 172.21.15.47:50882) 2024-02-19T15:26:12.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb16be00640 1 -- 172.21.15.47:0/1748079246 learned_addr learned my addr 172.21.15.47:0/1748079246 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:12.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb179a00640 1 --2- 172.21.15.47:0/1748079246 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb17415e280 0x7fb17409c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:12.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb179000640 1 --2- 172.21.15.47:0/1748079246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1740ce3b0 0x7fb17415b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:12.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb16be00640 1 -- 172.21.15.47:0/1748079246 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb17415e280 msgr2=0x7fb17409c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:12.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb16be00640 1 --2- 172.21.15.47:0/1748079246 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb17415e280 0x7fb17409c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb16be00640 1 -- 172.21.15.47:0/1748079246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1740ce3b0 msgr2=0x7fb17415b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:12.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb16be00640 1 --2- 172.21.15.47:0/1748079246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1740ce3b0 0x7fb17415b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb16be00640 1 -- 172.21.15.47:0/1748079246 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb164009420 con 0x7fb1740d2650 2024-02-19T15:26:12.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb16be00640 1 --2- 172.21.15.47:0/1748079246 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1740d2650 0x7fb1741f70e0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fb15c018f90 tx=0x7fb15c019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:12.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb171600640 1 -- 172.21.15.47:0/1748079246 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c0b4400 con 0x7fb1740d2650 2024-02-19T15:26:12.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb171600640 1 -- 172.21.15.47:0/1748079246 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15c0b4560 con 0x7fb1740d2650 2024-02-19T15:26:12.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb171600640 1 -- 172.21.15.47:0/1748079246 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c0b4910 con 0x7fb1740d2650 2024-02-19T15:26:12.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb17ba00640 1 -- 172.21.15.47:0/1748079246 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb15c009420 con 0x7fb1740d2650 2024-02-19T15:26:12.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb17ba00640 1 -- 172.21.15.47:0/1748079246 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb17409d780 con 0x7fb1740d2650 2024-02-19T15:26:12.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.436+0000 7fb17ba00640 1 -- 172.21.15.47:0/1748079246 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb17409bfa0 con 0x7fb1740d2650 2024-02-19T15:26:12.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.440+0000 7fb171600640 1 -- 172.21.15.47:0/1748079246 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb15c0b2040 con 0x7fb1740d2650 2024-02-19T15:26:12.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.440+0000 7fb171600640 1 --2- 172.21.15.47:0/1748079246 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb150066e40 0x7fb150069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:12.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.440+0000 7fb171600640 1 -- 172.21.15.47:0/1748079246 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb15c11e970 con 0x7fb1740d2650 2024-02-19T15:26:12.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.440+0000 7fb179000640 1 --2- 172.21.15.47:0/1748079246 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb150066e40 0x7fb150069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:12.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.440+0000 7fb179000640 1 --2- 172.21.15.47:0/1748079246 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb150066e40 0x7fb150069300 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb164009c50 tx=0x7fb164009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:12.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.444+0000 7fb171600640 1 -- 172.21.15.47:0/1748079246 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb17409bfa0 con 0x7fb1740d2650 2024-02-19T15:26:12.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.604+0000 7fb17ba00640 1 -- 172.21.15.47:0/1748079246 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb17409bfa0 con 0x7fb1740d2650 2024-02-19T15:26:12.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.608+0000 7fb171600640 1 -- 172.21.15.47:0/1748079246 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb17409bfa0 con 0x7fb1740d2650 2024-02-19T15:26:12.611 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:26:12.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.608+0000 7fb17ba00640 1 -- 172.21.15.47:0/1748079246 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb150066e40 msgr2=0x7fb150069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:12.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.608+0000 7fb17ba00640 1 --2- 172.21.15.47:0/1748079246 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb150066e40 0x7fb150069300 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb164009c50 tx=0x7fb164009190 comp rx=0 tx=0).stop 2024-02-19T15:26:12.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.608+0000 7fb17ba00640 1 -- 172.21.15.47:0/1748079246 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1740d2650 msgr2=0x7fb1741f70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:12.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.608+0000 7fb17ba00640 1 --2- 172.21.15.47:0/1748079246 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1740d2650 0x7fb1741f70e0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fb15c018f90 tx=0x7fb15c019120 comp rx=0 tx=0).stop 2024-02-19T15:26:12.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.608+0000 7fb17ba00640 1 -- 172.21.15.47:0/1748079246 shutdown_connections 2024-02-19T15:26:12.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.608+0000 7fb17ba00640 1 --2- 172.21.15.47:0/1748079246 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb150066e40 0x7fb150069300 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.608+0000 7fb17ba00640 1 --2- 172.21.15.47:0/1748079246 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb17415e280 0x7fb17409c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.608+0000 7fb17ba00640 1 --2- 172.21.15.47:0/1748079246 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1740d2650 0x7fb1741f70e0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.608+0000 7fb17ba00640 1 --2- 172.21.15.47:0/1748079246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1740ce3b0 0x7fb17415b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.608+0000 7fb17ba00640 1 -- 172.21.15.47:0/1748079246 >> 172.21.15.47:0/1748079246 conn(0x7fb1740d1b30 msgr2=0x7fb1740c57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:12.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.608+0000 7fb17ba00640 1 -- 172.21.15.47:0/1748079246 shutdown_connections 2024-02-19T15:26:12.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.608+0000 7fb17ba00640 1 -- 172.21.15.47:0/1748079246 wait complete. 2024-02-19T15:26:12.632 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:26:12.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.828+0000 7fead3600640 1 Processor -- start 2024-02-19T15:26:12.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.828+0000 7fead3600640 1 -- start start 2024-02-19T15:26:12.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.832+0000 7fead3600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feacc15e160 0x7feacc0d1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:12.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.832+0000 7fead3600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7feacc06f220 0x7feacc06dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:12.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.832+0000 7fead3600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feacc071a70 con 0x7feacc06e010 2024-02-19T15:26:12.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.832+0000 7fead3600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feacc06e440 con 0x7feacc06f220 2024-02-19T15:26:12.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.832+0000 7fead3600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feacc0ced30 con 0x7feacc15e160 2024-02-19T15:26:12.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.832+0000 7feac3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7feacc06f220 0x7feacc06dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:12.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.832+0000 7feac3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7feacc06f220 0x7feacc06dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50892/0 (socket says 172.21.15.47:50892) 2024-02-19T15:26:12.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.832+0000 7feac3e00640 1 -- 172.21.15.47:0/2482767879 learned_addr learned my addr 172.21.15.47:0/2482767879 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:12.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.832+0000 7fead0c00640 1 --2- 172.21.15.47:0/2482767879 >> v2:172.21.15.116:3300/0 conn(0x7feacc15e160 0x7feacc0d1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:12.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.832+0000 7feac3e00640 1 -- 172.21.15.47:0/2482767879 >> v2:172.21.15.116:3300/0 conn(0x7feacc15e160 msgr2=0x7feacc0d1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:12.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.832+0000 7feac3e00640 1 --2- 172.21.15.47:0/2482767879 >> v2:172.21.15.116:3300/0 conn(0x7feacc15e160 0x7feacc0d1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.832+0000 7feac3e00640 1 -- 172.21.15.47:0/2482767879 >> v1:172.21.15.116:6789/0 conn(0x7feacc06e010 legacy=0x7feacc06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:12.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.832+0000 7feac3e00640 1 -- 172.21.15.47:0/2482767879 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc06f6b0 con 0x7feacc06f220 2024-02-19T15:26:12.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.836+0000 7feac3e00640 1 --2- 172.21.15.47:0/2482767879 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feacc06f220 0x7feacc06dad0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7feacc074a90 tx=0x7feab401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad449a4c30ed0ff server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:12.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.836+0000 7feacbe00640 1 -- 172.21.15.47:0/2482767879 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab40aa660 con 0x7feacc06f220 2024-02-19T15:26:12.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.836+0000 7feacbe00640 1 -- 172.21.15.47:0/2482767879 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab40aa7c0 con 0x7feacc06f220 2024-02-19T15:26:12.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.836+0000 7feacbe00640 1 -- 172.21.15.47:0/2482767879 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab40aab50 con 0x7feacc06f220 2024-02-19T15:26:12.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.836+0000 7fead3600640 1 -- 172.21.15.47:0/2482767879 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feacc06f220 msgr2=0x7feacc06dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:12.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.836+0000 7fead3600640 1 --2- 172.21.15.47:0/2482767879 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feacc06f220 0x7feacc06dad0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7feacc074a90 tx=0x7feab401c500 comp rx=0 tx=0).stop 2024-02-19T15:26:12.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.836+0000 7fead3600640 1 -- 172.21.15.47:0/2482767879 shutdown_connections 2024-02-19T15:26:12.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.836+0000 7fead3600640 1 --2- 172.21.15.47:0/2482767879 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feacc06f220 0x7feacc06dad0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.836+0000 7fead3600640 1 --2- 172.21.15.47:0/2482767879 >> v2:172.21.15.116:3300/0 conn(0x7feacc15e160 0x7feacc0d1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.836+0000 7fead3600640 1 -- 172.21.15.47:0/2482767879 >> 172.21.15.47:0/2482767879 conn(0x7feacc0d0d30 msgr2=0x7feacc0d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:12.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.836+0000 7fead3600640 1 -- 172.21.15.47:0/2482767879 shutdown_connections 2024-02-19T15:26:12.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.840+0000 7fead3600640 1 -- 172.21.15.47:0/2482767879 wait complete. 2024-02-19T15:26:12.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.840+0000 7fead3600640 1 Processor -- start 2024-02-19T15:26:12.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.840+0000 7fead3600640 1 -- start start 2024-02-19T15:26:12.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.840+0000 7fead3600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feacc06e010 0x7feacc0b0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:12.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.840+0000 7fead3600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feacc06f220 0x7feacc157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:12.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.840+0000 7fead3600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feacc15e160 0x7feacc155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:12.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.840+0000 7fead3600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc155970 con 0x7feacc15e160 2024-02-19T15:26:12.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.840+0000 7fead3600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc155340 con 0x7feacc06e010 2024-02-19T15:26:12.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.840+0000 7fead3600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feacc0b3290 con 0x7feacc06f220 2024-02-19T15:26:12.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.840+0000 7fead1600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feacc15e160 0x7feacc155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:12.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.844+0000 7fead1600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feacc15e160 0x7feacc155e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50898/0 (socket says 172.21.15.47:50898) 2024-02-19T15:26:12.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.844+0000 7fead1600640 1 -- 172.21.15.47:0/1842620472 learned_addr learned my addr 172.21.15.47:0/1842620472 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:12.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.844+0000 7fead0c00640 1 --2- 172.21.15.47:0/1842620472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feacc06e010 0x7feacc0b0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:12.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.844+0000 7feac3e00640 1 --2- 172.21.15.47:0/1842620472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feacc06f220 0x7feacc157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:12.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.844+0000 7fead1600640 1 -- 172.21.15.47:0/1842620472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feacc06f220 msgr2=0x7feacc157c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:12.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.844+0000 7fead1600640 1 --2- 172.21.15.47:0/1842620472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feacc06f220 0x7feacc157c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.844+0000 7fead1600640 1 -- 172.21.15.47:0/1842620472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feacc06e010 msgr2=0x7feacc0b0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:12.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.844+0000 7fead1600640 1 --2- 172.21.15.47:0/1842620472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feacc06e010 0x7feacc0b0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:12.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.844+0000 7fead1600640 1 -- 172.21.15.47:0/1842620472 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab40094d0 con 0x7feacc15e160 2024-02-19T15:26:12.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.844+0000 7fead1600640 1 --2- 172.21.15.47:0/1842620472 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feacc15e160 0x7feacc155e00 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7feac400c770 tx=0x7feac400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:12.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.844+0000 7feac9600640 1 -- 172.21.15.47:0/1842620472 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac4003b80 con 0x7feacc15e160 2024-02-19T15:26:12.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.844+0000 7feac9600640 1 -- 172.21.15.47:0/1842620472 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac4003ce0 con 0x7feacc15e160 2024-02-19T15:26:12.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.844+0000 7fead3600640 1 -- 172.21.15.47:0/1842620472 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feacc0b3570 con 0x7feacc15e160 2024-02-19T15:26:12.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.844+0000 7feac9600640 1 -- 172.21.15.47:0/1842620472 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac400aa90 con 0x7feacc15e160 2024-02-19T15:26:12.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.844+0000 7fead3600640 1 -- 172.21.15.47:0/1842620472 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feacc0b0c70 con 0x7feacc15e160 2024-02-19T15:26:12.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.844+0000 7fead3600640 1 -- 172.21.15.47:0/1842620472 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feacc155ae0 con 0x7feacc15e160 2024-02-19T15:26:12.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.844+0000 7feac9600640 1 -- 172.21.15.47:0/1842620472 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7feac400abf0 con 0x7feacc15e160 2024-02-19T15:26:12.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.848+0000 7feac9600640 1 --2- 172.21.15.47:0/1842620472 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feaa8066e40 0x7feaa8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:12.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.848+0000 7feac9600640 1 -- 172.21.15.47:0/1842620472 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feac4084760 con 0x7feacc15e160 2024-02-19T15:26:12.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.848+0000 7fead0c00640 1 --2- 172.21.15.47:0/1842620472 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feaa8066e40 0x7feaa8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:12.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.848+0000 7fead0c00640 1 --2- 172.21.15.47:0/1842620472 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feaa8066e40 0x7feaa8069300 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7feabc0079c0 tx=0x7feabc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:12.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:12.852+0000 7feac9600640 1 -- 172.21.15.47:0/1842620472 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7feac4051600 con 0x7feacc15e160 2024-02-19T15:26:13.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.012+0000 7fead3600640 1 -- 172.21.15.47:0/1842620472 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7feacc075360 con 0x7feacc15e160 2024-02-19T15:26:13.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.016+0000 7feac9600640 1 -- 172.21.15.47:0/1842620472 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7feac4054cc0 con 0x7feacc15e160 2024-02-19T15:26:13.019 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:26:13.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.016+0000 7fead3600640 1 -- 172.21.15.47:0/1842620472 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feaa8066e40 msgr2=0x7feaa8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:13.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.016+0000 7fead3600640 1 --2- 172.21.15.47:0/1842620472 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feaa8066e40 0x7feaa8069300 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7feabc0079c0 tx=0x7feabc008040 comp rx=0 tx=0).stop 2024-02-19T15:26:13.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.016+0000 7fead3600640 1 -- 172.21.15.47:0/1842620472 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feacc15e160 msgr2=0x7feacc155e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:13.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.016+0000 7fead3600640 1 --2- 172.21.15.47:0/1842620472 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feacc15e160 0x7feacc155e00 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7feac400c770 tx=0x7feac400cc30 comp rx=0 tx=0).stop 2024-02-19T15:26:13.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.016+0000 7fead3600640 1 -- 172.21.15.47:0/1842620472 shutdown_connections 2024-02-19T15:26:13.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.016+0000 7fead3600640 1 --2- 172.21.15.47:0/1842620472 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feaa8066e40 0x7feaa8069300 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.016+0000 7fead3600640 1 --2- 172.21.15.47:0/1842620472 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feacc15e160 0x7feacc155e00 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.016+0000 7fead3600640 1 --2- 172.21.15.47:0/1842620472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feacc06f220 0x7feacc157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.016+0000 7fead3600640 1 --2- 172.21.15.47:0/1842620472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feacc06e010 0x7feacc0b0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.016+0000 7fead3600640 1 -- 172.21.15.47:0/1842620472 >> 172.21.15.47:0/1842620472 conn(0x7feacc0d0d30 msgr2=0x7feacc06aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:13.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.016+0000 7fead3600640 1 -- 172.21.15.47:0/1842620472 shutdown_connections 2024-02-19T15:26:13.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.016+0000 7fead3600640 1 -- 172.21.15.47:0/1842620472 wait complete. 2024-02-19T15:26:13.041 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:26:13.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.176+0000 7f309ac00640 1 Processor -- start 2024-02-19T15:26:13.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.176+0000 7f309ac00640 1 -- start start 2024-02-19T15:26:13.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.180+0000 7f309ac00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f309415bea0 0x7f30940cc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:13.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.180+0000 7f309ac00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3094072610 0x7f30940b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:13.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.180+0000 7f309ac00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f309406dd60 con 0x7f309406aeb0 2024-02-19T15:26:13.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.180+0000 7f309ac00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30940cd340 con 0x7f3094072610 2024-02-19T15:26:13.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.180+0000 7f309ac00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f309415c280 con 0x7f309415bea0 2024-02-19T15:26:13.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.180+0000 7f308b400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3094072610 0x7f30940b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:13.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.180+0000 7f3098c00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f309406aeb0 0x7f309406e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54934/0 (socket says 172.21.15.47:54934) 2024-02-19T15:26:13.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.180+0000 7f3098c00640 1 -- 172.21.15.47:0/610147433 learned_addr learned my addr 172.21.15.47:0/610147433 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:13.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.180+0000 7f3093e00640 1 --2- 172.21.15.47:0/610147433 >> v2:172.21.15.116:3300/0 conn(0x7f309415bea0 0x7f30940cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:13.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.180+0000 7f308b400640 1 -- 172.21.15.47:0/610147433 >> v2:172.21.15.116:3300/0 conn(0x7f309415bea0 msgr2=0x7f30940cc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:13.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.180+0000 7f308b400640 1 --2- 172.21.15.47:0/610147433 >> v2:172.21.15.116:3300/0 conn(0x7f309415bea0 0x7f30940cc780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.180+0000 7f308b400640 1 -- 172.21.15.47:0/610147433 >> v1:172.21.15.47:6789/0 conn(0x7f309406aeb0 legacy=0x7f309406e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:13.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.180+0000 7f308b400640 1 -- 172.21.15.47:0/610147433 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30940ccd80 con 0x7f3094072610 2024-02-19T15:26:13.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.184+0000 7f308b400640 1 --2- 172.21.15.47:0/610147433 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3094072610 0x7f30940b38e0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f30940ce2e0 tx=0x7f307c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6498502a850e9b37 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:13.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.184+0000 7f3093400640 1 -- 172.21.15.47:0/610147433 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307c0ad660 con 0x7f3094072610 2024-02-19T15:26:13.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.184+0000 7f3093400640 1 -- 172.21.15.47:0/610147433 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307c0ad7c0 con 0x7f3094072610 2024-02-19T15:26:13.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.184+0000 7f3093400640 1 -- 172.21.15.47:0/610147433 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307c0adb50 con 0x7f3094072610 2024-02-19T15:26:13.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.184+0000 7f309ac00640 1 -- 172.21.15.47:0/610147433 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3094072610 msgr2=0x7f30940b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:13.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.184+0000 7f309ac00640 1 --2- 172.21.15.47:0/610147433 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3094072610 0x7f30940b38e0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f30940ce2e0 tx=0x7f307c01c450 comp rx=0 tx=0).stop 2024-02-19T15:26:13.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.184+0000 7f309ac00640 1 -- 172.21.15.47:0/610147433 shutdown_connections 2024-02-19T15:26:13.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.184+0000 7f309ac00640 1 --2- 172.21.15.47:0/610147433 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3094072610 0x7f30940b38e0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.184+0000 7f309ac00640 1 --2- 172.21.15.47:0/610147433 >> v2:172.21.15.116:3300/0 conn(0x7f309415bea0 0x7f30940cc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.184+0000 7f309ac00640 1 -- 172.21.15.47:0/610147433 >> 172.21.15.47:0/610147433 conn(0x7f30940cd510 msgr2=0x7f30940cd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:13.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.184+0000 7f309ac00640 1 -- 172.21.15.47:0/610147433 shutdown_connections 2024-02-19T15:26:13.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.184+0000 7f309ac00640 1 -- 172.21.15.47:0/610147433 wait complete. 2024-02-19T15:26:13.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.184+0000 7f309ac00640 1 Processor -- start 2024-02-19T15:26:13.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.184+0000 7f309ac00640 1 -- start start 2024-02-19T15:26:13.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f309ac00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f309406aeb0 0x7f30940c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:13.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f309ac00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3094072610 0x7f3094151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:13.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f309ac00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f309415bea0 0x7f3094154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:13.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f309ac00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30940c60d0 con 0x7f309406aeb0 2024-02-19T15:26:13.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f309ac00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30940c4930 con 0x7f309415bea0 2024-02-19T15:26:13.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f309ac00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30940c5da0 con 0x7f3094072610 2024-02-19T15:26:13.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f3093e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f309406aeb0 0x7f30940c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:13.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f3093e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f309406aeb0 0x7f30940c43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50916/0 (socket says 172.21.15.47:50916) 2024-02-19T15:26:13.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f3093e00640 1 -- 172.21.15.47:0/208162446 learned_addr learned my addr 172.21.15.47:0/208162446 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:13.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f308b400640 1 --2- 172.21.15.47:0/208162446 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3094072610 0x7f3094151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:13.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f3098c00640 1 --2- 172.21.15.47:0/208162446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f309415bea0 0x7f3094154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:13.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f3093e00640 1 -- 172.21.15.47:0/208162446 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3094072610 msgr2=0x7f3094151cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:13.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f3093e00640 1 --2- 172.21.15.47:0/208162446 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3094072610 0x7f3094151cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f3093e00640 1 -- 172.21.15.47:0/208162446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f309415bea0 msgr2=0x7f3094154880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:13.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f3093e00640 1 --2- 172.21.15.47:0/208162446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f309415bea0 0x7f3094154880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f3093e00640 1 -- 172.21.15.47:0/208162446 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307c009420 con 0x7f309406aeb0 2024-02-19T15:26:13.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f3093e00640 1 --2- 172.21.15.47:0/208162446 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f309406aeb0 0x7f30940c43f0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f308400d5f0 tx=0x7f308400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:13.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f3090c00640 1 -- 172.21.15.47:0/208162446 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30840125f0 con 0x7f309406aeb0 2024-02-19T15:26:13.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f3090c00640 1 -- 172.21.15.47:0/208162446 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3084012750 con 0x7f309406aeb0 2024-02-19T15:26:13.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f309ac00640 1 -- 172.21.15.47:0/208162446 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f309406b850 con 0x7f309406aeb0 2024-02-19T15:26:13.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f3090c00640 1 -- 172.21.15.47:0/208162446 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3084012ae0 con 0x7f309406aeb0 2024-02-19T15:26:13.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.188+0000 7f309ac00640 1 -- 172.21.15.47:0/208162446 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3094153430 con 0x7f309406aeb0 2024-02-19T15:26:13.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.192+0000 7f309ac00640 1 -- 172.21.15.47:0/208162446 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3094068500 con 0x7f309406aeb0 2024-02-19T15:26:13.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.192+0000 7f3090c00640 1 -- 172.21.15.47:0/208162446 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3084012c40 con 0x7f309406aeb0 2024-02-19T15:26:13.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.192+0000 7f3090c00640 1 --2- 172.21.15.47:0/208162446 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3070066e40 0x7f3070069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:13.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.192+0000 7f3090c00640 1 -- 172.21.15.47:0/208162446 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3084084f20 con 0x7f309406aeb0 2024-02-19T15:26:13.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.192+0000 7f308b400640 1 --2- 172.21.15.47:0/208162446 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3070066e40 0x7f3070069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:13.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.192+0000 7f308b400640 1 --2- 172.21.15.47:0/208162446 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3070066e40 0x7f3070069300 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f307c0af000 tx=0x7f307c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:13.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.200+0000 7f3090c00640 1 -- 172.21.15.47:0/208162446 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f3084051c70 con 0x7f309406aeb0 2024-02-19T15:26:13.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.400+0000 7f309ac00640 1 -- 172.21.15.47:0/208162446 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3094075360 con 0x7f309406aeb0 2024-02-19T15:26:13.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.400+0000 7f3090c00640 1 -- 172.21.15.47:0/208162446 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4943 (secure 0 0 0) 0x7f3084007380 con 0x7f309406aeb0 2024-02-19T15:26:13.404 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:13.404 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5954,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1402677843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1402677843},{"type":"v1","addr":"172.21.15.116:6836","nonce":1402677843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":5957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1507554634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":1507554634},{"type":"v1","addr":"172.21.15.116:6837","nonce":1507554634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6771,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/888455604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":888455604},{"type":"v1","addr":"172.21.15.47:6833","nonce":888455604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:25:59.522224+0000","modified":"2024-02-19T15:26:02.535882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6774":{"gid":6774,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3598443511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3598443511},{"type":"v1","addr":"172.21.15.47:6835","nonce":3598443511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-19T15:26:13.404 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 49 2024-02-19T15:26:13.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.400+0000 7f309ac00640 1 -- 172.21.15.47:0/208162446 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3070066e40 msgr2=0x7f3070069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:13.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.400+0000 7f309ac00640 1 --2- 172.21.15.47:0/208162446 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3070066e40 0x7f3070069300 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f307c0af000 tx=0x7f307c008040 comp rx=0 tx=0).stop 2024-02-19T15:26:13.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.400+0000 7f309ac00640 1 -- 172.21.15.47:0/208162446 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f309406aeb0 msgr2=0x7f30940c43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:13.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.400+0000 7f309ac00640 1 --2- 172.21.15.47:0/208162446 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f309406aeb0 0x7f30940c43f0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f308400d5f0 tx=0x7f308400dab0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.400+0000 7f309ac00640 1 -- 172.21.15.47:0/208162446 shutdown_connections 2024-02-19T15:26:13.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.400+0000 7f309ac00640 1 --2- 172.21.15.47:0/208162446 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3070066e40 0x7f3070069300 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.400+0000 7f309ac00640 1 --2- 172.21.15.47:0/208162446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f309415bea0 0x7f3094154880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.400+0000 7f309ac00640 1 --2- 172.21.15.47:0/208162446 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3094072610 0x7f3094151cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.400+0000 7f309ac00640 1 --2- 172.21.15.47:0/208162446 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f309406aeb0 0x7f30940c43f0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.400+0000 7f309ac00640 1 -- 172.21.15.47:0/208162446 >> 172.21.15.47:0/208162446 conn(0x7f30940cd510 msgr2=0x7f30940cd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:13.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.400+0000 7f309ac00640 1 -- 172.21.15.47:0/208162446 shutdown_connections 2024-02-19T15:26:13.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.400+0000 7f309ac00640 1 -- 172.21.15.47:0/208162446 wait complete. 2024-02-19T15:26:13.425 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:26:13.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.560+0000 7f7227400640 1 Processor -- start 2024-02-19T15:26:13.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.560+0000 7f7227400640 1 -- start start 2024-02-19T15:26:13.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.560+0000 7f7227400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f72200cea80 0x7f72200727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:13.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.560+0000 7f7227400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f722015d280 con 0x7f722015c390 2024-02-19T15:26:13.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.560+0000 7f7227400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7220070ff0 con 0x7f7220145590 2024-02-19T15:26:13.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.560+0000 7f7227400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72200709e0 con 0x7f72200cea80 2024-02-19T15:26:13.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.564+0000 7f7225400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f722015c390 0x7f722015c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37692/0 (socket says 172.21.15.47:37692) 2024-02-19T15:26:13.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.564+0000 7f7225400640 1 -- 172.21.15.47:0/1827227297 learned_addr learned my addr 172.21.15.47:0/1827227297 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:13.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.564+0000 7f721fe00640 1 --2- 172.21.15.47:0/1827227297 >> v2:172.21.15.116:3300/0 conn(0x7f72200cea80 0x7f72200727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:13.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.564+0000 7f721f400640 1 -- 172.21.15.47:0/1827227297 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230906168 0 0) 0x7f722015d280 con 0x7f722015c390 2024-02-19T15:26:13.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.564+0000 7f721f400640 1 -- 172.21.15.47:0/1827227297 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7208003530 con 0x7f722015c390 2024-02-19T15:26:13.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.564+0000 7f721f400640 1 -- 172.21.15.47:0/1827227297 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255895924 0 0) 0x7f7220070ff0 con 0x7f7220145590 2024-02-19T15:26:13.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.564+0000 7f721f400640 1 -- 172.21.15.47:0/1827227297 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f722015d280 con 0x7f7220145590 2024-02-19T15:26:13.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.564+0000 7f721fe00640 1 -- 172.21.15.47:0/1827227297 >> v1:172.21.15.116:6789/0 conn(0x7f7220145590 legacy=0x7f722006c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:13.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.564+0000 7f721fe00640 1 -- 172.21.15.47:0/1827227297 >> v1:172.21.15.47:6789/0 conn(0x7f722015c390 legacy=0x7f722015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:13.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.564+0000 7f721fe00640 1 -- 172.21.15.47:0/1827227297 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72200cf020 con 0x7f72200cea80 2024-02-19T15:26:13.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.564+0000 7f721fe00640 1 --2- 172.21.15.47:0/1827227297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72200cea80 0x7f72200727b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f720c0097c0 tx=0x7f720c01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b656710af0e9034 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:13.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.568+0000 7f721f400640 1 -- 172.21.15.47:0/1827227297 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c0ac410 con 0x7f72200cea80 2024-02-19T15:26:13.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.568+0000 7f721f400640 1 -- 172.21.15.47:0/1827227297 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f720c0ac570 con 0x7f72200cea80 2024-02-19T15:26:13.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.568+0000 7f721f400640 1 -- 172.21.15.47:0/1827227297 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c0ac900 con 0x7f72200cea80 2024-02-19T15:26:13.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.568+0000 7f7227400640 1 -- 172.21.15.47:0/1827227297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72200cea80 msgr2=0x7f72200727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:13.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.568+0000 7f7227400640 1 --2- 172.21.15.47:0/1827227297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72200cea80 0x7f72200727b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f720c0097c0 tx=0x7f720c01c4a0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.568+0000 7f7227400640 1 -- 172.21.15.47:0/1827227297 shutdown_connections 2024-02-19T15:26:13.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.568+0000 7f7227400640 1 --2- 172.21.15.47:0/1827227297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72200cea80 0x7f72200727b0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.568+0000 7f7227400640 1 -- 172.21.15.47:0/1827227297 >> 172.21.15.47:0/1827227297 conn(0x7f72200b16a0 msgr2=0x7f72200b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:13.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.568+0000 7f7227400640 1 -- 172.21.15.47:0/1827227297 shutdown_connections 2024-02-19T15:26:13.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.568+0000 7f7227400640 1 -- 172.21.15.47:0/1827227297 wait complete. 2024-02-19T15:26:13.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.568+0000 7f7227400640 1 Processor -- start 2024-02-19T15:26:13.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.568+0000 7f7227400640 1 -- start start 2024-02-19T15:26:13.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f7227400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72200cea80 0x7f72201351e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:13.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f7227400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7220145590 0x7f722015bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:13.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f7227400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f722015c390 0x7f72200b4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:13.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f7227400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72200b3340 con 0x7f722015c390 2024-02-19T15:26:13.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f7227400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722015c1d0 con 0x7f7220145590 2024-02-19T15:26:13.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f7227400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72200b2730 con 0x7f72200cea80 2024-02-19T15:26:13.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f7225400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f722015c390 0x7f72200b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:13.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f7225400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f722015c390 0x7f72200b4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59180/0 (socket says 172.21.15.47:59180) 2024-02-19T15:26:13.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f7225400640 1 -- 172.21.15.47:0/1782215204 learned_addr learned my addr 172.21.15.47:0/1782215204 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:13.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f7224a00640 1 --2- 172.21.15.47:0/1782215204 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72200cea80 0x7f72201351e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:13.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f721fe00640 1 --2- 172.21.15.47:0/1782215204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7220145590 0x7f722015bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:13.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f7225400640 1 -- 172.21.15.47:0/1782215204 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72200cea80 msgr2=0x7f72201351e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:13.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f7225400640 1 --2- 172.21.15.47:0/1782215204 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72200cea80 0x7f72201351e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f7225400640 1 -- 172.21.15.47:0/1782215204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7220145590 msgr2=0x7f722015bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:13.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f7225400640 1 --2- 172.21.15.47:0/1782215204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7220145590 0x7f722015bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f7225400640 1 -- 172.21.15.47:0/1782215204 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f720c009420 con 0x7f722015c390 2024-02-19T15:26:13.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f7225400640 1 --2- 172.21.15.47:0/1782215204 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f722015c390 0x7f72200b4270 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f721800caa0 tx=0x7f721800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:13.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f721cc00640 1 -- 172.21.15.47:0/1782215204 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721800ac00 con 0x7f722015c390 2024-02-19T15:26:13.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f721cc00640 1 -- 172.21.15.47:0/1782215204 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f721800ad80 con 0x7f722015c390 2024-02-19T15:26:13.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f721cc00640 1 -- 172.21.15.47:0/1782215204 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7218012640 con 0x7f722015c390 2024-02-19T15:26:13.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f7227400640 1 -- 172.21.15.47:0/1782215204 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72200b2a10 con 0x7f722015c390 2024-02-19T15:26:13.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f7227400640 1 -- 172.21.15.47:0/1782215204 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72200b2fd0 con 0x7f722015c390 2024-02-19T15:26:13.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.572+0000 7f7227400640 1 -- 172.21.15.47:0/1782215204 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72200b34b0 con 0x7f722015c390 2024-02-19T15:26:13.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.576+0000 7f721cc00640 1 -- 172.21.15.47:0/1782215204 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7218007070 con 0x7f722015c390 2024-02-19T15:26:13.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.576+0000 7f721cc00640 1 --2- 172.21.15.47:0/1782215204 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7200066e70 0x7f7200069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:13.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.576+0000 7f721cc00640 1 -- 172.21.15.47:0/1782215204 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7218084980 con 0x7f722015c390 2024-02-19T15:26:13.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.576+0000 7f7224a00640 1 --2- 172.21.15.47:0/1782215204 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7200066e70 0x7f7200069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:13.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.576+0000 7f7224a00640 1 --2- 172.21.15.47:0/1782215204 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7200066e70 0x7f7200069330 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f72140017c0 tx=0x7f7214009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:13.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.580+0000 7f721cc00640 1 -- 172.21.15.47:0/1782215204 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7218051780 con 0x7f722015c390 2024-02-19T15:26:13.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.740+0000 7f7227400640 1 -- 172.21.15.47:0/1782215204 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7220075360 con 0x7f722015c390 2024-02-19T15:26:13.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.744+0000 7f721cc00640 1 -- 172.21.15.47:0/1782215204 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16672 (secure 0 0 0) 0x7f7218017090 con 0x7f722015c390 2024-02-19T15:26:13.747 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:13.747 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:26:01.137113+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:25:57.938354+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-02-19T15:25:58.514866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:26:13.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.744+0000 7f7227400640 1 -- 172.21.15.47:0/1782215204 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7200066e70 msgr2=0x7f7200069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:13.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.744+0000 7f7227400640 1 --2- 172.21.15.47:0/1782215204 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7200066e70 0x7f7200069330 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f72140017c0 tx=0x7f7214009040 comp rx=0 tx=0).stop 2024-02-19T15:26:13.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.744+0000 7f7227400640 1 -- 172.21.15.47:0/1782215204 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f722015c390 msgr2=0x7f72200b4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:13.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.744+0000 7f7227400640 1 --2- 172.21.15.47:0/1782215204 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f722015c390 0x7f72200b4270 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f721800caa0 tx=0x7f721800cf60 comp rx=0 tx=0).stop 2024-02-19T15:26:13.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.744+0000 7f7227400640 1 -- 172.21.15.47:0/1782215204 shutdown_connections 2024-02-19T15:26:13.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.744+0000 7f7227400640 1 --2- 172.21.15.47:0/1782215204 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7200066e70 0x7f7200069330 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.744+0000 7f7227400640 1 --2- 172.21.15.47:0/1782215204 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f722015c390 0x7f72200b4270 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.744+0000 7f7227400640 1 --2- 172.21.15.47:0/1782215204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7220145590 0x7f722015bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.744+0000 7f7227400640 1 --2- 172.21.15.47:0/1782215204 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72200cea80 0x7f72201351e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.744+0000 7f7227400640 1 -- 172.21.15.47:0/1782215204 >> 172.21.15.47:0/1782215204 conn(0x7f72200b16a0 msgr2=0x7f722013f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:13.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.744+0000 7f7227400640 1 -- 172.21.15.47:0/1782215204 shutdown_connections 2024-02-19T15:26:13.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.744+0000 7f7227400640 1 -- 172.21.15.47:0/1782215204 wait complete. 2024-02-19T15:26:13.777 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T15:26:13.777 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:26:13.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.912+0000 7f962e000640 1 Processor -- start 2024-02-19T15:26:13.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.912+0000 7f962e000640 1 -- start start 2024-02-19T15:26:13.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.912+0000 7f962e000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f962815bcd0 0x7f96280cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:13.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.912+0000 7f962e000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96280cedb0 0x7f96280cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:13.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.912+0000 7f962e000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96280cc700 con 0x7f96280d2710 2024-02-19T15:26:13.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.912+0000 7f962e000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96280cf5e0 con 0x7f962815bcd0 2024-02-19T15:26:13.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.912+0000 7f962e000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96280cf300 con 0x7f96280cedb0 2024-02-19T15:26:13.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.912+0000 7f9627400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f962815bcd0 0x7f96280cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:13.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.912+0000 7f961ea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96280cedb0 0x7f96280cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:13.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.912+0000 7f9627400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f962815bcd0 0x7f96280cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:38012/0 (socket says 172.21.15.47:38012) 2024-02-19T15:26:13.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.912+0000 7f9627400640 1 -- 172.21.15.47:0/4106616436 learned_addr learned my addr 172.21.15.47:0/4106616436 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:13.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.916+0000 7f961ea00640 1 -- 172.21.15.47:0/4106616436 >> v2:172.21.15.116:3300/0 conn(0x7f962815bcd0 msgr2=0x7f96280cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:13.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.916+0000 7f961ea00640 1 --2- 172.21.15.47:0/4106616436 >> v2:172.21.15.116:3300/0 conn(0x7f962815bcd0 0x7f96280cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.916+0000 7f961ea00640 1 -- 172.21.15.47:0/4106616436 >> v1:172.21.15.116:6789/0 conn(0x7f96280d2710 legacy=0x7f962815d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:13.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.916+0000 7f961ea00640 1 -- 172.21.15.47:0/4106616436 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96280cc9a0 con 0x7f96280cedb0 2024-02-19T15:26:13.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.916+0000 7f9627400640 1 --2- 172.21.15.47:0/4106616436 >> v2:172.21.15.116:3300/0 conn(0x7f962815bcd0 0x7f96280cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:13.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.916+0000 7f9626a00640 1 -- 172.21.15.47:0/4106616436 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425753093 0 0) 0x7f96280cc700 con 0x7f96280d2710 2024-02-19T15:26:13.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.916+0000 7f961ea00640 1 --2- 172.21.15.47:0/4106616436 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96280cedb0 0x7f96280cfc50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f96100097c0 tx=0x7f961001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=440582567b4b3ce0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:13.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.916+0000 7f9626a00640 1 -- 172.21.15.47:0/4106616436 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96100ad680 con 0x7f96280cedb0 2024-02-19T15:26:13.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.916+0000 7f9626a00640 1 -- 172.21.15.47:0/4106616436 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96100ad7e0 con 0x7f96280cedb0 2024-02-19T15:26:13.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.916+0000 7f9626a00640 1 -- 172.21.15.47:0/4106616436 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96100adb70 con 0x7f96280cedb0 2024-02-19T15:26:13.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f962e000640 1 -- 172.21.15.47:0/4106616436 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96280cedb0 msgr2=0x7f96280cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:13.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f962e000640 1 --2- 172.21.15.47:0/4106616436 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96280cedb0 0x7f96280cfc50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f96100097c0 tx=0x7f961001c470 comp rx=0 tx=0).stop 2024-02-19T15:26:13.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f962e000640 1 -- 172.21.15.47:0/4106616436 shutdown_connections 2024-02-19T15:26:13.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f962e000640 1 --2- 172.21.15.47:0/4106616436 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96280cedb0 0x7f96280cfc50 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f962e000640 1 --2- 172.21.15.47:0/4106616436 >> v2:172.21.15.116:3300/0 conn(0x7f962815bcd0 0x7f96280cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f962e000640 1 -- 172.21.15.47:0/4106616436 >> 172.21.15.47:0/4106616436 conn(0x7f96280d1bf0 msgr2=0x7f96280d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:13.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f962e000640 1 -- 172.21.15.47:0/4106616436 shutdown_connections 2024-02-19T15:26:13.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f962e000640 1 -- 172.21.15.47:0/4106616436 wait complete. 2024-02-19T15:26:13.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f962e000640 1 Processor -- start 2024-02-19T15:26:13.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f962e000640 1 -- start start 2024-02-19T15:26:13.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f962e000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96280cedb0 0x7f96280cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:13.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f962e000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96280d2710 0x7f96280c5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:13.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f962e000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f962815bcd0 0x7f96281f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:13.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f962e000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96281f72a0 con 0x7f962815bcd0 2024-02-19T15:26:13.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f962e000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96281f7540 con 0x7f96280cedb0 2024-02-19T15:26:13.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f962e000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96281f7820 con 0x7f96280d2710 2024-02-19T15:26:13.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f9627400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96280cedb0 0x7f96280cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:13.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f9627e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f962815bcd0 0x7f96281f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:13.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f961ea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96280d2710 0x7f96280c5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:13.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f9627400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96280cedb0 0x7f96280cbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:38018/0 (socket says 172.21.15.47:38018) 2024-02-19T15:26:13.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f9627400640 1 -- 172.21.15.47:0/2026237805 learned_addr learned my addr 172.21.15.47:0/2026237805 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:13.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f9627400640 1 -- 172.21.15.47:0/2026237805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96280d2710 msgr2=0x7f96280c5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:13.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f9627400640 1 --2- 172.21.15.47:0/2026237805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96280d2710 0x7f96280c5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f9627400640 1 -- 172.21.15.47:0/2026237805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f962815bcd0 msgr2=0x7f96281f6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:13.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f9627400640 1 --2- 172.21.15.47:0/2026237805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f962815bcd0 0x7f96281f6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:13.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f9627400640 1 -- 172.21.15.47:0/2026237805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96180094d0 con 0x7f96280cedb0 2024-02-19T15:26:13.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.920+0000 7f961ea00640 1 --2- 172.21.15.47:0/2026237805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96280d2710 0x7f96280c5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:13.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.924+0000 7f9627400640 1 --2- 172.21.15.47:0/2026237805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96280cedb0 0x7f96280cbb70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f961800e9d0 tx=0x7f961800ee90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:13.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.924+0000 7f961fe00640 1 -- 172.21.15.47:0/2026237805 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961800ca60 con 0x7f96280cedb0 2024-02-19T15:26:13.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.924+0000 7f961fe00640 1 -- 172.21.15.47:0/2026237805 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961800cbc0 con 0x7f96280cedb0 2024-02-19T15:26:13.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.924+0000 7f961fe00640 1 -- 172.21.15.47:0/2026237805 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618013470 con 0x7f96280cedb0 2024-02-19T15:26:13.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.924+0000 7f962e000640 1 -- 172.21.15.47:0/2026237805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9610009420 con 0x7f96280cedb0 2024-02-19T15:26:13.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.924+0000 7f962e000640 1 -- 172.21.15.47:0/2026237805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96280cdb40 con 0x7f96280cedb0 2024-02-19T15:26:13.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.924+0000 7f962e000640 1 -- 172.21.15.47:0/2026237805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96281f7d40 con 0x7f96280cedb0 2024-02-19T15:26:13.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.924+0000 7f961fe00640 1 -- 172.21.15.47:0/2026237805 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f96180059a0 con 0x7f96280cedb0 2024-02-19T15:26:13.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.924+0000 7f961fe00640 1 --2- 172.21.15.47:0/2026237805 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9604066e40 0x7f9604069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:13.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.924+0000 7f961fe00640 1 -- 172.21.15.47:0/2026237805 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9618084cb0 con 0x7f96280cedb0 2024-02-19T15:26:13.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.924+0000 7f961ea00640 1 --2- 172.21.15.47:0/2026237805 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9604066e40 0x7f9604069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:13.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.928+0000 7f961ea00640 1 --2- 172.21.15.47:0/2026237805 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9604066e40 0x7f9604069300 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f96100af000 tx=0x7f96100a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:13.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:13.928+0000 7f961fe00640 1 -- 172.21.15.47:0/2026237805 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f9618051bc0 con 0x7f96280cedb0 2024-02-19T15:26:14.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.124+0000 7f962e000640 1 -- 172.21.15.47:0/2026237805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9628075360 con 0x7f96280cedb0 2024-02-19T15:26:14.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.124+0000 7f961fe00640 1 -- 172.21.15.47:0/2026237805 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4943 (secure 0 0 0) 0x7f9618021020 con 0x7f96280cedb0 2024-02-19T15:26:14.129 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:14.129 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5954,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1402677843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1402677843},{"type":"v1","addr":"172.21.15.116:6836","nonce":1402677843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":5957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1507554634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":1507554634},{"type":"v1","addr":"172.21.15.116:6837","nonce":1507554634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6771,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/888455604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":888455604},{"type":"v1","addr":"172.21.15.47:6833","nonce":888455604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:25:59.522224+0000","modified":"2024-02-19T15:26:02.535882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6774":{"gid":6774,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3598443511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3598443511},{"type":"v1","addr":"172.21.15.47:6835","nonce":3598443511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-19T15:26:14.129 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 49 2024-02-19T15:26:14.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.128+0000 7f962e000640 1 -- 172.21.15.47:0/2026237805 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9604066e40 msgr2=0x7f9604069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:14.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.128+0000 7f962e000640 1 --2- 172.21.15.47:0/2026237805 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9604066e40 0x7f9604069300 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f96100af000 tx=0x7f96100a7000 comp rx=0 tx=0).stop 2024-02-19T15:26:14.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.128+0000 7f962e000640 1 -- 172.21.15.47:0/2026237805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96280cedb0 msgr2=0x7f96280cbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:14.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.128+0000 7f962e000640 1 --2- 172.21.15.47:0/2026237805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96280cedb0 0x7f96280cbb70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f961800e9d0 tx=0x7f961800ee90 comp rx=0 tx=0).stop 2024-02-19T15:26:14.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.128+0000 7f962e000640 1 -- 172.21.15.47:0/2026237805 shutdown_connections 2024-02-19T15:26:14.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.128+0000 7f962e000640 1 --2- 172.21.15.47:0/2026237805 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9604066e40 0x7f9604069300 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.128+0000 7f962e000640 1 --2- 172.21.15.47:0/2026237805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f962815bcd0 0x7f96281f6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.128+0000 7f962e000640 1 --2- 172.21.15.47:0/2026237805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96280d2710 0x7f96280c5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.128+0000 7f962e000640 1 --2- 172.21.15.47:0/2026237805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96280cedb0 0x7f96280cbb70 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.128+0000 7f962e000640 1 -- 172.21.15.47:0/2026237805 >> 172.21.15.47:0/2026237805 conn(0x7f96280d1bf0 msgr2=0x7f96280d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:14.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.128+0000 7f962e000640 1 -- 172.21.15.47:0/2026237805 shutdown_connections 2024-02-19T15:26:14.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.128+0000 7f962e000640 1 -- 172.21.15.47:0/2026237805 wait complete. 2024-02-19T15:26:14.151 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:26:14.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.288+0000 7f2a02e00640 1 Processor -- start 2024-02-19T15:26:14.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.288+0000 7f2a02e00640 1 -- start start 2024-02-19T15:26:14.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.288+0000 7f2a02e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f29fc145590 0x7f29fc0727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:14.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.288+0000 7f2a02e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29fc15d280 con 0x7f29fc15c390 2024-02-19T15:26:14.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.288+0000 7f2a02e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29fc070ff0 con 0x7f29fc0cea80 2024-02-19T15:26:14.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.288+0000 7f2a02e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29fc0709e0 con 0x7f29fc145590 2024-02-19T15:26:14.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.288+0000 7f2a00e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f29fc15c390 0x7f29fc15c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37700/0 (socket says 172.21.15.47:37700) 2024-02-19T15:26:14.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.288+0000 7f2a00e00640 1 -- 172.21.15.47:0/4018955777 learned_addr learned my addr 172.21.15.47:0/4018955777 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:14.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.288+0000 7f29fbe00640 1 --2- 172.21.15.47:0/4018955777 >> v2:172.21.15.116:3300/0 conn(0x7f29fc145590 0x7f29fc0727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:14.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.288+0000 7f29fb400640 1 -- 172.21.15.47:0/4018955777 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660157214 0 0) 0x7f29fc15d280 con 0x7f29fc15c390 2024-02-19T15:26:14.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.288+0000 7f29fb400640 1 -- 172.21.15.47:0/4018955777 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e0003530 con 0x7f29fc15c390 2024-02-19T15:26:14.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.288+0000 7f29fb400640 1 -- 172.21.15.47:0/4018955777 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789773461 0 0) 0x7f29fc070ff0 con 0x7f29fc0cea80 2024-02-19T15:26:14.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.288+0000 7f29fb400640 1 -- 172.21.15.47:0/4018955777 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29fc15d280 con 0x7f29fc0cea80 2024-02-19T15:26:14.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.288+0000 7f29fbe00640 1 -- 172.21.15.47:0/4018955777 >> v1:172.21.15.116:6789/0 conn(0x7f29fc0cea80 legacy=0x7f29fc06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:14.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.288+0000 7f29fbe00640 1 -- 172.21.15.47:0/4018955777 >> v1:172.21.15.47:6789/0 conn(0x7f29fc15c390 legacy=0x7f29fc15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:14.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.288+0000 7f29fbe00640 1 -- 172.21.15.47:0/4018955777 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29fc0cf020 con 0x7f29fc145590 2024-02-19T15:26:14.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.292+0000 7f29fbe00640 1 --2- 172.21.15.47:0/4018955777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29fc145590 0x7f29fc0727b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f29ec0097c0 tx=0x7f29ec01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d706cc2ea288f66 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:14.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.292+0000 7f29fb400640 1 -- 172.21.15.47:0/4018955777 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec0ac410 con 0x7f29fc145590 2024-02-19T15:26:14.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.292+0000 7f29fb400640 1 -- 172.21.15.47:0/4018955777 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ec0ac570 con 0x7f29fc145590 2024-02-19T15:26:14.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.292+0000 7f29fb400640 1 -- 172.21.15.47:0/4018955777 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec0ac900 con 0x7f29fc145590 2024-02-19T15:26:14.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.292+0000 7f2a02e00640 1 -- 172.21.15.47:0/4018955777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29fc145590 msgr2=0x7f29fc0727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:14.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.292+0000 7f2a02e00640 1 --2- 172.21.15.47:0/4018955777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29fc145590 0x7f29fc0727b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f29ec0097c0 tx=0x7f29ec01c4a0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.292+0000 7f2a02e00640 1 -- 172.21.15.47:0/4018955777 shutdown_connections 2024-02-19T15:26:14.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.292+0000 7f2a02e00640 1 --2- 172.21.15.47:0/4018955777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29fc145590 0x7f29fc0727b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.292+0000 7f2a02e00640 1 -- 172.21.15.47:0/4018955777 >> 172.21.15.47:0/4018955777 conn(0x7f29fc0b16a0 msgr2=0x7f29fc0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:14.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.292+0000 7f2a02e00640 1 -- 172.21.15.47:0/4018955777 shutdown_connections 2024-02-19T15:26:14.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.292+0000 7f2a02e00640 1 -- 172.21.15.47:0/4018955777 wait complete. 2024-02-19T15:26:14.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f2a02e00640 1 Processor -- start 2024-02-19T15:26:14.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f2a02e00640 1 -- start start 2024-02-19T15:26:14.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f2a02e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29fc0cea80 0x7f29fc13ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:14.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f2a02e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29fc145590 0x7f29fc15b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:14.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f2a02e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f29fc15c390 0x7f29fc1444b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:14.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f2a02e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29fc15b820 con 0x7f29fc0cea80 2024-02-19T15:26:14.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f2a02e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29fc13eaa0 con 0x7f29fc145590 2024-02-19T15:26:14.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f2a02e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29fc13de90 con 0x7f29fc15c390 2024-02-19T15:26:14.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f29fbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29fc0cea80 0x7f29fc13ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:14.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f29fbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29fc0cea80 0x7f29fc13ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59196/0 (socket says 172.21.15.47:59196) 2024-02-19T15:26:14.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f29fbe00640 1 -- 172.21.15.47:0/3126771778 learned_addr learned my addr 172.21.15.47:0/3126771778 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:14.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f29f3e00640 1 --2- 172.21.15.47:0/3126771778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29fc145590 0x7f29fc15b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:14.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f2a00e00640 1 --2- 172.21.15.47:0/3126771778 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f29fc15c390 0x7f29fc1444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:14.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f29fbe00640 1 -- 172.21.15.47:0/3126771778 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f29fc15c390 msgr2=0x7f29fc1444b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:14.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f29fbe00640 1 --2- 172.21.15.47:0/3126771778 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f29fc15c390 0x7f29fc1444b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f29fbe00640 1 -- 172.21.15.47:0/3126771778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29fc145590 msgr2=0x7f29fc15b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:14.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f29fbe00640 1 --2- 172.21.15.47:0/3126771778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29fc145590 0x7f29fc15b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f29fbe00640 1 -- 172.21.15.47:0/3126771778 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ec009420 con 0x7f29fc0cea80 2024-02-19T15:26:14.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f29fbe00640 1 --2- 172.21.15.47:0/3126771778 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29fc0cea80 0x7f29fc13ef30 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f29ec0097c0 tx=0x7f29ec019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:14.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f29f8c00640 1 -- 172.21.15.47:0/3126771778 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec006e20 con 0x7f29fc0cea80 2024-02-19T15:26:14.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f29f8c00640 1 -- 172.21.15.47:0/3126771778 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ec0b5410 con 0x7f29fc0cea80 2024-02-19T15:26:14.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f29f8c00640 1 -- 172.21.15.47:0/3126771778 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec0b57c0 con 0x7f29fc0cea80 2024-02-19T15:26:14.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f2a02e00640 1 -- 172.21.15.47:0/3126771778 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29fc13e170 con 0x7f29fc0cea80 2024-02-19T15:26:14.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.296+0000 7f2a02e00640 1 -- 172.21.15.47:0/3126771778 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29fc13e730 con 0x7f29fc0cea80 2024-02-19T15:26:14.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.300+0000 7f2a02e00640 1 -- 172.21.15.47:0/3126771778 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29fc0b2610 con 0x7f29fc0cea80 2024-02-19T15:26:14.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.300+0000 7f29f8c00640 1 -- 172.21.15.47:0/3126771778 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f29ec0b3040 con 0x7f29fc0cea80 2024-02-19T15:26:14.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.300+0000 7f29f8c00640 1 --2- 172.21.15.47:0/3126771778 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f29d8066e70 0x7f29d8069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:14.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.300+0000 7f29f8c00640 1 -- 172.21.15.47:0/3126771778 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29ec11e290 con 0x7f29fc0cea80 2024-02-19T15:26:14.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.300+0000 7f29f3e00640 1 --2- 172.21.15.47:0/3126771778 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f29d8066e70 0x7f29d8069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:14.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.300+0000 7f29f3e00640 1 --2- 172.21.15.47:0/3126771778 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f29d8066e70 0x7f29d8069330 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f29fc133cc0 tx=0x7f29e4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:14.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.304+0000 7f29f8c00640 1 -- 172.21.15.47:0/3126771778 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f29ec0eb080 con 0x7f29fc0cea80 2024-02-19T15:26:14.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.500+0000 7f2a02e00640 1 -- 172.21.15.47:0/3126771778 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29fc075360 con 0x7f29fc0cea80 2024-02-19T15:26:14.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.500+0000 7f29f8c00640 1 -- 172.21.15.47:0/3126771778 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4943 (secure 0 0 0) 0x7f29ec0ee740 con 0x7f29fc0cea80 2024-02-19T15:26:14.504 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:14.504 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5954,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1402677843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1402677843},{"type":"v1","addr":"172.21.15.116:6836","nonce":1402677843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":5957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1507554634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":1507554634},{"type":"v1","addr":"172.21.15.116:6837","nonce":1507554634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6771,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/888455604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":888455604},{"type":"v1","addr":"172.21.15.47:6833","nonce":888455604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:25:59.522224+0000","modified":"2024-02-19T15:26:02.535882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6774":{"gid":6774,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/3598443511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3598443511},{"type":"v1","addr":"172.21.15.47:6835","nonce":3598443511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-19T15:26:14.505 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 49 2024-02-19T15:26:14.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.500+0000 7f2a02e00640 1 -- 172.21.15.47:0/3126771778 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f29d8066e70 msgr2=0x7f29d8069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:14.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.500+0000 7f2a02e00640 1 --2- 172.21.15.47:0/3126771778 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f29d8066e70 0x7f29d8069330 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f29fc133cc0 tx=0x7f29e4009040 comp rx=0 tx=0).stop 2024-02-19T15:26:14.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.500+0000 7f2a02e00640 1 -- 172.21.15.47:0/3126771778 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29fc0cea80 msgr2=0x7f29fc13ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:14.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.500+0000 7f2a02e00640 1 --2- 172.21.15.47:0/3126771778 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29fc0cea80 0x7f29fc13ef30 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f29ec0097c0 tx=0x7f29ec019240 comp rx=0 tx=0).stop 2024-02-19T15:26:14.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.504+0000 7f2a02e00640 1 -- 172.21.15.47:0/3126771778 shutdown_connections 2024-02-19T15:26:14.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.504+0000 7f2a02e00640 1 --2- 172.21.15.47:0/3126771778 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f29d8066e70 0x7f29d8069330 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.504+0000 7f2a02e00640 1 --2- 172.21.15.47:0/3126771778 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f29fc15c390 0x7f29fc1444b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.504+0000 7f2a02e00640 1 --2- 172.21.15.47:0/3126771778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29fc145590 0x7f29fc15b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.504+0000 7f2a02e00640 1 --2- 172.21.15.47:0/3126771778 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29fc0cea80 0x7f29fc13ef30 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.504+0000 7f2a02e00640 1 -- 172.21.15.47:0/3126771778 >> 172.21.15.47:0/3126771778 conn(0x7f29fc0b16a0 msgr2=0x7f29fc141af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:14.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.504+0000 7f2a02e00640 1 -- 172.21.15.47:0/3126771778 shutdown_connections 2024-02-19T15:26:14.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.504+0000 7f2a02e00640 1 -- 172.21.15.47:0/3126771778 wait complete. 2024-02-19T15:26:14.536 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:26:14.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.672+0000 7f4e89600640 1 Processor -- start 2024-02-19T15:26:14.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.672+0000 7f4e89600640 1 -- start start 2024-02-19T15:26:14.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.672+0000 7f4e89600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4e841613a0 0x7f4e840adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:14.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.672+0000 7f4e89600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4e8406fdf0 0x7f4e8406e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:14.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.672+0000 7f4e89600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e84161780 con 0x7f4e8406e5f0 2024-02-19T15:26:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.672+0000 7f4e89600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e840b01a0 con 0x7f4e8406fdf0 2024-02-19T15:26:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.672+0000 7f4e89600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e840ae3c0 con 0x7f4e841613a0 2024-02-19T15:26:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.672+0000 7f4e7a000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4e8406fdf0 0x7f4e8406e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.672+0000 7f4e7a000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4e8406fdf0 0x7f4e8406e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59210/0 (socket says 172.21.15.47:59210) 2024-02-19T15:26:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.672+0000 7f4e7a000640 1 -- 172.21.15.47:0/579541238 learned_addr learned my addr 172.21.15.47:0/579541238 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.672+0000 7f4e82a00640 1 --2- 172.21.15.47:0/579541238 >> v2:172.21.15.116:3300/0 conn(0x7f4e841613a0 0x7f4e840adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.676+0000 7f4e7a000640 1 -- 172.21.15.47:0/579541238 >> v2:172.21.15.116:3300/0 conn(0x7f4e841613a0 msgr2=0x7f4e840adbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.676+0000 7f4e7a000640 1 --2- 172.21.15.47:0/579541238 >> v2:172.21.15.116:3300/0 conn(0x7f4e841613a0 0x7f4e840adbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.676+0000 7f4e7a000640 1 -- 172.21.15.47:0/579541238 >> v1:172.21.15.116:6789/0 conn(0x7f4e8406e5f0 legacy=0x7f4e84161a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.676+0000 7f4e7a000640 1 -- 172.21.15.47:0/579541238 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e840ae6a0 con 0x7f4e8406fdf0 2024-02-19T15:26:14.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.676+0000 7f4e7a000640 1 --2- 172.21.15.47:0/579541238 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e8406fdf0 0x7f4e8406e0b0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f4e840d1bd0 tx=0x7f4e6c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=570602953b109f94 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:14.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.676+0000 7f4e82000640 1 -- 172.21.15.47:0/579541238 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c0ab600 con 0x7f4e8406fdf0 2024-02-19T15:26:14.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.676+0000 7f4e82000640 1 -- 172.21.15.47:0/579541238 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6c0a7050 con 0x7f4e8406fdf0 2024-02-19T15:26:14.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.676+0000 7f4e82000640 1 -- 172.21.15.47:0/579541238 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c0ab980 con 0x7f4e8406fdf0 2024-02-19T15:26:14.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.676+0000 7f4e89600640 1 -- 172.21.15.47:0/579541238 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e8406fdf0 msgr2=0x7f4e8406e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:14.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.676+0000 7f4e89600640 1 --2- 172.21.15.47:0/579541238 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e8406fdf0 0x7f4e8406e0b0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f4e840d1bd0 tx=0x7f4e6c01c490 comp rx=0 tx=0).stop 2024-02-19T15:26:14.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.676+0000 7f4e89600640 1 -- 172.21.15.47:0/579541238 shutdown_connections 2024-02-19T15:26:14.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.676+0000 7f4e89600640 1 --2- 172.21.15.47:0/579541238 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e8406fdf0 0x7f4e8406e0b0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.676+0000 7f4e89600640 1 --2- 172.21.15.47:0/579541238 >> v2:172.21.15.116:3300/0 conn(0x7f4e841613a0 0x7f4e840adbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.676+0000 7f4e89600640 1 -- 172.21.15.47:0/579541238 >> 172.21.15.47:0/579541238 conn(0x7f4e840d0ad0 msgr2=0x7f4e840d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:14.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.676+0000 7f4e89600640 1 -- 172.21.15.47:0/579541238 shutdown_connections 2024-02-19T15:26:14.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.676+0000 7f4e89600640 1 -- 172.21.15.47:0/579541238 wait complete. 2024-02-19T15:26:14.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e89600640 1 Processor -- start 2024-02-19T15:26:14.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e89600640 1 -- start start 2024-02-19T15:26:14.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e89600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e8406e5f0 0x7f4e8414fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:14.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e89600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e8406fdf0 0x7f4e840b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:14.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e89600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e841613a0 0x7f4e8415ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:14.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e89600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e840cd240 con 0x7f4e8406fdf0 2024-02-19T15:26:14.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e89600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e840cd520 con 0x7f4e8406e5f0 2024-02-19T15:26:14.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e89600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e840cd800 con 0x7f4e841613a0 2024-02-19T15:26:14.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e7a000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e8406fdf0 0x7f4e840b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:14.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e7a000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e8406fdf0 0x7f4e840b4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59226/0 (socket says 172.21.15.47:59226) 2024-02-19T15:26:14.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e7a000640 1 -- 172.21.15.47:0/137117987 learned_addr learned my addr 172.21.15.47:0/137117987 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:14.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e82a00640 1 --2- 172.21.15.47:0/137117987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e8406e5f0 0x7f4e8414fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:14.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e83400640 1 --2- 172.21.15.47:0/137117987 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e841613a0 0x7f4e8415ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:14.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e7a000640 1 -- 172.21.15.47:0/137117987 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e841613a0 msgr2=0x7f4e8415ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:14.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e7a000640 1 --2- 172.21.15.47:0/137117987 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e841613a0 0x7f4e8415ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e7a000640 1 -- 172.21.15.47:0/137117987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e8406e5f0 msgr2=0x7f4e8414fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:14.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e7a000640 1 --2- 172.21.15.47:0/137117987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e8406e5f0 0x7f4e8414fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e7a000640 1 -- 172.21.15.47:0/137117987 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6c0094d0 con 0x7f4e8406fdf0 2024-02-19T15:26:14.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e7a000640 1 --2- 172.21.15.47:0/137117987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e8406fdf0 0x7f4e840b4b10 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c0190a0 tx=0x7f4e6c019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:14.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e7b400640 1 -- 172.21.15.47:0/137117987 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c0b37d0 con 0x7f4e8406fdf0 2024-02-19T15:26:14.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e7b400640 1 -- 172.21.15.47:0/137117987 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6c0b3930 con 0x7f4e8406fdf0 2024-02-19T15:26:14.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e7b400640 1 -- 172.21.15.47:0/137117987 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c0b3ca0 con 0x7f4e8406fdf0 2024-02-19T15:26:14.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e89600640 1 -- 172.21.15.47:0/137117987 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e840cdae0 con 0x7f4e8406fdf0 2024-02-19T15:26:14.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.680+0000 7f4e89600640 1 -- 172.21.15.47:0/137117987 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e840b5050 con 0x7f4e8406fdf0 2024-02-19T15:26:14.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.684+0000 7f4e89600640 1 -- 172.21.15.47:0/137117987 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e84150460 con 0x7f4e8406fdf0 2024-02-19T15:26:14.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.684+0000 7f4e7b400640 1 -- 172.21.15.47:0/137117987 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4e6c0b2070 con 0x7f4e8406fdf0 2024-02-19T15:26:14.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.684+0000 7f4e7b400640 1 --2- 172.21.15.47:0/137117987 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4e60066e40 0x7f4e60069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:14.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.684+0000 7f4e7b400640 1 -- 172.21.15.47:0/137117987 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4e6c11ed10 con 0x7f4e8406fdf0 2024-02-19T15:26:14.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.684+0000 7f4e82a00640 1 --2- 172.21.15.47:0/137117987 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4e60066e40 0x7f4e60069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:14.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.684+0000 7f4e82a00640 1 --2- 172.21.15.47:0/137117987 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4e60066e40 0x7f4e60069300 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f4e740079c0 tx=0x7f4e74008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:14.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.688+0000 7f4e7b400640 1 -- 172.21.15.47:0/137117987 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f4e84150460 con 0x7f4e8406fdf0 2024-02-19T15:26:14.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.852+0000 7f4e89600640 1 -- 172.21.15.47:0/137117987 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e84150460 con 0x7f4e8406fdf0 2024-02-19T15:26:14.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.852+0000 7f4e7b400640 1 -- 172.21.15.47:0/137117987 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16672 (secure 0 0 0) 0x7f4e84150460 con 0x7f4e8406fdf0 2024-02-19T15:26:14.857 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:14.857 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:26:01.137113+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:25:57.938354+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-02-19T15:25:58.514866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:26:14.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.856+0000 7f4e89600640 1 -- 172.21.15.47:0/137117987 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4e60066e40 msgr2=0x7f4e60069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:14.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.856+0000 7f4e89600640 1 --2- 172.21.15.47:0/137117987 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4e60066e40 0x7f4e60069300 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f4e740079c0 tx=0x7f4e74008040 comp rx=0 tx=0).stop 2024-02-19T15:26:14.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.856+0000 7f4e89600640 1 -- 172.21.15.47:0/137117987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e8406fdf0 msgr2=0x7f4e840b4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:14.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.856+0000 7f4e89600640 1 --2- 172.21.15.47:0/137117987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e8406fdf0 0x7f4e840b4b10 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c0190a0 tx=0x7f4e6c019230 comp rx=0 tx=0).stop 2024-02-19T15:26:14.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.856+0000 7f4e89600640 1 -- 172.21.15.47:0/137117987 shutdown_connections 2024-02-19T15:26:14.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.856+0000 7f4e89600640 1 --2- 172.21.15.47:0/137117987 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4e60066e40 0x7f4e60069300 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.856+0000 7f4e89600640 1 --2- 172.21.15.47:0/137117987 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e841613a0 0x7f4e8415ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.856+0000 7f4e89600640 1 --2- 172.21.15.47:0/137117987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e8406fdf0 0x7f4e840b4b10 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.856+0000 7f4e89600640 1 --2- 172.21.15.47:0/137117987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e8406e5f0 0x7f4e8414fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:14.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.856+0000 7f4e89600640 1 -- 172.21.15.47:0/137117987 >> 172.21.15.47:0/137117987 conn(0x7f4e840d0ad0 msgr2=0x7f4e840cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:14.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.856+0000 7f4e89600640 1 -- 172.21.15.47:0/137117987 shutdown_connections 2024-02-19T15:26:14.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:14.856+0000 7f4e89600640 1 -- 172.21.15.47:0/137117987 wait complete. 2024-02-19T15:26:14.878 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T15:26:15.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.012+0000 7f442b800640 1 Processor -- start 2024-02-19T15:26:15.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.012+0000 7f442b800640 1 -- start start 2024-02-19T15:26:15.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.016+0000 7f442b800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f442406d650 0x7f44240d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:15.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.016+0000 7f442b800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f44240ced10 0x7f442406bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:15.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.016+0000 7f442b800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44240cce30 con 0x7f44240cded0 2024-02-19T15:26:15.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.016+0000 7f442b800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44240cd110 con 0x7f44240ced10 2024-02-19T15:26:15.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.016+0000 7f442b800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44240cd3f0 con 0x7f442406d650 2024-02-19T15:26:15.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.016+0000 7f4429800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f44240ced10 0x7f442406bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:15.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.016+0000 7f4429800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f44240ced10 0x7f442406bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59238/0 (socket says 172.21.15.47:59238) 2024-02-19T15:26:15.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.016+0000 7f4429800640 1 -- 172.21.15.47:0/3713740019 learned_addr learned my addr 172.21.15.47:0/3713740019 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:15.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.016+0000 7f4423e00640 1 --1- 172.21.15.47:0/3713740019 >> v1:172.21.15.47:6789/0 conn(0x7f44240cded0 0x7f44240ce600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37710/0 (socket says 172.21.15.47:37710) 2024-02-19T15:26:15.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.016+0000 7f4428e00640 1 --2- 172.21.15.47:0/3713740019 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f442406d650 0x7f44240d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:15.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.016+0000 7f4429800640 1 -- 172.21.15.47:0/3713740019 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f442406d650 msgr2=0x7f44240d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:15.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.016+0000 7f4429800640 1 --2- 172.21.15.47:0/3713740019 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f442406d650 0x7f44240d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:15.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.016+0000 7f4429800640 1 -- 172.21.15.47:0/3713740019 >> v1:172.21.15.47:6789/0 conn(0x7f44240cded0 legacy=0x7f44240ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:15.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.016+0000 7f4429800640 1 -- 172.21.15.47:0/3713740019 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44240cd6d0 con 0x7f44240ced10 2024-02-19T15:26:15.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.020+0000 7f4429800640 1 --2- 172.21.15.47:0/3713740019 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44240ced10 0x7f442406bd60 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f441c009e50 tx=0x7f441c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2513ae3dc9da7f9b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:15.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.020+0000 7f4423400640 1 -- 172.21.15.47:0/3713740019 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441c0a5850 con 0x7f44240ced10 2024-02-19T15:26:15.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.020+0000 7f4423400640 1 -- 172.21.15.47:0/3713740019 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f441c0a59b0 con 0x7f44240ced10 2024-02-19T15:26:15.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.020+0000 7f4423400640 1 -- 172.21.15.47:0/3713740019 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441c0a5d40 con 0x7f44240ced10 2024-02-19T15:26:15.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.020+0000 7f442b800640 1 -- 172.21.15.47:0/3713740019 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44240ced10 msgr2=0x7f442406bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:15.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.020+0000 7f442b800640 1 --2- 172.21.15.47:0/3713740019 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44240ced10 0x7f442406bd60 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f441c009e50 tx=0x7f441c01c970 comp rx=0 tx=0).stop 2024-02-19T15:26:15.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.020+0000 7f442b800640 1 -- 172.21.15.47:0/3713740019 shutdown_connections 2024-02-19T15:26:15.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.020+0000 7f442b800640 1 --2- 172.21.15.47:0/3713740019 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44240ced10 0x7f442406bd60 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:15.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.020+0000 7f442b800640 1 --2- 172.21.15.47:0/3713740019 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f442406d650 0x7f44240d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:15.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.020+0000 7f442b800640 1 -- 172.21.15.47:0/3713740019 >> 172.21.15.47:0/3713740019 conn(0x7f44240d1bf0 msgr2=0x7f44240d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:15.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.020+0000 7f442b800640 1 -- 172.21.15.47:0/3713740019 shutdown_connections 2024-02-19T15:26:15.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.020+0000 7f442b800640 1 -- 172.21.15.47:0/3713740019 wait complete. 2024-02-19T15:26:15.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.020+0000 7f442b800640 1 Processor -- start 2024-02-19T15:26:15.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.020+0000 7f442b800640 1 -- start start 2024-02-19T15:26:15.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f442b800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442406d650 0x7f44240cbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:15.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f442b800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44240cded0 0x7f442415c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:15.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f442b800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f44240ced10 0x7f442414abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:15.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f442b800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44240c9570 con 0x7f44240cded0 2024-02-19T15:26:15.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f442b800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44240c7f30 con 0x7f442406d650 2024-02-19T15:26:15.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f442b800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44240c8210 con 0x7f44240ced10 2024-02-19T15:26:15.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f4423e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44240cded0 0x7f442415c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:15.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f4423e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44240cded0 0x7f442415c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59246/0 (socket says 172.21.15.47:59246) 2024-02-19T15:26:15.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f4423e00640 1 -- 172.21.15.47:0/3302443203 learned_addr learned my addr 172.21.15.47:0/3302443203 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:15.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f4428e00640 1 --2- 172.21.15.47:0/3302443203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442406d650 0x7f44240cbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:15.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f4429800640 1 --2- 172.21.15.47:0/3302443203 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f44240ced10 0x7f442414abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:15.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f4423e00640 1 -- 172.21.15.47:0/3302443203 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f44240ced10 msgr2=0x7f442414abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:15.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f4423e00640 1 --2- 172.21.15.47:0/3302443203 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f44240ced10 0x7f442414abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:15.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f4423e00640 1 -- 172.21.15.47:0/3302443203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442406d650 msgr2=0x7f44240cbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:15.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f4423e00640 1 --2- 172.21.15.47:0/3302443203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442406d650 0x7f44240cbd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:15.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f4423e00640 1 -- 172.21.15.47:0/3302443203 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f441c009b30 con 0x7f44240cded0 2024-02-19T15:26:15.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f4428e00640 1 --2- 172.21.15.47:0/3302443203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442406d650 0x7f44240cbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:15.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f4429800640 1 --2- 172.21.15.47:0/3302443203 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f44240ced10 0x7f442414abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:15.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f4423e00640 1 --2- 172.21.15.47:0/3302443203 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44240cded0 0x7f442415c980 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f441000c370 tx=0x7f441000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:15.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f4420c00640 1 -- 172.21.15.47:0/3302443203 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44100074e0 con 0x7f44240cded0 2024-02-19T15:26:15.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f4420c00640 1 -- 172.21.15.47:0/3302443203 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4410007640 con 0x7f44240cded0 2024-02-19T15:26:15.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f442b800640 1 -- 172.21.15.47:0/3302443203 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44240c84f0 con 0x7f44240cded0 2024-02-19T15:26:15.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f4420c00640 1 -- 172.21.15.47:0/3302443203 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4410004a30 con 0x7f44240cded0 2024-02-19T15:26:15.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f442b800640 1 -- 172.21.15.47:0/3302443203 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f442415cec0 con 0x7f44240cded0 2024-02-19T15:26:15.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f442b800640 1 -- 172.21.15.47:0/3302443203 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4424075360 con 0x7f44240cded0 2024-02-19T15:26:15.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.024+0000 7f4420c00640 1 -- 172.21.15.47:0/3302443203 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4410004b90 con 0x7f44240cded0 2024-02-19T15:26:15.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.028+0000 7f4420c00640 1 --2- 172.21.15.47:0/3302443203 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4404066e40 0x7f4404069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:15.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.028+0000 7f4420c00640 1 -- 172.21.15.47:0/3302443203 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44100843f0 con 0x7f44240cded0 2024-02-19T15:26:15.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.028+0000 7f4428e00640 1 --2- 172.21.15.47:0/3302443203 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4404066e40 0x7f4404069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:15.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.028+0000 7f4428e00640 1 --2- 172.21.15.47:0/3302443203 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4404066e40 0x7f4404069300 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f4418009a40 tx=0x7f4418008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:15.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.032+0000 7f4420c00640 1 -- 172.21.15.47:0/3302443203 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f44100511d0 con 0x7f44240cded0 2024-02-19T15:26:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.256+0000 7f442b800640 1 -- 172.21.15.47:0/3302443203 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f44240cf580 con 0x7f44240cded0 2024-02-19T15:26:15.259 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-02-19T15:26:15.256+0000 7fc100600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T15:26:15.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.256+0000 7f4420c00640 1 -- 172.21.15.47:0/3302443203 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v50) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4410054890 con 0x7f44240cded0 2024-02-19T15:26:15.261 INFO:teuthology.orchestra.run.smithi047.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T15:26:15.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.260+0000 7f442b800640 1 -- 172.21.15.47:0/3302443203 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4404066e40 msgr2=0x7f4404069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:15.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.260+0000 7f442b800640 1 --2- 172.21.15.47:0/3302443203 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4404066e40 0x7f4404069300 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f4418009a40 tx=0x7f4418008040 comp rx=0 tx=0).stop 2024-02-19T15:26:15.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.260+0000 7f442b800640 1 -- 172.21.15.47:0/3302443203 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44240cded0 msgr2=0x7f442415c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:15.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.260+0000 7f442b800640 1 --2- 172.21.15.47:0/3302443203 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44240cded0 0x7f442415c980 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f441000c370 tx=0x7f441000c830 comp rx=0 tx=0).stop 2024-02-19T15:26:15.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.260+0000 7f442b800640 1 -- 172.21.15.47:0/3302443203 shutdown_connections 2024-02-19T15:26:15.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.260+0000 7f442b800640 1 --2- 172.21.15.47:0/3302443203 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4404066e40 0x7f4404069300 secure :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f4418009a40 tx=0x7f4418008040 comp rx=0 tx=0).stop 2024-02-19T15:26:15.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.260+0000 7f442b800640 1 --2- 172.21.15.47:0/3302443203 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f44240ced10 0x7f442414abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:15.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.260+0000 7f442b800640 1 --2- 172.21.15.47:0/3302443203 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44240cded0 0x7f442415c980 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:15.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.260+0000 7f442b800640 1 --2- 172.21.15.47:0/3302443203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442406d650 0x7f44240cbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:15.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.260+0000 7f442b800640 1 -- 172.21.15.47:0/3302443203 >> 172.21.15.47:0/3302443203 conn(0x7f44240d1bf0 msgr2=0x7f44240b5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:15.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.260+0000 7f442b800640 1 -- 172.21.15.47:0/3302443203 shutdown_connections 2024-02-19T15:26:15.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.260+0000 7f442b800640 1 -- 172.21.15.47:0/3302443203 wait complete. 2024-02-19T15:26:15.272 INFO:tasks.ceph.mds.c.smithi047.stderr: -3995> 2024-02-19T15:26:02.108+0000 7f366e200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T15:26:15.286 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T15:26:15.322 INFO:tasks.ceph.mds.c.smithi047.stdout:starting mds.c at 2024-02-19T15:26:15.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.428+0000 7f25fca00640 1 Processor -- start 2024-02-19T15:26:15.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.428+0000 7f25fca00640 1 -- start start 2024-02-19T15:26:15.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.428+0000 7f25fca00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f25f80d0650 0x7f25f80ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:15.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.428+0000 7f25fca00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25f80d2480 0x7f25f80b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:15.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.428+0000 7f25fca00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f80b0db0 con 0x7f25f815bc60 2024-02-19T15:26:15.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.428+0000 7f25fca00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f80cf9e0 con 0x7f25f80d0650 2024-02-19T15:26:15.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.428+0000 7f25fca00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f80b0fe0 con 0x7f25f80d2480 2024-02-19T15:26:15.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.428+0000 7f25eea00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f25f80d0650 0x7f25f80ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:15.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.428+0000 7f25f7400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f25f815bc60 0x7f25f80cff40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37730/0 (socket says 172.21.15.47:37730) 2024-02-19T15:26:15.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.428+0000 7f25f7400640 1 -- 172.21.15.47:0/198106311 learned_addr learned my addr 172.21.15.47:0/198106311 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:15.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.428+0000 7f25eea00640 1 -- 172.21.15.47:0/198106311 >> v2:172.21.15.116:3300/0 conn(0x7f25f80d2480 msgr2=0x7f25f80b1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:15.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.428+0000 7f25eea00640 1 --2- 172.21.15.47:0/198106311 >> v2:172.21.15.116:3300/0 conn(0x7f25f80d2480 0x7f25f80b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:15.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.428+0000 7f25eea00640 1 -- 172.21.15.47:0/198106311 >> v1:172.21.15.47:6789/0 conn(0x7f25f815bc60 legacy=0x7f25f80cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:15.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.428+0000 7f25eea00640 1 -- 172.21.15.47:0/198106311 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f80b12c0 con 0x7f25f80d0650 2024-02-19T15:26:15.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.428+0000 7f25f6a00640 1 -- 172.21.15.47:0/198106311 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905250671 0 0) 0x7f25f80b0db0 con 0x7f25f815bc60 2024-02-19T15:26:15.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.432+0000 7f25eea00640 1 --2- 172.21.15.47:0/198106311 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25f80d0650 0x7f25f80ce9a0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f25f80cf440 tx=0x7f25e001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7d77e47c9ecd71d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:15.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.432+0000 7f25f6a00640 1 -- 172.21.15.47:0/198106311 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e00ac620 con 0x7f25f80d0650 2024-02-19T15:26:15.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.432+0000 7f25f6a00640 1 -- 172.21.15.47:0/198106311 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e00ac780 con 0x7f25f80d0650 2024-02-19T15:26:15.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.432+0000 7f25f6a00640 1 -- 172.21.15.47:0/198106311 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e00acb10 con 0x7f25f80d0650 2024-02-19T15:26:15.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.432+0000 7f25fca00640 1 -- 172.21.15.47:0/198106311 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25f80d0650 msgr2=0x7f25f80ce9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:15.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.432+0000 7f25fca00640 1 --2- 172.21.15.47:0/198106311 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25f80d0650 0x7f25f80ce9a0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f25f80cf440 tx=0x7f25e001c3c0 comp rx=0 tx=0).stop 2024-02-19T15:26:15.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.432+0000 7f25fca00640 1 -- 172.21.15.47:0/198106311 shutdown_connections 2024-02-19T15:26:15.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.432+0000 7f25fca00640 1 --2- 172.21.15.47:0/198106311 >> v2:172.21.15.116:3300/0 conn(0x7f25f80d2480 0x7f25f80b1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:15.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.432+0000 7f25fca00640 1 --2- 172.21.15.47:0/198106311 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25f80d0650 0x7f25f80ce9a0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:15.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.432+0000 7f25fca00640 1 -- 172.21.15.47:0/198106311 >> 172.21.15.47:0/198106311 conn(0x7f25f80d1960 msgr2=0x7f25f80d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:15.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.432+0000 7f25fca00640 1 -- 172.21.15.47:0/198106311 shutdown_connections 2024-02-19T15:26:15.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.432+0000 7f25fca00640 1 -- 172.21.15.47:0/198106311 wait complete. 2024-02-19T15:26:15.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.432+0000 7f25fca00640 1 Processor -- start 2024-02-19T15:26:15.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25fca00640 1 -- start start 2024-02-19T15:26:15.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25fca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25f80d0650 0x7f25f8165310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:15.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25fca00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25f80d2480 0x7f25f8163a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:15.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25fca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25f815bc60 0x7f25f8156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:15.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25fca00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f8165850 con 0x7f25f815bc60 2024-02-19T15:26:15.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25fca00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f8164270 con 0x7f25f80d0650 2024-02-19T15:26:15.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25fca00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f8164550 con 0x7f25f80d2480 2024-02-19T15:26:15.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25f7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25f815bc60 0x7f25f8156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:15.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25f7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25f815bc60 0x7f25f8156ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59274/0 (socket says 172.21.15.47:59274) 2024-02-19T15:26:15.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25f7e00640 1 -- 172.21.15.47:0/2472991825 learned_addr learned my addr 172.21.15.47:0/2472991825 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:15.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25f7400640 1 --2- 172.21.15.47:0/2472991825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25f80d0650 0x7f25f8165310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:15.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25eea00640 1 --2- 172.21.15.47:0/2472991825 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25f80d2480 0x7f25f8163a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:15.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25f7e00640 1 -- 172.21.15.47:0/2472991825 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25f80d2480 msgr2=0x7f25f8163a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:15.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25f7e00640 1 --2- 172.21.15.47:0/2472991825 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25f80d2480 0x7f25f8163a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:15.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25f7e00640 1 -- 172.21.15.47:0/2472991825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25f80d0650 msgr2=0x7f25f8165310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:15.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25f7e00640 1 --2- 172.21.15.47:0/2472991825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25f80d0650 0x7f25f8165310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:15.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25f7e00640 1 -- 172.21.15.47:0/2472991825 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e0009420 con 0x7f25f815bc60 2024-02-19T15:26:15.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25f7400640 1 --2- 172.21.15.47:0/2472991825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25f80d0650 0x7f25f8165310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:15.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25eea00640 1 --2- 172.21.15.47:0/2472991825 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25f80d2480 0x7f25f8163a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:15.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25f7e00640 1 --2- 172.21.15.47:0/2472991825 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25f815bc60 0x7f25f8156ae0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f25f000b9e0 tx=0x7f25f000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:15.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25efe00640 1 -- 172.21.15.47:0/2472991825 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f000cb20 con 0x7f25f815bc60 2024-02-19T15:26:15.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25efe00640 1 -- 172.21.15.47:0/2472991825 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25f000cc80 con 0x7f25f815bc60 2024-02-19T15:26:15.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25fca00640 1 -- 172.21.15.47:0/2472991825 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f8164830 con 0x7f25f815bc60 2024-02-19T15:26:15.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25efe00640 1 -- 172.21.15.47:0/2472991825 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f0014440 con 0x7f25f815bc60 2024-02-19T15:26:15.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25fca00640 1 -- 172.21.15.47:0/2472991825 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f8164df0 con 0x7f25f815bc60 2024-02-19T15:26:15.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.436+0000 7f25fca00640 1 -- 172.21.15.47:0/2472991825 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f80b07a0 con 0x7f25f815bc60 2024-02-19T15:26:15.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.440+0000 7f25efe00640 1 -- 172.21.15.47:0/2472991825 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f25f001d3b0 con 0x7f25f815bc60 2024-02-19T15:26:15.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.440+0000 7f25efe00640 1 --2- 172.21.15.47:0/2472991825 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f25d4066e70 0x7f25d4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:15.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.440+0000 7f25efe00640 1 -- 172.21.15.47:0/2472991825 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f25f0084960 con 0x7f25f815bc60 2024-02-19T15:26:15.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.440+0000 7f25f7400640 1 --2- 172.21.15.47:0/2472991825 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f25d4066e70 0x7f25d4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:15.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.440+0000 7f25f7400640 1 --2- 172.21.15.47:0/2472991825 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f25d4066e70 0x7f25d4069330 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f25e8007fd0 tx=0x7f25e8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:15.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.444+0000 7f25efe00640 1 -- 172.21.15.47:0/2472991825 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f25f00516e0 con 0x7f25f815bc60 2024-02-19T15:26:15.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:15.644+0000 7f25fca00640 1 -- 172.21.15.47:0/2472991825 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f25f8164fe0 con 0x7f25f815bc60 2024-02-19T15:26:16.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.260+0000 7f25efe00640 1 -- 172.21.15.47:0/2472991825 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v51) v1 ==== 104+0+0 (secure 0 0 0) 0x7f25f00190a0 con 0x7f25f815bc60 2024-02-19T15:26:16.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.260+0000 7f25fca00640 1 -- 172.21.15.47:0/2472991825 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f25d4066e70 msgr2=0x7f25d4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:16.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.260+0000 7f25fca00640 1 --2- 172.21.15.47:0/2472991825 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f25d4066e70 0x7f25d4069330 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f25e8007fd0 tx=0x7f25e8009040 comp rx=0 tx=0).stop 2024-02-19T15:26:16.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.260+0000 7f25fca00640 1 -- 172.21.15.47:0/2472991825 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25f815bc60 msgr2=0x7f25f8156ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:16.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.260+0000 7f25fca00640 1 --2- 172.21.15.47:0/2472991825 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25f815bc60 0x7f25f8156ae0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f25f000b9e0 tx=0x7f25f000bea0 comp rx=0 tx=0).stop 2024-02-19T15:26:16.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.260+0000 7f25fca00640 1 -- 172.21.15.47:0/2472991825 shutdown_connections 2024-02-19T15:26:16.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.260+0000 7f25fca00640 1 --2- 172.21.15.47:0/2472991825 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f25d4066e70 0x7f25d4069330 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:16.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.260+0000 7f25fca00640 1 --2- 172.21.15.47:0/2472991825 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25f815bc60 0x7f25f8156ae0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:16.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.260+0000 7f25fca00640 1 --2- 172.21.15.47:0/2472991825 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25f80d2480 0x7f25f8163a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:16.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.260+0000 7f25fca00640 1 --2- 172.21.15.47:0/2472991825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25f80d0650 0x7f25f8165310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:16.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.260+0000 7f25fca00640 1 -- 172.21.15.47:0/2472991825 >> 172.21.15.47:0/2472991825 conn(0x7f25f80d1960 msgr2=0x7f25f81582c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:16.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.260+0000 7f25fca00640 1 -- 172.21.15.47:0/2472991825 shutdown_connections 2024-02-19T15:26:16.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.260+0000 7f25fca00640 1 -- 172.21.15.47:0/2472991825 wait complete. 2024-02-19T15:26:16.287 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T15:26:16.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.476+0000 7f6a44a00640 1 Processor -- start 2024-02-19T15:26:16.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.476+0000 7f6a44a00640 1 -- start start 2024-02-19T15:26:16.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.476+0000 7f6a44a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6a4006e570 0x7f6a40071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:16.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.476+0000 7f6a44a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a400ca520 con 0x7f6a40071050 2024-02-19T15:26:16.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.476+0000 7f6a44a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a400ca800 con 0x7f6a4015abf0 2024-02-19T15:26:16.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.476+0000 7f6a44a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a400caae0 con 0x7f6a4006e570 2024-02-19T15:26:16.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.476+0000 7f6a35400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6a4006e570 0x7f6a40071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:16.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.476+0000 7f6a3e800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6a40071050 0x7f6a400ae490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37734/0 (socket says 172.21.15.47:37734) 2024-02-19T15:26:16.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.476+0000 7f6a3e800640 1 -- 172.21.15.47:0/3802905809 learned_addr learned my addr 172.21.15.47:0/3802905809 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:16.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.476+0000 7f6a35400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6a4006e570 0x7f6a40071f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59282/0 (socket says 172.21.15.47:59282) 2024-02-19T15:26:16.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.476+0000 7f6a35400640 1 -- 172.21.15.47:0/3802905809 >> v1:172.21.15.116:6789/0 conn(0x7f6a4015abf0 legacy=0x7f6a4006de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:16.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.476+0000 7f6a35400640 1 -- 172.21.15.47:0/3802905809 >> v1:172.21.15.47:6789/0 conn(0x7f6a40071050 legacy=0x7f6a400ae490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:16.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.476+0000 7f6a35400640 1 -- 172.21.15.47:0/3802905809 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a400cadc0 con 0x7f6a4006e570 2024-02-19T15:26:16.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.476+0000 7f6a3d400640 1 -- 172.21.15.47:0/3802905809 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929352424 0 0) 0x7f6a400ca520 con 0x7f6a40071050 2024-02-19T15:26:16.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.480+0000 7f6a35400640 1 --2- 172.21.15.47:0/3802905809 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a4006e570 0x7f6a40071f00 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f6a280097c0 tx=0x7f6a2801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f50250ad053a5d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:16.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.480+0000 7f6a3d400640 1 -- 172.21.15.47:0/3802905809 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a280aa5f0 con 0x7f6a4006e570 2024-02-19T15:26:16.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.480+0000 7f6a3d400640 1 -- 172.21.15.47:0/3802905809 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a280aa750 con 0x7f6a4006e570 2024-02-19T15:26:16.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.480+0000 7f6a3d400640 1 -- 172.21.15.47:0/3802905809 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a280aaae0 con 0x7f6a4006e570 2024-02-19T15:26:16.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.480+0000 7f6a44a00640 1 -- 172.21.15.47:0/3802905809 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a4006e570 msgr2=0x7f6a40071f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:16.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.480+0000 7f6a44a00640 1 --2- 172.21.15.47:0/3802905809 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a4006e570 0x7f6a40071f00 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f6a280097c0 tx=0x7f6a2801c470 comp rx=0 tx=0).stop 2024-02-19T15:26:16.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.480+0000 7f6a44a00640 1 -- 172.21.15.47:0/3802905809 shutdown_connections 2024-02-19T15:26:16.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.480+0000 7f6a44a00640 1 --2- 172.21.15.47:0/3802905809 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a4006e570 0x7f6a40071f00 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:16.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.480+0000 7f6a44a00640 1 -- 172.21.15.47:0/3802905809 >> 172.21.15.47:0/3802905809 conn(0x7f6a400cc3f0 msgr2=0x7f6a400cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:16.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.480+0000 7f6a44a00640 1 -- 172.21.15.47:0/3802905809 shutdown_connections 2024-02-19T15:26:16.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.480+0000 7f6a44a00640 1 -- 172.21.15.47:0/3802905809 wait complete. 2024-02-19T15:26:16.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a44a00640 1 Processor -- start 2024-02-19T15:26:16.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a44a00640 1 -- start start 2024-02-19T15:26:16.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a44a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a4006e570 0x7f6a400c46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:16.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a44a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a40071050 0x7f6a400b40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:16.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a44a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a4015abf0 0x7f6a401f70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:16.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a44a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a400c4210 con 0x7f6a4015abf0 2024-02-19T15:26:16.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a44a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a400b4630 con 0x7f6a4006e570 2024-02-19T15:26:16.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a44a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a401f7610 con 0x7f6a40071050 2024-02-19T15:26:16.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a3e800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a4015abf0 0x7f6a401f70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:16.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a3e800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a4015abf0 0x7f6a401f70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59298/0 (socket says 172.21.15.47:59298) 2024-02-19T15:26:16.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a3e800640 1 -- 172.21.15.47:0/3763956144 learned_addr learned my addr 172.21.15.47:0/3763956144 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:16.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a3de00640 1 --2- 172.21.15.47:0/3763956144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a4006e570 0x7f6a400c46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:16.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a35400640 1 --2- 172.21.15.47:0/3763956144 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a40071050 0x7f6a400b40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:16.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a3e800640 1 -- 172.21.15.47:0/3763956144 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a40071050 msgr2=0x7f6a400b40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:16.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a3e800640 1 --2- 172.21.15.47:0/3763956144 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a40071050 0x7f6a400b40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:16.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a3e800640 1 -- 172.21.15.47:0/3763956144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a4006e570 msgr2=0x7f6a400c46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:16.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a3e800640 1 --2- 172.21.15.47:0/3763956144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a4006e570 0x7f6a400c46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:16.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a3e800640 1 -- 172.21.15.47:0/3763956144 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a28009420 con 0x7f6a4015abf0 2024-02-19T15:26:16.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a3e800640 1 --2- 172.21.15.47:0/3763956144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a4015abf0 0x7f6a401f70d0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f6a3800c7a0 tx=0x7f6a3800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:16.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a36a00640 1 -- 172.21.15.47:0/3763956144 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3800a640 con 0x7f6a4015abf0 2024-02-19T15:26:16.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a36a00640 1 -- 172.21.15.47:0/3763956144 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a3800a7a0 con 0x7f6a4015abf0 2024-02-19T15:26:16.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a44a00640 1 -- 172.21.15.47:0/3763956144 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a400c3600 con 0x7f6a4015abf0 2024-02-19T15:26:16.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a36a00640 1 -- 172.21.15.47:0/3763956144 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3800abd0 con 0x7f6a4015abf0 2024-02-19T15:26:16.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.484+0000 7f6a44a00640 1 -- 172.21.15.47:0/3763956144 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a400c3bc0 con 0x7f6a4015abf0 2024-02-19T15:26:16.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.488+0000 7f6a44a00640 1 -- 172.21.15.47:0/3763956144 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a400c4380 con 0x7f6a4015abf0 2024-02-19T15:26:16.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.488+0000 7f6a36a00640 1 -- 172.21.15.47:0/3763956144 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6a3800ad30 con 0x7f6a4015abf0 2024-02-19T15:26:16.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.488+0000 7f6a36a00640 1 --2- 172.21.15.47:0/3763956144 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6a1c066e70 0x7f6a1c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:16.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.488+0000 7f6a36a00640 1 -- 172.21.15.47:0/3763956144 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6a38084170 con 0x7f6a4015abf0 2024-02-19T15:26:16.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.488+0000 7f6a3de00640 1 --2- 172.21.15.47:0/3763956144 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6a1c066e70 0x7f6a1c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:16.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.488+0000 7f6a3de00640 1 --2- 172.21.15.47:0/3763956144 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6a1c066e70 0x7f6a1c069330 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6a30007fd0 tx=0x7f6a30009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:16.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.492+0000 7f6a36a00640 1 -- 172.21.15.47:0/3763956144 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6a38051040 con 0x7f6a4015abf0 2024-02-19T15:26:16.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:16.656+0000 7f6a44a00640 1 -- 172.21.15.47:0/3763956144 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6a40075360 con 0x7f6a4015abf0 2024-02-19T15:26:17.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.264+0000 7f6a36a00640 1 -- 172.21.15.47:0/3763956144 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v84) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6a38054700 con 0x7f6a4015abf0 2024-02-19T15:26:17.269 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_metadata' removed 2024-02-19T15:26:17.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.268+0000 7f6a44a00640 1 -- 172.21.15.47:0/3763956144 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6a1c066e70 msgr2=0x7f6a1c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:17.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.268+0000 7f6a44a00640 1 --2- 172.21.15.47:0/3763956144 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6a1c066e70 0x7f6a1c069330 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6a30007fd0 tx=0x7f6a30009040 comp rx=0 tx=0).stop 2024-02-19T15:26:17.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.268+0000 7f6a44a00640 1 -- 172.21.15.47:0/3763956144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a4015abf0 msgr2=0x7f6a401f70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:17.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.268+0000 7f6a44a00640 1 --2- 172.21.15.47:0/3763956144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a4015abf0 0x7f6a401f70d0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f6a3800c7a0 tx=0x7f6a3800cc60 comp rx=0 tx=0).stop 2024-02-19T15:26:17.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.268+0000 7f6a44a00640 1 -- 172.21.15.47:0/3763956144 shutdown_connections 2024-02-19T15:26:17.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.268+0000 7f6a44a00640 1 --2- 172.21.15.47:0/3763956144 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6a1c066e70 0x7f6a1c069330 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:17.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.268+0000 7f6a44a00640 1 --2- 172.21.15.47:0/3763956144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a4015abf0 0x7f6a401f70d0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:17.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.268+0000 7f6a44a00640 1 --2- 172.21.15.47:0/3763956144 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a40071050 0x7f6a400b40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:17.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.268+0000 7f6a44a00640 1 --2- 172.21.15.47:0/3763956144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a4006e570 0x7f6a400c46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:17.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.268+0000 7f6a44a00640 1 -- 172.21.15.47:0/3763956144 >> 172.21.15.47:0/3763956144 conn(0x7f6a400cc3f0 msgr2=0x7f6a400af4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:17.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.272+0000 7f6a44a00640 1 -- 172.21.15.47:0/3763956144 shutdown_connections 2024-02-19T15:26:17.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.272+0000 7f6a44a00640 1 -- 172.21.15.47:0/3763956144 wait complete. 2024-02-19T15:26:17.299 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T15:26:17.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fcce4e00640 1 Processor -- start 2024-02-19T15:26:17.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fcce4e00640 1 -- start start 2024-02-19T15:26:17.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fcce4e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce00b5270 0x7fcce00719f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:17.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fcce4e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcce006c920 0x7fcce00c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:17.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fcce4e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce00b5650 con 0x7fcce006c4c0 2024-02-19T15:26:17.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fcce4e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce015f7b0 con 0x7fcce006c920 2024-02-19T15:26:17.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fcce4e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce0071560 con 0x7fcce00b5270 2024-02-19T15:26:17.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdd400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fcce006c4c0 0x7fcce006e020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37738/0 (socket says 172.21.15.47:37738) 2024-02-19T15:26:17.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdd400640 1 -- 172.21.15.47:0/3554800578 learned_addr learned my addr 172.21.15.47:0/3554800578 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:17.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccde800640 1 --2- 172.21.15.47:0/3554800578 >> v2:172.21.15.116:3300/0 conn(0x7fcce006c920 0x7fcce00c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:17.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdde00640 1 --2- 172.21.15.47:0/3554800578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce00b5270 0x7fcce00719f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:17.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdca00640 1 -- 172.21.15.47:0/3554800578 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174763740 0 0) 0x7fcce00b5650 con 0x7fcce006c4c0 2024-02-19T15:26:17.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdca00640 1 -- 172.21.15.47:0/3554800578 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc4007940 con 0x7fcce006c4c0 2024-02-19T15:26:17.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdca00640 1 -- 172.21.15.47:0/3554800578 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2278509997 0 0) 0x7fccc4007940 con 0x7fcce006c4c0 2024-02-19T15:26:17.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdca00640 1 -- 172.21.15.47:0/3554800578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce00b5270 msgr2=0x7fcce00719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:17.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdca00640 1 --2- 172.21.15.47:0/3554800578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce00b5270 0x7fcce00719f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:17.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdca00640 1 -- 172.21.15.47:0/3554800578 >> v2:172.21.15.116:3300/0 conn(0x7fcce006c920 msgr2=0x7fcce00c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:17.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdca00640 1 --2- 172.21.15.47:0/3554800578 >> v2:172.21.15.116:3300/0 conn(0x7fcce006c920 0x7fcce00c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:17.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdca00640 1 -- 172.21.15.47:0/3554800578 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce0071f30 con 0x7fcce006c4c0 2024-02-19T15:26:17.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdca00640 1 -- 172.21.15.47:0/3554800578 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7fcccc001680 con 0x7fcce006c4c0 2024-02-19T15:26:17.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdca00640 1 -- 172.21.15.47:0/3554800578 >> v1:172.21.15.47:6789/0 conn(0x7fcce006c4c0 legacy=0x7fcce006e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:17.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdca00640 1 --2- 172.21.15.47:0/3554800578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccc4007d90 0x7fccc40081b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:17.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdca00640 1 --2- 172.21.15.47:0/3554800578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccc4008be0 0x7fccc4009060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:17.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdca00640 1 --2- 172.21.15.47:0/3554800578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce00b5270 0x7fccc40095a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:17.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdca00640 1 -- 172.21.15.47:0/3554800578 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce015f7b0 con 0x7fccc4008be0 2024-02-19T15:26:17.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdca00640 1 -- 172.21.15.47:0/3554800578 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce0071560 con 0x7fccc4007d90 2024-02-19T15:26:17.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdca00640 1 -- 172.21.15.47:0/3554800578 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc400fda0 con 0x7fcce00b5270 2024-02-19T15:26:17.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdd400640 1 --2- 172.21.15.47:0/3554800578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccc4008be0 0x7fccc4009060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:17.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdde00640 1 --2- 172.21.15.47:0/3554800578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccc4007d90 0x7fccc40081b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:17.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccde800640 1 --2- 172.21.15.47:0/3554800578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce00b5270 0x7fccc40095a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:17.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdd400640 1 -- 172.21.15.47:0/3554800578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce00b5270 msgr2=0x7fccc40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:17.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdd400640 1 --2- 172.21.15.47:0/3554800578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce00b5270 0x7fccc40095a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:17.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdd400640 1 -- 172.21.15.47:0/3554800578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccc4007d90 msgr2=0x7fccc40081b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:17.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdd400640 1 --2- 172.21.15.47:0/3554800578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccc4007d90 0x7fccc40081b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:17.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdd400640 1 -- 172.21.15.47:0/3554800578 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc4007940 con 0x7fccc4008be0 2024-02-19T15:26:17.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.444+0000 7fccdde00640 1 --2- 172.21.15.47:0/3554800578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccc4007d90 0x7fccc40081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:17.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.448+0000 7fccdd400640 1 --2- 172.21.15.47:0/3554800578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccc4008be0 0x7fccc4009060 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00c110 tx=0x7fcccc01f2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1fcafe821124b01 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:17.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.448+0000 7fccdca00640 1 -- 172.21.15.47:0/3554800578 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc0a8c30 con 0x7fccc4008be0 2024-02-19T15:26:17.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.448+0000 7fccdca00640 1 -- 172.21.15.47:0/3554800578 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcccc0a8db0 con 0x7fccc4008be0 2024-02-19T15:26:17.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.448+0000 7fccdca00640 1 -- 172.21.15.47:0/3554800578 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc005b20 con 0x7fccc4008be0 2024-02-19T15:26:17.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.448+0000 7fcce4e00640 1 -- 172.21.15.47:0/3554800578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccc4008be0 msgr2=0x7fccc4009060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:17.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.448+0000 7fcce4e00640 1 --2- 172.21.15.47:0/3554800578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccc4008be0 0x7fccc4009060 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00c110 tx=0x7fcccc01f2b0 comp rx=0 tx=0).stop 2024-02-19T15:26:17.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.448+0000 7fccde800640 1 -- 172.21.15.47:0/3554800578 reap_dead start 2024-02-19T15:26:17.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.448+0000 7fcce4e00640 1 -- 172.21.15.47:0/3554800578 shutdown_connections 2024-02-19T15:26:17.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.448+0000 7fcce4e00640 1 -- 172.21.15.47:0/3554800578 >> 172.21.15.47:0/3554800578 conn(0x7fcce00cc3f0 msgr2=0x7fcce00cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:17.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.448+0000 7fcce4e00640 1 -- 172.21.15.47:0/3554800578 shutdown_connections 2024-02-19T15:26:17.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.448+0000 7fcce4e00640 1 -- 172.21.15.47:0/3554800578 wait complete. 2024-02-19T15:26:17.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fcce4e00640 1 Processor -- start 2024-02-19T15:26:17.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fcce4e00640 1 -- start start 2024-02-19T15:26:17.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fcce4e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccc4008be0 0x7fcce0151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:17.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fcce4e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce015e220 0x7fcce00c6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:17.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fcce4e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcce006dfe0 0x7fcce006e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:17.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fcce4e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce006e9d0 con 0x7fcce006dfe0 2024-02-19T15:26:17.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fcce4e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce0152230 con 0x7fccc4008be0 2024-02-19T15:26:17.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fcce4e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce00c6f80 con 0x7fcce015e220 2024-02-19T15:26:17.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fccde800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcce006dfe0 0x7fcce006e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:17.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fccde800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcce006dfe0 0x7fcce006e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59328/0 (socket says 172.21.15.47:59328) 2024-02-19T15:26:17.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fccde800640 1 -- 172.21.15.47:0/2268513805 learned_addr learned my addr 172.21.15.47:0/2268513805 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:17.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fccdde00640 1 --2- 172.21.15.47:0/2268513805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccc4008be0 0x7fcce0151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:17.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fccdd400640 1 --2- 172.21.15.47:0/2268513805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce015e220 0x7fcce00c6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:17.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fccde800640 1 -- 172.21.15.47:0/2268513805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce015e220 msgr2=0x7fcce00c6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:17.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fccde800640 1 --2- 172.21.15.47:0/2268513805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce015e220 0x7fcce00c6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:17.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fccde800640 1 -- 172.21.15.47:0/2268513805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccc4008be0 msgr2=0x7fcce0151cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:17.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fccde800640 1 --2- 172.21.15.47:0/2268513805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccc4008be0 0x7fcce0151cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:17.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fccde800640 1 -- 172.21.15.47:0/2268513805 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce00c7250 con 0x7fcce006dfe0 2024-02-19T15:26:17.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fccde800640 1 --2- 172.21.15.47:0/2268513805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcce006dfe0 0x7fcce006e490 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fcce006cd70 tx=0x7fccd80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:17.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fccd6000640 1 -- 172.21.15.47:0/2268513805 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd800e070 con 0x7fcce006dfe0 2024-02-19T15:26:17.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fccd6000640 1 -- 172.21.15.47:0/2268513805 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd8011ed0 con 0x7fcce006dfe0 2024-02-19T15:26:17.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fccd6000640 1 -- 172.21.15.47:0/2268513805 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd80039a0 con 0x7fcce006dfe0 2024-02-19T15:26:17.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fcce4e00640 1 -- 172.21.15.47:0/2268513805 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce00c7530 con 0x7fcce006dfe0 2024-02-19T15:26:17.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.452+0000 7fcce4e00640 1 -- 172.21.15.47:0/2268513805 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce00c7af0 con 0x7fcce006dfe0 2024-02-19T15:26:17.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.456+0000 7fcce4e00640 1 -- 172.21.15.47:0/2268513805 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce0075360 con 0x7fcce006dfe0 2024-02-19T15:26:17.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.456+0000 7fccd6000640 1 -- 172.21.15.47:0/2268513805 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fccd8003b00 con 0x7fcce006dfe0 2024-02-19T15:26:17.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.456+0000 7fccd6000640 1 --2- 172.21.15.47:0/2268513805 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fccc0066e40 0x7fccc0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:17.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.456+0000 7fccd6000640 1 -- 172.21.15.47:0/2268513805 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fccd8085120 con 0x7fcce006dfe0 2024-02-19T15:26:17.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.456+0000 7fccdde00640 1 --2- 172.21.15.47:0/2268513805 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fccc0066e40 0x7fccc0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:17.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.456+0000 7fccdde00640 1 --2- 172.21.15.47:0/2268513805 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fccc0066e40 0x7fccc0069300 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fccd0006f90 tx=0x7fccd0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:17.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.460+0000 7fccd6000640 1 -- 172.21.15.47:0/2268513805 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fccd8052080 con 0x7fcce006dfe0 2024-02-19T15:26:17.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:17.628+0000 7fcce4e00640 1 -- 172.21.15.47:0/2268513805 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcce00711c0 con 0x7fcce006dfe0 2024-02-19T15:26:18.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.268+0000 7fccd6000640 1 -- 172.21.15.47:0/2268513805 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v85) v1 ==== 169+0+0 (secure 0 0 0) 0x7fccd8055740 con 0x7fcce006dfe0 2024-02-19T15:26:18.272 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_data' removed 2024-02-19T15:26:18.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.272+0000 7fcce4e00640 1 -- 172.21.15.47:0/2268513805 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fccc0066e40 msgr2=0x7fccc0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:18.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.272+0000 7fcce4e00640 1 --2- 172.21.15.47:0/2268513805 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fccc0066e40 0x7fccc0069300 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fccd0006f90 tx=0x7fccd0008040 comp rx=0 tx=0).stop 2024-02-19T15:26:18.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.272+0000 7fcce4e00640 1 -- 172.21.15.47:0/2268513805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcce006dfe0 msgr2=0x7fcce006e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:18.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.272+0000 7fcce4e00640 1 --2- 172.21.15.47:0/2268513805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcce006dfe0 0x7fcce006e490 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fcce006cd70 tx=0x7fccd80115a0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.272+0000 7fcce4e00640 1 -- 172.21.15.47:0/2268513805 shutdown_connections 2024-02-19T15:26:18.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.272+0000 7fcce4e00640 1 --2- 172.21.15.47:0/2268513805 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fccc0066e40 0x7fccc0069300 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.272+0000 7fcce4e00640 1 --2- 172.21.15.47:0/2268513805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcce006dfe0 0x7fcce006e490 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.272+0000 7fcce4e00640 1 --2- 172.21.15.47:0/2268513805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcce015e220 0x7fcce00c6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.272+0000 7fcce4e00640 1 --2- 172.21.15.47:0/2268513805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccc4008be0 0x7fcce0151cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.272+0000 7fcce4e00640 1 -- 172.21.15.47:0/2268513805 >> 172.21.15.47:0/2268513805 conn(0x7fcce00cc3f0 msgr2=0x7fcce00c2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:18.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.276+0000 7fcce4e00640 1 -- 172.21.15.47:0/2268513805 shutdown_connections 2024-02-19T15:26:18.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.276+0000 7fcce4e00640 1 -- 172.21.15.47:0/2268513805 wait complete. 2024-02-19T15:26:18.308 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-19T15:26:18.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.448+0000 7f457b200640 1 Processor -- start 2024-02-19T15:26:18.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.448+0000 7f457b200640 1 -- start start 2024-02-19T15:26:18.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.448+0000 7f457b200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f457406b110 0x7f4574072f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:18.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.448+0000 7f457b200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f45740d0f40 0x7f457415df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:18.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.448+0000 7f457b200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f457406b4f0 0x7f457415e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:18.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.448+0000 7f457b200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4574073570 con 0x7f45740d0f40 2024-02-19T15:26:18.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.448+0000 7f457b200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45740b0210 con 0x7f457406b4f0 2024-02-19T15:26:18.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.448+0000 7f457b200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45740b0490 con 0x7f457406b110 2024-02-19T15:26:18.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.448+0000 7f456be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f45740d0f40 0x7f457415df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:18.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.448+0000 7f456be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f45740d0f40 0x7f457415df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59334/0 (socket says 172.21.15.47:59334) 2024-02-19T15:26:18.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.448+0000 7f4573e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f457406b110 0x7f4574072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:18.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.448+0000 7f456be00640 1 -- 172.21.15.47:0/681204335 learned_addr learned my addr 172.21.15.47:0/681204335 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:18.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.448+0000 7f4579200640 1 --2- 172.21.15.47:0/681204335 >> v2:172.21.15.116:3300/0 conn(0x7f457406b4f0 0x7f457415e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:18.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.452+0000 7f456be00640 1 -- 172.21.15.47:0/681204335 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f457406b110 msgr2=0x7f4574072f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:18.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.452+0000 7f456be00640 1 --2- 172.21.15.47:0/681204335 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f457406b110 0x7f4574072f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.452+0000 7f456be00640 1 -- 172.21.15.47:0/681204335 >> v2:172.21.15.116:3300/0 conn(0x7f457406b4f0 msgr2=0x7f457415e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:18.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.452+0000 7f456be00640 1 --2- 172.21.15.47:0/681204335 >> v2:172.21.15.116:3300/0 conn(0x7f457406b4f0 0x7f457415e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.452+0000 7f456be00640 1 -- 172.21.15.47:0/681204335 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4574066e00 con 0x7f45740d0f40 2024-02-19T15:26:18.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.452+0000 7f456be00640 1 --2- 172.21.15.47:0/681204335 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45740d0f40 0x7f457415df80 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f457406fa60 tx=0x7f455c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e5e8e5a6a2f3f87 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:18.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.452+0000 7f4573400640 1 -- 172.21.15.47:0/681204335 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455c0ab670 con 0x7f45740d0f40 2024-02-19T15:26:18.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.452+0000 7f4573400640 1 -- 172.21.15.47:0/681204335 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f455c0ab7d0 con 0x7f45740d0f40 2024-02-19T15:26:18.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.452+0000 7f4573400640 1 -- 172.21.15.47:0/681204335 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455c0abb60 con 0x7f45740d0f40 2024-02-19T15:26:18.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.452+0000 7f457b200640 1 -- 172.21.15.47:0/681204335 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45740d0f40 msgr2=0x7f457415df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:18.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.452+0000 7f457b200640 1 --2- 172.21.15.47:0/681204335 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45740d0f40 0x7f457415df80 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f457406fa60 tx=0x7f455c01c490 comp rx=0 tx=0).stop 2024-02-19T15:26:18.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f457b200640 1 -- 172.21.15.47:0/681204335 shutdown_connections 2024-02-19T15:26:18.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f457b200640 1 --2- 172.21.15.47:0/681204335 >> v2:172.21.15.116:3300/0 conn(0x7f457406b4f0 0x7f457415e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f457b200640 1 --2- 172.21.15.47:0/681204335 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45740d0f40 0x7f457415df80 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f457b200640 1 --2- 172.21.15.47:0/681204335 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f457406b110 0x7f4574072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f457b200640 1 -- 172.21.15.47:0/681204335 >> 172.21.15.47:0/681204335 conn(0x7f45740d0420 msgr2=0x7f45740d0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:18.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f457b200640 1 -- 172.21.15.47:0/681204335 shutdown_connections 2024-02-19T15:26:18.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f457b200640 1 -- 172.21.15.47:0/681204335 wait complete. 2024-02-19T15:26:18.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f457b200640 1 Processor -- start 2024-02-19T15:26:18.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f457b200640 1 -- start start 2024-02-19T15:26:18.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f457b200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f457406b110 0x7f45740c3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:18.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f457b200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f457406b4f0 0x7f45740af050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:18.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f457b200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f45740d0f40 0x7f45740ae840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:18.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f457b200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457409e050 con 0x7f457406b110 2024-02-19T15:26:18.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f457b200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4574158b50 con 0x7f457406b4f0 2024-02-19T15:26:18.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f457b200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45740af590 con 0x7f45740d0f40 2024-02-19T15:26:18.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f4573e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f457406b110 0x7f45740c3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:18.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f4573e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f457406b110 0x7f45740c3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59346/0 (socket says 172.21.15.47:59346) 2024-02-19T15:26:18.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f4573e00640 1 -- 172.21.15.47:0/3257985474 learned_addr learned my addr 172.21.15.47:0/3257985474 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:18.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f456be00640 1 --2- 172.21.15.47:0/3257985474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f457406b4f0 0x7f45740af050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:18.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f4579200640 1 --2- 172.21.15.47:0/3257985474 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f45740d0f40 0x7f45740ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:18.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f4573e00640 1 -- 172.21.15.47:0/3257985474 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f45740d0f40 msgr2=0x7f45740ae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:18.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f4573e00640 1 --2- 172.21.15.47:0/3257985474 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f45740d0f40 0x7f45740ae840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f4573e00640 1 -- 172.21.15.47:0/3257985474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f457406b4f0 msgr2=0x7f45740af050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:18.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f4573e00640 1 --2- 172.21.15.47:0/3257985474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f457406b4f0 0x7f45740af050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f4573e00640 1 -- 172.21.15.47:0/3257985474 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455c0094d0 con 0x7f457406b110 2024-02-19T15:26:18.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.456+0000 7f4573e00640 1 --2- 172.21.15.47:0/3257985474 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f457406b110 0x7f45740c3f50 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f456400e870 tx=0x7f456400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:18.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.460+0000 7f4570c00640 1 -- 172.21.15.47:0/3257985474 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456400ee60 con 0x7f457406b110 2024-02-19T15:26:18.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.460+0000 7f4570c00640 1 -- 172.21.15.47:0/3257985474 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f456400c940 con 0x7f457406b110 2024-02-19T15:26:18.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.460+0000 7f4570c00640 1 -- 172.21.15.47:0/3257985474 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456400ccf0 con 0x7f457406b110 2024-02-19T15:26:18.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.460+0000 7f457b200640 1 -- 172.21.15.47:0/3257985474 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4574157f40 con 0x7f457406b110 2024-02-19T15:26:18.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.460+0000 7f457b200640 1 -- 172.21.15.47:0/3257985474 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4574158500 con 0x7f457406b110 2024-02-19T15:26:18.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.460+0000 7f457b200640 1 -- 172.21.15.47:0/3257985474 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4574075360 con 0x7f457406b110 2024-02-19T15:26:18.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.460+0000 7f4570c00640 1 -- 172.21.15.47:0/3257985474 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4564013470 con 0x7f457406b110 2024-02-19T15:26:18.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.460+0000 7f4570c00640 1 --2- 172.21.15.47:0/3257985474 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4550066e40 0x7f4550069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:18.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.460+0000 7f4570c00640 1 -- 172.21.15.47:0/3257985474 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f45640847d0 con 0x7f457406b110 2024-02-19T15:26:18.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.460+0000 7f456be00640 1 --2- 172.21.15.47:0/3257985474 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4550066e40 0x7f4550069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:18.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.460+0000 7f456be00640 1 --2- 172.21.15.47:0/3257985474 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4550066e40 0x7f4550069300 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f455c0af000 tx=0x7f455c0b6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:18.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.464+0000 7f4570c00640 1 -- 172.21.15.47:0/3257985474 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f45640518d0 con 0x7f457406b110 2024-02-19T15:26:18.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.624+0000 7f457b200640 1 -- 172.21.15.47:0/3257985474 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f457415ab30 con 0x7f457406b110 2024-02-19T15:26:18.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.628+0000 7f4570c00640 1 -- 172.21.15.47:0/3257985474 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4564054f90 con 0x7f457406b110 2024-02-19T15:26:18.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.632+0000 7f457b200640 1 -- 172.21.15.47:0/3257985474 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4550066e40 msgr2=0x7f4550069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:18.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.632+0000 7f457b200640 1 --2- 172.21.15.47:0/3257985474 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4550066e40 0x7f4550069300 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f455c0af000 tx=0x7f455c0b6040 comp rx=0 tx=0).stop 2024-02-19T15:26:18.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.632+0000 7f457b200640 1 -- 172.21.15.47:0/3257985474 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f457406b110 msgr2=0x7f45740c3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:18.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.632+0000 7f457b200640 1 --2- 172.21.15.47:0/3257985474 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f457406b110 0x7f45740c3f50 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f456400e870 tx=0x7f456400ed30 comp rx=0 tx=0).stop 2024-02-19T15:26:18.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.632+0000 7f457b200640 1 -- 172.21.15.47:0/3257985474 shutdown_connections 2024-02-19T15:26:18.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.632+0000 7f457b200640 1 --2- 172.21.15.47:0/3257985474 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4550066e40 0x7f4550069300 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.632+0000 7f457b200640 1 --2- 172.21.15.47:0/3257985474 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f45740d0f40 0x7f45740ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.632+0000 7f457b200640 1 --2- 172.21.15.47:0/3257985474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f457406b4f0 0x7f45740af050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.632+0000 7f457b200640 1 --2- 172.21.15.47:0/3257985474 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f457406b110 0x7f45740c3f50 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.632+0000 7f457b200640 1 -- 172.21.15.47:0/3257985474 >> 172.21.15.47:0/3257985474 conn(0x7f45740d0420 msgr2=0x7f45740b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:18.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.632+0000 7f457b200640 1 -- 172.21.15.47:0/3257985474 shutdown_connections 2024-02-19T15:26:18.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.632+0000 7f457b200640 1 -- 172.21.15.47:0/3257985474 wait complete. 2024-02-19T15:26:18.655 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-02-19T15:26:18.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.788+0000 7fc1ad600640 1 Processor -- start 2024-02-19T15:26:18.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.792+0000 7fc1ad600640 1 -- start start 2024-02-19T15:26:18.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.792+0000 7fc1ad600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc1a8145590 0x7fc1a80727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:18.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.792+0000 7fc1ad600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc1a815c390 0x7fc1a815c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:18.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.792+0000 7fc1ad600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a815d280 con 0x7fc1a80cea80 2024-02-19T15:26:18.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.792+0000 7fc1ad600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1a8070ff0 con 0x7fc1a8145590 2024-02-19T15:26:18.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.792+0000 7fc1ad600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1a815cd40 con 0x7fc1a815c390 2024-02-19T15:26:18.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.792+0000 7fc1a6a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc1a8145590 0x7fc1a80727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:18.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.792+0000 7fc19e000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc1a80cea80 0x7fc1a806c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37746/0 (socket says 172.21.15.47:37746) 2024-02-19T15:26:18.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.792+0000 7fc19e000640 1 -- 172.21.15.47:0/2557465590 learned_addr learned my addr 172.21.15.47:0/2557465590 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:18.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.792+0000 7fc1a7400640 1 --2- 172.21.15.47:0/2557465590 >> v2:172.21.15.116:3300/0 conn(0x7fc1a815c390 0x7fc1a815c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:18.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.792+0000 7fc1a6a00640 1 -- 172.21.15.47:0/2557465590 >> v2:172.21.15.116:3300/0 conn(0x7fc1a815c390 msgr2=0x7fc1a815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:18.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.792+0000 7fc1a6a00640 1 --2- 172.21.15.47:0/2557465590 >> v2:172.21.15.116:3300/0 conn(0x7fc1a815c390 0x7fc1a815c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.792+0000 7fc1a6a00640 1 -- 172.21.15.47:0/2557465590 >> v1:172.21.15.47:6789/0 conn(0x7fc1a80cea80 legacy=0x7fc1a806c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:18.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.792+0000 7fc1a6a00640 1 -- 172.21.15.47:0/2557465590 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a80709e0 con 0x7fc1a8145590 2024-02-19T15:26:18.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.792+0000 7fc1a6000640 1 -- 172.21.15.47:0/2557465590 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534338729 0 0) 0x7fc1a815d280 con 0x7fc1a80cea80 2024-02-19T15:26:18.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.796+0000 7fc1a6a00640 1 --2- 172.21.15.47:0/2557465590 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1a8145590 0x7fc1a80727b0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8072380 tx=0x7fc19801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7d3926fc734029 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:18.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.796+0000 7fc1a6000640 1 -- 172.21.15.47:0/2557465590 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1980a58e0 con 0x7fc1a8145590 2024-02-19T15:26:18.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.796+0000 7fc1a6000640 1 -- 172.21.15.47:0/2557465590 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1980a5a40 con 0x7fc1a8145590 2024-02-19T15:26:18.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.796+0000 7fc1a6000640 1 -- 172.21.15.47:0/2557465590 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1980a5dd0 con 0x7fc1a8145590 2024-02-19T15:26:18.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.796+0000 7fc1ad600640 1 -- 172.21.15.47:0/2557465590 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1a8145590 msgr2=0x7fc1a80727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:18.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.796+0000 7fc1ad600640 1 --2- 172.21.15.47:0/2557465590 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1a8145590 0x7fc1a80727b0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8072380 tx=0x7fc19801c930 comp rx=0 tx=0).stop 2024-02-19T15:26:18.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.796+0000 7fc1ad600640 1 -- 172.21.15.47:0/2557465590 shutdown_connections 2024-02-19T15:26:18.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.796+0000 7fc1ad600640 1 --2- 172.21.15.47:0/2557465590 >> v2:172.21.15.116:3300/0 conn(0x7fc1a815c390 0x7fc1a815c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.796+0000 7fc1ad600640 1 --2- 172.21.15.47:0/2557465590 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1a8145590 0x7fc1a80727b0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.796+0000 7fc1ad600640 1 -- 172.21.15.47:0/2557465590 >> 172.21.15.47:0/2557465590 conn(0x7fc1a80b16a0 msgr2=0x7fc1a80b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:18.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.796+0000 7fc1ad600640 1 -- 172.21.15.47:0/2557465590 shutdown_connections 2024-02-19T15:26:18.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.796+0000 7fc1ad600640 1 -- 172.21.15.47:0/2557465590 wait complete. 2024-02-19T15:26:18.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.796+0000 7fc1ad600640 1 Processor -- start 2024-02-19T15:26:18.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.796+0000 7fc1ad600640 1 -- start start 2024-02-19T15:26:18.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc1ad600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1a80cea80 0x7fc1a80c4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:18.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc1ad600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1a8145590 0x7fc1a80b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:18.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc1ad600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1a815c390 0x7fc1a813e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:18.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc1ad600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a813eee0 con 0x7fc1a8145590 2024-02-19T15:26:18.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc1ad600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a813d4f0 con 0x7fc1a815c390 2024-02-19T15:26:18.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc1ad600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a813ec00 con 0x7fc1a80cea80 2024-02-19T15:26:18.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc19e000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1a8145590 0x7fc1a80b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:18.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc19e000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1a8145590 0x7fc1a80b3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59374/0 (socket says 172.21.15.47:59374) 2024-02-19T15:26:18.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc19e000640 1 -- 172.21.15.47:0/1194400855 learned_addr learned my addr 172.21.15.47:0/1194400855 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:18.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc1a7400640 1 --2- 172.21.15.47:0/1194400855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1a815c390 0x7fc1a813e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:18.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc1a6a00640 1 --2- 172.21.15.47:0/1194400855 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1a80cea80 0x7fc1a80c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:18.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc19e000640 1 -- 172.21.15.47:0/1194400855 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1a80cea80 msgr2=0x7fc1a80c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:18.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc19e000640 1 --2- 172.21.15.47:0/1194400855 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1a80cea80 0x7fc1a80c4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc19e000640 1 -- 172.21.15.47:0/1194400855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1a815c390 msgr2=0x7fc1a813e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:18.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc19e000640 1 --2- 172.21.15.47:0/1194400855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1a815c390 0x7fc1a813e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:18.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc19e000640 1 -- 172.21.15.47:0/1194400855 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc198009a70 con 0x7fc1a8145590 2024-02-19T15:26:18.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc19e000640 1 --2- 172.21.15.47:0/1194400855 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1a8145590 0x7fc1a80b3ee0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fc1900011f0 tx=0x7fc19000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:18.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc19f400640 1 -- 172.21.15.47:0/1194400855 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1900074e0 con 0x7fc1a8145590 2024-02-19T15:26:18.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc19f400640 1 -- 172.21.15.47:0/1194400855 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc190007640 con 0x7fc1a8145590 2024-02-19T15:26:18.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc19f400640 1 -- 172.21.15.47:0/1194400855 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc190004a10 con 0x7fc1a8145590 2024-02-19T15:26:18.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc1ad600640 1 -- 172.21.15.47:0/1194400855 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a8142290 con 0x7fc1a8145590 2024-02-19T15:26:18.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc1ad600640 1 -- 172.21.15.47:0/1194400855 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a8142850 con 0x7fc1a8145590 2024-02-19T15:26:18.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc1ad600640 1 -- 172.21.15.47:0/1194400855 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a8142a70 con 0x7fc1a8145590 2024-02-19T15:26:18.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.800+0000 7fc19f400640 1 -- 172.21.15.47:0/1194400855 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc190004b70 con 0x7fc1a8145590 2024-02-19T15:26:18.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.804+0000 7fc19f400640 1 --2- 172.21.15.47:0/1194400855 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc184066e40 0x7fc184069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:18.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.804+0000 7fc19f400640 1 -- 172.21.15.47:0/1194400855 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc190083c80 con 0x7fc1a8145590 2024-02-19T15:26:18.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.804+0000 7fc1a6a00640 1 --2- 172.21.15.47:0/1194400855 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc184066e40 0x7fc184069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:18.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.804+0000 7fc1a6a00640 1 --2- 172.21.15.47:0/1194400855 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc184066e40 0x7fc184069300 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc1980af000 tx=0x7fc1980a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:18.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:18.808+0000 7fc19f400640 1 -- 172.21.15.47:0/1194400855 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc190050d60 con 0x7fc1a8145590 2024-02-19T15:26:19.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.032+0000 7fc1ad600640 1 -- 172.21.15.47:0/1194400855 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7fc1a8075360 con 0x7fc1a8145590 2024-02-19T15:26:19.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.276+0000 7fc19f400640 1 -- 172.21.15.47:0/1194400855 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v246) v1 ==== 132+0+0 (secure 0 0 0) 0x7fc1900180c0 con 0x7fc1a8145590 2024-02-19T15:26:19.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.280+0000 7fc1ad600640 1 -- 172.21.15.47:0/1194400855 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc184066e40 msgr2=0x7fc184069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:19.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.280+0000 7fc1ad600640 1 --2- 172.21.15.47:0/1194400855 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc184066e40 0x7fc184069300 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc1980af000 tx=0x7fc1980a7070 comp rx=0 tx=0).stop 2024-02-19T15:26:19.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.280+0000 7fc1ad600640 1 -- 172.21.15.47:0/1194400855 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1a8145590 msgr2=0x7fc1a80b3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:19.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.280+0000 7fc1ad600640 1 --2- 172.21.15.47:0/1194400855 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1a8145590 0x7fc1a80b3ee0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fc1900011f0 tx=0x7fc19000c9b0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.280+0000 7fc1ad600640 1 -- 172.21.15.47:0/1194400855 shutdown_connections 2024-02-19T15:26:19.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.280+0000 7fc1ad600640 1 --2- 172.21.15.47:0/1194400855 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc184066e40 0x7fc184069300 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.280+0000 7fc1ad600640 1 --2- 172.21.15.47:0/1194400855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1a815c390 0x7fc1a813e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.280+0000 7fc1ad600640 1 --2- 172.21.15.47:0/1194400855 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1a8145590 0x7fc1a80b3ee0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.280+0000 7fc1ad600640 1 --2- 172.21.15.47:0/1194400855 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1a80cea80 0x7fc1a80c4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.280+0000 7fc1ad600640 1 -- 172.21.15.47:0/1194400855 >> 172.21.15.47:0/1194400855 conn(0x7fc1a80b16a0 msgr2=0x7fc1a815b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:19.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.280+0000 7fc1ad600640 1 -- 172.21.15.47:0/1194400855 shutdown_connections 2024-02-19T15:26:19.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.280+0000 7fc1ad600640 1 -- 172.21.15.47:0/1194400855 wait complete. 2024-02-19T15:26:19.305 INFO:tasks.cephfs_test_runner:purge stray file with fancy layout ... ok 2024-02-19T15:26:19.305 INFO:tasks.cephfs_test_runner:Starting test: test_files_throttle (tasks.cephfs.test_strays.TestStrays) 2024-02-19T15:26:19.305 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:26:19.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.440+0000 7fd196000640 1 Processor -- start 2024-02-19T15:26:19.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.440+0000 7fd196000640 1 -- start start 2024-02-19T15:26:19.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.444+0000 7fd196000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd19015b1a0 0x7fd19015b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:19.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.444+0000 7fd196000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd19006c9a0 0x7fd19015bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:19.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.444+0000 7fd196000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd19015c000 con 0x7fd190071350 2024-02-19T15:26:19.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.444+0000 7fd196000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1900718c0 con 0x7fd19006c9a0 2024-02-19T15:26:19.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.444+0000 7fd196000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd190071ba0 con 0x7fd19015b1a0 2024-02-19T15:26:19.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.444+0000 7fd18fe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd19006c9a0 0x7fd19015bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:19.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.444+0000 7fd18fe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd19006c9a0 0x7fd19015bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:38210/0 (socket says 172.21.15.47:38210) 2024-02-19T15:26:19.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.444+0000 7fd187e00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd190071350 0x7fd19015c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:40658/0 (socket says 172.21.15.47:40658) 2024-02-19T15:26:19.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.444+0000 7fd18f400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd19015b1a0 0x7fd19015b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:19.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.444+0000 7fd18fe00640 1 -- 172.21.15.47:0/1799335773 learned_addr learned my addr 172.21.15.47:0/1799335773 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:19.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.444+0000 7fd18fe00640 1 -- 172.21.15.47:0/1799335773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd19015b1a0 msgr2=0x7fd19015b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:19.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.444+0000 7fd18fe00640 1 --2- 172.21.15.47:0/1799335773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd19015b1a0 0x7fd19015b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.444+0000 7fd18fe00640 1 -- 172.21.15.47:0/1799335773 >> v1:172.21.15.116:6789/0 conn(0x7fd190071350 legacy=0x7fd19015c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:19.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.444+0000 7fd18fe00640 1 -- 172.21.15.47:0/1799335773 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd190071e80 con 0x7fd19006c9a0 2024-02-19T15:26:19.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.444+0000 7fd18f400640 1 --2- 172.21.15.47:0/1799335773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd19015b1a0 0x7fd19015b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:19.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.448+0000 7fd18fe00640 1 --2- 172.21.15.47:0/1799335773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd19006c9a0 0x7fd19015bac0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd188009e50 tx=0x7fd18801ca00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e2f96211394eacd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:19.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.448+0000 7fd18ea00640 1 -- 172.21.15.47:0/1799335773 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1880a58f0 con 0x7fd19006c9a0 2024-02-19T15:26:19.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.448+0000 7fd18ea00640 1 -- 172.21.15.47:0/1799335773 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1880a5a50 con 0x7fd19006c9a0 2024-02-19T15:26:19.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.448+0000 7fd18ea00640 1 -- 172.21.15.47:0/1799335773 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1880a5de0 con 0x7fd19006c9a0 2024-02-19T15:26:19.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.448+0000 7fd196000640 1 -- 172.21.15.47:0/1799335773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd19006c9a0 msgr2=0x7fd19015bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:19.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.448+0000 7fd196000640 1 --2- 172.21.15.47:0/1799335773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd19006c9a0 0x7fd19015bac0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd188009e50 tx=0x7fd18801ca00 comp rx=0 tx=0).stop 2024-02-19T15:26:19.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.448+0000 7fd196000640 1 -- 172.21.15.47:0/1799335773 shutdown_connections 2024-02-19T15:26:19.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.448+0000 7fd196000640 1 --2- 172.21.15.47:0/1799335773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd19006c9a0 0x7fd19015bac0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.448+0000 7fd196000640 1 --2- 172.21.15.47:0/1799335773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd19015b1a0 0x7fd19015b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.448+0000 7fd196000640 1 -- 172.21.15.47:0/1799335773 >> 172.21.15.47:0/1799335773 conn(0x7fd1900b16a0 msgr2=0x7fd1900b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:19.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.448+0000 7fd196000640 1 -- 172.21.15.47:0/1799335773 shutdown_connections 2024-02-19T15:26:19.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.448+0000 7fd196000640 1 -- 172.21.15.47:0/1799335773 wait complete. 2024-02-19T15:26:19.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.448+0000 7fd196000640 1 Processor -- start 2024-02-19T15:26:19.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.448+0000 7fd196000640 1 -- start start 2024-02-19T15:26:19.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd196000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd19006c9a0 0x7fd190139220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:19.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd196000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd190071350 0x7fd1900ae570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:19.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd196000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd19015b1a0 0x7fd1900aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:19.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd196000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1900ad840 con 0x7fd19015b1a0 2024-02-19T15:26:19.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd196000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1900ae0e0 con 0x7fd19006c9a0 2024-02-19T15:26:19.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd196000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd190139760 con 0x7fd190071350 2024-02-19T15:26:19.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd18fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd19015b1a0 0x7fd1900aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:19.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd18fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd19015b1a0 0x7fd1900aeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59384/0 (socket says 172.21.15.47:59384) 2024-02-19T15:26:19.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd18fe00640 1 -- 172.21.15.47:0/4115093336 learned_addr learned my addr 172.21.15.47:0/4115093336 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:19.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd18f400640 1 --2- 172.21.15.47:0/4115093336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd19006c9a0 0x7fd190139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:19.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd187e00640 1 --2- 172.21.15.47:0/4115093336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd190071350 0x7fd1900ae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:19.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd18fe00640 1 -- 172.21.15.47:0/4115093336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd190071350 msgr2=0x7fd1900ae570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:19.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd18fe00640 1 --2- 172.21.15.47:0/4115093336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd190071350 0x7fd1900ae570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd18fe00640 1 -- 172.21.15.47:0/4115093336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd19006c9a0 msgr2=0x7fd190139220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:19.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd18fe00640 1 --2- 172.21.15.47:0/4115093336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd19006c9a0 0x7fd190139220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd18fe00640 1 -- 172.21.15.47:0/4115093336 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd180009420 con 0x7fd19015b1a0 2024-02-19T15:26:19.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd187e00640 1 --2- 172.21.15.47:0/4115093336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd190071350 0x7fd1900ae570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:19.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd18fe00640 1 --2- 172.21.15.47:0/4115093336 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd19015b1a0 0x7fd1900aeab0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fd1880af040 tx=0x7fd1880197a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:19.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd187400640 1 -- 172.21.15.47:0/4115093336 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1880b6680 con 0x7fd19015b1a0 2024-02-19T15:26:19.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd187400640 1 -- 172.21.15.47:0/4115093336 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1880b67e0 con 0x7fd19015b1a0 2024-02-19T15:26:19.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd187400640 1 -- 172.21.15.47:0/4115093336 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1880b6b70 con 0x7fd19015b1a0 2024-02-19T15:26:19.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd196000640 1 -- 172.21.15.47:0/4115093336 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd188009b30 con 0x7fd19015b1a0 2024-02-19T15:26:19.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd196000640 1 -- 172.21.15.47:0/4115093336 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd190154ab0 con 0x7fd19015b1a0 2024-02-19T15:26:19.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd196000640 1 -- 172.21.15.47:0/4115093336 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd19006d1b0 con 0x7fd19015b1a0 2024-02-19T15:26:19.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.452+0000 7fd187400640 1 -- 172.21.15.47:0/4115093336 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd1880b6cd0 con 0x7fd19015b1a0 2024-02-19T15:26:19.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.456+0000 7fd187400640 1 --2- 172.21.15.47:0/4115093336 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd16c066e40 0x7fd16c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:19.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.456+0000 7fd187400640 1 -- 172.21.15.47:0/4115093336 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd18811f7b0 con 0x7fd19015b1a0 2024-02-19T15:26:19.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.456+0000 7fd18f400640 1 --2- 172.21.15.47:0/4115093336 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd16c066e40 0x7fd16c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:19.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.456+0000 7fd18f400640 1 --2- 172.21.15.47:0/4115093336 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd16c066e40 0x7fd16c069300 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fd180009cf0 tx=0x7fd180009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:19.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.460+0000 7fd187400640 1 -- 172.21.15.47:0/4115093336 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd1880ec890 con 0x7fd19015b1a0 2024-02-19T15:26:19.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.620+0000 7fd196000640 1 -- 172.21.15.47:0/4115093336 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd190075360 con 0x7fd19015b1a0 2024-02-19T15:26:19.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.624+0000 7fd187400640 1 -- 172.21.15.47:0/4115093336 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+13033 (secure 0 0 0) 0x7fd1880eff50 con 0x7fd19015b1a0 2024-02-19T15:26:19.626 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:19.627 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":85,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:26:18.269734+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:6835/3598443511":"2024-02-20T15:26:15.259811+0000","172.21.15.47:6834/3598443511":"2024-02-20T15:26:15.259811+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:26:19.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.624+0000 7fd196000640 1 -- 172.21.15.47:0/4115093336 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd16c066e40 msgr2=0x7fd16c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:19.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.624+0000 7fd196000640 1 --2- 172.21.15.47:0/4115093336 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd16c066e40 0x7fd16c069300 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fd180009cf0 tx=0x7fd180009190 comp rx=0 tx=0).stop 2024-02-19T15:26:19.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.624+0000 7fd196000640 1 -- 172.21.15.47:0/4115093336 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd19015b1a0 msgr2=0x7fd1900aeab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:19.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.624+0000 7fd196000640 1 --2- 172.21.15.47:0/4115093336 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd19015b1a0 0x7fd1900aeab0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fd1880af040 tx=0x7fd1880197a0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.624+0000 7fd196000640 1 -- 172.21.15.47:0/4115093336 shutdown_connections 2024-02-19T15:26:19.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.624+0000 7fd196000640 1 --2- 172.21.15.47:0/4115093336 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd16c066e40 0x7fd16c069300 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.624+0000 7fd196000640 1 --2- 172.21.15.47:0/4115093336 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd19015b1a0 0x7fd1900aeab0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.624+0000 7fd196000640 1 --2- 172.21.15.47:0/4115093336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd190071350 0x7fd1900ae570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.624+0000 7fd196000640 1 --2- 172.21.15.47:0/4115093336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd19006c9a0 0x7fd190139220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.624+0000 7fd196000640 1 -- 172.21.15.47:0/4115093336 >> 172.21.15.47:0/4115093336 conn(0x7fd1900b16a0 msgr2=0x7fd190070b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:19.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.624+0000 7fd196000640 1 -- 172.21.15.47:0/4115093336 shutdown_connections 2024-02-19T15:26:19.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.624+0000 7fd196000640 1 -- 172.21.15.47:0/4115093336 wait complete. 2024-02-19T15:26:19.649 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:26:19.649 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:26:19.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.788+0000 7fa57b600640 1 Processor -- start 2024-02-19T15:26:19.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.788+0000 7fa57b600640 1 -- start start 2024-02-19T15:26:19.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.792+0000 7fa57b600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa57415af10 0x7fa57406e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:19.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.792+0000 7fa57b600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa5740ca8f0 0x7fa5740cad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:19.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.792+0000 7fa57b600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa574071c70 con 0x7fa5740cb290 2024-02-19T15:26:19.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.792+0000 7fa57b600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa57406e8a0 con 0x7fa57415af10 2024-02-19T15:26:19.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.792+0000 7fa57b600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa57406cef0 con 0x7fa5740ca8f0 2024-02-19T15:26:19.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.792+0000 7fa578c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa57415af10 0x7fa57406e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:19.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.792+0000 7fa578c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa57415af10 0x7fa57406e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59400/0 (socket says 172.21.15.47:59400) 2024-02-19T15:26:19.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.792+0000 7fa578c00640 1 -- 172.21.15.47:0/2964226013 learned_addr learned my addr 172.21.15.47:0/2964226013 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:19.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.792+0000 7fa573e00640 1 --2- 172.21.15.47:0/2964226013 >> v2:172.21.15.116:3300/0 conn(0x7fa5740ca8f0 0x7fa5740cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:19.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.792+0000 7fa578c00640 1 -- 172.21.15.47:0/2964226013 >> v2:172.21.15.116:3300/0 conn(0x7fa5740ca8f0 msgr2=0x7fa5740cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:19.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.792+0000 7fa578c00640 1 --2- 172.21.15.47:0/2964226013 >> v2:172.21.15.116:3300/0 conn(0x7fa5740ca8f0 0x7fa5740cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.792+0000 7fa578c00640 1 -- 172.21.15.47:0/2964226013 >> v1:172.21.15.116:6789/0 conn(0x7fa5740cb290 legacy=0x7fa57415eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:19.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.792+0000 7fa578c00640 1 -- 172.21.15.47:0/2964226013 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5740ca610 con 0x7fa57415af10 2024-02-19T15:26:19.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.792+0000 7fa573e00640 1 --2- 172.21.15.47:0/2964226013 >> v2:172.21.15.116:3300/0 conn(0x7fa5740ca8f0 0x7fa5740cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:19.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.792+0000 7fa578c00640 1 --2- 172.21.15.47:0/2964226013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa57415af10 0x7fa57406e360 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fa5740b27b0 tx=0x7fa56801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e5eedc5db1f7b3a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:19.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa573400640 1 -- 172.21.15.47:0/2964226013 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5680ab620 con 0x7fa57415af10 2024-02-19T15:26:19.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa573400640 1 -- 172.21.15.47:0/2964226013 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5680ab780 con 0x7fa57415af10 2024-02-19T15:26:19.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa573400640 1 -- 172.21.15.47:0/2964226013 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5680abb10 con 0x7fa57415af10 2024-02-19T15:26:19.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa57b600640 1 -- 172.21.15.47:0/2964226013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa57415af10 msgr2=0x7fa57406e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:19.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa57b600640 1 --2- 172.21.15.47:0/2964226013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa57415af10 0x7fa57406e360 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fa5740b27b0 tx=0x7fa56801c420 comp rx=0 tx=0).stop 2024-02-19T15:26:19.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa57b600640 1 -- 172.21.15.47:0/2964226013 shutdown_connections 2024-02-19T15:26:19.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa57b600640 1 --2- 172.21.15.47:0/2964226013 >> v2:172.21.15.116:3300/0 conn(0x7fa5740ca8f0 0x7fa5740cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa57b600640 1 --2- 172.21.15.47:0/2964226013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa57415af10 0x7fa57406e360 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa57b600640 1 -- 172.21.15.47:0/2964226013 >> 172.21.15.47:0/2964226013 conn(0x7fa5740cfc20 msgr2=0x7fa5740d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:19.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa57b600640 1 -- 172.21.15.47:0/2964226013 shutdown_connections 2024-02-19T15:26:19.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa57b600640 1 -- 172.21.15.47:0/2964226013 wait complete. 2024-02-19T15:26:19.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa57b600640 1 Processor -- start 2024-02-19T15:26:19.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa57b600640 1 -- start start 2024-02-19T15:26:19.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa57b600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5740ca8f0 0x7fa57415de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:19.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa57b600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa5740cb290 0x7fa5740c9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:19.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa57b600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa57415af10 0x7fa574155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:19.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa57b600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57415e4d0 con 0x7fa57415af10 2024-02-19T15:26:19.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa57b600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5741567a0 con 0x7fa5740ca8f0 2024-02-19T15:26:19.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa57b600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5741561e0 con 0x7fa5740cb290 2024-02-19T15:26:19.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa579600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa57415af10 0x7fa574155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:19.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa579600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa57415af10 0x7fa574155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59408/0 (socket says 172.21.15.47:59408) 2024-02-19T15:26:19.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa579600640 1 -- 172.21.15.47:0/1987124141 learned_addr learned my addr 172.21.15.47:0/1987124141 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:19.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa578c00640 1 --2- 172.21.15.47:0/1987124141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5740ca8f0 0x7fa57415de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:19.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.796+0000 7fa573e00640 1 --2- 172.21.15.47:0/1987124141 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa5740cb290 0x7fa5740c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:19.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.800+0000 7fa579600640 1 -- 172.21.15.47:0/1987124141 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa5740cb290 msgr2=0x7fa5740c9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:19.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.800+0000 7fa579600640 1 --2- 172.21.15.47:0/1987124141 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa5740cb290 0x7fa5740c9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.800+0000 7fa579600640 1 -- 172.21.15.47:0/1987124141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5740ca8f0 msgr2=0x7fa57415de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:19.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.800+0000 7fa579600640 1 --2- 172.21.15.47:0/1987124141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5740ca8f0 0x7fa57415de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.800+0000 7fa579600640 1 -- 172.21.15.47:0/1987124141 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5680094d0 con 0x7fa57415af10 2024-02-19T15:26:19.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.800+0000 7fa578c00640 1 --2- 172.21.15.47:0/1987124141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5740ca8f0 0x7fa57415de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:19.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.800+0000 7fa579600640 1 --2- 172.21.15.47:0/1987124141 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa57415af10 0x7fa574155ca0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00b960 tx=0x7fa56c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:19.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.800+0000 7fa570c00640 1 -- 172.21.15.47:0/1987124141 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c00cb20 con 0x7fa57415af10 2024-02-19T15:26:19.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.800+0000 7fa570c00640 1 -- 172.21.15.47:0/1987124141 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56c00cc80 con 0x7fa57415af10 2024-02-19T15:26:19.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.800+0000 7fa570c00640 1 -- 172.21.15.47:0/1987124141 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c014440 con 0x7fa57415af10 2024-02-19T15:26:19.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.800+0000 7fa57b600640 1 -- 172.21.15.47:0/1987124141 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa57415c820 con 0x7fa57415af10 2024-02-19T15:26:19.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.800+0000 7fa57b600640 1 -- 172.21.15.47:0/1987124141 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57415cde0 con 0x7fa57415af10 2024-02-19T15:26:19.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.800+0000 7fa57b600640 1 -- 172.21.15.47:0/1987124141 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa574075360 con 0x7fa57415af10 2024-02-19T15:26:19.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.800+0000 7fa570c00640 1 -- 172.21.15.47:0/1987124141 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa56c01d3b0 con 0x7fa57415af10 2024-02-19T15:26:19.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.800+0000 7fa570c00640 1 --2- 172.21.15.47:0/1987124141 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa554066e40 0x7fa554069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:19.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.800+0000 7fa570c00640 1 -- 172.21.15.47:0/1987124141 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa56c084520 con 0x7fa57415af10 2024-02-19T15:26:19.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.804+0000 7fa578c00640 1 --2- 172.21.15.47:0/1987124141 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa554066e40 0x7fa554069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:19.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.804+0000 7fa578c00640 1 --2- 172.21.15.47:0/1987124141 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa554066e40 0x7fa554069300 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa5680af000 tx=0x7fa5680b6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:19.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.804+0000 7fa570c00640 1 -- 172.21.15.47:0/1987124141 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa56c0516b0 con 0x7fa57415af10 2024-02-19T15:26:19.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.968+0000 7fa57b600640 1 -- 172.21.15.47:0/1987124141 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa5740d0740 con 0x7fa57415af10 2024-02-19T15:26:19.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.968+0000 7fa570c00640 1 -- 172.21.15.47:0/1987124141 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa56c0190a0 con 0x7fa57415af10 2024-02-19T15:26:19.974 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:26:19.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.972+0000 7fa57b600640 1 -- 172.21.15.47:0/1987124141 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa554066e40 msgr2=0x7fa554069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:19.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.972+0000 7fa57b600640 1 --2- 172.21.15.47:0/1987124141 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa554066e40 0x7fa554069300 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa5680af000 tx=0x7fa5680b6040 comp rx=0 tx=0).stop 2024-02-19T15:26:19.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.972+0000 7fa57b600640 1 -- 172.21.15.47:0/1987124141 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa57415af10 msgr2=0x7fa574155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:19.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.972+0000 7fa57b600640 1 --2- 172.21.15.47:0/1987124141 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa57415af10 0x7fa574155ca0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00b960 tx=0x7fa56c00be20 comp rx=0 tx=0).stop 2024-02-19T15:26:19.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.972+0000 7fa57b600640 1 -- 172.21.15.47:0/1987124141 shutdown_connections 2024-02-19T15:26:19.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.972+0000 7fa57b600640 1 --2- 172.21.15.47:0/1987124141 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa554066e40 0x7fa554069300 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.972+0000 7fa57b600640 1 --2- 172.21.15.47:0/1987124141 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa57415af10 0x7fa574155ca0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.972+0000 7fa57b600640 1 --2- 172.21.15.47:0/1987124141 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa5740cb290 0x7fa5740c9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.972+0000 7fa57b600640 1 --2- 172.21.15.47:0/1987124141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5740ca8f0 0x7fa57415de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:19.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.972+0000 7fa57b600640 1 -- 172.21.15.47:0/1987124141 >> 172.21.15.47:0/1987124141 conn(0x7fa5740cfc20 msgr2=0x7fa5740b4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:19.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.972+0000 7fa57b600640 1 -- 172.21.15.47:0/1987124141 shutdown_connections 2024-02-19T15:26:19.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:19.972+0000 7fa57b600640 1 -- 172.21.15.47:0/1987124141 wait complete. 2024-02-19T15:26:19.995 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-02-19T15:26:20.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.132+0000 7fa295400640 1 Processor -- start 2024-02-19T15:26:20.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.132+0000 7fa295400640 1 -- start start 2024-02-19T15:26:20.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.132+0000 7fa295400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa29015be90 0x7fa2900d0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:20.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.132+0000 7fa295400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa2900d26b0 0x7fa2900ce460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:20.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.132+0000 7fa295400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2900ce9a0 0x7fa2900cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:20.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.132+0000 7fa295400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2900cedf0 con 0x7fa29015be90 2024-02-19T15:26:20.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.132+0000 7fa295400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2900cdae0 con 0x7fa2900d26b0 2024-02-19T15:26:20.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.132+0000 7fa295400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2900d0af0 con 0x7fa2900ce9a0 2024-02-19T15:26:20.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.132+0000 7fa28e800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa29015be90 0x7fa2900d0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:20.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.132+0000 7fa28e800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa29015be90 0x7fa2900d0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59412/0 (socket says 172.21.15.47:59412) 2024-02-19T15:26:20.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.132+0000 7fa28e800640 1 -- 172.21.15.47:0/820045419 learned_addr learned my addr 172.21.15.47:0/820045419 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:20.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.132+0000 7fa285e00640 1 --2- 172.21.15.47:0/820045419 >> v2:172.21.15.116:3300/0 conn(0x7fa2900d26b0 0x7fa2900ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:20.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.132+0000 7fa28f200640 1 --2- 172.21.15.47:0/820045419 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2900ce9a0 0x7fa2900cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:20.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.132+0000 7fa28e800640 1 -- 172.21.15.47:0/820045419 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2900ce9a0 msgr2=0x7fa2900cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:20.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.132+0000 7fa28e800640 1 --2- 172.21.15.47:0/820045419 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2900ce9a0 0x7fa2900cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:20.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.132+0000 7fa28e800640 1 -- 172.21.15.47:0/820045419 >> v2:172.21.15.116:3300/0 conn(0x7fa2900d26b0 msgr2=0x7fa2900ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:20.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.132+0000 7fa28e800640 1 --2- 172.21.15.47:0/820045419 >> v2:172.21.15.116:3300/0 conn(0x7fa2900d26b0 0x7fa2900ce460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:20.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.132+0000 7fa28e800640 1 -- 172.21.15.47:0/820045419 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2900d0810 con 0x7fa29015be90 2024-02-19T15:26:20.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.132+0000 7fa28f200640 1 --2- 172.21.15.47:0/820045419 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2900ce9a0 0x7fa2900cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:20.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.136+0000 7fa28e800640 1 --2- 172.21.15.47:0/820045419 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa29015be90 0x7fa2900d0170 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fa290071fe0 tx=0x7fa28001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a6eeccf673eff3f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:20.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.136+0000 7fa28de00640 1 -- 172.21.15.47:0/820045419 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2800a57b0 con 0x7fa29015be90 2024-02-19T15:26:20.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.136+0000 7fa28de00640 1 -- 172.21.15.47:0/820045419 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2800a5910 con 0x7fa29015be90 2024-02-19T15:26:20.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.136+0000 7fa28de00640 1 -- 172.21.15.47:0/820045419 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2800a5ca0 con 0x7fa29015be90 2024-02-19T15:26:20.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.136+0000 7fa295400640 1 -- 172.21.15.47:0/820045419 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa29015be90 msgr2=0x7fa2900d0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:20.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.136+0000 7fa295400640 1 --2- 172.21.15.47:0/820045419 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa29015be90 0x7fa2900d0170 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fa290071fe0 tx=0x7fa28001c930 comp rx=0 tx=0).stop 2024-02-19T15:26:20.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.136+0000 7fa295400640 1 -- 172.21.15.47:0/820045419 shutdown_connections 2024-02-19T15:26:20.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.136+0000 7fa295400640 1 --2- 172.21.15.47:0/820045419 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2900ce9a0 0x7fa2900cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:20.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.136+0000 7fa295400640 1 --2- 172.21.15.47:0/820045419 >> v2:172.21.15.116:3300/0 conn(0x7fa2900d26b0 0x7fa2900ce460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:20.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.136+0000 7fa295400640 1 --2- 172.21.15.47:0/820045419 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa29015be90 0x7fa2900d0170 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:20.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.136+0000 7fa295400640 1 -- 172.21.15.47:0/820045419 >> 172.21.15.47:0/820045419 conn(0x7fa2900d1b90 msgr2=0x7fa2900d1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:20.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.136+0000 7fa295400640 1 -- 172.21.15.47:0/820045419 shutdown_connections 2024-02-19T15:26:20.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.136+0000 7fa295400640 1 -- 172.21.15.47:0/820045419 wait complete. 2024-02-19T15:26:20.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa295400640 1 Processor -- start 2024-02-19T15:26:20.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa295400640 1 -- start start 2024-02-19T15:26:20.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa295400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2900ce9a0 0x7fa29014c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:20.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa295400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2900d26b0 0x7fa290152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:20.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa295400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa29015be90 0x7fa290155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:20.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa295400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa290154b10 con 0x7fa29015be90 2024-02-19T15:26:20.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa295400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa290155970 con 0x7fa2900ce9a0 2024-02-19T15:26:20.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa295400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29014c920 con 0x7fa2900d26b0 2024-02-19T15:26:20.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa28f200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa29015be90 0x7fa290155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:20.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa28e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2900ce9a0 0x7fa29014c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:20.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa28f200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa29015be90 0x7fa290155e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59424/0 (socket says 172.21.15.47:59424) 2024-02-19T15:26:20.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa28f200640 1 -- 172.21.15.47:0/3278875484 learned_addr learned my addr 172.21.15.47:0/3278875484 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:20.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa285e00640 1 --2- 172.21.15.47:0/3278875484 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2900d26b0 0x7fa290152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:20.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa28f200640 1 -- 172.21.15.47:0/3278875484 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2900d26b0 msgr2=0x7fa290152c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:20.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa28f200640 1 --2- 172.21.15.47:0/3278875484 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2900d26b0 0x7fa290152c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:20.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa28f200640 1 -- 172.21.15.47:0/3278875484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2900ce9a0 msgr2=0x7fa29014c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:20.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa28f200640 1 --2- 172.21.15.47:0/3278875484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2900ce9a0 0x7fa29014c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:20.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa28f200640 1 -- 172.21.15.47:0/3278875484 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa280009b30 con 0x7fa29015be90 2024-02-19T15:26:20.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa28e800640 1 --2- 172.21.15.47:0/3278875484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2900ce9a0 0x7fa29014c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:20.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa28f200640 1 --2- 172.21.15.47:0/3278875484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa29015be90 0x7fa290155e00 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fa28800c820 tx=0x7fa28800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:20.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa287400640 1 -- 172.21.15.47:0/3278875484 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa288007a30 con 0x7fa29015be90 2024-02-19T15:26:20.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa287400640 1 -- 172.21.15.47:0/3278875484 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa288007b90 con 0x7fa29015be90 2024-02-19T15:26:20.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa287400640 1 -- 172.21.15.47:0/3278875484 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28800ce10 con 0x7fa29015be90 2024-02-19T15:26:20.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa295400640 1 -- 172.21.15.47:0/3278875484 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa290154d60 con 0x7fa29015be90 2024-02-19T15:26:20.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.140+0000 7fa295400640 1 -- 172.21.15.47:0/3278875484 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa290155320 con 0x7fa29015be90 2024-02-19T15:26:20.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.144+0000 7fa295400640 1 -- 172.21.15.47:0/3278875484 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa290068500 con 0x7fa29015be90 2024-02-19T15:26:20.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.144+0000 7fa287400640 1 -- 172.21.15.47:0/3278875484 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa2880113f0 con 0x7fa29015be90 2024-02-19T15:26:20.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.144+0000 7fa287400640 1 --2- 172.21.15.47:0/3278875484 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa26c066e70 0x7fa26c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:20.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.144+0000 7fa287400640 1 -- 172.21.15.47:0/3278875484 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa288084820 con 0x7fa29015be90 2024-02-19T15:26:20.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.144+0000 7fa28e800640 1 --2- 172.21.15.47:0/3278875484 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa26c066e70 0x7fa26c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:20.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.144+0000 7fa28e800640 1 --2- 172.21.15.47:0/3278875484 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa26c066e70 0x7fa26c069330 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa2800ae000 tx=0x7fa2800a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:20.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.148+0000 7fa287400640 1 -- 172.21.15.47:0/3278875484 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa2880519e0 con 0x7fa29015be90 2024-02-19T15:26:20.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:20.368+0000 7fa295400640 1 -- 172.21.15.47:0/3278875484 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7fa2900d1170 con 0x7fa29015be90 2024-02-19T15:26:21.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.284+0000 7fa287400640 1 -- 172.21.15.47:0/3278875484 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v248) v1 ==== 137+0+0 (secure 0 0 0) 0x7fa2880550a0 con 0x7fa29015be90 2024-02-19T15:26:21.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.284+0000 7fa295400640 1 -- 172.21.15.47:0/3278875484 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa26c066e70 msgr2=0x7fa26c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:21.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.284+0000 7fa295400640 1 --2- 172.21.15.47:0/3278875484 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa26c066e70 0x7fa26c069330 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa2800ae000 tx=0x7fa2800a7070 comp rx=0 tx=0).stop 2024-02-19T15:26:21.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.284+0000 7fa295400640 1 -- 172.21.15.47:0/3278875484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa29015be90 msgr2=0x7fa290155e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:21.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.284+0000 7fa295400640 1 --2- 172.21.15.47:0/3278875484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa29015be90 0x7fa290155e00 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fa28800c820 tx=0x7fa28800cce0 comp rx=0 tx=0).stop 2024-02-19T15:26:21.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.284+0000 7fa295400640 1 -- 172.21.15.47:0/3278875484 shutdown_connections 2024-02-19T15:26:21.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.284+0000 7fa295400640 1 --2- 172.21.15.47:0/3278875484 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa26c066e70 0x7fa26c069330 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:21.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.284+0000 7fa295400640 1 --2- 172.21.15.47:0/3278875484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa29015be90 0x7fa290155e00 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:21.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.284+0000 7fa295400640 1 --2- 172.21.15.47:0/3278875484 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2900d26b0 0x7fa290152c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:21.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.284+0000 7fa295400640 1 --2- 172.21.15.47:0/3278875484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2900ce9a0 0x7fa29014c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:21.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.284+0000 7fa295400640 1 -- 172.21.15.47:0/3278875484 >> 172.21.15.47:0/3278875484 conn(0x7fa2900d1b90 msgr2=0x7fa29006eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:21.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.284+0000 7fa295400640 1 -- 172.21.15.47:0/3278875484 shutdown_connections 2024-02-19T15:26:21.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.288+0000 7fa295400640 1 -- 172.21.15.47:0/3278875484 wait complete. 2024-02-19T15:26:21.313 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-19T15:26:21.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.456+0000 7f5153e00640 1 Processor -- start 2024-02-19T15:26:21.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.456+0000 7f5153e00640 1 -- start start 2024-02-19T15:26:21.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.460+0000 7f5153e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f514c15bd90 0x7f514c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:21.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.460+0000 7f5153e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f514c0cb490 0x7f514c072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:21.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.460+0000 7f5153e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f514c072680 0x7f514c0ca270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:21.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.460+0000 7f5153e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f514c15fc90 con 0x7f514c072680 2024-02-19T15:26:21.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.460+0000 7f5153e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f514c15ff70 con 0x7f514c15bd90 2024-02-19T15:26:21.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.460+0000 7f5153e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514c160250 con 0x7f514c0cb490 2024-02-19T15:26:21.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.460+0000 7f5151e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f514c072680 0x7f514c0ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:21.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.460+0000 7f5151e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f514c072680 0x7f514c0ca270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59436/0 (socket says 172.21.15.47:59436) 2024-02-19T15:26:21.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.460+0000 7f5151e00640 1 -- 172.21.15.47:0/1931239110 learned_addr learned my addr 172.21.15.47:0/1931239110 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:21.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.460+0000 7f5150a00640 1 --2- 172.21.15.47:0/1931239110 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f514c0cb490 0x7f514c072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:21.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.460+0000 7f5151400640 1 --2- 172.21.15.47:0/1931239110 >> v2:172.21.15.116:3300/0 conn(0x7f514c15bd90 0x7f514c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:21.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.460+0000 7f5151e00640 1 -- 172.21.15.47:0/1931239110 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f514c0cb490 msgr2=0x7f514c072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:21.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.460+0000 7f5151e00640 1 --2- 172.21.15.47:0/1931239110 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f514c0cb490 0x7f514c072140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:21.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.460+0000 7f5151e00640 1 -- 172.21.15.47:0/1931239110 >> v2:172.21.15.116:3300/0 conn(0x7f514c15bd90 msgr2=0x7f514c070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:21.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.460+0000 7f5151e00640 1 --2- 172.21.15.47:0/1931239110 >> v2:172.21.15.116:3300/0 conn(0x7f514c15bd90 0x7f514c070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:21.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.460+0000 7f5151e00640 1 -- 172.21.15.47:0/1931239110 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c160530 con 0x7f514c072680 2024-02-19T15:26:21.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.464+0000 7f5151e00640 1 --2- 172.21.15.47:0/1931239110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f514c072680 0x7f514c0ca270 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f514c06c300 tx=0x7f514401c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7385e336e7fde72 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:21.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.464+0000 7f514be00640 1 -- 172.21.15.47:0/1931239110 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51440a8410 con 0x7f514c072680 2024-02-19T15:26:21.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.464+0000 7f514be00640 1 -- 172.21.15.47:0/1931239110 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51440a8570 con 0x7f514c072680 2024-02-19T15:26:21.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.464+0000 7f514be00640 1 -- 172.21.15.47:0/1931239110 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51440a8900 con 0x7f514c072680 2024-02-19T15:26:21.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.464+0000 7f5153e00640 1 -- 172.21.15.47:0/1931239110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f514c072680 msgr2=0x7f514c0ca270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:21.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.464+0000 7f5153e00640 1 --2- 172.21.15.47:0/1931239110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f514c072680 0x7f514c0ca270 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f514c06c300 tx=0x7f514401c5d0 comp rx=0 tx=0).stop 2024-02-19T15:26:21.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.464+0000 7f5153e00640 1 -- 172.21.15.47:0/1931239110 shutdown_connections 2024-02-19T15:26:21.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.464+0000 7f5153e00640 1 --2- 172.21.15.47:0/1931239110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f514c072680 0x7f514c0ca270 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:21.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.464+0000 7f5153e00640 1 --2- 172.21.15.47:0/1931239110 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f514c0cb490 0x7f514c072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:21.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.464+0000 7f5153e00640 1 --2- 172.21.15.47:0/1931239110 >> v2:172.21.15.116:3300/0 conn(0x7f514c15bd90 0x7f514c070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:21.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.464+0000 7f5153e00640 1 -- 172.21.15.47:0/1931239110 >> 172.21.15.47:0/1931239110 conn(0x7f514c0cfc20 msgr2=0x7f514c0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:21.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.464+0000 7f5153e00640 1 -- 172.21.15.47:0/1931239110 shutdown_connections 2024-02-19T15:26:21.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.464+0000 7f5153e00640 1 -- 172.21.15.47:0/1931239110 wait complete. 2024-02-19T15:26:21.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.464+0000 7f5153e00640 1 Processor -- start 2024-02-19T15:26:21.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.464+0000 7f5153e00640 1 -- start start 2024-02-19T15:26:21.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5153e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f514c072680 0x7f514c15c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:21.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5153e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f514c0cb490 0x7f514c144d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:21.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5153e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f514c15bd90 0x7f514c15dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:21.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5153e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c15cbd0 con 0x7f514c072680 2024-02-19T15:26:21.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5153e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c1579c0 con 0x7f514c0cb490 2024-02-19T15:26:21.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5153e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514c157090 con 0x7f514c15bd90 2024-02-19T15:26:21.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5151400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f514c072680 0x7f514c15c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:21.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5151400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f514c072680 0x7f514c15c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59440/0 (socket says 172.21.15.47:59440) 2024-02-19T15:26:21.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5151400640 1 -- 172.21.15.47:0/3342766130 learned_addr learned my addr 172.21.15.47:0/3342766130 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:21.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5150a00640 1 --2- 172.21.15.47:0/3342766130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f514c0cb490 0x7f514c144d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:21.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5151e00640 1 --2- 172.21.15.47:0/3342766130 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f514c15bd90 0x7f514c15dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:21.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5151400640 1 -- 172.21.15.47:0/3342766130 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f514c15bd90 msgr2=0x7f514c15dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:21.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5151400640 1 --2- 172.21.15.47:0/3342766130 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f514c15bd90 0x7f514c15dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:21.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5151400640 1 -- 172.21.15.47:0/3342766130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f514c0cb490 msgr2=0x7f514c144d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:21.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5151400640 1 --2- 172.21.15.47:0/3342766130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f514c0cb490 0x7f514c144d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:21.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5151400640 1 -- 172.21.15.47:0/3342766130 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5144009b30 con 0x7f514c072680 2024-02-19T15:26:21.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5151400640 1 --2- 172.21.15.47:0/3342766130 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f514c072680 0x7f514c15c380 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f514000d5f0 tx=0x7f514000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:21.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5149600640 1 -- 172.21.15.47:0/3342766130 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51400125f0 con 0x7f514c072680 2024-02-19T15:26:21.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5149600640 1 -- 172.21.15.47:0/3342766130 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5140012750 con 0x7f514c072680 2024-02-19T15:26:21.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5149600640 1 -- 172.21.15.47:0/3342766130 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5140012ae0 con 0x7f514c072680 2024-02-19T15:26:21.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5153e00640 1 -- 172.21.15.47:0/3342766130 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f514c157370 con 0x7f514c072680 2024-02-19T15:26:21.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5153e00640 1 -- 172.21.15.47:0/3342766130 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f514c15c8c0 con 0x7f514c072680 2024-02-19T15:26:21.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5153e00640 1 -- 172.21.15.47:0/3342766130 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f514c075360 con 0x7f514c072680 2024-02-19T15:26:21.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5149600640 1 -- 172.21.15.47:0/3342766130 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5140012c40 con 0x7f514c072680 2024-02-19T15:26:21.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.468+0000 7f5149600640 1 --2- 172.21.15.47:0/3342766130 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f512c066e70 0x7f512c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:21.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.472+0000 7f5149600640 1 -- 172.21.15.47:0/3342766130 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5140084cc0 con 0x7f514c072680 2024-02-19T15:26:21.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.472+0000 7f5150a00640 1 --2- 172.21.15.47:0/3342766130 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f512c066e70 0x7f512c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:21.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.472+0000 7f5150a00640 1 --2- 172.21.15.47:0/3342766130 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f512c066e70 0x7f512c069330 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f514c166990 tx=0x7f5138008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:21.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.472+0000 7f5149600640 1 -- 172.21.15.47:0/3342766130 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f514008b080 con 0x7f514c072680 2024-02-19T15:26:21.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.636+0000 7f5153e00640 1 -- 172.21.15.47:0/3342766130 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f514c06e800 con 0x7f514c072680 2024-02-19T15:26:21.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.640+0000 7f5149600640 1 -- 172.21.15.47:0/3342766130 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5140051d10 con 0x7f514c072680 2024-02-19T15:26:21.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.640+0000 7f5153e00640 1 -- 172.21.15.47:0/3342766130 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f512c066e70 msgr2=0x7f512c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:21.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.640+0000 7f5153e00640 1 --2- 172.21.15.47:0/3342766130 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f512c066e70 0x7f512c069330 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f514c166990 tx=0x7f5138008040 comp rx=0 tx=0).stop 2024-02-19T15:26:21.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.640+0000 7f5153e00640 1 -- 172.21.15.47:0/3342766130 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f514c072680 msgr2=0x7f514c15c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:21.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.640+0000 7f5153e00640 1 --2- 172.21.15.47:0/3342766130 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f514c072680 0x7f514c15c380 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f514000d5f0 tx=0x7f514000dab0 comp rx=0 tx=0).stop 2024-02-19T15:26:21.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.640+0000 7f5153e00640 1 -- 172.21.15.47:0/3342766130 shutdown_connections 2024-02-19T15:26:21.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.640+0000 7f5153e00640 1 --2- 172.21.15.47:0/3342766130 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f512c066e70 0x7f512c069330 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:21.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.640+0000 7f5153e00640 1 --2- 172.21.15.47:0/3342766130 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f514c15bd90 0x7f514c15dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:21.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.640+0000 7f5153e00640 1 --2- 172.21.15.47:0/3342766130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f514c0cb490 0x7f514c144d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:21.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.640+0000 7f5153e00640 1 --2- 172.21.15.47:0/3342766130 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f514c072680 0x7f514c15c380 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:21.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.640+0000 7f5153e00640 1 -- 172.21.15.47:0/3342766130 >> 172.21.15.47:0/3342766130 conn(0x7f514c0cfc20 msgr2=0x7f514c0cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:21.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.640+0000 7f5153e00640 1 -- 172.21.15.47:0/3342766130 shutdown_connections 2024-02-19T15:26:21.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:21.640+0000 7f5153e00640 1 -- 172.21.15.47:0/3342766130 wait complete. 2024-02-19T15:26:21.666 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T15:26:21.666 DEBUG:teuthology.orchestra.run.smithi047:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:26:21.725 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:26:21.789 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T15:26:21.790 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:26:21.790 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:26:21.798 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:26:21.799 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:26:21.799 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:26:21.808 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:26:21.808 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:26:21.808 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:26:21.850 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:26:21.850 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:26:21.851 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:26:21.859 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:26:21.859 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-02-19T15:26:21.859 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:26:22.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.028+0000 7f1b37400640 1 Processor -- start 2024-02-19T15:26:22.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.032+0000 7f1b37400640 1 -- start start 2024-02-19T15:26:22.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.032+0000 7f1b37400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1b3015bcd0 0x7f1b300cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:22.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.032+0000 7f1b37400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b300d0360 0x7f1b3015d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:22.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.032+0000 7f1b37400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b300cc700 con 0x7f1b300cedb0 2024-02-19T15:26:22.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.032+0000 7f1b37400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b300cf5e0 con 0x7f1b3015bcd0 2024-02-19T15:26:22.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.032+0000 7f1b37400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b300cf300 con 0x7f1b300d0360 2024-02-19T15:26:22.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.032+0000 7f1b34a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1b3015bcd0 0x7f1b300cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:22.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.032+0000 7f1b34a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1b3015bcd0 0x7f1b300cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59452/0 (socket says 172.21.15.47:59452) 2024-02-19T15:26:22.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.032+0000 7f1b34a00640 1 -- 172.21.15.47:0/3945181580 learned_addr learned my addr 172.21.15.47:0/3945181580 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:22.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.032+0000 7f1b34a00640 1 -- 172.21.15.47:0/3945181580 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b300d0360 msgr2=0x7f1b3015d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:22.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.032+0000 7f1b34a00640 1 --2- 172.21.15.47:0/3945181580 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b300d0360 0x7f1b3015d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:22.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.032+0000 7f1b34a00640 1 -- 172.21.15.47:0/3945181580 >> v1:172.21.15.116:6789/0 conn(0x7f1b300cedb0 legacy=0x7f1b300cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:22.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.032+0000 7f1b34a00640 1 -- 172.21.15.47:0/3945181580 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b300cc9a0 con 0x7f1b3015bcd0 2024-02-19T15:26:22.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.036+0000 7f1b34a00640 1 --2- 172.21.15.47:0/3945181580 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3015bcd0 0x7f1b300cd480 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f1b20009870 tx=0x7f1b2001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b39a7acbf780a87 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:22.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.036+0000 7f1b2fe00640 1 -- 172.21.15.47:0/3945181580 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b200aa430 con 0x7f1b3015bcd0 2024-02-19T15:26:22.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.036+0000 7f1b2fe00640 1 -- 172.21.15.47:0/3945181580 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b200aa590 con 0x7f1b3015bcd0 2024-02-19T15:26:22.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.036+0000 7f1b2fe00640 1 -- 172.21.15.47:0/3945181580 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b200aa920 con 0x7f1b3015bcd0 2024-02-19T15:26:22.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.036+0000 7f1b37400640 1 -- 172.21.15.47:0/3945181580 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3015bcd0 msgr2=0x7f1b300cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:22.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.036+0000 7f1b37400640 1 --2- 172.21.15.47:0/3945181580 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3015bcd0 0x7f1b300cd480 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f1b20009870 tx=0x7f1b2001c510 comp rx=0 tx=0).stop 2024-02-19T15:26:22.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.036+0000 7f1b37400640 1 -- 172.21.15.47:0/3945181580 shutdown_connections 2024-02-19T15:26:22.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.036+0000 7f1b37400640 1 --2- 172.21.15.47:0/3945181580 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b300d0360 0x7f1b3015d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:22.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.036+0000 7f1b37400640 1 --2- 172.21.15.47:0/3945181580 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3015bcd0 0x7f1b300cd480 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:22.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.036+0000 7f1b37400640 1 -- 172.21.15.47:0/3945181580 >> 172.21.15.47:0/3945181580 conn(0x7f1b300d1bf0 msgr2=0x7f1b300d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:22.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.036+0000 7f1b37400640 1 -- 172.21.15.47:0/3945181580 shutdown_connections 2024-02-19T15:26:22.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.036+0000 7f1b37400640 1 -- 172.21.15.47:0/3945181580 wait complete. 2024-02-19T15:26:22.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.036+0000 7f1b37400640 1 Processor -- start 2024-02-19T15:26:22.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.036+0000 7f1b37400640 1 -- start start 2024-02-19T15:26:22.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b37400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b300cedb0 0x7f1b301f7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:22.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b37400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b300d0360 0x7f1b301f73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:22.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b37400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3015bcd0 0x7f1b300c7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:22.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b37400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b300c8570 con 0x7f1b3015bcd0 2024-02-19T15:26:22.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b37400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b300c5ce0 con 0x7f1b300d0360 2024-02-19T15:26:22.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b37400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b300c53b0 con 0x7f1b300cedb0 2024-02-19T15:26:22.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b35400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3015bcd0 0x7f1b300c7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:22.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b35400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3015bcd0 0x7f1b300c7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59460/0 (socket says 172.21.15.47:59460) 2024-02-19T15:26:22.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b35400640 1 -- 172.21.15.47:0/1828343898 learned_addr learned my addr 172.21.15.47:0/1828343898 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:22.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b34a00640 1 --2- 172.21.15.47:0/1828343898 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b300cedb0 0x7f1b301f7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:22.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b35400640 1 -- 172.21.15.47:0/1828343898 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b300cedb0 msgr2=0x7f1b301f7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:22.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b35400640 1 --2- 172.21.15.47:0/1828343898 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b300cedb0 0x7f1b301f7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:22.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b35400640 1 -- 172.21.15.47:0/1828343898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b300d0360 msgr2=0x7f1b301f73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:22.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b35400640 1 --2- 172.21.15.47:0/1828343898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b300d0360 0x7f1b301f73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:22.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b35400640 1 -- 172.21.15.47:0/1828343898 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b200094d0 con 0x7f1b3015bcd0 2024-02-19T15:26:22.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b34a00640 1 --2- 172.21.15.47:0/1828343898 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b300cedb0 0x7f1b301f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:22.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b35400640 1 --2- 172.21.15.47:0/1828343898 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3015bcd0 0x7f1b300c7a40 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f1b2800ba10 tx=0x7f1b2800bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:22.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b2d600640 1 -- 172.21.15.47:0/1828343898 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2800cb20 con 0x7f1b3015bcd0 2024-02-19T15:26:22.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b2d600640 1 -- 172.21.15.47:0/1828343898 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2800cc80 con 0x7f1b3015bcd0 2024-02-19T15:26:22.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b2d600640 1 -- 172.21.15.47:0/1828343898 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b28014440 con 0x7f1b3015bcd0 2024-02-19T15:26:22.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b37400640 1 -- 172.21.15.47:0/1828343898 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b300cdb40 con 0x7f1b3015bcd0 2024-02-19T15:26:22.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b37400640 1 -- 172.21.15.47:0/1828343898 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b300c5970 con 0x7f1b3015bcd0 2024-02-19T15:26:22.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.040+0000 7f1b37400640 1 -- 172.21.15.47:0/1828343898 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b30075360 con 0x7f1b3015bcd0 2024-02-19T15:26:22.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.044+0000 7f1b2d600640 1 -- 172.21.15.47:0/1828343898 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1b2801d3b0 con 0x7f1b3015bcd0 2024-02-19T15:26:22.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.044+0000 7f1b2d600640 1 --2- 172.21.15.47:0/1828343898 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1b0c066e40 0x7f1b0c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:22.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.044+0000 7f1b2d600640 1 -- 172.21.15.47:0/1828343898 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1b28084600 con 0x7f1b3015bcd0 2024-02-19T15:26:22.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.044+0000 7f1b34a00640 1 --2- 172.21.15.47:0/1828343898 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1b0c066e40 0x7f1b0c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:22.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.044+0000 7f1b34a00640 1 --2- 172.21.15.47:0/1828343898 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1b0c066e40 0x7f1b0c069300 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f1b200ae000 tx=0x7f1b20008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:22.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.048+0000 7f1b2d600640 1 -- 172.21.15.47:0/1828343898 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1b280516e0 con 0x7f1b3015bcd0 2024-02-19T15:26:22.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.240+0000 7f1b37400640 1 -- 172.21.15.47:0/1828343898 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b300c5e20 con 0x7f1b3015bcd0 2024-02-19T15:26:22.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.240+0000 7f1b2d600640 1 -- 172.21.15.47:0/1828343898 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f1b28019090 con 0x7f1b3015bcd0 2024-02-19T15:26:22.246 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:22.247 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":51,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5954,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1402677843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1402677843},{"type":"v1","addr":"172.21.15.116:6836","nonce":1402677843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":5957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1507554634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":1507554634},{"type":"v1","addr":"172.21.15.116:6837","nonce":1507554634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6771,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/888455604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":888455604},{"type":"v1","addr":"172.21.15.47:6833","nonce":888455604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":7047,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/4172825099","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":4172825099},{"type":"v1","addr":"172.21.15.47:6835","nonce":4172825099}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[]} 2024-02-19T15:26:22.247 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 51 2024-02-19T15:26:22.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.244+0000 7f1b37400640 1 -- 172.21.15.47:0/1828343898 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1b0c066e40 msgr2=0x7f1b0c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:22.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.244+0000 7f1b37400640 1 --2- 172.21.15.47:0/1828343898 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1b0c066e40 0x7f1b0c069300 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f1b200ae000 tx=0x7f1b20008040 comp rx=0 tx=0).stop 2024-02-19T15:26:22.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.244+0000 7f1b37400640 1 -- 172.21.15.47:0/1828343898 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3015bcd0 msgr2=0x7f1b300c7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:22.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.244+0000 7f1b37400640 1 --2- 172.21.15.47:0/1828343898 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3015bcd0 0x7f1b300c7a40 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f1b2800ba10 tx=0x7f1b2800bed0 comp rx=0 tx=0).stop 2024-02-19T15:26:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.244+0000 7f1b37400640 1 -- 172.21.15.47:0/1828343898 shutdown_connections 2024-02-19T15:26:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.244+0000 7f1b37400640 1 --2- 172.21.15.47:0/1828343898 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1b0c066e40 0x7f1b0c069300 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.244+0000 7f1b37400640 1 --2- 172.21.15.47:0/1828343898 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3015bcd0 0x7f1b300c7a40 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.244+0000 7f1b37400640 1 --2- 172.21.15.47:0/1828343898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b300d0360 0x7f1b301f73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.244+0000 7f1b37400640 1 --2- 172.21.15.47:0/1828343898 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b300cedb0 0x7f1b301f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.244+0000 7f1b37400640 1 -- 172.21.15.47:0/1828343898 >> 172.21.15.47:0/1828343898 conn(0x7f1b300d1bf0 msgr2=0x7f1b300cfc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.244+0000 7f1b37400640 1 -- 172.21.15.47:0/1828343898 shutdown_connections 2024-02-19T15:26:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.244+0000 7f1b37400640 1 -- 172.21.15.47:0/1828343898 wait complete. 2024-02-19T15:26:22.268 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T15:26:22.268 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-19T15:26:22.268 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T15:26:22.268 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:26:22.269 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T15:26:22.269 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-19T15:26:22.269 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T15:26:22.269 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:26:22.269 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T15:26:22.269 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-19T15:26:22.269 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T15:26:22.270 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:26:22.270 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T15:26:22.270 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-19T15:26:22.270 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T15:26:22.270 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:26:22.280 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T15:26:22.280 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T15:26:22.281 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T15:26:22.281 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T15:26:22.284 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T15:26:22.284 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T15:26:22.285 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T15:26:22.285 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T15:26:22.328 INFO:tasks.ceph.mds.b:Started 2024-02-19T15:26:22.329 DEBUG:teuthology.parallel:result is None 2024-02-19T15:26:22.329 INFO:tasks.ceph.mds.a:Started 2024-02-19T15:26:22.329 DEBUG:teuthology.parallel:result is None 2024-02-19T15:26:22.330 INFO:tasks.ceph.mds.d:Started 2024-02-19T15:26:22.330 DEBUG:teuthology.parallel:result is None 2024-02-19T15:26:22.330 INFO:tasks.ceph.mds.c:Started 2024-02-19T15:26:22.331 DEBUG:teuthology.parallel:result is None 2024-02-19T15:26:22.331 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-19T15:26:22.415 INFO:tasks.ceph.mds.a.smithi047.stdout:starting mds.a at 2024-02-19T15:26:22.416 INFO:tasks.ceph.mds.b.smithi116.stdout:starting mds.b at 2024-02-19T15:26:22.416 INFO:tasks.ceph.mds.d.smithi116.stdout:starting mds.d at 2024-02-19T15:26:22.417 INFO:tasks.ceph.mds.c.smithi047.stdout:starting mds.c at 2024-02-19T15:26:22.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.472+0000 7feb53600640 1 Processor -- start 2024-02-19T15:26:22.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.472+0000 7feb53600640 1 -- start start 2024-02-19T15:26:22.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.472+0000 7feb53600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb4c0cea80 0x7feb4c0727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:22.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.472+0000 7feb53600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feb4c15c390 0x7feb4c15c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:22.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.472+0000 7feb53600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4c15d280 con 0x7feb4c145590 2024-02-19T15:26:22.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.472+0000 7feb53600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb4c070ff0 con 0x7feb4c15c390 2024-02-19T15:26:22.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.472+0000 7feb53600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c15cd60 con 0x7feb4c0cea80 2024-02-19T15:26:22.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.472+0000 7feb4be00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb4c0cea80 0x7feb4c0727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:22.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.472+0000 7feb51600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feb4c15c390 0x7feb4c15c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:22.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.472+0000 7feb4be00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb4c0cea80 0x7feb4c0727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:50632/0 (socket says 172.21.15.47:50632) 2024-02-19T15:26:22.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.472+0000 7feb50c00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7feb4c145590 0x7feb4c06c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:40686/0 (socket says 172.21.15.47:40686) 2024-02-19T15:26:22.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.472+0000 7feb4be00640 1 -- 172.21.15.47:0/45802662 learned_addr learned my addr 172.21.15.47:0/45802662 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:22.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.476+0000 7feb4be00640 1 -- 172.21.15.47:0/45802662 >> v2:172.21.15.116:3300/0 conn(0x7feb4c15c390 msgr2=0x7feb4c15c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:22.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.476+0000 7feb4be00640 1 --2- 172.21.15.47:0/45802662 >> v2:172.21.15.116:3300/0 conn(0x7feb4c15c390 0x7feb4c15c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:22.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.476+0000 7feb4be00640 1 -- 172.21.15.47:0/45802662 >> v1:172.21.15.116:6789/0 conn(0x7feb4c145590 legacy=0x7feb4c06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:22.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.476+0000 7feb4be00640 1 -- 172.21.15.47:0/45802662 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4c0709e0 con 0x7feb4c0cea80 2024-02-19T15:26:22.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.476+0000 7feb51600640 1 --2- 172.21.15.47:0/45802662 >> v2:172.21.15.116:3300/0 conn(0x7feb4c15c390 0x7feb4c15c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:22.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.476+0000 7feb4be00640 1 --2- 172.21.15.47:0/45802662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb4c0cea80 0x7feb4c0727b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7feb38009870 tx=0x7feb3801c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9556f15da8a70241 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:22.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.476+0000 7feb4b400640 1 -- 172.21.15.47:0/45802662 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb380ab650 con 0x7feb4c0cea80 2024-02-19T15:26:22.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.476+0000 7feb4b400640 1 -- 172.21.15.47:0/45802662 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb380ab7b0 con 0x7feb4c0cea80 2024-02-19T15:26:22.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.476+0000 7feb4b400640 1 -- 172.21.15.47:0/45802662 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb380abb40 con 0x7feb4c0cea80 2024-02-19T15:26:22.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.476+0000 7feb53600640 1 -- 172.21.15.47:0/45802662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb4c0cea80 msgr2=0x7feb4c0727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:22.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.476+0000 7feb53600640 1 --2- 172.21.15.47:0/45802662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb4c0cea80 0x7feb4c0727b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7feb38009870 tx=0x7feb3801c560 comp rx=0 tx=0).stop 2024-02-19T15:26:22.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb53600640 1 -- 172.21.15.47:0/45802662 shutdown_connections 2024-02-19T15:26:22.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb53600640 1 --2- 172.21.15.47:0/45802662 >> v2:172.21.15.116:3300/0 conn(0x7feb4c15c390 0x7feb4c15c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:22.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb53600640 1 --2- 172.21.15.47:0/45802662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb4c0cea80 0x7feb4c0727b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:22.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb53600640 1 -- 172.21.15.47:0/45802662 >> 172.21.15.47:0/45802662 conn(0x7feb4c0b16a0 msgr2=0x7feb4c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:22.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb53600640 1 -- 172.21.15.47:0/45802662 shutdown_connections 2024-02-19T15:26:22.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb53600640 1 -- 172.21.15.47:0/45802662 wait complete. 2024-02-19T15:26:22.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb53600640 1 Processor -- start 2024-02-19T15:26:22.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb53600640 1 -- start start 2024-02-19T15:26:22.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb53600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb4c0cea80 0x7feb4c1fa440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:22.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb53600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb4c145590 0x7feb4c13c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:22.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb53600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb4c15c390 0x7feb4c13ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:22.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb53600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c1fa980 con 0x7feb4c15c390 2024-02-19T15:26:22.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb53600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c14b0b0 con 0x7feb4c0cea80 2024-02-19T15:26:22.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb53600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c142f90 con 0x7feb4c145590 2024-02-19T15:26:22.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb50c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb4c0cea80 0x7feb4c1fa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:22.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb50c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb4c0cea80 0x7feb4c1fa440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:38358/0 (socket says 172.21.15.47:38358) 2024-02-19T15:26:22.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb51600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb4c15c390 0x7feb4c13ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:22.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb50c00640 1 -- 172.21.15.47:0/2909906951 learned_addr learned my addr 172.21.15.47:0/2909906951 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:22.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb4be00640 1 --2- 172.21.15.47:0/2909906951 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb4c145590 0x7feb4c13c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:22.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb50c00640 1 -- 172.21.15.47:0/2909906951 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb4c145590 msgr2=0x7feb4c13c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:22.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb50c00640 1 --2- 172.21.15.47:0/2909906951 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb4c145590 0x7feb4c13c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:22.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb50c00640 1 -- 172.21.15.47:0/2909906951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb4c15c390 msgr2=0x7feb4c13ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:22.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb50c00640 1 --2- 172.21.15.47:0/2909906951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb4c15c390 0x7feb4c13ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:22.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb50c00640 1 -- 172.21.15.47:0/2909906951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb44009a70 con 0x7feb4c0cea80 2024-02-19T15:26:22.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.480+0000 7feb50c00640 1 --2- 172.21.15.47:0/2909906951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb4c0cea80 0x7feb4c1fa440 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7feb4000c420 tx=0x7feb4000c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:22.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.484+0000 7feb48c00640 1 -- 172.21.15.47:0/2909906951 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb400074e0 con 0x7feb4c0cea80 2024-02-19T15:26:22.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.484+0000 7feb48c00640 1 -- 172.21.15.47:0/2909906951 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb40007640 con 0x7feb4c0cea80 2024-02-19T15:26:22.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.484+0000 7feb53600640 1 -- 172.21.15.47:0/2909906951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb380094d0 con 0x7feb4c0cea80 2024-02-19T15:26:22.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.484+0000 7feb48c00640 1 -- 172.21.15.47:0/2909906951 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb40004a30 con 0x7feb4c0cea80 2024-02-19T15:26:22.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.484+0000 7feb53600640 1 -- 172.21.15.47:0/2909906951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb4c142380 con 0x7feb4c0cea80 2024-02-19T15:26:22.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.484+0000 7feb53600640 1 -- 172.21.15.47:0/2909906951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb4c075360 con 0x7feb4c0cea80 2024-02-19T15:26:22.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.484+0000 7feb48c00640 1 -- 172.21.15.47:0/2909906951 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7feb40004b90 con 0x7feb4c0cea80 2024-02-19T15:26:22.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.484+0000 7feb48c00640 1 --2- 172.21.15.47:0/2909906951 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feb2c066e40 0x7feb2c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:22.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.484+0000 7feb48c00640 1 -- 172.21.15.47:0/2909906951 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7feb40084490 con 0x7feb4c0cea80 2024-02-19T15:26:22.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.484+0000 7feb4be00640 1 --2- 172.21.15.47:0/2909906951 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feb2c066e40 0x7feb2c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:22.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.488+0000 7feb4be00640 1 --2- 172.21.15.47:0/2909906951 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feb2c066e40 0x7feb2c069300 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7feb380af000 tx=0x7feb380b6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:22.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.488+0000 7feb48c00640 1 -- 172.21.15.47:0/2909906951 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7feb40051560 con 0x7feb4c0cea80 2024-02-19T15:26:22.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:22.660+0000 7feb53600640 1 -- 172.21.15.47:0/2909906951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7feb4c0b1d40 con 0x7feb4c0cea80 2024-02-19T15:26:23.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.296+0000 7feb48c00640 1 -- 172.21.15.47:0/2909906951 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v87) v1 ==== 97+0+0 (secure 0 0 0) 0x7feb40054c20 con 0x7feb4c0cea80 2024-02-19T15:26:23.303 INFO:teuthology.orchestra.run.smithi047.stderr: removed all blocklist entries 2024-02-19T15:26:23.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.300+0000 7feb53600640 1 -- 172.21.15.47:0/2909906951 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feb2c066e40 msgr2=0x7feb2c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:23.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.300+0000 7feb53600640 1 --2- 172.21.15.47:0/2909906951 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feb2c066e40 0x7feb2c069300 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7feb380af000 tx=0x7feb380b6040 comp rx=0 tx=0).stop 2024-02-19T15:26:23.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.300+0000 7feb53600640 1 -- 172.21.15.47:0/2909906951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb4c0cea80 msgr2=0x7feb4c1fa440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:23.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.300+0000 7feb53600640 1 --2- 172.21.15.47:0/2909906951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb4c0cea80 0x7feb4c1fa440 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7feb4000c420 tx=0x7feb4000c820 comp rx=0 tx=0).stop 2024-02-19T15:26:23.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.300+0000 7feb53600640 1 -- 172.21.15.47:0/2909906951 shutdown_connections 2024-02-19T15:26:23.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.300+0000 7feb53600640 1 --2- 172.21.15.47:0/2909906951 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feb2c066e40 0x7feb2c069300 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.300+0000 7feb53600640 1 --2- 172.21.15.47:0/2909906951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb4c15c390 0x7feb4c13ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.300+0000 7feb53600640 1 --2- 172.21.15.47:0/2909906951 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb4c145590 0x7feb4c13c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.300+0000 7feb53600640 1 --2- 172.21.15.47:0/2909906951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb4c0cea80 0x7feb4c1fa440 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.300+0000 7feb53600640 1 -- 172.21.15.47:0/2909906951 >> 172.21.15.47:0/2909906951 conn(0x7feb4c0b16a0 msgr2=0x7feb4c0b0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:23.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.300+0000 7feb53600640 1 -- 172.21.15.47:0/2909906951 shutdown_connections 2024-02-19T15:26:23.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.300+0000 7feb53600640 1 -- 172.21.15.47:0/2909906951 wait complete. 2024-02-19T15:26:23.328 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-19T15:26:23.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.468+0000 7fbd28c00640 1 Processor -- start 2024-02-19T15:26:23.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.468+0000 7fbd28c00640 1 -- start start 2024-02-19T15:26:23.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.468+0000 7fbd28c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbd2415d6e0 0x7fbd24162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:23.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.468+0000 7fbd28c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbd24071350 0x7fbd24164c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:23.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.468+0000 7fbd28c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd240718c0 con 0x7fbd2415b1a0 2024-02-19T15:26:23.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.468+0000 7fbd28c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd24071ba0 con 0x7fbd24071350 2024-02-19T15:26:23.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.468+0000 7fbd28c00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd24071e80 con 0x7fbd2415d6e0 2024-02-19T15:26:23.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.468+0000 7fbd22800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbd24071350 0x7fbd24164c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:23.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.468+0000 7fbd22800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbd24071350 0x7fbd24164c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58218/0 (socket says 172.21.15.47:58218) 2024-02-19T15:26:23.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.468+0000 7fbd22800640 1 -- 172.21.15.47:0/1993437291 learned_addr learned my addr 172.21.15.47:0/1993437291 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:23.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.468+0000 7fbd21400640 1 --2- 172.21.15.47:0/1993437291 >> v2:172.21.15.116:3300/0 conn(0x7fbd2415d6e0 0x7fbd24162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:23.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.468+0000 7fbd22800640 1 -- 172.21.15.47:0/1993437291 >> v2:172.21.15.116:3300/0 conn(0x7fbd2415d6e0 msgr2=0x7fbd24162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:23.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.468+0000 7fbd22800640 1 --2- 172.21.15.47:0/1993437291 >> v2:172.21.15.116:3300/0 conn(0x7fbd2415d6e0 0x7fbd24162740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.468+0000 7fbd22800640 1 -- 172.21.15.47:0/1993437291 >> v1:172.21.15.116:6789/0 conn(0x7fbd2415b1a0 legacy=0x7fbd2415b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:23.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.468+0000 7fbd22800640 1 -- 172.21.15.47:0/1993437291 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd24072160 con 0x7fbd24071350 2024-02-19T15:26:23.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.468+0000 7fbd21400640 1 --2- 172.21.15.47:0/1993437291 >> v2:172.21.15.116:3300/0 conn(0x7fbd2415d6e0 0x7fbd24162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:23.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.472+0000 7fbd22800640 1 --2- 172.21.15.47:0/1993437291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd24071350 0x7fbd24164c90 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009e50 tx=0x7fbd1c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b94634367b099f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:23.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.472+0000 7fbd20a00640 1 -- 172.21.15.47:0/1993437291 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0a58e0 con 0x7fbd24071350 2024-02-19T15:26:23.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.472+0000 7fbd20a00640 1 -- 172.21.15.47:0/1993437291 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1c0a5a40 con 0x7fbd24071350 2024-02-19T15:26:23.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.472+0000 7fbd20a00640 1 -- 172.21.15.47:0/1993437291 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0a5dd0 con 0x7fbd24071350 2024-02-19T15:26:23.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.472+0000 7fbd28c00640 1 -- 172.21.15.47:0/1993437291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd24071350 msgr2=0x7fbd24164c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:23.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.472+0000 7fbd28c00640 1 --2- 172.21.15.47:0/1993437291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd24071350 0x7fbd24164c90 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009e50 tx=0x7fbd1c01c9f0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.472+0000 7fbd28c00640 1 -- 172.21.15.47:0/1993437291 shutdown_connections 2024-02-19T15:26:23.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.472+0000 7fbd28c00640 1 --2- 172.21.15.47:0/1993437291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd24071350 0x7fbd24164c90 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.472+0000 7fbd28c00640 1 --2- 172.21.15.47:0/1993437291 >> v2:172.21.15.116:3300/0 conn(0x7fbd2415d6e0 0x7fbd24162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.472+0000 7fbd28c00640 1 -- 172.21.15.47:0/1993437291 >> 172.21.15.47:0/1993437291 conn(0x7fbd240b16a0 msgr2=0x7fbd240b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:23.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.472+0000 7fbd28c00640 1 -- 172.21.15.47:0/1993437291 shutdown_connections 2024-02-19T15:26:23.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.472+0000 7fbd28c00640 1 -- 172.21.15.47:0/1993437291 wait complete. 2024-02-19T15:26:23.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.472+0000 7fbd28c00640 1 Processor -- start 2024-02-19T15:26:23.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.472+0000 7fbd28c00640 1 -- start start 2024-02-19T15:26:23.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd28c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd24071350 0x7fbd240cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:23.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd28c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd2415b1a0 0x7fbd24156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:23.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd28c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd2415d6e0 0x7fbd24157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:23.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd28c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2415e860 con 0x7fbd2415b1a0 2024-02-19T15:26:23.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd28c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd241551b0 con 0x7fbd24071350 2024-02-19T15:26:23.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd28c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd24154ed0 con 0x7fbd2415d6e0 2024-02-19T15:26:23.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd21400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd2415b1a0 0x7fbd24156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:23.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd21400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd2415b1a0 0x7fbd24156e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58234/0 (socket says 172.21.15.47:58234) 2024-02-19T15:26:23.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd21400640 1 -- 172.21.15.47:0/2068401923 learned_addr learned my addr 172.21.15.47:0/2068401923 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:23.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd21e00640 1 --2- 172.21.15.47:0/2068401923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd24071350 0x7fbd240cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:23.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd22800640 1 --2- 172.21.15.47:0/2068401923 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd2415d6e0 0x7fbd24157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:23.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd21400640 1 -- 172.21.15.47:0/2068401923 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd2415d6e0 msgr2=0x7fbd24157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:23.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd21400640 1 --2- 172.21.15.47:0/2068401923 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd2415d6e0 0x7fbd24157350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd21400640 1 -- 172.21.15.47:0/2068401923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd24071350 msgr2=0x7fbd240cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:23.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd21400640 1 --2- 172.21.15.47:0/2068401923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd24071350 0x7fbd240cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd21400640 1 -- 172.21.15.47:0/2068401923 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd1c009b30 con 0x7fbd2415b1a0 2024-02-19T15:26:23.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd21400640 1 --2- 172.21.15.47:0/2068401923 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd2415b1a0 0x7fbd24156e10 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000e7e0 tx=0x7fbd1000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:23.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd1a000640 1 -- 172.21.15.47:0/2068401923 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1000c6d0 con 0x7fbd2415b1a0 2024-02-19T15:26:23.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd1a000640 1 -- 172.21.15.47:0/2068401923 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1000c830 con 0x7fbd2415b1a0 2024-02-19T15:26:23.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd1a000640 1 -- 172.21.15.47:0/2068401923 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1000cbe0 con 0x7fbd2415b1a0 2024-02-19T15:26:23.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd28c00640 1 -- 172.21.15.47:0/2068401923 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd240ccf60 con 0x7fbd2415b1a0 2024-02-19T15:26:23.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd28c00640 1 -- 172.21.15.47:0/2068401923 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd240cd520 con 0x7fbd2415b1a0 2024-02-19T15:26:23.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd28c00640 1 -- 172.21.15.47:0/2068401923 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd24072770 con 0x7fbd2415b1a0 2024-02-19T15:26:23.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.476+0000 7fbd1a000640 1 -- 172.21.15.47:0/2068401923 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fbd1000cd40 con 0x7fbd2415b1a0 2024-02-19T15:26:23.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.480+0000 7fbd1a000640 1 --2- 172.21.15.47:0/2068401923 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbd04066e40 0x7fbd04069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:23.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.480+0000 7fbd1a000640 1 -- 172.21.15.47:0/2068401923 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbd100845a0 con 0x7fbd2415b1a0 2024-02-19T15:26:23.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.480+0000 7fbd21e00640 1 --2- 172.21.15.47:0/2068401923 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbd04066e40 0x7fbd04069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:23.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.480+0000 7fbd21e00640 1 --2- 172.21.15.47:0/2068401923 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbd04066e40 0x7fbd04069300 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fbd24154e80 tx=0x7fbd14009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:23.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.484+0000 7fbd1a000640 1 -- 172.21.15.47:0/2068401923 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fbd100516f0 con 0x7fbd2415b1a0 2024-02-19T15:26:23.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.700+0000 7fbd28c00640 1 -- 172.21.15.47:0/2068401923 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbd2415b970 con 0x7fbd2415b1a0 2024-02-19T15:26:23.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.700+0000 7fbd1a000640 1 -- 172.21.15.47:0/2068401923 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fbd10054db0 con 0x7fbd2415b1a0 2024-02-19T15:26:23.706 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:23.706 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:26:23.706 INFO:teuthology.orchestra.run.smithi047.stdout: "auth_dump": [ 2024-02-19T15:26:23.706 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:23.706 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.a", 2024-02-19T15:26:23.706 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNl8BP+HRAAlJ3Jk+1k2XcZAnICzy++Ow==", 2024-02-19T15:26:23.706 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:26:23.706 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:26:23.707 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:26:23.707 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:26:23.707 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:26:23.707 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:23.707 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:23.707 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:23.707 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.b", 2024-02-19T15:26:23.707 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlIOh3KhAAlnU2ZFY4iv/XEw0WqxyZpw==", 2024-02-19T15:26:23.707 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:26:23.707 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:26:23.707 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:26:23.707 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:26:23.707 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:26:23.707 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:23.708 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:23.708 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:23.708 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.c", 2024-02-19T15:26:23.708 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNl48SHJRAAn4GCT5KnP+2PD8IXbGClCg==", 2024-02-19T15:26:23.708 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:26:23.708 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:26:23.708 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:26:23.708 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:26:23.708 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:26:23.708 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:23.708 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:23.708 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:23.708 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.d", 2024-02-19T15:26:23.708 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNljcekMRAA414655SzH43fBA0AtjpxZw==", 2024-02-19T15:26:23.708 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:26:23.709 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:26:23.709 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:26:23.709 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:26:23.709 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:26:23.709 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:23.709 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:23.709 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:23.709 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.0", 2024-02-19T15:26:23.709 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDlcdNl/bycFhAAmzYi/OxEPUW3tcRJC0DNEQ==", 2024-02-19T15:26:23.709 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:26:23.709 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:26:23.709 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:26:23.709 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:26:23.709 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:23.709 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:23.710 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:23.710 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.1", 2024-02-19T15:26:23.710 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDocdNluquhHBAA88aLHCvVol7JoZB0MtoutA==", 2024-02-19T15:26:23.710 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:26:23.710 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:26:23.710 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:26:23.710 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:26:23.710 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:23.710 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:23.710 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:23.710 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.2", 2024-02-19T15:26:23.710 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDrcdNlGAnWIhAAyELvCPKjtfgF0+kX6NQFIA==", 2024-02-19T15:26:23.710 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:26:23.710 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:26:23.710 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:26:23.710 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:26:23.710 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:23.711 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:23.711 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:23.711 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.3", 2024-02-19T15:26:23.711 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDucdNlROZ6KBAAxTCDuYsw5SRIZBcclzwxhg==", 2024-02-19T15:26:23.711 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:26:23.711 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:26:23.711 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:26:23.711 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:26:23.711 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:23.711 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:23.711 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:23.711 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.4", 2024-02-19T15:26:23.711 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD0cdNluOGILBAA66R9RVXD2vdyIw+oZ+iM1g==", 2024-02-19T15:26:23.711 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:26:23.712 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:26:23.712 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:26:23.712 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:26:23.712 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:23.712 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:23.712 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:23.712 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.5", 2024-02-19T15:26:23.712 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD3cdNlHg8dMhAA2LUzfpvvHR4i8e5RBCeGpQ==", 2024-02-19T15:26:23.712 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:26:23.712 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:26:23.712 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:26:23.712 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:26:23.712 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:23.712 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:23.713 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:23.713 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.6", 2024-02-19T15:26:23.713 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD6cdNlXD1EORAAhKGIt0wFeCyoE6QkkZ3caA==", 2024-02-19T15:26:23.713 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:26:23.713 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:26:23.713 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:26:23.713 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:26:23.713 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:23.713 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:23.713 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:23.713 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.7", 2024-02-19T15:26:23.713 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD+cdNlfW65AhAAVWj3ptC8uQHPh/UXcb/uDg==", 2024-02-19T15:26:23.713 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:26:23.713 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:26:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:26:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:26:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.0", 2024-02-19T15:26:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlb7eRNRAAk6ZLGfjhcdlR9GoXFCd9eA==", 2024-02-19T15:26:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:26:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:26:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow r", 2024-02-19T15:26:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-19T15:26:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:23.714 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.1", 2024-02-19T15:26:23.715 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlKBu+AxAAlLRU70SAs3ZJo5wFwRWa4w==", 2024-02-19T15:26:23.715 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:26:23.715 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:26:23.715 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:26:23.715 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:26:23.715 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:26:23.715 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:23.715 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:23.715 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:23.715 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.2", 2024-02-19T15:26:23.715 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlyDk9ABAAIAjKSd4qJO5oHhVnDc1Rgw==", 2024-02-19T15:26:23.715 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:26:23.715 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:26:23.715 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:26:23.715 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:26:23.715 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:26:23.716 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:23.716 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:23.716 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:23.716 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.3", 2024-02-19T15:26:23.716 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlUqK3CRAABmSIzoJvl0ymME6swrvnSQ==", 2024-02-19T15:26:23.716 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:26:23.716 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:26:23.716 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:26:23.716 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:26:23.716 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:26:23.716 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:23.716 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:23.716 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:23.716 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.admin", 2024-02-19T15:26:23.717 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNls3vPAhAA4x2yQTUCz4Dv9vqEWnSTyQ==", 2024-02-19T15:26:23.717 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:26:23.717 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:26:23.717 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:26:23.717 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:26:23.717 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:26:23.717 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:23.717 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:23.717 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:23.717 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mgr.x", 2024-02-19T15:26:23.717 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNliM78GRAABWBg4hMcKjV2pK5WVBjLMQ==", 2024-02-19T15:26:23.717 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:26:23.717 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:26:23.717 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile mgr", 2024-02-19T15:26:23.717 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:26:23.718 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:23.718 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:26:23.718 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:23.718 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mgr.y", 2024-02-19T15:26:23.718 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlwxs9FhAAWqP7YoKio/bbdx7SGiCJ9Q==", 2024-02-19T15:26:23.718 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:26:23.718 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:26:23.718 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile mgr", 2024-02-19T15:26:23.718 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:26:23.718 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:23.718 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:23.718 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-02-19T15:26:23.718 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:26:23.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.704+0000 7fbd28c00640 1 -- 172.21.15.47:0/2068401923 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbd04066e40 msgr2=0x7fbd04069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:23.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.704+0000 7fbd28c00640 1 --2- 172.21.15.47:0/2068401923 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbd04066e40 0x7fbd04069300 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fbd24154e80 tx=0x7fbd14009040 comp rx=0 tx=0).stop 2024-02-19T15:26:23.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.704+0000 7fbd28c00640 1 -- 172.21.15.47:0/2068401923 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd2415b1a0 msgr2=0x7fbd24156e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:23.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.704+0000 7fbd28c00640 1 --2- 172.21.15.47:0/2068401923 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd2415b1a0 0x7fbd24156e10 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000e7e0 tx=0x7fbd1000eca0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.704+0000 7fbd28c00640 1 -- 172.21.15.47:0/2068401923 shutdown_connections 2024-02-19T15:26:23.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.704+0000 7fbd28c00640 1 --2- 172.21.15.47:0/2068401923 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbd04066e40 0x7fbd04069300 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.704+0000 7fbd28c00640 1 --2- 172.21.15.47:0/2068401923 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd2415d6e0 0x7fbd24157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.704+0000 7fbd28c00640 1 --2- 172.21.15.47:0/2068401923 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd2415b1a0 0x7fbd24156e10 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.704+0000 7fbd28c00640 1 --2- 172.21.15.47:0/2068401923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd24071350 0x7fbd240cdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.704+0000 7fbd28c00640 1 -- 172.21.15.47:0/2068401923 >> 172.21.15.47:0/2068401923 conn(0x7fbd240b16a0 msgr2=0x7fbd2406d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:23.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.704+0000 7fbd28c00640 1 -- 172.21.15.47:0/2068401923 shutdown_connections 2024-02-19T15:26:23.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.704+0000 7fbd28c00640 1 -- 172.21.15.47:0/2068401923 wait complete. 2024-02-19T15:26:23.727 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:26:23.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.860+0000 7fb12f400640 1 Processor -- start 2024-02-19T15:26:23.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.864+0000 7fb12f400640 1 -- start start 2024-02-19T15:26:23.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.864+0000 7fb12f400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb12815dfa0 0x7fb12806fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:23.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.864+0000 7fb12f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb1280b4240 0x7fb1280d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:23.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.864+0000 7fb12f400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb1280ce400 0x7fb1280ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:23.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.864+0000 7fb12f400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1280cedd0 con 0x7fb1280b4240 2024-02-19T15:26:23.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.864+0000 7fb12f400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb12806ff60 con 0x7fb1280ce400 2024-02-19T15:26:23.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.864+0000 7fb12f400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1280d1070 con 0x7fb12815dfa0 2024-02-19T15:26:23.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.864+0000 7fb11fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb1280b4240 0x7fb1280d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:23.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.864+0000 7fb11fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb1280b4240 0x7fb1280d0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58238/0 (socket says 172.21.15.47:58238) 2024-02-19T15:26:23.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.864+0000 7fb11fe00640 1 -- 172.21.15.47:0/3876498930 learned_addr learned my addr 172.21.15.47:0/3876498930 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:23.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.864+0000 7fb12ca00640 1 --2- 172.21.15.47:0/3876498930 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb12815dfa0 0x7fb12806fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:23.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.864+0000 7fb12d400640 1 --2- 172.21.15.47:0/3876498930 >> v2:172.21.15.116:3300/0 conn(0x7fb1280ce400 0x7fb1280ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:23.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.864+0000 7fb11fe00640 1 -- 172.21.15.47:0/3876498930 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb12815dfa0 msgr2=0x7fb12806fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:23.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.864+0000 7fb11fe00640 1 --2- 172.21.15.47:0/3876498930 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb12815dfa0 0x7fb12806fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.864+0000 7fb11fe00640 1 -- 172.21.15.47:0/3876498930 >> v2:172.21.15.116:3300/0 conn(0x7fb1280ce400 msgr2=0x7fb1280ce890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:23.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.864+0000 7fb11fe00640 1 --2- 172.21.15.47:0/3876498930 >> v2:172.21.15.116:3300/0 conn(0x7fb1280ce400 0x7fb1280ce890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.864+0000 7fb11fe00640 1 -- 172.21.15.47:0/3876498930 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12806ec60 con 0x7fb1280b4240 2024-02-19T15:26:23.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.864+0000 7fb12ca00640 1 --2- 172.21.15.47:0/3876498930 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb12815dfa0 0x7fb12806fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:23.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.864+0000 7fb12d400640 1 --2- 172.21.15.47:0/3876498930 >> v2:172.21.15.116:3300/0 conn(0x7fb1280ce400 0x7fb1280ce890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:23.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.868+0000 7fb11fe00640 1 --2- 172.21.15.47:0/3876498930 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1280b4240 0x7fb1280d0820 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fb12806e100 tx=0x7fb11001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef54ee0da464109 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:23.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.868+0000 7fb127e00640 1 -- 172.21.15.47:0/3876498930 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1100aa610 con 0x7fb1280b4240 2024-02-19T15:26:23.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.868+0000 7fb127e00640 1 -- 172.21.15.47:0/3876498930 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1100a6040 con 0x7fb1280b4240 2024-02-19T15:26:23.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.868+0000 7fb127e00640 1 -- 172.21.15.47:0/3876498930 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1100aa990 con 0x7fb1280b4240 2024-02-19T15:26:23.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.868+0000 7fb12f400640 1 -- 172.21.15.47:0/3876498930 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1280b4240 msgr2=0x7fb1280d0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:23.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.868+0000 7fb12f400640 1 --2- 172.21.15.47:0/3876498930 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1280b4240 0x7fb1280d0820 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fb12806e100 tx=0x7fb11001c4f0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.868+0000 7fb12f400640 1 -- 172.21.15.47:0/3876498930 shutdown_connections 2024-02-19T15:26:23.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.868+0000 7fb12f400640 1 --2- 172.21.15.47:0/3876498930 >> v2:172.21.15.116:3300/0 conn(0x7fb1280ce400 0x7fb1280ce890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.868+0000 7fb12f400640 1 --2- 172.21.15.47:0/3876498930 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1280b4240 0x7fb1280d0820 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.868+0000 7fb12f400640 1 --2- 172.21.15.47:0/3876498930 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb12815dfa0 0x7fb12806fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.868+0000 7fb12f400640 1 -- 172.21.15.47:0/3876498930 >> 172.21.15.47:0/3876498930 conn(0x7fb1280b3690 msgr2=0x7fb1280b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:23.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.868+0000 7fb12f400640 1 -- 172.21.15.47:0/3876498930 shutdown_connections 2024-02-19T15:26:23.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.868+0000 7fb12f400640 1 -- 172.21.15.47:0/3876498930 wait complete. 2024-02-19T15:26:23.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.868+0000 7fb12f400640 1 Processor -- start 2024-02-19T15:26:23.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.868+0000 7fb12f400640 1 -- start start 2024-02-19T15:26:23.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb12f400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1280b4240 0x7fb1281560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:23.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb12f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1280ce400 0x7fb1280c56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:23.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb12f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb12815dfa0 0x7fb1280ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:23.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb12f400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1281565e0 con 0x7fb12815dfa0 2024-02-19T15:26:23.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb12f400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1280c6130 con 0x7fb1280ce400 2024-02-19T15:26:23.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb12f400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1280c6410 con 0x7fb1280b4240 2024-02-19T15:26:23.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb12d400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb12815dfa0 0x7fb1280ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:23.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb12d400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb12815dfa0 0x7fb1280ccd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58240/0 (socket says 172.21.15.47:58240) 2024-02-19T15:26:23.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb12d400640 1 -- 172.21.15.47:0/3178578418 learned_addr learned my addr 172.21.15.47:0/3178578418 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:23.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb11fe00640 1 --2- 172.21.15.47:0/3178578418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1280ce400 0x7fb1280c56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:23.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb12ca00640 1 --2- 172.21.15.47:0/3178578418 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1280b4240 0x7fb1281560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:23.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb12d400640 1 -- 172.21.15.47:0/3178578418 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1280b4240 msgr2=0x7fb1281560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:23.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb12d400640 1 --2- 172.21.15.47:0/3178578418 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1280b4240 0x7fb1281560a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb12d400640 1 -- 172.21.15.47:0/3178578418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1280ce400 msgr2=0x7fb1280c56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:23.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb12d400640 1 --2- 172.21.15.47:0/3178578418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1280ce400 0x7fb1280c56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:23.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb12d400640 1 -- 172.21.15.47:0/3178578418 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1100094d0 con 0x7fb12815dfa0 2024-02-19T15:26:23.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb11fe00640 1 --2- 172.21.15.47:0/3178578418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1280ce400 0x7fb1280c56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:23.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb12d400640 1 --2- 172.21.15.47:0/3178578418 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb12815dfa0 0x7fb1280ccd70 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fb12000cef0 tx=0x7fb12000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:23.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb125600640 1 -- 172.21.15.47:0/3178578418 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12000d840 con 0x7fb12815dfa0 2024-02-19T15:26:23.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb125600640 1 -- 172.21.15.47:0/3178578418 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12000d9c0 con 0x7fb12815dfa0 2024-02-19T15:26:23.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb125600640 1 -- 172.21.15.47:0/3178578418 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12000dd70 con 0x7fb12815dfa0 2024-02-19T15:26:23.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb12f400640 1 -- 172.21.15.47:0/3178578418 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1280c66f0 con 0x7fb12815dfa0 2024-02-19T15:26:23.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb12f400640 1 -- 172.21.15.47:0/3178578418 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1280c69d0 con 0x7fb12815dfa0 2024-02-19T15:26:23.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb12f400640 1 -- 172.21.15.47:0/3178578418 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb12815f6f0 con 0x7fb12815dfa0 2024-02-19T15:26:23.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.872+0000 7fb125600640 1 -- 172.21.15.47:0/3178578418 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb12001e3b0 con 0x7fb12815dfa0 2024-02-19T15:26:23.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.876+0000 7fb125600640 1 --2- 172.21.15.47:0/3178578418 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb104066e40 0x7fb104069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:23.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.876+0000 7fb125600640 1 -- 172.21.15.47:0/3178578418 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb120085260 con 0x7fb12815dfa0 2024-02-19T15:26:23.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.876+0000 7fb12ca00640 1 --2- 172.21.15.47:0/3178578418 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb104066e40 0x7fb104069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:23.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.876+0000 7fb12ca00640 1 --2- 172.21.15.47:0/3178578418 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb104066e40 0x7fb104069300 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb118006fd0 tx=0x7fb118008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:23.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:23.880+0000 7fb125600640 1 -- 172.21.15.47:0/3178578418 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb12815f6f0 con 0x7fb12815dfa0 2024-02-19T15:26:24.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.040+0000 7fb12f400640 1 -- 172.21.15.47:0/3178578418 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb12815f6f0 con 0x7fb12815dfa0 2024-02-19T15:26:24.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.040+0000 7fb125600640 1 -- 172.21.15.47:0/3178578418 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+12904 (secure 0 0 0) 0x7fb12815f6f0 con 0x7fb12815dfa0 2024-02-19T15:26:24.044 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:24.044 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":89,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:26:23.306625+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:26:24.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.040+0000 7fb12f400640 1 -- 172.21.15.47:0/3178578418 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb104066e40 msgr2=0x7fb104069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:24.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.040+0000 7fb12f400640 1 --2- 172.21.15.47:0/3178578418 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb104066e40 0x7fb104069300 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb118006fd0 tx=0x7fb118008040 comp rx=0 tx=0).stop 2024-02-19T15:26:24.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.040+0000 7fb12f400640 1 -- 172.21.15.47:0/3178578418 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb12815dfa0 msgr2=0x7fb1280ccd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:24.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.040+0000 7fb12f400640 1 --2- 172.21.15.47:0/3178578418 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb12815dfa0 0x7fb1280ccd70 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fb12000cef0 tx=0x7fb12000a710 comp rx=0 tx=0).stop 2024-02-19T15:26:24.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.040+0000 7fb12f400640 1 -- 172.21.15.47:0/3178578418 shutdown_connections 2024-02-19T15:26:24.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.040+0000 7fb12f400640 1 --2- 172.21.15.47:0/3178578418 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb104066e40 0x7fb104069300 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.040+0000 7fb12f400640 1 --2- 172.21.15.47:0/3178578418 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb12815dfa0 0x7fb1280ccd70 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.040+0000 7fb12f400640 1 --2- 172.21.15.47:0/3178578418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1280ce400 0x7fb1280c56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.040+0000 7fb12f400640 1 --2- 172.21.15.47:0/3178578418 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1280b4240 0x7fb1281560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.040+0000 7fb12f400640 1 -- 172.21.15.47:0/3178578418 >> 172.21.15.47:0/3178578418 conn(0x7fb1280b3690 msgr2=0x7fb12806c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:24.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.040+0000 7fb12f400640 1 -- 172.21.15.47:0/3178578418 shutdown_connections 2024-02-19T15:26:24.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.040+0000 7fb12f400640 1 -- 172.21.15.47:0/3178578418 wait complete. 2024-02-19T15:26:24.065 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:26:24.066 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:26:24.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.204+0000 7fb304a00640 1 Processor -- start 2024-02-19T15:26:24.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.204+0000 7fb304a00640 1 -- start start 2024-02-19T15:26:24.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.204+0000 7fb304a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb3000ce3b0 0x7fb3000ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.204+0000 7fb304a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3000d2650 0x7fb3000ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.204+0000 7fb304a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3000cc620 con 0x7fb30015e280 2024-02-19T15:26:24.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.204+0000 7fb304a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3000d12e0 con 0x7fb3000ce3b0 2024-02-19T15:26:24.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.204+0000 7fb304a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3000d1940 con 0x7fb3000d2650 2024-02-19T15:26:24.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.208+0000 7fb2fe800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb30015e280 0x7fb3000ccff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:40754/0 (socket says 172.21.15.47:40754) 2024-02-19T15:26:24.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.208+0000 7fb2fe800640 1 -- 172.21.15.47:0/3040871348 learned_addr learned my addr 172.21.15.47:0/3040871348 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:24.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.208+0000 7fb2fde00640 1 --2- 172.21.15.47:0/3040871348 >> v2:172.21.15.116:3300/0 conn(0x7fb3000ce3b0 0x7fb3000ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:24.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.208+0000 7fb2f7e00640 1 --2- 172.21.15.47:0/3040871348 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3000d2650 0x7fb3000ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:24.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.208+0000 7fb2fd400640 1 -- 172.21.15.47:0/3040871348 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764956754 0 0) 0x7fb3000cc620 con 0x7fb30015e280 2024-02-19T15:26:24.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.208+0000 7fb2fd400640 1 -- 172.21.15.47:0/3040871348 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2e4003530 con 0x7fb30015e280 2024-02-19T15:26:24.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.208+0000 7fb2fde00640 1 -- 172.21.15.47:0/3040871348 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3000d2650 msgr2=0x7fb3000ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:24.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.208+0000 7fb2fde00640 1 --2- 172.21.15.47:0/3040871348 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3000d2650 0x7fb3000ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.208+0000 7fb2fd400640 1 -- 172.21.15.47:0/3040871348 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1890614376 0 0) 0x7fb2e4003530 con 0x7fb30015e280 2024-02-19T15:26:24.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.208+0000 7fb2fde00640 1 -- 172.21.15.47:0/3040871348 >> v1:172.21.15.47:6789/0 conn(0x7fb30015e280 legacy=0x7fb3000ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:24.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.208+0000 7fb2fde00640 1 -- 172.21.15.47:0/3040871348 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3000cecd0 con 0x7fb3000ce3b0 2024-02-19T15:26:24.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.208+0000 7fb2fde00640 1 --2- 172.21.15.47:0/3040871348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3000ce3b0 0x7fb3000ce790 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb2f00097c0 tx=0x7fb2f001c2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f273b1c9498d8f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:24.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb2fd400640 1 -- 172.21.15.47:0/3040871348 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8001a00 con 0x7fb3000ce3b0 2024-02-19T15:26:24.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb2fd400640 1 -- 172.21.15.47:0/3040871348 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f00a7050 con 0x7fb3000ce3b0 2024-02-19T15:26:24.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb2fd400640 1 -- 172.21.15.47:0/3040871348 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f00ab620 con 0x7fb3000ce3b0 2024-02-19T15:26:24.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb304a00640 1 -- 172.21.15.47:0/3040871348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3000ce3b0 msgr2=0x7fb3000ce790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:24.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb304a00640 1 --2- 172.21.15.47:0/3040871348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3000ce3b0 0x7fb3000ce790 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb2f00097c0 tx=0x7fb2f001c2a0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb304a00640 1 -- 172.21.15.47:0/3040871348 shutdown_connections 2024-02-19T15:26:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb304a00640 1 --2- 172.21.15.47:0/3040871348 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3000d2650 0x7fb3000ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb304a00640 1 --2- 172.21.15.47:0/3040871348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3000ce3b0 0x7fb3000ce790 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb304a00640 1 -- 172.21.15.47:0/3040871348 >> 172.21.15.47:0/3040871348 conn(0x7fb3000d1b30 msgr2=0x7fb3000d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb304a00640 1 -- 172.21.15.47:0/3040871348 shutdown_connections 2024-02-19T15:26:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb304a00640 1 -- 172.21.15.47:0/3040871348 wait complete. 2024-02-19T15:26:24.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb304a00640 1 Processor -- start 2024-02-19T15:26:24.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb304a00640 1 -- start start 2024-02-19T15:26:24.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb304a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3000ce3b0 0x7fb3000c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb304a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3000d2650 0x7fb3000ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb304a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb30015e280 0x7fb300155c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb304a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30015b2c0 con 0x7fb3000ce3b0 2024-02-19T15:26:24.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb304a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb300158460 con 0x7fb3000d2650 2024-02-19T15:26:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb304a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3001572e0 con 0x7fb30015e280 2024-02-19T15:26:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb2fde00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3000ce3b0 0x7fb3000c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb2fde00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3000ce3b0 0x7fb3000c6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58254/0 (socket says 172.21.15.47:58254) 2024-02-19T15:26:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb2fde00640 1 -- 172.21.15.47:0/2941940400 learned_addr learned my addr 172.21.15.47:0/2941940400 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.212+0000 7fb2f7e00640 1 --2- 172.21.15.47:0/2941940400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3000d2650 0x7fb3000ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.216+0000 7fb2fe800640 1 --2- 172.21.15.47:0/2941940400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb30015e280 0x7fb300155c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.216+0000 7fb2fde00640 1 -- 172.21.15.47:0/2941940400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb30015e280 msgr2=0x7fb300155c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.216+0000 7fb2fde00640 1 --2- 172.21.15.47:0/2941940400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb30015e280 0x7fb300155c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.216+0000 7fb2fde00640 1 -- 172.21.15.47:0/2941940400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3000d2650 msgr2=0x7fb3000ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.216+0000 7fb2fde00640 1 --2- 172.21.15.47:0/2941940400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3000d2650 0x7fb3000ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.216+0000 7fb2fde00640 1 -- 172.21.15.47:0/2941940400 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f0009420 con 0x7fb3000ce3b0 2024-02-19T15:26:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.216+0000 7fb2fde00640 1 --2- 172.21.15.47:0/2941940400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3000ce3b0 0x7fb3000c6100 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fb2f001a950 tx=0x7fb2f001aae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.216+0000 7fb2f6000640 1 -- 172.21.15.47:0/2941940400 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f00b7450 con 0x7fb3000ce3b0 2024-02-19T15:26:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.216+0000 7fb2f6000640 1 -- 172.21.15.47:0/2941940400 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f00b75d0 con 0x7fb3000ce3b0 2024-02-19T15:26:24.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.216+0000 7fb2f6000640 1 -- 172.21.15.47:0/2941940400 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f00b7980 con 0x7fb3000ce3b0 2024-02-19T15:26:24.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.216+0000 7fb304a00640 1 -- 172.21.15.47:0/2941940400 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3001575c0 con 0x7fb3000ce3b0 2024-02-19T15:26:24.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.216+0000 7fb304a00640 1 -- 172.21.15.47:0/2941940400 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb300157f90 con 0x7fb3000ce3b0 2024-02-19T15:26:24.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.216+0000 7fb304a00640 1 -- 172.21.15.47:0/2941940400 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb300157a70 con 0x7fb3000ce3b0 2024-02-19T15:26:24.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.216+0000 7fb2f6000640 1 -- 172.21.15.47:0/2941940400 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb2f00b7ae0 con 0x7fb3000ce3b0 2024-02-19T15:26:24.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.216+0000 7fb2f6000640 1 --2- 172.21.15.47:0/2941940400 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb2dc066e40 0x7fb2dc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.216+0000 7fb2f6000640 1 -- 172.21.15.47:0/2941940400 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb2f011ee50 con 0x7fb3000ce3b0 2024-02-19T15:26:24.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.216+0000 7fb2f7e00640 1 --2- 172.21.15.47:0/2941940400 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb2dc066e40 0x7fb2dc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:24.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.220+0000 7fb2f7e00640 1 --2- 172.21.15.47:0/2941940400 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb2dc066e40 0x7fb2dc069300 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb3000b2600 tx=0x7fb2e8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:24.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.220+0000 7fb2f6000640 1 -- 172.21.15.47:0/2941940400 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb2f00ebf80 con 0x7fb3000ce3b0 2024-02-19T15:26:24.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.384+0000 7fb304a00640 1 -- 172.21.15.47:0/2941940400 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb300075360 con 0x7fb3000ce3b0 2024-02-19T15:26:24.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.384+0000 7fb2f6000640 1 -- 172.21.15.47:0/2941940400 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb2f00bb0a0 con 0x7fb3000ce3b0 2024-02-19T15:26:24.392 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:26:24.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.388+0000 7fb304a00640 1 -- 172.21.15.47:0/2941940400 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb2dc066e40 msgr2=0x7fb2dc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:24.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.388+0000 7fb304a00640 1 --2- 172.21.15.47:0/2941940400 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb2dc066e40 0x7fb2dc069300 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb3000b2600 tx=0x7fb2e8009190 comp rx=0 tx=0).stop 2024-02-19T15:26:24.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.388+0000 7fb304a00640 1 -- 172.21.15.47:0/2941940400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3000ce3b0 msgr2=0x7fb3000c6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:24.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.388+0000 7fb304a00640 1 --2- 172.21.15.47:0/2941940400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3000ce3b0 0x7fb3000c6100 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fb2f001a950 tx=0x7fb2f001aae0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.388+0000 7fb304a00640 1 -- 172.21.15.47:0/2941940400 shutdown_connections 2024-02-19T15:26:24.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.388+0000 7fb304a00640 1 --2- 172.21.15.47:0/2941940400 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb2dc066e40 0x7fb2dc069300 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.388+0000 7fb304a00640 1 --2- 172.21.15.47:0/2941940400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb30015e280 0x7fb300155c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.388+0000 7fb304a00640 1 --2- 172.21.15.47:0/2941940400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3000d2650 0x7fb3000ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.388+0000 7fb304a00640 1 --2- 172.21.15.47:0/2941940400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3000ce3b0 0x7fb3000c6100 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.388+0000 7fb304a00640 1 -- 172.21.15.47:0/2941940400 >> 172.21.15.47:0/2941940400 conn(0x7fb3000d1b30 msgr2=0x7fb30006fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:24.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.388+0000 7fb304a00640 1 -- 172.21.15.47:0/2941940400 shutdown_connections 2024-02-19T15:26:24.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.388+0000 7fb304a00640 1 -- 172.21.15.47:0/2941940400 wait complete. 2024-02-19T15:26:24.411 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T15:26:24.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac53e00640 1 Processor -- start 2024-02-19T15:26:24.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac53e00640 1 -- start start 2024-02-19T15:26:24.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac53e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac5415bb90 0x7fac540cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac53e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac540b3300 con 0x7fac540d2150 2024-02-19T15:26:24.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac53e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac540d0980 con 0x7fac5415bf70 2024-02-19T15:26:24.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac53e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac540d0c60 con 0x7fac5415bb90 2024-02-19T15:26:24.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac53400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fac540d2150 0x7fac540cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:40760/0 (socket says 172.21.15.47:40760) 2024-02-19T15:26:24.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac53400640 1 -- 172.21.15.47:0/1533603503 learned_addr learned my addr 172.21.15.47:0/1533603503 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:24.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac52a00640 1 --2- 172.21.15.47:0/1533603503 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac5415bb90 0x7fac540cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:24.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac52000640 1 -- 172.21.15.47:0/1533603503 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569216595 0 0) 0x7fac540b3300 con 0x7fac540d2150 2024-02-19T15:26:24.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac52000640 1 -- 172.21.15.47:0/1533603503 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac38003530 con 0x7fac540d2150 2024-02-19T15:26:24.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac52000640 1 -- 172.21.15.47:0/1533603503 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3011069012 0 0) 0x7fac38003530 con 0x7fac540d2150 2024-02-19T15:26:24.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac52000640 1 -- 172.21.15.47:0/1533603503 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac5415bb90 msgr2=0x7fac540cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:24.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac52000640 1 --2- 172.21.15.47:0/1533603503 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac5415bb90 0x7fac540cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac52000640 1 -- 172.21.15.47:0/1533603503 >> v1:172.21.15.116:6789/0 conn(0x7fac5415bf70 legacy=0x7fac540d1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:24.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac52000640 1 -- 172.21.15.47:0/1533603503 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac540cc9d0 con 0x7fac540d2150 2024-02-19T15:26:24.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac52000640 1 -- 172.21.15.47:0/1533603503 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7fac4c001680 con 0x7fac540d2150 2024-02-19T15:26:24.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac52000640 1 -- 172.21.15.47:0/1533603503 >> v1:172.21.15.47:6789/0 conn(0x7fac540d2150 legacy=0x7fac540cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:24.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac52000640 1 --2- 172.21.15.47:0/1533603503 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac380039b0 0x7fac38003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac52000640 1 --2- 172.21.15.47:0/1533603503 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac38004310 0x7fac380047f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac52000640 1 --2- 172.21.15.47:0/1533603503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac38004ff0 0x7fac3800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac52000640 1 -- 172.21.15.47:0/1533603503 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3800bed0 con 0x7fac38004310 2024-02-19T15:26:24.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac52000640 1 -- 172.21.15.47:0/1533603503 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3800c040 con 0x7fac38004ff0 2024-02-19T15:26:24.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac52000640 1 -- 172.21.15.47:0/1533603503 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac3800c200 con 0x7fac380039b0 2024-02-19T15:26:24.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac4a000640 1 --2- 172.21.15.47:0/1533603503 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac38004310 0x7fac380047f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:24.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac53400640 1 --2- 172.21.15.47:0/1533603503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac38004ff0 0x7fac3800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:24.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac52a00640 1 --2- 172.21.15.47:0/1533603503 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac380039b0 0x7fac38003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:24.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac4a000640 1 -- 172.21.15.47:0/1533603503 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac380039b0 msgr2=0x7fac38003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:24.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac4a000640 1 --2- 172.21.15.47:0/1533603503 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac380039b0 0x7fac38003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac4a000640 1 -- 172.21.15.47:0/1533603503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac38004ff0 msgr2=0x7fac3800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:24.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac4a000640 1 --2- 172.21.15.47:0/1533603503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac38004ff0 0x7fac3800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac4a000640 1 -- 172.21.15.47:0/1533603503 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac38003530 con 0x7fac38004310 2024-02-19T15:26:24.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac53400640 1 --2- 172.21.15.47:0/1533603503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac38004ff0 0x7fac3800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:24.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.548+0000 7fac52a00640 1 --2- 172.21.15.47:0/1533603503 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac380039b0 0x7fac38003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:24.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.552+0000 7fac4a000640 1 --2- 172.21.15.47:0/1533603503 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac38004310 0x7fac380047f0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fac3c001860 tx=0x7fac3c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2221fc68c1fd1604 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:24.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.552+0000 7fac52000640 1 -- 172.21.15.47:0/1533603503 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c0ab600 con 0x7fac38004310 2024-02-19T15:26:24.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.552+0000 7fac52000640 1 -- 172.21.15.47:0/1533603503 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3c0a7050 con 0x7fac38004310 2024-02-19T15:26:24.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.552+0000 7fac52000640 1 -- 172.21.15.47:0/1533603503 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c0ab9e0 con 0x7fac38004310 2024-02-19T15:26:24.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.552+0000 7fac53e00640 1 -- 172.21.15.47:0/1533603503 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac38004310 msgr2=0x7fac380047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:24.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.552+0000 7fac53e00640 1 --2- 172.21.15.47:0/1533603503 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac38004310 0x7fac380047f0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fac3c001860 tx=0x7fac3c01c4d0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.552+0000 7fac53400640 1 -- 172.21.15.47:0/1533603503 reap_dead start 2024-02-19T15:26:24.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.552+0000 7fac53e00640 1 -- 172.21.15.47:0/1533603503 shutdown_connections 2024-02-19T15:26:24.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.552+0000 7fac53e00640 1 -- 172.21.15.47:0/1533603503 >> 172.21.15.47:0/1533603503 conn(0x7fac5409a9d0 msgr2=0x7fac5409ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:24.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.552+0000 7fac53e00640 1 -- 172.21.15.47:0/1533603503 shutdown_connections 2024-02-19T15:26:24.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.552+0000 7fac53e00640 1 -- 172.21.15.47:0/1533603503 wait complete. 2024-02-19T15:26:24.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac53e00640 1 Processor -- start 2024-02-19T15:26:24.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac53e00640 1 -- start start 2024-02-19T15:26:24.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac53e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac38004310 0x7fac540c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac53e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac5415c370 0x7fac540cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac53e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac541f2110 0x7fac540ca370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac53e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac540c6340 con 0x7fac541f2110 2024-02-19T15:26:24.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac53e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5415c7f0 con 0x7fac5415c370 2024-02-19T15:26:24.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac53e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac540cb9d0 con 0x7fac38004310 2024-02-19T15:26:24.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac53400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac541f2110 0x7fac540ca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:24.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac53400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac541f2110 0x7fac540ca370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58276/0 (socket says 172.21.15.47:58276) 2024-02-19T15:26:24.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac53400640 1 -- 172.21.15.47:0/532879355 learned_addr learned my addr 172.21.15.47:0/532879355 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:24.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac52a00640 1 --2- 172.21.15.47:0/532879355 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac38004310 0x7fac540c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:24.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac4a000640 1 --2- 172.21.15.47:0/532879355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac5415c370 0x7fac540cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:24.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac53400640 1 -- 172.21.15.47:0/532879355 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac38004310 msgr2=0x7fac540c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:24.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac53400640 1 --2- 172.21.15.47:0/532879355 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac38004310 0x7fac540c5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac53400640 1 -- 172.21.15.47:0/532879355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac5415c370 msgr2=0x7fac540cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:24.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac53400640 1 --2- 172.21.15.47:0/532879355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac5415c370 0x7fac540cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac53400640 1 -- 172.21.15.47:0/532879355 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac540c6060 con 0x7fac541f2110 2024-02-19T15:26:24.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac53400640 1 --2- 172.21.15.47:0/532879355 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac541f2110 0x7fac540ca370 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fac5415f8f0 tx=0x7fac4c00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:24.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac4b400640 1 -- 172.21.15.47:0/532879355 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c00eed0 con 0x7fac541f2110 2024-02-19T15:26:24.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac4b400640 1 -- 172.21.15.47:0/532879355 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4c00c610 con 0x7fac541f2110 2024-02-19T15:26:24.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac4b400640 1 -- 172.21.15.47:0/532879355 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c00c9a0 con 0x7fac541f2110 2024-02-19T15:26:24.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac53e00640 1 -- 172.21.15.47:0/532879355 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac540c9890 con 0x7fac541f2110 2024-02-19T15:26:24.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.556+0000 7fac53e00640 1 -- 172.21.15.47:0/532879355 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac540c9e50 con 0x7fac541f2110 2024-02-19T15:26:24.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.560+0000 7fac53e00640 1 -- 172.21.15.47:0/532879355 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac540c96f0 con 0x7fac541f2110 2024-02-19T15:26:24.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.560+0000 7fac4b400640 1 -- 172.21.15.47:0/532879355 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fac4c00cb00 con 0x7fac541f2110 2024-02-19T15:26:24.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.560+0000 7fac4b400640 1 --2- 172.21.15.47:0/532879355 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fac30066e70 0x7fac30069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.560+0000 7fac4b400640 1 -- 172.21.15.47:0/532879355 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fac4c0842a0 con 0x7fac541f2110 2024-02-19T15:26:24.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.560+0000 7fac52a00640 1 --2- 172.21.15.47:0/532879355 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fac30066e70 0x7fac30069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:24.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.560+0000 7fac52a00640 1 --2- 172.21.15.47:0/532879355 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fac30066e70 0x7fac30069330 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fac540b0590 tx=0x7fac4400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:24.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.564+0000 7fac4b400640 1 -- 172.21.15.47:0/532879355 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fac540c96f0 con 0x7fac541f2110 2024-02-19T15:26:24.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.728+0000 7fac53e00640 1 -- 172.21.15.47:0/532879355 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fac540c96f0 con 0x7fac541f2110 2024-02-19T15:26:24.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.728+0000 7fac4b400640 1 -- 172.21.15.47:0/532879355 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v89) v1 ==== 84+0+65 (secure 0 0 0) 0x7fac540c96f0 con 0x7fac541f2110 2024-02-19T15:26:24.734 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:24.734 INFO:teuthology.orchestra.run.smithi047.stdout:[ 2024-02-19T15:26:24.734 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:26:24.734 INFO:teuthology.orchestra.run.smithi047.stdout: "poolnum": 1, 2024-02-19T15:26:24.734 INFO:teuthology.orchestra.run.smithi047.stdout: "poolname": ".mgr" 2024-02-19T15:26:24.734 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:26:24.734 INFO:teuthology.orchestra.run.smithi047.stdout:] 2024-02-19T15:26:24.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.732+0000 7fac53e00640 1 -- 172.21.15.47:0/532879355 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fac30066e70 msgr2=0x7fac30069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:24.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.732+0000 7fac53e00640 1 --2- 172.21.15.47:0/532879355 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fac30066e70 0x7fac30069330 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fac540b0590 tx=0x7fac4400b040 comp rx=0 tx=0).stop 2024-02-19T15:26:24.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.732+0000 7fac53e00640 1 -- 172.21.15.47:0/532879355 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac541f2110 msgr2=0x7fac540ca370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:24.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.732+0000 7fac53e00640 1 --2- 172.21.15.47:0/532879355 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac541f2110 0x7fac540ca370 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fac5415f8f0 tx=0x7fac4c00eb90 comp rx=0 tx=0).stop 2024-02-19T15:26:24.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.732+0000 7fac53e00640 1 -- 172.21.15.47:0/532879355 shutdown_connections 2024-02-19T15:26:24.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.732+0000 7fac53e00640 1 --2- 172.21.15.47:0/532879355 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fac30066e70 0x7fac30069330 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.732+0000 7fac53e00640 1 --2- 172.21.15.47:0/532879355 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac541f2110 0x7fac540ca370 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.732+0000 7fac53e00640 1 --2- 172.21.15.47:0/532879355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac5415c370 0x7fac540cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.732+0000 7fac53e00640 1 --2- 172.21.15.47:0/532879355 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac38004310 0x7fac540c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.732+0000 7fac53e00640 1 -- 172.21.15.47:0/532879355 >> 172.21.15.47:0/532879355 conn(0x7fac5409a9d0 msgr2=0x7fac5415aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:24.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.732+0000 7fac53e00640 1 -- 172.21.15.47:0/532879355 shutdown_connections 2024-02-19T15:26:24.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.732+0000 7fac53e00640 1 -- 172.21.15.47:0/532879355 wait complete. 2024-02-19T15:26:24.754 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T15:26:24.754 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T15:26:24.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.892+0000 7f97c3600640 1 Processor -- start 2024-02-19T15:26:24.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.896+0000 7f97c3600640 1 -- start start 2024-02-19T15:26:24.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.896+0000 7f97c3600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f97bc0b5270 0x7f97bc0719f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.896+0000 7f97c3600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f97bc06c4c0 0x7f97bc06e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.896+0000 7f97c3600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f97bc06c920 0x7f97bc06e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.896+0000 7f97c3600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97bc15f7b0 con 0x7f97bc06c4c0 2024-02-19T15:26:24.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.896+0000 7f97c3600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97bc0b5650 con 0x7f97bc0b5270 2024-02-19T15:26:24.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.896+0000 7f97c3600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc071560 con 0x7f97bc06c920 2024-02-19T15:26:24.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.896+0000 7f97b3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f97bc06c4c0 0x7f97bc06e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:24.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.896+0000 7f97b3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f97bc06c4c0 0x7f97bc06e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58286/0 (socket says 172.21.15.47:58286) 2024-02-19T15:26:24.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.896+0000 7f97b3e00640 1 -- 172.21.15.47:0/1978803395 learned_addr learned my addr 172.21.15.47:0/1978803395 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:24.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.896+0000 7f97c1600640 1 --2- 172.21.15.47:0/1978803395 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f97bc06c920 0x7f97bc06e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:24.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.896+0000 7f97c0c00640 1 --2- 172.21.15.47:0/1978803395 >> v2:172.21.15.116:3300/0 conn(0x7f97bc0b5270 0x7f97bc0719f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:24.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.896+0000 7f97b3e00640 1 -- 172.21.15.47:0/1978803395 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f97bc06c920 msgr2=0x7f97bc06e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:24.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.896+0000 7f97b3e00640 1 --2- 172.21.15.47:0/1978803395 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f97bc06c920 0x7f97bc06e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.896+0000 7f97b3e00640 1 -- 172.21.15.47:0/1978803395 >> v2:172.21.15.116:3300/0 conn(0x7f97bc0b5270 msgr2=0x7f97bc0719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:24.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.896+0000 7f97b3e00640 1 --2- 172.21.15.47:0/1978803395 >> v2:172.21.15.116:3300/0 conn(0x7f97bc0b5270 0x7f97bc0719f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.896+0000 7f97b3e00640 1 -- 172.21.15.47:0/1978803395 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97bc06eaa0 con 0x7f97bc06c4c0 2024-02-19T15:26:24.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.900+0000 7f97b3e00640 1 --2- 172.21.15.47:0/1978803395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97bc06c4c0 0x7f97bc06e020 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f97bc06bec0 tx=0x7f97a401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde5b7bf7170ebd5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:24.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.900+0000 7f97bbe00640 1 -- 172.21.15.47:0/1978803395 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a40ac630 con 0x7f97bc06c4c0 2024-02-19T15:26:24.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.900+0000 7f97bbe00640 1 -- 172.21.15.47:0/1978803395 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a40ac790 con 0x7f97bc06c4c0 2024-02-19T15:26:24.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.900+0000 7f97bbe00640 1 -- 172.21.15.47:0/1978803395 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a40acb20 con 0x7f97bc06c4c0 2024-02-19T15:26:24.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.900+0000 7f97c3600640 1 -- 172.21.15.47:0/1978803395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97bc06c4c0 msgr2=0x7f97bc06e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:24.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.900+0000 7f97c3600640 1 --2- 172.21.15.47:0/1978803395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97bc06c4c0 0x7f97bc06e020 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f97bc06bec0 tx=0x7f97a401c520 comp rx=0 tx=0).stop 2024-02-19T15:26:24.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.900+0000 7f97c3600640 1 -- 172.21.15.47:0/1978803395 shutdown_connections 2024-02-19T15:26:24.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.900+0000 7f97c3600640 1 --2- 172.21.15.47:0/1978803395 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f97bc06c920 0x7f97bc06e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.900+0000 7f97c3600640 1 --2- 172.21.15.47:0/1978803395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97bc06c4c0 0x7f97bc06e020 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.900+0000 7f97c3600640 1 --2- 172.21.15.47:0/1978803395 >> v2:172.21.15.116:3300/0 conn(0x7f97bc0b5270 0x7f97bc0719f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.900+0000 7f97c3600640 1 -- 172.21.15.47:0/1978803395 >> 172.21.15.47:0/1978803395 conn(0x7f97bc0cc3f0 msgr2=0x7f97bc0cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:24.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.900+0000 7f97c3600640 1 -- 172.21.15.47:0/1978803395 shutdown_connections 2024-02-19T15:26:24.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.900+0000 7f97c3600640 1 -- 172.21.15.47:0/1978803395 wait complete. 2024-02-19T15:26:24.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.900+0000 7f97c3600640 1 Processor -- start 2024-02-19T15:26:24.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.900+0000 7f97c3600640 1 -- start start 2024-02-19T15:26:24.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97c3600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f97bc06c4c0 0x7f97bc15a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97c3600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97bc06c920 0x7f97bc0c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97c3600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97bc0b5270 0x7f97bc0c5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97c3600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc15e5b0 con 0x7f97bc06c920 2024-02-19T15:26:24.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97c3600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc15e9d0 con 0x7f97bc0b5270 2024-02-19T15:26:24.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97c3600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc15d650 con 0x7f97bc06c4c0 2024-02-19T15:26:24.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97b3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97bc06c920 0x7f97bc0c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:24.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97b3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97bc06c920 0x7f97bc0c4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58302/0 (socket says 172.21.15.47:58302) 2024-02-19T15:26:24.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97b3e00640 1 -- 172.21.15.47:0/2852695722 learned_addr learned my addr 172.21.15.47:0/2852695722 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:24.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97b3e00640 1 -- 172.21.15.47:0/2852695722 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f97bc06c4c0 msgr2=0x7f97bc15a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:24.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97b3e00640 1 --2- 172.21.15.47:0/2852695722 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f97bc06c4c0 0x7f97bc15a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97b3e00640 1 -- 172.21.15.47:0/2852695722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97bc0b5270 msgr2=0x7f97bc0c5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:24.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97b3e00640 1 --2- 172.21.15.47:0/2852695722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97bc0b5270 0x7f97bc0c5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:24.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97b3e00640 1 -- 172.21.15.47:0/2852695722 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a40094d0 con 0x7f97bc06c920 2024-02-19T15:26:24.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97b3e00640 1 --2- 172.21.15.47:0/2852695722 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97bc06c920 0x7f97bc0c4e50 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f97a40af040 tx=0x7f97a40192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:24.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97b9600640 1 -- 172.21.15.47:0/2852695722 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a40b5660 con 0x7f97bc06c920 2024-02-19T15:26:24.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97b9600640 1 -- 172.21.15.47:0/2852695722 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a40b57c0 con 0x7f97bc06c920 2024-02-19T15:26:24.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97b9600640 1 -- 172.21.15.47:0/2852695722 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a40b5b50 con 0x7f97bc06c920 2024-02-19T15:26:24.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97c3600640 1 -- 172.21.15.47:0/2852695722 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97bc15d930 con 0x7f97bc06c920 2024-02-19T15:26:24.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97c3600640 1 -- 172.21.15.47:0/2852695722 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97bc15def0 con 0x7f97bc06c920 2024-02-19T15:26:24.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97c3600640 1 -- 172.21.15.47:0/2852695722 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97bc15e6f0 con 0x7f97bc06c920 2024-02-19T15:26:24.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.904+0000 7f97b9600640 1 -- 172.21.15.47:0/2852695722 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f97a40b3070 con 0x7f97bc06c920 2024-02-19T15:26:24.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.908+0000 7f97b9600640 1 --2- 172.21.15.47:0/2852695722 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9798066e40 0x7f9798069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:24.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.908+0000 7f97b9600640 1 -- 172.21.15.47:0/2852695722 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f97a411e7c0 con 0x7f97bc06c920 2024-02-19T15:26:24.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.908+0000 7f97c0c00640 1 --2- 172.21.15.47:0/2852695722 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9798066e40 0x7f9798069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:24.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.908+0000 7f97c0c00640 1 --2- 172.21.15.47:0/2852695722 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9798066e40 0x7f9798069300 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f97ac006aa0 tx=0x7f97ac006a30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:24.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:24.912+0000 7f97b9600640 1 -- 172.21.15.47:0/2852695722 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f97bc15e6f0 con 0x7f97bc06c920 2024-02-19T15:26:25.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.076+0000 7f97c3600640 1 -- 172.21.15.47:0/2852695722 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f97bc15e6f0 con 0x7f97bc06c920 2024-02-19T15:26:25.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.304+0000 7f97b9600640 1 -- 172.21.15.47:0/2852695722 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v90) v1 ==== 138+0+0 (secure 0 0 0) 0x7f97bc15e6f0 con 0x7f97bc06c920 2024-02-19T15:26:25.307 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_metadata' created 2024-02-19T15:26:25.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.304+0000 7f97c3600640 1 -- 172.21.15.47:0/2852695722 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9798066e40 msgr2=0x7f9798069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:25.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.304+0000 7f97c3600640 1 --2- 172.21.15.47:0/2852695722 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9798066e40 0x7f9798069300 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f97ac006aa0 tx=0x7f97ac006a30 comp rx=0 tx=0).stop 2024-02-19T15:26:25.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.304+0000 7f97c3600640 1 -- 172.21.15.47:0/2852695722 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97bc06c920 msgr2=0x7f97bc0c4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:25.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.304+0000 7f97c3600640 1 --2- 172.21.15.47:0/2852695722 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97bc06c920 0x7f97bc0c4e50 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f97a40af040 tx=0x7f97a40192c0 comp rx=0 tx=0).stop 2024-02-19T15:26:25.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.304+0000 7f97c3600640 1 -- 172.21.15.47:0/2852695722 shutdown_connections 2024-02-19T15:26:25.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.304+0000 7f97c3600640 1 --2- 172.21.15.47:0/2852695722 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9798066e40 0x7f9798069300 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:25.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.304+0000 7f97c3600640 1 --2- 172.21.15.47:0/2852695722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97bc0b5270 0x7f97bc0c5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:25.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.304+0000 7f97c3600640 1 --2- 172.21.15.47:0/2852695722 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97bc06c920 0x7f97bc0c4e50 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:25.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.304+0000 7f97c3600640 1 --2- 172.21.15.47:0/2852695722 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f97bc06c4c0 0x7f97bc15a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:25.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.304+0000 7f97c3600640 1 -- 172.21.15.47:0/2852695722 >> 172.21.15.47:0/2852695722 conn(0x7f97bc0cc3f0 msgr2=0x7f97bc0cc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:25.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.304+0000 7f97c3600640 1 -- 172.21.15.47:0/2852695722 shutdown_connections 2024-02-19T15:26:25.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.304+0000 7f97c3600640 1 -- 172.21.15.47:0/2852695722 wait complete. 2024-02-19T15:26:25.334 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T15:26:25.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.476+0000 7fb90ec00640 1 Processor -- start 2024-02-19T15:26:25.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.476+0000 7fb90ec00640 1 -- start start 2024-02-19T15:26:25.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.476+0000 7fb90ec00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb90815dec0 0x7fb9081618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:25.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.476+0000 7fb90ec00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb90806b5c0 0x7fb90806ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:25.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.476+0000 7fb90ec00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90806d960 con 0x7fb908161e20 2024-02-19T15:26:25.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.480+0000 7fb90ec00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb90806d3e0 con 0x7fb90806b5c0 2024-02-19T15:26:25.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.480+0000 7fb90ec00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9081623a0 con 0x7fb90815dec0 2024-02-19T15:26:25.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.480+0000 7fb90d800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb90815dec0 0x7fb9081618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:25.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.480+0000 7fb90e200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb90806b5c0 0x7fb90806ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:25.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.480+0000 7fb90ce00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb908161e20 0x7fb90806aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:53174/0 (socket says 172.21.15.47:53174) 2024-02-19T15:26:25.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.480+0000 7fb90ce00640 1 -- 172.21.15.47:0/1972315921 learned_addr learned my addr 172.21.15.47:0/1972315921 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:25.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.480+0000 7fb90d800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb90815dec0 0x7fb9081618e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:51740/0 (socket says 172.21.15.47:51740) 2024-02-19T15:26:25.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.480+0000 7fb907e00640 1 -- 172.21.15.47:0/1972315921 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153997191 0 0) 0x7fb90806d960 con 0x7fb908161e20 2024-02-19T15:26:25.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.480+0000 7fb90d800640 1 -- 172.21.15.47:0/1972315921 >> v2:172.21.15.116:3300/0 conn(0x7fb90806b5c0 msgr2=0x7fb90806ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:25.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.480+0000 7fb90d800640 1 --2- 172.21.15.47:0/1972315921 >> v2:172.21.15.116:3300/0 conn(0x7fb90806b5c0 0x7fb90806ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:25.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.480+0000 7fb90d800640 1 -- 172.21.15.47:0/1972315921 >> v1:172.21.15.116:6789/0 conn(0x7fb908161e20 legacy=0x7fb90806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:25.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.480+0000 7fb90d800640 1 -- 172.21.15.47:0/1972315921 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb908162680 con 0x7fb90815dec0 2024-02-19T15:26:25.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.480+0000 7fb90e200640 1 --2- 172.21.15.47:0/1972315921 >> v2:172.21.15.116:3300/0 conn(0x7fb90806b5c0 0x7fb90806ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:25.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90d800640 1 --2- 172.21.15.47:0/1972315921 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb90815dec0 0x7fb9081618e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc0097c0 tx=0x7fb8fc01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be198e92075abc63 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:25.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb907e00640 1 -- 172.21.15.47:0/1972315921 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc0ab600 con 0x7fb90815dec0 2024-02-19T15:26:25.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb907e00640 1 -- 172.21.15.47:0/1972315921 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8fc0ab760 con 0x7fb90815dec0 2024-02-19T15:26:25.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb907e00640 1 -- 172.21.15.47:0/1972315921 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc0abaf0 con 0x7fb90815dec0 2024-02-19T15:26:25.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90ec00640 1 -- 172.21.15.47:0/1972315921 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb90815dec0 msgr2=0x7fb9081618e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:25.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90ec00640 1 --2- 172.21.15.47:0/1972315921 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb90815dec0 0x7fb9081618e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc0097c0 tx=0x7fb8fc01c3e0 comp rx=0 tx=0).stop 2024-02-19T15:26:25.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90ec00640 1 -- 172.21.15.47:0/1972315921 shutdown_connections 2024-02-19T15:26:25.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90ec00640 1 --2- 172.21.15.47:0/1972315921 >> v2:172.21.15.116:3300/0 conn(0x7fb90806b5c0 0x7fb90806ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:25.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90ec00640 1 --2- 172.21.15.47:0/1972315921 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb90815dec0 0x7fb9081618e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:25.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90ec00640 1 -- 172.21.15.47:0/1972315921 >> 172.21.15.47:0/1972315921 conn(0x7fb9080cd2e0 msgr2=0x7fb9080cd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:25.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90ec00640 1 -- 172.21.15.47:0/1972315921 shutdown_connections 2024-02-19T15:26:25.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90ec00640 1 -- 172.21.15.47:0/1972315921 wait complete. 2024-02-19T15:26:25.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90ec00640 1 Processor -- start 2024-02-19T15:26:25.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90ec00640 1 -- start start 2024-02-19T15:26:25.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90ec00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb90806b5c0 0x7fb9081551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90ec00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb90815dec0 0x7fb908155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90ec00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb908161e20 0x7fb90815e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90ec00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9080affd0 con 0x7fb90806b5c0 2024-02-19T15:26:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90ec00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9080ade80 con 0x7fb90815dec0 2024-02-19T15:26:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90ec00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9080adba0 con 0x7fb908161e20 2024-02-19T15:26:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90d800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb90806b5c0 0x7fb9081551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90d800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb90806b5c0 0x7fb9081551f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58316/0 (socket says 172.21.15.47:58316) 2024-02-19T15:26:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90d800640 1 -- 172.21.15.47:0/3009796003 learned_addr learned my addr 172.21.15.47:0/3009796003 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90ce00640 1 --2- 172.21.15.47:0/3009796003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb90815dec0 0x7fb908155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.484+0000 7fb90e200640 1 --2- 172.21.15.47:0/3009796003 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb908161e20 0x7fb90815e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.488+0000 7fb90d800640 1 -- 172.21.15.47:0/3009796003 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb908161e20 msgr2=0x7fb90815e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.488+0000 7fb90d800640 1 --2- 172.21.15.47:0/3009796003 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb908161e20 0x7fb90815e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.488+0000 7fb90d800640 1 -- 172.21.15.47:0/3009796003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb90815dec0 msgr2=0x7fb908155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:25.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.488+0000 7fb90d800640 1 --2- 172.21.15.47:0/3009796003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb90815dec0 0x7fb908155730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:25.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.488+0000 7fb90d800640 1 -- 172.21.15.47:0/3009796003 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb900009a70 con 0x7fb90806b5c0 2024-02-19T15:26:25.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.488+0000 7fb90d800640 1 --2- 172.21.15.47:0/3009796003 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb90806b5c0 0x7fb9081551f0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc0ad040 tx=0x7fb8fc01ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:25.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.488+0000 7fb905600640 1 -- 172.21.15.47:0/3009796003 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc0bf610 con 0x7fb90806b5c0 2024-02-19T15:26:25.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.488+0000 7fb905600640 1 -- 172.21.15.47:0/3009796003 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8fc0bf7b0 con 0x7fb90806b5c0 2024-02-19T15:26:25.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.488+0000 7fb905600640 1 -- 172.21.15.47:0/3009796003 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc0bfb60 con 0x7fb90806b5c0 2024-02-19T15:26:25.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.488+0000 7fb90ec00640 1 -- 172.21.15.47:0/3009796003 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8fc009420 con 0x7fb90806b5c0 2024-02-19T15:26:25.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.488+0000 7fb90ec00640 1 -- 172.21.15.47:0/3009796003 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9080afcf0 con 0x7fb90806b5c0 2024-02-19T15:26:25.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.488+0000 7fb90ec00640 1 -- 172.21.15.47:0/3009796003 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb90815ac40 con 0x7fb90806b5c0 2024-02-19T15:26:25.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.488+0000 7fb905600640 1 -- 172.21.15.47:0/3009796003 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb8fc0b5400 con 0x7fb90806b5c0 2024-02-19T15:26:25.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.488+0000 7fb905600640 1 --2- 172.21.15.47:0/3009796003 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb8e8066e40 0x7fb8e8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:25.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.488+0000 7fb905600640 1 -- 172.21.15.47:0/3009796003 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fb8fc11f9e0 con 0x7fb90806b5c0 2024-02-19T15:26:25.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.492+0000 7fb90ce00640 1 --2- 172.21.15.47:0/3009796003 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb8e8066e40 0x7fb8e8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:25.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.492+0000 7fb90ce00640 1 --2- 172.21.15.47:0/3009796003 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb8e8066e40 0x7fb8e8069300 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb90806c770 tx=0x7fb8f4007d60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:25.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.496+0000 7fb905600640 1 -- 172.21.15.47:0/3009796003 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb90815ac40 con 0x7fb90806b5c0 2024-02-19T15:26:25.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:25.656+0000 7fb90ec00640 1 -- 172.21.15.47:0/3009796003 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb90815ac40 con 0x7fb90806b5c0 2024-02-19T15:26:26.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.308+0000 7fb905600640 1 -- 172.21.15.47:0/3009796003 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v91) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb90815ac40 con 0x7fb90806b5c0 2024-02-19T15:26:26.315 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_data' created 2024-02-19T15:26:26.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.312+0000 7fb90ec00640 1 -- 172.21.15.47:0/3009796003 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb8e8066e40 msgr2=0x7fb8e8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:26.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.312+0000 7fb90ec00640 1 --2- 172.21.15.47:0/3009796003 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb8e8066e40 0x7fb8e8069300 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb90806c770 tx=0x7fb8f4007d60 comp rx=0 tx=0).stop 2024-02-19T15:26:26.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.312+0000 7fb90ec00640 1 -- 172.21.15.47:0/3009796003 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb90806b5c0 msgr2=0x7fb9081551f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:26.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.312+0000 7fb90ec00640 1 --2- 172.21.15.47:0/3009796003 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb90806b5c0 0x7fb9081551f0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc0ad040 tx=0x7fb8fc01ac20 comp rx=0 tx=0).stop 2024-02-19T15:26:26.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.316+0000 7fb90ec00640 1 -- 172.21.15.47:0/3009796003 shutdown_connections 2024-02-19T15:26:26.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.316+0000 7fb90ec00640 1 --2- 172.21.15.47:0/3009796003 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb8e8066e40 0x7fb8e8069300 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:26.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.316+0000 7fb90ec00640 1 --2- 172.21.15.47:0/3009796003 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb908161e20 0x7fb90815e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:26.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.316+0000 7fb90ec00640 1 --2- 172.21.15.47:0/3009796003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb90815dec0 0x7fb908155730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:26.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.316+0000 7fb90ec00640 1 --2- 172.21.15.47:0/3009796003 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb90806b5c0 0x7fb9081551f0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:26.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.316+0000 7fb90ec00640 1 -- 172.21.15.47:0/3009796003 >> 172.21.15.47:0/3009796003 conn(0x7fb9080cd2e0 msgr2=0x7fb9080cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:26.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.316+0000 7fb90ec00640 1 -- 172.21.15.47:0/3009796003 shutdown_connections 2024-02-19T15:26:26.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.316+0000 7fb90ec00640 1 -- 172.21.15.47:0/3009796003 wait complete. 2024-02-19T15:26:26.359 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T15:26:26.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.500+0000 7f60d3800640 1 Processor -- start 2024-02-19T15:26:26.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.500+0000 7f60d3800640 1 -- start start 2024-02-19T15:26:26.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.504+0000 7f60d3800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f60cc15bd90 0x7f60cc070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:26.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.504+0000 7f60d3800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f60cc0cb490 0x7f60cc072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:26.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.504+0000 7f60d3800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60cc15fc90 con 0x7f60cc072680 2024-02-19T15:26:26.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.504+0000 7f60d3800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60cc15ff70 con 0x7f60cc0cb490 2024-02-19T15:26:26.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.504+0000 7f60d3800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60cc160250 con 0x7f60cc15bd90 2024-02-19T15:26:26.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.504+0000 7f60cbe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f60cc0cb490 0x7f60cc072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:26.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.504+0000 7f60cbe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f60cc0cb490 0x7f60cc072140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58326/0 (socket says 172.21.15.47:58326) 2024-02-19T15:26:26.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.504+0000 7f60cbe00640 1 -- 172.21.15.47:0/2236874937 learned_addr learned my addr 172.21.15.47:0/2236874937 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:26.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.504+0000 7f60d0e00640 1 --2- 172.21.15.47:0/2236874937 >> v2:172.21.15.116:3300/0 conn(0x7f60cc15bd90 0x7f60cc070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:26.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.504+0000 7f60cbe00640 1 -- 172.21.15.47:0/2236874937 >> v2:172.21.15.116:3300/0 conn(0x7f60cc15bd90 msgr2=0x7f60cc070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:26.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.504+0000 7f60cbe00640 1 --2- 172.21.15.47:0/2236874937 >> v2:172.21.15.116:3300/0 conn(0x7f60cc15bd90 0x7f60cc070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:26.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.504+0000 7f60cbe00640 1 -- 172.21.15.47:0/2236874937 >> v1:172.21.15.116:6789/0 conn(0x7f60cc072680 legacy=0x7f60cc0ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:26.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.504+0000 7f60cbe00640 1 -- 172.21.15.47:0/2236874937 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60cc160530 con 0x7f60cc0cb490 2024-02-19T15:26:26.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60cbe00640 1 --2- 172.21.15.47:0/2236874937 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60cc0cb490 0x7f60cc072140 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f60cc06c300 tx=0x7f60b801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ebdfe3e5c0743d1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:26.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60cb400640 1 -- 172.21.15.47:0/2236874937 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b80ab620 con 0x7f60cc0cb490 2024-02-19T15:26:26.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60cb400640 1 -- 172.21.15.47:0/2236874937 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b80ab780 con 0x7f60cc0cb490 2024-02-19T15:26:26.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60cb400640 1 -- 172.21.15.47:0/2236874937 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b80abb10 con 0x7f60cc0cb490 2024-02-19T15:26:26.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60d3800640 1 -- 172.21.15.47:0/2236874937 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60cc0cb490 msgr2=0x7f60cc072140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:26.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60d3800640 1 --2- 172.21.15.47:0/2236874937 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60cc0cb490 0x7f60cc072140 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f60cc06c300 tx=0x7f60b801c3c0 comp rx=0 tx=0).stop 2024-02-19T15:26:26.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60d3800640 1 -- 172.21.15.47:0/2236874937 shutdown_connections 2024-02-19T15:26:26.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60d3800640 1 --2- 172.21.15.47:0/2236874937 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60cc0cb490 0x7f60cc072140 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:26.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60d3800640 1 --2- 172.21.15.47:0/2236874937 >> v2:172.21.15.116:3300/0 conn(0x7f60cc15bd90 0x7f60cc070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:26.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60d3800640 1 -- 172.21.15.47:0/2236874937 >> 172.21.15.47:0/2236874937 conn(0x7f60cc0cfc20 msgr2=0x7f60cc0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:26.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60d3800640 1 -- 172.21.15.47:0/2236874937 shutdown_connections 2024-02-19T15:26:26.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60d3800640 1 -- 172.21.15.47:0/2236874937 wait complete. 2024-02-19T15:26:26.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60d3800640 1 Processor -- start 2024-02-19T15:26:26.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60d3800640 1 -- start start 2024-02-19T15:26:26.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60d3800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f60cc072680 0x7f60cc1f4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:26.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60d3800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60cc0cb490 0x7f60cc155100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:26.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60d3800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60cc15bd90 0x7f60cc158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:26.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60d3800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc155df0 con 0x7f60cc15bd90 2024-02-19T15:26:26.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60d3800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc158bf0 con 0x7f60cc0cb490 2024-02-19T15:26:26.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60d3800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc1560b0 con 0x7f60cc072680 2024-02-19T15:26:26.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60d1800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60cc15bd90 0x7f60cc158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:26.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60d1800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60cc15bd90 0x7f60cc158170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58334/0 (socket says 172.21.15.47:58334) 2024-02-19T15:26:26.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60d1800640 1 -- 172.21.15.47:0/2198396960 learned_addr learned my addr 172.21.15.47:0/2198396960 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:26.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60d0e00640 1 --2- 172.21.15.47:0/2198396960 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f60cc072680 0x7f60cc1f4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:26.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.508+0000 7f60cbe00640 1 --2- 172.21.15.47:0/2198396960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60cc0cb490 0x7f60cc155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:26.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.512+0000 7f60d1800640 1 -- 172.21.15.47:0/2198396960 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f60cc072680 msgr2=0x7f60cc1f4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:26.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.512+0000 7f60d1800640 1 --2- 172.21.15.47:0/2198396960 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f60cc072680 0x7f60cc1f4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:26.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.512+0000 7f60d1800640 1 -- 172.21.15.47:0/2198396960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60cc0cb490 msgr2=0x7f60cc155100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:26.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.512+0000 7f60d1800640 1 --2- 172.21.15.47:0/2198396960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60cc0cb490 0x7f60cc155100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:26.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.512+0000 7f60d1800640 1 -- 172.21.15.47:0/2198396960 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b80094d0 con 0x7f60cc15bd90 2024-02-19T15:26:26.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.512+0000 7f60d1800640 1 --2- 172.21.15.47:0/2198396960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60cc15bd90 0x7f60cc158170 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f60c400c730 tx=0x7f60c400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:26.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.512+0000 7f60c8c00640 1 -- 172.21.15.47:0/2198396960 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c4003b80 con 0x7f60cc15bd90 2024-02-19T15:26:26.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.512+0000 7f60c8c00640 1 -- 172.21.15.47:0/2198396960 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c4003ce0 con 0x7f60cc15bd90 2024-02-19T15:26:26.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.512+0000 7f60c8c00640 1 -- 172.21.15.47:0/2198396960 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c400aa30 con 0x7f60cc15bd90 2024-02-19T15:26:26.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.512+0000 7f60d3800640 1 -- 172.21.15.47:0/2198396960 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60cc156390 con 0x7f60cc15bd90 2024-02-19T15:26:26.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.512+0000 7f60d3800640 1 -- 172.21.15.47:0/2198396960 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60cc155640 con 0x7f60cc15bd90 2024-02-19T15:26:26.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.512+0000 7f60d3800640 1 -- 172.21.15.47:0/2198396960 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60cc075360 con 0x7f60cc15bd90 2024-02-19T15:26:26.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.512+0000 7f60c8c00640 1 -- 172.21.15.47:0/2198396960 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f60c400ab90 con 0x7f60cc15bd90 2024-02-19T15:26:26.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.516+0000 7f60c8c00640 1 --2- 172.21.15.47:0/2198396960 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f60ac066e40 0x7f60ac069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:26.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.516+0000 7f60c8c00640 1 -- 172.21.15.47:0/2198396960 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f60c4085180 con 0x7f60cc15bd90 2024-02-19T15:26:26.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.516+0000 7f60d0e00640 1 --2- 172.21.15.47:0/2198396960 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f60ac066e40 0x7f60ac069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:26.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.516+0000 7f60d0e00640 1 --2- 172.21.15.47:0/2198396960 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f60ac066e40 0x7f60ac069300 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f60cc1613e0 tx=0x7f60c0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:26.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.516+0000 7f60c8c00640 1 -- 172.21.15.47:0/2198396960 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f60c4052060 con 0x7f60cc15bd90 2024-02-19T15:26:26.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:26.716+0000 7f60d3800640 1 -- 172.21.15.47:0/2198396960 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f60cc156840 con 0x7f60cc15bd90 2024-02-19T15:26:27.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.312+0000 7f60c8c00640 1 -- 172.21.15.47:0/2198396960 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v54) v1 ==== 174+0+0 (secure 0 0 0) 0x7f60c4055720 con 0x7f60cc15bd90 2024-02-19T15:26:27.317 INFO:teuthology.orchestra.run.smithi047.stderr:new fs with metadata pool 14 and data pool 15 2024-02-19T15:26:27.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.316+0000 7f60d3800640 1 -- 172.21.15.47:0/2198396960 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f60ac066e40 msgr2=0x7f60ac069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:27.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.316+0000 7f60d3800640 1 --2- 172.21.15.47:0/2198396960 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f60ac066e40 0x7f60ac069300 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f60cc1613e0 tx=0x7f60c0008040 comp rx=0 tx=0).stop 2024-02-19T15:26:27.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.316+0000 7f60d3800640 1 -- 172.21.15.47:0/2198396960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60cc15bd90 msgr2=0x7f60cc158170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:27.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.316+0000 7f60d3800640 1 --2- 172.21.15.47:0/2198396960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60cc15bd90 0x7f60cc158170 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f60c400c730 tx=0x7f60c400cbf0 comp rx=0 tx=0).stop 2024-02-19T15:26:27.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.316+0000 7f60d3800640 1 -- 172.21.15.47:0/2198396960 shutdown_connections 2024-02-19T15:26:27.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.316+0000 7f60d3800640 1 --2- 172.21.15.47:0/2198396960 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f60ac066e40 0x7f60ac069300 secure :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f60cc1613e0 tx=0x7f60c0008040 comp rx=0 tx=0).stop 2024-02-19T15:26:27.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.316+0000 7f60d3800640 1 --2- 172.21.15.47:0/2198396960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60cc15bd90 0x7f60cc158170 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:27.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.316+0000 7f60d3800640 1 --2- 172.21.15.47:0/2198396960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60cc0cb490 0x7f60cc155100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:27.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.316+0000 7f60d3800640 1 --2- 172.21.15.47:0/2198396960 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f60cc072680 0x7f60cc1f4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:27.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.316+0000 7f60d3800640 1 -- 172.21.15.47:0/2198396960 >> 172.21.15.47:0/2198396960 conn(0x7f60cc0cfc20 msgr2=0x7f60cc0d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:27.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.316+0000 7f60d3800640 1 -- 172.21.15.47:0/2198396960 shutdown_connections 2024-02-19T15:26:27.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.316+0000 7f60d3800640 1 -- 172.21.15.47:0/2198396960 wait complete. 2024-02-19T15:26:27.371 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:26:27.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.508+0000 7fd550000640 1 Processor -- start 2024-02-19T15:26:27.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.508+0000 7fd550000640 1 -- start start 2024-02-19T15:26:27.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.508+0000 7fd550000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd548071350 0x7fd548165e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:27.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.508+0000 7fd550000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5480718c0 con 0x7fd54815d6e0 2024-02-19T15:26:27.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.508+0000 7fd550000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd548071ba0 con 0x7fd54815b1a0 2024-02-19T15:26:27.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.508+0000 7fd550000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd548071e80 con 0x7fd548071350 2024-02-19T15:26:27.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.512+0000 7fd54cc00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd54815d6e0 0x7fd548162740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:40764/0 (socket says 172.21.15.47:40764) 2024-02-19T15:26:27.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.512+0000 7fd54cc00640 1 -- 172.21.15.47:0/2842347654 learned_addr learned my addr 172.21.15.47:0/2842347654 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:27.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.512+0000 7fd54e000640 1 --2- 172.21.15.47:0/2842347654 >> v2:172.21.15.47:3300/0 conn(0x7fd548071350 0x7fd548165e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:27.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.512+0000 7fd547e00640 1 -- 172.21.15.47:0/2842347654 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002115473 0 0) 0x7fd5480718c0 con 0x7fd54815d6e0 2024-02-19T15:26:27.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.512+0000 7fd547e00640 1 -- 172.21.15.47:0/2842347654 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd530005740 con 0x7fd54815d6e0 2024-02-19T15:26:27.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.512+0000 7fd547e00640 1 -- 172.21.15.47:0/2842347654 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168973615 0 0) 0x7fd548071ba0 con 0x7fd54815b1a0 2024-02-19T15:26:27.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.512+0000 7fd547e00640 1 -- 172.21.15.47:0/2842347654 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5480718c0 con 0x7fd54815b1a0 2024-02-19T15:26:27.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.512+0000 7fd54e000640 1 -- 172.21.15.47:0/2842347654 >> v1:172.21.15.116:6789/0 conn(0x7fd54815b1a0 legacy=0x7fd54815b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:27.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.512+0000 7fd54e000640 1 -- 172.21.15.47:0/2842347654 >> v1:172.21.15.47:6789/0 conn(0x7fd54815d6e0 legacy=0x7fd548162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:27.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.512+0000 7fd54e000640 1 -- 172.21.15.47:0/2842347654 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd548072160 con 0x7fd548071350 2024-02-19T15:26:27.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.512+0000 7fd54e000640 1 --2- 172.21.15.47:0/2842347654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd548071350 0x7fd548165e70 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fd540009dc0 tx=0x7fd54001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=895c25b35bf018d4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:27.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.512+0000 7fd547e00640 1 -- 172.21.15.47:0/2842347654 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5400a5930 con 0x7fd548071350 2024-02-19T15:26:27.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.512+0000 7fd547e00640 1 -- 172.21.15.47:0/2842347654 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5400a5a90 con 0x7fd548071350 2024-02-19T15:26:27.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.512+0000 7fd547e00640 1 -- 172.21.15.47:0/2842347654 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5400a5e20 con 0x7fd548071350 2024-02-19T15:26:27.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd550000640 1 -- 172.21.15.47:0/2842347654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd548071350 msgr2=0x7fd548165e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:27.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd550000640 1 --2- 172.21.15.47:0/2842347654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd548071350 0x7fd548165e70 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fd540009dc0 tx=0x7fd54001c950 comp rx=0 tx=0).stop 2024-02-19T15:26:27.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd550000640 1 -- 172.21.15.47:0/2842347654 shutdown_connections 2024-02-19T15:26:27.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd550000640 1 --2- 172.21.15.47:0/2842347654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd548071350 0x7fd548165e70 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:27.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd550000640 1 -- 172.21.15.47:0/2842347654 >> 172.21.15.47:0/2842347654 conn(0x7fd5480b16a0 msgr2=0x7fd5480b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:27.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd550000640 1 -- 172.21.15.47:0/2842347654 shutdown_connections 2024-02-19T15:26:27.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd550000640 1 -- 172.21.15.47:0/2842347654 wait complete. 2024-02-19T15:26:27.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd550000640 1 Processor -- start 2024-02-19T15:26:27.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd550000640 1 -- start start 2024-02-19T15:26:27.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd550000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd548071350 0x7fd5480ce990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:27.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd550000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd54815b1a0 0x7fd5480ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:27.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd550000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd54815d6e0 0x7fd5481f6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:27.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd550000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd548146ab0 con 0x7fd548071350 2024-02-19T15:26:27.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd550000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd548146d90 con 0x7fd54815d6e0 2024-02-19T15:26:27.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd550000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd548147070 con 0x7fd54815b1a0 2024-02-19T15:26:27.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd54d600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd548071350 0x7fd5480ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:27.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd54d600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd548071350 0x7fd5480ce990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58362/0 (socket says 172.21.15.47:58362) 2024-02-19T15:26:27.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd54d600640 1 -- 172.21.15.47:0/671072142 learned_addr learned my addr 172.21.15.47:0/671072142 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:27.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd54e000640 1 --2- 172.21.15.47:0/671072142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd54815d6e0 0x7fd5481f6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:27.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd54cc00640 1 --2- 172.21.15.47:0/671072142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd54815b1a0 0x7fd5480ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:27.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd54d600640 1 -- 172.21.15.47:0/671072142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd54815b1a0 msgr2=0x7fd5480ceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:27.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd54d600640 1 --2- 172.21.15.47:0/671072142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd54815b1a0 0x7fd5480ceed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:27.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd54d600640 1 -- 172.21.15.47:0/671072142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd54815d6e0 msgr2=0x7fd5481f6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:27.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd54d600640 1 --2- 172.21.15.47:0/671072142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd54815d6e0 0x7fd5481f6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:27.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd54d600640 1 -- 172.21.15.47:0/671072142 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd540009a70 con 0x7fd548071350 2024-02-19T15:26:27.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.516+0000 7fd54e000640 1 --2- 172.21.15.47:0/671072142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd54815d6e0 0x7fd5481f6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:27.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.520+0000 7fd54d600640 1 --2- 172.21.15.47:0/671072142 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd548071350 0x7fd5480ce990 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fd53c00c490 tx=0x7fd53c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:27.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.520+0000 7fd545600640 1 -- 172.21.15.47:0/671072142 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c0052d0 con 0x7fd548071350 2024-02-19T15:26:27.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.520+0000 7fd545600640 1 -- 172.21.15.47:0/671072142 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53c005430 con 0x7fd548071350 2024-02-19T15:26:27.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.520+0000 7fd545600640 1 -- 172.21.15.47:0/671072142 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c012870 con 0x7fd548071350 2024-02-19T15:26:27.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.520+0000 7fd550000640 1 -- 172.21.15.47:0/671072142 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd548147350 con 0x7fd548071350 2024-02-19T15:26:27.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.520+0000 7fd550000640 1 -- 172.21.15.47:0/671072142 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54814c130 con 0x7fd548071350 2024-02-19T15:26:27.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.520+0000 7fd550000640 1 -- 172.21.15.47:0/671072142 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd548075360 con 0x7fd548071350 2024-02-19T15:26:27.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.520+0000 7fd545600640 1 -- 172.21.15.47:0/671072142 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd53c0129d0 con 0x7fd548071350 2024-02-19T15:26:27.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.520+0000 7fd545600640 1 --2- 172.21.15.47:0/671072142 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd528066e40 0x7fd528069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:27.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.520+0000 7fd545600640 1 -- 172.21.15.47:0/671072142 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd53c084730 con 0x7fd548071350 2024-02-19T15:26:27.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.524+0000 7fd54cc00640 1 --2- 172.21.15.47:0/671072142 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd528066e40 0x7fd528069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:27.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.524+0000 7fd54cc00640 1 --2- 172.21.15.47:0/671072142 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd528066e40 0x7fd528069300 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd548149170 tx=0x7fd534009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:27.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.524+0000 7fd545600640 1 -- 172.21.15.47:0/671072142 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd53c0513f0 con 0x7fd548071350 2024-02-19T15:26:27.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.692+0000 7fd550000640 1 -- 172.21.15.47:0/671072142 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5481591d0 con 0x7fd548071350 2024-02-19T15:26:27.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.696+0000 7fd545600640 1 -- 172.21.15.47:0/671072142 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+16633 (secure 0 0 0) 0x7fd53c054ab0 con 0x7fd548071350 2024-02-19T15:26:27.699 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:27.699 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":92,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:26:26.720761+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:26:25.124594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-02-19T15:26:25.706971+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:26:27.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.696+0000 7fd550000640 1 -- 172.21.15.47:0/671072142 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd528066e40 msgr2=0x7fd528069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:27.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.696+0000 7fd550000640 1 --2- 172.21.15.47:0/671072142 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd528066e40 0x7fd528069300 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd548149170 tx=0x7fd534009040 comp rx=0 tx=0).stop 2024-02-19T15:26:27.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.696+0000 7fd550000640 1 -- 172.21.15.47:0/671072142 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd548071350 msgr2=0x7fd5480ce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:27.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.696+0000 7fd550000640 1 --2- 172.21.15.47:0/671072142 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd548071350 0x7fd5480ce990 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fd53c00c490 tx=0x7fd53c00c950 comp rx=0 tx=0).stop 2024-02-19T15:26:27.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.696+0000 7fd550000640 1 -- 172.21.15.47:0/671072142 shutdown_connections 2024-02-19T15:26:27.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.696+0000 7fd550000640 1 --2- 172.21.15.47:0/671072142 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd528066e40 0x7fd528069300 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:27.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.696+0000 7fd550000640 1 --2- 172.21.15.47:0/671072142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd54815d6e0 0x7fd5481f6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:27.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.696+0000 7fd550000640 1 --2- 172.21.15.47:0/671072142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd54815b1a0 0x7fd5480ceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:27.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.696+0000 7fd550000640 1 --2- 172.21.15.47:0/671072142 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd548071350 0x7fd5480ce990 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:27.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.696+0000 7fd550000640 1 -- 172.21.15.47:0/671072142 >> 172.21.15.47:0/671072142 conn(0x7fd5480b16a0 msgr2=0x7fd5480cf940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:27.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.696+0000 7fd550000640 1 -- 172.21.15.47:0/671072142 shutdown_connections 2024-02-19T15:26:27.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.696+0000 7fd550000640 1 -- 172.21.15.47:0/671072142 wait complete. 2024-02-19T15:26:27.720 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:26:27.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.856+0000 7fe6b1200640 1 Processor -- start 2024-02-19T15:26:27.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.856+0000 7fe6b1200640 1 -- start start 2024-02-19T15:26:27.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.860+0000 7fe6b1200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe6ac0cea80 0x7fe6ac0727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:27.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.860+0000 7fe6b1200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe6ac15c390 0x7fe6ac15c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:27.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.860+0000 7fe6b1200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ac15d280 con 0x7fe6ac145590 2024-02-19T15:26:27.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.860+0000 7fe6b1200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6ac070ff0 con 0x7fe6ac0cea80 2024-02-19T15:26:27.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.860+0000 7fe6b1200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6ac15cd40 con 0x7fe6ac15c390 2024-02-19T15:26:27.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.860+0000 7fe6a1400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe6ac0cea80 0x7fe6ac0727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:27.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.860+0000 7fe6a1400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe6ac0cea80 0x7fe6ac0727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58374/0 (socket says 172.21.15.47:58374) 2024-02-19T15:26:27.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.860+0000 7fe6a1400640 1 -- 172.21.15.47:0/564096207 learned_addr learned my addr 172.21.15.47:0/564096207 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:27.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.860+0000 7fe6aa800640 1 --2- 172.21.15.47:0/564096207 >> v2:172.21.15.116:3300/0 conn(0x7fe6ac15c390 0x7fe6ac15c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:27.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.860+0000 7fe6a1400640 1 -- 172.21.15.47:0/564096207 >> v2:172.21.15.116:3300/0 conn(0x7fe6ac15c390 msgr2=0x7fe6ac15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:27.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.860+0000 7fe6a1400640 1 --2- 172.21.15.47:0/564096207 >> v2:172.21.15.116:3300/0 conn(0x7fe6ac15c390 0x7fe6ac15c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:27.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.860+0000 7fe6a1400640 1 -- 172.21.15.47:0/564096207 >> v1:172.21.15.116:6789/0 conn(0x7fe6ac145590 legacy=0x7fe6ac06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:27.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.860+0000 7fe6a1400640 1 -- 172.21.15.47:0/564096207 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6ac0709e0 con 0x7fe6ac0cea80 2024-02-19T15:26:27.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.860+0000 7fe6a1400640 1 --2- 172.21.15.47:0/564096207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6ac0cea80 0x7fe6ac0727b0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac072380 tx=0x7fe69401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc289992ea560fcf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:27.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6a9400640 1 -- 172.21.15.47:0/564096207 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6940ab610 con 0x7fe6ac0cea80 2024-02-19T15:26:27.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6a9400640 1 -- 172.21.15.47:0/564096207 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6940ab770 con 0x7fe6ac0cea80 2024-02-19T15:26:27.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6a9400640 1 -- 172.21.15.47:0/564096207 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6940abb00 con 0x7fe6ac0cea80 2024-02-19T15:26:27.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6b1200640 1 -- 172.21.15.47:0/564096207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6ac0cea80 msgr2=0x7fe6ac0727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:27.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6b1200640 1 --2- 172.21.15.47:0/564096207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6ac0cea80 0x7fe6ac0727b0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac072380 tx=0x7fe69401c520 comp rx=0 tx=0).stop 2024-02-19T15:26:27.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6b1200640 1 -- 172.21.15.47:0/564096207 shutdown_connections 2024-02-19T15:26:27.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6b1200640 1 --2- 172.21.15.47:0/564096207 >> v2:172.21.15.116:3300/0 conn(0x7fe6ac15c390 0x7fe6ac15c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:27.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6b1200640 1 --2- 172.21.15.47:0/564096207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6ac0cea80 0x7fe6ac0727b0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:27.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6b1200640 1 -- 172.21.15.47:0/564096207 >> 172.21.15.47:0/564096207 conn(0x7fe6ac0b16a0 msgr2=0x7fe6ac0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:27.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6b1200640 1 -- 172.21.15.47:0/564096207 shutdown_connections 2024-02-19T15:26:27.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6b1200640 1 -- 172.21.15.47:0/564096207 wait complete. 2024-02-19T15:26:27.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6b1200640 1 Processor -- start 2024-02-19T15:26:27.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6b1200640 1 -- start start 2024-02-19T15:26:27.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6b1200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6ac0cea80 0x7fe6ac13a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:27.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6b1200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6ac145590 0x7fe6ac0b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:27.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6b1200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe6ac15c390 0x7fe6ac13b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:27.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6b1200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ac0b40a0 con 0x7fe6ac0cea80 2024-02-19T15:26:27.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6b1200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ac0b3420 con 0x7fe6ac145590 2024-02-19T15:26:27.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6b1200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ac0b3700 con 0x7fe6ac15c390 2024-02-19T15:26:27.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6a9e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6ac0cea80 0x7fe6ac13a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:27.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6a9e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6ac0cea80 0x7fe6ac13a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58376/0 (socket says 172.21.15.47:58376) 2024-02-19T15:26:27.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6a9e00640 1 -- 172.21.15.47:0/3137912873 learned_addr learned my addr 172.21.15.47:0/3137912873 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:27.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6a1400640 1 --2- 172.21.15.47:0/3137912873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6ac145590 0x7fe6ac0b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:27.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.864+0000 7fe6aa800640 1 --2- 172.21.15.47:0/3137912873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe6ac15c390 0x7fe6ac13b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:27.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.868+0000 7fe6a9e00640 1 -- 172.21.15.47:0/3137912873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe6ac15c390 msgr2=0x7fe6ac13b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:27.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.868+0000 7fe6a9e00640 1 --2- 172.21.15.47:0/3137912873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe6ac15c390 0x7fe6ac13b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:27.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.868+0000 7fe6a9e00640 1 -- 172.21.15.47:0/3137912873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6ac145590 msgr2=0x7fe6ac0b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:27.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.868+0000 7fe6a9e00640 1 --2- 172.21.15.47:0/3137912873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6ac145590 0x7fe6ac0b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:27.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.868+0000 7fe6a9e00640 1 -- 172.21.15.47:0/3137912873 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6940094d0 con 0x7fe6ac0cea80 2024-02-19T15:26:27.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.868+0000 7fe6aa800640 1 --2- 172.21.15.47:0/3137912873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe6ac15c390 0x7fe6ac13b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:27.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.868+0000 7fe6a9e00640 1 --2- 172.21.15.47:0/3137912873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6ac0cea80 0x7fe6ac13a8d0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fe69c00c420 tx=0x7fe69c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:27.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.868+0000 7fe6a2a00640 1 -- 172.21.15.47:0/3137912873 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c0052d0 con 0x7fe6ac0cea80 2024-02-19T15:26:27.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.868+0000 7fe6a2a00640 1 -- 172.21.15.47:0/3137912873 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69c005430 con 0x7fe6ac0cea80 2024-02-19T15:26:27.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.868+0000 7fe6a2a00640 1 -- 172.21.15.47:0/3137912873 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c007260 con 0x7fe6ac0cea80 2024-02-19T15:26:27.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.868+0000 7fe6b1200640 1 -- 172.21.15.47:0/3137912873 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6ac0b39e0 con 0x7fe6ac0cea80 2024-02-19T15:26:27.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.868+0000 7fe6b1200640 1 -- 172.21.15.47:0/3137912873 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6ac0b4a00 con 0x7fe6ac0cea80 2024-02-19T15:26:27.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.868+0000 7fe6b1200640 1 -- 172.21.15.47:0/3137912873 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6ac075360 con 0x7fe6ac0cea80 2024-02-19T15:26:27.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.868+0000 7fe6a2a00640 1 -- 172.21.15.47:0/3137912873 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe69c0073c0 con 0x7fe6ac0cea80 2024-02-19T15:26:27.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.868+0000 7fe6a2a00640 1 --2- 172.21.15.47:0/3137912873 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe688066e40 0x7fe688069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:27.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.868+0000 7fe6a2a00640 1 -- 172.21.15.47:0/3137912873 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe69c083da0 con 0x7fe6ac0cea80 2024-02-19T15:26:27.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.868+0000 7fe6a1400640 1 --2- 172.21.15.47:0/3137912873 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe688066e40 0x7fe688069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:27.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.872+0000 7fe6a1400640 1 --2- 172.21.15.47:0/3137912873 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe688066e40 0x7fe688069300 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fe6940af000 tx=0x7fe6940b6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:27.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:27.872+0000 7fe6a2a00640 1 -- 172.21.15.47:0/3137912873 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe69c050c40 con 0x7fe6ac0cea80 2024-02-19T15:26:28.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.040+0000 7fe6b1200640 1 -- 172.21.15.47:0/3137912873 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6ac0b09a0 con 0x7fe6ac0cea80 2024-02-19T15:26:28.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.040+0000 7fe6a2a00640 1 -- 172.21.15.47:0/3137912873 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+16633 (secure 0 0 0) 0x7fe69c054300 con 0x7fe6ac0cea80 2024-02-19T15:26:28.044 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:28.044 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":92,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:26:26.720761+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:26:25.124594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-02-19T15:26:25.706971+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:26:28.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.040+0000 7fe6b1200640 1 -- 172.21.15.47:0/3137912873 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe688066e40 msgr2=0x7fe688069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:28.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.040+0000 7fe6b1200640 1 --2- 172.21.15.47:0/3137912873 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe688066e40 0x7fe688069300 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fe6940af000 tx=0x7fe6940b6040 comp rx=0 tx=0).stop 2024-02-19T15:26:28.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.040+0000 7fe6b1200640 1 -- 172.21.15.47:0/3137912873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6ac0cea80 msgr2=0x7fe6ac13a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:28.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.040+0000 7fe6b1200640 1 --2- 172.21.15.47:0/3137912873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6ac0cea80 0x7fe6ac13a8d0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fe69c00c420 tx=0x7fe69c00c8e0 comp rx=0 tx=0).stop 2024-02-19T15:26:28.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.040+0000 7fe6b1200640 1 -- 172.21.15.47:0/3137912873 shutdown_connections 2024-02-19T15:26:28.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.040+0000 7fe6b1200640 1 --2- 172.21.15.47:0/3137912873 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe688066e40 0x7fe688069300 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:28.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.040+0000 7fe6b1200640 1 --2- 172.21.15.47:0/3137912873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe6ac15c390 0x7fe6ac13b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:28.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.040+0000 7fe6b1200640 1 --2- 172.21.15.47:0/3137912873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6ac145590 0x7fe6ac0b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:28.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.040+0000 7fe6b1200640 1 --2- 172.21.15.47:0/3137912873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6ac0cea80 0x7fe6ac13a8d0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:28.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.040+0000 7fe6b1200640 1 -- 172.21.15.47:0/3137912873 >> 172.21.15.47:0/3137912873 conn(0x7fe6ac0b16a0 msgr2=0x7fe6ac06d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:28.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.040+0000 7fe6b1200640 1 -- 172.21.15.47:0/3137912873 shutdown_connections 2024-02-19T15:26:28.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.040+0000 7fe6b1200640 1 -- 172.21.15.47:0/3137912873 wait complete. 2024-02-19T15:26:28.065 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T15:26:28.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.204+0000 7fbd71c00640 1 Processor -- start 2024-02-19T15:26:28.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.204+0000 7fbd71c00640 1 -- start start 2024-02-19T15:26:28.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.204+0000 7fbd71c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd6c15af10 0x7fbd6c06e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:28.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.204+0000 7fbd71c00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd6c071c70 con 0x7fbd6c0cd880 2024-02-19T15:26:28.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.204+0000 7fbd71c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd6c06cef0 con 0x7fbd6c0ca8f0 2024-02-19T15:26:28.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.204+0000 7fbd71c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c06e8a0 con 0x7fbd6c15af10 2024-02-19T15:26:28.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.204+0000 7fbd6b400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbd6c0cd880 0x7fbd6c0cdd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:40768/0 (socket says 172.21.15.47:40768) 2024-02-19T15:26:28.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.204+0000 7fbd6b400640 1 -- 172.21.15.47:0/959518282 learned_addr learned my addr 172.21.15.47:0/959518282 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:28.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.204+0000 7fbd6aa00640 1 --2- 172.21.15.47:0/959518282 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd6c15af10 0x7fbd6c06e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:28.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.204+0000 7fbd6a000640 1 -- 172.21.15.47:0/959518282 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374353215 0 0) 0x7fbd6c071c70 con 0x7fbd6c0cd880 2024-02-19T15:26:28.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.204+0000 7fbd6a000640 1 -- 172.21.15.47:0/959518282 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd50003530 con 0x7fbd6c0cd880 2024-02-19T15:26:28.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.204+0000 7fbd6a000640 1 -- 172.21.15.47:0/959518282 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954338679 0 0) 0x7fbd6c06cef0 con 0x7fbd6c0ca8f0 2024-02-19T15:26:28.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.204+0000 7fbd6a000640 1 -- 172.21.15.47:0/959518282 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd6c071c70 con 0x7fbd6c0ca8f0 2024-02-19T15:26:28.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.204+0000 7fbd6aa00640 1 -- 172.21.15.47:0/959518282 >> v1:172.21.15.116:6789/0 conn(0x7fbd6c0ca8f0 legacy=0x7fbd6c0cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:28.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.204+0000 7fbd6aa00640 1 -- 172.21.15.47:0/959518282 >> v1:172.21.15.47:6789/0 conn(0x7fbd6c0cd880 legacy=0x7fbd6c0cdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:28.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.204+0000 7fbd6aa00640 1 -- 172.21.15.47:0/959518282 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6c0ca610 con 0x7fbd6c15af10 2024-02-19T15:26:28.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.208+0000 7fbd6aa00640 1 --2- 172.21.15.47:0/959518282 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd6c15af10 0x7fbd6c06e360 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c0097f0 tx=0x7fbd5c01c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bbefcadc0827269 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:28.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.208+0000 7fbd6a000640 1 -- 172.21.15.47:0/959518282 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c0ab620 con 0x7fbd6c15af10 2024-02-19T15:26:28.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.208+0000 7fbd6a000640 1 -- 172.21.15.47:0/959518282 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5c0ab780 con 0x7fbd6c15af10 2024-02-19T15:26:28.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.208+0000 7fbd6a000640 1 -- 172.21.15.47:0/959518282 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c0abb10 con 0x7fbd6c15af10 2024-02-19T15:26:28.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.208+0000 7fbd71c00640 1 -- 172.21.15.47:0/959518282 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd6c15af10 msgr2=0x7fbd6c06e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:28.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.208+0000 7fbd71c00640 1 --2- 172.21.15.47:0/959518282 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd6c15af10 0x7fbd6c06e360 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c0097f0 tx=0x7fbd5c01c420 comp rx=0 tx=0).stop 2024-02-19T15:26:28.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.208+0000 7fbd71c00640 1 -- 172.21.15.47:0/959518282 shutdown_connections 2024-02-19T15:26:28.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.208+0000 7fbd71c00640 1 --2- 172.21.15.47:0/959518282 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd6c15af10 0x7fbd6c06e360 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:28.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.208+0000 7fbd71c00640 1 -- 172.21.15.47:0/959518282 >> 172.21.15.47:0/959518282 conn(0x7fbd6c0cfc20 msgr2=0x7fbd6c0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:28.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.208+0000 7fbd71c00640 1 -- 172.21.15.47:0/959518282 shutdown_connections 2024-02-19T15:26:28.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.208+0000 7fbd71c00640 1 -- 172.21.15.47:0/959518282 wait complete. 2024-02-19T15:26:28.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd71c00640 1 Processor -- start 2024-02-19T15:26:28.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd71c00640 1 -- start start 2024-02-19T15:26:28.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd71c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd6c0ca8f0 0x7fbd6c154940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:28.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd71c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd6c0cd880 0x7fbd6c0b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:28.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd71c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd6c15af10 0x7fbd6c0b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:28.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd71c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c155d80 con 0x7fbd6c15af10 2024-02-19T15:26:28.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd71c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c155170 con 0x7fbd6c0ca8f0 2024-02-19T15:26:28.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd71c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c155450 con 0x7fbd6c0cd880 2024-02-19T15:26:28.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd6b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd6c15af10 0x7fbd6c0b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:28.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd6b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd6c15af10 0x7fbd6c0b4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58392/0 (socket says 172.21.15.47:58392) 2024-02-19T15:26:28.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd6b400640 1 -- 172.21.15.47:0/2242317056 learned_addr learned my addr 172.21.15.47:0/2242317056 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:28.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd6aa00640 1 --2- 172.21.15.47:0/2242317056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd6c0ca8f0 0x7fbd6c154940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:28.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd62000640 1 --2- 172.21.15.47:0/2242317056 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd6c0cd880 0x7fbd6c0b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:28.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd6b400640 1 -- 172.21.15.47:0/2242317056 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd6c0cd880 msgr2=0x7fbd6c0b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd6b400640 1 --2- 172.21.15.47:0/2242317056 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd6c0cd880 0x7fbd6c0b3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd6b400640 1 -- 172.21.15.47:0/2242317056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd6c0ca8f0 msgr2=0x7fbd6c154940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd6b400640 1 --2- 172.21.15.47:0/2242317056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd6c0ca8f0 0x7fbd6c154940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd6b400640 1 -- 172.21.15.47:0/2242317056 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c009420 con 0x7fbd6c15af10 2024-02-19T15:26:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd6aa00640 1 --2- 172.21.15.47:0/2242317056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd6c0ca8f0 0x7fbd6c154940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd62000640 1 --2- 172.21.15.47:0/2242317056 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd6c0cd880 0x7fbd6c0b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd6b400640 1 --2- 172.21.15.47:0/2242317056 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd6c15af10 0x7fbd6c0b4300 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400cbc0 tx=0x7fbd6400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd63400640 1 -- 172.21.15.47:0/2242317056 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6400ace0 con 0x7fbd6c15af10 2024-02-19T15:26:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd63400640 1 -- 172.21.15.47:0/2242317056 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6400ae40 con 0x7fbd6c15af10 2024-02-19T15:26:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd63400640 1 -- 172.21.15.47:0/2242317056 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd64012600 con 0x7fbd6c15af10 2024-02-19T15:26:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd71c00640 1 -- 172.21.15.47:0/2242317056 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd6c155730 con 0x7fbd6c15af10 2024-02-19T15:26:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.212+0000 7fbd71c00640 1 -- 172.21.15.47:0/2242317056 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd6c154e80 con 0x7fbd6c15af10 2024-02-19T15:26:28.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.216+0000 7fbd71c00640 1 -- 172.21.15.47:0/2242317056 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd6c155f30 con 0x7fbd6c15af10 2024-02-19T15:26:28.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.216+0000 7fbd63400640 1 -- 172.21.15.47:0/2242317056 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fbd64012820 con 0x7fbd6c15af10 2024-02-19T15:26:28.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.216+0000 7fbd63400640 1 --2- 172.21.15.47:0/2242317056 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbd48066e40 0x7fbd48069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:28.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.216+0000 7fbd63400640 1 -- 172.21.15.47:0/2242317056 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd640849b0 con 0x7fbd6c15af10 2024-02-19T15:26:28.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.216+0000 7fbd6aa00640 1 --2- 172.21.15.47:0/2242317056 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbd48066e40 0x7fbd48069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:28.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.216+0000 7fbd6aa00640 1 --2- 172.21.15.47:0/2242317056 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbd48066e40 0x7fbd48069300 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c0af000 tx=0x7fbd5c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:28.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.220+0000 7fbd63400640 1 -- 172.21.15.47:0/2242317056 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fbd6c155f30 con 0x7fbd6c15af10 2024-02-19T15:26:28.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:28.424+0000 7fbd71c00640 1 -- 172.21.15.47:0/2242317056 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fbd6c155ec0 con 0x7fbd6c15af10 2024-02-19T15:26:29.306 INFO:tasks.ceph.mds.d.smithi116.stderr:2024-02-19T15:26:29.305+0000 7fbc87400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T15:26:29.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.340+0000 7fbd63400640 1 -- 172.21.15.47:0/2242317056 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v57) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbd640170b0 con 0x7fbd6c15af10 2024-02-19T15:26:29.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.344+0000 7fbd71c00640 1 -- 172.21.15.47:0/2242317056 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbd48066e40 msgr2=0x7fbd48069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:29.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.344+0000 7fbd71c00640 1 --2- 172.21.15.47:0/2242317056 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbd48066e40 0x7fbd48069300 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c0af000 tx=0x7fbd5c008040 comp rx=0 tx=0).stop 2024-02-19T15:26:29.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.344+0000 7fbd71c00640 1 -- 172.21.15.47:0/2242317056 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd6c15af10 msgr2=0x7fbd6c0b4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:29.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.344+0000 7fbd71c00640 1 --2- 172.21.15.47:0/2242317056 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd6c15af10 0x7fbd6c0b4300 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400cbc0 tx=0x7fbd6400a430 comp rx=0 tx=0).stop 2024-02-19T15:26:29.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.344+0000 7fbd71c00640 1 -- 172.21.15.47:0/2242317056 shutdown_connections 2024-02-19T15:26:29.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.344+0000 7fbd71c00640 1 --2- 172.21.15.47:0/2242317056 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbd48066e40 0x7fbd48069300 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:29.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.344+0000 7fbd71c00640 1 --2- 172.21.15.47:0/2242317056 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd6c15af10 0x7fbd6c0b4300 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:29.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.344+0000 7fbd71c00640 1 --2- 172.21.15.47:0/2242317056 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd6c0cd880 0x7fbd6c0b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:29.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.344+0000 7fbd71c00640 1 --2- 172.21.15.47:0/2242317056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd6c0ca8f0 0x7fbd6c154940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:29.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.344+0000 7fbd71c00640 1 -- 172.21.15.47:0/2242317056 >> 172.21.15.47:0/2242317056 conn(0x7fbd6c0cfc20 msgr2=0x7fbd6c0c45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:29.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.344+0000 7fbd71c00640 1 -- 172.21.15.47:0/2242317056 shutdown_connections 2024-02-19T15:26:29.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.344+0000 7fbd71c00640 1 -- 172.21.15.47:0/2242317056 wait complete. 2024-02-19T15:26:29.370 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T15:26:29.371 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T15:26:29.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.512+0000 7fe1d8200640 1 Processor -- start 2024-02-19T15:26:29.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.512+0000 7fe1d8200640 1 -- start start 2024-02-19T15:26:29.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.516+0000 7fe1d8200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1d015af10 0x7fe1d006e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:29.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.516+0000 7fe1d8200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe1d00ca8f0 0x7fe1d00cad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:29.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.516+0000 7fe1d8200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe1d00cb290 0x7fe1d015eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:29.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.516+0000 7fe1d8200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d006e8a0 con 0x7fe1d00ca8f0 2024-02-19T15:26:29.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.516+0000 7fe1d8200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d0071c70 con 0x7fe1d00cb290 2024-02-19T15:26:29.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.516+0000 7fe1d8200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d006cef0 con 0x7fe1d015af10 2024-02-19T15:26:29.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.516+0000 7fe1d4e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe1d00ca8f0 0x7fe1d00cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:29.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.516+0000 7fe1d4e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe1d00ca8f0 0x7fe1d00cad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58408/0 (socket says 172.21.15.47:58408) 2024-02-19T15:26:29.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.516+0000 7fe1d4e00640 1 -- 172.21.15.47:0/171806698 learned_addr learned my addr 172.21.15.47:0/171806698 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:29.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.516+0000 7fe1d5800640 1 --2- 172.21.15.47:0/171806698 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1d015af10 0x7fe1d006e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:29.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.516+0000 7fe1d6200640 1 --2- 172.21.15.47:0/171806698 >> v2:172.21.15.116:3300/0 conn(0x7fe1d00cb290 0x7fe1d015eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:29.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.516+0000 7fe1d4e00640 1 -- 172.21.15.47:0/171806698 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1d015af10 msgr2=0x7fe1d006e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:29.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.516+0000 7fe1d4e00640 1 --2- 172.21.15.47:0/171806698 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1d015af10 0x7fe1d006e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:29.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.516+0000 7fe1d4e00640 1 -- 172.21.15.47:0/171806698 >> v2:172.21.15.116:3300/0 conn(0x7fe1d00cb290 msgr2=0x7fe1d015eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:29.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.516+0000 7fe1d4e00640 1 --2- 172.21.15.47:0/171806698 >> v2:172.21.15.116:3300/0 conn(0x7fe1d00cb290 0x7fe1d015eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:29.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.516+0000 7fe1d4e00640 1 -- 172.21.15.47:0/171806698 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d00ca610 con 0x7fe1d00ca8f0 2024-02-19T15:26:29.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.516+0000 7fe1d6200640 1 --2- 172.21.15.47:0/171806698 >> v2:172.21.15.116:3300/0 conn(0x7fe1d00cb290 0x7fe1d015eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:29.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1d4e00640 1 --2- 172.21.15.47:0/171806698 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d00ca8f0 0x7fe1d00cad50 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fe1d006c200 tx=0x7fe1bc01e870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=851f00225809c3eb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:29.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1cfe00640 1 -- 172.21.15.47:0/171806698 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc0a7900 con 0x7fe1d00ca8f0 2024-02-19T15:26:29.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1cfe00640 1 -- 172.21.15.47:0/171806698 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1bc0a7a60 con 0x7fe1d00ca8f0 2024-02-19T15:26:29.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1cfe00640 1 -- 172.21.15.47:0/171806698 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc0a7df0 con 0x7fe1d00ca8f0 2024-02-19T15:26:29.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1d8200640 1 -- 172.21.15.47:0/171806698 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d00ca8f0 msgr2=0x7fe1d00cad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:29.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1d8200640 1 --2- 172.21.15.47:0/171806698 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d00ca8f0 0x7fe1d00cad50 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fe1d006c200 tx=0x7fe1bc01e870 comp rx=0 tx=0).stop 2024-02-19T15:26:29.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1d8200640 1 -- 172.21.15.47:0/171806698 shutdown_connections 2024-02-19T15:26:29.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1d8200640 1 --2- 172.21.15.47:0/171806698 >> v2:172.21.15.116:3300/0 conn(0x7fe1d00cb290 0x7fe1d015eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:29.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1d8200640 1 --2- 172.21.15.47:0/171806698 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d00ca8f0 0x7fe1d00cad50 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:29.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1d8200640 1 --2- 172.21.15.47:0/171806698 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1d015af10 0x7fe1d006e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:29.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1d8200640 1 -- 172.21.15.47:0/171806698 >> 172.21.15.47:0/171806698 conn(0x7fe1d00cfc20 msgr2=0x7fe1d00d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:29.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1d8200640 1 -- 172.21.15.47:0/171806698 shutdown_connections 2024-02-19T15:26:29.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1d8200640 1 -- 172.21.15.47:0/171806698 wait complete. 2024-02-19T15:26:29.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1d8200640 1 Processor -- start 2024-02-19T15:26:29.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1d8200640 1 -- start start 2024-02-19T15:26:29.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1d8200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1d00ca8f0 0x7fe1d015c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:29.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1d8200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1d00cb290 0x7fe1d0155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:29.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1d8200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d015af10 0x7fe1d0164ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:29.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1d8200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d0165c20 con 0x7fe1d015af10 2024-02-19T15:26:29.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1d8200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d015bcb0 con 0x7fe1d00cb290 2024-02-19T15:26:29.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.520+0000 7fe1d8200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d015b6a0 con 0x7fe1d00ca8f0 2024-02-19T15:26:29.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.524+0000 7fe1d6200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d015af10 0x7fe1d0164ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:29.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.524+0000 7fe1d6200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d015af10 0x7fe1d0164ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58410/0 (socket says 172.21.15.47:58410) 2024-02-19T15:26:29.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.524+0000 7fe1d6200640 1 -- 172.21.15.47:0/3906445127 learned_addr learned my addr 172.21.15.47:0/3906445127 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:29.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.524+0000 7fe1d5800640 1 --2- 172.21.15.47:0/3906445127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1d00ca8f0 0x7fe1d015c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:29.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.524+0000 7fe1d4e00640 1 --2- 172.21.15.47:0/3906445127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1d00cb290 0x7fe1d0155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:29.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.524+0000 7fe1d6200640 1 -- 172.21.15.47:0/3906445127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1d00ca8f0 msgr2=0x7fe1d015c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:29.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.524+0000 7fe1d6200640 1 --2- 172.21.15.47:0/3906445127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1d00ca8f0 0x7fe1d015c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:29.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.524+0000 7fe1d6200640 1 -- 172.21.15.47:0/3906445127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1d00cb290 msgr2=0x7fe1d0155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:29.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.524+0000 7fe1d6200640 1 --2- 172.21.15.47:0/3906445127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1d00cb290 0x7fe1d0155750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:29.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.524+0000 7fe1d6200640 1 -- 172.21.15.47:0/3906445127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1bc00b770 con 0x7fe1d015af10 2024-02-19T15:26:29.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.524+0000 7fe1d6200640 1 --2- 172.21.15.47:0/3906445127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d015af10 0x7fe1d0164ff0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8007de0 tx=0x7fe1c800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:29.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.524+0000 7fe1cd600640 1 -- 172.21.15.47:0/3906445127 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8003ba0 con 0x7fe1d015af10 2024-02-19T15:26:29.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.524+0000 7fe1cd600640 1 -- 172.21.15.47:0/3906445127 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c800f040 con 0x7fe1d015af10 2024-02-19T15:26:29.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.524+0000 7fe1d8200640 1 -- 172.21.15.47:0/3906445127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d0165940 con 0x7fe1d015af10 2024-02-19T15:26:29.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.524+0000 7fe1cd600640 1 -- 172.21.15.47:0/3906445127 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c800de80 con 0x7fe1d015af10 2024-02-19T15:26:29.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.524+0000 7fe1d8200640 1 -- 172.21.15.47:0/3906445127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d0155c90 con 0x7fe1d015af10 2024-02-19T15:26:29.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.524+0000 7fe1d8200640 1 -- 172.21.15.47:0/3906445127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d015b500 con 0x7fe1d015af10 2024-02-19T15:26:29.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.524+0000 7fe1cd600640 1 -- 172.21.15.47:0/3906445127 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe1c8004030 con 0x7fe1d015af10 2024-02-19T15:26:29.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.524+0000 7fe1cd600640 1 --2- 172.21.15.47:0/3906445127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe1b0066e40 0x7fe1b0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:29.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.524+0000 7fe1cd600640 1 -- 172.21.15.47:0/3906445127 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1c8084e20 con 0x7fe1d015af10 2024-02-19T15:26:29.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.528+0000 7fe1d5800640 1 --2- 172.21.15.47:0/3906445127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe1b0066e40 0x7fe1b0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:29.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.528+0000 7fe1d5800640 1 --2- 172.21.15.47:0/3906445127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe1b0066e40 0x7fe1b0069300 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4009ac0 tx=0x7fe1c4009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:29.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.532+0000 7fe1cd600640 1 -- 172.21.15.47:0/3906445127 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe1d015b500 con 0x7fe1d015af10 2024-02-19T15:26:29.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:29.728+0000 7fe1d8200640 1 -- 172.21.15.47:0/3906445127 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe1d015b500 con 0x7fe1d015af10 2024-02-19T15:26:30.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.343+0000 7fe1cd600640 1 -- 172.21.15.47:0/3906445127 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v58) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe1d015b500 con 0x7fe1d015af10 2024-02-19T15:26:30.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.343+0000 7fe1d8200640 1 -- 172.21.15.47:0/3906445127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe1b0066e40 msgr2=0x7fe1b0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:30.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.343+0000 7fe1d8200640 1 --2- 172.21.15.47:0/3906445127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe1b0066e40 0x7fe1b0069300 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4009ac0 tx=0x7fe1c4009190 comp rx=0 tx=0).stop 2024-02-19T15:26:30.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.343+0000 7fe1d8200640 1 -- 172.21.15.47:0/3906445127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d015af10 msgr2=0x7fe1d0164ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:30.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.343+0000 7fe1d8200640 1 --2- 172.21.15.47:0/3906445127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d015af10 0x7fe1d0164ff0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8007de0 tx=0x7fe1c800d4a0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.343+0000 7fe1d8200640 1 -- 172.21.15.47:0/3906445127 shutdown_connections 2024-02-19T15:26:30.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.343+0000 7fe1d8200640 1 --2- 172.21.15.47:0/3906445127 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe1b0066e40 0x7fe1b0069300 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.343+0000 7fe1d8200640 1 --2- 172.21.15.47:0/3906445127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d015af10 0x7fe1d0164ff0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.343+0000 7fe1d8200640 1 --2- 172.21.15.47:0/3906445127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1d00cb290 0x7fe1d0155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.343+0000 7fe1d8200640 1 --2- 172.21.15.47:0/3906445127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1d00ca8f0 0x7fe1d015c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.343+0000 7fe1d8200640 1 -- 172.21.15.47:0/3906445127 >> 172.21.15.47:0/3906445127 conn(0x7fe1d00cfc20 msgr2=0x7fe1d00b5350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:30.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.343+0000 7fe1d8200640 1 -- 172.21.15.47:0/3906445127 shutdown_connections 2024-02-19T15:26:30.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.343+0000 7fe1d8200640 1 -- 172.21.15.47:0/3906445127 wait complete. 2024-02-19T15:26:30.372 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T15:26:30.372 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:26:30.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.511+0000 7f0bf8a00640 1 Processor -- start 2024-02-19T15:26:30.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.511+0000 7f0bf8a00640 1 -- start start 2024-02-19T15:26:30.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.511+0000 7f0bf8a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0bf415dd70 0x7f0bf40ae860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:30.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.511+0000 7f0bf8a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0bf40aeda0 0x7f0bf406d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:30.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.511+0000 7f0bf8a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf406dab0 con 0x7f0bf406d660 2024-02-19T15:26:30.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.511+0000 7f0bf8a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf415f3f0 con 0x7f0bf415dd70 2024-02-19T15:26:30.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.511+0000 7f0bf8a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf406c3e0 con 0x7f0bf40aeda0 2024-02-19T15:26:30.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.511+0000 7f0bf3400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0bf415dd70 0x7f0bf40ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:30.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.511+0000 7f0bf3400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0bf415dd70 0x7f0bf40ae860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58414/0 (socket says 172.21.15.47:58414) 2024-02-19T15:26:30.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.511+0000 7f0bf3400640 1 -- 172.21.15.47:0/2950041951 learned_addr learned my addr 172.21.15.47:0/2950041951 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:30.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.511+0000 7f0beaa00640 1 --2- 172.21.15.47:0/2950041951 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0bf40aeda0 0x7f0bf406d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:30.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.511+0000 7f0bf3400640 1 -- 172.21.15.47:0/2950041951 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0bf40aeda0 msgr2=0x7f0bf406d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:30.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.511+0000 7f0bf3400640 1 --2- 172.21.15.47:0/2950041951 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0bf40aeda0 0x7f0bf406d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.511+0000 7f0bf3400640 1 -- 172.21.15.47:0/2950041951 >> v1:172.21.15.116:6789/0 conn(0x7f0bf406d660 legacy=0x7f0bf40d19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:30.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.511+0000 7f0bf3400640 1 -- 172.21.15.47:0/2950041951 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf406c680 con 0x7f0bf415dd70 2024-02-19T15:26:30.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.511+0000 7f0beaa00640 1 --2- 172.21.15.47:0/2950041951 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0bf40aeda0 0x7f0bf406d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:30.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.515+0000 7f0bf3400640 1 --2- 172.21.15.47:0/2950041951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bf415dd70 0x7f0bf40ae860 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f0be4009870 tx=0x7f0be401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8fedff6864b3f6a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:30.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.515+0000 7f0bf2a00640 1 -- 172.21.15.47:0/2950041951 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be40a6040 con 0x7f0bf415dd70 2024-02-19T15:26:30.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.515+0000 7f0bf2a00640 1 -- 172.21.15.47:0/2950041951 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be40aa600 con 0x7f0bf415dd70 2024-02-19T15:26:30.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.515+0000 7f0bf2a00640 1 -- 172.21.15.47:0/2950041951 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be40aa990 con 0x7f0bf415dd70 2024-02-19T15:26:30.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.515+0000 7f0bf8a00640 1 -- 172.21.15.47:0/2950041951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bf415dd70 msgr2=0x7f0bf40ae860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:30.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.515+0000 7f0bf8a00640 1 --2- 172.21.15.47:0/2950041951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bf415dd70 0x7f0bf40ae860 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f0be4009870 tx=0x7f0be401c4a0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.515+0000 7f0bf8a00640 1 -- 172.21.15.47:0/2950041951 shutdown_connections 2024-02-19T15:26:30.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.515+0000 7f0bf8a00640 1 --2- 172.21.15.47:0/2950041951 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0bf40aeda0 0x7f0bf406d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.515+0000 7f0bf8a00640 1 --2- 172.21.15.47:0/2950041951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bf415dd70 0x7f0bf40ae860 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.515+0000 7f0bf8a00640 1 -- 172.21.15.47:0/2950041951 >> 172.21.15.47:0/2950041951 conn(0x7f0bf409a9d0 msgr2=0x7f0bf409ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:30.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.515+0000 7f0bf8a00640 1 -- 172.21.15.47:0/2950041951 shutdown_connections 2024-02-19T15:26:30.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.515+0000 7f0bf8a00640 1 -- 172.21.15.47:0/2950041951 wait complete. 2024-02-19T15:26:30.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf8a00640 1 Processor -- start 2024-02-19T15:26:30.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf8a00640 1 -- start start 2024-02-19T15:26:30.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf8a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0bf406d660 0x7f0bf4143720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:30.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf8a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bf40aeda0 0x7f0bf415a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:30.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf8a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bf415dd70 0x7f0bf40b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:30.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf8a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf415ac20 con 0x7f0bf415dd70 2024-02-19T15:26:30.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf8a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf4161a60 con 0x7f0bf40aeda0 2024-02-19T15:26:30.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf8a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf4161d40 con 0x7f0bf406d660 2024-02-19T15:26:30.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bf415dd70 0x7f0bf40b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:30.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf3400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0bf406d660 0x7f0bf4143720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:30.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0beaa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bf40aeda0 0x7f0bf415a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:30.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf3400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0bf406d660 0x7f0bf4143720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:51834/0 (socket says 172.21.15.47:51834) 2024-02-19T15:26:30.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf3400640 1 -- 172.21.15.47:0/3355573939 learned_addr learned my addr 172.21.15.47:0/3355573939 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:30.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bf415dd70 0x7f0bf40b6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58428/0 (socket says 172.21.15.47:58428) 2024-02-19T15:26:30.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf3e00640 1 -- 172.21.15.47:0/3355573939 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0bf406d660 msgr2=0x7f0bf4143720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:30.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf3e00640 1 --2- 172.21.15.47:0/3355573939 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0bf406d660 0x7f0bf4143720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf3e00640 1 -- 172.21.15.47:0/3355573939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bf40aeda0 msgr2=0x7f0bf415a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:30.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf3e00640 1 --2- 172.21.15.47:0/3355573939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bf40aeda0 0x7f0bf415a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf3e00640 1 -- 172.21.15.47:0/3355573939 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be40094d0 con 0x7f0bf415dd70 2024-02-19T15:26:30.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf3e00640 1 --2- 172.21.15.47:0/3355573939 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bf415dd70 0x7f0bf40b6de0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f0bec00c700 tx=0x7f0bec00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:30.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bebe00640 1 -- 172.21.15.47:0/3355573939 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec003b80 con 0x7f0bf415dd70 2024-02-19T15:26:30.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bebe00640 1 -- 172.21.15.47:0/3355573939 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bec003ce0 con 0x7f0bf415dd70 2024-02-19T15:26:30.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bebe00640 1 -- 172.21.15.47:0/3355573939 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec00aa30 con 0x7f0bf415dd70 2024-02-19T15:26:30.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf8a00640 1 -- 172.21.15.47:0/3355573939 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf4162020 con 0x7f0bf415dd70 2024-02-19T15:26:30.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.519+0000 7f0bf8a00640 1 -- 172.21.15.47:0/3355573939 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf41625e0 con 0x7f0bf415dd70 2024-02-19T15:26:30.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.523+0000 7f0bf8a00640 1 -- 172.21.15.47:0/3355573939 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf41627a0 con 0x7f0bf415dd70 2024-02-19T15:26:30.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.523+0000 7f0bebe00640 1 -- 172.21.15.47:0/3355573939 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0bec00ab90 con 0x7f0bf415dd70 2024-02-19T15:26:30.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.523+0000 7f0bebe00640 1 --2- 172.21.15.47:0/3355573939 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0bd0066e40 0x7f0bd0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:30.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.523+0000 7f0bebe00640 1 -- 172.21.15.47:0/3355573939 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0bec084640 con 0x7f0bf415dd70 2024-02-19T15:26:30.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.523+0000 7f0bf3400640 1 --2- 172.21.15.47:0/3355573939 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0bd0066e40 0x7f0bd0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:30.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.523+0000 7f0bf3400640 1 --2- 172.21.15.47:0/3355573939 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0bd0066e40 0x7f0bd0069300 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0be40ac040 tx=0x7f0be40b5040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:30.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.527+0000 7f0bebe00640 1 -- 172.21.15.47:0/3355573939 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0bec051570 con 0x7f0bf415dd70 2024-02-19T15:26:30.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.723+0000 7f0bf8a00640 1 -- 172.21.15.47:0/3355573939 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bf4075360 con 0x7f0bf415dd70 2024-02-19T15:26:30.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.723+0000 7f0bebe00640 1 -- 172.21.15.47:0/3355573939 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f0bec054c30 con 0x7f0bf415dd70 2024-02-19T15:26:30.729 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:30.729 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/2831452879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2831452879},{"type":"v1","addr":"172.21.15.116:6836","nonce":2831452879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6191,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/946400730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":946400730},{"type":"v1","addr":"172.21.15.47:6835","nonce":946400730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6844,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/68555355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":68555355},{"type":"v1","addr":"172.21.15.47:6833","nonce":68555355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:26:26.721109+0000","modified":"2024-02-19T15:26:29.735209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6850":{"gid":6850,"name":"d","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6837/547176070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":547176070},{"type":"v1","addr":"172.21.15.116:6837","nonce":547176070}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T15:26:30.729 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 58 2024-02-19T15:26:30.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.727+0000 7f0bf8a00640 1 -- 172.21.15.47:0/3355573939 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0bd0066e40 msgr2=0x7f0bd0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:30.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.727+0000 7f0bf8a00640 1 --2- 172.21.15.47:0/3355573939 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0bd0066e40 0x7f0bd0069300 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0be40ac040 tx=0x7f0be40b5040 comp rx=0 tx=0).stop 2024-02-19T15:26:30.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.727+0000 7f0bf8a00640 1 -- 172.21.15.47:0/3355573939 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bf415dd70 msgr2=0x7f0bf40b6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:30.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.727+0000 7f0bf8a00640 1 --2- 172.21.15.47:0/3355573939 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bf415dd70 0x7f0bf40b6de0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f0bec00c700 tx=0x7f0bec00cbc0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.727+0000 7f0bf8a00640 1 -- 172.21.15.47:0/3355573939 shutdown_connections 2024-02-19T15:26:30.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.727+0000 7f0bf8a00640 1 --2- 172.21.15.47:0/3355573939 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0bd0066e40 0x7f0bd0069300 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.727+0000 7f0bf8a00640 1 --2- 172.21.15.47:0/3355573939 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bf415dd70 0x7f0bf40b6de0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.727+0000 7f0bf8a00640 1 --2- 172.21.15.47:0/3355573939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bf40aeda0 0x7f0bf415a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.727+0000 7f0bf8a00640 1 --2- 172.21.15.47:0/3355573939 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0bf406d660 0x7f0bf4143720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.727+0000 7f0bf8a00640 1 -- 172.21.15.47:0/3355573939 >> 172.21.15.47:0/3355573939 conn(0x7f0bf409a9d0 msgr2=0x7f0bf40b04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:30.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.727+0000 7f0bf8a00640 1 -- 172.21.15.47:0/3355573939 shutdown_connections 2024-02-19T15:26:30.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.727+0000 7f0bf8a00640 1 -- 172.21.15.47:0/3355573939 wait complete. 2024-02-19T15:26:30.750 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:26:30.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.887+0000 7f2373200640 1 Processor -- start 2024-02-19T15:26:30.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.887+0000 7f2373200640 1 -- start start 2024-02-19T15:26:30.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.887+0000 7f2373200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f236c15bcd0 0x7f236c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:30.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.887+0000 7f2373200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f236c0cedb0 0x7f236c0cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:30.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.887+0000 7f2373200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f236c0cc700 con 0x7f236c0d2710 2024-02-19T15:26:30.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.887+0000 7f2373200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f236c0cf5e0 con 0x7f236c0cedb0 2024-02-19T15:26:30.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.887+0000 7f2373200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f236c0cf300 con 0x7f236c15bcd0 2024-02-19T15:26:30.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.887+0000 7f2363e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f236c0cedb0 0x7f236c0cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:30.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.887+0000 7f2371200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f236c0d2710 0x7f236c15d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:40776/0 (socket says 172.21.15.47:40776) 2024-02-19T15:26:30.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.887+0000 7f2371200640 1 -- 172.21.15.47:0/1356228213 learned_addr learned my addr 172.21.15.47:0/1356228213 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:30.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.887+0000 7f236be00640 1 --2- 172.21.15.47:0/1356228213 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f236c15bcd0 0x7f236c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:30.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.887+0000 7f2363e00640 1 -- 172.21.15.47:0/1356228213 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f236c15bcd0 msgr2=0x7f236c0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:30.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.887+0000 7f2363e00640 1 --2- 172.21.15.47:0/1356228213 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f236c15bcd0 0x7f236c0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.887+0000 7f2363e00640 1 -- 172.21.15.47:0/1356228213 >> v1:172.21.15.47:6789/0 conn(0x7f236c0d2710 legacy=0x7f236c15d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:30.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.887+0000 7f2363e00640 1 -- 172.21.15.47:0/1356228213 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236c0cc9a0 con 0x7f236c0cedb0 2024-02-19T15:26:30.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.887+0000 7f236b400640 1 -- 172.21.15.47:0/1356228213 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980723625 0 0) 0x7f236c0cc700 con 0x7f236c0d2710 2024-02-19T15:26:30.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.891+0000 7f2363e00640 1 --2- 172.21.15.47:0/1356228213 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f236c0cedb0 0x7f236c0cfc50 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f23540097c0 tx=0x7f235401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc862049214ec6e9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:30.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.891+0000 7f236b400640 1 -- 172.21.15.47:0/1356228213 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23540ad680 con 0x7f236c0cedb0 2024-02-19T15:26:30.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.891+0000 7f236b400640 1 -- 172.21.15.47:0/1356228213 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23540ad7e0 con 0x7f236c0cedb0 2024-02-19T15:26:30.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.891+0000 7f236b400640 1 -- 172.21.15.47:0/1356228213 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23540adb70 con 0x7f236c0cedb0 2024-02-19T15:26:30.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.891+0000 7f2373200640 1 -- 172.21.15.47:0/1356228213 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f236c0cedb0 msgr2=0x7f236c0cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:30.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.891+0000 7f2373200640 1 --2- 172.21.15.47:0/1356228213 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f236c0cedb0 0x7f236c0cfc50 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f23540097c0 tx=0x7f235401c470 comp rx=0 tx=0).stop 2024-02-19T15:26:30.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.891+0000 7f2373200640 1 -- 172.21.15.47:0/1356228213 shutdown_connections 2024-02-19T15:26:30.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.891+0000 7f2373200640 1 --2- 172.21.15.47:0/1356228213 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f236c0cedb0 0x7f236c0cfc50 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.891+0000 7f2373200640 1 --2- 172.21.15.47:0/1356228213 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f236c15bcd0 0x7f236c0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.891+0000 7f2373200640 1 -- 172.21.15.47:0/1356228213 >> 172.21.15.47:0/1356228213 conn(0x7f236c0d1bf0 msgr2=0x7f236c0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:30.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.891+0000 7f2373200640 1 -- 172.21.15.47:0/1356228213 shutdown_connections 2024-02-19T15:26:30.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.891+0000 7f2373200640 1 -- 172.21.15.47:0/1356228213 wait complete. 2024-02-19T15:26:30.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.891+0000 7f2373200640 1 Processor -- start 2024-02-19T15:26:30.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.891+0000 7f2373200640 1 -- start start 2024-02-19T15:26:30.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f2373200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f236c0cedb0 0x7f236c0ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:30.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f2373200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f236c0d2710 0x7f236c0cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:30.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f2373200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f236c15bcd0 0x7f236c154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:30.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f2373200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c0c9cb0 con 0x7f236c0cedb0 2024-02-19T15:26:30.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f2373200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c0c9030 con 0x7f236c15bcd0 2024-02-19T15:26:30.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f2373200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f236c0c9310 con 0x7f236c0d2710 2024-02-19T15:26:30.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f236be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f236c0cedb0 0x7f236c0ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:30.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f236be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f236c0cedb0 0x7f236c0ca0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58450/0 (socket says 172.21.15.47:58450) 2024-02-19T15:26:30.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f236be00640 1 -- 172.21.15.47:0/210998492 learned_addr learned my addr 172.21.15.47:0/210998492 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:30.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f2363e00640 1 --2- 172.21.15.47:0/210998492 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f236c0d2710 0x7f236c0cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:30.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f2371200640 1 --2- 172.21.15.47:0/210998492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f236c15bcd0 0x7f236c154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:30.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f236be00640 1 -- 172.21.15.47:0/210998492 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f236c0d2710 msgr2=0x7f236c0cb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:30.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f236be00640 1 --2- 172.21.15.47:0/210998492 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f236c0d2710 0x7f236c0cb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f236be00640 1 -- 172.21.15.47:0/210998492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f236c15bcd0 msgr2=0x7f236c154f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:30.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f236be00640 1 --2- 172.21.15.47:0/210998492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f236c15bcd0 0x7f236c154f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:30.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f236be00640 1 -- 172.21.15.47:0/210998492 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2354009420 con 0x7f236c0cedb0 2024-02-19T15:26:30.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f236be00640 1 --2- 172.21.15.47:0/210998492 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f236c0cedb0 0x7f236c0ca0d0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f235c00e930 tx=0x7f235c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:30.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f2368c00640 1 -- 172.21.15.47:0/210998492 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c009690 con 0x7f236c0cedb0 2024-02-19T15:26:30.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f2368c00640 1 -- 172.21.15.47:0/210998492 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235c00ca00 con 0x7f236c0cedb0 2024-02-19T15:26:30.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f2373200640 1 -- 172.21.15.47:0/210998492 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f236c0cdb40 con 0x7f236c0cedb0 2024-02-19T15:26:30.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f2368c00640 1 -- 172.21.15.47:0/210998492 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c00cd70 con 0x7f236c0cedb0 2024-02-19T15:26:30.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f2373200640 1 -- 172.21.15.47:0/210998492 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f236c0c98d0 con 0x7f236c0cedb0 2024-02-19T15:26:30.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.895+0000 7f2373200640 1 -- 172.21.15.47:0/210998492 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f236c0c9f20 con 0x7f236c0cedb0 2024-02-19T15:26:30.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.899+0000 7f2368c00640 1 -- 172.21.15.47:0/210998492 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f235c013470 con 0x7f236c0cedb0 2024-02-19T15:26:30.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.899+0000 7f2368c00640 1 --2- 172.21.15.47:0/210998492 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2348066e40 0x7f2348069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:30.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.899+0000 7f2368c00640 1 -- 172.21.15.47:0/210998492 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f235c084b80 con 0x7f236c0cedb0 2024-02-19T15:26:30.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.899+0000 7f2363e00640 1 --2- 172.21.15.47:0/210998492 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2348066e40 0x7f2348069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:30.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.899+0000 7f2363e00640 1 --2- 172.21.15.47:0/210998492 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2348066e40 0x7f2348069300 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f23540af000 tx=0x7f23540a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:30.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:30.903+0000 7f2368c00640 1 -- 172.21.15.47:0/210998492 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f235c051b70 con 0x7f236c0cedb0 2024-02-19T15:26:31.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.067+0000 7f2373200640 1 -- 172.21.15.47:0/210998492 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f236c075360 con 0x7f236c0cedb0 2024-02-19T15:26:31.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.067+0000 7f2368c00640 1 -- 172.21.15.47:0/210998492 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16620 (secure 0 0 0) 0x7f235c017090 con 0x7f236c0cedb0 2024-02-19T15:26:31.073 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:31.074 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":93,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:26:28.317179+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:26:25.124594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-02-19T15:26:25.706971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:26:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.071+0000 7f2373200640 1 -- 172.21.15.47:0/210998492 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2348066e40 msgr2=0x7f2348069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.071+0000 7f2373200640 1 --2- 172.21.15.47:0/210998492 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2348066e40 0x7f2348069300 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f23540af000 tx=0x7f23540a7000 comp rx=0 tx=0).stop 2024-02-19T15:26:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.071+0000 7f2373200640 1 -- 172.21.15.47:0/210998492 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f236c0cedb0 msgr2=0x7f236c0ca0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.071+0000 7f2373200640 1 --2- 172.21.15.47:0/210998492 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f236c0cedb0 0x7f236c0ca0d0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f235c00e930 tx=0x7f235c00edf0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.071+0000 7f2373200640 1 -- 172.21.15.47:0/210998492 shutdown_connections 2024-02-19T15:26:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.071+0000 7f2373200640 1 --2- 172.21.15.47:0/210998492 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2348066e40 0x7f2348069300 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.071+0000 7f2373200640 1 --2- 172.21.15.47:0/210998492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f236c15bcd0 0x7f236c154f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.071+0000 7f2373200640 1 --2- 172.21.15.47:0/210998492 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f236c0d2710 0x7f236c0cb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.071+0000 7f2373200640 1 --2- 172.21.15.47:0/210998492 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f236c0cedb0 0x7f236c0ca0d0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.071+0000 7f2373200640 1 -- 172.21.15.47:0/210998492 >> 172.21.15.47:0/210998492 conn(0x7f236c0d1bf0 msgr2=0x7f236c15b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.071+0000 7f2373200640 1 -- 172.21.15.47:0/210998492 shutdown_connections 2024-02-19T15:26:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.071+0000 7f2373200640 1 -- 172.21.15.47:0/210998492 wait complete. 2024-02-19T15:26:31.094 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T15:26:31.094 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:26:31.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.231+0000 7f2c6e200640 1 Processor -- start 2024-02-19T15:26:31.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.231+0000 7f2c6e200640 1 -- start start 2024-02-19T15:26:31.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.231+0000 7f2c6e200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c680cea80 0x7f2c680727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:31.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.231+0000 7f2c6e200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2c6815c390 0x7f2c6815c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:31.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.231+0000 7f2c6e200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6815d280 con 0x7f2c68145590 2024-02-19T15:26:31.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.231+0000 7f2c6e200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c68070ff0 con 0x7f2c6815c390 2024-02-19T15:26:31.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.231+0000 7f2c6e200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6815cd60 con 0x7f2c680cea80 2024-02-19T15:26:31.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.231+0000 7f2c5fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c680cea80 0x7f2c680727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:31.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.231+0000 7f2c67e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2c6815c390 0x7f2c6815c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:31.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.231+0000 7f2c5fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c680cea80 0x7f2c680727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:51868/0 (socket says 172.21.15.47:51868) 2024-02-19T15:26:31.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.231+0000 7f2c5fe00640 1 -- 172.21.15.47:0/27380585 learned_addr learned my addr 172.21.15.47:0/27380585 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:31.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.231+0000 7f2c67400640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2c68145590 0x7f2c6806c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:53230/0 (socket says 172.21.15.47:53230) 2024-02-19T15:26:31.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.231+0000 7f2c66a00640 1 -- 172.21.15.47:0/27380585 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004869075 0 0) 0x7f2c6815d280 con 0x7f2c68145590 2024-02-19T15:26:31.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.231+0000 7f2c66a00640 1 -- 172.21.15.47:0/27380585 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c4c003530 con 0x7f2c68145590 2024-02-19T15:26:31.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.231+0000 7f2c5fe00640 1 -- 172.21.15.47:0/27380585 >> v2:172.21.15.116:3300/0 conn(0x7f2c6815c390 msgr2=0x7f2c6815c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:31.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.231+0000 7f2c5fe00640 1 --2- 172.21.15.47:0/27380585 >> v2:172.21.15.116:3300/0 conn(0x7f2c6815c390 0x7f2c6815c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.231+0000 7f2c5fe00640 1 -- 172.21.15.47:0/27380585 >> v1:172.21.15.116:6789/0 conn(0x7f2c68145590 legacy=0x7f2c6806c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:31.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.231+0000 7f2c5fe00640 1 -- 172.21.15.47:0/27380585 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c680709e0 con 0x7f2c680cea80 2024-02-19T15:26:31.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.231+0000 7f2c67e00640 1 --2- 172.21.15.47:0/27380585 >> v2:172.21.15.116:3300/0 conn(0x7f2c6815c390 0x7f2c6815c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:31.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.235+0000 7f2c5fe00640 1 --2- 172.21.15.47:0/27380585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c680cea80 0x7f2c680727b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2c50009870 tx=0x7f2c5001c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6787634bb4f408c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:31.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.235+0000 7f2c66a00640 1 -- 172.21.15.47:0/27380585 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c500ab650 con 0x7f2c680cea80 2024-02-19T15:26:31.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.235+0000 7f2c66a00640 1 -- 172.21.15.47:0/27380585 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c500ab7b0 con 0x7f2c680cea80 2024-02-19T15:26:31.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.235+0000 7f2c66a00640 1 -- 172.21.15.47:0/27380585 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c500abb40 con 0x7f2c680cea80 2024-02-19T15:26:31.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.235+0000 7f2c6e200640 1 -- 172.21.15.47:0/27380585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c680cea80 msgr2=0x7f2c680727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:31.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.235+0000 7f2c6e200640 1 --2- 172.21.15.47:0/27380585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c680cea80 0x7f2c680727b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2c50009870 tx=0x7f2c5001c560 comp rx=0 tx=0).stop 2024-02-19T15:26:31.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.235+0000 7f2c6e200640 1 -- 172.21.15.47:0/27380585 shutdown_connections 2024-02-19T15:26:31.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.235+0000 7f2c6e200640 1 --2- 172.21.15.47:0/27380585 >> v2:172.21.15.116:3300/0 conn(0x7f2c6815c390 0x7f2c6815c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.235+0000 7f2c6e200640 1 --2- 172.21.15.47:0/27380585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c680cea80 0x7f2c680727b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.235+0000 7f2c6e200640 1 -- 172.21.15.47:0/27380585 >> 172.21.15.47:0/27380585 conn(0x7f2c680b16a0 msgr2=0x7f2c680b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:31.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.235+0000 7f2c6e200640 1 -- 172.21.15.47:0/27380585 shutdown_connections 2024-02-19T15:26:31.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.235+0000 7f2c6e200640 1 -- 172.21.15.47:0/27380585 wait complete. 2024-02-19T15:26:31.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c6e200640 1 Processor -- start 2024-02-19T15:26:31.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c6e200640 1 -- start start 2024-02-19T15:26:31.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c6e200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c680cea80 0x7f2c681412b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:31.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c6e200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c68145590 0x7f2c6815b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:31.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c6e200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c6815c390 0x7f2c6815bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:31.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c6e200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c680b3950 con 0x7f2c68145590 2024-02-19T15:26:31.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c6e200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c680b3c30 con 0x7f2c6815c390 2024-02-19T15:26:31.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c6e200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c680b3f10 con 0x7f2c680cea80 2024-02-19T15:26:31.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c5fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c68145590 0x7f2c6815b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:31.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c67400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c680cea80 0x7f2c681412b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:31.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c5fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c68145590 0x7f2c6815b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58452/0 (socket says 172.21.15.47:58452) 2024-02-19T15:26:31.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c5fe00640 1 -- 172.21.15.47:0/2517578809 learned_addr learned my addr 172.21.15.47:0/2517578809 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:31.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c67e00640 1 --2- 172.21.15.47:0/2517578809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c6815c390 0x7f2c6815bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:31.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c67400640 1 -- 172.21.15.47:0/2517578809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c6815c390 msgr2=0x7f2c6815bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:31.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c67400640 1 --2- 172.21.15.47:0/2517578809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c6815c390 0x7f2c6815bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c67400640 1 -- 172.21.15.47:0/2517578809 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c68145590 msgr2=0x7f2c6815b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:31.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c67400640 1 --2- 172.21.15.47:0/2517578809 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c68145590 0x7f2c6815b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c67400640 1 -- 172.21.15.47:0/2517578809 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c60009a70 con 0x7f2c680cea80 2024-02-19T15:26:31.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c5fe00640 1 --2- 172.21.15.47:0/2517578809 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c68145590 0x7f2c6815b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:31.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c67400640 1 --2- 172.21.15.47:0/2517578809 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c680cea80 0x7f2c681412b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2c5800c400 tx=0x7f2c5800c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:31.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c5f400640 1 -- 172.21.15.47:0/2517578809 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c580074e0 con 0x7f2c680cea80 2024-02-19T15:26:31.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c5f400640 1 -- 172.21.15.47:0/2517578809 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c58007640 con 0x7f2c680cea80 2024-02-19T15:26:31.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c5f400640 1 -- 172.21.15.47:0/2517578809 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c58004a10 con 0x7f2c680cea80 2024-02-19T15:26:31.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c6e200640 1 -- 172.21.15.47:0/2517578809 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c500094d0 con 0x7f2c680cea80 2024-02-19T15:26:31.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c6e200640 1 -- 172.21.15.47:0/2517578809 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c680b44d0 con 0x7f2c680cea80 2024-02-19T15:26:31.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.239+0000 7f2c6e200640 1 -- 172.21.15.47:0/2517578809 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c68075360 con 0x7f2c680cea80 2024-02-19T15:26:31.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.243+0000 7f2c5f400640 1 -- 172.21.15.47:0/2517578809 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2c58004b70 con 0x7f2c680cea80 2024-02-19T15:26:31.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.243+0000 7f2c5f400640 1 --2- 172.21.15.47:0/2517578809 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2c44066e40 0x7f2c44069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:31.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.243+0000 7f2c5f400640 1 -- 172.21.15.47:0/2517578809 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c58084740 con 0x7f2c680cea80 2024-02-19T15:26:31.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.243+0000 7f2c5fe00640 1 --2- 172.21.15.47:0/2517578809 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2c44066e40 0x7f2c44069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:31.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.243+0000 7f2c5fe00640 1 --2- 172.21.15.47:0/2517578809 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2c44066e40 0x7f2c44069300 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f2c5001c5a0 tx=0x7f2c500b6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:31.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.247+0000 7f2c5f400640 1 -- 172.21.15.47:0/2517578809 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2c58055040 con 0x7f2c680cea80 2024-02-19T15:26:31.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.407+0000 7f2c6e200640 1 -- 172.21.15.47:0/2517578809 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2c68071c70 con 0x7f2c44066e40 2024-02-19T15:26:31.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.415+0000 7f2c5f400640 1 -- 172.21.15.47:0/2517578809 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278233 (secure 0 0 0) 0x7f2c500abcd0 con 0x7f2c44066e40 2024-02-19T15:26:31.423 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:31.425 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:26:31.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.419+0000 7f2c6e200640 1 -- 172.21.15.47:0/2517578809 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2c44066e40 msgr2=0x7f2c44069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:31.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.419+0000 7f2c6e200640 1 --2- 172.21.15.47:0/2517578809 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2c44066e40 0x7f2c44069300 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f2c5001c5a0 tx=0x7f2c500b6040 comp rx=0 tx=0).stop 2024-02-19T15:26:31.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.419+0000 7f2c6e200640 1 -- 172.21.15.47:0/2517578809 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c680cea80 msgr2=0x7f2c681412b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:31.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.419+0000 7f2c6e200640 1 --2- 172.21.15.47:0/2517578809 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c680cea80 0x7f2c681412b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2c5800c400 tx=0x7f2c5800c800 comp rx=0 tx=0).stop 2024-02-19T15:26:31.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.419+0000 7f2c6e200640 1 -- 172.21.15.47:0/2517578809 shutdown_connections 2024-02-19T15:26:31.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.419+0000 7f2c6e200640 1 --2- 172.21.15.47:0/2517578809 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2c44066e40 0x7f2c44069300 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.419+0000 7f2c6e200640 1 --2- 172.21.15.47:0/2517578809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c6815c390 0x7f2c6815bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.419+0000 7f2c6e200640 1 --2- 172.21.15.47:0/2517578809 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c68145590 0x7f2c6815b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.419+0000 7f2c6e200640 1 --2- 172.21.15.47:0/2517578809 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c680cea80 0x7f2c681412b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.419+0000 7f2c6e200640 1 -- 172.21.15.47:0/2517578809 >> 172.21.15.47:0/2517578809 conn(0x7f2c680b16a0 msgr2=0x7f2c680b0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:31.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.419+0000 7f2c6e200640 1 -- 172.21.15.47:0/2517578809 shutdown_connections 2024-02-19T15:26:31.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.419+0000 7f2c6e200640 1 -- 172.21.15.47:0/2517578809 wait complete. 2024-02-19T15:26:31.443 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-02-19T15:26:31.210849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":84,"acting":84,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":263480,"kb_used_data":9844,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754711240,"statfs":{"total":773094113280,"available":772824309760,"internally_reserved":0,"allocated":10080256,"data_stored":4940203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":259575346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.902450"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.318242+0000","last_change":"2024-02-19T15:26:26.318242+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.318242+0000","last_undegraded":"2024-02-19T15:26:26.318242+0000","last_fullsized":"2024-02-19T15:26:26.318242+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.319202+0000","last_change":"2024-02-19T15:26:26.319202+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.319202+0000","last_undegraded":"2024-02-19T15:26:26.319202+0000","last_fullsized":"2024-02-19T15:26:26.319202+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.319262+0000","last_change":"2024-02-19T15:26:26.319262+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.319262+0000","last_undegraded":"2024-02-19T15:26:26.319262+0000","last_fullsized":"2024-02-19T15:26:26.319262+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.320398+0000","last_change":"2024-02-19T15:26:26.320398+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.320398+0000","last_undegraded":"2024-02-19T15:26:26.320398+0000","last_fullsized":"2024-02-19T15:26:26.320398+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315361+0000","last_change":"2024-02-19T15:26:25.315361+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315361+0000","last_undegraded":"2024-02-19T15:26:25.315361+0000","last_fullsized":"2024-02-19T15:26:25.315361+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315188+0000","last_change":"2024-02-19T15:26:25.315188+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315188+0000","last_undegraded":"2024-02-19T15:26:25.315188+0000","last_fullsized":"2024-02-19T15:26:25.315188+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315899+0000","last_change":"2024-02-19T15:26:25.315899+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315899+0000","last_undegraded":"2024-02-19T15:26:25.315899+0000","last_fullsized":"2024-02-19T15:26:25.315899+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.316019+0000","last_change":"2024-02-19T15:26:25.316019+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.316019+0000","last_undegraded":"2024-02-19T15:26:25.316019+0000","last_fullsized":"2024-02-19T15:26:25.316019+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.310338+0000","last_change":"2024-02-19T15:26:25.310338+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.310338+0000","last_undegraded":"2024-02-19T15:26:25.310338+0000","last_fullsized":"2024-02-19T15:26:25.310338+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.316341+0000","last_change":"2024-02-19T15:26:25.316341+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.316341+0000","last_undegraded":"2024-02-19T15:26:25.316341+0000","last_fullsized":"2024-02-19T15:26:25.316341+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-19T15:26:26.338255+0000","last_change":"2024-02-19T15:26:26.336796+0000","last_active":"2024-02-19T15:26:26.338255+0000","last_peered":"2024-02-19T15:26:26.338255+0000","last_clean":"2024-02-19T15:26:26.338255+0000","last_became_active":"2024-02-19T15:26:26.336376+0000","last_became_peered":"2024-02-19T15:26:26.336376+0000","last_unstale":"2024-02-19T15:26:26.338255+0000","last_undegraded":"2024-02-19T15:26:26.338255+0000","last_fullsized":"2024-02-19T15:26:26.338255+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:27:28.873521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.311184+0000","last_change":"2024-02-19T15:26:25.311184+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.311184+0000","last_undegraded":"2024-02-19T15:26:25.311184+0000","last_fullsized":"2024-02-19T15:26:25.311184+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.310305+0000","last_change":"2024-02-19T15:26:25.310305+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.310305+0000","last_undegraded":"2024-02-19T15:26:25.310305+0000","last_fullsized":"2024-02-19T15:26:25.310305+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.313986+0000","last_change":"2024-02-19T15:26:25.313986+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.313986+0000","last_undegraded":"2024-02-19T15:26:25.313986+0000","last_fullsized":"2024-02-19T15:26:25.313986+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.319592+0000","last_change":"2024-02-19T15:26:26.319592+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.319592+0000","last_undegraded":"2024-02-19T15:26:26.319592+0000","last_fullsized":"2024-02-19T15:26:26.319592+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.314616+0000","last_change":"2024-02-19T15:26:25.314616+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.314616+0000","last_undegraded":"2024-02-19T15:26:25.314616+0000","last_fullsized":"2024-02-19T15:26:25.314616+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315717+0000","last_change":"2024-02-19T15:26:25.315717+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315717+0000","last_undegraded":"2024-02-19T15:26:25.315717+0000","last_fullsized":"2024-02-19T15:26:25.315717+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.312770+0000","last_change":"2024-02-19T15:26:25.312770+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.312770+0000","last_undegraded":"2024-02-19T15:26:25.312770+0000","last_fullsized":"2024-02-19T15:26:25.312770+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.312441+0000","last_change":"2024-02-19T15:26:25.312441+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.312441+0000","last_undegraded":"2024-02-19T15:26:25.312441+0000","last_fullsized":"2024-02-19T15:26:25.312441+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315589+0000","last_change":"2024-02-19T15:26:25.315589+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315589+0000","last_undegraded":"2024-02-19T15:26:25.315589+0000","last_fullsized":"2024-02-19T15:26:25.315589+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.310289+0000","last_change":"2024-02-19T15:26:25.310289+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.310289+0000","last_undegraded":"2024-02-19T15:26:25.310289+0000","last_fullsized":"2024-02-19T15:26:25.310289+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.318671+0000","last_change":"2024-02-19T15:26:26.318671+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.318671+0000","last_undegraded":"2024-02-19T15:26:26.318671+0000","last_fullsized":"2024-02-19T15:26:26.318671+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315968+0000","last_change":"2024-02-19T15:26:25.315968+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315968+0000","last_undegraded":"2024-02-19T15:26:25.315968+0000","last_fullsized":"2024-02-19T15:26:25.315968+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-19T15:26:26.331522+0000","last_change":"2024-02-19T15:26:26.328022+0000","last_active":"2024-02-19T15:26:26.331522+0000","last_peered":"2024-02-19T15:26:26.331522+0000","last_clean":"2024-02-19T15:26:26.331522+0000","last_became_active":"2024-02-19T15:26:26.327714+0000","last_became_peered":"2024-02-19T15:26:26.327714+0000","last_unstale":"2024-02-19T15:26:26.331522+0000","last_undegraded":"2024-02-19T15:26:26.331522+0000","last_fullsized":"2024-02-19T15:26:26.331522+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:09:12.432031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315314+0000","last_change":"2024-02-19T15:26:25.315314+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315314+0000","last_undegraded":"2024-02-19T15:26:25.315314+0000","last_fullsized":"2024-02-19T15:26:25.315314+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.316123+0000","last_change":"2024-02-19T15:26:25.316123+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.316123+0000","last_undegraded":"2024-02-19T15:26:25.316123+0000","last_fullsized":"2024-02-19T15:26:25.316123+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.313762+0000","last_change":"2024-02-19T15:26:25.313762+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.313762+0000","last_undegraded":"2024-02-19T15:26:25.313762+0000","last_fullsized":"2024-02-19T15:26:25.313762+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.313774+0000","last_change":"2024-02-19T15:26:25.313774+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.313774+0000","last_undegraded":"2024-02-19T15:26:25.313774+0000","last_fullsized":"2024-02-19T15:26:25.313774+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.319175+0000","last_change":"2024-02-19T15:26:26.319175+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.319175+0000","last_undegraded":"2024-02-19T15:26:26.319175+0000","last_fullsized":"2024-02-19T15:26:26.319175+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315822+0000","last_change":"2024-02-19T15:26:25.315822+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315822+0000","last_undegraded":"2024-02-19T15:26:25.315822+0000","last_fullsized":"2024-02-19T15:26:25.315822+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.313928+0000","last_change":"2024-02-19T15:26:25.313928+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.313928+0000","last_undegraded":"2024-02-19T15:26:25.313928+0000","last_fullsized":"2024-02-19T15:26:25.313928+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315310+0000","last_change":"2024-02-19T15:26:25.315310+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315310+0000","last_undegraded":"2024-02-19T15:26:25.315310+0000","last_fullsized":"2024-02-19T15:26:25.315310+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315507+0000","last_change":"2024-02-19T15:26:25.315507+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315507+0000","last_undegraded":"2024-02-19T15:26:25.315507+0000","last_fullsized":"2024-02-19T15:26:25.315507+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.318591+0000","last_change":"2024-02-19T15:26:26.318591+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.318591+0000","last_undegraded":"2024-02-19T15:26:26.318591+0000","last_fullsized":"2024-02-19T15:26:26.318591+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-19T15:26:26.339710+0000","last_change":"2024-02-19T15:26:26.338468+0000","last_active":"2024-02-19T15:26:26.339710+0000","last_peered":"2024-02-19T15:26:26.339710+0000","last_clean":"2024-02-19T15:26:26.339710+0000","last_became_active":"2024-02-19T15:26:26.338119+0000","last_became_peered":"2024-02-19T15:26:26.338119+0000","last_unstale":"2024-02-19T15:26:26.339710+0000","last_undegraded":"2024-02-19T15:26:26.339710+0000","last_fullsized":"2024-02-19T15:26:26.339710+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:46:46.101170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315428+0000","last_change":"2024-02-19T15:26:25.315428+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315428+0000","last_undegraded":"2024-02-19T15:26:25.315428+0000","last_fullsized":"2024-02-19T15:26:25.315428+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-19T15:26:26.331603+0000","last_change":"2024-02-19T15:26:26.329556+0000","last_active":"2024-02-19T15:26:26.331603+0000","last_peered":"2024-02-19T15:26:26.331603+0000","last_clean":"2024-02-19T15:26:26.331603+0000","last_became_active":"2024-02-19T15:26:26.329110+0000","last_became_peered":"2024-02-19T15:26:26.329110+0000","last_unstale":"2024-02-19T15:26:26.331603+0000","last_undegraded":"2024-02-19T15:26:26.331603+0000","last_fullsized":"2024-02-19T15:26:26.331603+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:16:30.549123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.313959+0000","last_change":"2024-02-19T15:26:25.313959+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.313959+0000","last_undegraded":"2024-02-19T15:26:25.313959+0000","last_fullsized":"2024-02-19T15:26:25.313959+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.318242+0000","last_change":"2024-02-19T15:26:26.318242+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.318242+0000","last_undegraded":"2024-02-19T15:26:26.318242+0000","last_fullsized":"2024-02-19T15:26:26.318242+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.312292+0000","last_change":"2024-02-19T15:26:25.312292+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.312292+0000","last_undegraded":"2024-02-19T15:26:25.312292+0000","last_fullsized":"2024-02-19T15:26:25.312292+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.314635+0000","last_change":"2024-02-19T15:26:25.314635+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.314635+0000","last_undegraded":"2024-02-19T15:26:25.314635+0000","last_fullsized":"2024-02-19T15:26:25.314635+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":292,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-19T15:26:25.315555+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:26:25.315555+0000","last_peered":"2024-02-19T15:26:25.315555+0000","last_clean":"2024-02-19T15:26:25.315555+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:26:25.315555+0000","last_undegraded":"2024-02-19T15:26:25.315555+0000","last_fullsized":"2024-02-19T15:26:25.315555+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":1},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509498,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1122304,"data_stored":475424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.61399999999999999,"15min":0.61699999999999999},"min":{"1min":0.42899999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83599999999999997,"5min":1.022,"15min":1.022},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.68000000000000005,"15min":0.752},"min":{"1min":0.433,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.96699999999999997,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.64700000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.64500000000000002,"15min":0.65200000000000002},"min":{"1min":0.49299999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.81999999999999995,"5min":1.171,"15min":1.171},"last":0.5},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65600000000000003,"15min":0.65200000000000002},"min":{"1min":0.40300000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.51900000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.437,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.175,"5min":1.216,"15min":1.216},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.70599999999999996,"15min":0.69099999999999995},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.95099999999999996,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.54300000000000004}]},{"osd":3,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69499999999999995,"15min":0.68100000000000005},"min":{"1min":0.46899999999999997,"5min":0.438,"15min":0.438},"max":{"1min":0.90900000000000003,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67600000000000005,"15min":0.66200000000000003},"min":{"1min":0.48399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.87,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.72899999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.64900000000000002,"15min":0.61499999999999999},"min":{"1min":0.40400000000000003,"5min":0.309,"15min":0.309},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.752,"5min":0.66200000000000003,"15min":0.61499999999999999},"min":{"1min":0.41699999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.71099999999999997}]},{"osd":5,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.65700000000000003,"15min":0.61299999999999999},"min":{"1min":0.439,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.753},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.72199999999999998,"15min":0.68200000000000005},"min":{"1min":0.56899999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.73999999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.72199999999999998,"15min":0.66800000000000004},"min":{"1min":0.48899999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.74299999999999999,"15min":0.68899999999999995},"min":{"1min":0.67000000000000004,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.80800000000000005}]}]},{"osd":7,"up_from":14,"seq":60129542202,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1059517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.59699999999999998,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.87,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.67400000000000004},"min":{"1min":0.38800000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.77100000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.63500000000000001,"15min":0.60099999999999998},"min":{"1min":0.498,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86799999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.70399999999999996,"15min":0.72099999999999997},"min":{"1min":0.496,"5min":0.437,"15min":0.437},"max":{"1min":1.008,"5min":2.968,"15min":2.968},"last":0.73299999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.64700000000000002,"15min":0.63600000000000001},"min":{"1min":0.47599999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85399999999999998,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.497,"5min":0.438,"15min":0.438},"max":{"1min":0.97399999999999998,"5min":2.508,"15min":2.508},"last":0.68799999999999994}]},{"osd":3,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66300000000000003,"15min":0.65900000000000003},"min":{"1min":0.50900000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.91300000000000003,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.752},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.70999999999999996,"15min":0.69499999999999995},"min":{"1min":0.59399999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.94699999999999995,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":0.85699999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.59499999999999997,"15min":0.53900000000000003},"min":{"1min":0.432,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.61799999999999999,"15min":0.58199999999999996},"min":{"1min":0.40100000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93600000000000005,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.57199999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.65700000000000003,"15min":0.58399999999999996},"min":{"1min":0.53500000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.63800000000000001},"min":{"1min":0.54700000000000004,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.97299999999999998,"5min":1.046,"15min":1.046},"last":0.70899999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.67100000000000004,"15min":0.61299999999999999},"min":{"1min":0.46999999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.62},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72599999999999998,"15min":0.67500000000000004},"min":{"1min":0.59699999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.63700000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542203,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32268,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339572,"statfs":{"total":96636764160,"available":96603721728,"internally_reserved":0,"allocated":1110016,"data_stored":469149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.67200000000000004,"15min":0.64400000000000002},"min":{"1min":0.38300000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.97699999999999998,"5min":5.7789999999999999,"15min":5.7789999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.71099999999999997,"15min":0.63700000000000001},"min":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.86499999999999999,"5min":12.359999999999999,"15min":12.359999999999999},"last":0.53600000000000003}]},{"osd":1,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.73799999999999999,"15min":0.67000000000000004},"min":{"1min":0.432,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80000000000000004,"5min":10.938000000000001,"15min":10.938000000000001},"last":0.502},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.78900000000000003,"15min":0.71299999999999997},"min":{"1min":0.50600000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85599999999999998,"5min":10.531000000000001,"15min":10.531000000000001},"last":0.56499999999999995}]},{"osd":2,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.79700000000000004,"15min":0.72099999999999997},"min":{"1min":0.42999999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.89900000000000002,"5min":12.925000000000001,"15min":12.925000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.72999999999999998,"15min":0.64100000000000001},"min":{"1min":0.48899999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89000000000000001,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.46999999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.91100000000000003,"15min":0.73499999999999999},"min":{"1min":0.49399999999999999,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82299999999999995,"5min":26.161999999999999,"15min":26.161999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.91000000000000003,"15min":0.73999999999999999},"min":{"1min":0.47299999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1120000000000001,"5min":27.934999999999999,"15min":27.934999999999999},"last":0.52700000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.67500000000000004,"15min":0.59199999999999997},"min":{"1min":0.45700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83099999999999996,"5min":9.0549999999999997,"15min":9.0549999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.70899999999999996,"15min":0.67100000000000004},"min":{"1min":0.44600000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.92800000000000005,"5min":6.2549999999999999,"15min":6.2549999999999999},"last":0.33900000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.746,"15min":0.67000000000000004},"min":{"1min":0.50700000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.6539999999999999,"5min":6.5229999999999997,"15min":6.5229999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81899999999999995,"15min":0.749},"min":{"1min":0.63500000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.2310000000000001,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":0.48299999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.77600000000000002,"15min":0.69399999999999995},"min":{"1min":0.41499999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.95399999999999996,"5min":11.986000000000001,"15min":11.986000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.77700000000000002,"15min":0.71499999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.89100000000000001,"5min":8.9770000000000003,"15min":8.9770000000000003},"last":0.45200000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32268,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339572,"statfs":{"total":96636764160,"available":96603721728,"internally_reserved":0,"allocated":1110016,"data_stored":469149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.378,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.623,"15min":0.60499999999999998},"min":{"1min":0.47099999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85499999999999998,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.61199999999999999}]},{"osd":1,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64100000000000001,"15min":0.61899999999999999},"min":{"1min":0.48999999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.879,"5min":1.081,"15min":1.081},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64400000000000002,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83699999999999997,"5min":1.008,"15min":1.008},"last":0.56999999999999995}]},{"osd":2,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.65400000000000003,"15min":0.66000000000000003},"min":{"1min":0.49299999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.90700000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68200000000000005,"15min":0.67600000000000005},"min":{"1min":0.45100000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92000000000000004,"5min":1.002,"15min":1.002},"last":0.55400000000000005}]},{"osd":3,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.70799999999999996,"15min":0.68200000000000005},"min":{"1min":0.50600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94899999999999995,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.65900000000000003,"15min":0.65300000000000002},"min":{"1min":0.42799999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89600000000000002,"5min":1.153,"15min":1.153},"last":0.69099999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63600000000000001,"15min":0.60599999999999998},"min":{"1min":0.433,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85399999999999998,"5min":1.042,"15min":1.042},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70599999999999996,"15min":0.68200000000000005},"min":{"1min":0.54200000000000004,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.94799999999999995,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.749}]},{"osd":6,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.622,"15min":0.58299999999999996},"min":{"1min":0.376,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.92200000000000004,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68600000000000005,"15min":0.65500000000000003},"min":{"1min":0.52900000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90800000000000003,"5min":1.054,"15min":1.054},"last":0.63600000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.70199999999999996,"15min":0.67100000000000004},"min":{"1min":0.44800000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.98199999999999998,"5min":1.115,"15min":1.115},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.75600000000000001,"15min":0.73399999999999999},"min":{"1min":0.46999999999999997,"5min":0.443,"15min":0.443},"max":{"1min":1.0169999999999999,"5min":1.24,"15min":1.24},"last":0.78700000000000003}]}]},{"osd":3,"up_from":14,"seq":60129542201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32276,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339564,"statfs":{"total":96636764160,"available":96603713536,"internally_reserved":0,"allocated":1110016,"data_stored":469149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42899999999999999,"15min":0.44700000000000001},"min":{"1min":0.23699999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76600000000000001,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.54100000000000004,"15min":0.64600000000000002},"min":{"1min":0.27500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67600000000000005,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.53600000000000003}]},{"osd":1,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.433,"15min":0.442},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.66000000000000003,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.59699999999999998,"15min":0.70899999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.66400000000000003,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.67700000000000005}]},{"osd":2,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56799999999999995,"15min":0.61199999999999999},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.66500000000000004,"5min":1.008,"15min":1.008},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55000000000000004,"15min":0.58899999999999997},"min":{"1min":0.39900000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86199999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.82799999999999996}]},{"osd":4,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.70299999999999996,"15min":0.67900000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.90600000000000003,"5min":1.004,"15min":1.004},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.80100000000000005,"15min":0.86099999999999999},"min":{"1min":0.58199999999999996,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0269999999999999,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.63100000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.80100000000000005,"15min":0.85499999999999998},"min":{"1min":0.54700000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.96699999999999997,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.74199999999999999,"15min":0.748},"min":{"1min":0.53300000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.93899999999999995,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.86899999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:26:06 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.82499999999999996,"15min":0.89000000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92300000000000004,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.78500000000000003,"15min":0.81299999999999994},"min":{"1min":0.55400000000000005,"5min":0.502,"15min":0.502},"max":{"1min":1.0009999999999999,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.91700000000000004}]},{"osd":7,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76400000000000001,"15min":0.78000000000000003},"min":{"1min":0.52700000000000002,"5min":0.504,"15min":0.504},"max":{"1min":0.96799999999999997,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.84099999999999997,"15min":0.92000000000000004},"min":{"1min":0.57399999999999995,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0009999999999999,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.80100000000000005}]}]},{"osd":2,"up_from":14,"seq":60129542202,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1110016,"data_stored":469149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.66700000000000004,"15min":0.69699999999999995},"min":{"1min":0.27400000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.64400000000000002,"5min":15.42,"15min":15.42},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.71499999999999997,"15min":0.88500000000000001},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.76500000000000001,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.38100000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.75700000000000001,"15min":0.89700000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58999999999999997,"5min":9.5269999999999992,"15min":9.5269999999999992},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.94199999999999995,"15min":1.006},"min":{"1min":0.39700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.621,"5min":22.954999999999998,"15min":22.954999999999998},"last":0.41499999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.67100000000000004,"15min":0.78300000000000003},"min":{"1min":0.376,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66200000000000003,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.74299999999999999,"15min":0.82299999999999995},"min":{"1min":0.41799999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69399999999999995,"5min":7.734,"15min":7.734},"last":0.48499999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.91400000000000003,"15min":0.90300000000000002},"min":{"1min":0.52000000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.016,"5min":15.369,"15min":15.369},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.82699999999999996,"15min":0.89300000000000002},"min":{"1min":0.53800000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93700000000000006,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.59199999999999997}]},{"osd":5,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.871,"15min":0.88900000000000001},"min":{"1min":0.53900000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.95999999999999996,"5min":7.6520000000000001,"15min":7.6520000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.98699999999999999,"15min":0.97499999999999998},"min":{"1min":0.60199999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0660000000000001,"5min":15.391,"15min":15.391},"last":0.63300000000000001}]},{"osd":6,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.95799999999999996,"15min":0.94399999999999995},"min":{"1min":0.54900000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.90100000000000002,"5min":15.337,"15min":15.337},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.876,"15min":0.95799999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.98499999999999999,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.68899999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.90600000000000003,"15min":0.98099999999999998},"min":{"1min":0.58399999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.028,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.91600000000000004,"15min":0.95299999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.98099999999999998,"5min":7.6929999999999996,"15min":7.6929999999999996},"last":0.54800000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542203,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1110016,"data_stored":469149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.39200000000000002,"15min":0.42399999999999999},"min":{"1min":0.248,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56799999999999995,"5min":0.747,"15min":0.747},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.438,"15min":0.46600000000000003},"min":{"1min":0.28100000000000003,"5min":0.25,"15min":0.25},"max":{"1min":0.69599999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.49399999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.42599999999999999,"15min":0.44400000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.56100000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.48799999999999999,"15min":0.52600000000000002},"min":{"1min":0.36299999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.501,"5min":1.071,"15min":1.071},"last":0.379}]},{"osd":3,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58899999999999997,"15min":0.63300000000000001},"min":{"1min":0.44800000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.64600000000000002,"5min":1.365,"15min":1.365},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.57399999999999995,"15min":0.63},"min":{"1min":0.42999999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.61399999999999999,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.47099999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.65300000000000002,"15min":0.61299999999999999},"min":{"1min":0.52100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.99099999999999999,"5min":1.127,"15min":1.127},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.69299999999999995,"15min":0.68000000000000005},"min":{"1min":0.54900000000000004,"5min":0.435,"15min":0.435},"max":{"1min":0.98399999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.70299999999999996}]},{"osd":5,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70899999999999996,"15min":0.69699999999999995},"min":{"1min":0.54200000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.95599999999999996,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.70399999999999996,"15min":0.68500000000000005},"min":{"1min":0.503,"5min":0.443,"15min":0.443},"max":{"1min":0.94199999999999995,"5min":1.107,"15min":1.107},"last":0.80500000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.72899999999999998},"min":{"1min":0.57099999999999995,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.97399999999999998,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.74199999999999999,"15min":0.745},"min":{"1min":0.60299999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97199999999999998,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.73999999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.76800000000000002,"15min":0.78600000000000003},"min":{"1min":0.59999999999999998,"5min":0.496,"15min":0.496},"max":{"1min":0.95299999999999996,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.76400000000000001,"15min":0.748},"min":{"1min":0.58499999999999996,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0229999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.82799999999999996}]}]},{"osd":0,"up_from":14,"seq":60129542201,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37008,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334832,"statfs":{"total":96636764160,"available":96598867968,"internally_reserved":0,"allocated":1703936,"data_stored":1059517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.50700000000000001,"15min":0.628},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.66200000000000003,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.31},{"interface":"front","average":{"1min":0.443,"5min":0.46300000000000002,"15min":0.48299999999999998},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.76000000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.46899999999999997}]},{"osd":2,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.65400000000000003,"15min":0.78400000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.67500000000000004,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.61799999999999999,"15min":0.746},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69099999999999995,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.36199999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.56699999999999995,"15min":0.67900000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.61699999999999999,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.436},{"interface":"front","average":{"1min":0.495,"5min":0.54800000000000004,"15min":0.60999999999999999},"min":{"1min":0.38100000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.72499999999999998,"5min":1.345,"15min":1.345},"last":0.41499999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.745,"15min":0.74299999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.0700000000000001,"5min":1.095,"15min":1.095},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.84399999999999997,"15min":0.92500000000000004},"min":{"1min":0.64000000000000001,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0049999999999999,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.73299999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.73799999999999999,"15min":0.751},"min":{"1min":0.55900000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.97999999999999998,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.77300000000000002,"15min":0.85299999999999998},"min":{"1min":0.52900000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.05,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.55700000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.82099999999999995,"15min":0.91500000000000004},"min":{"1min":0.59799999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.95899999999999996,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73799999999999999,"15min":0.75700000000000001},"min":{"1min":0.57999999999999996,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.84999999999999998,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.57799999999999996}]},{"osd":7,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.85199999999999998,"15min":0.94199999999999995},"min":{"1min":0.58599999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.0229999999999999,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.83499999999999996,"15min":0.94099999999999995},"min":{"1min":0.55900000000000005,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.96499999999999997,"5min":4.915,"15min":4.915},"last":0.69499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:26:31.448 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:26:31.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.623+0000 7f63d4600640 1 Processor -- start 2024-02-19T15:26:31.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.623+0000 7f63d4600640 1 -- start start 2024-02-19T15:26:31.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.623+0000 7f63d4600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63cc1614e0 0x7f63cc15f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:31.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.627+0000 7f63d4600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f63cc0b2500 0x7f63cc06ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:31.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.627+0000 7f63d4600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63cc06b5d0 con 0x7f63cc15eee0 2024-02-19T15:26:31.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.627+0000 7f63d4600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63cc0b1c00 con 0x7f63cc0b2500 2024-02-19T15:26:31.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.627+0000 7f63d4600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc0b1640 con 0x7f63cc1614e0 2024-02-19T15:26:31.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.627+0000 7f63d1c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63cc1614e0 0x7f63cc15f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:31.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.627+0000 7f63d1c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63cc1614e0 0x7f63cc15f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:51882/0 (socket says 172.21.15.47:51882) 2024-02-19T15:26:31.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.627+0000 7f63d1c00640 1 -- 172.21.15.47:0/1160940850 learned_addr learned my addr 172.21.15.47:0/1160940850 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:31.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.627+0000 7f63d2600640 1 --2- 172.21.15.47:0/1160940850 >> v2:172.21.15.116:3300/0 conn(0x7f63cc0b2500 0x7f63cc06ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:31.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.627+0000 7f63d1c00640 1 -- 172.21.15.47:0/1160940850 >> v2:172.21.15.116:3300/0 conn(0x7f63cc0b2500 msgr2=0x7f63cc06ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:31.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.627+0000 7f63d1c00640 1 --2- 172.21.15.47:0/1160940850 >> v2:172.21.15.116:3300/0 conn(0x7f63cc0b2500 0x7f63cc06ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.627+0000 7f63d1c00640 1 -- 172.21.15.47:0/1160940850 >> v1:172.21.15.116:6789/0 conn(0x7f63cc15eee0 legacy=0x7f63cc0b1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:31.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.627+0000 7f63d1c00640 1 -- 172.21.15.47:0/1160940850 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc0b1920 con 0x7f63cc1614e0 2024-02-19T15:26:31.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.627+0000 7f63cbe00640 1 -- 172.21.15.47:0/1160940850 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470939213 0 0) 0x7f63cc06b5d0 con 0x7f63cc15eee0 2024-02-19T15:26:31.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.627+0000 7f63d1c00640 1 --2- 172.21.15.47:0/1160940850 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63cc1614e0 0x7f63cc15f620 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009870 tx=0x7f63c001c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f62604269b449c1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:31.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.627+0000 7f63cbe00640 1 -- 172.21.15.47:0/1160940850 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c00aa5f0 con 0x7f63cc1614e0 2024-02-19T15:26:31.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.627+0000 7f63cbe00640 1 -- 172.21.15.47:0/1160940850 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c00aa750 con 0x7f63cc1614e0 2024-02-19T15:26:31.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.627+0000 7f63cbe00640 1 -- 172.21.15.47:0/1160940850 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c00aaae0 con 0x7f63cc1614e0 2024-02-19T15:26:31.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d4600640 1 -- 172.21.15.47:0/1160940850 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63cc1614e0 msgr2=0x7f63cc15f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:31.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d4600640 1 --2- 172.21.15.47:0/1160940850 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63cc1614e0 0x7f63cc15f620 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009870 tx=0x7f63c001c560 comp rx=0 tx=0).stop 2024-02-19T15:26:31.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d4600640 1 -- 172.21.15.47:0/1160940850 shutdown_connections 2024-02-19T15:26:31.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d4600640 1 --2- 172.21.15.47:0/1160940850 >> v2:172.21.15.116:3300/0 conn(0x7f63cc0b2500 0x7f63cc06ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d4600640 1 --2- 172.21.15.47:0/1160940850 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63cc1614e0 0x7f63cc15f620 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d4600640 1 -- 172.21.15.47:0/1160940850 >> 172.21.15.47:0/1160940850 conn(0x7f63cc0b4fc0 msgr2=0x7f63cc0b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:31.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d4600640 1 -- 172.21.15.47:0/1160940850 shutdown_connections 2024-02-19T15:26:31.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d4600640 1 -- 172.21.15.47:0/1160940850 wait complete. 2024-02-19T15:26:31.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d4600640 1 Processor -- start 2024-02-19T15:26:31.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d4600640 1 -- start start 2024-02-19T15:26:31.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d4600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63cc0b2500 0x7f63cc15a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:31.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d4600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63cc15eee0 0x7f63cc157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:31.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d4600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63cc1614e0 0x7f63cc15cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:31.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d4600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc157fd0 con 0x7f63cc1614e0 2024-02-19T15:26:31.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d4600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc1582b0 con 0x7f63cc15eee0 2024-02-19T15:26:31.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d4600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc1597c0 con 0x7f63cc0b2500 2024-02-19T15:26:31.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d1c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63cc0b2500 0x7f63cc15a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:31.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d2600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63cc1614e0 0x7f63cc15cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:31.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d1c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63cc0b2500 0x7f63cc15a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:51888/0 (socket says 172.21.15.47:51888) 2024-02-19T15:26:31.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d1c00640 1 -- 172.21.15.47:0/974316789 learned_addr learned my addr 172.21.15.47:0/974316789 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:31.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d1200640 1 --2- 172.21.15.47:0/974316789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63cc15eee0 0x7f63cc157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:31.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d1c00640 1 -- 172.21.15.47:0/974316789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63cc15eee0 msgr2=0x7f63cc157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:31.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d1c00640 1 --2- 172.21.15.47:0/974316789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63cc15eee0 0x7f63cc157780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d1c00640 1 -- 172.21.15.47:0/974316789 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63cc1614e0 msgr2=0x7f63cc15cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:31.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d1c00640 1 --2- 172.21.15.47:0/974316789 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63cc1614e0 0x7f63cc15cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.631+0000 7f63d1c00640 1 -- 172.21.15.47:0/974316789 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c00094d0 con 0x7f63cc0b2500 2024-02-19T15:26:31.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.635+0000 7f63d1c00640 1 --2- 172.21.15.47:0/974316789 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63cc0b2500 0x7f63cc15a2a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f63c00ac040 tx=0x7f63c001ada0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:31.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.635+0000 7f63c9600640 1 -- 172.21.15.47:0/974316789 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c00bd5f0 con 0x7f63cc0b2500 2024-02-19T15:26:31.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.635+0000 7f63c9600640 1 -- 172.21.15.47:0/974316789 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c00bd770 con 0x7f63cc0b2500 2024-02-19T15:26:31.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.635+0000 7f63c9600640 1 -- 172.21.15.47:0/974316789 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c00bdb20 con 0x7f63cc0b2500 2024-02-19T15:26:31.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.635+0000 7f63d4600640 1 -- 172.21.15.47:0/974316789 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63cc159aa0 con 0x7f63cc0b2500 2024-02-19T15:26:31.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.635+0000 7f63d4600640 1 -- 172.21.15.47:0/974316789 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63cc157cc0 con 0x7f63cc0b2500 2024-02-19T15:26:31.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.635+0000 7f63d4600640 1 -- 172.21.15.47:0/974316789 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63cc0b1d70 con 0x7f63cc0b2500 2024-02-19T15:26:31.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.635+0000 7f63c9600640 1 -- 172.21.15.47:0/974316789 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f63c00b3600 con 0x7f63cc0b2500 2024-02-19T15:26:31.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.635+0000 7f63c9600640 1 --2- 172.21.15.47:0/974316789 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f63ac066e40 0x7f63ac069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:31.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.635+0000 7f63c9600640 1 -- 172.21.15.47:0/974316789 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63c00a7070 con 0x7f63cc0b2500 2024-02-19T15:26:31.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.635+0000 7f63d1200640 1 --2- 172.21.15.47:0/974316789 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f63ac066e40 0x7f63ac069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:31.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.639+0000 7f63d1200640 1 --2- 172.21.15.47:0/974316789 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f63ac066e40 0x7f63ac069300 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0721b0 tx=0x7f63b8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:31.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.639+0000 7f63c9600640 1 -- 172.21.15.47:0/974316789 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f63c00ec610 con 0x7f63cc0b2500 2024-02-19T15:26:31.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.799+0000 7f63d4600640 1 -- 172.21.15.47:0/974316789 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f63cc0b1fa0 con 0x7f63ac066e40 2024-02-19T15:26:31.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.807+0000 7f63c9600640 1 -- 172.21.15.47:0/974316789 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278233 (secure 0 0 0) 0x7f63cc0b1fa0 con 0x7f63ac066e40 2024-02-19T15:26:31.816 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:31.817 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:26:31.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.811+0000 7f63d4600640 1 -- 172.21.15.47:0/974316789 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f63ac066e40 msgr2=0x7f63ac069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:31.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.811+0000 7f63d4600640 1 --2- 172.21.15.47:0/974316789 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f63ac066e40 0x7f63ac069300 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0721b0 tx=0x7f63b8009040 comp rx=0 tx=0).stop 2024-02-19T15:26:31.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.811+0000 7f63d4600640 1 -- 172.21.15.47:0/974316789 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63cc0b2500 msgr2=0x7f63cc15a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:31.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.811+0000 7f63d4600640 1 --2- 172.21.15.47:0/974316789 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63cc0b2500 0x7f63cc15a2a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f63c00ac040 tx=0x7f63c001ada0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.811+0000 7f63d4600640 1 -- 172.21.15.47:0/974316789 shutdown_connections 2024-02-19T15:26:31.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.811+0000 7f63d4600640 1 --2- 172.21.15.47:0/974316789 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f63ac066e40 0x7f63ac069300 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.811+0000 7f63d4600640 1 --2- 172.21.15.47:0/974316789 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63cc1614e0 0x7f63cc15cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.811+0000 7f63d4600640 1 --2- 172.21.15.47:0/974316789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63cc15eee0 0x7f63cc157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.811+0000 7f63d4600640 1 --2- 172.21.15.47:0/974316789 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63cc0b2500 0x7f63cc15a2a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:31.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.811+0000 7f63d4600640 1 -- 172.21.15.47:0/974316789 >> 172.21.15.47:0/974316789 conn(0x7f63cc0b4fc0 msgr2=0x7f63cc168120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:31.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.811+0000 7f63d4600640 1 -- 172.21.15.47:0/974316789 shutdown_connections 2024-02-19T15:26:31.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:31.811+0000 7f63d4600640 1 -- 172.21.15.47:0/974316789 wait complete. 2024-02-19T15:26:31.836 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-02-19T15:26:31.210849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":84,"acting":84,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":263480,"kb_used_data":9844,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754711240,"statfs":{"total":773094113280,"available":772824309760,"internally_reserved":0,"allocated":10080256,"data_stored":4940203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":259575346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.902450"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.318242+0000","last_change":"2024-02-19T15:26:26.318242+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.318242+0000","last_undegraded":"2024-02-19T15:26:26.318242+0000","last_fullsized":"2024-02-19T15:26:26.318242+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.319202+0000","last_change":"2024-02-19T15:26:26.319202+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.319202+0000","last_undegraded":"2024-02-19T15:26:26.319202+0000","last_fullsized":"2024-02-19T15:26:26.319202+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.319262+0000","last_change":"2024-02-19T15:26:26.319262+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.319262+0000","last_undegraded":"2024-02-19T15:26:26.319262+0000","last_fullsized":"2024-02-19T15:26:26.319262+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.320398+0000","last_change":"2024-02-19T15:26:26.320398+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.320398+0000","last_undegraded":"2024-02-19T15:26:26.320398+0000","last_fullsized":"2024-02-19T15:26:26.320398+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315361+0000","last_change":"2024-02-19T15:26:25.315361+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315361+0000","last_undegraded":"2024-02-19T15:26:25.315361+0000","last_fullsized":"2024-02-19T15:26:25.315361+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315188+0000","last_change":"2024-02-19T15:26:25.315188+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315188+0000","last_undegraded":"2024-02-19T15:26:25.315188+0000","last_fullsized":"2024-02-19T15:26:25.315188+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315899+0000","last_change":"2024-02-19T15:26:25.315899+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315899+0000","last_undegraded":"2024-02-19T15:26:25.315899+0000","last_fullsized":"2024-02-19T15:26:25.315899+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.316019+0000","last_change":"2024-02-19T15:26:25.316019+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.316019+0000","last_undegraded":"2024-02-19T15:26:25.316019+0000","last_fullsized":"2024-02-19T15:26:25.316019+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.310338+0000","last_change":"2024-02-19T15:26:25.310338+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.310338+0000","last_undegraded":"2024-02-19T15:26:25.310338+0000","last_fullsized":"2024-02-19T15:26:25.310338+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.316341+0000","last_change":"2024-02-19T15:26:25.316341+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.316341+0000","last_undegraded":"2024-02-19T15:26:25.316341+0000","last_fullsized":"2024-02-19T15:26:25.316341+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-19T15:26:26.338255+0000","last_change":"2024-02-19T15:26:26.336796+0000","last_active":"2024-02-19T15:26:26.338255+0000","last_peered":"2024-02-19T15:26:26.338255+0000","last_clean":"2024-02-19T15:26:26.338255+0000","last_became_active":"2024-02-19T15:26:26.336376+0000","last_became_peered":"2024-02-19T15:26:26.336376+0000","last_unstale":"2024-02-19T15:26:26.338255+0000","last_undegraded":"2024-02-19T15:26:26.338255+0000","last_fullsized":"2024-02-19T15:26:26.338255+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:27:28.873521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.311184+0000","last_change":"2024-02-19T15:26:25.311184+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.311184+0000","last_undegraded":"2024-02-19T15:26:25.311184+0000","last_fullsized":"2024-02-19T15:26:25.311184+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.310305+0000","last_change":"2024-02-19T15:26:25.310305+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.310305+0000","last_undegraded":"2024-02-19T15:26:25.310305+0000","last_fullsized":"2024-02-19T15:26:25.310305+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.313986+0000","last_change":"2024-02-19T15:26:25.313986+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.313986+0000","last_undegraded":"2024-02-19T15:26:25.313986+0000","last_fullsized":"2024-02-19T15:26:25.313986+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.319592+0000","last_change":"2024-02-19T15:26:26.319592+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.319592+0000","last_undegraded":"2024-02-19T15:26:26.319592+0000","last_fullsized":"2024-02-19T15:26:26.319592+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.314616+0000","last_change":"2024-02-19T15:26:25.314616+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.314616+0000","last_undegraded":"2024-02-19T15:26:25.314616+0000","last_fullsized":"2024-02-19T15:26:25.314616+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315717+0000","last_change":"2024-02-19T15:26:25.315717+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315717+0000","last_undegraded":"2024-02-19T15:26:25.315717+0000","last_fullsized":"2024-02-19T15:26:25.315717+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.312770+0000","last_change":"2024-02-19T15:26:25.312770+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.312770+0000","last_undegraded":"2024-02-19T15:26:25.312770+0000","last_fullsized":"2024-02-19T15:26:25.312770+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.312441+0000","last_change":"2024-02-19T15:26:25.312441+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.312441+0000","last_undegraded":"2024-02-19T15:26:25.312441+0000","last_fullsized":"2024-02-19T15:26:25.312441+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315589+0000","last_change":"2024-02-19T15:26:25.315589+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315589+0000","last_undegraded":"2024-02-19T15:26:25.315589+0000","last_fullsized":"2024-02-19T15:26:25.315589+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.310289+0000","last_change":"2024-02-19T15:26:25.310289+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.310289+0000","last_undegraded":"2024-02-19T15:26:25.310289+0000","last_fullsized":"2024-02-19T15:26:25.310289+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.318671+0000","last_change":"2024-02-19T15:26:26.318671+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.318671+0000","last_undegraded":"2024-02-19T15:26:26.318671+0000","last_fullsized":"2024-02-19T15:26:26.318671+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315968+0000","last_change":"2024-02-19T15:26:25.315968+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315968+0000","last_undegraded":"2024-02-19T15:26:25.315968+0000","last_fullsized":"2024-02-19T15:26:25.315968+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-19T15:26:26.331522+0000","last_change":"2024-02-19T15:26:26.328022+0000","last_active":"2024-02-19T15:26:26.331522+0000","last_peered":"2024-02-19T15:26:26.331522+0000","last_clean":"2024-02-19T15:26:26.331522+0000","last_became_active":"2024-02-19T15:26:26.327714+0000","last_became_peered":"2024-02-19T15:26:26.327714+0000","last_unstale":"2024-02-19T15:26:26.331522+0000","last_undegraded":"2024-02-19T15:26:26.331522+0000","last_fullsized":"2024-02-19T15:26:26.331522+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:09:12.432031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315314+0000","last_change":"2024-02-19T15:26:25.315314+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315314+0000","last_undegraded":"2024-02-19T15:26:25.315314+0000","last_fullsized":"2024-02-19T15:26:25.315314+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.316123+0000","last_change":"2024-02-19T15:26:25.316123+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.316123+0000","last_undegraded":"2024-02-19T15:26:25.316123+0000","last_fullsized":"2024-02-19T15:26:25.316123+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.313762+0000","last_change":"2024-02-19T15:26:25.313762+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.313762+0000","last_undegraded":"2024-02-19T15:26:25.313762+0000","last_fullsized":"2024-02-19T15:26:25.313762+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.313774+0000","last_change":"2024-02-19T15:26:25.313774+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.313774+0000","last_undegraded":"2024-02-19T15:26:25.313774+0000","last_fullsized":"2024-02-19T15:26:25.313774+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.319175+0000","last_change":"2024-02-19T15:26:26.319175+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.319175+0000","last_undegraded":"2024-02-19T15:26:26.319175+0000","last_fullsized":"2024-02-19T15:26:26.319175+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315822+0000","last_change":"2024-02-19T15:26:25.315822+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315822+0000","last_undegraded":"2024-02-19T15:26:25.315822+0000","last_fullsized":"2024-02-19T15:26:25.315822+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.313928+0000","last_change":"2024-02-19T15:26:25.313928+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.313928+0000","last_undegraded":"2024-02-19T15:26:25.313928+0000","last_fullsized":"2024-02-19T15:26:25.313928+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315310+0000","last_change":"2024-02-19T15:26:25.315310+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315310+0000","last_undegraded":"2024-02-19T15:26:25.315310+0000","last_fullsized":"2024-02-19T15:26:25.315310+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315507+0000","last_change":"2024-02-19T15:26:25.315507+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315507+0000","last_undegraded":"2024-02-19T15:26:25.315507+0000","last_fullsized":"2024-02-19T15:26:25.315507+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.318591+0000","last_change":"2024-02-19T15:26:26.318591+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.318591+0000","last_undegraded":"2024-02-19T15:26:26.318591+0000","last_fullsized":"2024-02-19T15:26:26.318591+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-19T15:26:26.339710+0000","last_change":"2024-02-19T15:26:26.338468+0000","last_active":"2024-02-19T15:26:26.339710+0000","last_peered":"2024-02-19T15:26:26.339710+0000","last_clean":"2024-02-19T15:26:26.339710+0000","last_became_active":"2024-02-19T15:26:26.338119+0000","last_became_peered":"2024-02-19T15:26:26.338119+0000","last_unstale":"2024-02-19T15:26:26.339710+0000","last_undegraded":"2024-02-19T15:26:26.339710+0000","last_fullsized":"2024-02-19T15:26:26.339710+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:46:46.101170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315428+0000","last_change":"2024-02-19T15:26:25.315428+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315428+0000","last_undegraded":"2024-02-19T15:26:25.315428+0000","last_fullsized":"2024-02-19T15:26:25.315428+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-19T15:26:26.331603+0000","last_change":"2024-02-19T15:26:26.329556+0000","last_active":"2024-02-19T15:26:26.331603+0000","last_peered":"2024-02-19T15:26:26.331603+0000","last_clean":"2024-02-19T15:26:26.331603+0000","last_became_active":"2024-02-19T15:26:26.329110+0000","last_became_peered":"2024-02-19T15:26:26.329110+0000","last_unstale":"2024-02-19T15:26:26.331603+0000","last_undegraded":"2024-02-19T15:26:26.331603+0000","last_fullsized":"2024-02-19T15:26:26.331603+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:16:30.549123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.313959+0000","last_change":"2024-02-19T15:26:25.313959+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.313959+0000","last_undegraded":"2024-02-19T15:26:25.313959+0000","last_fullsized":"2024-02-19T15:26:25.313959+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.318242+0000","last_change":"2024-02-19T15:26:26.318242+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.318242+0000","last_undegraded":"2024-02-19T15:26:26.318242+0000","last_fullsized":"2024-02-19T15:26:26.318242+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.312292+0000","last_change":"2024-02-19T15:26:25.312292+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.312292+0000","last_undegraded":"2024-02-19T15:26:25.312292+0000","last_fullsized":"2024-02-19T15:26:25.312292+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.314635+0000","last_change":"2024-02-19T15:26:25.314635+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.314635+0000","last_undegraded":"2024-02-19T15:26:25.314635+0000","last_fullsized":"2024-02-19T15:26:25.314635+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":292,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-19T15:26:25.315555+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:26:25.315555+0000","last_peered":"2024-02-19T15:26:25.315555+0000","last_clean":"2024-02-19T15:26:25.315555+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:26:25.315555+0000","last_undegraded":"2024-02-19T15:26:25.315555+0000","last_fullsized":"2024-02-19T15:26:25.315555+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":1},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509498,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1122304,"data_stored":475424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.61399999999999999,"15min":0.61699999999999999},"min":{"1min":0.42899999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83599999999999997,"5min":1.022,"15min":1.022},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.68000000000000005,"15min":0.752},"min":{"1min":0.433,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.96699999999999997,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.64700000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.64500000000000002,"15min":0.65200000000000002},"min":{"1min":0.49299999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.81999999999999995,"5min":1.171,"15min":1.171},"last":0.5},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65600000000000003,"15min":0.65200000000000002},"min":{"1min":0.40300000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.51900000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.437,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.175,"5min":1.216,"15min":1.216},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.70599999999999996,"15min":0.69099999999999995},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.95099999999999996,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.54300000000000004}]},{"osd":3,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69499999999999995,"15min":0.68100000000000005},"min":{"1min":0.46899999999999997,"5min":0.438,"15min":0.438},"max":{"1min":0.90900000000000003,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67600000000000005,"15min":0.66200000000000003},"min":{"1min":0.48399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.87,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.72899999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.64900000000000002,"15min":0.61499999999999999},"min":{"1min":0.40400000000000003,"5min":0.309,"15min":0.309},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.752,"5min":0.66200000000000003,"15min":0.61499999999999999},"min":{"1min":0.41699999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.71099999999999997}]},{"osd":5,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.65700000000000003,"15min":0.61299999999999999},"min":{"1min":0.439,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.753},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.72199999999999998,"15min":0.68200000000000005},"min":{"1min":0.56899999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.73999999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.72199999999999998,"15min":0.66800000000000004},"min":{"1min":0.48899999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.74299999999999999,"15min":0.68899999999999995},"min":{"1min":0.67000000000000004,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.80800000000000005}]}]},{"osd":7,"up_from":14,"seq":60129542202,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1059517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.59699999999999998,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.87,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.67400000000000004},"min":{"1min":0.38800000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.77100000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.63500000000000001,"15min":0.60099999999999998},"min":{"1min":0.498,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86799999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.70399999999999996,"15min":0.72099999999999997},"min":{"1min":0.496,"5min":0.437,"15min":0.437},"max":{"1min":1.008,"5min":2.968,"15min":2.968},"last":0.73299999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.64700000000000002,"15min":0.63600000000000001},"min":{"1min":0.47599999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85399999999999998,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.497,"5min":0.438,"15min":0.438},"max":{"1min":0.97399999999999998,"5min":2.508,"15min":2.508},"last":0.68799999999999994}]},{"osd":3,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66300000000000003,"15min":0.65900000000000003},"min":{"1min":0.50900000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.91300000000000003,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.752},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.70999999999999996,"15min":0.69499999999999995},"min":{"1min":0.59399999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.94699999999999995,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":0.85699999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.59499999999999997,"15min":0.53900000000000003},"min":{"1min":0.432,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.61799999999999999,"15min":0.58199999999999996},"min":{"1min":0.40100000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93600000000000005,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.57199999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.65700000000000003,"15min":0.58399999999999996},"min":{"1min":0.53500000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.63800000000000001},"min":{"1min":0.54700000000000004,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.97299999999999998,"5min":1.046,"15min":1.046},"last":0.70899999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.67100000000000004,"15min":0.61299999999999999},"min":{"1min":0.46999999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.62},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72599999999999998,"15min":0.67500000000000004},"min":{"1min":0.59699999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.63700000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542203,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32268,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339572,"statfs":{"total":96636764160,"available":96603721728,"internally_reserved":0,"allocated":1110016,"data_stored":469149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.67200000000000004,"15min":0.64400000000000002},"min":{"1min":0.38300000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.97699999999999998,"5min":5.7789999999999999,"15min":5.7789999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.71099999999999997,"15min":0.63700000000000001},"min":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.86499999999999999,"5min":12.359999999999999,"15min":12.359999999999999},"last":0.53600000000000003}]},{"osd":1,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.73799999999999999,"15min":0.67000000000000004},"min":{"1min":0.432,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80000000000000004,"5min":10.938000000000001,"15min":10.938000000000001},"last":0.502},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.78900000000000003,"15min":0.71299999999999997},"min":{"1min":0.50600000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85599999999999998,"5min":10.531000000000001,"15min":10.531000000000001},"last":0.56499999999999995}]},{"osd":2,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.79700000000000004,"15min":0.72099999999999997},"min":{"1min":0.42999999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.89900000000000002,"5min":12.925000000000001,"15min":12.925000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.72999999999999998,"15min":0.64100000000000001},"min":{"1min":0.48899999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89000000000000001,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.46999999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.91100000000000003,"15min":0.73499999999999999},"min":{"1min":0.49399999999999999,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82299999999999995,"5min":26.161999999999999,"15min":26.161999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.91000000000000003,"15min":0.73999999999999999},"min":{"1min":0.47299999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1120000000000001,"5min":27.934999999999999,"15min":27.934999999999999},"last":0.52700000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.67500000000000004,"15min":0.59199999999999997},"min":{"1min":0.45700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83099999999999996,"5min":9.0549999999999997,"15min":9.0549999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.70899999999999996,"15min":0.67100000000000004},"min":{"1min":0.44600000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.92800000000000005,"5min":6.2549999999999999,"15min":6.2549999999999999},"last":0.33900000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.746,"15min":0.67000000000000004},"min":{"1min":0.50700000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.6539999999999999,"5min":6.5229999999999997,"15min":6.5229999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81899999999999995,"15min":0.749},"min":{"1min":0.63500000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.2310000000000001,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":0.48299999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.77600000000000002,"15min":0.69399999999999995},"min":{"1min":0.41499999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.95399999999999996,"5min":11.986000000000001,"15min":11.986000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.77700000000000002,"15min":0.71499999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.89100000000000001,"5min":8.9770000000000003,"15min":8.9770000000000003},"last":0.45200000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32268,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339572,"statfs":{"total":96636764160,"available":96603721728,"internally_reserved":0,"allocated":1110016,"data_stored":469149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.378,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.623,"15min":0.60499999999999998},"min":{"1min":0.47099999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85499999999999998,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.61199999999999999}]},{"osd":1,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64100000000000001,"15min":0.61899999999999999},"min":{"1min":0.48999999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.879,"5min":1.081,"15min":1.081},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64400000000000002,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83699999999999997,"5min":1.008,"15min":1.008},"last":0.56999999999999995}]},{"osd":2,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.65400000000000003,"15min":0.66000000000000003},"min":{"1min":0.49299999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.90700000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68200000000000005,"15min":0.67600000000000005},"min":{"1min":0.45100000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92000000000000004,"5min":1.002,"15min":1.002},"last":0.55400000000000005}]},{"osd":3,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.70799999999999996,"15min":0.68200000000000005},"min":{"1min":0.50600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94899999999999995,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.65900000000000003,"15min":0.65300000000000002},"min":{"1min":0.42799999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89600000000000002,"5min":1.153,"15min":1.153},"last":0.69099999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63600000000000001,"15min":0.60599999999999998},"min":{"1min":0.433,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85399999999999998,"5min":1.042,"15min":1.042},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70599999999999996,"15min":0.68200000000000005},"min":{"1min":0.54200000000000004,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.94799999999999995,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.749}]},{"osd":6,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.622,"15min":0.58299999999999996},"min":{"1min":0.376,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.92200000000000004,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68600000000000005,"15min":0.65500000000000003},"min":{"1min":0.52900000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90800000000000003,"5min":1.054,"15min":1.054},"last":0.63600000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.70199999999999996,"15min":0.67100000000000004},"min":{"1min":0.44800000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.98199999999999998,"5min":1.115,"15min":1.115},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.75600000000000001,"15min":0.73399999999999999},"min":{"1min":0.46999999999999997,"5min":0.443,"15min":0.443},"max":{"1min":1.0169999999999999,"5min":1.24,"15min":1.24},"last":0.78700000000000003}]}]},{"osd":3,"up_from":14,"seq":60129542201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32276,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339564,"statfs":{"total":96636764160,"available":96603713536,"internally_reserved":0,"allocated":1110016,"data_stored":469149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42899999999999999,"15min":0.44700000000000001},"min":{"1min":0.23699999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76600000000000001,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.54100000000000004,"15min":0.64600000000000002},"min":{"1min":0.27500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67600000000000005,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.53600000000000003}]},{"osd":1,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.433,"15min":0.442},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.66000000000000003,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.59699999999999998,"15min":0.70899999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.66400000000000003,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.67700000000000005}]},{"osd":2,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56799999999999995,"15min":0.61199999999999999},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.66500000000000004,"5min":1.008,"15min":1.008},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55000000000000004,"15min":0.58899999999999997},"min":{"1min":0.39900000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86199999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.82799999999999996}]},{"osd":4,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.70299999999999996,"15min":0.67900000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.90600000000000003,"5min":1.004,"15min":1.004},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.80100000000000005,"15min":0.86099999999999999},"min":{"1min":0.58199999999999996,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0269999999999999,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.63100000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.80100000000000005,"15min":0.85499999999999998},"min":{"1min":0.54700000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.96699999999999997,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.74199999999999999,"15min":0.748},"min":{"1min":0.53300000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.93899999999999995,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.86899999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:26:06 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.82499999999999996,"15min":0.89000000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92300000000000004,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.78500000000000003,"15min":0.81299999999999994},"min":{"1min":0.55400000000000005,"5min":0.502,"15min":0.502},"max":{"1min":1.0009999999999999,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.91700000000000004}]},{"osd":7,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76400000000000001,"15min":0.78000000000000003},"min":{"1min":0.52700000000000002,"5min":0.504,"15min":0.504},"max":{"1min":0.96799999999999997,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.84099999999999997,"15min":0.92000000000000004},"min":{"1min":0.57399999999999995,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0009999999999999,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.80100000000000005}]}]},{"osd":2,"up_from":14,"seq":60129542202,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1110016,"data_stored":469149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.66700000000000004,"15min":0.69699999999999995},"min":{"1min":0.27400000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.64400000000000002,"5min":15.42,"15min":15.42},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.71499999999999997,"15min":0.88500000000000001},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.76500000000000001,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.38100000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.75700000000000001,"15min":0.89700000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58999999999999997,"5min":9.5269999999999992,"15min":9.5269999999999992},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.94199999999999995,"15min":1.006},"min":{"1min":0.39700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.621,"5min":22.954999999999998,"15min":22.954999999999998},"last":0.41499999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.67100000000000004,"15min":0.78300000000000003},"min":{"1min":0.376,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66200000000000003,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.74299999999999999,"15min":0.82299999999999995},"min":{"1min":0.41799999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69399999999999995,"5min":7.734,"15min":7.734},"last":0.48499999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.91400000000000003,"15min":0.90300000000000002},"min":{"1min":0.52000000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.016,"5min":15.369,"15min":15.369},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.82699999999999996,"15min":0.89300000000000002},"min":{"1min":0.53800000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93700000000000006,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.59199999999999997}]},{"osd":5,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.871,"15min":0.88900000000000001},"min":{"1min":0.53900000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.95999999999999996,"5min":7.6520000000000001,"15min":7.6520000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.98699999999999999,"15min":0.97499999999999998},"min":{"1min":0.60199999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0660000000000001,"5min":15.391,"15min":15.391},"last":0.63300000000000001}]},{"osd":6,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.95799999999999996,"15min":0.94399999999999995},"min":{"1min":0.54900000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.90100000000000002,"5min":15.337,"15min":15.337},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.876,"15min":0.95799999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.98499999999999999,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.68899999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.90600000000000003,"15min":0.98099999999999998},"min":{"1min":0.58399999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.028,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.91600000000000004,"15min":0.95299999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.98099999999999998,"5min":7.6929999999999996,"15min":7.6929999999999996},"last":0.54800000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542203,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1110016,"data_stored":469149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.39200000000000002,"15min":0.42399999999999999},"min":{"1min":0.248,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56799999999999995,"5min":0.747,"15min":0.747},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.438,"15min":0.46600000000000003},"min":{"1min":0.28100000000000003,"5min":0.25,"15min":0.25},"max":{"1min":0.69599999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.49399999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.42599999999999999,"15min":0.44400000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.56100000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.48799999999999999,"15min":0.52600000000000002},"min":{"1min":0.36299999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.501,"5min":1.071,"15min":1.071},"last":0.379}]},{"osd":3,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58899999999999997,"15min":0.63300000000000001},"min":{"1min":0.44800000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.64600000000000002,"5min":1.365,"15min":1.365},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.57399999999999995,"15min":0.63},"min":{"1min":0.42999999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.61399999999999999,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.47099999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.65300000000000002,"15min":0.61299999999999999},"min":{"1min":0.52100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.99099999999999999,"5min":1.127,"15min":1.127},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.69299999999999995,"15min":0.68000000000000005},"min":{"1min":0.54900000000000004,"5min":0.435,"15min":0.435},"max":{"1min":0.98399999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.70299999999999996}]},{"osd":5,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70899999999999996,"15min":0.69699999999999995},"min":{"1min":0.54200000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.95599999999999996,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.70399999999999996,"15min":0.68500000000000005},"min":{"1min":0.503,"5min":0.443,"15min":0.443},"max":{"1min":0.94199999999999995,"5min":1.107,"15min":1.107},"last":0.80500000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.72899999999999998},"min":{"1min":0.57099999999999995,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.97399999999999998,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.74199999999999999,"15min":0.745},"min":{"1min":0.60299999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97199999999999998,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.73999999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.76800000000000002,"15min":0.78600000000000003},"min":{"1min":0.59999999999999998,"5min":0.496,"15min":0.496},"max":{"1min":0.95299999999999996,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.76400000000000001,"15min":0.748},"min":{"1min":0.58499999999999996,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0229999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.82799999999999996}]}]},{"osd":0,"up_from":14,"seq":60129542201,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37008,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334832,"statfs":{"total":96636764160,"available":96598867968,"internally_reserved":0,"allocated":1703936,"data_stored":1059517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.50700000000000001,"15min":0.628},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.66200000000000003,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.31},{"interface":"front","average":{"1min":0.443,"5min":0.46300000000000002,"15min":0.48299999999999998},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.76000000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.46899999999999997}]},{"osd":2,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.65400000000000003,"15min":0.78400000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.67500000000000004,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.61799999999999999,"15min":0.746},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69099999999999995,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.36199999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.56699999999999995,"15min":0.67900000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.61699999999999999,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.436},{"interface":"front","average":{"1min":0.495,"5min":0.54800000000000004,"15min":0.60999999999999999},"min":{"1min":0.38100000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.72499999999999998,"5min":1.345,"15min":1.345},"last":0.41499999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.745,"15min":0.74299999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.0700000000000001,"5min":1.095,"15min":1.095},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.84399999999999997,"15min":0.92500000000000004},"min":{"1min":0.64000000000000001,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0049999999999999,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.73299999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.73799999999999999,"15min":0.751},"min":{"1min":0.55900000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.97999999999999998,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.77300000000000002,"15min":0.85299999999999998},"min":{"1min":0.52900000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.05,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.55700000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.82099999999999995,"15min":0.91500000000000004},"min":{"1min":0.59799999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.95899999999999996,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73799999999999999,"15min":0.75700000000000001},"min":{"1min":0.57999999999999996,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.84999999999999998,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.57799999999999996}]},{"osd":7,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.85199999999999998,"15min":0.94199999999999995},"min":{"1min":0.58599999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.0229999999999999,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.83499999999999996,"15min":0.94099999999999995},"min":{"1min":0.55900000000000005,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.96499999999999997,"5min":4.915,"15min":4.915},"last":0.69499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:26:31.841 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:26:32.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.015+0000 7f7aac000640 1 Processor -- start 2024-02-19T15:26:32.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.015+0000 7f7aac000640 1 -- start start 2024-02-19T15:26:32.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.015+0000 7f7aac000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7aa40cb520 0x7f7aa40cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:32.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.015+0000 7f7aac000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7aa40c6780 0x7f7aa40c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:32.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.019+0000 7f7aac000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa40ad840 con 0x7f7aa406aeb0 2024-02-19T15:26:32.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.019+0000 7f7aac000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa40c62f0 con 0x7f7aa40c6780 2024-02-19T15:26:32.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.019+0000 7f7aac000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40cc810 con 0x7f7aa40cb520 2024-02-19T15:26:32.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.019+0000 7f7aa9600640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f7aa406aeb0 0x7f7aa406b290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:40788/0 (socket says 172.21.15.47:40788) 2024-02-19T15:26:32.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.019+0000 7f7aa9600640 1 -- 172.21.15.47:0/4248565374 learned_addr learned my addr 172.21.15.47:0/4248565374 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:32.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.019+0000 7f7aa8c00640 1 --2- 172.21.15.47:0/4248565374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7aa40cb520 0x7f7aa40cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:32.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.019+0000 7f7aaa000640 1 --2- 172.21.15.47:0/4248565374 >> v2:172.21.15.116:3300/0 conn(0x7f7aa40c6780 0x7f7aa40c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:32.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.019+0000 7f7aa8c00640 1 -- 172.21.15.47:0/4248565374 >> v2:172.21.15.116:3300/0 conn(0x7f7aa40c6780 msgr2=0x7f7aa40c6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:32.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.019+0000 7f7aa8c00640 1 --2- 172.21.15.47:0/4248565374 >> v2:172.21.15.116:3300/0 conn(0x7f7aa40c6780 0x7f7aa40c6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.019+0000 7f7aa3e00640 1 -- 172.21.15.47:0/4248565374 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 219232768 0 0) 0x7f7aa40ad840 con 0x7f7aa406aeb0 2024-02-19T15:26:32.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.019+0000 7f7aa8c00640 1 -- 172.21.15.47:0/4248565374 >> v1:172.21.15.47:6789/0 conn(0x7f7aa406aeb0 legacy=0x7f7aa406b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:32.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.019+0000 7f7aa8c00640 1 -- 172.21.15.47:0/4248565374 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa40cc250 con 0x7f7aa40cb520 2024-02-19T15:26:32.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.019+0000 7f7aa8c00640 1 --2- 172.21.15.47:0/4248565374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7aa40cb520 0x7f7aa40cb9a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7a900097c0 tx=0x7f7a9001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ac0189e2821ad6a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:32.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.019+0000 7f7aa3e00640 1 -- 172.21.15.47:0/4248565374 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa40c7960 con 0x7f7aa40cb520 2024-02-19T15:26:32.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.019+0000 7f7aa3e00640 1 -- 172.21.15.47:0/4248565374 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a900ab670 con 0x7f7aa40cb520 2024-02-19T15:26:32.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.019+0000 7f7aa3e00640 1 -- 172.21.15.47:0/4248565374 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a900aba00 con 0x7f7aa40cb520 2024-02-19T15:26:32.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aac000640 1 -- 172.21.15.47:0/4248565374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7aa40cb520 msgr2=0x7f7aa40cb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:32.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aac000640 1 --2- 172.21.15.47:0/4248565374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7aa40cb520 0x7f7aa40cb9a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7a900097c0 tx=0x7f7a9001c470 comp rx=0 tx=0).stop 2024-02-19T15:26:32.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aac000640 1 -- 172.21.15.47:0/4248565374 shutdown_connections 2024-02-19T15:26:32.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aac000640 1 --2- 172.21.15.47:0/4248565374 >> v2:172.21.15.116:3300/0 conn(0x7f7aa40c6780 0x7f7aa40c6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aac000640 1 --2- 172.21.15.47:0/4248565374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7aa40cb520 0x7f7aa40cb9a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aac000640 1 -- 172.21.15.47:0/4248565374 >> 172.21.15.47:0/4248565374 conn(0x7f7aa40d0ad0 msgr2=0x7f7aa40d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:32.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aac000640 1 -- 172.21.15.47:0/4248565374 shutdown_connections 2024-02-19T15:26:32.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aac000640 1 -- 172.21.15.47:0/4248565374 wait complete. 2024-02-19T15:26:32.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aac000640 1 Processor -- start 2024-02-19T15:26:32.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aac000640 1 -- start start 2024-02-19T15:26:32.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aac000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7aa406aeb0 0x7f7aa414b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:32.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aac000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7aa40c6780 0x7f7aa406c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:32.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aac000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aa40cb520 0x7f7aa406c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:32.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aac000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40b45c0 con 0x7f7aa40c6780 2024-02-19T15:26:32.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aac000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa414bcd0 con 0x7f7aa40cb520 2024-02-19T15:26:32.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aac000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40b49e0 con 0x7f7aa406aeb0 2024-02-19T15:26:32.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aa8c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7aa40c6780 0x7f7aa406c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:32.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aa9600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7aa406aeb0 0x7f7aa414b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:32.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aa9600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7aa406aeb0 0x7f7aa414b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:51894/0 (socket says 172.21.15.47:51894) 2024-02-19T15:26:32.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aa9600640 1 -- 172.21.15.47:0/2621364531 learned_addr learned my addr 172.21.15.47:0/2621364531 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:32.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aaa000640 1 --2- 172.21.15.47:0/2621364531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aa40cb520 0x7f7aa406c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:32.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aa9600640 1 -- 172.21.15.47:0/2621364531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aa40cb520 msgr2=0x7f7aa406c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:32.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aa9600640 1 --2- 172.21.15.47:0/2621364531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aa40cb520 0x7f7aa406c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aa9600640 1 -- 172.21.15.47:0/2621364531 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7aa40c6780 msgr2=0x7f7aa406c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:32.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aa9600640 1 --2- 172.21.15.47:0/2621364531 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7aa40c6780 0x7f7aa406c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.023+0000 7f7aa9600640 1 -- 172.21.15.47:0/2621364531 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a90009420 con 0x7f7aa406aeb0 2024-02-19T15:26:32.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.027+0000 7f7aa9600640 1 --2- 172.21.15.47:0/2621364531 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7aa406aeb0 0x7f7aa414b790 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7a9800c520 tx=0x7f7a9800c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:32.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.027+0000 7f7aa1600640 1 -- 172.21.15.47:0/2621364531 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a980074e0 con 0x7f7aa406aeb0 2024-02-19T15:26:32.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.027+0000 7f7aa1600640 1 -- 172.21.15.47:0/2621364531 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a98007640 con 0x7f7aa406aeb0 2024-02-19T15:26:32.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.027+0000 7f7aa1600640 1 -- 172.21.15.47:0/2621364531 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a980049f0 con 0x7f7aa406aeb0 2024-02-19T15:26:32.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.027+0000 7f7aac000640 1 -- 172.21.15.47:0/2621364531 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa40b3940 con 0x7f7aa406aeb0 2024-02-19T15:26:32.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.027+0000 7f7aac000640 1 -- 172.21.15.47:0/2621364531 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa40b3f00 con 0x7f7aa406aeb0 2024-02-19T15:26:32.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.027+0000 7f7aac000640 1 -- 172.21.15.47:0/2621364531 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa40b4830 con 0x7f7aa406aeb0 2024-02-19T15:26:32.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.027+0000 7f7aa1600640 1 -- 172.21.15.47:0/2621364531 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7a98004b50 con 0x7f7aa406aeb0 2024-02-19T15:26:32.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.027+0000 7f7aa1600640 1 --2- 172.21.15.47:0/2621364531 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a84066e40 0x7f7a84069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:32.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.027+0000 7f7aa1600640 1 -- 172.21.15.47:0/2621364531 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a98083c50 con 0x7f7aa406aeb0 2024-02-19T15:26:32.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.027+0000 7f7aa8c00640 1 --2- 172.21.15.47:0/2621364531 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a84066e40 0x7f7a84069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:32.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.031+0000 7f7aa8c00640 1 --2- 172.21.15.47:0/2621364531 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a84066e40 0x7f7a84069300 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7a900af000 tx=0x7f7a900a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:32.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.031+0000 7f7aa1600640 1 -- 172.21.15.47:0/2621364531 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7a98050ae0 con 0x7f7aa406aeb0 2024-02-19T15:26:32.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.191+0000 7f7aac000640 1 -- 172.21.15.47:0/2621364531 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7aa416ab80 con 0x7f7a84066e40 2024-02-19T15:26:32.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.203+0000 7f7aa1600640 1 -- 172.21.15.47:0/2621364531 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278233 (secure 0 0 0) 0x7f7a900ab860 con 0x7f7a84066e40 2024-02-19T15:26:32.207 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:32.209 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:26:32.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.203+0000 7f7aac000640 1 -- 172.21.15.47:0/2621364531 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a84066e40 msgr2=0x7f7a84069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:32.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.203+0000 7f7aac000640 1 --2- 172.21.15.47:0/2621364531 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a84066e40 0x7f7a84069300 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7a900af000 tx=0x7f7a900a7000 comp rx=0 tx=0).stop 2024-02-19T15:26:32.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.203+0000 7f7aac000640 1 -- 172.21.15.47:0/2621364531 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7aa406aeb0 msgr2=0x7f7aa414b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:32.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.203+0000 7f7aac000640 1 --2- 172.21.15.47:0/2621364531 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7aa406aeb0 0x7f7aa414b790 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7a9800c520 tx=0x7f7a9800c9e0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.203+0000 7f7aac000640 1 -- 172.21.15.47:0/2621364531 shutdown_connections 2024-02-19T15:26:32.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.203+0000 7f7aac000640 1 --2- 172.21.15.47:0/2621364531 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a84066e40 0x7f7a84069300 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.203+0000 7f7aac000640 1 --2- 172.21.15.47:0/2621364531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aa40cb520 0x7f7aa406c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.203+0000 7f7aac000640 1 --2- 172.21.15.47:0/2621364531 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7aa40c6780 0x7f7aa406c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.203+0000 7f7aac000640 1 --2- 172.21.15.47:0/2621364531 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7aa406aeb0 0x7f7aa414b790 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.203+0000 7f7aac000640 1 -- 172.21.15.47:0/2621364531 >> 172.21.15.47:0/2621364531 conn(0x7f7aa40d0ad0 msgr2=0x7f7aa40ceaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:32.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.203+0000 7f7aac000640 1 -- 172.21.15.47:0/2621364531 shutdown_connections 2024-02-19T15:26:32.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.203+0000 7f7aac000640 1 -- 172.21.15.47:0/2621364531 wait complete. 2024-02-19T15:26:32.227 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-02-19T15:26:31.210849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":84,"acting":84,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":263480,"kb_used_data":9844,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754711240,"statfs":{"total":773094113280,"available":772824309760,"internally_reserved":0,"allocated":10080256,"data_stored":4940203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":259575346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.902450"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.318242+0000","last_change":"2024-02-19T15:26:26.318242+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.318242+0000","last_undegraded":"2024-02-19T15:26:26.318242+0000","last_fullsized":"2024-02-19T15:26:26.318242+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.319202+0000","last_change":"2024-02-19T15:26:26.319202+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.319202+0000","last_undegraded":"2024-02-19T15:26:26.319202+0000","last_fullsized":"2024-02-19T15:26:26.319202+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.319262+0000","last_change":"2024-02-19T15:26:26.319262+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.319262+0000","last_undegraded":"2024-02-19T15:26:26.319262+0000","last_fullsized":"2024-02-19T15:26:26.319262+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.320398+0000","last_change":"2024-02-19T15:26:26.320398+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.320398+0000","last_undegraded":"2024-02-19T15:26:26.320398+0000","last_fullsized":"2024-02-19T15:26:26.320398+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315361+0000","last_change":"2024-02-19T15:26:25.315361+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315361+0000","last_undegraded":"2024-02-19T15:26:25.315361+0000","last_fullsized":"2024-02-19T15:26:25.315361+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315188+0000","last_change":"2024-02-19T15:26:25.315188+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315188+0000","last_undegraded":"2024-02-19T15:26:25.315188+0000","last_fullsized":"2024-02-19T15:26:25.315188+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315899+0000","last_change":"2024-02-19T15:26:25.315899+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315899+0000","last_undegraded":"2024-02-19T15:26:25.315899+0000","last_fullsized":"2024-02-19T15:26:25.315899+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.316019+0000","last_change":"2024-02-19T15:26:25.316019+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.316019+0000","last_undegraded":"2024-02-19T15:26:25.316019+0000","last_fullsized":"2024-02-19T15:26:25.316019+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.310338+0000","last_change":"2024-02-19T15:26:25.310338+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.310338+0000","last_undegraded":"2024-02-19T15:26:25.310338+0000","last_fullsized":"2024-02-19T15:26:25.310338+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.316341+0000","last_change":"2024-02-19T15:26:25.316341+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.316341+0000","last_undegraded":"2024-02-19T15:26:25.316341+0000","last_fullsized":"2024-02-19T15:26:25.316341+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-19T15:26:26.338255+0000","last_change":"2024-02-19T15:26:26.336796+0000","last_active":"2024-02-19T15:26:26.338255+0000","last_peered":"2024-02-19T15:26:26.338255+0000","last_clean":"2024-02-19T15:26:26.338255+0000","last_became_active":"2024-02-19T15:26:26.336376+0000","last_became_peered":"2024-02-19T15:26:26.336376+0000","last_unstale":"2024-02-19T15:26:26.338255+0000","last_undegraded":"2024-02-19T15:26:26.338255+0000","last_fullsized":"2024-02-19T15:26:26.338255+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:27:28.873521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.311184+0000","last_change":"2024-02-19T15:26:25.311184+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.311184+0000","last_undegraded":"2024-02-19T15:26:25.311184+0000","last_fullsized":"2024-02-19T15:26:25.311184+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.310305+0000","last_change":"2024-02-19T15:26:25.310305+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.310305+0000","last_undegraded":"2024-02-19T15:26:25.310305+0000","last_fullsized":"2024-02-19T15:26:25.310305+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.313986+0000","last_change":"2024-02-19T15:26:25.313986+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.313986+0000","last_undegraded":"2024-02-19T15:26:25.313986+0000","last_fullsized":"2024-02-19T15:26:25.313986+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.319592+0000","last_change":"2024-02-19T15:26:26.319592+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.319592+0000","last_undegraded":"2024-02-19T15:26:26.319592+0000","last_fullsized":"2024-02-19T15:26:26.319592+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.314616+0000","last_change":"2024-02-19T15:26:25.314616+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.314616+0000","last_undegraded":"2024-02-19T15:26:25.314616+0000","last_fullsized":"2024-02-19T15:26:25.314616+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315717+0000","last_change":"2024-02-19T15:26:25.315717+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315717+0000","last_undegraded":"2024-02-19T15:26:25.315717+0000","last_fullsized":"2024-02-19T15:26:25.315717+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.312770+0000","last_change":"2024-02-19T15:26:25.312770+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.312770+0000","last_undegraded":"2024-02-19T15:26:25.312770+0000","last_fullsized":"2024-02-19T15:26:25.312770+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.312441+0000","last_change":"2024-02-19T15:26:25.312441+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.312441+0000","last_undegraded":"2024-02-19T15:26:25.312441+0000","last_fullsized":"2024-02-19T15:26:25.312441+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315589+0000","last_change":"2024-02-19T15:26:25.315589+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315589+0000","last_undegraded":"2024-02-19T15:26:25.315589+0000","last_fullsized":"2024-02-19T15:26:25.315589+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.310289+0000","last_change":"2024-02-19T15:26:25.310289+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.310289+0000","last_undegraded":"2024-02-19T15:26:25.310289+0000","last_fullsized":"2024-02-19T15:26:25.310289+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.318671+0000","last_change":"2024-02-19T15:26:26.318671+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.318671+0000","last_undegraded":"2024-02-19T15:26:26.318671+0000","last_fullsized":"2024-02-19T15:26:26.318671+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315968+0000","last_change":"2024-02-19T15:26:25.315968+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315968+0000","last_undegraded":"2024-02-19T15:26:25.315968+0000","last_fullsized":"2024-02-19T15:26:25.315968+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-19T15:26:26.331522+0000","last_change":"2024-02-19T15:26:26.328022+0000","last_active":"2024-02-19T15:26:26.331522+0000","last_peered":"2024-02-19T15:26:26.331522+0000","last_clean":"2024-02-19T15:26:26.331522+0000","last_became_active":"2024-02-19T15:26:26.327714+0000","last_became_peered":"2024-02-19T15:26:26.327714+0000","last_unstale":"2024-02-19T15:26:26.331522+0000","last_undegraded":"2024-02-19T15:26:26.331522+0000","last_fullsized":"2024-02-19T15:26:26.331522+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:09:12.432031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315314+0000","last_change":"2024-02-19T15:26:25.315314+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315314+0000","last_undegraded":"2024-02-19T15:26:25.315314+0000","last_fullsized":"2024-02-19T15:26:25.315314+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.316123+0000","last_change":"2024-02-19T15:26:25.316123+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.316123+0000","last_undegraded":"2024-02-19T15:26:25.316123+0000","last_fullsized":"2024-02-19T15:26:25.316123+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.313762+0000","last_change":"2024-02-19T15:26:25.313762+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.313762+0000","last_undegraded":"2024-02-19T15:26:25.313762+0000","last_fullsized":"2024-02-19T15:26:25.313762+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.313774+0000","last_change":"2024-02-19T15:26:25.313774+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.313774+0000","last_undegraded":"2024-02-19T15:26:25.313774+0000","last_fullsized":"2024-02-19T15:26:25.313774+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.319175+0000","last_change":"2024-02-19T15:26:26.319175+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.319175+0000","last_undegraded":"2024-02-19T15:26:26.319175+0000","last_fullsized":"2024-02-19T15:26:26.319175+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315822+0000","last_change":"2024-02-19T15:26:25.315822+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315822+0000","last_undegraded":"2024-02-19T15:26:25.315822+0000","last_fullsized":"2024-02-19T15:26:25.315822+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.313928+0000","last_change":"2024-02-19T15:26:25.313928+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.313928+0000","last_undegraded":"2024-02-19T15:26:25.313928+0000","last_fullsized":"2024-02-19T15:26:25.313928+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315310+0000","last_change":"2024-02-19T15:26:25.315310+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315310+0000","last_undegraded":"2024-02-19T15:26:25.315310+0000","last_fullsized":"2024-02-19T15:26:25.315310+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315507+0000","last_change":"2024-02-19T15:26:25.315507+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315507+0000","last_undegraded":"2024-02-19T15:26:25.315507+0000","last_fullsized":"2024-02-19T15:26:25.315507+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.318591+0000","last_change":"2024-02-19T15:26:26.318591+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.318591+0000","last_undegraded":"2024-02-19T15:26:26.318591+0000","last_fullsized":"2024-02-19T15:26:26.318591+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-19T15:26:26.339710+0000","last_change":"2024-02-19T15:26:26.338468+0000","last_active":"2024-02-19T15:26:26.339710+0000","last_peered":"2024-02-19T15:26:26.339710+0000","last_clean":"2024-02-19T15:26:26.339710+0000","last_became_active":"2024-02-19T15:26:26.338119+0000","last_became_peered":"2024-02-19T15:26:26.338119+0000","last_unstale":"2024-02-19T15:26:26.339710+0000","last_undegraded":"2024-02-19T15:26:26.339710+0000","last_fullsized":"2024-02-19T15:26:26.339710+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:46:46.101170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315428+0000","last_change":"2024-02-19T15:26:25.315428+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315428+0000","last_undegraded":"2024-02-19T15:26:25.315428+0000","last_fullsized":"2024-02-19T15:26:25.315428+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-19T15:26:26.331603+0000","last_change":"2024-02-19T15:26:26.329556+0000","last_active":"2024-02-19T15:26:26.331603+0000","last_peered":"2024-02-19T15:26:26.331603+0000","last_clean":"2024-02-19T15:26:26.331603+0000","last_became_active":"2024-02-19T15:26:26.329110+0000","last_became_peered":"2024-02-19T15:26:26.329110+0000","last_unstale":"2024-02-19T15:26:26.331603+0000","last_undegraded":"2024-02-19T15:26:26.331603+0000","last_fullsized":"2024-02-19T15:26:26.331603+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:16:30.549123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.313959+0000","last_change":"2024-02-19T15:26:25.313959+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.313959+0000","last_undegraded":"2024-02-19T15:26:25.313959+0000","last_fullsized":"2024-02-19T15:26:25.313959+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.318242+0000","last_change":"2024-02-19T15:26:26.318242+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.318242+0000","last_undegraded":"2024-02-19T15:26:26.318242+0000","last_fullsized":"2024-02-19T15:26:26.318242+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.312292+0000","last_change":"2024-02-19T15:26:25.312292+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.312292+0000","last_undegraded":"2024-02-19T15:26:25.312292+0000","last_fullsized":"2024-02-19T15:26:25.312292+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.314635+0000","last_change":"2024-02-19T15:26:25.314635+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.314635+0000","last_undegraded":"2024-02-19T15:26:25.314635+0000","last_fullsized":"2024-02-19T15:26:25.314635+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":292,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-19T15:26:25.315555+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:26:25.315555+0000","last_peered":"2024-02-19T15:26:25.315555+0000","last_clean":"2024-02-19T15:26:25.315555+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:26:25.315555+0000","last_undegraded":"2024-02-19T15:26:25.315555+0000","last_fullsized":"2024-02-19T15:26:25.315555+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":1},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509498,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1122304,"data_stored":475424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.61399999999999999,"15min":0.61699999999999999},"min":{"1min":0.42899999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83599999999999997,"5min":1.022,"15min":1.022},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.68000000000000005,"15min":0.752},"min":{"1min":0.433,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.96699999999999997,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.64700000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.64500000000000002,"15min":0.65200000000000002},"min":{"1min":0.49299999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.81999999999999995,"5min":1.171,"15min":1.171},"last":0.5},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65600000000000003,"15min":0.65200000000000002},"min":{"1min":0.40300000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.51900000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.437,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.175,"5min":1.216,"15min":1.216},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.70599999999999996,"15min":0.69099999999999995},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.95099999999999996,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.54300000000000004}]},{"osd":3,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69499999999999995,"15min":0.68100000000000005},"min":{"1min":0.46899999999999997,"5min":0.438,"15min":0.438},"max":{"1min":0.90900000000000003,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67600000000000005,"15min":0.66200000000000003},"min":{"1min":0.48399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.87,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.72899999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.64900000000000002,"15min":0.61499999999999999},"min":{"1min":0.40400000000000003,"5min":0.309,"15min":0.309},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.752,"5min":0.66200000000000003,"15min":0.61499999999999999},"min":{"1min":0.41699999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.71099999999999997}]},{"osd":5,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.65700000000000003,"15min":0.61299999999999999},"min":{"1min":0.439,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.753},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.72199999999999998,"15min":0.68200000000000005},"min":{"1min":0.56899999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.73999999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.72199999999999998,"15min":0.66800000000000004},"min":{"1min":0.48899999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.74299999999999999,"15min":0.68899999999999995},"min":{"1min":0.67000000000000004,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.80800000000000005}]}]},{"osd":7,"up_from":14,"seq":60129542202,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1059517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.59699999999999998,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.87,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.67400000000000004},"min":{"1min":0.38800000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.77100000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.63500000000000001,"15min":0.60099999999999998},"min":{"1min":0.498,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86799999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.70399999999999996,"15min":0.72099999999999997},"min":{"1min":0.496,"5min":0.437,"15min":0.437},"max":{"1min":1.008,"5min":2.968,"15min":2.968},"last":0.73299999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.64700000000000002,"15min":0.63600000000000001},"min":{"1min":0.47599999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85399999999999998,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.497,"5min":0.438,"15min":0.438},"max":{"1min":0.97399999999999998,"5min":2.508,"15min":2.508},"last":0.68799999999999994}]},{"osd":3,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66300000000000003,"15min":0.65900000000000003},"min":{"1min":0.50900000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.91300000000000003,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.752},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.70999999999999996,"15min":0.69499999999999995},"min":{"1min":0.59399999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.94699999999999995,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":0.85699999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.59499999999999997,"15min":0.53900000000000003},"min":{"1min":0.432,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.61799999999999999,"15min":0.58199999999999996},"min":{"1min":0.40100000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93600000000000005,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.57199999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.65700000000000003,"15min":0.58399999999999996},"min":{"1min":0.53500000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.63800000000000001},"min":{"1min":0.54700000000000004,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.97299999999999998,"5min":1.046,"15min":1.046},"last":0.70899999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.67100000000000004,"15min":0.61299999999999999},"min":{"1min":0.46999999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.62},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72599999999999998,"15min":0.67500000000000004},"min":{"1min":0.59699999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.63700000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542203,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32268,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339572,"statfs":{"total":96636764160,"available":96603721728,"internally_reserved":0,"allocated":1110016,"data_stored":469149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.67200000000000004,"15min":0.64400000000000002},"min":{"1min":0.38300000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.97699999999999998,"5min":5.7789999999999999,"15min":5.7789999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.71099999999999997,"15min":0.63700000000000001},"min":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.86499999999999999,"5min":12.359999999999999,"15min":12.359999999999999},"last":0.53600000000000003}]},{"osd":1,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.73799999999999999,"15min":0.67000000000000004},"min":{"1min":0.432,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80000000000000004,"5min":10.938000000000001,"15min":10.938000000000001},"last":0.502},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.78900000000000003,"15min":0.71299999999999997},"min":{"1min":0.50600000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85599999999999998,"5min":10.531000000000001,"15min":10.531000000000001},"last":0.56499999999999995}]},{"osd":2,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.79700000000000004,"15min":0.72099999999999997},"min":{"1min":0.42999999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.89900000000000002,"5min":12.925000000000001,"15min":12.925000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.72999999999999998,"15min":0.64100000000000001},"min":{"1min":0.48899999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89000000000000001,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.46999999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.91100000000000003,"15min":0.73499999999999999},"min":{"1min":0.49399999999999999,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82299999999999995,"5min":26.161999999999999,"15min":26.161999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.91000000000000003,"15min":0.73999999999999999},"min":{"1min":0.47299999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1120000000000001,"5min":27.934999999999999,"15min":27.934999999999999},"last":0.52700000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.67500000000000004,"15min":0.59199999999999997},"min":{"1min":0.45700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83099999999999996,"5min":9.0549999999999997,"15min":9.0549999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.70899999999999996,"15min":0.67100000000000004},"min":{"1min":0.44600000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.92800000000000005,"5min":6.2549999999999999,"15min":6.2549999999999999},"last":0.33900000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.746,"15min":0.67000000000000004},"min":{"1min":0.50700000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.6539999999999999,"5min":6.5229999999999997,"15min":6.5229999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81899999999999995,"15min":0.749},"min":{"1min":0.63500000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.2310000000000001,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":0.48299999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.77600000000000002,"15min":0.69399999999999995},"min":{"1min":0.41499999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.95399999999999996,"5min":11.986000000000001,"15min":11.986000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.77700000000000002,"15min":0.71499999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.89100000000000001,"5min":8.9770000000000003,"15min":8.9770000000000003},"last":0.45200000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32268,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339572,"statfs":{"total":96636764160,"available":96603721728,"internally_reserved":0,"allocated":1110016,"data_stored":469149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.378,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.623,"15min":0.60499999999999998},"min":{"1min":0.47099999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85499999999999998,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.61199999999999999}]},{"osd":1,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64100000000000001,"15min":0.61899999999999999},"min":{"1min":0.48999999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.879,"5min":1.081,"15min":1.081},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64400000000000002,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83699999999999997,"5min":1.008,"15min":1.008},"last":0.56999999999999995}]},{"osd":2,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.65400000000000003,"15min":0.66000000000000003},"min":{"1min":0.49299999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.90700000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68200000000000005,"15min":0.67600000000000005},"min":{"1min":0.45100000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92000000000000004,"5min":1.002,"15min":1.002},"last":0.55400000000000005}]},{"osd":3,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.70799999999999996,"15min":0.68200000000000005},"min":{"1min":0.50600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94899999999999995,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.65900000000000003,"15min":0.65300000000000002},"min":{"1min":0.42799999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89600000000000002,"5min":1.153,"15min":1.153},"last":0.69099999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63600000000000001,"15min":0.60599999999999998},"min":{"1min":0.433,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85399999999999998,"5min":1.042,"15min":1.042},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70599999999999996,"15min":0.68200000000000005},"min":{"1min":0.54200000000000004,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.94799999999999995,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.749}]},{"osd":6,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.622,"15min":0.58299999999999996},"min":{"1min":0.376,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.92200000000000004,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68600000000000005,"15min":0.65500000000000003},"min":{"1min":0.52900000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90800000000000003,"5min":1.054,"15min":1.054},"last":0.63600000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.70199999999999996,"15min":0.67100000000000004},"min":{"1min":0.44800000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.98199999999999998,"5min":1.115,"15min":1.115},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.75600000000000001,"15min":0.73399999999999999},"min":{"1min":0.46999999999999997,"5min":0.443,"15min":0.443},"max":{"1min":1.0169999999999999,"5min":1.24,"15min":1.24},"last":0.78700000000000003}]}]},{"osd":3,"up_from":14,"seq":60129542201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32276,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339564,"statfs":{"total":96636764160,"available":96603713536,"internally_reserved":0,"allocated":1110016,"data_stored":469149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42899999999999999,"15min":0.44700000000000001},"min":{"1min":0.23699999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76600000000000001,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.54100000000000004,"15min":0.64600000000000002},"min":{"1min":0.27500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67600000000000005,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.53600000000000003}]},{"osd":1,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.433,"15min":0.442},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.66000000000000003,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.59699999999999998,"15min":0.70899999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.66400000000000003,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.67700000000000005}]},{"osd":2,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56799999999999995,"15min":0.61199999999999999},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.66500000000000004,"5min":1.008,"15min":1.008},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55000000000000004,"15min":0.58899999999999997},"min":{"1min":0.39900000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86199999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.82799999999999996}]},{"osd":4,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.70299999999999996,"15min":0.67900000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.90600000000000003,"5min":1.004,"15min":1.004},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.80100000000000005,"15min":0.86099999999999999},"min":{"1min":0.58199999999999996,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0269999999999999,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.63100000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.80100000000000005,"15min":0.85499999999999998},"min":{"1min":0.54700000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.96699999999999997,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.74199999999999999,"15min":0.748},"min":{"1min":0.53300000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.93899999999999995,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.86899999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:26:06 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.82499999999999996,"15min":0.89000000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92300000000000004,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.78500000000000003,"15min":0.81299999999999994},"min":{"1min":0.55400000000000005,"5min":0.502,"15min":0.502},"max":{"1min":1.0009999999999999,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.91700000000000004}]},{"osd":7,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76400000000000001,"15min":0.78000000000000003},"min":{"1min":0.52700000000000002,"5min":0.504,"15min":0.504},"max":{"1min":0.96799999999999997,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.84099999999999997,"15min":0.92000000000000004},"min":{"1min":0.57399999999999995,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0009999999999999,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.80100000000000005}]}]},{"osd":2,"up_from":14,"seq":60129542202,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1110016,"data_stored":469149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.66700000000000004,"15min":0.69699999999999995},"min":{"1min":0.27400000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.64400000000000002,"5min":15.42,"15min":15.42},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.71499999999999997,"15min":0.88500000000000001},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.76500000000000001,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.38100000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.75700000000000001,"15min":0.89700000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58999999999999997,"5min":9.5269999999999992,"15min":9.5269999999999992},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.94199999999999995,"15min":1.006},"min":{"1min":0.39700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.621,"5min":22.954999999999998,"15min":22.954999999999998},"last":0.41499999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.67100000000000004,"15min":0.78300000000000003},"min":{"1min":0.376,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66200000000000003,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.74299999999999999,"15min":0.82299999999999995},"min":{"1min":0.41799999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69399999999999995,"5min":7.734,"15min":7.734},"last":0.48499999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.91400000000000003,"15min":0.90300000000000002},"min":{"1min":0.52000000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.016,"5min":15.369,"15min":15.369},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.82699999999999996,"15min":0.89300000000000002},"min":{"1min":0.53800000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93700000000000006,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.59199999999999997}]},{"osd":5,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.871,"15min":0.88900000000000001},"min":{"1min":0.53900000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.95999999999999996,"5min":7.6520000000000001,"15min":7.6520000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.98699999999999999,"15min":0.97499999999999998},"min":{"1min":0.60199999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0660000000000001,"5min":15.391,"15min":15.391},"last":0.63300000000000001}]},{"osd":6,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.95799999999999996,"15min":0.94399999999999995},"min":{"1min":0.54900000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.90100000000000002,"5min":15.337,"15min":15.337},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.876,"15min":0.95799999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.98499999999999999,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.68899999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.90600000000000003,"15min":0.98099999999999998},"min":{"1min":0.58399999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.028,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.91600000000000004,"15min":0.95299999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.98099999999999998,"5min":7.6929999999999996,"15min":7.6929999999999996},"last":0.54800000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542203,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1110016,"data_stored":469149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.39200000000000002,"15min":0.42399999999999999},"min":{"1min":0.248,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56799999999999995,"5min":0.747,"15min":0.747},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.438,"15min":0.46600000000000003},"min":{"1min":0.28100000000000003,"5min":0.25,"15min":0.25},"max":{"1min":0.69599999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.49399999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.42599999999999999,"15min":0.44400000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.56100000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.48799999999999999,"15min":0.52600000000000002},"min":{"1min":0.36299999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.501,"5min":1.071,"15min":1.071},"last":0.379}]},{"osd":3,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58899999999999997,"15min":0.63300000000000001},"min":{"1min":0.44800000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.64600000000000002,"5min":1.365,"15min":1.365},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.57399999999999995,"15min":0.63},"min":{"1min":0.42999999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.61399999999999999,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.47099999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.65300000000000002,"15min":0.61299999999999999},"min":{"1min":0.52100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.99099999999999999,"5min":1.127,"15min":1.127},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.69299999999999995,"15min":0.68000000000000005},"min":{"1min":0.54900000000000004,"5min":0.435,"15min":0.435},"max":{"1min":0.98399999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.70299999999999996}]},{"osd":5,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70899999999999996,"15min":0.69699999999999995},"min":{"1min":0.54200000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.95599999999999996,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.70399999999999996,"15min":0.68500000000000005},"min":{"1min":0.503,"5min":0.443,"15min":0.443},"max":{"1min":0.94199999999999995,"5min":1.107,"15min":1.107},"last":0.80500000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.72899999999999998},"min":{"1min":0.57099999999999995,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.97399999999999998,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.74199999999999999,"15min":0.745},"min":{"1min":0.60299999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97199999999999998,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.73999999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.76800000000000002,"15min":0.78600000000000003},"min":{"1min":0.59999999999999998,"5min":0.496,"15min":0.496},"max":{"1min":0.95299999999999996,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.76400000000000001,"15min":0.748},"min":{"1min":0.58499999999999996,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0229999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.82799999999999996}]}]},{"osd":0,"up_from":14,"seq":60129542201,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37008,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334832,"statfs":{"total":96636764160,"available":96598867968,"internally_reserved":0,"allocated":1703936,"data_stored":1059517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.50700000000000001,"15min":0.628},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.66200000000000003,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.31},{"interface":"front","average":{"1min":0.443,"5min":0.46300000000000002,"15min":0.48299999999999998},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.76000000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.46899999999999997}]},{"osd":2,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.65400000000000003,"15min":0.78400000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.67500000000000004,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.61799999999999999,"15min":0.746},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69099999999999995,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.36199999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.56699999999999995,"15min":0.67900000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.61699999999999999,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.436},{"interface":"front","average":{"1min":0.495,"5min":0.54800000000000004,"15min":0.60999999999999999},"min":{"1min":0.38100000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.72499999999999998,"5min":1.345,"15min":1.345},"last":0.41499999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.745,"15min":0.74299999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.0700000000000001,"5min":1.095,"15min":1.095},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.84399999999999997,"15min":0.92500000000000004},"min":{"1min":0.64000000000000001,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0049999999999999,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.73299999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.73799999999999999,"15min":0.751},"min":{"1min":0.55900000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.97999999999999998,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.77300000000000002,"15min":0.85299999999999998},"min":{"1min":0.52900000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.05,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.55700000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.82099999999999995,"15min":0.91500000000000004},"min":{"1min":0.59799999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.95899999999999996,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73799999999999999,"15min":0.75700000000000001},"min":{"1min":0.57999999999999996,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.84999999999999998,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.57799999999999996}]},{"osd":7,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.85199999999999998,"15min":0.94199999999999995},"min":{"1min":0.58599999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.0229999999999999,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.83499999999999996,"15min":0.94099999999999995},"min":{"1min":0.55900000000000005,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.96499999999999997,"5min":4.915,"15min":4.915},"last":0.69499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:26:32.231 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3f is not active+clean 2024-02-19T15:26:32.232 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.232 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3e is not active+clean 2024-02-19T15:26:32.232 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.232 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3d is not active+clean 2024-02-19T15:26:32.232 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.232 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3c is not active+clean 2024-02-19T15:26:32.232 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.232 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3b is not active+clean 2024-02-19T15:26:32.232 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:26.318242+0000', 'last_change': '2024-02-19T15:26:26.318242+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.318242+0000', 'last_undegraded': '2024-02-19T15:26:26.318242+0000', 'last_fullsized': '2024-02-19T15:26:26.318242+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:26:32.232 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3a is not active+clean 2024-02-19T15:26:32.232 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.233 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.39 is not active+clean 2024-02-19T15:26:32.233 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.233 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.38 is not active+clean 2024-02-19T15:26:32.233 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.233 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.37 is not active+clean 2024-02-19T15:26:32.233 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.233 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.36 is not active+clean 2024-02-19T15:26:32.233 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:26.319202+0000', 'last_change': '2024-02-19T15:26:26.319202+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.319202+0000', 'last_undegraded': '2024-02-19T15:26:26.319202+0000', 'last_fullsized': '2024-02-19T15:26:26.319202+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:26:32.233 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.35 is not active+clean 2024-02-19T15:26:32.233 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.233 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.34 is not active+clean 2024-02-19T15:26:32.233 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.234 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.33 is not active+clean 2024-02-19T15:26:32.234 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.234 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.32 is not active+clean 2024-02-19T15:26:32.234 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.234 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.31 is not active+clean 2024-02-19T15:26:32.234 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.234 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.30 is not active+clean 2024-02-19T15:26:32.234 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.234 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.2f is not active+clean 2024-02-19T15:26:32.234 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.234 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.2e is not active+clean 2024-02-19T15:26:32.234 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.235 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.2d is not active+clean 2024-02-19T15:26:32.235 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.235 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.2c is not active+clean 2024-02-19T15:26:32.235 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.235 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.2b is not active+clean 2024-02-19T15:26:32.235 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.235 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.2a is not active+clean 2024-02-19T15:26:32.235 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.235 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.29 is not active+clean 2024-02-19T15:26:32.235 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.235 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.28 is not active+clean 2024-02-19T15:26:32.236 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.236 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.27 is not active+clean 2024-02-19T15:26:32.236 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:26.319262+0000', 'last_change': '2024-02-19T15:26:26.319262+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.319262+0000', 'last_undegraded': '2024-02-19T15:26:26.319262+0000', 'last_fullsized': '2024-02-19T15:26:26.319262+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:26:32.236 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.26 is not active+clean 2024-02-19T15:26:32.236 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.236 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.25 is not active+clean 2024-02-19T15:26:32.236 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.236 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.24 is not active+clean 2024-02-19T15:26:32.236 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:26.320398+0000', 'last_change': '2024-02-19T15:26:26.320398+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.320398+0000', 'last_undegraded': '2024-02-19T15:26:26.320398+0000', 'last_fullsized': '2024-02-19T15:26:26.320398+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:26:32.236 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.23 is not active+clean 2024-02-19T15:26:32.236 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.236 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.22 is not active+clean 2024-02-19T15:26:32.237 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.237 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.21 is not active+clean 2024-02-19T15:26:32.237 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.237 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.20 is not active+clean 2024-02-19T15:26:32.237 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.237 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1e is not active+clean 2024-02-19T15:26:32.237 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.237 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1f is not active+clean 2024-02-19T15:26:32.237 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.315361+0000', 'last_change': '2024-02-19T15:26:25.315361+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.315361+0000', 'last_undegraded': '2024-02-19T15:26:25.315361+0000', 'last_fullsized': '2024-02-19T15:26:25.315361+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:26:32.237 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1f is not active+clean 2024-02-19T15:26:32.237 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.237 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1e is not active+clean 2024-02-19T15:26:32.238 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.315188+0000', 'last_change': '2024-02-19T15:26:25.315188+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.315188+0000', 'last_undegraded': '2024-02-19T15:26:25.315188+0000', 'last_fullsized': '2024-02-19T15:26:25.315188+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T15:26:32.238 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1c is not active+clean 2024-02-19T15:26:32.238 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.238 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1d is not active+clean 2024-02-19T15:26:32.238 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.315899+0000', 'last_change': '2024-02-19T15:26:25.315899+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.315899+0000', 'last_undegraded': '2024-02-19T15:26:25.315899+0000', 'last_fullsized': '2024-02-19T15:26:25.315899+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:26:32.238 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1d is not active+clean 2024-02-19T15:26:32.238 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.238 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1c is not active+clean 2024-02-19T15:26:32.238 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.316019+0000', 'last_change': '2024-02-19T15:26:25.316019+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.316019+0000', 'last_undegraded': '2024-02-19T15:26:25.316019+0000', 'last_fullsized': '2024-02-19T15:26:25.316019+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:26:32.238 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1a is not active+clean 2024-02-19T15:26:32.238 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.238 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1b is not active+clean 2024-02-19T15:26:32.239 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.310338+0000', 'last_change': '2024-02-19T15:26:25.310338+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.310338+0000', 'last_undegraded': '2024-02-19T15:26:25.310338+0000', 'last_fullsized': '2024-02-19T15:26:25.310338+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T15:26:32.239 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1b is not active+clean 2024-02-19T15:26:32.239 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.239 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1a is not active+clean 2024-02-19T15:26:32.239 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.316341+0000', 'last_change': '2024-02-19T15:26:25.316341+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.316341+0000', 'last_undegraded': '2024-02-19T15:26:25.316341+0000', 'last_fullsized': '2024-02-19T15:26:25.316341+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:26:32.239 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.18 is not active+clean 2024-02-19T15:26:32.239 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.239 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.19 is not active+clean 2024-02-19T15:26:32.239 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.239 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.18 is not active+clean 2024-02-19T15:26:32.239 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.311184+0000', 'last_change': '2024-02-19T15:26:25.311184+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.311184+0000', 'last_undegraded': '2024-02-19T15:26:25.311184+0000', 'last_fullsized': '2024-02-19T15:26:25.311184+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T15:26:32.239 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.16 is not active+clean 2024-02-19T15:26:32.240 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.240 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.17 is not active+clean 2024-02-19T15:26:32.240 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.310305+0000', 'last_change': '2024-02-19T15:26:25.310305+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.310305+0000', 'last_undegraded': '2024-02-19T15:26:25.310305+0000', 'last_fullsized': '2024-02-19T15:26:25.310305+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T15:26:32.240 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.17 is not active+clean 2024-02-19T15:26:32.240 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.240 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.16 is not active+clean 2024-02-19T15:26:32.240 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.313986+0000', 'last_change': '2024-02-19T15:26:25.313986+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.313986+0000', 'last_undegraded': '2024-02-19T15:26:25.313986+0000', 'last_fullsized': '2024-02-19T15:26:25.313986+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T15:26:32.240 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.14 is not active+clean 2024-02-19T15:26:32.240 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:26.319592+0000', 'last_change': '2024-02-19T15:26:26.319592+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.319592+0000', 'last_undegraded': '2024-02-19T15:26:26.319592+0000', 'last_fullsized': '2024-02-19T15:26:26.319592+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:26:32.240 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.15 is not active+clean 2024-02-19T15:26:32.240 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.314616+0000', 'last_change': '2024-02-19T15:26:25.314616+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.314616+0000', 'last_undegraded': '2024-02-19T15:26:25.314616+0000', 'last_fullsized': '2024-02-19T15:26:25.314616+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T15:26:32.241 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.15 is not active+clean 2024-02-19T15:26:32.241 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.241 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.14 is not active+clean 2024-02-19T15:26:32.241 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.315717+0000', 'last_change': '2024-02-19T15:26:25.315717+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.315717+0000', 'last_undegraded': '2024-02-19T15:26:25.315717+0000', 'last_fullsized': '2024-02-19T15:26:25.315717+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T15:26:32.241 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.12 is not active+clean 2024-02-19T15:26:32.241 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.241 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.13 is not active+clean 2024-02-19T15:26:32.241 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.312770+0000', 'last_change': '2024-02-19T15:26:25.312770+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.312770+0000', 'last_undegraded': '2024-02-19T15:26:25.312770+0000', 'last_fullsized': '2024-02-19T15:26:25.312770+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T15:26:32.241 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.13 is not active+clean 2024-02-19T15:26:32.241 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.241 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.12 is not active+clean 2024-02-19T15:26:32.242 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.312441+0000', 'last_change': '2024-02-19T15:26:25.312441+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.312441+0000', 'last_undegraded': '2024-02-19T15:26:25.312441+0000', 'last_fullsized': '2024-02-19T15:26:25.312441+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T15:26:32.242 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.10 is not active+clean 2024-02-19T15:26:32.242 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.242 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.11 is not active+clean 2024-02-19T15:26:32.242 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.315589+0000', 'last_change': '2024-02-19T15:26:25.315589+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.315589+0000', 'last_undegraded': '2024-02-19T15:26:25.315589+0000', 'last_fullsized': '2024-02-19T15:26:25.315589+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T15:26:32.242 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.11 is not active+clean 2024-02-19T15:26:32.242 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.242 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.10 is not active+clean 2024-02-19T15:26:32.242 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.310289+0000', 'last_change': '2024-02-19T15:26:25.310289+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.310289+0000', 'last_undegraded': '2024-02-19T15:26:25.310289+0000', 'last_fullsized': '2024-02-19T15:26:25.310289+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T15:26:32.242 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.f is not active+clean 2024-02-19T15:26:32.242 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:26.318671+0000', 'last_change': '2024-02-19T15:26:26.318671+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.318671+0000', 'last_undegraded': '2024-02-19T15:26:26.318671+0000', 'last_fullsized': '2024-02-19T15:26:26.318671+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:26:32.243 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.e is not active+clean 2024-02-19T15:26:32.243 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.315968+0000', 'last_change': '2024-02-19T15:26:25.315968+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.315968+0000', 'last_undegraded': '2024-02-19T15:26:25.315968+0000', 'last_fullsized': '2024-02-19T15:26:25.315968+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T15:26:32.243 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.c is not active+clean 2024-02-19T15:26:32.243 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.243 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.d is not active+clean 2024-02-19T15:26:32.243 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.243 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.c is not active+clean 2024-02-19T15:26:32.243 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.315314+0000', 'last_change': '2024-02-19T15:26:25.315314+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.315314+0000', 'last_undegraded': '2024-02-19T15:26:25.315314+0000', 'last_fullsized': '2024-02-19T15:26:25.315314+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:26:32.243 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.a is not active+clean 2024-02-19T15:26:32.243 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.243 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.b is not active+clean 2024-02-19T15:26:32.244 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.316123+0000', 'last_change': '2024-02-19T15:26:25.316123+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.316123+0000', 'last_undegraded': '2024-02-19T15:26:25.316123+0000', 'last_fullsized': '2024-02-19T15:26:25.316123+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:26:32.244 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.b is not active+clean 2024-02-19T15:26:32.244 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.244 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.a is not active+clean 2024-02-19T15:26:32.244 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.313762+0000', 'last_change': '2024-02-19T15:26:25.313762+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.313762+0000', 'last_undegraded': '2024-02-19T15:26:25.313762+0000', 'last_fullsized': '2024-02-19T15:26:25.313762+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T15:26:32.244 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.8 is not active+clean 2024-02-19T15:26:32.244 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.244 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.9 is not active+clean 2024-02-19T15:26:32.244 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.313774+0000', 'last_change': '2024-02-19T15:26:25.313774+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.313774+0000', 'last_undegraded': '2024-02-19T15:26:25.313774+0000', 'last_fullsized': '2024-02-19T15:26:25.313774+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T15:26:32.244 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.9 is not active+clean 2024-02-19T15:26:32.244 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:26.319175+0000', 'last_change': '2024-02-19T15:26:26.319175+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.319175+0000', 'last_undegraded': '2024-02-19T15:26:26.319175+0000', 'last_fullsized': '2024-02-19T15:26:26.319175+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:26:32.245 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.8 is not active+clean 2024-02-19T15:26:32.245 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.315822+0000', 'last_change': '2024-02-19T15:26:25.315822+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.315822+0000', 'last_undegraded': '2024-02-19T15:26:25.315822+0000', 'last_fullsized': '2024-02-19T15:26:25.315822+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:26:32.245 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.6 is not active+clean 2024-02-19T15:26:32.245 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.245 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.7 is not active+clean 2024-02-19T15:26:32.245 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.313928+0000', 'last_change': '2024-02-19T15:26:25.313928+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.313928+0000', 'last_undegraded': '2024-02-19T15:26:25.313928+0000', 'last_fullsized': '2024-02-19T15:26:25.313928+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T15:26:32.245 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.7 is not active+clean 2024-02-19T15:26:32.245 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.245 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.6 is not active+clean 2024-02-19T15:26:32.245 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.315310+0000', 'last_change': '2024-02-19T15:26:25.315310+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.315310+0000', 'last_undegraded': '2024-02-19T15:26:25.315310+0000', 'last_fullsized': '2024-02-19T15:26:25.315310+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:26:32.245 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.4 is not active+clean 2024-02-19T15:26:32.246 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.246 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.5 is not active+clean 2024-02-19T15:26:32.246 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.315507+0000', 'last_change': '2024-02-19T15:26:25.315507+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.315507+0000', 'last_undegraded': '2024-02-19T15:26:25.315507+0000', 'last_fullsized': '2024-02-19T15:26:25.315507+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:26:32.246 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.5 is not active+clean 2024-02-19T15:26:32.246 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:26.318591+0000', 'last_change': '2024-02-19T15:26:26.318591+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.318591+0000', 'last_undegraded': '2024-02-19T15:26:26.318591+0000', 'last_fullsized': '2024-02-19T15:26:26.318591+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:26:32.246 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.2 is not active+clean 2024-02-19T15:26:32.246 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.246 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3 is not active+clean 2024-02-19T15:26:32.246 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.315428+0000', 'last_change': '2024-02-19T15:26:25.315428+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.315428+0000', 'last_undegraded': '2024-02-19T15:26:25.315428+0000', 'last_fullsized': '2024-02-19T15:26:25.315428+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:26:32.246 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3 is not active+clean 2024-02-19T15:26:32.246 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.247 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.0 is not active+clean 2024-02-19T15:26:32.247 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.247 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1 is not active+clean 2024-02-19T15:26:32.247 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.313959+0000', 'last_change': '2024-02-19T15:26:25.313959+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.313959+0000', 'last_undegraded': '2024-02-19T15:26:25.313959+0000', 'last_fullsized': '2024-02-19T15:26:25.313959+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T15:26:32.247 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1 is not active+clean 2024-02-19T15:26:32.247 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:26.318242+0000', 'last_change': '2024-02-19T15:26:26.318242+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.318242+0000', 'last_undegraded': '2024-02-19T15:26:26.318242+0000', 'last_fullsized': '2024-02-19T15:26:26.318242+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:26:32.247 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.0 is not active+clean 2024-02-19T15:26:32.247 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.312292+0000', 'last_change': '2024-02-19T15:26:25.312292+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.312292+0000', 'last_undegraded': '2024-02-19T15:26:25.312292+0000', 'last_fullsized': '2024-02-19T15:26:25.312292+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T15:26:32.247 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.e is not active+clean 2024-02-19T15:26:32.247 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:26:26.307873+0000', 'last_change': '2024-02-19T15:26:26.307873+0000', 'last_active': '2024-02-19T15:26:26.307873+0000', 'last_peered': '2024-02-19T15:26:26.307873+0000', 'last_clean': '2024-02-19T15:26:26.307873+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:26.307873+0000', 'last_undegraded': '2024-02-19T15:26:26.307873+0000', 'last_fullsized': '2024-02-19T15:26:26.307873+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:26.307873+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:26:32.247 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.f is not active+clean 2024-02-19T15:26:32.248 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:26:25.314635+0000', 'last_change': '2024-02-19T15:26:25.314635+0000', 'last_active': '2024-02-19T15:26:25.301961+0000', 'last_peered': '2024-02-19T15:26:25.301961+0000', 'last_clean': '2024-02-19T15:26:25.301961+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:26:25.314635+0000', 'last_undegraded': '2024-02-19T15:26:25.314635+0000', 'last_fullsized': '2024-02-19T15:26:25.314635+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'last_clean_scrub_stamp': '2024-02-19T15:26:25.301961+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T15:26:32.248 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-02-19T15:26:32.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.403+0000 7f34b7400640 1 Processor -- start 2024-02-19T15:26:32.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.403+0000 7f34b7400640 1 -- start start 2024-02-19T15:26:32.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.403+0000 7f34b7400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f34b015d6e0 0x7f34b0162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:32.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.403+0000 7f34b7400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b00718c0 con 0x7f34b0071350 2024-02-19T15:26:32.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.403+0000 7f34b7400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b0071ba0 con 0x7f34b015b1a0 2024-02-19T15:26:32.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.407+0000 7f34b7400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b0071e80 con 0x7f34b015d6e0 2024-02-19T15:26:32.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.407+0000 7f34afe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f34b015d6e0 0x7f34b0162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:32.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.407+0000 7f34b5400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f34b0071350 0x7f34b0164c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:40800/0 (socket says 172.21.15.47:40800) 2024-02-19T15:26:32.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.407+0000 7f34b5400640 1 -- 172.21.15.47:0/1766494766 learned_addr learned my addr 172.21.15.47:0/1766494766 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:32.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.407+0000 7f34afe00640 1 -- 172.21.15.47:0/1766494766 >> v1:172.21.15.116:6789/0 conn(0x7f34b015b1a0 legacy=0x7f34b015b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:32.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.407+0000 7f34afe00640 1 -- 172.21.15.47:0/1766494766 >> v1:172.21.15.47:6789/0 conn(0x7f34b0071350 legacy=0x7f34b0164c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:32.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.407+0000 7f34af400640 1 -- 172.21.15.47:0/1766494766 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997371800 0 0) 0x7f34b0071ba0 con 0x7f34b015b1a0 2024-02-19T15:26:32.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.407+0000 7f34afe00640 1 -- 172.21.15.47:0/1766494766 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b0072160 con 0x7f34b015d6e0 2024-02-19T15:26:32.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.407+0000 7f34afe00640 1 --2- 172.21.15.47:0/1766494766 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34b015d6e0 0x7f34b0162740 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f349c0097f0 tx=0x7f349c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad0a456246a3f43 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:32.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.407+0000 7f34af400640 1 -- 172.21.15.47:0/1766494766 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c0ac620 con 0x7f34b015d6e0 2024-02-19T15:26:32.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.407+0000 7f34af400640 1 -- 172.21.15.47:0/1766494766 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349c0ac780 con 0x7f34b015d6e0 2024-02-19T15:26:32.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.407+0000 7f34af400640 1 -- 172.21.15.47:0/1766494766 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c0acb10 con 0x7f34b015d6e0 2024-02-19T15:26:32.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34b7400640 1 -- 172.21.15.47:0/1766494766 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34b015d6e0 msgr2=0x7f34b0162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:32.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34b7400640 1 --2- 172.21.15.47:0/1766494766 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34b015d6e0 0x7f34b0162740 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f349c0097f0 tx=0x7f349c01c4a0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34b7400640 1 -- 172.21.15.47:0/1766494766 shutdown_connections 2024-02-19T15:26:32.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34b7400640 1 --2- 172.21.15.47:0/1766494766 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34b015d6e0 0x7f34b0162740 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34b7400640 1 -- 172.21.15.47:0/1766494766 >> 172.21.15.47:0/1766494766 conn(0x7f34b00b16a0 msgr2=0x7f34b00b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:32.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34b7400640 1 -- 172.21.15.47:0/1766494766 shutdown_connections 2024-02-19T15:26:32.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34b7400640 1 -- 172.21.15.47:0/1766494766 wait complete. 2024-02-19T15:26:32.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34b7400640 1 Processor -- start 2024-02-19T15:26:32.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34b7400640 1 -- start start 2024-02-19T15:26:32.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34b7400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34b0071350 0x7f34b0145b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:32.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34b7400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34b015b1a0 0x7f34b0148630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:32.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34b7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34b015d6e0 0x7f34b014dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:32.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34b7400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b01490a0 con 0x7f34b015b1a0 2024-02-19T15:26:32.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34b7400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b0149380 con 0x7f34b015d6e0 2024-02-19T15:26:32.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34b7400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b0149660 con 0x7f34b0071350 2024-02-19T15:26:32.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34afe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34b015b1a0 0x7f34b0148630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:32.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34afe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34b015b1a0 0x7f34b0148630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58476/0 (socket says 172.21.15.47:58476) 2024-02-19T15:26:32.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34afe00640 1 -- 172.21.15.47:0/2712737016 learned_addr learned my addr 172.21.15.47:0/2712737016 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:32.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34b4a00640 1 --2- 172.21.15.47:0/2712737016 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34b0071350 0x7f34b0145b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:32.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34b5400640 1 --2- 172.21.15.47:0/2712737016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34b015d6e0 0x7f34b014dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:32.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34afe00640 1 -- 172.21.15.47:0/2712737016 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34b0071350 msgr2=0x7f34b0145b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:32.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34afe00640 1 --2- 172.21.15.47:0/2712737016 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34b0071350 0x7f34b0145b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34afe00640 1 -- 172.21.15.47:0/2712737016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34b015d6e0 msgr2=0x7f34b014dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:32.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34afe00640 1 --2- 172.21.15.47:0/2712737016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34b015d6e0 0x7f34b014dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34afe00640 1 -- 172.21.15.47:0/2712737016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c009420 con 0x7f34b015b1a0 2024-02-19T15:26:32.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.411+0000 7f34afe00640 1 --2- 172.21.15.47:0/2712737016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34b015b1a0 0x7f34b0148630 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f349c0097f0 tx=0x7f349c019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:32.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.415+0000 7f34acc00640 1 -- 172.21.15.47:0/2712737016 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c0b6470 con 0x7f34b015b1a0 2024-02-19T15:26:32.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.415+0000 7f34acc00640 1 -- 172.21.15.47:0/2712737016 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349c0b65f0 con 0x7f34b015b1a0 2024-02-19T15:26:32.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.415+0000 7f34acc00640 1 -- 172.21.15.47:0/2712737016 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c0b69a0 con 0x7f34b015b1a0 2024-02-19T15:26:32.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.415+0000 7f34b7400640 1 -- 172.21.15.47:0/2712737016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b0149940 con 0x7f34b015b1a0 2024-02-19T15:26:32.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.415+0000 7f34b7400640 1 -- 172.21.15.47:0/2712737016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b0146530 con 0x7f34b015b1a0 2024-02-19T15:26:32.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.415+0000 7f34b7400640 1 -- 172.21.15.47:0/2712737016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b0075360 con 0x7f34b015b1a0 2024-02-19T15:26:32.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.415+0000 7f34acc00640 1 -- 172.21.15.47:0/2712737016 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f349c0b3040 con 0x7f34b015b1a0 2024-02-19T15:26:32.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.415+0000 7f34acc00640 1 --2- 172.21.15.47:0/2712737016 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3490066e40 0x7f3490069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:32.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.415+0000 7f34acc00640 1 -- 172.21.15.47:0/2712737016 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f349c11eec0 con 0x7f34b015b1a0 2024-02-19T15:26:32.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.415+0000 7f34b4a00640 1 --2- 172.21.15.47:0/2712737016 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3490066e40 0x7f3490069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:32.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.419+0000 7f34b4a00640 1 --2- 172.21.15.47:0/2712737016 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3490066e40 0x7f3490069300 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f34b00cec00 tx=0x7f34a4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:32.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.419+0000 7f34acc00640 1 -- 172.21.15.47:0/2712737016 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f349c0ebcc0 con 0x7f34b015b1a0 2024-02-19T15:26:32.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.643+0000 7f34b7400640 1 -- 172.21.15.47:0/2712737016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f34b0149df0 con 0x7f34b015b1a0 2024-02-19T15:26:32.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.643+0000 7f34acc00640 1 -- 172.21.15.47:0/2712737016 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1528 (secure 0 0 0) 0x7f349c0b1610 con 0x7f34b015b1a0 2024-02-19T15:26:32.647 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:32.647 INFO:teuthology.orchestra.run.smithi047.stdout:{"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":292,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":93,"num_osds":8,"num_up_osds":8,"osd_up_since":1708356111,"num_in_osds":8,"osd_in_since":1708356102,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":55},{"state_name":"creating+peering","count":37},{"state_name":"active+clean","count":5}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":269803520,"bytes_avail":772824309760,"bytes_total":773094113280,"unknown_pgs_ratio":0.56701028347015381,"inactive_pgs_ratio":0.38144329190254211},"fsmap":{"epoch":58,"id":7,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":7,"rank":0,"name":"d","status":"up:active","gid":6850}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-02-19T15:25:47.200497+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-02-19T15:26:32.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.643+0000 7f34b7400640 1 -- 172.21.15.47:0/2712737016 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3490066e40 msgr2=0x7f3490069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:32.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.643+0000 7f34b7400640 1 --2- 172.21.15.47:0/2712737016 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3490066e40 0x7f3490069300 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f34b00cec00 tx=0x7f34a4009040 comp rx=0 tx=0).stop 2024-02-19T15:26:32.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.643+0000 7f34b7400640 1 -- 172.21.15.47:0/2712737016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34b015b1a0 msgr2=0x7f34b0148630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:32.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.643+0000 7f34b7400640 1 --2- 172.21.15.47:0/2712737016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34b015b1a0 0x7f34b0148630 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f349c0097f0 tx=0x7f349c019240 comp rx=0 tx=0).stop 2024-02-19T15:26:32.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.643+0000 7f34b7400640 1 -- 172.21.15.47:0/2712737016 shutdown_connections 2024-02-19T15:26:32.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.643+0000 7f34b7400640 1 --2- 172.21.15.47:0/2712737016 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3490066e40 0x7f3490069300 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.643+0000 7f34b7400640 1 --2- 172.21.15.47:0/2712737016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34b015d6e0 0x7f34b014dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.643+0000 7f34b7400640 1 --2- 172.21.15.47:0/2712737016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34b015b1a0 0x7f34b0148630 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.643+0000 7f34b7400640 1 --2- 172.21.15.47:0/2712737016 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34b0071350 0x7f34b0145b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.643+0000 7f34b7400640 1 -- 172.21.15.47:0/2712737016 >> 172.21.15.47:0/2712737016 conn(0x7f34b00b16a0 msgr2=0x7f34b00cf940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:32.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.643+0000 7f34b7400640 1 -- 172.21.15.47:0/2712737016 shutdown_connections 2024-02-19T15:26:32.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.643+0000 7f34b7400640 1 -- 172.21.15.47:0/2712737016 wait complete. 2024-02-19T15:26:32.668 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-02-19T15:26:32.668 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:26:32.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.803+0000 7fa834400640 1 Processor -- start 2024-02-19T15:26:32.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.803+0000 7fa834400640 1 -- start start 2024-02-19T15:26:32.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.803+0000 7fa834400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa82c0d26b0 0x7fa82c0d0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:32.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.803+0000 7fa834400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82c0cdae0 con 0x7fa82c0ceb70 2024-02-19T15:26:32.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.803+0000 7fa834400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82c0d0af0 con 0x7fa82c15be90 2024-02-19T15:26:32.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.803+0000 7fa834400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa82c0d0810 con 0x7fa82c0d26b0 2024-02-19T15:26:32.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.803+0000 7fa831a00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa82c15be90 0x7fa82c0ce460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:53266/0 (socket says 172.21.15.47:53266) 2024-02-19T15:26:32.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.803+0000 7fa831a00640 1 -- 172.21.15.47:0/4124819019 learned_addr learned my addr 172.21.15.47:0/4124819019 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:32.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.803+0000 7fa832400640 1 --2- 172.21.15.47:0/4124819019 >> v2:172.21.15.47:3300/0 conn(0x7fa82c0d26b0 0x7fa82c0d0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:32.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.803+0000 7fa82be00640 1 -- 172.21.15.47:0/4124819019 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145913801 0 0) 0x7fa82c0d0af0 con 0x7fa82c15be90 2024-02-19T15:26:32.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.803+0000 7fa832400640 1 -- 172.21.15.47:0/4124819019 >> v1:172.21.15.116:6789/0 conn(0x7fa82c15be90 legacy=0x7fa82c0ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:32.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.803+0000 7fa832400640 1 -- 172.21.15.47:0/4124819019 >> v1:172.21.15.47:6789/0 conn(0x7fa82c0ceb70 legacy=0x7fa82c0b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:32.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.803+0000 7fa832400640 1 -- 172.21.15.47:0/4124819019 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82c0b4a80 con 0x7fa82c0d26b0 2024-02-19T15:26:32.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.807+0000 7fa832400640 1 --2- 172.21.15.47:0/4124819019 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa82c0d26b0 0x7fa82c0d0170 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fa824009d90 tx=0x7fa82401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbdbba0fddbd5a39 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:32.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.807+0000 7fa82be00640 1 -- 172.21.15.47:0/4124819019 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8240a5a00 con 0x7fa82c0d26b0 2024-02-19T15:26:32.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.807+0000 7fa82be00640 1 -- 172.21.15.47:0/4124819019 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8240a5b60 con 0x7fa82c0d26b0 2024-02-19T15:26:32.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.807+0000 7fa82be00640 1 -- 172.21.15.47:0/4124819019 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8240ad440 con 0x7fa82c0d26b0 2024-02-19T15:26:32.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.807+0000 7fa834400640 1 -- 172.21.15.47:0/4124819019 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa82c0d26b0 msgr2=0x7fa82c0d0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:32.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.807+0000 7fa834400640 1 --2- 172.21.15.47:0/4124819019 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa82c0d26b0 0x7fa82c0d0170 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fa824009d90 tx=0x7fa82401c940 comp rx=0 tx=0).stop 2024-02-19T15:26:32.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.807+0000 7fa834400640 1 -- 172.21.15.47:0/4124819019 shutdown_connections 2024-02-19T15:26:32.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.807+0000 7fa834400640 1 --2- 172.21.15.47:0/4124819019 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa82c0d26b0 0x7fa82c0d0170 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.807+0000 7fa834400640 1 -- 172.21.15.47:0/4124819019 >> 172.21.15.47:0/4124819019 conn(0x7fa82c0d1b90 msgr2=0x7fa82c0d1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:32.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.807+0000 7fa834400640 1 -- 172.21.15.47:0/4124819019 shutdown_connections 2024-02-19T15:26:32.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.807+0000 7fa834400640 1 -- 172.21.15.47:0/4124819019 wait complete. 2024-02-19T15:26:32.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa834400640 1 Processor -- start 2024-02-19T15:26:32.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa834400640 1 -- start start 2024-02-19T15:26:32.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa834400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa82c0ceb70 0x7fa82c0b6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:32.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa834400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa82c0d26b0 0x7fa82c14f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:32.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa834400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa82c15be90 0x7fa82c14f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:32.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa834400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c1512f0 con 0x7fa82c0ceb70 2024-02-19T15:26:32.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa834400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c14a3d0 con 0x7fa82c0d26b0 2024-02-19T15:26:32.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa834400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c14fde0 con 0x7fa82c15be90 2024-02-19T15:26:32.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa831a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa82c0ceb70 0x7fa82c0b6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:32.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa831a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa82c0ceb70 0x7fa82c0b6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58494/0 (socket says 172.21.15.47:58494) 2024-02-19T15:26:32.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa831a00640 1 -- 172.21.15.47:0/1199280780 learned_addr learned my addr 172.21.15.47:0/1199280780 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:32.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa831000640 1 --2- 172.21.15.47:0/1199280780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa82c0d26b0 0x7fa82c14f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:32.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa832400640 1 --2- 172.21.15.47:0/1199280780 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa82c15be90 0x7fa82c14f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:32.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa831a00640 1 -- 172.21.15.47:0/1199280780 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa82c15be90 msgr2=0x7fa82c14f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:32.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa831a00640 1 --2- 172.21.15.47:0/1199280780 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa82c15be90 0x7fa82c14f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa831a00640 1 -- 172.21.15.47:0/1199280780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa82c0d26b0 msgr2=0x7fa82c14f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:32.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa831a00640 1 --2- 172.21.15.47:0/1199280780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa82c0d26b0 0x7fa82c14f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa831a00640 1 -- 172.21.15.47:0/1199280780 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa824009a70 con 0x7fa82c0ceb70 2024-02-19T15:26:32.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa831a00640 1 --2- 172.21.15.47:0/1199280780 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa82c0ceb70 0x7fa82c0b6c20 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fa82000c410 tx=0x7fa82000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:32.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa829600640 1 -- 172.21.15.47:0/1199280780 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8200074e0 con 0x7fa82c0ceb70 2024-02-19T15:26:32.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa829600640 1 -- 172.21.15.47:0/1199280780 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa820007640 con 0x7fa82c0ceb70 2024-02-19T15:26:32.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa829600640 1 -- 172.21.15.47:0/1199280780 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa820004a30 con 0x7fa82c0ceb70 2024-02-19T15:26:32.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa834400640 1 -- 172.21.15.47:0/1199280780 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa82c06d3f0 con 0x7fa82c0ceb70 2024-02-19T15:26:32.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa834400640 1 -- 172.21.15.47:0/1199280780 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa82c06d9b0 con 0x7fa82c0ceb70 2024-02-19T15:26:32.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.811+0000 7fa834400640 1 -- 172.21.15.47:0/1199280780 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa82c068500 con 0x7fa82c0ceb70 2024-02-19T15:26:32.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.815+0000 7fa829600640 1 -- 172.21.15.47:0/1199280780 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa820004b90 con 0x7fa82c0ceb70 2024-02-19T15:26:32.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.815+0000 7fa829600640 1 --2- 172.21.15.47:0/1199280780 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa80c066e40 0x7fa80c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:32.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.815+0000 7fa829600640 1 -- 172.21.15.47:0/1199280780 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8200846b0 con 0x7fa82c0ceb70 2024-02-19T15:26:32.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.815+0000 7fa831000640 1 --2- 172.21.15.47:0/1199280780 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa80c066e40 0x7fa80c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:32.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.815+0000 7fa831000640 1 --2- 172.21.15.47:0/1199280780 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa80c066e40 0x7fa80c069300 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa82c14cb50 tx=0x7fa818009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:32.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.819+0000 7fa829600640 1 -- 172.21.15.47:0/1199280780 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa820051550 con 0x7fa82c0ceb70 2024-02-19T15:26:32.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.979+0000 7fa834400640 1 -- 172.21.15.47:0/1199280780 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa82c070a40 con 0x7fa80c066e40 2024-02-19T15:26:32.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.987+0000 7fa829600640 1 -- 172.21.15.47:0/1199280780 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278233 (secure 0 0 0) 0x7fa82c070a40 con 0x7fa80c066e40 2024-02-19T15:26:32.992 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:32.996 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:26:32.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.987+0000 7fa834400640 1 -- 172.21.15.47:0/1199280780 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa80c066e40 msgr2=0x7fa80c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:32.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.987+0000 7fa834400640 1 --2- 172.21.15.47:0/1199280780 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa80c066e40 0x7fa80c069300 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa82c14cb50 tx=0x7fa818009040 comp rx=0 tx=0).stop 2024-02-19T15:26:32.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.987+0000 7fa834400640 1 -- 172.21.15.47:0/1199280780 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa82c0ceb70 msgr2=0x7fa82c0b6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:32.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.987+0000 7fa834400640 1 --2- 172.21.15.47:0/1199280780 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa82c0ceb70 0x7fa82c0b6c20 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fa82000c410 tx=0x7fa82000c8d0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.987+0000 7fa834400640 1 -- 172.21.15.47:0/1199280780 shutdown_connections 2024-02-19T15:26:32.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.987+0000 7fa834400640 1 --2- 172.21.15.47:0/1199280780 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa80c066e40 0x7fa80c069300 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.987+0000 7fa834400640 1 --2- 172.21.15.47:0/1199280780 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa82c15be90 0x7fa82c14f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.987+0000 7fa834400640 1 --2- 172.21.15.47:0/1199280780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa82c0d26b0 0x7fa82c14f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.987+0000 7fa834400640 1 --2- 172.21.15.47:0/1199280780 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa82c0ceb70 0x7fa82c0b6c20 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:32.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.987+0000 7fa834400640 1 -- 172.21.15.47:0/1199280780 >> 172.21.15.47:0/1199280780 conn(0x7fa82c0d1b90 msgr2=0x7fa82c0d1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:32.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.987+0000 7fa834400640 1 -- 172.21.15.47:0/1199280780 shutdown_connections 2024-02-19T15:26:32.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:32.987+0000 7fa834400640 1 -- 172.21.15.47:0/1199280780 wait complete. 2024-02-19T15:26:33.015 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-02-19T15:26:31.210849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":84,"acting":84,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":263480,"kb_used_data":9844,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754711240,"statfs":{"total":773094113280,"available":772824309760,"internally_reserved":0,"allocated":10080256,"data_stored":4940203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":259575346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.902450"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.318242+0000","last_change":"2024-02-19T15:26:26.318242+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.318242+0000","last_undegraded":"2024-02-19T15:26:26.318242+0000","last_fullsized":"2024-02-19T15:26:26.318242+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.319202+0000","last_change":"2024-02-19T15:26:26.319202+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.319202+0000","last_undegraded":"2024-02-19T15:26:26.319202+0000","last_fullsized":"2024-02-19T15:26:26.319202+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.319262+0000","last_change":"2024-02-19T15:26:26.319262+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.319262+0000","last_undegraded":"2024-02-19T15:26:26.319262+0000","last_fullsized":"2024-02-19T15:26:26.319262+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.320398+0000","last_change":"2024-02-19T15:26:26.320398+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.320398+0000","last_undegraded":"2024-02-19T15:26:26.320398+0000","last_fullsized":"2024-02-19T15:26:26.320398+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315361+0000","last_change":"2024-02-19T15:26:25.315361+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315361+0000","last_undegraded":"2024-02-19T15:26:25.315361+0000","last_fullsized":"2024-02-19T15:26:25.315361+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315188+0000","last_change":"2024-02-19T15:26:25.315188+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315188+0000","last_undegraded":"2024-02-19T15:26:25.315188+0000","last_fullsized":"2024-02-19T15:26:25.315188+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315899+0000","last_change":"2024-02-19T15:26:25.315899+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315899+0000","last_undegraded":"2024-02-19T15:26:25.315899+0000","last_fullsized":"2024-02-19T15:26:25.315899+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.316019+0000","last_change":"2024-02-19T15:26:25.316019+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.316019+0000","last_undegraded":"2024-02-19T15:26:25.316019+0000","last_fullsized":"2024-02-19T15:26:25.316019+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.310338+0000","last_change":"2024-02-19T15:26:25.310338+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.310338+0000","last_undegraded":"2024-02-19T15:26:25.310338+0000","last_fullsized":"2024-02-19T15:26:25.310338+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.316341+0000","last_change":"2024-02-19T15:26:25.316341+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.316341+0000","last_undegraded":"2024-02-19T15:26:25.316341+0000","last_fullsized":"2024-02-19T15:26:25.316341+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-19T15:26:26.338255+0000","last_change":"2024-02-19T15:26:26.336796+0000","last_active":"2024-02-19T15:26:26.338255+0000","last_peered":"2024-02-19T15:26:26.338255+0000","last_clean":"2024-02-19T15:26:26.338255+0000","last_became_active":"2024-02-19T15:26:26.336376+0000","last_became_peered":"2024-02-19T15:26:26.336376+0000","last_unstale":"2024-02-19T15:26:26.338255+0000","last_undegraded":"2024-02-19T15:26:26.338255+0000","last_fullsized":"2024-02-19T15:26:26.338255+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:27:28.873521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.311184+0000","last_change":"2024-02-19T15:26:25.311184+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.311184+0000","last_undegraded":"2024-02-19T15:26:25.311184+0000","last_fullsized":"2024-02-19T15:26:25.311184+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.310305+0000","last_change":"2024-02-19T15:26:25.310305+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.310305+0000","last_undegraded":"2024-02-19T15:26:25.310305+0000","last_fullsized":"2024-02-19T15:26:25.310305+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.313986+0000","last_change":"2024-02-19T15:26:25.313986+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.313986+0000","last_undegraded":"2024-02-19T15:26:25.313986+0000","last_fullsized":"2024-02-19T15:26:25.313986+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.319592+0000","last_change":"2024-02-19T15:26:26.319592+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.319592+0000","last_undegraded":"2024-02-19T15:26:26.319592+0000","last_fullsized":"2024-02-19T15:26:26.319592+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.314616+0000","last_change":"2024-02-19T15:26:25.314616+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.314616+0000","last_undegraded":"2024-02-19T15:26:25.314616+0000","last_fullsized":"2024-02-19T15:26:25.314616+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315717+0000","last_change":"2024-02-19T15:26:25.315717+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315717+0000","last_undegraded":"2024-02-19T15:26:25.315717+0000","last_fullsized":"2024-02-19T15:26:25.315717+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.312770+0000","last_change":"2024-02-19T15:26:25.312770+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.312770+0000","last_undegraded":"2024-02-19T15:26:25.312770+0000","last_fullsized":"2024-02-19T15:26:25.312770+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.312441+0000","last_change":"2024-02-19T15:26:25.312441+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.312441+0000","last_undegraded":"2024-02-19T15:26:25.312441+0000","last_fullsized":"2024-02-19T15:26:25.312441+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315589+0000","last_change":"2024-02-19T15:26:25.315589+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315589+0000","last_undegraded":"2024-02-19T15:26:25.315589+0000","last_fullsized":"2024-02-19T15:26:25.315589+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.310289+0000","last_change":"2024-02-19T15:26:25.310289+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.310289+0000","last_undegraded":"2024-02-19T15:26:25.310289+0000","last_fullsized":"2024-02-19T15:26:25.310289+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.318671+0000","last_change":"2024-02-19T15:26:26.318671+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.318671+0000","last_undegraded":"2024-02-19T15:26:26.318671+0000","last_fullsized":"2024-02-19T15:26:26.318671+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315968+0000","last_change":"2024-02-19T15:26:25.315968+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315968+0000","last_undegraded":"2024-02-19T15:26:25.315968+0000","last_fullsized":"2024-02-19T15:26:25.315968+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-19T15:26:26.331522+0000","last_change":"2024-02-19T15:26:26.328022+0000","last_active":"2024-02-19T15:26:26.331522+0000","last_peered":"2024-02-19T15:26:26.331522+0000","last_clean":"2024-02-19T15:26:26.331522+0000","last_became_active":"2024-02-19T15:26:26.327714+0000","last_became_peered":"2024-02-19T15:26:26.327714+0000","last_unstale":"2024-02-19T15:26:26.331522+0000","last_undegraded":"2024-02-19T15:26:26.331522+0000","last_fullsized":"2024-02-19T15:26:26.331522+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:09:12.432031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315314+0000","last_change":"2024-02-19T15:26:25.315314+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315314+0000","last_undegraded":"2024-02-19T15:26:25.315314+0000","last_fullsized":"2024-02-19T15:26:25.315314+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.316123+0000","last_change":"2024-02-19T15:26:25.316123+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.316123+0000","last_undegraded":"2024-02-19T15:26:25.316123+0000","last_fullsized":"2024-02-19T15:26:25.316123+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.313762+0000","last_change":"2024-02-19T15:26:25.313762+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.313762+0000","last_undegraded":"2024-02-19T15:26:25.313762+0000","last_fullsized":"2024-02-19T15:26:25.313762+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.313774+0000","last_change":"2024-02-19T15:26:25.313774+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.313774+0000","last_undegraded":"2024-02-19T15:26:25.313774+0000","last_fullsized":"2024-02-19T15:26:25.313774+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.319175+0000","last_change":"2024-02-19T15:26:26.319175+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.319175+0000","last_undegraded":"2024-02-19T15:26:26.319175+0000","last_fullsized":"2024-02-19T15:26:26.319175+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315822+0000","last_change":"2024-02-19T15:26:25.315822+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315822+0000","last_undegraded":"2024-02-19T15:26:25.315822+0000","last_fullsized":"2024-02-19T15:26:25.315822+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.313928+0000","last_change":"2024-02-19T15:26:25.313928+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.313928+0000","last_undegraded":"2024-02-19T15:26:25.313928+0000","last_fullsized":"2024-02-19T15:26:25.313928+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315310+0000","last_change":"2024-02-19T15:26:25.315310+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315310+0000","last_undegraded":"2024-02-19T15:26:25.315310+0000","last_fullsized":"2024-02-19T15:26:25.315310+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315507+0000","last_change":"2024-02-19T15:26:25.315507+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315507+0000","last_undegraded":"2024-02-19T15:26:25.315507+0000","last_fullsized":"2024-02-19T15:26:25.315507+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.318591+0000","last_change":"2024-02-19T15:26:26.318591+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.318591+0000","last_undegraded":"2024-02-19T15:26:26.318591+0000","last_fullsized":"2024-02-19T15:26:26.318591+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-19T15:26:26.339710+0000","last_change":"2024-02-19T15:26:26.338468+0000","last_active":"2024-02-19T15:26:26.339710+0000","last_peered":"2024-02-19T15:26:26.339710+0000","last_clean":"2024-02-19T15:26:26.339710+0000","last_became_active":"2024-02-19T15:26:26.338119+0000","last_became_peered":"2024-02-19T15:26:26.338119+0000","last_unstale":"2024-02-19T15:26:26.339710+0000","last_undegraded":"2024-02-19T15:26:26.339710+0000","last_fullsized":"2024-02-19T15:26:26.339710+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:46:46.101170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.315428+0000","last_change":"2024-02-19T15:26:25.315428+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.315428+0000","last_undegraded":"2024-02-19T15:26:25.315428+0000","last_fullsized":"2024-02-19T15:26:25.315428+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-19T15:26:26.331603+0000","last_change":"2024-02-19T15:26:26.329556+0000","last_active":"2024-02-19T15:26:26.331603+0000","last_peered":"2024-02-19T15:26:26.331603+0000","last_clean":"2024-02-19T15:26:26.331603+0000","last_became_active":"2024-02-19T15:26:26.329110+0000","last_became_peered":"2024-02-19T15:26:26.329110+0000","last_unstale":"2024-02-19T15:26:26.331603+0000","last_undegraded":"2024-02-19T15:26:26.331603+0000","last_fullsized":"2024-02-19T15:26:26.331603+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:16:30.549123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.313959+0000","last_change":"2024-02-19T15:26:25.313959+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.313959+0000","last_undegraded":"2024-02-19T15:26:25.313959+0000","last_fullsized":"2024-02-19T15:26:25.313959+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-02-19T15:26:26.318242+0000","last_change":"2024-02-19T15:26:26.318242+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.318242+0000","last_undegraded":"2024-02-19T15:26:26.318242+0000","last_fullsized":"2024-02-19T15:26:26.318242+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.312292+0000","last_change":"2024-02-19T15:26:25.312292+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.312292+0000","last_undegraded":"2024-02-19T15:26:25.312292+0000","last_fullsized":"2024-02-19T15:26:25.312292+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:26:26.307873+0000","last_change":"2024-02-19T15:26:26.307873+0000","last_active":"2024-02-19T15:26:26.307873+0000","last_peered":"2024-02-19T15:26:26.307873+0000","last_clean":"2024-02-19T15:26:26.307873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:26.307873+0000","last_undegraded":"2024-02-19T15:26:26.307873+0000","last_fullsized":"2024-02-19T15:26:26.307873+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-02-19T15:26:25.314635+0000","last_change":"2024-02-19T15:26:25.314635+0000","last_active":"2024-02-19T15:26:25.301961+0000","last_peered":"2024-02-19T15:26:25.301961+0000","last_clean":"2024-02-19T15:26:25.301961+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:26:25.314635+0000","last_undegraded":"2024-02-19T15:26:25.314635+0000","last_fullsized":"2024-02-19T15:26:25.314635+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":292,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-02-19T15:26:25.315555+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:26:25.315555+0000","last_peered":"2024-02-19T15:26:25.315555+0000","last_clean":"2024-02-19T15:26:25.315555+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:26:25.315555+0000","last_undegraded":"2024-02-19T15:26:25.315555+0000","last_fullsized":"2024-02-19T15:26:25.315555+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":1},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509498,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1122304,"data_stored":475424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.61399999999999999,"15min":0.61699999999999999},"min":{"1min":0.42899999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83599999999999997,"5min":1.022,"15min":1.022},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.68000000000000005,"15min":0.752},"min":{"1min":0.433,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.96699999999999997,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.64700000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.64500000000000002,"15min":0.65200000000000002},"min":{"1min":0.49299999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.81999999999999995,"5min":1.171,"15min":1.171},"last":0.5},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65600000000000003,"15min":0.65200000000000002},"min":{"1min":0.40300000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.51900000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.437,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.175,"5min":1.216,"15min":1.216},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.70599999999999996,"15min":0.69099999999999995},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.95099999999999996,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.54300000000000004}]},{"osd":3,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69499999999999995,"15min":0.68100000000000005},"min":{"1min":0.46899999999999997,"5min":0.438,"15min":0.438},"max":{"1min":0.90900000000000003,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67600000000000005,"15min":0.66200000000000003},"min":{"1min":0.48399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.87,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.72899999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.64900000000000002,"15min":0.61499999999999999},"min":{"1min":0.40400000000000003,"5min":0.309,"15min":0.309},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.752,"5min":0.66200000000000003,"15min":0.61499999999999999},"min":{"1min":0.41699999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.71099999999999997}]},{"osd":5,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.65700000000000003,"15min":0.61299999999999999},"min":{"1min":0.439,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.753},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.72199999999999998,"15min":0.68200000000000005},"min":{"1min":0.56899999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.73999999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.72199999999999998,"15min":0.66800000000000004},"min":{"1min":0.48899999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.74299999999999999,"15min":0.68899999999999995},"min":{"1min":0.67000000000000004,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.80800000000000005}]}]},{"osd":7,"up_from":14,"seq":60129542202,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1059517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.59699999999999998,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.87,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.67400000000000004},"min":{"1min":0.38800000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.77100000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.63500000000000001,"15min":0.60099999999999998},"min":{"1min":0.498,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86799999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.70399999999999996,"15min":0.72099999999999997},"min":{"1min":0.496,"5min":0.437,"15min":0.437},"max":{"1min":1.008,"5min":2.968,"15min":2.968},"last":0.73299999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.64700000000000002,"15min":0.63600000000000001},"min":{"1min":0.47599999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85399999999999998,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.497,"5min":0.438,"15min":0.438},"max":{"1min":0.97399999999999998,"5min":2.508,"15min":2.508},"last":0.68799999999999994}]},{"osd":3,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66300000000000003,"15min":0.65900000000000003},"min":{"1min":0.50900000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.91300000000000003,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.752},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.70999999999999996,"15min":0.69499999999999995},"min":{"1min":0.59399999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.94699999999999995,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":0.85699999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.59499999999999997,"15min":0.53900000000000003},"min":{"1min":0.432,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.61799999999999999,"15min":0.58199999999999996},"min":{"1min":0.40100000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93600000000000005,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.57199999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.65700000000000003,"15min":0.58399999999999996},"min":{"1min":0.53500000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.63800000000000001},"min":{"1min":0.54700000000000004,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.97299999999999998,"5min":1.046,"15min":1.046},"last":0.70899999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.67100000000000004,"15min":0.61299999999999999},"min":{"1min":0.46999999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.62},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72599999999999998,"15min":0.67500000000000004},"min":{"1min":0.59699999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.63700000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542203,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32268,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339572,"statfs":{"total":96636764160,"available":96603721728,"internally_reserved":0,"allocated":1110016,"data_stored":469149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.67200000000000004,"15min":0.64400000000000002},"min":{"1min":0.38300000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.97699999999999998,"5min":5.7789999999999999,"15min":5.7789999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.71099999999999997,"15min":0.63700000000000001},"min":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.86499999999999999,"5min":12.359999999999999,"15min":12.359999999999999},"last":0.53600000000000003}]},{"osd":1,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.73799999999999999,"15min":0.67000000000000004},"min":{"1min":0.432,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80000000000000004,"5min":10.938000000000001,"15min":10.938000000000001},"last":0.502},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.78900000000000003,"15min":0.71299999999999997},"min":{"1min":0.50600000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85599999999999998,"5min":10.531000000000001,"15min":10.531000000000001},"last":0.56499999999999995}]},{"osd":2,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.79700000000000004,"15min":0.72099999999999997},"min":{"1min":0.42999999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.89900000000000002,"5min":12.925000000000001,"15min":12.925000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.72999999999999998,"15min":0.64100000000000001},"min":{"1min":0.48899999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89000000000000001,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.46999999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.91100000000000003,"15min":0.73499999999999999},"min":{"1min":0.49399999999999999,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82299999999999995,"5min":26.161999999999999,"15min":26.161999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.91000000000000003,"15min":0.73999999999999999},"min":{"1min":0.47299999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1120000000000001,"5min":27.934999999999999,"15min":27.934999999999999},"last":0.52700000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.67500000000000004,"15min":0.59199999999999997},"min":{"1min":0.45700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83099999999999996,"5min":9.0549999999999997,"15min":9.0549999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.70899999999999996,"15min":0.67100000000000004},"min":{"1min":0.44600000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.92800000000000005,"5min":6.2549999999999999,"15min":6.2549999999999999},"last":0.33900000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.746,"15min":0.67000000000000004},"min":{"1min":0.50700000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.6539999999999999,"5min":6.5229999999999997,"15min":6.5229999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81899999999999995,"15min":0.749},"min":{"1min":0.63500000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.2310000000000001,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":0.48299999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.77600000000000002,"15min":0.69399999999999995},"min":{"1min":0.41499999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.95399999999999996,"5min":11.986000000000001,"15min":11.986000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.77700000000000002,"15min":0.71499999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.89100000000000001,"5min":8.9770000000000003,"15min":8.9770000000000003},"last":0.45200000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32268,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339572,"statfs":{"total":96636764160,"available":96603721728,"internally_reserved":0,"allocated":1110016,"data_stored":469149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.378,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.623,"15min":0.60499999999999998},"min":{"1min":0.47099999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85499999999999998,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.61199999999999999}]},{"osd":1,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64100000000000001,"15min":0.61899999999999999},"min":{"1min":0.48999999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.879,"5min":1.081,"15min":1.081},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64400000000000002,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83699999999999997,"5min":1.008,"15min":1.008},"last":0.56999999999999995}]},{"osd":2,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.65400000000000003,"15min":0.66000000000000003},"min":{"1min":0.49299999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.90700000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68200000000000005,"15min":0.67600000000000005},"min":{"1min":0.45100000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92000000000000004,"5min":1.002,"15min":1.002},"last":0.55400000000000005}]},{"osd":3,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.70799999999999996,"15min":0.68200000000000005},"min":{"1min":0.50600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94899999999999995,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.65900000000000003,"15min":0.65300000000000002},"min":{"1min":0.42799999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89600000000000002,"5min":1.153,"15min":1.153},"last":0.69099999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63600000000000001,"15min":0.60599999999999998},"min":{"1min":0.433,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85399999999999998,"5min":1.042,"15min":1.042},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70599999999999996,"15min":0.68200000000000005},"min":{"1min":0.54200000000000004,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.94799999999999995,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.749}]},{"osd":6,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.622,"15min":0.58299999999999996},"min":{"1min":0.376,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.92200000000000004,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68600000000000005,"15min":0.65500000000000003},"min":{"1min":0.52900000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90800000000000003,"5min":1.054,"15min":1.054},"last":0.63600000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.70199999999999996,"15min":0.67100000000000004},"min":{"1min":0.44800000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.98199999999999998,"5min":1.115,"15min":1.115},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.75600000000000001,"15min":0.73399999999999999},"min":{"1min":0.46999999999999997,"5min":0.443,"15min":0.443},"max":{"1min":1.0169999999999999,"5min":1.24,"15min":1.24},"last":0.78700000000000003}]}]},{"osd":3,"up_from":14,"seq":60129542201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32276,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339564,"statfs":{"total":96636764160,"available":96603713536,"internally_reserved":0,"allocated":1110016,"data_stored":469149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42899999999999999,"15min":0.44700000000000001},"min":{"1min":0.23699999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76600000000000001,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.54100000000000004,"15min":0.64600000000000002},"min":{"1min":0.27500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67600000000000005,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.53600000000000003}]},{"osd":1,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.433,"15min":0.442},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.66000000000000003,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.59699999999999998,"15min":0.70899999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.66400000000000003,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.67700000000000005}]},{"osd":2,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56799999999999995,"15min":0.61199999999999999},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.66500000000000004,"5min":1.008,"15min":1.008},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55000000000000004,"15min":0.58899999999999997},"min":{"1min":0.39900000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86199999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.82799999999999996}]},{"osd":4,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.70299999999999996,"15min":0.67900000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.90600000000000003,"5min":1.004,"15min":1.004},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.80100000000000005,"15min":0.86099999999999999},"min":{"1min":0.58199999999999996,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0269999999999999,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.63100000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.80100000000000005,"15min":0.85499999999999998},"min":{"1min":0.54700000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.96699999999999997,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.74199999999999999,"15min":0.748},"min":{"1min":0.53300000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.93899999999999995,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.86899999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:26:06 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.82499999999999996,"15min":0.89000000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92300000000000004,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.78500000000000003,"15min":0.81299999999999994},"min":{"1min":0.55400000000000005,"5min":0.502,"15min":0.502},"max":{"1min":1.0009999999999999,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.91700000000000004}]},{"osd":7,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76400000000000001,"15min":0.78000000000000003},"min":{"1min":0.52700000000000002,"5min":0.504,"15min":0.504},"max":{"1min":0.96799999999999997,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.84099999999999997,"15min":0.92000000000000004},"min":{"1min":0.57399999999999995,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0009999999999999,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.80100000000000005}]}]},{"osd":2,"up_from":14,"seq":60129542202,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1110016,"data_stored":469149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.66700000000000004,"15min":0.69699999999999995},"min":{"1min":0.27400000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.64400000000000002,"5min":15.42,"15min":15.42},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.71499999999999997,"15min":0.88500000000000001},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.76500000000000001,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.38100000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.75700000000000001,"15min":0.89700000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58999999999999997,"5min":9.5269999999999992,"15min":9.5269999999999992},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.94199999999999995,"15min":1.006},"min":{"1min":0.39700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.621,"5min":22.954999999999998,"15min":22.954999999999998},"last":0.41499999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.67100000000000004,"15min":0.78300000000000003},"min":{"1min":0.376,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66200000000000003,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.74299999999999999,"15min":0.82299999999999995},"min":{"1min":0.41799999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69399999999999995,"5min":7.734,"15min":7.734},"last":0.48499999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.91400000000000003,"15min":0.90300000000000002},"min":{"1min":0.52000000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.016,"5min":15.369,"15min":15.369},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.82699999999999996,"15min":0.89300000000000002},"min":{"1min":0.53800000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93700000000000006,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.59199999999999997}]},{"osd":5,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.871,"15min":0.88900000000000001},"min":{"1min":0.53900000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.95999999999999996,"5min":7.6520000000000001,"15min":7.6520000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.98699999999999999,"15min":0.97499999999999998},"min":{"1min":0.60199999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0660000000000001,"5min":15.391,"15min":15.391},"last":0.63300000000000001}]},{"osd":6,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.95799999999999996,"15min":0.94399999999999995},"min":{"1min":0.54900000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.90100000000000002,"5min":15.337,"15min":15.337},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.876,"15min":0.95799999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.98499999999999999,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.68899999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.90600000000000003,"15min":0.98099999999999998},"min":{"1min":0.58399999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.028,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.91600000000000004,"15min":0.95299999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.98099999999999998,"5min":7.6929999999999996,"15min":7.6929999999999996},"last":0.54800000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542203,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1110016,"data_stored":469149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.39200000000000002,"15min":0.42399999999999999},"min":{"1min":0.248,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56799999999999995,"5min":0.747,"15min":0.747},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.438,"15min":0.46600000000000003},"min":{"1min":0.28100000000000003,"5min":0.25,"15min":0.25},"max":{"1min":0.69599999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.49399999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.42599999999999999,"15min":0.44400000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.56100000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.48799999999999999,"15min":0.52600000000000002},"min":{"1min":0.36299999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.501,"5min":1.071,"15min":1.071},"last":0.379}]},{"osd":3,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58899999999999997,"15min":0.63300000000000001},"min":{"1min":0.44800000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.64600000000000002,"5min":1.365,"15min":1.365},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.57399999999999995,"15min":0.63},"min":{"1min":0.42999999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.61399999999999999,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.47099999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.65300000000000002,"15min":0.61299999999999999},"min":{"1min":0.52100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.99099999999999999,"5min":1.127,"15min":1.127},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.69299999999999995,"15min":0.68000000000000005},"min":{"1min":0.54900000000000004,"5min":0.435,"15min":0.435},"max":{"1min":0.98399999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.70299999999999996}]},{"osd":5,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70899999999999996,"15min":0.69699999999999995},"min":{"1min":0.54200000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.95599999999999996,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.70399999999999996,"15min":0.68500000000000005},"min":{"1min":0.503,"5min":0.443,"15min":0.443},"max":{"1min":0.94199999999999995,"5min":1.107,"15min":1.107},"last":0.80500000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.72899999999999998},"min":{"1min":0.57099999999999995,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.97399999999999998,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.74199999999999999,"15min":0.745},"min":{"1min":0.60299999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97199999999999998,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.73999999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.76800000000000002,"15min":0.78600000000000003},"min":{"1min":0.59999999999999998,"5min":0.496,"15min":0.496},"max":{"1min":0.95299999999999996,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.76400000000000001,"15min":0.748},"min":{"1min":0.58499999999999996,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0229999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.82799999999999996}]}]},{"osd":0,"up_from":14,"seq":60129542201,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37008,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334832,"statfs":{"total":96636764160,"available":96598867968,"internally_reserved":0,"allocated":1703936,"data_stored":1059517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.50700000000000001,"15min":0.628},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.66200000000000003,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.31},{"interface":"front","average":{"1min":0.443,"5min":0.46300000000000002,"15min":0.48299999999999998},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.76000000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.46899999999999997}]},{"osd":2,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.65400000000000003,"15min":0.78400000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.67500000000000004,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.61799999999999999,"15min":0.746},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69099999999999995,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.36199999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.56699999999999995,"15min":0.67900000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.61699999999999999,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.436},{"interface":"front","average":{"1min":0.495,"5min":0.54800000000000004,"15min":0.60999999999999999},"min":{"1min":0.38100000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.72499999999999998,"5min":1.345,"15min":1.345},"last":0.41499999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.745,"15min":0.74299999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.0700000000000001,"5min":1.095,"15min":1.095},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.84399999999999997,"15min":0.92500000000000004},"min":{"1min":0.64000000000000001,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0049999999999999,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.73299999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.73799999999999999,"15min":0.751},"min":{"1min":0.55900000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.97999999999999998,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.77300000000000002,"15min":0.85299999999999998},"min":{"1min":0.52900000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.05,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.55700000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.82099999999999995,"15min":0.91500000000000004},"min":{"1min":0.59799999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.95899999999999996,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73799999999999999,"15min":0.75700000000000001},"min":{"1min":0.57999999999999996,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.84999999999999998,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.57799999999999996}]},{"osd":7,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.85199999999999998,"15min":0.94199999999999995},"min":{"1min":0.58599999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.0229999999999999,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.83499999999999996,"15min":0.94099999999999995},"min":{"1min":0.55900000000000005,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.96499999999999997,"5min":4.915,"15min":4.915},"last":0.69499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:26:35.642 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:26:35.646 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:26:36.020 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:26:36.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.167+0000 7fdc83800640 1 Processor -- start 2024-02-19T15:26:36.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.167+0000 7fdc83800640 1 -- start start 2024-02-19T15:26:36.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.167+0000 7fdc83800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdc7c145590 0x7fdc7c0727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:36.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.167+0000 7fdc83800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdc7c0cea80 0x7fdc7c06c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:36.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.167+0000 7fdc83800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc7c15c390 0x7fdc7c15c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:36.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.167+0000 7fdc83800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7c15d280 con 0x7fdc7c145590 2024-02-19T15:26:36.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.167+0000 7fdc83800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7c070ff0 con 0x7fdc7c0cea80 2024-02-19T15:26:36.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.167+0000 7fdc83800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c15cd60 con 0x7fdc7c15c390 2024-02-19T15:26:36.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.167+0000 7fdc80e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdc7c145590 0x7fdc7c0727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:36.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.167+0000 7fdc80e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdc7c145590 0x7fdc7c0727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40648/0 (socket says 172.21.15.47:40648) 2024-02-19T15:26:36.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.167+0000 7fdc80e00640 1 -- 172.21.15.47:0/46620821 learned_addr learned my addr 172.21.15.47:0/46620821 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:36.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.167+0000 7fdc7be00640 1 --2- 172.21.15.47:0/46620821 >> v2:172.21.15.116:3300/0 conn(0x7fdc7c0cea80 0x7fdc7c06c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:36.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.167+0000 7fdc81800640 1 --2- 172.21.15.47:0/46620821 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc7c15c390 0x7fdc7c15c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:36.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.167+0000 7fdc80e00640 1 -- 172.21.15.47:0/46620821 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc7c15c390 msgr2=0x7fdc7c15c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:36.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.167+0000 7fdc80e00640 1 --2- 172.21.15.47:0/46620821 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc7c15c390 0x7fdc7c15c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.167+0000 7fdc80e00640 1 -- 172.21.15.47:0/46620821 >> v2:172.21.15.116:3300/0 conn(0x7fdc7c0cea80 msgr2=0x7fdc7c06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:36.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.167+0000 7fdc80e00640 1 --2- 172.21.15.47:0/46620821 >> v2:172.21.15.116:3300/0 conn(0x7fdc7c0cea80 0x7fdc7c06c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.167+0000 7fdc80e00640 1 -- 172.21.15.47:0/46620821 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c0709e0 con 0x7fdc7c145590 2024-02-19T15:26:36.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.171+0000 7fdc80e00640 1 --2- 172.21.15.47:0/46620821 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc7c145590 0x7fdc7c0727b0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c06c770 tx=0x7fdc7001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=562a35f081cf0800 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:36.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.171+0000 7fdc7b400640 1 -- 172.21.15.47:0/46620821 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc700ac630 con 0x7fdc7c145590 2024-02-19T15:26:36.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.171+0000 7fdc7b400640 1 -- 172.21.15.47:0/46620821 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc700ac790 con 0x7fdc7c145590 2024-02-19T15:26:36.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.171+0000 7fdc7b400640 1 -- 172.21.15.47:0/46620821 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc700acb20 con 0x7fdc7c145590 2024-02-19T15:26:36.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.171+0000 7fdc83800640 1 -- 172.21.15.47:0/46620821 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc7c145590 msgr2=0x7fdc7c0727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:36.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.171+0000 7fdc83800640 1 --2- 172.21.15.47:0/46620821 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc7c145590 0x7fdc7c0727b0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c06c770 tx=0x7fdc7001c520 comp rx=0 tx=0).stop 2024-02-19T15:26:36.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.171+0000 7fdc83800640 1 -- 172.21.15.47:0/46620821 shutdown_connections 2024-02-19T15:26:36.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.171+0000 7fdc83800640 1 --2- 172.21.15.47:0/46620821 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc7c15c390 0x7fdc7c15c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.171+0000 7fdc83800640 1 --2- 172.21.15.47:0/46620821 >> v2:172.21.15.116:3300/0 conn(0x7fdc7c0cea80 0x7fdc7c06c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.171+0000 7fdc83800640 1 --2- 172.21.15.47:0/46620821 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc7c145590 0x7fdc7c0727b0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.171+0000 7fdc83800640 1 -- 172.21.15.47:0/46620821 >> 172.21.15.47:0/46620821 conn(0x7fdc7c0b16a0 msgr2=0x7fdc7c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:36.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.171+0000 7fdc83800640 1 -- 172.21.15.47:0/46620821 shutdown_connections 2024-02-19T15:26:36.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.171+0000 7fdc83800640 1 -- 172.21.15.47:0/46620821 wait complete. 2024-02-19T15:26:36.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc83800640 1 Processor -- start 2024-02-19T15:26:36.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc83800640 1 -- start start 2024-02-19T15:26:36.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc83800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc7c0cea80 0x7fdc7c13cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:36.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc83800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc7c145590 0x7fdc7c15a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:36.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc83800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc7c15c390 0x7fdc7c134ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:36.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc83800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c13d1e0 con 0x7fdc7c15c390 2024-02-19T15:26:36.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc83800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c13bc00 con 0x7fdc7c0cea80 2024-02-19T15:26:36.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc83800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c13bee0 con 0x7fdc7c145590 2024-02-19T15:26:36.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc81800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc7c15c390 0x7fdc7c134ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:36.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc80e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc7c0cea80 0x7fdc7c13cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:36.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc81800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc7c15c390 0x7fdc7c134ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40650/0 (socket says 172.21.15.47:40650) 2024-02-19T15:26:36.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc81800640 1 -- 172.21.15.47:0/1491803983 learned_addr learned my addr 172.21.15.47:0/1491803983 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:36.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc7be00640 1 --2- 172.21.15.47:0/1491803983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc7c145590 0x7fdc7c15a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:36.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc81800640 1 -- 172.21.15.47:0/1491803983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc7c145590 msgr2=0x7fdc7c15a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:36.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc81800640 1 --2- 172.21.15.47:0/1491803983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc7c145590 0x7fdc7c15a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc81800640 1 -- 172.21.15.47:0/1491803983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc7c0cea80 msgr2=0x7fdc7c13cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:36.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc81800640 1 --2- 172.21.15.47:0/1491803983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc7c0cea80 0x7fdc7c13cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc81800640 1 -- 172.21.15.47:0/1491803983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc700094d0 con 0x7fdc7c15c390 2024-02-19T15:26:36.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc80e00640 1 --2- 172.21.15.47:0/1491803983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc7c0cea80 0x7fdc7c13cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:36.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc81800640 1 --2- 172.21.15.47:0/1491803983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc7c15c390 0x7fdc7c134ea0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fdc7400efc0 tx=0x7fdc7400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:36.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc78c00640 1 -- 172.21.15.47:0/1491803983 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc74009ba0 con 0x7fdc7c15c390 2024-02-19T15:26:36.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc78c00640 1 -- 172.21.15.47:0/1491803983 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7400f040 con 0x7fdc7c15c390 2024-02-19T15:26:36.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc78c00640 1 -- 172.21.15.47:0/1491803983 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc740146c0 con 0x7fdc7c15c390 2024-02-19T15:26:36.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc83800640 1 -- 172.21.15.47:0/1491803983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7c13c1c0 con 0x7fdc7c15c390 2024-02-19T15:26:36.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.175+0000 7fdc83800640 1 -- 172.21.15.47:0/1491803983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc7c13c780 con 0x7fdc7c15c390 2024-02-19T15:26:36.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.179+0000 7fdc83800640 1 -- 172.21.15.47:0/1491803983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc7c13c9a0 con 0x7fdc7c15c390 2024-02-19T15:26:36.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.179+0000 7fdc78c00640 1 -- 172.21.15.47:0/1491803983 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fdc74014820 con 0x7fdc7c15c390 2024-02-19T15:26:36.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.179+0000 7fdc78c00640 1 --2- 172.21.15.47:0/1491803983 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdc5c066e40 0x7fdc5c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:36.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.179+0000 7fdc78c00640 1 -- 172.21.15.47:0/1491803983 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc74085540 con 0x7fdc7c15c390 2024-02-19T15:26:36.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.179+0000 7fdc80e00640 1 --2- 172.21.15.47:0/1491803983 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdc5c066e40 0x7fdc5c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:36.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.179+0000 7fdc80e00640 1 --2- 172.21.15.47:0/1491803983 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdc5c066e40 0x7fdc5c069300 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fdc700ae000 tx=0x7fdc70008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:36.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.183+0000 7fdc78c00640 1 -- 172.21.15.47:0/1491803983 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fdc74052340 con 0x7fdc7c15c390 2024-02-19T15:26:36.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.343+0000 7fdc83800640 1 -- 172.21.15.47:0/1491803983 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdc7c06e6d0 con 0x7fdc5c066e40 2024-02-19T15:26:36.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.355+0000 7fdc78c00640 1 -- 172.21.15.47:0/1491803983 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289229 (secure 0 0 0) 0x7fdc7c06e6d0 con 0x7fdc5c066e40 2024-02-19T15:26:36.360 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:36.362 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:26:36.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.355+0000 7fdc83800640 1 -- 172.21.15.47:0/1491803983 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdc5c066e40 msgr2=0x7fdc5c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:36.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.355+0000 7fdc83800640 1 --2- 172.21.15.47:0/1491803983 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdc5c066e40 0x7fdc5c069300 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fdc700ae000 tx=0x7fdc70008040 comp rx=0 tx=0).stop 2024-02-19T15:26:36.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.355+0000 7fdc83800640 1 -- 172.21.15.47:0/1491803983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc7c15c390 msgr2=0x7fdc7c134ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:36.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.355+0000 7fdc83800640 1 --2- 172.21.15.47:0/1491803983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc7c15c390 0x7fdc7c134ea0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fdc7400efc0 tx=0x7fdc7400c490 comp rx=0 tx=0).stop 2024-02-19T15:26:36.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.355+0000 7fdc83800640 1 -- 172.21.15.47:0/1491803983 shutdown_connections 2024-02-19T15:26:36.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.355+0000 7fdc83800640 1 --2- 172.21.15.47:0/1491803983 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdc5c066e40 0x7fdc5c069300 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.355+0000 7fdc83800640 1 --2- 172.21.15.47:0/1491803983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc7c15c390 0x7fdc7c134ea0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.355+0000 7fdc83800640 1 --2- 172.21.15.47:0/1491803983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc7c145590 0x7fdc7c15a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.355+0000 7fdc83800640 1 --2- 172.21.15.47:0/1491803983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc7c0cea80 0x7fdc7c13cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.355+0000 7fdc83800640 1 -- 172.21.15.47:0/1491803983 >> 172.21.15.47:0/1491803983 conn(0x7fdc7c0b16a0 msgr2=0x7fdc7c13a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:36.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.355+0000 7fdc83800640 1 -- 172.21.15.47:0/1491803983 shutdown_connections 2024-02-19T15:26:36.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.355+0000 7fdc83800640 1 -- 172.21.15.47:0/1491803983 wait complete. 2024-02-19T15:26:36.381 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-02-19T15:26:35.211576+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":184,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":267996,"kb_used_data":10184,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754706724,"statfs":{"total":773094113280,"available":772819685376,"internally_reserved":0,"allocated":10428416,"data_stored":5093548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":263835186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001477"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.323439+0000","last_change":"2024-02-19T15:26:27.337979+0000","last_active":"2024-02-19T15:26:28.323439+0000","last_peered":"2024-02-19T15:26:28.323439+0000","last_clean":"2024-02-19T15:26:28.323439+0000","last_became_active":"2024-02-19T15:26:27.337666+0000","last_became_peered":"2024-02-19T15:26:27.337666+0000","last_unstale":"2024-02-19T15:26:28.323439+0000","last_undegraded":"2024-02-19T15:26:28.323439+0000","last_fullsized":"2024-02-19T15:26:28.323439+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:48.248568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754565+0000","last_change":"2024-02-19T15:26:27.344900+0000","last_active":"2024-02-19T15:26:28.754565+0000","last_peered":"2024-02-19T15:26:28.754565+0000","last_clean":"2024-02-19T15:26:28.754565+0000","last_became_active":"2024-02-19T15:26:27.344549+0000","last_became_peered":"2024-02-19T15:26:27.344549+0000","last_unstale":"2024-02-19T15:26:28.754565+0000","last_undegraded":"2024-02-19T15:26:28.754565+0000","last_fullsized":"2024-02-19T15:26:28.754565+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:33:08.165205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.323642+0000","last_change":"2024-02-19T15:26:27.336881+0000","last_active":"2024-02-19T15:26:28.323642+0000","last_peered":"2024-02-19T15:26:28.323642+0000","last_clean":"2024-02-19T15:26:28.323642+0000","last_became_active":"2024-02-19T15:26:27.336517+0000","last_became_peered":"2024-02-19T15:26:27.336517+0000","last_unstale":"2024-02-19T15:26:28.323642+0000","last_undegraded":"2024-02-19T15:26:28.323642+0000","last_fullsized":"2024-02-19T15:26:28.323642+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:14:11.249352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754111+0000","last_change":"2024-02-19T15:26:27.339956+0000","last_active":"2024-02-19T15:26:28.754111+0000","last_peered":"2024-02-19T15:26:28.754111+0000","last_clean":"2024-02-19T15:26:28.754111+0000","last_became_active":"2024-02-19T15:26:27.339629+0000","last_became_peered":"2024-02-19T15:26:27.339629+0000","last_unstale":"2024-02-19T15:26:28.754111+0000","last_undegraded":"2024-02-19T15:26:28.754111+0000","last_fullsized":"2024-02-19T15:26:28.754111+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:30:30.078661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.755001+0000","last_change":"2024-02-19T15:26:27.355403+0000","last_active":"2024-02-19T15:26:28.755001+0000","last_peered":"2024-02-19T15:26:28.755001+0000","last_clean":"2024-02-19T15:26:28.755001+0000","last_became_active":"2024-02-19T15:26:27.354350+0000","last_became_peered":"2024-02-19T15:26:27.354350+0000","last_unstale":"2024-02-19T15:26:28.755001+0000","last_undegraded":"2024-02-19T15:26:28.755001+0000","last_fullsized":"2024-02-19T15:26:28.755001+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:26:02.430798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.323844+0000","last_change":"2024-02-19T15:26:27.345606+0000","last_active":"2024-02-19T15:26:28.323844+0000","last_peered":"2024-02-19T15:26:28.323844+0000","last_clean":"2024-02-19T15:26:28.323844+0000","last_became_active":"2024-02-19T15:26:27.345335+0000","last_became_peered":"2024-02-19T15:26:27.345335+0000","last_unstale":"2024-02-19T15:26:28.323844+0000","last_undegraded":"2024-02-19T15:26:28.323844+0000","last_fullsized":"2024-02-19T15:26:28.323844+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:02:24.609597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.323865+0000","last_change":"2024-02-19T15:26:27.343302+0000","last_active":"2024-02-19T15:26:28.323865+0000","last_peered":"2024-02-19T15:26:28.323865+0000","last_clean":"2024-02-19T15:26:28.323865+0000","last_became_active":"2024-02-19T15:26:27.343019+0000","last_became_peered":"2024-02-19T15:26:27.343019+0000","last_unstale":"2024-02-19T15:26:28.323865+0000","last_undegraded":"2024-02-19T15:26:28.323865+0000","last_fullsized":"2024-02-19T15:26:28.323865+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:28:10.152629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.753668+0000","last_change":"2024-02-19T15:26:27.351000+0000","last_active":"2024-02-19T15:26:28.753668+0000","last_peered":"2024-02-19T15:26:28.753668+0000","last_clean":"2024-02-19T15:26:28.753668+0000","last_became_active":"2024-02-19T15:26:27.350725+0000","last_became_peered":"2024-02-19T15:26:27.350725+0000","last_unstale":"2024-02-19T15:26:28.753668+0000","last_undegraded":"2024-02-19T15:26:28.753668+0000","last_fullsized":"2024-02-19T15:26:28.753668+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:21:28.895741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754726+0000","last_change":"2024-02-19T15:26:27.350219+0000","last_active":"2024-02-19T15:26:28.754726+0000","last_peered":"2024-02-19T15:26:28.754726+0000","last_clean":"2024-02-19T15:26:28.754726+0000","last_became_active":"2024-02-19T15:26:27.349831+0000","last_became_peered":"2024-02-19T15:26:27.349831+0000","last_unstale":"2024-02-19T15:26:28.754726+0000","last_undegraded":"2024-02-19T15:26:28.754726+0000","last_fullsized":"2024-02-19T15:26:28.754726+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:29:49.888433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.755011+0000","last_change":"2024-02-19T15:26:27.350862+0000","last_active":"2024-02-19T15:26:28.755011+0000","last_peered":"2024-02-19T15:26:28.755011+0000","last_clean":"2024-02-19T15:26:28.755011+0000","last_became_active":"2024-02-19T15:26:27.350507+0000","last_became_peered":"2024-02-19T15:26:27.350507+0000","last_unstale":"2024-02-19T15:26:28.755011+0000","last_undegraded":"2024-02-19T15:26:28.755011+0000","last_fullsized":"2024-02-19T15:26:28.755011+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:17:42.002372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.323804+0000","last_change":"2024-02-19T15:26:27.357980+0000","last_active":"2024-02-19T15:26:28.323804+0000","last_peered":"2024-02-19T15:26:28.323804+0000","last_clean":"2024-02-19T15:26:28.323804+0000","last_became_active":"2024-02-19T15:26:27.357051+0000","last_became_peered":"2024-02-19T15:26:27.357051+0000","last_unstale":"2024-02-19T15:26:28.323804+0000","last_undegraded":"2024-02-19T15:26:28.323804+0000","last_fullsized":"2024-02-19T15:26:28.323804+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:24:26.250640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754265+0000","last_change":"2024-02-19T15:26:27.346673+0000","last_active":"2024-02-19T15:26:28.754265+0000","last_peered":"2024-02-19T15:26:28.754265+0000","last_clean":"2024-02-19T15:26:28.754265+0000","last_became_active":"2024-02-19T15:26:27.346329+0000","last_became_peered":"2024-02-19T15:26:27.346329+0000","last_unstale":"2024-02-19T15:26:28.754265+0000","last_undegraded":"2024-02-19T15:26:28.754265+0000","last_fullsized":"2024-02-19T15:26:28.754265+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:24:09.729954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.753937+0000","last_change":"2024-02-19T15:26:27.354225+0000","last_active":"2024-02-19T15:26:28.753937+0000","last_peered":"2024-02-19T15:26:28.753937+0000","last_clean":"2024-02-19T15:26:28.753937+0000","last_became_active":"2024-02-19T15:26:27.353667+0000","last_became_peered":"2024-02-19T15:26:27.353667+0000","last_unstale":"2024-02-19T15:26:28.753937+0000","last_undegraded":"2024-02-19T15:26:28.753937+0000","last_fullsized":"2024-02-19T15:26:28.753937+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:50:12.415248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.321690+0000","last_change":"2024-02-19T15:26:27.346636+0000","last_active":"2024-02-19T15:26:28.321690+0000","last_peered":"2024-02-19T15:26:28.321690+0000","last_clean":"2024-02-19T15:26:28.321690+0000","last_became_active":"2024-02-19T15:26:27.346196+0000","last_became_peered":"2024-02-19T15:26:27.346196+0000","last_unstale":"2024-02-19T15:26:28.321690+0000","last_undegraded":"2024-02-19T15:26:28.321690+0000","last_fullsized":"2024-02-19T15:26:28.321690+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:18:09.372062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.753965+0000","last_change":"2024-02-19T15:26:27.339335+0000","last_active":"2024-02-19T15:26:28.753965+0000","last_peered":"2024-02-19T15:26:28.753965+0000","last_clean":"2024-02-19T15:26:28.753965+0000","last_became_active":"2024-02-19T15:26:27.339018+0000","last_became_peered":"2024-02-19T15:26:27.339018+0000","last_unstale":"2024-02-19T15:26:28.753965+0000","last_undegraded":"2024-02-19T15:26:28.753965+0000","last_fullsized":"2024-02-19T15:26:28.753965+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:37:41.351032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.324079+0000","last_change":"2024-02-19T15:26:27.353093+0000","last_active":"2024-02-19T15:26:28.324079+0000","last_peered":"2024-02-19T15:26:28.324079+0000","last_clean":"2024-02-19T15:26:28.324079+0000","last_became_active":"2024-02-19T15:26:27.348237+0000","last_became_peered":"2024-02-19T15:26:27.348237+0000","last_unstale":"2024-02-19T15:26:28.324079+0000","last_undegraded":"2024-02-19T15:26:28.324079+0000","last_fullsized":"2024-02-19T15:26:28.324079+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:32:54.677722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754703+0000","last_change":"2024-02-19T15:26:27.345579+0000","last_active":"2024-02-19T15:26:28.754703+0000","last_peered":"2024-02-19T15:26:28.754703+0000","last_clean":"2024-02-19T15:26:28.754703+0000","last_became_active":"2024-02-19T15:26:27.345310+0000","last_became_peered":"2024-02-19T15:26:27.345310+0000","last_unstale":"2024-02-19T15:26:28.754703+0000","last_undegraded":"2024-02-19T15:26:28.754703+0000","last_fullsized":"2024-02-19T15:26:28.754703+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:24:15.382737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754279+0000","last_change":"2024-02-19T15:26:27.340348+0000","last_active":"2024-02-19T15:26:28.754279+0000","last_peered":"2024-02-19T15:26:28.754279+0000","last_clean":"2024-02-19T15:26:28.754279+0000","last_became_active":"2024-02-19T15:26:27.340087+0000","last_became_peered":"2024-02-19T15:26:27.340087+0000","last_unstale":"2024-02-19T15:26:28.754279+0000","last_undegraded":"2024-02-19T15:26:28.754279+0000","last_fullsized":"2024-02-19T15:26:28.754279+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:43:47.353181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.322026+0000","last_change":"2024-02-19T15:26:27.340292+0000","last_active":"2024-02-19T15:26:28.322026+0000","last_peered":"2024-02-19T15:26:28.322026+0000","last_clean":"2024-02-19T15:26:28.322026+0000","last_became_active":"2024-02-19T15:26:27.339977+0000","last_became_peered":"2024-02-19T15:26:27.339977+0000","last_unstale":"2024-02-19T15:26:28.322026+0000","last_undegraded":"2024-02-19T15:26:28.322026+0000","last_fullsized":"2024-02-19T15:26:28.322026+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:32:06.466331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754474+0000","last_change":"2024-02-19T15:26:27.356099+0000","last_active":"2024-02-19T15:26:28.754474+0000","last_peered":"2024-02-19T15:26:28.754474+0000","last_clean":"2024-02-19T15:26:28.754474+0000","last_became_active":"2024-02-19T15:26:27.355788+0000","last_became_peered":"2024-02-19T15:26:27.355788+0000","last_unstale":"2024-02-19T15:26:28.754474+0000","last_undegraded":"2024-02-19T15:26:28.754474+0000","last_fullsized":"2024-02-19T15:26:28.754474+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:10:37.099067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.322054+0000","last_change":"2024-02-19T15:26:27.333395+0000","last_active":"2024-02-19T15:26:28.322054+0000","last_peered":"2024-02-19T15:26:28.322054+0000","last_clean":"2024-02-19T15:26:28.322054+0000","last_became_active":"2024-02-19T15:26:27.333023+0000","last_became_peered":"2024-02-19T15:26:27.333023+0000","last_unstale":"2024-02-19T15:26:28.322054+0000","last_undegraded":"2024-02-19T15:26:28.322054+0000","last_fullsized":"2024-02-19T15:26:28.322054+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:33:22.706979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.753598+0000","last_change":"2024-02-19T15:26:27.334493+0000","last_active":"2024-02-19T15:26:28.753598+0000","last_peered":"2024-02-19T15:26:28.753598+0000","last_clean":"2024-02-19T15:26:28.753598+0000","last_became_active":"2024-02-19T15:26:27.334186+0000","last_became_peered":"2024-02-19T15:26:27.334186+0000","last_unstale":"2024-02-19T15:26:28.753598+0000","last_undegraded":"2024-02-19T15:26:28.753598+0000","last_fullsized":"2024-02-19T15:26:28.753598+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:08:16.059511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.753902+0000","last_change":"2024-02-19T15:26:27.332026+0000","last_active":"2024-02-19T15:26:28.753902+0000","last_peered":"2024-02-19T15:26:28.753902+0000","last_clean":"2024-02-19T15:26:28.753902+0000","last_became_active":"2024-02-19T15:26:27.331759+0000","last_became_peered":"2024-02-19T15:26:27.331759+0000","last_unstale":"2024-02-19T15:26:28.753902+0000","last_undegraded":"2024-02-19T15:26:28.753902+0000","last_fullsized":"2024-02-19T15:26:28.753902+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:28:36.858701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.323861+0000","last_change":"2024-02-19T15:26:27.354543+0000","last_active":"2024-02-19T15:26:28.323861+0000","last_peered":"2024-02-19T15:26:28.323861+0000","last_clean":"2024-02-19T15:26:28.323861+0000","last_became_active":"2024-02-19T15:26:27.354073+0000","last_became_peered":"2024-02-19T15:26:27.354073+0000","last_unstale":"2024-02-19T15:26:28.323861+0000","last_undegraded":"2024-02-19T15:26:28.323861+0000","last_fullsized":"2024-02-19T15:26:28.323861+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:07:09.293876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.755153+0000","last_change":"2024-02-19T15:26:27.345314+0000","last_active":"2024-02-19T15:26:28.755153+0000","last_peered":"2024-02-19T15:26:28.755153+0000","last_clean":"2024-02-19T15:26:28.755153+0000","last_became_active":"2024-02-19T15:26:27.345008+0000","last_became_peered":"2024-02-19T15:26:27.345008+0000","last_unstale":"2024-02-19T15:26:28.755153+0000","last_undegraded":"2024-02-19T15:26:28.755153+0000","last_fullsized":"2024-02-19T15:26:28.755153+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:47:43.660844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754630+0000","last_change":"2024-02-19T15:26:27.341010+0000","last_active":"2024-02-19T15:26:28.754630+0000","last_peered":"2024-02-19T15:26:28.754630+0000","last_clean":"2024-02-19T15:26:28.754630+0000","last_became_active":"2024-02-19T15:26:27.340498+0000","last_became_peered":"2024-02-19T15:26:27.340498+0000","last_unstale":"2024-02-19T15:26:28.754630+0000","last_undegraded":"2024-02-19T15:26:28.754630+0000","last_fullsized":"2024-02-19T15:26:28.754630+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:10.643609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.323987+0000","last_change":"2024-02-19T15:26:27.350557+0000","last_active":"2024-02-19T15:26:28.323987+0000","last_peered":"2024-02-19T15:26:28.323987+0000","last_clean":"2024-02-19T15:26:28.323987+0000","last_became_active":"2024-02-19T15:26:27.349942+0000","last_became_peered":"2024-02-19T15:26:27.349942+0000","last_unstale":"2024-02-19T15:26:28.323987+0000","last_undegraded":"2024-02-19T15:26:28.323987+0000","last_fullsized":"2024-02-19T15:26:28.323987+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:09.551076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.755321+0000","last_change":"2024-02-19T15:26:27.355116+0000","last_active":"2024-02-19T15:26:28.755321+0000","last_peered":"2024-02-19T15:26:28.755321+0000","last_clean":"2024-02-19T15:26:28.755321+0000","last_became_active":"2024-02-19T15:26:27.354775+0000","last_became_peered":"2024-02-19T15:26:27.354775+0000","last_unstale":"2024-02-19T15:26:28.755321+0000","last_undegraded":"2024-02-19T15:26:28.755321+0000","last_fullsized":"2024-02-19T15:26:28.755321+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:34:06.236974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.753803+0000","last_change":"2024-02-19T15:26:27.339741+0000","last_active":"2024-02-19T15:26:28.753803+0000","last_peered":"2024-02-19T15:26:28.753803+0000","last_clean":"2024-02-19T15:26:28.753803+0000","last_became_active":"2024-02-19T15:26:27.338809+0000","last_became_peered":"2024-02-19T15:26:27.338809+0000","last_unstale":"2024-02-19T15:26:28.753803+0000","last_undegraded":"2024-02-19T15:26:28.753803+0000","last_fullsized":"2024-02-19T15:26:28.753803+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:12:31.107805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.321715+0000","last_change":"2024-02-19T15:26:27.333630+0000","last_active":"2024-02-19T15:26:28.321715+0000","last_peered":"2024-02-19T15:26:28.321715+0000","last_clean":"2024-02-19T15:26:28.321715+0000","last_became_active":"2024-02-19T15:26:27.333260+0000","last_became_peered":"2024-02-19T15:26:27.333260+0000","last_unstale":"2024-02-19T15:26:28.321715+0000","last_undegraded":"2024-02-19T15:26:28.321715+0000","last_fullsized":"2024-02-19T15:26:28.321715+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:57:23.494064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754140+0000","last_change":"2024-02-19T15:26:27.360382+0000","last_active":"2024-02-19T15:26:28.754140+0000","last_peered":"2024-02-19T15:26:28.754140+0000","last_clean":"2024-02-19T15:26:28.754140+0000","last_became_active":"2024-02-19T15:26:27.359801+0000","last_became_peered":"2024-02-19T15:26:27.359801+0000","last_unstale":"2024-02-19T15:26:28.754140+0000","last_undegraded":"2024-02-19T15:26:28.754140+0000","last_fullsized":"2024-02-19T15:26:28.754140+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:17:17.252984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.323011+0000","last_change":"2024-02-19T15:26:27.354903+0000","last_active":"2024-02-19T15:26:28.323011+0000","last_peered":"2024-02-19T15:26:28.323011+0000","last_clean":"2024-02-19T15:26:28.323011+0000","last_became_active":"2024-02-19T15:26:27.354643+0000","last_became_peered":"2024-02-19T15:26:27.354643+0000","last_unstale":"2024-02-19T15:26:28.323011+0000","last_undegraded":"2024-02-19T15:26:28.323011+0000","last_fullsized":"2024-02-19T15:26:28.323011+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:53:31.145629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.322371+0000","last_change":"2024-02-19T15:26:27.345179+0000","last_active":"2024-02-19T15:26:28.322371+0000","last_peered":"2024-02-19T15:26:28.322371+0000","last_clean":"2024-02-19T15:26:28.322371+0000","last_became_active":"2024-02-19T15:26:27.344664+0000","last_became_peered":"2024-02-19T15:26:27.344664+0000","last_unstale":"2024-02-19T15:26:28.322371+0000","last_undegraded":"2024-02-19T15:26:28.322371+0000","last_fullsized":"2024-02-19T15:26:28.322371+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:19:08.693592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"92'4","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754655+0000","last_change":"2024-02-19T15:26:26.334907+0000","last_active":"2024-02-19T15:26:28.754655+0000","last_peered":"2024-02-19T15:26:28.754655+0000","last_clean":"2024-02-19T15:26:28.754655+0000","last_became_active":"2024-02-19T15:26:26.334573+0000","last_became_peered":"2024-02-19T15:26:26.334573+0000","last_unstale":"2024-02-19T15:26:28.754655+0000","last_undegraded":"2024-02-19T15:26:28.754655+0000","last_fullsized":"2024-02-19T15:26:28.754655+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:32:07.607358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754376+0000","last_change":"2024-02-19T15:26:27.359153+0000","last_active":"2024-02-19T15:26:28.754376+0000","last_peered":"2024-02-19T15:26:28.754376+0000","last_clean":"2024-02-19T15:26:28.754376+0000","last_became_active":"2024-02-19T15:26:27.358791+0000","last_became_peered":"2024-02-19T15:26:27.358791+0000","last_unstale":"2024-02-19T15:26:28.754376+0000","last_undegraded":"2024-02-19T15:26:28.754376+0000","last_fullsized":"2024-02-19T15:26:28.754376+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:24:26.250640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.323783+0000","last_change":"2024-02-19T15:26:26.337773+0000","last_active":"2024-02-19T15:26:28.323783+0000","last_peered":"2024-02-19T15:26:28.323783+0000","last_clean":"2024-02-19T15:26:28.323783+0000","last_became_active":"2024-02-19T15:26:26.337410+0000","last_became_peered":"2024-02-19T15:26:26.337410+0000","last_unstale":"2024-02-19T15:26:28.323783+0000","last_undegraded":"2024-02-19T15:26:28.323783+0000","last_fullsized":"2024-02-19T15:26:28.323783+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:10:14.877959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.323366+0000","last_change":"2024-02-19T15:26:27.343061+0000","last_active":"2024-02-19T15:26:28.323366+0000","last_peered":"2024-02-19T15:26:28.323366+0000","last_clean":"2024-02-19T15:26:28.323366+0000","last_became_active":"2024-02-19T15:26:27.342782+0000","last_became_peered":"2024-02-19T15:26:27.342782+0000","last_unstale":"2024-02-19T15:26:28.323366+0000","last_undegraded":"2024-02-19T15:26:28.323366+0000","last_fullsized":"2024-02-19T15:26:28.323366+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:20:22.479873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754418+0000","last_change":"2024-02-19T15:26:26.345367+0000","last_active":"2024-02-19T15:26:28.754418+0000","last_peered":"2024-02-19T15:26:28.754418+0000","last_clean":"2024-02-19T15:26:28.754418+0000","last_became_active":"2024-02-19T15:26:26.345075+0000","last_became_peered":"2024-02-19T15:26:26.345075+0000","last_unstale":"2024-02-19T15:26:28.754418+0000","last_undegraded":"2024-02-19T15:26:28.754418+0000","last_fullsized":"2024-02-19T15:26:28.754418+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:34:41.784025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754418+0000","last_change":"2024-02-19T15:26:27.340411+0000","last_active":"2024-02-19T15:26:28.754418+0000","last_peered":"2024-02-19T15:26:28.754418+0000","last_clean":"2024-02-19T15:26:28.754418+0000","last_became_active":"2024-02-19T15:26:27.340120+0000","last_became_peered":"2024-02-19T15:26:27.340120+0000","last_unstale":"2024-02-19T15:26:28.754418+0000","last_undegraded":"2024-02-19T15:26:28.754418+0000","last_fullsized":"2024-02-19T15:26:28.754418+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:57:46.016622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.753787+0000","last_change":"2024-02-19T15:26:26.346242+0000","last_active":"2024-02-19T15:26:28.753787+0000","last_peered":"2024-02-19T15:26:28.753787+0000","last_clean":"2024-02-19T15:26:28.753787+0000","last_became_active":"2024-02-19T15:26:26.345912+0000","last_became_peered":"2024-02-19T15:26:26.345912+0000","last_unstale":"2024-02-19T15:26:28.753787+0000","last_undegraded":"2024-02-19T15:26:28.753787+0000","last_fullsized":"2024-02-19T15:26:28.753787+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:07:29.269364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754402+0000","last_change":"2024-02-19T15:26:27.346536+0000","last_active":"2024-02-19T15:26:28.754402+0000","last_peered":"2024-02-19T15:26:28.754402+0000","last_clean":"2024-02-19T15:26:28.754402+0000","last_became_active":"2024-02-19T15:26:27.346215+0000","last_became_peered":"2024-02-19T15:26:27.346215+0000","last_unstale":"2024-02-19T15:26:28.754402+0000","last_undegraded":"2024-02-19T15:26:28.754402+0000","last_fullsized":"2024-02-19T15:26:28.754402+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:31:10.832966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.323225+0000","last_change":"2024-02-19T15:26:26.326093+0000","last_active":"2024-02-19T15:26:28.323225+0000","last_peered":"2024-02-19T15:26:28.323225+0000","last_clean":"2024-02-19T15:26:28.323225+0000","last_became_active":"2024-02-19T15:26:26.325829+0000","last_became_peered":"2024-02-19T15:26:26.325829+0000","last_unstale":"2024-02-19T15:26:28.323225+0000","last_undegraded":"2024-02-19T15:26:28.323225+0000","last_fullsized":"2024-02-19T15:26:28.323225+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:33:21.701067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.324158+0000","last_change":"2024-02-19T15:26:27.353726+0000","last_active":"2024-02-19T15:26:28.324158+0000","last_peered":"2024-02-19T15:26:28.324158+0000","last_clean":"2024-02-19T15:26:28.324158+0000","last_became_active":"2024-02-19T15:26:27.352957+0000","last_became_peered":"2024-02-19T15:26:27.352957+0000","last_unstale":"2024-02-19T15:26:28.324158+0000","last_undegraded":"2024-02-19T15:26:28.324158+0000","last_fullsized":"2024-02-19T15:26:28.324158+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:58:21.002586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.753597+0000","last_change":"2024-02-19T15:26:26.343337+0000","last_active":"2024-02-19T15:26:28.753597+0000","last_peered":"2024-02-19T15:26:28.753597+0000","last_clean":"2024-02-19T15:26:28.753597+0000","last_became_active":"2024-02-19T15:26:26.342958+0000","last_became_peered":"2024-02-19T15:26:26.342958+0000","last_unstale":"2024-02-19T15:26:28.753597+0000","last_undegraded":"2024-02-19T15:26:28.753597+0000","last_fullsized":"2024-02-19T15:26:28.753597+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:52:48.004697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.753762+0000","last_change":"2024-02-19T15:26:27.333886+0000","last_active":"2024-02-19T15:26:28.753762+0000","last_peered":"2024-02-19T15:26:28.753762+0000","last_clean":"2024-02-19T15:26:28.753762+0000","last_became_active":"2024-02-19T15:26:27.333575+0000","last_became_peered":"2024-02-19T15:26:27.333575+0000","last_unstale":"2024-02-19T15:26:28.753762+0000","last_undegraded":"2024-02-19T15:26:28.753762+0000","last_fullsized":"2024-02-19T15:26:28.753762+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:17:33.969687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754787+0000","last_change":"2024-02-19T15:26:26.336796+0000","last_active":"2024-02-19T15:26:28.754787+0000","last_peered":"2024-02-19T15:26:28.754787+0000","last_clean":"2024-02-19T15:26:28.754787+0000","last_became_active":"2024-02-19T15:26:26.336376+0000","last_became_peered":"2024-02-19T15:26:26.336376+0000","last_unstale":"2024-02-19T15:26:28.754787+0000","last_undegraded":"2024-02-19T15:26:28.754787+0000","last_fullsized":"2024-02-19T15:26:28.754787+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:17:32.519492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.753997+0000","last_change":"2024-02-19T15:26:27.339790+0000","last_active":"2024-02-19T15:26:28.753997+0000","last_peered":"2024-02-19T15:26:28.753997+0000","last_clean":"2024-02-19T15:26:28.753997+0000","last_became_active":"2024-02-19T15:26:27.339502+0000","last_became_peered":"2024-02-19T15:26:27.339502+0000","last_unstale":"2024-02-19T15:26:28.753997+0000","last_undegraded":"2024-02-19T15:26:28.753997+0000","last_fullsized":"2024-02-19T15:26:28.753997+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:20:59.776623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.322967+0000","last_change":"2024-02-19T15:26:26.323675+0000","last_active":"2024-02-19T15:26:28.322967+0000","last_peered":"2024-02-19T15:26:28.322967+0000","last_clean":"2024-02-19T15:26:28.322967+0000","last_became_active":"2024-02-19T15:26:26.323440+0000","last_became_peered":"2024-02-19T15:26:26.323440+0000","last_unstale":"2024-02-19T15:26:28.322967+0000","last_undegraded":"2024-02-19T15:26:28.322967+0000","last_fullsized":"2024-02-19T15:26:28.322967+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:50:05.139020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754663+0000","last_change":"2024-02-19T15:26:27.348589+0000","last_active":"2024-02-19T15:26:28.754663+0000","last_peered":"2024-02-19T15:26:28.754663+0000","last_clean":"2024-02-19T15:26:28.754663+0000","last_became_active":"2024-02-19T15:26:27.347440+0000","last_became_peered":"2024-02-19T15:26:27.347440+0000","last_unstale":"2024-02-19T15:26:28.754663+0000","last_undegraded":"2024-02-19T15:26:28.754663+0000","last_fullsized":"2024-02-19T15:26:28.754663+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:17:22.714278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.323500+0000","last_change":"2024-02-19T15:26:26.332241+0000","last_active":"2024-02-19T15:26:28.323500+0000","last_peered":"2024-02-19T15:26:28.323500+0000","last_clean":"2024-02-19T15:26:28.323500+0000","last_became_active":"2024-02-19T15:26:26.331906+0000","last_became_peered":"2024-02-19T15:26:26.331906+0000","last_unstale":"2024-02-19T15:26:28.323500+0000","last_undegraded":"2024-02-19T15:26:28.323500+0000","last_fullsized":"2024-02-19T15:26:28.323500+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:19:07.687679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754799+0000","last_change":"2024-02-19T15:26:27.344707+0000","last_active":"2024-02-19T15:26:28.754799+0000","last_peered":"2024-02-19T15:26:28.754799+0000","last_clean":"2024-02-19T15:26:28.754799+0000","last_became_active":"2024-02-19T15:26:27.344399+0000","last_became_peered":"2024-02-19T15:26:27.344399+0000","last_unstale":"2024-02-19T15:26:28.754799+0000","last_undegraded":"2024-02-19T15:26:28.754799+0000","last_fullsized":"2024-02-19T15:26:28.754799+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:17:33.498173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754228+0000","last_change":"2024-02-19T15:26:26.340433+0000","last_active":"2024-02-19T15:26:28.754228+0000","last_peered":"2024-02-19T15:26:28.754228+0000","last_clean":"2024-02-19T15:26:28.754228+0000","last_became_active":"2024-02-19T15:26:26.340035+0000","last_became_peered":"2024-02-19T15:26:26.340035+0000","last_unstale":"2024-02-19T15:26:28.754228+0000","last_undegraded":"2024-02-19T15:26:28.754228+0000","last_fullsized":"2024-02-19T15:26:28.754228+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:17:32.492261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.755400+0000","last_change":"2024-02-19T15:26:27.342930+0000","last_active":"2024-02-19T15:26:28.755400+0000","last_peered":"2024-02-19T15:26:28.755400+0000","last_clean":"2024-02-19T15:26:28.755400+0000","last_became_active":"2024-02-19T15:26:27.342457+0000","last_became_peered":"2024-02-19T15:26:27.342457+0000","last_unstale":"2024-02-19T15:26:28.755400+0000","last_undegraded":"2024-02-19T15:26:28.755400+0000","last_fullsized":"2024-02-19T15:26:28.755400+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:52:17.121403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"92'3","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.324137+0000","last_change":"2024-02-19T15:26:26.338262+0000","last_active":"2024-02-19T15:26:28.324137+0000","last_peered":"2024-02-19T15:26:28.324137+0000","last_clean":"2024-02-19T15:26:28.324137+0000","last_became_active":"2024-02-19T15:26:26.337954+0000","last_became_peered":"2024-02-19T15:26:26.337954+0000","last_unstale":"2024-02-19T15:26:28.324137+0000","last_undegraded":"2024-02-19T15:26:28.324137+0000","last_fullsized":"2024-02-19T15:26:28.324137+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:18.518541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.324076+0000","last_change":"2024-02-19T15:26:27.356659+0000","last_active":"2024-02-19T15:26:28.324076+0000","last_peered":"2024-02-19T15:26:28.324076+0000","last_clean":"2024-02-19T15:26:28.324076+0000","last_became_active":"2024-02-19T15:26:27.350249+0000","last_became_peered":"2024-02-19T15:26:27.350249+0000","last_unstale":"2024-02-19T15:26:28.324076+0000","last_undegraded":"2024-02-19T15:26:28.324076+0000","last_fullsized":"2024-02-19T15:26:28.324076+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:56:46.242162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"92'6","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754391+0000","last_change":"2024-02-19T15:26:26.329794+0000","last_active":"2024-02-19T15:26:28.754391+0000","last_peered":"2024-02-19T15:26:28.754391+0000","last_clean":"2024-02-19T15:26:28.754391+0000","last_became_active":"2024-02-19T15:26:26.328998+0000","last_became_peered":"2024-02-19T15:26:26.328998+0000","last_unstale":"2024-02-19T15:26:28.754391+0000","last_undegraded":"2024-02-19T15:26:28.754391+0000","last_fullsized":"2024-02-19T15:26:28.754391+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:20:21.473960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.753942+0000","last_change":"2024-02-19T15:26:27.345807+0000","last_active":"2024-02-19T15:26:28.753942+0000","last_peered":"2024-02-19T15:26:28.753942+0000","last_clean":"2024-02-19T15:26:28.753942+0000","last_became_active":"2024-02-19T15:26:27.345537+0000","last_became_peered":"2024-02-19T15:26:27.345537+0000","last_unstale":"2024-02-19T15:26:28.753942+0000","last_undegraded":"2024-02-19T15:26:28.753942+0000","last_fullsized":"2024-02-19T15:26:28.753942+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:24:36.429437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.322144+0000","last_change":"2024-02-19T15:26:26.333070+0000","last_active":"2024-02-19T15:26:28.322144+0000","last_peered":"2024-02-19T15:26:28.322144+0000","last_clean":"2024-02-19T15:26:28.322144+0000","last_became_active":"2024-02-19T15:26:26.332803+0000","last_became_peered":"2024-02-19T15:26:26.332803+0000","last_unstale":"2024-02-19T15:26:28.322144+0000","last_undegraded":"2024-02-19T15:26:28.322144+0000","last_fullsized":"2024-02-19T15:26:28.322144+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:14:10.954235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.323295+0000","last_change":"2024-02-19T15:26:27.336531+0000","last_active":"2024-02-19T15:26:28.323295+0000","last_peered":"2024-02-19T15:26:28.323295+0000","last_clean":"2024-02-19T15:26:28.323295+0000","last_became_active":"2024-02-19T15:26:27.336219+0000","last_became_peered":"2024-02-19T15:26:27.336219+0000","last_unstale":"2024-02-19T15:26:28.323295+0000","last_undegraded":"2024-02-19T15:26:28.323295+0000","last_fullsized":"2024-02-19T15:26:28.323295+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:32.310022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.321892+0000","last_change":"2024-02-19T15:26:26.333724+0000","last_active":"2024-02-19T15:26:28.321892+0000","last_peered":"2024-02-19T15:26:28.321892+0000","last_clean":"2024-02-19T15:26:28.321892+0000","last_became_active":"2024-02-19T15:26:26.333404+0000","last_became_peered":"2024-02-19T15:26:26.333404+0000","last_unstale":"2024-02-19T15:26:28.321892+0000","last_undegraded":"2024-02-19T15:26:28.321892+0000","last_fullsized":"2024-02-19T15:26:28.321892+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:25:32.892788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.753770+0000","last_change":"2024-02-19T15:26:27.345055+0000","last_active":"2024-02-19T15:26:28.753770+0000","last_peered":"2024-02-19T15:26:28.753770+0000","last_clean":"2024-02-19T15:26:28.753770+0000","last_became_active":"2024-02-19T15:26:27.344792+0000","last_became_peered":"2024-02-19T15:26:27.344792+0000","last_unstale":"2024-02-19T15:26:28.753770+0000","last_undegraded":"2024-02-19T15:26:28.753770+0000","last_fullsized":"2024-02-19T15:26:28.753770+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:53:12.543784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754051+0000","last_change":"2024-02-19T15:26:26.326965+0000","last_active":"2024-02-19T15:26:28.754051+0000","last_peered":"2024-02-19T15:26:28.754051+0000","last_clean":"2024-02-19T15:26:28.754051+0000","last_became_active":"2024-02-19T15:26:26.326327+0000","last_became_peered":"2024-02-19T15:26:26.326327+0000","last_unstale":"2024-02-19T15:26:28.754051+0000","last_undegraded":"2024-02-19T15:26:28.754051+0000","last_fullsized":"2024-02-19T15:26:28.754051+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:31.304110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.321458+0000","last_change":"2024-02-19T15:26:27.333702+0000","last_active":"2024-02-19T15:26:28.321458+0000","last_peered":"2024-02-19T15:26:28.321458+0000","last_clean":"2024-02-19T15:26:28.321458+0000","last_became_active":"2024-02-19T15:26:27.333417+0000","last_became_peered":"2024-02-19T15:26:27.333417+0000","last_unstale":"2024-02-19T15:26:28.321458+0000","last_undegraded":"2024-02-19T15:26:28.321458+0000","last_fullsized":"2024-02-19T15:26:28.321458+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:45:01.092253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"92'8","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.323079+0000","last_change":"2024-02-19T15:26:26.334414+0000","last_active":"2024-02-19T15:26:28.323079+0000","last_peered":"2024-02-19T15:26:28.323079+0000","last_clean":"2024-02-19T15:26:28.323079+0000","last_became_active":"2024-02-19T15:26:26.333923+0000","last_became_peered":"2024-02-19T15:26:26.333923+0000","last_unstale":"2024-02-19T15:26:28.323079+0000","last_undegraded":"2024-02-19T15:26:28.323079+0000","last_fullsized":"2024-02-19T15:26:28.323079+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:54.938001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.755285+0000","last_change":"2024-02-19T15:26:27.351736+0000","last_active":"2024-02-19T15:26:28.755285+0000","last_peered":"2024-02-19T15:26:28.755285+0000","last_clean":"2024-02-19T15:26:28.755285+0000","last_became_active":"2024-02-19T15:26:27.351433+0000","last_became_peered":"2024-02-19T15:26:27.351433+0000","last_unstale":"2024-02-19T15:26:28.755285+0000","last_undegraded":"2024-02-19T15:26:28.755285+0000","last_fullsized":"2024-02-19T15:26:28.755285+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:52:41.699736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754371+0000","last_change":"2024-02-19T15:26:26.327967+0000","last_active":"2024-02-19T15:26:28.754371+0000","last_peered":"2024-02-19T15:26:28.754371+0000","last_clean":"2024-02-19T15:26:28.754371+0000","last_became_active":"2024-02-19T15:26:26.327674+0000","last_became_peered":"2024-02-19T15:26:26.327674+0000","last_unstale":"2024-02-19T15:26:28.754371+0000","last_undegraded":"2024-02-19T15:26:28.754371+0000","last_fullsized":"2024-02-19T15:26:28.754371+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:32.658574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.753840+0000","last_change":"2024-02-19T15:26:27.344246+0000","last_active":"2024-02-19T15:26:28.753840+0000","last_peered":"2024-02-19T15:26:28.753840+0000","last_clean":"2024-02-19T15:26:28.753840+0000","last_became_active":"2024-02-19T15:26:27.343879+0000","last_became_peered":"2024-02-19T15:26:27.343879+0000","last_unstale":"2024-02-19T15:26:28.753840+0000","last_undegraded":"2024-02-19T15:26:28.753840+0000","last_fullsized":"2024-02-19T15:26:28.753840+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:16:03.256159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.755051+0000","last_change":"2024-02-19T15:26:26.328022+0000","last_active":"2024-02-19T15:26:28.755051+0000","last_peered":"2024-02-19T15:26:28.755051+0000","last_clean":"2024-02-19T15:26:28.755051+0000","last_became_active":"2024-02-19T15:26:26.327714+0000","last_became_peered":"2024-02-19T15:26:26.327714+0000","last_unstale":"2024-02-19T15:26:28.755051+0000","last_undegraded":"2024-02-19T15:26:28.755051+0000","last_fullsized":"2024-02-19T15:26:28.755051+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:04.571445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.324163+0000","last_change":"2024-02-19T15:26:27.358053+0000","last_active":"2024-02-19T15:26:28.324163+0000","last_peered":"2024-02-19T15:26:28.324163+0000","last_clean":"2024-02-19T15:26:28.324163+0000","last_became_active":"2024-02-19T15:26:27.357723+0000","last_became_peered":"2024-02-19T15:26:27.357723+0000","last_unstale":"2024-02-19T15:26:28.324163+0000","last_undegraded":"2024-02-19T15:26:28.324163+0000","last_fullsized":"2024-02-19T15:26:28.324163+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:44:41.405654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754125+0000","last_change":"2024-02-19T15:26:26.346345+0000","last_active":"2024-02-19T15:26:28.754125+0000","last_peered":"2024-02-19T15:26:28.754125+0000","last_clean":"2024-02-19T15:26:28.754125+0000","last_became_active":"2024-02-19T15:26:26.345748+0000","last_became_peered":"2024-02-19T15:26:26.345748+0000","last_unstale":"2024-02-19T15:26:28.754125+0000","last_undegraded":"2024-02-19T15:26:28.754125+0000","last_fullsized":"2024-02-19T15:26:28.754125+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:07:31.927738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.321838+0000","last_change":"2024-02-19T15:26:27.340383+0000","last_active":"2024-02-19T15:26:28.321838+0000","last_peered":"2024-02-19T15:26:28.321838+0000","last_clean":"2024-02-19T15:26:28.321838+0000","last_became_active":"2024-02-19T15:26:27.340150+0000","last_became_peered":"2024-02-19T15:26:27.340150+0000","last_unstale":"2024-02-19T15:26:28.321838+0000","last_undegraded":"2024-02-19T15:26:28.321838+0000","last_fullsized":"2024-02-19T15:26:28.321838+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:16:03.630318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.753860+0000","last_change":"2024-02-19T15:26:26.348704+0000","last_active":"2024-02-19T15:26:28.753860+0000","last_peered":"2024-02-19T15:26:28.753860+0000","last_clean":"2024-02-19T15:26:28.753860+0000","last_became_active":"2024-02-19T15:26:26.348409+0000","last_became_peered":"2024-02-19T15:26:26.348409+0000","last_unstale":"2024-02-19T15:26:28.753860+0000","last_undegraded":"2024-02-19T15:26:28.753860+0000","last_fullsized":"2024-02-19T15:26:28.753860+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:05:52.119643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.322190+0000","last_change":"2024-02-19T15:26:27.333473+0000","last_active":"2024-02-19T15:26:28.322190+0000","last_peered":"2024-02-19T15:26:28.322190+0000","last_clean":"2024-02-19T15:26:28.322190+0000","last_became_active":"2024-02-19T15:26:27.333167+0000","last_became_peered":"2024-02-19T15:26:27.333167+0000","last_unstale":"2024-02-19T15:26:28.322190+0000","last_undegraded":"2024-02-19T15:26:28.322190+0000","last_fullsized":"2024-02-19T15:26:28.322190+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:50:06.144932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754448+0000","last_change":"2024-02-19T15:26:26.331652+0000","last_active":"2024-02-19T15:26:28.754448+0000","last_peered":"2024-02-19T15:26:28.754448+0000","last_clean":"2024-02-19T15:26:28.754448+0000","last_became_active":"2024-02-19T15:26:26.331351+0000","last_became_peered":"2024-02-19T15:26:26.331351+0000","last_unstale":"2024-02-19T15:26:28.754448+0000","last_undegraded":"2024-02-19T15:26:28.754448+0000","last_fullsized":"2024-02-19T15:26:28.754448+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:20:21.473960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.321461+0000","last_change":"2024-02-19T15:26:27.334553+0000","last_active":"2024-02-19T15:26:28.321461+0000","last_peered":"2024-02-19T15:26:28.321461+0000","last_clean":"2024-02-19T15:26:28.321461+0000","last_became_active":"2024-02-19T15:26:27.334285+0000","last_became_peered":"2024-02-19T15:26:27.334285+0000","last_unstale":"2024-02-19T15:26:28.321461+0000","last_undegraded":"2024-02-19T15:26:28.321461+0000","last_fullsized":"2024-02-19T15:26:28.321461+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:33:04.613131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.753665+0000","last_change":"2024-02-19T15:26:26.335622+0000","last_active":"2024-02-19T15:26:28.753665+0000","last_peered":"2024-02-19T15:26:28.753665+0000","last_clean":"2024-02-19T15:26:28.753665+0000","last_became_active":"2024-02-19T15:26:26.335315+0000","last_became_peered":"2024-02-19T15:26:26.335315+0000","last_unstale":"2024-02-19T15:26:28.753665+0000","last_undegraded":"2024-02-19T15:26:28.753665+0000","last_fullsized":"2024-02-19T15:26:28.753665+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:31.304110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754554+0000","last_change":"2024-02-19T15:26:27.350470+0000","last_active":"2024-02-19T15:26:28.754554+0000","last_peered":"2024-02-19T15:26:28.754554+0000","last_clean":"2024-02-19T15:26:28.754554+0000","last_became_active":"2024-02-19T15:26:27.350119+0000","last_became_peered":"2024-02-19T15:26:27.350119+0000","last_unstale":"2024-02-19T15:26:28.754554+0000","last_undegraded":"2024-02-19T15:26:28.754554+0000","last_fullsized":"2024-02-19T15:26:28.754554+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:46:26.832360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.753570+0000","last_change":"2024-02-19T15:26:26.343466+0000","last_active":"2024-02-19T15:26:28.753570+0000","last_peered":"2024-02-19T15:26:28.753570+0000","last_clean":"2024-02-19T15:26:28.753570+0000","last_became_active":"2024-02-19T15:26:26.342948+0000","last_became_peered":"2024-02-19T15:26:26.342948+0000","last_unstale":"2024-02-19T15:26:28.753570+0000","last_undegraded":"2024-02-19T15:26:28.753570+0000","last_fullsized":"2024-02-19T15:26:28.753570+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:10:00.665955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754765+0000","last_change":"2024-02-19T15:26:27.340772+0000","last_active":"2024-02-19T15:26:28.754765+0000","last_peered":"2024-02-19T15:26:28.754765+0000","last_clean":"2024-02-19T15:26:28.754765+0000","last_became_active":"2024-02-19T15:26:27.340162+0000","last_became_peered":"2024-02-19T15:26:27.340162+0000","last_unstale":"2024-02-19T15:26:28.754765+0000","last_undegraded":"2024-02-19T15:26:28.754765+0000","last_fullsized":"2024-02-19T15:26:28.754765+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:06:27.061083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754006+0000","last_change":"2024-02-19T15:26:26.333028+0000","last_active":"2024-02-19T15:26:28.754006+0000","last_peered":"2024-02-19T15:26:28.754006+0000","last_clean":"2024-02-19T15:26:28.754006+0000","last_became_active":"2024-02-19T15:26:26.332697+0000","last_became_peered":"2024-02-19T15:26:26.332697+0000","last_unstale":"2024-02-19T15:26:28.754006+0000","last_undegraded":"2024-02-19T15:26:28.754006+0000","last_fullsized":"2024-02-19T15:26:28.754006+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:32.658574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.323486+0000","last_change":"2024-02-19T15:26:27.335656+0000","last_active":"2024-02-19T15:26:28.323486+0000","last_peered":"2024-02-19T15:26:28.323486+0000","last_clean":"2024-02-19T15:26:28.323486+0000","last_became_active":"2024-02-19T15:26:27.335352+0000","last_became_peered":"2024-02-19T15:26:27.335352+0000","last_unstale":"2024-02-19T15:26:28.323486+0000","last_undegraded":"2024-02-19T15:26:28.323486+0000","last_fullsized":"2024-02-19T15:26:28.323486+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:30:10.209882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"93'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754755+0000","last_change":"2024-02-19T15:26:26.344380+0000","last_active":"2024-02-19T15:26:28.754755+0000","last_peered":"2024-02-19T15:26:28.754755+0000","last_clean":"2024-02-19T15:26:28.754755+0000","last_became_active":"2024-02-19T15:26:26.340741+0000","last_became_peered":"2024-02-19T15:26:26.340741+0000","last_unstale":"2024-02-19T15:26:28.754755+0000","last_undegraded":"2024-02-19T15:26:28.754755+0000","last_fullsized":"2024-02-19T15:26:28.754755+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:17:40.996459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.323439+0000","last_change":"2024-02-19T15:26:27.350336+0000","last_active":"2024-02-19T15:26:28.323439+0000","last_peered":"2024-02-19T15:26:28.323439+0000","last_clean":"2024-02-19T15:26:28.323439+0000","last_became_active":"2024-02-19T15:26:27.350063+0000","last_became_peered":"2024-02-19T15:26:27.350063+0000","last_unstale":"2024-02-19T15:26:28.323439+0000","last_undegraded":"2024-02-19T15:26:28.323439+0000","last_fullsized":"2024-02-19T15:26:28.323439+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:33.664486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754460+0000","last_change":"2024-02-19T15:26:26.341542+0000","last_active":"2024-02-19T15:26:28.754460+0000","last_peered":"2024-02-19T15:26:28.754460+0000","last_clean":"2024-02-19T15:26:28.754460+0000","last_became_active":"2024-02-19T15:26:26.341121+0000","last_became_peered":"2024-02-19T15:26:26.341121+0000","last_unstale":"2024-02-19T15:26:28.754460+0000","last_undegraded":"2024-02-19T15:26:28.754460+0000","last_fullsized":"2024-02-19T15:26:28.754460+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:26:01.424886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754982+0000","last_change":"2024-02-19T15:26:27.355445+0000","last_active":"2024-02-19T15:26:28.754982+0000","last_peered":"2024-02-19T15:26:28.754982+0000","last_clean":"2024-02-19T15:26:28.754982+0000","last_became_active":"2024-02-19T15:26:27.355166+0000","last_became_peered":"2024-02-19T15:26:27.355166+0000","last_unstale":"2024-02-19T15:26:28.754982+0000","last_undegraded":"2024-02-19T15:26:28.754982+0000","last_fullsized":"2024-02-19T15:26:28.754982+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:34:42.789937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.755441+0000","last_change":"2024-02-19T15:26:26.338468+0000","last_active":"2024-02-19T15:26:28.755441+0000","last_peered":"2024-02-19T15:26:28.755441+0000","last_clean":"2024-02-19T15:26:28.755441+0000","last_became_active":"2024-02-19T15:26:26.338119+0000","last_became_peered":"2024-02-19T15:26:26.338119+0000","last_unstale":"2024-02-19T15:26:28.755441+0000","last_undegraded":"2024-02-19T15:26:28.755441+0000","last_fullsized":"2024-02-19T15:26:28.755441+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:37:04.837745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.322019+0000","last_change":"2024-02-19T15:26:27.341131+0000","last_active":"2024-02-19T15:26:28.322019+0000","last_peered":"2024-02-19T15:26:28.322019+0000","last_clean":"2024-02-19T15:26:28.322019+0000","last_became_active":"2024-02-19T15:26:27.339753+0000","last_became_peered":"2024-02-19T15:26:27.339753+0000","last_unstale":"2024-02-19T15:26:28.322019+0000","last_undegraded":"2024-02-19T15:26:28.322019+0000","last_fullsized":"2024-02-19T15:26:28.322019+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:12:38.726596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.753960+0000","last_change":"2024-02-19T15:26:26.343964+0000","last_active":"2024-02-19T15:26:28.753960+0000","last_peered":"2024-02-19T15:26:28.753960+0000","last_clean":"2024-02-19T15:26:28.753960+0000","last_became_active":"2024-02-19T15:26:26.343666+0000","last_became_peered":"2024-02-19T15:26:26.343666+0000","last_unstale":"2024-02-19T15:26:28.753960+0000","last_undegraded":"2024-02-19T15:26:28.753960+0000","last_fullsized":"2024-02-19T15:26:28.753960+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:44:07.209291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754236+0000","last_change":"2024-02-19T15:26:27.351921+0000","last_active":"2024-02-19T15:26:28.754236+0000","last_peered":"2024-02-19T15:26:28.754236+0000","last_clean":"2024-02-19T15:26:28.754236+0000","last_became_active":"2024-02-19T15:26:27.351448+0000","last_became_peered":"2024-02-19T15:26:27.351448+0000","last_unstale":"2024-02-19T15:26:28.754236+0000","last_undegraded":"2024-02-19T15:26:28.754236+0000","last_fullsized":"2024-02-19T15:26:28.754236+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:11:33.597919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754868+0000","last_change":"2024-02-19T15:26:26.329556+0000","last_active":"2024-02-19T15:26:28.754868+0000","last_peered":"2024-02-19T15:26:28.754868+0000","last_clean":"2024-02-19T15:26:28.754868+0000","last_became_active":"2024-02-19T15:26:26.329110+0000","last_became_peered":"2024-02-19T15:26:26.329110+0000","last_unstale":"2024-02-19T15:26:28.754868+0000","last_undegraded":"2024-02-19T15:26:28.754868+0000","last_fullsized":"2024-02-19T15:26:28.754868+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:15:11.157558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.321454+0000","last_change":"2024-02-19T15:26:27.332396+0000","last_active":"2024-02-19T15:26:28.321454+0000","last_peered":"2024-02-19T15:26:28.321454+0000","last_clean":"2024-02-19T15:26:28.321454+0000","last_became_active":"2024-02-19T15:26:27.332034+0000","last_became_peered":"2024-02-19T15:26:27.332034+0000","last_unstale":"2024-02-19T15:26:28.321454+0000","last_undegraded":"2024-02-19T15:26:28.321454+0000","last_fullsized":"2024-02-19T15:26:28.321454+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:15:06.008658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754216+0000","last_change":"2024-02-19T15:26:26.331689+0000","last_active":"2024-02-19T15:26:28.754216+0000","last_peered":"2024-02-19T15:26:28.754216+0000","last_clean":"2024-02-19T15:26:28.754216+0000","last_became_active":"2024-02-19T15:26:26.331404+0000","last_became_peered":"2024-02-19T15:26:26.331404+0000","last_unstale":"2024-02-19T15:26:28.754216+0000","last_undegraded":"2024-02-19T15:26:28.754216+0000","last_fullsized":"2024-02-19T15:26:28.754216+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:24:08.724042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.754692+0000","last_change":"2024-02-19T15:26:27.354547+0000","last_active":"2024-02-19T15:26:28.754692+0000","last_peered":"2024-02-19T15:26:28.754692+0000","last_clean":"2024-02-19T15:26:28.754692+0000","last_became_active":"2024-02-19T15:26:27.354276+0000","last_became_peered":"2024-02-19T15:26:27.354276+0000","last_unstale":"2024-02-19T15:26:28.754692+0000","last_undegraded":"2024-02-19T15:26:28.754692+0000","last_fullsized":"2024-02-19T15:26:28.754692+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:32:08.613270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.321512+0000","last_change":"2024-02-19T15:26:26.345992+0000","last_active":"2024-02-19T15:26:28.321512+0000","last_peered":"2024-02-19T15:26:28.321512+0000","last_clean":"2024-02-19T15:26:28.321512+0000","last_became_active":"2024-02-19T15:26:26.345737+0000","last_became_peered":"2024-02-19T15:26:26.345737+0000","last_unstale":"2024-02-19T15:26:28.321512+0000","last_undegraded":"2024-02-19T15:26:28.321512+0000","last_fullsized":"2024-02-19T15:26:28.321512+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:02:48.353389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.322362+0000","last_change":"2024-02-19T15:26:27.344919+0000","last_active":"2024-02-19T15:26:28.322362+0000","last_peered":"2024-02-19T15:26:28.322362+0000","last_clean":"2024-02-19T15:26:28.322362+0000","last_became_active":"2024-02-19T15:26:27.344299+0000","last_became_peered":"2024-02-19T15:26:27.344299+0000","last_unstale":"2024-02-19T15:26:28.322362+0000","last_undegraded":"2024-02-19T15:26:28.322362+0000","last_fullsized":"2024-02-19T15:26:28.322362+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:26.307873+0000","last_clean_scrub_stamp":"2024-02-19T15:26:26.307873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:55.943914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.323859+0000","last_change":"2024-02-19T15:26:26.335747+0000","last_active":"2024-02-19T15:26:28.323859+0000","last_peered":"2024-02-19T15:26:28.323859+0000","last_clean":"2024-02-19T15:26:28.323859+0000","last_became_active":"2024-02-19T15:26:26.335456+0000","last_became_peered":"2024-02-19T15:26:26.335456+0000","last_unstale":"2024-02-19T15:26:28.323859+0000","last_undegraded":"2024-02-19T15:26:28.323859+0000","last_fullsized":"2024-02-19T15:26:28.323859+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:26:25.301961+0000","last_clean_scrub_stamp":"2024-02-19T15:26:25.301961+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:01:15.445329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":298,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-19T15:26:28.324061+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:26:28.324061+0000","last_peered":"2024-02-19T15:26:28.324061+0000","last_clean":"2024-02-19T15:26:28.324061+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:26:28.324061+0000","last_undegraded":"2024-02-19T15:26:28.324061+0000","last_fullsized":"2024-02-19T15:26:28.324061+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509499,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":1155072,"data_stored":489151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.61399999999999999,"15min":0.61699999999999999},"min":{"1min":0.42899999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83599999999999997,"5min":1.022,"15min":1.022},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.68000000000000005,"15min":0.752},"min":{"1min":0.433,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.96699999999999997,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.51700000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.64500000000000002,"15min":0.65200000000000002},"min":{"1min":0.49299999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.81999999999999995,"5min":1.171,"15min":1.171},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65600000000000003,"15min":0.65200000000000002},"min":{"1min":0.40300000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.55900000000000005}]},{"osd":2,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.437,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.175,"5min":1.216,"15min":1.216},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.70599999999999996,"15min":0.69099999999999995},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.95099999999999996,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.57399999999999995}]},{"osd":3,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69499999999999995,"15min":0.68100000000000005},"min":{"1min":0.46899999999999997,"5min":0.438,"15min":0.438},"max":{"1min":0.90900000000000003,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67600000000000005,"15min":0.66200000000000003},"min":{"1min":0.48399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.87,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.58099999999999996}]},{"osd":4,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.64900000000000002,"15min":0.61499999999999999},"min":{"1min":0.40400000000000003,"5min":0.309,"15min":0.309},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.66200000000000003,"15min":0.61499999999999999},"min":{"1min":0.41699999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.436}]},{"osd":5,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.65700000000000003,"15min":0.61299999999999999},"min":{"1min":0.439,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.72199999999999998,"15min":0.68200000000000005},"min":{"1min":0.56899999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.502}]},{"osd":7,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.72199999999999998,"15min":0.66800000000000004},"min":{"1min":0.48899999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.74299999999999999,"15min":0.68899999999999995},"min":{"1min":0.67000000000000004,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.48699999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542203,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32892,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338948,"statfs":{"total":96636764160,"available":96603082752,"internally_reserved":0,"allocated":1748992,"data_stored":1078979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.59699999999999998,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.87,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.67400000000000004},"min":{"1min":0.38800000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.69299999999999995}]},{"osd":1,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.63500000000000001,"15min":0.60099999999999998},"min":{"1min":0.498,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86799999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.70399999999999996,"15min":0.72099999999999997},"min":{"1min":0.496,"5min":0.437,"15min":0.437},"max":{"1min":1.008,"5min":2.968,"15min":2.968},"last":0.65800000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.64700000000000002,"15min":0.63600000000000001},"min":{"1min":0.47599999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85399999999999998,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.497,"5min":0.438,"15min":0.438},"max":{"1min":0.97399999999999998,"5min":2.508,"15min":2.508},"last":0.59399999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66300000000000003,"15min":0.65900000000000003},"min":{"1min":0.50900000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.91300000000000003,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.70999999999999996,"15min":0.69499999999999995},"min":{"1min":0.59399999999999997,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.94699999999999995,"5min":3.1720000000000002,"15min":3.1720000000000002},"last":0.53700000000000003}]},{"osd":4,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.59499999999999997,"15min":0.53900000000000003},"min":{"1min":0.432,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.61799999999999999,"15min":0.58199999999999996},"min":{"1min":0.40100000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93600000000000005,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.498}]},{"osd":5,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.65700000000000003,"15min":0.58399999999999996},"min":{"1min":0.53500000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.63800000000000001},"min":{"1min":0.54700000000000004,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.97299999999999998,"5min":1.046,"15min":1.046},"last":0.441}]},{"osd":6,"last update":"Mon Feb 19 15:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.67100000000000004,"15min":0.61299999999999999},"min":{"1min":0.46999999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.375},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72599999999999998,"15min":0.67500000000000004},"min":{"1min":0.59699999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.61899999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542204,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":1150976,"data_stored":488569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.67200000000000004,"15min":0.64400000000000002},"min":{"1min":0.38300000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.97699999999999998,"5min":5.7789999999999999,"15min":5.7789999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.71099999999999997,"15min":0.63700000000000001},"min":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.86499999999999999,"5min":12.359999999999999,"15min":12.359999999999999},"last":0.76700000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.73799999999999999,"15min":0.67000000000000004},"min":{"1min":0.432,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80000000000000004,"5min":10.938000000000001,"15min":10.938000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.78900000000000003,"15min":0.71299999999999997},"min":{"1min":0.50600000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85599999999999998,"5min":10.531000000000001,"15min":10.531000000000001},"last":0.78100000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.79700000000000004,"15min":0.72099999999999997},"min":{"1min":0.42999999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.89900000000000002,"5min":12.925000000000001,"15min":12.925000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.72999999999999998,"15min":0.64100000000000001},"min":{"1min":0.48899999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89000000000000001,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.81000000000000005}]},{"osd":3,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.91100000000000003,"15min":0.73499999999999999},"min":{"1min":0.49399999999999999,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82299999999999995,"5min":26.161999999999999,"15min":26.161999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.91000000000000003,"15min":0.73999999999999999},"min":{"1min":0.47299999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1120000000000001,"5min":27.934999999999999,"15min":27.934999999999999},"last":0.67600000000000005}]},{"osd":4,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.67500000000000004,"15min":0.59199999999999997},"min":{"1min":0.45700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83099999999999996,"5min":9.0549999999999997,"15min":9.0549999999999997},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.70899999999999996,"15min":0.67100000000000004},"min":{"1min":0.44600000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.92800000000000005,"5min":6.2549999999999999,"15min":6.2549999999999999},"last":0.51600000000000001}]},{"osd":6,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.746,"15min":0.67000000000000004},"min":{"1min":0.50700000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.6539999999999999,"5min":6.5229999999999997,"15min":6.5229999999999997},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81899999999999995,"15min":0.749},"min":{"1min":0.63500000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.2310000000000001,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":0.73999999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.77600000000000002,"15min":0.69399999999999995},"min":{"1min":0.41499999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.95399999999999996,"5min":11.986000000000001,"15min":11.986000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.77700000000000002,"15min":0.71499999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.89100000000000001,"5min":8.9770000000000003,"15min":8.9770000000000003},"last":0.754}]}]},{"osd":4,"up_from":14,"seq":60129542202,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32328,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339512,"statfs":{"total":96636764160,"available":96603660288,"internally_reserved":0,"allocated":1167360,"data_stored":490235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.378,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.432},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.623,"15min":0.60499999999999998},"min":{"1min":0.47099999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85499999999999998,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.54600000000000004}]},{"osd":1,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64100000000000001,"15min":0.61899999999999999},"min":{"1min":0.48999999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.879,"5min":1.081,"15min":1.081},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64400000000000002,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83699999999999997,"5min":1.008,"15min":1.008},"last":0.47399999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.65400000000000003,"15min":0.66000000000000003},"min":{"1min":0.49299999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.90700000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68200000000000005,"15min":0.67600000000000005},"min":{"1min":0.45100000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92000000000000004,"5min":1.002,"15min":1.002},"last":0.75600000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.70799999999999996,"15min":0.68200000000000005},"min":{"1min":0.50600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94899999999999995,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.65900000000000003,"15min":0.65300000000000002},"min":{"1min":0.42799999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89600000000000002,"5min":1.153,"15min":1.153},"last":0.624}]},{"osd":5,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63600000000000001,"15min":0.60599999999999998},"min":{"1min":0.433,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85399999999999998,"5min":1.042,"15min":1.042},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70599999999999996,"15min":0.68200000000000005},"min":{"1min":0.54200000000000004,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.94799999999999995,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.65200000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.622,"15min":0.58299999999999996},"min":{"1min":0.376,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.92200000000000004,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68600000000000005,"15min":0.65500000000000003},"min":{"1min":0.52900000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90800000000000003,"5min":1.054,"15min":1.054},"last":0.69999999999999996}]},{"osd":7,"last update":"Mon Feb 19 15:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.70199999999999996,"15min":0.67100000000000004},"min":{"1min":0.44800000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.98199999999999998,"5min":1.115,"15min":1.115},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.75600000000000001,"15min":0.73399999999999999},"min":{"1min":0.46999999999999997,"5min":0.443,"15min":0.443},"max":{"1min":1.0169999999999999,"5min":1.24,"15min":1.24},"last":0.73999999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542202,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36484,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335356,"statfs":{"total":96636764160,"available":96599404544,"internally_reserved":0,"allocated":1163264,"data_stored":490133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42899999999999999,"15min":0.44700000000000001},"min":{"1min":0.23699999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76600000000000001,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.438},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.54100000000000004,"15min":0.64600000000000002},"min":{"1min":0.27500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67600000000000005,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.41499999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.433,"15min":0.442},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.66000000000000003,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.59699999999999998,"15min":0.70899999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.66400000000000003,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56799999999999995,"15min":0.61199999999999999},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.66500000000000004,"5min":1.008,"15min":1.008},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55000000000000004,"15min":0.58899999999999997},"min":{"1min":0.39900000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86199999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.45200000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.70299999999999996,"15min":0.67900000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.90600000000000003,"5min":1.004,"15min":1.004},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.80100000000000005,"15min":0.86099999999999999},"min":{"1min":0.58199999999999996,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0269999999999999,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.84799999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.80100000000000005,"15min":0.85499999999999998},"min":{"1min":0.54700000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.96699999999999997,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.74199999999999999,"15min":0.748},"min":{"1min":0.53300000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.93899999999999995,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.78200000000000003}]},{"osd":6,"last update":"Mon Feb 19 15:26:06 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.82499999999999996,"15min":0.89000000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92300000000000004,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.78500000000000003,"15min":0.81299999999999994},"min":{"1min":0.55400000000000005,"5min":0.502,"15min":0.502},"max":{"1min":1.0009999999999999,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.755}]},{"osd":7,"last update":"Mon Feb 19 15:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76400000000000001,"15min":0.78000000000000003},"min":{"1min":0.52700000000000002,"5min":0.504,"15min":0.504},"max":{"1min":0.96799999999999997,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.84099999999999997,"15min":0.92000000000000004},"min":{"1min":0.57399999999999995,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0009999999999999,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.77000000000000002}]}]},{"osd":2,"up_from":14,"seq":60129542203,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":1150976,"data_stored":488533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.66700000000000004,"15min":0.69699999999999995},"min":{"1min":0.27400000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.64400000000000002,"5min":15.42,"15min":15.42},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.71499999999999997,"15min":0.88500000000000001},"min":{"1min":0.34000000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.76500000000000001,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.44400000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.75700000000000001,"15min":0.89700000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58999999999999997,"5min":9.5269999999999992,"15min":9.5269999999999992},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.94199999999999995,"15min":1.006},"min":{"1min":0.39700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.621,"5min":22.954999999999998,"15min":22.954999999999998},"last":0.51300000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.67100000000000004,"15min":0.78300000000000003},"min":{"1min":0.376,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66200000000000003,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.74299999999999999,"15min":0.82299999999999995},"min":{"1min":0.41799999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69399999999999995,"5min":7.734,"15min":7.734},"last":0.53400000000000003}]},{"osd":4,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.91400000000000003,"15min":0.90300000000000002},"min":{"1min":0.52000000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.016,"5min":15.369,"15min":15.369},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.82699999999999996,"15min":0.89300000000000002},"min":{"1min":0.53800000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93700000000000006,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.66500000000000004}]},{"osd":5,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.871,"15min":0.88900000000000001},"min":{"1min":0.53900000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.95999999999999996,"5min":7.6520000000000001,"15min":7.6520000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.98699999999999999,"15min":0.97499999999999998},"min":{"1min":0.60199999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0660000000000001,"5min":15.391,"15min":15.391},"last":0.63200000000000001}]},{"osd":6,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.95799999999999996,"15min":0.94399999999999995},"min":{"1min":0.54900000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.90100000000000002,"5min":15.337,"15min":15.337},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.876,"15min":0.95799999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.98499999999999999,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.61099999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.90600000000000003,"15min":0.98099999999999998},"min":{"1min":0.58399999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.028,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.91600000000000004,"15min":0.95299999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.98099999999999998,"5min":7.6929999999999996,"15min":7.6929999999999996},"last":0.49299999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542204,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32304,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339536,"statfs":{"total":96636764160,"available":96603684864,"internally_reserved":0,"allocated":1146880,"data_stored":488499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.39200000000000002,"15min":0.42399999999999999},"min":{"1min":0.248,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56799999999999995,"5min":0.747,"15min":0.747},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.438,"15min":0.46600000000000003},"min":{"1min":0.28100000000000003,"5min":0.25,"15min":0.25},"max":{"1min":0.69599999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.47999999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.42599999999999999,"15min":0.44400000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.56100000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.48799999999999999,"15min":0.52600000000000002},"min":{"1min":0.36299999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.501,"5min":1.071,"15min":1.071},"last":0.504}]},{"osd":3,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58899999999999997,"15min":0.63300000000000001},"min":{"1min":0.44800000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.64600000000000002,"5min":1.365,"15min":1.365},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.57399999999999995,"15min":0.63},"min":{"1min":0.42999999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.61399999999999999,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.59399999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.65300000000000002,"15min":0.61299999999999999},"min":{"1min":0.52100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.99099999999999999,"5min":1.127,"15min":1.127},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.69299999999999995,"15min":0.68000000000000005},"min":{"1min":0.54900000000000004,"5min":0.435,"15min":0.435},"max":{"1min":0.98399999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.56100000000000005}]},{"osd":5,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70899999999999996,"15min":0.69699999999999995},"min":{"1min":0.54200000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.95599999999999996,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.70399999999999996,"15min":0.68500000000000005},"min":{"1min":0.503,"5min":0.443,"15min":0.443},"max":{"1min":0.94199999999999995,"5min":1.107,"15min":1.107},"last":0.52700000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.72899999999999998},"min":{"1min":0.57099999999999995,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.97399999999999998,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.74199999999999999,"15min":0.745},"min":{"1min":0.60299999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97199999999999998,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.70099999999999996}]},{"osd":7,"last update":"Mon Feb 19 15:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.76800000000000002,"15min":0.78600000000000003},"min":{"1min":0.59999999999999998,"5min":0.496,"15min":0.496},"max":{"1min":0.95299999999999996,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.76400000000000001,"15min":0.748},"min":{"1min":0.58499999999999996,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0229999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.622}]}]},{"osd":0,"up_from":14,"seq":60129542202,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37052,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334788,"statfs":{"total":96636764160,"available":96598822912,"internally_reserved":0,"allocated":1744896,"data_stored":1079449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.50700000000000001,"15min":0.628},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.66200000000000003,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.443,"5min":0.46300000000000002,"15min":0.48299999999999998},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.76000000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.34200000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.65400000000000003,"15min":0.78400000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.67500000000000004,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.61799999999999999,"15min":0.746},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69099999999999995,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.57999999999999996}]},{"osd":3,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.56699999999999995,"15min":0.67900000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.61699999999999999,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.54800000000000004,"15min":0.60999999999999999},"min":{"1min":0.38100000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.72499999999999998,"5min":1.345,"15min":1.345},"last":0.42299999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.745,"15min":0.74299999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.0700000000000001,"5min":1.095,"15min":1.095},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.84399999999999997,"15min":0.92500000000000004},"min":{"1min":0.64000000000000001,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0049999999999999,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.55500000000000005}]},{"osd":5,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.73799999999999999,"15min":0.751},"min":{"1min":0.55900000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.97999999999999998,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.77300000000000002,"15min":0.85299999999999998},"min":{"1min":0.52900000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.05,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.64800000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.82099999999999995,"15min":0.91500000000000004},"min":{"1min":0.59799999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.95899999999999996,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73799999999999999,"15min":0.75700000000000001},"min":{"1min":0.57999999999999996,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.84999999999999998,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.50800000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.85199999999999998,"15min":0.94199999999999995},"min":{"1min":0.58599999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.0229999999999999,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.83499999999999996,"15min":0.94099999999999995},"min":{"1min":0.55900000000000005,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.96499999999999997,"5min":4.915,"15min":4.915},"last":0.68600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:26:36.386 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T15:26:36.386 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-19T15:26:36.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.563+0000 7f5d8a000640 1 Processor -- start 2024-02-19T15:26:36.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.563+0000 7f5d8a000640 1 -- start start 2024-02-19T15:26:36.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.563+0000 7f5d8a000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d8415abf0 0x7f5d84071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:36.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.563+0000 7f5d8a000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5d84071050 0x7f5d8406de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:36.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.563+0000 7f5d8a000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5d8406e3a0 0x7f5d840ae490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:36.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.563+0000 7f5d8a000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d8406e7f0 con 0x7f5d8406e3a0 2024-02-19T15:26:36.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.563+0000 7f5d8a000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d840ca520 con 0x7f5d84071050 2024-02-19T15:26:36.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.563+0000 7f5d8a000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d840ca800 con 0x7f5d8415abf0 2024-02-19T15:26:36.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.563+0000 7f5d83e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5d8406e3a0 0x7f5d840ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:36.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.563+0000 7f5d83400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d8415abf0 0x7f5d84071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:36.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.563+0000 7f5d7aa00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5d84071050 0x7f5d8406de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:36.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.563+0000 7f5d83e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5d8406e3a0 0x7f5d840ae490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40666/0 (socket says 172.21.15.47:40666) 2024-02-19T15:26:36.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.563+0000 7f5d83e00640 1 -- 172.21.15.47:0/762744323 learned_addr learned my addr 172.21.15.47:0/762744323 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:36.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.563+0000 7f5d83400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d8415abf0 0x7f5d84071f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:54742/0 (socket says 172.21.15.47:54742) 2024-02-19T15:26:36.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.563+0000 7f5d7aa00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5d84071050 0x7f5d8406de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:56154/0 (socket says 172.21.15.47:56154) 2024-02-19T15:26:36.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.567+0000 7f5d83e00640 1 -- 172.21.15.47:0/762744323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d8415abf0 msgr2=0x7f5d84071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:36.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.567+0000 7f5d83e00640 1 --2- 172.21.15.47:0/762744323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d8415abf0 0x7f5d84071f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.567+0000 7f5d83e00640 1 -- 172.21.15.47:0/762744323 >> v2:172.21.15.116:3300/0 conn(0x7f5d84071050 msgr2=0x7f5d8406de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:36.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.567+0000 7f5d83e00640 1 --2- 172.21.15.47:0/762744323 >> v2:172.21.15.116:3300/0 conn(0x7f5d84071050 0x7f5d8406de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.567+0000 7f5d83e00640 1 -- 172.21.15.47:0/762744323 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d840caae0 con 0x7f5d8406e3a0 2024-02-19T15:26:36.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.567+0000 7f5d83e00640 1 --2- 172.21.15.47:0/762744323 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d8406e3a0 0x7f5d840ae490 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f5d8406c040 tx=0x7f5d7c01ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d68193c65016b766 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:36.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.567+0000 7f5d82a00640 1 -- 172.21.15.47:0/762744323 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d7c00b2d0 con 0x7f5d8406e3a0 2024-02-19T15:26:36.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.567+0000 7f5d82a00640 1 -- 172.21.15.47:0/762744323 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d7c00b430 con 0x7f5d8406e3a0 2024-02-19T15:26:36.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.567+0000 7f5d82a00640 1 -- 172.21.15.47:0/762744323 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d7c0a7c70 con 0x7f5d8406e3a0 2024-02-19T15:26:36.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.567+0000 7f5d8a000640 1 -- 172.21.15.47:0/762744323 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d8406e3a0 msgr2=0x7f5d840ae490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:36.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.567+0000 7f5d8a000640 1 --2- 172.21.15.47:0/762744323 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d8406e3a0 0x7f5d840ae490 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f5d8406c040 tx=0x7f5d7c01ebc0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.567+0000 7f5d8a000640 1 -- 172.21.15.47:0/762744323 shutdown_connections 2024-02-19T15:26:36.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.567+0000 7f5d8a000640 1 --2- 172.21.15.47:0/762744323 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d8406e3a0 0x7f5d840ae490 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.567+0000 7f5d8a000640 1 --2- 172.21.15.47:0/762744323 >> v2:172.21.15.116:3300/0 conn(0x7f5d84071050 0x7f5d8406de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.567+0000 7f5d8a000640 1 --2- 172.21.15.47:0/762744323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d8415abf0 0x7f5d84071f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.567+0000 7f5d8a000640 1 -- 172.21.15.47:0/762744323 >> 172.21.15.47:0/762744323 conn(0x7f5d840cc3f0 msgr2=0x7f5d840cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:36.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d8a000640 1 -- 172.21.15.47:0/762744323 shutdown_connections 2024-02-19T15:26:36.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d8a000640 1 -- 172.21.15.47:0/762744323 wait complete. 2024-02-19T15:26:36.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d8a000640 1 Processor -- start 2024-02-19T15:26:36.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d8a000640 1 -- start start 2024-02-19T15:26:36.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d8a000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d8406e3a0 0x7f5d840c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:36.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d8a000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d84071050 0x7f5d84148e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:36.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d8a000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d8415abf0 0x7f5d840b07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:36.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d8a000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d840c6f40 con 0x7f5d84071050 2024-02-19T15:26:36.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d8a000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d840c7220 con 0x7f5d8406e3a0 2024-02-19T15:26:36.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d8a000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d840c7500 con 0x7f5d8415abf0 2024-02-19T15:26:36.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d7aa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d84071050 0x7f5d84148e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:36.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d83400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d8406e3a0 0x7f5d840c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:36.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d83400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d8406e3a0 0x7f5d840c97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:56158/0 (socket says 172.21.15.47:56158) 2024-02-19T15:26:36.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d83400640 1 -- 172.21.15.47:0/3052941899 learned_addr learned my addr 172.21.15.47:0/3052941899 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:36.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d7aa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d84071050 0x7f5d84148e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40672/0 (socket says 172.21.15.47:40672) 2024-02-19T15:26:36.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d83e00640 1 --2- 172.21.15.47:0/3052941899 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d8415abf0 0x7f5d840b07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:36.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d7aa00640 1 -- 172.21.15.47:0/3052941899 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d8415abf0 msgr2=0x7f5d840b07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:36.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d7aa00640 1 --2- 172.21.15.47:0/3052941899 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d8415abf0 0x7f5d840b07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d7aa00640 1 -- 172.21.15.47:0/3052941899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d8406e3a0 msgr2=0x7f5d840c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:36.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d7aa00640 1 --2- 172.21.15.47:0/3052941899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d8406e3a0 0x7f5d840c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d7aa00640 1 -- 172.21.15.47:0/3052941899 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c00bd40 con 0x7f5d84071050 2024-02-19T15:26:36.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d83400640 1 --2- 172.21.15.47:0/3052941899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d8406e3a0 0x7f5d840c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:36.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d7aa00640 1 --2- 172.21.15.47:0/3052941899 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d84071050 0x7f5d84148e20 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c00e7c0 tx=0x7f5d6c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:36.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d7be00640 1 -- 172.21.15.47:0/3052941899 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c00eed0 con 0x7f5d84071050 2024-02-19T15:26:36.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d7be00640 1 -- 172.21.15.47:0/3052941899 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d6c00c940 con 0x7f5d84071050 2024-02-19T15:26:36.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.571+0000 7f5d8a000640 1 -- 172.21.15.47:0/3052941899 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d840c77e0 con 0x7f5d84071050 2024-02-19T15:26:36.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.575+0000 7f5d7be00640 1 -- 172.21.15.47:0/3052941899 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c00ccb0 con 0x7f5d84071050 2024-02-19T15:26:36.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.575+0000 7f5d8a000640 1 -- 172.21.15.47:0/3052941899 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d84149360 con 0x7f5d84071050 2024-02-19T15:26:36.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.575+0000 7f5d8a000640 1 -- 172.21.15.47:0/3052941899 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d84075360 con 0x7f5d84071050 2024-02-19T15:26:36.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.575+0000 7f5d7be00640 1 -- 172.21.15.47:0/3052941899 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5d6c013470 con 0x7f5d84071050 2024-02-19T15:26:36.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.575+0000 7f5d7be00640 1 --2- 172.21.15.47:0/3052941899 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5d60066e40 0x7f5d60069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:36.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.575+0000 7f5d7be00640 1 -- 172.21.15.47:0/3052941899 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d6c084bf0 con 0x7f5d84071050 2024-02-19T15:26:36.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.575+0000 7f5d83400640 1 --2- 172.21.15.47:0/3052941899 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5d60066e40 0x7f5d60069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:36.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.575+0000 7f5d83400640 1 --2- 172.21.15.47:0/3052941899 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5d60066e40 0x7f5d60069300 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5d74011310 tx=0x7f5d74009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:36.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.579+0000 7f5d7be00640 1 -- 172.21.15.47:0/3052941899 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f5d6c051960 con 0x7f5d84071050 2024-02-19T15:26:36.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.795+0000 7f5d8a000640 1 -- 172.21.15.47:0/3052941899 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f5d84072220 con 0x7f5d84071050 2024-02-19T15:26:36.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.799+0000 7f5d7be00640 1 -- 172.21.15.47:0/3052941899 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7f5d6c0210e0 con 0x7f5d84071050 2024-02-19T15:26:36.803 INFO:teuthology.orchestra.run.smithi047.stdout:[client.0] 2024-02-19T15:26:36.803 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDjcdNlb7eRNRAAk6ZLGfjhcdlR9GoXFCd9eA== 2024-02-19T15:26:36.803 INFO:teuthology.orchestra.run.smithi047.stdout: caps mds = "allow" 2024-02-19T15:26:36.803 INFO:teuthology.orchestra.run.smithi047.stdout: caps mon = "allow r" 2024-02-19T15:26:36.803 INFO:teuthology.orchestra.run.smithi047.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-19T15:26:36.803 INFO:teuthology.orchestra.run.smithi047.stderr:updated caps for client.0 2024-02-19T15:26:36.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.799+0000 7f5d8a000640 1 -- 172.21.15.47:0/3052941899 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5d60066e40 msgr2=0x7f5d60069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:36.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.799+0000 7f5d8a000640 1 --2- 172.21.15.47:0/3052941899 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5d60066e40 0x7f5d60069300 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5d74011310 tx=0x7f5d74009190 comp rx=0 tx=0).stop 2024-02-19T15:26:36.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.799+0000 7f5d8a000640 1 -- 172.21.15.47:0/3052941899 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d84071050 msgr2=0x7f5d84148e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:36.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.799+0000 7f5d8a000640 1 --2- 172.21.15.47:0/3052941899 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d84071050 0x7f5d84148e20 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c00e7c0 tx=0x7f5d6c00ec80 comp rx=0 tx=0).stop 2024-02-19T15:26:36.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.799+0000 7f5d8a000640 1 -- 172.21.15.47:0/3052941899 shutdown_connections 2024-02-19T15:26:36.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.799+0000 7f5d8a000640 1 --2- 172.21.15.47:0/3052941899 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5d60066e40 0x7f5d60069300 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.799+0000 7f5d8a000640 1 --2- 172.21.15.47:0/3052941899 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d8415abf0 0x7f5d840b07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.799+0000 7f5d8a000640 1 --2- 172.21.15.47:0/3052941899 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d84071050 0x7f5d84148e20 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.799+0000 7f5d8a000640 1 --2- 172.21.15.47:0/3052941899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d8406e3a0 0x7f5d840c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.799+0000 7f5d8a000640 1 -- 172.21.15.47:0/3052941899 >> 172.21.15.47:0/3052941899 conn(0x7f5d840cc3f0 msgr2=0x7f5d840af450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:36.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.799+0000 7f5d8a000640 1 -- 172.21.15.47:0/3052941899 shutdown_connections 2024-02-19T15:26:36.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.799+0000 7f5d8a000640 1 -- 172.21.15.47:0/3052941899 wait complete. 2024-02-19T15:26:36.825 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:26:36.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.959+0000 7fa18be00640 1 Processor -- start 2024-02-19T15:26:36.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.959+0000 7fa18be00640 1 -- start start 2024-02-19T15:26:36.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.959+0000 7fa18be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa1840cf850 0x7fa1840cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:36.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.959+0000 7fa18be00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1840cd6c0 con 0x7fa18415c0d0 2024-02-19T15:26:36.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.959+0000 7fa18be00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1840b0fe0 con 0x7fa1840d0540 2024-02-19T15:26:36.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.959+0000 7fa18be00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa18415c4b0 con 0x7fa1840cf850 2024-02-19T15:26:36.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.959+0000 7fa189400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa18415c0d0 0x7fa1840cfe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39798/0 (socket says 172.21.15.47:39798) 2024-02-19T15:26:36.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.959+0000 7fa189400640 1 -- 172.21.15.47:0/3403937114 learned_addr learned my addr 172.21.15.47:0/3403937114 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:36.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.959+0000 7fa189e00640 1 --2- 172.21.15.47:0/3403937114 >> v2:172.21.15.47:3300/0 conn(0x7fa1840cf850 0x7fa1840cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:36.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.963+0000 7fa183e00640 1 -- 172.21.15.47:0/3403937114 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810160308 0 0) 0x7fa1840cd6c0 con 0x7fa18415c0d0 2024-02-19T15:26:36.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.963+0000 7fa183e00640 1 -- 172.21.15.47:0/3403937114 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa16c003530 con 0x7fa18415c0d0 2024-02-19T15:26:36.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.963+0000 7fa183e00640 1 -- 172.21.15.47:0/3403937114 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395469691 0 0) 0x7fa1840b0fe0 con 0x7fa1840d0540 2024-02-19T15:26:36.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.963+0000 7fa183e00640 1 -- 172.21.15.47:0/3403937114 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1840cd6c0 con 0x7fa1840d0540 2024-02-19T15:26:36.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.963+0000 7fa189e00640 1 -- 172.21.15.47:0/3403937114 >> v1:172.21.15.116:6789/0 conn(0x7fa1840d0540 legacy=0x7fa1840b1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:36.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.963+0000 7fa189e00640 1 -- 172.21.15.47:0/3403937114 >> v1:172.21.15.47:6789/0 conn(0x7fa18415c0d0 legacy=0x7fa1840cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:36.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.963+0000 7fa189e00640 1 -- 172.21.15.47:0/3403937114 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1840b1880 con 0x7fa1840cf850 2024-02-19T15:26:36.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.963+0000 7fa189e00640 1 --2- 172.21.15.47:0/3403937114 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1840cf850 0x7fa1840cd180 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fa17c009d90 tx=0x7fa17c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26e78f17bd644b5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:36.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.963+0000 7fa183e00640 1 -- 172.21.15.47:0/3403937114 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c0a5920 con 0x7fa1840cf850 2024-02-19T15:26:36.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.963+0000 7fa183e00640 1 -- 172.21.15.47:0/3403937114 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa17c0a5a80 con 0x7fa1840cf850 2024-02-19T15:26:36.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.963+0000 7fa183e00640 1 -- 172.21.15.47:0/3403937114 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c0a5e10 con 0x7fa1840cf850 2024-02-19T15:26:36.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.963+0000 7fa18be00640 1 -- 172.21.15.47:0/3403937114 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1840cf850 msgr2=0x7fa1840cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:36.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.963+0000 7fa18be00640 1 --2- 172.21.15.47:0/3403937114 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1840cf850 0x7fa1840cd180 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fa17c009d90 tx=0x7fa17c01c970 comp rx=0 tx=0).stop 2024-02-19T15:26:36.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa18be00640 1 -- 172.21.15.47:0/3403937114 shutdown_connections 2024-02-19T15:26:36.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa18be00640 1 --2- 172.21.15.47:0/3403937114 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1840cf850 0x7fa1840cd180 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa18be00640 1 -- 172.21.15.47:0/3403937114 >> 172.21.15.47:0/3403937114 conn(0x7fa1840d1ba0 msgr2=0x7fa1840d1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:36.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa18be00640 1 -- 172.21.15.47:0/3403937114 shutdown_connections 2024-02-19T15:26:36.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa18be00640 1 -- 172.21.15.47:0/3403937114 wait complete. 2024-02-19T15:26:36.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa18be00640 1 Processor -- start 2024-02-19T15:26:36.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa18be00640 1 -- start start 2024-02-19T15:26:36.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa18be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1840cf850 0x7fa184155060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:36.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa18be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1840d0540 0x7fa184155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:36.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa18be00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa18415c0d0 0x7fa184156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:36.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa18be00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa184157730 con 0x7fa1840cf850 2024-02-19T15:26:36.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa18be00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa184157a10 con 0x7fa1840d0540 2024-02-19T15:26:36.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa18be00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa184157cf0 con 0x7fa18415c0d0 2024-02-19T15:26:36.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa189400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1840cf850 0x7fa184155060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:36.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa189400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1840cf850 0x7fa184155060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40690/0 (socket says 172.21.15.47:40690) 2024-02-19T15:26:36.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa189400640 1 -- 172.21.15.47:0/1574492623 learned_addr learned my addr 172.21.15.47:0/1574492623 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:36.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa188a00640 1 --2- 172.21.15.47:0/1574492623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1840d0540 0x7fa184155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:36.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa189e00640 1 --2- 172.21.15.47:0/1574492623 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa18415c0d0 0x7fa184156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:36.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa189400640 1 -- 172.21.15.47:0/1574492623 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa18415c0d0 msgr2=0x7fa184156510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:36.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa189400640 1 --2- 172.21.15.47:0/1574492623 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa18415c0d0 0x7fa184156510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa189400640 1 -- 172.21.15.47:0/1574492623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1840d0540 msgr2=0x7fa184155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:36.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa189400640 1 --2- 172.21.15.47:0/1574492623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1840d0540 0x7fa184155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:36.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa189400640 1 -- 172.21.15.47:0/1574492623 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c009a70 con 0x7fa1840cf850 2024-02-19T15:26:36.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.967+0000 7fa189400640 1 --2- 172.21.15.47:0/1574492623 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1840cf850 0x7fa184155060 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fa17800c4d0 tx=0x7fa17800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:36.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.971+0000 7fa181600640 1 -- 172.21.15.47:0/1574492623 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa178005cf0 con 0x7fa1840cf850 2024-02-19T15:26:36.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.971+0000 7fa181600640 1 -- 172.21.15.47:0/1574492623 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa178005e50 con 0x7fa1840cf850 2024-02-19T15:26:36.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.971+0000 7fa181600640 1 -- 172.21.15.47:0/1574492623 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa178007270 con 0x7fa1840cf850 2024-02-19T15:26:36.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.971+0000 7fa18be00640 1 -- 172.21.15.47:0/1574492623 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa184157fd0 con 0x7fa1840cf850 2024-02-19T15:26:36.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.971+0000 7fa18be00640 1 -- 172.21.15.47:0/1574492623 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1841555a0 con 0x7fa1840cf850 2024-02-19T15:26:36.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.971+0000 7fa18be00640 1 -- 172.21.15.47:0/1574492623 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa184068500 con 0x7fa1840cf850 2024-02-19T15:26:36.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.971+0000 7fa181600640 1 -- 172.21.15.47:0/1574492623 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa1780073d0 con 0x7fa1840cf850 2024-02-19T15:26:36.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.971+0000 7fa181600640 1 --2- 172.21.15.47:0/1574492623 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa164066e40 0x7fa164069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:36.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.971+0000 7fa181600640 1 -- 172.21.15.47:0/1574492623 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa178083f10 con 0x7fa1840cf850 2024-02-19T15:26:36.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.971+0000 7fa188a00640 1 --2- 172.21.15.47:0/1574492623 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa164066e40 0x7fa164069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:36.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.975+0000 7fa188a00640 1 --2- 172.21.15.47:0/1574492623 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa164066e40 0x7fa164069300 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa184159b50 tx=0x7fa170009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:36.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:36.975+0000 7fa181600640 1 -- 172.21.15.47:0/1574492623 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa178050da0 con 0x7fa1840cf850 2024-02-19T15:26:37.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.171+0000 7fa18be00640 1 -- 172.21.15.47:0/1574492623 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1841584f0 con 0x7fa1840cf850 2024-02-19T15:26:37.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.171+0000 7fa181600640 1 -- 172.21.15.47:0/1574492623 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4934 (secure 0 0 0) 0x7fa1841584f0 con 0x7fa1840cf850 2024-02-19T15:26:37.176 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:37.176 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/2831452879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2831452879},{"type":"v1","addr":"172.21.15.116:6836","nonce":2831452879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6191,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/946400730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":946400730},{"type":"v1","addr":"172.21.15.47:6835","nonce":946400730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":6844,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/68555355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":68555355},{"type":"v1","addr":"172.21.15.47:6833","nonce":68555355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:26:26.721109+0000","modified":"2024-02-19T15:26:29.735209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6850":{"gid":6850,"name":"d","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6837/547176070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":547176070},{"type":"v1","addr":"172.21.15.116:6837","nonce":547176070}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T15:26:37.177 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 58 2024-02-19T15:26:37.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.171+0000 7fa18be00640 1 -- 172.21.15.47:0/1574492623 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa164066e40 msgr2=0x7fa164069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:37.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.171+0000 7fa18be00640 1 --2- 172.21.15.47:0/1574492623 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa164066e40 0x7fa164069300 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa184159b50 tx=0x7fa170009040 comp rx=0 tx=0).stop 2024-02-19T15:26:37.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.171+0000 7fa18be00640 1 -- 172.21.15.47:0/1574492623 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1840cf850 msgr2=0x7fa184155060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:37.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.171+0000 7fa18be00640 1 --2- 172.21.15.47:0/1574492623 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1840cf850 0x7fa184155060 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fa17800c4d0 tx=0x7fa17800c990 comp rx=0 tx=0).stop 2024-02-19T15:26:37.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.171+0000 7fa18be00640 1 -- 172.21.15.47:0/1574492623 shutdown_connections 2024-02-19T15:26:37.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.171+0000 7fa18be00640 1 --2- 172.21.15.47:0/1574492623 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa164066e40 0x7fa164069300 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.171+0000 7fa18be00640 1 --2- 172.21.15.47:0/1574492623 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa18415c0d0 0x7fa184156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.171+0000 7fa18be00640 1 --2- 172.21.15.47:0/1574492623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1840d0540 0x7fa184155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.171+0000 7fa18be00640 1 --2- 172.21.15.47:0/1574492623 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1840cf850 0x7fa184155060 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.171+0000 7fa18be00640 1 -- 172.21.15.47:0/1574492623 >> 172.21.15.47:0/1574492623 conn(0x7fa1840d1ba0 msgr2=0x7fa1841591d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:37.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.171+0000 7fa18be00640 1 -- 172.21.15.47:0/1574492623 shutdown_connections 2024-02-19T15:26:37.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.171+0000 7fa18be00640 1 -- 172.21.15.47:0/1574492623 wait complete. 2024-02-19T15:26:37.600 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:26:26.721109+0000', 'modified': '2024-02-19T15:26:29.735209+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6850}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6850': {'gid': 6850, 'name': 'd', 'rank': 0, 'incarnation': 54, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.116:6837/547176070', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 547176070}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 547176070}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:26:37.600 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T15:26:37.600 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-19T15:26:37.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.739+0000 7f9de3400640 1 Processor -- start 2024-02-19T15:26:37.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.739+0000 7f9de3400640 1 -- start start 2024-02-19T15:26:37.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.743+0000 7f9de3400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9ddc166b60 0x7f9ddc168f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.743+0000 7f9de3400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ddc0ad840 con 0x7f9ddc0cb520 2024-02-19T15:26:37.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.743+0000 7f9de3400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ddc0c62f0 con 0x7f9ddc06aeb0 2024-02-19T15:26:37.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.743+0000 7f9de3400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ddc0cc810 con 0x7f9ddc166b60 2024-02-19T15:26:37.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.743+0000 7f9de1400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9ddc166b60 0x7f9ddc168f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.743+0000 7f9de0a00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9ddc06aeb0 0x7f9ddc06b290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:40158/0 (socket says 172.21.15.47:40158) 2024-02-19T15:26:37.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.743+0000 7f9ddbe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f9ddc0cb520 0x7f9ddc0cb9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39812/0 (socket says 172.21.15.47:39812) 2024-02-19T15:26:37.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.743+0000 7f9de0a00640 1 -- 172.21.15.47:0/924552365 learned_addr learned my addr 172.21.15.47:0/924552365 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:37.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.743+0000 7f9de1400640 1 -- 172.21.15.47:0/924552365 >> v1:172.21.15.116:6789/0 conn(0x7f9ddc06aeb0 legacy=0x7f9ddc06b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:37.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.743+0000 7f9ddb400640 1 -- 172.21.15.47:0/924552365 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668713357 0 0) 0x7f9ddc0c62f0 con 0x7f9ddc06aeb0 2024-02-19T15:26:37.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.743+0000 7f9de1400640 1 -- 172.21.15.47:0/924552365 >> v1:172.21.15.47:6789/0 conn(0x7f9ddc0cb520 legacy=0x7f9ddc0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:37.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.743+0000 7f9de1400640 1 -- 172.21.15.47:0/924552365 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ddc0cc250 con 0x7f9ddc166b60 2024-02-19T15:26:37.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9de1400640 1 --2- 172.21.15.47:0/924552365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ddc166b60 0x7f9ddc168f50 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4009d90 tx=0x7f9dd401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f3e887d27ed673 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:37.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9ddb400640 1 -- 172.21.15.47:0/924552365 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd40a5a00 con 0x7f9ddc166b60 2024-02-19T15:26:37.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9ddb400640 1 -- 172.21.15.47:0/924552365 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd40a5b60 con 0x7f9ddc166b60 2024-02-19T15:26:37.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9ddb400640 1 -- 172.21.15.47:0/924552365 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd40ad440 con 0x7f9ddc166b60 2024-02-19T15:26:37.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9de3400640 1 -- 172.21.15.47:0/924552365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ddc166b60 msgr2=0x7f9ddc168f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:37.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9de3400640 1 --2- 172.21.15.47:0/924552365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ddc166b60 0x7f9ddc168f50 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4009d90 tx=0x7f9dd401c940 comp rx=0 tx=0).stop 2024-02-19T15:26:37.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9de3400640 1 -- 172.21.15.47:0/924552365 shutdown_connections 2024-02-19T15:26:37.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9de3400640 1 --2- 172.21.15.47:0/924552365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ddc166b60 0x7f9ddc168f50 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9de3400640 1 -- 172.21.15.47:0/924552365 >> 172.21.15.47:0/924552365 conn(0x7f9ddc0d0ad0 msgr2=0x7f9ddc0d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:37.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9de3400640 1 -- 172.21.15.47:0/924552365 shutdown_connections 2024-02-19T15:26:37.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9de3400640 1 -- 172.21.15.47:0/924552365 wait complete. 2024-02-19T15:26:37.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9de3400640 1 Processor -- start 2024-02-19T15:26:37.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9de3400640 1 -- start start 2024-02-19T15:26:37.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9de3400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ddc06aeb0 0x7f9ddc0ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9de3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ddc0cb520 0x7f9ddc153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9de3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ddc166b60 0x7f9ddc15e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9de3400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc0cab00 con 0x7f9ddc0cb520 2024-02-19T15:26:37.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9de3400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc1547d0 con 0x7f9ddc166b60 2024-02-19T15:26:37.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9de3400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc154ab0 con 0x7f9ddc06aeb0 2024-02-19T15:26:37.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9ddbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ddc0cb520 0x7f9ddc153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9ddbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ddc0cb520 0x7f9ddc153350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40714/0 (socket says 172.21.15.47:40714) 2024-02-19T15:26:37.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9ddbe00640 1 -- 172.21.15.47:0/1903528744 learned_addr learned my addr 172.21.15.47:0/1903528744 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:37.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9de0a00640 1 --2- 172.21.15.47:0/1903528744 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ddc06aeb0 0x7f9ddc0ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.747+0000 7f9de1400640 1 --2- 172.21.15.47:0/1903528744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ddc166b60 0x7f9ddc15e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.751+0000 7f9ddbe00640 1 -- 172.21.15.47:0/1903528744 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ddc06aeb0 msgr2=0x7f9ddc0ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:37.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.751+0000 7f9ddbe00640 1 --2- 172.21.15.47:0/1903528744 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ddc06aeb0 0x7f9ddc0ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.751+0000 7f9ddbe00640 1 -- 172.21.15.47:0/1903528744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ddc166b60 msgr2=0x7f9ddc15e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:37.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.751+0000 7f9ddbe00640 1 --2- 172.21.15.47:0/1903528744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ddc166b60 0x7f9ddc15e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.751+0000 7f9ddbe00640 1 -- 172.21.15.47:0/1903528744 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd4009a70 con 0x7f9ddc0cb520 2024-02-19T15:26:37.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.751+0000 7f9de1400640 1 --2- 172.21.15.47:0/1903528744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ddc166b60 0x7f9ddc15e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:37.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.751+0000 7f9ddbe00640 1 --2- 172.21.15.47:0/1903528744 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ddc0cb520 0x7f9ddc153350 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f9dc800c490 tx=0x7f9dc800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:37.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.751+0000 7f9dd8c00640 1 -- 172.21.15.47:0/1903528744 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8007480 con 0x7f9ddc0cb520 2024-02-19T15:26:37.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.751+0000 7f9dd8c00640 1 -- 172.21.15.47:0/1903528744 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc80075e0 con 0x7f9ddc0cb520 2024-02-19T15:26:37.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.751+0000 7f9de3400640 1 -- 172.21.15.47:0/1903528744 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ddc154d90 con 0x7f9ddc0cb520 2024-02-19T15:26:37.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.751+0000 7f9dd8c00640 1 -- 172.21.15.47:0/1903528744 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8004a10 con 0x7f9ddc0cb520 2024-02-19T15:26:37.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.751+0000 7f9de3400640 1 -- 172.21.15.47:0/1903528744 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ddc0ca7f0 con 0x7f9ddc0cb520 2024-02-19T15:26:37.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.751+0000 7f9dd8c00640 1 -- 172.21.15.47:0/1903528744 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9dc8004b70 con 0x7f9ddc0cb520 2024-02-19T15:26:37.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9dd8c00640 1 --2- 172.21.15.47:0/1903528744 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9dbc066e70 0x7f9dbc069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9dd8c00640 1 -- 172.21.15.47:0/1903528744 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9dc8083e80 con 0x7f9ddc0cb520 2024-02-19T15:26:37.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de68b9000 1 Processor -- start 2024-02-19T15:26:37.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de0a00640 1 --2- 172.21.15.47:0/1903528744 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9dbc066e70 0x7f9dbc069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de68b9000 1 -- start start 2024-02-19T15:26:37.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de0a00640 1 --2- 172.21.15.47:0/1903528744 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9dbc066e70 0x7f9dbc069330 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0007a90 tx=0x7f9dd00079e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:37.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de68b9000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed05f920 0x5624ed05fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de68b9000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed060240 0x5624ed0159f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de68b9000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed015f30 0x5624ed016310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de68b9000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624eceddd00 con 0x5624ed015f30 2024-02-19T15:26:37.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de68b9000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ece6f430 con 0x5624ed060240 2024-02-19T15:26:37.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de68b9000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x5624ece6c960 con 0x5624ed05f920 2024-02-19T15:26:37.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de1400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed015f30 0x5624ed016310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de0a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed05f920 0x5624ed05fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9ddbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed060240 0x5624ed0159f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de1400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed015f30 0x5624ed016310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40718/0 (socket says 172.21.15.47:40718) 2024-02-19T15:26:37.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de1400640 1 -- 172.21.15.47:0/3757790942 learned_addr learned my addr 172.21.15.47:0/3757790942 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:37.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de1400640 1 -- 172.21.15.47:0/3757790942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed05f920 msgr2=0x5624ed05fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:37.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de1400640 1 --2- 172.21.15.47:0/3757790942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed05f920 0x5624ed05fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de1400640 1 -- 172.21.15.47:0/3757790942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed060240 msgr2=0x5624ed0159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:37.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de1400640 1 --2- 172.21.15.47:0/3757790942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed060240 0x5624ed0159f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de1400640 1 -- 172.21.15.47:0/3757790942 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624ecb5e890 con 0x5624ed015f30 2024-02-19T15:26:37.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de1400640 1 --2- 172.21.15.47:0/3757790942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed015f30 0x5624ed016310 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4019550 tx=0x7f9dd4004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c757e9699bddd1f6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:37.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9dcd600640 1 -- 172.21.15.47:0/3757790942 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd40ada70 con 0x5624ed015f30 2024-02-19T15:26:37.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9dcd600640 1 -- 172.21.15.47:0/3757790942 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd40bc070 con 0x5624ed015f30 2024-02-19T15:26:37.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9dcd600640 1 -- 172.21.15.47:0/3757790942 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd40addd0 con 0x5624ed015f30 2024-02-19T15:26:37.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de68b9000 1 -- 172.21.15.47:0/3757790942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed015f30 msgr2=0x5624ed016310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:37.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de68b9000 1 --2- 172.21.15.47:0/3757790942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed015f30 0x5624ed016310 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4019550 tx=0x7f9dd4004ac0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de68b9000 1 -- 172.21.15.47:0/3757790942 shutdown_connections 2024-02-19T15:26:37.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de68b9000 1 --2- 172.21.15.47:0/3757790942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed015f30 0x5624ed016310 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de68b9000 1 --2- 172.21.15.47:0/3757790942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed060240 0x5624ed0159f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de68b9000 1 --2- 172.21.15.47:0/3757790942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed05f920 0x5624ed05fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de68b9000 1 -- 172.21.15.47:0/3757790942 >> 172.21.15.47:0/3757790942 conn(0x5624ed01c2a0 msgr2=0x5624ed0142d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:37.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de68b9000 1 -- 172.21.15.47:0/3757790942 shutdown_connections 2024-02-19T15:26:37.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.755+0000 7f9de68b9000 1 -- 172.21.15.47:0/3757790942 wait complete. 2024-02-19T15:26:37.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de68b9000 1 -- start start 2024-02-19T15:26:37.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de68b9000 1 Processor -- start 2024-02-19T15:26:37.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de68b9000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed015f30 0x5624ed01a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de68b9000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed05f920 0x5624ed060830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de68b9000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed060240 0x5624ed060d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de68b9000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ece68140 con 0x5624ed05f920 2024-02-19T15:26:37.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de68b9000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ece02630 con 0x5624ed060240 2024-02-19T15:26:37.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de68b9000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x5624ece022d0 con 0x5624ed015f30 2024-02-19T15:26:37.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de68b9000 10 client.0 _ll_register_callbacks cb 0x5624ecf181c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:26:37.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9ddbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed05f920 0x5624ed060830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de0a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed015f30 0x5624ed01a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de0a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed015f30 0x5624ed01a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:54798/0 (socket says 172.21.15.47:54798) 2024-02-19T15:26:37.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de0a00640 1 -- 172.21.15.47:0/3015547125 learned_addr learned my addr 172.21.15.47:0/3015547125 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:37.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de1400640 1 --2- 172.21.15.47:0/3015547125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed060240 0x5624ed060d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9ddbe00640 1 -- 172.21.15.47:0/3015547125 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed015f30 msgr2=0x5624ed01a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:37.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9ddbe00640 1 --2- 172.21.15.47:0/3015547125 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed015f30 0x5624ed01a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9ddbe00640 1 -- 172.21.15.47:0/3015547125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed060240 msgr2=0x5624ed060d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:37.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9ddbe00640 1 --2- 172.21.15.47:0/3015547125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed060240 0x5624ed060d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9ddbe00640 1 -- 172.21.15.47:0/3015547125 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ddc0cc250 con 0x5624ed05f920 2024-02-19T15:26:37.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de1400640 1 --2- 172.21.15.47:0/3015547125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed060240 0x5624ed060d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:37.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de0a00640 1 --2- 172.21.15.47:0/3015547125 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed015f30 0x5624ed01a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:37.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9ddbe00640 1 --2- 172.21.15.47:0/3015547125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed05f920 0x5624ed060830 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f9dc800c460 tx=0x7f9dc8058bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebbaa0a8a42c6e24 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:37.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9dc2000640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:26:37.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9dc2000640 1 -- 172.21.15.47:0/3015547125 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc806c1c0 con 0x5624ed05f920 2024-02-19T15:26:37.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de68b9000 1 -- 172.21.15.47:0/3015547125 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5624ece1b850 con 0x5624ed05f920 2024-02-19T15:26:37.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9dc2000640 1 -- 172.21.15.47:0/3015547125 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9dc806c320 con 0x5624ed05f920 2024-02-19T15:26:37.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9dc2000640 1 -- 172.21.15.47:0/3015547125 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9dc806c4e0 con 0x5624ed05f920 2024-02-19T15:26:37.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de68b9000 10 client.7284 fetch_fsmap learned FSMap version 58 2024-02-19T15:26:37.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de68b9000 1 -- 172.21.15.47:0/3015547125 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=2+}) v3 -- 0x5624ece27e70 con 0x5624ed05f920 2024-02-19T15:26:37.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9dc2000640 1 -- 172.21.15.47:0/3015547125 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc8070080 con 0x5624ed05f920 2024-02-19T15:26:37.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9dc2000640 1 -- 172.21.15.47:0/3015547125 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 58) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9dc8053e30 con 0x5624ed05f920 2024-02-19T15:26:37.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de68b9000 10 client.7284 fetch_fsmap finished waiting for FSMap version 58 2024-02-19T15:26:37.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de68b9000 10 client.7284 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T15:26:37.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de68b9000 20 client.7284 populate_metadata read hostname 'smithi047' 2024-02-19T15:26:37.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de68b9000 1 --2- 172.21.15.47:0/3015547125 >> [v2:172.21.15.116:6835/547176070,v1:172.21.15.116:6837/547176070] conn(0x5624ed01ac90 0x5624ed070bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de68b9000 4 client.7284 mds_command: new command op to 6850 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T15:26:37.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de68b9000 1 -- 172.21.15.47:0/3015547125 --> [v2:172.21.15.116:6835/547176070,v1:172.21.15.116:6837/547176070] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5624ece64f20 con 0x5624ed01ac90 2024-02-19T15:26:37.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.759+0000 7f9de0a00640 1 --2- 172.21.15.47:0/3015547125 >> [v2:172.21.15.116:6835/547176070,v1:172.21.15.116:6837/547176070] conn(0x5624ed01ac90 0x5624ed070bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.763+0000 7f9de0a00640 1 --2- 172.21.15.47:0/3015547125 >> [v2:172.21.15.116:6835/547176070,v1:172.21.15.116:6837/547176070] conn(0x5624ed01ac90 0x5624ed070bd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ea7dfe4641a5f4d server_cookie=9b2930256d8d860a in_seq=0 out_seq=0 2024-02-19T15:26:37.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.763+0000 7f9dc2000640 10 client.7284 ms_handle_connect on v2:172.21.15.116:6835/547176070 2024-02-19T15:26:37.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.763+0000 7f9dc2000640 1 -- 172.21.15.47:0/3015547125 <== mds.0 v2:172.21.15.116:6835/547176070 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5624ece64f20 con 0x5624ed01ac90 2024-02-19T15:26:37.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.763+0000 7f9dc2000640 10 client.7284 handle_command_reply: tid=0 2024-02-19T15:26:37.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.763+0000 7f9de68b9000 1 client.7284 shutdown 2024-02-19T15:26:37.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.763+0000 7f9de68b9000 1 -- 172.21.15.47:0/3015547125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed05f920 msgr2=0x5624ed060830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:37.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.763+0000 7f9de68b9000 1 --2- 172.21.15.47:0/3015547125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed05f920 0x5624ed060830 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f9dc800c460 tx=0x7f9dc8058bc0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.763+0000 7f9de68b9000 1 -- 172.21.15.47:0/3015547125 shutdown_connections 2024-02-19T15:26:37.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.763+0000 7f9de68b9000 1 --2- 172.21.15.47:0/3015547125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed060240 0x5624ed060d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.763+0000 7f9de68b9000 1 --2- 172.21.15.47:0/3015547125 >> [v2:172.21.15.116:6835/547176070,v1:172.21.15.116:6837/547176070] conn(0x5624ed01ac90 0x5624ed070bd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.763+0000 7f9de68b9000 1 --2- 172.21.15.47:0/3015547125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed05f920 0x5624ed060830 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.763+0000 7f9de68b9000 1 --2- 172.21.15.47:0/3015547125 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed015f30 0x5624ed01a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.763+0000 7f9de68b9000 1 -- 172.21.15.47:0/3015547125 >> 172.21.15.47:0/3015547125 conn(0x5624ed01c2a0 msgr2=0x5624ed0149e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:37.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.763+0000 7f9dc2000640 0 client.7284 ms_handle_reset on v2:172.21.15.116:6835/547176070 2024-02-19T15:26:37.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.763+0000 7f9de68b9000 1 -- 172.21.15.47:0/3015547125 shutdown_connections 2024-02-19T15:26:37.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.763+0000 7f9de68b9000 1 -- 172.21.15.47:0/3015547125 wait complete. 2024-02-19T15:26:37.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.763+0000 7f9de68b9000 20 client.7284 trim_cache size 0 max 16384 2024-02-19T15:26:37.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.779+0000 7f9de68b9000 1 Processor -- start 2024-02-19T15:26:37.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.779+0000 7f9de68b9000 1 -- start start 2024-02-19T15:26:37.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.779+0000 7f9de68b9000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed015f30 0x5624ed0731f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.779+0000 7f9de68b9000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed05f920 0x5624ed073730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.779+0000 7f9de68b9000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed060240 0x5624ed0842a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.779+0000 7f9de68b9000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ecb1f940 con 0x5624ed05f920 2024-02-19T15:26:37.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.779+0000 7f9de68b9000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ece2d2b0 con 0x5624ed015f30 2024-02-19T15:26:37.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.779+0000 7f9de68b9000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x5624ece07f10 con 0x5624ed060240 2024-02-19T15:26:37.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.779+0000 7f9ddbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed05f920 0x5624ed073730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.779+0000 7f9ddbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed05f920 0x5624ed073730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40732/0 (socket says 172.21.15.47:40732) 2024-02-19T15:26:37.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.779+0000 7f9ddbe00640 1 -- 172.21.15.47:0/3057754550 learned_addr learned my addr 172.21.15.47:0/3057754550 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:37.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.779+0000 7f9de0a00640 1 --2- 172.21.15.47:0/3057754550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed015f30 0x5624ed0731f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.779+0000 7f9de1400640 1 --2- 172.21.15.47:0/3057754550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed060240 0x5624ed0842a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.779+0000 7f9ddbe00640 1 -- 172.21.15.47:0/3057754550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed060240 msgr2=0x5624ed0842a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:37.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.779+0000 7f9ddbe00640 1 --2- 172.21.15.47:0/3057754550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed060240 0x5624ed0842a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.779+0000 7f9ddbe00640 1 -- 172.21.15.47:0/3057754550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed015f30 msgr2=0x5624ed0731f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:37.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.779+0000 7f9ddbe00640 1 --2- 172.21.15.47:0/3057754550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed015f30 0x5624ed0731f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.779+0000 7f9ddbe00640 1 -- 172.21.15.47:0/3057754550 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624ece0de10 con 0x5624ed05f920 2024-02-19T15:26:37.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9ddbe00640 1 --2- 172.21.15.47:0/3057754550 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed05f920 0x5624ed073730 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8052c90 tx=0x7f9dc8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=565326bd80dd8a6c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:37.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9dccc00640 1 -- 172.21.15.47:0/3057754550 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc800e040 con 0x5624ed05f920 2024-02-19T15:26:37.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9dccc00640 1 -- 172.21.15.47:0/3057754550 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5624ece1b850 con 0x5624ed05f920 2024-02-19T15:26:37.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9dccc00640 1 -- 172.21.15.47:0/3057754550 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc80549e0 con 0x5624ed05f920 2024-02-19T15:26:37.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de68b9000 1 -- 172.21.15.47:0/3057754550 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed05f920 msgr2=0x5624ed073730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:37.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de68b9000 1 --2- 172.21.15.47:0/3057754550 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed05f920 0x5624ed073730 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8052c90 tx=0x7f9dc8013000 comp rx=0 tx=0).stop 2024-02-19T15:26:37.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de68b9000 1 -- 172.21.15.47:0/3057754550 shutdown_connections 2024-02-19T15:26:37.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de68b9000 1 --2- 172.21.15.47:0/3057754550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed060240 0x5624ed0842a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de68b9000 1 --2- 172.21.15.47:0/3057754550 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed05f920 0x5624ed073730 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de68b9000 1 --2- 172.21.15.47:0/3057754550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed015f30 0x5624ed0731f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de68b9000 1 -- 172.21.15.47:0/3057754550 >> 172.21.15.47:0/3057754550 conn(0x5624ed01c2a0 msgr2=0x5624ed071ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:37.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de68b9000 1 -- 172.21.15.47:0/3057754550 shutdown_connections 2024-02-19T15:26:37.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de68b9000 1 -- 172.21.15.47:0/3057754550 wait complete. 2024-02-19T15:26:37.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de68b9000 1 -- start start 2024-02-19T15:26:37.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de68b9000 1 Processor -- start 2024-02-19T15:26:37.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de68b9000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed015f30 0x5624ed0b1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de68b9000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed05f920 0x5624ed0b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de68b9000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed060240 0x5624ed0b2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de68b9000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ece66fd0 con 0x5624ed015f30 2024-02-19T15:26:37.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de68b9000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ece6aca0 con 0x5624ed05f920 2024-02-19T15:26:37.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de68b9000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x5624ed060a80 con 0x5624ed060240 2024-02-19T15:26:37.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de68b9000 10 client.0 _ll_register_callbacks cb 0x7f9dc80545f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:26:37.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de0a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed015f30 0x5624ed0b1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de0a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed015f30 0x5624ed0b1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40746/0 (socket says 172.21.15.47:40746) 2024-02-19T15:26:37.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de0a00640 1 -- 172.21.15.47:0/1362407212 learned_addr learned my addr 172.21.15.47:0/1362407212 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:37.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de1400640 1 --2- 172.21.15.47:0/1362407212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed060240 0x5624ed0b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9ddbe00640 1 --2- 172.21.15.47:0/1362407212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed05f920 0x5624ed0b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de0a00640 1 -- 172.21.15.47:0/1362407212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed060240 msgr2=0x5624ed0b2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:37.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de0a00640 1 --2- 172.21.15.47:0/1362407212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed060240 0x5624ed0b2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de0a00640 1 -- 172.21.15.47:0/1362407212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed05f920 msgr2=0x5624ed0b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:37.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de0a00640 1 --2- 172.21.15.47:0/1362407212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed05f920 0x5624ed0b1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de0a00640 1 -- 172.21.15.47:0/1362407212 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5624ece27e70 con 0x5624ed015f30 2024-02-19T15:26:37.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9de0a00640 1 --2- 172.21.15.47:0/1362407212 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed015f30 0x5624ed0b1130 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x5624ece69680 tx=0x7f9dd0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b57b086deae38124 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:37.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9dc2000640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:26:37.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.783+0000 7f9dc2000640 1 -- 172.21.15.47:0/1362407212 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0029350 con 0x5624ed015f30 2024-02-19T15:26:37.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9dc2000640 1 -- 172.21.15.47:0/1362407212 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9dd001b040 con 0x5624ed015f30 2024-02-19T15:26:37.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 1 -- 172.21.15.47:0/1362407212 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5624ecdede80 con 0x5624ed015f30 2024-02-19T15:26:37.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9dc2000640 1 -- 172.21.15.47:0/1362407212 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9dd0013470 con 0x5624ed015f30 2024-02-19T15:26:37.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 10 client.7290 fetch_fsmap learned FSMap version 58 2024-02-19T15:26:37.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 1 -- 172.21.15.47:0/1362407212 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=2+}) v3 -- 0x5624ecb5ecc0 con 0x5624ed015f30 2024-02-19T15:26:37.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9dc2000640 1 -- 172.21.15.47:0/1362407212 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5624ecdede80 con 0x5624ed015f30 2024-02-19T15:26:37.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9dc2000640 1 -- 172.21.15.47:0/1362407212 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 58) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9dd0012070 con 0x5624ed015f30 2024-02-19T15:26:37.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 10 client.7290 fetch_fsmap finished waiting for FSMap version 58 2024-02-19T15:26:37.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 10 client.7290 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T15:26:37.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 20 client.7290 populate_metadata read hostname 'smithi047' 2024-02-19T15:26:37.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 1 --2- 172.21.15.47:0/1362407212 >> [v2:172.21.15.116:6835/547176070,v1:172.21.15.116:6837/547176070] conn(0x5624ed0b2550 0x5624ed0b2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 4 client.7290 mds_command: new command op to 6850 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T15:26:37.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 1 -- 172.21.15.47:0/1362407212 --> [v2:172.21.15.116:6835/547176070,v1:172.21.15.116:6837/547176070] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5624ece30fa0 con 0x5624ed0b2550 2024-02-19T15:26:37.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9ddbe00640 1 --2- 172.21.15.47:0/1362407212 >> [v2:172.21.15.116:6835/547176070,v1:172.21.15.116:6837/547176070] conn(0x5624ed0b2550 0x5624ed0b2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9dc2000640 10 client.7290 ms_handle_connect on v2:172.21.15.116:6835/547176070 2024-02-19T15:26:37.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9ddbe00640 1 --2- 172.21.15.47:0/1362407212 >> [v2:172.21.15.116:6835/547176070,v1:172.21.15.116:6837/547176070] conn(0x5624ed0b2550 0x5624ed0b2930 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21f65057dc50a5ed server_cookie=74d1e9462f0e41f4 in_seq=0 out_seq=0 2024-02-19T15:26:37.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9dc2000640 1 -- 172.21.15.47:0/1362407212 <== mds.0 v2:172.21.15.116:6835/547176070 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5624ece30fa0 con 0x5624ed0b2550 2024-02-19T15:26:37.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9dc2000640 10 client.7290 handle_command_reply: tid=0 2024-02-19T15:26:37.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 1 client.7290 shutdown 2024-02-19T15:26:37.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 1 -- 172.21.15.47:0/1362407212 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed015f30 msgr2=0x5624ed0b1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:37.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 1 --2- 172.21.15.47:0/1362407212 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed015f30 0x5624ed0b1130 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x5624ece69680 tx=0x7f9dd0013000 comp rx=0 tx=0).stop 2024-02-19T15:26:37.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 1 -- 172.21.15.47:0/1362407212 shutdown_connections 2024-02-19T15:26:37.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 1 --2- 172.21.15.47:0/1362407212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5624ed060240 0x5624ed0b2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 1 --2- 172.21.15.47:0/1362407212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624ed05f920 0x5624ed0b1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 1 --2- 172.21.15.47:0/1362407212 >> [v2:172.21.15.116:6835/547176070,v1:172.21.15.116:6837/547176070] conn(0x5624ed0b2550 0x5624ed0b2930 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 1 --2- 172.21.15.47:0/1362407212 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5624ed015f30 0x5624ed0b1130 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9dc2000640 0 client.7290 ms_handle_reset on v2:172.21.15.116:6835/547176070 2024-02-19T15:26:37.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 1 -- 172.21.15.47:0/1362407212 >> 172.21.15.47:0/1362407212 conn(0x5624ed01c2a0 msgr2=0x5624ed0721e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:37.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 1 -- 172.21.15.47:0/1362407212 shutdown_connections 2024-02-19T15:26:37.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 1 -- 172.21.15.47:0/1362407212 wait complete. 2024-02-19T15:26:37.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de68b9000 20 client.7290 trim_cache size 0 max 16384 2024-02-19T15:26:37.796 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:26:37.797 INFO:teuthology.orchestra.run.smithi047.stdout: "cluster_fsid": "ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e", 2024-02-19T15:26:37.797 INFO:teuthology.orchestra.run.smithi047.stdout: "whoami": 0, 2024-02-19T15:26:37.797 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 6850, 2024-02-19T15:26:37.797 INFO:teuthology.orchestra.run.smithi047.stdout: "want_state": "up:active", 2024-02-19T15:26:37.797 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "up:active", 2024-02-19T15:26:37.797 INFO:teuthology.orchestra.run.smithi047.stdout: "fs_name": "cephfs", 2024-02-19T15:26:37.797 INFO:teuthology.orchestra.run.smithi047.stdout: "rank_uptime": 10.473768166999999, 2024-02-19T15:26:37.797 INFO:teuthology.orchestra.run.smithi047.stdout: "mdsmap_epoch": 58, 2024-02-19T15:26:37.797 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch": 93, 2024-02-19T15:26:37.797 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch_barrier": 93, 2024-02-19T15:26:37.797 INFO:teuthology.orchestra.run.smithi047.stdout: "uptime": 15.372210129999999 2024-02-19T15:26:37.797 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:26:37.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de3400640 1 -- 172.21.15.47:0/1903528744 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9dbc066e70 msgr2=0x7f9dbc069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:37.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de3400640 1 --2- 172.21.15.47:0/1903528744 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9dbc066e70 0x7f9dbc069330 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0007a90 tx=0x7f9dd00079e0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de3400640 1 -- 172.21.15.47:0/1903528744 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ddc0cb520 msgr2=0x7f9ddc153350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:37.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de3400640 1 --2- 172.21.15.47:0/1903528744 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ddc0cb520 0x7f9ddc153350 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f9dc800c490 tx=0x7f9dc800c950 comp rx=0 tx=0).stop 2024-02-19T15:26:37.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de3400640 1 -- 172.21.15.47:0/1903528744 shutdown_connections 2024-02-19T15:26:37.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de3400640 1 --2- 172.21.15.47:0/1903528744 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9dbc066e70 0x7f9dbc069330 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de3400640 1 --2- 172.21.15.47:0/1903528744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ddc166b60 0x7f9ddc15e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de3400640 1 --2- 172.21.15.47:0/1903528744 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ddc0cb520 0x7f9ddc153350 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de3400640 1 --2- 172.21.15.47:0/1903528744 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ddc06aeb0 0x7f9ddc0ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de3400640 1 -- 172.21.15.47:0/1903528744 >> 172.21.15.47:0/1903528744 conn(0x7f9ddc0d0ad0 msgr2=0x7f9ddc06f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:37.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de3400640 1 -- 172.21.15.47:0/1903528744 shutdown_connections 2024-02-19T15:26:37.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.787+0000 7f9de3400640 1 -- 172.21.15.47:0/1903528744 wait complete. 2024-02-19T15:26:37.813 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-02-19T15:26:37.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.947+0000 7f51b2c00640 1 Processor -- start 2024-02-19T15:26:37.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.947+0000 7f51b2c00640 1 -- start start 2024-02-19T15:26:37.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.947+0000 7f51b2c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f51ac0cf670 0x7f51ac06aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.947+0000 7f51b2c00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51ac145a80 con 0x7f51ac1456a0 2024-02-19T15:26:37.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.947+0000 7f51b2c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51ac0d1880 con 0x7f51ac06b3f0 2024-02-19T15:26:37.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.947+0000 7f51b2c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51ac0d03c0 con 0x7f51ac0cf670 2024-02-19T15:26:37.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.947+0000 7f51abe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f51ac1456a0 0x7f51ac0cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39826/0 (socket says 172.21.15.47:39826) 2024-02-19T15:26:37.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.947+0000 7f51abe00640 1 -- 172.21.15.47:0/2353567981 learned_addr learned my addr 172.21.15.47:0/2353567981 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:37.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.947+0000 7f51a3400640 1 --2- 172.21.15.47:0/2353567981 >> v2:172.21.15.47:3300/0 conn(0x7f51ac0cf670 0x7f51ac06aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.947+0000 7f51ab400640 1 -- 172.21.15.47:0/2353567981 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330013765 0 0) 0x7f51ac145a80 con 0x7f51ac1456a0 2024-02-19T15:26:37.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.947+0000 7f51ab400640 1 -- 172.21.15.47:0/2353567981 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5190003530 con 0x7f51ac1456a0 2024-02-19T15:26:37.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.947+0000 7f51ab400640 1 -- 172.21.15.47:0/2353567981 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1475255238 0 0) 0x7f51ac0d1880 con 0x7f51ac06b3f0 2024-02-19T15:26:37.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.947+0000 7f51ab400640 1 -- 172.21.15.47:0/2353567981 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51ac145a80 con 0x7f51ac06b3f0 2024-02-19T15:26:37.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.947+0000 7f51a3400640 1 -- 172.21.15.47:0/2353567981 >> v1:172.21.15.116:6789/0 conn(0x7f51ac06b3f0 legacy=0x7f51ac06dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:37.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.947+0000 7f51a3400640 1 -- 172.21.15.47:0/2353567981 >> v1:172.21.15.47:6789/0 conn(0x7f51ac1456a0 legacy=0x7f51ac0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:37.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.947+0000 7f51a3400640 1 -- 172.21.15.47:0/2353567981 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51ac0cfb20 con 0x7f51ac0cf670 2024-02-19T15:26:37.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.951+0000 7f51a3400640 1 --2- 172.21.15.47:0/2353567981 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51ac0cf670 0x7f51ac06aeb0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f51940097c0 tx=0x7f519401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90574bab64f701ad server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:37.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.951+0000 7f51ab400640 1 -- 172.21.15.47:0/2353567981 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51940ab600 con 0x7f51ac0cf670 2024-02-19T15:26:37.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.951+0000 7f51ab400640 1 -- 172.21.15.47:0/2353567981 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51940a7050 con 0x7f51ac0cf670 2024-02-19T15:26:37.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.951+0000 7f51ab400640 1 -- 172.21.15.47:0/2353567981 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51940ab980 con 0x7f51ac0cf670 2024-02-19T15:26:37.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.951+0000 7f51b2c00640 1 -- 172.21.15.47:0/2353567981 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51ac0cf670 msgr2=0x7f51ac06aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:37.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.951+0000 7f51b2c00640 1 --2- 172.21.15.47:0/2353567981 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51ac0cf670 0x7f51ac06aeb0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f51940097c0 tx=0x7f519401c4a0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.951+0000 7f51b2c00640 1 -- 172.21.15.47:0/2353567981 shutdown_connections 2024-02-19T15:26:37.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.951+0000 7f51b2c00640 1 --2- 172.21.15.47:0/2353567981 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51ac0cf670 0x7f51ac06aeb0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.951+0000 7f51b2c00640 1 -- 172.21.15.47:0/2353567981 >> 172.21.15.47:0/2353567981 conn(0x7f51ac0d0ad0 msgr2=0x7f51ac0d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:37.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.951+0000 7f51b2c00640 1 -- 172.21.15.47:0/2353567981 shutdown_connections 2024-02-19T15:26:37.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.951+0000 7f51b2c00640 1 -- 172.21.15.47:0/2353567981 wait complete. 2024-02-19T15:26:37.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51b2c00640 1 Processor -- start 2024-02-19T15:26:37.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51b2c00640 1 -- start start 2024-02-19T15:26:37.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51b2c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51ac06b3f0 0x7f51ac13b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51b2c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51ac0cf670 0x7f51ac15db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51b2c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f51ac1456a0 0x7f51ac0afc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51b2c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac13bad0 con 0x7f51ac06b3f0 2024-02-19T15:26:37.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51b2c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac15d6a0 con 0x7f51ac0cf670 2024-02-19T15:26:37.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51b2c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac15e070 con 0x7f51ac1456a0 2024-02-19T15:26:37.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51abe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51ac06b3f0 0x7f51ac13b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51a3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51ac0cf670 0x7f51ac15db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51a3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51ac0cf670 0x7f51ac15db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:56202/0 (socket says 172.21.15.47:56202) 2024-02-19T15:26:37.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51a3400640 1 -- 172.21.15.47:0/2700114972 learned_addr learned my addr 172.21.15.47:0/2700114972 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:37.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51b0c00640 1 --2- 172.21.15.47:0/2700114972 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f51ac1456a0 0x7f51ac0afc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51abe00640 1 -- 172.21.15.47:0/2700114972 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f51ac1456a0 msgr2=0x7f51ac0afc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:37.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51abe00640 1 --2- 172.21.15.47:0/2700114972 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f51ac1456a0 0x7f51ac0afc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51abe00640 1 -- 172.21.15.47:0/2700114972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51ac0cf670 msgr2=0x7f51ac15db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:37.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51abe00640 1 --2- 172.21.15.47:0/2700114972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51ac0cf670 0x7f51ac15db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:37.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51abe00640 1 -- 172.21.15.47:0/2700114972 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5194009420 con 0x7f51ac06b3f0 2024-02-19T15:26:37.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51abe00640 1 --2- 172.21.15.47:0/2700114972 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51ac06b3f0 0x7f51ac13b590 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f519c00ca80 tx=0x7f519c00cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:37.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51a8c00640 1 -- 172.21.15.47:0/2700114972 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519c00ac30 con 0x7f51ac06b3f0 2024-02-19T15:26:37.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51a8c00640 1 -- 172.21.15.47:0/2700114972 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f519c00ad90 con 0x7f51ac06b3f0 2024-02-19T15:26:37.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51a8c00640 1 -- 172.21.15.47:0/2700114972 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519c012640 con 0x7f51ac06b3f0 2024-02-19T15:26:37.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51b2c00640 1 -- 172.21.15.47:0/2700114972 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51ac15ca90 con 0x7f51ac06b3f0 2024-02-19T15:26:37.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.955+0000 7f51b2c00640 1 -- 172.21.15.47:0/2700114972 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51ac15d050 con 0x7f51ac06b3f0 2024-02-19T15:26:37.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.959+0000 7f51b2c00640 1 -- 172.21.15.47:0/2700114972 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51ac075360 con 0x7f51ac06b3f0 2024-02-19T15:26:37.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.959+0000 7f51a8c00640 1 -- 172.21.15.47:0/2700114972 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f519c007070 con 0x7f51ac06b3f0 2024-02-19T15:26:37.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.959+0000 7f51a8c00640 1 --2- 172.21.15.47:0/2700114972 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5188066e90 0x7f5188069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:37.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.959+0000 7f51a8c00640 1 -- 172.21.15.47:0/2700114972 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f519c084830 con 0x7f51ac06b3f0 2024-02-19T15:26:37.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.959+0000 7f51a3400640 1 --2- 172.21.15.47:0/2700114972 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5188066e90 0x7f5188069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:37.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.959+0000 7f51a3400640 1 --2- 172.21.15.47:0/2700114972 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5188066e90 0x7f5188069350 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f519401a0d0 tx=0x7f5194008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:37.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:37.963+0000 7f51a8c00640 1 -- 172.21.15.47:0/2700114972 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f519c0517b0 con 0x7f51ac06b3f0 2024-02-19T15:26:38.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:38.159+0000 7f51b2c00640 1 -- 172.21.15.47:0/2700114972 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f51ac13bf00 con 0x7f51ac06b3f0 2024-02-19T15:26:38.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:38.159+0000 7f51a8c00640 1 -- 172.21.15.47:0/2700114972 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v58) v1 ==== 53+0+73 (secure 0 0 0) 0x7f519c054e70 con 0x7f51ac06b3f0 2024-02-19T15:26:38.166 INFO:teuthology.orchestra.run.smithi047.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-02-19T15:26:38.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:38.163+0000 7f51b2c00640 1 -- 172.21.15.47:0/2700114972 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5188066e90 msgr2=0x7f5188069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:38.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:38.163+0000 7f51b2c00640 1 --2- 172.21.15.47:0/2700114972 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5188066e90 0x7f5188069350 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f519401a0d0 tx=0x7f5194008040 comp rx=0 tx=0).stop 2024-02-19T15:26:38.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:38.163+0000 7f51b2c00640 1 -- 172.21.15.47:0/2700114972 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51ac06b3f0 msgr2=0x7f51ac13b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:38.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:38.163+0000 7f51b2c00640 1 --2- 172.21.15.47:0/2700114972 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51ac06b3f0 0x7f51ac13b590 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f519c00ca80 tx=0x7f519c00cf40 comp rx=0 tx=0).stop 2024-02-19T15:26:38.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:38.163+0000 7f51b2c00640 1 -- 172.21.15.47:0/2700114972 shutdown_connections 2024-02-19T15:26:38.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:38.163+0000 7f51b2c00640 1 --2- 172.21.15.47:0/2700114972 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5188066e90 0x7f5188069350 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:38.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:38.163+0000 7f51b2c00640 1 --2- 172.21.15.47:0/2700114972 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f51ac1456a0 0x7f51ac0afc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:38.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:38.163+0000 7f51b2c00640 1 --2- 172.21.15.47:0/2700114972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51ac0cf670 0x7f51ac15db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:38.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:38.163+0000 7f51b2c00640 1 --2- 172.21.15.47:0/2700114972 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51ac06b3f0 0x7f51ac13b590 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:38.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:38.163+0000 7f51b2c00640 1 -- 172.21.15.47:0/2700114972 >> 172.21.15.47:0/2700114972 conn(0x7f51ac0d0ad0 msgr2=0x7f51ac0cef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:38.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:38.163+0000 7f51b2c00640 1 -- 172.21.15.47:0/2700114972 shutdown_connections 2024-02-19T15:26:38.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:38.163+0000 7f51b2c00640 1 -- 172.21.15.47:0/2700114972 wait complete. 2024-02-19T15:26:38.186 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-02-19T15:26:38.186 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T15:26:38.186 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi047.front.sepia.ceph.com 2024-02-19T15:26:38.186 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-19T15:26:38.186 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-19T15:26:38.186 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-02-19T15:26:38.187 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-02-19T15:26:38.187 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T15:26:38.187 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:26:38.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:38.187 DEBUG:teuthology.orchestra.run.smithi047:> ip addr 2024-02-19T15:26:38.196 INFO:teuthology.orchestra.run.smithi047.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T15:26:38.196 INFO:teuthology.orchestra.run.smithi047.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T15:26:38.196 INFO:teuthology.orchestra.run.smithi047.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T15:26:38.196 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:26:38.196 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 ::1/128 scope host 2024-02-19T15:26:38.196 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:26:38.196 INFO:teuthology.orchestra.run.smithi047.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:26:38.196 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:78 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:26:38.196 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f0 2024-02-19T15:26:38.197 INFO:teuthology.orchestra.run.smithi047.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T15:26:38.197 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8e brd ff:ff:ff:ff:ff:ff 2024-02-19T15:26:38.197 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f0 2024-02-19T15:26:38.197 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.21.15.47/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-02-19T15:26:38.197 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft 42217sec preferred_lft 42217sec 2024-02-19T15:26:38.197 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::ec4:7aff:febd:148e/64 scope link 2024-02-19T15:26:38.197 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:26:38.197 INFO:teuthology.orchestra.run.smithi047.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:26:38.197 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:79 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:26:38.197 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f1 2024-02-19T15:26:38.197 INFO:teuthology.orchestra.run.smithi047.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:26:38.197 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8f brd ff:ff:ff:ff:ff:ff 2024-02-19T15:26:38.197 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f1 2024-02-19T15:26:38.197 INFO:teuthology.orchestra.run.smithi047.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-02-19T15:26:38.197 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 02:42:47:cf:b1:62 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:26:38.197 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-02-19T15:26:38.198 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:26:38.198 INFO:teuthology.orchestra.run.smithi047.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-02-19T15:26:38.198 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-02-19T15:26:38.198 INFO:teuthology.orchestra.run.smithi047.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-02-19T15:26:38.198 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:26:38.198 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-02-19T15:26:38.198 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:26:38.198 INFO:teuthology.orchestra.run.smithi047.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:26:38.198 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-19T15:26:38.198 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-02-19T15:26:38.198 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:26:38.198 INFO:teuthology.orchestra.run.smithi047.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:26:38.198 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-02-19T15:26:38.198 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-02-19T15:26:38.198 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:26:38.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:38.200 DEBUG:teuthology.orchestra.run.smithi047:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T15:26:38.259 INFO:teuthology.orchestra.run.smithi047.stdout:1 2024-02-19T15:26:38.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:38.262 DEBUG:teuthology.orchestra.run.smithi047:> route 2024-02-19T15:26:38.311 INFO:teuthology.orchestra.run.smithi047.stdout:Kernel IP routing table 2024-02-19T15:26:38.311 INFO:teuthology.orchestra.run.smithi047.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T15:26:38.311 INFO:teuthology.orchestra.run.smithi047.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-02-19T15:26:38.311 INFO:teuthology.orchestra.run.smithi047.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-02-19T15:26:38.311 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-02-19T15:26:38.312 INFO:teuthology.orchestra.run.smithi047.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T15:26:38.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:38.312 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2024-02-19T15:26:38.313 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2024-02-19T15:26:38.313 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-02-19T15:26:38.313 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-02-19T15:26:38.313 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-02-19T15:26:38.313 DEBUG:teuthology.orchestra.run.smithi047:> ') 2024-02-19T15:26:38.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:38.403 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list 2024-02-19T15:26:38.411 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-02-19T15:26:38.411 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-02-19T15:26:38.412 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:26:38.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:38.413 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set brx.0 up 2024-02-19T15:26:38.473 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:26:38.473 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-19T15:26:38.521 INFO:teuthology.orchestra.run.smithi047.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:26:38.522 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:26:38.522 DEBUG:teuthology.orchestra.run.smithi047:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:26:38.573 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-19T15:26:38.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:38.574 DEBUG:teuthology.orchestra.run.smithi047:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:26:38.663 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:26:38.663 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:26:38.663 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:26:38.678 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:26:38.678 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:26:38.678 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:26:38.678 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:26:38.678 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:26:38.687 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:26:38.687 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:26:38.687 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:26:38.687 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:26:38.687 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:26:38.687 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790" 2024-02-19T15:26:38.688 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790:/ 2024-02-19T15:26:38.688 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e" will pass to kernel 2024-02-19T15:26:38.691 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T15:26:38.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:38.691 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:26:38.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:38.712 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:26:38.772 INFO:teuthology.orchestra.run.smithi047.stdout:client6314 2024-02-19T15:26:38.772 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:26:38.773 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:26:38.823 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-19T15:26:38.873 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-19T15:26:38.912 INFO:teuthology.orchestra.run.smithi047.stdout:[global] 2024-02-19T15:26:38.912 INFO:teuthology.orchestra.run.smithi047.stdout: chdir = "" 2024-02-19T15:26:38.913 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-19T15:26:38.913 INFO:teuthology.orchestra.run.smithi047.stdout: auth supported = cephx 2024-02-19T15:26:38.913 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.913 INFO:teuthology.orchestra.run.smithi047.stdout: filestore xattr use omap = true 2024-02-19T15:26:38.913 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.913 INFO:teuthology.orchestra.run.smithi047.stdout: mon clock drift allowed = 1.000 2024-02-19T15:26:38.913 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.913 INFO:teuthology.orchestra.run.smithi047.stdout: osd crush chooseleaf type = 0 2024-02-19T15:26:38.913 INFO:teuthology.orchestra.run.smithi047.stdout: auth debug = true 2024-02-19T15:26:38.913 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.913 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on old message = true 2024-02-19T15:26:38.913 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on bug = true 2024-02-19T15:26:38.913 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.913 INFO:teuthology.orchestra.run.smithi047.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-19T15:26:38.914 INFO:teuthology.orchestra.run.smithi047.stdout: mon pg warn max object skew = 0 2024-02-19T15:26:38.914 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.914 INFO:teuthology.orchestra.run.smithi047.stdout: # disable pg_autoscaler by default for new pools 2024-02-19T15:26:38.914 INFO:teuthology.orchestra.run.smithi047.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-19T15:26:38.914 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.914 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default size = 2 2024-02-19T15:26:38.914 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.914 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow primary affinity = true 2024-02-19T15:26:38.914 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow pg remap = true 2024-02-19T15:26:38.914 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on legacy crush tunables = false 2024-02-19T15:26:38.914 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on crush straw calc version zero = false 2024-02-19T15:26:38.914 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on no sortbitwise = false 2024-02-19T15:26:38.914 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on osd down out interval zero = false 2024-02-19T15:26:38.914 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on too few osds = false 2024-02-19T15:26:38.914 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-19T15:26:38.915 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-19T15:26:38.915 INFO:teuthology.orchestra.run.smithi047.stdout: mon_allow_pool_size_one = true 2024-02-19T15:26:38.915 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.915 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-19T15:26:38.915 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.915 INFO:teuthology.orchestra.run.smithi047.stdout: osd default data pool replay window = 5 2024-02-19T15:26:38.915 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.915 INFO:teuthology.orchestra.run.smithi047.stdout: mon allow pool delete = true 2024-02-19T15:26:38.915 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.915 INFO:teuthology.orchestra.run.smithi047.stdout: mon cluster log file level = debug 2024-02-19T15:26:38.915 INFO:teuthology.orchestra.run.smithi047.stdout: debug asserts on shutdown = true 2024-02-19T15:26:38.915 INFO:teuthology.orchestra.run.smithi047.stdout: mon health detail to clog = false 2024-02-19T15:26:38.915 INFO:teuthology.orchestra.run.smithi047.stdout: mon host = "172.21.15.47,172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790]" 2024-02-19T15:26:38.915 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on skipped message = False 2024-02-19T15:26:38.915 INFO:teuthology.orchestra.run.smithi047.stdout: fsid = ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e 2024-02-19T15:26:38.916 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.916 INFO:teuthology.orchestra.run.smithi047.stdout:[osd] 2024-02-19T15:26:38.916 INFO:teuthology.orchestra.run.smithi047.stdout: osd journal size = 100 2024-02-19T15:26:38.916 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.916 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub load threshold = 5.0 2024-02-19T15:26:38.916 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub max interval = 600 2024-02-19T15:26:38.916 INFO:teuthology.orchestra.run.smithi047.stdout: osd mclock profile = high_recovery_ops 2024-02-19T15:26:38.916 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.916 INFO:teuthology.orchestra.run.smithi047.stdout: osd recover clone overlap = true 2024-02-19T15:26:38.916 INFO:teuthology.orchestra.run.smithi047.stdout: osd recovery max chunk = 1048576 2024-02-19T15:26:38.916 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.916 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug shutdown = true 2024-02-19T15:26:38.916 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug op order = true 2024-02-19T15:26:38.916 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify stray on activate = true 2024-02-19T15:26:38.917 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug trim objects = true 2024-02-19T15:26:38.917 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.917 INFO:teuthology.orchestra.run.smithi047.stdout: osd open classes on start = true 2024-02-19T15:26:38.917 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug pg log writeout = true 2024-02-19T15:26:38.917 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.917 INFO:teuthology.orchestra.run.smithi047.stdout: osd deep scrub update digest min age = 30 2024-02-19T15:26:38.917 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.917 INFO:teuthology.orchestra.run.smithi047.stdout: osd map max advance = 10 2024-02-19T15:26:38.917 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.917 INFO:teuthology.orchestra.run.smithi047.stdout: journal zero on create = true 2024-02-19T15:26:38.917 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.917 INFO:teuthology.orchestra.run.smithi047.stdout: filestore ondisk finisher threads = 3 2024-02-19T15:26:38.917 INFO:teuthology.orchestra.run.smithi047.stdout: filestore apply finisher threads = 3 2024-02-19T15:26:38.917 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.917 INFO:teuthology.orchestra.run.smithi047.stdout: bdev debug aio = true 2024-02-19T15:26:38.918 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug misdirected ops = true 2024-02-19T15:26:38.918 INFO:teuthology.orchestra.run.smithi047.stdout: bdev async discard = True 2024-02-19T15:26:38.918 INFO:teuthology.orchestra.run.smithi047.stdout: bdev enable discard = True 2024-02-19T15:26:38.918 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore allocator = bitmap 2024-02-19T15:26:38.918 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore block size = 96636764160 2024-02-19T15:26:38.918 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore fsck on mount = True 2024-02-19T15:26:38.918 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluefs = 1/20 2024-02-19T15:26:38.918 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluestore = 1/20 2024-02-19T15:26:38.918 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:26:38.918 INFO:teuthology.orchestra.run.smithi047.stdout: debug osd = 20 2024-02-19T15:26:38.918 INFO:teuthology.orchestra.run.smithi047.stdout: debug rocksdb = 4/10 2024-02-19T15:26:38.918 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd backfillfull_ratio = 0.85 2024-02-19T15:26:38.918 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd full ratio = 0.9 2024-02-19T15:26:38.918 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd nearfull ratio = 0.8 2024-02-19T15:26:38.918 INFO:teuthology.orchestra.run.smithi047.stdout: osd failsafe full ratio = 0.95 2024-02-19T15:26:38.918 INFO:teuthology.orchestra.run.smithi047.stdout: osd objectstore = bluestore 2024-02-19T15:26:38.919 INFO:teuthology.orchestra.run.smithi047.stdout: osd op complaint time = 180 2024-02-19T15:26:38.919 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.919 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr] 2024-02-19T15:26:38.919 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:26:38.919 INFO:teuthology.orchestra.run.smithi047.stdout: debug mgr = 20 2024-02-19T15:26:38.919 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-02-19T15:26:38.919 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-02-19T15:26:38.919 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:26:38.919 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:26:38.919 INFO:teuthology.orchestra.run.smithi047.stdout: mgr/telemetry/nag = false 2024-02-19T15:26:38.919 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.919 INFO:teuthology.orchestra.run.smithi047.stdout:[mon] 2024-02-19T15:26:38.919 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:26:38.919 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-02-19T15:26:38.919 INFO:teuthology.orchestra.run.smithi047.stdout: debug paxos = 20 2024-02-19T15:26:38.919 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-02-19T15:26:38.920 INFO:teuthology.orchestra.run.smithi047.stdout: mon data avail warn = 5 2024-02-19T15:26:38.920 INFO:teuthology.orchestra.run.smithi047.stdout: mon mgr mkfs grace = 240 2024-02-19T15:26:38.920 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:26:38.920 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd reporter subtree level = osd 2024-02-19T15:26:38.920 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd prime pg temp = true 2024-02-19T15:26:38.920 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:26:38.920 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.920 INFO:teuthology.orchestra.run.smithi047.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-19T15:26:38.920 INFO:teuthology.orchestra.run.smithi047.stdout: auth mon ticket ttl = 660 # 11m 2024-02-19T15:26:38.920 INFO:teuthology.orchestra.run.smithi047.stdout: auth service ticket ttl = 240 # 4m 2024-02-19T15:26:38.920 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.920 INFO:teuthology.orchestra.run.smithi047.stdout: # don't complain about insecure global_id in the test suite 2024-02-19T15:26:38.920 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-19T15:26:38.920 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-19T15:26:38.920 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.921 INFO:teuthology.orchestra.run.smithi047.stdout: # 1m isn't quite enough 2024-02-19T15:26:38.921 INFO:teuthology.orchestra.run.smithi047.stdout: mon_down_mkfs_grace = 2m 2024-02-19T15:26:38.921 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.921 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_filestore_osds = false 2024-02-19T15:26:38.921 INFO:teuthology.orchestra.run.smithi047.stdout: mon op complaint time = 120 2024-02-19T15:26:38.921 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:38.921 INFO:teuthology.orchestra.run.smithi047.stdout:[client] 2024-02-19T15:26:38.921 INFO:teuthology.orchestra.run.smithi047.stdout: rgw cache enabled = true 2024-02-19T15:26:38.921 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable ops log = true 2024-02-19T15:26:38.921 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable usage log = true 2024-02-19T15:26:38.921 INFO:teuthology.orchestra.run.smithi047.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-19T15:26:38.921 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-19T15:26:38.921 INFO:teuthology.orchestra.run.smithi047.stdout: client mount timeout = 600 2024-02-19T15:26:38.921 INFO:teuthology.orchestra.run.smithi047.stdout: debug client = 20 2024-02-19T15:26:38.921 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:26:38.921 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = "" 2024-02-19T15:26:38.922 INFO:teuthology.orchestra.run.smithi047.stdout: rados mon op timeout = 900 2024-02-19T15:26:38.922 INFO:teuthology.orchestra.run.smithi047.stdout: rados osd op timeout = 900 2024-02-19T15:26:38.922 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.a] 2024-02-19T15:26:38.922 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.b] 2024-02-19T15:26:38.922 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.c] 2024-02-19T15:26:38.922 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.a] 2024-02-19T15:26:38.922 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.c] 2024-02-19T15:26:38.922 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.b] 2024-02-19T15:26:38.922 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.d] 2024-02-19T15:26:38.922 INFO:teuthology.orchestra.run.smithi047.stdout:[mds] 2024-02-19T15:26:38.922 INFO:teuthology.orchestra.run.smithi047.stdout: debug mds = 20 2024-02-19T15:26:38.922 INFO:teuthology.orchestra.run.smithi047.stdout: debug mds balancer = 20 2024-02-19T15:26:38.922 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:26:38.922 INFO:teuthology.orchestra.run.smithi047.stdout: mds debug frag = True 2024-02-19T15:26:38.922 INFO:teuthology.orchestra.run.smithi047.stdout: mds debug scatterstat = True 2024-02-19T15:26:38.922 INFO:teuthology.orchestra.run.smithi047.stdout: mds op complaint time = 180 2024-02-19T15:26:38.923 INFO:teuthology.orchestra.run.smithi047.stdout: mds verify scatter = True 2024-02-19T15:26:38.923 INFO:teuthology.orchestra.run.smithi047.stdout: osd op complaint time = 180 2024-02-19T15:26:38.923 INFO:teuthology.orchestra.run.smithi047.stdout: rados mon op timeout = 900 2024-02-19T15:26:38.923 INFO:teuthology.orchestra.run.smithi047.stdout: rados osd op timeout = 900 2024-02-19T15:26:38.923 INFO:teuthology.orchestra.run.smithi047.stdout: mds_max_purge_files = 6 2024-02-19T15:26:38.923 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-02-19T15:26:38.924 INFO:teuthology.orchestra.run.smithi116.stdout: chdir = "" 2024-02-19T15:26:38.924 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-19T15:26:38.924 INFO:teuthology.orchestra.run.smithi116.stdout: auth supported = cephx 2024-02-19T15:26:38.924 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.924 INFO:teuthology.orchestra.run.smithi116.stdout: filestore xattr use omap = true 2024-02-19T15:26:38.924 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.924 INFO:teuthology.orchestra.run.smithi116.stdout: mon clock drift allowed = 1.000 2024-02-19T15:26:38.924 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.924 INFO:teuthology.orchestra.run.smithi116.stdout: osd crush chooseleaf type = 0 2024-02-19T15:26:38.924 INFO:teuthology.orchestra.run.smithi116.stdout: auth debug = true 2024-02-19T15:26:38.924 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.924 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on old message = true 2024-02-19T15:26:38.924 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on bug = true 2024-02-19T15:26:38.925 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.925 INFO:teuthology.orchestra.run.smithi116.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-19T15:26:38.925 INFO:teuthology.orchestra.run.smithi116.stdout: mon pg warn max object skew = 0 2024-02-19T15:26:38.925 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.925 INFO:teuthology.orchestra.run.smithi116.stdout: # disable pg_autoscaler by default for new pools 2024-02-19T15:26:38.925 INFO:teuthology.orchestra.run.smithi116.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-19T15:26:38.925 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.925 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default size = 2 2024-02-19T15:26:38.925 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.925 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow primary affinity = true 2024-02-19T15:26:38.925 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow pg remap = true 2024-02-19T15:26:38.926 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on legacy crush tunables = false 2024-02-19T15:26:38.926 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on crush straw calc version zero = false 2024-02-19T15:26:38.926 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on no sortbitwise = false 2024-02-19T15:26:38.926 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on osd down out interval zero = false 2024-02-19T15:26:38.926 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on too few osds = false 2024-02-19T15:26:38.926 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-19T15:26:38.926 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-19T15:26:38.926 INFO:teuthology.orchestra.run.smithi116.stdout: mon_allow_pool_size_one = true 2024-02-19T15:26:38.926 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.926 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-19T15:26:38.926 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.926 INFO:teuthology.orchestra.run.smithi116.stdout: osd default data pool replay window = 5 2024-02-19T15:26:38.926 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.926 INFO:teuthology.orchestra.run.smithi116.stdout: mon allow pool delete = true 2024-02-19T15:26:38.926 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.927 INFO:teuthology.orchestra.run.smithi116.stdout: mon cluster log file level = debug 2024-02-19T15:26:38.927 INFO:teuthology.orchestra.run.smithi116.stdout: debug asserts on shutdown = true 2024-02-19T15:26:38.927 INFO:teuthology.orchestra.run.smithi116.stdout: mon health detail to clog = false 2024-02-19T15:26:38.927 INFO:teuthology.orchestra.run.smithi116.stdout: mon host = "172.21.15.47,172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790]" 2024-02-19T15:26:38.927 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on skipped message = False 2024-02-19T15:26:38.927 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e 2024-02-19T15:26:38.927 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.927 INFO:teuthology.orchestra.run.smithi116.stdout:[osd] 2024-02-19T15:26:38.927 INFO:teuthology.orchestra.run.smithi116.stdout: osd journal size = 100 2024-02-19T15:26:38.927 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.927 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub load threshold = 5.0 2024-02-19T15:26:38.927 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub max interval = 600 2024-02-19T15:26:38.927 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock profile = high_recovery_ops 2024-02-19T15:26:38.927 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.927 INFO:teuthology.orchestra.run.smithi116.stdout: osd recover clone overlap = true 2024-02-19T15:26:38.927 INFO:teuthology.orchestra.run.smithi116.stdout: osd recovery max chunk = 1048576 2024-02-19T15:26:38.928 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.928 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug shutdown = true 2024-02-19T15:26:38.928 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug op order = true 2024-02-19T15:26:38.928 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify stray on activate = true 2024-02-19T15:26:38.928 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug trim objects = true 2024-02-19T15:26:38.928 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.928 INFO:teuthology.orchestra.run.smithi116.stdout: osd open classes on start = true 2024-02-19T15:26:38.928 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug pg log writeout = true 2024-02-19T15:26:38.928 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.928 INFO:teuthology.orchestra.run.smithi116.stdout: osd deep scrub update digest min age = 30 2024-02-19T15:26:38.928 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.928 INFO:teuthology.orchestra.run.smithi116.stdout: osd map max advance = 10 2024-02-19T15:26:38.928 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.928 INFO:teuthology.orchestra.run.smithi116.stdout: journal zero on create = true 2024-02-19T15:26:38.928 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.929 INFO:teuthology.orchestra.run.smithi116.stdout: filestore ondisk finisher threads = 3 2024-02-19T15:26:38.929 INFO:teuthology.orchestra.run.smithi116.stdout: filestore apply finisher threads = 3 2024-02-19T15:26:38.929 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.929 INFO:teuthology.orchestra.run.smithi116.stdout: bdev debug aio = true 2024-02-19T15:26:38.929 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug misdirected ops = true 2024-02-19T15:26:38.929 INFO:teuthology.orchestra.run.smithi116.stdout: bdev async discard = True 2024-02-19T15:26:38.929 INFO:teuthology.orchestra.run.smithi116.stdout: bdev enable discard = True 2024-02-19T15:26:38.929 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore allocator = bitmap 2024-02-19T15:26:38.929 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore block size = 96636764160 2024-02-19T15:26:38.929 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore fsck on mount = True 2024-02-19T15:26:38.929 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluefs = 1/20 2024-02-19T15:26:38.929 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluestore = 1/20 2024-02-19T15:26:38.929 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:26:38.929 INFO:teuthology.orchestra.run.smithi116.stdout: debug osd = 20 2024-02-19T15:26:38.929 INFO:teuthology.orchestra.run.smithi116.stdout: debug rocksdb = 4/10 2024-02-19T15:26:38.929 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd backfillfull_ratio = 0.85 2024-02-19T15:26:38.930 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd full ratio = 0.9 2024-02-19T15:26:38.930 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd nearfull ratio = 0.8 2024-02-19T15:26:38.930 INFO:teuthology.orchestra.run.smithi116.stdout: osd failsafe full ratio = 0.95 2024-02-19T15:26:38.930 INFO:teuthology.orchestra.run.smithi116.stdout: osd objectstore = bluestore 2024-02-19T15:26:38.930 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2024-02-19T15:26:38.930 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.930 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr] 2024-02-19T15:26:38.930 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:26:38.930 INFO:teuthology.orchestra.run.smithi116.stdout: debug mgr = 20 2024-02-19T15:26:38.930 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-02-19T15:26:38.930 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-02-19T15:26:38.930 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:26:38.930 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:26:38.930 INFO:teuthology.orchestra.run.smithi116.stdout: mgr/telemetry/nag = false 2024-02-19T15:26:38.930 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.930 INFO:teuthology.orchestra.run.smithi116.stdout:[mon] 2024-02-19T15:26:38.931 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:26:38.931 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-02-19T15:26:38.931 INFO:teuthology.orchestra.run.smithi116.stdout: debug paxos = 20 2024-02-19T15:26:38.931 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-02-19T15:26:38.931 INFO:teuthology.orchestra.run.smithi116.stdout: mon data avail warn = 5 2024-02-19T15:26:38.931 INFO:teuthology.orchestra.run.smithi116.stdout: mon mgr mkfs grace = 240 2024-02-19T15:26:38.931 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:26:38.931 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd reporter subtree level = osd 2024-02-19T15:26:38.931 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd prime pg temp = true 2024-02-19T15:26:38.931 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:26:38.931 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.931 INFO:teuthology.orchestra.run.smithi116.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-19T15:26:38.931 INFO:teuthology.orchestra.run.smithi116.stdout: auth mon ticket ttl = 660 # 11m 2024-02-19T15:26:38.931 INFO:teuthology.orchestra.run.smithi116.stdout: auth service ticket ttl = 240 # 4m 2024-02-19T15:26:38.931 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.932 INFO:teuthology.orchestra.run.smithi116.stdout: # don't complain about insecure global_id in the test suite 2024-02-19T15:26:38.932 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-19T15:26:38.932 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-19T15:26:38.932 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.932 INFO:teuthology.orchestra.run.smithi116.stdout: # 1m isn't quite enough 2024-02-19T15:26:38.932 INFO:teuthology.orchestra.run.smithi116.stdout: mon_down_mkfs_grace = 2m 2024-02-19T15:26:38.932 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.932 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_filestore_osds = false 2024-02-19T15:26:38.932 INFO:teuthology.orchestra.run.smithi116.stdout: mon op complaint time = 120 2024-02-19T15:26:38.932 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:38.932 INFO:teuthology.orchestra.run.smithi116.stdout:[client] 2024-02-19T15:26:38.932 INFO:teuthology.orchestra.run.smithi116.stdout: rgw cache enabled = true 2024-02-19T15:26:38.932 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable ops log = true 2024-02-19T15:26:38.932 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable usage log = true 2024-02-19T15:26:38.932 INFO:teuthology.orchestra.run.smithi116.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-19T15:26:38.932 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-19T15:26:38.933 INFO:teuthology.orchestra.run.smithi116.stdout: client mount timeout = 600 2024-02-19T15:26:38.933 INFO:teuthology.orchestra.run.smithi116.stdout: debug client = 20 2024-02-19T15:26:38.933 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:26:38.933 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = "" 2024-02-19T15:26:38.933 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2024-02-19T15:26:38.933 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2024-02-19T15:26:38.933 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.a] 2024-02-19T15:26:38.933 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.b] 2024-02-19T15:26:38.933 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.c] 2024-02-19T15:26:38.933 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a] 2024-02-19T15:26:38.933 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c] 2024-02-19T15:26:38.933 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b] 2024-02-19T15:26:38.933 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d] 2024-02-19T15:26:38.933 INFO:teuthology.orchestra.run.smithi116.stdout:[mds] 2024-02-19T15:26:38.933 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds = 20 2024-02-19T15:26:38.934 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds balancer = 20 2024-02-19T15:26:38.934 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:26:38.934 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug frag = True 2024-02-19T15:26:38.934 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug scatterstat = True 2024-02-19T15:26:38.934 INFO:teuthology.orchestra.run.smithi116.stdout: mds op complaint time = 180 2024-02-19T15:26:38.934 INFO:teuthology.orchestra.run.smithi116.stdout: mds verify scatter = True 2024-02-19T15:26:38.934 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2024-02-19T15:26:38.934 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2024-02-19T15:26:38.934 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2024-02-19T15:26:38.934 INFO:teuthology.orchestra.run.smithi116.stdout: mds_max_purge_files = 6 2024-02-19T15:26:38.936 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-19T15:26:38.985 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-19T15:26:39.025 INFO:teuthology.orchestra.run.smithi047.stdout:[global] 2024-02-19T15:26:39.025 INFO:teuthology.orchestra.run.smithi047.stdout: chdir = "" 2024-02-19T15:26:39.025 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-19T15:26:39.025 INFO:teuthology.orchestra.run.smithi047.stdout: auth supported = cephx 2024-02-19T15:26:39.025 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.025 INFO:teuthology.orchestra.run.smithi047.stdout: filestore xattr use omap = true 2024-02-19T15:26:39.025 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.025 INFO:teuthology.orchestra.run.smithi047.stdout: mon clock drift allowed = 1.000 2024-02-19T15:26:39.025 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.025 INFO:teuthology.orchestra.run.smithi047.stdout: osd crush chooseleaf type = 0 2024-02-19T15:26:39.026 INFO:teuthology.orchestra.run.smithi047.stdout: auth debug = true 2024-02-19T15:26:39.026 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.026 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on old message = true 2024-02-19T15:26:39.026 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on bug = true 2024-02-19T15:26:39.026 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.026 INFO:teuthology.orchestra.run.smithi047.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-19T15:26:39.026 INFO:teuthology.orchestra.run.smithi047.stdout: mon pg warn max object skew = 0 2024-02-19T15:26:39.026 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.026 INFO:teuthology.orchestra.run.smithi047.stdout: # disable pg_autoscaler by default for new pools 2024-02-19T15:26:39.026 INFO:teuthology.orchestra.run.smithi047.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-19T15:26:39.026 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.026 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default size = 2 2024-02-19T15:26:39.026 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.027 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow primary affinity = true 2024-02-19T15:26:39.027 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow pg remap = true 2024-02-19T15:26:39.027 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on legacy crush tunables = false 2024-02-19T15:26:39.027 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on crush straw calc version zero = false 2024-02-19T15:26:39.027 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on no sortbitwise = false 2024-02-19T15:26:39.027 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on osd down out interval zero = false 2024-02-19T15:26:39.027 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on too few osds = false 2024-02-19T15:26:39.027 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-19T15:26:39.027 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-19T15:26:39.027 INFO:teuthology.orchestra.run.smithi047.stdout: mon_allow_pool_size_one = true 2024-02-19T15:26:39.027 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.027 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-19T15:26:39.027 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.027 INFO:teuthology.orchestra.run.smithi047.stdout: osd default data pool replay window = 5 2024-02-19T15:26:39.027 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.028 INFO:teuthology.orchestra.run.smithi047.stdout: mon allow pool delete = true 2024-02-19T15:26:39.028 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.028 INFO:teuthology.orchestra.run.smithi047.stdout: mon cluster log file level = debug 2024-02-19T15:26:39.028 INFO:teuthology.orchestra.run.smithi047.stdout: debug asserts on shutdown = true 2024-02-19T15:26:39.028 INFO:teuthology.orchestra.run.smithi047.stdout: mon health detail to clog = false 2024-02-19T15:26:39.028 INFO:teuthology.orchestra.run.smithi047.stdout: mon host = "172.21.15.47,172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790]" 2024-02-19T15:26:39.028 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on skipped message = False 2024-02-19T15:26:39.028 INFO:teuthology.orchestra.run.smithi047.stdout: fsid = ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e 2024-02-19T15:26:39.028 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.028 INFO:teuthology.orchestra.run.smithi047.stdout:[osd] 2024-02-19T15:26:39.028 INFO:teuthology.orchestra.run.smithi047.stdout: osd journal size = 100 2024-02-19T15:26:39.028 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.028 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub load threshold = 5.0 2024-02-19T15:26:39.028 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub max interval = 600 2024-02-19T15:26:39.028 INFO:teuthology.orchestra.run.smithi047.stdout: osd mclock profile = high_recovery_ops 2024-02-19T15:26:39.029 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.029 INFO:teuthology.orchestra.run.smithi047.stdout: osd recover clone overlap = true 2024-02-19T15:26:39.029 INFO:teuthology.orchestra.run.smithi047.stdout: osd recovery max chunk = 1048576 2024-02-19T15:26:39.029 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.029 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug shutdown = true 2024-02-19T15:26:39.029 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug op order = true 2024-02-19T15:26:39.029 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify stray on activate = true 2024-02-19T15:26:39.029 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug trim objects = true 2024-02-19T15:26:39.029 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.029 INFO:teuthology.orchestra.run.smithi047.stdout: osd open classes on start = true 2024-02-19T15:26:39.029 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug pg log writeout = true 2024-02-19T15:26:39.029 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.029 INFO:teuthology.orchestra.run.smithi047.stdout: osd deep scrub update digest min age = 30 2024-02-19T15:26:39.029 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.029 INFO:teuthology.orchestra.run.smithi047.stdout: osd map max advance = 10 2024-02-19T15:26:39.029 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.030 INFO:teuthology.orchestra.run.smithi047.stdout: journal zero on create = true 2024-02-19T15:26:39.030 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.030 INFO:teuthology.orchestra.run.smithi047.stdout: filestore ondisk finisher threads = 3 2024-02-19T15:26:39.030 INFO:teuthology.orchestra.run.smithi047.stdout: filestore apply finisher threads = 3 2024-02-19T15:26:39.030 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.030 INFO:teuthology.orchestra.run.smithi047.stdout: bdev debug aio = true 2024-02-19T15:26:39.030 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug misdirected ops = true 2024-02-19T15:26:39.030 INFO:teuthology.orchestra.run.smithi047.stdout: bdev async discard = True 2024-02-19T15:26:39.030 INFO:teuthology.orchestra.run.smithi047.stdout: bdev enable discard = True 2024-02-19T15:26:39.030 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore allocator = bitmap 2024-02-19T15:26:39.030 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore block size = 96636764160 2024-02-19T15:26:39.030 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore fsck on mount = True 2024-02-19T15:26:39.030 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluefs = 1/20 2024-02-19T15:26:39.030 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluestore = 1/20 2024-02-19T15:26:39.030 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:26:39.031 INFO:teuthology.orchestra.run.smithi047.stdout: debug osd = 20 2024-02-19T15:26:39.031 INFO:teuthology.orchestra.run.smithi047.stdout: debug rocksdb = 4/10 2024-02-19T15:26:39.031 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd backfillfull_ratio = 0.85 2024-02-19T15:26:39.031 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd full ratio = 0.9 2024-02-19T15:26:39.031 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd nearfull ratio = 0.8 2024-02-19T15:26:39.031 INFO:teuthology.orchestra.run.smithi047.stdout: osd failsafe full ratio = 0.95 2024-02-19T15:26:39.031 INFO:teuthology.orchestra.run.smithi047.stdout: osd objectstore = bluestore 2024-02-19T15:26:39.031 INFO:teuthology.orchestra.run.smithi047.stdout: osd op complaint time = 180 2024-02-19T15:26:39.031 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.031 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr] 2024-02-19T15:26:39.031 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:26:39.031 INFO:teuthology.orchestra.run.smithi047.stdout: debug mgr = 20 2024-02-19T15:26:39.031 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-02-19T15:26:39.031 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-02-19T15:26:39.031 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:26:39.031 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:26:39.032 INFO:teuthology.orchestra.run.smithi047.stdout: mgr/telemetry/nag = false 2024-02-19T15:26:39.032 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.032 INFO:teuthology.orchestra.run.smithi047.stdout:[mon] 2024-02-19T15:26:39.032 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:26:39.032 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-02-19T15:26:39.032 INFO:teuthology.orchestra.run.smithi047.stdout: debug paxos = 20 2024-02-19T15:26:39.032 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-02-19T15:26:39.032 INFO:teuthology.orchestra.run.smithi047.stdout: mon data avail warn = 5 2024-02-19T15:26:39.032 INFO:teuthology.orchestra.run.smithi047.stdout: mon mgr mkfs grace = 240 2024-02-19T15:26:39.032 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:26:39.032 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd reporter subtree level = osd 2024-02-19T15:26:39.032 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd prime pg temp = true 2024-02-19T15:26:39.032 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:26:39.032 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.032 INFO:teuthology.orchestra.run.smithi047.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-19T15:26:39.033 INFO:teuthology.orchestra.run.smithi047.stdout: auth mon ticket ttl = 660 # 11m 2024-02-19T15:26:39.033 INFO:teuthology.orchestra.run.smithi047.stdout: auth service ticket ttl = 240 # 4m 2024-02-19T15:26:39.033 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.033 INFO:teuthology.orchestra.run.smithi047.stdout: # don't complain about insecure global_id in the test suite 2024-02-19T15:26:39.033 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-19T15:26:39.033 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-19T15:26:39.033 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.033 INFO:teuthology.orchestra.run.smithi047.stdout: # 1m isn't quite enough 2024-02-19T15:26:39.033 INFO:teuthology.orchestra.run.smithi047.stdout: mon_down_mkfs_grace = 2m 2024-02-19T15:26:39.033 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.033 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_filestore_osds = false 2024-02-19T15:26:39.033 INFO:teuthology.orchestra.run.smithi047.stdout: mon op complaint time = 120 2024-02-19T15:26:39.033 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.033 INFO:teuthology.orchestra.run.smithi047.stdout:[client] 2024-02-19T15:26:39.033 INFO:teuthology.orchestra.run.smithi047.stdout: rgw cache enabled = true 2024-02-19T15:26:39.034 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable ops log = true 2024-02-19T15:26:39.034 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable usage log = true 2024-02-19T15:26:39.034 INFO:teuthology.orchestra.run.smithi047.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-19T15:26:39.034 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-19T15:26:39.034 INFO:teuthology.orchestra.run.smithi047.stdout: client mount timeout = 600 2024-02-19T15:26:39.034 INFO:teuthology.orchestra.run.smithi047.stdout: debug client = 20 2024-02-19T15:26:39.034 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:26:39.034 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = "" 2024-02-19T15:26:39.034 INFO:teuthology.orchestra.run.smithi047.stdout: rados mon op timeout = 900 2024-02-19T15:26:39.034 INFO:teuthology.orchestra.run.smithi047.stdout: rados osd op timeout = 900 2024-02-19T15:26:39.034 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.a] 2024-02-19T15:26:39.034 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.b] 2024-02-19T15:26:39.034 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.c] 2024-02-19T15:26:39.034 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.a] 2024-02-19T15:26:39.034 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.c] 2024-02-19T15:26:39.034 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.b] 2024-02-19T15:26:39.035 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.d] 2024-02-19T15:26:39.035 INFO:teuthology.orchestra.run.smithi047.stdout:[mds] 2024-02-19T15:26:39.035 INFO:teuthology.orchestra.run.smithi047.stdout: debug mds = 20 2024-02-19T15:26:39.035 INFO:teuthology.orchestra.run.smithi047.stdout: debug mds balancer = 20 2024-02-19T15:26:39.035 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:26:39.035 INFO:teuthology.orchestra.run.smithi047.stdout: mds debug frag = True 2024-02-19T15:26:39.035 INFO:teuthology.orchestra.run.smithi047.stdout: mds debug scatterstat = True 2024-02-19T15:26:39.035 INFO:teuthology.orchestra.run.smithi047.stdout: mds op complaint time = 180 2024-02-19T15:26:39.035 INFO:teuthology.orchestra.run.smithi047.stdout: mds verify scatter = True 2024-02-19T15:26:39.035 INFO:teuthology.orchestra.run.smithi047.stdout: osd op complaint time = 180 2024-02-19T15:26:39.035 INFO:teuthology.orchestra.run.smithi047.stdout: rados mon op timeout = 900 2024-02-19T15:26:39.035 INFO:teuthology.orchestra.run.smithi047.stdout: rados osd op timeout = 900 2024-02-19T15:26:39.035 INFO:teuthology.orchestra.run.smithi047.stdout: mds_max_purge_files = 6 2024-02-19T15:26:39.035 INFO:teuthology.orchestra.run.smithi047.stdout: mds_max_purge_ops = 100000000 2024-02-19T15:26:39.036 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-02-19T15:26:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: chdir = "" 2024-02-19T15:26:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-19T15:26:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: auth supported = cephx 2024-02-19T15:26:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: filestore xattr use omap = true 2024-02-19T15:26:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: mon clock drift allowed = 1.000 2024-02-19T15:26:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: osd crush chooseleaf type = 0 2024-02-19T15:26:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: auth debug = true 2024-02-19T15:26:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on old message = true 2024-02-19T15:26:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on bug = true 2024-02-19T15:26:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-19T15:26:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: mon pg warn max object skew = 0 2024-02-19T15:26:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: # disable pg_autoscaler by default for new pools 2024-02-19T15:26:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-19T15:26:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default size = 2 2024-02-19T15:26:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow primary affinity = true 2024-02-19T15:26:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow pg remap = true 2024-02-19T15:26:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on legacy crush tunables = false 2024-02-19T15:26:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on crush straw calc version zero = false 2024-02-19T15:26:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on no sortbitwise = false 2024-02-19T15:26:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on osd down out interval zero = false 2024-02-19T15:26:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on too few osds = false 2024-02-19T15:26:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-19T15:26:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-19T15:26:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: mon_allow_pool_size_one = true 2024-02-19T15:26:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-19T15:26:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: osd default data pool replay window = 5 2024-02-19T15:26:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: mon allow pool delete = true 2024-02-19T15:26:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: mon cluster log file level = debug 2024-02-19T15:26:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: debug asserts on shutdown = true 2024-02-19T15:26:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: mon health detail to clog = false 2024-02-19T15:26:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: mon host = "172.21.15.47,172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790]" 2024-02-19T15:26:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on skipped message = False 2024-02-19T15:26:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e 2024-02-19T15:26:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.039 INFO:teuthology.orchestra.run.smithi116.stdout:[osd] 2024-02-19T15:26:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: osd journal size = 100 2024-02-19T15:26:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub load threshold = 5.0 2024-02-19T15:26:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub max interval = 600 2024-02-19T15:26:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock profile = high_recovery_ops 2024-02-19T15:26:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: osd recover clone overlap = true 2024-02-19T15:26:39.039 INFO:teuthology.orchestra.run.smithi116.stdout: osd recovery max chunk = 1048576 2024-02-19T15:26:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug shutdown = true 2024-02-19T15:26:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug op order = true 2024-02-19T15:26:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify stray on activate = true 2024-02-19T15:26:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug trim objects = true 2024-02-19T15:26:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: osd open classes on start = true 2024-02-19T15:26:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug pg log writeout = true 2024-02-19T15:26:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: osd deep scrub update digest min age = 30 2024-02-19T15:26:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: osd map max advance = 10 2024-02-19T15:26:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: journal zero on create = true 2024-02-19T15:26:39.040 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: filestore ondisk finisher threads = 3 2024-02-19T15:26:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: filestore apply finisher threads = 3 2024-02-19T15:26:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: bdev debug aio = true 2024-02-19T15:26:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug misdirected ops = true 2024-02-19T15:26:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: bdev async discard = True 2024-02-19T15:26:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: bdev enable discard = True 2024-02-19T15:26:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore allocator = bitmap 2024-02-19T15:26:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore block size = 96636764160 2024-02-19T15:26:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore fsck on mount = True 2024-02-19T15:26:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluefs = 1/20 2024-02-19T15:26:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluestore = 1/20 2024-02-19T15:26:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:26:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: debug osd = 20 2024-02-19T15:26:39.041 INFO:teuthology.orchestra.run.smithi116.stdout: debug rocksdb = 4/10 2024-02-19T15:26:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd backfillfull_ratio = 0.85 2024-02-19T15:26:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd full ratio = 0.9 2024-02-19T15:26:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd nearfull ratio = 0.8 2024-02-19T15:26:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: osd failsafe full ratio = 0.95 2024-02-19T15:26:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: osd objectstore = bluestore 2024-02-19T15:26:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2024-02-19T15:26:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.042 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr] 2024-02-19T15:26:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:26:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: debug mgr = 20 2024-02-19T15:26:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-02-19T15:26:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-02-19T15:26:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:26:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:26:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: mgr/telemetry/nag = false 2024-02-19T15:26:39.042 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.043 INFO:teuthology.orchestra.run.smithi116.stdout:[mon] 2024-02-19T15:26:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:26:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-02-19T15:26:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: debug paxos = 20 2024-02-19T15:26:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-02-19T15:26:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: mon data avail warn = 5 2024-02-19T15:26:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: mon mgr mkfs grace = 240 2024-02-19T15:26:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:26:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd reporter subtree level = osd 2024-02-19T15:26:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd prime pg temp = true 2024-02-19T15:26:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:26:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-19T15:26:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: auth mon ticket ttl = 660 # 11m 2024-02-19T15:26:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: auth service ticket ttl = 240 # 4m 2024-02-19T15:26:39.043 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: # don't complain about insecure global_id in the test suite 2024-02-19T15:26:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-19T15:26:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-19T15:26:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: # 1m isn't quite enough 2024-02-19T15:26:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: mon_down_mkfs_grace = 2m 2024-02-19T15:26:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_filestore_osds = false 2024-02-19T15:26:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: mon op complaint time = 120 2024-02-19T15:26:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.044 INFO:teuthology.orchestra.run.smithi116.stdout:[client] 2024-02-19T15:26:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: rgw cache enabled = true 2024-02-19T15:26:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable ops log = true 2024-02-19T15:26:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable usage log = true 2024-02-19T15:26:39.044 INFO:teuthology.orchestra.run.smithi116.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-19T15:26:39.045 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-19T15:26:39.045 INFO:teuthology.orchestra.run.smithi116.stdout: client mount timeout = 600 2024-02-19T15:26:39.045 INFO:teuthology.orchestra.run.smithi116.stdout: debug client = 20 2024-02-19T15:26:39.045 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:26:39.045 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = "" 2024-02-19T15:26:39.045 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2024-02-19T15:26:39.045 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2024-02-19T15:26:39.045 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.a] 2024-02-19T15:26:39.045 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.b] 2024-02-19T15:26:39.045 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.c] 2024-02-19T15:26:39.045 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a] 2024-02-19T15:26:39.045 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c] 2024-02-19T15:26:39.045 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b] 2024-02-19T15:26:39.045 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d] 2024-02-19T15:26:39.045 INFO:teuthology.orchestra.run.smithi116.stdout:[mds] 2024-02-19T15:26:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds = 20 2024-02-19T15:26:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds balancer = 20 2024-02-19T15:26:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:26:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug frag = True 2024-02-19T15:26:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug scatterstat = True 2024-02-19T15:26:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: mds op complaint time = 180 2024-02-19T15:26:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: mds verify scatter = True 2024-02-19T15:26:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2024-02-19T15:26:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2024-02-19T15:26:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2024-02-19T15:26:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: mds_max_purge_files = 6 2024-02-19T15:26:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: mds_max_purge_ops = 100000000 2024-02-19T15:26:39.048 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:26:39.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.227+0000 7fcc3f200640 1 Processor -- start 2024-02-19T15:26:39.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.227+0000 7fcc3f200640 1 -- start start 2024-02-19T15:26:39.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.231+0000 7fcc3f200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc400b7810 0x7fcc400d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.231+0000 7fcc3f200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcc400ce5b0 0x7fcc40160460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.231+0000 7fcc3f200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc400cc6e0 con 0x7fcc400ce150 2024-02-19T15:26:39.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.231+0000 7fcc3f200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc400cc980 con 0x7fcc400ce5b0 2024-02-19T15:26:39.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.231+0000 7fcc3f200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc400ccc60 con 0x7fcc400b7810 2024-02-19T15:26:39.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.231+0000 7fcc3e800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcc400ce5b0 0x7fcc40160460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.231+0000 7fcc3de00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc400b7810 0x7fcc400d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.231+0000 7fcc35400640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcc400ce150 0x7fcc4015fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:40182/0 (socket says 172.21.15.47:40182) 2024-02-19T15:26:39.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.231+0000 7fcc35400640 1 -- 172.21.15.47:0/595749127 learned_addr learned my addr 172.21.15.47:0/595749127 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:39.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.231+0000 7fcc3d400640 1 -- 172.21.15.47:0/595749127 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871131400 0 0) 0x7fcc400cc6e0 con 0x7fcc400ce150 2024-02-19T15:26:39.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.231+0000 7fcc3de00640 1 -- 172.21.15.47:0/595749127 >> v2:172.21.15.47:3300/0 conn(0x7fcc400ce5b0 msgr2=0x7fcc40160460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:39.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.231+0000 7fcc3de00640 1 --2- 172.21.15.47:0/595749127 >> v2:172.21.15.47:3300/0 conn(0x7fcc400ce5b0 0x7fcc40160460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.231+0000 7fcc3de00640 1 -- 172.21.15.47:0/595749127 >> v1:172.21.15.116:6789/0 conn(0x7fcc400ce150 legacy=0x7fcc4015fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.231+0000 7fcc3de00640 1 -- 172.21.15.47:0/595749127 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc400ccf40 con 0x7fcc400b7810 2024-02-19T15:26:39.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.231+0000 7fcc3e800640 1 --2- 172.21.15.47:0/595749127 >> v2:172.21.15.47:3300/0 conn(0x7fcc400ce5b0 0x7fcc40160460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:39.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.235+0000 7fcc3de00640 1 --2- 172.21.15.47:0/595749127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc400b7810 0x7fcc400d03d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcc30009dc0 tx=0x7fcc3001c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb8da06bb00a70cb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:39.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.235+0000 7fcc3d400640 1 -- 172.21.15.47:0/595749127 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc300a59e0 con 0x7fcc400b7810 2024-02-19T15:26:39.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.235+0000 7fcc3d400640 1 -- 172.21.15.47:0/595749127 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc300a5b40 con 0x7fcc400b7810 2024-02-19T15:26:39.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.235+0000 7fcc3d400640 1 -- 172.21.15.47:0/595749127 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc300ad440 con 0x7fcc400b7810 2024-02-19T15:26:39.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.235+0000 7fcc3f200640 1 -- 172.21.15.47:0/595749127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc400b7810 msgr2=0x7fcc400d03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.235+0000 7fcc3f200640 1 --2- 172.21.15.47:0/595749127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc400b7810 0x7fcc400d03d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcc30009dc0 tx=0x7fcc3001c920 comp rx=0 tx=0).stop 2024-02-19T15:26:39.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.235+0000 7fcc3f200640 1 -- 172.21.15.47:0/595749127 shutdown_connections 2024-02-19T15:26:39.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.235+0000 7fcc3f200640 1 --2- 172.21.15.47:0/595749127 >> v2:172.21.15.47:3300/0 conn(0x7fcc400ce5b0 0x7fcc40160460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.235+0000 7fcc3f200640 1 --2- 172.21.15.47:0/595749127 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc400b7810 0x7fcc400d03d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.235+0000 7fcc3f200640 1 -- 172.21.15.47:0/595749127 >> 172.21.15.47:0/595749127 conn(0x7fcc400d1b10 msgr2=0x7fcc400d1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:39.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.235+0000 7fcc3f200640 1 -- 172.21.15.47:0/595749127 shutdown_connections 2024-02-19T15:26:39.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.235+0000 7fcc3f200640 1 -- 172.21.15.47:0/595749127 wait complete. 2024-02-19T15:26:39.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.235+0000 7fcc3f200640 1 Processor -- start 2024-02-19T15:26:39.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.235+0000 7fcc3f200640 1 -- start start 2024-02-19T15:26:39.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.235+0000 7fcc3f200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc400b7810 0x7fcc401512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.235+0000 7fcc3f200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc400ce150 0x7fcc400c6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.235+0000 7fcc3f200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc400ce5b0 0x7fcc400c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.235+0000 7fcc3f200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc401517f0 con 0x7fcc400ce150 2024-02-19T15:26:39.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.235+0000 7fcc3f200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc400c7900 con 0x7fcc400ce5b0 2024-02-19T15:26:39.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.239+0000 7fcc3f200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc400c7be0 con 0x7fcc400b7810 2024-02-19T15:26:39.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.239+0000 7fcc35400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc400ce150 0x7fcc400c6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.239+0000 7fcc35400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc400ce150 0x7fcc400c6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40784/0 (socket says 172.21.15.47:40784) 2024-02-19T15:26:39.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.239+0000 7fcc35400640 1 -- 172.21.15.47:0/265773077 learned_addr learned my addr 172.21.15.47:0/265773077 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:39.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.239+0000 7fcc3de00640 1 --2- 172.21.15.47:0/265773077 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc400b7810 0x7fcc401512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.239+0000 7fcc3e800640 1 --2- 172.21.15.47:0/265773077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc400ce5b0 0x7fcc400c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.239+0000 7fcc35400640 1 -- 172.21.15.47:0/265773077 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc400b7810 msgr2=0x7fcc401512b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.239+0000 7fcc35400640 1 --2- 172.21.15.47:0/265773077 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc400b7810 0x7fcc401512b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.239+0000 7fcc35400640 1 -- 172.21.15.47:0/265773077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc400ce5b0 msgr2=0x7fcc400c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.239+0000 7fcc35400640 1 --2- 172.21.15.47:0/265773077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc400ce5b0 0x7fcc400c72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.239+0000 7fcc35400640 1 -- 172.21.15.47:0/265773077 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc38009420 con 0x7fcc400ce150 2024-02-19T15:26:39.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.239+0000 7fcc35400640 1 --2- 172.21.15.47:0/265773077 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc400ce150 0x7fcc400c6db0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fcc280011f0 tx=0x7fcc2800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:39.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.239+0000 7fcc36a00640 1 -- 172.21.15.47:0/265773077 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc280074e0 con 0x7fcc400ce150 2024-02-19T15:26:39.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.239+0000 7fcc36a00640 1 -- 172.21.15.47:0/265773077 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc28007640 con 0x7fcc400ce150 2024-02-19T15:26:39.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.239+0000 7fcc3f200640 1 -- 172.21.15.47:0/265773077 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc30009a70 con 0x7fcc400ce150 2024-02-19T15:26:39.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.239+0000 7fcc36a00640 1 -- 172.21.15.47:0/265773077 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc280049f0 con 0x7fcc400ce150 2024-02-19T15:26:39.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.239+0000 7fcc3f200640 1 -- 172.21.15.47:0/265773077 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc400c81a0 con 0x7fcc400ce150 2024-02-19T15:26:39.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.239+0000 7fcc3f200640 1 -- 172.21.15.47:0/265773077 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc40075450 con 0x7fcc400ce150 2024-02-19T15:26:39.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.239+0000 7fcc36a00640 1 -- 172.21.15.47:0/265773077 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fcc28004b50 con 0x7fcc400ce150 2024-02-19T15:26:39.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.239+0000 7fcc36a00640 1 --2- 172.21.15.47:0/265773077 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcc1c066e40 0x7fcc1c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.243+0000 7fcc36a00640 1 -- 172.21.15.47:0/265773077 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc28084870 con 0x7fcc400ce150 2024-02-19T15:26:39.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.243+0000 7fcc3de00640 1 --2- 172.21.15.47:0/265773077 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcc1c066e40 0x7fcc1c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.243+0000 7fcc3de00640 1 --2- 172.21.15.47:0/265773077 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcc1c066e40 0x7fcc1c069300 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fcc300af000 tx=0x7fcc300a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:39.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.243+0000 7fcc36a00640 1 -- 172.21.15.47:0/265773077 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fcc280515c0 con 0x7fcc400ce150 2024-02-19T15:26:39.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.407+0000 7fcc3f200640 1 -- 172.21.15.47:0/265773077 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcc400cf370 con 0x7fcc400ce150 2024-02-19T15:26:39.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.407+0000 7fcc36a00640 1 -- 172.21.15.47:0/265773077 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcc28054c80 con 0x7fcc400ce150 2024-02-19T15:26:39.412 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:26:39.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.407+0000 7fcc3f200640 1 -- 172.21.15.47:0/265773077 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcc1c066e40 msgr2=0x7fcc1c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.407+0000 7fcc3f200640 1 --2- 172.21.15.47:0/265773077 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcc1c066e40 0x7fcc1c069300 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fcc300af000 tx=0x7fcc300a7070 comp rx=0 tx=0).stop 2024-02-19T15:26:39.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.407+0000 7fcc3f200640 1 -- 172.21.15.47:0/265773077 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc400ce150 msgr2=0x7fcc400c6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.407+0000 7fcc3f200640 1 --2- 172.21.15.47:0/265773077 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc400ce150 0x7fcc400c6db0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fcc280011f0 tx=0x7fcc2800c8f0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.407+0000 7fcc3f200640 1 -- 172.21.15.47:0/265773077 shutdown_connections 2024-02-19T15:26:39.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.407+0000 7fcc3f200640 1 --2- 172.21.15.47:0/265773077 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcc1c066e40 0x7fcc1c069300 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.407+0000 7fcc3f200640 1 --2- 172.21.15.47:0/265773077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc400ce5b0 0x7fcc400c72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.407+0000 7fcc3f200640 1 --2- 172.21.15.47:0/265773077 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc400ce150 0x7fcc400c6db0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.407+0000 7fcc3f200640 1 --2- 172.21.15.47:0/265773077 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc400b7810 0x7fcc401512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.407+0000 7fcc3f200640 1 -- 172.21.15.47:0/265773077 >> 172.21.15.47:0/265773077 conn(0x7fcc400d1b10 msgr2=0x7fcc400d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:39.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.407+0000 7fcc3f200640 1 -- 172.21.15.47:0/265773077 shutdown_connections 2024-02-19T15:26:39.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.407+0000 7fcc3f200640 1 -- 172.21.15.47:0/265773077 wait complete. 2024-02-19T15:26:39.434 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-19T15:26:39.437 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-19T15:26:39.478 INFO:teuthology.orchestra.run.smithi047.stdout:[global] 2024-02-19T15:26:39.478 INFO:teuthology.orchestra.run.smithi047.stdout: chdir = "" 2024-02-19T15:26:39.478 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-19T15:26:39.478 INFO:teuthology.orchestra.run.smithi047.stdout: auth supported = cephx 2024-02-19T15:26:39.478 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.478 INFO:teuthology.orchestra.run.smithi047.stdout: filestore xattr use omap = true 2024-02-19T15:26:39.479 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.479 INFO:teuthology.orchestra.run.smithi047.stdout: mon clock drift allowed = 1.000 2024-02-19T15:26:39.479 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.479 INFO:teuthology.orchestra.run.smithi047.stdout: osd crush chooseleaf type = 0 2024-02-19T15:26:39.479 INFO:teuthology.orchestra.run.smithi047.stdout: auth debug = true 2024-02-19T15:26:39.479 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.479 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on old message = true 2024-02-19T15:26:39.479 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on bug = true 2024-02-19T15:26:39.479 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.479 INFO:teuthology.orchestra.run.smithi047.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-19T15:26:39.479 INFO:teuthology.orchestra.run.smithi047.stdout: mon pg warn max object skew = 0 2024-02-19T15:26:39.479 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.479 INFO:teuthology.orchestra.run.smithi047.stdout: # disable pg_autoscaler by default for new pools 2024-02-19T15:26:39.479 INFO:teuthology.orchestra.run.smithi047.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-19T15:26:39.479 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.479 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default size = 2 2024-02-19T15:26:39.480 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.480 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow primary affinity = true 2024-02-19T15:26:39.480 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow pg remap = true 2024-02-19T15:26:39.480 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on legacy crush tunables = false 2024-02-19T15:26:39.480 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on crush straw calc version zero = false 2024-02-19T15:26:39.480 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on no sortbitwise = false 2024-02-19T15:26:39.480 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on osd down out interval zero = false 2024-02-19T15:26:39.480 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on too few osds = false 2024-02-19T15:26:39.480 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-19T15:26:39.480 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-19T15:26:39.480 INFO:teuthology.orchestra.run.smithi047.stdout: mon_allow_pool_size_one = true 2024-02-19T15:26:39.480 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.480 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-19T15:26:39.480 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.481 INFO:teuthology.orchestra.run.smithi047.stdout: osd default data pool replay window = 5 2024-02-19T15:26:39.481 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.481 INFO:teuthology.orchestra.run.smithi047.stdout: mon allow pool delete = true 2024-02-19T15:26:39.481 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.481 INFO:teuthology.orchestra.run.smithi047.stdout: mon cluster log file level = debug 2024-02-19T15:26:39.481 INFO:teuthology.orchestra.run.smithi047.stdout: debug asserts on shutdown = true 2024-02-19T15:26:39.481 INFO:teuthology.orchestra.run.smithi047.stdout: mon health detail to clog = false 2024-02-19T15:26:39.481 INFO:teuthology.orchestra.run.smithi047.stdout: mon host = "172.21.15.47,172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790]" 2024-02-19T15:26:39.481 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on skipped message = False 2024-02-19T15:26:39.481 INFO:teuthology.orchestra.run.smithi047.stdout: fsid = ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e 2024-02-19T15:26:39.481 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.481 INFO:teuthology.orchestra.run.smithi047.stdout:[osd] 2024-02-19T15:26:39.481 INFO:teuthology.orchestra.run.smithi047.stdout: osd journal size = 100 2024-02-19T15:26:39.481 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.482 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub load threshold = 5.0 2024-02-19T15:26:39.482 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub max interval = 600 2024-02-19T15:26:39.482 INFO:teuthology.orchestra.run.smithi047.stdout: osd mclock profile = high_recovery_ops 2024-02-19T15:26:39.482 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.482 INFO:teuthology.orchestra.run.smithi047.stdout: osd recover clone overlap = true 2024-02-19T15:26:39.482 INFO:teuthology.orchestra.run.smithi047.stdout: osd recovery max chunk = 1048576 2024-02-19T15:26:39.482 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.482 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug shutdown = true 2024-02-19T15:26:39.482 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug op order = true 2024-02-19T15:26:39.482 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify stray on activate = true 2024-02-19T15:26:39.482 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug trim objects = true 2024-02-19T15:26:39.482 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.482 INFO:teuthology.orchestra.run.smithi047.stdout: osd open classes on start = true 2024-02-19T15:26:39.482 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug pg log writeout = true 2024-02-19T15:26:39.482 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.482 INFO:teuthology.orchestra.run.smithi047.stdout: osd deep scrub update digest min age = 30 2024-02-19T15:26:39.483 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.483 INFO:teuthology.orchestra.run.smithi047.stdout: osd map max advance = 10 2024-02-19T15:26:39.483 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.483 INFO:teuthology.orchestra.run.smithi047.stdout: journal zero on create = true 2024-02-19T15:26:39.483 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.483 INFO:teuthology.orchestra.run.smithi047.stdout: filestore ondisk finisher threads = 3 2024-02-19T15:26:39.483 INFO:teuthology.orchestra.run.smithi047.stdout: filestore apply finisher threads = 3 2024-02-19T15:26:39.483 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.483 INFO:teuthology.orchestra.run.smithi047.stdout: bdev debug aio = true 2024-02-19T15:26:39.483 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug misdirected ops = true 2024-02-19T15:26:39.483 INFO:teuthology.orchestra.run.smithi047.stdout: bdev async discard = True 2024-02-19T15:26:39.483 INFO:teuthology.orchestra.run.smithi047.stdout: bdev enable discard = True 2024-02-19T15:26:39.483 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore allocator = bitmap 2024-02-19T15:26:39.483 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore block size = 96636764160 2024-02-19T15:26:39.483 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore fsck on mount = True 2024-02-19T15:26:39.484 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluefs = 1/20 2024-02-19T15:26:39.484 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluestore = 1/20 2024-02-19T15:26:39.484 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:26:39.484 INFO:teuthology.orchestra.run.smithi047.stdout: debug osd = 20 2024-02-19T15:26:39.484 INFO:teuthology.orchestra.run.smithi047.stdout: debug rocksdb = 4/10 2024-02-19T15:26:39.484 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd backfillfull_ratio = 0.85 2024-02-19T15:26:39.484 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd full ratio = 0.9 2024-02-19T15:26:39.484 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd nearfull ratio = 0.8 2024-02-19T15:26:39.484 INFO:teuthology.orchestra.run.smithi047.stdout: osd failsafe full ratio = 0.95 2024-02-19T15:26:39.484 INFO:teuthology.orchestra.run.smithi047.stdout: osd objectstore = bluestore 2024-02-19T15:26:39.484 INFO:teuthology.orchestra.run.smithi047.stdout: osd op complaint time = 180 2024-02-19T15:26:39.484 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.484 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr] 2024-02-19T15:26:39.484 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:26:39.484 INFO:teuthology.orchestra.run.smithi047.stdout: debug mgr = 20 2024-02-19T15:26:39.485 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-02-19T15:26:39.485 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-02-19T15:26:39.485 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:26:39.485 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:26:39.485 INFO:teuthology.orchestra.run.smithi047.stdout: mgr/telemetry/nag = false 2024-02-19T15:26:39.485 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.485 INFO:teuthology.orchestra.run.smithi047.stdout:[mon] 2024-02-19T15:26:39.485 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:26:39.485 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-02-19T15:26:39.485 INFO:teuthology.orchestra.run.smithi047.stdout: debug paxos = 20 2024-02-19T15:26:39.485 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-02-19T15:26:39.485 INFO:teuthology.orchestra.run.smithi047.stdout: mon data avail warn = 5 2024-02-19T15:26:39.485 INFO:teuthology.orchestra.run.smithi047.stdout: mon mgr mkfs grace = 240 2024-02-19T15:26:39.485 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:26:39.485 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd reporter subtree level = osd 2024-02-19T15:26:39.485 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd prime pg temp = true 2024-02-19T15:26:39.486 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:26:39.486 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.486 INFO:teuthology.orchestra.run.smithi047.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-19T15:26:39.486 INFO:teuthology.orchestra.run.smithi047.stdout: auth mon ticket ttl = 660 # 11m 2024-02-19T15:26:39.486 INFO:teuthology.orchestra.run.smithi047.stdout: auth service ticket ttl = 240 # 4m 2024-02-19T15:26:39.486 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.486 INFO:teuthology.orchestra.run.smithi047.stdout: # don't complain about insecure global_id in the test suite 2024-02-19T15:26:39.486 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-19T15:26:39.486 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-19T15:26:39.486 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.486 INFO:teuthology.orchestra.run.smithi047.stdout: # 1m isn't quite enough 2024-02-19T15:26:39.486 INFO:teuthology.orchestra.run.smithi047.stdout: mon_down_mkfs_grace = 2m 2024-02-19T15:26:39.486 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.486 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_filestore_osds = false 2024-02-19T15:26:39.486 INFO:teuthology.orchestra.run.smithi047.stdout: mon op complaint time = 120 2024-02-19T15:26:39.486 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:39.486 INFO:teuthology.orchestra.run.smithi047.stdout:[client] 2024-02-19T15:26:39.487 INFO:teuthology.orchestra.run.smithi047.stdout: rgw cache enabled = true 2024-02-19T15:26:39.487 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable ops log = true 2024-02-19T15:26:39.487 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable usage log = true 2024-02-19T15:26:39.487 INFO:teuthology.orchestra.run.smithi047.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-19T15:26:39.487 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-19T15:26:39.487 INFO:teuthology.orchestra.run.smithi047.stdout: client mount timeout = 600 2024-02-19T15:26:39.487 INFO:teuthology.orchestra.run.smithi047.stdout: debug client = 20 2024-02-19T15:26:39.487 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:26:39.487 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = "" 2024-02-19T15:26:39.487 INFO:teuthology.orchestra.run.smithi047.stdout: rados mon op timeout = 900 2024-02-19T15:26:39.487 INFO:teuthology.orchestra.run.smithi047.stdout: rados osd op timeout = 900 2024-02-19T15:26:39.487 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.a] 2024-02-19T15:26:39.487 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.b] 2024-02-19T15:26:39.487 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.c] 2024-02-19T15:26:39.487 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.a] 2024-02-19T15:26:39.487 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.c] 2024-02-19T15:26:39.487 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.b] 2024-02-19T15:26:39.488 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.d] 2024-02-19T15:26:39.488 INFO:teuthology.orchestra.run.smithi047.stdout:[mds] 2024-02-19T15:26:39.488 INFO:teuthology.orchestra.run.smithi047.stdout: debug mds = 20 2024-02-19T15:26:39.488 INFO:teuthology.orchestra.run.smithi047.stdout: debug mds balancer = 20 2024-02-19T15:26:39.488 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:26:39.488 INFO:teuthology.orchestra.run.smithi047.stdout: mds debug frag = True 2024-02-19T15:26:39.488 INFO:teuthology.orchestra.run.smithi047.stdout: mds debug scatterstat = True 2024-02-19T15:26:39.488 INFO:teuthology.orchestra.run.smithi047.stdout: mds op complaint time = 180 2024-02-19T15:26:39.488 INFO:teuthology.orchestra.run.smithi047.stdout: mds verify scatter = True 2024-02-19T15:26:39.488 INFO:teuthology.orchestra.run.smithi047.stdout: osd op complaint time = 180 2024-02-19T15:26:39.488 INFO:teuthology.orchestra.run.smithi047.stdout: rados mon op timeout = 900 2024-02-19T15:26:39.488 INFO:teuthology.orchestra.run.smithi047.stdout: rados osd op timeout = 900 2024-02-19T15:26:39.488 INFO:teuthology.orchestra.run.smithi047.stdout: mds_max_purge_files = 6 2024-02-19T15:26:39.488 INFO:teuthology.orchestra.run.smithi047.stdout: mds_max_purge_ops = 100000000 2024-02-19T15:26:39.489 INFO:teuthology.orchestra.run.smithi047.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-02-19T15:26:39.489 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-02-19T15:26:39.489 INFO:teuthology.orchestra.run.smithi116.stdout: chdir = "" 2024-02-19T15:26:39.489 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-19T15:26:39.489 INFO:teuthology.orchestra.run.smithi116.stdout: auth supported = cephx 2024-02-19T15:26:39.489 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.489 INFO:teuthology.orchestra.run.smithi116.stdout: filestore xattr use omap = true 2024-02-19T15:26:39.490 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.490 INFO:teuthology.orchestra.run.smithi116.stdout: mon clock drift allowed = 1.000 2024-02-19T15:26:39.490 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.490 INFO:teuthology.orchestra.run.smithi116.stdout: osd crush chooseleaf type = 0 2024-02-19T15:26:39.490 INFO:teuthology.orchestra.run.smithi116.stdout: auth debug = true 2024-02-19T15:26:39.490 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.490 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on old message = true 2024-02-19T15:26:39.490 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on bug = true 2024-02-19T15:26:39.490 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.490 INFO:teuthology.orchestra.run.smithi116.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-19T15:26:39.490 INFO:teuthology.orchestra.run.smithi116.stdout: mon pg warn max object skew = 0 2024-02-19T15:26:39.490 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.490 INFO:teuthology.orchestra.run.smithi116.stdout: # disable pg_autoscaler by default for new pools 2024-02-19T15:26:39.490 INFO:teuthology.orchestra.run.smithi116.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-19T15:26:39.490 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.491 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default size = 2 2024-02-19T15:26:39.491 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.491 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow primary affinity = true 2024-02-19T15:26:39.491 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow pg remap = true 2024-02-19T15:26:39.491 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on legacy crush tunables = false 2024-02-19T15:26:39.491 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on crush straw calc version zero = false 2024-02-19T15:26:39.491 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on no sortbitwise = false 2024-02-19T15:26:39.491 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on osd down out interval zero = false 2024-02-19T15:26:39.491 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on too few osds = false 2024-02-19T15:26:39.491 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-19T15:26:39.491 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-19T15:26:39.491 INFO:teuthology.orchestra.run.smithi116.stdout: mon_allow_pool_size_one = true 2024-02-19T15:26:39.491 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.491 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-19T15:26:39.492 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.492 INFO:teuthology.orchestra.run.smithi116.stdout: osd default data pool replay window = 5 2024-02-19T15:26:39.492 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.492 INFO:teuthology.orchestra.run.smithi116.stdout: mon allow pool delete = true 2024-02-19T15:26:39.492 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.492 INFO:teuthology.orchestra.run.smithi116.stdout: mon cluster log file level = debug 2024-02-19T15:26:39.492 INFO:teuthology.orchestra.run.smithi116.stdout: debug asserts on shutdown = true 2024-02-19T15:26:39.492 INFO:teuthology.orchestra.run.smithi116.stdout: mon health detail to clog = false 2024-02-19T15:26:39.492 INFO:teuthology.orchestra.run.smithi116.stdout: mon host = "172.21.15.47,172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790]" 2024-02-19T15:26:39.492 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on skipped message = False 2024-02-19T15:26:39.492 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e 2024-02-19T15:26:39.492 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.492 INFO:teuthology.orchestra.run.smithi116.stdout:[osd] 2024-02-19T15:26:39.492 INFO:teuthology.orchestra.run.smithi116.stdout: osd journal size = 100 2024-02-19T15:26:39.493 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.493 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub load threshold = 5.0 2024-02-19T15:26:39.493 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub max interval = 600 2024-02-19T15:26:39.493 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock profile = high_recovery_ops 2024-02-19T15:26:39.493 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.493 INFO:teuthology.orchestra.run.smithi116.stdout: osd recover clone overlap = true 2024-02-19T15:26:39.493 INFO:teuthology.orchestra.run.smithi116.stdout: osd recovery max chunk = 1048576 2024-02-19T15:26:39.493 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.493 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug shutdown = true 2024-02-19T15:26:39.493 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug op order = true 2024-02-19T15:26:39.493 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify stray on activate = true 2024-02-19T15:26:39.493 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug trim objects = true 2024-02-19T15:26:39.493 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.493 INFO:teuthology.orchestra.run.smithi116.stdout: osd open classes on start = true 2024-02-19T15:26:39.494 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug pg log writeout = true 2024-02-19T15:26:39.494 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.494 INFO:teuthology.orchestra.run.smithi116.stdout: osd deep scrub update digest min age = 30 2024-02-19T15:26:39.494 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.494 INFO:teuthology.orchestra.run.smithi116.stdout: osd map max advance = 10 2024-02-19T15:26:39.494 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.494 INFO:teuthology.orchestra.run.smithi116.stdout: journal zero on create = true 2024-02-19T15:26:39.494 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.494 INFO:teuthology.orchestra.run.smithi116.stdout: filestore ondisk finisher threads = 3 2024-02-19T15:26:39.494 INFO:teuthology.orchestra.run.smithi116.stdout: filestore apply finisher threads = 3 2024-02-19T15:26:39.494 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.494 INFO:teuthology.orchestra.run.smithi116.stdout: bdev debug aio = true 2024-02-19T15:26:39.494 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug misdirected ops = true 2024-02-19T15:26:39.494 INFO:teuthology.orchestra.run.smithi116.stdout: bdev async discard = True 2024-02-19T15:26:39.494 INFO:teuthology.orchestra.run.smithi116.stdout: bdev enable discard = True 2024-02-19T15:26:39.494 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore allocator = bitmap 2024-02-19T15:26:39.495 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore block size = 96636764160 2024-02-19T15:26:39.495 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore fsck on mount = True 2024-02-19T15:26:39.495 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluefs = 1/20 2024-02-19T15:26:39.495 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluestore = 1/20 2024-02-19T15:26:39.495 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:26:39.495 INFO:teuthology.orchestra.run.smithi116.stdout: debug osd = 20 2024-02-19T15:26:39.495 INFO:teuthology.orchestra.run.smithi116.stdout: debug rocksdb = 4/10 2024-02-19T15:26:39.495 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd backfillfull_ratio = 0.85 2024-02-19T15:26:39.495 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd full ratio = 0.9 2024-02-19T15:26:39.495 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd nearfull ratio = 0.8 2024-02-19T15:26:39.495 INFO:teuthology.orchestra.run.smithi116.stdout: osd failsafe full ratio = 0.95 2024-02-19T15:26:39.495 INFO:teuthology.orchestra.run.smithi116.stdout: osd objectstore = bluestore 2024-02-19T15:26:39.495 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2024-02-19T15:26:39.495 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.495 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr] 2024-02-19T15:26:39.496 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:26:39.496 INFO:teuthology.orchestra.run.smithi116.stdout: debug mgr = 20 2024-02-19T15:26:39.496 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-02-19T15:26:39.496 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-02-19T15:26:39.496 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:26:39.496 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:26:39.496 INFO:teuthology.orchestra.run.smithi116.stdout: mgr/telemetry/nag = false 2024-02-19T15:26:39.496 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.496 INFO:teuthology.orchestra.run.smithi116.stdout:[mon] 2024-02-19T15:26:39.496 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:26:39.496 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-02-19T15:26:39.496 INFO:teuthology.orchestra.run.smithi116.stdout: debug paxos = 20 2024-02-19T15:26:39.496 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-02-19T15:26:39.496 INFO:teuthology.orchestra.run.smithi116.stdout: mon data avail warn = 5 2024-02-19T15:26:39.497 INFO:teuthology.orchestra.run.smithi116.stdout: mon mgr mkfs grace = 240 2024-02-19T15:26:39.497 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:26:39.497 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd reporter subtree level = osd 2024-02-19T15:26:39.497 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd prime pg temp = true 2024-02-19T15:26:39.497 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:26:39.497 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.497 INFO:teuthology.orchestra.run.smithi116.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-19T15:26:39.497 INFO:teuthology.orchestra.run.smithi116.stdout: auth mon ticket ttl = 660 # 11m 2024-02-19T15:26:39.497 INFO:teuthology.orchestra.run.smithi116.stdout: auth service ticket ttl = 240 # 4m 2024-02-19T15:26:39.497 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.497 INFO:teuthology.orchestra.run.smithi116.stdout: # don't complain about insecure global_id in the test suite 2024-02-19T15:26:39.497 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-19T15:26:39.497 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-19T15:26:39.497 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.497 INFO:teuthology.orchestra.run.smithi116.stdout: # 1m isn't quite enough 2024-02-19T15:26:39.498 INFO:teuthology.orchestra.run.smithi116.stdout: mon_down_mkfs_grace = 2m 2024-02-19T15:26:39.498 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.498 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_filestore_osds = false 2024-02-19T15:26:39.498 INFO:teuthology.orchestra.run.smithi116.stdout: mon op complaint time = 120 2024-02-19T15:26:39.498 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:26:39.498 INFO:teuthology.orchestra.run.smithi116.stdout:[client] 2024-02-19T15:26:39.498 INFO:teuthology.orchestra.run.smithi116.stdout: rgw cache enabled = true 2024-02-19T15:26:39.498 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable ops log = true 2024-02-19T15:26:39.498 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable usage log = true 2024-02-19T15:26:39.498 INFO:teuthology.orchestra.run.smithi116.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-19T15:26:39.498 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-19T15:26:39.498 INFO:teuthology.orchestra.run.smithi116.stdout: client mount timeout = 600 2024-02-19T15:26:39.498 INFO:teuthology.orchestra.run.smithi116.stdout: debug client = 20 2024-02-19T15:26:39.498 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:26:39.498 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = "" 2024-02-19T15:26:39.498 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2024-02-19T15:26:39.499 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2024-02-19T15:26:39.499 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.a] 2024-02-19T15:26:39.499 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.b] 2024-02-19T15:26:39.499 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.c] 2024-02-19T15:26:39.499 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a] 2024-02-19T15:26:39.499 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c] 2024-02-19T15:26:39.499 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b] 2024-02-19T15:26:39.499 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d] 2024-02-19T15:26:39.499 INFO:teuthology.orchestra.run.smithi116.stdout:[mds] 2024-02-19T15:26:39.499 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds = 20 2024-02-19T15:26:39.499 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds balancer = 20 2024-02-19T15:26:39.499 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:26:39.499 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug frag = True 2024-02-19T15:26:39.499 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug scatterstat = True 2024-02-19T15:26:39.499 INFO:teuthology.orchestra.run.smithi116.stdout: mds op complaint time = 180 2024-02-19T15:26:39.499 INFO:teuthology.orchestra.run.smithi116.stdout: mds verify scatter = True 2024-02-19T15:26:39.500 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2024-02-19T15:26:39.500 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2024-02-19T15:26:39.500 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2024-02-19T15:26:39.500 INFO:teuthology.orchestra.run.smithi116.stdout: mds_max_purge_files = 6 2024-02-19T15:26:39.500 INFO:teuthology.orchestra.run.smithi116.stdout: mds_max_purge_ops = 100000000 2024-02-19T15:26:39.500 INFO:teuthology.orchestra.run.smithi116.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-02-19T15:26:39.502 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T15:26:39.502 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:26:39.503 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T15:26:39.503 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:26:39.503 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T15:26:39.503 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:26:39.503 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T15:26:39.503 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:26:39.559 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T15:26:39.559 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-02-19T15:26:39.560 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T15:26:39.560 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-02-19T15:26:39.565 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T15:26:39.566 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-02-19T15:26:39.566 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T15:26:39.566 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-02-19T15:26:39.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f68e1a00640 1 Processor -- start 2024-02-19T15:26:39.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f1428800640 1 Processor -- start 2024-02-19T15:26:39.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f68e1a00640 1 -- start start 2024-02-19T15:26:39.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f1428800640 1 -- start start 2024-02-19T15:26:39.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f68e1a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68dc145790 0x7f68dc0728f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f68e1a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f68dc0cf760 0x7f68dc15c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f1428800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f142015e0d0 0x7f14200b1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f1428800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f142006c020 0x7f142006c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f1428800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14200d02b0 con 0x7f14200d1cf0 2024-02-19T15:26:39.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f1428800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f142006cac0 con 0x7f142006c020 2024-02-19T15:26:39.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f1428800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f142006cda0 con 0x7f142015e0d0 2024-02-19T15:26:39.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f68e1a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f68dc0b5190 0x7f68dc0b5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f68e1a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68dc1609b0 con 0x7f68dc0cf760 2024-02-19T15:26:39.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f68e1a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68dc0b5b40 con 0x7f68dc0b5190 2024-02-19T15:26:39.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f68e1a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68dc15fda0 con 0x7f68dc145790 2024-02-19T15:26:39.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f68da000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f68dc0cf760 0x7f68dc15c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f68daa00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68dc145790 0x7f68dc0728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f68da000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f68dc0cf760 0x7f68dc15c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40792/0 (socket says 172.21.15.47:40792) 2024-02-19T15:26:39.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f68da000640 1 -- 172.21.15.47:0/2898058365 learned_addr learned my addr 172.21.15.47:0/2898058365 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:39.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f68db400640 1 --2- 172.21.15.47:0/2898058365 >> v2:172.21.15.116:3300/0 conn(0x7f68dc0b5190 0x7f68dc0b5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f1425400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f142006c020 0x7f142006c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f1425400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f142006c020 0x7f142006c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40790/0 (socket says 172.21.15.47:40790) 2024-02-19T15:26:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f1425e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f142015e0d0 0x7f14200b1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f1425400640 1 -- 172.21.15.47:0/373205765 learned_addr learned my addr 172.21.15.47:0/373205765 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f68db400640 1 -- 172.21.15.47:0/2898058365 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68dc145790 msgr2=0x7f68dc0728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f68db400640 1 --2- 172.21.15.47:0/2898058365 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68dc145790 0x7f68dc0728f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f68db400640 1 -- 172.21.15.47:0/2898058365 >> v2:172.21.15.47:3300/0 conn(0x7f68dc0cf760 msgr2=0x7f68dc15c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f68db400640 1 --2- 172.21.15.47:0/2898058365 >> v2:172.21.15.47:3300/0 conn(0x7f68dc0cf760 0x7f68dc15c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f1425e00640 1 -- 172.21.15.47:0/373205765 >> v2:172.21.15.47:3300/0 conn(0x7f142006c020 msgr2=0x7f142006c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:39.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f1425e00640 1 --2- 172.21.15.47:0/373205765 >> v2:172.21.15.47:3300/0 conn(0x7f142006c020 0x7f142006c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f1425e00640 1 -- 172.21.15.47:0/373205765 >> v1:172.21.15.116:6789/0 conn(0x7f14200d1cf0 legacy=0x7f14200d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f1425e00640 1 -- 172.21.15.47:0/373205765 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142006d080 con 0x7f142015e0d0 2024-02-19T15:26:39.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.703+0000 7f68db400640 1 -- 172.21.15.47:0/2898058365 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68dc160080 con 0x7f68dc0b5190 2024-02-19T15:26:39.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.707+0000 7f1425e00640 1 --2- 172.21.15.47:0/373205765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f142015e0d0 0x7f14200b1790 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f14200cf8d0 tx=0x7f141401c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdf0b21912b957e5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:39.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.707+0000 7f1424a00640 1 -- 172.21.15.47:0/373205765 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14140ab670 con 0x7f142015e0d0 2024-02-19T15:26:39.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.707+0000 7f1424a00640 1 -- 172.21.15.47:0/373205765 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14140ab7d0 con 0x7f142015e0d0 2024-02-19T15:26:39.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.707+0000 7f1424a00640 1 -- 172.21.15.47:0/373205765 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14140abb60 con 0x7f142015e0d0 2024-02-19T15:26:39.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.707+0000 7f1428800640 1 -- 172.21.15.47:0/373205765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f142015e0d0 msgr2=0x7f14200b1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.707+0000 7f1428800640 1 --2- 172.21.15.47:0/373205765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f142015e0d0 0x7f14200b1790 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f14200cf8d0 tx=0x7f141401c450 comp rx=0 tx=0).stop 2024-02-19T15:26:39.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.707+0000 7f1428800640 1 -- 172.21.15.47:0/373205765 shutdown_connections 2024-02-19T15:26:39.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.707+0000 7f1428800640 1 --2- 172.21.15.47:0/373205765 >> v2:172.21.15.47:3300/0 conn(0x7f142006c020 0x7f142006c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.707+0000 7f1428800640 1 --2- 172.21.15.47:0/373205765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f142015e0d0 0x7f14200b1790 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.707+0000 7f1428800640 1 -- 172.21.15.47:0/373205765 >> 172.21.15.47:0/373205765 conn(0x7f142009aa70 msgr2=0x7f142009ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:39.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.707+0000 7f1428800640 1 -- 172.21.15.47:0/373205765 shutdown_connections 2024-02-19T15:26:39.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.707+0000 7f1428800640 1 -- 172.21.15.47:0/373205765 wait complete. 2024-02-19T15:26:39.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.707+0000 7f68db400640 1 --2- 172.21.15.47:0/2898058365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68dc0b5190 0x7f68dc0b5600 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f68dc0ccd20 tx=0x7f68d401c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37a25922288b27d7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:39.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f68d9600640 1 -- 172.21.15.47:0/2898058365 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d40a5720 con 0x7f68dc0b5190 2024-02-19T15:26:39.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f68d9600640 1 -- 172.21.15.47:0/2898058365 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d40a5880 con 0x7f68dc0b5190 2024-02-19T15:26:39.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f68d9600640 1 -- 172.21.15.47:0/2898058365 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d40a5c10 con 0x7f68dc0b5190 2024-02-19T15:26:39.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f68e1a00640 1 -- 172.21.15.47:0/2898058365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68dc0b5190 msgr2=0x7f68dc0b5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f68e1a00640 1 --2- 172.21.15.47:0/2898058365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68dc0b5190 0x7f68dc0b5600 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f68dc0ccd20 tx=0x7f68d401c8a0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f1428800640 1 Processor -- start 2024-02-19T15:26:39.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f68e1a00640 1 -- 172.21.15.47:0/2898058365 shutdown_connections 2024-02-19T15:26:39.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f68e1a00640 1 --2- 172.21.15.47:0/2898058365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68dc0b5190 0x7f68dc0b5600 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f68e1a00640 1 --2- 172.21.15.47:0/2898058365 >> v2:172.21.15.47:3300/0 conn(0x7f68dc0cf760 0x7f68dc15c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f68e1a00640 1 --2- 172.21.15.47:0/2898058365 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68dc145790 0x7f68dc0728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f68e1a00640 1 -- 172.21.15.47:0/2898058365 >> 172.21.15.47:0/2898058365 conn(0x7f68dc0b1b80 msgr2=0x7f68dc0b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f1428800640 1 -- start start 2024-02-19T15:26:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f68e1a00640 1 -- 172.21.15.47:0/2898058365 shutdown_connections 2024-02-19T15:26:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f68e1a00640 1 -- 172.21.15.47:0/2898058365 wait complete. 2024-02-19T15:26:39.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f1428800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f142006c020 0x7f14200c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f1428800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14200d1cf0 0x7f14200c7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f1428800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f142015e0d0 0x7f14200c5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f1428800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14201518d0 con 0x7f142006c020 2024-02-19T15:26:39.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f1428800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1420151ae0 con 0x7f14200d1cf0 2024-02-19T15:26:39.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f1428800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1420151dc0 con 0x7f142015e0d0 2024-02-19T15:26:39.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f1425e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f142006c020 0x7f14200c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f1425e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f142006c020 0x7f14200c6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40804/0 (socket says 172.21.15.47:40804) 2024-02-19T15:26:39.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f1425e00640 1 -- 172.21.15.47:0/3237161856 learned_addr learned my addr 172.21.15.47:0/3237161856 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:39.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f1425400640 1 --2- 172.21.15.47:0/3237161856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14200d1cf0 0x7f14200c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f1426800640 1 --2- 172.21.15.47:0/3237161856 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f142015e0d0 0x7f14200c5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f1425e00640 1 -- 172.21.15.47:0/3237161856 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f142015e0d0 msgr2=0x7f14200c5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f1425e00640 1 --2- 172.21.15.47:0/3237161856 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f142015e0d0 0x7f14200c5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f1425e00640 1 -- 172.21.15.47:0/3237161856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14200d1cf0 msgr2=0x7f14200c7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f1425e00640 1 --2- 172.21.15.47:0/3237161856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14200d1cf0 0x7f14200c7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f1425e00640 1 -- 172.21.15.47:0/3237161856 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1414009420 con 0x7f142006c020 2024-02-19T15:26:39.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f1425400640 1 --2- 172.21.15.47:0/3237161856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14200d1cf0 0x7f14200c7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:39.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f68e1a00640 1 Processor -- start 2024-02-19T15:26:39.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68e1a00640 1 -- start start 2024-02-19T15:26:39.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.711+0000 7f1425e00640 1 --2- 172.21.15.47:0/3237161856 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f142006c020 0x7f14200c6b30 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f1414019060 tx=0x7f141401ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:39.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f141a000640 1 -- 172.21.15.47:0/3237161856 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14140b5810 con 0x7f142006c020 2024-02-19T15:26:39.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f141a000640 1 -- 172.21.15.47:0/3237161856 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14140b5990 con 0x7f142006c020 2024-02-19T15:26:39.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f141a000640 1 -- 172.21.15.47:0/3237161856 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14140b5d40 con 0x7f142006c020 2024-02-19T15:26:39.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f1428800640 1 -- 172.21.15.47:0/3237161856 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14201520a0 con 0x7f142006c020 2024-02-19T15:26:39.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f1428800640 1 -- 172.21.15.47:0/3237161856 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14200b19f0 con 0x7f142006c020 2024-02-19T15:26:39.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68e1a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68dc0b5190 0x7f68dc15f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68e1a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68dc0cf760 0x7f68dc0b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68e1a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68dc145790 0x7f68dc0a94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68e1a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68dc15be40 con 0x7f68dc0b5190 2024-02-19T15:26:39.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68e1a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68dc15b880 con 0x7f68dc0cf760 2024-02-19T15:26:39.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68e1a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68dc15bb60 con 0x7f68dc145790 2024-02-19T15:26:39.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68daa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68dc0b5190 0x7f68dc15f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68daa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68dc0b5190 0x7f68dc15f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40820/0 (socket says 172.21.15.47:40820) 2024-02-19T15:26:39.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68daa00640 1 -- 172.21.15.47:0/1823662478 learned_addr learned my addr 172.21.15.47:0/1823662478 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:39.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68db400640 1 --2- 172.21.15.47:0/1823662478 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68dc145790 0x7f68dc0a94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68da000640 1 --2- 172.21.15.47:0/1823662478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68dc0cf760 0x7f68dc0b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68db400640 1 -- 172.21.15.47:0/1823662478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68dc0cf760 msgr2=0x7f68dc0b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68db400640 1 --2- 172.21.15.47:0/1823662478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68dc0cf760 0x7f68dc0b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f1428800640 1 -- 172.21.15.47:0/3237161856 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1420068120 con 0x7f142006c020 2024-02-19T15:26:39.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68db400640 1 -- 172.21.15.47:0/1823662478 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68dc0b5190 msgr2=0x7f68dc15f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68db400640 1 --2- 172.21.15.47:0/1823662478 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68dc0b5190 0x7f68dc15f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68db400640 1 -- 172.21.15.47:0/1823662478 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d4009a70 con 0x7f68dc145790 2024-02-19T15:26:39.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68db400640 1 --2- 172.21.15.47:0/1823662478 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68dc145790 0x7f68dc0a94d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f68dc0b2c40 tx=0x7f68d4019640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:39.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68d2a00640 1 -- 172.21.15.47:0/1823662478 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d40b9450 con 0x7f68dc145790 2024-02-19T15:26:39.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68d2a00640 1 -- 172.21.15.47:0/1823662478 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d40b95b0 con 0x7f68dc145790 2024-02-19T15:26:39.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68d2a00640 1 -- 172.21.15.47:0/1823662478 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d40b9960 con 0x7f68dc145790 2024-02-19T15:26:39.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68e1a00640 1 -- 172.21.15.47:0/1823662478 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68dc0ada30 con 0x7f68dc145790 2024-02-19T15:26:39.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f68e1a00640 1 -- 172.21.15.47:0/1823662478 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68dc13e600 con 0x7f68dc145790 2024-02-19T15:26:39.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.715+0000 7f141a000640 1 -- 172.21.15.47:0/3237161856 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f14140bf410 con 0x7f142006c020 2024-02-19T15:26:39.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.719+0000 7f141a000640 1 --2- 172.21.15.47:0/3237161856 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1404066e40 0x7f1404069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.719+0000 7f68e1a00640 1 -- 172.21.15.47:0/1823662478 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68dc0ad870 con 0x7f68dc145790 2024-02-19T15:26:39.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.719+0000 7f141a000640 1 -- 172.21.15.47:0/3237161856 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f141411f6e0 con 0x7f142006c020 2024-02-19T15:26:39.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.719+0000 7f68d2a00640 1 -- 172.21.15.47:0/1823662478 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f68d40b9ac0 con 0x7f68dc145790 2024-02-19T15:26:39.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.719+0000 7f1425400640 1 --2- 172.21.15.47:0/3237161856 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1404066e40 0x7f1404069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.719+0000 7f68d2a00640 1 --2- 172.21.15.47:0/1823662478 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f68bc066e40 0x7f68bc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.719+0000 7f1425400640 1 --2- 172.21.15.47:0/3237161856 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1404066e40 0x7f1404069300 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f14200690e0 tx=0x7f1410009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:39.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.723+0000 7f68d2a00640 1 -- 172.21.15.47:0/1823662478 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68d411f2f0 con 0x7f68dc145790 2024-02-19T15:26:39.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.723+0000 7f68daa00640 1 --2- 172.21.15.47:0/1823662478 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f68bc066e40 0x7f68bc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.723+0000 7f68daa00640 1 --2- 172.21.15.47:0/1823662478 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f68bc066e40 0x7f68bc069300 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f68cc009b00 tx=0x7f68cc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:39.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.723+0000 7f141a000640 1 -- 172.21.15.47:0/3237161856 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f14140ec4e0 con 0x7f142006c020 2024-02-19T15:26:39.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.723+0000 7f68d2a00640 1 -- 172.21.15.47:0/1823662478 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f68d40ec290 con 0x7f68dc145790 2024-02-19T15:26:39.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.751+0000 7f346ce00640 1 Processor -- start 2024-02-19T15:26:39.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7f346ce00640 1 -- start start 2024-02-19T15:26:39.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7fe11a800640 1 Processor -- start 2024-02-19T15:26:39.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7fe11a800640 1 -- start start 2024-02-19T15:26:39.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7f346ce00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f346806df30 0x7f346806e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7f346ce00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34680ce770 0x7f34680cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7f346ce00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346806e8d0 con 0x7f3468158a30 2024-02-19T15:26:39.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7f346ce00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3468158e10 con 0x7f346806df30 2024-02-19T15:26:39.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7f346ce00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346815ec20 con 0x7f34680ce770 2024-02-19T15:26:39.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7fe11a800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1140b2e80 0x7fe11406df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7fe11a800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1140b3870 con 0x7fe1140b32a0 2024-02-19T15:26:39.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7fe11a800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1140b3b50 con 0x7fe114162450 2024-02-19T15:26:39.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7fe11a800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1140b3e30 con 0x7fe1140b2e80 2024-02-19T15:26:39.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7f3465e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f3468158a30 0x7f346815bdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39828/0 (socket says 172.21.15.47:39828) 2024-02-19T15:26:39.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7f3465e00640 1 -- 172.21.15.47:0/597041177 learned_addr learned my addr 172.21.15.47:0/597041177 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:39.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7f3465400640 1 --2- 172.21.15.47:0/597041177 >> v2:172.21.15.116:3300/0 conn(0x7f346806df30 0x7f346806e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7f3466800640 1 --2- 172.21.15.47:0/597041177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34680ce770 0x7f34680cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7fe112a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1140b2e80 0x7fe11406df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7fe112a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1140b2e80 0x7fe11406df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:54862/0 (socket says 172.21.15.47:54862) 2024-02-19T15:26:39.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7fe113400640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe114162450 0x7fe114071630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:40210/0 (socket says 172.21.15.47:40210) 2024-02-19T15:26:39.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7fe112a00640 1 -- 172.21.15.47:0/2722784673 learned_addr learned my addr 172.21.15.47:0/2722784673 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:39.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7fe113e00640 1 --1- 172.21.15.47:0/2722784673 >> v1:172.21.15.47:6789/0 conn(0x7fe1140b32a0 0x7fe114163bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39832/0 (socket says 172.21.15.47:39832) 2024-02-19T15:26:39.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7fe112a00640 1 -- 172.21.15.47:0/2722784673 >> v1:172.21.15.116:6789/0 conn(0x7fe114162450 legacy=0x7fe114071630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:39.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7fe112a00640 1 -- 172.21.15.47:0/2722784673 >> v1:172.21.15.47:6789/0 conn(0x7fe1140b32a0 legacy=0x7fe114163bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7fe112a00640 1 -- 172.21.15.47:0/2722784673 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe114070e30 con 0x7fe1140b2e80 2024-02-19T15:26:39.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7f3466800640 1 -- 172.21.15.47:0/597041177 >> v2:172.21.15.116:3300/0 conn(0x7f346806df30 msgr2=0x7f346806e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:39.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7f3466800640 1 --2- 172.21.15.47:0/597041177 >> v2:172.21.15.116:3300/0 conn(0x7f346806df30 0x7f346806e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7f3466800640 1 -- 172.21.15.47:0/597041177 >> v1:172.21.15.47:6789/0 conn(0x7f3468158a30 legacy=0x7f346815bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.755+0000 7f3466800640 1 -- 172.21.15.47:0/597041177 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346815ef00 con 0x7f34680ce770 2024-02-19T15:26:39.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.759+0000 7fe112a00640 1 --2- 172.21.15.47:0/2722784673 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1140b2e80 0x7fe11406df30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc0098a0 tx=0x7fe0fc01c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f71b5cc66e66347c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:39.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.759+0000 7f3466800640 1 --2- 172.21.15.47:0/597041177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34680ce770 0x7f34680cec00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3460009d90 tx=0x7f346001c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22373c21a7bd2c94 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:39.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.759+0000 7f3464a00640 1 -- 172.21.15.47:0/597041177 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34600a5980 con 0x7f34680ce770 2024-02-19T15:26:39.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.759+0000 7f3464a00640 1 -- 172.21.15.47:0/597041177 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34600a5ae0 con 0x7f34680ce770 2024-02-19T15:26:39.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.759+0000 7fe112000640 1 -- 172.21.15.47:0/2722784673 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc0a95f0 con 0x7fe1140b2e80 2024-02-19T15:26:39.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.759+0000 7fe112000640 1 -- 172.21.15.47:0/2722784673 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0fc0a9750 con 0x7fe1140b2e80 2024-02-19T15:26:39.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.759+0000 7fe112000640 1 -- 172.21.15.47:0/2722784673 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc0a9ae0 con 0x7fe1140b2e80 2024-02-19T15:26:39.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.759+0000 7f3464a00640 1 -- 172.21.15.47:0/597041177 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34600a5e70 con 0x7f34680ce770 2024-02-19T15:26:39.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.763+0000 7f346ce00640 1 -- 172.21.15.47:0/597041177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34680ce770 msgr2=0x7f34680cec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.763+0000 7f346ce00640 1 --2- 172.21.15.47:0/597041177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34680ce770 0x7f34680cec00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3460009d90 tx=0x7f346001c910 comp rx=0 tx=0).stop 2024-02-19T15:26:39.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.763+0000 7fe11a800640 1 -- 172.21.15.47:0/2722784673 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1140b2e80 msgr2=0x7fe11406df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.763+0000 7fe11a800640 1 --2- 172.21.15.47:0/2722784673 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1140b2e80 0x7fe11406df30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc0098a0 tx=0x7fe0fc01c510 comp rx=0 tx=0).stop 2024-02-19T15:26:39.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.763+0000 7fe11a800640 1 -- 172.21.15.47:0/2722784673 shutdown_connections 2024-02-19T15:26:39.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.763+0000 7fe11a800640 1 --2- 172.21.15.47:0/2722784673 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1140b2e80 0x7fe11406df30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.763+0000 7fe11a800640 1 -- 172.21.15.47:0/2722784673 >> 172.21.15.47:0/2722784673 conn(0x7fe1140b1b80 msgr2=0x7fe1140b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:39.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.763+0000 7f346ce00640 1 -- 172.21.15.47:0/597041177 shutdown_connections 2024-02-19T15:26:39.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.763+0000 7f346ce00640 1 --2- 172.21.15.47:0/597041177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f34680ce770 0x7f34680cec00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.763+0000 7f346ce00640 1 --2- 172.21.15.47:0/597041177 >> v2:172.21.15.116:3300/0 conn(0x7f346806df30 0x7f346806e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.763+0000 7f346ce00640 1 -- 172.21.15.47:0/597041177 >> 172.21.15.47:0/597041177 conn(0x7f34680b1b80 msgr2=0x7f34680b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:39.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.763+0000 7f346ce00640 1 -- 172.21.15.47:0/597041177 shutdown_connections 2024-02-19T15:26:39.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.763+0000 7fe11a800640 1 -- 172.21.15.47:0/2722784673 shutdown_connections 2024-02-19T15:26:39.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.763+0000 7f346ce00640 1 -- 172.21.15.47:0/597041177 wait complete. 2024-02-19T15:26:39.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.763+0000 7fe11a800640 1 -- 172.21.15.47:0/2722784673 wait complete. 2024-02-19T15:26:39.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.763+0000 7fe11a800640 1 Processor -- start 2024-02-19T15:26:39.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.763+0000 7f346ce00640 1 Processor -- start 2024-02-19T15:26:39.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.763+0000 7fe11a800640 1 -- start start 2024-02-19T15:26:39.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.763+0000 7f346ce00640 1 -- start start 2024-02-19T15:26:39.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe11a800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1140b2e80 0x7fe1140cd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f346ce00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f346806df30 0x7f346814c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe11a800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1140b32a0 0x7fe11414b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe11a800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe114162450 0x7fe114152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe11a800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11414bb50 con 0x7fe1140b32a0 2024-02-19T15:26:39.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe11a800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe114156c80 con 0x7fe114162450 2024-02-19T15:26:39.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe11a800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1141566c0 con 0x7fe1140b2e80 2024-02-19T15:26:39.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f346ce00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34680ce770 0x7f346813e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f346ce00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3468158a30 0x7f3468157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f346ce00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346814d620 con 0x7f34680ce770 2024-02-19T15:26:39.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f346ce00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346814d900 con 0x7f346806df30 2024-02-19T15:26:39.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f346ce00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346814dbe0 con 0x7f3468158a30 2024-02-19T15:26:39.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f3465400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34680ce770 0x7f346813e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe113e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe114162450 0x7fe114152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe113400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1140b2e80 0x7fe1140cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe113e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe114162450 0x7fe114152920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:56292/0 (socket says 172.21.15.47:56292) 2024-02-19T15:26:39.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe113e00640 1 -- 172.21.15.47:0/3813465731 learned_addr learned my addr 172.21.15.47:0/3813465731 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:39.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f3466800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3468158a30 0x7f3468157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f3465400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34680ce770 0x7f346813e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40828/0 (socket says 172.21.15.47:40828) 2024-02-19T15:26:39.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f3465400640 1 -- 172.21.15.47:0/3502890149 learned_addr learned my addr 172.21.15.47:0/3502890149 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:39.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f3465e00640 1 --2- 172.21.15.47:0/3502890149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f346806df30 0x7f346814c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe112a00640 1 --2- 172.21.15.47:0/3813465731 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1140b32a0 0x7fe11414b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f3466800640 1 -- 172.21.15.47:0/3502890149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f346806df30 msgr2=0x7f346814c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f3466800640 1 --2- 172.21.15.47:0/3502890149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f346806df30 0x7f346814c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f3466800640 1 -- 172.21.15.47:0/3502890149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34680ce770 msgr2=0x7f346813e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f3466800640 1 --2- 172.21.15.47:0/3502890149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34680ce770 0x7f346813e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f3466800640 1 -- 172.21.15.47:0/3502890149 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3460009a70 con 0x7f3468158a30 2024-02-19T15:26:39.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe113e00640 1 -- 172.21.15.47:0/3813465731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1140b2e80 msgr2=0x7fe1140cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe113e00640 1 --2- 172.21.15.47:0/3813465731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1140b2e80 0x7fe1140cd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe113e00640 1 -- 172.21.15.47:0/3813465731 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1140b32a0 msgr2=0x7fe11414b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe113e00640 1 --2- 172.21.15.47:0/3813465731 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1140b32a0 0x7fe11414b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe113e00640 1 -- 172.21.15.47:0/3813465731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0fc0094d0 con 0x7fe114162450 2024-02-19T15:26:39.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f3466800640 1 --2- 172.21.15.47:0/3502890149 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3468158a30 0x7f3468157080 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f34600ac000 tx=0x7f34600196b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:39.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe113e00640 1 --2- 172.21.15.47:0/3813465731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe114162450 0x7fe114152920 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe10400c410 tx=0x7fe10400c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:39.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f345e000640 1 -- 172.21.15.47:0/3502890149 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34600b9860 con 0x7f3468158a30 2024-02-19T15:26:39.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f345e000640 1 -- 172.21.15.47:0/3502890149 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34600b99c0 con 0x7f3468158a30 2024-02-19T15:26:39.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f345e000640 1 -- 172.21.15.47:0/3502890149 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34600b9d50 con 0x7f3468158a30 2024-02-19T15:26:39.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f346ce00640 1 -- 172.21.15.47:0/3502890149 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f346814dec0 con 0x7f3468158a30 2024-02-19T15:26:39.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe10b400640 1 -- 172.21.15.47:0/3813465731 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1040074e0 con 0x7fe114162450 2024-02-19T15:26:39.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe10b400640 1 -- 172.21.15.47:0/3813465731 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe104007640 con 0x7fe114162450 2024-02-19T15:26:39.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe10b400640 1 -- 172.21.15.47:0/3813465731 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe104004a10 con 0x7fe114162450 2024-02-19T15:26:39.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7f346ce00640 1 -- 172.21.15.47:0/3502890149 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f346813ea30 con 0x7f3468158a30 2024-02-19T15:26:39.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe11a800640 1 -- 172.21.15.47:0/3813465731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1141569a0 con 0x7fe114162450 2024-02-19T15:26:39.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.767+0000 7fe11a800640 1 -- 172.21.15.47:0/3813465731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe114152400 con 0x7fe114162450 2024-02-19T15:26:39.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.771+0000 7f346ce00640 1 -- 172.21.15.47:0/3502890149 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3468068120 con 0x7f3468158a30 2024-02-19T15:26:39.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.771+0000 7fe11a800640 1 -- 172.21.15.47:0/3813465731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1141525f0 con 0x7fe114162450 2024-02-19T15:26:39.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.771+0000 7f345e000640 1 -- 172.21.15.47:0/3502890149 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f34600ad440 con 0x7f3468158a30 2024-02-19T15:26:39.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.771+0000 7f345e000640 1 --2- 172.21.15.47:0/3502890149 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3448066e90 0x7f3448069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.771+0000 7f345e000640 1 -- 172.21.15.47:0/3502890149 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f346011f1c0 con 0x7f3468158a30 2024-02-19T15:26:39.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.771+0000 7f3465e00640 1 --2- 172.21.15.47:0/3502890149 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3448066e90 0x7f3448069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.771+0000 7fe10b400640 1 -- 172.21.15.47:0/3813465731 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe104004b70 con 0x7fe114162450 2024-02-19T15:26:39.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.771+0000 7f3465e00640 1 --2- 172.21.15.47:0/3502890149 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3448066e90 0x7f3448069350 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f3458000f70 tx=0x7f3458009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:39.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.771+0000 7fe10b400640 1 --2- 172.21.15.47:0/3813465731 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe0f0066e90 0x7fe0f0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:39.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.771+0000 7fe10b400640 1 -- 172.21.15.47:0/3813465731 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe104084400 con 0x7fe114162450 2024-02-19T15:26:39.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.771+0000 7fe113400640 1 --2- 172.21.15.47:0/3813465731 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe0f0066e90 0x7fe0f0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:39.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.775+0000 7fe113400640 1 --2- 172.21.15.47:0/3813465731 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe0f0066e90 0x7fe0f0069350 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00c440 tx=0x7fe10c009690 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:39.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.775+0000 7f345e000640 1 -- 172.21.15.47:0/3502890149 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f34600ec050 con 0x7f3468158a30 2024-02-19T15:26:39.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.779+0000 7fe10b400640 1 -- 172.21.15.47:0/3813465731 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe1040511d0 con 0x7fe114162450 2024-02-19T15:26:39.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.955+0000 7f1428800640 1 -- 172.21.15.47:0/3237161856 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f1420156f80 con 0x7f142006c020 2024-02-19T15:26:39.960 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-02-19T15:26:39.955+0000 7fc100600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T15:26:39.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.955+0000 7f141a000640 1 -- 172.21.15.47:0/3237161856 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6850 v59) v1 ==== 95+0+0 (secure 0 0 0) 0x7f14140ab630 con 0x7f142006c020 2024-02-19T15:26:39.962 INFO:teuthology.orchestra.run.smithi047.stderr:failed mds gid 6850 2024-02-19T15:26:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.959+0000 7f1428800640 1 -- 172.21.15.47:0/3237161856 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1404066e40 msgr2=0x7f1404069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.959+0000 7f1428800640 1 --2- 172.21.15.47:0/3237161856 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1404066e40 0x7f1404069300 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f14200690e0 tx=0x7f1410009240 comp rx=0 tx=0).stop 2024-02-19T15:26:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.959+0000 7f1428800640 1 -- 172.21.15.47:0/3237161856 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f142006c020 msgr2=0x7f14200c6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.959+0000 7f1428800640 1 --2- 172.21.15.47:0/3237161856 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f142006c020 0x7f14200c6b30 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f1414019060 tx=0x7f141401ac90 comp rx=0 tx=0).stop 2024-02-19T15:26:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.959+0000 7f1428800640 1 -- 172.21.15.47:0/3237161856 shutdown_connections 2024-02-19T15:26:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.959+0000 7f1428800640 1 --2- 172.21.15.47:0/3237161856 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1404066e40 0x7f1404069300 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.959+0000 7f1428800640 1 --2- 172.21.15.47:0/3237161856 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f142015e0d0 0x7f14200c5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.959+0000 7f1428800640 1 --2- 172.21.15.47:0/3237161856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14200d1cf0 0x7f14200c7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.959+0000 7f1428800640 1 --2- 172.21.15.47:0/3237161856 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f142006c020 0x7f14200c6b30 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.959+0000 7f1428800640 1 -- 172.21.15.47:0/3237161856 >> 172.21.15.47:0/3237161856 conn(0x7f142009aa70 msgr2=0x7f14200cd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.959+0000 7f1428800640 1 -- 172.21.15.47:0/3237161856 shutdown_connections 2024-02-19T15:26:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.959+0000 7f1428800640 1 -- 172.21.15.47:0/3237161856 wait complete. 2024-02-19T15:26:39.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:39.967+0000 7f68e1a00640 1 -- 172.21.15.47:0/1823662478 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f68dc15bca0 con 0x7f68dc145790 2024-02-19T15:26:39.997 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T15:26:39.997 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T15:26:40.000 INFO:tasks.ceph.mds.d:Started 2024-02-19T15:26:40.000 DEBUG:teuthology.parallel:result is None 2024-02-19T15:26:40.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.007+0000 7fe11a800640 1 -- 172.21.15.47:0/3813465731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7fe114068120 con 0x7fe114162450 2024-02-19T15:26:40.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.019+0000 7f346ce00640 1 -- 172.21.15.47:0/3502890149 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f34680cdd50 con 0x7f3468158a30 2024-02-19T15:26:40.083 INFO:tasks.ceph.mds.d.smithi116.stdout:starting mds.d at 2024-02-19T15:26:40.819 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-02-19T15:26:40.815+0000 7fc0fd400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T15:26:40.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.819+0000 7fe10b400640 1 -- 172.21.15.47:0/3813465731 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 6844 v61) v1 ==== 95+0+0 (secure 0 0 0) 0x7fe104054890 con 0x7fe114162450 2024-02-19T15:26:40.825 INFO:teuthology.orchestra.run.smithi047.stderr:failed mds gid 6844 2024-02-19T15:26:40.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.819+0000 7f68d2a00640 1 -- 172.21.15.47:0/1823662478 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6188 v61) v1 ==== 95+0+0 (secure 0 0 0) 0x7f68dc15bca0 con 0x7f68dc145790 2024-02-19T15:26:40.826 INFO:teuthology.orchestra.run.smithi047.stderr:failed mds gid 6188 2024-02-19T15:26:40.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7fe11a800640 1 -- 172.21.15.47:0/3813465731 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe0f0066e90 msgr2=0x7fe0f0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:40.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7fe11a800640 1 --2- 172.21.15.47:0/3813465731 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe0f0066e90 0x7fe0f0069350 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00c440 tx=0x7fe10c009690 comp rx=0 tx=0).stop 2024-02-19T15:26:40.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7fe11a800640 1 -- 172.21.15.47:0/3813465731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe114162450 msgr2=0x7fe114152920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:40.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7fe11a800640 1 --2- 172.21.15.47:0/3813465731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe114162450 0x7fe114152920 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe10400c410 tx=0x7fe10400c8d0 comp rx=0 tx=0).stop 2024-02-19T15:26:40.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7fe11a800640 1 -- 172.21.15.47:0/3813465731 shutdown_connections 2024-02-19T15:26:40.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7fe11a800640 1 --2- 172.21.15.47:0/3813465731 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe0f0066e90 0x7fe0f0069350 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:40.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7fe11a800640 1 --2- 172.21.15.47:0/3813465731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe114162450 0x7fe114152920 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:40.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7fe11a800640 1 --2- 172.21.15.47:0/3813465731 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1140b32a0 0x7fe11414b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:40.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7fe11a800640 1 --2- 172.21.15.47:0/3813465731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1140b2e80 0x7fe1140cd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:40.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7fe11a800640 1 -- 172.21.15.47:0/3813465731 >> 172.21.15.47:0/3813465731 conn(0x7fe1140b1b80 msgr2=0x7fe114158200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:40.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7fe11a800640 1 -- 172.21.15.47:0/3813465731 shutdown_connections 2024-02-19T15:26:40.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7fe11a800640 1 -- 172.21.15.47:0/3813465731 wait complete. 2024-02-19T15:26:40.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7f68e1a00640 1 -- 172.21.15.47:0/1823662478 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f68bc066e40 msgr2=0x7f68bc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:40.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7f68e1a00640 1 --2- 172.21.15.47:0/1823662478 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f68bc066e40 0x7f68bc069300 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f68cc009b00 tx=0x7f68cc009190 comp rx=0 tx=0).stop 2024-02-19T15:26:40.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7f68e1a00640 1 -- 172.21.15.47:0/1823662478 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68dc145790 msgr2=0x7f68dc0a94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:40.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7f68e1a00640 1 --2- 172.21.15.47:0/1823662478 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68dc145790 0x7f68dc0a94d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f68dc0b2c40 tx=0x7f68d4019640 comp rx=0 tx=0).stop 2024-02-19T15:26:40.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7f68e1a00640 1 -- 172.21.15.47:0/1823662478 shutdown_connections 2024-02-19T15:26:40.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7f68e1a00640 1 --2- 172.21.15.47:0/1823662478 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f68bc066e40 0x7f68bc069300 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:40.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7f68e1a00640 1 --2- 172.21.15.47:0/1823662478 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68dc145790 0x7f68dc0a94d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:40.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7f68e1a00640 1 --2- 172.21.15.47:0/1823662478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68dc0cf760 0x7f68dc0b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:40.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7f68e1a00640 1 --2- 172.21.15.47:0/1823662478 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68dc0b5190 0x7f68dc15f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:40.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.823+0000 7f68e1a00640 1 -- 172.21.15.47:0/1823662478 >> 172.21.15.47:0/1823662478 conn(0x7f68dc0b1b80 msgr2=0x7f68dc072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:40.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.827+0000 7f68e1a00640 1 -- 172.21.15.47:0/1823662478 shutdown_connections 2024-02-19T15:26:40.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.827+0000 7f68e1a00640 1 -- 172.21.15.47:0/1823662478 wait complete. 2024-02-19T15:26:40.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.831+0000 7f345e000640 1 -- 172.21.15.47:0/3502890149 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 6191 v62) v1 ==== 95+0+0 (secure 0 0 0) 0x7f34600ef710 con 0x7f3468158a30 2024-02-19T15:26:40.835 INFO:teuthology.orchestra.run.smithi047.stderr:failed mds gid 6191 2024-02-19T15:26:40.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.831+0000 7f346ce00640 1 -- 172.21.15.47:0/3502890149 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3448066e90 msgr2=0x7f3448069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:40.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.831+0000 7f346ce00640 1 --2- 172.21.15.47:0/3502890149 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3448066e90 0x7f3448069350 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f3458000f70 tx=0x7f3458009040 comp rx=0 tx=0).stop 2024-02-19T15:26:40.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.831+0000 7f346ce00640 1 -- 172.21.15.47:0/3502890149 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3468158a30 msgr2=0x7f3468157080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:40.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.831+0000 7f346ce00640 1 --2- 172.21.15.47:0/3502890149 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3468158a30 0x7f3468157080 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f34600ac000 tx=0x7f34600196b0 comp rx=0 tx=0).stop 2024-02-19T15:26:40.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.831+0000 7f346ce00640 1 -- 172.21.15.47:0/3502890149 shutdown_connections 2024-02-19T15:26:40.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.831+0000 7f346ce00640 1 --2- 172.21.15.47:0/3502890149 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3448066e90 0x7f3448069350 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:40.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.831+0000 7f346ce00640 1 --2- 172.21.15.47:0/3502890149 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3468158a30 0x7f3468157080 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:40.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.831+0000 7f346ce00640 1 --2- 172.21.15.47:0/3502890149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34680ce770 0x7f346813e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:40.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.831+0000 7f346ce00640 1 --2- 172.21.15.47:0/3502890149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f346806df30 0x7f346814c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:40.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.831+0000 7f346ce00640 1 -- 172.21.15.47:0/3502890149 >> 172.21.15.47:0/3502890149 conn(0x7f34680b1b80 msgr2=0x7f34680b0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:40.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.831+0000 7f346ce00640 1 -- 172.21.15.47:0/3502890149 shutdown_connections 2024-02-19T15:26:40.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:40.831+0000 7f346ce00640 1 -- 172.21.15.47:0/3502890149 wait complete. 2024-02-19T15:26:40.862 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T15:26:40.862 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T15:26:40.863 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T15:26:40.863 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T15:26:40.867 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T15:26:40.867 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T15:26:40.867 INFO:tasks.ceph.mds.b:Started 2024-02-19T15:26:40.868 DEBUG:teuthology.parallel:result is None 2024-02-19T15:26:40.912 INFO:tasks.ceph.mds.a:Started 2024-02-19T15:26:40.912 DEBUG:teuthology.parallel:result is None 2024-02-19T15:26:40.914 INFO:tasks.ceph.mds.c:Started 2024-02-19T15:26:40.914 DEBUG:teuthology.parallel:result is None 2024-02-19T15:26:40.915 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:26:40.949 INFO:tasks.ceph.mds.b.smithi116.stdout:starting mds.b at 2024-02-19T15:26:41.001 INFO:tasks.ceph.mds.c.smithi047.stdout:starting mds.c at 2024-02-19T15:26:41.001 INFO:tasks.ceph.mds.a.smithi047.stdout:starting mds.a at 2024-02-19T15:26:41.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.055+0000 7fe2b9600640 1 Processor -- start 2024-02-19T15:26:41.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.055+0000 7fe2b9600640 1 -- start start 2024-02-19T15:26:41.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.055+0000 7fe2b9600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2b406df30 0x7fe2b406e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:41.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.055+0000 7fe2b9600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe2b40ce770 0x7fe2b40cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:41.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.055+0000 7fe2b9600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b406e8d0 con 0x7fe2b4158a30 2024-02-19T15:26:41.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.055+0000 7fe2b9600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b4158e10 con 0x7fe2b40ce770 2024-02-19T15:26:41.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.055+0000 7fe2b9600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b415ec20 con 0x7fe2b406df30 2024-02-19T15:26:41.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.055+0000 7fe2b2a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fe2b4158a30 0x7fe2b415bdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39850/0 (socket says 172.21.15.47:39850) 2024-02-19T15:26:41.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.055+0000 7fe2b2a00640 1 -- 172.21.15.47:0/70112040 learned_addr learned my addr 172.21.15.47:0/70112040 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:41.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.055+0000 7fe2b3400640 1 --2- 172.21.15.47:0/70112040 >> v2:172.21.15.47:3300/0 conn(0x7fe2b40ce770 0x7fe2b40cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:41.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.055+0000 7fe2aa000640 1 --2- 172.21.15.47:0/70112040 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2b406df30 0x7fe2b406e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:41.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.055+0000 7fe2b2000640 1 -- 172.21.15.47:0/70112040 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010094760 0 0) 0x7fe2b406e8d0 con 0x7fe2b4158a30 2024-02-19T15:26:41.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.055+0000 7fe2b3400640 1 -- 172.21.15.47:0/70112040 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2b406df30 msgr2=0x7fe2b406e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:41.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.055+0000 7fe2b3400640 1 --2- 172.21.15.47:0/70112040 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2b406df30 0x7fe2b406e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:41.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.055+0000 7fe2b3400640 1 -- 172.21.15.47:0/70112040 >> v1:172.21.15.47:6789/0 conn(0x7fe2b4158a30 legacy=0x7fe2b415bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:41.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.055+0000 7fe2b3400640 1 -- 172.21.15.47:0/70112040 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b415ef00 con 0x7fe2b40ce770 2024-02-19T15:26:41.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.059+0000 7fe2b3400640 1 --2- 172.21.15.47:0/70112040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2b40ce770 0x7fe2b40cec00 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac009d90 tx=0x7fe2ac01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ab144896d6ab17 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:41.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.059+0000 7fe2b2000640 1 -- 172.21.15.47:0/70112040 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac0a5980 con 0x7fe2b40ce770 2024-02-19T15:26:41.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.059+0000 7fe2b2000640 1 -- 172.21.15.47:0/70112040 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ac0a5ae0 con 0x7fe2b40ce770 2024-02-19T15:26:41.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.059+0000 7fe2b2000640 1 -- 172.21.15.47:0/70112040 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac0a5e70 con 0x7fe2b40ce770 2024-02-19T15:26:41.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.059+0000 7fe2b9600640 1 -- 172.21.15.47:0/70112040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2b40ce770 msgr2=0x7fe2b40cec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:41.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.059+0000 7fe2b9600640 1 --2- 172.21.15.47:0/70112040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2b40ce770 0x7fe2b40cec00 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac009d90 tx=0x7fe2ac01c910 comp rx=0 tx=0).stop 2024-02-19T15:26:41.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.059+0000 7fe2b9600640 1 -- 172.21.15.47:0/70112040 shutdown_connections 2024-02-19T15:26:41.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.059+0000 7fe2b9600640 1 --2- 172.21.15.47:0/70112040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2b40ce770 0x7fe2b40cec00 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:41.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.059+0000 7fe2b9600640 1 --2- 172.21.15.47:0/70112040 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2b406df30 0x7fe2b406e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:41.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.059+0000 7fe2b9600640 1 -- 172.21.15.47:0/70112040 >> 172.21.15.47:0/70112040 conn(0x7fe2b40b1b80 msgr2=0x7fe2b40b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:41.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.059+0000 7fe2b9600640 1 -- 172.21.15.47:0/70112040 shutdown_connections 2024-02-19T15:26:41.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.059+0000 7fe2b9600640 1 -- 172.21.15.47:0/70112040 wait complete. 2024-02-19T15:26:41.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2b9600640 1 Processor -- start 2024-02-19T15:26:41.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2b9600640 1 -- start start 2024-02-19T15:26:41.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2b9600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2b406df30 0x7fe2b413d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:41.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2b9600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2b40ce770 0x7fe2b414a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:41.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2b9600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2b4158a30 0x7fe2b4071a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:41.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2b9600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b41497d0 con 0x7fe2b406df30 2024-02-19T15:26:41.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2b9600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b4149ab0 con 0x7fe2b40ce770 2024-02-19T15:26:41.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2b9600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b4149d90 con 0x7fe2b4158a30 2024-02-19T15:26:41.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2b2a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2b406df30 0x7fe2b413d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:41.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2b2a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2b406df30 0x7fe2b413d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40886/0 (socket says 172.21.15.47:40886) 2024-02-19T15:26:41.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2b2a00640 1 -- 172.21.15.47:0/38435323 learned_addr learned my addr 172.21.15.47:0/38435323 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:41.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2b3400640 1 --2- 172.21.15.47:0/38435323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2b4158a30 0x7fe2b4071a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:41.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2aa000640 1 --2- 172.21.15.47:0/38435323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2b40ce770 0x7fe2b414a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:41.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2b2a00640 1 -- 172.21.15.47:0/38435323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2b4158a30 msgr2=0x7fe2b4071a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:41.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2b2a00640 1 --2- 172.21.15.47:0/38435323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2b4158a30 0x7fe2b4071a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:41.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2b2a00640 1 -- 172.21.15.47:0/38435323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2b40ce770 msgr2=0x7fe2b414a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:41.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2b2a00640 1 --2- 172.21.15.47:0/38435323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2b40ce770 0x7fe2b414a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:41.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2b2a00640 1 -- 172.21.15.47:0/38435323 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2ac009a70 con 0x7fe2b406df30 2024-02-19T15:26:41.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2b2a00640 1 --2- 172.21.15.47:0/38435323 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2b406df30 0x7fe2b413d9a0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fe2a400c480 tx=0x7fe2a400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:41.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2ab400640 1 -- 172.21.15.47:0/38435323 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a40074e0 con 0x7fe2b406df30 2024-02-19T15:26:41.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2ab400640 1 -- 172.21.15.47:0/38435323 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a4007660 con 0x7fe2b406df30 2024-02-19T15:26:41.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2b9600640 1 -- 172.21.15.47:0/38435323 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b414a070 con 0x7fe2b406df30 2024-02-19T15:26:41.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2ab400640 1 -- 172.21.15.47:0/38435323 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a4004a30 con 0x7fe2b406df30 2024-02-19T15:26:41.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.063+0000 7fe2b9600640 1 -- 172.21.15.47:0/38435323 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b413dee0 con 0x7fe2b406df30 2024-02-19T15:26:41.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.067+0000 7fe2b9600640 1 -- 172.21.15.47:0/38435323 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b40b3ad0 con 0x7fe2b406df30 2024-02-19T15:26:41.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.067+0000 7fe2ab400640 1 -- 172.21.15.47:0/38435323 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe2a4004b90 con 0x7fe2b406df30 2024-02-19T15:26:41.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.067+0000 7fe2ab400640 1 --2- 172.21.15.47:0/38435323 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe290066e70 0x7fe290069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:41.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.067+0000 7fe2ab400640 1 -- 172.21.15.47:0/38435323 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fe2a4083f70 con 0x7fe2b406df30 2024-02-19T15:26:41.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.067+0000 7fe2aa000640 1 --2- 172.21.15.47:0/38435323 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe290066e70 0x7fe290069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:41.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.067+0000 7fe2aa000640 1 --2- 172.21.15.47:0/38435323 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe290066e70 0x7fe290069330 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe2b41580b0 tx=0x7fe29c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:41.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.071+0000 7fe2ab400640 1 -- 172.21.15.47:0/38435323 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe2a4050d50 con 0x7fe2b406df30 2024-02-19T15:26:41.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.267+0000 7fe2b9600640 1 -- 172.21.15.47:0/38435323 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2b4068120 con 0x7fe2b406df30 2024-02-19T15:26:41.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.267+0000 7fe2ab400640 1 -- 172.21.15.47:0/38435323 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2572 (secure 0 0 0) 0x7fe2a4018020 con 0x7fe2b406df30 2024-02-19T15:26:41.274 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:41.274 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":62,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:26:26.721109+0000","modified":"2024-02-19T15:26:40.824224+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7054":{"gid":7054,"name":"d","rank":0,"incarnation":62,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6835/3852677195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3852677195},{"type":"v1","addr":"172.21.15.116:6835","nonce":3852677195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T15:26:41.274 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 62 2024-02-19T15:26:41.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.271+0000 7fe2b9600640 1 -- 172.21.15.47:0/38435323 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe290066e70 msgr2=0x7fe290069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:41.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.271+0000 7fe2b9600640 1 --2- 172.21.15.47:0/38435323 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe290066e70 0x7fe290069330 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe2b41580b0 tx=0x7fe29c009190 comp rx=0 tx=0).stop 2024-02-19T15:26:41.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.271+0000 7fe2b9600640 1 -- 172.21.15.47:0/38435323 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2b406df30 msgr2=0x7fe2b413d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:41.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.271+0000 7fe2b9600640 1 --2- 172.21.15.47:0/38435323 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2b406df30 0x7fe2b413d9a0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fe2a400c480 tx=0x7fe2a400c940 comp rx=0 tx=0).stop 2024-02-19T15:26:41.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.271+0000 7fe2b9600640 1 -- 172.21.15.47:0/38435323 shutdown_connections 2024-02-19T15:26:41.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.271+0000 7fe2b9600640 1 --2- 172.21.15.47:0/38435323 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe290066e70 0x7fe290069330 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:41.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.271+0000 7fe2b9600640 1 --2- 172.21.15.47:0/38435323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2b4158a30 0x7fe2b4071a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:41.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.271+0000 7fe2b9600640 1 --2- 172.21.15.47:0/38435323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2b40ce770 0x7fe2b414a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:41.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.271+0000 7fe2b9600640 1 --2- 172.21.15.47:0/38435323 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2b406df30 0x7fe2b413d9a0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:41.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.271+0000 7fe2b9600640 1 -- 172.21.15.47:0/38435323 >> 172.21.15.47:0/38435323 conn(0x7fe2b40b1b80 msgr2=0x7fe2b40b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:41.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.271+0000 7fe2b9600640 1 -- 172.21.15.47:0/38435323 shutdown_connections 2024-02-19T15:26:41.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:41.271+0000 7fe2b9600640 1 -- 172.21.15.47:0/38435323 wait complete. 2024-02-19T15:26:41.698 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:26:26.721109+0000', 'modified': '2024-02-19T15:26:40.824224+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 96, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7054}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7054': {'gid': 7054, 'name': 'd', 'rank': 0, 'incarnation': 62, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.116:6835/3852677195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6834', 'nonce': 3852677195}, {'type': 'v1', 'addr': '172.21.15.116:6835', 'nonce': 3852677195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:26:41.698 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7054:up:replay 2024-02-19T15:26:42.699 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:26:42.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.847+0000 7fd093e00640 1 Processor -- start 2024-02-19T15:26:42.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.847+0000 7fd093e00640 1 -- start start 2024-02-19T15:26:42.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.847+0000 7fd093e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd08c158a30 0x7fd08c15bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:42.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.847+0000 7fd093e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd08c06df30 0x7fd08c06e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:42.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.847+0000 7fd093e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd08c071dc0 0x7fd08c072250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:42.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.847+0000 7fd093e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd08c158e10 con 0x7fd08c06df30 2024-02-19T15:26:42.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.847+0000 7fd093e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd08c06e8d0 con 0x7fd08c158a30 2024-02-19T15:26:42.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.847+0000 7fd093e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c15c300 con 0x7fd08c071dc0 2024-02-19T15:26:42.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.847+0000 7fd090a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd08c06df30 0x7fd08c06e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:42.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.847+0000 7fd090a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd08c06df30 0x7fd08c06e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40888/0 (socket says 172.21.15.47:40888) 2024-02-19T15:26:42.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.847+0000 7fd090a00640 1 -- 172.21.15.47:0/770956791 learned_addr learned my addr 172.21.15.47:0/770956791 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:42.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.847+0000 7fd091e00640 1 --2- 172.21.15.47:0/770956791 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd08c071dc0 0x7fd08c072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:42.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.847+0000 7fd091400640 1 --2- 172.21.15.47:0/770956791 >> v2:172.21.15.116:3300/0 conn(0x7fd08c158a30 0x7fd08c15bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:42.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.847+0000 7fd090a00640 1 -- 172.21.15.47:0/770956791 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd08c071dc0 msgr2=0x7fd08c072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:42.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.847+0000 7fd090a00640 1 --2- 172.21.15.47:0/770956791 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd08c071dc0 0x7fd08c072250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:42.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.847+0000 7fd090a00640 1 -- 172.21.15.47:0/770956791 >> v2:172.21.15.116:3300/0 conn(0x7fd08c158a30 msgr2=0x7fd08c15bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:42.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.847+0000 7fd090a00640 1 --2- 172.21.15.47:0/770956791 >> v2:172.21.15.116:3300/0 conn(0x7fd08c158a30 0x7fd08c15bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:42.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.847+0000 7fd090a00640 1 -- 172.21.15.47:0/770956791 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd08c15ec20 con 0x7fd08c06df30 2024-02-19T15:26:42.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.847+0000 7fd091e00640 1 --2- 172.21.15.47:0/770956791 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd08c071dc0 0x7fd08c072250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:42.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.851+0000 7fd090a00640 1 --2- 172.21.15.47:0/770956791 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd08c06df30 0x7fd08c06e390 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fd08c0b6110 tx=0x7fd07801e700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f03273c33fec575 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:42.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.851+0000 7fd08be00640 1 -- 172.21.15.47:0/770956791 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0780b2600 con 0x7fd08c06df30 2024-02-19T15:26:42.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.851+0000 7fd08be00640 1 -- 172.21.15.47:0/770956791 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0780b2760 con 0x7fd08c06df30 2024-02-19T15:26:42.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.851+0000 7fd08be00640 1 -- 172.21.15.47:0/770956791 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0780b2af0 con 0x7fd08c06df30 2024-02-19T15:26:42.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.851+0000 7fd093e00640 1 -- 172.21.15.47:0/770956791 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd08c06df30 msgr2=0x7fd08c06e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:42.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.851+0000 7fd093e00640 1 --2- 172.21.15.47:0/770956791 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd08c06df30 0x7fd08c06e390 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fd08c0b6110 tx=0x7fd07801e700 comp rx=0 tx=0).stop 2024-02-19T15:26:42.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.851+0000 7fd093e00640 1 -- 172.21.15.47:0/770956791 shutdown_connections 2024-02-19T15:26:42.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.851+0000 7fd093e00640 1 --2- 172.21.15.47:0/770956791 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd08c071dc0 0x7fd08c072250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:42.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.851+0000 7fd093e00640 1 --2- 172.21.15.47:0/770956791 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd08c06df30 0x7fd08c06e390 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:42.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.851+0000 7fd093e00640 1 --2- 172.21.15.47:0/770956791 >> v2:172.21.15.116:3300/0 conn(0x7fd08c158a30 0x7fd08c15bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:42.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.851+0000 7fd093e00640 1 -- 172.21.15.47:0/770956791 >> 172.21.15.47:0/770956791 conn(0x7fd08c0b1b80 msgr2=0x7fd08c0b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:42.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.851+0000 7fd093e00640 1 -- 172.21.15.47:0/770956791 shutdown_connections 2024-02-19T15:26:42.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.851+0000 7fd093e00640 1 -- 172.21.15.47:0/770956791 wait complete. 2024-02-19T15:26:42.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd093e00640 1 Processor -- start 2024-02-19T15:26:42.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd093e00640 1 -- start start 2024-02-19T15:26:42.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd093e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c06df30 0x7fd08c156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:42.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd093e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd08c071dc0 0x7fd08c1463e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:42.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd093e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd08c158a30 0x7fd08c15cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:42.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd093e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c14c830 con 0x7fd08c071dc0 2024-02-19T15:26:42.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd093e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c1561d0 con 0x7fd08c06df30 2024-02-19T15:26:42.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd093e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c1564b0 con 0x7fd08c158a30 2024-02-19T15:26:42.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd090a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd08c071dc0 0x7fd08c1463e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:42.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd090a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd08c071dc0 0x7fd08c1463e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40890/0 (socket says 172.21.15.47:40890) 2024-02-19T15:26:42.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd090a00640 1 -- 172.21.15.47:0/3734402731 learned_addr learned my addr 172.21.15.47:0/3734402731 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:42.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd091400640 1 --2- 172.21.15.47:0/3734402731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c06df30 0x7fd08c156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:42.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd091e00640 1 --2- 172.21.15.47:0/3734402731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd08c158a30 0x7fd08c15cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:42.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd090a00640 1 -- 172.21.15.47:0/3734402731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd08c158a30 msgr2=0x7fd08c15cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:42.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd090a00640 1 --2- 172.21.15.47:0/3734402731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd08c158a30 0x7fd08c15cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:42.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd090a00640 1 -- 172.21.15.47:0/3734402731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c06df30 msgr2=0x7fd08c156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:42.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd090a00640 1 --2- 172.21.15.47:0/3734402731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c06df30 0x7fd08c156cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:42.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd090a00640 1 -- 172.21.15.47:0/3734402731 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0780094d0 con 0x7fd08c071dc0 2024-02-19T15:26:42.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd091400640 1 --2- 172.21.15.47:0/3734402731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c06df30 0x7fd08c156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:42.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd091e00640 1 --2- 172.21.15.47:0/3734402731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd08c158a30 0x7fd08c15cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:42.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd090a00640 1 --2- 172.21.15.47:0/3734402731 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd08c071dc0 0x7fd08c1463e0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fd07801cdb0 tx=0x7fd07801b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:42.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd089600640 1 -- 172.21.15.47:0/3734402731 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0780b2600 con 0x7fd08c071dc0 2024-02-19T15:26:42.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd089600640 1 -- 172.21.15.47:0/3734402731 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0780b28d0 con 0x7fd08c071dc0 2024-02-19T15:26:42.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd089600640 1 -- 172.21.15.47:0/3734402731 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0780b2c40 con 0x7fd08c071dc0 2024-02-19T15:26:42.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd093e00640 1 -- 172.21.15.47:0/3734402731 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd08c156790 con 0x7fd08c071dc0 2024-02-19T15:26:42.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.855+0000 7fd093e00640 1 -- 172.21.15.47:0/3734402731 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd08c146920 con 0x7fd08c071dc0 2024-02-19T15:26:42.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.859+0000 7fd093e00640 1 -- 172.21.15.47:0/3734402731 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd08c156030 con 0x7fd08c071dc0 2024-02-19T15:26:42.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.859+0000 7fd089600640 1 -- 172.21.15.47:0/3734402731 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd0780bb3b0 con 0x7fd08c071dc0 2024-02-19T15:26:42.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.859+0000 7fd089600640 1 --2- 172.21.15.47:0/3734402731 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd06c066e70 0x7fd06c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:42.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.859+0000 7fd089600640 1 -- 172.21.15.47:0/3734402731 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fd07811f200 con 0x7fd08c071dc0 2024-02-19T15:26:42.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.859+0000 7fd091400640 1 --2- 172.21.15.47:0/3734402731 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd06c066e70 0x7fd06c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:42.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.859+0000 7fd091400640 1 --2- 172.21.15.47:0/3734402731 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd06c066e70 0x7fd06c069330 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd080009840 tx=0x7fd080008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:42.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:42.863+0000 7fd089600640 1 -- 172.21.15.47:0/3734402731 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd08c156030 con 0x7fd08c071dc0 2024-02-19T15:26:43.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:43.063+0000 7fd093e00640 1 -- 172.21.15.47:0/3734402731 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd08c156030 con 0x7fd08c071dc0 2024-02-19T15:26:43.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:43.063+0000 7fd089600640 1 -- 172.21.15.47:0/3734402731 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4947 (secure 0 0 0) 0x7fd08c156030 con 0x7fd08c071dc0 2024-02-19T15:26:43.068 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:43.068 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":64,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7060,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/4006350936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":4006350936},{"type":"v1","addr":"172.21.15.116:6837","nonce":4006350936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/2191173523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2191173523},{"type":"v1","addr":"172.21.15.47:6834","nonce":2191173523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/1012301665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":1012301665},{"type":"v1","addr":"172.21.15.47:6835","nonce":1012301665}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:26:26.721109+0000","modified":"2024-02-19T15:26:41.838479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7054":{"gid":7054,"name":"d","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.116:6835/3852677195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3852677195},{"type":"v1","addr":"172.21.15.116:6835","nonce":3852677195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T15:26:43.068 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 64 2024-02-19T15:26:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:43.063+0000 7fd093e00640 1 -- 172.21.15.47:0/3734402731 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd06c066e70 msgr2=0x7fd06c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:43.063+0000 7fd093e00640 1 --2- 172.21.15.47:0/3734402731 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd06c066e70 0x7fd06c069330 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd080009840 tx=0x7fd080008040 comp rx=0 tx=0).stop 2024-02-19T15:26:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:43.063+0000 7fd093e00640 1 -- 172.21.15.47:0/3734402731 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd08c071dc0 msgr2=0x7fd08c1463e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:43.063+0000 7fd093e00640 1 --2- 172.21.15.47:0/3734402731 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd08c071dc0 0x7fd08c1463e0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fd07801cdb0 tx=0x7fd07801b4a0 comp rx=0 tx=0).stop 2024-02-19T15:26:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:43.063+0000 7fd093e00640 1 -- 172.21.15.47:0/3734402731 shutdown_connections 2024-02-19T15:26:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:43.063+0000 7fd093e00640 1 --2- 172.21.15.47:0/3734402731 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd06c066e70 0x7fd06c069330 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:43.063+0000 7fd093e00640 1 --2- 172.21.15.47:0/3734402731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd08c158a30 0x7fd08c15cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:43.063+0000 7fd093e00640 1 --2- 172.21.15.47:0/3734402731 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd08c071dc0 0x7fd08c1463e0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:43.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:43.063+0000 7fd093e00640 1 --2- 172.21.15.47:0/3734402731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c06df30 0x7fd08c156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:43.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:43.063+0000 7fd093e00640 1 -- 172.21.15.47:0/3734402731 >> 172.21.15.47:0/3734402731 conn(0x7fd08c0b1b80 msgr2=0x7fd08c0ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:43.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:43.063+0000 7fd093e00640 1 -- 172.21.15.47:0/3734402731 shutdown_connections 2024-02-19T15:26:43.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:43.063+0000 7fd093e00640 1 -- 172.21.15.47:0/3734402731 wait complete. 2024-02-19T15:26:43.493 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:26:26.721109+0000', 'modified': '2024-02-19T15:26:41.838479+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 96, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7054}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7054': {'gid': 7054, 'name': 'd', 'rank': 0, 'incarnation': 62, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.116:6835/3852677195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6834', 'nonce': 3852677195}, {'type': 'v1', 'addr': '172.21.15.116:6835', 'nonce': 3852677195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:26:43.493 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7054:up:rejoin 2024-02-19T15:26:44.494 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:26:44.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.639+0000 7fc8ea000640 1 Processor -- start 2024-02-19T15:26:44.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.639+0000 7fc8ea000640 1 -- start start 2024-02-19T15:26:44.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.643+0000 7fc8ea000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc8e406df30 0x7fc8e406e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:44.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.643+0000 7fc8ea000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc8e40ce770 0x7fc8e40cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:44.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.643+0000 7fc8ea000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8e406e8d0 con 0x7fc8e4158a30 2024-02-19T15:26:44.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.643+0000 7fc8ea000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8e4158e10 con 0x7fc8e406df30 2024-02-19T15:26:44.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.643+0000 7fc8ea000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8e415ec20 con 0x7fc8e40ce770 2024-02-19T15:26:44.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.643+0000 7fc8e3400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc8e4158a30 0x7fc8e415bdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37536/0 (socket says 172.21.15.47:37536) 2024-02-19T15:26:44.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.643+0000 7fc8e3400640 1 -- 172.21.15.47:0/3512197795 learned_addr learned my addr 172.21.15.47:0/3512197795 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:44.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.643+0000 7fc8daa00640 1 --2- 172.21.15.47:0/3512197795 >> v2:172.21.15.47:3300/0 conn(0x7fc8e406df30 0x7fc8e406e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:44.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.643+0000 7fc8e3e00640 1 --2- 172.21.15.47:0/3512197795 >> v2:172.21.15.116:3300/0 conn(0x7fc8e40ce770 0x7fc8e40cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:44.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.643+0000 7fc8e2a00640 1 -- 172.21.15.47:0/3512197795 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543928169 0 0) 0x7fc8e406e8d0 con 0x7fc8e4158a30 2024-02-19T15:26:44.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.643+0000 7fc8daa00640 1 -- 172.21.15.47:0/3512197795 >> v2:172.21.15.116:3300/0 conn(0x7fc8e40ce770 msgr2=0x7fc8e40cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:44.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.643+0000 7fc8daa00640 1 --2- 172.21.15.47:0/3512197795 >> v2:172.21.15.116:3300/0 conn(0x7fc8e40ce770 0x7fc8e40cec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:44.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.643+0000 7fc8daa00640 1 -- 172.21.15.47:0/3512197795 >> v1:172.21.15.47:6789/0 conn(0x7fc8e4158a30 legacy=0x7fc8e415bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:44.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.643+0000 7fc8daa00640 1 -- 172.21.15.47:0/3512197795 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e415ef00 con 0x7fc8e406df30 2024-02-19T15:26:44.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.647+0000 7fc8daa00640 1 --2- 172.21.15.47:0/3512197795 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e406df30 0x7fc8e406e390 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fc8e40b33c0 tx=0x7fc8cc01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ec5b1650f0ef60 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:44.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.647+0000 7fc8e2a00640 1 -- 172.21.15.47:0/3512197795 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc0ac410 con 0x7fc8e406df30 2024-02-19T15:26:44.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.647+0000 7fc8e2a00640 1 -- 172.21.15.47:0/3512197795 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8cc0ac570 con 0x7fc8e406df30 2024-02-19T15:26:44.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.647+0000 7fc8e2a00640 1 -- 172.21.15.47:0/3512197795 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc0ac900 con 0x7fc8e406df30 2024-02-19T15:26:44.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.647+0000 7fc8ea000640 1 -- 172.21.15.47:0/3512197795 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e406df30 msgr2=0x7fc8e406e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:44.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.647+0000 7fc8ea000640 1 --2- 172.21.15.47:0/3512197795 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e406df30 0x7fc8e406e390 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fc8e40b33c0 tx=0x7fc8cc01c420 comp rx=0 tx=0).stop 2024-02-19T15:26:44.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.647+0000 7fc8ea000640 1 -- 172.21.15.47:0/3512197795 shutdown_connections 2024-02-19T15:26:44.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.647+0000 7fc8ea000640 1 --2- 172.21.15.47:0/3512197795 >> v2:172.21.15.116:3300/0 conn(0x7fc8e40ce770 0x7fc8e40cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:44.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.647+0000 7fc8ea000640 1 --2- 172.21.15.47:0/3512197795 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e406df30 0x7fc8e406e390 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:44.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.647+0000 7fc8ea000640 1 -- 172.21.15.47:0/3512197795 >> 172.21.15.47:0/3512197795 conn(0x7fc8e40b1b80 msgr2=0x7fc8e40b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:44.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.647+0000 7fc8ea000640 1 -- 172.21.15.47:0/3512197795 shutdown_connections 2024-02-19T15:26:44.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.647+0000 7fc8ea000640 1 -- 172.21.15.47:0/3512197795 wait complete. 2024-02-19T15:26:44.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.647+0000 7fc8ea000640 1 Processor -- start 2024-02-19T15:26:44.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.647+0000 7fc8ea000640 1 -- start start 2024-02-19T15:26:44.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8ea000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8e406df30 0x7fc8e41442a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:44.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8ea000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e40ce770 0x7fc8e414d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:44.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8ea000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e4158a30 0x7fc8e40721d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:44.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8ea000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e414e1f0 con 0x7fc8e4158a30 2024-02-19T15:26:44.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8ea000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e414e4d0 con 0x7fc8e406df30 2024-02-19T15:26:44.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8ea000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e414e7b0 con 0x7fc8e40ce770 2024-02-19T15:26:44.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8e3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e4158a30 0x7fc8e40721d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:44.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8e3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e4158a30 0x7fc8e40721d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:47282/0 (socket says 172.21.15.47:47282) 2024-02-19T15:26:44.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8e3e00640 1 -- 172.21.15.47:0/2034970626 learned_addr learned my addr 172.21.15.47:0/2034970626 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:44.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8e3400640 1 --2- 172.21.15.47:0/2034970626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8e406df30 0x7fc8e41442a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:44.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8daa00640 1 --2- 172.21.15.47:0/2034970626 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e40ce770 0x7fc8e414d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:44.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8e3e00640 1 -- 172.21.15.47:0/2034970626 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e40ce770 msgr2=0x7fc8e414d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:44.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8e3e00640 1 --2- 172.21.15.47:0/2034970626 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e40ce770 0x7fc8e414d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:44.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8e3e00640 1 -- 172.21.15.47:0/2034970626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8e406df30 msgr2=0x7fc8e41442a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:44.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8e3e00640 1 --2- 172.21.15.47:0/2034970626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8e406df30 0x7fc8e41442a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:44.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8e3e00640 1 -- 172.21.15.47:0/2034970626 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc009420 con 0x7fc8e4158a30 2024-02-19T15:26:44.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8e3400640 1 --2- 172.21.15.47:0/2034970626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8e406df30 0x7fc8e41442a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:26:44.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8e3e00640 1 --2- 172.21.15.47:0/2034970626 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e4158a30 0x7fc8e40721d0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc007e40 tx=0x7fc8dc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:44.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8dbe00640 1 -- 172.21.15.47:0/2034970626 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc014620 con 0x7fc8e4158a30 2024-02-19T15:26:44.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8dbe00640 1 -- 172.21.15.47:0/2034970626 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8dc00ee90 con 0x7fc8e4158a30 2024-02-19T15:26:44.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8dbe00640 1 -- 172.21.15.47:0/2034970626 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc0149e0 con 0x7fc8e4158a30 2024-02-19T15:26:44.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8ea000640 1 -- 172.21.15.47:0/2034970626 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8e414ea90 con 0x7fc8e4158a30 2024-02-19T15:26:44.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8ea000640 1 -- 172.21.15.47:0/2034970626 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8e414df00 con 0x7fc8e4158a30 2024-02-19T15:26:44.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8ea000640 1 -- 172.21.15.47:0/2034970626 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8e41482a0 con 0x7fc8e4158a30 2024-02-19T15:26:44.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.651+0000 7fc8dbe00640 1 -- 172.21.15.47:0/2034970626 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc8dc014b40 con 0x7fc8e4158a30 2024-02-19T15:26:44.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.655+0000 7fc8dbe00640 1 --2- 172.21.15.47:0/2034970626 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc8c0066e40 0x7fc8c0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:44.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.655+0000 7fc8dbe00640 1 -- 172.21.15.47:0/2034970626 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fc8dc085590 con 0x7fc8e4158a30 2024-02-19T15:26:44.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.655+0000 7fc8e3400640 1 --2- 172.21.15.47:0/2034970626 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc8c0066e40 0x7fc8c0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:44.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.655+0000 7fc8e3400640 1 --2- 172.21.15.47:0/2034970626 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc8c0066e40 0x7fc8c0069300 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4000f70 tx=0x7fc8d4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:44.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.659+0000 7fc8dbe00640 1 -- 172.21.15.47:0/2034970626 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc8e41482a0 con 0x7fc8e4158a30 2024-02-19T15:26:44.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.855+0000 7fc8ea000640 1 -- 172.21.15.47:0/2034970626 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8e41482a0 con 0x7fc8e4158a30 2024-02-19T15:26:44.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.855+0000 7fc8dbe00640 1 -- 172.21.15.47:0/2034970626 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4947 (secure 0 0 0) 0x7fc8e41482a0 con 0x7fc8e4158a30 2024-02-19T15:26:44.862 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:44.862 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":66,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7060,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/4006350936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":4006350936},{"type":"v1","addr":"172.21.15.116:6837","nonce":4006350936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/2191173523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2191173523},{"type":"v1","addr":"172.21.15.47:6834","nonce":2191173523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/1012301665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":1012301665},{"type":"v1","addr":"172.21.15.47:6835","nonce":1012301665}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:26:26.721109+0000","modified":"2024-02-19T15:26:43.866082+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7054":{"gid":7054,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6835/3852677195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3852677195},{"type":"v1","addr":"172.21.15.116:6835","nonce":3852677195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T15:26:44.863 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 66 2024-02-19T15:26:44.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.859+0000 7fc8ea000640 1 -- 172.21.15.47:0/2034970626 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc8c0066e40 msgr2=0x7fc8c0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:44.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.859+0000 7fc8ea000640 1 --2- 172.21.15.47:0/2034970626 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc8c0066e40 0x7fc8c0069300 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4000f70 tx=0x7fc8d4009040 comp rx=0 tx=0).stop 2024-02-19T15:26:44.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.859+0000 7fc8ea000640 1 -- 172.21.15.47:0/2034970626 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e4158a30 msgr2=0x7fc8e40721d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:44.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.859+0000 7fc8ea000640 1 --2- 172.21.15.47:0/2034970626 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e4158a30 0x7fc8e40721d0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc007e40 tx=0x7fc8dc00e4b0 comp rx=0 tx=0).stop 2024-02-19T15:26:44.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.859+0000 7fc8ea000640 1 -- 172.21.15.47:0/2034970626 shutdown_connections 2024-02-19T15:26:44.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.859+0000 7fc8ea000640 1 --2- 172.21.15.47:0/2034970626 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc8c0066e40 0x7fc8c0069300 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:44.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.859+0000 7fc8ea000640 1 --2- 172.21.15.47:0/2034970626 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e4158a30 0x7fc8e40721d0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:44.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.859+0000 7fc8ea000640 1 --2- 172.21.15.47:0/2034970626 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e40ce770 0x7fc8e414d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:44.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.859+0000 7fc8ea000640 1 --2- 172.21.15.47:0/2034970626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8e406df30 0x7fc8e41442a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:44.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.859+0000 7fc8ea000640 1 -- 172.21.15.47:0/2034970626 >> 172.21.15.47:0/2034970626 conn(0x7fc8e40b1b80 msgr2=0x7fc8e4070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:44.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.859+0000 7fc8ea000640 1 -- 172.21.15.47:0/2034970626 shutdown_connections 2024-02-19T15:26:44.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:44.859+0000 7fc8ea000640 1 -- 172.21.15.47:0/2034970626 wait complete. 2024-02-19T15:26:45.285 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:26:26.721109+0000', 'modified': '2024-02-19T15:26:43.866082+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 96, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7054}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7054': {'gid': 7054, 'name': 'd', 'rank': 0, 'incarnation': 62, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.116:6835/3852677195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6834', 'nonce': 3852677195}, {'type': 'v1', 'addr': '172.21.15.116:6835', 'nonce': 3852677195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:26:45.286 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T15:26:45.286 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-19T15:26:45.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.427+0000 7f9773400640 1 Processor -- start 2024-02-19T15:26:45.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.427+0000 7f9773400640 1 -- start start 2024-02-19T15:26:45.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.427+0000 7f9773400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f976c15e460 0x7f976c0ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:45.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.427+0000 7f9773400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f976c0ceb20 0x7f976c06f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:45.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.427+0000 7f9773400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f976c06c3f0 con 0x7f976c15e080 2024-02-19T15:26:45.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.427+0000 7f9773400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f976c0cefd0 con 0x7f976c0ceb20 2024-02-19T15:26:45.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.427+0000 7f9773400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f976c0cf290 con 0x7f976c15e460 2024-02-19T15:26:45.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.427+0000 7f9770a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f976c15e080 0x7f976c0cff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37548/0 (socket says 172.21.15.47:37548) 2024-02-19T15:26:45.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.427+0000 7f9770a00640 1 -- 172.21.15.47:0/1700450817 learned_addr learned my addr 172.21.15.47:0/1700450817 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:45.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.427+0000 7f9771400640 1 --2- 172.21.15.47:0/1700450817 >> v2:172.21.15.47:3300/0 conn(0x7f976c0ceb20 0x7f976c06f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:45.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.427+0000 7f976be00640 1 --2- 172.21.15.47:0/1700450817 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f976c15e460 0x7f976c0ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:45.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.427+0000 7f976b400640 1 -- 172.21.15.47:0/1700450817 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203864346 0 0) 0x7f976c06c3f0 con 0x7f976c15e080 2024-02-19T15:26:45.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.427+0000 7f976b400640 1 -- 172.21.15.47:0/1700450817 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9754003530 con 0x7f976c15e080 2024-02-19T15:26:45.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.427+0000 7f9771400640 1 -- 172.21.15.47:0/1700450817 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f976c15e460 msgr2=0x7f976c0ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:45.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.427+0000 7f9771400640 1 --2- 172.21.15.47:0/1700450817 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f976c15e460 0x7f976c0ce5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.427+0000 7f9771400640 1 -- 172.21.15.47:0/1700450817 >> v1:172.21.15.47:6789/0 conn(0x7f976c15e080 legacy=0x7f976c0cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:45.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.427+0000 7f9771400640 1 -- 172.21.15.47:0/1700450817 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976c0cf570 con 0x7f976c0ceb20 2024-02-19T15:26:45.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.431+0000 7f9771400640 1 --2- 172.21.15.47:0/1700450817 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f976c0ceb20 0x7f976c06f7c0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f9764009d90 tx=0x7f976401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c2e9d8180132a7e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:45.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.431+0000 7f976b400640 1 -- 172.21.15.47:0/1700450817 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97640a5910 con 0x7f976c0ceb20 2024-02-19T15:26:45.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.431+0000 7f976b400640 1 -- 172.21.15.47:0/1700450817 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97640a5a70 con 0x7f976c0ceb20 2024-02-19T15:26:45.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.431+0000 7f976b400640 1 -- 172.21.15.47:0/1700450817 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97640a5e00 con 0x7f976c0ceb20 2024-02-19T15:26:45.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.431+0000 7f9773400640 1 -- 172.21.15.47:0/1700450817 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f976c0ceb20 msgr2=0x7f976c06f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:45.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.431+0000 7f9773400640 1 --2- 172.21.15.47:0/1700450817 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f976c0ceb20 0x7f976c06f7c0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f9764009d90 tx=0x7f976401c940 comp rx=0 tx=0).stop 2024-02-19T15:26:45.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.431+0000 7f9773400640 1 -- 172.21.15.47:0/1700450817 shutdown_connections 2024-02-19T15:26:45.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.431+0000 7f9773400640 1 --2- 172.21.15.47:0/1700450817 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f976c0ceb20 0x7f976c06f7c0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.431+0000 7f9773400640 1 --2- 172.21.15.47:0/1700450817 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f976c15e460 0x7f976c0ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.431+0000 7f9773400640 1 -- 172.21.15.47:0/1700450817 >> 172.21.15.47:0/1700450817 conn(0x7f976c09aa70 msgr2=0x7f976c09ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:45.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.431+0000 7f9773400640 1 -- 172.21.15.47:0/1700450817 shutdown_connections 2024-02-19T15:26:45.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.431+0000 7f9773400640 1 -- 172.21.15.47:0/1700450817 wait complete. 2024-02-19T15:26:45.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.431+0000 7f9773400640 1 Processor -- start 2024-02-19T15:26:45.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.431+0000 7f9773400640 1 -- start start 2024-02-19T15:26:45.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f9773400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f976c0ceb20 0x7f976c0cde90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:45.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f9773400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f976c15e080 0x7f976c158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:45.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f9773400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f976c15e460 0x7f976c159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:45.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f9773400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976c0cb440 con 0x7f976c15e080 2024-02-19T15:26:45.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f9773400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976c0caba0 con 0x7f976c0ceb20 2024-02-19T15:26:45.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f9773400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f976c0cae80 con 0x7f976c15e460 2024-02-19T15:26:45.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f976be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f976c15e080 0x7f976c158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:45.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f976be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f976c15e080 0x7f976c158e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:47300/0 (socket says 172.21.15.47:47300) 2024-02-19T15:26:45.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f976be00640 1 -- 172.21.15.47:0/310169534 learned_addr learned my addr 172.21.15.47:0/310169534 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:45.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f9771400640 1 --2- 172.21.15.47:0/310169534 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f976c15e460 0x7f976c159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:45.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f9770a00640 1 --2- 172.21.15.47:0/310169534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f976c0ceb20 0x7f976c0cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:45.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f976be00640 1 -- 172.21.15.47:0/310169534 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f976c15e460 msgr2=0x7f976c159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:45.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f976be00640 1 --2- 172.21.15.47:0/310169534 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f976c15e460 0x7f976c159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f976be00640 1 -- 172.21.15.47:0/310169534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f976c0ceb20 msgr2=0x7f976c0cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:45.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f976be00640 1 --2- 172.21.15.47:0/310169534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f976c0ceb20 0x7f976c0cde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f976be00640 1 -- 172.21.15.47:0/310169534 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9764009a70 con 0x7f976c15e080 2024-02-19T15:26:45.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f9770a00640 1 --2- 172.21.15.47:0/310169534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f976c0ceb20 0x7f976c0cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:45.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f976be00640 1 --2- 172.21.15.47:0/310169534 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f976c15e080 0x7f976c158e20 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f975800e870 tx=0x7f975800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:45.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f9768c00640 1 -- 172.21.15.47:0/310169534 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9758009510 con 0x7f976c15e080 2024-02-19T15:26:45.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f9768c00640 1 -- 172.21.15.47:0/310169534 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975800eec0 con 0x7f976c15e080 2024-02-19T15:26:45.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f9773400640 1 -- 172.21.15.47:0/310169534 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f976c0cb160 con 0x7f976c15e080 2024-02-19T15:26:45.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f9768c00640 1 -- 172.21.15.47:0/310169534 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975800cb40 con 0x7f976c15e080 2024-02-19T15:26:45.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.435+0000 7f9773400640 1 -- 172.21.15.47:0/310169534 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f976c0ca800 con 0x7f976c15e080 2024-02-19T15:26:45.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f9768c00640 1 -- 172.21.15.47:0/310169534 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f975800ccc0 con 0x7f976c15e080 2024-02-19T15:26:45.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f9768c00640 1 --2- 172.21.15.47:0/310169534 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f974c066e40 0x7f974c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:45.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f9768c00640 1 -- 172.21.15.47:0/310169534 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f9758084a90 con 0x7f976c15e080 2024-02-19T15:26:45.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f977691c000 1 Processor -- start 2024-02-19T15:26:45.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f9770a00640 1 --2- 172.21.15.47:0/310169534 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f974c066e40 0x7f974c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:45.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f977691c000 1 -- start start 2024-02-19T15:26:45.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f9770a00640 1 --2- 172.21.15.47:0/310169534 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f974c066e40 0x7f974c069300 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f9760005890 tx=0x7f9760009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:45.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f977691c000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7fa39920 0x560c7fa39d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:45.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f977691c000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa3a240 0x560c7f9ef9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:45.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f977691c000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7f9eff30 0x560c7f9f0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:45.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f977691c000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c7f8b7d00 con 0x560c7fa3a240 2024-02-19T15:26:45.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f977691c000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c7f849430 con 0x560c7f9eff30 2024-02-19T15:26:45.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f977691c000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c7f846960 con 0x560c7fa39920 2024-02-19T15:26:45.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f976be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa3a240 0x560c7f9ef9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:45.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f9770a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7fa39920 0x560c7fa39d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:45.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f976be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa3a240 0x560c7f9ef9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:47304/0 (socket says 172.21.15.47:47304) 2024-02-19T15:26:45.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f976be00640 1 -- 172.21.15.47:0/1823530805 learned_addr learned my addr 172.21.15.47:0/1823530805 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:45.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f9771400640 1 --2- 172.21.15.47:0/1823530805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7f9eff30 0x560c7f9f0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:45.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f976be00640 1 -- 172.21.15.47:0/1823530805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7fa39920 msgr2=0x560c7fa39d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:45.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f976be00640 1 --2- 172.21.15.47:0/1823530805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7fa39920 0x560c7fa39d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f976be00640 1 -- 172.21.15.47:0/1823530805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7f9eff30 msgr2=0x560c7f9f0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:45.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f976be00640 1 --2- 172.21.15.47:0/1823530805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7f9eff30 0x560c7f9f0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f976be00640 1 -- 172.21.15.47:0/1823530805 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c7f538890 con 0x560c7fa3a240 2024-02-19T15:26:45.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f976be00640 1 --2- 172.21.15.47:0/1823530805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa3a240 0x560c7f9ef9f0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f9758082fc0 tx=0x7f97580557f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea305e387252a0d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:45.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.439+0000 7f975d600640 1 -- 172.21.15.47:0/1823530805 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9758053970 con 0x560c7fa3a240 2024-02-19T15:26:45.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f975d600640 1 -- 172.21.15.47:0/1823530805 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9758051ce0 con 0x560c7fa3a240 2024-02-19T15:26:45.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f975d600640 1 -- 172.21.15.47:0/1823530805 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975807cd40 con 0x560c7fa3a240 2024-02-19T15:26:45.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f977691c000 1 -- 172.21.15.47:0/1823530805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa3a240 msgr2=0x560c7f9ef9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:45.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f977691c000 1 --2- 172.21.15.47:0/1823530805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa3a240 0x560c7f9ef9f0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f9758082fc0 tx=0x7f97580557f0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f977691c000 1 -- 172.21.15.47:0/1823530805 shutdown_connections 2024-02-19T15:26:45.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f977691c000 1 --2- 172.21.15.47:0/1823530805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7f9eff30 0x560c7f9f0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f977691c000 1 --2- 172.21.15.47:0/1823530805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa3a240 0x560c7f9ef9f0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f977691c000 1 --2- 172.21.15.47:0/1823530805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7fa39920 0x560c7fa39d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f977691c000 1 -- 172.21.15.47:0/1823530805 >> 172.21.15.47:0/1823530805 conn(0x560c7f9f62a0 msgr2=0x560c7f9ee2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:45.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f977691c000 1 -- 172.21.15.47:0/1823530805 shutdown_connections 2024-02-19T15:26:45.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f977691c000 1 -- 172.21.15.47:0/1823530805 wait complete. 2024-02-19T15:26:45.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f977691c000 1 -- start start 2024-02-19T15:26:45.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f977691c000 1 Processor -- start 2024-02-19T15:26:45.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f977691c000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7f9eff30 0x560c7fa392c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:45.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f977691c000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7fa39920 0x560c7f9f4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:45.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f977691c000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa3a240 0x560c7f9f4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:45.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f977691c000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c7f842140 con 0x560c7fa3a240 2024-02-19T15:26:45.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f977691c000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c7f7dc630 con 0x560c7f9eff30 2024-02-19T15:26:45.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f977691c000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c7f7dc2d0 con 0x560c7fa39920 2024-02-19T15:26:45.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f977691c000 10 client.0 _ll_register_callbacks cb 0x560c7f8f21c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:26:45.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f9771400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa3a240 0x560c7f9f4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:45.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f976be00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7fa39920 0x560c7f9f4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:45.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f9771400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa3a240 0x560c7f9f4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:47320/0 (socket says 172.21.15.47:47320) 2024-02-19T15:26:45.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f9771400640 1 -- 172.21.15.47:0/1859975959 learned_addr learned my addr 172.21.15.47:0/1859975959 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:45.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f9770a00640 1 --2- 172.21.15.47:0/1859975959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7f9eff30 0x560c7fa392c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:45.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f976be00640 1 -- 172.21.15.47:0/1859975959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7f9eff30 msgr2=0x560c7fa392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:45.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f976be00640 1 --2- 172.21.15.47:0/1859975959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7f9eff30 0x560c7fa392c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f976be00640 1 -- 172.21.15.47:0/1859975959 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa3a240 msgr2=0x560c7f9f4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:45.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f976be00640 1 --2- 172.21.15.47:0/1859975959 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa3a240 0x560c7f9f4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f976be00640 1 -- 172.21.15.47:0/1859975959 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f976c0ca800 con 0x560c7fa39920 2024-02-19T15:26:45.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f9771400640 1 --2- 172.21.15.47:0/1859975959 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa3a240 0x560c7f9f4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:45.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f976be00640 1 --2- 172.21.15.47:0/1859975959 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7fa39920 0x560c7f9f4530 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9758000c10 tx=0x7f975800cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d1d96b2aeac9f44 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f9752000640 10 client.0 ms_handle_connect on v2:172.21.15.116:3301/0 2024-02-19T15:26:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f9752000640 1 -- 172.21.15.47:0/1859975959 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9758072070 con 0x560c7fa39920 2024-02-19T15:26:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.443+0000 7f977691c000 1 -- 172.21.15.47:0/1859975959 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560c7f7f5850 con 0x560c7fa39920 2024-02-19T15:26:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f9752000640 1 -- 172.21.15.47:0/1859975959 <== mon.2 v2:172.21.15.116:3301/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f975807cb40 con 0x560c7fa39920 2024-02-19T15:26:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f9752000640 1 -- 172.21.15.47:0/1859975959 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f975807cd00 con 0x560c7fa39920 2024-02-19T15:26:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f977691c000 10 client.6386 fetch_fsmap learned FSMap version 66 2024-02-19T15:26:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f977691c000 1 -- 172.21.15.47:0/1859975959 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x560c7f801e70 con 0x560c7fa39920 2024-02-19T15:26:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f9752000640 1 -- 172.21.15.47:0/1859975959 <== mon.2 v2:172.21.15.116:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560c7f7f5850 con 0x560c7fa39920 2024-02-19T15:26:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f9752000640 1 -- 172.21.15.47:0/1859975959 <== mon.2 v2:172.21.15.116:3301/0 5 ==== fsmap(e 66) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9758079020 con 0x560c7fa39920 2024-02-19T15:26:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f977691c000 10 client.6386 fetch_fsmap finished waiting for FSMap version 66 2024-02-19T15:26:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f977691c000 10 client.6386 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T15:26:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f977691c000 20 client.6386 populate_metadata read hostname 'smithi047' 2024-02-19T15:26:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f977691c000 1 --2- 172.21.15.47:0/1859975959 >> [v2:172.21.15.116:6834/3852677195,v1:172.21.15.116:6835/3852677195] conn(0x560c7fa3aaf0 0x560c7fa3aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f977691c000 4 client.6386 mds_command: new command op to 7054 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T15:26:45.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f977691c000 1 -- 172.21.15.47:0/1859975959 --> [v2:172.21.15.116:6834/3852677195,v1:172.21.15.116:6835/3852677195] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560c7f83ef20 con 0x560c7fa3aaf0 2024-02-19T15:26:45.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f9770a00640 1 --2- 172.21.15.47:0/1859975959 >> [v2:172.21.15.116:6834/3852677195,v1:172.21.15.116:6835/3852677195] conn(0x560c7fa3aaf0 0x560c7fa3aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:45.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f9770a00640 1 --2- 172.21.15.47:0/1859975959 >> [v2:172.21.15.116:6834/3852677195,v1:172.21.15.116:6835/3852677195] conn(0x560c7fa3aaf0 0x560c7fa3aed0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cdbcceb3167d8f5 server_cookie=5733dcc7bef989dc in_seq=0 out_seq=0 2024-02-19T15:26:45.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f9752000640 10 client.6386 ms_handle_connect on v2:172.21.15.116:6834/3852677195 2024-02-19T15:26:45.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f9752000640 1 -- 172.21.15.47:0/1859975959 <== mds.0 v2:172.21.15.116:6834/3852677195 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560c7f83ef20 con 0x560c7fa3aaf0 2024-02-19T15:26:45.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f9752000640 10 client.6386 handle_command_reply: tid=0 2024-02-19T15:26:45.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f977691c000 1 client.6386 shutdown 2024-02-19T15:26:45.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f977691c000 1 -- 172.21.15.47:0/1859975959 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7fa39920 msgr2=0x560c7f9f4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:45.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f977691c000 1 --2- 172.21.15.47:0/1859975959 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7fa39920 0x560c7f9f4530 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9758000c10 tx=0x7f975800cf70 comp rx=0 tx=0).stop 2024-02-19T15:26:45.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f977691c000 1 -- 172.21.15.47:0/1859975959 shutdown_connections 2024-02-19T15:26:45.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f977691c000 1 --2- 172.21.15.47:0/1859975959 >> [v2:172.21.15.116:6834/3852677195,v1:172.21.15.116:6835/3852677195] conn(0x560c7fa3aaf0 0x560c7fa3aed0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f977691c000 1 --2- 172.21.15.47:0/1859975959 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa3a240 0x560c7f9f4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f977691c000 1 --2- 172.21.15.47:0/1859975959 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7fa39920 0x560c7f9f4530 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f9752000640 0 client.6386 ms_handle_reset on v2:172.21.15.116:6834/3852677195 2024-02-19T15:26:45.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f977691c000 1 --2- 172.21.15.47:0/1859975959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7f9eff30 0x560c7fa392c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f977691c000 1 -- 172.21.15.47:0/1859975959 >> 172.21.15.47:0/1859975959 conn(0x560c7f9f62a0 msgr2=0x560c7f9ee9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f977691c000 1 -- 172.21.15.47:0/1859975959 shutdown_connections 2024-02-19T15:26:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f977691c000 1 -- 172.21.15.47:0/1859975959 wait complete. 2024-02-19T15:26:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.447+0000 7f977691c000 20 client.6386 trim_cache size 0 max 16384 2024-02-19T15:26:45.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.463+0000 7f977691c000 1 Processor -- start 2024-02-19T15:26:45.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.463+0000 7f977691c000 1 -- start start 2024-02-19T15:26:45.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7f9eff30 0x560c7fa45ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:45.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa39920 0x560c7fa46400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:45.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7fa3a240 0x560c7fa4c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:45.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c7f4f9940 con 0x560c7fa39920 2024-02-19T15:26:45.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c7f83c2c0 con 0x560c7fa3a240 2024-02-19T15:26:45.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c7f9f5aa0 con 0x560c7f9eff30 2024-02-19T15:26:45.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f976be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa39920 0x560c7fa46400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:45.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f976be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa39920 0x560c7fa46400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:47334/0 (socket says 172.21.15.47:47334) 2024-02-19T15:26:45.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f9770a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7f9eff30 0x560c7fa45ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:45.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f976be00640 1 -- 172.21.15.47:0/518786919 learned_addr learned my addr 172.21.15.47:0/518786919 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:45.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f9771400640 1 --2- 172.21.15.47:0/518786919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7fa3a240 0x560c7fa4c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:45.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f9770a00640 1 -- 172.21.15.47:0/518786919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7fa3a240 msgr2=0x560c7fa4c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:45.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f9770a00640 1 --2- 172.21.15.47:0/518786919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7fa3a240 0x560c7fa4c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f9770a00640 1 -- 172.21.15.47:0/518786919 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa39920 msgr2=0x560c7fa46400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:45.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f9770a00640 1 --2- 172.21.15.47:0/518786919 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa39920 0x560c7fa46400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f9770a00640 1 -- 172.21.15.47:0/518786919 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c7f7e7e10 con 0x560c7f9eff30 2024-02-19T15:26:45.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f976be00640 1 --2- 172.21.15.47:0/518786919 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa39920 0x560c7fa46400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:45.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f9770a00640 1 --2- 172.21.15.47:0/518786919 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7f9eff30 0x560c7fa45ec0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x560c7f7e7650 tx=0x7f9760011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3aa9d336dc2cf25f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:45.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f975cc00640 1 -- 172.21.15.47:0/518786919 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97600281d0 con 0x560c7f9eff30 2024-02-19T15:26:45.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f975cc00640 1 -- 172.21.15.47:0/518786919 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9760028330 con 0x560c7f9eff30 2024-02-19T15:26:45.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f975cc00640 1 -- 172.21.15.47:0/518786919 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9760028880 con 0x560c7f9eff30 2024-02-19T15:26:45.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 -- 172.21.15.47:0/518786919 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7f9eff30 msgr2=0x560c7fa45ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:45.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 --2- 172.21.15.47:0/518786919 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7f9eff30 0x560c7fa45ec0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x560c7f7e7650 tx=0x7f9760011000 comp rx=0 tx=0).stop 2024-02-19T15:26:45.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 -- 172.21.15.47:0/518786919 shutdown_connections 2024-02-19T15:26:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 --2- 172.21.15.47:0/518786919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7fa3a240 0x560c7fa4c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 --2- 172.21.15.47:0/518786919 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7fa39920 0x560c7fa46400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 --2- 172.21.15.47:0/518786919 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7f9eff30 0x560c7fa45ec0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 -- 172.21.15.47:0/518786919 >> 172.21.15.47:0/518786919 conn(0x560c7f9f62a0 msgr2=0x560c7fa457b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 -- 172.21.15.47:0/518786919 shutdown_connections 2024-02-19T15:26:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 -- 172.21.15.47:0/518786919 wait complete. 2024-02-19T15:26:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 -- start start 2024-02-19T15:26:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 Processor -- start 2024-02-19T15:26:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7f9eff30 0x560c7fa82d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7fa39920 0x560c7fa83280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7fa3a240 0x560c7fa837c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c7f5391b0 con 0x560c7f9eff30 2024-02-19T15:26:45.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c7f7e1f10 con 0x560c7fa39920 2024-02-19T15:26:45.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c7fa4f130 con 0x560c7fa3a240 2024-02-19T15:26:45.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f977691c000 10 client.0 _ll_register_callbacks cb 0x7f97540036c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:26:45.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f9770a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7f9eff30 0x560c7fa82d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:45.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f9770a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7f9eff30 0x560c7fa82d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:47338/0 (socket says 172.21.15.47:47338) 2024-02-19T15:26:45.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f9770a00640 1 -- 172.21.15.47:0/1535937689 learned_addr learned my addr 172.21.15.47:0/1535937689 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:45.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.467+0000 7f9771400640 1 --2- 172.21.15.47:0/1535937689 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7fa3a240 0x560c7fa837c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:45.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f976be00640 1 --2- 172.21.15.47:0/1535937689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7fa39920 0x560c7fa83280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:45.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9770a00640 1 -- 172.21.15.47:0/1535937689 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7fa3a240 msgr2=0x560c7fa837c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:45.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9770a00640 1 --2- 172.21.15.47:0/1535937689 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7fa3a240 0x560c7fa837c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9770a00640 1 -- 172.21.15.47:0/1535937689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7fa39920 msgr2=0x560c7fa83280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:45.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9770a00640 1 --2- 172.21.15.47:0/1535937689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7fa39920 0x560c7fa83280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9770a00640 1 -- 172.21.15.47:0/1535937689 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c7f801e70 con 0x560c7f9eff30 2024-02-19T15:26:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9770a00640 1 --2- 172.21.15.47:0/1535937689 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7f9eff30 0x560c7fa82d40 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f9760025040 tx=0x7f97600010b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92b2fe2a7fcf99ce server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9752000640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:26:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9752000640 1 -- 172.21.15.47:0/1535937689 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9760028a90 con 0x560c7f9eff30 2024-02-19T15:26:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9752000640 1 -- 172.21.15.47:0/1535937689 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f9760028bf0 con 0x560c7f9eff30 2024-02-19T15:26:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 1 -- 172.21.15.47:0/1535937689 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560c7f80afa0 con 0x560c7f9eff30 2024-02-19T15:26:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9752000640 1 -- 172.21.15.47:0/1535937689 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9760028db0 con 0x560c7f9eff30 2024-02-19T15:26:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 10 client.7395 fetch_fsmap learned FSMap version 66 2024-02-19T15:26:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 1 -- 172.21.15.47:0/1535937689 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x7f976000f040 con 0x560c7f9eff30 2024-02-19T15:26:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9752000640 1 -- 172.21.15.47:0/1535937689 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560c7f80afa0 con 0x560c7f9eff30 2024-02-19T15:26:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9752000640 1 -- 172.21.15.47:0/1535937689 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 66) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9760020070 con 0x560c7f9eff30 2024-02-19T15:26:45.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 10 client.7395 fetch_fsmap finished waiting for FSMap version 66 2024-02-19T15:26:45.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 10 client.7395 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T15:26:45.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 20 client.7395 populate_metadata read hostname 'smithi047' 2024-02-19T15:26:45.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 1 --2- 172.21.15.47:0/1535937689 >> [v2:172.21.15.116:6834/3852677195,v1:172.21.15.116:6835/3852677195] conn(0x560c7fa83d00 0x560c7fa840e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:45.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 4 client.7395 mds_command: new command op to 7054 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T15:26:45.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 1 -- 172.21.15.47:0/1535937689 --> [v2:172.21.15.116:6834/3852677195,v1:172.21.15.116:6835/3852677195] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560c7f83ab60 con 0x560c7fa83d00 2024-02-19T15:26:45.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f976be00640 1 --2- 172.21.15.47:0/1535937689 >> [v2:172.21.15.116:6834/3852677195,v1:172.21.15.116:6835/3852677195] conn(0x560c7fa83d00 0x560c7fa840e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:45.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f976be00640 1 --2- 172.21.15.47:0/1535937689 >> [v2:172.21.15.116:6834/3852677195,v1:172.21.15.116:6835/3852677195] conn(0x560c7fa83d00 0x560c7fa840e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d738a4f232f1688 server_cookie=5997e6b620f976c1 in_seq=0 out_seq=0 2024-02-19T15:26:45.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9752000640 10 client.7395 ms_handle_connect on v2:172.21.15.116:6834/3852677195 2024-02-19T15:26:45.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9752000640 1 -- 172.21.15.47:0/1535937689 <== mds.0 v2:172.21.15.116:6834/3852677195 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x560c7f83ab60 con 0x560c7fa83d00 2024-02-19T15:26:45.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9752000640 10 client.7395 handle_command_reply: tid=0 2024-02-19T15:26:45.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 1 client.7395 shutdown 2024-02-19T15:26:45.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 1 -- 172.21.15.47:0/1535937689 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7f9eff30 msgr2=0x560c7fa82d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:45.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 1 --2- 172.21.15.47:0/1535937689 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7f9eff30 0x560c7fa82d40 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f9760025040 tx=0x7f97600010b0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 1 -- 172.21.15.47:0/1535937689 shutdown_connections 2024-02-19T15:26:45.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 1 --2- 172.21.15.47:0/1535937689 >> [v2:172.21.15.116:6834/3852677195,v1:172.21.15.116:6835/3852677195] conn(0x560c7fa83d00 0x560c7fa840e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 1 --2- 172.21.15.47:0/1535937689 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560c7fa3a240 0x560c7fa837c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 1 --2- 172.21.15.47:0/1535937689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7fa39920 0x560c7fa83280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 1 --2- 172.21.15.47:0/1535937689 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560c7f9eff30 0x560c7fa82d40 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9752000640 0 client.7395 ms_handle_reset on v2:172.21.15.116:6834/3852677195 2024-02-19T15:26:45.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 1 -- 172.21.15.47:0/1535937689 >> 172.21.15.47:0/1535937689 conn(0x560c7f9f62a0 msgr2=0x560c7fa45ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:45.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 1 -- 172.21.15.47:0/1535937689 shutdown_connections 2024-02-19T15:26:45.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 1 -- 172.21.15.47:0/1535937689 wait complete. 2024-02-19T15:26:45.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f977691c000 20 client.7395 trim_cache size 0 max 16384 2024-02-19T15:26:45.489 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:26:45.489 INFO:teuthology.orchestra.run.smithi047.stdout: "cluster_fsid": "ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e", 2024-02-19T15:26:45.489 INFO:teuthology.orchestra.run.smithi047.stdout: "whoami": 0, 2024-02-19T15:26:45.489 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 7054, 2024-02-19T15:26:45.489 INFO:teuthology.orchestra.run.smithi047.stdout: "want_state": "up:active", 2024-02-19T15:26:45.489 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "up:active", 2024-02-19T15:26:45.489 INFO:teuthology.orchestra.run.smithi047.stdout: "fs_name": "cephfs", 2024-02-19T15:26:45.490 INFO:teuthology.orchestra.run.smithi047.stdout: "rank_uptime": 4.6440501520000002, 2024-02-19T15:26:45.490 INFO:teuthology.orchestra.run.smithi047.stdout: "mdsmap_epoch": 66, 2024-02-19T15:26:45.490 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch": 97, 2024-02-19T15:26:45.490 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch_barrier": 97, 2024-02-19T15:26:45.490 INFO:teuthology.orchestra.run.smithi047.stdout: "uptime": 5.3909095090000001 2024-02-19T15:26:45.490 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:26:45.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9773400640 1 -- 172.21.15.47:0/310169534 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f974c066e40 msgr2=0x7f974c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:45.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9773400640 1 --2- 172.21.15.47:0/310169534 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f974c066e40 0x7f974c069300 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f9760005890 tx=0x7f9760009040 comp rx=0 tx=0).stop 2024-02-19T15:26:45.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9773400640 1 -- 172.21.15.47:0/310169534 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f976c15e080 msgr2=0x7f976c158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:45.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9773400640 1 --2- 172.21.15.47:0/310169534 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f976c15e080 0x7f976c158e20 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f975800e870 tx=0x7f975800ec70 comp rx=0 tx=0).stop 2024-02-19T15:26:45.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9773400640 1 -- 172.21.15.47:0/310169534 shutdown_connections 2024-02-19T15:26:45.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9773400640 1 --2- 172.21.15.47:0/310169534 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f974c066e40 0x7f974c069300 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9773400640 1 --2- 172.21.15.47:0/310169534 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f976c15e460 0x7f976c159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9773400640 1 --2- 172.21.15.47:0/310169534 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f976c15e080 0x7f976c158e20 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9773400640 1 --2- 172.21.15.47:0/310169534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f976c0ceb20 0x7f976c0cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:45.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9773400640 1 -- 172.21.15.47:0/310169534 >> 172.21.15.47:0/310169534 conn(0x7f976c09aa70 msgr2=0x7f976c0cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:45.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9773400640 1 -- 172.21.15.47:0/310169534 shutdown_connections 2024-02-19T15:26:45.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:45.471+0000 7f9773400640 1 -- 172.21.15.47:0/310169534 wait complete. 2024-02-19T15:26:45.497 DEBUG:teuthology.orchestra.run.smithi047:> stdin-killer -- python3 -c ' 2024-02-19T15:26:45.497 DEBUG:teuthology.orchestra.run.smithi047:> import os 2024-02-19T15:26:45.497 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:26:45.497 DEBUG:teuthology.orchestra.run.smithi047:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-02-19T15:26:45.497 DEBUG:teuthology.orchestra.run.smithi047:> subdir = "delete_me" 2024-02-19T15:26:45.497 DEBUG:teuthology.orchestra.run.smithi047:> size_unit = 1024 2024-02-19T15:26:45.497 DEBUG:teuthology.orchestra.run.smithi047:> file_multiplier = 200 2024-02-19T15:26:45.497 DEBUG:teuthology.orchestra.run.smithi047:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-02-19T15:26:45.497 DEBUG:teuthology.orchestra.run.smithi047:> for i in range(0, file_multiplier): 2024-02-19T15:26:45.498 DEBUG:teuthology.orchestra.run.smithi047:> for size in range(0, 16*size_unit, size_unit): 2024-02-19T15:26:45.498 DEBUG:teuthology.orchestra.run.smithi047:> filename = "{0}_{1}.bin".format(i, size // size_unit) 2024-02-19T15:26:45.498 DEBUG:teuthology.orchestra.run.smithi047:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-02-19T15:26:45.498 DEBUG:teuthology.orchestra.run.smithi047:> f.write(size * '"'"'x'"'"') 2024-02-19T15:26:45.498 DEBUG:teuthology.orchestra.run.smithi047:> ' 2024-02-19T15:26:46.831 INFO:tasks.ceph.mds.d.smithi116.stderr:2024-02-19T15:26:46.829+0000 7f1277400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T15:26:48.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-02-19T15:26:48.969 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get mds_max_purge_ops 2024-02-19T15:26:48.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:26:48.969 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-02-19T15:26:49.174 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_max_purge_ops":"100000000"} 2024-02-19T15:26:49.174 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_ops": "100000000" } 2024-02-19T15:26:49.175 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config get mds_max_purge_files 2024-02-19T15:26:49.393 INFO:teuthology.orchestra.run.smithi116.stdout:{"mds_max_purge_files":"6"} 2024-02-19T15:26:49.394 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_files": "6" } 2024-02-19T15:26:49.394 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:26:49.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.547+0000 7fd609400640 1 Processor -- start 2024-02-19T15:26:49.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.547+0000 7fd609400640 1 -- start start 2024-02-19T15:26:49.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.547+0000 7fd609400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd60415ec60 0x7fd6040b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:49.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.547+0000 7fd609400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd60406b620 0x7fd60406e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:49.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.547+0000 7fd609400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6040b44b0 con 0x7fd60415f040 2024-02-19T15:26:49.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.547+0000 7fd609400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6040b4fb0 con 0x7fd60406b620 2024-02-19T15:26:49.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.547+0000 7fd609400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6040b4710 con 0x7fd60415ec60 2024-02-19T15:26:49.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.547+0000 7fd5f9e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd60415f040 0x7fd60406af10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37562/0 (socket says 172.21.15.47:37562) 2024-02-19T15:26:49.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.547+0000 7fd5f9e00640 1 -- 172.21.15.47:0/1169496302 learned_addr learned my addr 172.21.15.47:0/1169496302 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:49.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.547+0000 7fd602800640 1 --2- 172.21.15.47:0/1169496302 >> v2:172.21.15.116:3300/0 conn(0x7fd60415ec60 0x7fd6040b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:49.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.547+0000 7fd603200640 1 --2- 172.21.15.47:0/1169496302 >> v2:172.21.15.47:3300/0 conn(0x7fd60406b620 0x7fd60406e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:49.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.551+0000 7fd603200640 1 -- 172.21.15.47:0/1169496302 >> v2:172.21.15.116:3300/0 conn(0x7fd60415ec60 msgr2=0x7fd6040b3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:49.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.551+0000 7fd603200640 1 --2- 172.21.15.47:0/1169496302 >> v2:172.21.15.116:3300/0 conn(0x7fd60415ec60 0x7fd6040b3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:49.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.551+0000 7fd603200640 1 -- 172.21.15.47:0/1169496302 >> v1:172.21.15.47:6789/0 conn(0x7fd60415f040 legacy=0x7fd60406af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:49.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.551+0000 7fd601e00640 1 -- 172.21.15.47:0/1169496302 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700603637 0 0) 0x7fd6040b44b0 con 0x7fd60415f040 2024-02-19T15:26:49.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.551+0000 7fd603200640 1 -- 172.21.15.47:0/1169496302 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6040b49f0 con 0x7fd60406b620 2024-02-19T15:26:49.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.551+0000 7fd602800640 1 --2- 172.21.15.47:0/1169496302 >> v2:172.21.15.116:3300/0 conn(0x7fd60415ec60 0x7fd6040b3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:49.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.551+0000 7fd603200640 1 --2- 172.21.15.47:0/1169496302 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd60406b620 0x7fd60406e680 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fd6040b3980 tx=0x7fd5fc01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=207dce6844fd86b0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:49.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.551+0000 7fd601e00640 1 -- 172.21.15.47:0/1169496302 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc0ac620 con 0x7fd60406b620 2024-02-19T15:26:49.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.551+0000 7fd601e00640 1 -- 172.21.15.47:0/1169496302 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5fc0ac780 con 0x7fd60406b620 2024-02-19T15:26:49.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.551+0000 7fd601e00640 1 -- 172.21.15.47:0/1169496302 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc0acb10 con 0x7fd60406b620 2024-02-19T15:26:49.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.551+0000 7fd609400640 1 -- 172.21.15.47:0/1169496302 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd60406b620 msgr2=0x7fd60406e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:49.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.551+0000 7fd609400640 1 --2- 172.21.15.47:0/1169496302 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd60406b620 0x7fd60406e680 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fd6040b3980 tx=0x7fd5fc01c3c0 comp rx=0 tx=0).stop 2024-02-19T15:26:49.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.551+0000 7fd609400640 1 -- 172.21.15.47:0/1169496302 shutdown_connections 2024-02-19T15:26:49.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.551+0000 7fd609400640 1 --2- 172.21.15.47:0/1169496302 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd60406b620 0x7fd60406e680 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:49.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.551+0000 7fd609400640 1 --2- 172.21.15.47:0/1169496302 >> v2:172.21.15.116:3300/0 conn(0x7fd60415ec60 0x7fd6040b3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:49.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd609400640 1 -- 172.21.15.47:0/1169496302 >> 172.21.15.47:0/1169496302 conn(0x7fd60406c9b0 msgr2=0x7fd6040b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:49.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd609400640 1 -- 172.21.15.47:0/1169496302 shutdown_connections 2024-02-19T15:26:49.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd609400640 1 -- 172.21.15.47:0/1169496302 wait complete. 2024-02-19T15:26:49.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd609400640 1 Processor -- start 2024-02-19T15:26:49.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd609400640 1 -- start start 2024-02-19T15:26:49.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd609400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd60406b620 0x7fd604158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:49.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd609400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd60415ec60 0x7fd60414da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:49.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd609400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd60415f040 0x7fd6040cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:49.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd609400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60414cd50 con 0x7fd60415f040 2024-02-19T15:26:49.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd609400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60414d5f0 con 0x7fd60415ec60 2024-02-19T15:26:49.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd609400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd604157c70 con 0x7fd60406b620 2024-02-19T15:26:49.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd603200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd60415f040 0x7fd6040cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:49.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd603200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd60415f040 0x7fd6040cf6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:47354/0 (socket says 172.21.15.47:47354) 2024-02-19T15:26:49.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd603200640 1 -- 172.21.15.47:0/4270871229 learned_addr learned my addr 172.21.15.47:0/4270871229 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:49.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd602800640 1 --2- 172.21.15.47:0/4270871229 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd60406b620 0x7fd604158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:49.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd5f9e00640 1 --2- 172.21.15.47:0/4270871229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd60415ec60 0x7fd60414da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:49.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd603200640 1 -- 172.21.15.47:0/4270871229 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd60406b620 msgr2=0x7fd604158c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:49.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd603200640 1 --2- 172.21.15.47:0/4270871229 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd60406b620 0x7fd604158c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:49.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd603200640 1 -- 172.21.15.47:0/4270871229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd60415ec60 msgr2=0x7fd60414da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:49.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd603200640 1 --2- 172.21.15.47:0/4270871229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd60415ec60 0x7fd60414da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:49.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd603200640 1 -- 172.21.15.47:0/4270871229 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f4009420 con 0x7fd60415f040 2024-02-19T15:26:49.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd603200640 1 --2- 172.21.15.47:0/4270871229 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd60415f040 0x7fd6040cf6b0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc019ff0 tx=0x7fd5fc019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:49.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd5fb400640 1 -- 172.21.15.47:0/4270871229 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc0b5410 con 0x7fd60415f040 2024-02-19T15:26:49.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd5fb400640 1 -- 172.21.15.47:0/4270871229 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5fc0b5590 con 0x7fd60415f040 2024-02-19T15:26:49.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd609400640 1 -- 172.21.15.47:0/4270871229 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5fc009420 con 0x7fd60415f040 2024-02-19T15:26:49.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.555+0000 7fd5fb400640 1 -- 172.21.15.47:0/4270871229 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc0b5940 con 0x7fd60415f040 2024-02-19T15:26:49.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.559+0000 7fd609400640 1 -- 172.21.15.47:0/4270871229 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd60414d280 con 0x7fd60415f040 2024-02-19T15:26:49.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.559+0000 7fd609400640 1 -- 172.21.15.47:0/4270871229 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd604068120 con 0x7fd60415f040 2024-02-19T15:26:49.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.559+0000 7fd5fb400640 1 -- 172.21.15.47:0/4270871229 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd5fc0b3040 con 0x7fd60415f040 2024-02-19T15:26:49.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.559+0000 7fd5fb400640 1 --2- 172.21.15.47:0/4270871229 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd5e0066e40 0x7fd5e0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:49.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.559+0000 7fd5fb400640 1 -- 172.21.15.47:0/4270871229 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fd5fc11ec00 con 0x7fd60415f040 2024-02-19T15:26:49.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.559+0000 7fd602800640 1 --2- 172.21.15.47:0/4270871229 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd5e0066e40 0x7fd5e0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:49.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.559+0000 7fd602800640 1 --2- 172.21.15.47:0/4270871229 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd5e0066e40 0x7fd5e0069300 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4009cf0 tx=0x7fd5f4009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:49.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.563+0000 7fd5fb400640 1 -- 172.21.15.47:0/4270871229 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd5fc0eb9e0 con 0x7fd60415f040 2024-02-19T15:26:49.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.787+0000 7fd609400640 1 -- 172.21.15.47:0/4270871229 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd60414d730 con 0x7fd60415f040 2024-02-19T15:26:49.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.787+0000 7fd5fb400640 1 -- 172.21.15.47:0/4270871229 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4947 (secure 0 0 0) 0x7fd5fc0ef0a0 con 0x7fd60415f040 2024-02-19T15:26:49.793 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:49.793 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":66,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7060,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/4006350936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":4006350936},{"type":"v1","addr":"172.21.15.116:6837","nonce":4006350936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/2191173523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2191173523},{"type":"v1","addr":"172.21.15.47:6834","nonce":2191173523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/1012301665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":1012301665},{"type":"v1","addr":"172.21.15.47:6835","nonce":1012301665}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:26:26.721109+0000","modified":"2024-02-19T15:26:43.866082+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7054":{"gid":7054,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6835/3852677195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3852677195},{"type":"v1","addr":"172.21.15.116:6835","nonce":3852677195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T15:26:49.793 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 66 2024-02-19T15:26:49.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.791+0000 7fd609400640 1 -- 172.21.15.47:0/4270871229 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd5e0066e40 msgr2=0x7fd5e0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:49.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.791+0000 7fd609400640 1 --2- 172.21.15.47:0/4270871229 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd5e0066e40 0x7fd5e0069300 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4009cf0 tx=0x7fd5f4009190 comp rx=0 tx=0).stop 2024-02-19T15:26:49.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.791+0000 7fd609400640 1 -- 172.21.15.47:0/4270871229 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd60415f040 msgr2=0x7fd6040cf6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:49.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.791+0000 7fd609400640 1 --2- 172.21.15.47:0/4270871229 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd60415f040 0x7fd6040cf6b0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc019ff0 tx=0x7fd5fc019160 comp rx=0 tx=0).stop 2024-02-19T15:26:49.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.791+0000 7fd609400640 1 -- 172.21.15.47:0/4270871229 shutdown_connections 2024-02-19T15:26:49.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.791+0000 7fd609400640 1 --2- 172.21.15.47:0/4270871229 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd5e0066e40 0x7fd5e0069300 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:49.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.791+0000 7fd609400640 1 --2- 172.21.15.47:0/4270871229 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd60415f040 0x7fd6040cf6b0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:49.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.791+0000 7fd609400640 1 --2- 172.21.15.47:0/4270871229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd60415ec60 0x7fd60414da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:49.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.791+0000 7fd609400640 1 --2- 172.21.15.47:0/4270871229 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd60406b620 0x7fd604158c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:49.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.791+0000 7fd609400640 1 -- 172.21.15.47:0/4270871229 >> 172.21.15.47:0/4270871229 conn(0x7fd60406c9b0 msgr2=0x7fd6040cea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:49.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.791+0000 7fd609400640 1 -- 172.21.15.47:0/4270871229 shutdown_connections 2024-02-19T15:26:49.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:49.791+0000 7fd609400640 1 -- 172.21.15.47:0/4270871229 wait complete. 2024-02-19T15:26:49.814 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump 2024-02-19T15:26:49.992 INFO:teuthology.orchestra.run.smithi116.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":45,"msgr_send_messages":47,"msgr_recv_bytes":18012,"msgr_send_bytes":5720935,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.035473751,"msgr_running_send_time":0.021689841,"msgr_running_recv_time":0.006602778,"msgr_running_fast_dispatch_time":0.004640631,"msgr_send_messages_queue_lat":{"avgcount":46,"sum":0.012425638,"avgtime":0.000270122},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000170,"avgtime":0.000000085},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":90832},"AsyncMessenger::Worker-1":{"msgr_recv_messages":6513,"msgr_send_messages":13126,"msgr_recv_bytes":2158407,"msgr_send_bytes":7137406,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.500549459,"msgr_running_send_time":0.307807862,"msgr_running_recv_time":0.549455385,"msgr_running_fast_dispatch_time":0.000587766,"msgr_send_messages_queue_lat":{"avgcount":13126,"sum":0.236064625,"avgtime":0.000017984},"msgr_handle_ack_lat":{"avgcount":10958,"sum":0.001984375,"avgtime":0.000000181},"msgr_recv_encrypted_bytes":103840,"msgr_send_encrypted_bytes":6064},"AsyncMessenger::Worker-2":{"msgr_recv_messages":18,"msgr_send_messages":21,"msgr_recv_bytes":14332,"msgr_send_bytes":1424465,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.015831056,"msgr_running_send_time":0.009246112,"msgr_running_recv_time":0.004162741,"msgr_running_fast_dispatch_time":0.002058916,"msgr_send_messages_queue_lat":{"avgcount":20,"sum":0.007058456,"avgtime":0.000352922},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":77712},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":71,"sum":1.970763298,"avgtime":0.027757229}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000145208,"avgtime":0.000036302}},"mds":{"request":3487,"reply":3485,"reply_latency":{"avgcount":3485,"sum":381.848438017,"avgtime":0.109569135},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3419,"inodes_top":0,"inodes_bottom":3419,"inodes_pin_tail":0,"inodes_pinned":3419,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":3488,"traverse_hit":3488,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3199,"root_rbytes":24516608,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":1,"ceph_cap_op_grant":3315,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2993,"handle_client_caps_dirty":2992,"handle_client_cap_release":0,"process_request_cap_release":284},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":276,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":276,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":6480,"evex":0,"evtrm":0,"ev":6483,"evexg":0,"evexd":0,"segadd":9,"segex":0,"segtrm":0,"seg":11,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":17304419,"rdpos":4195244,"jlat":{"avgcount":11,"sum":0.627229132,"avgtime":0.057020830},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":3487,"dn+":3487,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":99856,"heap":248112},"mds_server":{"dispatch_client_request":3487,"dispatch_server_request":0,"handle_client_request":3487,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":381.004064656,"avgtime":0.119063770},"req_getattr_latency":{"avgcount":2,"sum":0.167551542,"avgtime":0.083775771},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003020866,"avgtime":0.003020866},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.395814412,"avgtime":0.065969068},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":276,"sum":0.277986541,"avgtime":0.001007197},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3387,"average_load":3387,"avg_session_uptime":985,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":237205,"buffer_anon_items":67,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":51696,"mds_co_items":189,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":62,"op_send_bytes":12889538,"op_resend":0,"op_reply":62,"op_latency":{"avgcount":62,"sum":0.215997928,"avgtime":0.003483837},"op_inflight":0,"oplen_avg":{"avgcount":62,"sum":113},"op":62,"op_r":36,"op_w":26,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":11,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":8,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":10,"omap_rd":48,"omap_del":0},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":260,"max":104857600,"get_started":0,"get":6572,"get_sum":1700664,"get_or_fail_fail":0,"get_or_fail_success":6572,"take":0,"take_sum":0,"put":6571,"put_sum":1700404,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":62,"take_sum":12890472,"put":62,"put_sum":12890472,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":62,"take_sum":62,"put":62,"put_sum":62,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55fa43f90dc0":{"val":527203,"max":3758096384,"get_started":0,"get":6480,"get_sum":13108917,"get_or_fail_fail":0,"get_or_fail_success":6480,"take":0,"take_sum":0,"put":11,"put_sum":12581714,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-02-19T15:26:49.993 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 8.5e-08, "sum": 1.7e-07 }, "msgr_recv_bytes": 18012, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 45, "msgr_running_fast_dispatch_time": 0.004640631, "msgr_running_recv_time": 0.006602778, "msgr_running_send_time": 0.021689841, "msgr_running_total_time": 0.035473751, "msgr_send_bytes": 5720935, "msgr_send_encrypted_bytes": 90832, "msgr_send_messages": 47, "msgr_send_messages_queue_lat": { "avgcount": 46, "avgtime": 0.000270122, "sum": 0.012425638 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 10958, "avgtime": 1.81e-07, "sum": 0.001984375 }, "msgr_recv_bytes": 2158407, "msgr_recv_encrypted_bytes": 103840, "msgr_recv_messages": 6513, "msgr_running_fast_dispatch_time": 0.000587766, "msgr_running_recv_time": 0.549455385, "msgr_running_send_time": 0.307807862, "msgr_running_total_time": 0.500549459, "msgr_send_bytes": 7137406, "msgr_send_encrypted_bytes": 6064, "msgr_send_messages": 13126, "msgr_send_messages_queue_lat": { "avgcount": 13126, "avgtime": 1.7984e-05, "sum": 0.236064625 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 14332, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 18, "msgr_running_fast_dispatch_time": 0.002058916, "msgr_running_recv_time": 0.004162741, "msgr_running_send_time": 0.009246112, "msgr_running_total_time": 0.015831056, "msgr_send_bytes": 1424465, "msgr_send_encrypted_bytes": 77712, "msgr_send_messages": 21, "msgr_send_messages_queue_lat": { "avgcount": 20, "avgtime": 0.000352922, "sum": 0.007058456 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 71, "avgtime": 0.027757229, "sum": 1.970763298 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 3.6302e-05, "sum": 0.000145208 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3315, "ceph_cap_op_revoke": 1, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 2993, "handle_client_caps_dirty": 2992, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3419, "inodes_bottom": 3419, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3419, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 284, "q": 0, "reply": 3485, "reply_latency": { "avgcount": 3485, "avgtime": 0.109569135, "sum": 381.848438017 }, "request": 3487, "root_rbytes": 24516608, "root_rfiles": 3199, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3488, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3488, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 276, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 276, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 6483, "evadd": 6480, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 11, "avgtime": 0.05702083, "sum": 0.627229132 }, "rdpos": 4195244, "replayed": 3, "seg": 11, "segadd": 9, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 17304419 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 3487, "dn+": 3487, "dn-": 0, "heap": 248112, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 99856 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3487, "dispatch_server_request": 0, "handle_client_request": 3487, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.11906377, "sum": 381.004064656 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.083775771, "sum": 0.167551542 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003020866, "sum": 0.003020866 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.065969068, "sum": 0.395814412 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 276, "avgtime": 0.001007197, "sum": 0.277986541 } }, "mds_sessions": { "average_load": 3387, "avg_session_uptime": 985, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3387 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 237205, "buffer_anon_items": 67, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 51696, "mds_co_items": 189, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 0, "omap_rd": 48, "omap_wr": 10, "op": 62, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 62, "avgtime": 0.003483837, "sum": 0.215997928 }, "op_pg": 0, "op_r": 36, "op_reply": 62, "op_resend": 0, "op_rmw": 0, "op_send": 62, "op_send_bytes": 12889538, "op_w": 26, "oplen_avg": { "avgcount": 62, "sum": 113 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 8, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 11, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 6572, "get_or_fail_fail": 0, "get_or_fail_success": 6572, "get_started": 0, "get_sum": 1700664, "max": 104857600, "put": 6571, "put_sum": 1700404, "take": 0, "take_sum": 0, "val": 260, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 62, "put_sum": 12890472, "take": 62, "take_sum": 12890472, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 62, "put_sum": 62, "take": 62, "take_sum": 62, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55fa43f90dc0": { "get": 6480, "get_or_fail_fail": 0, "get_or_fail_success": 6480, "get_started": 0, "get_sum": 13108917, "max": 3758096384, "put": 11, "put_sum": 12581714, "take": 0, "take_sum": 0, "val": 527203, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-02-19T15:26:49.993 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/276, 0/276 2024-02-19T15:26:50.995 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:26:51.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.135+0000 7f1073800640 1 Processor -- start 2024-02-19T15:26:51.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.135+0000 7f1073800640 1 -- start start 2024-02-19T15:26:51.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.135+0000 7f1073800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f106c15c320 0x7f106c0b0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:51.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.135+0000 7f1073800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f106c071680 0x7f106c0cf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:51.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.135+0000 7f1073800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f106c0cfae0 0x7f106c0acff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:51.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.135+0000 7f1073800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f106c15c700 con 0x7f106c15c320 2024-02-19T15:26:51.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.135+0000 7f1073800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f106c073280 con 0x7f106c071680 2024-02-19T15:26:51.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.135+0000 7f1073800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106c0b2550 con 0x7f106c0cfae0 2024-02-19T15:26:51.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.135+0000 7f1070e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f106c15c320 0x7f106c0b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:51.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.135+0000 7f1070e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f106c15c320 0x7f106c0b0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:47368/0 (socket says 172.21.15.47:47368) 2024-02-19T15:26:51.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.135+0000 7f1070e00640 1 -- 172.21.15.47:0/3042683173 learned_addr learned my addr 172.21.15.47:0/3042683173 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:51.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.135+0000 7f106be00640 1 --2- 172.21.15.47:0/3042683173 >> v2:172.21.15.116:3300/0 conn(0x7f106c071680 0x7f106c0cf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:51.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.135+0000 7f1071800640 1 --2- 172.21.15.47:0/3042683173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f106c0cfae0 0x7f106c0acff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:51.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.135+0000 7f1070e00640 1 -- 172.21.15.47:0/3042683173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f106c0cfae0 msgr2=0x7f106c0acff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:51.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.135+0000 7f1070e00640 1 --2- 172.21.15.47:0/3042683173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f106c0cfae0 0x7f106c0acff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:51.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.135+0000 7f1070e00640 1 -- 172.21.15.47:0/3042683173 >> v2:172.21.15.116:3300/0 conn(0x7f106c071680 msgr2=0x7f106c0cf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:51.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.135+0000 7f1070e00640 1 --2- 172.21.15.47:0/3042683173 >> v2:172.21.15.116:3300/0 conn(0x7f106c071680 0x7f106c0cf5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:51.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.135+0000 7f1070e00640 1 -- 172.21.15.47:0/3042683173 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c0b2830 con 0x7f106c15c320 2024-02-19T15:26:51.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.139+0000 7f1070e00640 1 --2- 172.21.15.47:0/3042683173 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f106c15c320 0x7f106c0b0690 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f106c0d1010 tx=0x7f106001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eae384d252d6708 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:51.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.139+0000 7f106b400640 1 -- 172.21.15.47:0/3042683173 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10600ac640 con 0x7f106c15c320 2024-02-19T15:26:51.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.139+0000 7f106b400640 1 -- 172.21.15.47:0/3042683173 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10600ac7a0 con 0x7f106c15c320 2024-02-19T15:26:51.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.139+0000 7f106b400640 1 -- 172.21.15.47:0/3042683173 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10600acb30 con 0x7f106c15c320 2024-02-19T15:26:51.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.139+0000 7f1073800640 1 -- 172.21.15.47:0/3042683173 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f106c15c320 msgr2=0x7f106c0b0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:51.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.139+0000 7f1073800640 1 --2- 172.21.15.47:0/3042683173 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f106c15c320 0x7f106c0b0690 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f106c0d1010 tx=0x7f106001c400 comp rx=0 tx=0).stop 2024-02-19T15:26:51.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.139+0000 7f1073800640 1 -- 172.21.15.47:0/3042683173 shutdown_connections 2024-02-19T15:26:51.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.139+0000 7f1073800640 1 --2- 172.21.15.47:0/3042683173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f106c0cfae0 0x7f106c0acff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:51.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.139+0000 7f1073800640 1 --2- 172.21.15.47:0/3042683173 >> v2:172.21.15.116:3300/0 conn(0x7f106c071680 0x7f106c0cf5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:51.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.139+0000 7f1073800640 1 --2- 172.21.15.47:0/3042683173 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f106c15c320 0x7f106c0b0690 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:51.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.139+0000 7f1073800640 1 -- 172.21.15.47:0/3042683173 >> 172.21.15.47:0/3042683173 conn(0x7f106c0b5750 msgr2=0x7f106c0b1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:51.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.139+0000 7f1073800640 1 -- 172.21.15.47:0/3042683173 shutdown_connections 2024-02-19T15:26:51.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.139+0000 7f1073800640 1 -- 172.21.15.47:0/3042683173 wait complete. 2024-02-19T15:26:51.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1073800640 1 Processor -- start 2024-02-19T15:26:51.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1073800640 1 -- start start 2024-02-19T15:26:51.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1073800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f106c071680 0x7f106c153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:51.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1073800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f106c0cfae0 0x7f106c1f56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:51.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1073800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f106c15c320 0x7f106c1569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:51.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1073800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c1f6700 con 0x7f106c071680 2024-02-19T15:26:51.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1073800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c1537d0 con 0x7f106c15c320 2024-02-19T15:26:51.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1073800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106c1531c0 con 0x7f106c0cfae0 2024-02-19T15:26:51.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1070e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f106c071680 0x7f106c153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:51.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1070e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f106c071680 0x7f106c153c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:47374/0 (socket says 172.21.15.47:47374) 2024-02-19T15:26:51.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1070e00640 1 -- 172.21.15.47:0/4161804406 learned_addr learned my addr 172.21.15.47:0/4161804406 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:51.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1071800640 1 --2- 172.21.15.47:0/4161804406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f106c15c320 0x7f106c1569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:51.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f106be00640 1 --2- 172.21.15.47:0/4161804406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f106c0cfae0 0x7f106c1f56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:51.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1070e00640 1 -- 172.21.15.47:0/4161804406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f106c0cfae0 msgr2=0x7f106c1f56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:51.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1070e00640 1 --2- 172.21.15.47:0/4161804406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f106c0cfae0 0x7f106c1f56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:51.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1070e00640 1 -- 172.21.15.47:0/4161804406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f106c15c320 msgr2=0x7f106c1569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:51.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1070e00640 1 --2- 172.21.15.47:0/4161804406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f106c15c320 0x7f106c1569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:51.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1070e00640 1 -- 172.21.15.47:0/4161804406 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10600094d0 con 0x7f106c071680 2024-02-19T15:26:51.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f106be00640 1 --2- 172.21.15.47:0/4161804406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f106c0cfae0 0x7f106c1f56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:26:51.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1070e00640 1 --2- 172.21.15.47:0/4161804406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f106c071680 0x7f106c153c60 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f106001a030 tx=0x7f106001ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:51.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1068c00640 1 -- 172.21.15.47:0/4161804406 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10600b7620 con 0x7f106c071680 2024-02-19T15:26:51.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1068c00640 1 -- 172.21.15.47:0/4161804406 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10600b7780 con 0x7f106c071680 2024-02-19T15:26:51.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1073800640 1 -- 172.21.15.47:0/4161804406 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f106c1f6140 con 0x7f106c071680 2024-02-19T15:26:51.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1068c00640 1 -- 172.21.15.47:0/4161804406 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10600b7b10 con 0x7f106c071680 2024-02-19T15:26:51.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1073800640 1 -- 172.21.15.47:0/4161804406 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106c153460 con 0x7f106c071680 2024-02-19T15:26:51.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.143+0000 7f1073800640 1 -- 172.21.15.47:0/4161804406 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f106c068e70 con 0x7f106c071680 2024-02-19T15:26:51.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.147+0000 7f1068c00640 1 -- 172.21.15.47:0/4161804406 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f10600b3040 con 0x7f106c071680 2024-02-19T15:26:51.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.147+0000 7f1068c00640 1 --2- 172.21.15.47:0/4161804406 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f104c066e40 0x7f104c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:51.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.147+0000 7f1068c00640 1 -- 172.21.15.47:0/4161804406 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f106011f0f0 con 0x7f106c071680 2024-02-19T15:26:51.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.147+0000 7f106be00640 1 --2- 172.21.15.47:0/4161804406 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f104c066e40 0x7f104c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:51.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.147+0000 7f106be00640 1 --2- 172.21.15.47:0/4161804406 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f104c066e40 0x7f104c069300 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f106c151030 tx=0x7f1058008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:51.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.151+0000 7f1068c00640 1 -- 172.21.15.47:0/4161804406 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f10600ebeb0 con 0x7f106c071680 2024-02-19T15:26:51.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.363+0000 7f1073800640 1 -- 172.21.15.47:0/4161804406 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f106c073060 con 0x7f106c071680 2024-02-19T15:26:51.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.363+0000 7f1068c00640 1 -- 172.21.15.47:0/4161804406 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4947 (secure 0 0 0) 0x7f10600ef570 con 0x7f106c071680 2024-02-19T15:26:51.370 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:51.370 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":66,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7060,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/4006350936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":4006350936},{"type":"v1","addr":"172.21.15.116:6837","nonce":4006350936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/2191173523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2191173523},{"type":"v1","addr":"172.21.15.47:6834","nonce":2191173523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/1012301665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":1012301665},{"type":"v1","addr":"172.21.15.47:6835","nonce":1012301665}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:26:26.721109+0000","modified":"2024-02-19T15:26:43.866082+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7054":{"gid":7054,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6835/3852677195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3852677195},{"type":"v1","addr":"172.21.15.116:6835","nonce":3852677195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T15:26:51.370 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 66 2024-02-19T15:26:51.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.367+0000 7f1073800640 1 -- 172.21.15.47:0/4161804406 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f104c066e40 msgr2=0x7f104c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:51.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.367+0000 7f1073800640 1 --2- 172.21.15.47:0/4161804406 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f104c066e40 0x7f104c069300 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f106c151030 tx=0x7f1058008040 comp rx=0 tx=0).stop 2024-02-19T15:26:51.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.367+0000 7f1073800640 1 -- 172.21.15.47:0/4161804406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f106c071680 msgr2=0x7f106c153c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:51.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.367+0000 7f1073800640 1 --2- 172.21.15.47:0/4161804406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f106c071680 0x7f106c153c60 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f106001a030 tx=0x7f106001ac40 comp rx=0 tx=0).stop 2024-02-19T15:26:51.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.367+0000 7f1073800640 1 -- 172.21.15.47:0/4161804406 shutdown_connections 2024-02-19T15:26:51.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.367+0000 7f1073800640 1 --2- 172.21.15.47:0/4161804406 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f104c066e40 0x7f104c069300 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:51.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.367+0000 7f1073800640 1 --2- 172.21.15.47:0/4161804406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f106c15c320 0x7f106c1569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:51.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.367+0000 7f1073800640 1 --2- 172.21.15.47:0/4161804406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f106c0cfae0 0x7f106c1f56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:51.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.367+0000 7f1073800640 1 --2- 172.21.15.47:0/4161804406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f106c071680 0x7f106c153c60 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:51.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.367+0000 7f1073800640 1 -- 172.21.15.47:0/4161804406 >> 172.21.15.47:0/4161804406 conn(0x7f106c0b5750 msgr2=0x7f106c0ce530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:51.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.367+0000 7f1073800640 1 -- 172.21.15.47:0/4161804406 shutdown_connections 2024-02-19T15:26:51.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:51.367+0000 7f1073800640 1 -- 172.21.15.47:0/4161804406 wait complete. 2024-02-19T15:26:51.392 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump 2024-02-19T15:26:51.579 INFO:teuthology.orchestra.run.smithi116.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":47,"msgr_send_messages":49,"msgr_recv_bytes":18592,"msgr_send_bytes":6509920,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.037808637,"msgr_running_send_time":0.023597595,"msgr_running_recv_time":0.006642187,"msgr_running_fast_dispatch_time":0.005013103,"msgr_send_messages_queue_lat":{"avgcount":48,"sum":0.012598981,"avgtime":0.000262478},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000170,"avgtime":0.000000085},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":90832},"AsyncMessenger::Worker-1":{"msgr_recv_messages":7922,"msgr_send_messages":14407,"msgr_recv_bytes":2627912,"msgr_send_bytes":7660865,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.576948808,"msgr_running_send_time":0.350654977,"msgr_running_recv_time":0.640026867,"msgr_running_fast_dispatch_time":0.000665509,"msgr_send_messages_queue_lat":{"avgcount":14407,"sum":0.273940158,"avgtime":0.000019014},"msgr_handle_ack_lat":{"avgcount":11999,"sum":0.002179774,"avgtime":0.000000181},"msgr_recv_encrypted_bytes":103840,"msgr_send_encrypted_bytes":6064},"AsyncMessenger::Worker-2":{"msgr_recv_messages":18,"msgr_send_messages":21,"msgr_recv_bytes":14332,"msgr_send_bytes":1424465,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.015831056,"msgr_running_send_time":0.009246112,"msgr_running_recv_time":0.004162741,"msgr_running_fast_dispatch_time":0.002058916,"msgr_send_messages_queue_lat":{"avgcount":20,"sum":0.007058456,"avgtime":0.000352922},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":77712},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":74,"sum":2.370357793,"avgtime":0.032031862}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000145208,"avgtime":0.000036302}},"mds":{"request":4243,"reply":4240,"reply_latency":{"avgcount":4240,"sum":383.111687921,"avgtime":0.090356530},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3785,"inodes_top":0,"inodes_bottom":3785,"inodes_pin_tail":0,"inodes_pinned":3785,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":4244,"traverse_hit":4244,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3199,"root_rbytes":24516608,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":1,"ceph_cap_op_grant":3602,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3598,"handle_client_caps_dirty":2996,"handle_client_cap_release":46,"process_request_cap_release":1040},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1032,"num_strays_delayed":109,"num_strays_enqueuing":0,"strays_created":1032,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":7815,"evex":0,"evtrm":0,"ev":7818,"evexg":0,"evexd":0,"segadd":10,"segex":0,"segtrm":0,"seg":12,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":20801440,"rdpos":4195244,"jlat":{"avgcount":12,"sum":0.634784199,"avgtime":0.052898683},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":4242,"dn+":4242,"dn-":0,"cap":3045,"cap+":3202,"cap-":157,"rss":88108,"heap":248112},"mds_server":{"dispatch_client_request":4243,"dispatch_server_request":0,"handle_client_request":4243,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":381.004064656,"avgtime":0.119063770},"req_getattr_latency":{"avgcount":2,"sum":0.167551542,"avgtime":0.083775771},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003020866,"avgtime":0.003020866},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.395814412,"avgtime":0.065969068},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1031,"sum":1.541236445,"avgtime":0.001494894},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4074,"average_load":4074,"avg_session_uptime":990,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":5016679,"buffer_anon_items":1501,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":11451917,"mds_co_items":83510,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":64,"op_send_bytes":13677821,"op_resend":0,"op_reply":64,"op_latency":{"avgcount":64,"sum":0.231997775,"avgtime":0.003624965},"op_inflight":0,"oplen_avg":{"avgcount":64,"sum":117},"op":64,"op_r":36,"op_w":28,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":12,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":8,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":12,"omap_rd":48,"omap_del":1},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":7982,"get_sum":2066043,"get_or_fail_fail":0,"get_or_fail_success":7982,"take":0,"take_sum":0,"put":7982,"put_sum":2066043,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":64,"take_sum":13678755,"put":64,"put_sum":13678755,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":64,"take_sum":64,"put":64,"put_sum":64,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55fa43f90dc0":{"val":3257877,"max":3758096384,"get_started":0,"get":7816,"get_sum":16609162,"get_or_fail_fail":0,"get_or_fail_success":7816,"take":0,"take_sum":0,"put":12,"put_sum":13351285,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-02-19T15:26:51.581 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 8.5e-08, "sum": 1.7e-07 }, "msgr_recv_bytes": 18592, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 47, "msgr_running_fast_dispatch_time": 0.005013103, "msgr_running_recv_time": 0.006642187, "msgr_running_send_time": 0.023597595, "msgr_running_total_time": 0.037808637, "msgr_send_bytes": 6509920, "msgr_send_encrypted_bytes": 90832, "msgr_send_messages": 49, "msgr_send_messages_queue_lat": { "avgcount": 48, "avgtime": 0.000262478, "sum": 0.012598981 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 11999, "avgtime": 1.81e-07, "sum": 0.002179774 }, "msgr_recv_bytes": 2627912, "msgr_recv_encrypted_bytes": 103840, "msgr_recv_messages": 7922, "msgr_running_fast_dispatch_time": 0.000665509, "msgr_running_recv_time": 0.640026867, "msgr_running_send_time": 0.350654977, "msgr_running_total_time": 0.576948808, "msgr_send_bytes": 7660865, "msgr_send_encrypted_bytes": 6064, "msgr_send_messages": 14407, "msgr_send_messages_queue_lat": { "avgcount": 14407, "avgtime": 1.9014e-05, "sum": 0.273940158 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 14332, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 18, "msgr_running_fast_dispatch_time": 0.002058916, "msgr_running_recv_time": 0.004162741, "msgr_running_send_time": 0.009246112, "msgr_running_total_time": 0.015831056, "msgr_send_bytes": 1424465, "msgr_send_encrypted_bytes": 77712, "msgr_send_messages": 21, "msgr_send_messages_queue_lat": { "avgcount": 20, "avgtime": 0.000352922, "sum": 0.007058456 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 74, "avgtime": 0.032031862, "sum": 2.370357793 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 3.6302e-05, "sum": 0.000145208 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3602, "ceph_cap_op_revoke": 1, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 46, "handle_client_caps": 3598, "handle_client_caps_dirty": 2996, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3785, "inodes_bottom": 3785, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3785, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1040, "q": 0, "reply": 4240, "reply_latency": { "avgcount": 4240, "avgtime": 0.09035653, "sum": 383.111687921 }, "request": 4243, "root_rbytes": 24516608, "root_rfiles": 3199, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 4244, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4244, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1032, "num_strays_delayed": 109, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1032, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 7818, "evadd": 7815, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 12, "avgtime": 0.052898683, "sum": 0.634784199 }, "rdpos": 4195244, "replayed": 3, "seg": 12, "segadd": 10, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 20801440 }, "mds_mem": { "cap": 3045, "cap+": 3202, "cap-": 157, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4242, "dn+": 4242, "dn-": 0, "heap": 248112, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 88108 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4243, "dispatch_server_request": 0, "handle_client_request": 4243, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.11906377, "sum": 381.004064656 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.083775771, "sum": 0.167551542 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003020866, "sum": 0.003020866 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.065969068, "sum": 0.395814412 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1031, "avgtime": 0.001494894, "sum": 1.541236445 } }, "mds_sessions": { "average_load": 4074, "avg_session_uptime": 990, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4074 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 5016679, "buffer_anon_items": 1501, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 11451917, "mds_co_items": 83510, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 1, "omap_rd": 48, "omap_wr": 12, "op": 64, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 64, "avgtime": 0.003624965, "sum": 0.231997775 }, "op_pg": 0, "op_r": 36, "op_reply": 64, "op_resend": 0, "op_rmw": 0, "op_send": 64, "op_send_bytes": 13677821, "op_w": 28, "oplen_avg": { "avgcount": 64, "sum": 117 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 8, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 12, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 7982, "get_or_fail_fail": 0, "get_or_fail_success": 7982, "get_started": 0, "get_sum": 2066043, "max": 104857600, "put": 7982, "put_sum": 2066043, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 64, "put_sum": 13678755, "take": 64, "take_sum": 13678755, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 64, "put_sum": 64, "take": 64, "take_sum": 64, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55fa43f90dc0": { "get": 7816, "get_or_fail_fail": 0, "get_or_fail_success": 7816, "get_started": 0, "get_sum": 16609162, "max": 3758096384, "put": 12, "put_sum": 13351285, "take": 0, "take_sum": 0, "val": 3257877, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-02-19T15:26:51.581 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1032, 0/1032 2024-02-19T15:26:52.581 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:26:52.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.719+0000 7f400d000640 1 Processor -- start 2024-02-19T15:26:52.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.719+0000 7f400d000640 1 -- start start 2024-02-19T15:26:52.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.719+0000 7f400d000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4008158a30 0x7f400815bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:52.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.719+0000 7f400d000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f400806df30 0x7f400806e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:52.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.719+0000 7f400d000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4008071dc0 0x7f4008072250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:52.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.719+0000 7f400d000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4008158e10 con 0x7f4008071dc0 2024-02-19T15:26:52.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.719+0000 7f400d000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f400806e8d0 con 0x7f4008158a30 2024-02-19T15:26:52.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.719+0000 7f400d000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f400815c300 con 0x7f400806df30 2024-02-19T15:26:52.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.719+0000 7f4006800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4008071dc0 0x7f4008072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:52.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.719+0000 7f3ffd400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f400806df30 0x7f400806e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:52.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.719+0000 7f4005e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4008158a30 0x7f400815bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:52.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.719+0000 7f3ffd400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f400806df30 0x7f400806e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:32788/0 (socket says 172.21.15.47:32788) 2024-02-19T15:26:52.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.719+0000 7f3ffd400640 1 -- 172.21.15.47:0/2370734080 learned_addr learned my addr 172.21.15.47:0/2370734080 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:52.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.719+0000 7f4006800640 1 -- 172.21.15.47:0/2370734080 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f400806df30 msgr2=0x7f400806e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:52.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.719+0000 7f4006800640 1 --2- 172.21.15.47:0/2370734080 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f400806df30 0x7f400806e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:52.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.719+0000 7f4006800640 1 -- 172.21.15.47:0/2370734080 >> v2:172.21.15.116:3300/0 conn(0x7f4008158a30 msgr2=0x7f400815bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:52.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.719+0000 7f4006800640 1 --2- 172.21.15.47:0/2370734080 >> v2:172.21.15.116:3300/0 conn(0x7f4008158a30 0x7f400815bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:52.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.719+0000 7f4006800640 1 -- 172.21.15.47:0/2370734080 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400815ec20 con 0x7f4008071dc0 2024-02-19T15:26:52.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.723+0000 7f4006800640 1 --2- 172.21.15.47:0/2370734080 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4008071dc0 0x7f4008072250 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f40080b6110 tx=0x7f400001eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2daa8374f48f90f6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:52.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.723+0000 7f4005400640 1 -- 172.21.15.47:0/2370734080 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40000a7ab0 con 0x7f4008071dc0 2024-02-19T15:26:52.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.723+0000 7f4005400640 1 -- 172.21.15.47:0/2370734080 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40000a7c10 con 0x7f4008071dc0 2024-02-19T15:26:52.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.723+0000 7f4005400640 1 -- 172.21.15.47:0/2370734080 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40000af440 con 0x7f4008071dc0 2024-02-19T15:26:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.723+0000 7f400d000640 1 -- 172.21.15.47:0/2370734080 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4008071dc0 msgr2=0x7f4008072250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.723+0000 7f400d000640 1 --2- 172.21.15.47:0/2370734080 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4008071dc0 0x7f4008072250 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f40080b6110 tx=0x7f400001eb20 comp rx=0 tx=0).stop 2024-02-19T15:26:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.723+0000 7f400d000640 1 -- 172.21.15.47:0/2370734080 shutdown_connections 2024-02-19T15:26:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.723+0000 7f400d000640 1 --2- 172.21.15.47:0/2370734080 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4008071dc0 0x7f4008072250 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.723+0000 7f400d000640 1 --2- 172.21.15.47:0/2370734080 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f400806df30 0x7f400806e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.723+0000 7f400d000640 1 --2- 172.21.15.47:0/2370734080 >> v2:172.21.15.116:3300/0 conn(0x7f4008158a30 0x7f400815bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.723+0000 7f400d000640 1 -- 172.21.15.47:0/2370734080 >> 172.21.15.47:0/2370734080 conn(0x7f40080b1b80 msgr2=0x7f40080b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.723+0000 7f400d000640 1 -- 172.21.15.47:0/2370734080 shutdown_connections 2024-02-19T15:26:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.723+0000 7f400d000640 1 -- 172.21.15.47:0/2370734080 wait complete. 2024-02-19T15:26:52.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f400d000640 1 Processor -- start 2024-02-19T15:26:52.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f400d000640 1 -- start start 2024-02-19T15:26:52.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f400d000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f400806df30 0x7f4008158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:52.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f400d000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4008071dc0 0x7f4008148950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:52.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f400d000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4008158a30 0x7f400815b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:52.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f400d000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400815cb30 con 0x7f400806df30 2024-02-19T15:26:52.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f400d000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40081598d0 con 0x7f4008158a30 2024-02-19T15:26:52.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f400d000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f400815c4e0 con 0x7f4008071dc0 2024-02-19T15:26:52.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f4005e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f400806df30 0x7f4008158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:52.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f4005e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f400806df30 0x7f4008158f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:47392/0 (socket says 172.21.15.47:47392) 2024-02-19T15:26:52.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f4005e00640 1 -- 172.21.15.47:0/2293315882 learned_addr learned my addr 172.21.15.47:0/2293315882 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f4006800640 1 --2- 172.21.15.47:0/2293315882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4008158a30 0x7f400815b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f3ffd400640 1 --2- 172.21.15.47:0/2293315882 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4008071dc0 0x7f4008148950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f4005e00640 1 -- 172.21.15.47:0/2293315882 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4008071dc0 msgr2=0x7f4008148950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f4005e00640 1 --2- 172.21.15.47:0/2293315882 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4008071dc0 0x7f4008148950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f4005e00640 1 -- 172.21.15.47:0/2293315882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4008158a30 msgr2=0x7f400815b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f4005e00640 1 --2- 172.21.15.47:0/2293315882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4008158a30 0x7f400815b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f4005e00640 1 -- 172.21.15.47:0/2293315882 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400000b6c0 con 0x7f400806df30 2024-02-19T15:26:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f3ffd400640 1 --2- 172.21.15.47:0/2293315882 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4008071dc0 0x7f4008148950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:26:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f4005e00640 1 --2- 172.21.15.47:0/2293315882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f400806df30 0x7f4008158f60 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f3ff800efc0 tx=0x7f3ff800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f3ffea00640 1 -- 172.21.15.47:0/2293315882 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8014480 con 0x7f400806df30 2024-02-19T15:26:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f3ffea00640 1 -- 172.21.15.47:0/2293315882 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff800f040 con 0x7f400806df30 2024-02-19T15:26:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f400d000640 1 -- 172.21.15.47:0/2293315882 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4008148e90 con 0x7f400806df30 2024-02-19T15:26:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f3ffea00640 1 -- 172.21.15.47:0/2293315882 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8014800 con 0x7f400806df30 2024-02-19T15:26:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.727+0000 7f400d000640 1 -- 172.21.15.47:0/2293315882 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f400815c750 con 0x7f400806df30 2024-02-19T15:26:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.731+0000 7f400d000640 1 -- 172.21.15.47:0/2293315882 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4008068120 con 0x7f400806df30 2024-02-19T15:26:52.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.731+0000 7f3ffea00640 1 -- 172.21.15.47:0/2293315882 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3ff8014960 con 0x7f400806df30 2024-02-19T15:26:52.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.731+0000 7f3ffea00640 1 --2- 172.21.15.47:0/2293315882 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3fe4066e40 0x7f3fe4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:52.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.731+0000 7f3ffea00640 1 -- 172.21.15.47:0/2293315882 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f3ff8085970 con 0x7f400806df30 2024-02-19T15:26:52.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.731+0000 7f3ffd400640 1 --2- 172.21.15.47:0/2293315882 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3fe4066e40 0x7f3fe4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:52.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.731+0000 7f3ffd400640 1 --2- 172.21.15.47:0/2293315882 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3fe4066e40 0x7f3fe4069300 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f40081f5e30 tx=0x7f3ff0009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:52.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.735+0000 7f3ffea00640 1 -- 172.21.15.47:0/2293315882 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f3ff80526c0 con 0x7f400806df30 2024-02-19T15:26:52.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.955+0000 7f400d000640 1 -- 172.21.15.47:0/2293315882 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f400815cda0 con 0x7f400806df30 2024-02-19T15:26:52.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.959+0000 7f3ffea00640 1 -- 172.21.15.47:0/2293315882 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4947 (secure 0 0 0) 0x7f3ff8085070 con 0x7f400806df30 2024-02-19T15:26:52.963 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:52.963 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":66,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7060,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/4006350936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":4006350936},{"type":"v1","addr":"172.21.15.116:6837","nonce":4006350936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/2191173523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2191173523},{"type":"v1","addr":"172.21.15.47:6834","nonce":2191173523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/1012301665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":1012301665},{"type":"v1","addr":"172.21.15.47:6835","nonce":1012301665}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:26:26.721109+0000","modified":"2024-02-19T15:26:43.866082+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7054":{"gid":7054,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6835/3852677195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3852677195},{"type":"v1","addr":"172.21.15.116:6835","nonce":3852677195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T15:26:52.963 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 66 2024-02-19T15:26:52.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.959+0000 7f400d000640 1 -- 172.21.15.47:0/2293315882 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3fe4066e40 msgr2=0x7f3fe4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:52.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.959+0000 7f400d000640 1 --2- 172.21.15.47:0/2293315882 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3fe4066e40 0x7f3fe4069300 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f40081f5e30 tx=0x7f3ff0009240 comp rx=0 tx=0).stop 2024-02-19T15:26:52.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.959+0000 7f400d000640 1 -- 172.21.15.47:0/2293315882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f400806df30 msgr2=0x7f4008158f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:52.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.959+0000 7f400d000640 1 --2- 172.21.15.47:0/2293315882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f400806df30 0x7f4008158f60 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f3ff800efc0 tx=0x7f3ff800c490 comp rx=0 tx=0).stop 2024-02-19T15:26:52.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.959+0000 7f400d000640 1 -- 172.21.15.47:0/2293315882 shutdown_connections 2024-02-19T15:26:52.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.959+0000 7f400d000640 1 --2- 172.21.15.47:0/2293315882 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3fe4066e40 0x7f3fe4069300 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:52.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.959+0000 7f400d000640 1 --2- 172.21.15.47:0/2293315882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4008158a30 0x7f400815b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:52.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.959+0000 7f400d000640 1 --2- 172.21.15.47:0/2293315882 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4008071dc0 0x7f4008148950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:52.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.959+0000 7f400d000640 1 --2- 172.21.15.47:0/2293315882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f400806df30 0x7f4008158f60 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:52.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.959+0000 7f400d000640 1 -- 172.21.15.47:0/2293315882 >> 172.21.15.47:0/2293315882 conn(0x7f40080b1b80 msgr2=0x7f40080b13a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:52.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.959+0000 7f400d000640 1 -- 172.21.15.47:0/2293315882 shutdown_connections 2024-02-19T15:26:52.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:52.959+0000 7f400d000640 1 -- 172.21.15.47:0/2293315882 wait complete. 2024-02-19T15:26:52.995 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump 2024-02-19T15:26:53.234 INFO:teuthology.orchestra.run.smithi116.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":181,"msgr_send_messages":184,"msgr_recv_bytes":52668,"msgr_send_bytes":7752309,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.059779038,"msgr_running_send_time":0.037251234,"msgr_running_recv_time":0.009596076,"msgr_running_fast_dispatch_time":0.009592651,"msgr_send_messages_queue_lat":{"avgcount":183,"sum":0.021957305,"avgtime":0.000119985},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000170,"avgtime":0.000000085},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":92624},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9153,"msgr_send_messages":17097,"msgr_recv_bytes":3083839,"msgr_send_bytes":8655415,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.683092931,"msgr_running_send_time":0.417128473,"msgr_running_recv_time":0.680757225,"msgr_running_fast_dispatch_time":0.004073726,"msgr_send_messages_queue_lat":{"avgcount":17097,"sum":0.350603741,"avgtime":0.000020506},"msgr_handle_ack_lat":{"avgcount":13802,"sum":0.002510458,"avgtime":0.000000181},"msgr_recv_encrypted_bytes":104112,"msgr_send_encrypted_bytes":6688},"AsyncMessenger::Worker-2":{"msgr_recv_messages":242,"msgr_send_messages":245,"msgr_recv_bytes":71212,"msgr_send_bytes":1495905,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.035459362,"msgr_running_send_time":0.017316182,"msgr_running_recv_time":0.009185441,"msgr_running_fast_dispatch_time":0.007033805,"msgr_send_messages_queue_lat":{"avgcount":244,"sum":0.024187878,"avgtime":0.000099130},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":77712},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":78,"sum":3.599260129,"avgtime":0.046144360}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":842,"sum":0.286330264,"avgtime":0.000340059}},"mds":{"request":5324,"reply":5321,"reply_latency":{"avgcount":5321,"sum":384.394214951,"avgtime":0.072240972},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":5123,"inodes_top":0,"inodes_bottom":5123,"inodes_pin_tail":0,"inodes_pinned":5123,"inodes_expired":0,"inodes_with_caps":2725,"caps":2725,"subtrees":2,"traverse":5325,"traverse_hit":5325,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2138,"root_rbytes":16566272,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":1,"ceph_cap_op_grant":4333,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3601,"handle_client_caps_dirty":2999,"handle_client_cap_release":48,"process_request_cap_release":2121},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1635,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":2112,"strays_enqueued":477,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":9556,"evex":0,"evtrm":0,"ev":9559,"evexg":0,"evexd":0,"segadd":12,"segex":0,"segtrm":0,"seg":14,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":24764802,"rdpos":4195244,"jlat":{"avgcount":13,"sum":0.661394010,"avgtime":0.050876462},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":5323,"dn+":5323,"dn-":0,"cap":2725,"cap+":3202,"cap-":477,"rss":103980,"heap":248112},"mds_server":{"dispatch_client_request":5324,"dispatch_server_request":0,"handle_client_request":5324,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":381.004064656,"avgtime":0.119063770},"req_getattr_latency":{"avgcount":2,"sum":0.167551542,"avgtime":0.083775771},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003020866,"avgtime":0.003020866},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.395814412,"avgtime":0.065969068},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2112,"sum":2.823763475,"avgtime":0.001337009},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5068,"average_load":5068,"avg_session_uptime":990,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":5016679,"buffer_anon_items":1501,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":11451917,"mds_co_items":83510,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":565,"op_send_bytes":17161771,"op_resend":0,"op_reply":565,"op_latency":{"avgcount":565,"sum":1.379986773,"avgtime":0.002442454},"op_inflight":0,"oplen_avg":{"avgcount":565,"sum":620},"op":565,"op_r":42,"op_w":523,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":14,"osdop_write":19,"osdop_writefull":4,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":492,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":14,"omap_rd":48,"omap_del":2},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":477,"pq_executed":477,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":260,"max":104857600,"get_started":0,"get":9571,"get_sum":2487256,"get_or_fail_fail":0,"get_or_fail_success":9571,"take":0,"take_sum":0,"put":9570,"put_sum":2486996,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":565,"take_sum":17210882,"put":565,"put_sum":17210882,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":565,"take_sum":565,"put":565,"put_sum":565,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":477,"get_sum":48177,"get_or_fail_fail":0,"get_or_fail_success":477,"take":0,"take_sum":0,"put":6,"put_sum":48177,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55fa43f90dc0":{"val":3793282,"max":3758096384,"get_started":0,"get":9556,"get_sum":20569300,"get_or_fail_fail":0,"get_or_fail_success":9556,"take":0,"take_sum":0,"put":13,"put_sum":16776018,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-02-19T15:26:53.235 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 8.5e-08, "sum": 1.7e-07 }, "msgr_recv_bytes": 52668, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 181, "msgr_running_fast_dispatch_time": 0.009592651, "msgr_running_recv_time": 0.009596076, "msgr_running_send_time": 0.037251234, "msgr_running_total_time": 0.059779038, "msgr_send_bytes": 7752309, "msgr_send_encrypted_bytes": 92624, "msgr_send_messages": 184, "msgr_send_messages_queue_lat": { "avgcount": 183, "avgtime": 0.000119985, "sum": 0.021957305 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 13802, "avgtime": 1.81e-07, "sum": 0.002510458 }, "msgr_recv_bytes": 3083839, "msgr_recv_encrypted_bytes": 104112, "msgr_recv_messages": 9153, "msgr_running_fast_dispatch_time": 0.004073726, "msgr_running_recv_time": 0.680757225, "msgr_running_send_time": 0.417128473, "msgr_running_total_time": 0.683092931, "msgr_send_bytes": 8655415, "msgr_send_encrypted_bytes": 6688, "msgr_send_messages": 17097, "msgr_send_messages_queue_lat": { "avgcount": 17097, "avgtime": 2.0506e-05, "sum": 0.350603741 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 71212, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 242, "msgr_running_fast_dispatch_time": 0.007033805, "msgr_running_recv_time": 0.009185441, "msgr_running_send_time": 0.017316182, "msgr_running_total_time": 0.035459362, "msgr_send_bytes": 1495905, "msgr_send_encrypted_bytes": 77712, "msgr_send_messages": 245, "msgr_send_messages_queue_lat": { "avgcount": 244, "avgtime": 9.913e-05, "sum": 0.024187878 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 78, "avgtime": 0.04614436, "sum": 3.599260129 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 842, "avgtime": 0.000340059, "sum": 0.286330264 }, "queue_len": 0 }, "mds": { "caps": 2725, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4333, "ceph_cap_op_revoke": 1, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 48, "handle_client_caps": 3601, "handle_client_caps_dirty": 2999, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5123, "inodes_bottom": 5123, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5123, "inodes_top": 0, "inodes_with_caps": 2725, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 2121, "q": 0, "reply": 5321, "reply_latency": { "avgcount": 5321, "avgtime": 0.072240972, "sum": 384.394214951 }, "request": 5324, "root_rbytes": 16566272, "root_rfiles": 2138, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5325, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5325, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1635, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2112, "strays_enqueued": 477, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 9559, "evadd": 9556, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 13, "avgtime": 0.050876462, "sum": 0.66139401 }, "rdpos": 4195244, "replayed": 3, "seg": 14, "segadd": 12, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 24764802 }, "mds_mem": { "cap": 2725, "cap+": 3202, "cap-": 477, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5323, "dn+": 5323, "dn-": 0, "heap": 248112, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 103980 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5324, "dispatch_server_request": 0, "handle_client_request": 5324, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.11906377, "sum": 381.004064656 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.083775771, "sum": 0.167551542 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003020866, "sum": 0.003020866 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.065969068, "sum": 0.395814412 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2112, "avgtime": 0.001337009, "sum": 2.823763475 } }, "mds_sessions": { "average_load": 5068, "avg_session_uptime": 990, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5068 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 5016679, "buffer_anon_items": 1501, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 11451917, "mds_co_items": 83510, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 2, "omap_rd": 48, "omap_wr": 14, "op": 565, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 565, "avgtime": 0.002442454, "sum": 1.379986773 }, "op_pg": 0, "op_r": 42, "op_reply": 565, "op_resend": 0, "op_rmw": 0, "op_send": 565, "op_send_bytes": 17161771, "op_w": 523, "oplen_avg": { "avgcount": 565, "sum": 620 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 492, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 14, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 19, "osdop_writefull": 4, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 477, "pq_executed_ops": 477, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 9571, "get_or_fail_fail": 0, "get_or_fail_success": 9571, "get_started": 0, "get_sum": 2487256, "max": 104857600, "put": 9570, "put_sum": 2486996, "take": 0, "take_sum": 0, "val": 260, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 565, "put_sum": 17210882, "take": 565, "take_sum": 17210882, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 565, "put_sum": 565, "take": 565, "take_sum": 565, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 477, "get_or_fail_fail": 0, "get_or_fail_success": 477, "get_started": 0, "get_sum": 48177, "max": 3758096384, "put": 6, "put_sum": 48177, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55fa43f90dc0": { "get": 9556, "get_or_fail_fail": 0, "get_or_fail_success": 9556, "get_started": 0, "get_sum": 20569300, "max": 3758096384, "put": 13, "put_sum": 16776018, "take": 0, "take_sum": 0, "val": 3793282, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-02-19T15:26:53.235 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1635, 477/2112 2024-02-19T15:26:54.236 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:26:54.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.375+0000 7fb83e800640 1 Processor -- start 2024-02-19T15:26:54.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.375+0000 7fb83e800640 1 -- start start 2024-02-19T15:26:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.375+0000 7fb83e800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb8380ce4b0 0x7fb8380d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.375+0000 7fb83e800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb8380b2e90 0x7fb8380d2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.375+0000 7fb83e800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8380d06d0 con 0x7fb83815bbf0 2024-02-19T15:26:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.375+0000 7fb83e800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8380d2570 con 0x7fb8380ce4b0 2024-02-19T15:26:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.375+0000 7fb83e800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8380ce970 con 0x7fb8380b2e90 2024-02-19T15:26:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.379+0000 7fb837400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb83815bbf0 0x7fb8381603f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53032/0 (socket says 172.21.15.47:53032) 2024-02-19T15:26:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.379+0000 7fb837400640 1 -- 172.21.15.47:0/2834550090 learned_addr learned my addr 172.21.15.47:0/2834550090 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.379+0000 7fb837e00640 1 --2- 172.21.15.47:0/2834550090 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb8380b2e90 0x7fb8380d2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.379+0000 7fb82fe00640 1 --2- 172.21.15.47:0/2834550090 >> v2:172.21.15.116:3300/0 conn(0x7fb8380ce4b0 0x7fb8380d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.379+0000 7fb836a00640 1 -- 172.21.15.47:0/2834550090 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239719490 0 0) 0x7fb8380d06d0 con 0x7fb83815bbf0 2024-02-19T15:26:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.379+0000 7fb836a00640 1 -- 172.21.15.47:0/2834550090 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb81c003530 con 0x7fb83815bbf0 2024-02-19T15:26:54.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.379+0000 7fb837e00640 1 -- 172.21.15.47:0/2834550090 >> v2:172.21.15.116:3300/0 conn(0x7fb8380ce4b0 msgr2=0x7fb8380d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:54.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.379+0000 7fb837e00640 1 --2- 172.21.15.47:0/2834550090 >> v2:172.21.15.116:3300/0 conn(0x7fb8380ce4b0 0x7fb8380d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:54.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.379+0000 7fb837e00640 1 -- 172.21.15.47:0/2834550090 >> v1:172.21.15.47:6789/0 conn(0x7fb83815bbf0 legacy=0x7fb8381603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:54.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.379+0000 7fb837e00640 1 -- 172.21.15.47:0/2834550090 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8380cec50 con 0x7fb8380b2e90 2024-02-19T15:26:54.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.379+0000 7fb837e00640 1 --2- 172.21.15.47:0/2834550090 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb8380b2e90 0x7fb8380d2030 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb830009d90 tx=0x7fb83001c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c22bfe3aa3eeb7c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:54.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb836a00640 1 -- 172.21.15.47:0/2834550090 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8300a5a00 con 0x7fb8380b2e90 2024-02-19T15:26:54.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb836a00640 1 -- 172.21.15.47:0/2834550090 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8300a5b60 con 0x7fb8380b2e90 2024-02-19T15:26:54.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb836a00640 1 -- 172.21.15.47:0/2834550090 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8300ad440 con 0x7fb8380b2e90 2024-02-19T15:26:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb83e800640 1 -- 172.21.15.47:0/2834550090 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb8380b2e90 msgr2=0x7fb8380d2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb83e800640 1 --2- 172.21.15.47:0/2834550090 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb8380b2e90 0x7fb8380d2030 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb830009d90 tx=0x7fb83001c940 comp rx=0 tx=0).stop 2024-02-19T15:26:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb83e800640 1 -- 172.21.15.47:0/2834550090 shutdown_connections 2024-02-19T15:26:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb83e800640 1 --2- 172.21.15.47:0/2834550090 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb8380b2e90 0x7fb8380d2030 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb83e800640 1 --2- 172.21.15.47:0/2834550090 >> v2:172.21.15.116:3300/0 conn(0x7fb8380ce4b0 0x7fb8380d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb83e800640 1 -- 172.21.15.47:0/2834550090 >> 172.21.15.47:0/2834550090 conn(0x7fb83809aa70 msgr2=0x7fb83809ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:54.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb83e800640 1 -- 172.21.15.47:0/2834550090 shutdown_connections 2024-02-19T15:26:54.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb83e800640 1 -- 172.21.15.47:0/2834550090 wait complete. 2024-02-19T15:26:54.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb83e800640 1 Processor -- start 2024-02-19T15:26:54.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb83e800640 1 -- start start 2024-02-19T15:26:54.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb83e800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8380b2e90 0x7fb83815b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:54.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb83e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8380ce4b0 0x7fb8380c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:54.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb83e800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb83815bbf0 0x7fb8380c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:54.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb83e800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb838159d10 con 0x7fb8380b2e90 2024-02-19T15:26:54.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb83e800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb838159ff0 con 0x7fb8380ce4b0 2024-02-19T15:26:54.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb83e800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83815a2d0 con 0x7fb83815bbf0 2024-02-19T15:26:54.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb837400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8380b2e90 0x7fb83815b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:54.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb837400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8380b2e90 0x7fb83815b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:38482/0 (socket says 172.21.15.47:38482) 2024-02-19T15:26:54.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb837400640 1 -- 172.21.15.47:0/3791585249 learned_addr learned my addr 172.21.15.47:0/3791585249 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:54.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb82fe00640 1 --2- 172.21.15.47:0/3791585249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8380ce4b0 0x7fb8380c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:54.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.383+0000 7fb837e00640 1 --2- 172.21.15.47:0/3791585249 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb83815bbf0 0x7fb8380c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:54.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.387+0000 7fb837400640 1 -- 172.21.15.47:0/3791585249 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb83815bbf0 msgr2=0x7fb8380c93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:54.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.387+0000 7fb837400640 1 --2- 172.21.15.47:0/3791585249 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb83815bbf0 0x7fb8380c93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:54.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.387+0000 7fb837400640 1 -- 172.21.15.47:0/3791585249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8380ce4b0 msgr2=0x7fb8380c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:54.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.387+0000 7fb837400640 1 --2- 172.21.15.47:0/3791585249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8380ce4b0 0x7fb8380c8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:54.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.387+0000 7fb837400640 1 -- 172.21.15.47:0/3791585249 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb830009a70 con 0x7fb8380b2e90 2024-02-19T15:26:54.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.387+0000 7fb837400640 1 --2- 172.21.15.47:0/3791585249 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8380b2e90 0x7fb83815b090 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fb82800c3c0 tx=0x7fb82800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:54.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.387+0000 7fb82f400640 1 -- 172.21.15.47:0/3791585249 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb828005cf0 con 0x7fb8380b2e90 2024-02-19T15:26:54.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.387+0000 7fb82f400640 1 -- 172.21.15.47:0/3791585249 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb828005e50 con 0x7fb8380b2e90 2024-02-19T15:26:54.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.387+0000 7fb82f400640 1 -- 172.21.15.47:0/3791585249 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8280072f0 con 0x7fb8380b2e90 2024-02-19T15:26:54.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.387+0000 7fb83e800640 1 -- 172.21.15.47:0/3791585249 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb83815a5b0 con 0x7fb8380b2e90 2024-02-19T15:26:54.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.387+0000 7fb83e800640 1 -- 172.21.15.47:0/3791585249 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb83815ab70 con 0x7fb8380b2e90 2024-02-19T15:26:54.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.387+0000 7fb83e800640 1 -- 172.21.15.47:0/3791585249 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8380b3450 con 0x7fb8380b2e90 2024-02-19T15:26:54.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.387+0000 7fb82f400640 1 -- 172.21.15.47:0/3791585249 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb828007450 con 0x7fb8380b2e90 2024-02-19T15:26:54.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.387+0000 7fb82f400640 1 --2- 172.21.15.47:0/3791585249 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb814066e40 0x7fb814069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:54.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.387+0000 7fb82f400640 1 -- 172.21.15.47:0/3791585249 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fb828084190 con 0x7fb8380b2e90 2024-02-19T15:26:54.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.391+0000 7fb82fe00640 1 --2- 172.21.15.47:0/3791585249 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb814066e40 0x7fb814069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:54.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.391+0000 7fb82fe00640 1 --2- 172.21.15.47:0/3791585249 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb814066e40 0x7fb814069300 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb838159c40 tx=0x7fb820009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:54.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.391+0000 7fb82f400640 1 -- 172.21.15.47:0/3791585249 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb828050e50 con 0x7fb8380b2e90 2024-02-19T15:26:54.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.587+0000 7fb83e800640 1 -- 172.21.15.47:0/3791585249 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb838068120 con 0x7fb8380b2e90 2024-02-19T15:26:54.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.587+0000 7fb82f400640 1 -- 172.21.15.47:0/3791585249 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4947 (secure 0 0 0) 0x7fb828054510 con 0x7fb8380b2e90 2024-02-19T15:26:54.594 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:54.594 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":66,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7060,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/4006350936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":4006350936},{"type":"v1","addr":"172.21.15.116:6837","nonce":4006350936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/2191173523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2191173523},{"type":"v1","addr":"172.21.15.47:6834","nonce":2191173523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/1012301665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":1012301665},{"type":"v1","addr":"172.21.15.47:6835","nonce":1012301665}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:26:26.721109+0000","modified":"2024-02-19T15:26:43.866082+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7054":{"gid":7054,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6835/3852677195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3852677195},{"type":"v1","addr":"172.21.15.116:6835","nonce":3852677195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T15:26:54.594 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 66 2024-02-19T15:26:54.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.591+0000 7fb83e800640 1 -- 172.21.15.47:0/3791585249 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb814066e40 msgr2=0x7fb814069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:54.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.591+0000 7fb83e800640 1 --2- 172.21.15.47:0/3791585249 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb814066e40 0x7fb814069300 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb838159c40 tx=0x7fb820009190 comp rx=0 tx=0).stop 2024-02-19T15:26:54.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.591+0000 7fb83e800640 1 -- 172.21.15.47:0/3791585249 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8380b2e90 msgr2=0x7fb83815b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:54.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.591+0000 7fb83e800640 1 --2- 172.21.15.47:0/3791585249 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8380b2e90 0x7fb83815b090 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fb82800c3c0 tx=0x7fb82800c880 comp rx=0 tx=0).stop 2024-02-19T15:26:54.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.591+0000 7fb83e800640 1 -- 172.21.15.47:0/3791585249 shutdown_connections 2024-02-19T15:26:54.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.591+0000 7fb83e800640 1 --2- 172.21.15.47:0/3791585249 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb814066e40 0x7fb814069300 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:54.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.591+0000 7fb83e800640 1 --2- 172.21.15.47:0/3791585249 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb83815bbf0 0x7fb8380c93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:54.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.591+0000 7fb83e800640 1 --2- 172.21.15.47:0/3791585249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8380ce4b0 0x7fb8380c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:54.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.591+0000 7fb83e800640 1 --2- 172.21.15.47:0/3791585249 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8380b2e90 0x7fb83815b090 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:54.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.591+0000 7fb83e800640 1 -- 172.21.15.47:0/3791585249 >> 172.21.15.47:0/3791585249 conn(0x7fb83809aa70 msgr2=0x7fb8380cc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:54.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.591+0000 7fb83e800640 1 -- 172.21.15.47:0/3791585249 shutdown_connections 2024-02-19T15:26:54.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:54.591+0000 7fb83e800640 1 -- 172.21.15.47:0/3791585249 wait complete. 2024-02-19T15:26:54.615 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump 2024-02-19T15:26:54.833 INFO:teuthology.orchestra.run.smithi116.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":222,"msgr_send_messages":225,"msgr_recv_bytes":63154,"msgr_send_bytes":9527188,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.078088205,"msgr_running_send_time":0.051526959,"msgr_running_recv_time":0.010476449,"msgr_running_fast_dispatch_time":0.012500754,"msgr_send_messages_queue_lat":{"avgcount":224,"sum":0.025084551,"avgtime":0.000111984},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000170,"avgtime":0.000000085},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":92624},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10205,"msgr_send_messages":19918,"msgr_recv_bytes":3456399,"msgr_send_bytes":11058225,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.782151804,"msgr_running_send_time":0.482533871,"msgr_running_recv_time":0.715313427,"msgr_running_fast_dispatch_time":0.006302959,"msgr_send_messages_queue_lat":{"avgcount":19918,"sum":0.427219134,"avgtime":0.000021448},"msgr_handle_ack_lat":{"avgcount":15657,"sum":0.002841974,"avgtime":0.000000181},"msgr_recv_encrypted_bytes":104112,"msgr_send_encrypted_bytes":6688},"AsyncMessenger::Worker-2":{"msgr_recv_messages":328,"msgr_send_messages":331,"msgr_recv_bytes":93048,"msgr_send_bytes":1523331,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.042482907,"msgr_running_send_time":0.019933291,"msgr_running_recv_time":0.010990595,"msgr_running_fast_dispatch_time":0.009089303,"msgr_send_messages_queue_lat":{"avgcount":330,"sum":0.029427688,"avgtime":0.000089174},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":77712},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1824,"complete_latency":{"avgcount":80,"sum":3.599749431,"avgtime":0.044996867}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":1154,"sum":0.506953115,"avgtime":0.000439300}},"mds":{"request":6314,"reply":6312,"reply_latency":{"avgcount":6312,"sum":385.651451046,"avgtime":0.061098138},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":5837,"inodes_top":0,"inodes_bottom":5837,"inodes_pin_tail":0,"inodes_pinned":5837,"inodes_expired":0,"inodes_with_caps":2379,"caps":2379,"subtrees":2,"traverse":6315,"traverse_hit":6315,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1502,"root_rbytes":11617280,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":1,"ceph_cap_op_grant":5219,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3602,"handle_client_caps_dirty":3000,"handle_client_cap_release":51,"process_request_cap_release":3111},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":2456,"num_strays_delayed":176,"num_strays_enqueuing":0,"strays_created":3103,"strays_enqueued":647,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":10880,"evex":0,"evtrm":0,"ev":10883,"evexg":0,"evexd":0,"segadd":14,"segex":0,"segtrm":0,"seg":16,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":28231545,"rdpos":4195244,"jlat":{"avgcount":14,"sum":0.696575162,"avgtime":0.049755368},"replayed":3},"mds_mem":{"ino":2737,"ino+":3214,"ino-":477,"dir":13,"dir+":13,"dir-":0,"dn":5837,"dn+":6314,"dn-":477,"cap":2379,"cap+":3202,"cap-":823,"rss":114988,"heap":248112},"mds_server":{"dispatch_client_request":6314,"dispatch_server_request":0,"handle_client_request":6314,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":381.004064656,"avgtime":0.119063770},"req_getattr_latency":{"avgcount":2,"sum":0.167551542,"avgtime":0.083775771},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003020866,"avgtime":0.003020866},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.395814412,"avgtime":0.065969068},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3103,"sum":4.080999570,"avgtime":0.001315178},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5956,"average_load":5956,"avg_session_uptime":990,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":5016679,"buffer_anon_items":1501,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":11451917,"mds_co_items":83510,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":746,"op_send_bytes":22874176,"op_resend":0,"op_reply":746,"op_latency":{"avgcount":746,"sum":1.775982979,"avgtime":0.002380674},"op_inflight":0,"oplen_avg":{"avgcount":746,"sum":803},"op":746,"op_r":45,"op_w":701,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":17,"osdop_write":24,"osdop_writefull":5,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":663,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":16,"omap_rd":48,"omap_del":3},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":647,"pq_executed":647,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":261,"max":104857600,"get_started":0,"get":10750,"get_sum":2801996,"get_or_fail_fail":0,"get_or_fail_success":10750,"take":0,"take_sum":0,"put":10749,"put_sum":2801735,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":746,"take_sum":22940457,"put":746,"put_sum":22940457,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":746,"take_sum":746,"put":746,"put_sum":746,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":647,"get_sum":65347,"get_or_fail_fail":0,"get_or_fail_success":647,"take":0,"take_sum":0,"put":9,"put_sum":65347,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55fa43f90dc0":{"val":1618287,"max":3758096384,"get_started":0,"get":10880,"get_sum":24036043,"get_or_fail_fail":0,"get_or_fail_success":10880,"take":0,"take_sum":0,"put":15,"put_sum":22417756,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-02-19T15:26:54.834 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 8.5e-08, "sum": 1.7e-07 }, "msgr_recv_bytes": 63154, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 222, "msgr_running_fast_dispatch_time": 0.012500754, "msgr_running_recv_time": 0.010476449, "msgr_running_send_time": 0.051526959, "msgr_running_total_time": 0.078088205, "msgr_send_bytes": 9527188, "msgr_send_encrypted_bytes": 92624, "msgr_send_messages": 225, "msgr_send_messages_queue_lat": { "avgcount": 224, "avgtime": 0.000111984, "sum": 0.025084551 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 15657, "avgtime": 1.81e-07, "sum": 0.002841974 }, "msgr_recv_bytes": 3456399, "msgr_recv_encrypted_bytes": 104112, "msgr_recv_messages": 10205, "msgr_running_fast_dispatch_time": 0.006302959, "msgr_running_recv_time": 0.715313427, "msgr_running_send_time": 0.482533871, "msgr_running_total_time": 0.782151804, "msgr_send_bytes": 11058225, "msgr_send_encrypted_bytes": 6688, "msgr_send_messages": 19918, "msgr_send_messages_queue_lat": { "avgcount": 19918, "avgtime": 2.1448e-05, "sum": 0.427219134 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 93048, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 328, "msgr_running_fast_dispatch_time": 0.009089303, "msgr_running_recv_time": 0.010990595, "msgr_running_send_time": 0.019933291, "msgr_running_total_time": 0.042482907, "msgr_send_bytes": 1523331, "msgr_send_encrypted_bytes": 77712, "msgr_send_messages": 331, "msgr_send_messages_queue_lat": { "avgcount": 330, "avgtime": 8.9174e-05, "sum": 0.029427688 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 80, "avgtime": 0.044996867, "sum": 3.599749431 }, "queue_len": 1824 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 1154, "avgtime": 0.0004393, "sum": 0.506953115 }, "queue_len": 0 }, "mds": { "caps": 2379, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 5219, "ceph_cap_op_revoke": 1, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 51, "handle_client_caps": 3602, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5837, "inodes_bottom": 5837, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5837, "inodes_top": 0, "inodes_with_caps": 2379, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3111, "q": 0, "reply": 6312, "reply_latency": { "avgcount": 6312, "avgtime": 0.061098138, "sum": 385.651451046 }, "request": 6314, "root_rbytes": 11617280, "root_rfiles": 1502, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6315, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6315, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 2456, "num_strays_delayed": 176, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3103, "strays_enqueued": 647, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 10883, "evadd": 10880, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 14, "avgtime": 0.049755368, "sum": 0.696575162 }, "rdpos": 4195244, "replayed": 3, "seg": 16, "segadd": 14, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 28231545 }, "mds_mem": { "cap": 2379, "cap+": 3202, "cap-": 823, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5837, "dn+": 6314, "dn-": 477, "heap": 248112, "ino": 2737, "ino+": 3214, "ino-": 477, "rss": 114988 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6314, "dispatch_server_request": 0, "handle_client_request": 6314, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.11906377, "sum": 381.004064656 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.083775771, "sum": 0.167551542 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003020866, "sum": 0.003020866 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.065969068, "sum": 0.395814412 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3103, "avgtime": 0.001315178, "sum": 4.08099957 } }, "mds_sessions": { "average_load": 5956, "avg_session_uptime": 990, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5956 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 5016679, "buffer_anon_items": 1501, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 11451917, "mds_co_items": 83510, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 3, "omap_rd": 48, "omap_wr": 16, "op": 746, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 746, "avgtime": 0.002380674, "sum": 1.775982979 }, "op_pg": 0, "op_r": 45, "op_reply": 746, "op_resend": 0, "op_rmw": 0, "op_send": 746, "op_send_bytes": 22874176, "op_w": 701, "oplen_avg": { "avgcount": 746, "sum": 803 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 663, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 17, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 24, "osdop_writefull": 5, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 647, "pq_executed_ops": 647, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 10750, "get_or_fail_fail": 0, "get_or_fail_success": 10750, "get_started": 0, "get_sum": 2801996, "max": 104857600, "put": 10749, "put_sum": 2801735, "take": 0, "take_sum": 0, "val": 261, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 746, "put_sum": 22940457, "take": 746, "take_sum": 22940457, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 746, "put_sum": 746, "take": 746, "take_sum": 746, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 647, "get_or_fail_fail": 0, "get_or_fail_success": 647, "get_started": 0, "get_sum": 65347, "max": 3758096384, "put": 9, "put_sum": 65347, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55fa43f90dc0": { "get": 10880, "get_or_fail_fail": 0, "get_or_fail_success": 10880, "get_started": 0, "get_sum": 24036043, "max": 3758096384, "put": 15, "put_sum": 22417756, "take": 0, "take_sum": 0, "val": 1618287, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-02-19T15:26:54.834 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/2456, 647/3103 2024-02-19T15:26:55.489 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:26:55.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.623+0000 7fcc43600640 1 Processor -- start 2024-02-19T15:26:55.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.623+0000 7fcc43600640 1 -- start start 2024-02-19T15:26:55.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.627+0000 7fcc43600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3c061a50 0x7fcc3c0ae2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:55.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.627+0000 7fcc43600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcc3c0aefc0 0x7fcc3c0af450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:55.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.627+0000 7fcc43600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc3c06eed0 con 0x7fcc3c15e080 2024-02-19T15:26:55.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.627+0000 7fcc43600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc3c0af990 con 0x7fcc3c0aefc0 2024-02-19T15:26:55.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.627+0000 7fcc43600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c15e460 con 0x7fcc3c061a50 2024-02-19T15:26:55.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.627+0000 7fcc40c00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fcc3c15e080 0x7fcc3c0b35a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53034/0 (socket says 172.21.15.47:53034) 2024-02-19T15:26:55.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.627+0000 7fcc40c00640 1 -- 172.21.15.47:0/2417579471 learned_addr learned my addr 172.21.15.47:0/2417579471 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:26:55.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.627+0000 7fcc33e00640 1 --2- 172.21.15.47:0/2417579471 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3c061a50 0x7fcc3c0ae2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:55.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.627+0000 7fcc41600640 1 --2- 172.21.15.47:0/2417579471 >> v2:172.21.15.116:3300/0 conn(0x7fcc3c0aefc0 0x7fcc3c0af450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:55.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.627+0000 7fcc33e00640 1 -- 172.21.15.47:0/2417579471 >> v2:172.21.15.116:3300/0 conn(0x7fcc3c0aefc0 msgr2=0x7fcc3c0af450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:55.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.627+0000 7fcc33e00640 1 --2- 172.21.15.47:0/2417579471 >> v2:172.21.15.116:3300/0 conn(0x7fcc3c0aefc0 0x7fcc3c0af450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:55.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.627+0000 7fcc33e00640 1 -- 172.21.15.47:0/2417579471 >> v1:172.21.15.47:6789/0 conn(0x7fcc3c15e080 legacy=0x7fcc3c0b35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:55.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.627+0000 7fcc33e00640 1 -- 172.21.15.47:0/2417579471 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3c0aebf0 con 0x7fcc3c061a50 2024-02-19T15:26:55.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.627+0000 7fcc3be00640 1 -- 172.21.15.47:0/2417579471 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183795382 0 0) 0x7fcc3c06eed0 con 0x7fcc3c15e080 2024-02-19T15:26:55.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc33e00640 1 --2- 172.21.15.47:0/2417579471 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3c061a50 0x7fcc3c0ae2f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcc240097c0 tx=0x7fcc2401c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d1e390ae1ee844b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:55.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc3be00640 1 -- 172.21.15.47:0/2417579471 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc240ab670 con 0x7fcc3c061a50 2024-02-19T15:26:55.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc3be00640 1 -- 172.21.15.47:0/2417579471 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc240ab7d0 con 0x7fcc3c061a50 2024-02-19T15:26:55.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc3be00640 1 -- 172.21.15.47:0/2417579471 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc240abb60 con 0x7fcc3c061a50 2024-02-19T15:26:55.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc43600640 1 -- 172.21.15.47:0/2417579471 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3c061a50 msgr2=0x7fcc3c0ae2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:55.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc43600640 1 --2- 172.21.15.47:0/2417579471 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3c061a50 0x7fcc3c0ae2f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcc240097c0 tx=0x7fcc2401c470 comp rx=0 tx=0).stop 2024-02-19T15:26:55.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc43600640 1 -- 172.21.15.47:0/2417579471 shutdown_connections 2024-02-19T15:26:55.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc43600640 1 --2- 172.21.15.47:0/2417579471 >> v2:172.21.15.116:3300/0 conn(0x7fcc3c0aefc0 0x7fcc3c0af450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:55.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc43600640 1 --2- 172.21.15.47:0/2417579471 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3c061a50 0x7fcc3c0ae2f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:55.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc43600640 1 -- 172.21.15.47:0/2417579471 >> 172.21.15.47:0/2417579471 conn(0x7fcc3c072470 msgr2=0x7fcc3c06af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:55.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc43600640 1 -- 172.21.15.47:0/2417579471 shutdown_connections 2024-02-19T15:26:55.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc43600640 1 -- 172.21.15.47:0/2417579471 wait complete. 2024-02-19T15:26:55.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc43600640 1 Processor -- start 2024-02-19T15:26:55.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc43600640 1 -- start start 2024-02-19T15:26:55.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc43600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3c061a50 0x7fcc3c0b0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:55.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc43600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc3c0aefc0 0x7fcc3c159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:55.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc43600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc3c15e080 0x7fcc3c14c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:55.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc43600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c0b20b0 con 0x7fcc3c15e080 2024-02-19T15:26:55.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc43600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c0b1c20 con 0x7fcc3c0aefc0 2024-02-19T15:26:55.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc43600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c0b1010 con 0x7fcc3c061a50 2024-02-19T15:26:55.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc40c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3c061a50 0x7fcc3c0b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:55.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.631+0000 7fcc41600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc3c15e080 0x7fcc3c14c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:55.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.635+0000 7fcc40c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3c061a50 0x7fcc3c0b0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:46792/0 (socket says 172.21.15.47:46792) 2024-02-19T15:26:55.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.635+0000 7fcc40c00640 1 -- 172.21.15.47:0/1777720755 learned_addr learned my addr 172.21.15.47:0/1777720755 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:55.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.635+0000 7fcc33e00640 1 --2- 172.21.15.47:0/1777720755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc3c0aefc0 0x7fcc3c159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:55.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.635+0000 7fcc40c00640 1 -- 172.21.15.47:0/1777720755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc3c0aefc0 msgr2=0x7fcc3c159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:55.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.635+0000 7fcc40c00640 1 --2- 172.21.15.47:0/1777720755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc3c0aefc0 0x7fcc3c159880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:55.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.635+0000 7fcc40c00640 1 -- 172.21.15.47:0/1777720755 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc3c15e080 msgr2=0x7fcc3c14c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:55.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.635+0000 7fcc40c00640 1 --2- 172.21.15.47:0/1777720755 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc3c15e080 0x7fcc3c14c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:55.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.635+0000 7fcc40c00640 1 -- 172.21.15.47:0/1777720755 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc24009420 con 0x7fcc3c061a50 2024-02-19T15:26:55.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.635+0000 7fcc40c00640 1 --2- 172.21.15.47:0/1777720755 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3c061a50 0x7fcc3c0b0710 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c00c4b0 tx=0x7fcc2c00c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:55.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.635+0000 7fcc39600640 1 -- 172.21.15.47:0/1777720755 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c0074e0 con 0x7fcc3c061a50 2024-02-19T15:26:55.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.635+0000 7fcc39600640 1 -- 172.21.15.47:0/1777720755 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2c007660 con 0x7fcc3c061a50 2024-02-19T15:26:55.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.635+0000 7fcc39600640 1 -- 172.21.15.47:0/1777720755 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c004a30 con 0x7fcc3c061a50 2024-02-19T15:26:55.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.635+0000 7fcc43600640 1 -- 172.21.15.47:0/1777720755 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3c0b12f0 con 0x7fcc3c061a50 2024-02-19T15:26:55.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.635+0000 7fcc43600640 1 -- 172.21.15.47:0/1777720755 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3c0b18b0 con 0x7fcc3c061a50 2024-02-19T15:26:55.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.635+0000 7fcc43600640 1 -- 172.21.15.47:0/1777720755 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc3c068120 con 0x7fcc3c061a50 2024-02-19T15:26:55.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.635+0000 7fcc39600640 1 -- 172.21.15.47:0/1777720755 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fcc2c01c3b0 con 0x7fcc3c061a50 2024-02-19T15:26:55.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.639+0000 7fcc39600640 1 --2- 172.21.15.47:0/1777720755 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcc18066e70 0x7fcc18069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:55.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.639+0000 7fcc39600640 1 -- 172.21.15.47:0/1777720755 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fcc2c084270 con 0x7fcc3c061a50 2024-02-19T15:26:55.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.639+0000 7fcc33e00640 1 --2- 172.21.15.47:0/1777720755 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcc18066e70 0x7fcc18069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:55.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.639+0000 7fcc33e00640 1 --2- 172.21.15.47:0/1777720755 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcc18066e70 0x7fcc18069330 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fcc240af000 tx=0x7fcc240a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:55.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:55.647+0000 7fcc39600640 1 -- 172.21.15.47:0/1777720755 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fcc2c051040 con 0x7fcc3c061a50 2024-02-19T15:26:55.835 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:26:56.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.099+0000 7fcc43600640 1 -- 172.21.15.47:0/1777720755 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc3c062110 con 0x7fcc3c061a50 2024-02-19T15:26:56.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.103+0000 7fcc39600640 1 -- 172.21.15.47:0/1777720755 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4947 (secure 0 0 0) 0x7fcc2c054700 con 0x7fcc3c061a50 2024-02-19T15:26:56.109 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:56.110 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":66,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7060,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/4006350936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":4006350936},{"type":"v1","addr":"172.21.15.116:6837","nonce":4006350936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/2191173523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2191173523},{"type":"v1","addr":"172.21.15.47:6834","nonce":2191173523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/1012301665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":1012301665},{"type":"v1","addr":"172.21.15.47:6835","nonce":1012301665}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:26:26.721109+0000","modified":"2024-02-19T15:26:43.866082+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7054":{"gid":7054,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6835/3852677195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3852677195},{"type":"v1","addr":"172.21.15.116:6835","nonce":3852677195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T15:26:56.110 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 66 2024-02-19T15:26:56.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.107+0000 7fcc43600640 1 -- 172.21.15.47:0/1777720755 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcc18066e70 msgr2=0x7fcc18069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:56.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.107+0000 7fcc43600640 1 --2- 172.21.15.47:0/1777720755 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcc18066e70 0x7fcc18069330 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fcc240af000 tx=0x7fcc240a7000 comp rx=0 tx=0).stop 2024-02-19T15:26:56.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.107+0000 7fcc43600640 1 -- 172.21.15.47:0/1777720755 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3c061a50 msgr2=0x7fcc3c0b0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:56.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.107+0000 7fcc43600640 1 --2- 172.21.15.47:0/1777720755 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3c061a50 0x7fcc3c0b0710 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c00c4b0 tx=0x7fcc2c00c970 comp rx=0 tx=0).stop 2024-02-19T15:26:56.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.107+0000 7fcc43600640 1 -- 172.21.15.47:0/1777720755 shutdown_connections 2024-02-19T15:26:56.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.107+0000 7fcc43600640 1 --2- 172.21.15.47:0/1777720755 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcc18066e70 0x7fcc18069330 secure :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fcc240af000 tx=0x7fcc240a7000 comp rx=0 tx=0).stop 2024-02-19T15:26:56.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.107+0000 7fcc43600640 1 --2- 172.21.15.47:0/1777720755 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc3c15e080 0x7fcc3c14c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:56.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.107+0000 7fcc43600640 1 --2- 172.21.15.47:0/1777720755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc3c0aefc0 0x7fcc3c159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:56.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.107+0000 7fcc43600640 1 --2- 172.21.15.47:0/1777720755 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3c061a50 0x7fcc3c0b0710 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:56.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.107+0000 7fcc43600640 1 -- 172.21.15.47:0/1777720755 >> 172.21.15.47:0/1777720755 conn(0x7fcc3c072470 msgr2=0x7fcc3c06af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:56.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.111+0000 7fcc43600640 1 -- 172.21.15.47:0/1777720755 shutdown_connections 2024-02-19T15:26:56.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.115+0000 7fcc43600640 1 -- 172.21.15.47:0/1777720755 wait complete. 2024-02-19T15:26:56.146 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2024-02-19T15:26:56.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.175+0000 7ff038600640 1 Processor -- start 2024-02-19T15:26:56.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.175+0000 7ff038600640 1 -- start start 2024-02-19T15:26:56.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.175+0000 7ff038600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff030070170 0x7ff0300cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:56.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.175+0000 7ff038600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff030070720 0x7ff0300cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:56.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.175+0000 7ff038600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0300d0830 con 0x7ff03015f860 2024-02-19T15:26:56.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.175+0000 7ff038600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0300628b0 con 0x7ff030070170 2024-02-19T15:26:56.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.175+0000 7ff038600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0300cf7b0 con 0x7ff030070720 2024-02-19T15:26:56.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.175+0000 7ff036600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff030070720 0x7ff0300cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:56.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.175+0000 7ff035200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff030070170 0x7ff0300cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:56.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.175+0000 7ff036600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff030070720 0x7ff0300cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:46796/0 (socket says 172.21.15.47:46796) 2024-02-19T15:26:56.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.175+0000 7ff036600640 1 -- 172.21.15.47:0/1853512668 learned_addr learned my addr 172.21.15.47:0/1853512668 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:56.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.179+0000 7ff036600640 1 -- 172.21.15.47:0/1853512668 >> v2:172.21.15.116:3300/0 conn(0x7ff030070170 msgr2=0x7ff0300cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:26:56.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.179+0000 7ff036600640 1 --2- 172.21.15.47:0/1853512668 >> v2:172.21.15.116:3300/0 conn(0x7ff030070170 0x7ff0300cee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:56.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.179+0000 7ff036600640 1 -- 172.21.15.47:0/1853512668 >> v1:172.21.15.116:6789/0 conn(0x7ff03015f860 legacy=0x7ff0300ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:56.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.179+0000 7ff036600640 1 -- 172.21.15.47:0/1853512668 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0300cf4d0 con 0x7ff030070720 2024-02-19T15:26:56.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.179+0000 7ff036600640 1 --2- 172.21.15.47:0/1853512668 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff030070720 0x7ff0300cd210 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff028009e50 tx=0x7ff02801c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bea975e9d46c358 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:56.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.179+0000 7ff02fe00640 1 -- 172.21.15.47:0/1853512668 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0280a5850 con 0x7ff030070720 2024-02-19T15:26:56.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.179+0000 7ff02fe00640 1 -- 172.21.15.47:0/1853512668 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0280a59b0 con 0x7ff030070720 2024-02-19T15:26:56.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.179+0000 7ff02fe00640 1 -- 172.21.15.47:0/1853512668 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0280a5d40 con 0x7ff030070720 2024-02-19T15:26:56.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.179+0000 7ff038600640 1 -- 172.21.15.47:0/1853512668 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff030070720 msgr2=0x7ff0300cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:56.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.179+0000 7ff038600640 1 --2- 172.21.15.47:0/1853512668 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff030070720 0x7ff0300cd210 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff028009e50 tx=0x7ff02801c970 comp rx=0 tx=0).stop 2024-02-19T15:26:56.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff038600640 1 -- 172.21.15.47:0/1853512668 shutdown_connections 2024-02-19T15:26:56.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff038600640 1 --2- 172.21.15.47:0/1853512668 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff030070720 0x7ff0300cd210 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:56.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff038600640 1 --2- 172.21.15.47:0/1853512668 >> v2:172.21.15.116:3300/0 conn(0x7ff030070170 0x7ff0300cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:56.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff038600640 1 -- 172.21.15.47:0/1853512668 >> 172.21.15.47:0/1853512668 conn(0x7ff0300b38d0 msgr2=0x7ff0300b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:56.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff038600640 1 -- 172.21.15.47:0/1853512668 shutdown_connections 2024-02-19T15:26:56.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff038600640 1 -- 172.21.15.47:0/1853512668 wait complete. 2024-02-19T15:26:56.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff038600640 1 Processor -- start 2024-02-19T15:26:56.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff038600640 1 -- start start 2024-02-19T15:26:56.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff038600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff030070170 0x7ff0300b7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:56.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff038600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff030070720 0x7ff03014c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:56.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff038600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff03015f860 0x7ff0300c7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:56.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff038600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0300b63b0 con 0x7ff030070170 2024-02-19T15:26:56.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff038600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0300b75a0 con 0x7ff03015f860 2024-02-19T15:26:56.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff038600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0300b57a0 con 0x7ff030070720 2024-02-19T15:26:56.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff035c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff030070170 0x7ff0300b7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:56.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff035c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff030070170 0x7ff0300b7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:38500/0 (socket says 172.21.15.47:38500) 2024-02-19T15:26:56.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff035c00640 1 -- 172.21.15.47:0/668182705 learned_addr learned my addr 172.21.15.47:0/668182705 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:26:56.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff035200640 1 --2- 172.21.15.47:0/668182705 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff030070720 0x7ff03014c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:56.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff036600640 1 --2- 172.21.15.47:0/668182705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff03015f860 0x7ff0300c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:56.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff035c00640 1 -- 172.21.15.47:0/668182705 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff030070720 msgr2=0x7ff03014c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:56.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff035c00640 1 --2- 172.21.15.47:0/668182705 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff030070720 0x7ff03014c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:56.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff035c00640 1 -- 172.21.15.47:0/668182705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff03015f860 msgr2=0x7ff0300c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:56.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff035c00640 1 --2- 172.21.15.47:0/668182705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff03015f860 0x7ff0300c7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:56.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff035c00640 1 -- 172.21.15.47:0/668182705 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff028009b30 con 0x7ff030070170 2024-02-19T15:26:56.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff035c00640 1 --2- 172.21.15.47:0/668182705 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff030070170 0x7ff0300b7060 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7ff02400c370 tx=0x7ff02400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:56.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.183+0000 7ff02d600640 1 -- 172.21.15.47:0/668182705 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0240052d0 con 0x7ff030070170 2024-02-19T15:26:56.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.187+0000 7ff02d600640 1 -- 172.21.15.47:0/668182705 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff024005430 con 0x7ff030070170 2024-02-19T15:26:56.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.187+0000 7ff02d600640 1 -- 172.21.15.47:0/668182705 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff024007260 con 0x7ff030070170 2024-02-19T15:26:56.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.187+0000 7ff038600640 1 -- 172.21.15.47:0/668182705 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0300b5a80 con 0x7ff030070170 2024-02-19T15:26:56.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.187+0000 7ff038600640 1 -- 172.21.15.47:0/668182705 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0300b6040 con 0x7ff030070170 2024-02-19T15:26:56.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.187+0000 7ff038600640 1 -- 172.21.15.47:0/668182705 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff030068120 con 0x7ff030070170 2024-02-19T15:26:56.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.187+0000 7ff02d600640 1 -- 172.21.15.47:0/668182705 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff0240073c0 con 0x7ff030070170 2024-02-19T15:26:56.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.187+0000 7ff02d600640 1 --2- 172.21.15.47:0/668182705 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff010066e70 0x7ff010069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:26:56.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.187+0000 7ff02d600640 1 -- 172.21.15.47:0/668182705 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7ff024083da0 con 0x7ff030070170 2024-02-19T15:26:56.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.187+0000 7ff035200640 1 --2- 172.21.15.47:0/668182705 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff010066e70 0x7ff010069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:26:56.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.187+0000 7ff035200640 1 --2- 172.21.15.47:0/668182705 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff010066e70 0x7ff010069330 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff03014fe10 tx=0x7ff01c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:26:56.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.191+0000 7ff02d600640 1 -- 172.21.15.47:0/668182705 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff024050b80 con 0x7ff030070170 2024-02-19T15:26:56.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.403+0000 7ff038600640 1 -- 172.21.15.47:0/668182705 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff03015dfc0 con 0x7ff030070170 2024-02-19T15:26:56.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.403+0000 7ff02d600640 1 -- 172.21.15.47:0/668182705 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4947 (secure 0 0 0) 0x7ff024054240 con 0x7ff030070170 2024-02-19T15:26:56.409 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:26:56.409 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":66,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7060,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/4006350936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":4006350936},{"type":"v1","addr":"172.21.15.116:6837","nonce":4006350936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/2191173523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2191173523},{"type":"v1","addr":"172.21.15.47:6834","nonce":2191173523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/1012301665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":1012301665},{"type":"v1","addr":"172.21.15.47:6835","nonce":1012301665}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:26:26.721109+0000","modified":"2024-02-19T15:26:43.866082+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7054":{"gid":7054,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6835/3852677195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3852677195},{"type":"v1","addr":"172.21.15.116:6835","nonce":3852677195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T15:26:56.409 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 66 2024-02-19T15:26:56.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.407+0000 7ff038600640 1 -- 172.21.15.47:0/668182705 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff010066e70 msgr2=0x7ff010069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:56.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.407+0000 7ff038600640 1 --2- 172.21.15.47:0/668182705 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff010066e70 0x7ff010069330 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff03014fe10 tx=0x7ff01c009190 comp rx=0 tx=0).stop 2024-02-19T15:26:56.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.407+0000 7ff038600640 1 -- 172.21.15.47:0/668182705 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff030070170 msgr2=0x7ff0300b7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:26:56.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.407+0000 7ff038600640 1 --2- 172.21.15.47:0/668182705 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff030070170 0x7ff0300b7060 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7ff02400c370 tx=0x7ff02400c830 comp rx=0 tx=0).stop 2024-02-19T15:26:56.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.407+0000 7ff038600640 1 -- 172.21.15.47:0/668182705 shutdown_connections 2024-02-19T15:26:56.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.407+0000 7ff038600640 1 --2- 172.21.15.47:0/668182705 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff010066e70 0x7ff010069330 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:56.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.407+0000 7ff038600640 1 --2- 172.21.15.47:0/668182705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff03015f860 0x7ff0300c7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:56.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.407+0000 7ff038600640 1 --2- 172.21.15.47:0/668182705 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff030070720 0x7ff03014c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:56.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.407+0000 7ff038600640 1 --2- 172.21.15.47:0/668182705 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff030070170 0x7ff0300b7060 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:26:56.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.407+0000 7ff038600640 1 -- 172.21.15.47:0/668182705 >> 172.21.15.47:0/668182705 conn(0x7ff0300b38d0 msgr2=0x7ff03015d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:26:56.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.407+0000 7ff038600640 1 -- 172.21.15.47:0/668182705 shutdown_connections 2024-02-19T15:26:56.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:26:56.407+0000 7ff038600640 1 -- 172.21.15.47:0/668182705 wait complete. 2024-02-19T15:26:56.432 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump 2024-02-19T15:27:00.493 INFO:teuthology.orchestra.run.smithi116.stdout:{"message":"","return_code":0} 2024-02-19T15:27:00.493 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-02-19T15:27:00.507 INFO:teuthology.orchestra.run.smithi116.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":956,"msgr_send_messages":960,"msgr_recv_bytes":275405,"msgr_send_bytes":12428678,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.155074912,"msgr_running_send_time":0.092793510,"msgr_running_recv_time":0.022320033,"msgr_running_fast_dispatch_time":0.031928702,"msgr_send_messages_queue_lat":{"avgcount":959,"sum":0.343892854,"avgtime":0.000358595},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000170,"avgtime":0.000000085},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":94416},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13766,"msgr_send_messages":24273,"msgr_recv_bytes":4604135,"msgr_send_bytes":15251319,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.989864612,"msgr_running_send_time":0.592936841,"msgr_running_recv_time":1.024731276,"msgr_running_fast_dispatch_time":0.023546605,"msgr_send_messages_queue_lat":{"avgcount":24273,"sum":0.979751879,"avgtime":0.000040363},"msgr_handle_ack_lat":{"avgcount":17992,"sum":0.003370273,"avgtime":0.000000187},"msgr_recv_encrypted_bytes":104656,"msgr_send_encrypted_bytes":7936},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1492,"msgr_send_messages":1495,"msgr_recv_bytes":428527,"msgr_send_bytes":2505771,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.126787030,"msgr_running_send_time":0.055160045,"msgr_running_recv_time":0.029521681,"msgr_running_fast_dispatch_time":0.035334748,"msgr_send_messages_queue_lat":{"avgcount":1494,"sum":2.281387563,"avgtime":0.001527033},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":77712},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":127,"sum":6.565337948,"avgtime":0.051695574}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":3819,"sum":1.149007252,"avgtime":0.000300865}},"mds":{"request":6412,"reply":6410,"reply_latency":{"avgcount":6410,"sum":386.294306555,"avgtime":0.060264322},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":2054,"inodes_top":0,"inodes_bottom":2054,"inodes_pin_tail":0,"inodes_pinned":2054,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6413,"traverse_hit":6413,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":4,"ceph_cap_op_grant":7512,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5315,"handle_client_caps_dirty":3000,"handle_client_cap_release":1070,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1020,"num_strays_delayed":1020,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":2181,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":14241,"evex":13222,"evtrm":12198,"ev":14243,"evexg":0,"evexd":13221,"segadd":17,"segex":18,"segtrm":17,"seg":2,"segmjr":1,"segexg":0,"segexd":17,"expos":30209192,"wrpos":32798735,"rdpos":4195244,"jlat":{"avgcount":23,"sum":3.103890750,"avgtime":0.134951771},"replayed":3},"mds_mem":{"ino":2056,"ino+":3214,"ino-":1158,"dir":12,"dir+":13,"dir-":1,"dn":2054,"dn+":6412,"dn-":4358,"cap":1,"cap+":3202,"cap-":3201,"rss":106244,"heap":248112},"mds_server":{"dispatch_client_request":6415,"dispatch_server_request":0,"handle_client_request":6412,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":381.004064656,"avgtime":0.119063770},"req_getattr_latency":{"avgcount":2,"sum":0.167551542,"avgtime":0.083775771},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003020866,"avgtime":0.003020866},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.395814412,"avgtime":0.065969068},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.525413336,"avgtime":0.525413336},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":4.198441743,"avgtime":0.001312013},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":6007,"average_load":6007,"avg_session_uptime":1000,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":20788514,"buffer_anon_items":5204,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":8347315,"mds_co_items":129718,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3368,"op_send_bytes":30077249,"op_resend":0,"op_reply":3368,"op_latency":{"avgcount":3368,"sum":99.775046192,"avgtime":0.029624419},"op_inflight":0,"oplen_avg":{"avgcount":3368,"sum":5513},"op":3368,"op_r":53,"op_w":3315,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1030,"osdop_read":25,"osdop_write":40,"osdop_writefull":15,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":2204,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":2060,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":46,"omap_rd":48,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":2181,"pq_executed":2181,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":16209,"get_sum":4051408,"get_or_fail_fail":0,"get_or_fail_success":16209,"take":0,"take_sum":0,"put":16209,"put_sum":4051408,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3368,"take_sum":30298464,"put":3368,"put_sum":30298464,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3368,"take_sum":3368,"put":3368,"put_sum":3368,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":2181,"get_sum":220281,"get_or_fail_fail":0,"get_or_fail_success":2181,"take":0,"take_sum":0,"put":17,"put_sum":220281,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55fa43f90dc0":{"val":0,"max":3758096384,"get_started":0,"get":14241,"get_sum":28603233,"get_or_fail_fail":0,"get_or_fail_success":14241,"take":0,"take_sum":0,"put":23,"put_sum":28603233,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-02-19T15:27:00.508 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 8.5e-08, "sum": 1.7e-07 }, "msgr_recv_bytes": 275405, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 956, "msgr_running_fast_dispatch_time": 0.031928702, "msgr_running_recv_time": 0.022320033, "msgr_running_send_time": 0.09279351, "msgr_running_total_time": 0.155074912, "msgr_send_bytes": 12428678, "msgr_send_encrypted_bytes": 94416, "msgr_send_messages": 960, "msgr_send_messages_queue_lat": { "avgcount": 959, "avgtime": 0.000358595, "sum": 0.343892854 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 17992, "avgtime": 1.87e-07, "sum": 0.003370273 }, "msgr_recv_bytes": 4604135, "msgr_recv_encrypted_bytes": 104656, "msgr_recv_messages": 13766, "msgr_running_fast_dispatch_time": 0.023546605, "msgr_running_recv_time": 1.024731276, "msgr_running_send_time": 0.592936841, "msgr_running_total_time": 0.989864612, "msgr_send_bytes": 15251319, "msgr_send_encrypted_bytes": 7936, "msgr_send_messages": 24273, "msgr_send_messages_queue_lat": { "avgcount": 24273, "avgtime": 4.0363e-05, "sum": 0.979751879 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 428527, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 1492, "msgr_running_fast_dispatch_time": 0.035334748, "msgr_running_recv_time": 0.029521681, "msgr_running_send_time": 0.055160045, "msgr_running_total_time": 0.12678703, "msgr_send_bytes": 2505771, "msgr_send_encrypted_bytes": 77712, "msgr_send_messages": 1495, "msgr_send_messages_queue_lat": { "avgcount": 1494, "avgtime": 0.001527033, "sum": 2.281387563 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 127, "avgtime": 0.051695574, "sum": 6.565337948 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 3819, "avgtime": 0.000300865, "sum": 1.149007252 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7512, "ceph_cap_op_revoke": 4, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 1070, "handle_client_caps": 5315, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 2054, "inodes_bottom": 2054, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 2054, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.060264322, "sum": 386.294306555 }, "request": 6412, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6413, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6413, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1020, "num_strays_delayed": 1020, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 2181, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 14243, "evadd": 14241, "evex": 13222, "evexd": 13221, "evexg": 0, "evlrg": 0, "evtrm": 12198, "expos": 30209192, "jlat": { "avgcount": 23, "avgtime": 0.134951771, "sum": 3.10389075 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 17, "segex": 18, "segexd": 17, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 32798735 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 12, "dir+": 13, "dir-": 1, "dn": 2054, "dn+": 6412, "dn-": 4358, "heap": 248112, "ino": 2056, "ino+": 3214, "ino-": 1158, "rss": 106244 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6415, "dispatch_server_request": 0, "handle_client_request": 6412, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.11906377, "sum": 381.004064656 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.083775771, "sum": 0.167551542 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003020866, "sum": 0.003020866 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.065969068, "sum": 0.395814412 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.525413336, "sum": 0.525413336 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001312013, "sum": 4.198441743 } }, "mds_sessions": { "average_load": 6007, "avg_session_uptime": 1000, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 6007 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 20788514, "buffer_anon_items": 5204, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 8347315, "mds_co_items": 129718, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 48, "omap_wr": 46, "op": 3368, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3368, "avgtime": 0.029624419, "sum": 99.775046192 }, "op_pg": 0, "op_r": 53, "op_reply": 3368, "op_resend": 0, "op_rmw": 0, "op_send": 3368, "op_send_bytes": 30077249, "op_w": 3315, "oplen_avg": { "avgcount": 3368, "sum": 5513 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1030, "osdop_delete": 2204, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 25, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 2060, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 40, "osdop_writefull": 15, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 2181, "pq_executed_ops": 2181, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 16209, "get_or_fail_fail": 0, "get_or_fail_success": 16209, "get_started": 0, "get_sum": 4051408, "max": 104857600, "put": 16209, "put_sum": 4051408, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3368, "put_sum": 30298464, "take": 3368, "take_sum": 30298464, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3368, "put_sum": 3368, "take": 3368, "take_sum": 3368, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 2181, "get_or_fail_fail": 0, "get_or_fail_success": 2181, "get_started": 0, "get_sum": 220281, "max": 3758096384, "put": 17, "put_sum": 220281, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55fa43f90dc0": { "get": 14241, "get_or_fail_fail": 0, "get_or_fail_success": 14241, "get_started": 0, "get_sum": 28603233, "max": 3758096384, "put": 23, "put_sum": 28603233, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-02-19T15:27:00.509 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1020, 2181/3201 2024-02-19T15:27:01.510 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:27:01.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.643+0000 7f8c0d800640 1 Processor -- start 2024-02-19T15:27:01.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.647+0000 7f8c0d800640 1 -- start start 2024-02-19T15:27:01.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.647+0000 7f8c0d800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c0806c180 0x7f8c08162500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:01.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.647+0000 7f8c0d800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8c0815e530 0x7f8c0806f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:01.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.647+0000 7f8c0d800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c080b4a30 con 0x7f8c0806c560 2024-02-19T15:27:01.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.647+0000 7f8c0d800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c080aee00 con 0x7f8c0815e530 2024-02-19T15:27:01.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.647+0000 7f8c0d800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c08162a40 con 0x7f8c0806c180 2024-02-19T15:27:01.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.647+0000 7f8bffe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8c0815e530 0x7f8c0806f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:01.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.647+0000 7f8c0ce00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8c0806c560 0x7f8c0806f8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53046/0 (socket says 172.21.15.47:53046) 2024-02-19T15:27:01.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.647+0000 7f8c0ce00640 1 -- 172.21.15.47:0/787192692 learned_addr learned my addr 172.21.15.47:0/787192692 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:01.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.647+0000 7f8c07e00640 1 --2- 172.21.15.47:0/787192692 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c0806c180 0x7f8c08162500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:01.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.647+0000 7f8bffe00640 1 -- 172.21.15.47:0/787192692 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c0806c180 msgr2=0x7f8c08162500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:01.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.647+0000 7f8bffe00640 1 --2- 172.21.15.47:0/787192692 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c0806c180 0x7f8c08162500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:01.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.647+0000 7f8bffe00640 1 -- 172.21.15.47:0/787192692 >> v1:172.21.15.47:6789/0 conn(0x7f8c0806c560 legacy=0x7f8c0806f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:01.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.647+0000 7f8bffe00640 1 -- 172.21.15.47:0/787192692 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0815ea60 con 0x7f8c0815e530 2024-02-19T15:27:01.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.647+0000 7f8c07400640 1 -- 172.21.15.47:0/787192692 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 103854596 0 0) 0x7f8c080b4a30 con 0x7f8c0806c560 2024-02-19T15:27:01.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.651+0000 7f8bffe00640 1 --2- 172.21.15.47:0/787192692 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c0815e530 0x7f8c0806f370 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f8bf00097c0 tx=0x7f8bf001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa36ccfb3ebe6d57 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:01.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.651+0000 7f8c07400640 1 -- 172.21.15.47:0/787192692 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf00ac410 con 0x7f8c0815e530 2024-02-19T15:27:01.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.651+0000 7f8c07400640 1 -- 172.21.15.47:0/787192692 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf00ac570 con 0x7f8c0815e530 2024-02-19T15:27:01.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.651+0000 7f8c07400640 1 -- 172.21.15.47:0/787192692 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf00ac900 con 0x7f8c0815e530 2024-02-19T15:27:01.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.651+0000 7f8c0d800640 1 -- 172.21.15.47:0/787192692 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c0815e530 msgr2=0x7f8c0806f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:01.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.651+0000 7f8c0d800640 1 --2- 172.21.15.47:0/787192692 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c0815e530 0x7f8c0806f370 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f8bf00097c0 tx=0x7f8bf001c4a0 comp rx=0 tx=0).stop 2024-02-19T15:27:01.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.651+0000 7f8c0d800640 1 -- 172.21.15.47:0/787192692 shutdown_connections 2024-02-19T15:27:01.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.651+0000 7f8c0d800640 1 --2- 172.21.15.47:0/787192692 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c0815e530 0x7f8c0806f370 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:01.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.651+0000 7f8c0d800640 1 --2- 172.21.15.47:0/787192692 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c0806c180 0x7f8c08162500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:01.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.651+0000 7f8c0d800640 1 -- 172.21.15.47:0/787192692 >> 172.21.15.47:0/787192692 conn(0x7f8c08071430 msgr2=0x7f8c08071840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:01.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.651+0000 7f8c0d800640 1 -- 172.21.15.47:0/787192692 shutdown_connections 2024-02-19T15:27:01.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.651+0000 7f8c0d800640 1 -- 172.21.15.47:0/787192692 wait complete. 2024-02-19T15:27:01.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.651+0000 7f8c0d800640 1 Processor -- start 2024-02-19T15:27:01.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.651+0000 7f8c0d800640 1 -- start start 2024-02-19T15:27:01.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c0d800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c0806c180 0x7f8c080adb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:01.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c0d800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c0806c560 0x7f8c08153220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:01.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c0d800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c0815e530 0x7f8c081f9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:01.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c0d800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c08154fd0 con 0x7f8c0815e530 2024-02-19T15:27:01.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c0d800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0815b050 con 0x7f8c0806c560 2024-02-19T15:27:01.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c0d800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0815b330 con 0x7f8c0806c180 2024-02-19T15:27:01.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c0ce00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c0815e530 0x7f8c081f9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:01.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c07e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c0806c180 0x7f8c080adb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:01.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c0ce00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c0815e530 0x7f8c081f9fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:38522/0 (socket says 172.21.15.47:38522) 2024-02-19T15:27:01.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c0ce00640 1 -- 172.21.15.47:0/4011043462 learned_addr learned my addr 172.21.15.47:0/4011043462 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:01.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8bffe00640 1 --2- 172.21.15.47:0/4011043462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c0806c560 0x7f8c08153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:01.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c07e00640 1 -- 172.21.15.47:0/4011043462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c0806c560 msgr2=0x7f8c08153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:01.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c07e00640 1 --2- 172.21.15.47:0/4011043462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c0806c560 0x7f8c08153220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:01.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c07e00640 1 -- 172.21.15.47:0/4011043462 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c0815e530 msgr2=0x7f8c081f9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:01.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c07e00640 1 --2- 172.21.15.47:0/4011043462 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c0815e530 0x7f8c081f9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:01.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c07e00640 1 -- 172.21.15.47:0/4011043462 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf0009420 con 0x7f8c0806c180 2024-02-19T15:27:01.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c07e00640 1 --2- 172.21.15.47:0/4011043462 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c0806c180 0x7f8c080adb40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8bf800e870 tx=0x7f8bf800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:01.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c04c00640 1 -- 172.21.15.47:0/4011043462 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf800ee60 con 0x7f8c0806c180 2024-02-19T15:27:01.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c04c00640 1 -- 172.21.15.47:0/4011043462 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf800c940 con 0x7f8c0806c180 2024-02-19T15:27:01.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c0d800640 1 -- 172.21.15.47:0/4011043462 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0815b610 con 0x7f8c0806c180 2024-02-19T15:27:01.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c04c00640 1 -- 172.21.15.47:0/4011043462 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf800ccd0 con 0x7f8c0806c180 2024-02-19T15:27:01.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c0d800640 1 -- 172.21.15.47:0/4011043462 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c080ae080 con 0x7f8c0806c180 2024-02-19T15:27:01.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c0d800640 1 -- 172.21.15.47:0/4011043462 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c08061c10 con 0x7f8c0806c180 2024-02-19T15:27:01.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.655+0000 7f8c04c00640 1 -- 172.21.15.47:0/4011043462 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8bf8013470 con 0x7f8c0806c180 2024-02-19T15:27:01.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.659+0000 7f8c04c00640 1 --2- 172.21.15.47:0/4011043462 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8be4066e40 0x7f8be4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:01.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.659+0000 7f8c04c00640 1 -- 172.21.15.47:0/4011043462 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f8bf8084b20 con 0x7f8c0806c180 2024-02-19T15:27:01.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.659+0000 7f8bffe00640 1 --2- 172.21.15.47:0/4011043462 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8be4066e40 0x7f8be4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:01.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.659+0000 7f8bffe00640 1 --2- 172.21.15.47:0/4011043462 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8be4066e40 0x7f8be4069300 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8bf00ae000 tx=0x7f8bf00a6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:01.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.663+0000 7f8c04c00640 1 -- 172.21.15.47:0/4011043462 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8bf8051870 con 0x7f8c0806c180 2024-02-19T15:27:01.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.859+0000 7f8c0d800640 1 -- 172.21.15.47:0/4011043462 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c08068e70 con 0x7f8c0806c180 2024-02-19T15:27:01.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.859+0000 7f8c04c00640 1 -- 172.21.15.47:0/4011043462 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4947 (secure 0 0 0) 0x7f8bf8017090 con 0x7f8c0806c180 2024-02-19T15:27:01.864 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:01.864 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":66,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7060,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/4006350936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":4006350936},{"type":"v1","addr":"172.21.15.116:6837","nonce":4006350936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/2191173523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2191173523},{"type":"v1","addr":"172.21.15.47:6834","nonce":2191173523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/1012301665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":1012301665},{"type":"v1","addr":"172.21.15.47:6835","nonce":1012301665}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:26:26.721109+0000","modified":"2024-02-19T15:26:43.866082+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7054":{"gid":7054,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6835/3852677195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3852677195},{"type":"v1","addr":"172.21.15.116:6835","nonce":3852677195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T15:27:01.864 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 66 2024-02-19T15:27:01.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.859+0000 7f8c0d800640 1 -- 172.21.15.47:0/4011043462 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8be4066e40 msgr2=0x7f8be4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:01.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.859+0000 7f8c0d800640 1 --2- 172.21.15.47:0/4011043462 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8be4066e40 0x7f8be4069300 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8bf00ae000 tx=0x7f8bf00a6040 comp rx=0 tx=0).stop 2024-02-19T15:27:01.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.859+0000 7f8c0d800640 1 -- 172.21.15.47:0/4011043462 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c0806c180 msgr2=0x7f8c080adb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:01.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.859+0000 7f8c0d800640 1 --2- 172.21.15.47:0/4011043462 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c0806c180 0x7f8c080adb40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8bf800e870 tx=0x7f8bf800ed30 comp rx=0 tx=0).stop 2024-02-19T15:27:01.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.859+0000 7f8c0d800640 1 -- 172.21.15.47:0/4011043462 shutdown_connections 2024-02-19T15:27:01.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.859+0000 7f8c0d800640 1 --2- 172.21.15.47:0/4011043462 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8be4066e40 0x7f8be4069300 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:01.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.859+0000 7f8c0d800640 1 --2- 172.21.15.47:0/4011043462 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c0815e530 0x7f8c081f9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:01.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.859+0000 7f8c0d800640 1 --2- 172.21.15.47:0/4011043462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c0806c560 0x7f8c08153220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:01.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.859+0000 7f8c0d800640 1 --2- 172.21.15.47:0/4011043462 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c0806c180 0x7f8c080adb40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:01.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.859+0000 7f8c0d800640 1 -- 172.21.15.47:0/4011043462 >> 172.21.15.47:0/4011043462 conn(0x7f8c08071430 msgr2=0x7f8c080cf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:01.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.859+0000 7f8c0d800640 1 -- 172.21.15.47:0/4011043462 shutdown_connections 2024-02-19T15:27:01.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:01.859+0000 7f8c0d800640 1 -- 172.21.15.47:0/4011043462 wait complete. 2024-02-19T15:27:01.885 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump 2024-02-19T15:27:02.054 INFO:teuthology.orchestra.run.smithi116.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1256,"msgr_send_messages":1261,"msgr_recv_bytes":351601,"msgr_send_bytes":12526208,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.178300502,"msgr_running_send_time":0.101744309,"msgr_running_recv_time":0.028514729,"msgr_running_fast_dispatch_time":0.038300525,"msgr_send_messages_queue_lat":{"avgcount":1260,"sum":0.353983785,"avgtime":0.000280939},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000170,"avgtime":0.000000085},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":96208},"AsyncMessenger::Worker-1":{"msgr_recv_messages":14031,"msgr_send_messages":24537,"msgr_recv_bytes":4774557,"msgr_send_bytes":15438589,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":1.011541393,"msgr_running_send_time":0.601357876,"msgr_running_recv_time":1.030412105,"msgr_running_fast_dispatch_time":0.029627084,"msgr_send_messages_queue_lat":{"avgcount":24537,"sum":0.989539904,"avgtime":0.000040328},"msgr_handle_ack_lat":{"avgcount":17992,"sum":0.003370273,"avgtime":0.000000187},"msgr_recv_encrypted_bytes":104656,"msgr_send_encrypted_bytes":7936},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1956,"msgr_send_messages":1959,"msgr_recv_bytes":546383,"msgr_send_bytes":2653787,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.161489819,"msgr_running_send_time":0.068319028,"msgr_running_recv_time":0.038770983,"msgr_running_fast_dispatch_time":0.045105082,"msgr_send_messages_queue_lat":{"avgcount":1958,"sum":2.302137837,"avgtime":0.001175759},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":77712},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":128,"sum":6.565348948,"avgtime":0.051291788}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5712,"sum":1.243101258,"avgtime":0.000217629}},"mds":{"request":6412,"reply":6410,"reply_latency":{"avgcount":6410,"sum":386.294306555,"avgtime":0.060264322},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":2054,"inodes_top":0,"inodes_bottom":2054,"inodes_pin_tail":0,"inodes_pinned":2054,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6413,"traverse_hit":6413,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":4,"ceph_cap_op_grant":7512,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5315,"handle_client_caps_dirty":3000,"handle_client_cap_release":1070,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15262,"evex":13222,"evtrm":12198,"ev":3067,"evexg":0,"evexd":13221,"segadd":18,"segex":18,"segtrm":17,"seg":3,"segmjr":1,"segexg":0,"segexd":17,"expos":30209192,"wrpos":33409763,"rdpos":4195244,"jlat":{"avgcount":23,"sum":3.103890750,"avgtime":0.134951771},"replayed":3},"mds_mem":{"ino":1033,"ino+":3214,"ino-":2181,"dir":12,"dir+":13,"dir-":1,"dn":1031,"dn+":6412,"dn-":5381,"cap":1,"cap+":3202,"cap-":3201,"rss":108804,"heap":248112},"mds_server":{"dispatch_client_request":6415,"dispatch_server_request":0,"handle_client_request":6412,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":381.004064656,"avgtime":0.119063770},"req_getattr_latency":{"avgcount":2,"sum":0.167551542,"avgtime":0.083775771},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003020866,"avgtime":0.003020866},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.395814412,"avgtime":0.065969068},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.525413336,"avgtime":0.525413336},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":4.198441743,"avgtime":0.001312013},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":6007,"average_load":6007,"avg_session_uptime":1000,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":20788514,"buffer_anon_items":5204,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":8347315,"mds_co_items":129718,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":4396,"op_send_bytes":30180401,"op_resend":0,"op_reply":4396,"op_latency":{"avgcount":4396,"sum":101.959025320,"avgtime":0.023193590},"op_inflight":0,"oplen_avg":{"avgcount":4396,"sum":6541},"op":4396,"op_r":56,"op_w":4340,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1030,"osdop_read":28,"osdop_write":43,"osdop_writefull":16,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3224,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":2060,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":47,"omap_rd":48,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":17238,"get_sum":4323288,"get_or_fail_fail":0,"get_or_fail_success":17238,"take":0,"take_sum":0,"put":17238,"put_sum":4323288,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4396,"take_sum":30504636,"put":4396,"put_sum":30504636,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4396,"take_sum":4396,"put":4396,"put_sum":4396,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":20,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55fa43f90dc0":{"val":611028,"max":3758096384,"get_started":0,"get":15262,"get_sum":29214261,"get_or_fail_fail":0,"get_or_fail_success":15262,"take":0,"take_sum":0,"put":23,"put_sum":28603233,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-02-19T15:27:02.056 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 8.5e-08, "sum": 1.7e-07 }, "msgr_recv_bytes": 351601, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 1256, "msgr_running_fast_dispatch_time": 0.038300525, "msgr_running_recv_time": 0.028514729, "msgr_running_send_time": 0.101744309, "msgr_running_total_time": 0.178300502, "msgr_send_bytes": 12526208, "msgr_send_encrypted_bytes": 96208, "msgr_send_messages": 1261, "msgr_send_messages_queue_lat": { "avgcount": 1260, "avgtime": 0.000280939, "sum": 0.353983785 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 17992, "avgtime": 1.87e-07, "sum": 0.003370273 }, "msgr_recv_bytes": 4774557, "msgr_recv_encrypted_bytes": 104656, "msgr_recv_messages": 14031, "msgr_running_fast_dispatch_time": 0.029627084, "msgr_running_recv_time": 1.030412105, "msgr_running_send_time": 0.601357876, "msgr_running_total_time": 1.011541393, "msgr_send_bytes": 15438589, "msgr_send_encrypted_bytes": 7936, "msgr_send_messages": 24537, "msgr_send_messages_queue_lat": { "avgcount": 24537, "avgtime": 4.0328e-05, "sum": 0.989539904 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 546383, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 1956, "msgr_running_fast_dispatch_time": 0.045105082, "msgr_running_recv_time": 0.038770983, "msgr_running_send_time": 0.068319028, "msgr_running_total_time": 0.161489819, "msgr_send_bytes": 2653787, "msgr_send_encrypted_bytes": 77712, "msgr_send_messages": 1959, "msgr_send_messages_queue_lat": { "avgcount": 1958, "avgtime": 0.001175759, "sum": 2.302137837 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 128, "avgtime": 0.051291788, "sum": 6.565348948 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5712, "avgtime": 0.000217629, "sum": 1.243101258 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7512, "ceph_cap_op_revoke": 4, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 1070, "handle_client_caps": 5315, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 2054, "inodes_bottom": 2054, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 2054, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.060264322, "sum": 386.294306555 }, "request": 6412, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6413, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6413, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 3067, "evadd": 15262, "evex": 13222, "evexd": 13221, "evexg": 0, "evlrg": 0, "evtrm": 12198, "expos": 30209192, "jlat": { "avgcount": 23, "avgtime": 0.134951771, "sum": 3.10389075 }, "rdpos": 4195244, "replayed": 3, "seg": 3, "segadd": 18, "segex": 18, "segexd": 17, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 33409763 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 12, "dir+": 13, "dir-": 1, "dn": 1031, "dn+": 6412, "dn-": 5381, "heap": 248112, "ino": 1033, "ino+": 3214, "ino-": 2181, "rss": 108804 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6415, "dispatch_server_request": 0, "handle_client_request": 6412, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.11906377, "sum": 381.004064656 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.083775771, "sum": 0.167551542 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003020866, "sum": 0.003020866 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.065969068, "sum": 0.395814412 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.525413336, "sum": 0.525413336 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001312013, "sum": 4.198441743 } }, "mds_sessions": { "average_load": 6007, "avg_session_uptime": 1000, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 6007 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 20788514, "buffer_anon_items": 5204, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 8347315, "mds_co_items": 129718, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 48, "omap_wr": 47, "op": 4396, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 4396, "avgtime": 0.02319359, "sum": 101.95902532 }, "op_pg": 0, "op_r": 56, "op_reply": 4396, "op_resend": 0, "op_rmw": 0, "op_send": 4396, "op_send_bytes": 30180401, "op_w": 4340, "oplen_avg": { "avgcount": 4396, "sum": 6541 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1030, "osdop_delete": 3224, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 28, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 2060, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 43, "osdop_writefull": 16, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 17238, "get_or_fail_fail": 0, "get_or_fail_success": 17238, "get_started": 0, "get_sum": 4323288, "max": 104857600, "put": 17238, "put_sum": 4323288, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 4396, "put_sum": 30504636, "take": 4396, "take_sum": 30504636, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 4396, "put_sum": 4396, "take": 4396, "take_sum": 4396, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 20, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55fa43f90dc0": { "get": 15262, "get_or_fail_fail": 0, "get_or_fail_success": 15262, "get_started": 0, "get_sum": 29214261, "max": 3758096384, "put": 23, "put_sum": 28603233, "take": 0, "take_sum": 0, "val": 611028, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-02-19T15:27:02.057 INFO:tasks.cephfs.test_strays:Complete purge in 5 seconds 2024-02-19T15:27:02.057 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:27:02.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.195+0000 7ff766400640 1 Processor -- start 2024-02-19T15:27:02.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.195+0000 7ff766400640 1 -- start start 2024-02-19T15:27:02.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.195+0000 7ff766400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff7600b5620 0x7ff7600b5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:02.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.195+0000 7ff766400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7600b05d0 0x7ff7600b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:02.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.195+0000 7ff766400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7600b0140 con 0x7ff7601599b0 2024-02-19T15:27:02.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.195+0000 7ff766400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7600b6010 con 0x7ff7600b5620 2024-02-19T15:27:02.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.195+0000 7ff766400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76015d480 con 0x7ff7600b05d0 2024-02-19T15:27:02.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.195+0000 7ff765000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff7601599b0 0x7ff7600b4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53050/0 (socket says 172.21.15.47:53050) 2024-02-19T15:27:02.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.195+0000 7ff765000640 1 -- 172.21.15.47:0/193080451 learned_addr learned my addr 172.21.15.47:0/193080451 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:02.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.195+0000 7ff765a00640 1 --2- 172.21.15.47:0/193080451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7600b05d0 0x7ff7600b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:02.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.195+0000 7ff757e00640 1 --2- 172.21.15.47:0/193080451 >> v2:172.21.15.116:3300/0 conn(0x7ff7600b5620 0x7ff7600b5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:02.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.195+0000 7ff75fe00640 1 -- 172.21.15.47:0/193080451 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3543423699 0 0) 0x7ff7600b0140 con 0x7ff7601599b0 2024-02-19T15:27:02.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.195+0000 7ff75fe00640 1 -- 172.21.15.47:0/193080451 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff744003530 con 0x7ff7601599b0 2024-02-19T15:27:02.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.195+0000 7ff765a00640 1 -- 172.21.15.47:0/193080451 >> v2:172.21.15.116:3300/0 conn(0x7ff7600b5620 msgr2=0x7ff7600b5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:02.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.195+0000 7ff765a00640 1 --2- 172.21.15.47:0/193080451 >> v2:172.21.15.116:3300/0 conn(0x7ff7600b5620 0x7ff7600b5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:02.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.195+0000 7ff765a00640 1 -- 172.21.15.47:0/193080451 >> v1:172.21.15.47:6789/0 conn(0x7ff7601599b0 legacy=0x7ff7600b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:02.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.195+0000 7ff765a00640 1 -- 172.21.15.47:0/193080451 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7600af290 con 0x7ff7600b05d0 2024-02-19T15:27:02.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.195+0000 7ff757e00640 1 --2- 172.21.15.47:0/193080451 >> v2:172.21.15.116:3300/0 conn(0x7ff7600b5620 0x7ff7600b5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:27:02.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.199+0000 7ff765a00640 1 --2- 172.21.15.47:0/193080451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7600b05d0 0x7ff7600b0a00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff758009d90 tx=0x7ff75801c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b597026143e15c24 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:02.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.199+0000 7ff75fe00640 1 -- 172.21.15.47:0/193080451 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7580a5a00 con 0x7ff7600b05d0 2024-02-19T15:27:02.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.199+0000 7ff75fe00640 1 -- 172.21.15.47:0/193080451 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7580a5b60 con 0x7ff7600b05d0 2024-02-19T15:27:02.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.199+0000 7ff75fe00640 1 -- 172.21.15.47:0/193080451 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7580ad440 con 0x7ff7600b05d0 2024-02-19T15:27:02.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.199+0000 7ff766400640 1 -- 172.21.15.47:0/193080451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7600b05d0 msgr2=0x7ff7600b0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:02.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.199+0000 7ff766400640 1 --2- 172.21.15.47:0/193080451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7600b05d0 0x7ff7600b0a00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff758009d90 tx=0x7ff75801c940 comp rx=0 tx=0).stop 2024-02-19T15:27:02.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.199+0000 7ff766400640 1 -- 172.21.15.47:0/193080451 shutdown_connections 2024-02-19T15:27:02.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.199+0000 7ff766400640 1 --2- 172.21.15.47:0/193080451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7600b05d0 0x7ff7600b0a00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:02.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.199+0000 7ff766400640 1 --2- 172.21.15.47:0/193080451 >> v2:172.21.15.116:3300/0 conn(0x7ff7600b5620 0x7ff7600b5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:02.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.199+0000 7ff766400640 1 -- 172.21.15.47:0/193080451 >> 172.21.15.47:0/193080451 conn(0x7ff76009aa70 msgr2=0x7ff76009ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:02.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.199+0000 7ff766400640 1 -- 172.21.15.47:0/193080451 shutdown_connections 2024-02-19T15:27:02.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.199+0000 7ff766400640 1 -- 172.21.15.47:0/193080451 wait complete. 2024-02-19T15:27:02.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff766400640 1 Processor -- start 2024-02-19T15:27:02.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff766400640 1 -- start start 2024-02-19T15:27:02.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff766400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7600b05d0 0x7ff76014f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:02.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff766400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7600b5620 0x7ff760151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:02.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff766400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7601599b0 0x7ff76006fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:02.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff766400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff760158c60 con 0x7ff7600b05d0 2024-02-19T15:27:02.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff766400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff760150190 con 0x7ff7600b5620 2024-02-19T15:27:02.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff766400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7601503e0 con 0x7ff7601599b0 2024-02-19T15:27:02.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff765000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7600b05d0 0x7ff76014f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:02.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff765000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7600b05d0 0x7ff76014f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:38528/0 (socket says 172.21.15.47:38528) 2024-02-19T15:27:02.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff765000640 1 -- 172.21.15.47:0/1356805955 learned_addr learned my addr 172.21.15.47:0/1356805955 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:02.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff757e00640 1 --2- 172.21.15.47:0/1356805955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7600b5620 0x7ff760151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:02.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff765a00640 1 --2- 172.21.15.47:0/1356805955 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7601599b0 0x7ff76006fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:02.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff765000640 1 -- 172.21.15.47:0/1356805955 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7601599b0 msgr2=0x7ff76006fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:02.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff765000640 1 --2- 172.21.15.47:0/1356805955 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7601599b0 0x7ff76006fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:02.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff765000640 1 -- 172.21.15.47:0/1356805955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7600b5620 msgr2=0x7ff760151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:02.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff765000640 1 --2- 172.21.15.47:0/1356805955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7600b5620 0x7ff760151480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:02.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff765000640 1 -- 172.21.15.47:0/1356805955 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff748009420 con 0x7ff7600b05d0 2024-02-19T15:27:02.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff765000640 1 --2- 172.21.15.47:0/1356805955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7600b05d0 0x7ff76014f9f0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7ff75000c450 tx=0x7ff75000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:02.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff75d600640 1 -- 172.21.15.47:0/1356805955 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff750005cf0 con 0x7ff7600b05d0 2024-02-19T15:27:02.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff75d600640 1 -- 172.21.15.47:0/1356805955 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff750005e50 con 0x7ff7600b05d0 2024-02-19T15:27:02.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff75d600640 1 -- 172.21.15.47:0/1356805955 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff750007290 con 0x7ff7600b05d0 2024-02-19T15:27:02.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff766400640 1 -- 172.21.15.47:0/1356805955 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff758009a70 con 0x7ff7600b05d0 2024-02-19T15:27:02.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff766400640 1 -- 172.21.15.47:0/1356805955 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7601509a0 con 0x7ff7600b05d0 2024-02-19T15:27:02.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.203+0000 7ff766400640 1 -- 172.21.15.47:0/1356805955 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff760068040 con 0x7ff7600b05d0 2024-02-19T15:27:02.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.207+0000 7ff75d600640 1 -- 172.21.15.47:0/1356805955 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff7500073f0 con 0x7ff7600b05d0 2024-02-19T15:27:02.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.207+0000 7ff75d600640 1 --2- 172.21.15.47:0/1356805955 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff73c066e40 0x7ff73c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:02.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.207+0000 7ff75d600640 1 -- 172.21.15.47:0/1356805955 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7ff750083ff0 con 0x7ff7600b05d0 2024-02-19T15:27:02.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.207+0000 7ff757e00640 1 --2- 172.21.15.47:0/1356805955 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff73c066e40 0x7ff73c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:02.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.207+0000 7ff757e00640 1 --2- 172.21.15.47:0/1356805955 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff73c066e40 0x7ff73c069300 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff760070a70 tx=0x7ff748009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:02.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.211+0000 7ff75d600640 1 -- 172.21.15.47:0/1356805955 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff7500b61c0 con 0x7ff7600b05d0 2024-02-19T15:27:02.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.407+0000 7ff766400640 1 -- 172.21.15.47:0/1356805955 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff76014ff30 con 0x7ff7600b05d0 2024-02-19T15:27:02.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.407+0000 7ff75d600640 1 -- 172.21.15.47:0/1356805955 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4947 (secure 0 0 0) 0x7ff750017070 con 0x7ff7600b05d0 2024-02-19T15:27:02.411 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:02.412 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":66,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7060,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/4006350936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":4006350936},{"type":"v1","addr":"172.21.15.116:6837","nonce":4006350936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/2191173523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2191173523},{"type":"v1","addr":"172.21.15.47:6834","nonce":2191173523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/1012301665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":1012301665},{"type":"v1","addr":"172.21.15.47:6835","nonce":1012301665}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:26:26.721109+0000","modified":"2024-02-19T15:26:43.866082+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7054":{"gid":7054,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6835/3852677195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3852677195},{"type":"v1","addr":"172.21.15.116:6835","nonce":3852677195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T15:27:02.412 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 66 2024-02-19T15:27:02.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.407+0000 7ff766400640 1 -- 172.21.15.47:0/1356805955 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff73c066e40 msgr2=0x7ff73c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:02.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.407+0000 7ff766400640 1 --2- 172.21.15.47:0/1356805955 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff73c066e40 0x7ff73c069300 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff760070a70 tx=0x7ff748009190 comp rx=0 tx=0).stop 2024-02-19T15:27:02.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.407+0000 7ff766400640 1 -- 172.21.15.47:0/1356805955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7600b05d0 msgr2=0x7ff76014f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:02.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.407+0000 7ff766400640 1 --2- 172.21.15.47:0/1356805955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7600b05d0 0x7ff76014f9f0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7ff75000c450 tx=0x7ff75000c910 comp rx=0 tx=0).stop 2024-02-19T15:27:02.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.407+0000 7ff766400640 1 -- 172.21.15.47:0/1356805955 shutdown_connections 2024-02-19T15:27:02.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.407+0000 7ff766400640 1 --2- 172.21.15.47:0/1356805955 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff73c066e40 0x7ff73c069300 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:02.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.407+0000 7ff766400640 1 --2- 172.21.15.47:0/1356805955 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7601599b0 0x7ff76006fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:02.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.407+0000 7ff766400640 1 --2- 172.21.15.47:0/1356805955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7600b5620 0x7ff760151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:02.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.407+0000 7ff766400640 1 --2- 172.21.15.47:0/1356805955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff7600b05d0 0x7ff76014f9f0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:02.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.407+0000 7ff766400640 1 -- 172.21.15.47:0/1356805955 >> 172.21.15.47:0/1356805955 conn(0x7ff76009aa70 msgr2=0x7ff76006f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:02.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.407+0000 7ff766400640 1 -- 172.21.15.47:0/1356805955 shutdown_connections 2024-02-19T15:27:02.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.407+0000 7ff766400640 1 -- 172.21.15.47:0/1356805955 wait complete. 2024-02-19T15:27:02.433 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump 2024-02-19T15:27:02.590 INFO:teuthology.orchestra.run.smithi116.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1256,"msgr_send_messages":1261,"msgr_recv_bytes":351601,"msgr_send_bytes":12526208,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.178300502,"msgr_running_send_time":0.101744309,"msgr_running_recv_time":0.028514729,"msgr_running_fast_dispatch_time":0.038300525,"msgr_send_messages_queue_lat":{"avgcount":1260,"sum":0.353983785,"avgtime":0.000280939},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000170,"avgtime":0.000000085},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":96208},"AsyncMessenger::Worker-1":{"msgr_recv_messages":14032,"msgr_send_messages":24537,"msgr_recv_bytes":4774959,"msgr_send_bytes":15438589,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":1.011655861,"msgr_running_send_time":0.601383710,"msgr_running_recv_time":1.030445681,"msgr_running_fast_dispatch_time":0.029679609,"msgr_send_messages_queue_lat":{"avgcount":24537,"sum":0.989539904,"avgtime":0.000040328},"msgr_handle_ack_lat":{"avgcount":17992,"sum":0.003370273,"avgtime":0.000000187},"msgr_recv_encrypted_bytes":104656,"msgr_send_encrypted_bytes":7936},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1956,"msgr_send_messages":1959,"msgr_recv_bytes":546383,"msgr_send_bytes":2653787,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.161489819,"msgr_running_send_time":0.068319028,"msgr_running_recv_time":0.038770983,"msgr_running_fast_dispatch_time":0.045105082,"msgr_send_messages_queue_lat":{"avgcount":1958,"sum":2.302137837,"avgtime":0.001175759},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":77712},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":128,"sum":6.565348948,"avgtime":0.051291788}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5712,"sum":1.243101258,"avgtime":0.000217629}},"mds":{"request":6412,"reply":6410,"reply_latency":{"avgcount":6410,"sum":386.294306555,"avgtime":0.060264322},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":2054,"inodes_top":0,"inodes_bottom":2054,"inodes_pin_tail":0,"inodes_pinned":2054,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6413,"traverse_hit":6413,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":4,"ceph_cap_op_grant":7512,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5315,"handle_client_caps_dirty":3000,"handle_client_cap_release":1070,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15262,"evex":13222,"evtrm":12198,"ev":3067,"evexg":0,"evexd":13221,"segadd":18,"segex":18,"segtrm":17,"seg":3,"segmjr":1,"segexg":0,"segexd":17,"expos":30209192,"wrpos":33409763,"rdpos":4195244,"jlat":{"avgcount":23,"sum":3.103890750,"avgtime":0.134951771},"replayed":3},"mds_mem":{"ino":1033,"ino+":3214,"ino-":2181,"dir":12,"dir+":13,"dir-":1,"dn":1031,"dn+":6412,"dn-":5381,"cap":1,"cap+":3202,"cap-":3201,"rss":108932,"heap":248112},"mds_server":{"dispatch_client_request":6415,"dispatch_server_request":0,"handle_client_request":6412,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":381.004064656,"avgtime":0.119063770},"req_getattr_latency":{"avgcount":2,"sum":0.167551542,"avgtime":0.083775771},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003020866,"avgtime":0.003020866},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.395814412,"avgtime":0.065969068},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.525413336,"avgtime":0.525413336},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":4.198441743,"avgtime":0.001312013},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":6007,"average_load":6007,"avg_session_uptime":1000,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":20788514,"buffer_anon_items":5204,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":8347315,"mds_co_items":129718,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":4396,"op_send_bytes":30180401,"op_resend":0,"op_reply":4396,"op_latency":{"avgcount":4396,"sum":101.959025320,"avgtime":0.023193590},"op_inflight":0,"oplen_avg":{"avgcount":4396,"sum":6541},"op":4396,"op_r":56,"op_w":4340,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1030,"osdop_read":28,"osdop_write":43,"osdop_writefull":16,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3224,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":2060,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":47,"omap_rd":48,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":17239,"get_sum":4323616,"get_or_fail_fail":0,"get_or_fail_success":17239,"take":0,"take_sum":0,"put":17239,"put_sum":4323616,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4396,"take_sum":30504636,"put":4396,"put_sum":30504636,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4396,"take_sum":4396,"put":4396,"put_sum":4396,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":20,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55fa43f90dc0":{"val":611028,"max":3758096384,"get_started":0,"get":15262,"get_sum":29214261,"get_or_fail_fail":0,"get_or_fail_success":15262,"take":0,"take_sum":0,"put":23,"put_sum":28603233,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-02-19T15:27:02.592 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 8.5e-08, "sum": 1.7e-07 }, "msgr_recv_bytes": 351601, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 1256, "msgr_running_fast_dispatch_time": 0.038300525, "msgr_running_recv_time": 0.028514729, "msgr_running_send_time": 0.101744309, "msgr_running_total_time": 0.178300502, "msgr_send_bytes": 12526208, "msgr_send_encrypted_bytes": 96208, "msgr_send_messages": 1261, "msgr_send_messages_queue_lat": { "avgcount": 1260, "avgtime": 0.000280939, "sum": 0.353983785 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 17992, "avgtime": 1.87e-07, "sum": 0.003370273 }, "msgr_recv_bytes": 4774959, "msgr_recv_encrypted_bytes": 104656, "msgr_recv_messages": 14032, "msgr_running_fast_dispatch_time": 0.029679609, "msgr_running_recv_time": 1.030445681, "msgr_running_send_time": 0.60138371, "msgr_running_total_time": 1.011655861, "msgr_send_bytes": 15438589, "msgr_send_encrypted_bytes": 7936, "msgr_send_messages": 24537, "msgr_send_messages_queue_lat": { "avgcount": 24537, "avgtime": 4.0328e-05, "sum": 0.989539904 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 546383, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 1956, "msgr_running_fast_dispatch_time": 0.045105082, "msgr_running_recv_time": 0.038770983, "msgr_running_send_time": 0.068319028, "msgr_running_total_time": 0.161489819, "msgr_send_bytes": 2653787, "msgr_send_encrypted_bytes": 77712, "msgr_send_messages": 1959, "msgr_send_messages_queue_lat": { "avgcount": 1958, "avgtime": 0.001175759, "sum": 2.302137837 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 128, "avgtime": 0.051291788, "sum": 6.565348948 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5712, "avgtime": 0.000217629, "sum": 1.243101258 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7512, "ceph_cap_op_revoke": 4, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 1070, "handle_client_caps": 5315, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 2054, "inodes_bottom": 2054, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 2054, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.060264322, "sum": 386.294306555 }, "request": 6412, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6413, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6413, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 3067, "evadd": 15262, "evex": 13222, "evexd": 13221, "evexg": 0, "evlrg": 0, "evtrm": 12198, "expos": 30209192, "jlat": { "avgcount": 23, "avgtime": 0.134951771, "sum": 3.10389075 }, "rdpos": 4195244, "replayed": 3, "seg": 3, "segadd": 18, "segex": 18, "segexd": 17, "segexg": 0, "segmjr": 1, "segtrm": 17, "wrpos": 33409763 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 12, "dir+": 13, "dir-": 1, "dn": 1031, "dn+": 6412, "dn-": 5381, "heap": 248112, "ino": 1033, "ino+": 3214, "ino-": 2181, "rss": 108932 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6415, "dispatch_server_request": 0, "handle_client_request": 6412, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.11906377, "sum": 381.004064656 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.083775771, "sum": 0.167551542 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003020866, "sum": 0.003020866 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.065969068, "sum": 0.395814412 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.525413336, "sum": 0.525413336 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001312013, "sum": 4.198441743 } }, "mds_sessions": { "average_load": 6007, "avg_session_uptime": 1000, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 6007 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 20788514, "buffer_anon_items": 5204, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 8347315, "mds_co_items": 129718, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 48, "omap_wr": 47, "op": 4396, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 4396, "avgtime": 0.02319359, "sum": 101.95902532 }, "op_pg": 0, "op_r": 56, "op_reply": 4396, "op_resend": 0, "op_rmw": 0, "op_send": 4396, "op_send_bytes": 30180401, "op_w": 4340, "oplen_avg": { "avgcount": 4396, "sum": 6541 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1030, "osdop_delete": 3224, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 28, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 2060, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 43, "osdop_writefull": 16, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 17239, "get_or_fail_fail": 0, "get_or_fail_success": 17239, "get_started": 0, "get_sum": 4323616, "max": 104857600, "put": 17239, "put_sum": 4323616, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 4396, "put_sum": 30504636, "take": 4396, "take_sum": 30504636, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 4396, "put_sum": 4396, "take": 4396, "take_sum": 4396, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 20, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55fa43f90dc0": { "get": 15262, "get_or_fail_fail": 0, "get_or_fail_success": 15262, "get_started": 0, "get_sum": 29214261, "max": 3758096384, "put": 23, "put_sum": 28603233, "take": 0, "take_sum": 0, "val": 611028, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-02-19T15:27:02.592 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T15:27:02.592 DEBUG:teuthology.orchestra.run.smithi047:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:27:02.673 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:27:02.744 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T15:27:02.744 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:27:02.744 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:27:02.753 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:27:02.753 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:27:02.802 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:27:02.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.983+0000 7f6724200640 1 Processor -- start 2024-02-19T15:27:02.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.983+0000 7f6724200640 1 -- start start 2024-02-19T15:27:02.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.983+0000 7f6724200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f671c158a30 0x7f671c15bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:02.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.983+0000 7f6724200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f671c06df30 0x7f671c06e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:02.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.983+0000 7f6724200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c06e8d0 con 0x7f671c071dc0 2024-02-19T15:27:02.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.983+0000 7f6724200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f671c158e10 con 0x7f671c06df30 2024-02-19T15:27:02.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.983+0000 7f6724200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f671c15c300 con 0x7f671c158a30 2024-02-19T15:27:02.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.983+0000 7f6720e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f671c06df30 0x7f671c06e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:02.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.983+0000 7f6721800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f671c158a30 0x7f671c15bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:02.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.983+0000 7f6720e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f671c06df30 0x7f671c06e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:38544/0 (socket says 172.21.15.47:38544) 2024-02-19T15:27:02.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.983+0000 7f6720e00640 1 -- 172.21.15.47:0/3871679152 learned_addr learned my addr 172.21.15.47:0/3871679152 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:02.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.983+0000 7f6720e00640 1 -- 172.21.15.47:0/3871679152 >> v2:172.21.15.116:3300/0 conn(0x7f671c158a30 msgr2=0x7f671c15bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:02.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.983+0000 7f6720e00640 1 --2- 172.21.15.47:0/3871679152 >> v2:172.21.15.116:3300/0 conn(0x7f671c158a30 0x7f671c15bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:02.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.983+0000 7f6720e00640 1 -- 172.21.15.47:0/3871679152 >> v1:172.21.15.116:6789/0 conn(0x7f671c071dc0 legacy=0x7f671c072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:02.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.983+0000 7f6720e00640 1 -- 172.21.15.47:0/3871679152 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c0cf780 con 0x7f671c06df30 2024-02-19T15:27:02.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.987+0000 7f6720e00640 1 --2- 172.21.15.47:0/3871679152 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f671c06df30 0x7f671c06e390 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f671c0b6110 tx=0x7f670801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5270ac5d7cc706b0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:02.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.987+0000 7f671be00640 1 -- 172.21.15.47:0/3871679152 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67080aa650 con 0x7f671c06df30 2024-02-19T15:27:02.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.987+0000 7f671be00640 1 -- 172.21.15.47:0/3871679152 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67080aa7b0 con 0x7f671c06df30 2024-02-19T15:27:02.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.987+0000 7f671be00640 1 -- 172.21.15.47:0/3871679152 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67080aab40 con 0x7f671c06df30 2024-02-19T15:27:02.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.987+0000 7f6724200640 1 -- 172.21.15.47:0/3871679152 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f671c06df30 msgr2=0x7f671c06e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:02.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.987+0000 7f6724200640 1 --2- 172.21.15.47:0/3871679152 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f671c06df30 0x7f671c06e390 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f671c0b6110 tx=0x7f670801c4f0 comp rx=0 tx=0).stop 2024-02-19T15:27:02.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.987+0000 7f6724200640 1 -- 172.21.15.47:0/3871679152 shutdown_connections 2024-02-19T15:27:02.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.987+0000 7f6724200640 1 --2- 172.21.15.47:0/3871679152 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f671c06df30 0x7f671c06e390 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:02.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.987+0000 7f6724200640 1 --2- 172.21.15.47:0/3871679152 >> v2:172.21.15.116:3300/0 conn(0x7f671c158a30 0x7f671c15bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:02.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.987+0000 7f6724200640 1 -- 172.21.15.47:0/3871679152 >> 172.21.15.47:0/3871679152 conn(0x7f671c0b1b80 msgr2=0x7f671c0b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:02.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.987+0000 7f6724200640 1 -- 172.21.15.47:0/3871679152 shutdown_connections 2024-02-19T15:27:02.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.987+0000 7f6724200640 1 -- 172.21.15.47:0/3871679152 wait complete. 2024-02-19T15:27:02.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.987+0000 7f6724200640 1 Processor -- start 2024-02-19T15:27:02.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.987+0000 7f6724200640 1 -- start start 2024-02-19T15:27:02.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6724200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f671c06df30 0x7f671c136850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:02.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6724200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f671c071dc0 0x7f671c0b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:02.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6724200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f671c158a30 0x7f671c133e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:02.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6724200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c14fa60 con 0x7f671c158a30 2024-02-19T15:27:02.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6724200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c1511c0 con 0x7f671c06df30 2024-02-19T15:27:02.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6724200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f671c1514a0 con 0x7f671c071dc0 2024-02-19T15:27:02.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6722200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f671c158a30 0x7f671c133e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:02.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6721800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f671c06df30 0x7f671c136850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:02.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6721800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f671c06df30 0x7f671c136850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:60610/0 (socket says 172.21.15.47:60610) 2024-02-19T15:27:02.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6722200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f671c158a30 0x7f671c133e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:38558/0 (socket says 172.21.15.47:38558) 2024-02-19T15:27:02.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6721800640 1 -- 172.21.15.47:0/3374861562 learned_addr learned my addr 172.21.15.47:0/3374861562 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:02.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6720e00640 1 --2- 172.21.15.47:0/3374861562 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f671c071dc0 0x7f671c0b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:02.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6721800640 1 -- 172.21.15.47:0/3374861562 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f671c071dc0 msgr2=0x7f671c0b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:02.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6721800640 1 --2- 172.21.15.47:0/3374861562 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f671c071dc0 0x7f671c0b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:02.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6721800640 1 -- 172.21.15.47:0/3374861562 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f671c158a30 msgr2=0x7f671c133e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:02.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6721800640 1 --2- 172.21.15.47:0/3374861562 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f671c158a30 0x7f671c133e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:02.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6721800640 1 -- 172.21.15.47:0/3374861562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67080094d0 con 0x7f671c06df30 2024-02-19T15:27:02.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6721800640 1 --2- 172.21.15.47:0/3374861562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f671c06df30 0x7f671c136850 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f671000e930 tx=0x7f671000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:02.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6719600640 1 -- 172.21.15.47:0/3374861562 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6710009690 con 0x7f671c06df30 2024-02-19T15:27:02.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6719600640 1 -- 172.21.15.47:0/3374861562 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f671000ca00 con 0x7f671c06df30 2024-02-19T15:27:02.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6724200640 1 -- 172.21.15.47:0/3374861562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671c151780 con 0x7f671c06df30 2024-02-19T15:27:02.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6719600640 1 -- 172.21.15.47:0/3374861562 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671000cd70 con 0x7f671c06df30 2024-02-19T15:27:02.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6724200640 1 -- 172.21.15.47:0/3374861562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671c136d90 con 0x7f671c06df30 2024-02-19T15:27:02.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.991+0000 7f6724200640 1 -- 172.21.15.47:0/3374861562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f671c0b3420 con 0x7f671c06df30 2024-02-19T15:27:02.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.995+0000 7f6719600640 1 -- 172.21.15.47:0/3374861562 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6710013470 con 0x7f671c06df30 2024-02-19T15:27:02.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.995+0000 7f6719600640 1 --2- 172.21.15.47:0/3374861562 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f66fc066e40 0x7f66fc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:02.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.995+0000 7f6719600640 1 -- 172.21.15.47:0/3374861562 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f6710084d70 con 0x7f671c06df30 2024-02-19T15:27:02.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.995+0000 7f6720e00640 1 --2- 172.21.15.47:0/3374861562 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f66fc066e40 0x7f66fc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:03.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.995+0000 7f6720e00640 1 --2- 172.21.15.47:0/3374861562 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f66fc066e40 0x7f66fc069300 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f67080ae000 tx=0x7f67080b5040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:03.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:02.999+0000 7f6719600640 1 -- 172.21.15.47:0/3374861562 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6710051b80 con 0x7f671c06df30 2024-02-19T15:27:03.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.163+0000 7f6724200640 1 -- 172.21.15.47:0/3374861562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f671c068100 con 0x7f671c06df30 2024-02-19T15:27:03.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.167+0000 7f6719600640 1 -- 172.21.15.47:0/3374861562 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16875 (secure 0 0 0) 0x7f6710017090 con 0x7f671c06df30 2024-02-19T15:27:03.172 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:03.172 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":97,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:26:40.823963+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:26:25.124594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-02-19T15:26:25.706971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:6833/68555355":"2024-02-20T15:26:40.818800+0000","172.21.15.47:6832/68555355":"2024-02-20T15:26:40.818800+0000","172.21.15.116:6837/547176070":"2024-02-20T15:26:39.960651+0000","172.21.15.116:6835/547176070":"2024-02-20T15:26:39.960651+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:27:03.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.167+0000 7f6724200640 1 -- 172.21.15.47:0/3374861562 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f66fc066e40 msgr2=0x7f66fc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:03.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.167+0000 7f6724200640 1 --2- 172.21.15.47:0/3374861562 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f66fc066e40 0x7f66fc069300 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f67080ae000 tx=0x7f67080b5040 comp rx=0 tx=0).stop 2024-02-19T15:27:03.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.167+0000 7f6724200640 1 -- 172.21.15.47:0/3374861562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f671c06df30 msgr2=0x7f671c136850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:03.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.167+0000 7f6724200640 1 --2- 172.21.15.47:0/3374861562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f671c06df30 0x7f671c136850 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f671000e930 tx=0x7f671000edf0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.167+0000 7f6724200640 1 -- 172.21.15.47:0/3374861562 shutdown_connections 2024-02-19T15:27:03.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.167+0000 7f6724200640 1 --2- 172.21.15.47:0/3374861562 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f66fc066e40 0x7f66fc069300 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.167+0000 7f6724200640 1 --2- 172.21.15.47:0/3374861562 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f671c158a30 0x7f671c133e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.167+0000 7f6724200640 1 --2- 172.21.15.47:0/3374861562 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f671c071dc0 0x7f671c0b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.167+0000 7f6724200640 1 --2- 172.21.15.47:0/3374861562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f671c06df30 0x7f671c136850 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.167+0000 7f6724200640 1 -- 172.21.15.47:0/3374861562 >> 172.21.15.47:0/3374861562 conn(0x7f671c0b1b80 msgr2=0x7f671c155670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:03.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.167+0000 7f6724200640 1 -- 172.21.15.47:0/3374861562 shutdown_connections 2024-02-19T15:27:03.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.167+0000 7f6724200640 1 -- 172.21.15.47:0/3374861562 wait complete. 2024-02-19T15:27:03.193 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:26:25.124594+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:26:25.706971+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:27:03.193 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:27:03.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.331+0000 7f0a3a000640 1 Processor -- start 2024-02-19T15:27:03.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.331+0000 7f0a3a000640 1 -- start start 2024-02-19T15:27:03.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.335+0000 7f0a3a000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a34145790 0x7f0a340728f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:03.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.335+0000 7f0a3a000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0a340b5190 0x7f0a340b5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:03.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.335+0000 7f0a3a000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a3415fda0 con 0x7f0a340cf760 2024-02-19T15:27:03.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.335+0000 7f0a3a000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a341609b0 con 0x7f0a340b5190 2024-02-19T15:27:03.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.335+0000 7f0a3a000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a340b5b40 con 0x7f0a34145790 2024-02-19T15:27:03.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.335+0000 7f0a33400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a34145790 0x7f0a340728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:03.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.335+0000 7f0a33e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0a340b5190 0x7f0a340b5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:03.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.335+0000 7f0a33400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a34145790 0x7f0a340728f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:46860/0 (socket says 172.21.15.47:46860) 2024-02-19T15:27:03.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.335+0000 7f0a33400640 1 -- 172.21.15.47:0/3156921726 learned_addr learned my addr 172.21.15.47:0/3156921726 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:03.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.335+0000 7f0a33400640 1 -- 172.21.15.47:0/3156921726 >> v2:172.21.15.116:3300/0 conn(0x7f0a340b5190 msgr2=0x7f0a340b5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:03.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.335+0000 7f0a33400640 1 --2- 172.21.15.47:0/3156921726 >> v2:172.21.15.116:3300/0 conn(0x7f0a340b5190 0x7f0a340b5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.335+0000 7f0a33400640 1 -- 172.21.15.47:0/3156921726 >> v1:172.21.15.116:6789/0 conn(0x7f0a340cf760 legacy=0x7f0a3415c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:03.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.335+0000 7f0a33400640 1 -- 172.21.15.47:0/3156921726 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a34160080 con 0x7f0a34145790 2024-02-19T15:27:03.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.335+0000 7f0a33400640 1 --2- 172.21.15.47:0/3156921726 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a34145790 0x7f0a340728f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0a24009870 tx=0x7f0a2401c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=919913ea6b73222d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:03.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a32a00640 1 -- 172.21.15.47:0/3156921726 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a240ac620 con 0x7f0a34145790 2024-02-19T15:27:03.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a32a00640 1 -- 172.21.15.47:0/3156921726 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a240ac780 con 0x7f0a34145790 2024-02-19T15:27:03.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a32a00640 1 -- 172.21.15.47:0/3156921726 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a240acb10 con 0x7f0a34145790 2024-02-19T15:27:03.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a3a000640 1 -- 172.21.15.47:0/3156921726 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a34145790 msgr2=0x7f0a340728f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:03.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a3a000640 1 --2- 172.21.15.47:0/3156921726 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a34145790 0x7f0a340728f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0a24009870 tx=0x7f0a2401c4d0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a3a000640 1 -- 172.21.15.47:0/3156921726 shutdown_connections 2024-02-19T15:27:03.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a3a000640 1 --2- 172.21.15.47:0/3156921726 >> v2:172.21.15.116:3300/0 conn(0x7f0a340b5190 0x7f0a340b5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a3a000640 1 --2- 172.21.15.47:0/3156921726 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a34145790 0x7f0a340728f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a3a000640 1 -- 172.21.15.47:0/3156921726 >> 172.21.15.47:0/3156921726 conn(0x7f0a340b1b80 msgr2=0x7f0a340b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:03.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a3a000640 1 -- 172.21.15.47:0/3156921726 shutdown_connections 2024-02-19T15:27:03.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a3a000640 1 -- 172.21.15.47:0/3156921726 wait complete. 2024-02-19T15:27:03.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a3a000640 1 Processor -- start 2024-02-19T15:27:03.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a3a000640 1 -- start start 2024-02-19T15:27:03.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a3a000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a340b5190 0x7f0a341395e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:03.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a3a000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a340cf760 0x7f0a34137750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:03.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a3a000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a34145790 0x7f0a34142ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:03.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a3a000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a34159040 con 0x7f0a34145790 2024-02-19T15:27:03.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a3a000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a34139b20 con 0x7f0a340cf760 2024-02-19T15:27:03.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a3a000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a340afc20 con 0x7f0a340b5190 2024-02-19T15:27:03.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a33e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a34145790 0x7f0a34142ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:03.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a33400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a340b5190 0x7f0a341395e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:03.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a2aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a340cf760 0x7f0a34137750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:03.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a33400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a340b5190 0x7f0a341395e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:40682/0 (socket says 172.21.15.47:40682) 2024-02-19T15:27:03.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a33400640 1 -- 172.21.15.47:0/2057517766 learned_addr learned my addr 172.21.15.47:0/2057517766 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:03.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a33400640 1 -- 172.21.15.47:0/2057517766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a340cf760 msgr2=0x7f0a34137750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:03.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a33400640 1 --2- 172.21.15.47:0/2057517766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a340cf760 0x7f0a34137750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a33400640 1 -- 172.21.15.47:0/2057517766 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a34145790 msgr2=0x7f0a34142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:03.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.339+0000 7f0a33400640 1 --2- 172.21.15.47:0/2057517766 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a34145790 0x7f0a34142ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.343+0000 7f0a33400640 1 -- 172.21.15.47:0/2057517766 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a240094d0 con 0x7f0a340b5190 2024-02-19T15:27:03.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.343+0000 7f0a33400640 1 --2- 172.21.15.47:0/2057517766 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a340b5190 0x7f0a341395e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0a240af040 tx=0x7f0a24019270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:03.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.343+0000 7f0a2be00640 1 -- 172.21.15.47:0/2057517766 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a240b8450 con 0x7f0a340b5190 2024-02-19T15:27:03.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.343+0000 7f0a2be00640 1 -- 172.21.15.47:0/2057517766 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a240b85b0 con 0x7f0a340b5190 2024-02-19T15:27:03.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.343+0000 7f0a3a000640 1 -- 172.21.15.47:0/2057517766 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a340aff00 con 0x7f0a340b5190 2024-02-19T15:27:03.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.343+0000 7f0a2be00640 1 -- 172.21.15.47:0/2057517766 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a240b8940 con 0x7f0a340b5190 2024-02-19T15:27:03.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.343+0000 7f0a3a000640 1 -- 172.21.15.47:0/2057517766 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a340b04c0 con 0x7f0a340b5190 2024-02-19T15:27:03.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.343+0000 7f0a3a000640 1 -- 172.21.15.47:0/2057517766 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a34068120 con 0x7f0a340b5190 2024-02-19T15:27:03.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.343+0000 7f0a2be00640 1 -- 172.21.15.47:0/2057517766 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0a240aa070 con 0x7f0a340b5190 2024-02-19T15:27:03.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.343+0000 7f0a2be00640 1 --2- 172.21.15.47:0/2057517766 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a10066e40 0x7f0a10069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:03.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.343+0000 7f0a2be00640 1 -- 172.21.15.47:0/2057517766 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f0a2411efe0 con 0x7f0a340b5190 2024-02-19T15:27:03.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.343+0000 7f0a2aa00640 1 --2- 172.21.15.47:0/2057517766 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a10066e40 0x7f0a10069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:03.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.347+0000 7f0a2aa00640 1 --2- 172.21.15.47:0/2057517766 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a10066e40 0x7f0a10069300 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0a340b1790 tx=0x7f0a1c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:03.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.347+0000 7f0a2be00640 1 -- 172.21.15.47:0/2057517766 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0a240ebdf0 con 0x7f0a340b5190 2024-02-19T15:27:03.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.511+0000 7f0a3a000640 1 -- 172.21.15.47:0/2057517766 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0a340afd60 con 0x7f0a340b5190 2024-02-19T15:27:03.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.511+0000 7f0a2be00640 1 -- 172.21.15.47:0/2057517766 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0a340afd60 con 0x7f0a340b5190 2024-02-19T15:27:03.518 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:27:03.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.515+0000 7f0a3a000640 1 -- 172.21.15.47:0/2057517766 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a10066e40 msgr2=0x7f0a10069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:03.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.515+0000 7f0a3a000640 1 --2- 172.21.15.47:0/2057517766 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a10066e40 0x7f0a10069300 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0a340b1790 tx=0x7f0a1c009040 comp rx=0 tx=0).stop 2024-02-19T15:27:03.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.515+0000 7f0a3a000640 1 -- 172.21.15.47:0/2057517766 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a340b5190 msgr2=0x7f0a341395e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:03.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.515+0000 7f0a3a000640 1 --2- 172.21.15.47:0/2057517766 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a340b5190 0x7f0a341395e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0a240af040 tx=0x7f0a24019270 comp rx=0 tx=0).stop 2024-02-19T15:27:03.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.515+0000 7f0a3a000640 1 -- 172.21.15.47:0/2057517766 shutdown_connections 2024-02-19T15:27:03.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.515+0000 7f0a3a000640 1 --2- 172.21.15.47:0/2057517766 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0a10066e40 0x7f0a10069300 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.515+0000 7f0a3a000640 1 --2- 172.21.15.47:0/2057517766 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a34145790 0x7f0a34142ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.515+0000 7f0a3a000640 1 --2- 172.21.15.47:0/2057517766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a340cf760 0x7f0a34137750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.515+0000 7f0a3a000640 1 --2- 172.21.15.47:0/2057517766 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a340b5190 0x7f0a341395e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.515+0000 7f0a3a000640 1 -- 172.21.15.47:0/2057517766 >> 172.21.15.47:0/2057517766 conn(0x7f0a340b1b80 msgr2=0x7f0a340b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:03.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.515+0000 7f0a3a000640 1 -- 172.21.15.47:0/2057517766 shutdown_connections 2024-02-19T15:27:03.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.515+0000 7f0a3a000640 1 -- 172.21.15.47:0/2057517766 wait complete. 2024-02-19T15:27:03.538 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:27:03.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.675+0000 7f9b57000640 1 Processor -- start 2024-02-19T15:27:03.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.675+0000 7f9b57000640 1 -- start start 2024-02-19T15:27:03.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.679+0000 7f9b57000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b501599b0 0x7f9b500b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:03.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.679+0000 7f9b57000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9b500b5050 0x7f9b500b5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:03.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.679+0000 7f9b57000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9b500b5b60 0x7f9b5015faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:03.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.679+0000 7f9b57000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b500b5fb0 con 0x7f9b500b5b60 2024-02-19T15:27:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.679+0000 7f9b57000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b500b0140 con 0x7f9b500b5050 2024-02-19T15:27:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.679+0000 7f9b57000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5015d480 con 0x7f9b501599b0 2024-02-19T15:27:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.679+0000 7f9b56600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9b500b5b60 0x7f9b5015faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.679+0000 7f9b55c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b501599b0 0x7f9b500b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.679+0000 7f9b55200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9b500b5050 0x7f9b500b5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.679+0000 7f9b56600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9b500b5b60 0x7f9b5015faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56380/0 (socket says 172.21.15.47:56380) 2024-02-19T15:27:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.679+0000 7f9b56600640 1 -- 172.21.15.47:0/4036290868 learned_addr learned my addr 172.21.15.47:0/4036290868 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.679+0000 7f9b56600640 1 -- 172.21.15.47:0/4036290868 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b501599b0 msgr2=0x7f9b500b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:03.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.679+0000 7f9b56600640 1 --2- 172.21.15.47:0/4036290868 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b501599b0 0x7f9b500b4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.679+0000 7f9b56600640 1 -- 172.21.15.47:0/4036290868 >> v2:172.21.15.116:3300/0 conn(0x7f9b500b5050 msgr2=0x7f9b500b5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:03.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.679+0000 7f9b56600640 1 --2- 172.21.15.47:0/4036290868 >> v2:172.21.15.116:3300/0 conn(0x7f9b500b5050 0x7f9b500b5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.679+0000 7f9b56600640 1 -- 172.21.15.47:0/4036290868 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b500af290 con 0x7f9b500b5b60 2024-02-19T15:27:03.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.679+0000 7f9b56600640 1 --2- 172.21.15.47:0/4036290868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b500b5b60 0x7f9b5015faa0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f9b5006bef0 tx=0x7f9b4801c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f52a9374a7c70aca server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:03.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b4fe00640 1 -- 172.21.15.47:0/4036290868 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b480a5580 con 0x7f9b500b5b60 2024-02-19T15:27:03.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b4fe00640 1 -- 172.21.15.47:0/4036290868 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b480a56e0 con 0x7f9b500b5b60 2024-02-19T15:27:03.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b4fe00640 1 -- 172.21.15.47:0/4036290868 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b480a5a70 con 0x7f9b500b5b60 2024-02-19T15:27:03.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b57000640 1 -- 172.21.15.47:0/4036290868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b500b5b60 msgr2=0x7f9b5015faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:03.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b57000640 1 --2- 172.21.15.47:0/4036290868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b500b5b60 0x7f9b5015faa0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f9b5006bef0 tx=0x7f9b4801c700 comp rx=0 tx=0).stop 2024-02-19T15:27:03.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b57000640 1 -- 172.21.15.47:0/4036290868 shutdown_connections 2024-02-19T15:27:03.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b57000640 1 --2- 172.21.15.47:0/4036290868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b500b5b60 0x7f9b5015faa0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b57000640 1 --2- 172.21.15.47:0/4036290868 >> v2:172.21.15.116:3300/0 conn(0x7f9b500b5050 0x7f9b500b5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b57000640 1 --2- 172.21.15.47:0/4036290868 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b501599b0 0x7f9b500b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b57000640 1 -- 172.21.15.47:0/4036290868 >> 172.21.15.47:0/4036290868 conn(0x7f9b5009aa70 msgr2=0x7f9b5009ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:03.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b57000640 1 -- 172.21.15.47:0/4036290868 shutdown_connections 2024-02-19T15:27:03.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b57000640 1 -- 172.21.15.47:0/4036290868 wait complete. 2024-02-19T15:27:03.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b57000640 1 Processor -- start 2024-02-19T15:27:03.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b57000640 1 -- start start 2024-02-19T15:27:03.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b57000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b500b5050 0x7f9b5013a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:03.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b57000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b500b5b60 0x7f9b5014c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:03.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b57000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b501599b0 0x7f9b5013af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:03.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b57000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5014b7d0 con 0x7f9b500b5050 2024-02-19T15:27:03.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b57000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5014bab0 con 0x7f9b501599b0 2024-02-19T15:27:03.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b57000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5014bd90 con 0x7f9b500b5b60 2024-02-19T15:27:03.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b55c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b500b5050 0x7f9b5013a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:03.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b55c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b500b5050 0x7f9b5013a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56388/0 (socket says 172.21.15.47:56388) 2024-02-19T15:27:03.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b55c00640 1 -- 172.21.15.47:0/974808325 learned_addr learned my addr 172.21.15.47:0/974808325 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:03.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b55200640 1 --2- 172.21.15.47:0/974808325 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b500b5b60 0x7f9b5014c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:03.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.683+0000 7f9b56600640 1 --2- 172.21.15.47:0/974808325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b501599b0 0x7f9b5013af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:03.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.687+0000 7f9b55c00640 1 -- 172.21.15.47:0/974808325 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b500b5b60 msgr2=0x7f9b5014c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:03.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.687+0000 7f9b55c00640 1 --2- 172.21.15.47:0/974808325 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b500b5b60 0x7f9b5014c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.687+0000 7f9b55c00640 1 -- 172.21.15.47:0/974808325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b501599b0 msgr2=0x7f9b5013af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:03.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.687+0000 7f9b55c00640 1 --2- 172.21.15.47:0/974808325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b501599b0 0x7f9b5013af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.687+0000 7f9b55c00640 1 -- 172.21.15.47:0/974808325 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b48009b30 con 0x7f9b500b5050 2024-02-19T15:27:03.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.687+0000 7f9b55c00640 1 --2- 172.21.15.47:0/974808325 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b500b5050 0x7f9b5013a730 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f9b4400e930 tx=0x7f9b4400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:03.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.687+0000 7f9b4d600640 1 -- 172.21.15.47:0/974808325 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b44009690 con 0x7f9b500b5050 2024-02-19T15:27:03.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.687+0000 7f9b4d600640 1 -- 172.21.15.47:0/974808325 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b4400ca00 con 0x7f9b500b5050 2024-02-19T15:27:03.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.687+0000 7f9b4d600640 1 -- 172.21.15.47:0/974808325 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4400cd70 con 0x7f9b500b5050 2024-02-19T15:27:03.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.687+0000 7f9b57000640 1 -- 172.21.15.47:0/974808325 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b5014c070 con 0x7f9b500b5050 2024-02-19T15:27:03.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.687+0000 7f9b57000640 1 -- 172.21.15.47:0/974808325 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b5014b340 con 0x7f9b500b5050 2024-02-19T15:27:03.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.687+0000 7f9b57000640 1 -- 172.21.15.47:0/974808325 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b5014b590 con 0x7f9b500b5050 2024-02-19T15:27:03.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.687+0000 7f9b4d600640 1 -- 172.21.15.47:0/974808325 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9b44013470 con 0x7f9b500b5050 2024-02-19T15:27:03.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.687+0000 7f9b4d600640 1 --2- 172.21.15.47:0/974808325 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9b30066e40 0x7f9b30069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:03.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.687+0000 7f9b4d600640 1 -- 172.21.15.47:0/974808325 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f9b44084d60 con 0x7f9b500b5050 2024-02-19T15:27:03.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.691+0000 7f9b55200640 1 --2- 172.21.15.47:0/974808325 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9b30066e40 0x7f9b30069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:03.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.691+0000 7f9b55200640 1 --2- 172.21.15.47:0/974808325 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9b30066e40 0x7f9b30069300 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f9b50153ae0 tx=0x7f9b3c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:03.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.691+0000 7f9b4d600640 1 -- 172.21.15.47:0/974808325 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f9b5014b590 con 0x7f9b500b5050 2024-02-19T15:27:03.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.851+0000 7f9b57000640 1 -- 172.21.15.47:0/974808325 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9b5014b590 con 0x7f9b500b5050 2024-02-19T15:27:03.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.855+0000 7f9b4d600640 1 -- 172.21.15.47:0/974808325 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9b44017020 con 0x7f9b500b5050 2024-02-19T15:27:03.858 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:27:03.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.855+0000 7f9b57000640 1 -- 172.21.15.47:0/974808325 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9b30066e40 msgr2=0x7f9b30069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:03.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.855+0000 7f9b57000640 1 --2- 172.21.15.47:0/974808325 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9b30066e40 0x7f9b30069300 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f9b50153ae0 tx=0x7f9b3c009190 comp rx=0 tx=0).stop 2024-02-19T15:27:03.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.855+0000 7f9b57000640 1 -- 172.21.15.47:0/974808325 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b500b5050 msgr2=0x7f9b5013a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:03.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.855+0000 7f9b57000640 1 --2- 172.21.15.47:0/974808325 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b500b5050 0x7f9b5013a730 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f9b4400e930 tx=0x7f9b4400edf0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.855+0000 7f9b57000640 1 -- 172.21.15.47:0/974808325 shutdown_connections 2024-02-19T15:27:03.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.855+0000 7f9b57000640 1 --2- 172.21.15.47:0/974808325 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9b30066e40 0x7f9b30069300 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.855+0000 7f9b57000640 1 --2- 172.21.15.47:0/974808325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b501599b0 0x7f9b5013af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.855+0000 7f9b57000640 1 --2- 172.21.15.47:0/974808325 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b500b5b60 0x7f9b5014c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.855+0000 7f9b57000640 1 --2- 172.21.15.47:0/974808325 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b500b5050 0x7f9b5013a730 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:03.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.855+0000 7f9b57000640 1 -- 172.21.15.47:0/974808325 >> 172.21.15.47:0/974808325 conn(0x7f9b5009aa70 msgr2=0x7f9b5015a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:03.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.855+0000 7f9b57000640 1 -- 172.21.15.47:0/974808325 shutdown_connections 2024-02-19T15:27:03.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:03.855+0000 7f9b57000640 1 -- 172.21.15.47:0/974808325 wait complete. 2024-02-19T15:27:03.878 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:27:04.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.015+0000 7ff0d6800640 1 Processor -- start 2024-02-19T15:27:04.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.019+0000 7ff0d6800640 1 -- start start 2024-02-19T15:27:04.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.019+0000 7ff0d6800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0d006b620 0x7ff0d0169050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:04.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.019+0000 7ff0d6800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d00b4fb0 con 0x7ff0d015ec60 2024-02-19T15:27:04.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.019+0000 7ff0d6800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d00b4710 con 0x7ff0d00b3f70 2024-02-19T15:27:04.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.019+0000 7ff0d6800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d00b49f0 con 0x7ff0d006b620 2024-02-19T15:27:04.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.019+0000 7ff0cf400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff0d015ec60 0x7ff0d015f040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58494/0 (socket says 172.21.15.47:58494) 2024-02-19T15:27:04.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.019+0000 7ff0cf400640 1 -- 172.21.15.47:0/1742414251 learned_addr learned my addr 172.21.15.47:0/1742414251 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:04.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.019+0000 7ff0cfe00640 1 --2- 172.21.15.47:0/1742414251 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0d006b620 0x7ff0d0169050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:04.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.019+0000 7ff0cea00640 1 -- 172.21.15.47:0/1742414251 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174916013 0 0) 0x7ff0d00b4fb0 con 0x7ff0d015ec60 2024-02-19T15:27:04.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.019+0000 7ff0cea00640 1 -- 172.21.15.47:0/1742414251 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0b4003530 con 0x7ff0d015ec60 2024-02-19T15:27:04.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.019+0000 7ff0cea00640 1 -- 172.21.15.47:0/1742414251 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036488839 0 0) 0x7ff0d00b4710 con 0x7ff0d00b3f70 2024-02-19T15:27:04.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.019+0000 7ff0cea00640 1 -- 172.21.15.47:0/1742414251 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d00b4fb0 con 0x7ff0d00b3f70 2024-02-19T15:27:04.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.019+0000 7ff0cfe00640 1 -- 172.21.15.47:0/1742414251 >> v1:172.21.15.116:6789/0 conn(0x7ff0d00b3f70 legacy=0x7ff0d006af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:04.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.019+0000 7ff0cfe00640 1 -- 172.21.15.47:0/1742414251 >> v1:172.21.15.47:6789/0 conn(0x7ff0d015ec60 legacy=0x7ff0d015f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:04.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.019+0000 7ff0cfe00640 1 -- 172.21.15.47:0/1742414251 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d00b4cd0 con 0x7ff0d006b620 2024-02-19T15:27:04.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.023+0000 7ff0cfe00640 1 --2- 172.21.15.47:0/1742414251 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0d006b620 0x7ff0d0169050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8009dc0 tx=0x7ff0c801c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb3dc3f01b2236cf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:04.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.023+0000 7ff0cea00640 1 -- 172.21.15.47:0/1742414251 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c80a5920 con 0x7ff0d006b620 2024-02-19T15:27:04.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.023+0000 7ff0cea00640 1 -- 172.21.15.47:0/1742414251 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c80a5a80 con 0x7ff0d006b620 2024-02-19T15:27:04.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.023+0000 7ff0cea00640 1 -- 172.21.15.47:0/1742414251 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c80a5e10 con 0x7ff0d006b620 2024-02-19T15:27:04.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.023+0000 7ff0d6800640 1 -- 172.21.15.47:0/1742414251 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0d006b620 msgr2=0x7ff0d0169050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:04.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.023+0000 7ff0d6800640 1 --2- 172.21.15.47:0/1742414251 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0d006b620 0x7ff0d0169050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8009dc0 tx=0x7ff0c801c900 comp rx=0 tx=0).stop 2024-02-19T15:27:04.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.023+0000 7ff0d6800640 1 -- 172.21.15.47:0/1742414251 shutdown_connections 2024-02-19T15:27:04.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.023+0000 7ff0d6800640 1 --2- 172.21.15.47:0/1742414251 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0d006b620 0x7ff0d0169050 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.023+0000 7ff0d6800640 1 -- 172.21.15.47:0/1742414251 >> 172.21.15.47:0/1742414251 conn(0x7ff0d006c9b0 msgr2=0x7ff0d00b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:04.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.023+0000 7ff0d6800640 1 -- 172.21.15.47:0/1742414251 shutdown_connections 2024-02-19T15:27:04.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.023+0000 7ff0d6800640 1 -- 172.21.15.47:0/1742414251 wait complete. 2024-02-19T15:27:04.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.023+0000 7ff0d6800640 1 Processor -- start 2024-02-19T15:27:04.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.023+0000 7ff0d6800640 1 -- start start 2024-02-19T15:27:04.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0d6800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d006b620 0x7ff0d014bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:04.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0d6800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0d00b3f70 0x7ff0d0135130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:04.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0d6800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0d015ec60 0x7ff0d014b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:04.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0d6800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d0135670 con 0x7ff0d015ec60 2024-02-19T15:27:04.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0d6800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d0137a10 con 0x7ff0d006b620 2024-02-19T15:27:04.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0d6800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d0137cf0 con 0x7ff0d00b3f70 2024-02-19T15:27:04.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0cfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0d015ec60 0x7ff0d014b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0cfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0d015ec60 0x7ff0d014b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56402/0 (socket says 172.21.15.47:56402) 2024-02-19T15:27:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0cfe00640 1 -- 172.21.15.47:0/4226289217 learned_addr learned my addr 172.21.15.47:0/4226289217 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0cf400640 1 --2- 172.21.15.47:0/4226289217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d006b620 0x7ff0d014bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0c6a00640 1 --2- 172.21.15.47:0/4226289217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0d00b3f70 0x7ff0d0135130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0cfe00640 1 -- 172.21.15.47:0/4226289217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0d00b3f70 msgr2=0x7ff0d0135130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0cfe00640 1 --2- 172.21.15.47:0/4226289217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0d00b3f70 0x7ff0d0135130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0cfe00640 1 -- 172.21.15.47:0/4226289217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d006b620 msgr2=0x7ff0d014bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0cfe00640 1 --2- 172.21.15.47:0/4226289217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d006b620 0x7ff0d014bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0cfe00640 1 -- 172.21.15.47:0/4226289217 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c8009a70 con 0x7ff0d015ec60 2024-02-19T15:27:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0cfe00640 1 --2- 172.21.15.47:0/4226289217 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0d015ec60 0x7ff0d014b020 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7ff0c801a530 tx=0x7ff0c80196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0c7e00640 1 -- 172.21.15.47:0/4226289217 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c80aec30 con 0x7ff0d015ec60 2024-02-19T15:27:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0c7e00640 1 -- 172.21.15.47:0/4226289217 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c80aedb0 con 0x7ff0d015ec60 2024-02-19T15:27:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0d6800640 1 -- 172.21.15.47:0/4226289217 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d0137fd0 con 0x7ff0d015ec60 2024-02-19T15:27:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0c7e00640 1 -- 172.21.15.47:0/4226289217 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c80b7650 con 0x7ff0d015ec60 2024-02-19T15:27:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0d6800640 1 -- 172.21.15.47:0/4226289217 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d014b560 con 0x7ff0d015ec60 2024-02-19T15:27:04.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0d6800640 1 -- 172.21.15.47:0/4226289217 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d014e7c0 con 0x7ff0d015ec60 2024-02-19T15:27:04.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.027+0000 7ff0c7e00640 1 -- 172.21.15.47:0/4226289217 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff0c80b4050 con 0x7ff0d015ec60 2024-02-19T15:27:04.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.031+0000 7ff0c7e00640 1 --2- 172.21.15.47:0/4226289217 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff0ac066e40 0x7ff0ac069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:04.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.031+0000 7ff0c7e00640 1 -- 172.21.15.47:0/4226289217 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7ff0c811fc80 con 0x7ff0d015ec60 2024-02-19T15:27:04.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.031+0000 7ff0cf400640 1 --2- 172.21.15.47:0/4226289217 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff0ac066e40 0x7ff0ac069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:04.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.031+0000 7ff0cf400640 1 --2- 172.21.15.47:0/4226289217 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff0ac066e40 0x7ff0ac069300 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0000f70 tx=0x7ff0c0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:04.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.035+0000 7ff0c7e00640 1 -- 172.21.15.47:0/4226289217 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff0d014e7c0 con 0x7ff0d015ec60 2024-02-19T15:27:04.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.195+0000 7ff0d6800640 1 -- 172.21.15.47:0/4226289217 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff0d014e7c0 con 0x7ff0d015ec60 2024-02-19T15:27:04.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.199+0000 7ff0c7e00640 1 -- 172.21.15.47:0/4226289217 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff0d014e7c0 con 0x7ff0d015ec60 2024-02-19T15:27:04.202 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:27:04.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.199+0000 7ff0d6800640 1 -- 172.21.15.47:0/4226289217 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff0ac066e40 msgr2=0x7ff0ac069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:04.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.199+0000 7ff0d6800640 1 --2- 172.21.15.47:0/4226289217 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff0ac066e40 0x7ff0ac069300 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0000f70 tx=0x7ff0c0009040 comp rx=0 tx=0).stop 2024-02-19T15:27:04.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.199+0000 7ff0d6800640 1 -- 172.21.15.47:0/4226289217 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0d015ec60 msgr2=0x7ff0d014b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:04.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.199+0000 7ff0d6800640 1 --2- 172.21.15.47:0/4226289217 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0d015ec60 0x7ff0d014b020 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7ff0c801a530 tx=0x7ff0c80196a0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.199+0000 7ff0d6800640 1 -- 172.21.15.47:0/4226289217 shutdown_connections 2024-02-19T15:27:04.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.199+0000 7ff0d6800640 1 --2- 172.21.15.47:0/4226289217 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff0ac066e40 0x7ff0ac069300 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.199+0000 7ff0d6800640 1 --2- 172.21.15.47:0/4226289217 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0d015ec60 0x7ff0d014b020 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.199+0000 7ff0d6800640 1 --2- 172.21.15.47:0/4226289217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0d00b3f70 0x7ff0d0135130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.199+0000 7ff0d6800640 1 --2- 172.21.15.47:0/4226289217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d006b620 0x7ff0d014bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.199+0000 7ff0d6800640 1 -- 172.21.15.47:0/4226289217 >> 172.21.15.47:0/4226289217 conn(0x7ff0d006c9b0 msgr2=0x7ff0d015f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:04.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.199+0000 7ff0d6800640 1 -- 172.21.15.47:0/4226289217 shutdown_connections 2024-02-19T15:27:04.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.199+0000 7ff0d6800640 1 -- 172.21.15.47:0/4226289217 wait complete. 2024-02-19T15:27:04.224 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-19T15:27:04.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.359+0000 7f275d200640 1 Processor -- start 2024-02-19T15:27:04.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.363+0000 7f275d200640 1 -- start start 2024-02-19T15:27:04.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.363+0000 7f275d200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f275815e080 0x7f27580b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.363+0000 7f275d200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27580ad060 0x7f27580af1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.363+0000 7f275d200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27580ad590 con 0x7f275815e460 2024-02-19T15:27:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.363+0000 7f275d200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27580ae210 con 0x7f275815e080 2024-02-19T15:27:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.363+0000 7f275d200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27580ad870 con 0x7f27580ad060 2024-02-19T15:27:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.363+0000 7f2755e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f275815e080 0x7f27580b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.363+0000 7f2755e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f275815e080 0x7f27580b4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:39988/0 (socket says 172.21.15.47:39988) 2024-02-19T15:27:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.363+0000 7f2755e00640 1 -- 172.21.15.47:0/957599132 learned_addr learned my addr 172.21.15.47:0/957599132 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.363+0000 7f2756800640 1 --2- 172.21.15.47:0/957599132 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27580ad060 0x7f27580af1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.363+0000 7f274fe00640 1 --1- 172.21.15.47:0/957599132 >> v1:172.21.15.116:6789/0 conn(0x7f275815e460 0x7f27580ae630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:49306/0 (socket says 172.21.15.47:49306) 2024-02-19T15:27:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.363+0000 7f2755e00640 1 -- 172.21.15.47:0/957599132 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27580ad060 msgr2=0x7f27580af1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.363+0000 7f2755e00640 1 --2- 172.21.15.47:0/957599132 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27580ad060 0x7f27580af1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.363+0000 7f2755e00640 1 -- 172.21.15.47:0/957599132 >> v1:172.21.15.116:6789/0 conn(0x7f275815e460 legacy=0x7f27580ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.363+0000 7f2755e00640 1 -- 172.21.15.47:0/957599132 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27580adb50 con 0x7f275815e080 2024-02-19T15:27:04.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.367+0000 7f2755e00640 1 --2- 172.21.15.47:0/957599132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f275815e080 0x7f27580b4f80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2748009870 tx=0x7f274801c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b0b0fa07a3309e1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:04.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.367+0000 7f2755400640 1 -- 172.21.15.47:0/957599132 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27480ac640 con 0x7f275815e080 2024-02-19T15:27:04.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.367+0000 7f2755400640 1 -- 172.21.15.47:0/957599132 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27480ac7a0 con 0x7f275815e080 2024-02-19T15:27:04.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.367+0000 7f2755400640 1 -- 172.21.15.47:0/957599132 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27480acb30 con 0x7f275815e080 2024-02-19T15:27:04.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.367+0000 7f275d200640 1 -- 172.21.15.47:0/957599132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f275815e080 msgr2=0x7f27580b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:04.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.367+0000 7f275d200640 1 --2- 172.21.15.47:0/957599132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f275815e080 0x7f27580b4f80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2748009870 tx=0x7f274801c4d0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.367+0000 7f275d200640 1 -- 172.21.15.47:0/957599132 shutdown_connections 2024-02-19T15:27:04.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.367+0000 7f275d200640 1 --2- 172.21.15.47:0/957599132 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27580ad060 0x7f27580af1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.367+0000 7f275d200640 1 --2- 172.21.15.47:0/957599132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f275815e080 0x7f27580b4f80 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.367+0000 7f275d200640 1 -- 172.21.15.47:0/957599132 >> 172.21.15.47:0/957599132 conn(0x7f27580b5d10 msgr2=0x7f2758072450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:04.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.367+0000 7f275d200640 1 -- 172.21.15.47:0/957599132 shutdown_connections 2024-02-19T15:27:04.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.367+0000 7f275d200640 1 -- 172.21.15.47:0/957599132 wait complete. 2024-02-19T15:27:04.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.367+0000 7f275d200640 1 Processor -- start 2024-02-19T15:27:04.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.367+0000 7f275d200640 1 -- start start 2024-02-19T15:27:04.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f275d200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27580ad060 0x7f2758153080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:04.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f275d200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f275815e080 0x7f275809a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:04.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f275d200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f275815e460 0x7f275815cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:04.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f275d200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2758154b60 con 0x7f27580ad060 2024-02-19T15:27:04.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f275d200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275809c4b0 con 0x7f275815e460 2024-02-19T15:27:04.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f275d200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27581535c0 con 0x7f275815e080 2024-02-19T15:27:04.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f2755e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27580ad060 0x7f2758153080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:04.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f2755e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27580ad060 0x7f2758153080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56408/0 (socket says 172.21.15.47:56408) 2024-02-19T15:27:04.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f2755e00640 1 -- 172.21.15.47:0/718256659 learned_addr learned my addr 172.21.15.47:0/718256659 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:04.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f274fe00640 1 --2- 172.21.15.47:0/718256659 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f275815e080 0x7f275809a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:04.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f2756800640 1 --2- 172.21.15.47:0/718256659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f275815e460 0x7f275815cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:04.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f2755e00640 1 -- 172.21.15.47:0/718256659 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f275815e080 msgr2=0x7f275809a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:04.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f2755e00640 1 --2- 172.21.15.47:0/718256659 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f275815e080 0x7f275809a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f2755e00640 1 -- 172.21.15.47:0/718256659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f275815e460 msgr2=0x7f275815cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:04.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f2755e00640 1 --2- 172.21.15.47:0/718256659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f275815e460 0x7f275815cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f2755e00640 1 -- 172.21.15.47:0/718256659 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27480094d0 con 0x7f27580ad060 2024-02-19T15:27:04.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f2755e00640 1 --2- 172.21.15.47:0/718256659 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27580ad060 0x7f2758153080 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f274801a100 tx=0x7f274801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:04.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f274e000640 1 -- 172.21.15.47:0/718256659 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27480b7620 con 0x7f27580ad060 2024-02-19T15:27:04.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f274e000640 1 -- 172.21.15.47:0/718256659 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27480b7780 con 0x7f27580ad060 2024-02-19T15:27:04.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f274e000640 1 -- 172.21.15.47:0/718256659 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27480b7b30 con 0x7f27580ad060 2024-02-19T15:27:04.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f275d200640 1 -- 172.21.15.47:0/718256659 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f275809bea0 con 0x7f27580ad060 2024-02-19T15:27:04.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f275d200640 1 -- 172.21.15.47:0/718256659 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2758154880 con 0x7f27580ad060 2024-02-19T15:27:04.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f275d200640 1 -- 172.21.15.47:0/718256659 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27580b5800 con 0x7f27580ad060 2024-02-19T15:27:04.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.371+0000 7f274e000640 1 -- 172.21.15.47:0/718256659 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f27480b3040 con 0x7f27580ad060 2024-02-19T15:27:04.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.375+0000 7f274e000640 1 --2- 172.21.15.47:0/718256659 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2734066e40 0x7f2734069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:04.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.375+0000 7f274e000640 1 -- 172.21.15.47:0/718256659 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f274811ee40 con 0x7f27580ad060 2024-02-19T15:27:04.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.375+0000 7f274fe00640 1 --2- 172.21.15.47:0/718256659 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2734066e40 0x7f2734069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:04.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.375+0000 7f274fe00640 1 --2- 172.21.15.47:0/718256659 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2734066e40 0x7f2734069300 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2758168c20 tx=0x7f2740007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:04.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.379+0000 7f274e000640 1 -- 172.21.15.47:0/718256659 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f27480ebcb0 con 0x7f27580ad060 2024-02-19T15:27:04.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.551+0000 7f275d200640 1 -- 172.21.15.47:0/718256659 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2758068120 con 0x7f27580ad060 2024-02-19T15:27:04.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.551+0000 7f274e000640 1 -- 172.21.15.47:0/718256659 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v97) v1 ==== 80+0+240 (secure 0 0 0) 0x7f27480b1100 con 0x7f27580ad060 2024-02-19T15:27:04.556 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:6833/68555355 2024-02-20T15:26:40.818800+0000 2024-02-19T15:27:04.557 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:6832/68555355 2024-02-20T15:26:40.818800+0000 2024-02-19T15:27:04.557 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.116:6837/547176070 2024-02-20T15:26:39.960651+0000 2024-02-19T15:27:04.557 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.116:6835/547176070 2024-02-20T15:26:39.960651+0000 2024-02-19T15:27:04.560 INFO:teuthology.orchestra.run.smithi047.stderr:listed 4 entries 2024-02-19T15:27:04.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.551+0000 7f275d200640 1 -- 172.21.15.47:0/718256659 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2734066e40 msgr2=0x7f2734069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:04.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.551+0000 7f275d200640 1 --2- 172.21.15.47:0/718256659 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2734066e40 0x7f2734069300 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2758168c20 tx=0x7f2740007e90 comp rx=0 tx=0).stop 2024-02-19T15:27:04.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.551+0000 7f275d200640 1 -- 172.21.15.47:0/718256659 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27580ad060 msgr2=0x7f2758153080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:04.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.551+0000 7f275d200640 1 --2- 172.21.15.47:0/718256659 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27580ad060 0x7f2758153080 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f274801a100 tx=0x7f274801ad10 comp rx=0 tx=0).stop 2024-02-19T15:27:04.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.551+0000 7f275d200640 1 -- 172.21.15.47:0/718256659 shutdown_connections 2024-02-19T15:27:04.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.551+0000 7f275d200640 1 --2- 172.21.15.47:0/718256659 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2734066e40 0x7f2734069300 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.551+0000 7f275d200640 1 --2- 172.21.15.47:0/718256659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f275815e460 0x7f275815cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.551+0000 7f275d200640 1 --2- 172.21.15.47:0/718256659 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f275815e080 0x7f275809a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.551+0000 7f275d200640 1 --2- 172.21.15.47:0/718256659 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27580ad060 0x7f2758153080 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.551+0000 7f275d200640 1 -- 172.21.15.47:0/718256659 >> 172.21.15.47:0/718256659 conn(0x7f27580b5d10 msgr2=0x7f27580afca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:04.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.551+0000 7f275d200640 1 -- 172.21.15.47:0/718256659 shutdown_connections 2024-02-19T15:27:04.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.551+0000 7f275d200640 1 -- 172.21.15.47:0/718256659 wait complete. 2024-02-19T15:27:04.577 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-02-19T15:27:04.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:04.577 DEBUG:teuthology.orchestra.run.smithi047:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-02-19T15:27:04.667 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi047.front.sepia.ceph.com 2024-02-19T15:27:04.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:04.667 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-19T15:27:04.677 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:27:04.677 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:27:04.696 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:27:04.697 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:27:04.697 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:27:04.730 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:27:04.730 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:27:04.730 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:27:04.749 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:27:04.749 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:27:04.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.907+0000 7f68fa200640 1 Processor -- start 2024-02-19T15:27:04.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.907+0000 7f68fa200640 1 -- start start 2024-02-19T15:27:04.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.907+0000 7f68fa200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68f4061a50 0x7f68f40ae2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:04.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.907+0000 7f68fa200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f68f40aefc0 0x7f68f40af450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:04.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.907+0000 7f68fa200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68f406eed0 con 0x7f68f415e080 2024-02-19T15:27:04.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.907+0000 7f68fa200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68f40af990 con 0x7f68f40aefc0 2024-02-19T15:27:04.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.907+0000 7f68fa200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f415e460 con 0x7f68f4061a50 2024-02-19T15:27:04.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.911+0000 7f68f3e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f68f40aefc0 0x7f68f40af450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:04.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.911+0000 7f68eaa00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68f4061a50 0x7f68f40ae2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:04.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.911+0000 7f68f3e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f68f40aefc0 0x7f68f40af450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:40010/0 (socket says 172.21.15.47:40010) 2024-02-19T15:27:04.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.911+0000 7f68f3e00640 1 -- 172.21.15.47:0/3122245829 learned_addr learned my addr 172.21.15.47:0/3122245829 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:04.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.911+0000 7f68eaa00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68f4061a50 0x7f68f40ae2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:40728/0 (socket says 172.21.15.47:40728) 2024-02-19T15:27:04.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.911+0000 7f68f3e00640 1 -- 172.21.15.47:0/3122245829 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68f4061a50 msgr2=0x7f68f40ae2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:04.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.911+0000 7f68f3e00640 1 --2- 172.21.15.47:0/3122245829 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68f4061a50 0x7f68f40ae2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.911+0000 7f68f3e00640 1 -- 172.21.15.47:0/3122245829 >> v1:172.21.15.116:6789/0 conn(0x7f68f415e080 legacy=0x7f68f40b35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:04.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.911+0000 7f68f3e00640 1 -- 172.21.15.47:0/3122245829 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f40aebf0 con 0x7f68f40aefc0 2024-02-19T15:27:04.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.911+0000 7f68f2a00640 1 -- 172.21.15.47:0/3122245829 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775738425 0 0) 0x7f68f406eed0 con 0x7f68f415e080 2024-02-19T15:27:04.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.911+0000 7f68eaa00640 1 --2- 172.21.15.47:0/3122245829 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68f4061a50 0x7f68f40ae2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:27:04.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.911+0000 7f68f3e00640 1 --2- 172.21.15.47:0/3122245829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f40aefc0 0x7f68f40af450 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f68ec009e50 tx=0x7f68ec01ca00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2a9f41b9d0d66e7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:04.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.911+0000 7f68f2a00640 1 -- 172.21.15.47:0/3122245829 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec0a58d0 con 0x7f68f40aefc0 2024-02-19T15:27:04.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.911+0000 7f68f2a00640 1 -- 172.21.15.47:0/3122245829 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ec0a5a30 con 0x7f68f40aefc0 2024-02-19T15:27:04.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.911+0000 7f68f2a00640 1 -- 172.21.15.47:0/3122245829 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec0a5dc0 con 0x7f68f40aefc0 2024-02-19T15:27:04.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68fa200640 1 -- 172.21.15.47:0/3122245829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f40aefc0 msgr2=0x7f68f40af450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:04.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68fa200640 1 --2- 172.21.15.47:0/3122245829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f40aefc0 0x7f68f40af450 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f68ec009e50 tx=0x7f68ec01ca00 comp rx=0 tx=0).stop 2024-02-19T15:27:04.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68fa200640 1 -- 172.21.15.47:0/3122245829 shutdown_connections 2024-02-19T15:27:04.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68fa200640 1 --2- 172.21.15.47:0/3122245829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f40aefc0 0x7f68f40af450 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68fa200640 1 --2- 172.21.15.47:0/3122245829 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68f4061a50 0x7f68f40ae2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68fa200640 1 -- 172.21.15.47:0/3122245829 >> 172.21.15.47:0/3122245829 conn(0x7f68f4072470 msgr2=0x7f68f406af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:04.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68fa200640 1 -- 172.21.15.47:0/3122245829 shutdown_connections 2024-02-19T15:27:04.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68fa200640 1 -- 172.21.15.47:0/3122245829 wait complete. 2024-02-19T15:27:04.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68fa200640 1 Processor -- start 2024-02-19T15:27:04.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68fa200640 1 -- start start 2024-02-19T15:27:04.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68fa200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68f4061a50 0x7f68f4145030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:04.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68fa200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68f40aefc0 0x7f68f4155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:04.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68fa200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f415e080 0x7f68f414c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:04.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68fa200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f4146250 con 0x7f68f40aefc0 2024-02-19T15:27:04.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68fa200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f414d810 con 0x7f68f415e080 2024-02-19T15:27:04.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68fa200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f414daf0 con 0x7f68f4061a50 2024-02-19T15:27:04.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68eaa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68f40aefc0 0x7f68f4155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:04.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68eaa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68f40aefc0 0x7f68f4155c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56424/0 (socket says 172.21.15.47:56424) 2024-02-19T15:27:04.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68eaa00640 1 -- 172.21.15.47:0/128874117 learned_addr learned my addr 172.21.15.47:0/128874117 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:04.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68f3400640 1 --2- 172.21.15.47:0/128874117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68f4061a50 0x7f68f4145030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:04.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68f3e00640 1 --2- 172.21.15.47:0/128874117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f415e080 0x7f68f414c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:04.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68eaa00640 1 -- 172.21.15.47:0/128874117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68f4061a50 msgr2=0x7f68f4145030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:04.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68eaa00640 1 --2- 172.21.15.47:0/128874117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68f4061a50 0x7f68f4145030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68eaa00640 1 -- 172.21.15.47:0/128874117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f415e080 msgr2=0x7f68f414c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:04.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.915+0000 7f68eaa00640 1 --2- 172.21.15.47:0/128874117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f415e080 0x7f68f414c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:04.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.919+0000 7f68eaa00640 1 -- 172.21.15.47:0/128874117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68dc009420 con 0x7f68f40aefc0 2024-02-19T15:27:04.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.919+0000 7f68eaa00640 1 --2- 172.21.15.47:0/128874117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68f40aefc0 0x7f68f4155c70 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f68dc00ebe0 tx=0x7f68dc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:04.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.919+0000 7f68ebe00640 1 -- 172.21.15.47:0/128874117 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68dc00eee0 con 0x7f68f40aefc0 2024-02-19T15:27:04.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.919+0000 7f68ebe00640 1 -- 172.21.15.47:0/128874117 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68dc00cc30 con 0x7f68f40aefc0 2024-02-19T15:27:04.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.919+0000 7f68ebe00640 1 -- 172.21.15.47:0/128874117 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68dc013470 con 0x7f68f40aefc0 2024-02-19T15:27:04.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.919+0000 7f68fa200640 1 -- 172.21.15.47:0/128874117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ec009b30 con 0x7f68f40aefc0 2024-02-19T15:27:04.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.919+0000 7f68fa200640 1 -- 172.21.15.47:0/128874117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f41464a0 con 0x7f68f40aefc0 2024-02-19T15:27:04.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.919+0000 7f68fa200640 1 -- 172.21.15.47:0/128874117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68f414e010 con 0x7f68f40aefc0 2024-02-19T15:27:04.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.919+0000 7f68ebe00640 1 -- 172.21.15.47:0/128874117 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f68dc0135d0 con 0x7f68f40aefc0 2024-02-19T15:27:04.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.919+0000 7f68ebe00640 1 --2- 172.21.15.47:0/128874117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f68d0066e70 0x7f68d0069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:04.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.919+0000 7f68ebe00640 1 -- 172.21.15.47:0/128874117 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f68dc084c80 con 0x7f68f40aefc0 2024-02-19T15:27:04.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.919+0000 7f68f3400640 1 --2- 172.21.15.47:0/128874117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f68d0066e70 0x7f68d0069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:04.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.923+0000 7f68f3400640 1 --2- 172.21.15.47:0/128874117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f68d0066e70 0x7f68d0069330 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f68e4007fd0 tx=0x7f68e4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:04.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:04.923+0000 7f68ebe00640 1 -- 172.21.15.47:0/128874117 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f68dc08a020 con 0x7f68f40aefc0 2024-02-19T15:27:05.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.119+0000 7f68fa200640 1 -- 172.21.15.47:0/128874117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68f4068120 con 0x7f68f40aefc0 2024-02-19T15:27:05.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.123+0000 7f68ebe00640 1 -- 172.21.15.47:0/128874117 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4947 (secure 0 0 0) 0x7f68dc051ae0 con 0x7f68f40aefc0 2024-02-19T15:27:05.127 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:05.127 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":66,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7060,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/4006350936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":4006350936},{"type":"v1","addr":"172.21.15.116:6837","nonce":4006350936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/2191173523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2191173523},{"type":"v1","addr":"172.21.15.47:6834","nonce":2191173523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/1012301665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":1012301665},{"type":"v1","addr":"172.21.15.47:6835","nonce":1012301665}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:26:26.721109+0000","modified":"2024-02-19T15:26:43.866082+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7054":{"gid":7054,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6835/3852677195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3852677195},{"type":"v1","addr":"172.21.15.116:6835","nonce":3852677195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T15:27:05.127 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 66 2024-02-19T15:27:05.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.123+0000 7f68fa200640 1 -- 172.21.15.47:0/128874117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f68d0066e70 msgr2=0x7f68d0069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:05.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.123+0000 7f68fa200640 1 --2- 172.21.15.47:0/128874117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f68d0066e70 0x7f68d0069330 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f68e4007fd0 tx=0x7f68e4009040 comp rx=0 tx=0).stop 2024-02-19T15:27:05.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.123+0000 7f68fa200640 1 -- 172.21.15.47:0/128874117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68f40aefc0 msgr2=0x7f68f4155c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:05.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.123+0000 7f68fa200640 1 --2- 172.21.15.47:0/128874117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68f40aefc0 0x7f68f4155c70 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f68dc00ebe0 tx=0x7f68dc00c520 comp rx=0 tx=0).stop 2024-02-19T15:27:05.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.123+0000 7f68fa200640 1 -- 172.21.15.47:0/128874117 shutdown_connections 2024-02-19T15:27:05.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.123+0000 7f68fa200640 1 --2- 172.21.15.47:0/128874117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f68d0066e70 0x7f68d0069330 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.123+0000 7f68fa200640 1 --2- 172.21.15.47:0/128874117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f415e080 0x7f68f414c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.123+0000 7f68fa200640 1 --2- 172.21.15.47:0/128874117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68f40aefc0 0x7f68f4155c70 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.123+0000 7f68fa200640 1 --2- 172.21.15.47:0/128874117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68f4061a50 0x7f68f4145030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.123+0000 7f68fa200640 1 -- 172.21.15.47:0/128874117 >> 172.21.15.47:0/128874117 conn(0x7f68f4072470 msgr2=0x7f68f415bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:05.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.123+0000 7f68fa200640 1 -- 172.21.15.47:0/128874117 shutdown_connections 2024-02-19T15:27:05.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.123+0000 7f68fa200640 1 -- 172.21.15.47:0/128874117 wait complete. 2024-02-19T15:27:05.148 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:27:05.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.283+0000 7ffb4b600640 1 Processor -- start 2024-02-19T15:27:05.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.283+0000 7ffb4b600640 1 -- start start 2024-02-19T15:27:05.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.283+0000 7ffb4b600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ffb44145790 0x7ffb440728f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:05.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.283+0000 7ffb4b600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb440cf760 0x7ffb4415c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:05.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.283+0000 7ffb4b600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4415fda0 con 0x7ffb440b5190 2024-02-19T15:27:05.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.283+0000 7ffb4b600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb441609b0 con 0x7ffb44145790 2024-02-19T15:27:05.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.283+0000 7ffb4b600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb44160080 con 0x7ffb440cf760 2024-02-19T15:27:05.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.283+0000 7ffb48c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ffb44145790 0x7ffb440728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:05.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.283+0000 7ffb48c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ffb44145790 0x7ffb440728f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56432/0 (socket says 172.21.15.47:56432) 2024-02-19T15:27:05.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.283+0000 7ffb43e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb440cf760 0x7ffb4415c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:05.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.283+0000 7ffb48c00640 1 -- 172.21.15.47:0/2374903236 learned_addr learned my addr 172.21.15.47:0/2374903236 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:05.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.283+0000 7ffb48c00640 1 -- 172.21.15.47:0/2374903236 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb440cf760 msgr2=0x7ffb4415c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:05.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.283+0000 7ffb48c00640 1 --2- 172.21.15.47:0/2374903236 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb440cf760 0x7ffb4415c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.283+0000 7ffb48c00640 1 -- 172.21.15.47:0/2374903236 >> v1:172.21.15.116:6789/0 conn(0x7ffb440b5190 legacy=0x7ffb440cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:05.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.283+0000 7ffb48c00640 1 -- 172.21.15.47:0/2374903236 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb44160360 con 0x7ffb44145790 2024-02-19T15:27:05.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.287+0000 7ffb48c00640 1 --2- 172.21.15.47:0/2374903236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb44145790 0x7ffb440728f0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7ffb38009870 tx=0x7ffb3801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992a3f00e47621f1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:05.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.287+0000 7ffb43400640 1 -- 172.21.15.47:0/2374903236 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb380ac620 con 0x7ffb44145790 2024-02-19T15:27:05.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.287+0000 7ffb43400640 1 -- 172.21.15.47:0/2374903236 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb380ac780 con 0x7ffb44145790 2024-02-19T15:27:05.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.287+0000 7ffb43400640 1 -- 172.21.15.47:0/2374903236 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb380acb10 con 0x7ffb44145790 2024-02-19T15:27:05.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.287+0000 7ffb4b600640 1 -- 172.21.15.47:0/2374903236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb44145790 msgr2=0x7ffb440728f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:05.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.287+0000 7ffb4b600640 1 --2- 172.21.15.47:0/2374903236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb44145790 0x7ffb440728f0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7ffb38009870 tx=0x7ffb3801c4d0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.287+0000 7ffb4b600640 1 -- 172.21.15.47:0/2374903236 shutdown_connections 2024-02-19T15:27:05.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.287+0000 7ffb4b600640 1 --2- 172.21.15.47:0/2374903236 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb440cf760 0x7ffb4415c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.287+0000 7ffb4b600640 1 --2- 172.21.15.47:0/2374903236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb44145790 0x7ffb440728f0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.287+0000 7ffb4b600640 1 -- 172.21.15.47:0/2374903236 >> 172.21.15.47:0/2374903236 conn(0x7ffb440b1b80 msgr2=0x7ffb440b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:05.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.287+0000 7ffb4b600640 1 -- 172.21.15.47:0/2374903236 shutdown_connections 2024-02-19T15:27:05.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.287+0000 7ffb4b600640 1 -- 172.21.15.47:0/2374903236 wait complete. 2024-02-19T15:27:05.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb4b600640 1 Processor -- start 2024-02-19T15:27:05.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb4b600640 1 -- start start 2024-02-19T15:27:05.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb4b600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb440b5190 0x7ffb44140460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:05.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb4b600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb440cf760 0x7ffb4413d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:05.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb4b600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb44145790 0x7ffb44160af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:05.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb4b600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4413fb70 con 0x7ffb440b5190 2024-02-19T15:27:05.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb4b600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4413ef60 con 0x7ffb44145790 2024-02-19T15:27:05.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb4b600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4413f240 con 0x7ffb440cf760 2024-02-19T15:27:05.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb48c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb440b5190 0x7ffb44140460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:05.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb48c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb440b5190 0x7ffb44140460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56440/0 (socket says 172.21.15.47:56440) 2024-02-19T15:27:05.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb48c00640 1 -- 172.21.15.47:0/3238724235 learned_addr learned my addr 172.21.15.47:0/3238724235 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:05.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb43e00640 1 --2- 172.21.15.47:0/3238724235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb440cf760 0x7ffb4413d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:05.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb49600640 1 --2- 172.21.15.47:0/3238724235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb44145790 0x7ffb44160af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:05.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb48c00640 1 -- 172.21.15.47:0/3238724235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb440cf760 msgr2=0x7ffb4413d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:05.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb48c00640 1 --2- 172.21.15.47:0/3238724235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb440cf760 0x7ffb4413d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb48c00640 1 -- 172.21.15.47:0/3238724235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb44145790 msgr2=0x7ffb44160af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:05.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb48c00640 1 --2- 172.21.15.47:0/3238724235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb44145790 0x7ffb44160af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb48c00640 1 -- 172.21.15.47:0/3238724235 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb380094d0 con 0x7ffb440b5190 2024-02-19T15:27:05.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb48c00640 1 --2- 172.21.15.47:0/3238724235 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb440b5190 0x7ffb44140460 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7ffb380af040 tx=0x7ffb3801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:05.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb40c00640 1 -- 172.21.15.47:0/3238724235 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb380b7420 con 0x7ffb440b5190 2024-02-19T15:27:05.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb40c00640 1 -- 172.21.15.47:0/3238724235 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb380b7580 con 0x7ffb440b5190 2024-02-19T15:27:05.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb40c00640 1 -- 172.21.15.47:0/3238724235 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb380b7910 con 0x7ffb440b5190 2024-02-19T15:27:05.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb4b600640 1 -- 172.21.15.47:0/3238724235 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4413f520 con 0x7ffb440b5190 2024-02-19T15:27:05.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.291+0000 7ffb4b600640 1 -- 172.21.15.47:0/3238724235 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb4413dc10 con 0x7ffb440b5190 2024-02-19T15:27:05.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.295+0000 7ffb4b600640 1 -- 172.21.15.47:0/3238724235 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb44068120 con 0x7ffb440b5190 2024-02-19T15:27:05.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.295+0000 7ffb40c00640 1 -- 172.21.15.47:0/3238724235 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ffb380b3040 con 0x7ffb440b5190 2024-02-19T15:27:05.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.295+0000 7ffb40c00640 1 --2- 172.21.15.47:0/3238724235 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffb24066e40 0x7ffb24069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:05.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.295+0000 7ffb40c00640 1 -- 172.21.15.47:0/3238724235 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7ffb3811ef80 con 0x7ffb440b5190 2024-02-19T15:27:05.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.295+0000 7ffb43e00640 1 --2- 172.21.15.47:0/3238724235 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffb24066e40 0x7ffb24069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:05.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.295+0000 7ffb43e00640 1 --2- 172.21.15.47:0/3238724235 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffb24066e40 0x7ffb24069300 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ffb440b1aa0 tx=0x7ffb30009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:05.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.299+0000 7ffb40c00640 1 -- 172.21.15.47:0/3238724235 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ffb380ebcd0 con 0x7ffb440b5190 2024-02-19T15:27:05.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.463+0000 7ffb4b600640 1 -- 172.21.15.47:0/3238724235 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb44164910 con 0x7ffb440b5190 2024-02-19T15:27:05.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.463+0000 7ffb40c00640 1 -- 172.21.15.47:0/3238724235 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16875 (secure 0 0 0) 0x7ffb380ef390 con 0x7ffb440b5190 2024-02-19T15:27:05.469 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:05.469 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":97,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:26:40.823963+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:26:25.124594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-02-19T15:26:25.706971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:6833/68555355":"2024-02-20T15:26:40.818800+0000","172.21.15.47:6832/68555355":"2024-02-20T15:26:40.818800+0000","172.21.15.116:6837/547176070":"2024-02-20T15:26:39.960651+0000","172.21.15.116:6835/547176070":"2024-02-20T15:26:39.960651+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:27:05.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.467+0000 7ffb4b600640 1 -- 172.21.15.47:0/3238724235 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffb24066e40 msgr2=0x7ffb24069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:05.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.467+0000 7ffb4b600640 1 --2- 172.21.15.47:0/3238724235 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffb24066e40 0x7ffb24069300 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ffb440b1aa0 tx=0x7ffb30009190 comp rx=0 tx=0).stop 2024-02-19T15:27:05.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.467+0000 7ffb4b600640 1 -- 172.21.15.47:0/3238724235 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb440b5190 msgr2=0x7ffb44140460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:05.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.467+0000 7ffb4b600640 1 --2- 172.21.15.47:0/3238724235 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb440b5190 0x7ffb44140460 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7ffb380af040 tx=0x7ffb3801ad10 comp rx=0 tx=0).stop 2024-02-19T15:27:05.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.467+0000 7ffb4b600640 1 -- 172.21.15.47:0/3238724235 shutdown_connections 2024-02-19T15:27:05.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.467+0000 7ffb4b600640 1 --2- 172.21.15.47:0/3238724235 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffb24066e40 0x7ffb24069300 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.467+0000 7ffb4b600640 1 --2- 172.21.15.47:0/3238724235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb44145790 0x7ffb44160af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.467+0000 7ffb4b600640 1 --2- 172.21.15.47:0/3238724235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb440cf760 0x7ffb4413d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.467+0000 7ffb4b600640 1 --2- 172.21.15.47:0/3238724235 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb440b5190 0x7ffb44140460 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.467+0000 7ffb4b600640 1 -- 172.21.15.47:0/3238724235 >> 172.21.15.47:0/3238724235 conn(0x7ffb440b1b80 msgr2=0x7ffb4415cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:05.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.467+0000 7ffb4b600640 1 -- 172.21.15.47:0/3238724235 shutdown_connections 2024-02-19T15:27:05.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.467+0000 7ffb4b600640 1 -- 172.21.15.47:0/3238724235 wait complete. 2024-02-19T15:27:05.490 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:26:25.124594+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:26:25.706971+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:27:05.490 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:27:05.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.627+0000 7ff3da800640 1 Processor -- start 2024-02-19T15:27:05.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.627+0000 7ff3da800640 1 -- start start 2024-02-19T15:27:05.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.631+0000 7ff3da800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff3d4158a30 0x7ff3d415bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:05.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.631+0000 7ff3da800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d415c300 con 0x7ff3d406df30 2024-02-19T15:27:05.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.631+0000 7ff3da800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d4072be0 con 0x7ff3d406e390 2024-02-19T15:27:05.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.631+0000 7ff3da800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3d4158e10 con 0x7ff3d4158a30 2024-02-19T15:27:05.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.631+0000 7ff3d3400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff3d4158a30 0x7ff3d415bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:05.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.631+0000 7ff3d3400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff3d4158a30 0x7ff3d415bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56456/0 (socket says 172.21.15.47:56456) 2024-02-19T15:27:05.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.631+0000 7ff3d3400640 1 -- 172.21.15.47:0/1155904962 learned_addr learned my addr 172.21.15.47:0/1155904962 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:05.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.631+0000 7ff3d3400640 1 -- 172.21.15.47:0/1155904962 >> v1:172.21.15.116:6789/0 conn(0x7ff3d406e390 legacy=0x7ff3d40724d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:05.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.631+0000 7ff3d3400640 1 -- 172.21.15.47:0/1155904962 >> v1:172.21.15.47:6789/0 conn(0x7ff3d406df30 legacy=0x7ff3d4071dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:05.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.631+0000 7ff3d3400640 1 -- 172.21.15.47:0/1155904962 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d40cf780 con 0x7ff3d4158a30 2024-02-19T15:27:05.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.631+0000 7ff3d3400640 1 --2- 172.21.15.47:0/1155904962 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3d4158a30 0x7ff3d415bdc0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4009870 tx=0x7ff3c401c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe5b99f1e6452cff server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:05.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.631+0000 7ff3d2a00640 1 -- 172.21.15.47:0/1155904962 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c40a95f0 con 0x7ff3d4158a30 2024-02-19T15:27:05.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.631+0000 7ff3d2a00640 1 -- 172.21.15.47:0/1155904962 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c40a9750 con 0x7ff3d4158a30 2024-02-19T15:27:05.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.631+0000 7ff3d2a00640 1 -- 172.21.15.47:0/1155904962 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c40a9ae0 con 0x7ff3d4158a30 2024-02-19T15:27:05.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.635+0000 7ff3da800640 1 -- 172.21.15.47:0/1155904962 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3d4158a30 msgr2=0x7ff3d415bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:05.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.635+0000 7ff3da800640 1 --2- 172.21.15.47:0/1155904962 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3d4158a30 0x7ff3d415bdc0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4009870 tx=0x7ff3c401c530 comp rx=0 tx=0).stop 2024-02-19T15:27:05.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.635+0000 7ff3da800640 1 -- 172.21.15.47:0/1155904962 shutdown_connections 2024-02-19T15:27:05.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.635+0000 7ff3da800640 1 --2- 172.21.15.47:0/1155904962 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3d4158a30 0x7ff3d415bdc0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.635+0000 7ff3da800640 1 -- 172.21.15.47:0/1155904962 >> 172.21.15.47:0/1155904962 conn(0x7ff3d40b1b80 msgr2=0x7ff3d40b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:05.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.635+0000 7ff3da800640 1 -- 172.21.15.47:0/1155904962 shutdown_connections 2024-02-19T15:27:05.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.635+0000 7ff3da800640 1 -- 172.21.15.47:0/1155904962 wait complete. 2024-02-19T15:27:05.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.635+0000 7ff3da800640 1 Processor -- start 2024-02-19T15:27:05.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.635+0000 7ff3da800640 1 -- start start 2024-02-19T15:27:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.635+0000 7ff3da800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3d406df30 0x7ff3d4157670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.635+0000 7ff3da800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff3d406e390 0x7ff3d4144160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.635+0000 7ff3da800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3d4158a30 0x7ff3d40b4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.635+0000 7ff3da800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d41446a0 con 0x7ff3d406df30 2024-02-19T15:27:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.635+0000 7ff3da800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d4143710 con 0x7ff3d4158a30 2024-02-19T15:27:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.635+0000 7ff3da800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d4156e00 con 0x7ff3d406e390 2024-02-19T15:27:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.635+0000 7ff3d3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3d406df30 0x7ff3d4157670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.635+0000 7ff3d3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3d406df30 0x7ff3d4157670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56458/0 (socket says 172.21.15.47:56458) 2024-02-19T15:27:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.635+0000 7ff3d3400640 1 -- 172.21.15.47:0/3661922846 learned_addr learned my addr 172.21.15.47:0/3661922846 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.635+0000 7ff3caa00640 1 --2- 172.21.15.47:0/3661922846 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff3d406e390 0x7ff3d4144160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.635+0000 7ff3d3e00640 1 --2- 172.21.15.47:0/3661922846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3d4158a30 0x7ff3d40b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:05.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.639+0000 7ff3d3400640 1 -- 172.21.15.47:0/3661922846 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff3d406e390 msgr2=0x7ff3d4144160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:05.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.639+0000 7ff3d3400640 1 --2- 172.21.15.47:0/3661922846 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff3d406e390 0x7ff3d4144160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.639+0000 7ff3d3400640 1 -- 172.21.15.47:0/3661922846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3d4158a30 msgr2=0x7ff3d40b4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:05.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.639+0000 7ff3d3400640 1 --2- 172.21.15.47:0/3661922846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3d4158a30 0x7ff3d40b4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.639+0000 7ff3d3400640 1 -- 172.21.15.47:0/3661922846 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c40094d0 con 0x7ff3d406df30 2024-02-19T15:27:05.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.639+0000 7ff3d3400640 1 --2- 172.21.15.47:0/3661922846 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3d406df30 0x7ff3d4157670 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7ff3c401a160 tx=0x7ff3c401ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:05.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.639+0000 7ff3cbe00640 1 -- 172.21.15.47:0/3661922846 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c40b1800 con 0x7ff3d406df30 2024-02-19T15:27:05.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.639+0000 7ff3cbe00640 1 -- 172.21.15.47:0/3661922846 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c40b1960 con 0x7ff3d406df30 2024-02-19T15:27:05.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.639+0000 7ff3cbe00640 1 -- 172.21.15.47:0/3661922846 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c40b1cf0 con 0x7ff3d406df30 2024-02-19T15:27:05.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.639+0000 7ff3da800640 1 -- 172.21.15.47:0/3661922846 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d41570e0 con 0x7ff3d406df30 2024-02-19T15:27:05.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.639+0000 7ff3da800640 1 -- 172.21.15.47:0/3661922846 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d4143c40 con 0x7ff3d406df30 2024-02-19T15:27:05.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.639+0000 7ff3da800640 1 -- 172.21.15.47:0/3661922846 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d4157bb0 con 0x7ff3d406df30 2024-02-19T15:27:05.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.639+0000 7ff3cbe00640 1 -- 172.21.15.47:0/3661922846 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff3c40c53f0 con 0x7ff3d406df30 2024-02-19T15:27:05.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.639+0000 7ff3cbe00640 1 --2- 172.21.15.47:0/3661922846 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff3b0066e40 0x7ff3b0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:05.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.639+0000 7ff3cbe00640 1 -- 172.21.15.47:0/3661922846 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7ff3c411f8d0 con 0x7ff3d406df30 2024-02-19T15:27:05.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.639+0000 7ff3caa00640 1 --2- 172.21.15.47:0/3661922846 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff3b0066e40 0x7ff3b0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:05.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.643+0000 7ff3caa00640 1 --2- 172.21.15.47:0/3661922846 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff3b0066e40 0x7ff3b0069300 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff3d414a1a0 tx=0x7ff3bc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:05.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.643+0000 7ff3cbe00640 1 -- 172.21.15.47:0/3661922846 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff3c40ec620 con 0x7ff3d406df30 2024-02-19T15:27:05.700 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:27:05.703 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:27:05.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.807+0000 7ff3da800640 1 -- 172.21.15.47:0/3661922846 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff3d4068120 con 0x7ff3d406df30 2024-02-19T15:27:05.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.807+0000 7ff3cbe00640 1 -- 172.21.15.47:0/3661922846 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff3c40efce0 con 0x7ff3d406df30 2024-02-19T15:27:05.812 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:27:05.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.807+0000 7ff3da800640 1 -- 172.21.15.47:0/3661922846 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff3b0066e40 msgr2=0x7ff3b0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:05.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.807+0000 7ff3da800640 1 --2- 172.21.15.47:0/3661922846 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff3b0066e40 0x7ff3b0069300 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff3d414a1a0 tx=0x7ff3bc009040 comp rx=0 tx=0).stop 2024-02-19T15:27:05.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.807+0000 7ff3da800640 1 -- 172.21.15.47:0/3661922846 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3d406df30 msgr2=0x7ff3d4157670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:05.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.807+0000 7ff3da800640 1 --2- 172.21.15.47:0/3661922846 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3d406df30 0x7ff3d4157670 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7ff3c401a160 tx=0x7ff3c401ad70 comp rx=0 tx=0).stop 2024-02-19T15:27:05.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.807+0000 7ff3da800640 1 -- 172.21.15.47:0/3661922846 shutdown_connections 2024-02-19T15:27:05.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.807+0000 7ff3da800640 1 --2- 172.21.15.47:0/3661922846 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff3b0066e40 0x7ff3b0069300 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.807+0000 7ff3da800640 1 --2- 172.21.15.47:0/3661922846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3d4158a30 0x7ff3d40b4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.807+0000 7ff3da800640 1 --2- 172.21.15.47:0/3661922846 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff3d406e390 0x7ff3d4144160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.807+0000 7ff3da800640 1 --2- 172.21.15.47:0/3661922846 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3d406df30 0x7ff3d4157670 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.807+0000 7ff3da800640 1 -- 172.21.15.47:0/3661922846 >> 172.21.15.47:0/3661922846 conn(0x7ff3d40b1b80 msgr2=0x7ff3d40afb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:05.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.807+0000 7ff3da800640 1 -- 172.21.15.47:0/3661922846 shutdown_connections 2024-02-19T15:27:05.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.807+0000 7ff3da800640 1 -- 172.21.15.47:0/3661922846 wait complete. 2024-02-19T15:27:05.832 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:27:05.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.971+0000 7f2e51c00640 1 Processor -- start 2024-02-19T15:27:05.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.971+0000 7f2e51c00640 1 -- start start 2024-02-19T15:27:05.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.971+0000 7f2e51c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e4c06e640 0x7f2e4c162750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:05.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.971+0000 7f2e51c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2e4c0ae2f0 0x7f2e4c071180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:05.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.971+0000 7f2e51c00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e4c0b3240 con 0x7f2e4c1478b0 2024-02-19T15:27:05.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.971+0000 7f2e51c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e4c161c70 con 0x7f2e4c0ae2f0 2024-02-19T15:27:05.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.971+0000 7f2e51c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c161f50 con 0x7f2e4c06e640 2024-02-19T15:27:05.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.971+0000 7f2e4aa00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f2e4c1478b0 0x7f2e4c06df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58514/0 (socket says 172.21.15.47:58514) 2024-02-19T15:27:05.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.971+0000 7f2e4aa00640 1 -- 172.21.15.47:0/2937392576 learned_addr learned my addr 172.21.15.47:0/2937392576 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:05.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.971+0000 7f2e4b400640 1 --2- 172.21.15.47:0/2937392576 >> v2:172.21.15.47:3300/0 conn(0x7f2e4c0ae2f0 0x7f2e4c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:05.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.971+0000 7f2e43e00640 1 --2- 172.21.15.47:0/2937392576 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e4c06e640 0x7f2e4c162750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:05.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.971+0000 7f2e4a000640 1 -- 172.21.15.47:0/2937392576 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371474743 0 0) 0x7f2e4c0b3240 con 0x7f2e4c1478b0 2024-02-19T15:27:05.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.971+0000 7f2e4a000640 1 -- 172.21.15.47:0/2937392576 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e30003530 con 0x7f2e4c1478b0 2024-02-19T15:27:05.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.971+0000 7f2e4b400640 1 -- 172.21.15.47:0/2937392576 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e4c06e640 msgr2=0x7f2e4c162750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:05.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.971+0000 7f2e4b400640 1 --2- 172.21.15.47:0/2937392576 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e4c06e640 0x7f2e4c162750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.971+0000 7f2e4b400640 1 -- 172.21.15.47:0/2937392576 >> v1:172.21.15.47:6789/0 conn(0x7f2e4c1478b0 legacy=0x7f2e4c06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:05.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.971+0000 7f2e4b400640 1 -- 172.21.15.47:0/2937392576 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4c162230 con 0x7f2e4c0ae2f0 2024-02-19T15:27:05.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.975+0000 7f2e4b400640 1 --2- 172.21.15.47:0/2937392576 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0ae2f0 0x7f2e4c071180 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f2e44009d90 tx=0x7f2e4401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7fd0e946d9fa277 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:05.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.975+0000 7f2e4a000640 1 -- 172.21.15.47:0/2937392576 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e440a5960 con 0x7f2e4c0ae2f0 2024-02-19T15:27:05.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.975+0000 7f2e4a000640 1 -- 172.21.15.47:0/2937392576 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e440a5ac0 con 0x7f2e4c0ae2f0 2024-02-19T15:27:05.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.975+0000 7f2e4a000640 1 -- 172.21.15.47:0/2937392576 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e440a5e50 con 0x7f2e4c0ae2f0 2024-02-19T15:27:05.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.975+0000 7f2e51c00640 1 -- 172.21.15.47:0/2937392576 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0ae2f0 msgr2=0x7f2e4c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:05.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.975+0000 7f2e51c00640 1 --2- 172.21.15.47:0/2937392576 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0ae2f0 0x7f2e4c071180 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f2e44009d90 tx=0x7f2e4401c910 comp rx=0 tx=0).stop 2024-02-19T15:27:05.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.975+0000 7f2e51c00640 1 -- 172.21.15.47:0/2937392576 shutdown_connections 2024-02-19T15:27:05.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.975+0000 7f2e51c00640 1 --2- 172.21.15.47:0/2937392576 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0ae2f0 0x7f2e4c071180 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.975+0000 7f2e51c00640 1 --2- 172.21.15.47:0/2937392576 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e4c06e640 0x7f2e4c162750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.975+0000 7f2e51c00640 1 -- 172.21.15.47:0/2937392576 >> 172.21.15.47:0/2937392576 conn(0x7f2e4c0b1b80 msgr2=0x7f2e4c0b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:05.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.975+0000 7f2e51c00640 1 -- 172.21.15.47:0/2937392576 shutdown_connections 2024-02-19T15:27:05.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.975+0000 7f2e51c00640 1 -- 172.21.15.47:0/2937392576 wait complete. 2024-02-19T15:27:05.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e51c00640 1 Processor -- start 2024-02-19T15:27:05.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e51c00640 1 -- start start 2024-02-19T15:27:05.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e51c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e4c06e640 0x7f2e4c1452a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:05.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e51c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0ae2f0 0x7f2e4c13df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:05.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e51c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e4c1478b0 0x7f2e4c13e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:05.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e51c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c1457e0 con 0x7f2e4c0ae2f0 2024-02-19T15:27:05.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e51c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c145dd0 con 0x7f2e4c1478b0 2024-02-19T15:27:05.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e51c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c1460b0 con 0x7f2e4c06e640 2024-02-19T15:27:05.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e43e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0ae2f0 0x7f2e4c13df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:05.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e43e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0ae2f0 0x7f2e4c13df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56468/0 (socket says 172.21.15.47:56468) 2024-02-19T15:27:05.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e43e00640 1 -- 172.21.15.47:0/3248157862 learned_addr learned my addr 172.21.15.47:0/3248157862 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:05.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e4aa00640 1 --2- 172.21.15.47:0/3248157862 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e4c06e640 0x7f2e4c1452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:05.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e4b400640 1 --2- 172.21.15.47:0/3248157862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e4c1478b0 0x7f2e4c13e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:05.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e43e00640 1 -- 172.21.15.47:0/3248157862 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e4c06e640 msgr2=0x7f2e4c1452a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:05.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e43e00640 1 --2- 172.21.15.47:0/3248157862 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e4c06e640 0x7f2e4c1452a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e43e00640 1 -- 172.21.15.47:0/3248157862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e4c1478b0 msgr2=0x7f2e4c13e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:05.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e43e00640 1 --2- 172.21.15.47:0/3248157862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e4c1478b0 0x7f2e4c13e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:05.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e43e00640 1 -- 172.21.15.47:0/3248157862 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e44009a70 con 0x7f2e4c0ae2f0 2024-02-19T15:27:05.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e43e00640 1 --2- 172.21.15.47:0/3248157862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0ae2f0 0x7f2e4c13df00 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400e9b0 tx=0x7f2e3400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:05.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e42a00640 1 -- 172.21.15.47:0/3248157862 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e340096b0 con 0x7f2e4c0ae2f0 2024-02-19T15:27:05.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e42a00640 1 -- 172.21.15.47:0/3248157862 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3400c9a0 con 0x7f2e4c0ae2f0 2024-02-19T15:27:05.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e42a00640 1 -- 172.21.15.47:0/3248157862 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3400cd30 con 0x7f2e4c0ae2f0 2024-02-19T15:27:05.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e51c00640 1 -- 172.21.15.47:0/3248157862 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e4c146390 con 0x7f2e4c0ae2f0 2024-02-19T15:27:05.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.979+0000 7f2e51c00640 1 -- 172.21.15.47:0/3248157862 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e4c146950 con 0x7f2e4c0ae2f0 2024-02-19T15:27:05.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.983+0000 7f2e51c00640 1 -- 172.21.15.47:0/3248157862 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e4c068120 con 0x7f2e4c0ae2f0 2024-02-19T15:27:05.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.983+0000 7f2e42a00640 1 -- 172.21.15.47:0/3248157862 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2e34013470 con 0x7f2e4c0ae2f0 2024-02-19T15:27:05.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.983+0000 7f2e42a00640 1 --2- 172.21.15.47:0/3248157862 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2e28066e40 0x7f2e28069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:05.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.983+0000 7f2e42a00640 1 -- 172.21.15.47:0/3248157862 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f2e34084cf0 con 0x7f2e4c0ae2f0 2024-02-19T15:27:05.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.983+0000 7f2e4aa00640 1 --2- 172.21.15.47:0/3248157862 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2e28066e40 0x7f2e28069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:05.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.983+0000 7f2e4aa00640 1 --2- 172.21.15.47:0/3248157862 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2e28066e40 0x7f2e28069300 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c007fd0 tx=0x7f2e3c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:05.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:05.987+0000 7f2e42a00640 1 -- 172.21.15.47:0/3248157862 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2e34089070 con 0x7f2e4c0ae2f0 2024-02-19T15:27:06.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.151+0000 7f2e51c00640 1 -- 172.21.15.47:0/3248157862 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2e4c13ed10 con 0x7f2e4c0ae2f0 2024-02-19T15:27:06.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.151+0000 7f2e42a00640 1 -- 172.21.15.47:0/3248157862 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2e34017020 con 0x7f2e4c0ae2f0 2024-02-19T15:27:06.155 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:27:06.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.151+0000 7f2e51c00640 1 -- 172.21.15.47:0/3248157862 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2e28066e40 msgr2=0x7f2e28069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:06.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.151+0000 7f2e51c00640 1 --2- 172.21.15.47:0/3248157862 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2e28066e40 0x7f2e28069300 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c007fd0 tx=0x7f2e3c009040 comp rx=0 tx=0).stop 2024-02-19T15:27:06.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.151+0000 7f2e51c00640 1 -- 172.21.15.47:0/3248157862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0ae2f0 msgr2=0x7f2e4c13df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:06.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.151+0000 7f2e51c00640 1 --2- 172.21.15.47:0/3248157862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0ae2f0 0x7f2e4c13df00 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400e9b0 tx=0x7f2e3400ee70 comp rx=0 tx=0).stop 2024-02-19T15:27:06.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.151+0000 7f2e51c00640 1 -- 172.21.15.47:0/3248157862 shutdown_connections 2024-02-19T15:27:06.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.151+0000 7f2e51c00640 1 --2- 172.21.15.47:0/3248157862 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2e28066e40 0x7f2e28069300 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.151+0000 7f2e51c00640 1 --2- 172.21.15.47:0/3248157862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e4c1478b0 0x7f2e4c13e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.151+0000 7f2e51c00640 1 --2- 172.21.15.47:0/3248157862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e4c0ae2f0 0x7f2e4c13df00 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.151+0000 7f2e51c00640 1 --2- 172.21.15.47:0/3248157862 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e4c06e640 0x7f2e4c1452a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.151+0000 7f2e51c00640 1 -- 172.21.15.47:0/3248157862 >> 172.21.15.47:0/3248157862 conn(0x7f2e4c0b1b80 msgr2=0x7f2e4c06de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:06.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.151+0000 7f2e51c00640 1 -- 172.21.15.47:0/3248157862 shutdown_connections 2024-02-19T15:27:06.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.151+0000 7f2e51c00640 1 -- 172.21.15.47:0/3248157862 wait complete. 2024-02-19T15:27:06.176 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:27:06.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.315+0000 7f6ce2c00640 1 Processor -- start 2024-02-19T15:27:06.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.315+0000 7f6ce2c00640 1 -- start start 2024-02-19T15:27:06.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.315+0000 7f6ce2c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6cdc06df30 0x7f6cdc06e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:06.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.315+0000 7f6ce2c00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc06e8d0 con 0x7f6cdc0ce770 2024-02-19T15:27:06.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.315+0000 7f6ce2c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc15ec20 con 0x7f6cdc158a30 2024-02-19T15:27:06.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.315+0000 7f6ce2c00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cdc158e10 con 0x7f6cdc06df30 2024-02-19T15:27:06.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.315+0000 7f6ce0c00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6cdc0ce770 0x7f6cdc0cec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58528/0 (socket says 172.21.15.47:58528) 2024-02-19T15:27:06.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.315+0000 7f6ce0c00640 1 -- 172.21.15.47:0/991396384 learned_addr learned my addr 172.21.15.47:0/991396384 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:06.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.315+0000 7f6cd3400640 1 --2- 172.21.15.47:0/991396384 >> v2:172.21.15.116:3300/0 conn(0x7f6cdc06df30 0x7f6cdc06e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:06.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.315+0000 7f6cdb400640 1 -- 172.21.15.47:0/991396384 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308416256 0 0) 0x7f6cdc06e8d0 con 0x7f6cdc0ce770 2024-02-19T15:27:06.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.315+0000 7f6cdb400640 1 -- 172.21.15.47:0/991396384 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc0003530 con 0x7f6cdc0ce770 2024-02-19T15:27:06.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.315+0000 7f6cdb400640 1 -- 172.21.15.47:0/991396384 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198354698 0 0) 0x7f6cdc15ec20 con 0x7f6cdc158a30 2024-02-19T15:27:06.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.315+0000 7f6cdb400640 1 -- 172.21.15.47:0/991396384 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cdc06e8d0 con 0x7f6cdc158a30 2024-02-19T15:27:06.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.315+0000 7f6cd3400640 1 -- 172.21.15.47:0/991396384 >> v1:172.21.15.116:6789/0 conn(0x7f6cdc158a30 legacy=0x7f6cdc15bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:06.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.315+0000 7f6cd3400640 1 -- 172.21.15.47:0/991396384 >> v1:172.21.15.47:6789/0 conn(0x7f6cdc0ce770 legacy=0x7f6cdc0cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:06.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.315+0000 7f6cd3400640 1 -- 172.21.15.47:0/991396384 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc15ef00 con 0x7f6cdc06df30 2024-02-19T15:27:06.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.319+0000 7f6cd3400640 1 --2- 172.21.15.47:0/991396384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cdc06df30 0x7f6cdc06e390 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f6cc40097c0 tx=0x7f6cc401c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75c43872f39ceb49 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:06.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.319+0000 7f6cdb400640 1 -- 172.21.15.47:0/991396384 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc40ab670 con 0x7f6cdc06df30 2024-02-19T15:27:06.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.319+0000 7f6cdb400640 1 -- 172.21.15.47:0/991396384 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc40ab7d0 con 0x7f6cdc06df30 2024-02-19T15:27:06.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.319+0000 7f6cdb400640 1 -- 172.21.15.47:0/991396384 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc40abb60 con 0x7f6cdc06df30 2024-02-19T15:27:06.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.319+0000 7f6ce2c00640 1 -- 172.21.15.47:0/991396384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cdc06df30 msgr2=0x7f6cdc06e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:06.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.319+0000 7f6ce2c00640 1 --2- 172.21.15.47:0/991396384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cdc06df30 0x7f6cdc06e390 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f6cc40097c0 tx=0x7f6cc401c470 comp rx=0 tx=0).stop 2024-02-19T15:27:06.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.319+0000 7f6ce2c00640 1 -- 172.21.15.47:0/991396384 shutdown_connections 2024-02-19T15:27:06.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.319+0000 7f6ce2c00640 1 --2- 172.21.15.47:0/991396384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cdc06df30 0x7f6cdc06e390 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.319+0000 7f6ce2c00640 1 -- 172.21.15.47:0/991396384 >> 172.21.15.47:0/991396384 conn(0x7f6cdc0b1b80 msgr2=0x7f6cdc0b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:06.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.319+0000 7f6ce2c00640 1 -- 172.21.15.47:0/991396384 shutdown_connections 2024-02-19T15:27:06.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.319+0000 7f6ce2c00640 1 -- 172.21.15.47:0/991396384 wait complete. 2024-02-19T15:27:06.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6ce2c00640 1 Processor -- start 2024-02-19T15:27:06.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6ce2c00640 1 -- start start 2024-02-19T15:27:06.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6ce2c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cdc06df30 0x7f6cdc149ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:06.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6ce2c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cdc0ce770 0x7f6cdc151ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:06.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6ce2c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cdc158a30 0x7f6cdc148120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:06.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6ce2c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc159ee0 con 0x7f6cdc0ce770 2024-02-19T15:27:06.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6ce2c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc15a1c0 con 0x7f6cdc158a30 2024-02-19T15:27:06.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6ce2c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc15a4a0 con 0x7f6cdc06df30 2024-02-19T15:27:06.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6cd3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cdc0ce770 0x7f6cdc151ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:06.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6cd3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cdc0ce770 0x7f6cdc151ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56474/0 (socket says 172.21.15.47:56474) 2024-02-19T15:27:06.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6cd3400640 1 -- 172.21.15.47:0/3482529035 learned_addr learned my addr 172.21.15.47:0/3482529035 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:06.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6ce0c00640 1 --2- 172.21.15.47:0/3482529035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cdc158a30 0x7f6cdc148120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:06.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6cdbe00640 1 --2- 172.21.15.47:0/3482529035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cdc06df30 0x7f6cdc149ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:06.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6cd3400640 1 -- 172.21.15.47:0/3482529035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cdc06df30 msgr2=0x7f6cdc149ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:06.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6cd3400640 1 --2- 172.21.15.47:0/3482529035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cdc06df30 0x7f6cdc149ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6cd3400640 1 -- 172.21.15.47:0/3482529035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cdc158a30 msgr2=0x7f6cdc148120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:06.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6cd3400640 1 --2- 172.21.15.47:0/3482529035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cdc158a30 0x7f6cdc148120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6cd3400640 1 -- 172.21.15.47:0/3482529035 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc4009420 con 0x7f6cdc0ce770 2024-02-19T15:27:06.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6cd3400640 1 --2- 172.21.15.47:0/3482529035 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cdc0ce770 0x7f6cdc151ca0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f6cc40ad040 tx=0x7f6cc401acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:06.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6cd8c00640 1 -- 172.21.15.47:0/3482529035 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc40bf610 con 0x7f6cdc0ce770 2024-02-19T15:27:06.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6cd8c00640 1 -- 172.21.15.47:0/3482529035 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc40bf770 con 0x7f6cdc0ce770 2024-02-19T15:27:06.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6ce2c00640 1 -- 172.21.15.47:0/3482529035 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cdc15a780 con 0x7f6cdc0ce770 2024-02-19T15:27:06.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6cd8c00640 1 -- 172.21.15.47:0/3482529035 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc40bfb20 con 0x7f6cdc0ce770 2024-02-19T15:27:06.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.323+0000 7f6ce2c00640 1 -- 172.21.15.47:0/3482529035 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cdc14a3e0 con 0x7f6cdc0ce770 2024-02-19T15:27:06.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.327+0000 7f6ce2c00640 1 -- 172.21.15.47:0/3482529035 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cdc0b3ad0 con 0x7f6cdc0ce770 2024-02-19T15:27:06.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.327+0000 7f6cd8c00640 1 -- 172.21.15.47:0/3482529035 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6cc40bfc80 con 0x7f6cdc0ce770 2024-02-19T15:27:06.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.327+0000 7f6cd8c00640 1 --2- 172.21.15.47:0/3482529035 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6cb8066e40 0x7f6cb8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:06.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.327+0000 7f6cd8c00640 1 -- 172.21.15.47:0/3482529035 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f6cc411f890 con 0x7f6cdc0ce770 2024-02-19T15:27:06.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.327+0000 7f6cdbe00640 1 --2- 172.21.15.47:0/3482529035 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6cb8066e40 0x7f6cb8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:06.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.327+0000 7f6cdbe00640 1 --2- 172.21.15.47:0/3482529035 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6cb8066e40 0x7f6cb8069300 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0017c0 tx=0x7f6ccc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:06.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.331+0000 7f6cd8c00640 1 -- 172.21.15.47:0/3482529035 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6cc40ec780 con 0x7f6cdc0ce770 2024-02-19T15:27:06.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.495+0000 7f6ce2c00640 1 -- 172.21.15.47:0/3482529035 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6cdc068120 con 0x7f6cdc0ce770 2024-02-19T15:27:06.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.495+0000 7f6cd8c00640 1 -- 172.21.15.47:0/3482529035 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6cc40efe40 con 0x7f6cdc0ce770 2024-02-19T15:27:06.499 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:27:06.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.495+0000 7f6ce2c00640 1 -- 172.21.15.47:0/3482529035 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6cb8066e40 msgr2=0x7f6cb8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:06.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.495+0000 7f6ce2c00640 1 --2- 172.21.15.47:0/3482529035 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6cb8066e40 0x7f6cb8069300 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0017c0 tx=0x7f6ccc009040 comp rx=0 tx=0).stop 2024-02-19T15:27:06.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.495+0000 7f6ce2c00640 1 -- 172.21.15.47:0/3482529035 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cdc0ce770 msgr2=0x7f6cdc151ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:06.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.495+0000 7f6ce2c00640 1 --2- 172.21.15.47:0/3482529035 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cdc0ce770 0x7f6cdc151ca0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f6cc40ad040 tx=0x7f6cc401acb0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.495+0000 7f6ce2c00640 1 -- 172.21.15.47:0/3482529035 shutdown_connections 2024-02-19T15:27:06.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.495+0000 7f6ce2c00640 1 --2- 172.21.15.47:0/3482529035 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6cb8066e40 0x7f6cb8069300 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.495+0000 7f6ce2c00640 1 --2- 172.21.15.47:0/3482529035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cdc158a30 0x7f6cdc148120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.495+0000 7f6ce2c00640 1 --2- 172.21.15.47:0/3482529035 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cdc0ce770 0x7f6cdc151ca0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.495+0000 7f6ce2c00640 1 --2- 172.21.15.47:0/3482529035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cdc06df30 0x7f6cdc149ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.495+0000 7f6ce2c00640 1 -- 172.21.15.47:0/3482529035 >> 172.21.15.47:0/3482529035 conn(0x7f6cdc0b1b80 msgr2=0x7f6cdc0cec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:06.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.495+0000 7f6ce2c00640 1 -- 172.21.15.47:0/3482529035 shutdown_connections 2024-02-19T15:27:06.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.495+0000 7f6ce2c00640 1 -- 172.21.15.47:0/3482529035 wait complete. 2024-02-19T15:27:06.521 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:27:06.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.655+0000 7fe50ae00640 1 Processor -- start 2024-02-19T15:27:06.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.655+0000 7fe50ae00640 1 -- start start 2024-02-19T15:27:06.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.659+0000 7fe50ae00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe50415e080 0x7fe5040b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:06.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.659+0000 7fe50ae00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe50415e460 0x7fe5040ae630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:06.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.659+0000 7fe50ae00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5040ad590 con 0x7fe5040aeb70 2024-02-19T15:27:06.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.659+0000 7fe50ae00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5040ae210 con 0x7fe50415e080 2024-02-19T15:27:06.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.659+0000 7fe50ae00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5040ad870 con 0x7fe50415e460 2024-02-19T15:27:06.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.659+0000 7fe503e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe50415e080 0x7fe5040b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:06.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.659+0000 7fe4fb400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe50415e460 0x7fe5040ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:06.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.659+0000 7fe503e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe50415e080 0x7fe5040b4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56476/0 (socket says 172.21.15.47:56476) 2024-02-19T15:27:06.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.659+0000 7fe503e00640 1 -- 172.21.15.47:0/3103170041 learned_addr learned my addr 172.21.15.47:0/3103170041 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:06.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.659+0000 7fe508e00640 1 --1- 172.21.15.47:0/3103170041 >> v1:172.21.15.47:6789/0 conn(0x7fe5040aeb70 0x7fe5040af1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58540/0 (socket says 172.21.15.47:58540) 2024-02-19T15:27:06.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.659+0000 7fe503e00640 1 -- 172.21.15.47:0/3103170041 >> v2:172.21.15.116:3300/0 conn(0x7fe50415e460 msgr2=0x7fe5040ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:06.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.659+0000 7fe503e00640 1 --2- 172.21.15.47:0/3103170041 >> v2:172.21.15.116:3300/0 conn(0x7fe50415e460 0x7fe5040ae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.659+0000 7fe503e00640 1 -- 172.21.15.47:0/3103170041 >> v1:172.21.15.47:6789/0 conn(0x7fe5040aeb70 legacy=0x7fe5040af1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:06.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.659+0000 7fe503e00640 1 -- 172.21.15.47:0/3103170041 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5040adb50 con 0x7fe50415e080 2024-02-19T15:27:06.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.663+0000 7fe503e00640 1 --2- 172.21.15.47:0/3103170041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe50415e080 0x7fe5040b4f80 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fe504068800 tx=0x7fe4f401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=165e696fc73f22f9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:06.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.663+0000 7fe503400640 1 -- 172.21.15.47:0/3103170041 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f40aa610 con 0x7fe50415e080 2024-02-19T15:27:06.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.663+0000 7fe503400640 1 -- 172.21.15.47:0/3103170041 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f40a6040 con 0x7fe50415e080 2024-02-19T15:27:06.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.663+0000 7fe503400640 1 -- 172.21.15.47:0/3103170041 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f40aa990 con 0x7fe50415e080 2024-02-19T15:27:06.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.663+0000 7fe50ae00640 1 -- 172.21.15.47:0/3103170041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe50415e080 msgr2=0x7fe5040b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:06.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.663+0000 7fe50ae00640 1 --2- 172.21.15.47:0/3103170041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe50415e080 0x7fe5040b4f80 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fe504068800 tx=0x7fe4f401c490 comp rx=0 tx=0).stop 2024-02-19T15:27:06.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.663+0000 7fe50ae00640 1 -- 172.21.15.47:0/3103170041 shutdown_connections 2024-02-19T15:27:06.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.663+0000 7fe50ae00640 1 --2- 172.21.15.47:0/3103170041 >> v2:172.21.15.116:3300/0 conn(0x7fe50415e460 0x7fe5040ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.663+0000 7fe50ae00640 1 --2- 172.21.15.47:0/3103170041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe50415e080 0x7fe5040b4f80 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.663+0000 7fe50ae00640 1 -- 172.21.15.47:0/3103170041 >> 172.21.15.47:0/3103170041 conn(0x7fe5040b5d10 msgr2=0x7fe504072450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:06.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.663+0000 7fe50ae00640 1 -- 172.21.15.47:0/3103170041 shutdown_connections 2024-02-19T15:27:06.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.663+0000 7fe50ae00640 1 -- 172.21.15.47:0/3103170041 wait complete. 2024-02-19T15:27:06.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.663+0000 7fe50ae00640 1 Processor -- start 2024-02-19T15:27:06.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.663+0000 7fe50ae00640 1 -- start start 2024-02-19T15:27:06.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe50ae00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5040aeb70 0x7fe504149910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:06.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe50ae00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe50415e080 0x7fe50415e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:06.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe50ae00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe50415e460 0x7fe50414be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:06.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe50ae00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50415d8e0 con 0x7fe50415e460 2024-02-19T15:27:06.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe50ae00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe504155b30 con 0x7fe5040aeb70 2024-02-19T15:27:06.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe50ae00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe504155540 con 0x7fe50415e080 2024-02-19T15:27:06.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe508e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe50415e460 0x7fe50414be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:06.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe508e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe50415e460 0x7fe50414be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56478/0 (socket says 172.21.15.47:56478) 2024-02-19T15:27:06.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe508e00640 1 -- 172.21.15.47:0/1360126342 learned_addr learned my addr 172.21.15.47:0/1360126342 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:06.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe503e00640 1 --2- 172.21.15.47:0/1360126342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5040aeb70 0x7fe504149910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:06.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe4fb400640 1 --2- 172.21.15.47:0/1360126342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe50415e080 0x7fe50415e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:06.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe508e00640 1 -- 172.21.15.47:0/1360126342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe50415e080 msgr2=0x7fe50415e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:06.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe508e00640 1 --2- 172.21.15.47:0/1360126342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe50415e080 0x7fe50415e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe508e00640 1 -- 172.21.15.47:0/1360126342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5040aeb70 msgr2=0x7fe504149910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:06.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe508e00640 1 --2- 172.21.15.47:0/1360126342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5040aeb70 0x7fe504149910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe508e00640 1 -- 172.21.15.47:0/1360126342 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f40094d0 con 0x7fe50415e460 2024-02-19T15:27:06.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe508e00640 1 --2- 172.21.15.47:0/1360126342 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe50415e460 0x7fe50414be70 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc00c7d0 tx=0x7fe4fc00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:06.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe500c00640 1 -- 172.21.15.47:0/1360126342 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc00a640 con 0x7fe50415e460 2024-02-19T15:27:06.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe500c00640 1 -- 172.21.15.47:0/1360126342 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4fc00a7a0 con 0x7fe50415e460 2024-02-19T15:27:06.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe500c00640 1 -- 172.21.15.47:0/1360126342 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc00ab10 con 0x7fe50415e460 2024-02-19T15:27:06.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe50ae00640 1 -- 172.21.15.47:0/1360126342 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe504156200 con 0x7fe50415e460 2024-02-19T15:27:06.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe50ae00640 1 -- 172.21.15.47:0/1360126342 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5041567c0 con 0x7fe50415e460 2024-02-19T15:27:06.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe50ae00640 1 -- 172.21.15.47:0/1360126342 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe504155ca0 con 0x7fe50415e460 2024-02-19T15:27:06.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.667+0000 7fe500c00640 1 -- 172.21.15.47:0/1360126342 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe4fc00ac70 con 0x7fe50415e460 2024-02-19T15:27:06.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.671+0000 7fe500c00640 1 --2- 172.21.15.47:0/1360126342 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe4e0066e40 0x7fe4e0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:06.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.671+0000 7fe500c00640 1 -- 172.21.15.47:0/1360126342 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fe4fc0841e0 con 0x7fe50415e460 2024-02-19T15:27:06.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.671+0000 7fe503e00640 1 --2- 172.21.15.47:0/1360126342 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe4e0066e40 0x7fe4e0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:06.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.671+0000 7fe503e00640 1 --2- 172.21.15.47:0/1360126342 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe4e0066e40 0x7fe4e0069300 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe4f40ae000 tx=0x7fe4f40b5040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:06.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.675+0000 7fe500c00640 1 -- 172.21.15.47:0/1360126342 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe4fc050fc0 con 0x7fe50415e460 2024-02-19T15:27:06.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.867+0000 7fe50ae00640 1 -- 172.21.15.47:0/1360126342 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe504071300 con 0x7fe50415e460 2024-02-19T15:27:06.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.867+0000 7fe500c00640 1 -- 172.21.15.47:0/1360126342 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4947 (secure 0 0 0) 0x7fe4fc054680 con 0x7fe50415e460 2024-02-19T15:27:06.874 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:06.874 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":66,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7060,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/4006350936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":4006350936},{"type":"v1","addr":"172.21.15.116:6837","nonce":4006350936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/2191173523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2191173523},{"type":"v1","addr":"172.21.15.47:6834","nonce":2191173523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/1012301665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":1012301665},{"type":"v1","addr":"172.21.15.47:6835","nonce":1012301665}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:26:26.721109+0000","modified":"2024-02-19T15:26:43.866082+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7054":{"gid":7054,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6835/3852677195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3852677195},{"type":"v1","addr":"172.21.15.116:6835","nonce":3852677195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T15:27:06.874 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 66 2024-02-19T15:27:06.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.871+0000 7fe50ae00640 1 -- 172.21.15.47:0/1360126342 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe4e0066e40 msgr2=0x7fe4e0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:06.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.871+0000 7fe50ae00640 1 --2- 172.21.15.47:0/1360126342 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe4e0066e40 0x7fe4e0069300 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe4f40ae000 tx=0x7fe4f40b5040 comp rx=0 tx=0).stop 2024-02-19T15:27:06.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.871+0000 7fe50ae00640 1 -- 172.21.15.47:0/1360126342 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe50415e460 msgr2=0x7fe50414be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:06.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.871+0000 7fe50ae00640 1 --2- 172.21.15.47:0/1360126342 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe50415e460 0x7fe50414be70 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc00c7d0 tx=0x7fe4fc00cc90 comp rx=0 tx=0).stop 2024-02-19T15:27:06.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.871+0000 7fe50ae00640 1 -- 172.21.15.47:0/1360126342 shutdown_connections 2024-02-19T15:27:06.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.871+0000 7fe50ae00640 1 --2- 172.21.15.47:0/1360126342 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe4e0066e40 0x7fe4e0069300 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.871+0000 7fe50ae00640 1 --2- 172.21.15.47:0/1360126342 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe50415e460 0x7fe50414be70 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.871+0000 7fe50ae00640 1 --2- 172.21.15.47:0/1360126342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe50415e080 0x7fe50415e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.871+0000 7fe50ae00640 1 --2- 172.21.15.47:0/1360126342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5040aeb70 0x7fe504149910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:06.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.871+0000 7fe50ae00640 1 -- 172.21.15.47:0/1360126342 >> 172.21.15.47:0/1360126342 conn(0x7fe5040b5d10 msgr2=0x7fe5040b0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:06.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.871+0000 7fe50ae00640 1 -- 172.21.15.47:0/1360126342 shutdown_connections 2024-02-19T15:27:06.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:06.871+0000 7fe50ae00640 1 -- 172.21.15.47:0/1360126342 wait complete. 2024-02-19T15:27:06.894 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:27:07.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.031+0000 7f7d0a200640 1 Processor -- start 2024-02-19T15:27:07.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.031+0000 7f7d0a200640 1 -- start start 2024-02-19T15:27:07.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.031+0000 7f7d0a200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d041603a0 0x7f7d040cd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:07.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.031+0000 7f7d0a200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d04160b10 con 0x7f7d0415c1e0 2024-02-19T15:27:07.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.031+0000 7f7d0a200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0415c5c0 con 0x7f7d040b4420 2024-02-19T15:27:07.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.031+0000 7f7d0a200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d040cddd0 con 0x7f7d041603a0 2024-02-19T15:27:07.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.031+0000 7f7d03400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f7d0415c1e0 0x7f7d040d0600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58556/0 (socket says 172.21.15.47:58556) 2024-02-19T15:27:07.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.031+0000 7f7d03400640 1 -- 172.21.15.47:0/744407333 learned_addr learned my addr 172.21.15.47:0/744407333 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:07.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.031+0000 7f7d03e00640 1 --2- 172.21.15.47:0/744407333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d041603a0 0x7f7d040cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:07.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.031+0000 7f7d02000640 1 -- 172.21.15.47:0/744407333 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465487174 0 0) 0x7f7d04160b10 con 0x7f7d0415c1e0 2024-02-19T15:27:07.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.031+0000 7f7d02000640 1 -- 172.21.15.47:0/744407333 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ce8003530 con 0x7f7d0415c1e0 2024-02-19T15:27:07.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.031+0000 7f7d02000640 1 -- 172.21.15.47:0/744407333 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96365937 0 0) 0x7f7d0415c5c0 con 0x7f7d040b4420 2024-02-19T15:27:07.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.031+0000 7f7d02000640 1 -- 172.21.15.47:0/744407333 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d04160b10 con 0x7f7d040b4420 2024-02-19T15:27:07.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.031+0000 7f7d03e00640 1 -- 172.21.15.47:0/744407333 >> v1:172.21.15.116:6789/0 conn(0x7f7d040b4420 legacy=0x7f7d0415fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:07.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.031+0000 7f7d03e00640 1 -- 172.21.15.47:0/744407333 >> v1:172.21.15.47:6789/0 conn(0x7f7d0415c1e0 legacy=0x7f7d040d0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:07.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.031+0000 7f7d03e00640 1 -- 172.21.15.47:0/744407333 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d040d0d10 con 0x7f7d041603a0 2024-02-19T15:27:07.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.035+0000 7f7d03e00640 1 --2- 172.21.15.47:0/744407333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d041603a0 0x7f7d040cd430 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc009dc0 tx=0x7f7cfc01f170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89a37fb0f39cd4e4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:07.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.035+0000 7f7d02000640 1 -- 172.21.15.47:0/744407333 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc0af600 con 0x7f7d041603a0 2024-02-19T15:27:07.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.035+0000 7f7d02000640 1 -- 172.21.15.47:0/744407333 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cfc0ab050 con 0x7f7d041603a0 2024-02-19T15:27:07.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.035+0000 7f7d02000640 1 -- 172.21.15.47:0/744407333 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc0af980 con 0x7f7d041603a0 2024-02-19T15:27:07.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.035+0000 7f7d0a200640 1 -- 172.21.15.47:0/744407333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d041603a0 msgr2=0x7f7d040cd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:07.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.035+0000 7f7d0a200640 1 --2- 172.21.15.47:0/744407333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d041603a0 0x7f7d040cd430 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc009dc0 tx=0x7f7cfc01f170 comp rx=0 tx=0).stop 2024-02-19T15:27:07.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.035+0000 7f7d0a200640 1 -- 172.21.15.47:0/744407333 shutdown_connections 2024-02-19T15:27:07.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.035+0000 7f7d0a200640 1 --2- 172.21.15.47:0/744407333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d041603a0 0x7f7d040cd430 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.035+0000 7f7d0a200640 1 -- 172.21.15.47:0/744407333 >> 172.21.15.47:0/744407333 conn(0x7f7d040b3900 msgr2=0x7f7d040b3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:07.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.035+0000 7f7d0a200640 1 -- 172.21.15.47:0/744407333 shutdown_connections 2024-02-19T15:27:07.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.035+0000 7f7d0a200640 1 -- 172.21.15.47:0/744407333 wait complete. 2024-02-19T15:27:07.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.035+0000 7f7d0a200640 1 Processor -- start 2024-02-19T15:27:07.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d0a200640 1 -- start start 2024-02-19T15:27:07.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d0a200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d040b4420 0x7f7d041533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:07.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d0a200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d0415c1e0 0x7f7d040d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:07.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d0a200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d041603a0 0x7f7d040cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:07.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d0a200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0415d910 con 0x7f7d041603a0 2024-02-19T15:27:07.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d0a200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d040c9a30 con 0x7f7d0415c1e0 2024-02-19T15:27:07.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d0a200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d040c9d10 con 0x7f7d040b4420 2024-02-19T15:27:07.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d03e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d041603a0 0x7f7d040cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:07.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d03e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d041603a0 0x7f7d040cab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56480/0 (socket says 172.21.15.47:56480) 2024-02-19T15:27:07.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d03e00640 1 -- 172.21.15.47:0/2399460537 learned_addr learned my addr 172.21.15.47:0/2399460537 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:07.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d02a00640 1 --2- 172.21.15.47:0/2399460537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d0415c1e0 0x7f7d040d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:07.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d03400640 1 --2- 172.21.15.47:0/2399460537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d040b4420 0x7f7d041533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:07.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d03e00640 1 -- 172.21.15.47:0/2399460537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d040b4420 msgr2=0x7f7d041533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:07.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d03e00640 1 --2- 172.21.15.47:0/2399460537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d040b4420 0x7f7d041533e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d03e00640 1 -- 172.21.15.47:0/2399460537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d0415c1e0 msgr2=0x7f7d040d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:07.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d03e00640 1 --2- 172.21.15.47:0/2399460537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d0415c1e0 0x7f7d040d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d03e00640 1 -- 172.21.15.47:0/2399460537 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cfc009a70 con 0x7f7d041603a0 2024-02-19T15:27:07.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d03e00640 1 --2- 172.21.15.47:0/2399460537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d041603a0 0x7f7d040cab70 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc01bd80 tx=0x7f7cfc01bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:07.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7cfb400640 1 -- 172.21.15.47:0/2399460537 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc0afe90 con 0x7f7d041603a0 2024-02-19T15:27:07.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7cfb400640 1 -- 172.21.15.47:0/2399460537 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cfc009450 con 0x7f7d041603a0 2024-02-19T15:27:07.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7cfb400640 1 -- 172.21.15.47:0/2399460537 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc0b9650 con 0x7f7d041603a0 2024-02-19T15:27:07.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d0a200640 1 -- 172.21.15.47:0/2399460537 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d040c9ff0 con 0x7f7d041603a0 2024-02-19T15:27:07.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d0a200640 1 -- 172.21.15.47:0/2399460537 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d040d2c00 con 0x7f7d041603a0 2024-02-19T15:27:07.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.039+0000 7f7d0a200640 1 -- 172.21.15.47:0/2399460537 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d0415da80 con 0x7f7d041603a0 2024-02-19T15:27:07.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.043+0000 7f7cfb400640 1 -- 172.21.15.47:0/2399460537 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7cfc0b6050 con 0x7f7d041603a0 2024-02-19T15:27:07.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.043+0000 7f7cfb400640 1 --2- 172.21.15.47:0/2399460537 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ce4066e40 0x7f7ce4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:07.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.043+0000 7f7cfb400640 1 -- 172.21.15.47:0/2399460537 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f7cfc122dd0 con 0x7f7d041603a0 2024-02-19T15:27:07.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.043+0000 7f7d03400640 1 --2- 172.21.15.47:0/2399460537 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ce4066e40 0x7f7ce4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:07.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.043+0000 7f7d03400640 1 --2- 172.21.15.47:0/2399460537 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ce4066e40 0x7f7ce4069300 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4007fd0 tx=0x7f7cf4007d60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:07.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.047+0000 7f7cfb400640 1 -- 172.21.15.47:0/2399460537 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7cfc0f3790 con 0x7f7d041603a0 2024-02-19T15:27:07.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.211+0000 7f7d0a200640 1 -- 172.21.15.47:0/2399460537 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d04068120 con 0x7f7d041603a0 2024-02-19T15:27:07.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.211+0000 7f7cfb400640 1 -- 172.21.15.47:0/2399460537 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16875 (secure 0 0 0) 0x7f7cfc0b4020 con 0x7f7d041603a0 2024-02-19T15:27:07.216 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:07.216 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":97,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:26:40.823963+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:26:25.124594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-02-19T15:26:25.706971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:6833/68555355":"2024-02-20T15:26:40.818800+0000","172.21.15.47:6832/68555355":"2024-02-20T15:26:40.818800+0000","172.21.15.116:6837/547176070":"2024-02-20T15:26:39.960651+0000","172.21.15.116:6835/547176070":"2024-02-20T15:26:39.960651+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:27:07.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.215+0000 7f7d0a200640 1 -- 172.21.15.47:0/2399460537 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ce4066e40 msgr2=0x7f7ce4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:07.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.215+0000 7f7d0a200640 1 --2- 172.21.15.47:0/2399460537 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ce4066e40 0x7f7ce4069300 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4007fd0 tx=0x7f7cf4007d60 comp rx=0 tx=0).stop 2024-02-19T15:27:07.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.215+0000 7f7d0a200640 1 -- 172.21.15.47:0/2399460537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d041603a0 msgr2=0x7f7d040cab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:07.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.215+0000 7f7d0a200640 1 --2- 172.21.15.47:0/2399460537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d041603a0 0x7f7d040cab70 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc01bd80 tx=0x7f7cfc01bf10 comp rx=0 tx=0).stop 2024-02-19T15:27:07.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.215+0000 7f7d0a200640 1 -- 172.21.15.47:0/2399460537 shutdown_connections 2024-02-19T15:27:07.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.215+0000 7f7d0a200640 1 --2- 172.21.15.47:0/2399460537 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7ce4066e40 0x7f7ce4069300 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.215+0000 7f7d0a200640 1 --2- 172.21.15.47:0/2399460537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d041603a0 0x7f7d040cab70 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.215+0000 7f7d0a200640 1 --2- 172.21.15.47:0/2399460537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d0415c1e0 0x7f7d040d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.215+0000 7f7d0a200640 1 --2- 172.21.15.47:0/2399460537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d040b4420 0x7f7d041533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.215+0000 7f7d0a200640 1 -- 172.21.15.47:0/2399460537 >> 172.21.15.47:0/2399460537 conn(0x7f7d040b3900 msgr2=0x7f7d040b16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:07.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.215+0000 7f7d0a200640 1 -- 172.21.15.47:0/2399460537 shutdown_connections 2024-02-19T15:27:07.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.215+0000 7f7d0a200640 1 -- 172.21.15.47:0/2399460537 wait complete. 2024-02-19T15:27:07.237 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T15:27:07.237 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:27:07.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.375+0000 7fd51e000640 1 Processor -- start 2024-02-19T15:27:07.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.375+0000 7fd51e000640 1 -- start start 2024-02-19T15:27:07.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.375+0000 7fd51e000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd51806cb70 0x7fd5180b54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:07.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.375+0000 7fd51e000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd5180b5a30 0x7fd518158640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:07.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.375+0000 7fd51e000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5180b5e40 con 0x7fd51806cf50 2024-02-19T15:27:07.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.375+0000 7fd51e000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd518158b80 con 0x7fd51806cb70 2024-02-19T15:27:07.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.375+0000 7fd51e000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5180b6070 con 0x7fd5180b5a30 2024-02-19T15:27:07.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.375+0000 7fd51cc00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd51806cb70 0x7fd5180b54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:07.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.375+0000 7fd51cc00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd51806cb70 0x7fd5180b54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56486/0 (socket says 172.21.15.47:56486) 2024-02-19T15:27:07.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.375+0000 7fd51cc00640 1 -- 172.21.15.47:0/1150470228 learned_addr learned my addr 172.21.15.47:0/1150470228 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:07.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.375+0000 7fd51d600640 1 --2- 172.21.15.47:0/1150470228 >> v2:172.21.15.116:3300/0 conn(0x7fd5180b5a30 0x7fd518158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:07.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.379+0000 7fd51cc00640 1 -- 172.21.15.47:0/1150470228 >> v2:172.21.15.116:3300/0 conn(0x7fd5180b5a30 msgr2=0x7fd518158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:07.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.379+0000 7fd51cc00640 1 --2- 172.21.15.47:0/1150470228 >> v2:172.21.15.116:3300/0 conn(0x7fd5180b5a30 0x7fd518158640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.379+0000 7fd51cc00640 1 -- 172.21.15.47:0/1150470228 >> v1:172.21.15.116:6789/0 conn(0x7fd51806cf50 legacy=0x7fd518157f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:07.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.379+0000 7fd51cc00640 1 -- 172.21.15.47:0/1150470228 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5180b6350 con 0x7fd51806cb70 2024-02-19T15:27:07.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.379+0000 7fd51cc00640 1 --2- 172.21.15.47:0/1150470228 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd51806cb70 0x7fd5180b54f0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fd518072300 tx=0x7fd50c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e3f8de023f9f99c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:07.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.379+0000 7fd517400640 1 -- 172.21.15.47:0/1150470228 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c0aa650 con 0x7fd51806cb70 2024-02-19T15:27:07.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.379+0000 7fd517400640 1 -- 172.21.15.47:0/1150470228 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd50c0aa7b0 con 0x7fd51806cb70 2024-02-19T15:27:07.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.379+0000 7fd517400640 1 -- 172.21.15.47:0/1150470228 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c0aab40 con 0x7fd51806cb70 2024-02-19T15:27:07.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.379+0000 7fd51e000640 1 -- 172.21.15.47:0/1150470228 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd51806cb70 msgr2=0x7fd5180b54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:07.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.379+0000 7fd51e000640 1 --2- 172.21.15.47:0/1150470228 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd51806cb70 0x7fd5180b54f0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fd518072300 tx=0x7fd50c01c4f0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.379+0000 7fd51e000640 1 -- 172.21.15.47:0/1150470228 shutdown_connections 2024-02-19T15:27:07.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.379+0000 7fd51e000640 1 --2- 172.21.15.47:0/1150470228 >> v2:172.21.15.116:3300/0 conn(0x7fd5180b5a30 0x7fd518158640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.379+0000 7fd51e000640 1 --2- 172.21.15.47:0/1150470228 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd51806cb70 0x7fd5180b54f0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.379+0000 7fd51e000640 1 -- 172.21.15.47:0/1150470228 >> 172.21.15.47:0/1150470228 conn(0x7fd51809aa70 msgr2=0x7fd51809ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:07.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.379+0000 7fd51e000640 1 -- 172.21.15.47:0/1150470228 shutdown_connections 2024-02-19T15:27:07.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.379+0000 7fd51e000640 1 -- 172.21.15.47:0/1150470228 wait complete. 2024-02-19T15:27:07.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd51e000640 1 Processor -- start 2024-02-19T15:27:07.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd51e000640 1 -- start start 2024-02-19T15:27:07.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd51e000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd51806cb70 0x7fd518149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:07.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd51e000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd51806cf50 0x7fd518153db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:07.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd51e000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd5180b5a30 0x7fd5181f7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:07.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd51e000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd518149e80 con 0x7fd51806cb70 2024-02-19T15:27:07.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd51e000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5181542f0 con 0x7fd51806cf50 2024-02-19T15:27:07.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd51e000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5181545c0 con 0x7fd5180b5a30 2024-02-19T15:27:07.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd51cc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd51806cb70 0x7fd518149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:07.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd51cc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd51806cb70 0x7fd518149940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56492/0 (socket says 172.21.15.47:56492) 2024-02-19T15:27:07.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd51cc00640 1 -- 172.21.15.47:0/1177095955 learned_addr learned my addr 172.21.15.47:0/1177095955 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:07.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd51d600640 1 --2- 172.21.15.47:0/1177095955 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd5180b5a30 0x7fd5181f7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:07.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd517e00640 1 --2- 172.21.15.47:0/1177095955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd51806cf50 0x7fd518153db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:07.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd51cc00640 1 -- 172.21.15.47:0/1177095955 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd5180b5a30 msgr2=0x7fd5181f7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:07.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd51cc00640 1 --2- 172.21.15.47:0/1177095955 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd5180b5a30 0x7fd5181f7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd51cc00640 1 -- 172.21.15.47:0/1177095955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd51806cf50 msgr2=0x7fd518153db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:07.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd51cc00640 1 --2- 172.21.15.47:0/1177095955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd51806cf50 0x7fd518153db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd51cc00640 1 -- 172.21.15.47:0/1177095955 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c0094d0 con 0x7fd51806cb70 2024-02-19T15:27:07.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd51cc00640 1 --2- 172.21.15.47:0/1177095955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd51806cb70 0x7fd518149940 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fd50c019100 tx=0x7fd50c01ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:07.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd514c00640 1 -- 172.21.15.47:0/1177095955 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c0b3800 con 0x7fd51806cb70 2024-02-19T15:27:07.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd514c00640 1 -- 172.21.15.47:0/1177095955 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd50c0b3980 con 0x7fd51806cb70 2024-02-19T15:27:07.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd51e000640 1 -- 172.21.15.47:0/1177095955 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5181548a0 con 0x7fd51806cb70 2024-02-19T15:27:07.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd514c00640 1 -- 172.21.15.47:0/1177095955 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c0b3d30 con 0x7fd51806cb70 2024-02-19T15:27:07.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.383+0000 7fd51e000640 1 -- 172.21.15.47:0/1177095955 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5181f6990 con 0x7fd51806cb70 2024-02-19T15:27:07.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.387+0000 7fd51e000640 1 -- 172.21.15.47:0/1177095955 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5180d2c40 con 0x7fd51806cb70 2024-02-19T15:27:07.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.387+0000 7fd514c00640 1 -- 172.21.15.47:0/1177095955 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd50c0c6440 con 0x7fd51806cb70 2024-02-19T15:27:07.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.387+0000 7fd514c00640 1 --2- 172.21.15.47:0/1177095955 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd4f8066e40 0x7fd4f8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:07.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.387+0000 7fd514c00640 1 -- 172.21.15.47:0/1177095955 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fd50c11f690 con 0x7fd51806cb70 2024-02-19T15:27:07.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.387+0000 7fd517e00640 1 --2- 172.21.15.47:0/1177095955 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd4f8066e40 0x7fd4f8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:07.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.387+0000 7fd517e00640 1 --2- 172.21.15.47:0/1177095955 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd4f8066e40 0x7fd4f8069300 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd5181f9d50 tx=0x7fd504009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:07.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.391+0000 7fd514c00640 1 -- 172.21.15.47:0/1177095955 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd50c0ec470 con 0x7fd51806cb70 2024-02-19T15:27:07.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.587+0000 7fd51e000640 1 -- 172.21.15.47:0/1177095955 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd518068040 con 0x7fd51806cb70 2024-02-19T15:27:07.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.587+0000 7fd514c00640 1 -- 172.21.15.47:0/1177095955 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4947 (secure 0 0 0) 0x7fd50c0aa610 con 0x7fd51806cb70 2024-02-19T15:27:07.594 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:07.594 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":66,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7060,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/4006350936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":4006350936},{"type":"v1","addr":"172.21.15.116:6837","nonce":4006350936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/2191173523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2191173523},{"type":"v1","addr":"172.21.15.47:6834","nonce":2191173523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/1012301665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":1012301665},{"type":"v1","addr":"172.21.15.47:6835","nonce":1012301665}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:26:26.721109+0000","modified":"2024-02-19T15:26:43.866082+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7054":{"gid":7054,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6835/3852677195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3852677195},{"type":"v1","addr":"172.21.15.116:6835","nonce":3852677195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T15:27:07.594 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 66 2024-02-19T15:27:07.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.591+0000 7fd51e000640 1 -- 172.21.15.47:0/1177095955 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd4f8066e40 msgr2=0x7fd4f8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:07.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.591+0000 7fd51e000640 1 --2- 172.21.15.47:0/1177095955 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd4f8066e40 0x7fd4f8069300 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd5181f9d50 tx=0x7fd504009040 comp rx=0 tx=0).stop 2024-02-19T15:27:07.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.591+0000 7fd51e000640 1 -- 172.21.15.47:0/1177095955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd51806cb70 msgr2=0x7fd518149940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:07.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.591+0000 7fd51e000640 1 --2- 172.21.15.47:0/1177095955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd51806cb70 0x7fd518149940 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fd50c019100 tx=0x7fd50c01ad30 comp rx=0 tx=0).stop 2024-02-19T15:27:07.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.591+0000 7fd51e000640 1 -- 172.21.15.47:0/1177095955 shutdown_connections 2024-02-19T15:27:07.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.591+0000 7fd51e000640 1 --2- 172.21.15.47:0/1177095955 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd4f8066e40 0x7fd4f8069300 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.591+0000 7fd51e000640 1 --2- 172.21.15.47:0/1177095955 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd5180b5a30 0x7fd5181f7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.591+0000 7fd51e000640 1 --2- 172.21.15.47:0/1177095955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd51806cf50 0x7fd518153db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.591+0000 7fd51e000640 1 --2- 172.21.15.47:0/1177095955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd51806cb70 0x7fd518149940 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.591+0000 7fd51e000640 1 -- 172.21.15.47:0/1177095955 >> 172.21.15.47:0/1177095955 conn(0x7fd51809aa70 msgr2=0x7fd51806ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:07.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.591+0000 7fd51e000640 1 -- 172.21.15.47:0/1177095955 shutdown_connections 2024-02-19T15:27:07.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.591+0000 7fd51e000640 1 -- 172.21.15.47:0/1177095955 wait complete. 2024-02-19T15:27:07.614 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:27:07.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.747+0000 7eff5f600640 1 Processor -- start 2024-02-19T15:27:07.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.747+0000 7eff5f600640 1 -- start start 2024-02-19T15:27:07.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.751+0000 7eff5f600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff58158a30 0x7eff5815bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:07.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.751+0000 7eff5f600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7eff5806e390 0x7eff580724d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:07.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.751+0000 7eff5f600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff5815c300 con 0x7eff5806df30 2024-02-19T15:27:07.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.751+0000 7eff5f600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff58158e10 con 0x7eff5806e390 2024-02-19T15:27:07.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.751+0000 7eff5f600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff580cf780 con 0x7eff58158a30 2024-02-19T15:27:07.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.751+0000 7eff5d600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7eff5806e390 0x7eff580724d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:07.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.751+0000 7eff57e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7eff5806df30 0x7eff58071dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58566/0 (socket says 172.21.15.47:58566) 2024-02-19T15:27:07.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.751+0000 7eff5cc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff58158a30 0x7eff5815bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:07.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.751+0000 7eff57e00640 1 -- 172.21.15.47:0/3706780606 learned_addr learned my addr 172.21.15.47:0/3706780606 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:07.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.751+0000 7eff5d600640 1 -- 172.21.15.47:0/3706780606 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff58158a30 msgr2=0x7eff5815bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:07.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.751+0000 7eff5d600640 1 --2- 172.21.15.47:0/3706780606 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff58158a30 0x7eff5815bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.751+0000 7eff5d600640 1 -- 172.21.15.47:0/3706780606 >> v1:172.21.15.47:6789/0 conn(0x7eff5806df30 legacy=0x7eff58071dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:07.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.751+0000 7eff5d600640 1 -- 172.21.15.47:0/3706780606 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5815ec20 con 0x7eff5806e390 2024-02-19T15:27:07.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.751+0000 7eff57400640 1 -- 172.21.15.47:0/3706780606 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787649187 0 0) 0x7eff5815c300 con 0x7eff5806df30 2024-02-19T15:27:07.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.751+0000 7eff5cc00640 1 --2- 172.21.15.47:0/3706780606 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff58158a30 0x7eff5815bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:27:07.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.755+0000 7eff5d600640 1 --2- 172.21.15.47:0/3706780606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5806e390 0x7eff580724d0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7eff50009d90 tx=0x7eff5001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69edd14950d88f00 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:07.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.755+0000 7eff57400640 1 -- 172.21.15.47:0/3706780606 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff500a5a00 con 0x7eff5806e390 2024-02-19T15:27:07.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.755+0000 7eff57400640 1 -- 172.21.15.47:0/3706780606 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff500a5b60 con 0x7eff5806e390 2024-02-19T15:27:07.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.755+0000 7eff57400640 1 -- 172.21.15.47:0/3706780606 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff500ad440 con 0x7eff5806e390 2024-02-19T15:27:07.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.755+0000 7eff5f600640 1 -- 172.21.15.47:0/3706780606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5806e390 msgr2=0x7eff580724d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:07.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.755+0000 7eff5f600640 1 --2- 172.21.15.47:0/3706780606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5806e390 0x7eff580724d0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7eff50009d90 tx=0x7eff5001c940 comp rx=0 tx=0).stop 2024-02-19T15:27:07.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.755+0000 7eff5f600640 1 -- 172.21.15.47:0/3706780606 shutdown_connections 2024-02-19T15:27:07.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.755+0000 7eff5f600640 1 --2- 172.21.15.47:0/3706780606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5806e390 0x7eff580724d0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.755+0000 7eff5f600640 1 --2- 172.21.15.47:0/3706780606 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff58158a30 0x7eff5815bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.755+0000 7eff5f600640 1 -- 172.21.15.47:0/3706780606 >> 172.21.15.47:0/3706780606 conn(0x7eff580b1b80 msgr2=0x7eff580b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:07.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.755+0000 7eff5f600640 1 -- 172.21.15.47:0/3706780606 shutdown_connections 2024-02-19T15:27:07.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.755+0000 7eff5f600640 1 -- 172.21.15.47:0/3706780606 wait complete. 2024-02-19T15:27:07.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.755+0000 7eff5f600640 1 Processor -- start 2024-02-19T15:27:07.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.755+0000 7eff5f600640 1 -- start start 2024-02-19T15:27:07.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.755+0000 7eff5f600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff5806df30 0x7eff58144920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:07.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.755+0000 7eff5f600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5806e390 0x7eff5813c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:07.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.755+0000 7eff5f600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff58158a30 0x7eff58072250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:07.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.755+0000 7eff5f600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff58145d80 con 0x7eff5806e390 2024-02-19T15:27:07.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.755+0000 7eff5f600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff58145170 con 0x7eff58158a30 2024-02-19T15:27:07.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.755+0000 7eff5f600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff58145450 con 0x7eff5806df30 2024-02-19T15:27:07.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.759+0000 7eff57e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5806e390 0x7eff5813c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:07.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.759+0000 7eff5cc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff5806df30 0x7eff58144920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:07.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.759+0000 7eff57e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5806e390 0x7eff5813c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56508/0 (socket says 172.21.15.47:56508) 2024-02-19T15:27:07.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.759+0000 7eff57e00640 1 -- 172.21.15.47:0/3370043054 learned_addr learned my addr 172.21.15.47:0/3370043054 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:07.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.759+0000 7eff5d600640 1 --2- 172.21.15.47:0/3370043054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff58158a30 0x7eff58072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:07.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.759+0000 7eff5cc00640 1 -- 172.21.15.47:0/3370043054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff58158a30 msgr2=0x7eff58072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:07.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.759+0000 7eff5cc00640 1 --2- 172.21.15.47:0/3370043054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff58158a30 0x7eff58072250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.759+0000 7eff5cc00640 1 -- 172.21.15.47:0/3370043054 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5806e390 msgr2=0x7eff5813c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:07.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.759+0000 7eff5cc00640 1 --2- 172.21.15.47:0/3370043054 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5806e390 0x7eff5813c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.759+0000 7eff5cc00640 1 -- 172.21.15.47:0/3370043054 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff4c009420 con 0x7eff5806df30 2024-02-19T15:27:07.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.759+0000 7eff5cc00640 1 --2- 172.21.15.47:0/3370043054 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff5806df30 0x7eff58144920 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7eff4c00e9b0 tx=0x7eff4c00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:07.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.759+0000 7eff54c00640 1 -- 172.21.15.47:0/3370043054 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c00c9a0 con 0x7eff5806df30 2024-02-19T15:27:07.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.759+0000 7eff54c00640 1 -- 172.21.15.47:0/3370043054 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff4c00cb00 con 0x7eff5806df30 2024-02-19T15:27:07.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.759+0000 7eff54c00640 1 -- 172.21.15.47:0/3370043054 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c013470 con 0x7eff5806df30 2024-02-19T15:27:07.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.759+0000 7eff5f600640 1 -- 172.21.15.47:0/3370043054 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff50009a70 con 0x7eff5806df30 2024-02-19T15:27:07.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.759+0000 7eff5f600640 1 -- 172.21.15.47:0/3370043054 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff58145a10 con 0x7eff5806df30 2024-02-19T15:27:07.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.759+0000 7eff5f600640 1 -- 172.21.15.47:0/3370043054 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff58068120 con 0x7eff5806df30 2024-02-19T15:27:07.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.759+0000 7eff54c00640 1 -- 172.21.15.47:0/3370043054 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7eff4c0135d0 con 0x7eff5806df30 2024-02-19T15:27:07.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.763+0000 7eff54c00640 1 --2- 172.21.15.47:0/3370043054 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7eff38066e40 0x7eff38069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:07.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.763+0000 7eff54c00640 1 -- 172.21.15.47:0/3370043054 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7eff4c084cb0 con 0x7eff5806df30 2024-02-19T15:27:07.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.763+0000 7eff57e00640 1 --2- 172.21.15.47:0/3370043054 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7eff38066e40 0x7eff38069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:07.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.763+0000 7eff57e00640 1 --2- 172.21.15.47:0/3370043054 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7eff38066e40 0x7eff38069300 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7eff58146970 tx=0x7eff44007eb0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:07.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.767+0000 7eff54c00640 1 -- 172.21.15.47:0/3370043054 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7eff4c051a90 con 0x7eff5806df30 2024-02-19T15:27:07.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.959+0000 7eff5f600640 1 -- 172.21.15.47:0/3370043054 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff58145bd0 con 0x7eff5806df30 2024-02-19T15:27:07.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.963+0000 7eff54c00640 1 -- 172.21.15.47:0/3370043054 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4947 (secure 0 0 0) 0x7eff4c055150 con 0x7eff5806df30 2024-02-19T15:27:07.966 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:07.966 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":66,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7060,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/4006350936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":4006350936},{"type":"v1","addr":"172.21.15.116:6837","nonce":4006350936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/2191173523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2191173523},{"type":"v1","addr":"172.21.15.47:6834","nonce":2191173523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/1012301665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":1012301665},{"type":"v1","addr":"172.21.15.47:6835","nonce":1012301665}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:26:26.721109+0000","modified":"2024-02-19T15:26:43.866082+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7054":{"gid":7054,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6835/3852677195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3852677195},{"type":"v1","addr":"172.21.15.116:6835","nonce":3852677195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T15:27:07.966 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 66 2024-02-19T15:27:07.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.963+0000 7eff5f600640 1 -- 172.21.15.47:0/3370043054 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7eff38066e40 msgr2=0x7eff38069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:07.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.963+0000 7eff5f600640 1 --2- 172.21.15.47:0/3370043054 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7eff38066e40 0x7eff38069300 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7eff58146970 tx=0x7eff44007eb0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.963+0000 7eff5f600640 1 -- 172.21.15.47:0/3370043054 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff5806df30 msgr2=0x7eff58144920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.963+0000 7eff5f600640 1 --2- 172.21.15.47:0/3370043054 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff5806df30 0x7eff58144920 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7eff4c00e9b0 tx=0x7eff4c00ee70 comp rx=0 tx=0).stop 2024-02-19T15:27:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.963+0000 7eff5f600640 1 -- 172.21.15.47:0/3370043054 shutdown_connections 2024-02-19T15:27:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.963+0000 7eff5f600640 1 --2- 172.21.15.47:0/3370043054 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7eff38066e40 0x7eff38069300 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.963+0000 7eff5f600640 1 --2- 172.21.15.47:0/3370043054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff58158a30 0x7eff58072250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.963+0000 7eff5f600640 1 --2- 172.21.15.47:0/3370043054 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5806e390 0x7eff5813c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.963+0000 7eff5f600640 1 --2- 172.21.15.47:0/3370043054 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff5806df30 0x7eff58144920 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.963+0000 7eff5f600640 1 -- 172.21.15.47:0/3370043054 >> 172.21.15.47:0/3370043054 conn(0x7eff580b1b80 msgr2=0x7eff581473e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.963+0000 7eff5f600640 1 -- 172.21.15.47:0/3370043054 shutdown_connections 2024-02-19T15:27:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:07.963+0000 7eff5f600640 1 -- 172.21.15.47:0/3370043054 wait complete. 2024-02-19T15:27:07.988 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:27:08.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.123+0000 7f8c20200640 1 Processor -- start 2024-02-19T15:27:08.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.123+0000 7f8c20200640 1 -- start start 2024-02-19T15:27:08.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.123+0000 7f8c20200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8c181478b0 0x7f8c18162750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:08.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.123+0000 7f8c20200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8c180ae2f0 0x7f8c1806df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:08.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.123+0000 7f8c20200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c1806e470 0x7f8c18071180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:08.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.123+0000 7f8c20200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c1806e8e0 con 0x7f8c181478b0 2024-02-19T15:27:08.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.123+0000 7f8c20200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c180b3240 con 0x7f8c180ae2f0 2024-02-19T15:27:08.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.123+0000 7f8c20200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c18161c70 con 0x7f8c1806e470 2024-02-19T15:27:08.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.123+0000 7f8c1d800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8c181478b0 0x7f8c18162750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:08.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.123+0000 7f8c1d800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8c181478b0 0x7f8c18162750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56510/0 (socket says 172.21.15.47:56510) 2024-02-19T15:27:08.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.123+0000 7f8c1d800640 1 -- 172.21.15.47:0/3572064902 learned_addr learned my addr 172.21.15.47:0/3572064902 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:08.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.123+0000 7f8c1ce00640 1 --2- 172.21.15.47:0/3572064902 >> v2:172.21.15.116:3300/0 conn(0x7f8c180ae2f0 0x7f8c1806df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:08.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.123+0000 7f8c1e200640 1 --2- 172.21.15.47:0/3572064902 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c1806e470 0x7f8c18071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:08.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.127+0000 7f8c1d800640 1 -- 172.21.15.47:0/3572064902 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c1806e470 msgr2=0x7f8c18071180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:08.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.127+0000 7f8c1d800640 1 --2- 172.21.15.47:0/3572064902 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c1806e470 0x7f8c18071180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.127+0000 7f8c1d800640 1 -- 172.21.15.47:0/3572064902 >> v2:172.21.15.116:3300/0 conn(0x7f8c180ae2f0 msgr2=0x7f8c1806df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:08.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.127+0000 7f8c1d800640 1 --2- 172.21.15.47:0/3572064902 >> v2:172.21.15.116:3300/0 conn(0x7f8c180ae2f0 0x7f8c1806df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.127+0000 7f8c1d800640 1 -- 172.21.15.47:0/3572064902 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c18161f50 con 0x7f8c181478b0 2024-02-19T15:27:08.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.127+0000 7f8c1ce00640 1 --2- 172.21.15.47:0/3572064902 >> v2:172.21.15.116:3300/0 conn(0x7f8c180ae2f0 0x7f8c1806df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:27:08.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.127+0000 7f8c1d800640 1 --2- 172.21.15.47:0/3572064902 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c181478b0 0x7f8c18162750 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f8c18073740 tx=0x7f8c0c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45ad34d0c61943d6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:08.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.127+0000 7f8c17e00640 1 -- 172.21.15.47:0/3572064902 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c0ab640 con 0x7f8c181478b0 2024-02-19T15:27:08.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.127+0000 7f8c17e00640 1 -- 172.21.15.47:0/3572064902 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0c0a7070 con 0x7f8c181478b0 2024-02-19T15:27:08.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.127+0000 7f8c17e00640 1 -- 172.21.15.47:0/3572064902 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c0ab9c0 con 0x7f8c181478b0 2024-02-19T15:27:08.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.127+0000 7f8c20200640 1 -- 172.21.15.47:0/3572064902 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c181478b0 msgr2=0x7f8c18162750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:08.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.127+0000 7f8c20200640 1 --2- 172.21.15.47:0/3572064902 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c181478b0 0x7f8c18162750 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f8c18073740 tx=0x7f8c0c01c4f0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c20200640 1 -- 172.21.15.47:0/3572064902 shutdown_connections 2024-02-19T15:27:08.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c20200640 1 --2- 172.21.15.47:0/3572064902 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c1806e470 0x7f8c18071180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c20200640 1 --2- 172.21.15.47:0/3572064902 >> v2:172.21.15.116:3300/0 conn(0x7f8c180ae2f0 0x7f8c1806df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c20200640 1 --2- 172.21.15.47:0/3572064902 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c181478b0 0x7f8c18162750 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c20200640 1 -- 172.21.15.47:0/3572064902 >> 172.21.15.47:0/3572064902 conn(0x7f8c180b1b80 msgr2=0x7f8c180b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:08.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c20200640 1 -- 172.21.15.47:0/3572064902 shutdown_connections 2024-02-19T15:27:08.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c20200640 1 -- 172.21.15.47:0/3572064902 wait complete. 2024-02-19T15:27:08.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c20200640 1 Processor -- start 2024-02-19T15:27:08.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c20200640 1 -- start start 2024-02-19T15:27:08.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c20200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c1806e470 0x7f8c18160c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:08.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c20200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c180ae2f0 0x7f8c1813d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:08.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c20200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c181478b0 0x7f8c18129ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:08.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c20200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c181433b0 con 0x7f8c181478b0 2024-02-19T15:27:08.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c20200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c18143690 con 0x7f8c180ae2f0 2024-02-19T15:27:08.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c20200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c18143970 con 0x7f8c1806e470 2024-02-19T15:27:08.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c1e200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c181478b0 0x7f8c18129ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:08.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c1e200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c181478b0 0x7f8c18129ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56512/0 (socket says 172.21.15.47:56512) 2024-02-19T15:27:08.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c1e200640 1 -- 172.21.15.47:0/2383741054 learned_addr learned my addr 172.21.15.47:0/2383741054 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:08.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c1ce00640 1 --2- 172.21.15.47:0/2383741054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c180ae2f0 0x7f8c1813d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:08.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c1d800640 1 --2- 172.21.15.47:0/2383741054 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c1806e470 0x7f8c18160c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:08.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c1ce00640 1 -- 172.21.15.47:0/2383741054 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c1806e470 msgr2=0x7f8c18160c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:08.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c1ce00640 1 --2- 172.21.15.47:0/2383741054 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c1806e470 0x7f8c18160c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c1ce00640 1 -- 172.21.15.47:0/2383741054 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c181478b0 msgr2=0x7f8c18129ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:08.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c1ce00640 1 --2- 172.21.15.47:0/2383741054 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c181478b0 0x7f8c18129ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c1ce00640 1 -- 172.21.15.47:0/2383741054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c04009420 con 0x7f8c180ae2f0 2024-02-19T15:27:08.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c1e200640 1 --2- 172.21.15.47:0/2383741054 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c181478b0 0x7f8c18129ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:27:08.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c1ce00640 1 --2- 172.21.15.47:0/2383741054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c180ae2f0 0x7f8c1813d2c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400e9b0 tx=0x7f8c0400ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:08.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.131+0000 7f8c15600640 1 -- 172.21.15.47:0/2383741054 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0400c9a0 con 0x7f8c180ae2f0 2024-02-19T15:27:08.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.135+0000 7f8c15600640 1 -- 172.21.15.47:0/2383741054 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0400cb00 con 0x7f8c180ae2f0 2024-02-19T15:27:08.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.135+0000 7f8c15600640 1 -- 172.21.15.47:0/2383741054 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04013470 con 0x7f8c180ae2f0 2024-02-19T15:27:08.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.135+0000 7f8c20200640 1 -- 172.21.15.47:0/2383741054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0c0094d0 con 0x7f8c180ae2f0 2024-02-19T15:27:08.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.135+0000 7f8c20200640 1 -- 172.21.15.47:0/2383741054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c18143f30 con 0x7f8c180ae2f0 2024-02-19T15:27:08.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.135+0000 7f8c20200640 1 -- 172.21.15.47:0/2383741054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c18068120 con 0x7f8c180ae2f0 2024-02-19T15:27:08.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.135+0000 7f8c15600640 1 -- 172.21.15.47:0/2383741054 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8c040135d0 con 0x7f8c180ae2f0 2024-02-19T15:27:08.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.135+0000 7f8c15600640 1 --2- 172.21.15.47:0/2383741054 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8bf8066e70 0x7f8bf8069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:08.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.135+0000 7f8c15600640 1 -- 172.21.15.47:0/2383741054 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f8c04084be0 con 0x7f8c180ae2f0 2024-02-19T15:27:08.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.135+0000 7f8c1d800640 1 --2- 172.21.15.47:0/2383741054 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8bf8066e70 0x7f8bf8069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:08.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.135+0000 7f8c1d800640 1 --2- 172.21.15.47:0/2383741054 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8bf8066e70 0x7f8bf8069330 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0af000 tx=0x7f8c0c0b6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:08.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.139+0000 7f8c15600640 1 -- 172.21.15.47:0/2383741054 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8c040519c0 con 0x7f8c180ae2f0 2024-02-19T15:27:08.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.303+0000 7f8c20200640 1 -- 172.21.15.47:0/2383741054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c180cc670 con 0x7f8c180ae2f0 2024-02-19T15:27:08.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.307+0000 7f8c15600640 1 -- 172.21.15.47:0/2383741054 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16875 (secure 0 0 0) 0x7f8c04055080 con 0x7f8c180ae2f0 2024-02-19T15:27:08.311 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:08.311 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":97,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:26:40.823963+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:26:25.124594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-02-19T15:26:25.706971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:6833/68555355":"2024-02-20T15:26:40.818800+0000","172.21.15.47:6832/68555355":"2024-02-20T15:26:40.818800+0000","172.21.15.116:6837/547176070":"2024-02-20T15:26:39.960651+0000","172.21.15.116:6835/547176070":"2024-02-20T15:26:39.960651+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:27:08.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.307+0000 7f8c20200640 1 -- 172.21.15.47:0/2383741054 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8bf8066e70 msgr2=0x7f8bf8069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:08.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.307+0000 7f8c20200640 1 --2- 172.21.15.47:0/2383741054 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8bf8066e70 0x7f8bf8069330 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0af000 tx=0x7f8c0c0b6040 comp rx=0 tx=0).stop 2024-02-19T15:27:08.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.307+0000 7f8c20200640 1 -- 172.21.15.47:0/2383741054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c180ae2f0 msgr2=0x7f8c1813d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:08.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.307+0000 7f8c20200640 1 --2- 172.21.15.47:0/2383741054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c180ae2f0 0x7f8c1813d2c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400e9b0 tx=0x7f8c0400ee70 comp rx=0 tx=0).stop 2024-02-19T15:27:08.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.307+0000 7f8c20200640 1 -- 172.21.15.47:0/2383741054 shutdown_connections 2024-02-19T15:27:08.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.307+0000 7f8c20200640 1 --2- 172.21.15.47:0/2383741054 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8bf8066e70 0x7f8bf8069330 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.307+0000 7f8c20200640 1 --2- 172.21.15.47:0/2383741054 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c181478b0 0x7f8c18129ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.307+0000 7f8c20200640 1 --2- 172.21.15.47:0/2383741054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c180ae2f0 0x7f8c1813d2c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.307+0000 7f8c20200640 1 --2- 172.21.15.47:0/2383741054 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c1806e470 0x7f8c18160c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.307+0000 7f8c20200640 1 -- 172.21.15.47:0/2383741054 >> 172.21.15.47:0/2383741054 conn(0x7f8c180b1b80 msgr2=0x7f8c180b1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:08.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.307+0000 7f8c20200640 1 -- 172.21.15.47:0/2383741054 shutdown_connections 2024-02-19T15:27:08.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.307+0000 7f8c20200640 1 -- 172.21.15.47:0/2383741054 wait complete. 2024-02-19T15:27:08.333 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T15:27:08.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.467+0000 7f9629a00640 1 Processor -- start 2024-02-19T15:27:08.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.467+0000 7f9629a00640 1 -- start start 2024-02-19T15:27:08.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.471+0000 7f9629a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f96240cf760 0x7f96240728f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:08.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.471+0000 7f9629a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962415fda0 con 0x7f9624145790 2024-02-19T15:27:08.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.471+0000 7f9629a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9624160080 con 0x7f96240b5190 2024-02-19T15:27:08.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.471+0000 7f9629a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96241609b0 con 0x7f96240cf760 2024-02-19T15:27:08.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.471+0000 7f961be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f96240cf760 0x7f96240728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:08.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.471+0000 7f961be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f96240cf760 0x7f96240728f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56514/0 (socket says 172.21.15.47:56514) 2024-02-19T15:27:08.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.471+0000 7f961be00640 1 -- 172.21.15.47:0/1326032572 learned_addr learned my addr 172.21.15.47:0/1326032572 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:08.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.471+0000 7f9622a00640 1 --1- 172.21.15.47:0/1326032572 >> v1:172.21.15.47:6789/0 conn(0x7f9624145790 0x7f962415c410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58582/0 (socket says 172.21.15.47:58582) 2024-02-19T15:27:08.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.471+0000 7f961be00640 1 -- 172.21.15.47:0/1326032572 >> v1:172.21.15.116:6789/0 conn(0x7f96240b5190 legacy=0x7f96240b55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:08.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.471+0000 7f961be00640 1 -- 172.21.15.47:0/1326032572 >> v1:172.21.15.47:6789/0 conn(0x7f9624145790 legacy=0x7f962415c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:08.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.471+0000 7f961be00640 1 -- 172.21.15.47:0/1326032572 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9624160360 con 0x7f96240cf760 2024-02-19T15:27:08.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.471+0000 7f961be00640 1 --2- 172.21.15.47:0/1326032572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96240cf760 0x7f96240728f0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f960c009870 tx=0x7f960c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d47f063fecedf8f5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:08.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f9622000640 1 -- 172.21.15.47:0/1326032572 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c0ac640 con 0x7f96240cf760 2024-02-19T15:27:08.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f9622000640 1 -- 172.21.15.47:0/1326032572 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960c0ac7a0 con 0x7f96240cf760 2024-02-19T15:27:08.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f9622000640 1 -- 172.21.15.47:0/1326032572 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c0acb30 con 0x7f96240cf760 2024-02-19T15:27:08.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f9629a00640 1 -- 172.21.15.47:0/1326032572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96240cf760 msgr2=0x7f96240728f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:08.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f9629a00640 1 --2- 172.21.15.47:0/1326032572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96240cf760 0x7f96240728f0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f960c009870 tx=0x7f960c01c4d0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f9629a00640 1 -- 172.21.15.47:0/1326032572 shutdown_connections 2024-02-19T15:27:08.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f9629a00640 1 --2- 172.21.15.47:0/1326032572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96240cf760 0x7f96240728f0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f9629a00640 1 -- 172.21.15.47:0/1326032572 >> 172.21.15.47:0/1326032572 conn(0x7f96240b1b80 msgr2=0x7f96240b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:08.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f9629a00640 1 -- 172.21.15.47:0/1326032572 shutdown_connections 2024-02-19T15:27:08.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f9629a00640 1 -- 172.21.15.47:0/1326032572 wait complete. 2024-02-19T15:27:08.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f9629a00640 1 Processor -- start 2024-02-19T15:27:08.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f9629a00640 1 -- start start 2024-02-19T15:27:08.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f9629a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96240b5190 0x7f962415b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:08.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f9629a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96240cf760 0x7f9624149200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:08.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f9629a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9624145790 0x7f9624143490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:08.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f9629a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9624146140 con 0x7f96240cf760 2024-02-19T15:27:08.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f9629a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9624145e30 con 0x7f96240b5190 2024-02-19T15:27:08.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f9629a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9624149740 con 0x7f9624145790 2024-02-19T15:27:08.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f961be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96240cf760 0x7f9624149200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:08.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f961be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96240cf760 0x7f9624149200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56524/0 (socket says 172.21.15.47:56524) 2024-02-19T15:27:08.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f961be00640 1 -- 172.21.15.47:0/3163434033 learned_addr learned my addr 172.21.15.47:0/3163434033 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:08.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f9622a00640 1 --2- 172.21.15.47:0/3163434033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96240b5190 0x7f962415b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:08.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.475+0000 7f9623400640 1 --2- 172.21.15.47:0/3163434033 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9624145790 0x7f9624143490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:08.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.479+0000 7f9622a00640 1 -- 172.21.15.47:0/3163434033 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9624145790 msgr2=0x7f9624143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:08.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.479+0000 7f9622a00640 1 --2- 172.21.15.47:0/3163434033 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9624145790 0x7f9624143490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.479+0000 7f9622a00640 1 -- 172.21.15.47:0/3163434033 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96240cf760 msgr2=0x7f9624149200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:08.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.479+0000 7f9622a00640 1 --2- 172.21.15.47:0/3163434033 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96240cf760 0x7f9624149200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.479+0000 7f9622a00640 1 -- 172.21.15.47:0/3163434033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960c0094d0 con 0x7f96240b5190 2024-02-19T15:27:08.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.479+0000 7f961be00640 1 --2- 172.21.15.47:0/3163434033 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96240cf760 0x7f9624149200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:27:08.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.479+0000 7f9622a00640 1 --2- 172.21.15.47:0/3163434033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96240b5190 0x7f962415b990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f961400c390 tx=0x7f961400c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:08.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.479+0000 7f961aa00640 1 -- 172.21.15.47:0/3163434033 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96140074e0 con 0x7f96240b5190 2024-02-19T15:27:08.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.479+0000 7f961aa00640 1 -- 172.21.15.47:0/3163434033 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9614007640 con 0x7f96240b5190 2024-02-19T15:27:08.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.479+0000 7f961aa00640 1 -- 172.21.15.47:0/3163434033 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9614004a10 con 0x7f96240b5190 2024-02-19T15:27:08.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.479+0000 7f9629a00640 1 -- 172.21.15.47:0/3163434033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96241499e0 con 0x7f96240b5190 2024-02-19T15:27:08.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.479+0000 7f9629a00640 1 -- 172.21.15.47:0/3163434033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9624149fa0 con 0x7f96240b5190 2024-02-19T15:27:08.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.479+0000 7f9629a00640 1 -- 172.21.15.47:0/3163434033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f962414a190 con 0x7f96240b5190 2024-02-19T15:27:08.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.479+0000 7f961aa00640 1 -- 172.21.15.47:0/3163434033 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9614004b70 con 0x7f96240b5190 2024-02-19T15:27:08.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.479+0000 7f961aa00640 1 --2- 172.21.15.47:0/3163434033 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9600066e40 0x7f9600069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:08.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.479+0000 7f961aa00640 1 -- 172.21.15.47:0/3163434033 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f9614084360 con 0x7f96240b5190 2024-02-19T15:27:08.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.479+0000 7f961be00640 1 --2- 172.21.15.47:0/3163434033 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9600066e40 0x7f9600069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:08.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.483+0000 7f961be00640 1 --2- 172.21.15.47:0/3163434033 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9600066e40 0x7f9600069300 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f960c009870 tx=0x7f960c0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:08.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.483+0000 7f961aa00640 1 -- 172.21.15.47:0/3163434033 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f962414a190 con 0x7f96240b5190 2024-02-19T15:27:08.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.683+0000 7f9629a00640 1 -- 172.21.15.47:0/3163434033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f962414a190 con 0x7f96240b5190 2024-02-19T15:27:08.689 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-02-19T15:27:08.683+0000 7fc100600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T15:27:08.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.695+0000 7f961aa00640 1 -- 172.21.15.47:0/3163434033 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v67) v1 ==== 161+0+0 (secure 0 0 0) 0x7f962414a190 con 0x7f96240b5190 2024-02-19T15:27:08.702 INFO:teuthology.orchestra.run.smithi047.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T15:27:08.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.699+0000 7f9629a00640 1 -- 172.21.15.47:0/3163434033 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9600066e40 msgr2=0x7f9600069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:08.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.699+0000 7f9629a00640 1 --2- 172.21.15.47:0/3163434033 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9600066e40 0x7f9600069300 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f960c009870 tx=0x7f960c0a7000 comp rx=0 tx=0).stop 2024-02-19T15:27:08.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.699+0000 7f9629a00640 1 -- 172.21.15.47:0/3163434033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96240b5190 msgr2=0x7f962415b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:08.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.699+0000 7f9629a00640 1 --2- 172.21.15.47:0/3163434033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96240b5190 0x7f962415b990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f961400c390 tx=0x7f961400c850 comp rx=0 tx=0).stop 2024-02-19T15:27:08.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.699+0000 7f9629a00640 1 -- 172.21.15.47:0/3163434033 shutdown_connections 2024-02-19T15:27:08.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.699+0000 7f9629a00640 1 --2- 172.21.15.47:0/3163434033 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9600066e40 0x7f9600069300 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.699+0000 7f9629a00640 1 --2- 172.21.15.47:0/3163434033 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9624145790 0x7f9624143490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.699+0000 7f9629a00640 1 --2- 172.21.15.47:0/3163434033 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96240cf760 0x7f9624149200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.699+0000 7f9629a00640 1 --2- 172.21.15.47:0/3163434033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96240b5190 0x7f962415b990 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.699+0000 7f9629a00640 1 -- 172.21.15.47:0/3163434033 >> 172.21.15.47:0/3163434033 conn(0x7f96240b1b80 msgr2=0x7f962415c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:08.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.699+0000 7f9629a00640 1 -- 172.21.15.47:0/3163434033 shutdown_connections 2024-02-19T15:27:08.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.699+0000 7f9629a00640 1 -- 172.21.15.47:0/3163434033 wait complete. 2024-02-19T15:27:08.723 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T15:27:08.762 INFO:tasks.ceph.mds.d.smithi116.stdout:starting mds.d at 2024-02-19T15:27:08.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.859+0000 7fd254e00640 1 Processor -- start 2024-02-19T15:27:08.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.859+0000 7fd254e00640 1 -- start start 2024-02-19T15:27:08.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.859+0000 7fd254e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd250158a30 0x7fd25015bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:08.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.859+0000 7fd254e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd25015c300 con 0x7fd25006e390 2024-02-19T15:27:08.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.859+0000 7fd254e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd250072be0 con 0x7fd25006df30 2024-02-19T15:27:08.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.859+0000 7fd254e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd250158e10 con 0x7fd250158a30 2024-02-19T15:27:08.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.859+0000 7fd24e800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd25006e390 0x7fd2500724d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58598/0 (socket says 172.21.15.47:58598) 2024-02-19T15:27:08.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.859+0000 7fd24e800640 1 -- 172.21.15.47:0/2047124829 learned_addr learned my addr 172.21.15.47:0/2047124829 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:08.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.859+0000 7fd24de00640 1 --2- 172.21.15.47:0/2047124829 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd250158a30 0x7fd25015bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:08.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.859+0000 7fd24d400640 1 -- 172.21.15.47:0/2047124829 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891880776 0 0) 0x7fd25015c300 con 0x7fd25006e390 2024-02-19T15:27:08.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.863+0000 7fd24d400640 1 -- 172.21.15.47:0/2047124829 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd234003530 con 0x7fd25006e390 2024-02-19T15:27:08.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.863+0000 7fd24d400640 1 -- 172.21.15.47:0/2047124829 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1179477761 0 0) 0x7fd250072be0 con 0x7fd25006df30 2024-02-19T15:27:08.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.863+0000 7fd24d400640 1 -- 172.21.15.47:0/2047124829 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd25015c300 con 0x7fd25006df30 2024-02-19T15:27:08.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.863+0000 7fd24de00640 1 -- 172.21.15.47:0/2047124829 >> v1:172.21.15.116:6789/0 conn(0x7fd25006df30 legacy=0x7fd250071dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:08.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.863+0000 7fd24de00640 1 -- 172.21.15.47:0/2047124829 >> v1:172.21.15.47:6789/0 conn(0x7fd25006e390 legacy=0x7fd2500724d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:08.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.863+0000 7fd24de00640 1 -- 172.21.15.47:0/2047124829 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2500cf780 con 0x7fd250158a30 2024-02-19T15:27:08.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.863+0000 7fd24de00640 1 --2- 172.21.15.47:0/2047124829 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd250158a30 0x7fd25015bdc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd2400097f0 tx=0x7fd24001c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=882bb5b83aa20668 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:08.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.863+0000 7fd24d400640 1 -- 172.21.15.47:0/2047124829 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2400aa610 con 0x7fd250158a30 2024-02-19T15:27:08.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.863+0000 7fd24d400640 1 -- 172.21.15.47:0/2047124829 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2400a6040 con 0x7fd250158a30 2024-02-19T15:27:08.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.863+0000 7fd24d400640 1 -- 172.21.15.47:0/2047124829 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2400aa990 con 0x7fd250158a30 2024-02-19T15:27:08.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.863+0000 7fd254e00640 1 -- 172.21.15.47:0/2047124829 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd250158a30 msgr2=0x7fd25015bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:08.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.863+0000 7fd254e00640 1 --2- 172.21.15.47:0/2047124829 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd250158a30 0x7fd25015bdc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd2400097f0 tx=0x7fd24001c4b0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.863+0000 7fd254e00640 1 -- 172.21.15.47:0/2047124829 shutdown_connections 2024-02-19T15:27:08.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.863+0000 7fd254e00640 1 --2- 172.21.15.47:0/2047124829 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd250158a30 0x7fd25015bdc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.863+0000 7fd254e00640 1 -- 172.21.15.47:0/2047124829 >> 172.21.15.47:0/2047124829 conn(0x7fd2500b1b80 msgr2=0x7fd2500b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:08.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.863+0000 7fd254e00640 1 -- 172.21.15.47:0/2047124829 shutdown_connections 2024-02-19T15:27:08.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.863+0000 7fd254e00640 1 -- 172.21.15.47:0/2047124829 wait complete. 2024-02-19T15:27:08.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd254e00640 1 Processor -- start 2024-02-19T15:27:08.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd254e00640 1 -- start start 2024-02-19T15:27:08.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd254e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd25006df30 0x7fd250144940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:08.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd254e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd25006e390 0x7fd25016ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:08.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd254e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd250158a30 0x7fd2500ceb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:08.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd254e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25015ae00 con 0x7fd250158a30 2024-02-19T15:27:08.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd254e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25014ebc0 con 0x7fd25006df30 2024-02-19T15:27:08.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd254e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd25014e5b0 con 0x7fd25006e390 2024-02-19T15:27:08.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd24e800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd250158a30 0x7fd2500ceb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:08.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd24de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd25006df30 0x7fd250144940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:08.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd24de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd25006df30 0x7fd250144940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:40184/0 (socket says 172.21.15.47:40184) 2024-02-19T15:27:08.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd24de00640 1 -- 172.21.15.47:0/3604205039 learned_addr learned my addr 172.21.15.47:0/3604205039 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:08.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd245400640 1 --2- 172.21.15.47:0/3604205039 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd25006e390 0x7fd25016ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:08.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd24de00640 1 -- 172.21.15.47:0/3604205039 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd25006e390 msgr2=0x7fd25016ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:08.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd24de00640 1 --2- 172.21.15.47:0/3604205039 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd25006e390 0x7fd25016ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd24de00640 1 -- 172.21.15.47:0/3604205039 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd250158a30 msgr2=0x7fd2500ceb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:08.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd24de00640 1 --2- 172.21.15.47:0/3604205039 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd250158a30 0x7fd2500ceb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:08.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd24de00640 1 -- 172.21.15.47:0/3604205039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd240009420 con 0x7fd25006df30 2024-02-19T15:27:08.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd245400640 1 --2- 172.21.15.47:0/3604205039 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd25006e390 0x7fd25016ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:27:08.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd24e800640 1 --2- 172.21.15.47:0/3604205039 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd250158a30 0x7fd2500ceb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:27:08.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd24de00640 1 --2- 172.21.15.47:0/3604205039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd25006df30 0x7fd250144940 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd2400190c0 tx=0x7fd24001acf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:08.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd246a00640 1 -- 172.21.15.47:0/3604205039 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2400be410 con 0x7fd25006df30 2024-02-19T15:27:08.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd246a00640 1 -- 172.21.15.47:0/3604205039 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2400be570 con 0x7fd25006df30 2024-02-19T15:27:08.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd254e00640 1 -- 172.21.15.47:0/3604205039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd25015ab20 con 0x7fd25006df30 2024-02-19T15:27:08.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd246a00640 1 -- 172.21.15.47:0/3604205039 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2400be920 con 0x7fd25006df30 2024-02-19T15:27:08.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.867+0000 7fd254e00640 1 -- 172.21.15.47:0/3604205039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd250144e80 con 0x7fd25006df30 2024-02-19T15:27:08.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.871+0000 7fd254e00640 1 -- 172.21.15.47:0/3604205039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd25014e410 con 0x7fd25006df30 2024-02-19T15:27:08.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.871+0000 7fd246a00640 1 -- 172.21.15.47:0/3604205039 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd2400b4400 con 0x7fd25006df30 2024-02-19T15:27:08.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.871+0000 7fd246a00640 1 --2- 172.21.15.47:0/3604205039 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd22c066e70 0x7fd22c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:08.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.871+0000 7fd246a00640 1 -- 172.21.15.47:0/3604205039 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fd24011f530 con 0x7fd25006df30 2024-02-19T15:27:08.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.871+0000 7fd245400640 1 --2- 172.21.15.47:0/3604205039 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd22c066e70 0x7fd22c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:08.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.871+0000 7fd245400640 1 --2- 172.21.15.47:0/3604205039 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd22c066e70 0x7fd22c069330 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fd2500b00e0 tx=0x7fd238009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:08.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:08.875+0000 7fd246a00640 1 -- 172.21.15.47:0/3604205039 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd25014e410 con 0x7fd25006df30 2024-02-19T15:27:09.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.075+0000 7fd254e00640 1 -- 172.21.15.47:0/3604205039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd25014e410 con 0x7fd25006df30 2024-02-19T15:27:09.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.691+0000 7fd246a00640 1 -- 172.21.15.47:0/3604205039 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v68) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd25014e410 con 0x7fd25006df30 2024-02-19T15:27:09.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.695+0000 7fd254e00640 1 -- 172.21.15.47:0/3604205039 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd22c066e70 msgr2=0x7fd22c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:09.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.695+0000 7fd254e00640 1 --2- 172.21.15.47:0/3604205039 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd22c066e70 0x7fd22c069330 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fd2500b00e0 tx=0x7fd238009040 comp rx=0 tx=0).stop 2024-02-19T15:27:09.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.695+0000 7fd254e00640 1 -- 172.21.15.47:0/3604205039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd25006df30 msgr2=0x7fd250144940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:09.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.695+0000 7fd254e00640 1 --2- 172.21.15.47:0/3604205039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd25006df30 0x7fd250144940 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd2400190c0 tx=0x7fd24001acf0 comp rx=0 tx=0).stop 2024-02-19T15:27:09.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.695+0000 7fd254e00640 1 -- 172.21.15.47:0/3604205039 shutdown_connections 2024-02-19T15:27:09.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.695+0000 7fd254e00640 1 --2- 172.21.15.47:0/3604205039 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd22c066e70 0x7fd22c069330 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:09.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.695+0000 7fd254e00640 1 --2- 172.21.15.47:0/3604205039 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd250158a30 0x7fd2500ceb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:09.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.695+0000 7fd254e00640 1 --2- 172.21.15.47:0/3604205039 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd25006e390 0x7fd25016ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:09.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.695+0000 7fd254e00640 1 --2- 172.21.15.47:0/3604205039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd25006df30 0x7fd250144940 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:09.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.695+0000 7fd254e00640 1 -- 172.21.15.47:0/3604205039 >> 172.21.15.47:0/3604205039 conn(0x7fd2500b1b80 msgr2=0x7fd250145fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:09.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.695+0000 7fd254e00640 1 -- 172.21.15.47:0/3604205039 shutdown_connections 2024-02-19T15:27:09.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.695+0000 7fd254e00640 1 -- 172.21.15.47:0/3604205039 wait complete. 2024-02-19T15:27:09.723 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T15:27:09.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.867+0000 7fde98000640 1 Processor -- start 2024-02-19T15:27:09.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.867+0000 7fde98000640 1 -- start start 2024-02-19T15:27:09.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.867+0000 7fde98000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fde90070170 0x7fde900cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:09.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.867+0000 7fde98000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde90070720 0x7fde900cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:09.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.867+0000 7fde98000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde900d0830 con 0x7fde9015f860 2024-02-19T15:27:09.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.867+0000 7fde98000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde900628b0 con 0x7fde90070170 2024-02-19T15:27:09.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.867+0000 7fde98000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde900cf7b0 con 0x7fde90070720 2024-02-19T15:27:09.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.867+0000 7fde95600640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fde9015f860 0x7fde900ce750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58614/0 (socket says 172.21.15.47:58614) 2024-02-19T15:27:09.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.867+0000 7fde95600640 1 -- 172.21.15.47:0/649586029 learned_addr learned my addr 172.21.15.47:0/649586029 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:09.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.867+0000 7fde96000640 1 --2- 172.21.15.47:0/649586029 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde90070720 0x7fde900cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:09.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.867+0000 7fde94c00640 1 --2- 172.21.15.47:0/649586029 >> v2:172.21.15.116:3300/0 conn(0x7fde90070170 0x7fde900cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:09.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.867+0000 7fde8fe00640 1 -- 172.21.15.47:0/649586029 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460032683 0 0) 0x7fde900d0830 con 0x7fde9015f860 2024-02-19T15:27:09.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.867+0000 7fde8fe00640 1 -- 172.21.15.47:0/649586029 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde78003530 con 0x7fde9015f860 2024-02-19T15:27:09.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.871+0000 7fde8fe00640 1 -- 172.21.15.47:0/649586029 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 666279971 0 0) 0x7fde78003530 con 0x7fde9015f860 2024-02-19T15:27:09.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.871+0000 7fde96000640 1 -- 172.21.15.47:0/649586029 >> v2:172.21.15.116:3300/0 conn(0x7fde90070170 msgr2=0x7fde900cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:09.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.871+0000 7fde96000640 1 --2- 172.21.15.47:0/649586029 >> v2:172.21.15.116:3300/0 conn(0x7fde90070170 0x7fde900cee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:09.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.871+0000 7fde96000640 1 -- 172.21.15.47:0/649586029 >> v1:172.21.15.47:6789/0 conn(0x7fde9015f860 legacy=0x7fde900ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:09.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.871+0000 7fde96000640 1 -- 172.21.15.47:0/649586029 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde900cf4d0 con 0x7fde90070720 2024-02-19T15:27:09.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.871+0000 7fde96000640 1 --2- 172.21.15.47:0/649586029 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde90070720 0x7fde900cd210 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fde88009d90 tx=0x7fde8801c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b26e07d061d0b7fc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:09.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.871+0000 7fde8fe00640 1 -- 172.21.15.47:0/649586029 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde84001680 con 0x7fde90070720 2024-02-19T15:27:09.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.871+0000 7fde8fe00640 1 -- 172.21.15.47:0/649586029 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde880a55f0 con 0x7fde90070720 2024-02-19T15:27:09.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.871+0000 7fde8fe00640 1 -- 172.21.15.47:0/649586029 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde880a5980 con 0x7fde90070720 2024-02-19T15:27:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.871+0000 7fde98000640 1 -- 172.21.15.47:0/649586029 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde90070720 msgr2=0x7fde900cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.871+0000 7fde98000640 1 --2- 172.21.15.47:0/649586029 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde90070720 0x7fde900cd210 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fde88009d90 tx=0x7fde8801c7d0 comp rx=0 tx=0).stop 2024-02-19T15:27:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.871+0000 7fde98000640 1 -- 172.21.15.47:0/649586029 shutdown_connections 2024-02-19T15:27:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.871+0000 7fde98000640 1 --2- 172.21.15.47:0/649586029 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde90070720 0x7fde900cd210 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.871+0000 7fde98000640 1 --2- 172.21.15.47:0/649586029 >> v2:172.21.15.116:3300/0 conn(0x7fde90070170 0x7fde900cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.871+0000 7fde98000640 1 -- 172.21.15.47:0/649586029 >> 172.21.15.47:0/649586029 conn(0x7fde900b38d0 msgr2=0x7fde900b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.871+0000 7fde98000640 1 -- 172.21.15.47:0/649586029 shutdown_connections 2024-02-19T15:27:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.871+0000 7fde98000640 1 -- 172.21.15.47:0/649586029 wait complete. 2024-02-19T15:27:09.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde98000640 1 Processor -- start 2024-02-19T15:27:09.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde98000640 1 -- start start 2024-02-19T15:27:09.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde98000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde90070170 0x7fde9014c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde98000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde90070720 0x7fde9015a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde98000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde9015f860 0x7fde900b56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde98000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9014c840 con 0x7fde90070170 2024-02-19T15:27:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde98000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde900c7930 con 0x7fde9015f860 2024-02-19T15:27:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde98000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde900c6d20 con 0x7fde90070720 2024-02-19T15:27:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde95600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde90070170 0x7fde9014c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde95600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde90070170 0x7fde9014c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56544/0 (socket says 172.21.15.47:56544) 2024-02-19T15:27:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde95600640 1 -- 172.21.15.47:0/386297910 learned_addr learned my addr 172.21.15.47:0/386297910 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde94c00640 1 --2- 172.21.15.47:0/386297910 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde90070720 0x7fde9015a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde96000640 1 --2- 172.21.15.47:0/386297910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde9015f860 0x7fde900b56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde95600640 1 -- 172.21.15.47:0/386297910 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde90070720 msgr2=0x7fde9015a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde95600640 1 --2- 172.21.15.47:0/386297910 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde90070720 0x7fde9015a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde95600640 1 -- 172.21.15.47:0/386297910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde9015f860 msgr2=0x7fde900b56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde95600640 1 --2- 172.21.15.47:0/386297910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde9015f860 0x7fde900b56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde95600640 1 -- 172.21.15.47:0/386297910 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde88009a70 con 0x7fde90070170 2024-02-19T15:27:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde95600640 1 --2- 172.21.15.47:0/386297910 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde90070170 0x7fde9014c300 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fde8400c750 tx=0x7fde8400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde8d600640 1 -- 172.21.15.47:0/386297910 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde84005ae0 con 0x7fde90070170 2024-02-19T15:27:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde8d600640 1 -- 172.21.15.47:0/386297910 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde84005c40 con 0x7fde90070170 2024-02-19T15:27:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde8d600640 1 -- 172.21.15.47:0/386297910 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde84007070 con 0x7fde90070170 2024-02-19T15:27:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde98000640 1 -- 172.21.15.47:0/386297910 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde900c7000 con 0x7fde90070170 2024-02-19T15:27:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.875+0000 7fde98000640 1 -- 172.21.15.47:0/386297910 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde900c75c0 con 0x7fde90070170 2024-02-19T15:27:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.879+0000 7fde98000640 1 -- 172.21.15.47:0/386297910 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde90068120 con 0x7fde90070170 2024-02-19T15:27:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.879+0000 7fde8d600640 1 -- 172.21.15.47:0/386297910 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fde840071d0 con 0x7fde90070170 2024-02-19T15:27:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.879+0000 7fde8d600640 1 --2- 172.21.15.47:0/386297910 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fde70066e40 0x7fde70069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.879+0000 7fde8d600640 1 -- 172.21.15.47:0/386297910 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fde840842a0 con 0x7fde90070170 2024-02-19T15:27:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.879+0000 7fde94c00640 1 --2- 172.21.15.47:0/386297910 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fde70066e40 0x7fde70069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.879+0000 7fde94c00640 1 --2- 172.21.15.47:0/386297910 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fde70066e40 0x7fde70069300 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fde900c68d0 tx=0x7fde7c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:09.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:09.883+0000 7fde8d600640 1 -- 172.21.15.47:0/386297910 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fde84050fa0 con 0x7fde90070170 2024-02-19T15:27:10.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:10.051+0000 7fde98000640 1 -- 172.21.15.47:0/386297910 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fde900b34f0 con 0x7fde90070170 2024-02-19T15:27:10.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:10.959+0000 7fde8d600640 1 -- 172.21.15.47:0/386297910 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v99) v1 ==== 181+0+0 (secure 0 0 0) 0x7fde84054660 con 0x7fde90070170 2024-02-19T15:27:10.965 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_metadata' removed 2024-02-19T15:27:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:10.967+0000 7fde98000640 1 -- 172.21.15.47:0/386297910 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fde70066e40 msgr2=0x7fde70069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:10.967+0000 7fde98000640 1 --2- 172.21.15.47:0/386297910 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fde70066e40 0x7fde70069300 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fde900c68d0 tx=0x7fde7c009190 comp rx=0 tx=0).stop 2024-02-19T15:27:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:10.967+0000 7fde98000640 1 -- 172.21.15.47:0/386297910 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde90070170 msgr2=0x7fde9014c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:10.967+0000 7fde98000640 1 --2- 172.21.15.47:0/386297910 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde90070170 0x7fde9014c300 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fde8400c750 tx=0x7fde8400cc10 comp rx=0 tx=0).stop 2024-02-19T15:27:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:10.967+0000 7fde98000640 1 -- 172.21.15.47:0/386297910 shutdown_connections 2024-02-19T15:27:10.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:10.967+0000 7fde98000640 1 --2- 172.21.15.47:0/386297910 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fde70066e40 0x7fde70069300 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:10.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:10.967+0000 7fde98000640 1 --2- 172.21.15.47:0/386297910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde9015f860 0x7fde900b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:10.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:10.967+0000 7fde98000640 1 --2- 172.21.15.47:0/386297910 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fde90070720 0x7fde9015a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:10.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:10.967+0000 7fde98000640 1 --2- 172.21.15.47:0/386297910 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde90070170 0x7fde9014c300 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:10.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:10.967+0000 7fde98000640 1 -- 172.21.15.47:0/386297910 >> 172.21.15.47:0/386297910 conn(0x7fde900b38d0 msgr2=0x7fde9006d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:10.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:10.967+0000 7fde98000640 1 -- 172.21.15.47:0/386297910 shutdown_connections 2024-02-19T15:27:10.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:10.967+0000 7fde98000640 1 -- 172.21.15.47:0/386297910 wait complete. 2024-02-19T15:27:10.998 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T15:27:11.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.147+0000 7fa845400640 1 Processor -- start 2024-02-19T15:27:11.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.147+0000 7fa845400640 1 -- start start 2024-02-19T15:27:11.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.147+0000 7fa845400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa84015dfe0 0x7fa8400b6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:11.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.147+0000 7fa845400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa84006fce0 0x7fa840167b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:11.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.147+0000 7fa845400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa84006e040 con 0x7fa8400b69d0 2024-02-19T15:27:11.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.147+0000 7fa845400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa84006c020 con 0x7fa84015dfe0 2024-02-19T15:27:11.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.147+0000 7fa845400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa84006c270 con 0x7fa84006fce0 2024-02-19T15:27:11.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.147+0000 7fa83fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa84015dfe0 0x7fa8400b6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:11.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.147+0000 7fa837e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa8400b69d0 0x7fa84006f5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58618/0 (socket says 172.21.15.47:58618) 2024-02-19T15:27:11.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.147+0000 7fa837e00640 1 -- 172.21.15.47:0/1652390089 learned_addr learned my addr 172.21.15.47:0/1652390089 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:11.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.151+0000 7fa844a00640 1 --2- 172.21.15.47:0/1652390089 >> v2:172.21.15.116:3300/0 conn(0x7fa84006fce0 0x7fa840167b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:11.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.151+0000 7fa83fe00640 1 -- 172.21.15.47:0/1652390089 >> v2:172.21.15.116:3300/0 conn(0x7fa84006fce0 msgr2=0x7fa840167b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:11.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.151+0000 7fa83fe00640 1 --2- 172.21.15.47:0/1652390089 >> v2:172.21.15.116:3300/0 conn(0x7fa84006fce0 0x7fa840167b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:11.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.151+0000 7fa83fe00640 1 -- 172.21.15.47:0/1652390089 >> v1:172.21.15.47:6789/0 conn(0x7fa8400b69d0 legacy=0x7fa84006f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:11.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.151+0000 7fa83fe00640 1 -- 172.21.15.47:0/1652390089 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84006c550 con 0x7fa84015dfe0 2024-02-19T15:27:11.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.151+0000 7fa83f400640 1 -- 172.21.15.47:0/1652390089 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063870577 0 0) 0x7fa84006e040 con 0x7fa8400b69d0 2024-02-19T15:27:11.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.151+0000 7fa844a00640 1 --2- 172.21.15.47:0/1652390089 >> v2:172.21.15.116:3300/0 conn(0x7fa84006fce0 0x7fa840167b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:11.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.151+0000 7fa83fe00640 1 --2- 172.21.15.47:0/1652390089 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa84015dfe0 0x7fa8400b6490 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fa8400b6e20 tx=0x7fa83001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec20a6f8f7070b5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:11.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.151+0000 7fa83f400640 1 -- 172.21.15.47:0/1652390089 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8300a8970 con 0x7fa84015dfe0 2024-02-19T15:27:11.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.151+0000 7fa83f400640 1 -- 172.21.15.47:0/1652390089 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8300a8ad0 con 0x7fa84015dfe0 2024-02-19T15:27:11.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.151+0000 7fa83f400640 1 -- 172.21.15.47:0/1652390089 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8300a8e60 con 0x7fa84015dfe0 2024-02-19T15:27:11.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.151+0000 7fa845400640 1 -- 172.21.15.47:0/1652390089 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa84015dfe0 msgr2=0x7fa8400b6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:11.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.151+0000 7fa845400640 1 --2- 172.21.15.47:0/1652390089 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa84015dfe0 0x7fa8400b6490 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fa8400b6e20 tx=0x7fa83001ec70 comp rx=0 tx=0).stop 2024-02-19T15:27:11.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa845400640 1 -- 172.21.15.47:0/1652390089 shutdown_connections 2024-02-19T15:27:11.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa845400640 1 --2- 172.21.15.47:0/1652390089 >> v2:172.21.15.116:3300/0 conn(0x7fa84006fce0 0x7fa840167b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:11.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa845400640 1 --2- 172.21.15.47:0/1652390089 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa84015dfe0 0x7fa8400b6490 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:11.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa845400640 1 -- 172.21.15.47:0/1652390089 >> 172.21.15.47:0/1652390089 conn(0x7fa84009aa70 msgr2=0x7fa84009ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:11.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa845400640 1 -- 172.21.15.47:0/1652390089 shutdown_connections 2024-02-19T15:27:11.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa845400640 1 -- 172.21.15.47:0/1652390089 wait complete. 2024-02-19T15:27:11.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa845400640 1 Processor -- start 2024-02-19T15:27:11.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa845400640 1 -- start start 2024-02-19T15:27:11.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa845400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa84006fce0 0x7fa840152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:11.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa845400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa8400b69d0 0x7fa84015c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:11.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa845400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa84015dfe0 0x7fa84009e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:11.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa845400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84009c030 con 0x7fa84015dfe0 2024-02-19T15:27:11.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa845400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84009c310 con 0x7fa84006fce0 2024-02-19T15:27:11.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa845400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa84009c5f0 con 0x7fa8400b69d0 2024-02-19T15:27:11.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa844a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa84015dfe0 0x7fa84009e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:11.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa844a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa84015dfe0 0x7fa84009e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56552/0 (socket says 172.21.15.47:56552) 2024-02-19T15:27:11.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa844a00640 1 -- 172.21.15.47:0/3537281390 learned_addr learned my addr 172.21.15.47:0/3537281390 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:11.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa83fe00640 1 --2- 172.21.15.47:0/3537281390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa84006fce0 0x7fa840152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:11.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa837e00640 1 --2- 172.21.15.47:0/3537281390 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa8400b69d0 0x7fa84015c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:11.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa844a00640 1 -- 172.21.15.47:0/3537281390 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa8400b69d0 msgr2=0x7fa84015c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:11.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa844a00640 1 --2- 172.21.15.47:0/3537281390 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa8400b69d0 0x7fa84015c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:11.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa844a00640 1 -- 172.21.15.47:0/3537281390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa84006fce0 msgr2=0x7fa840152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:11.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa844a00640 1 --2- 172.21.15.47:0/3537281390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa84006fce0 0x7fa840152a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:11.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa844a00640 1 -- 172.21.15.47:0/3537281390 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa830009420 con 0x7fa84015dfe0 2024-02-19T15:27:11.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa837e00640 1 --2- 172.21.15.47:0/3537281390 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa8400b69d0 0x7fa84015c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:27:11.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.155+0000 7fa844a00640 1 --2- 172.21.15.47:0/3537281390 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa84015dfe0 0x7fa84009e560 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fa83800ce40 tx=0x7fa83800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:11.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.159+0000 7fa83cc00640 1 -- 172.21.15.47:0/3537281390 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83800d840 con 0x7fa84015dfe0 2024-02-19T15:27:11.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.159+0000 7fa83cc00640 1 -- 172.21.15.47:0/3537281390 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa83800d9a0 con 0x7fa84015dfe0 2024-02-19T15:27:11.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.159+0000 7fa83cc00640 1 -- 172.21.15.47:0/3537281390 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83800dd50 con 0x7fa84015dfe0 2024-02-19T15:27:11.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.159+0000 7fa845400640 1 -- 172.21.15.47:0/3537281390 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84009c8d0 con 0x7fa84015dfe0 2024-02-19T15:27:11.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.159+0000 7fa845400640 1 -- 172.21.15.47:0/3537281390 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa840152fa0 con 0x7fa84015dfe0 2024-02-19T15:27:11.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.159+0000 7fa845400640 1 -- 172.21.15.47:0/3537281390 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa840068040 con 0x7fa84015dfe0 2024-02-19T15:27:11.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.159+0000 7fa83cc00640 1 -- 172.21.15.47:0/3537281390 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa83801e3b0 con 0x7fa84015dfe0 2024-02-19T15:27:11.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.159+0000 7fa83cc00640 1 --2- 172.21.15.47:0/3537281390 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa81c066e40 0x7fa81c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:11.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.159+0000 7fa83cc00640 1 -- 172.21.15.47:0/3537281390 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5529+0+0 (secure 0 0 0) 0x7fa8380853a0 con 0x7fa84015dfe0 2024-02-19T15:27:11.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.159+0000 7fa83fe00640 1 --2- 172.21.15.47:0/3537281390 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa81c066e40 0x7fa81c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:11.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.159+0000 7fa83fe00640 1 --2- 172.21.15.47:0/3537281390 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa81c066e40 0x7fa81c069300 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fa8300b2000 tx=0x7fa830008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:11.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.163+0000 7fa83cc00640 1 -- 172.21.15.47:0/3537281390 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa838052250 con 0x7fa84015dfe0 2024-02-19T15:27:11.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.327+0000 7fa845400640 1 -- 172.21.15.47:0/3537281390 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa8400d2530 con 0x7fa84015dfe0 2024-02-19T15:27:11.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.967+0000 7fa83cc00640 1 -- 172.21.15.47:0/3537281390 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v100) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa838055910 con 0x7fa84015dfe0 2024-02-19T15:27:11.972 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_data' removed 2024-02-19T15:27:11.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.971+0000 7fa845400640 1 -- 172.21.15.47:0/3537281390 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa81c066e40 msgr2=0x7fa81c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:11.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.971+0000 7fa845400640 1 --2- 172.21.15.47:0/3537281390 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa81c066e40 0x7fa81c069300 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fa8300b2000 tx=0x7fa830008040 comp rx=0 tx=0).stop 2024-02-19T15:27:11.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.971+0000 7fa845400640 1 -- 172.21.15.47:0/3537281390 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa84015dfe0 msgr2=0x7fa84009e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:11.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.971+0000 7fa845400640 1 --2- 172.21.15.47:0/3537281390 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa84015dfe0 0x7fa84009e560 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fa83800ce40 tx=0x7fa83800a710 comp rx=0 tx=0).stop 2024-02-19T15:27:11.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.971+0000 7fa845400640 1 -- 172.21.15.47:0/3537281390 shutdown_connections 2024-02-19T15:27:11.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.971+0000 7fa845400640 1 --2- 172.21.15.47:0/3537281390 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa81c066e40 0x7fa81c069300 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:11.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.971+0000 7fa845400640 1 --2- 172.21.15.47:0/3537281390 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa84015dfe0 0x7fa84009e560 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:11.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.971+0000 7fa845400640 1 --2- 172.21.15.47:0/3537281390 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa8400b69d0 0x7fa84015c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:11.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.971+0000 7fa845400640 1 --2- 172.21.15.47:0/3537281390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa84006fce0 0x7fa840152a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:11.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.971+0000 7fa845400640 1 -- 172.21.15.47:0/3537281390 >> 172.21.15.47:0/3537281390 conn(0x7fa84009aa70 msgr2=0x7fa8400b2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:11.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.971+0000 7fa845400640 1 -- 172.21.15.47:0/3537281390 shutdown_connections 2024-02-19T15:27:11.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:11.971+0000 7fa845400640 1 -- 172.21.15.47:0/3537281390 wait complete. 2024-02-19T15:27:12.003 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-19T15:27:12.006 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-19T15:27:12.049 INFO:teuthology.orchestra.run.smithi047.stdout:[global] 2024-02-19T15:27:12.049 INFO:teuthology.orchestra.run.smithi047.stdout: chdir = "" 2024-02-19T15:27:12.049 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-19T15:27:12.049 INFO:teuthology.orchestra.run.smithi047.stdout: auth supported = cephx 2024-02-19T15:27:12.049 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.049 INFO:teuthology.orchestra.run.smithi047.stdout: filestore xattr use omap = true 2024-02-19T15:27:12.049 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.049 INFO:teuthology.orchestra.run.smithi047.stdout: mon clock drift allowed = 1.000 2024-02-19T15:27:12.049 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.049 INFO:teuthology.orchestra.run.smithi047.stdout: osd crush chooseleaf type = 0 2024-02-19T15:27:12.050 INFO:teuthology.orchestra.run.smithi047.stdout: auth debug = true 2024-02-19T15:27:12.050 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.050 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on old message = true 2024-02-19T15:27:12.050 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on bug = true 2024-02-19T15:27:12.050 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.050 INFO:teuthology.orchestra.run.smithi047.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-19T15:27:12.050 INFO:teuthology.orchestra.run.smithi047.stdout: mon pg warn max object skew = 0 2024-02-19T15:27:12.050 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.050 INFO:teuthology.orchestra.run.smithi047.stdout: # disable pg_autoscaler by default for new pools 2024-02-19T15:27:12.050 INFO:teuthology.orchestra.run.smithi047.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-19T15:27:12.050 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.050 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default size = 2 2024-02-19T15:27:12.050 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.050 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow primary affinity = true 2024-02-19T15:27:12.050 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow pg remap = true 2024-02-19T15:27:12.050 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on legacy crush tunables = false 2024-02-19T15:27:12.051 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on crush straw calc version zero = false 2024-02-19T15:27:12.051 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on no sortbitwise = false 2024-02-19T15:27:12.051 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on osd down out interval zero = false 2024-02-19T15:27:12.051 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on too few osds = false 2024-02-19T15:27:12.051 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-19T15:27:12.051 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-19T15:27:12.051 INFO:teuthology.orchestra.run.smithi047.stdout: mon_allow_pool_size_one = true 2024-02-19T15:27:12.051 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.051 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-19T15:27:12.051 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.051 INFO:teuthology.orchestra.run.smithi047.stdout: osd default data pool replay window = 5 2024-02-19T15:27:12.051 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.051 INFO:teuthology.orchestra.run.smithi047.stdout: mon allow pool delete = true 2024-02-19T15:27:12.051 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.051 INFO:teuthology.orchestra.run.smithi047.stdout: mon cluster log file level = debug 2024-02-19T15:27:12.051 INFO:teuthology.orchestra.run.smithi047.stdout: debug asserts on shutdown = true 2024-02-19T15:27:12.051 INFO:teuthology.orchestra.run.smithi047.stdout: mon health detail to clog = false 2024-02-19T15:27:12.052 INFO:teuthology.orchestra.run.smithi047.stdout: mon host = "172.21.15.47,172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790]" 2024-02-19T15:27:12.052 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on skipped message = False 2024-02-19T15:27:12.052 INFO:teuthology.orchestra.run.smithi047.stdout: fsid = ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e 2024-02-19T15:27:12.052 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.052 INFO:teuthology.orchestra.run.smithi047.stdout:[osd] 2024-02-19T15:27:12.052 INFO:teuthology.orchestra.run.smithi047.stdout: osd journal size = 100 2024-02-19T15:27:12.052 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.052 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub load threshold = 5.0 2024-02-19T15:27:12.052 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub max interval = 600 2024-02-19T15:27:12.052 INFO:teuthology.orchestra.run.smithi047.stdout: osd mclock profile = high_recovery_ops 2024-02-19T15:27:12.052 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.052 INFO:teuthology.orchestra.run.smithi047.stdout: osd recover clone overlap = true 2024-02-19T15:27:12.052 INFO:teuthology.orchestra.run.smithi047.stdout: osd recovery max chunk = 1048576 2024-02-19T15:27:12.052 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.052 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug shutdown = true 2024-02-19T15:27:12.053 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug op order = true 2024-02-19T15:27:12.053 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify stray on activate = true 2024-02-19T15:27:12.053 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug trim objects = true 2024-02-19T15:27:12.053 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.053 INFO:teuthology.orchestra.run.smithi047.stdout: osd open classes on start = true 2024-02-19T15:27:12.053 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug pg log writeout = true 2024-02-19T15:27:12.053 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.053 INFO:teuthology.orchestra.run.smithi047.stdout: osd deep scrub update digest min age = 30 2024-02-19T15:27:12.053 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.053 INFO:teuthology.orchestra.run.smithi047.stdout: osd map max advance = 10 2024-02-19T15:27:12.053 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.053 INFO:teuthology.orchestra.run.smithi047.stdout: journal zero on create = true 2024-02-19T15:27:12.053 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.053 INFO:teuthology.orchestra.run.smithi047.stdout: filestore ondisk finisher threads = 3 2024-02-19T15:27:12.053 INFO:teuthology.orchestra.run.smithi047.stdout: filestore apply finisher threads = 3 2024-02-19T15:27:12.054 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.054 INFO:teuthology.orchestra.run.smithi047.stdout: bdev debug aio = true 2024-02-19T15:27:12.054 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug misdirected ops = true 2024-02-19T15:27:12.054 INFO:teuthology.orchestra.run.smithi047.stdout: bdev async discard = True 2024-02-19T15:27:12.054 INFO:teuthology.orchestra.run.smithi047.stdout: bdev enable discard = True 2024-02-19T15:27:12.054 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore allocator = bitmap 2024-02-19T15:27:12.054 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore block size = 96636764160 2024-02-19T15:27:12.054 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore fsck on mount = True 2024-02-19T15:27:12.054 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluefs = 1/20 2024-02-19T15:27:12.054 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluestore = 1/20 2024-02-19T15:27:12.054 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:27:12.054 INFO:teuthology.orchestra.run.smithi047.stdout: debug osd = 20 2024-02-19T15:27:12.054 INFO:teuthology.orchestra.run.smithi047.stdout: debug rocksdb = 4/10 2024-02-19T15:27:12.054 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd backfillfull_ratio = 0.85 2024-02-19T15:27:12.055 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd full ratio = 0.9 2024-02-19T15:27:12.055 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd nearfull ratio = 0.8 2024-02-19T15:27:12.055 INFO:teuthology.orchestra.run.smithi047.stdout: osd failsafe full ratio = 0.95 2024-02-19T15:27:12.055 INFO:teuthology.orchestra.run.smithi047.stdout: osd objectstore = bluestore 2024-02-19T15:27:12.055 INFO:teuthology.orchestra.run.smithi047.stdout: osd op complaint time = 180 2024-02-19T15:27:12.055 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.055 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr] 2024-02-19T15:27:12.055 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:27:12.055 INFO:teuthology.orchestra.run.smithi047.stdout: debug mgr = 20 2024-02-19T15:27:12.055 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-02-19T15:27:12.055 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-02-19T15:27:12.055 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:27:12.055 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:27:12.055 INFO:teuthology.orchestra.run.smithi047.stdout: mgr/telemetry/nag = false 2024-02-19T15:27:12.055 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.056 INFO:teuthology.orchestra.run.smithi047.stdout:[mon] 2024-02-19T15:27:12.056 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:27:12.056 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-02-19T15:27:12.056 INFO:teuthology.orchestra.run.smithi047.stdout: debug paxos = 20 2024-02-19T15:27:12.056 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-02-19T15:27:12.056 INFO:teuthology.orchestra.run.smithi047.stdout: mon data avail warn = 5 2024-02-19T15:27:12.056 INFO:teuthology.orchestra.run.smithi047.stdout: mon mgr mkfs grace = 240 2024-02-19T15:27:12.056 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:27:12.056 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd reporter subtree level = osd 2024-02-19T15:27:12.056 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd prime pg temp = true 2024-02-19T15:27:12.056 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:27:12.056 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.056 INFO:teuthology.orchestra.run.smithi047.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-19T15:27:12.056 INFO:teuthology.orchestra.run.smithi047.stdout: auth mon ticket ttl = 660 # 11m 2024-02-19T15:27:12.056 INFO:teuthology.orchestra.run.smithi047.stdout: auth service ticket ttl = 240 # 4m 2024-02-19T15:27:12.057 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.057 INFO:teuthology.orchestra.run.smithi047.stdout: # don't complain about insecure global_id in the test suite 2024-02-19T15:27:12.057 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-19T15:27:12.057 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-19T15:27:12.057 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.057 INFO:teuthology.orchestra.run.smithi047.stdout: # 1m isn't quite enough 2024-02-19T15:27:12.057 INFO:teuthology.orchestra.run.smithi047.stdout: mon_down_mkfs_grace = 2m 2024-02-19T15:27:12.057 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.057 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_filestore_osds = false 2024-02-19T15:27:12.057 INFO:teuthology.orchestra.run.smithi047.stdout: mon op complaint time = 120 2024-02-19T15:27:12.057 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.057 INFO:teuthology.orchestra.run.smithi047.stdout:[client] 2024-02-19T15:27:12.057 INFO:teuthology.orchestra.run.smithi047.stdout: rgw cache enabled = true 2024-02-19T15:27:12.057 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable ops log = true 2024-02-19T15:27:12.057 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable usage log = true 2024-02-19T15:27:12.057 INFO:teuthology.orchestra.run.smithi047.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-19T15:27:12.058 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-19T15:27:12.058 INFO:teuthology.orchestra.run.smithi047.stdout: client mount timeout = 600 2024-02-19T15:27:12.058 INFO:teuthology.orchestra.run.smithi047.stdout: debug client = 20 2024-02-19T15:27:12.058 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:27:12.058 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = "" 2024-02-19T15:27:12.058 INFO:teuthology.orchestra.run.smithi047.stdout: rados mon op timeout = 900 2024-02-19T15:27:12.058 INFO:teuthology.orchestra.run.smithi047.stdout: rados osd op timeout = 900 2024-02-19T15:27:12.058 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.a] 2024-02-19T15:27:12.058 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.b] 2024-02-19T15:27:12.058 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.c] 2024-02-19T15:27:12.058 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.a] 2024-02-19T15:27:12.058 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.c] 2024-02-19T15:27:12.058 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.b] 2024-02-19T15:27:12.058 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.d] 2024-02-19T15:27:12.058 INFO:teuthology.orchestra.run.smithi047.stdout:[mds] 2024-02-19T15:27:12.058 INFO:teuthology.orchestra.run.smithi047.stdout: debug mds = 20 2024-02-19T15:27:12.058 INFO:teuthology.orchestra.run.smithi047.stdout: debug mds balancer = 20 2024-02-19T15:27:12.059 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:27:12.059 INFO:teuthology.orchestra.run.smithi047.stdout: mds debug frag = True 2024-02-19T15:27:12.059 INFO:teuthology.orchestra.run.smithi047.stdout: mds debug scatterstat = True 2024-02-19T15:27:12.059 INFO:teuthology.orchestra.run.smithi047.stdout: mds op complaint time = 180 2024-02-19T15:27:12.059 INFO:teuthology.orchestra.run.smithi047.stdout: mds verify scatter = True 2024-02-19T15:27:12.059 INFO:teuthology.orchestra.run.smithi047.stdout: osd op complaint time = 180 2024-02-19T15:27:12.059 INFO:teuthology.orchestra.run.smithi047.stdout: rados mon op timeout = 900 2024-02-19T15:27:12.059 INFO:teuthology.orchestra.run.smithi047.stdout: rados osd op timeout = 900 2024-02-19T15:27:12.059 INFO:teuthology.orchestra.run.smithi047.stdout: mds_max_purge_files = 6 2024-02-19T15:27:12.059 INFO:teuthology.orchestra.run.smithi047.stdout: mds_max_purge_ops = 100000000 2024-02-19T15:27:12.059 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-02-19T15:27:12.060 INFO:teuthology.orchestra.run.smithi116.stdout: chdir = "" 2024-02-19T15:27:12.060 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-19T15:27:12.060 INFO:teuthology.orchestra.run.smithi116.stdout: auth supported = cephx 2024-02-19T15:27:12.060 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.060 INFO:teuthology.orchestra.run.smithi116.stdout: filestore xattr use omap = true 2024-02-19T15:27:12.060 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.060 INFO:teuthology.orchestra.run.smithi116.stdout: mon clock drift allowed = 1.000 2024-02-19T15:27:12.060 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.060 INFO:teuthology.orchestra.run.smithi116.stdout: osd crush chooseleaf type = 0 2024-02-19T15:27:12.060 INFO:teuthology.orchestra.run.smithi116.stdout: auth debug = true 2024-02-19T15:27:12.060 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.060 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on old message = true 2024-02-19T15:27:12.060 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on bug = true 2024-02-19T15:27:12.060 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.061 INFO:teuthology.orchestra.run.smithi116.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-19T15:27:12.061 INFO:teuthology.orchestra.run.smithi116.stdout: mon pg warn max object skew = 0 2024-02-19T15:27:12.061 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.061 INFO:teuthology.orchestra.run.smithi116.stdout: # disable pg_autoscaler by default for new pools 2024-02-19T15:27:12.061 INFO:teuthology.orchestra.run.smithi116.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-19T15:27:12.061 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.061 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default size = 2 2024-02-19T15:27:12.061 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.061 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow primary affinity = true 2024-02-19T15:27:12.061 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow pg remap = true 2024-02-19T15:27:12.061 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on legacy crush tunables = false 2024-02-19T15:27:12.061 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on crush straw calc version zero = false 2024-02-19T15:27:12.061 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on no sortbitwise = false 2024-02-19T15:27:12.061 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on osd down out interval zero = false 2024-02-19T15:27:12.061 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on too few osds = false 2024-02-19T15:27:12.062 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-19T15:27:12.062 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-19T15:27:12.062 INFO:teuthology.orchestra.run.smithi116.stdout: mon_allow_pool_size_one = true 2024-02-19T15:27:12.062 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.062 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-19T15:27:12.062 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.062 INFO:teuthology.orchestra.run.smithi116.stdout: osd default data pool replay window = 5 2024-02-19T15:27:12.062 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.062 INFO:teuthology.orchestra.run.smithi116.stdout: mon allow pool delete = true 2024-02-19T15:27:12.062 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.062 INFO:teuthology.orchestra.run.smithi116.stdout: mon cluster log file level = debug 2024-02-19T15:27:12.062 INFO:teuthology.orchestra.run.smithi116.stdout: debug asserts on shutdown = true 2024-02-19T15:27:12.062 INFO:teuthology.orchestra.run.smithi116.stdout: mon health detail to clog = false 2024-02-19T15:27:12.062 INFO:teuthology.orchestra.run.smithi116.stdout: mon host = "172.21.15.47,172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790]" 2024-02-19T15:27:12.062 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on skipped message = False 2024-02-19T15:27:12.062 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e 2024-02-19T15:27:12.063 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.063 INFO:teuthology.orchestra.run.smithi116.stdout:[osd] 2024-02-19T15:27:12.063 INFO:teuthology.orchestra.run.smithi116.stdout: osd journal size = 100 2024-02-19T15:27:12.063 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.063 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub load threshold = 5.0 2024-02-19T15:27:12.063 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub max interval = 600 2024-02-19T15:27:12.063 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock profile = high_recovery_ops 2024-02-19T15:27:12.063 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.063 INFO:teuthology.orchestra.run.smithi116.stdout: osd recover clone overlap = true 2024-02-19T15:27:12.063 INFO:teuthology.orchestra.run.smithi116.stdout: osd recovery max chunk = 1048576 2024-02-19T15:27:12.063 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.063 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug shutdown = true 2024-02-19T15:27:12.063 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug op order = true 2024-02-19T15:27:12.063 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify stray on activate = true 2024-02-19T15:27:12.063 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug trim objects = true 2024-02-19T15:27:12.063 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.064 INFO:teuthology.orchestra.run.smithi116.stdout: osd open classes on start = true 2024-02-19T15:27:12.064 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug pg log writeout = true 2024-02-19T15:27:12.064 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.064 INFO:teuthology.orchestra.run.smithi116.stdout: osd deep scrub update digest min age = 30 2024-02-19T15:27:12.064 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.064 INFO:teuthology.orchestra.run.smithi116.stdout: osd map max advance = 10 2024-02-19T15:27:12.064 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.064 INFO:teuthology.orchestra.run.smithi116.stdout: journal zero on create = true 2024-02-19T15:27:12.064 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.064 INFO:teuthology.orchestra.run.smithi116.stdout: filestore ondisk finisher threads = 3 2024-02-19T15:27:12.064 INFO:teuthology.orchestra.run.smithi116.stdout: filestore apply finisher threads = 3 2024-02-19T15:27:12.064 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.064 INFO:teuthology.orchestra.run.smithi116.stdout: bdev debug aio = true 2024-02-19T15:27:12.064 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug misdirected ops = true 2024-02-19T15:27:12.064 INFO:teuthology.orchestra.run.smithi116.stdout: bdev async discard = True 2024-02-19T15:27:12.064 INFO:teuthology.orchestra.run.smithi116.stdout: bdev enable discard = True 2024-02-19T15:27:12.065 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore allocator = bitmap 2024-02-19T15:27:12.065 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore block size = 96636764160 2024-02-19T15:27:12.065 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore fsck on mount = True 2024-02-19T15:27:12.065 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluefs = 1/20 2024-02-19T15:27:12.065 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluestore = 1/20 2024-02-19T15:27:12.065 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:27:12.065 INFO:teuthology.orchestra.run.smithi116.stdout: debug osd = 20 2024-02-19T15:27:12.065 INFO:teuthology.orchestra.run.smithi116.stdout: debug rocksdb = 4/10 2024-02-19T15:27:12.065 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd backfillfull_ratio = 0.85 2024-02-19T15:27:12.065 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd full ratio = 0.9 2024-02-19T15:27:12.065 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd nearfull ratio = 0.8 2024-02-19T15:27:12.065 INFO:teuthology.orchestra.run.smithi116.stdout: osd failsafe full ratio = 0.95 2024-02-19T15:27:12.065 INFO:teuthology.orchestra.run.smithi116.stdout: osd objectstore = bluestore 2024-02-19T15:27:12.065 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2024-02-19T15:27:12.065 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.066 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr] 2024-02-19T15:27:12.066 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:27:12.066 INFO:teuthology.orchestra.run.smithi116.stdout: debug mgr = 20 2024-02-19T15:27:12.066 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-02-19T15:27:12.066 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-02-19T15:27:12.066 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:27:12.066 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:27:12.066 INFO:teuthology.orchestra.run.smithi116.stdout: mgr/telemetry/nag = false 2024-02-19T15:27:12.066 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.066 INFO:teuthology.orchestra.run.smithi116.stdout:[mon] 2024-02-19T15:27:12.066 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:27:12.066 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-02-19T15:27:12.066 INFO:teuthology.orchestra.run.smithi116.stdout: debug paxos = 20 2024-02-19T15:27:12.066 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-02-19T15:27:12.066 INFO:teuthology.orchestra.run.smithi116.stdout: mon data avail warn = 5 2024-02-19T15:27:12.067 INFO:teuthology.orchestra.run.smithi116.stdout: mon mgr mkfs grace = 240 2024-02-19T15:27:12.067 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:27:12.067 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd reporter subtree level = osd 2024-02-19T15:27:12.067 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd prime pg temp = true 2024-02-19T15:27:12.067 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:27:12.067 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.067 INFO:teuthology.orchestra.run.smithi116.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-19T15:27:12.067 INFO:teuthology.orchestra.run.smithi116.stdout: auth mon ticket ttl = 660 # 11m 2024-02-19T15:27:12.067 INFO:teuthology.orchestra.run.smithi116.stdout: auth service ticket ttl = 240 # 4m 2024-02-19T15:27:12.067 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.067 INFO:teuthology.orchestra.run.smithi116.stdout: # don't complain about insecure global_id in the test suite 2024-02-19T15:27:12.067 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-19T15:27:12.067 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-19T15:27:12.067 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.067 INFO:teuthology.orchestra.run.smithi116.stdout: # 1m isn't quite enough 2024-02-19T15:27:12.067 INFO:teuthology.orchestra.run.smithi116.stdout: mon_down_mkfs_grace = 2m 2024-02-19T15:27:12.068 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.068 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_filestore_osds = false 2024-02-19T15:27:12.068 INFO:teuthology.orchestra.run.smithi116.stdout: mon op complaint time = 120 2024-02-19T15:27:12.068 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.068 INFO:teuthology.orchestra.run.smithi116.stdout:[client] 2024-02-19T15:27:12.068 INFO:teuthology.orchestra.run.smithi116.stdout: rgw cache enabled = true 2024-02-19T15:27:12.068 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable ops log = true 2024-02-19T15:27:12.068 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable usage log = true 2024-02-19T15:27:12.068 INFO:teuthology.orchestra.run.smithi116.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-19T15:27:12.068 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-19T15:27:12.068 INFO:teuthology.orchestra.run.smithi116.stdout: client mount timeout = 600 2024-02-19T15:27:12.068 INFO:teuthology.orchestra.run.smithi116.stdout: debug client = 20 2024-02-19T15:27:12.068 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:27:12.069 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = "" 2024-02-19T15:27:12.069 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2024-02-19T15:27:12.069 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2024-02-19T15:27:12.069 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.a] 2024-02-19T15:27:12.069 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.b] 2024-02-19T15:27:12.069 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.c] 2024-02-19T15:27:12.069 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a] 2024-02-19T15:27:12.069 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c] 2024-02-19T15:27:12.069 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b] 2024-02-19T15:27:12.069 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d] 2024-02-19T15:27:12.069 INFO:teuthology.orchestra.run.smithi116.stdout:[mds] 2024-02-19T15:27:12.069 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds = 20 2024-02-19T15:27:12.069 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds balancer = 20 2024-02-19T15:27:12.069 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:27:12.069 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug frag = True 2024-02-19T15:27:12.070 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug scatterstat = True 2024-02-19T15:27:12.070 INFO:teuthology.orchestra.run.smithi116.stdout: mds op complaint time = 180 2024-02-19T15:27:12.070 INFO:teuthology.orchestra.run.smithi116.stdout: mds verify scatter = True 2024-02-19T15:27:12.070 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2024-02-19T15:27:12.070 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2024-02-19T15:27:12.070 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2024-02-19T15:27:12.070 INFO:teuthology.orchestra.run.smithi116.stdout: mds_max_purge_files = 6 2024-02-19T15:27:12.070 INFO:teuthology.orchestra.run.smithi116.stdout: mds_max_purge_ops = 100000000 2024-02-19T15:27:12.072 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-19T15:27:12.116 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-19T15:27:12.157 INFO:teuthology.orchestra.run.smithi047.stdout:[global] 2024-02-19T15:27:12.157 INFO:teuthology.orchestra.run.smithi047.stdout: chdir = "" 2024-02-19T15:27:12.157 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-19T15:27:12.158 INFO:teuthology.orchestra.run.smithi047.stdout: auth supported = cephx 2024-02-19T15:27:12.158 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.158 INFO:teuthology.orchestra.run.smithi047.stdout: filestore xattr use omap = true 2024-02-19T15:27:12.158 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.158 INFO:teuthology.orchestra.run.smithi047.stdout: mon clock drift allowed = 1.000 2024-02-19T15:27:12.158 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.158 INFO:teuthology.orchestra.run.smithi047.stdout: osd crush chooseleaf type = 0 2024-02-19T15:27:12.158 INFO:teuthology.orchestra.run.smithi047.stdout: auth debug = true 2024-02-19T15:27:12.158 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.158 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on old message = true 2024-02-19T15:27:12.158 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on bug = true 2024-02-19T15:27:12.158 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.159 INFO:teuthology.orchestra.run.smithi047.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-19T15:27:12.159 INFO:teuthology.orchestra.run.smithi047.stdout: mon pg warn max object skew = 0 2024-02-19T15:27:12.159 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.159 INFO:teuthology.orchestra.run.smithi047.stdout: # disable pg_autoscaler by default for new pools 2024-02-19T15:27:12.159 INFO:teuthology.orchestra.run.smithi047.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-19T15:27:12.159 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.159 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default size = 2 2024-02-19T15:27:12.159 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.159 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow primary affinity = true 2024-02-19T15:27:12.159 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow pg remap = true 2024-02-19T15:27:12.159 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on legacy crush tunables = false 2024-02-19T15:27:12.159 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on crush straw calc version zero = false 2024-02-19T15:27:12.159 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on no sortbitwise = false 2024-02-19T15:27:12.159 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on osd down out interval zero = false 2024-02-19T15:27:12.160 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on too few osds = false 2024-02-19T15:27:12.160 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-19T15:27:12.160 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-19T15:27:12.160 INFO:teuthology.orchestra.run.smithi047.stdout: mon_allow_pool_size_one = true 2024-02-19T15:27:12.160 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.160 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-19T15:27:12.160 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.160 INFO:teuthology.orchestra.run.smithi047.stdout: osd default data pool replay window = 5 2024-02-19T15:27:12.160 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.160 INFO:teuthology.orchestra.run.smithi047.stdout: mon allow pool delete = true 2024-02-19T15:27:12.160 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.160 INFO:teuthology.orchestra.run.smithi047.stdout: mon cluster log file level = debug 2024-02-19T15:27:12.160 INFO:teuthology.orchestra.run.smithi047.stdout: debug asserts on shutdown = true 2024-02-19T15:27:12.160 INFO:teuthology.orchestra.run.smithi047.stdout: mon health detail to clog = false 2024-02-19T15:27:12.160 INFO:teuthology.orchestra.run.smithi047.stdout: mon host = "172.21.15.47,172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790]" 2024-02-19T15:27:12.161 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on skipped message = False 2024-02-19T15:27:12.161 INFO:teuthology.orchestra.run.smithi047.stdout: fsid = ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e 2024-02-19T15:27:12.161 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.161 INFO:teuthology.orchestra.run.smithi047.stdout:[osd] 2024-02-19T15:27:12.161 INFO:teuthology.orchestra.run.smithi047.stdout: osd journal size = 100 2024-02-19T15:27:12.161 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.161 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub load threshold = 5.0 2024-02-19T15:27:12.161 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub max interval = 600 2024-02-19T15:27:12.161 INFO:teuthology.orchestra.run.smithi047.stdout: osd mclock profile = high_recovery_ops 2024-02-19T15:27:12.161 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.161 INFO:teuthology.orchestra.run.smithi047.stdout: osd recover clone overlap = true 2024-02-19T15:27:12.161 INFO:teuthology.orchestra.run.smithi047.stdout: osd recovery max chunk = 1048576 2024-02-19T15:27:12.161 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.161 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug shutdown = true 2024-02-19T15:27:12.161 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug op order = true 2024-02-19T15:27:12.162 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify stray on activate = true 2024-02-19T15:27:12.162 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug trim objects = true 2024-02-19T15:27:12.162 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.162 INFO:teuthology.orchestra.run.smithi047.stdout: osd open classes on start = true 2024-02-19T15:27:12.162 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug pg log writeout = true 2024-02-19T15:27:12.162 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.162 INFO:teuthology.orchestra.run.smithi047.stdout: osd deep scrub update digest min age = 30 2024-02-19T15:27:12.162 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.162 INFO:teuthology.orchestra.run.smithi047.stdout: osd map max advance = 10 2024-02-19T15:27:12.162 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.162 INFO:teuthology.orchestra.run.smithi047.stdout: journal zero on create = true 2024-02-19T15:27:12.162 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.162 INFO:teuthology.orchestra.run.smithi047.stdout: filestore ondisk finisher threads = 3 2024-02-19T15:27:12.162 INFO:teuthology.orchestra.run.smithi047.stdout: filestore apply finisher threads = 3 2024-02-19T15:27:12.162 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.162 INFO:teuthology.orchestra.run.smithi047.stdout: bdev debug aio = true 2024-02-19T15:27:12.162 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug misdirected ops = true 2024-02-19T15:27:12.163 INFO:teuthology.orchestra.run.smithi047.stdout: bdev async discard = True 2024-02-19T15:27:12.163 INFO:teuthology.orchestra.run.smithi047.stdout: bdev enable discard = True 2024-02-19T15:27:12.163 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore allocator = bitmap 2024-02-19T15:27:12.163 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore block size = 96636764160 2024-02-19T15:27:12.163 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore fsck on mount = True 2024-02-19T15:27:12.163 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluefs = 1/20 2024-02-19T15:27:12.163 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluestore = 1/20 2024-02-19T15:27:12.163 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:27:12.163 INFO:teuthology.orchestra.run.smithi047.stdout: debug osd = 20 2024-02-19T15:27:12.163 INFO:teuthology.orchestra.run.smithi047.stdout: debug rocksdb = 4/10 2024-02-19T15:27:12.163 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd backfillfull_ratio = 0.85 2024-02-19T15:27:12.163 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd full ratio = 0.9 2024-02-19T15:27:12.163 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd nearfull ratio = 0.8 2024-02-19T15:27:12.163 INFO:teuthology.orchestra.run.smithi047.stdout: osd failsafe full ratio = 0.95 2024-02-19T15:27:12.163 INFO:teuthology.orchestra.run.smithi047.stdout: osd objectstore = bluestore 2024-02-19T15:27:12.163 INFO:teuthology.orchestra.run.smithi047.stdout: osd op complaint time = 180 2024-02-19T15:27:12.163 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.164 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr] 2024-02-19T15:27:12.164 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:27:12.164 INFO:teuthology.orchestra.run.smithi047.stdout: debug mgr = 20 2024-02-19T15:27:12.164 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-02-19T15:27:12.164 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-02-19T15:27:12.164 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:27:12.164 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:27:12.164 INFO:teuthology.orchestra.run.smithi047.stdout: mgr/telemetry/nag = false 2024-02-19T15:27:12.164 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.164 INFO:teuthology.orchestra.run.smithi047.stdout:[mon] 2024-02-19T15:27:12.164 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:27:12.164 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-02-19T15:27:12.164 INFO:teuthology.orchestra.run.smithi047.stdout: debug paxos = 20 2024-02-19T15:27:12.164 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-02-19T15:27:12.164 INFO:teuthology.orchestra.run.smithi047.stdout: mon data avail warn = 5 2024-02-19T15:27:12.164 INFO:teuthology.orchestra.run.smithi047.stdout: mon mgr mkfs grace = 240 2024-02-19T15:27:12.165 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:27:12.165 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd reporter subtree level = osd 2024-02-19T15:27:12.165 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd prime pg temp = true 2024-02-19T15:27:12.165 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:27:12.165 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.165 INFO:teuthology.orchestra.run.smithi047.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-19T15:27:12.165 INFO:teuthology.orchestra.run.smithi047.stdout: auth mon ticket ttl = 660 # 11m 2024-02-19T15:27:12.165 INFO:teuthology.orchestra.run.smithi047.stdout: auth service ticket ttl = 240 # 4m 2024-02-19T15:27:12.165 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.165 INFO:teuthology.orchestra.run.smithi047.stdout: # don't complain about insecure global_id in the test suite 2024-02-19T15:27:12.165 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-19T15:27:12.165 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-19T15:27:12.165 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.165 INFO:teuthology.orchestra.run.smithi047.stdout: # 1m isn't quite enough 2024-02-19T15:27:12.165 INFO:teuthology.orchestra.run.smithi047.stdout: mon_down_mkfs_grace = 2m 2024-02-19T15:27:12.165 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.166 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_filestore_osds = false 2024-02-19T15:27:12.166 INFO:teuthology.orchestra.run.smithi047.stdout: mon op complaint time = 120 2024-02-19T15:27:12.166 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.166 INFO:teuthology.orchestra.run.smithi047.stdout:[client] 2024-02-19T15:27:12.166 INFO:teuthology.orchestra.run.smithi047.stdout: rgw cache enabled = true 2024-02-19T15:27:12.166 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable ops log = true 2024-02-19T15:27:12.166 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable usage log = true 2024-02-19T15:27:12.166 INFO:teuthology.orchestra.run.smithi047.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-19T15:27:12.166 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-19T15:27:12.166 INFO:teuthology.orchestra.run.smithi047.stdout: client mount timeout = 600 2024-02-19T15:27:12.166 INFO:teuthology.orchestra.run.smithi047.stdout: debug client = 20 2024-02-19T15:27:12.166 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:27:12.166 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = "" 2024-02-19T15:27:12.166 INFO:teuthology.orchestra.run.smithi047.stdout: rados mon op timeout = 900 2024-02-19T15:27:12.166 INFO:teuthology.orchestra.run.smithi047.stdout: rados osd op timeout = 900 2024-02-19T15:27:12.167 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.a] 2024-02-19T15:27:12.167 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.b] 2024-02-19T15:27:12.167 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.c] 2024-02-19T15:27:12.167 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.a] 2024-02-19T15:27:12.167 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.c] 2024-02-19T15:27:12.167 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.b] 2024-02-19T15:27:12.167 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.d] 2024-02-19T15:27:12.167 INFO:teuthology.orchestra.run.smithi047.stdout:[mds] 2024-02-19T15:27:12.167 INFO:teuthology.orchestra.run.smithi047.stdout: debug mds = 20 2024-02-19T15:27:12.167 INFO:teuthology.orchestra.run.smithi047.stdout: debug mds balancer = 20 2024-02-19T15:27:12.167 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:27:12.167 INFO:teuthology.orchestra.run.smithi047.stdout: mds debug frag = True 2024-02-19T15:27:12.167 INFO:teuthology.orchestra.run.smithi047.stdout: mds debug scatterstat = True 2024-02-19T15:27:12.167 INFO:teuthology.orchestra.run.smithi047.stdout: mds op complaint time = 180 2024-02-19T15:27:12.167 INFO:teuthology.orchestra.run.smithi047.stdout: mds verify scatter = True 2024-02-19T15:27:12.167 INFO:teuthology.orchestra.run.smithi047.stdout: osd op complaint time = 180 2024-02-19T15:27:12.167 INFO:teuthology.orchestra.run.smithi047.stdout: rados mon op timeout = 900 2024-02-19T15:27:12.167 INFO:teuthology.orchestra.run.smithi047.stdout: rados osd op timeout = 900 2024-02-19T15:27:12.168 INFO:teuthology.orchestra.run.smithi047.stdout: mds_max_purge_files = 6 2024-02-19T15:27:12.168 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-02-19T15:27:12.168 INFO:teuthology.orchestra.run.smithi116.stdout: chdir = "" 2024-02-19T15:27:12.168 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-19T15:27:12.168 INFO:teuthology.orchestra.run.smithi116.stdout: auth supported = cephx 2024-02-19T15:27:12.168 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.168 INFO:teuthology.orchestra.run.smithi116.stdout: filestore xattr use omap = true 2024-02-19T15:27:12.168 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.168 INFO:teuthology.orchestra.run.smithi116.stdout: mon clock drift allowed = 1.000 2024-02-19T15:27:12.168 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.168 INFO:teuthology.orchestra.run.smithi116.stdout: osd crush chooseleaf type = 0 2024-02-19T15:27:12.169 INFO:teuthology.orchestra.run.smithi116.stdout: auth debug = true 2024-02-19T15:27:12.169 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.169 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on old message = true 2024-02-19T15:27:12.169 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on bug = true 2024-02-19T15:27:12.169 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.169 INFO:teuthology.orchestra.run.smithi116.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-19T15:27:12.169 INFO:teuthology.orchestra.run.smithi116.stdout: mon pg warn max object skew = 0 2024-02-19T15:27:12.169 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.169 INFO:teuthology.orchestra.run.smithi116.stdout: # disable pg_autoscaler by default for new pools 2024-02-19T15:27:12.169 INFO:teuthology.orchestra.run.smithi116.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-19T15:27:12.169 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.169 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default size = 2 2024-02-19T15:27:12.169 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.169 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow primary affinity = true 2024-02-19T15:27:12.169 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow pg remap = true 2024-02-19T15:27:12.169 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on legacy crush tunables = false 2024-02-19T15:27:12.170 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on crush straw calc version zero = false 2024-02-19T15:27:12.170 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on no sortbitwise = false 2024-02-19T15:27:12.170 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on osd down out interval zero = false 2024-02-19T15:27:12.170 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on too few osds = false 2024-02-19T15:27:12.170 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-19T15:27:12.170 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-19T15:27:12.170 INFO:teuthology.orchestra.run.smithi116.stdout: mon_allow_pool_size_one = true 2024-02-19T15:27:12.170 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.170 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-19T15:27:12.170 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.170 INFO:teuthology.orchestra.run.smithi116.stdout: osd default data pool replay window = 5 2024-02-19T15:27:12.170 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.170 INFO:teuthology.orchestra.run.smithi116.stdout: mon allow pool delete = true 2024-02-19T15:27:12.170 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.170 INFO:teuthology.orchestra.run.smithi116.stdout: mon cluster log file level = debug 2024-02-19T15:27:12.171 INFO:teuthology.orchestra.run.smithi116.stdout: debug asserts on shutdown = true 2024-02-19T15:27:12.171 INFO:teuthology.orchestra.run.smithi116.stdout: mon health detail to clog = false 2024-02-19T15:27:12.171 INFO:teuthology.orchestra.run.smithi116.stdout: mon host = "172.21.15.47,172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790]" 2024-02-19T15:27:12.171 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on skipped message = False 2024-02-19T15:27:12.171 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e 2024-02-19T15:27:12.171 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.171 INFO:teuthology.orchestra.run.smithi116.stdout:[osd] 2024-02-19T15:27:12.171 INFO:teuthology.orchestra.run.smithi116.stdout: osd journal size = 100 2024-02-19T15:27:12.171 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.171 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub load threshold = 5.0 2024-02-19T15:27:12.171 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub max interval = 600 2024-02-19T15:27:12.171 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock profile = high_recovery_ops 2024-02-19T15:27:12.171 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.171 INFO:teuthology.orchestra.run.smithi116.stdout: osd recover clone overlap = true 2024-02-19T15:27:12.171 INFO:teuthology.orchestra.run.smithi116.stdout: osd recovery max chunk = 1048576 2024-02-19T15:27:12.171 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.172 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug shutdown = true 2024-02-19T15:27:12.172 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug op order = true 2024-02-19T15:27:12.172 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify stray on activate = true 2024-02-19T15:27:12.172 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug trim objects = true 2024-02-19T15:27:12.172 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.172 INFO:teuthology.orchestra.run.smithi116.stdout: osd open classes on start = true 2024-02-19T15:27:12.172 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug pg log writeout = true 2024-02-19T15:27:12.172 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.172 INFO:teuthology.orchestra.run.smithi116.stdout: osd deep scrub update digest min age = 30 2024-02-19T15:27:12.172 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.172 INFO:teuthology.orchestra.run.smithi116.stdout: osd map max advance = 10 2024-02-19T15:27:12.172 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.172 INFO:teuthology.orchestra.run.smithi116.stdout: journal zero on create = true 2024-02-19T15:27:12.172 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.172 INFO:teuthology.orchestra.run.smithi116.stdout: filestore ondisk finisher threads = 3 2024-02-19T15:27:12.173 INFO:teuthology.orchestra.run.smithi116.stdout: filestore apply finisher threads = 3 2024-02-19T15:27:12.173 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.173 INFO:teuthology.orchestra.run.smithi116.stdout: bdev debug aio = true 2024-02-19T15:27:12.173 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug misdirected ops = true 2024-02-19T15:27:12.173 INFO:teuthology.orchestra.run.smithi116.stdout: bdev async discard = True 2024-02-19T15:27:12.173 INFO:teuthology.orchestra.run.smithi116.stdout: bdev enable discard = True 2024-02-19T15:27:12.173 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore allocator = bitmap 2024-02-19T15:27:12.173 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore block size = 96636764160 2024-02-19T15:27:12.173 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore fsck on mount = True 2024-02-19T15:27:12.173 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluefs = 1/20 2024-02-19T15:27:12.173 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluestore = 1/20 2024-02-19T15:27:12.173 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:27:12.173 INFO:teuthology.orchestra.run.smithi116.stdout: debug osd = 20 2024-02-19T15:27:12.173 INFO:teuthology.orchestra.run.smithi116.stdout: debug rocksdb = 4/10 2024-02-19T15:27:12.173 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd backfillfull_ratio = 0.85 2024-02-19T15:27:12.174 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd full ratio = 0.9 2024-02-19T15:27:12.174 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd nearfull ratio = 0.8 2024-02-19T15:27:12.174 INFO:teuthology.orchestra.run.smithi116.stdout: osd failsafe full ratio = 0.95 2024-02-19T15:27:12.174 INFO:teuthology.orchestra.run.smithi116.stdout: osd objectstore = bluestore 2024-02-19T15:27:12.174 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2024-02-19T15:27:12.174 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.174 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr] 2024-02-19T15:27:12.174 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:27:12.174 INFO:teuthology.orchestra.run.smithi116.stdout: debug mgr = 20 2024-02-19T15:27:12.174 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-02-19T15:27:12.174 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-02-19T15:27:12.174 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:27:12.174 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:27:12.174 INFO:teuthology.orchestra.run.smithi116.stdout: mgr/telemetry/nag = false 2024-02-19T15:27:12.174 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.174 INFO:teuthology.orchestra.run.smithi116.stdout:[mon] 2024-02-19T15:27:12.175 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:27:12.175 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-02-19T15:27:12.175 INFO:teuthology.orchestra.run.smithi116.stdout: debug paxos = 20 2024-02-19T15:27:12.175 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-02-19T15:27:12.175 INFO:teuthology.orchestra.run.smithi116.stdout: mon data avail warn = 5 2024-02-19T15:27:12.175 INFO:teuthology.orchestra.run.smithi116.stdout: mon mgr mkfs grace = 240 2024-02-19T15:27:12.175 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:27:12.175 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd reporter subtree level = osd 2024-02-19T15:27:12.175 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd prime pg temp = true 2024-02-19T15:27:12.175 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:27:12.175 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.175 INFO:teuthology.orchestra.run.smithi116.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-19T15:27:12.175 INFO:teuthology.orchestra.run.smithi116.stdout: auth mon ticket ttl = 660 # 11m 2024-02-19T15:27:12.175 INFO:teuthology.orchestra.run.smithi116.stdout: auth service ticket ttl = 240 # 4m 2024-02-19T15:27:12.176 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.176 INFO:teuthology.orchestra.run.smithi116.stdout: # don't complain about insecure global_id in the test suite 2024-02-19T15:27:12.176 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-19T15:27:12.176 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-19T15:27:12.176 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.176 INFO:teuthology.orchestra.run.smithi116.stdout: # 1m isn't quite enough 2024-02-19T15:27:12.176 INFO:teuthology.orchestra.run.smithi116.stdout: mon_down_mkfs_grace = 2m 2024-02-19T15:27:12.176 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.176 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_filestore_osds = false 2024-02-19T15:27:12.176 INFO:teuthology.orchestra.run.smithi116.stdout: mon op complaint time = 120 2024-02-19T15:27:12.176 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.176 INFO:teuthology.orchestra.run.smithi116.stdout:[client] 2024-02-19T15:27:12.176 INFO:teuthology.orchestra.run.smithi116.stdout: rgw cache enabled = true 2024-02-19T15:27:12.176 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable ops log = true 2024-02-19T15:27:12.177 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable usage log = true 2024-02-19T15:27:12.177 INFO:teuthology.orchestra.run.smithi116.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-19T15:27:12.177 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-19T15:27:12.177 INFO:teuthology.orchestra.run.smithi116.stdout: client mount timeout = 600 2024-02-19T15:27:12.177 INFO:teuthology.orchestra.run.smithi116.stdout: debug client = 20 2024-02-19T15:27:12.177 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:27:12.177 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = "" 2024-02-19T15:27:12.177 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2024-02-19T15:27:12.177 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2024-02-19T15:27:12.177 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.a] 2024-02-19T15:27:12.177 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.b] 2024-02-19T15:27:12.177 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.c] 2024-02-19T15:27:12.177 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a] 2024-02-19T15:27:12.177 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c] 2024-02-19T15:27:12.177 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b] 2024-02-19T15:27:12.178 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d] 2024-02-19T15:27:12.178 INFO:teuthology.orchestra.run.smithi116.stdout:[mds] 2024-02-19T15:27:12.178 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds = 20 2024-02-19T15:27:12.178 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds balancer = 20 2024-02-19T15:27:12.178 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:27:12.178 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug frag = True 2024-02-19T15:27:12.178 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug scatterstat = True 2024-02-19T15:27:12.178 INFO:teuthology.orchestra.run.smithi116.stdout: mds op complaint time = 180 2024-02-19T15:27:12.178 INFO:teuthology.orchestra.run.smithi116.stdout: mds verify scatter = True 2024-02-19T15:27:12.178 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2024-02-19T15:27:12.178 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2024-02-19T15:27:12.178 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2024-02-19T15:27:12.178 INFO:teuthology.orchestra.run.smithi116.stdout: mds_max_purge_files = 6 2024-02-19T15:27:12.180 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-19T15:27:12.228 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-19T15:27:12.268 INFO:teuthology.orchestra.run.smithi047.stdout:[global] 2024-02-19T15:27:12.268 INFO:teuthology.orchestra.run.smithi047.stdout: chdir = "" 2024-02-19T15:27:12.268 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-19T15:27:12.268 INFO:teuthology.orchestra.run.smithi047.stdout: auth supported = cephx 2024-02-19T15:27:12.268 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.268 INFO:teuthology.orchestra.run.smithi047.stdout: filestore xattr use omap = true 2024-02-19T15:27:12.268 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.268 INFO:teuthology.orchestra.run.smithi047.stdout: mon clock drift allowed = 1.000 2024-02-19T15:27:12.268 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.268 INFO:teuthology.orchestra.run.smithi047.stdout: osd crush chooseleaf type = 0 2024-02-19T15:27:12.269 INFO:teuthology.orchestra.run.smithi047.stdout: auth debug = true 2024-02-19T15:27:12.269 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.269 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on old message = true 2024-02-19T15:27:12.269 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on bug = true 2024-02-19T15:27:12.269 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.269 INFO:teuthology.orchestra.run.smithi047.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-19T15:27:12.269 INFO:teuthology.orchestra.run.smithi047.stdout: mon pg warn max object skew = 0 2024-02-19T15:27:12.269 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.269 INFO:teuthology.orchestra.run.smithi047.stdout: # disable pg_autoscaler by default for new pools 2024-02-19T15:27:12.269 INFO:teuthology.orchestra.run.smithi047.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-19T15:27:12.269 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.269 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default size = 2 2024-02-19T15:27:12.269 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.269 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow primary affinity = true 2024-02-19T15:27:12.269 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow pg remap = true 2024-02-19T15:27:12.269 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on legacy crush tunables = false 2024-02-19T15:27:12.270 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on crush straw calc version zero = false 2024-02-19T15:27:12.270 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on no sortbitwise = false 2024-02-19T15:27:12.270 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on osd down out interval zero = false 2024-02-19T15:27:12.270 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on too few osds = false 2024-02-19T15:27:12.270 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-19T15:27:12.270 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-19T15:27:12.270 INFO:teuthology.orchestra.run.smithi047.stdout: mon_allow_pool_size_one = true 2024-02-19T15:27:12.270 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.270 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-19T15:27:12.270 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.270 INFO:teuthology.orchestra.run.smithi047.stdout: osd default data pool replay window = 5 2024-02-19T15:27:12.270 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.270 INFO:teuthology.orchestra.run.smithi047.stdout: mon allow pool delete = true 2024-02-19T15:27:12.270 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.270 INFO:teuthology.orchestra.run.smithi047.stdout: mon cluster log file level = debug 2024-02-19T15:27:12.270 INFO:teuthology.orchestra.run.smithi047.stdout: debug asserts on shutdown = true 2024-02-19T15:27:12.270 INFO:teuthology.orchestra.run.smithi047.stdout: mon health detail to clog = false 2024-02-19T15:27:12.271 INFO:teuthology.orchestra.run.smithi047.stdout: mon host = "172.21.15.47,172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790]" 2024-02-19T15:27:12.271 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on skipped message = False 2024-02-19T15:27:12.271 INFO:teuthology.orchestra.run.smithi047.stdout: fsid = ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e 2024-02-19T15:27:12.271 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.271 INFO:teuthology.orchestra.run.smithi047.stdout:[osd] 2024-02-19T15:27:12.271 INFO:teuthology.orchestra.run.smithi047.stdout: osd journal size = 100 2024-02-19T15:27:12.271 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.271 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub load threshold = 5.0 2024-02-19T15:27:12.271 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub max interval = 600 2024-02-19T15:27:12.271 INFO:teuthology.orchestra.run.smithi047.stdout: osd mclock profile = high_recovery_ops 2024-02-19T15:27:12.271 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.271 INFO:teuthology.orchestra.run.smithi047.stdout: osd recover clone overlap = true 2024-02-19T15:27:12.271 INFO:teuthology.orchestra.run.smithi047.stdout: osd recovery max chunk = 1048576 2024-02-19T15:27:12.271 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.271 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug shutdown = true 2024-02-19T15:27:12.272 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug op order = true 2024-02-19T15:27:12.272 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify stray on activate = true 2024-02-19T15:27:12.272 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug trim objects = true 2024-02-19T15:27:12.272 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.272 INFO:teuthology.orchestra.run.smithi047.stdout: osd open classes on start = true 2024-02-19T15:27:12.272 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug pg log writeout = true 2024-02-19T15:27:12.272 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.272 INFO:teuthology.orchestra.run.smithi047.stdout: osd deep scrub update digest min age = 30 2024-02-19T15:27:12.272 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.272 INFO:teuthology.orchestra.run.smithi047.stdout: osd map max advance = 10 2024-02-19T15:27:12.272 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.272 INFO:teuthology.orchestra.run.smithi047.stdout: journal zero on create = true 2024-02-19T15:27:12.272 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.272 INFO:teuthology.orchestra.run.smithi047.stdout: filestore ondisk finisher threads = 3 2024-02-19T15:27:12.272 INFO:teuthology.orchestra.run.smithi047.stdout: filestore apply finisher threads = 3 2024-02-19T15:27:12.272 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.272 INFO:teuthology.orchestra.run.smithi047.stdout: bdev debug aio = true 2024-02-19T15:27:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug misdirected ops = true 2024-02-19T15:27:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: bdev async discard = True 2024-02-19T15:27:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: bdev enable discard = True 2024-02-19T15:27:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore allocator = bitmap 2024-02-19T15:27:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore block size = 96636764160 2024-02-19T15:27:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore fsck on mount = True 2024-02-19T15:27:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluefs = 1/20 2024-02-19T15:27:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluestore = 1/20 2024-02-19T15:27:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:27:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: debug osd = 20 2024-02-19T15:27:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: debug rocksdb = 4/10 2024-02-19T15:27:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd backfillfull_ratio = 0.85 2024-02-19T15:27:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd full ratio = 0.9 2024-02-19T15:27:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd nearfull ratio = 0.8 2024-02-19T15:27:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: osd failsafe full ratio = 0.95 2024-02-19T15:27:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: osd objectstore = bluestore 2024-02-19T15:27:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: osd op complaint time = 180 2024-02-19T15:27:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.274 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr] 2024-02-19T15:27:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:27:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: debug mgr = 20 2024-02-19T15:27:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-02-19T15:27:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-02-19T15:27:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:27:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:27:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: mgr/telemetry/nag = false 2024-02-19T15:27:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.274 INFO:teuthology.orchestra.run.smithi047.stdout:[mon] 2024-02-19T15:27:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:27:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-02-19T15:27:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: debug paxos = 20 2024-02-19T15:27:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-02-19T15:27:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: mon data avail warn = 5 2024-02-19T15:27:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: mon mgr mkfs grace = 240 2024-02-19T15:27:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:27:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd reporter subtree level = osd 2024-02-19T15:27:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd prime pg temp = true 2024-02-19T15:27:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:27:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-19T15:27:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: auth mon ticket ttl = 660 # 11m 2024-02-19T15:27:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: auth service ticket ttl = 240 # 4m 2024-02-19T15:27:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: # don't complain about insecure global_id in the test suite 2024-02-19T15:27:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-19T15:27:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-19T15:27:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: # 1m isn't quite enough 2024-02-19T15:27:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: mon_down_mkfs_grace = 2m 2024-02-19T15:27:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_filestore_osds = false 2024-02-19T15:27:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: mon op complaint time = 120 2024-02-19T15:27:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:12.276 INFO:teuthology.orchestra.run.smithi047.stdout:[client] 2024-02-19T15:27:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: rgw cache enabled = true 2024-02-19T15:27:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable ops log = true 2024-02-19T15:27:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable usage log = true 2024-02-19T15:27:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-19T15:27:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-19T15:27:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: client mount timeout = 600 2024-02-19T15:27:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: debug client = 20 2024-02-19T15:27:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:27:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = "" 2024-02-19T15:27:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: rados mon op timeout = 900 2024-02-19T15:27:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: rados osd op timeout = 900 2024-02-19T15:27:12.277 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.a] 2024-02-19T15:27:12.277 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.b] 2024-02-19T15:27:12.277 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.c] 2024-02-19T15:27:12.277 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.a] 2024-02-19T15:27:12.277 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.c] 2024-02-19T15:27:12.277 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.b] 2024-02-19T15:27:12.277 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.d] 2024-02-19T15:27:12.277 INFO:teuthology.orchestra.run.smithi047.stdout:[mds] 2024-02-19T15:27:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: debug mds = 20 2024-02-19T15:27:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: debug mds balancer = 20 2024-02-19T15:27:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-02-19T15:27:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: mds debug frag = True 2024-02-19T15:27:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: mds debug scatterstat = True 2024-02-19T15:27:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: mds op complaint time = 180 2024-02-19T15:27:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: mds verify scatter = True 2024-02-19T15:27:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: osd op complaint time = 180 2024-02-19T15:27:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: rados mon op timeout = 900 2024-02-19T15:27:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: rados osd op timeout = 900 2024-02-19T15:27:12.278 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-02-19T15:27:12.278 INFO:teuthology.orchestra.run.smithi116.stdout: chdir = "" 2024-02-19T15:27:12.278 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-19T15:27:12.278 INFO:teuthology.orchestra.run.smithi116.stdout: auth supported = cephx 2024-02-19T15:27:12.279 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.279 INFO:teuthology.orchestra.run.smithi116.stdout: filestore xattr use omap = true 2024-02-19T15:27:12.279 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.279 INFO:teuthology.orchestra.run.smithi116.stdout: mon clock drift allowed = 1.000 2024-02-19T15:27:12.279 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.279 INFO:teuthology.orchestra.run.smithi116.stdout: osd crush chooseleaf type = 0 2024-02-19T15:27:12.279 INFO:teuthology.orchestra.run.smithi116.stdout: auth debug = true 2024-02-19T15:27:12.279 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.279 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on old message = true 2024-02-19T15:27:12.279 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on bug = true 2024-02-19T15:27:12.279 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.279 INFO:teuthology.orchestra.run.smithi116.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-19T15:27:12.279 INFO:teuthology.orchestra.run.smithi116.stdout: mon pg warn max object skew = 0 2024-02-19T15:27:12.279 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.279 INFO:teuthology.orchestra.run.smithi116.stdout: # disable pg_autoscaler by default for new pools 2024-02-19T15:27:12.279 INFO:teuthology.orchestra.run.smithi116.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-19T15:27:12.280 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.280 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default size = 2 2024-02-19T15:27:12.280 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.280 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow primary affinity = true 2024-02-19T15:27:12.280 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow pg remap = true 2024-02-19T15:27:12.280 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on legacy crush tunables = false 2024-02-19T15:27:12.280 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on crush straw calc version zero = false 2024-02-19T15:27:12.280 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on no sortbitwise = false 2024-02-19T15:27:12.280 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on osd down out interval zero = false 2024-02-19T15:27:12.280 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on too few osds = false 2024-02-19T15:27:12.280 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-19T15:27:12.280 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-19T15:27:12.280 INFO:teuthology.orchestra.run.smithi116.stdout: mon_allow_pool_size_one = true 2024-02-19T15:27:12.280 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.280 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-19T15:27:12.280 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.281 INFO:teuthology.orchestra.run.smithi116.stdout: osd default data pool replay window = 5 2024-02-19T15:27:12.281 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.281 INFO:teuthology.orchestra.run.smithi116.stdout: mon allow pool delete = true 2024-02-19T15:27:12.281 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.281 INFO:teuthology.orchestra.run.smithi116.stdout: mon cluster log file level = debug 2024-02-19T15:27:12.281 INFO:teuthology.orchestra.run.smithi116.stdout: debug asserts on shutdown = true 2024-02-19T15:27:12.281 INFO:teuthology.orchestra.run.smithi116.stdout: mon health detail to clog = false 2024-02-19T15:27:12.281 INFO:teuthology.orchestra.run.smithi116.stdout: mon host = "172.21.15.47,172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790]" 2024-02-19T15:27:12.281 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on skipped message = False 2024-02-19T15:27:12.281 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e 2024-02-19T15:27:12.281 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.281 INFO:teuthology.orchestra.run.smithi116.stdout:[osd] 2024-02-19T15:27:12.281 INFO:teuthology.orchestra.run.smithi116.stdout: osd journal size = 100 2024-02-19T15:27:12.281 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.281 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub load threshold = 5.0 2024-02-19T15:27:12.281 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub max interval = 600 2024-02-19T15:27:12.282 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock profile = high_recovery_ops 2024-02-19T15:27:12.282 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.282 INFO:teuthology.orchestra.run.smithi116.stdout: osd recover clone overlap = true 2024-02-19T15:27:12.282 INFO:teuthology.orchestra.run.smithi116.stdout: osd recovery max chunk = 1048576 2024-02-19T15:27:12.282 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.282 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug shutdown = true 2024-02-19T15:27:12.282 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug op order = true 2024-02-19T15:27:12.282 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify stray on activate = true 2024-02-19T15:27:12.282 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug trim objects = true 2024-02-19T15:27:12.282 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.282 INFO:teuthology.orchestra.run.smithi116.stdout: osd open classes on start = true 2024-02-19T15:27:12.282 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug pg log writeout = true 2024-02-19T15:27:12.282 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.282 INFO:teuthology.orchestra.run.smithi116.stdout: osd deep scrub update digest min age = 30 2024-02-19T15:27:12.282 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.282 INFO:teuthology.orchestra.run.smithi116.stdout: osd map max advance = 10 2024-02-19T15:27:12.282 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.282 INFO:teuthology.orchestra.run.smithi116.stdout: journal zero on create = true 2024-02-19T15:27:12.283 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.283 INFO:teuthology.orchestra.run.smithi116.stdout: filestore ondisk finisher threads = 3 2024-02-19T15:27:12.283 INFO:teuthology.orchestra.run.smithi116.stdout: filestore apply finisher threads = 3 2024-02-19T15:27:12.283 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.283 INFO:teuthology.orchestra.run.smithi116.stdout: bdev debug aio = true 2024-02-19T15:27:12.283 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug misdirected ops = true 2024-02-19T15:27:12.283 INFO:teuthology.orchestra.run.smithi116.stdout: bdev async discard = True 2024-02-19T15:27:12.283 INFO:teuthology.orchestra.run.smithi116.stdout: bdev enable discard = True 2024-02-19T15:27:12.283 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore allocator = bitmap 2024-02-19T15:27:12.283 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore block size = 96636764160 2024-02-19T15:27:12.283 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore fsck on mount = True 2024-02-19T15:27:12.283 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluefs = 1/20 2024-02-19T15:27:12.283 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluestore = 1/20 2024-02-19T15:27:12.283 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:27:12.283 INFO:teuthology.orchestra.run.smithi116.stdout: debug osd = 20 2024-02-19T15:27:12.283 INFO:teuthology.orchestra.run.smithi116.stdout: debug rocksdb = 4/10 2024-02-19T15:27:12.284 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd backfillfull_ratio = 0.85 2024-02-19T15:27:12.284 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd full ratio = 0.9 2024-02-19T15:27:12.284 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd nearfull ratio = 0.8 2024-02-19T15:27:12.284 INFO:teuthology.orchestra.run.smithi116.stdout: osd failsafe full ratio = 0.95 2024-02-19T15:27:12.284 INFO:teuthology.orchestra.run.smithi116.stdout: osd objectstore = bluestore 2024-02-19T15:27:12.284 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2024-02-19T15:27:12.284 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.284 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr] 2024-02-19T15:27:12.284 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:27:12.284 INFO:teuthology.orchestra.run.smithi116.stdout: debug mgr = 20 2024-02-19T15:27:12.284 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-02-19T15:27:12.284 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-02-19T15:27:12.284 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:27:12.284 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:27:12.284 INFO:teuthology.orchestra.run.smithi116.stdout: mgr/telemetry/nag = false 2024-02-19T15:27:12.284 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.285 INFO:teuthology.orchestra.run.smithi116.stdout:[mon] 2024-02-19T15:27:12.285 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:27:12.285 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-02-19T15:27:12.285 INFO:teuthology.orchestra.run.smithi116.stdout: debug paxos = 20 2024-02-19T15:27:12.285 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-02-19T15:27:12.285 INFO:teuthology.orchestra.run.smithi116.stdout: mon data avail warn = 5 2024-02-19T15:27:12.285 INFO:teuthology.orchestra.run.smithi116.stdout: mon mgr mkfs grace = 240 2024-02-19T15:27:12.285 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-02-19T15:27:12.285 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd reporter subtree level = osd 2024-02-19T15:27:12.285 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd prime pg temp = true 2024-02-19T15:27:12.285 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-02-19T15:27:12.285 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.285 INFO:teuthology.orchestra.run.smithi116.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-19T15:27:12.285 INFO:teuthology.orchestra.run.smithi116.stdout: auth mon ticket ttl = 660 # 11m 2024-02-19T15:27:12.285 INFO:teuthology.orchestra.run.smithi116.stdout: auth service ticket ttl = 240 # 4m 2024-02-19T15:27:12.285 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.285 INFO:teuthology.orchestra.run.smithi116.stdout: # don't complain about insecure global_id in the test suite 2024-02-19T15:27:12.286 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-19T15:27:12.286 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-19T15:27:12.286 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.286 INFO:teuthology.orchestra.run.smithi116.stdout: # 1m isn't quite enough 2024-02-19T15:27:12.286 INFO:teuthology.orchestra.run.smithi116.stdout: mon_down_mkfs_grace = 2m 2024-02-19T15:27:12.286 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.286 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_filestore_osds = false 2024-02-19T15:27:12.286 INFO:teuthology.orchestra.run.smithi116.stdout: mon op complaint time = 120 2024-02-19T15:27:12.286 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-02-19T15:27:12.286 INFO:teuthology.orchestra.run.smithi116.stdout:[client] 2024-02-19T15:27:12.286 INFO:teuthology.orchestra.run.smithi116.stdout: rgw cache enabled = true 2024-02-19T15:27:12.286 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable ops log = true 2024-02-19T15:27:12.286 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable usage log = true 2024-02-19T15:27:12.286 INFO:teuthology.orchestra.run.smithi116.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-19T15:27:12.286 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-19T15:27:12.286 INFO:teuthology.orchestra.run.smithi116.stdout: client mount timeout = 600 2024-02-19T15:27:12.286 INFO:teuthology.orchestra.run.smithi116.stdout: debug client = 20 2024-02-19T15:27:12.287 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:27:12.287 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = "" 2024-02-19T15:27:12.287 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2024-02-19T15:27:12.287 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2024-02-19T15:27:12.287 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.a] 2024-02-19T15:27:12.287 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.b] 2024-02-19T15:27:12.287 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.c] 2024-02-19T15:27:12.287 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a] 2024-02-19T15:27:12.287 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c] 2024-02-19T15:27:12.287 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b] 2024-02-19T15:27:12.287 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d] 2024-02-19T15:27:12.287 INFO:teuthology.orchestra.run.smithi116.stdout:[mds] 2024-02-19T15:27:12.287 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds = 20 2024-02-19T15:27:12.287 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds balancer = 20 2024-02-19T15:27:12.287 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-02-19T15:27:12.287 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug frag = True 2024-02-19T15:27:12.287 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug scatterstat = True 2024-02-19T15:27:12.288 INFO:teuthology.orchestra.run.smithi116.stdout: mds op complaint time = 180 2024-02-19T15:27:12.288 INFO:teuthology.orchestra.run.smithi116.stdout: mds verify scatter = True 2024-02-19T15:27:12.288 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2024-02-19T15:27:12.288 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2024-02-19T15:27:12.288 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2024-02-19T15:27:12.289 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-19T15:27:12.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.475+0000 7f0dff800640 1 Processor -- start 2024-02-19T15:27:12.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.475+0000 7f0dff800640 1 -- start start 2024-02-19T15:27:12.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.479+0000 7f0dff800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0df815e210 0x7f0df80ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:12.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.479+0000 7f0dff800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0df80cc180 0x7f0df80ce8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:12.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.479+0000 7f0dff800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df80b2fd0 con 0x7f0df80d2650 2024-02-19T15:27:12.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.479+0000 7f0dff800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0df80cfe00 con 0x7f0df815e210 2024-02-19T15:27:12.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.479+0000 7f0dff800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0df80cff90 con 0x7f0df80cc180 2024-02-19T15:27:12.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.479+0000 7f0dfce00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0df815e210 0x7f0df80ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:12.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.479+0000 7f0dfce00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0df815e210 0x7f0df80ce3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56558/0 (socket says 172.21.15.47:56558) 2024-02-19T15:27:12.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.479+0000 7f0dfce00640 1 -- 172.21.15.47:0/3694338029 learned_addr learned my addr 172.21.15.47:0/3694338029 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:12.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.479+0000 7f0df7e00640 1 --2- 172.21.15.47:0/3694338029 >> v2:172.21.15.116:3300/0 conn(0x7f0df80cc180 0x7f0df80ce8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:12.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.479+0000 7f0dfce00640 1 -- 172.21.15.47:0/3694338029 >> v2:172.21.15.116:3300/0 conn(0x7f0df80cc180 msgr2=0x7f0df80ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:12.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.479+0000 7f0dfce00640 1 --2- 172.21.15.47:0/3694338029 >> v2:172.21.15.116:3300/0 conn(0x7f0df80cc180 0x7f0df80ce8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:12.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.479+0000 7f0dfce00640 1 -- 172.21.15.47:0/3694338029 >> v1:172.21.15.116:6789/0 conn(0x7f0df80d2650 legacy=0x7f0df80d03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:12.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.479+0000 7f0dfce00640 1 -- 172.21.15.47:0/3694338029 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df80b2cf0 con 0x7f0df815e210 2024-02-19T15:27:12.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.479+0000 7f0df7e00640 1 --2- 172.21.15.47:0/3694338029 >> v2:172.21.15.116:3300/0 conn(0x7f0df80cc180 0x7f0df80ce8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:27:12.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.483+0000 7f0dfce00640 1 --2- 172.21.15.47:0/3694338029 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0df815e210 0x7f0df80ce3b0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f0df8070d40 tx=0x7f0dec01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e27560a989acea server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:12.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.483+0000 7f0df7400640 1 -- 172.21.15.47:0/3694338029 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dec0ab660 con 0x7f0df815e210 2024-02-19T15:27:12.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.483+0000 7f0df7400640 1 -- 172.21.15.47:0/3694338029 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dec0ab7c0 con 0x7f0df815e210 2024-02-19T15:27:12.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.483+0000 7f0df7400640 1 -- 172.21.15.47:0/3694338029 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dec0abb50 con 0x7f0df815e210 2024-02-19T15:27:12.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.483+0000 7f0dff800640 1 -- 172.21.15.47:0/3694338029 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0df815e210 msgr2=0x7f0df80ce3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:12.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.483+0000 7f0dff800640 1 --2- 172.21.15.47:0/3694338029 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0df815e210 0x7f0df80ce3b0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f0df8070d40 tx=0x7f0dec01c460 comp rx=0 tx=0).stop 2024-02-19T15:27:12.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.483+0000 7f0dff800640 1 -- 172.21.15.47:0/3694338029 shutdown_connections 2024-02-19T15:27:12.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.483+0000 7f0dff800640 1 --2- 172.21.15.47:0/3694338029 >> v2:172.21.15.116:3300/0 conn(0x7f0df80cc180 0x7f0df80ce8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:12.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.483+0000 7f0dff800640 1 --2- 172.21.15.47:0/3694338029 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0df815e210 0x7f0df80ce3b0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:12.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.483+0000 7f0dff800640 1 -- 172.21.15.47:0/3694338029 >> 172.21.15.47:0/3694338029 conn(0x7f0df80d1b30 msgr2=0x7f0df80d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:12.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.483+0000 7f0dff800640 1 -- 172.21.15.47:0/3694338029 shutdown_connections 2024-02-19T15:27:12.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.483+0000 7f0dff800640 1 -- 172.21.15.47:0/3694338029 wait complete. 2024-02-19T15:27:12.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.483+0000 7f0dff800640 1 Processor -- start 2024-02-19T15:27:12.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.483+0000 7f0dff800640 1 -- start start 2024-02-19T15:27:12.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0dff800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0df80cc180 0x7f0df81552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:12.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0dff800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0df80d2650 0x7f0df80c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:12.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0dff800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0df815e210 0x7f0df80c6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:12.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0dff800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df815b6c0 con 0x7f0df80cc180 2024-02-19T15:27:12.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0dff800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df8155e70 con 0x7f0df80d2650 2024-02-19T15:27:12.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0dff800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df80c7330 con 0x7f0df815e210 2024-02-19T15:27:12.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0dfce00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0df80cc180 0x7f0df81552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:12.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0dfce00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0df80cc180 0x7f0df81552e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56566/0 (socket says 172.21.15.47:56566) 2024-02-19T15:27:12.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0dfce00640 1 -- 172.21.15.47:0/1686933754 learned_addr learned my addr 172.21.15.47:0/1686933754 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:12.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0df7e00640 1 --2- 172.21.15.47:0/1686933754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0df80d2650 0x7f0df80c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:12.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0dfd800640 1 --2- 172.21.15.47:0/1686933754 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0df815e210 0x7f0df80c6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:12.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0dfce00640 1 -- 172.21.15.47:0/1686933754 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0df815e210 msgr2=0x7f0df80c6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:12.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0dfce00640 1 --2- 172.21.15.47:0/1686933754 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0df815e210 0x7f0df80c6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:12.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0dfce00640 1 -- 172.21.15.47:0/1686933754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0df80d2650 msgr2=0x7f0df80c67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:12.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0dfce00640 1 --2- 172.21.15.47:0/1686933754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0df80d2650 0x7f0df80c67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:12.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0dfce00640 1 -- 172.21.15.47:0/1686933754 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dec0094d0 con 0x7f0df80cc180 2024-02-19T15:27:12.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0dfd800640 1 --2- 172.21.15.47:0/1686933754 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0df815e210 0x7f0df80c6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:27:12.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0dfce00640 1 --2- 172.21.15.47:0/1686933754 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0df80cc180 0x7f0df81552e0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f0dec019070 tx=0x7f0dec01aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:12.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0df4c00640 1 -- 172.21.15.47:0/1686933754 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dec0b4800 con 0x7f0df80cc180 2024-02-19T15:27:12.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0df4c00640 1 -- 172.21.15.47:0/1686933754 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dec0b4960 con 0x7f0df80cc180 2024-02-19T15:27:12.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0df4c00640 1 -- 172.21.15.47:0/1686933754 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dec0b4d10 con 0x7f0df80cc180 2024-02-19T15:27:12.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0dff800640 1 -- 172.21.15.47:0/1686933754 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df80c7610 con 0x7f0df80cc180 2024-02-19T15:27:12.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0dff800640 1 -- 172.21.15.47:0/1686933754 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df8155820 con 0x7f0df80cc180 2024-02-19T15:27:12.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0dff800640 1 -- 172.21.15.47:0/1686933754 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df8155a10 con 0x7f0df80cc180 2024-02-19T15:27:12.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.487+0000 7f0df4c00640 1 -- 172.21.15.47:0/1686933754 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0dec0a9070 con 0x7f0df80cc180 2024-02-19T15:27:12.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.491+0000 7f0df4c00640 1 --2- 172.21.15.47:0/1686933754 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0dd8066e40 0x7f0dd8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:12.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.491+0000 7f0df4c00640 1 -- 172.21.15.47:0/1686933754 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f0dec11f0e0 con 0x7f0df80cc180 2024-02-19T15:27:12.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.491+0000 7f0df7e00640 1 --2- 172.21.15.47:0/1686933754 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0dd8066e40 0x7f0dd8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:12.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.491+0000 7f0df7e00640 1 --2- 172.21.15.47:0/1686933754 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0dd8066e40 0x7f0dd8069300 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0df80cbcc0 tx=0x7f0de4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:12.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.495+0000 7f0df4c00640 1 -- 172.21.15.47:0/1686933754 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0df8155a10 con 0x7f0df80cc180 2024-02-19T15:27:12.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.655+0000 7f0dff800640 1 -- 172.21.15.47:0/1686933754 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f0df815b1b0 con 0x7f0df80cc180 2024-02-19T15:27:12.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.659+0000 7f0df4c00640 1 -- 172.21.15.47:0/1686933754 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0df815b1b0 con 0x7f0df80cc180 2024-02-19T15:27:12.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.663+0000 7f0dff800640 1 -- 172.21.15.47:0/1686933754 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0dd8066e40 msgr2=0x7f0dd8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:12.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.663+0000 7f0dff800640 1 --2- 172.21.15.47:0/1686933754 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0dd8066e40 0x7f0dd8069300 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0df80cbcc0 tx=0x7f0de4008040 comp rx=0 tx=0).stop 2024-02-19T15:27:12.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.663+0000 7f0dff800640 1 -- 172.21.15.47:0/1686933754 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0df80cc180 msgr2=0x7f0df81552e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:12.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.663+0000 7f0dff800640 1 --2- 172.21.15.47:0/1686933754 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0df80cc180 0x7f0df81552e0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f0dec019070 tx=0x7f0dec01aca0 comp rx=0 tx=0).stop 2024-02-19T15:27:12.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.663+0000 7f0dff800640 1 -- 172.21.15.47:0/1686933754 shutdown_connections 2024-02-19T15:27:12.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.663+0000 7f0dff800640 1 --2- 172.21.15.47:0/1686933754 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0dd8066e40 0x7f0dd8069300 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:12.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.663+0000 7f0dff800640 1 --2- 172.21.15.47:0/1686933754 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0df815e210 0x7f0df80c6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:12.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.663+0000 7f0dff800640 1 --2- 172.21.15.47:0/1686933754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0df80d2650 0x7f0df80c67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:12.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.663+0000 7f0dff800640 1 --2- 172.21.15.47:0/1686933754 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0df80cc180 0x7f0df81552e0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:12.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.663+0000 7f0dff800640 1 -- 172.21.15.47:0/1686933754 >> 172.21.15.47:0/1686933754 conn(0x7f0df80d1b30 msgr2=0x7f0df8159e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:12.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.663+0000 7f0dff800640 1 -- 172.21.15.47:0/1686933754 shutdown_connections 2024-02-19T15:27:12.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.663+0000 7f0dff800640 1 -- 172.21.15.47:0/1686933754 wait complete. 2024-02-19T15:27:12.686 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-02-19T15:27:12.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.823+0000 7f2b66000640 1 Processor -- start 2024-02-19T15:27:12.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.823+0000 7f2b66000640 1 -- start start 2024-02-19T15:27:12.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.823+0000 7f2b66000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2b6015dd70 0x7f2b600ae860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:12.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.823+0000 7f2b66000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2b600aeda0 0x7f2b6006d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:12.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.823+0000 7f2b66000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b6006d660 0x7f2b600d19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:12.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.823+0000 7f2b66000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b6015f3f0 con 0x7f2b600aeda0 2024-02-19T15:27:12.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.823+0000 7f2b66000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b6006dab0 con 0x7f2b6015dd70 2024-02-19T15:27:12.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.823+0000 7f2b66000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6006c3e0 con 0x7f2b6006d660 2024-02-19T15:27:12.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.823+0000 7f2b5fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2b600aeda0 0x7f2b6006d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:12.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.823+0000 7f2b5fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2b600aeda0 0x7f2b6006d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56578/0 (socket says 172.21.15.47:56578) 2024-02-19T15:27:12.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.823+0000 7f2b5fe00640 1 -- 172.21.15.47:0/690121848 learned_addr learned my addr 172.21.15.47:0/690121848 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:12.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.823+0000 7f2b64c00640 1 --2- 172.21.15.47:0/690121848 >> v2:172.21.15.116:3300/0 conn(0x7f2b6015dd70 0x7f2b600ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:12.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.823+0000 7f2b65600640 1 --2- 172.21.15.47:0/690121848 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b6006d660 0x7f2b600d19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:12.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.823+0000 7f2b5fe00640 1 -- 172.21.15.47:0/690121848 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b6006d660 msgr2=0x7f2b600d19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:12.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.823+0000 7f2b5fe00640 1 --2- 172.21.15.47:0/690121848 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b6006d660 0x7f2b600d19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:12.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.823+0000 7f2b5fe00640 1 -- 172.21.15.47:0/690121848 >> v2:172.21.15.116:3300/0 conn(0x7f2b6015dd70 msgr2=0x7f2b600ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:12.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.823+0000 7f2b5fe00640 1 --2- 172.21.15.47:0/690121848 >> v2:172.21.15.116:3300/0 conn(0x7f2b6015dd70 0x7f2b600ae860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:12.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.823+0000 7f2b5fe00640 1 -- 172.21.15.47:0/690121848 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6006c680 con 0x7f2b600aeda0 2024-02-19T15:27:12.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.823+0000 7f2b64c00640 1 --2- 172.21.15.47:0/690121848 >> v2:172.21.15.116:3300/0 conn(0x7f2b6015dd70 0x7f2b600ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:27:12.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.823+0000 7f2b65600640 1 --2- 172.21.15.47:0/690121848 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b6006d660 0x7f2b600d19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:27:12.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.827+0000 7f2b5fe00640 1 --2- 172.21.15.47:0/690121848 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b600aeda0 0x7f2b6006d120 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f2b6006e880 tx=0x7f2b4c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b2c87282424992 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:12.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.827+0000 7f2b5f400640 1 -- 172.21.15.47:0/690121848 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c0ab600 con 0x7f2b600aeda0 2024-02-19T15:27:12.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.827+0000 7f2b5f400640 1 -- 172.21.15.47:0/690121848 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b4c0a7050 con 0x7f2b600aeda0 2024-02-19T15:27:12.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.827+0000 7f2b5f400640 1 -- 172.21.15.47:0/690121848 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c0ab980 con 0x7f2b600aeda0 2024-02-19T15:27:12.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.827+0000 7f2b66000640 1 -- 172.21.15.47:0/690121848 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b600aeda0 msgr2=0x7f2b6006d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:12.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.827+0000 7f2b66000640 1 --2- 172.21.15.47:0/690121848 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b600aeda0 0x7f2b6006d120 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f2b6006e880 tx=0x7f2b4c01c460 comp rx=0 tx=0).stop 2024-02-19T15:27:12.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.827+0000 7f2b66000640 1 -- 172.21.15.47:0/690121848 shutdown_connections 2024-02-19T15:27:12.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.827+0000 7f2b66000640 1 --2- 172.21.15.47:0/690121848 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b6006d660 0x7f2b600d19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:12.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.827+0000 7f2b66000640 1 --2- 172.21.15.47:0/690121848 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b600aeda0 0x7f2b6006d120 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:12.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.827+0000 7f2b66000640 1 --2- 172.21.15.47:0/690121848 >> v2:172.21.15.116:3300/0 conn(0x7f2b6015dd70 0x7f2b600ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:12.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.827+0000 7f2b66000640 1 -- 172.21.15.47:0/690121848 >> 172.21.15.47:0/690121848 conn(0x7f2b6009a9d0 msgr2=0x7f2b6009ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:12.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.827+0000 7f2b66000640 1 -- 172.21.15.47:0/690121848 shutdown_connections 2024-02-19T15:27:12.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.827+0000 7f2b66000640 1 -- 172.21.15.47:0/690121848 wait complete. 2024-02-19T15:27:12.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.827+0000 7f2b66000640 1 Processor -- start 2024-02-19T15:27:12.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.827+0000 7f2b66000640 1 -- start start 2024-02-19T15:27:12.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b66000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b6006d660 0x7f2b6015c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:12.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b66000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b600aeda0 0x7f2b60151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:12.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b66000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b6015dd70 0x7f2b60161ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:12.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b66000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6015c4c0 con 0x7f2b6015dd70 2024-02-19T15:27:12.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b66000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6015b840 con 0x7f2b6006d660 2024-02-19T15:27:12.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b66000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6015bb20 con 0x7f2b600aeda0 2024-02-19T15:27:12.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b65600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b6015dd70 0x7f2b60161ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:12.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b65600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b6015dd70 0x7f2b60161ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56590/0 (socket says 172.21.15.47:56590) 2024-02-19T15:27:12.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b65600640 1 -- 172.21.15.47:0/1719406632 learned_addr learned my addr 172.21.15.47:0/1719406632 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:12.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b64c00640 1 --2- 172.21.15.47:0/1719406632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b6006d660 0x7f2b6015c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:12.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b5fe00640 1 --2- 172.21.15.47:0/1719406632 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b600aeda0 0x7f2b60151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:12.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b65600640 1 -- 172.21.15.47:0/1719406632 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b600aeda0 msgr2=0x7f2b60151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:12.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b65600640 1 --2- 172.21.15.47:0/1719406632 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b600aeda0 0x7f2b60151d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:12.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b65600640 1 -- 172.21.15.47:0/1719406632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b6006d660 msgr2=0x7f2b6015c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:12.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b65600640 1 --2- 172.21.15.47:0/1719406632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b6006d660 0x7f2b6015c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:12.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b65600640 1 -- 172.21.15.47:0/1719406632 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b4c0094d0 con 0x7f2b6015dd70 2024-02-19T15:27:12.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b65600640 1 --2- 172.21.15.47:0/1719406632 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b6015dd70 0x7f2b60161ec0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f2b58007df0 tx=0x7f2b5800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:12.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b5cc00640 1 -- 172.21.15.47:0/1719406632 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b580039a0 con 0x7f2b6015dd70 2024-02-19T15:27:12.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b5cc00640 1 -- 172.21.15.47:0/1719406632 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5800f040 con 0x7f2b6015dd70 2024-02-19T15:27:12.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b5cc00640 1 -- 172.21.15.47:0/1719406632 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b58003d40 con 0x7f2b6015dd70 2024-02-19T15:27:12.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b66000640 1 -- 172.21.15.47:0/1719406632 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b6015be00 con 0x7f2b6015dd70 2024-02-19T15:27:12.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b66000640 1 -- 172.21.15.47:0/1719406632 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b6015ce20 con 0x7f2b6015dd70 2024-02-19T15:27:12.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.831+0000 7f2b66000640 1 -- 172.21.15.47:0/1719406632 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b60075360 con 0x7f2b6015dd70 2024-02-19T15:27:12.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.835+0000 7f2b5cc00640 1 -- 172.21.15.47:0/1719406632 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2b58004030 con 0x7f2b6015dd70 2024-02-19T15:27:12.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.835+0000 7f2b5cc00640 1 --2- 172.21.15.47:0/1719406632 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2b40066e40 0x7f2b40069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:12.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.835+0000 7f2b5cc00640 1 -- 172.21.15.47:0/1719406632 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f2b58084b40 con 0x7f2b6015dd70 2024-02-19T15:27:12.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.835+0000 7f2b64c00640 1 --2- 172.21.15.47:0/1719406632 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2b40066e40 0x7f2b40069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:12.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.835+0000 7f2b64c00640 1 --2- 172.21.15.47:0/1719406632 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2b40066e40 0x7f2b40069300 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2b60156c70 tx=0x7f2b54008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:12.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:12.839+0000 7f2b5cc00640 1 -- 172.21.15.47:0/1719406632 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2b58051b70 con 0x7f2b6015dd70 2024-02-19T15:27:13.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:13.059+0000 7f2b66000640 1 -- 172.21.15.47:0/1719406632 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7f2b600d27c0 con 0x7f2b6015dd70 2024-02-19T15:27:13.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:13.975+0000 7f2b5cc00640 1 -- 172.21.15.47:0/1719406632 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v295) v1 ==== 134+0+0 (secure 0 0 0) 0x7f2b5801a0c0 con 0x7f2b6015dd70 2024-02-19T15:27:13.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:13.975+0000 7f2b66000640 1 -- 172.21.15.47:0/1719406632 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2b40066e40 msgr2=0x7f2b40069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:14.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:13.975+0000 7f2b66000640 1 --2- 172.21.15.47:0/1719406632 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2b40066e40 0x7f2b40069300 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2b60156c70 tx=0x7f2b54008040 comp rx=0 tx=0).stop 2024-02-19T15:27:14.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:13.975+0000 7f2b66000640 1 -- 172.21.15.47:0/1719406632 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b6015dd70 msgr2=0x7f2b60161ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:14.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:13.975+0000 7f2b66000640 1 --2- 172.21.15.47:0/1719406632 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b6015dd70 0x7f2b60161ec0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f2b58007df0 tx=0x7f2b5800d4a0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:13.975+0000 7f2b66000640 1 -- 172.21.15.47:0/1719406632 shutdown_connections 2024-02-19T15:27:14.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:13.975+0000 7f2b66000640 1 --2- 172.21.15.47:0/1719406632 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2b40066e40 0x7f2b40069300 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:13.975+0000 7f2b66000640 1 --2- 172.21.15.47:0/1719406632 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b6015dd70 0x7f2b60161ec0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:13.975+0000 7f2b66000640 1 --2- 172.21.15.47:0/1719406632 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b600aeda0 0x7f2b60151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:13.975+0000 7f2b66000640 1 --2- 172.21.15.47:0/1719406632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b6006d660 0x7f2b6015c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:13.975+0000 7f2b66000640 1 -- 172.21.15.47:0/1719406632 >> 172.21.15.47:0/1719406632 conn(0x7f2b6009a9d0 msgr2=0x7f2b600b6b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:14.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:13.975+0000 7f2b66000640 1 -- 172.21.15.47:0/1719406632 shutdown_connections 2024-02-19T15:27:14.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:13.975+0000 7f2b66000640 1 -- 172.21.15.47:0/1719406632 wait complete. 2024-02-19T15:27:14.004 INFO:tasks.cephfs_test_runner:test_files_throttle (tasks.cephfs.test_strays.TestStrays) ... ok 2024-02-19T15:27:14.005 INFO:tasks.cephfs_test_runner:Starting test: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) That removal of primary dentry of hardlinked inode results 2024-02-19T15:27:14.005 INFO:tasks.cephfs_test_runner:test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) 2024-02-19T15:27:14.005 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:27:14.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.147+0000 7f7a1a200640 1 Processor -- start 2024-02-19T15:27:14.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.147+0000 7f7a1a200640 1 -- start start 2024-02-19T15:27:14.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.147+0000 7f7a1a200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7a140acec0 0x7f7a1406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:14.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.147+0000 7f7a1a200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a140cdf90 0x7f7a140ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:14.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.147+0000 7f7a1a200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a1406e290 con 0x7f7a1406e470 2024-02-19T15:27:14.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.147+0000 7f7a1a200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a1409e010 con 0x7f7a140acec0 2024-02-19T15:27:14.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.147+0000 7f7a1a200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1409dd30 con 0x7f7a140cdf90 2024-02-19T15:27:14.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.147+0000 7f7a18e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7a140acec0 0x7f7a1406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:14.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.147+0000 7f7a18e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7a140acec0 0x7f7a1406dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56602/0 (socket says 172.21.15.47:56602) 2024-02-19T15:27:14.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.147+0000 7f7a18e00640 1 -- 172.21.15.47:0/3875946849 learned_addr learned my addr 172.21.15.47:0/3875946849 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:14.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.147+0000 7f7a19800640 1 --2- 172.21.15.47:0/3875946849 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a140cdf90 0x7f7a140ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:14.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.147+0000 7f7a18e00640 1 -- 172.21.15.47:0/3875946849 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a140cdf90 msgr2=0x7f7a140ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:14.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.147+0000 7f7a18e00640 1 --2- 172.21.15.47:0/3875946849 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a140cdf90 0x7f7a140ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.147+0000 7f7a18e00640 1 -- 172.21.15.47:0/3875946849 >> v1:172.21.15.116:6789/0 conn(0x7f7a1406e470 legacy=0x7f7a140ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:14.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.147+0000 7f7a18e00640 1 -- 172.21.15.47:0/3875946849 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1406cf90 con 0x7f7a140acec0 2024-02-19T15:27:14.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.151+0000 7f7a18e00640 1 --2- 172.21.15.47:0/3875946849 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a140acec0 0x7f7a1406dd50 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f7a140736c0 tx=0x7f7a0801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=451834f65c1a7e94 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:14.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.151+0000 7f7a13400640 1 -- 172.21.15.47:0/3875946849 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a080ad640 con 0x7f7a140acec0 2024-02-19T15:27:14.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.151+0000 7f7a13400640 1 -- 172.21.15.47:0/3875946849 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a080ad7a0 con 0x7f7a140acec0 2024-02-19T15:27:14.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.151+0000 7f7a13400640 1 -- 172.21.15.47:0/3875946849 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a080adb30 con 0x7f7a140acec0 2024-02-19T15:27:14.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.151+0000 7f7a1a200640 1 -- 172.21.15.47:0/3875946849 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a140acec0 msgr2=0x7f7a1406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:14.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.151+0000 7f7a1a200640 1 --2- 172.21.15.47:0/3875946849 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a140acec0 0x7f7a1406dd50 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f7a140736c0 tx=0x7f7a0801c4d0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.151+0000 7f7a1a200640 1 -- 172.21.15.47:0/3875946849 shutdown_connections 2024-02-19T15:27:14.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.151+0000 7f7a1a200640 1 --2- 172.21.15.47:0/3875946849 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a140cdf90 0x7f7a140ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.151+0000 7f7a1a200640 1 --2- 172.21.15.47:0/3875946849 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a140acec0 0x7f7a1406dd50 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.151+0000 7f7a1a200640 1 -- 172.21.15.47:0/3875946849 >> 172.21.15.47:0/3875946849 conn(0x7f7a140b1be0 msgr2=0x7f7a140b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:14.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.151+0000 7f7a1a200640 1 -- 172.21.15.47:0/3875946849 shutdown_connections 2024-02-19T15:27:14.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.151+0000 7f7a1a200640 1 -- 172.21.15.47:0/3875946849 wait complete. 2024-02-19T15:27:14.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a1a200640 1 Processor -- start 2024-02-19T15:27:14.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a1a200640 1 -- start start 2024-02-19T15:27:14.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a1a200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a1406e470 0x7f7a140afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:14.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a1a200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a140acec0 0x7f7a140c8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:14.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a1a200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a140cdf90 0x7f7a140c2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:14.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a1a200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a141601a0 con 0x7f7a140cdf90 2024-02-19T15:27:14.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a1a200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1415f900 con 0x7f7a140acec0 2024-02-19T15:27:14.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a1a200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1415fbe0 con 0x7f7a1406e470 2024-02-19T15:27:14.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a19800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a140cdf90 0x7f7a140c2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:14.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a18e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a1406e470 0x7f7a140afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:14.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a19800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a140cdf90 0x7f7a140c2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56618/0 (socket says 172.21.15.47:56618) 2024-02-19T15:27:14.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a19800640 1 -- 172.21.15.47:0/3349609886 learned_addr learned my addr 172.21.15.47:0/3349609886 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:14.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a13e00640 1 --2- 172.21.15.47:0/3349609886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a140acec0 0x7f7a140c8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:14.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a19800640 1 -- 172.21.15.47:0/3349609886 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a1406e470 msgr2=0x7f7a140afcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:14.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a19800640 1 --2- 172.21.15.47:0/3349609886 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a1406e470 0x7f7a140afcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a19800640 1 -- 172.21.15.47:0/3349609886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a140acec0 msgr2=0x7f7a140c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:14.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a19800640 1 --2- 172.21.15.47:0/3349609886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a140acec0 0x7f7a140c8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a19800640 1 -- 172.21.15.47:0/3349609886 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a080094d0 con 0x7f7a140cdf90 2024-02-19T15:27:14.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a19800640 1 --2- 172.21.15.47:0/3349609886 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a140cdf90 0x7f7a140c2b00 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c007e40 tx=0x7f7a0c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:14.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a10c00640 1 -- 172.21.15.47:0/3349609886 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c014620 con 0x7f7a140cdf90 2024-02-19T15:27:14.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a10c00640 1 -- 172.21.15.47:0/3349609886 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0c00ee90 con 0x7f7a140cdf90 2024-02-19T15:27:14.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a1a200640 1 -- 172.21.15.47:0/3349609886 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a1415fec0 con 0x7f7a140cdf90 2024-02-19T15:27:14.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a10c00640 1 -- 172.21.15.47:0/3349609886 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c0149e0 con 0x7f7a140cdf90 2024-02-19T15:27:14.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a1a200640 1 -- 172.21.15.47:0/3349609886 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a140b01f0 con 0x7f7a140cdf90 2024-02-19T15:27:14.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.155+0000 7f7a1a200640 1 -- 172.21.15.47:0/3349609886 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a140b03e0 con 0x7f7a140cdf90 2024-02-19T15:27:14.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.159+0000 7f7a10c00640 1 -- 172.21.15.47:0/3349609886 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7a0c014b40 con 0x7f7a140cdf90 2024-02-19T15:27:14.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.159+0000 7f7a10c00640 1 --2- 172.21.15.47:0/3349609886 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f79f4066e40 0x7f79f4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:14.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.159+0000 7f7a10c00640 1 -- 172.21.15.47:0/3349609886 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f7a0c085080 con 0x7f7a140cdf90 2024-02-19T15:27:14.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.159+0000 7f7a18e00640 1 --2- 172.21.15.47:0/3349609886 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f79f4066e40 0x7f79f4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:14.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.159+0000 7f7a18e00640 1 --2- 172.21.15.47:0/3349609886 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f79f4066e40 0x7f79f4069300 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7a080af000 tx=0x7f7a08008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:14.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.163+0000 7f7a10c00640 1 -- 172.21.15.47:0/3349609886 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7a0c0b71c0 con 0x7f7a140cdf90 2024-02-19T15:27:14.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.323+0000 7f7a1a200640 1 -- 172.21.15.47:0/3349609886 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a14075360 con 0x7f7a140cdf90 2024-02-19T15:27:14.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.323+0000 7f7a10c00640 1 -- 172.21.15.47:0/3349609886 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+13292 (secure 0 0 0) 0x7f7a0c052160 con 0x7f7a140cdf90 2024-02-19T15:27:14.329 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:14.329 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:27:11.966782+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6835/3852677195":"2024-02-20T15:27:08.690016+0000","172.21.15.116:6834/3852677195":"2024-02-20T15:27:08.690016+0000","172.21.15.47:6833/68555355":"2024-02-20T15:26:40.818800+0000","172.21.15.47:6832/68555355":"2024-02-20T15:26:40.818800+0000","172.21.15.116:6837/547176070":"2024-02-20T15:26:39.960651+0000","172.21.15.116:6835/547176070":"2024-02-20T15:26:39.960651+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:27:14.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.327+0000 7f7a1a200640 1 -- 172.21.15.47:0/3349609886 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f79f4066e40 msgr2=0x7f79f4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:14.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.327+0000 7f7a1a200640 1 --2- 172.21.15.47:0/3349609886 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f79f4066e40 0x7f79f4069300 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7a080af000 tx=0x7f7a08008040 comp rx=0 tx=0).stop 2024-02-19T15:27:14.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.327+0000 7f7a1a200640 1 -- 172.21.15.47:0/3349609886 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a140cdf90 msgr2=0x7f7a140c2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:14.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.327+0000 7f7a1a200640 1 --2- 172.21.15.47:0/3349609886 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a140cdf90 0x7f7a140c2b00 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c007e40 tx=0x7f7a0c00e4b0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.327+0000 7f7a1a200640 1 -- 172.21.15.47:0/3349609886 shutdown_connections 2024-02-19T15:27:14.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.327+0000 7f7a1a200640 1 --2- 172.21.15.47:0/3349609886 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f79f4066e40 0x7f79f4069300 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.327+0000 7f7a1a200640 1 --2- 172.21.15.47:0/3349609886 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a140cdf90 0x7f7a140c2b00 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.327+0000 7f7a1a200640 1 --2- 172.21.15.47:0/3349609886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a140acec0 0x7f7a140c8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.327+0000 7f7a1a200640 1 --2- 172.21.15.47:0/3349609886 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a1406e470 0x7f7a140afcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.327+0000 7f7a1a200640 1 -- 172.21.15.47:0/3349609886 >> 172.21.15.47:0/3349609886 conn(0x7f7a140b1be0 msgr2=0x7f7a140cc3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:14.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.327+0000 7f7a1a200640 1 -- 172.21.15.47:0/3349609886 shutdown_connections 2024-02-19T15:27:14.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.327+0000 7f7a1a200640 1 -- 172.21.15.47:0/3349609886 wait complete. 2024-02-19T15:27:14.351 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:27:14.351 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:27:14.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.491+0000 7f2e1e200640 1 Processor -- start 2024-02-19T15:27:14.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.491+0000 7f2e1e200640 1 -- start start 2024-02-19T15:27:14.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.491+0000 7f2e1e200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2e180cb490 0x7f2e18070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:14.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.491+0000 7f2e1e200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e18072850 0x7f2e1806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:14.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.491+0000 7f2e1e200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e1815fc90 con 0x7f2e1815bd90 2024-02-19T15:27:14.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.491+0000 7f2e1e200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e1815ff70 con 0x7f2e180cb490 2024-02-19T15:27:14.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.491+0000 7f2e1e200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e18160250 con 0x7f2e18072850 2024-02-19T15:27:14.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.491+0000 7f2e0ea00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2e180cb490 0x7f2e18070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:14.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.491+0000 7f2e0ea00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2e180cb490 0x7f2e18070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56624/0 (socket says 172.21.15.47:56624) 2024-02-19T15:27:14.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.491+0000 7f2e0ea00640 1 -- 172.21.15.47:0/1143182307 learned_addr learned my addr 172.21.15.47:0/1143182307 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:14.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.491+0000 7f2e17400640 1 --1- 172.21.15.47:0/1143182307 >> v1:172.21.15.47:6789/0 conn(0x7f2e1815bd90 0x7f2e18072140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49006/0 (socket says 172.21.15.47:49006) 2024-02-19T15:27:14.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.491+0000 7f2e17e00640 1 --2- 172.21.15.47:0/1143182307 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e18072850 0x7f2e1806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:14.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.491+0000 7f2e0ea00640 1 -- 172.21.15.47:0/1143182307 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e18072850 msgr2=0x7f2e1806cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:14.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.491+0000 7f2e0ea00640 1 --2- 172.21.15.47:0/1143182307 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e18072850 0x7f2e1806cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.491+0000 7f2e0ea00640 1 -- 172.21.15.47:0/1143182307 >> v1:172.21.15.47:6789/0 conn(0x7f2e1815bd90 legacy=0x7f2e18072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:14.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.491+0000 7f2e0ea00640 1 -- 172.21.15.47:0/1143182307 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e18160530 con 0x7f2e180cb490 2024-02-19T15:27:14.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.495+0000 7f2e0ea00640 1 --2- 172.21.15.47:0/1143182307 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e180cb490 0x7f2e18070960 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f2e00009870 tx=0x7f2e0001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38897bafb2098e2b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:14.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.495+0000 7f2e16a00640 1 -- 172.21.15.47:0/1143182307 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e00004230 con 0x7f2e180cb490 2024-02-19T15:27:14.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.495+0000 7f2e16a00640 1 -- 172.21.15.47:0/1143182307 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e000a7050 con 0x7f2e180cb490 2024-02-19T15:27:14.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.495+0000 7f2e16a00640 1 -- 172.21.15.47:0/1143182307 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e00003620 con 0x7f2e180cb490 2024-02-19T15:27:14.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.495+0000 7f2e1e200640 1 -- 172.21.15.47:0/1143182307 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e180cb490 msgr2=0x7f2e18070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:14.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.495+0000 7f2e1e200640 1 --2- 172.21.15.47:0/1143182307 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e180cb490 0x7f2e18070960 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f2e00009870 tx=0x7f2e0001c400 comp rx=0 tx=0).stop 2024-02-19T15:27:14.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.495+0000 7f2e1e200640 1 -- 172.21.15.47:0/1143182307 shutdown_connections 2024-02-19T15:27:14.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.495+0000 7f2e1e200640 1 --2- 172.21.15.47:0/1143182307 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e18072850 0x7f2e1806cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.495+0000 7f2e1e200640 1 --2- 172.21.15.47:0/1143182307 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e180cb490 0x7f2e18070960 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.495+0000 7f2e1e200640 1 -- 172.21.15.47:0/1143182307 >> 172.21.15.47:0/1143182307 conn(0x7f2e180cfc20 msgr2=0x7f2e180d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:14.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.495+0000 7f2e1e200640 1 -- 172.21.15.47:0/1143182307 shutdown_connections 2024-02-19T15:27:14.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.495+0000 7f2e1e200640 1 -- 172.21.15.47:0/1143182307 wait complete. 2024-02-19T15:27:14.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e1e200640 1 Processor -- start 2024-02-19T15:27:14.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e1e200640 1 -- start start 2024-02-19T15:27:14.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e1e200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e18072850 0x7f2e181480a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:14.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e1e200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e180cb490 0x7f2e180c6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:14.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e1e200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e1815bd90 0x7f2e180c88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:14.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e1e200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1815c870 con 0x7f2e180cb490 2024-02-19T15:27:14.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e1e200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e181485e0 con 0x7f2e18072850 2024-02-19T15:27:14.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e1e200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e180c9650 con 0x7f2e1815bd90 2024-02-19T15:27:14.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e0ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e180cb490 0x7f2e180c6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:14.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e0ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e180cb490 0x7f2e180c6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56626/0 (socket says 172.21.15.47:56626) 2024-02-19T15:27:14.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e0ea00640 1 -- 172.21.15.47:0/496472346 learned_addr learned my addr 172.21.15.47:0/496472346 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:14.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e0ea00640 1 -- 172.21.15.47:0/496472346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e1815bd90 msgr2=0x7f2e180c88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:14.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e0ea00640 1 --2- 172.21.15.47:0/496472346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e1815bd90 0x7f2e180c88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e0ea00640 1 -- 172.21.15.47:0/496472346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e18072850 msgr2=0x7f2e181480a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:14.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e0ea00640 1 --2- 172.21.15.47:0/496472346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e18072850 0x7f2e181480a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e0ea00640 1 -- 172.21.15.47:0/496472346 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e000094d0 con 0x7f2e180cb490 2024-02-19T15:27:14.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e0ea00640 1 --2- 172.21.15.47:0/496472346 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e180cb490 0x7f2e180c6cf0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f2e000ae040 tx=0x7f2e000191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:14.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e0fe00640 1 -- 172.21.15.47:0/496472346 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e00003bf0 con 0x7f2e180cb490 2024-02-19T15:27:14.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e0fe00640 1 -- 172.21.15.47:0/496472346 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e00003d90 con 0x7f2e180cb490 2024-02-19T15:27:14.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e0fe00640 1 -- 172.21.15.47:0/496472346 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e000ab680 con 0x7f2e180cb490 2024-02-19T15:27:14.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e1e200640 1 -- 172.21.15.47:0/496472346 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e180c9930 con 0x7f2e180cb490 2024-02-19T15:27:14.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e1e200640 1 -- 172.21.15.47:0/496472346 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e180c9ef0 con 0x7f2e180cb490 2024-02-19T15:27:14.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.499+0000 7f2e1e200640 1 -- 172.21.15.47:0/496472346 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e18075360 con 0x7f2e180cb490 2024-02-19T15:27:14.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.503+0000 7f2e0fe00640 1 -- 172.21.15.47:0/496472346 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2e000b2040 con 0x7f2e180cb490 2024-02-19T15:27:14.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.503+0000 7f2e0fe00640 1 --2- 172.21.15.47:0/496472346 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2df4066e40 0x7f2df4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:14.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.503+0000 7f2e0fe00640 1 -- 172.21.15.47:0/496472346 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f2e0011ed50 con 0x7f2e180cb490 2024-02-19T15:27:14.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.503+0000 7f2e17400640 1 --2- 172.21.15.47:0/496472346 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2df4066e40 0x7f2df4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:14.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.503+0000 7f2e17400640 1 --2- 172.21.15.47:0/496472346 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2df4066e40 0x7f2df4069300 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2e08000f70 tx=0x7f2e08007a10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:14.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.507+0000 7f2e0fe00640 1 -- 172.21.15.47:0/496472346 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2e000ebd70 con 0x7f2e180cb490 2024-02-19T15:27:14.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.671+0000 7f2e1e200640 1 -- 172.21.15.47:0/496472346 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2e1815c9b0 con 0x7f2e180cb490 2024-02-19T15:27:14.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.671+0000 7f2e0fe00640 1 -- 172.21.15.47:0/496472346 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2e1815c9b0 con 0x7f2e180cb490 2024-02-19T15:27:14.676 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:27:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.675+0000 7f2e1e200640 1 -- 172.21.15.47:0/496472346 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2df4066e40 msgr2=0x7f2df4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.675+0000 7f2e1e200640 1 --2- 172.21.15.47:0/496472346 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2df4066e40 0x7f2df4069300 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2e08000f70 tx=0x7f2e08007a10 comp rx=0 tx=0).stop 2024-02-19T15:27:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.675+0000 7f2e1e200640 1 -- 172.21.15.47:0/496472346 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e180cb490 msgr2=0x7f2e180c6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.675+0000 7f2e1e200640 1 --2- 172.21.15.47:0/496472346 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e180cb490 0x7f2e180c6cf0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f2e000ae040 tx=0x7f2e000191a0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.675+0000 7f2e1e200640 1 -- 172.21.15.47:0/496472346 shutdown_connections 2024-02-19T15:27:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.675+0000 7f2e1e200640 1 --2- 172.21.15.47:0/496472346 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2df4066e40 0x7f2df4069300 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.675+0000 7f2e1e200640 1 --2- 172.21.15.47:0/496472346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e1815bd90 0x7f2e180c88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.675+0000 7f2e1e200640 1 --2- 172.21.15.47:0/496472346 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e180cb490 0x7f2e180c6cf0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.675+0000 7f2e1e200640 1 --2- 172.21.15.47:0/496472346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e18072850 0x7f2e181480a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.675+0000 7f2e1e200640 1 -- 172.21.15.47:0/496472346 >> 172.21.15.47:0/496472346 conn(0x7f2e180cfc20 msgr2=0x7f2e18157fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.675+0000 7f2e1e200640 1 -- 172.21.15.47:0/496472346 shutdown_connections 2024-02-19T15:27:14.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.675+0000 7f2e1e200640 1 -- 172.21.15.47:0/496472346 wait complete. 2024-02-19T15:27:14.698 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-02-19T15:27:14.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.831+0000 7f3578c00640 1 Processor -- start 2024-02-19T15:27:14.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.835+0000 7f3578c00640 1 -- start start 2024-02-19T15:27:14.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.835+0000 7f3578c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f35740cea80 0x7f35740727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:14.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.835+0000 7f3578c00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f357415d280 con 0x7f3574145590 2024-02-19T15:27:14.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.835+0000 7f3578c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3574070ff0 con 0x7f357415c390 2024-02-19T15:27:14.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.835+0000 7f3578c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35740709e0 con 0x7f35740cea80 2024-02-19T15:27:14.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.835+0000 7f3569400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f35740cea80 0x7f35740727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:14.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.835+0000 7f3569400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f35740cea80 0x7f35740727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56642/0 (socket says 172.21.15.47:56642) 2024-02-19T15:27:14.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.835+0000 7f3569400640 1 -- 172.21.15.47:0/3997863521 learned_addr learned my addr 172.21.15.47:0/3997863521 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:14.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.835+0000 7f3571e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f3574145590 0x7f357406c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49022/0 (socket says 172.21.15.47:49022) 2024-02-19T15:27:14.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.835+0000 7f3569400640 1 -- 172.21.15.47:0/3997863521 >> v1:172.21.15.116:6789/0 conn(0x7f357415c390 legacy=0x7f357415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:14.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.835+0000 7f3569400640 1 -- 172.21.15.47:0/3997863521 >> v1:172.21.15.47:6789/0 conn(0x7f3574145590 legacy=0x7f357406c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:14.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.835+0000 7f3569400640 1 -- 172.21.15.47:0/3997863521 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35740cf020 con 0x7f35740cea80 2024-02-19T15:27:14.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.835+0000 7f3571400640 1 -- 172.21.15.47:0/3997863521 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1781082170 0 0) 0x7f357415d280 con 0x7f3574145590 2024-02-19T15:27:14.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.839+0000 7f3569400640 1 --2- 172.21.15.47:0/3997863521 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35740cea80 0x7f35740727b0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f355c009870 tx=0x7f355c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceec43e1b70b686c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:14.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.839+0000 7f3571400640 1 -- 172.21.15.47:0/3997863521 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c0a7070 con 0x7f35740cea80 2024-02-19T15:27:14.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.839+0000 7f3571400640 1 -- 172.21.15.47:0/3997863521 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355c0ab430 con 0x7f35740cea80 2024-02-19T15:27:14.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.839+0000 7f3571400640 1 -- 172.21.15.47:0/3997863521 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c0ab7c0 con 0x7f35740cea80 2024-02-19T15:27:14.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.839+0000 7f3578c00640 1 -- 172.21.15.47:0/3997863521 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35740cea80 msgr2=0x7f35740727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:14.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.839+0000 7f3578c00640 1 --2- 172.21.15.47:0/3997863521 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35740cea80 0x7f35740727b0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f355c009870 tx=0x7f355c01c520 comp rx=0 tx=0).stop 2024-02-19T15:27:14.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.839+0000 7f3578c00640 1 -- 172.21.15.47:0/3997863521 shutdown_connections 2024-02-19T15:27:14.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.839+0000 7f3578c00640 1 --2- 172.21.15.47:0/3997863521 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35740cea80 0x7f35740727b0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.839+0000 7f3578c00640 1 -- 172.21.15.47:0/3997863521 >> 172.21.15.47:0/3997863521 conn(0x7f35740b16a0 msgr2=0x7f35740b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:14.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.839+0000 7f3578c00640 1 -- 172.21.15.47:0/3997863521 shutdown_connections 2024-02-19T15:27:14.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.839+0000 7f3578c00640 1 -- 172.21.15.47:0/3997863521 wait complete. 2024-02-19T15:27:14.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.839+0000 7f3578c00640 1 Processor -- start 2024-02-19T15:27:14.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.839+0000 7f3578c00640 1 -- start start 2024-02-19T15:27:14.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f3578c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35740cea80 0x7f35740c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:14.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f3578c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3574145590 0x7f35740ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:14.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f3578c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f357415c390 0x7f3574143370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:14.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f3578c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35740c58f0 con 0x7f35740cea80 2024-02-19T15:27:14.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f3578c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35740cabd0 con 0x7f357415c390 2024-02-19T15:27:14.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f3578c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35740cae50 con 0x7f3574145590 2024-02-19T15:27:14.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f3571e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35740cea80 0x7f35740c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:14.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f3571e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35740cea80 0x7f35740c53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56656/0 (socket says 172.21.15.47:56656) 2024-02-19T15:27:14.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f3571e00640 1 -- 172.21.15.47:0/3409640825 learned_addr learned my addr 172.21.15.47:0/3409640825 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:14.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f3569400640 1 --2- 172.21.15.47:0/3409640825 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3574145590 0x7f35740ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:14.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f3572800640 1 --2- 172.21.15.47:0/3409640825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f357415c390 0x7f3574143370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:14.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f3571e00640 1 -- 172.21.15.47:0/3409640825 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3574145590 msgr2=0x7f35740ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:14.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f3571e00640 1 --2- 172.21.15.47:0/3409640825 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3574145590 0x7f35740ca690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f3571e00640 1 -- 172.21.15.47:0/3409640825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f357415c390 msgr2=0x7f3574143370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:14.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f3571e00640 1 --2- 172.21.15.47:0/3409640825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f357415c390 0x7f3574143370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:14.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f3571e00640 1 -- 172.21.15.47:0/3409640825 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c0094d0 con 0x7f35740cea80 2024-02-19T15:27:14.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f3571e00640 1 --2- 172.21.15.47:0/3409640825 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35740cea80 0x7f35740c53b0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f3564001570 tx=0x7f356400cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:14.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f356aa00640 1 -- 172.21.15.47:0/3409640825 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356400a640 con 0x7f35740cea80 2024-02-19T15:27:14.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f356aa00640 1 -- 172.21.15.47:0/3409640825 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356400a7c0 con 0x7f35740cea80 2024-02-19T15:27:14.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f356aa00640 1 -- 172.21.15.47:0/3409640825 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356400ab70 con 0x7f35740cea80 2024-02-19T15:27:14.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f3578c00640 1 -- 172.21.15.47:0/3409640825 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35740cb130 con 0x7f35740cea80 2024-02-19T15:27:14.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f3578c00640 1 -- 172.21.15.47:0/3409640825 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35740cb6f0 con 0x7f35740cea80 2024-02-19T15:27:14.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f3578c00640 1 -- 172.21.15.47:0/3409640825 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3574075360 con 0x7f35740cea80 2024-02-19T15:27:14.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.843+0000 7f356aa00640 1 -- 172.21.15.47:0/3409640825 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f356400acd0 con 0x7f35740cea80 2024-02-19T15:27:14.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.847+0000 7f356aa00640 1 --2- 172.21.15.47:0/3409640825 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3550066e40 0x7f3550069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:14.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.847+0000 7f356aa00640 1 -- 172.21.15.47:0/3409640825 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f3564083ef0 con 0x7f35740cea80 2024-02-19T15:27:14.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.847+0000 7f3569400640 1 --2- 172.21.15.47:0/3409640825 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3550066e40 0x7f3550069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:14.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.847+0000 7f3569400640 1 --2- 172.21.15.47:0/3409640825 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3550066e40 0x7f3550069300 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f355c0af000 tx=0x7f355c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:14.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:14.851+0000 7f356aa00640 1 -- 172.21.15.47:0/3409640825 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f3564050e90 con 0x7f35740cea80 2024-02-19T15:27:15.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:15.071+0000 7f3578c00640 1 -- 172.21.15.47:0/3409640825 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f357406e320 con 0x7f35740cea80 2024-02-19T15:27:15.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:15.979+0000 7f356aa00640 1 -- 172.21.15.47:0/3409640825 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v297) v1 ==== 145+0+0 (secure 0 0 0) 0x7f3564018020 con 0x7f35740cea80 2024-02-19T15:27:15.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:15.983+0000 7f3578c00640 1 -- 172.21.15.47:0/3409640825 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3550066e40 msgr2=0x7f3550069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:15.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:15.983+0000 7f3578c00640 1 --2- 172.21.15.47:0/3409640825 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3550066e40 0x7f3550069300 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f355c0af000 tx=0x7f355c008040 comp rx=0 tx=0).stop 2024-02-19T15:27:15.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:15.983+0000 7f3578c00640 1 -- 172.21.15.47:0/3409640825 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35740cea80 msgr2=0x7f35740c53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:15.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:15.983+0000 7f3578c00640 1 --2- 172.21.15.47:0/3409640825 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35740cea80 0x7f35740c53b0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f3564001570 tx=0x7f356400cd20 comp rx=0 tx=0).stop 2024-02-19T15:27:15.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:15.983+0000 7f3578c00640 1 -- 172.21.15.47:0/3409640825 shutdown_connections 2024-02-19T15:27:15.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:15.983+0000 7f3578c00640 1 --2- 172.21.15.47:0/3409640825 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3550066e40 0x7f3550069300 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:15.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:15.983+0000 7f3578c00640 1 --2- 172.21.15.47:0/3409640825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f357415c390 0x7f3574143370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:15.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:15.983+0000 7f3578c00640 1 --2- 172.21.15.47:0/3409640825 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3574145590 0x7f35740ca690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:15.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:15.983+0000 7f3578c00640 1 --2- 172.21.15.47:0/3409640825 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35740cea80 0x7f35740c53b0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:15.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:15.983+0000 7f3578c00640 1 -- 172.21.15.47:0/3409640825 >> 172.21.15.47:0/3409640825 conn(0x7f35740b16a0 msgr2=0x7f35740723d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:15.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:15.983+0000 7f3578c00640 1 -- 172.21.15.47:0/3409640825 shutdown_connections 2024-02-19T15:27:15.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:15.983+0000 7f3578c00640 1 -- 172.21.15.47:0/3409640825 wait complete. 2024-02-19T15:27:16.012 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-19T15:27:16.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.155+0000 7f3b9f800640 1 Processor -- start 2024-02-19T15:27:16.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.155+0000 7f3b9f800640 1 -- start start 2024-02-19T15:27:16.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.155+0000 7f3b9f800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3b980cb490 0x7f3b98070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:16.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.155+0000 7f3b9f800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3b98072850 0x7f3b9806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:16.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.155+0000 7f3b9f800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9815fc90 con 0x7f3b9815bd90 2024-02-19T15:27:16.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.155+0000 7f3b9f800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b9815ff70 con 0x7f3b98072850 2024-02-19T15:27:16.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.155+0000 7f3b9f800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b98160250 con 0x7f3b980cb490 2024-02-19T15:27:16.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.155+0000 7f3b9d800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3b98072850 0x7f3b9806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:16.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.155+0000 7f3b9d800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3b98072850 0x7f3b9806cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56670/0 (socket says 172.21.15.47:56670) 2024-02-19T15:27:16.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.155+0000 7f3b9d800640 1 -- 172.21.15.47:0/126397281 learned_addr learned my addr 172.21.15.47:0/126397281 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:16.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.155+0000 7f3b97e00640 1 --2- 172.21.15.47:0/126397281 >> v2:172.21.15.116:3300/0 conn(0x7f3b980cb490 0x7f3b98070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:16.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.159+0000 7f3b9d800640 1 -- 172.21.15.47:0/126397281 >> v2:172.21.15.116:3300/0 conn(0x7f3b980cb490 msgr2=0x7f3b98070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:16.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.159+0000 7f3b9d800640 1 --2- 172.21.15.47:0/126397281 >> v2:172.21.15.116:3300/0 conn(0x7f3b980cb490 0x7f3b98070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.159+0000 7f3b9d800640 1 -- 172.21.15.47:0/126397281 >> v1:172.21.15.116:6789/0 conn(0x7f3b9815bd90 legacy=0x7f3b98072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:16.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.159+0000 7f3b9d800640 1 -- 172.21.15.47:0/126397281 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b98160530 con 0x7f3b98072850 2024-02-19T15:27:16.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.159+0000 7f3b9d800640 1 --2- 172.21.15.47:0/126397281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b98072850 0x7f3b9806cc60 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f3b980b5620 tx=0x7f3b9001c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc9acfb93d0ec6c6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:16.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.159+0000 7f3b97400640 1 -- 172.21.15.47:0/126397281 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b900a5660 con 0x7f3b98072850 2024-02-19T15:27:16.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.159+0000 7f3b97400640 1 -- 172.21.15.47:0/126397281 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b900a57c0 con 0x7f3b98072850 2024-02-19T15:27:16.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.159+0000 7f3b97400640 1 -- 172.21.15.47:0/126397281 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b900a5b50 con 0x7f3b98072850 2024-02-19T15:27:16.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.159+0000 7f3b9f800640 1 -- 172.21.15.47:0/126397281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b98072850 msgr2=0x7f3b9806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:16.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.159+0000 7f3b9f800640 1 --2- 172.21.15.47:0/126397281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b98072850 0x7f3b9806cc60 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f3b980b5620 tx=0x7f3b9001c890 comp rx=0 tx=0).stop 2024-02-19T15:27:16.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.159+0000 7f3b9f800640 1 -- 172.21.15.47:0/126397281 shutdown_connections 2024-02-19T15:27:16.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.159+0000 7f3b9f800640 1 --2- 172.21.15.47:0/126397281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b98072850 0x7f3b9806cc60 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.159+0000 7f3b9f800640 1 --2- 172.21.15.47:0/126397281 >> v2:172.21.15.116:3300/0 conn(0x7f3b980cb490 0x7f3b98070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.159+0000 7f3b9f800640 1 -- 172.21.15.47:0/126397281 >> 172.21.15.47:0/126397281 conn(0x7f3b980cfc20 msgr2=0x7f3b980d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:16.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.159+0000 7f3b9f800640 1 -- 172.21.15.47:0/126397281 shutdown_connections 2024-02-19T15:27:16.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.159+0000 7f3b9f800640 1 -- 172.21.15.47:0/126397281 wait complete. 2024-02-19T15:27:16.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9f800640 1 Processor -- start 2024-02-19T15:27:16.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9f800640 1 -- start start 2024-02-19T15:27:16.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9f800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b98072850 0x7f3b980c6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:16.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9f800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b980cb490 0x7f3b98154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:16.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9f800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b9815bd90 0x7f3b980b4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:16.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9f800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b98155510 con 0x7f3b9815bd90 2024-02-19T15:27:16.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9f800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b981557f0 con 0x7f3b980cb490 2024-02-19T15:27:16.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9f800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b98155ad0 con 0x7f3b98072850 2024-02-19T15:27:16.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9d800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b9815bd90 0x7f3b980b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:16.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9d800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b9815bd90 0x7f3b980b4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56678/0 (socket says 172.21.15.47:56678) 2024-02-19T15:27:16.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9d800640 1 -- 172.21.15.47:0/1505276998 learned_addr learned my addr 172.21.15.47:0/1505276998 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:16.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9ce00640 1 --2- 172.21.15.47:0/1505276998 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b98072850 0x7f3b980c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:16.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b97e00640 1 --2- 172.21.15.47:0/1505276998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b980cb490 0x7f3b98154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:16.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9d800640 1 -- 172.21.15.47:0/1505276998 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b98072850 msgr2=0x7f3b980c6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:16.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9d800640 1 --2- 172.21.15.47:0/1505276998 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b98072850 0x7f3b980c6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9d800640 1 -- 172.21.15.47:0/1505276998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b980cb490 msgr2=0x7f3b98154ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:16.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9d800640 1 --2- 172.21.15.47:0/1505276998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b980cb490 0x7f3b98154ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9d800640 1 -- 172.21.15.47:0/1505276998 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b90009b30 con 0x7f3b9815bd90 2024-02-19T15:27:16.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9ce00640 1 --2- 172.21.15.47:0/1505276998 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b98072850 0x7f3b980c6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:27:16.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9d800640 1 --2- 172.21.15.47:0/1505276998 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b9815bd90 0x7f3b980b4810 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f3b9001a4c0 tx=0x7f3b90019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:16.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b94c00640 1 -- 172.21.15.47:0/1505276998 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b900b8670 con 0x7f3b9815bd90 2024-02-19T15:27:16.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b94c00640 1 -- 172.21.15.47:0/1505276998 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b900b87f0 con 0x7f3b9815bd90 2024-02-19T15:27:16.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9f800640 1 -- 172.21.15.47:0/1505276998 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b980b42f0 con 0x7f3b9815bd90 2024-02-19T15:27:16.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b94c00640 1 -- 172.21.15.47:0/1505276998 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b900b8ba0 con 0x7f3b9815bd90 2024-02-19T15:27:16.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.163+0000 7f3b9f800640 1 -- 172.21.15.47:0/1505276998 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b98152c90 con 0x7f3b9815bd90 2024-02-19T15:27:16.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.167+0000 7f3b9f800640 1 -- 172.21.15.47:0/1505276998 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b980b44e0 con 0x7f3b9815bd90 2024-02-19T15:27:16.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.167+0000 7f3b94c00640 1 -- 172.21.15.47:0/1505276998 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3b900b8d00 con 0x7f3b9815bd90 2024-02-19T15:27:16.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.167+0000 7f3b94c00640 1 --2- 172.21.15.47:0/1505276998 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3b78066e40 0x7f3b78069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:16.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.167+0000 7f3b94c00640 1 -- 172.21.15.47:0/1505276998 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f3b9011fd40 con 0x7f3b9815bd90 2024-02-19T15:27:16.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.167+0000 7f3b9ce00640 1 --2- 172.21.15.47:0/1505276998 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3b78066e40 0x7f3b78069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:16.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.167+0000 7f3b9ce00640 1 --2- 172.21.15.47:0/1505276998 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3b78066e40 0x7f3b78069300 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c0098b0 tx=0x7f3b8c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:16.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.171+0000 7f3b94c00640 1 -- 172.21.15.47:0/1505276998 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f3b900ece00 con 0x7f3b9815bd90 2024-02-19T15:27:16.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.335+0000 7f3b9f800640 1 -- 172.21.15.47:0/1505276998 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3b98075360 con 0x7f3b9815bd90 2024-02-19T15:27:16.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.339+0000 7f3b94c00640 1 -- 172.21.15.47:0/1505276998 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3b900bc0c0 con 0x7f3b9815bd90 2024-02-19T15:27:16.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.339+0000 7f3b9f800640 1 -- 172.21.15.47:0/1505276998 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3b78066e40 msgr2=0x7f3b78069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:16.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.339+0000 7f3b9f800640 1 --2- 172.21.15.47:0/1505276998 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3b78066e40 0x7f3b78069300 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c0098b0 tx=0x7f3b8c008040 comp rx=0 tx=0).stop 2024-02-19T15:27:16.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.339+0000 7f3b9f800640 1 -- 172.21.15.47:0/1505276998 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b9815bd90 msgr2=0x7f3b980b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:16.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.339+0000 7f3b9f800640 1 --2- 172.21.15.47:0/1505276998 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b9815bd90 0x7f3b980b4810 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f3b9001a4c0 tx=0x7f3b90019630 comp rx=0 tx=0).stop 2024-02-19T15:27:16.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.339+0000 7f3b9f800640 1 -- 172.21.15.47:0/1505276998 shutdown_connections 2024-02-19T15:27:16.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.339+0000 7f3b9f800640 1 --2- 172.21.15.47:0/1505276998 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3b78066e40 0x7f3b78069300 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.339+0000 7f3b9f800640 1 --2- 172.21.15.47:0/1505276998 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b9815bd90 0x7f3b980b4810 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.339+0000 7f3b9f800640 1 --2- 172.21.15.47:0/1505276998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b980cb490 0x7f3b98154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.339+0000 7f3b9f800640 1 --2- 172.21.15.47:0/1505276998 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b98072850 0x7f3b980c6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.339+0000 7f3b9f800640 1 -- 172.21.15.47:0/1505276998 >> 172.21.15.47:0/1505276998 conn(0x7f3b980cfc20 msgr2=0x7f3b980d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:16.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.339+0000 7f3b9f800640 1 -- 172.21.15.47:0/1505276998 shutdown_connections 2024-02-19T15:27:16.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.339+0000 7f3b9f800640 1 -- 172.21.15.47:0/1505276998 wait complete. 2024-02-19T15:27:16.365 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T15:27:16.365 DEBUG:teuthology.orchestra.run.smithi047:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:27:16.440 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:27:16.528 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T15:27:16.529 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:27:16.529 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:27:16.537 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:27:16.537 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:27:16.538 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:27:16.547 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:27:16.547 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:27:16.547 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:27:16.589 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:27:16.590 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:27:16.590 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:27:16.599 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:27:16.599 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-02-19T15:27:16.599 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:27:16.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.771+0000 7f950fc00640 1 Processor -- start 2024-02-19T15:27:16.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.771+0000 7f950fc00640 1 -- start start 2024-02-19T15:27:16.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.771+0000 7f950fc00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f950815e060 0x7f950806b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:16.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.771+0000 7f950fc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95080729a0 0x7f95080cc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:16.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.771+0000 7f950fc00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f95080aff90 0x7f95080ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:16.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.771+0000 7f950fc00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f950815e440 con 0x7f95080aff90 2024-02-19T15:27:16.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.771+0000 7f950fc00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9508071990 con 0x7f950815e060 2024-02-19T15:27:16.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.771+0000 7f950fc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9508072510 con 0x7f95080729a0 2024-02-19T15:27:16.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.771+0000 7f950dc00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f95080aff90 0x7f95080ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:16.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.771+0000 7f950dc00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f95080aff90 0x7f95080ccbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56694/0 (socket says 172.21.15.47:56694) 2024-02-19T15:27:16.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.771+0000 7f950dc00640 1 -- 172.21.15.47:0/2784375790 learned_addr learned my addr 172.21.15.47:0/2784375790 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:16.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.771+0000 7f950d200640 1 --2- 172.21.15.47:0/2784375790 >> v2:172.21.15.116:3300/0 conn(0x7f950815e060 0x7f950806b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:16.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.771+0000 7f94ffe00640 1 --2- 172.21.15.47:0/2784375790 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95080729a0 0x7f95080cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:16.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.771+0000 7f950dc00640 1 -- 172.21.15.47:0/2784375790 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95080729a0 msgr2=0x7f95080cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:16.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.771+0000 7f950dc00640 1 --2- 172.21.15.47:0/2784375790 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95080729a0 0x7f95080cc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.771+0000 7f950dc00640 1 -- 172.21.15.47:0/2784375790 >> v2:172.21.15.116:3300/0 conn(0x7f950815e060 msgr2=0x7f950806b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:16.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.771+0000 7f950dc00640 1 --2- 172.21.15.47:0/2784375790 >> v2:172.21.15.116:3300/0 conn(0x7f950815e060 0x7f950806b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.771+0000 7f950dc00640 1 -- 172.21.15.47:0/2784375790 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95080cd100 con 0x7f95080aff90 2024-02-19T15:27:16.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.775+0000 7f950dc00640 1 --2- 172.21.15.47:0/2784375790 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95080aff90 0x7f95080ccbc0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f9508075070 tx=0x7f950001c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5086757414faaf6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:16.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.775+0000 7f9507e00640 1 -- 172.21.15.47:0/2784375790 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95000a5840 con 0x7f95080aff90 2024-02-19T15:27:16.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.775+0000 7f9507e00640 1 -- 172.21.15.47:0/2784375790 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95000a59a0 con 0x7f95080aff90 2024-02-19T15:27:16.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.775+0000 7f9507e00640 1 -- 172.21.15.47:0/2784375790 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95000a5d30 con 0x7f95080aff90 2024-02-19T15:27:16.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.775+0000 7f950fc00640 1 -- 172.21.15.47:0/2784375790 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95080aff90 msgr2=0x7f95080ccbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:16.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.775+0000 7f950fc00640 1 --2- 172.21.15.47:0/2784375790 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95080aff90 0x7f95080ccbc0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f9508075070 tx=0x7f950001c9c0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.775+0000 7f950fc00640 1 -- 172.21.15.47:0/2784375790 shutdown_connections 2024-02-19T15:27:16.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.775+0000 7f950fc00640 1 --2- 172.21.15.47:0/2784375790 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95080aff90 0x7f95080ccbc0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.775+0000 7f950fc00640 1 --2- 172.21.15.47:0/2784375790 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95080729a0 0x7f95080cc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.775+0000 7f950fc00640 1 --2- 172.21.15.47:0/2784375790 >> v2:172.21.15.116:3300/0 conn(0x7f950815e060 0x7f950806b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.775+0000 7f950fc00640 1 -- 172.21.15.47:0/2784375790 >> 172.21.15.47:0/2784375790 conn(0x7f95080d0b10 msgr2=0x7f95080d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:16.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.775+0000 7f950fc00640 1 -- 172.21.15.47:0/2784375790 shutdown_connections 2024-02-19T15:27:16.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.775+0000 7f950fc00640 1 -- 172.21.15.47:0/2784375790 wait complete. 2024-02-19T15:27:16.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.775+0000 7f950fc00640 1 Processor -- start 2024-02-19T15:27:16.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.775+0000 7f950fc00640 1 -- start start 2024-02-19T15:27:16.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f950fc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95080729a0 0x7f95080af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:16.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f950fc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95080aff90 0x7f95080c2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:16.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f950fc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f950815e060 0x7f95080c28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:16.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f950fc00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95080afa60 con 0x7f95080aff90 2024-02-19T15:27:16.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f950fc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950809ede0 con 0x7f950815e060 2024-02-19T15:27:16.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f950fc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950809f0c0 con 0x7f95080729a0 2024-02-19T15:27:16.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f94ffe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95080aff90 0x7f95080c2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:16.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f94ffe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95080aff90 0x7f95080c2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56700/0 (socket says 172.21.15.47:56700) 2024-02-19T15:27:16.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f94ffe00640 1 -- 172.21.15.47:0/4016132872 learned_addr learned my addr 172.21.15.47:0/4016132872 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:16.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f950d200640 1 --2- 172.21.15.47:0/4016132872 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95080729a0 0x7f95080af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:16.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f950dc00640 1 --2- 172.21.15.47:0/4016132872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f950815e060 0x7f95080c28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:16.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f94ffe00640 1 -- 172.21.15.47:0/4016132872 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95080729a0 msgr2=0x7f95080af520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:16.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f94ffe00640 1 --2- 172.21.15.47:0/4016132872 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95080729a0 0x7f95080af520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f94ffe00640 1 -- 172.21.15.47:0/4016132872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f950815e060 msgr2=0x7f95080c28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:16.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f94ffe00640 1 --2- 172.21.15.47:0/4016132872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f950815e060 0x7f95080c28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f94ffe00640 1 -- 172.21.15.47:0/4016132872 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9500009b30 con 0x7f95080aff90 2024-02-19T15:27:16.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f94ffe00640 1 --2- 172.21.15.47:0/4016132872 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95080aff90 0x7f95080c2370 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f94f000d800 tx=0x7f94f000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:16.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f9505600640 1 -- 172.21.15.47:0/4016132872 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f00096a0 con 0x7f95080aff90 2024-02-19T15:27:16.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f9505600640 1 -- 172.21.15.47:0/4016132872 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f0009800 con 0x7f95080aff90 2024-02-19T15:27:16.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f9505600640 1 -- 172.21.15.47:0/4016132872 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f0009b70 con 0x7f95080aff90 2024-02-19T15:27:16.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f950fc00640 1 -- 172.21.15.47:0/4016132872 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950809f3a0 con 0x7f95080aff90 2024-02-19T15:27:16.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f950fc00640 1 -- 172.21.15.47:0/4016132872 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950809f960 con 0x7f95080aff90 2024-02-19T15:27:16.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.779+0000 7f950fc00640 1 -- 172.21.15.47:0/4016132872 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95080b23e0 con 0x7f95080aff90 2024-02-19T15:27:16.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.783+0000 7f9505600640 1 -- 172.21.15.47:0/4016132872 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f94f0009cd0 con 0x7f95080aff90 2024-02-19T15:27:16.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.783+0000 7f9505600640 1 --2- 172.21.15.47:0/4016132872 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f94e4066e40 0x7f94e4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:16.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.783+0000 7f9505600640 1 -- 172.21.15.47:0/4016132872 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f94f0084ff0 con 0x7f95080aff90 2024-02-19T15:27:16.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.783+0000 7f950d200640 1 --2- 172.21.15.47:0/4016132872 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f94e4066e40 0x7f94e4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:16.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.783+0000 7f950d200640 1 --2- 172.21.15.47:0/4016132872 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f94e4066e40 0x7f94e4069300 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f94f8009a50 tx=0x7f94f8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:16.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.787+0000 7f9505600640 1 -- 172.21.15.47:0/4016132872 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f94f00b71c0 con 0x7f95080aff90 2024-02-19T15:27:16.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.983+0000 7f950fc00640 1 -- 172.21.15.47:0/4016132872 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95080c4380 con 0x7f95080aff90 2024-02-19T15:27:16.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.983+0000 7f9505600640 1 -- 172.21.15.47:0/4016132872 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f94f0051fb0 con 0x7f95080aff90 2024-02-19T15:27:16.990 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:16.990 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":68,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7060,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/4006350936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":4006350936},{"type":"v1","addr":"172.21.15.116:6837","nonce":4006350936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/175660992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":175660992},{"type":"v1","addr":"172.21.15.116:6835","nonce":175660992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/2191173523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2191173523},{"type":"v1","addr":"172.21.15.47:6834","nonce":2191173523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/1012301665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":1012301665},{"type":"v1","addr":"172.21.15.47:6835","nonce":1012301665}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[]} 2024-02-19T15:27:16.990 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 68 2024-02-19T15:27:16.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.987+0000 7f950fc00640 1 -- 172.21.15.47:0/4016132872 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f94e4066e40 msgr2=0x7f94e4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:16.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.987+0000 7f950fc00640 1 --2- 172.21.15.47:0/4016132872 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f94e4066e40 0x7f94e4069300 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f94f8009a50 tx=0x7f94f8008040 comp rx=0 tx=0).stop 2024-02-19T15:27:16.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.987+0000 7f950fc00640 1 -- 172.21.15.47:0/4016132872 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95080aff90 msgr2=0x7f95080c2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:16.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.987+0000 7f950fc00640 1 --2- 172.21.15.47:0/4016132872 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95080aff90 0x7f95080c2370 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f94f000d800 tx=0x7f94f000dcc0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.987+0000 7f950fc00640 1 -- 172.21.15.47:0/4016132872 shutdown_connections 2024-02-19T15:27:16.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.987+0000 7f950fc00640 1 --2- 172.21.15.47:0/4016132872 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f94e4066e40 0x7f94e4069300 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.987+0000 7f950fc00640 1 --2- 172.21.15.47:0/4016132872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f950815e060 0x7f95080c28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.987+0000 7f950fc00640 1 --2- 172.21.15.47:0/4016132872 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95080aff90 0x7f95080c2370 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.987+0000 7f950fc00640 1 --2- 172.21.15.47:0/4016132872 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95080729a0 0x7f95080af520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:16.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.987+0000 7f950fc00640 1 -- 172.21.15.47:0/4016132872 >> 172.21.15.47:0/4016132872 conn(0x7f95080d0b10 msgr2=0x7f95080b1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:16.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.987+0000 7f950fc00640 1 -- 172.21.15.47:0/4016132872 shutdown_connections 2024-02-19T15:27:16.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:16.987+0000 7f950fc00640 1 -- 172.21.15.47:0/4016132872 wait complete. 2024-02-19T15:27:17.012 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T15:27:17.012 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-19T15:27:17.012 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T15:27:17.013 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:27:17.013 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T15:27:17.013 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-19T15:27:17.013 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T15:27:17.014 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:27:17.014 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T15:27:17.014 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-19T15:27:17.014 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T15:27:17.014 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:27:17.014 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T15:27:17.014 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-19T15:27:17.014 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T15:27:17.015 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:27:17.025 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T15:27:17.026 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T15:27:17.026 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T15:27:17.026 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T15:27:17.028 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T15:27:17.028 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T15:27:17.030 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T15:27:17.030 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T15:27:17.072 INFO:tasks.ceph.mds.c:Started 2024-02-19T15:27:17.073 DEBUG:teuthology.parallel:result is None 2024-02-19T15:27:17.074 INFO:tasks.ceph.mds.a:Started 2024-02-19T15:27:17.074 DEBUG:teuthology.parallel:result is None 2024-02-19T15:27:17.079 INFO:tasks.ceph.mds.b:Started 2024-02-19T15:27:17.080 DEBUG:teuthology.parallel:result is None 2024-02-19T15:27:17.082 INFO:tasks.ceph.mds.d:Started 2024-02-19T15:27:17.082 DEBUG:teuthology.parallel:result is None 2024-02-19T15:27:17.082 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-19T15:27:17.159 INFO:tasks.ceph.mds.a.smithi047.stdout:starting mds.a at 2024-02-19T15:27:17.159 INFO:tasks.ceph.mds.c.smithi047.stdout:starting mds.c at 2024-02-19T15:27:17.171 INFO:tasks.ceph.mds.b.smithi116.stdout:starting mds.b at 2024-02-19T15:27:17.171 INFO:tasks.ceph.mds.d.smithi116.stdout:starting mds.d at 2024-02-19T15:27:17.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T15:27:17.168+0000 7f9d4e400640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-02-19T15:27:17.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.223+0000 7f5bc8a00640 1 Processor -- start 2024-02-19T15:27:17.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.227+0000 7f5bc8a00640 1 -- start start 2024-02-19T15:27:17.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.227+0000 7f5bc8a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc4145590 0x7f5bc40727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:17.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.227+0000 7f5bc8a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5bc415c390 0x7f5bc415c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:17.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.227+0000 7f5bc8a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc415d280 con 0x7f5bc40cea80 2024-02-19T15:27:17.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.227+0000 7f5bc8a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bc4070ff0 con 0x7f5bc415c390 2024-02-19T15:27:17.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.227+0000 7f5bc8a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc415cd60 con 0x7f5bc4145590 2024-02-19T15:27:17.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.227+0000 7f5bc2800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5bc415c390 0x7f5bc415c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:17.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.227+0000 7f5bc2800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5bc415c390 0x7f5bc415c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56746/0 (socket says 172.21.15.47:56746) 2024-02-19T15:27:17.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.227+0000 7f5bc2800640 1 -- 172.21.15.47:0/1271964546 learned_addr learned my addr 172.21.15.47:0/1271964546 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:17.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.227+0000 7f5bc1e00640 1 --2- 172.21.15.47:0/1271964546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc4145590 0x7f5bc40727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:17.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.227+0000 7f5bc2800640 1 -- 172.21.15.47:0/1271964546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc4145590 msgr2=0x7f5bc40727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:17.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.227+0000 7f5bc2800640 1 --2- 172.21.15.47:0/1271964546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc4145590 0x7f5bc40727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:17.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.227+0000 7f5bc2800640 1 -- 172.21.15.47:0/1271964546 >> v1:172.21.15.116:6789/0 conn(0x7f5bc40cea80 legacy=0x7f5bc406c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:17.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.227+0000 7f5bc2800640 1 -- 172.21.15.47:0/1271964546 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc40709e0 con 0x7f5bc415c390 2024-02-19T15:27:17.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.231+0000 7f5bc2800640 1 --2- 172.21.15.47:0/1271964546 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5bc415c390 0x7f5bc415c820 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc009870 tx=0x7f5bbc01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a40dbf36f4d5205e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:17.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.231+0000 7f5bc1400640 1 -- 172.21.15.47:0/1271964546 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc0ab670 con 0x7f5bc415c390 2024-02-19T15:27:17.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.231+0000 7f5bc1400640 1 -- 172.21.15.47:0/1271964546 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bbc0ab7d0 con 0x7f5bc415c390 2024-02-19T15:27:17.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.231+0000 7f5bc1400640 1 -- 172.21.15.47:0/1271964546 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc0abb60 con 0x7f5bc415c390 2024-02-19T15:27:17.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.231+0000 7f5bc8a00640 1 -- 172.21.15.47:0/1271964546 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5bc415c390 msgr2=0x7f5bc415c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:17.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.231+0000 7f5bc8a00640 1 --2- 172.21.15.47:0/1271964546 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5bc415c390 0x7f5bc415c820 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc009870 tx=0x7f5bbc01c560 comp rx=0 tx=0).stop 2024-02-19T15:27:17.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.231+0000 7f5bc8a00640 1 -- 172.21.15.47:0/1271964546 shutdown_connections 2024-02-19T15:27:17.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.231+0000 7f5bc8a00640 1 --2- 172.21.15.47:0/1271964546 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5bc415c390 0x7f5bc415c820 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:17.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.231+0000 7f5bc8a00640 1 --2- 172.21.15.47:0/1271964546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc4145590 0x7f5bc40727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:17.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.231+0000 7f5bc8a00640 1 -- 172.21.15.47:0/1271964546 >> 172.21.15.47:0/1271964546 conn(0x7f5bc40b16a0 msgr2=0x7f5bc40b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:17.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.231+0000 7f5bc8a00640 1 -- 172.21.15.47:0/1271964546 shutdown_connections 2024-02-19T15:27:17.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.231+0000 7f5bc8a00640 1 -- 172.21.15.47:0/1271964546 wait complete. 2024-02-19T15:27:17.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.231+0000 7f5bc8a00640 1 Processor -- start 2024-02-19T15:27:17.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.231+0000 7f5bc8a00640 1 -- start start 2024-02-19T15:27:17.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bc8a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bc40cea80 0x7f5bc406d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:17.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bc8a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5bc4145590 0x7f5bc41376f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:17.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bc8a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc415c390 0x7f5bc4137c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:17.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bc8a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc413da60 con 0x7f5bc4145590 2024-02-19T15:27:17.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bc8a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc413dcd0 con 0x7f5bc40cea80 2024-02-19T15:27:17.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bc8a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc413dfb0 con 0x7f5bc415c390 2024-02-19T15:27:17.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bb9400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5bc4145590 0x7f5bc41376f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:17.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bb9400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5bc4145590 0x7f5bc41376f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56754/0 (socket says 172.21.15.47:56754) 2024-02-19T15:27:17.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bb9400640 1 -- 172.21.15.47:0/2768816433 learned_addr learned my addr 172.21.15.47:0/2768816433 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:17.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bc2800640 1 --2- 172.21.15.47:0/2768816433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc415c390 0x7f5bc4137c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:17.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bc1e00640 1 --2- 172.21.15.47:0/2768816433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bc40cea80 0x7f5bc406d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:17.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bb9400640 1 -- 172.21.15.47:0/2768816433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc415c390 msgr2=0x7f5bc4137c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:17.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bb9400640 1 --2- 172.21.15.47:0/2768816433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc415c390 0x7f5bc4137c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:17.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bb9400640 1 -- 172.21.15.47:0/2768816433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bc40cea80 msgr2=0x7f5bc406d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:17.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bb9400640 1 --2- 172.21.15.47:0/2768816433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bc40cea80 0x7f5bc406d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:17.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bb9400640 1 -- 172.21.15.47:0/2768816433 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bbc0094d0 con 0x7f5bc4145590 2024-02-19T15:27:17.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bb9400640 1 --2- 172.21.15.47:0/2768816433 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5bc4145590 0x7f5bc41376f0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00c420 tx=0x7f5bac00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:17.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bbaa00640 1 -- 172.21.15.47:0/2768816433 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac0052d0 con 0x7f5bc4145590 2024-02-19T15:27:17.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bbaa00640 1 -- 172.21.15.47:0/2768816433 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bac005430 con 0x7f5bc4145590 2024-02-19T15:27:17.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bbaa00640 1 -- 172.21.15.47:0/2768816433 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac007260 con 0x7f5bc4145590 2024-02-19T15:27:17.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bc8a00640 1 -- 172.21.15.47:0/2768816433 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc413e290 con 0x7f5bc4145590 2024-02-19T15:27:17.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bc8a00640 1 -- 172.21.15.47:0/2768816433 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc413e850 con 0x7f5bc4145590 2024-02-19T15:27:17.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bc8a00640 1 -- 172.21.15.47:0/2768816433 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc4075360 con 0x7f5bc4145590 2024-02-19T15:27:17.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.235+0000 7f5bbaa00640 1 -- 172.21.15.47:0/2768816433 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5bac0073c0 con 0x7f5bc4145590 2024-02-19T15:27:17.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.239+0000 7f5bbaa00640 1 --2- 172.21.15.47:0/2768816433 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5ba0066e70 0x7f5ba0069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:17.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.239+0000 7f5bbaa00640 1 -- 172.21.15.47:0/2768816433 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f5bac083b50 con 0x7f5bc4145590 2024-02-19T15:27:17.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.239+0000 7f5bc1e00640 1 --2- 172.21.15.47:0/2768816433 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5ba0066e70 0x7f5ba0069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:17.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.239+0000 7f5bc1e00640 1 --2- 172.21.15.47:0/2768816433 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5ba0066e70 0x7f5ba0069330 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f5bc406e5b0 tx=0x7f5bb40098d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:17.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.243+0000 7f5bbaa00640 1 -- 172.21.15.47:0/2768816433 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f5bac050c30 con 0x7f5bc4145590 2024-02-19T15:27:17.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.415+0000 7f5bc8a00640 1 -- 172.21.15.47:0/2768816433 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5bc4166760 con 0x7f5bc4145590 2024-02-19T15:27:18.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.995+0000 7f5bbaa00640 1 -- 172.21.15.47:0/2768816433 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v102) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5bac0542f0 con 0x7f5bc4145590 2024-02-19T15:27:18.000 INFO:teuthology.orchestra.run.smithi047.stderr: removed all blocklist entries 2024-02-19T15:27:18.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.999+0000 7f5bc8a00640 1 -- 172.21.15.47:0/2768816433 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5ba0066e70 msgr2=0x7f5ba0069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:18.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.999+0000 7f5bc8a00640 1 --2- 172.21.15.47:0/2768816433 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5ba0066e70 0x7f5ba0069330 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f5bc406e5b0 tx=0x7f5bb40098d0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.999+0000 7f5bc8a00640 1 -- 172.21.15.47:0/2768816433 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5bc4145590 msgr2=0x7f5bc41376f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:18.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.999+0000 7f5bc8a00640 1 --2- 172.21.15.47:0/2768816433 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5bc4145590 0x7f5bc41376f0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00c420 tx=0x7f5bac00c8e0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.999+0000 7f5bc8a00640 1 -- 172.21.15.47:0/2768816433 shutdown_connections 2024-02-19T15:27:18.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.999+0000 7f5bc8a00640 1 --2- 172.21.15.47:0/2768816433 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5ba0066e70 0x7f5ba0069330 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.999+0000 7f5bc8a00640 1 --2- 172.21.15.47:0/2768816433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc415c390 0x7f5bc4137c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.999+0000 7f5bc8a00640 1 --2- 172.21.15.47:0/2768816433 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5bc4145590 0x7f5bc41376f0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.999+0000 7f5bc8a00640 1 --2- 172.21.15.47:0/2768816433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bc40cea80 0x7f5bc406d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.999+0000 7f5bc8a00640 1 -- 172.21.15.47:0/2768816433 >> 172.21.15.47:0/2768816433 conn(0x7f5bc40b16a0 msgr2=0x7f5bc40afa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:18.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.999+0000 7f5bc8a00640 1 -- 172.21.15.47:0/2768816433 shutdown_connections 2024-02-19T15:27:18.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:17.999+0000 7f5bc8a00640 1 -- 172.21.15.47:0/2768816433 wait complete. 2024-02-19T15:27:18.030 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-19T15:27:18.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.171+0000 7f397c000640 1 Processor -- start 2024-02-19T15:27:18.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.171+0000 7f397c000640 1 -- start start 2024-02-19T15:27:18.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.171+0000 7f397c000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f39740b29a0 0x7f39740721b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:18.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.171+0000 7f397c000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39740b2e30 0x7f39740726f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:18.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.171+0000 7f397c000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3974070e10 con 0x7f39741585c0 2024-02-19T15:27:18.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.171+0000 7f397c000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39741589a0 con 0x7f39740b29a0 2024-02-19T15:27:18.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.171+0000 7f397c000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3974070ae0 con 0x7f39740b2e30 2024-02-19T15:27:18.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.171+0000 7f3978c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f39740b29a0 0x7f39740721b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:18.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.171+0000 7f3979600640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f39741585c0 0x7f397406ce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49034/0 (socket says 172.21.15.47:49034) 2024-02-19T15:27:18.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.171+0000 7f3979600640 1 -- 172.21.15.47:0/1411953968 learned_addr learned my addr 172.21.15.47:0/1411953968 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:18.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.171+0000 7f397a000640 1 --2- 172.21.15.47:0/1411953968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39740b2e30 0x7f39740726f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:18.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.171+0000 7f3978c00640 1 -- 172.21.15.47:0/1411953968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39740b2e30 msgr2=0x7f39740726f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:18.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.171+0000 7f3978c00640 1 --2- 172.21.15.47:0/1411953968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39740b2e30 0x7f39740726f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.171+0000 7f3978c00640 1 -- 172.21.15.47:0/1411953968 >> v1:172.21.15.47:6789/0 conn(0x7f39741585c0 legacy=0x7f397406ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:18.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.171+0000 7f3978c00640 1 -- 172.21.15.47:0/1411953968 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39740b3340 con 0x7f39740b29a0 2024-02-19T15:27:18.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.171+0000 7f397a000640 1 --2- 172.21.15.47:0/1411953968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39740b2e30 0x7f39740726f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:27:18.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.175+0000 7f3978c00640 1 --2- 172.21.15.47:0/1411953968 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39740b29a0 0x7f39740721b0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f39600097c0 tx=0x7f396001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9961d95c6885931d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:18.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.175+0000 7f3973e00640 1 -- 172.21.15.47:0/1411953968 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39600ab600 con 0x7f39740b29a0 2024-02-19T15:27:18.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.175+0000 7f3973e00640 1 -- 172.21.15.47:0/1411953968 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39600a7050 con 0x7f39740b29a0 2024-02-19T15:27:18.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.175+0000 7f3973e00640 1 -- 172.21.15.47:0/1411953968 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39600ab980 con 0x7f39740b29a0 2024-02-19T15:27:18.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.175+0000 7f397c000640 1 -- 172.21.15.47:0/1411953968 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39740b29a0 msgr2=0x7f39740721b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:18.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.175+0000 7f397c000640 1 --2- 172.21.15.47:0/1411953968 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39740b29a0 0x7f39740721b0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f39600097c0 tx=0x7f396001c3e0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.175+0000 7f397c000640 1 -- 172.21.15.47:0/1411953968 shutdown_connections 2024-02-19T15:27:18.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.175+0000 7f397c000640 1 --2- 172.21.15.47:0/1411953968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39740b2e30 0x7f39740726f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.175+0000 7f397c000640 1 --2- 172.21.15.47:0/1411953968 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39740b29a0 0x7f39740721b0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.175+0000 7f397c000640 1 -- 172.21.15.47:0/1411953968 >> 172.21.15.47:0/1411953968 conn(0x7f39740b16a0 msgr2=0x7f39740b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:18.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.175+0000 7f397c000640 1 -- 172.21.15.47:0/1411953968 shutdown_connections 2024-02-19T15:27:18.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.175+0000 7f397c000640 1 -- 172.21.15.47:0/1411953968 wait complete. 2024-02-19T15:27:18.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.175+0000 7f397c000640 1 Processor -- start 2024-02-19T15:27:18.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f397c000640 1 -- start start 2024-02-19T15:27:18.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f397c000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39740b29a0 0x7f397415d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:18.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f397c000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39740b2e30 0x7f3974157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:18.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f397c000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39741585c0 0x7f397415b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:18.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f397c000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397415c540 con 0x7f39740b2e30 2024-02-19T15:27:18.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f397c000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397415c820 con 0x7f39740b29a0 2024-02-19T15:27:18.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f397c000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f397415cb00 con 0x7f39741585c0 2024-02-19T15:27:18.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f3978c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39740b2e30 0x7f3974157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:18.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f3978c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39740b2e30 0x7f3974157a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56758/0 (socket says 172.21.15.47:56758) 2024-02-19T15:27:18.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f3978c00640 1 -- 172.21.15.47:0/723103117 learned_addr learned my addr 172.21.15.47:0/723103117 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:18.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f397a000640 1 --2- 172.21.15.47:0/723103117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39741585c0 0x7f397415b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:18.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f3979600640 1 --2- 172.21.15.47:0/723103117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39740b29a0 0x7f397415d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:18.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f3978c00640 1 -- 172.21.15.47:0/723103117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39741585c0 msgr2=0x7f397415b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:18.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f3978c00640 1 --2- 172.21.15.47:0/723103117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39741585c0 0x7f397415b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f3978c00640 1 -- 172.21.15.47:0/723103117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39740b29a0 msgr2=0x7f397415d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:18.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f3978c00640 1 --2- 172.21.15.47:0/723103117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39740b29a0 0x7f397415d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f3978c00640 1 -- 172.21.15.47:0/723103117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3960009420 con 0x7f39740b2e30 2024-02-19T15:27:18.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f3978c00640 1 --2- 172.21.15.47:0/723103117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39740b2e30 0x7f3974157a30 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f3960018ff0 tx=0x7f3960019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:18.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f3971600640 1 -- 172.21.15.47:0/723103117 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39600b4600 con 0x7f39740b2e30 2024-02-19T15:27:18.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f3971600640 1 -- 172.21.15.47:0/723103117 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39600b4760 con 0x7f39740b2e30 2024-02-19T15:27:18.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f397c000640 1 -- 172.21.15.47:0/723103117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397415cde0 con 0x7f39740b2e30 2024-02-19T15:27:18.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f3971600640 1 -- 172.21.15.47:0/723103117 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39600b4b10 con 0x7f39740b2e30 2024-02-19T15:27:18.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f397c000640 1 -- 172.21.15.47:0/723103117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f397414e630 con 0x7f39740b2e30 2024-02-19T15:27:18.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.179+0000 7f397c000640 1 -- 172.21.15.47:0/723103117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3974067bd0 con 0x7f39740b2e30 2024-02-19T15:27:18.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.183+0000 7f3971600640 1 -- 172.21.15.47:0/723103117 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f39600b2040 con 0x7f39740b2e30 2024-02-19T15:27:18.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.183+0000 7f3971600640 1 --2- 172.21.15.47:0/723103117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3954066e40 0x7f3954069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:18.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.183+0000 7f3971600640 1 -- 172.21.15.47:0/723103117 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f396011e340 con 0x7f39740b2e30 2024-02-19T15:27:18.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.183+0000 7f3979600640 1 --2- 172.21.15.47:0/723103117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3954066e40 0x7f3954069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:18.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.183+0000 7f3979600640 1 --2- 172.21.15.47:0/723103117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3954066e40 0x7f3954069300 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3968000f70 tx=0x7f3968007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:18.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.187+0000 7f3971600640 1 -- 172.21.15.47:0/723103117 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f39600eb490 con 0x7f39740b2e30 2024-02-19T15:27:18.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.403+0000 7f397c000640 1 -- 172.21.15.47:0/723103117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3974075360 con 0x7f39740b2e30 2024-02-19T15:27:18.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.403+0000 7f3971600640 1 -- 172.21.15.47:0/723103117 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f39600eeb50 con 0x7f39740b2e30 2024-02-19T15:27:18.409 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:18.409 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:27:18.410 INFO:teuthology.orchestra.run.smithi047.stdout: "auth_dump": [ 2024-02-19T15:27:18.410 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:18.410 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.a", 2024-02-19T15:27:18.410 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNl8BP+HRAAlJ3Jk+1k2XcZAnICzy++Ow==", 2024-02-19T15:27:18.410 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:27:18.410 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:27:18.411 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:27:18.411 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:27:18.411 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:27:18.411 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:18.411 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:27:18.411 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:18.412 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.b", 2024-02-19T15:27:18.412 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlIOh3KhAAlnU2ZFY4iv/XEw0WqxyZpw==", 2024-02-19T15:27:18.412 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:27:18.412 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:27:18.412 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:27:18.412 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:27:18.413 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:27:18.413 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:18.413 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:27:18.413 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:18.413 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.c", 2024-02-19T15:27:18.413 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNl48SHJRAAn4GCT5KnP+2PD8IXbGClCg==", 2024-02-19T15:27:18.414 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:27:18.414 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:27:18.414 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:27:18.414 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:27:18.414 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:27:18.414 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:18.415 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:27:18.415 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:18.415 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.d", 2024-02-19T15:27:18.415 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNljcekMRAA414655SzH43fBA0AtjpxZw==", 2024-02-19T15:27:18.415 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:27:18.415 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:27:18.416 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:27:18.416 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:27:18.416 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:27:18.416 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:18.416 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:27:18.416 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:18.417 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.0", 2024-02-19T15:27:18.417 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDlcdNl/bycFhAAmzYi/OxEPUW3tcRJC0DNEQ==", 2024-02-19T15:27:18.417 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:27:18.417 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:27:18.417 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:27:18.417 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:27:18.418 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:18.418 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:27:18.418 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:18.418 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.1", 2024-02-19T15:27:18.418 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDocdNluquhHBAA88aLHCvVol7JoZB0MtoutA==", 2024-02-19T15:27:18.418 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:27:18.419 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:27:18.419 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:27:18.419 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:27:18.419 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:18.419 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:27:18.420 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:18.420 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.2", 2024-02-19T15:27:18.420 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDrcdNlGAnWIhAAyELvCPKjtfgF0+kX6NQFIA==", 2024-02-19T15:27:18.420 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:27:18.420 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:27:18.421 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:27:18.421 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:27:18.421 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:18.421 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:27:18.422 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:18.422 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.3", 2024-02-19T15:27:18.422 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDucdNlROZ6KBAAxTCDuYsw5SRIZBcclzwxhg==", 2024-02-19T15:27:18.422 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:27:18.422 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:27:18.422 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:27:18.423 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:27:18.423 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:18.423 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:27:18.423 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:18.423 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.4", 2024-02-19T15:27:18.424 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD0cdNluOGILBAA66R9RVXD2vdyIw+oZ+iM1g==", 2024-02-19T15:27:18.424 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:27:18.424 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:27:18.424 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:27:18.424 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:27:18.424 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:18.425 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:27:18.425 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:18.425 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.5", 2024-02-19T15:27:18.425 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD3cdNlHg8dMhAA2LUzfpvvHR4i8e5RBCeGpQ==", 2024-02-19T15:27:18.425 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:27:18.425 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:27:18.426 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:27:18.426 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:27:18.426 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:18.426 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:27:18.426 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:18.426 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.6", 2024-02-19T15:27:18.427 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD6cdNlXD1EORAAhKGIt0wFeCyoE6QkkZ3caA==", 2024-02-19T15:27:18.427 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:27:18.427 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:27:18.427 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:27:18.427 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:27:18.427 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:18.428 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:27:18.428 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:18.428 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.7", 2024-02-19T15:27:18.428 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD+cdNlfW65AhAAVWj3ptC8uQHPh/UXcb/uDg==", 2024-02-19T15:27:18.428 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:27:18.428 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:27:18.429 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:27:18.429 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:27:18.429 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:18.429 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:27:18.429 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:18.429 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.0", 2024-02-19T15:27:18.430 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlb7eRNRAAk6ZLGfjhcdlR9GoXFCd9eA==", 2024-02-19T15:27:18.430 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:27:18.430 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:27:18.430 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow r", 2024-02-19T15:27:18.430 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-19T15:27:18.430 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:18.431 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:27:18.431 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:18.431 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.1", 2024-02-19T15:27:18.431 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlKBu+AxAAlLRU70SAs3ZJo5wFwRWa4w==", 2024-02-19T15:27:18.431 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:27:18.431 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:27:18.431 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:27:18.431 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:27:18.431 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:27:18.431 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:18.431 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:27:18.432 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:18.432 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.2", 2024-02-19T15:27:18.432 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlyDk9ABAAIAjKSd4qJO5oHhVnDc1Rgw==", 2024-02-19T15:27:18.432 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:27:18.432 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:27:18.432 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:27:18.432 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:27:18.432 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:27:18.432 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:18.432 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:27:18.433 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:18.433 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.3", 2024-02-19T15:27:18.433 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlUqK3CRAABmSIzoJvl0ymME6swrvnSQ==", 2024-02-19T15:27:18.433 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:27:18.433 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:27:18.433 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:27:18.433 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:27:18.433 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:27:18.433 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:18.433 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:27:18.433 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:18.433 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.admin", 2024-02-19T15:27:18.433 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNls3vPAhAA4x2yQTUCz4Dv9vqEWnSTyQ==", 2024-02-19T15:27:18.433 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:27:18.434 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:27:18.434 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:27:18.434 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:27:18.434 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:27:18.434 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:18.434 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:27:18.434 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:18.434 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mgr.x", 2024-02-19T15:27:18.434 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNliM78GRAABWBg4hMcKjV2pK5WVBjLMQ==", 2024-02-19T15:27:18.434 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:27:18.434 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:27:18.434 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile mgr", 2024-02-19T15:27:18.434 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:27:18.434 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:18.434 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:27:18.435 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:18.435 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mgr.y", 2024-02-19T15:27:18.435 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlwxs9FhAAWqP7YoKio/bbdx7SGiCJ9Q==", 2024-02-19T15:27:18.435 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:27:18.435 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:27:18.435 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile mgr", 2024-02-19T15:27:18.435 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:27:18.435 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:18.435 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:18.435 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-02-19T15:27:18.435 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:27:18.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.407+0000 7f397c000640 1 -- 172.21.15.47:0/723103117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3954066e40 msgr2=0x7f3954069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:18.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.407+0000 7f397c000640 1 --2- 172.21.15.47:0/723103117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3954066e40 0x7f3954069300 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3968000f70 tx=0x7f3968007e90 comp rx=0 tx=0).stop 2024-02-19T15:27:18.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.407+0000 7f397c000640 1 -- 172.21.15.47:0/723103117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39740b2e30 msgr2=0x7f3974157a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:18.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.407+0000 7f397c000640 1 --2- 172.21.15.47:0/723103117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39740b2e30 0x7f3974157a30 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f3960018ff0 tx=0x7f3960019180 comp rx=0 tx=0).stop 2024-02-19T15:27:18.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.407+0000 7f397c000640 1 -- 172.21.15.47:0/723103117 shutdown_connections 2024-02-19T15:27:18.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.407+0000 7f397c000640 1 --2- 172.21.15.47:0/723103117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3954066e40 0x7f3954069300 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.407+0000 7f397c000640 1 --2- 172.21.15.47:0/723103117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f39741585c0 0x7f397415b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.407+0000 7f397c000640 1 --2- 172.21.15.47:0/723103117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39740b2e30 0x7f3974157a30 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.407+0000 7f397c000640 1 --2- 172.21.15.47:0/723103117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39740b29a0 0x7f397415d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.407+0000 7f397c000640 1 -- 172.21.15.47:0/723103117 >> 172.21.15.47:0/723103117 conn(0x7f39740b16a0 msgr2=0x7f3974071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:18.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.407+0000 7f397c000640 1 -- 172.21.15.47:0/723103117 shutdown_connections 2024-02-19T15:27:18.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.407+0000 7f397c000640 1 -- 172.21.15.47:0/723103117 wait complete. 2024-02-19T15:27:18.438 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:27:18.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.575+0000 7fd496000640 1 Processor -- start 2024-02-19T15:27:18.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.575+0000 7fd496000640 1 -- start start 2024-02-19T15:27:18.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.575+0000 7fd496000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd4901614e0 0x7fd49015f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:18.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.575+0000 7fd496000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4900b2500 0x7fd49006ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:18.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.575+0000 7fd496000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49006b5d0 con 0x7fd49015eee0 2024-02-19T15:27:18.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.575+0000 7fd496000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4900b1c00 con 0x7fd4901614e0 2024-02-19T15:27:18.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.575+0000 7fd496000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4900b1640 con 0x7fd4900b2500 2024-02-19T15:27:18.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.575+0000 7fd48f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd4901614e0 0x7fd49015f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:18.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.575+0000 7fd48f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd4901614e0 0x7fd49015f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56768/0 (socket says 172.21.15.47:56768) 2024-02-19T15:27:18.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.575+0000 7fd48f400640 1 -- 172.21.15.47:0/2739547645 learned_addr learned my addr 172.21.15.47:0/2739547645 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:18.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.575+0000 7fd48fe00640 1 --2- 172.21.15.47:0/2739547645 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4900b2500 0x7fd49006ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:18.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.575+0000 7fd48f400640 1 -- 172.21.15.47:0/2739547645 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4900b2500 msgr2=0x7fd49006ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:18.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.575+0000 7fd48f400640 1 --2- 172.21.15.47:0/2739547645 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4900b2500 0x7fd49006ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.575+0000 7fd48f400640 1 -- 172.21.15.47:0/2739547645 >> v1:172.21.15.47:6789/0 conn(0x7fd49015eee0 legacy=0x7fd4900b1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:18.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.575+0000 7fd48f400640 1 -- 172.21.15.47:0/2739547645 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4900b1920 con 0x7fd4901614e0 2024-02-19T15:27:18.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.575+0000 7fd48ea00640 1 -- 172.21.15.47:0/2739547645 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730976078 0 0) 0x7fd49006b5d0 con 0x7fd49015eee0 2024-02-19T15:27:18.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.575+0000 7fd48fe00640 1 --2- 172.21.15.47:0/2739547645 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4900b2500 0x7fd49006ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:18.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.579+0000 7fd48f400640 1 --2- 172.21.15.47:0/2739547645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4901614e0 0x7fd49015f620 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fd480009870 tx=0x7fd48001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937b88a8d8a65ead server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:18.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.579+0000 7fd48ea00640 1 -- 172.21.15.47:0/2739547645 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4800a9640 con 0x7fd4901614e0 2024-02-19T15:27:18.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.579+0000 7fd48ea00640 1 -- 172.21.15.47:0/2739547645 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4800a97a0 con 0x7fd4901614e0 2024-02-19T15:27:18.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.579+0000 7fd48ea00640 1 -- 172.21.15.47:0/2739547645 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4800a9b30 con 0x7fd4901614e0 2024-02-19T15:27:18.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.579+0000 7fd496000640 1 -- 172.21.15.47:0/2739547645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4901614e0 msgr2=0x7fd49015f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:18.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.579+0000 7fd496000640 1 --2- 172.21.15.47:0/2739547645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4901614e0 0x7fd49015f620 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fd480009870 tx=0x7fd48001c560 comp rx=0 tx=0).stop 2024-02-19T15:27:18.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.579+0000 7fd496000640 1 -- 172.21.15.47:0/2739547645 shutdown_connections 2024-02-19T15:27:18.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.579+0000 7fd496000640 1 --2- 172.21.15.47:0/2739547645 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4900b2500 0x7fd49006ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.579+0000 7fd496000640 1 --2- 172.21.15.47:0/2739547645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4901614e0 0x7fd49015f620 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.579+0000 7fd496000640 1 -- 172.21.15.47:0/2739547645 >> 172.21.15.47:0/2739547645 conn(0x7fd4900b4fc0 msgr2=0x7fd4900b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:18.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.579+0000 7fd496000640 1 -- 172.21.15.47:0/2739547645 shutdown_connections 2024-02-19T15:27:18.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.579+0000 7fd496000640 1 -- 172.21.15.47:0/2739547645 wait complete. 2024-02-19T15:27:18.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd496000640 1 Processor -- start 2024-02-19T15:27:18.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd496000640 1 -- start start 2024-02-19T15:27:18.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd496000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4900b2500 0x7fd490161c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:18.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd496000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd49015eee0 0x7fd49015af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:18.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd496000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4901614e0 0x7fd49014c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:18.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd496000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49014e280 con 0x7fd49015eee0 2024-02-19T15:27:18.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd496000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49014d600 con 0x7fd4901614e0 2024-02-19T15:27:18.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd496000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd49014d8e0 con 0x7fd4900b2500 2024-02-19T15:27:18.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd486a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd49015eee0 0x7fd49015af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:18.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd486a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd49015eee0 0x7fd49015af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56778/0 (socket says 172.21.15.47:56778) 2024-02-19T15:27:18.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd486a00640 1 -- 172.21.15.47:0/3033999877 learned_addr learned my addr 172.21.15.47:0/3033999877 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:18.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd48f400640 1 --2- 172.21.15.47:0/3033999877 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4900b2500 0x7fd490161c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:18.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd48fe00640 1 --2- 172.21.15.47:0/3033999877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4901614e0 0x7fd49014c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:18.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd486a00640 1 -- 172.21.15.47:0/3033999877 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4900b2500 msgr2=0x7fd490161c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:18.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd486a00640 1 --2- 172.21.15.47:0/3033999877 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4900b2500 0x7fd490161c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd486a00640 1 -- 172.21.15.47:0/3033999877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4901614e0 msgr2=0x7fd49014c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:18.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd486a00640 1 --2- 172.21.15.47:0/3033999877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4901614e0 0x7fd49014c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd486a00640 1 -- 172.21.15.47:0/3033999877 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4800094d0 con 0x7fd49015eee0 2024-02-19T15:27:18.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd486a00640 1 --2- 172.21.15.47:0/3033999877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd49015eee0 0x7fd49015af60 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fd47800c520 tx=0x7fd47800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:18.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd487e00640 1 -- 172.21.15.47:0/3033999877 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd478003600 con 0x7fd49015eee0 2024-02-19T15:27:18.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd487e00640 1 -- 172.21.15.47:0/3033999877 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd478003780 con 0x7fd49015eee0 2024-02-19T15:27:18.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd487e00640 1 -- 172.21.15.47:0/3033999877 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd478003b30 con 0x7fd49015eee0 2024-02-19T15:27:18.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd496000640 1 -- 172.21.15.47:0/3033999877 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd49014dbc0 con 0x7fd49015eee0 2024-02-19T15:27:18.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd496000640 1 -- 172.21.15.47:0/3033999877 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd49014e6a0 con 0x7fd49015eee0 2024-02-19T15:27:18.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.583+0000 7fd496000640 1 -- 172.21.15.47:0/3033999877 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd49014e3c0 con 0x7fd49015eee0 2024-02-19T15:27:18.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.587+0000 7fd487e00640 1 -- 172.21.15.47:0/3033999877 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd478003c90 con 0x7fd49015eee0 2024-02-19T15:27:18.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.587+0000 7fd487e00640 1 --2- 172.21.15.47:0/3033999877 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd46c066e90 0x7fd46c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:18.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.587+0000 7fd487e00640 1 -- 172.21.15.47:0/3033999877 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd478083c70 con 0x7fd49015eee0 2024-02-19T15:27:18.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.587+0000 7fd48f400640 1 --2- 172.21.15.47:0/3033999877 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd46c066e90 0x7fd46c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:18.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.587+0000 7fd48f400640 1 --2- 172.21.15.47:0/3033999877 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd46c066e90 0x7fd46c069350 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd4800ae000 tx=0x7fd4800b3040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:18.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.591+0000 7fd487e00640 1 -- 172.21.15.47:0/3033999877 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd49014e3c0 con 0x7fd49015eee0 2024-02-19T15:27:18.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.755+0000 7fd496000640 1 -- 172.21.15.47:0/3033999877 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd49014e3c0 con 0x7fd49015eee0 2024-02-19T15:27:18.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.755+0000 7fd487e00640 1 -- 172.21.15.47:0/3033999877 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+12905 (secure 0 0 0) 0x7fd49014e3c0 con 0x7fd49015eee0 2024-02-19T15:27:18.759 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:18.759 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":104,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:27:18.008169+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:27:18.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.755+0000 7fd496000640 1 -- 172.21.15.47:0/3033999877 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd46c066e90 msgr2=0x7fd46c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:18.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.755+0000 7fd496000640 1 --2- 172.21.15.47:0/3033999877 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd46c066e90 0x7fd46c069350 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd4800ae000 tx=0x7fd4800b3040 comp rx=0 tx=0).stop 2024-02-19T15:27:18.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.755+0000 7fd496000640 1 -- 172.21.15.47:0/3033999877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd49015eee0 msgr2=0x7fd49015af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:18.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.755+0000 7fd496000640 1 --2- 172.21.15.47:0/3033999877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd49015eee0 0x7fd49015af60 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fd47800c520 tx=0x7fd47800c9e0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.755+0000 7fd496000640 1 -- 172.21.15.47:0/3033999877 shutdown_connections 2024-02-19T15:27:18.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.755+0000 7fd496000640 1 --2- 172.21.15.47:0/3033999877 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd46c066e90 0x7fd46c069350 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.755+0000 7fd496000640 1 --2- 172.21.15.47:0/3033999877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4901614e0 0x7fd49014c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.755+0000 7fd496000640 1 --2- 172.21.15.47:0/3033999877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd49015eee0 0x7fd49015af60 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.755+0000 7fd496000640 1 --2- 172.21.15.47:0/3033999877 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4900b2500 0x7fd490161c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.755+0000 7fd496000640 1 -- 172.21.15.47:0/3033999877 >> 172.21.15.47:0/3033999877 conn(0x7fd4900b4fc0 msgr2=0x7fd4900b1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:18.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.755+0000 7fd496000640 1 -- 172.21.15.47:0/3033999877 shutdown_connections 2024-02-19T15:27:18.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.755+0000 7fd496000640 1 -- 172.21.15.47:0/3033999877 wait complete. 2024-02-19T15:27:18.780 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:27:18.781 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:27:18.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.919+0000 7f298d000640 1 Processor -- start 2024-02-19T15:27:18.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.919+0000 7f298d000640 1 -- start start 2024-02-19T15:27:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.919+0000 7f298d000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f298815ded0 0x7f2988066cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.919+0000 7f298d000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f29880b1280 0x7f298806aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.919+0000 7f298d000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f298815e2b0 0x7f298806b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.919+0000 7f298d000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2988067210 con 0x7f298815ded0 2024-02-19T15:27:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.919+0000 7f298d000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29880b04c0 con 0x7f298815e2b0 2024-02-19T15:27:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.919+0000 7f298d000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29880b07a0 con 0x7f29880b1280 2024-02-19T15:27:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.919+0000 7f2985e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f298815ded0 0x7f2988066cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.919+0000 7f2985e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f298815ded0 0x7f2988066cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56792/0 (socket says 172.21.15.47:56792) 2024-02-19T15:27:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.919+0000 7f2985e00640 1 -- 172.21.15.47:0/2507509577 learned_addr learned my addr 172.21.15.47:0/2507509577 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.919+0000 7f297fe00640 1 --2- 172.21.15.47:0/2507509577 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f29880b1280 0x7f298806aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.919+0000 7f2986800640 1 --2- 172.21.15.47:0/2507509577 >> v2:172.21.15.116:3300/0 conn(0x7f298815e2b0 0x7f298806b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:18.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.923+0000 7f2985e00640 1 -- 172.21.15.47:0/2507509577 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f29880b1280 msgr2=0x7f298806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:18.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.923+0000 7f2985e00640 1 --2- 172.21.15.47:0/2507509577 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f29880b1280 0x7f298806aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.923+0000 7f2985e00640 1 -- 172.21.15.47:0/2507509577 >> v2:172.21.15.116:3300/0 conn(0x7f298815e2b0 msgr2=0x7f298806b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:18.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.923+0000 7f2985e00640 1 --2- 172.21.15.47:0/2507509577 >> v2:172.21.15.116:3300/0 conn(0x7f298815e2b0 0x7f298806b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.923+0000 7f2985e00640 1 -- 172.21.15.47:0/2507509577 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29880b0a80 con 0x7f298815ded0 2024-02-19T15:27:18.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.923+0000 7f2985e00640 1 --2- 172.21.15.47:0/2507509577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f298815ded0 0x7f2988066cd0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f2978009870 tx=0x7f297801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196214fb87171ee3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:18.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.923+0000 7f2985400640 1 -- 172.21.15.47:0/2507509577 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29780aa430 con 0x7f298815ded0 2024-02-19T15:27:18.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.923+0000 7f2985400640 1 -- 172.21.15.47:0/2507509577 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29780aa590 con 0x7f298815ded0 2024-02-19T15:27:18.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.923+0000 7f2985400640 1 -- 172.21.15.47:0/2507509577 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29780aa920 con 0x7f298815ded0 2024-02-19T15:27:18.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.923+0000 7f298d000640 1 -- 172.21.15.47:0/2507509577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f298815ded0 msgr2=0x7f2988066cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:18.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.923+0000 7f298d000640 1 --2- 172.21.15.47:0/2507509577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f298815ded0 0x7f2988066cd0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f2978009870 tx=0x7f297801c4f0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.923+0000 7f298d000640 1 -- 172.21.15.47:0/2507509577 shutdown_connections 2024-02-19T15:27:18.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.923+0000 7f298d000640 1 --2- 172.21.15.47:0/2507509577 >> v2:172.21.15.116:3300/0 conn(0x7f298815e2b0 0x7f298806b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.923+0000 7f298d000640 1 --2- 172.21.15.47:0/2507509577 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f29880b1280 0x7f298806aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.923+0000 7f298d000640 1 --2- 172.21.15.47:0/2507509577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f298815ded0 0x7f2988066cd0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.923+0000 7f298d000640 1 -- 172.21.15.47:0/2507509577 >> 172.21.15.47:0/2507509577 conn(0x7f29880b32f0 msgr2=0x7f29880b36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:18.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.923+0000 7f298d000640 1 -- 172.21.15.47:0/2507509577 shutdown_connections 2024-02-19T15:27:18.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.923+0000 7f298d000640 1 -- 172.21.15.47:0/2507509577 wait complete. 2024-02-19T15:27:18.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f298d000640 1 Processor -- start 2024-02-19T15:27:18.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f298d000640 1 -- start start 2024-02-19T15:27:18.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f298d000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f29880b1280 0x7f2988156160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:18.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f298d000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f298815ded0 0x7f298815ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:18.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f298d000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f298815e2b0 0x7f298815bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:18.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f298d000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298815a570 con 0x7f298815ded0 2024-02-19T15:27:18.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f298d000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298815a850 con 0x7f298815e2b0 2024-02-19T15:27:18.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f298d000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f298815ab30 con 0x7f29880b1280 2024-02-19T15:27:18.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f297fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f298815ded0 0x7f298815ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:18.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f297fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f298815ded0 0x7f298815ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56802/0 (socket says 172.21.15.47:56802) 2024-02-19T15:27:18.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f297fe00640 1 -- 172.21.15.47:0/2797279014 learned_addr learned my addr 172.21.15.47:0/2797279014 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:18.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f2986800640 1 --2- 172.21.15.47:0/2797279014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f298815e2b0 0x7f298815bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:18.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f297fe00640 1 -- 172.21.15.47:0/2797279014 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f29880b1280 msgr2=0x7f2988156160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:18.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f2985e00640 1 --2- 172.21.15.47:0/2797279014 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f29880b1280 0x7f2988156160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:18.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f297fe00640 1 --2- 172.21.15.47:0/2797279014 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f29880b1280 0x7f2988156160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f297fe00640 1 -- 172.21.15.47:0/2797279014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f298815e2b0 msgr2=0x7f298815bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:18.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f297fe00640 1 --2- 172.21.15.47:0/2797279014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f298815e2b0 0x7f298815bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:18.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f297fe00640 1 -- 172.21.15.47:0/2797279014 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29780094d0 con 0x7f298815ded0 2024-02-19T15:27:18.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f2986800640 1 --2- 172.21.15.47:0/2797279014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f298815e2b0 0x7f298815bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:18.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f2985e00640 1 --2- 172.21.15.47:0/2797279014 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f29880b1280 0x7f2988156160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:18.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f297fe00640 1 --2- 172.21.15.47:0/2797279014 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f298815ded0 0x7f298815ba40 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f297000d7d0 tx=0x7f297000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:18.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f297e000640 1 -- 172.21.15.47:0/2797279014 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29700096a0 con 0x7f298815ded0 2024-02-19T15:27:18.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f297e000640 1 -- 172.21.15.47:0/2797279014 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2970009800 con 0x7f298815ded0 2024-02-19T15:27:18.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f297e000640 1 -- 172.21.15.47:0/2797279014 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2970009bb0 con 0x7f298815ded0 2024-02-19T15:27:18.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.927+0000 7f298d000640 1 -- 172.21.15.47:0/2797279014 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f298815ae10 con 0x7f298815ded0 2024-02-19T15:27:18.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.931+0000 7f298d000640 1 -- 172.21.15.47:0/2797279014 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29880af1f0 con 0x7f298815ded0 2024-02-19T15:27:18.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.931+0000 7f298d000640 1 -- 172.21.15.47:0/2797279014 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2988075360 con 0x7f298815ded0 2024-02-19T15:27:18.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.931+0000 7f297e000640 1 -- 172.21.15.47:0/2797279014 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2970009d10 con 0x7f298815ded0 2024-02-19T15:27:18.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.931+0000 7f297e000640 1 --2- 172.21.15.47:0/2797279014 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2964066e40 0x7f2964069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:18.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.931+0000 7f297e000640 1 -- 172.21.15.47:0/2797279014 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2970084c90 con 0x7f298815ded0 2024-02-19T15:27:18.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.931+0000 7f2985e00640 1 --2- 172.21.15.47:0/2797279014 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2964066e40 0x7f2964069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:18.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.931+0000 7f2985e00640 1 --2- 172.21.15.47:0/2797279014 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2964066e40 0x7f2964069300 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f29780ae000 tx=0x7f2978008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:18.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:18.935+0000 7f297e000640 1 -- 172.21.15.47:0/2797279014 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2970051d30 con 0x7f298815ded0 2024-02-19T15:27:19.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.099+0000 7f298d000640 1 -- 172.21.15.47:0/2797279014 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f298815a6b0 con 0x7f298815ded0 2024-02-19T15:27:19.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.099+0000 7f297e000640 1 -- 172.21.15.47:0/2797279014 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7f298815a6b0 con 0x7f298815ded0 2024-02-19T15:27:19.104 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:27:19.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.103+0000 7f298d000640 1 -- 172.21.15.47:0/2797279014 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2964066e40 msgr2=0x7f2964069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:19.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.103+0000 7f298d000640 1 --2- 172.21.15.47:0/2797279014 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2964066e40 0x7f2964069300 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f29780ae000 tx=0x7f2978008040 comp rx=0 tx=0).stop 2024-02-19T15:27:19.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.103+0000 7f298d000640 1 -- 172.21.15.47:0/2797279014 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f298815ded0 msgr2=0x7f298815ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:19.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.103+0000 7f298d000640 1 --2- 172.21.15.47:0/2797279014 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f298815ded0 0x7f298815ba40 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f297000d7d0 tx=0x7f297000dc90 comp rx=0 tx=0).stop 2024-02-19T15:27:19.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.103+0000 7f298d000640 1 -- 172.21.15.47:0/2797279014 shutdown_connections 2024-02-19T15:27:19.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.103+0000 7f298d000640 1 --2- 172.21.15.47:0/2797279014 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2964066e40 0x7f2964069300 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:19.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.103+0000 7f298d000640 1 --2- 172.21.15.47:0/2797279014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f298815e2b0 0x7f298815bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:19.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.103+0000 7f298d000640 1 --2- 172.21.15.47:0/2797279014 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f298815ded0 0x7f298815ba40 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:19.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.103+0000 7f298d000640 1 --2- 172.21.15.47:0/2797279014 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f29880b1280 0x7f2988156160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:19.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.103+0000 7f298d000640 1 -- 172.21.15.47:0/2797279014 >> 172.21.15.47:0/2797279014 conn(0x7f29880b32f0 msgr2=0x7f298815c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:19.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.103+0000 7f298d000640 1 -- 172.21.15.47:0/2797279014 shutdown_connections 2024-02-19T15:27:19.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.103+0000 7f298d000640 1 -- 172.21.15.47:0/2797279014 wait complete. 2024-02-19T15:27:19.128 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T15:27:19.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.263+0000 7f26b2800640 1 Processor -- start 2024-02-19T15:27:19.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.263+0000 7f26b2800640 1 -- start start 2024-02-19T15:27:19.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.263+0000 7f26b2800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26ac15b1a0 0x7f26ac15b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:19.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.263+0000 7f26b2800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f26ac071350 0x7f26ac15bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:19.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.263+0000 7f26b2800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ac15c000 con 0x7f26ac06c9a0 2024-02-19T15:27:19.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.263+0000 7f26b2800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26ac0718c0 con 0x7f26ac071350 2024-02-19T15:27:19.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.263+0000 7f26b2800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26ac071ba0 con 0x7f26ac15b1a0 2024-02-19T15:27:19.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.263+0000 7f26a3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f26ac071350 0x7f26ac15bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:19.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.263+0000 7f26ab400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26ac15b1a0 0x7f26ac15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:19.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.263+0000 7f26a3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f26ac071350 0x7f26ac15bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56812/0 (socket says 172.21.15.47:56812) 2024-02-19T15:27:19.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.263+0000 7f26a3e00640 1 -- 172.21.15.47:0/1673770316 learned_addr learned my addr 172.21.15.47:0/1673770316 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:19.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.267+0000 7f26a3e00640 1 -- 172.21.15.47:0/1673770316 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26ac15b1a0 msgr2=0x7f26ac15b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:19.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.267+0000 7f26a3e00640 1 --2- 172.21.15.47:0/1673770316 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26ac15b1a0 0x7f26ac15b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:19.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.267+0000 7f26a3e00640 1 -- 172.21.15.47:0/1673770316 >> v1:172.21.15.47:6789/0 conn(0x7f26ac06c9a0 legacy=0x7f26ac15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:19.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.267+0000 7f26a3e00640 1 -- 172.21.15.47:0/1673770316 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ac071e80 con 0x7f26ac071350 2024-02-19T15:27:19.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.267+0000 7f26a3e00640 1 --2- 172.21.15.47:0/1673770316 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26ac071350 0x7f26ac15bac0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f2694009870 tx=0x7f269401c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2134736df03b0b82 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:19.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.267+0000 7f26aaa00640 1 -- 172.21.15.47:0/1673770316 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26940a9610 con 0x7f26ac071350 2024-02-19T15:27:19.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.267+0000 7f26aaa00640 1 -- 172.21.15.47:0/1673770316 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26940a9770 con 0x7f26ac071350 2024-02-19T15:27:19.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.267+0000 7f26aaa00640 1 -- 172.21.15.47:0/1673770316 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26940a9b00 con 0x7f26ac071350 2024-02-19T15:27:19.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.267+0000 7f26b2800640 1 -- 172.21.15.47:0/1673770316 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26ac071350 msgr2=0x7f26ac15bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:19.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.267+0000 7f26b2800640 1 --2- 172.21.15.47:0/1673770316 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26ac071350 0x7f26ac15bac0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f2694009870 tx=0x7f269401c530 comp rx=0 tx=0).stop 2024-02-19T15:27:19.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.267+0000 7f26b2800640 1 -- 172.21.15.47:0/1673770316 shutdown_connections 2024-02-19T15:27:19.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.267+0000 7f26b2800640 1 --2- 172.21.15.47:0/1673770316 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26ac071350 0x7f26ac15bac0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:19.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.267+0000 7f26b2800640 1 --2- 172.21.15.47:0/1673770316 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26ac15b1a0 0x7f26ac15b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:19.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.267+0000 7f26b2800640 1 -- 172.21.15.47:0/1673770316 >> 172.21.15.47:0/1673770316 conn(0x7f26ac0b16a0 msgr2=0x7f26ac0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:19.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.267+0000 7f26b2800640 1 -- 172.21.15.47:0/1673770316 shutdown_connections 2024-02-19T15:27:19.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.267+0000 7f26b2800640 1 -- 172.21.15.47:0/1673770316 wait complete. 2024-02-19T15:27:19.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26b2800640 1 Processor -- start 2024-02-19T15:27:19.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26b2800640 1 -- start start 2024-02-19T15:27:19.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26b2800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26ac06c9a0 0x7f26ac139a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:19.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26b2800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26ac071350 0x7f26ac0cd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:19.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26b2800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26ac15b1a0 0x7f26ac155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:19.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26b2800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ac0ce850 con 0x7f26ac06c9a0 2024-02-19T15:27:19.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26b2800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ac0ce290 con 0x7f26ac071350 2024-02-19T15:27:19.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26b2800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26ac0ce570 con 0x7f26ac15b1a0 2024-02-19T15:27:19.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26ab400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26ac06c9a0 0x7f26ac139a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:19.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26ab400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26ac06c9a0 0x7f26ac139a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56814/0 (socket says 172.21.15.47:56814) 2024-02-19T15:27:19.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26ab400640 1 -- 172.21.15.47:0/260296666 learned_addr learned my addr 172.21.15.47:0/260296666 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:19.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26abe00640 1 --2- 172.21.15.47:0/260296666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26ac15b1a0 0x7f26ac155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:19.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26a3e00640 1 --2- 172.21.15.47:0/260296666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26ac071350 0x7f26ac0cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:19.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26ab400640 1 -- 172.21.15.47:0/260296666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26ac15b1a0 msgr2=0x7f26ac155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:19.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26ab400640 1 --2- 172.21.15.47:0/260296666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26ac15b1a0 0x7f26ac155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:19.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26ab400640 1 -- 172.21.15.47:0/260296666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26ac071350 msgr2=0x7f26ac0cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:19.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26ab400640 1 --2- 172.21.15.47:0/260296666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26ac071350 0x7f26ac0cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:19.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26ab400640 1 -- 172.21.15.47:0/260296666 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26940094d0 con 0x7f26ac06c9a0 2024-02-19T15:27:19.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26ab400640 1 --2- 172.21.15.47:0/260296666 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26ac06c9a0 0x7f26ac139a90 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f269c00e930 tx=0x7f269c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:19.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26a3400640 1 -- 172.21.15.47:0/260296666 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c009690 con 0x7f26ac06c9a0 2024-02-19T15:27:19.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26a3400640 1 -- 172.21.15.47:0/260296666 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269c00ca00 con 0x7f26ac06c9a0 2024-02-19T15:27:19.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26a3400640 1 -- 172.21.15.47:0/260296666 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c00cd70 con 0x7f26ac06c9a0 2024-02-19T15:27:19.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26b2800640 1 -- 172.21.15.47:0/260296666 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26ac13a4d0 con 0x7f26ac06c9a0 2024-02-19T15:27:19.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.271+0000 7f26b2800640 1 -- 172.21.15.47:0/260296666 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26ac139fd0 con 0x7f26ac06c9a0 2024-02-19T15:27:19.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.275+0000 7f26b2800640 1 -- 172.21.15.47:0/260296666 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26ac075360 con 0x7f26ac06c9a0 2024-02-19T15:27:19.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.275+0000 7f26a3400640 1 -- 172.21.15.47:0/260296666 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f269c013470 con 0x7f26ac06c9a0 2024-02-19T15:27:19.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.275+0000 7f26a3400640 1 --2- 172.21.15.47:0/260296666 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2688066e40 0x7f2688069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:19.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.275+0000 7f26a3400640 1 -- 172.21.15.47:0/260296666 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f269c084a70 con 0x7f26ac06c9a0 2024-02-19T15:27:19.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.275+0000 7f26a3e00640 1 --2- 172.21.15.47:0/260296666 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2688066e40 0x7f2688069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:19.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.275+0000 7f26a3e00640 1 --2- 172.21.15.47:0/260296666 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2688066e40 0x7f2688069300 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f26940ae000 tx=0x7f2694009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:19.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.279+0000 7f26a3400640 1 -- 172.21.15.47:0/260296666 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f269c051b70 con 0x7f26ac06c9a0 2024-02-19T15:27:19.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.443+0000 7f26b2800640 1 -- 172.21.15.47:0/260296666 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f26ac158540 con 0x7f26ac06c9a0 2024-02-19T15:27:19.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.443+0000 7f26a3400640 1 -- 172.21.15.47:0/260296666 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v104) v1 ==== 84+0+65 (secure 0 0 0) 0x7f269c017090 con 0x7f26ac06c9a0 2024-02-19T15:27:19.449 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:19.449 INFO:teuthology.orchestra.run.smithi047.stdout:[ 2024-02-19T15:27:19.449 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:19.449 INFO:teuthology.orchestra.run.smithi047.stdout: "poolnum": 1, 2024-02-19T15:27:19.450 INFO:teuthology.orchestra.run.smithi047.stdout: "poolname": ".mgr" 2024-02-19T15:27:19.450 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:19.450 INFO:teuthology.orchestra.run.smithi047.stdout:] 2024-02-19T15:27:19.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.447+0000 7f26b2800640 1 -- 172.21.15.47:0/260296666 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2688066e40 msgr2=0x7f2688069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:19.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.447+0000 7f26b2800640 1 --2- 172.21.15.47:0/260296666 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2688066e40 0x7f2688069300 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f26940ae000 tx=0x7f2694009040 comp rx=0 tx=0).stop 2024-02-19T15:27:19.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.447+0000 7f26b2800640 1 -- 172.21.15.47:0/260296666 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26ac06c9a0 msgr2=0x7f26ac139a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:19.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.447+0000 7f26b2800640 1 --2- 172.21.15.47:0/260296666 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26ac06c9a0 0x7f26ac139a90 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f269c00e930 tx=0x7f269c00edf0 comp rx=0 tx=0).stop 2024-02-19T15:27:19.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.447+0000 7f26b2800640 1 -- 172.21.15.47:0/260296666 shutdown_connections 2024-02-19T15:27:19.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.447+0000 7f26b2800640 1 --2- 172.21.15.47:0/260296666 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2688066e40 0x7f2688069300 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:19.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.447+0000 7f26b2800640 1 --2- 172.21.15.47:0/260296666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26ac15b1a0 0x7f26ac155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:19.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.447+0000 7f26b2800640 1 --2- 172.21.15.47:0/260296666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26ac071350 0x7f26ac0cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:19.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.447+0000 7f26b2800640 1 --2- 172.21.15.47:0/260296666 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26ac06c9a0 0x7f26ac139a90 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:19.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.447+0000 7f26b2800640 1 -- 172.21.15.47:0/260296666 >> 172.21.15.47:0/260296666 conn(0x7f26ac0b16a0 msgr2=0x7f26ac0af680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:19.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.447+0000 7f26b2800640 1 -- 172.21.15.47:0/260296666 shutdown_connections 2024-02-19T15:27:19.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.447+0000 7f26b2800640 1 -- 172.21.15.47:0/260296666 wait complete. 2024-02-19T15:27:19.470 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T15:27:19.470 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T15:27:19.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.607+0000 7f256f400640 1 Processor -- start 2024-02-19T15:27:19.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f256f400640 1 -- start start 2024-02-19T15:27:19.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f256f400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f256806c4c0 0x7f256806e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:19.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f256f400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25680b5650 con 0x7f256806c920 2024-02-19T15:27:19.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f256f400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2568071560 con 0x7f25680b5270 2024-02-19T15:27:19.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f256f400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256815f7b0 con 0x7f256806c4c0 2024-02-19T15:27:19.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f256d400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f256806c920 0x7f25680c9fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49054/0 (socket says 172.21.15.47:49054) 2024-02-19T15:27:19.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f256d400640 1 -- 172.21.15.47:0/1586545513 learned_addr learned my addr 172.21.15.47:0/1586545513 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:19.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f2567e00640 1 --2- 172.21.15.47:0/1586545513 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f256806c4c0 0x7f256806e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:19.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f2567400640 1 -- 172.21.15.47:0/1586545513 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839701692 0 0) 0x7f25680b5650 con 0x7f256806c920 2024-02-19T15:27:19.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f2567400640 1 -- 172.21.15.47:0/1586545513 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2550003530 con 0x7f256806c920 2024-02-19T15:27:19.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f2567400640 1 -- 172.21.15.47:0/1586545513 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2699897908 0 0) 0x7f2550003530 con 0x7f256806c920 2024-02-19T15:27:19.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f2567400640 1 -- 172.21.15.47:0/1586545513 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f256806c4c0 msgr2=0x7f256806e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:19.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f2567400640 1 --2- 172.21.15.47:0/1586545513 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f256806c4c0 0x7f256806e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:19.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f2567400640 1 -- 172.21.15.47:0/1586545513 >> v1:172.21.15.116:6789/0 conn(0x7f25680b5270 legacy=0x7f25680719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f2567400640 1 -- 172.21.15.47:0/1586545513 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2568072320 con 0x7f256806c920 2024-02-19T15:27:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f2567400640 1 -- 172.21.15.47:0/1586545513 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f2560001a00 con 0x7f256806c920 2024-02-19T15:27:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f2567e00640 1 --2- 172.21.15.47:0/1586545513 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f256806c4c0 0x7f256806e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:27:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f2567400640 1 -- 172.21.15.47:0/1586545513 >> v1:172.21.15.47:6789/0 conn(0x7f256806c920 legacy=0x7f25680c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f2567400640 1 --2- 172.21.15.47:0/1586545513 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25500039b0 0x7f2550003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f2567400640 1 --2- 172.21.15.47:0/1586545513 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f256806c4c0 0x7f25500047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f2567400640 1 --2- 172.21.15.47:0/1586545513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2550004d10 0x7f2550005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f2567400640 1 -- 172.21.15.47:0/1586545513 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256815f7b0 con 0x7f25500039b0 2024-02-19T15:27:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f2567400640 1 -- 172.21.15.47:0/1586545513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255000b990 con 0x7f2550004d10 2024-02-19T15:27:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f2567400640 1 -- 172.21.15.47:0/1586545513 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f255000bb50 con 0x7f256806c4c0 2024-02-19T15:27:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f256ca00640 1 --2- 172.21.15.47:0/1586545513 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25500039b0 0x7f2550003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f2567e00640 1 --2- 172.21.15.47:0/1586545513 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f256806c4c0 0x7f25500047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f256d400640 1 --2- 172.21.15.47:0/1586545513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2550004d10 0x7f2550005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f256ca00640 1 -- 172.21.15.47:0/1586545513 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f256806c4c0 msgr2=0x7f25500047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f256ca00640 1 --2- 172.21.15.47:0/1586545513 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f256806c4c0 0x7f25500047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:19.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f256ca00640 1 -- 172.21.15.47:0/1586545513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2550004d10 msgr2=0x7f2550005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:19.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f256ca00640 1 --2- 172.21.15.47:0/1586545513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2550004d10 0x7f2550005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:19.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.611+0000 7f256ca00640 1 -- 172.21.15.47:0/1586545513 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2554009420 con 0x7f25500039b0 2024-02-19T15:27:19.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.615+0000 7f256ca00640 1 --2- 172.21.15.47:0/1586545513 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25500039b0 0x7f2550003d90 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f255c001860 tx=0x7f255c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f7a4d43aefc5cf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:19.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.615+0000 7f2567400640 1 -- 172.21.15.47:0/1586545513 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c0ab600 con 0x7f25500039b0 2024-02-19T15:27:19.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.615+0000 7f2567400640 1 -- 172.21.15.47:0/1586545513 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f255c0a7050 con 0x7f25500039b0 2024-02-19T15:27:19.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.615+0000 7f2567400640 1 -- 172.21.15.47:0/1586545513 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c0ab9a0 con 0x7f25500039b0 2024-02-19T15:27:19.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.615+0000 7f256f400640 1 -- 172.21.15.47:0/1586545513 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25500039b0 msgr2=0x7f2550003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:19.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.615+0000 7f256f400640 1 --2- 172.21.15.47:0/1586545513 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25500039b0 0x7f2550003d90 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f255c001860 tx=0x7f255c01c520 comp rx=0 tx=0).stop 2024-02-19T15:27:19.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.615+0000 7f256d400640 1 -- 172.21.15.47:0/1586545513 reap_dead start 2024-02-19T15:27:19.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.615+0000 7f256f400640 1 -- 172.21.15.47:0/1586545513 shutdown_connections 2024-02-19T15:27:19.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.615+0000 7f256f400640 1 -- 172.21.15.47:0/1586545513 >> 172.21.15.47:0/1586545513 conn(0x7f25680cc3f0 msgr2=0x7f25680cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:19.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.615+0000 7f256f400640 1 -- 172.21.15.47:0/1586545513 shutdown_connections 2024-02-19T15:27:19.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.615+0000 7f256f400640 1 -- 172.21.15.47:0/1586545513 wait complete. 2024-02-19T15:27:19.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f256f400640 1 Processor -- start 2024-02-19T15:27:19.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f256f400640 1 -- start start 2024-02-19T15:27:19.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f256f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25500039b0 0x7f25681f7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:19.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f256f400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f256806b7d0 0x7f25681f9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:19.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f256f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25681f69c0 0x7f256815dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:19.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f256f400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25680b24c0 con 0x7f25500039b0 2024-02-19T15:27:19.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f256f400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25681f9d90 con 0x7f25681f69c0 2024-02-19T15:27:19.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f256f400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25681f7340 con 0x7f256806b7d0 2024-02-19T15:27:19.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f256ca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25500039b0 0x7f25681f7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:19.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f256ca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25500039b0 0x7f25681f7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56842/0 (socket says 172.21.15.47:56842) 2024-02-19T15:27:19.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f256ca00640 1 -- 172.21.15.47:0/4132077093 learned_addr learned my addr 172.21.15.47:0/4132077093 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:19.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f256ca00640 1 -- 172.21.15.47:0/4132077093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f256806b7d0 msgr2=0x7f25681f9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:19.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f256ca00640 1 --2- 172.21.15.47:0/4132077093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f256806b7d0 0x7f25681f9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:19.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f256ca00640 1 -- 172.21.15.47:0/4132077093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25681f69c0 msgr2=0x7f256815dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:19.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f256ca00640 1 --2- 172.21.15.47:0/4132077093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25681f69c0 0x7f256815dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:19.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f256ca00640 1 -- 172.21.15.47:0/4132077093 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25681f7620 con 0x7f25500039b0 2024-02-19T15:27:19.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f256ca00640 1 --2- 172.21.15.47:0/4132077093 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25500039b0 0x7f25681f7bd0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f255c01c350 tx=0x7f2550005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:19.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f2564c00640 1 -- 172.21.15.47:0/4132077093 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c005270 con 0x7f25500039b0 2024-02-19T15:27:19.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f2564c00640 1 -- 172.21.15.47:0/4132077093 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f255c0053d0 con 0x7f25500039b0 2024-02-19T15:27:19.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f256f400640 1 -- 172.21.15.47:0/4132077093 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25680b2730 con 0x7f25500039b0 2024-02-19T15:27:19.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f2564c00640 1 -- 172.21.15.47:0/4132077093 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c007240 con 0x7f25500039b0 2024-02-19T15:27:19.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f256f400640 1 -- 172.21.15.47:0/4132077093 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25681f6e00 con 0x7f25500039b0 2024-02-19T15:27:19.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.619+0000 7f256f400640 1 -- 172.21.15.47:0/4132077093 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25680b2420 con 0x7f25500039b0 2024-02-19T15:27:19.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.623+0000 7f2564c00640 1 -- 172.21.15.47:0/4132077093 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f255c0b2050 con 0x7f25500039b0 2024-02-19T15:27:19.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.623+0000 7f2564c00640 1 --2- 172.21.15.47:0/4132077093 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2548066e70 0x7f2548069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:19.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.623+0000 7f2564c00640 1 -- 172.21.15.47:0/4132077093 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f255c11e440 con 0x7f25500039b0 2024-02-19T15:27:19.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.623+0000 7f2567e00640 1 --2- 172.21.15.47:0/4132077093 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2548066e70 0x7f2548069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:19.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.623+0000 7f2567e00640 1 --2- 172.21.15.47:0/4132077093 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2548066e70 0x7f2548069330 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f25681f4600 tx=0x7f2554006c40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:19.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.627+0000 7f2564c00640 1 -- 172.21.15.47:0/4132077093 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f255c0eb590 con 0x7f25500039b0 2024-02-19T15:27:19.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.787+0000 7f256f400640 1 -- 172.21.15.47:0/4132077093 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2568075360 con 0x7f25500039b0 2024-02-19T15:27:20.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:19.999+0000 7f2564c00640 1 -- 172.21.15.47:0/4132077093 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v105) v1 ==== 138+0+0 (secure 0 0 0) 0x7f255c006e60 con 0x7f25500039b0 2024-02-19T15:27:20.004 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_metadata' created 2024-02-19T15:27:20.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.003+0000 7f256f400640 1 -- 172.21.15.47:0/4132077093 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2548066e70 msgr2=0x7f2548069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:20.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.003+0000 7f256f400640 1 --2- 172.21.15.47:0/4132077093 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2548066e70 0x7f2548069330 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f25681f4600 tx=0x7f2554006c40 comp rx=0 tx=0).stop 2024-02-19T15:27:20.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.003+0000 7f256f400640 1 -- 172.21.15.47:0/4132077093 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25500039b0 msgr2=0x7f25681f7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:20.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.003+0000 7f256f400640 1 --2- 172.21.15.47:0/4132077093 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25500039b0 0x7f25681f7bd0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f255c01c350 tx=0x7f2550005800 comp rx=0 tx=0).stop 2024-02-19T15:27:20.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.003+0000 7f256f400640 1 -- 172.21.15.47:0/4132077093 shutdown_connections 2024-02-19T15:27:20.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.003+0000 7f256f400640 1 --2- 172.21.15.47:0/4132077093 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2548066e70 0x7f2548069330 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:20.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.003+0000 7f256f400640 1 --2- 172.21.15.47:0/4132077093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25681f69c0 0x7f256815dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:20.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.003+0000 7f256f400640 1 --2- 172.21.15.47:0/4132077093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f256806b7d0 0x7f25681f9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:20.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.003+0000 7f256f400640 1 --2- 172.21.15.47:0/4132077093 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25500039b0 0x7f25681f7bd0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:20.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.003+0000 7f256f400640 1 -- 172.21.15.47:0/4132077093 >> 172.21.15.47:0/4132077093 conn(0x7f25680cc3f0 msgr2=0x7f25680ad850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:20.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.003+0000 7f256f400640 1 -- 172.21.15.47:0/4132077093 shutdown_connections 2024-02-19T15:27:20.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.003+0000 7f256f400640 1 -- 172.21.15.47:0/4132077093 wait complete. 2024-02-19T15:27:20.030 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T15:27:20.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.175+0000 7f4047000640 1 Processor -- start 2024-02-19T15:27:20.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.175+0000 7f4047000640 1 -- start start 2024-02-19T15:27:20.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.175+0000 7f4047000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f40400cec80 0x7f40400d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:20.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.175+0000 7f4047000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f404015f190 0x7f404006cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:20.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.175+0000 7f4047000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40400d0a50 con 0x7f404015f8f0 2024-02-19T15:27:20.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.175+0000 7f4047000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f404006d470 con 0x7f40400cec80 2024-02-19T15:27:20.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.175+0000 7f4047000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40400cfe80 con 0x7f404015f190 2024-02-19T15:27:20.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.175+0000 7f4045000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f404015f8f0 0x7f404015fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49068/0 (socket says 172.21.15.47:49068) 2024-02-19T15:27:20.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.175+0000 7f4045000640 1 -- 172.21.15.47:0/556166822 learned_addr learned my addr 172.21.15.47:0/556166822 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:20.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.175+0000 7f4037e00640 1 --2- 172.21.15.47:0/556166822 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f404015f190 0x7f404006cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:20.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.175+0000 7f403fe00640 1 --2- 172.21.15.47:0/556166822 >> v2:172.21.15.116:3300/0 conn(0x7f40400cec80 0x7f40400d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:20.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.175+0000 7f403f400640 1 -- 172.21.15.47:0/556166822 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639392687 0 0) 0x7f40400d0a50 con 0x7f404015f8f0 2024-02-19T15:27:20.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.175+0000 7f403f400640 1 -- 172.21.15.47:0/556166822 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4024003530 con 0x7f404015f8f0 2024-02-19T15:27:20.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.175+0000 7f4037e00640 1 -- 172.21.15.47:0/556166822 >> v2:172.21.15.116:3300/0 conn(0x7f40400cec80 msgr2=0x7f40400d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:20.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.175+0000 7f4037e00640 1 --2- 172.21.15.47:0/556166822 >> v2:172.21.15.116:3300/0 conn(0x7f40400cec80 0x7f40400d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:20.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.175+0000 7f4037e00640 1 -- 172.21.15.47:0/556166822 >> v1:172.21.15.47:6789/0 conn(0x7f404015f8f0 legacy=0x7f404015fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:20.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.175+0000 7f403f400640 1 -- 172.21.15.47:0/556166822 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 647867412 0 0) 0x7f4024003530 con 0x7f404015f8f0 2024-02-19T15:27:20.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.175+0000 7f4037e00640 1 -- 172.21.15.47:0/556166822 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40400d0120 con 0x7f404015f190 2024-02-19T15:27:20.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.179+0000 7f4037e00640 1 --2- 172.21.15.47:0/556166822 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f404015f190 0x7f404006cbe0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f40280097c0 tx=0x7f402801c390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8a3b200236a803b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:20.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.179+0000 7f403f400640 1 -- 172.21.15.47:0/556166822 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4038001a00 con 0x7f404015f190 2024-02-19T15:27:20.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.179+0000 7f403f400640 1 -- 172.21.15.47:0/556166822 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40280a7050 con 0x7f404015f190 2024-02-19T15:27:20.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.179+0000 7f403f400640 1 -- 172.21.15.47:0/556166822 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40280ab620 con 0x7f404015f190 2024-02-19T15:27:20.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.179+0000 7f4047000640 1 -- 172.21.15.47:0/556166822 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f404015f190 msgr2=0x7f404006cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:20.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.179+0000 7f4047000640 1 --2- 172.21.15.47:0/556166822 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f404015f190 0x7f404006cbe0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f40280097c0 tx=0x7f402801c390 comp rx=0 tx=0).stop 2024-02-19T15:27:20.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.179+0000 7f4047000640 1 -- 172.21.15.47:0/556166822 shutdown_connections 2024-02-19T15:27:20.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.179+0000 7f4047000640 1 --2- 172.21.15.47:0/556166822 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f404015f190 0x7f404006cbe0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:20.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.179+0000 7f4047000640 1 --2- 172.21.15.47:0/556166822 >> v2:172.21.15.116:3300/0 conn(0x7f40400cec80 0x7f40400d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:20.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.179+0000 7f4047000640 1 -- 172.21.15.47:0/556166822 >> 172.21.15.47:0/556166822 conn(0x7f404009aed0 msgr2=0x7f40400d1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:20.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.179+0000 7f4047000640 1 -- 172.21.15.47:0/556166822 shutdown_connections 2024-02-19T15:27:20.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.179+0000 7f4047000640 1 -- 172.21.15.47:0/556166822 wait complete. 2024-02-19T15:27:20.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f4047000640 1 Processor -- start 2024-02-19T15:27:20.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f4047000640 1 -- start start 2024-02-19T15:27:20.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f4047000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40400cec80 0x7f40400cb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:20.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f4047000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f404015f190 0x7f40400c7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:20.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f4047000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f404015f8f0 0x7f4040159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:20.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f4047000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40400c6a30 con 0x7f40400cec80 2024-02-19T15:27:20.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f4047000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40400c9070 con 0x7f404015f8f0 2024-02-19T15:27:20.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f4047000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40400c6c80 con 0x7f404015f190 2024-02-19T15:27:20.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f403fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40400cec80 0x7f40400cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:20.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f403fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40400cec80 0x7f40400cb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56844/0 (socket says 172.21.15.47:56844) 2024-02-19T15:27:20.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f403fe00640 1 -- 172.21.15.47:0/4098748508 learned_addr learned my addr 172.21.15.47:0/4098748508 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:20.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f4037e00640 1 --2- 172.21.15.47:0/4098748508 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f404015f190 0x7f40400c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:20.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f4045000640 1 --2- 172.21.15.47:0/4098748508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f404015f8f0 0x7f4040159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:20.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f403fe00640 1 -- 172.21.15.47:0/4098748508 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f404015f190 msgr2=0x7f40400c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:20.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f403fe00640 1 --2- 172.21.15.47:0/4098748508 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f404015f190 0x7f40400c7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:20.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f403fe00640 1 -- 172.21.15.47:0/4098748508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f404015f8f0 msgr2=0x7f4040159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:20.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f403fe00640 1 --2- 172.21.15.47:0/4098748508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f404015f8f0 0x7f4040159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:20.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f403fe00640 1 -- 172.21.15.47:0/4098748508 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4028009420 con 0x7f40400cec80 2024-02-19T15:27:20.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f4045000640 1 --2- 172.21.15.47:0/4098748508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f404015f8f0 0x7f4040159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:27:20.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f403fe00640 1 --2- 172.21.15.47:0/4098748508 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40400cec80 0x7f40400cb690 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f403000e930 tx=0x7f403000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:20.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f403cc00640 1 -- 172.21.15.47:0/4098748508 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4030009690 con 0x7f40400cec80 2024-02-19T15:27:20.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f403cc00640 1 -- 172.21.15.47:0/4098748508 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403000ca00 con 0x7f40400cec80 2024-02-19T15:27:20.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f403cc00640 1 -- 172.21.15.47:0/4098748508 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403000cd70 con 0x7f40400cec80 2024-02-19T15:27:20.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f4047000640 1 -- 172.21.15.47:0/4098748508 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40400c6f60 con 0x7f40400cec80 2024-02-19T15:27:20.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.183+0000 7f4047000640 1 -- 172.21.15.47:0/4098748508 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40400c7ca0 con 0x7f40400cec80 2024-02-19T15:27:20.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.187+0000 7f4047000640 1 -- 172.21.15.47:0/4098748508 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4040075360 con 0x7f40400cec80 2024-02-19T15:27:20.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.187+0000 7f403cc00640 1 -- 172.21.15.47:0/4098748508 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4030013470 con 0x7f40400cec80 2024-02-19T15:27:20.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.187+0000 7f403cc00640 1 --2- 172.21.15.47:0/4098748508 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f401c066e40 0x7f401c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:20.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.187+0000 7f403cc00640 1 -- 172.21.15.47:0/4098748508 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f40300852f0 con 0x7f40400cec80 2024-02-19T15:27:20.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.187+0000 7f4037e00640 1 --2- 172.21.15.47:0/4098748508 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f401c066e40 0x7f401c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:20.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.187+0000 7f4037e00640 1 --2- 172.21.15.47:0/4098748508 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f401c066e40 0x7f401c069300 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f40280ad000 tx=0x7f4028008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:20.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.191+0000 7f403cc00640 1 -- 172.21.15.47:0/4098748508 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f4030052410 con 0x7f40400cec80 2024-02-19T15:27:20.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:20.355+0000 7f4047000640 1 -- 172.21.15.47:0/4098748508 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f404006b280 con 0x7f40400cec80 2024-02-19T15:27:21.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.003+0000 7f403cc00640 1 -- 172.21.15.47:0/4098748508 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v106) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4030055ad0 con 0x7f40400cec80 2024-02-19T15:27:21.007 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_data' created 2024-02-19T15:27:21.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.007+0000 7f4047000640 1 -- 172.21.15.47:0/4098748508 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f401c066e40 msgr2=0x7f401c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:21.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.007+0000 7f4047000640 1 --2- 172.21.15.47:0/4098748508 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f401c066e40 0x7f401c069300 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f40280ad000 tx=0x7f4028008040 comp rx=0 tx=0).stop 2024-02-19T15:27:21.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.007+0000 7f4047000640 1 -- 172.21.15.47:0/4098748508 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40400cec80 msgr2=0x7f40400cb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:21.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.007+0000 7f4047000640 1 --2- 172.21.15.47:0/4098748508 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40400cec80 0x7f40400cb690 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f403000e930 tx=0x7f403000edf0 comp rx=0 tx=0).stop 2024-02-19T15:27:21.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.007+0000 7f4047000640 1 -- 172.21.15.47:0/4098748508 shutdown_connections 2024-02-19T15:27:21.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.007+0000 7f4047000640 1 --2- 172.21.15.47:0/4098748508 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f401c066e40 0x7f401c069300 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:21.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.007+0000 7f4047000640 1 --2- 172.21.15.47:0/4098748508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f404015f8f0 0x7f4040159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:21.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.007+0000 7f4047000640 1 --2- 172.21.15.47:0/4098748508 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f404015f190 0x7f40400c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:21.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.007+0000 7f4047000640 1 --2- 172.21.15.47:0/4098748508 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40400cec80 0x7f40400cb690 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:21.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.007+0000 7f4047000640 1 -- 172.21.15.47:0/4098748508 >> 172.21.15.47:0/4098748508 conn(0x7f404009aed0 msgr2=0x7f40401621f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:21.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.011+0000 7f4047000640 1 -- 172.21.15.47:0/4098748508 shutdown_connections 2024-02-19T15:27:21.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.011+0000 7f4047000640 1 -- 172.21.15.47:0/4098748508 wait complete. 2024-02-19T15:27:21.053 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T15:27:21.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.195+0000 7f071ee00640 1 Processor -- start 2024-02-19T15:27:21.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.195+0000 7f071ee00640 1 -- start start 2024-02-19T15:27:21.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.195+0000 7f071ee00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f071815ded0 0x7f0718066cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:21.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.195+0000 7f071ee00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0718067210 con 0x7f07180b1280 2024-02-19T15:27:21.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.195+0000 7f071ee00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07180b04c0 con 0x7f071806b5c0 2024-02-19T15:27:21.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.195+0000 7f071ee00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07180b07a0 con 0x7f071815ded0 2024-02-19T15:27:21.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.195+0000 7f0717e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f071815ded0 0x7f0718066cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:21.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.195+0000 7f0717e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f071815ded0 0x7f0718066cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56852/0 (socket says 172.21.15.47:56852) 2024-02-19T15:27:21.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.195+0000 7f070f400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f07180b1280 0x7f071806aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49082/0 (socket says 172.21.15.47:49082) 2024-02-19T15:27:21.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.195+0000 7f0717e00640 1 -- 172.21.15.47:0/2897911968 learned_addr learned my addr 172.21.15.47:0/2897911968 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:21.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.199+0000 7f0717e00640 1 -- 172.21.15.47:0/2897911968 >> v1:172.21.15.116:6789/0 conn(0x7f071806b5c0 legacy=0x7f0718167c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:21.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.199+0000 7f0717e00640 1 -- 172.21.15.47:0/2897911968 >> v1:172.21.15.47:6789/0 conn(0x7f07180b1280 legacy=0x7f071806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:21.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.199+0000 7f0717e00640 1 -- 172.21.15.47:0/2897911968 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07180b0a80 con 0x7f071815ded0 2024-02-19T15:27:21.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.199+0000 7f0717400640 1 -- 172.21.15.47:0/2897911968 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062855417 0 0) 0x7f0718067210 con 0x7f07180b1280 2024-02-19T15:27:21.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.199+0000 7f0717e00640 1 --2- 172.21.15.47:0/2897911968 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f071815ded0 0x7f0718066cd0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f0708009e50 tx=0x7f070801ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=133ee6e04053f3ca server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:21.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.199+0000 7f0717400640 1 -- 172.21.15.47:0/2897911968 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07080a58f0 con 0x7f071815ded0 2024-02-19T15:27:21.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.199+0000 7f0717400640 1 -- 172.21.15.47:0/2897911968 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07080a5a50 con 0x7f071815ded0 2024-02-19T15:27:21.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.199+0000 7f0717400640 1 -- 172.21.15.47:0/2897911968 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07080a5de0 con 0x7f071815ded0 2024-02-19T15:27:21.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.199+0000 7f071ee00640 1 -- 172.21.15.47:0/2897911968 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f071815ded0 msgr2=0x7f0718066cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:21.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.199+0000 7f071ee00640 1 --2- 172.21.15.47:0/2897911968 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f071815ded0 0x7f0718066cd0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f0708009e50 tx=0x7f070801ca00 comp rx=0 tx=0).stop 2024-02-19T15:27:21.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.199+0000 7f071ee00640 1 -- 172.21.15.47:0/2897911968 shutdown_connections 2024-02-19T15:27:21.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.199+0000 7f071ee00640 1 --2- 172.21.15.47:0/2897911968 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f071815ded0 0x7f0718066cd0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:21.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.199+0000 7f071ee00640 1 -- 172.21.15.47:0/2897911968 >> 172.21.15.47:0/2897911968 conn(0x7f07180b32f0 msgr2=0x7f07180b36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:21.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.199+0000 7f071ee00640 1 -- 172.21.15.47:0/2897911968 shutdown_connections 2024-02-19T15:27:21.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f071ee00640 1 -- 172.21.15.47:0/2897911968 wait complete. 2024-02-19T15:27:21.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f071ee00640 1 Processor -- start 2024-02-19T15:27:21.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f071ee00640 1 -- start start 2024-02-19T15:27:21.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f071ee00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f071806b5c0 0x7f071815d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:21.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f071ee00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07180b1280 0x7f0718094d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:21.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f071ee00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f071815ded0 0x7f0718154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:21.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f071ee00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07180952a0 con 0x7f071806b5c0 2024-02-19T15:27:21.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f071ee00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071815c2c0 con 0x7f07180b1280 2024-02-19T15:27:21.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f071ee00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f071815c5a0 con 0x7f071815ded0 2024-02-19T15:27:21.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f0717e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f071806b5c0 0x7f071815d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:21.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f0717e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f071806b5c0 0x7f071815d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56854/0 (socket says 172.21.15.47:56854) 2024-02-19T15:27:21.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f0717e00640 1 -- 172.21.15.47:0/4164836204 learned_addr learned my addr 172.21.15.47:0/4164836204 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:21.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f070f400640 1 --2- 172.21.15.47:0/4164836204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07180b1280 0x7f0718094d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:21.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f071ce00640 1 --2- 172.21.15.47:0/4164836204 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f071815ded0 0x7f0718154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:21.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f0717e00640 1 -- 172.21.15.47:0/4164836204 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f071815ded0 msgr2=0x7f0718154920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:21.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f0717e00640 1 --2- 172.21.15.47:0/4164836204 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f071815ded0 0x7f0718154920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:21.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f0717e00640 1 -- 172.21.15.47:0/4164836204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07180b1280 msgr2=0x7f0718094d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:21.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f0717e00640 1 --2- 172.21.15.47:0/4164836204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07180b1280 0x7f0718094d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:21.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f0717e00640 1 -- 172.21.15.47:0/4164836204 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0708009b30 con 0x7f071806b5c0 2024-02-19T15:27:21.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f071ce00640 1 --2- 172.21.15.47:0/4164836204 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f071815ded0 0x7f0718154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:21.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f070f400640 1 --2- 172.21.15.47:0/4164836204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07180b1280 0x7f0718094d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:21.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f0717e00640 1 --2- 172.21.15.47:0/4164836204 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f071806b5c0 0x7f071815d360 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f0708019610 tx=0x7f07080197a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:21.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f0714c00640 1 -- 172.21.15.47:0/4164836204 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07080bd070 con 0x7f071806b5c0 2024-02-19T15:27:21.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f0714c00640 1 -- 172.21.15.47:0/4164836204 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07080b8680 con 0x7f071806b5c0 2024-02-19T15:27:21.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f0714c00640 1 -- 172.21.15.47:0/4164836204 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07080b8a50 con 0x7f071806b5c0 2024-02-19T15:27:21.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.203+0000 7f071ee00640 1 -- 172.21.15.47:0/4164836204 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f071815c880 con 0x7f071806b5c0 2024-02-19T15:27:21.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.207+0000 7f071ee00640 1 -- 172.21.15.47:0/4164836204 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f071815ce40 con 0x7f071806b5c0 2024-02-19T15:27:21.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.207+0000 7f071ee00640 1 -- 172.21.15.47:0/4164836204 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0718075360 con 0x7f071806b5c0 2024-02-19T15:27:21.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.207+0000 7f0714c00640 1 -- 172.21.15.47:0/4164836204 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f07080b8bb0 con 0x7f071806b5c0 2024-02-19T15:27:21.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.207+0000 7f0714c00640 1 --2- 172.21.15.47:0/4164836204 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f06f4066e40 0x7f06f4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:21.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.207+0000 7f0714c00640 1 -- 172.21.15.47:0/4164836204 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f07081202c0 con 0x7f071806b5c0 2024-02-19T15:27:21.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.207+0000 7f070f400640 1 --2- 172.21.15.47:0/4164836204 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f06f4066e40 0x7f06f4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:21.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.211+0000 7f070f400640 1 --2- 172.21.15.47:0/4164836204 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f06f4066e40 0x7f06f4069300 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f0700000f70 tx=0x7f0700009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:21.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.211+0000 7f0714c00640 1 -- 172.21.15.47:0/4164836204 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f07080ed0f0 con 0x7f071806b5c0 2024-02-19T15:27:21.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:21.407+0000 7f071ee00640 1 -- 172.21.15.47:0/4164836204 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f071815c6e0 con 0x7f071806b5c0 2024-02-19T15:27:22.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.019+0000 7f0714c00640 1 -- 172.21.15.47:0/4164836204 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v71) v1 ==== 174+0+0 (secure 0 0 0) 0x7f071815c6e0 con 0x7f071806b5c0 2024-02-19T15:27:22.023 INFO:teuthology.orchestra.run.smithi047.stderr:new fs with metadata pool 16 and data pool 17 2024-02-19T15:27:22.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.019+0000 7f071ee00640 1 -- 172.21.15.47:0/4164836204 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f06f4066e40 msgr2=0x7f06f4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:22.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.019+0000 7f071ee00640 1 --2- 172.21.15.47:0/4164836204 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f06f4066e40 0x7f06f4069300 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f0700000f70 tx=0x7f0700009040 comp rx=0 tx=0).stop 2024-02-19T15:27:22.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.019+0000 7f071ee00640 1 -- 172.21.15.47:0/4164836204 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f071806b5c0 msgr2=0x7f071815d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:22.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.019+0000 7f071ee00640 1 --2- 172.21.15.47:0/4164836204 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f071806b5c0 0x7f071815d360 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f0708019610 tx=0x7f07080197a0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.023+0000 7f071ee00640 1 -- 172.21.15.47:0/4164836204 shutdown_connections 2024-02-19T15:27:22.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.023+0000 7f071ee00640 1 --2- 172.21.15.47:0/4164836204 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f06f4066e40 0x7f06f4069300 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.023+0000 7f071ee00640 1 --2- 172.21.15.47:0/4164836204 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f071815ded0 0x7f0718154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.023+0000 7f071ee00640 1 --2- 172.21.15.47:0/4164836204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07180b1280 0x7f0718094d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.023+0000 7f071ee00640 1 --2- 172.21.15.47:0/4164836204 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f071806b5c0 0x7f071815d360 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.023+0000 7f071ee00640 1 -- 172.21.15.47:0/4164836204 >> 172.21.15.47:0/4164836204 conn(0x7f07180b32f0 msgr2=0x7f071806d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:22.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.023+0000 7f071ee00640 1 -- 172.21.15.47:0/4164836204 shutdown_connections 2024-02-19T15:27:22.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.023+0000 7f071ee00640 1 -- 172.21.15.47:0/4164836204 wait complete. 2024-02-19T15:27:22.083 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:27:22.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.219+0000 7faf9be00640 1 Processor -- start 2024-02-19T15:27:22.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.219+0000 7faf9be00640 1 -- start start 2024-02-19T15:27:22.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.223+0000 7faf9be00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf94071350 0x7faf94165e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:22.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.223+0000 7faf9be00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf940718c0 con 0x7faf9415b1a0 2024-02-19T15:27:22.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.223+0000 7faf9be00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf94071ba0 con 0x7faf9415d6e0 2024-02-19T15:27:22.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.223+0000 7faf9be00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf94071e80 con 0x7faf94071350 2024-02-19T15:27:22.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.223+0000 7faf99400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7faf9415b1a0 0x7faf9415b580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49090/0 (socket says 172.21.15.47:49090) 2024-02-19T15:27:22.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.223+0000 7faf99400640 1 -- 172.21.15.47:0/614986293 learned_addr learned my addr 172.21.15.47:0/614986293 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:22.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.223+0000 7faf99e00640 1 --2- 172.21.15.47:0/614986293 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf94071350 0x7faf94165e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:22.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.223+0000 7faf93e00640 1 -- 172.21.15.47:0/614986293 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109926167 0 0) 0x7faf940718c0 con 0x7faf9415b1a0 2024-02-19T15:27:22.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.223+0000 7faf93e00640 1 -- 172.21.15.47:0/614986293 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf7c003530 con 0x7faf9415b1a0 2024-02-19T15:27:22.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.223+0000 7faf93e00640 1 -- 172.21.15.47:0/614986293 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955560548 0 0) 0x7faf94071ba0 con 0x7faf9415d6e0 2024-02-19T15:27:22.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.223+0000 7faf93e00640 1 -- 172.21.15.47:0/614986293 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf940718c0 con 0x7faf9415d6e0 2024-02-19T15:27:22.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.223+0000 7faf99e00640 1 -- 172.21.15.47:0/614986293 >> v1:172.21.15.116:6789/0 conn(0x7faf9415d6e0 legacy=0x7faf94162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:22.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.223+0000 7faf99e00640 1 -- 172.21.15.47:0/614986293 >> v1:172.21.15.47:6789/0 conn(0x7faf9415b1a0 legacy=0x7faf9415b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:22.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.223+0000 7faf99e00640 1 -- 172.21.15.47:0/614986293 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf94072160 con 0x7faf94071350 2024-02-19T15:27:22.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.227+0000 7faf99e00640 1 --2- 172.21.15.47:0/614986293 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf94071350 0x7faf94165e70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7faf8c009df0 tx=0x7faf8c01c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2263da4d099327d5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:22.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.227+0000 7faf93e00640 1 -- 172.21.15.47:0/614986293 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8c0a5a70 con 0x7faf94071350 2024-02-19T15:27:22.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.227+0000 7faf93e00640 1 -- 172.21.15.47:0/614986293 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8c0a5bd0 con 0x7faf94071350 2024-02-19T15:27:22.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.227+0000 7faf93e00640 1 -- 172.21.15.47:0/614986293 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8c0ad440 con 0x7faf94071350 2024-02-19T15:27:22.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.227+0000 7faf9be00640 1 -- 172.21.15.47:0/614986293 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf94071350 msgr2=0x7faf94165e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:22.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.227+0000 7faf9be00640 1 --2- 172.21.15.47:0/614986293 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf94071350 0x7faf94165e70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7faf8c009df0 tx=0x7faf8c01c990 comp rx=0 tx=0).stop 2024-02-19T15:27:22.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.227+0000 7faf9be00640 1 -- 172.21.15.47:0/614986293 shutdown_connections 2024-02-19T15:27:22.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.227+0000 7faf9be00640 1 --2- 172.21.15.47:0/614986293 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf94071350 0x7faf94165e70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.227+0000 7faf9be00640 1 -- 172.21.15.47:0/614986293 >> 172.21.15.47:0/614986293 conn(0x7faf940b16a0 msgr2=0x7faf940b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:22.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.227+0000 7faf9be00640 1 -- 172.21.15.47:0/614986293 shutdown_connections 2024-02-19T15:27:22.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.227+0000 7faf9be00640 1 -- 172.21.15.47:0/614986293 wait complete. 2024-02-19T15:27:22.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.227+0000 7faf9be00640 1 Processor -- start 2024-02-19T15:27:22.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.227+0000 7faf9be00640 1 -- start start 2024-02-19T15:27:22.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.227+0000 7faf9be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf94071350 0x7faf9414b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:22.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf9be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf9415b1a0 0x7faf9414fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:22.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf9be00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf9415d6e0 0x7faf9414d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:22.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf9be00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf940ccec0 con 0x7faf94071350 2024-02-19T15:27:22.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf9be00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9414bea0 con 0x7faf9415b1a0 2024-02-19T15:27:22.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf9be00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf94150090 con 0x7faf9415d6e0 2024-02-19T15:27:22.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf99400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf94071350 0x7faf9414b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:22.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf99400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf94071350 0x7faf9414b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56860/0 (socket says 172.21.15.47:56860) 2024-02-19T15:27:22.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf99400640 1 -- 172.21.15.47:0/2721605259 learned_addr learned my addr 172.21.15.47:0/2721605259 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:22.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf99e00640 1 --2- 172.21.15.47:0/2721605259 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf9415d6e0 0x7faf9414d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:22.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf98a00640 1 --2- 172.21.15.47:0/2721605259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf9415b1a0 0x7faf9414fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:22.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf99400640 1 -- 172.21.15.47:0/2721605259 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf9415d6e0 msgr2=0x7faf9414d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:22.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf99400640 1 --2- 172.21.15.47:0/2721605259 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf9415d6e0 0x7faf9414d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf99400640 1 -- 172.21.15.47:0/2721605259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf9415b1a0 msgr2=0x7faf9414fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:22.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf99400640 1 --2- 172.21.15.47:0/2721605259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf9415b1a0 0x7faf9414fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf99400640 1 -- 172.21.15.47:0/2721605259 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8c009a70 con 0x7faf94071350 2024-02-19T15:27:22.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf98a00640 1 --2- 172.21.15.47:0/2721605259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf9415b1a0 0x7faf9414fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:27:22.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf99400640 1 --2- 172.21.15.47:0/2721605259 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf94071350 0x7faf9414b960 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7faf8800c500 tx=0x7faf8800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:22.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf91600640 1 -- 172.21.15.47:0/2721605259 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf88005cf0 con 0x7faf94071350 2024-02-19T15:27:22.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf91600640 1 -- 172.21.15.47:0/2721605259 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf88005e50 con 0x7faf94071350 2024-02-19T15:27:22.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf91600640 1 -- 172.21.15.47:0/2721605259 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf88007330 con 0x7faf94071350 2024-02-19T15:27:22.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf9be00640 1 -- 172.21.15.47:0/2721605259 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf9414d710 con 0x7faf94071350 2024-02-19T15:27:22.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf9be00640 1 -- 172.21.15.47:0/2721605259 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf940cd3b0 con 0x7faf94071350 2024-02-19T15:27:22.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf9be00640 1 -- 172.21.15.47:0/2721605259 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf94075360 con 0x7faf94071350 2024-02-19T15:27:22.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.231+0000 7faf91600640 1 -- 172.21.15.47:0/2721605259 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7faf88007490 con 0x7faf94071350 2024-02-19T15:27:22.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.235+0000 7faf91600640 1 --2- 172.21.15.47:0/2721605259 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faf74066e40 0x7faf74069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:22.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.235+0000 7faf91600640 1 -- 172.21.15.47:0/2721605259 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faf88084060 con 0x7faf94071350 2024-02-19T15:27:22.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.235+0000 7faf98a00640 1 --2- 172.21.15.47:0/2721605259 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faf74066e40 0x7faf74069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:22.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.235+0000 7faf98a00640 1 --2- 172.21.15.47:0/2721605259 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faf74066e40 0x7faf74069300 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7faf94150840 tx=0x7faf80009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:22.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.239+0000 7faf91600640 1 -- 172.21.15.47:0/2721605259 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7faf880b61c0 con 0x7faf94071350 2024-02-19T15:27:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.399+0000 7faf9be00640 1 -- 172.21.15.47:0/2721605259 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf940729c0 con 0x7faf94071350 2024-02-19T15:27:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.403+0000 7faf91600640 1 -- 172.21.15.47:0/2721605259 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16652 (secure 0 0 0) 0x7faf88017090 con 0x7faf94071350 2024-02-19T15:27:22.406 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:22.406 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":107,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:27:21.414960+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:27:19.839144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-02-19T15:27:20.405924+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:27:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.403+0000 7faf9be00640 1 -- 172.21.15.47:0/2721605259 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faf74066e40 msgr2=0x7faf74069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.403+0000 7faf9be00640 1 --2- 172.21.15.47:0/2721605259 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faf74066e40 0x7faf74069300 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7faf94150840 tx=0x7faf80009040 comp rx=0 tx=0).stop 2024-02-19T15:27:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.403+0000 7faf9be00640 1 -- 172.21.15.47:0/2721605259 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf94071350 msgr2=0x7faf9414b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.403+0000 7faf9be00640 1 --2- 172.21.15.47:0/2721605259 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf94071350 0x7faf9414b960 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7faf8800c500 tx=0x7faf8800c9c0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.403+0000 7faf9be00640 1 -- 172.21.15.47:0/2721605259 shutdown_connections 2024-02-19T15:27:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.403+0000 7faf9be00640 1 --2- 172.21.15.47:0/2721605259 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faf74066e40 0x7faf74069300 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.403+0000 7faf9be00640 1 --2- 172.21.15.47:0/2721605259 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf9415d6e0 0x7faf9414d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.403+0000 7faf9be00640 1 --2- 172.21.15.47:0/2721605259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf9415b1a0 0x7faf9414fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.403+0000 7faf9be00640 1 --2- 172.21.15.47:0/2721605259 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf94071350 0x7faf9414b960 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.403+0000 7faf9be00640 1 -- 172.21.15.47:0/2721605259 >> 172.21.15.47:0/2721605259 conn(0x7faf940b16a0 msgr2=0x7faf940d01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.403+0000 7faf9be00640 1 -- 172.21.15.47:0/2721605259 shutdown_connections 2024-02-19T15:27:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.403+0000 7faf9be00640 1 -- 172.21.15.47:0/2721605259 wait complete. 2024-02-19T15:27:22.428 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:27:22.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.563+0000 7f8277000640 1 Processor -- start 2024-02-19T15:27:22.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.563+0000 7f8277000640 1 -- start start 2024-02-19T15:27:22.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.563+0000 7f8277000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8270145590 0x7f82700727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:22.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.563+0000 7f8277000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f82700cea80 0x7f827006c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:22.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.563+0000 7f8277000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f827015c390 0x7f827015c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:22.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.563+0000 7f8277000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f827015d280 con 0x7f82700cea80 2024-02-19T15:27:22.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.563+0000 7f8277000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8270070ff0 con 0x7f827015c390 2024-02-19T15:27:22.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.563+0000 7f8277000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f827015cd60 con 0x7f8270145590 2024-02-19T15:27:22.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.563+0000 7f8267e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f82700cea80 0x7f827006c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:22.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.563+0000 7f8267e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f82700cea80 0x7f827006c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56864/0 (socket says 172.21.15.47:56864) 2024-02-19T15:27:22.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.563+0000 7f8267e00640 1 -- 172.21.15.47:0/1848584657 learned_addr learned my addr 172.21.15.47:0/1848584657 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:22.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.563+0000 7f826fe00640 1 --2- 172.21.15.47:0/1848584657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8270145590 0x7f82700727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:22.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.563+0000 7f8275000640 1 --2- 172.21.15.47:0/1848584657 >> v2:172.21.15.116:3300/0 conn(0x7f827015c390 0x7f827015c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:22.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.567+0000 7f8267e00640 1 -- 172.21.15.47:0/1848584657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8270145590 msgr2=0x7f82700727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:22.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.567+0000 7f8267e00640 1 --2- 172.21.15.47:0/1848584657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8270145590 0x7f82700727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.567+0000 7f8267e00640 1 -- 172.21.15.47:0/1848584657 >> v2:172.21.15.116:3300/0 conn(0x7f827015c390 msgr2=0x7f827015c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:22.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.567+0000 7f8267e00640 1 --2- 172.21.15.47:0/1848584657 >> v2:172.21.15.116:3300/0 conn(0x7f827015c390 0x7f827015c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.567+0000 7f8267e00640 1 -- 172.21.15.47:0/1848584657 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82700709e0 con 0x7f82700cea80 2024-02-19T15:27:22.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.567+0000 7f826fe00640 1 --2- 172.21.15.47:0/1848584657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8270145590 0x7f82700727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:27:22.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.567+0000 7f8267e00640 1 --2- 172.21.15.47:0/1848584657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82700cea80 0x7f827006c9a0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f8258009870 tx=0x7f825801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc8b997b2fe7c15 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:22.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.567+0000 7f826f400640 1 -- 172.21.15.47:0/1848584657 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82580ac630 con 0x7f82700cea80 2024-02-19T15:27:22.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.567+0000 7f826f400640 1 -- 172.21.15.47:0/1848584657 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82580ac790 con 0x7f82700cea80 2024-02-19T15:27:22.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.567+0000 7f826f400640 1 -- 172.21.15.47:0/1848584657 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82580acb20 con 0x7f82700cea80 2024-02-19T15:27:22.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.567+0000 7f8277000640 1 -- 172.21.15.47:0/1848584657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82700cea80 msgr2=0x7f827006c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:22.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.567+0000 7f8277000640 1 --2- 172.21.15.47:0/1848584657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82700cea80 0x7f827006c9a0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f8258009870 tx=0x7f825801c520 comp rx=0 tx=0).stop 2024-02-19T15:27:22.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.567+0000 7f8277000640 1 -- 172.21.15.47:0/1848584657 shutdown_connections 2024-02-19T15:27:22.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.567+0000 7f8277000640 1 --2- 172.21.15.47:0/1848584657 >> v2:172.21.15.116:3300/0 conn(0x7f827015c390 0x7f827015c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.567+0000 7f8277000640 1 --2- 172.21.15.47:0/1848584657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82700cea80 0x7f827006c9a0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.567+0000 7f8277000640 1 --2- 172.21.15.47:0/1848584657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8270145590 0x7f82700727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.567+0000 7f8277000640 1 -- 172.21.15.47:0/1848584657 >> 172.21.15.47:0/1848584657 conn(0x7f82700b16a0 msgr2=0x7f82700b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:22.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.567+0000 7f8277000640 1 -- 172.21.15.47:0/1848584657 shutdown_connections 2024-02-19T15:27:22.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.567+0000 7f8277000640 1 -- 172.21.15.47:0/1848584657 wait complete. 2024-02-19T15:27:22.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f8277000640 1 Processor -- start 2024-02-19T15:27:22.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f8277000640 1 -- start start 2024-02-19T15:27:22.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f8277000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82700cea80 0x7f827014d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:22.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f8277000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8270145590 0x7f82700b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:22.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f8277000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f827015c390 0x7f82700b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:22.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f8277000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8270140ed0 con 0x7f8270145590 2024-02-19T15:27:22.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f8277000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82700b48a0 con 0x7f82700cea80 2024-02-19T15:27:22.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f8277000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f827013f140 con 0x7f827015c390 2024-02-19T15:27:22.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f8267e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8270145590 0x7f82700b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:22.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f8267e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8270145590 0x7f82700b3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56868/0 (socket says 172.21.15.47:56868) 2024-02-19T15:27:22.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f8267e00640 1 -- 172.21.15.47:0/194270003 learned_addr learned my addr 172.21.15.47:0/194270003 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:22.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f826fe00640 1 --2- 172.21.15.47:0/194270003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82700cea80 0x7f827014d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:22.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f8275000640 1 --2- 172.21.15.47:0/194270003 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f827015c390 0x7f82700b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:22.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f8267e00640 1 -- 172.21.15.47:0/194270003 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f827015c390 msgr2=0x7f82700b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:22.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f8267e00640 1 --2- 172.21.15.47:0/194270003 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f827015c390 0x7f82700b4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f8267e00640 1 -- 172.21.15.47:0/194270003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82700cea80 msgr2=0x7f827014d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:22.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f8267e00640 1 --2- 172.21.15.47:0/194270003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82700cea80 0x7f827014d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f8267e00640 1 -- 172.21.15.47:0/194270003 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82580094d0 con 0x7f8270145590 2024-02-19T15:27:22.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f8267e00640 1 --2- 172.21.15.47:0/194270003 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8270145590 0x7f82700b3ce0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f82580af040 tx=0x7f82580192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:22.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f826cc00640 1 -- 172.21.15.47:0/194270003 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82580b5850 con 0x7f8270145590 2024-02-19T15:27:22.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f826cc00640 1 -- 172.21.15.47:0/194270003 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82580b59b0 con 0x7f8270145590 2024-02-19T15:27:22.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f826cc00640 1 -- 172.21.15.47:0/194270003 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82580b5d60 con 0x7f8270145590 2024-02-19T15:27:22.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f8277000640 1 -- 172.21.15.47:0/194270003 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f827013f420 con 0x7f8270145590 2024-02-19T15:27:22.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.571+0000 7f8277000640 1 -- 172.21.15.47:0/194270003 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f827015a940 con 0x7f8270145590 2024-02-19T15:27:22.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.575+0000 7f8277000640 1 -- 172.21.15.47:0/194270003 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f827013efa0 con 0x7f8270145590 2024-02-19T15:27:22.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.575+0000 7f826cc00640 1 -- 172.21.15.47:0/194270003 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f82580b3070 con 0x7f8270145590 2024-02-19T15:27:22.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.575+0000 7f826cc00640 1 --2- 172.21.15.47:0/194270003 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f824c066e40 0x7f824c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:22.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.575+0000 7f826cc00640 1 -- 172.21.15.47:0/194270003 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f825811eff0 con 0x7f8270145590 2024-02-19T15:27:22.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.575+0000 7f826fe00640 1 --2- 172.21.15.47:0/194270003 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f824c066e40 0x7f824c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:22.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.579+0000 7f826fe00640 1 --2- 172.21.15.47:0/194270003 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f824c066e40 0x7f824c069300 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f826000c880 tx=0x7f8260008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:22.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.579+0000 7f826cc00640 1 -- 172.21.15.47:0/194270003 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f82580ebe70 con 0x7f8270145590 2024-02-19T15:27:22.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.747+0000 7f8277000640 1 -- 172.21.15.47:0/194270003 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f827013efa0 con 0x7f8270145590 2024-02-19T15:27:22.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.747+0000 7f826cc00640 1 -- 172.21.15.47:0/194270003 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16652 (secure 0 0 0) 0x7f827013efa0 con 0x7f8270145590 2024-02-19T15:27:22.751 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:22.752 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":107,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:27:21.414960+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:27:19.839144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-02-19T15:27:20.405924+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:27:22.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.747+0000 7f8277000640 1 -- 172.21.15.47:0/194270003 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f824c066e40 msgr2=0x7f824c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:22.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.747+0000 7f8277000640 1 --2- 172.21.15.47:0/194270003 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f824c066e40 0x7f824c069300 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f826000c880 tx=0x7f8260008040 comp rx=0 tx=0).stop 2024-02-19T15:27:22.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.747+0000 7f8277000640 1 -- 172.21.15.47:0/194270003 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8270145590 msgr2=0x7f82700b3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:22.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.747+0000 7f8277000640 1 --2- 172.21.15.47:0/194270003 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8270145590 0x7f82700b3ce0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f82580af040 tx=0x7f82580192c0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.747+0000 7f8277000640 1 -- 172.21.15.47:0/194270003 shutdown_connections 2024-02-19T15:27:22.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.747+0000 7f8277000640 1 --2- 172.21.15.47:0/194270003 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f824c066e40 0x7f824c069300 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.747+0000 7f8277000640 1 --2- 172.21.15.47:0/194270003 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f827015c390 0x7f82700b4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.747+0000 7f8277000640 1 --2- 172.21.15.47:0/194270003 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8270145590 0x7f82700b3ce0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.747+0000 7f8277000640 1 --2- 172.21.15.47:0/194270003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82700cea80 0x7f827014d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.747+0000 7f8277000640 1 -- 172.21.15.47:0/194270003 >> 172.21.15.47:0/194270003 conn(0x7f82700b16a0 msgr2=0x7f8270141af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:22.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.751+0000 7f8277000640 1 -- 172.21.15.47:0/194270003 shutdown_connections 2024-02-19T15:27:22.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.751+0000 7f8277000640 1 -- 172.21.15.47:0/194270003 wait complete. 2024-02-19T15:27:22.772 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T15:27:22.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.911+0000 7f14ee000640 1 Processor -- start 2024-02-19T15:27:22.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.911+0000 7f14ee000640 1 -- start start 2024-02-19T15:27:22.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.911+0000 7f14ee000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f14e806e360 0x7f14e80cb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:22.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.911+0000 7f14ee000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e80cb540 con 0x7f14e815af10 2024-02-19T15:27:22.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.911+0000 7f14ee000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e8071c70 con 0x7f14e80cd880 2024-02-19T15:27:22.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.911+0000 7f14ee000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e806cef0 con 0x7f14e806e360 2024-02-19T15:27:22.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.911+0000 7f14e7400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f14e815af10 0x7f14e80ca8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49096/0 (socket says 172.21.15.47:49096) 2024-02-19T15:27:22.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.911+0000 7f14e7400640 1 -- 172.21.15.47:0/754164433 learned_addr learned my addr 172.21.15.47:0/754164433 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:22.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14dfe00640 1 --2- 172.21.15.47:0/754164433 >> v2:172.21.15.116:3300/0 conn(0x7f14e806e360 0x7f14e80cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:22.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14e6a00640 1 -- 172.21.15.47:0/754164433 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774730537 0 0) 0x7f14e80cb540 con 0x7f14e815af10 2024-02-19T15:27:22.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14e6a00640 1 -- 172.21.15.47:0/754164433 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14cc003530 con 0x7f14e815af10 2024-02-19T15:27:22.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14e6a00640 1 -- 172.21.15.47:0/754164433 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1019741879 0 0) 0x7f14cc003530 con 0x7f14e815af10 2024-02-19T15:27:22.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14e6a00640 1 -- 172.21.15.47:0/754164433 >> v2:172.21.15.116:3300/0 conn(0x7f14e806e360 msgr2=0x7f14e80cb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:22.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14e6a00640 1 --2- 172.21.15.47:0/754164433 >> v2:172.21.15.116:3300/0 conn(0x7f14e806e360 0x7f14e80cb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14e6a00640 1 -- 172.21.15.47:0/754164433 >> v1:172.21.15.116:6789/0 conn(0x7f14e80cd880 legacy=0x7f14e80cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:22.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14e6a00640 1 -- 172.21.15.47:0/754164433 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e80ca610 con 0x7f14e815af10 2024-02-19T15:27:22.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14e6a00640 1 -- 172.21.15.47:0/754164433 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f14d8001680 con 0x7f14e815af10 2024-02-19T15:27:22.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14e6a00640 1 -- 172.21.15.47:0/754164433 >> v1:172.21.15.47:6789/0 conn(0x7f14e815af10 legacy=0x7f14e80ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:22.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14e6a00640 1 --2- 172.21.15.47:0/754164433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14cc003980 0x7f14cc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:22.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14e6a00640 1 --2- 172.21.15.47:0/754164433 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14cc0047d0 0x7f14cc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:22.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14e6a00640 1 --2- 172.21.15.47:0/754164433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14cc005190 0x7f14cc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:22.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14e6a00640 1 -- 172.21.15.47:0/754164433 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e806cef0 con 0x7f14cc0047d0 2024-02-19T15:27:22.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14e6a00640 1 -- 172.21.15.47:0/754164433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc00c130 con 0x7f14cc005190 2024-02-19T15:27:22.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14e6a00640 1 -- 172.21.15.47:0/754164433 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc00c2f0 con 0x7f14cc003980 2024-02-19T15:27:22.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14dfe00640 1 --2- 172.21.15.47:0/754164433 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14cc0047d0 0x7f14cc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:22.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14e7400640 1 --2- 172.21.15.47:0/754164433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14cc003980 0x7f14cc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:22.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14e7e00640 1 --2- 172.21.15.47:0/754164433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14cc005190 0x7f14cc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:22.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14dfe00640 1 -- 172.21.15.47:0/754164433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14cc003980 msgr2=0x7f14cc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:22.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14dfe00640 1 --2- 172.21.15.47:0/754164433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14cc003980 0x7f14cc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14dfe00640 1 -- 172.21.15.47:0/754164433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14cc005190 msgr2=0x7f14cc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:22.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14dfe00640 1 --2- 172.21.15.47:0/754164433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14cc005190 0x7f14cc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14dfe00640 1 -- 172.21.15.47:0/754164433 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14cc003530 con 0x7f14cc0047d0 2024-02-19T15:27:22.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14e7400640 1 --2- 172.21.15.47:0/754164433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14cc003980 0x7f14cc003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:27:22.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14e7e00640 1 --2- 172.21.15.47:0/754164433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14cc005190 0x7f14cc00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:27:22.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.915+0000 7f14e7e00640 1 -- 172.21.15.47:0/754164433 reap_dead start 2024-02-19T15:27:22.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.919+0000 7f14dfe00640 1 --2- 172.21.15.47:0/754164433 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14cc0047d0 0x7f14cc004c50 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f14d0009ae0 tx=0x7f14d001ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72afc18296d5751b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:22.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.919+0000 7f14e6a00640 1 -- 172.21.15.47:0/754164433 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d00a5610 con 0x7f14cc0047d0 2024-02-19T15:27:22.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.919+0000 7f14e6a00640 1 -- 172.21.15.47:0/754164433 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d00a5770 con 0x7f14cc0047d0 2024-02-19T15:27:22.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.919+0000 7f14e6a00640 1 -- 172.21.15.47:0/754164433 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d00a5b20 con 0x7f14cc0047d0 2024-02-19T15:27:22.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.919+0000 7f14ee000640 1 -- 172.21.15.47:0/754164433 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14cc0047d0 msgr2=0x7f14cc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:22.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.919+0000 7f14ee000640 1 --2- 172.21.15.47:0/754164433 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14cc0047d0 0x7f14cc004c50 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f14d0009ae0 tx=0x7f14d001ca40 comp rx=0 tx=0).stop 2024-02-19T15:27:22.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.919+0000 7f14ee000640 1 -- 172.21.15.47:0/754164433 shutdown_connections 2024-02-19T15:27:22.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.919+0000 7f14ee000640 1 --2- 172.21.15.47:0/754164433 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14cc0047d0 0x7f14cc004c50 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.919+0000 7f14ee000640 1 -- 172.21.15.47:0/754164433 >> 172.21.15.47:0/754164433 conn(0x7f14e80cfc20 msgr2=0x7f14e80d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:22.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.919+0000 7f14ee000640 1 -- 172.21.15.47:0/754164433 shutdown_connections 2024-02-19T15:27:22.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.919+0000 7f14ee000640 1 -- 172.21.15.47:0/754164433 wait complete. 2024-02-19T15:27:22.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.919+0000 7f14ee000640 1 Processor -- start 2024-02-19T15:27:22.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.919+0000 7f14ee000640 1 -- start start 2024-02-19T15:27:22.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.919+0000 7f14ee000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14cc0047d0 0x7f14e81fa3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:22.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.919+0000 7f14ee000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14e81f3b90 0x7f14e81f4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:22.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.919+0000 7f14ee000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14e80c8050 0x7f14e80c84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:22.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.919+0000 7f14ee000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e80ab5a0 con 0x7f14e80c8050 2024-02-19T15:27:22.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.919+0000 7f14ee000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e80b3660 con 0x7f14e81f3b90 2024-02-19T15:27:22.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.919+0000 7f14ee000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e81f4550 con 0x7f14cc0047d0 2024-02-19T15:27:22.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.923+0000 7f14e7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14e80c8050 0x7f14e80c84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:22.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.923+0000 7f14e7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14e80c8050 0x7f14e80c84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56876/0 (socket says 172.21.15.47:56876) 2024-02-19T15:27:22.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.923+0000 7f14e7e00640 1 -- 172.21.15.47:0/3450948113 learned_addr learned my addr 172.21.15.47:0/3450948113 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:22.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.923+0000 7f14e7400640 1 --2- 172.21.15.47:0/3450948113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14cc0047d0 0x7f14e81fa3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:22.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.923+0000 7f14dfe00640 1 --2- 172.21.15.47:0/3450948113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14e81f3b90 0x7f14e81f4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:22.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.923+0000 7f14e7e00640 1 -- 172.21.15.47:0/3450948113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14cc0047d0 msgr2=0x7f14e81fa3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:22.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.923+0000 7f14e7e00640 1 --2- 172.21.15.47:0/3450948113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14cc0047d0 0x7f14e81fa3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.923+0000 7f14e7e00640 1 -- 172.21.15.47:0/3450948113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14e81f3b90 msgr2=0x7f14e81f4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:22.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.923+0000 7f14e7e00640 1 --2- 172.21.15.47:0/3450948113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14e81f3b90 0x7f14e81f4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:22.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.923+0000 7f14e7e00640 1 -- 172.21.15.47:0/3450948113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e80c8d80 con 0x7f14e80c8050 2024-02-19T15:27:22.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.923+0000 7f14dfe00640 1 --2- 172.21.15.47:0/3450948113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14e81f3b90 0x7f14e81f4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:27:22.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.923+0000 7f14e7e00640 1 --2- 172.21.15.47:0/3450948113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14e80c8050 0x7f14e80c84d0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f14e815fb60 tx=0x7f14e000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:22.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.923+0000 7f14df400640 1 -- 172.21.15.47:0/3450948113 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e000eed0 con 0x7f14e80c8050 2024-02-19T15:27:22.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.923+0000 7f14df400640 1 -- 172.21.15.47:0/3450948113 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e0003980 con 0x7f14e80c8050 2024-02-19T15:27:22.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.923+0000 7f14ee000640 1 -- 172.21.15.47:0/3450948113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e80c9060 con 0x7f14e80c8050 2024-02-19T15:27:22.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.923+0000 7f14df400640 1 -- 172.21.15.47:0/3450948113 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e0003d10 con 0x7f14e80c8050 2024-02-19T15:27:22.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.923+0000 7f14ee000640 1 -- 172.21.15.47:0/3450948113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e80c9620 con 0x7f14e80c8050 2024-02-19T15:27:22.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.923+0000 7f14ee000640 1 -- 172.21.15.47:0/3450948113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e8070a00 con 0x7f14e80c8050 2024-02-19T15:27:22.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.923+0000 7f14df400640 1 -- 172.21.15.47:0/3450948113 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f14e00050d0 con 0x7f14e80c8050 2024-02-19T15:27:22.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.923+0000 7f14df400640 1 --2- 172.21.15.47:0/3450948113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f14c4066e40 0x7f14c4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:22.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.927+0000 7f14df400640 1 -- 172.21.15.47:0/3450948113 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14e00847b0 con 0x7f14e80c8050 2024-02-19T15:27:22.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.927+0000 7f14e7400640 1 --2- 172.21.15.47:0/3450948113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f14c4066e40 0x7f14c4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:22.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.927+0000 7f14e7400640 1 --2- 172.21.15.47:0/3450948113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f14c4066e40 0x7f14c4069300 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f14d8001ca0 tx=0x7f14d8007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:22.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:22.931+0000 7f14df400640 1 -- 172.21.15.47:0/3450948113 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f14e0051500 con 0x7f14e80c8050 2024-02-19T15:27:23.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:23.147+0000 7f14ee000640 1 -- 172.21.15.47:0/3450948113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f14e8075360 con 0x7f14e80c8050 2024-02-19T15:27:24.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.043+0000 7f14df400640 1 -- 172.21.15.47:0/3450948113 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v74) v1 ==== 118+0+0 (secure 0 0 0) 0x7f14e0054bc0 con 0x7f14e80c8050 2024-02-19T15:27:24.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.047+0000 7f14ee000640 1 -- 172.21.15.47:0/3450948113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f14c4066e40 msgr2=0x7f14c4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:24.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.047+0000 7f14ee000640 1 --2- 172.21.15.47:0/3450948113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f14c4066e40 0x7f14c4069300 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f14d8001ca0 tx=0x7f14d8007040 comp rx=0 tx=0).stop 2024-02-19T15:27:24.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.047+0000 7f14ee000640 1 -- 172.21.15.47:0/3450948113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14e80c8050 msgr2=0x7f14e80c84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:24.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.047+0000 7f14ee000640 1 --2- 172.21.15.47:0/3450948113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14e80c8050 0x7f14e80c84d0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f14e815fb60 tx=0x7f14e000eb90 comp rx=0 tx=0).stop 2024-02-19T15:27:24.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.047+0000 7f14ee000640 1 -- 172.21.15.47:0/3450948113 shutdown_connections 2024-02-19T15:27:24.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.047+0000 7f14ee000640 1 --2- 172.21.15.47:0/3450948113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f14c4066e40 0x7f14c4069300 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:24.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.047+0000 7f14ee000640 1 --2- 172.21.15.47:0/3450948113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f14e80c8050 0x7f14e80c84d0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:24.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.047+0000 7f14ee000640 1 --2- 172.21.15.47:0/3450948113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14e81f3b90 0x7f14e81f4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:24.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.047+0000 7f14ee000640 1 --2- 172.21.15.47:0/3450948113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14cc0047d0 0x7f14e81fa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:24.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.047+0000 7f14ee000640 1 -- 172.21.15.47:0/3450948113 >> 172.21.15.47:0/3450948113 conn(0x7f14e80cfc20 msgr2=0x7f14e815a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:24.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.047+0000 7f14ee000640 1 -- 172.21.15.47:0/3450948113 shutdown_connections 2024-02-19T15:27:24.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.047+0000 7f14ee000640 1 -- 172.21.15.47:0/3450948113 wait complete. 2024-02-19T15:27:24.070 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T15:27:24.070 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T15:27:24.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.211+0000 7fd75e400640 1 Processor -- start 2024-02-19T15:27:24.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.211+0000 7fd75e400640 1 -- start start 2024-02-19T15:27:24.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.211+0000 7fd75e400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd758158bc0 0x7fd75815be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:24.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.211+0000 7fd75e400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd7580cf770 0x7fd758072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:24.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.211+0000 7fd75e400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75815c380 con 0x7fd7580cf310 2024-02-19T15:27:24.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.211+0000 7fd75e400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd75815f8f0 con 0x7fd7580cf770 2024-02-19T15:27:24.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.211+0000 7fd75e400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd75806dad0 con 0x7fd758158bc0 2024-02-19T15:27:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.211+0000 7fd757e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd7580cf770 0x7fd758072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.211+0000 7fd757400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd758158bc0 0x7fd75815be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.211+0000 7fd757e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd7580cf770 0x7fd758072180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:38260/0 (socket says 172.21.15.47:38260) 2024-02-19T15:27:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.211+0000 7fd757e00640 1 -- 172.21.15.47:0/4067089579 learned_addr learned my addr 172.21.15.47:0/4067089579 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.211+0000 7fd757400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd758158bc0 0x7fd75815be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:52722/0 (socket says 172.21.15.47:52722) 2024-02-19T15:27:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.211+0000 7fd74fe00640 1 --1- 172.21.15.47:0/4067089579 >> v1:172.21.15.116:6789/0 conn(0x7fd7580cf310 0x7fd758071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:60084/0 (socket says 172.21.15.47:60084) 2024-02-19T15:27:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.211+0000 7fd757400640 1 -- 172.21.15.47:0/4067089579 >> v2:172.21.15.116:3300/0 conn(0x7fd7580cf770 msgr2=0x7fd758072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.211+0000 7fd757400640 1 --2- 172.21.15.47:0/4067089579 >> v2:172.21.15.116:3300/0 conn(0x7fd7580cf770 0x7fd758072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:24.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.211+0000 7fd757400640 1 -- 172.21.15.47:0/4067089579 >> v1:172.21.15.116:6789/0 conn(0x7fd7580cf310 legacy=0x7fd758071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:24.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.211+0000 7fd757400640 1 -- 172.21.15.47:0/4067089579 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75815ece0 con 0x7fd758158bc0 2024-02-19T15:27:24.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.211+0000 7fd757e00640 1 --2- 172.21.15.47:0/4067089579 >> v2:172.21.15.116:3300/0 conn(0x7fd7580cf770 0x7fd758072180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:27:24.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.215+0000 7fd757400640 1 --2- 172.21.15.47:0/4067089579 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd758158bc0 0x7fd75815be40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd7480097c0 tx=0x7fd74801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a6a4199e24b1e0a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.215+0000 7fd756a00640 1 -- 172.21.15.47:0/4067089579 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7480ac640 con 0x7fd758158bc0 2024-02-19T15:27:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.215+0000 7fd756a00640 1 -- 172.21.15.47:0/4067089579 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7480ac7a0 con 0x7fd758158bc0 2024-02-19T15:27:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.215+0000 7fd756a00640 1 -- 172.21.15.47:0/4067089579 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7480acb30 con 0x7fd758158bc0 2024-02-19T15:27:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.215+0000 7fd75e400640 1 -- 172.21.15.47:0/4067089579 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd758158bc0 msgr2=0x7fd75815be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.215+0000 7fd75e400640 1 --2- 172.21.15.47:0/4067089579 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd758158bc0 0x7fd75815be40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd7480097c0 tx=0x7fd74801c3e0 comp rx=0 tx=0).stop 2024-02-19T15:27:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.215+0000 7fd75e400640 1 -- 172.21.15.47:0/4067089579 shutdown_connections 2024-02-19T15:27:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.215+0000 7fd75e400640 1 --2- 172.21.15.47:0/4067089579 >> v2:172.21.15.116:3300/0 conn(0x7fd7580cf770 0x7fd758072180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.215+0000 7fd75e400640 1 --2- 172.21.15.47:0/4067089579 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd758158bc0 0x7fd75815be40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.215+0000 7fd75e400640 1 -- 172.21.15.47:0/4067089579 >> 172.21.15.47:0/4067089579 conn(0x7fd7580d0d30 msgr2=0x7fd7580d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.215+0000 7fd75e400640 1 -- 172.21.15.47:0/4067089579 shutdown_connections 2024-02-19T15:27:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.215+0000 7fd75e400640 1 -- 172.21.15.47:0/4067089579 wait complete. 2024-02-19T15:27:24.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.215+0000 7fd75e400640 1 Processor -- start 2024-02-19T15:27:24.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd75e400640 1 -- start start 2024-02-19T15:27:24.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd75e400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7580cf310 0x7fd758149d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:24.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd75e400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd7580cf770 0x7fd7580c87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:24.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd75e400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd758158bc0 0x7fd75815d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:24.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd75e400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7581594f0 con 0x7fd7580cf310 2024-02-19T15:27:24.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd75e400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7580c92b0 con 0x7fd758158bc0 2024-02-19T15:27:24.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd75e400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7580c9590 con 0x7fd7580cf770 2024-02-19T15:27:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd757400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7580cf310 0x7fd758149d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd757400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7580cf310 0x7fd758149d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55238/0 (socket says 172.21.15.47:55238) 2024-02-19T15:27:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd757400640 1 -- 172.21.15.47:0/3011367395 learned_addr learned my addr 172.21.15.47:0/3011367395 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd74fe00640 1 --2- 172.21.15.47:0/3011367395 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd7580cf770 0x7fd7580c87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd757e00640 1 --2- 172.21.15.47:0/3011367395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd758158bc0 0x7fd75815d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd757400640 1 -- 172.21.15.47:0/3011367395 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd7580cf770 msgr2=0x7fd7580c87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd757400640 1 --2- 172.21.15.47:0/3011367395 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd7580cf770 0x7fd7580c87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd757400640 1 -- 172.21.15.47:0/3011367395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd758158bc0 msgr2=0x7fd75815d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd757400640 1 --2- 172.21.15.47:0/3011367395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd758158bc0 0x7fd75815d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:24.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd757400640 1 -- 172.21.15.47:0/3011367395 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd750009b30 con 0x7fd7580cf310 2024-02-19T15:27:24.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd757400640 1 --2- 172.21.15.47:0/3011367395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7580cf310 0x7fd758149d00 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fd74801aa90 tx=0x7fd748019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:24.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd74f400640 1 -- 172.21.15.47:0/3011367395 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7480b6670 con 0x7fd7580cf310 2024-02-19T15:27:24.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd75e400640 1 -- 172.21.15.47:0/3011367395 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd748009420 con 0x7fd7580cf310 2024-02-19T15:27:24.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd74f400640 1 -- 172.21.15.47:0/3011367395 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7480b67d0 con 0x7fd7580cf310 2024-02-19T15:27:24.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd74f400640 1 -- 172.21.15.47:0/3011367395 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7480b6b80 con 0x7fd7580cf310 2024-02-19T15:27:24.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd75e400640 1 -- 172.21.15.47:0/3011367395 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7580c9b50 con 0x7fd7580cf310 2024-02-19T15:27:24.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.219+0000 7fd75e400640 1 -- 172.21.15.47:0/3011367395 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd758067a10 con 0x7fd7580cf310 2024-02-19T15:27:24.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.223+0000 7fd74f400640 1 -- 172.21.15.47:0/3011367395 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd7480b3040 con 0x7fd7580cf310 2024-02-19T15:27:24.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.223+0000 7fd74f400640 1 --2- 172.21.15.47:0/3011367395 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd734066e40 0x7fd734069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:24.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.223+0000 7fd74f400640 1 -- 172.21.15.47:0/3011367395 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd74811fcf0 con 0x7fd7580cf310 2024-02-19T15:27:24.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.223+0000 7fd74fe00640 1 --2- 172.21.15.47:0/3011367395 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd734066e40 0x7fd734069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:24.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.223+0000 7fd74fe00640 1 --2- 172.21.15.47:0/3011367395 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd734066e40 0x7fd734069300 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fd7580c6310 tx=0x7fd740007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:24.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.227+0000 7fd74f400640 1 -- 172.21.15.47:0/3011367395 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd7480eccb0 con 0x7fd7580cf310 2024-02-19T15:27:24.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:24.423+0000 7fd75e400640 1 -- 172.21.15.47:0/3011367395 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd7580cd5b0 con 0x7fd7580cf310 2024-02-19T15:27:25.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.043+0000 7fd74f400640 1 -- 172.21.15.47:0/3011367395 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v75) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd7480f0370 con 0x7fd7580cf310 2024-02-19T15:27:25.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.047+0000 7fd75e400640 1 -- 172.21.15.47:0/3011367395 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd734066e40 msgr2=0x7fd734069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:25.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.047+0000 7fd75e400640 1 --2- 172.21.15.47:0/3011367395 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd734066e40 0x7fd734069300 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fd7580c6310 tx=0x7fd740007e90 comp rx=0 tx=0).stop 2024-02-19T15:27:25.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.047+0000 7fd75e400640 1 -- 172.21.15.47:0/3011367395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7580cf310 msgr2=0x7fd758149d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:25.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.047+0000 7fd75e400640 1 --2- 172.21.15.47:0/3011367395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7580cf310 0x7fd758149d00 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fd74801aa90 tx=0x7fd748019180 comp rx=0 tx=0).stop 2024-02-19T15:27:25.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.047+0000 7fd75e400640 1 -- 172.21.15.47:0/3011367395 shutdown_connections 2024-02-19T15:27:25.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.047+0000 7fd75e400640 1 --2- 172.21.15.47:0/3011367395 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd734066e40 0x7fd734069300 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.047+0000 7fd75e400640 1 --2- 172.21.15.47:0/3011367395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd758158bc0 0x7fd75815d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.047+0000 7fd75e400640 1 --2- 172.21.15.47:0/3011367395 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd7580cf770 0x7fd7580c87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.047+0000 7fd75e400640 1 --2- 172.21.15.47:0/3011367395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7580cf310 0x7fd758149d00 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.047+0000 7fd75e400640 1 -- 172.21.15.47:0/3011367395 >> 172.21.15.47:0/3011367395 conn(0x7fd7580d0d30 msgr2=0x7fd758157bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:25.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.047+0000 7fd75e400640 1 -- 172.21.15.47:0/3011367395 shutdown_connections 2024-02-19T15:27:25.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.047+0000 7fd75e400640 1 -- 172.21.15.47:0/3011367395 wait complete. 2024-02-19T15:27:25.072 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T15:27:25.072 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:27:25.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.207+0000 7f7d36e00640 1 Processor -- start 2024-02-19T15:27:25.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.207+0000 7f7d36e00640 1 -- start start 2024-02-19T15:27:25.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.211+0000 7f7d36e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7d3015dd70 0x7f7d300ae860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:25.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.211+0000 7f7d36e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7d300aeda0 0x7f7d3006d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:25.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.211+0000 7f7d36e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d3006d660 0x7f7d300d19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:25.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.211+0000 7f7d36e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d3015f3f0 con 0x7f7d300aeda0 2024-02-19T15:27:25.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.211+0000 7f7d36e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d3006dab0 con 0x7f7d3015dd70 2024-02-19T15:27:25.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.211+0000 7f7d36e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3006c3e0 con 0x7f7d3006d660 2024-02-19T15:27:25.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.211+0000 7f7d35000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7d300aeda0 0x7f7d3006d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:25.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.211+0000 7f7d35000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7d300aeda0 0x7f7d3006d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55252/0 (socket says 172.21.15.47:55252) 2024-02-19T15:27:25.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.211+0000 7f7d35000640 1 -- 172.21.15.47:0/3667960625 learned_addr learned my addr 172.21.15.47:0/3667960625 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:25.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.211+0000 7f7d35a00640 1 --2- 172.21.15.47:0/3667960625 >> v2:172.21.15.116:3300/0 conn(0x7f7d3015dd70 0x7f7d300ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:25.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.211+0000 7f7d36400640 1 --2- 172.21.15.47:0/3667960625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d3006d660 0x7f7d300d19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:25.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.211+0000 7f7d35000640 1 -- 172.21.15.47:0/3667960625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d3006d660 msgr2=0x7f7d300d19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:25.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.211+0000 7f7d35000640 1 --2- 172.21.15.47:0/3667960625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d3006d660 0x7f7d300d19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.211+0000 7f7d35000640 1 -- 172.21.15.47:0/3667960625 >> v2:172.21.15.116:3300/0 conn(0x7f7d3015dd70 msgr2=0x7f7d300ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:25.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.211+0000 7f7d35000640 1 --2- 172.21.15.47:0/3667960625 >> v2:172.21.15.116:3300/0 conn(0x7f7d3015dd70 0x7f7d300ae860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.211+0000 7f7d35000640 1 -- 172.21.15.47:0/3667960625 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3006c680 con 0x7f7d300aeda0 2024-02-19T15:27:25.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.211+0000 7f7d35a00640 1 --2- 172.21.15.47:0/3667960625 >> v2:172.21.15.116:3300/0 conn(0x7f7d3015dd70 0x7f7d300ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:25.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.211+0000 7f7d36400640 1 --2- 172.21.15.47:0/3667960625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d3006d660 0x7f7d300d19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:25.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d35000640 1 --2- 172.21.15.47:0/3667960625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d300aeda0 0x7f7d3006d120 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f7d3006e880 tx=0x7f7d1c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=290e710e420808d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:25.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d2fe00640 1 -- 172.21.15.47:0/3667960625 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c0ab600 con 0x7f7d300aeda0 2024-02-19T15:27:25.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d2fe00640 1 -- 172.21.15.47:0/3667960625 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1c0a7050 con 0x7f7d300aeda0 2024-02-19T15:27:25.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d2fe00640 1 -- 172.21.15.47:0/3667960625 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c0ab980 con 0x7f7d300aeda0 2024-02-19T15:27:25.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d36e00640 1 -- 172.21.15.47:0/3667960625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d300aeda0 msgr2=0x7f7d3006d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:25.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d36e00640 1 --2- 172.21.15.47:0/3667960625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d300aeda0 0x7f7d3006d120 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f7d3006e880 tx=0x7f7d1c01c460 comp rx=0 tx=0).stop 2024-02-19T15:27:25.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d36e00640 1 -- 172.21.15.47:0/3667960625 shutdown_connections 2024-02-19T15:27:25.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d36e00640 1 --2- 172.21.15.47:0/3667960625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d3006d660 0x7f7d300d19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d36e00640 1 --2- 172.21.15.47:0/3667960625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d300aeda0 0x7f7d3006d120 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d36e00640 1 --2- 172.21.15.47:0/3667960625 >> v2:172.21.15.116:3300/0 conn(0x7f7d3015dd70 0x7f7d300ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d36e00640 1 -- 172.21.15.47:0/3667960625 >> 172.21.15.47:0/3667960625 conn(0x7f7d3009a9d0 msgr2=0x7f7d3009ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:25.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d36e00640 1 -- 172.21.15.47:0/3667960625 shutdown_connections 2024-02-19T15:27:25.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d36e00640 1 -- 172.21.15.47:0/3667960625 wait complete. 2024-02-19T15:27:25.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d36e00640 1 Processor -- start 2024-02-19T15:27:25.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d36e00640 1 -- start start 2024-02-19T15:27:25.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d36e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d3006d660 0x7f7d30159680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:25.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d36e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d300aeda0 0x7f7d30161a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:25.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d36e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d3015dd70 0x7f7d30161f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:25.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d36e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3015b170 con 0x7f7d300aeda0 2024-02-19T15:27:25.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d36e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d30159bc0 con 0x7f7d3015dd70 2024-02-19T15:27:25.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.215+0000 7f7d36e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3015abb0 con 0x7f7d3006d660 2024-02-19T15:27:25.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.219+0000 7f7d35000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d300aeda0 0x7f7d30161a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:25.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.219+0000 7f7d35a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d3006d660 0x7f7d30159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:25.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.219+0000 7f7d35000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d300aeda0 0x7f7d30161a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55264/0 (socket says 172.21.15.47:55264) 2024-02-19T15:27:25.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.219+0000 7f7d35000640 1 -- 172.21.15.47:0/2782638938 learned_addr learned my addr 172.21.15.47:0/2782638938 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:25.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.219+0000 7f7d36400640 1 --2- 172.21.15.47:0/2782638938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d3015dd70 0x7f7d30161f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:25.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.219+0000 7f7d35000640 1 -- 172.21.15.47:0/2782638938 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d3006d660 msgr2=0x7f7d30159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:25.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.219+0000 7f7d35000640 1 --2- 172.21.15.47:0/2782638938 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d3006d660 0x7f7d30159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.219+0000 7f7d35000640 1 -- 172.21.15.47:0/2782638938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d3015dd70 msgr2=0x7f7d30161f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:25.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.219+0000 7f7d35000640 1 --2- 172.21.15.47:0/2782638938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d3015dd70 0x7f7d30161f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.219+0000 7f7d35000640 1 -- 172.21.15.47:0/2782638938 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1c0094d0 con 0x7f7d300aeda0 2024-02-19T15:27:25.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.219+0000 7f7d35000640 1 --2- 172.21.15.47:0/2782638938 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d300aeda0 0x7f7d30161a30 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c0ae040 tx=0x7f7d1c019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:25.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.219+0000 7f7d2d600640 1 -- 172.21.15.47:0/2782638938 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c0b4640 con 0x7f7d300aeda0 2024-02-19T15:27:25.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.219+0000 7f7d2d600640 1 -- 172.21.15.47:0/2782638938 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1c0b47c0 con 0x7f7d300aeda0 2024-02-19T15:27:25.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.219+0000 7f7d36e00640 1 -- 172.21.15.47:0/2782638938 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3015ae90 con 0x7f7d300aeda0 2024-02-19T15:27:25.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.219+0000 7f7d2d600640 1 -- 172.21.15.47:0/2782638938 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c0b4b70 con 0x7f7d300aeda0 2024-02-19T15:27:25.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.219+0000 7f7d36e00640 1 -- 172.21.15.47:0/2782638938 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3009bdf0 con 0x7f7d300aeda0 2024-02-19T15:27:25.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.219+0000 7f7d36e00640 1 -- 172.21.15.47:0/2782638938 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3009b970 con 0x7f7d300aeda0 2024-02-19T15:27:25.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.219+0000 7f7d2d600640 1 -- 172.21.15.47:0/2782638938 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7d1c0b2070 con 0x7f7d300aeda0 2024-02-19T15:27:25.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.219+0000 7f7d2d600640 1 --2- 172.21.15.47:0/2782638938 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7d10066e40 0x7f7d10069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:25.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.223+0000 7f7d2d600640 1 -- 172.21.15.47:0/2782638938 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d1c11e960 con 0x7f7d300aeda0 2024-02-19T15:27:25.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.223+0000 7f7d35a00640 1 --2- 172.21.15.47:0/2782638938 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7d10066e40 0x7f7d10069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:25.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.223+0000 7f7d35a00640 1 --2- 172.21.15.47:0/2782638938 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7d10066e40 0x7f7d10069300 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400aae0 tx=0x7f7d24008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:25.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.227+0000 7f7d2d600640 1 -- 172.21.15.47:0/2782638938 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7d3009b970 con 0x7f7d300aeda0 2024-02-19T15:27:25.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.555+0000 7f7d36e00640 1 -- 172.21.15.47:0/2782638938 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d3009b970 con 0x7f7d300aeda0 2024-02-19T15:27:25.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.555+0000 7f7d2d600640 1 -- 172.21.15.47:0/2782638938 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4937 (secure 0 0 0) 0x7f7d3009b970 con 0x7f7d300aeda0 2024-02-19T15:27:25.560 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:25.560 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":75,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6578,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/408259797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":408259797},{"type":"v1","addr":"172.21.15.116:6836","nonce":408259797}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7327,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3795002232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3795002232},{"type":"v1","addr":"172.21.15.47:6835","nonce":3795002232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7330,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/819751716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":819751716},{"type":"v1","addr":"172.21.15.116:6837","nonce":819751716}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:24.428431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6834/897369244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":897369244},{"type":"v1","addr":"172.21.15.47:6834","nonce":897369244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:27:25.560 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 75 2024-02-19T15:27:25.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.559+0000 7f7d36e00640 1 -- 172.21.15.47:0/2782638938 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7d10066e40 msgr2=0x7f7d10069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:25.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.559+0000 7f7d36e00640 1 --2- 172.21.15.47:0/2782638938 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7d10066e40 0x7f7d10069300 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400aae0 tx=0x7f7d24008040 comp rx=0 tx=0).stop 2024-02-19T15:27:25.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.559+0000 7f7d36e00640 1 -- 172.21.15.47:0/2782638938 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d300aeda0 msgr2=0x7f7d30161a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:25.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.559+0000 7f7d36e00640 1 --2- 172.21.15.47:0/2782638938 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d300aeda0 0x7f7d30161a30 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c0ae040 tx=0x7f7d1c019200 comp rx=0 tx=0).stop 2024-02-19T15:27:25.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.559+0000 7f7d36e00640 1 -- 172.21.15.47:0/2782638938 shutdown_connections 2024-02-19T15:27:25.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.559+0000 7f7d36e00640 1 --2- 172.21.15.47:0/2782638938 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7d10066e40 0x7f7d10069300 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.559+0000 7f7d36e00640 1 --2- 172.21.15.47:0/2782638938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d3015dd70 0x7f7d30161f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.559+0000 7f7d36e00640 1 --2- 172.21.15.47:0/2782638938 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d300aeda0 0x7f7d30161a30 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.559+0000 7f7d36e00640 1 --2- 172.21.15.47:0/2782638938 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d3006d660 0x7f7d30159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.559+0000 7f7d36e00640 1 -- 172.21.15.47:0/2782638938 >> 172.21.15.47:0/2782638938 conn(0x7f7d3009a9d0 msgr2=0x7f7d300b3cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:25.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.559+0000 7f7d36e00640 1 -- 172.21.15.47:0/2782638938 shutdown_connections 2024-02-19T15:27:25.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.559+0000 7f7d36e00640 1 -- 172.21.15.47:0/2782638938 wait complete. 2024-02-19T15:27:25.582 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:27:25.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.719+0000 7f1876000640 1 Processor -- start 2024-02-19T15:27:25.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.719+0000 7f1876000640 1 -- start start 2024-02-19T15:27:25.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.719+0000 7f1876000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f187006d120 0x7f187006d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:25.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.719+0000 7f1876000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18700d19e0 0x7f18700d1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:25.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.719+0000 7f1876000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18700d2350 con 0x7f187015dd70 2024-02-19T15:27:25.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.719+0000 7f1876000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f187006daf0 con 0x7f187006d120 2024-02-19T15:27:25.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.719+0000 7f1876000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f187015f3f0 con 0x7f18700d19e0 2024-02-19T15:27:25.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.719+0000 7f186fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f187006d120 0x7f187006d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:25.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.719+0000 7f186fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f187006d120 0x7f187006d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55270/0 (socket says 172.21.15.47:55270) 2024-02-19T15:27:25.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.719+0000 7f186fe00640 1 -- 172.21.15.47:0/3324329774 learned_addr learned my addr 172.21.15.47:0/3324329774 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:25.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.719+0000 7f1875600640 1 --2- 172.21.15.47:0/3324329774 >> v2:172.21.15.116:3300/0 conn(0x7f18700d19e0 0x7f18700d1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:25.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.719+0000 7f186fe00640 1 -- 172.21.15.47:0/3324329774 >> v2:172.21.15.116:3300/0 conn(0x7f18700d19e0 msgr2=0x7f18700d1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:25.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.719+0000 7f186fe00640 1 --2- 172.21.15.47:0/3324329774 >> v2:172.21.15.116:3300/0 conn(0x7f18700d19e0 0x7f18700d1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.719+0000 7f186fe00640 1 -- 172.21.15.47:0/3324329774 >> v1:172.21.15.116:6789/0 conn(0x7f187015dd70 legacy=0x7f18700ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:25.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.719+0000 7f186fe00640 1 -- 172.21.15.47:0/3324329774 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187006c3e0 con 0x7f187006d120 2024-02-19T15:27:25.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.719+0000 7f1875600640 1 --2- 172.21.15.47:0/3324329774 >> v2:172.21.15.116:3300/0 conn(0x7f18700d19e0 0x7f18700d1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:25.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.723+0000 7f186fe00640 1 --2- 172.21.15.47:0/3324329774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f187006d120 0x7f187006d5b0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f18700d0020 tx=0x7f185c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac3c27fe91f1cb3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:25.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.723+0000 7f186f400640 1 -- 172.21.15.47:0/3324329774 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c0ab600 con 0x7f187006d120 2024-02-19T15:27:25.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.723+0000 7f186f400640 1 -- 172.21.15.47:0/3324329774 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185c0a7050 con 0x7f187006d120 2024-02-19T15:27:25.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.723+0000 7f186f400640 1 -- 172.21.15.47:0/3324329774 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c0ab980 con 0x7f187006d120 2024-02-19T15:27:25.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.723+0000 7f1876000640 1 -- 172.21.15.47:0/3324329774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f187006d120 msgr2=0x7f187006d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:25.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.723+0000 7f1876000640 1 --2- 172.21.15.47:0/3324329774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f187006d120 0x7f187006d5b0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f18700d0020 tx=0x7f185c01c450 comp rx=0 tx=0).stop 2024-02-19T15:27:25.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.723+0000 7f1876000640 1 -- 172.21.15.47:0/3324329774 shutdown_connections 2024-02-19T15:27:25.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.723+0000 7f1876000640 1 --2- 172.21.15.47:0/3324329774 >> v2:172.21.15.116:3300/0 conn(0x7f18700d19e0 0x7f18700d1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.723+0000 7f1876000640 1 --2- 172.21.15.47:0/3324329774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f187006d120 0x7f187006d5b0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.723+0000 7f1876000640 1 -- 172.21.15.47:0/3324329774 >> 172.21.15.47:0/3324329774 conn(0x7f187009a9d0 msgr2=0x7f187009ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:25.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.723+0000 7f1876000640 1 -- 172.21.15.47:0/3324329774 shutdown_connections 2024-02-19T15:27:25.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.723+0000 7f1876000640 1 -- 172.21.15.47:0/3324329774 wait complete. 2024-02-19T15:27:25.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f1876000640 1 Processor -- start 2024-02-19T15:27:25.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f1876000640 1 -- start start 2024-02-19T15:27:25.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f1876000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f187006d120 0x7f18700b60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:25.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f1876000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18700d19e0 0x7f1870159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:25.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f1876000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f187015dd70 0x7f1870154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:25.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f1876000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187015f9f0 con 0x7f18700d19e0 2024-02-19T15:27:25.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f1876000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1870159d50 con 0x7f187006d120 2024-02-19T15:27:25.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f1876000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18700b6d80 con 0x7f187015dd70 2024-02-19T15:27:25.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f186fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18700d19e0 0x7f1870159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:25.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f186fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18700d19e0 0x7f1870159810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55280/0 (socket says 172.21.15.47:55280) 2024-02-19T15:27:25.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f186fe00640 1 -- 172.21.15.47:0/928099624 learned_addr learned my addr 172.21.15.47:0/928099624 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:25.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f1875600640 1 --2- 172.21.15.47:0/928099624 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f187015dd70 0x7f1870154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:25.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f1874c00640 1 --2- 172.21.15.47:0/928099624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f187006d120 0x7f18700b60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:25.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f186fe00640 1 -- 172.21.15.47:0/928099624 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f187015dd70 msgr2=0x7f1870154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:25.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f186fe00640 1 --2- 172.21.15.47:0/928099624 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f187015dd70 0x7f1870154ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f186fe00640 1 -- 172.21.15.47:0/928099624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f187006d120 msgr2=0x7f18700b60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:25.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f186fe00640 1 --2- 172.21.15.47:0/928099624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f187006d120 0x7f18700b60c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f186fe00640 1 -- 172.21.15.47:0/928099624 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c0094d0 con 0x7f18700d19e0 2024-02-19T15:27:25.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f186fe00640 1 --2- 172.21.15.47:0/928099624 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18700d19e0 0x7f1870159810 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f185c01a080 tx=0x7f185c0191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:25.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f186cc00640 1 -- 172.21.15.47:0/928099624 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c0b4420 con 0x7f18700d19e0 2024-02-19T15:27:25.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f186cc00640 1 -- 172.21.15.47:0/928099624 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185c0b45a0 con 0x7f18700d19e0 2024-02-19T15:27:25.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f186cc00640 1 -- 172.21.15.47:0/928099624 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c0b4950 con 0x7f18700d19e0 2024-02-19T15:27:25.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f1876000640 1 -- 172.21.15.47:0/928099624 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18700b67c0 con 0x7f18700d19e0 2024-02-19T15:27:25.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.727+0000 7f1876000640 1 -- 172.21.15.47:0/928099624 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f187015f680 con 0x7f18700d19e0 2024-02-19T15:27:25.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.731+0000 7f1876000640 1 -- 172.21.15.47:0/928099624 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f187015fba0 con 0x7f18700d19e0 2024-02-19T15:27:25.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.731+0000 7f186cc00640 1 -- 172.21.15.47:0/928099624 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f185c0b2070 con 0x7f18700d19e0 2024-02-19T15:27:25.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.731+0000 7f186cc00640 1 --2- 172.21.15.47:0/928099624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1850066e40 0x7f1850069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:25.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.731+0000 7f186cc00640 1 -- 172.21.15.47:0/928099624 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f185c11e2c0 con 0x7f18700d19e0 2024-02-19T15:27:25.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.731+0000 7f1874c00640 1 --2- 172.21.15.47:0/928099624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1850066e40 0x7f1850069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:25.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.731+0000 7f1874c00640 1 --2- 172.21.15.47:0/928099624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1850066e40 0x7f1850069300 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f1870157690 tx=0x7f1864008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:25.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.739+0000 7f186cc00640 1 -- 172.21.15.47:0/928099624 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f187015fba0 con 0x7f18700d19e0 2024-02-19T15:27:25.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.903+0000 7f1876000640 1 -- 172.21.15.47:0/928099624 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f187015fba0 con 0x7f18700d19e0 2024-02-19T15:27:25.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.907+0000 7f186cc00640 1 -- 172.21.15.47:0/928099624 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16639 (secure 0 0 0) 0x7f187015fba0 con 0x7f18700d19e0 2024-02-19T15:27:25.911 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:25.911 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":108,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:27:23.037320+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:27:19.839144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-02-19T15:27:20.405924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:27:25.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.907+0000 7f1876000640 1 -- 172.21.15.47:0/928099624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1850066e40 msgr2=0x7f1850069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:25.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.907+0000 7f1876000640 1 --2- 172.21.15.47:0/928099624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1850066e40 0x7f1850069300 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f1870157690 tx=0x7f1864008040 comp rx=0 tx=0).stop 2024-02-19T15:27:25.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.907+0000 7f1876000640 1 -- 172.21.15.47:0/928099624 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18700d19e0 msgr2=0x7f1870159810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:25.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.907+0000 7f1876000640 1 --2- 172.21.15.47:0/928099624 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18700d19e0 0x7f1870159810 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f185c01a080 tx=0x7f185c0191f0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.907+0000 7f1876000640 1 -- 172.21.15.47:0/928099624 shutdown_connections 2024-02-19T15:27:25.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.907+0000 7f1876000640 1 --2- 172.21.15.47:0/928099624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1850066e40 0x7f1850069300 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.907+0000 7f1876000640 1 --2- 172.21.15.47:0/928099624 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f187015dd70 0x7f1870154ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.907+0000 7f1876000640 1 --2- 172.21.15.47:0/928099624 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18700d19e0 0x7f1870159810 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.907+0000 7f1876000640 1 --2- 172.21.15.47:0/928099624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f187006d120 0x7f18700b60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:25.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.907+0000 7f1876000640 1 -- 172.21.15.47:0/928099624 >> 172.21.15.47:0/928099624 conn(0x7f187009a9d0 msgr2=0x7f18700ae860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:25.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.907+0000 7f1876000640 1 -- 172.21.15.47:0/928099624 shutdown_connections 2024-02-19T15:27:25.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:25.907+0000 7f1876000640 1 -- 172.21.15.47:0/928099624 wait complete. 2024-02-19T15:27:25.941 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T15:27:25.941 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:27:26.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.079+0000 7f06b9400640 1 Processor -- start 2024-02-19T15:27:26.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.079+0000 7f06b9400640 1 -- start start 2024-02-19T15:27:26.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.079+0000 7f06b9400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f06b4145590 0x7f06b40727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:26.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.079+0000 7f06b9400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b415d280 con 0x7f06b415c390 2024-02-19T15:27:26.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.079+0000 7f06b9400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b4070ff0 con 0x7f06b40cea80 2024-02-19T15:27:26.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.079+0000 7f06b9400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b40709e0 con 0x7f06b4145590 2024-02-19T15:27:26.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.079+0000 7f06b3200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f06b415c390 0x7f06b415c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39792/0 (socket says 172.21.15.47:39792) 2024-02-19T15:27:26.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.079+0000 7f06b3200640 1 -- 172.21.15.47:0/741735461 learned_addr learned my addr 172.21.15.47:0/741735461 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:26.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.079+0000 7f06b2800640 1 --2- 172.21.15.47:0/741735461 >> v2:172.21.15.116:3300/0 conn(0x7f06b4145590 0x7f06b40727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:26.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.083+0000 7f06b1e00640 1 -- 172.21.15.47:0/741735461 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108639507 0 0) 0x7f06b415d280 con 0x7f06b415c390 2024-02-19T15:27:26.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.083+0000 7f06b1e00640 1 -- 172.21.15.47:0/741735461 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0698003530 con 0x7f06b415c390 2024-02-19T15:27:26.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.083+0000 7f06b1e00640 1 -- 172.21.15.47:0/741735461 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859827196 0 0) 0x7f06b4070ff0 con 0x7f06b40cea80 2024-02-19T15:27:26.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.083+0000 7f06b1e00640 1 -- 172.21.15.47:0/741735461 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06b415d280 con 0x7f06b40cea80 2024-02-19T15:27:26.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.083+0000 7f06b2800640 1 -- 172.21.15.47:0/741735461 >> v1:172.21.15.116:6789/0 conn(0x7f06b40cea80 legacy=0x7f06b406c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:26.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.083+0000 7f06b2800640 1 -- 172.21.15.47:0/741735461 >> v1:172.21.15.47:6789/0 conn(0x7f06b415c390 legacy=0x7f06b415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:26.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.083+0000 7f06b2800640 1 -- 172.21.15.47:0/741735461 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b40cf020 con 0x7f06b4145590 2024-02-19T15:27:26.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.083+0000 7f06b2800640 1 --2- 172.21.15.47:0/741735461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06b4145590 0x7f06b40727b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f06a40097c0 tx=0x7f06a401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b9ab353247bee9b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:26.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.083+0000 7f06b1e00640 1 -- 172.21.15.47:0/741735461 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a40ac410 con 0x7f06b4145590 2024-02-19T15:27:26.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.083+0000 7f06b1e00640 1 -- 172.21.15.47:0/741735461 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a40ac570 con 0x7f06b4145590 2024-02-19T15:27:26.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.083+0000 7f06b1e00640 1 -- 172.21.15.47:0/741735461 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a40ac900 con 0x7f06b4145590 2024-02-19T15:27:26.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.083+0000 7f06b9400640 1 -- 172.21.15.47:0/741735461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06b4145590 msgr2=0x7f06b40727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:26.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.083+0000 7f06b9400640 1 --2- 172.21.15.47:0/741735461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06b4145590 0x7f06b40727b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f06a40097c0 tx=0x7f06a401c4a0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b9400640 1 -- 172.21.15.47:0/741735461 shutdown_connections 2024-02-19T15:27:26.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b9400640 1 --2- 172.21.15.47:0/741735461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06b4145590 0x7f06b40727b0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b9400640 1 -- 172.21.15.47:0/741735461 >> 172.21.15.47:0/741735461 conn(0x7f06b40b16a0 msgr2=0x7f06b40b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:26.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b9400640 1 -- 172.21.15.47:0/741735461 shutdown_connections 2024-02-19T15:27:26.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b9400640 1 -- 172.21.15.47:0/741735461 wait complete. 2024-02-19T15:27:26.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b9400640 1 Processor -- start 2024-02-19T15:27:26.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b9400640 1 -- start start 2024-02-19T15:27:26.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b9400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06b40cea80 0x7f06b415b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:26.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b9400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06b4145590 0x7f06b40b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:26.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b9400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06b415c390 0x7f06b413bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:26.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b9400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b40b3970 con 0x7f06b415c390 2024-02-19T15:27:26.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b9400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b40b3c50 con 0x7f06b40cea80 2024-02-19T15:27:26.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b9400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b40b3f30 con 0x7f06b4145590 2024-02-19T15:27:26.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b3200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06b415c390 0x7f06b413bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:26.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b3200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06b415c390 0x7f06b413bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55292/0 (socket says 172.21.15.47:55292) 2024-02-19T15:27:26.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b3200640 1 -- 172.21.15.47:0/937018591 learned_addr learned my addr 172.21.15.47:0/937018591 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:26.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b2800640 1 --2- 172.21.15.47:0/937018591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06b40cea80 0x7f06b415b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:26.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b3200640 1 -- 172.21.15.47:0/937018591 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06b4145590 msgr2=0x7f06b40b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:26.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06a9e00640 1 --2- 172.21.15.47:0/937018591 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06b4145590 0x7f06b40b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:26.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b3200640 1 --2- 172.21.15.47:0/937018591 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06b4145590 0x7f06b40b2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b3200640 1 -- 172.21.15.47:0/937018591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06b40cea80 msgr2=0x7f06b415b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:26.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b3200640 1 --2- 172.21.15.47:0/937018591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06b40cea80 0x7f06b415b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b3200640 1 -- 172.21.15.47:0/937018591 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a4009420 con 0x7f06b415c390 2024-02-19T15:27:26.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b2800640 1 --2- 172.21.15.47:0/937018591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06b40cea80 0x7f06b415b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:26.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06a9e00640 1 --2- 172.21.15.47:0/937018591 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06b4145590 0x7f06b40b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:26.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06b3200640 1 --2- 172.21.15.47:0/937018591 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06b415c390 0x7f06b413bd90 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f06ac00cb10 tx=0x7f06ac00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:26.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.087+0000 7f06ab400640 1 -- 172.21.15.47:0/937018591 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ac00ac70 con 0x7f06b415c390 2024-02-19T15:27:26.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.091+0000 7f06ab400640 1 -- 172.21.15.47:0/937018591 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ac00add0 con 0x7f06b415c390 2024-02-19T15:27:26.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.091+0000 7f06ab400640 1 -- 172.21.15.47:0/937018591 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ac012600 con 0x7f06b415c390 2024-02-19T15:27:26.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.091+0000 7f06b9400640 1 -- 172.21.15.47:0/937018591 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b40b4210 con 0x7f06b415c390 2024-02-19T15:27:26.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.091+0000 7f06b9400640 1 -- 172.21.15.47:0/937018591 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b415b9c0 con 0x7f06b415c390 2024-02-19T15:27:26.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.091+0000 7f06b9400640 1 -- 172.21.15.47:0/937018591 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b40b4730 con 0x7f06b415c390 2024-02-19T15:27:26.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.091+0000 7f06ab400640 1 -- 172.21.15.47:0/937018591 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f06ac007070 con 0x7f06b415c390 2024-02-19T15:27:26.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.095+0000 7f06ab400640 1 --2- 172.21.15.47:0/937018591 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0690066e40 0x7f0690069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:26.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.095+0000 7f06ab400640 1 -- 172.21.15.47:0/937018591 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06ac084900 con 0x7f06b415c390 2024-02-19T15:27:26.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.095+0000 7f06b2800640 1 --2- 172.21.15.47:0/937018591 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0690066e40 0x7f0690069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:26.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.095+0000 7f06b2800640 1 --2- 172.21.15.47:0/937018591 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0690066e40 0x7f0690069300 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f06a40ae000 tx=0x7f06a40a6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:26.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.099+0000 7f06ab400640 1 -- 172.21.15.47:0/937018591 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f06b40b4730 con 0x7f06b415c390 2024-02-19T15:27:26.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.255+0000 7f06b9400640 1 -- 172.21.15.47:0/937018591 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f06b40b4730 con 0x7f0690066e40 2024-02-19T15:27:26.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.263+0000 7f06ab400640 1 -- 172.21.15.47:0/937018591 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284066 (secure 0 0 0) 0x7f06b40b4730 con 0x7f0690066e40 2024-02-19T15:27:26.272 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:26.274 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:27:26.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.267+0000 7f06b9400640 1 -- 172.21.15.47:0/937018591 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0690066e40 msgr2=0x7f0690069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:26.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.267+0000 7f06b9400640 1 --2- 172.21.15.47:0/937018591 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0690066e40 0x7f0690069300 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f06a40ae000 tx=0x7f06a40a6040 comp rx=0 tx=0).stop 2024-02-19T15:27:26.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.267+0000 7f06b9400640 1 -- 172.21.15.47:0/937018591 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06b415c390 msgr2=0x7f06b413bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:26.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.267+0000 7f06b9400640 1 --2- 172.21.15.47:0/937018591 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06b415c390 0x7f06b413bd90 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f06ac00cb10 tx=0x7f06ac00a060 comp rx=0 tx=0).stop 2024-02-19T15:27:26.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.267+0000 7f06b9400640 1 -- 172.21.15.47:0/937018591 shutdown_connections 2024-02-19T15:27:26.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.267+0000 7f06b9400640 1 --2- 172.21.15.47:0/937018591 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0690066e40 0x7f0690069300 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.267+0000 7f06b9400640 1 --2- 172.21.15.47:0/937018591 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06b415c390 0x7f06b413bd90 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.267+0000 7f06b9400640 1 --2- 172.21.15.47:0/937018591 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06b4145590 0x7f06b40b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.267+0000 7f06b9400640 1 --2- 172.21.15.47:0/937018591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06b40cea80 0x7f06b415b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.267+0000 7f06b9400640 1 -- 172.21.15.47:0/937018591 >> 172.21.15.47:0/937018591 conn(0x7f06b40b16a0 msgr2=0x7f06b4141af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:26.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.267+0000 7f06b9400640 1 -- 172.21.15.47:0/937018591 shutdown_connections 2024-02-19T15:27:26.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.267+0000 7f06b9400640 1 -- 172.21.15.47:0/937018591 wait complete. 2024-02-19T15:27:26.293 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-02-19T15:27:25.221972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590438,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":160,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":20,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":306604,"kb_used_data":11372,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754668116,"statfs":{"total":773094113280,"available":772780150784,"internally_reserved":0,"allocated":11644928,"data_stored":5659569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":302173746},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.216184"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.031361+0000","last_change":"2024-02-19T15:27:21.031361+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.031361+0000","last_undegraded":"2024-02-19T15:27:21.031361+0000","last_fullsized":"2024-02-19T15:27:21.031361+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.030203+0000","last_change":"2024-02-19T15:27:21.030203+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.030203+0000","last_undegraded":"2024-02-19T15:27:21.030203+0000","last_fullsized":"2024-02-19T15:27:21.030203+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.016897+0000","last_change":"2024-02-19T15:27:21.016897+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.016897+0000","last_undegraded":"2024-02-19T15:27:21.016897+0000","last_fullsized":"2024-02-19T15:27:21.016897+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.030394+0000","last_change":"2024-02-19T15:27:21.030394+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.030394+0000","last_undegraded":"2024-02-19T15:27:21.030394+0000","last_fullsized":"2024-02-19T15:27:21.030394+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.023740+0000","last_change":"2024-02-19T15:27:21.023740+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.023740+0000","last_undegraded":"2024-02-19T15:27:21.023740+0000","last_fullsized":"2024-02-19T15:27:21.023740+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.011541+0000","last_change":"2024-02-19T15:27:21.011541+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.011541+0000","last_undegraded":"2024-02-19T15:27:21.011541+0000","last_fullsized":"2024-02-19T15:27:21.011541+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.026169+0000","last_change":"2024-02-19T15:27:21.026169+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.026169+0000","last_undegraded":"2024-02-19T15:27:21.026169+0000","last_fullsized":"2024-02-19T15:27:21.026169+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.016613+0000","last_change":"2024-02-19T15:27:21.016613+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.016613+0000","last_undegraded":"2024-02-19T15:27:21.016613+0000","last_fullsized":"2024-02-19T15:27:21.016613+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.030819+0000","last_change":"2024-02-19T15:27:21.030819+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.030819+0000","last_undegraded":"2024-02-19T15:27:21.030819+0000","last_fullsized":"2024-02-19T15:27:21.030819+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.029180+0000","last_change":"2024-02-19T15:27:21.029180+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.029180+0000","last_undegraded":"2024-02-19T15:27:21.029180+0000","last_fullsized":"2024-02-19T15:27:21.029180+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.026903+0000","last_change":"2024-02-19T15:27:21.026903+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.026903+0000","last_undegraded":"2024-02-19T15:27:21.026903+0000","last_fullsized":"2024-02-19T15:27:21.026903+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.028102+0000","last_change":"2024-02-19T15:27:21.028102+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.028102+0000","last_undegraded":"2024-02-19T15:27:21.028102+0000","last_fullsized":"2024-02-19T15:27:21.028102+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.015426+0000","last_change":"2024-02-19T15:27:21.015426+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.015426+0000","last_undegraded":"2024-02-19T15:27:21.015426+0000","last_fullsized":"2024-02-19T15:27:21.015426+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.016518+0000","last_change":"2024-02-19T15:27:21.016518+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.016518+0000","last_undegraded":"2024-02-19T15:27:21.016518+0000","last_fullsized":"2024-02-19T15:27:21.016518+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.011705+0000","last_change":"2024-02-19T15:27:21.011705+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.011705+0000","last_undegraded":"2024-02-19T15:27:21.011705+0000","last_fullsized":"2024-02-19T15:27:21.011705+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.026401+0000","last_change":"2024-02-19T15:27:21.026401+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.026401+0000","last_undegraded":"2024-02-19T15:27:21.026401+0000","last_fullsized":"2024-02-19T15:27:21.026401+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.025923+0000","last_change":"2024-02-19T15:27:21.025923+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.025923+0000","last_undegraded":"2024-02-19T15:27:21.025923+0000","last_fullsized":"2024-02-19T15:27:21.025923+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.042266+0000","last_change":"2024-02-19T15:27:22.036471+0000","last_active":"2024-02-19T15:27:22.042266+0000","last_peered":"2024-02-19T15:27:22.042266+0000","last_clean":"2024-02-19T15:27:22.042266+0000","last_became_active":"2024-02-19T15:27:22.036201+0000","last_became_peered":"2024-02-19T15:27:22.036201+0000","last_unstale":"2024-02-19T15:27:22.042266+0000","last_undegraded":"2024-02-19T15:27:22.042266+0000","last_fullsized":"2024-02-19T15:27:22.042266+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:28:59.516714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.016248+0000","last_change":"2024-02-19T15:27:21.016248+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.016248+0000","last_undegraded":"2024-02-19T15:27:21.016248+0000","last_fullsized":"2024-02-19T15:27:21.016248+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024313+0000","last_change":"2024-02-19T15:27:21.024313+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024313+0000","last_undegraded":"2024-02-19T15:27:21.024313+0000","last_fullsized":"2024-02-19T15:27:21.024313+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024688+0000","last_change":"2024-02-19T15:27:21.024688+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024688+0000","last_undegraded":"2024-02-19T15:27:21.024688+0000","last_fullsized":"2024-02-19T15:27:21.024688+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.033511+0000","last_change":"2024-02-19T15:27:21.033511+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.033511+0000","last_undegraded":"2024-02-19T15:27:21.033511+0000","last_fullsized":"2024-02-19T15:27:21.033511+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024070+0000","last_change":"2024-02-19T15:27:21.024070+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024070+0000","last_undegraded":"2024-02-19T15:27:21.024070+0000","last_fullsized":"2024-02-19T15:27:21.024070+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.033844+0000","last_change":"2024-02-19T15:27:21.033844+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.033844+0000","last_undegraded":"2024-02-19T15:27:21.033844+0000","last_fullsized":"2024-02-19T15:27:21.033844+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024026+0000","last_change":"2024-02-19T15:27:21.024026+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024026+0000","last_undegraded":"2024-02-19T15:27:21.024026+0000","last_fullsized":"2024-02-19T15:27:21.024026+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.025203+0000","last_change":"2024-02-19T15:27:21.025203+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.025203+0000","last_undegraded":"2024-02-19T15:27:21.025203+0000","last_fullsized":"2024-02-19T15:27:21.025203+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.012320+0000","last_change":"2024-02-19T15:27:21.012320+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.012320+0000","last_undegraded":"2024-02-19T15:27:21.012320+0000","last_fullsized":"2024-02-19T15:27:21.012320+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.033395+0000","last_change":"2024-02-19T15:27:21.033395+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.033395+0000","last_undegraded":"2024-02-19T15:27:21.033395+0000","last_fullsized":"2024-02-19T15:27:21.033395+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024077+0000","last_change":"2024-02-19T15:27:21.024077+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024077+0000","last_undegraded":"2024-02-19T15:27:21.024077+0000","last_fullsized":"2024-02-19T15:27:21.024077+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.021450+0000","last_change":"2024-02-19T15:27:21.021450+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.021450+0000","last_undegraded":"2024-02-19T15:27:21.021450+0000","last_fullsized":"2024-02-19T15:27:21.021450+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.026040+0000","last_change":"2024-02-19T15:27:21.026040+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.026040+0000","last_undegraded":"2024-02-19T15:27:21.026040+0000","last_fullsized":"2024-02-19T15:27:21.026040+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.011531+0000","last_change":"2024-02-19T15:27:21.011531+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.011531+0000","last_undegraded":"2024-02-19T15:27:21.011531+0000","last_fullsized":"2024-02-19T15:27:21.011531+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.013560+0000","last_change":"2024-02-19T15:27:21.013560+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.013560+0000","last_undegraded":"2024-02-19T15:27:21.013560+0000","last_fullsized":"2024-02-19T15:27:21.013560+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.033484+0000","last_change":"2024-02-19T15:27:21.030063+0000","last_active":"2024-02-19T15:27:21.033484+0000","last_peered":"2024-02-19T15:27:21.033484+0000","last_clean":"2024-02-19T15:27:21.033484+0000","last_became_active":"2024-02-19T15:27:21.029780+0000","last_became_peered":"2024-02-19T15:27:21.029780+0000","last_unstale":"2024-02-19T15:27:21.033484+0000","last_undegraded":"2024-02-19T15:27:21.033484+0000","last_fullsized":"2024-02-19T15:27:21.033484+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:15.014562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.054040+0000","last_change":"2024-02-19T15:27:22.048726+0000","last_active":"2024-02-19T15:27:22.054040+0000","last_peered":"2024-02-19T15:27:22.054040+0000","last_clean":"2024-02-19T15:27:22.054040+0000","last_became_active":"2024-02-19T15:27:22.048437+0000","last_became_peered":"2024-02-19T15:27:22.048437+0000","last_unstale":"2024-02-19T15:27:22.054040+0000","last_undegraded":"2024-02-19T15:27:22.054040+0000","last_fullsized":"2024-02-19T15:27:22.054040+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:50:16.869685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.030517+0000","last_change":"2024-02-19T15:27:21.027135+0000","last_active":"2024-02-19T15:27:21.030517+0000","last_peered":"2024-02-19T15:27:21.030517+0000","last_clean":"2024-02-19T15:27:21.030517+0000","last_became_active":"2024-02-19T15:27:21.026870+0000","last_became_peered":"2024-02-19T15:27:21.026870+0000","last_unstale":"2024-02-19T15:27:21.030517+0000","last_undegraded":"2024-02-19T15:27:21.030517+0000","last_fullsized":"2024-02-19T15:27:21.030517+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:51.292175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.012534+0000","last_change":"2024-02-19T15:27:21.012534+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.012534+0000","last_undegraded":"2024-02-19T15:27:21.012534+0000","last_fullsized":"2024-02-19T15:27:21.012534+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.038196+0000","last_change":"2024-02-19T15:27:21.029872+0000","last_active":"2024-02-19T15:27:21.038196+0000","last_peered":"2024-02-19T15:27:21.038196+0000","last_clean":"2024-02-19T15:27:21.038196+0000","last_became_active":"2024-02-19T15:27:21.029590+0000","last_became_peered":"2024-02-19T15:27:21.029590+0000","last_unstale":"2024-02-19T15:27:21.038196+0000","last_undegraded":"2024-02-19T15:27:21.038196+0000","last_fullsized":"2024-02-19T15:27:21.038196+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:32:04.524076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.015800+0000","last_change":"2024-02-19T15:27:21.015800+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.015800+0000","last_undegraded":"2024-02-19T15:27:21.015800+0000","last_fullsized":"2024-02-19T15:27:21.015800+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.043779+0000","last_change":"2024-02-19T15:27:21.042870+0000","last_active":"2024-02-19T15:27:21.043779+0000","last_peered":"2024-02-19T15:27:21.043779+0000","last_clean":"2024-02-19T15:27:21.043779+0000","last_became_active":"2024-02-19T15:27:21.041700+0000","last_became_peered":"2024-02-19T15:27:21.041700+0000","last_unstale":"2024-02-19T15:27:21.043779+0000","last_undegraded":"2024-02-19T15:27:21.043779+0000","last_fullsized":"2024-02-19T15:27:21.043779+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:21:53.467732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.022487+0000","last_change":"2024-02-19T15:27:21.022487+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.022487+0000","last_undegraded":"2024-02-19T15:27:21.022487+0000","last_fullsized":"2024-02-19T15:27:21.022487+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.041473+0000","last_change":"2024-02-19T15:27:21.040738+0000","last_active":"2024-02-19T15:27:21.041473+0000","last_peered":"2024-02-19T15:27:21.041473+0000","last_clean":"2024-02-19T15:27:21.041473+0000","last_became_active":"2024-02-19T15:27:21.040492+0000","last_became_peered":"2024-02-19T15:27:21.040492+0000","last_unstale":"2024-02-19T15:27:21.041473+0000","last_undegraded":"2024-02-19T15:27:21.041473+0000","last_fullsized":"2024-02-19T15:27:21.041473+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:38:35.042142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.023070+0000","last_change":"2024-02-19T15:27:21.023070+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.023070+0000","last_undegraded":"2024-02-19T15:27:21.023070+0000","last_fullsized":"2024-02-19T15:27:21.023070+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.039620+0000","last_change":"2024-02-19T15:27:21.037989+0000","last_active":"2024-02-19T15:27:21.039620+0000","last_peered":"2024-02-19T15:27:21.039620+0000","last_clean":"2024-02-19T15:27:21.039620+0000","last_became_active":"2024-02-19T15:27:21.037527+0000","last_became_peered":"2024-02-19T15:27:21.037527+0000","last_unstale":"2024-02-19T15:27:21.039620+0000","last_undegraded":"2024-02-19T15:27:21.039620+0000","last_fullsized":"2024-02-19T15:27:21.039620+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:59:16.122985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024991+0000","last_change":"2024-02-19T15:27:21.024991+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024991+0000","last_undegraded":"2024-02-19T15:27:21.024991+0000","last_fullsized":"2024-02-19T15:27:21.024991+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.032693+0000","last_change":"2024-02-19T15:27:21.031349+0000","last_active":"2024-02-19T15:27:21.032693+0000","last_peered":"2024-02-19T15:27:21.032693+0000","last_clean":"2024-02-19T15:27:21.032693+0000","last_became_active":"2024-02-19T15:27:21.031075+0000","last_became_peered":"2024-02-19T15:27:21.031075+0000","last_unstale":"2024-02-19T15:27:21.032693+0000","last_undegraded":"2024-02-19T15:27:21.032693+0000","last_fullsized":"2024-02-19T15:27:21.032693+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:25:19.941750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.023512+0000","last_change":"2024-02-19T15:27:21.023512+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.023512+0000","last_undegraded":"2024-02-19T15:27:21.023512+0000","last_fullsized":"2024-02-19T15:27:21.023512+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.030110+0000","last_change":"2024-02-19T15:27:21.025268+0000","last_active":"2024-02-19T15:27:21.030110+0000","last_peered":"2024-02-19T15:27:21.030110+0000","last_clean":"2024-02-19T15:27:21.030110+0000","last_became_active":"2024-02-19T15:27:21.022655+0000","last_became_peered":"2024-02-19T15:27:21.022655+0000","last_unstale":"2024-02-19T15:27:21.030110+0000","last_undegraded":"2024-02-19T15:27:21.030110+0000","last_fullsized":"2024-02-19T15:27:21.030110+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:44:25.168118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.021162+0000","last_change":"2024-02-19T15:27:21.021162+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.021162+0000","last_undegraded":"2024-02-19T15:27:21.021162+0000","last_fullsized":"2024-02-19T15:27:21.021162+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.043234+0000","last_change":"2024-02-19T15:27:21.041198+0000","last_active":"2024-02-19T15:27:21.043234+0000","last_peered":"2024-02-19T15:27:21.043234+0000","last_clean":"2024-02-19T15:27:21.043234+0000","last_became_active":"2024-02-19T15:27:21.040821+0000","last_became_peered":"2024-02-19T15:27:21.040821+0000","last_unstale":"2024-02-19T15:27:21.043234+0000","last_undegraded":"2024-02-19T15:27:21.043234+0000","last_fullsized":"2024-02-19T15:27:21.043234+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:15:15.162474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.050040+0000","last_change":"2024-02-19T15:27:22.043233+0000","last_active":"2024-02-19T15:27:22.050040+0000","last_peered":"2024-02-19T15:27:22.050040+0000","last_clean":"2024-02-19T15:27:22.050040+0000","last_became_active":"2024-02-19T15:27:22.042812+0000","last_became_peered":"2024-02-19T15:27:22.042812+0000","last_unstale":"2024-02-19T15:27:22.050040+0000","last_undegraded":"2024-02-19T15:27:22.050040+0000","last_fullsized":"2024-02-19T15:27:22.050040+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:02:10.136155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.039592+0000","last_change":"2024-02-19T15:27:21.036912+0000","last_active":"2024-02-19T15:27:21.039592+0000","last_peered":"2024-02-19T15:27:21.039592+0000","last_clean":"2024-02-19T15:27:21.039592+0000","last_became_active":"2024-02-19T15:27:21.036227+0000","last_became_peered":"2024-02-19T15:27:21.036227+0000","last_unstale":"2024-02-19T15:27:21.039592+0000","last_undegraded":"2024-02-19T15:27:21.039592+0000","last_fullsized":"2024-02-19T15:27:21.039592+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:15:01.148828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.032945+0000","last_change":"2024-02-19T15:27:21.032945+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.032945+0000","last_undegraded":"2024-02-19T15:27:21.032945+0000","last_fullsized":"2024-02-19T15:27:21.032945+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.023292+0000","last_change":"2024-02-19T15:27:21.022250+0000","last_active":"2024-02-19T15:27:21.023292+0000","last_peered":"2024-02-19T15:27:21.023292+0000","last_clean":"2024-02-19T15:27:21.023292+0000","last_became_active":"2024-02-19T15:27:21.021690+0000","last_became_peered":"2024-02-19T15:27:21.021690+0000","last_unstale":"2024-02-19T15:27:21.023292+0000","last_undegraded":"2024-02-19T15:27:21.023292+0000","last_fullsized":"2024-02-19T15:27:21.023292+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:41.432608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.021714+0000","last_change":"2024-02-19T15:27:21.021714+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.021714+0000","last_undegraded":"2024-02-19T15:27:21.021714+0000","last_fullsized":"2024-02-19T15:27:21.021714+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.037376+0000","last_change":"2024-02-19T15:27:21.031559+0000","last_active":"2024-02-19T15:27:21.037376+0000","last_peered":"2024-02-19T15:27:21.037376+0000","last_clean":"2024-02-19T15:27:21.037376+0000","last_became_active":"2024-02-19T15:27:21.031280+0000","last_became_peered":"2024-02-19T15:27:21.031280+0000","last_unstale":"2024-02-19T15:27:21.037376+0000","last_undegraded":"2024-02-19T15:27:21.037376+0000","last_fullsized":"2024-02-19T15:27:21.037376+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:48.463248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.029223+0000","last_change":"2024-02-19T15:27:21.029223+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.029223+0000","last_undegraded":"2024-02-19T15:27:21.029223+0000","last_fullsized":"2024-02-19T15:27:21.029223+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.025611+0000","last_change":"2024-02-19T15:27:21.019044+0000","last_active":"2024-02-19T15:27:21.025611+0000","last_peered":"2024-02-19T15:27:21.025611+0000","last_clean":"2024-02-19T15:27:21.025611+0000","last_became_active":"2024-02-19T15:27:21.018732+0000","last_became_peered":"2024-02-19T15:27:21.018732+0000","last_unstale":"2024-02-19T15:27:21.025611+0000","last_undegraded":"2024-02-19T15:27:21.025611+0000","last_fullsized":"2024-02-19T15:27:21.025611+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:02:10.142351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.033846+0000","last_change":"2024-02-19T15:27:21.033846+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.033846+0000","last_undegraded":"2024-02-19T15:27:21.033846+0000","last_fullsized":"2024-02-19T15:27:21.033846+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.027358+0000","last_change":"2024-02-19T15:27:21.022352+0000","last_active":"2024-02-19T15:27:22.027358+0000","last_peered":"2024-02-19T15:27:22.027358+0000","last_clean":"2024-02-19T15:27:22.027358+0000","last_became_active":"2024-02-19T15:27:21.021353+0000","last_became_peered":"2024-02-19T15:27:21.021353+0000","last_unstale":"2024-02-19T15:27:22.027358+0000","last_undegraded":"2024-02-19T15:27:22.027358+0000","last_fullsized":"2024-02-19T15:27:22.027358+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:05:11.130486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.029752+0000","last_change":"2024-02-19T15:27:21.029752+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.029752+0000","last_undegraded":"2024-02-19T15:27:21.029752+0000","last_fullsized":"2024-02-19T15:27:21.029752+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.043774+0000","last_change":"2024-02-19T15:27:21.042396+0000","last_active":"2024-02-19T15:27:21.043774+0000","last_peered":"2024-02-19T15:27:21.043774+0000","last_clean":"2024-02-19T15:27:21.043774+0000","last_became_active":"2024-02-19T15:27:21.042017+0000","last_became_peered":"2024-02-19T15:27:21.042017+0000","last_unstale":"2024-02-19T15:27:21.043774+0000","last_undegraded":"2024-02-19T15:27:21.043774+0000","last_fullsized":"2024-02-19T15:27:21.043774+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:54:06.234893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024250+0000","last_change":"2024-02-19T15:27:21.024250+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024250+0000","last_undegraded":"2024-02-19T15:27:21.024250+0000","last_fullsized":"2024-02-19T15:27:21.024250+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.031024+0000","last_change":"2024-02-19T15:27:21.027023+0000","last_active":"2024-02-19T15:27:21.031024+0000","last_peered":"2024-02-19T15:27:21.031024+0000","last_clean":"2024-02-19T15:27:21.031024+0000","last_became_active":"2024-02-19T15:27:21.026565+0000","last_became_peered":"2024-02-19T15:27:21.026565+0000","last_unstale":"2024-02-19T15:27:21.031024+0000","last_undegraded":"2024-02-19T15:27:21.031024+0000","last_fullsized":"2024-02-19T15:27:21.031024+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:52:51.983435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.013930+0000","last_change":"2024-02-19T15:27:21.013930+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.013930+0000","last_undegraded":"2024-02-19T15:27:21.013930+0000","last_fullsized":"2024-02-19T15:27:21.013930+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.039502+0000","last_change":"2024-02-19T15:27:21.033163+0000","last_active":"2024-02-19T15:27:21.039502+0000","last_peered":"2024-02-19T15:27:21.039502+0000","last_clean":"2024-02-19T15:27:21.039502+0000","last_became_active":"2024-02-19T15:27:21.032778+0000","last_became_peered":"2024-02-19T15:27:21.032778+0000","last_unstale":"2024-02-19T15:27:21.039502+0000","last_undegraded":"2024-02-19T15:27:21.039502+0000","last_fullsized":"2024-02-19T15:27:21.039502+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:35:31.450513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.015439+0000","last_change":"2024-02-19T15:27:21.015439+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.015439+0000","last_undegraded":"2024-02-19T15:27:21.015439+0000","last_fullsized":"2024-02-19T15:27:21.015439+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.047598+0000","last_change":"2024-02-19T15:27:21.033698+0000","last_active":"2024-02-19T15:27:22.047598+0000","last_peered":"2024-02-19T15:27:22.047598+0000","last_clean":"2024-02-19T15:27:22.047598+0000","last_became_active":"2024-02-19T15:27:21.033354+0000","last_became_peered":"2024-02-19T15:27:21.033354+0000","last_unstale":"2024-02-19T15:27:22.047598+0000","last_undegraded":"2024-02-19T15:27:22.047598+0000","last_fullsized":"2024-02-19T15:27:22.047598+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:42.028837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.015175+0000","last_change":"2024-02-19T15:27:21.015175+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.015175+0000","last_undegraded":"2024-02-19T15:27:21.015175+0000","last_fullsized":"2024-02-19T15:27:21.015175+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.033403+0000","last_change":"2024-02-19T15:27:21.031371+0000","last_active":"2024-02-19T15:27:21.033403+0000","last_peered":"2024-02-19T15:27:21.033403+0000","last_clean":"2024-02-19T15:27:21.033403+0000","last_became_active":"2024-02-19T15:27:21.031066+0000","last_became_peered":"2024-02-19T15:27:21.031066+0000","last_unstale":"2024-02-19T15:27:21.033403+0000","last_undegraded":"2024-02-19T15:27:21.033403+0000","last_fullsized":"2024-02-19T15:27:21.033403+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:27.660797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.015962+0000","last_change":"2024-02-19T15:27:21.015962+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.015962+0000","last_undegraded":"2024-02-19T15:27:21.015962+0000","last_fullsized":"2024-02-19T15:27:21.015962+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.039981+0000","last_change":"2024-02-19T15:27:21.026803+0000","last_active":"2024-02-19T15:27:21.039981+0000","last_peered":"2024-02-19T15:27:21.039981+0000","last_clean":"2024-02-19T15:27:21.039981+0000","last_became_active":"2024-02-19T15:27:21.025817+0000","last_became_peered":"2024-02-19T15:27:21.025817+0000","last_unstale":"2024-02-19T15:27:21.039981+0000","last_undegraded":"2024-02-19T15:27:21.039981+0000","last_fullsized":"2024-02-19T15:27:21.039981+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:15.014562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.023713+0000","last_change":"2024-02-19T15:27:21.023713+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.023713+0000","last_undegraded":"2024-02-19T15:27:21.023713+0000","last_fullsized":"2024-02-19T15:27:21.023713+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.043399+0000","last_change":"2024-02-19T15:27:21.042086+0000","last_active":"2024-02-19T15:27:21.043399+0000","last_peered":"2024-02-19T15:27:21.043399+0000","last_clean":"2024-02-19T15:27:21.043399+0000","last_became_active":"2024-02-19T15:27:21.041580+0000","last_became_peered":"2024-02-19T15:27:21.041580+0000","last_unstale":"2024-02-19T15:27:21.043399+0000","last_undegraded":"2024-02-19T15:27:21.043399+0000","last_fullsized":"2024-02-19T15:27:21.043399+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:20:44.224840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.016067+0000","last_change":"2024-02-19T15:27:21.016067+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.016067+0000","last_undegraded":"2024-02-19T15:27:21.016067+0000","last_fullsized":"2024-02-19T15:27:21.016067+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.039839+0000","last_change":"2024-02-19T15:27:21.038892+0000","last_active":"2024-02-19T15:27:21.039839+0000","last_peered":"2024-02-19T15:27:21.039839+0000","last_clean":"2024-02-19T15:27:21.039839+0000","last_became_active":"2024-02-19T15:27:21.038595+0000","last_became_peered":"2024-02-19T15:27:21.038595+0000","last_unstale":"2024-02-19T15:27:21.039839+0000","last_undegraded":"2024-02-19T15:27:21.039839+0000","last_fullsized":"2024-02-19T15:27:21.039839+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:27.660797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.027042+0000","last_change":"2024-02-19T15:27:21.027042+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.027042+0000","last_undegraded":"2024-02-19T15:27:21.027042+0000","last_fullsized":"2024-02-19T15:27:21.027042+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.018777+0000","last_change":"2024-02-19T15:27:21.017652+0000","last_active":"2024-02-19T15:27:21.018777+0000","last_peered":"2024-02-19T15:27:21.018777+0000","last_clean":"2024-02-19T15:27:21.018777+0000","last_became_active":"2024-02-19T15:27:21.017345+0000","last_became_peered":"2024-02-19T15:27:21.017345+0000","last_unstale":"2024-02-19T15:27:21.018777+0000","last_undegraded":"2024-02-19T15:27:21.018777+0000","last_fullsized":"2024-02-19T15:27:21.018777+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:21:04.018088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.042243+0000","last_change":"2024-02-19T15:27:22.035845+0000","last_active":"2024-02-19T15:27:22.042243+0000","last_peered":"2024-02-19T15:27:22.042243+0000","last_clean":"2024-02-19T15:27:22.042243+0000","last_became_active":"2024-02-19T15:27:22.035521+0000","last_became_peered":"2024-02-19T15:27:22.035521+0000","last_unstale":"2024-02-19T15:27:22.042243+0000","last_undegraded":"2024-02-19T15:27:22.042243+0000","last_fullsized":"2024-02-19T15:27:22.042243+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:10:54.441676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.038834+0000","last_change":"2024-02-19T15:27:21.034876+0000","last_active":"2024-02-19T15:27:21.038834+0000","last_peered":"2024-02-19T15:27:21.038834+0000","last_clean":"2024-02-19T15:27:21.038834+0000","last_became_active":"2024-02-19T15:27:21.032593+0000","last_became_peered":"2024-02-19T15:27:21.032593+0000","last_unstale":"2024-02-19T15:27:21.038834+0000","last_undegraded":"2024-02-19T15:27:21.038834+0000","last_fullsized":"2024-02-19T15:27:21.038834+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:38:39.139291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.022396+0000","last_change":"2024-02-19T15:27:21.022396+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.022396+0000","last_undegraded":"2024-02-19T15:27:21.022396+0000","last_fullsized":"2024-02-19T15:27:21.022396+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.029072+0000","last_change":"2024-02-19T15:27:21.025533+0000","last_active":"2024-02-19T15:27:22.029072+0000","last_peered":"2024-02-19T15:27:22.029072+0000","last_clean":"2024-02-19T15:27:22.029072+0000","last_became_active":"2024-02-19T15:27:21.025161+0000","last_became_peered":"2024-02-19T15:27:21.025161+0000","last_unstale":"2024-02-19T15:27:22.029072+0000","last_undegraded":"2024-02-19T15:27:22.029072+0000","last_fullsized":"2024-02-19T15:27:22.029072+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:59:09.129286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.023032+0000","last_change":"2024-02-19T15:27:21.023032+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.023032+0000","last_undegraded":"2024-02-19T15:27:21.023032+0000","last_fullsized":"2024-02-19T15:27:21.023032+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.023272+0000","last_change":"2024-02-19T15:27:21.021965+0000","last_active":"2024-02-19T15:27:21.023272+0000","last_peered":"2024-02-19T15:27:21.023272+0000","last_clean":"2024-02-19T15:27:21.023272+0000","last_became_active":"2024-02-19T15:27:21.021319+0000","last_became_peered":"2024-02-19T15:27:21.021319+0000","last_unstale":"2024-02-19T15:27:21.023272+0000","last_undegraded":"2024-02-19T15:27:21.023272+0000","last_fullsized":"2024-02-19T15:27:21.023272+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:58:20.368186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.041845+0000","last_change":"2024-02-19T15:27:22.037785+0000","last_active":"2024-02-19T15:27:22.041845+0000","last_peered":"2024-02-19T15:27:22.041845+0000","last_clean":"2024-02-19T15:27:22.041845+0000","last_became_active":"2024-02-19T15:27:22.037460+0000","last_became_peered":"2024-02-19T15:27:22.037460+0000","last_unstale":"2024-02-19T15:27:22.041845+0000","last_undegraded":"2024-02-19T15:27:22.041845+0000","last_fullsized":"2024-02-19T15:27:22.041845+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:46.258004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.031658+0000","last_change":"2024-02-19T15:27:21.028129+0000","last_active":"2024-02-19T15:27:21.031658+0000","last_peered":"2024-02-19T15:27:21.031658+0000","last_clean":"2024-02-19T15:27:21.031658+0000","last_became_active":"2024-02-19T15:27:21.027822+0000","last_became_peered":"2024-02-19T15:27:21.027822+0000","last_unstale":"2024-02-19T15:27:21.031658+0000","last_undegraded":"2024-02-19T15:27:21.031658+0000","last_fullsized":"2024-02-19T15:27:21.031658+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:26.546966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.021775+0000","last_change":"2024-02-19T15:27:21.021775+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.021775+0000","last_undegraded":"2024-02-19T15:27:21.021775+0000","last_fullsized":"2024-02-19T15:27:21.021775+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.033732+0000","last_change":"2024-02-19T15:27:21.032501+0000","last_active":"2024-02-19T15:27:21.033732+0000","last_peered":"2024-02-19T15:27:21.033732+0000","last_clean":"2024-02-19T15:27:21.033732+0000","last_became_active":"2024-02-19T15:27:21.032206+0000","last_became_peered":"2024-02-19T15:27:21.032206+0000","last_unstale":"2024-02-19T15:27:21.033732+0000","last_undegraded":"2024-02-19T15:27:21.033732+0000","last_fullsized":"2024-02-19T15:27:21.033732+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:25:30.120547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.022269+0000","last_change":"2024-02-19T15:27:21.022269+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.022269+0000","last_undegraded":"2024-02-19T15:27:21.022269+0000","last_fullsized":"2024-02-19T15:27:21.022269+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.036528+0000","last_change":"2024-02-19T15:27:21.035917+0000","last_active":"2024-02-19T15:27:21.036528+0000","last_peered":"2024-02-19T15:27:21.036528+0000","last_clean":"2024-02-19T15:27:21.036528+0000","last_became_active":"2024-02-19T15:27:21.035628+0000","last_became_peered":"2024-02-19T15:27:21.035628+0000","last_unstale":"2024-02-19T15:27:21.036528+0000","last_undegraded":"2024-02-19T15:27:21.036528+0000","last_fullsized":"2024-02-19T15:27:21.036528+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:03:18.300707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.014923+0000","last_change":"2024-02-19T15:27:21.014923+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.014923+0000","last_undegraded":"2024-02-19T15:27:21.014923+0000","last_fullsized":"2024-02-19T15:27:21.014923+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.032053+0000","last_change":"2024-02-19T15:27:21.029497+0000","last_active":"2024-02-19T15:27:21.032053+0000","last_peered":"2024-02-19T15:27:21.032053+0000","last_clean":"2024-02-19T15:27:21.032053+0000","last_became_active":"2024-02-19T15:27:21.029251+0000","last_became_peered":"2024-02-19T15:27:21.029251+0000","last_unstale":"2024-02-19T15:27:21.032053+0000","last_undegraded":"2024-02-19T15:27:21.032053+0000","last_fullsized":"2024-02-19T15:27:21.032053+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:16:56.947269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.022312+0000","last_change":"2024-02-19T15:27:21.022312+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.022312+0000","last_undegraded":"2024-02-19T15:27:21.022312+0000","last_fullsized":"2024-02-19T15:27:21.022312+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.027420+0000","last_change":"2024-02-19T15:27:21.024536+0000","last_active":"2024-02-19T15:27:21.027420+0000","last_peered":"2024-02-19T15:27:21.027420+0000","last_clean":"2024-02-19T15:27:21.027420+0000","last_became_active":"2024-02-19T15:27:21.020949+0000","last_became_peered":"2024-02-19T15:27:21.020949+0000","last_unstale":"2024-02-19T15:27:21.027420+0000","last_undegraded":"2024-02-19T15:27:21.027420+0000","last_fullsized":"2024-02-19T15:27:21.027420+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:16:06.605936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.025938+0000","last_change":"2024-02-19T15:27:21.025938+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.025938+0000","last_undegraded":"2024-02-19T15:27:21.025938+0000","last_fullsized":"2024-02-19T15:27:21.025938+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.036666+0000","last_change":"2024-02-19T15:27:21.036160+0000","last_active":"2024-02-19T15:27:21.036666+0000","last_peered":"2024-02-19T15:27:21.036666+0000","last_clean":"2024-02-19T15:27:21.036666+0000","last_became_active":"2024-02-19T15:27:21.035805+0000","last_became_peered":"2024-02-19T15:27:21.035805+0000","last_unstale":"2024-02-19T15:27:21.036666+0000","last_undegraded":"2024-02-19T15:27:21.036666+0000","last_fullsized":"2024-02-19T15:27:21.036666+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:51:06.106358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":320,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.018304+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:27:21.018304+0000","last_peered":"2024-02-19T15:27:21.018304+0000","last_clean":"2024-02-19T15:27:21.018304+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:27:21.018304+0000","last_undegraded":"2024-02-19T15:27:21.018304+0000","last_fullsized":"2024-02-19T15:27:21.018304+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509509,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1314816,"data_stored":563776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.622,"15min":0.62},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.66100000000000003,"15min":0.746},"min":{"1min":0.39100000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.8100000000000001,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.76100000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63800000000000001,"15min":0.65000000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.44,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.774,"5min":2.774,"15min":2.774},"last":0.58099999999999996}]},{"osd":2,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63100000000000001,"15min":0.63200000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.69899999999999995,"15min":0.68899999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.69299999999999995}]},{"osd":3,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":0.83499999999999996,"15min":0.72699999999999998},"min":{"1min":0.46600000000000003,"5min":0.438,"15min":0.438},"max":{"1min":14.045999999999999,"5min":14.045999999999999,"15min":14.045999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.67600000000000005,"15min":0.66200000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.85999999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.64500000000000002,"15min":0.61299999999999999},"min":{"1min":0.32400000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.69099999999999995,"15min":0.625},"min":{"1min":0.35599999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":2.8500000000000001,"5min":2.8500000000000001,"15min":2.8500000000000001},"last":0.83599999999999997}]},{"osd":5,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.61899999999999999},"min":{"1min":0.38200000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.72899999999999998,"15min":0.68500000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.73199999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.75900000000000001,"15min":0.68100000000000005},"min":{"1min":0.42799999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":3.0539999999999998,"5min":3.0539999999999998,"15min":3.0539999999999998},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.755,"15min":0.69299999999999995},"min":{"1min":0.439,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.256,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.91100000000000003}]}]},{"osd":7,"up_from":14,"seq":60129542213,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37200,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334640,"statfs":{"total":96636764160,"available":96598671360,"internally_reserved":0,"allocated":1896448,"data_stored":1147869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":2.016,"5min":0.88300000000000001,"15min":0.68500000000000005},"min":{"1min":0.51800000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":15.127000000000001,"5min":15.127000000000001,"15min":15.127000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.71899999999999997,"15min":0.69399999999999995},"min":{"1min":0.47499999999999998,"5min":0.376,"15min":0.376},"max":{"1min":6.0890000000000004,"5min":6.0890000000000004,"15min":6.0890000000000004},"last":0.63700000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":2.0910000000000002,"5min":0.93600000000000005,"15min":0.70199999999999996},"min":{"1min":0.47499999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":15.196999999999999,"5min":15.196999999999999,"15min":15.196999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.3100000000000001,"5min":0.81999999999999995,"15min":0.76000000000000001},"min":{"1min":0.45600000000000002,"5min":0.437,"15min":0.437},"max":{"1min":10.933999999999999,"5min":10.933999999999999,"15min":10.933999999999999},"last":0.80800000000000005}]},{"osd":2,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":0.79600000000000004,"15min":0.68600000000000005},"min":{"1min":0.45700000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":10.84,"5min":10.84,"15min":10.84},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.71199999999999997,"15min":0.69999999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":2.3170000000000002,"5min":2.508,"15min":2.508},"last":0.77700000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":1.6539999999999999,"5min":0.86199999999999999,"15min":0.72599999999999998},"min":{"1min":0.49199999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":17.094000000000001,"5min":17.094000000000001,"15min":17.094000000000001},"last":0.746},{"interface":"front","average":{"1min":1.226,"5min":0.81799999999999995,"15min":0.73099999999999998},"min":{"1min":0.46300000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":7.7190000000000003,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.85299999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.61299999999999999,"15min":0.54500000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.876},{"interface":"front","average":{"1min":2.266,"5min":0.95799999999999996,"15min":0.69499999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":26.625,"5min":26.625,"15min":26.625},"last":0.79600000000000004}]},{"osd":5,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":2.5350000000000001,"5min":1.0549999999999999,"15min":0.71699999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":31.844999999999999,"5min":31.844999999999999,"15min":31.844999999999999},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.6100000000000001,"5min":0.876,"15min":0.70399999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":16.776,"5min":16.776,"15min":16.776},"last":0.83499999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.70499999999999996,"15min":0.625},"min":{"1min":0.375,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.76400000000000001,"15min":0.68799999999999994},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":3.5099999999999998,"5min":3.5099999999999998,"15min":3.5099999999999998},"last":0.97599999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542214,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36616,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335224,"statfs":{"total":96636764160,"available":96599269376,"internally_reserved":0,"allocated":1302528,"data_stored":557501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.69699999999999995,"15min":0.65300000000000002},"min":{"1min":0.48399999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":2.6659999999999999,"5min":5.7789999999999999,"15min":5.7789999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73799999999999999,"15min":0.64600000000000002},"min":{"1min":0.45000000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.2370000000000001,"5min":12.359999999999999,"15min":12.359999999999999},"last":0.91700000000000004}]},{"osd":1,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.76600000000000001,"15min":0.67900000000000005},"min":{"1min":0.496,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.6499999999999999,"5min":10.938000000000001,"15min":10.938000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.82199999999999995,"15min":0.72399999999999998},"min":{"1min":0.45800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":3.625,"5min":10.531000000000001,"15min":10.531000000000001},"last":0.89500000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.81499999999999995,"15min":0.72699999999999998},"min":{"1min":0.503,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":2.589,"5min":12.925000000000001,"15min":12.925000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.74099999999999999,"15min":0.64500000000000002},"min":{"1min":0.46999999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89100000000000001,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.77300000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.96499999999999997,"15min":0.753},"min":{"1min":0.51000000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":5.492,"5min":26.161999999999999,"15min":26.161999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.76200000000000001},"min":{"1min":0.52700000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":6.4939999999999998,"5min":27.934999999999999,"15min":27.934999999999999},"last":0.86399999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":0.79900000000000004,"15min":0.63300000000000001},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":11.73,"5min":11.73,"15min":11.73},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.117,"5min":0.80200000000000005,"15min":0.70199999999999996},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":10.731,"5min":10.731,"15min":10.731},"last":0.68600000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.76900000000000002,"15min":0.67700000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.6320000000000001,"5min":6.5229999999999997,"15min":6.5229999999999997},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.248,"5min":0.92600000000000005,"15min":0.78400000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":10.432,"5min":10.432,"15min":10.432},"last":0.73499999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.78700000000000003,"15min":0.69799999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.9450000000000001,"5min":11.986000000000001,"15min":11.986000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.83099999999999996,"15min":0.73299999999999998},"min":{"1min":0.42999999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":6.5339999999999998,"5min":8.9770000000000003,"15min":8.9770000000000003},"last":0.749}]}]},{"osd":4,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36616,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335224,"statfs":{"total":96636764160,"available":96599269376,"internally_reserved":0,"allocated":1302528,"data_stored":557501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":4.1779999999999999,"5min":1.3089999999999999,"15min":0.83099999999999996},"min":{"1min":0.38,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":76.117000000000004,"5min":76.117000000000004,"15min":76.117000000000004},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.746,"5min":0.65300000000000002,"15min":0.61499999999999999},"min":{"1min":0.40000000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.2709999999999999,"5min":3.2709999999999999,"15min":3.2709999999999999},"last":0.42799999999999999}]},{"osd":1,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":4.4470000000000001,"5min":1.409,"15min":0.875},"min":{"1min":0.45800000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":80.513000000000005,"5min":80.513000000000005,"15min":80.513000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.70799999999999996,"15min":0.65500000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":7.181,"5min":7.181,"15min":7.181},"last":0.54100000000000004}]},{"osd":2,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":0.77900000000000003,"15min":0.70199999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":14.362,"5min":14.362,"15min":14.362},"last":0.67200000000000004},{"interface":"front","average":{"1min":4.6100000000000003,"5min":1.4690000000000001,"15min":0.93899999999999995},"min":{"1min":0.42299999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":84.093000000000004,"5min":84.093000000000004,"15min":84.093000000000004},"last":0.51100000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":4.6539999999999999,"5min":1.5049999999999999,"15min":0.94799999999999995},"min":{"1min":0.504,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":84.140000000000001,"5min":84.140000000000001,"15min":84.140000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.44400000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.56499999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":0.76500000000000001,"15min":0.64900000000000002},"min":{"1min":0.379,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":14.318,"5min":14.318,"15min":14.318},"last":0.69299999999999995},{"interface":"front","average":{"1min":4.274,"5min":1.427,"15min":0.92300000000000004},"min":{"1min":0.41099999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":76.164000000000001,"5min":76.164000000000001,"15min":76.164000000000001},"last":0.626}]},{"osd":6,"last update":"Mon Feb 19 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64100000000000001,"15min":0.58899999999999997},"min":{"1min":0.33500000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":3.5019999999999998,"5min":1.2589999999999999,"15min":0.84599999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":60.151000000000003,"5min":60.151000000000003,"15min":60.151000000000003},"last":0.60699999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":2.044,"5min":0.97999999999999998,"15min":0.76400000000000001},"min":{"1min":0.40699999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":29.036000000000001,"5min":29.036000000000001,"15min":29.036000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":1.0449999999999999,"5min":0.82099999999999995,"15min":0.755},"min":{"1min":0.50700000000000001,"5min":0.443,"15min":0.443},"max":{"1min":7.101,"5min":7.101,"15min":7.101},"last":0.64700000000000002}]}]},{"osd":3,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40772,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331068,"statfs":{"total":96636764160,"available":96595013632,"internally_reserved":0,"allocated":1302528,"data_stored":557501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41799999999999998,"15min":0.44400000000000001},"min":{"1min":0.23499999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.82699999999999996,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.5,"15min":0.63200000000000001},"min":{"1min":0.28899999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.0840000000000001,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.51200000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.437,"15min":0.443},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.54400000000000004,"15min":0.69199999999999995},"min":{"1min":0.316,"5min":0.31,"15min":0.31},"max":{"1min":0.85099999999999998,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.55400000000000005}]},{"osd":2,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55500000000000005,"15min":0.60699999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53700000000000003,"15min":0.58499999999999996},"min":{"1min":0.38400000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.57599999999999996}]},{"osd":4,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71299999999999997,"15min":0.68200000000000005},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.97999999999999998,"5min":1.004,"15min":1.004},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.75800000000000001,"15min":0.84599999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0169999999999999,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.88600000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.77000000000000002,"15min":0.84399999999999997},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.99099999999999999,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.73499999999999999,"15min":0.745},"min":{"1min":0.46999999999999997,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89700000000000002,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.92000000000000004}]},{"osd":6,"last update":"Mon Feb 19 15:27:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.79300000000000004,"15min":0.879},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.3180000000000001,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":1.052},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.77500000000000002,"15min":0.81000000000000005},"min":{"1min":0.53900000000000003,"5min":0.502,"15min":0.502},"max":{"1min":1.206,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":1.0349999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.755,"15min":0.77700000000000002},"min":{"1min":0.57099999999999995,"5min":0.504,"15min":0.504},"max":{"1min":0.996,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.80400000000000005,"15min":0.90800000000000003},"min":{"1min":0.54800000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.1919999999999999,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.98299999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1314816,"data_stored":563776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.60099999999999998,"15min":0.67500000000000004},"min":{"1min":0.26600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.48899999999999999,"5min":15.42,"15min":15.42},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.61199999999999999,"15min":0.85099999999999998},"min":{"1min":0.28199999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68700000000000006,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.38500000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.65500000000000003,"15min":0.86299999999999999},"min":{"1min":0.29399999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72799999999999998,"5min":9.5269999999999992,"15min":9.5269999999999992},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.83999999999999997,"15min":0.97199999999999998},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.75600000000000001,"5min":22.954999999999998,"15min":22.954999999999998},"last":0.53700000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.61499999999999999,"15min":0.76500000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.75,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.68700000000000006,"15min":0.80400000000000005},"min":{"1min":0.42399999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81399999999999995,"5min":7.734,"15min":7.734},"last":0.63700000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.86399999999999999,"15min":0.88600000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.97999999999999998,"5min":15.369,"15min":15.369},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.77800000000000002,"15min":0.876},"min":{"1min":0.51900000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93600000000000005,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.78800000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.83299999999999996,"15min":0.876},"min":{"1min":0.47199999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.006,"5min":7.6520000000000001,"15min":7.6520000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.93400000000000005,"15min":0.95699999999999996},"min":{"1min":0.496,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.026,"5min":15.391,"15min":15.391},"last":0.87}]},{"osd":6,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.90900000000000003,"15min":0.92800000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.92000000000000004,"5min":15.337,"15min":15.337},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.81399999999999995,"15min":0.93799999999999994},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.93700000000000006,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.93899999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.85399999999999998,"15min":0.96399999999999997},"min":{"1min":0.57099999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.98099999999999998,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.86099999999999999,"15min":0.93500000000000005},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94299999999999995,"5min":7.6929999999999996,"15min":7.6929999999999996},"last":0.85299999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542214,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40784,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331056,"statfs":{"total":96636764160,"available":96595001344,"internally_reserved":0,"allocated":1314816,"data_stored":563776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38600000000000001,"15min":0.42199999999999999},"min":{"1min":0.25700000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.68999999999999995,"5min":0.747,"15min":0.747},"last":0.432},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.434,"15min":0.46500000000000002},"min":{"1min":0.312,"5min":0.25,"15min":0.25},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.33200000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.44500000000000001},"min":{"1min":0.30499999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.47799999999999998,"15min":0.52300000000000002},"min":{"1min":0.32400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.74099999999999999,"5min":1.071,"15min":1.071},"last":0.66100000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57999999999999996,"15min":0.63},"min":{"1min":0.45500000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.1200000000000001,"5min":1.365,"15min":1.365},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.55800000000000005,"15min":0.625},"min":{"1min":0.35399999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0069999999999999,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.59299999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.67400000000000004,"15min":0.62},"min":{"1min":0.434,"5min":0.377,"15min":0.377},"max":{"1min":1.0409999999999999,"5min":1.127,"15min":1.127},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.69999999999999996,"15min":0.68200000000000005},"min":{"1min":0.56100000000000005,"5min":0.435,"15min":0.435},"max":{"1min":1.083,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.82899999999999996}]},{"osd":5,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.71199999999999997,"15min":0.69799999999999995},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.88800000000000001,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70899999999999996,"15min":0.68700000000000006},"min":{"1min":0.51200000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.96399999999999997,"5min":1.107,"15min":1.107},"last":0.745}]},{"osd":6,"last update":"Mon Feb 19 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.73199999999999998,"15min":0.72899999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.97799999999999998,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.755,"5min":0.74399999999999999,"15min":0.746},"min":{"1min":0.56299999999999994,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.1559999999999999,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.97099999999999997}]},{"osd":7,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.78400000000000003},"min":{"1min":0.52700000000000002,"5min":0.496,"15min":0.496},"max":{"1min":0.94599999999999995,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":1.0800000000000001},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.75},"min":{"1min":0.54500000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.99299999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":1.0509999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542212,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41360,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330480,"statfs":{"total":96636764160,"available":96594411520,"internally_reserved":0,"allocated":1896448,"data_stored":1147869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.45300000000000001,"15min":0.60999999999999999},"min":{"1min":0.24299999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.745,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.45800000000000002,"15min":0.48199999999999998},"min":{"1min":0.34200000000000003,"5min":0.252,"15min":0.252},"max":{"1min":0.77300000000000002,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.61399999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59999999999999998,"15min":0.76600000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.82399999999999995,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.56100000000000005,"15min":0.72699999999999998},"min":{"1min":0.35699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.85899999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.42699999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52000000000000002,"15min":0.66300000000000003},"min":{"1min":0.307,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.93799999999999994,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53300000000000003,"15min":0.60499999999999998},"min":{"1min":0.376,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.052,"5min":1.345,"15min":1.345},"last":0.40799999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.73199999999999998,"15min":0.73799999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.091,"5min":1.095,"15min":1.095},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.79500000000000004,"15min":0.90900000000000003},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.085,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.69199999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.71399999999999997,"15min":0.74299999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.036,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.73699999999999999,"15min":0.84099999999999997},"min":{"1min":0.53600000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.169,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.5}]},{"osd":6,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.76400000000000001,"15min":0.89600000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.1200000000000001,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.71999999999999997,"15min":0.751},"min":{"1min":0.50800000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.70799999999999996}]},{"osd":7,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.79600000000000004,"15min":0.92400000000000004},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.1399999999999999,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.78100000000000003,"15min":0.92300000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.149,"5min":4.915,"15min":4.915},"last":0.65200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:27:26.324 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:27:26.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.475+0000 7fb9cd400640 1 Processor -- start 2024-02-19T15:27:26.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.475+0000 7fb9cd400640 1 -- start start 2024-02-19T15:27:26.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.479+0000 7fb9cd400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb9c806d650 0x7fb9c80d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:26.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.479+0000 7fb9cd400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb9c80cded0 0x7fb9c80ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:26.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.479+0000 7fb9cd400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c80cce30 con 0x7fb9c80ceb40 2024-02-19T15:27:26.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.479+0000 7fb9cd400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c80cefd0 con 0x7fb9c80cded0 2024-02-19T15:27:26.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.479+0000 7fb9cd400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c80cd110 con 0x7fb9c806d650 2024-02-19T15:27:26.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.479+0000 7fb9bfe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb9c80cded0 0x7fb9c80ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:26.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.479+0000 7fb9bfe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb9c80cded0 0x7fb9c80ce600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55304/0 (socket says 172.21.15.47:55304) 2024-02-19T15:27:26.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.479+0000 7fb9bfe00640 1 -- 172.21.15.47:0/3616275108 learned_addr learned my addr 172.21.15.47:0/3616275108 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:26.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.479+0000 7fb9c6800640 1 --2- 172.21.15.47:0/3616275108 >> v2:172.21.15.116:3300/0 conn(0x7fb9c806d650 0x7fb9c80d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:26.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.479+0000 7fb9bfe00640 1 -- 172.21.15.47:0/3616275108 >> v2:172.21.15.116:3300/0 conn(0x7fb9c806d650 msgr2=0x7fb9c80d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:26.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.479+0000 7fb9bfe00640 1 --2- 172.21.15.47:0/3616275108 >> v2:172.21.15.116:3300/0 conn(0x7fb9c806d650 0x7fb9c80d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.479+0000 7fb9bfe00640 1 -- 172.21.15.47:0/3616275108 >> v1:172.21.15.116:6789/0 conn(0x7fb9c80ceb40 legacy=0x7fb9c8161d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:26.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.479+0000 7fb9bfe00640 1 -- 172.21.15.47:0/3616275108 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c80cd3f0 con 0x7fb9c80cded0 2024-02-19T15:27:26.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.483+0000 7fb9bfe00640 1 --2- 172.21.15.47:0/3616275108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9c80cded0 0x7fb9c80ce600 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8072160 tx=0x7fb9b001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44b7aae2d98a9a0e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:26.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.483+0000 7fb9c5e00640 1 -- 172.21.15.47:0/3616275108 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b00ad680 con 0x7fb9c80cded0 2024-02-19T15:27:26.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.483+0000 7fb9c5e00640 1 -- 172.21.15.47:0/3616275108 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b00ad7e0 con 0x7fb9c80cded0 2024-02-19T15:27:26.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.483+0000 7fb9c5e00640 1 -- 172.21.15.47:0/3616275108 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b00adb70 con 0x7fb9c80cded0 2024-02-19T15:27:26.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.483+0000 7fb9cd400640 1 -- 172.21.15.47:0/3616275108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9c80cded0 msgr2=0x7fb9c80ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:26.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.483+0000 7fb9cd400640 1 --2- 172.21.15.47:0/3616275108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9c80cded0 0x7fb9c80ce600 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8072160 tx=0x7fb9b001c460 comp rx=0 tx=0).stop 2024-02-19T15:27:26.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.483+0000 7fb9cd400640 1 -- 172.21.15.47:0/3616275108 shutdown_connections 2024-02-19T15:27:26.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.483+0000 7fb9cd400640 1 --2- 172.21.15.47:0/3616275108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9c80cded0 0x7fb9c80ce600 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.483+0000 7fb9cd400640 1 --2- 172.21.15.47:0/3616275108 >> v2:172.21.15.116:3300/0 conn(0x7fb9c806d650 0x7fb9c80d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.483+0000 7fb9cd400640 1 -- 172.21.15.47:0/3616275108 >> 172.21.15.47:0/3616275108 conn(0x7fb9c80d1bf0 msgr2=0x7fb9c80d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:26.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.483+0000 7fb9cd400640 1 -- 172.21.15.47:0/3616275108 shutdown_connections 2024-02-19T15:27:26.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.483+0000 7fb9cd400640 1 -- 172.21.15.47:0/3616275108 wait complete. 2024-02-19T15:27:26.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.483+0000 7fb9cd400640 1 Processor -- start 2024-02-19T15:27:26.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.483+0000 7fb9cd400640 1 -- start start 2024-02-19T15:27:26.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9cd400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9c806d650 0x7fb9c815ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:26.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9cd400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9c80cded0 0x7fb9c80c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:26.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9cd400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9c80ceb40 0x7fb9c815bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:26.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9cd400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80c7220 con 0x7fb9c80ceb40 2024-02-19T15:27:26.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9cd400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80c6980 con 0x7fb9c80cded0 2024-02-19T15:27:26.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9cd400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80c6c60 con 0x7fb9c806d650 2024-02-19T15:27:26.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9c7200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9c80ceb40 0x7fb9c815bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:26.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9c7200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9c80ceb40 0x7fb9c815bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55320/0 (socket says 172.21.15.47:55320) 2024-02-19T15:27:26.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9c7200640 1 -- 172.21.15.47:0/2322237289 learned_addr learned my addr 172.21.15.47:0/2322237289 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:26.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9c6800640 1 --2- 172.21.15.47:0/2322237289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9c806d650 0x7fb9c815ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:26.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9bfe00640 1 --2- 172.21.15.47:0/2322237289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9c80cded0 0x7fb9c80c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:26.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9c7200640 1 -- 172.21.15.47:0/2322237289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9c806d650 msgr2=0x7fb9c815ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:26.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9c7200640 1 --2- 172.21.15.47:0/2322237289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9c806d650 0x7fb9c815ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9c7200640 1 -- 172.21.15.47:0/2322237289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9c80cded0 msgr2=0x7fb9c80c86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:26.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9c7200640 1 --2- 172.21.15.47:0/2322237289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9c80cded0 0x7fb9c80c86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9c7200640 1 -- 172.21.15.47:0/2322237289 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b00094d0 con 0x7fb9c80ceb40 2024-02-19T15:27:26.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9c6800640 1 --2- 172.21.15.47:0/2322237289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9c806d650 0x7fb9c815ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:26.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9bfe00640 1 --2- 172.21.15.47:0/2322237289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9c80cded0 0x7fb9c80c86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:26.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9c7200640 1 --2- 172.21.15.47:0/2322237289 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9c80ceb40 0x7fb9c815bb80 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fb9c000c730 tx=0x7fb9c000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:26.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9bea00640 1 -- 172.21.15.47:0/2322237289 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c0003b80 con 0x7fb9c80ceb40 2024-02-19T15:27:26.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9bea00640 1 -- 172.21.15.47:0/2322237289 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c0003ce0 con 0x7fb9c80ceb40 2024-02-19T15:27:26.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9bea00640 1 -- 172.21.15.47:0/2322237289 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c000aa30 con 0x7fb9c80ceb40 2024-02-19T15:27:26.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9cd400640 1 -- 172.21.15.47:0/2322237289 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c80c6f40 con 0x7fb9c80ceb40 2024-02-19T15:27:26.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9cd400640 1 -- 172.21.15.47:0/2322237289 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c815c0c0 con 0x7fb9c80ceb40 2024-02-19T15:27:26.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9cd400640 1 -- 172.21.15.47:0/2322237289 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c815c2b0 con 0x7fb9c80ceb40 2024-02-19T15:27:26.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.487+0000 7fb9bea00640 1 -- 172.21.15.47:0/2322237289 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb9c000ab90 con 0x7fb9c80ceb40 2024-02-19T15:27:26.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.491+0000 7fb9bea00640 1 --2- 172.21.15.47:0/2322237289 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb9a4066e70 0x7fb9a4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:26.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.491+0000 7fb9bea00640 1 -- 172.21.15.47:0/2322237289 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb9c00847b0 con 0x7fb9c80ceb40 2024-02-19T15:27:26.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.491+0000 7fb9c6800640 1 --2- 172.21.15.47:0/2322237289 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb9a4066e70 0x7fb9a4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:26.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.491+0000 7fb9c6800640 1 --2- 172.21.15.47:0/2322237289 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb9a4066e70 0x7fb9a4069330 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fb9b80097a0 tx=0x7fb9b8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:26.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.495+0000 7fb9bea00640 1 -- 172.21.15.47:0/2322237289 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb9c815c2b0 con 0x7fb9c80ceb40 2024-02-19T15:27:26.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.651+0000 7fb9cd400640 1 -- 172.21.15.47:0/2322237289 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb9c80ccae0 con 0x7fb9a4066e70 2024-02-19T15:27:26.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.659+0000 7fb9bea00640 1 -- 172.21.15.47:0/2322237289 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284066 (secure 0 0 0) 0x7fb9b8009d80 con 0x7fb9a4066e70 2024-02-19T15:27:26.668 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:26.669 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:27:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.663+0000 7fb9cd400640 1 -- 172.21.15.47:0/2322237289 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb9a4066e70 msgr2=0x7fb9a4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.663+0000 7fb9cd400640 1 --2- 172.21.15.47:0/2322237289 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb9a4066e70 0x7fb9a4069330 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fb9b80097a0 tx=0x7fb9b8008040 comp rx=0 tx=0).stop 2024-02-19T15:27:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.663+0000 7fb9cd400640 1 -- 172.21.15.47:0/2322237289 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9c80ceb40 msgr2=0x7fb9c815bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.663+0000 7fb9cd400640 1 --2- 172.21.15.47:0/2322237289 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9c80ceb40 0x7fb9c815bb80 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fb9c000c730 tx=0x7fb9c000cbf0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.667+0000 7fb9cd400640 1 -- 172.21.15.47:0/2322237289 shutdown_connections 2024-02-19T15:27:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.667+0000 7fb9cd400640 1 --2- 172.21.15.47:0/2322237289 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb9a4066e70 0x7fb9a4069330 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.667+0000 7fb9cd400640 1 --2- 172.21.15.47:0/2322237289 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb9c80ceb40 0x7fb9c815bb80 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.667+0000 7fb9cd400640 1 --2- 172.21.15.47:0/2322237289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9c80cded0 0x7fb9c80c86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.667+0000 7fb9cd400640 1 --2- 172.21.15.47:0/2322237289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9c806d650 0x7fb9c815ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.667+0000 7fb9cd400640 1 -- 172.21.15.47:0/2322237289 >> 172.21.15.47:0/2322237289 conn(0x7fb9c80d1bf0 msgr2=0x7fb9c80cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.667+0000 7fb9cd400640 1 -- 172.21.15.47:0/2322237289 shutdown_connections 2024-02-19T15:27:26.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.667+0000 7fb9cd400640 1 -- 172.21.15.47:0/2322237289 wait complete. 2024-02-19T15:27:26.689 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-02-19T15:27:25.221972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590438,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":160,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":20,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":306604,"kb_used_data":11372,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754668116,"statfs":{"total":773094113280,"available":772780150784,"internally_reserved":0,"allocated":11644928,"data_stored":5659569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":302173746},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.216184"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.031361+0000","last_change":"2024-02-19T15:27:21.031361+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.031361+0000","last_undegraded":"2024-02-19T15:27:21.031361+0000","last_fullsized":"2024-02-19T15:27:21.031361+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.030203+0000","last_change":"2024-02-19T15:27:21.030203+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.030203+0000","last_undegraded":"2024-02-19T15:27:21.030203+0000","last_fullsized":"2024-02-19T15:27:21.030203+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.016897+0000","last_change":"2024-02-19T15:27:21.016897+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.016897+0000","last_undegraded":"2024-02-19T15:27:21.016897+0000","last_fullsized":"2024-02-19T15:27:21.016897+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.030394+0000","last_change":"2024-02-19T15:27:21.030394+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.030394+0000","last_undegraded":"2024-02-19T15:27:21.030394+0000","last_fullsized":"2024-02-19T15:27:21.030394+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.023740+0000","last_change":"2024-02-19T15:27:21.023740+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.023740+0000","last_undegraded":"2024-02-19T15:27:21.023740+0000","last_fullsized":"2024-02-19T15:27:21.023740+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.011541+0000","last_change":"2024-02-19T15:27:21.011541+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.011541+0000","last_undegraded":"2024-02-19T15:27:21.011541+0000","last_fullsized":"2024-02-19T15:27:21.011541+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.026169+0000","last_change":"2024-02-19T15:27:21.026169+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.026169+0000","last_undegraded":"2024-02-19T15:27:21.026169+0000","last_fullsized":"2024-02-19T15:27:21.026169+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.016613+0000","last_change":"2024-02-19T15:27:21.016613+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.016613+0000","last_undegraded":"2024-02-19T15:27:21.016613+0000","last_fullsized":"2024-02-19T15:27:21.016613+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.030819+0000","last_change":"2024-02-19T15:27:21.030819+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.030819+0000","last_undegraded":"2024-02-19T15:27:21.030819+0000","last_fullsized":"2024-02-19T15:27:21.030819+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.029180+0000","last_change":"2024-02-19T15:27:21.029180+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.029180+0000","last_undegraded":"2024-02-19T15:27:21.029180+0000","last_fullsized":"2024-02-19T15:27:21.029180+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.026903+0000","last_change":"2024-02-19T15:27:21.026903+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.026903+0000","last_undegraded":"2024-02-19T15:27:21.026903+0000","last_fullsized":"2024-02-19T15:27:21.026903+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.028102+0000","last_change":"2024-02-19T15:27:21.028102+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.028102+0000","last_undegraded":"2024-02-19T15:27:21.028102+0000","last_fullsized":"2024-02-19T15:27:21.028102+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.015426+0000","last_change":"2024-02-19T15:27:21.015426+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.015426+0000","last_undegraded":"2024-02-19T15:27:21.015426+0000","last_fullsized":"2024-02-19T15:27:21.015426+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.016518+0000","last_change":"2024-02-19T15:27:21.016518+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.016518+0000","last_undegraded":"2024-02-19T15:27:21.016518+0000","last_fullsized":"2024-02-19T15:27:21.016518+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.011705+0000","last_change":"2024-02-19T15:27:21.011705+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.011705+0000","last_undegraded":"2024-02-19T15:27:21.011705+0000","last_fullsized":"2024-02-19T15:27:21.011705+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.026401+0000","last_change":"2024-02-19T15:27:21.026401+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.026401+0000","last_undegraded":"2024-02-19T15:27:21.026401+0000","last_fullsized":"2024-02-19T15:27:21.026401+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.025923+0000","last_change":"2024-02-19T15:27:21.025923+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.025923+0000","last_undegraded":"2024-02-19T15:27:21.025923+0000","last_fullsized":"2024-02-19T15:27:21.025923+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.042266+0000","last_change":"2024-02-19T15:27:22.036471+0000","last_active":"2024-02-19T15:27:22.042266+0000","last_peered":"2024-02-19T15:27:22.042266+0000","last_clean":"2024-02-19T15:27:22.042266+0000","last_became_active":"2024-02-19T15:27:22.036201+0000","last_became_peered":"2024-02-19T15:27:22.036201+0000","last_unstale":"2024-02-19T15:27:22.042266+0000","last_undegraded":"2024-02-19T15:27:22.042266+0000","last_fullsized":"2024-02-19T15:27:22.042266+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:28:59.516714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.016248+0000","last_change":"2024-02-19T15:27:21.016248+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.016248+0000","last_undegraded":"2024-02-19T15:27:21.016248+0000","last_fullsized":"2024-02-19T15:27:21.016248+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024313+0000","last_change":"2024-02-19T15:27:21.024313+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024313+0000","last_undegraded":"2024-02-19T15:27:21.024313+0000","last_fullsized":"2024-02-19T15:27:21.024313+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024688+0000","last_change":"2024-02-19T15:27:21.024688+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024688+0000","last_undegraded":"2024-02-19T15:27:21.024688+0000","last_fullsized":"2024-02-19T15:27:21.024688+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.033511+0000","last_change":"2024-02-19T15:27:21.033511+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.033511+0000","last_undegraded":"2024-02-19T15:27:21.033511+0000","last_fullsized":"2024-02-19T15:27:21.033511+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024070+0000","last_change":"2024-02-19T15:27:21.024070+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024070+0000","last_undegraded":"2024-02-19T15:27:21.024070+0000","last_fullsized":"2024-02-19T15:27:21.024070+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.033844+0000","last_change":"2024-02-19T15:27:21.033844+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.033844+0000","last_undegraded":"2024-02-19T15:27:21.033844+0000","last_fullsized":"2024-02-19T15:27:21.033844+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024026+0000","last_change":"2024-02-19T15:27:21.024026+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024026+0000","last_undegraded":"2024-02-19T15:27:21.024026+0000","last_fullsized":"2024-02-19T15:27:21.024026+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.025203+0000","last_change":"2024-02-19T15:27:21.025203+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.025203+0000","last_undegraded":"2024-02-19T15:27:21.025203+0000","last_fullsized":"2024-02-19T15:27:21.025203+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.012320+0000","last_change":"2024-02-19T15:27:21.012320+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.012320+0000","last_undegraded":"2024-02-19T15:27:21.012320+0000","last_fullsized":"2024-02-19T15:27:21.012320+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.033395+0000","last_change":"2024-02-19T15:27:21.033395+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.033395+0000","last_undegraded":"2024-02-19T15:27:21.033395+0000","last_fullsized":"2024-02-19T15:27:21.033395+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024077+0000","last_change":"2024-02-19T15:27:21.024077+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024077+0000","last_undegraded":"2024-02-19T15:27:21.024077+0000","last_fullsized":"2024-02-19T15:27:21.024077+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.021450+0000","last_change":"2024-02-19T15:27:21.021450+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.021450+0000","last_undegraded":"2024-02-19T15:27:21.021450+0000","last_fullsized":"2024-02-19T15:27:21.021450+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.026040+0000","last_change":"2024-02-19T15:27:21.026040+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.026040+0000","last_undegraded":"2024-02-19T15:27:21.026040+0000","last_fullsized":"2024-02-19T15:27:21.026040+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.011531+0000","last_change":"2024-02-19T15:27:21.011531+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.011531+0000","last_undegraded":"2024-02-19T15:27:21.011531+0000","last_fullsized":"2024-02-19T15:27:21.011531+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.013560+0000","last_change":"2024-02-19T15:27:21.013560+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.013560+0000","last_undegraded":"2024-02-19T15:27:21.013560+0000","last_fullsized":"2024-02-19T15:27:21.013560+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.033484+0000","last_change":"2024-02-19T15:27:21.030063+0000","last_active":"2024-02-19T15:27:21.033484+0000","last_peered":"2024-02-19T15:27:21.033484+0000","last_clean":"2024-02-19T15:27:21.033484+0000","last_became_active":"2024-02-19T15:27:21.029780+0000","last_became_peered":"2024-02-19T15:27:21.029780+0000","last_unstale":"2024-02-19T15:27:21.033484+0000","last_undegraded":"2024-02-19T15:27:21.033484+0000","last_fullsized":"2024-02-19T15:27:21.033484+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:15.014562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.054040+0000","last_change":"2024-02-19T15:27:22.048726+0000","last_active":"2024-02-19T15:27:22.054040+0000","last_peered":"2024-02-19T15:27:22.054040+0000","last_clean":"2024-02-19T15:27:22.054040+0000","last_became_active":"2024-02-19T15:27:22.048437+0000","last_became_peered":"2024-02-19T15:27:22.048437+0000","last_unstale":"2024-02-19T15:27:22.054040+0000","last_undegraded":"2024-02-19T15:27:22.054040+0000","last_fullsized":"2024-02-19T15:27:22.054040+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:50:16.869685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.030517+0000","last_change":"2024-02-19T15:27:21.027135+0000","last_active":"2024-02-19T15:27:21.030517+0000","last_peered":"2024-02-19T15:27:21.030517+0000","last_clean":"2024-02-19T15:27:21.030517+0000","last_became_active":"2024-02-19T15:27:21.026870+0000","last_became_peered":"2024-02-19T15:27:21.026870+0000","last_unstale":"2024-02-19T15:27:21.030517+0000","last_undegraded":"2024-02-19T15:27:21.030517+0000","last_fullsized":"2024-02-19T15:27:21.030517+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:51.292175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.012534+0000","last_change":"2024-02-19T15:27:21.012534+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.012534+0000","last_undegraded":"2024-02-19T15:27:21.012534+0000","last_fullsized":"2024-02-19T15:27:21.012534+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.038196+0000","last_change":"2024-02-19T15:27:21.029872+0000","last_active":"2024-02-19T15:27:21.038196+0000","last_peered":"2024-02-19T15:27:21.038196+0000","last_clean":"2024-02-19T15:27:21.038196+0000","last_became_active":"2024-02-19T15:27:21.029590+0000","last_became_peered":"2024-02-19T15:27:21.029590+0000","last_unstale":"2024-02-19T15:27:21.038196+0000","last_undegraded":"2024-02-19T15:27:21.038196+0000","last_fullsized":"2024-02-19T15:27:21.038196+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:32:04.524076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.015800+0000","last_change":"2024-02-19T15:27:21.015800+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.015800+0000","last_undegraded":"2024-02-19T15:27:21.015800+0000","last_fullsized":"2024-02-19T15:27:21.015800+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.043779+0000","last_change":"2024-02-19T15:27:21.042870+0000","last_active":"2024-02-19T15:27:21.043779+0000","last_peered":"2024-02-19T15:27:21.043779+0000","last_clean":"2024-02-19T15:27:21.043779+0000","last_became_active":"2024-02-19T15:27:21.041700+0000","last_became_peered":"2024-02-19T15:27:21.041700+0000","last_unstale":"2024-02-19T15:27:21.043779+0000","last_undegraded":"2024-02-19T15:27:21.043779+0000","last_fullsized":"2024-02-19T15:27:21.043779+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:21:53.467732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.022487+0000","last_change":"2024-02-19T15:27:21.022487+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.022487+0000","last_undegraded":"2024-02-19T15:27:21.022487+0000","last_fullsized":"2024-02-19T15:27:21.022487+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.041473+0000","last_change":"2024-02-19T15:27:21.040738+0000","last_active":"2024-02-19T15:27:21.041473+0000","last_peered":"2024-02-19T15:27:21.041473+0000","last_clean":"2024-02-19T15:27:21.041473+0000","last_became_active":"2024-02-19T15:27:21.040492+0000","last_became_peered":"2024-02-19T15:27:21.040492+0000","last_unstale":"2024-02-19T15:27:21.041473+0000","last_undegraded":"2024-02-19T15:27:21.041473+0000","last_fullsized":"2024-02-19T15:27:21.041473+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:38:35.042142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.023070+0000","last_change":"2024-02-19T15:27:21.023070+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.023070+0000","last_undegraded":"2024-02-19T15:27:21.023070+0000","last_fullsized":"2024-02-19T15:27:21.023070+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.039620+0000","last_change":"2024-02-19T15:27:21.037989+0000","last_active":"2024-02-19T15:27:21.039620+0000","last_peered":"2024-02-19T15:27:21.039620+0000","last_clean":"2024-02-19T15:27:21.039620+0000","last_became_active":"2024-02-19T15:27:21.037527+0000","last_became_peered":"2024-02-19T15:27:21.037527+0000","last_unstale":"2024-02-19T15:27:21.039620+0000","last_undegraded":"2024-02-19T15:27:21.039620+0000","last_fullsized":"2024-02-19T15:27:21.039620+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:59:16.122985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024991+0000","last_change":"2024-02-19T15:27:21.024991+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024991+0000","last_undegraded":"2024-02-19T15:27:21.024991+0000","last_fullsized":"2024-02-19T15:27:21.024991+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.032693+0000","last_change":"2024-02-19T15:27:21.031349+0000","last_active":"2024-02-19T15:27:21.032693+0000","last_peered":"2024-02-19T15:27:21.032693+0000","last_clean":"2024-02-19T15:27:21.032693+0000","last_became_active":"2024-02-19T15:27:21.031075+0000","last_became_peered":"2024-02-19T15:27:21.031075+0000","last_unstale":"2024-02-19T15:27:21.032693+0000","last_undegraded":"2024-02-19T15:27:21.032693+0000","last_fullsized":"2024-02-19T15:27:21.032693+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:25:19.941750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.023512+0000","last_change":"2024-02-19T15:27:21.023512+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.023512+0000","last_undegraded":"2024-02-19T15:27:21.023512+0000","last_fullsized":"2024-02-19T15:27:21.023512+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.030110+0000","last_change":"2024-02-19T15:27:21.025268+0000","last_active":"2024-02-19T15:27:21.030110+0000","last_peered":"2024-02-19T15:27:21.030110+0000","last_clean":"2024-02-19T15:27:21.030110+0000","last_became_active":"2024-02-19T15:27:21.022655+0000","last_became_peered":"2024-02-19T15:27:21.022655+0000","last_unstale":"2024-02-19T15:27:21.030110+0000","last_undegraded":"2024-02-19T15:27:21.030110+0000","last_fullsized":"2024-02-19T15:27:21.030110+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:44:25.168118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.021162+0000","last_change":"2024-02-19T15:27:21.021162+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.021162+0000","last_undegraded":"2024-02-19T15:27:21.021162+0000","last_fullsized":"2024-02-19T15:27:21.021162+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.043234+0000","last_change":"2024-02-19T15:27:21.041198+0000","last_active":"2024-02-19T15:27:21.043234+0000","last_peered":"2024-02-19T15:27:21.043234+0000","last_clean":"2024-02-19T15:27:21.043234+0000","last_became_active":"2024-02-19T15:27:21.040821+0000","last_became_peered":"2024-02-19T15:27:21.040821+0000","last_unstale":"2024-02-19T15:27:21.043234+0000","last_undegraded":"2024-02-19T15:27:21.043234+0000","last_fullsized":"2024-02-19T15:27:21.043234+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:15:15.162474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.050040+0000","last_change":"2024-02-19T15:27:22.043233+0000","last_active":"2024-02-19T15:27:22.050040+0000","last_peered":"2024-02-19T15:27:22.050040+0000","last_clean":"2024-02-19T15:27:22.050040+0000","last_became_active":"2024-02-19T15:27:22.042812+0000","last_became_peered":"2024-02-19T15:27:22.042812+0000","last_unstale":"2024-02-19T15:27:22.050040+0000","last_undegraded":"2024-02-19T15:27:22.050040+0000","last_fullsized":"2024-02-19T15:27:22.050040+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:02:10.136155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.039592+0000","last_change":"2024-02-19T15:27:21.036912+0000","last_active":"2024-02-19T15:27:21.039592+0000","last_peered":"2024-02-19T15:27:21.039592+0000","last_clean":"2024-02-19T15:27:21.039592+0000","last_became_active":"2024-02-19T15:27:21.036227+0000","last_became_peered":"2024-02-19T15:27:21.036227+0000","last_unstale":"2024-02-19T15:27:21.039592+0000","last_undegraded":"2024-02-19T15:27:21.039592+0000","last_fullsized":"2024-02-19T15:27:21.039592+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:15:01.148828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.032945+0000","last_change":"2024-02-19T15:27:21.032945+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.032945+0000","last_undegraded":"2024-02-19T15:27:21.032945+0000","last_fullsized":"2024-02-19T15:27:21.032945+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.023292+0000","last_change":"2024-02-19T15:27:21.022250+0000","last_active":"2024-02-19T15:27:21.023292+0000","last_peered":"2024-02-19T15:27:21.023292+0000","last_clean":"2024-02-19T15:27:21.023292+0000","last_became_active":"2024-02-19T15:27:21.021690+0000","last_became_peered":"2024-02-19T15:27:21.021690+0000","last_unstale":"2024-02-19T15:27:21.023292+0000","last_undegraded":"2024-02-19T15:27:21.023292+0000","last_fullsized":"2024-02-19T15:27:21.023292+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:41.432608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.021714+0000","last_change":"2024-02-19T15:27:21.021714+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.021714+0000","last_undegraded":"2024-02-19T15:27:21.021714+0000","last_fullsized":"2024-02-19T15:27:21.021714+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.037376+0000","last_change":"2024-02-19T15:27:21.031559+0000","last_active":"2024-02-19T15:27:21.037376+0000","last_peered":"2024-02-19T15:27:21.037376+0000","last_clean":"2024-02-19T15:27:21.037376+0000","last_became_active":"2024-02-19T15:27:21.031280+0000","last_became_peered":"2024-02-19T15:27:21.031280+0000","last_unstale":"2024-02-19T15:27:21.037376+0000","last_undegraded":"2024-02-19T15:27:21.037376+0000","last_fullsized":"2024-02-19T15:27:21.037376+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:48.463248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.029223+0000","last_change":"2024-02-19T15:27:21.029223+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.029223+0000","last_undegraded":"2024-02-19T15:27:21.029223+0000","last_fullsized":"2024-02-19T15:27:21.029223+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.025611+0000","last_change":"2024-02-19T15:27:21.019044+0000","last_active":"2024-02-19T15:27:21.025611+0000","last_peered":"2024-02-19T15:27:21.025611+0000","last_clean":"2024-02-19T15:27:21.025611+0000","last_became_active":"2024-02-19T15:27:21.018732+0000","last_became_peered":"2024-02-19T15:27:21.018732+0000","last_unstale":"2024-02-19T15:27:21.025611+0000","last_undegraded":"2024-02-19T15:27:21.025611+0000","last_fullsized":"2024-02-19T15:27:21.025611+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:02:10.142351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.033846+0000","last_change":"2024-02-19T15:27:21.033846+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.033846+0000","last_undegraded":"2024-02-19T15:27:21.033846+0000","last_fullsized":"2024-02-19T15:27:21.033846+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.027358+0000","last_change":"2024-02-19T15:27:21.022352+0000","last_active":"2024-02-19T15:27:22.027358+0000","last_peered":"2024-02-19T15:27:22.027358+0000","last_clean":"2024-02-19T15:27:22.027358+0000","last_became_active":"2024-02-19T15:27:21.021353+0000","last_became_peered":"2024-02-19T15:27:21.021353+0000","last_unstale":"2024-02-19T15:27:22.027358+0000","last_undegraded":"2024-02-19T15:27:22.027358+0000","last_fullsized":"2024-02-19T15:27:22.027358+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:05:11.130486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.029752+0000","last_change":"2024-02-19T15:27:21.029752+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.029752+0000","last_undegraded":"2024-02-19T15:27:21.029752+0000","last_fullsized":"2024-02-19T15:27:21.029752+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.043774+0000","last_change":"2024-02-19T15:27:21.042396+0000","last_active":"2024-02-19T15:27:21.043774+0000","last_peered":"2024-02-19T15:27:21.043774+0000","last_clean":"2024-02-19T15:27:21.043774+0000","last_became_active":"2024-02-19T15:27:21.042017+0000","last_became_peered":"2024-02-19T15:27:21.042017+0000","last_unstale":"2024-02-19T15:27:21.043774+0000","last_undegraded":"2024-02-19T15:27:21.043774+0000","last_fullsized":"2024-02-19T15:27:21.043774+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:54:06.234893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024250+0000","last_change":"2024-02-19T15:27:21.024250+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024250+0000","last_undegraded":"2024-02-19T15:27:21.024250+0000","last_fullsized":"2024-02-19T15:27:21.024250+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.031024+0000","last_change":"2024-02-19T15:27:21.027023+0000","last_active":"2024-02-19T15:27:21.031024+0000","last_peered":"2024-02-19T15:27:21.031024+0000","last_clean":"2024-02-19T15:27:21.031024+0000","last_became_active":"2024-02-19T15:27:21.026565+0000","last_became_peered":"2024-02-19T15:27:21.026565+0000","last_unstale":"2024-02-19T15:27:21.031024+0000","last_undegraded":"2024-02-19T15:27:21.031024+0000","last_fullsized":"2024-02-19T15:27:21.031024+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:52:51.983435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.013930+0000","last_change":"2024-02-19T15:27:21.013930+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.013930+0000","last_undegraded":"2024-02-19T15:27:21.013930+0000","last_fullsized":"2024-02-19T15:27:21.013930+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.039502+0000","last_change":"2024-02-19T15:27:21.033163+0000","last_active":"2024-02-19T15:27:21.039502+0000","last_peered":"2024-02-19T15:27:21.039502+0000","last_clean":"2024-02-19T15:27:21.039502+0000","last_became_active":"2024-02-19T15:27:21.032778+0000","last_became_peered":"2024-02-19T15:27:21.032778+0000","last_unstale":"2024-02-19T15:27:21.039502+0000","last_undegraded":"2024-02-19T15:27:21.039502+0000","last_fullsized":"2024-02-19T15:27:21.039502+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:35:31.450513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.015439+0000","last_change":"2024-02-19T15:27:21.015439+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.015439+0000","last_undegraded":"2024-02-19T15:27:21.015439+0000","last_fullsized":"2024-02-19T15:27:21.015439+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.047598+0000","last_change":"2024-02-19T15:27:21.033698+0000","last_active":"2024-02-19T15:27:22.047598+0000","last_peered":"2024-02-19T15:27:22.047598+0000","last_clean":"2024-02-19T15:27:22.047598+0000","last_became_active":"2024-02-19T15:27:21.033354+0000","last_became_peered":"2024-02-19T15:27:21.033354+0000","last_unstale":"2024-02-19T15:27:22.047598+0000","last_undegraded":"2024-02-19T15:27:22.047598+0000","last_fullsized":"2024-02-19T15:27:22.047598+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:42.028837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.015175+0000","last_change":"2024-02-19T15:27:21.015175+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.015175+0000","last_undegraded":"2024-02-19T15:27:21.015175+0000","last_fullsized":"2024-02-19T15:27:21.015175+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.033403+0000","last_change":"2024-02-19T15:27:21.031371+0000","last_active":"2024-02-19T15:27:21.033403+0000","last_peered":"2024-02-19T15:27:21.033403+0000","last_clean":"2024-02-19T15:27:21.033403+0000","last_became_active":"2024-02-19T15:27:21.031066+0000","last_became_peered":"2024-02-19T15:27:21.031066+0000","last_unstale":"2024-02-19T15:27:21.033403+0000","last_undegraded":"2024-02-19T15:27:21.033403+0000","last_fullsized":"2024-02-19T15:27:21.033403+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:27.660797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.015962+0000","last_change":"2024-02-19T15:27:21.015962+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.015962+0000","last_undegraded":"2024-02-19T15:27:21.015962+0000","last_fullsized":"2024-02-19T15:27:21.015962+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.039981+0000","last_change":"2024-02-19T15:27:21.026803+0000","last_active":"2024-02-19T15:27:21.039981+0000","last_peered":"2024-02-19T15:27:21.039981+0000","last_clean":"2024-02-19T15:27:21.039981+0000","last_became_active":"2024-02-19T15:27:21.025817+0000","last_became_peered":"2024-02-19T15:27:21.025817+0000","last_unstale":"2024-02-19T15:27:21.039981+0000","last_undegraded":"2024-02-19T15:27:21.039981+0000","last_fullsized":"2024-02-19T15:27:21.039981+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:15.014562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.023713+0000","last_change":"2024-02-19T15:27:21.023713+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.023713+0000","last_undegraded":"2024-02-19T15:27:21.023713+0000","last_fullsized":"2024-02-19T15:27:21.023713+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.043399+0000","last_change":"2024-02-19T15:27:21.042086+0000","last_active":"2024-02-19T15:27:21.043399+0000","last_peered":"2024-02-19T15:27:21.043399+0000","last_clean":"2024-02-19T15:27:21.043399+0000","last_became_active":"2024-02-19T15:27:21.041580+0000","last_became_peered":"2024-02-19T15:27:21.041580+0000","last_unstale":"2024-02-19T15:27:21.043399+0000","last_undegraded":"2024-02-19T15:27:21.043399+0000","last_fullsized":"2024-02-19T15:27:21.043399+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:20:44.224840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.016067+0000","last_change":"2024-02-19T15:27:21.016067+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.016067+0000","last_undegraded":"2024-02-19T15:27:21.016067+0000","last_fullsized":"2024-02-19T15:27:21.016067+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.039839+0000","last_change":"2024-02-19T15:27:21.038892+0000","last_active":"2024-02-19T15:27:21.039839+0000","last_peered":"2024-02-19T15:27:21.039839+0000","last_clean":"2024-02-19T15:27:21.039839+0000","last_became_active":"2024-02-19T15:27:21.038595+0000","last_became_peered":"2024-02-19T15:27:21.038595+0000","last_unstale":"2024-02-19T15:27:21.039839+0000","last_undegraded":"2024-02-19T15:27:21.039839+0000","last_fullsized":"2024-02-19T15:27:21.039839+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:27.660797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.027042+0000","last_change":"2024-02-19T15:27:21.027042+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.027042+0000","last_undegraded":"2024-02-19T15:27:21.027042+0000","last_fullsized":"2024-02-19T15:27:21.027042+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.018777+0000","last_change":"2024-02-19T15:27:21.017652+0000","last_active":"2024-02-19T15:27:21.018777+0000","last_peered":"2024-02-19T15:27:21.018777+0000","last_clean":"2024-02-19T15:27:21.018777+0000","last_became_active":"2024-02-19T15:27:21.017345+0000","last_became_peered":"2024-02-19T15:27:21.017345+0000","last_unstale":"2024-02-19T15:27:21.018777+0000","last_undegraded":"2024-02-19T15:27:21.018777+0000","last_fullsized":"2024-02-19T15:27:21.018777+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:21:04.018088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.042243+0000","last_change":"2024-02-19T15:27:22.035845+0000","last_active":"2024-02-19T15:27:22.042243+0000","last_peered":"2024-02-19T15:27:22.042243+0000","last_clean":"2024-02-19T15:27:22.042243+0000","last_became_active":"2024-02-19T15:27:22.035521+0000","last_became_peered":"2024-02-19T15:27:22.035521+0000","last_unstale":"2024-02-19T15:27:22.042243+0000","last_undegraded":"2024-02-19T15:27:22.042243+0000","last_fullsized":"2024-02-19T15:27:22.042243+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:10:54.441676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.038834+0000","last_change":"2024-02-19T15:27:21.034876+0000","last_active":"2024-02-19T15:27:21.038834+0000","last_peered":"2024-02-19T15:27:21.038834+0000","last_clean":"2024-02-19T15:27:21.038834+0000","last_became_active":"2024-02-19T15:27:21.032593+0000","last_became_peered":"2024-02-19T15:27:21.032593+0000","last_unstale":"2024-02-19T15:27:21.038834+0000","last_undegraded":"2024-02-19T15:27:21.038834+0000","last_fullsized":"2024-02-19T15:27:21.038834+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:38:39.139291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.022396+0000","last_change":"2024-02-19T15:27:21.022396+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.022396+0000","last_undegraded":"2024-02-19T15:27:21.022396+0000","last_fullsized":"2024-02-19T15:27:21.022396+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.029072+0000","last_change":"2024-02-19T15:27:21.025533+0000","last_active":"2024-02-19T15:27:22.029072+0000","last_peered":"2024-02-19T15:27:22.029072+0000","last_clean":"2024-02-19T15:27:22.029072+0000","last_became_active":"2024-02-19T15:27:21.025161+0000","last_became_peered":"2024-02-19T15:27:21.025161+0000","last_unstale":"2024-02-19T15:27:22.029072+0000","last_undegraded":"2024-02-19T15:27:22.029072+0000","last_fullsized":"2024-02-19T15:27:22.029072+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:59:09.129286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.023032+0000","last_change":"2024-02-19T15:27:21.023032+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.023032+0000","last_undegraded":"2024-02-19T15:27:21.023032+0000","last_fullsized":"2024-02-19T15:27:21.023032+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.023272+0000","last_change":"2024-02-19T15:27:21.021965+0000","last_active":"2024-02-19T15:27:21.023272+0000","last_peered":"2024-02-19T15:27:21.023272+0000","last_clean":"2024-02-19T15:27:21.023272+0000","last_became_active":"2024-02-19T15:27:21.021319+0000","last_became_peered":"2024-02-19T15:27:21.021319+0000","last_unstale":"2024-02-19T15:27:21.023272+0000","last_undegraded":"2024-02-19T15:27:21.023272+0000","last_fullsized":"2024-02-19T15:27:21.023272+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:58:20.368186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.041845+0000","last_change":"2024-02-19T15:27:22.037785+0000","last_active":"2024-02-19T15:27:22.041845+0000","last_peered":"2024-02-19T15:27:22.041845+0000","last_clean":"2024-02-19T15:27:22.041845+0000","last_became_active":"2024-02-19T15:27:22.037460+0000","last_became_peered":"2024-02-19T15:27:22.037460+0000","last_unstale":"2024-02-19T15:27:22.041845+0000","last_undegraded":"2024-02-19T15:27:22.041845+0000","last_fullsized":"2024-02-19T15:27:22.041845+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:46.258004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.031658+0000","last_change":"2024-02-19T15:27:21.028129+0000","last_active":"2024-02-19T15:27:21.031658+0000","last_peered":"2024-02-19T15:27:21.031658+0000","last_clean":"2024-02-19T15:27:21.031658+0000","last_became_active":"2024-02-19T15:27:21.027822+0000","last_became_peered":"2024-02-19T15:27:21.027822+0000","last_unstale":"2024-02-19T15:27:21.031658+0000","last_undegraded":"2024-02-19T15:27:21.031658+0000","last_fullsized":"2024-02-19T15:27:21.031658+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:26.546966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.021775+0000","last_change":"2024-02-19T15:27:21.021775+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.021775+0000","last_undegraded":"2024-02-19T15:27:21.021775+0000","last_fullsized":"2024-02-19T15:27:21.021775+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.033732+0000","last_change":"2024-02-19T15:27:21.032501+0000","last_active":"2024-02-19T15:27:21.033732+0000","last_peered":"2024-02-19T15:27:21.033732+0000","last_clean":"2024-02-19T15:27:21.033732+0000","last_became_active":"2024-02-19T15:27:21.032206+0000","last_became_peered":"2024-02-19T15:27:21.032206+0000","last_unstale":"2024-02-19T15:27:21.033732+0000","last_undegraded":"2024-02-19T15:27:21.033732+0000","last_fullsized":"2024-02-19T15:27:21.033732+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:25:30.120547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.022269+0000","last_change":"2024-02-19T15:27:21.022269+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.022269+0000","last_undegraded":"2024-02-19T15:27:21.022269+0000","last_fullsized":"2024-02-19T15:27:21.022269+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.036528+0000","last_change":"2024-02-19T15:27:21.035917+0000","last_active":"2024-02-19T15:27:21.036528+0000","last_peered":"2024-02-19T15:27:21.036528+0000","last_clean":"2024-02-19T15:27:21.036528+0000","last_became_active":"2024-02-19T15:27:21.035628+0000","last_became_peered":"2024-02-19T15:27:21.035628+0000","last_unstale":"2024-02-19T15:27:21.036528+0000","last_undegraded":"2024-02-19T15:27:21.036528+0000","last_fullsized":"2024-02-19T15:27:21.036528+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:03:18.300707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.014923+0000","last_change":"2024-02-19T15:27:21.014923+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.014923+0000","last_undegraded":"2024-02-19T15:27:21.014923+0000","last_fullsized":"2024-02-19T15:27:21.014923+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.032053+0000","last_change":"2024-02-19T15:27:21.029497+0000","last_active":"2024-02-19T15:27:21.032053+0000","last_peered":"2024-02-19T15:27:21.032053+0000","last_clean":"2024-02-19T15:27:21.032053+0000","last_became_active":"2024-02-19T15:27:21.029251+0000","last_became_peered":"2024-02-19T15:27:21.029251+0000","last_unstale":"2024-02-19T15:27:21.032053+0000","last_undegraded":"2024-02-19T15:27:21.032053+0000","last_fullsized":"2024-02-19T15:27:21.032053+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:16:56.947269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.022312+0000","last_change":"2024-02-19T15:27:21.022312+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.022312+0000","last_undegraded":"2024-02-19T15:27:21.022312+0000","last_fullsized":"2024-02-19T15:27:21.022312+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.027420+0000","last_change":"2024-02-19T15:27:21.024536+0000","last_active":"2024-02-19T15:27:21.027420+0000","last_peered":"2024-02-19T15:27:21.027420+0000","last_clean":"2024-02-19T15:27:21.027420+0000","last_became_active":"2024-02-19T15:27:21.020949+0000","last_became_peered":"2024-02-19T15:27:21.020949+0000","last_unstale":"2024-02-19T15:27:21.027420+0000","last_undegraded":"2024-02-19T15:27:21.027420+0000","last_fullsized":"2024-02-19T15:27:21.027420+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:16:06.605936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.025938+0000","last_change":"2024-02-19T15:27:21.025938+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.025938+0000","last_undegraded":"2024-02-19T15:27:21.025938+0000","last_fullsized":"2024-02-19T15:27:21.025938+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.036666+0000","last_change":"2024-02-19T15:27:21.036160+0000","last_active":"2024-02-19T15:27:21.036666+0000","last_peered":"2024-02-19T15:27:21.036666+0000","last_clean":"2024-02-19T15:27:21.036666+0000","last_became_active":"2024-02-19T15:27:21.035805+0000","last_became_peered":"2024-02-19T15:27:21.035805+0000","last_unstale":"2024-02-19T15:27:21.036666+0000","last_undegraded":"2024-02-19T15:27:21.036666+0000","last_fullsized":"2024-02-19T15:27:21.036666+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:51:06.106358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":320,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.018304+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:27:21.018304+0000","last_peered":"2024-02-19T15:27:21.018304+0000","last_clean":"2024-02-19T15:27:21.018304+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:27:21.018304+0000","last_undegraded":"2024-02-19T15:27:21.018304+0000","last_fullsized":"2024-02-19T15:27:21.018304+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509509,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1314816,"data_stored":563776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.622,"15min":0.62},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.66100000000000003,"15min":0.746},"min":{"1min":0.39100000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.8100000000000001,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.76100000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63800000000000001,"15min":0.65000000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.44,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.774,"5min":2.774,"15min":2.774},"last":0.58099999999999996}]},{"osd":2,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63100000000000001,"15min":0.63200000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.69899999999999995,"15min":0.68899999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.69299999999999995}]},{"osd":3,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":0.83499999999999996,"15min":0.72699999999999998},"min":{"1min":0.46600000000000003,"5min":0.438,"15min":0.438},"max":{"1min":14.045999999999999,"5min":14.045999999999999,"15min":14.045999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.67600000000000005,"15min":0.66200000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.85999999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.64500000000000002,"15min":0.61299999999999999},"min":{"1min":0.32400000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.69099999999999995,"15min":0.625},"min":{"1min":0.35599999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":2.8500000000000001,"5min":2.8500000000000001,"15min":2.8500000000000001},"last":0.83599999999999997}]},{"osd":5,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.61899999999999999},"min":{"1min":0.38200000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.72899999999999998,"15min":0.68500000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.73199999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.75900000000000001,"15min":0.68100000000000005},"min":{"1min":0.42799999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":3.0539999999999998,"5min":3.0539999999999998,"15min":3.0539999999999998},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.755,"15min":0.69299999999999995},"min":{"1min":0.439,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.256,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.91100000000000003}]}]},{"osd":7,"up_from":14,"seq":60129542213,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37200,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334640,"statfs":{"total":96636764160,"available":96598671360,"internally_reserved":0,"allocated":1896448,"data_stored":1147869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":2.016,"5min":0.88300000000000001,"15min":0.68500000000000005},"min":{"1min":0.51800000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":15.127000000000001,"5min":15.127000000000001,"15min":15.127000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.71899999999999997,"15min":0.69399999999999995},"min":{"1min":0.47499999999999998,"5min":0.376,"15min":0.376},"max":{"1min":6.0890000000000004,"5min":6.0890000000000004,"15min":6.0890000000000004},"last":0.63700000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":2.0910000000000002,"5min":0.93600000000000005,"15min":0.70199999999999996},"min":{"1min":0.47499999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":15.196999999999999,"5min":15.196999999999999,"15min":15.196999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.3100000000000001,"5min":0.81999999999999995,"15min":0.76000000000000001},"min":{"1min":0.45600000000000002,"5min":0.437,"15min":0.437},"max":{"1min":10.933999999999999,"5min":10.933999999999999,"15min":10.933999999999999},"last":0.80800000000000005}]},{"osd":2,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":0.79600000000000004,"15min":0.68600000000000005},"min":{"1min":0.45700000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":10.84,"5min":10.84,"15min":10.84},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.71199999999999997,"15min":0.69999999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":2.3170000000000002,"5min":2.508,"15min":2.508},"last":0.77700000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":1.6539999999999999,"5min":0.86199999999999999,"15min":0.72599999999999998},"min":{"1min":0.49199999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":17.094000000000001,"5min":17.094000000000001,"15min":17.094000000000001},"last":0.746},{"interface":"front","average":{"1min":1.226,"5min":0.81799999999999995,"15min":0.73099999999999998},"min":{"1min":0.46300000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":7.7190000000000003,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.85299999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.61299999999999999,"15min":0.54500000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.876},{"interface":"front","average":{"1min":2.266,"5min":0.95799999999999996,"15min":0.69499999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":26.625,"5min":26.625,"15min":26.625},"last":0.79600000000000004}]},{"osd":5,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":2.5350000000000001,"5min":1.0549999999999999,"15min":0.71699999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":31.844999999999999,"5min":31.844999999999999,"15min":31.844999999999999},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.6100000000000001,"5min":0.876,"15min":0.70399999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":16.776,"5min":16.776,"15min":16.776},"last":0.83499999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.70499999999999996,"15min":0.625},"min":{"1min":0.375,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.76400000000000001,"15min":0.68799999999999994},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":3.5099999999999998,"5min":3.5099999999999998,"15min":3.5099999999999998},"last":0.97599999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542214,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36616,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335224,"statfs":{"total":96636764160,"available":96599269376,"internally_reserved":0,"allocated":1302528,"data_stored":557501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.69699999999999995,"15min":0.65300000000000002},"min":{"1min":0.48399999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":2.6659999999999999,"5min":5.7789999999999999,"15min":5.7789999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73799999999999999,"15min":0.64600000000000002},"min":{"1min":0.45000000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.2370000000000001,"5min":12.359999999999999,"15min":12.359999999999999},"last":0.91700000000000004}]},{"osd":1,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.76600000000000001,"15min":0.67900000000000005},"min":{"1min":0.496,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.6499999999999999,"5min":10.938000000000001,"15min":10.938000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.82199999999999995,"15min":0.72399999999999998},"min":{"1min":0.45800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":3.625,"5min":10.531000000000001,"15min":10.531000000000001},"last":0.89500000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.81499999999999995,"15min":0.72699999999999998},"min":{"1min":0.503,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":2.589,"5min":12.925000000000001,"15min":12.925000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.74099999999999999,"15min":0.64500000000000002},"min":{"1min":0.46999999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89100000000000001,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.77300000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.96499999999999997,"15min":0.753},"min":{"1min":0.51000000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":5.492,"5min":26.161999999999999,"15min":26.161999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.76200000000000001},"min":{"1min":0.52700000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":6.4939999999999998,"5min":27.934999999999999,"15min":27.934999999999999},"last":0.86399999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":0.79900000000000004,"15min":0.63300000000000001},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":11.73,"5min":11.73,"15min":11.73},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.117,"5min":0.80200000000000005,"15min":0.70199999999999996},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":10.731,"5min":10.731,"15min":10.731},"last":0.68600000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.76900000000000002,"15min":0.67700000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.6320000000000001,"5min":6.5229999999999997,"15min":6.5229999999999997},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.248,"5min":0.92600000000000005,"15min":0.78400000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":10.432,"5min":10.432,"15min":10.432},"last":0.73499999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.78700000000000003,"15min":0.69799999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.9450000000000001,"5min":11.986000000000001,"15min":11.986000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.83099999999999996,"15min":0.73299999999999998},"min":{"1min":0.42999999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":6.5339999999999998,"5min":8.9770000000000003,"15min":8.9770000000000003},"last":0.749}]}]},{"osd":4,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36616,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335224,"statfs":{"total":96636764160,"available":96599269376,"internally_reserved":0,"allocated":1302528,"data_stored":557501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":4.1779999999999999,"5min":1.3089999999999999,"15min":0.83099999999999996},"min":{"1min":0.38,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":76.117000000000004,"5min":76.117000000000004,"15min":76.117000000000004},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.746,"5min":0.65300000000000002,"15min":0.61499999999999999},"min":{"1min":0.40000000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.2709999999999999,"5min":3.2709999999999999,"15min":3.2709999999999999},"last":0.42799999999999999}]},{"osd":1,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":4.4470000000000001,"5min":1.409,"15min":0.875},"min":{"1min":0.45800000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":80.513000000000005,"5min":80.513000000000005,"15min":80.513000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.70799999999999996,"15min":0.65500000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":7.181,"5min":7.181,"15min":7.181},"last":0.54100000000000004}]},{"osd":2,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":0.77900000000000003,"15min":0.70199999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":14.362,"5min":14.362,"15min":14.362},"last":0.67200000000000004},{"interface":"front","average":{"1min":4.6100000000000003,"5min":1.4690000000000001,"15min":0.93899999999999995},"min":{"1min":0.42299999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":84.093000000000004,"5min":84.093000000000004,"15min":84.093000000000004},"last":0.51100000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":4.6539999999999999,"5min":1.5049999999999999,"15min":0.94799999999999995},"min":{"1min":0.504,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":84.140000000000001,"5min":84.140000000000001,"15min":84.140000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.44400000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.56499999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":0.76500000000000001,"15min":0.64900000000000002},"min":{"1min":0.379,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":14.318,"5min":14.318,"15min":14.318},"last":0.69299999999999995},{"interface":"front","average":{"1min":4.274,"5min":1.427,"15min":0.92300000000000004},"min":{"1min":0.41099999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":76.164000000000001,"5min":76.164000000000001,"15min":76.164000000000001},"last":0.626}]},{"osd":6,"last update":"Mon Feb 19 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64100000000000001,"15min":0.58899999999999997},"min":{"1min":0.33500000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":3.5019999999999998,"5min":1.2589999999999999,"15min":0.84599999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":60.151000000000003,"5min":60.151000000000003,"15min":60.151000000000003},"last":0.60699999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":2.044,"5min":0.97999999999999998,"15min":0.76400000000000001},"min":{"1min":0.40699999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":29.036000000000001,"5min":29.036000000000001,"15min":29.036000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":1.0449999999999999,"5min":0.82099999999999995,"15min":0.755},"min":{"1min":0.50700000000000001,"5min":0.443,"15min":0.443},"max":{"1min":7.101,"5min":7.101,"15min":7.101},"last":0.64700000000000002}]}]},{"osd":3,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40772,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331068,"statfs":{"total":96636764160,"available":96595013632,"internally_reserved":0,"allocated":1302528,"data_stored":557501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41799999999999998,"15min":0.44400000000000001},"min":{"1min":0.23499999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.82699999999999996,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.5,"15min":0.63200000000000001},"min":{"1min":0.28899999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.0840000000000001,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.51200000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.437,"15min":0.443},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.54400000000000004,"15min":0.69199999999999995},"min":{"1min":0.316,"5min":0.31,"15min":0.31},"max":{"1min":0.85099999999999998,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.55400000000000005}]},{"osd":2,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55500000000000005,"15min":0.60699999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53700000000000003,"15min":0.58499999999999996},"min":{"1min":0.38400000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.57599999999999996}]},{"osd":4,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71299999999999997,"15min":0.68200000000000005},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.97999999999999998,"5min":1.004,"15min":1.004},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.75800000000000001,"15min":0.84599999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0169999999999999,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.88600000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.77000000000000002,"15min":0.84399999999999997},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.99099999999999999,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.73499999999999999,"15min":0.745},"min":{"1min":0.46999999999999997,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89700000000000002,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.92000000000000004}]},{"osd":6,"last update":"Mon Feb 19 15:27:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.79300000000000004,"15min":0.879},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.3180000000000001,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":1.052},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.77500000000000002,"15min":0.81000000000000005},"min":{"1min":0.53900000000000003,"5min":0.502,"15min":0.502},"max":{"1min":1.206,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":1.0349999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.755,"15min":0.77700000000000002},"min":{"1min":0.57099999999999995,"5min":0.504,"15min":0.504},"max":{"1min":0.996,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.80400000000000005,"15min":0.90800000000000003},"min":{"1min":0.54800000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.1919999999999999,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.98299999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1314816,"data_stored":563776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.60099999999999998,"15min":0.67500000000000004},"min":{"1min":0.26600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.48899999999999999,"5min":15.42,"15min":15.42},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.61199999999999999,"15min":0.85099999999999998},"min":{"1min":0.28199999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68700000000000006,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.38500000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.65500000000000003,"15min":0.86299999999999999},"min":{"1min":0.29399999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72799999999999998,"5min":9.5269999999999992,"15min":9.5269999999999992},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.83999999999999997,"15min":0.97199999999999998},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.75600000000000001,"5min":22.954999999999998,"15min":22.954999999999998},"last":0.53700000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.61499999999999999,"15min":0.76500000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.75,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.68700000000000006,"15min":0.80400000000000005},"min":{"1min":0.42399999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81399999999999995,"5min":7.734,"15min":7.734},"last":0.63700000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.86399999999999999,"15min":0.88600000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.97999999999999998,"5min":15.369,"15min":15.369},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.77800000000000002,"15min":0.876},"min":{"1min":0.51900000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93600000000000005,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.78800000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.83299999999999996,"15min":0.876},"min":{"1min":0.47199999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.006,"5min":7.6520000000000001,"15min":7.6520000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.93400000000000005,"15min":0.95699999999999996},"min":{"1min":0.496,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.026,"5min":15.391,"15min":15.391},"last":0.87}]},{"osd":6,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.90900000000000003,"15min":0.92800000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.92000000000000004,"5min":15.337,"15min":15.337},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.81399999999999995,"15min":0.93799999999999994},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.93700000000000006,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.93899999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.85399999999999998,"15min":0.96399999999999997},"min":{"1min":0.57099999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.98099999999999998,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.86099999999999999,"15min":0.93500000000000005},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94299999999999995,"5min":7.6929999999999996,"15min":7.6929999999999996},"last":0.85299999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542214,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40784,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331056,"statfs":{"total":96636764160,"available":96595001344,"internally_reserved":0,"allocated":1314816,"data_stored":563776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38600000000000001,"15min":0.42199999999999999},"min":{"1min":0.25700000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.68999999999999995,"5min":0.747,"15min":0.747},"last":0.432},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.434,"15min":0.46500000000000002},"min":{"1min":0.312,"5min":0.25,"15min":0.25},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.33200000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.44500000000000001},"min":{"1min":0.30499999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.47799999999999998,"15min":0.52300000000000002},"min":{"1min":0.32400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.74099999999999999,"5min":1.071,"15min":1.071},"last":0.66100000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57999999999999996,"15min":0.63},"min":{"1min":0.45500000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.1200000000000001,"5min":1.365,"15min":1.365},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.55800000000000005,"15min":0.625},"min":{"1min":0.35399999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0069999999999999,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.59299999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.67400000000000004,"15min":0.62},"min":{"1min":0.434,"5min":0.377,"15min":0.377},"max":{"1min":1.0409999999999999,"5min":1.127,"15min":1.127},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.69999999999999996,"15min":0.68200000000000005},"min":{"1min":0.56100000000000005,"5min":0.435,"15min":0.435},"max":{"1min":1.083,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.82899999999999996}]},{"osd":5,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.71199999999999997,"15min":0.69799999999999995},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.88800000000000001,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70899999999999996,"15min":0.68700000000000006},"min":{"1min":0.51200000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.96399999999999997,"5min":1.107,"15min":1.107},"last":0.745}]},{"osd":6,"last update":"Mon Feb 19 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.73199999999999998,"15min":0.72899999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.97799999999999998,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.755,"5min":0.74399999999999999,"15min":0.746},"min":{"1min":0.56299999999999994,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.1559999999999999,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.97099999999999997}]},{"osd":7,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.78400000000000003},"min":{"1min":0.52700000000000002,"5min":0.496,"15min":0.496},"max":{"1min":0.94599999999999995,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":1.0800000000000001},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.75},"min":{"1min":0.54500000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.99299999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":1.0509999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542212,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41360,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330480,"statfs":{"total":96636764160,"available":96594411520,"internally_reserved":0,"allocated":1896448,"data_stored":1147869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.45300000000000001,"15min":0.60999999999999999},"min":{"1min":0.24299999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.745,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.45800000000000002,"15min":0.48199999999999998},"min":{"1min":0.34200000000000003,"5min":0.252,"15min":0.252},"max":{"1min":0.77300000000000002,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.61399999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59999999999999998,"15min":0.76600000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.82399999999999995,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.56100000000000005,"15min":0.72699999999999998},"min":{"1min":0.35699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.85899999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.42699999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52000000000000002,"15min":0.66300000000000003},"min":{"1min":0.307,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.93799999999999994,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53300000000000003,"15min":0.60499999999999998},"min":{"1min":0.376,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.052,"5min":1.345,"15min":1.345},"last":0.40799999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.73199999999999998,"15min":0.73799999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.091,"5min":1.095,"15min":1.095},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.79500000000000004,"15min":0.90900000000000003},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.085,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.69199999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.71399999999999997,"15min":0.74299999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.036,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.73699999999999999,"15min":0.84099999999999997},"min":{"1min":0.53600000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.169,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.5}]},{"osd":6,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.76400000000000001,"15min":0.89600000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.1200000000000001,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.71999999999999997,"15min":0.751},"min":{"1min":0.50800000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.70799999999999996}]},{"osd":7,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.79600000000000004,"15min":0.92400000000000004},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.1399999999999999,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.78100000000000003,"15min":0.92300000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.149,"5min":4.915,"15min":4.915},"last":0.65200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:27:26.693 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:27:26.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.863+0000 7f57a7e00640 1 Processor -- start 2024-02-19T15:27:26.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.867+0000 7f57a7e00640 1 -- start start 2024-02-19T15:27:26.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.867+0000 7f57a7e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f57a006d650 0x7f57a00d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:26.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.867+0000 7f57a7e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a00cded0 0x7f57a00ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:26.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.867+0000 7f57a7e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a00cce30 con 0x7f57a00ceb40 2024-02-19T15:27:26.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.867+0000 7f57a7e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57a00cefd0 con 0x7f57a006d650 2024-02-19T15:27:26.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.867+0000 7f57a7e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a00cd110 con 0x7f57a00cded0 2024-02-19T15:27:26.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.867+0000 7f57a5e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f57a00ceb40 0x7f57a0161d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39806/0 (socket says 172.21.15.47:39806) 2024-02-19T15:27:26.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.867+0000 7f57a5e00640 1 -- 172.21.15.47:0/3706437711 learned_addr learned my addr 172.21.15.47:0/3706437711 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:26.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.867+0000 7f57a5400640 1 --2- 172.21.15.47:0/3706437711 >> v2:172.21.15.116:3300/0 conn(0x7f57a006d650 0x7f57a00d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:26.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.867+0000 7f57a4a00640 1 --2- 172.21.15.47:0/3706437711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a00cded0 0x7f57a00ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:26.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.867+0000 7f579fe00640 1 -- 172.21.15.47:0/3706437711 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165961394 0 0) 0x7f57a00cce30 con 0x7f57a00ceb40 2024-02-19T15:27:26.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.867+0000 7f579fe00640 1 -- 172.21.15.47:0/3706437711 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5788003530 con 0x7f57a00ceb40 2024-02-19T15:27:26.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.867+0000 7f57a5400640 1 -- 172.21.15.47:0/3706437711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a00cded0 msgr2=0x7f57a00ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:26.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.867+0000 7f57a5400640 1 --2- 172.21.15.47:0/3706437711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a00cded0 0x7f57a00ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.867+0000 7f57a5400640 1 -- 172.21.15.47:0/3706437711 >> v1:172.21.15.47:6789/0 conn(0x7f57a00ceb40 legacy=0x7f57a0161d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:26.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.867+0000 7f57a5400640 1 -- 172.21.15.47:0/3706437711 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a00cd3f0 con 0x7f57a006d650 2024-02-19T15:27:26.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.871+0000 7f57a5400640 1 --2- 172.21.15.47:0/3706437711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a006d650 0x7f57a00d08b0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f57900097c0 tx=0x7f579001c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d20af432522938b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:26.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.871+0000 7f579fe00640 1 -- 172.21.15.47:0/3706437711 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57900ab600 con 0x7f57a006d650 2024-02-19T15:27:26.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.871+0000 7f579fe00640 1 -- 172.21.15.47:0/3706437711 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57900a7050 con 0x7f57a006d650 2024-02-19T15:27:26.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.871+0000 7f579fe00640 1 -- 172.21.15.47:0/3706437711 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57900ab980 con 0x7f57a006d650 2024-02-19T15:27:26.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.871+0000 7f57a7e00640 1 -- 172.21.15.47:0/3706437711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a006d650 msgr2=0x7f57a00d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:26.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.871+0000 7f57a7e00640 1 --2- 172.21.15.47:0/3706437711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a006d650 0x7f57a00d08b0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f57900097c0 tx=0x7f579001c3e0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.871+0000 7f57a7e00640 1 -- 172.21.15.47:0/3706437711 shutdown_connections 2024-02-19T15:27:26.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.871+0000 7f57a7e00640 1 --2- 172.21.15.47:0/3706437711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a00cded0 0x7f57a00ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.871+0000 7f57a7e00640 1 --2- 172.21.15.47:0/3706437711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a006d650 0x7f57a00d08b0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.871+0000 7f57a7e00640 1 -- 172.21.15.47:0/3706437711 >> 172.21.15.47:0/3706437711 conn(0x7f57a00d1bf0 msgr2=0x7f57a00d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:26.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.871+0000 7f57a7e00640 1 -- 172.21.15.47:0/3706437711 shutdown_connections 2024-02-19T15:27:26.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.871+0000 7f57a7e00640 1 -- 172.21.15.47:0/3706437711 wait complete. 2024-02-19T15:27:26.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.871+0000 7f57a7e00640 1 Processor -- start 2024-02-19T15:27:26.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a7e00640 1 -- start start 2024-02-19T15:27:26.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a006d650 0x7f57a0157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:26.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57a00cded0 0x7f57a00c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:26.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a7e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a00ceb40 0x7f57a0158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:26.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a7e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a00c8870 con 0x7f57a00cded0 2024-02-19T15:27:26.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a7e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a0159490 con 0x7f57a006d650 2024-02-19T15:27:26.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a7e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a015a1d0 con 0x7f57a00ceb40 2024-02-19T15:27:26.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a4a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57a00cded0 0x7f57a00c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:26.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a4a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57a00cded0 0x7f57a00c5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55326/0 (socket says 172.21.15.47:55326) 2024-02-19T15:27:26.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a4a00640 1 -- 172.21.15.47:0/1559698686 learned_addr learned my addr 172.21.15.47:0/1559698686 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:26.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a5400640 1 --2- 172.21.15.47:0/1559698686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a006d650 0x7f57a0157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:26.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a5e00640 1 --2- 172.21.15.47:0/1559698686 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a00ceb40 0x7f57a0158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:26.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a4a00640 1 -- 172.21.15.47:0/1559698686 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a00ceb40 msgr2=0x7f57a0158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:26.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a4a00640 1 --2- 172.21.15.47:0/1559698686 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a00ceb40 0x7f57a0158850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a4a00640 1 -- 172.21.15.47:0/1559698686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a006d650 msgr2=0x7f57a0157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:26.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a4a00640 1 --2- 172.21.15.47:0/1559698686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a006d650 0x7f57a0157590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:26.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a4a00640 1 -- 172.21.15.47:0/1559698686 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5790009420 con 0x7f57a00cded0 2024-02-19T15:27:26.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a4a00640 1 --2- 172.21.15.47:0/1559698686 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57a00cded0 0x7f57a00c5650 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f579400e870 tx=0x7f579400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:26.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f579d600640 1 -- 172.21.15.47:0/1559698686 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5794009510 con 0x7f57a00cded0 2024-02-19T15:27:26.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f579d600640 1 -- 172.21.15.47:0/1559698686 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f579400eec0 con 0x7f57a00cded0 2024-02-19T15:27:26.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a7e00640 1 -- 172.21.15.47:0/1559698686 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a015a4b0 con 0x7f57a00cded0 2024-02-19T15:27:26.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f579d600640 1 -- 172.21.15.47:0/1559698686 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579400cb40 con 0x7f57a00cded0 2024-02-19T15:27:26.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a7e00640 1 -- 172.21.15.47:0/1559698686 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a00c8490 con 0x7f57a00cded0 2024-02-19T15:27:26.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.875+0000 7f57a7e00640 1 -- 172.21.15.47:0/1559698686 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a00c8010 con 0x7f57a00cded0 2024-02-19T15:27:26.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.879+0000 7f579d600640 1 -- 172.21.15.47:0/1559698686 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f579400cca0 con 0x7f57a00cded0 2024-02-19T15:27:26.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.879+0000 7f579d600640 1 --2- 172.21.15.47:0/1559698686 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5784066e40 0x7f5784069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:26.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.879+0000 7f579d600640 1 -- 172.21.15.47:0/1559698686 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5794084b40 con 0x7f57a00cded0 2024-02-19T15:27:26.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.879+0000 7f57a5400640 1 --2- 172.21.15.47:0/1559698686 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5784066e40 0x7f5784069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:26.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.879+0000 7f57a5400640 1 --2- 172.21.15.47:0/1559698686 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5784066e40 0x7f5784069300 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f57900ad000 tx=0x7f5790008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:26.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:26.883+0000 7f579d600640 1 -- 172.21.15.47:0/1559698686 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f57a00c8010 con 0x7f57a00cded0 2024-02-19T15:27:27.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.143+0000 7f57a7e00640 1 -- 172.21.15.47:0/1559698686 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f57a00d0df0 con 0x7f5784066e40 2024-02-19T15:27:27.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.151+0000 7f579d600640 1 -- 172.21.15.47:0/1559698686 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284066 (secure 0 0 0) 0x7f57900bc090 con 0x7f5784066e40 2024-02-19T15:27:27.158 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:27.159 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:27:27.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.155+0000 7f57a7e00640 1 -- 172.21.15.47:0/1559698686 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5784066e40 msgr2=0x7f5784069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:27.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.155+0000 7f57a7e00640 1 --2- 172.21.15.47:0/1559698686 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5784066e40 0x7f5784069300 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f57900ad000 tx=0x7f5790008040 comp rx=0 tx=0).stop 2024-02-19T15:27:27.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.155+0000 7f57a7e00640 1 -- 172.21.15.47:0/1559698686 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57a00cded0 msgr2=0x7f57a00c5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:27.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.155+0000 7f57a7e00640 1 --2- 172.21.15.47:0/1559698686 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57a00cded0 0x7f57a00c5650 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f579400e870 tx=0x7f579400ec70 comp rx=0 tx=0).stop 2024-02-19T15:27:27.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.155+0000 7f57a7e00640 1 -- 172.21.15.47:0/1559698686 shutdown_connections 2024-02-19T15:27:27.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.155+0000 7f57a7e00640 1 --2- 172.21.15.47:0/1559698686 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5784066e40 0x7f5784069300 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:27.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.155+0000 7f57a7e00640 1 --2- 172.21.15.47:0/1559698686 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a00ceb40 0x7f57a0158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:27.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.155+0000 7f57a7e00640 1 --2- 172.21.15.47:0/1559698686 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57a00cded0 0x7f57a00c5650 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:27.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.155+0000 7f57a7e00640 1 --2- 172.21.15.47:0/1559698686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a006d650 0x7f57a0157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:27.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.155+0000 7f57a7e00640 1 -- 172.21.15.47:0/1559698686 >> 172.21.15.47:0/1559698686 conn(0x7f57a00d1bf0 msgr2=0x7f57a006fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:27.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.155+0000 7f57a7e00640 1 -- 172.21.15.47:0/1559698686 shutdown_connections 2024-02-19T15:27:27.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.155+0000 7f57a7e00640 1 -- 172.21.15.47:0/1559698686 wait complete. 2024-02-19T15:27:27.179 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-02-19T15:27:25.221972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590438,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":160,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":20,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":306604,"kb_used_data":11372,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754668116,"statfs":{"total":773094113280,"available":772780150784,"internally_reserved":0,"allocated":11644928,"data_stored":5659569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":302173746},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.216184"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.031361+0000","last_change":"2024-02-19T15:27:21.031361+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.031361+0000","last_undegraded":"2024-02-19T15:27:21.031361+0000","last_fullsized":"2024-02-19T15:27:21.031361+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.030203+0000","last_change":"2024-02-19T15:27:21.030203+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.030203+0000","last_undegraded":"2024-02-19T15:27:21.030203+0000","last_fullsized":"2024-02-19T15:27:21.030203+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.016897+0000","last_change":"2024-02-19T15:27:21.016897+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.016897+0000","last_undegraded":"2024-02-19T15:27:21.016897+0000","last_fullsized":"2024-02-19T15:27:21.016897+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.030394+0000","last_change":"2024-02-19T15:27:21.030394+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.030394+0000","last_undegraded":"2024-02-19T15:27:21.030394+0000","last_fullsized":"2024-02-19T15:27:21.030394+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.023740+0000","last_change":"2024-02-19T15:27:21.023740+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.023740+0000","last_undegraded":"2024-02-19T15:27:21.023740+0000","last_fullsized":"2024-02-19T15:27:21.023740+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.011541+0000","last_change":"2024-02-19T15:27:21.011541+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.011541+0000","last_undegraded":"2024-02-19T15:27:21.011541+0000","last_fullsized":"2024-02-19T15:27:21.011541+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.026169+0000","last_change":"2024-02-19T15:27:21.026169+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.026169+0000","last_undegraded":"2024-02-19T15:27:21.026169+0000","last_fullsized":"2024-02-19T15:27:21.026169+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.016613+0000","last_change":"2024-02-19T15:27:21.016613+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.016613+0000","last_undegraded":"2024-02-19T15:27:21.016613+0000","last_fullsized":"2024-02-19T15:27:21.016613+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.030819+0000","last_change":"2024-02-19T15:27:21.030819+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.030819+0000","last_undegraded":"2024-02-19T15:27:21.030819+0000","last_fullsized":"2024-02-19T15:27:21.030819+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.029180+0000","last_change":"2024-02-19T15:27:21.029180+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.029180+0000","last_undegraded":"2024-02-19T15:27:21.029180+0000","last_fullsized":"2024-02-19T15:27:21.029180+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.026903+0000","last_change":"2024-02-19T15:27:21.026903+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.026903+0000","last_undegraded":"2024-02-19T15:27:21.026903+0000","last_fullsized":"2024-02-19T15:27:21.026903+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.028102+0000","last_change":"2024-02-19T15:27:21.028102+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.028102+0000","last_undegraded":"2024-02-19T15:27:21.028102+0000","last_fullsized":"2024-02-19T15:27:21.028102+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.015426+0000","last_change":"2024-02-19T15:27:21.015426+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.015426+0000","last_undegraded":"2024-02-19T15:27:21.015426+0000","last_fullsized":"2024-02-19T15:27:21.015426+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.016518+0000","last_change":"2024-02-19T15:27:21.016518+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.016518+0000","last_undegraded":"2024-02-19T15:27:21.016518+0000","last_fullsized":"2024-02-19T15:27:21.016518+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.011705+0000","last_change":"2024-02-19T15:27:21.011705+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.011705+0000","last_undegraded":"2024-02-19T15:27:21.011705+0000","last_fullsized":"2024-02-19T15:27:21.011705+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.026401+0000","last_change":"2024-02-19T15:27:21.026401+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.026401+0000","last_undegraded":"2024-02-19T15:27:21.026401+0000","last_fullsized":"2024-02-19T15:27:21.026401+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.025923+0000","last_change":"2024-02-19T15:27:21.025923+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.025923+0000","last_undegraded":"2024-02-19T15:27:21.025923+0000","last_fullsized":"2024-02-19T15:27:21.025923+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.042266+0000","last_change":"2024-02-19T15:27:22.036471+0000","last_active":"2024-02-19T15:27:22.042266+0000","last_peered":"2024-02-19T15:27:22.042266+0000","last_clean":"2024-02-19T15:27:22.042266+0000","last_became_active":"2024-02-19T15:27:22.036201+0000","last_became_peered":"2024-02-19T15:27:22.036201+0000","last_unstale":"2024-02-19T15:27:22.042266+0000","last_undegraded":"2024-02-19T15:27:22.042266+0000","last_fullsized":"2024-02-19T15:27:22.042266+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:28:59.516714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.016248+0000","last_change":"2024-02-19T15:27:21.016248+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.016248+0000","last_undegraded":"2024-02-19T15:27:21.016248+0000","last_fullsized":"2024-02-19T15:27:21.016248+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024313+0000","last_change":"2024-02-19T15:27:21.024313+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024313+0000","last_undegraded":"2024-02-19T15:27:21.024313+0000","last_fullsized":"2024-02-19T15:27:21.024313+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024688+0000","last_change":"2024-02-19T15:27:21.024688+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024688+0000","last_undegraded":"2024-02-19T15:27:21.024688+0000","last_fullsized":"2024-02-19T15:27:21.024688+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.033511+0000","last_change":"2024-02-19T15:27:21.033511+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.033511+0000","last_undegraded":"2024-02-19T15:27:21.033511+0000","last_fullsized":"2024-02-19T15:27:21.033511+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024070+0000","last_change":"2024-02-19T15:27:21.024070+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024070+0000","last_undegraded":"2024-02-19T15:27:21.024070+0000","last_fullsized":"2024-02-19T15:27:21.024070+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.033844+0000","last_change":"2024-02-19T15:27:21.033844+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.033844+0000","last_undegraded":"2024-02-19T15:27:21.033844+0000","last_fullsized":"2024-02-19T15:27:21.033844+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024026+0000","last_change":"2024-02-19T15:27:21.024026+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024026+0000","last_undegraded":"2024-02-19T15:27:21.024026+0000","last_fullsized":"2024-02-19T15:27:21.024026+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.025203+0000","last_change":"2024-02-19T15:27:21.025203+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.025203+0000","last_undegraded":"2024-02-19T15:27:21.025203+0000","last_fullsized":"2024-02-19T15:27:21.025203+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.012320+0000","last_change":"2024-02-19T15:27:21.012320+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.012320+0000","last_undegraded":"2024-02-19T15:27:21.012320+0000","last_fullsized":"2024-02-19T15:27:21.012320+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.033395+0000","last_change":"2024-02-19T15:27:21.033395+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.033395+0000","last_undegraded":"2024-02-19T15:27:21.033395+0000","last_fullsized":"2024-02-19T15:27:21.033395+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024077+0000","last_change":"2024-02-19T15:27:21.024077+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024077+0000","last_undegraded":"2024-02-19T15:27:21.024077+0000","last_fullsized":"2024-02-19T15:27:21.024077+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.021450+0000","last_change":"2024-02-19T15:27:21.021450+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.021450+0000","last_undegraded":"2024-02-19T15:27:21.021450+0000","last_fullsized":"2024-02-19T15:27:21.021450+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.026040+0000","last_change":"2024-02-19T15:27:21.026040+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.026040+0000","last_undegraded":"2024-02-19T15:27:21.026040+0000","last_fullsized":"2024-02-19T15:27:21.026040+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.011531+0000","last_change":"2024-02-19T15:27:21.011531+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.011531+0000","last_undegraded":"2024-02-19T15:27:21.011531+0000","last_fullsized":"2024-02-19T15:27:21.011531+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.013560+0000","last_change":"2024-02-19T15:27:21.013560+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.013560+0000","last_undegraded":"2024-02-19T15:27:21.013560+0000","last_fullsized":"2024-02-19T15:27:21.013560+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.033484+0000","last_change":"2024-02-19T15:27:21.030063+0000","last_active":"2024-02-19T15:27:21.033484+0000","last_peered":"2024-02-19T15:27:21.033484+0000","last_clean":"2024-02-19T15:27:21.033484+0000","last_became_active":"2024-02-19T15:27:21.029780+0000","last_became_peered":"2024-02-19T15:27:21.029780+0000","last_unstale":"2024-02-19T15:27:21.033484+0000","last_undegraded":"2024-02-19T15:27:21.033484+0000","last_fullsized":"2024-02-19T15:27:21.033484+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:15.014562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.054040+0000","last_change":"2024-02-19T15:27:22.048726+0000","last_active":"2024-02-19T15:27:22.054040+0000","last_peered":"2024-02-19T15:27:22.054040+0000","last_clean":"2024-02-19T15:27:22.054040+0000","last_became_active":"2024-02-19T15:27:22.048437+0000","last_became_peered":"2024-02-19T15:27:22.048437+0000","last_unstale":"2024-02-19T15:27:22.054040+0000","last_undegraded":"2024-02-19T15:27:22.054040+0000","last_fullsized":"2024-02-19T15:27:22.054040+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:50:16.869685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.030517+0000","last_change":"2024-02-19T15:27:21.027135+0000","last_active":"2024-02-19T15:27:21.030517+0000","last_peered":"2024-02-19T15:27:21.030517+0000","last_clean":"2024-02-19T15:27:21.030517+0000","last_became_active":"2024-02-19T15:27:21.026870+0000","last_became_peered":"2024-02-19T15:27:21.026870+0000","last_unstale":"2024-02-19T15:27:21.030517+0000","last_undegraded":"2024-02-19T15:27:21.030517+0000","last_fullsized":"2024-02-19T15:27:21.030517+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:51.292175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.012534+0000","last_change":"2024-02-19T15:27:21.012534+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.012534+0000","last_undegraded":"2024-02-19T15:27:21.012534+0000","last_fullsized":"2024-02-19T15:27:21.012534+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.038196+0000","last_change":"2024-02-19T15:27:21.029872+0000","last_active":"2024-02-19T15:27:21.038196+0000","last_peered":"2024-02-19T15:27:21.038196+0000","last_clean":"2024-02-19T15:27:21.038196+0000","last_became_active":"2024-02-19T15:27:21.029590+0000","last_became_peered":"2024-02-19T15:27:21.029590+0000","last_unstale":"2024-02-19T15:27:21.038196+0000","last_undegraded":"2024-02-19T15:27:21.038196+0000","last_fullsized":"2024-02-19T15:27:21.038196+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:32:04.524076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.015800+0000","last_change":"2024-02-19T15:27:21.015800+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.015800+0000","last_undegraded":"2024-02-19T15:27:21.015800+0000","last_fullsized":"2024-02-19T15:27:21.015800+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.043779+0000","last_change":"2024-02-19T15:27:21.042870+0000","last_active":"2024-02-19T15:27:21.043779+0000","last_peered":"2024-02-19T15:27:21.043779+0000","last_clean":"2024-02-19T15:27:21.043779+0000","last_became_active":"2024-02-19T15:27:21.041700+0000","last_became_peered":"2024-02-19T15:27:21.041700+0000","last_unstale":"2024-02-19T15:27:21.043779+0000","last_undegraded":"2024-02-19T15:27:21.043779+0000","last_fullsized":"2024-02-19T15:27:21.043779+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:21:53.467732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.022487+0000","last_change":"2024-02-19T15:27:21.022487+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.022487+0000","last_undegraded":"2024-02-19T15:27:21.022487+0000","last_fullsized":"2024-02-19T15:27:21.022487+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.041473+0000","last_change":"2024-02-19T15:27:21.040738+0000","last_active":"2024-02-19T15:27:21.041473+0000","last_peered":"2024-02-19T15:27:21.041473+0000","last_clean":"2024-02-19T15:27:21.041473+0000","last_became_active":"2024-02-19T15:27:21.040492+0000","last_became_peered":"2024-02-19T15:27:21.040492+0000","last_unstale":"2024-02-19T15:27:21.041473+0000","last_undegraded":"2024-02-19T15:27:21.041473+0000","last_fullsized":"2024-02-19T15:27:21.041473+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:38:35.042142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.023070+0000","last_change":"2024-02-19T15:27:21.023070+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.023070+0000","last_undegraded":"2024-02-19T15:27:21.023070+0000","last_fullsized":"2024-02-19T15:27:21.023070+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.039620+0000","last_change":"2024-02-19T15:27:21.037989+0000","last_active":"2024-02-19T15:27:21.039620+0000","last_peered":"2024-02-19T15:27:21.039620+0000","last_clean":"2024-02-19T15:27:21.039620+0000","last_became_active":"2024-02-19T15:27:21.037527+0000","last_became_peered":"2024-02-19T15:27:21.037527+0000","last_unstale":"2024-02-19T15:27:21.039620+0000","last_undegraded":"2024-02-19T15:27:21.039620+0000","last_fullsized":"2024-02-19T15:27:21.039620+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:59:16.122985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024991+0000","last_change":"2024-02-19T15:27:21.024991+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024991+0000","last_undegraded":"2024-02-19T15:27:21.024991+0000","last_fullsized":"2024-02-19T15:27:21.024991+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.032693+0000","last_change":"2024-02-19T15:27:21.031349+0000","last_active":"2024-02-19T15:27:21.032693+0000","last_peered":"2024-02-19T15:27:21.032693+0000","last_clean":"2024-02-19T15:27:21.032693+0000","last_became_active":"2024-02-19T15:27:21.031075+0000","last_became_peered":"2024-02-19T15:27:21.031075+0000","last_unstale":"2024-02-19T15:27:21.032693+0000","last_undegraded":"2024-02-19T15:27:21.032693+0000","last_fullsized":"2024-02-19T15:27:21.032693+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:25:19.941750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.023512+0000","last_change":"2024-02-19T15:27:21.023512+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.023512+0000","last_undegraded":"2024-02-19T15:27:21.023512+0000","last_fullsized":"2024-02-19T15:27:21.023512+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.030110+0000","last_change":"2024-02-19T15:27:21.025268+0000","last_active":"2024-02-19T15:27:21.030110+0000","last_peered":"2024-02-19T15:27:21.030110+0000","last_clean":"2024-02-19T15:27:21.030110+0000","last_became_active":"2024-02-19T15:27:21.022655+0000","last_became_peered":"2024-02-19T15:27:21.022655+0000","last_unstale":"2024-02-19T15:27:21.030110+0000","last_undegraded":"2024-02-19T15:27:21.030110+0000","last_fullsized":"2024-02-19T15:27:21.030110+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:44:25.168118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.021162+0000","last_change":"2024-02-19T15:27:21.021162+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.021162+0000","last_undegraded":"2024-02-19T15:27:21.021162+0000","last_fullsized":"2024-02-19T15:27:21.021162+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.043234+0000","last_change":"2024-02-19T15:27:21.041198+0000","last_active":"2024-02-19T15:27:21.043234+0000","last_peered":"2024-02-19T15:27:21.043234+0000","last_clean":"2024-02-19T15:27:21.043234+0000","last_became_active":"2024-02-19T15:27:21.040821+0000","last_became_peered":"2024-02-19T15:27:21.040821+0000","last_unstale":"2024-02-19T15:27:21.043234+0000","last_undegraded":"2024-02-19T15:27:21.043234+0000","last_fullsized":"2024-02-19T15:27:21.043234+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:15:15.162474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.050040+0000","last_change":"2024-02-19T15:27:22.043233+0000","last_active":"2024-02-19T15:27:22.050040+0000","last_peered":"2024-02-19T15:27:22.050040+0000","last_clean":"2024-02-19T15:27:22.050040+0000","last_became_active":"2024-02-19T15:27:22.042812+0000","last_became_peered":"2024-02-19T15:27:22.042812+0000","last_unstale":"2024-02-19T15:27:22.050040+0000","last_undegraded":"2024-02-19T15:27:22.050040+0000","last_fullsized":"2024-02-19T15:27:22.050040+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:02:10.136155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.039592+0000","last_change":"2024-02-19T15:27:21.036912+0000","last_active":"2024-02-19T15:27:21.039592+0000","last_peered":"2024-02-19T15:27:21.039592+0000","last_clean":"2024-02-19T15:27:21.039592+0000","last_became_active":"2024-02-19T15:27:21.036227+0000","last_became_peered":"2024-02-19T15:27:21.036227+0000","last_unstale":"2024-02-19T15:27:21.039592+0000","last_undegraded":"2024-02-19T15:27:21.039592+0000","last_fullsized":"2024-02-19T15:27:21.039592+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:15:01.148828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.032945+0000","last_change":"2024-02-19T15:27:21.032945+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.032945+0000","last_undegraded":"2024-02-19T15:27:21.032945+0000","last_fullsized":"2024-02-19T15:27:21.032945+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.023292+0000","last_change":"2024-02-19T15:27:21.022250+0000","last_active":"2024-02-19T15:27:21.023292+0000","last_peered":"2024-02-19T15:27:21.023292+0000","last_clean":"2024-02-19T15:27:21.023292+0000","last_became_active":"2024-02-19T15:27:21.021690+0000","last_became_peered":"2024-02-19T15:27:21.021690+0000","last_unstale":"2024-02-19T15:27:21.023292+0000","last_undegraded":"2024-02-19T15:27:21.023292+0000","last_fullsized":"2024-02-19T15:27:21.023292+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:41.432608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.021714+0000","last_change":"2024-02-19T15:27:21.021714+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.021714+0000","last_undegraded":"2024-02-19T15:27:21.021714+0000","last_fullsized":"2024-02-19T15:27:21.021714+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.037376+0000","last_change":"2024-02-19T15:27:21.031559+0000","last_active":"2024-02-19T15:27:21.037376+0000","last_peered":"2024-02-19T15:27:21.037376+0000","last_clean":"2024-02-19T15:27:21.037376+0000","last_became_active":"2024-02-19T15:27:21.031280+0000","last_became_peered":"2024-02-19T15:27:21.031280+0000","last_unstale":"2024-02-19T15:27:21.037376+0000","last_undegraded":"2024-02-19T15:27:21.037376+0000","last_fullsized":"2024-02-19T15:27:21.037376+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:48.463248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.029223+0000","last_change":"2024-02-19T15:27:21.029223+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.029223+0000","last_undegraded":"2024-02-19T15:27:21.029223+0000","last_fullsized":"2024-02-19T15:27:21.029223+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.025611+0000","last_change":"2024-02-19T15:27:21.019044+0000","last_active":"2024-02-19T15:27:21.025611+0000","last_peered":"2024-02-19T15:27:21.025611+0000","last_clean":"2024-02-19T15:27:21.025611+0000","last_became_active":"2024-02-19T15:27:21.018732+0000","last_became_peered":"2024-02-19T15:27:21.018732+0000","last_unstale":"2024-02-19T15:27:21.025611+0000","last_undegraded":"2024-02-19T15:27:21.025611+0000","last_fullsized":"2024-02-19T15:27:21.025611+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:02:10.142351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.033846+0000","last_change":"2024-02-19T15:27:21.033846+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.033846+0000","last_undegraded":"2024-02-19T15:27:21.033846+0000","last_fullsized":"2024-02-19T15:27:21.033846+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.027358+0000","last_change":"2024-02-19T15:27:21.022352+0000","last_active":"2024-02-19T15:27:22.027358+0000","last_peered":"2024-02-19T15:27:22.027358+0000","last_clean":"2024-02-19T15:27:22.027358+0000","last_became_active":"2024-02-19T15:27:21.021353+0000","last_became_peered":"2024-02-19T15:27:21.021353+0000","last_unstale":"2024-02-19T15:27:22.027358+0000","last_undegraded":"2024-02-19T15:27:22.027358+0000","last_fullsized":"2024-02-19T15:27:22.027358+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:05:11.130486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.029752+0000","last_change":"2024-02-19T15:27:21.029752+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.029752+0000","last_undegraded":"2024-02-19T15:27:21.029752+0000","last_fullsized":"2024-02-19T15:27:21.029752+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.043774+0000","last_change":"2024-02-19T15:27:21.042396+0000","last_active":"2024-02-19T15:27:21.043774+0000","last_peered":"2024-02-19T15:27:21.043774+0000","last_clean":"2024-02-19T15:27:21.043774+0000","last_became_active":"2024-02-19T15:27:21.042017+0000","last_became_peered":"2024-02-19T15:27:21.042017+0000","last_unstale":"2024-02-19T15:27:21.043774+0000","last_undegraded":"2024-02-19T15:27:21.043774+0000","last_fullsized":"2024-02-19T15:27:21.043774+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:54:06.234893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.024250+0000","last_change":"2024-02-19T15:27:21.024250+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.024250+0000","last_undegraded":"2024-02-19T15:27:21.024250+0000","last_fullsized":"2024-02-19T15:27:21.024250+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.031024+0000","last_change":"2024-02-19T15:27:21.027023+0000","last_active":"2024-02-19T15:27:21.031024+0000","last_peered":"2024-02-19T15:27:21.031024+0000","last_clean":"2024-02-19T15:27:21.031024+0000","last_became_active":"2024-02-19T15:27:21.026565+0000","last_became_peered":"2024-02-19T15:27:21.026565+0000","last_unstale":"2024-02-19T15:27:21.031024+0000","last_undegraded":"2024-02-19T15:27:21.031024+0000","last_fullsized":"2024-02-19T15:27:21.031024+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:52:51.983435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.013930+0000","last_change":"2024-02-19T15:27:21.013930+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.013930+0000","last_undegraded":"2024-02-19T15:27:21.013930+0000","last_fullsized":"2024-02-19T15:27:21.013930+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.039502+0000","last_change":"2024-02-19T15:27:21.033163+0000","last_active":"2024-02-19T15:27:21.039502+0000","last_peered":"2024-02-19T15:27:21.039502+0000","last_clean":"2024-02-19T15:27:21.039502+0000","last_became_active":"2024-02-19T15:27:21.032778+0000","last_became_peered":"2024-02-19T15:27:21.032778+0000","last_unstale":"2024-02-19T15:27:21.039502+0000","last_undegraded":"2024-02-19T15:27:21.039502+0000","last_fullsized":"2024-02-19T15:27:21.039502+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:35:31.450513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.015439+0000","last_change":"2024-02-19T15:27:21.015439+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.015439+0000","last_undegraded":"2024-02-19T15:27:21.015439+0000","last_fullsized":"2024-02-19T15:27:21.015439+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.047598+0000","last_change":"2024-02-19T15:27:21.033698+0000","last_active":"2024-02-19T15:27:22.047598+0000","last_peered":"2024-02-19T15:27:22.047598+0000","last_clean":"2024-02-19T15:27:22.047598+0000","last_became_active":"2024-02-19T15:27:21.033354+0000","last_became_peered":"2024-02-19T15:27:21.033354+0000","last_unstale":"2024-02-19T15:27:22.047598+0000","last_undegraded":"2024-02-19T15:27:22.047598+0000","last_fullsized":"2024-02-19T15:27:22.047598+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:42.028837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.015175+0000","last_change":"2024-02-19T15:27:21.015175+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.015175+0000","last_undegraded":"2024-02-19T15:27:21.015175+0000","last_fullsized":"2024-02-19T15:27:21.015175+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.033403+0000","last_change":"2024-02-19T15:27:21.031371+0000","last_active":"2024-02-19T15:27:21.033403+0000","last_peered":"2024-02-19T15:27:21.033403+0000","last_clean":"2024-02-19T15:27:21.033403+0000","last_became_active":"2024-02-19T15:27:21.031066+0000","last_became_peered":"2024-02-19T15:27:21.031066+0000","last_unstale":"2024-02-19T15:27:21.033403+0000","last_undegraded":"2024-02-19T15:27:21.033403+0000","last_fullsized":"2024-02-19T15:27:21.033403+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:27.660797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.015962+0000","last_change":"2024-02-19T15:27:21.015962+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.015962+0000","last_undegraded":"2024-02-19T15:27:21.015962+0000","last_fullsized":"2024-02-19T15:27:21.015962+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.039981+0000","last_change":"2024-02-19T15:27:21.026803+0000","last_active":"2024-02-19T15:27:21.039981+0000","last_peered":"2024-02-19T15:27:21.039981+0000","last_clean":"2024-02-19T15:27:21.039981+0000","last_became_active":"2024-02-19T15:27:21.025817+0000","last_became_peered":"2024-02-19T15:27:21.025817+0000","last_unstale":"2024-02-19T15:27:21.039981+0000","last_undegraded":"2024-02-19T15:27:21.039981+0000","last_fullsized":"2024-02-19T15:27:21.039981+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:15.014562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.023713+0000","last_change":"2024-02-19T15:27:21.023713+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.023713+0000","last_undegraded":"2024-02-19T15:27:21.023713+0000","last_fullsized":"2024-02-19T15:27:21.023713+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.043399+0000","last_change":"2024-02-19T15:27:21.042086+0000","last_active":"2024-02-19T15:27:21.043399+0000","last_peered":"2024-02-19T15:27:21.043399+0000","last_clean":"2024-02-19T15:27:21.043399+0000","last_became_active":"2024-02-19T15:27:21.041580+0000","last_became_peered":"2024-02-19T15:27:21.041580+0000","last_unstale":"2024-02-19T15:27:21.043399+0000","last_undegraded":"2024-02-19T15:27:21.043399+0000","last_fullsized":"2024-02-19T15:27:21.043399+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:20:44.224840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.016067+0000","last_change":"2024-02-19T15:27:21.016067+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.016067+0000","last_undegraded":"2024-02-19T15:27:21.016067+0000","last_fullsized":"2024-02-19T15:27:21.016067+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.039839+0000","last_change":"2024-02-19T15:27:21.038892+0000","last_active":"2024-02-19T15:27:21.039839+0000","last_peered":"2024-02-19T15:27:21.039839+0000","last_clean":"2024-02-19T15:27:21.039839+0000","last_became_active":"2024-02-19T15:27:21.038595+0000","last_became_peered":"2024-02-19T15:27:21.038595+0000","last_unstale":"2024-02-19T15:27:21.039839+0000","last_undegraded":"2024-02-19T15:27:21.039839+0000","last_fullsized":"2024-02-19T15:27:21.039839+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:27.660797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.027042+0000","last_change":"2024-02-19T15:27:21.027042+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.027042+0000","last_undegraded":"2024-02-19T15:27:21.027042+0000","last_fullsized":"2024-02-19T15:27:21.027042+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.018777+0000","last_change":"2024-02-19T15:27:21.017652+0000","last_active":"2024-02-19T15:27:21.018777+0000","last_peered":"2024-02-19T15:27:21.018777+0000","last_clean":"2024-02-19T15:27:21.018777+0000","last_became_active":"2024-02-19T15:27:21.017345+0000","last_became_peered":"2024-02-19T15:27:21.017345+0000","last_unstale":"2024-02-19T15:27:21.018777+0000","last_undegraded":"2024-02-19T15:27:21.018777+0000","last_fullsized":"2024-02-19T15:27:21.018777+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:21:04.018088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.042243+0000","last_change":"2024-02-19T15:27:22.035845+0000","last_active":"2024-02-19T15:27:22.042243+0000","last_peered":"2024-02-19T15:27:22.042243+0000","last_clean":"2024-02-19T15:27:22.042243+0000","last_became_active":"2024-02-19T15:27:22.035521+0000","last_became_peered":"2024-02-19T15:27:22.035521+0000","last_unstale":"2024-02-19T15:27:22.042243+0000","last_undegraded":"2024-02-19T15:27:22.042243+0000","last_fullsized":"2024-02-19T15:27:22.042243+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:10:54.441676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.038834+0000","last_change":"2024-02-19T15:27:21.034876+0000","last_active":"2024-02-19T15:27:21.038834+0000","last_peered":"2024-02-19T15:27:21.038834+0000","last_clean":"2024-02-19T15:27:21.038834+0000","last_became_active":"2024-02-19T15:27:21.032593+0000","last_became_peered":"2024-02-19T15:27:21.032593+0000","last_unstale":"2024-02-19T15:27:21.038834+0000","last_undegraded":"2024-02-19T15:27:21.038834+0000","last_fullsized":"2024-02-19T15:27:21.038834+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:38:39.139291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.022396+0000","last_change":"2024-02-19T15:27:21.022396+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.022396+0000","last_undegraded":"2024-02-19T15:27:21.022396+0000","last_fullsized":"2024-02-19T15:27:21.022396+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.029072+0000","last_change":"2024-02-19T15:27:21.025533+0000","last_active":"2024-02-19T15:27:22.029072+0000","last_peered":"2024-02-19T15:27:22.029072+0000","last_clean":"2024-02-19T15:27:22.029072+0000","last_became_active":"2024-02-19T15:27:21.025161+0000","last_became_peered":"2024-02-19T15:27:21.025161+0000","last_unstale":"2024-02-19T15:27:22.029072+0000","last_undegraded":"2024-02-19T15:27:22.029072+0000","last_fullsized":"2024-02-19T15:27:22.029072+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:59:09.129286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.023032+0000","last_change":"2024-02-19T15:27:21.023032+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.023032+0000","last_undegraded":"2024-02-19T15:27:21.023032+0000","last_fullsized":"2024-02-19T15:27:21.023032+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.023272+0000","last_change":"2024-02-19T15:27:21.021965+0000","last_active":"2024-02-19T15:27:21.023272+0000","last_peered":"2024-02-19T15:27:21.023272+0000","last_clean":"2024-02-19T15:27:21.023272+0000","last_became_active":"2024-02-19T15:27:21.021319+0000","last_became_peered":"2024-02-19T15:27:21.021319+0000","last_unstale":"2024-02-19T15:27:21.023272+0000","last_undegraded":"2024-02-19T15:27:21.023272+0000","last_fullsized":"2024-02-19T15:27:21.023272+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:58:20.368186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.041845+0000","last_change":"2024-02-19T15:27:22.037785+0000","last_active":"2024-02-19T15:27:22.041845+0000","last_peered":"2024-02-19T15:27:22.041845+0000","last_clean":"2024-02-19T15:27:22.041845+0000","last_became_active":"2024-02-19T15:27:22.037460+0000","last_became_peered":"2024-02-19T15:27:22.037460+0000","last_unstale":"2024-02-19T15:27:22.041845+0000","last_undegraded":"2024-02-19T15:27:22.041845+0000","last_fullsized":"2024-02-19T15:27:22.041845+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:46.258004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.031658+0000","last_change":"2024-02-19T15:27:21.028129+0000","last_active":"2024-02-19T15:27:21.031658+0000","last_peered":"2024-02-19T15:27:21.031658+0000","last_clean":"2024-02-19T15:27:21.031658+0000","last_became_active":"2024-02-19T15:27:21.027822+0000","last_became_peered":"2024-02-19T15:27:21.027822+0000","last_unstale":"2024-02-19T15:27:21.031658+0000","last_undegraded":"2024-02-19T15:27:21.031658+0000","last_fullsized":"2024-02-19T15:27:21.031658+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:26.546966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.021775+0000","last_change":"2024-02-19T15:27:21.021775+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.021775+0000","last_undegraded":"2024-02-19T15:27:21.021775+0000","last_fullsized":"2024-02-19T15:27:21.021775+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.033732+0000","last_change":"2024-02-19T15:27:21.032501+0000","last_active":"2024-02-19T15:27:21.033732+0000","last_peered":"2024-02-19T15:27:21.033732+0000","last_clean":"2024-02-19T15:27:21.033732+0000","last_became_active":"2024-02-19T15:27:21.032206+0000","last_became_peered":"2024-02-19T15:27:21.032206+0000","last_unstale":"2024-02-19T15:27:21.033732+0000","last_undegraded":"2024-02-19T15:27:21.033732+0000","last_fullsized":"2024-02-19T15:27:21.033732+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:25:30.120547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.022269+0000","last_change":"2024-02-19T15:27:21.022269+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.022269+0000","last_undegraded":"2024-02-19T15:27:21.022269+0000","last_fullsized":"2024-02-19T15:27:21.022269+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.036528+0000","last_change":"2024-02-19T15:27:21.035917+0000","last_active":"2024-02-19T15:27:21.036528+0000","last_peered":"2024-02-19T15:27:21.036528+0000","last_clean":"2024-02-19T15:27:21.036528+0000","last_became_active":"2024-02-19T15:27:21.035628+0000","last_became_peered":"2024-02-19T15:27:21.035628+0000","last_unstale":"2024-02-19T15:27:21.036528+0000","last_undegraded":"2024-02-19T15:27:21.036528+0000","last_fullsized":"2024-02-19T15:27:21.036528+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:03:18.300707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.014923+0000","last_change":"2024-02-19T15:27:21.014923+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.014923+0000","last_undegraded":"2024-02-19T15:27:21.014923+0000","last_fullsized":"2024-02-19T15:27:21.014923+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.032053+0000","last_change":"2024-02-19T15:27:21.029497+0000","last_active":"2024-02-19T15:27:21.032053+0000","last_peered":"2024-02-19T15:27:21.032053+0000","last_clean":"2024-02-19T15:27:21.032053+0000","last_became_active":"2024-02-19T15:27:21.029251+0000","last_became_peered":"2024-02-19T15:27:21.029251+0000","last_unstale":"2024-02-19T15:27:21.032053+0000","last_undegraded":"2024-02-19T15:27:21.032053+0000","last_fullsized":"2024-02-19T15:27:21.032053+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:16:56.947269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.022312+0000","last_change":"2024-02-19T15:27:21.022312+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.022312+0000","last_undegraded":"2024-02-19T15:27:21.022312+0000","last_fullsized":"2024-02-19T15:27:21.022312+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.027420+0000","last_change":"2024-02-19T15:27:21.024536+0000","last_active":"2024-02-19T15:27:21.027420+0000","last_peered":"2024-02-19T15:27:21.027420+0000","last_clean":"2024-02-19T15:27:21.027420+0000","last_became_active":"2024-02-19T15:27:21.020949+0000","last_became_peered":"2024-02-19T15:27:21.020949+0000","last_unstale":"2024-02-19T15:27:21.027420+0000","last_undegraded":"2024-02-19T15:27:21.027420+0000","last_fullsized":"2024-02-19T15:27:21.027420+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:16:06.605936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":106,"state":"creating+peering","last_fresh":"2024-02-19T15:27:21.025938+0000","last_change":"2024-02-19T15:27:21.025938+0000","last_active":"2024-02-19T15:27:21.004796+0000","last_peered":"2024-02-19T15:27:21.004796+0000","last_clean":"2024-02-19T15:27:21.004796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:27:21.025938+0000","last_undegraded":"2024-02-19T15:27:21.025938+0000","last_fullsized":"2024-02-19T15:27:21.025938+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.036666+0000","last_change":"2024-02-19T15:27:21.036160+0000","last_active":"2024-02-19T15:27:21.036666+0000","last_peered":"2024-02-19T15:27:21.036666+0000","last_clean":"2024-02-19T15:27:21.036666+0000","last_became_active":"2024-02-19T15:27:21.035805+0000","last_became_peered":"2024-02-19T15:27:21.035805+0000","last_unstale":"2024-02-19T15:27:21.036666+0000","last_undegraded":"2024-02-19T15:27:21.036666+0000","last_fullsized":"2024-02-19T15:27:21.036666+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:51:06.106358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":320,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-19T15:27:21.018304+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:27:21.018304+0000","last_peered":"2024-02-19T15:27:21.018304+0000","last_clean":"2024-02-19T15:27:21.018304+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:27:21.018304+0000","last_undegraded":"2024-02-19T15:27:21.018304+0000","last_fullsized":"2024-02-19T15:27:21.018304+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509509,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1314816,"data_stored":563776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.622,"15min":0.62},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.66100000000000003,"15min":0.746},"min":{"1min":0.39100000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.8100000000000001,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.76100000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63800000000000001,"15min":0.65000000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.44,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.774,"5min":2.774,"15min":2.774},"last":0.58099999999999996}]},{"osd":2,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63100000000000001,"15min":0.63200000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.69899999999999995,"15min":0.68899999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.69299999999999995}]},{"osd":3,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":0.83499999999999996,"15min":0.72699999999999998},"min":{"1min":0.46600000000000003,"5min":0.438,"15min":0.438},"max":{"1min":14.045999999999999,"5min":14.045999999999999,"15min":14.045999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.67600000000000005,"15min":0.66200000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.85999999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.64500000000000002,"15min":0.61299999999999999},"min":{"1min":0.32400000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.69099999999999995,"15min":0.625},"min":{"1min":0.35599999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":2.8500000000000001,"5min":2.8500000000000001,"15min":2.8500000000000001},"last":0.83599999999999997}]},{"osd":5,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.61899999999999999},"min":{"1min":0.38200000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.72899999999999998,"15min":0.68500000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.73199999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.75900000000000001,"15min":0.68100000000000005},"min":{"1min":0.42799999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":3.0539999999999998,"5min":3.0539999999999998,"15min":3.0539999999999998},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.755,"15min":0.69299999999999995},"min":{"1min":0.439,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.256,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.91100000000000003}]}]},{"osd":7,"up_from":14,"seq":60129542213,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37200,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334640,"statfs":{"total":96636764160,"available":96598671360,"internally_reserved":0,"allocated":1896448,"data_stored":1147869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":2.016,"5min":0.88300000000000001,"15min":0.68500000000000005},"min":{"1min":0.51800000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":15.127000000000001,"5min":15.127000000000001,"15min":15.127000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.71899999999999997,"15min":0.69399999999999995},"min":{"1min":0.47499999999999998,"5min":0.376,"15min":0.376},"max":{"1min":6.0890000000000004,"5min":6.0890000000000004,"15min":6.0890000000000004},"last":0.63700000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":2.0910000000000002,"5min":0.93600000000000005,"15min":0.70199999999999996},"min":{"1min":0.47499999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":15.196999999999999,"5min":15.196999999999999,"15min":15.196999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.3100000000000001,"5min":0.81999999999999995,"15min":0.76000000000000001},"min":{"1min":0.45600000000000002,"5min":0.437,"15min":0.437},"max":{"1min":10.933999999999999,"5min":10.933999999999999,"15min":10.933999999999999},"last":0.80800000000000005}]},{"osd":2,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":0.79600000000000004,"15min":0.68600000000000005},"min":{"1min":0.45700000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":10.84,"5min":10.84,"15min":10.84},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.71199999999999997,"15min":0.69999999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":2.3170000000000002,"5min":2.508,"15min":2.508},"last":0.77700000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":1.6539999999999999,"5min":0.86199999999999999,"15min":0.72599999999999998},"min":{"1min":0.49199999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":17.094000000000001,"5min":17.094000000000001,"15min":17.094000000000001},"last":0.746},{"interface":"front","average":{"1min":1.226,"5min":0.81799999999999995,"15min":0.73099999999999998},"min":{"1min":0.46300000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":7.7190000000000003,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.85299999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.61299999999999999,"15min":0.54500000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.876},{"interface":"front","average":{"1min":2.266,"5min":0.95799999999999996,"15min":0.69499999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":26.625,"5min":26.625,"15min":26.625},"last":0.79600000000000004}]},{"osd":5,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":2.5350000000000001,"5min":1.0549999999999999,"15min":0.71699999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":31.844999999999999,"5min":31.844999999999999,"15min":31.844999999999999},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.6100000000000001,"5min":0.876,"15min":0.70399999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":16.776,"5min":16.776,"15min":16.776},"last":0.83499999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.70499999999999996,"15min":0.625},"min":{"1min":0.375,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.76400000000000001,"15min":0.68799999999999994},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":3.5099999999999998,"5min":3.5099999999999998,"15min":3.5099999999999998},"last":0.97599999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542214,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36616,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335224,"statfs":{"total":96636764160,"available":96599269376,"internally_reserved":0,"allocated":1302528,"data_stored":557501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.69699999999999995,"15min":0.65300000000000002},"min":{"1min":0.48399999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":2.6659999999999999,"5min":5.7789999999999999,"15min":5.7789999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73799999999999999,"15min":0.64600000000000002},"min":{"1min":0.45000000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.2370000000000001,"5min":12.359999999999999,"15min":12.359999999999999},"last":0.91700000000000004}]},{"osd":1,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.76600000000000001,"15min":0.67900000000000005},"min":{"1min":0.496,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.6499999999999999,"5min":10.938000000000001,"15min":10.938000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.82199999999999995,"15min":0.72399999999999998},"min":{"1min":0.45800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":3.625,"5min":10.531000000000001,"15min":10.531000000000001},"last":0.89500000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.81499999999999995,"15min":0.72699999999999998},"min":{"1min":0.503,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":2.589,"5min":12.925000000000001,"15min":12.925000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.74099999999999999,"15min":0.64500000000000002},"min":{"1min":0.46999999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89100000000000001,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.77300000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.96499999999999997,"15min":0.753},"min":{"1min":0.51000000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":5.492,"5min":26.161999999999999,"15min":26.161999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.76200000000000001},"min":{"1min":0.52700000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":6.4939999999999998,"5min":27.934999999999999,"15min":27.934999999999999},"last":0.86399999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":0.79900000000000004,"15min":0.63300000000000001},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":11.73,"5min":11.73,"15min":11.73},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.117,"5min":0.80200000000000005,"15min":0.70199999999999996},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":10.731,"5min":10.731,"15min":10.731},"last":0.68600000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.76900000000000002,"15min":0.67700000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.6320000000000001,"5min":6.5229999999999997,"15min":6.5229999999999997},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.248,"5min":0.92600000000000005,"15min":0.78400000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":10.432,"5min":10.432,"15min":10.432},"last":0.73499999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.78700000000000003,"15min":0.69799999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.9450000000000001,"5min":11.986000000000001,"15min":11.986000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.83099999999999996,"15min":0.73299999999999998},"min":{"1min":0.42999999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":6.5339999999999998,"5min":8.9770000000000003,"15min":8.9770000000000003},"last":0.749}]}]},{"osd":4,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36616,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335224,"statfs":{"total":96636764160,"available":96599269376,"internally_reserved":0,"allocated":1302528,"data_stored":557501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":4.1779999999999999,"5min":1.3089999999999999,"15min":0.83099999999999996},"min":{"1min":0.38,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":76.117000000000004,"5min":76.117000000000004,"15min":76.117000000000004},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.746,"5min":0.65300000000000002,"15min":0.61499999999999999},"min":{"1min":0.40000000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.2709999999999999,"5min":3.2709999999999999,"15min":3.2709999999999999},"last":0.42799999999999999}]},{"osd":1,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":4.4470000000000001,"5min":1.409,"15min":0.875},"min":{"1min":0.45800000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":80.513000000000005,"5min":80.513000000000005,"15min":80.513000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.70799999999999996,"15min":0.65500000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":7.181,"5min":7.181,"15min":7.181},"last":0.54100000000000004}]},{"osd":2,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":0.77900000000000003,"15min":0.70199999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":14.362,"5min":14.362,"15min":14.362},"last":0.67200000000000004},{"interface":"front","average":{"1min":4.6100000000000003,"5min":1.4690000000000001,"15min":0.93899999999999995},"min":{"1min":0.42299999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":84.093000000000004,"5min":84.093000000000004,"15min":84.093000000000004},"last":0.51100000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":4.6539999999999999,"5min":1.5049999999999999,"15min":0.94799999999999995},"min":{"1min":0.504,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":84.140000000000001,"5min":84.140000000000001,"15min":84.140000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.44400000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.56499999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":0.76500000000000001,"15min":0.64900000000000002},"min":{"1min":0.379,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":14.318,"5min":14.318,"15min":14.318},"last":0.69299999999999995},{"interface":"front","average":{"1min":4.274,"5min":1.427,"15min":0.92300000000000004},"min":{"1min":0.41099999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":76.164000000000001,"5min":76.164000000000001,"15min":76.164000000000001},"last":0.626}]},{"osd":6,"last update":"Mon Feb 19 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64100000000000001,"15min":0.58899999999999997},"min":{"1min":0.33500000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":3.5019999999999998,"5min":1.2589999999999999,"15min":0.84599999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":60.151000000000003,"5min":60.151000000000003,"15min":60.151000000000003},"last":0.60699999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":2.044,"5min":0.97999999999999998,"15min":0.76400000000000001},"min":{"1min":0.40699999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":29.036000000000001,"5min":29.036000000000001,"15min":29.036000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":1.0449999999999999,"5min":0.82099999999999995,"15min":0.755},"min":{"1min":0.50700000000000001,"5min":0.443,"15min":0.443},"max":{"1min":7.101,"5min":7.101,"15min":7.101},"last":0.64700000000000002}]}]},{"osd":3,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40772,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331068,"statfs":{"total":96636764160,"available":96595013632,"internally_reserved":0,"allocated":1302528,"data_stored":557501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41799999999999998,"15min":0.44400000000000001},"min":{"1min":0.23499999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.82699999999999996,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.5,"15min":0.63200000000000001},"min":{"1min":0.28899999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.0840000000000001,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.51200000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.437,"15min":0.443},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.54400000000000004,"15min":0.69199999999999995},"min":{"1min":0.316,"5min":0.31,"15min":0.31},"max":{"1min":0.85099999999999998,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.55400000000000005}]},{"osd":2,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55500000000000005,"15min":0.60699999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53700000000000003,"15min":0.58499999999999996},"min":{"1min":0.38400000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.57599999999999996}]},{"osd":4,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71299999999999997,"15min":0.68200000000000005},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.97999999999999998,"5min":1.004,"15min":1.004},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.75800000000000001,"15min":0.84599999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0169999999999999,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.88600000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.77000000000000002,"15min":0.84399999999999997},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.99099999999999999,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.73499999999999999,"15min":0.745},"min":{"1min":0.46999999999999997,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89700000000000002,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.92000000000000004}]},{"osd":6,"last update":"Mon Feb 19 15:27:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.79300000000000004,"15min":0.879},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.3180000000000001,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":1.052},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.77500000000000002,"15min":0.81000000000000005},"min":{"1min":0.53900000000000003,"5min":0.502,"15min":0.502},"max":{"1min":1.206,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":1.0349999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.755,"15min":0.77700000000000002},"min":{"1min":0.57099999999999995,"5min":0.504,"15min":0.504},"max":{"1min":0.996,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.80400000000000005,"15min":0.90800000000000003},"min":{"1min":0.54800000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.1919999999999999,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.98299999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1314816,"data_stored":563776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.60099999999999998,"15min":0.67500000000000004},"min":{"1min":0.26600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.48899999999999999,"5min":15.42,"15min":15.42},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.61199999999999999,"15min":0.85099999999999998},"min":{"1min":0.28199999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68700000000000006,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.38500000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.65500000000000003,"15min":0.86299999999999999},"min":{"1min":0.29399999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72799999999999998,"5min":9.5269999999999992,"15min":9.5269999999999992},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.83999999999999997,"15min":0.97199999999999998},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.75600000000000001,"5min":22.954999999999998,"15min":22.954999999999998},"last":0.53700000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.61499999999999999,"15min":0.76500000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.75,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.68700000000000006,"15min":0.80400000000000005},"min":{"1min":0.42399999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81399999999999995,"5min":7.734,"15min":7.734},"last":0.63700000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.86399999999999999,"15min":0.88600000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.97999999999999998,"5min":15.369,"15min":15.369},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.77800000000000002,"15min":0.876},"min":{"1min":0.51900000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93600000000000005,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.78800000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.83299999999999996,"15min":0.876},"min":{"1min":0.47199999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.006,"5min":7.6520000000000001,"15min":7.6520000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.93400000000000005,"15min":0.95699999999999996},"min":{"1min":0.496,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.026,"5min":15.391,"15min":15.391},"last":0.87}]},{"osd":6,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.90900000000000003,"15min":0.92800000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.92000000000000004,"5min":15.337,"15min":15.337},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.81399999999999995,"15min":0.93799999999999994},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.93700000000000006,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.93899999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.85399999999999998,"15min":0.96399999999999997},"min":{"1min":0.57099999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.98099999999999998,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.86099999999999999,"15min":0.93500000000000005},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94299999999999995,"5min":7.6929999999999996,"15min":7.6929999999999996},"last":0.85299999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542214,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40784,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331056,"statfs":{"total":96636764160,"available":96595001344,"internally_reserved":0,"allocated":1314816,"data_stored":563776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38600000000000001,"15min":0.42199999999999999},"min":{"1min":0.25700000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.68999999999999995,"5min":0.747,"15min":0.747},"last":0.432},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.434,"15min":0.46500000000000002},"min":{"1min":0.312,"5min":0.25,"15min":0.25},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.33200000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.44500000000000001},"min":{"1min":0.30499999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.47799999999999998,"15min":0.52300000000000002},"min":{"1min":0.32400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.74099999999999999,"5min":1.071,"15min":1.071},"last":0.66100000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57999999999999996,"15min":0.63},"min":{"1min":0.45500000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.1200000000000001,"5min":1.365,"15min":1.365},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.55800000000000005,"15min":0.625},"min":{"1min":0.35399999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0069999999999999,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.59299999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.67400000000000004,"15min":0.62},"min":{"1min":0.434,"5min":0.377,"15min":0.377},"max":{"1min":1.0409999999999999,"5min":1.127,"15min":1.127},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.69999999999999996,"15min":0.68200000000000005},"min":{"1min":0.56100000000000005,"5min":0.435,"15min":0.435},"max":{"1min":1.083,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.82899999999999996}]},{"osd":5,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.71199999999999997,"15min":0.69799999999999995},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.88800000000000001,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70899999999999996,"15min":0.68700000000000006},"min":{"1min":0.51200000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.96399999999999997,"5min":1.107,"15min":1.107},"last":0.745}]},{"osd":6,"last update":"Mon Feb 19 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.73199999999999998,"15min":0.72899999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.97799999999999998,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.755,"5min":0.74399999999999999,"15min":0.746},"min":{"1min":0.56299999999999994,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.1559999999999999,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.97099999999999997}]},{"osd":7,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.78400000000000003},"min":{"1min":0.52700000000000002,"5min":0.496,"15min":0.496},"max":{"1min":0.94599999999999995,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":1.0800000000000001},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.75},"min":{"1min":0.54500000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.99299999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":1.0509999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542212,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41360,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330480,"statfs":{"total":96636764160,"available":96594411520,"internally_reserved":0,"allocated":1896448,"data_stored":1147869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.45300000000000001,"15min":0.60999999999999999},"min":{"1min":0.24299999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.745,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.45800000000000002,"15min":0.48199999999999998},"min":{"1min":0.34200000000000003,"5min":0.252,"15min":0.252},"max":{"1min":0.77300000000000002,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.61399999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59999999999999998,"15min":0.76600000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.82399999999999995,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.56100000000000005,"15min":0.72699999999999998},"min":{"1min":0.35699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.85899999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.42699999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52000000000000002,"15min":0.66300000000000003},"min":{"1min":0.307,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.93799999999999994,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53300000000000003,"15min":0.60499999999999998},"min":{"1min":0.376,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.052,"5min":1.345,"15min":1.345},"last":0.40799999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.73199999999999998,"15min":0.73799999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.091,"5min":1.095,"15min":1.095},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.79500000000000004,"15min":0.90900000000000003},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.085,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.69199999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.71399999999999997,"15min":0.74299999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.036,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.73699999999999999,"15min":0.84099999999999997},"min":{"1min":0.53600000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.169,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.5}]},{"osd":6,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.76400000000000001,"15min":0.89600000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.1200000000000001,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.71999999999999997,"15min":0.751},"min":{"1min":0.50800000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.70799999999999996}]},{"osd":7,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.79600000000000004,"15min":0.92400000000000004},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.1399999999999999,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.78100000000000003,"15min":0.92300000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.149,"5min":4.915,"15min":4.915},"last":0.65200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:27:27.183 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3f is not active+clean 2024-02-19T15:27:27.183 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.031361+0000', 'last_change': '2024-02-19T15:27:21.031361+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.031361+0000', 'last_undegraded': '2024-02-19T15:27:21.031361+0000', 'last_fullsized': '2024-02-19T15:27:21.031361+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:27:27.184 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3e is not active+clean 2024-02-19T15:27:27.184 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.030203+0000', 'last_change': '2024-02-19T15:27:21.030203+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.030203+0000', 'last_undegraded': '2024-02-19T15:27:21.030203+0000', 'last_fullsized': '2024-02-19T15:27:21.030203+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T15:27:27.184 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3d is not active+clean 2024-02-19T15:27:27.184 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.016897+0000', 'last_change': '2024-02-19T15:27:21.016897+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.016897+0000', 'last_undegraded': '2024-02-19T15:27:21.016897+0000', 'last_fullsized': '2024-02-19T15:27:21.016897+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T15:27:27.184 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3c is not active+clean 2024-02-19T15:27:27.184 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.030394+0000', 'last_change': '2024-02-19T15:27:21.030394+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.030394+0000', 'last_undegraded': '2024-02-19T15:27:21.030394+0000', 'last_fullsized': '2024-02-19T15:27:21.030394+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:27:27.184 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3b is not active+clean 2024-02-19T15:27:27.184 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.023740+0000', 'last_change': '2024-02-19T15:27:21.023740+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.023740+0000', 'last_undegraded': '2024-02-19T15:27:21.023740+0000', 'last_fullsized': '2024-02-19T15:27:21.023740+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T15:27:27.184 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3a is not active+clean 2024-02-19T15:27:27.184 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.011541+0000', 'last_change': '2024-02-19T15:27:21.011541+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.011541+0000', 'last_undegraded': '2024-02-19T15:27:21.011541+0000', 'last_fullsized': '2024-02-19T15:27:21.011541+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T15:27:27.185 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.39 is not active+clean 2024-02-19T15:27:27.185 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.026169+0000', 'last_change': '2024-02-19T15:27:21.026169+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.026169+0000', 'last_undegraded': '2024-02-19T15:27:21.026169+0000', 'last_fullsized': '2024-02-19T15:27:21.026169+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T15:27:27.185 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.38 is not active+clean 2024-02-19T15:27:27.185 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.016613+0000', 'last_change': '2024-02-19T15:27:21.016613+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.016613+0000', 'last_undegraded': '2024-02-19T15:27:21.016613+0000', 'last_fullsized': '2024-02-19T15:27:21.016613+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T15:27:27.185 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.37 is not active+clean 2024-02-19T15:27:27.185 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.030819+0000', 'last_change': '2024-02-19T15:27:21.030819+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.030819+0000', 'last_undegraded': '2024-02-19T15:27:21.030819+0000', 'last_fullsized': '2024-02-19T15:27:21.030819+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:27:27.185 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.36 is not active+clean 2024-02-19T15:27:27.185 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.029180+0000', 'last_change': '2024-02-19T15:27:21.029180+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.029180+0000', 'last_undegraded': '2024-02-19T15:27:21.029180+0000', 'last_fullsized': '2024-02-19T15:27:21.029180+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:27:27.185 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.35 is not active+clean 2024-02-19T15:27:27.185 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.026903+0000', 'last_change': '2024-02-19T15:27:21.026903+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.026903+0000', 'last_undegraded': '2024-02-19T15:27:21.026903+0000', 'last_fullsized': '2024-02-19T15:27:21.026903+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T15:27:27.185 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.34 is not active+clean 2024-02-19T15:27:27.186 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.028102+0000', 'last_change': '2024-02-19T15:27:21.028102+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.028102+0000', 'last_undegraded': '2024-02-19T15:27:21.028102+0000', 'last_fullsized': '2024-02-19T15:27:21.028102+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:27:27.186 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.33 is not active+clean 2024-02-19T15:27:27.186 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.015426+0000', 'last_change': '2024-02-19T15:27:21.015426+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.015426+0000', 'last_undegraded': '2024-02-19T15:27:21.015426+0000', 'last_fullsized': '2024-02-19T15:27:21.015426+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T15:27:27.186 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.32 is not active+clean 2024-02-19T15:27:27.186 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.016518+0000', 'last_change': '2024-02-19T15:27:21.016518+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.016518+0000', 'last_undegraded': '2024-02-19T15:27:21.016518+0000', 'last_fullsized': '2024-02-19T15:27:21.016518+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T15:27:27.186 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.31 is not active+clean 2024-02-19T15:27:27.186 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.011705+0000', 'last_change': '2024-02-19T15:27:21.011705+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.011705+0000', 'last_undegraded': '2024-02-19T15:27:21.011705+0000', 'last_fullsized': '2024-02-19T15:27:21.011705+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T15:27:27.186 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.30 is not active+clean 2024-02-19T15:27:27.186 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.026401+0000', 'last_change': '2024-02-19T15:27:21.026401+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.026401+0000', 'last_undegraded': '2024-02-19T15:27:21.026401+0000', 'last_fullsized': '2024-02-19T15:27:21.026401+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:27:27.186 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2f is not active+clean 2024-02-19T15:27:27.186 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.025923+0000', 'last_change': '2024-02-19T15:27:21.025923+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.025923+0000', 'last_undegraded': '2024-02-19T15:27:21.025923+0000', 'last_fullsized': '2024-02-19T15:27:21.025923+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T15:27:27.187 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2d is not active+clean 2024-02-19T15:27:27.187 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.016248+0000', 'last_change': '2024-02-19T15:27:21.016248+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.016248+0000', 'last_undegraded': '2024-02-19T15:27:21.016248+0000', 'last_fullsized': '2024-02-19T15:27:21.016248+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T15:27:27.187 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2c is not active+clean 2024-02-19T15:27:27.187 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.024313+0000', 'last_change': '2024-02-19T15:27:21.024313+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.024313+0000', 'last_undegraded': '2024-02-19T15:27:21.024313+0000', 'last_fullsized': '2024-02-19T15:27:21.024313+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T15:27:27.187 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2b is not active+clean 2024-02-19T15:27:27.187 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.024688+0000', 'last_change': '2024-02-19T15:27:21.024688+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.024688+0000', 'last_undegraded': '2024-02-19T15:27:21.024688+0000', 'last_fullsized': '2024-02-19T15:27:21.024688+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T15:27:27.187 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2a is not active+clean 2024-02-19T15:27:27.187 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.033511+0000', 'last_change': '2024-02-19T15:27:21.033511+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.033511+0000', 'last_undegraded': '2024-02-19T15:27:21.033511+0000', 'last_fullsized': '2024-02-19T15:27:21.033511+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T15:27:27.187 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.29 is not active+clean 2024-02-19T15:27:27.187 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.024070+0000', 'last_change': '2024-02-19T15:27:21.024070+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.024070+0000', 'last_undegraded': '2024-02-19T15:27:21.024070+0000', 'last_fullsized': '2024-02-19T15:27:21.024070+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T15:27:27.187 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.28 is not active+clean 2024-02-19T15:27:27.187 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.033844+0000', 'last_change': '2024-02-19T15:27:21.033844+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.033844+0000', 'last_undegraded': '2024-02-19T15:27:21.033844+0000', 'last_fullsized': '2024-02-19T15:27:21.033844+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T15:27:27.188 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.27 is not active+clean 2024-02-19T15:27:27.188 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.024026+0000', 'last_change': '2024-02-19T15:27:21.024026+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.024026+0000', 'last_undegraded': '2024-02-19T15:27:21.024026+0000', 'last_fullsized': '2024-02-19T15:27:21.024026+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T15:27:27.188 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.26 is not active+clean 2024-02-19T15:27:27.188 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.025203+0000', 'last_change': '2024-02-19T15:27:21.025203+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.025203+0000', 'last_undegraded': '2024-02-19T15:27:21.025203+0000', 'last_fullsized': '2024-02-19T15:27:21.025203+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:27:27.188 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.25 is not active+clean 2024-02-19T15:27:27.188 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.012320+0000', 'last_change': '2024-02-19T15:27:21.012320+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.012320+0000', 'last_undegraded': '2024-02-19T15:27:21.012320+0000', 'last_fullsized': '2024-02-19T15:27:21.012320+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T15:27:27.188 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.24 is not active+clean 2024-02-19T15:27:27.188 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.033395+0000', 'last_change': '2024-02-19T15:27:21.033395+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.033395+0000', 'last_undegraded': '2024-02-19T15:27:21.033395+0000', 'last_fullsized': '2024-02-19T15:27:21.033395+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T15:27:27.188 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.23 is not active+clean 2024-02-19T15:27:27.188 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.024077+0000', 'last_change': '2024-02-19T15:27:21.024077+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.024077+0000', 'last_undegraded': '2024-02-19T15:27:21.024077+0000', 'last_fullsized': '2024-02-19T15:27:21.024077+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T15:27:27.189 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.22 is not active+clean 2024-02-19T15:27:27.189 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.021450+0000', 'last_change': '2024-02-19T15:27:21.021450+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.021450+0000', 'last_undegraded': '2024-02-19T15:27:21.021450+0000', 'last_fullsized': '2024-02-19T15:27:21.021450+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T15:27:27.189 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.21 is not active+clean 2024-02-19T15:27:27.189 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.026040+0000', 'last_change': '2024-02-19T15:27:21.026040+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.026040+0000', 'last_undegraded': '2024-02-19T15:27:21.026040+0000', 'last_fullsized': '2024-02-19T15:27:21.026040+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:27:27.189 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.20 is not active+clean 2024-02-19T15:27:27.189 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.011531+0000', 'last_change': '2024-02-19T15:27:21.011531+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.011531+0000', 'last_undegraded': '2024-02-19T15:27:21.011531+0000', 'last_fullsized': '2024-02-19T15:27:21.011531+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T15:27:27.189 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1e is not active+clean 2024-02-19T15:27:27.189 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.013560+0000', 'last_change': '2024-02-19T15:27:21.013560+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.013560+0000', 'last_undegraded': '2024-02-19T15:27:21.013560+0000', 'last_fullsized': '2024-02-19T15:27:21.013560+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T15:27:27.189 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1c is not active+clean 2024-02-19T15:27:27.189 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.012534+0000', 'last_change': '2024-02-19T15:27:21.012534+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.012534+0000', 'last_undegraded': '2024-02-19T15:27:21.012534+0000', 'last_fullsized': '2024-02-19T15:27:21.012534+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T15:27:27.189 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1d is not active+clean 2024-02-19T15:27:27.189 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.015800+0000', 'last_change': '2024-02-19T15:27:21.015800+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.015800+0000', 'last_undegraded': '2024-02-19T15:27:21.015800+0000', 'last_fullsized': '2024-02-19T15:27:21.015800+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T15:27:27.190 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1a is not active+clean 2024-02-19T15:27:27.190 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.022487+0000', 'last_change': '2024-02-19T15:27:21.022487+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.022487+0000', 'last_undegraded': '2024-02-19T15:27:21.022487+0000', 'last_fullsized': '2024-02-19T15:27:21.022487+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:27:27.190 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1b is not active+clean 2024-02-19T15:27:27.190 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.023070+0000', 'last_change': '2024-02-19T15:27:21.023070+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.023070+0000', 'last_undegraded': '2024-02-19T15:27:21.023070+0000', 'last_fullsized': '2024-02-19T15:27:21.023070+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T15:27:27.190 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.18 is not active+clean 2024-02-19T15:27:27.190 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.024991+0000', 'last_change': '2024-02-19T15:27:21.024991+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.024991+0000', 'last_undegraded': '2024-02-19T15:27:21.024991+0000', 'last_fullsized': '2024-02-19T15:27:21.024991+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:27:27.190 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.19 is not active+clean 2024-02-19T15:27:27.190 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.023512+0000', 'last_change': '2024-02-19T15:27:21.023512+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.023512+0000', 'last_undegraded': '2024-02-19T15:27:21.023512+0000', 'last_fullsized': '2024-02-19T15:27:21.023512+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T15:27:27.190 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.16 is not active+clean 2024-02-19T15:27:27.190 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.021162+0000', 'last_change': '2024-02-19T15:27:21.021162+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.021162+0000', 'last_undegraded': '2024-02-19T15:27:21.021162+0000', 'last_fullsized': '2024-02-19T15:27:21.021162+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T15:27:27.190 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.14 is not active+clean 2024-02-19T15:27:27.190 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.032945+0000', 'last_change': '2024-02-19T15:27:21.032945+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.032945+0000', 'last_undegraded': '2024-02-19T15:27:21.032945+0000', 'last_fullsized': '2024-02-19T15:27:21.032945+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T15:27:27.190 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.15 is not active+clean 2024-02-19T15:27:27.191 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.021714+0000', 'last_change': '2024-02-19T15:27:21.021714+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.021714+0000', 'last_undegraded': '2024-02-19T15:27:21.021714+0000', 'last_fullsized': '2024-02-19T15:27:21.021714+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T15:27:27.191 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.12 is not active+clean 2024-02-19T15:27:27.191 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.029223+0000', 'last_change': '2024-02-19T15:27:21.029223+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.029223+0000', 'last_undegraded': '2024-02-19T15:27:21.029223+0000', 'last_fullsized': '2024-02-19T15:27:21.029223+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T15:27:27.191 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.13 is not active+clean 2024-02-19T15:27:27.191 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.033846+0000', 'last_change': '2024-02-19T15:27:21.033846+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.033846+0000', 'last_undegraded': '2024-02-19T15:27:21.033846+0000', 'last_fullsized': '2024-02-19T15:27:21.033846+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T15:27:27.191 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.11 is not active+clean 2024-02-19T15:27:27.191 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.029752+0000', 'last_change': '2024-02-19T15:27:21.029752+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.029752+0000', 'last_undegraded': '2024-02-19T15:27:21.029752+0000', 'last_fullsized': '2024-02-19T15:27:21.029752+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T15:27:27.191 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.e is not active+clean 2024-02-19T15:27:27.191 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.024250+0000', 'last_change': '2024-02-19T15:27:21.024250+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.024250+0000', 'last_undegraded': '2024-02-19T15:27:21.024250+0000', 'last_fullsized': '2024-02-19T15:27:21.024250+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T15:27:27.191 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.f is not active+clean 2024-02-19T15:27:27.191 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.013930+0000', 'last_change': '2024-02-19T15:27:21.013930+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.013930+0000', 'last_undegraded': '2024-02-19T15:27:21.013930+0000', 'last_fullsized': '2024-02-19T15:27:21.013930+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T15:27:27.192 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.c is not active+clean 2024-02-19T15:27:27.192 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.015439+0000', 'last_change': '2024-02-19T15:27:21.015439+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.015439+0000', 'last_undegraded': '2024-02-19T15:27:21.015439+0000', 'last_fullsized': '2024-02-19T15:27:21.015439+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T15:27:27.192 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.d is not active+clean 2024-02-19T15:27:27.192 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.015175+0000', 'last_change': '2024-02-19T15:27:21.015175+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.015175+0000', 'last_undegraded': '2024-02-19T15:27:21.015175+0000', 'last_fullsized': '2024-02-19T15:27:21.015175+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T15:27:27.192 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.a is not active+clean 2024-02-19T15:27:27.192 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.015962+0000', 'last_change': '2024-02-19T15:27:21.015962+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.015962+0000', 'last_undegraded': '2024-02-19T15:27:21.015962+0000', 'last_fullsized': '2024-02-19T15:27:21.015962+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T15:27:27.192 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.b is not active+clean 2024-02-19T15:27:27.192 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.023713+0000', 'last_change': '2024-02-19T15:27:21.023713+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.023713+0000', 'last_undegraded': '2024-02-19T15:27:21.023713+0000', 'last_fullsized': '2024-02-19T15:27:21.023713+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:27:27.192 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.8 is not active+clean 2024-02-19T15:27:27.192 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.016067+0000', 'last_change': '2024-02-19T15:27:21.016067+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.016067+0000', 'last_undegraded': '2024-02-19T15:27:21.016067+0000', 'last_fullsized': '2024-02-19T15:27:21.016067+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T15:27:27.192 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.9 is not active+clean 2024-02-19T15:27:27.193 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.027042+0000', 'last_change': '2024-02-19T15:27:21.027042+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.027042+0000', 'last_undegraded': '2024-02-19T15:27:21.027042+0000', 'last_fullsized': '2024-02-19T15:27:21.027042+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T15:27:27.193 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.7 is not active+clean 2024-02-19T15:27:27.193 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.022396+0000', 'last_change': '2024-02-19T15:27:21.022396+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.022396+0000', 'last_undegraded': '2024-02-19T15:27:21.022396+0000', 'last_fullsized': '2024-02-19T15:27:21.022396+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:27:27.193 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.4 is not active+clean 2024-02-19T15:27:27.193 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.023032+0000', 'last_change': '2024-02-19T15:27:21.023032+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.023032+0000', 'last_undegraded': '2024-02-19T15:27:21.023032+0000', 'last_fullsized': '2024-02-19T15:27:21.023032+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:27:27.193 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2 is not active+clean 2024-02-19T15:27:27.193 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.021775+0000', 'last_change': '2024-02-19T15:27:21.021775+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.021775+0000', 'last_undegraded': '2024-02-19T15:27:21.021775+0000', 'last_fullsized': '2024-02-19T15:27:21.021775+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T15:27:27.193 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3 is not active+clean 2024-02-19T15:27:27.193 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.022269+0000', 'last_change': '2024-02-19T15:27:21.022269+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.022269+0000', 'last_undegraded': '2024-02-19T15:27:21.022269+0000', 'last_fullsized': '2024-02-19T15:27:21.022269+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T15:27:27.193 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.0 is not active+clean 2024-02-19T15:27:27.193 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.014923+0000', 'last_change': '2024-02-19T15:27:21.014923+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.014923+0000', 'last_undegraded': '2024-02-19T15:27:21.014923+0000', 'last_fullsized': '2024-02-19T15:27:21.014923+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T15:27:27.193 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1 is not active+clean 2024-02-19T15:27:27.193 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.022312+0000', 'last_change': '2024-02-19T15:27:21.022312+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.022312+0000', 'last_undegraded': '2024-02-19T15:27:21.022312+0000', 'last_fullsized': '2024-02-19T15:27:21.022312+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:27:27.194 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.10 is not active+clean 2024-02-19T15:27:27.194 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 106, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:27:21.025938+0000', 'last_change': '2024-02-19T15:27:21.025938+0000', 'last_active': '2024-02-19T15:27:21.004796+0000', 'last_peered': '2024-02-19T15:27:21.004796+0000', 'last_clean': '2024-02-19T15:27:21.004796+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:27:21.025938+0000', 'last_undegraded': '2024-02-19T15:27:21.025938+0000', 'last_fullsized': '2024-02-19T15:27:21.025938+0000', 'mapping_epoch': 106, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 106, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'last_clean_scrub_stamp': '2024-02-19T15:27:21.004796+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T15:27:27.194 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-02-19T15:27:27.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.403+0000 7efd6c000640 1 Processor -- start 2024-02-19T15:27:27.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.403+0000 7efd6c000640 1 -- start start 2024-02-19T15:27:27.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.403+0000 7efd6c000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efd6415dfa0 0x7efd6406fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:27.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.403+0000 7efd6c000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd640b4240 0x7efd640d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:27.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.403+0000 7efd6c000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd6406ff60 con 0x7efd640ce400 2024-02-19T15:27:27.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.403+0000 7efd6c000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd640d1070 con 0x7efd6415dfa0 2024-02-19T15:27:27.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.403+0000 7efd6c000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd6406ec60 con 0x7efd640b4240 2024-02-19T15:27:27.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.403+0000 7efd69600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efd6415dfa0 0x7efd6406fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:27.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.403+0000 7efd69600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efd6415dfa0 0x7efd6406fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55334/0 (socket says 172.21.15.47:55334) 2024-02-19T15:27:27.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.403+0000 7efd69600640 1 -- 172.21.15.47:0/1483739198 learned_addr learned my addr 172.21.15.47:0/1483739198 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:27.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.407+0000 7efd69600640 1 -- 172.21.15.47:0/1483739198 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd640b4240 msgr2=0x7efd640d0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:27.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.403+0000 7efd68c00640 1 --2- 172.21.15.47:0/1483739198 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd640b4240 0x7efd640d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:27.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.407+0000 7efd69600640 1 --2- 172.21.15.47:0/1483739198 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd640b4240 0x7efd640d0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:27.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.407+0000 7efd69600640 1 -- 172.21.15.47:0/1483739198 >> v1:172.21.15.47:6789/0 conn(0x7efd640ce400 legacy=0x7efd640cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:27.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.407+0000 7efd69600640 1 -- 172.21.15.47:0/1483739198 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6406ef40 con 0x7efd6415dfa0 2024-02-19T15:27:27.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.407+0000 7efd68c00640 1 --2- 172.21.15.47:0/1483739198 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd640b4240 0x7efd640d0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:27.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.407+0000 7efd69600640 1 --2- 172.21.15.47:0/1483739198 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd6415dfa0 0x7efd6406fa20 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7efd58009870 tx=0x7efd5801c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1eb06c132677a7a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:27.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.407+0000 7efd63e00640 1 -- 172.21.15.47:0/1483739198 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd580ab640 con 0x7efd6415dfa0 2024-02-19T15:27:27.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.407+0000 7efd63e00640 1 -- 172.21.15.47:0/1483739198 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd580a7070 con 0x7efd6415dfa0 2024-02-19T15:27:27.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.407+0000 7efd63e00640 1 -- 172.21.15.47:0/1483739198 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd580ab9c0 con 0x7efd6415dfa0 2024-02-19T15:27:27.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.407+0000 7efd6c000640 1 -- 172.21.15.47:0/1483739198 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd6415dfa0 msgr2=0x7efd6406fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:27.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.407+0000 7efd6c000640 1 --2- 172.21.15.47:0/1483739198 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd6415dfa0 0x7efd6406fa20 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7efd58009870 tx=0x7efd5801c530 comp rx=0 tx=0).stop 2024-02-19T15:27:27.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.407+0000 7efd6c000640 1 -- 172.21.15.47:0/1483739198 shutdown_connections 2024-02-19T15:27:27.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.407+0000 7efd6c000640 1 --2- 172.21.15.47:0/1483739198 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd640b4240 0x7efd640d0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:27.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.407+0000 7efd6c000640 1 --2- 172.21.15.47:0/1483739198 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd6415dfa0 0x7efd6406fa20 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:27.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.407+0000 7efd6c000640 1 -- 172.21.15.47:0/1483739198 >> 172.21.15.47:0/1483739198 conn(0x7efd640b3690 msgr2=0x7efd640b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:27.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.407+0000 7efd6c000640 1 -- 172.21.15.47:0/1483739198 shutdown_connections 2024-02-19T15:27:27.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd6c000640 1 -- 172.21.15.47:0/1483739198 wait complete. 2024-02-19T15:27:27.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd6c000640 1 Processor -- start 2024-02-19T15:27:27.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd6c000640 1 -- start start 2024-02-19T15:27:27.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd6c000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd640b4240 0x7efd641552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:27.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd6c000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd640ce400 0x7efd64153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:27.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd6c000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd6415dfa0 0x7efd6415c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:27.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd6c000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd64156880 con 0x7efd640ce400 2024-02-19T15:27:27.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd6c000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd64156b60 con 0x7efd6415dfa0 2024-02-19T15:27:27.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd6c000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd64156e40 con 0x7efd640b4240 2024-02-19T15:27:27.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd68c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd640ce400 0x7efd64153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:27.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd68c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd640ce400 0x7efd64153570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55338/0 (socket says 172.21.15.47:55338) 2024-02-19T15:27:27.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd68c00640 1 -- 172.21.15.47:0/2153034637 learned_addr learned my addr 172.21.15.47:0/2153034637 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:27.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd69600640 1 --2- 172.21.15.47:0/2153034637 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd640b4240 0x7efd641552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:27.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd6a000640 1 --2- 172.21.15.47:0/2153034637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd6415dfa0 0x7efd6415c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:27.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd68c00640 1 -- 172.21.15.47:0/2153034637 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd640b4240 msgr2=0x7efd641552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:27.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd68c00640 1 --2- 172.21.15.47:0/2153034637 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd640b4240 0x7efd641552f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:27.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd68c00640 1 -- 172.21.15.47:0/2153034637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd6415dfa0 msgr2=0x7efd6415c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:27.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd68c00640 1 --2- 172.21.15.47:0/2153034637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd6415dfa0 0x7efd6415c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:27.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd68c00640 1 -- 172.21.15.47:0/2153034637 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd580094d0 con 0x7efd640ce400 2024-02-19T15:27:27.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd68c00640 1 --2- 172.21.15.47:0/2153034637 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd640ce400 0x7efd64153570 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7efd5000c7c0 tx=0x7efd5000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:27.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd61600640 1 -- 172.21.15.47:0/2153034637 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd50007a30 con 0x7efd640ce400 2024-02-19T15:27:27.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd61600640 1 -- 172.21.15.47:0/2153034637 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd50007b90 con 0x7efd640ce400 2024-02-19T15:27:27.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd61600640 1 -- 172.21.15.47:0/2153034637 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5000ce70 con 0x7efd640ce400 2024-02-19T15:27:27.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd6c000640 1 -- 172.21.15.47:0/2153034637 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd640ceda0 con 0x7efd640ce400 2024-02-19T15:27:27.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.411+0000 7efd6c000640 1 -- 172.21.15.47:0/2153034637 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd640aea10 con 0x7efd640ce400 2024-02-19T15:27:27.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.415+0000 7efd6c000640 1 -- 172.21.15.47:0/2153034637 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd640aec00 con 0x7efd640ce400 2024-02-19T15:27:27.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.415+0000 7efd61600640 1 -- 172.21.15.47:0/2153034637 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efd500113f0 con 0x7efd640ce400 2024-02-19T15:27:27.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.415+0000 7efd61600640 1 --2- 172.21.15.47:0/2153034637 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd44066e40 0x7efd44069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:27.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.415+0000 7efd61600640 1 -- 172.21.15.47:0/2153034637 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd50084f10 con 0x7efd640ce400 2024-02-19T15:27:27.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.415+0000 7efd69600640 1 --2- 172.21.15.47:0/2153034637 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd44066e40 0x7efd44069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:27.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.415+0000 7efd69600640 1 --2- 172.21.15.47:0/2153034637 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd44066e40 0x7efd44069300 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7efd580af000 tx=0x7efd580a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:27.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.419+0000 7efd61600640 1 -- 172.21.15.47:0/2153034637 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7efd50051d00 con 0x7efd640ce400 2024-02-19T15:27:27.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.643+0000 7efd6c000640 1 -- 172.21.15.47:0/2153034637 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7efd64075360 con 0x7efd640ce400 2024-02-19T15:27:27.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.643+0000 7efd61600640 1 -- 172.21.15.47:0/2153034637 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1517 (secure 0 0 0) 0x7efd5001e600 con 0x7efd640ce400 2024-02-19T15:27:27.647 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:27.647 INFO:teuthology.orchestra.run.smithi047.stdout:{"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":347,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":108,"num_osds":8,"num_up_osds":8,"osd_up_since":1708356111,"num_in_osds":8,"osd_in_since":1708356102,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":59},{"state_name":"active+clean","count":38}],"num_pgs":97,"num_pools":3,"num_objects":3,"data_bytes":590438,"bytes_used":313962496,"bytes_avail":772780150784,"bytes_total":773094113280,"inactive_pgs_ratio":0.60824739933013916,"write_bytes_sec":196,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":75,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"c","status":"up:active","gid":7635}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-02-19T15:27:09.218830+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-02-19T15:27:27.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.643+0000 7efd6c000640 1 -- 172.21.15.47:0/2153034637 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd44066e40 msgr2=0x7efd44069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:27.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.643+0000 7efd6c000640 1 --2- 172.21.15.47:0/2153034637 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd44066e40 0x7efd44069300 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7efd580af000 tx=0x7efd580a7000 comp rx=0 tx=0).stop 2024-02-19T15:27:27.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.643+0000 7efd6c000640 1 -- 172.21.15.47:0/2153034637 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd640ce400 msgr2=0x7efd64153570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:27.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.643+0000 7efd6c000640 1 --2- 172.21.15.47:0/2153034637 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd640ce400 0x7efd64153570 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7efd5000c7c0 tx=0x7efd5000cc80 comp rx=0 tx=0).stop 2024-02-19T15:27:27.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.643+0000 7efd6c000640 1 -- 172.21.15.47:0/2153034637 shutdown_connections 2024-02-19T15:27:27.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.643+0000 7efd6c000640 1 --2- 172.21.15.47:0/2153034637 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd44066e40 0x7efd44069300 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:27.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.643+0000 7efd6c000640 1 --2- 172.21.15.47:0/2153034637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd6415dfa0 0x7efd6415c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:27.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.643+0000 7efd6c000640 1 --2- 172.21.15.47:0/2153034637 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd640ce400 0x7efd64153570 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:27.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.643+0000 7efd6c000640 1 --2- 172.21.15.47:0/2153034637 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd640b4240 0x7efd641552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:27.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.643+0000 7efd6c000640 1 -- 172.21.15.47:0/2153034637 >> 172.21.15.47:0/2153034637 conn(0x7efd640b3690 msgr2=0x7efd640cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:27.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.643+0000 7efd6c000640 1 -- 172.21.15.47:0/2153034637 shutdown_connections 2024-02-19T15:27:27.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.643+0000 7efd6c000640 1 -- 172.21.15.47:0/2153034637 wait complete. 2024-02-19T15:27:27.668 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-02-19T15:27:27.669 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:27:27.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.827+0000 7f01fde00640 1 Processor -- start 2024-02-19T15:27:27.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.827+0000 7f01fde00640 1 -- start start 2024-02-19T15:27:27.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.831+0000 7f01fde00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f01f80cb520 0x7f01f80cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:27.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.831+0000 7f01fde00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f01f80c6780 0x7f01f80c6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:27.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.831+0000 7f01fde00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f80ad840 con 0x7f01f806aeb0 2024-02-19T15:27:27.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.831+0000 7f01fde00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f80c62f0 con 0x7f01f80cb520 2024-02-19T15:27:27.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.831+0000 7f01fde00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f80cc810 con 0x7f01f80c6780 2024-02-19T15:27:27.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.831+0000 7f01f6000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f01f80cb520 0x7f01f80cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:27.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.831+0000 7f01f6a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f01f806aeb0 0x7f01f806b290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39828/0 (socket says 172.21.15.47:39828) 2024-02-19T15:27:27.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.831+0000 7f01f6a00640 1 -- 172.21.15.47:0/976399560 learned_addr learned my addr 172.21.15.47:0/976399560 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:27.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.831+0000 7f01f7400640 1 --2- 172.21.15.47:0/976399560 >> v2:172.21.15.116:3300/0 conn(0x7f01f80c6780 0x7f01f80c6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:27.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.831+0000 7f01f5600640 1 -- 172.21.15.47:0/976399560 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305921047 0 0) 0x7f01f80ad840 con 0x7f01f806aeb0 2024-02-19T15:27:27.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.831+0000 7f01f5600640 1 -- 172.21.15.47:0/976399560 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01dc003530 con 0x7f01f806aeb0 2024-02-19T15:27:27.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.831+0000 7f01f6000640 1 -- 172.21.15.47:0/976399560 >> v2:172.21.15.116:3300/0 conn(0x7f01f80c6780 msgr2=0x7f01f80c6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:27.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.831+0000 7f01f6000640 1 --2- 172.21.15.47:0/976399560 >> v2:172.21.15.116:3300/0 conn(0x7f01f80c6780 0x7f01f80c6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:27.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.831+0000 7f01f6000640 1 -- 172.21.15.47:0/976399560 >> v1:172.21.15.47:6789/0 conn(0x7f01f806aeb0 legacy=0x7f01f806b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:27.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.831+0000 7f01f6000640 1 -- 172.21.15.47:0/976399560 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f80cc250 con 0x7f01f80cb520 2024-02-19T15:27:27.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.835+0000 7f01f6000640 1 --2- 172.21.15.47:0/976399560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01f80cb520 0x7f01f80cb9a0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f01f806d430 tx=0x7f01e401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83c261a40e29781a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:27.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.835+0000 7f01f5600640 1 -- 172.21.15.47:0/976399560 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e40ab650 con 0x7f01f80cb520 2024-02-19T15:27:27.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.835+0000 7f01f5600640 1 -- 172.21.15.47:0/976399560 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e40ab7b0 con 0x7f01f80cb520 2024-02-19T15:27:27.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.835+0000 7f01f5600640 1 -- 172.21.15.47:0/976399560 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e40abb40 con 0x7f01f80cb520 2024-02-19T15:27:27.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.835+0000 7f01fde00640 1 -- 172.21.15.47:0/976399560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01f80cb520 msgr2=0x7f01f80cb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:27.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.835+0000 7f01fde00640 1 --2- 172.21.15.47:0/976399560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01f80cb520 0x7f01f80cb9a0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f01f806d430 tx=0x7f01e401c450 comp rx=0 tx=0).stop 2024-02-19T15:27:27.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.835+0000 7f01fde00640 1 -- 172.21.15.47:0/976399560 shutdown_connections 2024-02-19T15:27:27.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.835+0000 7f01fde00640 1 --2- 172.21.15.47:0/976399560 >> v2:172.21.15.116:3300/0 conn(0x7f01f80c6780 0x7f01f80c6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:27.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.835+0000 7f01fde00640 1 --2- 172.21.15.47:0/976399560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01f80cb520 0x7f01f80cb9a0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:27.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.835+0000 7f01fde00640 1 -- 172.21.15.47:0/976399560 >> 172.21.15.47:0/976399560 conn(0x7f01f80d0ad0 msgr2=0x7f01f80d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:27.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.835+0000 7f01fde00640 1 -- 172.21.15.47:0/976399560 shutdown_connections 2024-02-19T15:27:27.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.835+0000 7f01fde00640 1 -- 172.21.15.47:0/976399560 wait complete. 2024-02-19T15:27:27.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01fde00640 1 Processor -- start 2024-02-19T15:27:27.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01fde00640 1 -- start start 2024-02-19T15:27:27.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01fde00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01f806aeb0 0x7f01f815ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:27.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01fde00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01f80c6780 0x7f01f80718e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01fde00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01f80cb520 0x7f01f8071e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01fde00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f815f8b0 con 0x7f01f806aeb0 2024-02-19T15:27:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01fde00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f80b6b80 con 0x7f01f80c6780 2024-02-19T15:27:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01fde00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f80b62e0 con 0x7f01f80cb520 2024-02-19T15:27:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01f6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01f806aeb0 0x7f01f815ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01f6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01f806aeb0 0x7f01f815ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55360/0 (socket says 172.21.15.47:55360) 2024-02-19T15:27:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01f6a00640 1 -- 172.21.15.47:0/3465347198 learned_addr learned my addr 172.21.15.47:0/3465347198 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01f6000640 1 --2- 172.21.15.47:0/3465347198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01f80c6780 0x7f01f80718e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01f7400640 1 --2- 172.21.15.47:0/3465347198 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01f80cb520 0x7f01f8071e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01f6a00640 1 -- 172.21.15.47:0/3465347198 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01f80cb520 msgr2=0x7f01f8071e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01f6a00640 1 --2- 172.21.15.47:0/3465347198 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01f80cb520 0x7f01f8071e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01f6a00640 1 -- 172.21.15.47:0/3465347198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01f80c6780 msgr2=0x7f01f80718e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01f6a00640 1 --2- 172.21.15.47:0/3465347198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01f80c6780 0x7f01f80718e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01f6a00640 1 -- 172.21.15.47:0/3465347198 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e4009420 con 0x7f01f806aeb0 2024-02-19T15:27:27.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01f6000640 1 --2- 172.21.15.47:0/3465347198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01f80c6780 0x7f01f80718e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:27:27.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01f6a00640 1 --2- 172.21.15.47:0/3465347198 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01f806aeb0 0x7f01f815ae10 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f01e800cb40 tx=0x7f01e800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:27.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01ef400640 1 -- 172.21.15.47:0/3465347198 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e800aca0 con 0x7f01f806aeb0 2024-02-19T15:27:27.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01ef400640 1 -- 172.21.15.47:0/3465347198 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e800ae00 con 0x7f01f806aeb0 2024-02-19T15:27:27.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01ef400640 1 -- 172.21.15.47:0/3465347198 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e8012620 con 0x7f01f806aeb0 2024-02-19T15:27:27.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01fde00640 1 -- 172.21.15.47:0/3465347198 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f80b65c0 con 0x7f01f806aeb0 2024-02-19T15:27:27.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.839+0000 7f01fde00640 1 -- 172.21.15.47:0/3465347198 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f80b5f90 con 0x7f01f806aeb0 2024-02-19T15:27:27.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.843+0000 7f01fde00640 1 -- 172.21.15.47:0/3465347198 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f8075360 con 0x7f01f806aeb0 2024-02-19T15:27:27.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.843+0000 7f01ef400640 1 -- 172.21.15.47:0/3465347198 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f01e8007070 con 0x7f01f806aeb0 2024-02-19T15:27:27.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.843+0000 7f01ef400640 1 --2- 172.21.15.47:0/3465347198 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f01d8066e40 0x7f01d8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:27.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.843+0000 7f01ef400640 1 -- 172.21.15.47:0/3465347198 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01e8084b50 con 0x7f01f806aeb0 2024-02-19T15:27:27.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.843+0000 7f01f6000640 1 --2- 172.21.15.47:0/3465347198 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f01d8066e40 0x7f01d8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:27.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.843+0000 7f01f6000640 1 --2- 172.21.15.47:0/3465347198 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f01d8066e40 0x7f01d8069300 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f01e40af000 tx=0x7f01e40b6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:27.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:27.847+0000 7f01ef400640 1 -- 172.21.15.47:0/3465347198 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f01e80518a0 con 0x7f01f806aeb0 2024-02-19T15:27:28.005 INFO:tasks.ceph.mds.c.smithi047.stderr:2024-02-19T15:27:27.999+0000 7f7ff6200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T15:27:28.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:28.007+0000 7f01fde00640 1 -- 172.21.15.47:0/3465347198 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f01f80b0370 con 0x7f01d8066e40 2024-02-19T15:27:28.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:28.015+0000 7f01ef400640 1 -- 172.21.15.47:0/3465347198 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289012 (secure 0 0 0) 0x7f01f80b0370 con 0x7f01d8066e40 2024-02-19T15:27:28.023 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:28.024 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:27:28.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:28.019+0000 7f01fde00640 1 -- 172.21.15.47:0/3465347198 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f01d8066e40 msgr2=0x7f01d8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:28.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:28.019+0000 7f01fde00640 1 --2- 172.21.15.47:0/3465347198 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f01d8066e40 0x7f01d8069300 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f01e40af000 tx=0x7f01e40b6040 comp rx=0 tx=0).stop 2024-02-19T15:27:28.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:28.019+0000 7f01fde00640 1 -- 172.21.15.47:0/3465347198 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01f806aeb0 msgr2=0x7f01f815ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:28.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:28.019+0000 7f01fde00640 1 --2- 172.21.15.47:0/3465347198 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01f806aeb0 0x7f01f815ae10 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f01e800cb40 tx=0x7f01e800a060 comp rx=0 tx=0).stop 2024-02-19T15:27:28.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:28.019+0000 7f01fde00640 1 -- 172.21.15.47:0/3465347198 shutdown_connections 2024-02-19T15:27:28.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:28.019+0000 7f01fde00640 1 --2- 172.21.15.47:0/3465347198 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f01d8066e40 0x7f01d8069300 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:28.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:28.019+0000 7f01fde00640 1 --2- 172.21.15.47:0/3465347198 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01f80cb520 0x7f01f8071e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:28.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:28.019+0000 7f01fde00640 1 --2- 172.21.15.47:0/3465347198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01f80c6780 0x7f01f80718e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:28.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:28.019+0000 7f01fde00640 1 --2- 172.21.15.47:0/3465347198 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01f806aeb0 0x7f01f815ae10 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:28.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:28.019+0000 7f01fde00640 1 -- 172.21.15.47:0/3465347198 >> 172.21.15.47:0/3465347198 conn(0x7f01f80d0ad0 msgr2=0x7f01f80ce660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:28.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:28.019+0000 7f01fde00640 1 -- 172.21.15.47:0/3465347198 shutdown_connections 2024-02-19T15:27:28.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:28.019+0000 7f01fde00640 1 -- 172.21.15.47:0/3465347198 wait complete. 2024-02-19T15:27:28.043 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-02-19T15:27:27.222385+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":306888,"kb_used_data":11660,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754667832,"statfs":{"total":773094113280,"available":772779859968,"internally_reserved":0,"allocated":11939840,"data_stored":5787219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":302173746},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.213062"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651409+0000","last_change":"2024-02-19T15:27:22.044103+0000","last_active":"2024-02-19T15:27:23.651409+0000","last_peered":"2024-02-19T15:27:23.651409+0000","last_clean":"2024-02-19T15:27:23.651409+0000","last_became_active":"2024-02-19T15:27:22.043723+0000","last_became_peered":"2024-02-19T15:27:22.043723+0000","last_unstale":"2024-02-19T15:27:23.651409+0000","last_undegraded":"2024-02-19T15:27:23.651409+0000","last_fullsized":"2024-02-19T15:27:23.651409+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:59:04.067243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045480+0000","last_change":"2024-02-19T15:27:22.060740+0000","last_active":"2024-02-19T15:27:23.045480+0000","last_peered":"2024-02-19T15:27:23.045480+0000","last_clean":"2024-02-19T15:27:23.045480+0000","last_became_active":"2024-02-19T15:27:22.060471+0000","last_became_peered":"2024-02-19T15:27:22.060471+0000","last_unstale":"2024-02-19T15:27:23.045480+0000","last_undegraded":"2024-02-19T15:27:23.045480+0000","last_fullsized":"2024-02-19T15:27:23.045480+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:39:12.144358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650032+0000","last_change":"2024-02-19T15:27:22.048031+0000","last_active":"2024-02-19T15:27:23.650032+0000","last_peered":"2024-02-19T15:27:23.650032+0000","last_clean":"2024-02-19T15:27:23.650032+0000","last_became_active":"2024-02-19T15:27:22.047724+0000","last_became_peered":"2024-02-19T15:27:22.047724+0000","last_unstale":"2024-02-19T15:27:23.650032+0000","last_undegraded":"2024-02-19T15:27:23.650032+0000","last_fullsized":"2024-02-19T15:27:23.650032+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:25:25.436463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650600+0000","last_change":"2024-02-19T15:27:22.052619+0000","last_active":"2024-02-19T15:27:23.650600+0000","last_peered":"2024-02-19T15:27:23.650600+0000","last_clean":"2024-02-19T15:27:23.650600+0000","last_became_active":"2024-02-19T15:27:22.052289+0000","last_became_peered":"2024-02-19T15:27:22.052289+0000","last_unstale":"2024-02-19T15:27:23.650600+0000","last_undegraded":"2024-02-19T15:27:23.650600+0000","last_fullsized":"2024-02-19T15:27:23.650600+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:53:18.030121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650775+0000","last_change":"2024-02-19T15:27:22.053770+0000","last_active":"2024-02-19T15:27:23.650775+0000","last_peered":"2024-02-19T15:27:23.650775+0000","last_clean":"2024-02-19T15:27:23.650775+0000","last_became_active":"2024-02-19T15:27:22.053497+0000","last_became_peered":"2024-02-19T15:27:22.053497+0000","last_unstale":"2024-02-19T15:27:23.650775+0000","last_undegraded":"2024-02-19T15:27:23.650775+0000","last_fullsized":"2024-02-19T15:27:23.650775+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:28.666610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045744+0000","last_change":"2024-02-19T15:27:22.063229+0000","last_active":"2024-02-19T15:27:23.045744+0000","last_peered":"2024-02-19T15:27:23.045744+0000","last_clean":"2024-02-19T15:27:23.045744+0000","last_became_active":"2024-02-19T15:27:22.062899+0000","last_became_peered":"2024-02-19T15:27:22.062899+0000","last_unstale":"2024-02-19T15:27:23.045744+0000","last_undegraded":"2024-02-19T15:27:23.045744+0000","last_fullsized":"2024-02-19T15:27:23.045744+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:59:15.699509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649795+0000","last_change":"2024-02-19T15:27:22.040230+0000","last_active":"2024-02-19T15:27:23.649795+0000","last_peered":"2024-02-19T15:27:23.649795+0000","last_clean":"2024-02-19T15:27:23.649795+0000","last_became_active":"2024-02-19T15:27:22.039959+0000","last_became_peered":"2024-02-19T15:27:22.039959+0000","last_unstale":"2024-02-19T15:27:23.649795+0000","last_undegraded":"2024-02-19T15:27:23.649795+0000","last_fullsized":"2024-02-19T15:27:23.649795+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:45:36.102577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649861+0000","last_change":"2024-02-19T15:27:22.060882+0000","last_active":"2024-02-19T15:27:23.649861+0000","last_peered":"2024-02-19T15:27:23.649861+0000","last_clean":"2024-02-19T15:27:23.649861+0000","last_became_active":"2024-02-19T15:27:22.060570+0000","last_became_peered":"2024-02-19T15:27:22.060570+0000","last_unstale":"2024-02-19T15:27:23.649861+0000","last_undegraded":"2024-02-19T15:27:23.649861+0000","last_fullsized":"2024-02-19T15:27:23.649861+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:34.686973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651562+0000","last_change":"2024-02-19T15:27:22.050561+0000","last_active":"2024-02-19T15:27:23.651562+0000","last_peered":"2024-02-19T15:27:23.651562+0000","last_clean":"2024-02-19T15:27:23.651562+0000","last_became_active":"2024-02-19T15:27:22.049431+0000","last_became_peered":"2024-02-19T15:27:22.049431+0000","last_unstale":"2024-02-19T15:27:23.651562+0000","last_undegraded":"2024-02-19T15:27:23.651562+0000","last_fullsized":"2024-02-19T15:27:23.651562+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:58:39.839056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651522+0000","last_change":"2024-02-19T15:27:22.057693+0000","last_active":"2024-02-19T15:27:23.651522+0000","last_peered":"2024-02-19T15:27:23.651522+0000","last_clean":"2024-02-19T15:27:23.651522+0000","last_became_active":"2024-02-19T15:27:22.057412+0000","last_became_peered":"2024-02-19T15:27:22.057412+0000","last_unstale":"2024-02-19T15:27:23.651522+0000","last_undegraded":"2024-02-19T15:27:23.651522+0000","last_fullsized":"2024-02-19T15:27:23.651522+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:53:13.793384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045193+0000","last_change":"2024-02-19T15:27:22.045843+0000","last_active":"2024-02-19T15:27:23.045193+0000","last_peered":"2024-02-19T15:27:23.045193+0000","last_clean":"2024-02-19T15:27:23.045193+0000","last_became_active":"2024-02-19T15:27:22.045550+0000","last_became_peered":"2024-02-19T15:27:22.045550+0000","last_unstale":"2024-02-19T15:27:23.045193+0000","last_undegraded":"2024-02-19T15:27:23.045193+0000","last_fullsized":"2024-02-19T15:27:23.045193+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:27.552780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650768+0000","last_change":"2024-02-19T15:27:22.051357+0000","last_active":"2024-02-19T15:27:23.650768+0000","last_peered":"2024-02-19T15:27:23.650768+0000","last_clean":"2024-02-19T15:27:23.650768+0000","last_became_active":"2024-02-19T15:27:22.051073+0000","last_became_peered":"2024-02-19T15:27:22.051073+0000","last_unstale":"2024-02-19T15:27:23.650768+0000","last_undegraded":"2024-02-19T15:27:23.650768+0000","last_fullsized":"2024-02-19T15:27:23.650768+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:16:39.463746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650232+0000","last_change":"2024-02-19T15:27:22.054976+0000","last_active":"2024-02-19T15:27:23.650232+0000","last_peered":"2024-02-19T15:27:23.650232+0000","last_clean":"2024-02-19T15:27:23.650232+0000","last_became_active":"2024-02-19T15:27:22.054427+0000","last_became_peered":"2024-02-19T15:27:22.054427+0000","last_unstale":"2024-02-19T15:27:23.650232+0000","last_undegraded":"2024-02-19T15:27:23.650232+0000","last_fullsized":"2024-02-19T15:27:23.650232+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:58:10.943380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649945+0000","last_change":"2024-02-19T15:27:22.046799+0000","last_active":"2024-02-19T15:27:23.649945+0000","last_peered":"2024-02-19T15:27:23.649945+0000","last_clean":"2024-02-19T15:27:23.649945+0000","last_became_active":"2024-02-19T15:27:22.046301+0000","last_became_peered":"2024-02-19T15:27:22.046301+0000","last_unstale":"2024-02-19T15:27:23.649945+0000","last_undegraded":"2024-02-19T15:27:23.649945+0000","last_fullsized":"2024-02-19T15:27:23.649945+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:58:18.347673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045487+0000","last_change":"2024-02-19T15:27:22.060116+0000","last_active":"2024-02-19T15:27:23.045487+0000","last_peered":"2024-02-19T15:27:23.045487+0000","last_clean":"2024-02-19T15:27:23.045487+0000","last_became_active":"2024-02-19T15:27:22.059885+0000","last_became_peered":"2024-02-19T15:27:22.059885+0000","last_unstale":"2024-02-19T15:27:23.045487+0000","last_undegraded":"2024-02-19T15:27:23.045487+0000","last_fullsized":"2024-02-19T15:27:23.045487+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:33:49.374645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650143+0000","last_change":"2024-02-19T15:27:22.057325+0000","last_active":"2024-02-19T15:27:23.650143+0000","last_peered":"2024-02-19T15:27:23.650143+0000","last_clean":"2024-02-19T15:27:23.650143+0000","last_became_active":"2024-02-19T15:27:22.050946+0000","last_became_peered":"2024-02-19T15:27:22.050946+0000","last_unstale":"2024-02-19T15:27:23.650143+0000","last_undegraded":"2024-02-19T15:27:23.650143+0000","last_fullsized":"2024-02-19T15:27:23.650143+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:03:35.602958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045349+0000","last_change":"2024-02-19T15:27:22.041551+0000","last_active":"2024-02-19T15:27:23.045349+0000","last_peered":"2024-02-19T15:27:23.045349+0000","last_clean":"2024-02-19T15:27:23.045349+0000","last_became_active":"2024-02-19T15:27:22.040360+0000","last_became_peered":"2024-02-19T15:27:22.040360+0000","last_unstale":"2024-02-19T15:27:23.045349+0000","last_undegraded":"2024-02-19T15:27:23.045349+0000","last_fullsized":"2024-02-19T15:27:23.045349+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:59:17.128798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.042266+0000","last_change":"2024-02-19T15:27:22.036471+0000","last_active":"2024-02-19T15:27:22.042266+0000","last_peered":"2024-02-19T15:27:22.042266+0000","last_clean":"2024-02-19T15:27:22.042266+0000","last_became_active":"2024-02-19T15:27:22.036201+0000","last_became_peered":"2024-02-19T15:27:22.036201+0000","last_unstale":"2024-02-19T15:27:22.042266+0000","last_undegraded":"2024-02-19T15:27:22.042266+0000","last_fullsized":"2024-02-19T15:27:22.042266+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:28:59.516714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650241+0000","last_change":"2024-02-19T15:27:22.055260+0000","last_active":"2024-02-19T15:27:23.650241+0000","last_peered":"2024-02-19T15:27:23.650241+0000","last_clean":"2024-02-19T15:27:23.650241+0000","last_became_active":"2024-02-19T15:27:22.054795+0000","last_became_peered":"2024-02-19T15:27:22.054795+0000","last_unstale":"2024-02-19T15:27:23.650241+0000","last_undegraded":"2024-02-19T15:27:23.650241+0000","last_fullsized":"2024-02-19T15:27:23.650241+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:24:06.367087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.047110+0000","last_change":"2024-02-19T15:27:22.048745+0000","last_active":"2024-02-19T15:27:23.047110+0000","last_peered":"2024-02-19T15:27:23.047110+0000","last_clean":"2024-02-19T15:27:23.047110+0000","last_became_active":"2024-02-19T15:27:22.043577+0000","last_became_peered":"2024-02-19T15:27:22.043577+0000","last_unstale":"2024-02-19T15:27:23.047110+0000","last_undegraded":"2024-02-19T15:27:23.047110+0000","last_fullsized":"2024-02-19T15:27:23.047110+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:58.251712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045286+0000","last_change":"2024-02-19T15:27:22.048453+0000","last_active":"2024-02-19T15:27:23.045286+0000","last_peered":"2024-02-19T15:27:23.045286+0000","last_clean":"2024-02-19T15:27:23.045286+0000","last_became_active":"2024-02-19T15:27:22.046650+0000","last_became_peered":"2024-02-19T15:27:22.046650+0000","last_unstale":"2024-02-19T15:27:23.045286+0000","last_undegraded":"2024-02-19T15:27:23.045286+0000","last_fullsized":"2024-02-19T15:27:23.045286+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:15:02.154641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650028+0000","last_change":"2024-02-19T15:27:22.048023+0000","last_active":"2024-02-19T15:27:23.650028+0000","last_peered":"2024-02-19T15:27:23.650028+0000","last_clean":"2024-02-19T15:27:23.650028+0000","last_became_active":"2024-02-19T15:27:22.047669+0000","last_became_peered":"2024-02-19T15:27:22.047669+0000","last_unstale":"2024-02-19T15:27:23.650028+0000","last_undegraded":"2024-02-19T15:27:23.650028+0000","last_fullsized":"2024-02-19T15:27:23.650028+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:25:25.436463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045036+0000","last_change":"2024-02-19T15:27:22.052131+0000","last_active":"2024-02-19T15:27:23.045036+0000","last_peered":"2024-02-19T15:27:23.045036+0000","last_clean":"2024-02-19T15:27:23.045036+0000","last_became_active":"2024-02-19T15:27:22.051851+0000","last_became_peered":"2024-02-19T15:27:22.051851+0000","last_unstale":"2024-02-19T15:27:23.045036+0000","last_undegraded":"2024-02-19T15:27:23.045036+0000","last_fullsized":"2024-02-19T15:27:23.045036+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:27:08.938061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650176+0000","last_change":"2024-02-19T15:27:22.056052+0000","last_active":"2024-02-19T15:27:23.650176+0000","last_peered":"2024-02-19T15:27:23.650176+0000","last_clean":"2024-02-19T15:27:23.650176+0000","last_became_active":"2024-02-19T15:27:22.054103+0000","last_became_peered":"2024-02-19T15:27:22.054103+0000","last_unstale":"2024-02-19T15:27:23.650176+0000","last_undegraded":"2024-02-19T15:27:23.650176+0000","last_fullsized":"2024-02-19T15:27:23.650176+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:58:10.943380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045393+0000","last_change":"2024-02-19T15:27:22.041777+0000","last_active":"2024-02-19T15:27:23.045393+0000","last_peered":"2024-02-19T15:27:23.045393+0000","last_clean":"2024-02-19T15:27:23.045393+0000","last_became_active":"2024-02-19T15:27:22.040747+0000","last_became_peered":"2024-02-19T15:27:22.040747+0000","last_unstale":"2024-02-19T15:27:23.045393+0000","last_undegraded":"2024-02-19T15:27:23.045393+0000","last_fullsized":"2024-02-19T15:27:23.045393+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:08:48.016401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651670+0000","last_change":"2024-02-19T15:27:22.045024+0000","last_active":"2024-02-19T15:27:23.651670+0000","last_peered":"2024-02-19T15:27:23.651670+0000","last_clean":"2024-02-19T15:27:23.651670+0000","last_became_active":"2024-02-19T15:27:22.044758+0000","last_became_peered":"2024-02-19T15:27:22.044758+0000","last_unstale":"2024-02-19T15:27:23.651670+0000","last_undegraded":"2024-02-19T15:27:23.651670+0000","last_fullsized":"2024-02-19T15:27:23.651670+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:52:21.606372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.046021+0000","last_change":"2024-02-19T15:27:22.062389+0000","last_active":"2024-02-19T15:27:23.046021+0000","last_peered":"2024-02-19T15:27:23.046021+0000","last_clean":"2024-02-19T15:27:23.046021+0000","last_became_active":"2024-02-19T15:27:22.058599+0000","last_became_peered":"2024-02-19T15:27:22.058599+0000","last_unstale":"2024-02-19T15:27:23.046021+0000","last_undegraded":"2024-02-19T15:27:23.046021+0000","last_fullsized":"2024-02-19T15:27:23.046021+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:27:13.600503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649959+0000","last_change":"2024-02-19T15:27:22.060877+0000","last_active":"2024-02-19T15:27:23.649959+0000","last_peered":"2024-02-19T15:27:23.649959+0000","last_clean":"2024-02-19T15:27:23.649959+0000","last_became_active":"2024-02-19T15:27:22.060529+0000","last_became_peered":"2024-02-19T15:27:22.060529+0000","last_unstale":"2024-02-19T15:27:23.649959+0000","last_undegraded":"2024-02-19T15:27:23.649959+0000","last_fullsized":"2024-02-19T15:27:23.649959+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:58:18.347673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045474+0000","last_change":"2024-02-19T15:27:22.043723+0000","last_active":"2024-02-19T15:27:23.045474+0000","last_peered":"2024-02-19T15:27:23.045474+0000","last_clean":"2024-02-19T15:27:23.045474+0000","last_became_active":"2024-02-19T15:27:22.043463+0000","last_became_peered":"2024-02-19T15:27:22.043463+0000","last_unstale":"2024-02-19T15:27:23.045474+0000","last_undegraded":"2024-02-19T15:27:23.045474+0000","last_fullsized":"2024-02-19T15:27:23.045474+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:52.297989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650158+0000","last_change":"2024-02-19T15:27:22.045324+0000","last_active":"2024-02-19T15:27:23.650158+0000","last_peered":"2024-02-19T15:27:23.650158+0000","last_clean":"2024-02-19T15:27:23.650158+0000","last_became_active":"2024-02-19T15:27:22.044981+0000","last_became_peered":"2024-02-19T15:27:22.044981+0000","last_unstale":"2024-02-19T15:27:23.650158+0000","last_undegraded":"2024-02-19T15:27:23.650158+0000","last_fullsized":"2024-02-19T15:27:23.650158+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:27:13.600503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649846+0000","last_change":"2024-02-19T15:27:22.045853+0000","last_active":"2024-02-19T15:27:23.649846+0000","last_peered":"2024-02-19T15:27:23.649846+0000","last_clean":"2024-02-19T15:27:23.649846+0000","last_became_active":"2024-02-19T15:27:22.045474+0000","last_became_peered":"2024-02-19T15:27:22.045474+0000","last_unstale":"2024-02-19T15:27:23.649846+0000","last_undegraded":"2024-02-19T15:27:23.649846+0000","last_fullsized":"2024-02-19T15:27:23.649846+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:21:04.976920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045408+0000","last_change":"2024-02-19T15:27:22.055671+0000","last_active":"2024-02-19T15:27:23.045408+0000","last_peered":"2024-02-19T15:27:23.045408+0000","last_clean":"2024-02-19T15:27:23.045408+0000","last_became_active":"2024-02-19T15:27:22.055359+0000","last_became_peered":"2024-02-19T15:27:22.055359+0000","last_unstale":"2024-02-19T15:27:23.045408+0000","last_undegraded":"2024-02-19T15:27:23.045408+0000","last_fullsized":"2024-02-19T15:27:23.045408+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:57:40.939085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.046102+0000","last_change":"2024-02-19T15:27:22.059901+0000","last_active":"2024-02-19T15:27:23.046102+0000","last_peered":"2024-02-19T15:27:23.046102+0000","last_clean":"2024-02-19T15:27:23.046102+0000","last_became_active":"2024-02-19T15:27:22.059581+0000","last_became_peered":"2024-02-19T15:27:22.059581+0000","last_unstale":"2024-02-19T15:27:23.046102+0000","last_undegraded":"2024-02-19T15:27:23.046102+0000","last_fullsized":"2024-02-19T15:27:23.046102+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:40:09.571306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"107'4","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650737+0000","last_change":"2024-02-19T15:27:21.030063+0000","last_active":"2024-02-19T15:27:23.650737+0000","last_peered":"2024-02-19T15:27:23.650737+0000","last_clean":"2024-02-19T15:27:23.650737+0000","last_became_active":"2024-02-19T15:27:21.029780+0000","last_became_peered":"2024-02-19T15:27:21.029780+0000","last_unstale":"2024-02-19T15:27:23.650737+0000","last_undegraded":"2024-02-19T15:27:23.650737+0000","last_fullsized":"2024-02-19T15:27:23.650737+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:44:41.044290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.054040+0000","last_change":"2024-02-19T15:27:22.048726+0000","last_active":"2024-02-19T15:27:22.054040+0000","last_peered":"2024-02-19T15:27:22.054040+0000","last_clean":"2024-02-19T15:27:22.054040+0000","last_became_active":"2024-02-19T15:27:22.048437+0000","last_became_peered":"2024-02-19T15:27:22.048437+0000","last_unstale":"2024-02-19T15:27:22.054040+0000","last_undegraded":"2024-02-19T15:27:22.054040+0000","last_fullsized":"2024-02-19T15:27:22.054040+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:50:16.869685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650367+0000","last_change":"2024-02-19T15:27:21.027135+0000","last_active":"2024-02-19T15:27:23.650367+0000","last_peered":"2024-02-19T15:27:23.650367+0000","last_clean":"2024-02-19T15:27:23.650367+0000","last_became_active":"2024-02-19T15:27:21.026870+0000","last_became_peered":"2024-02-19T15:27:21.026870+0000","last_unstale":"2024-02-19T15:27:23.650367+0000","last_undegraded":"2024-02-19T15:27:23.650367+0000","last_fullsized":"2024-02-19T15:27:23.650367+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:54.672402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045992+0000","last_change":"2024-02-19T15:27:22.059124+0000","last_active":"2024-02-19T15:27:23.045992+0000","last_peered":"2024-02-19T15:27:23.045992+0000","last_clean":"2024-02-19T15:27:23.045992+0000","last_became_active":"2024-02-19T15:27:22.057067+0000","last_became_peered":"2024-02-19T15:27:22.057067+0000","last_unstale":"2024-02-19T15:27:23.045992+0000","last_undegraded":"2024-02-19T15:27:23.045992+0000","last_fullsized":"2024-02-19T15:27:23.045992+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:45:36.102577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651208+0000","last_change":"2024-02-19T15:27:21.029872+0000","last_active":"2024-02-19T15:27:23.651208+0000","last_peered":"2024-02-19T15:27:23.651208+0000","last_clean":"2024-02-19T15:27:23.651208+0000","last_became_active":"2024-02-19T15:27:21.029590+0000","last_became_peered":"2024-02-19T15:27:21.029590+0000","last_unstale":"2024-02-19T15:27:23.651208+0000","last_undegraded":"2024-02-19T15:27:23.651208+0000","last_fullsized":"2024-02-19T15:27:23.651208+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:25:03.421064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650459+0000","last_change":"2024-02-19T15:27:22.054621+0000","last_active":"2024-02-19T15:27:23.650459+0000","last_peered":"2024-02-19T15:27:23.650459+0000","last_clean":"2024-02-19T15:27:23.650459+0000","last_became_active":"2024-02-19T15:27:22.054395+0000","last_became_peered":"2024-02-19T15:27:22.054395+0000","last_unstale":"2024-02-19T15:27:23.650459+0000","last_undegraded":"2024-02-19T15:27:23.650459+0000","last_fullsized":"2024-02-19T15:27:23.650459+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:43:10.829112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650470+0000","last_change":"2024-02-19T15:27:21.042870+0000","last_active":"2024-02-19T15:27:23.650470+0000","last_peered":"2024-02-19T15:27:23.650470+0000","last_clean":"2024-02-19T15:27:23.650470+0000","last_became_active":"2024-02-19T15:27:21.041700+0000","last_became_peered":"2024-02-19T15:27:21.041700+0000","last_unstale":"2024-02-19T15:27:23.650470+0000","last_undegraded":"2024-02-19T15:27:23.650470+0000","last_fullsized":"2024-02-19T15:27:23.650470+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:58:39.707731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650691+0000","last_change":"2024-02-19T15:27:22.043230+0000","last_active":"2024-02-19T15:27:23.650691+0000","last_peered":"2024-02-19T15:27:23.650691+0000","last_clean":"2024-02-19T15:27:23.650691+0000","last_became_active":"2024-02-19T15:27:22.042919+0000","last_became_peered":"2024-02-19T15:27:22.042919+0000","last_unstale":"2024-02-19T15:27:23.650691+0000","last_undegraded":"2024-02-19T15:27:23.650691+0000","last_fullsized":"2024-02-19T15:27:23.650691+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:22:19.119378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649731+0000","last_change":"2024-02-19T15:27:21.040738+0000","last_active":"2024-02-19T15:27:23.649731+0000","last_peered":"2024-02-19T15:27:23.649731+0000","last_clean":"2024-02-19T15:27:23.649731+0000","last_became_active":"2024-02-19T15:27:21.040492+0000","last_became_peered":"2024-02-19T15:27:21.040492+0000","last_unstale":"2024-02-19T15:27:23.649731+0000","last_undegraded":"2024-02-19T15:27:23.649731+0000","last_fullsized":"2024-02-19T15:27:23.649731+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:31:23.769771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045610+0000","last_change":"2024-02-19T15:27:22.066931+0000","last_active":"2024-02-19T15:27:23.045610+0000","last_peered":"2024-02-19T15:27:23.045610+0000","last_clean":"2024-02-19T15:27:23.045610+0000","last_became_active":"2024-02-19T15:27:22.066614+0000","last_became_peered":"2024-02-19T15:27:22.066614+0000","last_unstale":"2024-02-19T15:27:23.045610+0000","last_undegraded":"2024-02-19T15:27:23.045610+0000","last_fullsized":"2024-02-19T15:27:23.045610+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:55.678215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650275+0000","last_change":"2024-02-19T15:27:21.037989+0000","last_active":"2024-02-19T15:27:23.650275+0000","last_peered":"2024-02-19T15:27:23.650275+0000","last_clean":"2024-02-19T15:27:23.650275+0000","last_became_active":"2024-02-19T15:27:21.037527+0000","last_became_peered":"2024-02-19T15:27:21.037527+0000","last_unstale":"2024-02-19T15:27:23.650275+0000","last_undegraded":"2024-02-19T15:27:23.650275+0000","last_fullsized":"2024-02-19T15:27:23.650275+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:08:47.010588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650103+0000","last_change":"2024-02-19T15:27:22.065841+0000","last_active":"2024-02-19T15:27:23.650103+0000","last_peered":"2024-02-19T15:27:23.650103+0000","last_clean":"2024-02-19T15:27:23.650103+0000","last_became_active":"2024-02-19T15:27:22.065572+0000","last_became_peered":"2024-02-19T15:27:22.065572+0000","last_unstale":"2024-02-19T15:27:23.650103+0000","last_undegraded":"2024-02-19T15:27:23.650103+0000","last_fullsized":"2024-02-19T15:27:23.650103+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:51:18.750175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.044991+0000","last_change":"2024-02-19T15:27:21.031349+0000","last_active":"2024-02-19T15:27:23.044991+0000","last_peered":"2024-02-19T15:27:23.044991+0000","last_clean":"2024-02-19T15:27:23.044991+0000","last_became_active":"2024-02-19T15:27:21.031075+0000","last_became_peered":"2024-02-19T15:27:21.031075+0000","last_unstale":"2024-02-19T15:27:23.044991+0000","last_undegraded":"2024-02-19T15:27:23.044991+0000","last_fullsized":"2024-02-19T15:27:23.044991+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:29:03.843739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649714+0000","last_change":"2024-02-19T15:27:22.042301+0000","last_active":"2024-02-19T15:27:23.649714+0000","last_peered":"2024-02-19T15:27:23.649714+0000","last_clean":"2024-02-19T15:27:23.649714+0000","last_became_active":"2024-02-19T15:27:22.042015+0000","last_became_peered":"2024-02-19T15:27:22.042015+0000","last_unstale":"2024-02-19T15:27:23.649714+0000","last_undegraded":"2024-02-19T15:27:23.649714+0000","last_fullsized":"2024-02-19T15:27:23.649714+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:59:15.699509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650207+0000","last_change":"2024-02-19T15:27:21.025268+0000","last_active":"2024-02-19T15:27:23.650207+0000","last_peered":"2024-02-19T15:27:23.650207+0000","last_clean":"2024-02-19T15:27:23.650207+0000","last_became_active":"2024-02-19T15:27:21.022655+0000","last_became_peered":"2024-02-19T15:27:21.022655+0000","last_unstale":"2024-02-19T15:27:23.650207+0000","last_undegraded":"2024-02-19T15:27:23.650207+0000","last_fullsized":"2024-02-19T15:27:23.650207+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:09.503853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650580+0000","last_change":"2024-02-19T15:27:22.050190+0000","last_active":"2024-02-19T15:27:23.650580+0000","last_peered":"2024-02-19T15:27:23.650580+0000","last_clean":"2024-02-19T15:27:23.650580+0000","last_became_active":"2024-02-19T15:27:22.049824+0000","last_became_peered":"2024-02-19T15:27:22.049824+0000","last_unstale":"2024-02-19T15:27:23.650580+0000","last_undegraded":"2024-02-19T15:27:23.650580+0000","last_fullsized":"2024-02-19T15:27:23.650580+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:40:09.571306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650204+0000","last_change":"2024-02-19T15:27:21.041198+0000","last_active":"2024-02-19T15:27:23.650204+0000","last_peered":"2024-02-19T15:27:23.650204+0000","last_clean":"2024-02-19T15:27:23.650204+0000","last_became_active":"2024-02-19T15:27:21.040821+0000","last_became_peered":"2024-02-19T15:27:21.040821+0000","last_unstale":"2024-02-19T15:27:23.650204+0000","last_undegraded":"2024-02-19T15:27:23.650204+0000","last_fullsized":"2024-02-19T15:27:23.650204+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:15:13.215563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.050040+0000","last_change":"2024-02-19T15:27:22.043233+0000","last_active":"2024-02-19T15:27:22.050040+0000","last_peered":"2024-02-19T15:27:22.050040+0000","last_clean":"2024-02-19T15:27:22.050040+0000","last_became_active":"2024-02-19T15:27:22.042812+0000","last_became_peered":"2024-02-19T15:27:22.042812+0000","last_unstale":"2024-02-19T15:27:22.050040+0000","last_undegraded":"2024-02-19T15:27:22.050040+0000","last_fullsized":"2024-02-19T15:27:22.050040+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:02:10.136155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651530+0000","last_change":"2024-02-19T15:27:21.036912+0000","last_active":"2024-02-19T15:27:23.651530+0000","last_peered":"2024-02-19T15:27:23.651530+0000","last_clean":"2024-02-19T15:27:23.651530+0000","last_became_active":"2024-02-19T15:27:21.036227+0000","last_became_peered":"2024-02-19T15:27:21.036227+0000","last_unstale":"2024-02-19T15:27:23.651530+0000","last_undegraded":"2024-02-19T15:27:23.651530+0000","last_fullsized":"2024-02-19T15:27:23.651530+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:39:11.138544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651228+0000","last_change":"2024-02-19T15:27:22.054682+0000","last_active":"2024-02-19T15:27:23.651228+0000","last_peered":"2024-02-19T15:27:23.651228+0000","last_clean":"2024-02-19T15:27:23.651228+0000","last_became_active":"2024-02-19T15:27:22.054457+0000","last_became_peered":"2024-02-19T15:27:22.054457+0000","last_unstale":"2024-02-19T15:27:23.651228+0000","last_undegraded":"2024-02-19T15:27:23.651228+0000","last_fullsized":"2024-02-19T15:27:23.651228+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:10:38.351264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"107'3","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651245+0000","last_change":"2024-02-19T15:27:21.022250+0000","last_active":"2024-02-19T15:27:23.651245+0000","last_peered":"2024-02-19T15:27:23.651245+0000","last_clean":"2024-02-19T15:27:23.651245+0000","last_became_active":"2024-02-19T15:27:21.021690+0000","last_became_peered":"2024-02-19T15:27:21.021690+0000","last_unstale":"2024-02-19T15:27:23.651245+0000","last_undegraded":"2024-02-19T15:27:23.651245+0000","last_fullsized":"2024-02-19T15:27:23.651245+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:21:10.257063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650229+0000","last_change":"2024-02-19T15:27:22.045625+0000","last_active":"2024-02-19T15:27:23.650229+0000","last_peered":"2024-02-19T15:27:23.650229+0000","last_clean":"2024-02-19T15:27:23.650229+0000","last_became_active":"2024-02-19T15:27:22.045360+0000","last_became_peered":"2024-02-19T15:27:22.045360+0000","last_unstale":"2024-02-19T15:27:23.650229+0000","last_undegraded":"2024-02-19T15:27:23.650229+0000","last_fullsized":"2024-02-19T15:27:23.650229+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:26:01.935420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"107'6","reported_seq":20,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651108+0000","last_change":"2024-02-19T15:27:21.031559+0000","last_active":"2024-02-19T15:27:23.651108+0000","last_peered":"2024-02-19T15:27:23.651108+0000","last_clean":"2024-02-19T15:27:23.651108+0000","last_became_active":"2024-02-19T15:27:21.031280+0000","last_became_peered":"2024-02-19T15:27:21.031280+0000","last_unstale":"2024-02-19T15:27:23.651108+0000","last_undegraded":"2024-02-19T15:27:23.651108+0000","last_fullsized":"2024-02-19T15:27:23.651108+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:36:18.076454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650180+0000","last_change":"2024-02-19T15:27:22.044977+0000","last_active":"2024-02-19T15:27:23.650180+0000","last_peered":"2024-02-19T15:27:23.650180+0000","last_clean":"2024-02-19T15:27:23.650180+0000","last_became_active":"2024-02-19T15:27:22.044719+0000","last_became_peered":"2024-02-19T15:27:22.044719+0000","last_unstale":"2024-02-19T15:27:23.650180+0000","last_undegraded":"2024-02-19T15:27:23.650180+0000","last_fullsized":"2024-02-19T15:27:23.650180+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:24:06.367087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045792+0000","last_change":"2024-02-19T15:27:21.019044+0000","last_active":"2024-02-19T15:27:23.045792+0000","last_peered":"2024-02-19T15:27:23.045792+0000","last_clean":"2024-02-19T15:27:23.045792+0000","last_became_active":"2024-02-19T15:27:21.018732+0000","last_became_peered":"2024-02-19T15:27:21.018732+0000","last_unstale":"2024-02-19T15:27:23.045792+0000","last_undegraded":"2024-02-19T15:27:23.045792+0000","last_fullsized":"2024-02-19T15:27:23.045792+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:51:06.106358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650178+0000","last_change":"2024-02-19T15:27:22.052449+0000","last_active":"2024-02-19T15:27:23.650178+0000","last_peered":"2024-02-19T15:27:23.650178+0000","last_clean":"2024-02-19T15:27:23.650178+0000","last_became_active":"2024-02-19T15:27:22.051990+0000","last_became_peered":"2024-02-19T15:27:22.051990+0000","last_unstale":"2024-02-19T15:27:23.650178+0000","last_undegraded":"2024-02-19T15:27:23.650178+0000","last_fullsized":"2024-02-19T15:27:23.650178+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:57:49.784754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.027358+0000","last_change":"2024-02-19T15:27:21.022352+0000","last_active":"2024-02-19T15:27:22.027358+0000","last_peered":"2024-02-19T15:27:22.027358+0000","last_clean":"2024-02-19T15:27:22.027358+0000","last_became_active":"2024-02-19T15:27:21.021353+0000","last_became_peered":"2024-02-19T15:27:21.021353+0000","last_unstale":"2024-02-19T15:27:22.027358+0000","last_undegraded":"2024-02-19T15:27:22.027358+0000","last_fullsized":"2024-02-19T15:27:22.027358+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:05:11.130486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649860+0000","last_change":"2024-02-19T15:27:22.046349+0000","last_active":"2024-02-19T15:27:23.649860+0000","last_peered":"2024-02-19T15:27:23.649860+0000","last_clean":"2024-02-19T15:27:23.649860+0000","last_became_active":"2024-02-19T15:27:22.045913+0000","last_became_peered":"2024-02-19T15:27:22.045913+0000","last_unstale":"2024-02-19T15:27:23.649860+0000","last_undegraded":"2024-02-19T15:27:23.649860+0000","last_fullsized":"2024-02-19T15:27:23.649860+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:34.686973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"107'8","reported_seq":22,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650398+0000","last_change":"2024-02-19T15:27:21.042396+0000","last_active":"2024-02-19T15:27:23.650398+0000","last_peered":"2024-02-19T15:27:23.650398+0000","last_clean":"2024-02-19T15:27:23.650398+0000","last_became_active":"2024-02-19T15:27:21.042017+0000","last_became_peered":"2024-02-19T15:27:21.042017+0000","last_unstale":"2024-02-19T15:27:23.650398+0000","last_undegraded":"2024-02-19T15:27:23.650398+0000","last_fullsized":"2024-02-19T15:27:23.650398+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:18:27.189283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650690+0000","last_change":"2024-02-19T15:27:22.038153+0000","last_active":"2024-02-19T15:27:23.650690+0000","last_peered":"2024-02-19T15:27:23.650690+0000","last_clean":"2024-02-19T15:27:23.650690+0000","last_became_active":"2024-02-19T15:27:22.037885+0000","last_became_peered":"2024-02-19T15:27:22.037885+0000","last_unstale":"2024-02-19T15:27:23.650690+0000","last_undegraded":"2024-02-19T15:27:23.650690+0000","last_fullsized":"2024-02-19T15:27:23.650690+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:16.020376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651868+0000","last_change":"2024-02-19T15:27:21.027023+0000","last_active":"2024-02-19T15:27:23.651868+0000","last_peered":"2024-02-19T15:27:23.651868+0000","last_clean":"2024-02-19T15:27:23.651868+0000","last_became_active":"2024-02-19T15:27:21.026565+0000","last_became_peered":"2024-02-19T15:27:21.026565+0000","last_unstale":"2024-02-19T15:27:23.651868+0000","last_undegraded":"2024-02-19T15:27:23.651868+0000","last_fullsized":"2024-02-19T15:27:23.651868+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:04:34.220187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.046040+0000","last_change":"2024-02-19T15:27:22.058919+0000","last_active":"2024-02-19T15:27:23.046040+0000","last_peered":"2024-02-19T15:27:23.046040+0000","last_clean":"2024-02-19T15:27:23.046040+0000","last_became_active":"2024-02-19T15:27:22.058619+0000","last_became_peered":"2024-02-19T15:27:22.058619+0000","last_unstale":"2024-02-19T15:27:23.046040+0000","last_undegraded":"2024-02-19T15:27:23.046040+0000","last_fullsized":"2024-02-19T15:27:23.046040+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:26:01.935420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650785+0000","last_change":"2024-02-19T15:27:21.033163+0000","last_active":"2024-02-19T15:27:23.650785+0000","last_peered":"2024-02-19T15:27:23.650785+0000","last_clean":"2024-02-19T15:27:23.650785+0000","last_became_active":"2024-02-19T15:27:21.032778+0000","last_became_peered":"2024-02-19T15:27:21.032778+0000","last_unstale":"2024-02-19T15:27:23.650785+0000","last_undegraded":"2024-02-19T15:27:23.650785+0000","last_fullsized":"2024-02-19T15:27:23.650785+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:22:19.544810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650504+0000","last_change":"2024-02-19T15:27:22.050672+0000","last_active":"2024-02-19T15:27:23.650504+0000","last_peered":"2024-02-19T15:27:23.650504+0000","last_clean":"2024-02-19T15:27:23.650504+0000","last_became_active":"2024-02-19T15:27:22.050405+0000","last_became_peered":"2024-02-19T15:27:22.050405+0000","last_unstale":"2024-02-19T15:27:23.650504+0000","last_undegraded":"2024-02-19T15:27:23.650504+0000","last_fullsized":"2024-02-19T15:27:23.650504+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:20:30.719772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.047598+0000","last_change":"2024-02-19T15:27:21.033698+0000","last_active":"2024-02-19T15:27:22.047598+0000","last_peered":"2024-02-19T15:27:22.047598+0000","last_clean":"2024-02-19T15:27:22.047598+0000","last_became_active":"2024-02-19T15:27:21.033354+0000","last_became_peered":"2024-02-19T15:27:21.033354+0000","last_unstale":"2024-02-19T15:27:22.047598+0000","last_undegraded":"2024-02-19T15:27:22.047598+0000","last_fullsized":"2024-02-19T15:27:22.047598+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:42.028837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650634+0000","last_change":"2024-02-19T15:27:22.049020+0000","last_active":"2024-02-19T15:27:23.650634+0000","last_peered":"2024-02-19T15:27:23.650634+0000","last_clean":"2024-02-19T15:27:23.650634+0000","last_became_active":"2024-02-19T15:27:22.048713+0000","last_became_peered":"2024-02-19T15:27:22.048713+0000","last_unstale":"2024-02-19T15:27:23.650634+0000","last_undegraded":"2024-02-19T15:27:23.650634+0000","last_fullsized":"2024-02-19T15:27:23.650634+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:59:30.423613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650092+0000","last_change":"2024-02-19T15:27:21.031371+0000","last_active":"2024-02-19T15:27:23.650092+0000","last_peered":"2024-02-19T15:27:23.650092+0000","last_clean":"2024-02-19T15:27:23.650092+0000","last_became_active":"2024-02-19T15:27:21.031066+0000","last_became_peered":"2024-02-19T15:27:21.031066+0000","last_unstale":"2024-02-19T15:27:23.650092+0000","last_undegraded":"2024-02-19T15:27:23.650092+0000","last_fullsized":"2024-02-19T15:27:23.650092+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:03:18.300707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650512+0000","last_change":"2024-02-19T15:27:22.048519+0000","last_active":"2024-02-19T15:27:23.650512+0000","last_peered":"2024-02-19T15:27:23.650512+0000","last_clean":"2024-02-19T15:27:23.650512+0000","last_became_active":"2024-02-19T15:27:22.048290+0000","last_became_peered":"2024-02-19T15:27:22.048290+0000","last_unstale":"2024-02-19T15:27:23.650512+0000","last_undegraded":"2024-02-19T15:27:23.650512+0000","last_fullsized":"2024-02-19T15:27:23.650512+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:07:37.708947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045798+0000","last_change":"2024-02-19T15:27:21.026803+0000","last_active":"2024-02-19T15:27:23.045798+0000","last_peered":"2024-02-19T15:27:23.045798+0000","last_clean":"2024-02-19T15:27:23.045798+0000","last_became_active":"2024-02-19T15:27:21.025817+0000","last_became_peered":"2024-02-19T15:27:21.025817+0000","last_unstale":"2024-02-19T15:27:23.045798+0000","last_undegraded":"2024-02-19T15:27:23.045798+0000","last_fullsized":"2024-02-19T15:27:23.045798+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:31:23.769771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650975+0000","last_change":"2024-02-19T15:27:22.058638+0000","last_active":"2024-02-19T15:27:23.650975+0000","last_peered":"2024-02-19T15:27:23.650975+0000","last_clean":"2024-02-19T15:27:23.650975+0000","last_became_active":"2024-02-19T15:27:22.057438+0000","last_became_peered":"2024-02-19T15:27:22.057438+0000","last_unstale":"2024-02-19T15:27:23.650975+0000","last_undegraded":"2024-02-19T15:27:23.650975+0000","last_fullsized":"2024-02-19T15:27:23.650975+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:42:11.448198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649997+0000","last_change":"2024-02-19T15:27:21.042086+0000","last_active":"2024-02-19T15:27:23.649997+0000","last_peered":"2024-02-19T15:27:23.649997+0000","last_clean":"2024-02-19T15:27:23.649997+0000","last_became_active":"2024-02-19T15:27:21.041580+0000","last_became_peered":"2024-02-19T15:27:21.041580+0000","last_unstale":"2024-02-19T15:27:23.649997+0000","last_undegraded":"2024-02-19T15:27:23.649997+0000","last_fullsized":"2024-02-19T15:27:23.649997+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:30:43.579542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650400+0000","last_change":"2024-02-19T15:27:22.052442+0000","last_active":"2024-02-19T15:27:23.650400+0000","last_peered":"2024-02-19T15:27:23.650400+0000","last_clean":"2024-02-19T15:27:23.650400+0000","last_became_active":"2024-02-19T15:27:22.051952+0000","last_became_peered":"2024-02-19T15:27:22.051952+0000","last_unstale":"2024-02-19T15:27:23.650400+0000","last_undegraded":"2024-02-19T15:27:23.650400+0000","last_fullsized":"2024-02-19T15:27:23.650400+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:10:38.351264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045448+0000","last_change":"2024-02-19T15:27:21.038892+0000","last_active":"2024-02-19T15:27:23.045448+0000","last_peered":"2024-02-19T15:27:23.045448+0000","last_clean":"2024-02-19T15:27:23.045448+0000","last_became_active":"2024-02-19T15:27:21.038595+0000","last_became_peered":"2024-02-19T15:27:21.038595+0000","last_unstale":"2024-02-19T15:27:23.045448+0000","last_undegraded":"2024-02-19T15:27:23.045448+0000","last_fullsized":"2024-02-19T15:27:23.045448+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:25:30.120547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651487+0000","last_change":"2024-02-19T15:27:22.046134+0000","last_active":"2024-02-19T15:27:23.651487+0000","last_peered":"2024-02-19T15:27:23.651487+0000","last_clean":"2024-02-19T15:27:23.651487+0000","last_became_active":"2024-02-19T15:27:22.045608+0000","last_became_peered":"2024-02-19T15:27:22.045608+0000","last_unstale":"2024-02-19T15:27:23.651487+0000","last_undegraded":"2024-02-19T15:27:23.651487+0000","last_fullsized":"2024-02-19T15:27:23.651487+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:43:10.829112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650299+0000","last_change":"2024-02-19T15:27:21.017652+0000","last_active":"2024-02-19T15:27:23.650299+0000","last_peered":"2024-02-19T15:27:23.650299+0000","last_clean":"2024-02-19T15:27:23.650299+0000","last_became_active":"2024-02-19T15:27:21.017345+0000","last_became_peered":"2024-02-19T15:27:21.017345+0000","last_unstale":"2024-02-19T15:27:23.650299+0000","last_undegraded":"2024-02-19T15:27:23.650299+0000","last_fullsized":"2024-02-19T15:27:23.650299+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:43:48.389820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.042243+0000","last_change":"2024-02-19T15:27:22.035845+0000","last_active":"2024-02-19T15:27:22.042243+0000","last_peered":"2024-02-19T15:27:22.042243+0000","last_clean":"2024-02-19T15:27:22.042243+0000","last_became_active":"2024-02-19T15:27:22.035521+0000","last_became_peered":"2024-02-19T15:27:22.035521+0000","last_unstale":"2024-02-19T15:27:22.042243+0000","last_undegraded":"2024-02-19T15:27:22.042243+0000","last_fullsized":"2024-02-19T15:27:22.042243+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:10:54.441676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651663+0000","last_change":"2024-02-19T15:27:21.034876+0000","last_active":"2024-02-19T15:27:23.651663+0000","last_peered":"2024-02-19T15:27:23.651663+0000","last_clean":"2024-02-19T15:27:23.651663+0000","last_became_active":"2024-02-19T15:27:21.032593+0000","last_became_peered":"2024-02-19T15:27:21.032593+0000","last_unstale":"2024-02-19T15:27:23.651663+0000","last_undegraded":"2024-02-19T15:27:23.651663+0000","last_fullsized":"2024-02-19T15:27:23.651663+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:48:01.413719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651710+0000","last_change":"2024-02-19T15:27:22.047285+0000","last_active":"2024-02-19T15:27:23.651710+0000","last_peered":"2024-02-19T15:27:23.651710+0000","last_clean":"2024-02-19T15:27:23.651710+0000","last_became_active":"2024-02-19T15:27:22.046953+0000","last_became_peered":"2024-02-19T15:27:22.046953+0000","last_unstale":"2024-02-19T15:27:23.651710+0000","last_undegraded":"2024-02-19T15:27:23.651710+0000","last_fullsized":"2024-02-19T15:27:23.651710+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:47:04.051444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.029072+0000","last_change":"2024-02-19T15:27:21.025533+0000","last_active":"2024-02-19T15:27:22.029072+0000","last_peered":"2024-02-19T15:27:22.029072+0000","last_clean":"2024-02-19T15:27:22.029072+0000","last_became_active":"2024-02-19T15:27:21.025161+0000","last_became_peered":"2024-02-19T15:27:21.025161+0000","last_unstale":"2024-02-19T15:27:22.029072+0000","last_undegraded":"2024-02-19T15:27:22.029072+0000","last_fullsized":"2024-02-19T15:27:22.029072+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:59:09.129286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650976+0000","last_change":"2024-02-19T15:27:22.066060+0000","last_active":"2024-02-19T15:27:23.650976+0000","last_peered":"2024-02-19T15:27:23.650976+0000","last_clean":"2024-02-19T15:27:23.650976+0000","last_became_active":"2024-02-19T15:27:22.065704+0000","last_became_peered":"2024-02-19T15:27:22.065704+0000","last_unstale":"2024-02-19T15:27:23.650976+0000","last_undegraded":"2024-02-19T15:27:23.650976+0000","last_fullsized":"2024-02-19T15:27:23.650976+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:39:24.288533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651282+0000","last_change":"2024-02-19T15:27:21.021965+0000","last_active":"2024-02-19T15:27:23.651282+0000","last_peered":"2024-02-19T15:27:23.651282+0000","last_clean":"2024-02-19T15:27:23.651282+0000","last_became_active":"2024-02-19T15:27:21.021319+0000","last_became_peered":"2024-02-19T15:27:21.021319+0000","last_unstale":"2024-02-19T15:27:23.651282+0000","last_undegraded":"2024-02-19T15:27:23.651282+0000","last_fullsized":"2024-02-19T15:27:23.651282+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:58:28.270645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-02-19T15:27:22.041845+0000","last_change":"2024-02-19T15:27:22.037785+0000","last_active":"2024-02-19T15:27:22.041845+0000","last_peered":"2024-02-19T15:27:22.041845+0000","last_clean":"2024-02-19T15:27:22.041845+0000","last_became_active":"2024-02-19T15:27:22.037460+0000","last_became_peered":"2024-02-19T15:27:22.037460+0000","last_unstale":"2024-02-19T15:27:22.041845+0000","last_undegraded":"2024-02-19T15:27:22.041845+0000","last_fullsized":"2024-02-19T15:27:22.041845+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:46.258004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651412+0000","last_change":"2024-02-19T15:27:21.028129+0000","last_active":"2024-02-19T15:27:23.651412+0000","last_peered":"2024-02-19T15:27:23.651412+0000","last_clean":"2024-02-19T15:27:23.651412+0000","last_became_active":"2024-02-19T15:27:21.027822+0000","last_became_peered":"2024-02-19T15:27:21.027822+0000","last_unstale":"2024-02-19T15:27:23.651412+0000","last_undegraded":"2024-02-19T15:27:23.651412+0000","last_fullsized":"2024-02-19T15:27:23.651412+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:51.292175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045206+0000","last_change":"2024-02-19T15:27:22.045423+0000","last_active":"2024-02-19T15:27:23.045206+0000","last_peered":"2024-02-19T15:27:23.045206+0000","last_clean":"2024-02-19T15:27:23.045206+0000","last_became_active":"2024-02-19T15:27:22.045159+0000","last_became_peered":"2024-02-19T15:27:22.045159+0000","last_unstale":"2024-02-19T15:27:23.045206+0000","last_undegraded":"2024-02-19T15:27:23.045206+0000","last_fullsized":"2024-02-19T15:27:23.045206+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:38:40.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649872+0000","last_change":"2024-02-19T15:27:21.032501+0000","last_active":"2024-02-19T15:27:23.649872+0000","last_peered":"2024-02-19T15:27:23.649872+0000","last_clean":"2024-02-19T15:27:23.649872+0000","last_became_active":"2024-02-19T15:27:21.032206+0000","last_became_peered":"2024-02-19T15:27:21.032206+0000","last_unstale":"2024-02-19T15:27:23.649872+0000","last_undegraded":"2024-02-19T15:27:23.649872+0000","last_fullsized":"2024-02-19T15:27:23.649872+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:18:10.944093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045734+0000","last_change":"2024-02-19T15:27:22.051965+0000","last_active":"2024-02-19T15:27:23.045734+0000","last_peered":"2024-02-19T15:27:23.045734+0000","last_clean":"2024-02-19T15:27:23.045734+0000","last_became_active":"2024-02-19T15:27:22.051580+0000","last_became_peered":"2024-02-19T15:27:22.051580+0000","last_unstale":"2024-02-19T15:27:23.045734+0000","last_undegraded":"2024-02-19T15:27:23.045734+0000","last_fullsized":"2024-02-19T15:27:23.045734+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:48:02.419533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045376+0000","last_change":"2024-02-19T15:27:21.035917+0000","last_active":"2024-02-19T15:27:23.045376+0000","last_peered":"2024-02-19T15:27:23.045376+0000","last_clean":"2024-02-19T15:27:23.045376+0000","last_became_active":"2024-02-19T15:27:21.035628+0000","last_became_peered":"2024-02-19T15:27:21.035628+0000","last_unstale":"2024-02-19T15:27:23.045376+0000","last_undegraded":"2024-02-19T15:27:23.045376+0000","last_fullsized":"2024-02-19T15:27:23.045376+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:08:02.984985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650053+0000","last_change":"2024-02-19T15:27:22.047614+0000","last_active":"2024-02-19T15:27:23.650053+0000","last_peered":"2024-02-19T15:27:23.650053+0000","last_clean":"2024-02-19T15:27:23.650053+0000","last_became_active":"2024-02-19T15:27:22.047282+0000","last_became_peered":"2024-02-19T15:27:22.047282+0000","last_unstale":"2024-02-19T15:27:23.650053+0000","last_undegraded":"2024-02-19T15:27:23.650053+0000","last_fullsized":"2024-02-19T15:27:23.650053+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:57:49.784754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650918+0000","last_change":"2024-02-19T15:27:21.029497+0000","last_active":"2024-02-19T15:27:23.650918+0000","last_peered":"2024-02-19T15:27:23.650918+0000","last_clean":"2024-02-19T15:27:23.650918+0000","last_became_active":"2024-02-19T15:27:21.029251+0000","last_became_peered":"2024-02-19T15:27:21.029251+0000","last_unstale":"2024-02-19T15:27:23.650918+0000","last_undegraded":"2024-02-19T15:27:23.650918+0000","last_fullsized":"2024-02-19T15:27:23.650918+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:11:30.790177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650642+0000","last_change":"2024-02-19T15:27:22.068736+0000","last_active":"2024-02-19T15:27:23.650642+0000","last_peered":"2024-02-19T15:27:23.650642+0000","last_clean":"2024-02-19T15:27:23.650642+0000","last_became_active":"2024-02-19T15:27:22.068450+0000","last_became_peered":"2024-02-19T15:27:22.068450+0000","last_unstale":"2024-02-19T15:27:23.650642+0000","last_undegraded":"2024-02-19T15:27:23.650642+0000","last_fullsized":"2024-02-19T15:27:23.650642+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:48.996005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650393+0000","last_change":"2024-02-19T15:27:21.024536+0000","last_active":"2024-02-19T15:27:23.650393+0000","last_peered":"2024-02-19T15:27:23.650393+0000","last_clean":"2024-02-19T15:27:23.650393+0000","last_became_active":"2024-02-19T15:27:21.020949+0000","last_became_peered":"2024-02-19T15:27:21.020949+0000","last_unstale":"2024-02-19T15:27:23.650393+0000","last_undegraded":"2024-02-19T15:27:23.650393+0000","last_fullsized":"2024-02-19T15:27:23.650393+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:20:37.538132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649727+0000","last_change":"2024-02-19T15:27:22.052196+0000","last_active":"2024-02-19T15:27:23.649727+0000","last_peered":"2024-02-19T15:27:23.649727+0000","last_clean":"2024-02-19T15:27:23.649727+0000","last_became_active":"2024-02-19T15:27:22.051895+0000","last_became_peered":"2024-02-19T15:27:22.051895+0000","last_unstale":"2024-02-19T15:27:23.649727+0000","last_undegraded":"2024-02-19T15:27:23.649727+0000","last_fullsized":"2024-02-19T15:27:23.649727+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:53:30.741750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649841+0000","last_change":"2024-02-19T15:27:21.036160+0000","last_active":"2024-02-19T15:27:23.649841+0000","last_peered":"2024-02-19T15:27:23.649841+0000","last_clean":"2024-02-19T15:27:23.649841+0000","last_became_active":"2024-02-19T15:27:21.035805+0000","last_became_peered":"2024-02-19T15:27:21.035805+0000","last_unstale":"2024-02-19T15:27:23.649841+0000","last_undegraded":"2024-02-19T15:27:23.649841+0000","last_fullsized":"2024-02-19T15:27:23.649841+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:25:19.941750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":324,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650271+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:27:23.650271+0000","last_peered":"2024-02-19T15:27:23.650271+0000","last_clean":"2024-02-19T15:27:23.650271+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:27:23.650271+0000","last_undegraded":"2024-02-19T15:27:23.650271+0000","last_fullsized":"2024-02-19T15:27:23.650271+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509509,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1314816,"data_stored":563776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.622,"15min":0.62},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.66100000000000003,"15min":0.746},"min":{"1min":0.39100000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.8100000000000001,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.76100000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63800000000000001,"15min":0.65000000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.44,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.774,"5min":2.774,"15min":2.774},"last":0.58099999999999996}]},{"osd":2,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63100000000000001,"15min":0.63200000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.69899999999999995,"15min":0.68899999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.69299999999999995}]},{"osd":3,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":0.83499999999999996,"15min":0.72699999999999998},"min":{"1min":0.46600000000000003,"5min":0.438,"15min":0.438},"max":{"1min":14.045999999999999,"5min":14.045999999999999,"15min":14.045999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.67600000000000005,"15min":0.66200000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.85999999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.64500000000000002,"15min":0.61299999999999999},"min":{"1min":0.32400000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.69099999999999995,"15min":0.625},"min":{"1min":0.35599999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":2.8500000000000001,"5min":2.8500000000000001,"15min":2.8500000000000001},"last":0.83599999999999997}]},{"osd":5,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.61899999999999999},"min":{"1min":0.38200000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.72899999999999998,"15min":0.68500000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.73199999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.75900000000000001,"15min":0.68100000000000005},"min":{"1min":0.42799999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":3.0539999999999998,"5min":3.0539999999999998,"15min":3.0539999999999998},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.755,"15min":0.69299999999999995},"min":{"1min":0.439,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.256,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.91100000000000003}]}]},{"osd":7,"up_from":14,"seq":60129542214,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37244,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334596,"statfs":{"total":96636764160,"available":96598626304,"internally_reserved":0,"allocated":1941504,"data_stored":1167835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":2.016,"5min":0.88300000000000001,"15min":0.68500000000000005},"min":{"1min":0.51800000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":15.127000000000001,"5min":15.127000000000001,"15min":15.127000000000001},"last":0.752},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.71899999999999997,"15min":0.69399999999999995},"min":{"1min":0.47499999999999998,"5min":0.376,"15min":0.376},"max":{"1min":6.0890000000000004,"5min":6.0890000000000004,"15min":6.0890000000000004},"last":0.61199999999999999}]},{"osd":1,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":2.0910000000000002,"5min":0.93600000000000005,"15min":0.70199999999999996},"min":{"1min":0.47499999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":15.196999999999999,"5min":15.196999999999999,"15min":15.196999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":1.3100000000000001,"5min":0.81999999999999995,"15min":0.76000000000000001},"min":{"1min":0.45600000000000002,"5min":0.437,"15min":0.437},"max":{"1min":10.933999999999999,"5min":10.933999999999999,"15min":10.933999999999999},"last":0.73699999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":0.79600000000000004,"15min":0.68600000000000005},"min":{"1min":0.45700000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":10.84,"5min":10.84,"15min":10.84},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.71199999999999997,"15min":0.69999999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":2.3170000000000002,"5min":2.508,"15min":2.508},"last":0.71499999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":1.6539999999999999,"5min":0.86199999999999999,"15min":0.72599999999999998},"min":{"1min":0.49199999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":17.094000000000001,"5min":17.094000000000001,"15min":17.094000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.226,"5min":0.81799999999999995,"15min":0.73099999999999998},"min":{"1min":0.46300000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":7.7190000000000003,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.64600000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.61299999999999999,"15min":0.54500000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":2.266,"5min":0.95799999999999996,"15min":0.69499999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":26.625,"5min":26.625,"15min":26.625},"last":0.59199999999999997}]},{"osd":5,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":2.5350000000000001,"5min":1.0549999999999999,"15min":0.71699999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":31.844999999999999,"5min":31.844999999999999,"15min":31.844999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.6100000000000001,"5min":0.876,"15min":0.70399999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":16.776,"5min":16.776,"15min":16.776},"last":0.629}]},{"osd":6,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.70499999999999996,"15min":0.625},"min":{"1min":0.375,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.76400000000000001,"15min":0.68799999999999994},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":3.5099999999999998,"5min":3.5099999999999998,"15min":3.5099999999999998},"last":0.82299999999999995}]}]},{"osd":5,"up_from":14,"seq":60129542215,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36656,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335184,"statfs":{"total":96636764160,"available":96599228416,"internally_reserved":0,"allocated":1347584,"data_stored":577455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.69699999999999995,"15min":0.65300000000000002},"min":{"1min":0.48399999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":2.6659999999999999,"5min":5.7789999999999999,"15min":5.7789999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73799999999999999,"15min":0.64600000000000002},"min":{"1min":0.45000000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.2370000000000001,"5min":12.359999999999999,"15min":12.359999999999999},"last":0.52700000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.76600000000000001,"15min":0.67900000000000005},"min":{"1min":0.496,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.6499999999999999,"5min":10.938000000000001,"15min":10.938000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.82199999999999995,"15min":0.72399999999999998},"min":{"1min":0.45800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":3.625,"5min":10.531000000000001,"15min":10.531000000000001},"last":0.69299999999999995}]},{"osd":2,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.81499999999999995,"15min":0.72699999999999998},"min":{"1min":0.503,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":2.589,"5min":12.925000000000001,"15min":12.925000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.74099999999999999,"15min":0.64500000000000002},"min":{"1min":0.46999999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89100000000000001,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.85899999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.96499999999999997,"15min":0.753},"min":{"1min":0.51000000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":5.492,"5min":26.161999999999999,"15min":26.161999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.76200000000000001},"min":{"1min":0.52700000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":6.4939999999999998,"5min":27.934999999999999,"15min":27.934999999999999},"last":0.66600000000000004}]},{"osd":4,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":0.79900000000000004,"15min":0.63300000000000001},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":11.73,"5min":11.73,"15min":11.73},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.117,"5min":0.80200000000000005,"15min":0.70199999999999996},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":10.731,"5min":10.731,"15min":10.731},"last":0.80400000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.76900000000000002,"15min":0.67700000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.6320000000000001,"5min":6.5229999999999997,"15min":6.5229999999999997},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.248,"5min":0.92600000000000005,"15min":0.78400000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":10.432,"5min":10.432,"15min":10.432},"last":0.92900000000000005}]},{"osd":7,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.78700000000000003,"15min":0.69799999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.9450000000000001,"5min":11.986000000000001,"15min":11.986000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.83099999999999996,"15min":0.73299999999999998},"min":{"1min":0.42999999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":6.5339999999999998,"5min":8.9770000000000003,"15min":8.9770000000000003},"last":0.78000000000000003}]}]},{"osd":4,"up_from":14,"seq":60129542213,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36656,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335184,"statfs":{"total":96636764160,"available":96599228416,"internally_reserved":0,"allocated":1339392,"data_stored":576851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":4.1779999999999999,"5min":1.3089999999999999,"15min":0.83099999999999996},"min":{"1min":0.38,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":76.117000000000004,"5min":76.117000000000004,"15min":76.117000000000004},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.746,"5min":0.65300000000000002,"15min":0.61499999999999999},"min":{"1min":0.40000000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.2709999999999999,"5min":3.2709999999999999,"15min":3.2709999999999999},"last":0.46800000000000003}]},{"osd":1,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":4.4470000000000001,"5min":1.409,"15min":0.875},"min":{"1min":0.45800000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":80.513000000000005,"5min":80.513000000000005,"15min":80.513000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.70799999999999996,"15min":0.65500000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":7.181,"5min":7.181,"15min":7.181},"last":0.70399999999999996}]},{"osd":2,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":0.77900000000000003,"15min":0.70199999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":14.362,"5min":14.362,"15min":14.362},"last":0.748},{"interface":"front","average":{"1min":4.6100000000000003,"5min":1.4690000000000001,"15min":0.93899999999999995},"min":{"1min":0.42299999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":84.093000000000004,"5min":84.093000000000004,"15min":84.093000000000004},"last":0.42499999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":4.6539999999999999,"5min":1.5049999999999999,"15min":0.94799999999999995},"min":{"1min":0.504,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":84.140000000000001,"5min":84.140000000000001,"15min":84.140000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.44400000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.57399999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":0.76500000000000001,"15min":0.64900000000000002},"min":{"1min":0.379,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":14.318,"5min":14.318,"15min":14.318},"last":0.77800000000000002},{"interface":"front","average":{"1min":4.274,"5min":1.427,"15min":0.92300000000000004},"min":{"1min":0.41099999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":76.164000000000001,"5min":76.164000000000001,"15min":76.164000000000001},"last":0.65900000000000003}]},{"osd":6,"last update":"Mon Feb 19 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64100000000000001,"15min":0.58899999999999997},"min":{"1min":0.33500000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":3.5019999999999998,"5min":1.2589999999999999,"15min":0.84599999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":60.151000000000003,"5min":60.151000000000003,"15min":60.151000000000003},"last":0.80300000000000005}]},{"osd":7,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":2.044,"5min":0.97999999999999998,"15min":0.76400000000000001},"min":{"1min":0.40699999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":29.036000000000001,"5min":29.036000000000001,"15min":29.036000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.0449999999999999,"5min":0.82099999999999995,"15min":0.755},"min":{"1min":0.50700000000000001,"5min":0.443,"15min":0.443},"max":{"1min":7.101,"5min":7.101,"15min":7.101},"last":0.82499999999999996}]}]},{"osd":3,"up_from":14,"seq":60129542213,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40824,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331016,"statfs":{"total":96636764160,"available":96594960384,"internally_reserved":0,"allocated":1355776,"data_stored":578533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41799999999999998,"15min":0.44400000000000001},"min":{"1min":0.23499999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.82699999999999996,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.5,"15min":0.63200000000000001},"min":{"1min":0.28899999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.0840000000000001,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.503}]},{"osd":1,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.437,"15min":0.443},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.54400000000000004,"15min":0.69199999999999995},"min":{"1min":0.316,"5min":0.31,"15min":0.31},"max":{"1min":0.85099999999999998,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.41299999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55500000000000005,"15min":0.60699999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53700000000000003,"15min":0.58499999999999996},"min":{"1min":0.38400000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.53800000000000003}]},{"osd":4,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71299999999999997,"15min":0.68200000000000005},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.97999999999999998,"5min":1.004,"15min":1.004},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.75800000000000001,"15min":0.84599999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0169999999999999,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.63700000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.77000000000000002,"15min":0.84399999999999997},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.99099999999999999,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.73499999999999999,"15min":0.745},"min":{"1min":0.46999999999999997,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89700000000000002,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.86199999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:27:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.79300000000000004,"15min":0.879},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.3180000000000001,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.77500000000000002,"15min":0.81000000000000005},"min":{"1min":0.53900000000000003,"5min":0.502,"15min":0.502},"max":{"1min":1.206,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.76600000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.755,"15min":0.77700000000000002},"min":{"1min":0.57099999999999995,"5min":0.504,"15min":0.504},"max":{"1min":0.996,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.80400000000000005,"15min":0.90800000000000003},"min":{"1min":0.54800000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.1919999999999999,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.746}]}]},{"osd":2,"up_from":14,"seq":60129542214,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36648,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335192,"statfs":{"total":96636764160,"available":96599236608,"internally_reserved":0,"allocated":1339392,"data_stored":576851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.60099999999999998,"15min":0.67500000000000004},"min":{"1min":0.26600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.48899999999999999,"5min":15.42,"15min":15.42},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.61199999999999999,"15min":0.85099999999999998},"min":{"1min":0.28199999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68700000000000006,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.53800000000000003}]},{"osd":1,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.65500000000000003,"15min":0.86299999999999999},"min":{"1min":0.29399999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72799999999999998,"5min":9.5269999999999992,"15min":9.5269999999999992},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.83999999999999997,"15min":0.97199999999999998},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.75600000000000001,"5min":22.954999999999998,"15min":22.954999999999998},"last":0.51500000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.61499999999999999,"15min":0.76500000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.75,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.498},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.68700000000000006,"15min":0.80400000000000005},"min":{"1min":0.42399999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81399999999999995,"5min":7.734,"15min":7.734},"last":0.55200000000000005}]},{"osd":4,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.86399999999999999,"15min":0.88600000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.97999999999999998,"5min":15.369,"15min":15.369},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.77800000000000002,"15min":0.876},"min":{"1min":0.51900000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93600000000000005,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.76900000000000002}]},{"osd":5,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.83299999999999996,"15min":0.876},"min":{"1min":0.47199999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.006,"5min":7.6520000000000001,"15min":7.6520000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.93400000000000005,"15min":0.95699999999999996},"min":{"1min":0.496,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.026,"5min":15.391,"15min":15.391},"last":0.96099999999999997}]},{"osd":6,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.90900000000000003,"15min":0.92800000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.92000000000000004,"5min":15.337,"15min":15.337},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.81399999999999995,"15min":0.93799999999999994},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.93700000000000006,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.81000000000000005}]},{"osd":7,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.85399999999999998,"15min":0.96399999999999997},"min":{"1min":0.57099999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.98099999999999998,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.86099999999999999,"15min":0.93500000000000005},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94299999999999995,"5min":7.6929999999999996,"15min":7.6929999999999996},"last":0.997}]}]},{"osd":1,"up_from":14,"seq":60129542215,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40824,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331016,"statfs":{"total":96636764160,"available":96594960384,"internally_reserved":0,"allocated":1347584,"data_stored":576975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38600000000000001,"15min":0.42199999999999999},"min":{"1min":0.25700000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.68999999999999995,"5min":0.747,"15min":0.747},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.434,"15min":0.46500000000000002},"min":{"1min":0.312,"5min":0.25,"15min":0.25},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.48799999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.44500000000000001},"min":{"1min":0.30499999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.47799999999999998,"15min":0.52300000000000002},"min":{"1min":0.32400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.74099999999999999,"5min":1.071,"15min":1.071},"last":0.47099999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57999999999999996,"15min":0.63},"min":{"1min":0.45500000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.1200000000000001,"5min":1.365,"15min":1.365},"last":0.503},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.55800000000000005,"15min":0.625},"min":{"1min":0.35399999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0069999999999999,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.52900000000000003}]},{"osd":4,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.67400000000000004,"15min":0.62},"min":{"1min":0.434,"5min":0.377,"15min":0.377},"max":{"1min":1.0409999999999999,"5min":1.127,"15min":1.127},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.69999999999999996,"15min":0.68200000000000005},"min":{"1min":0.56100000000000005,"5min":0.435,"15min":0.435},"max":{"1min":1.083,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.54900000000000004}]},{"osd":5,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.71199999999999997,"15min":0.69799999999999995},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.88800000000000001,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70899999999999996,"15min":0.68700000000000006},"min":{"1min":0.51200000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.96399999999999997,"5min":1.107,"15min":1.107},"last":0.64700000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.73199999999999998,"15min":0.72899999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.97799999999999998,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.74399999999999999,"15min":0.746},"min":{"1min":0.56299999999999994,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.1559999999999999,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.67600000000000005}]},{"osd":7,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.78400000000000003},"min":{"1min":0.52700000000000002,"5min":0.496,"15min":0.496},"max":{"1min":0.94599999999999995,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.75},"min":{"1min":0.54500000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.99299999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.59099999999999997}]}]},{"osd":0,"up_from":14,"seq":60129542213,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41408,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330432,"statfs":{"total":96636764160,"available":96594362368,"internally_reserved":0,"allocated":1953792,"data_stored":1168943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.45300000000000001,"15min":0.60999999999999999},"min":{"1min":0.24299999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.745,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.45800000000000002,"15min":0.48199999999999998},"min":{"1min":0.34200000000000003,"5min":0.252,"15min":0.252},"max":{"1min":0.77300000000000002,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.46899999999999997}]},{"osd":2,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59999999999999998,"15min":0.76600000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.82399999999999995,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.56100000000000005,"15min":0.72699999999999998},"min":{"1min":0.35699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.85899999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.51900000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52000000000000002,"15min":0.66300000000000003},"min":{"1min":0.307,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.93799999999999994,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.495},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53300000000000003,"15min":0.60499999999999998},"min":{"1min":0.376,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.052,"5min":1.345,"15min":1.345},"last":0.53900000000000003}]},{"osd":4,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.73199999999999998,"15min":0.73799999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.091,"5min":1.095,"15min":1.095},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.79500000000000004,"15min":0.90900000000000003},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.085,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.66600000000000004}]},{"osd":5,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.71399999999999997,"15min":0.74299999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.036,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.73699999999999999,"15min":0.84099999999999997},"min":{"1min":0.53600000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.169,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.57599999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.76400000000000001,"15min":0.89600000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.1200000000000001,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.624},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.71999999999999997,"15min":0.751},"min":{"1min":0.50800000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.746}]},{"osd":7,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.79600000000000004,"15min":0.92400000000000004},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.1399999999999999,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.78100000000000003,"15min":0.92300000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.149,"5min":4.915,"15min":4.915},"last":0.76000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:27:31.049 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:27:31.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.183+0000 7fd3d7200640 1 Processor -- start 2024-02-19T15:27:31.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.187+0000 7fd3d7200640 1 -- start start 2024-02-19T15:27:31.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.187+0000 7fd3d7200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3d815ba70 0x7fd3d806dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:31.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.187+0000 7fd3d7200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd3d80727d0 0x7fd3d815be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:31.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.187+0000 7fd3d7200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d806e6a0 con 0x7fd3d80b1ee0 2024-02-19T15:27:31.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.187+0000 7fd3d7200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3d806e980 con 0x7fd3d80727d0 2024-02-19T15:27:31.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.187+0000 7fd3d7200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d80b1400 con 0x7fd3d815ba70 2024-02-19T15:27:31.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.187+0000 7fd3d6800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd3d80727d0 0x7fd3d815be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:31.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.187+0000 7fd3d6800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd3d80727d0 0x7fd3d815be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55376/0 (socket says 172.21.15.47:55376) 2024-02-19T15:27:31.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.187+0000 7fd3d6800640 1 -- 172.21.15.47:0/1914989055 learned_addr learned my addr 172.21.15.47:0/1914989055 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:31.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.187+0000 7fd3d5e00640 1 --2- 172.21.15.47:0/1914989055 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3d815ba70 0x7fd3d806dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:31.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.187+0000 7fd3d6800640 1 -- 172.21.15.47:0/1914989055 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3d815ba70 msgr2=0x7fd3d806dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:31.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.187+0000 7fd3d6800640 1 --2- 172.21.15.47:0/1914989055 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3d815ba70 0x7fd3d806dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.187+0000 7fd3d6800640 1 -- 172.21.15.47:0/1914989055 >> v1:172.21.15.116:6789/0 conn(0x7fd3d80b1ee0 legacy=0x7fd3d80720c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:31.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.187+0000 7fd3d6800640 1 -- 172.21.15.47:0/1914989055 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d80b16e0 con 0x7fd3d80727d0 2024-02-19T15:27:31.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.191+0000 7fd3d6800640 1 --2- 172.21.15.47:0/1914989055 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3d80727d0 0x7fd3d815be50 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0009e50 tx=0x7fd3d001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd94c6b48ece1e2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:31.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.191+0000 7fd3d5400640 1 -- 172.21.15.47:0/1914989055 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d00a57d0 con 0x7fd3d80727d0 2024-02-19T15:27:31.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.191+0000 7fd3d5400640 1 -- 172.21.15.47:0/1914989055 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d00a5930 con 0x7fd3d80727d0 2024-02-19T15:27:31.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.191+0000 7fd3d5400640 1 -- 172.21.15.47:0/1914989055 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d00a5cc0 con 0x7fd3d80727d0 2024-02-19T15:27:31.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.191+0000 7fd3d7200640 1 -- 172.21.15.47:0/1914989055 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3d80727d0 msgr2=0x7fd3d815be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:31.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.191+0000 7fd3d7200640 1 --2- 172.21.15.47:0/1914989055 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3d80727d0 0x7fd3d815be50 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0009e50 tx=0x7fd3d001c930 comp rx=0 tx=0).stop 2024-02-19T15:27:31.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.191+0000 7fd3d7200640 1 -- 172.21.15.47:0/1914989055 shutdown_connections 2024-02-19T15:27:31.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.191+0000 7fd3d7200640 1 --2- 172.21.15.47:0/1914989055 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3d80727d0 0x7fd3d815be50 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.191+0000 7fd3d7200640 1 --2- 172.21.15.47:0/1914989055 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3d815ba70 0x7fd3d806dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.191+0000 7fd3d7200640 1 -- 172.21.15.47:0/1914989055 >> 172.21.15.47:0/1914989055 conn(0x7fd3d80cf7c0 msgr2=0x7fd3d80cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:31.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.191+0000 7fd3d7200640 1 -- 172.21.15.47:0/1914989055 shutdown_connections 2024-02-19T15:27:31.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.191+0000 7fd3d7200640 1 -- 172.21.15.47:0/1914989055 wait complete. 2024-02-19T15:27:31.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3d7200640 1 Processor -- start 2024-02-19T15:27:31.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3d7200640 1 -- start start 2024-02-19T15:27:31.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3d7200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3d80727d0 0x7fd3d81572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:31.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3d7200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3d80b1ee0 0x7fd3d81532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:31.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3d7200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3d815ba70 0x7fd3d814d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:31.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3d7200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d81537e0 con 0x7fd3d80727d0 2024-02-19T15:27:31.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3d7200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d80b2520 con 0x7fd3d815ba70 2024-02-19T15:27:31.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3d7200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d8157cd0 con 0x7fd3d80b1ee0 2024-02-19T15:27:31.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3d5e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3d80727d0 0x7fd3d81572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:31.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3d5e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3d80727d0 0x7fd3d81572f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55378/0 (socket says 172.21.15.47:55378) 2024-02-19T15:27:31.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3d5e00640 1 -- 172.21.15.47:0/697422589 learned_addr learned my addr 172.21.15.47:0/697422589 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:31.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3d6800640 1 --2- 172.21.15.47:0/697422589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3d815ba70 0x7fd3d814d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:31.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3d5e00640 1 -- 172.21.15.47:0/697422589 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3d80b1ee0 msgr2=0x7fd3d81532a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:31.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3cfe00640 1 --2- 172.21.15.47:0/697422589 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3d80b1ee0 0x7fd3d81532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:31.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3d5e00640 1 --2- 172.21.15.47:0/697422589 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3d80b1ee0 0x7fd3d81532a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3d5e00640 1 -- 172.21.15.47:0/697422589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3d815ba70 msgr2=0x7fd3d814d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:31.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3d5e00640 1 --2- 172.21.15.47:0/697422589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3d815ba70 0x7fd3d814d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3d5e00640 1 -- 172.21.15.47:0/697422589 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d0009b30 con 0x7fd3d80727d0 2024-02-19T15:27:31.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3d6800640 1 --2- 172.21.15.47:0/697422589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3d815ba70 0x7fd3d814d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:31.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3cfe00640 1 --2- 172.21.15.47:0/697422589 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3d80b1ee0 0x7fd3d81532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:31.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3d5e00640 1 --2- 172.21.15.47:0/697422589 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3d80727d0 0x7fd3d81572f0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fd3c800d5f0 tx=0x7fd3c800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:31.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3ce000640 1 -- 172.21.15.47:0/697422589 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c80125f0 con 0x7fd3d80727d0 2024-02-19T15:27:31.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3ce000640 1 -- 172.21.15.47:0/697422589 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c8012750 con 0x7fd3d80727d0 2024-02-19T15:27:31.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3d7200640 1 -- 172.21.15.47:0/697422589 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d81579f0 con 0x7fd3d80727d0 2024-02-19T15:27:31.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.195+0000 7fd3ce000640 1 -- 172.21.15.47:0/697422589 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8012ae0 con 0x7fd3d80727d0 2024-02-19T15:27:31.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.199+0000 7fd3d7200640 1 -- 172.21.15.47:0/697422589 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d8155080 con 0x7fd3d80727d0 2024-02-19T15:27:31.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.199+0000 7fd3d7200640 1 -- 172.21.15.47:0/697422589 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d8154c00 con 0x7fd3d80727d0 2024-02-19T15:27:31.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.199+0000 7fd3ce000640 1 -- 172.21.15.47:0/697422589 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd3c8012c40 con 0x7fd3d80727d0 2024-02-19T15:27:31.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.199+0000 7fd3ce000640 1 --2- 172.21.15.47:0/697422589 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd3b4066e40 0x7fd3b4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:31.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.199+0000 7fd3ce000640 1 -- 172.21.15.47:0/697422589 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3c8084f40 con 0x7fd3d80727d0 2024-02-19T15:27:31.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.199+0000 7fd3cfe00640 1 --2- 172.21.15.47:0/697422589 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd3b4066e40 0x7fd3b4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:31.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.199+0000 7fd3cfe00640 1 --2- 172.21.15.47:0/697422589 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd3b4066e40 0x7fd3b4069300 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fd3d80b0cf0 tx=0x7fd3c0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:31.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.203+0000 7fd3ce000640 1 -- 172.21.15.47:0/697422589 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd3d8154c00 con 0x7fd3d80727d0 2024-02-19T15:27:31.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.367+0000 7fd3d7200640 1 -- 172.21.15.47:0/697422589 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd3d8168a00 con 0x7fd3b4066e40 2024-02-19T15:27:31.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.375+0000 7fd3ce000640 1 -- 172.21.15.47:0/697422589 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289005 (secure 0 0 0) 0x7fd3c00053f0 con 0x7fd3b4066e40 2024-02-19T15:27:31.382 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:31.384 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:27:31.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.379+0000 7fd3d7200640 1 -- 172.21.15.47:0/697422589 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd3b4066e40 msgr2=0x7fd3b4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:31.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.379+0000 7fd3d7200640 1 --2- 172.21.15.47:0/697422589 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd3b4066e40 0x7fd3b4069300 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fd3d80b0cf0 tx=0x7fd3c0009040 comp rx=0 tx=0).stop 2024-02-19T15:27:31.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.379+0000 7fd3d7200640 1 -- 172.21.15.47:0/697422589 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3d80727d0 msgr2=0x7fd3d81572f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:31.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.379+0000 7fd3d7200640 1 --2- 172.21.15.47:0/697422589 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3d80727d0 0x7fd3d81572f0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fd3c800d5f0 tx=0x7fd3c800dab0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.379+0000 7fd3d7200640 1 -- 172.21.15.47:0/697422589 shutdown_connections 2024-02-19T15:27:31.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.379+0000 7fd3d7200640 1 --2- 172.21.15.47:0/697422589 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd3b4066e40 0x7fd3b4069300 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.379+0000 7fd3d7200640 1 --2- 172.21.15.47:0/697422589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3d815ba70 0x7fd3d814d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.379+0000 7fd3d7200640 1 --2- 172.21.15.47:0/697422589 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3d80b1ee0 0x7fd3d81532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.379+0000 7fd3d7200640 1 --2- 172.21.15.47:0/697422589 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3d80727d0 0x7fd3d81572f0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.379+0000 7fd3d7200640 1 -- 172.21.15.47:0/697422589 >> 172.21.15.47:0/697422589 conn(0x7fd3d80cf7c0 msgr2=0x7fd3d815e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:31.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.379+0000 7fd3d7200640 1 -- 172.21.15.47:0/697422589 shutdown_connections 2024-02-19T15:27:31.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.379+0000 7fd3d7200640 1 -- 172.21.15.47:0/697422589 wait complete. 2024-02-19T15:27:31.401 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-02-19T15:27:31.223552+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":306916,"kb_used_data":11688,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754667804,"statfs":{"total":773094113280,"available":772779831296,"internally_reserved":0,"allocated":11968512,"data_stored":5800364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":302173746},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.198248"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651409+0000","last_change":"2024-02-19T15:27:22.044103+0000","last_active":"2024-02-19T15:27:23.651409+0000","last_peered":"2024-02-19T15:27:23.651409+0000","last_clean":"2024-02-19T15:27:23.651409+0000","last_became_active":"2024-02-19T15:27:22.043723+0000","last_became_peered":"2024-02-19T15:27:22.043723+0000","last_unstale":"2024-02-19T15:27:23.651409+0000","last_undegraded":"2024-02-19T15:27:23.651409+0000","last_fullsized":"2024-02-19T15:27:23.651409+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:59:04.067243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045480+0000","last_change":"2024-02-19T15:27:22.060740+0000","last_active":"2024-02-19T15:27:23.045480+0000","last_peered":"2024-02-19T15:27:23.045480+0000","last_clean":"2024-02-19T15:27:23.045480+0000","last_became_active":"2024-02-19T15:27:22.060471+0000","last_became_peered":"2024-02-19T15:27:22.060471+0000","last_unstale":"2024-02-19T15:27:23.045480+0000","last_undegraded":"2024-02-19T15:27:23.045480+0000","last_fullsized":"2024-02-19T15:27:23.045480+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:39:12.144358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650032+0000","last_change":"2024-02-19T15:27:22.048031+0000","last_active":"2024-02-19T15:27:23.650032+0000","last_peered":"2024-02-19T15:27:23.650032+0000","last_clean":"2024-02-19T15:27:23.650032+0000","last_became_active":"2024-02-19T15:27:22.047724+0000","last_became_peered":"2024-02-19T15:27:22.047724+0000","last_unstale":"2024-02-19T15:27:23.650032+0000","last_undegraded":"2024-02-19T15:27:23.650032+0000","last_fullsized":"2024-02-19T15:27:23.650032+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:25:25.436463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650600+0000","last_change":"2024-02-19T15:27:22.052619+0000","last_active":"2024-02-19T15:27:23.650600+0000","last_peered":"2024-02-19T15:27:23.650600+0000","last_clean":"2024-02-19T15:27:23.650600+0000","last_became_active":"2024-02-19T15:27:22.052289+0000","last_became_peered":"2024-02-19T15:27:22.052289+0000","last_unstale":"2024-02-19T15:27:23.650600+0000","last_undegraded":"2024-02-19T15:27:23.650600+0000","last_fullsized":"2024-02-19T15:27:23.650600+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:53:18.030121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650775+0000","last_change":"2024-02-19T15:27:22.053770+0000","last_active":"2024-02-19T15:27:23.650775+0000","last_peered":"2024-02-19T15:27:23.650775+0000","last_clean":"2024-02-19T15:27:23.650775+0000","last_became_active":"2024-02-19T15:27:22.053497+0000","last_became_peered":"2024-02-19T15:27:22.053497+0000","last_unstale":"2024-02-19T15:27:23.650775+0000","last_undegraded":"2024-02-19T15:27:23.650775+0000","last_fullsized":"2024-02-19T15:27:23.650775+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:28.666610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045744+0000","last_change":"2024-02-19T15:27:22.063229+0000","last_active":"2024-02-19T15:27:23.045744+0000","last_peered":"2024-02-19T15:27:23.045744+0000","last_clean":"2024-02-19T15:27:23.045744+0000","last_became_active":"2024-02-19T15:27:22.062899+0000","last_became_peered":"2024-02-19T15:27:22.062899+0000","last_unstale":"2024-02-19T15:27:23.045744+0000","last_undegraded":"2024-02-19T15:27:23.045744+0000","last_fullsized":"2024-02-19T15:27:23.045744+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:59:15.699509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649795+0000","last_change":"2024-02-19T15:27:22.040230+0000","last_active":"2024-02-19T15:27:23.649795+0000","last_peered":"2024-02-19T15:27:23.649795+0000","last_clean":"2024-02-19T15:27:23.649795+0000","last_became_active":"2024-02-19T15:27:22.039959+0000","last_became_peered":"2024-02-19T15:27:22.039959+0000","last_unstale":"2024-02-19T15:27:23.649795+0000","last_undegraded":"2024-02-19T15:27:23.649795+0000","last_fullsized":"2024-02-19T15:27:23.649795+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:45:36.102577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649861+0000","last_change":"2024-02-19T15:27:22.060882+0000","last_active":"2024-02-19T15:27:23.649861+0000","last_peered":"2024-02-19T15:27:23.649861+0000","last_clean":"2024-02-19T15:27:23.649861+0000","last_became_active":"2024-02-19T15:27:22.060570+0000","last_became_peered":"2024-02-19T15:27:22.060570+0000","last_unstale":"2024-02-19T15:27:23.649861+0000","last_undegraded":"2024-02-19T15:27:23.649861+0000","last_fullsized":"2024-02-19T15:27:23.649861+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:34.686973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651562+0000","last_change":"2024-02-19T15:27:22.050561+0000","last_active":"2024-02-19T15:27:23.651562+0000","last_peered":"2024-02-19T15:27:23.651562+0000","last_clean":"2024-02-19T15:27:23.651562+0000","last_became_active":"2024-02-19T15:27:22.049431+0000","last_became_peered":"2024-02-19T15:27:22.049431+0000","last_unstale":"2024-02-19T15:27:23.651562+0000","last_undegraded":"2024-02-19T15:27:23.651562+0000","last_fullsized":"2024-02-19T15:27:23.651562+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:58:39.839056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651522+0000","last_change":"2024-02-19T15:27:22.057693+0000","last_active":"2024-02-19T15:27:23.651522+0000","last_peered":"2024-02-19T15:27:23.651522+0000","last_clean":"2024-02-19T15:27:23.651522+0000","last_became_active":"2024-02-19T15:27:22.057412+0000","last_became_peered":"2024-02-19T15:27:22.057412+0000","last_unstale":"2024-02-19T15:27:23.651522+0000","last_undegraded":"2024-02-19T15:27:23.651522+0000","last_fullsized":"2024-02-19T15:27:23.651522+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:53:13.793384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045193+0000","last_change":"2024-02-19T15:27:22.045843+0000","last_active":"2024-02-19T15:27:23.045193+0000","last_peered":"2024-02-19T15:27:23.045193+0000","last_clean":"2024-02-19T15:27:23.045193+0000","last_became_active":"2024-02-19T15:27:22.045550+0000","last_became_peered":"2024-02-19T15:27:22.045550+0000","last_unstale":"2024-02-19T15:27:23.045193+0000","last_undegraded":"2024-02-19T15:27:23.045193+0000","last_fullsized":"2024-02-19T15:27:23.045193+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:27.552780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650768+0000","last_change":"2024-02-19T15:27:22.051357+0000","last_active":"2024-02-19T15:27:23.650768+0000","last_peered":"2024-02-19T15:27:23.650768+0000","last_clean":"2024-02-19T15:27:23.650768+0000","last_became_active":"2024-02-19T15:27:22.051073+0000","last_became_peered":"2024-02-19T15:27:22.051073+0000","last_unstale":"2024-02-19T15:27:23.650768+0000","last_undegraded":"2024-02-19T15:27:23.650768+0000","last_fullsized":"2024-02-19T15:27:23.650768+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:16:39.463746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650232+0000","last_change":"2024-02-19T15:27:22.054976+0000","last_active":"2024-02-19T15:27:23.650232+0000","last_peered":"2024-02-19T15:27:23.650232+0000","last_clean":"2024-02-19T15:27:23.650232+0000","last_became_active":"2024-02-19T15:27:22.054427+0000","last_became_peered":"2024-02-19T15:27:22.054427+0000","last_unstale":"2024-02-19T15:27:23.650232+0000","last_undegraded":"2024-02-19T15:27:23.650232+0000","last_fullsized":"2024-02-19T15:27:23.650232+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:58:10.943380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649945+0000","last_change":"2024-02-19T15:27:22.046799+0000","last_active":"2024-02-19T15:27:23.649945+0000","last_peered":"2024-02-19T15:27:23.649945+0000","last_clean":"2024-02-19T15:27:23.649945+0000","last_became_active":"2024-02-19T15:27:22.046301+0000","last_became_peered":"2024-02-19T15:27:22.046301+0000","last_unstale":"2024-02-19T15:27:23.649945+0000","last_undegraded":"2024-02-19T15:27:23.649945+0000","last_fullsized":"2024-02-19T15:27:23.649945+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:58:18.347673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045487+0000","last_change":"2024-02-19T15:27:22.060116+0000","last_active":"2024-02-19T15:27:23.045487+0000","last_peered":"2024-02-19T15:27:23.045487+0000","last_clean":"2024-02-19T15:27:23.045487+0000","last_became_active":"2024-02-19T15:27:22.059885+0000","last_became_peered":"2024-02-19T15:27:22.059885+0000","last_unstale":"2024-02-19T15:27:23.045487+0000","last_undegraded":"2024-02-19T15:27:23.045487+0000","last_fullsized":"2024-02-19T15:27:23.045487+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:33:49.374645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650143+0000","last_change":"2024-02-19T15:27:22.057325+0000","last_active":"2024-02-19T15:27:23.650143+0000","last_peered":"2024-02-19T15:27:23.650143+0000","last_clean":"2024-02-19T15:27:23.650143+0000","last_became_active":"2024-02-19T15:27:22.050946+0000","last_became_peered":"2024-02-19T15:27:22.050946+0000","last_unstale":"2024-02-19T15:27:23.650143+0000","last_undegraded":"2024-02-19T15:27:23.650143+0000","last_fullsized":"2024-02-19T15:27:23.650143+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:03:35.602958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045349+0000","last_change":"2024-02-19T15:27:22.041551+0000","last_active":"2024-02-19T15:27:23.045349+0000","last_peered":"2024-02-19T15:27:23.045349+0000","last_clean":"2024-02-19T15:27:23.045349+0000","last_became_active":"2024-02-19T15:27:22.040360+0000","last_became_peered":"2024-02-19T15:27:22.040360+0000","last_unstale":"2024-02-19T15:27:23.045349+0000","last_undegraded":"2024-02-19T15:27:23.045349+0000","last_fullsized":"2024-02-19T15:27:23.045349+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:59:17.128798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.046751+0000","last_change":"2024-02-19T15:27:22.036471+0000","last_active":"2024-02-19T15:27:23.046751+0000","last_peered":"2024-02-19T15:27:23.046751+0000","last_clean":"2024-02-19T15:27:23.046751+0000","last_became_active":"2024-02-19T15:27:22.036201+0000","last_became_peered":"2024-02-19T15:27:22.036201+0000","last_unstale":"2024-02-19T15:27:23.046751+0000","last_undegraded":"2024-02-19T15:27:23.046751+0000","last_fullsized":"2024-02-19T15:27:23.046751+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:09:50.065235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650241+0000","last_change":"2024-02-19T15:27:22.055260+0000","last_active":"2024-02-19T15:27:23.650241+0000","last_peered":"2024-02-19T15:27:23.650241+0000","last_clean":"2024-02-19T15:27:23.650241+0000","last_became_active":"2024-02-19T15:27:22.054795+0000","last_became_peered":"2024-02-19T15:27:22.054795+0000","last_unstale":"2024-02-19T15:27:23.650241+0000","last_undegraded":"2024-02-19T15:27:23.650241+0000","last_fullsized":"2024-02-19T15:27:23.650241+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:24:06.367087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.047110+0000","last_change":"2024-02-19T15:27:22.048745+0000","last_active":"2024-02-19T15:27:23.047110+0000","last_peered":"2024-02-19T15:27:23.047110+0000","last_clean":"2024-02-19T15:27:23.047110+0000","last_became_active":"2024-02-19T15:27:22.043577+0000","last_became_peered":"2024-02-19T15:27:22.043577+0000","last_unstale":"2024-02-19T15:27:23.047110+0000","last_undegraded":"2024-02-19T15:27:23.047110+0000","last_fullsized":"2024-02-19T15:27:23.047110+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:58.251712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045286+0000","last_change":"2024-02-19T15:27:22.048453+0000","last_active":"2024-02-19T15:27:23.045286+0000","last_peered":"2024-02-19T15:27:23.045286+0000","last_clean":"2024-02-19T15:27:23.045286+0000","last_became_active":"2024-02-19T15:27:22.046650+0000","last_became_peered":"2024-02-19T15:27:22.046650+0000","last_unstale":"2024-02-19T15:27:23.045286+0000","last_undegraded":"2024-02-19T15:27:23.045286+0000","last_fullsized":"2024-02-19T15:27:23.045286+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:15:02.154641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650028+0000","last_change":"2024-02-19T15:27:22.048023+0000","last_active":"2024-02-19T15:27:23.650028+0000","last_peered":"2024-02-19T15:27:23.650028+0000","last_clean":"2024-02-19T15:27:23.650028+0000","last_became_active":"2024-02-19T15:27:22.047669+0000","last_became_peered":"2024-02-19T15:27:22.047669+0000","last_unstale":"2024-02-19T15:27:23.650028+0000","last_undegraded":"2024-02-19T15:27:23.650028+0000","last_fullsized":"2024-02-19T15:27:23.650028+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:25:25.436463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045036+0000","last_change":"2024-02-19T15:27:22.052131+0000","last_active":"2024-02-19T15:27:23.045036+0000","last_peered":"2024-02-19T15:27:23.045036+0000","last_clean":"2024-02-19T15:27:23.045036+0000","last_became_active":"2024-02-19T15:27:22.051851+0000","last_became_peered":"2024-02-19T15:27:22.051851+0000","last_unstale":"2024-02-19T15:27:23.045036+0000","last_undegraded":"2024-02-19T15:27:23.045036+0000","last_fullsized":"2024-02-19T15:27:23.045036+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:27:08.938061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650176+0000","last_change":"2024-02-19T15:27:22.056052+0000","last_active":"2024-02-19T15:27:23.650176+0000","last_peered":"2024-02-19T15:27:23.650176+0000","last_clean":"2024-02-19T15:27:23.650176+0000","last_became_active":"2024-02-19T15:27:22.054103+0000","last_became_peered":"2024-02-19T15:27:22.054103+0000","last_unstale":"2024-02-19T15:27:23.650176+0000","last_undegraded":"2024-02-19T15:27:23.650176+0000","last_fullsized":"2024-02-19T15:27:23.650176+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:58:10.943380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045393+0000","last_change":"2024-02-19T15:27:22.041777+0000","last_active":"2024-02-19T15:27:23.045393+0000","last_peered":"2024-02-19T15:27:23.045393+0000","last_clean":"2024-02-19T15:27:23.045393+0000","last_became_active":"2024-02-19T15:27:22.040747+0000","last_became_peered":"2024-02-19T15:27:22.040747+0000","last_unstale":"2024-02-19T15:27:23.045393+0000","last_undegraded":"2024-02-19T15:27:23.045393+0000","last_fullsized":"2024-02-19T15:27:23.045393+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:08:48.016401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651670+0000","last_change":"2024-02-19T15:27:22.045024+0000","last_active":"2024-02-19T15:27:23.651670+0000","last_peered":"2024-02-19T15:27:23.651670+0000","last_clean":"2024-02-19T15:27:23.651670+0000","last_became_active":"2024-02-19T15:27:22.044758+0000","last_became_peered":"2024-02-19T15:27:22.044758+0000","last_unstale":"2024-02-19T15:27:23.651670+0000","last_undegraded":"2024-02-19T15:27:23.651670+0000","last_fullsized":"2024-02-19T15:27:23.651670+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:52:21.606372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.046021+0000","last_change":"2024-02-19T15:27:22.062389+0000","last_active":"2024-02-19T15:27:23.046021+0000","last_peered":"2024-02-19T15:27:23.046021+0000","last_clean":"2024-02-19T15:27:23.046021+0000","last_became_active":"2024-02-19T15:27:22.058599+0000","last_became_peered":"2024-02-19T15:27:22.058599+0000","last_unstale":"2024-02-19T15:27:23.046021+0000","last_undegraded":"2024-02-19T15:27:23.046021+0000","last_fullsized":"2024-02-19T15:27:23.046021+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:27:13.600503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649959+0000","last_change":"2024-02-19T15:27:22.060877+0000","last_active":"2024-02-19T15:27:23.649959+0000","last_peered":"2024-02-19T15:27:23.649959+0000","last_clean":"2024-02-19T15:27:23.649959+0000","last_became_active":"2024-02-19T15:27:22.060529+0000","last_became_peered":"2024-02-19T15:27:22.060529+0000","last_unstale":"2024-02-19T15:27:23.649959+0000","last_undegraded":"2024-02-19T15:27:23.649959+0000","last_fullsized":"2024-02-19T15:27:23.649959+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:58:18.347673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045474+0000","last_change":"2024-02-19T15:27:22.043723+0000","last_active":"2024-02-19T15:27:23.045474+0000","last_peered":"2024-02-19T15:27:23.045474+0000","last_clean":"2024-02-19T15:27:23.045474+0000","last_became_active":"2024-02-19T15:27:22.043463+0000","last_became_peered":"2024-02-19T15:27:22.043463+0000","last_unstale":"2024-02-19T15:27:23.045474+0000","last_undegraded":"2024-02-19T15:27:23.045474+0000","last_fullsized":"2024-02-19T15:27:23.045474+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:52.297989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650158+0000","last_change":"2024-02-19T15:27:22.045324+0000","last_active":"2024-02-19T15:27:23.650158+0000","last_peered":"2024-02-19T15:27:23.650158+0000","last_clean":"2024-02-19T15:27:23.650158+0000","last_became_active":"2024-02-19T15:27:22.044981+0000","last_became_peered":"2024-02-19T15:27:22.044981+0000","last_unstale":"2024-02-19T15:27:23.650158+0000","last_undegraded":"2024-02-19T15:27:23.650158+0000","last_fullsized":"2024-02-19T15:27:23.650158+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:27:13.600503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649846+0000","last_change":"2024-02-19T15:27:22.045853+0000","last_active":"2024-02-19T15:27:23.649846+0000","last_peered":"2024-02-19T15:27:23.649846+0000","last_clean":"2024-02-19T15:27:23.649846+0000","last_became_active":"2024-02-19T15:27:22.045474+0000","last_became_peered":"2024-02-19T15:27:22.045474+0000","last_unstale":"2024-02-19T15:27:23.649846+0000","last_undegraded":"2024-02-19T15:27:23.649846+0000","last_fullsized":"2024-02-19T15:27:23.649846+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:21:04.976920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045408+0000","last_change":"2024-02-19T15:27:22.055671+0000","last_active":"2024-02-19T15:27:23.045408+0000","last_peered":"2024-02-19T15:27:23.045408+0000","last_clean":"2024-02-19T15:27:23.045408+0000","last_became_active":"2024-02-19T15:27:22.055359+0000","last_became_peered":"2024-02-19T15:27:22.055359+0000","last_unstale":"2024-02-19T15:27:23.045408+0000","last_undegraded":"2024-02-19T15:27:23.045408+0000","last_fullsized":"2024-02-19T15:27:23.045408+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:57:40.939085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.046102+0000","last_change":"2024-02-19T15:27:22.059901+0000","last_active":"2024-02-19T15:27:23.046102+0000","last_peered":"2024-02-19T15:27:23.046102+0000","last_clean":"2024-02-19T15:27:23.046102+0000","last_became_active":"2024-02-19T15:27:22.059581+0000","last_became_peered":"2024-02-19T15:27:22.059581+0000","last_unstale":"2024-02-19T15:27:23.046102+0000","last_undegraded":"2024-02-19T15:27:23.046102+0000","last_fullsized":"2024-02-19T15:27:23.046102+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:40:09.571306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"107'4","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650737+0000","last_change":"2024-02-19T15:27:21.030063+0000","last_active":"2024-02-19T15:27:23.650737+0000","last_peered":"2024-02-19T15:27:23.650737+0000","last_clean":"2024-02-19T15:27:23.650737+0000","last_became_active":"2024-02-19T15:27:21.029780+0000","last_became_peered":"2024-02-19T15:27:21.029780+0000","last_unstale":"2024-02-19T15:27:23.650737+0000","last_undegraded":"2024-02-19T15:27:23.650737+0000","last_fullsized":"2024-02-19T15:27:23.650737+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:44:41.044290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.047302+0000","last_change":"2024-02-19T15:27:22.048726+0000","last_active":"2024-02-19T15:27:23.047302+0000","last_peered":"2024-02-19T15:27:23.047302+0000","last_clean":"2024-02-19T15:27:23.047302+0000","last_became_active":"2024-02-19T15:27:22.048437+0000","last_became_peered":"2024-02-19T15:27:22.048437+0000","last_unstale":"2024-02-19T15:27:23.047302+0000","last_undegraded":"2024-02-19T15:27:23.047302+0000","last_fullsized":"2024-02-19T15:27:23.047302+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:13:20.145608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650367+0000","last_change":"2024-02-19T15:27:21.027135+0000","last_active":"2024-02-19T15:27:23.650367+0000","last_peered":"2024-02-19T15:27:23.650367+0000","last_clean":"2024-02-19T15:27:23.650367+0000","last_became_active":"2024-02-19T15:27:21.026870+0000","last_became_peered":"2024-02-19T15:27:21.026870+0000","last_unstale":"2024-02-19T15:27:23.650367+0000","last_undegraded":"2024-02-19T15:27:23.650367+0000","last_fullsized":"2024-02-19T15:27:23.650367+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:54.672402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045992+0000","last_change":"2024-02-19T15:27:22.059124+0000","last_active":"2024-02-19T15:27:23.045992+0000","last_peered":"2024-02-19T15:27:23.045992+0000","last_clean":"2024-02-19T15:27:23.045992+0000","last_became_active":"2024-02-19T15:27:22.057067+0000","last_became_peered":"2024-02-19T15:27:22.057067+0000","last_unstale":"2024-02-19T15:27:23.045992+0000","last_undegraded":"2024-02-19T15:27:23.045992+0000","last_fullsized":"2024-02-19T15:27:23.045992+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:45:36.102577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651208+0000","last_change":"2024-02-19T15:27:21.029872+0000","last_active":"2024-02-19T15:27:23.651208+0000","last_peered":"2024-02-19T15:27:23.651208+0000","last_clean":"2024-02-19T15:27:23.651208+0000","last_became_active":"2024-02-19T15:27:21.029590+0000","last_became_peered":"2024-02-19T15:27:21.029590+0000","last_unstale":"2024-02-19T15:27:23.651208+0000","last_undegraded":"2024-02-19T15:27:23.651208+0000","last_fullsized":"2024-02-19T15:27:23.651208+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:25:03.421064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650459+0000","last_change":"2024-02-19T15:27:22.054621+0000","last_active":"2024-02-19T15:27:23.650459+0000","last_peered":"2024-02-19T15:27:23.650459+0000","last_clean":"2024-02-19T15:27:23.650459+0000","last_became_active":"2024-02-19T15:27:22.054395+0000","last_became_peered":"2024-02-19T15:27:22.054395+0000","last_unstale":"2024-02-19T15:27:23.650459+0000","last_undegraded":"2024-02-19T15:27:23.650459+0000","last_fullsized":"2024-02-19T15:27:23.650459+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:43:10.829112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650470+0000","last_change":"2024-02-19T15:27:21.042870+0000","last_active":"2024-02-19T15:27:23.650470+0000","last_peered":"2024-02-19T15:27:23.650470+0000","last_clean":"2024-02-19T15:27:23.650470+0000","last_became_active":"2024-02-19T15:27:21.041700+0000","last_became_peered":"2024-02-19T15:27:21.041700+0000","last_unstale":"2024-02-19T15:27:23.650470+0000","last_undegraded":"2024-02-19T15:27:23.650470+0000","last_fullsized":"2024-02-19T15:27:23.650470+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:58:39.707731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650691+0000","last_change":"2024-02-19T15:27:22.043230+0000","last_active":"2024-02-19T15:27:23.650691+0000","last_peered":"2024-02-19T15:27:23.650691+0000","last_clean":"2024-02-19T15:27:23.650691+0000","last_became_active":"2024-02-19T15:27:22.042919+0000","last_became_peered":"2024-02-19T15:27:22.042919+0000","last_unstale":"2024-02-19T15:27:23.650691+0000","last_undegraded":"2024-02-19T15:27:23.650691+0000","last_fullsized":"2024-02-19T15:27:23.650691+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:22:19.119378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649731+0000","last_change":"2024-02-19T15:27:21.040738+0000","last_active":"2024-02-19T15:27:23.649731+0000","last_peered":"2024-02-19T15:27:23.649731+0000","last_clean":"2024-02-19T15:27:23.649731+0000","last_became_active":"2024-02-19T15:27:21.040492+0000","last_became_peered":"2024-02-19T15:27:21.040492+0000","last_unstale":"2024-02-19T15:27:23.649731+0000","last_undegraded":"2024-02-19T15:27:23.649731+0000","last_fullsized":"2024-02-19T15:27:23.649731+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:31:23.769771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045610+0000","last_change":"2024-02-19T15:27:22.066931+0000","last_active":"2024-02-19T15:27:23.045610+0000","last_peered":"2024-02-19T15:27:23.045610+0000","last_clean":"2024-02-19T15:27:23.045610+0000","last_became_active":"2024-02-19T15:27:22.066614+0000","last_became_peered":"2024-02-19T15:27:22.066614+0000","last_unstale":"2024-02-19T15:27:23.045610+0000","last_undegraded":"2024-02-19T15:27:23.045610+0000","last_fullsized":"2024-02-19T15:27:23.045610+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:55.678215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650275+0000","last_change":"2024-02-19T15:27:21.037989+0000","last_active":"2024-02-19T15:27:23.650275+0000","last_peered":"2024-02-19T15:27:23.650275+0000","last_clean":"2024-02-19T15:27:23.650275+0000","last_became_active":"2024-02-19T15:27:21.037527+0000","last_became_peered":"2024-02-19T15:27:21.037527+0000","last_unstale":"2024-02-19T15:27:23.650275+0000","last_undegraded":"2024-02-19T15:27:23.650275+0000","last_fullsized":"2024-02-19T15:27:23.650275+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:08:47.010588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650103+0000","last_change":"2024-02-19T15:27:22.065841+0000","last_active":"2024-02-19T15:27:23.650103+0000","last_peered":"2024-02-19T15:27:23.650103+0000","last_clean":"2024-02-19T15:27:23.650103+0000","last_became_active":"2024-02-19T15:27:22.065572+0000","last_became_peered":"2024-02-19T15:27:22.065572+0000","last_unstale":"2024-02-19T15:27:23.650103+0000","last_undegraded":"2024-02-19T15:27:23.650103+0000","last_fullsized":"2024-02-19T15:27:23.650103+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:51:18.750175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.044991+0000","last_change":"2024-02-19T15:27:21.031349+0000","last_active":"2024-02-19T15:27:23.044991+0000","last_peered":"2024-02-19T15:27:23.044991+0000","last_clean":"2024-02-19T15:27:23.044991+0000","last_became_active":"2024-02-19T15:27:21.031075+0000","last_became_peered":"2024-02-19T15:27:21.031075+0000","last_unstale":"2024-02-19T15:27:23.044991+0000","last_undegraded":"2024-02-19T15:27:23.044991+0000","last_fullsized":"2024-02-19T15:27:23.044991+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:29:03.843739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649714+0000","last_change":"2024-02-19T15:27:22.042301+0000","last_active":"2024-02-19T15:27:23.649714+0000","last_peered":"2024-02-19T15:27:23.649714+0000","last_clean":"2024-02-19T15:27:23.649714+0000","last_became_active":"2024-02-19T15:27:22.042015+0000","last_became_peered":"2024-02-19T15:27:22.042015+0000","last_unstale":"2024-02-19T15:27:23.649714+0000","last_undegraded":"2024-02-19T15:27:23.649714+0000","last_fullsized":"2024-02-19T15:27:23.649714+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:59:15.699509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650207+0000","last_change":"2024-02-19T15:27:21.025268+0000","last_active":"2024-02-19T15:27:23.650207+0000","last_peered":"2024-02-19T15:27:23.650207+0000","last_clean":"2024-02-19T15:27:23.650207+0000","last_became_active":"2024-02-19T15:27:21.022655+0000","last_became_peered":"2024-02-19T15:27:21.022655+0000","last_unstale":"2024-02-19T15:27:23.650207+0000","last_undegraded":"2024-02-19T15:27:23.650207+0000","last_fullsized":"2024-02-19T15:27:23.650207+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:09.503853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650580+0000","last_change":"2024-02-19T15:27:22.050190+0000","last_active":"2024-02-19T15:27:23.650580+0000","last_peered":"2024-02-19T15:27:23.650580+0000","last_clean":"2024-02-19T15:27:23.650580+0000","last_became_active":"2024-02-19T15:27:22.049824+0000","last_became_peered":"2024-02-19T15:27:22.049824+0000","last_unstale":"2024-02-19T15:27:23.650580+0000","last_undegraded":"2024-02-19T15:27:23.650580+0000","last_fullsized":"2024-02-19T15:27:23.650580+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:40:09.571306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650204+0000","last_change":"2024-02-19T15:27:21.041198+0000","last_active":"2024-02-19T15:27:23.650204+0000","last_peered":"2024-02-19T15:27:23.650204+0000","last_clean":"2024-02-19T15:27:23.650204+0000","last_became_active":"2024-02-19T15:27:21.040821+0000","last_became_peered":"2024-02-19T15:27:21.040821+0000","last_unstale":"2024-02-19T15:27:23.650204+0000","last_undegraded":"2024-02-19T15:27:23.650204+0000","last_fullsized":"2024-02-19T15:27:23.650204+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:15:13.215563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.047316+0000","last_change":"2024-02-19T15:27:22.043233+0000","last_active":"2024-02-19T15:27:23.047316+0000","last_peered":"2024-02-19T15:27:23.047316+0000","last_clean":"2024-02-19T15:27:23.047316+0000","last_became_active":"2024-02-19T15:27:22.042812+0000","last_became_peered":"2024-02-19T15:27:22.042812+0000","last_unstale":"2024-02-19T15:27:23.047316+0000","last_undegraded":"2024-02-19T15:27:23.047316+0000","last_fullsized":"2024-02-19T15:27:23.047316+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:52.686803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651530+0000","last_change":"2024-02-19T15:27:21.036912+0000","last_active":"2024-02-19T15:27:23.651530+0000","last_peered":"2024-02-19T15:27:23.651530+0000","last_clean":"2024-02-19T15:27:23.651530+0000","last_became_active":"2024-02-19T15:27:21.036227+0000","last_became_peered":"2024-02-19T15:27:21.036227+0000","last_unstale":"2024-02-19T15:27:23.651530+0000","last_undegraded":"2024-02-19T15:27:23.651530+0000","last_fullsized":"2024-02-19T15:27:23.651530+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:39:11.138544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651228+0000","last_change":"2024-02-19T15:27:22.054682+0000","last_active":"2024-02-19T15:27:23.651228+0000","last_peered":"2024-02-19T15:27:23.651228+0000","last_clean":"2024-02-19T15:27:23.651228+0000","last_became_active":"2024-02-19T15:27:22.054457+0000","last_became_peered":"2024-02-19T15:27:22.054457+0000","last_unstale":"2024-02-19T15:27:23.651228+0000","last_undegraded":"2024-02-19T15:27:23.651228+0000","last_fullsized":"2024-02-19T15:27:23.651228+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:10:38.351264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"107'3","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651245+0000","last_change":"2024-02-19T15:27:21.022250+0000","last_active":"2024-02-19T15:27:23.651245+0000","last_peered":"2024-02-19T15:27:23.651245+0000","last_clean":"2024-02-19T15:27:23.651245+0000","last_became_active":"2024-02-19T15:27:21.021690+0000","last_became_peered":"2024-02-19T15:27:21.021690+0000","last_unstale":"2024-02-19T15:27:23.651245+0000","last_undegraded":"2024-02-19T15:27:23.651245+0000","last_fullsized":"2024-02-19T15:27:23.651245+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:21:10.257063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650229+0000","last_change":"2024-02-19T15:27:22.045625+0000","last_active":"2024-02-19T15:27:23.650229+0000","last_peered":"2024-02-19T15:27:23.650229+0000","last_clean":"2024-02-19T15:27:23.650229+0000","last_became_active":"2024-02-19T15:27:22.045360+0000","last_became_peered":"2024-02-19T15:27:22.045360+0000","last_unstale":"2024-02-19T15:27:23.650229+0000","last_undegraded":"2024-02-19T15:27:23.650229+0000","last_fullsized":"2024-02-19T15:27:23.650229+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:26:01.935420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"107'6","reported_seq":20,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651108+0000","last_change":"2024-02-19T15:27:21.031559+0000","last_active":"2024-02-19T15:27:23.651108+0000","last_peered":"2024-02-19T15:27:23.651108+0000","last_clean":"2024-02-19T15:27:23.651108+0000","last_became_active":"2024-02-19T15:27:21.031280+0000","last_became_peered":"2024-02-19T15:27:21.031280+0000","last_unstale":"2024-02-19T15:27:23.651108+0000","last_undegraded":"2024-02-19T15:27:23.651108+0000","last_fullsized":"2024-02-19T15:27:23.651108+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:36:18.076454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650180+0000","last_change":"2024-02-19T15:27:22.044977+0000","last_active":"2024-02-19T15:27:23.650180+0000","last_peered":"2024-02-19T15:27:23.650180+0000","last_clean":"2024-02-19T15:27:23.650180+0000","last_became_active":"2024-02-19T15:27:22.044719+0000","last_became_peered":"2024-02-19T15:27:22.044719+0000","last_unstale":"2024-02-19T15:27:23.650180+0000","last_undegraded":"2024-02-19T15:27:23.650180+0000","last_fullsized":"2024-02-19T15:27:23.650180+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:24:06.367087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045792+0000","last_change":"2024-02-19T15:27:21.019044+0000","last_active":"2024-02-19T15:27:23.045792+0000","last_peered":"2024-02-19T15:27:23.045792+0000","last_clean":"2024-02-19T15:27:23.045792+0000","last_became_active":"2024-02-19T15:27:21.018732+0000","last_became_peered":"2024-02-19T15:27:21.018732+0000","last_unstale":"2024-02-19T15:27:23.045792+0000","last_undegraded":"2024-02-19T15:27:23.045792+0000","last_fullsized":"2024-02-19T15:27:23.045792+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:51:06.106358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650178+0000","last_change":"2024-02-19T15:27:22.052449+0000","last_active":"2024-02-19T15:27:23.650178+0000","last_peered":"2024-02-19T15:27:23.650178+0000","last_clean":"2024-02-19T15:27:23.650178+0000","last_became_active":"2024-02-19T15:27:22.051990+0000","last_became_peered":"2024-02-19T15:27:22.051990+0000","last_unstale":"2024-02-19T15:27:23.650178+0000","last_undegraded":"2024-02-19T15:27:23.650178+0000","last_fullsized":"2024-02-19T15:27:23.650178+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:57:49.784754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.046544+0000","last_change":"2024-02-19T15:27:21.022352+0000","last_active":"2024-02-19T15:27:23.046544+0000","last_peered":"2024-02-19T15:27:23.046544+0000","last_clean":"2024-02-19T15:27:23.046544+0000","last_became_active":"2024-02-19T15:27:21.021353+0000","last_became_peered":"2024-02-19T15:27:21.021353+0000","last_unstale":"2024-02-19T15:27:23.046544+0000","last_undegraded":"2024-02-19T15:27:23.046544+0000","last_fullsized":"2024-02-19T15:27:23.046544+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:05:11.130486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649860+0000","last_change":"2024-02-19T15:27:22.046349+0000","last_active":"2024-02-19T15:27:23.649860+0000","last_peered":"2024-02-19T15:27:23.649860+0000","last_clean":"2024-02-19T15:27:23.649860+0000","last_became_active":"2024-02-19T15:27:22.045913+0000","last_became_peered":"2024-02-19T15:27:22.045913+0000","last_unstale":"2024-02-19T15:27:23.649860+0000","last_undegraded":"2024-02-19T15:27:23.649860+0000","last_fullsized":"2024-02-19T15:27:23.649860+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:34.686973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"107'8","reported_seq":22,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650398+0000","last_change":"2024-02-19T15:27:21.042396+0000","last_active":"2024-02-19T15:27:23.650398+0000","last_peered":"2024-02-19T15:27:23.650398+0000","last_clean":"2024-02-19T15:27:23.650398+0000","last_became_active":"2024-02-19T15:27:21.042017+0000","last_became_peered":"2024-02-19T15:27:21.042017+0000","last_unstale":"2024-02-19T15:27:23.650398+0000","last_undegraded":"2024-02-19T15:27:23.650398+0000","last_fullsized":"2024-02-19T15:27:23.650398+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:18:27.189283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650690+0000","last_change":"2024-02-19T15:27:22.038153+0000","last_active":"2024-02-19T15:27:23.650690+0000","last_peered":"2024-02-19T15:27:23.650690+0000","last_clean":"2024-02-19T15:27:23.650690+0000","last_became_active":"2024-02-19T15:27:22.037885+0000","last_became_peered":"2024-02-19T15:27:22.037885+0000","last_unstale":"2024-02-19T15:27:23.650690+0000","last_undegraded":"2024-02-19T15:27:23.650690+0000","last_fullsized":"2024-02-19T15:27:23.650690+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:16.020376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651868+0000","last_change":"2024-02-19T15:27:21.027023+0000","last_active":"2024-02-19T15:27:23.651868+0000","last_peered":"2024-02-19T15:27:23.651868+0000","last_clean":"2024-02-19T15:27:23.651868+0000","last_became_active":"2024-02-19T15:27:21.026565+0000","last_became_peered":"2024-02-19T15:27:21.026565+0000","last_unstale":"2024-02-19T15:27:23.651868+0000","last_undegraded":"2024-02-19T15:27:23.651868+0000","last_fullsized":"2024-02-19T15:27:23.651868+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:04:34.220187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.046040+0000","last_change":"2024-02-19T15:27:22.058919+0000","last_active":"2024-02-19T15:27:23.046040+0000","last_peered":"2024-02-19T15:27:23.046040+0000","last_clean":"2024-02-19T15:27:23.046040+0000","last_became_active":"2024-02-19T15:27:22.058619+0000","last_became_peered":"2024-02-19T15:27:22.058619+0000","last_unstale":"2024-02-19T15:27:23.046040+0000","last_undegraded":"2024-02-19T15:27:23.046040+0000","last_fullsized":"2024-02-19T15:27:23.046040+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:26:01.935420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650785+0000","last_change":"2024-02-19T15:27:21.033163+0000","last_active":"2024-02-19T15:27:23.650785+0000","last_peered":"2024-02-19T15:27:23.650785+0000","last_clean":"2024-02-19T15:27:23.650785+0000","last_became_active":"2024-02-19T15:27:21.032778+0000","last_became_peered":"2024-02-19T15:27:21.032778+0000","last_unstale":"2024-02-19T15:27:23.650785+0000","last_undegraded":"2024-02-19T15:27:23.650785+0000","last_fullsized":"2024-02-19T15:27:23.650785+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:22:19.544810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650504+0000","last_change":"2024-02-19T15:27:22.050672+0000","last_active":"2024-02-19T15:27:23.650504+0000","last_peered":"2024-02-19T15:27:23.650504+0000","last_clean":"2024-02-19T15:27:23.650504+0000","last_became_active":"2024-02-19T15:27:22.050405+0000","last_became_peered":"2024-02-19T15:27:22.050405+0000","last_unstale":"2024-02-19T15:27:23.650504+0000","last_undegraded":"2024-02-19T15:27:23.650504+0000","last_fullsized":"2024-02-19T15:27:23.650504+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:20:30.719772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.046667+0000","last_change":"2024-02-19T15:27:21.033698+0000","last_active":"2024-02-19T15:27:23.046667+0000","last_peered":"2024-02-19T15:27:23.046667+0000","last_clean":"2024-02-19T15:27:23.046667+0000","last_became_active":"2024-02-19T15:27:21.033354+0000","last_became_peered":"2024-02-19T15:27:21.033354+0000","last_unstale":"2024-02-19T15:27:23.046667+0000","last_undegraded":"2024-02-19T15:27:23.046667+0000","last_fullsized":"2024-02-19T15:27:23.046667+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:42.028837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650634+0000","last_change":"2024-02-19T15:27:22.049020+0000","last_active":"2024-02-19T15:27:23.650634+0000","last_peered":"2024-02-19T15:27:23.650634+0000","last_clean":"2024-02-19T15:27:23.650634+0000","last_became_active":"2024-02-19T15:27:22.048713+0000","last_became_peered":"2024-02-19T15:27:22.048713+0000","last_unstale":"2024-02-19T15:27:23.650634+0000","last_undegraded":"2024-02-19T15:27:23.650634+0000","last_fullsized":"2024-02-19T15:27:23.650634+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:59:30.423613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650092+0000","last_change":"2024-02-19T15:27:21.031371+0000","last_active":"2024-02-19T15:27:23.650092+0000","last_peered":"2024-02-19T15:27:23.650092+0000","last_clean":"2024-02-19T15:27:23.650092+0000","last_became_active":"2024-02-19T15:27:21.031066+0000","last_became_peered":"2024-02-19T15:27:21.031066+0000","last_unstale":"2024-02-19T15:27:23.650092+0000","last_undegraded":"2024-02-19T15:27:23.650092+0000","last_fullsized":"2024-02-19T15:27:23.650092+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:03:18.300707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650512+0000","last_change":"2024-02-19T15:27:22.048519+0000","last_active":"2024-02-19T15:27:23.650512+0000","last_peered":"2024-02-19T15:27:23.650512+0000","last_clean":"2024-02-19T15:27:23.650512+0000","last_became_active":"2024-02-19T15:27:22.048290+0000","last_became_peered":"2024-02-19T15:27:22.048290+0000","last_unstale":"2024-02-19T15:27:23.650512+0000","last_undegraded":"2024-02-19T15:27:23.650512+0000","last_fullsized":"2024-02-19T15:27:23.650512+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:07:37.708947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045798+0000","last_change":"2024-02-19T15:27:21.026803+0000","last_active":"2024-02-19T15:27:23.045798+0000","last_peered":"2024-02-19T15:27:23.045798+0000","last_clean":"2024-02-19T15:27:23.045798+0000","last_became_active":"2024-02-19T15:27:21.025817+0000","last_became_peered":"2024-02-19T15:27:21.025817+0000","last_unstale":"2024-02-19T15:27:23.045798+0000","last_undegraded":"2024-02-19T15:27:23.045798+0000","last_fullsized":"2024-02-19T15:27:23.045798+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:31:23.769771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650975+0000","last_change":"2024-02-19T15:27:22.058638+0000","last_active":"2024-02-19T15:27:23.650975+0000","last_peered":"2024-02-19T15:27:23.650975+0000","last_clean":"2024-02-19T15:27:23.650975+0000","last_became_active":"2024-02-19T15:27:22.057438+0000","last_became_peered":"2024-02-19T15:27:22.057438+0000","last_unstale":"2024-02-19T15:27:23.650975+0000","last_undegraded":"2024-02-19T15:27:23.650975+0000","last_fullsized":"2024-02-19T15:27:23.650975+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:42:11.448198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649997+0000","last_change":"2024-02-19T15:27:21.042086+0000","last_active":"2024-02-19T15:27:23.649997+0000","last_peered":"2024-02-19T15:27:23.649997+0000","last_clean":"2024-02-19T15:27:23.649997+0000","last_became_active":"2024-02-19T15:27:21.041580+0000","last_became_peered":"2024-02-19T15:27:21.041580+0000","last_unstale":"2024-02-19T15:27:23.649997+0000","last_undegraded":"2024-02-19T15:27:23.649997+0000","last_fullsized":"2024-02-19T15:27:23.649997+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:30:43.579542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650400+0000","last_change":"2024-02-19T15:27:22.052442+0000","last_active":"2024-02-19T15:27:23.650400+0000","last_peered":"2024-02-19T15:27:23.650400+0000","last_clean":"2024-02-19T15:27:23.650400+0000","last_became_active":"2024-02-19T15:27:22.051952+0000","last_became_peered":"2024-02-19T15:27:22.051952+0000","last_unstale":"2024-02-19T15:27:23.650400+0000","last_undegraded":"2024-02-19T15:27:23.650400+0000","last_fullsized":"2024-02-19T15:27:23.650400+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:10:38.351264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045448+0000","last_change":"2024-02-19T15:27:21.038892+0000","last_active":"2024-02-19T15:27:23.045448+0000","last_peered":"2024-02-19T15:27:23.045448+0000","last_clean":"2024-02-19T15:27:23.045448+0000","last_became_active":"2024-02-19T15:27:21.038595+0000","last_became_peered":"2024-02-19T15:27:21.038595+0000","last_unstale":"2024-02-19T15:27:23.045448+0000","last_undegraded":"2024-02-19T15:27:23.045448+0000","last_fullsized":"2024-02-19T15:27:23.045448+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:25:30.120547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651487+0000","last_change":"2024-02-19T15:27:22.046134+0000","last_active":"2024-02-19T15:27:23.651487+0000","last_peered":"2024-02-19T15:27:23.651487+0000","last_clean":"2024-02-19T15:27:23.651487+0000","last_became_active":"2024-02-19T15:27:22.045608+0000","last_became_peered":"2024-02-19T15:27:22.045608+0000","last_unstale":"2024-02-19T15:27:23.651487+0000","last_undegraded":"2024-02-19T15:27:23.651487+0000","last_fullsized":"2024-02-19T15:27:23.651487+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:43:10.829112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650299+0000","last_change":"2024-02-19T15:27:21.017652+0000","last_active":"2024-02-19T15:27:23.650299+0000","last_peered":"2024-02-19T15:27:23.650299+0000","last_clean":"2024-02-19T15:27:23.650299+0000","last_became_active":"2024-02-19T15:27:21.017345+0000","last_became_peered":"2024-02-19T15:27:21.017345+0000","last_unstale":"2024-02-19T15:27:23.650299+0000","last_undegraded":"2024-02-19T15:27:23.650299+0000","last_fullsized":"2024-02-19T15:27:23.650299+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:43:48.389820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.046871+0000","last_change":"2024-02-19T15:27:22.035845+0000","last_active":"2024-02-19T15:27:23.046871+0000","last_peered":"2024-02-19T15:27:23.046871+0000","last_clean":"2024-02-19T15:27:23.046871+0000","last_became_active":"2024-02-19T15:27:22.035521+0000","last_became_peered":"2024-02-19T15:27:22.035521+0000","last_unstale":"2024-02-19T15:27:23.046871+0000","last_undegraded":"2024-02-19T15:27:23.046871+0000","last_fullsized":"2024-02-19T15:27:23.046871+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:16:07.683195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651663+0000","last_change":"2024-02-19T15:27:21.034876+0000","last_active":"2024-02-19T15:27:23.651663+0000","last_peered":"2024-02-19T15:27:23.651663+0000","last_clean":"2024-02-19T15:27:23.651663+0000","last_became_active":"2024-02-19T15:27:21.032593+0000","last_became_peered":"2024-02-19T15:27:21.032593+0000","last_unstale":"2024-02-19T15:27:23.651663+0000","last_undegraded":"2024-02-19T15:27:23.651663+0000","last_fullsized":"2024-02-19T15:27:23.651663+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:48:01.413719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651710+0000","last_change":"2024-02-19T15:27:22.047285+0000","last_active":"2024-02-19T15:27:23.651710+0000","last_peered":"2024-02-19T15:27:23.651710+0000","last_clean":"2024-02-19T15:27:23.651710+0000","last_became_active":"2024-02-19T15:27:22.046953+0000","last_became_peered":"2024-02-19T15:27:22.046953+0000","last_unstale":"2024-02-19T15:27:23.651710+0000","last_undegraded":"2024-02-19T15:27:23.651710+0000","last_fullsized":"2024-02-19T15:27:23.651710+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:47:04.051444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"108'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.046953+0000","last_change":"2024-02-19T15:27:21.025533+0000","last_active":"2024-02-19T15:27:23.046953+0000","last_peered":"2024-02-19T15:27:23.046953+0000","last_clean":"2024-02-19T15:27:23.046953+0000","last_became_active":"2024-02-19T15:27:21.025161+0000","last_became_peered":"2024-02-19T15:27:21.025161+0000","last_unstale":"2024-02-19T15:27:23.046953+0000","last_undegraded":"2024-02-19T15:27:23.046953+0000","last_fullsized":"2024-02-19T15:27:23.046953+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:59:09.129286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650976+0000","last_change":"2024-02-19T15:27:22.066060+0000","last_active":"2024-02-19T15:27:23.650976+0000","last_peered":"2024-02-19T15:27:23.650976+0000","last_clean":"2024-02-19T15:27:23.650976+0000","last_became_active":"2024-02-19T15:27:22.065704+0000","last_became_peered":"2024-02-19T15:27:22.065704+0000","last_unstale":"2024-02-19T15:27:23.650976+0000","last_undegraded":"2024-02-19T15:27:23.650976+0000","last_fullsized":"2024-02-19T15:27:23.650976+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:39:24.288533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651282+0000","last_change":"2024-02-19T15:27:21.021965+0000","last_active":"2024-02-19T15:27:23.651282+0000","last_peered":"2024-02-19T15:27:23.651282+0000","last_clean":"2024-02-19T15:27:23.651282+0000","last_became_active":"2024-02-19T15:27:21.021319+0000","last_became_peered":"2024-02-19T15:27:21.021319+0000","last_unstale":"2024-02-19T15:27:23.651282+0000","last_undegraded":"2024-02-19T15:27:23.651282+0000","last_fullsized":"2024-02-19T15:27:23.651282+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:58:28.270645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.046587+0000","last_change":"2024-02-19T15:27:22.037785+0000","last_active":"2024-02-19T15:27:23.046587+0000","last_peered":"2024-02-19T15:27:23.046587+0000","last_clean":"2024-02-19T15:27:23.046587+0000","last_became_active":"2024-02-19T15:27:22.037460+0000","last_became_peered":"2024-02-19T15:27:22.037460+0000","last_unstale":"2024-02-19T15:27:23.046587+0000","last_undegraded":"2024-02-19T15:27:23.046587+0000","last_fullsized":"2024-02-19T15:27:23.046587+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:25:01.649887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.651412+0000","last_change":"2024-02-19T15:27:21.028129+0000","last_active":"2024-02-19T15:27:23.651412+0000","last_peered":"2024-02-19T15:27:23.651412+0000","last_clean":"2024-02-19T15:27:23.651412+0000","last_became_active":"2024-02-19T15:27:21.027822+0000","last_became_peered":"2024-02-19T15:27:21.027822+0000","last_unstale":"2024-02-19T15:27:23.651412+0000","last_undegraded":"2024-02-19T15:27:23.651412+0000","last_fullsized":"2024-02-19T15:27:23.651412+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:51.292175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045206+0000","last_change":"2024-02-19T15:27:22.045423+0000","last_active":"2024-02-19T15:27:23.045206+0000","last_peered":"2024-02-19T15:27:23.045206+0000","last_clean":"2024-02-19T15:27:23.045206+0000","last_became_active":"2024-02-19T15:27:22.045159+0000","last_became_peered":"2024-02-19T15:27:22.045159+0000","last_unstale":"2024-02-19T15:27:23.045206+0000","last_undegraded":"2024-02-19T15:27:23.045206+0000","last_fullsized":"2024-02-19T15:27:23.045206+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:38:40.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649872+0000","last_change":"2024-02-19T15:27:21.032501+0000","last_active":"2024-02-19T15:27:23.649872+0000","last_peered":"2024-02-19T15:27:23.649872+0000","last_clean":"2024-02-19T15:27:23.649872+0000","last_became_active":"2024-02-19T15:27:21.032206+0000","last_became_peered":"2024-02-19T15:27:21.032206+0000","last_unstale":"2024-02-19T15:27:23.649872+0000","last_undegraded":"2024-02-19T15:27:23.649872+0000","last_fullsized":"2024-02-19T15:27:23.649872+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:18:10.944093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045734+0000","last_change":"2024-02-19T15:27:22.051965+0000","last_active":"2024-02-19T15:27:23.045734+0000","last_peered":"2024-02-19T15:27:23.045734+0000","last_clean":"2024-02-19T15:27:23.045734+0000","last_became_active":"2024-02-19T15:27:22.051580+0000","last_became_peered":"2024-02-19T15:27:22.051580+0000","last_unstale":"2024-02-19T15:27:23.045734+0000","last_undegraded":"2024-02-19T15:27:23.045734+0000","last_fullsized":"2024-02-19T15:27:23.045734+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:48:02.419533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.045376+0000","last_change":"2024-02-19T15:27:21.035917+0000","last_active":"2024-02-19T15:27:23.045376+0000","last_peered":"2024-02-19T15:27:23.045376+0000","last_clean":"2024-02-19T15:27:23.045376+0000","last_became_active":"2024-02-19T15:27:21.035628+0000","last_became_peered":"2024-02-19T15:27:21.035628+0000","last_unstale":"2024-02-19T15:27:23.045376+0000","last_undegraded":"2024-02-19T15:27:23.045376+0000","last_fullsized":"2024-02-19T15:27:23.045376+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:08:02.984985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650053+0000","last_change":"2024-02-19T15:27:22.047614+0000","last_active":"2024-02-19T15:27:23.650053+0000","last_peered":"2024-02-19T15:27:23.650053+0000","last_clean":"2024-02-19T15:27:23.650053+0000","last_became_active":"2024-02-19T15:27:22.047282+0000","last_became_peered":"2024-02-19T15:27:22.047282+0000","last_unstale":"2024-02-19T15:27:23.650053+0000","last_undegraded":"2024-02-19T15:27:23.650053+0000","last_fullsized":"2024-02-19T15:27:23.650053+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:57:49.784754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650918+0000","last_change":"2024-02-19T15:27:21.029497+0000","last_active":"2024-02-19T15:27:23.650918+0000","last_peered":"2024-02-19T15:27:23.650918+0000","last_clean":"2024-02-19T15:27:23.650918+0000","last_became_active":"2024-02-19T15:27:21.029251+0000","last_became_peered":"2024-02-19T15:27:21.029251+0000","last_unstale":"2024-02-19T15:27:23.650918+0000","last_undegraded":"2024-02-19T15:27:23.650918+0000","last_fullsized":"2024-02-19T15:27:23.650918+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:11:30.790177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650642+0000","last_change":"2024-02-19T15:27:22.068736+0000","last_active":"2024-02-19T15:27:23.650642+0000","last_peered":"2024-02-19T15:27:23.650642+0000","last_clean":"2024-02-19T15:27:23.650642+0000","last_became_active":"2024-02-19T15:27:22.068450+0000","last_became_peered":"2024-02-19T15:27:22.068450+0000","last_unstale":"2024-02-19T15:27:23.650642+0000","last_undegraded":"2024-02-19T15:27:23.650642+0000","last_fullsized":"2024-02-19T15:27:23.650642+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:48.996005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650393+0000","last_change":"2024-02-19T15:27:21.024536+0000","last_active":"2024-02-19T15:27:23.650393+0000","last_peered":"2024-02-19T15:27:23.650393+0000","last_clean":"2024-02-19T15:27:23.650393+0000","last_became_active":"2024-02-19T15:27:21.020949+0000","last_became_peered":"2024-02-19T15:27:21.020949+0000","last_unstale":"2024-02-19T15:27:23.650393+0000","last_undegraded":"2024-02-19T15:27:23.650393+0000","last_fullsized":"2024-02-19T15:27:23.650393+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:20:37.538132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649727+0000","last_change":"2024-02-19T15:27:22.052196+0000","last_active":"2024-02-19T15:27:23.649727+0000","last_peered":"2024-02-19T15:27:23.649727+0000","last_clean":"2024-02-19T15:27:23.649727+0000","last_became_active":"2024-02-19T15:27:22.051895+0000","last_became_peered":"2024-02-19T15:27:22.051895+0000","last_unstale":"2024-02-19T15:27:23.649727+0000","last_undegraded":"2024-02-19T15:27:23.649727+0000","last_fullsized":"2024-02-19T15:27:23.649727+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:21.004796+0000","last_clean_scrub_stamp":"2024-02-19T15:27:21.004796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:53:30.741750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.649841+0000","last_change":"2024-02-19T15:27:21.036160+0000","last_active":"2024-02-19T15:27:23.649841+0000","last_peered":"2024-02-19T15:27:23.649841+0000","last_clean":"2024-02-19T15:27:23.649841+0000","last_became_active":"2024-02-19T15:27:21.035805+0000","last_became_peered":"2024-02-19T15:27:21.035805+0000","last_unstale":"2024-02-19T15:27:23.649841+0000","last_undegraded":"2024-02-19T15:27:23.649841+0000","last_fullsized":"2024-02-19T15:27:23.649841+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:27:19.998983+0000","last_clean_scrub_stamp":"2024-02-19T15:27:19.998983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:25:19.941750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":324,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-02-19T15:27:23.650271+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:27:23.650271+0000","last_peered":"2024-02-19T15:27:23.650271+0000","last_clean":"2024-02-19T15:27:23.650271+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:27:23.650271+0000","last_undegraded":"2024-02-19T15:27:23.650271+0000","last_fullsized":"2024-02-19T15:27:23.650271+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509510,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36656,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335184,"statfs":{"total":96636764160,"available":96599228416,"internally_reserved":0,"allocated":1343488,"data_stored":576921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.622,"15min":0.62},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.66100000000000003,"15min":0.746},"min":{"1min":0.39100000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.8100000000000001,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.48899999999999999}]},{"osd":1,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63800000000000001,"15min":0.65000000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.44,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.774,"5min":2.774,"15min":2.774},"last":0.77200000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63100000000000001,"15min":0.63200000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.69899999999999995,"15min":0.68899999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.54500000000000004}]},{"osd":3,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":0.83499999999999996,"15min":0.72699999999999998},"min":{"1min":0.46600000000000003,"5min":0.438,"15min":0.438},"max":{"1min":14.045999999999999,"5min":14.045999999999999,"15min":14.045999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.67600000000000005,"15min":0.66200000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.752}]},{"osd":4,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.64500000000000002,"15min":0.61299999999999999},"min":{"1min":0.32400000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.69099999999999995,"15min":0.625},"min":{"1min":0.35599999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":2.8500000000000001,"5min":2.8500000000000001,"15min":2.8500000000000001},"last":0.82099999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.61899999999999999},"min":{"1min":0.38200000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.72899999999999998,"15min":0.68500000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.80200000000000005}]},{"osd":7,"last update":"Mon Feb 19 15:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.75900000000000001,"15min":0.68100000000000005},"min":{"1min":0.42799999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":3.0539999999999998,"5min":3.0539999999999998,"15min":3.0539999999999998},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.755,"15min":0.69299999999999995},"min":{"1min":0.439,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.256,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.83399999999999996}]}]},{"osd":7,"up_from":14,"seq":60129542214,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37244,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334596,"statfs":{"total":96636764160,"available":96598626304,"internally_reserved":0,"allocated":1941504,"data_stored":1167835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":2.016,"5min":0.88300000000000001,"15min":0.68500000000000005},"min":{"1min":0.51800000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":15.127000000000001,"5min":15.127000000000001,"15min":15.127000000000001},"last":0.752},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.71899999999999997,"15min":0.69399999999999995},"min":{"1min":0.47499999999999998,"5min":0.376,"15min":0.376},"max":{"1min":6.0890000000000004,"5min":6.0890000000000004,"15min":6.0890000000000004},"last":0.61199999999999999}]},{"osd":1,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":2.0910000000000002,"5min":0.93600000000000005,"15min":0.70199999999999996},"min":{"1min":0.47499999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":15.196999999999999,"5min":15.196999999999999,"15min":15.196999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":1.3100000000000001,"5min":0.81999999999999995,"15min":0.76000000000000001},"min":{"1min":0.45600000000000002,"5min":0.437,"15min":0.437},"max":{"1min":10.933999999999999,"5min":10.933999999999999,"15min":10.933999999999999},"last":0.73699999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":0.79600000000000004,"15min":0.68600000000000005},"min":{"1min":0.45700000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":10.84,"5min":10.84,"15min":10.84},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.71199999999999997,"15min":0.69999999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":2.3170000000000002,"5min":2.508,"15min":2.508},"last":0.71499999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":1.6539999999999999,"5min":0.86199999999999999,"15min":0.72599999999999998},"min":{"1min":0.49199999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":17.094000000000001,"5min":17.094000000000001,"15min":17.094000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.226,"5min":0.81799999999999995,"15min":0.73099999999999998},"min":{"1min":0.46300000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":7.7190000000000003,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.64600000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.61299999999999999,"15min":0.54500000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":2.266,"5min":0.95799999999999996,"15min":0.69499999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":26.625,"5min":26.625,"15min":26.625},"last":0.59199999999999997}]},{"osd":5,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":2.5350000000000001,"5min":1.0549999999999999,"15min":0.71699999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":31.844999999999999,"5min":31.844999999999999,"15min":31.844999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.6100000000000001,"5min":0.876,"15min":0.70399999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":16.776,"5min":16.776,"15min":16.776},"last":0.629}]},{"osd":6,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.70499999999999996,"15min":0.625},"min":{"1min":0.375,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.76400000000000001,"15min":0.68799999999999994},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":3.5099999999999998,"5min":3.5099999999999998,"15min":3.5099999999999998},"last":0.82299999999999995}]}]},{"osd":5,"up_from":14,"seq":60129542215,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36656,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335184,"statfs":{"total":96636764160,"available":96599228416,"internally_reserved":0,"allocated":1347584,"data_stored":577455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.69699999999999995,"15min":0.65300000000000002},"min":{"1min":0.48399999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":2.6659999999999999,"5min":5.7789999999999999,"15min":5.7789999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73799999999999999,"15min":0.64600000000000002},"min":{"1min":0.45000000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.2370000000000001,"5min":12.359999999999999,"15min":12.359999999999999},"last":0.52700000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.76600000000000001,"15min":0.67900000000000005},"min":{"1min":0.496,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.6499999999999999,"5min":10.938000000000001,"15min":10.938000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.82199999999999995,"15min":0.72399999999999998},"min":{"1min":0.45800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":3.625,"5min":10.531000000000001,"15min":10.531000000000001},"last":0.69299999999999995}]},{"osd":2,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.81499999999999995,"15min":0.72699999999999998},"min":{"1min":0.503,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":2.589,"5min":12.925000000000001,"15min":12.925000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.74099999999999999,"15min":0.64500000000000002},"min":{"1min":0.46999999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89100000000000001,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.85899999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.96499999999999997,"15min":0.753},"min":{"1min":0.51000000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":5.492,"5min":26.161999999999999,"15min":26.161999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.76200000000000001},"min":{"1min":0.52700000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":6.4939999999999998,"5min":27.934999999999999,"15min":27.934999999999999},"last":0.66600000000000004}]},{"osd":4,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":0.79900000000000004,"15min":0.63300000000000001},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":11.73,"5min":11.73,"15min":11.73},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.117,"5min":0.80200000000000005,"15min":0.70199999999999996},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":10.731,"5min":10.731,"15min":10.731},"last":0.80400000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.76900000000000002,"15min":0.67700000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.6320000000000001,"5min":6.5229999999999997,"15min":6.5229999999999997},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.248,"5min":0.92600000000000005,"15min":0.78400000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":10.432,"5min":10.432,"15min":10.432},"last":0.92900000000000005}]},{"osd":7,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.78700000000000003,"15min":0.69799999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.9450000000000001,"5min":11.986000000000001,"15min":11.986000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.83099999999999996,"15min":0.73299999999999998},"min":{"1min":0.42999999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":6.5339999999999998,"5min":8.9770000000000003,"15min":8.9770000000000003},"last":0.78000000000000003}]}]},{"osd":4,"up_from":14,"seq":60129542213,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36656,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335184,"statfs":{"total":96636764160,"available":96599228416,"internally_reserved":0,"allocated":1339392,"data_stored":576851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":4.1779999999999999,"5min":1.3089999999999999,"15min":0.83099999999999996},"min":{"1min":0.38,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":76.117000000000004,"5min":76.117000000000004,"15min":76.117000000000004},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.746,"5min":0.65300000000000002,"15min":0.61499999999999999},"min":{"1min":0.40000000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.2709999999999999,"5min":3.2709999999999999,"15min":3.2709999999999999},"last":0.46800000000000003}]},{"osd":1,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":4.4470000000000001,"5min":1.409,"15min":0.875},"min":{"1min":0.45800000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":80.513000000000005,"5min":80.513000000000005,"15min":80.513000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.70799999999999996,"15min":0.65500000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":7.181,"5min":7.181,"15min":7.181},"last":0.70399999999999996}]},{"osd":2,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":0.77900000000000003,"15min":0.70199999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":14.362,"5min":14.362,"15min":14.362},"last":0.748},{"interface":"front","average":{"1min":4.6100000000000003,"5min":1.4690000000000001,"15min":0.93899999999999995},"min":{"1min":0.42299999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":84.093000000000004,"5min":84.093000000000004,"15min":84.093000000000004},"last":0.42499999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":4.6539999999999999,"5min":1.5049999999999999,"15min":0.94799999999999995},"min":{"1min":0.504,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":84.140000000000001,"5min":84.140000000000001,"15min":84.140000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.44400000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.57399999999999995}]},{"osd":5,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":0.76500000000000001,"15min":0.64900000000000002},"min":{"1min":0.379,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":14.318,"5min":14.318,"15min":14.318},"last":0.77800000000000002},{"interface":"front","average":{"1min":4.274,"5min":1.427,"15min":0.92300000000000004},"min":{"1min":0.41099999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":76.164000000000001,"5min":76.164000000000001,"15min":76.164000000000001},"last":0.65900000000000003}]},{"osd":6,"last update":"Mon Feb 19 15:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64100000000000001,"15min":0.58899999999999997},"min":{"1min":0.33500000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":3.5019999999999998,"5min":1.2589999999999999,"15min":0.84599999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":60.151000000000003,"5min":60.151000000000003,"15min":60.151000000000003},"last":0.80300000000000005}]},{"osd":7,"last update":"Mon Feb 19 15:26:55 2024","interfaces":[{"interface":"back","average":{"1min":2.044,"5min":0.97999999999999998,"15min":0.76400000000000001},"min":{"1min":0.40699999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":29.036000000000001,"5min":29.036000000000001,"15min":29.036000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.0449999999999999,"5min":0.82099999999999995,"15min":0.755},"min":{"1min":0.50700000000000001,"5min":0.443,"15min":0.443},"max":{"1min":7.101,"5min":7.101,"15min":7.101},"last":0.82499999999999996}]}]},{"osd":3,"up_from":14,"seq":60129542213,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40824,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331016,"statfs":{"total":96636764160,"available":96594960384,"internally_reserved":0,"allocated":1355776,"data_stored":578533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41799999999999998,"15min":0.44400000000000001},"min":{"1min":0.23499999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.82699999999999996,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.5,"15min":0.63200000000000001},"min":{"1min":0.28899999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.0840000000000001,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.503}]},{"osd":1,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.437,"15min":0.443},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.54400000000000004,"15min":0.69199999999999995},"min":{"1min":0.316,"5min":0.31,"15min":0.31},"max":{"1min":0.85099999999999998,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":0.41299999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55500000000000005,"15min":0.60699999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53700000000000003,"15min":0.58499999999999996},"min":{"1min":0.38400000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.53800000000000003}]},{"osd":4,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71299999999999997,"15min":0.68200000000000005},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.97999999999999998,"5min":1.004,"15min":1.004},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.75800000000000001,"15min":0.84599999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0169999999999999,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.63700000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.77000000000000002,"15min":0.84399999999999997},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.99099999999999999,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.73499999999999999,"15min":0.745},"min":{"1min":0.46999999999999997,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89700000000000002,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.86199999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:27:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.79300000000000004,"15min":0.879},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.3180000000000001,"5min":4.0940000000000003,"15min":4.0940000000000003},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.77500000000000002,"15min":0.81000000000000005},"min":{"1min":0.53900000000000003,"5min":0.502,"15min":0.502},"max":{"1min":1.206,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.76600000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.755,"15min":0.77700000000000002},"min":{"1min":0.57099999999999995,"5min":0.504,"15min":0.504},"max":{"1min":0.996,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.80400000000000005,"15min":0.90800000000000003},"min":{"1min":0.54800000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.1919999999999999,"5min":3.9660000000000002,"15min":3.9660000000000002},"last":0.746}]}]},{"osd":2,"up_from":14,"seq":60129542214,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36648,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335192,"statfs":{"total":96636764160,"available":96599236608,"internally_reserved":0,"allocated":1339392,"data_stored":576851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.60099999999999998,"15min":0.67500000000000004},"min":{"1min":0.26600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.48899999999999999,"5min":15.42,"15min":15.42},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.61199999999999999,"15min":0.85099999999999998},"min":{"1min":0.28199999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68700000000000006,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":0.53800000000000003}]},{"osd":1,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.65500000000000003,"15min":0.86299999999999999},"min":{"1min":0.29399999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72799999999999998,"5min":9.5269999999999992,"15min":9.5269999999999992},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.83999999999999997,"15min":0.97199999999999998},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.75600000000000001,"5min":22.954999999999998,"15min":22.954999999999998},"last":0.51500000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.61499999999999999,"15min":0.76500000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.75,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.498},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.68700000000000006,"15min":0.80400000000000005},"min":{"1min":0.42399999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81399999999999995,"5min":7.734,"15min":7.734},"last":0.55200000000000005}]},{"osd":4,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.86399999999999999,"15min":0.88600000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.97999999999999998,"5min":15.369,"15min":15.369},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.77800000000000002,"15min":0.876},"min":{"1min":0.51900000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93600000000000005,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.76900000000000002}]},{"osd":5,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.83299999999999996,"15min":0.876},"min":{"1min":0.47199999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.006,"5min":7.6520000000000001,"15min":7.6520000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.93400000000000005,"15min":0.95699999999999996},"min":{"1min":0.496,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.026,"5min":15.391,"15min":15.391},"last":0.96099999999999997}]},{"osd":6,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.90900000000000003,"15min":0.92800000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.92000000000000004,"5min":15.337,"15min":15.337},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.81399999999999995,"15min":0.93799999999999994},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.93700000000000006,"5min":4.4320000000000004,"15min":4.4320000000000004},"last":0.81000000000000005}]},{"osd":7,"last update":"Mon Feb 19 15:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.85399999999999998,"15min":0.96399999999999997},"min":{"1min":0.57099999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.98099999999999998,"5min":4.4829999999999997,"15min":4.4829999999999997},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.86099999999999999,"15min":0.93500000000000005},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94299999999999995,"5min":7.6929999999999996,"15min":7.6929999999999996},"last":0.997}]}]},{"osd":1,"up_from":14,"seq":60129542215,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40824,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331016,"statfs":{"total":96636764160,"available":96594960384,"internally_reserved":0,"allocated":1347584,"data_stored":576975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38600000000000001,"15min":0.42199999999999999},"min":{"1min":0.25700000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.68999999999999995,"5min":0.747,"15min":0.747},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.434,"15min":0.46500000000000002},"min":{"1min":0.312,"5min":0.25,"15min":0.25},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.48799999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.44500000000000001},"min":{"1min":0.30499999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.47799999999999998,"15min":0.52300000000000002},"min":{"1min":0.32400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.74099999999999999,"5min":1.071,"15min":1.071},"last":0.47099999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57999999999999996,"15min":0.63},"min":{"1min":0.45500000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.1200000000000001,"5min":1.365,"15min":1.365},"last":0.503},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.55800000000000005,"15min":0.625},"min":{"1min":0.35399999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0069999999999999,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.52900000000000003}]},{"osd":4,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.67400000000000004,"15min":0.62},"min":{"1min":0.434,"5min":0.377,"15min":0.377},"max":{"1min":1.0409999999999999,"5min":1.127,"15min":1.127},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.69999999999999996,"15min":0.68200000000000005},"min":{"1min":0.56100000000000005,"5min":0.435,"15min":0.435},"max":{"1min":1.083,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.54900000000000004}]},{"osd":5,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.71199999999999997,"15min":0.69799999999999995},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.88800000000000001,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70899999999999996,"15min":0.68700000000000006},"min":{"1min":0.51200000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.96399999999999997,"5min":1.107,"15min":1.107},"last":0.64700000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.73199999999999998,"15min":0.72899999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.97799999999999998,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.74399999999999999,"15min":0.746},"min":{"1min":0.56299999999999994,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.1559999999999999,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.67600000000000005}]},{"osd":7,"last update":"Mon Feb 19 15:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.78400000000000003},"min":{"1min":0.52700000000000002,"5min":0.496,"15min":0.496},"max":{"1min":0.94599999999999995,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.75},"min":{"1min":0.54500000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.99299999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.59099999999999997}]}]},{"osd":0,"up_from":14,"seq":60129542213,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41408,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330432,"statfs":{"total":96636764160,"available":96594362368,"internally_reserved":0,"allocated":1953792,"data_stored":1168943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.45300000000000001,"15min":0.60999999999999999},"min":{"1min":0.24299999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.745,"5min":5.0199999999999996,"15min":5.0199999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.45800000000000002,"15min":0.48199999999999998},"min":{"1min":0.34200000000000003,"5min":0.252,"15min":0.252},"max":{"1min":0.77300000000000002,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.46899999999999997}]},{"osd":2,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59999999999999998,"15min":0.76600000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.82399999999999995,"5min":5.6669999999999998,"15min":5.6669999999999998},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.56100000000000005,"15min":0.72699999999999998},"min":{"1min":0.35699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.85899999999999999,"5min":5.4649999999999999,"15min":5.4649999999999999},"last":0.51900000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52000000000000002,"15min":0.66300000000000003},"min":{"1min":0.307,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.93799999999999994,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.495},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53300000000000003,"15min":0.60499999999999998},"min":{"1min":0.376,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.052,"5min":1.345,"15min":1.345},"last":0.53900000000000003}]},{"osd":4,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.73199999999999998,"15min":0.73799999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.091,"5min":1.095,"15min":1.095},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.79500000000000004,"15min":0.90900000000000003},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.085,"5min":4.7880000000000003,"15min":4.7880000000000003},"last":0.66600000000000004}]},{"osd":5,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.71399999999999997,"15min":0.74299999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.036,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.73699999999999999,"15min":0.84099999999999997},"min":{"1min":0.53600000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.169,"5min":4.8810000000000002,"15min":4.8810000000000002},"last":0.57599999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.76400000000000001,"15min":0.89600000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.1200000000000001,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.624},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.71999999999999997,"15min":0.751},"min":{"1min":0.50800000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.746}]},{"osd":7,"last update":"Mon Feb 19 15:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.79600000000000004,"15min":0.92400000000000004},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.1399999999999999,"5min":4.9809999999999999,"15min":4.9809999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.78100000000000003,"15min":0.92300000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.149,"5min":4.915,"15min":4.915},"last":0.76000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:27:31.406 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T15:27:31.406 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-19T15:27:31.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.587+0000 7fdc49600640 1 Processor -- start 2024-02-19T15:27:31.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.587+0000 7fdc49600640 1 -- start start 2024-02-19T15:27:31.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.587+0000 7fdc49600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdc4415f420 0x7fdc4406fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:31.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.587+0000 7fdc49600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc44070040 0x7fdc4406aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:31.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.587+0000 7fdc49600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4415fbe0 con 0x7fdc4406b3f0 2024-02-19T15:27:31.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.587+0000 7fdc49600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc4406b860 con 0x7fdc4415f420 2024-02-19T15:27:31.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.587+0000 7fdc49600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4415fe50 con 0x7fdc44070040 2024-02-19T15:27:31.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.587+0000 7fdc42a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdc4415f420 0x7fdc4406fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:31.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.587+0000 7fdc42a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdc4415f420 0x7fdc4406fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55392/0 (socket says 172.21.15.47:55392) 2024-02-19T15:27:31.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.587+0000 7fdc42a00640 1 -- 172.21.15.47:0/778165699 learned_addr learned my addr 172.21.15.47:0/778165699 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:31.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.587+0000 7fdc3be00640 1 --2- 172.21.15.47:0/778165699 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc44070040 0x7fdc4406aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:31.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.587+0000 7fdc42a00640 1 -- 172.21.15.47:0/778165699 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc44070040 msgr2=0x7fdc4406aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:31.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.587+0000 7fdc42a00640 1 --2- 172.21.15.47:0/778165699 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc44070040 0x7fdc4406aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.587+0000 7fdc42a00640 1 -- 172.21.15.47:0/778165699 >> v1:172.21.15.116:6789/0 conn(0x7fdc4406b3f0 legacy=0x7fdc44071850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:31.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.587+0000 7fdc42a00640 1 -- 172.21.15.47:0/778165699 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc44160130 con 0x7fdc4415f420 2024-02-19T15:27:31.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.591+0000 7fdc42a00640 1 --2- 172.21.15.47:0/778165699 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc4415f420 0x7fdc4406fb00 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fdc340098a0 tx=0x7fdc3401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcacd95e7d34ff8a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:31.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.591+0000 7fdc42000640 1 -- 172.21.15.47:0/778165699 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc340aa5f0 con 0x7fdc4415f420 2024-02-19T15:27:31.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.591+0000 7fdc42000640 1 -- 172.21.15.47:0/778165699 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc340aa750 con 0x7fdc4415f420 2024-02-19T15:27:31.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.591+0000 7fdc42000640 1 -- 172.21.15.47:0/778165699 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc340aaae0 con 0x7fdc4415f420 2024-02-19T15:27:31.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.591+0000 7fdc49600640 1 -- 172.21.15.47:0/778165699 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc4415f420 msgr2=0x7fdc4406fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:31.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.591+0000 7fdc49600640 1 --2- 172.21.15.47:0/778165699 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc4415f420 0x7fdc4406fb00 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fdc340098a0 tx=0x7fdc3401c4d0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.591+0000 7fdc49600640 1 -- 172.21.15.47:0/778165699 shutdown_connections 2024-02-19T15:27:31.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.591+0000 7fdc49600640 1 --2- 172.21.15.47:0/778165699 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc44070040 0x7fdc4406aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.591+0000 7fdc49600640 1 --2- 172.21.15.47:0/778165699 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc4415f420 0x7fdc4406fb00 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.591+0000 7fdc49600640 1 -- 172.21.15.47:0/778165699 >> 172.21.15.47:0/778165699 conn(0x7fdc440b2580 msgr2=0x7fdc440b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:31.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.591+0000 7fdc49600640 1 -- 172.21.15.47:0/778165699 shutdown_connections 2024-02-19T15:27:31.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.591+0000 7fdc49600640 1 -- 172.21.15.47:0/778165699 wait complete. 2024-02-19T15:27:31.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc49600640 1 Processor -- start 2024-02-19T15:27:31.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc49600640 1 -- start start 2024-02-19T15:27:31.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc49600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc4406b3f0 0x7fdc44151400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:31.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc49600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc4415f420 0x7fdc441fbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:31.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc49600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc441f72b0 0x7fdc44155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:31.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc49600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc440c5520 con 0x7fdc4406b3f0 2024-02-19T15:27:31.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc49600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc441fbfe0 con 0x7fdc4415f420 2024-02-19T15:27:31.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc49600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc440c4910 con 0x7fdc441f72b0 2024-02-19T15:27:31.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc42a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc4406b3f0 0x7fdc44151400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:31.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc42a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc4406b3f0 0x7fdc44151400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55398/0 (socket says 172.21.15.47:55398) 2024-02-19T15:27:31.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc42a00640 1 -- 172.21.15.47:0/2963395182 learned_addr learned my addr 172.21.15.47:0/2963395182 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:31.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc42a00640 1 -- 172.21.15.47:0/2963395182 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc441f72b0 msgr2=0x7fdc44155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:31.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc42a00640 1 --2- 172.21.15.47:0/2963395182 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc441f72b0 0x7fdc44155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc42a00640 1 -- 172.21.15.47:0/2963395182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc4415f420 msgr2=0x7fdc441fbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:31.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc42a00640 1 --2- 172.21.15.47:0/2963395182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc4415f420 0x7fdc441fbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc42a00640 1 -- 172.21.15.47:0/2963395182 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc340094d0 con 0x7fdc4406b3f0 2024-02-19T15:27:31.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc42a00640 1 --2- 172.21.15.47:0/2963395182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc4406b3f0 0x7fdc44151400 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fdc340ac040 tx=0x7fdc3401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:31.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc3aa00640 1 -- 172.21.15.47:0/2963395182 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc340b3800 con 0x7fdc4406b3f0 2024-02-19T15:27:31.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc3aa00640 1 -- 172.21.15.47:0/2963395182 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc340b3980 con 0x7fdc4406b3f0 2024-02-19T15:27:31.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc3aa00640 1 -- 172.21.15.47:0/2963395182 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc340b3d30 con 0x7fdc4406b3f0 2024-02-19T15:27:31.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc49600640 1 -- 172.21.15.47:0/2963395182 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc440c4bf0 con 0x7fdc4406b3f0 2024-02-19T15:27:31.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.595+0000 7fdc49600640 1 -- 172.21.15.47:0/2963395182 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc440c51b0 con 0x7fdc4406b3f0 2024-02-19T15:27:31.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.599+0000 7fdc49600640 1 -- 172.21.15.47:0/2963395182 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc44166da0 con 0x7fdc4406b3f0 2024-02-19T15:27:31.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.599+0000 7fdc3aa00640 1 -- 172.21.15.47:0/2963395182 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fdc340c6390 con 0x7fdc4406b3f0 2024-02-19T15:27:31.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.599+0000 7fdc3aa00640 1 --2- 172.21.15.47:0/2963395182 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdc20066e70 0x7fdc20069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:31.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.599+0000 7fdc3aa00640 1 -- 172.21.15.47:0/2963395182 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc340a7070 con 0x7fdc4406b3f0 2024-02-19T15:27:31.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.599+0000 7fdc3be00640 1 --2- 172.21.15.47:0/2963395182 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdc20066e70 0x7fdc20069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:31.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.599+0000 7fdc3be00640 1 --2- 172.21.15.47:0/2963395182 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdc20066e70 0x7fdc20069330 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fdc440cfd40 tx=0x7fdc2c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:31.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.603+0000 7fdc3aa00640 1 -- 172.21.15.47:0/2963395182 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fdc340ec100 con 0x7fdc4406b3f0 2024-02-19T15:27:31.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.823+0000 7fdc49600640 1 -- 172.21.15.47:0/2963395182 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fdc440c42c0 con 0x7fdc4406b3f0 2024-02-19T15:27:31.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.827+0000 7fdc3aa00640 1 -- 172.21.15.47:0/2963395182 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7fdc340ef7c0 con 0x7fdc4406b3f0 2024-02-19T15:27:31.830 INFO:teuthology.orchestra.run.smithi047.stdout:[client.0] 2024-02-19T15:27:31.830 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDjcdNlb7eRNRAAk6ZLGfjhcdlR9GoXFCd9eA== 2024-02-19T15:27:31.830 INFO:teuthology.orchestra.run.smithi047.stdout: caps mds = "allow" 2024-02-19T15:27:31.830 INFO:teuthology.orchestra.run.smithi047.stdout: caps mon = "allow r" 2024-02-19T15:27:31.830 INFO:teuthology.orchestra.run.smithi047.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-19T15:27:31.831 INFO:teuthology.orchestra.run.smithi047.stderr:updated caps for client.0 2024-02-19T15:27:31.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.827+0000 7fdc49600640 1 -- 172.21.15.47:0/2963395182 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdc20066e70 msgr2=0x7fdc20069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:31.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.827+0000 7fdc49600640 1 --2- 172.21.15.47:0/2963395182 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdc20066e70 0x7fdc20069330 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fdc440cfd40 tx=0x7fdc2c008040 comp rx=0 tx=0).stop 2024-02-19T15:27:31.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.827+0000 7fdc49600640 1 -- 172.21.15.47:0/2963395182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc4406b3f0 msgr2=0x7fdc44151400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:31.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.827+0000 7fdc49600640 1 --2- 172.21.15.47:0/2963395182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc4406b3f0 0x7fdc44151400 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fdc340ac040 tx=0x7fdc3401ad10 comp rx=0 tx=0).stop 2024-02-19T15:27:31.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.827+0000 7fdc49600640 1 -- 172.21.15.47:0/2963395182 shutdown_connections 2024-02-19T15:27:31.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.827+0000 7fdc49600640 1 --2- 172.21.15.47:0/2963395182 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdc20066e70 0x7fdc20069330 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.827+0000 7fdc49600640 1 --2- 172.21.15.47:0/2963395182 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdc441f72b0 0x7fdc44155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.827+0000 7fdc49600640 1 --2- 172.21.15.47:0/2963395182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc4415f420 0x7fdc441fbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.827+0000 7fdc49600640 1 --2- 172.21.15.47:0/2963395182 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc4406b3f0 0x7fdc44151400 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.827+0000 7fdc49600640 1 -- 172.21.15.47:0/2963395182 >> 172.21.15.47:0/2963395182 conn(0x7fdc440b2580 msgr2=0x7fdc440ad850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:31.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.827+0000 7fdc49600640 1 -- 172.21.15.47:0/2963395182 shutdown_connections 2024-02-19T15:27:31.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.827+0000 7fdc49600640 1 -- 172.21.15.47:0/2963395182 wait complete. 2024-02-19T15:27:31.852 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:27:31.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.987+0000 7f9480600640 1 Processor -- start 2024-02-19T15:27:31.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.987+0000 7f9480600640 1 -- start start 2024-02-19T15:27:31.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.991+0000 7f9480600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f947815c0d0 0x7f94780cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:31.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.991+0000 7f9480600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94780d0540 0x7f94780b1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:31.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.991+0000 7f9480600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94780b0fe0 con 0x7f94780cd180 2024-02-19T15:27:31.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.991+0000 7f9480600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f947815c4b0 con 0x7f947815c0d0 2024-02-19T15:27:31.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.991+0000 7f9480600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94780b1880 con 0x7f94780d0540 2024-02-19T15:27:31.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.991+0000 7f947dc00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f947815c0d0 0x7f94780cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:31.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.991+0000 7f947d200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f94780cd180 0x7f94780cfe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39830/0 (socket says 172.21.15.47:39830) 2024-02-19T15:27:31.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.991+0000 7f947d200640 1 -- 172.21.15.47:0/3857850148 learned_addr learned my addr 172.21.15.47:0/3857850148 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:31.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.991+0000 7f947e600640 1 --2- 172.21.15.47:0/3857850148 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94780d0540 0x7f94780b1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:31.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.991+0000 7f947dc00640 1 -- 172.21.15.47:0/3857850148 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94780d0540 msgr2=0x7f94780b1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:31.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.991+0000 7f947dc00640 1 --2- 172.21.15.47:0/3857850148 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94780d0540 0x7f94780b1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.991+0000 7f947dc00640 1 -- 172.21.15.47:0/3857850148 >> v1:172.21.15.47:6789/0 conn(0x7f94780cd180 legacy=0x7f94780cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:31.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.991+0000 7f947dc00640 1 -- 172.21.15.47:0/3857850148 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94780cd8f0 con 0x7f947815c0d0 2024-02-19T15:27:31.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.991+0000 7f947e600640 1 --2- 172.21.15.47:0/3857850148 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94780d0540 0x7f94780b1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:31.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.991+0000 7f9477e00640 1 -- 172.21.15.47:0/3857850148 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823655503 0 0) 0x7f94780b0fe0 con 0x7f94780cd180 2024-02-19T15:27:31.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.991+0000 7f947dc00640 1 --2- 172.21.15.47:0/3857850148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f947815c0d0 0x7f94780cf850 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f946c0097c0 tx=0x7f946c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30defd51bc16a37c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:31.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f9477e00640 1 -- 172.21.15.47:0/3857850148 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c0ab670 con 0x7f947815c0d0 2024-02-19T15:27:31.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f9477e00640 1 -- 172.21.15.47:0/3857850148 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946c0ab7d0 con 0x7f947815c0d0 2024-02-19T15:27:31.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f9477e00640 1 -- 172.21.15.47:0/3857850148 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c0abb60 con 0x7f947815c0d0 2024-02-19T15:27:31.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f9480600640 1 -- 172.21.15.47:0/3857850148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f947815c0d0 msgr2=0x7f94780cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:31.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f9480600640 1 --2- 172.21.15.47:0/3857850148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f947815c0d0 0x7f94780cf850 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f946c0097c0 tx=0x7f946c01c470 comp rx=0 tx=0).stop 2024-02-19T15:27:31.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f9480600640 1 -- 172.21.15.47:0/3857850148 shutdown_connections 2024-02-19T15:27:31.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f9480600640 1 --2- 172.21.15.47:0/3857850148 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94780d0540 0x7f94780b1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f9480600640 1 --2- 172.21.15.47:0/3857850148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f947815c0d0 0x7f94780cf850 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:31.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f9480600640 1 -- 172.21.15.47:0/3857850148 >> 172.21.15.47:0/3857850148 conn(0x7f94780d1ba0 msgr2=0x7f94780d1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:31.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f9480600640 1 -- 172.21.15.47:0/3857850148 shutdown_connections 2024-02-19T15:27:31.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f9480600640 1 -- 172.21.15.47:0/3857850148 wait complete. 2024-02-19T15:27:32.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f9480600640 1 Processor -- start 2024-02-19T15:27:32.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f9480600640 1 -- start start 2024-02-19T15:27:32.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f9480600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94780cd180 0x7f947815ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f9480600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94780d0540 0x7f947815d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f9480600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f947815c0d0 0x7f947815b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f9480600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947815b940 con 0x7f947815c0d0 2024-02-19T15:27:32.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f9480600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947815f440 con 0x7f94780cd180 2024-02-19T15:27:32.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f9480600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f947815de70 con 0x7f94780d0540 2024-02-19T15:27:32.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f947e600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f947815c0d0 0x7f947815b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f947e600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f947815c0d0 0x7f947815b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55412/0 (socket says 172.21.15.47:55412) 2024-02-19T15:27:32.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f947e600640 1 -- 172.21.15.47:0/3705733840 learned_addr learned my addr 172.21.15.47:0/3705733840 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:32.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.995+0000 7f947d200640 1 --2- 172.21.15.47:0/3705733840 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94780d0540 0x7f947815d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.999+0000 7f947dc00640 1 --2- 172.21.15.47:0/3705733840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94780cd180 0x7f947815ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.999+0000 7f947e600640 1 -- 172.21.15.47:0/3705733840 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94780d0540 msgr2=0x7f947815d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:32.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.999+0000 7f947e600640 1 --2- 172.21.15.47:0/3705733840 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94780d0540 0x7f947815d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.999+0000 7f947e600640 1 -- 172.21.15.47:0/3705733840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94780cd180 msgr2=0x7f947815ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:32.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.999+0000 7f947e600640 1 --2- 172.21.15.47:0/3705733840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94780cd180 0x7f947815ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.999+0000 7f947e600640 1 -- 172.21.15.47:0/3705733840 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946c009420 con 0x7f947815c0d0 2024-02-19T15:27:32.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.999+0000 7f947e600640 1 --2- 172.21.15.47:0/3705733840 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f947815c0d0 0x7f947815b0f0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f947000cf20 tx=0x7f947000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:32.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.999+0000 7f9475600640 1 -- 172.21.15.47:0/3705733840 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f947000d840 con 0x7f947815c0d0 2024-02-19T15:27:32.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.999+0000 7f9475600640 1 -- 172.21.15.47:0/3705733840 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f947000d9a0 con 0x7f947815c0d0 2024-02-19T15:27:32.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.999+0000 7f9475600640 1 -- 172.21.15.47:0/3705733840 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f947000dd50 con 0x7f947815c0d0 2024-02-19T15:27:32.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.999+0000 7f9480600640 1 -- 172.21.15.47:0/3705733840 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f947815e140 con 0x7f947815c0d0 2024-02-19T15:27:32.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.999+0000 7f9480600640 1 -- 172.21.15.47:0/3705733840 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f947815e700 con 0x7f947815c0d0 2024-02-19T15:27:32.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.999+0000 7f9480600640 1 -- 172.21.15.47:0/3705733840 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9478068500 con 0x7f947815c0d0 2024-02-19T15:27:32.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:31.999+0000 7f9475600640 1 -- 172.21.15.47:0/3705733840 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f947001e3b0 con 0x7f947815c0d0 2024-02-19T15:27:32.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.003+0000 7f9475600640 1 --2- 172.21.15.47:0/3705733840 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9458066e90 0x7f9458069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.003+0000 7f9475600640 1 -- 172.21.15.47:0/3705733840 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94700853c0 con 0x7f947815c0d0 2024-02-19T15:27:32.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.003+0000 7f947dc00640 1 --2- 172.21.15.47:0/3705733840 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9458066e90 0x7f9458069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.003+0000 7f947dc00640 1 --2- 172.21.15.47:0/3705733840 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9458066e90 0x7f9458069350 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f946c0af000 tx=0x7f946c0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:32.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.007+0000 7f9475600640 1 -- 172.21.15.47:0/3705733840 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f9470052320 con 0x7f947815c0d0 2024-02-19T15:27:32.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.203+0000 7f9480600640 1 -- 172.21.15.47:0/3705733840 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9478075360 con 0x7f947815c0d0 2024-02-19T15:27:32.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.203+0000 7f9475600640 1 -- 172.21.15.47:0/3705733840 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4937 (secure 0 0 0) 0x7f94700559e0 con 0x7f947815c0d0 2024-02-19T15:27:32.208 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:32.208 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":75,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6578,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/408259797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":408259797},{"type":"v1","addr":"172.21.15.116:6836","nonce":408259797}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7327,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3795002232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3795002232},{"type":"v1","addr":"172.21.15.47:6835","nonce":3795002232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7330,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/819751716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":819751716},{"type":"v1","addr":"172.21.15.116:6837","nonce":819751716}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:24.428431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6834/897369244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":897369244},{"type":"v1","addr":"172.21.15.47:6834","nonce":897369244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:27:32.208 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 75 2024-02-19T15:27:32.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.203+0000 7f9480600640 1 -- 172.21.15.47:0/3705733840 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9458066e90 msgr2=0x7f9458069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:32.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.203+0000 7f9480600640 1 --2- 172.21.15.47:0/3705733840 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9458066e90 0x7f9458069350 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f946c0af000 tx=0x7f946c0a7000 comp rx=0 tx=0).stop 2024-02-19T15:27:32.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.203+0000 7f9480600640 1 -- 172.21.15.47:0/3705733840 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f947815c0d0 msgr2=0x7f947815b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:32.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.203+0000 7f9480600640 1 --2- 172.21.15.47:0/3705733840 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f947815c0d0 0x7f947815b0f0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f947000cf20 tx=0x7f947000a7d0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.203+0000 7f9480600640 1 -- 172.21.15.47:0/3705733840 shutdown_connections 2024-02-19T15:27:32.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.203+0000 7f9480600640 1 --2- 172.21.15.47:0/3705733840 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9458066e90 0x7f9458069350 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.203+0000 7f9480600640 1 --2- 172.21.15.47:0/3705733840 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f947815c0d0 0x7f947815b0f0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.203+0000 7f9480600640 1 --2- 172.21.15.47:0/3705733840 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94780d0540 0x7f947815d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.203+0000 7f9480600640 1 --2- 172.21.15.47:0/3705733840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94780cd180 0x7f947815ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.203+0000 7f9480600640 1 -- 172.21.15.47:0/3705733840 >> 172.21.15.47:0/3705733840 conn(0x7f94780d1ba0 msgr2=0x7f94780cddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:32.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.203+0000 7f9480600640 1 -- 172.21.15.47:0/3705733840 shutdown_connections 2024-02-19T15:27:32.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.203+0000 7f9480600640 1 -- 172.21.15.47:0/3705733840 wait complete. 2024-02-19T15:27:32.631 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 75, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:27:21.415321+0000', 'modified': '2024-02-19T15:27:24.428431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7635}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7635': {'gid': 7635, 'name': 'c', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6834/897369244', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6832', 'nonce': 897369244}, {'type': 'v1', 'addr': '172.21.15.47:6834', 'nonce': 897369244}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:27:32.631 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T15:27:32.631 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T15:27:32.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.767+0000 7fcb4c200640 1 Processor -- start 2024-02-19T15:27:32.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.771+0000 7fcb4c200640 1 -- start start 2024-02-19T15:27:32.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.771+0000 7fcb4c200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcb440cedb0 0x7fcb440cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.771+0000 7fcb4c200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcb440d2710 0x7fcb440d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.771+0000 7fcb4c200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb440cc700 con 0x7fcb4415bcd0 2024-02-19T15:27:32.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.771+0000 7fcb4c200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb440cf5e0 con 0x7fcb440cedb0 2024-02-19T15:27:32.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.771+0000 7fcb4c200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb440cdb90 con 0x7fcb440d2710 2024-02-19T15:27:32.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.771+0000 7fcb48e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcb440cedb0 0x7fcb440cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.771+0000 7fcb49800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fcb4415bcd0 0x7fcb440cd480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39834/0 (socket says 172.21.15.47:39834) 2024-02-19T15:27:32.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.771+0000 7fcb49800640 1 -- 172.21.15.47:0/1316544908 learned_addr learned my addr 172.21.15.47:0/1316544908 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:32.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.771+0000 7fcb48e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcb440cedb0 0x7fcb440cfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55414/0 (socket says 172.21.15.47:55414) 2024-02-19T15:27:32.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.771+0000 7fcb4a200640 1 --2- 172.21.15.47:0/1316544908 >> v2:172.21.15.116:3300/0 conn(0x7fcb440d2710 0x7fcb440d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.771+0000 7fcb48e00640 1 -- 172.21.15.47:0/1316544908 >> v2:172.21.15.116:3300/0 conn(0x7fcb440d2710 msgr2=0x7fcb440d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:32.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.771+0000 7fcb48e00640 1 --2- 172.21.15.47:0/1316544908 >> v2:172.21.15.116:3300/0 conn(0x7fcb440d2710 0x7fcb440d0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.771+0000 7fcb48e00640 1 -- 172.21.15.47:0/1316544908 >> v1:172.21.15.47:6789/0 conn(0x7fcb4415bcd0 legacy=0x7fcb440cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:32.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.771+0000 7fcb48e00640 1 -- 172.21.15.47:0/1316544908 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb440cf300 con 0x7fcb440cedb0 2024-02-19T15:27:32.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.771+0000 7fcb43e00640 1 -- 172.21.15.47:0/1316544908 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663577574 0 0) 0x7fcb440cc700 con 0x7fcb4415bcd0 2024-02-19T15:27:32.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.775+0000 7fcb48e00640 1 --2- 172.21.15.47:0/1316544908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb440cedb0 0x7fcb440cfc50 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fcb440b7c80 tx=0x7fcb3001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67af590cb374959 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:32.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.775+0000 7fcb43e00640 1 -- 172.21.15.47:0/1316544908 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb300ab650 con 0x7fcb440cedb0 2024-02-19T15:27:32.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.775+0000 7fcb43e00640 1 -- 172.21.15.47:0/1316544908 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb300ab7b0 con 0x7fcb440cedb0 2024-02-19T15:27:32.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.775+0000 7fcb43e00640 1 -- 172.21.15.47:0/1316544908 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb300abb40 con 0x7fcb440cedb0 2024-02-19T15:27:32.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.775+0000 7fcb4c200640 1 -- 172.21.15.47:0/1316544908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb440cedb0 msgr2=0x7fcb440cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:32.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.775+0000 7fcb4c200640 1 --2- 172.21.15.47:0/1316544908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb440cedb0 0x7fcb440cfc50 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fcb440b7c80 tx=0x7fcb3001c450 comp rx=0 tx=0).stop 2024-02-19T15:27:32.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.775+0000 7fcb4c200640 1 -- 172.21.15.47:0/1316544908 shutdown_connections 2024-02-19T15:27:32.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.775+0000 7fcb4c200640 1 --2- 172.21.15.47:0/1316544908 >> v2:172.21.15.116:3300/0 conn(0x7fcb440d2710 0x7fcb440d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.775+0000 7fcb4c200640 1 --2- 172.21.15.47:0/1316544908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb440cedb0 0x7fcb440cfc50 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.775+0000 7fcb4c200640 1 -- 172.21.15.47:0/1316544908 >> 172.21.15.47:0/1316544908 conn(0x7fcb440d1bf0 msgr2=0x7fcb440d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:32.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.775+0000 7fcb4c200640 1 -- 172.21.15.47:0/1316544908 shutdown_connections 2024-02-19T15:27:32.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.775+0000 7fcb4c200640 1 -- 172.21.15.47:0/1316544908 wait complete. 2024-02-19T15:27:32.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb4c200640 1 Processor -- start 2024-02-19T15:27:32.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb4c200640 1 -- start start 2024-02-19T15:27:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb4c200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb440cedb0 0x7fcb44152c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb4c200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb440d2710 0x7fcb440b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb4c200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb4415bcd0 0x7fcb440b6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb4c200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb44153d10 con 0x7fcb440cedb0 2024-02-19T15:27:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb4c200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb440c6ce0 con 0x7fcb440d2710 2024-02-19T15:27:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb4c200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb44153140 con 0x7fcb4415bcd0 2024-02-19T15:27:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb49800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb440cedb0 0x7fcb44152c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb49800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb440cedb0 0x7fcb44152c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55420/0 (socket says 172.21.15.47:55420) 2024-02-19T15:27:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb49800640 1 -- 172.21.15.47:0/127754975 learned_addr learned my addr 172.21.15.47:0/127754975 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb4a200640 1 --2- 172.21.15.47:0/127754975 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb4415bcd0 0x7fcb440b6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb48e00640 1 --2- 172.21.15.47:0/127754975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb440d2710 0x7fcb440b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb49800640 1 -- 172.21.15.47:0/127754975 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb4415bcd0 msgr2=0x7fcb440b6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb49800640 1 --2- 172.21.15.47:0/127754975 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb4415bcd0 0x7fcb440b6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb49800640 1 -- 172.21.15.47:0/127754975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb440d2710 msgr2=0x7fcb440b6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb49800640 1 --2- 172.21.15.47:0/127754975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb440d2710 0x7fcb440b6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb49800640 1 -- 172.21.15.47:0/127754975 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb30009420 con 0x7fcb440cedb0 2024-02-19T15:27:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb48e00640 1 --2- 172.21.15.47:0/127754975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb440d2710 0x7fcb440b6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:27:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb49800640 1 --2- 172.21.15.47:0/127754975 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb440cedb0 0x7fcb44152c00 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800c4b0 tx=0x7fcb3800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb41600640 1 -- 172.21.15.47:0/127754975 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb380074e0 con 0x7fcb440cedb0 2024-02-19T15:27:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb41600640 1 -- 172.21.15.47:0/127754975 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb38007660 con 0x7fcb440cedb0 2024-02-19T15:27:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb41600640 1 -- 172.21.15.47:0/127754975 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb38004a30 con 0x7fcb440cedb0 2024-02-19T15:27:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb4c200640 1 -- 172.21.15.47:0/127754975 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4415d240 con 0x7fcb440cedb0 2024-02-19T15:27:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.779+0000 7fcb4c200640 1 -- 172.21.15.47:0/127754975 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb441533e0 con 0x7fcb440cedb0 2024-02-19T15:27:32.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.783+0000 7fcb41600640 1 -- 172.21.15.47:0/127754975 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fcb38004b90 con 0x7fcb440cedb0 2024-02-19T15:27:32.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.783+0000 7fcb4f78e000 1 Processor -- start 2024-02-19T15:27:32.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.783+0000 7fcb4f78e000 1 -- start start 2024-02-19T15:27:32.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.783+0000 7fcb41600640 1 --2- 172.21.15.47:0/127754975 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcb24066e90 0x7fcb24069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.783+0000 7fcb41600640 1 -- 172.21.15.47:0/127754975 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb38083ef0 con 0x7fcb440cedb0 2024-02-19T15:27:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb4a200640 1 --2- 172.21.15.47:0/127754975 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcb24066e90 0x7fcb24069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb4f78e000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa64376920 0x55aa64376d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb4f78e000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa6432c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb4f78e000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55aa6432cf30 0x55aa6432d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb4f78e000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa641f4d00 con 0x55aa64377240 2024-02-19T15:27:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb4f78e000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa64186430 con 0x55aa64376920 2024-02-19T15:27:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb4f78e000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aa64183960 con 0x55aa6432cf30 2024-02-19T15:27:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb48e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa6432c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb48e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa6432c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55430/0 (socket says 172.21.15.47:55430) 2024-02-19T15:27:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb48e00640 1 -- 172.21.15.47:0/2740058091 learned_addr learned my addr 172.21.15.47:0/2740058091 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb49800640 1 --2- 172.21.15.47:0/2740058091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa64376920 0x55aa64376d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb4a200640 1 --2- 172.21.15.47:0/2740058091 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55aa6432cf30 0x55aa6432d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb48e00640 1 -- 172.21.15.47:0/2740058091 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55aa6432cf30 msgr2=0x55aa6432d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb48e00640 1 --2- 172.21.15.47:0/2740058091 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55aa6432cf30 0x55aa6432d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb48e00640 1 -- 172.21.15.47:0/2740058091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa64376920 msgr2=0x55aa64376d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb48e00640 1 --2- 172.21.15.47:0/2740058091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa64376920 0x55aa64376d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb48e00640 1 -- 172.21.15.47:0/2740058091 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aa63e75890 con 0x55aa64377240 2024-02-19T15:27:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb4a200640 1 --2- 172.21.15.47:0/127754975 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcb24066e90 0x7fcb24069350 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c009f90 tx=0x7fcb3c007700 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb48e00640 1 --2- 172.21.15.47:0/2740058091 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa6432c9f0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fcb3001a080 tx=0x7fcb30004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c27b28553d53fce server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb36000640 1 -- 172.21.15.47:0/2740058091 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb300ab920 con 0x55aa64377240 2024-02-19T15:27:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb36000640 1 -- 172.21.15.47:0/2740058091 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb300aba80 con 0x55aa64377240 2024-02-19T15:27:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb36000640 1 -- 172.21.15.47:0/2740058091 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb300abe10 con 0x55aa64377240 2024-02-19T15:27:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb4f78e000 1 -- 172.21.15.47:0/2740058091 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 msgr2=0x55aa6432c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb4f78e000 1 --2- 172.21.15.47:0/2740058091 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa6432c9f0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fcb3001a080 tx=0x7fcb30004740 comp rx=0 tx=0).stop 2024-02-19T15:27:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb4f78e000 1 -- 172.21.15.47:0/2740058091 shutdown_connections 2024-02-19T15:27:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb4f78e000 1 --2- 172.21.15.47:0/2740058091 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55aa6432cf30 0x55aa6432d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb4f78e000 1 --2- 172.21.15.47:0/2740058091 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa6432c9f0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb4f78e000 1 --2- 172.21.15.47:0/2740058091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa64376920 0x55aa64376d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb4f78e000 1 -- 172.21.15.47:0/2740058091 >> 172.21.15.47:0/2740058091 conn(0x55aa643332a0 msgr2=0x55aa6432b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:32.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb4f78e000 1 -- 172.21.15.47:0/2740058091 shutdown_connections 2024-02-19T15:27:32.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb4f78e000 1 -- 172.21.15.47:0/2740058091 wait complete. 2024-02-19T15:27:32.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb4f78e000 1 -- start start 2024-02-19T15:27:32.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.787+0000 7fcb4f78e000 1 Processor -- start 2024-02-19T15:27:32.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4f78e000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa6432cf30 0x55aa64363a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4f78e000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55aa64376920 0x55aa64330aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4f78e000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa64330fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4f78e000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa6417f140 con 0x55aa64377240 2024-02-19T15:27:32.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4f78e000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa64119630 con 0x55aa6432cf30 2024-02-19T15:27:32.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4f78e000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aa641192d0 con 0x55aa64376920 2024-02-19T15:27:32.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4f78e000 10 client.0 _ll_register_callbacks cb 0x55aa6422f1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:27:32.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4a200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa64330fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb49800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa6432cf30 0x55aa64363a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4a200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa64330fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55436/0 (socket says 172.21.15.47:55436) 2024-02-19T15:27:32.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb48e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55aa64376920 0x55aa64330aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4a200640 1 -- 172.21.15.47:0/2468598478 learned_addr learned my addr 172.21.15.47:0/2468598478 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:32.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4a200640 1 -- 172.21.15.47:0/2468598478 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55aa64376920 msgr2=0x55aa64330aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:32.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4a200640 1 --2- 172.21.15.47:0/2468598478 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55aa64376920 0x55aa64330aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4a200640 1 -- 172.21.15.47:0/2468598478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa6432cf30 msgr2=0x55aa64363a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:32.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4a200640 1 --2- 172.21.15.47:0/2468598478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa6432cf30 0x55aa64363a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4a200640 1 -- 172.21.15.47:0/2468598478 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb440cf300 con 0x55aa64377240 2024-02-19T15:27:32.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4a200640 1 --2- 172.21.15.47:0/2468598478 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa64330fe0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c018b80 tx=0x7fcb3c004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c878e4dfd015b0d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:32.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb2aa00640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:27:32.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb2aa00640 1 -- 172.21.15.47:0/2468598478 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c019990 con 0x55aa64377240 2024-02-19T15:27:32.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb2aa00640 1 -- 172.21.15.47:0/2468598478 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb3c019af0 con 0x55aa64377240 2024-02-19T15:27:32.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4f78e000 1 -- 172.21.15.47:0/2468598478 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aa64132850 con 0x55aa64377240 2024-02-19T15:27:32.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb2aa00640 1 -- 172.21.15.47:0/2468598478 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb3c019cb0 con 0x55aa64377240 2024-02-19T15:27:32.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4f78e000 10 client.7818 fetch_fsmap learned FSMap version 75 2024-02-19T15:27:32.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4f78e000 1 -- 172.21.15.47:0/2468598478 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=2+}) v3 -- 0x55aa6413ee70 con 0x55aa64377240 2024-02-19T15:27:32.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb2aa00640 1 -- 172.21.15.47:0/2468598478 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55aa64132850 con 0x55aa64377240 2024-02-19T15:27:32.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb2aa00640 1 -- 172.21.15.47:0/2468598478 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 75) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fcb3c023410 con 0x55aa64377240 2024-02-19T15:27:32.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4f78e000 10 client.7818 fetch_fsmap finished waiting for FSMap version 75 2024-02-19T15:27:32.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4f78e000 10 client.7818 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T15:27:32.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4f78e000 20 client.7818 populate_metadata read hostname 'smithi047' 2024-02-19T15:27:32.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4f78e000 1 --2- 172.21.15.47:0/2468598478 >> [v2:172.21.15.47:6832/897369244,v1:172.21.15.47:6834/897369244] conn(0x55aa643319f0 0x55aa64331dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4f78e000 4 client.7818 mds_command: new command op to 7635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T15:27:32.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb4f78e000 1 -- 172.21.15.47:0/2468598478 --> [v2:172.21.15.47:6832/897369244,v1:172.21.15.47:6834/897369244] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aa6417bf20 con 0x55aa643319f0 2024-02-19T15:27:32.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb49800640 1 --2- 172.21.15.47:0/2468598478 >> [v2:172.21.15.47:6832/897369244,v1:172.21.15.47:6834/897369244] conn(0x55aa643319f0 0x55aa64331dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb2aa00640 10 client.7818 ms_handle_connect on v2:172.21.15.47:6832/897369244 2024-02-19T15:27:32.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.791+0000 7fcb49800640 1 --2- 172.21.15.47:0/2468598478 >> [v2:172.21.15.47:6832/897369244,v1:172.21.15.47:6834/897369244] conn(0x55aa643319f0 0x55aa64331dd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b76ad3352ef8374c server_cookie=76e27c72432852ca in_seq=0 out_seq=0 2024-02-19T15:27:32.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.795+0000 7fcb2aa00640 1 -- 172.21.15.47:0/2468598478 <== mds.0 v2:172.21.15.47:6832/897369244 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55aa6417bf20 con 0x55aa643319f0 2024-02-19T15:27:32.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.795+0000 7fcb2aa00640 10 client.7818 handle_command_reply: tid=0 2024-02-19T15:27:32.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.795+0000 7fcb4f78e000 1 client.7818 shutdown 2024-02-19T15:27:32.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.795+0000 7fcb4f78e000 1 -- 172.21.15.47:0/2468598478 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 msgr2=0x55aa64330fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:32.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.795+0000 7fcb4f78e000 1 --2- 172.21.15.47:0/2468598478 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa64330fe0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c018b80 tx=0x7fcb3c004ac0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.795+0000 7fcb4f78e000 1 -- 172.21.15.47:0/2468598478 shutdown_connections 2024-02-19T15:27:32.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.795+0000 7fcb4f78e000 1 --2- 172.21.15.47:0/2468598478 >> [v2:172.21.15.47:6832/897369244,v1:172.21.15.47:6834/897369244] conn(0x55aa643319f0 0x55aa64331dd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.795+0000 7fcb4f78e000 1 --2- 172.21.15.47:0/2468598478 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa64330fe0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.795+0000 7fcb4f78e000 1 --2- 172.21.15.47:0/2468598478 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55aa64376920 0x55aa64330aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.795+0000 7fcb4f78e000 1 --2- 172.21.15.47:0/2468598478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa6432cf30 0x55aa64363a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.795+0000 7fcb4f78e000 1 -- 172.21.15.47:0/2468598478 >> 172.21.15.47:0/2468598478 conn(0x55aa643332a0 msgr2=0x55aa6432b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:32.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.795+0000 7fcb2aa00640 0 client.7818 ms_handle_reset on v2:172.21.15.47:6832/897369244 2024-02-19T15:27:32.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.795+0000 7fcb4f78e000 1 -- 172.21.15.47:0/2468598478 shutdown_connections 2024-02-19T15:27:32.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.795+0000 7fcb4f78e000 1 -- 172.21.15.47:0/2468598478 wait complete. 2024-02-19T15:27:32.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.795+0000 7fcb4f78e000 20 client.7818 trim_cache size 0 max 16384 2024-02-19T15:27:32.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.827+0000 7fcb4f78e000 1 Processor -- start 2024-02-19T15:27:32.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.827+0000 7fcb4f78e000 1 -- start start 2024-02-19T15:27:32.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4f78e000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa6432cf30 0x55aa643da900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4f78e000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55aa64376920 0x55aa643dae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4f78e000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa643db380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4f78e000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa63e36940 con 0x55aa64377240 2024-02-19T15:27:32.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4f78e000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa6411ef10 con 0x55aa6432cf30 2024-02-19T15:27:32.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4f78e000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aa6413e8c0 con 0x55aa64376920 2024-02-19T15:27:32.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4a200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa643db380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4a200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa643db380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55444/0 (socket says 172.21.15.47:55444) 2024-02-19T15:27:32.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4a200640 1 -- 172.21.15.47:0/1682135599 learned_addr learned my addr 172.21.15.47:0/1682135599 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:32.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb49800640 1 --2- 172.21.15.47:0/1682135599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa6432cf30 0x55aa643da900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb48e00640 1 --2- 172.21.15.47:0/1682135599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55aa64376920 0x55aa643dae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4a200640 1 -- 172.21.15.47:0/1682135599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55aa64376920 msgr2=0x55aa643dae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:32.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4a200640 1 --2- 172.21.15.47:0/1682135599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55aa64376920 0x55aa643dae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4a200640 1 -- 172.21.15.47:0/1682135599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa6432cf30 msgr2=0x55aa643da900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:32.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4a200640 1 --2- 172.21.15.47:0/1682135599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa6432cf30 0x55aa643da900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4a200640 1 -- 172.21.15.47:0/1682135599 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aa64124e10 con 0x55aa64377240 2024-02-19T15:27:32.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4a200640 1 --2- 172.21.15.47:0/1682135599 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa643db380 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c007f60 tx=0x7fcb3c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7ab59352630a17f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:32.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb35600640 1 -- 172.21.15.47:0/1682135599 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c00d050 con 0x55aa64377240 2024-02-19T15:27:32.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb35600640 1 -- 172.21.15.47:0/1682135599 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3c023a60 con 0x55aa64377240 2024-02-19T15:27:32.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb35600640 1 -- 172.21.15.47:0/1682135599 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c0187b0 con 0x55aa64377240 2024-02-19T15:27:32.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4f78e000 1 -- 172.21.15.47:0/1682135599 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 msgr2=0x55aa643db380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:32.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4f78e000 1 --2- 172.21.15.47:0/1682135599 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa643db380 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c007f60 tx=0x7fcb3c009040 comp rx=0 tx=0).stop 2024-02-19T15:27:32.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4f78e000 1 -- 172.21.15.47:0/1682135599 shutdown_connections 2024-02-19T15:27:32.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4f78e000 1 --2- 172.21.15.47:0/1682135599 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa643db380 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4f78e000 1 --2- 172.21.15.47:0/1682135599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55aa64376920 0x55aa643dae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4f78e000 1 --2- 172.21.15.47:0/1682135599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa6432cf30 0x55aa643da900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4f78e000 1 -- 172.21.15.47:0/1682135599 >> 172.21.15.47:0/1682135599 conn(0x55aa643332a0 msgr2=0x55aa64362030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:32.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4f78e000 1 -- 172.21.15.47:0/1682135599 shutdown_connections 2024-02-19T15:27:32.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4f78e000 1 -- 172.21.15.47:0/1682135599 wait complete. 2024-02-19T15:27:32.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4f78e000 1 -- start start 2024-02-19T15:27:32.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.831+0000 7fcb4f78e000 1 Processor -- start 2024-02-19T15:27:32.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4f78e000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55aa6432cf30 0x55aa643c2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4f78e000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa64376920 0x55aa643863a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4f78e000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa643868e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4f78e000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa64186e80 con 0x55aa64377240 2024-02-19T15:27:32.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4f78e000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa64154980 con 0x55aa64376920 2024-02-19T15:27:32.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4f78e000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aa63e761b0 con 0x55aa6432cf30 2024-02-19T15:27:32.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4f78e000 10 client.0 _ll_register_callbacks cb 0x7fcb3c007f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:27:32.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4a200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa643868e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4a200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa643868e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55450/0 (socket says 172.21.15.47:55450) 2024-02-19T15:27:32.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4a200640 1 -- 172.21.15.47:0/4131866737 learned_addr learned my addr 172.21.15.47:0/4131866737 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:32.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb48e00640 1 --2- 172.21.15.47:0/4131866737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa64376920 0x55aa643863a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb49800640 1 --2- 172.21.15.47:0/4131866737 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55aa6432cf30 0x55aa643c2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4a200640 1 -- 172.21.15.47:0/4131866737 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55aa6432cf30 msgr2=0x55aa643c2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:32.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4a200640 1 --2- 172.21.15.47:0/4131866737 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55aa6432cf30 0x55aa643c2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4a200640 1 -- 172.21.15.47:0/4131866737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa64376920 msgr2=0x55aa643863a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:32.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4a200640 1 --2- 172.21.15.47:0/4131866737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa64376920 0x55aa643863a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4a200640 1 -- 172.21.15.47:0/4131866737 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aa6413ee70 con 0x55aa64377240 2024-02-19T15:27:32.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4a200640 1 --2- 172.21.15.47:0/4131866737 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa643868e0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c023970 tx=0x7fcb3c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfe4b4a3c08f998e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:32.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb2aa00640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:27:32.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb2aa00640 1 -- 172.21.15.47:0/4131866737 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c00d040 con 0x55aa64377240 2024-02-19T15:27:32.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb2aa00640 1 -- 172.21.15.47:0/4131866737 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb3c0189c0 con 0x55aa64377240 2024-02-19T15:27:32.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4f78e000 1 -- 172.21.15.47:0/4131866737 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aa64104e80 con 0x55aa64377240 2024-02-19T15:27:32.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb2aa00640 1 -- 172.21.15.47:0/4131866737 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb3c023a60 con 0x55aa64377240 2024-02-19T15:27:32.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4f78e000 10 client.7824 fetch_fsmap learned FSMap version 75 2024-02-19T15:27:32.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4f78e000 1 -- 172.21.15.47:0/4131866737 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=2+}) v3 -- 0x55aa63e75cc0 con 0x55aa64377240 2024-02-19T15:27:32.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb2aa00640 1 -- 172.21.15.47:0/4131866737 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55aa64104e80 con 0x55aa64377240 2024-02-19T15:27:32.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb2aa00640 1 -- 172.21.15.47:0/4131866737 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 75) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fcb3c039050 con 0x55aa64377240 2024-02-19T15:27:32.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4f78e000 10 client.7824 fetch_fsmap finished waiting for FSMap version 75 2024-02-19T15:27:32.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4f78e000 10 client.7824 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T15:27:32.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4f78e000 20 client.7824 populate_metadata read hostname 'smithi047' 2024-02-19T15:27:32.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4f78e000 1 --2- 172.21.15.47:0/4131866737 >> [v2:172.21.15.47:6832/897369244,v1:172.21.15.47:6834/897369244] conn(0x55aa64386e20 0x55aa64387200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:32.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4f78e000 4 client.7824 mds_command: new command op to 7635 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T15:27:32.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.835+0000 7fcb4f78e000 1 -- 172.21.15.47:0/4131866737 --> [v2:172.21.15.47:6832/897369244,v1:172.21.15.47:6834/897369244] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55aa64147fa0 con 0x55aa64386e20 2024-02-19T15:27:32.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb49800640 1 --2- 172.21.15.47:0/4131866737 >> [v2:172.21.15.47:6832/897369244,v1:172.21.15.47:6834/897369244] conn(0x55aa64386e20 0x55aa64387200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:32.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb2aa00640 10 client.7824 ms_handle_connect on v2:172.21.15.47:6832/897369244 2024-02-19T15:27:32.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb49800640 1 --2- 172.21.15.47:0/4131866737 >> [v2:172.21.15.47:6832/897369244,v1:172.21.15.47:6834/897369244] conn(0x55aa64386e20 0x55aa64387200 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68d62c9fe6c07a08 server_cookie=a31190d820351ae5 in_seq=0 out_seq=0 2024-02-19T15:27:32.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb2aa00640 1 -- 172.21.15.47:0/4131866737 <== mds.0 v2:172.21.15.47:6832/897369244 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+320 (crc 0 0 0) 0x55aa64147fa0 con 0x55aa64386e20 2024-02-19T15:27:32.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb2aa00640 10 client.7824 handle_command_reply: tid=0 2024-02-19T15:27:32.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4f78e000 1 client.7824 shutdown 2024-02-19T15:27:32.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4f78e000 1 -- 172.21.15.47:0/4131866737 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 msgr2=0x55aa643868e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:32.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4f78e000 1 --2- 172.21.15.47:0/4131866737 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa643868e0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c023970 tx=0x7fcb3c017000 comp rx=0 tx=0).stop 2024-02-19T15:27:32.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4f78e000 1 -- 172.21.15.47:0/4131866737 shutdown_connections 2024-02-19T15:27:32.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4f78e000 1 --2- 172.21.15.47:0/4131866737 >> [v2:172.21.15.47:6832/897369244,v1:172.21.15.47:6834/897369244] conn(0x55aa64386e20 0x55aa64387200 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4f78e000 1 --2- 172.21.15.47:0/4131866737 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aa64377240 0x55aa643868e0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4f78e000 1 --2- 172.21.15.47:0/4131866737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa64376920 0x55aa643863a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4f78e000 1 --2- 172.21.15.47:0/4131866737 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55aa6432cf30 0x55aa643c2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb2aa00640 0 client.7824 ms_handle_reset on v2:172.21.15.47:6832/897369244 2024-02-19T15:27:32.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4f78e000 1 -- 172.21.15.47:0/4131866737 >> 172.21.15.47:0/4131866737 conn(0x55aa643332a0 msgr2=0x55aa64362fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:32.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4f78e000 1 -- 172.21.15.47:0/4131866737 shutdown_connections 2024-02-19T15:27:32.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4f78e000 1 -- 172.21.15.47:0/4131866737 wait complete. 2024-02-19T15:27:32.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4f78e000 20 client.7824 trim_cache size 0 max 16384 2024-02-19T15:27:32.846 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:27:32.846 INFO:teuthology.orchestra.run.smithi047.stdout: "cluster_fsid": "ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e", 2024-02-19T15:27:32.846 INFO:teuthology.orchestra.run.smithi047.stdout: "whoami": 0, 2024-02-19T15:27:32.846 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 7635, 2024-02-19T15:27:32.846 INFO:teuthology.orchestra.run.smithi047.stdout: "want_state": "up:active", 2024-02-19T15:27:32.846 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "up:active", 2024-02-19T15:27:32.846 INFO:teuthology.orchestra.run.smithi047.stdout: "fs_name": "cephfs", 2024-02-19T15:27:32.846 INFO:teuthology.orchestra.run.smithi047.stdout: "rank_uptime": 10.81916642, 2024-02-19T15:27:32.846 INFO:teuthology.orchestra.run.smithi047.stdout: "mdsmap_epoch": 75, 2024-02-19T15:27:32.846 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch": 108, 2024-02-19T15:27:32.847 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch_barrier": 108, 2024-02-19T15:27:32.847 INFO:teuthology.orchestra.run.smithi047.stdout: "uptime": 15.680591153 2024-02-19T15:27:32.847 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:27:32.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4c200640 1 -- 172.21.15.47:0/127754975 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcb24066e90 msgr2=0x7fcb24069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:32.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4c200640 1 --2- 172.21.15.47:0/127754975 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcb24066e90 0x7fcb24069350 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c009f90 tx=0x7fcb3c007700 comp rx=0 tx=0).stop 2024-02-19T15:27:32.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4c200640 1 -- 172.21.15.47:0/127754975 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb440cedb0 msgr2=0x7fcb44152c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:32.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4c200640 1 --2- 172.21.15.47:0/127754975 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb440cedb0 0x7fcb44152c00 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800c4b0 tx=0x7fcb3800c970 comp rx=0 tx=0).stop 2024-02-19T15:27:32.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4c200640 1 -- 172.21.15.47:0/127754975 shutdown_connections 2024-02-19T15:27:32.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4c200640 1 --2- 172.21.15.47:0/127754975 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcb24066e90 0x7fcb24069350 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4c200640 1 --2- 172.21.15.47:0/127754975 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb4415bcd0 0x7fcb440b6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4c200640 1 --2- 172.21.15.47:0/127754975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb440d2710 0x7fcb440b6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4c200640 1 --2- 172.21.15.47:0/127754975 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb440cedb0 0x7fcb44152c00 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:32.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4c200640 1 -- 172.21.15.47:0/127754975 >> 172.21.15.47:0/127754975 conn(0x7fcb440d1bf0 msgr2=0x7fcb440b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:32.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4c200640 1 -- 172.21.15.47:0/127754975 shutdown_connections 2024-02-19T15:27:32.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.839+0000 7fcb4c200640 1 -- 172.21.15.47:0/127754975 wait complete. 2024-02-19T15:27:32.862 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-02-19T15:27:33.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.999+0000 7efd48400640 1 Processor -- start 2024-02-19T15:27:33.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.999+0000 7efd48400640 1 -- start start 2024-02-19T15:27:33.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.999+0000 7efd48400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd4006aeb0 0x7efd4006b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:33.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.999+0000 7efd48400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efd400c6780 0x7efd400c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:33.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.999+0000 7efd48400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd400ad840 con 0x7efd400cb520 2024-02-19T15:27:33.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.999+0000 7efd48400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd400c62f0 con 0x7efd400c6780 2024-02-19T15:27:33.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.999+0000 7efd48400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd4006b7d0 con 0x7efd4006aeb0 2024-02-19T15:27:33.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.999+0000 7efd46400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efd400c6780 0x7efd400c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:33.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.999+0000 7efd45000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7efd400cb520 0x7efd400cb9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39848/0 (socket says 172.21.15.47:39848) 2024-02-19T15:27:33.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.999+0000 7efd45000640 1 -- 172.21.15.47:0/1643192896 learned_addr learned my addr 172.21.15.47:0/1643192896 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:33.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.999+0000 7efd45a00640 1 --2- 172.21.15.47:0/1643192896 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd4006aeb0 0x7efd4006b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:33.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.999+0000 7efd46400640 1 -- 172.21.15.47:0/1643192896 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd4006aeb0 msgr2=0x7efd4006b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:33.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.999+0000 7efd46400640 1 --2- 172.21.15.47:0/1643192896 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd4006aeb0 0x7efd4006b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:33.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.999+0000 7efd46400640 1 -- 172.21.15.47:0/1643192896 >> v1:172.21.15.47:6789/0 conn(0x7efd400cb520 legacy=0x7efd400cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:33.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.999+0000 7efd46400640 1 -- 172.21.15.47:0/1643192896 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd400cc810 con 0x7efd400c6780 2024-02-19T15:27:33.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:32.999+0000 7efd3fe00640 1 -- 172.21.15.47:0/1643192896 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366435335 0 0) 0x7efd400ad840 con 0x7efd400cb520 2024-02-19T15:27:33.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.003+0000 7efd46400640 1 --2- 172.21.15.47:0/1643192896 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd400c6780 0x7efd400c6c10 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7efd38009d90 tx=0x7efd3801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cf87b6426f88855 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:33.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.003+0000 7efd3fe00640 1 -- 172.21.15.47:0/1643192896 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd400c7960 con 0x7efd400c6780 2024-02-19T15:27:33.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.003+0000 7efd3fe00640 1 -- 172.21.15.47:0/1643192896 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd380a5a00 con 0x7efd400c6780 2024-02-19T15:27:33.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.003+0000 7efd3fe00640 1 -- 172.21.15.47:0/1643192896 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd380a5d90 con 0x7efd400c6780 2024-02-19T15:27:33.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.003+0000 7efd48400640 1 -- 172.21.15.47:0/1643192896 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd400c6780 msgr2=0x7efd400c6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:33.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.003+0000 7efd48400640 1 --2- 172.21.15.47:0/1643192896 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd400c6780 0x7efd400c6c10 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7efd38009d90 tx=0x7efd3801c940 comp rx=0 tx=0).stop 2024-02-19T15:27:33.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.003+0000 7efd48400640 1 -- 172.21.15.47:0/1643192896 shutdown_connections 2024-02-19T15:27:33.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.003+0000 7efd48400640 1 --2- 172.21.15.47:0/1643192896 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd400c6780 0x7efd400c6c10 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:33.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.003+0000 7efd48400640 1 --2- 172.21.15.47:0/1643192896 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd4006aeb0 0x7efd4006b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:33.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.003+0000 7efd48400640 1 -- 172.21.15.47:0/1643192896 >> 172.21.15.47:0/1643192896 conn(0x7efd400d0ad0 msgr2=0x7efd400d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:33.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.003+0000 7efd48400640 1 -- 172.21.15.47:0/1643192896 shutdown_connections 2024-02-19T15:27:33.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.003+0000 7efd48400640 1 -- 172.21.15.47:0/1643192896 wait complete. 2024-02-19T15:27:33.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd48400640 1 Processor -- start 2024-02-19T15:27:33.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd48400640 1 -- start start 2024-02-19T15:27:33.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd48400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd4006aeb0 0x7efd400ad840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:33.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd48400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd400c6780 0x7efd40159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:33.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd48400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd400cb520 0x7efd40159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:33.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd48400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd40159a80 con 0x7efd400cb520 2024-02-19T15:27:33.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd48400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd40161440 con 0x7efd4006aeb0 2024-02-19T15:27:33.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd48400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd40154cc0 con 0x7efd400c6780 2024-02-19T15:27:33.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd46400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd400cb520 0x7efd40159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:33.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd46400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd400cb520 0x7efd40159540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55472/0 (socket says 172.21.15.47:55472) 2024-02-19T15:27:33.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd46400640 1 -- 172.21.15.47:0/690270974 learned_addr learned my addr 172.21.15.47:0/690270974 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:33.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd45000640 1 --2- 172.21.15.47:0/690270974 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd400c6780 0x7efd40159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:33.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd45a00640 1 --2- 172.21.15.47:0/690270974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd4006aeb0 0x7efd400ad840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:33.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd46400640 1 -- 172.21.15.47:0/690270974 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd400c6780 msgr2=0x7efd40159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:33.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd46400640 1 --2- 172.21.15.47:0/690270974 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd400c6780 0x7efd40159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:33.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd46400640 1 -- 172.21.15.47:0/690270974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd4006aeb0 msgr2=0x7efd400ad840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:33.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd46400640 1 --2- 172.21.15.47:0/690270974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd4006aeb0 0x7efd400ad840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:33.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd46400640 1 -- 172.21.15.47:0/690270974 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd38009a70 con 0x7efd400cb520 2024-02-19T15:27:33.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd46400640 1 --2- 172.21.15.47:0/690270974 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd400cb520 0x7efd40159540 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7efd380b0040 tx=0x7efd380196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:33.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd3d600640 1 -- 172.21.15.47:0/690270974 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd380b9660 con 0x7efd400cb520 2024-02-19T15:27:33.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd3d600640 1 -- 172.21.15.47:0/690270974 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd380b97c0 con 0x7efd400cb520 2024-02-19T15:27:33.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd3d600640 1 -- 172.21.15.47:0/690270974 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd380b9b50 con 0x7efd400cb520 2024-02-19T15:27:33.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd48400640 1 -- 172.21.15.47:0/690270974 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd400add80 con 0x7efd400cb520 2024-02-19T15:27:33.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd48400640 1 -- 172.21.15.47:0/690270974 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd4015f2a0 con 0x7efd400cb520 2024-02-19T15:27:33.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.007+0000 7efd48400640 1 -- 172.21.15.47:0/690270974 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd40075360 con 0x7efd400cb520 2024-02-19T15:27:33.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.011+0000 7efd3d600640 1 -- 172.21.15.47:0/690270974 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efd380b9cb0 con 0x7efd400cb520 2024-02-19T15:27:33.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.011+0000 7efd3d600640 1 --2- 172.21.15.47:0/690270974 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd20066e40 0x7efd20069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:33.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.011+0000 7efd3d600640 1 -- 172.21.15.47:0/690270974 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd3811fc80 con 0x7efd400cb520 2024-02-19T15:27:33.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.011+0000 7efd45a00640 1 --2- 172.21.15.47:0/690270974 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd20066e40 0x7efd20069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:33.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.011+0000 7efd45a00640 1 --2- 172.21.15.47:0/690270974 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd20066e40 0x7efd20069300 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7efd3400b9d0 tx=0x7efd34008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:33.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.015+0000 7efd3d600640 1 -- 172.21.15.47:0/690270974 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7efd380eca60 con 0x7efd400cb520 2024-02-19T15:27:33.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.215+0000 7efd48400640 1 -- 172.21.15.47:0/690270974 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7efd4006ef20 con 0x7efd400cb520 2024-02-19T15:27:33.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.215+0000 7efd3d600640 1 -- 172.21.15.47:0/690270974 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v75) v1 ==== 53+0+73 (secure 0 0 0) 0x7efd380f0120 con 0x7efd400cb520 2024-02-19T15:27:33.220 INFO:teuthology.orchestra.run.smithi047.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-02-19T15:27:33.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.219+0000 7efd48400640 1 -- 172.21.15.47:0/690270974 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd20066e40 msgr2=0x7efd20069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:33.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.219+0000 7efd48400640 1 --2- 172.21.15.47:0/690270974 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd20066e40 0x7efd20069300 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7efd3400b9d0 tx=0x7efd34008040 comp rx=0 tx=0).stop 2024-02-19T15:27:33.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.219+0000 7efd48400640 1 -- 172.21.15.47:0/690270974 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd400cb520 msgr2=0x7efd40159540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:33.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.219+0000 7efd48400640 1 --2- 172.21.15.47:0/690270974 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd400cb520 0x7efd40159540 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7efd380b0040 tx=0x7efd380196e0 comp rx=0 tx=0).stop 2024-02-19T15:27:33.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.219+0000 7efd48400640 1 -- 172.21.15.47:0/690270974 shutdown_connections 2024-02-19T15:27:33.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.219+0000 7efd48400640 1 --2- 172.21.15.47:0/690270974 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd20066e40 0x7efd20069300 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:33.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.219+0000 7efd48400640 1 --2- 172.21.15.47:0/690270974 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd400cb520 0x7efd40159540 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:33.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.219+0000 7efd48400640 1 --2- 172.21.15.47:0/690270974 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd400c6780 0x7efd40159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:33.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.219+0000 7efd48400640 1 --2- 172.21.15.47:0/690270974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd4006aeb0 0x7efd400ad840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:33.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.219+0000 7efd48400640 1 -- 172.21.15.47:0/690270974 >> 172.21.15.47:0/690270974 conn(0x7efd400d0ad0 msgr2=0x7efd400ce660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:33.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.219+0000 7efd48400640 1 -- 172.21.15.47:0/690270974 shutdown_connections 2024-02-19T15:27:33.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:33.219+0000 7efd48400640 1 -- 172.21.15.47:0/690270974 wait complete. 2024-02-19T15:27:33.241 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-02-19T15:27:33.241 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T15:27:33.241 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi047.front.sepia.ceph.com 2024-02-19T15:27:33.241 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-19T15:27:33.241 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-19T15:27:33.241 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-02-19T15:27:33.242 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-02-19T15:27:33.242 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T15:27:33.242 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:27:33.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:33.242 DEBUG:teuthology.orchestra.run.smithi047:> ip addr 2024-02-19T15:27:33.251 INFO:teuthology.orchestra.run.smithi047.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T15:27:33.251 INFO:teuthology.orchestra.run.smithi047.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T15:27:33.251 INFO:teuthology.orchestra.run.smithi047.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T15:27:33.251 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:27:33.252 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 ::1/128 scope host 2024-02-19T15:27:33.252 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:27:33.252 INFO:teuthology.orchestra.run.smithi047.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:27:33.252 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:78 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:27:33.252 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f0 2024-02-19T15:27:33.252 INFO:teuthology.orchestra.run.smithi047.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T15:27:33.252 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8e brd ff:ff:ff:ff:ff:ff 2024-02-19T15:27:33.252 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f0 2024-02-19T15:27:33.252 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.21.15.47/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-02-19T15:27:33.252 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft 42162sec preferred_lft 42162sec 2024-02-19T15:27:33.252 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::ec4:7aff:febd:148e/64 scope link 2024-02-19T15:27:33.252 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:27:33.252 INFO:teuthology.orchestra.run.smithi047.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:27:33.252 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:79 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:27:33.253 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f1 2024-02-19T15:27:33.253 INFO:teuthology.orchestra.run.smithi047.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:27:33.253 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8f brd ff:ff:ff:ff:ff:ff 2024-02-19T15:27:33.253 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f1 2024-02-19T15:27:33.253 INFO:teuthology.orchestra.run.smithi047.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-02-19T15:27:33.253 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 02:42:47:cf:b1:62 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:27:33.253 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-02-19T15:27:33.253 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:27:33.253 INFO:teuthology.orchestra.run.smithi047.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-02-19T15:27:33.253 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-02-19T15:27:33.253 INFO:teuthology.orchestra.run.smithi047.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-02-19T15:27:33.253 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:27:33.253 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-02-19T15:27:33.253 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:27:33.254 INFO:teuthology.orchestra.run.smithi047.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:27:33.254 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-19T15:27:33.254 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-02-19T15:27:33.254 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:27:33.254 INFO:teuthology.orchestra.run.smithi047.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:27:33.254 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-02-19T15:27:33.254 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-02-19T15:27:33.254 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:27:33.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:33.254 DEBUG:teuthology.orchestra.run.smithi047:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T15:27:33.310 INFO:teuthology.orchestra.run.smithi047.stdout:1 2024-02-19T15:27:33.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:33.313 DEBUG:teuthology.orchestra.run.smithi047:> route 2024-02-19T15:27:33.362 INFO:teuthology.orchestra.run.smithi047.stdout:Kernel IP routing table 2024-02-19T15:27:33.362 INFO:teuthology.orchestra.run.smithi047.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T15:27:33.362 INFO:teuthology.orchestra.run.smithi047.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-02-19T15:27:33.362 INFO:teuthology.orchestra.run.smithi047.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-02-19T15:27:33.362 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-02-19T15:27:33.363 INFO:teuthology.orchestra.run.smithi047.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T15:27:33.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:33.363 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2024-02-19T15:27:33.363 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2024-02-19T15:27:33.363 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-02-19T15:27:33.363 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-02-19T15:27:33.363 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-02-19T15:27:33.363 DEBUG:teuthology.orchestra.run.smithi047:> ') 2024-02-19T15:27:33.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:33.455 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list 2024-02-19T15:27:33.464 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-02-19T15:27:33.464 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-02-19T15:27:33.464 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:27:33.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:33.464 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set brx.0 up 2024-02-19T15:27:33.524 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:27:33.524 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-19T15:27:33.573 INFO:teuthology.orchestra.run.smithi047.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:27:33.573 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:27:33.573 DEBUG:teuthology.orchestra.run.smithi047:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:27:33.624 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-19T15:27:33.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:33.625 DEBUG:teuthology.orchestra.run.smithi047:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:27:33.713 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:27:33.713 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:27:33.713 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:27:33.728 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:27:33.728 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:27:33.728 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:27:33.728 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:27:33.728 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:27:33.737 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:27:33.737 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:27:33.737 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:27:33.738 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:27:33.738 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:27:33.738 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790" 2024-02-19T15:27:33.738 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790:/ 2024-02-19T15:27:33.738 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e" will pass to kernel 2024-02-19T15:27:33.742 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T15:27:33.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:33.742 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:27:33.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:33.763 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:27:33.821 INFO:teuthology.orchestra.run.smithi047.stdout:client6686 2024-02-19T15:27:33.821 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:27:33.821 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:27:33.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:33.877 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_1) 2024-02-19T15:27:33.934 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:27:33.934 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:27:33.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:33.985 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=dir_1/file_a bs=1M conv=fdatasync count=8 seek=0) 2024-02-19T15:27:34.166 INFO:teuthology.orchestra.run.smithi047.stderr:8+0 records in 2024-02-19T15:27:34.166 INFO:teuthology.orchestra.run.smithi047.stderr:8+0 records out 2024-02-19T15:27:34.166 INFO:teuthology.orchestra.run.smithi047.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.127843 s, 65.6 MB/s 2024-02-19T15:27:34.167 DEBUG:teuthology.orchestra.run.smithi047:> stdin-killer -- python3 -c ' 2024-02-19T15:27:34.167 DEBUG:teuthology.orchestra.run.smithi047:> import os 2024-02-19T15:27:34.167 DEBUG:teuthology.orchestra.run.smithi047:> import stat 2024-02-19T15:27:34.167 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:27:34.167 DEBUG:teuthology.orchestra.run.smithi047:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_1/file_a").st_ino) 2024-02-19T15:27:34.167 DEBUG:teuthology.orchestra.run.smithi047:> ' 2024-02-19T15:27:34.232 INFO:teuthology.orchestra.run.smithi047.stdout:1099511627777 2024-02-19T15:27:34.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-02-19T15:27:34.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:34.242 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_2) 2024-02-19T15:27:34.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:34.295 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_1/file_a dir_2/file_b) 2024-02-19T15:27:34.349 DEBUG:teuthology.orchestra.run.smithi047:> stdin-killer -- python3 -c ' 2024-02-19T15:27:34.350 DEBUG:teuthology.orchestra.run.smithi047:> import os 2024-02-19T15:27:34.350 DEBUG:teuthology.orchestra.run.smithi047:> import stat 2024-02-19T15:27:34.350 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:27:34.350 DEBUG:teuthology.orchestra.run.smithi047:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_2/file_b").st_ino) 2024-02-19T15:27:34.350 DEBUG:teuthology.orchestra.run.smithi047:> ' 2024-02-19T15:27:34.459 INFO:teuthology.orchestra.run.smithi047.stdout:1099511627777 2024-02-19T15:27:34.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-02-19T15:27:34.469 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:27:34.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.643+0000 7f62c4000640 1 Processor -- start 2024-02-19T15:27:34.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.643+0000 7f62c4000640 1 -- start start 2024-02-19T15:27:34.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.643+0000 7f62c4000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f62bc1613a0 0x7f62bc0adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:34.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.643+0000 7f62c4000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62bc06fdf0 0x7f62bc06e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:34.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.643+0000 7f62c4000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f62bc06e5f0 0x7f62bc06ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:34.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.643+0000 7f62c4000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62bc161780 con 0x7f62bc06e5f0 2024-02-19T15:27:34.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.643+0000 7f62c4000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62bc0b01a0 con 0x7f62bc1613a0 2024-02-19T15:27:34.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.643+0000 7f62c4000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62bc0ae3c0 con 0x7f62bc06fdf0 2024-02-19T15:27:34.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.643+0000 7f62c2000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f62bc06e5f0 0x7f62bc06ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:34.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.647+0000 7f62c2000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f62bc06e5f0 0x7f62bc06ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60790/0 (socket says 172.21.15.47:60790) 2024-02-19T15:27:34.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.647+0000 7f62c2000640 1 -- 172.21.15.47:0/2785881955 learned_addr learned my addr 172.21.15.47:0/2785881955 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:34.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.647+0000 7f62c1600640 1 --2- 172.21.15.47:0/2785881955 >> v2:172.21.15.116:3300/0 conn(0x7f62bc1613a0 0x7f62bc0adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:34.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.647+0000 7f62c0c00640 1 --2- 172.21.15.47:0/2785881955 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62bc06fdf0 0x7f62bc06e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:34.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.647+0000 7f62c2000640 1 -- 172.21.15.47:0/2785881955 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62bc06fdf0 msgr2=0x7f62bc06e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:34.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.647+0000 7f62c2000640 1 --2- 172.21.15.47:0/2785881955 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62bc06fdf0 0x7f62bc06e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:34.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.647+0000 7f62c2000640 1 -- 172.21.15.47:0/2785881955 >> v2:172.21.15.116:3300/0 conn(0x7f62bc1613a0 msgr2=0x7f62bc0adbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:34.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.647+0000 7f62c2000640 1 --2- 172.21.15.47:0/2785881955 >> v2:172.21.15.116:3300/0 conn(0x7f62bc1613a0 0x7f62bc0adbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:34.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.647+0000 7f62c2000640 1 -- 172.21.15.47:0/2785881955 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62bc0ae6a0 con 0x7f62bc06e5f0 2024-02-19T15:27:34.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.647+0000 7f62c2000640 1 --2- 172.21.15.47:0/2785881955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62bc06e5f0 0x7f62bc06ea80 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f62bc0d1bd0 tx=0x7f62b401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bf2b7ea48478e33 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:34.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.647+0000 7f62bbe00640 1 -- 172.21.15.47:0/2785881955 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b40a56f0 con 0x7f62bc06e5f0 2024-02-19T15:27:34.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.647+0000 7f62bbe00640 1 -- 172.21.15.47:0/2785881955 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b40a5850 con 0x7f62bc06e5f0 2024-02-19T15:27:34.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.647+0000 7f62bbe00640 1 -- 172.21.15.47:0/2785881955 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b40a5be0 con 0x7f62bc06e5f0 2024-02-19T15:27:34.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.647+0000 7f62c4000640 1 -- 172.21.15.47:0/2785881955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62bc06e5f0 msgr2=0x7f62bc06ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:34.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.647+0000 7f62c4000640 1 --2- 172.21.15.47:0/2785881955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62bc06e5f0 0x7f62bc06ea80 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f62bc0d1bd0 tx=0x7f62b401c960 comp rx=0 tx=0).stop 2024-02-19T15:27:34.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c4000640 1 -- 172.21.15.47:0/2785881955 shutdown_connections 2024-02-19T15:27:34.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c4000640 1 --2- 172.21.15.47:0/2785881955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62bc06e5f0 0x7f62bc06ea80 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:34.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c4000640 1 --2- 172.21.15.47:0/2785881955 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62bc06fdf0 0x7f62bc06e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:34.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c4000640 1 --2- 172.21.15.47:0/2785881955 >> v2:172.21.15.116:3300/0 conn(0x7f62bc1613a0 0x7f62bc0adbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:34.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c4000640 1 -- 172.21.15.47:0/2785881955 >> 172.21.15.47:0/2785881955 conn(0x7f62bc0d0ad0 msgr2=0x7f62bc0d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:34.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c4000640 1 -- 172.21.15.47:0/2785881955 shutdown_connections 2024-02-19T15:27:34.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c4000640 1 -- 172.21.15.47:0/2785881955 wait complete. 2024-02-19T15:27:34.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c4000640 1 Processor -- start 2024-02-19T15:27:34.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c4000640 1 -- start start 2024-02-19T15:27:34.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c4000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62bc06e5f0 0x7f62bc157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:34.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c4000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62bc06fdf0 0x7f62bc148bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:34.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c4000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62bc1613a0 0x7f62bc14d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:34.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c4000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62bc14df30 con 0x7f62bc1613a0 2024-02-19T15:27:34.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c4000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62bc14dcb0 con 0x7f62bc06fdf0 2024-02-19T15:27:34.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c4000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62bc14e2d0 con 0x7f62bc06e5f0 2024-02-19T15:27:34.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c2000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62bc1613a0 0x7f62bc14d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:34.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c2000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62bc1613a0 0x7f62bc14d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60798/0 (socket says 172.21.15.47:60798) 2024-02-19T15:27:34.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c2000640 1 -- 172.21.15.47:0/1393826063 learned_addr learned my addr 172.21.15.47:0/1393826063 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:34.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c0c00640 1 --2- 172.21.15.47:0/1393826063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62bc06fdf0 0x7f62bc148bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:34.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c1600640 1 --2- 172.21.15.47:0/1393826063 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62bc06e5f0 0x7f62bc157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:34.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c2000640 1 -- 172.21.15.47:0/1393826063 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62bc06e5f0 msgr2=0x7f62bc157d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:34.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c2000640 1 --2- 172.21.15.47:0/1393826063 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62bc06e5f0 0x7f62bc157d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:34.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c2000640 1 -- 172.21.15.47:0/1393826063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62bc06fdf0 msgr2=0x7f62bc148bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:34.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c2000640 1 --2- 172.21.15.47:0/1393826063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62bc06fdf0 0x7f62bc148bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:34.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c2000640 1 -- 172.21.15.47:0/1393826063 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b4009b30 con 0x7f62bc1613a0 2024-02-19T15:27:34.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.651+0000 7f62c2000640 1 --2- 172.21.15.47:0/1393826063 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62bc1613a0 0x7f62bc14d770 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f62b401b010 tx=0x7f62b4019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:34.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.655+0000 7f62b9600640 1 -- 172.21.15.47:0/1393826063 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b40b8670 con 0x7f62bc1613a0 2024-02-19T15:27:34.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.655+0000 7f62b9600640 1 -- 172.21.15.47:0/1393826063 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b40b87d0 con 0x7f62bc1613a0 2024-02-19T15:27:34.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.655+0000 7f62c4000640 1 -- 172.21.15.47:0/1393826063 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62bc14e5b0 con 0x7f62bc1613a0 2024-02-19T15:27:34.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.655+0000 7f62b9600640 1 -- 172.21.15.47:0/1393826063 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b40b8bc0 con 0x7f62bc1613a0 2024-02-19T15:27:34.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.655+0000 7f62c4000640 1 -- 172.21.15.47:0/1393826063 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62bc158310 con 0x7f62bc1613a0 2024-02-19T15:27:34.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.655+0000 7f62c4000640 1 -- 172.21.15.47:0/1393826063 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62bc149230 con 0x7f62bc1613a0 2024-02-19T15:27:34.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.655+0000 7f62b9600640 1 -- 172.21.15.47:0/1393826063 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f62b40b8d20 con 0x7f62bc1613a0 2024-02-19T15:27:34.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.655+0000 7f62b9600640 1 --2- 172.21.15.47:0/1393826063 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f629c066e40 0x7f629c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:34.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.655+0000 7f62b9600640 1 -- 172.21.15.47:0/1393826063 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62b411ff80 con 0x7f62bc1613a0 2024-02-19T15:27:34.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.655+0000 7f62c1600640 1 --2- 172.21.15.47:0/1393826063 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f629c066e40 0x7f629c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:34.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.659+0000 7f62c1600640 1 --2- 172.21.15.47:0/1393826063 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f629c066e40 0x7f629c069300 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f62b0007940 tx=0x7f62b0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:34.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.659+0000 7f62b9600640 1 -- 172.21.15.47:0/1393826063 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f62b40ecf40 con 0x7f62bc1613a0 2024-02-19T15:27:34.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.855+0000 7f62c4000640 1 -- 172.21.15.47:0/1393826063 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62bc0d0810 con 0x7f62bc1613a0 2024-02-19T15:27:34.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.855+0000 7f62b9600640 1 -- 172.21.15.47:0/1393826063 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4937 (secure 0 0 0) 0x7f62b40f0600 con 0x7f62bc1613a0 2024-02-19T15:27:34.859 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:34.859 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":75,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6578,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/408259797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":408259797},{"type":"v1","addr":"172.21.15.116:6836","nonce":408259797}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7327,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3795002232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3795002232},{"type":"v1","addr":"172.21.15.47:6835","nonce":3795002232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7330,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/819751716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":819751716},{"type":"v1","addr":"172.21.15.116:6837","nonce":819751716}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:24.428431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6834/897369244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":897369244},{"type":"v1","addr":"172.21.15.47:6834","nonce":897369244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:27:34.859 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 75 2024-02-19T15:27:34.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.855+0000 7f62c4000640 1 -- 172.21.15.47:0/1393826063 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f629c066e40 msgr2=0x7f629c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:34.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.855+0000 7f62c4000640 1 --2- 172.21.15.47:0/1393826063 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f629c066e40 0x7f629c069300 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f62b0007940 tx=0x7f62b0008040 comp rx=0 tx=0).stop 2024-02-19T15:27:34.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.855+0000 7f62c4000640 1 -- 172.21.15.47:0/1393826063 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62bc1613a0 msgr2=0x7f62bc14d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:34.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.855+0000 7f62c4000640 1 --2- 172.21.15.47:0/1393826063 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62bc1613a0 0x7f62bc14d770 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f62b401b010 tx=0x7f62b4019700 comp rx=0 tx=0).stop 2024-02-19T15:27:34.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.855+0000 7f62c4000640 1 -- 172.21.15.47:0/1393826063 shutdown_connections 2024-02-19T15:27:34.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.855+0000 7f62c4000640 1 --2- 172.21.15.47:0/1393826063 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f629c066e40 0x7f629c069300 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:34.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.855+0000 7f62c4000640 1 --2- 172.21.15.47:0/1393826063 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62bc1613a0 0x7f62bc14d770 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:34.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.855+0000 7f62c4000640 1 --2- 172.21.15.47:0/1393826063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62bc06fdf0 0x7f62bc148bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:34.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.855+0000 7f62c4000640 1 --2- 172.21.15.47:0/1393826063 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62bc06e5f0 0x7f62bc157d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:34.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.855+0000 7f62c4000640 1 -- 172.21.15.47:0/1393826063 >> 172.21.15.47:0/1393826063 conn(0x7f62bc0d0ad0 msgr2=0x7f62bc0b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:34.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.855+0000 7f62c4000640 1 -- 172.21.15.47:0/1393826063 shutdown_connections 2024-02-19T15:27:34.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:34.855+0000 7f62c4000640 1 -- 172.21.15.47:0/1393826063 wait complete. 2024-02-19T15:27:34.880 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-02-19T15:27:35.050 INFO:teuthology.orchestra.run.smithi047.stdout:{"message":"","return_code":0} 2024-02-19T15:27:35.050 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-02-19T15:27:35.050 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-02-19T15:27:35.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.091+0000 7f0463647080 1 Processor -- start 2024-02-19T15:27:35.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.091+0000 7f0463647080 1 -- start start 2024-02-19T15:27:35.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f0463647080 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55ce4ea2b990 0x55ce4e9b6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f0463647080 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ce4ea2bd70 0x55ce4e9ba180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f0463647080 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce4ea2c250 0x55ce4e9ba6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f0463647080 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce4e8ba1e0 con 0x55ce4ea2bd70 2024-02-19T15:27:35.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f0463647080 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce4e8b9e80 con 0x55ce4ea2c250 2024-02-19T15:27:35.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f0463647080 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ce4e8dd610 con 0x55ce4ea2b990 2024-02-19T15:27:35.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f045f200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ce4ea2bd70 0x55ce4e9ba180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f0460600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce4ea2c250 0x55ce4e9ba6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f0460600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce4ea2c250 0x55ce4e9ba6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:56102/0 (socket says 172.21.15.47:56102) 2024-02-19T15:27:35.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f045f200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ce4ea2bd70 0x55ce4e9ba180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60824/0 (socket says 172.21.15.47:60824) 2024-02-19T15:27:35.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f0460600640 1 -- 172.21.15.47:0/574685875 learned_addr learned my addr 172.21.15.47:0/574685875 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:35.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f045fc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55ce4ea2b990 0x55ce4e9b6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f045f200640 1 -- 172.21.15.47:0/574685875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55ce4ea2b990 msgr2=0x55ce4e9b6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:35.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f045f200640 1 --2- 172.21.15.47:0/574685875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55ce4ea2b990 0x55ce4e9b6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f045f200640 1 -- 172.21.15.47:0/574685875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce4ea2c250 msgr2=0x55ce4e9ba6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:35.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f045f200640 1 --2- 172.21.15.47:0/574685875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce4ea2c250 0x55ce4e9ba6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f045f200640 1 -- 172.21.15.47:0/574685875 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0450005df0 con 0x55ce4ea2bd70 2024-02-19T15:27:35.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f045f200640 1 --2- 172.21.15.47:0/574685875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ce4ea2bd70 0x55ce4e9ba180 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f044c001d30 tx=0x7f044c00b5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c554ab6f619874 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:35.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f045de00640 1 -- 172.21.15.47:0/574685875 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044c014070 con 0x55ce4ea2bd70 2024-02-19T15:27:35.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f045de00640 1 -- 172.21.15.47:0/574685875 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044c00f410 con 0x55ce4ea2bd70 2024-02-19T15:27:35.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f045de00640 1 -- 172.21.15.47:0/574685875 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044c00f7a0 con 0x55ce4ea2bd70 2024-02-19T15:27:35.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f0463647080 1 -- 172.21.15.47:0/574685875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ce4ea2bd70 msgr2=0x55ce4e9ba180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:35.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f0463647080 1 --2- 172.21.15.47:0/574685875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ce4ea2bd70 0x55ce4e9ba180 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f044c001d30 tx=0x7f044c00b5c0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f0463647080 1 -- 172.21.15.47:0/574685875 shutdown_connections 2024-02-19T15:27:35.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f0463647080 1 --2- 172.21.15.47:0/574685875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce4ea2c250 0x55ce4e9ba6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f0463647080 1 --2- 172.21.15.47:0/574685875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ce4ea2bd70 0x55ce4e9ba180 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f0463647080 1 --2- 172.21.15.47:0/574685875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55ce4ea2b990 0x55ce4e9b6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f0463647080 1 -- 172.21.15.47:0/574685875 >> 172.21.15.47:0/574685875 conn(0x55ce4e973670 msgr2=0x55ce4e974a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:35.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f0463647080 1 -- 172.21.15.47:0/574685875 shutdown_connections 2024-02-19T15:27:35.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f0463647080 1 -- 172.21.15.47:0/574685875 wait complete. 2024-02-19T15:27:35.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f0463647080 1 Processor -- start 2024-02-19T15:27:35.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.095+0000 7f0463647080 1 -- start start 2024-02-19T15:27:35.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f0463647080 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce4ea2b990 0x55ce4e996bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f0463647080 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ce4ea2bd70 0x55ce4ea4b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f0463647080 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55ce4ea2c250 0x55ce4eae3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f0463647080 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce4e8ad740 con 0x55ce4ea2bd70 2024-02-19T15:27:35.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f0463647080 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce4e995ca0 con 0x55ce4ea2b990 2024-02-19T15:27:35.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f0463647080 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ce4e995f40 con 0x55ce4ea2c250 2024-02-19T15:27:35.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f045f200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ce4ea2bd70 0x55ce4ea4b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f045f200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ce4ea2bd70 0x55ce4ea4b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60828/0 (socket says 172.21.15.47:60828) 2024-02-19T15:27:35.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f045f200640 1 -- 172.21.15.47:0/4117671911 learned_addr learned my addr 172.21.15.47:0/4117671911 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:35.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f0460600640 1 --2- 172.21.15.47:0/4117671911 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55ce4ea2c250 0x55ce4eae3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f045fc00640 1 --2- 172.21.15.47:0/4117671911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce4ea2b990 0x55ce4e996bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f045f200640 1 -- 172.21.15.47:0/4117671911 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55ce4ea2c250 msgr2=0x55ce4eae3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:35.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f045f200640 1 --2- 172.21.15.47:0/4117671911 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55ce4ea2c250 0x55ce4eae3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f045f200640 1 -- 172.21.15.47:0/4117671911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce4ea2b990 msgr2=0x55ce4e996bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:35.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f045f200640 1 --2- 172.21.15.47:0/4117671911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce4ea2b990 0x55ce4e996bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f045f200640 1 -- 172.21.15.47:0/4117671911 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce4e8b8ba0 con 0x55ce4ea2bd70 2024-02-19T15:27:35.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f045f200640 1 --2- 172.21.15.47:0/4117671911 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ce4ea2bd70 0x55ce4ea4b670 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f044c00bac0 tx=0x7f044c000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:35.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f045ca00640 1 -- 172.21.15.47:0/4117671911 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044c014040 con 0x55ce4ea2bd70 2024-02-19T15:27:35.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f045ca00640 1 -- 172.21.15.47:0/4117671911 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044c01b840 con 0x55ce4ea2bd70 2024-02-19T15:27:35.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f0463647080 1 -- 172.21.15.47:0/4117671911 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55ce4e8c5d80 con 0x55ce4ea2bd70 2024-02-19T15:27:35.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f045ca00640 1 -- 172.21.15.47:0/4117671911 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044c01bbd0 con 0x55ce4ea2bd70 2024-02-19T15:27:35.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f0463647080 1 -- 172.21.15.47:0/4117671911 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ce4e8b8f30 con 0x55ce4ea2bd70 2024-02-19T15:27:35.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.099+0000 7f045ca00640 1 -- 172.21.15.47:0/4117671911 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f044c01bd30 con 0x55ce4ea2bd70 2024-02-19T15:27:35.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.103+0000 7f045ca00640 1 --2- 172.21.15.47:0/4117671911 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0440065850 0x7f0440067d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.103+0000 7f045ca00640 1 -- 172.21.15.47:0/4117671911 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f044c084e30 con 0x55ce4ea2bd70 2024-02-19T15:27:35.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.103+0000 7f0463647080 1 --2- 172.21.15.47:0/4117671911 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x55ce4e997100 0x55ce4eaeef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.103+0000 7f0463647080 1 -- 172.21.15.47:0/4117671911 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108) v8 -- 0x55ce4ea52720 con 0x55ce4e997100 2024-02-19T15:27:35.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.103+0000 7f045fc00640 1 --2- 172.21.15.47:0/4117671911 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0440065850 0x7f0440067d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.103+0000 7f0460600640 1 --2- 172.21.15.47:0/4117671911 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x55ce4e997100 0x55ce4eaeef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.103+0000 7f0460600640 1 --2- 172.21.15.47:0/4117671911 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x55ce4e997100 0x55ce4eaeef60 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:35.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.103+0000 7f045fc00640 1 --2- 172.21.15.47:0/4117671911 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0440065850 0x7f0440067d10 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f04500ac000 tx=0x7f04500a5000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:35.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.103+0000 7f0460600640 1 -- 172.21.15.47:0/4117671911 <== osd.0 v2:172.21.15.47:6809/1968337531 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f045800ba40 con 0x55ce4e997100 2024-02-19T15:27:35.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.103+0000 7f0463647080 1 -- 172.21.15.47:0/4117671911 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x55ce4e997100 msgr2=0x55ce4eaeef60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:35.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.103+0000 7f0463647080 1 --2- 172.21.15.47:0/4117671911 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x55ce4e997100 0x55ce4eaeef60 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.103+0000 7f0463647080 1 -- 172.21.15.47:0/4117671911 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0440065850 msgr2=0x7f0440067d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:35.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.103+0000 7f0463647080 1 --2- 172.21.15.47:0/4117671911 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0440065850 0x7f0440067d10 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f04500ac000 tx=0x7f04500a5000 comp rx=0 tx=0).stop 2024-02-19T15:27:35.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.103+0000 7f0463647080 1 -- 172.21.15.47:0/4117671911 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ce4ea2bd70 msgr2=0x55ce4ea4b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:35.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.103+0000 7f0463647080 1 --2- 172.21.15.47:0/4117671911 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ce4ea2bd70 0x55ce4ea4b670 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f044c00bac0 tx=0x7f044c000bd0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.103+0000 7f0460600640 1 -- 172.21.15.47:0/4117671911 reap_dead start 2024-02-19T15:27:35.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.103+0000 7f0463647080 1 -- 172.21.15.47:0/4117671911 shutdown_connections 2024-02-19T15:27:35.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.103+0000 7f0463647080 1 -- 172.21.15.47:0/4117671911 >> 172.21.15.47:0/4117671911 conn(0x55ce4e973670 msgr2=0x55ce4e9b6750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:35.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.103+0000 7f0463647080 1 -- 172.21.15.47:0/4117671911 shutdown_connections 2024-02-19T15:27:35.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.103+0000 7f0463647080 1 -- 172.21.15.47:0/4117671911 wait complete. 2024-02-19T15:27:35.111 INFO:teuthology.orchestra.run.smithi047.stdout:Wfile_adir_1 2024-02-19T15:27:35.112 DEBUG:teuthology.orchestra.run.smithi047:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-02-19T15:27:35.205 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:27:35.205 INFO:teuthology.orchestra.run.smithi047.stdout: "ino": 1099511627777, 2024-02-19T15:27:35.205 INFO:teuthology.orchestra.run.smithi047.stdout: "ancestors": [ 2024-02-19T15:27:35.205 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:35.205 INFO:teuthology.orchestra.run.smithi047.stdout: "dirino": 1099511627776, 2024-02-19T15:27:35.205 INFO:teuthology.orchestra.run.smithi047.stdout: "dname": "file_a", 2024-02-19T15:27:35.205 INFO:teuthology.orchestra.run.smithi047.stdout: "version": 8 2024-02-19T15:27:35.205 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:27:35.205 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:27:35.205 INFO:teuthology.orchestra.run.smithi047.stdout: "dirino": 1, 2024-02-19T15:27:35.205 INFO:teuthology.orchestra.run.smithi047.stdout: "dname": "dir_1", 2024-02-19T15:27:35.205 INFO:teuthology.orchestra.run.smithi047.stdout: "version": 15 2024-02-19T15:27:35.205 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:27:35.205 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-02-19T15:27:35.206 INFO:teuthology.orchestra.run.smithi047.stdout: "pool": 17, 2024-02-19T15:27:35.206 INFO:teuthology.orchestra.run.smithi047.stdout: "old_pools": [] 2024-02-19T15:27:35.206 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:27:35.206 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:35.209 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:27:35.209 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:27:35.261 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:27:35.261 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:27:35.313 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:27:35.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.491+0000 7f487fe00640 1 Processor -- start 2024-02-19T15:27:35.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.491+0000 7f487fe00640 1 -- start start 2024-02-19T15:27:35.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.491+0000 7f487fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f487815dfa0 0x7f487806fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.491+0000 7f487fe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f48780b4240 0x7f48780d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.491+0000 7f487fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f48780ce400 0x7f48780ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.491+0000 7f487fe00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48780cedd0 con 0x7f48780ce400 2024-02-19T15:27:35.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.491+0000 7f487fe00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f487806ff60 con 0x7f48780b4240 2024-02-19T15:27:35.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.491+0000 7f487fe00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48780d1070 con 0x7f487815dfa0 2024-02-19T15:27:35.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.491+0000 7f487de00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f48780ce400 0x7f48780ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.491+0000 7f487de00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f48780ce400 0x7f48780ce890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60838/0 (socket says 172.21.15.47:60838) 2024-02-19T15:27:35.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.491+0000 7f487de00640 1 -- 172.21.15.47:0/1148772094 learned_addr learned my addr 172.21.15.47:0/1148772094 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:35.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.491+0000 7f487d400640 1 --2- 172.21.15.47:0/1148772094 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f487815dfa0 0x7f487806fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.491+0000 7f487ca00640 1 --2- 172.21.15.47:0/1148772094 >> v2:172.21.15.116:3300/0 conn(0x7f48780b4240 0x7f48780d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.491+0000 7f487de00640 1 -- 172.21.15.47:0/1148772094 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f487815dfa0 msgr2=0x7f487806fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:35.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.491+0000 7f487de00640 1 --2- 172.21.15.47:0/1148772094 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f487815dfa0 0x7f487806fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.491+0000 7f487de00640 1 -- 172.21.15.47:0/1148772094 >> v2:172.21.15.116:3300/0 conn(0x7f48780b4240 msgr2=0x7f48780d0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:35.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.491+0000 7f487de00640 1 --2- 172.21.15.47:0/1148772094 >> v2:172.21.15.116:3300/0 conn(0x7f48780b4240 0x7f48780d0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.491+0000 7f487de00640 1 -- 172.21.15.47:0/1148772094 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487806ec60 con 0x7f48780ce400 2024-02-19T15:27:35.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.495+0000 7f487de00640 1 --2- 172.21.15.47:0/1148772094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48780ce400 0x7f48780ce890 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f4878071a80 tx=0x7f487001c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b94a0f2277ddf4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:35.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.495+0000 7f4877e00640 1 -- 172.21.15.47:0/1148772094 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48700a5750 con 0x7f48780ce400 2024-02-19T15:27:35.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.495+0000 7f4877e00640 1 -- 172.21.15.47:0/1148772094 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48700a58b0 con 0x7f48780ce400 2024-02-19T15:27:35.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.495+0000 7f4877e00640 1 -- 172.21.15.47:0/1148772094 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48700a5c40 con 0x7f48780ce400 2024-02-19T15:27:35.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.495+0000 7f487fe00640 1 -- 172.21.15.47:0/1148772094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48780ce400 msgr2=0x7f48780ce890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:35.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.495+0000 7f487fe00640 1 --2- 172.21.15.47:0/1148772094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48780ce400 0x7f48780ce890 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f4878071a80 tx=0x7f487001c9c0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.495+0000 7f487fe00640 1 -- 172.21.15.47:0/1148772094 shutdown_connections 2024-02-19T15:27:35.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.495+0000 7f487fe00640 1 --2- 172.21.15.47:0/1148772094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48780ce400 0x7f48780ce890 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.495+0000 7f487fe00640 1 --2- 172.21.15.47:0/1148772094 >> v2:172.21.15.116:3300/0 conn(0x7f48780b4240 0x7f48780d0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.495+0000 7f487fe00640 1 --2- 172.21.15.47:0/1148772094 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f487815dfa0 0x7f487806fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.495+0000 7f487fe00640 1 -- 172.21.15.47:0/1148772094 >> 172.21.15.47:0/1148772094 conn(0x7f48780b3690 msgr2=0x7f48780b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:35.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.495+0000 7f487fe00640 1 -- 172.21.15.47:0/1148772094 shutdown_connections 2024-02-19T15:27:35.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.495+0000 7f487fe00640 1 -- 172.21.15.47:0/1148772094 wait complete. 2024-02-19T15:27:35.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.499+0000 7f487fe00640 1 Processor -- start 2024-02-19T15:27:35.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.499+0000 7f487fe00640 1 -- start start 2024-02-19T15:27:35.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.499+0000 7f487fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48780b4240 0x7f48780c6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.499+0000 7f487fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48780ce400 0x7f48781fa1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.499+0000 7f487fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f487815dfa0 0x7f48780c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.499+0000 7f487fe00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48781fb1c0 con 0x7f487815dfa0 2024-02-19T15:27:35.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.503+0000 7f487fe00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48780c6c90 con 0x7f48780ce400 2024-02-19T15:27:35.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.503+0000 7f487fe00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48780c6690 con 0x7f48780b4240 2024-02-19T15:27:35.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.503+0000 7f487de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f487815dfa0 0x7f48780c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.503+0000 7f487ca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48780ce400 0x7f48781fa1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.503+0000 7f487de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f487815dfa0 0x7f48780c9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60842/0 (socket says 172.21.15.47:60842) 2024-02-19T15:27:35.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.503+0000 7f487de00640 1 -- 172.21.15.47:0/2037354315 learned_addr learned my addr 172.21.15.47:0/2037354315 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:35.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.503+0000 7f487d400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48780b4240 0x7f48780c6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.503+0000 7f487de00640 1 -- 172.21.15.47:0/2037354315 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48780b4240 msgr2=0x7f48780c6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:35.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.503+0000 7f487de00640 1 --2- 172.21.15.47:0/2037354315 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48780b4240 0x7f48780c6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.503+0000 7f487de00640 1 -- 172.21.15.47:0/2037354315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48780ce400 msgr2=0x7f48781fa1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:35.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.503+0000 7f487de00640 1 --2- 172.21.15.47:0/2037354315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48780ce400 0x7f48781fa1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.503+0000 7f487de00640 1 -- 172.21.15.47:0/2037354315 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4870009b30 con 0x7f487815dfa0 2024-02-19T15:27:35.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.503+0000 7f487de00640 1 --2- 172.21.15.47:0/2037354315 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f487815dfa0 0x7f48780c9770 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f48700af040 tx=0x7f4870019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:35.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.503+0000 7f4875600640 1 -- 172.21.15.47:0/2037354315 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48700b8670 con 0x7f487815dfa0 2024-02-19T15:27:35.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.503+0000 7f4875600640 1 -- 172.21.15.47:0/2037354315 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48700b87d0 con 0x7f487815dfa0 2024-02-19T15:27:35.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.503+0000 7f4875600640 1 -- 172.21.15.47:0/2037354315 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48700b8b80 con 0x7f487815dfa0 2024-02-19T15:27:35.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.503+0000 7f487fe00640 1 -- 172.21.15.47:0/2037354315 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48781fa920 con 0x7f487815dfa0 2024-02-19T15:27:35.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.503+0000 7f487fe00640 1 -- 172.21.15.47:0/2037354315 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48781faee0 con 0x7f487815dfa0 2024-02-19T15:27:35.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.503+0000 7f487fe00640 1 -- 172.21.15.47:0/2037354315 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48780c69f0 con 0x7f487815dfa0 2024-02-19T15:27:35.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.503+0000 7f4875600640 1 -- 172.21.15.47:0/2037354315 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f48700b8ce0 con 0x7f487815dfa0 2024-02-19T15:27:35.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.507+0000 7f4875600640 1 --2- 172.21.15.47:0/2037354315 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4858066e40 0x7f4858069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.507+0000 7f4875600640 1 -- 172.21.15.47:0/2037354315 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f48701200d0 con 0x7f487815dfa0 2024-02-19T15:27:35.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.507+0000 7f487d400640 1 --2- 172.21.15.47:0/2037354315 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4858066e40 0x7f4858069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.507+0000 7f487d400640 1 --2- 172.21.15.47:0/2037354315 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4858066e40 0x7f4858069300 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f48780cdd60 tx=0x7f486c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:35.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.511+0000 7f4875600640 1 -- 172.21.15.47:0/2037354315 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f48780c69f0 con 0x7f487815dfa0 2024-02-19T15:27:35.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.675+0000 7f487fe00640 1 -- 172.21.15.47:0/2037354315 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48780c69f0 con 0x7f487815dfa0 2024-02-19T15:27:35.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.675+0000 7f4875600640 1 -- 172.21.15.47:0/2037354315 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16639 (secure 0 0 0) 0x7f48780c69f0 con 0x7f487815dfa0 2024-02-19T15:27:35.680 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:35.681 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":108,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:27:23.037320+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:27:19.839144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-02-19T15:27:20.405924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:27:35.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.679+0000 7f487fe00640 1 -- 172.21.15.47:0/2037354315 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4858066e40 msgr2=0x7f4858069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:35.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.679+0000 7f487fe00640 1 --2- 172.21.15.47:0/2037354315 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4858066e40 0x7f4858069300 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f48780cdd60 tx=0x7f486c008040 comp rx=0 tx=0).stop 2024-02-19T15:27:35.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.679+0000 7f487fe00640 1 -- 172.21.15.47:0/2037354315 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f487815dfa0 msgr2=0x7f48780c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:35.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.679+0000 7f487fe00640 1 --2- 172.21.15.47:0/2037354315 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f487815dfa0 0x7f48780c9770 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f48700af040 tx=0x7f4870019760 comp rx=0 tx=0).stop 2024-02-19T15:27:35.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.679+0000 7f487fe00640 1 -- 172.21.15.47:0/2037354315 shutdown_connections 2024-02-19T15:27:35.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.679+0000 7f487fe00640 1 --2- 172.21.15.47:0/2037354315 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4858066e40 0x7f4858069300 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.679+0000 7f487fe00640 1 --2- 172.21.15.47:0/2037354315 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f487815dfa0 0x7f48780c9770 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.679+0000 7f487fe00640 1 --2- 172.21.15.47:0/2037354315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48780ce400 0x7f48781fa1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.679+0000 7f487fe00640 1 --2- 172.21.15.47:0/2037354315 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48780b4240 0x7f48780c6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.679+0000 7f487fe00640 1 -- 172.21.15.47:0/2037354315 >> 172.21.15.47:0/2037354315 conn(0x7f48780b3690 msgr2=0x7f487806bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:35.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.679+0000 7f487fe00640 1 -- 172.21.15.47:0/2037354315 shutdown_connections 2024-02-19T15:27:35.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.679+0000 7f487fe00640 1 -- 172.21.15.47:0/2037354315 wait complete. 2024-02-19T15:27:35.702 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:27:19.839144+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:27:20.405924+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:27:35.702 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:27:35.770 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:27:35.773 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:27:35.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.843+0000 7fedab800640 1 Processor -- start 2024-02-19T15:27:35.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.843+0000 7fedab800640 1 -- start start 2024-02-19T15:27:35.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.843+0000 7fedab800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feda40c7e80 0x7feda40d0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.843+0000 7fedab800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7feda40c8260 0x7feda40d0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.843+0000 7fedab800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda40cd750 con 0x7feda40cfbf0 2024-02-19T15:27:35.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.843+0000 7fedab800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feda40ceee0 con 0x7feda40c8260 2024-02-19T15:27:35.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.843+0000 7fedab800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda40cdc60 con 0x7feda40c7e80 2024-02-19T15:27:35.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.843+0000 7feda3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7feda40c8260 0x7feda40d0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.843+0000 7feda3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7feda40c8260 0x7feda40d0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60846/0 (socket says 172.21.15.47:60846) 2024-02-19T15:27:35.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.843+0000 7feda3e00640 1 -- 172.21.15.47:0/3895160873 learned_addr learned my addr 172.21.15.47:0/3895160873 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:35.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.843+0000 7feda8e00640 1 --2- 172.21.15.47:0/3895160873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feda40c7e80 0x7feda40d0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.843+0000 7feda3e00640 1 -- 172.21.15.47:0/3895160873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feda40c7e80 msgr2=0x7feda40d0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:35.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.843+0000 7feda3e00640 1 --2- 172.21.15.47:0/3895160873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feda40c7e80 0x7feda40d0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.843+0000 7feda3e00640 1 -- 172.21.15.47:0/3895160873 >> v1:172.21.15.116:6789/0 conn(0x7feda40cfbf0 legacy=0x7feda40c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:35.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.843+0000 7feda3e00640 1 -- 172.21.15.47:0/3895160873 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda40d2a40 con 0x7feda40c8260 2024-02-19T15:27:35.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.847+0000 7feda3e00640 1 --2- 172.21.15.47:0/3895160873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feda40c8260 0x7feda40d0650 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fed90009870 tx=0x7fed9001c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8db125ec9bf68a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:35.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.847+0000 7feda3400640 1 -- 172.21.15.47:0/3895160873 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed900a95f0 con 0x7feda40c8260 2024-02-19T15:27:35.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.847+0000 7feda3400640 1 -- 172.21.15.47:0/3895160873 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed900a9750 con 0x7feda40c8260 2024-02-19T15:27:35.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.847+0000 7feda3400640 1 -- 172.21.15.47:0/3895160873 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed900a9ae0 con 0x7feda40c8260 2024-02-19T15:27:35.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.847+0000 7fedab800640 1 -- 172.21.15.47:0/3895160873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feda40c8260 msgr2=0x7feda40d0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:35.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.847+0000 7fedab800640 1 --2- 172.21.15.47:0/3895160873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feda40c8260 0x7feda40d0650 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fed90009870 tx=0x7fed9001c530 comp rx=0 tx=0).stop 2024-02-19T15:27:35.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.847+0000 7fedab800640 1 -- 172.21.15.47:0/3895160873 shutdown_connections 2024-02-19T15:27:35.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.847+0000 7fedab800640 1 --2- 172.21.15.47:0/3895160873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feda40c8260 0x7feda40d0650 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.847+0000 7fedab800640 1 --2- 172.21.15.47:0/3895160873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feda40c7e80 0x7feda40d0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.847+0000 7fedab800640 1 -- 172.21.15.47:0/3895160873 >> 172.21.15.47:0/3895160873 conn(0x7feda40d1b30 msgr2=0x7feda40d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:35.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.847+0000 7fedab800640 1 -- 172.21.15.47:0/3895160873 shutdown_connections 2024-02-19T15:27:35.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.847+0000 7fedab800640 1 -- 172.21.15.47:0/3895160873 wait complete. 2024-02-19T15:27:35.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7fedab800640 1 Processor -- start 2024-02-19T15:27:35.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7fedab800640 1 -- start start 2024-02-19T15:27:35.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7fedab800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feda40c7e80 0x7feda415acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7fedab800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feda40c8260 0x7feda40cab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7fedab800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feda40cfbf0 0x7feda4149b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7fedab800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda415b510 con 0x7feda40c8260 2024-02-19T15:27:35.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7fedab800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda415b7f0 con 0x7feda40c7e80 2024-02-19T15:27:35.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7fedab800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda415bad0 con 0x7feda40cfbf0 2024-02-19T15:27:35.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7feda3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feda40c8260 0x7feda40cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7feda3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feda40c8260 0x7feda40cab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60854/0 (socket says 172.21.15.47:60854) 2024-02-19T15:27:35.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7feda3e00640 1 -- 172.21.15.47:0/2557602511 learned_addr learned my addr 172.21.15.47:0/2557602511 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:35.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7feda9800640 1 --2- 172.21.15.47:0/2557602511 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feda40cfbf0 0x7feda4149b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7feda8e00640 1 --2- 172.21.15.47:0/2557602511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feda40c7e80 0x7feda415acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7feda3e00640 1 -- 172.21.15.47:0/2557602511 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feda40cfbf0 msgr2=0x7feda4149b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:35.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7feda3e00640 1 --2- 172.21.15.47:0/2557602511 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feda40cfbf0 0x7feda4149b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7feda3e00640 1 -- 172.21.15.47:0/2557602511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feda40c7e80 msgr2=0x7feda415acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:35.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7feda3e00640 1 --2- 172.21.15.47:0/2557602511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feda40c7e80 0x7feda415acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:35.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7feda3e00640 1 -- 172.21.15.47:0/2557602511 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed900094d0 con 0x7feda40c8260 2024-02-19T15:27:35.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7feda3e00640 1 --2- 172.21.15.47:0/2557602511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feda40c8260 0x7feda40cab20 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fed9001abe0 tx=0x7fed900192d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:35.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7feda0c00640 1 -- 172.21.15.47:0/2557602511 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed900b1800 con 0x7feda40c8260 2024-02-19T15:27:35.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7feda0c00640 1 -- 172.21.15.47:0/2557602511 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed900b1960 con 0x7feda40c8260 2024-02-19T15:27:35.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7feda0c00640 1 -- 172.21.15.47:0/2557602511 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed900b1d10 con 0x7feda40c8260 2024-02-19T15:27:35.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7fedab800640 1 -- 172.21.15.47:0/2557602511 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feda415bdb0 con 0x7feda40c8260 2024-02-19T15:27:35.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7fedab800640 1 -- 172.21.15.47:0/2557602511 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feda415b200 con 0x7feda40c8260 2024-02-19T15:27:35.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.851+0000 7fedab800640 1 -- 172.21.15.47:0/2557602511 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda40d1180 con 0x7feda40c8260 2024-02-19T15:27:35.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.855+0000 7feda0c00640 1 -- 172.21.15.47:0/2557602511 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fed900bb3f0 con 0x7feda40c8260 2024-02-19T15:27:35.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.855+0000 7feda0c00640 1 --2- 172.21.15.47:0/2557602511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fed84066e40 0x7fed84069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:35.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.855+0000 7feda0c00640 1 -- 172.21.15.47:0/2557602511 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed9011f0a0 con 0x7feda40c8260 2024-02-19T15:27:35.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.855+0000 7feda8e00640 1 --2- 172.21.15.47:0/2557602511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fed84066e40 0x7fed84069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:35.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.855+0000 7feda8e00640 1 --2- 172.21.15.47:0/2557602511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fed84066e40 0x7fed84069300 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fed98009a50 tx=0x7fed98008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:35.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:35.859+0000 7feda0c00640 1 -- 172.21.15.47:0/2557602511 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fed900ebdf0 con 0x7feda40c8260 2024-02-19T15:27:36.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.023+0000 7fedab800640 1 -- 172.21.15.47:0/2557602511 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feda415bc10 con 0x7feda40c8260 2024-02-19T15:27:36.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.023+0000 7feda0c00640 1 -- 172.21.15.47:0/2557602511 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v108) v1 ==== 93+0+10 (secure 0 0 0) 0x7feda415bc10 con 0x7feda40c8260 2024-02-19T15:27:36.027 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:27:36.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.023+0000 7fedab800640 1 -- 172.21.15.47:0/2557602511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fed84066e40 msgr2=0x7fed84069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:36.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.023+0000 7fedab800640 1 --2- 172.21.15.47:0/2557602511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fed84066e40 0x7fed84069300 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fed98009a50 tx=0x7fed98008040 comp rx=0 tx=0).stop 2024-02-19T15:27:36.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.023+0000 7fedab800640 1 -- 172.21.15.47:0/2557602511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feda40c8260 msgr2=0x7feda40cab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:36.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.023+0000 7fedab800640 1 --2- 172.21.15.47:0/2557602511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feda40c8260 0x7feda40cab20 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fed9001abe0 tx=0x7fed900192d0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.023+0000 7fedab800640 1 -- 172.21.15.47:0/2557602511 shutdown_connections 2024-02-19T15:27:36.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.023+0000 7fedab800640 1 --2- 172.21.15.47:0/2557602511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fed84066e40 0x7fed84069300 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.023+0000 7fedab800640 1 --2- 172.21.15.47:0/2557602511 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feda40cfbf0 0x7feda4149b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.023+0000 7fedab800640 1 --2- 172.21.15.47:0/2557602511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feda40c8260 0x7feda40cab20 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.023+0000 7fedab800640 1 --2- 172.21.15.47:0/2557602511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feda40c7e80 0x7feda415acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.023+0000 7fedab800640 1 -- 172.21.15.47:0/2557602511 >> 172.21.15.47:0/2557602511 conn(0x7feda40d1b30 msgr2=0x7feda406f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:36.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.023+0000 7fedab800640 1 -- 172.21.15.47:0/2557602511 shutdown_connections 2024-02-19T15:27:36.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.023+0000 7fedab800640 1 -- 172.21.15.47:0/2557602511 wait complete. 2024-02-19T15:27:36.049 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:27:36.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.231+0000 7fa914400640 1 Processor -- start 2024-02-19T15:27:36.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.231+0000 7fa914400640 1 -- start start 2024-02-19T15:27:36.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.231+0000 7fa914400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa90c0c7e80 0x7fa90c0d0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:36.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.231+0000 7fa914400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa90c0cd750 con 0x7fa90c0cfbf0 2024-02-19T15:27:36.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.231+0000 7fa914400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa90c0ceee0 con 0x7fa90c0c8260 2024-02-19T15:27:36.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.231+0000 7fa914400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa90c0cdc60 con 0x7fa90c0c7e80 2024-02-19T15:27:36.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.235+0000 7fa911a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa90c0c7e80 0x7fa90c0d0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:36.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.235+0000 7fa912400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa90c0cfbf0 0x7fa90c0c5d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35264/0 (socket says 172.21.15.47:35264) 2024-02-19T15:27:36.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.235+0000 7fa912400640 1 -- 172.21.15.47:0/3686171169 learned_addr learned my addr 172.21.15.47:0/3686171169 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:36.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.235+0000 7fa911a00640 1 -- 172.21.15.47:0/3686171169 >> v1:172.21.15.116:6789/0 conn(0x7fa90c0c8260 legacy=0x7fa90c0c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:36.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.235+0000 7fa911a00640 1 -- 172.21.15.47:0/3686171169 >> v1:172.21.15.47:6789/0 conn(0x7fa90c0cfbf0 legacy=0x7fa90c0c5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:36.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.235+0000 7fa911a00640 1 -- 172.21.15.47:0/3686171169 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90c0d2a40 con 0x7fa90c0c7e80 2024-02-19T15:27:36.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.235+0000 7fa911a00640 1 --2- 172.21.15.47:0/3686171169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa90c0c7e80 0x7fa90c0d0110 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fa9000097c0 tx=0x7fa90001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1f200cda6601735 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:36.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.235+0000 7fa90be00640 1 -- 172.21.15.47:0/3686171169 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9000a6040 con 0x7fa90c0c7e80 2024-02-19T15:27:36.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.235+0000 7fa90be00640 1 -- 172.21.15.47:0/3686171169 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9000aa600 con 0x7fa90c0c7e80 2024-02-19T15:27:36.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.235+0000 7fa90be00640 1 -- 172.21.15.47:0/3686171169 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9000aa990 con 0x7fa90c0c7e80 2024-02-19T15:27:36.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa914400640 1 -- 172.21.15.47:0/3686171169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa90c0c7e80 msgr2=0x7fa90c0d0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:36.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa914400640 1 --2- 172.21.15.47:0/3686171169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa90c0c7e80 0x7fa90c0d0110 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fa9000097c0 tx=0x7fa90001c4a0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa914400640 1 -- 172.21.15.47:0/3686171169 shutdown_connections 2024-02-19T15:27:36.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa914400640 1 --2- 172.21.15.47:0/3686171169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa90c0c7e80 0x7fa90c0d0110 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa914400640 1 -- 172.21.15.47:0/3686171169 >> 172.21.15.47:0/3686171169 conn(0x7fa90c0d1b30 msgr2=0x7fa90c0d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:36.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa914400640 1 -- 172.21.15.47:0/3686171169 shutdown_connections 2024-02-19T15:27:36.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa914400640 1 -- 172.21.15.47:0/3686171169 wait complete. 2024-02-19T15:27:36.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa914400640 1 Processor -- start 2024-02-19T15:27:36.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa914400640 1 -- start start 2024-02-19T15:27:36.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa914400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa90c0c7e80 0x7fa90c0ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:36.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa914400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa90c0c8260 0x7fa90c153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:36.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa914400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa90c0cfbf0 0x7fa90c157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:36.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa914400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c0cac90 con 0x7fa90c0cfbf0 2024-02-19T15:27:36.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa914400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c0caf70 con 0x7fa90c0c8260 2024-02-19T15:27:36.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa914400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c0cb250 con 0x7fa90c0c7e80 2024-02-19T15:27:36.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa912400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa90c0cfbf0 0x7fa90c157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:36.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa912400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa90c0cfbf0 0x7fa90c157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60870/0 (socket says 172.21.15.47:60870) 2024-02-19T15:27:36.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa912400640 1 -- 172.21.15.47:0/159726522 learned_addr learned my addr 172.21.15.47:0/159726522 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:36.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa911000640 1 --2- 172.21.15.47:0/159726522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa90c0c8260 0x7fa90c153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:36.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa911a00640 1 --2- 172.21.15.47:0/159726522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa90c0c7e80 0x7fa90c0ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:36.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa912400640 1 -- 172.21.15.47:0/159726522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa90c0c7e80 msgr2=0x7fa90c0ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:36.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.239+0000 7fa912400640 1 --2- 172.21.15.47:0/159726522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa90c0c7e80 0x7fa90c0ca400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.243+0000 7fa912400640 1 -- 172.21.15.47:0/159726522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa90c0c8260 msgr2=0x7fa90c153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:36.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.243+0000 7fa912400640 1 --2- 172.21.15.47:0/159726522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa90c0c8260 0x7fa90c153290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.243+0000 7fa912400640 1 -- 172.21.15.47:0/159726522 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa900009420 con 0x7fa90c0cfbf0 2024-02-19T15:27:36.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.243+0000 7fa911000640 1 --2- 172.21.15.47:0/159726522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa90c0c8260 0x7fa90c153290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:27:36.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.243+0000 7fa912400640 1 --2- 172.21.15.47:0/159726522 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa90c0cfbf0 0x7fa90c157b90 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fa90400ca80 tx=0x7fa90400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:36.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.243+0000 7fa909600640 1 -- 172.21.15.47:0/159726522 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90400ac30 con 0x7fa90c0cfbf0 2024-02-19T15:27:36.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.243+0000 7fa909600640 1 -- 172.21.15.47:0/159726522 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90400ad90 con 0x7fa90c0cfbf0 2024-02-19T15:27:36.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.243+0000 7fa914400640 1 -- 172.21.15.47:0/159726522 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa90c157390 con 0x7fa90c0cfbf0 2024-02-19T15:27:36.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.243+0000 7fa909600640 1 -- 172.21.15.47:0/159726522 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa904012600 con 0x7fa90c0cfbf0 2024-02-19T15:27:36.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.243+0000 7fa914400640 1 -- 172.21.15.47:0/159726522 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa90c0ca940 con 0x7fa90c0cfbf0 2024-02-19T15:27:36.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.243+0000 7fa914400640 1 -- 172.21.15.47:0/159726522 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa90c0d1100 con 0x7fa90c0cfbf0 2024-02-19T15:27:36.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.243+0000 7fa909600640 1 -- 172.21.15.47:0/159726522 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa904007070 con 0x7fa90c0cfbf0 2024-02-19T15:27:36.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.243+0000 7fa909600640 1 --2- 172.21.15.47:0/159726522 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa8ec066e40 0x7fa8ec069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:36.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.243+0000 7fa909600640 1 -- 172.21.15.47:0/159726522 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9040847e0 con 0x7fa90c0cfbf0 2024-02-19T15:27:36.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.247+0000 7fa911a00640 1 --2- 172.21.15.47:0/159726522 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa8ec066e40 0x7fa8ec069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:36.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.247+0000 7fa911a00640 1 --2- 172.21.15.47:0/159726522 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa8ec066e40 0x7fa8ec069300 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fa9000ae000 tx=0x7fa9000b5040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:36.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.247+0000 7fa909600640 1 -- 172.21.15.47:0/159726522 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa904051680 con 0x7fa90c0cfbf0 2024-02-19T15:27:36.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.415+0000 7fa914400640 1 -- 172.21.15.47:0/159726522 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa90c075360 con 0x7fa90c0cfbf0 2024-02-19T15:27:36.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.415+0000 7fa909600640 1 -- 172.21.15.47:0/159726522 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v108) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa9040170a0 con 0x7fa90c0cfbf0 2024-02-19T15:27:36.419 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:27:36.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.415+0000 7fa914400640 1 -- 172.21.15.47:0/159726522 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa8ec066e40 msgr2=0x7fa8ec069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:36.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.415+0000 7fa914400640 1 --2- 172.21.15.47:0/159726522 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa8ec066e40 0x7fa8ec069300 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fa9000ae000 tx=0x7fa9000b5040 comp rx=0 tx=0).stop 2024-02-19T15:27:36.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.415+0000 7fa914400640 1 -- 172.21.15.47:0/159726522 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa90c0cfbf0 msgr2=0x7fa90c157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:36.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.415+0000 7fa914400640 1 --2- 172.21.15.47:0/159726522 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa90c0cfbf0 0x7fa90c157b90 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fa90400ca80 tx=0x7fa90400cf40 comp rx=0 tx=0).stop 2024-02-19T15:27:36.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.415+0000 7fa914400640 1 -- 172.21.15.47:0/159726522 shutdown_connections 2024-02-19T15:27:36.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.415+0000 7fa914400640 1 --2- 172.21.15.47:0/159726522 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa8ec066e40 0x7fa8ec069300 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.415+0000 7fa914400640 1 --2- 172.21.15.47:0/159726522 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa90c0cfbf0 0x7fa90c157b90 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.415+0000 7fa914400640 1 --2- 172.21.15.47:0/159726522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa90c0c8260 0x7fa90c153290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.415+0000 7fa914400640 1 --2- 172.21.15.47:0/159726522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa90c0c7e80 0x7fa90c0ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.415+0000 7fa914400640 1 -- 172.21.15.47:0/159726522 >> 172.21.15.47:0/159726522 conn(0x7fa90c0d1b30 msgr2=0x7fa90c15d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:36.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.415+0000 7fa914400640 1 -- 172.21.15.47:0/159726522 shutdown_connections 2024-02-19T15:27:36.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.415+0000 7fa914400640 1 -- 172.21.15.47:0/159726522 wait complete. 2024-02-19T15:27:36.440 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:27:36.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.579+0000 7f7b96800640 1 Processor -- start 2024-02-19T15:27:36.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.579+0000 7f7b96800640 1 -- start start 2024-02-19T15:27:36.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.579+0000 7f7b96800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7b9015e280 0x7f7b9006c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:36.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.579+0000 7f7b96800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b900cc620 con 0x7f7b900d2650 2024-02-19T15:27:36.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.579+0000 7f7b96800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b900d12e0 con 0x7f7b900ce3b0 2024-02-19T15:27:36.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.579+0000 7f7b96800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b900d1940 con 0x7f7b9015e280 2024-02-19T15:27:36.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.583+0000 7f7b8fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7b9015e280 0x7f7b9006c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:36.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.583+0000 7f7b8fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7b9015e280 0x7f7b9006c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60882/0 (socket says 172.21.15.47:60882) 2024-02-19T15:27:36.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.583+0000 7f7b8fe00640 1 -- 172.21.15.47:0/244216624 learned_addr learned my addr 172.21.15.47:0/244216624 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:36.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.583+0000 7f7b87e00640 1 --1- 172.21.15.47:0/244216624 >> v1:172.21.15.47:6789/0 conn(0x7f7b900d2650 0x7f7b9006c1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35272/0 (socket says 172.21.15.47:35272) 2024-02-19T15:27:36.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.583+0000 7f7b8fe00640 1 -- 172.21.15.47:0/244216624 >> v1:172.21.15.116:6789/0 conn(0x7f7b900ce3b0 legacy=0x7f7b900ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:36.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.583+0000 7f7b8fe00640 1 -- 172.21.15.47:0/244216624 >> v1:172.21.15.47:6789/0 conn(0x7f7b900d2650 legacy=0x7f7b9006c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:36.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.583+0000 7f7b8ea00640 1 -- 172.21.15.47:0/244216624 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589836908 0 0) 0x7f7b900cc620 con 0x7f7b900d2650 2024-02-19T15:27:36.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.583+0000 7f7b8fe00640 1 -- 172.21.15.47:0/244216624 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b900d1000 con 0x7f7b9015e280 2024-02-19T15:27:36.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.583+0000 7f7b8fe00640 1 --2- 172.21.15.47:0/244216624 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b9015e280 0x7f7b9006c8e0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009e50 tx=0x7f7b8801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ec3ca44d552306 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:36.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.583+0000 7f7b8ea00640 1 -- 172.21.15.47:0/244216624 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b880a5660 con 0x7f7b9015e280 2024-02-19T15:27:36.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.583+0000 7f7b8ea00640 1 -- 172.21.15.47:0/244216624 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b880a57c0 con 0x7f7b9015e280 2024-02-19T15:27:36.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.583+0000 7f7b8ea00640 1 -- 172.21.15.47:0/244216624 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b880a5b50 con 0x7f7b9015e280 2024-02-19T15:27:36.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b96800640 1 -- 172.21.15.47:0/244216624 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b9015e280 msgr2=0x7f7b9006c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:36.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b96800640 1 --2- 172.21.15.47:0/244216624 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b9015e280 0x7f7b9006c8e0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009e50 tx=0x7f7b8801c910 comp rx=0 tx=0).stop 2024-02-19T15:27:36.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b96800640 1 -- 172.21.15.47:0/244216624 shutdown_connections 2024-02-19T15:27:36.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b96800640 1 --2- 172.21.15.47:0/244216624 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b9015e280 0x7f7b9006c8e0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b96800640 1 -- 172.21.15.47:0/244216624 >> 172.21.15.47:0/244216624 conn(0x7f7b900d1b30 msgr2=0x7f7b900d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:36.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b96800640 1 -- 172.21.15.47:0/244216624 shutdown_connections 2024-02-19T15:27:36.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b96800640 1 -- 172.21.15.47:0/244216624 wait complete. 2024-02-19T15:27:36.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b96800640 1 Processor -- start 2024-02-19T15:27:36.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b96800640 1 -- start start 2024-02-19T15:27:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b96800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b900ce3b0 0x7f7b90150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b96800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b900d2650 0x7f7b9015eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b96800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b9015e280 0x7f7b900c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b96800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b90150550 con 0x7f7b900d2650 2024-02-19T15:27:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b96800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b901541a0 con 0x7f7b9015e280 2024-02-19T15:27:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b96800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b90154440 con 0x7f7b900ce3b0 2024-02-19T15:27:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b87e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b900d2650 0x7f7b9015eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b87e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b900d2650 0x7f7b9015eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60898/0 (socket says 172.21.15.47:60898) 2024-02-19T15:27:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b87e00640 1 -- 172.21.15.47:0/143902379 learned_addr learned my addr 172.21.15.47:0/143902379 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b8fe00640 1 --2- 172.21.15.47:0/143902379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b9015e280 0x7f7b900c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b8f400640 1 --2- 172.21.15.47:0/143902379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b900ce3b0 0x7f7b90150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b87e00640 1 -- 172.21.15.47:0/143902379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b900ce3b0 msgr2=0x7f7b90150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b87e00640 1 --2- 172.21.15.47:0/143902379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b900ce3b0 0x7f7b90150010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b87e00640 1 -- 172.21.15.47:0/143902379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b9015e280 msgr2=0x7f7b900c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b87e00640 1 --2- 172.21.15.47:0/143902379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b9015e280 0x7f7b900c8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.587+0000 7f7b87e00640 1 -- 172.21.15.47:0/143902379 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b88009b30 con 0x7f7b900d2650 2024-02-19T15:27:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.591+0000 7f7b87e00640 1 --2- 172.21.15.47:0/143902379 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b900d2650 0x7f7b9015eee0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f7b7800c410 tx=0x7f7b7800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.591+0000 7f7b87400640 1 -- 172.21.15.47:0/143902379 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b780074e0 con 0x7f7b900d2650 2024-02-19T15:27:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.591+0000 7f7b87400640 1 -- 172.21.15.47:0/143902379 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b78007640 con 0x7f7b900d2650 2024-02-19T15:27:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.591+0000 7f7b87400640 1 -- 172.21.15.47:0/143902379 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b78004a30 con 0x7f7b900d2650 2024-02-19T15:27:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.591+0000 7f7b96800640 1 -- 172.21.15.47:0/143902379 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b90154720 con 0x7f7b900d2650 2024-02-19T15:27:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.591+0000 7f7b96800640 1 -- 172.21.15.47:0/143902379 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b90154ce0 con 0x7f7b900d2650 2024-02-19T15:27:36.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.591+0000 7f7b96800640 1 -- 172.21.15.47:0/143902379 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b90154ed0 con 0x7f7b900d2650 2024-02-19T15:27:36.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.591+0000 7f7b87400640 1 -- 172.21.15.47:0/143902379 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7b78004b90 con 0x7f7b900d2650 2024-02-19T15:27:36.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.595+0000 7f7b87400640 1 --2- 172.21.15.47:0/143902379 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7b6c066e40 0x7f7b6c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:36.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.595+0000 7f7b87400640 1 -- 172.21.15.47:0/143902379 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b78084720 con 0x7f7b900d2650 2024-02-19T15:27:36.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.595+0000 7f7b8f400640 1 --2- 172.21.15.47:0/143902379 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7b6c066e40 0x7f7b6c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:36.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.595+0000 7f7b8f400640 1 --2- 172.21.15.47:0/143902379 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7b6c066e40 0x7f7b6c069300 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7b80000f70 tx=0x7f7b80009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:36.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.599+0000 7f7b87400640 1 -- 172.21.15.47:0/143902379 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7b78051580 con 0x7f7b900d2650 2024-02-19T15:27:36.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.759+0000 7f7b96800640 1 -- 172.21.15.47:0/143902379 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7b90075360 con 0x7f7b900d2650 2024-02-19T15:27:36.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.759+0000 7f7b87400640 1 -- 172.21.15.47:0/143902379 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v108) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7b78054c40 con 0x7f7b900d2650 2024-02-19T15:27:36.763 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:27:36.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.759+0000 7f7b96800640 1 -- 172.21.15.47:0/143902379 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7b6c066e40 msgr2=0x7f7b6c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:36.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.759+0000 7f7b96800640 1 --2- 172.21.15.47:0/143902379 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7b6c066e40 0x7f7b6c069300 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7b80000f70 tx=0x7f7b80009040 comp rx=0 tx=0).stop 2024-02-19T15:27:36.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.759+0000 7f7b96800640 1 -- 172.21.15.47:0/143902379 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b900d2650 msgr2=0x7f7b9015eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:36.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.759+0000 7f7b96800640 1 --2- 172.21.15.47:0/143902379 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b900d2650 0x7f7b9015eee0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f7b7800c410 tx=0x7f7b7800c8d0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.759+0000 7f7b96800640 1 -- 172.21.15.47:0/143902379 shutdown_connections 2024-02-19T15:27:36.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.759+0000 7f7b96800640 1 --2- 172.21.15.47:0/143902379 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7b6c066e40 0x7f7b6c069300 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.759+0000 7f7b96800640 1 --2- 172.21.15.47:0/143902379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b9015e280 0x7f7b900c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.759+0000 7f7b96800640 1 --2- 172.21.15.47:0/143902379 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b900d2650 0x7f7b9015eee0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.759+0000 7f7b96800640 1 --2- 172.21.15.47:0/143902379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b900ce3b0 0x7f7b90150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.759+0000 7f7b96800640 1 -- 172.21.15.47:0/143902379 >> 172.21.15.47:0/143902379 conn(0x7f7b900d1b30 msgr2=0x7f7b900c7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:36.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.759+0000 7f7b96800640 1 -- 172.21.15.47:0/143902379 shutdown_connections 2024-02-19T15:27:36.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.759+0000 7f7b96800640 1 -- 172.21.15.47:0/143902379 wait complete. 2024-02-19T15:27:36.784 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-19T15:27:36.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.919+0000 7fe8aec00640 1 Processor -- start 2024-02-19T15:27:36.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.919+0000 7fe8aec00640 1 -- start start 2024-02-19T15:27:36.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.919+0000 7fe8aec00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe8a80cedb0 0x7fe8a80cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:36.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.919+0000 7fe8aec00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe8a80d2710 0x7fe8a80d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:36.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.919+0000 7fe8aec00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8a80cc700 con 0x7fe8a815bcd0 2024-02-19T15:27:36.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.919+0000 7fe8aec00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a80cf5e0 con 0x7fe8a80d2710 2024-02-19T15:27:36.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.919+0000 7fe8aec00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a80cdb90 con 0x7fe8a80cedb0 2024-02-19T15:27:36.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.923+0000 7fe8acc00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe8a80d2710 0x7fe8a80d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:36.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.923+0000 7fe89fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe8a80cedb0 0x7fe8a80cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:36.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.923+0000 7fe89fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe8a80cedb0 0x7fe8a80cfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:37530/0 (socket says 172.21.15.47:37530) 2024-02-19T15:27:36.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.923+0000 7fe89fe00640 1 -- 172.21.15.47:0/2382794687 learned_addr learned my addr 172.21.15.47:0/2382794687 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:36.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.923+0000 7fe8acc00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe8a80d2710 0x7fe8a80d0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:56158/0 (socket says 172.21.15.47:56158) 2024-02-19T15:27:36.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.923+0000 7fe89fe00640 1 -- 172.21.15.47:0/2382794687 >> v2:172.21.15.116:3300/0 conn(0x7fe8a80d2710 msgr2=0x7fe8a80d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:36.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.923+0000 7fe89fe00640 1 --2- 172.21.15.47:0/2382794687 >> v2:172.21.15.116:3300/0 conn(0x7fe8a80d2710 0x7fe8a80d0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.923+0000 7fe89fe00640 1 -- 172.21.15.47:0/2382794687 >> v1:172.21.15.116:6789/0 conn(0x7fe8a815bcd0 legacy=0x7fe8a80cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:36.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.923+0000 7fe89fe00640 1 -- 172.21.15.47:0/2382794687 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a80cf300 con 0x7fe8a80cedb0 2024-02-19T15:27:36.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.923+0000 7fe89fe00640 1 --2- 172.21.15.47:0/2382794687 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe8a80cedb0 0x7fe8a80cfc50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe890009870 tx=0x7fe89001c530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad1ff699e670d54a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:36.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.927+0000 7fe8a7400640 1 -- 172.21.15.47:0/2382794687 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8900a95f0 con 0x7fe8a80cedb0 2024-02-19T15:27:36.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.927+0000 7fe8a7400640 1 -- 172.21.15.47:0/2382794687 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8900a9750 con 0x7fe8a80cedb0 2024-02-19T15:27:36.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.927+0000 7fe8a7400640 1 -- 172.21.15.47:0/2382794687 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8900a9ae0 con 0x7fe8a80cedb0 2024-02-19T15:27:36.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.927+0000 7fe8aec00640 1 -- 172.21.15.47:0/2382794687 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe8a80cedb0 msgr2=0x7fe8a80cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:36.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.927+0000 7fe8aec00640 1 --2- 172.21.15.47:0/2382794687 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe8a80cedb0 0x7fe8a80cfc50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe890009870 tx=0x7fe89001c530 comp rx=0 tx=0).stop 2024-02-19T15:27:36.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.927+0000 7fe8aec00640 1 -- 172.21.15.47:0/2382794687 shutdown_connections 2024-02-19T15:27:36.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.927+0000 7fe8aec00640 1 --2- 172.21.15.47:0/2382794687 >> v2:172.21.15.116:3300/0 conn(0x7fe8a80d2710 0x7fe8a80d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.927+0000 7fe8aec00640 1 --2- 172.21.15.47:0/2382794687 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe8a80cedb0 0x7fe8a80cfc50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.927+0000 7fe8aec00640 1 -- 172.21.15.47:0/2382794687 >> 172.21.15.47:0/2382794687 conn(0x7fe8a80d1bf0 msgr2=0x7fe8a80d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:36.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.927+0000 7fe8aec00640 1 -- 172.21.15.47:0/2382794687 shutdown_connections 2024-02-19T15:27:36.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.927+0000 7fe8aec00640 1 -- 172.21.15.47:0/2382794687 wait complete. 2024-02-19T15:27:36.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.927+0000 7fe8aec00640 1 Processor -- start 2024-02-19T15:27:36.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.927+0000 7fe8aec00640 1 -- start start 2024-02-19T15:27:36.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.927+0000 7fe8aec00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8a80cedb0 0x7fe8a8150670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:36.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.927+0000 7fe8aec00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe8a80d2710 0x7fe8a81fa020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:36.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.927+0000 7fe8aec00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8a815bcd0 0x7fe8a81fa560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:36.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.927+0000 7fe8aec00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a814d5b0 con 0x7fe8a815bcd0 2024-02-19T15:27:36.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.927+0000 7fe8aec00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a80c9a90 con 0x7fe8a80cedb0 2024-02-19T15:27:36.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.927+0000 7fe8aec00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a80c9d70 con 0x7fe8a80d2710 2024-02-19T15:27:36.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.931+0000 7fe8a7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8a80cedb0 0x7fe8a8150670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:36.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.931+0000 7fe8acc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8a815bcd0 0x7fe8a81fa560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:36.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.931+0000 7fe8a7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8a80cedb0 0x7fe8a8150670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:56174/0 (socket says 172.21.15.47:56174) 2024-02-19T15:27:36.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.931+0000 7fe8a7e00640 1 -- 172.21.15.47:0/1324511743 learned_addr learned my addr 172.21.15.47:0/1324511743 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:36.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.931+0000 7fe89fe00640 1 --2- 172.21.15.47:0/1324511743 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe8a80d2710 0x7fe8a81fa020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:36.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.931+0000 7fe8acc00640 1 -- 172.21.15.47:0/1324511743 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe8a80d2710 msgr2=0x7fe8a81fa020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:36.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.931+0000 7fe8acc00640 1 --2- 172.21.15.47:0/1324511743 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe8a80d2710 0x7fe8a81fa020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.931+0000 7fe8acc00640 1 -- 172.21.15.47:0/1324511743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8a80cedb0 msgr2=0x7fe8a8150670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:36.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.931+0000 7fe8acc00640 1 --2- 172.21.15.47:0/1324511743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8a80cedb0 0x7fe8a8150670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:36.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.931+0000 7fe8acc00640 1 -- 172.21.15.47:0/1324511743 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8900094d0 con 0x7fe8a815bcd0 2024-02-19T15:27:36.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.931+0000 7fe8a7e00640 1 --2- 172.21.15.47:0/1324511743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8a80cedb0 0x7fe8a8150670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:27:36.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.931+0000 7fe8acc00640 1 --2- 172.21.15.47:0/1324511743 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8a815bcd0 0x7fe8a81fa560 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fe8a000efc0 tx=0x7fe8a000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:36.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.931+0000 7fe8a4c00640 1 -- 172.21.15.47:0/1324511743 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a0009c00 con 0x7fe8a815bcd0 2024-02-19T15:27:36.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.931+0000 7fe8a4c00640 1 -- 172.21.15.47:0/1324511743 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8a000f040 con 0x7fe8a815bcd0 2024-02-19T15:27:36.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.931+0000 7fe8aec00640 1 -- 172.21.15.47:0/1324511743 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a815d240 con 0x7fe8a815bcd0 2024-02-19T15:27:36.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.931+0000 7fe8a4c00640 1 -- 172.21.15.47:0/1324511743 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a00146c0 con 0x7fe8a815bcd0 2024-02-19T15:27:36.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.931+0000 7fe8aec00640 1 -- 172.21.15.47:0/1324511743 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a80ca330 con 0x7fe8a815bcd0 2024-02-19T15:27:36.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.931+0000 7fe8aec00640 1 -- 172.21.15.47:0/1324511743 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a8075360 con 0x7fe8a815bcd0 2024-02-19T15:27:36.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.931+0000 7fe8a4c00640 1 -- 172.21.15.47:0/1324511743 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe8a0014820 con 0x7fe8a815bcd0 2024-02-19T15:27:36.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.931+0000 7fe8a4c00640 1 --2- 172.21.15.47:0/1324511743 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe884066e40 0x7fe884069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:36.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.935+0000 7fe8a4c00640 1 -- 172.21.15.47:0/1324511743 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe8a0085780 con 0x7fe8a815bcd0 2024-02-19T15:27:36.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.935+0000 7fe8a7e00640 1 --2- 172.21.15.47:0/1324511743 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe884066e40 0x7fe884069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:36.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.935+0000 7fe8a7e00640 1 --2- 172.21.15.47:0/1324511743 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe884066e40 0x7fe884069300 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fe898007fd0 tx=0x7fe898009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:36.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:36.939+0000 7fe8a4c00640 1 -- 172.21.15.47:0/1324511743 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe8a0052610 con 0x7fe8a815bcd0 2024-02-19T15:27:37.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.115+0000 7fe8aec00640 1 -- 172.21.15.47:0/1324511743 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe8a815b830 con 0x7fe8a815bcd0 2024-02-19T15:27:37.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.115+0000 7fe8a4c00640 1 -- 172.21.15.47:0/1324511743 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v108) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe8a0018080 con 0x7fe8a815bcd0 2024-02-19T15:27:37.119 INFO:teuthology.orchestra.run.smithi047.stderr:listed 0 entries 2024-02-19T15:27:37.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.115+0000 7fe8aec00640 1 -- 172.21.15.47:0/1324511743 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe884066e40 msgr2=0x7fe884069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:37.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.115+0000 7fe8aec00640 1 --2- 172.21.15.47:0/1324511743 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe884066e40 0x7fe884069300 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fe898007fd0 tx=0x7fe898009040 comp rx=0 tx=0).stop 2024-02-19T15:27:37.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.115+0000 7fe8aec00640 1 -- 172.21.15.47:0/1324511743 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8a815bcd0 msgr2=0x7fe8a81fa560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:37.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.115+0000 7fe8aec00640 1 --2- 172.21.15.47:0/1324511743 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8a815bcd0 0x7fe8a81fa560 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fe8a000efc0 tx=0x7fe8a000c490 comp rx=0 tx=0).stop 2024-02-19T15:27:37.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.115+0000 7fe8aec00640 1 -- 172.21.15.47:0/1324511743 shutdown_connections 2024-02-19T15:27:37.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.115+0000 7fe8aec00640 1 --2- 172.21.15.47:0/1324511743 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe884066e40 0x7fe884069300 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:37.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.115+0000 7fe8aec00640 1 --2- 172.21.15.47:0/1324511743 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8a815bcd0 0x7fe8a81fa560 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:37.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.115+0000 7fe8aec00640 1 --2- 172.21.15.47:0/1324511743 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe8a80d2710 0x7fe8a81fa020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:37.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.115+0000 7fe8aec00640 1 --2- 172.21.15.47:0/1324511743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8a80cedb0 0x7fe8a8150670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:37.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.115+0000 7fe8aec00640 1 -- 172.21.15.47:0/1324511743 >> 172.21.15.47:0/1324511743 conn(0x7fe8a80d1bf0 msgr2=0x7fe8a815b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:37.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.115+0000 7fe8aec00640 1 -- 172.21.15.47:0/1324511743 shutdown_connections 2024-02-19T15:27:37.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.119+0000 7fe8aec00640 1 -- 172.21.15.47:0/1324511743 wait complete. 2024-02-19T15:27:37.140 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-02-19T15:27:37.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:37.141 DEBUG:teuthology.orchestra.run.smithi047:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-02-19T15:27:37.198 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi047.front.sepia.ceph.com 2024-02-19T15:27:37.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:37.199 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-19T15:27:37.216 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:27:37.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.391+0000 7f3b5f200640 1 Processor -- start 2024-02-19T15:27:37.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.391+0000 7f3b5f200640 1 -- start start 2024-02-19T15:27:37.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.395+0000 7f3b5f200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b5815bcd0 0x7f3b580cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:37.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.395+0000 7f3b5f200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3b580cedb0 0x7f3b580cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:37.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.395+0000 7f3b5f200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b580cc700 con 0x7f3b580d2710 2024-02-19T15:27:37.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.395+0000 7f3b5f200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b580cf5e0 con 0x7f3b580cedb0 2024-02-19T15:27:37.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.395+0000 7f3b5f200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b580cf300 con 0x7f3b5815bcd0 2024-02-19T15:27:37.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.395+0000 7f3b4f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3b580cedb0 0x7f3b580cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:37.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.395+0000 7f3b5d200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f3b580d2710 0x7f3b5815d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35288/0 (socket says 172.21.15.47:35288) 2024-02-19T15:27:37.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.395+0000 7f3b5d200640 1 -- 172.21.15.47:0/4286428049 learned_addr learned my addr 172.21.15.47:0/4286428049 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:37.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.395+0000 7f3b57e00640 1 --2- 172.21.15.47:0/4286428049 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b5815bcd0 0x7f3b580cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:37.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.395+0000 7f3b4f400640 1 -- 172.21.15.47:0/4286428049 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b5815bcd0 msgr2=0x7f3b580cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:37.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.395+0000 7f3b4f400640 1 --2- 172.21.15.47:0/4286428049 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b5815bcd0 0x7f3b580cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:37.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.395+0000 7f3b4f400640 1 -- 172.21.15.47:0/4286428049 >> v1:172.21.15.47:6789/0 conn(0x7f3b580d2710 legacy=0x7f3b5815d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:37.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.395+0000 7f3b4f400640 1 -- 172.21.15.47:0/4286428049 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b580cc9a0 con 0x7f3b580cedb0 2024-02-19T15:27:37.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.395+0000 7f3b4f400640 1 --2- 172.21.15.47:0/4286428049 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b580cedb0 0x7f3b580cfc50 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f3b400097c0 tx=0x7f3b4001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8858fed4f422be6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:37.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b57400640 1 -- 172.21.15.47:0/4286428049 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b400ad680 con 0x7f3b580cedb0 2024-02-19T15:27:37.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b57400640 1 -- 172.21.15.47:0/4286428049 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b400ad7e0 con 0x7f3b580cedb0 2024-02-19T15:27:37.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b57400640 1 -- 172.21.15.47:0/4286428049 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b400adb70 con 0x7f3b580cedb0 2024-02-19T15:27:37.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b5f200640 1 -- 172.21.15.47:0/4286428049 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b580cedb0 msgr2=0x7f3b580cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:37.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b5f200640 1 --2- 172.21.15.47:0/4286428049 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b580cedb0 0x7f3b580cfc50 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f3b400097c0 tx=0x7f3b4001c470 comp rx=0 tx=0).stop 2024-02-19T15:27:37.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b5f200640 1 -- 172.21.15.47:0/4286428049 shutdown_connections 2024-02-19T15:27:37.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b5f200640 1 --2- 172.21.15.47:0/4286428049 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b580cedb0 0x7f3b580cfc50 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:37.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b5f200640 1 --2- 172.21.15.47:0/4286428049 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b5815bcd0 0x7f3b580cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:37.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b5f200640 1 -- 172.21.15.47:0/4286428049 >> 172.21.15.47:0/4286428049 conn(0x7f3b580d1bf0 msgr2=0x7f3b580d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:37.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b5f200640 1 -- 172.21.15.47:0/4286428049 shutdown_connections 2024-02-19T15:27:37.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b5f200640 1 -- 172.21.15.47:0/4286428049 wait complete. 2024-02-19T15:27:37.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b5f200640 1 Processor -- start 2024-02-19T15:27:37.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b5f200640 1 -- start start 2024-02-19T15:27:37.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b5f200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b580cedb0 0x7f3b5806d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:37.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b5f200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b580d2710 0x7f3b580c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:37.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b5f200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b5815bcd0 0x7f3b580c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:37.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b5f200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5806d980 con 0x7f3b580d2710 2024-02-19T15:27:37.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b5f200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5806cfb0 con 0x7f3b5815bcd0 2024-02-19T15:27:37.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b5f200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5806c3a0 con 0x7f3b580cedb0 2024-02-19T15:27:37.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b4f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b580d2710 0x7f3b580c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:37.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b4f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b580d2710 0x7f3b580c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60940/0 (socket says 172.21.15.47:60940) 2024-02-19T15:27:37.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b4f400640 1 -- 172.21.15.47:0/17290218 learned_addr learned my addr 172.21.15.47:0/17290218 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:37.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.399+0000 7f3b57e00640 1 --2- 172.21.15.47:0/17290218 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b580cedb0 0x7f3b5806d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:37.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.403+0000 7f3b5d200640 1 --2- 172.21.15.47:0/17290218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b5815bcd0 0x7f3b580c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:37.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.403+0000 7f3b4f400640 1 -- 172.21.15.47:0/17290218 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b580cedb0 msgr2=0x7f3b5806d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:37.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.403+0000 7f3b4f400640 1 --2- 172.21.15.47:0/17290218 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b580cedb0 0x7f3b5806d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:37.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.403+0000 7f3b4f400640 1 -- 172.21.15.47:0/17290218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b5815bcd0 msgr2=0x7f3b580c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:37.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.403+0000 7f3b4f400640 1 --2- 172.21.15.47:0/17290218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b5815bcd0 0x7f3b580c9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:37.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.403+0000 7f3b4f400640 1 -- 172.21.15.47:0/17290218 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b40009420 con 0x7f3b580d2710 2024-02-19T15:27:37.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.403+0000 7f3b4f400640 1 --2- 172.21.15.47:0/17290218 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b580d2710 0x7f3b580c8cc0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f3b4001ab20 tx=0x7f3b4001acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:37.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.403+0000 7f3b54c00640 1 -- 172.21.15.47:0/17290218 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b400add60 con 0x7f3b580d2710 2024-02-19T15:27:37.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.403+0000 7f3b54c00640 1 -- 172.21.15.47:0/17290218 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b400b8450 con 0x7f3b580d2710 2024-02-19T15:27:37.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.403+0000 7f3b54c00640 1 -- 172.21.15.47:0/17290218 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b400b8800 con 0x7f3b580d2710 2024-02-19T15:27:37.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.403+0000 7f3b5f200640 1 -- 172.21.15.47:0/17290218 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b580cdb40 con 0x7f3b580d2710 2024-02-19T15:27:37.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.403+0000 7f3b5f200640 1 -- 172.21.15.47:0/17290218 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5806c960 con 0x7f3b580d2710 2024-02-19T15:27:37.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.403+0000 7f3b5f200640 1 -- 172.21.15.47:0/17290218 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b5806d140 con 0x7f3b580d2710 2024-02-19T15:27:37.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.403+0000 7f3b54c00640 1 -- 172.21.15.47:0/17290218 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3b400b4070 con 0x7f3b580d2710 2024-02-19T15:27:37.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.403+0000 7f3b54c00640 1 --2- 172.21.15.47:0/17290218 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3b34066e40 0x7f3b34069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:37.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.403+0000 7f3b54c00640 1 -- 172.21.15.47:0/17290218 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b4011efc0 con 0x7f3b580d2710 2024-02-19T15:27:37.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.407+0000 7f3b57e00640 1 --2- 172.21.15.47:0/17290218 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3b34066e40 0x7f3b34069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:37.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.407+0000 7f3b57e00640 1 --2- 172.21.15.47:0/17290218 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3b34066e40 0x7f3b34069300 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f3b48009af0 tx=0x7f3b48008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:37.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.407+0000 7f3b54c00640 1 -- 172.21.15.47:0/17290218 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f3b400ebe40 con 0x7f3b580d2710 2024-02-19T15:27:37.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.607+0000 7f3b5f200640 1 -- 172.21.15.47:0/17290218 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b58075360 con 0x7f3b580d2710 2024-02-19T15:27:37.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.607+0000 7f3b54c00640 1 -- 172.21.15.47:0/17290218 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4937 (secure 0 0 0) 0x7f3b400ef500 con 0x7f3b580d2710 2024-02-19T15:27:37.612 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:37.612 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":75,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6578,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/408259797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":408259797},{"type":"v1","addr":"172.21.15.116:6836","nonce":408259797}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7327,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/3795002232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":3795002232},{"type":"v1","addr":"172.21.15.47:6835","nonce":3795002232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7330,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/819751716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":819751716},{"type":"v1","addr":"172.21.15.116:6837","nonce":819751716}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:24.428431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"c","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6834/897369244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":897369244},{"type":"v1","addr":"172.21.15.47:6834","nonce":897369244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:27:37.613 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 75 2024-02-19T15:27:37.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.611+0000 7f3b5f200640 1 -- 172.21.15.47:0/17290218 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3b34066e40 msgr2=0x7f3b34069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:37.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.611+0000 7f3b5f200640 1 --2- 172.21.15.47:0/17290218 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3b34066e40 0x7f3b34069300 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f3b48009af0 tx=0x7f3b48008040 comp rx=0 tx=0).stop 2024-02-19T15:27:37.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.611+0000 7f3b5f200640 1 -- 172.21.15.47:0/17290218 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b580d2710 msgr2=0x7f3b580c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:37.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.611+0000 7f3b5f200640 1 --2- 172.21.15.47:0/17290218 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b580d2710 0x7f3b580c8cc0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f3b4001ab20 tx=0x7f3b4001acb0 comp rx=0 tx=0).stop 2024-02-19T15:27:37.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.611+0000 7f3b5f200640 1 -- 172.21.15.47:0/17290218 shutdown_connections 2024-02-19T15:27:37.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.611+0000 7f3b5f200640 1 --2- 172.21.15.47:0/17290218 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3b34066e40 0x7f3b34069300 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:37.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.611+0000 7f3b5f200640 1 --2- 172.21.15.47:0/17290218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b5815bcd0 0x7f3b580c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:37.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.611+0000 7f3b5f200640 1 --2- 172.21.15.47:0/17290218 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b580d2710 0x7f3b580c8cc0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:37.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.611+0000 7f3b5f200640 1 --2- 172.21.15.47:0/17290218 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b580cedb0 0x7f3b5806d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:37.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.611+0000 7f3b5f200640 1 -- 172.21.15.47:0/17290218 >> 172.21.15.47:0/17290218 conn(0x7f3b580d1bf0 msgr2=0x7f3b580b4810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:37.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.611+0000 7f3b5f200640 1 -- 172.21.15.47:0/17290218 shutdown_connections 2024-02-19T15:27:37.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.611+0000 7f3b5f200640 1 -- 172.21.15.47:0/17290218 wait complete. 2024-02-19T15:27:37.635 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-02-19T15:27:37.801 INFO:teuthology.orchestra.run.smithi047.stdout:{"message":"","return_code":0} 2024-02-19T15:27:37.801 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-02-19T15:27:37.802 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T15:27:37.802 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:27:37.802 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T15:27:37.802 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:27:37.803 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T15:27:37.803 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:27:37.803 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T15:27:37.803 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:27:37.816 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T15:27:37.816 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-02-19T15:27:37.818 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T15:27:37.818 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-02-19T15:27:37.819 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T15:27:37.819 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-02-19T15:27:37.820 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T15:27:37.820 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-02-19T15:27:38.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.999+0000 7f6d17400640 1 Processor -- start 2024-02-19T15:27:38.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:37.999+0000 7f6d17400640 1 -- start start 2024-02-19T15:27:38.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f6d17400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6d100cedb0 0x7f6d100cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f6d17400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d100d2710 0x7f6d100d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f6d17400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d100cc700 con 0x7f6d1015bcd0 2024-02-19T15:27:38.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f6d17400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d100cf5e0 con 0x7f6d100cedb0 2024-02-19T15:27:38.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f6d17400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d100cdb90 con 0x7f6d100d2710 2024-02-19T15:27:38.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f6d0fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6d100cedb0 0x7f6d100cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f6d0fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6d100cedb0 0x7f6d100cfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60950/0 (socket says 172.21.15.47:60950) 2024-02-19T15:27:38.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f6d0fe00640 1 -- 172.21.15.47:0/355873065 learned_addr learned my addr 172.21.15.47:0/355873065 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:38.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f2c28200640 1 Processor -- start 2024-02-19T15:27:38.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f6d15400640 1 --2- 172.21.15.47:0/355873065 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d100d2710 0x7f6d100d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f2c28200640 1 -- start start 2024-02-19T15:27:38.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f6d0fe00640 1 -- 172.21.15.47:0/355873065 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d100d2710 msgr2=0x7f6d100d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:38.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f6d0fe00640 1 --2- 172.21.15.47:0/355873065 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d100d2710 0x7f6d100d0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f6d0fe00640 1 -- 172.21.15.47:0/355873065 >> v1:172.21.15.116:6789/0 conn(0x7f6d1015bcd0 legacy=0x7f6d100cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f6d0fe00640 1 -- 172.21.15.47:0/355873065 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d100cf300 con 0x7f6d100cedb0 2024-02-19T15:27:38.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f2c28200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c200cfd00 0x7f2c200acf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f2c28200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c20067330 con 0x7f2c2015e070 2024-02-19T15:27:38.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f2c28200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2006c740 con 0x7f2c200b2330 2024-02-19T15:27:38.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f2c28200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c20066d70 con 0x7f2c200cfd00 2024-02-19T15:27:38.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f2c25800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f2c2015e070 0x7f2c200b1c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35296/0 (socket says 172.21.15.47:35296) 2024-02-19T15:27:38.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f2c25800640 1 -- 172.21.15.47:0/1924091385 learned_addr learned my addr 172.21.15.47:0/1924091385 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:38.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f2c26200640 1 --2- 172.21.15.47:0/1924091385 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c200cfd00 0x7f2c200acf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f2c1fe00640 1 -- 172.21.15.47:0/1924091385 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918402918 0 0) 0x7f2c2006c740 con 0x7f2c200b2330 2024-02-19T15:27:38.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f2c26200640 1 -- 172.21.15.47:0/1924091385 >> v1:172.21.15.116:6789/0 conn(0x7f2c200b2330 legacy=0x7f2c200ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f2c26200640 1 -- 172.21.15.47:0/1924091385 >> v1:172.21.15.47:6789/0 conn(0x7f2c2015e070 legacy=0x7f2c200b1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.003+0000 7f2c26200640 1 -- 172.21.15.47:0/1924091385 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c20067050 con 0x7f2c200cfd00 2024-02-19T15:27:38.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f7c23a00640 1 Processor -- start 2024-02-19T15:27:38.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f8364c00640 1 Processor -- start 2024-02-19T15:27:38.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f7c23a00640 1 -- start start 2024-02-19T15:27:38.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f8364c00640 1 -- start start 2024-02-19T15:27:38.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f6d0fe00640 1 --2- 172.21.15.47:0/355873065 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d100cedb0 0x7f6d100cfc50 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc0098a0 tx=0x7f6cfc01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d6ffff3125a5935 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:38.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f6d0f400640 1 -- 172.21.15.47:0/355873065 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc0ab680 con 0x7f6d100cedb0 2024-02-19T15:27:38.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f6d0f400640 1 -- 172.21.15.47:0/355873065 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cfc0ab7e0 con 0x7f6d100cedb0 2024-02-19T15:27:38.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f6d0f400640 1 -- 172.21.15.47:0/355873065 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc0abb70 con 0x7f6d100cedb0 2024-02-19T15:27:38.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f8364c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f83601581f0 0x7f83600b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f8364c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83600adfc0 0x7f83600b55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f8364c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8360161aa0 con 0x7f83601585d0 2024-02-19T15:27:38.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f8364c00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8360162400 con 0x7f83601581f0 2024-02-19T15:27:38.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f8364c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83600b37f0 con 0x7f83600adfc0 2024-02-19T15:27:38.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f7c23a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7c1c0cea80 0x7f7c1c0727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f6d17400640 1 -- 172.21.15.47:0/355873065 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d100cedb0 msgr2=0x7f6d100cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f6d17400640 1 --2- 172.21.15.47:0/355873065 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d100cedb0 0x7f6d100cfc50 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc0098a0 tx=0x7f6cfc01c480 comp rx=0 tx=0).stop 2024-02-19T15:27:38.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f7c23a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1c15d280 con 0x7f7c1c145590 2024-02-19T15:27:38.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f7c23a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1c070ff0 con 0x7f7c1c15c390 2024-02-19T15:27:38.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f7c23a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c1c0709e0 con 0x7f7c1c0cea80 2024-02-19T15:27:38.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f6d17400640 1 -- 172.21.15.47:0/355873065 shutdown_connections 2024-02-19T15:27:38.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f6d17400640 1 --2- 172.21.15.47:0/355873065 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d100d2710 0x7f6d100d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f6d17400640 1 --2- 172.21.15.47:0/355873065 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d100cedb0 0x7f6d100cfc50 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f6d17400640 1 -- 172.21.15.47:0/355873065 >> 172.21.15.47:0/355873065 conn(0x7f6d100d1bf0 msgr2=0x7f6d100d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:38.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f7c21000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f7c1c145590 0x7f7c1c06c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35310/0 (socket says 172.21.15.47:35310) 2024-02-19T15:27:38.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f7c21000640 1 -- 172.21.15.47:0/3773129135 learned_addr learned my addr 172.21.15.47:0/3773129135 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:38.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f7c21a00640 1 --1- 172.21.15.47:0/3773129135 >> v1:172.21.15.116:6789/0 conn(0x7f7c1c15c390 0x7f7c1c15c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:53794/0 (socket says 172.21.15.47:53794) 2024-02-19T15:27:38.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f835f400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f83601581f0 0x7f83600b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f835fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83600adfc0 0x7f83600b55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f835f400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f83601581f0 0x7f83600b50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:56190/0 (socket says 172.21.15.47:56190) 2024-02-19T15:27:38.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f835f400640 1 -- 172.21.15.47:0/564303347 learned_addr learned my addr 172.21.15.47:0/564303347 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:38.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f7c1be00640 1 --2- 172.21.15.47:0/3773129135 >> v2:172.21.15.116:3300/0 conn(0x7f7c1c0cea80 0x7f7c1c0727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f6d17400640 1 -- 172.21.15.47:0/355873065 shutdown_connections 2024-02-19T15:27:38.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f6d17400640 1 -- 172.21.15.47:0/355873065 wait complete. 2024-02-19T15:27:38.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f835f400640 1 -- 172.21.15.47:0/564303347 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83600adfc0 msgr2=0x7f83600b55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:38.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f835f400640 1 --2- 172.21.15.47:0/564303347 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83600adfc0 0x7f83600b55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f835f400640 1 -- 172.21.15.47:0/564303347 >> v1:172.21.15.116:6789/0 conn(0x7f83601585d0 legacy=0x7f83600ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f835f400640 1 -- 172.21.15.47:0/564303347 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f836015f310 con 0x7f83601581f0 2024-02-19T15:27:38.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f835ea00640 1 -- 172.21.15.47:0/564303347 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067075360 0 0) 0x7f8360161aa0 con 0x7f83601585d0 2024-02-19T15:27:38.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f7c1b400640 1 -- 172.21.15.47:0/3773129135 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020147034 0 0) 0x7f7c1c070ff0 con 0x7f7c1c15c390 2024-02-19T15:27:38.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f7c1b400640 1 -- 172.21.15.47:0/3773129135 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c04003530 con 0x7f7c1c15c390 2024-02-19T15:27:38.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f7c1b400640 1 -- 172.21.15.47:0/3773129135 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782902541 0 0) 0x7f7c1c15d280 con 0x7f7c1c145590 2024-02-19T15:27:38.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f7c1b400640 1 -- 172.21.15.47:0/3773129135 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c1c070ff0 con 0x7f7c1c145590 2024-02-19T15:27:38.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f7c1be00640 1 -- 172.21.15.47:0/3773129135 >> v1:172.21.15.116:6789/0 conn(0x7f7c1c15c390 legacy=0x7f7c1c15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f7c1be00640 1 -- 172.21.15.47:0/3773129135 >> v1:172.21.15.47:6789/0 conn(0x7f7c1c145590 legacy=0x7f7c1c06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f7c1be00640 1 -- 172.21.15.47:0/3773129135 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c0cf020 con 0x7f7c1c0cea80 2024-02-19T15:27:38.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.007+0000 7f2c26200640 1 --2- 172.21.15.47:0/1924091385 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c200cfd00 0x7f2c200acf30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2c140097f0 tx=0x7f2c1401c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1dd928b0c0ce4f0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:38.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f2c1fe00640 1 -- 172.21.15.47:0/1924091385 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c140ab600 con 0x7f2c200cfd00 2024-02-19T15:27:38.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f2c1fe00640 1 -- 172.21.15.47:0/1924091385 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c140a7050 con 0x7f2c200cfd00 2024-02-19T15:27:38.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f2c1fe00640 1 -- 172.21.15.47:0/1924091385 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c140ab980 con 0x7f2c200cfd00 2024-02-19T15:27:38.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d17400640 1 Processor -- start 2024-02-19T15:27:38.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d17400640 1 -- start start 2024-02-19T15:27:38.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f2c28200640 1 -- 172.21.15.47:0/1924091385 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c200cfd00 msgr2=0x7f2c200acf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f2c28200640 1 --2- 172.21.15.47:0/1924091385 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c200cfd00 0x7f2c200acf30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2c140097f0 tx=0x7f2c1401c340 comp rx=0 tx=0).stop 2024-02-19T15:27:38.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f2c28200640 1 -- 172.21.15.47:0/1924091385 shutdown_connections 2024-02-19T15:27:38.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f2c28200640 1 --2- 172.21.15.47:0/1924091385 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c200cfd00 0x7f2c200acf30 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f2c28200640 1 -- 172.21.15.47:0/1924091385 >> 172.21.15.47:0/1924091385 conn(0x7f2c200ccc20 msgr2=0x7f2c200cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:38.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f2c28200640 1 -- 172.21.15.47:0/1924091385 shutdown_connections 2024-02-19T15:27:38.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f2c28200640 1 -- 172.21.15.47:0/1924091385 wait complete. 2024-02-19T15:27:38.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d17400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d100cedb0 0x7f6d10154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d17400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d100d2710 0x7f6d100c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d17400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d1015bcd0 0x7f6d100c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d17400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d100c82f0 con 0x7f6d100cedb0 2024-02-19T15:27:38.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d17400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d100c8020 con 0x7f6d100d2710 2024-02-19T15:27:38.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d17400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d100c5420 con 0x7f6d1015bcd0 2024-02-19T15:27:38.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d14a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d100cedb0 0x7f6d10154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d14a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d100cedb0 0x7f6d10154d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60964/0 (socket says 172.21.15.47:60964) 2024-02-19T15:27:38.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d14a00640 1 -- 172.21.15.47:0/1445348920 learned_addr learned my addr 172.21.15.47:0/1445348920 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:38.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d15400640 1 --2- 172.21.15.47:0/1445348920 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d1015bcd0 0x7f6d100c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d0fe00640 1 --2- 172.21.15.47:0/1445348920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d100d2710 0x7f6d100c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d14a00640 1 -- 172.21.15.47:0/1445348920 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d1015bcd0 msgr2=0x7f6d100c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d14a00640 1 --2- 172.21.15.47:0/1445348920 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d1015bcd0 0x7f6d100c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d14a00640 1 -- 172.21.15.47:0/1445348920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d100d2710 msgr2=0x7f6d100c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d14a00640 1 --2- 172.21.15.47:0/1445348920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d100d2710 0x7f6d100c61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d14a00640 1 -- 172.21.15.47:0/1445348920 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cfc0094d0 con 0x7f6d100cedb0 2024-02-19T15:27:38.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f7c1be00640 1 --2- 172.21.15.47:0/3773129135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c1c0cea80 0x7f7c1c0727b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7c080097c0 tx=0x7f7c0801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1017e5bce1a0b4a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:38.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d14a00640 1 --2- 172.21.15.47:0/1445348920 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d100cedb0 0x7f6d10154d40 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f6d0400c420 tx=0x7f6d0400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:38.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f7c1b400640 1 -- 172.21.15.47:0/3773129135 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c080ac410 con 0x7f7c1c0cea80 2024-02-19T15:27:38.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d0cc00640 1 -- 172.21.15.47:0/1445348920 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d040052d0 con 0x7f6d100cedb0 2024-02-19T15:27:38.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f7c1b400640 1 -- 172.21.15.47:0/3773129135 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c080ac570 con 0x7f7c1c0cea80 2024-02-19T15:27:38.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d17400640 1 -- 172.21.15.47:0/1445348920 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d1015d240 con 0x7f6d100cedb0 2024-02-19T15:27:38.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f835f400640 1 --2- 172.21.15.47:0/564303347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83601581f0 0x7f83600b50a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8350009e50 tx=0x7f835001c9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=995c2000f9042c24 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:38.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d0cc00640 1 -- 172.21.15.47:0/1445348920 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d04005430 con 0x7f6d100cedb0 2024-02-19T15:27:38.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d0cc00640 1 -- 172.21.15.47:0/1445348920 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d04007260 con 0x7f6d100cedb0 2024-02-19T15:27:38.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f6d17400640 1 -- 172.21.15.47:0/1445348920 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d100c59e0 con 0x7f6d100cedb0 2024-02-19T15:27:38.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.011+0000 7f7c1b400640 1 -- 172.21.15.47:0/3773129135 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c080ac900 con 0x7f7c1c0cea80 2024-02-19T15:27:38.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c28200640 1 Processor -- start 2024-02-19T15:27:38.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f835ea00640 1 -- 172.21.15.47:0/564303347 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83500a5750 con 0x7f83601581f0 2024-02-19T15:27:38.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f835ea00640 1 -- 172.21.15.47:0/564303347 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83500a58b0 con 0x7f83601581f0 2024-02-19T15:27:38.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f835ea00640 1 -- 172.21.15.47:0/564303347 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83500a5c40 con 0x7f83601581f0 2024-02-19T15:27:38.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c28200640 1 -- start start 2024-02-19T15:27:38.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f7c23a00640 1 -- 172.21.15.47:0/3773129135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c1c0cea80 msgr2=0x7f7c1c0727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f7c23a00640 1 --2- 172.21.15.47:0/3773129135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c1c0cea80 0x7f7c1c0727b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7c080097c0 tx=0x7f7c0801c4a0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f8364c00640 1 -- 172.21.15.47:0/564303347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83601581f0 msgr2=0x7f83600b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f8364c00640 1 --2- 172.21.15.47:0/564303347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83601581f0 0x7f83600b50a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8350009e50 tx=0x7f835001c9a0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f7c23a00640 1 -- 172.21.15.47:0/3773129135 shutdown_connections 2024-02-19T15:27:38.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f7c23a00640 1 --2- 172.21.15.47:0/3773129135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c1c0cea80 0x7f7c1c0727b0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f7c23a00640 1 -- 172.21.15.47:0/3773129135 >> 172.21.15.47:0/3773129135 conn(0x7f7c1c0b16a0 msgr2=0x7f7c1c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:38.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f6d17400640 1 -- 172.21.15.47:0/1445348920 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d10075360 con 0x7f6d100cedb0 2024-02-19T15:27:38.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f8364c00640 1 -- 172.21.15.47:0/564303347 shutdown_connections 2024-02-19T15:27:38.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f8364c00640 1 --2- 172.21.15.47:0/564303347 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83600adfc0 0x7f83600b55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f8364c00640 1 --2- 172.21.15.47:0/564303347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83601581f0 0x7f83600b50a0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f8364c00640 1 -- 172.21.15.47:0/564303347 >> 172.21.15.47:0/564303347 conn(0x7f83600b24f0 msgr2=0x7f83600b2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:38.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f7c23a00640 1 -- 172.21.15.47:0/3773129135 shutdown_connections 2024-02-19T15:27:38.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f8364c00640 1 -- 172.21.15.47:0/564303347 shutdown_connections 2024-02-19T15:27:38.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c28200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c200b2330 0x7f2c201503b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c28200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c200cfd00 0x7f2c201610f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f8364c00640 1 -- 172.21.15.47:0/564303347 wait complete. 2024-02-19T15:27:38.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c28200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c2015e070 0x7f2c2014d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c28200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2009c6d0 con 0x7f2c2015e070 2024-02-19T15:27:38.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c28200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2009b910 con 0x7f2c200cfd00 2024-02-19T15:27:38.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c28200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2009bbf0 con 0x7f2c200b2330 2024-02-19T15:27:38.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c24e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c200cfd00 0x7f2c201610f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c24e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c200cfd00 0x7f2c201610f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:56206/0 (socket says 172.21.15.47:56206) 2024-02-19T15:27:38.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c24e00640 1 -- 172.21.15.47:0/2972980835 learned_addr learned my addr 172.21.15.47:0/2972980835 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:38.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c25800640 1 --2- 172.21.15.47:0/2972980835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c200b2330 0x7f2c201503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c26200640 1 --2- 172.21.15.47:0/2972980835 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c2015e070 0x7f2c2014d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c25800640 1 -- 172.21.15.47:0/2972980835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c200cfd00 msgr2=0x7f2c201610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c25800640 1 --2- 172.21.15.47:0/2972980835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c200cfd00 0x7f2c201610f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c25800640 1 -- 172.21.15.47:0/2972980835 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c2015e070 msgr2=0x7f2c2014d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c25800640 1 --2- 172.21.15.47:0/2972980835 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c2015e070 0x7f2c2014d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c25800640 1 -- 172.21.15.47:0/2972980835 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c14009420 con 0x7f2c200b2330 2024-02-19T15:27:38.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c25800640 1 --2- 172.21.15.47:0/2972980835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c200b2330 0x7f2c201503b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2c1800c8c0 tx=0x7f2c1800cd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:38.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f6d0cc00640 1 -- 172.21.15.47:0/1445348920 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6d040073c0 con 0x7f6d100cedb0 2024-02-19T15:27:38.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f7c23a00640 1 -- 172.21.15.47:0/3773129135 wait complete. 2024-02-19T15:27:38.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f8364c00640 1 Processor -- start 2024-02-19T15:27:38.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f7c23a00640 1 Processor -- start 2024-02-19T15:27:38.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f7c23a00640 1 -- start start 2024-02-19T15:27:38.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c1d600640 1 -- 172.21.15.47:0/2972980835 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1800a640 con 0x7f2c200b2330 2024-02-19T15:27:38.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c1d600640 1 -- 172.21.15.47:0/2972980835 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c1800a7a0 con 0x7f2c200b2330 2024-02-19T15:27:38.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c1d600640 1 -- 172.21.15.47:0/2972980835 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1800ab90 con 0x7f2c200b2330 2024-02-19T15:27:38.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f2c28200640 1 -- 172.21.15.47:0/2972980835 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c2009bed0 con 0x7f2c200b2330 2024-02-19T15:27:38.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.015+0000 7f8364c00640 1 -- start start 2024-02-19T15:27:38.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f2c28200640 1 -- 172.21.15.47:0/2972980835 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c2009b5d0 con 0x7f2c200b2330 2024-02-19T15:27:38.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f8364c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83600adfc0 0x7f8360155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f8364c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83601581f0 0x7f8360160a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f8364c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83601585d0 0x7f83600cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f8364c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8360160f80 con 0x7f83601581f0 2024-02-19T15:27:38.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f8364c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83601553e0 con 0x7f83601585d0 2024-02-19T15:27:38.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f8364c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83600cd8e0 con 0x7f83600adfc0 2024-02-19T15:27:38.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f2c28200640 1 -- 172.21.15.47:0/2972980835 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c20075360 con 0x7f2c200b2330 2024-02-19T15:27:38.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f7c23a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c1c0cea80 0x7f7c1c206db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f7c23a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c1c145590 0x7f7c1c2072f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f7c23a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c1c15c390 0x7f7c1c1334c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f7c23a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c135380 con 0x7f7c1c0cea80 2024-02-19T15:27:38.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f7c23a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c135550 con 0x7f7c1c15c390 2024-02-19T15:27:38.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f7c23a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c1350a0 con 0x7f7c1c145590 2024-02-19T15:27:38.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f6d0cc00640 1 --2- 172.21.15.47:0/1445348920 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6cf0066e40 0x7f6cf0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f6d0cc00640 1 -- 172.21.15.47:0/1445348920 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d04084650 con 0x7f6d100cedb0 2024-02-19T15:27:38.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f835f400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83600adfc0 0x7f8360155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f835f400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83600adfc0 0x7f8360155870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:37614/0 (socket says 172.21.15.47:37614) 2024-02-19T15:27:38.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f835f400640 1 -- 172.21.15.47:0/4114489209 learned_addr learned my addr 172.21.15.47:0/4114489209 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:38.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f7c21000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c1c0cea80 0x7f7c1c206db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f7c21a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c1c15c390 0x7f7c1c1334c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f7c1be00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c1c145590 0x7f7c1c2072f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f7c21000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c1c0cea80 0x7f7c1c206db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60988/0 (socket says 172.21.15.47:60988) 2024-02-19T15:27:38.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f8356a00640 1 --2- 172.21.15.47:0/4114489209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83601581f0 0x7f8360160a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f7c21000640 1 -- 172.21.15.47:0/861274493 learned_addr learned my addr 172.21.15.47:0/861274493 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:38.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f835fe00640 1 --2- 172.21.15.47:0/4114489209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83601585d0 0x7f83600cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f7c21a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c1c15c390 0x7f7c1c1334c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:56214/0 (socket says 172.21.15.47:56214) 2024-02-19T15:27:38.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f835f400640 1 -- 172.21.15.47:0/4114489209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83601585d0 msgr2=0x7f83600cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f835f400640 1 --2- 172.21.15.47:0/4114489209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83601585d0 0x7f83600cd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f835f400640 1 -- 172.21.15.47:0/4114489209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83601581f0 msgr2=0x7f8360160a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f835f400640 1 --2- 172.21.15.47:0/4114489209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83601581f0 0x7f8360160a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f835f400640 1 -- 172.21.15.47:0/4114489209 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8350009b30 con 0x7f83600adfc0 2024-02-19T15:27:38.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f8356a00640 1 --2- 172.21.15.47:0/4114489209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83601581f0 0x7f8360160a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:38.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f835f400640 1 --2- 172.21.15.47:0/4114489209 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83600adfc0 0x7f8360155870 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f835001b050 tx=0x7f8350019740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:38.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f7c1be00640 1 -- 172.21.15.47:0/861274493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c1c15c390 msgr2=0x7f7c1c1334c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f7c1be00640 1 --2- 172.21.15.47:0/861274493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c1c15c390 0x7f7c1c1334c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f7c1be00640 1 -- 172.21.15.47:0/861274493 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c1c0cea80 msgr2=0x7f7c1c206db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f7c1be00640 1 --2- 172.21.15.47:0/861274493 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c1c0cea80 0x7f7c1c206db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f7c1be00640 1 -- 172.21.15.47:0/861274493 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c08009420 con 0x7f7c1c145590 2024-02-19T15:27:38.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f7c21000640 1 --2- 172.21.15.47:0/861274493 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c1c0cea80 0x7f7c1c206db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:27:38.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f6d0fe00640 1 --2- 172.21.15.47:0/1445348920 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6cf0066e40 0x7f6cf0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f7c1be00640 1 --2- 172.21.15.47:0/861274493 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c1c145590 0x7f7c1c2072f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7c080ac000 tx=0x7f7c08019240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:38.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f2c1d600640 1 -- 172.21.15.47:0/2972980835 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2c1800acf0 con 0x7f2c200b2330 2024-02-19T15:27:38.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.019+0000 7f6d0fe00640 1 --2- 172.21.15.47:0/1445348920 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6cf0066e40 0x7f6cf0069300 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc0af000 tx=0x7f6cfc0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:38.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.023+0000 7f2c1d600640 1 --2- 172.21.15.47:0/2972980835 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2c04066e40 0x7f2c04069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.023+0000 7f2c1d600640 1 -- 172.21.15.47:0/2972980835 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c18084420 con 0x7f2c200b2330 2024-02-19T15:27:38.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.023+0000 7f2c24e00640 1 --2- 172.21.15.47:0/2972980835 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2c04066e40 0x7f2c04069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.023+0000 7f2c24e00640 1 --2- 172.21.15.47:0/2972980835 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2c04066e40 0x7f2c04069300 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2c2006d4e0 tx=0x7f2c10009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:38.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.023+0000 7f6d0cc00640 1 -- 172.21.15.47:0/1445348920 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6d040514d0 con 0x7f6d100cedb0 2024-02-19T15:27:38.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.023+0000 7f8357e00640 1 -- 172.21.15.47:0/4114489209 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83500b8470 con 0x7f83600adfc0 2024-02-19T15:27:38.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.023+0000 7f8357e00640 1 -- 172.21.15.47:0/4114489209 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83500b8610 con 0x7f83600adfc0 2024-02-19T15:27:38.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.023+0000 7f8357e00640 1 -- 172.21.15.47:0/4114489209 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83500b89c0 con 0x7f83600adfc0 2024-02-19T15:27:38.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.023+0000 7f7c18c00640 1 -- 172.21.15.47:0/861274493 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c080ba070 con 0x7f7c1c145590 2024-02-19T15:27:38.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.023+0000 7f7c18c00640 1 -- 172.21.15.47:0/861274493 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c080b5430 con 0x7f7c1c145590 2024-02-19T15:27:38.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.023+0000 7f2c1d600640 1 -- 172.21.15.47:0/2972980835 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2c180512b0 con 0x7f2c200b2330 2024-02-19T15:27:38.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.023+0000 7f8364c00640 1 -- 172.21.15.47:0/4114489209 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8360161490 con 0x7f83600adfc0 2024-02-19T15:27:38.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.023+0000 7f8364c00640 1 -- 172.21.15.47:0/4114489209 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8360154d90 con 0x7f83600adfc0 2024-02-19T15:27:38.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.023+0000 7f7c18c00640 1 -- 172.21.15.47:0/861274493 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c080b55e0 con 0x7f7c1c145590 2024-02-19T15:27:38.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.023+0000 7f7c23a00640 1 -- 172.21.15.47:0/861274493 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c1c06d940 con 0x7f7c1c145590 2024-02-19T15:27:38.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.023+0000 7f7c23a00640 1 -- 172.21.15.47:0/861274493 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c1c207830 con 0x7f7c1c145590 2024-02-19T15:27:38.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.027+0000 7f8364c00640 1 -- 172.21.15.47:0/4114489209 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83601541d0 con 0x7f83600adfc0 2024-02-19T15:27:38.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.027+0000 7f8357e00640 1 -- 172.21.15.47:0/4114489209 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f83500b8b20 con 0x7f83600adfc0 2024-02-19T15:27:38.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.027+0000 7f7c23a00640 1 -- 172.21.15.47:0/861274493 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c1c075360 con 0x7f7c1c145590 2024-02-19T15:27:38.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.027+0000 7f8357e00640 1 --2- 172.21.15.47:0/4114489209 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f833c066e90 0x7f833c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.027+0000 7f8357e00640 1 -- 172.21.15.47:0/4114489209 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f835011ffc0 con 0x7f83600adfc0 2024-02-19T15:27:38.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.027+0000 7f7c18c00640 1 -- 172.21.15.47:0/861274493 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7c080b3040 con 0x7f7c1c145590 2024-02-19T15:27:38.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.027+0000 7f8356a00640 1 --2- 172.21.15.47:0/4114489209 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f833c066e90 0x7f833c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.027+0000 7f7c18c00640 1 --2- 172.21.15.47:0/861274493 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7bfc066e40 0x7f7bfc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:38.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.027+0000 7f7c18c00640 1 -- 172.21.15.47:0/861274493 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c0811e3c0 con 0x7f7c1c145590 2024-02-19T15:27:38.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.027+0000 7f7c21000640 1 --2- 172.21.15.47:0/861274493 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7bfc066e40 0x7f7bfc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:38.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.027+0000 7f8356a00640 1 --2- 172.21.15.47:0/4114489209 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f833c066e90 0x7f833c069350 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8360153340 tx=0x7f8348009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:38.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.031+0000 7f7c21000640 1 --2- 172.21.15.47:0/861274493 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7bfc066e40 0x7f7bfc069300 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f7c10005890 tx=0x7f7c10009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:38.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.031+0000 7f7c18c00640 1 -- 172.21.15.47:0/861274493 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7c080eb260 con 0x7f7c1c145590 2024-02-19T15:27:38.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.035+0000 7f8357e00640 1 -- 172.21.15.47:0/4114489209 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f83500ecda0 con 0x7f83600adfc0 2024-02-19T15:27:38.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.255+0000 7f2c28200640 1 -- 172.21.15.47:0/2972980835 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f2c2006b190 con 0x7f2c200b2330 2024-02-19T15:27:38.262 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-02-19T15:27:38.259+0000 7fc100600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T15:27:38.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.259+0000 7f6d17400640 1 -- 172.21.15.47:0/1445348920 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f6d100cff90 con 0x7f6d100cedb0 2024-02-19T15:27:38.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.263+0000 7f7c23a00640 1 -- 172.21.15.47:0/861274493 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f7c1c135970 con 0x7f7c1c145590 2024-02-19T15:27:38.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.263+0000 7f8364c00640 1 -- 172.21.15.47:0/4114489209 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f8360075360 con 0x7f83600adfc0 2024-02-19T15:27:38.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.263+0000 7f2c1d600640 1 -- 172.21.15.47:0/2972980835 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7635 v76) v1 ==== 95+0+0 (secure 0 0 0) 0x7f2c18018090 con 0x7f2c200b2330 2024-02-19T15:27:38.269 INFO:teuthology.orchestra.run.smithi047.stderr:failed mds gid 7635 2024-02-19T15:27:38.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.267+0000 7f6d0cc00640 1 -- 172.21.15.47:0/1445348920 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7327 v77) v1 ==== 95+0+0 (secure 0 0 0) 0x7f6d04020090 con 0x7f6d100cedb0 2024-02-19T15:27:38.270 INFO:teuthology.orchestra.run.smithi047.stderr:failed mds gid 7327 2024-02-19T15:27:38.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.267+0000 7f2c28200640 1 -- 172.21.15.47:0/2972980835 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2c04066e40 msgr2=0x7f2c04069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.267+0000 7f2c28200640 1 --2- 172.21.15.47:0/2972980835 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2c04066e40 0x7f2c04069300 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2c2006d4e0 tx=0x7f2c10009040 comp rx=0 tx=0).stop 2024-02-19T15:27:38.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.267+0000 7f2c28200640 1 -- 172.21.15.47:0/2972980835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c200b2330 msgr2=0x7f2c201503b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.267+0000 7f2c28200640 1 --2- 172.21.15.47:0/2972980835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c200b2330 0x7f2c201503b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2c1800c8c0 tx=0x7f2c1800cd80 comp rx=0 tx=0).stop 2024-02-19T15:27:38.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.267+0000 7f2c28200640 1 -- 172.21.15.47:0/2972980835 shutdown_connections 2024-02-19T15:27:38.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.267+0000 7f2c28200640 1 --2- 172.21.15.47:0/2972980835 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2c04066e40 0x7f2c04069300 secure :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2c2006d4e0 tx=0x7f2c10009040 comp rx=0 tx=0).stop 2024-02-19T15:27:38.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.267+0000 7f2c28200640 1 --2- 172.21.15.47:0/2972980835 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c2015e070 0x7f2c2014d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.267+0000 7f2c28200640 1 --2- 172.21.15.47:0/2972980835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c200cfd00 0x7f2c201610f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.267+0000 7f2c28200640 1 --2- 172.21.15.47:0/2972980835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c200b2330 0x7f2c201503b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.267+0000 7f2c28200640 1 -- 172.21.15.47:0/2972980835 >> 172.21.15.47:0/2972980835 conn(0x7f2c200ccc20 msgr2=0x7f2c200717c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:38.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.267+0000 7f2c28200640 1 -- 172.21.15.47:0/2972980835 shutdown_connections 2024-02-19T15:27:38.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.267+0000 7f2c28200640 1 -- 172.21.15.47:0/2972980835 wait complete. 2024-02-19T15:27:38.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.267+0000 7f6d17400640 1 -- 172.21.15.47:0/1445348920 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6cf0066e40 msgr2=0x7f6cf0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.267+0000 7f6d17400640 1 --2- 172.21.15.47:0/1445348920 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6cf0066e40 0x7f6cf0069300 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc0af000 tx=0x7f6cfc0a7000 comp rx=0 tx=0).stop 2024-02-19T15:27:38.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.271+0000 7f6d17400640 1 -- 172.21.15.47:0/1445348920 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d100cedb0 msgr2=0x7f6d10154d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:38.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.271+0000 7f6d17400640 1 --2- 172.21.15.47:0/1445348920 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d100cedb0 0x7f6d10154d40 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f6d0400c420 tx=0x7f6d0400c8e0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.271+0000 7f6d17400640 1 -- 172.21.15.47:0/1445348920 shutdown_connections 2024-02-19T15:27:38.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.271+0000 7f6d17400640 1 --2- 172.21.15.47:0/1445348920 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6cf0066e40 0x7f6cf0069300 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.271+0000 7f6d17400640 1 --2- 172.21.15.47:0/1445348920 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d1015bcd0 0x7f6d100c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.271+0000 7f6d17400640 1 --2- 172.21.15.47:0/1445348920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d100d2710 0x7f6d100c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.271+0000 7f6d17400640 1 --2- 172.21.15.47:0/1445348920 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d100cedb0 0x7f6d10154d40 secure :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f6d0400c420 tx=0x7f6d0400c8e0 comp rx=0 tx=0).stop 2024-02-19T15:27:38.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.271+0000 7f6d17400640 1 -- 172.21.15.47:0/1445348920 >> 172.21.15.47:0/1445348920 conn(0x7f6d100d1bf0 msgr2=0x7f6d100b4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:38.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.271+0000 7f6d17400640 1 -- 172.21.15.47:0/1445348920 shutdown_connections 2024-02-19T15:27:38.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:38.271+0000 7f6d17400640 1 -- 172.21.15.47:0/1445348920 wait complete. 2024-02-19T15:27:38.294 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T15:27:38.295 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T15:27:38.298 INFO:tasks.ceph.mds.c:Started 2024-02-19T15:27:38.299 DEBUG:teuthology.parallel:result is None 2024-02-19T15:27:38.299 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T15:27:38.299 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T15:27:38.344 INFO:tasks.ceph.mds.a:Started 2024-02-19T15:27:38.344 DEBUG:teuthology.parallel:result is None 2024-02-19T15:27:38.383 INFO:tasks.ceph.mds.c.smithi047.stdout:starting mds.c at 2024-02-19T15:27:38.430 INFO:tasks.ceph.mds.a.smithi047.stdout:starting mds.a at 2024-02-19T15:27:39.101 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-02-19T15:27:39.095+0000 7fc0fd400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T15:27:39.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.107+0000 7f8357e00640 1 -- 172.21.15.47:0/4114489209 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6578 v78) v1 ==== 95+0+0 (secure 0 0 0) 0x7f83500f0460 con 0x7f83600adfc0 2024-02-19T15:27:39.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.107+0000 7f7c18c00640 1 -- 172.21.15.47:0/861274493 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 7330 v78) v1 ==== 95+0+0 (secure 0 0 0) 0x7f7c080ee920 con 0x7f7c1c145590 2024-02-19T15:27:39.112 INFO:teuthology.orchestra.run.smithi047.stderr:failed mds gid 6578 2024-02-19T15:27:39.112 INFO:teuthology.orchestra.run.smithi047.stderr:failed mds gid 7330 2024-02-19T15:27:39.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f7c23a00640 1 -- 172.21.15.47:0/861274493 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7bfc066e40 msgr2=0x7f7bfc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:39.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f7c23a00640 1 --2- 172.21.15.47:0/861274493 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7bfc066e40 0x7f7bfc069300 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f7c10005890 tx=0x7f7c10009040 comp rx=0 tx=0).stop 2024-02-19T15:27:39.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f7c23a00640 1 -- 172.21.15.47:0/861274493 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c1c145590 msgr2=0x7f7c1c2072f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:39.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f7c23a00640 1 --2- 172.21.15.47:0/861274493 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c1c145590 0x7f7c1c2072f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7c080ac000 tx=0x7f7c08019240 comp rx=0 tx=0).stop 2024-02-19T15:27:39.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f7c23a00640 1 -- 172.21.15.47:0/861274493 shutdown_connections 2024-02-19T15:27:39.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f7c23a00640 1 --2- 172.21.15.47:0/861274493 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7bfc066e40 0x7f7bfc069300 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:39.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f7c23a00640 1 --2- 172.21.15.47:0/861274493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c1c15c390 0x7f7c1c1334c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:39.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f7c23a00640 1 --2- 172.21.15.47:0/861274493 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c1c145590 0x7f7c1c2072f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:39.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f7c23a00640 1 --2- 172.21.15.47:0/861274493 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c1c0cea80 0x7f7c1c206db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:39.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f7c23a00640 1 -- 172.21.15.47:0/861274493 >> 172.21.15.47:0/861274493 conn(0x7f7c1c0b16a0 msgr2=0x7f7c1c0b0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:39.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f7c23a00640 1 -- 172.21.15.47:0/861274493 shutdown_connections 2024-02-19T15:27:39.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f8364c00640 1 -- 172.21.15.47:0/4114489209 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f833c066e90 msgr2=0x7f833c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:39.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f8364c00640 1 --2- 172.21.15.47:0/4114489209 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f833c066e90 0x7f833c069350 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8360153340 tx=0x7f8348009040 comp rx=0 tx=0).stop 2024-02-19T15:27:39.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f8364c00640 1 -- 172.21.15.47:0/4114489209 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83600adfc0 msgr2=0x7f8360155870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:39.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f8364c00640 1 --2- 172.21.15.47:0/4114489209 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83600adfc0 0x7f8360155870 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f835001b050 tx=0x7f8350019740 comp rx=0 tx=0).stop 2024-02-19T15:27:39.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f7c23a00640 1 -- 172.21.15.47:0/861274493 wait complete. 2024-02-19T15:27:39.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f8364c00640 1 -- 172.21.15.47:0/4114489209 shutdown_connections 2024-02-19T15:27:39.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f8364c00640 1 --2- 172.21.15.47:0/4114489209 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f833c066e90 0x7f833c069350 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:39.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f8364c00640 1 --2- 172.21.15.47:0/4114489209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83601585d0 0x7f83600cd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:39.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f8364c00640 1 --2- 172.21.15.47:0/4114489209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f83601581f0 0x7f8360160a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:39.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f8364c00640 1 --2- 172.21.15.47:0/4114489209 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83600adfc0 0x7f8360155870 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:39.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f8364c00640 1 -- 172.21.15.47:0/4114489209 >> 172.21.15.47:0/4114489209 conn(0x7f83600b24f0 msgr2=0x7f8360153ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:39.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f8364c00640 1 -- 172.21.15.47:0/4114489209 shutdown_connections 2024-02-19T15:27:39.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.111+0000 7f8364c00640 1 -- 172.21.15.47:0/4114489209 wait complete. 2024-02-19T15:27:39.147 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T15:27:39.147 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T15:27:39.148 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T15:27:39.148 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T15:27:39.151 INFO:tasks.ceph.mds.b:Started 2024-02-19T15:27:39.151 DEBUG:teuthology.parallel:result is None 2024-02-19T15:27:39.152 INFO:tasks.ceph.mds.d:Started 2024-02-19T15:27:39.152 DEBUG:teuthology.parallel:result is None 2024-02-19T15:27:39.153 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:27:39.237 INFO:tasks.ceph.mds.b.smithi116.stdout:starting mds.b at 2024-02-19T15:27:39.237 INFO:tasks.ceph.mds.d.smithi116.stdout:starting mds.d at 2024-02-19T15:27:39.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.331+0000 7fae89000640 1 Processor -- start 2024-02-19T15:27:39.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.331+0000 7fae89000640 1 -- start start 2024-02-19T15:27:39.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.331+0000 7fae89000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fae840cea80 0x7fae840727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:39.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.331+0000 7fae89000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fae8415c390 0x7fae8415c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:39.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.331+0000 7fae89000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8415d280 con 0x7fae84145590 2024-02-19T15:27:39.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.331+0000 7fae89000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae84070ff0 con 0x7fae8415c390 2024-02-19T15:27:39.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.331+0000 7fae89000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8415cd40 con 0x7fae840cea80 2024-02-19T15:27:39.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.331+0000 7fae82800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fae8415c390 0x7fae8415c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:39.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.331+0000 7fae79400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fae840cea80 0x7fae840727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:39.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.331+0000 7fae82800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fae8415c390 0x7fae8415c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:32784/0 (socket says 172.21.15.47:32784) 2024-02-19T15:27:39.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.331+0000 7fae82800640 1 -- 172.21.15.47:0/32028836 learned_addr learned my addr 172.21.15.47:0/32028836 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:39.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.331+0000 7fae81e00640 1 --1- 172.21.15.47:0/32028836 >> v1:172.21.15.47:6789/0 conn(0x7fae84145590 0x7fae8406c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35328/0 (socket says 172.21.15.47:35328) 2024-02-19T15:27:39.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.331+0000 7fae82800640 1 -- 172.21.15.47:0/32028836 >> v2:172.21.15.116:3300/0 conn(0x7fae840cea80 msgr2=0x7fae840727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:39.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.331+0000 7fae82800640 1 --2- 172.21.15.47:0/32028836 >> v2:172.21.15.116:3300/0 conn(0x7fae840cea80 0x7fae840727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:39.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.331+0000 7fae82800640 1 -- 172.21.15.47:0/32028836 >> v1:172.21.15.47:6789/0 conn(0x7fae84145590 legacy=0x7fae8406c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:39.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.331+0000 7fae82800640 1 -- 172.21.15.47:0/32028836 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae840709e0 con 0x7fae8415c390 2024-02-19T15:27:39.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.335+0000 7fae82800640 1 --2- 172.21.15.47:0/32028836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae8415c390 0x7fae8415c800 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fae8406c770 tx=0x7fae7c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a368e5bbea00033 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:39.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.335+0000 7fae81400640 1 -- 172.21.15.47:0/32028836 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c0a57a0 con 0x7fae8415c390 2024-02-19T15:27:39.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.335+0000 7fae81400640 1 -- 172.21.15.47:0/32028836 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7c0a5900 con 0x7fae8415c390 2024-02-19T15:27:39.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.335+0000 7fae81400640 1 -- 172.21.15.47:0/32028836 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c0a5c90 con 0x7fae8415c390 2024-02-19T15:27:39.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.335+0000 7fae89000640 1 -- 172.21.15.47:0/32028836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae8415c390 msgr2=0x7fae8415c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:39.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.335+0000 7fae89000640 1 --2- 172.21.15.47:0/32028836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae8415c390 0x7fae8415c800 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fae8406c770 tx=0x7fae7c01c9f0 comp rx=0 tx=0).stop 2024-02-19T15:27:39.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.335+0000 7fae89000640 1 -- 172.21.15.47:0/32028836 shutdown_connections 2024-02-19T15:27:39.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.335+0000 7fae89000640 1 --2- 172.21.15.47:0/32028836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae8415c390 0x7fae8415c800 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:39.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.335+0000 7fae89000640 1 --2- 172.21.15.47:0/32028836 >> v2:172.21.15.116:3300/0 conn(0x7fae840cea80 0x7fae840727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:39.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.335+0000 7fae89000640 1 -- 172.21.15.47:0/32028836 >> 172.21.15.47:0/32028836 conn(0x7fae840b16a0 msgr2=0x7fae840b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:39.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.335+0000 7fae89000640 1 -- 172.21.15.47:0/32028836 shutdown_connections 2024-02-19T15:27:39.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.335+0000 7fae89000640 1 -- 172.21.15.47:0/32028836 wait complete. 2024-02-19T15:27:39.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae89000640 1 Processor -- start 2024-02-19T15:27:39.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae89000640 1 -- start start 2024-02-19T15:27:39.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae89000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae840cea80 0x7fae84157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:39.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae89000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae84145590 0x7fae8413b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:39.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae89000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae8415c390 0x7fae8406d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:39.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae89000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8413fc90 con 0x7fae840cea80 2024-02-19T15:27:39.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae89000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8413cea0 con 0x7fae84145590 2024-02-19T15:27:39.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae89000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae8413d330 con 0x7fae8415c390 2024-02-19T15:27:39.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae81e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae840cea80 0x7fae84157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:39.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae81e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae840cea80 0x7fae84157750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:32794/0 (socket says 172.21.15.47:32794) 2024-02-19T15:27:39.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae81e00640 1 -- 172.21.15.47:0/32532281 learned_addr learned my addr 172.21.15.47:0/32532281 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:39.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae82800640 1 --2- 172.21.15.47:0/32532281 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae8415c390 0x7fae8406d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:39.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae79400640 1 --2- 172.21.15.47:0/32532281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae84145590 0x7fae8413b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:39.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae81e00640 1 -- 172.21.15.47:0/32532281 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae8415c390 msgr2=0x7fae8406d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:39.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae81e00640 1 --2- 172.21.15.47:0/32532281 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae8415c390 0x7fae8406d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:39.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae81e00640 1 -- 172.21.15.47:0/32532281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae84145590 msgr2=0x7fae8413b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:39.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae81e00640 1 --2- 172.21.15.47:0/32532281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae84145590 0x7fae8413b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:39.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae81e00640 1 -- 172.21.15.47:0/32532281 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7c009b30 con 0x7fae840cea80 2024-02-19T15:27:39.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae79400640 1 --2- 172.21.15.47:0/32532281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae84145590 0x7fae8413b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:27:39.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae81e00640 1 --2- 172.21.15.47:0/32532281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae840cea80 0x7fae84157750 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fae7400c420 tx=0x7fae7400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:39.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae7aa00640 1 -- 172.21.15.47:0/32532281 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae740074e0 con 0x7fae840cea80 2024-02-19T15:27:39.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae7aa00640 1 -- 172.21.15.47:0/32532281 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae74007640 con 0x7fae840cea80 2024-02-19T15:27:39.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae7aa00640 1 -- 172.21.15.47:0/32532281 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae74004a30 con 0x7fae840cea80 2024-02-19T15:27:39.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae89000640 1 -- 172.21.15.47:0/32532281 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae8413c290 con 0x7fae840cea80 2024-02-19T15:27:39.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.339+0000 7fae89000640 1 -- 172.21.15.47:0/32532281 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8413c850 con 0x7fae840cea80 2024-02-19T15:27:39.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.343+0000 7fae89000640 1 -- 172.21.15.47:0/32532281 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae84075360 con 0x7fae840cea80 2024-02-19T15:27:39.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.343+0000 7fae7aa00640 1 -- 172.21.15.47:0/32532281 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fae74004b90 con 0x7fae840cea80 2024-02-19T15:27:39.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.343+0000 7fae7aa00640 1 --2- 172.21.15.47:0/32532281 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fae60066e40 0x7fae60069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:39.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.343+0000 7fae7aa00640 1 -- 172.21.15.47:0/32532281 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fae74084470 con 0x7fae840cea80 2024-02-19T15:27:39.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.343+0000 7fae79400640 1 --2- 172.21.15.47:0/32532281 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fae60066e40 0x7fae60069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:39.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.343+0000 7fae79400640 1 --2- 172.21.15.47:0/32532281 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fae60066e40 0x7fae60069300 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fae84144d20 tx=0x7fae6c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:39.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.347+0000 7fae7aa00640 1 -- 172.21.15.47:0/32532281 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fae740511a0 con 0x7fae840cea80 2024-02-19T15:27:39.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.547+0000 7fae89000640 1 -- 172.21.15.47:0/32532281 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae8406cbb0 con 0x7fae840cea80 2024-02-19T15:27:39.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.547+0000 7fae7aa00640 1 -- 172.21.15.47:0/32532281 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+3357 (secure 0 0 0) 0x7fae74054860 con 0x7fae840cea80 2024-02-19T15:27:39.551 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:39.552 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":79,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:39.109800+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:replay","state_seq":1,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:27:39.552 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 79 2024-02-19T15:27:39.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.547+0000 7fae89000640 1 -- 172.21.15.47:0/32532281 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fae60066e40 msgr2=0x7fae60069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:39.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.547+0000 7fae89000640 1 --2- 172.21.15.47:0/32532281 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fae60066e40 0x7fae60069300 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fae84144d20 tx=0x7fae6c008040 comp rx=0 tx=0).stop 2024-02-19T15:27:39.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.547+0000 7fae89000640 1 -- 172.21.15.47:0/32532281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae840cea80 msgr2=0x7fae84157750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:39.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.547+0000 7fae89000640 1 --2- 172.21.15.47:0/32532281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae840cea80 0x7fae84157750 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fae7400c420 tx=0x7fae7400c8e0 comp rx=0 tx=0).stop 2024-02-19T15:27:39.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.547+0000 7fae89000640 1 -- 172.21.15.47:0/32532281 shutdown_connections 2024-02-19T15:27:39.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.547+0000 7fae89000640 1 --2- 172.21.15.47:0/32532281 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fae60066e40 0x7fae60069300 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:39.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.547+0000 7fae89000640 1 --2- 172.21.15.47:0/32532281 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae8415c390 0x7fae8406d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:39.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.547+0000 7fae89000640 1 --2- 172.21.15.47:0/32532281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae84145590 0x7fae8413b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:39.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.547+0000 7fae89000640 1 --2- 172.21.15.47:0/32532281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae840cea80 0x7fae84157750 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:39.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.547+0000 7fae89000640 1 -- 172.21.15.47:0/32532281 >> 172.21.15.47:0/32532281 conn(0x7fae840b16a0 msgr2=0x7fae8413a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:39.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.551+0000 7fae89000640 1 -- 172.21.15.47:0/32532281 shutdown_connections 2024-02-19T15:27:39.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:39.551+0000 7fae89000640 1 -- 172.21.15.47:0/32532281 wait complete. 2024-02-19T15:27:39.976 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:27:21.415321+0000', 'modified': '2024-02-19T15:27:39.109800+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7920}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7920': {'gid': 7920, 'name': 'a', 'rank': 0, 'incarnation': 79, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.47:6835/3636833601', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6834', 'nonce': 3636833601}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 3636833601}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:27:39.977 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7920:up:replay 2024-02-19T15:27:40.977 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:27:41.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.127+0000 7fd33e000640 1 Processor -- start 2024-02-19T15:27:41.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.127+0000 7fd33e000640 1 -- start start 2024-02-19T15:27:41.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.127+0000 7fd33e000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd33815c390 0x7fd3380b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:41.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.127+0000 7fd33e000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd3380d0830 0x7fd3380cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:41.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.127+0000 7fd33e000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3380d2800 con 0x7fd3380cf920 2024-02-19T15:27:41.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.127+0000 7fd33e000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3380d2ae0 con 0x7fd33815c390 2024-02-19T15:27:41.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.127+0000 7fd33e000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3380cfdb0 con 0x7fd3380d0830 2024-02-19T15:27:41.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.127+0000 7fd337400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd33815c390 0x7fd3380b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:41.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.127+0000 7fd337400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd33815c390 0x7fd3380b2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:32810/0 (socket says 172.21.15.47:32810) 2024-02-19T15:27:41.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.127+0000 7fd337400640 1 -- 172.21.15.47:0/2605472392 learned_addr learned my addr 172.21.15.47:0/2605472392 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:41.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.127+0000 7fd337e00640 1 --2- 172.21.15.47:0/2605472392 >> v2:172.21.15.116:3300/0 conn(0x7fd3380d0830 0x7fd3380cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:41.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.127+0000 7fd337400640 1 -- 172.21.15.47:0/2605472392 >> v2:172.21.15.116:3300/0 conn(0x7fd3380d0830 msgr2=0x7fd3380cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:41.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.127+0000 7fd337400640 1 --2- 172.21.15.47:0/2605472392 >> v2:172.21.15.116:3300/0 conn(0x7fd3380d0830 0x7fd3380cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:41.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.127+0000 7fd337400640 1 -- 172.21.15.47:0/2605472392 >> v1:172.21.15.116:6789/0 conn(0x7fd3380cf920 legacy=0x7fd3380d0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:41.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.127+0000 7fd337400640 1 -- 172.21.15.47:0/2605472392 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3380b2650 con 0x7fd33815c390 2024-02-19T15:27:41.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.127+0000 7fd337e00640 1 --2- 172.21.15.47:0/2605472392 >> v2:172.21.15.116:3300/0 conn(0x7fd3380d0830 0x7fd3380cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:41.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.131+0000 7fd337400640 1 --2- 172.21.15.47:0/2605472392 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd33815c390 0x7fd3380b2c10 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fd3380b13f0 tx=0x7fd32801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bd513c1d4b5c358 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:41.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.131+0000 7fd336000640 1 -- 172.21.15.47:0/2605472392 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3280ab640 con 0x7fd33815c390 2024-02-19T15:27:41.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.131+0000 7fd336000640 1 -- 172.21.15.47:0/2605472392 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3280a7070 con 0x7fd33815c390 2024-02-19T15:27:41.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.131+0000 7fd336000640 1 -- 172.21.15.47:0/2605472392 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3280ab9c0 con 0x7fd33815c390 2024-02-19T15:27:41.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.131+0000 7fd33e000640 1 -- 172.21.15.47:0/2605472392 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd33815c390 msgr2=0x7fd3380b2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:41.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.131+0000 7fd33e000640 1 --2- 172.21.15.47:0/2605472392 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd33815c390 0x7fd3380b2c10 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fd3380b13f0 tx=0x7fd32801c420 comp rx=0 tx=0).stop 2024-02-19T15:27:41.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.131+0000 7fd33e000640 1 -- 172.21.15.47:0/2605472392 shutdown_connections 2024-02-19T15:27:41.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.131+0000 7fd33e000640 1 --2- 172.21.15.47:0/2605472392 >> v2:172.21.15.116:3300/0 conn(0x7fd3380d0830 0x7fd3380cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:41.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.131+0000 7fd33e000640 1 --2- 172.21.15.47:0/2605472392 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd33815c390 0x7fd3380b2c10 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:41.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.131+0000 7fd33e000640 1 -- 172.21.15.47:0/2605472392 >> 172.21.15.47:0/2605472392 conn(0x7fd3380d1b40 msgr2=0x7fd3380d1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:41.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.131+0000 7fd33e000640 1 -- 172.21.15.47:0/2605472392 shutdown_connections 2024-02-19T15:27:41.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.131+0000 7fd33e000640 1 -- 172.21.15.47:0/2605472392 wait complete. 2024-02-19T15:27:41.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.131+0000 7fd33e000640 1 Processor -- start 2024-02-19T15:27:41.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.131+0000 7fd33e000640 1 -- start start 2024-02-19T15:27:41.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd33e000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3380d0830 0x7fd338152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:41.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd33e000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd33815c390 0x7fd3381f6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:41.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd33e000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd338153110 0x7fd33806c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:41.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd33e000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33806f5a0 con 0x7fd338153110 2024-02-19T15:27:41.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd33e000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33806ecb0 con 0x7fd33815c390 2024-02-19T15:27:41.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd33e000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33806ef50 con 0x7fd3380d0830 2024-02-19T15:27:41.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd337e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd338153110 0x7fd33806c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:41.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd337e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd338153110 0x7fd33806c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:32824/0 (socket says 172.21.15.47:32824) 2024-02-19T15:27:41.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd337e00640 1 -- 172.21.15.47:0/2014740511 learned_addr learned my addr 172.21.15.47:0/2014740511 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:41.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd337400640 1 --2- 172.21.15.47:0/2014740511 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3380d0830 0x7fd338152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:41.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd336a00640 1 --2- 172.21.15.47:0/2014740511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd33815c390 0x7fd3381f6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:41.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd337e00640 1 -- 172.21.15.47:0/2014740511 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3380d0830 msgr2=0x7fd338152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:41.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd337e00640 1 --2- 172.21.15.47:0/2014740511 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3380d0830 0x7fd338152960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:41.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd337e00640 1 -- 172.21.15.47:0/2014740511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd33815c390 msgr2=0x7fd3381f6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:41.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd337e00640 1 --2- 172.21.15.47:0/2014740511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd33815c390 0x7fd3381f6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:41.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd337e00640 1 -- 172.21.15.47:0/2014740511 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3280094d0 con 0x7fd338153110 2024-02-19T15:27:41.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd337e00640 1 --2- 172.21.15.47:0/2014740511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd338153110 0x7fd33806c9b0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fd33000cef0 tx=0x7fd33000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:41.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd32f400640 1 -- 172.21.15.47:0/2014740511 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33000d840 con 0x7fd338153110 2024-02-19T15:27:41.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd32f400640 1 -- 172.21.15.47:0/2014740511 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33000d9c0 con 0x7fd338153110 2024-02-19T15:27:41.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd33e000640 1 -- 172.21.15.47:0/2014740511 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33806f230 con 0x7fd338153110 2024-02-19T15:27:41.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd32f400640 1 -- 172.21.15.47:0/2014740511 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33000dd70 con 0x7fd338153110 2024-02-19T15:27:41.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd33e000640 1 -- 172.21.15.47:0/2014740511 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3380c70e0 con 0x7fd338153110 2024-02-19T15:27:41.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.135+0000 7fd33e000640 1 -- 172.21.15.47:0/2014740511 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33806f750 con 0x7fd338153110 2024-02-19T15:27:41.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.139+0000 7fd32f400640 1 -- 172.21.15.47:0/2014740511 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd33001e3b0 con 0x7fd338153110 2024-02-19T15:27:41.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.139+0000 7fd32f400640 1 --2- 172.21.15.47:0/2014740511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd318066e70 0x7fd318069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:41.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.139+0000 7fd32f400640 1 -- 172.21.15.47:0/2014740511 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fd330085490 con 0x7fd338153110 2024-02-19T15:27:41.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.139+0000 7fd337400640 1 --2- 172.21.15.47:0/2014740511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd318066e70 0x7fd318069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:41.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.139+0000 7fd337400640 1 --2- 172.21.15.47:0/2014740511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd318066e70 0x7fd318069330 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd3280af000 tx=0x7fd3280b6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:41.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.143+0000 7fd32f400640 1 -- 172.21.15.47:0/2014740511 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd33806f750 con 0x7fd338153110 2024-02-19T15:27:41.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.335+0000 7fd33e000640 1 -- 172.21.15.47:0/2014740511 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd338075360 con 0x7fd338153110 2024-02-19T15:27:41.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.335+0000 7fd32f400640 1 -- 172.21.15.47:0/2014740511 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4942 (secure 0 0 0) 0x7fd330052330 con 0x7fd338153110 2024-02-19T15:27:41.342 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:41.342 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":81,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:40.121540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:27:41.342 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 81 2024-02-19T15:27:41.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.339+0000 7fd33e000640 1 -- 172.21.15.47:0/2014740511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd318066e70 msgr2=0x7fd318069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:41.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.339+0000 7fd33e000640 1 --2- 172.21.15.47:0/2014740511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd318066e70 0x7fd318069330 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd3280af000 tx=0x7fd3280b6040 comp rx=0 tx=0).stop 2024-02-19T15:27:41.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.339+0000 7fd33e000640 1 -- 172.21.15.47:0/2014740511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd338153110 msgr2=0x7fd33806c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:41.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.339+0000 7fd33e000640 1 --2- 172.21.15.47:0/2014740511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd338153110 0x7fd33806c9b0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fd33000cef0 tx=0x7fd33000a710 comp rx=0 tx=0).stop 2024-02-19T15:27:41.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.339+0000 7fd33e000640 1 -- 172.21.15.47:0/2014740511 shutdown_connections 2024-02-19T15:27:41.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.339+0000 7fd33e000640 1 --2- 172.21.15.47:0/2014740511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd318066e70 0x7fd318069330 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:41.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.339+0000 7fd33e000640 1 --2- 172.21.15.47:0/2014740511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd338153110 0x7fd33806c9b0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:41.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.339+0000 7fd33e000640 1 --2- 172.21.15.47:0/2014740511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd33815c390 0x7fd3381f6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:41.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.339+0000 7fd33e000640 1 --2- 172.21.15.47:0/2014740511 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3380d0830 0x7fd338152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:41.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.339+0000 7fd33e000640 1 -- 172.21.15.47:0/2014740511 >> 172.21.15.47:0/2014740511 conn(0x7fd3380d1b40 msgr2=0x7fd3380d0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:41.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.339+0000 7fd33e000640 1 -- 172.21.15.47:0/2014740511 shutdown_connections 2024-02-19T15:27:41.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:41.339+0000 7fd33e000640 1 -- 172.21.15.47:0/2014740511 wait complete. 2024-02-19T15:27:41.765 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:27:21.415321+0000', 'modified': '2024-02-19T15:27:40.121540+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7920}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7920': {'gid': 7920, 'name': 'a', 'rank': 0, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.47:6835/3636833601', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6834', 'nonce': 3636833601}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 3636833601}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:27:41.765 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7920:up:rejoin 2024-02-19T15:27:42.765 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:27:42.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.903+0000 7fc4e9c00640 1 Processor -- start 2024-02-19T15:27:42.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.903+0000 7fc4e9c00640 1 -- start start 2024-02-19T15:27:42.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.903+0000 7fc4e9c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc4e406b5c0 0x7fc4e4166c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:42.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.903+0000 7fc4e9c00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4e415fca0 con 0x7fc4e406d760 2024-02-19T15:27:42.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.903+0000 7fc4e9c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4e415ff80 con 0x7fc4e4144b00 2024-02-19T15:27:42.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.903+0000 7fc4e9c00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4e41608b0 con 0x7fc4e406b5c0 2024-02-19T15:27:42.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.903+0000 7fc4e2000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc4e406d760 0x7fc4e406aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35330/0 (socket says 172.21.15.47:35330) 2024-02-19T15:27:42.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.903+0000 7fc4e2000640 1 -- 172.21.15.47:0/3216871064 learned_addr learned my addr 172.21.15.47:0/3216871064 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:42.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.903+0000 7fc4e3400640 1 --2- 172.21.15.47:0/3216871064 >> v2:172.21.15.116:3300/0 conn(0x7fc4e406b5c0 0x7fc4e4166c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:42.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.903+0000 7fc4e1600640 1 -- 172.21.15.47:0/3216871064 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803947777 0 0) 0x7fc4e415fca0 con 0x7fc4e406d760 2024-02-19T15:27:42.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.903+0000 7fc4e1600640 1 -- 172.21.15.47:0/3216871064 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4c8003530 con 0x7fc4e406d760 2024-02-19T15:27:42.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.903+0000 7fc4e3400640 1 -- 172.21.15.47:0/3216871064 >> v1:172.21.15.116:6789/0 conn(0x7fc4e4144b00 legacy=0x7fc4e4144ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:42.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.903+0000 7fc4e3400640 1 -- 172.21.15.47:0/3216871064 >> v1:172.21.15.47:6789/0 conn(0x7fc4e406d760 legacy=0x7fc4e406aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:42.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.907+0000 7fc4e3400640 1 -- 172.21.15.47:0/3216871064 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e4160260 con 0x7fc4e406b5c0 2024-02-19T15:27:42.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.907+0000 7fc4e3400640 1 --2- 172.21.15.47:0/3216871064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e406b5c0 0x7fc4e4166c30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc009dc0 tx=0x7fc4dc01c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca41a08f8c2537ef server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:42.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.907+0000 7fc4e1600640 1 -- 172.21.15.47:0/3216871064 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc0a5920 con 0x7fc4e406b5c0 2024-02-19T15:27:42.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.907+0000 7fc4e1600640 1 -- 172.21.15.47:0/3216871064 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4dc0a5a80 con 0x7fc4e406b5c0 2024-02-19T15:27:42.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.907+0000 7fc4e1600640 1 -- 172.21.15.47:0/3216871064 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc0a5e10 con 0x7fc4e406b5c0 2024-02-19T15:27:42.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.907+0000 7fc4e9c00640 1 -- 172.21.15.47:0/3216871064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e406b5c0 msgr2=0x7fc4e4166c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:42.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.907+0000 7fc4e9c00640 1 --2- 172.21.15.47:0/3216871064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e406b5c0 0x7fc4e4166c30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc009dc0 tx=0x7fc4dc01c950 comp rx=0 tx=0).stop 2024-02-19T15:27:42.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e9c00640 1 -- 172.21.15.47:0/3216871064 shutdown_connections 2024-02-19T15:27:42.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e9c00640 1 --2- 172.21.15.47:0/3216871064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e406b5c0 0x7fc4e4166c30 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:42.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e9c00640 1 -- 172.21.15.47:0/3216871064 >> 172.21.15.47:0/3216871064 conn(0x7fc4e40d0ad0 msgr2=0x7fc4e40d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:42.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e9c00640 1 -- 172.21.15.47:0/3216871064 shutdown_connections 2024-02-19T15:27:42.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e9c00640 1 -- 172.21.15.47:0/3216871064 wait complete. 2024-02-19T15:27:42.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e9c00640 1 Processor -- start 2024-02-19T15:27:42.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e9c00640 1 -- start start 2024-02-19T15:27:42.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e9c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4e406b5c0 0x7fc4e40c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:42.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e9c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4e406d760 0x7fc4e41669e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:42.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e9c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e4144b00 0x7fc4e41403e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:42.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e9c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e406e000 con 0x7fc4e406d760 2024-02-19T15:27:42.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e9c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e4164e00 con 0x7fc4e4144b00 2024-02-19T15:27:42.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e9c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e40c80e0 con 0x7fc4e406b5c0 2024-02-19T15:27:42.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e2000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4e406d760 0x7fc4e41669e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:42.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e2000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4e406d760 0x7fc4e41669e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:32830/0 (socket says 172.21.15.47:32830) 2024-02-19T15:27:42.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e2000640 1 -- 172.21.15.47:0/992010346 learned_addr learned my addr 172.21.15.47:0/992010346 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:42.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e3400640 1 --2- 172.21.15.47:0/992010346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e4144b00 0x7fc4e41403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:42.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e2a00640 1 --2- 172.21.15.47:0/992010346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4e406b5c0 0x7fc4e40c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:42.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e2000640 1 -- 172.21.15.47:0/992010346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4e406b5c0 msgr2=0x7fc4e40c7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:42.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e2000640 1 --2- 172.21.15.47:0/992010346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4e406b5c0 0x7fc4e40c7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:42.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e2000640 1 -- 172.21.15.47:0/992010346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e4144b00 msgr2=0x7fc4e41403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:42.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e2000640 1 --2- 172.21.15.47:0/992010346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e4144b00 0x7fc4e41403e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:42.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e2000640 1 -- 172.21.15.47:0/992010346 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc009a70 con 0x7fc4e406d760 2024-02-19T15:27:42.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e3400640 1 --2- 172.21.15.47:0/992010346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e4144b00 0x7fc4e41403e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:27:42.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4e2000640 1 --2- 172.21.15.47:0/992010346 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4e406d760 0x7fc4e41669e0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fc4d000c4d0 tx=0x7fc4d000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:42.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.911+0000 7fc4daa00640 1 -- 172.21.15.47:0/992010346 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0005cf0 con 0x7fc4e406d760 2024-02-19T15:27:42.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.915+0000 7fc4e9c00640 1 -- 172.21.15.47:0/992010346 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4e415c210 con 0x7fc4e406d760 2024-02-19T15:27:42.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.915+0000 7fc4daa00640 1 -- 172.21.15.47:0/992010346 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d0005e50 con 0x7fc4e406d760 2024-02-19T15:27:42.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.915+0000 7fc4daa00640 1 -- 172.21.15.47:0/992010346 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0007270 con 0x7fc4e406d760 2024-02-19T15:27:42.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.915+0000 7fc4e9c00640 1 -- 172.21.15.47:0/992010346 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4e415c7d0 con 0x7fc4e406d760 2024-02-19T15:27:42.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.915+0000 7fc4e9c00640 1 -- 172.21.15.47:0/992010346 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4e4164f70 con 0x7fc4e406d760 2024-02-19T15:27:42.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.915+0000 7fc4daa00640 1 -- 172.21.15.47:0/992010346 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc4d00073d0 con 0x7fc4e406d760 2024-02-19T15:27:42.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.915+0000 7fc4daa00640 1 --2- 172.21.15.47:0/992010346 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc4c4066e40 0x7fc4c4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:42.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.915+0000 7fc4daa00640 1 -- 172.21.15.47:0/992010346 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fc4d00840f0 con 0x7fc4e406d760 2024-02-19T15:27:42.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.915+0000 7fc4e2a00640 1 --2- 172.21.15.47:0/992010346 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc4c4066e40 0x7fc4c4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:42.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.915+0000 7fc4e2a00640 1 --2- 172.21.15.47:0/992010346 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc4c4066e40 0x7fc4c4069300 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4000f70 tx=0x7fc4d4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:42.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:42.919+0000 7fc4daa00640 1 -- 172.21.15.47:0/992010346 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc4e4164f70 con 0x7fc4e406d760 2024-02-19T15:27:43.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.119+0000 7fc4e9c00640 1 -- 172.21.15.47:0/992010346 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4e4164f70 con 0x7fc4e406d760 2024-02-19T15:27:43.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.119+0000 7fc4daa00640 1 -- 172.21.15.47:0/992010346 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7fc4e4164f70 con 0x7fc4e406d760 2024-02-19T15:27:43.124 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:43.124 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:27:43.124 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:27:43.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.123+0000 7fc4e9c00640 1 -- 172.21.15.47:0/992010346 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc4c4066e40 msgr2=0x7fc4c4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:43.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.123+0000 7fc4e9c00640 1 --2- 172.21.15.47:0/992010346 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc4c4066e40 0x7fc4c4069300 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4000f70 tx=0x7fc4d4009040 comp rx=0 tx=0).stop 2024-02-19T15:27:43.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.123+0000 7fc4e9c00640 1 -- 172.21.15.47:0/992010346 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4e406d760 msgr2=0x7fc4e41669e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:43.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.123+0000 7fc4e9c00640 1 --2- 172.21.15.47:0/992010346 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4e406d760 0x7fc4e41669e0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fc4d000c4d0 tx=0x7fc4d000c990 comp rx=0 tx=0).stop 2024-02-19T15:27:43.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.123+0000 7fc4e9c00640 1 -- 172.21.15.47:0/992010346 shutdown_connections 2024-02-19T15:27:43.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.123+0000 7fc4e9c00640 1 --2- 172.21.15.47:0/992010346 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc4c4066e40 0x7fc4c4069300 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.123+0000 7fc4e9c00640 1 --2- 172.21.15.47:0/992010346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e4144b00 0x7fc4e41403e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.123+0000 7fc4e9c00640 1 --2- 172.21.15.47:0/992010346 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4e406d760 0x7fc4e41669e0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.123+0000 7fc4e9c00640 1 --2- 172.21.15.47:0/992010346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4e406b5c0 0x7fc4e40c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.123+0000 7fc4e9c00640 1 -- 172.21.15.47:0/992010346 >> 172.21.15.47:0/992010346 conn(0x7fc4e40d0ad0 msgr2=0x7fc4e40ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:43.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.123+0000 7fc4e9c00640 1 -- 172.21.15.47:0/992010346 shutdown_connections 2024-02-19T15:27:43.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.123+0000 7fc4e9c00640 1 -- 172.21.15.47:0/992010346 wait complete. 2024-02-19T15:27:43.549 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:27:21.415321+0000', 'modified': '2024-02-19T15:27:41.129893+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7920}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7920': {'gid': 7920, 'name': 'a', 'rank': 0, 'incarnation': 79, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.47:6835/3636833601', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6834', 'nonce': 3636833601}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 3636833601}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:27:43.550 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T15:27:43.550 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-19T15:27:43.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.687+0000 7efe86a00640 1 Processor -- start 2024-02-19T15:27:43.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.687+0000 7efe86a00640 1 -- start start 2024-02-19T15:27:43.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.687+0000 7efe86a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe8015bcd0 0x7efe800cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.687+0000 7efe86a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efe800cedb0 0x7efe800cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.687+0000 7efe86a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efe800d2710 0x7efe800d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.687+0000 7efe86a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe800cf5e0 con 0x7efe800cedb0 2024-02-19T15:27:43.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.687+0000 7efe86a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe800cc700 con 0x7efe800d2710 2024-02-19T15:27:43.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.687+0000 7efe86a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe800cf300 con 0x7efe8015bcd0 2024-02-19T15:27:43.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.687+0000 7efe7f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efe800cedb0 0x7efe800cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.687+0000 7efe7f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efe800cedb0 0x7efe800cfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36698/0 (socket says 172.21.15.47:36698) 2024-02-19T15:27:43.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.687+0000 7efe7f400640 1 -- 172.21.15.47:0/3661832131 learned_addr learned my addr 172.21.15.47:0/3661832131 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:43.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.687+0000 7efe84a00640 1 --2- 172.21.15.47:0/3661832131 >> v2:172.21.15.116:3300/0 conn(0x7efe800d2710 0x7efe800d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.687+0000 7efe7fe00640 1 --2- 172.21.15.47:0/3661832131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe8015bcd0 0x7efe800cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.687+0000 7efe7f400640 1 -- 172.21.15.47:0/3661832131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe8015bcd0 msgr2=0x7efe800cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:43.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.687+0000 7efe7f400640 1 --2- 172.21.15.47:0/3661832131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe8015bcd0 0x7efe800cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.687+0000 7efe7f400640 1 -- 172.21.15.47:0/3661832131 >> v2:172.21.15.116:3300/0 conn(0x7efe800d2710 msgr2=0x7efe800d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:43.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.687+0000 7efe7f400640 1 --2- 172.21.15.47:0/3661832131 >> v2:172.21.15.116:3300/0 conn(0x7efe800d2710 0x7efe800d0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.687+0000 7efe7f400640 1 -- 172.21.15.47:0/3661832131 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe800cc9a0 con 0x7efe800cedb0 2024-02-19T15:27:43.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.691+0000 7efe7f400640 1 --2- 172.21.15.47:0/3661832131 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe800cedb0 0x7efe800cfc50 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7efe800ce100 tx=0x7efe6c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d71e31c4a35d0b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:43.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.691+0000 7efe7ea00640 1 -- 172.21.15.47:0/3661832131 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c0aa610 con 0x7efe800cedb0 2024-02-19T15:27:43.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.691+0000 7efe7ea00640 1 -- 172.21.15.47:0/3661832131 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6c0a6040 con 0x7efe800cedb0 2024-02-19T15:27:43.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.691+0000 7efe7ea00640 1 -- 172.21.15.47:0/3661832131 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c0aa990 con 0x7efe800cedb0 2024-02-19T15:27:43.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.691+0000 7efe86a00640 1 -- 172.21.15.47:0/3661832131 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe800cedb0 msgr2=0x7efe800cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:43.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.691+0000 7efe86a00640 1 --2- 172.21.15.47:0/3661832131 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe800cedb0 0x7efe800cfc50 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7efe800ce100 tx=0x7efe6c01c4f0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.691+0000 7efe86a00640 1 -- 172.21.15.47:0/3661832131 shutdown_connections 2024-02-19T15:27:43.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.691+0000 7efe86a00640 1 --2- 172.21.15.47:0/3661832131 >> v2:172.21.15.116:3300/0 conn(0x7efe800d2710 0x7efe800d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.691+0000 7efe86a00640 1 --2- 172.21.15.47:0/3661832131 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe800cedb0 0x7efe800cfc50 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.691+0000 7efe86a00640 1 --2- 172.21.15.47:0/3661832131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe8015bcd0 0x7efe800cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.691+0000 7efe86a00640 1 -- 172.21.15.47:0/3661832131 >> 172.21.15.47:0/3661832131 conn(0x7efe800d1bf0 msgr2=0x7efe800d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:43.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.691+0000 7efe86a00640 1 -- 172.21.15.47:0/3661832131 shutdown_connections 2024-02-19T15:27:43.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.691+0000 7efe86a00640 1 -- 172.21.15.47:0/3661832131 wait complete. 2024-02-19T15:27:43.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe86a00640 1 Processor -- start 2024-02-19T15:27:43.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe86a00640 1 -- start start 2024-02-19T15:27:43.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe86a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe800cedb0 0x7efe8014f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe86a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe800d2710 0x7efe8014e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe86a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe8015bcd0 0x7efe80151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe86a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe800b5fe0 con 0x7efe800d2710 2024-02-19T15:27:43.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe86a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe800b62c0 con 0x7efe800cedb0 2024-02-19T15:27:43.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe86a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe800b65a0 con 0x7efe8015bcd0 2024-02-19T15:27:43.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe7f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe800d2710 0x7efe8014e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe7f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe800d2710 0x7efe8014e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36702/0 (socket says 172.21.15.47:36702) 2024-02-19T15:27:43.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe7f400640 1 -- 172.21.15.47:0/362747770 learned_addr learned my addr 172.21.15.47:0/362747770 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:43.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe7fe00640 1 --2- 172.21.15.47:0/362747770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe800cedb0 0x7efe8014f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe84a00640 1 --2- 172.21.15.47:0/362747770 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe8015bcd0 0x7efe80151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe7f400640 1 -- 172.21.15.47:0/362747770 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe8015bcd0 msgr2=0x7efe80151c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:43.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe7f400640 1 --2- 172.21.15.47:0/362747770 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe8015bcd0 0x7efe80151c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe7f400640 1 -- 172.21.15.47:0/362747770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe800cedb0 msgr2=0x7efe8014f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:43.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe7f400640 1 --2- 172.21.15.47:0/362747770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe800cedb0 0x7efe8014f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe7f400640 1 -- 172.21.15.47:0/362747770 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6c0094d0 con 0x7efe800d2710 2024-02-19T15:27:43.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe7f400640 1 --2- 172.21.15.47:0/362747770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe800d2710 0x7efe8014e600 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7efe6c01aba0 tx=0x7efe6c01ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:43.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe77e00640 1 -- 172.21.15.47:0/362747770 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c0bd5f0 con 0x7efe800d2710 2024-02-19T15:27:43.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe77e00640 1 -- 172.21.15.47:0/362747770 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6c0bd750 con 0x7efe800d2710 2024-02-19T15:27:43.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe77e00640 1 -- 172.21.15.47:0/362747770 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c0bdb00 con 0x7efe800d2710 2024-02-19T15:27:43.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe86a00640 1 -- 172.21.15.47:0/362747770 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe800cdb40 con 0x7efe800d2710 2024-02-19T15:27:43.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.695+0000 7efe86a00640 1 -- 172.21.15.47:0/362747770 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe800b6b60 con 0x7efe800d2710 2024-02-19T15:27:43.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.699+0000 7efe77e00640 1 -- 172.21.15.47:0/362747770 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efe6c0b33f0 con 0x7efe800d2710 2024-02-19T15:27:43.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.699+0000 7efe77e00640 1 --2- 172.21.15.47:0/362747770 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efe60066e40 0x7efe60069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.699+0000 7efe77e00640 1 -- 172.21.15.47:0/362747770 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5881+0+0 (secure 0 0 0) 0x7efe6c11fa70 con 0x7efe800d2710 2024-02-19T15:27:43.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.699+0000 7efe7fe00640 1 --2- 172.21.15.47:0/362747770 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efe60066e40 0x7efe60069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.699+0000 7efe7fe00640 1 --2- 172.21.15.47:0/362747770 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efe60066e40 0x7efe60069300 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7efe800b7bf0 tx=0x7efe70008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:43.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.699+0000 7efe89f2d000 1 Processor -- start 2024-02-19T15:27:43.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.699+0000 7efe89f2d000 1 -- start start 2024-02-19T15:27:43.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe89f2d000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c876920 0x56510c876d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe89f2d000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c877240 0x56510c82c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe89f2d000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c82cf30 0x56510c82d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe89f2d000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510c6f4d00 con 0x56510c876920 2024-02-19T15:27:43.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe89f2d000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510c686430 con 0x56510c82cf30 2024-02-19T15:27:43.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe89f2d000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x56510c683960 con 0x56510c877240 2024-02-19T15:27:43.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe7fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c876920 0x56510c876d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe7fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c876920 0x56510c876d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36710/0 (socket says 172.21.15.47:36710) 2024-02-19T15:27:43.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe7fe00640 1 -- 172.21.15.47:0/1236364925 learned_addr learned my addr 172.21.15.47:0/1236364925 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:43.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe7f400640 1 --2- 172.21.15.47:0/1236364925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c877240 0x56510c82c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe84a00640 1 --2- 172.21.15.47:0/1236364925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c82cf30 0x56510c82d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe7fe00640 1 -- 172.21.15.47:0/1236364925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c877240 msgr2=0x56510c82c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:43.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe7fe00640 1 --2- 172.21.15.47:0/1236364925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c877240 0x56510c82c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe7fe00640 1 -- 172.21.15.47:0/1236364925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c82cf30 msgr2=0x56510c82d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:43.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe7fe00640 1 --2- 172.21.15.47:0/1236364925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c82cf30 0x56510c82d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe7fe00640 1 -- 172.21.15.47:0/1236364925 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56510c375890 con 0x56510c876920 2024-02-19T15:27:43.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe84a00640 1 --2- 172.21.15.47:0/1236364925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c82cf30 0x56510c82d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:27:43.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe7fe00640 1 --2- 172.21.15.47:0/1236364925 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c876920 0x56510c876d00 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7efe70007720 tx=0x7efe70013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b28a757476235581 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:43.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe74c00640 1 -- 172.21.15.47:0/1236364925 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe70016c10 con 0x56510c876920 2024-02-19T15:27:43.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe74c00640 1 -- 172.21.15.47:0/1236364925 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe7001a070 con 0x56510c876920 2024-02-19T15:27:43.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe74c00640 1 -- 172.21.15.47:0/1236364925 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe70021440 con 0x56510c876920 2024-02-19T15:27:43.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe89f2d000 1 -- 172.21.15.47:0/1236364925 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c876920 msgr2=0x56510c876d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:43.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe89f2d000 1 --2- 172.21.15.47:0/1236364925 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c876920 0x56510c876d00 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7efe70007720 tx=0x7efe70013000 comp rx=0 tx=0).stop 2024-02-19T15:27:43.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe89f2d000 1 -- 172.21.15.47:0/1236364925 shutdown_connections 2024-02-19T15:27:43.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe89f2d000 1 --2- 172.21.15.47:0/1236364925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c82cf30 0x56510c82d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe89f2d000 1 --2- 172.21.15.47:0/1236364925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c877240 0x56510c82c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe89f2d000 1 --2- 172.21.15.47:0/1236364925 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c876920 0x56510c876d00 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe89f2d000 1 -- 172.21.15.47:0/1236364925 >> 172.21.15.47:0/1236364925 conn(0x56510c8332a0 msgr2=0x56510c82b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:43.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe89f2d000 1 -- 172.21.15.47:0/1236364925 shutdown_connections 2024-02-19T15:27:43.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.703+0000 7efe89f2d000 1 -- 172.21.15.47:0/1236364925 wait complete. 2024-02-19T15:27:43.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe89f2d000 1 -- start start 2024-02-19T15:27:43.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe89f2d000 1 Processor -- start 2024-02-19T15:27:43.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe89f2d000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c82cf30 0x56510c831910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe89f2d000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c876920 0x56510c877830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe89f2d000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c877240 0x56510c877d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe89f2d000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510c67f140 con 0x56510c877240 2024-02-19T15:27:43.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe89f2d000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510c619630 con 0x56510c876920 2024-02-19T15:27:43.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe89f2d000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x56510c6192d0 con 0x56510c82cf30 2024-02-19T15:27:43.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe89f2d000 10 client.0 _ll_register_callbacks cb 0x56510c72f1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:27:43.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe84a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c877240 0x56510c877d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe7f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c876920 0x56510c877830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe84a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c877240 0x56510c877d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36726/0 (socket says 172.21.15.47:36726) 2024-02-19T15:27:43.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe84a00640 1 -- 172.21.15.47:0/2038412679 learned_addr learned my addr 172.21.15.47:0/2038412679 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:43.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe7fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c82cf30 0x56510c831910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe84a00640 1 -- 172.21.15.47:0/2038412679 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c82cf30 msgr2=0x56510c831910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:43.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe84a00640 1 --2- 172.21.15.47:0/2038412679 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c82cf30 0x56510c831910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe84a00640 1 -- 172.21.15.47:0/2038412679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c876920 msgr2=0x56510c877830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:43.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe84a00640 1 --2- 172.21.15.47:0/2038412679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c876920 0x56510c877830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe84a00640 1 -- 172.21.15.47:0/2038412679 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7000e040 con 0x56510c877240 2024-02-19T15:27:43.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe84a00640 1 --2- 172.21.15.47:0/2038412679 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c877240 0x56510c877d70 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7efe780042f0 tx=0x7efe780047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d225e5f923a64d06 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:43.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe69600640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:27:43.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe69600640 1 -- 172.21.15.47:0/2038412679 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe78009e20 con 0x56510c877240 2024-02-19T15:27:43.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe89f2d000 1 -- 172.21.15.47:0/2038412679 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56510c632850 con 0x56510c877240 2024-02-19T15:27:43.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe69600640 1 -- 172.21.15.47:0/2038412679 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 5881+0+0 (secure 0 0 0) 0x7efe78007b20 con 0x56510c877240 2024-02-19T15:27:43.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe69600640 1 -- 172.21.15.47:0/2038412679 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7efe78007ce0 con 0x56510c877240 2024-02-19T15:27:43.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe89f2d000 10 client.7965 fetch_fsmap learned FSMap version 82 2024-02-19T15:27:43.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe89f2d000 1 -- 172.21.15.47:0/2038412679 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x56510c63ee70 con 0x56510c877240 2024-02-19T15:27:43.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe69600640 1 -- 172.21.15.47:0/2038412679 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56510c632850 con 0x56510c877240 2024-02-19T15:27:43.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe69600640 1 -- 172.21.15.47:0/2038412679 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 82) v1 ==== 3294+0+0 (secure 0 0 0) 0x7efe78016400 con 0x56510c877240 2024-02-19T15:27:43.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe89f2d000 10 client.7965 fetch_fsmap finished waiting for FSMap version 82 2024-02-19T15:27:43.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe89f2d000 10 client.7965 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-19T15:27:43.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe89f2d000 20 client.7965 populate_metadata read hostname 'smithi047' 2024-02-19T15:27:43.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe89f2d000 1 --2- 172.21.15.47:0/2038412679 >> [v2:172.21.15.47:6834/3636833601,v1:172.21.15.47:6835/3636833601] conn(0x56510c8818d0 0x56510c881cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe89f2d000 4 client.7965 mds_command: new command op to 7920 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T15:27:43.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.707+0000 7efe89f2d000 1 -- 172.21.15.47:0/2038412679 --> [v2:172.21.15.47:6834/3636833601,v1:172.21.15.47:6835/3636833601] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56510c67bf20 con 0x56510c8818d0 2024-02-19T15:27:43.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.711+0000 7efe7fe00640 1 --2- 172.21.15.47:0/2038412679 >> [v2:172.21.15.47:6834/3636833601,v1:172.21.15.47:6835/3636833601] conn(0x56510c8818d0 0x56510c881cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.711+0000 7efe7fe00640 1 --2- 172.21.15.47:0/2038412679 >> [v2:172.21.15.47:6834/3636833601,v1:172.21.15.47:6835/3636833601] conn(0x56510c8818d0 0x56510c881cb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a1f3c7d48fc73b2 server_cookie=efaa5dd67547ab7a in_seq=0 out_seq=0 2024-02-19T15:27:43.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.711+0000 7efe69600640 10 client.7965 ms_handle_connect on v2:172.21.15.47:6834/3636833601 2024-02-19T15:27:43.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.711+0000 7efe69600640 1 -- 172.21.15.47:0/2038412679 <== mds.0 v2:172.21.15.47:6834/3636833601 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56510c67bf20 con 0x56510c8818d0 2024-02-19T15:27:43.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.711+0000 7efe69600640 10 client.7965 handle_command_reply: tid=0 2024-02-19T15:27:43.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.711+0000 7efe89f2d000 1 client.7965 shutdown 2024-02-19T15:27:43.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.711+0000 7efe89f2d000 1 -- 172.21.15.47:0/2038412679 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c877240 msgr2=0x56510c877d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:43.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.711+0000 7efe89f2d000 1 --2- 172.21.15.47:0/2038412679 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c877240 0x56510c877d70 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7efe780042f0 tx=0x7efe780047b0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.711+0000 7efe89f2d000 1 -- 172.21.15.47:0/2038412679 shutdown_connections 2024-02-19T15:27:43.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.711+0000 7efe89f2d000 1 --2- 172.21.15.47:0/2038412679 >> [v2:172.21.15.47:6834/3636833601,v1:172.21.15.47:6835/3636833601] conn(0x56510c8818d0 0x56510c881cb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.711+0000 7efe89f2d000 1 --2- 172.21.15.47:0/2038412679 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c877240 0x56510c877d70 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.711+0000 7efe89f2d000 1 --2- 172.21.15.47:0/2038412679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c876920 0x56510c877830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.711+0000 7efe89f2d000 1 --2- 172.21.15.47:0/2038412679 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c82cf30 0x56510c831910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.711+0000 7efe69600640 0 client.7965 ms_handle_reset on v2:172.21.15.47:6834/3636833601 2024-02-19T15:27:43.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.711+0000 7efe89f2d000 1 -- 172.21.15.47:0/2038412679 >> 172.21.15.47:0/2038412679 conn(0x56510c8332a0 msgr2=0x56510c82b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:43.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.711+0000 7efe89f2d000 1 -- 172.21.15.47:0/2038412679 shutdown_connections 2024-02-19T15:27:43.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.711+0000 7efe89f2d000 1 -- 172.21.15.47:0/2038412679 wait complete. 2024-02-19T15:27:43.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.711+0000 7efe89f2d000 20 client.7965 trim_cache size 0 max 16384 2024-02-19T15:27:43.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.727+0000 7efe89f2d000 1 Processor -- start 2024-02-19T15:27:43.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.727+0000 7efe89f2d000 1 -- start start 2024-02-19T15:27:43.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.727+0000 7efe89f2d000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c82cf30 0x56510c830910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.727+0000 7efe89f2d000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c876920 0x56510c830e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.727+0000 7efe89f2d000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c877240 0x56510c883cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.727+0000 7efe89f2d000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510c336940 con 0x56510c877240 2024-02-19T15:27:43.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.727+0000 7efe89f2d000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510c683140 con 0x56510c82cf30 2024-02-19T15:27:43.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.727+0000 7efe89f2d000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x56510c7377d0 con 0x56510c876920 2024-02-19T15:27:43.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe84a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c877240 0x56510c883cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe84a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c877240 0x56510c883cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36738/0 (socket says 172.21.15.47:36738) 2024-02-19T15:27:43.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe84a00640 1 -- 172.21.15.47:0/1702951497 learned_addr learned my addr 172.21.15.47:0/1702951497 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:43.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe7f400640 1 --2- 172.21.15.47:0/1702951497 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c876920 0x56510c830e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe7fe00640 1 --2- 172.21.15.47:0/1702951497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c82cf30 0x56510c830910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe84a00640 1 -- 172.21.15.47:0/1702951497 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c876920 msgr2=0x56510c830e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:43.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe84a00640 1 --2- 172.21.15.47:0/1702951497 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c876920 0x56510c830e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe84a00640 1 -- 172.21.15.47:0/1702951497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c82cf30 msgr2=0x56510c830910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:43.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe84a00640 1 --2- 172.21.15.47:0/1702951497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c82cf30 0x56510c830910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe84a00640 1 -- 172.21.15.47:0/1702951497 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56510c624e10 con 0x56510c877240 2024-02-19T15:27:43.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe7f400640 1 --2- 172.21.15.47:0/1702951497 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c876920 0x56510c830e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:43.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe7fe00640 1 --2- 172.21.15.47:0/1702951497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c82cf30 0x56510c830910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:43.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe84a00640 1 --2- 172.21.15.47:0/1702951497 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c877240 0x56510c883cc0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7efe7800aeb0 tx=0x7efe78010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=758f9ebd79c09c1c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:43.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe6be00640 1 -- 172.21.15.47:0/1702951497 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe78016480 con 0x56510c877240 2024-02-19T15:27:43.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe6be00640 1 -- 172.21.15.47:0/1702951497 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe7800d020 con 0x56510c877240 2024-02-19T15:27:43.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe6be00640 1 -- 172.21.15.47:0/1702951497 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe780090c0 con 0x56510c877240 2024-02-19T15:27:43.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe89f2d000 1 -- 172.21.15.47:0/1702951497 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c877240 msgr2=0x56510c883cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:43.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe89f2d000 1 --2- 172.21.15.47:0/1702951497 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c877240 0x56510c883cc0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7efe7800aeb0 tx=0x7efe78010000 comp rx=0 tx=0).stop 2024-02-19T15:27:43.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe89f2d000 1 -- 172.21.15.47:0/1702951497 shutdown_connections 2024-02-19T15:27:43.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe89f2d000 1 --2- 172.21.15.47:0/1702951497 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c877240 0x56510c883cc0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe89f2d000 1 --2- 172.21.15.47:0/1702951497 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c876920 0x56510c830e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe89f2d000 1 --2- 172.21.15.47:0/1702951497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c82cf30 0x56510c830910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe89f2d000 1 -- 172.21.15.47:0/1702951497 >> 172.21.15.47:0/1702951497 conn(0x56510c8332a0 msgr2=0x56510c832630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:43.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe89f2d000 1 -- 172.21.15.47:0/1702951497 shutdown_connections 2024-02-19T15:27:43.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe89f2d000 1 -- 172.21.15.47:0/1702951497 wait complete. 2024-02-19T15:27:43.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe89f2d000 1 -- start start 2024-02-19T15:27:43.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe89f2d000 1 Processor -- start 2024-02-19T15:27:43.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe89f2d000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c82cf30 0x56510c887040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe89f2d000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c876920 0x56510c887580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe89f2d000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c877240 0x56510c8dccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe89f2d000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510c6442b0 con 0x56510c876920 2024-02-19T15:27:43.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe89f2d000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510c3761b0 con 0x56510c82cf30 2024-02-19T15:27:43.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe89f2d000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x56510c604b40 con 0x56510c877240 2024-02-19T15:27:43.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe89f2d000 10 client.0 _ll_register_callbacks cb 0x7efe78009bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:27:43.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe7f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c876920 0x56510c887580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe7fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c82cf30 0x56510c887040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe7f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c876920 0x56510c887580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36748/0 (socket says 172.21.15.47:36748) 2024-02-19T15:27:43.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.731+0000 7efe7f400640 1 -- 172.21.15.47:0/2280602973 learned_addr learned my addr 172.21.15.47:0/2280602973 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:43.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe84a00640 1 --2- 172.21.15.47:0/2280602973 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c877240 0x56510c8dccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe7f400640 1 -- 172.21.15.47:0/2280602973 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c877240 msgr2=0x56510c8dccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:43.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe7f400640 1 --2- 172.21.15.47:0/2280602973 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c877240 0x56510c8dccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe7f400640 1 -- 172.21.15.47:0/2280602973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c82cf30 msgr2=0x56510c887040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:43.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe7f400640 1 --2- 172.21.15.47:0/2280602973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c82cf30 0x56510c887040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe7f400640 1 -- 172.21.15.47:0/2280602973 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56510c63ee70 con 0x56510c876920 2024-02-19T15:27:43.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe7fe00640 1 --2- 172.21.15.47:0/2280602973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c82cf30 0x56510c887040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:27:43.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe7f400640 1 --2- 172.21.15.47:0/2280602973 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c876920 0x56510c887580 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7efe6c0f02a0 tx=0x7efe6c0f0010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9318bd37adfc7af8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:43.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe69600640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:27:43.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe69600640 1 -- 172.21.15.47:0/2280602973 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c0ee960 con 0x56510c876920 2024-02-19T15:27:43.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe69600640 1 -- 172.21.15.47:0/2280602973 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 5881+0+0 (secure 0 0 0) 0x7efe6c0ef9f0 con 0x56510c876920 2024-02-19T15:27:43.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 1 -- 172.21.15.47:0/2280602973 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56510c604e80 con 0x56510c876920 2024-02-19T15:27:43.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe69600640 1 -- 172.21.15.47:0/2280602973 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7efe6c0efbb0 con 0x56510c876920 2024-02-19T15:27:43.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 10 client.7971 fetch_fsmap learned FSMap version 82 2024-02-19T15:27:43.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 1 -- 172.21.15.47:0/2280602973 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x56510c375cc0 con 0x56510c876920 2024-02-19T15:27:43.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe69600640 1 -- 172.21.15.47:0/2280602973 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56510c604e80 con 0x56510c876920 2024-02-19T15:27:43.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe69600640 1 -- 172.21.15.47:0/2280602973 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 82) v1 ==== 3294+0+0 (secure 0 0 0) 0x7efe6c0ec2f0 con 0x56510c876920 2024-02-19T15:27:43.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 10 client.7971 fetch_fsmap finished waiting for FSMap version 82 2024-02-19T15:27:43.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 10 client.7971 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-19T15:27:43.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 20 client.7971 populate_metadata read hostname 'smithi047' 2024-02-19T15:27:43.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 1 --2- 172.21.15.47:0/2280602973 >> [v2:172.21.15.47:6834/3636833601,v1:172.21.15.47:6835/3636833601] conn(0x56510c8dd230 0x56510c8dd610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 4 client.7971 mds_command: new command op to 7920 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T15:27:43.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 1 -- 172.21.15.47:0/2280602973 --> [v2:172.21.15.47:6834/3636833601,v1:172.21.15.47:6835/3636833601] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56510c647fa0 con 0x56510c8dd230 2024-02-19T15:27:43.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe7fe00640 1 --2- 172.21.15.47:0/2280602973 >> [v2:172.21.15.47:6834/3636833601,v1:172.21.15.47:6835/3636833601] conn(0x56510c8dd230 0x56510c8dd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe7fe00640 1 --2- 172.21.15.47:0/2280602973 >> [v2:172.21.15.47:6834/3636833601,v1:172.21.15.47:6835/3636833601] conn(0x56510c8dd230 0x56510c8dd610 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f9bedd9cdcbadde server_cookie=6bd50f14b24f9047 in_seq=0 out_seq=0 2024-02-19T15:27:43.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe69600640 10 client.7971 ms_handle_connect on v2:172.21.15.47:6834/3636833601 2024-02-19T15:27:43.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe69600640 1 -- 172.21.15.47:0/2280602973 <== mds.0 v2:172.21.15.47:6834/3636833601 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x56510c647fa0 con 0x56510c8dd230 2024-02-19T15:27:43.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe69600640 10 client.7971 handle_command_reply: tid=0 2024-02-19T15:27:43.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 1 client.7971 shutdown 2024-02-19T15:27:43.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 1 -- 172.21.15.47:0/2280602973 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c876920 msgr2=0x56510c887580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:43.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 1 --2- 172.21.15.47:0/2280602973 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c876920 0x56510c887580 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7efe6c0f02a0 tx=0x7efe6c0f0010 comp rx=0 tx=0).stop 2024-02-19T15:27:43.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 1 -- 172.21.15.47:0/2280602973 shutdown_connections 2024-02-19T15:27:43.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 1 --2- 172.21.15.47:0/2280602973 >> [v2:172.21.15.47:6834/3636833601,v1:172.21.15.47:6835/3636833601] conn(0x56510c8dd230 0x56510c8dd610 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 1 --2- 172.21.15.47:0/2280602973 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56510c877240 0x56510c8dccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 1 --2- 172.21.15.47:0/2280602973 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56510c876920 0x56510c887580 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 1 --2- 172.21.15.47:0/2280602973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56510c82cf30 0x56510c887040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe69600640 0 client.7971 ms_handle_reset on v2:172.21.15.47:6834/3636833601 2024-02-19T15:27:43.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 1 -- 172.21.15.47:0/2280602973 >> 172.21.15.47:0/2280602973 conn(0x56510c8332a0 msgr2=0x56510c82f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:43.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 1 -- 172.21.15.47:0/2280602973 shutdown_connections 2024-02-19T15:27:43.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 1 -- 172.21.15.47:0/2280602973 wait complete. 2024-02-19T15:27:43.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe89f2d000 20 client.7971 trim_cache size 0 max 16384 2024-02-19T15:27:43.745 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:27:43.745 INFO:teuthology.orchestra.run.smithi047.stdout: "cluster_fsid": "ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e", 2024-02-19T15:27:43.745 INFO:teuthology.orchestra.run.smithi047.stdout: "whoami": 0, 2024-02-19T15:27:43.745 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 7920, 2024-02-19T15:27:43.745 INFO:teuthology.orchestra.run.smithi047.stdout: "want_state": "up:active", 2024-02-19T15:27:43.745 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "up:active", 2024-02-19T15:27:43.745 INFO:teuthology.orchestra.run.smithi047.stdout: "fs_name": "cephfs", 2024-02-19T15:27:43.745 INFO:teuthology.orchestra.run.smithi047.stdout: "rank_uptime": 4.6239773050000004, 2024-02-19T15:27:43.746 INFO:teuthology.orchestra.run.smithi047.stdout: "mdsmap_epoch": 82, 2024-02-19T15:27:43.746 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch": 112, 2024-02-19T15:27:43.746 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch_barrier": 112, 2024-02-19T15:27:43.746 INFO:teuthology.orchestra.run.smithi047.stdout: "uptime": 5.3074559419999998 2024-02-19T15:27:43.746 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:27:43.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe86a00640 1 -- 172.21.15.47:0/362747770 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efe60066e40 msgr2=0x7efe60069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:43.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe86a00640 1 --2- 172.21.15.47:0/362747770 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efe60066e40 0x7efe60069300 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7efe800b7bf0 tx=0x7efe70008040 comp rx=0 tx=0).stop 2024-02-19T15:27:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe86a00640 1 -- 172.21.15.47:0/362747770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe800d2710 msgr2=0x7efe8014e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe86a00640 1 --2- 172.21.15.47:0/362747770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe800d2710 0x7efe8014e600 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7efe6c01aba0 tx=0x7efe6c01ad30 comp rx=0 tx=0).stop 2024-02-19T15:27:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe86a00640 1 -- 172.21.15.47:0/362747770 shutdown_connections 2024-02-19T15:27:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe86a00640 1 --2- 172.21.15.47:0/362747770 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efe60066e40 0x7efe60069300 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe86a00640 1 --2- 172.21.15.47:0/362747770 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe8015bcd0 0x7efe80151c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe86a00640 1 --2- 172.21.15.47:0/362747770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe800d2710 0x7efe8014e600 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe86a00640 1 --2- 172.21.15.47:0/362747770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe800cedb0 0x7efe8014f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe86a00640 1 -- 172.21.15.47:0/362747770 >> 172.21.15.47:0/362747770 conn(0x7efe800d1bf0 msgr2=0x7efe800b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe86a00640 1 -- 172.21.15.47:0/362747770 shutdown_connections 2024-02-19T15:27:43.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.735+0000 7efe86a00640 1 -- 172.21.15.47:0/362747770 wait complete. 2024-02-19T15:27:43.760 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-02-19T15:27:43.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.895+0000 7f8bba800640 1 Processor -- start 2024-02-19T15:27:43.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.895+0000 7f8bba800640 1 -- start start 2024-02-19T15:27:43.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.895+0000 7f8bba800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8bb415bcd0 0x7f8bb40cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.895+0000 7f8bba800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb40cc700 con 0x7f8bb40d0360 2024-02-19T15:27:43.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.895+0000 7f8bba800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb40cf300 con 0x7f8bb40cedb0 2024-02-19T15:27:43.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.895+0000 7f8bba800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bb40cf5e0 con 0x7f8bb415bcd0 2024-02-19T15:27:43.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.899+0000 7f8bb3400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8bb415bcd0 0x7f8bb40cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.899+0000 7f8bb3e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8bb40d0360 0x7f8bb415d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:48186/0 (socket says 172.21.15.47:48186) 2024-02-19T15:27:43.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.899+0000 7f8bb3e00640 1 -- 172.21.15.47:0/1980767351 learned_addr learned my addr 172.21.15.47:0/1980767351 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:43.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.899+0000 7f8bb2a00640 1 -- 172.21.15.47:0/1980767351 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463791989 0 0) 0x7f8bb40cc700 con 0x7f8bb40d0360 2024-02-19T15:27:43.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.899+0000 7f8bb2a00640 1 -- 172.21.15.47:0/1980767351 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b98003530 con 0x7f8bb40d0360 2024-02-19T15:27:43.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.899+0000 7f8bb3400640 1 -- 172.21.15.47:0/1980767351 >> v1:172.21.15.116:6789/0 conn(0x7f8bb40cedb0 legacy=0x7f8bb40cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:43.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.899+0000 7f8bb3400640 1 -- 172.21.15.47:0/1980767351 >> v1:172.21.15.47:6789/0 conn(0x7f8bb40d0360 legacy=0x7f8bb415d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:43.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.899+0000 7f8bb3400640 1 -- 172.21.15.47:0/1980767351 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb40cc9a0 con 0x7f8bb415bcd0 2024-02-19T15:27:43.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.899+0000 7f8bb3400640 1 --2- 172.21.15.47:0/1980767351 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bb415bcd0 0x7f8bb40cd480 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40097c0 tx=0x7f8ba401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b0c87c9fb1d64da server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:43.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.899+0000 7f8bb2a00640 1 -- 172.21.15.47:0/1980767351 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba40ab670 con 0x7f8bb415bcd0 2024-02-19T15:27:43.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.899+0000 7f8bb2a00640 1 -- 172.21.15.47:0/1980767351 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba40ab7d0 con 0x7f8bb415bcd0 2024-02-19T15:27:43.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.899+0000 7f8bb2a00640 1 -- 172.21.15.47:0/1980767351 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba40abb40 con 0x7f8bb415bcd0 2024-02-19T15:27:43.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8bba800640 1 -- 172.21.15.47:0/1980767351 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bb415bcd0 msgr2=0x7f8bb40cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:43.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8bba800640 1 --2- 172.21.15.47:0/1980767351 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bb415bcd0 0x7f8bb40cd480 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40097c0 tx=0x7f8ba401c470 comp rx=0 tx=0).stop 2024-02-19T15:27:43.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8bba800640 1 -- 172.21.15.47:0/1980767351 shutdown_connections 2024-02-19T15:27:43.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8bba800640 1 --2- 172.21.15.47:0/1980767351 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bb415bcd0 0x7f8bb40cd480 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8bba800640 1 -- 172.21.15.47:0/1980767351 >> 172.21.15.47:0/1980767351 conn(0x7f8bb40d1bf0 msgr2=0x7f8bb40d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:43.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8bba800640 1 -- 172.21.15.47:0/1980767351 shutdown_connections 2024-02-19T15:27:43.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8bba800640 1 -- 172.21.15.47:0/1980767351 wait complete. 2024-02-19T15:27:43.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8bba800640 1 Processor -- start 2024-02-19T15:27:43.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8bba800640 1 -- start start 2024-02-19T15:27:43.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8bba800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb40cedb0 0x7f8bb41577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8bba800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bb40d0360 0x7f8bb4158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8bba800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8bb415bcd0 0x7f8bb415a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8bba800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb40c8cc0 con 0x7f8bb40d0360 2024-02-19T15:27:43.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8bba800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb40c8f10 con 0x7f8bb40cedb0 2024-02-19T15:27:43.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8bba800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb40c91f0 con 0x7f8bb415bcd0 2024-02-19T15:27:43.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8babe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bb40d0360 0x7f8bb4158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8babe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bb40d0360 0x7f8bb4158070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36752/0 (socket says 172.21.15.47:36752) 2024-02-19T15:27:43.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8babe00640 1 -- 172.21.15.47:0/1910625331 learned_addr learned my addr 172.21.15.47:0/1910625331 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:43.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8bb3400640 1 --2- 172.21.15.47:0/1910625331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb40cedb0 0x7f8bb41577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8bb3e00640 1 --2- 172.21.15.47:0/1910625331 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8bb415bcd0 0x7f8bb415a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.903+0000 7f8babe00640 1 -- 172.21.15.47:0/1910625331 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8bb415bcd0 msgr2=0x7f8bb415a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:43.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.907+0000 7f8babe00640 1 --2- 172.21.15.47:0/1910625331 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8bb415bcd0 0x7f8bb415a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.907+0000 7f8babe00640 1 -- 172.21.15.47:0/1910625331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb40cedb0 msgr2=0x7f8bb41577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:43.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.907+0000 7f8babe00640 1 --2- 172.21.15.47:0/1910625331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb40cedb0 0x7f8bb41577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:43.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.907+0000 7f8babe00640 1 -- 172.21.15.47:0/1910625331 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba4009420 con 0x7f8bb40d0360 2024-02-19T15:27:43.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.907+0000 7f8bb3e00640 1 --2- 172.21.15.47:0/1910625331 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8bb415bcd0 0x7f8bb415a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:27:43.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.907+0000 7f8babe00640 1 --2- 172.21.15.47:0/1910625331 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bb40d0360 0x7f8bb4158070 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c00c3c0 tx=0x7f8b9c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:43.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.907+0000 7f8bab400640 1 -- 172.21.15.47:0/1910625331 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c0052d0 con 0x7f8bb40d0360 2024-02-19T15:27:43.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.907+0000 7f8bab400640 1 -- 172.21.15.47:0/1910625331 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9c005430 con 0x7f8bb40d0360 2024-02-19T15:27:43.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.907+0000 7f8bba800640 1 -- 172.21.15.47:0/1910625331 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb40cdb40 con 0x7f8bb40d0360 2024-02-19T15:27:43.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.907+0000 7f8bab400640 1 -- 172.21.15.47:0/1910625331 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c003840 con 0x7f8bb40d0360 2024-02-19T15:27:43.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.907+0000 7f8bba800640 1 -- 172.21.15.47:0/1910625331 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb40c97b0 con 0x7f8bb40d0360 2024-02-19T15:27:43.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.907+0000 7f8bba800640 1 -- 172.21.15.47:0/1910625331 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bb40c99d0 con 0x7f8bb40d0360 2024-02-19T15:27:43.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.907+0000 7f8bab400640 1 -- 172.21.15.47:0/1910625331 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8b9c0039a0 con 0x7f8bb40d0360 2024-02-19T15:27:43.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.907+0000 7f8bab400640 1 --2- 172.21.15.47:0/1910625331 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8b90066e70 0x7f8b90069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:43.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.907+0000 7f8bab400640 1 -- 172.21.15.47:0/1910625331 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f8b9c083ff0 con 0x7f8bb40d0360 2024-02-19T15:27:43.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.907+0000 7f8bb3400640 1 --2- 172.21.15.47:0/1910625331 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8b90066e70 0x7f8b90069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:43.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.911+0000 7f8bb3400640 1 --2- 172.21.15.47:0/1910625331 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8b90066e70 0x7f8b90069330 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40af000 tx=0x7f8ba40a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:43.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:43.911+0000 7f8bab400640 1 -- 172.21.15.47:0/1910625331 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8b9c0b61c0 con 0x7f8bb40d0360 2024-02-19T15:27:44.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:44.111+0000 7f8bba800640 1 -- 172.21.15.47:0/1910625331 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8bb4075360 con 0x7f8bb40d0360 2024-02-19T15:27:44.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:44.111+0000 7f8bab400640 1 -- 172.21.15.47:0/1910625331 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v82) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8b9c017060 con 0x7f8bb40d0360 2024-02-19T15:27:44.116 INFO:teuthology.orchestra.run.smithi047.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-02-19T15:27:44.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:44.115+0000 7f8bba800640 1 -- 172.21.15.47:0/1910625331 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8b90066e70 msgr2=0x7f8b90069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:44.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:44.115+0000 7f8bba800640 1 --2- 172.21.15.47:0/1910625331 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8b90066e70 0x7f8b90069330 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40af000 tx=0x7f8ba40a7000 comp rx=0 tx=0).stop 2024-02-19T15:27:44.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:44.115+0000 7f8bba800640 1 -- 172.21.15.47:0/1910625331 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bb40d0360 msgr2=0x7f8bb4158070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:44.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:44.115+0000 7f8bba800640 1 --2- 172.21.15.47:0/1910625331 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bb40d0360 0x7f8bb4158070 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c00c3c0 tx=0x7f8b9c00c880 comp rx=0 tx=0).stop 2024-02-19T15:27:44.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:44.115+0000 7f8bba800640 1 -- 172.21.15.47:0/1910625331 shutdown_connections 2024-02-19T15:27:44.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:44.115+0000 7f8bba800640 1 --2- 172.21.15.47:0/1910625331 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8b90066e70 0x7f8b90069330 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:44.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:44.115+0000 7f8bba800640 1 --2- 172.21.15.47:0/1910625331 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8bb415bcd0 0x7f8bb415a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:44.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:44.115+0000 7f8bba800640 1 --2- 172.21.15.47:0/1910625331 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bb40d0360 0x7f8bb4158070 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:44.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:44.115+0000 7f8bba800640 1 --2- 172.21.15.47:0/1910625331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb40cedb0 0x7f8bb41577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:44.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:44.115+0000 7f8bba800640 1 -- 172.21.15.47:0/1910625331 >> 172.21.15.47:0/1910625331 conn(0x7f8bb40d1bf0 msgr2=0x7f8bb40703c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:44.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:44.115+0000 7f8bba800640 1 -- 172.21.15.47:0/1910625331 shutdown_connections 2024-02-19T15:27:44.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:44.115+0000 7f8bba800640 1 -- 172.21.15.47:0/1910625331 wait complete. 2024-02-19T15:27:44.139 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-02-19T15:27:44.139 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T15:27:44.139 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi047.front.sepia.ceph.com 2024-02-19T15:27:44.139 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-19T15:27:44.139 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-19T15:27:44.139 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-02-19T15:27:44.139 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-02-19T15:27:44.139 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T15:27:44.139 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:27:44.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:44.139 DEBUG:teuthology.orchestra.run.smithi047:> ip addr 2024-02-19T15:27:44.149 INFO:teuthology.orchestra.run.smithi047.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T15:27:44.149 INFO:teuthology.orchestra.run.smithi047.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T15:27:44.149 INFO:teuthology.orchestra.run.smithi047.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T15:27:44.149 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:27:44.149 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 ::1/128 scope host 2024-02-19T15:27:44.149 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:27:44.149 INFO:teuthology.orchestra.run.smithi047.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:27:44.149 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:78 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:27:44.149 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f0 2024-02-19T15:27:44.149 INFO:teuthology.orchestra.run.smithi047.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T15:27:44.149 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8e brd ff:ff:ff:ff:ff:ff 2024-02-19T15:27:44.149 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f0 2024-02-19T15:27:44.150 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.21.15.47/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-02-19T15:27:44.150 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft 42151sec preferred_lft 42151sec 2024-02-19T15:27:44.150 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::ec4:7aff:febd:148e/64 scope link 2024-02-19T15:27:44.150 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:27:44.150 INFO:teuthology.orchestra.run.smithi047.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:27:44.150 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:79 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:27:44.150 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f1 2024-02-19T15:27:44.150 INFO:teuthology.orchestra.run.smithi047.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:27:44.150 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8f brd ff:ff:ff:ff:ff:ff 2024-02-19T15:27:44.150 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f1 2024-02-19T15:27:44.150 INFO:teuthology.orchestra.run.smithi047.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-02-19T15:27:44.150 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 02:42:47:cf:b1:62 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:27:44.150 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-02-19T15:27:44.150 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:27:44.150 INFO:teuthology.orchestra.run.smithi047.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-02-19T15:27:44.151 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-02-19T15:27:44.151 INFO:teuthology.orchestra.run.smithi047.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-02-19T15:27:44.151 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:27:44.151 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-02-19T15:27:44.151 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:27:44.151 INFO:teuthology.orchestra.run.smithi047.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:27:44.151 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-19T15:27:44.151 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-02-19T15:27:44.151 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:27:44.151 INFO:teuthology.orchestra.run.smithi047.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:27:44.151 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-02-19T15:27:44.151 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-02-19T15:27:44.151 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:27:44.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:44.152 DEBUG:teuthology.orchestra.run.smithi047:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T15:27:44.211 INFO:teuthology.orchestra.run.smithi047.stdout:1 2024-02-19T15:27:44.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:44.213 DEBUG:teuthology.orchestra.run.smithi047:> route 2024-02-19T15:27:44.266 INFO:teuthology.orchestra.run.smithi047.stdout:Kernel IP routing table 2024-02-19T15:27:44.266 INFO:teuthology.orchestra.run.smithi047.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T15:27:44.266 INFO:teuthology.orchestra.run.smithi047.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-02-19T15:27:44.266 INFO:teuthology.orchestra.run.smithi047.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-02-19T15:27:44.266 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-02-19T15:27:44.266 INFO:teuthology.orchestra.run.smithi047.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T15:27:44.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:44.266 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2024-02-19T15:27:44.267 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2024-02-19T15:27:44.267 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-02-19T15:27:44.267 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-02-19T15:27:44.267 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-02-19T15:27:44.267 DEBUG:teuthology.orchestra.run.smithi047:> ') 2024-02-19T15:27:44.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:44.359 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list 2024-02-19T15:27:44.368 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-02-19T15:27:44.368 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-02-19T15:27:44.369 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:27:44.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:44.369 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set brx.0 up 2024-02-19T15:27:44.428 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:27:44.429 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-19T15:27:44.481 INFO:teuthology.orchestra.run.smithi047.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:27:44.481 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:27:44.481 DEBUG:teuthology.orchestra.run.smithi047:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:27:44.532 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-19T15:27:44.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:44.533 DEBUG:teuthology.orchestra.run.smithi047:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:27:44.621 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:27:44.621 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:27:44.621 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:27:44.636 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:27:44.636 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:27:44.636 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:27:44.636 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:27:44.636 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:27:44.645 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:27:44.645 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:27:44.646 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:27:44.646 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:27:44.646 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:27:44.646 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790" 2024-02-19T15:27:44.646 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790:/ 2024-02-19T15:27:44.646 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e" will pass to kernel 2024-02-19T15:27:44.651 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T15:27:44.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:44.651 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:27:44.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:44.670 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:27:44.728 INFO:teuthology.orchestra.run.smithi047.stdout:client7983 2024-02-19T15:27:44.828 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:27:44.828 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:27:44.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:27:44.837 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_1/file_a) 2024-02-19T15:27:44.892 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:27:45.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.087+0000 7ff131600640 1 Processor -- start 2024-02-19T15:27:45.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.087+0000 7ff131600640 1 -- start start 2024-02-19T15:27:45.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.091+0000 7ff131600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff12c145590 0x7ff12c0727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:45.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.091+0000 7ff131600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff12c15c390 0x7ff12c15c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:45.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.091+0000 7ff131600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff12c15d280 con 0x7ff12c0cea80 2024-02-19T15:27:45.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.091+0000 7ff131600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff12c070ff0 con 0x7ff12c145590 2024-02-19T15:27:45.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.091+0000 7ff131600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff12c15cd40 con 0x7ff12c15c390 2024-02-19T15:27:45.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.091+0000 7ff12aa00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff12c145590 0x7ff12c0727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:45.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.091+0000 7ff12aa00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff12c145590 0x7ff12c0727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36764/0 (socket says 172.21.15.47:36764) 2024-02-19T15:27:45.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.091+0000 7ff12aa00640 1 -- 172.21.15.47:0/3864448478 learned_addr learned my addr 172.21.15.47:0/3864448478 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:45.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.091+0000 7ff12b400640 1 --2- 172.21.15.47:0/3864448478 >> v2:172.21.15.116:3300/0 conn(0x7ff12c15c390 0x7ff12c15c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:45.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.091+0000 7ff12aa00640 1 -- 172.21.15.47:0/3864448478 >> v2:172.21.15.116:3300/0 conn(0x7ff12c15c390 msgr2=0x7ff12c15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:45.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.091+0000 7ff12aa00640 1 --2- 172.21.15.47:0/3864448478 >> v2:172.21.15.116:3300/0 conn(0x7ff12c15c390 0x7ff12c15c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:45.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.091+0000 7ff12aa00640 1 -- 172.21.15.47:0/3864448478 >> v1:172.21.15.116:6789/0 conn(0x7ff12c0cea80 legacy=0x7ff12c06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:45.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.091+0000 7ff12a000640 1 -- 172.21.15.47:0/3864448478 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757211200 0 0) 0x7ff12c15d280 con 0x7ff12c0cea80 2024-02-19T15:27:45.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.091+0000 7ff12aa00640 1 -- 172.21.15.47:0/3864448478 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff12c0709e0 con 0x7ff12c145590 2024-02-19T15:27:45.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.095+0000 7ff12aa00640 1 --2- 172.21.15.47:0/3864448478 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff12c145590 0x7ff12c0727b0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff12c072380 tx=0x7ff11c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c822d5cdada1ff7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:45.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.095+0000 7ff12a000640 1 -- 172.21.15.47:0/3864448478 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11c0a57a0 con 0x7ff12c145590 2024-02-19T15:27:45.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.095+0000 7ff12a000640 1 -- 172.21.15.47:0/3864448478 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff11c0a5900 con 0x7ff12c145590 2024-02-19T15:27:45.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.095+0000 7ff12a000640 1 -- 172.21.15.47:0/3864448478 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11c0a5c90 con 0x7ff12c145590 2024-02-19T15:27:45.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.095+0000 7ff131600640 1 -- 172.21.15.47:0/3864448478 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff12c145590 msgr2=0x7ff12c0727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:45.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.095+0000 7ff131600640 1 --2- 172.21.15.47:0/3864448478 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff12c145590 0x7ff12c0727b0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff12c072380 tx=0x7ff11c01c9f0 comp rx=0 tx=0).stop 2024-02-19T15:27:45.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.095+0000 7ff131600640 1 -- 172.21.15.47:0/3864448478 shutdown_connections 2024-02-19T15:27:45.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.095+0000 7ff131600640 1 --2- 172.21.15.47:0/3864448478 >> v2:172.21.15.116:3300/0 conn(0x7ff12c15c390 0x7ff12c15c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:45.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.095+0000 7ff131600640 1 --2- 172.21.15.47:0/3864448478 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff12c145590 0x7ff12c0727b0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:45.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.095+0000 7ff131600640 1 -- 172.21.15.47:0/3864448478 >> 172.21.15.47:0/3864448478 conn(0x7ff12c0b16a0 msgr2=0x7ff12c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:45.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.095+0000 7ff131600640 1 -- 172.21.15.47:0/3864448478 shutdown_connections 2024-02-19T15:27:45.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.095+0000 7ff131600640 1 -- 172.21.15.47:0/3864448478 wait complete. 2024-02-19T15:27:45.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.095+0000 7ff131600640 1 Processor -- start 2024-02-19T15:27:45.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.095+0000 7ff131600640 1 -- start start 2024-02-19T15:27:45.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff131600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff12c0cea80 0x7ff12c13a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:45.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff131600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff12c145590 0x7ff12c15afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:45.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff131600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff12c15c390 0x7ff12c15b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:45.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff131600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12c15ba20 con 0x7ff12c145590 2024-02-19T15:27:45.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff131600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12c13b580 con 0x7ff12c15c390 2024-02-19T15:27:45.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff131600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff12c13ace0 con 0x7ff12c0cea80 2024-02-19T15:27:45.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff122000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff12c145590 0x7ff12c15afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:45.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff122000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff12c145590 0x7ff12c15afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36778/0 (socket says 172.21.15.47:36778) 2024-02-19T15:27:45.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff122000640 1 -- 172.21.15.47:0/3185972606 learned_addr learned my addr 172.21.15.47:0/3185972606 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:45.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff12aa00640 1 --2- 172.21.15.47:0/3185972606 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff12c0cea80 0x7ff12c13a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:45.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff12b400640 1 --2- 172.21.15.47:0/3185972606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff12c15c390 0x7ff12c15b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:45.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff122000640 1 -- 172.21.15.47:0/3185972606 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff12c0cea80 msgr2=0x7ff12c13a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:45.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff122000640 1 --2- 172.21.15.47:0/3185972606 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff12c0cea80 0x7ff12c13a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:45.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff122000640 1 -- 172.21.15.47:0/3185972606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff12c15c390 msgr2=0x7ff12c15b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:45.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff122000640 1 --2- 172.21.15.47:0/3185972606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff12c15c390 0x7ff12c15b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:45.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff122000640 1 -- 172.21.15.47:0/3185972606 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11c009b30 con 0x7ff12c145590 2024-02-19T15:27:45.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff122000640 1 --2- 172.21.15.47:0/3185972606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff12c145590 0x7ff12c15afa0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff11400c480 tx=0x7ff11400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:45.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff123400640 1 -- 172.21.15.47:0/3185972606 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff114005480 con 0x7ff12c145590 2024-02-19T15:27:45.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff123400640 1 -- 172.21.15.47:0/3185972606 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff114003600 con 0x7ff12c145590 2024-02-19T15:27:45.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff123400640 1 -- 172.21.15.47:0/3185972606 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1140039b0 con 0x7ff12c145590 2024-02-19T15:27:45.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff131600640 1 -- 172.21.15.47:0/3185972606 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff12c13afc0 con 0x7ff12c145590 2024-02-19T15:27:45.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff131600640 1 -- 172.21.15.47:0/3185972606 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff12c0b28e0 con 0x7ff12c145590 2024-02-19T15:27:45.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff131600640 1 -- 172.21.15.47:0/3185972606 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff12c15cb40 con 0x7ff12c145590 2024-02-19T15:27:45.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.099+0000 7ff123400640 1 -- 172.21.15.47:0/3185972606 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff114003b10 con 0x7ff12c145590 2024-02-19T15:27:45.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.103+0000 7ff123400640 1 --2- 172.21.15.47:0/3185972606 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff108066e40 0x7ff108069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:45.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.103+0000 7ff123400640 1 -- 172.21.15.47:0/3185972606 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5881+0+0 (secure 0 0 0) 0x7ff114084620 con 0x7ff12c145590 2024-02-19T15:27:45.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.103+0000 7ff12aa00640 1 --2- 172.21.15.47:0/3185972606 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff108066e40 0x7ff108069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:45.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.103+0000 7ff12aa00640 1 --2- 172.21.15.47:0/3185972606 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff108066e40 0x7ff108069300 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff11c0ae000 tx=0x7ff11c0a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:45.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.107+0000 7ff123400640 1 -- 172.21.15.47:0/3185972606 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff114051350 con 0x7ff12c145590 2024-02-19T15:27:45.115 INFO:tasks.ceph.mds.a.smithi047.stderr:2024-02-19T15:27:45.111+0000 7fbbfd200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T15:27:45.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.331+0000 7ff131600640 1 -- 172.21.15.47:0/3185972606 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff12c075360 con 0x7ff12c145590 2024-02-19T15:27:45.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.331+0000 7ff123400640 1 -- 172.21.15.47:0/3185972606 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7ff114054a10 con 0x7ff12c145590 2024-02-19T15:27:45.335 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:45.335 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:27:45.335 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:27:45.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.331+0000 7ff131600640 1 -- 172.21.15.47:0/3185972606 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff108066e40 msgr2=0x7ff108069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:45.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.331+0000 7ff131600640 1 --2- 172.21.15.47:0/3185972606 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff108066e40 0x7ff108069300 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff11c0ae000 tx=0x7ff11c0a7070 comp rx=0 tx=0).stop 2024-02-19T15:27:45.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.331+0000 7ff131600640 1 -- 172.21.15.47:0/3185972606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff12c145590 msgr2=0x7ff12c15afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:45.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.331+0000 7ff131600640 1 --2- 172.21.15.47:0/3185972606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff12c145590 0x7ff12c15afa0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff11400c480 tx=0x7ff11400c940 comp rx=0 tx=0).stop 2024-02-19T15:27:45.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.331+0000 7ff131600640 1 -- 172.21.15.47:0/3185972606 shutdown_connections 2024-02-19T15:27:45.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.331+0000 7ff131600640 1 --2- 172.21.15.47:0/3185972606 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff108066e40 0x7ff108069300 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:45.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.331+0000 7ff131600640 1 --2- 172.21.15.47:0/3185972606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff12c15c390 0x7ff12c15b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:45.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.331+0000 7ff131600640 1 --2- 172.21.15.47:0/3185972606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff12c145590 0x7ff12c15afa0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:45.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.331+0000 7ff131600640 1 --2- 172.21.15.47:0/3185972606 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff12c0cea80 0x7ff12c13a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:45.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.331+0000 7ff131600640 1 -- 172.21.15.47:0/3185972606 >> 172.21.15.47:0/3185972606 conn(0x7ff12c0b16a0 msgr2=0x7ff12c06d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:45.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.331+0000 7ff131600640 1 -- 172.21.15.47:0/3185972606 shutdown_connections 2024-02-19T15:27:45.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.331+0000 7ff131600640 1 -- 172.21.15.47:0/3185972606 wait complete. 2024-02-19T15:27:45.357 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-02-19T15:27:45.597 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"num_strays":1}} 2024-02-19T15:27:45.597 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-02-19T15:27:45.598 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:27:45.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.735+0000 7f3562800640 1 Processor -- start 2024-02-19T15:27:45.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.735+0000 7f3562800640 1 -- start start 2024-02-19T15:27:45.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.735+0000 7f3562800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f355c0cded0 0x7f355c0ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:45.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.735+0000 7f3562800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f355c0ceb40 0x7f355c06bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:45.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.735+0000 7f3562800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355c0cce30 con 0x7f355c06d650 2024-02-19T15:27:45.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.735+0000 7f3562800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f355c0cefd0 con 0x7f355c0cded0 2024-02-19T15:27:45.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.735+0000 7f3562800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355c0cd110 con 0x7f355c0ceb40 2024-02-19T15:27:45.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.735+0000 7f355b400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f355c06d650 0x7f355c0d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:48188/0 (socket says 172.21.15.47:48188) 2024-02-19T15:27:45.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.735+0000 7f355b400640 1 -- 172.21.15.47:0/3598403398 learned_addr learned my addr 172.21.15.47:0/3598403398 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:27:45.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.735+0000 7f3553e00640 1 --2- 172.21.15.47:0/3598403398 >> v2:172.21.15.116:3300/0 conn(0x7f355c0cded0 0x7f355c0ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:45.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.735+0000 7f355be00640 1 --2- 172.21.15.47:0/3598403398 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f355c0ceb40 0x7f355c06bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:45.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.735+0000 7f355aa00640 1 -- 172.21.15.47:0/3598403398 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897937873 0 0) 0x7f355c0cce30 con 0x7f355c06d650 2024-02-19T15:27:45.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.735+0000 7f355aa00640 1 -- 172.21.15.47:0/3598403398 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3540003530 con 0x7f355c06d650 2024-02-19T15:27:45.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.735+0000 7f355aa00640 1 -- 172.21.15.47:0/3598403398 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1733823338 0 0) 0x7f3540003530 con 0x7f355c06d650 2024-02-19T15:27:45.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.735+0000 7f355aa00640 1 -- 172.21.15.47:0/3598403398 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f355c0ceb40 msgr2=0x7f355c06bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:45.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.735+0000 7f355aa00640 1 --2- 172.21.15.47:0/3598403398 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f355c0ceb40 0x7f355c06bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:45.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.735+0000 7f355aa00640 1 -- 172.21.15.47:0/3598403398 >> v2:172.21.15.116:3300/0 conn(0x7f355c0cded0 msgr2=0x7f355c0ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:45.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.735+0000 7f355aa00640 1 --2- 172.21.15.47:0/3598403398 >> v2:172.21.15.116:3300/0 conn(0x7f355c0cded0 0x7f355c0ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:45.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.735+0000 7f355aa00640 1 -- 172.21.15.47:0/3598403398 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c0cd3f0 con 0x7f355c06d650 2024-02-19T15:27:45.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.735+0000 7f355aa00640 1 -- 172.21.15.47:0/3598403398 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f354c001680 con 0x7f355c06d650 2024-02-19T15:27:45.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.739+0000 7f355aa00640 1 -- 172.21.15.47:0/3598403398 >> v1:172.21.15.47:6789/0 conn(0x7f355c06d650 legacy=0x7f355c0d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:45.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.739+0000 7f355aa00640 1 --2- 172.21.15.47:0/3598403398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3540003980 0x7f3540003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:45.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.739+0000 7f355aa00640 1 --2- 172.21.15.47:0/3598403398 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f355c0ceb40 0x7f35400047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:45.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.739+0000 7f355aa00640 1 --2- 172.21.15.47:0/3598403398 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3540004d10 0x7f3540005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:45.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.739+0000 7f355aa00640 1 -- 172.21.15.47:0/3598403398 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c0cefd0 con 0x7f3540004d10 2024-02-19T15:27:45.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.739+0000 7f355aa00640 1 -- 172.21.15.47:0/3598403398 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c0cd110 con 0x7f3540003980 2024-02-19T15:27:45.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.739+0000 7f355aa00640 1 -- 172.21.15.47:0/3598403398 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f354000b9c0 con 0x7f355c0ceb40 2024-02-19T15:27:45.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.739+0000 7f355be00640 1 --2- 172.21.15.47:0/3598403398 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3540004d10 0x7f3540005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:45.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.739+0000 7f3553e00640 1 --2- 172.21.15.47:0/3598403398 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f355c0ceb40 0x7f35400047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:45.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.739+0000 7f355b400640 1 --2- 172.21.15.47:0/3598403398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3540003980 0x7f3540003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:45.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.739+0000 7f355be00640 1 -- 172.21.15.47:0/3598403398 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f355c0ceb40 msgr2=0x7f35400047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:45.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.739+0000 7f355be00640 1 --2- 172.21.15.47:0/3598403398 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f355c0ceb40 0x7f35400047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:45.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.739+0000 7f355be00640 1 -- 172.21.15.47:0/3598403398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3540003980 msgr2=0x7f3540003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:27:45.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.739+0000 7f355be00640 1 --2- 172.21.15.47:0/3598403398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3540003980 0x7f3540003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:45.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.739+0000 7f355be00640 1 -- 172.21.15.47:0/3598403398 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3540003530 con 0x7f3540004d10 2024-02-19T15:27:45.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.743+0000 7f355be00640 1 --2- 172.21.15.47:0/3598403398 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3540004d10 0x7f3540005190 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f3554009fc0 tx=0x7f355401cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31dd367c9cd42bef server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:45.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.743+0000 7f355aa00640 1 -- 172.21.15.47:0/3598403398 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35540a5c90 con 0x7f3540004d10 2024-02-19T15:27:45.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.743+0000 7f355aa00640 1 -- 172.21.15.47:0/3598403398 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355400f040 con 0x7f3540004d10 2024-02-19T15:27:45.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.743+0000 7f355aa00640 1 -- 172.21.15.47:0/3598403398 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35540ad440 con 0x7f3540004d10 2024-02-19T15:27:45.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.743+0000 7f3562800640 1 -- 172.21.15.47:0/3598403398 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3540004d10 msgr2=0x7f3540005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:45.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.743+0000 7f3562800640 1 --2- 172.21.15.47:0/3598403398 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3540004d10 0x7f3540005190 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f3554009fc0 tx=0x7f355401cc80 comp rx=0 tx=0).stop 2024-02-19T15:27:45.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.743+0000 7f355be00640 1 -- 172.21.15.47:0/3598403398 reap_dead start 2024-02-19T15:27:45.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.743+0000 7f3562800640 1 -- 172.21.15.47:0/3598403398 shutdown_connections 2024-02-19T15:27:45.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.743+0000 7f3562800640 1 -- 172.21.15.47:0/3598403398 >> 172.21.15.47:0/3598403398 conn(0x7f355c0d1bf0 msgr2=0x7f355c0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:45.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.743+0000 7f3562800640 1 -- 172.21.15.47:0/3598403398 shutdown_connections 2024-02-19T15:27:45.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.743+0000 7f3562800640 1 -- 172.21.15.47:0/3598403398 wait complete. 2024-02-19T15:27:45.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.747+0000 7f3562800640 1 Processor -- start 2024-02-19T15:27:45.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.747+0000 7f3562800640 1 -- start start 2024-02-19T15:27:45.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.747+0000 7f3562800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3540004d10 0x7f355c14ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:45.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.747+0000 7f3562800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f355c0c7b50 0x7f355c0ae860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:45.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.747+0000 7f3562800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f355c0c7fd0 0x7f355c0aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:45.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.747+0000 7f3562800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c0af2e0 con 0x7f3540004d10 2024-02-19T15:27:45.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.747+0000 7f3562800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c0b0c10 con 0x7f355c0c7b50 2024-02-19T15:27:45.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.747+0000 7f3562800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355c0b0ef0 con 0x7f355c0c7fd0 2024-02-19T15:27:45.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.747+0000 7f355b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3540004d10 0x7f355c14ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:45.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.747+0000 7f355b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3540004d10 0x7f355c14ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36798/0 (socket says 172.21.15.47:36798) 2024-02-19T15:27:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.747+0000 7f355b400640 1 -- 172.21.15.47:0/1908360940 learned_addr learned my addr 172.21.15.47:0/1908360940 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:27:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.747+0000 7f3553e00640 1 --2- 172.21.15.47:0/1908360940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f355c0c7b50 0x7f355c0ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.747+0000 7f355be00640 1 --2- 172.21.15.47:0/1908360940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f355c0c7fd0 0x7f355c0aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.747+0000 7f355b400640 1 -- 172.21.15.47:0/1908360940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f355c0c7fd0 msgr2=0x7f355c0aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.747+0000 7f355b400640 1 --2- 172.21.15.47:0/1908360940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f355c0c7fd0 0x7f355c0aeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.751+0000 7f355b400640 1 -- 172.21.15.47:0/1908360940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f355c0c7b50 msgr2=0x7f355c0ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.751+0000 7f355b400640 1 --2- 172.21.15.47:0/1908360940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f355c0c7b50 0x7f355c0ae860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.751+0000 7f355b400640 1 -- 172.21.15.47:0/1908360940 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c0b11d0 con 0x7f3540004d10 2024-02-19T15:27:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.751+0000 7f355b400640 1 --2- 172.21.15.47:0/1908360940 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3540004d10 0x7f355c14ff20 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f354c012510 tx=0x7f354c012910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.751+0000 7f3553400640 1 -- 172.21.15.47:0/1908360940 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c014750 con 0x7f3540004d10 2024-02-19T15:27:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.751+0000 7f3553400640 1 -- 172.21.15.47:0/1908360940 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354c0148b0 con 0x7f3540004d10 2024-02-19T15:27:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.751+0000 7f3553400640 1 -- 172.21.15.47:0/1908360940 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c014c60 con 0x7f3540004d10 2024-02-19T15:27:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.751+0000 7f3562800640 1 -- 172.21.15.47:0/1908360940 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355c0b14b0 con 0x7f3540004d10 2024-02-19T15:27:45.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.751+0000 7f3562800640 1 -- 172.21.15.47:0/1908360940 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f355c14f1a0 con 0x7f3540004d10 2024-02-19T15:27:45.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.751+0000 7f3562800640 1 -- 172.21.15.47:0/1908360940 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355c0ccaa0 con 0x7f3540004d10 2024-02-19T15:27:45.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.751+0000 7f3553400640 1 -- 172.21.15.47:0/1908360940 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f354c0153e0 con 0x7f3540004d10 2024-02-19T15:27:45.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.755+0000 7f3553400640 1 --2- 172.21.15.47:0/1908360940 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3538066e40 0x7f3538069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:27:45.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.755+0000 7f3553400640 1 -- 172.21.15.47:0/1908360940 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f354c0872e0 con 0x7f3540004d10 2024-02-19T15:27:45.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.755+0000 7f3553e00640 1 --2- 172.21.15.47:0/1908360940 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3538066e40 0x7f3538069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:27:45.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.755+0000 7f3553e00640 1 --2- 172.21.15.47:0/1908360940 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3538066e40 0x7f3538069300 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f355c1f6ab0 tx=0x7f3544009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:27:45.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.759+0000 7f3553400640 1 -- 172.21.15.47:0/1908360940 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f354c054030 con 0x7f3540004d10 2024-02-19T15:27:45.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.955+0000 7f3562800640 1 -- 172.21.15.47:0/1908360940 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f355c0b0d50 con 0x7f3540004d10 2024-02-19T15:27:45.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.955+0000 7f3553400640 1 -- 172.21.15.47:0/1908360940 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f355c0b0d50 con 0x7f3540004d10 2024-02-19T15:27:45.958 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:27:45.958 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:27:45.959 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:27:45.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.955+0000 7f3562800640 1 -- 172.21.15.47:0/1908360940 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3538066e40 msgr2=0x7f3538069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:45.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.955+0000 7f3562800640 1 --2- 172.21.15.47:0/1908360940 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3538066e40 0x7f3538069300 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f355c1f6ab0 tx=0x7f3544009190 comp rx=0 tx=0).stop 2024-02-19T15:27:45.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.955+0000 7f3562800640 1 -- 172.21.15.47:0/1908360940 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3540004d10 msgr2=0x7f355c14ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:27:45.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.955+0000 7f3562800640 1 --2- 172.21.15.47:0/1908360940 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3540004d10 0x7f355c14ff20 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f354c012510 tx=0x7f354c012910 comp rx=0 tx=0).stop 2024-02-19T15:27:45.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.955+0000 7f3562800640 1 -- 172.21.15.47:0/1908360940 shutdown_connections 2024-02-19T15:27:45.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.955+0000 7f3562800640 1 --2- 172.21.15.47:0/1908360940 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3538066e40 0x7f3538069300 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:45.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.955+0000 7f3562800640 1 --2- 172.21.15.47:0/1908360940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f355c0c7fd0 0x7f355c0aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:45.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.955+0000 7f3562800640 1 --2- 172.21.15.47:0/1908360940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f355c0c7b50 0x7f355c0ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:45.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.955+0000 7f3562800640 1 --2- 172.21.15.47:0/1908360940 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3540004d10 0x7f355c14ff20 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:27:45.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.955+0000 7f3562800640 1 -- 172.21.15.47:0/1908360940 >> 172.21.15.47:0/1908360940 conn(0x7f355c0d1bf0 msgr2=0x7f355c0b6cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:27:45.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.955+0000 7f3562800640 1 -- 172.21.15.47:0/1908360940 shutdown_connections 2024-02-19T15:27:45.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:27:45.955+0000 7f3562800640 1 -- 172.21.15.47:0/1908360940 wait complete. 2024-02-19T15:27:45.979 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-02-19T15:27:46.144 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"strays_created":1}} 2024-02-19T15:27:46.144 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-02-19T15:28:05.835 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:28:05.838 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:28:16.146 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-02-19T15:28:16.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5ff3db080 1 Processor -- start 2024-02-19T15:28:16.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5ff3db080 1 -- start start 2024-02-19T15:28:16.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5ff3db080 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56094b46a990 0x56094b3f7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5ff3db080 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56094b46ad70 0x56094b3f7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5ff3db080 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56094b46b250 0x56094b3fa5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5ff3db080 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x56094b2f91e0 con 0x56094b46b250 2024-02-19T15:28:16.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5ff3db080 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56094b2f8e80 con 0x56094b46a990 2024-02-19T15:28:16.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5ff3db080 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x56094b31c610 con 0x56094b46ad70 2024-02-19T15:28:16.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5fd000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56094b46b250 0x56094b3fa5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5fd000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56094b46b250 0x56094b3fa5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39338/0 (socket says 172.21.15.47:39338) 2024-02-19T15:28:16.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5fd000640 1 -- 172.21.15.47:0/1904521649 learned_addr learned my addr 172.21.15.47:0/1904521649 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:16.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5fc600640 1 --2- 172.21.15.47:0/1904521649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56094b46a990 0x56094b3f7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5fbc00640 1 --2- 172.21.15.47:0/1904521649 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56094b46ad70 0x56094b3f7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5fd000640 1 -- 172.21.15.47:0/1904521649 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56094b46ad70 msgr2=0x56094b3f7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:16.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5fd000640 1 --2- 172.21.15.47:0/1904521649 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56094b46ad70 0x56094b3f7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5fd000640 1 -- 172.21.15.47:0/1904521649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56094b46a990 msgr2=0x56094b3f7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:16.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5fd000640 1 --2- 172.21.15.47:0/1904521649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56094b46a990 0x56094b3f7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5fd000640 1 -- 172.21.15.47:0/1904521649 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f400a100 con 0x56094b46b250 2024-02-19T15:28:16.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5fc600640 1 --2- 172.21.15.47:0/1904521649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56094b46a990 0x56094b3f7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:16.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5fd000640 1 --2- 172.21.15.47:0/1904521649 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56094b46b250 0x56094b3fa5a0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400a3f0 tx=0x7ff5f40033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fedcaf679990c3a0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:16.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5fa800640 1 -- 172.21.15.47:0/1904521649 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f40a3e50 con 0x56094b46b250 2024-02-19T15:28:16.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5fa800640 1 -- 172.21.15.47:0/1904521649 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f40ab5b0 con 0x56094b46b250 2024-02-19T15:28:16.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5fa800640 1 -- 172.21.15.47:0/1904521649 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f40ab920 con 0x56094b46b250 2024-02-19T15:28:16.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1904521649 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56094b46b250 msgr2=0x56094b3fa5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5ff3db080 1 --2- 172.21.15.47:0/1904521649 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56094b46b250 0x56094b3fa5a0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400a3f0 tx=0x7ff5f40033b0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1904521649 shutdown_connections 2024-02-19T15:28:16.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5ff3db080 1 --2- 172.21.15.47:0/1904521649 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56094b46b250 0x56094b3fa5a0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5ff3db080 1 --2- 172.21.15.47:0/1904521649 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56094b46ad70 0x56094b3f7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5ff3db080 1 --2- 172.21.15.47:0/1904521649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56094b46a990 0x56094b3f7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1904521649 >> 172.21.15.47:0/1904521649 conn(0x56094b3b2670 msgr2=0x56094b3b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:16.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.190+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1904521649 shutdown_connections 2024-02-19T15:28:16.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1904521649 wait complete. 2024-02-19T15:28:16.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5ff3db080 1 Processor -- start 2024-02-19T15:28:16.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5ff3db080 1 -- start start 2024-02-19T15:28:16.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5ff3db080 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56094b46a990 0x56094b469dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5ff3db080 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56094b46ad70 0x56094b46a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5ff3db080 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56094b46b250 0x56094b48b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5ff3db080 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x56094b2ec740 con 0x56094b46a990 2024-02-19T15:28:16.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5ff3db080 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56094b46a850 con 0x56094b46b250 2024-02-19T15:28:16.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5ff3db080 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x56094b484190 con 0x56094b46ad70 2024-02-19T15:28:16.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5fc600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56094b46a990 0x56094b469dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5fc600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56094b46a990 0x56094b469dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39348/0 (socket says 172.21.15.47:39348) 2024-02-19T15:28:16.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 learned_addr learned my addr 172.21.15.47:0/1377796456 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:16.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5fbc00640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56094b46ad70 0x56094b46a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5fd000640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56094b46b250 0x56094b48b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56094b46ad70 msgr2=0x56094b46a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5fc600640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x56094b46ad70 0x56094b46a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56094b46b250 msgr2=0x56094b48b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5fc600640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56094b46b250 0x56094b48b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56094b31f0d0 con 0x56094b46a990 2024-02-19T15:28:16.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5fd000640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56094b46b250 0x56094b48b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:16.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.194+0000 7ff5fc600640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56094b46a990 0x56094b469dd0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec008740 tx=0x7ff5ec008c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:16.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.198+0000 7ff5f9400640 1 -- 172.21.15.47:0/1377796456 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec006b70 con 0x56094b46a990 2024-02-19T15:28:16.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.198+0000 7ff5f9400640 1 -- 172.21.15.47:0/1377796456 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ec010040 con 0x56094b46a990 2024-02-19T15:28:16.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.198+0000 7ff5f9400640 1 -- 172.21.15.47:0/1377796456 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec0143f0 con 0x56094b46a990 2024-02-19T15:28:16.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.198+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56094b304d80 con 0x56094b46a990 2024-02-19T15:28:16.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.198+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56094b2f7f30 con 0x56094b46a990 2024-02-19T15:28:16.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.198+0000 7ff5f9400640 1 -- 172.21.15.47:0/1377796456 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff5ec014550 con 0x56094b46a990 2024-02-19T15:28:16.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.198+0000 7ff5f9400640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff5dc065820 0x7ff5dc067ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.198+0000 7ff5f9400640 1 -- 172.21.15.47:0/1377796456 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7ff5ec083cc0 con 0x56094b46a990 2024-02-19T15:28:16.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.198+0000 7ff5ff3db080 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] conn(0x56094b484e10 0x56094b48be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.198+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x56094b484e10 2024-02-19T15:28:16.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.198+0000 7ff5fbc00640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff5dc065820 0x7ff5dc067ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.198+0000 7ff5fd000640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] conn(0x56094b484e10 0x56094b48be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.202+0000 7ff5fbc00640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff5dc065820 0x7ff5dc067ce0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8000f70 tx=0x7ff5e8001820 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:16.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.202+0000 7ff5fd000640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] conn(0x56094b484e10 0x56094b48be00 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:16.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.202+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.7 v2:172.21.15.116:6800/3212743377 1 ==== osd_op_reply(1 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40aba00 con 0x56094b484e10 2024-02-19T15:28:16.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.202+0000 7ff5fd000640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] conn(0x7ff5f40b4430 0x7ff5f40b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.202+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5f40b4430 2024-02-19T15:28:16.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.202+0000 7ff5fc600640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] conn(0x7ff5f40b4430 0x7ff5f40b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.202+0000 7ff5fc600640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] conn(0x7ff5f40b4430 0x7ff5f40b48b0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:16.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.202+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 <== osd.1 v2:172.21.15.47:6800/1591346632 1 ==== osd_op_reply(2 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5ec01a020 con 0x7ff5f40b4430 2024-02-19T15:28:16.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.202+0000 7ff5fc600640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x7ff5ec009040 0x7ff5ec0535d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.202+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec009040 2024-02-19T15:28:16.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.202+0000 7ff5fbc00640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x7ff5ec009040 0x7ff5ec0535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.202+0000 7ff5fbc00640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x7ff5ec009040 0x7ff5ec0535d0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:16.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.202+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 <== osd.3 v2:172.21.15.47:6824/4149334728 1 ==== osd_op_reply(3 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5e8006030 con 0x7ff5ec009040 2024-02-19T15:28:16.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.202+0000 7ff5fbc00640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x7ff5e801a040 0x7ff5e801c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.202+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5e801a040 2024-02-19T15:28:16.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.202+0000 7ff5fd000640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x7ff5e801a040 0x7ff5e801c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.206+0000 7ff5fd000640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x7ff5e801a040 0x7ff5e801c950 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:16.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.206+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.0 v2:172.21.15.47:6809/1968337531 1 ==== osd_op_reply(4 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40aba00 con 0x7ff5e801a040 2024-02-19T15:28:16.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.206+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x56094b484e10 2024-02-19T15:28:16.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.206+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.7 v2:172.21.15.116:6800/3212743377 2 ==== osd_op_reply(5 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40aba00 con 0x56094b484e10 2024-02-19T15:28:16.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.206+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec009040 2024-02-19T15:28:16.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.206+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 <== osd.3 v2:172.21.15.47:6824/4149334728 2 ==== osd_op_reply(6 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5e8006030 con 0x7ff5ec009040 2024-02-19T15:28:16.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.206+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5e801a040 2024-02-19T15:28:16.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.206+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.0 v2:172.21.15.47:6809/1968337531 2 ==== osd_op_reply(7 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40aba00 con 0x7ff5e801a040 2024-02-19T15:28:16.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.206+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x56094b484e10 2024-02-19T15:28:16.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.206+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.7 v2:172.21.15.116:6800/3212743377 3 ==== osd_op_reply(8 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40aba00 con 0x56094b484e10 2024-02-19T15:28:16.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.206+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5e801a040 2024-02-19T15:28:16.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.206+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.0 v2:172.21.15.47:6809/1968337531 3 ==== osd_op_reply(9 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40aba00 con 0x7ff5e801a040 2024-02-19T15:28:16.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.206+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec009040 2024-02-19T15:28:16.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.206+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 <== osd.3 v2:172.21.15.47:6824/4149334728 3 ==== osd_op_reply(10 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5e8006030 con 0x7ff5ec009040 2024-02-19T15:28:16.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.206+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec009040 2024-02-19T15:28:16.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.206+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 <== osd.3 v2:172.21.15.47:6824/4149334728 4 ==== osd_op_reply(11 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5e8006030 con 0x7ff5ec009040 2024-02-19T15:28:16.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.206+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5e801a040 2024-02-19T15:28:16.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.206+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.0 v2:172.21.15.47:6809/1968337531 4 ==== osd_op_reply(12 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40aba00 con 0x7ff5e801a040 2024-02-19T15:28:16.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.206+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x56094b484e10 2024-02-19T15:28:16.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.210+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.7 v2:172.21.15.116:6800/3212743377 4 ==== osd_op_reply(13 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40aba00 con 0x56094b484e10 2024-02-19T15:28:16.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.210+0000 7ff5fd000640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] conn(0x7ff5f40c4910 0x7ff5f40bf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.210+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5f40c4910 2024-02-19T15:28:16.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.210+0000 7ff5fc600640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] conn(0x7ff5f40c4910 0x7ff5f40bf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.210+0000 7ff5fc600640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] conn(0x7ff5f40c4910 0x7ff5f40bf3d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:16.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.210+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 <== osd.4 v2:172.21.15.116:6809/3366344088 1 ==== osd_op_reply(14 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5ec01a020 con 0x7ff5f40c4910 2024-02-19T15:28:16.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.210+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5f40b4430 2024-02-19T15:28:16.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.210+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 <== osd.1 v2:172.21.15.47:6800/1591346632 2 ==== osd_op_reply(15 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5ec01a020 con 0x7ff5f40b4430 2024-02-19T15:28:16.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.210+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5e801a040 2024-02-19T15:28:16.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.210+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.0 v2:172.21.15.47:6809/1968337531 5 ==== osd_op_reply(16 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40aba00 con 0x7ff5e801a040 2024-02-19T15:28:16.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.210+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5f40c4910 2024-02-19T15:28:16.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.210+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 <== osd.4 v2:172.21.15.116:6809/3366344088 2 ==== osd_op_reply(17 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5ec01a020 con 0x7ff5f40c4910 2024-02-19T15:28:16.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.210+0000 7ff5fc600640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] conn(0x7ff5ec054970 0x7ff5ec059780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.210+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec054970 2024-02-19T15:28:16.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.210+0000 7ff5fbc00640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] conn(0x7ff5ec054970 0x7ff5ec059780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.210+0000 7ff5fbc00640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] conn(0x7ff5ec054970 0x7ff5ec059780 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:16.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.214+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 <== osd.5 v2:172.21.15.116:6808/3478542733 1 ==== osd_op_reply(18 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5e8025040 con 0x7ff5ec054970 2024-02-19T15:28:16.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.214+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec009040 2024-02-19T15:28:16.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.214+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 <== osd.3 v2:172.21.15.47:6824/4149334728 5 ==== osd_op_reply(19 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5e8025040 con 0x7ff5ec009040 2024-02-19T15:28:16.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.214+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x56094b484e10 2024-02-19T15:28:16.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.214+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.7 v2:172.21.15.116:6800/3212743377 5 ==== osd_op_reply(20 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40aba00 con 0x56094b484e10 2024-02-19T15:28:16.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.214+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x56094b484e10 2024-02-19T15:28:16.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.214+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.7 v2:172.21.15.116:6800/3212743377 6 ==== osd_op_reply(21 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40aba00 con 0x56094b484e10 2024-02-19T15:28:16.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.214+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec009040 2024-02-19T15:28:16.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.214+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 <== osd.3 v2:172.21.15.47:6824/4149334728 6 ==== osd_op_reply(22 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5e8025040 con 0x7ff5ec009040 2024-02-19T15:28:16.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.214+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5e801a040 2024-02-19T15:28:16.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.214+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.0 v2:172.21.15.47:6809/1968337531 6 ==== osd_op_reply(23 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40aba00 con 0x7ff5e801a040 2024-02-19T15:28:16.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.214+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5f40b4430 2024-02-19T15:28:16.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.214+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 <== osd.1 v2:172.21.15.47:6800/1591346632 3 ==== osd_op_reply(24 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5ec01a020 con 0x7ff5f40b4430 2024-02-19T15:28:16.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.214+0000 7ff5fc600640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] conn(0x7ff5ec05c990 0x7ff5ec052cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.214+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec05c990 2024-02-19T15:28:16.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.214+0000 7ff5fd000640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] conn(0x7ff5ec05c990 0x7ff5ec052cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.214+0000 7ff5fd000640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] conn(0x7ff5ec05c990 0x7ff5ec052cd0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:16.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.214+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.6 v2:172.21.15.116:6824/2921645124 1 ==== osd_op_reply(25 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40dc040 con 0x7ff5ec05c990 2024-02-19T15:28:16.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.218+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5e801a040 2024-02-19T15:28:16.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.218+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.0 v2:172.21.15.47:6809/1968337531 7 ==== osd_op_reply(26 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40dc040 con 0x7ff5e801a040 2024-02-19T15:28:16.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.218+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec054970 2024-02-19T15:28:16.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.218+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 <== osd.5 v2:172.21.15.116:6808/3478542733 2 ==== osd_op_reply(27 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5e8025040 con 0x7ff5ec054970 2024-02-19T15:28:16.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.218+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec054970 2024-02-19T15:28:16.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.218+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 <== osd.5 v2:172.21.15.116:6808/3478542733 3 ==== osd_op_reply(28 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5e8025040 con 0x7ff5ec054970 2024-02-19T15:28:16.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.218+0000 7ff5fbc00640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] conn(0x7ff5e802e440 0x7ff5e802e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.218+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5e802e440 2024-02-19T15:28:16.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.218+0000 7ff5fc600640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] conn(0x7ff5e802e440 0x7ff5e802e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.218+0000 7ff5fc600640 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] conn(0x7ff5e802e440 0x7ff5e802e8a0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:16.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.218+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 <== osd.2 v2:172.21.15.47:6808/2881016873 1 ==== osd_op_reply(29 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5ec01a020 con 0x7ff5e802e440 2024-02-19T15:28:16.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.218+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec05c990 2024-02-19T15:28:16.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.218+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.6 v2:172.21.15.116:6824/2921645124 2 ==== osd_op_reply(30 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40dc040 con 0x7ff5ec05c990 2024-02-19T15:28:16.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.218+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5f40b4430 2024-02-19T15:28:16.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.218+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 <== osd.1 v2:172.21.15.47:6800/1591346632 4 ==== osd_op_reply(31 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5ec01a020 con 0x7ff5f40b4430 2024-02-19T15:28:16.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.218+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5f40c4910 2024-02-19T15:28:16.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.218+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 <== osd.4 v2:172.21.15.116:6809/3366344088 3 ==== osd_op_reply(32 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5ec01a020 con 0x7ff5f40c4910 2024-02-19T15:28:16.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.218+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5e801a040 2024-02-19T15:28:16.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.218+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.0 v2:172.21.15.47:6809/1968337531 8 ==== osd_op_reply(33 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40dc040 con 0x7ff5e801a040 2024-02-19T15:28:16.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.218+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5e801a040 2024-02-19T15:28:16.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.222+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.0 v2:172.21.15.47:6809/1968337531 9 ==== osd_op_reply(34 [pgnls start_epoch 114 out=49b] v108'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40dc040 con 0x7ff5e801a040 2024-02-19T15:28:16.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.222+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec009040 2024-02-19T15:28:16.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.222+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 <== osd.3 v2:172.21.15.47:6824/4149334728 7 ==== osd_op_reply(35 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5e8025040 con 0x7ff5ec009040 2024-02-19T15:28:16.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.222+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5f40b4430 2024-02-19T15:28:16.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.222+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 <== osd.1 v2:172.21.15.47:6800/1591346632 5 ==== osd_op_reply(36 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5ec01a020 con 0x7ff5f40b4430 2024-02-19T15:28:16.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.222+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec009040 2024-02-19T15:28:16.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.222+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 <== osd.3 v2:172.21.15.47:6824/4149334728 8 ==== osd_op_reply(37 [pgnls start_epoch 114 out=49b] v108'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5e8025040 con 0x7ff5ec009040 2024-02-19T15:28:16.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.222+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5f40c4910 2024-02-19T15:28:16.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.222+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 <== osd.4 v2:172.21.15.116:6809/3366344088 4 ==== osd_op_reply(38 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5ec01a020 con 0x7ff5f40c4910 2024-02-19T15:28:16.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.222+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec054970 2024-02-19T15:28:16.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.222+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 <== osd.5 v2:172.21.15.116:6808/3478542733 4 ==== osd_op_reply(39 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5e8025040 con 0x7ff5ec054970 2024-02-19T15:28:16.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.222+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec054970 2024-02-19T15:28:16.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.222+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 <== osd.5 v2:172.21.15.116:6808/3478542733 5 ==== osd_op_reply(40 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5e8025040 con 0x7ff5ec054970 2024-02-19T15:28:16.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.222+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec05c990 2024-02-19T15:28:16.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.222+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.6 v2:172.21.15.116:6824/2921645124 3 ==== osd_op_reply(41 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40dc040 con 0x7ff5ec05c990 2024-02-19T15:28:16.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.222+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5f40b4430 2024-02-19T15:28:16.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.222+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 <== osd.1 v2:172.21.15.47:6800/1591346632 6 ==== osd_op_reply(42 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5ec01a020 con 0x7ff5f40b4430 2024-02-19T15:28:16.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.222+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec054970 2024-02-19T15:28:16.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.222+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 <== osd.5 v2:172.21.15.116:6808/3478542733 6 ==== osd_op_reply(43 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5e8025040 con 0x7ff5ec054970 2024-02-19T15:28:16.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.222+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5f40c4910 2024-02-19T15:28:16.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.226+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 <== osd.4 v2:172.21.15.116:6809/3366344088 5 ==== osd_op_reply(44 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5ec01a020 con 0x7ff5f40c4910 2024-02-19T15:28:16.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.226+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x56094b484e10 2024-02-19T15:28:16.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.226+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.7 v2:172.21.15.116:6800/3212743377 7 ==== osd_op_reply(45 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40dc040 con 0x56094b484e10 2024-02-19T15:28:16.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.226+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x56094b484e10 2024-02-19T15:28:16.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.226+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.7 v2:172.21.15.116:6800/3212743377 8 ==== osd_op_reply(46 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40dc040 con 0x56094b484e10 2024-02-19T15:28:16.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.226+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x56094b484e10 2024-02-19T15:28:16.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.226+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.7 v2:172.21.15.116:6800/3212743377 9 ==== osd_op_reply(47 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40dc040 con 0x56094b484e10 2024-02-19T15:28:16.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.226+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x56094b484e10 2024-02-19T15:28:16.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.226+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.7 v2:172.21.15.116:6800/3212743377 10 ==== osd_op_reply(48 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40dc040 con 0x56094b484e10 2024-02-19T15:28:16.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.226+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5f40c4910 2024-02-19T15:28:16.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.226+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 <== osd.4 v2:172.21.15.116:6809/3366344088 6 ==== osd_op_reply(49 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5ec01a020 con 0x7ff5f40c4910 2024-02-19T15:28:16.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.226+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5f40c4910 2024-02-19T15:28:16.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.226+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 <== osd.4 v2:172.21.15.116:6809/3366344088 7 ==== osd_op_reply(50 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5ec01a020 con 0x7ff5f40c4910 2024-02-19T15:28:16.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.226+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec009040 2024-02-19T15:28:16.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.226+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 <== osd.3 v2:172.21.15.47:6824/4149334728 9 ==== osd_op_reply(51 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5e8025040 con 0x7ff5ec009040 2024-02-19T15:28:16.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.226+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x56094b484e10 2024-02-19T15:28:16.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.226+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.7 v2:172.21.15.116:6800/3212743377 11 ==== osd_op_reply(52 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40dc040 con 0x56094b484e10 2024-02-19T15:28:16.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.226+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5e801a040 2024-02-19T15:28:16.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.226+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.0 v2:172.21.15.47:6809/1968337531 10 ==== osd_op_reply(53 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40dc040 con 0x7ff5e801a040 2024-02-19T15:28:16.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.226+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5f40c4910 2024-02-19T15:28:16.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.230+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 <== osd.4 v2:172.21.15.116:6809/3366344088 8 ==== osd_op_reply(54 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5ec01a020 con 0x7ff5f40c4910 2024-02-19T15:28:16.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.230+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5f40c4910 2024-02-19T15:28:16.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.230+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 <== osd.4 v2:172.21.15.116:6809/3366344088 9 ==== osd_op_reply(55 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5ec01a020 con 0x7ff5f40c4910 2024-02-19T15:28:16.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.230+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5e802e440 2024-02-19T15:28:16.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.230+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 <== osd.2 v2:172.21.15.47:6808/2881016873 2 ==== osd_op_reply(56 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5ec01a020 con 0x7ff5e802e440 2024-02-19T15:28:16.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.230+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5e801a040 2024-02-19T15:28:16.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.230+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.0 v2:172.21.15.47:6809/1968337531 11 ==== osd_op_reply(57 [pgnls start_epoch 114 out=81b] v108'8 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7ff5f40dc040 con 0x7ff5e801a040 2024-02-19T15:28:16.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.230+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5f40c4910 2024-02-19T15:28:16.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.230+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 <== osd.4 v2:172.21.15.116:6809/3366344088 10 ==== osd_op_reply(58 [pgnls start_epoch 114 out=49b] v108'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5ec01a020 con 0x7ff5f40c4910 2024-02-19T15:28:16.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.230+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec05c990 2024-02-19T15:28:16.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.230+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.6 v2:172.21.15.116:6824/2921645124 4 ==== osd_op_reply(59 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40dc040 con 0x7ff5ec05c990 2024-02-19T15:28:16.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.230+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5e801a040 2024-02-19T15:28:16.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.230+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.0 v2:172.21.15.47:6809/1968337531 12 ==== osd_op_reply(60 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40dc040 con 0x7ff5e801a040 2024-02-19T15:28:16.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.230+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5f40b4430 2024-02-19T15:28:16.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.230+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 <== osd.1 v2:172.21.15.47:6800/1591346632 7 ==== osd_op_reply(61 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5ec01a020 con 0x7ff5f40b4430 2024-02-19T15:28:16.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.230+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5f40c4910 2024-02-19T15:28:16.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.230+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 <== osd.4 v2:172.21.15.116:6809/3366344088 11 ==== osd_op_reply(62 [pgnls start_epoch 114 out=81b] v108'9 uv1 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7ff5ec01a020 con 0x7ff5f40c4910 2024-02-19T15:28:16.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.230+0000 7ff5fc600640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec05c990 2024-02-19T15:28:16.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.230+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 <== osd.6 v2:172.21.15.116:6824/2921645124 5 ==== osd_op_reply(63 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5f40e6040 con 0x7ff5ec05c990 2024-02-19T15:28:16.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.230+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 --> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x56094b483000 con 0x7ff5ec054970 2024-02-19T15:28:16.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5fbc00640 1 -- 172.21.15.47:0/1377796456 <== osd.5 v2:172.21.15.116:6808/3478542733 7 ==== osd_op_reply(64 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff5e8025040 con 0x7ff5ec054970 2024-02-19T15:28:16.233 INFO:teuthology.orchestra.run.smithi047.stdout:10000000001.00000000 2024-02-19T15:28:16.234 INFO:teuthology.orchestra.run.smithi047.stdout:10000000001.00000001 2024-02-19T15:28:16.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x7ff5e801a040 msgr2=0x7ff5e801c950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x7ff5e801a040 0x7ff5e801c950 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] conn(0x7ff5f40b4430 msgr2=0x7ff5f40b48b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] conn(0x7ff5f40b4430 0x7ff5f40b48b0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] conn(0x7ff5e802e440 msgr2=0x7ff5e802e8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] conn(0x7ff5e802e440 0x7ff5e802e8a0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x7ff5ec009040 msgr2=0x7ff5ec0535d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x7ff5ec009040 0x7ff5ec0535d0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] conn(0x7ff5f40c4910 msgr2=0x7ff5f40bf3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] conn(0x7ff5f40c4910 0x7ff5f40bf3d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 reap_dead start 2024-02-19T15:28:16.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] conn(0x7ff5ec054970 msgr2=0x7ff5ec059780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] conn(0x7ff5ec054970 0x7ff5ec059780 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] conn(0x7ff5ec05c990 msgr2=0x7ff5ec052cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] conn(0x7ff5ec05c990 0x7ff5ec052cd0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] conn(0x56094b484e10 msgr2=0x56094b48be00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] conn(0x56094b484e10 0x56094b48be00 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 reap_dead start 2024-02-19T15:28:16.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 reap_dead start 2024-02-19T15:28:16.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5fd000640 1 -- 172.21.15.47:0/1377796456 reap_dead start 2024-02-19T15:28:16.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff5dc065820 msgr2=0x7ff5dc067ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff5dc065820 0x7ff5dc067ce0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8000f70 tx=0x7ff5e8001820 comp rx=0 tx=0).stop 2024-02-19T15:28:16.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56094b46a990 msgr2=0x56094b469dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56094b46a990 0x56094b469dd0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec008740 tx=0x7ff5ec008c00 comp rx=0 tx=0).stop 2024-02-19T15:28:16.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1377796456 shutdown_connections 2024-02-19T15:28:16.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff5dc065820 0x7ff5dc067ce0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 --2- 172.21.15.47:0/1377796456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56094b46a990 0x56094b469dd0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1377796456 >> 172.21.15.47:0/1377796456 conn(0x56094b3b2670 msgr2=0x56094b3f8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:16.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1377796456 shutdown_connections 2024-02-19T15:28:16.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.234+0000 7ff5ff3db080 1 -- 172.21.15.47:0/1377796456 wait complete. 2024-02-19T15:28:16.239 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 found 2024-02-19T15:28:16.239 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:16.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.418+0000 7f821d200640 1 Processor -- start 2024-02-19T15:28:16.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.418+0000 7f821d200640 1 -- start start 2024-02-19T15:28:16.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.422+0000 7f821d200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f821815df50 0x7f82180d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.422+0000 7f821d200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f82180d0f90 0x7f8218167d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.422+0000 7f821d200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82180d2da0 con 0x7f82180d4a70 2024-02-19T15:28:16.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.422+0000 7f821d200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82180b6df0 con 0x7f82180d0f90 2024-02-19T15:28:16.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.422+0000 7f821d200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82180b70d0 con 0x7f821815df50 2024-02-19T15:28:16.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.422+0000 7f8216800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f82180d0f90 0x7f8218167d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.422+0000 7f820d400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f82180d4a70 0x7f82180d0880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60974/0 (socket says 172.21.15.47:60974) 2024-02-19T15:28:16.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.422+0000 7f820d400640 1 -- 172.21.15.47:0/1670609215 learned_addr learned my addr 172.21.15.47:0/1670609215 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:16.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.422+0000 7f8215e00640 1 --2- 172.21.15.47:0/1670609215 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f821815df50 0x7f82180d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.422+0000 7f8216800640 1 -- 172.21.15.47:0/1670609215 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f821815df50 msgr2=0x7f82180d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:16.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.422+0000 7f8216800640 1 --2- 172.21.15.47:0/1670609215 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f821815df50 0x7f82180d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.422+0000 7f8216800640 1 -- 172.21.15.47:0/1670609215 >> v1:172.21.15.47:6789/0 conn(0x7f82180d4a70 legacy=0x7f82180d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.422+0000 7f8216800640 1 -- 172.21.15.47:0/1670609215 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82180b73b0 con 0x7f82180d0f90 2024-02-19T15:28:16.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.422+0000 7f8215400640 1 -- 172.21.15.47:0/1670609215 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286349848 0 0) 0x7f82180d2da0 con 0x7f82180d4a70 2024-02-19T15:28:16.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.426+0000 7f8216800640 1 --2- 172.21.15.47:0/1670609215 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82180d0f90 0x7f8218167d90 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f8210009d90 tx=0x7f821001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a31a9a57c5359b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:16.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.426+0000 7f8215400640 1 -- 172.21.15.47:0/1670609215 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82100a5910 con 0x7f82180d0f90 2024-02-19T15:28:16.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.426+0000 7f8215400640 1 -- 172.21.15.47:0/1670609215 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82100a5a70 con 0x7f82180d0f90 2024-02-19T15:28:16.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.426+0000 7f8215400640 1 -- 172.21.15.47:0/1670609215 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82100a5e00 con 0x7f82180d0f90 2024-02-19T15:28:16.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.426+0000 7f821d200640 1 -- 172.21.15.47:0/1670609215 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82180d0f90 msgr2=0x7f8218167d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.426+0000 7f821d200640 1 --2- 172.21.15.47:0/1670609215 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82180d0f90 0x7f8218167d90 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f8210009d90 tx=0x7f821001c940 comp rx=0 tx=0).stop 2024-02-19T15:28:16.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.426+0000 7f821d200640 1 -- 172.21.15.47:0/1670609215 shutdown_connections 2024-02-19T15:28:16.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.426+0000 7f821d200640 1 --2- 172.21.15.47:0/1670609215 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82180d0f90 0x7f8218167d90 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.426+0000 7f821d200640 1 --2- 172.21.15.47:0/1670609215 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f821815df50 0x7f82180d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.426+0000 7f821d200640 1 -- 172.21.15.47:0/1670609215 >> 172.21.15.47:0/1670609215 conn(0x7f82180d3e70 msgr2=0x7f82180d4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:16.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.426+0000 7f821d200640 1 -- 172.21.15.47:0/1670609215 shutdown_connections 2024-02-19T15:28:16.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.426+0000 7f821d200640 1 -- 172.21.15.47:0/1670609215 wait complete. 2024-02-19T15:28:16.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f821d200640 1 Processor -- start 2024-02-19T15:28:16.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f821d200640 1 -- start start 2024-02-19T15:28:16.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f821d200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82180d0f90 0x7f82181566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f821d200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82180d4a70 0x7f821806c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f821d200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f821815df50 0x7f821815f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f821d200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8218157000 con 0x7f82180d4a70 2024-02-19T15:28:16.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f821d200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82181572e0 con 0x7f821815df50 2024-02-19T15:28:16.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f821d200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f821815e9d0 con 0x7f82180d0f90 2024-02-19T15:28:16.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f820d400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82180d4a70 0x7f821806c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f820d400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82180d4a70 0x7f821806c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39366/0 (socket says 172.21.15.47:39366) 2024-02-19T15:28:16.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f820d400640 1 -- 172.21.15.47:0/1349553048 learned_addr learned my addr 172.21.15.47:0/1349553048 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:16.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f8215e00640 1 --2- 172.21.15.47:0/1349553048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82180d0f90 0x7f82181566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f8216800640 1 --2- 172.21.15.47:0/1349553048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f821815df50 0x7f821815f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f820d400640 1 -- 172.21.15.47:0/1349553048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82180d0f90 msgr2=0x7f82181566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f820d400640 1 --2- 172.21.15.47:0/1349553048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82180d0f90 0x7f82181566d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f820d400640 1 -- 172.21.15.47:0/1349553048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f821815df50 msgr2=0x7f821815f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f820d400640 1 --2- 172.21.15.47:0/1349553048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f821815df50 0x7f821815f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f820d400640 1 -- 172.21.15.47:0/1349553048 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8210009a70 con 0x7f82180d4a70 2024-02-19T15:28:16.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f820d400640 1 --2- 172.21.15.47:0/1349553048 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82180d4a70 0x7f821806c030 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f82000011f0 tx=0x7f820000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:16.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f820ea00640 1 -- 172.21.15.47:0/1349553048 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82000074e0 con 0x7f82180d4a70 2024-02-19T15:28:16.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f820ea00640 1 -- 172.21.15.47:0/1349553048 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8200007640 con 0x7f82180d4a70 2024-02-19T15:28:16.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f820ea00640 1 -- 172.21.15.47:0/1349553048 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8200004a10 con 0x7f82180d4a70 2024-02-19T15:28:16.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.430+0000 7f821d200640 1 -- 172.21.15.47:0/1349553048 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f821815ecb0 con 0x7f82180d4a70 2024-02-19T15:28:16.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.434+0000 7f821d200640 1 -- 172.21.15.47:0/1349553048 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f821806c570 con 0x7f82180d4a70 2024-02-19T15:28:16.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.434+0000 7f821d200640 1 -- 172.21.15.47:0/1349553048 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f821815eef0 con 0x7f82180d4a70 2024-02-19T15:28:16.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.434+0000 7f820ea00640 1 -- 172.21.15.47:0/1349553048 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8200004b70 con 0x7f82180d4a70 2024-02-19T15:28:16.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.434+0000 7f820ea00640 1 --2- 172.21.15.47:0/1349553048 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f81f4066e40 0x7f81f4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.434+0000 7f820ea00640 1 -- 172.21.15.47:0/1349553048 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f8200083fc0 con 0x7f82180d4a70 2024-02-19T15:28:16.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.438+0000 7f8215e00640 1 --2- 172.21.15.47:0/1349553048 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f81f4066e40 0x7f81f4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.438+0000 7f8215e00640 1 --2- 172.21.15.47:0/1349553048 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f81f4066e40 0x7f81f4069300 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f82181fa250 tx=0x7f8208009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:16.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.442+0000 7f820ea00640 1 -- 172.21.15.47:0/1349553048 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f82000b61c0 con 0x7f82180d4a70 2024-02-19T15:28:16.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.638+0000 7f821d200640 1 -- 172.21.15.47:0/1349553048 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8218075360 con 0x7f82180d4a70 2024-02-19T15:28:16.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.638+0000 7f820ea00640 1 -- 172.21.15.47:0/1349553048 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f8200018020 con 0x7f82180d4a70 2024-02-19T15:28:16.638 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:16.638 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:16.639 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:16.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.638+0000 7f821d200640 1 -- 172.21.15.47:0/1349553048 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f81f4066e40 msgr2=0x7f81f4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.638+0000 7f821d200640 1 --2- 172.21.15.47:0/1349553048 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f81f4066e40 0x7f81f4069300 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f82181fa250 tx=0x7f8208009190 comp rx=0 tx=0).stop 2024-02-19T15:28:16.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.638+0000 7f821d200640 1 -- 172.21.15.47:0/1349553048 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82180d4a70 msgr2=0x7f821806c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.638+0000 7f821d200640 1 --2- 172.21.15.47:0/1349553048 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82180d4a70 0x7f821806c030 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f82000011f0 tx=0x7f820000c9b0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.638+0000 7f821d200640 1 -- 172.21.15.47:0/1349553048 shutdown_connections 2024-02-19T15:28:16.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.638+0000 7f821d200640 1 --2- 172.21.15.47:0/1349553048 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f81f4066e40 0x7f81f4069300 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.638+0000 7f821d200640 1 --2- 172.21.15.47:0/1349553048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f821815df50 0x7f821815f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.638+0000 7f821d200640 1 --2- 172.21.15.47:0/1349553048 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82180d4a70 0x7f821806c030 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.638+0000 7f821d200640 1 --2- 172.21.15.47:0/1349553048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82180d0f90 0x7f82181566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.638+0000 7f821d200640 1 -- 172.21.15.47:0/1349553048 >> 172.21.15.47:0/1349553048 conn(0x7f82180d3e70 msgr2=0x7f82180b87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:16.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.638+0000 7f821d200640 1 -- 172.21.15.47:0/1349553048 shutdown_connections 2024-02-19T15:28:16.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.638+0000 7f821d200640 1 -- 172.21.15.47:0/1349553048 wait complete. 2024-02-19T15:28:16.660 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-02-19T15:28:16.822 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-02-19T15:28:16.822 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-02-19T15:28:16.823 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:16.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.962+0000 7f06d8400640 1 Processor -- start 2024-02-19T15:28:16.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.962+0000 7f06d8400640 1 -- start start 2024-02-19T15:28:16.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.962+0000 7f06d8400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f06d015df50 0x7f06d00d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.962+0000 7f06d8400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d00d2da0 con 0x7f06d00d4a70 2024-02-19T15:28:16.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.962+0000 7f06d8400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d00b6df0 con 0x7f06d00d0f90 2024-02-19T15:28:16.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.962+0000 7f06d8400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d00b70d0 con 0x7f06d015df50 2024-02-19T15:28:16.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.962+0000 7f06d5000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f06d00d4a70 0x7f06d00d0880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60982/0 (socket says 172.21.15.47:60982) 2024-02-19T15:28:16.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.962+0000 7f06d5000640 1 -- 172.21.15.47:0/2651809747 learned_addr learned my addr 172.21.15.47:0/2651809747 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:16.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.962+0000 7f06d5a00640 1 --2- 172.21.15.47:0/2651809747 >> v2:172.21.15.116:3300/0 conn(0x7f06d015df50 0x7f06d00d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.962+0000 7f06cfe00640 1 -- 172.21.15.47:0/2651809747 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309476409 0 0) 0x7f06d00d2da0 con 0x7f06d00d4a70 2024-02-19T15:28:16.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.962+0000 7f06cfe00640 1 -- 172.21.15.47:0/2651809747 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06b8003530 con 0x7f06d00d4a70 2024-02-19T15:28:16.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.962+0000 7f06cfe00640 1 -- 172.21.15.47:0/2651809747 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423508289 0 0) 0x7f06d00b6df0 con 0x7f06d00d0f90 2024-02-19T15:28:16.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.962+0000 7f06cfe00640 1 -- 172.21.15.47:0/2651809747 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06d00d2da0 con 0x7f06d00d0f90 2024-02-19T15:28:16.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.966+0000 7f06d5a00640 1 -- 172.21.15.47:0/2651809747 >> v1:172.21.15.116:6789/0 conn(0x7f06d00d0f90 legacy=0x7f06d0167d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.966+0000 7f06d5a00640 1 -- 172.21.15.47:0/2651809747 >> v1:172.21.15.47:6789/0 conn(0x7f06d00d4a70 legacy=0x7f06d00d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.966+0000 7f06d5a00640 1 -- 172.21.15.47:0/2651809747 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d00b73b0 con 0x7f06d015df50 2024-02-19T15:28:16.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.966+0000 7f06d5a00640 1 --2- 172.21.15.47:0/2651809747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d015df50 0x7f06d00d2450 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f06c40097c0 tx=0x7f06c401c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=262853a12a7439fa server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:16.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.966+0000 7f06cfe00640 1 -- 172.21.15.47:0/2651809747 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c40ad680 con 0x7f06d015df50 2024-02-19T15:28:16.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.966+0000 7f06cfe00640 1 -- 172.21.15.47:0/2651809747 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c40ad7e0 con 0x7f06d015df50 2024-02-19T15:28:16.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.966+0000 7f06cfe00640 1 -- 172.21.15.47:0/2651809747 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c40adb70 con 0x7f06d015df50 2024-02-19T15:28:16.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.966+0000 7f06d8400640 1 -- 172.21.15.47:0/2651809747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d015df50 msgr2=0x7f06d00d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.966+0000 7f06d8400640 1 --2- 172.21.15.47:0/2651809747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d015df50 0x7f06d00d2450 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f06c40097c0 tx=0x7f06c401c470 comp rx=0 tx=0).stop 2024-02-19T15:28:16.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.966+0000 7f06d8400640 1 -- 172.21.15.47:0/2651809747 shutdown_connections 2024-02-19T15:28:16.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.966+0000 7f06d8400640 1 --2- 172.21.15.47:0/2651809747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d015df50 0x7f06d00d2450 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.966+0000 7f06d8400640 1 -- 172.21.15.47:0/2651809747 >> 172.21.15.47:0/2651809747 conn(0x7f06d00d3e70 msgr2=0x7f06d00d4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:16.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d8400640 1 -- 172.21.15.47:0/2651809747 shutdown_connections 2024-02-19T15:28:16.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d8400640 1 -- 172.21.15.47:0/2651809747 wait complete. 2024-02-19T15:28:16.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d8400640 1 Processor -- start 2024-02-19T15:28:16.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d8400640 1 -- start start 2024-02-19T15:28:16.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d8400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06d00d0f90 0x7f06d0152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d8400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d00d4a70 0x7f06d01600a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d8400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06d015df50 0x7f06d01510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d8400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d0153870 con 0x7f06d015df50 2024-02-19T15:28:16.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d8400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d0152c60 con 0x7f06d00d4a70 2024-02-19T15:28:16.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d8400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d0152f40 con 0x7f06d00d0f90 2024-02-19T15:28:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d6400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06d015df50 0x7f06d01510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d6400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06d015df50 0x7f06d01510e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39376/0 (socket says 172.21.15.47:39376) 2024-02-19T15:28:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d6400640 1 -- 172.21.15.47:0/652744356 learned_addr learned my addr 172.21.15.47:0/652744356 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d5a00640 1 --2- 172.21.15.47:0/652744356 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06d00d0f90 0x7f06d0152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d5000640 1 --2- 172.21.15.47:0/652744356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d00d4a70 0x7f06d01600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d6400640 1 -- 172.21.15.47:0/652744356 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06d00d0f90 msgr2=0x7f06d0152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d6400640 1 --2- 172.21.15.47:0/652744356 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06d00d0f90 0x7f06d0152650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d6400640 1 -- 172.21.15.47:0/652744356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d00d4a70 msgr2=0x7f06d01600a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d6400640 1 --2- 172.21.15.47:0/652744356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d00d4a70 0x7f06d01600a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d6400640 1 -- 172.21.15.47:0/652744356 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c4009420 con 0x7f06d015df50 2024-02-19T15:28:16.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d5000640 1 --2- 172.21.15.47:0/652744356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d00d4a70 0x7f06d01600a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:16.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06d6400640 1 --2- 172.21.15.47:0/652744356 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06d015df50 0x7f06d01510e0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f06c800c7a0 tx=0x7f06c800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:16.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.970+0000 7f06cd600640 1 -- 172.21.15.47:0/652744356 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c8003b80 con 0x7f06d015df50 2024-02-19T15:28:16.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.974+0000 7f06cd600640 1 -- 172.21.15.47:0/652744356 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c8003ce0 con 0x7f06d015df50 2024-02-19T15:28:16.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.974+0000 7f06cd600640 1 -- 172.21.15.47:0/652744356 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c800aa90 con 0x7f06d015df50 2024-02-19T15:28:16.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.974+0000 7f06d8400640 1 -- 172.21.15.47:0/652744356 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d0153220 con 0x7f06d015df50 2024-02-19T15:28:16.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.974+0000 7f06d8400640 1 -- 172.21.15.47:0/652744356 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d00d2710 con 0x7f06d015df50 2024-02-19T15:28:16.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.974+0000 7f06d8400640 1 -- 172.21.15.47:0/652744356 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d01539e0 con 0x7f06d015df50 2024-02-19T15:28:16.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.974+0000 7f06cd600640 1 -- 172.21.15.47:0/652744356 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f06c800abf0 con 0x7f06d015df50 2024-02-19T15:28:16.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.974+0000 7f06cd600640 1 --2- 172.21.15.47:0/652744356 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f06b0066e40 0x7f06b0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:16.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.974+0000 7f06cd600640 1 -- 172.21.15.47:0/652744356 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f06c8084a40 con 0x7f06d015df50 2024-02-19T15:28:16.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.974+0000 7f06d5a00640 1 --2- 172.21.15.47:0/652744356 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f06b0066e40 0x7f06b0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:16.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.974+0000 7f06d5a00640 1 --2- 172.21.15.47:0/652744356 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f06b0066e40 0x7f06b0069300 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f06c40af000 tx=0x7f06c4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:16.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:16.978+0000 7f06cd600640 1 -- 172.21.15.47:0/652744356 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f06c8051730 con 0x7f06d015df50 2024-02-19T15:28:17.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.182+0000 7f06d8400640 1 -- 172.21.15.47:0/652744356 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06d0075360 con 0x7f06d015df50 2024-02-19T15:28:17.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.182+0000 7f06cd600640 1 -- 172.21.15.47:0/652744356 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f06c8054df0 con 0x7f06d015df50 2024-02-19T15:28:17.183 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:17.184 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:17.184 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:17.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.186+0000 7f06d8400640 1 -- 172.21.15.47:0/652744356 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f06b0066e40 msgr2=0x7f06b0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:17.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.186+0000 7f06d8400640 1 --2- 172.21.15.47:0/652744356 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f06b0066e40 0x7f06b0069300 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f06c40af000 tx=0x7f06c4008040 comp rx=0 tx=0).stop 2024-02-19T15:28:17.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.186+0000 7f06d8400640 1 -- 172.21.15.47:0/652744356 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06d015df50 msgr2=0x7f06d01510e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:17.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.186+0000 7f06d8400640 1 --2- 172.21.15.47:0/652744356 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06d015df50 0x7f06d01510e0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f06c800c7a0 tx=0x7f06c800cc60 comp rx=0 tx=0).stop 2024-02-19T15:28:17.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.186+0000 7f06d8400640 1 -- 172.21.15.47:0/652744356 shutdown_connections 2024-02-19T15:28:17.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.186+0000 7f06d8400640 1 --2- 172.21.15.47:0/652744356 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f06b0066e40 0x7f06b0069300 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.186+0000 7f06d8400640 1 --2- 172.21.15.47:0/652744356 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06d015df50 0x7f06d01510e0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.186+0000 7f06d8400640 1 --2- 172.21.15.47:0/652744356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d00d4a70 0x7f06d01600a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.186+0000 7f06d8400640 1 --2- 172.21.15.47:0/652744356 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06d00d0f90 0x7f06d0152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.186+0000 7f06d8400640 1 -- 172.21.15.47:0/652744356 >> 172.21.15.47:0/652744356 conn(0x7f06d00d3e70 msgr2=0x7f06d006c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:17.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.186+0000 7f06d8400640 1 -- 172.21.15.47:0/652744356 shutdown_connections 2024-02-19T15:28:17.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.186+0000 7f06d8400640 1 -- 172.21.15.47:0/652744356 wait complete. 2024-02-19T15:28:17.208 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-02-19T15:28:17.387 INFO:teuthology.orchestra.run.smithi047.stdout:{"message":"","return_code":0} 2024-02-19T15:28:17.387 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-02-19T15:28:17.387 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-02-19T15:28:17.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f654d647080 1 Processor -- start 2024-02-19T15:28:17.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f654d647080 1 -- start start 2024-02-19T15:28:17.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f654d647080 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5608cd98e990 0x5608cd91d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:17.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f654d647080 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5608cd98ed70 0x5608cd991470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:17.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f654d647080 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608cd98f250 0x5608cd9ae350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:17.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f654d647080 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608cd885f90 con 0x5608cd98e990 2024-02-19T15:28:17.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f654d647080 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608cd81d1e0 con 0x5608cd98f250 2024-02-19T15:28:17.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f654d647080 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x5608cd81ce80 con 0x5608cd98ed70 2024-02-19T15:28:17.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f6549c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5608cd98e990 0x5608cd91d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:17.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f6549c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5608cd98e990 0x5608cd91d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39396/0 (socket says 172.21.15.47:39396) 2024-02-19T15:28:17.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f6549c00640 1 -- 172.21.15.47:0/2101620458 learned_addr learned my addr 172.21.15.47:0/2101620458 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:17.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f6549200640 1 --2- 172.21.15.47:0/2101620458 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5608cd98ed70 0x5608cd991470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:17.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f654a600640 1 --2- 172.21.15.47:0/2101620458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608cd98f250 0x5608cd9ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:17.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f6549c00640 1 -- 172.21.15.47:0/2101620458 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5608cd98ed70 msgr2=0x5608cd991470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:17.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f6549c00640 1 --2- 172.21.15.47:0/2101620458 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5608cd98ed70 0x5608cd991470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f6549c00640 1 -- 172.21.15.47:0/2101620458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608cd98f250 msgr2=0x5608cd9ae350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:17.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f6549c00640 1 --2- 172.21.15.47:0/2101620458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608cd98f250 0x5608cd9ae350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f6549c00640 1 -- 172.21.15.47:0/2101620458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6538005d30 con 0x5608cd98e990 2024-02-19T15:28:17.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f6549c00640 1 --2- 172.21.15.47:0/2101620458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5608cd98e990 0x5608cd91d580 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f653c00b2d0 tx=0x7f653c00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce3d2c1f6b4643a5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:17.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f6543400640 1 -- 172.21.15.47:0/2101620458 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c014070 con 0x5608cd98e990 2024-02-19T15:28:17.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f6543400640 1 -- 172.21.15.47:0/2101620458 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653c00d510 con 0x5608cd98e990 2024-02-19T15:28:17.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f6543400640 1 -- 172.21.15.47:0/2101620458 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c00d8c0 con 0x5608cd98e990 2024-02-19T15:28:17.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f654d647080 1 -- 172.21.15.47:0/2101620458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5608cd98e990 msgr2=0x5608cd91d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:17.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f654d647080 1 --2- 172.21.15.47:0/2101620458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5608cd98e990 0x5608cd91d580 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f653c00b2d0 tx=0x7f653c00b790 comp rx=0 tx=0).stop 2024-02-19T15:28:17.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f654d647080 1 -- 172.21.15.47:0/2101620458 shutdown_connections 2024-02-19T15:28:17.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f654d647080 1 --2- 172.21.15.47:0/2101620458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608cd98f250 0x5608cd9ae350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f654d647080 1 --2- 172.21.15.47:0/2101620458 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5608cd98ed70 0x5608cd991470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f654d647080 1 --2- 172.21.15.47:0/2101620458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5608cd98e990 0x5608cd91d580 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f654d647080 1 -- 172.21.15.47:0/2101620458 >> 172.21.15.47:0/2101620458 conn(0x5608cd8d6670 msgr2=0x5608cd916ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:17.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f654d647080 1 -- 172.21.15.47:0/2101620458 shutdown_connections 2024-02-19T15:28:17.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.434+0000 7f654d647080 1 -- 172.21.15.47:0/2101620458 wait complete. 2024-02-19T15:28:17.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f654d647080 1 Processor -- start 2024-02-19T15:28:17.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f654d647080 1 -- start start 2024-02-19T15:28:17.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f654d647080 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5608cd98e990 0x5608cd9af450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:17.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f654d647080 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5608cd98ed70 0x5608cd9b0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:17.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f654d647080 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608cd98f250 0x5608cd916ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:17.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f654d647080 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608cd840610 con 0x5608cd98ed70 2024-02-19T15:28:17.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f654d647080 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608cd810740 con 0x5608cd98f250 2024-02-19T15:28:17.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f654d647080 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x5608cd9b1fb0 con 0x5608cd98e990 2024-02-19T15:28:17.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f6549200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5608cd98ed70 0x5608cd9b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:17.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f6549200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5608cd98ed70 0x5608cd9b0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39412/0 (socket says 172.21.15.47:39412) 2024-02-19T15:28:17.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f6549200640 1 -- 172.21.15.47:0/4219125624 learned_addr learned my addr 172.21.15.47:0/4219125624 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:17.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f6549c00640 1 --2- 172.21.15.47:0/4219125624 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5608cd98e990 0x5608cd9af450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:17.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f654a600640 1 --2- 172.21.15.47:0/4219125624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608cd98f250 0x5608cd916ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:17.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f6549200640 1 -- 172.21.15.47:0/4219125624 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5608cd98e990 msgr2=0x5608cd9af450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:17.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f6549200640 1 --2- 172.21.15.47:0/4219125624 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x5608cd98e990 0x5608cd9af450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f6549200640 1 -- 172.21.15.47:0/4219125624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608cd98f250 msgr2=0x5608cd916ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:17.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f6549200640 1 --2- 172.21.15.47:0/4219125624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608cd98f250 0x5608cd916ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f6549200640 1 -- 172.21.15.47:0/4219125624 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653c001bb0 con 0x5608cd98ed70 2024-02-19T15:28:17.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f6549200640 1 --2- 172.21.15.47:0/4219125624 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5608cd98ed70 0x5608cd9b0030 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f65380ad040 tx=0x7f6538018030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:17.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f6542000640 1 -- 172.21.15.47:0/4219125624 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6538009dc0 con 0x5608cd98ed70 2024-02-19T15:28:17.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f6542000640 1 -- 172.21.15.47:0/4219125624 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65380bd400 con 0x5608cd98ed70 2024-02-19T15:28:17.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f654d647080 1 -- 172.21.15.47:0/4219125624 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5608cd81bf30 con 0x5608cd98ed70 2024-02-19T15:28:17.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f6542000640 1 -- 172.21.15.47:0/4219125624 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65380bd770 con 0x5608cd98ed70 2024-02-19T15:28:17.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.438+0000 7f654d647080 1 -- 172.21.15.47:0/4219125624 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5608cd81b990 con 0x5608cd98ed70 2024-02-19T15:28:17.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.442+0000 7f6542000640 1 -- 172.21.15.47:0/4219125624 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f65380b1040 con 0x5608cd98ed70 2024-02-19T15:28:17.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.442+0000 7f6542000640 1 --2- 172.21.15.47:0/4219125624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f652c065850 0x7f652c067d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:17.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.442+0000 7f6542000640 1 -- 172.21.15.47:0/4219125624 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f653811e910 con 0x5608cd98ed70 2024-02-19T15:28:17.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.442+0000 7f654d647080 1 --2- 172.21.15.47:0/4219125624 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x5608cd917400 0x5608cda49d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:17.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.442+0000 7f654d647080 1 -- 172.21.15.47:0/4219125624 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v8 -- 0x5608cd9b28e0 con 0x5608cd917400 2024-02-19T15:28:17.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.442+0000 7f6549c00640 1 --2- 172.21.15.47:0/4219125624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f652c065850 0x7f652c067d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:17.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.442+0000 7f654a600640 1 --2- 172.21.15.47:0/4219125624 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x5608cd917400 0x5608cda49d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:17.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.442+0000 7f654a600640 1 --2- 172.21.15.47:0/4219125624 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x5608cd917400 0x5608cda49d40 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:17.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.442+0000 7f6549c00640 1 --2- 172.21.15.47:0/4219125624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f652c065850 0x7f652c067d10 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f653c00b2d0 tx=0x7f653c009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:17.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.442+0000 7f654a600640 1 -- 172.21.15.47:0/4219125624 <== osd.0 v2:172.21.15.47:6809/1968337531 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (99) out=99b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+99 (crc 0 0 0) 0x7f6544002590 con 0x5608cd917400 2024-02-19T15:28:17.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.442+0000 7f654d647080 1 -- 172.21.15.47:0/4219125624 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x5608cd917400 msgr2=0x5608cda49d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:17.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.442+0000 7f654d647080 1 --2- 172.21.15.47:0/4219125624 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x5608cd917400 0x5608cda49d40 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.442+0000 7f654d647080 1 -- 172.21.15.47:0/4219125624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f652c065850 msgr2=0x7f652c067d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:17.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.446+0000 7f654d647080 1 --2- 172.21.15.47:0/4219125624 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f652c065850 0x7f652c067d10 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f653c00b2d0 tx=0x7f653c009000 comp rx=0 tx=0).stop 2024-02-19T15:28:17.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.446+0000 7f654d647080 1 -- 172.21.15.47:0/4219125624 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5608cd98ed70 msgr2=0x5608cd9b0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:17.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.446+0000 7f654d647080 1 --2- 172.21.15.47:0/4219125624 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5608cd98ed70 0x5608cd9b0030 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f65380ad040 tx=0x7f6538018030 comp rx=0 tx=0).stop 2024-02-19T15:28:17.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.446+0000 7f654a600640 1 -- 172.21.15.47:0/4219125624 reap_dead start 2024-02-19T15:28:17.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.446+0000 7f654d647080 1 -- 172.21.15.47:0/4219125624 shutdown_connections 2024-02-19T15:28:17.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.446+0000 7f654d647080 1 -- 172.21.15.47:0/4219125624 >> 172.21.15.47:0/4219125624 conn(0x5608cd8d6670 msgr2=0x5608cd901c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:17.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.446+0000 7f654d647080 1 -- 172.21.15.47:0/4219125624 shutdown_connections 2024-02-19T15:28:17.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.446+0000 7f654d647080 1 -- 172.21.15.47:0/4219125624 wait complete. 2024-02-19T15:28:17.449 INFO:teuthology.orchestra.run.smithi047.stdout:] 10000000001 stray0 2024-02-19T15:28:17.450 DEBUG:teuthology.orchestra.run.smithi047:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-02-19T15:28:17.497 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:28:17.497 INFO:teuthology.orchestra.run.smithi047.stdout: "ino": 1099511627777, 2024-02-19T15:28:17.497 INFO:teuthology.orchestra.run.smithi047.stdout: "ancestors": [ 2024-02-19T15:28:17.497 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:17.497 INFO:teuthology.orchestra.run.smithi047.stdout: "dirino": 1536, 2024-02-19T15:28:17.497 INFO:teuthology.orchestra.run.smithi047.stdout: "dname": "10000000001", 2024-02-19T15:28:17.497 INFO:teuthology.orchestra.run.smithi047.stdout: "version": 13 2024-02-19T15:28:17.498 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:17.498 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:17.498 INFO:teuthology.orchestra.run.smithi047.stdout: "dirino": 256, 2024-02-19T15:28:17.498 INFO:teuthology.orchestra.run.smithi047.stdout: "dname": "stray0", 2024-02-19T15:28:17.498 INFO:teuthology.orchestra.run.smithi047.stdout: "version": 5 2024-02-19T15:28:17.498 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:17.498 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-02-19T15:28:17.498 INFO:teuthology.orchestra.run.smithi047.stdout: "pool": 17, 2024-02-19T15:28:17.498 INFO:teuthology.orchestra.run.smithi047.stdout: "old_pools": [] 2024-02-19T15:28:17.498 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:28:17.498 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:17.503 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:17.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.686+0000 7f1e22a00640 1 Processor -- start 2024-02-19T15:28:17.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.686+0000 7f1e22a00640 1 -- start start 2024-02-19T15:28:17.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.686+0000 7f1e22a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1e1c1456a0 0x7f1e1c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:17.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.686+0000 7f1e22a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e1c0cf4a0 0x7f1e1c06aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:17.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.686+0000 7f1e22a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1e1c06b3f0 0x7f1e1c15f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:17.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.686+0000 7f1e22a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e1c0d03c0 con 0x7f1e1c1456a0 2024-02-19T15:28:17.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.686+0000 7f1e22a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e1c0cf900 con 0x7f1e1c06b3f0 2024-02-19T15:28:17.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.686+0000 7f1e22a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c06b850 con 0x7f1e1c0cf4a0 2024-02-19T15:28:17.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.686+0000 7f1e1be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1e1c1456a0 0x7f1e1c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:17.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.686+0000 7f1e1be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1e1c1456a0 0x7f1e1c0cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39428/0 (socket says 172.21.15.47:39428) 2024-02-19T15:28:17.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.686+0000 7f1e1be00640 1 -- 172.21.15.47:0/1416794374 learned_addr learned my addr 172.21.15.47:0/1416794374 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:17.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.690+0000 7f1e1be00640 1 -- 172.21.15.47:0/1416794374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e1c0cf4a0 msgr2=0x7f1e1c06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:17.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.690+0000 7f1e1be00640 1 --2- 172.21.15.47:0/1416794374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e1c0cf4a0 0x7f1e1c06aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.690+0000 7f1e1be00640 1 -- 172.21.15.47:0/1416794374 >> v2:172.21.15.116:3300/0 conn(0x7f1e1c06b3f0 msgr2=0x7f1e1c15f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:17.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.690+0000 7f1e1be00640 1 --2- 172.21.15.47:0/1416794374 >> v2:172.21.15.116:3300/0 conn(0x7f1e1c06b3f0 0x7f1e1c15f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.690+0000 7f1e1be00640 1 -- 172.21.15.47:0/1416794374 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1c145a80 con 0x7f1e1c1456a0 2024-02-19T15:28:17.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.690+0000 7f1e1be00640 1 --2- 172.21.15.47:0/1416794374 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e1c1456a0 0x7f1e1c0cef60 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c067cb0 tx=0x7f1e0c01c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e521ac2ef6db8fc1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:17.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.690+0000 7f1e1b400640 1 -- 172.21.15.47:0/1416794374 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c0ab600 con 0x7f1e1c1456a0 2024-02-19T15:28:17.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.690+0000 7f1e1b400640 1 -- 172.21.15.47:0/1416794374 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0c0a7050 con 0x7f1e1c1456a0 2024-02-19T15:28:17.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.690+0000 7f1e1b400640 1 -- 172.21.15.47:0/1416794374 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c0ab980 con 0x7f1e1c1456a0 2024-02-19T15:28:17.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.690+0000 7f1e22a00640 1 -- 172.21.15.47:0/1416794374 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e1c1456a0 msgr2=0x7f1e1c0cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:17.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.690+0000 7f1e22a00640 1 --2- 172.21.15.47:0/1416794374 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e1c1456a0 0x7f1e1c0cef60 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c067cb0 tx=0x7f1e0c01c260 comp rx=0 tx=0).stop 2024-02-19T15:28:17.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.690+0000 7f1e22a00640 1 -- 172.21.15.47:0/1416794374 shutdown_connections 2024-02-19T15:28:17.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.690+0000 7f1e22a00640 1 --2- 172.21.15.47:0/1416794374 >> v2:172.21.15.116:3300/0 conn(0x7f1e1c06b3f0 0x7f1e1c15f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.690+0000 7f1e22a00640 1 --2- 172.21.15.47:0/1416794374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e1c0cf4a0 0x7f1e1c06aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.690+0000 7f1e22a00640 1 --2- 172.21.15.47:0/1416794374 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e1c1456a0 0x7f1e1c0cef60 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.690+0000 7f1e22a00640 1 -- 172.21.15.47:0/1416794374 >> 172.21.15.47:0/1416794374 conn(0x7f1e1c0d0ad0 msgr2=0x7f1e1c0d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:17.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e22a00640 1 -- 172.21.15.47:0/1416794374 shutdown_connections 2024-02-19T15:28:17.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e22a00640 1 -- 172.21.15.47:0/1416794374 wait complete. 2024-02-19T15:28:17.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e22a00640 1 Processor -- start 2024-02-19T15:28:17.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e22a00640 1 -- start start 2024-02-19T15:28:17.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e22a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e1c06b3f0 0x7f1e1c15a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:17.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e22a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e1c0cf4a0 0x7f1e1c159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:17.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e22a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e1c1456a0 0x7f1e1c161230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:17.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e22a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c1443c0 con 0x7f1e1c1456a0 2024-02-19T15:28:17.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e22a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c06f3f0 con 0x7f1e1c06b3f0 2024-02-19T15:28:17.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e22a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c06f6d0 con 0x7f1e1c0cf4a0 2024-02-19T15:28:17.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e20a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e1c1456a0 0x7f1e1c161230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:17.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e20a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e1c1456a0 0x7f1e1c161230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39432/0 (socket says 172.21.15.47:39432) 2024-02-19T15:28:17.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e20a00640 1 -- 172.21.15.47:0/2541685285 learned_addr learned my addr 172.21.15.47:0/2541685285 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:17.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e1be00640 1 --2- 172.21.15.47:0/2541685285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e1c06b3f0 0x7f1e1c15a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:17.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e13e00640 1 --2- 172.21.15.47:0/2541685285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e1c0cf4a0 0x7f1e1c159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:17.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e20a00640 1 -- 172.21.15.47:0/2541685285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e1c0cf4a0 msgr2=0x7f1e1c159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:17.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e20a00640 1 --2- 172.21.15.47:0/2541685285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e1c0cf4a0 0x7f1e1c159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e20a00640 1 -- 172.21.15.47:0/2541685285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e1c06b3f0 msgr2=0x7f1e1c15a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:17.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e20a00640 1 --2- 172.21.15.47:0/2541685285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e1c06b3f0 0x7f1e1c15a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e20a00640 1 -- 172.21.15.47:0/2541685285 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0c0094d0 con 0x7f1e1c1456a0 2024-02-19T15:28:17.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e20a00640 1 --2- 172.21.15.47:0/2541685285 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e1c1456a0 0x7f1e1c161230 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f1e1400b970 tx=0x7f1e1400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:17.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e18c00640 1 -- 172.21.15.47:0/2541685285 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1400cb40 con 0x7f1e1c1456a0 2024-02-19T15:28:17.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e18c00640 1 -- 172.21.15.47:0/2541685285 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1400ccc0 con 0x7f1e1c1456a0 2024-02-19T15:28:17.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.694+0000 7f1e18c00640 1 -- 172.21.15.47:0/2541685285 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e14014680 con 0x7f1e1c1456a0 2024-02-19T15:28:17.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.698+0000 7f1e22a00640 1 -- 172.21.15.47:0/2541685285 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e1c06f9b0 con 0x7f1e1c1456a0 2024-02-19T15:28:17.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.698+0000 7f1e22a00640 1 -- 172.21.15.47:0/2541685285 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e1c15a220 con 0x7f1e1c1456a0 2024-02-19T15:28:17.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.698+0000 7f1e22a00640 1 -- 172.21.15.47:0/2541685285 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e1c144530 con 0x7f1e1c1456a0 2024-02-19T15:28:17.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.698+0000 7f1e18c00640 1 -- 172.21.15.47:0/2541685285 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1e1401d3b0 con 0x7f1e1c1456a0 2024-02-19T15:28:17.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.698+0000 7f1e18c00640 1 --2- 172.21.15.47:0/2541685285 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1df8066e40 0x7f1df8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:17.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.698+0000 7f1e18c00640 1 -- 172.21.15.47:0/2541685285 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f1e140849f0 con 0x7f1e1c1456a0 2024-02-19T15:28:17.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.698+0000 7f1e1be00640 1 --2- 172.21.15.47:0/2541685285 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1df8066e40 0x7f1df8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:17.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.702+0000 7f1e1be00640 1 --2- 172.21.15.47:0/2541685285 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1df8066e40 0x7f1df8069300 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c0ad000 tx=0x7f1e0c0a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:17.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.702+0000 7f1e18c00640 1 -- 172.21.15.47:0/2541685285 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1e14051790 con 0x7f1e1c1456a0 2024-02-19T15:28:17.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.898+0000 7f1e22a00640 1 -- 172.21.15.47:0/2541685285 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e1c075360 con 0x7f1e1c1456a0 2024-02-19T15:28:17.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.902+0000 7f1e18c00640 1 -- 172.21.15.47:0/2541685285 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f1e14019020 con 0x7f1e1c1456a0 2024-02-19T15:28:17.901 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:17.901 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:17.901 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:17.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.902+0000 7f1e22a00640 1 -- 172.21.15.47:0/2541685285 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1df8066e40 msgr2=0x7f1df8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:17.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.902+0000 7f1e22a00640 1 --2- 172.21.15.47:0/2541685285 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1df8066e40 0x7f1df8069300 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c0ad000 tx=0x7f1e0c0a6000 comp rx=0 tx=0).stop 2024-02-19T15:28:17.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.902+0000 7f1e22a00640 1 -- 172.21.15.47:0/2541685285 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e1c1456a0 msgr2=0x7f1e1c161230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:17.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.902+0000 7f1e22a00640 1 --2- 172.21.15.47:0/2541685285 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e1c1456a0 0x7f1e1c161230 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f1e1400b970 tx=0x7f1e1400be30 comp rx=0 tx=0).stop 2024-02-19T15:28:17.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.902+0000 7f1e22a00640 1 -- 172.21.15.47:0/2541685285 shutdown_connections 2024-02-19T15:28:17.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.902+0000 7f1e22a00640 1 --2- 172.21.15.47:0/2541685285 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1df8066e40 0x7f1df8069300 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.902+0000 7f1e22a00640 1 --2- 172.21.15.47:0/2541685285 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e1c1456a0 0x7f1e1c161230 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.902+0000 7f1e22a00640 1 --2- 172.21.15.47:0/2541685285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e1c0cf4a0 0x7f1e1c159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.902+0000 7f1e22a00640 1 --2- 172.21.15.47:0/2541685285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e1c06b3f0 0x7f1e1c15a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:17.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.902+0000 7f1e22a00640 1 -- 172.21.15.47:0/2541685285 >> 172.21.15.47:0/2541685285 conn(0x7f1e1c0d0ad0 msgr2=0x7f1e1c15af20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:17.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.902+0000 7f1e22a00640 1 -- 172.21.15.47:0/2541685285 shutdown_connections 2024-02-19T15:28:17.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:17.902+0000 7f1e22a00640 1 -- 172.21.15.47:0/2541685285 wait complete. 2024-02-19T15:28:17.933 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_reintegrated 2024-02-19T15:28:18.121 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"strays_reintegrated":0}} 2024-02-19T15:28:18.121 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 0 } } 2024-02-19T15:28:18.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:28:18.121 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec mv dir_2/file_b dir_2/file_c) 2024-02-19T15:28:18.139 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:18.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.322+0000 7f9afe000640 1 Processor -- start 2024-02-19T15:28:18.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.322+0000 7f9afe000640 1 -- start start 2024-02-19T15:28:18.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.322+0000 7f9afe000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9af806fdf0 0x7f9af80adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:18.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.322+0000 7f9afe000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af8161780 con 0x7f9af81613a0 2024-02-19T15:28:18.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.322+0000 7f9afe000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af80b01a0 con 0x7f9af806e7c0 2024-02-19T15:28:18.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.322+0000 7f9afe000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af80ae3c0 con 0x7f9af806fdf0 2024-02-19T15:28:18.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.322+0000 7f9af7400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f9af81613a0 0x7f9af806e0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:32768/0 (socket says 172.21.15.47:32768) 2024-02-19T15:28:18.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.322+0000 7f9af7400640 1 -- 172.21.15.47:0/1384128090 learned_addr learned my addr 172.21.15.47:0/1384128090 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:18.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.326+0000 7f9aefe00640 1 --2- 172.21.15.47:0/1384128090 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9af806fdf0 0x7f9af80adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:18.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.326+0000 7f9af6a00640 1 -- 172.21.15.47:0/1384128090 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232046330 0 0) 0x7f9af8161780 con 0x7f9af81613a0 2024-02-19T15:28:18.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.326+0000 7f9af6a00640 1 -- 172.21.15.47:0/1384128090 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9adc003530 con 0x7f9af81613a0 2024-02-19T15:28:18.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.326+0000 7f9aefe00640 1 -- 172.21.15.47:0/1384128090 >> v1:172.21.15.116:6789/0 conn(0x7f9af806e7c0 legacy=0x7f9af806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:18.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.326+0000 7f9af6a00640 1 -- 172.21.15.47:0/1384128090 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800644792 0 0) 0x7f9af80b01a0 con 0x7f9af806e7c0 2024-02-19T15:28:18.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.326+0000 7f9aefe00640 1 -- 172.21.15.47:0/1384128090 >> v1:172.21.15.47:6789/0 conn(0x7f9af81613a0 legacy=0x7f9af806e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:18.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.326+0000 7f9aefe00640 1 -- 172.21.15.47:0/1384128090 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af80ae6a0 con 0x7f9af806fdf0 2024-02-19T15:28:18.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.326+0000 7f9aefe00640 1 --2- 172.21.15.47:0/1384128090 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9af806fdf0 0x7f9af80adbb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9ae00097f0 tx=0x7f9ae001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1b9c0b38d50e597 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:18.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.326+0000 7f9af6a00640 1 -- 172.21.15.47:0/1384128090 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae00ad660 con 0x7f9af806fdf0 2024-02-19T15:28:18.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.326+0000 7f9af6a00640 1 -- 172.21.15.47:0/1384128090 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae00ad7c0 con 0x7f9af806fdf0 2024-02-19T15:28:18.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.326+0000 7f9af6a00640 1 -- 172.21.15.47:0/1384128090 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae00adb50 con 0x7f9af806fdf0 2024-02-19T15:28:18.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.330+0000 7f9afe000640 1 -- 172.21.15.47:0/1384128090 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9af806fdf0 msgr2=0x7f9af80adbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:18.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.330+0000 7f9afe000640 1 --2- 172.21.15.47:0/1384128090 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9af806fdf0 0x7f9af80adbb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9ae00097f0 tx=0x7f9ae001c450 comp rx=0 tx=0).stop 2024-02-19T15:28:18.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.330+0000 7f9afe000640 1 -- 172.21.15.47:0/1384128090 shutdown_connections 2024-02-19T15:28:18.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.330+0000 7f9afe000640 1 --2- 172.21.15.47:0/1384128090 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9af806fdf0 0x7f9af80adbb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:18.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.330+0000 7f9afe000640 1 -- 172.21.15.47:0/1384128090 >> 172.21.15.47:0/1384128090 conn(0x7f9af80d0ad0 msgr2=0x7f9af80d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:18.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.330+0000 7f9afe000640 1 -- 172.21.15.47:0/1384128090 shutdown_connections 2024-02-19T15:28:18.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.330+0000 7f9afe000640 1 -- 172.21.15.47:0/1384128090 wait complete. 2024-02-19T15:28:18.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.330+0000 7f9afe000640 1 Processor -- start 2024-02-19T15:28:18.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.330+0000 7f9afe000640 1 -- start start 2024-02-19T15:28:18.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.330+0000 7f9afe000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9af806e7c0 0x7f9af81442f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:18.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.330+0000 7f9afe000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9af806fdf0 0x7f9af8151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:18.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.330+0000 7f9afe000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9af81613a0 0x7f9af81524f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:18.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.330+0000 7f9afe000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af8150a80 con 0x7f9af81613a0 2024-02-19T15:28:18.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.330+0000 7f9afe000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af8150d60 con 0x7f9af806fdf0 2024-02-19T15:28:18.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.330+0000 7f9afe000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af8151040 con 0x7f9af806e7c0 2024-02-19T15:28:18.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.330+0000 7f9af7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9af81613a0 0x7f9af81524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:18.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.330+0000 7f9af7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9af81613a0 0x7f9af81524f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39448/0 (socket says 172.21.15.47:39448) 2024-02-19T15:28:18.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.330+0000 7f9af7e00640 1 -- 172.21.15.47:0/3967675177 learned_addr learned my addr 172.21.15.47:0/3967675177 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:18.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.330+0000 7f9af7400640 1 --2- 172.21.15.47:0/3967675177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9af806e7c0 0x7f9af81442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:18.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.330+0000 7f9aefe00640 1 --2- 172.21.15.47:0/3967675177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9af806fdf0 0x7f9af8151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:18.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.334+0000 7f9af7e00640 1 -- 172.21.15.47:0/3967675177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9af806e7c0 msgr2=0x7f9af81442f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:18.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.334+0000 7f9af7e00640 1 --2- 172.21.15.47:0/3967675177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9af806e7c0 0x7f9af81442f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:18.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.334+0000 7f9af7e00640 1 -- 172.21.15.47:0/3967675177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9af806fdf0 msgr2=0x7f9af8151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:18.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.334+0000 7f9af7e00640 1 --2- 172.21.15.47:0/3967675177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9af806fdf0 0x7f9af8151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:18.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.334+0000 7f9af7e00640 1 -- 172.21.15.47:0/3967675177 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae0009420 con 0x7f9af81613a0 2024-02-19T15:28:18.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.334+0000 7f9af7e00640 1 --2- 172.21.15.47:0/3967675177 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9af81613a0 0x7f9af81524f0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f9af000c780 tx=0x7f9af000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:18.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.334+0000 7f9aef400640 1 -- 172.21.15.47:0/3967675177 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af0003db0 con 0x7f9af81613a0 2024-02-19T15:28:18.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.334+0000 7f9aef400640 1 -- 172.21.15.47:0/3967675177 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af000ce90 con 0x7f9af81613a0 2024-02-19T15:28:18.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.334+0000 7f9afe000640 1 -- 172.21.15.47:0/3967675177 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af8151320 con 0x7f9af81613a0 2024-02-19T15:28:18.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.334+0000 7f9aef400640 1 -- 172.21.15.47:0/3967675177 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af00052b0 con 0x7f9af81613a0 2024-02-19T15:28:18.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.334+0000 7f9afe000640 1 -- 172.21.15.47:0/3967675177 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af8144830 con 0x7f9af81613a0 2024-02-19T15:28:18.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.334+0000 7f9afe000640 1 -- 172.21.15.47:0/3967675177 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af8075360 con 0x7f9af81613a0 2024-02-19T15:28:18.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.334+0000 7f9aef400640 1 -- 172.21.15.47:0/3967675177 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9af0005410 con 0x7f9af81613a0 2024-02-19T15:28:18.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.334+0000 7f9aef400640 1 --2- 172.21.15.47:0/3967675177 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9ad4066e40 0x7f9ad4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:18.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.338+0000 7f9aef400640 1 -- 172.21.15.47:0/3967675177 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f9af00847e0 con 0x7f9af81613a0 2024-02-19T15:28:18.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.338+0000 7f9af7400640 1 --2- 172.21.15.47:0/3967675177 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9ad4066e40 0x7f9ad4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:18.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.338+0000 7f9af7400640 1 --2- 172.21.15.47:0/3967675177 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9ad4066e40 0x7f9ad4069300 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8007fd0 tx=0x7f9ae8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:18.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.346+0000 7f9aef400640 1 -- 172.21.15.47:0/3967675177 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f9af0051620 con 0x7f9af81613a0 2024-02-19T15:28:18.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.594+0000 7f9afe000640 1 -- 172.21.15.47:0/3967675177 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9af80d15f0 con 0x7f9af81613a0 2024-02-19T15:28:18.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.594+0000 7f9aef400640 1 -- 172.21.15.47:0/3967675177 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f9af0054ce0 con 0x7f9af81613a0 2024-02-19T15:28:18.596 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:18.596 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:18.597 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:18.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.598+0000 7f9afe000640 1 -- 172.21.15.47:0/3967675177 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9ad4066e40 msgr2=0x7f9ad4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:18.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.598+0000 7f9afe000640 1 --2- 172.21.15.47:0/3967675177 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9ad4066e40 0x7f9ad4069300 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8007fd0 tx=0x7f9ae8009040 comp rx=0 tx=0).stop 2024-02-19T15:28:18.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.598+0000 7f9afe000640 1 -- 172.21.15.47:0/3967675177 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9af81613a0 msgr2=0x7f9af81524f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:18.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.598+0000 7f9afe000640 1 --2- 172.21.15.47:0/3967675177 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9af81613a0 0x7f9af81524f0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f9af000c780 tx=0x7f9af000cc40 comp rx=0 tx=0).stop 2024-02-19T15:28:18.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.598+0000 7f9afe000640 1 -- 172.21.15.47:0/3967675177 shutdown_connections 2024-02-19T15:28:18.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.598+0000 7f9afe000640 1 --2- 172.21.15.47:0/3967675177 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9ad4066e40 0x7f9ad4069300 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:18.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.598+0000 7f9afe000640 1 --2- 172.21.15.47:0/3967675177 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9af81613a0 0x7f9af81524f0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:18.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.598+0000 7f9afe000640 1 --2- 172.21.15.47:0/3967675177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9af806fdf0 0x7f9af8151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:18.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.598+0000 7f9afe000640 1 --2- 172.21.15.47:0/3967675177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9af806e7c0 0x7f9af81442f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:18.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.598+0000 7f9afe000640 1 -- 172.21.15.47:0/3967675177 >> 172.21.15.47:0/3967675177 conn(0x7f9af80d0ad0 msgr2=0x7f9af806dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:18.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.598+0000 7f9afe000640 1 -- 172.21.15.47:0/3967675177 shutdown_connections 2024-02-19T15:28:18.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.598+0000 7f9afe000640 1 -- 172.21.15.47:0/3967675177 wait complete. 2024-02-19T15:28:18.627 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-02-19T15:28:18.787 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"num_strays":0}} 2024-02-19T15:28:18.788 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-02-19T15:28:18.788 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:18.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.926+0000 7fd956400640 1 Processor -- start 2024-02-19T15:28:18.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.926+0000 7fd956400640 1 -- start start 2024-02-19T15:28:18.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.926+0000 7fd956400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd95015bcd0 0x7fd9500cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:18.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.926+0000 7fd956400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd9500d0360 0x7fd95015d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:18.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.926+0000 7fd956400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9500cc700 con 0x7fd9500cedb0 2024-02-19T15:28:18.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.926+0000 7fd956400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9500cf5e0 con 0x7fd95015bcd0 2024-02-19T15:28:18.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.926+0000 7fd956400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9500cf300 con 0x7fd9500d0360 2024-02-19T15:28:18.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.926+0000 7fd94f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd95015bcd0 0x7fd9500cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:18.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.930+0000 7fd946a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd9500cedb0 0x7fd9500cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:32780/0 (socket says 172.21.15.47:32780) 2024-02-19T15:28:18.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.930+0000 7fd946a00640 1 -- 172.21.15.47:0/3385646223 learned_addr learned my addr 172.21.15.47:0/3385646223 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:18.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.930+0000 7fd94fe00640 1 --2- 172.21.15.47:0/3385646223 >> v2:172.21.15.116:3300/0 conn(0x7fd9500d0360 0x7fd95015d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:18.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.930+0000 7fd94f400640 1 -- 172.21.15.47:0/3385646223 >> v2:172.21.15.116:3300/0 conn(0x7fd9500d0360 msgr2=0x7fd95015d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:18.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.930+0000 7fd94f400640 1 --2- 172.21.15.47:0/3385646223 >> v2:172.21.15.116:3300/0 conn(0x7fd9500d0360 0x7fd95015d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:18.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.930+0000 7fd94f400640 1 -- 172.21.15.47:0/3385646223 >> v1:172.21.15.47:6789/0 conn(0x7fd9500cedb0 legacy=0x7fd9500cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:18.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.930+0000 7fd94f400640 1 -- 172.21.15.47:0/3385646223 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9500cc9a0 con 0x7fd95015bcd0 2024-02-19T15:28:18.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.930+0000 7fd94fe00640 1 --2- 172.21.15.47:0/3385646223 >> v2:172.21.15.116:3300/0 conn(0x7fd9500d0360 0x7fd95015d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:18.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.930+0000 7fd94ea00640 1 -- 172.21.15.47:0/3385646223 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391421835 0 0) 0x7fd9500cc700 con 0x7fd9500cedb0 2024-02-19T15:28:18.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.930+0000 7fd94f400640 1 --2- 172.21.15.47:0/3385646223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd95015bcd0 0x7fd9500cd480 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fd9500b7c80 tx=0x7fd94001c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b41707e1d91076 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:18.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.930+0000 7fd94ea00640 1 -- 172.21.15.47:0/3385646223 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9400a59c0 con 0x7fd95015bcd0 2024-02-19T15:28:18.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.930+0000 7fd94ea00640 1 -- 172.21.15.47:0/3385646223 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9400a5b20 con 0x7fd95015bcd0 2024-02-19T15:28:18.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.930+0000 7fd94ea00640 1 -- 172.21.15.47:0/3385646223 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9400ad440 con 0x7fd95015bcd0 2024-02-19T15:28:18.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd956400640 1 -- 172.21.15.47:0/3385646223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd95015bcd0 msgr2=0x7fd9500cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:18.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd956400640 1 --2- 172.21.15.47:0/3385646223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd95015bcd0 0x7fd9500cd480 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fd9500b7c80 tx=0x7fd94001c900 comp rx=0 tx=0).stop 2024-02-19T15:28:18.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd956400640 1 -- 172.21.15.47:0/3385646223 shutdown_connections 2024-02-19T15:28:18.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd956400640 1 --2- 172.21.15.47:0/3385646223 >> v2:172.21.15.116:3300/0 conn(0x7fd9500d0360 0x7fd95015d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:18.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd956400640 1 --2- 172.21.15.47:0/3385646223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd95015bcd0 0x7fd9500cd480 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:18.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd956400640 1 -- 172.21.15.47:0/3385646223 >> 172.21.15.47:0/3385646223 conn(0x7fd9500d1bf0 msgr2=0x7fd9500d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:18.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd956400640 1 -- 172.21.15.47:0/3385646223 shutdown_connections 2024-02-19T15:28:18.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd956400640 1 -- 172.21.15.47:0/3385646223 wait complete. 2024-02-19T15:28:18.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd956400640 1 Processor -- start 2024-02-19T15:28:18.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd956400640 1 -- start start 2024-02-19T15:28:18.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd956400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9500cedb0 0x7fd9500cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:18.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd956400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9500d0360 0x7fd950147e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:18.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd956400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd95015bcd0 0x7fd9500c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:18.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd956400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9500ca070 con 0x7fd9500cedb0 2024-02-19T15:28:18.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd956400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95006dfd0 con 0x7fd95015bcd0 2024-02-19T15:28:18.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd956400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9500cc000 con 0x7fd9500d0360 2024-02-19T15:28:18.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd94f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9500cedb0 0x7fd9500cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:18.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd94f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9500cedb0 0x7fd9500cbac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39474/0 (socket says 172.21.15.47:39474) 2024-02-19T15:28:18.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd94f400640 1 -- 172.21.15.47:0/3346593891 learned_addr learned my addr 172.21.15.47:0/3346593891 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:18.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd946a00640 1 --2- 172.21.15.47:0/3346593891 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9500d0360 0x7fd950147e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:18.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd94f400640 1 -- 172.21.15.47:0/3346593891 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9500d0360 msgr2=0x7fd950147e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:18.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd94fe00640 1 --2- 172.21.15.47:0/3346593891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd95015bcd0 0x7fd9500c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:18.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd94f400640 1 --2- 172.21.15.47:0/3346593891 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9500d0360 0x7fd950147e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:18.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd94f400640 1 -- 172.21.15.47:0/3346593891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd95015bcd0 msgr2=0x7fd9500c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:18.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd94f400640 1 --2- 172.21.15.47:0/3346593891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd95015bcd0 0x7fd9500c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:18.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd94f400640 1 -- 172.21.15.47:0/3346593891 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd940009a70 con 0x7fd9500cedb0 2024-02-19T15:28:18.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd946a00640 1 --2- 172.21.15.47:0/3346593891 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9500d0360 0x7fd950147e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:18.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd94fe00640 1 --2- 172.21.15.47:0/3346593891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd95015bcd0 0x7fd9500c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:18.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.934+0000 7fd94f400640 1 --2- 172.21.15.47:0/3346593891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9500cedb0 0x7fd9500cbac0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fd9400b0040 tx=0x7fd9400196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:18.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.938+0000 7fd947e00640 1 -- 172.21.15.47:0/3346593891 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9400be070 con 0x7fd9500cedb0 2024-02-19T15:28:18.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.938+0000 7fd947e00640 1 -- 172.21.15.47:0/3346593891 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9400adca0 con 0x7fd9500cedb0 2024-02-19T15:28:18.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.938+0000 7fd947e00640 1 -- 172.21.15.47:0/3346593891 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9400b9650 con 0x7fd9500cedb0 2024-02-19T15:28:18.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.938+0000 7fd956400640 1 -- 172.21.15.47:0/3346593891 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9500cdb40 con 0x7fd9500cedb0 2024-02-19T15:28:18.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.938+0000 7fd956400640 1 -- 172.21.15.47:0/3346593891 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9500b6d20 con 0x7fd9500cedb0 2024-02-19T15:28:18.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.938+0000 7fd956400640 1 -- 172.21.15.47:0/3346593891 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd950075360 con 0x7fd9500cedb0 2024-02-19T15:28:18.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.938+0000 7fd947e00640 1 -- 172.21.15.47:0/3346593891 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd9400b97b0 con 0x7fd9500cedb0 2024-02-19T15:28:18.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.938+0000 7fd947e00640 1 --2- 172.21.15.47:0/3346593891 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd92c066e40 0x7fd92c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:18.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.938+0000 7fd947e00640 1 -- 172.21.15.47:0/3346593891 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fd94011fe90 con 0x7fd9500cedb0 2024-02-19T15:28:18.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.938+0000 7fd946a00640 1 --2- 172.21.15.47:0/3346593891 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd92c066e40 0x7fd92c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:18.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.942+0000 7fd946a00640 1 --2- 172.21.15.47:0/3346593891 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd92c066e40 0x7fd92c069300 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fd938007fd0 tx=0x7fd938009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:18.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:18.942+0000 7fd947e00640 1 -- 172.21.15.47:0/3346593891 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd9400ecaf0 con 0x7fd9500cedb0 2024-02-19T15:28:19.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.226+0000 7fd956400640 1 -- 172.21.15.47:0/3346593891 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9500b3330 con 0x7fd9500cedb0 2024-02-19T15:28:19.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.226+0000 7fd947e00640 1 -- 172.21.15.47:0/3346593891 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7fd9400b2090 con 0x7fd9500cedb0 2024-02-19T15:28:19.228 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:19.228 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:19.228 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:19.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.230+0000 7fd956400640 1 -- 172.21.15.47:0/3346593891 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd92c066e40 msgr2=0x7fd92c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:19.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.230+0000 7fd956400640 1 --2- 172.21.15.47:0/3346593891 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd92c066e40 0x7fd92c069300 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fd938007fd0 tx=0x7fd938009040 comp rx=0 tx=0).stop 2024-02-19T15:28:19.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.230+0000 7fd956400640 1 -- 172.21.15.47:0/3346593891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9500cedb0 msgr2=0x7fd9500cbac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:19.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.230+0000 7fd956400640 1 --2- 172.21.15.47:0/3346593891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9500cedb0 0x7fd9500cbac0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fd9400b0040 tx=0x7fd9400196a0 comp rx=0 tx=0).stop 2024-02-19T15:28:19.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.230+0000 7fd956400640 1 -- 172.21.15.47:0/3346593891 shutdown_connections 2024-02-19T15:28:19.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.230+0000 7fd956400640 1 --2- 172.21.15.47:0/3346593891 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd92c066e40 0x7fd92c069300 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:19.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.230+0000 7fd956400640 1 --2- 172.21.15.47:0/3346593891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd95015bcd0 0x7fd9500c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:19.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.230+0000 7fd956400640 1 --2- 172.21.15.47:0/3346593891 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9500d0360 0x7fd950147e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:19.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.230+0000 7fd956400640 1 --2- 172.21.15.47:0/3346593891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9500cedb0 0x7fd9500cbac0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:19.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.230+0000 7fd956400640 1 -- 172.21.15.47:0/3346593891 >> 172.21.15.47:0/3346593891 conn(0x7fd9500d1bf0 msgr2=0x7fd9500b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:19.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.230+0000 7fd956400640 1 -- 172.21.15.47:0/3346593891 shutdown_connections 2024-02-19T15:28:19.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.230+0000 7fd956400640 1 -- 172.21.15.47:0/3346593891 wait complete. 2024-02-19T15:28:19.249 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_reintegrated 2024-02-19T15:28:19.411 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"strays_reintegrated":1}} 2024-02-19T15:28:19.411 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 1 } } 2024-02-19T15:28:19.411 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:19.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.602+0000 7f8988e00640 1 Processor -- start 2024-02-19T15:28:19.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.602+0000 7f8988e00640 1 -- start start 2024-02-19T15:28:19.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.602+0000 7f8988e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f898415f620 0x7f89840b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:19.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.602+0000 7f8988e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89840b2330 0x7f898406ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:19.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.602+0000 7f8988e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f898406b5d0 con 0x7f89841614e0 2024-02-19T15:28:19.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8988e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89840b1c00 con 0x7f898415f620 2024-02-19T15:28:19.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8988e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89840b27e0 con 0x7f89840b2330 2024-02-19T15:28:19.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8981e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f89841614e0 0x7f898415eee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:32784/0 (socket says 172.21.15.47:32784) 2024-02-19T15:28:19.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8981e00640 1 -- 172.21.15.47:0/1396210976 learned_addr learned my addr 172.21.15.47:0/1396210976 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:19.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8981400640 1 -- 172.21.15.47:0/1396210976 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732278425 0 0) 0x7f898406b5d0 con 0x7f89841614e0 2024-02-19T15:28:19.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8982800640 1 --2- 172.21.15.47:0/1396210976 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89840b2330 0x7f898406ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:19.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f897be00640 1 --2- 172.21.15.47:0/1396210976 >> v2:172.21.15.116:3300/0 conn(0x7f898415f620 0x7f89840b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:19.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8981400640 1 -- 172.21.15.47:0/1396210976 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8968003530 con 0x7f89841614e0 2024-02-19T15:28:19.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8981400640 1 -- 172.21.15.47:0/1396210976 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 861428031 0 0) 0x7f8968003530 con 0x7f89841614e0 2024-02-19T15:28:19.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8981400640 1 -- 172.21.15.47:0/1396210976 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89840b2330 msgr2=0x7f898406ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:19.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8981400640 1 --2- 172.21.15.47:0/1396210976 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89840b2330 0x7f898406ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:19.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8981400640 1 -- 172.21.15.47:0/1396210976 >> v2:172.21.15.116:3300/0 conn(0x7f898415f620 msgr2=0x7f89840b1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:19.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8981400640 1 --2- 172.21.15.47:0/1396210976 >> v2:172.21.15.116:3300/0 conn(0x7f898415f620 0x7f89840b1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:19.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8981400640 1 -- 172.21.15.47:0/1396210976 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89840b1640 con 0x7f89841614e0 2024-02-19T15:28:19.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8981400640 1 -- 172.21.15.47:0/1396210976 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f8974001680 con 0x7f89841614e0 2024-02-19T15:28:19.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8981400640 1 -- 172.21.15.47:0/1396210976 >> v1:172.21.15.47:6789/0 conn(0x7f89841614e0 legacy=0x7f898415eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:19.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8981400640 1 --2- 172.21.15.47:0/1396210976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8968003980 0x7f8968003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:19.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8981400640 1 --2- 172.21.15.47:0/1396210976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89680047d0 0x7f8968004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:19.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8981400640 1 --2- 172.21.15.47:0/1396210976 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89840b2330 0x7f8968005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:19.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8981400640 1 -- 172.21.15.47:0/1396210976 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89840b1c00 con 0x7f89680047d0 2024-02-19T15:28:19.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8981400640 1 -- 172.21.15.47:0/1396210976 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89840b27e0 con 0x7f8968003980 2024-02-19T15:28:19.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8981400640 1 -- 172.21.15.47:0/1396210976 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f896800b990 con 0x7f89840b2330 2024-02-19T15:28:19.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8981e00640 1 --2- 172.21.15.47:0/1396210976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8968003980 0x7f8968003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:19.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f897be00640 1 --2- 172.21.15.47:0/1396210976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89680047d0 0x7f8968004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:19.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f8982800640 1 --2- 172.21.15.47:0/1396210976 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89840b2330 0x7f8968005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:19.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f897be00640 1 -- 172.21.15.47:0/1396210976 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89840b2330 msgr2=0x7f8968005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:19.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f897be00640 1 --2- 172.21.15.47:0/1396210976 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89840b2330 0x7f8968005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:19.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f897be00640 1 -- 172.21.15.47:0/1396210976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8968003980 msgr2=0x7f8968003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:19.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f897be00640 1 --2- 172.21.15.47:0/1396210976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8968003980 0x7f8968003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:19.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.606+0000 7f897be00640 1 -- 172.21.15.47:0/1396210976 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8968003530 con 0x7f89680047d0 2024-02-19T15:28:19.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.610+0000 7f897be00640 1 --2- 172.21.15.47:0/1396210976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89680047d0 0x7f8968004c50 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f896c009ab0 tx=0x7f896c01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c2bb01dffc0a138 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:19.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.610+0000 7f8981400640 1 -- 172.21.15.47:0/1396210976 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896c0a55a0 con 0x7f89680047d0 2024-02-19T15:28:19.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.610+0000 7f8981400640 1 -- 172.21.15.47:0/1396210976 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f896c0a5700 con 0x7f89680047d0 2024-02-19T15:28:19.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.610+0000 7f8981400640 1 -- 172.21.15.47:0/1396210976 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896c0a5a70 con 0x7f89680047d0 2024-02-19T15:28:19.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.610+0000 7f8988e00640 1 -- 172.21.15.47:0/1396210976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89680047d0 msgr2=0x7f8968004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:19.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.610+0000 7f8988e00640 1 --2- 172.21.15.47:0/1396210976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89680047d0 0x7f8968004c50 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f896c009ab0 tx=0x7f896c01ca90 comp rx=0 tx=0).stop 2024-02-19T15:28:19.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.610+0000 7f8982800640 1 -- 172.21.15.47:0/1396210976 reap_dead start 2024-02-19T15:28:19.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.610+0000 7f8988e00640 1 -- 172.21.15.47:0/1396210976 shutdown_connections 2024-02-19T15:28:19.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.610+0000 7f8988e00640 1 -- 172.21.15.47:0/1396210976 >> 172.21.15.47:0/1396210976 conn(0x7f89840b4fc0 msgr2=0x7f89840b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:19.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.610+0000 7f8988e00640 1 -- 172.21.15.47:0/1396210976 shutdown_connections 2024-02-19T15:28:19.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.610+0000 7f8988e00640 1 -- 172.21.15.47:0/1396210976 wait complete. 2024-02-19T15:28:19.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f8988e00640 1 Processor -- start 2024-02-19T15:28:19.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f8988e00640 1 -- start start 2024-02-19T15:28:19.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f8988e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89680047d0 0x7f898409c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:19.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f8988e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f898409a250 0x7f898409a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:19.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f8988e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f898414a460 0x7f898414a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:19.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f8988e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898409abf0 con 0x7f898414a460 2024-02-19T15:28:19.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f8988e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898409c660 con 0x7f898409a250 2024-02-19T15:28:19.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f8988e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89841493c0 con 0x7f89680047d0 2024-02-19T15:28:19.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f8982800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f898414a460 0x7f898414a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:19.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f8982800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f898414a460 0x7f898414a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39486/0 (socket says 172.21.15.47:39486) 2024-02-19T15:28:19.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f8982800640 1 -- 172.21.15.47:0/1705633081 learned_addr learned my addr 172.21.15.47:0/1705633081 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:19.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f8981e00640 1 --2- 172.21.15.47:0/1705633081 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89680047d0 0x7f898409c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:19.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f897be00640 1 --2- 172.21.15.47:0/1705633081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f898409a250 0x7f898409a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f8982800640 1 -- 172.21.15.47:0/1705633081 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89680047d0 msgr2=0x7f898409c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f8982800640 1 --2- 172.21.15.47:0/1705633081 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89680047d0 0x7f898409c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f8982800640 1 -- 172.21.15.47:0/1705633081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f898409a250 msgr2=0x7f898409a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f8982800640 1 --2- 172.21.15.47:0/1705633081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f898409a250 0x7f898409a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f8982800640 1 -- 172.21.15.47:0/1705633081 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89841496a0 con 0x7f898414a460 2024-02-19T15:28:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f897be00640 1 --2- 172.21.15.47:0/1705633081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f898409a250 0x7f898409a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f8982800640 1 --2- 172.21.15.47:0/1705633081 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f898414a460 0x7f898414a8e0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f8984071120 tx=0x7f897c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f897a000640 1 -- 172.21.15.47:0/1705633081 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897c00e040 con 0x7f898414a460 2024-02-19T15:28:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f897a000640 1 -- 172.21.15.47:0/1705633081 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f897c00f040 con 0x7f898414a460 2024-02-19T15:28:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f8988e00640 1 -- 172.21.15.47:0/1705633081 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8984149980 con 0x7f898414a460 2024-02-19T15:28:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.614+0000 7f897a000640 1 -- 172.21.15.47:0/1705633081 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897c015430 con 0x7f898414a460 2024-02-19T15:28:19.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.618+0000 7f8988e00640 1 -- 172.21.15.47:0/1705633081 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8984149f40 con 0x7f898414a460 2024-02-19T15:28:19.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.618+0000 7f8988e00640 1 -- 172.21.15.47:0/1705633081 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8984075360 con 0x7f898414a460 2024-02-19T15:28:19.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.618+0000 7f897a000640 1 -- 172.21.15.47:0/1705633081 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f897c015590 con 0x7f898414a460 2024-02-19T15:28:19.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.618+0000 7f897a000640 1 --2- 172.21.15.47:0/1705633081 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8960066e70 0x7f8960069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:19.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.618+0000 7f897a000640 1 -- 172.21.15.47:0/1705633081 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f897c085250 con 0x7f898414a460 2024-02-19T15:28:19.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.618+0000 7f8981e00640 1 --2- 172.21.15.47:0/1705633081 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8960066e70 0x7f8960069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:19.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.622+0000 7f8981e00640 1 --2- 172.21.15.47:0/1705633081 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8960066e70 0x7f8960069330 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f89740050c0 tx=0x7f8974009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:19.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.622+0000 7f897a000640 1 -- 172.21.15.47:0/1705633081 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f897c051eb0 con 0x7f898414a460 2024-02-19T15:28:19.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.822+0000 7f8988e00640 1 -- 172.21.15.47:0/1705633081 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f898406b230 con 0x7f898414a460 2024-02-19T15:28:19.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.822+0000 7f897a000640 1 -- 172.21.15.47:0/1705633081 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f897c055570 con 0x7f898414a460 2024-02-19T15:28:19.823 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:19.823 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:19.823 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:19.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.822+0000 7f8988e00640 1 -- 172.21.15.47:0/1705633081 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8960066e70 msgr2=0x7f8960069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:19.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.826+0000 7f8988e00640 1 --2- 172.21.15.47:0/1705633081 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8960066e70 0x7f8960069330 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f89740050c0 tx=0x7f8974009040 comp rx=0 tx=0).stop 2024-02-19T15:28:19.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.826+0000 7f8988e00640 1 -- 172.21.15.47:0/1705633081 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f898414a460 msgr2=0x7f898414a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:19.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.826+0000 7f8988e00640 1 --2- 172.21.15.47:0/1705633081 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f898414a460 0x7f898414a8e0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f8984071120 tx=0x7f897c00f580 comp rx=0 tx=0).stop 2024-02-19T15:28:19.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.826+0000 7f8988e00640 1 -- 172.21.15.47:0/1705633081 shutdown_connections 2024-02-19T15:28:19.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.826+0000 7f8988e00640 1 --2- 172.21.15.47:0/1705633081 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8960066e70 0x7f8960069330 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:19.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.826+0000 7f8988e00640 1 --2- 172.21.15.47:0/1705633081 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f898414a460 0x7f898414a8e0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:19.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.826+0000 7f8988e00640 1 --2- 172.21.15.47:0/1705633081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f898409a250 0x7f898409a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:19.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.826+0000 7f8988e00640 1 --2- 172.21.15.47:0/1705633081 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89680047d0 0x7f898409c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:19.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.826+0000 7f8988e00640 1 -- 172.21.15.47:0/1705633081 >> 172.21.15.47:0/1705633081 conn(0x7f89840b4fc0 msgr2=0x7f89840b5e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:19.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.826+0000 7f8988e00640 1 -- 172.21.15.47:0/1705633081 shutdown_connections 2024-02-19T15:28:19.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:19.826+0000 7f8988e00640 1 -- 172.21.15.47:0/1705633081 wait complete. 2024-02-19T15:28:19.845 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-02-19T15:28:20.011 INFO:teuthology.orchestra.run.smithi047.stdout:{"message":"","return_code":0} 2024-02-19T15:28:20.012 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-02-19T15:28:20.012 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-02-19T15:28:20.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.058+0000 7fea3a1c6080 1 Processor -- start 2024-02-19T15:28:20.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.058+0000 7fea3a1c6080 1 -- start start 2024-02-19T15:28:20.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.058+0000 7fea3a1c6080 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f879b39ab0 0x55f879b21a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:20.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.058+0000 7fea3a1c6080 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f879b39e90 0x55f879aef690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:20.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.058+0000 7fea3a1c6080 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f879bc5bd0 0x55f879bc61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:20.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.058+0000 7fea3a1c6080 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f879a9df90 con 0x55f879b39e90 2024-02-19T15:28:20.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.058+0000 7fea3a1c6080 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f879a351e0 con 0x55f879b39ab0 2024-02-19T15:28:20.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.058+0000 7fea3a1c6080 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f879a34e80 con 0x55f879bc5bd0 2024-02-19T15:28:20.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.058+0000 7fea36a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f879b39e90 0x55f879aef690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:20.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.058+0000 7fea36a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f879b39e90 0x55f879aef690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39494/0 (socket says 172.21.15.47:39494) 2024-02-19T15:28:20.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.058+0000 7fea36a00640 1 -- 172.21.15.47:0/2749120013 learned_addr learned my addr 172.21.15.47:0/2749120013 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:20.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.058+0000 7fea37e00640 1 --2- 172.21.15.47:0/2749120013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f879bc5bd0 0x55f879bc61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:20.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.058+0000 7fea37400640 1 --2- 172.21.15.47:0/2749120013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f879b39ab0 0x55f879b21a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:20.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.058+0000 7fea36a00640 1 -- 172.21.15.47:0/2749120013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f879bc5bd0 msgr2=0x55f879bc61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:20.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.058+0000 7fea36a00640 1 --2- 172.21.15.47:0/2749120013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f879bc5bd0 0x55f879bc61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.058+0000 7fea36a00640 1 -- 172.21.15.47:0/2749120013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f879b39ab0 msgr2=0x55f879b21a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:20.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.058+0000 7fea36a00640 1 --2- 172.21.15.47:0/2749120013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f879b39ab0 0x55f879b21a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.058+0000 7fea36a00640 1 -- 172.21.15.47:0/2749120013 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3000a100 con 0x55f879b39e90 2024-02-19T15:28:20.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.058+0000 7fea37400640 1 --2- 172.21.15.47:0/2749120013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f879b39ab0 0x55f879b21a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:20.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea36a00640 1 --2- 172.21.15.47:0/2749120013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f879b39e90 0x55f879aef690 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fea24001ce0 tx=0x7fea2400b6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11db986988785ef7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:20.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea35600640 1 -- 172.21.15.47:0/2749120013 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2400e480 con 0x55f879b39e90 2024-02-19T15:28:20.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea35600640 1 -- 172.21.15.47:0/2749120013 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2400e5e0 con 0x55f879b39e90 2024-02-19T15:28:20.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea35600640 1 -- 172.21.15.47:0/2749120013 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2400e970 con 0x55f879b39e90 2024-02-19T15:28:20.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea3a1c6080 1 -- 172.21.15.47:0/2749120013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f879b39e90 msgr2=0x55f879aef690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:20.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea3a1c6080 1 --2- 172.21.15.47:0/2749120013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f879b39e90 0x55f879aef690 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fea24001ce0 tx=0x7fea2400b6d0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea3a1c6080 1 -- 172.21.15.47:0/2749120013 shutdown_connections 2024-02-19T15:28:20.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea3a1c6080 1 --2- 172.21.15.47:0/2749120013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f879bc5bd0 0x55f879bc61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea3a1c6080 1 --2- 172.21.15.47:0/2749120013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f879b39e90 0x55f879aef690 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea3a1c6080 1 --2- 172.21.15.47:0/2749120013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f879b39ab0 0x55f879b21a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea3a1c6080 1 -- 172.21.15.47:0/2749120013 >> 172.21.15.47:0/2749120013 conn(0x55f879b1c680 msgr2=0x55f879bc7620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:20.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea3a1c6080 1 -- 172.21.15.47:0/2749120013 shutdown_connections 2024-02-19T15:28:20.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea3a1c6080 1 -- 172.21.15.47:0/2749120013 wait complete. 2024-02-19T15:28:20.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea3a1c6080 1 Processor -- start 2024-02-19T15:28:20.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea3a1c6080 1 -- start start 2024-02-19T15:28:20.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea3a1c6080 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f879b39ab0 0x55f879c60970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:20.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea3a1c6080 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f879b39e90 0x55f879bbfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:20.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea3a1c6080 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f879bc5bd0 0x55f879bc14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:20.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea3a1c6080 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f879a58610 con 0x55f879b39e90 2024-02-19T15:28:20.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea3a1c6080 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f879a28740 con 0x55f879b39ab0 2024-02-19T15:28:20.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea3a1c6080 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f879c60eb0 con 0x55f879bc5bd0 2024-02-19T15:28:20.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.062+0000 7fea36a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f879b39e90 0x55f879bbfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:20.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.066+0000 7fea36a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f879b39e90 0x55f879bbfa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39506/0 (socket says 172.21.15.47:39506) 2024-02-19T15:28:20.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.066+0000 7fea36a00640 1 -- 172.21.15.47:0/4051102663 learned_addr learned my addr 172.21.15.47:0/4051102663 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:20.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.066+0000 7fea37400640 1 --2- 172.21.15.47:0/4051102663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f879b39ab0 0x55f879c60970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:20.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.066+0000 7fea37e00640 1 --2- 172.21.15.47:0/4051102663 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f879bc5bd0 0x55f879bc14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:20.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.066+0000 7fea36a00640 1 -- 172.21.15.47:0/4051102663 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f879bc5bd0 msgr2=0x55f879bc14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:20.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.066+0000 7fea36a00640 1 --2- 172.21.15.47:0/4051102663 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55f879bc5bd0 0x55f879bc14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.066+0000 7fea36a00640 1 -- 172.21.15.47:0/4051102663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f879b39ab0 msgr2=0x55f879c60970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:20.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.066+0000 7fea36a00640 1 --2- 172.21.15.47:0/4051102663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f879b39ab0 0x55f879c60970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.066+0000 7fea36a00640 1 -- 172.21.15.47:0/4051102663 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f879a33ba0 con 0x55f879b39e90 2024-02-19T15:28:20.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.066+0000 7fea36a00640 1 --2- 172.21.15.47:0/4051102663 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f879b39e90 0x55f879bbfa10 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fea2400bbd0 tx=0x7fea24000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:20.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.066+0000 7fea2fe00640 1 -- 172.21.15.47:0/4051102663 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2401e070 con 0x55f879b39e90 2024-02-19T15:28:20.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.066+0000 7fea2fe00640 1 -- 172.21.15.47:0/4051102663 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2400ee20 con 0x55f879b39e90 2024-02-19T15:28:20.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.066+0000 7fea2fe00640 1 -- 172.21.15.47:0/4051102663 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2400e480 con 0x55f879b39e90 2024-02-19T15:28:20.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.066+0000 7fea3a1c6080 1 -- 172.21.15.47:0/4051102663 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f879a33f30 con 0x55f879b39e90 2024-02-19T15:28:20.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.066+0000 7fea3a1c6080 1 -- 172.21.15.47:0/4051102663 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f879a33990 con 0x55f879b39e90 2024-02-19T15:28:20.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.066+0000 7fea2fe00640 1 -- 172.21.15.47:0/4051102663 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fea2400a040 con 0x55f879b39e90 2024-02-19T15:28:20.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.066+0000 7fea2fe00640 1 --2- 172.21.15.47:0/4051102663 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fea18065870 0x7fea18067d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:20.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.066+0000 7fea2fe00640 1 -- 172.21.15.47:0/4051102663 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fea240853d0 con 0x55f879b39e90 2024-02-19T15:28:20.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.070+0000 7fea37400640 1 --2- 172.21.15.47:0/4051102663 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fea18065870 0x7fea18067d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:20.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.070+0000 7fea3a1c6080 1 --2- 172.21.15.47:0/4051102663 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x55f879c5de90 0x55f879c5e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:20.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.070+0000 7fea3a1c6080 1 -- 172.21.15.47:0/4051102663 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v8 -- 0x55f879bbff50 con 0x55f879c5de90 2024-02-19T15:28:20.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.070+0000 7fea37e00640 1 --2- 172.21.15.47:0/4051102663 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x55f879c5de90 0x55f879c5e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:20.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.070+0000 7fea37400640 1 --2- 172.21.15.47:0/4051102663 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fea18065870 0x7fea18067d30 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fea28003f20 tx=0x7fea28002870 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:20.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.070+0000 7fea37e00640 1 --2- 172.21.15.47:0/4051102663 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x55f879c5de90 0x55f879c5e2a0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:20.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.070+0000 7fea37e00640 1 -- 172.21.15.47:0/4051102663 <== osd.0 v2:172.21.15.47:6809/1968337531 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7fea300b1040 con 0x55f879c5de90 2024-02-19T15:28:20.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.070+0000 7fea3a1c6080 1 -- 172.21.15.47:0/4051102663 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x55f879c5de90 msgr2=0x55f879c5e2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:20.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.070+0000 7fea3a1c6080 1 --2- 172.21.15.47:0/4051102663 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x55f879c5de90 0x55f879c5e2a0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.070+0000 7fea3a1c6080 1 -- 172.21.15.47:0/4051102663 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fea18065870 msgr2=0x7fea18067d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:20.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.070+0000 7fea3a1c6080 1 --2- 172.21.15.47:0/4051102663 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fea18065870 0x7fea18067d30 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fea28003f20 tx=0x7fea28002870 comp rx=0 tx=0).stop 2024-02-19T15:28:20.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.070+0000 7fea3a1c6080 1 -- 172.21.15.47:0/4051102663 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f879b39e90 msgr2=0x55f879bbfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:20.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.070+0000 7fea3a1c6080 1 --2- 172.21.15.47:0/4051102663 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f879b39e90 0x55f879bbfa10 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fea2400bbd0 tx=0x7fea24000bd0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.070+0000 7fea37e00640 1 -- 172.21.15.47:0/4051102663 reap_dead start 2024-02-19T15:28:20.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.070+0000 7fea3a1c6080 1 -- 172.21.15.47:0/4051102663 shutdown_connections 2024-02-19T15:28:20.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.070+0000 7fea3a1c6080 1 -- 172.21.15.47:0/4051102663 >> 172.21.15.47:0/4051102663 conn(0x55f879b1c680 msgr2=0x55f879aef690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:20.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.070+0000 7fea3a1c6080 1 -- 172.21.15.47:0/4051102663 shutdown_connections 2024-02-19T15:28:20.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.070+0000 7fea3a1c6080 1 -- 172.21.15.47:0/4051102663 wait complete. 2024-02-19T15:28:20.077 INFO:teuthology.orchestra.run.smithi047.stdout:W�file_cdir_2# 2024-02-19T15:28:20.077 DEBUG:teuthology.orchestra.run.smithi047:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-02-19T15:28:20.165 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:28:20.165 INFO:teuthology.orchestra.run.smithi047.stdout: "ino": 1099511627777, 2024-02-19T15:28:20.165 INFO:teuthology.orchestra.run.smithi047.stdout: "ancestors": [ 2024-02-19T15:28:20.165 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:20.165 INFO:teuthology.orchestra.run.smithi047.stdout: "dirino": 1099511628278, 2024-02-19T15:28:20.165 INFO:teuthology.orchestra.run.smithi047.stdout: "dname": "file_c", 2024-02-19T15:28:20.165 INFO:teuthology.orchestra.run.smithi047.stdout: "version": 17 2024-02-19T15:28:20.165 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:20.166 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:20.166 INFO:teuthology.orchestra.run.smithi047.stdout: "dirino": 1, 2024-02-19T15:28:20.166 INFO:teuthology.orchestra.run.smithi047.stdout: "dname": "dir_2", 2024-02-19T15:28:20.166 INFO:teuthology.orchestra.run.smithi047.stdout: "version": 35 2024-02-19T15:28:20.166 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:20.166 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-02-19T15:28:20.166 INFO:teuthology.orchestra.run.smithi047.stdout: "pool": 17, 2024-02-19T15:28:20.166 INFO:teuthology.orchestra.run.smithi047.stdout: "old_pools": [] 2024-02-19T15:28:20.166 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:28:20.166 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:20.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:28:20.169 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_2/file_c dir_2/file_d) 2024-02-19T15:28:20.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:28:20.228 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_c) 2024-02-19T15:28:20.279 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:20.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.458+0000 7fa78da00640 1 Processor -- start 2024-02-19T15:28:20.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.458+0000 7fa78da00640 1 -- start start 2024-02-19T15:28:20.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.462+0000 7fa78da00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa78806ef50 0x7fa78806d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:20.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.462+0000 7fa78da00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa78806f330 0x7fa78815aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.462+0000 7fa78da00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa78815c650 0x7fa78815cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.462+0000 7fa78da00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa78815c350 con 0x7fa78806ef50 2024-02-19T15:28:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.462+0000 7fa78da00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa78806dbe0 con 0x7fa78806f330 2024-02-19T15:28:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.462+0000 7fa78da00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa78806ddf0 con 0x7fa78815c650 2024-02-19T15:28:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.462+0000 7fa786a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa78806ef50 0x7fa78806d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.462+0000 7fa786a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa78806ef50 0x7fa78806d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39520/0 (socket says 172.21.15.47:39520) 2024-02-19T15:28:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.462+0000 7fa786a00640 1 -- 172.21.15.47:0/1866116557 learned_addr learned my addr 172.21.15.47:0/1866116557 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.462+0000 7fa786000640 1 --2- 172.21.15.47:0/1866116557 >> v2:172.21.15.116:3300/0 conn(0x7fa78806f330 0x7fa78815aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.462+0000 7fa787400640 1 --2- 172.21.15.47:0/1866116557 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa78815c650 0x7fa78815cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.462+0000 7fa786a00640 1 -- 172.21.15.47:0/1866116557 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa78815c650 msgr2=0x7fa78815cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.462+0000 7fa786a00640 1 --2- 172.21.15.47:0/1866116557 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa78815c650 0x7fa78815cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.462+0000 7fa786a00640 1 -- 172.21.15.47:0/1866116557 >> v2:172.21.15.116:3300/0 conn(0x7fa78806f330 msgr2=0x7fa78815aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:20.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.462+0000 7fa786a00640 1 --2- 172.21.15.47:0/1866116557 >> v2:172.21.15.116:3300/0 conn(0x7fa78806f330 0x7fa78815aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.462+0000 7fa786a00640 1 -- 172.21.15.47:0/1866116557 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78806e0d0 con 0x7fa78806ef50 2024-02-19T15:28:20.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.462+0000 7fa786000640 1 --2- 172.21.15.47:0/1866116557 >> v2:172.21.15.116:3300/0 conn(0x7fa78806f330 0x7fa78815aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:20.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.466+0000 7fa786a00640 1 --2- 172.21.15.47:0/1866116557 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa78806ef50 0x7fa78806d6a0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fa7880704d0 tx=0x7fa77801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e4296ff3286802d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:20.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.466+0000 7fa785600640 1 -- 172.21.15.47:0/1866116557 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7780aa610 con 0x7fa78806ef50 2024-02-19T15:28:20.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.466+0000 7fa785600640 1 -- 172.21.15.47:0/1866116557 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7780a6040 con 0x7fa78806ef50 2024-02-19T15:28:20.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.466+0000 7fa785600640 1 -- 172.21.15.47:0/1866116557 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7780aa990 con 0x7fa78806ef50 2024-02-19T15:28:20.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.466+0000 7fa78da00640 1 -- 172.21.15.47:0/1866116557 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa78806ef50 msgr2=0x7fa78806d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:20.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.466+0000 7fa78da00640 1 --2- 172.21.15.47:0/1866116557 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa78806ef50 0x7fa78806d6a0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fa7880704d0 tx=0x7fa77801c4f0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.466+0000 7fa78da00640 1 -- 172.21.15.47:0/1866116557 shutdown_connections 2024-02-19T15:28:20.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.466+0000 7fa78da00640 1 --2- 172.21.15.47:0/1866116557 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa78815c650 0x7fa78815cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.466+0000 7fa78da00640 1 --2- 172.21.15.47:0/1866116557 >> v2:172.21.15.116:3300/0 conn(0x7fa78806f330 0x7fa78815aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.466+0000 7fa78da00640 1 --2- 172.21.15.47:0/1866116557 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa78806ef50 0x7fa78806d6a0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.466+0000 7fa78da00640 1 -- 172.21.15.47:0/1866116557 >> 172.21.15.47:0/1866116557 conn(0x7fa7880d0ad0 msgr2=0x7fa7880d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:20.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.466+0000 7fa78da00640 1 -- 172.21.15.47:0/1866116557 shutdown_connections 2024-02-19T15:28:20.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.466+0000 7fa78da00640 1 -- 172.21.15.47:0/1866116557 wait complete. 2024-02-19T15:28:20.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa78da00640 1 Processor -- start 2024-02-19T15:28:20.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa78da00640 1 -- start start 2024-02-19T15:28:20.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa78da00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa78806ef50 0x7fa78814c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:20.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa78da00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa78806f330 0x7fa788155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:20.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa78da00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa78815c650 0x7fa788156280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:20.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa78da00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7880cec60 con 0x7fa78806ef50 2024-02-19T15:28:20.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa78da00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7880cef40 con 0x7fa78806f330 2024-02-19T15:28:20.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa78da00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7880cf220 con 0x7fa78815c650 2024-02-19T15:28:20.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa786a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa78806ef50 0x7fa78814c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:20.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa786a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa78806ef50 0x7fa78814c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39530/0 (socket says 172.21.15.47:39530) 2024-02-19T15:28:20.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa786a00640 1 -- 172.21.15.47:0/3756846272 learned_addr learned my addr 172.21.15.47:0/3756846272 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:20.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa786000640 1 --2- 172.21.15.47:0/3756846272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa78806f330 0x7fa788155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:20.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa787400640 1 --2- 172.21.15.47:0/3756846272 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa78815c650 0x7fa788156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:20.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa786a00640 1 -- 172.21.15.47:0/3756846272 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa78815c650 msgr2=0x7fa788156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:20.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa786a00640 1 --2- 172.21.15.47:0/3756846272 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa78815c650 0x7fa788156280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa786a00640 1 -- 172.21.15.47:0/3756846272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa78806f330 msgr2=0x7fa788155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:20.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa786a00640 1 --2- 172.21.15.47:0/3756846272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa78806f330 0x7fa788155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa786a00640 1 -- 172.21.15.47:0/3756846272 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7780094d0 con 0x7fa78806ef50 2024-02-19T15:28:20.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa787400640 1 --2- 172.21.15.47:0/3756846272 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa78815c650 0x7fa788156280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:20.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa786a00640 1 --2- 172.21.15.47:0/3756846272 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa78806ef50 0x7fa78814c0a0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fa77801aba0 tx=0x7fa77801ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:20.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa77ea00640 1 -- 172.21.15.47:0/3756846272 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7780bd5f0 con 0x7fa78806ef50 2024-02-19T15:28:20.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa77ea00640 1 -- 172.21.15.47:0/3756846272 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7780bd750 con 0x7fa78806ef50 2024-02-19T15:28:20.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa78da00640 1 -- 172.21.15.47:0/3756846272 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7880cf500 con 0x7fa78806ef50 2024-02-19T15:28:20.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa77ea00640 1 -- 172.21.15.47:0/3756846272 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7780bdb00 con 0x7fa78806ef50 2024-02-19T15:28:20.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.470+0000 7fa78da00640 1 -- 172.21.15.47:0/3756846272 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa78814c5e0 con 0x7fa78806ef50 2024-02-19T15:28:20.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.474+0000 7fa78da00640 1 -- 172.21.15.47:0/3756846272 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa788075360 con 0x7fa78806ef50 2024-02-19T15:28:20.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.474+0000 7fa77ea00640 1 -- 172.21.15.47:0/3756846272 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa78815d830 con 0x7fa78806ef50 2024-02-19T15:28:20.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.474+0000 7fa77ea00640 1 --2- 172.21.15.47:0/3756846272 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa768066e70 0x7fa768069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:20.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.474+0000 7fa77ea00640 1 -- 172.21.15.47:0/3756846272 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fa77811f870 con 0x7fa78806ef50 2024-02-19T15:28:20.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.474+0000 7fa786000640 1 --2- 172.21.15.47:0/3756846272 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa768066e70 0x7fa768069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:20.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.478+0000 7fa786000640 1 --2- 172.21.15.47:0/3756846272 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa768066e70 0x7fa768069330 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa7880d0120 tx=0x7fa774008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:20.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.478+0000 7fa77ea00640 1 -- 172.21.15.47:0/3756846272 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa7780ec570 con 0x7fa78806ef50 2024-02-19T15:28:20.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.674+0000 7fa78da00640 1 -- 172.21.15.47:0/3756846272 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7880cf360 con 0x7fa78806ef50 2024-02-19T15:28:20.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.674+0000 7fa77ea00640 1 -- 172.21.15.47:0/3756846272 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7fa7880cf360 con 0x7fa78806ef50 2024-02-19T15:28:20.675 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:20.676 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:20.676 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:20.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.678+0000 7fa78da00640 1 -- 172.21.15.47:0/3756846272 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa768066e70 msgr2=0x7fa768069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:20.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.678+0000 7fa78da00640 1 --2- 172.21.15.47:0/3756846272 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa768066e70 0x7fa768069330 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa7880d0120 tx=0x7fa774008040 comp rx=0 tx=0).stop 2024-02-19T15:28:20.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.678+0000 7fa78da00640 1 -- 172.21.15.47:0/3756846272 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa78806ef50 msgr2=0x7fa78814c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:20.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.678+0000 7fa78da00640 1 --2- 172.21.15.47:0/3756846272 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa78806ef50 0x7fa78814c0a0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fa77801aba0 tx=0x7fa77801ad30 comp rx=0 tx=0).stop 2024-02-19T15:28:20.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.678+0000 7fa78da00640 1 -- 172.21.15.47:0/3756846272 shutdown_connections 2024-02-19T15:28:20.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.678+0000 7fa78da00640 1 --2- 172.21.15.47:0/3756846272 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa768066e70 0x7fa768069330 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.678+0000 7fa78da00640 1 --2- 172.21.15.47:0/3756846272 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa78815c650 0x7fa788156280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.678+0000 7fa78da00640 1 --2- 172.21.15.47:0/3756846272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa78806f330 0x7fa788155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.678+0000 7fa78da00640 1 --2- 172.21.15.47:0/3756846272 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa78806ef50 0x7fa78814c0a0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:20.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.678+0000 7fa78da00640 1 -- 172.21.15.47:0/3756846272 >> 172.21.15.47:0/3756846272 conn(0x7fa7880d0ad0 msgr2=0x7fa7880b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:20.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.678+0000 7fa78da00640 1 -- 172.21.15.47:0/3756846272 shutdown_connections 2024-02-19T15:28:20.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:20.678+0000 7fa78da00640 1 -- 172.21.15.47:0/3756846272 wait complete. 2024-02-19T15:28:20.698 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-02-19T15:28:20.858 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"num_strays":1}} 2024-02-19T15:28:20.858 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-02-19T15:28:20.858 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 0, waiting (timeout=60)... 2024-02-19T15:28:25.859 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:26.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:25.998+0000 7f27c0e00640 1 Processor -- start 2024-02-19T15:28:26.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:25.998+0000 7f27c0e00640 1 -- start start 2024-02-19T15:28:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.002+0000 7f27c0e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f27bc15b1a0 0x7f27bc15b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.002+0000 7f27c0e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27bc06c9a0 0x7f27bc15bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.002+0000 7f27c0e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc15c000 con 0x7f27bc071350 2024-02-19T15:28:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.002+0000 7f27c0e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27bc0718c0 con 0x7f27bc15b1a0 2024-02-19T15:28:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.002+0000 7f27c0e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc071ba0 con 0x7f27bc06c9a0 2024-02-19T15:28:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.002+0000 7f27b3e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f27bc071350 0x7f27bc15c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41492/0 (socket says 172.21.15.47:41492) 2024-02-19T15:28:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.002+0000 7f27b3e00640 1 -- 172.21.15.47:0/3595827467 learned_addr learned my addr 172.21.15.47:0/3595827467 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.002+0000 7f27b9e00640 1 --2- 172.21.15.47:0/3595827467 >> v2:172.21.15.116:3300/0 conn(0x7f27bc15b1a0 0x7f27bc15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.002+0000 7f27ba800640 1 --2- 172.21.15.47:0/3595827467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27bc06c9a0 0x7f27bc15bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.002+0000 7f27b9400640 1 -- 172.21.15.47:0/3595827467 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603265596 0 0) 0x7f27bc15c000 con 0x7f27bc071350 2024-02-19T15:28:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.002+0000 7f27b9400640 1 -- 172.21.15.47:0/3595827467 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27a0003530 con 0x7f27bc071350 2024-02-19T15:28:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.002+0000 7f27b9e00640 1 -- 172.21.15.47:0/3595827467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27bc06c9a0 msgr2=0x7f27bc15bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.002+0000 7f27b9e00640 1 --2- 172.21.15.47:0/3595827467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27bc06c9a0 0x7f27bc15bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.002+0000 7f27b9e00640 1 -- 172.21.15.47:0/3595827467 >> v1:172.21.15.47:6789/0 conn(0x7f27bc071350 legacy=0x7f27bc15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.002+0000 7f27b9400640 1 -- 172.21.15.47:0/3595827467 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1239821087 0 0) 0x7f27a0003530 con 0x7f27bc071350 2024-02-19T15:28:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.002+0000 7f27b9e00640 1 -- 172.21.15.47:0/3595827467 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc071e80 con 0x7f27bc15b1a0 2024-02-19T15:28:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.002+0000 7f27ba800640 1 --2- 172.21.15.47:0/3595827467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27bc06c9a0 0x7f27bc15bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:28:26.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.006+0000 7f27b9e00640 1 --2- 172.21.15.47:0/3595827467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27bc15b1a0 0x7f27bc15b580 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f27ac0097f0 tx=0x7f27ac01c390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=957fd5192101c91e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:26.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.006+0000 7f27b9400640 1 -- 172.21.15.47:0/3595827467 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a4001680 con 0x7f27bc15b1a0 2024-02-19T15:28:26.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.006+0000 7f27b9400640 1 -- 172.21.15.47:0/3595827467 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ac0a7050 con 0x7f27bc15b1a0 2024-02-19T15:28:26.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.006+0000 7f27b9400640 1 -- 172.21.15.47:0/3595827467 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ac0ab620 con 0x7f27bc15b1a0 2024-02-19T15:28:26.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.006+0000 7f27c0e00640 1 -- 172.21.15.47:0/3595827467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27bc15b1a0 msgr2=0x7f27bc15b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:26.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.006+0000 7f27c0e00640 1 --2- 172.21.15.47:0/3595827467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27bc15b1a0 0x7f27bc15b580 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f27ac0097f0 tx=0x7f27ac01c390 comp rx=0 tx=0).stop 2024-02-19T15:28:26.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.006+0000 7f27c0e00640 1 -- 172.21.15.47:0/3595827467 shutdown_connections 2024-02-19T15:28:26.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.006+0000 7f27c0e00640 1 --2- 172.21.15.47:0/3595827467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27bc06c9a0 0x7f27bc15bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:26.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.006+0000 7f27c0e00640 1 --2- 172.21.15.47:0/3595827467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27bc15b1a0 0x7f27bc15b580 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:26.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.006+0000 7f27c0e00640 1 -- 172.21.15.47:0/3595827467 >> 172.21.15.47:0/3595827467 conn(0x7f27bc0b16a0 msgr2=0x7f27bc0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:26.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.006+0000 7f27c0e00640 1 -- 172.21.15.47:0/3595827467 shutdown_connections 2024-02-19T15:28:26.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.006+0000 7f27c0e00640 1 -- 172.21.15.47:0/3595827467 wait complete. 2024-02-19T15:28:26.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.006+0000 7f27c0e00640 1 Processor -- start 2024-02-19T15:28:26.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.006+0000 7f27c0e00640 1 -- start start 2024-02-19T15:28:26.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.006+0000 7f27c0e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27bc06c9a0 0x7f27bc14d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:26.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.006+0000 7f27c0e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27bc071350 0x7f27bc157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:26.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.006+0000 7f27c0e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27bc15b1a0 0x7f27bc1f6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:26.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.006+0000 7f27c0e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc14e630 con 0x7f27bc15b1a0 2024-02-19T15:28:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.006+0000 7f27c0e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0b4750 con 0x7f27bc06c9a0 2024-02-19T15:28:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.006+0000 7f27c0e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc14dab0 con 0x7f27bc071350 2024-02-19T15:28:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.010+0000 7f27ba800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27bc15b1a0 0x7f27bc1f6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.010+0000 7f27ba800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27bc15b1a0 0x7f27bc1f6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45212/0 (socket says 172.21.15.47:45212) 2024-02-19T15:28:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.010+0000 7f27ba800640 1 -- 172.21.15.47:0/1059717139 learned_addr learned my addr 172.21.15.47:0/1059717139 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.010+0000 7f27b9e00640 1 --2- 172.21.15.47:0/1059717139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27bc06c9a0 0x7f27bc14d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.010+0000 7f27b3e00640 1 --2- 172.21.15.47:0/1059717139 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27bc071350 0x7f27bc157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.010+0000 7f27ba800640 1 -- 172.21.15.47:0/1059717139 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27bc071350 msgr2=0x7f27bc157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.010+0000 7f27ba800640 1 --2- 172.21.15.47:0/1059717139 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27bc071350 0x7f27bc157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.010+0000 7f27ba800640 1 -- 172.21.15.47:0/1059717139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27bc06c9a0 msgr2=0x7f27bc14d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.010+0000 7f27ba800640 1 --2- 172.21.15.47:0/1059717139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27bc06c9a0 0x7f27bc14d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.010+0000 7f27ba800640 1 -- 172.21.15.47:0/1059717139 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b4009a70 con 0x7f27bc15b1a0 2024-02-19T15:28:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.010+0000 7f27ba800640 1 --2- 172.21.15.47:0/1059717139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27bc15b1a0 0x7f27bc1f6a60 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f27b400efc0 tx=0x7f27b400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:26.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.010+0000 7f27b2000640 1 -- 172.21.15.47:0/1059717139 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b4009cd0 con 0x7f27bc15b1a0 2024-02-19T15:28:26.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.010+0000 7f27b2000640 1 -- 172.21.15.47:0/1059717139 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b400f040 con 0x7f27bc15b1a0 2024-02-19T15:28:26.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.010+0000 7f27b2000640 1 -- 172.21.15.47:0/1059717139 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b40039a0 con 0x7f27bc15b1a0 2024-02-19T15:28:26.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.010+0000 7f27c0e00640 1 -- 172.21.15.47:0/1059717139 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27ac009420 con 0x7f27bc15b1a0 2024-02-19T15:28:26.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.010+0000 7f27c0e00640 1 -- 172.21.15.47:0/1059717139 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27bc14e070 con 0x7f27bc15b1a0 2024-02-19T15:28:26.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.010+0000 7f27c0e00640 1 -- 172.21.15.47:0/1059717139 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27bc0b27e0 con 0x7f27bc15b1a0 2024-02-19T15:28:26.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.010+0000 7f27b2000640 1 -- 172.21.15.47:0/1059717139 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f27b4014480 con 0x7f27bc15b1a0 2024-02-19T15:28:26.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.014+0000 7f27b2000640 1 --2- 172.21.15.47:0/1059717139 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2798066e40 0x7f2798069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:26.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.014+0000 7f27b2000640 1 -- 172.21.15.47:0/1059717139 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f27b40855e0 con 0x7f27bc15b1a0 2024-02-19T15:28:26.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.014+0000 7f27b9e00640 1 --2- 172.21.15.47:0/1059717139 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2798066e40 0x7f2798069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:26.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.014+0000 7f27b9e00640 1 --2- 172.21.15.47:0/1059717139 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2798066e40 0x7f2798069300 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f27ac0ad000 tx=0x7f27ac0a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:26.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.018+0000 7f27b2000640 1 -- 172.21.15.47:0/1059717139 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f27b4052380 con 0x7f27bc15b1a0 2024-02-19T15:28:26.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.222+0000 7f27c0e00640 1 -- 172.21.15.47:0/1059717139 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27bc075360 con 0x7f27bc15b1a0 2024-02-19T15:28:26.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.222+0000 7f27b2000640 1 -- 172.21.15.47:0/1059717139 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f27b4055a40 con 0x7f27bc15b1a0 2024-02-19T15:28:26.223 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:26.223 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:26.224 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:26.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.222+0000 7f27c0e00640 1 -- 172.21.15.47:0/1059717139 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2798066e40 msgr2=0x7f2798069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:26.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.222+0000 7f27c0e00640 1 --2- 172.21.15.47:0/1059717139 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2798066e40 0x7f2798069300 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f27ac0ad000 tx=0x7f27ac0a6000 comp rx=0 tx=0).stop 2024-02-19T15:28:26.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.222+0000 7f27c0e00640 1 -- 172.21.15.47:0/1059717139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27bc15b1a0 msgr2=0x7f27bc1f6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:26.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.222+0000 7f27c0e00640 1 --2- 172.21.15.47:0/1059717139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27bc15b1a0 0x7f27bc1f6a60 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f27b400efc0 tx=0x7f27b400c490 comp rx=0 tx=0).stop 2024-02-19T15:28:26.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.226+0000 7f27c0e00640 1 -- 172.21.15.47:0/1059717139 shutdown_connections 2024-02-19T15:28:26.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.226+0000 7f27c0e00640 1 --2- 172.21.15.47:0/1059717139 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2798066e40 0x7f2798069300 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:26.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.226+0000 7f27c0e00640 1 --2- 172.21.15.47:0/1059717139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27bc15b1a0 0x7f27bc1f6a60 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:26.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.226+0000 7f27c0e00640 1 --2- 172.21.15.47:0/1059717139 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27bc071350 0x7f27bc157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:26.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.226+0000 7f27c0e00640 1 --2- 172.21.15.47:0/1059717139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27bc06c9a0 0x7f27bc14d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:26.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.226+0000 7f27c0e00640 1 -- 172.21.15.47:0/1059717139 >> 172.21.15.47:0/1059717139 conn(0x7f27bc0b16a0 msgr2=0x7f27bc0af5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:26.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.226+0000 7f27c0e00640 1 -- 172.21.15.47:0/1059717139 shutdown_connections 2024-02-19T15:28:26.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.226+0000 7f27c0e00640 1 -- 172.21.15.47:0/1059717139 wait complete. 2024-02-19T15:28:26.245 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-02-19T15:28:26.409 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"num_strays":0}} 2024-02-19T15:28:26.409 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-02-19T15:28:26.409 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:26.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.546+0000 7ff4ba600640 1 Processor -- start 2024-02-19T15:28:26.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.546+0000 7ff4ba600640 1 -- start start 2024-02-19T15:28:26.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4ba600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff4b415e570 0x7ff4b40cc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:26.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4ba600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b40d2ad0 con 0x7ff4b40b2e20 2024-02-19T15:28:26.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4ba600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b40cc190 con 0x7ff4b40ce3c0 2024-02-19T15:28:26.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4ba600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4b40d0ac0 con 0x7ff4b415e570 2024-02-19T15:28:26.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b3e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff4b40b2e20 0x7ff4b40ccb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41494/0 (socket says 172.21.15.47:41494) 2024-02-19T15:28:26.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b3e00640 1 -- 172.21.15.47:0/2443128643 learned_addr learned my addr 172.21.15.47:0/2443128643 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:26.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4abe00640 1 --2- 172.21.15.47:0/2443128643 >> v2:172.21.15.116:3300/0 conn(0x7ff4b415e570 0x7ff4b40cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:26.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b2a00640 1 -- 172.21.15.47:0/2443128643 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612296987 0 0) 0x7ff4ac0014a0 con 0x7ff4b40b2e20 2024-02-19T15:28:26.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b2a00640 1 -- 172.21.15.47:0/2443128643 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff498003530 con 0x7ff4b40b2e20 2024-02-19T15:28:26.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b2a00640 1 -- 172.21.15.47:0/2443128643 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 724395075 0 0) 0x7ff498003530 con 0x7ff4b40b2e20 2024-02-19T15:28:26.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b2a00640 1 -- 172.21.15.47:0/2443128643 >> v2:172.21.15.116:3300/0 conn(0x7ff4b415e570 msgr2=0x7ff4b40cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:26.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b2a00640 1 --2- 172.21.15.47:0/2443128643 >> v2:172.21.15.116:3300/0 conn(0x7ff4b415e570 0x7ff4b40cc620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:26.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b2a00640 1 -- 172.21.15.47:0/2443128643 >> v1:172.21.15.116:6789/0 conn(0x7ff4b40ce3c0 legacy=0x7ff4b40ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:26.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b2a00640 1 -- 172.21.15.47:0/2443128643 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b40d0da0 con 0x7ff4b40b2e20 2024-02-19T15:28:26.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b2a00640 1 -- 172.21.15.47:0/2443128643 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7ff4ac001b90 con 0x7ff4b40b2e20 2024-02-19T15:28:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b2a00640 1 -- 172.21.15.47:0/2443128643 >> v1:172.21.15.47:6789/0 conn(0x7ff4b40b2e20 legacy=0x7ff4b40ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b2a00640 1 --2- 172.21.15.47:0/2443128643 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff498003980 0x7ff498003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b2a00640 1 --2- 172.21.15.47:0/2443128643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4980047d0 0x7ff498004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b2a00640 1 --2- 172.21.15.47:0/2443128643 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff498005190 0x7ff49800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b2a00640 1 -- 172.21.15.47:0/2443128643 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b40d0ac0 con 0x7ff498003980 2024-02-19T15:28:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b2a00640 1 -- 172.21.15.47:0/2443128643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49800c030 con 0x7ff4980047d0 2024-02-19T15:28:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b2a00640 1 -- 172.21.15.47:0/2443128643 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff49800c1f0 con 0x7ff498005190 2024-02-19T15:28:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b3400640 1 --2- 172.21.15.47:0/2443128643 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff498003980 0x7ff498003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4abe00640 1 --2- 172.21.15.47:0/2443128643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4980047d0 0x7ff498004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b3e00640 1 --2- 172.21.15.47:0/2443128643 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff498005190 0x7ff49800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b3400640 1 -- 172.21.15.47:0/2443128643 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff498005190 msgr2=0x7ff49800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b3400640 1 --2- 172.21.15.47:0/2443128643 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff498005190 0x7ff49800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b3400640 1 -- 172.21.15.47:0/2443128643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4980047d0 msgr2=0x7ff498004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b3400640 1 --2- 172.21.15.47:0/2443128643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4980047d0 0x7ff498004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b3400640 1 -- 172.21.15.47:0/2443128643 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff498003530 con 0x7ff498003980 2024-02-19T15:28:26.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.550+0000 7ff4b3e00640 1 -- 172.21.15.47:0/2443128643 reap_dead start 2024-02-19T15:28:26.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.554+0000 7ff4b3400640 1 --2- 172.21.15.47:0/2443128643 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff498003980 0x7ff498003da0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4001780 tx=0x7ff4a401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff67255f9c4e46 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:26.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4b2a00640 1 -- 172.21.15.47:0/2443128643 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a40052d0 con 0x7ff498003980 2024-02-19T15:28:26.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4b2a00640 1 -- 172.21.15.47:0/2443128643 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a40a7050 con 0x7ff498003980 2024-02-19T15:28:26.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4b2a00640 1 -- 172.21.15.47:0/2443128643 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4003600 con 0x7ff498003980 2024-02-19T15:28:26.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4ba600640 1 -- 172.21.15.47:0/2443128643 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff498003980 msgr2=0x7ff498003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:26.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4ba600640 1 --2- 172.21.15.47:0/2443128643 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff498003980 0x7ff498003da0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4001780 tx=0x7ff4a401c400 comp rx=0 tx=0).stop 2024-02-19T15:28:26.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4ba600640 1 -- 172.21.15.47:0/2443128643 shutdown_connections 2024-02-19T15:28:26.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4ba600640 1 --2- 172.21.15.47:0/2443128643 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff498003980 0x7ff498003da0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:26.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4ba600640 1 -- 172.21.15.47:0/2443128643 >> 172.21.15.47:0/2443128643 conn(0x7ff4b40d1b40 msgr2=0x7ff4b40d1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:26.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4ba600640 1 -- 172.21.15.47:0/2443128643 shutdown_connections 2024-02-19T15:28:26.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4ba600640 1 -- 172.21.15.47:0/2443128643 wait complete. 2024-02-19T15:28:26.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4ba600640 1 Processor -- start 2024-02-19T15:28:26.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4ba600640 1 -- start start 2024-02-19T15:28:26.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4ba600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff498003980 0x7ff4b41590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4ba600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4b41534c0 0x7ff4b4153930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4ba600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff4b41545b0 0x7ff4b4153e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4ba600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b4159a70 con 0x7ff4b41545b0 2024-02-19T15:28:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4ba600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b4159d50 con 0x7ff498003980 2024-02-19T15:28:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4ba600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b415a030 con 0x7ff4b41534c0 2024-02-19T15:28:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4b3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff4b41545b0 0x7ff4b4153e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4b3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff4b41545b0 0x7ff4b4153e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45230/0 (socket says 172.21.15.47:45230) 2024-02-19T15:28:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4b3e00640 1 -- 172.21.15.47:0/3527419863 learned_addr learned my addr 172.21.15.47:0/3527419863 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4b3400640 1 --2- 172.21.15.47:0/3527419863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff498003980 0x7ff4b41590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.558+0000 7ff4abe00640 1 --2- 172.21.15.47:0/3527419863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4b41534c0 0x7ff4b4153930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.562+0000 7ff4b3e00640 1 -- 172.21.15.47:0/3527419863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4b41534c0 msgr2=0x7ff4b4153930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.562+0000 7ff4b3e00640 1 --2- 172.21.15.47:0/3527419863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4b41534c0 0x7ff4b4153930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.562+0000 7ff4b3e00640 1 -- 172.21.15.47:0/3527419863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff498003980 msgr2=0x7ff4b41590f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:26.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.562+0000 7ff4b3e00640 1 --2- 172.21.15.47:0/3527419863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff498003980 0x7ff4b41590f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:26.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.562+0000 7ff4b3e00640 1 -- 172.21.15.47:0/3527419863 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b415a310 con 0x7ff4b41545b0 2024-02-19T15:28:26.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.562+0000 7ff4b3400640 1 --2- 172.21.15.47:0/3527419863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff498003980 0x7ff4b41590f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:26.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.562+0000 7ff4b3e00640 1 --2- 172.21.15.47:0/3527419863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff4b41545b0 0x7ff4b4153e70 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0021b0 tx=0x7ff4ac00ae20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:26.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.562+0000 7ff4ab400640 1 -- 172.21.15.47:0/3527419863 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac0077a0 con 0x7ff4b41545b0 2024-02-19T15:28:26.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.562+0000 7ff4ab400640 1 -- 172.21.15.47:0/3527419863 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4ac007940 con 0x7ff4b41545b0 2024-02-19T15:28:26.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.562+0000 7ff4ab400640 1 -- 172.21.15.47:0/3527419863 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac013630 con 0x7ff4b41545b0 2024-02-19T15:28:26.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.562+0000 7ff4ba600640 1 -- 172.21.15.47:0/3527419863 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b415a5f0 con 0x7ff4b41545b0 2024-02-19T15:28:26.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.562+0000 7ff4ba600640 1 -- 172.21.15.47:0/3527419863 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b4159800 con 0x7ff4b41545b0 2024-02-19T15:28:26.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.562+0000 7ff4ba600640 1 -- 172.21.15.47:0/3527419863 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b4075360 con 0x7ff4b41545b0 2024-02-19T15:28:26.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.562+0000 7ff4ab400640 1 -- 172.21.15.47:0/3527419863 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff4ac013790 con 0x7ff4b41545b0 2024-02-19T15:28:26.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.562+0000 7ff4ab400640 1 --2- 172.21.15.47:0/3527419863 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff490066e70 0x7ff490069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:26.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.562+0000 7ff4ab400640 1 -- 172.21.15.47:0/3527419863 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7ff4ac085260 con 0x7ff4b41545b0 2024-02-19T15:28:26.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.566+0000 7ff4b3400640 1 --2- 172.21.15.47:0/3527419863 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff490066e70 0x7ff490069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:26.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.566+0000 7ff4b3400640 1 --2- 172.21.15.47:0/3527419863 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff490066e70 0x7ff490069330 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4001750 tx=0x7ff4a40011a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:26.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.566+0000 7ff4ab400640 1 -- 172.21.15.47:0/3527419863 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff4ac051ec0 con 0x7ff4b41545b0 2024-02-19T15:28:26.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.762+0000 7ff4ba600640 1 -- 172.21.15.47:0/3527419863 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4b40d2ad0 con 0x7ff4b41545b0 2024-02-19T15:28:26.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.766+0000 7ff4ab400640 1 -- 172.21.15.47:0/3527419863 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7ff4b40d2ad0 con 0x7ff4b41545b0 2024-02-19T15:28:26.766 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:26.766 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:26.766 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:26.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.766+0000 7ff4ba600640 1 -- 172.21.15.47:0/3527419863 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff490066e70 msgr2=0x7ff490069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:26.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.766+0000 7ff4ba600640 1 --2- 172.21.15.47:0/3527419863 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff490066e70 0x7ff490069330 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4001750 tx=0x7ff4a40011a0 comp rx=0 tx=0).stop 2024-02-19T15:28:26.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.766+0000 7ff4ba600640 1 -- 172.21.15.47:0/3527419863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff4b41545b0 msgr2=0x7ff4b4153e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:26.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.766+0000 7ff4ba600640 1 --2- 172.21.15.47:0/3527419863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff4b41545b0 0x7ff4b4153e70 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0021b0 tx=0x7ff4ac00ae20 comp rx=0 tx=0).stop 2024-02-19T15:28:26.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.766+0000 7ff4ba600640 1 -- 172.21.15.47:0/3527419863 shutdown_connections 2024-02-19T15:28:26.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.766+0000 7ff4ba600640 1 --2- 172.21.15.47:0/3527419863 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff490066e70 0x7ff490069330 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:26.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.766+0000 7ff4ba600640 1 --2- 172.21.15.47:0/3527419863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff4b41545b0 0x7ff4b4153e70 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:26.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.766+0000 7ff4ba600640 1 --2- 172.21.15.47:0/3527419863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4b41534c0 0x7ff4b4153930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:26.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.766+0000 7ff4ba600640 1 --2- 172.21.15.47:0/3527419863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff498003980 0x7ff4b41590f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:26.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.766+0000 7ff4ba600640 1 -- 172.21.15.47:0/3527419863 >> 172.21.15.47:0/3527419863 conn(0x7ff4b40d1b40 msgr2=0x7ff4b415d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:26.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.766+0000 7ff4ba600640 1 -- 172.21.15.47:0/3527419863 shutdown_connections 2024-02-19T15:28:26.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:26.766+0000 7ff4ba600640 1 -- 172.21.15.47:0/3527419863 wait complete. 2024-02-19T15:28:26.787 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_reintegrated 2024-02-19T15:28:26.947 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"strays_reintegrated":2}} 2024-02-19T15:28:26.947 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 2 } } 2024-02-19T15:28:26.947 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:27.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.086+0000 7f6a7de00640 1 Processor -- start 2024-02-19T15:28:27.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.086+0000 7f6a7de00640 1 -- start start 2024-02-19T15:28:27.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.086+0000 7f6a7de00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a7815bef0 0x7f6a780b2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.086+0000 7f6a7de00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6a7806b4f0 0x7f6a7806b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.086+0000 7f6a7de00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a780b2d40 con 0x7f6a780b5d90 2024-02-19T15:28:27.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.086+0000 7f6a7de00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a7806bec0 con 0x7f6a7806b4f0 2024-02-19T15:28:27.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.086+0000 7f6a7de00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a780d00f0 con 0x7f6a7815bef0 2024-02-19T15:28:27.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.086+0000 7f6a77400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6a7806b4f0 0x7f6a7806b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.086+0000 7f6a76000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6a780b5d90 0x7f6a780d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41502/0 (socket says 172.21.15.47:41502) 2024-02-19T15:28:27.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.086+0000 7f6a76000640 1 -- 172.21.15.47:0/2746779845 learned_addr learned my addr 172.21.15.47:0/2746779845 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:27.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.086+0000 7f6a76a00640 1 --2- 172.21.15.47:0/2746779845 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a7815bef0 0x7f6a780b2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.086+0000 7f6a75600640 1 -- 172.21.15.47:0/2746779845 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1651643099 0 0) 0x7f6a780b2d40 con 0x7f6a780b5d90 2024-02-19T15:28:27.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.086+0000 7f6a77400640 1 -- 172.21.15.47:0/2746779845 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a7815bef0 msgr2=0x7f6a780b2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:27.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.086+0000 7f6a77400640 1 --2- 172.21.15.47:0/2746779845 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a7815bef0 0x7f6a780b2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.086+0000 7f6a77400640 1 -- 172.21.15.47:0/2746779845 >> v1:172.21.15.47:6789/0 conn(0x7f6a780b5d90 legacy=0x7f6a780d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:27.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.086+0000 7f6a77400640 1 -- 172.21.15.47:0/2746779845 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a780af690 con 0x7f6a7806b4f0 2024-02-19T15:28:27.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.090+0000 7f6a77400640 1 --2- 172.21.15.47:0/2746779845 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a7806b4f0 0x7f6a7806b980 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f6a70009dc0 tx=0x7f6a7001c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2baaef6f97eff4d2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:27.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.090+0000 7f6a75600640 1 -- 172.21.15.47:0/2746779845 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a700a5950 con 0x7f6a7806b4f0 2024-02-19T15:28:27.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.090+0000 7f6a75600640 1 -- 172.21.15.47:0/2746779845 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a700a5ab0 con 0x7f6a7806b4f0 2024-02-19T15:28:27.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.090+0000 7f6a75600640 1 -- 172.21.15.47:0/2746779845 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a700a5e40 con 0x7f6a7806b4f0 2024-02-19T15:28:27.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.090+0000 7f6a7de00640 1 -- 172.21.15.47:0/2746779845 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a7806b4f0 msgr2=0x7f6a7806b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:27.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.090+0000 7f6a7de00640 1 --2- 172.21.15.47:0/2746779845 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a7806b4f0 0x7f6a7806b980 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f6a70009dc0 tx=0x7f6a7001c890 comp rx=0 tx=0).stop 2024-02-19T15:28:27.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.090+0000 7f6a7de00640 1 -- 172.21.15.47:0/2746779845 shutdown_connections 2024-02-19T15:28:27.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.090+0000 7f6a7de00640 1 --2- 172.21.15.47:0/2746779845 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a7806b4f0 0x7f6a7806b980 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.090+0000 7f6a7de00640 1 --2- 172.21.15.47:0/2746779845 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a7815bef0 0x7f6a780b2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.090+0000 7f6a7de00640 1 -- 172.21.15.47:0/2746779845 >> 172.21.15.47:0/2746779845 conn(0x7f6a780b4fc0 msgr2=0x7f6a780b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:27.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.090+0000 7f6a7de00640 1 -- 172.21.15.47:0/2746779845 shutdown_connections 2024-02-19T15:28:27.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.090+0000 7f6a7de00640 1 -- 172.21.15.47:0/2746779845 wait complete. 2024-02-19T15:28:27.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a7de00640 1 Processor -- start 2024-02-19T15:28:27.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a7de00640 1 -- start start 2024-02-19T15:28:27.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a7de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a7806b4f0 0x7f6a780ad260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a7de00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a780b5d90 0x7f6a780997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a7de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a7815bef0 0x7f6a780aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a7de00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a780ad7a0 con 0x7f6a7815bef0 2024-02-19T15:28:27.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a7de00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a780ad9d0 con 0x7f6a7806b4f0 2024-02-19T15:28:27.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a7de00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a780adcb0 con 0x7f6a780b5d90 2024-02-19T15:28:27.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a77400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a7815bef0 0x7f6a780aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a77400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a7815bef0 0x7f6a780aea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45250/0 (socket says 172.21.15.47:45250) 2024-02-19T15:28:27.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a77400640 1 -- 172.21.15.47:0/151839359 learned_addr learned my addr 172.21.15.47:0/151839359 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:27.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a76a00640 1 --2- 172.21.15.47:0/151839359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a7806b4f0 0x7f6a780ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a76000640 1 --2- 172.21.15.47:0/151839359 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a780b5d90 0x7f6a780997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a77400640 1 -- 172.21.15.47:0/151839359 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a780b5d90 msgr2=0x7f6a780997e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:27.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a77400640 1 --2- 172.21.15.47:0/151839359 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a780b5d90 0x7f6a780997e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a77400640 1 -- 172.21.15.47:0/151839359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a7806b4f0 msgr2=0x7f6a780ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:27.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a77400640 1 --2- 172.21.15.47:0/151839359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a7806b4f0 0x7f6a780ad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a77400640 1 -- 172.21.15.47:0/151839359 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a70009a70 con 0x7f6a7815bef0 2024-02-19T15:28:27.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a77400640 1 --2- 172.21.15.47:0/151839359 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a7815bef0 0x7f6a780aea70 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f6a7001a4c0 tx=0x7f6a70019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:27.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a6f400640 1 -- 172.21.15.47:0/151839359 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a700addf0 con 0x7f6a7815bef0 2024-02-19T15:28:27.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a6f400640 1 -- 172.21.15.47:0/151839359 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a70009450 con 0x7f6a7815bef0 2024-02-19T15:28:27.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a6f400640 1 -- 172.21.15.47:0/151839359 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a70003ba0 con 0x7f6a7815bef0 2024-02-19T15:28:27.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a7de00640 1 -- 172.21.15.47:0/151839359 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a780adf90 con 0x7f6a7815bef0 2024-02-19T15:28:27.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.094+0000 7f6a7de00640 1 -- 172.21.15.47:0/151839359 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a780ae550 con 0x7f6a7815bef0 2024-02-19T15:28:27.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.098+0000 7f6a7de00640 1 -- 172.21.15.47:0/151839359 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a780ae740 con 0x7f6a7815bef0 2024-02-19T15:28:27.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.098+0000 7f6a6f400640 1 -- 172.21.15.47:0/151839359 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6a700b9450 con 0x7f6a7815bef0 2024-02-19T15:28:27.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.098+0000 7f6a6f400640 1 --2- 172.21.15.47:0/151839359 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6a58066e40 0x7f6a58069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.098+0000 7f6a6f400640 1 -- 172.21.15.47:0/151839359 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f6a7011fd60 con 0x7f6a7815bef0 2024-02-19T15:28:27.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.098+0000 7f6a76a00640 1 --2- 172.21.15.47:0/151839359 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6a58066e40 0x7f6a58069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.102+0000 7f6a76a00640 1 --2- 172.21.15.47:0/151839359 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6a58066e40 0x7f6a58069300 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009ac0 tx=0x7f6a68009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:27.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.102+0000 7f6a6f400640 1 -- 172.21.15.47:0/151839359 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6a700eca70 con 0x7f6a7815bef0 2024-02-19T15:28:27.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.298+0000 7f6a7de00640 1 -- 172.21.15.47:0/151839359 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a78075360 con 0x7f6a7815bef0 2024-02-19T15:28:27.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.298+0000 7f6a6f400640 1 -- 172.21.15.47:0/151839359 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f6a700f0130 con 0x7f6a7815bef0 2024-02-19T15:28:27.301 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:27.301 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:27.301 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:27.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.302+0000 7f6a7de00640 1 -- 172.21.15.47:0/151839359 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6a58066e40 msgr2=0x7f6a58069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:27.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.302+0000 7f6a7de00640 1 --2- 172.21.15.47:0/151839359 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6a58066e40 0x7f6a58069300 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009ac0 tx=0x7f6a68009190 comp rx=0 tx=0).stop 2024-02-19T15:28:27.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.302+0000 7f6a7de00640 1 -- 172.21.15.47:0/151839359 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a7815bef0 msgr2=0x7f6a780aea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:27.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.302+0000 7f6a7de00640 1 --2- 172.21.15.47:0/151839359 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a7815bef0 0x7f6a780aea70 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f6a7001a4c0 tx=0x7f6a70019630 comp rx=0 tx=0).stop 2024-02-19T15:28:27.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.302+0000 7f6a7de00640 1 -- 172.21.15.47:0/151839359 shutdown_connections 2024-02-19T15:28:27.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.302+0000 7f6a7de00640 1 --2- 172.21.15.47:0/151839359 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6a58066e40 0x7f6a58069300 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.302+0000 7f6a7de00640 1 --2- 172.21.15.47:0/151839359 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a7815bef0 0x7f6a780aea70 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.302+0000 7f6a7de00640 1 --2- 172.21.15.47:0/151839359 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a780b5d90 0x7f6a780997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.302+0000 7f6a7de00640 1 --2- 172.21.15.47:0/151839359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a7806b4f0 0x7f6a780ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.302+0000 7f6a7de00640 1 -- 172.21.15.47:0/151839359 >> 172.21.15.47:0/151839359 conn(0x7f6a780b4fc0 msgr2=0x7f6a78158d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:27.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.302+0000 7f6a7de00640 1 -- 172.21.15.47:0/151839359 shutdown_connections 2024-02-19T15:28:27.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.302+0000 7f6a7de00640 1 -- 172.21.15.47:0/151839359 wait complete. 2024-02-19T15:28:27.322 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-02-19T15:28:27.490 INFO:teuthology.orchestra.run.smithi047.stdout:{"message":"","return_code":0} 2024-02-19T15:28:27.490 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-02-19T15:28:27.490 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-02-19T15:28:27.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.534+0000 7f25b7849080 1 Processor -- start 2024-02-19T15:28:27.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b7849080 1 -- start start 2024-02-19T15:28:27.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b7849080 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c81579b990 0x55c8157bb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b7849080 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c81579bd70 0x55c81579e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b7849080 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c81579ed90 0x55c81579a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b7849080 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c815692f90 con 0x55c81579bd70 2024-02-19T15:28:27.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b7849080 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c81562a1e0 con 0x55c81579ed90 2024-02-19T15:28:27.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b7849080 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c815629e80 con 0x55c81579b990 2024-02-19T15:28:27.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b3600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c81579bd70 0x55c81579e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b3600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c81579bd70 0x55c81579e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45262/0 (socket says 172.21.15.47:45262) 2024-02-19T15:28:27.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b3600640 1 -- 172.21.15.47:0/1772339733 learned_addr learned my addr 172.21.15.47:0/1772339733 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:27.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b4000640 1 --2- 172.21.15.47:0/1772339733 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c81579b990 0x55c8157bb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b4a00640 1 --2- 172.21.15.47:0/1772339733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c81579ed90 0x55c81579a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b3600640 1 -- 172.21.15.47:0/1772339733 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c81579b990 msgr2=0x55c8157bb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:27.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b3600640 1 --2- 172.21.15.47:0/1772339733 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c81579b990 0x55c8157bb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b3600640 1 -- 172.21.15.47:0/1772339733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c81579ed90 msgr2=0x55c81579a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:27.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b3600640 1 --2- 172.21.15.47:0/1772339733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c81579ed90 0x55c81579a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b3600640 1 -- 172.21.15.47:0/1772339733 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac00a100 con 0x55c81579bd70 2024-02-19T15:28:27.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b3600640 1 --2- 172.21.15.47:0/1772339733 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c81579bd70 0x55c81579e470 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f25a0001d60 tx=0x7f25a000b6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c760c120277d4f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:27.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b2200640 1 -- 172.21.15.47:0/1772339733 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a000bea0 con 0x55c81579bd70 2024-02-19T15:28:27.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b2200640 1 -- 172.21.15.47:0/1772339733 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a000e690 con 0x55c81579bd70 2024-02-19T15:28:27.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b2200640 1 -- 172.21.15.47:0/1772339733 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a000ea40 con 0x55c81579bd70 2024-02-19T15:28:27.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b7849080 1 -- 172.21.15.47:0/1772339733 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c81579bd70 msgr2=0x55c81579e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:27.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b7849080 1 --2- 172.21.15.47:0/1772339733 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c81579bd70 0x55c81579e470 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f25a0001d60 tx=0x7f25a000b6d0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b7849080 1 -- 172.21.15.47:0/1772339733 shutdown_connections 2024-02-19T15:28:27.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b7849080 1 --2- 172.21.15.47:0/1772339733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c81579ed90 0x55c81579a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b7849080 1 --2- 172.21.15.47:0/1772339733 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c81579bd70 0x55c81579e470 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b7849080 1 --2- 172.21.15.47:0/1772339733 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c81579b990 0x55c8157bb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b7849080 1 -- 172.21.15.47:0/1772339733 >> 172.21.15.47:0/1772339733 conn(0x55c8156e3670 msgr2=0x55c81579b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:27.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b7849080 1 -- 172.21.15.47:0/1772339733 shutdown_connections 2024-02-19T15:28:27.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b7849080 1 -- 172.21.15.47:0/1772339733 wait complete. 2024-02-19T15:28:27.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b7849080 1 Processor -- start 2024-02-19T15:28:27.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.538+0000 7f25b7849080 1 -- start start 2024-02-19T15:28:27.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b7849080 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c81579b990 0x55c815729a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b7849080 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c81579bd70 0x55c815727150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b7849080 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c81579ed90 0x55c8157b3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b7849080 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c81564d610 con 0x55c81579bd70 2024-02-19T15:28:27.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b7849080 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c81561d740 con 0x55c81579b990 2024-02-19T15:28:27.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b7849080 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c815727940 con 0x55c81579ed90 2024-02-19T15:28:27.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b3600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c81579bd70 0x55c815727150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b3600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c81579bd70 0x55c815727150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45278/0 (socket says 172.21.15.47:45278) 2024-02-19T15:28:27.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b3600640 1 -- 172.21.15.47:0/1379533662 learned_addr learned my addr 172.21.15.47:0/1379533662 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:27.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b4000640 1 --2- 172.21.15.47:0/1379533662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c81579b990 0x55c815729a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b4a00640 1 --2- 172.21.15.47:0/1379533662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c81579ed90 0x55c8157b3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b3600640 1 -- 172.21.15.47:0/1379533662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c81579ed90 msgr2=0x55c8157b3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:27.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b3600640 1 --2- 172.21.15.47:0/1379533662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55c81579ed90 0x55c8157b3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b3600640 1 -- 172.21.15.47:0/1379533662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c81579b990 msgr2=0x55c815729a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:27.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b3600640 1 --2- 172.21.15.47:0/1379533662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c81579b990 0x55c815729a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b3600640 1 -- 172.21.15.47:0/1379533662 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2598001e20 con 0x55c81579bd70 2024-02-19T15:28:27.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b3600640 1 --2- 172.21.15.47:0/1379533662 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c81579bd70 0x55c815727150 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f25a000bbd0 tx=0x7f25a0000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:27.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b0e00640 1 -- 172.21.15.47:0/1379533662 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a001e070 con 0x55c81579bd70 2024-02-19T15:28:27.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b7849080 1 -- 172.21.15.47:0/1379533662 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c815628f30 con 0x55c81579bd70 2024-02-19T15:28:27.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b0e00640 1 -- 172.21.15.47:0/1379533662 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a000bea0 con 0x55c81579bd70 2024-02-19T15:28:27.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b0e00640 1 -- 172.21.15.47:0/1379533662 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a00192e0 con 0x55c81579bd70 2024-02-19T15:28:27.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.542+0000 7f25b7849080 1 -- 172.21.15.47:0/1379533662 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c815628990 con 0x55c81579bd70 2024-02-19T15:28:27.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b0e00640 1 -- 172.21.15.47:0/1379533662 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f25a000a040 con 0x55c81579bd70 2024-02-19T15:28:27.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b0e00640 1 --2- 172.21.15.47:0/1379533662 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2594065850 0x7f2594067d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b0e00640 1 -- 172.21.15.47:0/1379533662 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f25a00849f0 con 0x55c81579bd70 2024-02-19T15:28:27.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b7849080 1 --2- 172.21.15.47:0/1379533662 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x55c8157b3cc0 0x55c81579ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b7849080 1 -- 172.21.15.47:0/1379533662 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v8 -- 0x55c815724fe0 con 0x55c8157b3cc0 2024-02-19T15:28:27.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b4000640 1 --2- 172.21.15.47:0/1379533662 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2594065850 0x7f2594067d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b4a00640 1 --2- 172.21.15.47:0/1379533662 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x55c8157b3cc0 0x55c81579ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b4000640 1 --2- 172.21.15.47:0/1379533662 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2594065850 0x7f2594067d10 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f25a4011570 tx=0x7f25a40017a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:27.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b4a00640 1 --2- 172.21.15.47:0/1379533662 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x55c8157b3cc0 0x55c81579ad90 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:27.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b4a00640 1 -- 172.21.15.47:0/1379533662 <== osd.0 v2:172.21.15.47:6809/1968337531 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f25ac0ab5b0 con 0x55c8157b3cc0 2024-02-19T15:28:27.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b7849080 1 -- 172.21.15.47:0/1379533662 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x55c8157b3cc0 msgr2=0x55c81579ad90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:27.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b7849080 1 --2- 172.21.15.47:0/1379533662 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x55c8157b3cc0 0x55c81579ad90 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b7849080 1 -- 172.21.15.47:0/1379533662 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2594065850 msgr2=0x7f2594067d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:27.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b7849080 1 --2- 172.21.15.47:0/1379533662 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2594065850 0x7f2594067d10 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f25a4011570 tx=0x7f25a40017a0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b7849080 1 -- 172.21.15.47:0/1379533662 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c81579bd70 msgr2=0x55c815727150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:27.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b7849080 1 --2- 172.21.15.47:0/1379533662 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c81579bd70 0x55c815727150 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f25a000bbd0 tx=0x7f25a0000ee0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b4a00640 1 -- 172.21.15.47:0/1379533662 reap_dead start 2024-02-19T15:28:27.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b7849080 1 -- 172.21.15.47:0/1379533662 shutdown_connections 2024-02-19T15:28:27.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b7849080 1 -- 172.21.15.47:0/1379533662 >> 172.21.15.47:0/1379533662 conn(0x55c8156e3670 msgr2=0x55c8157ba910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:27.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b7849080 1 -- 172.21.15.47:0/1379533662 shutdown_connections 2024-02-19T15:28:27.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.546+0000 7f25b7849080 1 -- 172.21.15.47:0/1379533662 wait complete. 2024-02-19T15:28:27.553 INFO:teuthology.orchestra.run.smithi047.stdout:W�file_ddir_2. 2024-02-19T15:28:27.553 DEBUG:teuthology.orchestra.run.smithi047:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-02-19T15:28:27.601 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:28:27.601 INFO:teuthology.orchestra.run.smithi047.stdout: "ino": 1099511627777, 2024-02-19T15:28:27.601 INFO:teuthology.orchestra.run.smithi047.stdout: "ancestors": [ 2024-02-19T15:28:27.601 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:27.601 INFO:teuthology.orchestra.run.smithi047.stdout: "dirino": 1099511628278, 2024-02-19T15:28:27.601 INFO:teuthology.orchestra.run.smithi047.stdout: "dname": "file_d", 2024-02-19T15:28:27.602 INFO:teuthology.orchestra.run.smithi047.stdout: "version": 27 2024-02-19T15:28:27.602 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:27.602 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:27.602 INFO:teuthology.orchestra.run.smithi047.stdout: "dirino": 1, 2024-02-19T15:28:27.602 INFO:teuthology.orchestra.run.smithi047.stdout: "dname": "dir_2", 2024-02-19T15:28:27.602 INFO:teuthology.orchestra.run.smithi047.stdout: "version": 46 2024-02-19T15:28:27.602 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:27.602 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-02-19T15:28:27.602 INFO:teuthology.orchestra.run.smithi047.stdout: "pool": 17, 2024-02-19T15:28:27.602 INFO:teuthology.orchestra.run.smithi047.stdout: "old_pools": [] 2024-02-19T15:28:27.602 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:28:27.602 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:27.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:28:27.605 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_d) 2024-02-19T15:28:27.659 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:27.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.838+0000 7f1d70c00640 1 Processor -- start 2024-02-19T15:28:27.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.838+0000 7f1d70c00640 1 -- start start 2024-02-19T15:28:27.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.838+0000 7f1d70c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1d6c15bcd0 0x7f1d6c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.838+0000 7f1d70c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d6c0cedb0 0x7f1d6c0cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.838+0000 7f1d70c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1d6c0d2710 0x7f1d6c0d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.838+0000 7f1d70c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6c0cf5e0 con 0x7f1d6c0d2710 2024-02-19T15:28:27.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.838+0000 7f1d70c00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6c0cc700 con 0x7f1d6c15bcd0 2024-02-19T15:28:27.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.838+0000 7f1d70c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c0cf300 con 0x7f1d6c0cedb0 2024-02-19T15:28:27.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.838+0000 7f1d6a800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1d6c0d2710 0x7f1d6c0d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.838+0000 7f1d6a800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1d6c0d2710 0x7f1d6c0d0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45292/0 (socket says 172.21.15.47:45292) 2024-02-19T15:28:27.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.838+0000 7f1d6a800640 1 -- 172.21.15.47:0/395411621 learned_addr learned my addr 172.21.15.47:0/395411621 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:27.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.838+0000 7f1d69e00640 1 --2- 172.21.15.47:0/395411621 >> v2:172.21.15.116:3300/0 conn(0x7f1d6c15bcd0 0x7f1d6c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.838+0000 7f1d63e00640 1 --2- 172.21.15.47:0/395411621 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d6c0cedb0 0x7f1d6c0cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.838+0000 7f1d6a800640 1 -- 172.21.15.47:0/395411621 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d6c0cedb0 msgr2=0x7f1d6c0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:27.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.838+0000 7f1d6a800640 1 --2- 172.21.15.47:0/395411621 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d6c0cedb0 0x7f1d6c0cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.838+0000 7f1d6a800640 1 -- 172.21.15.47:0/395411621 >> v2:172.21.15.116:3300/0 conn(0x7f1d6c15bcd0 msgr2=0x7f1d6c0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:27.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.838+0000 7f1d6a800640 1 --2- 172.21.15.47:0/395411621 >> v2:172.21.15.116:3300/0 conn(0x7f1d6c15bcd0 0x7f1d6c0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.838+0000 7f1d6a800640 1 -- 172.21.15.47:0/395411621 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c0cc9a0 con 0x7f1d6c0d2710 2024-02-19T15:28:27.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.842+0000 7f1d6a800640 1 --2- 172.21.15.47:0/395411621 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d6c0d2710 0x7f1d6c0d0190 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c071a60 tx=0x7f1d6401c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eca19b7d194f86c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:27.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.842+0000 7f1d69400640 1 -- 172.21.15.47:0/395411621 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d640a5840 con 0x7f1d6c0d2710 2024-02-19T15:28:27.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.842+0000 7f1d69400640 1 -- 172.21.15.47:0/395411621 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d640a59a0 con 0x7f1d6c0d2710 2024-02-19T15:28:27.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.842+0000 7f1d69400640 1 -- 172.21.15.47:0/395411621 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d640a5d30 con 0x7f1d6c0d2710 2024-02-19T15:28:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.842+0000 7f1d70c00640 1 -- 172.21.15.47:0/395411621 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d6c0d2710 msgr2=0x7f1d6c0d0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.842+0000 7f1d70c00640 1 --2- 172.21.15.47:0/395411621 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d6c0d2710 0x7f1d6c0d0190 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c071a60 tx=0x7f1d6401c9c0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.842+0000 7f1d70c00640 1 -- 172.21.15.47:0/395411621 shutdown_connections 2024-02-19T15:28:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.842+0000 7f1d70c00640 1 --2- 172.21.15.47:0/395411621 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d6c0d2710 0x7f1d6c0d0190 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.842+0000 7f1d70c00640 1 --2- 172.21.15.47:0/395411621 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d6c0cedb0 0x7f1d6c0cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.842+0000 7f1d70c00640 1 --2- 172.21.15.47:0/395411621 >> v2:172.21.15.116:3300/0 conn(0x7f1d6c15bcd0 0x7f1d6c0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.842+0000 7f1d70c00640 1 -- 172.21.15.47:0/395411621 >> 172.21.15.47:0/395411621 conn(0x7f1d6c0d1bf0 msgr2=0x7f1d6c0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.842+0000 7f1d70c00640 1 -- 172.21.15.47:0/395411621 shutdown_connections 2024-02-19T15:28:27.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.842+0000 7f1d70c00640 1 -- 172.21.15.47:0/395411621 wait complete. 2024-02-19T15:28:27.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d70c00640 1 Processor -- start 2024-02-19T15:28:27.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d70c00640 1 -- start start 2024-02-19T15:28:27.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d70c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d6c0cedb0 0x7f1d6c0c59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d70c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d6c0d2710 0x7f1d6c0b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d70c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d6c15bcd0 0x7f1d6c0ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d70c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c0b7660 con 0x7f1d6c0cedb0 2024-02-19T15:28:27.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d70c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c0b4790 con 0x7f1d6c15bcd0 2024-02-19T15:28:27.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d70c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c1fa810 con 0x7f1d6c0d2710 2024-02-19T15:28:27.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d69e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d6c0cedb0 0x7f1d6c0c59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d69e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d6c0cedb0 0x7f1d6c0c59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45298/0 (socket says 172.21.15.47:45298) 2024-02-19T15:28:27.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d69e00640 1 -- 172.21.15.47:0/2091019493 learned_addr learned my addr 172.21.15.47:0/2091019493 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:27.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d69e00640 1 -- 172.21.15.47:0/2091019493 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d6c0d2710 msgr2=0x7f1d6c0b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:27.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d63e00640 1 --2- 172.21.15.47:0/2091019493 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d6c0d2710 0x7f1d6c0b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d6a800640 1 --2- 172.21.15.47:0/2091019493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d6c15bcd0 0x7f1d6c0ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d69e00640 1 --2- 172.21.15.47:0/2091019493 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d6c0d2710 0x7f1d6c0b7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d69e00640 1 -- 172.21.15.47:0/2091019493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d6c15bcd0 msgr2=0x7f1d6c0ca800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:27.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d69e00640 1 --2- 172.21.15.47:0/2091019493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d6c15bcd0 0x7f1d6c0ca800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:27.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d69e00640 1 -- 172.21.15.47:0/2091019493 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d64009b30 con 0x7f1d6c0cedb0 2024-02-19T15:28:27.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d6a800640 1 --2- 172.21.15.47:0/2091019493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d6c15bcd0 0x7f1d6c0ca800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:27.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d63e00640 1 --2- 172.21.15.47:0/2091019493 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d6c0d2710 0x7f1d6c0b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:27.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d69e00640 1 --2- 172.21.15.47:0/2091019493 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d6c0cedb0 0x7f1d6c0c59e0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c00e870 tx=0x7f1d5c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:27.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d62000640 1 -- 172.21.15.47:0/2091019493 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c009510 con 0x7f1d6c0cedb0 2024-02-19T15:28:27.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d62000640 1 -- 172.21.15.47:0/2091019493 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5c00eec0 con 0x7f1d6c0cedb0 2024-02-19T15:28:27.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d70c00640 1 -- 172.21.15.47:0/2091019493 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d6c0cdb40 con 0x7f1d6c0cedb0 2024-02-19T15:28:27.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d62000640 1 -- 172.21.15.47:0/2091019493 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c00cb40 con 0x7f1d6c0cedb0 2024-02-19T15:28:27.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.846+0000 7f1d70c00640 1 -- 172.21.15.47:0/2091019493 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d6c0b4d10 con 0x7f1d6c0cedb0 2024-02-19T15:28:27.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.850+0000 7f1d70c00640 1 -- 172.21.15.47:0/2091019493 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d6c075360 con 0x7f1d6c0cedb0 2024-02-19T15:28:27.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.850+0000 7f1d62000640 1 -- 172.21.15.47:0/2091019493 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1d5c00cca0 con 0x7f1d6c0cedb0 2024-02-19T15:28:27.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.854+0000 7f1d62000640 1 --2- 172.21.15.47:0/2091019493 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d48066e40 0x7f1d48069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:27.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.854+0000 7f1d62000640 1 -- 172.21.15.47:0/2091019493 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f1d5c084c90 con 0x7f1d6c0cedb0 2024-02-19T15:28:27.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.854+0000 7f1d63e00640 1 --2- 172.21.15.47:0/2091019493 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d48066e40 0x7f1d48069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:27.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.854+0000 7f1d63e00640 1 --2- 172.21.15.47:0/2091019493 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d48066e40 0x7f1d48069300 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c166660 tx=0x7f1d54008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:27.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:27.858+0000 7f1d62000640 1 -- 172.21.15.47:0/2091019493 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1d5c0518f0 con 0x7f1d6c0cedb0 2024-02-19T15:28:28.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:28.050+0000 7f1d70c00640 1 -- 172.21.15.47:0/2091019493 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d6c15abb0 con 0x7f1d6c0cedb0 2024-02-19T15:28:28.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:28.054+0000 7f1d62000640 1 -- 172.21.15.47:0/2091019493 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f1d5c017090 con 0x7f1d6c0cedb0 2024-02-19T15:28:28.053 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:28.054 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:28.054 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:28.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:28.054+0000 7f1d70c00640 1 -- 172.21.15.47:0/2091019493 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d48066e40 msgr2=0x7f1d48069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:28.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:28.054+0000 7f1d70c00640 1 --2- 172.21.15.47:0/2091019493 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d48066e40 0x7f1d48069300 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c166660 tx=0x7f1d54008040 comp rx=0 tx=0).stop 2024-02-19T15:28:28.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:28.054+0000 7f1d70c00640 1 -- 172.21.15.47:0/2091019493 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d6c0cedb0 msgr2=0x7f1d6c0c59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:28.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:28.054+0000 7f1d70c00640 1 --2- 172.21.15.47:0/2091019493 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d6c0cedb0 0x7f1d6c0c59e0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c00e870 tx=0x7f1d5c00ec70 comp rx=0 tx=0).stop 2024-02-19T15:28:28.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:28.054+0000 7f1d70c00640 1 -- 172.21.15.47:0/2091019493 shutdown_connections 2024-02-19T15:28:28.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:28.054+0000 7f1d70c00640 1 --2- 172.21.15.47:0/2091019493 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d48066e40 0x7f1d48069300 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:28.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:28.054+0000 7f1d70c00640 1 --2- 172.21.15.47:0/2091019493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d6c15bcd0 0x7f1d6c0ca800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:28.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:28.054+0000 7f1d70c00640 1 --2- 172.21.15.47:0/2091019493 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d6c0d2710 0x7f1d6c0b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:28.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:28.054+0000 7f1d70c00640 1 --2- 172.21.15.47:0/2091019493 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d6c0cedb0 0x7f1d6c0c59e0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:28.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:28.054+0000 7f1d70c00640 1 -- 172.21.15.47:0/2091019493 >> 172.21.15.47:0/2091019493 conn(0x7f1d6c0d1bf0 msgr2=0x7f1d6c06eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:28.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:28.054+0000 7f1d70c00640 1 -- 172.21.15.47:0/2091019493 shutdown_connections 2024-02-19T15:28:28.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:28.054+0000 7f1d70c00640 1 -- 172.21.15.47:0/2091019493 wait complete. 2024-02-19T15:28:28.074 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-02-19T15:28:28.235 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-02-19T15:28:28.236 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-02-19T15:28:28.236 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 1, waiting (timeout=60)... 2024-02-19T15:28:33.237 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:33.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff253400640 1 Processor -- start 2024-02-19T15:28:33.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff253400640 1 -- start start 2024-02-19T15:28:33.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff253400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff24c15c0d0 0x7ff24c0cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:33.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff253400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c0b0fe0 con 0x7ff24c0d0540 2024-02-19T15:28:33.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff253400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c0b1880 con 0x7ff24c0cd180 2024-02-19T15:28:33.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff253400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff24c15c4b0 con 0x7ff24c15c0d0 2024-02-19T15:28:33.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff251400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff24c0d0540 0x7ff24c0b1d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43610/0 (socket says 172.21.15.47:43610) 2024-02-19T15:28:33.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff251400640 1 -- 172.21.15.47:0/641921320 learned_addr learned my addr 172.21.15.47:0/641921320 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:33.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff250a00640 1 --2- 172.21.15.47:0/641921320 >> v2:172.21.15.116:3300/0 conn(0x7ff24c15c0d0 0x7ff24c0cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:33.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff24b400640 1 -- 172.21.15.47:0/641921320 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850376850 0 0) 0x7ff24c0b0fe0 con 0x7ff24c0d0540 2024-02-19T15:28:33.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff24b400640 1 -- 172.21.15.47:0/641921320 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff234003530 con 0x7ff24c0d0540 2024-02-19T15:28:33.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff24b400640 1 -- 172.21.15.47:0/641921320 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 926166575 0 0) 0x7ff234003530 con 0x7ff24c0d0540 2024-02-19T15:28:33.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff24b400640 1 -- 172.21.15.47:0/641921320 >> v2:172.21.15.116:3300/0 conn(0x7ff24c15c0d0 msgr2=0x7ff24c0cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:33.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff24b400640 1 --2- 172.21.15.47:0/641921320 >> v2:172.21.15.116:3300/0 conn(0x7ff24c15c0d0 0x7ff24c0cf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:33.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff24b400640 1 -- 172.21.15.47:0/641921320 >> v1:172.21.15.116:6789/0 conn(0x7ff24c0cd180 legacy=0x7ff24c0cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:33.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff24b400640 1 -- 172.21.15.47:0/641921320 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c0cd8f0 con 0x7ff24c0d0540 2024-02-19T15:28:33.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff24b400640 1 -- 172.21.15.47:0/641921320 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7ff244001a00 con 0x7ff24c0d0540 2024-02-19T15:28:33.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff24b400640 1 -- 172.21.15.47:0/641921320 >> v1:172.21.15.47:6789/0 conn(0x7ff24c0d0540 legacy=0x7ff24c0b1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:33.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff24b400640 1 --2- 172.21.15.47:0/641921320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff234003980 0x7ff234003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:33.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff24b400640 1 --2- 172.21.15.47:0/641921320 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2340047d0 0x7ff234004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:33.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff24b400640 1 --2- 172.21.15.47:0/641921320 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff234005190 0x7ff23400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:33.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff24b400640 1 -- 172.21.15.47:0/641921320 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c15c4b0 con 0x7ff2340047d0 2024-02-19T15:28:33.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff24b400640 1 -- 172.21.15.47:0/641921320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23400c030 con 0x7ff234003980 2024-02-19T15:28:33.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.394+0000 7ff24b400640 1 -- 172.21.15.47:0/641921320 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23400c1f0 con 0x7ff234005190 2024-02-19T15:28:33.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.398+0000 7ff250a00640 1 --2- 172.21.15.47:0/641921320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff234003980 0x7ff234003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:33.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.398+0000 7ff24be00640 1 --2- 172.21.15.47:0/641921320 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2340047d0 0x7ff234004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:33.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.398+0000 7ff251400640 1 --2- 172.21.15.47:0/641921320 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff234005190 0x7ff23400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:33.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.398+0000 7ff24be00640 1 -- 172.21.15.47:0/641921320 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff234005190 msgr2=0x7ff23400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:33.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.398+0000 7ff24be00640 1 --2- 172.21.15.47:0/641921320 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff234005190 0x7ff23400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:33.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.398+0000 7ff24be00640 1 -- 172.21.15.47:0/641921320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff234003980 msgr2=0x7ff234003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:33.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.398+0000 7ff24be00640 1 --2- 172.21.15.47:0/641921320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff234003980 0x7ff234003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:33.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.398+0000 7ff24be00640 1 -- 172.21.15.47:0/641921320 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff234003530 con 0x7ff2340047d0 2024-02-19T15:28:33.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.398+0000 7ff251400640 1 -- 172.21.15.47:0/641921320 reap_dead start 2024-02-19T15:28:33.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.398+0000 7ff24be00640 1 --2- 172.21.15.47:0/641921320 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2340047d0 0x7ff234004c50 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7ff238001780 tx=0x7ff23801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d1292ae48840ed server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:33.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.398+0000 7ff24b400640 1 -- 172.21.15.47:0/641921320 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2380052d0 con 0x7ff2340047d0 2024-02-19T15:28:33.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.398+0000 7ff24b400640 1 -- 172.21.15.47:0/641921320 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2380a7050 con 0x7ff2340047d0 2024-02-19T15:28:33.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.398+0000 7ff24b400640 1 -- 172.21.15.47:0/641921320 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff238003600 con 0x7ff2340047d0 2024-02-19T15:28:33.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff253400640 1 -- 172.21.15.47:0/641921320 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2340047d0 msgr2=0x7ff234004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:33.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff253400640 1 --2- 172.21.15.47:0/641921320 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2340047d0 0x7ff234004c50 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7ff238001780 tx=0x7ff23801c400 comp rx=0 tx=0).stop 2024-02-19T15:28:33.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff253400640 1 -- 172.21.15.47:0/641921320 shutdown_connections 2024-02-19T15:28:33.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff253400640 1 --2- 172.21.15.47:0/641921320 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2340047d0 0x7ff234004c50 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:33.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff253400640 1 -- 172.21.15.47:0/641921320 >> 172.21.15.47:0/641921320 conn(0x7ff24c0d1ba0 msgr2=0x7ff24c0d1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:33.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff253400640 1 -- 172.21.15.47:0/641921320 shutdown_connections 2024-02-19T15:28:33.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff253400640 1 -- 172.21.15.47:0/641921320 wait complete. 2024-02-19T15:28:33.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff253400640 1 Processor -- start 2024-02-19T15:28:33.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff253400640 1 -- start start 2024-02-19T15:28:33.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff253400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2340047d0 0x7ff24c1f3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:33.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff253400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff24c15eee0 0x7ff24c156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:33.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff253400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff24c0c7140 0x7ff24c0c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:33.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff253400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c1f43a0 con 0x7ff24c15eee0 2024-02-19T15:28:33.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff253400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c0c7b00 con 0x7ff2340047d0 2024-02-19T15:28:33.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff253400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c15d220 con 0x7ff24c0c7140 2024-02-19T15:28:33.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff24be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff24c15eee0 0x7ff24c156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:33.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff24be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff24c15eee0 0x7ff24c156f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59806/0 (socket says 172.21.15.47:59806) 2024-02-19T15:28:33.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff24be00640 1 -- 172.21.15.47:0/2671440711 learned_addr learned my addr 172.21.15.47:0/2671440711 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:33.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff250a00640 1 --2- 172.21.15.47:0/2671440711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2340047d0 0x7ff24c1f3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:33.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff251400640 1 --2- 172.21.15.47:0/2671440711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff24c0c7140 0x7ff24c0c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:33.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff24be00640 1 -- 172.21.15.47:0/2671440711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff24c0c7140 msgr2=0x7ff24c0c75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:33.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff24be00640 1 --2- 172.21.15.47:0/2671440711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff24c0c7140 0x7ff24c0c75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:33.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff24be00640 1 -- 172.21.15.47:0/2671440711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2340047d0 msgr2=0x7ff24c1f3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:33.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff24be00640 1 --2- 172.21.15.47:0/2671440711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2340047d0 0x7ff24c1f3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:33.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff24be00640 1 -- 172.21.15.47:0/2671440711 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c15d4a0 con 0x7ff24c15eee0 2024-02-19T15:28:33.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.402+0000 7ff24be00640 1 --2- 172.21.15.47:0/2671440711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff24c15eee0 0x7ff24c156f50 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7ff23801c230 tx=0x7ff234005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:33.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.406+0000 7ff248c00640 1 -- 172.21.15.47:0/2671440711 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff238003e40 con 0x7ff24c15eee0 2024-02-19T15:28:33.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.406+0000 7ff248c00640 1 -- 172.21.15.47:0/2671440711 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff238007030 con 0x7ff24c15eee0 2024-02-19T15:28:33.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.406+0000 7ff248c00640 1 -- 172.21.15.47:0/2671440711 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2380073e0 con 0x7ff24c15eee0 2024-02-19T15:28:33.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.406+0000 7ff253400640 1 -- 172.21.15.47:0/2671440711 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff24c15d780 con 0x7ff24c15eee0 2024-02-19T15:28:33.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.406+0000 7ff253400640 1 -- 172.21.15.47:0/2671440711 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff24c15dd40 con 0x7ff24c15eee0 2024-02-19T15:28:33.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.406+0000 7ff253400640 1 -- 172.21.15.47:0/2671440711 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff24c068500 con 0x7ff24c15eee0 2024-02-19T15:28:33.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.406+0000 7ff248c00640 1 -- 172.21.15.47:0/2671440711 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff238007540 con 0x7ff24c15eee0 2024-02-19T15:28:33.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.410+0000 7ff248c00640 1 --2- 172.21.15.47:0/2671440711 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff22c066e40 0x7ff22c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:33.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.410+0000 7ff248c00640 1 -- 172.21.15.47:0/2671440711 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7ff23811eff0 con 0x7ff24c15eee0 2024-02-19T15:28:33.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.410+0000 7ff250a00640 1 --2- 172.21.15.47:0/2671440711 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff22c066e40 0x7ff22c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:33.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.410+0000 7ff250a00640 1 --2- 172.21.15.47:0/2671440711 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff22c066e40 0x7ff22c069300 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff240009ba0 tx=0x7ff240008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.418+0000 7ff248c00640 1 -- 172.21.15.47:0/2671440711 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff2380ebc50 con 0x7ff24c15eee0 2024-02-19T15:28:33.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.638+0000 7ff253400640 1 -- 172.21.15.47:0/2671440711 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff24c075360 con 0x7ff24c15eee0 2024-02-19T15:28:33.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.638+0000 7ff248c00640 1 -- 172.21.15.47:0/2671440711 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7ff2380ef310 con 0x7ff24c15eee0 2024-02-19T15:28:33.639 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:33.639 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:33.640 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:33.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.642+0000 7ff253400640 1 -- 172.21.15.47:0/2671440711 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff22c066e40 msgr2=0x7ff22c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:33.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.642+0000 7ff253400640 1 --2- 172.21.15.47:0/2671440711 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff22c066e40 0x7ff22c069300 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff240009ba0 tx=0x7ff240008040 comp rx=0 tx=0).stop 2024-02-19T15:28:33.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.642+0000 7ff253400640 1 -- 172.21.15.47:0/2671440711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff24c15eee0 msgr2=0x7ff24c156f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:33.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.642+0000 7ff253400640 1 --2- 172.21.15.47:0/2671440711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff24c15eee0 0x7ff24c156f50 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7ff23801c230 tx=0x7ff234005800 comp rx=0 tx=0).stop 2024-02-19T15:28:33.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.642+0000 7ff253400640 1 -- 172.21.15.47:0/2671440711 shutdown_connections 2024-02-19T15:28:33.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.642+0000 7ff253400640 1 --2- 172.21.15.47:0/2671440711 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff22c066e40 0x7ff22c069300 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:33.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.642+0000 7ff253400640 1 --2- 172.21.15.47:0/2671440711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff24c0c7140 0x7ff24c0c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:33.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.642+0000 7ff253400640 1 --2- 172.21.15.47:0/2671440711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff24c15eee0 0x7ff24c156f50 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:33.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.642+0000 7ff253400640 1 --2- 172.21.15.47:0/2671440711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2340047d0 0x7ff24c1f3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:33.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.642+0000 7ff253400640 1 -- 172.21.15.47:0/2671440711 >> 172.21.15.47:0/2671440711 conn(0x7ff24c0d1ba0 msgr2=0x7ff24c15b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:33.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.642+0000 7ff253400640 1 -- 172.21.15.47:0/2671440711 shutdown_connections 2024-02-19T15:28:33.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.642+0000 7ff253400640 1 -- 172.21.15.47:0/2671440711 wait complete. 2024-02-19T15:28:33.661 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-02-19T15:28:33.822 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-02-19T15:28:33.823 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-02-19T15:28:33.823 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:33.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.966+0000 7f114ac00640 1 Processor -- start 2024-02-19T15:28:33.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.966+0000 7f114ac00640 1 -- start start 2024-02-19T15:28:33.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.966+0000 7f114ac00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11441613a0 0x7f11440adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:33.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.966+0000 7f114ac00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f114406fdf0 0x7f114406e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:33.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.966+0000 7f114ac00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f114406e5f0 0x7f114406ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:33.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.966+0000 7f114ac00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1144161780 con 0x7f114406fdf0 2024-02-19T15:28:33.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.966+0000 7f114ac00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11440b01a0 con 0x7f114406e5f0 2024-02-19T15:28:33.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.966+0000 7f114ac00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11440ae3c0 con 0x7f11441613a0 2024-02-19T15:28:33.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.966+0000 7f113be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f114406fdf0 0x7f114406e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:33.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.966+0000 7f113be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f114406fdf0 0x7f114406e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59808/0 (socket says 172.21.15.47:59808) 2024-02-19T15:28:33.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.966+0000 7f113be00640 1 -- 172.21.15.47:0/3242715934 learned_addr learned my addr 172.21.15.47:0/3242715934 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:33.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.966+0000 7f1148c00640 1 --2- 172.21.15.47:0/3242715934 >> v2:172.21.15.116:3300/0 conn(0x7f114406e5f0 0x7f114406ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:33.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.966+0000 7f1143e00640 1 --2- 172.21.15.47:0/3242715934 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11441613a0 0x7f11440adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:33.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.970+0000 7f113be00640 1 -- 172.21.15.47:0/3242715934 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11441613a0 msgr2=0x7f11440adbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:33.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.970+0000 7f113be00640 1 --2- 172.21.15.47:0/3242715934 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11441613a0 0x7f11440adbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:33.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.970+0000 7f113be00640 1 -- 172.21.15.47:0/3242715934 >> v2:172.21.15.116:3300/0 conn(0x7f114406e5f0 msgr2=0x7f114406ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:33.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.970+0000 7f113be00640 1 --2- 172.21.15.47:0/3242715934 >> v2:172.21.15.116:3300/0 conn(0x7f114406e5f0 0x7f114406ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:33.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.970+0000 7f113be00640 1 -- 172.21.15.47:0/3242715934 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11440ae6a0 con 0x7f114406fdf0 2024-02-19T15:28:33.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.970+0000 7f1143e00640 1 --2- 172.21.15.47:0/3242715934 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11441613a0 0x7f11440adbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:33.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.974+0000 7f113be00640 1 --2- 172.21.15.47:0/3242715934 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f114406fdf0 0x7f114406e0b0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f11440677b0 tx=0x7f112c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0152d073c2c5ad4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:33.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.974+0000 7f1143400640 1 -- 172.21.15.47:0/3242715934 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c0ad660 con 0x7f114406fdf0 2024-02-19T15:28:33.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.974+0000 7f1143400640 1 -- 172.21.15.47:0/3242715934 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112c0ad7c0 con 0x7f114406fdf0 2024-02-19T15:28:33.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.974+0000 7f1143400640 1 -- 172.21.15.47:0/3242715934 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c0adb50 con 0x7f114406fdf0 2024-02-19T15:28:33.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.974+0000 7f114ac00640 1 -- 172.21.15.47:0/3242715934 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f114406fdf0 msgr2=0x7f114406e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:33.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.974+0000 7f114ac00640 1 --2- 172.21.15.47:0/3242715934 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f114406fdf0 0x7f114406e0b0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f11440677b0 tx=0x7f112c01c490 comp rx=0 tx=0).stop 2024-02-19T15:28:33.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.974+0000 7f114ac00640 1 -- 172.21.15.47:0/3242715934 shutdown_connections 2024-02-19T15:28:33.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.974+0000 7f114ac00640 1 --2- 172.21.15.47:0/3242715934 >> v2:172.21.15.116:3300/0 conn(0x7f114406e5f0 0x7f114406ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:33.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.974+0000 7f114ac00640 1 --2- 172.21.15.47:0/3242715934 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f114406fdf0 0x7f114406e0b0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:33.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.974+0000 7f114ac00640 1 --2- 172.21.15.47:0/3242715934 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f11441613a0 0x7f11440adbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:33.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.974+0000 7f114ac00640 1 -- 172.21.15.47:0/3242715934 >> 172.21.15.47:0/3242715934 conn(0x7f11440d0ad0 msgr2=0x7f11440d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:33.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.974+0000 7f114ac00640 1 -- 172.21.15.47:0/3242715934 shutdown_connections 2024-02-19T15:28:33.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.974+0000 7f114ac00640 1 -- 172.21.15.47:0/3242715934 wait complete. 2024-02-19T15:28:33.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.978+0000 7f114ac00640 1 Processor -- start 2024-02-19T15:28:33.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.978+0000 7f114ac00640 1 -- start start 2024-02-19T15:28:33.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.978+0000 7f114ac00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f114406e5f0 0x7f1144139670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:33.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.978+0000 7f114ac00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f114406fdf0 0x7f114413dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:33.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.978+0000 7f114ac00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11441613a0 0x7f114413e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:33.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.978+0000 7f114ac00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114413b760 con 0x7f114406fdf0 2024-02-19T15:28:33.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.978+0000 7f114ac00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11441590e0 con 0x7f11441613a0 2024-02-19T15:28:33.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.978+0000 7f114ac00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1144139bb0 con 0x7f114406e5f0 2024-02-19T15:28:33.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.978+0000 7f113be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f114406fdf0 0x7f114413dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:33.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.978+0000 7f113be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f114406fdf0 0x7f114413dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59824/0 (socket says 172.21.15.47:59824) 2024-02-19T15:28:33.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.978+0000 7f113be00640 1 -- 172.21.15.47:0/3930372113 learned_addr learned my addr 172.21.15.47:0/3930372113 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:33.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.978+0000 7f1143e00640 1 --2- 172.21.15.47:0/3930372113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f114406e5f0 0x7f1144139670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:33.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.978+0000 7f1148c00640 1 --2- 172.21.15.47:0/3930372113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11441613a0 0x7f114413e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:33.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.978+0000 7f113be00640 1 -- 172.21.15.47:0/3930372113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f114406e5f0 msgr2=0x7f1144139670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:33.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.978+0000 7f113be00640 1 --2- 172.21.15.47:0/3930372113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f114406e5f0 0x7f1144139670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:33.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.978+0000 7f113be00640 1 -- 172.21.15.47:0/3930372113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11441613a0 msgr2=0x7f114413e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:33.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.978+0000 7f113be00640 1 --2- 172.21.15.47:0/3930372113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11441613a0 0x7f114413e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:33.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.978+0000 7f113be00640 1 -- 172.21.15.47:0/3930372113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112c0094d0 con 0x7f114406fdf0 2024-02-19T15:28:33.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.978+0000 7f1148c00640 1 --2- 172.21.15.47:0/3930372113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11441613a0 0x7f114413e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:33.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.978+0000 7f113be00640 1 --2- 172.21.15.47:0/3930372113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f114406fdf0 0x7f114413dc80 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f112c01ab40 tx=0x7f112c01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:33.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.982+0000 7f1140c00640 1 -- 172.21.15.47:0/3930372113 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c0b8620 con 0x7f114406fdf0 2024-02-19T15:28:33.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.982+0000 7f1140c00640 1 -- 172.21.15.47:0/3930372113 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112c0b8780 con 0x7f114406fdf0 2024-02-19T15:28:33.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.982+0000 7f114ac00640 1 -- 172.21.15.47:0/3930372113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f114413f400 con 0x7f114406fdf0 2024-02-19T15:28:33.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.982+0000 7f1140c00640 1 -- 172.21.15.47:0/3930372113 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c0b8b30 con 0x7f114406fdf0 2024-02-19T15:28:33.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.982+0000 7f114ac00640 1 -- 172.21.15.47:0/3930372113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1144159330 con 0x7f114406fdf0 2024-02-19T15:28:33.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.982+0000 7f114ac00640 1 -- 172.21.15.47:0/3930372113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1144075360 con 0x7f114406fdf0 2024-02-19T15:28:33.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.982+0000 7f1140c00640 1 -- 172.21.15.47:0/3930372113 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f112c0b4040 con 0x7f114406fdf0 2024-02-19T15:28:33.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.982+0000 7f1140c00640 1 --2- 172.21.15.47:0/3930372113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1120066e40 0x7f1120069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:33.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.982+0000 7f1140c00640 1 -- 172.21.15.47:0/3930372113 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f112c11f520 con 0x7f114406fdf0 2024-02-19T15:28:33.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.982+0000 7f1143e00640 1 --2- 172.21.15.47:0/3930372113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1120066e40 0x7f1120069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:33.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.986+0000 7f1143e00640 1 --2- 172.21.15.47:0/3930372113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1120066e40 0x7f1120069300 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f113400b930 tx=0x7f1134008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:33.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:33.986+0000 7f1140c00640 1 -- 172.21.15.47:0/3930372113 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f112c0ec390 con 0x7f114406fdf0 2024-02-19T15:28:34.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.194+0000 7f114ac00640 1 -- 172.21.15.47:0/3930372113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11440d15f0 con 0x7f114406fdf0 2024-02-19T15:28:34.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.194+0000 7f1140c00640 1 -- 172.21.15.47:0/3930372113 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f112c0efa50 con 0x7f114406fdf0 2024-02-19T15:28:34.193 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:34.194 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:34.194 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:34.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.194+0000 7f114ac00640 1 -- 172.21.15.47:0/3930372113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1120066e40 msgr2=0x7f1120069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:34.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.194+0000 7f114ac00640 1 --2- 172.21.15.47:0/3930372113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1120066e40 0x7f1120069300 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f113400b930 tx=0x7f1134008040 comp rx=0 tx=0).stop 2024-02-19T15:28:34.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.194+0000 7f114ac00640 1 -- 172.21.15.47:0/3930372113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f114406fdf0 msgr2=0x7f114413dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:34.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.194+0000 7f114ac00640 1 --2- 172.21.15.47:0/3930372113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f114406fdf0 0x7f114413dc80 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f112c01ab40 tx=0x7f112c01acd0 comp rx=0 tx=0).stop 2024-02-19T15:28:34.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.194+0000 7f114ac00640 1 -- 172.21.15.47:0/3930372113 shutdown_connections 2024-02-19T15:28:34.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.194+0000 7f114ac00640 1 --2- 172.21.15.47:0/3930372113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1120066e40 0x7f1120069300 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:34.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.194+0000 7f114ac00640 1 --2- 172.21.15.47:0/3930372113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11441613a0 0x7f114413e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:34.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.194+0000 7f114ac00640 1 --2- 172.21.15.47:0/3930372113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f114406fdf0 0x7f114413dc80 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:34.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.194+0000 7f114ac00640 1 --2- 172.21.15.47:0/3930372113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f114406e5f0 0x7f1144139670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:34.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.194+0000 7f114ac00640 1 -- 172.21.15.47:0/3930372113 >> 172.21.15.47:0/3930372113 conn(0x7f11440d0ad0 msgr2=0x7f1144071920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:34.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.194+0000 7f114ac00640 1 -- 172.21.15.47:0/3930372113 shutdown_connections 2024-02-19T15:28:34.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.194+0000 7f114ac00640 1 -- 172.21.15.47:0/3930372113 wait complete. 2024-02-19T15:28:34.216 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2024-02-19T15:28:34.398 INFO:teuthology.orchestra.run.smithi047.stdout:{"purge_queue":{"pq_executed":1}} 2024-02-19T15:28:34.398 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1 } } 2024-02-19T15:28:34.399 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:34.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.538+0000 7f1870e00640 1 Processor -- start 2024-02-19T15:28:34.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.538+0000 7f1870e00640 1 -- start start 2024-02-19T15:28:34.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.542+0000 7f1870e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f186c15b1a0 0x7f186c15b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:34.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.542+0000 7f1870e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f186c06c9a0 0x7f186c15bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:34.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.542+0000 7f1870e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186c15c000 con 0x7f186c071350 2024-02-19T15:28:34.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.542+0000 7f1870e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f186c0718c0 con 0x7f186c06c9a0 2024-02-19T15:28:34.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.542+0000 7f1870e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f186c071ba0 con 0x7f186c15b1a0 2024-02-19T15:28:34.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.542+0000 7f186a800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f186c06c9a0 0x7f186c15bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:34.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.542+0000 7f186a800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f186c06c9a0 0x7f186c15bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59832/0 (socket says 172.21.15.47:59832) 2024-02-19T15:28:34.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.542+0000 7f186a800640 1 -- 172.21.15.47:0/969769306 learned_addr learned my addr 172.21.15.47:0/969769306 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:34.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.542+0000 7f1869e00640 1 --2- 172.21.15.47:0/969769306 >> v2:172.21.15.116:3300/0 conn(0x7f186c15b1a0 0x7f186c15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:34.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.542+0000 7f186a800640 1 -- 172.21.15.47:0/969769306 >> v2:172.21.15.116:3300/0 conn(0x7f186c15b1a0 msgr2=0x7f186c15b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:34.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.542+0000 7f186a800640 1 --2- 172.21.15.47:0/969769306 >> v2:172.21.15.116:3300/0 conn(0x7f186c15b1a0 0x7f186c15b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:34.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.542+0000 7f186a800640 1 -- 172.21.15.47:0/969769306 >> v1:172.21.15.116:6789/0 conn(0x7f186c071350 legacy=0x7f186c15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:34.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.542+0000 7f186a800640 1 -- 172.21.15.47:0/969769306 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186c071e80 con 0x7f186c06c9a0 2024-02-19T15:28:34.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.546+0000 7f186a800640 1 --2- 172.21.15.47:0/969769306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f186c06c9a0 0x7f186c15bac0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f1864009e50 tx=0x7f186401c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c80261f90a9970 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:34.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.546+0000 7f1869400640 1 -- 172.21.15.47:0/969769306 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18640a58a0 con 0x7f186c06c9a0 2024-02-19T15:28:34.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.546+0000 7f1869400640 1 -- 172.21.15.47:0/969769306 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18640a5a00 con 0x7f186c06c9a0 2024-02-19T15:28:34.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.546+0000 7f1869400640 1 -- 172.21.15.47:0/969769306 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18640a5d90 con 0x7f186c06c9a0 2024-02-19T15:28:34.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.546+0000 7f1870e00640 1 -- 172.21.15.47:0/969769306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f186c06c9a0 msgr2=0x7f186c15bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:34.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.546+0000 7f1870e00640 1 --2- 172.21.15.47:0/969769306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f186c06c9a0 0x7f186c15bac0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f1864009e50 tx=0x7f186401c9c0 comp rx=0 tx=0).stop 2024-02-19T15:28:34.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.546+0000 7f1870e00640 1 -- 172.21.15.47:0/969769306 shutdown_connections 2024-02-19T15:28:34.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.546+0000 7f1870e00640 1 --2- 172.21.15.47:0/969769306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f186c06c9a0 0x7f186c15bac0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:34.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.546+0000 7f1870e00640 1 --2- 172.21.15.47:0/969769306 >> v2:172.21.15.116:3300/0 conn(0x7f186c15b1a0 0x7f186c15b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:34.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.546+0000 7f1870e00640 1 -- 172.21.15.47:0/969769306 >> 172.21.15.47:0/969769306 conn(0x7f186c0b16a0 msgr2=0x7f186c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:34.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.546+0000 7f1870e00640 1 -- 172.21.15.47:0/969769306 shutdown_connections 2024-02-19T15:28:34.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.546+0000 7f1870e00640 1 -- 172.21.15.47:0/969769306 wait complete. 2024-02-19T15:28:34.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1870e00640 1 Processor -- start 2024-02-19T15:28:34.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1870e00640 1 -- start start 2024-02-19T15:28:34.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1870e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f186c06c9a0 0x7f186c137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:34.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1870e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f186c071350 0x7f186c147e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:34.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1870e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f186c15b1a0 0x7f186c149200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:34.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1870e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186c138300 con 0x7f186c071350 2024-02-19T15:28:34.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1870e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186c137370 con 0x7f186c15b1a0 2024-02-19T15:28:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1870e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f186c148870 con 0x7f186c06c9a0 2024-02-19T15:28:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1863e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f186c071350 0x7f186c147e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1869e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f186c06c9a0 0x7f186c137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1863e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f186c071350 0x7f186c147e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59840/0 (socket says 172.21.15.47:59840) 2024-02-19T15:28:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1863e00640 1 -- 172.21.15.47:0/3014579840 learned_addr learned my addr 172.21.15.47:0/3014579840 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f186a800640 1 --2- 172.21.15.47:0/3014579840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f186c15b1a0 0x7f186c149200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:34.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1863e00640 1 -- 172.21.15.47:0/3014579840 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f186c06c9a0 msgr2=0x7f186c137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:34.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1863e00640 1 --2- 172.21.15.47:0/3014579840 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f186c06c9a0 0x7f186c137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:34.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1863e00640 1 -- 172.21.15.47:0/3014579840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f186c15b1a0 msgr2=0x7f186c149200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:34.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1863e00640 1 --2- 172.21.15.47:0/3014579840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f186c15b1a0 0x7f186c149200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:34.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1863e00640 1 -- 172.21.15.47:0/3014579840 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1864009b30 con 0x7f186c071350 2024-02-19T15:28:34.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1863e00640 1 --2- 172.21.15.47:0/3014579840 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f186c071350 0x7f186c147e20 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f185400c420 tx=0x7f185400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:34.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1862000640 1 -- 172.21.15.47:0/3014579840 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18540052d0 con 0x7f186c071350 2024-02-19T15:28:34.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1862000640 1 -- 172.21.15.47:0/3014579840 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1854005430 con 0x7f186c071350 2024-02-19T15:28:34.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1862000640 1 -- 172.21.15.47:0/3014579840 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854007260 con 0x7f186c071350 2024-02-19T15:28:34.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1870e00640 1 -- 172.21.15.47:0/3014579840 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f186c148b50 con 0x7f186c071350 2024-02-19T15:28:34.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1870e00640 1 -- 172.21.15.47:0/3014579840 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f186c1378a0 con 0x7f186c071350 2024-02-19T15:28:34.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.550+0000 7f1870e00640 1 -- 172.21.15.47:0/3014579840 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f186c075360 con 0x7f186c071350 2024-02-19T15:28:34.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.554+0000 7f1862000640 1 -- 172.21.15.47:0/3014579840 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f18540073c0 con 0x7f186c071350 2024-02-19T15:28:34.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.554+0000 7f1862000640 1 --2- 172.21.15.47:0/3014579840 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1848066e40 0x7f1848069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:34.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.554+0000 7f1862000640 1 -- 172.21.15.47:0/3014579840 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f1854083f40 con 0x7f186c071350 2024-02-19T15:28:34.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.554+0000 7f1869e00640 1 --2- 172.21.15.47:0/3014579840 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1848066e40 0x7f1848069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:34.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.554+0000 7f1869e00640 1 --2- 172.21.15.47:0/3014579840 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1848066e40 0x7f1848069300 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f185c009b60 tx=0x7f185c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:34.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.558+0000 7f1862000640 1 -- 172.21.15.47:0/3014579840 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1854050c50 con 0x7f186c071350 2024-02-19T15:28:34.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.754+0000 7f1870e00640 1 -- 172.21.15.47:0/3014579840 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f186c071130 con 0x7f186c071350 2024-02-19T15:28:34.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.758+0000 7f1862000640 1 -- 172.21.15.47:0/3014579840 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f1854054310 con 0x7f186c071350 2024-02-19T15:28:34.757 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:34.757 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:34.757 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:34.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.758+0000 7f1870e00640 1 -- 172.21.15.47:0/3014579840 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1848066e40 msgr2=0x7f1848069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:34.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.758+0000 7f1870e00640 1 --2- 172.21.15.47:0/3014579840 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1848066e40 0x7f1848069300 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f185c009b60 tx=0x7f185c009190 comp rx=0 tx=0).stop 2024-02-19T15:28:34.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.758+0000 7f1870e00640 1 -- 172.21.15.47:0/3014579840 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f186c071350 msgr2=0x7f186c147e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:34.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.758+0000 7f1870e00640 1 --2- 172.21.15.47:0/3014579840 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f186c071350 0x7f186c147e20 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f185400c420 tx=0x7f185400c8e0 comp rx=0 tx=0).stop 2024-02-19T15:28:34.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.758+0000 7f1870e00640 1 -- 172.21.15.47:0/3014579840 shutdown_connections 2024-02-19T15:28:34.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.758+0000 7f1870e00640 1 --2- 172.21.15.47:0/3014579840 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1848066e40 0x7f1848069300 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:34.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.758+0000 7f1870e00640 1 --2- 172.21.15.47:0/3014579840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f186c15b1a0 0x7f186c149200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:34.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.758+0000 7f1870e00640 1 --2- 172.21.15.47:0/3014579840 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f186c071350 0x7f186c147e20 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:34.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.758+0000 7f1870e00640 1 --2- 172.21.15.47:0/3014579840 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f186c06c9a0 0x7f186c137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:34.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.758+0000 7f1870e00640 1 -- 172.21.15.47:0/3014579840 >> 172.21.15.47:0/3014579840 conn(0x7f186c0b16a0 msgr2=0x7f186c0af610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:34.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.758+0000 7f1870e00640 1 -- 172.21.15.47:0/3014579840 shutdown_connections 2024-02-19T15:28:34.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:34.758+0000 7f1870e00640 1 -- 172.21.15.47:0/3014579840 wait complete. 2024-02-19T15:28:34.779 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache 2024-02-19T15:28:34.939 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3,"strays_enqueued":1,"strays_reintegrated":2,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0}} 2024-02-19T15:28:34.939 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3, "strays_enqueued": 1, "strays_migrated": 0, "strays_reintegrated": 2 } } 2024-02-19T15:28:34.940 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:35.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.078+0000 7f248e800640 1 Processor -- start 2024-02-19T15:28:35.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.078+0000 7f248e800640 1 -- start start 2024-02-19T15:28:35.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.082+0000 7f248e800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f248806d650 0x7f24880d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.082+0000 7f248e800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24880cce30 con 0x7f24880ced10 2024-02-19T15:28:35.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.082+0000 7f248e800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24880cd110 con 0x7f24880cded0 2024-02-19T15:28:35.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.082+0000 7f248e800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24880cd3f0 con 0x7f248806d650 2024-02-19T15:28:35.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.082+0000 7f2487e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f24880ced10 0x7f248806bd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43618/0 (socket says 172.21.15.47:43618) 2024-02-19T15:28:35.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.082+0000 7f2487e00640 1 -- 172.21.15.47:0/1929530269 learned_addr learned my addr 172.21.15.47:0/1929530269 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:35.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.082+0000 7f2487400640 1 --2- 172.21.15.47:0/1929530269 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f248806d650 0x7f24880d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.082+0000 7f2486a00640 1 -- 172.21.15.47:0/1929530269 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046109522 0 0) 0x7f24880cce30 con 0x7f24880ced10 2024-02-19T15:28:35.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.082+0000 7f2486a00640 1 -- 172.21.15.47:0/1929530269 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f246c003530 con 0x7f24880ced10 2024-02-19T15:28:35.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.082+0000 7f2487400640 1 -- 172.21.15.47:0/1929530269 >> v1:172.21.15.116:6789/0 conn(0x7f24880cded0 legacy=0x7f24880ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.082+0000 7f2487400640 1 -- 172.21.15.47:0/1929530269 >> v1:172.21.15.47:6789/0 conn(0x7f24880ced10 legacy=0x7f248806bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.082+0000 7f2487400640 1 -- 172.21.15.47:0/1929530269 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24880cd6d0 con 0x7f248806d650 2024-02-19T15:28:35.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f2487400640 1 --2- 172.21.15.47:0/1929530269 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f248806d650 0x7f24880d08b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f24780097f0 tx=0x7f247801c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2f6c677e2e83ffc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:35.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f2486a00640 1 -- 172.21.15.47:0/1929530269 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24780ab640 con 0x7f248806d650 2024-02-19T15:28:35.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f2486a00640 1 -- 172.21.15.47:0/1929530269 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24780a7070 con 0x7f248806d650 2024-02-19T15:28:35.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f2486a00640 1 -- 172.21.15.47:0/1929530269 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24780ab9c0 con 0x7f248806d650 2024-02-19T15:28:35.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f248e800640 1 -- 172.21.15.47:0/1929530269 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f248806d650 msgr2=0x7f24880d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f248e800640 1 --2- 172.21.15.47:0/1929530269 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f248806d650 0x7f24880d08b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f24780097f0 tx=0x7f247801c420 comp rx=0 tx=0).stop 2024-02-19T15:28:35.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f248e800640 1 -- 172.21.15.47:0/1929530269 shutdown_connections 2024-02-19T15:28:35.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f248e800640 1 --2- 172.21.15.47:0/1929530269 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f248806d650 0x7f24880d08b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f248e800640 1 -- 172.21.15.47:0/1929530269 >> 172.21.15.47:0/1929530269 conn(0x7f24880d1bf0 msgr2=0x7f24880d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:35.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f248e800640 1 -- 172.21.15.47:0/1929530269 shutdown_connections 2024-02-19T15:28:35.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f248e800640 1 -- 172.21.15.47:0/1929530269 wait complete. 2024-02-19T15:28:35.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f248e800640 1 Processor -- start 2024-02-19T15:28:35.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f248e800640 1 -- start start 2024-02-19T15:28:35.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f248e800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f248806d650 0x7f248815c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f248e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24880cded0 0x7f24881595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f248e800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24880ced10 0x7f24880b5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f248e800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24880b57f0 con 0x7f24880ced10 2024-02-19T15:28:35.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f248e800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2488159af0 con 0x7f24880cded0 2024-02-19T15:28:35.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f248e800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24880c96a0 con 0x7f248806d650 2024-02-19T15:28:35.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f2487e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24880ced10 0x7f24880b5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f2487e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24880ced10 0x7f24880b5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59856/0 (socket says 172.21.15.47:59856) 2024-02-19T15:28:35.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.086+0000 7f2487e00640 1 -- 172.21.15.47:0/2799410035 learned_addr learned my addr 172.21.15.47:0/2799410035 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:35.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.090+0000 7f2487400640 1 --2- 172.21.15.47:0/2799410035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f248806d650 0x7f248815c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.090+0000 7f247fe00640 1 --2- 172.21.15.47:0/2799410035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24880cded0 0x7f24881595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.090+0000 7f2487e00640 1 -- 172.21.15.47:0/2799410035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f248806d650 msgr2=0x7f248815c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.090+0000 7f2487e00640 1 --2- 172.21.15.47:0/2799410035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f248806d650 0x7f248815c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.090+0000 7f2487e00640 1 -- 172.21.15.47:0/2799410035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24880cded0 msgr2=0x7f24881595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.090+0000 7f2487e00640 1 --2- 172.21.15.47:0/2799410035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24880cded0 0x7f24881595b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.090+0000 7f2487e00640 1 -- 172.21.15.47:0/2799410035 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2478009420 con 0x7f24880ced10 2024-02-19T15:28:35.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.090+0000 7f2487400640 1 --2- 172.21.15.47:0/2799410035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f248806d650 0x7f248815c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:35.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.090+0000 7f247fe00640 1 --2- 172.21.15.47:0/2799410035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24880cded0 0x7f24881595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:35.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.090+0000 7f2487e00640 1 --2- 172.21.15.47:0/2799410035 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24880ced10 0x7f24880b5d30 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f248000cb40 tx=0x7f248000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:35.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.090+0000 7f247f400640 1 -- 172.21.15.47:0/2799410035 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248000aca0 con 0x7f24880ced10 2024-02-19T15:28:35.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.090+0000 7f247f400640 1 -- 172.21.15.47:0/2799410035 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248000ae00 con 0x7f24880ced10 2024-02-19T15:28:35.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.090+0000 7f247f400640 1 -- 172.21.15.47:0/2799410035 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2480012620 con 0x7f24880ced10 2024-02-19T15:28:35.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.090+0000 7f248e800640 1 -- 172.21.15.47:0/2799410035 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24880c9920 con 0x7f24880ced10 2024-02-19T15:28:35.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.090+0000 7f248e800640 1 -- 172.21.15.47:0/2799410035 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24880c9ee0 con 0x7f24880ced10 2024-02-19T15:28:35.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.090+0000 7f248e800640 1 -- 172.21.15.47:0/2799410035 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2488075360 con 0x7f24880ced10 2024-02-19T15:28:35.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.090+0000 7f247f400640 1 -- 172.21.15.47:0/2799410035 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2480007070 con 0x7f24880ced10 2024-02-19T15:28:35.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.094+0000 7f247f400640 1 --2- 172.21.15.47:0/2799410035 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2464066e40 0x7f2464069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.094+0000 7f247f400640 1 -- 172.21.15.47:0/2799410035 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f2480084b10 con 0x7f24880ced10 2024-02-19T15:28:35.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.094+0000 7f2487400640 1 --2- 172.21.15.47:0/2799410035 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2464066e40 0x7f2464069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.094+0000 7f2487400640 1 --2- 172.21.15.47:0/2799410035 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2464066e40 0x7f2464069300 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f24780af000 tx=0x7f2478008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:35.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.098+0000 7f247f400640 1 -- 172.21.15.47:0/2799410035 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f24800518a0 con 0x7f24880ced10 2024-02-19T15:28:35.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.330+0000 7f248e800640 1 -- 172.21.15.47:0/2799410035 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24880d13d0 con 0x7f24880ced10 2024-02-19T15:28:35.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.334+0000 7f247f400640 1 -- 172.21.15.47:0/2799410035 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f2480017090 con 0x7f24880ced10 2024-02-19T15:28:35.333 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:35.333 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:35.334 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:35.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.334+0000 7f248e800640 1 -- 172.21.15.47:0/2799410035 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2464066e40 msgr2=0x7f2464069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.334+0000 7f248e800640 1 --2- 172.21.15.47:0/2799410035 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2464066e40 0x7f2464069300 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f24780af000 tx=0x7f2478008040 comp rx=0 tx=0).stop 2024-02-19T15:28:35.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.334+0000 7f248e800640 1 -- 172.21.15.47:0/2799410035 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24880ced10 msgr2=0x7f24880b5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.334+0000 7f248e800640 1 --2- 172.21.15.47:0/2799410035 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24880ced10 0x7f24880b5d30 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f248000cb40 tx=0x7f248000a060 comp rx=0 tx=0).stop 2024-02-19T15:28:35.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.334+0000 7f248e800640 1 -- 172.21.15.47:0/2799410035 shutdown_connections 2024-02-19T15:28:35.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.334+0000 7f248e800640 1 --2- 172.21.15.47:0/2799410035 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2464066e40 0x7f2464069300 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.334+0000 7f248e800640 1 --2- 172.21.15.47:0/2799410035 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24880ced10 0x7f24880b5d30 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.334+0000 7f248e800640 1 --2- 172.21.15.47:0/2799410035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24880cded0 0x7f24881595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.334+0000 7f248e800640 1 --2- 172.21.15.47:0/2799410035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f248806d650 0x7f248815c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.334+0000 7f248e800640 1 -- 172.21.15.47:0/2799410035 >> 172.21.15.47:0/2799410035 conn(0x7f24880d1bf0 msgr2=0x7f24880b48e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:35.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.334+0000 7f248e800640 1 -- 172.21.15.47:0/2799410035 shutdown_connections 2024-02-19T15:28:35.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.334+0000 7f248e800640 1 -- 172.21.15.47:0/2799410035 wait complete. 2024-02-19T15:28:35.355 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue 2024-02-19T15:28:35.518 INFO:teuthology.orchestra.run.smithi047.stdout:{"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":2,"pq_executing":0,"pq_executing_high_water":1,"pq_executed_ops":2,"pq_executed":1,"pq_item_in_journal":0}} 2024-02-19T15:28:35.518 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1, "pq_executed_ops": 2, "pq_executing": 0, "pq_executing_high_water": 1, "pq_executing_ops": 0, "pq_executing_ops_high_water": 2, "pq_item_in_journal": 0 } } 2024-02-19T15:28:35.518 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-02-19T15:28:35.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.562+0000 7ffa62447080 1 Processor -- start 2024-02-19T15:28:35.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.562+0000 7ffa62447080 1 -- start start 2024-02-19T15:28:35.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.562+0000 7ffa62447080 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa44002430 0x55e250d4f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.562+0000 7ffa62447080 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e250d31070 0x55e250de0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.562+0000 7ffa62447080 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55e250d31450 0x55e250de08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.562+0000 7ffa62447080 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e250c23f90 con 0x55e250d31070 2024-02-19T15:28:35.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.562+0000 7ffa62447080 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e250bbb1e0 con 0x7ffa44002430 2024-02-19T15:28:35.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.562+0000 7ffa62447080 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e250bbae80 con 0x55e250d31450 2024-02-19T15:28:35.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.562+0000 7ffa5ec00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e250d31070 0x55e250de0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.562+0000 7ffa5f600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa44002430 0x55e250d4f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.562+0000 7ffa60000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55e250d31450 0x55e250de08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.562+0000 7ffa5ec00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e250d31070 0x55e250de0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59876/0 (socket says 172.21.15.47:59876) 2024-02-19T15:28:35.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.562+0000 7ffa5ec00640 1 -- 172.21.15.47:0/1168086158 learned_addr learned my addr 172.21.15.47:0/1168086158 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:35.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.562+0000 7ffa5ec00640 1 -- 172.21.15.47:0/1168086158 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55e250d31450 msgr2=0x55e250de08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:35.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.566+0000 7ffa5ec00640 1 --2- 172.21.15.47:0/1168086158 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55e250d31450 0x55e250de08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.566+0000 7ffa5ec00640 1 -- 172.21.15.47:0/1168086158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa44002430 msgr2=0x55e250d4f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:35.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.566+0000 7ffa5ec00640 1 --2- 172.21.15.47:0/1168086158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa44002430 0x55e250d4f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.566+0000 7ffa5ec00640 1 -- 172.21.15.47:0/1168086158 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e250bb9f30 con 0x55e250d31070 2024-02-19T15:28:35.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.566+0000 7ffa5ec00640 1 --2- 172.21.15.47:0/1168086158 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e250d31070 0x55e250de0370 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c001770 tx=0x7ffa4c0099d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dc198c32b24a060 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:35.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.566+0000 7ffa5d800640 1 -- 172.21.15.47:0/1168086158 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c00ea00 con 0x55e250d31070 2024-02-19T15:28:35.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.566+0000 7ffa5d800640 1 -- 172.21.15.47:0/1168086158 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa4c00eb60 con 0x55e250d31070 2024-02-19T15:28:35.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.566+0000 7ffa5d800640 1 -- 172.21.15.47:0/1168086158 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c018400 con 0x55e250d31070 2024-02-19T15:28:35.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.566+0000 7ffa62447080 1 -- 172.21.15.47:0/1168086158 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e250d31070 msgr2=0x55e250de0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.566+0000 7ffa62447080 1 --2- 172.21.15.47:0/1168086158 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e250d31070 0x55e250de0370 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c001770 tx=0x7ffa4c0099d0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.566+0000 7ffa62447080 1 -- 172.21.15.47:0/1168086158 shutdown_connections 2024-02-19T15:28:35.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.566+0000 7ffa62447080 1 --2- 172.21.15.47:0/1168086158 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55e250d31450 0x55e250de08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.566+0000 7ffa62447080 1 --2- 172.21.15.47:0/1168086158 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e250d31070 0x55e250de0370 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.566+0000 7ffa62447080 1 --2- 172.21.15.47:0/1168086158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa44002430 0x55e250d4f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.566+0000 7ffa62447080 1 -- 172.21.15.47:0/1168086158 >> 172.21.15.47:0/1168086158 conn(0x55e250ca2680 msgr2=0x55e250d4f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:35.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.566+0000 7ffa62447080 1 -- 172.21.15.47:0/1168086158 shutdown_connections 2024-02-19T15:28:35.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.566+0000 7ffa62447080 1 -- 172.21.15.47:0/1168086158 wait complete. 2024-02-19T15:28:35.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.566+0000 7ffa62447080 1 Processor -- start 2024-02-19T15:28:35.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.566+0000 7ffa62447080 1 -- start start 2024-02-19T15:28:35.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa62447080 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55e250d31070 0x55e250de3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa62447080 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e250d31450 0x55e250ca66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa62447080 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa44002430 0x55e250ca6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa62447080 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e250bde610 con 0x55e250d31450 2024-02-19T15:28:35.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa62447080 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e250bae740 con 0x7ffa44002430 2024-02-19T15:28:35.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa62447080 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e250de4200 con 0x55e250d31070 2024-02-19T15:28:35.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa5f600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55e250d31070 0x55e250de3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa5f600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55e250d31070 0x55e250de3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:34254/0 (socket says 172.21.15.47:34254) 2024-02-19T15:28:35.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 learned_addr learned my addr 172.21.15.47:0/2075973713 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:35.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa5ec00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e250d31450 0x55e250ca66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa60000640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa44002430 0x55e250ca6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa44002430 msgr2=0x55e250ca6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa5f600640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa44002430 0x55e250ca6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e250d31450 msgr2=0x55e250ca66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa5f600640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e250d31450 0x55e250ca66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e250bc6d80 con 0x55e250d31070 2024-02-19T15:28:35.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa5ec00640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e250d31450 0x55e250ca66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:28:35.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa5f600640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55e250d31070 0x55e250de3cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ffa50096f10 tx=0x7ffa50005a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:35.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa57e00640 1 -- 172.21.15.47:0/2075973713 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa500ba070 con 0x55e250d31070 2024-02-19T15:28:35.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa57e00640 1 -- 172.21.15.47:0/2075973713 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa500aaad0 con 0x55e250d31070 2024-02-19T15:28:35.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa57e00640 1 -- 172.21.15.47:0/2075973713 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa500aae60 con 0x55e250d31070 2024-02-19T15:28:35.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa62447080 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e250bb9990 con 0x55e250d31070 2024-02-19T15:28:35.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa62447080 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e250cba820 con 0x55e250d31070 2024-02-19T15:28:35.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.570+0000 7ffa57e00640 1 -- 172.21.15.47:0/2075973713 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ffa500b5410 con 0x55e250d31070 2024-02-19T15:28:35.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.574+0000 7ffa57e00640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffa40065820 0x7ffa40067ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.574+0000 7ffa57e00640 1 -- 172.21.15.47:0/2075973713 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7ffa5011e7d0 con 0x55e250d31070 2024-02-19T15:28:35.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.574+0000 7ffa5ec00640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffa40065820 0x7ffa40067ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.574+0000 7ffa62447080 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] conn(0x55e250cb7870 0x55e250cb7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.574+0000 7ffa62447080 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cb7870 2024-02-19T15:28:35.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.574+0000 7ffa60000640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] conn(0x55e250cb7870 0x55e250cb7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.574+0000 7ffa5ec00640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffa40065820 0x7ffa40067ce0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c001660 tx=0x7ffa4c00b000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:35.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.574+0000 7ffa60000640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] conn(0x55e250cb7870 0x55e250cb7c80 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:35.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.574+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.7 v2:172.21.15.116:6800/3212743377 1 ==== osd_op_reply(1 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x55e250cb7870 2024-02-19T15:28:35.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.574+0000 7ffa60000640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] conn(0x7ffa58015040 0x7ffa58003040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.574+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa58015040 2024-02-19T15:28:35.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.574+0000 7ffa5f600640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] conn(0x7ffa58015040 0x7ffa58003040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.578+0000 7ffa5f600640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] conn(0x7ffa58015040 0x7ffa58003040 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:35.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.578+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 <== osd.1 v2:172.21.15.47:6800/1591346632 1 ==== osd_op_reply(2 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa500b0020 con 0x7ffa58015040 2024-02-19T15:28:35.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.578+0000 7ffa5f600640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x7ffa500c8060 0x7ffa500ee100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.578+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa500c8060 2024-02-19T15:28:35.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.578+0000 7ffa5ec00640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x7ffa500c8060 0x7ffa500ee100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.578+0000 7ffa5ec00640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x7ffa500c8060 0x7ffa500ee100 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:35.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.578+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 <== osd.3 v2:172.21.15.47:6824/4149334728 1 ==== osd_op_reply(3 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa4c0197c0 con 0x7ffa500c8060 2024-02-19T15:28:35.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.578+0000 7ffa5ec00640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x7ffa4c018020 0x7ffa4c025a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.578+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa4c018020 2024-02-19T15:28:35.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.578+0000 7ffa60000640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x7ffa4c018020 0x7ffa4c025a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.578+0000 7ffa60000640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x7ffa4c018020 0x7ffa4c025a70 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:35.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.578+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.0 v2:172.21.15.47:6809/1968337531 1 ==== osd_op_reply(4 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x7ffa4c018020 2024-02-19T15:28:35.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.578+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cb7870 2024-02-19T15:28:35.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.578+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.7 v2:172.21.15.116:6800/3212743377 2 ==== osd_op_reply(5 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x55e250cb7870 2024-02-19T15:28:35.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.578+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa500c8060 2024-02-19T15:28:35.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.578+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 <== osd.3 v2:172.21.15.47:6824/4149334728 2 ==== osd_op_reply(6 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa4c0197c0 con 0x7ffa500c8060 2024-02-19T15:28:35.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.578+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa4c018020 2024-02-19T15:28:35.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.578+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.0 v2:172.21.15.47:6809/1968337531 2 ==== osd_op_reply(7 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x7ffa4c018020 2024-02-19T15:28:35.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.578+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cb7870 2024-02-19T15:28:35.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.582+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.7 v2:172.21.15.116:6800/3212743377 3 ==== osd_op_reply(8 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x55e250cb7870 2024-02-19T15:28:35.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.582+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa4c018020 2024-02-19T15:28:35.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.582+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.0 v2:172.21.15.47:6809/1968337531 3 ==== osd_op_reply(9 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x7ffa4c018020 2024-02-19T15:28:35.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.582+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa500c8060 2024-02-19T15:28:35.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.582+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 <== osd.3 v2:172.21.15.47:6824/4149334728 3 ==== osd_op_reply(10 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa4c0197c0 con 0x7ffa500c8060 2024-02-19T15:28:35.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.582+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa500c8060 2024-02-19T15:28:35.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.582+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 <== osd.3 v2:172.21.15.47:6824/4149334728 4 ==== osd_op_reply(11 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa4c0197c0 con 0x7ffa500c8060 2024-02-19T15:28:35.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.582+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa4c018020 2024-02-19T15:28:35.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.582+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.0 v2:172.21.15.47:6809/1968337531 4 ==== osd_op_reply(12 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x7ffa4c018020 2024-02-19T15:28:35.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.582+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cb7870 2024-02-19T15:28:35.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.582+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.7 v2:172.21.15.116:6800/3212743377 4 ==== osd_op_reply(13 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x55e250cb7870 2024-02-19T15:28:35.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.582+0000 7ffa60000640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] conn(0x55e250cbfab0 0x7ffa58017940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.582+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cbfab0 2024-02-19T15:28:35.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.582+0000 7ffa5f600640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] conn(0x55e250cbfab0 0x7ffa58017940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.582+0000 7ffa5f600640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] conn(0x55e250cbfab0 0x7ffa58017940 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:35.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.582+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 <== osd.4 v2:172.21.15.116:6809/3366344088 1 ==== osd_op_reply(14 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa500b0020 con 0x55e250cbfab0 2024-02-19T15:28:35.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.582+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa58015040 2024-02-19T15:28:35.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.582+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 <== osd.1 v2:172.21.15.47:6800/1591346632 2 ==== osd_op_reply(15 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa500b0020 con 0x7ffa58015040 2024-02-19T15:28:35.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.582+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa4c018020 2024-02-19T15:28:35.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.586+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.0 v2:172.21.15.47:6809/1968337531 5 ==== osd_op_reply(16 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x7ffa4c018020 2024-02-19T15:28:35.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.586+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cbfab0 2024-02-19T15:28:35.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.586+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 <== osd.4 v2:172.21.15.116:6809/3366344088 2 ==== osd_op_reply(17 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa500b0020 con 0x55e250cbfab0 2024-02-19T15:28:35.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.586+0000 7ffa5f600640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] conn(0x7ffa50009040 0x7ffa5011c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.586+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa50009040 2024-02-19T15:28:35.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.586+0000 7ffa5ec00640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] conn(0x7ffa50009040 0x7ffa5011c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.586+0000 7ffa5ec00640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] conn(0x7ffa50009040 0x7ffa5011c1a0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:35.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.586+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 <== osd.5 v2:172.21.15.116:6808/3478542733 1 ==== osd_op_reply(18 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa4c0197c0 con 0x7ffa50009040 2024-02-19T15:28:35.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.586+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa500c8060 2024-02-19T15:28:35.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.586+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 <== osd.3 v2:172.21.15.47:6824/4149334728 5 ==== osd_op_reply(19 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa4c0197c0 con 0x7ffa500c8060 2024-02-19T15:28:35.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.586+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cb7870 2024-02-19T15:28:35.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.586+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.7 v2:172.21.15.116:6800/3212743377 5 ==== osd_op_reply(20 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x55e250cb7870 2024-02-19T15:28:35.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.586+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cb7870 2024-02-19T15:28:35.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.586+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.7 v2:172.21.15.116:6800/3212743377 6 ==== osd_op_reply(21 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x55e250cb7870 2024-02-19T15:28:35.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.586+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa500c8060 2024-02-19T15:28:35.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.586+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 <== osd.3 v2:172.21.15.47:6824/4149334728 6 ==== osd_op_reply(22 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa4c0197c0 con 0x7ffa500c8060 2024-02-19T15:28:35.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.586+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa4c018020 2024-02-19T15:28:35.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.590+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.0 v2:172.21.15.47:6809/1968337531 6 ==== osd_op_reply(23 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x7ffa4c018020 2024-02-19T15:28:35.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.590+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa58015040 2024-02-19T15:28:35.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.590+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 <== osd.1 v2:172.21.15.47:6800/1591346632 3 ==== osd_op_reply(24 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa500b0020 con 0x7ffa58015040 2024-02-19T15:28:35.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.590+0000 7ffa5f600640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] conn(0x7ffa500ef4a0 0x7ffa50116710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.590+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa500ef4a0 2024-02-19T15:28:35.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.590+0000 7ffa60000640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] conn(0x7ffa500ef4a0 0x7ffa50116710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.590+0000 7ffa60000640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] conn(0x7ffa500ef4a0 0x7ffa50116710 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:35.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.590+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.6 v2:172.21.15.116:6824/2921645124 1 ==== osd_op_reply(25 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x7ffa500ef4a0 2024-02-19T15:28:35.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.590+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa4c018020 2024-02-19T15:28:35.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.590+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.0 v2:172.21.15.47:6809/1968337531 7 ==== osd_op_reply(26 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x7ffa4c018020 2024-02-19T15:28:35.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.590+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa50009040 2024-02-19T15:28:35.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.590+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 <== osd.5 v2:172.21.15.116:6808/3478542733 2 ==== osd_op_reply(27 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa4c0197c0 con 0x7ffa50009040 2024-02-19T15:28:35.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.590+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa50009040 2024-02-19T15:28:35.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.590+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 <== osd.5 v2:172.21.15.116:6808/3478542733 3 ==== osd_op_reply(28 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa4c0197c0 con 0x7ffa50009040 2024-02-19T15:28:35.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.590+0000 7ffa5ec00640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] conn(0x7ffa4c0173b0 0x7ffa4c017810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.590+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa4c0173b0 2024-02-19T15:28:35.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.590+0000 7ffa5f600640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] conn(0x7ffa4c0173b0 0x7ffa4c017810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.590+0000 7ffa5f600640 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] conn(0x7ffa4c0173b0 0x7ffa4c017810 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:35.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.590+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 <== osd.2 v2:172.21.15.47:6808/2881016873 1 ==== osd_op_reply(29 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa500b0020 con 0x7ffa4c0173b0 2024-02-19T15:28:35.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.590+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa500ef4a0 2024-02-19T15:28:35.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.594+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.6 v2:172.21.15.116:6824/2921645124 2 ==== osd_op_reply(30 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x7ffa500ef4a0 2024-02-19T15:28:35.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.594+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa58015040 2024-02-19T15:28:35.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.594+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 <== osd.1 v2:172.21.15.47:6800/1591346632 4 ==== osd_op_reply(31 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa500b0020 con 0x7ffa58015040 2024-02-19T15:28:35.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.594+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cbfab0 2024-02-19T15:28:35.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.594+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 <== osd.4 v2:172.21.15.116:6809/3366344088 3 ==== osd_op_reply(32 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa500b0020 con 0x55e250cbfab0 2024-02-19T15:28:35.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.594+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa4c018020 2024-02-19T15:28:35.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.594+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.0 v2:172.21.15.47:6809/1968337531 8 ==== osd_op_reply(33 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x7ffa4c018020 2024-02-19T15:28:35.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.594+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa4c018020 2024-02-19T15:28:35.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.594+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.0 v2:172.21.15.47:6809/1968337531 9 ==== osd_op_reply(34 [pgnls start_epoch 114 out=49b] v108'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x7ffa4c018020 2024-02-19T15:28:35.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.594+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa500c8060 2024-02-19T15:28:35.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.594+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 <== osd.3 v2:172.21.15.47:6824/4149334728 7 ==== osd_op_reply(35 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa4c0197c0 con 0x7ffa500c8060 2024-02-19T15:28:35.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.594+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa58015040 2024-02-19T15:28:35.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.594+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 <== osd.1 v2:172.21.15.47:6800/1591346632 5 ==== osd_op_reply(36 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa500b0020 con 0x7ffa58015040 2024-02-19T15:28:35.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.594+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa500c8060 2024-02-19T15:28:35.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.594+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 <== osd.3 v2:172.21.15.47:6824/4149334728 8 ==== osd_op_reply(37 [pgnls start_epoch 114 out=49b] v108'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa4c0197c0 con 0x7ffa500c8060 2024-02-19T15:28:35.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.594+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cbfab0 2024-02-19T15:28:35.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.594+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 <== osd.4 v2:172.21.15.116:6809/3366344088 4 ==== osd_op_reply(38 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa500b0020 con 0x55e250cbfab0 2024-02-19T15:28:35.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.594+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa50009040 2024-02-19T15:28:35.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 <== osd.5 v2:172.21.15.116:6808/3478542733 4 ==== osd_op_reply(39 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa4c04b040 con 0x7ffa50009040 2024-02-19T15:28:35.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa50009040 2024-02-19T15:28:35.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 <== osd.5 v2:172.21.15.116:6808/3478542733 5 ==== osd_op_reply(40 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa4c04b040 con 0x7ffa50009040 2024-02-19T15:28:35.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa500ef4a0 2024-02-19T15:28:35.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.6 v2:172.21.15.116:6824/2921645124 3 ==== osd_op_reply(41 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x7ffa500ef4a0 2024-02-19T15:28:35.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa58015040 2024-02-19T15:28:35.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 <== osd.1 v2:172.21.15.47:6800/1591346632 6 ==== osd_op_reply(42 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa500b0020 con 0x7ffa58015040 2024-02-19T15:28:35.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa50009040 2024-02-19T15:28:35.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 <== osd.5 v2:172.21.15.116:6808/3478542733 6 ==== osd_op_reply(43 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa4c04b040 con 0x7ffa50009040 2024-02-19T15:28:35.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cbfab0 2024-02-19T15:28:35.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 <== osd.4 v2:172.21.15.116:6809/3366344088 5 ==== osd_op_reply(44 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa500b0020 con 0x55e250cbfab0 2024-02-19T15:28:35.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cb7870 2024-02-19T15:28:35.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.7 v2:172.21.15.116:6800/3212743377 7 ==== osd_op_reply(45 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x55e250cb7870 2024-02-19T15:28:35.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cb7870 2024-02-19T15:28:35.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.7 v2:172.21.15.116:6800/3212743377 8 ==== osd_op_reply(46 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x55e250cb7870 2024-02-19T15:28:35.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cb7870 2024-02-19T15:28:35.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.7 v2:172.21.15.116:6800/3212743377 9 ==== osd_op_reply(47 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x55e250cb7870 2024-02-19T15:28:35.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cb7870 2024-02-19T15:28:35.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.7 v2:172.21.15.116:6800/3212743377 10 ==== osd_op_reply(48 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x55e250cb7870 2024-02-19T15:28:35.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cbfab0 2024-02-19T15:28:35.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.598+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 <== osd.4 v2:172.21.15.116:6809/3366344088 6 ==== osd_op_reply(49 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa500b0020 con 0x55e250cbfab0 2024-02-19T15:28:35.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cbfab0 2024-02-19T15:28:35.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 <== osd.4 v2:172.21.15.116:6809/3366344088 7 ==== osd_op_reply(50 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa500b0020 con 0x55e250cbfab0 2024-02-19T15:28:35.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa500c8060 2024-02-19T15:28:35.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 <== osd.3 v2:172.21.15.47:6824/4149334728 9 ==== osd_op_reply(51 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa4c04b040 con 0x7ffa500c8060 2024-02-19T15:28:35.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cb7870 2024-02-19T15:28:35.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.7 v2:172.21.15.116:6800/3212743377 11 ==== osd_op_reply(52 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x55e250cb7870 2024-02-19T15:28:35.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa4c018020 2024-02-19T15:28:35.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.0 v2:172.21.15.47:6809/1968337531 10 ==== osd_op_reply(53 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x7ffa4c018020 2024-02-19T15:28:35.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cbfab0 2024-02-19T15:28:35.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 <== osd.4 v2:172.21.15.116:6809/3366344088 8 ==== osd_op_reply(54 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa500b0020 con 0x55e250cbfab0 2024-02-19T15:28:35.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cbfab0 2024-02-19T15:28:35.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 <== osd.4 v2:172.21.15.116:6809/3366344088 9 ==== osd_op_reply(55 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa500b0020 con 0x55e250cbfab0 2024-02-19T15:28:35.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa4c0173b0 2024-02-19T15:28:35.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 <== osd.2 v2:172.21.15.47:6808/2881016873 2 ==== osd_op_reply(56 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa500b0020 con 0x7ffa4c0173b0 2024-02-19T15:28:35.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa4c018020 2024-02-19T15:28:35.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.0 v2:172.21.15.47:6809/1968337531 11 ==== osd_op_reply(57 [pgnls start_epoch 114 out=49b] v114'12 uv12 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x7ffa4c018020 2024-02-19T15:28:35.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cbfab0 2024-02-19T15:28:35.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 <== osd.4 v2:172.21.15.116:6809/3366344088 10 ==== osd_op_reply(58 [pgnls start_epoch 114 out=49b] v108'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa500b0020 con 0x55e250cbfab0 2024-02-19T15:28:35.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa500ef4a0 2024-02-19T15:28:35.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.6 v2:172.21.15.116:6824/2921645124 4 ==== osd_op_reply(59 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x7ffa500ef4a0 2024-02-19T15:28:35.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.602+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa4c018020 2024-02-19T15:28:35.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.0 v2:172.21.15.47:6809/1968337531 12 ==== osd_op_reply(60 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x7ffa4c018020 2024-02-19T15:28:35.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa58015040 2024-02-19T15:28:35.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 <== osd.1 v2:172.21.15.47:6800/1591346632 7 ==== osd_op_reply(61 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa500b0020 con 0x7ffa58015040 2024-02-19T15:28:35.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x55e250cbfab0 2024-02-19T15:28:35.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 <== osd.4 v2:172.21.15.116:6809/3366344088 11 ==== osd_op_reply(62 [pgnls start_epoch 114 out=49b] v114'10 uv10 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa500b0020 con 0x55e250cbfab0 2024-02-19T15:28:35.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa5f600640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa500ef4a0 2024-02-19T15:28:35.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 <== osd.6 v2:172.21.15.116:6824/2921645124 5 ==== osd_op_reply(63 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa58002b20 con 0x7ffa500ef4a0 2024-02-19T15:28:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 --> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v8 -- 0x55e250cb81c0 con 0x7ffa50009040 2024-02-19T15:28:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa5ec00640 1 -- 172.21.15.47:0/2075973713 <== osd.5 v2:172.21.15.116:6808/3478542733 7 ==== osd_op_reply(64 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa4c04b040 con 0x7ffa50009040 2024-02-19T15:28:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 -- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x7ffa4c018020 msgr2=0x7ffa4c025a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:6809/1968337531,v1:172.21.15.47:6811/1968337531] conn(0x7ffa4c018020 0x7ffa4c025a70 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 -- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] conn(0x7ffa58015040 msgr2=0x7ffa58003040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:6800/1591346632,v1:172.21.15.47:6801/1591346632] conn(0x7ffa58015040 0x7ffa58003040 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 -- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] conn(0x7ffa4c0173b0 msgr2=0x7ffa4c017810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:6808/2881016873,v1:172.21.15.47:6810/2881016873] conn(0x7ffa4c0173b0 0x7ffa4c017810 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 -- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x7ffa500c8060 msgr2=0x7ffa500ee100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.47:6824/4149334728,v1:172.21.15.47:6825/4149334728] conn(0x7ffa500c8060 0x7ffa500ee100 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 -- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] conn(0x55e250cbfab0 msgr2=0x7ffa58017940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6809/3366344088,v1:172.21.15.116:6812/3366344088] conn(0x55e250cbfab0 0x7ffa58017940 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 -- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] conn(0x7ffa50009040 msgr2=0x7ffa5011c1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6808/3478542733,v1:172.21.15.116:6810/3478542733] conn(0x7ffa50009040 0x7ffa5011c1a0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 -- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] conn(0x7ffa500ef4a0 msgr2=0x7ffa50116710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6824/2921645124,v1:172.21.15.116:6825/2921645124] conn(0x7ffa500ef4a0 0x7ffa50116710 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 -- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] conn(0x55e250cb7870 msgr2=0x55e250cb7c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 reap_dead start 2024-02-19T15:28:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] conn(0x55e250cb7870 0x55e250cb7c80 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa60000640 1 -- 172.21.15.47:0/2075973713 reap_dead start 2024-02-19T15:28:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 -- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffa40065820 msgr2=0x7ffa40067ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffa40065820 0x7ffa40067ce0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c001660 tx=0x7ffa4c00b000 comp rx=0 tx=0).stop 2024-02-19T15:28:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 -- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55e250d31070 msgr2=0x55e250de3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55e250d31070 0x55e250de3cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ffa50096f10 tx=0x7ffa50005a10 comp rx=0 tx=0).stop 2024-02-19T15:28:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 -- 172.21.15.47:0/2075973713 shutdown_connections 2024-02-19T15:28:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6800/3212743377,v1:172.21.15.116:6801/3212743377] conn(0x55e250cb7870 0x55e250cb7c80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffa40065820 0x7ffa40067ce0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 --2- 172.21.15.47:0/2075973713 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x55e250d31070 0x55e250de3cc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 -- 172.21.15.47:0/2075973713 >> 172.21.15.47:0/2075973713 conn(0x55e250ca2680 msgr2=0x55e250d35270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:35.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 -- 172.21.15.47:0/2075973713 shutdown_connections 2024-02-19T15:28:35.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.606+0000 7ffa62447080 1 -- 172.21.15.47:0/2075973713 wait complete. 2024-02-19T15:28:35.612 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 are absent 2024-02-19T15:28:35.612 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:35.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.794+0000 7fa9b7000640 1 Processor -- start 2024-02-19T15:28:35.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.794+0000 7fa9b7000640 1 -- start start 2024-02-19T15:28:35.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.794+0000 7fa9b7000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa9b00cea80 0x7fa9b00727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.794+0000 7fa9b7000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b015c390 0x7fa9b015c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.794+0000 7fa9b7000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b015d280 con 0x7fa9b0145590 2024-02-19T15:28:35.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.794+0000 7fa9b7000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b0070ff0 con 0x7fa9b00cea80 2024-02-19T15:28:35.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.794+0000 7fa9b7000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b015cd60 con 0x7fa9b015c390 2024-02-19T15:28:35.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.794+0000 7fa9af400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa9b00cea80 0x7fa9b00727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.794+0000 7fa9af400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa9b00cea80 0x7fa9b00727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59894/0 (socket says 172.21.15.47:59894) 2024-02-19T15:28:35.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.794+0000 7fa9af400640 1 -- 172.21.15.47:0/3337148119 learned_addr learned my addr 172.21.15.47:0/3337148119 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:35.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.794+0000 7fa9b5000640 1 --2- 172.21.15.47:0/3337148119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b015c390 0x7fa9b015c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.794+0000 7fa9af400640 1 -- 172.21.15.47:0/3337148119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b015c390 msgr2=0x7fa9b015c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:35.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.794+0000 7fa9af400640 1 --2- 172.21.15.47:0/3337148119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b015c390 0x7fa9b015c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.794+0000 7fa9af400640 1 -- 172.21.15.47:0/3337148119 >> v1:172.21.15.116:6789/0 conn(0x7fa9b0145590 legacy=0x7fa9b006c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:35.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.794+0000 7fa9af400640 1 -- 172.21.15.47:0/3337148119 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b00709e0 con 0x7fa9b00cea80 2024-02-19T15:28:35.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.798+0000 7fa9af400640 1 --2- 172.21.15.47:0/3337148119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00cea80 0x7fa9b00727b0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fa99c009870 tx=0x7fa99c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1416d9fb80270b75 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:35.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.798+0000 7fa9aea00640 1 -- 172.21.15.47:0/3337148119 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99c0ab650 con 0x7fa9b00cea80 2024-02-19T15:28:35.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.798+0000 7fa9aea00640 1 -- 172.21.15.47:0/3337148119 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99c0ab7b0 con 0x7fa9b00cea80 2024-02-19T15:28:35.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.798+0000 7fa9aea00640 1 -- 172.21.15.47:0/3337148119 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99c0abb40 con 0x7fa9b00cea80 2024-02-19T15:28:35.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.798+0000 7fa9b7000640 1 -- 172.21.15.47:0/3337148119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00cea80 msgr2=0x7fa9b00727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.798+0000 7fa9b7000640 1 --2- 172.21.15.47:0/3337148119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00cea80 0x7fa9b00727b0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fa99c009870 tx=0x7fa99c01c560 comp rx=0 tx=0).stop 2024-02-19T15:28:35.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.798+0000 7fa9b7000640 1 -- 172.21.15.47:0/3337148119 shutdown_connections 2024-02-19T15:28:35.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.798+0000 7fa9b7000640 1 --2- 172.21.15.47:0/3337148119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b015c390 0x7fa9b015c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.798+0000 7fa9b7000640 1 --2- 172.21.15.47:0/3337148119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00cea80 0x7fa9b00727b0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.798+0000 7fa9b7000640 1 -- 172.21.15.47:0/3337148119 >> 172.21.15.47:0/3337148119 conn(0x7fa9b00b16a0 msgr2=0x7fa9b00b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:35.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.798+0000 7fa9b7000640 1 -- 172.21.15.47:0/3337148119 shutdown_connections 2024-02-19T15:28:35.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.798+0000 7fa9b7000640 1 -- 172.21.15.47:0/3337148119 wait complete. 2024-02-19T15:28:35.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.798+0000 7fa9b7000640 1 Processor -- start 2024-02-19T15:28:35.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.798+0000 7fa9b7000640 1 -- start start 2024-02-19T15:28:35.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9b7000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9b00cea80 0x7fa9b013c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9b7000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b0145590 0x7fa9b006d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9b7000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b015c390 0x7fa9b006d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9b7000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b0143fc0 con 0x7fa9b0145590 2024-02-19T15:28:35.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9b7000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b01433b0 con 0x7fa9b00cea80 2024-02-19T15:28:35.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9b7000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b0143690 con 0x7fa9b015c390 2024-02-19T15:28:35.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9af400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b0145590 0x7fa9b006d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9af400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b0145590 0x7fa9b006d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59900/0 (socket says 172.21.15.47:59900) 2024-02-19T15:28:35.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9af400640 1 -- 172.21.15.47:0/291869130 learned_addr learned my addr 172.21.15.47:0/291869130 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:35.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9afe00640 1 --2- 172.21.15.47:0/291869130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9b00cea80 0x7fa9b013c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9b5000640 1 --2- 172.21.15.47:0/291869130 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b015c390 0x7fa9b006d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9af400640 1 -- 172.21.15.47:0/291869130 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b015c390 msgr2=0x7fa9b006d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9af400640 1 --2- 172.21.15.47:0/291869130 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b015c390 0x7fa9b006d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9af400640 1 -- 172.21.15.47:0/291869130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9b00cea80 msgr2=0x7fa9b013c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:35.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9af400640 1 --2- 172.21.15.47:0/291869130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9b00cea80 0x7fa9b013c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:35.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9af400640 1 -- 172.21.15.47:0/291869130 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c0094d0 con 0x7fa9b0145590 2024-02-19T15:28:35.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9af400640 1 --2- 172.21.15.47:0/291869130 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b0145590 0x7fa9b006d100 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fa99c01ce00 tx=0x7fa99c01ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:35.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9a7e00640 1 -- 172.21.15.47:0/291869130 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99c0ab650 con 0x7fa9b0145590 2024-02-19T15:28:35.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9a7e00640 1 -- 172.21.15.47:0/291869130 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99c0b4800 con 0x7fa9b0145590 2024-02-19T15:28:35.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9a7e00640 1 -- 172.21.15.47:0/291869130 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99c0b4b90 con 0x7fa9b0145590 2024-02-19T15:28:35.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9b7000640 1 -- 172.21.15.47:0/291869130 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b0143970 con 0x7fa9b0145590 2024-02-19T15:28:35.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9b7000640 1 -- 172.21.15.47:0/291869130 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b013c940 con 0x7fa9b0145590 2024-02-19T15:28:35.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9b7000640 1 -- 172.21.15.47:0/291869130 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b0144130 con 0x7fa9b0145590 2024-02-19T15:28:35.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.802+0000 7fa9a7e00640 1 -- 172.21.15.47:0/291869130 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa99c0b4cf0 con 0x7fa9b0145590 2024-02-19T15:28:35.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.806+0000 7fa9a7e00640 1 --2- 172.21.15.47:0/291869130 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa990066e40 0x7fa990069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:35.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.806+0000 7fa9a7e00640 1 -- 172.21.15.47:0/291869130 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fa99c11f0d0 con 0x7fa9b0145590 2024-02-19T15:28:35.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.806+0000 7fa9afe00640 1 --2- 172.21.15.47:0/291869130 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa990066e40 0x7fa990069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:35.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.806+0000 7fa9afe00640 1 --2- 172.21.15.47:0/291869130 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa990066e40 0x7fa990069300 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0001810 tx=0x7fa9a0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:35.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:35.810+0000 7fa9a7e00640 1 -- 172.21.15.47:0/291869130 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa99c0ebde0 con 0x7fa9b0145590 2024-02-19T15:28:35.899 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:28:35.902 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:28:36.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.006+0000 7fa9b7000640 1 -- 172.21.15.47:0/291869130 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9b01437d0 con 0x7fa9b0145590 2024-02-19T15:28:36.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.006+0000 7fa9a7e00640 1 -- 172.21.15.47:0/291869130 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7fa9b01437d0 con 0x7fa9b0145590 2024-02-19T15:28:36.008 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:36.008 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:36.008 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:36.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.010+0000 7fa9b7000640 1 -- 172.21.15.47:0/291869130 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa990066e40 msgr2=0x7fa990069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:36.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.010+0000 7fa9b7000640 1 --2- 172.21.15.47:0/291869130 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa990066e40 0x7fa990069300 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0001810 tx=0x7fa9a0009040 comp rx=0 tx=0).stop 2024-02-19T15:28:36.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.010+0000 7fa9b7000640 1 -- 172.21.15.47:0/291869130 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b0145590 msgr2=0x7fa9b006d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:36.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.010+0000 7fa9b7000640 1 --2- 172.21.15.47:0/291869130 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b0145590 0x7fa9b006d100 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fa99c01ce00 tx=0x7fa99c01ada0 comp rx=0 tx=0).stop 2024-02-19T15:28:36.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.010+0000 7fa9b7000640 1 -- 172.21.15.47:0/291869130 shutdown_connections 2024-02-19T15:28:36.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.010+0000 7fa9b7000640 1 --2- 172.21.15.47:0/291869130 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa990066e40 0x7fa990069300 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:36.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.010+0000 7fa9b7000640 1 --2- 172.21.15.47:0/291869130 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b015c390 0x7fa9b006d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:36.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.010+0000 7fa9b7000640 1 --2- 172.21.15.47:0/291869130 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b0145590 0x7fa9b006d100 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:36.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.010+0000 7fa9b7000640 1 --2- 172.21.15.47:0/291869130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9b00cea80 0x7fa9b013c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:36.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.010+0000 7fa9b7000640 1 -- 172.21.15.47:0/291869130 >> 172.21.15.47:0/291869130 conn(0x7fa9b00b16a0 msgr2=0x7fa9b00af510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:36.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.010+0000 7fa9b7000640 1 -- 172.21.15.47:0/291869130 shutdown_connections 2024-02-19T15:28:36.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.010+0000 7fa9b7000640 1 -- 172.21.15.47:0/291869130 wait complete. 2024-02-19T15:28:36.037 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-02-19T15:28:36.198 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"strays_created":3}} 2024-02-19T15:28:36.198 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 3 } } 2024-02-19T15:28:36.199 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:36.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f5321c00640 1 Processor -- start 2024-02-19T15:28:36.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f5321c00640 1 -- start start 2024-02-19T15:28:36.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f5321c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f531c166b60 0x7f531c168f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:36.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f5321c00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531c0ad840 con 0x7f531c06aeb0 2024-02-19T15:28:36.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f5321c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531c0c62f0 con 0x7f531c0cb520 2024-02-19T15:28:36.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f5321c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531c0cc810 con 0x7f531c166b60 2024-02-19T15:28:36.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f531aa00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f531c06aeb0 0x7f531c06b290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43634/0 (socket says 172.21.15.47:43634) 2024-02-19T15:28:36.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f531aa00640 1 -- 172.21.15.47:0/3287744863 learned_addr learned my addr 172.21.15.47:0/3287744863 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:36.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f531b400640 1 --2- 172.21.15.47:0/3287744863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f531c166b60 0x7f531c168f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:36.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f531a000640 1 -- 172.21.15.47:0/3287744863 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344800941 0 0) 0x7f531c0ad840 con 0x7f531c06aeb0 2024-02-19T15:28:36.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f531a000640 1 -- 172.21.15.47:0/3287744863 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5300003530 con 0x7f531c06aeb0 2024-02-19T15:28:36.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f531a000640 1 -- 172.21.15.47:0/3287744863 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102987780 0 0) 0x7f531c0c62f0 con 0x7f531c0cb520 2024-02-19T15:28:36.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f531a000640 1 -- 172.21.15.47:0/3287744863 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f531c0ad840 con 0x7f531c0cb520 2024-02-19T15:28:36.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f531a000640 1 -- 172.21.15.47:0/3287744863 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4112083964 0 0) 0x7f5300003530 con 0x7f531c06aeb0 2024-02-19T15:28:36.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f531a000640 1 -- 172.21.15.47:0/3287744863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f531c166b60 msgr2=0x7f531c168f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:36.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f531a000640 1 --2- 172.21.15.47:0/3287744863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f531c166b60 0x7f531c168f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:36.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f531a000640 1 -- 172.21.15.47:0/3287744863 >> v1:172.21.15.116:6789/0 conn(0x7f531c0cb520 legacy=0x7f531c0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:36.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f531a000640 1 -- 172.21.15.47:0/3287744863 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531c0cc250 con 0x7f531c06aeb0 2024-02-19T15:28:36.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f531a000640 1 -- 172.21.15.47:0/3287744863 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f530c001680 con 0x7f531c06aeb0 2024-02-19T15:28:36.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f531a000640 1 -- 172.21.15.47:0/3287744863 >> v1:172.21.15.47:6789/0 conn(0x7f531c06aeb0 legacy=0x7f531c06b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:36.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f531a000640 1 --2- 172.21.15.47:0/3287744863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5300003b40 0x7f5300003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:36.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f531a000640 1 --2- 172.21.15.47:0/3287744863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53000047d0 0x7f5300004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:36.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f531a000640 1 --2- 172.21.15.47:0/3287744863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f531c166b60 0x7f5300005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:36.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f531a000640 1 -- 172.21.15.47:0/3287744863 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c0cc810 con 0x7f53000047d0 2024-02-19T15:28:36.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f531a000640 1 -- 172.21.15.47:0/3287744863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530000ba70 con 0x7f5300003b40 2024-02-19T15:28:36.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.338+0000 7f531a000640 1 -- 172.21.15.47:0/3287744863 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f530000bc30 con 0x7f531c166b60 2024-02-19T15:28:36.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.342+0000 7f5312000640 1 --2- 172.21.15.47:0/3287744863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53000047d0 0x7f5300004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:36.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.342+0000 7f531aa00640 1 --2- 172.21.15.47:0/3287744863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5300003b40 0x7f5300003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:36.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.342+0000 7f531b400640 1 --2- 172.21.15.47:0/3287744863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f531c166b60 0x7f5300005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:36.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.342+0000 7f5312000640 1 -- 172.21.15.47:0/3287744863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f531c166b60 msgr2=0x7f5300005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:36.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.342+0000 7f5312000640 1 --2- 172.21.15.47:0/3287744863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f531c166b60 0x7f5300005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:36.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.342+0000 7f5312000640 1 -- 172.21.15.47:0/3287744863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5300003b40 msgr2=0x7f5300003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:36.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.342+0000 7f5312000640 1 --2- 172.21.15.47:0/3287744863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5300003b40 0x7f5300003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:36.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.342+0000 7f5312000640 1 -- 172.21.15.47:0/3287744863 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5300003530 con 0x7f53000047d0 2024-02-19T15:28:36.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.342+0000 7f5312000640 1 --2- 172.21.15.47:0/3287744863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53000047d0 0x7f5300004c50 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f5304007fd0 tx=0x7f530401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e010b9cf53f7548 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:36.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.342+0000 7f531a000640 1 -- 172.21.15.47:0/3287744863 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53040052d0 con 0x7f53000047d0 2024-02-19T15:28:36.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.342+0000 7f531a000640 1 -- 172.21.15.47:0/3287744863 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5304005450 con 0x7f53000047d0 2024-02-19T15:28:36.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.342+0000 7f531a000640 1 -- 172.21.15.47:0/3287744863 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53040a8910 con 0x7f53000047d0 2024-02-19T15:28:36.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5321c00640 1 -- 172.21.15.47:0/3287744863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53000047d0 msgr2=0x7f5300004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:36.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5321c00640 1 --2- 172.21.15.47:0/3287744863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53000047d0 0x7f5300004c50 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f5304007fd0 tx=0x7f530401ecb0 comp rx=0 tx=0).stop 2024-02-19T15:28:36.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f531b400640 1 -- 172.21.15.47:0/3287744863 reap_dead start 2024-02-19T15:28:36.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5321c00640 1 -- 172.21.15.47:0/3287744863 shutdown_connections 2024-02-19T15:28:36.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5321c00640 1 -- 172.21.15.47:0/3287744863 >> 172.21.15.47:0/3287744863 conn(0x7f531c0d0ad0 msgr2=0x7f531c0d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:36.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5321c00640 1 -- 172.21.15.47:0/3287744863 shutdown_connections 2024-02-19T15:28:36.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5321c00640 1 -- 172.21.15.47:0/3287744863 wait complete. 2024-02-19T15:28:36.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5321c00640 1 Processor -- start 2024-02-19T15:28:36.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5321c00640 1 -- start start 2024-02-19T15:28:36.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5321c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53000047d0 0x7f531c0b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:36.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5321c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f531c15ac80 0x7f531c157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:36.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5321c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f531c15d480 0x7f531c15d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:36.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5321c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c15de40 con 0x7f531c15ac80 2024-02-19T15:28:36.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5321c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c0b5530 con 0x7f53000047d0 2024-02-19T15:28:36.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5321c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531c15b100 con 0x7f531c15d480 2024-02-19T15:28:36.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5312000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f531c15ac80 0x7f531c157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:36.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5312000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f531c15ac80 0x7f531c157e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59926/0 (socket says 172.21.15.47:59926) 2024-02-19T15:28:36.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5312000640 1 -- 172.21.15.47:0/2556626169 learned_addr learned my addr 172.21.15.47:0/2556626169 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:36.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f531aa00640 1 --2- 172.21.15.47:0/2556626169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53000047d0 0x7f531c0b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:36.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f531b400640 1 --2- 172.21.15.47:0/2556626169 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f531c15d480 0x7f531c15d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:36.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5312000640 1 -- 172.21.15.47:0/2556626169 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f531c15d480 msgr2=0x7f531c15d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:36.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5312000640 1 --2- 172.21.15.47:0/2556626169 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f531c15d480 0x7f531c15d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:36.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5312000640 1 -- 172.21.15.47:0/2556626169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53000047d0 msgr2=0x7f531c0b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:36.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5312000640 1 --2- 172.21.15.47:0/2556626169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53000047d0 0x7f531c0b4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:36.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f5312000640 1 -- 172.21.15.47:0/2556626169 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531c0b4920 con 0x7f531c15ac80 2024-02-19T15:28:36.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.346+0000 7f531b400640 1 --2- 172.21.15.47:0/2556626169 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f531c15d480 0x7f531c15d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:36.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.350+0000 7f5312000640 1 --2- 172.21.15.47:0/2556626169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f531c15ac80 0x7f531c157e50 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f5304005040 tx=0x7f5300005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:36.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.350+0000 7f5313400640 1 -- 172.21.15.47:0/2556626169 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53040a9da0 con 0x7f531c15ac80 2024-02-19T15:28:36.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.350+0000 7f5313400640 1 -- 172.21.15.47:0/2556626169 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53040050d0 con 0x7f531c15ac80 2024-02-19T15:28:36.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.350+0000 7f5313400640 1 -- 172.21.15.47:0/2556626169 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5304003600 con 0x7f531c15ac80 2024-02-19T15:28:36.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.350+0000 7f5321c00640 1 -- 172.21.15.47:0/2556626169 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f531c0b4c00 con 0x7f531c15ac80 2024-02-19T15:28:36.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.350+0000 7f5321c00640 1 -- 172.21.15.47:0/2556626169 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f531c0b51c0 con 0x7f531c15ac80 2024-02-19T15:28:36.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.350+0000 7f5321c00640 1 -- 172.21.15.47:0/2556626169 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f531c0b5380 con 0x7f531c15ac80 2024-02-19T15:28:36.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.350+0000 7f5313400640 1 -- 172.21.15.47:0/2556626169 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5304003760 con 0x7f531c15ac80 2024-02-19T15:28:36.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.350+0000 7f5313400640 1 --2- 172.21.15.47:0/2556626169 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52f8066e70 0x7f52f8069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:36.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.350+0000 7f5313400640 1 -- 172.21.15.47:0/2556626169 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f5304121780 con 0x7f531c15ac80 2024-02-19T15:28:36.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.350+0000 7f531aa00640 1 --2- 172.21.15.47:0/2556626169 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52f8066e70 0x7f52f8069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:36.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.354+0000 7f531aa00640 1 --2- 172.21.15.47:0/2556626169 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52f8066e70 0x7f52f8069330 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f530c001060 tx=0x7f530c001e30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:36.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.358+0000 7f5313400640 1 -- 172.21.15.47:0/2556626169 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f53040ee3e0 con 0x7f531c15ac80 2024-02-19T15:28:36.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.562+0000 7f5321c00640 1 -- 172.21.15.47:0/2556626169 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f531c0b5380 con 0x7f531c15ac80 2024-02-19T15:28:36.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.562+0000 7f5313400640 1 -- 172.21.15.47:0/2556626169 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f53040f1aa0 con 0x7f531c15ac80 2024-02-19T15:28:36.564 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:36.564 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:36.565 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:36.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.566+0000 7f5321c00640 1 -- 172.21.15.47:0/2556626169 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52f8066e70 msgr2=0x7f52f8069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:36.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.566+0000 7f5321c00640 1 --2- 172.21.15.47:0/2556626169 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52f8066e70 0x7f52f8069330 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f530c001060 tx=0x7f530c001e30 comp rx=0 tx=0).stop 2024-02-19T15:28:36.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.566+0000 7f5321c00640 1 -- 172.21.15.47:0/2556626169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f531c15ac80 msgr2=0x7f531c157e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:36.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.566+0000 7f5321c00640 1 --2- 172.21.15.47:0/2556626169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f531c15ac80 0x7f531c157e50 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f5304005040 tx=0x7f5300005800 comp rx=0 tx=0).stop 2024-02-19T15:28:36.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.566+0000 7f5321c00640 1 -- 172.21.15.47:0/2556626169 shutdown_connections 2024-02-19T15:28:36.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.566+0000 7f5321c00640 1 --2- 172.21.15.47:0/2556626169 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52f8066e70 0x7f52f8069330 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:36.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.566+0000 7f5321c00640 1 --2- 172.21.15.47:0/2556626169 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f531c15d480 0x7f531c15d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:36.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.566+0000 7f5321c00640 1 --2- 172.21.15.47:0/2556626169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f531c15ac80 0x7f531c157e50 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:36.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.566+0000 7f5321c00640 1 --2- 172.21.15.47:0/2556626169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53000047d0 0x7f531c0b4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:36.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.566+0000 7f5321c00640 1 -- 172.21.15.47:0/2556626169 >> 172.21.15.47:0/2556626169 conn(0x7f531c0d0ad0 msgr2=0x7f531c0b12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:36.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.566+0000 7f5321c00640 1 -- 172.21.15.47:0/2556626169 shutdown_connections 2024-02-19T15:28:36.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:36.566+0000 7f5321c00640 1 -- 172.21.15.47:0/2556626169 wait complete. 2024-02-19T15:28:36.585 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-02-19T15:28:36.746 INFO:teuthology.orchestra.run.smithi047.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-02-19T15:28:36.746 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-02-19T15:28:36.747 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T15:28:36.747 DEBUG:teuthology.orchestra.run.smithi047:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:28:36.815 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:28:36.915 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T15:28:36.916 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:28:36.916 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:28:36.924 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:28:36.925 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:28:36.976 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:28:37.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.154+0000 7f7543600640 1 Processor -- start 2024-02-19T15:28:37.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.154+0000 7f7543600640 1 -- start start 2024-02-19T15:28:37.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.158+0000 7f7543600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f753c06d650 0x7f753c0d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:37.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.158+0000 7f7543600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f753c0cce30 con 0x7f753c0ced10 2024-02-19T15:28:37.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.158+0000 7f7543600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f753c0cd110 con 0x7f753c0cded0 2024-02-19T15:28:37.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.158+0000 7f7543600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f753c0cd3f0 con 0x7f753c06d650 2024-02-19T15:28:37.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.158+0000 7f7541600640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f753c0ced10 0x7f753c06bd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43646/0 (socket says 172.21.15.47:43646) 2024-02-19T15:28:37.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.158+0000 7f7541600640 1 -- 172.21.15.47:0/2477215876 learned_addr learned my addr 172.21.15.47:0/2477215876 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:37.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.158+0000 7f7540c00640 1 --2- 172.21.15.47:0/2477215876 >> v2:172.21.15.116:3300/0 conn(0x7f753c06d650 0x7f753c0d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:37.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.158+0000 7f753b400640 1 -- 172.21.15.47:0/2477215876 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208251990 0 0) 0x7f753c0cce30 con 0x7f753c0ced10 2024-02-19T15:28:37.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.158+0000 7f753b400640 1 -- 172.21.15.47:0/2477215876 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7524003530 con 0x7f753c0ced10 2024-02-19T15:28:37.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.158+0000 7f753b400640 1 -- 172.21.15.47:0/2477215876 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113759158 0 0) 0x7f753c0cd110 con 0x7f753c0cded0 2024-02-19T15:28:37.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.158+0000 7f753b400640 1 -- 172.21.15.47:0/2477215876 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f753c0cce30 con 0x7f753c0cded0 2024-02-19T15:28:37.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.158+0000 7f753b400640 1 -- 172.21.15.47:0/2477215876 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3643150240 0 0) 0x7f7524003530 con 0x7f753c0ced10 2024-02-19T15:28:37.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.158+0000 7f7540c00640 1 -- 172.21.15.47:0/2477215876 >> v1:172.21.15.116:6789/0 conn(0x7f753c0cded0 legacy=0x7f753c0ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:37.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.158+0000 7f7540c00640 1 -- 172.21.15.47:0/2477215876 >> v1:172.21.15.47:6789/0 conn(0x7f753c0ced10 legacy=0x7f753c06bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:37.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.158+0000 7f7540c00640 1 -- 172.21.15.47:0/2477215876 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f753c0cd6d0 con 0x7f753c06d650 2024-02-19T15:28:37.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.162+0000 7f7540c00640 1 --2- 172.21.15.47:0/2477215876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f753c06d650 0x7f753c0d08b0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f75300097f0 tx=0x7f753001c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec3963ea26bb6415 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:37.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.162+0000 7f753b400640 1 -- 172.21.15.47:0/2477215876 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7534001a00 con 0x7f753c06d650 2024-02-19T15:28:37.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.162+0000 7f753b400640 1 -- 172.21.15.47:0/2477215876 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75300a7050 con 0x7f753c06d650 2024-02-19T15:28:37.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.162+0000 7f753b400640 1 -- 172.21.15.47:0/2477215876 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75300ab620 con 0x7f753c06d650 2024-02-19T15:28:37.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.162+0000 7f7543600640 1 -- 172.21.15.47:0/2477215876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f753c06d650 msgr2=0x7f753c0d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:37.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.162+0000 7f7543600640 1 --2- 172.21.15.47:0/2477215876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f753c06d650 0x7f753c0d08b0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f75300097f0 tx=0x7f753001c300 comp rx=0 tx=0).stop 2024-02-19T15:28:37.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.162+0000 7f7543600640 1 -- 172.21.15.47:0/2477215876 shutdown_connections 2024-02-19T15:28:37.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.162+0000 7f7543600640 1 --2- 172.21.15.47:0/2477215876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f753c06d650 0x7f753c0d08b0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.162+0000 7f7543600640 1 -- 172.21.15.47:0/2477215876 >> 172.21.15.47:0/2477215876 conn(0x7f753c0d1bf0 msgr2=0x7f753c0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:37.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.162+0000 7f7543600640 1 -- 172.21.15.47:0/2477215876 shutdown_connections 2024-02-19T15:28:37.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.162+0000 7f7543600640 1 -- 172.21.15.47:0/2477215876 wait complete. 2024-02-19T15:28:37.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.162+0000 7f7543600640 1 Processor -- start 2024-02-19T15:28:37.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.162+0000 7f7543600640 1 -- start start 2024-02-19T15:28:37.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7543600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f753c06d650 0x7f753c152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:37.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7543600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f753c0cded0 0x7f753c1539c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:37.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7543600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f753c0ced10 0x7f753c153f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:37.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7543600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f753c153730 con 0x7f753c0ced10 2024-02-19T15:28:37.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7543600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f753c155830 con 0x7f753c0cded0 2024-02-19T15:28:37.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7543600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f753c0c56f0 con 0x7f753c06d650 2024-02-19T15:28:37.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7541600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f753c0ced10 0x7f753c153f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:37.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7541600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f753c0ced10 0x7f753c153f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59942/0 (socket says 172.21.15.47:59942) 2024-02-19T15:28:37.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7541600640 1 -- 172.21.15.47:0/2870927734 learned_addr learned my addr 172.21.15.47:0/2870927734 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:37.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f753be00640 1 --2- 172.21.15.47:0/2870927734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f753c0cded0 0x7f753c1539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:37.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7540c00640 1 --2- 172.21.15.47:0/2870927734 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f753c06d650 0x7f753c152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:37.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7541600640 1 -- 172.21.15.47:0/2870927734 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f753c06d650 msgr2=0x7f753c152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:37.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7541600640 1 --2- 172.21.15.47:0/2870927734 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f753c06d650 0x7f753c152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7541600640 1 -- 172.21.15.47:0/2870927734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f753c0cded0 msgr2=0x7f753c1539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:37.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7541600640 1 --2- 172.21.15.47:0/2870927734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f753c0cded0 0x7f753c1539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7541600640 1 -- 172.21.15.47:0/2870927734 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7530009420 con 0x7f753c0ced10 2024-02-19T15:28:37.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7541600640 1 --2- 172.21.15.47:0/2870927734 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f753c0ced10 0x7f753c153f00 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f753400cc70 tx=0x7f753400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:37.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7538c00640 1 -- 172.21.15.47:0/2870927734 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753400ce80 con 0x7f753c0ced10 2024-02-19T15:28:37.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7538c00640 1 -- 172.21.15.47:0/2870927734 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7534012400 con 0x7f753c0ced10 2024-02-19T15:28:37.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7538c00640 1 -- 172.21.15.47:0/2870927734 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7534012790 con 0x7f753c0ced10 2024-02-19T15:28:37.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7543600640 1 -- 172.21.15.47:0/2870927734 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f753c0c59d0 con 0x7f753c0ced10 2024-02-19T15:28:37.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7543600640 1 -- 172.21.15.47:0/2870927734 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f753c153180 con 0x7f753c0ced10 2024-02-19T15:28:37.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7543600640 1 -- 172.21.15.47:0/2870927734 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f753c0c61d0 con 0x7f753c0ced10 2024-02-19T15:28:37.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.166+0000 7f7538c00640 1 -- 172.21.15.47:0/2870927734 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7534007070 con 0x7f753c0ced10 2024-02-19T15:28:37.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.170+0000 7f7538c00640 1 --2- 172.21.15.47:0/2870927734 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f751c066e40 0x7f751c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:37.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.170+0000 7f7538c00640 1 -- 172.21.15.47:0/2870927734 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f7534084c50 con 0x7f753c0ced10 2024-02-19T15:28:37.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.170+0000 7f7540c00640 1 --2- 172.21.15.47:0/2870927734 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f751c066e40 0x7f751c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:37.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.170+0000 7f7540c00640 1 --2- 172.21.15.47:0/2870927734 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f751c066e40 0x7f751c069300 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f75300ad000 tx=0x7f75300a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:37.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.174+0000 7f7538c00640 1 -- 172.21.15.47:0/2870927734 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f753c0c61d0 con 0x7f753c0ced10 2024-02-19T15:28:37.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.342+0000 7f7543600640 1 -- 172.21.15.47:0/2870927734 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f753c0c61d0 con 0x7f753c0ced10 2024-02-19T15:28:37.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.342+0000 7f7538c00640 1 -- 172.21.15.47:0/2870927734 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+16990 (secure 0 0 0) 0x7f753c0c61d0 con 0x7f753c0ced10 2024-02-19T15:28:37.344 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:37.345 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":114,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:27:47.287065+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:27:19.839144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-02-19T15:27:20.405924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.36","mappings":[{"from":0,"to":5}]},{"pgid":"17.3f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6837/819751716":"2024-02-20T15:27:39.101989+0000","172.21.15.116:6835/819751716":"2024-02-20T15:27:39.101989+0000","172.21.15.47:6834/897369244":"2024-02-20T15:27:38.263490+0000","172.21.15.47:6832/897369244":"2024-02-20T15:27:38.263490+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:28:37.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.346+0000 7f7543600640 1 -- 172.21.15.47:0/2870927734 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f751c066e40 msgr2=0x7f751c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:37.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.346+0000 7f7543600640 1 --2- 172.21.15.47:0/2870927734 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f751c066e40 0x7f751c069300 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f75300ad000 tx=0x7f75300a6000 comp rx=0 tx=0).stop 2024-02-19T15:28:37.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.346+0000 7f7543600640 1 -- 172.21.15.47:0/2870927734 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f753c0ced10 msgr2=0x7f753c153f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:37.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.346+0000 7f7543600640 1 --2- 172.21.15.47:0/2870927734 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f753c0ced10 0x7f753c153f00 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f753400cc70 tx=0x7f753400a430 comp rx=0 tx=0).stop 2024-02-19T15:28:37.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.346+0000 7f7543600640 1 -- 172.21.15.47:0/2870927734 shutdown_connections 2024-02-19T15:28:37.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.346+0000 7f7543600640 1 --2- 172.21.15.47:0/2870927734 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f751c066e40 0x7f751c069300 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.346+0000 7f7543600640 1 --2- 172.21.15.47:0/2870927734 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f753c0ced10 0x7f753c153f00 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.346+0000 7f7543600640 1 --2- 172.21.15.47:0/2870927734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f753c0cded0 0x7f753c1539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.346+0000 7f7543600640 1 --2- 172.21.15.47:0/2870927734 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f753c06d650 0x7f753c152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.346+0000 7f7543600640 1 -- 172.21.15.47:0/2870927734 >> 172.21.15.47:0/2870927734 conn(0x7f753c0d1bf0 msgr2=0x7f753c159390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:37.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.346+0000 7f7543600640 1 -- 172.21.15.47:0/2870927734 shutdown_connections 2024-02-19T15:28:37.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.346+0000 7f7543600640 1 -- 172.21.15.47:0/2870927734 wait complete. 2024-02-19T15:28:37.366 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:27:19.839144+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:27:20.405924+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:28:37.367 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:28:37.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.510+0000 7fe966000640 1 Processor -- start 2024-02-19T15:28:37.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.510+0000 7fe966000640 1 -- start start 2024-02-19T15:28:37.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.510+0000 7fe966000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe9600ce3b0 0x7fe9600ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:37.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.510+0000 7fe966000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe96015e280 0x7fe9600cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:37.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.510+0000 7fe966000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9600cc620 con 0x7fe9600d2650 2024-02-19T15:28:37.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.510+0000 7fe966000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9600d12e0 con 0x7fe9600ce3b0 2024-02-19T15:28:37.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.510+0000 7fe966000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9600d1940 con 0x7fe96015e280 2024-02-19T15:28:37.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.510+0000 7fe95f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe9600ce3b0 0x7fe9600ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:37.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.510+0000 7fe95f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe9600ce3b0 0x7fe9600ce790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59954/0 (socket says 172.21.15.47:59954) 2024-02-19T15:28:37.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.510+0000 7fe95f400640 1 -- 172.21.15.47:0/3696691631 learned_addr learned my addr 172.21.15.47:0/3696691631 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:37.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.510+0000 7fe95fe00640 1 --2- 172.21.15.47:0/3696691631 >> v2:172.21.15.116:3300/0 conn(0x7fe96015e280 0x7fe9600cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:37.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.514+0000 7fe95f400640 1 -- 172.21.15.47:0/3696691631 >> v2:172.21.15.116:3300/0 conn(0x7fe96015e280 msgr2=0x7fe9600cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:37.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.514+0000 7fe95f400640 1 --2- 172.21.15.47:0/3696691631 >> v2:172.21.15.116:3300/0 conn(0x7fe96015e280 0x7fe9600cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.514+0000 7fe95f400640 1 -- 172.21.15.47:0/3696691631 >> v1:172.21.15.116:6789/0 conn(0x7fe9600d2650 legacy=0x7fe9600ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:37.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.514+0000 7fe95f400640 1 -- 172.21.15.47:0/3696691631 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9600cecd0 con 0x7fe9600ce3b0 2024-02-19T15:28:37.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.518+0000 7fe95f400640 1 --2- 172.21.15.47:0/3696691631 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9600ce3b0 0x7fe9600ce790 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fe960067b60 tx=0x7fe95001c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fdb355e41af27ec server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:37.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.518+0000 7fe95ea00640 1 -- 172.21.15.47:0/3696691631 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9500a5680 con 0x7fe9600ce3b0 2024-02-19T15:28:37.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.518+0000 7fe95ea00640 1 -- 172.21.15.47:0/3696691631 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9500a57e0 con 0x7fe9600ce3b0 2024-02-19T15:28:37.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.518+0000 7fe95ea00640 1 -- 172.21.15.47:0/3696691631 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9500a5b70 con 0x7fe9600ce3b0 2024-02-19T15:28:37.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.518+0000 7fe966000640 1 -- 172.21.15.47:0/3696691631 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9600ce3b0 msgr2=0x7fe9600ce790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:37.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.518+0000 7fe966000640 1 --2- 172.21.15.47:0/3696691631 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9600ce3b0 0x7fe9600ce790 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fe960067b60 tx=0x7fe95001c8d0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.518+0000 7fe966000640 1 -- 172.21.15.47:0/3696691631 shutdown_connections 2024-02-19T15:28:37.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.518+0000 7fe966000640 1 --2- 172.21.15.47:0/3696691631 >> v2:172.21.15.116:3300/0 conn(0x7fe96015e280 0x7fe9600cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.518+0000 7fe966000640 1 --2- 172.21.15.47:0/3696691631 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9600ce3b0 0x7fe9600ce790 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.518+0000 7fe966000640 1 -- 172.21.15.47:0/3696691631 >> 172.21.15.47:0/3696691631 conn(0x7fe9600d1b30 msgr2=0x7fe9600d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:37.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.518+0000 7fe966000640 1 -- 172.21.15.47:0/3696691631 shutdown_connections 2024-02-19T15:28:37.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.518+0000 7fe966000640 1 -- 172.21.15.47:0/3696691631 wait complete. 2024-02-19T15:28:37.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.518+0000 7fe966000640 1 Processor -- start 2024-02-19T15:28:37.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.518+0000 7fe966000640 1 -- start start 2024-02-19T15:28:37.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe966000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9600ce3b0 0x7fe9601561a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:37.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe966000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9600d2650 0x7fe96009b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:37.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe966000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe96015e280 0x7fe960159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:37.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe966000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9601f7380 con 0x7fe9600d2650 2024-02-19T15:28:37.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe966000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9601566e0 con 0x7fe96015e280 2024-02-19T15:28:37.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe966000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe96009e730 con 0x7fe9600ce3b0 2024-02-19T15:28:37.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe956a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9600d2650 0x7fe96009b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:37.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe956a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9600d2650 0x7fe96009b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59970/0 (socket says 172.21.15.47:59970) 2024-02-19T15:28:37.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe956a00640 1 -- 172.21.15.47:0/3405435200 learned_addr learned my addr 172.21.15.47:0/3405435200 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:37.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe95f400640 1 --2- 172.21.15.47:0/3405435200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9600ce3b0 0x7fe9601561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:37.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe95fe00640 1 --2- 172.21.15.47:0/3405435200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe96015e280 0x7fe960159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:37.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe956a00640 1 -- 172.21.15.47:0/3405435200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9600ce3b0 msgr2=0x7fe9601561a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:37.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe956a00640 1 --2- 172.21.15.47:0/3405435200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9600ce3b0 0x7fe9601561a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe956a00640 1 -- 172.21.15.47:0/3405435200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe96015e280 msgr2=0x7fe960159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:37.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe956a00640 1 --2- 172.21.15.47:0/3405435200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe96015e280 0x7fe960159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe956a00640 1 -- 172.21.15.47:0/3405435200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe950009b30 con 0x7fe9600d2650 2024-02-19T15:28:37.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe956a00640 1 --2- 172.21.15.47:0/3405435200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9600d2650 0x7fe96009b8e0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fe94800c370 tx=0x7fe94800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:37.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe957e00640 1 -- 172.21.15.47:0/3405435200 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9480052d0 con 0x7fe9600d2650 2024-02-19T15:28:37.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe957e00640 1 -- 172.21.15.47:0/3405435200 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe948005430 con 0x7fe9600d2650 2024-02-19T15:28:37.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe966000640 1 -- 172.21.15.47:0/3405435200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe96009ea10 con 0x7fe9600d2650 2024-02-19T15:28:37.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe957e00640 1 -- 172.21.15.47:0/3405435200 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe948007260 con 0x7fe9600d2650 2024-02-19T15:28:37.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe966000640 1 -- 172.21.15.47:0/3405435200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9601f75c0 con 0x7fe9600d2650 2024-02-19T15:28:37.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.522+0000 7fe966000640 1 -- 172.21.15.47:0/3405435200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe96009ef30 con 0x7fe9600d2650 2024-02-19T15:28:37.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.526+0000 7fe957e00640 1 -- 172.21.15.47:0/3405435200 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe9480073c0 con 0x7fe9600d2650 2024-02-19T15:28:37.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.526+0000 7fe957e00640 1 --2- 172.21.15.47:0/3405435200 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe93c066e40 0x7fe93c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:37.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.526+0000 7fe957e00640 1 -- 172.21.15.47:0/3405435200 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fe948083e50 con 0x7fe9600d2650 2024-02-19T15:28:37.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.526+0000 7fe95f400640 1 --2- 172.21.15.47:0/3405435200 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe93c066e40 0x7fe93c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:37.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.526+0000 7fe95f400640 1 --2- 172.21.15.47:0/3405435200 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe93c066e40 0x7fe93c069300 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fe9500ae000 tx=0x7fe9500a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:37.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.534+0000 7fe957e00640 1 -- 172.21.15.47:0/3405435200 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe96009ef30 con 0x7fe9600d2650 2024-02-19T15:28:37.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.702+0000 7fe966000640 1 -- 172.21.15.47:0/3405435200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe96009ef30 con 0x7fe9600d2650 2024-02-19T15:28:37.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.702+0000 7fe957e00640 1 -- 172.21.15.47:0/3405435200 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe96009ef30 con 0x7fe9600d2650 2024-02-19T15:28:37.703 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:28:37.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.702+0000 7fe966000640 1 -- 172.21.15.47:0/3405435200 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe93c066e40 msgr2=0x7fe93c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:37.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.702+0000 7fe966000640 1 --2- 172.21.15.47:0/3405435200 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe93c066e40 0x7fe93c069300 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fe9500ae000 tx=0x7fe9500a7070 comp rx=0 tx=0).stop 2024-02-19T15:28:37.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.702+0000 7fe966000640 1 -- 172.21.15.47:0/3405435200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9600d2650 msgr2=0x7fe96009b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:37.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.702+0000 7fe966000640 1 --2- 172.21.15.47:0/3405435200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9600d2650 0x7fe96009b8e0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fe94800c370 tx=0x7fe94800c830 comp rx=0 tx=0).stop 2024-02-19T15:28:37.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.702+0000 7fe966000640 1 -- 172.21.15.47:0/3405435200 shutdown_connections 2024-02-19T15:28:37.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.702+0000 7fe966000640 1 --2- 172.21.15.47:0/3405435200 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe93c066e40 0x7fe93c069300 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.702+0000 7fe966000640 1 --2- 172.21.15.47:0/3405435200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe96015e280 0x7fe960159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.702+0000 7fe966000640 1 --2- 172.21.15.47:0/3405435200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9600d2650 0x7fe96009b8e0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.702+0000 7fe966000640 1 --2- 172.21.15.47:0/3405435200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9600ce3b0 0x7fe9601561a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.702+0000 7fe966000640 1 -- 172.21.15.47:0/3405435200 >> 172.21.15.47:0/3405435200 conn(0x7fe9600d1b30 msgr2=0x7fe9600c7a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:37.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.702+0000 7fe966000640 1 -- 172.21.15.47:0/3405435200 shutdown_connections 2024-02-19T15:28:37.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.702+0000 7fe966000640 1 -- 172.21.15.47:0/3405435200 wait complete. 2024-02-19T15:28:37.723 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:28:37.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.866+0000 7f6cb7200640 1 Processor -- start 2024-02-19T15:28:37.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.866+0000 7f6cb7200640 1 -- start start 2024-02-19T15:28:37.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.870+0000 7f6cb7200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6cb815dfb0 0x7f6cb815e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:37.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.870+0000 7f6cb7200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cb80ce8f0 0x7f6cb80cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:37.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.870+0000 7f6cb7200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb80cbfb0 con 0x7f6cb80d2480 2024-02-19T15:28:37.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.870+0000 7f6cb7200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cb80d0980 con 0x7f6cb815dfb0 2024-02-19T15:28:37.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.870+0000 7f6cb7200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb80d0c60 con 0x7f6cb80ce8f0 2024-02-19T15:28:37.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.870+0000 7f6cb5e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6cb815dfb0 0x7f6cb815e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:37.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.870+0000 7f6cb5400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6cb80d2480 0x7f6cb80ce1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43652/0 (socket says 172.21.15.47:43652) 2024-02-19T15:28:37.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.870+0000 7f6cb5400640 1 -- 172.21.15.47:0/273527507 learned_addr learned my addr 172.21.15.47:0/273527507 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:37.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.870+0000 7f6cb6800640 1 --2- 172.21.15.47:0/273527507 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cb80ce8f0 0x7f6cb80cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:37.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.870+0000 7f6cb5e00640 1 -- 172.21.15.47:0/273527507 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cb80ce8f0 msgr2=0x7f6cb80cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:37.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.870+0000 7f6cb5e00640 1 --2- 172.21.15.47:0/273527507 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cb80ce8f0 0x7f6cb80cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.870+0000 7f6cb5e00640 1 -- 172.21.15.47:0/273527507 >> v1:172.21.15.47:6789/0 conn(0x7f6cb80d2480 legacy=0x7f6cb80ce1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:37.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.870+0000 7f6cb5e00640 1 -- 172.21.15.47:0/273527507 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb80d0f40 con 0x7f6cb815dfb0 2024-02-19T15:28:37.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.870+0000 7f6cb4a00640 1 -- 172.21.15.47:0/273527507 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112995137 0 0) 0x7f6cb80cbfb0 con 0x7f6cb80d2480 2024-02-19T15:28:37.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.874+0000 7f6cb5e00640 1 --2- 172.21.15.47:0/273527507 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cb815dfb0 0x7f6cb815e390 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f6ca80097c0 tx=0x7f6ca801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3948e03659cd7150 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:37.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.874+0000 7f6cb4a00640 1 -- 172.21.15.47:0/273527507 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca80aa620 con 0x7f6cb815dfb0 2024-02-19T15:28:37.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.874+0000 7f6cb4a00640 1 -- 172.21.15.47:0/273527507 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca80aa780 con 0x7f6cb815dfb0 2024-02-19T15:28:37.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.874+0000 7f6cb4a00640 1 -- 172.21.15.47:0/273527507 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca80aab10 con 0x7f6cb815dfb0 2024-02-19T15:28:37.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.874+0000 7f6cb7200640 1 -- 172.21.15.47:0/273527507 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cb815dfb0 msgr2=0x7f6cb815e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:37.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.874+0000 7f6cb7200640 1 --2- 172.21.15.47:0/273527507 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cb815dfb0 0x7f6cb815e390 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f6ca80097c0 tx=0x7f6ca801c4a0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.874+0000 7f6cb7200640 1 -- 172.21.15.47:0/273527507 shutdown_connections 2024-02-19T15:28:37.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.874+0000 7f6cb7200640 1 --2- 172.21.15.47:0/273527507 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cb80ce8f0 0x7f6cb80cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.874+0000 7f6cb7200640 1 --2- 172.21.15.47:0/273527507 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cb815dfb0 0x7f6cb815e390 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.874+0000 7f6cb7200640 1 -- 172.21.15.47:0/273527507 >> 172.21.15.47:0/273527507 conn(0x7f6cb80d1960 msgr2=0x7f6cb80d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:37.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.874+0000 7f6cb7200640 1 -- 172.21.15.47:0/273527507 shutdown_connections 2024-02-19T15:28:37.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.874+0000 7f6cb7200640 1 -- 172.21.15.47:0/273527507 wait complete. 2024-02-19T15:28:37.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb7200640 1 Processor -- start 2024-02-19T15:28:37.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb7200640 1 -- start start 2024-02-19T15:28:37.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb7200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cb80ce8f0 0x7f6cb815a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:37.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb7200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cb815dfb0 0x7f6cb815eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:37.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb7200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cb81553a0 0x7f6cb815fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:37.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb7200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb81546b0 con 0x7f6cb815dfb0 2024-02-19T15:28:37.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb7200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb8155f80 con 0x7f6cb80ce8f0 2024-02-19T15:28:37.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb7200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb815f410 con 0x7f6cb81553a0 2024-02-19T15:28:37.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb5400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cb815dfb0 0x7f6cb815eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:37.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb5400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cb815dfb0 0x7f6cb815eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59980/0 (socket says 172.21.15.47:59980) 2024-02-19T15:28:37.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb5400640 1 -- 172.21.15.47:0/548663409 learned_addr learned my addr 172.21.15.47:0/548663409 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:37.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb6800640 1 --2- 172.21.15.47:0/548663409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cb81553a0 0x7f6cb815fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:37.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb5e00640 1 --2- 172.21.15.47:0/548663409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cb80ce8f0 0x7f6cb815a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:37.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb5400640 1 -- 172.21.15.47:0/548663409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cb81553a0 msgr2=0x7f6cb815fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:37.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb5400640 1 --2- 172.21.15.47:0/548663409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cb81553a0 0x7f6cb815fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb5400640 1 -- 172.21.15.47:0/548663409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cb80ce8f0 msgr2=0x7f6cb815a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:37.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb5400640 1 --2- 172.21.15.47:0/548663409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cb80ce8f0 0x7f6cb815a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:37.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb5400640 1 -- 172.21.15.47:0/548663409 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca8009420 con 0x7f6cb815dfb0 2024-02-19T15:28:37.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb5400640 1 --2- 172.21.15.47:0/548663409 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cb815dfb0 0x7f6cb815eda0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f6ca40011f0 tx=0x7f6ca400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:37.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cae000640 1 -- 172.21.15.47:0/548663409 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca40074e0 con 0x7f6cb815dfb0 2024-02-19T15:28:37.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cae000640 1 -- 172.21.15.47:0/548663409 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca4007640 con 0x7f6cb815dfb0 2024-02-19T15:28:37.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb7200640 1 -- 172.21.15.47:0/548663409 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb81548c0 con 0x7f6cb815dfb0 2024-02-19T15:28:37.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cae000640 1 -- 172.21.15.47:0/548663409 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4004a10 con 0x7f6cb815dfb0 2024-02-19T15:28:37.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.878+0000 7f6cb7200640 1 -- 172.21.15.47:0/548663409 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb8154e80 con 0x7f6cb815dfb0 2024-02-19T15:28:37.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.882+0000 7f6cb7200640 1 -- 172.21.15.47:0/548663409 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb81550a0 con 0x7f6cb815dfb0 2024-02-19T15:28:37.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.882+0000 7f6cae000640 1 -- 172.21.15.47:0/548663409 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6ca4004b70 con 0x7f6cb815dfb0 2024-02-19T15:28:37.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.882+0000 7f6cae000640 1 --2- 172.21.15.47:0/548663409 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6c98066e40 0x7f6c98069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:37.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.882+0000 7f6cae000640 1 -- 172.21.15.47:0/548663409 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f6ca4083f20 con 0x7f6cb815dfb0 2024-02-19T15:28:37.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.882+0000 7f6cb5e00640 1 --2- 172.21.15.47:0/548663409 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6c98066e40 0x7f6c98069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:37.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.886+0000 7f6cb5e00640 1 --2- 172.21.15.47:0/548663409 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6c98066e40 0x7f6c98069300 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f6ca80ae000 tx=0x7f6ca80a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:37.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:37.886+0000 7f6cae000640 1 -- 172.21.15.47:0/548663409 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6ca4050d60 con 0x7f6cb815dfb0 2024-02-19T15:28:38.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.050+0000 7f6cb7200640 1 -- 172.21.15.47:0/548663409 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6cb804f260 con 0x7f6cb815dfb0 2024-02-19T15:28:38.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.050+0000 7f6cae000640 1 -- 172.21.15.47:0/548663409 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6ca40180a0 con 0x7f6cb815dfb0 2024-02-19T15:28:38.050 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:28:38.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.050+0000 7f6cb7200640 1 -- 172.21.15.47:0/548663409 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6c98066e40 msgr2=0x7f6c98069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:38.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.050+0000 7f6cb7200640 1 --2- 172.21.15.47:0/548663409 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6c98066e40 0x7f6c98069300 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f6ca80ae000 tx=0x7f6ca80a7000 comp rx=0 tx=0).stop 2024-02-19T15:28:38.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.050+0000 7f6cb7200640 1 -- 172.21.15.47:0/548663409 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cb815dfb0 msgr2=0x7f6cb815eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:38.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.050+0000 7f6cb7200640 1 --2- 172.21.15.47:0/548663409 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cb815dfb0 0x7f6cb815eda0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f6ca40011f0 tx=0x7f6ca400c9b0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.050+0000 7f6cb7200640 1 -- 172.21.15.47:0/548663409 shutdown_connections 2024-02-19T15:28:38.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.050+0000 7f6cb7200640 1 --2- 172.21.15.47:0/548663409 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6c98066e40 0x7f6c98069300 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.050+0000 7f6cb7200640 1 --2- 172.21.15.47:0/548663409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cb81553a0 0x7f6cb815fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.050+0000 7f6cb7200640 1 --2- 172.21.15.47:0/548663409 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cb815dfb0 0x7f6cb815eda0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.050+0000 7f6cb7200640 1 --2- 172.21.15.47:0/548663409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cb80ce8f0 0x7f6cb815a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.050+0000 7f6cb7200640 1 -- 172.21.15.47:0/548663409 >> 172.21.15.47:0/548663409 conn(0x7f6cb80d1960 msgr2=0x7f6cb806be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:38.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.050+0000 7f6cb7200640 1 -- 172.21.15.47:0/548663409 shutdown_connections 2024-02-19T15:28:38.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.050+0000 7f6cb7200640 1 -- 172.21.15.47:0/548663409 wait complete. 2024-02-19T15:28:38.071 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:28:38.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.214+0000 7fa2fce00640 1 Processor -- start 2024-02-19T15:28:38.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.214+0000 7fa2fce00640 1 -- start start 2024-02-19T15:28:38.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.214+0000 7fa2fce00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa2f80ce3b0 0x7fa2f80ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:38.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.214+0000 7fa2fce00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa2f80d2650 0x7fa2f80ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:38.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.214+0000 7fa2fce00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2f815e280 0x7fa2f80ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:38.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.214+0000 7fa2fce00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f80cc620 con 0x7fa2f80ce3b0 2024-02-19T15:28:38.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.214+0000 7fa2fce00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f80d12e0 con 0x7fa2f80d2650 2024-02-19T15:28:38.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.214+0000 7fa2fce00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f80d1940 con 0x7fa2f815e280 2024-02-19T15:28:38.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.214+0000 7fa2f5e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa2f80ce3b0 0x7fa2f80ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:38.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.214+0000 7fa2f5e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa2f80ce3b0 0x7fa2f80ce790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:59990/0 (socket says 172.21.15.47:59990) 2024-02-19T15:28:38.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.214+0000 7fa2f5e00640 1 -- 172.21.15.47:0/3692414946 learned_addr learned my addr 172.21.15.47:0/3692414946 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:38.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.214+0000 7fa2efe00640 1 --2- 172.21.15.47:0/3692414946 >> v2:172.21.15.116:3300/0 conn(0x7fa2f80d2650 0x7fa2f80ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:38.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.214+0000 7fa2f6800640 1 --2- 172.21.15.47:0/3692414946 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2f815e280 0x7fa2f80ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:38.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.214+0000 7fa2f5e00640 1 -- 172.21.15.47:0/3692414946 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2f815e280 msgr2=0x7fa2f80ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:38.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.214+0000 7fa2f5e00640 1 --2- 172.21.15.47:0/3692414946 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2f815e280 0x7fa2f80ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.214+0000 7fa2f5e00640 1 -- 172.21.15.47:0/3692414946 >> v2:172.21.15.116:3300/0 conn(0x7fa2f80d2650 msgr2=0x7fa2f80ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:38.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.214+0000 7fa2f5e00640 1 --2- 172.21.15.47:0/3692414946 >> v2:172.21.15.116:3300/0 conn(0x7fa2f80d2650 0x7fa2f80ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.214+0000 7fa2f5e00640 1 -- 172.21.15.47:0/3692414946 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f80cecd0 con 0x7fa2f80ce3b0 2024-02-19T15:28:38.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.218+0000 7fa2f5e00640 1 --2- 172.21.15.47:0/3692414946 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2f80ce3b0 0x7fa2f80ce790 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8071dd0 tx=0x7fa2e801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40e03e1380d5029 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:38.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.218+0000 7fa2f5400640 1 -- 172.21.15.47:0/3692414946 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e80ab600 con 0x7fa2f80ce3b0 2024-02-19T15:28:38.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.218+0000 7fa2f5400640 1 -- 172.21.15.47:0/3692414946 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e80a7050 con 0x7fa2f80ce3b0 2024-02-19T15:28:38.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.218+0000 7fa2f5400640 1 -- 172.21.15.47:0/3692414946 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e80ab980 con 0x7fa2f80ce3b0 2024-02-19T15:28:38.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.218+0000 7fa2fce00640 1 -- 172.21.15.47:0/3692414946 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2f80ce3b0 msgr2=0x7fa2f80ce790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:38.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.218+0000 7fa2fce00640 1 --2- 172.21.15.47:0/3692414946 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2f80ce3b0 0x7fa2f80ce790 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8071dd0 tx=0x7fa2e801c400 comp rx=0 tx=0).stop 2024-02-19T15:28:38.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.218+0000 7fa2fce00640 1 -- 172.21.15.47:0/3692414946 shutdown_connections 2024-02-19T15:28:38.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.218+0000 7fa2fce00640 1 --2- 172.21.15.47:0/3692414946 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2f815e280 0x7fa2f80ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.218+0000 7fa2fce00640 1 --2- 172.21.15.47:0/3692414946 >> v2:172.21.15.116:3300/0 conn(0x7fa2f80d2650 0x7fa2f80ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.218+0000 7fa2fce00640 1 --2- 172.21.15.47:0/3692414946 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2f80ce3b0 0x7fa2f80ce790 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.218+0000 7fa2fce00640 1 -- 172.21.15.47:0/3692414946 >> 172.21.15.47:0/3692414946 conn(0x7fa2f80d1b30 msgr2=0x7fa2f80d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:38.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.218+0000 7fa2fce00640 1 -- 172.21.15.47:0/3692414946 shutdown_connections 2024-02-19T15:28:38.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.218+0000 7fa2fce00640 1 -- 172.21.15.47:0/3692414946 wait complete. 2024-02-19T15:28:38.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2fce00640 1 Processor -- start 2024-02-19T15:28:38.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2fce00640 1 -- start start 2024-02-19T15:28:38.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2fce00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2f80ce3b0 0x7fa2f8154dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:38.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2fce00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f80d2650 0x7fa2f80cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:38.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2fce00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2f815e280 0x7fa2f8155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:38.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2fce00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f80cabb0 con 0x7fa2f815e280 2024-02-19T15:28:38.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2fce00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f8158ac0 con 0x7fa2f80d2650 2024-02-19T15:28:38.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2fce00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f8158570 con 0x7fa2f80ce3b0 2024-02-19T15:28:38.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2f6800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2f815e280 0x7fa2f8155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:38.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2f6800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2f815e280 0x7fa2f8155670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60006/0 (socket says 172.21.15.47:60006) 2024-02-19T15:28:38.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2f6800640 1 -- 172.21.15.47:0/1895846564 learned_addr learned my addr 172.21.15.47:0/1895846564 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:38.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2f5e00640 1 --2- 172.21.15.47:0/1895846564 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2f80ce3b0 0x7fa2f8154dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:38.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2efe00640 1 --2- 172.21.15.47:0/1895846564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f80d2650 0x7fa2f80cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:38.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2f6800640 1 -- 172.21.15.47:0/1895846564 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2f80ce3b0 msgr2=0x7fa2f8154dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:38.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2f6800640 1 --2- 172.21.15.47:0/1895846564 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2f80ce3b0 0x7fa2f8154dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2f6800640 1 -- 172.21.15.47:0/1895846564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f80d2650 msgr2=0x7fa2f80cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:38.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2f6800640 1 --2- 172.21.15.47:0/1895846564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f80d2650 0x7fa2f80cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2f6800640 1 -- 172.21.15.47:0/1895846564 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e80094d0 con 0x7fa2f815e280 2024-02-19T15:28:38.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2f6800640 1 --2- 172.21.15.47:0/1895846564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2f815e280 0x7fa2f8155670 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0007e90 tx=0x7fa2f000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:38.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2ee000640 1 -- 172.21.15.47:0/1895846564 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0014420 con 0x7fa2f815e280 2024-02-19T15:28:38.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2ee000640 1 -- 172.21.15.47:0/1895846564 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f0014580 con 0x7fa2f815e280 2024-02-19T15:28:38.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2ee000640 1 -- 172.21.15.47:0/1895846564 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0014930 con 0x7fa2f815e280 2024-02-19T15:28:38.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2fce00640 1 -- 172.21.15.47:0/1895846564 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f81587e0 con 0x7fa2f815e280 2024-02-19T15:28:38.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.222+0000 7fa2fce00640 1 -- 172.21.15.47:0/1895846564 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f81606d0 con 0x7fa2f815e280 2024-02-19T15:28:38.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.226+0000 7fa2fce00640 1 -- 172.21.15.47:0/1895846564 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f8075360 con 0x7fa2f815e280 2024-02-19T15:28:38.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.226+0000 7fa2ee000640 1 -- 172.21.15.47:0/1895846564 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa2f0014a90 con 0x7fa2f815e280 2024-02-19T15:28:38.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.226+0000 7fa2ee000640 1 --2- 172.21.15.47:0/1895846564 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa2d4066e40 0x7fa2d4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:38.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.226+0000 7fa2ee000640 1 -- 172.21.15.47:0/1895846564 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fa2f0085300 con 0x7fa2f815e280 2024-02-19T15:28:38.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.226+0000 7fa2f5e00640 1 --2- 172.21.15.47:0/1895846564 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa2d4066e40 0x7fa2d4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:38.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.230+0000 7fa2f5e00640 1 --2- 172.21.15.47:0/1895846564 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa2d4066e40 0x7fa2d4069300 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fa2e80ad000 tx=0x7fa2e8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:38.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.230+0000 7fa2ee000640 1 -- 172.21.15.47:0/1895846564 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa2f00520a0 con 0x7fa2f815e280 2024-02-19T15:28:38.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.394+0000 7fa2fce00640 1 -- 172.21.15.47:0/1895846564 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa2f80cbe60 con 0x7fa2f815e280 2024-02-19T15:28:38.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.394+0000 7fa2ee000640 1 -- 172.21.15.47:0/1895846564 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa2f0055760 con 0x7fa2f815e280 2024-02-19T15:28:38.395 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:28:38.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.398+0000 7fa2fce00640 1 -- 172.21.15.47:0/1895846564 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa2d4066e40 msgr2=0x7fa2d4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:38.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.398+0000 7fa2fce00640 1 --2- 172.21.15.47:0/1895846564 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa2d4066e40 0x7fa2d4069300 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fa2e80ad000 tx=0x7fa2e8008040 comp rx=0 tx=0).stop 2024-02-19T15:28:38.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.398+0000 7fa2fce00640 1 -- 172.21.15.47:0/1895846564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2f815e280 msgr2=0x7fa2f8155670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:38.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.398+0000 7fa2fce00640 1 --2- 172.21.15.47:0/1895846564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2f815e280 0x7fa2f8155670 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0007e90 tx=0x7fa2f000e4b0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.398+0000 7fa2fce00640 1 -- 172.21.15.47:0/1895846564 shutdown_connections 2024-02-19T15:28:38.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.398+0000 7fa2fce00640 1 --2- 172.21.15.47:0/1895846564 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa2d4066e40 0x7fa2d4069300 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.398+0000 7fa2fce00640 1 --2- 172.21.15.47:0/1895846564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2f815e280 0x7fa2f8155670 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.398+0000 7fa2fce00640 1 --2- 172.21.15.47:0/1895846564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f80d2650 0x7fa2f80cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.398+0000 7fa2fce00640 1 --2- 172.21.15.47:0/1895846564 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2f80ce3b0 0x7fa2f8154dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.398+0000 7fa2fce00640 1 -- 172.21.15.47:0/1895846564 >> 172.21.15.47:0/1895846564 conn(0x7fa2f80d1b30 msgr2=0x7fa2f80d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:38.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.398+0000 7fa2fce00640 1 -- 172.21.15.47:0/1895846564 shutdown_connections 2024-02-19T15:28:38.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.398+0000 7fa2fce00640 1 -- 172.21.15.47:0/1895846564 wait complete. 2024-02-19T15:28:38.418 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-19T15:28:38.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.558+0000 7f0c3aa00640 1 Processor -- start 2024-02-19T15:28:38.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.562+0000 7f0c3aa00640 1 -- start start 2024-02-19T15:28:38.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.562+0000 7f0c3aa00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0c340cded0 0x7f0c340ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:38.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.562+0000 7f0c3aa00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c340ceb40 0x7f0c3406bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:38.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.562+0000 7f0c3aa00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c340cce30 con 0x7f0c3406d650 2024-02-19T15:28:38.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.562+0000 7f0c3aa00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c340cefd0 con 0x7f0c340cded0 2024-02-19T15:28:38.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.562+0000 7f0c3aa00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c340cd110 con 0x7f0c340ceb40 2024-02-19T15:28:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.562+0000 7f0c38a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c340ceb40 0x7f0c3406bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.562+0000 7f0c33400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0c340cded0 0x7f0c340ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.562+0000 7f0c33e00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0c3406d650 0x7f0c340d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:40938/0 (socket says 172.21.15.47:40938) 2024-02-19T15:28:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.562+0000 7f0c33400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0c340cded0 0x7f0c340ce600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:45814/0 (socket says 172.21.15.47:45814) 2024-02-19T15:28:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.562+0000 7f0c33e00640 1 -- 172.21.15.47:0/2054278453 learned_addr learned my addr 172.21.15.47:0/2054278453 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.562+0000 7f0c38a00640 1 -- 172.21.15.47:0/2054278453 >> v2:172.21.15.116:3300/0 conn(0x7f0c340cded0 msgr2=0x7f0c340ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.562+0000 7f0c38a00640 1 --2- 172.21.15.47:0/2054278453 >> v2:172.21.15.116:3300/0 conn(0x7f0c340cded0 0x7f0c340ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.562+0000 7f0c38a00640 1 -- 172.21.15.47:0/2054278453 >> v1:172.21.15.116:6789/0 conn(0x7f0c3406d650 legacy=0x7f0c340d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:38.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.562+0000 7f0c38a00640 1 -- 172.21.15.47:0/2054278453 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c340cd3f0 con 0x7f0c340ceb40 2024-02-19T15:28:38.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.562+0000 7f0c33400640 1 --2- 172.21.15.47:0/2054278453 >> v2:172.21.15.116:3300/0 conn(0x7f0c340cded0 0x7f0c340ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:28:38.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.566+0000 7f0c38a00640 1 --2- 172.21.15.47:0/2054278453 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c340ceb40 0x7f0c3406bd60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c009d90 tx=0x7f0c2c01c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d577e72b0ff73ee2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:38.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.566+0000 7f0c32a00640 1 -- 172.21.15.47:0/2054278453 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c0a5970 con 0x7f0c340ceb40 2024-02-19T15:28:38.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.566+0000 7f0c32a00640 1 -- 172.21.15.47:0/2054278453 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2c0a5ad0 con 0x7f0c340ceb40 2024-02-19T15:28:38.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.566+0000 7f0c32a00640 1 -- 172.21.15.47:0/2054278453 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c0a5e60 con 0x7f0c340ceb40 2024-02-19T15:28:38.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.566+0000 7f0c3aa00640 1 -- 172.21.15.47:0/2054278453 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c340ceb40 msgr2=0x7f0c3406bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:38.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.566+0000 7f0c3aa00640 1 --2- 172.21.15.47:0/2054278453 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c340ceb40 0x7f0c3406bd60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c009d90 tx=0x7f0c2c01c8b0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.566+0000 7f0c3aa00640 1 -- 172.21.15.47:0/2054278453 shutdown_connections 2024-02-19T15:28:38.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.566+0000 7f0c3aa00640 1 --2- 172.21.15.47:0/2054278453 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c340ceb40 0x7f0c3406bd60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.566+0000 7f0c3aa00640 1 --2- 172.21.15.47:0/2054278453 >> v2:172.21.15.116:3300/0 conn(0x7f0c340cded0 0x7f0c340ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.566+0000 7f0c3aa00640 1 -- 172.21.15.47:0/2054278453 >> 172.21.15.47:0/2054278453 conn(0x7f0c340d1bf0 msgr2=0x7f0c340d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:38.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.566+0000 7f0c3aa00640 1 -- 172.21.15.47:0/2054278453 shutdown_connections 2024-02-19T15:28:38.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.566+0000 7f0c3aa00640 1 -- 172.21.15.47:0/2054278453 wait complete. 2024-02-19T15:28:38.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c3aa00640 1 Processor -- start 2024-02-19T15:28:38.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c3aa00640 1 -- start start 2024-02-19T15:28:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c3aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c3406d650 0x7f0c340c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c3aa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c340cded0 0x7f0c340b22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c3aa00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c340ceb40 0x7f0c3415a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c3aa00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c340c6130 con 0x7f0c340cded0 2024-02-19T15:28:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c3aa00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c340c9c80 con 0x7f0c3406d650 2024-02-19T15:28:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c3aa00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c340c96c0 con 0x7f0c340ceb40 2024-02-19T15:28:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c33400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c340cded0 0x7f0c340b22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:38.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c33400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c340cded0 0x7f0c340b22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60022/0 (socket says 172.21.15.47:60022) 2024-02-19T15:28:38.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c33400640 1 -- 172.21.15.47:0/2293817847 learned_addr learned my addr 172.21.15.47:0/2293817847 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:38.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c33e00640 1 --2- 172.21.15.47:0/2293817847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c3406d650 0x7f0c340c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:38.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c33400640 1 -- 172.21.15.47:0/2293817847 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c340ceb40 msgr2=0x7f0c3415a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:38.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c33400640 1 --2- 172.21.15.47:0/2293817847 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c340ceb40 0x7f0c3415a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c33400640 1 -- 172.21.15.47:0/2293817847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c3406d650 msgr2=0x7f0c340c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:38.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c33400640 1 --2- 172.21.15.47:0/2293817847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c3406d650 0x7f0c340c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c33400640 1 -- 172.21.15.47:0/2293817847 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c20009420 con 0x7f0c340cded0 2024-02-19T15:28:38.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c33e00640 1 --2- 172.21.15.47:0/2293817847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c3406d650 0x7f0c340c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:38.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c33400640 1 --2- 172.21.15.47:0/2293817847 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c340cded0 0x7f0c340b22b0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000e980 tx=0x7f0c2000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:38.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c2be00640 1 -- 172.21.15.47:0/2293817847 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c200096e0 con 0x7f0c340cded0 2024-02-19T15:28:38.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c2be00640 1 -- 172.21.15.47:0/2293817847 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2000c9a0 con 0x7f0c340cded0 2024-02-19T15:28:38.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c2be00640 1 -- 172.21.15.47:0/2293817847 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2000cd30 con 0x7f0c340cded0 2024-02-19T15:28:38.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c3aa00640 1 -- 172.21.15.47:0/2293817847 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c2c009a70 con 0x7f0c340cded0 2024-02-19T15:28:38.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c3aa00640 1 -- 172.21.15.47:0/2293817847 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c340c6310 con 0x7f0c340cded0 2024-02-19T15:28:38.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.570+0000 7f0c3aa00640 1 -- 172.21.15.47:0/2293817847 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c34075360 con 0x7f0c340cded0 2024-02-19T15:28:38.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.574+0000 7f0c2be00640 1 -- 172.21.15.47:0/2293817847 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0c20013470 con 0x7f0c340cded0 2024-02-19T15:28:38.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.574+0000 7f0c2be00640 1 --2- 172.21.15.47:0/2293817847 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0c14066e40 0x7f0c14069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:38.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.574+0000 7f0c2be00640 1 -- 172.21.15.47:0/2293817847 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f0c20084e00 con 0x7f0c340cded0 2024-02-19T15:28:38.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.574+0000 7f0c33e00640 1 --2- 172.21.15.47:0/2293817847 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0c14066e40 0x7f0c14069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:38.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.574+0000 7f0c33e00640 1 --2- 172.21.15.47:0/2293817847 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0c14066e40 0x7f0c14069300 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f0c24005890 tx=0x7f0c24009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:38.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.578+0000 7f0c2be00640 1 -- 172.21.15.47:0/2293817847 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0c20051af0 con 0x7f0c340cded0 2024-02-19T15:28:38.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.754+0000 7f0c3aa00640 1 -- 172.21.15.47:0/2293817847 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0c340708f0 con 0x7f0c340cded0 2024-02-19T15:28:38.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.754+0000 7f0c2be00640 1 -- 172.21.15.47:0/2293817847 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v114) v1 ==== 80+0+242 (secure 0 0 0) 0x7f0c20021110 con 0x7f0c340cded0 2024-02-19T15:28:38.754 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.116:6837/819751716 2024-02-20T15:27:39.101989+0000 2024-02-19T15:28:38.754 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.116:6835/819751716 2024-02-20T15:27:39.101989+0000 2024-02-19T15:28:38.754 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:6834/897369244 2024-02-20T15:27:38.263490+0000 2024-02-19T15:28:38.754 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47:6832/897369244 2024-02-20T15:27:38.263490+0000 2024-02-19T15:28:38.755 INFO:teuthology.orchestra.run.smithi047.stderr:listed 4 entries 2024-02-19T15:28:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.754+0000 7f0c3aa00640 1 -- 172.21.15.47:0/2293817847 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0c14066e40 msgr2=0x7f0c14069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.754+0000 7f0c3aa00640 1 --2- 172.21.15.47:0/2293817847 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0c14066e40 0x7f0c14069300 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f0c24005890 tx=0x7f0c24009040 comp rx=0 tx=0).stop 2024-02-19T15:28:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.754+0000 7f0c3aa00640 1 -- 172.21.15.47:0/2293817847 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c340cded0 msgr2=0x7f0c340b22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.754+0000 7f0c3aa00640 1 --2- 172.21.15.47:0/2293817847 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c340cded0 0x7f0c340b22b0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000e980 tx=0x7f0c2000ee40 comp rx=0 tx=0).stop 2024-02-19T15:28:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.754+0000 7f0c3aa00640 1 -- 172.21.15.47:0/2293817847 shutdown_connections 2024-02-19T15:28:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.754+0000 7f0c3aa00640 1 --2- 172.21.15.47:0/2293817847 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0c14066e40 0x7f0c14069300 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.754+0000 7f0c3aa00640 1 --2- 172.21.15.47:0/2293817847 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c340ceb40 0x7f0c3415a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.754+0000 7f0c3aa00640 1 --2- 172.21.15.47:0/2293817847 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c340cded0 0x7f0c340b22b0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.754+0000 7f0c3aa00640 1 --2- 172.21.15.47:0/2293817847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c3406d650 0x7f0c340c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.754+0000 7f0c3aa00640 1 -- 172.21.15.47:0/2293817847 >> 172.21.15.47:0/2293817847 conn(0x7f0c340d1bf0 msgr2=0x7f0c340d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.754+0000 7f0c3aa00640 1 -- 172.21.15.47:0/2293817847 shutdown_connections 2024-02-19T15:28:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:38.754+0000 7f0c3aa00640 1 -- 172.21.15.47:0/2293817847 wait complete. 2024-02-19T15:28:38.776 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-02-19T15:28:38.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:28:38.777 DEBUG:teuthology.orchestra.run.smithi047:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-02-19T15:28:38.801 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi047.front.sepia.ceph.com 2024-02-19T15:28:38.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:28:38.802 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-19T15:28:38.854 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:28:38.854 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:28:38.864 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:28:38.864 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:28:38.864 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:28:38.905 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:28:38.905 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:28:38.905 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:28:38.914 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:28:38.915 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:39.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.086+0000 7f415d400640 1 Processor -- start 2024-02-19T15:28:39.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.086+0000 7f415d400640 1 -- start start 2024-02-19T15:28:39.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.090+0000 7f415d400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f41580d2150 0x7f41580ce330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:39.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.090+0000 7f415d400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41580b1b10 con 0x7f415815ddd0 2024-02-19T15:28:39.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.090+0000 7f415d400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41580cea30 con 0x7f415815e1b0 2024-02-19T15:28:39.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.090+0000 7f415d400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41580ced10 con 0x7f41580d2150 2024-02-19T15:28:39.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.090+0000 7f414f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f41580d2150 0x7f41580ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:39.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.090+0000 7f414f400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f41580d2150 0x7f41580ce330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60038/0 (socket says 172.21.15.47:60038) 2024-02-19T15:28:39.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.090+0000 7f414f400640 1 -- 172.21.15.47:0/3024347140 learned_addr learned my addr 172.21.15.47:0/3024347140 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:39.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.090+0000 7f4157e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f415815ddd0 0x7f41580d1a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43658/0 (socket says 172.21.15.47:43658) 2024-02-19T15:28:39.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.090+0000 7f414f400640 1 -- 172.21.15.47:0/3024347140 >> v1:172.21.15.116:6789/0 conn(0x7f415815e1b0 legacy=0x7f41580b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:39.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.090+0000 7f414f400640 1 -- 172.21.15.47:0/3024347140 >> v1:172.21.15.47:6789/0 conn(0x7f415815ddd0 legacy=0x7f41580d1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:39.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.090+0000 7f414f400640 1 -- 172.21.15.47:0/3024347140 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41580ceff0 con 0x7f41580d2150 2024-02-19T15:28:39.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.094+0000 7f414f400640 1 --2- 172.21.15.47:0/3024347140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41580d2150 0x7f41580ce330 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f4140009870 tx=0x7f414001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6162748f081c8a79 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:39.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.094+0000 7f4157400640 1 -- 172.21.15.47:0/3024347140 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41400a9400 con 0x7f41580d2150 2024-02-19T15:28:39.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.094+0000 7f4157400640 1 -- 172.21.15.47:0/3024347140 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41400a9560 con 0x7f41580d2150 2024-02-19T15:28:39.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.094+0000 7f4157400640 1 -- 172.21.15.47:0/3024347140 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41400a98f0 con 0x7f41580d2150 2024-02-19T15:28:39.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.094+0000 7f415d400640 1 -- 172.21.15.47:0/3024347140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41580d2150 msgr2=0x7f41580ce330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:39.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.094+0000 7f415d400640 1 --2- 172.21.15.47:0/3024347140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41580d2150 0x7f41580ce330 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f4140009870 tx=0x7f414001c520 comp rx=0 tx=0).stop 2024-02-19T15:28:39.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.094+0000 7f415d400640 1 -- 172.21.15.47:0/3024347140 shutdown_connections 2024-02-19T15:28:39.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.094+0000 7f415d400640 1 --2- 172.21.15.47:0/3024347140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41580d2150 0x7f41580ce330 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.094+0000 7f415d400640 1 -- 172.21.15.47:0/3024347140 >> 172.21.15.47:0/3024347140 conn(0x7f415809a9d0 msgr2=0x7f415809ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:39.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.094+0000 7f415d400640 1 -- 172.21.15.47:0/3024347140 shutdown_connections 2024-02-19T15:28:39.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.094+0000 7f415d400640 1 -- 172.21.15.47:0/3024347140 wait complete. 2024-02-19T15:28:39.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.094+0000 7f415d400640 1 Processor -- start 2024-02-19T15:28:39.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.094+0000 7f415d400640 1 -- start start 2024-02-19T15:28:39.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.094+0000 7f415d400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41580d2150 0x7f41580c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:39.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.094+0000 7f415d400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f415815ddd0 0x7f415814f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:39.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.094+0000 7f415d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f415815e1b0 0x7f41580b4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:39.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.094+0000 7f415d400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41580c7020 con 0x7f41580d2150 2024-02-19T15:28:39.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.094+0000 7f415d400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415814f8f0 con 0x7f415815e1b0 2024-02-19T15:28:39.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.094+0000 7f415d400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41580b51c0 con 0x7f415815ddd0 2024-02-19T15:28:39.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.098+0000 7f4157e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41580d2150 0x7f41580c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:39.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.098+0000 7f4157e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41580d2150 0x7f41580c62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60050/0 (socket says 172.21.15.47:60050) 2024-02-19T15:28:39.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.098+0000 7f4157e00640 1 -- 172.21.15.47:0/224379159 learned_addr learned my addr 172.21.15.47:0/224379159 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:39.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.098+0000 7f414f400640 1 --2- 172.21.15.47:0/224379159 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f415815ddd0 0x7f415814f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.098+0000 7f415ca00640 1 --2- 172.21.15.47:0/224379159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f415815e1b0 0x7f41580b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.098+0000 7f4157e00640 1 -- 172.21.15.47:0/224379159 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f415815ddd0 msgr2=0x7f415814f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.098+0000 7f4157e00640 1 --2- 172.21.15.47:0/224379159 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f415815ddd0 0x7f415814f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.098+0000 7f4157e00640 1 -- 172.21.15.47:0/224379159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f415815e1b0 msgr2=0x7f41580b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.098+0000 7f4157e00640 1 --2- 172.21.15.47:0/224379159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f415815e1b0 0x7f41580b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.098+0000 7f4157e00640 1 -- 172.21.15.47:0/224379159 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41400094d0 con 0x7f41580d2150 2024-02-19T15:28:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.098+0000 7f414f400640 1 --2- 172.21.15.47:0/224379159 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f415815ddd0 0x7f415814f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.098+0000 7f415ca00640 1 --2- 172.21.15.47:0/224379159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f415815e1b0 0x7f41580b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.098+0000 7f4157e00640 1 --2- 172.21.15.47:0/224379159 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41580d2150 0x7f41580c62e0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f414800c420 tx=0x7f414800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.098+0000 7f4154c00640 1 -- 172.21.15.47:0/224379159 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41480074e0 con 0x7f41580d2150 2024-02-19T15:28:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.098+0000 7f4154c00640 1 -- 172.21.15.47:0/224379159 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4148007640 con 0x7f41580d2150 2024-02-19T15:28:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.098+0000 7f4154c00640 1 -- 172.21.15.47:0/224379159 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4148004a30 con 0x7f41580d2150 2024-02-19T15:28:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.098+0000 7f415d400640 1 -- 172.21.15.47:0/224379159 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41580b54a0 con 0x7f41580d2150 2024-02-19T15:28:39.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.098+0000 7f415d400640 1 -- 172.21.15.47:0/224379159 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41580c6cb0 con 0x7f41580d2150 2024-02-19T15:28:39.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.098+0000 7f415d400640 1 -- 172.21.15.47:0/224379159 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41580af350 con 0x7f41580d2150 2024-02-19T15:28:39.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.102+0000 7f4154c00640 1 -- 172.21.15.47:0/224379159 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4148004b90 con 0x7f41580d2150 2024-02-19T15:28:39.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.102+0000 7f4154c00640 1 --2- 172.21.15.47:0/224379159 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4134066e40 0x7f4134069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:39.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.102+0000 7f4154c00640 1 -- 172.21.15.47:0/224379159 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f41480845b0 con 0x7f41580d2150 2024-02-19T15:28:39.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.102+0000 7f414f400640 1 --2- 172.21.15.47:0/224379159 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4134066e40 0x7f4134069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:39.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.102+0000 7f414f400640 1 --2- 172.21.15.47:0/224379159 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4134066e40 0x7f4134069300 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f41400ae000 tx=0x7f41400b3040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:39.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.106+0000 7f4154c00640 1 -- 172.21.15.47:0/224379159 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f41480512b0 con 0x7f41580d2150 2024-02-19T15:28:39.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.302+0000 7f415d400640 1 -- 172.21.15.47:0/224379159 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41580b5690 con 0x7f41580d2150 2024-02-19T15:28:39.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.302+0000 7f4154c00640 1 -- 172.21.15.47:0/224379159 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f4148054970 con 0x7f41580d2150 2024-02-19T15:28:39.304 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:39.304 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:39.304 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:39.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.306+0000 7f415d400640 1 -- 172.21.15.47:0/224379159 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4134066e40 msgr2=0x7f4134069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:39.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.306+0000 7f415d400640 1 --2- 172.21.15.47:0/224379159 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4134066e40 0x7f4134069300 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f41400ae000 tx=0x7f41400b3040 comp rx=0 tx=0).stop 2024-02-19T15:28:39.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.306+0000 7f415d400640 1 -- 172.21.15.47:0/224379159 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41580d2150 msgr2=0x7f41580c62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:39.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.306+0000 7f415d400640 1 --2- 172.21.15.47:0/224379159 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41580d2150 0x7f41580c62e0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f414800c420 tx=0x7f414800c8e0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.306+0000 7f415d400640 1 -- 172.21.15.47:0/224379159 shutdown_connections 2024-02-19T15:28:39.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.306+0000 7f415d400640 1 --2- 172.21.15.47:0/224379159 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4134066e40 0x7f4134069300 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.306+0000 7f415d400640 1 --2- 172.21.15.47:0/224379159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f415815e1b0 0x7f41580b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.306+0000 7f415d400640 1 --2- 172.21.15.47:0/224379159 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f415815ddd0 0x7f415814f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.306+0000 7f415d400640 1 --2- 172.21.15.47:0/224379159 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41580d2150 0x7f41580c62e0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.306+0000 7f415d400640 1 -- 172.21.15.47:0/224379159 >> 172.21.15.47:0/224379159 conn(0x7f415809a9d0 msgr2=0x7f4158155dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:39.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.306+0000 7f415d400640 1 -- 172.21.15.47:0/224379159 shutdown_connections 2024-02-19T15:28:39.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.306+0000 7f415d400640 1 -- 172.21.15.47:0/224379159 wait complete. 2024-02-19T15:28:39.326 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:28:39.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.470+0000 7ff6de200640 1 Processor -- start 2024-02-19T15:28:39.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.470+0000 7ff6de200640 1 -- start start 2024-02-19T15:28:39.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.470+0000 7ff6de200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff6d80cd020 0x7ff6d80ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:39.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.470+0000 7ff6de200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d80ce6e0 con 0x7ff6d815e100 2024-02-19T15:28:39.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.470+0000 7ff6de200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d80acec0 con 0x7ff6d815dd20 2024-02-19T15:28:39.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.470+0000 7ff6de200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6d806c300 con 0x7ff6d80cd020 2024-02-19T15:28:39.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.470+0000 7ff6d7e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff6d80cd020 0x7ff6d80ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:39.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.470+0000 7ff6dd800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff6d815e100 0x7ff6d80ad3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43672/0 (socket says 172.21.15.47:43672) 2024-02-19T15:28:39.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.470+0000 7ff6dd800640 1 -- 172.21.15.47:0/1944779755 learned_addr learned my addr 172.21.15.47:0/1944779755 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:39.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.474+0000 7ff6d7400640 1 -- 172.21.15.47:0/1944779755 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292829494 0 0) 0x7ff6d80ce6e0 con 0x7ff6d815e100 2024-02-19T15:28:39.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.474+0000 7ff6d7400640 1 -- 172.21.15.47:0/1944779755 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c0003530 con 0x7ff6d815e100 2024-02-19T15:28:39.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.474+0000 7ff6d7e00640 1 -- 172.21.15.47:0/1944779755 >> v1:172.21.15.116:6789/0 conn(0x7ff6d815dd20 legacy=0x7ff6d80cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:39.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.474+0000 7ff6d7400640 1 -- 172.21.15.47:0/1944779755 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84745992 0 0) 0x7ff6d80acec0 con 0x7ff6d815dd20 2024-02-19T15:28:39.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.474+0000 7ff6d7e00640 1 -- 172.21.15.47:0/1944779755 >> v1:172.21.15.47:6789/0 conn(0x7ff6d815e100 legacy=0x7ff6d80ad3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:39.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.474+0000 7ff6d7e00640 1 -- 172.21.15.47:0/1944779755 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d806c5e0 con 0x7ff6d80cd020 2024-02-19T15:28:39.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.478+0000 7ff6d7e00640 1 --2- 172.21.15.47:0/1944779755 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6d80cd020 0x7ff6d80ca6e0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7ff6c40097c0 tx=0x7ff6c401c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d7ce60f6fd4cee1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:39.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.478+0000 7ff6d7400640 1 -- 172.21.15.47:0/1944779755 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c40ab600 con 0x7ff6d80cd020 2024-02-19T15:28:39.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.478+0000 7ff6d7400640 1 -- 172.21.15.47:0/1944779755 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c40a7050 con 0x7ff6d80cd020 2024-02-19T15:28:39.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.478+0000 7ff6d7400640 1 -- 172.21.15.47:0/1944779755 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c40ab980 con 0x7ff6d80cd020 2024-02-19T15:28:39.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.478+0000 7ff6de200640 1 -- 172.21.15.47:0/1944779755 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6d80cd020 msgr2=0x7ff6d80ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:39.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.478+0000 7ff6de200640 1 --2- 172.21.15.47:0/1944779755 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6d80cd020 0x7ff6d80ca6e0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7ff6c40097c0 tx=0x7ff6c401c350 comp rx=0 tx=0).stop 2024-02-19T15:28:39.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.478+0000 7ff6de200640 1 -- 172.21.15.47:0/1944779755 shutdown_connections 2024-02-19T15:28:39.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.478+0000 7ff6de200640 1 --2- 172.21.15.47:0/1944779755 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6d80cd020 0x7ff6d80ca6e0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.478+0000 7ff6de200640 1 -- 172.21.15.47:0/1944779755 >> 172.21.15.47:0/1944779755 conn(0x7ff6d80b1be0 msgr2=0x7ff6d80b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:39.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.478+0000 7ff6de200640 1 -- 172.21.15.47:0/1944779755 shutdown_connections 2024-02-19T15:28:39.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.478+0000 7ff6de200640 1 -- 172.21.15.47:0/1944779755 wait complete. 2024-02-19T15:28:39.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.478+0000 7ff6de200640 1 Processor -- start 2024-02-19T15:28:39.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.478+0000 7ff6de200640 1 -- start start 2024-02-19T15:28:39.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6de200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d80cd020 0x7ff6d8155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:39.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6de200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d815dd20 0x7ff6d8072790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:39.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6de200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6d815e100 0x7ff6d8072cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:39.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6de200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d809ab70 con 0x7ff6d815e100 2024-02-19T15:28:39.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6de200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d809ae50 con 0x7ff6d815dd20 2024-02-19T15:28:39.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6de200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d809b130 con 0x7ff6d80cd020 2024-02-19T15:28:39.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6dd800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6d815e100 0x7ff6d8072cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:39.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6dd800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6d815e100 0x7ff6d8072cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60060/0 (socket says 172.21.15.47:60060) 2024-02-19T15:28:39.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6dd800640 1 -- 172.21.15.47:0/136352843 learned_addr learned my addr 172.21.15.47:0/136352843 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:39.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6dce00640 1 --2- 172.21.15.47:0/136352843 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d80cd020 0x7ff6d8155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:39.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6d7e00640 1 --2- 172.21.15.47:0/136352843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d815dd20 0x7ff6d8072790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:39.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6dd800640 1 -- 172.21.15.47:0/136352843 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d80cd020 msgr2=0x7ff6d8155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:39.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6dd800640 1 --2- 172.21.15.47:0/136352843 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d80cd020 0x7ff6d8155de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6dd800640 1 -- 172.21.15.47:0/136352843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d815dd20 msgr2=0x7ff6d8072790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:39.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6dd800640 1 --2- 172.21.15.47:0/136352843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d815dd20 0x7ff6d8072790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6dd800640 1 -- 172.21.15.47:0/136352843 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c4009420 con 0x7ff6d815e100 2024-02-19T15:28:39.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6dce00640 1 --2- 172.21.15.47:0/136352843 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d80cd020 0x7ff6d8155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:39.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6dd800640 1 --2- 172.21.15.47:0/136352843 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6d815e100 0x7ff6d8072cd0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000caa0 tx=0x7ff6d000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:39.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6d4c00640 1 -- 172.21.15.47:0/136352843 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d000ac00 con 0x7ff6d815e100 2024-02-19T15:28:39.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6d4c00640 1 -- 172.21.15.47:0/136352843 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d000ad60 con 0x7ff6d815e100 2024-02-19T15:28:39.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6d4c00640 1 -- 172.21.15.47:0/136352843 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0012620 con 0x7ff6d815e100 2024-02-19T15:28:39.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6de200640 1 -- 172.21.15.47:0/136352843 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d809b410 con 0x7ff6d815e100 2024-02-19T15:28:39.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6de200640 1 -- 172.21.15.47:0/136352843 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d8156320 con 0x7ff6d815e100 2024-02-19T15:28:39.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.482+0000 7ff6de200640 1 -- 172.21.15.47:0/136352843 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d80cfd00 con 0x7ff6d815e100 2024-02-19T15:28:39.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.486+0000 7ff6d4c00640 1 -- 172.21.15.47:0/136352843 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff6d0007070 con 0x7ff6d815e100 2024-02-19T15:28:39.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.486+0000 7ff6d4c00640 1 --2- 172.21.15.47:0/136352843 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff6b8066e40 0x7ff6b8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:39.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.486+0000 7ff6d4c00640 1 -- 172.21.15.47:0/136352843 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7ff6d0084990 con 0x7ff6d815e100 2024-02-19T15:28:39.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.486+0000 7ff6dce00640 1 --2- 172.21.15.47:0/136352843 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff6b8066e40 0x7ff6b8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:39.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.486+0000 7ff6dce00640 1 --2- 172.21.15.47:0/136352843 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff6b8066e40 0x7ff6b8069300 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80b0fa0 tx=0x7ff6cc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:39.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.490+0000 7ff6d4c00640 1 -- 172.21.15.47:0/136352843 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff6d00517d0 con 0x7ff6d815e100 2024-02-19T15:28:39.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.682+0000 7ff6de200640 1 -- 172.21.15.47:0/136352843 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6d8075360 con 0x7ff6d815e100 2024-02-19T15:28:39.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.682+0000 7ff6d4c00640 1 -- 172.21.15.47:0/136352843 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+16990 (secure 0 0 0) 0x7ff6d0017020 con 0x7ff6d815e100 2024-02-19T15:28:39.682 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:39.683 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":114,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:27:47.287065+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:27:19.839144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-02-19T15:27:20.405924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.36","mappings":[{"from":0,"to":5}]},{"pgid":"17.3f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6837/819751716":"2024-02-20T15:27:39.101989+0000","172.21.15.116:6835/819751716":"2024-02-20T15:27:39.101989+0000","172.21.15.47:6834/897369244":"2024-02-20T15:27:38.263490+0000","172.21.15.47:6832/897369244":"2024-02-20T15:27:38.263490+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:28:39.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.686+0000 7ff6de200640 1 -- 172.21.15.47:0/136352843 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff6b8066e40 msgr2=0x7ff6b8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:39.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.686+0000 7ff6de200640 1 --2- 172.21.15.47:0/136352843 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff6b8066e40 0x7ff6b8069300 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80b0fa0 tx=0x7ff6cc009040 comp rx=0 tx=0).stop 2024-02-19T15:28:39.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.686+0000 7ff6de200640 1 -- 172.21.15.47:0/136352843 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6d815e100 msgr2=0x7ff6d8072cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:39.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.686+0000 7ff6de200640 1 --2- 172.21.15.47:0/136352843 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6d815e100 0x7ff6d8072cd0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000caa0 tx=0x7ff6d000cf60 comp rx=0 tx=0).stop 2024-02-19T15:28:39.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.686+0000 7ff6de200640 1 -- 172.21.15.47:0/136352843 shutdown_connections 2024-02-19T15:28:39.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.686+0000 7ff6de200640 1 --2- 172.21.15.47:0/136352843 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff6b8066e40 0x7ff6b8069300 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.686+0000 7ff6de200640 1 --2- 172.21.15.47:0/136352843 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff6d815e100 0x7ff6d8072cd0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.686+0000 7ff6de200640 1 --2- 172.21.15.47:0/136352843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d815dd20 0x7ff6d8072790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.686+0000 7ff6de200640 1 --2- 172.21.15.47:0/136352843 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d80cd020 0x7ff6d8155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.686+0000 7ff6de200640 1 -- 172.21.15.47:0/136352843 >> 172.21.15.47:0/136352843 conn(0x7ff6d80b1be0 msgr2=0x7ff6d80b04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:39.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.686+0000 7ff6de200640 1 -- 172.21.15.47:0/136352843 shutdown_connections 2024-02-19T15:28:39.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.686+0000 7ff6de200640 1 -- 172.21.15.47:0/136352843 wait complete. 2024-02-19T15:28:39.705 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:27:19.839144+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:27:20.405924+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:28:39.706 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:28:39.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.846+0000 7fac09c00640 1 Processor -- start 2024-02-19T15:28:39.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.846+0000 7fac09c00640 1 -- start start 2024-02-19T15:28:39.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac09c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fac0415bc60 0x7fac040ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:39.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac09c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac040d0650 0x7fac040b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:39.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac09c00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac040b0db0 con 0x7fac040d2480 2024-02-19T15:28:39.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac09c00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac040cf9e0 con 0x7fac0415bc60 2024-02-19T15:28:39.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac09c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac040b0fe0 con 0x7fac040d0650 2024-02-19T15:28:39.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fabfb400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fac040d2480 0x7fac040cff40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43678/0 (socket says 172.21.15.47:43678) 2024-02-19T15:28:39.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fabfb400640 1 -- 172.21.15.47:0/1291312565 learned_addr learned my addr 172.21.15.47:0/1291312565 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:39.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac09200640 1 --2- 172.21.15.47:0/1291312565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac040d0650 0x7fac040b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:39.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac03e00640 1 --2- 172.21.15.47:0/1291312565 >> v2:172.21.15.116:3300/0 conn(0x7fac0415bc60 0x7fac040ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:39.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac03400640 1 -- 172.21.15.47:0/1291312565 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662537224 0 0) 0x7fac040b0db0 con 0x7fac040d2480 2024-02-19T15:28:39.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac03400640 1 -- 172.21.15.47:0/1291312565 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe8003530 con 0x7fac040d2480 2024-02-19T15:28:39.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac03400640 1 -- 172.21.15.47:0/1291312565 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2853799731 0 0) 0x7fabe8003530 con 0x7fac040d2480 2024-02-19T15:28:39.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac03400640 1 -- 172.21.15.47:0/1291312565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac040d0650 msgr2=0x7fac040b1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:39.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac03400640 1 --2- 172.21.15.47:0/1291312565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac040d0650 0x7fac040b1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac03400640 1 -- 172.21.15.47:0/1291312565 >> v2:172.21.15.116:3300/0 conn(0x7fac0415bc60 msgr2=0x7fac040ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:39.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac03400640 1 --2- 172.21.15.47:0/1291312565 >> v2:172.21.15.116:3300/0 conn(0x7fac0415bc60 0x7fac040ce9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac03400640 1 -- 172.21.15.47:0/1291312565 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac040b12c0 con 0x7fac040d2480 2024-02-19T15:28:39.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac03400640 1 -- 172.21.15.47:0/1291312565 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7fabec001680 con 0x7fac040d2480 2024-02-19T15:28:39.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac03400640 1 -- 172.21.15.47:0/1291312565 >> v1:172.21.15.47:6789/0 conn(0x7fac040d2480 legacy=0x7fac040cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:39.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac03400640 1 --2- 172.21.15.47:0/1291312565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac040d0650 0x7fabe8003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:39.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac03400640 1 --2- 172.21.15.47:0/1291312565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fabe8003f50 0x7fabe80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:39.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac03400640 1 --2- 172.21.15.47:0/1291312565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe8004d10 0x7fabe8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:39.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac03400640 1 -- 172.21.15.47:0/1291312565 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac040cf9e0 con 0x7fabe8003f50 2024-02-19T15:28:39.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac03400640 1 -- 172.21.15.47:0/1291312565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac040b0fe0 con 0x7fabe8004d10 2024-02-19T15:28:39.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac03400640 1 -- 172.21.15.47:0/1291312565 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe800fe50 con 0x7fac040d0650 2024-02-19T15:28:39.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fabfb400640 1 --2- 172.21.15.47:0/1291312565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fabe8003f50 0x7fabe80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:39.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac03e00640 1 --2- 172.21.15.47:0/1291312565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac040d0650 0x7fabe8003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:39.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fac09200640 1 --2- 172.21.15.47:0/1291312565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe8004d10 0x7fabe8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:39.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fabfb400640 1 -- 172.21.15.47:0/1291312565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac040d0650 msgr2=0x7fabe8003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:39.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fabfb400640 1 --2- 172.21.15.47:0/1291312565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac040d0650 0x7fabe8003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fabfb400640 1 -- 172.21.15.47:0/1291312565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe8004d10 msgr2=0x7fabe8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:39.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fabfb400640 1 --2- 172.21.15.47:0/1291312565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe8004d10 0x7fabe8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.850+0000 7fabfb400640 1 -- 172.21.15.47:0/1291312565 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe8003530 con 0x7fabe8003f50 2024-02-19T15:28:39.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.854+0000 7fabfb400640 1 --2- 172.21.15.47:0/1291312565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fabe8003f50 0x7fabe80047d0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fabec00c0e0 tx=0x7fabec01cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d68253a3b27c85ee server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:39.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.854+0000 7fac03400640 1 -- 172.21.15.47:0/1291312565 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabec0058e0 con 0x7fabe8003f50 2024-02-19T15:28:39.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.854+0000 7fac03400640 1 -- 172.21.15.47:0/1291312565 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabec005a60 con 0x7fabe8003f50 2024-02-19T15:28:39.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.854+0000 7fac03400640 1 -- 172.21.15.47:0/1291312565 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabec005e50 con 0x7fabe8003f50 2024-02-19T15:28:39.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.854+0000 7fac09c00640 1 -- 172.21.15.47:0/1291312565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fabe8003f50 msgr2=0x7fabe80047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:39.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.854+0000 7fac09c00640 1 --2- 172.21.15.47:0/1291312565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fabe8003f50 0x7fabe80047d0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fabec00c0e0 tx=0x7fabec01cb50 comp rx=0 tx=0).stop 2024-02-19T15:28:39.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.854+0000 7fac09200640 1 -- 172.21.15.47:0/1291312565 reap_dead start 2024-02-19T15:28:39.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.854+0000 7fac09c00640 1 -- 172.21.15.47:0/1291312565 shutdown_connections 2024-02-19T15:28:39.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.854+0000 7fac09c00640 1 -- 172.21.15.47:0/1291312565 >> 172.21.15.47:0/1291312565 conn(0x7fac040d1960 msgr2=0x7fac040d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:39.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.854+0000 7fac09c00640 1 -- 172.21.15.47:0/1291312565 shutdown_connections 2024-02-19T15:28:39.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.854+0000 7fac09c00640 1 -- 172.21.15.47:0/1291312565 wait complete. 2024-02-19T15:28:39.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.854+0000 7fac09c00640 1 Processor -- start 2024-02-19T15:28:39.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.854+0000 7fac09c00640 1 -- start start 2024-02-19T15:28:39.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fac09c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabe8003f50 0x7fac040b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:39.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fac09c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac04158b30 0x7fac040b0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:39.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fac09c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac040d0be0 0x7fac040d1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:39.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fac09c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac040d1580 con 0x7fac04158b30 2024-02-19T15:28:39.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fac09c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac040b5570 con 0x7fac040d0be0 2024-02-19T15:28:39.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fac09c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac040b5850 con 0x7fabe8003f50 2024-02-19T15:28:39.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fabfb400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac04158b30 0x7fac040b0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:39.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fabfb400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac04158b30 0x7fac040b0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60072/0 (socket says 172.21.15.47:60072) 2024-02-19T15:28:39.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fabfb400640 1 -- 172.21.15.47:0/4146515805 learned_addr learned my addr 172.21.15.47:0/4146515805 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:39.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fabfb400640 1 -- 172.21.15.47:0/4146515805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabe8003f50 msgr2=0x7fac040b4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:39.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fabfb400640 1 --2- 172.21.15.47:0/4146515805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabe8003f50 0x7fac040b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fabfb400640 1 -- 172.21.15.47:0/4146515805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac040d0be0 msgr2=0x7fac040d1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:39.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fabfb400640 1 --2- 172.21.15.47:0/4146515805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac040d0be0 0x7fac040d1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:39.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fabfb400640 1 -- 172.21.15.47:0/4146515805 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac040b5b30 con 0x7fac04158b30 2024-02-19T15:28:39.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fabfb400640 1 --2- 172.21.15.47:0/4146515805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac04158b30 0x7fac040b0190 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fabec007040 tx=0x7fabe8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:39.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fac00c00640 1 -- 172.21.15.47:0/4146515805 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabec0047f0 con 0x7fac04158b30 2024-02-19T15:28:39.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fac00c00640 1 -- 172.21.15.47:0/4146515805 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabec004950 con 0x7fac04158b30 2024-02-19T15:28:39.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fac00c00640 1 -- 172.21.15.47:0/4146515805 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabec004ce0 con 0x7fac04158b30 2024-02-19T15:28:39.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fac09c00640 1 -- 172.21.15.47:0/4146515805 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac040b5e10 con 0x7fac04158b30 2024-02-19T15:28:39.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fac09c00640 1 -- 172.21.15.47:0/4146515805 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac040b06d0 con 0x7fac04158b30 2024-02-19T15:28:39.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.858+0000 7fac09c00640 1 -- 172.21.15.47:0/4146515805 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac0404f260 con 0x7fac04158b30 2024-02-19T15:28:39.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.862+0000 7fac00c00640 1 -- 172.21.15.47:0/4146515805 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fabec007070 con 0x7fac04158b30 2024-02-19T15:28:39.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.862+0000 7fac00c00640 1 --2- 172.21.15.47:0/4146515805 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fabe0066e40 0x7fabe0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:39.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.862+0000 7fac00c00640 1 -- 172.21.15.47:0/4146515805 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fabec11e410 con 0x7fac04158b30 2024-02-19T15:28:39.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.862+0000 7fac03e00640 1 --2- 172.21.15.47:0/4146515805 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fabe0066e40 0x7fabe0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:39.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.862+0000 7fac03e00640 1 --2- 172.21.15.47:0/4146515805 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fabe0066e40 0x7fabe0069300 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fabf4006b20 tx=0x7fabf4006ab0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:39.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:39.866+0000 7fac00c00640 1 -- 172.21.15.47:0/4146515805 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fabec0eb110 con 0x7fac04158b30 2024-02-19T15:28:40.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.026+0000 7fac09c00640 1 -- 172.21.15.47:0/4146515805 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fac040d0350 con 0x7fac04158b30 2024-02-19T15:28:40.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.030+0000 7fac00c00640 1 -- 172.21.15.47:0/4146515805 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7fabec0c0e80 con 0x7fac04158b30 2024-02-19T15:28:40.029 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:28:40.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.030+0000 7fac09c00640 1 -- 172.21.15.47:0/4146515805 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fabe0066e40 msgr2=0x7fabe0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:40.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.030+0000 7fac09c00640 1 --2- 172.21.15.47:0/4146515805 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fabe0066e40 0x7fabe0069300 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fabf4006b20 tx=0x7fabf4006ab0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.030+0000 7fac09c00640 1 -- 172.21.15.47:0/4146515805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac04158b30 msgr2=0x7fac040b0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:40.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.030+0000 7fac09c00640 1 --2- 172.21.15.47:0/4146515805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac04158b30 0x7fac040b0190 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fabec007040 tx=0x7fabe8005800 comp rx=0 tx=0).stop 2024-02-19T15:28:40.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.030+0000 7fac09c00640 1 -- 172.21.15.47:0/4146515805 shutdown_connections 2024-02-19T15:28:40.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.030+0000 7fac09c00640 1 --2- 172.21.15.47:0/4146515805 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fabe0066e40 0x7fabe0069300 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.030+0000 7fac09c00640 1 --2- 172.21.15.47:0/4146515805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac040d0be0 0x7fac040d1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.030+0000 7fac09c00640 1 --2- 172.21.15.47:0/4146515805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac04158b30 0x7fac040b0190 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.030+0000 7fac09c00640 1 --2- 172.21.15.47:0/4146515805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabe8003f50 0x7fac040b4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.030+0000 7fac09c00640 1 -- 172.21.15.47:0/4146515805 >> 172.21.15.47:0/4146515805 conn(0x7fac040d1960 msgr2=0x7fac040caa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:40.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.030+0000 7fac09c00640 1 -- 172.21.15.47:0/4146515805 shutdown_connections 2024-02-19T15:28:40.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.030+0000 7fac09c00640 1 -- 172.21.15.47:0/4146515805 wait complete. 2024-02-19T15:28:40.051 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:28:40.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.194+0000 7fc914c00640 1 Processor -- start 2024-02-19T15:28:40.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.194+0000 7fc914c00640 1 -- start start 2024-02-19T15:28:40.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.194+0000 7fc914c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9100c7e80 0x7fc9100d0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:40.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.194+0000 7fc914c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc9100c8260 0x7fc9100d0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:40.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.194+0000 7fc914c00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9100cd750 con 0x7fc9100cfbf0 2024-02-19T15:28:40.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.194+0000 7fc914c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9100ceee0 con 0x7fc9100c8260 2024-02-19T15:28:40.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.194+0000 7fc914c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9100cdc60 con 0x7fc9100c7e80 2024-02-19T15:28:40.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.194+0000 7fc905400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc9100c8260 0x7fc9100d0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:40.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.194+0000 7fc905400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc9100c8260 0x7fc9100d0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60074/0 (socket says 172.21.15.47:60074) 2024-02-19T15:28:40.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.194+0000 7fc905400640 1 -- 172.21.15.47:0/2819068331 learned_addr learned my addr 172.21.15.47:0/2819068331 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:40.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.194+0000 7fc90e800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc9100cfbf0 0x7fc9100c5680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43686/0 (socket says 172.21.15.47:43686) 2024-02-19T15:28:40.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.194+0000 7fc90de00640 1 --2- 172.21.15.47:0/2819068331 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9100c7e80 0x7fc9100d0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:40.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.194+0000 7fc905400640 1 -- 172.21.15.47:0/2819068331 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9100c7e80 msgr2=0x7fc9100d0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:40.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.194+0000 7fc905400640 1 --2- 172.21.15.47:0/2819068331 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9100c7e80 0x7fc9100d0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.194+0000 7fc905400640 1 -- 172.21.15.47:0/2819068331 >> v1:172.21.15.47:6789/0 conn(0x7fc9100cfbf0 legacy=0x7fc9100c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:40.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.194+0000 7fc905400640 1 -- 172.21.15.47:0/2819068331 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9100d2a40 con 0x7fc9100c8260 2024-02-19T15:28:40.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.198+0000 7fc905400640 1 --2- 172.21.15.47:0/2819068331 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9100c8260 0x7fc9100d0650 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8009870 tx=0x7fc8f801c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc47468905b5e0a1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:40.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.198+0000 7fc90d400640 1 -- 172.21.15.47:0/2819068331 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f80a95f0 con 0x7fc9100c8260 2024-02-19T15:28:40.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.198+0000 7fc90d400640 1 -- 172.21.15.47:0/2819068331 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f80a9750 con 0x7fc9100c8260 2024-02-19T15:28:40.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.198+0000 7fc90d400640 1 -- 172.21.15.47:0/2819068331 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f80a9ae0 con 0x7fc9100c8260 2024-02-19T15:28:40.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.198+0000 7fc914c00640 1 -- 172.21.15.47:0/2819068331 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9100c8260 msgr2=0x7fc9100d0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:40.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.198+0000 7fc914c00640 1 --2- 172.21.15.47:0/2819068331 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9100c8260 0x7fc9100d0650 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8009870 tx=0x7fc8f801c530 comp rx=0 tx=0).stop 2024-02-19T15:28:40.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.198+0000 7fc914c00640 1 -- 172.21.15.47:0/2819068331 shutdown_connections 2024-02-19T15:28:40.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.198+0000 7fc914c00640 1 --2- 172.21.15.47:0/2819068331 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9100c8260 0x7fc9100d0650 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.198+0000 7fc914c00640 1 --2- 172.21.15.47:0/2819068331 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9100c7e80 0x7fc9100d0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.198+0000 7fc914c00640 1 -- 172.21.15.47:0/2819068331 >> 172.21.15.47:0/2819068331 conn(0x7fc9100d1b30 msgr2=0x7fc9100d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:40.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.198+0000 7fc914c00640 1 -- 172.21.15.47:0/2819068331 shutdown_connections 2024-02-19T15:28:40.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.198+0000 7fc914c00640 1 -- 172.21.15.47:0/2819068331 wait complete. 2024-02-19T15:28:40.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc914c00640 1 Processor -- start 2024-02-19T15:28:40.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc914c00640 1 -- start start 2024-02-19T15:28:40.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc914c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9100c7e80 0x7fc910157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:40.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc914c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9100c8260 0x7fc91006c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:40.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc914c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9100cfbf0 0x7fc91015dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:40.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc914c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9100ae940 con 0x7fc9100cfbf0 2024-02-19T15:28:40.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc914c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9100aead0 con 0x7fc9100c7e80 2024-02-19T15:28:40.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc914c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9101575a0 con 0x7fc9100c8260 2024-02-19T15:28:40.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc90e800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9100cfbf0 0x7fc91015dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:40.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc90e800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9100cfbf0 0x7fc91015dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60088/0 (socket says 172.21.15.47:60088) 2024-02-19T15:28:40.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc90e800640 1 -- 172.21.15.47:0/956455767 learned_addr learned my addr 172.21.15.47:0/956455767 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:40.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc905400640 1 --2- 172.21.15.47:0/956455767 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9100c8260 0x7fc91006c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:40.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc90de00640 1 --2- 172.21.15.47:0/956455767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9100c7e80 0x7fc910157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:40.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc90e800640 1 -- 172.21.15.47:0/956455767 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9100c8260 msgr2=0x7fc91006c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:40.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc90e800640 1 --2- 172.21.15.47:0/956455767 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9100c8260 0x7fc91006c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc90e800640 1 -- 172.21.15.47:0/956455767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9100c7e80 msgr2=0x7fc910157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:40.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc90e800640 1 --2- 172.21.15.47:0/956455767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9100c7e80 0x7fc910157060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc90e800640 1 -- 172.21.15.47:0/956455767 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f80094d0 con 0x7fc9100cfbf0 2024-02-19T15:28:40.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc90de00640 1 --2- 172.21.15.47:0/956455767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9100c7e80 0x7fc910157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:40.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc90e800640 1 --2- 172.21.15.47:0/956455767 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9100cfbf0 0x7fc91015dab0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fc90800c6f0 tx=0x7fc90800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:40.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc906a00640 1 -- 172.21.15.47:0/956455767 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc908005db0 con 0x7fc9100cfbf0 2024-02-19T15:28:40.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc906a00640 1 -- 172.21.15.47:0/956455767 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90800a640 con 0x7fc9100cfbf0 2024-02-19T15:28:40.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc906a00640 1 -- 172.21.15.47:0/956455767 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90800a9b0 con 0x7fc9100cfbf0 2024-02-19T15:28:40.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc914c00640 1 -- 172.21.15.47:0/956455767 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc91006d490 con 0x7fc9100cfbf0 2024-02-19T15:28:40.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc914c00640 1 -- 172.21.15.47:0/956455767 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc91006da50 con 0x7fc9100cfbf0 2024-02-19T15:28:40.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.202+0000 7fc914c00640 1 -- 172.21.15.47:0/956455767 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9100d1180 con 0x7fc9100cfbf0 2024-02-19T15:28:40.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.206+0000 7fc906a00640 1 -- 172.21.15.47:0/956455767 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc90800ab10 con 0x7fc9100cfbf0 2024-02-19T15:28:40.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.206+0000 7fc906a00640 1 --2- 172.21.15.47:0/956455767 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc8ec066e70 0x7fc8ec069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:40.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.206+0000 7fc906a00640 1 -- 172.21.15.47:0/956455767 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fc908084290 con 0x7fc9100cfbf0 2024-02-19T15:28:40.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.206+0000 7fc90de00640 1 --2- 172.21.15.47:0/956455767 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc8ec066e70 0x7fc8ec069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:40.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.206+0000 7fc90de00640 1 --2- 172.21.15.47:0/956455767 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc8ec066e70 0x7fc8ec069330 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fc900009a40 tx=0x7fc900008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:40.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.210+0000 7fc906a00640 1 -- 172.21.15.47:0/956455767 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc908050fa0 con 0x7fc9100cfbf0 2024-02-19T15:28:40.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.370+0000 7fc914c00640 1 -- 172.21.15.47:0/956455767 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc9100aec10 con 0x7fc9100cfbf0 2024-02-19T15:28:40.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.370+0000 7fc906a00640 1 -- 172.21.15.47:0/956455767 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc908054660 con 0x7fc9100cfbf0 2024-02-19T15:28:40.372 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:28:40.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.374+0000 7fc914c00640 1 -- 172.21.15.47:0/956455767 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc8ec066e70 msgr2=0x7fc8ec069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:40.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.374+0000 7fc914c00640 1 --2- 172.21.15.47:0/956455767 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc8ec066e70 0x7fc8ec069330 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fc900009a40 tx=0x7fc900008040 comp rx=0 tx=0).stop 2024-02-19T15:28:40.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.374+0000 7fc914c00640 1 -- 172.21.15.47:0/956455767 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9100cfbf0 msgr2=0x7fc91015dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:40.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.374+0000 7fc914c00640 1 --2- 172.21.15.47:0/956455767 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9100cfbf0 0x7fc91015dab0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fc90800c6f0 tx=0x7fc90800cbb0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.374+0000 7fc914c00640 1 -- 172.21.15.47:0/956455767 shutdown_connections 2024-02-19T15:28:40.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.374+0000 7fc914c00640 1 --2- 172.21.15.47:0/956455767 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc8ec066e70 0x7fc8ec069330 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.374+0000 7fc914c00640 1 --2- 172.21.15.47:0/956455767 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9100cfbf0 0x7fc91015dab0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.374+0000 7fc914c00640 1 --2- 172.21.15.47:0/956455767 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9100c8260 0x7fc91006c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.374+0000 7fc914c00640 1 --2- 172.21.15.47:0/956455767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9100c7e80 0x7fc910157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.374+0000 7fc914c00640 1 -- 172.21.15.47:0/956455767 >> 172.21.15.47:0/956455767 conn(0x7fc9100d1b30 msgr2=0x7fc9100cde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:40.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.374+0000 7fc914c00640 1 -- 172.21.15.47:0/956455767 shutdown_connections 2024-02-19T15:28:40.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.374+0000 7fc914c00640 1 -- 172.21.15.47:0/956455767 wait complete. 2024-02-19T15:28:40.393 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:28:40.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.534+0000 7fa334800640 1 Processor -- start 2024-02-19T15:28:40.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.534+0000 7fa334800640 1 -- start start 2024-02-19T15:28:40.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.534+0000 7fa334800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa32c15e280 0x7fa32c06c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:40.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.534+0000 7fa334800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa32c0cc620 con 0x7fa32c0ce3b0 2024-02-19T15:28:40.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.534+0000 7fa334800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa32c0d12e0 con 0x7fa32c0d2650 2024-02-19T15:28:40.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.534+0000 7fa334800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c0d1940 con 0x7fa32c15e280 2024-02-19T15:28:40.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.538+0000 7fa331e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa32c0ce3b0 0x7fa32c0ce790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43700/0 (socket says 172.21.15.47:43700) 2024-02-19T15:28:40.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.538+0000 7fa331e00640 1 -- 172.21.15.47:0/3804305666 learned_addr learned my addr 172.21.15.47:0/3804305666 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:40.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.538+0000 7fa332800640 1 --2- 172.21.15.47:0/3804305666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa32c15e280 0x7fa32c06c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:40.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.538+0000 7fa330a00640 1 -- 172.21.15.47:0/3804305666 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252601746 0 0) 0x7fa32c0cc620 con 0x7fa32c0ce3b0 2024-02-19T15:28:40.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.538+0000 7fa330a00640 1 -- 172.21.15.47:0/3804305666 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa314003530 con 0x7fa32c0ce3b0 2024-02-19T15:28:40.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.538+0000 7fa330a00640 1 -- 172.21.15.47:0/3804305666 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246923578 0 0) 0x7fa32c0d12e0 con 0x7fa32c0d2650 2024-02-19T15:28:40.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.538+0000 7fa330a00640 1 -- 172.21.15.47:0/3804305666 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa32c0cc620 con 0x7fa32c0d2650 2024-02-19T15:28:40.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.538+0000 7fa332800640 1 -- 172.21.15.47:0/3804305666 >> v1:172.21.15.116:6789/0 conn(0x7fa32c0d2650 legacy=0x7fa32c06c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:40.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.538+0000 7fa332800640 1 -- 172.21.15.47:0/3804305666 >> v1:172.21.15.47:6789/0 conn(0x7fa32c0ce3b0 legacy=0x7fa32c0ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:40.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.538+0000 7fa332800640 1 -- 172.21.15.47:0/3804305666 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32c0d1000 con 0x7fa32c15e280 2024-02-19T15:28:40.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.538+0000 7fa332800640 1 --2- 172.21.15.47:0/3804305666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa32c15e280 0x7fa32c06c8e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa328009dc0 tx=0x7fa32801c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b840935d4e163bb7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:40.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.538+0000 7fa330a00640 1 -- 172.21.15.47:0/3804305666 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3280a57c0 con 0x7fa32c15e280 2024-02-19T15:28:40.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa330a00640 1 -- 172.21.15.47:0/3804305666 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3280a5920 con 0x7fa32c15e280 2024-02-19T15:28:40.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa330a00640 1 -- 172.21.15.47:0/3804305666 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3280a5c90 con 0x7fa32c15e280 2024-02-19T15:28:40.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa334800640 1 -- 172.21.15.47:0/3804305666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa32c15e280 msgr2=0x7fa32c06c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:40.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa334800640 1 --2- 172.21.15.47:0/3804305666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa32c15e280 0x7fa32c06c8e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa328009dc0 tx=0x7fa32801c870 comp rx=0 tx=0).stop 2024-02-19T15:28:40.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa334800640 1 -- 172.21.15.47:0/3804305666 shutdown_connections 2024-02-19T15:28:40.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa334800640 1 --2- 172.21.15.47:0/3804305666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa32c15e280 0x7fa32c06c8e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa334800640 1 -- 172.21.15.47:0/3804305666 >> 172.21.15.47:0/3804305666 conn(0x7fa32c0d1b30 msgr2=0x7fa32c0d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:40.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa334800640 1 -- 172.21.15.47:0/3804305666 shutdown_connections 2024-02-19T15:28:40.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa334800640 1 -- 172.21.15.47:0/3804305666 wait complete. 2024-02-19T15:28:40.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa334800640 1 Processor -- start 2024-02-19T15:28:40.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa334800640 1 -- start start 2024-02-19T15:28:40.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa334800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa32c0ce3b0 0x7fa32c1530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:40.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa334800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa32c0d2650 0x7fa32c0cb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:40.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa334800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa32c15e280 0x7fa32c157fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:40.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa334800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c1554c0 con 0x7fa32c0d2650 2024-02-19T15:28:40.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa334800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c1548b0 con 0x7fa32c0ce3b0 2024-02-19T15:28:40.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa334800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c154b90 con 0x7fa32c15e280 2024-02-19T15:28:40.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa331400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa32c0d2650 0x7fa32c0cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:40.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa331400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa32c0d2650 0x7fa32c0cb830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60092/0 (socket says 172.21.15.47:60092) 2024-02-19T15:28:40.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa331400640 1 -- 172.21.15.47:0/1750657867 learned_addr learned my addr 172.21.15.47:0/1750657867 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:40.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa331e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa32c0ce3b0 0x7fa32c1530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:40.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.542+0000 7fa332800640 1 --2- 172.21.15.47:0/1750657867 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa32c15e280 0x7fa32c157fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:40.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.546+0000 7fa331400640 1 -- 172.21.15.47:0/1750657867 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa32c15e280 msgr2=0x7fa32c157fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:40.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.546+0000 7fa331400640 1 --2- 172.21.15.47:0/1750657867 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa32c15e280 0x7fa32c157fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.546+0000 7fa331400640 1 -- 172.21.15.47:0/1750657867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa32c0ce3b0 msgr2=0x7fa32c1530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:40.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.546+0000 7fa331400640 1 --2- 172.21.15.47:0/1750657867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa32c0ce3b0 0x7fa32c1530d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.546+0000 7fa331400640 1 -- 172.21.15.47:0/1750657867 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa328009a70 con 0x7fa32c0d2650 2024-02-19T15:28:40.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.546+0000 7fa331400640 1 --2- 172.21.15.47:0/1750657867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa32c0d2650 0x7fa32c0cb830 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00c430 tx=0x7fa31c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:40.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.546+0000 7fa326000640 1 -- 172.21.15.47:0/1750657867 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31c0052d0 con 0x7fa32c0d2650 2024-02-19T15:28:40.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.546+0000 7fa326000640 1 -- 172.21.15.47:0/1750657867 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa31c005450 con 0x7fa32c0d2650 2024-02-19T15:28:40.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.546+0000 7fa334800640 1 -- 172.21.15.47:0/1750657867 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa32c154e70 con 0x7fa32c0d2650 2024-02-19T15:28:40.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.546+0000 7fa326000640 1 -- 172.21.15.47:0/1750657867 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31c003840 con 0x7fa32c0d2650 2024-02-19T15:28:40.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.546+0000 7fa334800640 1 -- 172.21.15.47:0/1750657867 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa32c153610 con 0x7fa32c0d2650 2024-02-19T15:28:40.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.546+0000 7fa334800640 1 -- 172.21.15.47:0/1750657867 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa32c155630 con 0x7fa32c0d2650 2024-02-19T15:28:40.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.546+0000 7fa326000640 1 -- 172.21.15.47:0/1750657867 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa31c0039a0 con 0x7fa32c0d2650 2024-02-19T15:28:40.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.550+0000 7fa326000640 1 --2- 172.21.15.47:0/1750657867 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa310066e40 0x7fa310069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:40.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.550+0000 7fa326000640 1 -- 172.21.15.47:0/1750657867 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fa31c0841f0 con 0x7fa32c0d2650 2024-02-19T15:28:40.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.550+0000 7fa331e00640 1 --2- 172.21.15.47:0/1750657867 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa310066e40 0x7fa310069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:40.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.550+0000 7fa331e00640 1 --2- 172.21.15.47:0/1750657867 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa310066e40 0x7fa310069300 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa320000f70 tx=0x7fa320007d40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:40.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.554+0000 7fa326000640 1 -- 172.21.15.47:0/1750657867 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa31c050e70 con 0x7fa32c0d2650 2024-02-19T15:28:40.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.714+0000 7fa334800640 1 -- 172.21.15.47:0/1750657867 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa32c075360 con 0x7fa32c0d2650 2024-02-19T15:28:40.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.714+0000 7fa326000640 1 -- 172.21.15.47:0/1750657867 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa31c054530 con 0x7fa32c0d2650 2024-02-19T15:28:40.715 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:28:40.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.718+0000 7fa334800640 1 -- 172.21.15.47:0/1750657867 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa310066e40 msgr2=0x7fa310069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:40.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.718+0000 7fa334800640 1 --2- 172.21.15.47:0/1750657867 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa310066e40 0x7fa310069300 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa320000f70 tx=0x7fa320007d40 comp rx=0 tx=0).stop 2024-02-19T15:28:40.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.718+0000 7fa334800640 1 -- 172.21.15.47:0/1750657867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa32c0d2650 msgr2=0x7fa32c0cb830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:40.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.718+0000 7fa334800640 1 --2- 172.21.15.47:0/1750657867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa32c0d2650 0x7fa32c0cb830 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00c430 tx=0x7fa31c00c8f0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.718+0000 7fa334800640 1 -- 172.21.15.47:0/1750657867 shutdown_connections 2024-02-19T15:28:40.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.718+0000 7fa334800640 1 --2- 172.21.15.47:0/1750657867 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa310066e40 0x7fa310069300 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.718+0000 7fa334800640 1 --2- 172.21.15.47:0/1750657867 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa32c15e280 0x7fa32c157fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.718+0000 7fa334800640 1 --2- 172.21.15.47:0/1750657867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa32c0d2650 0x7fa32c0cb830 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.718+0000 7fa334800640 1 --2- 172.21.15.47:0/1750657867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa32c0ce3b0 0x7fa32c1530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.718+0000 7fa334800640 1 -- 172.21.15.47:0/1750657867 >> 172.21.15.47:0/1750657867 conn(0x7fa32c0d1b30 msgr2=0x7fa32c1563c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:40.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.718+0000 7fa334800640 1 -- 172.21.15.47:0/1750657867 shutdown_connections 2024-02-19T15:28:40.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.718+0000 7fa334800640 1 -- 172.21.15.47:0/1750657867 wait complete. 2024-02-19T15:28:40.737 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:40.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.878+0000 7f4875200640 1 Processor -- start 2024-02-19T15:28:40.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.878+0000 7f4875200640 1 -- start start 2024-02-19T15:28:40.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.878+0000 7f4875200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f487015bcd0 0x7f48700cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:40.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.878+0000 7f4875200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48700cc700 con 0x7f48700cedb0 2024-02-19T15:28:40.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.878+0000 7f4875200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48700cf300 con 0x7f48700d0360 2024-02-19T15:28:40.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.878+0000 7f4875200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48700cf5e0 con 0x7f487015bcd0 2024-02-19T15:28:40.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.878+0000 7f4867e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f48700cedb0 0x7f48700cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43706/0 (socket says 172.21.15.47:43706) 2024-02-19T15:28:40.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.878+0000 7f4867e00640 1 -- 172.21.15.47:0/274728629 learned_addr learned my addr 172.21.15.47:0/274728629 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:40.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.878+0000 7f486de00640 1 --2- 172.21.15.47:0/274728629 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f487015bcd0 0x7f48700cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:40.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.878+0000 7f486d400640 1 -- 172.21.15.47:0/274728629 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985018550 0 0) 0x7f48700cc700 con 0x7f48700cedb0 2024-02-19T15:28:40.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.878+0000 7f486d400640 1 -- 172.21.15.47:0/274728629 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4854003530 con 0x7f48700cedb0 2024-02-19T15:28:40.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.878+0000 7f486d400640 1 -- 172.21.15.47:0/274728629 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395109239 0 0) 0x7f48700cf300 con 0x7f48700d0360 2024-02-19T15:28:40.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.878+0000 7f486d400640 1 -- 172.21.15.47:0/274728629 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48700cc700 con 0x7f48700d0360 2024-02-19T15:28:40.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.878+0000 7f486d400640 1 -- 172.21.15.47:0/274728629 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1354748809 0 0) 0x7f4854003530 con 0x7f48700cedb0 2024-02-19T15:28:40.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.878+0000 7f486de00640 1 -- 172.21.15.47:0/274728629 >> v1:172.21.15.116:6789/0 conn(0x7f48700d0360 legacy=0x7f487015d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:40.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.878+0000 7f486de00640 1 -- 172.21.15.47:0/274728629 >> v1:172.21.15.47:6789/0 conn(0x7f48700cedb0 legacy=0x7f48700cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:40.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.878+0000 7f486de00640 1 -- 172.21.15.47:0/274728629 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48700cc9a0 con 0x7f487015bcd0 2024-02-19T15:28:40.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.882+0000 7f486de00640 1 --2- 172.21.15.47:0/274728629 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f487015bcd0 0x7f48700cd480 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f48600097f0 tx=0x7f486001c3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=559a4c77b3bce118 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:40.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.882+0000 7f486d400640 1 -- 172.21.15.47:0/274728629 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4858001680 con 0x7f487015bcd0 2024-02-19T15:28:40.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.882+0000 7f486d400640 1 -- 172.21.15.47:0/274728629 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48600a7050 con 0x7f487015bcd0 2024-02-19T15:28:40.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.882+0000 7f486d400640 1 -- 172.21.15.47:0/274728629 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860004250 con 0x7f487015bcd0 2024-02-19T15:28:40.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.882+0000 7f4875200640 1 -- 172.21.15.47:0/274728629 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f487015bcd0 msgr2=0x7f48700cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:40.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.882+0000 7f4875200640 1 --2- 172.21.15.47:0/274728629 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f487015bcd0 0x7f48700cd480 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f48600097f0 tx=0x7f486001c3d0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.882+0000 7f4875200640 1 -- 172.21.15.47:0/274728629 shutdown_connections 2024-02-19T15:28:40.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.882+0000 7f4875200640 1 --2- 172.21.15.47:0/274728629 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f487015bcd0 0x7f48700cd480 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.882+0000 7f4875200640 1 -- 172.21.15.47:0/274728629 >> 172.21.15.47:0/274728629 conn(0x7f48700d1bf0 msgr2=0x7f48700d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:40.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.882+0000 7f4875200640 1 -- 172.21.15.47:0/274728629 shutdown_connections 2024-02-19T15:28:40.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.882+0000 7f4875200640 1 -- 172.21.15.47:0/274728629 wait complete. 2024-02-19T15:28:40.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f4875200640 1 Processor -- start 2024-02-19T15:28:40.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f4875200640 1 -- start start 2024-02-19T15:28:40.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f4875200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48700cedb0 0x7f487006cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:40.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f4875200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48700d0360 0x7f4870153b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:40.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f4875200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f487015bcd0 0x7f4870157e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:40.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f4875200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48701546c0 con 0x7f487015bcd0 2024-02-19T15:28:40.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f4875200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48700c9d10 con 0x7f48700cedb0 2024-02-19T15:28:40.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f4875200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487006d3e0 con 0x7f48700d0360 2024-02-19T15:28:40.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f486e800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f487015bcd0 0x7f4870157e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:40.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f486e800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f487015bcd0 0x7f4870157e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60098/0 (socket says 172.21.15.47:60098) 2024-02-19T15:28:40.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f486e800640 1 -- 172.21.15.47:0/926202942 learned_addr learned my addr 172.21.15.47:0/926202942 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:40.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f486de00640 1 --2- 172.21.15.47:0/926202942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48700cedb0 0x7f487006cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:40.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f4867e00640 1 --2- 172.21.15.47:0/926202942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48700d0360 0x7f4870153b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:40.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f486e800640 1 -- 172.21.15.47:0/926202942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48700d0360 msgr2=0x7f4870153b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:40.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f486e800640 1 --2- 172.21.15.47:0/926202942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48700d0360 0x7f4870153b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f486e800640 1 -- 172.21.15.47:0/926202942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48700cedb0 msgr2=0x7f487006cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:40.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f486e800640 1 --2- 172.21.15.47:0/926202942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48700cedb0 0x7f487006cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:40.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f486e800640 1 -- 172.21.15.47:0/926202942 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4860009420 con 0x7f487015bcd0 2024-02-19T15:28:40.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f486de00640 1 --2- 172.21.15.47:0/926202942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48700cedb0 0x7f487006cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:40.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f4867e00640 1 --2- 172.21.15.47:0/926202942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48700d0360 0x7f4870153b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:40.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f486e800640 1 --2- 172.21.15.47:0/926202942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f487015bcd0 0x7f4870157e70 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f486800c7d0 tx=0x7f486800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:40.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f4866000640 1 -- 172.21.15.47:0/926202942 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4868003b80 con 0x7f487015bcd0 2024-02-19T15:28:40.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f4866000640 1 -- 172.21.15.47:0/926202942 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4868003ce0 con 0x7f487015bcd0 2024-02-19T15:28:40.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f4875200640 1 -- 172.21.15.47:0/926202942 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48700cdb40 con 0x7f487015bcd0 2024-02-19T15:28:40.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f4866000640 1 -- 172.21.15.47:0/926202942 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486800aa90 con 0x7f487015bcd0 2024-02-19T15:28:40.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.886+0000 7f4875200640 1 -- 172.21.15.47:0/926202942 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f487006d930 con 0x7f487015bcd0 2024-02-19T15:28:40.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.890+0000 7f4875200640 1 -- 172.21.15.47:0/926202942 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48700c9ea0 con 0x7f487015bcd0 2024-02-19T15:28:40.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.890+0000 7f4866000640 1 -- 172.21.15.47:0/926202942 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f486800abf0 con 0x7f487015bcd0 2024-02-19T15:28:40.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.890+0000 7f4866000640 1 --2- 172.21.15.47:0/926202942 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f484c066e40 0x7f484c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:40.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.890+0000 7f4866000640 1 -- 172.21.15.47:0/926202942 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f4868084a40 con 0x7f487015bcd0 2024-02-19T15:28:40.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.890+0000 7f486de00640 1 --2- 172.21.15.47:0/926202942 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f484c066e40 0x7f484c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:40.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.890+0000 7f486de00640 1 --2- 172.21.15.47:0/926202942 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f484c066e40 0x7f484c069300 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f48600ad000 tx=0x7f48600a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:40.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:40.898+0000 7f4866000640 1 -- 172.21.15.47:0/926202942 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f4868051730 con 0x7f487015bcd0 2024-02-19T15:28:41.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.098+0000 7f4875200640 1 -- 172.21.15.47:0/926202942 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4870075360 con 0x7f487015bcd0 2024-02-19T15:28:41.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.098+0000 7f4866000640 1 -- 172.21.15.47:0/926202942 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f4868054df0 con 0x7f487015bcd0 2024-02-19T15:28:41.099 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:41.099 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:41.099 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:41.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.102+0000 7f4875200640 1 -- 172.21.15.47:0/926202942 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f484c066e40 msgr2=0x7f484c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:41.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.102+0000 7f4875200640 1 --2- 172.21.15.47:0/926202942 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f484c066e40 0x7f484c069300 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f48600ad000 tx=0x7f48600a6000 comp rx=0 tx=0).stop 2024-02-19T15:28:41.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.102+0000 7f4875200640 1 -- 172.21.15.47:0/926202942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f487015bcd0 msgr2=0x7f4870157e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:41.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.102+0000 7f4875200640 1 --2- 172.21.15.47:0/926202942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f487015bcd0 0x7f4870157e70 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f486800c7d0 tx=0x7f486800cc90 comp rx=0 tx=0).stop 2024-02-19T15:28:41.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.102+0000 7f4875200640 1 -- 172.21.15.47:0/926202942 shutdown_connections 2024-02-19T15:28:41.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.102+0000 7f4875200640 1 --2- 172.21.15.47:0/926202942 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f484c066e40 0x7f484c069300 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.102+0000 7f4875200640 1 --2- 172.21.15.47:0/926202942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f487015bcd0 0x7f4870157e70 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.102+0000 7f4875200640 1 --2- 172.21.15.47:0/926202942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48700d0360 0x7f4870153b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.102+0000 7f4875200640 1 --2- 172.21.15.47:0/926202942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48700cedb0 0x7f487006cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.102+0000 7f4875200640 1 -- 172.21.15.47:0/926202942 >> 172.21.15.47:0/926202942 conn(0x7f48700d1bf0 msgr2=0x7f4870070310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:41.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.102+0000 7f4875200640 1 -- 172.21.15.47:0/926202942 shutdown_connections 2024-02-19T15:28:41.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.102+0000 7f4875200640 1 -- 172.21.15.47:0/926202942 wait complete. 2024-02-19T15:28:41.124 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:28:41.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.262+0000 7f9961400640 1 Processor -- start 2024-02-19T15:28:41.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.262+0000 7f9961400640 1 -- start start 2024-02-19T15:28:41.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.266+0000 7f9961400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f995c1456a0 0x7f995c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:41.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.266+0000 7f9961400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f995c06b5c0 0x7f995c06dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:41.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.266+0000 7f9961400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f995c145a80 con 0x7f995c0cf4a0 2024-02-19T15:28:41.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.266+0000 7f9961400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f995c0d03c0 con 0x7f995c06b5c0 2024-02-19T15:28:41.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.266+0000 7f9961400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f995c0cf900 con 0x7f995c1456a0 2024-02-19T15:28:41.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.266+0000 7f995b200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f995c06b5c0 0x7f995c06dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:41.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.266+0000 7f995b200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f995c06b5c0 0x7f995c06dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60114/0 (socket says 172.21.15.47:60114) 2024-02-19T15:28:41.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.266+0000 7f995b200640 1 -- 172.21.15.47:0/937873370 learned_addr learned my addr 172.21.15.47:0/937873370 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:41.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.266+0000 7f995a800640 1 --2- 172.21.15.47:0/937873370 >> v2:172.21.15.116:3300/0 conn(0x7f995c1456a0 0x7f995c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:41.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.266+0000 7f995b200640 1 -- 172.21.15.47:0/937873370 >> v2:172.21.15.116:3300/0 conn(0x7f995c1456a0 msgr2=0x7f995c0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:41.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.266+0000 7f995b200640 1 --2- 172.21.15.47:0/937873370 >> v2:172.21.15.116:3300/0 conn(0x7f995c1456a0 0x7f995c0cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.266+0000 7f995b200640 1 -- 172.21.15.47:0/937873370 >> v1:172.21.15.116:6789/0 conn(0x7f995c0cf4a0 legacy=0x7f995c06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:41.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.266+0000 7f995b200640 1 -- 172.21.15.47:0/937873370 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f995c0d1880 con 0x7f995c06b5c0 2024-02-19T15:28:41.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.266+0000 7f995b200640 1 --2- 172.21.15.47:0/937873370 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f995c06b5c0 0x7f995c06dd60 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f995c0b6b80 tx=0x7f995401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2773d533adc8b0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:41.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9959e00640 1 -- 172.21.15.47:0/937873370 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99540ab610 con 0x7f995c06b5c0 2024-02-19T15:28:41.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9959e00640 1 -- 172.21.15.47:0/937873370 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99540ab770 con 0x7f995c06b5c0 2024-02-19T15:28:41.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9959e00640 1 -- 172.21.15.47:0/937873370 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99540abb00 con 0x7f995c06b5c0 2024-02-19T15:28:41.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9961400640 1 -- 172.21.15.47:0/937873370 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f995c06b5c0 msgr2=0x7f995c06dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:41.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9961400640 1 --2- 172.21.15.47:0/937873370 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f995c06b5c0 0x7f995c06dd60 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f995c0b6b80 tx=0x7f995401c520 comp rx=0 tx=0).stop 2024-02-19T15:28:41.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9961400640 1 -- 172.21.15.47:0/937873370 shutdown_connections 2024-02-19T15:28:41.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9961400640 1 --2- 172.21.15.47:0/937873370 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f995c06b5c0 0x7f995c06dd60 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9961400640 1 --2- 172.21.15.47:0/937873370 >> v2:172.21.15.116:3300/0 conn(0x7f995c1456a0 0x7f995c0cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9961400640 1 -- 172.21.15.47:0/937873370 >> 172.21.15.47:0/937873370 conn(0x7f995c0d0ad0 msgr2=0x7f995c0d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:41.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9961400640 1 -- 172.21.15.47:0/937873370 shutdown_connections 2024-02-19T15:28:41.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9961400640 1 -- 172.21.15.47:0/937873370 wait complete. 2024-02-19T15:28:41.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9961400640 1 Processor -- start 2024-02-19T15:28:41.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9961400640 1 -- start start 2024-02-19T15:28:41.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9961400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f995c06b5c0 0x7f995c157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:41.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9961400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f995c0cf4a0 0x7f995c155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:41.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9961400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f995c1456a0 0x7f995c15c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:41.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9961400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995c15ba50 con 0x7f995c0cf4a0 2024-02-19T15:28:41.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9961400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995c159ab0 con 0x7f995c06b5c0 2024-02-19T15:28:41.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9961400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f995c159d20 con 0x7f995c1456a0 2024-02-19T15:28:41.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9951e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f995c0cf4a0 0x7f995c155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:41.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f995a800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f995c06b5c0 0x7f995c157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:41.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9951e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f995c0cf4a0 0x7f995c155700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60124/0 (socket says 172.21.15.47:60124) 2024-02-19T15:28:41.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f9951e00640 1 -- 172.21.15.47:0/3679724061 learned_addr learned my addr 172.21.15.47:0/3679724061 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:41.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.270+0000 7f995b200640 1 --2- 172.21.15.47:0/3679724061 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f995c1456a0 0x7f995c15c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:41.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.274+0000 7f9951e00640 1 -- 172.21.15.47:0/3679724061 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f995c1456a0 msgr2=0x7f995c15c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:41.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.274+0000 7f9951e00640 1 --2- 172.21.15.47:0/3679724061 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f995c1456a0 0x7f995c15c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.274+0000 7f9951e00640 1 -- 172.21.15.47:0/3679724061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f995c06b5c0 msgr2=0x7f995c157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:41.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.274+0000 7f9951e00640 1 --2- 172.21.15.47:0/3679724061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f995c06b5c0 0x7f995c157160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.274+0000 7f9951e00640 1 -- 172.21.15.47:0/3679724061 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99540094d0 con 0x7f995c0cf4a0 2024-02-19T15:28:41.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.274+0000 7f9951e00640 1 --2- 172.21.15.47:0/3679724061 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f995c0cf4a0 0x7f995c155700 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f994400c7c0 tx=0x7f994400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:41.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.274+0000 7f9953400640 1 -- 172.21.15.47:0/3679724061 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9944006d60 con 0x7f995c0cf4a0 2024-02-19T15:28:41.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.274+0000 7f9953400640 1 -- 172.21.15.47:0/3679724061 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9944003980 con 0x7f995c0cf4a0 2024-02-19T15:28:41.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.274+0000 7f9953400640 1 -- 172.21.15.47:0/3679724061 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9944003d30 con 0x7f995c0cf4a0 2024-02-19T15:28:41.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.274+0000 7f9961400640 1 -- 172.21.15.47:0/3679724061 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f995c15bca0 con 0x7f995c0cf4a0 2024-02-19T15:28:41.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.274+0000 7f9961400640 1 -- 172.21.15.47:0/3679724061 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f995c157910 con 0x7f995c0cf4a0 2024-02-19T15:28:41.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.274+0000 7f9961400640 1 -- 172.21.15.47:0/3679724061 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f995c075360 con 0x7f995c0cf4a0 2024-02-19T15:28:41.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.274+0000 7f9953400640 1 -- 172.21.15.47:0/3679724061 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f994400a850 con 0x7f995c0cf4a0 2024-02-19T15:28:41.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.274+0000 7f9953400640 1 --2- 172.21.15.47:0/3679724061 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9938066e40 0x7f9938069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:41.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.274+0000 7f9953400640 1 -- 172.21.15.47:0/3679724061 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f9944084610 con 0x7f995c0cf4a0 2024-02-19T15:28:41.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.278+0000 7f995a800640 1 --2- 172.21.15.47:0/3679724061 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9938066e40 0x7f9938069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:41.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.278+0000 7f995a800640 1 --2- 172.21.15.47:0/3679724061 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9938066e40 0x7f9938069300 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f994c009ac0 tx=0x7f994c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:41.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.282+0000 7f9953400640 1 -- 172.21.15.47:0/3679724061 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f99440513b0 con 0x7f995c0cf4a0 2024-02-19T15:28:41.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.446+0000 7f9961400640 1 -- 172.21.15.47:0/3679724061 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f995c15c150 con 0x7f995c0cf4a0 2024-02-19T15:28:41.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.450+0000 7f9953400640 1 -- 172.21.15.47:0/3679724061 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+16990 (secure 0 0 0) 0x7f9944054a70 con 0x7f995c0cf4a0 2024-02-19T15:28:41.449 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:41.450 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":114,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:27:47.287065+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:27:19.839144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-02-19T15:27:20.405924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.36","mappings":[{"from":0,"to":5}]},{"pgid":"17.3f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6837/819751716":"2024-02-20T15:27:39.101989+0000","172.21.15.116:6835/819751716":"2024-02-20T15:27:39.101989+0000","172.21.15.47:6834/897369244":"2024-02-20T15:27:38.263490+0000","172.21.15.47:6832/897369244":"2024-02-20T15:27:38.263490+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:28:41.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.450+0000 7f9961400640 1 -- 172.21.15.47:0/3679724061 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9938066e40 msgr2=0x7f9938069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:41.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.450+0000 7f9961400640 1 --2- 172.21.15.47:0/3679724061 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9938066e40 0x7f9938069300 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f994c009ac0 tx=0x7f994c009190 comp rx=0 tx=0).stop 2024-02-19T15:28:41.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.450+0000 7f9961400640 1 -- 172.21.15.47:0/3679724061 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f995c0cf4a0 msgr2=0x7f995c155700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:41.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.450+0000 7f9961400640 1 --2- 172.21.15.47:0/3679724061 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f995c0cf4a0 0x7f995c155700 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f994400c7c0 tx=0x7f994400cc80 comp rx=0 tx=0).stop 2024-02-19T15:28:41.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.450+0000 7f9961400640 1 -- 172.21.15.47:0/3679724061 shutdown_connections 2024-02-19T15:28:41.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.450+0000 7f9961400640 1 --2- 172.21.15.47:0/3679724061 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9938066e40 0x7f9938069300 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.450+0000 7f9961400640 1 --2- 172.21.15.47:0/3679724061 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f995c1456a0 0x7f995c15c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.450+0000 7f9961400640 1 --2- 172.21.15.47:0/3679724061 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f995c0cf4a0 0x7f995c155700 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.450+0000 7f9961400640 1 --2- 172.21.15.47:0/3679724061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f995c06b5c0 0x7f995c157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.450+0000 7f9961400640 1 -- 172.21.15.47:0/3679724061 >> 172.21.15.47:0/3679724061 conn(0x7f995c0d0ad0 msgr2=0x7f995c157dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:41.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.450+0000 7f9961400640 1 -- 172.21.15.47:0/3679724061 shutdown_connections 2024-02-19T15:28:41.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.450+0000 7f9961400640 1 -- 172.21.15.47:0/3679724061 wait complete. 2024-02-19T15:28:41.472 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T15:28:41.472 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:41.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.610+0000 7fb01d800640 1 Processor -- start 2024-02-19T15:28:41.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.610+0000 7fb01d800640 1 -- start start 2024-02-19T15:28:41.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.610+0000 7fb01d800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb01815bcd0 0x7fb0180cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:41.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.610+0000 7fb01d800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb0180d0360 0x7fb01815d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:41.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.610+0000 7fb01d800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0180cc700 con 0x7fb0180cedb0 2024-02-19T15:28:41.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.610+0000 7fb01d800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0180cf5e0 con 0x7fb0180d0360 2024-02-19T15:28:41.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.610+0000 7fb01d800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0180cf300 con 0x7fb01815bcd0 2024-02-19T15:28:41.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.614+0000 7fb017400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb0180d0360 0x7fb01815d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:41.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.614+0000 7fb017400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb0180d0360 0x7fb01815d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60126/0 (socket says 172.21.15.47:60126) 2024-02-19T15:28:41.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.614+0000 7fb017400640 1 -- 172.21.15.47:0/2878235033 learned_addr learned my addr 172.21.15.47:0/2878235033 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:41.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.614+0000 7fb016a00640 1 --2- 172.21.15.47:0/2878235033 >> v2:172.21.15.116:3300/0 conn(0x7fb01815bcd0 0x7fb0180cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:41.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.614+0000 7fb017400640 1 -- 172.21.15.47:0/2878235033 >> v2:172.21.15.116:3300/0 conn(0x7fb01815bcd0 msgr2=0x7fb0180cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:41.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.614+0000 7fb017400640 1 --2- 172.21.15.47:0/2878235033 >> v2:172.21.15.116:3300/0 conn(0x7fb01815bcd0 0x7fb0180cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.614+0000 7fb017400640 1 -- 172.21.15.47:0/2878235033 >> v1:172.21.15.116:6789/0 conn(0x7fb0180cedb0 legacy=0x7fb0180cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:41.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.614+0000 7fb017400640 1 -- 172.21.15.47:0/2878235033 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0180cc9a0 con 0x7fb0180d0360 2024-02-19T15:28:41.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.614+0000 7fb017400640 1 --2- 172.21.15.47:0/2878235033 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0180d0360 0x7fb01815d4f0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fb018071a60 tx=0x7fb01001c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18100bf52ac26fcd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:41.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.614+0000 7fb016000640 1 -- 172.21.15.47:0/2878235033 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0100a5880 con 0x7fb0180d0360 2024-02-19T15:28:41.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.614+0000 7fb016000640 1 -- 172.21.15.47:0/2878235033 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0100a59e0 con 0x7fb0180d0360 2024-02-19T15:28:41.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.614+0000 7fb016000640 1 -- 172.21.15.47:0/2878235033 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0100a5d70 con 0x7fb0180d0360 2024-02-19T15:28:41.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb01d800640 1 -- 172.21.15.47:0/2878235033 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0180d0360 msgr2=0x7fb01815d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:41.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb01d800640 1 --2- 172.21.15.47:0/2878235033 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0180d0360 0x7fb01815d4f0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fb018071a60 tx=0x7fb01001c9c0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb01d800640 1 -- 172.21.15.47:0/2878235033 shutdown_connections 2024-02-19T15:28:41.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb01d800640 1 --2- 172.21.15.47:0/2878235033 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0180d0360 0x7fb01815d4f0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb01d800640 1 --2- 172.21.15.47:0/2878235033 >> v2:172.21.15.116:3300/0 conn(0x7fb01815bcd0 0x7fb0180cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb01d800640 1 -- 172.21.15.47:0/2878235033 >> 172.21.15.47:0/2878235033 conn(0x7fb0180d1bf0 msgr2=0x7fb0180d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:41.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb01d800640 1 -- 172.21.15.47:0/2878235033 shutdown_connections 2024-02-19T15:28:41.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb01d800640 1 -- 172.21.15.47:0/2878235033 wait complete. 2024-02-19T15:28:41.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb01d800640 1 Processor -- start 2024-02-19T15:28:41.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb01d800640 1 -- start start 2024-02-19T15:28:41.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb01d800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0180cedb0 0x7fb0180c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:41.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb01d800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0180d0360 0x7fb0180b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:41.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb01d800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb01815bcd0 0x7fb0180b62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:41.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb01d800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180b7210 con 0x7fb0180cedb0 2024-02-19T15:28:41.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb01d800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180c7eb0 con 0x7fb0180d0360 2024-02-19T15:28:41.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb01d800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180b6b50 con 0x7fb01815bcd0 2024-02-19T15:28:41.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb016a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0180cedb0 0x7fb0180c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:41.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb016a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0180cedb0 0x7fb0180c78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60134/0 (socket says 172.21.15.47:60134) 2024-02-19T15:28:41.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb016a00640 1 -- 172.21.15.47:0/2520630071 learned_addr learned my addr 172.21.15.47:0/2520630071 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:41.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb00e000640 1 --2- 172.21.15.47:0/2520630071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0180d0360 0x7fb0180b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:41.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb017400640 1 --2- 172.21.15.47:0/2520630071 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb01815bcd0 0x7fb0180b62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:41.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb016a00640 1 -- 172.21.15.47:0/2520630071 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb01815bcd0 msgr2=0x7fb0180b62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:41.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb016a00640 1 --2- 172.21.15.47:0/2520630071 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb01815bcd0 0x7fb0180b62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb016a00640 1 -- 172.21.15.47:0/2520630071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0180d0360 msgr2=0x7fb0180b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:41.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb016a00640 1 --2- 172.21.15.47:0/2520630071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0180d0360 0x7fb0180b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.618+0000 7fb016a00640 1 -- 172.21.15.47:0/2520630071 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb010009b30 con 0x7fb0180cedb0 2024-02-19T15:28:41.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.622+0000 7fb016a00640 1 --2- 172.21.15.47:0/2520630071 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0180cedb0 0x7fb0180c78d0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fb00800e930 tx=0x7fb00800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:41.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.622+0000 7fb00f400640 1 -- 172.21.15.47:0/2520630071 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb008009690 con 0x7fb0180cedb0 2024-02-19T15:28:41.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.622+0000 7fb00f400640 1 -- 172.21.15.47:0/2520630071 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00800ca00 con 0x7fb0180cedb0 2024-02-19T15:28:41.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.622+0000 7fb00f400640 1 -- 172.21.15.47:0/2520630071 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00800cd70 con 0x7fb0180cedb0 2024-02-19T15:28:41.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.622+0000 7fb01d800640 1 -- 172.21.15.47:0/2520630071 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0180cdb40 con 0x7fb0180cedb0 2024-02-19T15:28:41.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.622+0000 7fb01d800640 1 -- 172.21.15.47:0/2520630071 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb01814d790 con 0x7fb0180cedb0 2024-02-19T15:28:41.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.622+0000 7fb01d800640 1 -- 172.21.15.47:0/2520630071 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb018075360 con 0x7fb0180cedb0 2024-02-19T15:28:41.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.622+0000 7fb00f400640 1 -- 172.21.15.47:0/2520630071 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb008013470 con 0x7fb0180cedb0 2024-02-19T15:28:41.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.622+0000 7fb00f400640 1 --2- 172.21.15.47:0/2520630071 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faff4066e90 0x7faff4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:41.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.622+0000 7fb00f400640 1 -- 172.21.15.47:0/2520630071 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fb008084c90 con 0x7fb0180cedb0 2024-02-19T15:28:41.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.622+0000 7fb00e000640 1 --2- 172.21.15.47:0/2520630071 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faff4066e90 0x7faff4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:41.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.626+0000 7fb00e000640 1 --2- 172.21.15.47:0/2520630071 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faff4066e90 0x7faff4069350 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fb0180cb890 tx=0x7fb000009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:41.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.626+0000 7fb00f400640 1 -- 172.21.15.47:0/2520630071 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb008051b70 con 0x7fb0180cedb0 2024-02-19T15:28:41.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.822+0000 7fb01d800640 1 -- 172.21.15.47:0/2520630071 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb01815cdf0 con 0x7fb0180cedb0 2024-02-19T15:28:41.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.822+0000 7fb00f400640 1 -- 172.21.15.47:0/2520630071 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7fb008017090 con 0x7fb0180cedb0 2024-02-19T15:28:41.823 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:41.823 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:41.823 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.822+0000 7fb01d800640 1 -- 172.21.15.47:0/2520630071 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faff4066e90 msgr2=0x7faff4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.822+0000 7fb01d800640 1 --2- 172.21.15.47:0/2520630071 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faff4066e90 0x7faff4069350 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fb0180cb890 tx=0x7fb000009040 comp rx=0 tx=0).stop 2024-02-19T15:28:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.826+0000 7fb01d800640 1 -- 172.21.15.47:0/2520630071 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0180cedb0 msgr2=0x7fb0180c78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.826+0000 7fb01d800640 1 --2- 172.21.15.47:0/2520630071 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0180cedb0 0x7fb0180c78d0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fb00800e930 tx=0x7fb00800edf0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.826+0000 7fb01d800640 1 -- 172.21.15.47:0/2520630071 shutdown_connections 2024-02-19T15:28:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.826+0000 7fb01d800640 1 --2- 172.21.15.47:0/2520630071 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faff4066e90 0x7faff4069350 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.826+0000 7fb01d800640 1 --2- 172.21.15.47:0/2520630071 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb01815bcd0 0x7fb0180b62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.826+0000 7fb01d800640 1 --2- 172.21.15.47:0/2520630071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0180d0360 0x7fb0180b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.826+0000 7fb01d800640 1 --2- 172.21.15.47:0/2520630071 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0180cedb0 0x7fb0180c78d0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.826+0000 7fb01d800640 1 -- 172.21.15.47:0/2520630071 >> 172.21.15.47:0/2520630071 conn(0x7fb0180d1bf0 msgr2=0x7fb018156a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:41.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.826+0000 7fb01d800640 1 -- 172.21.15.47:0/2520630071 shutdown_connections 2024-02-19T15:28:41.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.826+0000 7fb01d800640 1 -- 172.21.15.47:0/2520630071 wait complete. 2024-02-19T15:28:41.844 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:41.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.982+0000 7f35cb600640 1 Processor -- start 2024-02-19T15:28:41.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.982+0000 7f35cb600640 1 -- start start 2024-02-19T15:28:41.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.982+0000 7f35cb600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f35c415f590 0x7f35c40d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:41.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.982+0000 7f35cb600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f35c40b0a80 0x7f35c40b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:41.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.982+0000 7f35cb600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c40b0280 con 0x7f35c40b1a10 2024-02-19T15:28:41.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.982+0000 7f35cb600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35c40cc530 con 0x7f35c40b0a80 2024-02-19T15:28:41.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.982+0000 7f35cb600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35c40b1e20 con 0x7f35c415f590 2024-02-19T15:28:41.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.986+0000 7f35c3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f35c40b0a80 0x7f35c40b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:41.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.986+0000 7f35c3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f35c40b0a80 0x7f35c40b14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60138/0 (socket says 172.21.15.47:60138) 2024-02-19T15:28:41.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.986+0000 7f35c3e00640 1 -- 172.21.15.47:0/1371375435 learned_addr learned my addr 172.21.15.47:0/1371375435 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:41.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.986+0000 7f35c8c00640 1 --2- 172.21.15.47:0/1371375435 >> v2:172.21.15.116:3300/0 conn(0x7f35c415f590 0x7f35c40d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:41.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.986+0000 7f35c3e00640 1 -- 172.21.15.47:0/1371375435 >> v2:172.21.15.116:3300/0 conn(0x7f35c415f590 msgr2=0x7f35c40d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:41.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.986+0000 7f35c3e00640 1 --2- 172.21.15.47:0/1371375435 >> v2:172.21.15.116:3300/0 conn(0x7f35c415f590 0x7f35c40d2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.986+0000 7f35c3e00640 1 -- 172.21.15.47:0/1371375435 >> v1:172.21.15.116:6789/0 conn(0x7f35c40b1a10 legacy=0x7f35c40ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:41.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.986+0000 7f35c3e00640 1 -- 172.21.15.47:0/1371375435 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c40b04d0 con 0x7f35c40b0a80 2024-02-19T15:28:41.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.986+0000 7f35c3e00640 1 --2- 172.21.15.47:0/1371375435 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35c40b0a80 0x7f35c40b14d0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f35c406be80 tx=0x7f35b001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f30565bc02a6c3d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:41.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.986+0000 7f35c3400640 1 -- 172.21.15.47:0/1371375435 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b00aa650 con 0x7f35c40b0a80 2024-02-19T15:28:41.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.986+0000 7f35c3400640 1 -- 172.21.15.47:0/1371375435 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b00aa7b0 con 0x7f35c40b0a80 2024-02-19T15:28:41.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.986+0000 7f35c3400640 1 -- 172.21.15.47:0/1371375435 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b00aab40 con 0x7f35c40b0a80 2024-02-19T15:28:41.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35cb600640 1 -- 172.21.15.47:0/1371375435 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35c40b0a80 msgr2=0x7f35c40b14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:41.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35cb600640 1 --2- 172.21.15.47:0/1371375435 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35c40b0a80 0x7f35c40b14d0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f35c406be80 tx=0x7f35b001c4f0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35cb600640 1 -- 172.21.15.47:0/1371375435 shutdown_connections 2024-02-19T15:28:41.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35cb600640 1 --2- 172.21.15.47:0/1371375435 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35c40b0a80 0x7f35c40b14d0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35cb600640 1 --2- 172.21.15.47:0/1371375435 >> v2:172.21.15.116:3300/0 conn(0x7f35c415f590 0x7f35c40d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35cb600640 1 -- 172.21.15.47:0/1371375435 >> 172.21.15.47:0/1371375435 conn(0x7f35c40d1b70 msgr2=0x7f35c40d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:41.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35cb600640 1 -- 172.21.15.47:0/1371375435 shutdown_connections 2024-02-19T15:28:41.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35cb600640 1 -- 172.21.15.47:0/1371375435 wait complete. 2024-02-19T15:28:41.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35cb600640 1 Processor -- start 2024-02-19T15:28:41.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35cb600640 1 -- start start 2024-02-19T15:28:41.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35cb600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35c40b0a80 0x7f35c40c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:41.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35cb600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35c40b1a10 0x7f35c415cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:41.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35cb600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35c415f590 0x7f35c415d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:41.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35cb600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c415a3b0 con 0x7f35c415f590 2024-02-19T15:28:41.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35cb600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c4159b10 con 0x7f35c40b1a10 2024-02-19T15:28:41.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35cb600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c4159df0 con 0x7f35c40b0a80 2024-02-19T15:28:41.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35c9600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35c415f590 0x7f35c415d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:41.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35c9600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35c415f590 0x7f35c415d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60148/0 (socket says 172.21.15.47:60148) 2024-02-19T15:28:41.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35c9600640 1 -- 172.21.15.47:0/2366248458 learned_addr learned my addr 172.21.15.47:0/2366248458 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:41.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35c8c00640 1 --2- 172.21.15.47:0/2366248458 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35c40b0a80 0x7f35c40c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:41.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35c3e00640 1 --2- 172.21.15.47:0/2366248458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35c40b1a10 0x7f35c415cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:41.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35c9600640 1 -- 172.21.15.47:0/2366248458 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35c40b0a80 msgr2=0x7f35c40c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:41.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.990+0000 7f35c9600640 1 --2- 172.21.15.47:0/2366248458 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35c40b0a80 0x7f35c40c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.994+0000 7f35c9600640 1 -- 172.21.15.47:0/2366248458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35c40b1a10 msgr2=0x7f35c415cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:41.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.994+0000 7f35c9600640 1 --2- 172.21.15.47:0/2366248458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35c40b1a10 0x7f35c415cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:41.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.994+0000 7f35c9600640 1 -- 172.21.15.47:0/2366248458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b00094d0 con 0x7f35c415f590 2024-02-19T15:28:41.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.994+0000 7f35c9600640 1 --2- 172.21.15.47:0/2366248458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35c415f590 0x7f35c415d0b0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f35bc00c730 tx=0x7f35bc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:41.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.994+0000 7f35c0c00640 1 -- 172.21.15.47:0/2366248458 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35bc003b80 con 0x7f35c415f590 2024-02-19T15:28:41.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.994+0000 7f35c0c00640 1 -- 172.21.15.47:0/2366248458 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35bc003ce0 con 0x7f35c415f590 2024-02-19T15:28:41.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.994+0000 7f35c0c00640 1 -- 172.21.15.47:0/2366248458 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35bc00aa30 con 0x7f35c415f590 2024-02-19T15:28:41.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.994+0000 7f35cb600640 1 -- 172.21.15.47:0/2366248458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c415a0d0 con 0x7f35c415f590 2024-02-19T15:28:41.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.994+0000 7f35cb600640 1 -- 172.21.15.47:0/2366248458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c4159730 con 0x7f35c415f590 2024-02-19T15:28:41.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.994+0000 7f35cb600640 1 -- 172.21.15.47:0/2366248458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c4075360 con 0x7f35c415f590 2024-02-19T15:28:41.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.994+0000 7f35c0c00640 1 -- 172.21.15.47:0/2366248458 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f35bc00ab90 con 0x7f35c415f590 2024-02-19T15:28:41.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.994+0000 7f35c0c00640 1 --2- 172.21.15.47:0/2366248458 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f35a4066e40 0x7f35a4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:41.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.994+0000 7f35c0c00640 1 -- 172.21.15.47:0/2366248458 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f35bc084860 con 0x7f35c415f590 2024-02-19T15:28:41.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.994+0000 7f35c8c00640 1 --2- 172.21.15.47:0/2366248458 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f35a4066e40 0x7f35a4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:41.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.998+0000 7f35c8c00640 1 --2- 172.21.15.47:0/2366248458 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f35a4066e40 0x7f35a4069300 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f35b8009b60 tx=0x7f35b8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:42.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:41.998+0000 7f35c0c00640 1 -- 172.21.15.47:0/2366248458 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f35bc051570 con 0x7f35c415f590 2024-02-19T15:28:42.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.198+0000 7f35cb600640 1 -- 172.21.15.47:0/2366248458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35c40c5a10 con 0x7f35c415f590 2024-02-19T15:28:42.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.198+0000 7f35c0c00640 1 -- 172.21.15.47:0/2366248458 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f35bc054c30 con 0x7f35c415f590 2024-02-19T15:28:42.201 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:42.201 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:27:21.415321+0000","modified":"2024-02-19T15:27:41.129893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.47:6835/3636833601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":3636833601},{"type":"v1","addr":"172.21.15.47:6835","nonce":3636833601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T15:28:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 82 2024-02-19T15:28:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.202+0000 7f35cb600640 1 -- 172.21.15.47:0/2366248458 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f35a4066e40 msgr2=0x7f35a4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.202+0000 7f35cb600640 1 --2- 172.21.15.47:0/2366248458 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f35a4066e40 0x7f35a4069300 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f35b8009b60 tx=0x7f35b8009190 comp rx=0 tx=0).stop 2024-02-19T15:28:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.202+0000 7f35cb600640 1 -- 172.21.15.47:0/2366248458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35c415f590 msgr2=0x7f35c415d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.202+0000 7f35cb600640 1 --2- 172.21.15.47:0/2366248458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35c415f590 0x7f35c415d0b0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f35bc00c730 tx=0x7f35bc00cbf0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.202+0000 7f35cb600640 1 -- 172.21.15.47:0/2366248458 shutdown_connections 2024-02-19T15:28:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.202+0000 7f35cb600640 1 --2- 172.21.15.47:0/2366248458 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f35a4066e40 0x7f35a4069300 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.202+0000 7f35cb600640 1 --2- 172.21.15.47:0/2366248458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35c415f590 0x7f35c415d0b0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.202+0000 7f35cb600640 1 --2- 172.21.15.47:0/2366248458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35c40b1a10 0x7f35c415cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.202+0000 7f35cb600640 1 --2- 172.21.15.47:0/2366248458 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35c40b0a80 0x7f35c40c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.202+0000 7f35cb600640 1 -- 172.21.15.47:0/2366248458 >> 172.21.15.47:0/2366248458 conn(0x7f35c40d1b70 msgr2=0x7f35c40d1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.202+0000 7f35cb600640 1 -- 172.21.15.47:0/2366248458 shutdown_connections 2024-02-19T15:28:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.202+0000 7f35cb600640 1 -- 172.21.15.47:0/2366248458 wait complete. 2024-02-19T15:28:42.222 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:28:42.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.362+0000 7efccca00640 1 Processor -- start 2024-02-19T15:28:42.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.362+0000 7efccca00640 1 -- start start 2024-02-19T15:28:42.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.362+0000 7efccca00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efcc81476e0 0x7efcc80ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:42.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.362+0000 7efccca00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efcc80b33c0 0x7efcc8071950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:42.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.362+0000 7efccca00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc80af1b0 con 0x7efcc806c9a0 2024-02-19T15:28:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.362+0000 7efccca00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc8071530 con 0x7efcc80b33c0 2024-02-19T15:28:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.362+0000 7efccca00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc8070980 con 0x7efcc81476e0 2024-02-19T15:28:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.362+0000 7efcc6800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efcc80b33c0 0x7efcc8071950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.362+0000 7efcc6800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efcc80b33c0 0x7efcc8071950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60156/0 (socket says 172.21.15.47:60156) 2024-02-19T15:28:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.362+0000 7efcc6800640 1 -- 172.21.15.47:0/3441678844 learned_addr learned my addr 172.21.15.47:0/3441678844 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.362+0000 7efcc5e00640 1 --2- 172.21.15.47:0/3441678844 >> v2:172.21.15.116:3300/0 conn(0x7efcc81476e0 0x7efcc80ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.362+0000 7efcc6800640 1 -- 172.21.15.47:0/3441678844 >> v2:172.21.15.116:3300/0 conn(0x7efcc81476e0 msgr2=0x7efcc80ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.362+0000 7efcc6800640 1 --2- 172.21.15.47:0/3441678844 >> v2:172.21.15.116:3300/0 conn(0x7efcc81476e0 0x7efcc80ae730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.362+0000 7efcc6800640 1 -- 172.21.15.47:0/3441678844 >> v1:172.21.15.116:6789/0 conn(0x7efcc806c9a0 legacy=0x7efcc80b2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.362+0000 7efcc6800640 1 -- 172.21.15.47:0/3441678844 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc8070b90 con 0x7efcc80b33c0 2024-02-19T15:28:42.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.366+0000 7efcc6800640 1 --2- 172.21.15.47:0/3441678844 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcc80b33c0 0x7efcc8071950 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7efcc8070610 tx=0x7efcc001c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2d940401eb0bb44 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:42.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.366+0000 7efcc4a00640 1 -- 172.21.15.47:0/3441678844 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc00a5880 con 0x7efcc80b33c0 2024-02-19T15:28:42.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.366+0000 7efcc4a00640 1 -- 172.21.15.47:0/3441678844 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc00a59e0 con 0x7efcc80b33c0 2024-02-19T15:28:42.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.366+0000 7efcc4a00640 1 -- 172.21.15.47:0/3441678844 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc00a5d70 con 0x7efcc80b33c0 2024-02-19T15:28:42.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.366+0000 7efccca00640 1 -- 172.21.15.47:0/3441678844 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcc80b33c0 msgr2=0x7efcc8071950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:42.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.366+0000 7efccca00640 1 --2- 172.21.15.47:0/3441678844 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcc80b33c0 0x7efcc8071950 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7efcc8070610 tx=0x7efcc001c9c0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.366+0000 7efccca00640 1 -- 172.21.15.47:0/3441678844 shutdown_connections 2024-02-19T15:28:42.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.366+0000 7efccca00640 1 --2- 172.21.15.47:0/3441678844 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcc80b33c0 0x7efcc8071950 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.366+0000 7efccca00640 1 --2- 172.21.15.47:0/3441678844 >> v2:172.21.15.116:3300/0 conn(0x7efcc81476e0 0x7efcc80ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.366+0000 7efccca00640 1 -- 172.21.15.47:0/3441678844 >> 172.21.15.47:0/3441678844 conn(0x7efcc80b16a0 msgr2=0x7efcc80b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:42.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.366+0000 7efccca00640 1 -- 172.21.15.47:0/3441678844 shutdown_connections 2024-02-19T15:28:42.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.366+0000 7efccca00640 1 -- 172.21.15.47:0/3441678844 wait complete. 2024-02-19T15:28:42.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.370+0000 7efccca00640 1 Processor -- start 2024-02-19T15:28:42.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.370+0000 7efccca00640 1 -- start start 2024-02-19T15:28:42.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.370+0000 7efccca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcc806c9a0 0x7efcc815c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.370+0000 7efccca00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcc80b33c0 0x7efcc8142880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.370+0000 7efccca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcc81476e0 0x7efcc8142dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.370+0000 7efccca00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc815c840 con 0x7efcc81476e0 2024-02-19T15:28:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.370+0000 7efccca00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc813bf40 con 0x7efcc806c9a0 2024-02-19T15:28:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.370+0000 7efccca00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc813c1e0 con 0x7efcc80b33c0 2024-02-19T15:28:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.370+0000 7efcc6800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcc81476e0 0x7efcc8142dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.370+0000 7efcc6800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcc81476e0 0x7efcc8142dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60166/0 (socket says 172.21.15.47:60166) 2024-02-19T15:28:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.370+0000 7efcc6800640 1 -- 172.21.15.47:0/3319222740 learned_addr learned my addr 172.21.15.47:0/3319222740 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.374+0000 7efcc5e00640 1 --2- 172.21.15.47:0/3319222740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcc806c9a0 0x7efcc815c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.370+0000 7efcc5400640 1 --2- 172.21.15.47:0/3319222740 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcc80b33c0 0x7efcc8142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.374+0000 7efcc6800640 1 -- 172.21.15.47:0/3319222740 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcc80b33c0 msgr2=0x7efcc8142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.374+0000 7efcc6800640 1 --2- 172.21.15.47:0/3319222740 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcc80b33c0 0x7efcc8142880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.374+0000 7efcc6800640 1 -- 172.21.15.47:0/3319222740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcc806c9a0 msgr2=0x7efcc815c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:42.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.374+0000 7efcc6800640 1 --2- 172.21.15.47:0/3319222740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcc806c9a0 0x7efcc815c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.374+0000 7efcc6800640 1 -- 172.21.15.47:0/3319222740 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc0009b30 con 0x7efcc81476e0 2024-02-19T15:28:42.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.374+0000 7efcc6800640 1 --2- 172.21.15.47:0/3319222740 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcc81476e0 0x7efcc8142dc0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7efcc00af040 tx=0x7efcc0019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:42.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.374+0000 7efcbe000640 1 -- 172.21.15.47:0/3319222740 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc00b8670 con 0x7efcc81476e0 2024-02-19T15:28:42.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.374+0000 7efcbe000640 1 -- 172.21.15.47:0/3319222740 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc00b87d0 con 0x7efcc81476e0 2024-02-19T15:28:42.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.374+0000 7efccca00640 1 -- 172.21.15.47:0/3319222740 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc813c4c0 con 0x7efcc81476e0 2024-02-19T15:28:42.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.374+0000 7efcbe000640 1 -- 172.21.15.47:0/3319222740 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc00b8b80 con 0x7efcc81476e0 2024-02-19T15:28:42.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.374+0000 7efccca00640 1 -- 172.21.15.47:0/3319222740 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc813ca80 con 0x7efcc81476e0 2024-02-19T15:28:42.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.374+0000 7efccca00640 1 -- 172.21.15.47:0/3319222740 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc8075360 con 0x7efcc81476e0 2024-02-19T15:28:42.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.378+0000 7efcbe000640 1 -- 172.21.15.47:0/3319222740 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efcc00b8ce0 con 0x7efcc81476e0 2024-02-19T15:28:42.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.378+0000 7efcbe000640 1 --2- 172.21.15.47:0/3319222740 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efca8066e40 0x7efca8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:42.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.378+0000 7efcbe000640 1 -- 172.21.15.47:0/3319222740 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7efcc011fff0 con 0x7efcc81476e0 2024-02-19T15:28:42.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.378+0000 7efcc5e00640 1 --2- 172.21.15.47:0/3319222740 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efca8066e40 0x7efca8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:42.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.378+0000 7efcc5e00640 1 --2- 172.21.15.47:0/3319222740 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efca8066e40 0x7efca8069300 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7efcc81405f0 tx=0x7efcb8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:42.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.382+0000 7efcbe000640 1 -- 172.21.15.47:0/3319222740 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7efcc00ecf20 con 0x7efcc81476e0 2024-02-19T15:28:42.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.546+0000 7efccca00640 1 -- 172.21.15.47:0/3319222740 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efcc80b23e0 con 0x7efcc81476e0 2024-02-19T15:28:42.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.546+0000 7efcbe000640 1 -- 172.21.15.47:0/3319222740 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+16990 (secure 0 0 0) 0x7efcc00f05e0 con 0x7efcc81476e0 2024-02-19T15:28:42.548 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:42.548 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":114,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:27:47.287065+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:27:19.839144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-02-19T15:27:20.405924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.36","mappings":[{"from":0,"to":5}]},{"pgid":"17.3f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6837/819751716":"2024-02-20T15:27:39.101989+0000","172.21.15.116:6835/819751716":"2024-02-20T15:27:39.101989+0000","172.21.15.47:6834/897369244":"2024-02-20T15:27:38.263490+0000","172.21.15.47:6832/897369244":"2024-02-20T15:27:38.263490+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:28:42.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.550+0000 7efccca00640 1 -- 172.21.15.47:0/3319222740 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efca8066e40 msgr2=0x7efca8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:42.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.550+0000 7efccca00640 1 --2- 172.21.15.47:0/3319222740 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efca8066e40 0x7efca8069300 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7efcc81405f0 tx=0x7efcb8009190 comp rx=0 tx=0).stop 2024-02-19T15:28:42.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.550+0000 7efccca00640 1 -- 172.21.15.47:0/3319222740 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcc81476e0 msgr2=0x7efcc8142dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:42.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.550+0000 7efccca00640 1 --2- 172.21.15.47:0/3319222740 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcc81476e0 0x7efcc8142dc0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7efcc00af040 tx=0x7efcc0019760 comp rx=0 tx=0).stop 2024-02-19T15:28:42.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.550+0000 7efccca00640 1 -- 172.21.15.47:0/3319222740 shutdown_connections 2024-02-19T15:28:42.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.550+0000 7efccca00640 1 --2- 172.21.15.47:0/3319222740 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efca8066e40 0x7efca8069300 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.550+0000 7efccca00640 1 --2- 172.21.15.47:0/3319222740 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcc81476e0 0x7efcc8142dc0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.550+0000 7efccca00640 1 --2- 172.21.15.47:0/3319222740 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcc80b33c0 0x7efcc8142880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.550+0000 7efccca00640 1 --2- 172.21.15.47:0/3319222740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcc806c9a0 0x7efcc815c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.550+0000 7efccca00640 1 -- 172.21.15.47:0/3319222740 >> 172.21.15.47:0/3319222740 conn(0x7efcc80b16a0 msgr2=0x7efcc8141010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:42.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.550+0000 7efccca00640 1 -- 172.21.15.47:0/3319222740 shutdown_connections 2024-02-19T15:28:42.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.550+0000 7efccca00640 1 -- 172.21.15.47:0/3319222740 wait complete. 2024-02-19T15:28:42.569 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T15:28:42.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.710+0000 7fa533600640 1 Processor -- start 2024-02-19T15:28:42.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.710+0000 7fa533600640 1 -- start start 2024-02-19T15:28:42.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.710+0000 7fa533600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa52c06fcb0 0x7fa52c06aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:42.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.710+0000 7fa533600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa52c06b3f0 0x7fa52c0b3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:42.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.710+0000 7fa533600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa52c070350 con 0x7fa52c15ca70 2024-02-19T15:28:42.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.710+0000 7fa533600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa52c06b880 con 0x7fa52c06fcb0 2024-02-19T15:28:42.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.710+0000 7fa533600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c06d860 con 0x7fa52c06b3f0 2024-02-19T15:28:42.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.710+0000 7fa530c00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa52c15ca70 0x7fa52c163290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43708/0 (socket says 172.21.15.47:43708) 2024-02-19T15:28:42.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.710+0000 7fa530c00640 1 -- 172.21.15.47:0/2606108508 learned_addr learned my addr 172.21.15.47:0/2606108508 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:42.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.710+0000 7fa531600640 1 --2- 172.21.15.47:0/2606108508 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa52c06b3f0 0x7fa52c0b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:42.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.710+0000 7fa52be00640 1 --2- 172.21.15.47:0/2606108508 >> v2:172.21.15.116:3300/0 conn(0x7fa52c06fcb0 0x7fa52c06aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:42.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.710+0000 7fa52b400640 1 -- 172.21.15.47:0/2606108508 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2589484448 0 0) 0x7fa52c070350 con 0x7fa52c15ca70 2024-02-19T15:28:42.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.710+0000 7fa52b400640 1 -- 172.21.15.47:0/2606108508 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa514003530 con 0x7fa52c15ca70 2024-02-19T15:28:42.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.710+0000 7fa531600640 1 -- 172.21.15.47:0/2606108508 >> v2:172.21.15.116:3300/0 conn(0x7fa52c06fcb0 msgr2=0x7fa52c06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:42.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.710+0000 7fa531600640 1 --2- 172.21.15.47:0/2606108508 >> v2:172.21.15.116:3300/0 conn(0x7fa52c06fcb0 0x7fa52c06aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.710+0000 7fa531600640 1 -- 172.21.15.47:0/2606108508 >> v1:172.21.15.47:6789/0 conn(0x7fa52c15ca70 legacy=0x7fa52c163290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:42.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.710+0000 7fa52b400640 1 -- 172.21.15.47:0/2606108508 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3449261941 0 0) 0x7fa514003530 con 0x7fa52c15ca70 2024-02-19T15:28:42.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.710+0000 7fa531600640 1 -- 172.21.15.47:0/2606108508 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52c06e1e0 con 0x7fa52c06b3f0 2024-02-19T15:28:42.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.714+0000 7fa531600640 1 --2- 172.21.15.47:0/2606108508 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa52c06b3f0 0x7fa52c0b3940 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa524009d90 tx=0x7fa52401c770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=183cdc40264d06a9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:42.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.714+0000 7fa52b400640 1 -- 172.21.15.47:0/2606108508 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa520001680 con 0x7fa52c06b3f0 2024-02-19T15:28:42.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.714+0000 7fa52b400640 1 -- 172.21.15.47:0/2606108508 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5240a6410 con 0x7fa52c06b3f0 2024-02-19T15:28:42.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.714+0000 7fa52b400640 1 -- 172.21.15.47:0/2606108508 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5240a67a0 con 0x7fa52c06b3f0 2024-02-19T15:28:42.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.714+0000 7fa533600640 1 -- 172.21.15.47:0/2606108508 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa52c06b3f0 msgr2=0x7fa52c0b3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:42.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.714+0000 7fa533600640 1 --2- 172.21.15.47:0/2606108508 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa52c06b3f0 0x7fa52c0b3940 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa524009d90 tx=0x7fa52401c770 comp rx=0 tx=0).stop 2024-02-19T15:28:42.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.714+0000 7fa533600640 1 -- 172.21.15.47:0/2606108508 shutdown_connections 2024-02-19T15:28:42.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.714+0000 7fa533600640 1 --2- 172.21.15.47:0/2606108508 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa52c06b3f0 0x7fa52c0b3940 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.714+0000 7fa533600640 1 --2- 172.21.15.47:0/2606108508 >> v2:172.21.15.116:3300/0 conn(0x7fa52c06fcb0 0x7fa52c06aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.714+0000 7fa533600640 1 -- 172.21.15.47:0/2606108508 >> 172.21.15.47:0/2606108508 conn(0x7fa52c0d0ad0 msgr2=0x7fa52c0d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:42.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.714+0000 7fa533600640 1 -- 172.21.15.47:0/2606108508 shutdown_connections 2024-02-19T15:28:42.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.714+0000 7fa533600640 1 -- 172.21.15.47:0/2606108508 wait complete. 2024-02-19T15:28:42.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa533600640 1 Processor -- start 2024-02-19T15:28:42.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa533600640 1 -- start start 2024-02-19T15:28:42.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa533600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa52c06b3f0 0x7fa52c1616e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:42.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa533600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa52c06fcb0 0x7fa52c15b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:42.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa533600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa52c15ca70 0x7fa52c0ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:42.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa533600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c15abf0 con 0x7fa52c15ca70 2024-02-19T15:28:42.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa533600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c159fe0 con 0x7fa52c06fcb0 2024-02-19T15:28:42.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa533600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c15a2c0 con 0x7fa52c06b3f0 2024-02-19T15:28:42.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa531600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa52c15ca70 0x7fa52c0ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:42.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa531600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa52c15ca70 0x7fa52c0ccaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60174/0 (socket says 172.21.15.47:60174) 2024-02-19T15:28:42.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa531600640 1 -- 172.21.15.47:0/1134896783 learned_addr learned my addr 172.21.15.47:0/1134896783 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:42.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa52be00640 1 --2- 172.21.15.47:0/1134896783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa52c06fcb0 0x7fa52c15b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:42.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa530c00640 1 --2- 172.21.15.47:0/1134896783 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa52c06b3f0 0x7fa52c1616e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:42.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa531600640 1 -- 172.21.15.47:0/1134896783 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa52c06b3f0 msgr2=0x7fa52c1616e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:42.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa531600640 1 --2- 172.21.15.47:0/1134896783 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa52c06b3f0 0x7fa52c1616e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa531600640 1 -- 172.21.15.47:0/1134896783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa52c06fcb0 msgr2=0x7fa52c15b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:42.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa531600640 1 --2- 172.21.15.47:0/1134896783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa52c06fcb0 0x7fa52c15b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa531600640 1 -- 172.21.15.47:0/1134896783 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa524009a70 con 0x7fa52c15ca70 2024-02-19T15:28:42.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa531600640 1 --2- 172.21.15.47:0/1134896783 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa52c15ca70 0x7fa52c0ccaf0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fa524009d90 tx=0x7fa524019510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:42.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa528c00640 1 -- 172.21.15.47:0/1134896783 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5240b6070 con 0x7fa52c15ca70 2024-02-19T15:28:42.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa528c00640 1 -- 172.21.15.47:0/1134896783 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa524009450 con 0x7fa52c15ca70 2024-02-19T15:28:42.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa533600640 1 -- 172.21.15.47:0/1134896783 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa52c15a5a0 con 0x7fa52c15ca70 2024-02-19T15:28:42.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa528c00640 1 -- 172.21.15.47:0/1134896783 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5240bb6b0 con 0x7fa52c15ca70 2024-02-19T15:28:42.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.718+0000 7fa533600640 1 -- 172.21.15.47:0/1134896783 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa52c159c00 con 0x7fa52c15ca70 2024-02-19T15:28:42.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.722+0000 7fa533600640 1 -- 172.21.15.47:0/1134896783 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa52c15d770 con 0x7fa52c15ca70 2024-02-19T15:28:42.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.722+0000 7fa528c00640 1 -- 172.21.15.47:0/1134896783 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa5240b1450 con 0x7fa52c15ca70 2024-02-19T15:28:42.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.722+0000 7fa528c00640 1 --2- 172.21.15.47:0/1134896783 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa50c066e40 0x7fa50c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:42.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.722+0000 7fa528c00640 1 -- 172.21.15.47:0/1134896783 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fa52411f600 con 0x7fa52c15ca70 2024-02-19T15:28:42.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.722+0000 7fa530c00640 1 --2- 172.21.15.47:0/1134896783 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa50c066e40 0x7fa50c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:42.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.722+0000 7fa530c00640 1 --2- 172.21.15.47:0/1134896783 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa50c066e40 0x7fa50c069300 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fa520000fc0 tx=0x7fa520009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:42.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.726+0000 7fa528c00640 1 -- 172.21.15.47:0/1134896783 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa5240ec380 con 0x7fa52c15ca70 2024-02-19T15:28:42.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.926+0000 7fa533600640 1 -- 172.21.15.47:0/1134896783 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa52c075360 con 0x7fa52c15ca70 2024-02-19T15:28:42.926 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-02-19T15:28:42.926+0000 7fc100600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T15:28:42.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.926+0000 7fa528c00640 1 -- 172.21.15.47:0/1134896783 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v83) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa5240efa40 con 0x7fa52c15ca70 2024-02-19T15:28:42.928 INFO:teuthology.orchestra.run.smithi047.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T15:28:42.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.930+0000 7fa533600640 1 -- 172.21.15.47:0/1134896783 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa50c066e40 msgr2=0x7fa50c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:42.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.930+0000 7fa533600640 1 --2- 172.21.15.47:0/1134896783 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa50c066e40 0x7fa50c069300 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fa520000fc0 tx=0x7fa520009040 comp rx=0 tx=0).stop 2024-02-19T15:28:42.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.930+0000 7fa533600640 1 -- 172.21.15.47:0/1134896783 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa52c15ca70 msgr2=0x7fa52c0ccaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:42.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.930+0000 7fa533600640 1 --2- 172.21.15.47:0/1134896783 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa52c15ca70 0x7fa52c0ccaf0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fa524009d90 tx=0x7fa524019510 comp rx=0 tx=0).stop 2024-02-19T15:28:42.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.930+0000 7fa533600640 1 -- 172.21.15.47:0/1134896783 shutdown_connections 2024-02-19T15:28:42.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.930+0000 7fa533600640 1 --2- 172.21.15.47:0/1134896783 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa50c066e40 0x7fa50c069300 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.930+0000 7fa533600640 1 --2- 172.21.15.47:0/1134896783 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa52c15ca70 0x7fa52c0ccaf0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.930+0000 7fa533600640 1 --2- 172.21.15.47:0/1134896783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa52c06fcb0 0x7fa52c15b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.930+0000 7fa533600640 1 --2- 172.21.15.47:0/1134896783 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa52c06b3f0 0x7fa52c1616e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:42.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.930+0000 7fa533600640 1 -- 172.21.15.47:0/1134896783 >> 172.21.15.47:0/1134896783 conn(0x7fa52c0d0ad0 msgr2=0x7fa52c0d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:42.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.930+0000 7fa533600640 1 -- 172.21.15.47:0/1134896783 shutdown_connections 2024-02-19T15:28:42.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:42.930+0000 7fa533600640 1 -- 172.21.15.47:0/1134896783 wait complete. 2024-02-19T15:28:42.945 INFO:tasks.ceph.mds.a.smithi047.stderr: -7226> 2024-02-19T15:27:45.111+0000 7fbbfd200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T15:28:42.951 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T15:28:42.997 INFO:tasks.ceph.mds.a.smithi047.stdout:starting mds.a at 2024-02-19T15:28:43.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.098+0000 7fe951800640 1 Processor -- start 2024-02-19T15:28:43.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.098+0000 7fe951800640 1 -- start start 2024-02-19T15:28:43.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.098+0000 7fe951800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe94c15bc60 0x7fe94c0ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:43.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.098+0000 7fe951800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe94c0d2480 0x7fe94c0cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:43.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.098+0000 7fe951800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94c0b0db0 con 0x7fe94c0d0480 2024-02-19T15:28:43.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.098+0000 7fe951800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe94c0d0930 con 0x7fe94c15bc60 2024-02-19T15:28:43.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.098+0000 7fe951800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c0cf9e0 con 0x7fe94c0d2480 2024-02-19T15:28:43.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.098+0000 7fe94be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe94c15bc60 0x7fe94c0ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:43.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.098+0000 7fe94be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe94c15bc60 0x7fe94c0ce9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60208/0 (socket says 172.21.15.47:60208) 2024-02-19T15:28:43.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.098+0000 7fe94be00640 1 -- 172.21.15.47:0/3955666570 learned_addr learned my addr 172.21.15.47:0/3955666570 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:43.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.102+0000 7fe943e00640 1 --2- 172.21.15.47:0/3955666570 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe94c0d2480 0x7fe94c0cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:43.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.102+0000 7fe94be00640 1 -- 172.21.15.47:0/3955666570 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe94c0d2480 msgr2=0x7fe94c0cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:43.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.102+0000 7fe94be00640 1 --2- 172.21.15.47:0/3955666570 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe94c0d2480 0x7fe94c0cff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:43.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.102+0000 7fe94be00640 1 -- 172.21.15.47:0/3955666570 >> v1:172.21.15.116:6789/0 conn(0x7fe94c0d0480 legacy=0x7fe94c0cc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:43.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.102+0000 7fe94be00640 1 -- 172.21.15.47:0/3955666570 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94c0b0fe0 con 0x7fe94c15bc60 2024-02-19T15:28:43.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.102+0000 7fe943e00640 1 --2- 172.21.15.47:0/3955666570 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe94c0d2480 0x7fe94c0cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:43.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.102+0000 7fe94be00640 1 --2- 172.21.15.47:0/3955666570 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe94c15bc60 0x7fe94c0ce9a0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fe93c009870 tx=0x7fe93c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e2613bb731b69bf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:43.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.102+0000 7fe94b400640 1 -- 172.21.15.47:0/3955666570 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c0ab620 con 0x7fe94c15bc60 2024-02-19T15:28:43.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.102+0000 7fe94b400640 1 -- 172.21.15.47:0/3955666570 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93c0ab780 con 0x7fe94c15bc60 2024-02-19T15:28:43.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.102+0000 7fe94b400640 1 -- 172.21.15.47:0/3955666570 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c0abb10 con 0x7fe94c15bc60 2024-02-19T15:28:43.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.102+0000 7fe951800640 1 -- 172.21.15.47:0/3955666570 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe94c15bc60 msgr2=0x7fe94c0ce9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:43.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.102+0000 7fe951800640 1 --2- 172.21.15.47:0/3955666570 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe94c15bc60 0x7fe94c0ce9a0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fe93c009870 tx=0x7fe93c01c460 comp rx=0 tx=0).stop 2024-02-19T15:28:43.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe951800640 1 -- 172.21.15.47:0/3955666570 shutdown_connections 2024-02-19T15:28:43.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe951800640 1 --2- 172.21.15.47:0/3955666570 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe94c0d2480 0x7fe94c0cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:43.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe951800640 1 --2- 172.21.15.47:0/3955666570 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe94c15bc60 0x7fe94c0ce9a0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:43.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe951800640 1 -- 172.21.15.47:0/3955666570 >> 172.21.15.47:0/3955666570 conn(0x7fe94c0d1960 msgr2=0x7fe94c0d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:43.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe951800640 1 -- 172.21.15.47:0/3955666570 shutdown_connections 2024-02-19T15:28:43.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe951800640 1 -- 172.21.15.47:0/3955666570 wait complete. 2024-02-19T15:28:43.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe951800640 1 Processor -- start 2024-02-19T15:28:43.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe951800640 1 -- start start 2024-02-19T15:28:43.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe951800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe94c0d0480 0x7fe94c1599a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:43.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe951800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe94c0d2480 0x7fe94c152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:43.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe951800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe94c15bc60 0x7fe94c0af480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:43.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe951800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c15a600 con 0x7fe94c15bc60 2024-02-19T15:28:43.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe951800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c15dbf0 con 0x7fe94c0d0480 2024-02-19T15:28:43.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe951800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c15ded0 con 0x7fe94c0d2480 2024-02-19T15:28:43.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe950e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe94c15bc60 0x7fe94c0af480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:43.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe950e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe94c15bc60 0x7fe94c0af480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60220/0 (socket says 172.21.15.47:60220) 2024-02-19T15:28:43.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe950e00640 1 -- 172.21.15.47:0/3981769683 learned_addr learned my addr 172.21.15.47:0/3981769683 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:43.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe943e00640 1 --2- 172.21.15.47:0/3981769683 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe94c0d2480 0x7fe94c152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:43.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe94be00640 1 --2- 172.21.15.47:0/3981769683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe94c0d0480 0x7fe94c1599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:43.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe950e00640 1 -- 172.21.15.47:0/3981769683 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe94c0d2480 msgr2=0x7fe94c152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:43.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe950e00640 1 --2- 172.21.15.47:0/3981769683 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe94c0d2480 0x7fe94c152cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:43.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe950e00640 1 -- 172.21.15.47:0/3981769683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe94c0d0480 msgr2=0x7fe94c1599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:43.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe950e00640 1 --2- 172.21.15.47:0/3981769683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe94c0d0480 0x7fe94c1599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:43.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe950e00640 1 -- 172.21.15.47:0/3981769683 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c0094d0 con 0x7fe94c15bc60 2024-02-19T15:28:43.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.106+0000 7fe950e00640 1 --2- 172.21.15.47:0/3981769683 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe94c15bc60 0x7fe94c0af480 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fe94400c780 tx=0x7fe94400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:43.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.110+0000 7fe948c00640 1 -- 172.21.15.47:0/3981769683 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe944004b70 con 0x7fe94c15bc60 2024-02-19T15:28:43.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.110+0000 7fe948c00640 1 -- 172.21.15.47:0/3981769683 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94400ce90 con 0x7fe94c15bc60 2024-02-19T15:28:43.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.110+0000 7fe948c00640 1 -- 172.21.15.47:0/3981769683 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94400a850 con 0x7fe94c15bc60 2024-02-19T15:28:43.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.110+0000 7fe951800640 1 -- 172.21.15.47:0/3981769683 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe94c15e1b0 con 0x7fe94c15bc60 2024-02-19T15:28:43.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.110+0000 7fe951800640 1 -- 172.21.15.47:0/3981769683 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe94c15e770 con 0x7fe94c15bc60 2024-02-19T15:28:43.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.110+0000 7fe951800640 1 -- 172.21.15.47:0/3981769683 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe94c15e960 con 0x7fe94c15bc60 2024-02-19T15:28:43.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.110+0000 7fe948c00640 1 -- 172.21.15.47:0/3981769683 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe94400a9b0 con 0x7fe94c15bc60 2024-02-19T15:28:43.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.110+0000 7fe948c00640 1 --2- 172.21.15.47:0/3981769683 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe928066e40 0x7fe928069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:43.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.110+0000 7fe948c00640 1 -- 172.21.15.47:0/3981769683 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fe944084b30 con 0x7fe94c15bc60 2024-02-19T15:28:43.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.110+0000 7fe94be00640 1 --2- 172.21.15.47:0/3981769683 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe928066e40 0x7fe928069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:43.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.114+0000 7fe94be00640 1 --2- 172.21.15.47:0/3981769683 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe928066e40 0x7fe928069300 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe93c0af000 tx=0x7fe93c0b6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:43.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.114+0000 7fe948c00640 1 -- 172.21.15.47:0/3981769683 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe944051920 con 0x7fe94c15bc60 2024-02-19T15:28:43.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.314+0000 7fe951800640 1 -- 172.21.15.47:0/3981769683 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe94c04f260 con 0x7fe94c15bc60 2024-02-19T15:28:43.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.934+0000 7fe948c00640 1 -- 172.21.15.47:0/3981769683 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v84) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe944054fe0 con 0x7fe94c15bc60 2024-02-19T15:28:43.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.938+0000 7fe951800640 1 -- 172.21.15.47:0/3981769683 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe928066e40 msgr2=0x7fe928069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:43.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.938+0000 7fe951800640 1 --2- 172.21.15.47:0/3981769683 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe928066e40 0x7fe928069300 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe93c0af000 tx=0x7fe93c0b6040 comp rx=0 tx=0).stop 2024-02-19T15:28:43.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.938+0000 7fe951800640 1 -- 172.21.15.47:0/3981769683 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe94c15bc60 msgr2=0x7fe94c0af480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:43.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.938+0000 7fe951800640 1 --2- 172.21.15.47:0/3981769683 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe94c15bc60 0x7fe94c0af480 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fe94400c780 tx=0x7fe94400cc40 comp rx=0 tx=0).stop 2024-02-19T15:28:43.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.938+0000 7fe951800640 1 -- 172.21.15.47:0/3981769683 shutdown_connections 2024-02-19T15:28:43.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.938+0000 7fe951800640 1 --2- 172.21.15.47:0/3981769683 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe928066e40 0x7fe928069300 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:43.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.938+0000 7fe951800640 1 --2- 172.21.15.47:0/3981769683 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe94c15bc60 0x7fe94c0af480 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:43.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.938+0000 7fe951800640 1 --2- 172.21.15.47:0/3981769683 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe94c0d2480 0x7fe94c152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:43.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.938+0000 7fe951800640 1 --2- 172.21.15.47:0/3981769683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe94c0d0480 0x7fe94c1599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:43.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.938+0000 7fe951800640 1 -- 172.21.15.47:0/3981769683 >> 172.21.15.47:0/3981769683 conn(0x7fe94c0d1960 msgr2=0x7fe94c0b2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:43.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.938+0000 7fe951800640 1 -- 172.21.15.47:0/3981769683 shutdown_connections 2024-02-19T15:28:43.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:43.938+0000 7fe951800640 1 -- 172.21.15.47:0/3981769683 wait complete. 2024-02-19T15:28:43.962 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T15:28:44.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.158+0000 7f55dcc00640 1 Processor -- start 2024-02-19T15:28:44.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.158+0000 7f55dcc00640 1 -- start start 2024-02-19T15:28:44.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.158+0000 7f55dcc00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f55d80d0800 0x7f55d80d2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:44.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.158+0000 7f55dcc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f55d815d860 0x7f55d80ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:44.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.158+0000 7f55dcc00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d80cc690 con 0x7f55d80ceb20 2024-02-19T15:28:44.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.158+0000 7f55dcc00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55d80cfca0 con 0x7f55d80d0800 2024-02-19T15:28:44.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.158+0000 7f55dcc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d80cc950 con 0x7f55d815d860 2024-02-19T15:28:44.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.158+0000 7f55d5400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f55d80d0800 0x7f55d80d2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:44.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.158+0000 7f55d5400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f55d80d0800 0x7f55d80d2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40640/0 (socket says 172.21.15.47:40640) 2024-02-19T15:28:44.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.158+0000 7f55d5400640 1 -- 172.21.15.47:0/935823313 learned_addr learned my addr 172.21.15.47:0/935823313 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:44.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.158+0000 7f55d5400640 1 -- 172.21.15.47:0/935823313 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f55d815d860 msgr2=0x7f55d80ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:44.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.158+0000 7f55d6800640 1 --2- 172.21.15.47:0/935823313 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f55d815d860 0x7f55d80ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:44.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.158+0000 7f55d5400640 1 --2- 172.21.15.47:0/935823313 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f55d815d860 0x7f55d80ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:44.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.158+0000 7f55d5400640 1 -- 172.21.15.47:0/935823313 >> v1:172.21.15.116:6789/0 conn(0x7f55d80ceb20 legacy=0x7f55d80d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:44.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.158+0000 7f55d5400640 1 -- 172.21.15.47:0/935823313 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d80cef00 con 0x7f55d80d0800 2024-02-19T15:28:44.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.158+0000 7f55d6800640 1 --2- 172.21.15.47:0/935823313 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f55d815d860 0x7f55d80ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:44.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.162+0000 7f55d5400640 1 --2- 172.21.15.47:0/935823313 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55d80d0800 0x7f55d80d2670 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009870 tx=0x7f55c401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43107666cf59f884 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:44.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.162+0000 7f55d4a00640 1 -- 172.21.15.47:0/935823313 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c40ab600 con 0x7f55d80d0800 2024-02-19T15:28:44.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.162+0000 7f55d4a00640 1 -- 172.21.15.47:0/935823313 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c40a7050 con 0x7f55d80d0800 2024-02-19T15:28:44.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.162+0000 7f55d4a00640 1 -- 172.21.15.47:0/935823313 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c40ab980 con 0x7f55d80d0800 2024-02-19T15:28:44.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.162+0000 7f55dcc00640 1 -- 172.21.15.47:0/935823313 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55d80d0800 msgr2=0x7f55d80d2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:44.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.162+0000 7f55dcc00640 1 --2- 172.21.15.47:0/935823313 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55d80d0800 0x7f55d80d2670 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009870 tx=0x7f55c401c490 comp rx=0 tx=0).stop 2024-02-19T15:28:44.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.162+0000 7f55dcc00640 1 -- 172.21.15.47:0/935823313 shutdown_connections 2024-02-19T15:28:44.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.162+0000 7f55dcc00640 1 --2- 172.21.15.47:0/935823313 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f55d815d860 0x7f55d80ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:44.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.162+0000 7f55dcc00640 1 --2- 172.21.15.47:0/935823313 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55d80d0800 0x7f55d80d2670 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:44.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.162+0000 7f55dcc00640 1 -- 172.21.15.47:0/935823313 >> 172.21.15.47:0/935823313 conn(0x7f55d809aed0 msgr2=0x7f55d80d1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:44.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.162+0000 7f55dcc00640 1 -- 172.21.15.47:0/935823313 shutdown_connections 2024-02-19T15:28:44.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.162+0000 7f55dcc00640 1 -- 172.21.15.47:0/935823313 wait complete. 2024-02-19T15:28:44.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55dcc00640 1 Processor -- start 2024-02-19T15:28:44.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55dcc00640 1 -- start start 2024-02-19T15:28:44.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55dcc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55d80ceb20 0x7f55d80c9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:44.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55dcc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f55d80d0800 0x7f55d8153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:44.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55dcc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55d815d860 0x7f55d80c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:44.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55dcc00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d81543d0 con 0x7f55d80ceb20 2024-02-19T15:28:44.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55dcc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d80c96b0 con 0x7f55d815d860 2024-02-19T15:28:44.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55dcc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d80c8d80 con 0x7f55d80d0800 2024-02-19T15:28:44.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55d5e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55d80ceb20 0x7f55d80c9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:44.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55d5e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55d80ceb20 0x7f55d80c9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40656/0 (socket says 172.21.15.47:40656) 2024-02-19T15:28:44.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55d5e00640 1 -- 172.21.15.47:0/2378078891 learned_addr learned my addr 172.21.15.47:0/2378078891 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:44.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55d6800640 1 --2- 172.21.15.47:0/2378078891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55d815d860 0x7f55d80c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:44.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55d5400640 1 --2- 172.21.15.47:0/2378078891 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f55d80d0800 0x7f55d8153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:44.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55d5e00640 1 -- 172.21.15.47:0/2378078891 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f55d80d0800 msgr2=0x7f55d8153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:44.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55d5e00640 1 --2- 172.21.15.47:0/2378078891 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f55d80d0800 0x7f55d8153e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:44.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55d5e00640 1 -- 172.21.15.47:0/2378078891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55d815d860 msgr2=0x7f55d80c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:44.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55d5e00640 1 --2- 172.21.15.47:0/2378078891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55d815d860 0x7f55d80c7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:44.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55d5e00640 1 -- 172.21.15.47:0/2378078891 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c40094d0 con 0x7f55d80ceb20 2024-02-19T15:28:44.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55d6800640 1 --2- 172.21.15.47:0/2378078891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55d815d860 0x7f55d80c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:44.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55d5e00640 1 --2- 172.21.15.47:0/2378078891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55d80ceb20 0x7f55d80c9b40 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f55c800c450 tx=0x7f55c800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:44.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55ce000640 1 -- 172.21.15.47:0/2378078891 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c8012830 con 0x7f55d80ceb20 2024-02-19T15:28:44.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55ce000640 1 -- 172.21.15.47:0/2378078891 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c80129b0 con 0x7f55d80ceb20 2024-02-19T15:28:44.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55ce000640 1 -- 172.21.15.47:0/2378078891 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c8012d60 con 0x7f55d80ceb20 2024-02-19T15:28:44.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55dcc00640 1 -- 172.21.15.47:0/2378078891 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d80c9060 con 0x7f55d80ceb20 2024-02-19T15:28:44.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.166+0000 7f55dcc00640 1 -- 172.21.15.47:0/2378078891 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d80c7a90 con 0x7f55d80ceb20 2024-02-19T15:28:44.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.170+0000 7f55dcc00640 1 -- 172.21.15.47:0/2378078891 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d8075360 con 0x7f55d80ceb20 2024-02-19T15:28:44.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.170+0000 7f55ce000640 1 -- 172.21.15.47:0/2378078891 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f55c801c460 con 0x7f55d80ceb20 2024-02-19T15:28:44.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.170+0000 7f55ce000640 1 --2- 172.21.15.47:0/2378078891 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f55b8066e40 0x7f55b8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:44.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.170+0000 7f55ce000640 1 -- 172.21.15.47:0/2378078891 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f55c8084a80 con 0x7f55d80ceb20 2024-02-19T15:28:44.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.170+0000 7f55d5400640 1 --2- 172.21.15.47:0/2378078891 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f55b8066e40 0x7f55b8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:44.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.170+0000 7f55d5400640 1 --2- 172.21.15.47:0/2378078891 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f55b8066e40 0x7f55b8069300 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f55d8072500 tx=0x7f55c4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:44.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.174+0000 7f55ce000640 1 -- 172.21.15.47:0/2378078891 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f55c8051690 con 0x7f55d80ceb20 2024-02-19T15:28:44.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.334+0000 7f55dcc00640 1 -- 172.21.15.47:0/2378078891 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f55d80cfed0 con 0x7f55d80ceb20 2024-02-19T15:28:44.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.934+0000 7f55ce000640 1 -- 172.21.15.47:0/2378078891 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v116) v1 ==== 181+0+0 (secure 0 0 0) 0x7f55c8054d50 con 0x7f55d80ceb20 2024-02-19T15:28:44.937 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_metadata' removed 2024-02-19T15:28:44.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.938+0000 7f55dcc00640 1 -- 172.21.15.47:0/2378078891 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f55b8066e40 msgr2=0x7f55b8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:44.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.938+0000 7f55dcc00640 1 --2- 172.21.15.47:0/2378078891 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f55b8066e40 0x7f55b8069300 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f55d8072500 tx=0x7f55c4008040 comp rx=0 tx=0).stop 2024-02-19T15:28:44.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.938+0000 7f55dcc00640 1 -- 172.21.15.47:0/2378078891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55d80ceb20 msgr2=0x7f55d80c9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:44.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.938+0000 7f55dcc00640 1 --2- 172.21.15.47:0/2378078891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55d80ceb20 0x7f55d80c9b40 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f55c800c450 tx=0x7f55c800c910 comp rx=0 tx=0).stop 2024-02-19T15:28:44.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.938+0000 7f55dcc00640 1 -- 172.21.15.47:0/2378078891 shutdown_connections 2024-02-19T15:28:44.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.938+0000 7f55dcc00640 1 --2- 172.21.15.47:0/2378078891 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f55b8066e40 0x7f55b8069300 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:44.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.938+0000 7f55dcc00640 1 --2- 172.21.15.47:0/2378078891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55d815d860 0x7f55d80c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:44.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.938+0000 7f55dcc00640 1 --2- 172.21.15.47:0/2378078891 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f55d80d0800 0x7f55d8153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:44.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.938+0000 7f55dcc00640 1 --2- 172.21.15.47:0/2378078891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55d80ceb20 0x7f55d80c9b40 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:44.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.938+0000 7f55dcc00640 1 -- 172.21.15.47:0/2378078891 >> 172.21.15.47:0/2378078891 conn(0x7f55d809aed0 msgr2=0x7f55d80cdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:44.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.938+0000 7f55dcc00640 1 -- 172.21.15.47:0/2378078891 shutdown_connections 2024-02-19T15:28:44.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:44.938+0000 7f55dcc00640 1 -- 172.21.15.47:0/2378078891 wait complete. 2024-02-19T15:28:44.964 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T15:28:45.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.110+0000 7f675e800640 1 Processor -- start 2024-02-19T15:28:45.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.110+0000 7f675e800640 1 -- start start 2024-02-19T15:28:45.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.110+0000 7f675e800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f675815f190 0x7f67580d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:45.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.110+0000 7f675e800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f675815f8f0 0x7f675815fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:45.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.110+0000 7f675e800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67580d0a50 con 0x7f67580cec80 2024-02-19T15:28:45.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.110+0000 7f675e800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f675806d470 con 0x7f675815f8f0 2024-02-19T15:28:45.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.110+0000 7f675e800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67580cfe80 con 0x7f675815f190 2024-02-19T15:28:45.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.110+0000 7f6757e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f675815f8f0 0x7f675815fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:45.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.110+0000 7f6757e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f675815f8f0 0x7f675815fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40668/0 (socket says 172.21.15.47:40668) 2024-02-19T15:28:45.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.110+0000 7f6757e00640 1 -- 172.21.15.47:0/3205096950 learned_addr learned my addr 172.21.15.47:0/3205096950 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:45.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.110+0000 7f6757400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f67580cec80 0x7f675806cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60720/0 (socket says 172.21.15.47:60720) 2024-02-19T15:28:45.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.110+0000 7f674ea00640 1 --2- 172.21.15.47:0/3205096950 >> v2:172.21.15.116:3300/0 conn(0x7f675815f190 0x7f67580d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:45.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.110+0000 7f6757e00640 1 -- 172.21.15.47:0/3205096950 >> v2:172.21.15.116:3300/0 conn(0x7f675815f190 msgr2=0x7f67580d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:45.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.110+0000 7f6757e00640 1 --2- 172.21.15.47:0/3205096950 >> v2:172.21.15.116:3300/0 conn(0x7f675815f190 0x7f67580d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:45.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.110+0000 7f6757e00640 1 -- 172.21.15.47:0/3205096950 >> v1:172.21.15.47:6789/0 conn(0x7f67580cec80 legacy=0x7f675806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:45.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.110+0000 7f6757e00640 1 -- 172.21.15.47:0/3205096950 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67580d0120 con 0x7f675815f8f0 2024-02-19T15:28:45.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.114+0000 7f6757e00640 1 --2- 172.21.15.47:0/3205096950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f675815f8f0 0x7f675815fd80 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f675806e1a0 tx=0x7f675001c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd2823cd57efe4d5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:45.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.114+0000 7f6756a00640 1 -- 172.21.15.47:0/3205096950 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67500a5790 con 0x7f675815f8f0 2024-02-19T15:28:45.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.114+0000 7f6756a00640 1 -- 172.21.15.47:0/3205096950 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67500a58f0 con 0x7f675815f8f0 2024-02-19T15:28:45.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.114+0000 7f6756a00640 1 -- 172.21.15.47:0/3205096950 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67500a5c80 con 0x7f675815f8f0 2024-02-19T15:28:45.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.114+0000 7f675e800640 1 -- 172.21.15.47:0/3205096950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f675815f8f0 msgr2=0x7f675815fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:45.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.114+0000 7f675e800640 1 --2- 172.21.15.47:0/3205096950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f675815f8f0 0x7f675815fd80 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f675806e1a0 tx=0x7f675001c9c0 comp rx=0 tx=0).stop 2024-02-19T15:28:45.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.114+0000 7f675e800640 1 -- 172.21.15.47:0/3205096950 shutdown_connections 2024-02-19T15:28:45.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.114+0000 7f675e800640 1 --2- 172.21.15.47:0/3205096950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f675815f8f0 0x7f675815fd80 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:45.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.114+0000 7f675e800640 1 --2- 172.21.15.47:0/3205096950 >> v2:172.21.15.116:3300/0 conn(0x7f675815f190 0x7f67580d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:45.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.114+0000 7f675e800640 1 -- 172.21.15.47:0/3205096950 >> 172.21.15.47:0/3205096950 conn(0x7f675809aed0 msgr2=0x7f67580d1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:45.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.114+0000 7f675e800640 1 -- 172.21.15.47:0/3205096950 shutdown_connections 2024-02-19T15:28:45.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.114+0000 7f675e800640 1 -- 172.21.15.47:0/3205096950 wait complete. 2024-02-19T15:28:45.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f675e800640 1 Processor -- start 2024-02-19T15:28:45.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f675e800640 1 -- start start 2024-02-19T15:28:45.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f675e800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f67580cec80 0x7f6758152f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:45.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f675e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f675815f190 0x7f67580c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:45.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f675e800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f675815f8f0 0x7f67580c59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:45.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f675e800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67580b4d60 con 0x7f675815f8f0 2024-02-19T15:28:45.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f675e800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67580b5040 con 0x7f675815f190 2024-02-19T15:28:45.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f675e800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67580b5320 con 0x7f67580cec80 2024-02-19T15:28:45.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f6757e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f675815f8f0 0x7f67580c59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:45.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f6757e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f675815f8f0 0x7f67580c59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40672/0 (socket says 172.21.15.47:40672) 2024-02-19T15:28:45.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f6757e00640 1 -- 172.21.15.47:0/605227566 learned_addr learned my addr 172.21.15.47:0/605227566 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:45.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f674ea00640 1 --2- 172.21.15.47:0/605227566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f675815f190 0x7f67580c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:45.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f6757400640 1 --2- 172.21.15.47:0/605227566 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f67580cec80 0x7f6758152f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:45.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f6757e00640 1 -- 172.21.15.47:0/605227566 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f67580cec80 msgr2=0x7f6758152f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:45.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f6757e00640 1 --2- 172.21.15.47:0/605227566 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f67580cec80 0x7f6758152f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:45.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f6757e00640 1 -- 172.21.15.47:0/605227566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f675815f190 msgr2=0x7f67580c54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:45.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f6757e00640 1 --2- 172.21.15.47:0/605227566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f675815f190 0x7f67580c54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:45.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f6757e00640 1 -- 172.21.15.47:0/605227566 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6750009b30 con 0x7f675815f8f0 2024-02-19T15:28:45.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f674ea00640 1 --2- 172.21.15.47:0/605227566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f675815f190 0x7f67580c54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:45.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f6757e00640 1 --2- 172.21.15.47:0/605227566 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f675815f8f0 0x7f67580c59f0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f67500195d0 tx=0x7f6750019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:45.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f674fe00640 1 -- 172.21.15.47:0/605227566 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67500b8670 con 0x7f675815f8f0 2024-02-19T15:28:45.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f674fe00640 1 -- 172.21.15.47:0/605227566 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67500b87d0 con 0x7f675815f8f0 2024-02-19T15:28:45.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f674fe00640 1 -- 172.21.15.47:0/605227566 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67500b8b60 con 0x7f675815f8f0 2024-02-19T15:28:45.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f675e800640 1 -- 172.21.15.47:0/605227566 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67580b5600 con 0x7f675815f8f0 2024-02-19T15:28:45.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.118+0000 7f675e800640 1 -- 172.21.15.47:0/605227566 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6758153440 con 0x7f675815f8f0 2024-02-19T15:28:45.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.122+0000 7f675e800640 1 -- 172.21.15.47:0/605227566 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6758075360 con 0x7f675815f8f0 2024-02-19T15:28:45.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.122+0000 7f674fe00640 1 -- 172.21.15.47:0/605227566 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f67500b8cc0 con 0x7f675815f8f0 2024-02-19T15:28:45.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.122+0000 7f674fe00640 1 --2- 172.21.15.47:0/605227566 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6734066e40 0x7f6734069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:45.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.122+0000 7f674fe00640 1 -- 172.21.15.47:0/605227566 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5587+0+0 (secure 0 0 0) 0x7f6750120200 con 0x7f675815f8f0 2024-02-19T15:28:45.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.122+0000 7f6757400640 1 --2- 172.21.15.47:0/605227566 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6734066e40 0x7f6734069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:45.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.122+0000 7f6757400640 1 --2- 172.21.15.47:0/605227566 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6734066e40 0x7f6734069300 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6748000f70 tx=0x7f6748007cd0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:45.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.126+0000 7f674fe00640 1 -- 172.21.15.47:0/605227566 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f67500ed170 con 0x7f675815f8f0 2024-02-19T15:28:45.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.290+0000 7f675e800640 1 -- 172.21.15.47:0/605227566 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f67580d2220 con 0x7f675815f8f0 2024-02-19T15:28:45.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.942+0000 7f674fe00640 1 -- 172.21.15.47:0/605227566 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v117) v1 ==== 169+0+0 (secure 0 0 0) 0x7f67500f0830 con 0x7f675815f8f0 2024-02-19T15:28:45.942 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_data' removed 2024-02-19T15:28:45.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.942+0000 7f675e800640 1 -- 172.21.15.47:0/605227566 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6734066e40 msgr2=0x7f6734069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:45.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.942+0000 7f675e800640 1 --2- 172.21.15.47:0/605227566 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6734066e40 0x7f6734069300 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6748000f70 tx=0x7f6748007cd0 comp rx=0 tx=0).stop 2024-02-19T15:28:45.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.946+0000 7f675e800640 1 -- 172.21.15.47:0/605227566 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f675815f8f0 msgr2=0x7f67580c59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:45.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.946+0000 7f675e800640 1 --2- 172.21.15.47:0/605227566 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f675815f8f0 0x7f67580c59f0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f67500195d0 tx=0x7f6750019760 comp rx=0 tx=0).stop 2024-02-19T15:28:45.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.946+0000 7f675e800640 1 -- 172.21.15.47:0/605227566 shutdown_connections 2024-02-19T15:28:45.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.946+0000 7f675e800640 1 --2- 172.21.15.47:0/605227566 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6734066e40 0x7f6734069300 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:45.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.946+0000 7f675e800640 1 --2- 172.21.15.47:0/605227566 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f675815f8f0 0x7f67580c59f0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:45.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.946+0000 7f675e800640 1 --2- 172.21.15.47:0/605227566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f675815f190 0x7f67580c54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:45.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.946+0000 7f675e800640 1 --2- 172.21.15.47:0/605227566 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f67580cec80 0x7f6758152f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:45.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.946+0000 7f675e800640 1 -- 172.21.15.47:0/605227566 >> 172.21.15.47:0/605227566 conn(0x7f675809aed0 msgr2=0x7f6758070280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:45.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.946+0000 7f675e800640 1 -- 172.21.15.47:0/605227566 shutdown_connections 2024-02-19T15:28:45.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:45.946+0000 7f675e800640 1 -- 172.21.15.47:0/605227566 wait complete. 2024-02-19T15:28:45.968 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-19T15:28:46.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.118+0000 7f4c01a00640 1 Processor -- start 2024-02-19T15:28:46.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.118+0000 7f4c01a00640 1 -- start start 2024-02-19T15:28:46.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.122+0000 7f4c01a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4bfc15af10 0x7f4bfc06e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:46.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.122+0000 7f4c01a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4bfc0cd880 0x7f4bfc0cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:46.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.122+0000 7f4c01a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bfc071c70 con 0x7f4bfc0ca8f0 2024-02-19T15:28:46.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.122+0000 7f4c01a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bfc06e8a0 con 0x7f4bfc0cd880 2024-02-19T15:28:46.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.122+0000 7f4c01a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bfc06cef0 con 0x7f4bfc15af10 2024-02-19T15:28:46.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.122+0000 7f4bfb400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4bfc0cd880 0x7f4bfc0cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:46.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.122+0000 7f4bfb400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4bfc0cd880 0x7f4bfc0cdcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40688/0 (socket says 172.21.15.47:40688) 2024-02-19T15:28:46.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.122+0000 7f4bfb400640 1 -- 172.21.15.47:0/724477209 learned_addr learned my addr 172.21.15.47:0/724477209 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:46.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.122+0000 7f4bf2000640 1 --1- 172.21.15.47:0/724477209 >> v1:172.21.15.47:6789/0 conn(0x7f4bfc0ca8f0 0x7f4bfc0cad50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60724/0 (socket says 172.21.15.47:60724) 2024-02-19T15:28:46.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.122+0000 7f4bfaa00640 1 --2- 172.21.15.47:0/724477209 >> v2:172.21.15.116:3300/0 conn(0x7f4bfc15af10 0x7f4bfc06e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:46.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.122+0000 7f4bfb400640 1 -- 172.21.15.47:0/724477209 >> v2:172.21.15.116:3300/0 conn(0x7f4bfc15af10 msgr2=0x7f4bfc06e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:46.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.122+0000 7f4bfb400640 1 --2- 172.21.15.47:0/724477209 >> v2:172.21.15.116:3300/0 conn(0x7f4bfc15af10 0x7f4bfc06e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:46.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.122+0000 7f4bfb400640 1 -- 172.21.15.47:0/724477209 >> v1:172.21.15.47:6789/0 conn(0x7f4bfc0ca8f0 legacy=0x7f4bfc0cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:46.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.122+0000 7f4bfb400640 1 -- 172.21.15.47:0/724477209 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bfc0ca610 con 0x7f4bfc0cd880 2024-02-19T15:28:46.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4bfb400640 1 --2- 172.21.15.47:0/724477209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4bfc0cd880 0x7f4bfc0cdcf0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc073660 tx=0x7f4bf401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c93814f45cf28b11 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:46.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4bfa000640 1 -- 172.21.15.47:0/724477209 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf40a57f0 con 0x7f4bfc0cd880 2024-02-19T15:28:46.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4bfa000640 1 -- 172.21.15.47:0/724477209 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf40a5950 con 0x7f4bfc0cd880 2024-02-19T15:28:46.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4bfa000640 1 -- 172.21.15.47:0/724477209 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf40a5ce0 con 0x7f4bfc0cd880 2024-02-19T15:28:46.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4c01a00640 1 -- 172.21.15.47:0/724477209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4bfc0cd880 msgr2=0x7f4bfc0cdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:46.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4c01a00640 1 --2- 172.21.15.47:0/724477209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4bfc0cd880 0x7f4bfc0cdcf0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc073660 tx=0x7f4bf401c930 comp rx=0 tx=0).stop 2024-02-19T15:28:46.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4c01a00640 1 -- 172.21.15.47:0/724477209 shutdown_connections 2024-02-19T15:28:46.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4c01a00640 1 --2- 172.21.15.47:0/724477209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4bfc0cd880 0x7f4bfc0cdcf0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:46.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4c01a00640 1 --2- 172.21.15.47:0/724477209 >> v2:172.21.15.116:3300/0 conn(0x7f4bfc15af10 0x7f4bfc06e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:46.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4c01a00640 1 -- 172.21.15.47:0/724477209 >> 172.21.15.47:0/724477209 conn(0x7f4bfc0cfc20 msgr2=0x7f4bfc0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:46.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4c01a00640 1 -- 172.21.15.47:0/724477209 shutdown_connections 2024-02-19T15:28:46.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4c01a00640 1 -- 172.21.15.47:0/724477209 wait complete. 2024-02-19T15:28:46.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4c01a00640 1 Processor -- start 2024-02-19T15:28:46.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4c01a00640 1 -- start start 2024-02-19T15:28:46.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4c01a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4bfc0ca8f0 0x7f4bfc156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:46.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4c01a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bfc0cd880 0x7f4bfc145190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:46.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4c01a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4bfc15af10 0x7f4bfc15d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:46.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4c01a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc0c8710 con 0x7f4bfc15af10 2024-02-19T15:28:46.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4c01a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc157300 con 0x7f4bfc0cd880 2024-02-19T15:28:46.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4c01a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc1575e0 con 0x7f4bfc0ca8f0 2024-02-19T15:28:46.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.126+0000 7f4bfb400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4bfc15af10 0x7f4bfc15d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:46.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.130+0000 7f4bfb400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4bfc15af10 0x7f4bfc15d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40690/0 (socket says 172.21.15.47:40690) 2024-02-19T15:28:46.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.130+0000 7f4bfb400640 1 -- 172.21.15.47:0/276929889 learned_addr learned my addr 172.21.15.47:0/276929889 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:46.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.130+0000 7f4bf2000640 1 --2- 172.21.15.47:0/276929889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bfc0cd880 0x7f4bfc145190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:46.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.130+0000 7f4bfaa00640 1 --2- 172.21.15.47:0/276929889 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4bfc0ca8f0 0x7f4bfc156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:46.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.130+0000 7f4bfb400640 1 -- 172.21.15.47:0/276929889 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4bfc0ca8f0 msgr2=0x7f4bfc156d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:46.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.130+0000 7f4bfb400640 1 --2- 172.21.15.47:0/276929889 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4bfc0ca8f0 0x7f4bfc156d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:46.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.130+0000 7f4bfb400640 1 -- 172.21.15.47:0/276929889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bfc0cd880 msgr2=0x7f4bfc145190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:46.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.130+0000 7f4bfb400640 1 --2- 172.21.15.47:0/276929889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bfc0cd880 0x7f4bfc145190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:46.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.130+0000 7f4bfb400640 1 -- 172.21.15.47:0/276929889 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf4009b30 con 0x7f4bfc15af10 2024-02-19T15:28:46.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.130+0000 7f4bfb400640 1 --2- 172.21.15.47:0/276929889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4bfc15af10 0x7f4bfc15d640 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f4bf40af040 tx=0x7f4bf40196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:46.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.130+0000 7f4bf3400640 1 -- 172.21.15.47:0/276929889 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf40b8670 con 0x7f4bfc15af10 2024-02-19T15:28:46.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.130+0000 7f4bf3400640 1 -- 172.21.15.47:0/276929889 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf40b87d0 con 0x7f4bfc15af10 2024-02-19T15:28:46.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.130+0000 7f4c01a00640 1 -- 172.21.15.47:0/276929889 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bfc1578c0 con 0x7f4bfc15af10 2024-02-19T15:28:46.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.130+0000 7f4bf3400640 1 -- 172.21.15.47:0/276929889 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf40b8b80 con 0x7f4bfc15af10 2024-02-19T15:28:46.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.130+0000 7f4c01a00640 1 -- 172.21.15.47:0/276929889 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bfc157e80 con 0x7f4bfc15af10 2024-02-19T15:28:46.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.130+0000 7f4c01a00640 1 -- 172.21.15.47:0/276929889 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bfc158070 con 0x7f4bfc15af10 2024-02-19T15:28:46.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.130+0000 7f4bf3400640 1 -- 172.21.15.47:0/276929889 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4bf40b8ce0 con 0x7f4bfc15af10 2024-02-19T15:28:46.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.130+0000 7f4bf3400640 1 --2- 172.21.15.47:0/276929889 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4bd8066e40 0x7f4bd8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:46.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.130+0000 7f4bf3400640 1 -- 172.21.15.47:0/276929889 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f4bf411fed0 con 0x7f4bfc15af10 2024-02-19T15:28:46.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.134+0000 7f4bfaa00640 1 --2- 172.21.15.47:0/276929889 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4bd8066e40 0x7f4bd8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:46.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.134+0000 7f4bfaa00640 1 --2- 172.21.15.47:0/276929889 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4bd8066e40 0x7f4bd8069300 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0716e0 tx=0x7f4bec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:46.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.134+0000 7f4bf3400640 1 -- 172.21.15.47:0/276929889 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f4bf40ecf30 con 0x7f4bfc15af10 2024-02-19T15:28:46.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.298+0000 7f4c01a00640 1 -- 172.21.15.47:0/276929889 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f4bfc075360 con 0x7f4bfc15af10 2024-02-19T15:28:46.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.302+0000 7f4bf3400640 1 -- 172.21.15.47:0/276929889 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4bf40f05f0 con 0x7f4bfc15af10 2024-02-19T15:28:46.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.306+0000 7f4c01a00640 1 -- 172.21.15.47:0/276929889 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4bd8066e40 msgr2=0x7f4bd8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:46.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.306+0000 7f4c01a00640 1 --2- 172.21.15.47:0/276929889 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4bd8066e40 0x7f4bd8069300 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0716e0 tx=0x7f4bec008040 comp rx=0 tx=0).stop 2024-02-19T15:28:46.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.306+0000 7f4c01a00640 1 -- 172.21.15.47:0/276929889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4bfc15af10 msgr2=0x7f4bfc15d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:46.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.306+0000 7f4c01a00640 1 --2- 172.21.15.47:0/276929889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4bfc15af10 0x7f4bfc15d640 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f4bf40af040 tx=0x7f4bf40196d0 comp rx=0 tx=0).stop 2024-02-19T15:28:46.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.306+0000 7f4c01a00640 1 -- 172.21.15.47:0/276929889 shutdown_connections 2024-02-19T15:28:46.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.306+0000 7f4c01a00640 1 --2- 172.21.15.47:0/276929889 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4bd8066e40 0x7f4bd8069300 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:46.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.306+0000 7f4c01a00640 1 --2- 172.21.15.47:0/276929889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4bfc15af10 0x7f4bfc15d640 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:46.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.306+0000 7f4c01a00640 1 --2- 172.21.15.47:0/276929889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bfc0cd880 0x7f4bfc145190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:46.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.306+0000 7f4c01a00640 1 --2- 172.21.15.47:0/276929889 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4bfc0ca8f0 0x7f4bfc156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:46.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.306+0000 7f4c01a00640 1 -- 172.21.15.47:0/276929889 >> 172.21.15.47:0/276929889 conn(0x7f4bfc0cfc20 msgr2=0x7f4bfc0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:46.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.306+0000 7f4c01a00640 1 -- 172.21.15.47:0/276929889 shutdown_connections 2024-02-19T15:28:46.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.306+0000 7f4c01a00640 1 -- 172.21.15.47:0/276929889 wait complete. 2024-02-19T15:28:46.328 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-02-19T15:28:46.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.470+0000 7f5744000640 1 Processor -- start 2024-02-19T15:28:46.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.470+0000 7f5744000640 1 -- start start 2024-02-19T15:28:46.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.470+0000 7f5744000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f573c0cded0 0x7f573c0ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:46.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.470+0000 7f5744000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f573c0ceb40 0x7f573c06bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:46.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.470+0000 7f5744000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c0cce30 con 0x7f573c06d650 2024-02-19T15:28:46.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.470+0000 7f5744000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f573c0cefd0 con 0x7f573c0ceb40 2024-02-19T15:28:46.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.470+0000 7f5744000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0cd110 con 0x7f573c0cded0 2024-02-19T15:28:46.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.470+0000 7f5741600640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f573c06d650 0x7f573c0d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60736/0 (socket says 172.21.15.47:60736) 2024-02-19T15:28:46.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.470+0000 7f5741600640 1 -- 172.21.15.47:0/394207476 learned_addr learned my addr 172.21.15.47:0/394207476 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:46.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.470+0000 7f5742000640 1 --2- 172.21.15.47:0/394207476 >> v2:172.21.15.47:3300/0 conn(0x7f573c0ceb40 0x7f573c06bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:46.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.470+0000 7f5740c00640 1 --2- 172.21.15.47:0/394207476 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f573c0cded0 0x7f573c0ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:46.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.470+0000 7f573be00640 1 -- 172.21.15.47:0/394207476 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103108144 0 0) 0x7f573c0cce30 con 0x7f573c06d650 2024-02-19T15:28:46.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.470+0000 7f5742000640 1 -- 172.21.15.47:0/394207476 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f573c0cded0 msgr2=0x7f573c0ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:46.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.470+0000 7f5742000640 1 --2- 172.21.15.47:0/394207476 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f573c0cded0 0x7f573c0ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:46.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.470+0000 7f5742000640 1 -- 172.21.15.47:0/394207476 >> v1:172.21.15.47:6789/0 conn(0x7f573c06d650 legacy=0x7f573c0d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:46.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.470+0000 7f5742000640 1 -- 172.21.15.47:0/394207476 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c0cd3f0 con 0x7f573c0ceb40 2024-02-19T15:28:46.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.474+0000 7f5742000640 1 --2- 172.21.15.47:0/394207476 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f573c0ceb40 0x7f573c06bd60 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f5734009d90 tx=0x7f573401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82d50afbd63630f1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:46.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.474+0000 7f573be00640 1 -- 172.21.15.47:0/394207476 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57340a5970 con 0x7f573c0ceb40 2024-02-19T15:28:46.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.474+0000 7f573be00640 1 -- 172.21.15.47:0/394207476 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57340a5ad0 con 0x7f573c0ceb40 2024-02-19T15:28:46.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.474+0000 7f573be00640 1 -- 172.21.15.47:0/394207476 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57340a5e60 con 0x7f573c0ceb40 2024-02-19T15:28:46.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.474+0000 7f5744000640 1 -- 172.21.15.47:0/394207476 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f573c0ceb40 msgr2=0x7f573c06bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:46.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.474+0000 7f5744000640 1 --2- 172.21.15.47:0/394207476 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f573c0ceb40 0x7f573c06bd60 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f5734009d90 tx=0x7f573401c8b0 comp rx=0 tx=0).stop 2024-02-19T15:28:46.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.474+0000 7f5744000640 1 -- 172.21.15.47:0/394207476 shutdown_connections 2024-02-19T15:28:46.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.474+0000 7f5744000640 1 --2- 172.21.15.47:0/394207476 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f573c0ceb40 0x7f573c06bd60 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:46.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.474+0000 7f5744000640 1 --2- 172.21.15.47:0/394207476 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f573c0cded0 0x7f573c0ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:46.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.474+0000 7f5744000640 1 -- 172.21.15.47:0/394207476 >> 172.21.15.47:0/394207476 conn(0x7f573c0d1bf0 msgr2=0x7f573c0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:46.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.474+0000 7f5744000640 1 -- 172.21.15.47:0/394207476 shutdown_connections 2024-02-19T15:28:46.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.474+0000 7f5744000640 1 -- 172.21.15.47:0/394207476 wait complete. 2024-02-19T15:28:46.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5744000640 1 Processor -- start 2024-02-19T15:28:46.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5744000640 1 -- start start 2024-02-19T15:28:46.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5744000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f573c06d650 0x7f573c148290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:46.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5744000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f573c0cded0 0x7f573c0c7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:46.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5744000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f573c0ceb40 0x7f573c09b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:46.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5744000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c1487d0 con 0x7f573c06d650 2024-02-19T15:28:46.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5744000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0c85f0 con 0x7f573c0cded0 2024-02-19T15:28:46.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5744000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573c09da10 con 0x7f573c0ceb40 2024-02-19T15:28:46.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5741600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f573c06d650 0x7f573c148290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:46.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5741600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f573c06d650 0x7f573c148290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40708/0 (socket says 172.21.15.47:40708) 2024-02-19T15:28:46.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5741600640 1 -- 172.21.15.47:0/3442442673 learned_addr learned my addr 172.21.15.47:0/3442442673 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:46.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5740c00640 1 --2- 172.21.15.47:0/3442442673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f573c0cded0 0x7f573c0c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:46.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5742000640 1 --2- 172.21.15.47:0/3442442673 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f573c0ceb40 0x7f573c09b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:46.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5741600640 1 -- 172.21.15.47:0/3442442673 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f573c0ceb40 msgr2=0x7f573c09b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:46.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5741600640 1 --2- 172.21.15.47:0/3442442673 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f573c0ceb40 0x7f573c09b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:46.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5741600640 1 -- 172.21.15.47:0/3442442673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f573c0cded0 msgr2=0x7f573c0c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:46.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5741600640 1 --2- 172.21.15.47:0/3442442673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f573c0cded0 0x7f573c0c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:46.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5741600640 1 -- 172.21.15.47:0/3442442673 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5734009a70 con 0x7f573c06d650 2024-02-19T15:28:46.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5740c00640 1 --2- 172.21.15.47:0/3442442673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f573c0cded0 0x7f573c0c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:46.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5741600640 1 --2- 172.21.15.47:0/3442442673 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f573c06d650 0x7f573c148290 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f573000c480 tx=0x7f573000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:46.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5739600640 1 -- 172.21.15.47:0/3442442673 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57300074e0 con 0x7f573c06d650 2024-02-19T15:28:46.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5739600640 1 -- 172.21.15.47:0/3442442673 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5730007660 con 0x7f573c06d650 2024-02-19T15:28:46.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5744000640 1 -- 172.21.15.47:0/3442442673 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573c0c8030 con 0x7f573c06d650 2024-02-19T15:28:46.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5739600640 1 -- 172.21.15.47:0/3442442673 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5730004a30 con 0x7f573c06d650 2024-02-19T15:28:46.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5744000640 1 -- 172.21.15.47:0/3442442673 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573c09dcb0 con 0x7f573c06d650 2024-02-19T15:28:46.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.478+0000 7f5744000640 1 -- 172.21.15.47:0/3442442673 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573c075360 con 0x7f573c06d650 2024-02-19T15:28:46.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.482+0000 7f5739600640 1 -- 172.21.15.47:0/3442442673 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5730004b90 con 0x7f573c06d650 2024-02-19T15:28:46.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.482+0000 7f5739600640 1 --2- 172.21.15.47:0/3442442673 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f571c066e40 0x7f571c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:46.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.482+0000 7f5739600640 1 -- 172.21.15.47:0/3442442673 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f5730083cd0 con 0x7f573c06d650 2024-02-19T15:28:46.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.482+0000 7f5740c00640 1 --2- 172.21.15.47:0/3442442673 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f571c066e40 0x7f571c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:46.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.482+0000 7f5740c00640 1 --2- 172.21.15.47:0/3442442673 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f571c066e40 0x7f571c069300 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f573c1fa300 tx=0x7f5728008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:46.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.486+0000 7f5739600640 1 -- 172.21.15.47:0/3442442673 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f57300548e0 con 0x7f573c06d650 2024-02-19T15:28:46.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.710+0000 7f5744000640 1 -- 172.21.15.47:0/3442442673 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f573c0d13a0 con 0x7f573c06d650 2024-02-19T15:28:46.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.942+0000 7f5739600640 1 -- 172.21.15.47:0/3442442673 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v366) v1 ==== 142+0+0 (secure 0 0 0) 0x7f5730018020 con 0x7f573c06d650 2024-02-19T15:28:46.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.946+0000 7f5744000640 1 -- 172.21.15.47:0/3442442673 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f571c066e40 msgr2=0x7f571c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:46.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.946+0000 7f5744000640 1 --2- 172.21.15.47:0/3442442673 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f571c066e40 0x7f571c069300 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f573c1fa300 tx=0x7f5728008040 comp rx=0 tx=0).stop 2024-02-19T15:28:46.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.946+0000 7f5744000640 1 -- 172.21.15.47:0/3442442673 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f573c06d650 msgr2=0x7f573c148290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:46.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.946+0000 7f5744000640 1 --2- 172.21.15.47:0/3442442673 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f573c06d650 0x7f573c148290 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f573000c480 tx=0x7f573000c940 comp rx=0 tx=0).stop 2024-02-19T15:28:46.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.946+0000 7f5744000640 1 -- 172.21.15.47:0/3442442673 shutdown_connections 2024-02-19T15:28:46.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.946+0000 7f5744000640 1 --2- 172.21.15.47:0/3442442673 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f571c066e40 0x7f571c069300 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:46.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.946+0000 7f5744000640 1 --2- 172.21.15.47:0/3442442673 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f573c0ceb40 0x7f573c09b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:46.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.946+0000 7f5744000640 1 --2- 172.21.15.47:0/3442442673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f573c0cded0 0x7f573c0c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:46.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.946+0000 7f5744000640 1 --2- 172.21.15.47:0/3442442673 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f573c06d650 0x7f573c148290 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:46.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.946+0000 7f5744000640 1 -- 172.21.15.47:0/3442442673 >> 172.21.15.47:0/3442442673 conn(0x7f573c0d1bf0 msgr2=0x7f573c0b51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:46.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.946+0000 7f5744000640 1 -- 172.21.15.47:0/3442442673 shutdown_connections 2024-02-19T15:28:46.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:46.946+0000 7f5744000640 1 -- 172.21.15.47:0/3442442673 wait complete. 2024-02-19T15:28:46.966 INFO:tasks.cephfs_test_runner:That removal of primary dentry of hardlinked inode results ... ok 2024-02-19T15:28:46.966 INFO:tasks.cephfs_test_runner:Starting test: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) Reproduce https://tracker.ceph.com/issues/53597 2024-02-19T15:28:46.966 INFO:tasks.cephfs_test_runner:test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-02-19T15:28:46.967 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:28:47.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.106+0000 7f3679a00640 1 Processor -- start 2024-02-19T15:28:47.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.106+0000 7f3679a00640 1 -- start start 2024-02-19T15:28:47.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.106+0000 7f3679a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f367415f510 0x7f3674070700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:47.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.106+0000 7f3679a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36740ae970 con 0x7f367415fa90 2024-02-19T15:28:47.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.106+0000 7f3679a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3674070c40 con 0x7f3674160620 2024-02-19T15:28:47.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.106+0000 7f3679a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36740cffc0 con 0x7f367415f510 2024-02-19T15:28:47.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.106+0000 7f3672a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f367415f510 0x7f3674070700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:47.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.106+0000 7f3672a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f367415f510 0x7f3674070700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40718/0 (socket says 172.21.15.47:40718) 2024-02-19T15:28:47.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.106+0000 7f3672a00640 1 -- 172.21.15.47:0/996076257 learned_addr learned my addr 172.21.15.47:0/996076257 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:47.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.106+0000 7f366be00640 1 --1- 172.21.15.47:0/996076257 >> v1:172.21.15.47:6789/0 conn(0x7f367415fa90 0x7f367415ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60738/0 (socket says 172.21.15.47:60738) 2024-02-19T15:28:47.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.106+0000 7f3672a00640 1 -- 172.21.15.47:0/996076257 >> v1:172.21.15.116:6789/0 conn(0x7f3674160620 legacy=0x7f36740af0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:47.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.106+0000 7f3672a00640 1 -- 172.21.15.47:0/996076257 >> v1:172.21.15.47:6789/0 conn(0x7f367415fa90 legacy=0x7f367415ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:47.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.106+0000 7f3672a00640 1 -- 172.21.15.47:0/996076257 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36740d0200 con 0x7f367415f510 2024-02-19T15:28:47.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.110+0000 7f3672a00640 1 --2- 172.21.15.47:0/996076257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f367415f510 0x7f3674070700 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f3664009870 tx=0x7f366401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=325fd2aa18809a1c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:47.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.110+0000 7f3672000640 1 -- 172.21.15.47:0/996076257 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36640ad680 con 0x7f367415f510 2024-02-19T15:28:47.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.110+0000 7f3672000640 1 -- 172.21.15.47:0/996076257 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36640ad7e0 con 0x7f367415f510 2024-02-19T15:28:47.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.110+0000 7f3672000640 1 -- 172.21.15.47:0/996076257 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36640adb70 con 0x7f367415f510 2024-02-19T15:28:47.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.110+0000 7f3679a00640 1 -- 172.21.15.47:0/996076257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f367415f510 msgr2=0x7f3674070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:47.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.110+0000 7f3679a00640 1 --2- 172.21.15.47:0/996076257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f367415f510 0x7f3674070700 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f3664009870 tx=0x7f366401c490 comp rx=0 tx=0).stop 2024-02-19T15:28:47.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.110+0000 7f3679a00640 1 -- 172.21.15.47:0/996076257 shutdown_connections 2024-02-19T15:28:47.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.110+0000 7f3679a00640 1 --2- 172.21.15.47:0/996076257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f367415f510 0x7f3674070700 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.110+0000 7f3679a00640 1 -- 172.21.15.47:0/996076257 >> 172.21.15.47:0/996076257 conn(0x7f36740b3690 msgr2=0x7f36740b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:47.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.110+0000 7f3679a00640 1 -- 172.21.15.47:0/996076257 shutdown_connections 2024-02-19T15:28:47.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.110+0000 7f3679a00640 1 -- 172.21.15.47:0/996076257 wait complete. 2024-02-19T15:28:47.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f3679a00640 1 Processor -- start 2024-02-19T15:28:47.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f3679a00640 1 -- start start 2024-02-19T15:28:47.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f3679a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f367415f510 0x7f3674167300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:47.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f3679a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f367415fa90 0x7f3674164f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:47.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f3679a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3674160620 0x7f36741654d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:47.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f3679a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3674165f10 con 0x7f367415f510 2024-02-19T15:28:47.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f3679a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3674167c50 con 0x7f3674160620 2024-02-19T15:28:47.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f3679a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367406d170 con 0x7f367415fa90 2024-02-19T15:28:47.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f3672a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f367415f510 0x7f3674167300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:47.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f3672a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f367415f510 0x7f3674167300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40722/0 (socket says 172.21.15.47:40722) 2024-02-19T15:28:47.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f3672a00640 1 -- 172.21.15.47:0/646091551 learned_addr learned my addr 172.21.15.47:0/646091551 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:47.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f366be00640 1 --2- 172.21.15.47:0/646091551 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f367415fa90 0x7f3674164f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:47.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f3673400640 1 --2- 172.21.15.47:0/646091551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3674160620 0x7f36741654d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:47.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f3672a00640 1 -- 172.21.15.47:0/646091551 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f367415fa90 msgr2=0x7f3674164f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:47.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f3672a00640 1 --2- 172.21.15.47:0/646091551 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f367415fa90 0x7f3674164f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f3672a00640 1 -- 172.21.15.47:0/646091551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3674160620 msgr2=0x7f36741654d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:47.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f3672a00640 1 --2- 172.21.15.47:0/646091551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3674160620 0x7f36741654d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f3672a00640 1 -- 172.21.15.47:0/646091551 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36640094d0 con 0x7f367415f510 2024-02-19T15:28:47.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f3672a00640 1 --2- 172.21.15.47:0/646091551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f367415f510 0x7f3674167300 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f36640097c0 tx=0x7f366401acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:47.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f366aa00640 1 -- 172.21.15.47:0/646091551 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36640b8450 con 0x7f367415f510 2024-02-19T15:28:47.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f366aa00640 1 -- 172.21.15.47:0/646091551 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36640b85b0 con 0x7f367415f510 2024-02-19T15:28:47.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f366aa00640 1 -- 172.21.15.47:0/646091551 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36640b8940 con 0x7f367415f510 2024-02-19T15:28:47.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f3679a00640 1 -- 172.21.15.47:0/646091551 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f367406d410 con 0x7f367415f510 2024-02-19T15:28:47.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.114+0000 7f3679a00640 1 -- 172.21.15.47:0/646091551 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3674167840 con 0x7f367415f510 2024-02-19T15:28:47.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.118+0000 7f3679a00640 1 -- 172.21.15.47:0/646091551 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3674075360 con 0x7f367415f510 2024-02-19T15:28:47.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.118+0000 7f366aa00640 1 -- 172.21.15.47:0/646091551 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f36640b4070 con 0x7f367415f510 2024-02-19T15:28:47.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.118+0000 7f366aa00640 1 --2- 172.21.15.47:0/646091551 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3650066e40 0x7f3650069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:47.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.118+0000 7f366aa00640 1 -- 172.21.15.47:0/646091551 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f366411ef50 con 0x7f367415f510 2024-02-19T15:28:47.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.118+0000 7f366be00640 1 --2- 172.21.15.47:0/646091551 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3650066e40 0x7f3650069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:47.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.118+0000 7f366be00640 1 --2- 172.21.15.47:0/646091551 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3650066e40 0x7f3650069300 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f36740c53b0 tx=0x7f365c007cf0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:47.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.122+0000 7f366aa00640 1 -- 172.21.15.47:0/646091551 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f36640ebef0 con 0x7f367415f510 2024-02-19T15:28:47.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.290+0000 7f3679a00640 1 -- 172.21.15.47:0/646091551 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36740b7b00 con 0x7f367415f510 2024-02-19T15:28:47.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.290+0000 7f366aa00640 1 -- 172.21.15.47:0/646091551 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+13300 (secure 0 0 0) 0x7f36640ef5b0 con 0x7f367415f510 2024-02-19T15:28:47.290 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:47.290 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":117,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:28:45.939317+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.47:6834/3636833601":"2024-02-20T15:28:42.928139+0000","172.21.15.116:6837/819751716":"2024-02-20T15:27:39.101989+0000","172.21.15.116:6835/819751716":"2024-02-20T15:27:39.101989+0000","172.21.15.47:6835/3636833601":"2024-02-20T15:28:42.928139+0000","172.21.15.47:6834/897369244":"2024-02-20T15:27:38.263490+0000","172.21.15.47:6832/897369244":"2024-02-20T15:27:38.263490+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:28:47.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.290+0000 7f3679a00640 1 -- 172.21.15.47:0/646091551 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3650066e40 msgr2=0x7f3650069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:47.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.290+0000 7f3679a00640 1 --2- 172.21.15.47:0/646091551 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3650066e40 0x7f3650069300 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f36740c53b0 tx=0x7f365c007cf0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.290+0000 7f3679a00640 1 -- 172.21.15.47:0/646091551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f367415f510 msgr2=0x7f3674167300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:47.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.290+0000 7f3679a00640 1 --2- 172.21.15.47:0/646091551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f367415f510 0x7f3674167300 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f36640097c0 tx=0x7f366401acd0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.290+0000 7f3679a00640 1 -- 172.21.15.47:0/646091551 shutdown_connections 2024-02-19T15:28:47.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.290+0000 7f3679a00640 1 --2- 172.21.15.47:0/646091551 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3650066e40 0x7f3650069300 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.290+0000 7f3679a00640 1 --2- 172.21.15.47:0/646091551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3674160620 0x7f36741654d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.290+0000 7f3679a00640 1 --2- 172.21.15.47:0/646091551 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f367415fa90 0x7f3674164f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.290+0000 7f3679a00640 1 --2- 172.21.15.47:0/646091551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f367415f510 0x7f3674167300 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.290+0000 7f3679a00640 1 -- 172.21.15.47:0/646091551 >> 172.21.15.47:0/646091551 conn(0x7f36740b3690 msgr2=0x7f36740c5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:47.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.290+0000 7f3679a00640 1 -- 172.21.15.47:0/646091551 shutdown_connections 2024-02-19T15:28:47.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.290+0000 7f3679a00640 1 -- 172.21.15.47:0/646091551 wait complete. 2024-02-19T15:28:47.311 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:28:47.311 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:28:47.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.454+0000 7f3c01600640 1 Processor -- start 2024-02-19T15:28:47.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.454+0000 7f3c01600640 1 -- start start 2024-02-19T15:28:47.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.454+0000 7f3c01600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3bfc15af10 0x7f3bfc06e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:47.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.454+0000 7f3c01600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc0cd880 0x7f3bfc0cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:47.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.454+0000 7f3c01600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc071c70 con 0x7f3bfc0ca8f0 2024-02-19T15:28:47.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.454+0000 7f3c01600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bfc06e8a0 con 0x7f3bfc15af10 2024-02-19T15:28:47.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.454+0000 7f3c01600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc06cef0 con 0x7f3bfc0cd880 2024-02-19T15:28:47.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.454+0000 7f3bfaa00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3bfc15af10 0x7f3bfc06e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:47.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.454+0000 7f3bfb400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc0cd880 0x7f3bfc0cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:47.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.454+0000 7f3bfaa00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3bfc15af10 0x7f3bfc06e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:53286/0 (socket says 172.21.15.47:53286) 2024-02-19T15:28:47.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.454+0000 7f3bfaa00640 1 -- 172.21.15.47:0/1597524108 learned_addr learned my addr 172.21.15.47:0/1597524108 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:47.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.454+0000 7f3bfb400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc0cd880 0x7f3bfc0cdd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:57358/0 (socket says 172.21.15.47:57358) 2024-02-19T15:28:47.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.458+0000 7f3bfb400640 1 -- 172.21.15.47:0/1597524108 >> v2:172.21.15.116:3300/0 conn(0x7f3bfc15af10 msgr2=0x7f3bfc06e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:47.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.458+0000 7f3bfb400640 1 --2- 172.21.15.47:0/1597524108 >> v2:172.21.15.116:3300/0 conn(0x7f3bfc15af10 0x7f3bfc06e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.458+0000 7f3bfb400640 1 -- 172.21.15.47:0/1597524108 >> v1:172.21.15.116:6789/0 conn(0x7f3bfc0ca8f0 legacy=0x7f3bfc0cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:47.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.458+0000 7f3bfb400640 1 -- 172.21.15.47:0/1597524108 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc0ca610 con 0x7f3bfc0cd880 2024-02-19T15:28:47.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.458+0000 7f3bfaa00640 1 --2- 172.21.15.47:0/1597524108 >> v2:172.21.15.116:3300/0 conn(0x7f3bfc15af10 0x7f3bfc06e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:28:47.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.458+0000 7f3bfb400640 1 --2- 172.21.15.47:0/1597524108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc0cd880 0x7f3bfc0cdd10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4009d90 tx=0x7f3bf401c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19880752fd11f2d5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:47.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.458+0000 7f3bf9600640 1 -- 172.21.15.47:0/1597524108 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf40a5970 con 0x7f3bfc0cd880 2024-02-19T15:28:47.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.458+0000 7f3bf9600640 1 -- 172.21.15.47:0/1597524108 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf40a5ad0 con 0x7f3bfc0cd880 2024-02-19T15:28:47.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.458+0000 7f3bf9600640 1 -- 172.21.15.47:0/1597524108 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf40a5e60 con 0x7f3bfc0cd880 2024-02-19T15:28:47.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.458+0000 7f3c01600640 1 -- 172.21.15.47:0/1597524108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc0cd880 msgr2=0x7f3bfc0cdd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:47.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3c01600640 1 --2- 172.21.15.47:0/1597524108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc0cd880 0x7f3bfc0cdd10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4009d90 tx=0x7f3bf401c8b0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3c01600640 1 -- 172.21.15.47:0/1597524108 shutdown_connections 2024-02-19T15:28:47.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3c01600640 1 --2- 172.21.15.47:0/1597524108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc0cd880 0x7f3bfc0cdd10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3c01600640 1 --2- 172.21.15.47:0/1597524108 >> v2:172.21.15.116:3300/0 conn(0x7f3bfc15af10 0x7f3bfc06e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3c01600640 1 -- 172.21.15.47:0/1597524108 >> 172.21.15.47:0/1597524108 conn(0x7f3bfc0cfc20 msgr2=0x7f3bfc0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:47.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3c01600640 1 -- 172.21.15.47:0/1597524108 shutdown_connections 2024-02-19T15:28:47.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3c01600640 1 -- 172.21.15.47:0/1597524108 wait complete. 2024-02-19T15:28:47.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3c01600640 1 Processor -- start 2024-02-19T15:28:47.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3c01600640 1 -- start start 2024-02-19T15:28:47.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3c01600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bfc0ca8f0 0x7f3bfc153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:47.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3c01600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc0cd880 0x7f3bfc0c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:47.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3c01600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bfc15af10 0x7f3bfc0c8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:47.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3c01600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc15a7f0 con 0x7f3bfc15af10 2024-02-19T15:28:47.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3c01600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc159a30 con 0x7f3bfc0ca8f0 2024-02-19T15:28:47.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3c01600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc159d10 con 0x7f3bfc0cd880 2024-02-19T15:28:47.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3bfb400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bfc15af10 0x7f3bfc0c8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:47.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3bfb400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bfc15af10 0x7f3bfc0c8830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40728/0 (socket says 172.21.15.47:40728) 2024-02-19T15:28:47.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3bfb400640 1 -- 172.21.15.47:0/3878443785 learned_addr learned my addr 172.21.15.47:0/3878443785 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:47.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3bfaa00640 1 --2- 172.21.15.47:0/3878443785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bfc0ca8f0 0x7f3bfc153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:47.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3bfa000640 1 --2- 172.21.15.47:0/3878443785 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc0cd880 0x7f3bfc0c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:47.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3bfb400640 1 -- 172.21.15.47:0/3878443785 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc0cd880 msgr2=0x7f3bfc0c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:47.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3bfb400640 1 --2- 172.21.15.47:0/3878443785 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc0cd880 0x7f3bfc0c82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3bfb400640 1 -- 172.21.15.47:0/3878443785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bfc0ca8f0 msgr2=0x7f3bfc153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:47.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3bfb400640 1 --2- 172.21.15.47:0/3878443785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bfc0ca8f0 0x7f3bfc153150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3bfb400640 1 -- 172.21.15.47:0/3878443785 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bec00b770 con 0x7f3bfc15af10 2024-02-19T15:28:47.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.462+0000 7f3bfb400640 1 --2- 172.21.15.47:0/3878443785 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bfc15af10 0x7f3bfc0c8830 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f3bf401a4e0 tx=0x7f3bf4019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:47.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.466+0000 7f3bf2a00640 1 -- 172.21.15.47:0/3878443785 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf40add70 con 0x7f3bfc15af10 2024-02-19T15:28:47.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.466+0000 7f3bf2a00640 1 -- 172.21.15.47:0/3878443785 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf4009450 con 0x7f3bfc15af10 2024-02-19T15:28:47.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.466+0000 7f3bf2a00640 1 -- 172.21.15.47:0/3878443785 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf40b9690 con 0x7f3bfc15af10 2024-02-19T15:28:47.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.466+0000 7f3c01600640 1 -- 172.21.15.47:0/3878443785 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf4009a70 con 0x7f3bfc15af10 2024-02-19T15:28:47.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.466+0000 7f3c01600640 1 -- 172.21.15.47:0/3878443785 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bfc15a2d0 con 0x7f3bfc15af10 2024-02-19T15:28:47.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.466+0000 7f3c01600640 1 -- 172.21.15.47:0/3878443785 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bfc075360 con 0x7f3bfc15af10 2024-02-19T15:28:47.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.466+0000 7f3bf2a00640 1 -- 172.21.15.47:0/3878443785 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3bfc0cea60 con 0x7f3bfc15af10 2024-02-19T15:28:47.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.466+0000 7f3bf2a00640 1 --2- 172.21.15.47:0/3878443785 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3bdc066e70 0x7f3bdc069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:47.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.466+0000 7f3bf2a00640 1 -- 172.21.15.47:0/3878443785 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f3bf411fc90 con 0x7f3bfc15af10 2024-02-19T15:28:47.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.466+0000 7f3bfaa00640 1 --2- 172.21.15.47:0/3878443785 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3bdc066e70 0x7f3bdc069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:47.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.470+0000 7f3bfaa00640 1 --2- 172.21.15.47:0/3878443785 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3bdc066e70 0x7f3bdc069330 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f3bec006cb0 tx=0x7f3bec00b450 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:47.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.470+0000 7f3bf2a00640 1 -- 172.21.15.47:0/3878443785 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f3bf40ecd80 con 0x7f3bfc15af10 2024-02-19T15:28:47.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.634+0000 7f3c01600640 1 -- 172.21.15.47:0/3878443785 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3bfc0d0740 con 0x7f3bfc15af10 2024-02-19T15:28:47.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.634+0000 7f3bf2a00640 1 -- 172.21.15.47:0/3878443785 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3bf40bd0a0 con 0x7f3bfc15af10 2024-02-19T15:28:47.635 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:28:47.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.634+0000 7f3c01600640 1 -- 172.21.15.47:0/3878443785 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3bdc066e70 msgr2=0x7f3bdc069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:47.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.634+0000 7f3c01600640 1 --2- 172.21.15.47:0/3878443785 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3bdc066e70 0x7f3bdc069330 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f3bec006cb0 tx=0x7f3bec00b450 comp rx=0 tx=0).stop 2024-02-19T15:28:47.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.634+0000 7f3c01600640 1 -- 172.21.15.47:0/3878443785 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bfc15af10 msgr2=0x7f3bfc0c8830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:47.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.634+0000 7f3c01600640 1 --2- 172.21.15.47:0/3878443785 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bfc15af10 0x7f3bfc0c8830 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f3bf401a4e0 tx=0x7f3bf4019650 comp rx=0 tx=0).stop 2024-02-19T15:28:47.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.638+0000 7f3c01600640 1 -- 172.21.15.47:0/3878443785 shutdown_connections 2024-02-19T15:28:47.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.638+0000 7f3c01600640 1 --2- 172.21.15.47:0/3878443785 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3bdc066e70 0x7f3bdc069330 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.638+0000 7f3c01600640 1 --2- 172.21.15.47:0/3878443785 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bfc15af10 0x7f3bfc0c8830 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.638+0000 7f3c01600640 1 --2- 172.21.15.47:0/3878443785 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc0cd880 0x7f3bfc0c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.638+0000 7f3c01600640 1 --2- 172.21.15.47:0/3878443785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bfc0ca8f0 0x7f3bfc153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.638+0000 7f3c01600640 1 -- 172.21.15.47:0/3878443785 >> 172.21.15.47:0/3878443785 conn(0x7f3bfc0cfc20 msgr2=0x7f3bfc0c51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:47.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.638+0000 7f3c01600640 1 -- 172.21.15.47:0/3878443785 shutdown_connections 2024-02-19T15:28:47.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.638+0000 7f3c01600640 1 -- 172.21.15.47:0/3878443785 wait complete. 2024-02-19T15:28:47.656 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-02-19T15:28:47.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.794+0000 7fbfc5c00640 1 Processor -- start 2024-02-19T15:28:47.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.794+0000 7fbfc5c00640 1 -- start start 2024-02-19T15:28:47.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.798+0000 7fbfc5c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbfc00717e0 0x7fbfc00b1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:47.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.798+0000 7fbfc5c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbfc015ba70 0x7fbfc015a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:47.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.798+0000 7fbfc5c00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc00b1a50 con 0x7fbfc0071400 2024-02-19T15:28:47.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.798+0000 7fbfc5c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfc00b2010 con 0x7fbfc015ba70 2024-02-19T15:28:47.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.798+0000 7fbfc5c00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfc00b1d30 con 0x7fbfc00717e0 2024-02-19T15:28:47.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.798+0000 7fbfc5200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbfc015ba70 0x7fbfc015a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:47.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.798+0000 7fbfbfe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbfc0071400 0x7fbfc0070990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60746/0 (socket says 172.21.15.47:60746) 2024-02-19T15:28:47.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.798+0000 7fbfbfe00640 1 -- 172.21.15.47:0/3390427100 learned_addr learned my addr 172.21.15.47:0/3390427100 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:47.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.798+0000 7fbfb7e00640 1 --2- 172.21.15.47:0/3390427100 >> v2:172.21.15.116:3300/0 conn(0x7fbfc00717e0 0x7fbfc00b1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:47.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.798+0000 7fbfc5200640 1 -- 172.21.15.47:0/3390427100 >> v2:172.21.15.116:3300/0 conn(0x7fbfc00717e0 msgr2=0x7fbfc00b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:47.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.798+0000 7fbfc5200640 1 --2- 172.21.15.47:0/3390427100 >> v2:172.21.15.116:3300/0 conn(0x7fbfc00717e0 0x7fbfc00b1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.798+0000 7fbfc5200640 1 -- 172.21.15.47:0/3390427100 >> v1:172.21.15.47:6789/0 conn(0x7fbfc0071400 legacy=0x7fbfc0070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:47.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.798+0000 7fbfc5200640 1 -- 172.21.15.47:0/3390427100 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc015b270 con 0x7fbfc015ba70 2024-02-19T15:28:47.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.798+0000 7fbfb7e00640 1 --2- 172.21.15.47:0/3390427100 >> v2:172.21.15.116:3300/0 conn(0x7fbfc00717e0 0x7fbfc00b1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:47.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.802+0000 7fbfc5200640 1 --2- 172.21.15.47:0/3390427100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfc015ba70 0x7fbfc015a4d0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fbfc00cd1a0 tx=0x7fbfb801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ce033ea7383b3b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:47.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.802+0000 7fbfbf400640 1 -- 172.21.15.47:0/3390427100 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb80a59e0 con 0x7fbfc015ba70 2024-02-19T15:28:47.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.802+0000 7fbfbf400640 1 -- 172.21.15.47:0/3390427100 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb80a5b40 con 0x7fbfc015ba70 2024-02-19T15:28:47.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.802+0000 7fbfbf400640 1 -- 172.21.15.47:0/3390427100 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb80ad440 con 0x7fbfc015ba70 2024-02-19T15:28:47.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.802+0000 7fbfc5c00640 1 -- 172.21.15.47:0/3390427100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfc015ba70 msgr2=0x7fbfc015a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:47.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.802+0000 7fbfc5c00640 1 --2- 172.21.15.47:0/3390427100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfc015ba70 0x7fbfc015a4d0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fbfc00cd1a0 tx=0x7fbfb801c900 comp rx=0 tx=0).stop 2024-02-19T15:28:47.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.802+0000 7fbfc5c00640 1 -- 172.21.15.47:0/3390427100 shutdown_connections 2024-02-19T15:28:47.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.802+0000 7fbfc5c00640 1 --2- 172.21.15.47:0/3390427100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfc015ba70 0x7fbfc015a4d0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.802+0000 7fbfc5c00640 1 --2- 172.21.15.47:0/3390427100 >> v2:172.21.15.116:3300/0 conn(0x7fbfc00717e0 0x7fbfc00b1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.802+0000 7fbfc5c00640 1 -- 172.21.15.47:0/3390427100 >> 172.21.15.47:0/3390427100 conn(0x7fbfc00cf7c0 msgr2=0x7fbfc00cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:47.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.802+0000 7fbfc5c00640 1 -- 172.21.15.47:0/3390427100 shutdown_connections 2024-02-19T15:28:47.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.802+0000 7fbfc5c00640 1 -- 172.21.15.47:0/3390427100 wait complete. 2024-02-19T15:28:47.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.802+0000 7fbfc5c00640 1 Processor -- start 2024-02-19T15:28:47.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.802+0000 7fbfc5c00640 1 -- start start 2024-02-19T15:28:47.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.802+0000 7fbfc5c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfc0071400 0x7fbfc015d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:47.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.802+0000 7fbfc5c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfc00717e0 0x7fbfc01441b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:47.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.802+0000 7fbfc5c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfc015ba70 0x7fbfc0154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:47.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.802+0000 7fbfc5c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc015c360 con 0x7fbfc0071400 2024-02-19T15:28:47.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.802+0000 7fbfc5c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc015c600 con 0x7fbfc00717e0 2024-02-19T15:28:47.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.802+0000 7fbfc5c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc015c8e0 con 0x7fbfc015ba70 2024-02-19T15:28:47.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.806+0000 7fbfbfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfc0071400 0x7fbfc015d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:47.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.806+0000 7fbfbfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfc0071400 0x7fbfc015d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40734/0 (socket says 172.21.15.47:40734) 2024-02-19T15:28:47.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.806+0000 7fbfbfe00640 1 -- 172.21.15.47:0/2592030542 learned_addr learned my addr 172.21.15.47:0/2592030542 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:47.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.806+0000 7fbfb7e00640 1 --2- 172.21.15.47:0/2592030542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfc00717e0 0x7fbfc01441b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:47.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.806+0000 7fbfc5200640 1 --2- 172.21.15.47:0/2592030542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfc015ba70 0x7fbfc0154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:47.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.806+0000 7fbfbfe00640 1 -- 172.21.15.47:0/2592030542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfc015ba70 msgr2=0x7fbfc0154bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:47.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.806+0000 7fbfbfe00640 1 --2- 172.21.15.47:0/2592030542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfc015ba70 0x7fbfc0154bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.806+0000 7fbfbfe00640 1 -- 172.21.15.47:0/2592030542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfc00717e0 msgr2=0x7fbfc01441b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:47.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.806+0000 7fbfbfe00640 1 --2- 172.21.15.47:0/2592030542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfc00717e0 0x7fbfc01441b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:47.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.806+0000 7fbfbfe00640 1 -- 172.21.15.47:0/2592030542 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb8009a70 con 0x7fbfc0071400 2024-02-19T15:28:47.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.806+0000 7fbfbfe00640 1 --2- 172.21.15.47:0/2592030542 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfc0071400 0x7fbfc015d3c0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fbfb000c420 tx=0x7fbfb000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:47.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.806+0000 7fbfbcc00640 1 -- 172.21.15.47:0/2592030542 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0005cf0 con 0x7fbfc0071400 2024-02-19T15:28:47.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.806+0000 7fbfbcc00640 1 -- 172.21.15.47:0/2592030542 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb0005e50 con 0x7fbfc0071400 2024-02-19T15:28:47.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.806+0000 7fbfbcc00640 1 -- 172.21.15.47:0/2592030542 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb00072f0 con 0x7fbfc0071400 2024-02-19T15:28:47.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.806+0000 7fbfc5c00640 1 -- 172.21.15.47:0/2592030542 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc015cbc0 con 0x7fbfc0071400 2024-02-19T15:28:47.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.806+0000 7fbfc5c00640 1 -- 172.21.15.47:0/2592030542 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc01446f0 con 0x7fbfc0071400 2024-02-19T15:28:47.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.806+0000 7fbfc5c00640 1 -- 172.21.15.47:0/2592030542 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc0075360 con 0x7fbfc0071400 2024-02-19T15:28:47.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.806+0000 7fbfbcc00640 1 -- 172.21.15.47:0/2592030542 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fbfb0007450 con 0x7fbfc0071400 2024-02-19T15:28:47.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.806+0000 7fbfbcc00640 1 --2- 172.21.15.47:0/2592030542 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbf9c066e90 0x7fbf9c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:47.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.810+0000 7fbfbcc00640 1 -- 172.21.15.47:0/2592030542 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fbfb0083d00 con 0x7fbfc0071400 2024-02-19T15:28:47.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.810+0000 7fbfb7e00640 1 --2- 172.21.15.47:0/2592030542 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbf9c066e90 0x7fbf9c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:47.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.810+0000 7fbfb7e00640 1 --2- 172.21.15.47:0/2592030542 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbf9c066e90 0x7fbf9c069350 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fbfc00cf0a0 tx=0x7fbfa8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:47.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:47.810+0000 7fbfbcc00640 1 -- 172.21.15.47:0/2592030542 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fbfb0050e60 con 0x7fbfc0071400 2024-02-19T15:28:48.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:48.034+0000 7fbfc5c00640 1 -- 172.21.15.47:0/2592030542 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) v1 -- 0x7fbfc006e000 con 0x7fbfc0071400 2024-02-19T15:28:48.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:48.950+0000 7fbfbcc00640 1 -- 172.21.15.47:0/2592030542 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v368) v1 ==== 143+0+0 (secure 0 0 0) 0x7fbfb0017020 con 0x7fbfc0071400 2024-02-19T15:28:48.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:48.954+0000 7fbfc5c00640 1 -- 172.21.15.47:0/2592030542 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbf9c066e90 msgr2=0x7fbf9c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:48.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:48.954+0000 7fbfc5c00640 1 --2- 172.21.15.47:0/2592030542 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbf9c066e90 0x7fbf9c069350 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fbfc00cf0a0 tx=0x7fbfa8008040 comp rx=0 tx=0).stop 2024-02-19T15:28:48.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:48.954+0000 7fbfc5c00640 1 -- 172.21.15.47:0/2592030542 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfc0071400 msgr2=0x7fbfc015d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:48.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:48.954+0000 7fbfc5c00640 1 --2- 172.21.15.47:0/2592030542 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfc0071400 0x7fbfc015d3c0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fbfb000c420 tx=0x7fbfb000c8e0 comp rx=0 tx=0).stop 2024-02-19T15:28:48.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:48.954+0000 7fbfc5c00640 1 -- 172.21.15.47:0/2592030542 shutdown_connections 2024-02-19T15:28:48.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:48.954+0000 7fbfc5c00640 1 --2- 172.21.15.47:0/2592030542 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbf9c066e90 0x7fbf9c069350 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:48.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:48.954+0000 7fbfc5c00640 1 --2- 172.21.15.47:0/2592030542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfc015ba70 0x7fbfc0154bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:48.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:48.954+0000 7fbfc5c00640 1 --2- 172.21.15.47:0/2592030542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfc00717e0 0x7fbfc01441b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:48.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:48.954+0000 7fbfc5c00640 1 --2- 172.21.15.47:0/2592030542 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfc0071400 0x7fbfc015d3c0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:48.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:48.954+0000 7fbfc5c00640 1 -- 172.21.15.47:0/2592030542 >> 172.21.15.47:0/2592030542 conn(0x7fbfc00cf7c0 msgr2=0x7fbfc00c8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:48.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:48.954+0000 7fbfc5c00640 1 -- 172.21.15.47:0/2592030542 shutdown_connections 2024-02-19T15:28:48.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:48.954+0000 7fbfc5c00640 1 -- 172.21.15.47:0/2592030542 wait complete. 2024-02-19T15:28:48.978 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-19T15:28:49.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.126+0000 7f9d42800640 1 Processor -- start 2024-02-19T15:28:49.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.126+0000 7f9d42800640 1 -- start start 2024-02-19T15:28:49.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.126+0000 7f9d42800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d3c0d2650 0x7f9d3c06c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:49.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.126+0000 7f9d42800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d3c0cc620 con 0x7f9d3c0ce3b0 2024-02-19T15:28:49.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.126+0000 7f9d42800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d3c0d12e0 con 0x7f9d3c15e280 2024-02-19T15:28:49.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.126+0000 7f9d42800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c0d1940 con 0x7f9d3c0d2650 2024-02-19T15:28:49.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.126+0000 7f9d3b400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f9d3c0ce3b0 0x7f9d3c0ce790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60750/0 (socket says 172.21.15.47:60750) 2024-02-19T15:28:49.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.126+0000 7f9d3b400640 1 -- 172.21.15.47:0/1879418294 learned_addr learned my addr 172.21.15.47:0/1879418294 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:49.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.126+0000 7f9d33e00640 1 --2- 172.21.15.47:0/1879418294 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d3c0d2650 0x7f9d3c06c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:49.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.126+0000 7f9d3aa00640 1 -- 172.21.15.47:0/1879418294 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111694203 0 0) 0x7f9d3c0cc620 con 0x7f9d3c0ce3b0 2024-02-19T15:28:49.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.126+0000 7f9d3aa00640 1 -- 172.21.15.47:0/1879418294 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d20003530 con 0x7f9d3c0ce3b0 2024-02-19T15:28:49.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.126+0000 7f9d33e00640 1 -- 172.21.15.47:0/1879418294 >> v1:172.21.15.116:6789/0 conn(0x7f9d3c15e280 legacy=0x7f9d3c06c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:49.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.126+0000 7f9d3aa00640 1 -- 172.21.15.47:0/1879418294 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 350817499 0 0) 0x7f9d20003530 con 0x7f9d3c0ce3b0 2024-02-19T15:28:49.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.126+0000 7f9d33e00640 1 -- 172.21.15.47:0/1879418294 >> v1:172.21.15.47:6789/0 conn(0x7f9d3c0ce3b0 legacy=0x7f9d3c0ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:49.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.126+0000 7f9d33e00640 1 -- 172.21.15.47:0/1879418294 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3c0d1000 con 0x7f9d3c0d2650 2024-02-19T15:28:49.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.130+0000 7f9d33e00640 1 --2- 172.21.15.47:0/1879418294 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d3c0d2650 0x7f9d3c06c1d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9d24009770 tx=0x7f9d2401be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b02705ece352bf63 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:49.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.130+0000 7f9d3aa00640 1 -- 172.21.15.47:0/1879418294 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c001680 con 0x7f9d3c0d2650 2024-02-19T15:28:49.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.130+0000 7f9d3aa00640 1 -- 172.21.15.47:0/1879418294 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d240a8440 con 0x7f9d3c0d2650 2024-02-19T15:28:49.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.130+0000 7f9d3aa00640 1 -- 172.21.15.47:0/1879418294 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d240a87d0 con 0x7f9d3c0d2650 2024-02-19T15:28:49.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.130+0000 7f9d42800640 1 -- 172.21.15.47:0/1879418294 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d3c0d2650 msgr2=0x7f9d3c06c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:49.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.130+0000 7f9d42800640 1 --2- 172.21.15.47:0/1879418294 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d3c0d2650 0x7f9d3c06c1d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9d24009770 tx=0x7f9d2401be80 comp rx=0 tx=0).stop 2024-02-19T15:28:49.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.130+0000 7f9d42800640 1 -- 172.21.15.47:0/1879418294 shutdown_connections 2024-02-19T15:28:49.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.130+0000 7f9d42800640 1 --2- 172.21.15.47:0/1879418294 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d3c0d2650 0x7f9d3c06c1d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:49.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.130+0000 7f9d42800640 1 -- 172.21.15.47:0/1879418294 >> 172.21.15.47:0/1879418294 conn(0x7f9d3c0d1b30 msgr2=0x7f9d3c0d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:49.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.130+0000 7f9d42800640 1 -- 172.21.15.47:0/1879418294 shutdown_connections 2024-02-19T15:28:49.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.130+0000 7f9d42800640 1 -- 172.21.15.47:0/1879418294 wait complete. 2024-02-19T15:28:49.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.134+0000 7f9d42800640 1 Processor -- start 2024-02-19T15:28:49.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.134+0000 7f9d42800640 1 -- start start 2024-02-19T15:28:49.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.134+0000 7f9d42800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d3c0ce3b0 0x7f9d3c14fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:49.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.134+0000 7f9d42800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d3c0d2650 0x7f9d3c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:49.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.134+0000 7f9d42800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d3c15e280 0x7f9d3c0cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:49.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.134+0000 7f9d42800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c15f360 con 0x7f9d3c0ce3b0 2024-02-19T15:28:49.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.134+0000 7f9d42800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c152bb0 con 0x7f9d3c0d2650 2024-02-19T15:28:49.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.134+0000 7f9d42800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c161900 con 0x7f9d3c15e280 2024-02-19T15:28:49.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.134+0000 7f9d3b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d3c0ce3b0 0x7f9d3c14fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:49.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.134+0000 7f9d3b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d3c0ce3b0 0x7f9d3c14fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40738/0 (socket says 172.21.15.47:40738) 2024-02-19T15:28:49.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.134+0000 7f9d3b400640 1 -- 172.21.15.47:0/2943636546 learned_addr learned my addr 172.21.15.47:0/2943636546 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:49.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.134+0000 7f9d33e00640 1 --2- 172.21.15.47:0/2943636546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d3c0d2650 0x7f9d3c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:49.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.134+0000 7f9d3be00640 1 --2- 172.21.15.47:0/2943636546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d3c15e280 0x7f9d3c0cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:49.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.134+0000 7f9d3b400640 1 -- 172.21.15.47:0/2943636546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d3c15e280 msgr2=0x7f9d3c0cae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:49.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.134+0000 7f9d3b400640 1 --2- 172.21.15.47:0/2943636546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d3c15e280 0x7f9d3c0cae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:49.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.134+0000 7f9d3b400640 1 -- 172.21.15.47:0/2943636546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d3c0d2650 msgr2=0x7f9d3c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:49.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.134+0000 7f9d3b400640 1 --2- 172.21.15.47:0/2943636546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d3c0d2650 0x7f9d3c15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:49.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.134+0000 7f9d3b400640 1 -- 172.21.15.47:0/2943636546 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d24009420 con 0x7f9d3c0ce3b0 2024-02-19T15:28:49.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.134+0000 7f9d3b400640 1 --2- 172.21.15.47:0/2943636546 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d3c0ce3b0 0x7f9d3c14fb50 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c00c550 tx=0x7f9d2c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:49.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.134+0000 7f9d33400640 1 -- 172.21.15.47:0/2943636546 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c0074e0 con 0x7f9d3c0ce3b0 2024-02-19T15:28:49.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.138+0000 7f9d33400640 1 -- 172.21.15.47:0/2943636546 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2c007640 con 0x7f9d3c0ce3b0 2024-02-19T15:28:49.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.138+0000 7f9d33400640 1 -- 172.21.15.47:0/2943636546 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c004a30 con 0x7f9d3c0ce3b0 2024-02-19T15:28:49.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.138+0000 7f9d42800640 1 -- 172.21.15.47:0/2943636546 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d3c161620 con 0x7f9d3c0ce3b0 2024-02-19T15:28:49.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.138+0000 7f9d42800640 1 -- 172.21.15.47:0/2943636546 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d3c0c9d10 con 0x7f9d3c0ce3b0 2024-02-19T15:28:49.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.138+0000 7f9d42800640 1 -- 172.21.15.47:0/2943636546 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3c075360 con 0x7f9d3c0ce3b0 2024-02-19T15:28:49.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.138+0000 7f9d33400640 1 -- 172.21.15.47:0/2943636546 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9d2c004b90 con 0x7f9d3c0ce3b0 2024-02-19T15:28:49.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.138+0000 7f9d33400640 1 --2- 172.21.15.47:0/2943636546 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9d18066e40 0x7f9d18069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:49.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.138+0000 7f9d33400640 1 -- 172.21.15.47:0/2943636546 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f9d2c083a30 con 0x7f9d3c0ce3b0 2024-02-19T15:28:49.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.142+0000 7f9d33e00640 1 --2- 172.21.15.47:0/2943636546 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9d18066e40 0x7f9d18069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:49.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.142+0000 7f9d33e00640 1 --2- 172.21.15.47:0/2943636546 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9d18066e40 0x7f9d18069300 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9d240ad000 tx=0x7f9d24008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:49.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.146+0000 7f9d33400640 1 -- 172.21.15.47:0/2943636546 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f9d2c050a70 con 0x7f9d3c0ce3b0 2024-02-19T15:28:49.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.306+0000 7f9d42800640 1 -- 172.21.15.47:0/2943636546 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9d3c0cdf80 con 0x7f9d3c0ce3b0 2024-02-19T15:28:49.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.310+0000 7f9d33400640 1 -- 172.21.15.47:0/2943636546 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9d2c018020 con 0x7f9d3c0ce3b0 2024-02-19T15:28:49.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.314+0000 7f9d42800640 1 -- 172.21.15.47:0/2943636546 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9d18066e40 msgr2=0x7f9d18069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:49.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.314+0000 7f9d42800640 1 --2- 172.21.15.47:0/2943636546 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9d18066e40 0x7f9d18069300 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9d240ad000 tx=0x7f9d24008040 comp rx=0 tx=0).stop 2024-02-19T15:28:49.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.314+0000 7f9d42800640 1 -- 172.21.15.47:0/2943636546 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d3c0ce3b0 msgr2=0x7f9d3c14fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:49.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.314+0000 7f9d42800640 1 --2- 172.21.15.47:0/2943636546 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d3c0ce3b0 0x7f9d3c14fb50 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c00c550 tx=0x7f9d2c00ca10 comp rx=0 tx=0).stop 2024-02-19T15:28:49.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.314+0000 7f9d42800640 1 -- 172.21.15.47:0/2943636546 shutdown_connections 2024-02-19T15:28:49.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.314+0000 7f9d42800640 1 --2- 172.21.15.47:0/2943636546 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9d18066e40 0x7f9d18069300 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:49.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.314+0000 7f9d42800640 1 --2- 172.21.15.47:0/2943636546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d3c15e280 0x7f9d3c0cae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:49.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.314+0000 7f9d42800640 1 --2- 172.21.15.47:0/2943636546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d3c0d2650 0x7f9d3c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:49.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.314+0000 7f9d42800640 1 --2- 172.21.15.47:0/2943636546 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d3c0ce3b0 0x7f9d3c14fb50 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:49.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.314+0000 7f9d42800640 1 -- 172.21.15.47:0/2943636546 >> 172.21.15.47:0/2943636546 conn(0x7f9d3c0d1b30 msgr2=0x7f9d3c0c6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:49.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.314+0000 7f9d42800640 1 -- 172.21.15.47:0/2943636546 shutdown_connections 2024-02-19T15:28:49.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.314+0000 7f9d42800640 1 -- 172.21.15.47:0/2943636546 wait complete. 2024-02-19T15:28:49.333 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T15:28:49.333 DEBUG:teuthology.orchestra.run.smithi047:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:28:49.435 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:28:49.524 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T15:28:49.524 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:28:49.524 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:28:49.533 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:28:49.533 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:28:49.533 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:28:49.542 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:28:49.542 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:28:49.542 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:28:49.584 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3908 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-02-19T15:28:49.585 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-02-19T15:28:49.585 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:28:49.593 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354616k,nr_inodes=4088654,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8557 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-02-19T15:28:49.593 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-02-19T15:28:49.594 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:49.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.766+0000 7fa803e00640 1 Processor -- start 2024-02-19T15:28:49.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.766+0000 7fa803e00640 1 -- start start 2024-02-19T15:28:49.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.770+0000 7fa803e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa7fc15b1a0 0x7fa7fc15b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:49.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.770+0000 7fa803e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7fc0718c0 con 0x7fa7fc071350 2024-02-19T15:28:49.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.770+0000 7fa803e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7fc071ba0 con 0x7fa7fc06c9a0 2024-02-19T15:28:49.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.770+0000 7fa803e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7fc071e80 con 0x7fa7fc15b1a0 2024-02-19T15:28:49.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.770+0000 7fa801400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa7fc15b1a0 0x7fa7fc15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:49.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.770+0000 7fa801400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa7fc15b1a0 0x7fa7fc15b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40746/0 (socket says 172.21.15.47:40746) 2024-02-19T15:28:49.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.770+0000 7fa800a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa7fc071350 0x7fa7fc15c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60758/0 (socket says 172.21.15.47:60758) 2024-02-19T15:28:49.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.770+0000 7fa801400640 1 -- 172.21.15.47:0/760958345 learned_addr learned my addr 172.21.15.47:0/760958345 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:49.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.770+0000 7fa801400640 1 -- 172.21.15.47:0/760958345 >> v1:172.21.15.116:6789/0 conn(0x7fa7fc06c9a0 legacy=0x7fa7fc15cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:49.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.770+0000 7fa801400640 1 -- 172.21.15.47:0/760958345 >> v1:172.21.15.47:6789/0 conn(0x7fa7fc071350 legacy=0x7fa7fc15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:49.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.770+0000 7fa801400640 1 -- 172.21.15.47:0/760958345 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7fc072160 con 0x7fa7fc15b1a0 2024-02-19T15:28:49.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.774+0000 7fa801400640 1 --2- 172.21.15.47:0/760958345 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7fc15b1a0 0x7fa7fc15b580 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fa7f00098a0 tx=0x7fa7f001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79da0b1eccb9d932 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:49.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.774+0000 7fa7fbe00640 1 -- 172.21.15.47:0/760958345 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f00aa5f0 con 0x7fa7fc15b1a0 2024-02-19T15:28:49.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.774+0000 7fa7fbe00640 1 -- 172.21.15.47:0/760958345 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f00aa750 con 0x7fa7fc15b1a0 2024-02-19T15:28:49.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.774+0000 7fa7fbe00640 1 -- 172.21.15.47:0/760958345 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f00aaae0 con 0x7fa7fc15b1a0 2024-02-19T15:28:49.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.774+0000 7fa803e00640 1 -- 172.21.15.47:0/760958345 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7fc15b1a0 msgr2=0x7fa7fc15b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:49.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.774+0000 7fa803e00640 1 --2- 172.21.15.47:0/760958345 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7fc15b1a0 0x7fa7fc15b580 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fa7f00098a0 tx=0x7fa7f001c500 comp rx=0 tx=0).stop 2024-02-19T15:28:49.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.774+0000 7fa803e00640 1 -- 172.21.15.47:0/760958345 shutdown_connections 2024-02-19T15:28:49.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.774+0000 7fa803e00640 1 --2- 172.21.15.47:0/760958345 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7fc15b1a0 0x7fa7fc15b580 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:49.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.774+0000 7fa803e00640 1 -- 172.21.15.47:0/760958345 >> 172.21.15.47:0/760958345 conn(0x7fa7fc0b16a0 msgr2=0x7fa7fc0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:49.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.774+0000 7fa803e00640 1 -- 172.21.15.47:0/760958345 shutdown_connections 2024-02-19T15:28:49.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.774+0000 7fa803e00640 1 -- 172.21.15.47:0/760958345 wait complete. 2024-02-19T15:28:49.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.774+0000 7fa803e00640 1 Processor -- start 2024-02-19T15:28:49.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.774+0000 7fa803e00640 1 -- start start 2024-02-19T15:28:49.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa803e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7fc06c9a0 0x7fa7fc1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:49.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa803e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7fc071350 0x7fa7fc139e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:49.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa803e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7fc15b1a0 0x7fa7fc15c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:49.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa803e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc14d7d0 con 0x7fa7fc06c9a0 2024-02-19T15:28:49.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa803e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc0cf230 con 0x7fa7fc071350 2024-02-19T15:28:49.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa803e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc139b30 con 0x7fa7fc15b1a0 2024-02-19T15:28:49.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa801400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7fc06c9a0 0x7fa7fc1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:49.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa801400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7fc06c9a0 0x7fa7fc1395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40756/0 (socket says 172.21.15.47:40756) 2024-02-19T15:28:49.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa801400640 1 -- 172.21.15.47:0/3543941494 learned_addr learned my addr 172.21.15.47:0/3543941494 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:49.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa800a00640 1 --2- 172.21.15.47:0/3543941494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7fc071350 0x7fa7fc139e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:49.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa801e00640 1 --2- 172.21.15.47:0/3543941494 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7fc15b1a0 0x7fa7fc15c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:49.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa801400640 1 -- 172.21.15.47:0/3543941494 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7fc15b1a0 msgr2=0x7fa7fc15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:49.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa801400640 1 --2- 172.21.15.47:0/3543941494 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7fc15b1a0 0x7fa7fc15c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:49.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa801400640 1 -- 172.21.15.47:0/3543941494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7fc071350 msgr2=0x7fa7fc139e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:49.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa801400640 1 --2- 172.21.15.47:0/3543941494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7fc071350 0x7fa7fc139e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:49.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa801400640 1 -- 172.21.15.47:0/3543941494 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f00094d0 con 0x7fa7fc06c9a0 2024-02-19T15:28:49.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa801e00640 1 --2- 172.21.15.47:0/3543941494 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7fc15b1a0 0x7fa7fc15c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:49.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa801400640 1 --2- 172.21.15.47:0/3543941494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7fc06c9a0 0x7fa7fc1395f0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fa7f00ac040 tx=0x7fa7f001ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:49.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa7f9600640 1 -- 172.21.15.47:0/3543941494 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f00aa5f0 con 0x7fa7fc06c9a0 2024-02-19T15:28:49.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa7f9600640 1 -- 172.21.15.47:0/3543941494 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f00b3800 con 0x7fa7fc06c9a0 2024-02-19T15:28:49.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa7f9600640 1 -- 172.21.15.47:0/3543941494 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f00b3bb0 con 0x7fa7fc06c9a0 2024-02-19T15:28:49.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa803e00640 1 -- 172.21.15.47:0/3543941494 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7fc0cec70 con 0x7fa7fc06c9a0 2024-02-19T15:28:49.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa803e00640 1 -- 172.21.15.47:0/3543941494 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7fc14ed60 con 0x7fa7fc06c9a0 2024-02-19T15:28:49.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa803e00640 1 -- 172.21.15.47:0/3543941494 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7fc075360 con 0x7fa7fc06c9a0 2024-02-19T15:28:49.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.778+0000 7fa7f9600640 1 -- 172.21.15.47:0/3543941494 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa7f00b3d10 con 0x7fa7fc06c9a0 2024-02-19T15:28:49.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.782+0000 7fa7f9600640 1 --2- 172.21.15.47:0/3543941494 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa7dc066e90 0x7fa7dc069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:49.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.782+0000 7fa7f9600640 1 -- 172.21.15.47:0/3543941494 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fa7f00a7070 con 0x7fa7fc06c9a0 2024-02-19T15:28:49.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.782+0000 7fa800a00640 1 --2- 172.21.15.47:0/3543941494 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa7dc066e90 0x7fa7dc069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:49.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.782+0000 7fa800a00640 1 --2- 172.21.15.47:0/3543941494 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa7dc066e90 0x7fa7dc069350 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc14e2e0 tx=0x7fa7e8007cd0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:49.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.786+0000 7fa7f9600640 1 -- 172.21.15.47:0/3543941494 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa7f00ec2f0 con 0x7fa7fc06c9a0 2024-02-19T15:28:49.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.982+0000 7fa803e00640 1 -- 172.21.15.47:0/3543941494 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7fc154090 con 0x7fa7fc06c9a0 2024-02-19T15:28:49.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.982+0000 7fa7f9600640 1 -- 172.21.15.47:0/3543941494 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+3699 (secure 0 0 0) 0x7fa7f00ef9b0 con 0x7fa7fc06c9a0 2024-02-19T15:28:49.984 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:49.984 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":84,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7552,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/3447418179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3447418179},{"type":"v1","addr":"172.21.15.116:6836","nonce":3447418179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7555,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/183664699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":183664699},{"type":"v1","addr":"172.21.15.116:6837","nonce":183664699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7914,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/720669924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":720669924},{"type":"v1","addr":"172.21.15.47:6833","nonce":720669924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":8271,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2272085647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2272085647},{"type":"v1","addr":"172.21.15.47:6835","nonce":2272085647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":84}],"filesystems":[]} 2024-02-19T15:28:49.984 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 84 2024-02-19T15:28:49.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.986+0000 7fa803e00640 1 -- 172.21.15.47:0/3543941494 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa7dc066e90 msgr2=0x7fa7dc069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:49.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.986+0000 7fa803e00640 1 --2- 172.21.15.47:0/3543941494 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa7dc066e90 0x7fa7dc069350 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc14e2e0 tx=0x7fa7e8007cd0 comp rx=0 tx=0).stop 2024-02-19T15:28:49.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.986+0000 7fa803e00640 1 -- 172.21.15.47:0/3543941494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7fc06c9a0 msgr2=0x7fa7fc1395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:49.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.986+0000 7fa803e00640 1 --2- 172.21.15.47:0/3543941494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7fc06c9a0 0x7fa7fc1395f0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fa7f00ac040 tx=0x7fa7f001ad40 comp rx=0 tx=0).stop 2024-02-19T15:28:49.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.986+0000 7fa803e00640 1 -- 172.21.15.47:0/3543941494 shutdown_connections 2024-02-19T15:28:49.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.986+0000 7fa803e00640 1 --2- 172.21.15.47:0/3543941494 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa7dc066e90 0x7fa7dc069350 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:49.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.986+0000 7fa803e00640 1 --2- 172.21.15.47:0/3543941494 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7fc15b1a0 0x7fa7fc15c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:49.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.986+0000 7fa803e00640 1 --2- 172.21.15.47:0/3543941494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7fc071350 0x7fa7fc139e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:49.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.986+0000 7fa803e00640 1 --2- 172.21.15.47:0/3543941494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7fc06c9a0 0x7fa7fc1395f0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:49.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.986+0000 7fa803e00640 1 -- 172.21.15.47:0/3543941494 >> 172.21.15.47:0/3543941494 conn(0x7fa7fc0b16a0 msgr2=0x7fa7fc0aed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:49.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.986+0000 7fa803e00640 1 -- 172.21.15.47:0/3543941494 shutdown_connections 2024-02-19T15:28:49.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:49.986+0000 7fa803e00640 1 -- 172.21.15.47:0/3543941494 wait complete. 2024-02-19T15:28:50.005 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T15:28:50.005 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-19T15:28:50.005 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T15:28:50.005 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:28:50.006 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T15:28:50.006 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-19T15:28:50.006 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T15:28:50.006 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:28:50.006 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T15:28:50.006 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-19T15:28:50.006 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T15:28:50.007 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:28:50.007 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T15:28:50.007 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-19T15:28:50.007 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T15:28:50.007 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:28:50.019 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T15:28:50.019 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T15:28:50.021 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T15:28:50.021 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T15:28:50.023 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T15:28:50.023 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T15:28:50.024 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T15:28:50.024 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T15:28:50.066 INFO:tasks.ceph.mds.a:Started 2024-02-19T15:28:50.066 DEBUG:teuthology.parallel:result is None 2024-02-19T15:28:50.067 INFO:tasks.ceph.mds.d:Started 2024-02-19T15:28:50.067 DEBUG:teuthology.parallel:result is None 2024-02-19T15:28:50.069 INFO:tasks.ceph.mds.c:Started 2024-02-19T15:28:50.069 DEBUG:teuthology.parallel:result is None 2024-02-19T15:28:50.069 INFO:tasks.ceph.mds.b:Started 2024-02-19T15:28:50.070 DEBUG:teuthology.parallel:result is None 2024-02-19T15:28:50.070 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-19T15:28:50.154 INFO:tasks.ceph.mds.a.smithi047.stdout:starting mds.a at 2024-02-19T15:28:50.154 INFO:tasks.ceph.mds.c.smithi047.stdout:starting mds.c at 2024-02-19T15:28:50.157 INFO:tasks.ceph.mds.b.smithi116.stdout:starting mds.b at 2024-02-19T15:28:50.157 INFO:tasks.ceph.mds.d.smithi116.stdout:starting mds.d at 2024-02-19T15:28:50.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.214+0000 7f1d66e00640 1 Processor -- start 2024-02-19T15:28:50.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.214+0000 7f1d66e00640 1 -- start start 2024-02-19T15:28:50.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.218+0000 7f1d66e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1d6015caa0 0x7f1d600727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:50.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.218+0000 7f1d66e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6015d280 con 0x7f1d600cea80 2024-02-19T15:28:50.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.218+0000 7f1d66e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d60070ff0 con 0x7f1d60145590 2024-02-19T15:28:50.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.218+0000 7f1d66e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d600709e0 con 0x7f1d6015caa0 2024-02-19T15:28:50.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.218+0000 7f1d57400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1d600cea80 0x7f1d6015c390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60770/0 (socket says 172.21.15.47:60770) 2024-02-19T15:28:50.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.218+0000 7f1d57400640 1 -- 172.21.15.47:0/1867711326 learned_addr learned my addr 172.21.15.47:0/1867711326 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:50.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.218+0000 7f1d64e00640 1 --2- 172.21.15.47:0/1867711326 >> v2:172.21.15.116:3300/0 conn(0x7f1d6015caa0 0x7f1d600727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:50.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.218+0000 7f1d5f400640 1 -- 172.21.15.47:0/1867711326 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062202535 0 0) 0x7f1d6015d280 con 0x7f1d600cea80 2024-02-19T15:28:50.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.218+0000 7f1d5f400640 1 -- 172.21.15.47:0/1867711326 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d44003530 con 0x7f1d600cea80 2024-02-19T15:28:50.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.218+0000 7f1d64e00640 1 -- 172.21.15.47:0/1867711326 >> v1:172.21.15.116:6789/0 conn(0x7f1d60145590 legacy=0x7f1d6006c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:50.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.218+0000 7f1d64e00640 1 -- 172.21.15.47:0/1867711326 >> v1:172.21.15.47:6789/0 conn(0x7f1d600cea80 legacy=0x7f1d6015c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:50.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.218+0000 7f1d64e00640 1 -- 172.21.15.47:0/1867711326 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d600cf020 con 0x7f1d6015caa0 2024-02-19T15:28:50.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.218+0000 7f1d64e00640 1 --2- 172.21.15.47:0/1867711326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d6015caa0 0x7f1d600727b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009d90 tx=0x7f1d5801c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4d81063b6da5d8a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:50.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d5f400640 1 -- 172.21.15.47:0/1867711326 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d580a5940 con 0x7f1d6015caa0 2024-02-19T15:28:50.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d5f400640 1 -- 172.21.15.47:0/1867711326 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d580a5aa0 con 0x7f1d6015caa0 2024-02-19T15:28:50.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d5f400640 1 -- 172.21.15.47:0/1867711326 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d580a5e10 con 0x7f1d6015caa0 2024-02-19T15:28:50.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d66e00640 1 -- 172.21.15.47:0/1867711326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d6015caa0 msgr2=0x7f1d600727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:50.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d66e00640 1 --2- 172.21.15.47:0/1867711326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d6015caa0 0x7f1d600727b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009d90 tx=0x7f1d5801c970 comp rx=0 tx=0).stop 2024-02-19T15:28:50.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d66e00640 1 -- 172.21.15.47:0/1867711326 shutdown_connections 2024-02-19T15:28:50.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d66e00640 1 --2- 172.21.15.47:0/1867711326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d6015caa0 0x7f1d600727b0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:50.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d66e00640 1 -- 172.21.15.47:0/1867711326 >> 172.21.15.47:0/1867711326 conn(0x7f1d600b16a0 msgr2=0x7f1d600b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:50.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d66e00640 1 -- 172.21.15.47:0/1867711326 shutdown_connections 2024-02-19T15:28:50.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d66e00640 1 -- 172.21.15.47:0/1867711326 wait complete. 2024-02-19T15:28:50.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d66e00640 1 Processor -- start 2024-02-19T15:28:50.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d66e00640 1 -- start start 2024-02-19T15:28:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d66e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d600cea80 0x7f1d6015af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d66e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d60145590 0x7f1d6013aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d66e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d6015caa0 0x7f1d60154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d66e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6015c0a0 con 0x7f1d60145590 2024-02-19T15:28:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d66e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6015bb20 con 0x7f1d6015caa0 2024-02-19T15:28:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d66e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6015bdc0 con 0x7f1d600cea80 2024-02-19T15:28:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d57400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d60145590 0x7f1d6013aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d57400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d60145590 0x7f1d6013aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40812/0 (socket says 172.21.15.47:40812) 2024-02-19T15:28:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d57400640 1 -- 172.21.15.47:0/2014090987 learned_addr learned my addr 172.21.15.47:0/2014090987 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d64e00640 1 --2- 172.21.15.47:0/2014090987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d6015caa0 0x7f1d60154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d5fe00640 1 --2- 172.21.15.47:0/2014090987 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d600cea80 0x7f1d6015af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:50.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d57400640 1 -- 172.21.15.47:0/2014090987 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d600cea80 msgr2=0x7f1d6015af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:50.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d57400640 1 --2- 172.21.15.47:0/2014090987 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d600cea80 0x7f1d6015af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:50.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d57400640 1 -- 172.21.15.47:0/2014090987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d6015caa0 msgr2=0x7f1d60154cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:50.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d57400640 1 --2- 172.21.15.47:0/2014090987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d6015caa0 0x7f1d60154cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:50.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d57400640 1 -- 172.21.15.47:0/2014090987 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d58009a70 con 0x7f1d60145590 2024-02-19T15:28:50.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.222+0000 7f1d5fe00640 1 --2- 172.21.15.47:0/2014090987 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d600cea80 0x7f1d6015af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:50.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.226+0000 7f1d57400640 1 --2- 172.21.15.47:0/2014090987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d60145590 0x7f1d6013aa50 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f1d4800c4d0 tx=0x7f1d4800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:50.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.226+0000 7f1d5cc00640 1 -- 172.21.15.47:0/2014090987 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d48005cf0 con 0x7f1d60145590 2024-02-19T15:28:50.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.226+0000 7f1d5cc00640 1 -- 172.21.15.47:0/2014090987 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d48005e50 con 0x7f1d60145590 2024-02-19T15:28:50.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.226+0000 7f1d66e00640 1 -- 172.21.15.47:0/2014090987 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d6015a750 con 0x7f1d60145590 2024-02-19T15:28:50.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.226+0000 7f1d5cc00640 1 -- 172.21.15.47:0/2014090987 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d48007270 con 0x7f1d60145590 2024-02-19T15:28:50.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.226+0000 7f1d66e00640 1 -- 172.21.15.47:0/2014090987 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d6013dd60 con 0x7f1d60145590 2024-02-19T15:28:50.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.226+0000 7f1d66e00640 1 -- 172.21.15.47:0/2014090987 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d6015a5b0 con 0x7f1d60145590 2024-02-19T15:28:50.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.226+0000 7f1d5cc00640 1 -- 172.21.15.47:0/2014090987 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1d480073d0 con 0x7f1d60145590 2024-02-19T15:28:50.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.226+0000 7f1d5cc00640 1 --2- 172.21.15.47:0/2014090987 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d3c066e40 0x7f1d3c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:50.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.226+0000 7f1d5cc00640 1 -- 172.21.15.47:0/2014090987 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f1d48083d70 con 0x7f1d60145590 2024-02-19T15:28:50.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.226+0000 7f1d5fe00640 1 --2- 172.21.15.47:0/2014090987 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d3c066e40 0x7f1d3c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:50.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.230+0000 7f1d5fe00640 1 --2- 172.21.15.47:0/2014090987 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d3c066e40 0x7f1d3c069300 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f1d50001710 tx=0x7f1d50009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:50.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.230+0000 7f1d5cc00640 1 -- 172.21.15.47:0/2014090987 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1d6015a5b0 con 0x7f1d60145590 2024-02-19T15:28:50.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.406+0000 7f1d66e00640 1 -- 172.21.15.47:0/2014090987 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f1d6015a5b0 con 0x7f1d60145590 2024-02-19T15:28:50.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.966+0000 7f1d5cc00640 1 -- 172.21.15.47:0/2014090987 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v119) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1d6015a5b0 con 0x7f1d60145590 2024-02-19T15:28:50.968 INFO:teuthology.orchestra.run.smithi047.stderr: removed all blocklist entries 2024-02-19T15:28:50.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.970+0000 7f1d66e00640 1 -- 172.21.15.47:0/2014090987 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d3c066e40 msgr2=0x7f1d3c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:50.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.970+0000 7f1d66e00640 1 --2- 172.21.15.47:0/2014090987 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d3c066e40 0x7f1d3c069300 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f1d50001710 tx=0x7f1d50009040 comp rx=0 tx=0).stop 2024-02-19T15:28:50.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.970+0000 7f1d66e00640 1 -- 172.21.15.47:0/2014090987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d60145590 msgr2=0x7f1d6013aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:50.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.970+0000 7f1d66e00640 1 --2- 172.21.15.47:0/2014090987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d60145590 0x7f1d6013aa50 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f1d4800c4d0 tx=0x7f1d4800c990 comp rx=0 tx=0).stop 2024-02-19T15:28:50.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.970+0000 7f1d66e00640 1 -- 172.21.15.47:0/2014090987 shutdown_connections 2024-02-19T15:28:50.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.970+0000 7f1d66e00640 1 --2- 172.21.15.47:0/2014090987 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1d3c066e40 0x7f1d3c069300 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:50.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.970+0000 7f1d66e00640 1 --2- 172.21.15.47:0/2014090987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d6015caa0 0x7f1d60154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:50.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.970+0000 7f1d66e00640 1 --2- 172.21.15.47:0/2014090987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d60145590 0x7f1d6013aa50 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:50.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.970+0000 7f1d66e00640 1 --2- 172.21.15.47:0/2014090987 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d600cea80 0x7f1d6015af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:50.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.970+0000 7f1d66e00640 1 -- 172.21.15.47:0/2014090987 >> 172.21.15.47:0/2014090987 conn(0x7f1d600b16a0 msgr2=0x7f1d60142d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:50.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.970+0000 7f1d66e00640 1 -- 172.21.15.47:0/2014090987 shutdown_connections 2024-02-19T15:28:50.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:50.970+0000 7f1d66e00640 1 -- 172.21.15.47:0/2014090987 wait complete. 2024-02-19T15:28:50.999 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-19T15:28:51.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.142+0000 7fec1ec00640 1 Processor -- start 2024-02-19T15:28:51.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.142+0000 7fec1ec00640 1 -- start start 2024-02-19T15:28:51.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.146+0000 7fec1ec00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fec1806fdf0 0x7fec180adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:51.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.146+0000 7fec1ec00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec1806e7c0 0x7fec1806aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:51.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.146+0000 7fec1ec00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec18161780 con 0x7fec181613a0 2024-02-19T15:28:51.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.146+0000 7fec1ec00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec180b01a0 con 0x7fec1806fdf0 2024-02-19T15:28:51.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.146+0000 7fec1ec00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec180ae3c0 con 0x7fec1806e7c0 2024-02-19T15:28:51.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.146+0000 7fec17e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fec181613a0 0x7fec1806e0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60784/0 (socket says 172.21.15.47:60784) 2024-02-19T15:28:51.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.146+0000 7fec17e00640 1 -- 172.21.15.47:0/4051347352 learned_addr learned my addr 172.21.15.47:0/4051347352 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:51.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.146+0000 7fec0fe00640 1 --2- 172.21.15.47:0/4051347352 >> v2:172.21.15.116:3300/0 conn(0x7fec1806fdf0 0x7fec180adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:51.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.146+0000 7fec1cc00640 1 --2- 172.21.15.47:0/4051347352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec1806e7c0 0x7fec1806aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:51.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.146+0000 7fec17400640 1 -- 172.21.15.47:0/4051347352 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586262774 0 0) 0x7fec18161780 con 0x7fec181613a0 2024-02-19T15:28:51.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.146+0000 7fec17400640 1 -- 172.21.15.47:0/4051347352 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febfc003530 con 0x7fec181613a0 2024-02-19T15:28:51.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.146+0000 7fec0fe00640 1 -- 172.21.15.47:0/4051347352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec1806e7c0 msgr2=0x7fec1806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:51.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.146+0000 7fec0fe00640 1 --2- 172.21.15.47:0/4051347352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec1806e7c0 0x7fec1806aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.146+0000 7fec17400640 1 -- 172.21.15.47:0/4051347352 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3514944805 0 0) 0x7febfc003530 con 0x7fec181613a0 2024-02-19T15:28:51.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.146+0000 7fec0fe00640 1 -- 172.21.15.47:0/4051347352 >> v1:172.21.15.47:6789/0 conn(0x7fec181613a0 legacy=0x7fec1806e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:51.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.146+0000 7fec0fe00640 1 -- 172.21.15.47:0/4051347352 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec180ae6a0 con 0x7fec1806fdf0 2024-02-19T15:28:51.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.146+0000 7fec1cc00640 1 --2- 172.21.15.47:0/4051347352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec1806e7c0 0x7fec1806aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:28:51.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.150+0000 7fec0fe00640 1 --2- 172.21.15.47:0/4051347352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec1806fdf0 0x7fec180adbb0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fec000097c0 tx=0x7fec0001c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8151ce803e8e3ab server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:51.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.150+0000 7fec17400640 1 -- 172.21.15.47:0/4051347352 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec08001680 con 0x7fec1806fdf0 2024-02-19T15:28:51.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.150+0000 7fec17400640 1 -- 172.21.15.47:0/4051347352 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec000a7050 con 0x7fec1806fdf0 2024-02-19T15:28:51.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.150+0000 7fec17400640 1 -- 172.21.15.47:0/4051347352 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec000ab620 con 0x7fec1806fdf0 2024-02-19T15:28:51.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.150+0000 7fec1ec00640 1 -- 172.21.15.47:0/4051347352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec1806fdf0 msgr2=0x7fec180adbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:51.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.150+0000 7fec1ec00640 1 --2- 172.21.15.47:0/4051347352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec1806fdf0 0x7fec180adbb0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fec000097c0 tx=0x7fec0001c330 comp rx=0 tx=0).stop 2024-02-19T15:28:51.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.150+0000 7fec1ec00640 1 -- 172.21.15.47:0/4051347352 shutdown_connections 2024-02-19T15:28:51.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.150+0000 7fec1ec00640 1 --2- 172.21.15.47:0/4051347352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec1806e7c0 0x7fec1806aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.150+0000 7fec1ec00640 1 --2- 172.21.15.47:0/4051347352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec1806fdf0 0x7fec180adbb0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.150+0000 7fec1ec00640 1 -- 172.21.15.47:0/4051347352 >> 172.21.15.47:0/4051347352 conn(0x7fec180d0ad0 msgr2=0x7fec180d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:51.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.150+0000 7fec1ec00640 1 -- 172.21.15.47:0/4051347352 shutdown_connections 2024-02-19T15:28:51.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.150+0000 7fec1ec00640 1 -- 172.21.15.47:0/4051347352 wait complete. 2024-02-19T15:28:51.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.150+0000 7fec1ec00640 1 Processor -- start 2024-02-19T15:28:51.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.150+0000 7fec1ec00640 1 -- start start 2024-02-19T15:28:51.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec1ec00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec1806e7c0 0x7fec18153300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:51.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec1ec00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1806fdf0 0x7fec18153840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:51.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec1ec00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec181613a0 0x7fec18153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:51.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec1ec00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec180cd1e0 con 0x7fec1806fdf0 2024-02-19T15:28:51.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec1ec00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec18159a30 con 0x7fec181613a0 2024-02-19T15:28:51.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec1ec00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec18159d10 con 0x7fec1806e7c0 2024-02-19T15:28:51.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec0fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1806fdf0 0x7fec18153840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:51.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec0fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1806fdf0 0x7fec18153840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40816/0 (socket says 172.21.15.47:40816) 2024-02-19T15:28:51.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec0fe00640 1 -- 172.21.15.47:0/449969572 learned_addr learned my addr 172.21.15.47:0/449969572 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:51.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec1cc00640 1 --2- 172.21.15.47:0/449969572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec181613a0 0x7fec18153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:51.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec17e00640 1 --2- 172.21.15.47:0/449969572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec1806e7c0 0x7fec18153300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:51.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec0fe00640 1 -- 172.21.15.47:0/449969572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec1806e7c0 msgr2=0x7fec18153300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:51.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec0fe00640 1 --2- 172.21.15.47:0/449969572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec1806e7c0 0x7fec18153300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec0fe00640 1 -- 172.21.15.47:0/449969572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec181613a0 msgr2=0x7fec18153d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:51.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec0fe00640 1 --2- 172.21.15.47:0/449969572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec181613a0 0x7fec18153d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec0fe00640 1 -- 172.21.15.47:0/449969572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec10009a70 con 0x7fec1806fdf0 2024-02-19T15:28:51.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec17e00640 1 --2- 172.21.15.47:0/449969572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec1806e7c0 0x7fec18153300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:51.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec0fe00640 1 --2- 172.21.15.47:0/449969572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1806fdf0 0x7fec18153840 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fec000ae040 tx=0x7fec0001ab70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:51.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec14c00640 1 -- 172.21.15.47:0/449969572 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec00006e20 con 0x7fec1806fdf0 2024-02-19T15:28:51.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec14c00640 1 -- 172.21.15.47:0/449969572 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec000b7450 con 0x7fec1806fdf0 2024-02-19T15:28:51.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec14c00640 1 -- 172.21.15.47:0/449969572 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec000b7800 con 0x7fec1806fdf0 2024-02-19T15:28:51.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec1ec00640 1 -- 172.21.15.47:0/449969572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec00009420 con 0x7fec1806fdf0 2024-02-19T15:28:51.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec1ec00640 1 -- 172.21.15.47:0/449969572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec180cd4c0 con 0x7fec1806fdf0 2024-02-19T15:28:51.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.154+0000 7fec1ec00640 1 -- 172.21.15.47:0/449969572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec18075360 con 0x7fec1806fdf0 2024-02-19T15:28:51.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.158+0000 7fec14c00640 1 -- 172.21.15.47:0/449969572 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fec000b7960 con 0x7fec1806fdf0 2024-02-19T15:28:51.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.158+0000 7fec14c00640 1 --2- 172.21.15.47:0/449969572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7febf4066e40 0x7febf4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:51.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.158+0000 7fec14c00640 1 -- 172.21.15.47:0/449969572 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fec0011eea0 con 0x7fec1806fdf0 2024-02-19T15:28:51.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.158+0000 7fec17e00640 1 --2- 172.21.15.47:0/449969572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7febf4066e40 0x7febf4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:51.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.158+0000 7fec17e00640 1 --2- 172.21.15.47:0/449969572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7febf4066e40 0x7febf4069300 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fec08007a10 tx=0x7fec08009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:51.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.162+0000 7fec14c00640 1 -- 172.21.15.47:0/449969572 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fec000ebf40 con 0x7fec1806fdf0 2024-02-19T15:28:51.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.382+0000 7fec1ec00640 1 -- 172.21.15.47:0/449969572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fec1815a1c0 con 0x7fec1806fdf0 2024-02-19T15:28:51.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.382+0000 7fec14c00640 1 -- 172.21.15.47:0/449969572 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fec000ef600 con 0x7fec1806fdf0 2024-02-19T15:28:51.383 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:51.383 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:28:51.383 INFO:teuthology.orchestra.run.smithi047.stdout: "auth_dump": [ 2024-02-19T15:28:51.383 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:51.383 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.a", 2024-02-19T15:28:51.383 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNl8BP+HRAAlJ3Jk+1k2XcZAnICzy++Ow==", 2024-02-19T15:28:51.383 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:28:51.383 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:28:51.383 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:28:51.383 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:28:51.384 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:28:51.384 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:51.384 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:51.384 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:51.384 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.b", 2024-02-19T15:28:51.384 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlIOh3KhAAlnU2ZFY4iv/XEw0WqxyZpw==", 2024-02-19T15:28:51.384 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:28:51.384 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:28:51.384 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:28:51.384 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:28:51.384 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:28:51.384 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:51.384 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:51.385 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:51.385 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.c", 2024-02-19T15:28:51.385 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNl48SHJRAAn4GCT5KnP+2PD8IXbGClCg==", 2024-02-19T15:28:51.385 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:28:51.385 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:28:51.385 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:28:51.385 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:28:51.385 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:28:51.385 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:51.385 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:51.385 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:51.385 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mds.d", 2024-02-19T15:28:51.385 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNljcekMRAA414655SzH43fBA0AtjpxZw==", 2024-02-19T15:28:51.385 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:28:51.385 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:28:51.386 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:28:51.386 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:28:51.386 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:28:51.386 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:51.386 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:51.386 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:51.386 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.0", 2024-02-19T15:28:51.386 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDlcdNl/bycFhAAmzYi/OxEPUW3tcRJC0DNEQ==", 2024-02-19T15:28:51.386 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:28:51.386 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:28:51.386 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:28:51.386 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:28:51.386 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:51.386 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:51.386 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:51.386 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.1", 2024-02-19T15:28:51.387 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDocdNluquhHBAA88aLHCvVol7JoZB0MtoutA==", 2024-02-19T15:28:51.387 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:28:51.387 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:28:51.387 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:28:51.387 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:28:51.387 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:51.387 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:51.387 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:51.387 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.2", 2024-02-19T15:28:51.387 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDrcdNlGAnWIhAAyELvCPKjtfgF0+kX6NQFIA==", 2024-02-19T15:28:51.387 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:28:51.387 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:28:51.387 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:28:51.387 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:28:51.387 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:51.387 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:51.388 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:51.388 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.3", 2024-02-19T15:28:51.388 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDucdNlROZ6KBAAxTCDuYsw5SRIZBcclzwxhg==", 2024-02-19T15:28:51.388 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:28:51.388 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:28:51.388 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:28:51.388 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:28:51.388 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:51.388 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:51.388 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:51.388 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.4", 2024-02-19T15:28:51.388 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD0cdNluOGILBAA66R9RVXD2vdyIw+oZ+iM1g==", 2024-02-19T15:28:51.388 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:28:51.388 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:28:51.388 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:28:51.388 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:28:51.389 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:51.389 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:51.389 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:51.389 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.5", 2024-02-19T15:28:51.389 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD3cdNlHg8dMhAA2LUzfpvvHR4i8e5RBCeGpQ==", 2024-02-19T15:28:51.389 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:28:51.389 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:28:51.389 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:28:51.389 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:28:51.389 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:51.389 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:51.389 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:51.390 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.6", 2024-02-19T15:28:51.390 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD6cdNlXD1EORAAhKGIt0wFeCyoE6QkkZ3caA==", 2024-02-19T15:28:51.390 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:28:51.390 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:28:51.390 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:28:51.390 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:28:51.390 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:51.391 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:51.391 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:51.391 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "osd.7", 2024-02-19T15:28:51.391 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQD+cdNlfW65AhAAVWj3ptC8uQHPh/UXcb/uDg==", 2024-02-19T15:28:51.391 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:28:51.391 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow profile osd", 2024-02-19T15:28:51.392 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile osd", 2024-02-19T15:28:51.392 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:28:51.392 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:51.392 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:51.392 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:51.392 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.0", 2024-02-19T15:28:51.393 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlb7eRNRAAk6ZLGfjhcdlR9GoXFCd9eA==", 2024-02-19T15:28:51.393 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:28:51.393 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:28:51.393 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow r", 2024-02-19T15:28:51.393 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-19T15:28:51.393 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:51.393 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:51.393 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:51.393 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.1", 2024-02-19T15:28:51.393 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlKBu+AxAAlLRU70SAs3ZJo5wFwRWa4w==", 2024-02-19T15:28:51.393 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:28:51.393 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:28:51.393 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:28:51.393 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:28:51.394 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:28:51.394 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:51.394 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:51.394 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:51.394 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.2", 2024-02-19T15:28:51.394 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlyDk9ABAAIAjKSd4qJO5oHhVnDc1Rgw==", 2024-02-19T15:28:51.394 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:28:51.394 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:28:51.394 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:28:51.394 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:28:51.394 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:28:51.394 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:51.394 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:51.394 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:51.394 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.3", 2024-02-19T15:28:51.394 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDkcdNlUqK3CRAABmSIzoJvl0ymME6swrvnSQ==", 2024-02-19T15:28:51.395 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:28:51.395 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow", 2024-02-19T15:28:51.395 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow r", 2024-02-19T15:28:51.395 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow rw", 2024-02-19T15:28:51.395 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow rwx" 2024-02-19T15:28:51.395 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:51.395 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:51.395 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:51.395 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "client.admin", 2024-02-19T15:28:51.395 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNls3vPAhAA4x2yQTUCz4Dv9vqEWnSTyQ==", 2024-02-19T15:28:51.395 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:28:51.395 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:28:51.395 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": "allow *", 2024-02-19T15:28:51.395 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow *", 2024-02-19T15:28:51.395 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:28:51.396 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:51.396 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:51.396 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:51.396 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mgr.x", 2024-02-19T15:28:51.396 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNliM78GRAABWBg4hMcKjV2pK5WVBjLMQ==", 2024-02-19T15:28:51.396 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:28:51.396 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:28:51.396 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile mgr", 2024-02-19T15:28:51.396 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:28:51.396 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:51.396 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-02-19T15:28:51.396 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:51.396 INFO:teuthology.orchestra.run.smithi047.stdout: "entity": "mgr.y", 2024-02-19T15:28:51.396 INFO:teuthology.orchestra.run.smithi047.stdout: "key": "AQDjcdNlwxs9FhAAWqP7YoKio/bbdx7SGiCJ9Q==", 2024-02-19T15:28:51.396 INFO:teuthology.orchestra.run.smithi047.stdout: "caps": { 2024-02-19T15:28:51.396 INFO:teuthology.orchestra.run.smithi047.stdout: "mds": "allow *", 2024-02-19T15:28:51.397 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": "allow profile mgr", 2024-02-19T15:28:51.397 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": "allow *" 2024-02-19T15:28:51.397 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:51.397 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:51.397 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-02-19T15:28:51.397 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:28:51.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.386+0000 7fec1ec00640 1 -- 172.21.15.47:0/449969572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7febf4066e40 msgr2=0x7febf4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:51.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.386+0000 7fec1ec00640 1 --2- 172.21.15.47:0/449969572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7febf4066e40 0x7febf4069300 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fec08007a10 tx=0x7fec08009040 comp rx=0 tx=0).stop 2024-02-19T15:28:51.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.386+0000 7fec1ec00640 1 -- 172.21.15.47:0/449969572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1806fdf0 msgr2=0x7fec18153840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:51.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.386+0000 7fec1ec00640 1 --2- 172.21.15.47:0/449969572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1806fdf0 0x7fec18153840 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fec000ae040 tx=0x7fec0001ab70 comp rx=0 tx=0).stop 2024-02-19T15:28:51.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.386+0000 7fec1ec00640 1 -- 172.21.15.47:0/449969572 shutdown_connections 2024-02-19T15:28:51.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.386+0000 7fec1ec00640 1 --2- 172.21.15.47:0/449969572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7febf4066e40 0x7febf4069300 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.386+0000 7fec1ec00640 1 --2- 172.21.15.47:0/449969572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec181613a0 0x7fec18153d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.386+0000 7fec1ec00640 1 --2- 172.21.15.47:0/449969572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1806fdf0 0x7fec18153840 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.386+0000 7fec1ec00640 1 --2- 172.21.15.47:0/449969572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec1806e7c0 0x7fec18153300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.386+0000 7fec1ec00640 1 -- 172.21.15.47:0/449969572 >> 172.21.15.47:0/449969572 conn(0x7fec180d0ad0 msgr2=0x7fec18160660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:51.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.386+0000 7fec1ec00640 1 -- 172.21.15.47:0/449969572 shutdown_connections 2024-02-19T15:28:51.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.386+0000 7fec1ec00640 1 -- 172.21.15.47:0/449969572 wait complete. 2024-02-19T15:28:51.405 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:28:51.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.546+0000 7f28c9600640 1 Processor -- start 2024-02-19T15:28:51.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.546+0000 7f28c9600640 1 -- start start 2024-02-19T15:28:51.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.546+0000 7f28c9600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f28c415e270 0x7f28c40d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:51.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.546+0000 7f28c9600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f28c40d1760 0x7f28c40ce320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:51.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.546+0000 7f28c9600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c40b3240 con 0x7f28c40cff60 2024-02-19T15:28:51.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.546+0000 7f28c9600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28c40b3520 con 0x7f28c415e270 2024-02-19T15:28:51.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.546+0000 7f28c9600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28c406cb50 con 0x7f28c40d1760 2024-02-19T15:28:51.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.546+0000 7f28c2000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f28c415e270 0x7f28c40d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:51.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.546+0000 7f28c2000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f28c415e270 0x7f28c40d0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40822/0 (socket says 172.21.15.47:40822) 2024-02-19T15:28:51.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.546+0000 7f28c2000640 1 -- 172.21.15.47:0/1862380588 learned_addr learned my addr 172.21.15.47:0/1862380588 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:51.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.546+0000 7f28c3400640 1 --2- 172.21.15.47:0/1862380588 >> v2:172.21.15.116:3300/0 conn(0x7f28c40d1760 0x7f28c40ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:51.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.546+0000 7f28c2000640 1 -- 172.21.15.47:0/1862380588 >> v2:172.21.15.116:3300/0 conn(0x7f28c40d1760 msgr2=0x7f28c40ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:51.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.546+0000 7f28c2000640 1 --2- 172.21.15.47:0/1862380588 >> v2:172.21.15.116:3300/0 conn(0x7f28c40d1760 0x7f28c40ce320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.546+0000 7f28c2000640 1 -- 172.21.15.47:0/1862380588 >> v1:172.21.15.116:6789/0 conn(0x7f28c40cff60 legacy=0x7f28c40d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:51.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.546+0000 7f28c2000640 1 -- 172.21.15.47:0/1862380588 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c406ce30 con 0x7f28c415e270 2024-02-19T15:28:51.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.550+0000 7f28c2000640 1 --2- 172.21.15.47:0/1862380588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28c415e270 0x7f28c40d0dc0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f28c4071ed0 tx=0x7f28b001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=152414bf930c5531 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:51.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.550+0000 7f28c1600640 1 -- 172.21.15.47:0/1862380588 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b00ab670 con 0x7f28c415e270 2024-02-19T15:28:51.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.550+0000 7f28c1600640 1 -- 172.21.15.47:0/1862380588 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b00ab7d0 con 0x7f28c415e270 2024-02-19T15:28:51.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.550+0000 7f28c1600640 1 -- 172.21.15.47:0/1862380588 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b00abb60 con 0x7f28c415e270 2024-02-19T15:28:51.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.550+0000 7f28c9600640 1 -- 172.21.15.47:0/1862380588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28c415e270 msgr2=0x7f28c40d0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:51.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.550+0000 7f28c9600640 1 --2- 172.21.15.47:0/1862380588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28c415e270 0x7f28c40d0dc0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f28c4071ed0 tx=0x7f28b001c450 comp rx=0 tx=0).stop 2024-02-19T15:28:51.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.550+0000 7f28c9600640 1 -- 172.21.15.47:0/1862380588 shutdown_connections 2024-02-19T15:28:51.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.550+0000 7f28c9600640 1 --2- 172.21.15.47:0/1862380588 >> v2:172.21.15.116:3300/0 conn(0x7f28c40d1760 0x7f28c40ce320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.550+0000 7f28c9600640 1 --2- 172.21.15.47:0/1862380588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28c415e270 0x7f28c40d0dc0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.550+0000 7f28c9600640 1 -- 172.21.15.47:0/1862380588 >> 172.21.15.47:0/1862380588 conn(0x7f28c40d1b90 msgr2=0x7f28c40d1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:51.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.550+0000 7f28c9600640 1 -- 172.21.15.47:0/1862380588 shutdown_connections 2024-02-19T15:28:51.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.550+0000 7f28c9600640 1 -- 172.21.15.47:0/1862380588 wait complete. 2024-02-19T15:28:51.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.550+0000 7f28c9600640 1 Processor -- start 2024-02-19T15:28:51.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.550+0000 7f28c9600640 1 -- start start 2024-02-19T15:28:51.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c9600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28c40cff60 0x7f28c41552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:51.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c9600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28c40d1760 0x7f28c414db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:51.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c9600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28c415e270 0x7f28c414f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:51.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c9600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c41520f0 con 0x7f28c40d1760 2024-02-19T15:28:51.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c9600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c414e0b0 con 0x7f28c40cff60 2024-02-19T15:28:51.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c9600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c41500b0 con 0x7f28c415e270 2024-02-19T15:28:51.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c2000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28c40d1760 0x7f28c414db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:51.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c2000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28c40d1760 0x7f28c414db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40830/0 (socket says 172.21.15.47:40830) 2024-02-19T15:28:51.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c2000640 1 -- 172.21.15.47:0/902093442 learned_addr learned my addr 172.21.15.47:0/902093442 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:51.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c3400640 1 --2- 172.21.15.47:0/902093442 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28c415e270 0x7f28c414f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:51.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c2a00640 1 --2- 172.21.15.47:0/902093442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28c40cff60 0x7f28c41552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:51.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c2000640 1 -- 172.21.15.47:0/902093442 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28c415e270 msgr2=0x7f28c414f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:51.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c2000640 1 --2- 172.21.15.47:0/902093442 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28c415e270 0x7f28c414f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c2000640 1 -- 172.21.15.47:0/902093442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28c40cff60 msgr2=0x7f28c41552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:51.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c2000640 1 --2- 172.21.15.47:0/902093442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28c40cff60 0x7f28c41552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c2000640 1 -- 172.21.15.47:0/902093442 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b00094d0 con 0x7f28c40d1760 2024-02-19T15:28:51.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c3400640 1 --2- 172.21.15.47:0/902093442 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28c415e270 0x7f28c414f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:51.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c2a00640 1 --2- 172.21.15.47:0/902093442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28c40cff60 0x7f28c41552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:51.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c2000640 1 --2- 172.21.15.47:0/902093442 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28c40d1760 0x7f28c414db70 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f28b001a080 tx=0x7f28b001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:51.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28baa00640 1 -- 172.21.15.47:0/902093442 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b00b4800 con 0x7f28c40d1760 2024-02-19T15:28:51.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28baa00640 1 -- 172.21.15.47:0/902093442 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b00b4980 con 0x7f28c40d1760 2024-02-19T15:28:51.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28baa00640 1 -- 172.21.15.47:0/902093442 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b00b4d30 con 0x7f28c40d1760 2024-02-19T15:28:51.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c9600640 1 -- 172.21.15.47:0/902093442 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c4150390 con 0x7f28c40d1760 2024-02-19T15:28:51.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c9600640 1 -- 172.21.15.47:0/902093442 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c4150950 con 0x7f28c40d1760 2024-02-19T15:28:51.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28c9600640 1 -- 172.21.15.47:0/902093442 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c4068500 con 0x7f28c40d1760 2024-02-19T15:28:51.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.554+0000 7f28baa00640 1 -- 172.21.15.47:0/902093442 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f28b00b3380 con 0x7f28c40d1760 2024-02-19T15:28:51.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.558+0000 7f28baa00640 1 --2- 172.21.15.47:0/902093442 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f28a4066e40 0x7f28a4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:51.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.558+0000 7f28baa00640 1 -- 172.21.15.47:0/902093442 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f28b011ed10 con 0x7f28c40d1760 2024-02-19T15:28:51.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.558+0000 7f28c2a00640 1 --2- 172.21.15.47:0/902093442 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f28a4066e40 0x7f28a4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:51.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.558+0000 7f28c2a00640 1 --2- 172.21.15.47:0/902093442 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f28a4066e40 0x7f28a4069300 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f28b4003ec0 tx=0x7f28b4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:51.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.562+0000 7f28baa00640 1 -- 172.21.15.47:0/902093442 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f28b00ebf80 con 0x7f28c40d1760 2024-02-19T15:28:51.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.726+0000 7f28c9600640 1 -- 172.21.15.47:0/902093442 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28c406c840 con 0x7f28c40d1760 2024-02-19T15:28:51.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.726+0000 7f28baa00640 1 -- 172.21.15.47:0/902093442 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+12913 (secure 0 0 0) 0x7f28b00ef640 con 0x7f28c40d1760 2024-02-19T15:28:51.726 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:51.727 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":121,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:28:50.977934+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:28:51.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.730+0000 7f28c9600640 1 -- 172.21.15.47:0/902093442 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f28a4066e40 msgr2=0x7f28a4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:51.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.730+0000 7f28c9600640 1 --2- 172.21.15.47:0/902093442 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f28a4066e40 0x7f28a4069300 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f28b4003ec0 tx=0x7f28b4009040 comp rx=0 tx=0).stop 2024-02-19T15:28:51.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.730+0000 7f28c9600640 1 -- 172.21.15.47:0/902093442 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28c40d1760 msgr2=0x7f28c414db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:51.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.730+0000 7f28c9600640 1 --2- 172.21.15.47:0/902093442 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28c40d1760 0x7f28c414db70 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f28b001a080 tx=0x7f28b001ac90 comp rx=0 tx=0).stop 2024-02-19T15:28:51.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.730+0000 7f28c9600640 1 -- 172.21.15.47:0/902093442 shutdown_connections 2024-02-19T15:28:51.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.730+0000 7f28c9600640 1 --2- 172.21.15.47:0/902093442 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f28a4066e40 0x7f28a4069300 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.730+0000 7f28c9600640 1 --2- 172.21.15.47:0/902093442 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28c415e270 0x7f28c414f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.730+0000 7f28c9600640 1 --2- 172.21.15.47:0/902093442 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28c40d1760 0x7f28c414db70 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.730+0000 7f28c9600640 1 --2- 172.21.15.47:0/902093442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28c40cff60 0x7f28c41552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.730+0000 7f28c9600640 1 -- 172.21.15.47:0/902093442 >> 172.21.15.47:0/902093442 conn(0x7f28c40d1b90 msgr2=0x7f28c406eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:51.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.730+0000 7f28c9600640 1 -- 172.21.15.47:0/902093442 shutdown_connections 2024-02-19T15:28:51.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.730+0000 7f28c9600640 1 -- 172.21.15.47:0/902093442 wait complete. 2024-02-19T15:28:51.749 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:28:51.749 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:28:51.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.894+0000 7efc0c000640 1 Processor -- start 2024-02-19T15:28:51.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.894+0000 7efc0c000640 1 -- start start 2024-02-19T15:28:51.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.894+0000 7efc0c000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efc0415af10 0x7efc0406e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:51.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.894+0000 7efc0c000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc040ca8f0 0x7efc040cad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:51.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.894+0000 7efc0c000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc04071c70 con 0x7efc040cb290 2024-02-19T15:28:51.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.894+0000 7efc0c000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc0406e8a0 con 0x7efc0415af10 2024-02-19T15:28:51.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.894+0000 7efc0c000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc0406cef0 con 0x7efc040ca8f0 2024-02-19T15:28:51.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.894+0000 7efc09600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efc0415af10 0x7efc0406e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:51.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.894+0000 7efc0a000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7efc040cb290 0x7efc0415eaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60788/0 (socket says 172.21.15.47:60788) 2024-02-19T15:28:51.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.894+0000 7efc0a000640 1 -- 172.21.15.47:0/3709618171 learned_addr learned my addr 172.21.15.47:0/3709618171 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:51.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.894+0000 7efc08c00640 1 --2- 172.21.15.47:0/3709618171 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc040ca8f0 0x7efc040cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:51.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.894+0000 7efc09600640 1 -- 172.21.15.47:0/3709618171 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc040ca8f0 msgr2=0x7efc040cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:51.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.894+0000 7efc09600640 1 --2- 172.21.15.47:0/3709618171 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc040ca8f0 0x7efc040cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.894+0000 7efc09600640 1 -- 172.21.15.47:0/3709618171 >> v1:172.21.15.47:6789/0 conn(0x7efc040cb290 legacy=0x7efc0415eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:51.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.894+0000 7efc09600640 1 -- 172.21.15.47:0/3709618171 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc040ca610 con 0x7efc0415af10 2024-02-19T15:28:51.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.898+0000 7efc09600640 1 --2- 172.21.15.47:0/3709618171 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc0415af10 0x7efc0406e360 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7efbf80097c0 tx=0x7efbf801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e0b4472b349652f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:51.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.898+0000 7efc03e00640 1 -- 172.21.15.47:0/3709618171 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf80ac640 con 0x7efc0415af10 2024-02-19T15:28:51.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.898+0000 7efc03e00640 1 -- 172.21.15.47:0/3709618171 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf80ac7a0 con 0x7efc0415af10 2024-02-19T15:28:51.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.898+0000 7efc03e00640 1 -- 172.21.15.47:0/3709618171 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf80acb30 con 0x7efc0415af10 2024-02-19T15:28:51.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.898+0000 7efc0c000640 1 -- 172.21.15.47:0/3709618171 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc0415af10 msgr2=0x7efc0406e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:51.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.898+0000 7efc0c000640 1 --2- 172.21.15.47:0/3709618171 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc0415af10 0x7efc0406e360 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7efbf80097c0 tx=0x7efbf801c3e0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.898+0000 7efc0c000640 1 -- 172.21.15.47:0/3709618171 shutdown_connections 2024-02-19T15:28:51.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.898+0000 7efc0c000640 1 --2- 172.21.15.47:0/3709618171 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc040ca8f0 0x7efc040cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.898+0000 7efc0c000640 1 --2- 172.21.15.47:0/3709618171 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc0415af10 0x7efc0406e360 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.898+0000 7efc0c000640 1 -- 172.21.15.47:0/3709618171 >> 172.21.15.47:0/3709618171 conn(0x7efc040cfc20 msgr2=0x7efc040d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:51.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.898+0000 7efc0c000640 1 -- 172.21.15.47:0/3709618171 shutdown_connections 2024-02-19T15:28:51.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.898+0000 7efc0c000640 1 -- 172.21.15.47:0/3709618171 wait complete. 2024-02-19T15:28:51.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc0c000640 1 Processor -- start 2024-02-19T15:28:51.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc0c000640 1 -- start start 2024-02-19T15:28:51.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc0c000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc040ca8f0 0x7efc04153130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:51.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc0c000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc040cb290 0x7efc040c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc0c000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc0415af10 0x7efc040c9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc0c000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc0415c740 con 0x7efc0415af10 2024-02-19T15:28:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc0c000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc041576c0 con 0x7efc040ca8f0 2024-02-19T15:28:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc0c000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc041579a0 con 0x7efc040cb290 2024-02-19T15:28:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc0a000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc0415af10 0x7efc040c9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc0a000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc0415af10 0x7efc040c9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40840/0 (socket says 172.21.15.47:40840) 2024-02-19T15:28:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc0a000640 1 -- 172.21.15.47:0/2259558469 learned_addr learned my addr 172.21.15.47:0/2259558469 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc08c00640 1 --2- 172.21.15.47:0/2259558469 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc040cb290 0x7efc040c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc09600640 1 --2- 172.21.15.47:0/2259558469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc040ca8f0 0x7efc04153130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc0a000640 1 -- 172.21.15.47:0/2259558469 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc040cb290 msgr2=0x7efc040c8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc0a000640 1 --2- 172.21.15.47:0/2259558469 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc040cb290 0x7efc040c8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc0a000640 1 -- 172.21.15.47:0/2259558469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc040ca8f0 msgr2=0x7efc04153130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc0a000640 1 --2- 172.21.15.47:0/2259558469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc040ca8f0 0x7efc04153130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc0a000640 1 -- 172.21.15.47:0/2259558469 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf8009420 con 0x7efc0415af10 2024-02-19T15:28:51.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc0a000640 1 --2- 172.21.15.47:0/2259558469 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc0415af10 0x7efc040c9240 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7efbfc00caa0 tx=0x7efbfc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:51.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc01600640 1 -- 172.21.15.47:0/2259558469 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc00ac00 con 0x7efc0415af10 2024-02-19T15:28:51.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc01600640 1 -- 172.21.15.47:0/2259558469 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbfc00ad60 con 0x7efc0415af10 2024-02-19T15:28:51.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc01600640 1 -- 172.21.15.47:0/2259558469 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc012600 con 0x7efc0415af10 2024-02-19T15:28:51.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc0c000640 1 -- 172.21.15.47:0/2259558469 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc0415c9e0 con 0x7efc0415af10 2024-02-19T15:28:51.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc0c000640 1 -- 172.21.15.47:0/2259558469 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc041551b0 con 0x7efc0415af10 2024-02-19T15:28:51.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.902+0000 7efc0c000640 1 -- 172.21.15.47:0/2259558469 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc0415cbd0 con 0x7efc0415af10 2024-02-19T15:28:51.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.906+0000 7efc01600640 1 -- 172.21.15.47:0/2259558469 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efbfc007070 con 0x7efc0415af10 2024-02-19T15:28:51.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.906+0000 7efc01600640 1 --2- 172.21.15.47:0/2259558469 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efbe4066e40 0x7efbe4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:51.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.906+0000 7efc01600640 1 -- 172.21.15.47:0/2259558469 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efbfc0844a0 con 0x7efc0415af10 2024-02-19T15:28:51.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.906+0000 7efc09600640 1 --2- 172.21.15.47:0/2259558469 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efbe4066e40 0x7efbe4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:51.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.906+0000 7efc09600640 1 --2- 172.21.15.47:0/2259558469 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efbe4066e40 0x7efbe4069300 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7efbf80ae000 tx=0x7efbf80a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:51.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:51.910+0000 7efc01600640 1 -- 172.21.15.47:0/2259558469 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7efbfc051680 con 0x7efc0415af10 2024-02-19T15:28:52.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.078+0000 7efc0c000640 1 -- 172.21.15.47:0/2259558469 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efc04075360 con 0x7efc0415af10 2024-02-19T15:28:52.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.078+0000 7efc01600640 1 -- 172.21.15.47:0/2259558469 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v121) v1 ==== 93+0+10 (secure 0 0 0) 0x7efbfc0170a0 con 0x7efc0415af10 2024-02-19T15:28:52.079 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:28:52.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.082+0000 7efc0c000640 1 -- 172.21.15.47:0/2259558469 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efbe4066e40 msgr2=0x7efbe4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:52.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.082+0000 7efc0c000640 1 --2- 172.21.15.47:0/2259558469 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efbe4066e40 0x7efbe4069300 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7efbf80ae000 tx=0x7efbf80a7000 comp rx=0 tx=0).stop 2024-02-19T15:28:52.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.082+0000 7efc0c000640 1 -- 172.21.15.47:0/2259558469 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc0415af10 msgr2=0x7efc040c9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:52.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.082+0000 7efc0c000640 1 --2- 172.21.15.47:0/2259558469 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc0415af10 0x7efc040c9240 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7efbfc00caa0 tx=0x7efbfc00cf60 comp rx=0 tx=0).stop 2024-02-19T15:28:52.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.082+0000 7efc0c000640 1 -- 172.21.15.47:0/2259558469 shutdown_connections 2024-02-19T15:28:52.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.082+0000 7efc0c000640 1 --2- 172.21.15.47:0/2259558469 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efbe4066e40 0x7efbe4069300 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.082+0000 7efc0c000640 1 --2- 172.21.15.47:0/2259558469 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc0415af10 0x7efc040c9240 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.082+0000 7efc0c000640 1 --2- 172.21.15.47:0/2259558469 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc040cb290 0x7efc040c8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.082+0000 7efc0c000640 1 --2- 172.21.15.47:0/2259558469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc040ca8f0 0x7efc04153130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.082+0000 7efc0c000640 1 -- 172.21.15.47:0/2259558469 >> 172.21.15.47:0/2259558469 conn(0x7efc040cfc20 msgr2=0x7efc041587b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:52.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.082+0000 7efc0c000640 1 -- 172.21.15.47:0/2259558469 shutdown_connections 2024-02-19T15:28:52.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.082+0000 7efc0c000640 1 -- 172.21.15.47:0/2259558469 wait complete. 2024-02-19T15:28:52.101 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T15:28:52.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.238+0000 7ff954000640 1 Processor -- start 2024-02-19T15:28:52.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.238+0000 7ff954000640 1 -- start start 2024-02-19T15:28:52.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.242+0000 7ff954000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff94c0b2b70 0x7ff94c0cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:52.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.242+0000 7ff954000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff94c0d0420 0x7ff94c0d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:52.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.242+0000 7ff954000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff94c0b32c0 con 0x7ff94c0ce510 2024-02-19T15:28:52.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.242+0000 7ff954000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff94c0cf6f0 con 0x7ff94c0d0420 2024-02-19T15:28:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.242+0000 7ff954000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c0cff90 con 0x7ff94c0b2b70 2024-02-19T15:28:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.242+0000 7ff950c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff94c0b2b70 0x7ff94c0cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.242+0000 7ff952000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff94c0d0420 0x7ff94c0d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.242+0000 7ff950c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff94c0b2b70 0x7ff94c0cd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:57510/0 (socket says 172.21.15.47:57510) 2024-02-19T15:28:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.242+0000 7ff950c00640 1 -- 172.21.15.47:0/4206610021 learned_addr learned my addr 172.21.15.47:0/4206610021 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.242+0000 7ff952000640 1 -- 172.21.15.47:0/4206610021 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff94c0b2b70 msgr2=0x7ff94c0cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.242+0000 7ff952000640 1 --2- 172.21.15.47:0/4206610021 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff94c0b2b70 0x7ff94c0cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.242+0000 7ff952000640 1 -- 172.21.15.47:0/4206610021 >> v1:172.21.15.116:6789/0 conn(0x7ff94c0ce510 legacy=0x7ff94c0ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.242+0000 7ff952000640 1 -- 172.21.15.47:0/4206610021 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94c0cd9f0 con 0x7ff94c0d0420 2024-02-19T15:28:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.242+0000 7ff950c00640 1 --2- 172.21.15.47:0/4206610021 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff94c0b2b70 0x7ff94c0cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:28:52.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.246+0000 7ff952000640 1 --2- 172.21.15.47:0/4206610021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff94c0d0420 0x7ff94c0d08b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff944009d90 tx=0x7ff94401c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b38822b38f1ea7bb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:52.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.246+0000 7ff94be00640 1 -- 172.21.15.47:0/4206610021 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9440a58b0 con 0x7ff94c0d0420 2024-02-19T15:28:52.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.246+0000 7ff94be00640 1 -- 172.21.15.47:0/4206610021 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9440a5a10 con 0x7ff94c0d0420 2024-02-19T15:28:52.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.246+0000 7ff94be00640 1 -- 172.21.15.47:0/4206610021 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9440a5da0 con 0x7ff94c0d0420 2024-02-19T15:28:52.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.246+0000 7ff954000640 1 -- 172.21.15.47:0/4206610021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff94c0d0420 msgr2=0x7ff94c0d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:52.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.246+0000 7ff954000640 1 --2- 172.21.15.47:0/4206610021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff94c0d0420 0x7ff94c0d08b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff944009d90 tx=0x7ff94401c8e0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.246+0000 7ff954000640 1 -- 172.21.15.47:0/4206610021 shutdown_connections 2024-02-19T15:28:52.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.246+0000 7ff954000640 1 --2- 172.21.15.47:0/4206610021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff94c0d0420 0x7ff94c0d08b0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.246+0000 7ff954000640 1 --2- 172.21.15.47:0/4206610021 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff94c0b2b70 0x7ff94c0cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.246+0000 7ff954000640 1 -- 172.21.15.47:0/4206610021 >> 172.21.15.47:0/4206610021 conn(0x7ff94c0d1b20 msgr2=0x7ff94c0d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:52.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.246+0000 7ff954000640 1 -- 172.21.15.47:0/4206610021 shutdown_connections 2024-02-19T15:28:52.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.246+0000 7ff954000640 1 -- 172.21.15.47:0/4206610021 wait complete. 2024-02-19T15:28:52.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.246+0000 7ff954000640 1 Processor -- start 2024-02-19T15:28:52.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.246+0000 7ff954000640 1 -- start start 2024-02-19T15:28:52.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff954000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff94c0b2b70 0x7ff94c150b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:52.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff954000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff94c0ce510 0x7ff94c15cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:52.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff954000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff94c0d0420 0x7ff94c15c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:52.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff954000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c15d3a0 con 0x7ff94c0d0420 2024-02-19T15:28:52.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff954000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c15d680 con 0x7ff94c0ce510 2024-02-19T15:28:52.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff954000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c15d960 con 0x7ff94c0b2b70 2024-02-19T15:28:52.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff952000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff94c0d0420 0x7ff94c15c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:52.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff950c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff94c0ce510 0x7ff94c15cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:52.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff950c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff94c0ce510 0x7ff94c15cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:53460/0 (socket says 172.21.15.47:53460) 2024-02-19T15:28:52.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff950c00640 1 -- 172.21.15.47:0/344060040 learned_addr learned my addr 172.21.15.47:0/344060040 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:52.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff951600640 1 --2- 172.21.15.47:0/344060040 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff94c0b2b70 0x7ff94c150b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:52.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff950c00640 1 -- 172.21.15.47:0/344060040 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff94c0b2b70 msgr2=0x7ff94c150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:52.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff950c00640 1 --2- 172.21.15.47:0/344060040 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff94c0b2b70 0x7ff94c150b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff950c00640 1 -- 172.21.15.47:0/344060040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff94c0d0420 msgr2=0x7ff94c15c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:52.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff950c00640 1 --2- 172.21.15.47:0/344060040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff94c0d0420 0x7ff94c15c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff950c00640 1 -- 172.21.15.47:0/344060040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9380094d0 con 0x7ff94c0ce510 2024-02-19T15:28:52.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff952000640 1 --2- 172.21.15.47:0/344060040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff94c0d0420 0x7ff94c15c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:28:52.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff950c00640 1 --2- 172.21.15.47:0/344060040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff94c0ce510 0x7ff94c15cd00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff93800e930 tx=0x7ff93800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:52.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff949600640 1 -- 172.21.15.47:0/344060040 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93800ca00 con 0x7ff94c0ce510 2024-02-19T15:28:52.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff949600640 1 -- 172.21.15.47:0/344060040 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93800cb60 con 0x7ff94c0ce510 2024-02-19T15:28:52.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff949600640 1 -- 172.21.15.47:0/344060040 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff938013470 con 0x7ff94c0ce510 2024-02-19T15:28:52.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff954000640 1 -- 172.21.15.47:0/344060040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff944009a70 con 0x7ff94c0ce510 2024-02-19T15:28:52.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff954000640 1 -- 172.21.15.47:0/344060040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94c15df20 con 0x7ff94c0ce510 2024-02-19T15:28:52.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff954000640 1 -- 172.21.15.47:0/344060040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff94c075360 con 0x7ff94c0ce510 2024-02-19T15:28:52.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.250+0000 7ff949600640 1 -- 172.21.15.47:0/344060040 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff9380135d0 con 0x7ff94c0ce510 2024-02-19T15:28:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.254+0000 7ff949600640 1 --2- 172.21.15.47:0/344060040 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff92c066e40 0x7ff92c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.254+0000 7ff949600640 1 -- 172.21.15.47:0/344060040 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff9380848f0 con 0x7ff94c0ce510 2024-02-19T15:28:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.254+0000 7ff951600640 1 --2- 172.21.15.47:0/344060040 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff92c066e40 0x7ff92c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.254+0000 7ff951600640 1 --2- 172.21.15.47:0/344060040 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff92c066e40 0x7ff92c069300 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7ff94c160760 tx=0x7ff940009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:52.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.258+0000 7ff949600640 1 -- 172.21.15.47:0/344060040 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff938051ad0 con 0x7ff94c0ce510 2024-02-19T15:28:52.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.422+0000 7ff954000640 1 -- 172.21.15.47:0/344060040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff94c15d7c0 con 0x7ff94c0ce510 2024-02-19T15:28:52.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.422+0000 7ff949600640 1 -- 172.21.15.47:0/344060040 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v121) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff938017060 con 0x7ff94c0ce510 2024-02-19T15:28:52.422 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:52.422 INFO:teuthology.orchestra.run.smithi047.stdout:[ 2024-02-19T15:28:52.422 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-02-19T15:28:52.422 INFO:teuthology.orchestra.run.smithi047.stdout: "poolnum": 1, 2024-02-19T15:28:52.422 INFO:teuthology.orchestra.run.smithi047.stdout: "poolname": ".mgr" 2024-02-19T15:28:52.422 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-02-19T15:28:52.422 INFO:teuthology.orchestra.run.smithi047.stdout:] 2024-02-19T15:28:52.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.422+0000 7ff954000640 1 -- 172.21.15.47:0/344060040 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff92c066e40 msgr2=0x7ff92c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:52.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.422+0000 7ff954000640 1 --2- 172.21.15.47:0/344060040 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff92c066e40 0x7ff92c069300 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7ff94c160760 tx=0x7ff940009040 comp rx=0 tx=0).stop 2024-02-19T15:28:52.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.422+0000 7ff954000640 1 -- 172.21.15.47:0/344060040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff94c0ce510 msgr2=0x7ff94c15cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:52.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.422+0000 7ff954000640 1 --2- 172.21.15.47:0/344060040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff94c0ce510 0x7ff94c15cd00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff93800e930 tx=0x7ff93800edf0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.422+0000 7ff954000640 1 -- 172.21.15.47:0/344060040 shutdown_connections 2024-02-19T15:28:52.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.422+0000 7ff954000640 1 --2- 172.21.15.47:0/344060040 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff92c066e40 0x7ff92c069300 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.422+0000 7ff954000640 1 --2- 172.21.15.47:0/344060040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff94c0d0420 0x7ff94c15c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.422+0000 7ff954000640 1 --2- 172.21.15.47:0/344060040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff94c0ce510 0x7ff94c15cd00 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.422+0000 7ff954000640 1 --2- 172.21.15.47:0/344060040 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff94c0b2b70 0x7ff94c150b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.422+0000 7ff954000640 1 -- 172.21.15.47:0/344060040 >> 172.21.15.47:0/344060040 conn(0x7ff94c0d1b20 msgr2=0x7ff94c06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:52.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.422+0000 7ff954000640 1 -- 172.21.15.47:0/344060040 shutdown_connections 2024-02-19T15:28:52.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.422+0000 7ff954000640 1 -- 172.21.15.47:0/344060040 wait complete. 2024-02-19T15:28:52.443 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T15:28:52.443 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T15:28:52.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.586+0000 7f3145200640 1 Processor -- start 2024-02-19T15:28:52.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.586+0000 7f3145200640 1 -- start start 2024-02-19T15:28:52.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.590+0000 7f3145200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f314006c920 0x7f31400c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:52.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.590+0000 7f3145200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31400b5650 con 0x7f31400b5270 2024-02-19T15:28:52.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.590+0000 7f3145200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3140071560 con 0x7f314006c4c0 2024-02-19T15:28:52.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.590+0000 7f3145200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f314015f7b0 con 0x7f314006c920 2024-02-19T15:28:52.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.590+0000 7f313de00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f31400b5270 0x7f31400719f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60804/0 (socket says 172.21.15.47:60804) 2024-02-19T15:28:52.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.590+0000 7f313de00640 1 -- 172.21.15.47:0/2023664264 learned_addr learned my addr 172.21.15.47:0/2023664264 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:52.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.590+0000 7f313e800640 1 --2- 172.21.15.47:0/2023664264 >> v2:172.21.15.116:3300/0 conn(0x7f314006c920 0x7f31400c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:52.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.590+0000 7f313d400640 1 -- 172.21.15.47:0/2023664264 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917784844 0 0) 0x7f31400b5650 con 0x7f31400b5270 2024-02-19T15:28:52.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.590+0000 7f313d400640 1 -- 172.21.15.47:0/2023664264 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3124003530 con 0x7f31400b5270 2024-02-19T15:28:52.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.590+0000 7f313d400640 1 -- 172.21.15.47:0/2023664264 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774201827 0 0) 0x7f3140071560 con 0x7f314006c4c0 2024-02-19T15:28:52.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.590+0000 7f313d400640 1 -- 172.21.15.47:0/2023664264 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31400b5650 con 0x7f314006c4c0 2024-02-19T15:28:52.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.590+0000 7f313e800640 1 -- 172.21.15.47:0/2023664264 >> v1:172.21.15.116:6789/0 conn(0x7f314006c4c0 legacy=0x7f314006e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:52.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.590+0000 7f313e800640 1 -- 172.21.15.47:0/2023664264 >> v1:172.21.15.47:6789/0 conn(0x7f31400b5270 legacy=0x7f31400719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:52.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.590+0000 7f313d400640 1 -- 172.21.15.47:0/2023664264 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2916745544 0 0) 0x7f3124003530 con 0x7f31400b5270 2024-02-19T15:28:52.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.590+0000 7f313e800640 1 -- 172.21.15.47:0/2023664264 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3140072320 con 0x7f314006c920 2024-02-19T15:28:52.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f313e800640 1 --2- 172.21.15.47:0/2023664264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f314006c920 0x7f31400c9fc0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f31380097c0 tx=0x7f313801c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=545955fe18d5328a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:52.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f313d400640 1 -- 172.21.15.47:0/2023664264 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3130001a00 con 0x7f314006c920 2024-02-19T15:28:52.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f313d400640 1 -- 172.21.15.47:0/2023664264 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31380a7050 con 0x7f314006c920 2024-02-19T15:28:52.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f313d400640 1 -- 172.21.15.47:0/2023664264 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31380ab620 con 0x7f314006c920 2024-02-19T15:28:52.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f3145200640 1 -- 172.21.15.47:0/2023664264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f314006c920 msgr2=0x7f31400c9fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:52.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f3145200640 1 --2- 172.21.15.47:0/2023664264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f314006c920 0x7f31400c9fc0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f31380097c0 tx=0x7f313801c3c0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f3145200640 1 -- 172.21.15.47:0/2023664264 shutdown_connections 2024-02-19T15:28:52.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f3145200640 1 --2- 172.21.15.47:0/2023664264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f314006c920 0x7f31400c9fc0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f3145200640 1 -- 172.21.15.47:0/2023664264 >> 172.21.15.47:0/2023664264 conn(0x7f31400cc3f0 msgr2=0x7f31400cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:52.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f3145200640 1 -- 172.21.15.47:0/2023664264 shutdown_connections 2024-02-19T15:28:52.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f3145200640 1 -- 172.21.15.47:0/2023664264 wait complete. 2024-02-19T15:28:52.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f3145200640 1 Processor -- start 2024-02-19T15:28:52.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f3145200640 1 -- start start 2024-02-19T15:28:52.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f3145200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f314006c4c0 0x7f314015a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:52.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f3145200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f314006c920 0x7f31400c5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:52.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f3145200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31400b5270 0x7f314015b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:52.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f3145200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31400c4390 con 0x7f314006c4c0 2024-02-19T15:28:52.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f3145200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31400c34a0 con 0x7f314006c920 2024-02-19T15:28:52.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f3145200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31400c3780 con 0x7f31400b5270 2024-02-19T15:28:52.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f313de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f314006c4c0 0x7f314015a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:52.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f313de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f314006c4c0 0x7f314015a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40866/0 (socket says 172.21.15.47:40866) 2024-02-19T15:28:52.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.594+0000 7f313de00640 1 -- 172.21.15.47:0/3561046641 learned_addr learned my addr 172.21.15.47:0/3561046641 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:52.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.598+0000 7f313e800640 1 --2- 172.21.15.47:0/3561046641 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31400b5270 0x7f314015b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:52.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.598+0000 7f313de00640 1 -- 172.21.15.47:0/3561046641 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31400b5270 msgr2=0x7f314015b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:52.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.598+0000 7f3135400640 1 --2- 172.21.15.47:0/3561046641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f314006c920 0x7f31400c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:52.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.598+0000 7f313de00640 1 --2- 172.21.15.47:0/3561046641 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31400b5270 0x7f314015b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.598+0000 7f313de00640 1 -- 172.21.15.47:0/3561046641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f314006c920 msgr2=0x7f31400c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:52.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.598+0000 7f313de00640 1 --2- 172.21.15.47:0/3561046641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f314006c920 0x7f31400c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.598+0000 7f313de00640 1 -- 172.21.15.47:0/3561046641 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3138009420 con 0x7f314006c4c0 2024-02-19T15:28:52.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.598+0000 7f313e800640 1 --2- 172.21.15.47:0/3561046641 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31400b5270 0x7f314015b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:52.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.598+0000 7f3135400640 1 --2- 172.21.15.47:0/3561046641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f314006c920 0x7f31400c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:52.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.598+0000 7f313de00640 1 --2- 172.21.15.47:0/3561046641 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f314006c4c0 0x7f314015a4f0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f3130001f90 tx=0x7f313000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:52.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.598+0000 7f3136a00640 1 -- 172.21.15.47:0/3561046641 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3130012400 con 0x7f314006c4c0 2024-02-19T15:28:52.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.598+0000 7f3136a00640 1 -- 172.21.15.47:0/3561046641 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3130012560 con 0x7f314006c4c0 2024-02-19T15:28:52.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.598+0000 7f3136a00640 1 -- 172.21.15.47:0/3561046641 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3130012910 con 0x7f314006c4c0 2024-02-19T15:28:52.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.598+0000 7f3145200640 1 -- 172.21.15.47:0/3561046641 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31400c3a60 con 0x7f314006c4c0 2024-02-19T15:28:52.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.598+0000 7f3145200640 1 -- 172.21.15.47:0/3561046641 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31400c4020 con 0x7f314006c4c0 2024-02-19T15:28:52.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.598+0000 7f3145200640 1 -- 172.21.15.47:0/3561046641 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3140075360 con 0x7f314006c4c0 2024-02-19T15:28:52.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.598+0000 7f3136a00640 1 -- 172.21.15.47:0/3561046641 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3130007070 con 0x7f314006c4c0 2024-02-19T15:28:52.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.598+0000 7f3136a00640 1 --2- 172.21.15.47:0/3561046641 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f311c066e40 0x7f311c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:52.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.598+0000 7f3136a00640 1 -- 172.21.15.47:0/3561046641 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f31300849a0 con 0x7f314006c4c0 2024-02-19T15:28:52.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.602+0000 7f3135400640 1 --2- 172.21.15.47:0/3561046641 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f311c066e40 0x7f311c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:52.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.602+0000 7f3135400640 1 --2- 172.21.15.47:0/3561046641 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f311c066e40 0x7f311c069300 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f314015ae90 tx=0x7f3128009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:52.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.602+0000 7f3136a00640 1 -- 172.21.15.47:0/3561046641 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f3130051a60 con 0x7f314006c4c0 2024-02-19T15:28:52.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.766+0000 7f3145200640 1 -- 172.21.15.47:0/3561046641 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f31400cd300 con 0x7f314006c4c0 2024-02-19T15:28:52.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.978+0000 7f3136a00640 1 -- 172.21.15.47:0/3561046641 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v122) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3130017070 con 0x7f314006c4c0 2024-02-19T15:28:52.980 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_metadata' created 2024-02-19T15:28:52.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.982+0000 7f3145200640 1 -- 172.21.15.47:0/3561046641 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f311c066e40 msgr2=0x7f311c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:52.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.982+0000 7f3145200640 1 --2- 172.21.15.47:0/3561046641 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f311c066e40 0x7f311c069300 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f314015ae90 tx=0x7f3128009040 comp rx=0 tx=0).stop 2024-02-19T15:28:52.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.982+0000 7f3145200640 1 -- 172.21.15.47:0/3561046641 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f314006c4c0 msgr2=0x7f314015a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:52.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.982+0000 7f3145200640 1 --2- 172.21.15.47:0/3561046641 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f314006c4c0 0x7f314015a4f0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f3130001f90 tx=0x7f313000a430 comp rx=0 tx=0).stop 2024-02-19T15:28:52.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.982+0000 7f3145200640 1 -- 172.21.15.47:0/3561046641 shutdown_connections 2024-02-19T15:28:52.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.982+0000 7f3145200640 1 --2- 172.21.15.47:0/3561046641 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f311c066e40 0x7f311c069300 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.982+0000 7f3145200640 1 --2- 172.21.15.47:0/3561046641 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31400b5270 0x7f314015b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.982+0000 7f3145200640 1 --2- 172.21.15.47:0/3561046641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f314006c920 0x7f31400c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.982+0000 7f3145200640 1 --2- 172.21.15.47:0/3561046641 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f314006c4c0 0x7f314015a4f0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:52.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.982+0000 7f3145200640 1 -- 172.21.15.47:0/3561046641 >> 172.21.15.47:0/3561046641 conn(0x7f31400cc3f0 msgr2=0x7f31401579d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:52.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.982+0000 7f3145200640 1 -- 172.21.15.47:0/3561046641 shutdown_connections 2024-02-19T15:28:52.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:52.982+0000 7f3145200640 1 -- 172.21.15.47:0/3561046641 wait complete. 2024-02-19T15:28:53.008 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T15:28:53.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.154+0000 7f531d200640 1 Processor -- start 2024-02-19T15:28:53.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.154+0000 7f531d200640 1 -- start start 2024-02-19T15:28:53.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.154+0000 7f531d200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f53180d0150 0x7f531806aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:53.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.154+0000 7f531d200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5318162460 0x7f53180ae4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:53.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.154+0000 7f531d200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53180ce660 con 0x7f531806b3f0 2024-02-19T15:28:53.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.154+0000 7f531d200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5318162d60 con 0x7f5318162460 2024-02-19T15:28:53.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.154+0000 7f531d200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5318162a80 con 0x7f53180d0150 2024-02-19T15:28:53.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.154+0000 7f5316800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5318162460 0x7f53180ae4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:53.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.154+0000 7f5316800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5318162460 0x7f53180ae4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40874/0 (socket says 172.21.15.47:40874) 2024-02-19T15:28:53.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.154+0000 7f5316800640 1 -- 172.21.15.47:0/420803073 learned_addr learned my addr 172.21.15.47:0/420803073 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:53.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.154+0000 7f5315e00640 1 --2- 172.21.15.47:0/420803073 >> v2:172.21.15.116:3300/0 conn(0x7f53180d0150 0x7f531806aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:53.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.154+0000 7f5316800640 1 -- 172.21.15.47:0/420803073 >> v2:172.21.15.116:3300/0 conn(0x7f53180d0150 msgr2=0x7f531806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:53.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.154+0000 7f5316800640 1 --2- 172.21.15.47:0/420803073 >> v2:172.21.15.116:3300/0 conn(0x7f53180d0150 0x7f531806aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:53.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.154+0000 7f5316800640 1 -- 172.21.15.47:0/420803073 >> v1:172.21.15.47:6789/0 conn(0x7f531806b3f0 legacy=0x7f5318161d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:53.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.154+0000 7f5315400640 1 -- 172.21.15.47:0/420803073 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806824796 0 0) 0x7f53180ce660 con 0x7f531806b3f0 2024-02-19T15:28:53.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.154+0000 7f5316800640 1 -- 172.21.15.47:0/420803073 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531806b850 con 0x7f5318162460 2024-02-19T15:28:53.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.158+0000 7f5316800640 1 --2- 172.21.15.47:0/420803073 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5318162460 0x7f53180ae4d0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f5318067610 tx=0x7f531001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53644429e9de8e47 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:53.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.158+0000 7f5315400640 1 -- 172.21.15.47:0/420803073 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53100a57a0 con 0x7f5318162460 2024-02-19T15:28:53.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.158+0000 7f5315400640 1 -- 172.21.15.47:0/420803073 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53100a5900 con 0x7f5318162460 2024-02-19T15:28:53.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.158+0000 7f5315400640 1 -- 172.21.15.47:0/420803073 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53100a5c90 con 0x7f5318162460 2024-02-19T15:28:53.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.158+0000 7f531d200640 1 -- 172.21.15.47:0/420803073 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5318162460 msgr2=0x7f53180ae4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:53.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.158+0000 7f531d200640 1 --2- 172.21.15.47:0/420803073 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5318162460 0x7f53180ae4d0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f5318067610 tx=0x7f531001c9f0 comp rx=0 tx=0).stop 2024-02-19T15:28:53.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.158+0000 7f531d200640 1 -- 172.21.15.47:0/420803073 shutdown_connections 2024-02-19T15:28:53.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.158+0000 7f531d200640 1 --2- 172.21.15.47:0/420803073 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5318162460 0x7f53180ae4d0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:53.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.158+0000 7f531d200640 1 --2- 172.21.15.47:0/420803073 >> v2:172.21.15.116:3300/0 conn(0x7f53180d0150 0x7f531806aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:53.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.158+0000 7f531d200640 1 -- 172.21.15.47:0/420803073 >> 172.21.15.47:0/420803073 conn(0x7f53180b2580 msgr2=0x7f53180b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:53.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.158+0000 7f531d200640 1 -- 172.21.15.47:0/420803073 shutdown_connections 2024-02-19T15:28:53.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.158+0000 7f531d200640 1 -- 172.21.15.47:0/420803073 wait complete. 2024-02-19T15:28:53.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f531d200640 1 Processor -- start 2024-02-19T15:28:53.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f531d200640 1 -- start start 2024-02-19T15:28:53.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f531d200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f531806b3f0 0x7f531815a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:53.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f531d200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53180d0150 0x7f531815d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:53.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f531d200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5318162460 0x7f531815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:53.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f531d200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5318150e70 con 0x7f5318162460 2024-02-19T15:28:53.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f531d200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531815c720 con 0x7f531806b3f0 2024-02-19T15:28:53.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f531d200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531815ca00 con 0x7f53180d0150 2024-02-19T15:28:53.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f5316800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5318162460 0x7f531815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:53.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f5316800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5318162460 0x7f531815fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:40890/0 (socket says 172.21.15.47:40890) 2024-02-19T15:28:53.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f5316800640 1 -- 172.21.15.47:0/2096696908 learned_addr learned my addr 172.21.15.47:0/2096696908 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:53.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f5315e00640 1 --2- 172.21.15.47:0/2096696908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f531806b3f0 0x7f531815a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:53.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f530fe00640 1 --2- 172.21.15.47:0/2096696908 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53180d0150 0x7f531815d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:53.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f5316800640 1 -- 172.21.15.47:0/2096696908 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53180d0150 msgr2=0x7f531815d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:53.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f5316800640 1 --2- 172.21.15.47:0/2096696908 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53180d0150 0x7f531815d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:53.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f5316800640 1 -- 172.21.15.47:0/2096696908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f531806b3f0 msgr2=0x7f531815a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:53.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f5316800640 1 --2- 172.21.15.47:0/2096696908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f531806b3f0 0x7f531815a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:53.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f5316800640 1 -- 172.21.15.47:0/2096696908 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5310009b30 con 0x7f5318162460 2024-02-19T15:28:53.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f5316800640 1 --2- 172.21.15.47:0/2096696908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5318162460 0x7f531815fe30 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f5310019600 tx=0x7f5310019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:53.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f530e000640 1 -- 172.21.15.47:0/2096696908 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53100b8470 con 0x7f5318162460 2024-02-19T15:28:53.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f530e000640 1 -- 172.21.15.47:0/2096696908 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53100b85d0 con 0x7f5318162460 2024-02-19T15:28:53.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f530e000640 1 -- 172.21.15.47:0/2096696908 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53100b8940 con 0x7f5318162460 2024-02-19T15:28:53.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.162+0000 7f531d200640 1 -- 172.21.15.47:0/2096696908 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f531815cce0 con 0x7f5318162460 2024-02-19T15:28:53.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.166+0000 7f531d200640 1 -- 172.21.15.47:0/2096696908 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f531815d2a0 con 0x7f5318162460 2024-02-19T15:28:53.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.166+0000 7f531d200640 1 -- 172.21.15.47:0/2096696908 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5318075360 con 0x7f5318162460 2024-02-19T15:28:53.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.166+0000 7f530e000640 1 -- 172.21.15.47:0/2096696908 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f53100b8aa0 con 0x7f5318162460 2024-02-19T15:28:53.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.166+0000 7f530e000640 1 --2- 172.21.15.47:0/2096696908 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52f4066e40 0x7f52f4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:53.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.166+0000 7f530e000640 1 -- 172.21.15.47:0/2096696908 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f53101200a0 con 0x7f5318162460 2024-02-19T15:28:53.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.170+0000 7f5315e00640 1 --2- 172.21.15.47:0/2096696908 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52f4066e40 0x7f52f4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:53.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.170+0000 7f5315e00640 1 --2- 172.21.15.47:0/2096696908 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52f4066e40 0x7f52f4069300 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f5308009b60 tx=0x7f5308009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:53.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.170+0000 7f530e000640 1 -- 172.21.15.47:0/2096696908 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f53100ed190 con 0x7f5318162460 2024-02-19T15:28:53.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.334+0000 7f531d200640 1 -- 172.21.15.47:0/2096696908 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f53180b3420 con 0x7f5318162460 2024-02-19T15:28:53.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.986+0000 7f530e000640 1 -- 172.21.15.47:0/2096696908 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v123) v1 ==== 170+0+0 (secure 0 0 0) 0x7f53100f0850 con 0x7f5318162460 2024-02-19T15:28:53.988 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'cephfs_data' created 2024-02-19T15:28:53.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.990+0000 7f531d200640 1 -- 172.21.15.47:0/2096696908 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52f4066e40 msgr2=0x7f52f4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:53.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.990+0000 7f531d200640 1 --2- 172.21.15.47:0/2096696908 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52f4066e40 0x7f52f4069300 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f5308009b60 tx=0x7f5308009190 comp rx=0 tx=0).stop 2024-02-19T15:28:53.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.990+0000 7f531d200640 1 -- 172.21.15.47:0/2096696908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5318162460 msgr2=0x7f531815fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:53.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.990+0000 7f531d200640 1 --2- 172.21.15.47:0/2096696908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5318162460 0x7f531815fe30 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f5310019600 tx=0x7f5310019790 comp rx=0 tx=0).stop 2024-02-19T15:28:53.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.990+0000 7f531d200640 1 -- 172.21.15.47:0/2096696908 shutdown_connections 2024-02-19T15:28:53.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.990+0000 7f531d200640 1 --2- 172.21.15.47:0/2096696908 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f52f4066e40 0x7f52f4069300 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:53.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.990+0000 7f531d200640 1 --2- 172.21.15.47:0/2096696908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5318162460 0x7f531815fe30 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:53.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.990+0000 7f531d200640 1 --2- 172.21.15.47:0/2096696908 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53180d0150 0x7f531815d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:53.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.990+0000 7f531d200640 1 --2- 172.21.15.47:0/2096696908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f531806b3f0 0x7f531815a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:53.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.990+0000 7f531d200640 1 -- 172.21.15.47:0/2096696908 >> 172.21.15.47:0/2096696908 conn(0x7f53180b2580 msgr2=0x7f53180c3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:53.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.990+0000 7f531d200640 1 -- 172.21.15.47:0/2096696908 shutdown_connections 2024-02-19T15:28:53.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:53.990+0000 7f531d200640 1 -- 172.21.15.47:0/2096696908 wait complete. 2024-02-19T15:28:54.029 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T15:28:54.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.178+0000 7fbe97e00640 1 Processor -- start 2024-02-19T15:28:54.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.178+0000 7fbe97e00640 1 -- start start 2024-02-19T15:28:54.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.178+0000 7fbe97e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe980d2480 0x7fbe980ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:54.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.178+0000 7fbe97e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe980b0db0 con 0x7fbe9815bc60 2024-02-19T15:28:54.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.178+0000 7fbe97e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe980cf9e0 con 0x7fbe980d0650 2024-02-19T15:28:54.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.178+0000 7fbe97e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe980b0fe0 con 0x7fbe980d2480 2024-02-19T15:28:54.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.178+0000 7fbe96a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbe9815bc60 0x7fbe980cff40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41120/0 (socket says 172.21.15.47:41120) 2024-02-19T15:28:54.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.178+0000 7fbe96a00640 1 -- 172.21.15.47:0/3830616257 learned_addr learned my addr 172.21.15.47:0/3830616257 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:54.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.178+0000 7fbe97400640 1 --2- 172.21.15.47:0/3830616257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe980d2480 0x7fbe980ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:54.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.178+0000 7fbe95600640 1 -- 172.21.15.47:0/3830616257 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250642442 0 0) 0x7fbe980b0db0 con 0x7fbe9815bc60 2024-02-19T15:28:54.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.178+0000 7fbe95600640 1 -- 172.21.15.47:0/3830616257 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe7c003530 con 0x7fbe9815bc60 2024-02-19T15:28:54.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.178+0000 7fbe95600640 1 -- 172.21.15.47:0/3830616257 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561039395 0 0) 0x7fbe980cf9e0 con 0x7fbe980d0650 2024-02-19T15:28:54.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.178+0000 7fbe95600640 1 -- 172.21.15.47:0/3830616257 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe980b0db0 con 0x7fbe980d0650 2024-02-19T15:28:54.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.178+0000 7fbe97400640 1 -- 172.21.15.47:0/3830616257 >> v1:172.21.15.116:6789/0 conn(0x7fbe980d0650 legacy=0x7fbe980b1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:54.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.178+0000 7fbe97400640 1 -- 172.21.15.47:0/3830616257 >> v1:172.21.15.47:6789/0 conn(0x7fbe9815bc60 legacy=0x7fbe980cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:54.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.178+0000 7fbe97400640 1 -- 172.21.15.47:0/3830616257 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe980b12c0 con 0x7fbe980d2480 2024-02-19T15:28:54.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.182+0000 7fbe97400640 1 --2- 172.21.15.47:0/3830616257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe980d2480 0x7fbe980ce9a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbe90009dc0 tx=0x7fbe9001c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=587ed804fd3661d1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:54.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.182+0000 7fbe95600640 1 -- 172.21.15.47:0/3830616257 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe900a5990 con 0x7fbe980d2480 2024-02-19T15:28:54.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.182+0000 7fbe95600640 1 -- 172.21.15.47:0/3830616257 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe900a5af0 con 0x7fbe980d2480 2024-02-19T15:28:54.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.182+0000 7fbe95600640 1 -- 172.21.15.47:0/3830616257 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe900a5e60 con 0x7fbe980d2480 2024-02-19T15:28:54.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.182+0000 7fbe97e00640 1 -- 172.21.15.47:0/3830616257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe980d2480 msgr2=0x7fbe980ce9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:54.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.182+0000 7fbe97e00640 1 --2- 172.21.15.47:0/3830616257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe980d2480 0x7fbe980ce9a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbe90009dc0 tx=0x7fbe9001c8d0 comp rx=0 tx=0).stop 2024-02-19T15:28:54.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.182+0000 7fbe97e00640 1 -- 172.21.15.47:0/3830616257 shutdown_connections 2024-02-19T15:28:54.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.182+0000 7fbe97e00640 1 --2- 172.21.15.47:0/3830616257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe980d2480 0x7fbe980ce9a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:54.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.182+0000 7fbe97e00640 1 -- 172.21.15.47:0/3830616257 >> 172.21.15.47:0/3830616257 conn(0x7fbe980d1960 msgr2=0x7fbe980d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:54.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.182+0000 7fbe97e00640 1 -- 172.21.15.47:0/3830616257 shutdown_connections 2024-02-19T15:28:54.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.182+0000 7fbe97e00640 1 -- 172.21.15.47:0/3830616257 wait complete. 2024-02-19T15:28:54.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe97e00640 1 Processor -- start 2024-02-19T15:28:54.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe97e00640 1 -- start start 2024-02-19T15:28:54.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe97e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe980d0650 0x7fbe98156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:54.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe97e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe980d2480 0x7fbe980b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:54.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe97e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe9815bc60 0x7fbe981579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:54.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe97e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe98156aa0 con 0x7fbe980d2480 2024-02-19T15:28:54.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe97e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9815d420 con 0x7fbe980d0650 2024-02-19T15:28:54.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe97e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9815d700 con 0x7fbe9815bc60 2024-02-19T15:28:54.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe96000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe980d2480 0x7fbe980b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:54.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe96000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe980d2480 0x7fbe980b4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51008/0 (socket says 172.21.15.47:51008) 2024-02-19T15:28:54.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe96000640 1 -- 172.21.15.47:0/2214430652 learned_addr learned my addr 172.21.15.47:0/2214430652 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:54.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe96a00640 1 --2- 172.21.15.47:0/2214430652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe980d0650 0x7fbe98156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:54.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe97400640 1 --2- 172.21.15.47:0/2214430652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe9815bc60 0x7fbe981579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:54.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe96000640 1 -- 172.21.15.47:0/2214430652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe9815bc60 msgr2=0x7fbe981579c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:54.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe96000640 1 --2- 172.21.15.47:0/2214430652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe9815bc60 0x7fbe981579c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:54.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe96000640 1 -- 172.21.15.47:0/2214430652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe980d0650 msgr2=0x7fbe98156560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:54.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe96000640 1 --2- 172.21.15.47:0/2214430652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe980d0650 0x7fbe98156560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:54.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe96000640 1 -- 172.21.15.47:0/2214430652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe90009a70 con 0x7fbe980d2480 2024-02-19T15:28:54.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe97400640 1 --2- 172.21.15.47:0/2214430652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe9815bc60 0x7fbe981579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:54.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe96a00640 1 --2- 172.21.15.47:0/2214430652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe980d0650 0x7fbe98156560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:54.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe96000640 1 --2- 172.21.15.47:0/2214430652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe980d2480 0x7fbe980b4f90 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400c430 tx=0x7fbe8400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:54.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe8ea00640 1 -- 172.21.15.47:0/2214430652 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe840052d0 con 0x7fbe980d2480 2024-02-19T15:28:54.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe8ea00640 1 -- 172.21.15.47:0/2214430652 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe84005450 con 0x7fbe980d2480 2024-02-19T15:28:54.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe8ea00640 1 -- 172.21.15.47:0/2214430652 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe84003840 con 0x7fbe980d2480 2024-02-19T15:28:54.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe97e00640 1 -- 172.21.15.47:0/2214430652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9815d9e0 con 0x7fbe980d2480 2024-02-19T15:28:54.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe97e00640 1 -- 172.21.15.47:0/2214430652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe980b54d0 con 0x7fbe980d2480 2024-02-19T15:28:54.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.186+0000 7fbe97e00640 1 -- 172.21.15.47:0/2214430652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9804f260 con 0x7fbe980d2480 2024-02-19T15:28:54.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.190+0000 7fbe8ea00640 1 -- 172.21.15.47:0/2214430652 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fbe840039a0 con 0x7fbe980d2480 2024-02-19T15:28:54.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.190+0000 7fbe8ea00640 1 --2- 172.21.15.47:0/2214430652 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbe78066e70 0x7fbe78069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:54.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.190+0000 7fbe8ea00640 1 -- 172.21.15.47:0/2214430652 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fbe84084090 con 0x7fbe980d2480 2024-02-19T15:28:54.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.190+0000 7fbe96a00640 1 --2- 172.21.15.47:0/2214430652 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbe78066e70 0x7fbe78069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:54.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.190+0000 7fbe96a00640 1 --2- 172.21.15.47:0/2214430652 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbe78066e70 0x7fbe78069330 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fbe880011f0 tx=0x7fbe88009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:54.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.194+0000 7fbe8ea00640 1 -- 172.21.15.47:0/2214430652 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fbe84050e40 con 0x7fbe980d2480 2024-02-19T15:28:54.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.406+0000 7fbe97e00640 1 -- 172.21.15.47:0/2214430652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fbe9815de90 con 0x7fbe980d2480 2024-02-19T15:28:54.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.990+0000 7fbe8ea00640 1 -- 172.21.15.47:0/2214430652 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v87) v1 ==== 174+0+0 (secure 0 0 0) 0x7fbe84054500 con 0x7fbe980d2480 2024-02-19T15:28:54.991 INFO:teuthology.orchestra.run.smithi047.stderr:new fs with metadata pool 18 and data pool 19 2024-02-19T15:28:54.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.994+0000 7fbe97e00640 1 -- 172.21.15.47:0/2214430652 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbe78066e70 msgr2=0x7fbe78069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:54.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.994+0000 7fbe97e00640 1 --2- 172.21.15.47:0/2214430652 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbe78066e70 0x7fbe78069330 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fbe880011f0 tx=0x7fbe88009040 comp rx=0 tx=0).stop 2024-02-19T15:28:54.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.994+0000 7fbe97e00640 1 -- 172.21.15.47:0/2214430652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe980d2480 msgr2=0x7fbe980b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:54.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.994+0000 7fbe97e00640 1 --2- 172.21.15.47:0/2214430652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe980d2480 0x7fbe980b4f90 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400c430 tx=0x7fbe8400c8f0 comp rx=0 tx=0).stop 2024-02-19T15:28:54.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.994+0000 7fbe97e00640 1 -- 172.21.15.47:0/2214430652 shutdown_connections 2024-02-19T15:28:54.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.994+0000 7fbe97e00640 1 --2- 172.21.15.47:0/2214430652 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbe78066e70 0x7fbe78069330 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:54.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.994+0000 7fbe97e00640 1 --2- 172.21.15.47:0/2214430652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe9815bc60 0x7fbe981579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:54.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.994+0000 7fbe97e00640 1 --2- 172.21.15.47:0/2214430652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbe980d2480 0x7fbe980b4f90 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:54.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.994+0000 7fbe97e00640 1 --2- 172.21.15.47:0/2214430652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe980d0650 0x7fbe98156560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:54.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.994+0000 7fbe97e00640 1 -- 172.21.15.47:0/2214430652 >> 172.21.15.47:0/2214430652 conn(0x7fbe980d1960 msgr2=0x7fbe980ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:54.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.994+0000 7fbe97e00640 1 -- 172.21.15.47:0/2214430652 shutdown_connections 2024-02-19T15:28:54.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:54.994+0000 7fbe97e00640 1 -- 172.21.15.47:0/2214430652 wait complete. 2024-02-19T15:28:55.041 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:28:55.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.182+0000 7fd6b1000640 1 Processor -- start 2024-02-19T15:28:55.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.182+0000 7fd6b1000640 1 -- start start 2024-02-19T15:28:55.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.186+0000 7fd6b1000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd6ac1581f0 0x7fd6ac0b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:55.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.186+0000 7fd6b1000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6ac1585d0 0x7fd6ac0b55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:55.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.186+0000 7fd6b1000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ac161aa0 con 0x7fd6ac0ad8b0 2024-02-19T15:28:55.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.186+0000 7fd6b1000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6ac162400 con 0x7fd6ac1581f0 2024-02-19T15:28:55.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.186+0000 7fd6b1000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac0b37f0 con 0x7fd6ac1585d0 2024-02-19T15:28:55.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.186+0000 7fd6ab400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd6ac1581f0 0x7fd6ac0b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:55.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.186+0000 7fd6ab400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd6ac1581f0 0x7fd6ac0b50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51010/0 (socket says 172.21.15.47:51010) 2024-02-19T15:28:55.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.186+0000 7fd6ab400640 1 -- 172.21.15.47:0/3535640312 learned_addr learned my addr 172.21.15.47:0/3535640312 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:55.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.186+0000 7fd6a2a00640 1 --2- 172.21.15.47:0/3535640312 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6ac1585d0 0x7fd6ac0b55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:55.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.186+0000 7fd6ab400640 1 -- 172.21.15.47:0/3535640312 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6ac1585d0 msgr2=0x7fd6ac0b55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:55.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.186+0000 7fd6ab400640 1 --2- 172.21.15.47:0/3535640312 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6ac1585d0 0x7fd6ac0b55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.186+0000 7fd6ab400640 1 -- 172.21.15.47:0/3535640312 >> v1:172.21.15.116:6789/0 conn(0x7fd6ac0ad8b0 legacy=0x7fd6ac0add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:55.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.186+0000 7fd6ab400640 1 -- 172.21.15.47:0/3535640312 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ac15f310 con 0x7fd6ac1581f0 2024-02-19T15:28:55.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.186+0000 7fd6a2a00640 1 --2- 172.21.15.47:0/3535640312 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6ac1585d0 0x7fd6ac0b55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:55.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.186+0000 7fd6ab400640 1 --2- 172.21.15.47:0/3535640312 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6ac1581f0 0x7fd6ac0b50a0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fd69c009870 tx=0x7fd69c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d898c0a6c6e47ac server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:55.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6aaa00640 1 -- 172.21.15.47:0/3535640312 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c0ab600 con 0x7fd6ac1581f0 2024-02-19T15:28:55.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6aaa00640 1 -- 172.21.15.47:0/3535640312 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69c0a7050 con 0x7fd6ac1581f0 2024-02-19T15:28:55.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6aaa00640 1 -- 172.21.15.47:0/3535640312 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c0ab980 con 0x7fd6ac1581f0 2024-02-19T15:28:55.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6b1000640 1 -- 172.21.15.47:0/3535640312 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6ac1581f0 msgr2=0x7fd6ac0b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:55.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6b1000640 1 --2- 172.21.15.47:0/3535640312 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6ac1581f0 0x7fd6ac0b50a0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fd69c009870 tx=0x7fd69c01c490 comp rx=0 tx=0).stop 2024-02-19T15:28:55.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6b1000640 1 -- 172.21.15.47:0/3535640312 shutdown_connections 2024-02-19T15:28:55.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6b1000640 1 --2- 172.21.15.47:0/3535640312 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6ac1585d0 0x7fd6ac0b55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6b1000640 1 --2- 172.21.15.47:0/3535640312 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6ac1581f0 0x7fd6ac0b50a0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6b1000640 1 -- 172.21.15.47:0/3535640312 >> 172.21.15.47:0/3535640312 conn(0x7fd6ac0b24f0 msgr2=0x7fd6ac0b2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:55.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6b1000640 1 -- 172.21.15.47:0/3535640312 shutdown_connections 2024-02-19T15:28:55.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6b1000640 1 -- 172.21.15.47:0/3535640312 wait complete. 2024-02-19T15:28:55.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6b1000640 1 Processor -- start 2024-02-19T15:28:55.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6b1000640 1 -- start start 2024-02-19T15:28:55.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6b1000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6ac0ad8b0 0x7fd6ac155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:55.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6b1000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6ac1581f0 0x7fd6ac06aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:55.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6b1000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6ac1585d0 0x7fd6ac06b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:55.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6b1000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac14e5e0 con 0x7fd6ac0ad8b0 2024-02-19T15:28:55.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6b1000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac156660 con 0x7fd6ac1581f0 2024-02-19T15:28:55.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6b1000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac14e880 con 0x7fd6ac1585d0 2024-02-19T15:28:55.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6ab400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6ac0ad8b0 0x7fd6ac155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:55.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6ab400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6ac0ad8b0 0x7fd6ac155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51026/0 (socket says 172.21.15.47:51026) 2024-02-19T15:28:55.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.190+0000 7fd6ab400640 1 -- 172.21.15.47:0/2200789895 learned_addr learned my addr 172.21.15.47:0/2200789895 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:55.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.194+0000 7fd6abe00640 1 --2- 172.21.15.47:0/2200789895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6ac1585d0 0x7fd6ac06b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:55.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.194+0000 7fd6a2a00640 1 --2- 172.21.15.47:0/2200789895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6ac1581f0 0x7fd6ac06aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:55.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.194+0000 7fd6ab400640 1 -- 172.21.15.47:0/2200789895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6ac1585d0 msgr2=0x7fd6ac06b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:55.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.194+0000 7fd6ab400640 1 --2- 172.21.15.47:0/2200789895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6ac1585d0 0x7fd6ac06b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.194+0000 7fd6ab400640 1 -- 172.21.15.47:0/2200789895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6ac1581f0 msgr2=0x7fd6ac06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:55.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.194+0000 7fd6ab400640 1 --2- 172.21.15.47:0/2200789895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6ac1581f0 0x7fd6ac06aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.194+0000 7fd6ab400640 1 -- 172.21.15.47:0/2200789895 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69c0094d0 con 0x7fd6ac0ad8b0 2024-02-19T15:28:55.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.194+0000 7fd6ab400640 1 --2- 172.21.15.47:0/2200789895 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6ac0ad8b0 0x7fd6ac155cf0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fd69c01cd30 tx=0x7fd69c019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:55.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.194+0000 7fd6a3e00640 1 -- 172.21.15.47:0/2200789895 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c0b6640 con 0x7fd6ac0ad8b0 2024-02-19T15:28:55.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.194+0000 7fd6a3e00640 1 -- 172.21.15.47:0/2200789895 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69c0b67a0 con 0x7fd6ac0ad8b0 2024-02-19T15:28:55.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.194+0000 7fd6a3e00640 1 -- 172.21.15.47:0/2200789895 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c0b6b50 con 0x7fd6ac0ad8b0 2024-02-19T15:28:55.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.194+0000 7fd6b1000640 1 -- 172.21.15.47:0/2200789895 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6ac14eb60 con 0x7fd6ac0ad8b0 2024-02-19T15:28:55.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.194+0000 7fd6b1000640 1 -- 172.21.15.47:0/2200789895 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6ac156230 con 0x7fd6ac0ad8b0 2024-02-19T15:28:55.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.194+0000 7fd6b1000640 1 -- 172.21.15.47:0/2200789895 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6ac075360 con 0x7fd6ac0ad8b0 2024-02-19T15:28:55.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.194+0000 7fd6a3e00640 1 -- 172.21.15.47:0/2200789895 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd69c0b2040 con 0x7fd6ac0ad8b0 2024-02-19T15:28:55.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.194+0000 7fd6a3e00640 1 --2- 172.21.15.47:0/2200789895 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd688066e40 0x7fd688069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:55.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.194+0000 7fd6a3e00640 1 -- 172.21.15.47:0/2200789895 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd69c11f960 con 0x7fd6ac0ad8b0 2024-02-19T15:28:55.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.198+0000 7fd6a2a00640 1 --2- 172.21.15.47:0/2200789895 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd688066e40 0x7fd688069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:55.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.198+0000 7fd6a2a00640 1 --2- 172.21.15.47:0/2200789895 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd688066e40 0x7fd688069300 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd694006fd0 tx=0x7fd694008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:55.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.198+0000 7fd6a3e00640 1 -- 172.21.15.47:0/2200789895 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd69c0ec7c0 con 0x7fd6ac0ad8b0 2024-02-19T15:28:55.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.362+0000 7fd6b1000640 1 -- 172.21.15.47:0/2200789895 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6ac154390 con 0x7fd6ac0ad8b0 2024-02-19T15:28:55.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.366+0000 7fd6a3e00640 1 -- 172.21.15.47:0/2200789895 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+16704 (secure 0 0 0) 0x7fd69c0b6cb0 con 0x7fd6ac0ad8b0 2024-02-19T15:28:55.365 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:55.366 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":124,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:28:54.408957+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:28:52.814262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-02-19T15:28:53.381287+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:28:55.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.366+0000 7fd6b1000640 1 -- 172.21.15.47:0/2200789895 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd688066e40 msgr2=0x7fd688069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:55.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.366+0000 7fd6b1000640 1 --2- 172.21.15.47:0/2200789895 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd688066e40 0x7fd688069300 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd694006fd0 tx=0x7fd694008040 comp rx=0 tx=0).stop 2024-02-19T15:28:55.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.366+0000 7fd6b1000640 1 -- 172.21.15.47:0/2200789895 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6ac0ad8b0 msgr2=0x7fd6ac155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:55.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.366+0000 7fd6b1000640 1 --2- 172.21.15.47:0/2200789895 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6ac0ad8b0 0x7fd6ac155cf0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fd69c01cd30 tx=0x7fd69c019230 comp rx=0 tx=0).stop 2024-02-19T15:28:55.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.366+0000 7fd6b1000640 1 -- 172.21.15.47:0/2200789895 shutdown_connections 2024-02-19T15:28:55.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.366+0000 7fd6b1000640 1 --2- 172.21.15.47:0/2200789895 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd688066e40 0x7fd688069300 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.366+0000 7fd6b1000640 1 --2- 172.21.15.47:0/2200789895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6ac1585d0 0x7fd6ac06b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.366+0000 7fd6b1000640 1 --2- 172.21.15.47:0/2200789895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6ac1581f0 0x7fd6ac06aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.366+0000 7fd6b1000640 1 --2- 172.21.15.47:0/2200789895 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6ac0ad8b0 0x7fd6ac155cf0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.366+0000 7fd6b1000640 1 -- 172.21.15.47:0/2200789895 >> 172.21.15.47:0/2200789895 conn(0x7fd6ac0b24f0 msgr2=0x7fd6ac1548b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:55.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.366+0000 7fd6b1000640 1 -- 172.21.15.47:0/2200789895 shutdown_connections 2024-02-19T15:28:55.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.366+0000 7fd6b1000640 1 -- 172.21.15.47:0/2200789895 wait complete. 2024-02-19T15:28:55.387 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:28:55.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.526+0000 7fecaa000640 1 Processor -- start 2024-02-19T15:28:55.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.526+0000 7fecaa000640 1 -- start start 2024-02-19T15:28:55.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.526+0000 7fecaa000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feca415b1a0 0x7feca415b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:55.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.526+0000 7fecaa000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feca4071350 0x7feca415bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:55.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.526+0000 7fecaa000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7feca406c9a0 0x7feca415c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:55.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.526+0000 7fecaa000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca415c000 con 0x7feca406c9a0 2024-02-19T15:28:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.526+0000 7fecaa000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca40718c0 con 0x7feca415b1a0 2024-02-19T15:28:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.526+0000 7fecaa000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca4071ba0 con 0x7feca4071350 2024-02-19T15:28:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.526+0000 7feca3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7feca406c9a0 0x7feca415c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.526+0000 7feca3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7feca406c9a0 0x7feca415c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51036/0 (socket says 172.21.15.47:51036) 2024-02-19T15:28:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.526+0000 7feca3e00640 1 -- 172.21.15.47:0/3523170487 learned_addr learned my addr 172.21.15.47:0/3523170487 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.526+0000 7fec9aa00640 1 --2- 172.21.15.47:0/3523170487 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feca4071350 0x7feca415bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.526+0000 7feca3400640 1 --2- 172.21.15.47:0/3523170487 >> v2:172.21.15.116:3300/0 conn(0x7feca415b1a0 0x7feca415b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.526+0000 7feca3e00640 1 -- 172.21.15.47:0/3523170487 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feca4071350 msgr2=0x7feca415bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.526+0000 7feca3e00640 1 --2- 172.21.15.47:0/3523170487 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feca4071350 0x7feca415bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.526+0000 7feca3e00640 1 -- 172.21.15.47:0/3523170487 >> v2:172.21.15.116:3300/0 conn(0x7feca415b1a0 msgr2=0x7feca415b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.526+0000 7feca3e00640 1 --2- 172.21.15.47:0/3523170487 >> v2:172.21.15.116:3300/0 conn(0x7feca415b1a0 0x7feca415b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.526+0000 7feca3e00640 1 -- 172.21.15.47:0/3523170487 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca4071e80 con 0x7feca406c9a0 2024-02-19T15:28:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.526+0000 7fec9aa00640 1 --2- 172.21.15.47:0/3523170487 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feca4071350 0x7feca415bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:55.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.530+0000 7feca3e00640 1 --2- 172.21.15.47:0/3523170487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feca406c9a0 0x7feca415c6d0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7feca4067b40 tx=0x7fec9c01c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e0114b4f9e95e38 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:55.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.530+0000 7feca2a00640 1 -- 172.21.15.47:0/3523170487 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c0a5860 con 0x7feca406c9a0 2024-02-19T15:28:55.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.530+0000 7feca2a00640 1 -- 172.21.15.47:0/3523170487 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9c0a59c0 con 0x7feca406c9a0 2024-02-19T15:28:55.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.530+0000 7feca2a00640 1 -- 172.21.15.47:0/3523170487 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c0a5d50 con 0x7feca406c9a0 2024-02-19T15:28:55.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.530+0000 7fecaa000640 1 -- 172.21.15.47:0/3523170487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feca406c9a0 msgr2=0x7feca415c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:55.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.530+0000 7fecaa000640 1 --2- 172.21.15.47:0/3523170487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feca406c9a0 0x7feca415c6d0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7feca4067b40 tx=0x7fec9c01c9c0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.530+0000 7fecaa000640 1 -- 172.21.15.47:0/3523170487 shutdown_connections 2024-02-19T15:28:55.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.530+0000 7fecaa000640 1 --2- 172.21.15.47:0/3523170487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feca406c9a0 0x7feca415c6d0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.530+0000 7fecaa000640 1 --2- 172.21.15.47:0/3523170487 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feca4071350 0x7feca415bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.530+0000 7fecaa000640 1 --2- 172.21.15.47:0/3523170487 >> v2:172.21.15.116:3300/0 conn(0x7feca415b1a0 0x7feca415b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.530+0000 7fecaa000640 1 -- 172.21.15.47:0/3523170487 >> 172.21.15.47:0/3523170487 conn(0x7feca40b16a0 msgr2=0x7feca40b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:55.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.530+0000 7fecaa000640 1 -- 172.21.15.47:0/3523170487 shutdown_connections 2024-02-19T15:28:55.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.530+0000 7fecaa000640 1 -- 172.21.15.47:0/3523170487 wait complete. 2024-02-19T15:28:55.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7fecaa000640 1 Processor -- start 2024-02-19T15:28:55.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7fecaa000640 1 -- start start 2024-02-19T15:28:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7fecaa000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feca406c9a0 0x7feca414cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7fecaa000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feca4071350 0x7feca40b3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7fecaa000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feca415b1a0 0x7feca40cde20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7fecaa000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca414d1d0 con 0x7feca406c9a0 2024-02-19T15:28:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7fecaa000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca40ceaf0 con 0x7feca415b1a0 2024-02-19T15:28:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7fecaa000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca40cedd0 con 0x7feca4071350 2024-02-19T15:28:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7feca3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feca406c9a0 0x7feca414cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7feca3400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feca406c9a0 0x7feca414cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51050/0 (socket says 172.21.15.47:51050) 2024-02-19T15:28:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7feca3400640 1 -- 172.21.15.47:0/3674024897 learned_addr learned my addr 172.21.15.47:0/3674024897 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7fec9aa00640 1 --2- 172.21.15.47:0/3674024897 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feca4071350 0x7feca40b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7feca3e00640 1 --2- 172.21.15.47:0/3674024897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feca415b1a0 0x7feca40cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7feca3400640 1 -- 172.21.15.47:0/3674024897 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feca4071350 msgr2=0x7feca40b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7feca3400640 1 --2- 172.21.15.47:0/3674024897 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feca4071350 0x7feca40b3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7feca3400640 1 -- 172.21.15.47:0/3674024897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feca415b1a0 msgr2=0x7feca40cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7feca3400640 1 --2- 172.21.15.47:0/3674024897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feca415b1a0 0x7feca40cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7feca3400640 1 -- 172.21.15.47:0/3674024897 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9c009b30 con 0x7feca406c9a0 2024-02-19T15:28:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7feca3e00640 1 --2- 172.21.15.47:0/3674024897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feca415b1a0 0x7feca40cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7feca3400640 1 --2- 172.21.15.47:0/3674024897 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feca406c9a0 0x7feca414cc90 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fec9400d800 tx=0x7fec9400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7fec9be00640 1 -- 172.21.15.47:0/3674024897 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9400b6c0 con 0x7feca406c9a0 2024-02-19T15:28:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7fec9be00640 1 -- 172.21.15.47:0/3674024897 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9400b820 con 0x7feca406c9a0 2024-02-19T15:28:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7fec9be00640 1 -- 172.21.15.47:0/3674024897 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9400bb90 con 0x7feca406c9a0 2024-02-19T15:28:55.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.534+0000 7fecaa000640 1 -- 172.21.15.47:0/3674024897 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca40cf0b0 con 0x7feca406c9a0 2024-02-19T15:28:55.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.538+0000 7fecaa000640 1 -- 172.21.15.47:0/3674024897 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca414d5e0 con 0x7feca406c9a0 2024-02-19T15:28:55.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.538+0000 7fecaa000640 1 -- 172.21.15.47:0/3674024897 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca4075360 con 0x7feca406c9a0 2024-02-19T15:28:55.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.538+0000 7fec9be00640 1 -- 172.21.15.47:0/3674024897 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fec9400bcf0 con 0x7feca406c9a0 2024-02-19T15:28:55.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.538+0000 7fec9be00640 1 --2- 172.21.15.47:0/3674024897 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fec80066e40 0x7fec80069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:55.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.538+0000 7fec9be00640 1 -- 172.21.15.47:0/3674024897 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec94084f60 con 0x7feca406c9a0 2024-02-19T15:28:55.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.538+0000 7fec9aa00640 1 --2- 172.21.15.47:0/3674024897 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fec80066e40 0x7fec80069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:55.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.538+0000 7fec9aa00640 1 --2- 172.21.15.47:0/3674024897 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fec80066e40 0x7fec80069300 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7feca4150b90 tx=0x7fec8c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:55.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.542+0000 7fec9be00640 1 -- 172.21.15.47:0/3674024897 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fec94051d40 con 0x7feca406c9a0 2024-02-19T15:28:55.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.706+0000 7fecaa000640 1 -- 172.21.15.47:0/3674024897 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feca4150fc0 con 0x7feca406c9a0 2024-02-19T15:28:55.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.706+0000 7fec9be00640 1 -- 172.21.15.47:0/3674024897 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+16704 (secure 0 0 0) 0x7fec94055400 con 0x7feca406c9a0 2024-02-19T15:28:55.708 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:55.708 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":124,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:28:54.408957+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:28:52.814262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-02-19T15:28:53.381287+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:28:55.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.710+0000 7fecaa000640 1 -- 172.21.15.47:0/3674024897 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fec80066e40 msgr2=0x7fec80069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:55.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.710+0000 7fecaa000640 1 --2- 172.21.15.47:0/3674024897 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fec80066e40 0x7fec80069300 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7feca4150b90 tx=0x7fec8c008040 comp rx=0 tx=0).stop 2024-02-19T15:28:55.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.710+0000 7fecaa000640 1 -- 172.21.15.47:0/3674024897 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feca406c9a0 msgr2=0x7feca414cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:55.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.710+0000 7fecaa000640 1 --2- 172.21.15.47:0/3674024897 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feca406c9a0 0x7feca414cc90 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fec9400d800 tx=0x7fec9400dcc0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.710+0000 7fecaa000640 1 -- 172.21.15.47:0/3674024897 shutdown_connections 2024-02-19T15:28:55.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.710+0000 7fecaa000640 1 --2- 172.21.15.47:0/3674024897 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fec80066e40 0x7fec80069300 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.710+0000 7fecaa000640 1 --2- 172.21.15.47:0/3674024897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feca415b1a0 0x7feca40cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.710+0000 7fecaa000640 1 --2- 172.21.15.47:0/3674024897 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feca4071350 0x7feca40b3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.710+0000 7fecaa000640 1 --2- 172.21.15.47:0/3674024897 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feca406c9a0 0x7feca414cc90 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.710+0000 7fecaa000640 1 -- 172.21.15.47:0/3674024897 >> 172.21.15.47:0/3674024897 conn(0x7feca40b16a0 msgr2=0x7feca415a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:55.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.710+0000 7fecaa000640 1 -- 172.21.15.47:0/3674024897 shutdown_connections 2024-02-19T15:28:55.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.710+0000 7fecaa000640 1 -- 172.21.15.47:0/3674024897 wait complete. 2024-02-19T15:28:55.729 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T15:28:55.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.870+0000 7fd75c000640 1 Processor -- start 2024-02-19T15:28:55.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.870+0000 7fd75c000640 1 -- start start 2024-02-19T15:28:55.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.874+0000 7fd75c000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd7540b1280 0x7fd75406aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:55.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.874+0000 7fd75c000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd75415e2b0 0x7fd75406b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:55.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.874+0000 7fd75c000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7540b04c0 con 0x7fd75415ded0 2024-02-19T15:28:55.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.874+0000 7fd75c000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7540b07a0 con 0x7fd75415e2b0 2024-02-19T15:28:55.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.874+0000 7fd75c000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7540b0a80 con 0x7fd7540b1280 2024-02-19T15:28:55.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.874+0000 7fd759600640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd75415ded0 0x7fd754066cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41124/0 (socket says 172.21.15.47:41124) 2024-02-19T15:28:55.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.874+0000 7fd759600640 1 -- 172.21.15.47:0/456957863 learned_addr learned my addr 172.21.15.47:0/456957863 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:55.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.874+0000 7fd75a000640 1 --2- 172.21.15.47:0/456957863 >> v2:172.21.15.47:3300/0 conn(0x7fd75415e2b0 0x7fd75406b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.874+0000 7fd758c00640 1 --2- 172.21.15.47:0/456957863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd7540b1280 0x7fd75406aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.874+0000 7fd753e00640 1 -- 172.21.15.47:0/456957863 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379830726 0 0) 0x7fd7540b04c0 con 0x7fd75415ded0 2024-02-19T15:28:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.874+0000 7fd753e00640 1 -- 172.21.15.47:0/456957863 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd73c003530 con 0x7fd75415ded0 2024-02-19T15:28:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.874+0000 7fd75a000640 1 -- 172.21.15.47:0/456957863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd7540b1280 msgr2=0x7fd75406aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.874+0000 7fd75a000640 1 --2- 172.21.15.47:0/456957863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd7540b1280 0x7fd75406aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.874+0000 7fd75a000640 1 -- 172.21.15.47:0/456957863 >> v1:172.21.15.47:6789/0 conn(0x7fd75415ded0 legacy=0x7fd754066cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:55.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.874+0000 7fd75a000640 1 -- 172.21.15.47:0/456957863 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7540b0d60 con 0x7fd75415e2b0 2024-02-19T15:28:55.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.878+0000 7fd75a000640 1 --2- 172.21.15.47:0/456957863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd75415e2b0 0x7fd75406b3f0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fd74c009d90 tx=0x7fd74c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a8f0ee1820260f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:55.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.878+0000 7fd753e00640 1 -- 172.21.15.47:0/456957863 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c0a5a50 con 0x7fd75415e2b0 2024-02-19T15:28:55.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.878+0000 7fd753e00640 1 -- 172.21.15.47:0/456957863 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74c0a5bb0 con 0x7fd75415e2b0 2024-02-19T15:28:55.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.878+0000 7fd753e00640 1 -- 172.21.15.47:0/456957863 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c0ad440 con 0x7fd75415e2b0 2024-02-19T15:28:55.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.878+0000 7fd75c000640 1 -- 172.21.15.47:0/456957863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd75415e2b0 msgr2=0x7fd75406b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:55.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.878+0000 7fd75c000640 1 --2- 172.21.15.47:0/456957863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd75415e2b0 0x7fd75406b3f0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fd74c009d90 tx=0x7fd74c01c970 comp rx=0 tx=0).stop 2024-02-19T15:28:55.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.878+0000 7fd75c000640 1 -- 172.21.15.47:0/456957863 shutdown_connections 2024-02-19T15:28:55.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.878+0000 7fd75c000640 1 --2- 172.21.15.47:0/456957863 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd75415e2b0 0x7fd75406b3f0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.878+0000 7fd75c000640 1 --2- 172.21.15.47:0/456957863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd7540b1280 0x7fd75406aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.878+0000 7fd75c000640 1 -- 172.21.15.47:0/456957863 >> 172.21.15.47:0/456957863 conn(0x7fd7540b32f0 msgr2=0x7fd7540b36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:55.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.878+0000 7fd75c000640 1 -- 172.21.15.47:0/456957863 shutdown_connections 2024-02-19T15:28:55.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.878+0000 7fd75c000640 1 -- 172.21.15.47:0/456957863 wait complete. 2024-02-19T15:28:55.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.878+0000 7fd75c000640 1 Processor -- start 2024-02-19T15:28:55.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.878+0000 7fd75c000640 1 -- start start 2024-02-19T15:28:55.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd75c000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7540b1280 0x7fd754154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:55.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd75c000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd75415ded0 0x7fd75415be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:55.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd75c000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75415e2b0 0x7fd75415c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:55.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd75c000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd754155880 con 0x7fd7540b1280 2024-02-19T15:28:55.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd75c000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd754157660 con 0x7fd75415e2b0 2024-02-19T15:28:55.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd75c000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd754155f70 con 0x7fd75415ded0 2024-02-19T15:28:55.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd759600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7540b1280 0x7fd754154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:55.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd759600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7540b1280 0x7fd754154f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51060/0 (socket says 172.21.15.47:51060) 2024-02-19T15:28:55.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd759600640 1 -- 172.21.15.47:0/1332389721 learned_addr learned my addr 172.21.15.47:0/1332389721 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:55.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd75a000640 1 --2- 172.21.15.47:0/1332389721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75415e2b0 0x7fd75415c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:55.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd758c00640 1 --2- 172.21.15.47:0/1332389721 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd75415ded0 0x7fd75415be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:55.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd759600640 1 -- 172.21.15.47:0/1332389721 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd75415ded0 msgr2=0x7fd75415be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:55.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd759600640 1 --2- 172.21.15.47:0/1332389721 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd75415ded0 0x7fd75415be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd759600640 1 -- 172.21.15.47:0/1332389721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75415e2b0 msgr2=0x7fd75415c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:55.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd759600640 1 --2- 172.21.15.47:0/1332389721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75415e2b0 0x7fd75415c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:55.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd759600640 1 -- 172.21.15.47:0/1332389721 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74c009a70 con 0x7fd7540b1280 2024-02-19T15:28:55.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd758c00640 1 --2- 172.21.15.47:0/1332389721 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd75415ded0 0x7fd75415be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:55.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd759600640 1 --2- 172.21.15.47:0/1332389721 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7540b1280 0x7fd754154f20 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fd74800c450 tx=0x7fd74800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:55.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd751600640 1 -- 172.21.15.47:0/1332389721 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd748005cf0 con 0x7fd7540b1280 2024-02-19T15:28:55.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd751600640 1 -- 172.21.15.47:0/1332389721 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd748005e50 con 0x7fd7540b1280 2024-02-19T15:28:55.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd751600640 1 -- 172.21.15.47:0/1332389721 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd748007290 con 0x7fd7540b1280 2024-02-19T15:28:55.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd75c000640 1 -- 172.21.15.47:0/1332389721 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd754155ad0 con 0x7fd7540b1280 2024-02-19T15:28:55.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd75c000640 1 -- 172.21.15.47:0/1332389721 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd754156390 con 0x7fd7540b1280 2024-02-19T15:28:55.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd75c000640 1 -- 172.21.15.47:0/1332389721 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd754075360 con 0x7fd7540b1280 2024-02-19T15:28:55.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.882+0000 7fd751600640 1 -- 172.21.15.47:0/1332389721 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd7480073f0 con 0x7fd7540b1280 2024-02-19T15:28:55.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.886+0000 7fd751600640 1 --2- 172.21.15.47:0/1332389721 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd734066e40 0x7fd734069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:55.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.886+0000 7fd751600640 1 -- 172.21.15.47:0/1332389721 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd748083fe0 con 0x7fd7540b1280 2024-02-19T15:28:55.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.886+0000 7fd758c00640 1 --2- 172.21.15.47:0/1332389721 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd734066e40 0x7fd734069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:55.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.886+0000 7fd758c00640 1 --2- 172.21.15.47:0/1332389721 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd734066e40 0x7fd734069300 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fd75409d4f0 tx=0x7fd740008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:55.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:55.890+0000 7fd751600640 1 -- 172.21.15.47:0/1332389721 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd748050de0 con 0x7fd7540b1280 2024-02-19T15:28:56.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:56.094+0000 7fd75c000640 1 -- 172.21.15.47:0/1332389721 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd7540cf710 con 0x7fd7540b1280 2024-02-19T15:28:56.972 INFO:tasks.ceph.mds.a.smithi047.stderr:2024-02-19T15:28:56.970+0000 7f9239400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T15:28:57.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.014+0000 7fd751600640 1 -- 172.21.15.47:0/1332389721 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v90) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd748017020 con 0x7fd7540b1280 2024-02-19T15:28:57.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.014+0000 7fd75c000640 1 -- 172.21.15.47:0/1332389721 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd734066e40 msgr2=0x7fd734069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:57.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.014+0000 7fd75c000640 1 --2- 172.21.15.47:0/1332389721 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd734066e40 0x7fd734069300 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fd75409d4f0 tx=0x7fd740008040 comp rx=0 tx=0).stop 2024-02-19T15:28:57.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.014+0000 7fd75c000640 1 -- 172.21.15.47:0/1332389721 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7540b1280 msgr2=0x7fd754154f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:57.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.014+0000 7fd75c000640 1 --2- 172.21.15.47:0/1332389721 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7540b1280 0x7fd754154f20 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fd74800c450 tx=0x7fd74800c910 comp rx=0 tx=0).stop 2024-02-19T15:28:57.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.014+0000 7fd75c000640 1 -- 172.21.15.47:0/1332389721 shutdown_connections 2024-02-19T15:28:57.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.014+0000 7fd75c000640 1 --2- 172.21.15.47:0/1332389721 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd734066e40 0x7fd734069300 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:57.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.014+0000 7fd75c000640 1 --2- 172.21.15.47:0/1332389721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75415e2b0 0x7fd75415c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:57.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.014+0000 7fd75c000640 1 --2- 172.21.15.47:0/1332389721 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd75415ded0 0x7fd75415be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:57.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.014+0000 7fd75c000640 1 --2- 172.21.15.47:0/1332389721 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd7540b1280 0x7fd754154f20 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:57.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.014+0000 7fd75c000640 1 -- 172.21.15.47:0/1332389721 >> 172.21.15.47:0/1332389721 conn(0x7fd7540b32f0 msgr2=0x7fd754156be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:57.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.014+0000 7fd75c000640 1 -- 172.21.15.47:0/1332389721 shutdown_connections 2024-02-19T15:28:57.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.018+0000 7fd75c000640 1 -- 172.21.15.47:0/1332389721 wait complete. 2024-02-19T15:28:57.037 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T15:28:57.037 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T15:28:57.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.182+0000 7f0b7e000640 1 Processor -- start 2024-02-19T15:28:57.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.182+0000 7f0b7e000640 1 -- start start 2024-02-19T15:28:57.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.182+0000 7f0b7e000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0b780ce3b0 0x7f0b780ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:57.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.182+0000 7f0b7e000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b780cc620 con 0x7f0b7815e280 2024-02-19T15:28:57.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.182+0000 7f0b7e000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b780d12e0 con 0x7f0b780d2650 2024-02-19T15:28:57.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.182+0000 7f0b7e000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b780d1940 con 0x7f0b780ce3b0 2024-02-19T15:28:57.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.182+0000 7f0b77e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0b7815e280 0x7f0b780cd1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41134/0 (socket says 172.21.15.47:41134) 2024-02-19T15:28:57.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.182+0000 7f0b77e00640 1 -- 172.21.15.47:0/1216119664 learned_addr learned my addr 172.21.15.47:0/1216119664 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:57.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.182+0000 7f0b77400640 1 --2- 172.21.15.47:0/1216119664 >> v2:172.21.15.116:3300/0 conn(0x7f0b780ce3b0 0x7f0b780ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:57.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.182+0000 7f0b76a00640 1 -- 172.21.15.47:0/1216119664 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490129808 0 0) 0x7f0b780cc620 con 0x7f0b7815e280 2024-02-19T15:28:57.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.182+0000 7f0b76a00640 1 -- 172.21.15.47:0/1216119664 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b5c003530 con 0x7f0b7815e280 2024-02-19T15:28:57.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.182+0000 7f0b76a00640 1 -- 172.21.15.47:0/1216119664 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254395979 0 0) 0x7f0b780d12e0 con 0x7f0b780d2650 2024-02-19T15:28:57.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.182+0000 7f0b77400640 1 -- 172.21.15.47:0/1216119664 >> v1:172.21.15.116:6789/0 conn(0x7f0b780d2650 legacy=0x7f0b780ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:57.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.182+0000 7f0b77400640 1 -- 172.21.15.47:0/1216119664 >> v1:172.21.15.47:6789/0 conn(0x7f0b7815e280 legacy=0x7f0b780cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:57.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.182+0000 7f0b77400640 1 -- 172.21.15.47:0/1216119664 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b780cecd0 con 0x7f0b780ce3b0 2024-02-19T15:28:57.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.186+0000 7f0b77400640 1 --2- 172.21.15.47:0/1216119664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b780ce3b0 0x7f0b780ce790 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f0b680097c0 tx=0x7f0b6801c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7a46c81991161ad server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:57.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.186+0000 7f0b76a00640 1 -- 172.21.15.47:0/1216119664 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b680ab600 con 0x7f0b780ce3b0 2024-02-19T15:28:57.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.186+0000 7f0b76a00640 1 -- 172.21.15.47:0/1216119664 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b680a7050 con 0x7f0b780ce3b0 2024-02-19T15:28:57.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.186+0000 7f0b76a00640 1 -- 172.21.15.47:0/1216119664 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b680ab960 con 0x7f0b780ce3b0 2024-02-19T15:28:57.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.186+0000 7f0b7e000640 1 -- 172.21.15.47:0/1216119664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b780ce3b0 msgr2=0x7f0b780ce790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:57.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.186+0000 7f0b7e000640 1 --2- 172.21.15.47:0/1216119664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b780ce3b0 0x7f0b780ce790 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f0b680097c0 tx=0x7f0b6801c380 comp rx=0 tx=0).stop 2024-02-19T15:28:57.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.186+0000 7f0b7e000640 1 -- 172.21.15.47:0/1216119664 shutdown_connections 2024-02-19T15:28:57.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.186+0000 7f0b7e000640 1 --2- 172.21.15.47:0/1216119664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b780ce3b0 0x7f0b780ce790 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:57.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.186+0000 7f0b7e000640 1 -- 172.21.15.47:0/1216119664 >> 172.21.15.47:0/1216119664 conn(0x7f0b780d1b30 msgr2=0x7f0b780d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:57.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.186+0000 7f0b7e000640 1 -- 172.21.15.47:0/1216119664 shutdown_connections 2024-02-19T15:28:57.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.186+0000 7f0b7e000640 1 -- 172.21.15.47:0/1216119664 wait complete. 2024-02-19T15:28:57.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.190+0000 7f0b7e000640 1 Processor -- start 2024-02-19T15:28:57.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.190+0000 7f0b7e000640 1 -- start start 2024-02-19T15:28:57.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.190+0000 7f0b7e000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b780ce3b0 0x7f0b78149bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:57.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.190+0000 7f0b7e000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b780d2650 0x7f0b781476b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:57.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.190+0000 7f0b7e000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b7815e280 0x7f0b78154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:57.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.190+0000 7f0b7e000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7814bad0 con 0x7f0b780d2650 2024-02-19T15:28:57.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.190+0000 7f0b7e000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b78158c20 con 0x7f0b7815e280 2024-02-19T15:28:57.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.190+0000 7f0b7e000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7814b1a0 con 0x7f0b780ce3b0 2024-02-19T15:28:57.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.190+0000 7f0b6fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b780d2650 0x7f0b781476b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:57.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.190+0000 7f0b6fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b780d2650 0x7f0b781476b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51064/0 (socket says 172.21.15.47:51064) 2024-02-19T15:28:57.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.190+0000 7f0b6fe00640 1 -- 172.21.15.47:0/71349952 learned_addr learned my addr 172.21.15.47:0/71349952 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:57.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.190+0000 7f0b77400640 1 --2- 172.21.15.47:0/71349952 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b780ce3b0 0x7f0b78149bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:57.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.190+0000 7f0b77e00640 1 --2- 172.21.15.47:0/71349952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b7815e280 0x7f0b78154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:57.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.194+0000 7f0b6fe00640 1 -- 172.21.15.47:0/71349952 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b780ce3b0 msgr2=0x7f0b78149bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:57.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.194+0000 7f0b6fe00640 1 --2- 172.21.15.47:0/71349952 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b780ce3b0 0x7f0b78149bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:57.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.194+0000 7f0b6fe00640 1 -- 172.21.15.47:0/71349952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b7815e280 msgr2=0x7f0b78154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:57.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.194+0000 7f0b6fe00640 1 --2- 172.21.15.47:0/71349952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b7815e280 0x7f0b78154ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:57.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.194+0000 7f0b6fe00640 1 -- 172.21.15.47:0/71349952 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b68009420 con 0x7f0b780d2650 2024-02-19T15:28:57.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.194+0000 7f0b6fe00640 1 --2- 172.21.15.47:0/71349952 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b780d2650 0x7f0b781476b0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f0b6000c410 tx=0x7f0b6000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:57.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.194+0000 7f0b6f400640 1 -- 172.21.15.47:0/71349952 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b600054e0 con 0x7f0b780d2650 2024-02-19T15:28:57.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.194+0000 7f0b6f400640 1 -- 172.21.15.47:0/71349952 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b60003600 con 0x7f0b780d2650 2024-02-19T15:28:57.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.194+0000 7f0b6f400640 1 -- 172.21.15.47:0/71349952 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b600039b0 con 0x7f0b780d2650 2024-02-19T15:28:57.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.194+0000 7f0b7e000640 1 -- 172.21.15.47:0/71349952 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7814b480 con 0x7f0b780d2650 2024-02-19T15:28:57.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.194+0000 7f0b7e000640 1 -- 172.21.15.47:0/71349952 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b78147bf0 con 0x7f0b780d2650 2024-02-19T15:28:57.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.194+0000 7f0b7e000640 1 -- 172.21.15.47:0/71349952 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7814bc40 con 0x7f0b780d2650 2024-02-19T15:28:57.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.194+0000 7f0b6f400640 1 -- 172.21.15.47:0/71349952 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0b60003b10 con 0x7f0b780d2650 2024-02-19T15:28:57.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.194+0000 7f0b6f400640 1 --2- 172.21.15.47:0/71349952 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0b54066e40 0x7f0b54069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:57.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.194+0000 7f0b6f400640 1 -- 172.21.15.47:0/71349952 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b60083ff0 con 0x7f0b780d2650 2024-02-19T15:28:57.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.198+0000 7f0b77400640 1 --2- 172.21.15.47:0/71349952 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0b54066e40 0x7f0b54069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:57.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.198+0000 7f0b77400640 1 --2- 172.21.15.47:0/71349952 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0b54066e40 0x7f0b54069300 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f0b680ad000 tx=0x7f0b68008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:57.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.202+0000 7f0b6f400640 1 -- 172.21.15.47:0/71349952 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0b600b61c0 con 0x7f0b780d2650 2024-02-19T15:28:57.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:57.398+0000 7f0b7e000640 1 -- 172.21.15.47:0/71349952 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f0b78075360 con 0x7f0b780d2650 2024-02-19T15:28:58.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.018+0000 7f0b6f400640 1 -- 172.21.15.47:0/71349952 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v91) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0b60050e30 con 0x7f0b780d2650 2024-02-19T15:28:58.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.022+0000 7f0b7e000640 1 -- 172.21.15.47:0/71349952 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0b54066e40 msgr2=0x7f0b54069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:58.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.022+0000 7f0b7e000640 1 --2- 172.21.15.47:0/71349952 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0b54066e40 0x7f0b54069300 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f0b680ad000 tx=0x7f0b68008040 comp rx=0 tx=0).stop 2024-02-19T15:28:58.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.022+0000 7f0b7e000640 1 -- 172.21.15.47:0/71349952 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b780d2650 msgr2=0x7f0b781476b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:58.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.022+0000 7f0b7e000640 1 --2- 172.21.15.47:0/71349952 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b780d2650 0x7f0b781476b0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f0b6000c410 tx=0x7f0b6000c8d0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.022+0000 7f0b7e000640 1 -- 172.21.15.47:0/71349952 shutdown_connections 2024-02-19T15:28:58.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.022+0000 7f0b7e000640 1 --2- 172.21.15.47:0/71349952 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0b54066e40 0x7f0b54069300 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.022+0000 7f0b7e000640 1 --2- 172.21.15.47:0/71349952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b7815e280 0x7f0b78154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.022+0000 7f0b7e000640 1 --2- 172.21.15.47:0/71349952 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b780d2650 0x7f0b781476b0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.022+0000 7f0b7e000640 1 --2- 172.21.15.47:0/71349952 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b780ce3b0 0x7f0b78149bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.022+0000 7f0b7e000640 1 -- 172.21.15.47:0/71349952 >> 172.21.15.47:0/71349952 conn(0x7f0b780d1b30 msgr2=0x7f0b780c6150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:58.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.022+0000 7f0b7e000640 1 -- 172.21.15.47:0/71349952 shutdown_connections 2024-02-19T15:28:58.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.022+0000 7f0b7e000640 1 -- 172.21.15.47:0/71349952 wait complete. 2024-02-19T15:28:58.041 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T15:28:58.041 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:28:58.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.214+0000 7f019d000640 1 Processor -- start 2024-02-19T15:28:58.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.214+0000 7f019d000640 1 -- start start 2024-02-19T15:28:58.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.214+0000 7f019d000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f019806d650 0x7f01980d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.214+0000 7f019d000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01980ced10 0x7f019806bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.214+0000 7f019d000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01980cce30 con 0x7f01980cded0 2024-02-19T15:28:58.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.214+0000 7f019d000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01980cd110 con 0x7f019806d650 2024-02-19T15:28:58.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.214+0000 7f019d000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01980cd3f0 con 0x7f01980ced10 2024-02-19T15:28:58.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.214+0000 7f018fe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f01980cded0 0x7f01980ce600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41138/0 (socket says 172.21.15.47:41138) 2024-02-19T15:28:58.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.214+0000 7f018fe00640 1 -- 172.21.15.47:0/472924681 learned_addr learned my addr 172.21.15.47:0/472924681 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:58.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.214+0000 7f0195e00640 1 --2- 172.21.15.47:0/472924681 >> v2:172.21.15.116:3300/0 conn(0x7f019806d650 0x7f01980d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.214+0000 7f0196800640 1 --2- 172.21.15.47:0/472924681 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01980ced10 0x7f019806bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.214+0000 7f0195400640 1 -- 172.21.15.47:0/472924681 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936799606 0 0) 0x7f01980cce30 con 0x7f01980cded0 2024-02-19T15:28:58.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.214+0000 7f0195400640 1 -- 172.21.15.47:0/472924681 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f017c003530 con 0x7f01980cded0 2024-02-19T15:28:58.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0195400640 1 -- 172.21.15.47:0/472924681 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1859464124 0 0) 0x7f017c003530 con 0x7f01980cded0 2024-02-19T15:28:58.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0195400640 1 -- 172.21.15.47:0/472924681 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01980ced10 msgr2=0x7f019806bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:58.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0195400640 1 --2- 172.21.15.47:0/472924681 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01980ced10 0x7f019806bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0195400640 1 -- 172.21.15.47:0/472924681 >> v2:172.21.15.116:3300/0 conn(0x7f019806d650 msgr2=0x7f01980d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:58.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0195400640 1 --2- 172.21.15.47:0/472924681 >> v2:172.21.15.116:3300/0 conn(0x7f019806d650 0x7f01980d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0195400640 1 -- 172.21.15.47:0/472924681 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01980cd6d0 con 0x7f01980cded0 2024-02-19T15:28:58.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0195400640 1 -- 172.21.15.47:0/472924681 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f0180001680 con 0x7f01980cded0 2024-02-19T15:28:58.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0195400640 1 -- 172.21.15.47:0/472924681 >> v1:172.21.15.47:6789/0 conn(0x7f01980cded0 legacy=0x7f01980ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:58.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0195400640 1 --2- 172.21.15.47:0/472924681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f017c003980 0x7f017c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0195400640 1 --2- 172.21.15.47:0/472924681 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01980ced10 0x7f017c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0195400640 1 --2- 172.21.15.47:0/472924681 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f017c004d10 0x7f017c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0195400640 1 -- 172.21.15.47:0/472924681 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01980cd110 con 0x7f017c004d10 2024-02-19T15:28:58.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0195400640 1 -- 172.21.15.47:0/472924681 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01980cd3f0 con 0x7f017c003980 2024-02-19T15:28:58.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0195400640 1 -- 172.21.15.47:0/472924681 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017c00ba40 con 0x7f01980ced10 2024-02-19T15:28:58.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0195e00640 1 --2- 172.21.15.47:0/472924681 >> v2:172.21.15.116:3300/0 conn(0x7f019806d650 0x7f01980d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:28:58.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0196800640 1 --2- 172.21.15.47:0/472924681 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f017c004d10 0x7f017c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f018fe00640 1 --2- 172.21.15.47:0/472924681 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01980ced10 0x7f017c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0195e00640 1 --2- 172.21.15.47:0/472924681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f017c003980 0x7f017c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0196800640 1 -- 172.21.15.47:0/472924681 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01980ced10 msgr2=0x7f017c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:58.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0196800640 1 --2- 172.21.15.47:0/472924681 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01980ced10 0x7f017c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0196800640 1 -- 172.21.15.47:0/472924681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f017c003980 msgr2=0x7f017c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:58.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0196800640 1 --2- 172.21.15.47:0/472924681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f017c003980 0x7f017c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0196800640 1 -- 172.21.15.47:0/472924681 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017c003530 con 0x7f017c004d10 2024-02-19T15:28:58.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.218+0000 7f0196800640 1 --2- 172.21.15.47:0/472924681 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f017c004d10 0x7f017c005190 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f0190009fc0 tx=0x7f019001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=461e3693baaa8fa8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:58.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f0195400640 1 -- 172.21.15.47:0/472924681 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01900a5c90 con 0x7f017c004d10 2024-02-19T15:28:58.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f0195400640 1 -- 172.21.15.47:0/472924681 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019000f040 con 0x7f017c004d10 2024-02-19T15:28:58.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f0195400640 1 -- 172.21.15.47:0/472924681 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01900ad680 con 0x7f017c004d10 2024-02-19T15:28:58.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f019d000640 1 -- 172.21.15.47:0/472924681 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f017c004d10 msgr2=0x7f017c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:58.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f019d000640 1 --2- 172.21.15.47:0/472924681 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f017c004d10 0x7f017c005190 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f0190009fc0 tx=0x7f019001cc80 comp rx=0 tx=0).stop 2024-02-19T15:28:58.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f0196800640 1 -- 172.21.15.47:0/472924681 reap_dead start 2024-02-19T15:28:58.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f019d000640 1 -- 172.21.15.47:0/472924681 shutdown_connections 2024-02-19T15:28:58.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f019d000640 1 -- 172.21.15.47:0/472924681 >> 172.21.15.47:0/472924681 conn(0x7f01980d1bf0 msgr2=0x7f01980d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:58.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f019d000640 1 -- 172.21.15.47:0/472924681 shutdown_connections 2024-02-19T15:28:58.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f019d000640 1 -- 172.21.15.47:0/472924681 wait complete. 2024-02-19T15:28:58.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f019d000640 1 Processor -- start 2024-02-19T15:28:58.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f019d000640 1 -- start start 2024-02-19T15:28:58.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f019d000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f017c004d10 0x7f01980b72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f019d000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0198156c40 0x7f01981570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f019d000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01980c7250 0x7f01980c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f019d000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0198157600 con 0x7f017c004d10 2024-02-19T15:28:58.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f019d000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0198159b50 con 0x7f01980c7250 2024-02-19T15:28:58.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f019d000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0198159e30 con 0x7f0198156c40 2024-02-19T15:28:58.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f0195e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f017c004d10 0x7f01980b72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f0195e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f017c004d10 0x7f01980b72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51094/0 (socket says 172.21.15.47:51094) 2024-02-19T15:28:58.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.222+0000 7f0195e00640 1 -- 172.21.15.47:0/2492241557 learned_addr learned my addr 172.21.15.47:0/2492241557 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:58.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.226+0000 7f0196800640 1 --2- 172.21.15.47:0/2492241557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01980c7250 0x7f01980c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.226+0000 7f0195e00640 1 -- 172.21.15.47:0/2492241557 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0198156c40 msgr2=0x7f01981570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:58.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.226+0000 7f018fe00640 1 --2- 172.21.15.47:0/2492241557 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0198156c40 0x7f01981570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.226+0000 7f0195e00640 1 --2- 172.21.15.47:0/2492241557 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0198156c40 0x7f01981570c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.226+0000 7f0195e00640 1 -- 172.21.15.47:0/2492241557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01980c7250 msgr2=0x7f01980c7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:58.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.226+0000 7f0195e00640 1 --2- 172.21.15.47:0/2492241557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01980c7250 0x7f01980c7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.226+0000 7f0195e00640 1 -- 172.21.15.47:0/2492241557 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019815a110 con 0x7f017c004d10 2024-02-19T15:28:58.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.226+0000 7f0196800640 1 --2- 172.21.15.47:0/2492241557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01980c7250 0x7f01980c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:58.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.226+0000 7f018fe00640 1 --2- 172.21.15.47:0/2492241557 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0198156c40 0x7f01981570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:28:58.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.226+0000 7f0195e00640 1 --2- 172.21.15.47:0/2492241557 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f017c004d10 0x7f01980b72d0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f0188011a00 tx=0x7f0188011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:58.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.226+0000 7f018e000640 1 -- 172.21.15.47:0/2492241557 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018800fc10 con 0x7f017c004d10 2024-02-19T15:28:58.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.226+0000 7f018e000640 1 -- 172.21.15.47:0/2492241557 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f018800fd70 con 0x7f017c004d10 2024-02-19T15:28:58.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.226+0000 7f018e000640 1 -- 172.21.15.47:0/2492241557 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018800fee0 con 0x7f017c004d10 2024-02-19T15:28:58.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.226+0000 7f019d000640 1 -- 172.21.15.47:0/2492241557 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f019815a3f0 con 0x7f017c004d10 2024-02-19T15:28:58.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.226+0000 7f019d000640 1 -- 172.21.15.47:0/2492241557 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01980c7c40 con 0x7f017c004d10 2024-02-19T15:28:58.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.226+0000 7f019d000640 1 -- 172.21.15.47:0/2492241557 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01980ccaa0 con 0x7f017c004d10 2024-02-19T15:28:58.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.226+0000 7f018e000640 1 -- 172.21.15.47:0/2492241557 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0188004260 con 0x7f017c004d10 2024-02-19T15:28:58.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.226+0000 7f018e000640 1 --2- 172.21.15.47:0/2492241557 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0174066e40 0x7f0174069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.226+0000 7f018e000640 1 -- 172.21.15.47:0/2492241557 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01880851d0 con 0x7f017c004d10 2024-02-19T15:28:58.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.230+0000 7f018fe00640 1 --2- 172.21.15.47:0/2492241557 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0174066e40 0x7f0174069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.230+0000 7f018fe00640 1 --2- 172.21.15.47:0/2492241557 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0174066e40 0x7f0174069300 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f0180001060 tx=0x7f0180001de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:58.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.230+0000 7f018e000640 1 -- 172.21.15.47:0/2492241557 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0188051f20 con 0x7f017c004d10 2024-02-19T15:28:58.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.430+0000 7f019d000640 1 -- 172.21.15.47:0/2492241557 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01980d0da0 con 0x7f017c004d10 2024-02-19T15:28:58.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.430+0000 7f018e000640 1 -- 172.21.15.47:0/2492241557 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+4943 (secure 0 0 0) 0x7f01880555e0 con 0x7f017c004d10 2024-02-19T15:28:58.430 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:58.431 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":91,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7028,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/809721153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":809721153},{"type":"v1","addr":"172.21.15.47:6834","nonce":809721153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7034,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1553163168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1553163168},{"type":"v1","addr":"172.21.15.116:6836","nonce":1553163168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":7037,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:28:57.400659+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8361},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:28:58.431 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 91 2024-02-19T15:28:58.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.430+0000 7f019d000640 1 -- 172.21.15.47:0/2492241557 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0174066e40 msgr2=0x7f0174069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:58.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.434+0000 7f019d000640 1 --2- 172.21.15.47:0/2492241557 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0174066e40 0x7f0174069300 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f0180001060 tx=0x7f0180001de0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.434+0000 7f019d000640 1 -- 172.21.15.47:0/2492241557 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f017c004d10 msgr2=0x7f01980b72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:58.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.434+0000 7f019d000640 1 --2- 172.21.15.47:0/2492241557 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f017c004d10 0x7f01980b72d0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f0188011a00 tx=0x7f0188011ec0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.434+0000 7f019d000640 1 -- 172.21.15.47:0/2492241557 shutdown_connections 2024-02-19T15:28:58.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.434+0000 7f019d000640 1 --2- 172.21.15.47:0/2492241557 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0174066e40 0x7f0174069300 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.434+0000 7f019d000640 1 --2- 172.21.15.47:0/2492241557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01980c7250 0x7f01980c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.434+0000 7f019d000640 1 --2- 172.21.15.47:0/2492241557 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0198156c40 0x7f01981570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.434+0000 7f019d000640 1 --2- 172.21.15.47:0/2492241557 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f017c004d10 0x7f01980b72d0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.434+0000 7f019d000640 1 -- 172.21.15.47:0/2492241557 >> 172.21.15.47:0/2492241557 conn(0x7f01980d1bf0 msgr2=0x7f01980d05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:58.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.434+0000 7f019d000640 1 -- 172.21.15.47:0/2492241557 shutdown_connections 2024-02-19T15:28:58.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.434+0000 7f019d000640 1 -- 172.21.15.47:0/2492241557 wait complete. 2024-02-19T15:28:58.452 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:28:58.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.590+0000 7f250e400640 1 Processor -- start 2024-02-19T15:28:58.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.590+0000 7f250e400640 1 -- start start 2024-02-19T15:28:58.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.590+0000 7f250e400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25080cb520 0x7f25080cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.590+0000 7f250e400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25080c6780 0x7f25080c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.590+0000 7f250e400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25080ad840 con 0x7f250806aeb0 2024-02-19T15:28:58.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.590+0000 7f250e400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25080c62f0 con 0x7f25080cb520 2024-02-19T15:28:58.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.590+0000 7f250e400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25080cc810 con 0x7f25080c6780 2024-02-19T15:28:58.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.594+0000 7f2507400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f250806aeb0 0x7f250806b290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41148/0 (socket says 172.21.15.47:41148) 2024-02-19T15:28:58.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.594+0000 7f2507400640 1 -- 172.21.15.47:0/758523400 learned_addr learned my addr 172.21.15.47:0/758523400 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:58.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.594+0000 7f2507e00640 1 --2- 172.21.15.47:0/758523400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25080c6780 0x7f25080c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.594+0000 7f2506a00640 1 --2- 172.21.15.47:0/758523400 >> v2:172.21.15.116:3300/0 conn(0x7f25080cb520 0x7f25080cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.594+0000 7f2506000640 1 -- 172.21.15.47:0/758523400 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724917243 0 0) 0x7f25080ad840 con 0x7f250806aeb0 2024-02-19T15:28:58.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.594+0000 7f2506000640 1 -- 172.21.15.47:0/758523400 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24ec003530 con 0x7f250806aeb0 2024-02-19T15:28:58.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.594+0000 7f2507e00640 1 -- 172.21.15.47:0/758523400 >> v2:172.21.15.116:3300/0 conn(0x7f25080cb520 msgr2=0x7f25080cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:58.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.594+0000 7f2507e00640 1 --2- 172.21.15.47:0/758523400 >> v2:172.21.15.116:3300/0 conn(0x7f25080cb520 0x7f25080cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.594+0000 7f2507e00640 1 -- 172.21.15.47:0/758523400 >> v1:172.21.15.47:6789/0 conn(0x7f250806aeb0 legacy=0x7f250806b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:58.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.594+0000 7f2507e00640 1 -- 172.21.15.47:0/758523400 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25080cc250 con 0x7f25080c6780 2024-02-19T15:28:58.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.594+0000 7f2507e00640 1 --2- 172.21.15.47:0/758523400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25080c6780 0x7f25080c6c10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2500009d90 tx=0x7f250001c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b657c11f9d33b9d5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:58.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.594+0000 7f2506000640 1 -- 172.21.15.47:0/758523400 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25080c7960 con 0x7f25080c6780 2024-02-19T15:28:58.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f2506000640 1 -- 172.21.15.47:0/758523400 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25000a5a00 con 0x7f25080c6780 2024-02-19T15:28:58.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f2506000640 1 -- 172.21.15.47:0/758523400 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25000a5d90 con 0x7f25080c6780 2024-02-19T15:28:58.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f250e400640 1 -- 172.21.15.47:0/758523400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25080c6780 msgr2=0x7f25080c6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:58.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f250e400640 1 --2- 172.21.15.47:0/758523400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25080c6780 0x7f25080c6c10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2500009d90 tx=0x7f250001c940 comp rx=0 tx=0).stop 2024-02-19T15:28:58.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f250e400640 1 -- 172.21.15.47:0/758523400 shutdown_connections 2024-02-19T15:28:58.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f250e400640 1 --2- 172.21.15.47:0/758523400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25080c6780 0x7f25080c6c10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f250e400640 1 --2- 172.21.15.47:0/758523400 >> v2:172.21.15.116:3300/0 conn(0x7f25080cb520 0x7f25080cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f250e400640 1 -- 172.21.15.47:0/758523400 >> 172.21.15.47:0/758523400 conn(0x7f25080d0ad0 msgr2=0x7f25080d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:58.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f250e400640 1 -- 172.21.15.47:0/758523400 shutdown_connections 2024-02-19T15:28:58.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f250e400640 1 -- 172.21.15.47:0/758523400 wait complete. 2024-02-19T15:28:58.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f250e400640 1 Processor -- start 2024-02-19T15:28:58.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f250e400640 1 -- start start 2024-02-19T15:28:58.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f250e400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f250806aeb0 0x7f25081474e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f250e400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25080c6780 0x7f250815f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f250e400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25080cb520 0x7f250815f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f250e400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2508147a20 con 0x7f25080cb520 2024-02-19T15:28:58.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f250e400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2508157b30 con 0x7f250806aeb0 2024-02-19T15:28:58.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f250e400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25081610a0 con 0x7f25080c6780 2024-02-19T15:28:58.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f2507e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25080cb520 0x7f250815f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f2507e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25080cb520 0x7f250815f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51108/0 (socket says 172.21.15.47:51108) 2024-02-19T15:28:58.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f2507e00640 1 -- 172.21.15.47:0/1680741177 learned_addr learned my addr 172.21.15.47:0/1680741177 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:58.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f2506a00640 1 --2- 172.21.15.47:0/1680741177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25080c6780 0x7f250815f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f2507400640 1 --2- 172.21.15.47:0/1680741177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f250806aeb0 0x7f25081474e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f2507e00640 1 -- 172.21.15.47:0/1680741177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25080c6780 msgr2=0x7f250815f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:58.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f2507e00640 1 --2- 172.21.15.47:0/1680741177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25080c6780 0x7f250815f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f2507e00640 1 -- 172.21.15.47:0/1680741177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f250806aeb0 msgr2=0x7f25081474e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:58.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f2507e00640 1 --2- 172.21.15.47:0/1680741177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f250806aeb0 0x7f25081474e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.598+0000 7f2507e00640 1 -- 172.21.15.47:0/1680741177 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2500009a70 con 0x7f25080cb520 2024-02-19T15:28:58.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.602+0000 7f2507e00640 1 --2- 172.21.15.47:0/1680741177 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25080cb520 0x7f250815f540 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f250001aff0 tx=0x7f25000196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:58.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.602+0000 7f24ff400640 1 -- 172.21.15.47:0/1680741177 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25000b9860 con 0x7f25080cb520 2024-02-19T15:28:58.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.602+0000 7f24ff400640 1 -- 172.21.15.47:0/1680741177 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25000b99c0 con 0x7f25080cb520 2024-02-19T15:28:58.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.602+0000 7f250e400640 1 -- 172.21.15.47:0/1680741177 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2508155fd0 con 0x7f25080cb520 2024-02-19T15:28:58.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.602+0000 7f24ff400640 1 -- 172.21.15.47:0/1680741177 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25000b9d50 con 0x7f25080cb520 2024-02-19T15:28:58.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.602+0000 7f250e400640 1 -- 172.21.15.47:0/1680741177 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2508156590 con 0x7f25080cb520 2024-02-19T15:28:58.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.602+0000 7f250e400640 1 -- 172.21.15.47:0/1680741177 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2508156780 con 0x7f25080cb520 2024-02-19T15:28:58.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.602+0000 7f24ff400640 1 -- 172.21.15.47:0/1680741177 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f25000ad440 con 0x7f25080cb520 2024-02-19T15:28:58.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.602+0000 7f24ff400640 1 --2- 172.21.15.47:0/1680741177 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f24e8066e40 0x7f24e8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.602+0000 7f24ff400640 1 -- 172.21.15.47:0/1680741177 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2500120080 con 0x7f25080cb520 2024-02-19T15:28:58.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.602+0000 7f2507400640 1 --2- 172.21.15.47:0/1680741177 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f24e8066e40 0x7f24e8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.606+0000 7f2507400640 1 --2- 172.21.15.47:0/1680741177 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f24e8066e40 0x7f24e8069300 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f24f8007fd0 tx=0x7f24f8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:58.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.606+0000 7f24ff400640 1 -- 172.21.15.47:0/1680741177 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f25000ed090 con 0x7f25080cb520 2024-02-19T15:28:58.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.770+0000 7f250e400640 1 -- 172.21.15.47:0/1680741177 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2508075360 con 0x7f25080cb520 2024-02-19T15:28:58.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.770+0000 7f24ff400640 1 -- 172.21.15.47:0/1680741177 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+16691 (secure 0 0 0) 0x7f25000f0750 con 0x7f25080cb520 2024-02-19T15:28:58.770 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:58.770 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":125,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:28:55.992738+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:28:52.814262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-02-19T15:28:53.381287+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:28:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.770+0000 7f250e400640 1 -- 172.21.15.47:0/1680741177 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f24e8066e40 msgr2=0x7f24e8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.770+0000 7f250e400640 1 --2- 172.21.15.47:0/1680741177 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f24e8066e40 0x7f24e8069300 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f24f8007fd0 tx=0x7f24f8009040 comp rx=0 tx=0).stop 2024-02-19T15:28:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.770+0000 7f250e400640 1 -- 172.21.15.47:0/1680741177 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25080cb520 msgr2=0x7f250815f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.770+0000 7f250e400640 1 --2- 172.21.15.47:0/1680741177 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25080cb520 0x7f250815f540 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f250001aff0 tx=0x7f25000196e0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.770+0000 7f250e400640 1 -- 172.21.15.47:0/1680741177 shutdown_connections 2024-02-19T15:28:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.770+0000 7f250e400640 1 --2- 172.21.15.47:0/1680741177 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f24e8066e40 0x7f24e8069300 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.770+0000 7f250e400640 1 --2- 172.21.15.47:0/1680741177 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25080cb520 0x7f250815f540 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.770+0000 7f250e400640 1 --2- 172.21.15.47:0/1680741177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25080c6780 0x7f250815f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.770+0000 7f250e400640 1 --2- 172.21.15.47:0/1680741177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f250806aeb0 0x7f25081474e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.770+0000 7f250e400640 1 -- 172.21.15.47:0/1680741177 >> 172.21.15.47:0/1680741177 conn(0x7f25080d0ad0 msgr2=0x7f25080b0590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:58.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.770+0000 7f250e400640 1 -- 172.21.15.47:0/1680741177 shutdown_connections 2024-02-19T15:28:58.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.770+0000 7f250e400640 1 -- 172.21.15.47:0/1680741177 wait complete. 2024-02-19T15:28:58.792 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T15:28:58.792 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:28:58.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.934+0000 7f6ff7800640 1 Processor -- start 2024-02-19T15:28:58.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.934+0000 7f6ff7800640 1 -- start start 2024-02-19T15:28:58.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.934+0000 7f6ff7800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6ff015f620 0x7f6ff00b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.934+0000 7f6ff7800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ff00b2330 0x7f6ff006ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.934+0000 7f6ff7800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff006b5d0 con 0x7f6ff01614e0 2024-02-19T15:28:58.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.934+0000 7f6ff7800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ff00b1c00 con 0x7f6ff015f620 2024-02-19T15:28:58.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.934+0000 7f6ff7800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff00b27e0 con 0x7f6ff00b2330 2024-02-19T15:28:58.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.934+0000 7f6fefe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6ff015f620 0x7f6ff00b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.934+0000 7f6fefe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6ff015f620 0x7f6ff00b1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51118/0 (socket says 172.21.15.47:51118) 2024-02-19T15:28:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.934+0000 7f6fefe00640 1 -- 172.21.15.47:0/2646800749 learned_addr learned my addr 172.21.15.47:0/2646800749 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.934+0000 7f6ff5800640 1 --2- 172.21.15.47:0/2646800749 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ff00b2330 0x7f6ff006ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.934+0000 7f6fefe00640 1 -- 172.21.15.47:0/2646800749 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ff00b2330 msgr2=0x7f6ff006ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.934+0000 7f6fefe00640 1 --2- 172.21.15.47:0/2646800749 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ff00b2330 0x7f6ff006ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.934+0000 7f6fefe00640 1 -- 172.21.15.47:0/2646800749 >> v1:172.21.15.47:6789/0 conn(0x7f6ff01614e0 legacy=0x7f6ff015eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.934+0000 7f6fefe00640 1 -- 172.21.15.47:0/2646800749 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff00b1640 con 0x7f6ff015f620 2024-02-19T15:28:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.934+0000 7f6ff5800640 1 --2- 172.21.15.47:0/2646800749 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ff00b2330 0x7f6ff006ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:28:58.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.938+0000 7f6fefe00640 1 --2- 172.21.15.47:0/2646800749 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ff015f620 0x7f6ff00b1df0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009870 tx=0x7f6fdc01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=167dc2c2db046e17 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:58.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.938+0000 7f6fef400640 1 -- 172.21.15.47:0/2646800749 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc0a9620 con 0x7f6ff015f620 2024-02-19T15:28:58.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.938+0000 7f6fef400640 1 -- 172.21.15.47:0/2646800749 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fdc0a9780 con 0x7f6ff015f620 2024-02-19T15:28:58.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.938+0000 7f6fef400640 1 -- 172.21.15.47:0/2646800749 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc0a9b10 con 0x7f6ff015f620 2024-02-19T15:28:58.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff7800640 1 -- 172.21.15.47:0/2646800749 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ff015f620 msgr2=0x7f6ff00b1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:58.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff7800640 1 --2- 172.21.15.47:0/2646800749 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ff015f620 0x7f6ff00b1df0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009870 tx=0x7f6fdc01c560 comp rx=0 tx=0).stop 2024-02-19T15:28:58.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff7800640 1 -- 172.21.15.47:0/2646800749 shutdown_connections 2024-02-19T15:28:58.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff7800640 1 --2- 172.21.15.47:0/2646800749 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ff00b2330 0x7f6ff006ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff7800640 1 --2- 172.21.15.47:0/2646800749 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ff015f620 0x7f6ff00b1df0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff7800640 1 -- 172.21.15.47:0/2646800749 >> 172.21.15.47:0/2646800749 conn(0x7f6ff00b4fc0 msgr2=0x7f6ff00b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:58.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff7800640 1 -- 172.21.15.47:0/2646800749 shutdown_connections 2024-02-19T15:28:58.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff7800640 1 -- 172.21.15.47:0/2646800749 wait complete. 2024-02-19T15:28:58.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff7800640 1 Processor -- start 2024-02-19T15:28:58.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff7800640 1 -- start start 2024-02-19T15:28:58.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff7800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ff00b2330 0x7f6ff0160580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff7800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ff015f620 0x7f6ff01472e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff7800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ff01614e0 0x7f6ff0147820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff7800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff0147d60 con 0x7f6ff01614e0 2024-02-19T15:28:58.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff7800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff00ca5a0 con 0x7f6ff00b2330 2024-02-19T15:28:58.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff7800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff0158610 con 0x7f6ff015f620 2024-02-19T15:28:58.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff5800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ff01614e0 0x7f6ff0147820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff5800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ff01614e0 0x7f6ff0147820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51122/0 (socket says 172.21.15.47:51122) 2024-02-19T15:28:58.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff5800640 1 -- 172.21.15.47:0/334418535 learned_addr learned my addr 172.21.15.47:0/334418535 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:58.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6fefe00640 1 --2- 172.21.15.47:0/334418535 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ff015f620 0x7f6ff01472e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff4e00640 1 --2- 172.21.15.47:0/334418535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ff00b2330 0x7f6ff0160580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff5800640 1 -- 172.21.15.47:0/334418535 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ff015f620 msgr2=0x7f6ff01472e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:58.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff5800640 1 --2- 172.21.15.47:0/334418535 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ff015f620 0x7f6ff01472e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff5800640 1 -- 172.21.15.47:0/334418535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ff00b2330 msgr2=0x7f6ff0160580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:58.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff5800640 1 --2- 172.21.15.47:0/334418535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ff00b2330 0x7f6ff0160580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:58.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.942+0000 7f6ff5800640 1 -- 172.21.15.47:0/334418535 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc0094d0 con 0x7f6ff01614e0 2024-02-19T15:28:58.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.946+0000 7f6ff5800640 1 --2- 172.21.15.47:0/334418535 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ff01614e0 0x7f6ff0147820 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8007ef0 tx=0x7f6fe800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:58.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.946+0000 7f6fecc00640 1 -- 172.21.15.47:0/334418535 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8003ba0 con 0x7f6ff01614e0 2024-02-19T15:28:58.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.946+0000 7f6fecc00640 1 -- 172.21.15.47:0/334418535 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe800f040 con 0x7f6ff01614e0 2024-02-19T15:28:58.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.946+0000 7f6fecc00640 1 -- 172.21.15.47:0/334418535 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe800de80 con 0x7f6ff01614e0 2024-02-19T15:28:58.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.946+0000 7f6ff7800640 1 -- 172.21.15.47:0/334418535 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff01588f0 con 0x7f6ff01614e0 2024-02-19T15:28:58.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.946+0000 7f6ff7800640 1 -- 172.21.15.47:0/334418535 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff00ca860 con 0x7f6ff01614e0 2024-02-19T15:28:58.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.946+0000 7f6ff7800640 1 -- 172.21.15.47:0/334418535 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff00caa50 con 0x7f6ff01614e0 2024-02-19T15:28:58.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.946+0000 7f6fecc00640 1 -- 172.21.15.47:0/334418535 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6fe8004030 con 0x7f6ff01614e0 2024-02-19T15:28:58.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.946+0000 7f6fecc00640 1 --2- 172.21.15.47:0/334418535 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6fd0066e40 0x7f6fd0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:58.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.946+0000 7f6fecc00640 1 -- 172.21.15.47:0/334418535 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6fe8084d60 con 0x7f6ff01614e0 2024-02-19T15:28:58.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.946+0000 7f6ff4e00640 1 --2- 172.21.15.47:0/334418535 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6fd0066e40 0x7f6fd0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:58.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.950+0000 7f6ff4e00640 1 --2- 172.21.15.47:0/334418535 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6fd0066e40 0x7f6fd0069300 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40015b0 tx=0x7f6fe4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:58.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:58.950+0000 7f6fecc00640 1 -- 172.21.15.47:0/334418535 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6fe8051b60 con 0x7f6ff01614e0 2024-02-19T15:28:59.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.134+0000 7f6ff7800640 1 -- 172.21.15.47:0/334418535 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6ff0075360 con 0x7f6fd0066e40 2024-02-19T15:28:59.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.142+0000 7f6fecc00640 1 -- 172.21.15.47:0/334418535 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287384 (secure 0 0 0) 0x7f6ff0075360 con 0x7f6fd0066e40 2024-02-19T15:28:59.146 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:59.148 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:28:59.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.146+0000 7f6ff7800640 1 -- 172.21.15.47:0/334418535 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6fd0066e40 msgr2=0x7f6fd0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:59.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.146+0000 7f6ff7800640 1 --2- 172.21.15.47:0/334418535 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6fd0066e40 0x7f6fd0069300 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40015b0 tx=0x7f6fe4009040 comp rx=0 tx=0).stop 2024-02-19T15:28:59.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.146+0000 7f6ff7800640 1 -- 172.21.15.47:0/334418535 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ff01614e0 msgr2=0x7f6ff0147820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:59.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.146+0000 7f6ff7800640 1 --2- 172.21.15.47:0/334418535 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ff01614e0 0x7f6ff0147820 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8007ef0 tx=0x7f6fe800d4a0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.146+0000 7f6ff7800640 1 -- 172.21.15.47:0/334418535 shutdown_connections 2024-02-19T15:28:59.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.146+0000 7f6ff7800640 1 --2- 172.21.15.47:0/334418535 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6fd0066e40 0x7f6fd0069300 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.146+0000 7f6ff7800640 1 --2- 172.21.15.47:0/334418535 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ff01614e0 0x7f6ff0147820 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.146+0000 7f6ff7800640 1 --2- 172.21.15.47:0/334418535 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ff015f620 0x7f6ff01472e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.146+0000 7f6ff7800640 1 --2- 172.21.15.47:0/334418535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ff00b2330 0x7f6ff0160580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.146+0000 7f6ff7800640 1 -- 172.21.15.47:0/334418535 >> 172.21.15.47:0/334418535 conn(0x7f6ff00b4fc0 msgr2=0x7f6ff015fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:59.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.146+0000 7f6ff7800640 1 -- 172.21.15.47:0/334418535 shutdown_connections 2024-02-19T15:28:59.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.146+0000 7f6ff7800640 1 -- 172.21.15.47:0/334418535 wait complete. 2024-02-19T15:28:59.168 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-02-19T15:28:57.246287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591166,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":185,"num_write_kb":2427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":172,"acting":172,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":308488,"kb_used_data":13252,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754666232,"statfs":{"total":773094113280,"available":772778221568,"internally_reserved":0,"allocated":13570048,"data_stored":6541865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":302173746},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":798,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.263301"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997573+0000","last_change":"2024-02-19T15:28:55.012834+0000","last_active":"2024-02-19T15:28:55.997573+0000","last_peered":"2024-02-19T15:28:55.997573+0000","last_clean":"2024-02-19T15:28:55.997573+0000","last_became_active":"2024-02-19T15:28:55.012200+0000","last_became_peered":"2024-02-19T15:28:55.012200+0000","last_unstale":"2024-02-19T15:28:55.997573+0000","last_undegraded":"2024-02-19T15:28:55.997573+0000","last_fullsized":"2024-02-19T15:28:55.997573+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:22:03.698617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.998347+0000","last_change":"2024-02-19T15:28:55.011248+0000","last_active":"2024-02-19T15:28:55.998347+0000","last_peered":"2024-02-19T15:28:55.998347+0000","last_clean":"2024-02-19T15:28:55.998347+0000","last_became_active":"2024-02-19T15:28:55.010739+0000","last_became_peered":"2024-02-19T15:28:55.010739+0000","last_unstale":"2024-02-19T15:28:55.998347+0000","last_undegraded":"2024-02-19T15:28:55.998347+0000","last_fullsized":"2024-02-19T15:28:55.998347+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:06:35.962884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:28:53.983641+0000","last_change":"2024-02-19T15:28:53.983641+0000","last_active":"2024-02-19T15:28:53.983641+0000","last_peered":"2024-02-19T15:28:53.983641+0000","last_clean":"2024-02-19T15:28:53.983641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:28:53.983641+0000","last_undegraded":"2024-02-19T15:28:53.983641+0000","last_fullsized":"2024-02-19T15:28:53.983641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.996941+0000","last_change":"2024-02-19T15:28:55.016650+0000","last_active":"2024-02-19T15:28:55.996941+0000","last_peered":"2024-02-19T15:28:55.996941+0000","last_clean":"2024-02-19T15:28:55.996941+0000","last_became_active":"2024-02-19T15:28:55.016324+0000","last_became_peered":"2024-02-19T15:28:55.016324+0000","last_unstale":"2024-02-19T15:28:55.996941+0000","last_undegraded":"2024-02-19T15:28:55.996941+0000","last_fullsized":"2024-02-19T15:28:55.996941+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:55:03.720594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.024398+0000","last_change":"2024-02-19T15:28:55.020595+0000","last_active":"2024-02-19T15:28:55.024398+0000","last_peered":"2024-02-19T15:28:55.024398+0000","last_clean":"2024-02-19T15:28:55.024398+0000","last_became_active":"2024-02-19T15:28:55.020240+0000","last_became_peered":"2024-02-19T15:28:55.020240+0000","last_unstale":"2024-02-19T15:28:55.024398+0000","last_undegraded":"2024-02-19T15:28:55.024398+0000","last_fullsized":"2024-02-19T15:28:55.024398+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:15.873342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:56.195924+0000","last_change":"2024-02-19T15:28:55.020804+0000","last_active":"2024-02-19T15:28:56.195924+0000","last_peered":"2024-02-19T15:28:56.195924+0000","last_clean":"2024-02-19T15:28:56.195924+0000","last_became_active":"2024-02-19T15:28:55.019838+0000","last_became_peered":"2024-02-19T15:28:55.019838+0000","last_unstale":"2024-02-19T15:28:56.195924+0000","last_undegraded":"2024-02-19T15:28:56.195924+0000","last_fullsized":"2024-02-19T15:28:56.195924+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:59:45.677859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:56.195670+0000","last_change":"2024-02-19T15:28:55.016093+0000","last_active":"2024-02-19T15:28:56.195670+0000","last_peered":"2024-02-19T15:28:56.195670+0000","last_clean":"2024-02-19T15:28:56.195670+0000","last_became_active":"2024-02-19T15:28:55.015790+0000","last_became_peered":"2024-02-19T15:28:55.015790+0000","last_unstale":"2024-02-19T15:28:56.195670+0000","last_undegraded":"2024-02-19T15:28:56.195670+0000","last_fullsized":"2024-02-19T15:28:56.195670+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:06:35.962884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:28:53.983641+0000","last_change":"2024-02-19T15:28:53.983641+0000","last_active":"2024-02-19T15:28:53.983641+0000","last_peered":"2024-02-19T15:28:53.983641+0000","last_clean":"2024-02-19T15:28:53.983641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:28:53.983641+0000","last_undegraded":"2024-02-19T15:28:53.983641+0000","last_fullsized":"2024-02-19T15:28:53.983641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:28:53.983641+0000","last_change":"2024-02-19T15:28:53.983641+0000","last_active":"2024-02-19T15:28:53.983641+0000","last_peered":"2024-02-19T15:28:53.983641+0000","last_clean":"2024-02-19T15:28:53.983641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:28:53.983641+0000","last_undegraded":"2024-02-19T15:28:53.983641+0000","last_fullsized":"2024-02-19T15:28:53.983641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.998391+0000","last_change":"2024-02-19T15:28:55.011009+0000","last_active":"2024-02-19T15:28:55.998391+0000","last_peered":"2024-02-19T15:28:55.998391+0000","last_clean":"2024-02-19T15:28:55.998391+0000","last_became_active":"2024-02-19T15:28:55.010328+0000","last_became_peered":"2024-02-19T15:28:55.010328+0000","last_unstale":"2024-02-19T15:28:55.998391+0000","last_undegraded":"2024-02-19T15:28:55.998391+0000","last_fullsized":"2024-02-19T15:28:55.998391+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:19:18.262726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.026272+0000","last_change":"2024-02-19T15:28:55.025024+0000","last_active":"2024-02-19T15:28:55.026272+0000","last_peered":"2024-02-19T15:28:55.026272+0000","last_clean":"2024-02-19T15:28:55.026272+0000","last_became_active":"2024-02-19T15:28:55.024729+0000","last_became_peered":"2024-02-19T15:28:55.024729+0000","last_unstale":"2024-02-19T15:28:55.026272+0000","last_undegraded":"2024-02-19T15:28:55.026272+0000","last_fullsized":"2024-02-19T15:28:55.026272+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:41:57.334120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:28:53.983641+0000","last_change":"2024-02-19T15:28:53.983641+0000","last_active":"2024-02-19T15:28:53.983641+0000","last_peered":"2024-02-19T15:28:53.983641+0000","last_clean":"2024-02-19T15:28:53.983641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:28:53.983641+0000","last_undegraded":"2024-02-19T15:28:53.983641+0000","last_fullsized":"2024-02-19T15:28:53.983641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997044+0000","last_change":"2024-02-19T15:28:55.010569+0000","last_active":"2024-02-19T15:28:55.997044+0000","last_peered":"2024-02-19T15:28:55.997044+0000","last_clean":"2024-02-19T15:28:55.997044+0000","last_became_active":"2024-02-19T15:28:55.010227+0000","last_became_peered":"2024-02-19T15:28:55.010227+0000","last_unstale":"2024-02-19T15:28:55.997044+0000","last_undegraded":"2024-02-19T15:28:55.997044+0000","last_fullsized":"2024-02-19T15:28:55.997044+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:59:51.326518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.998048+0000","last_change":"2024-02-19T15:28:55.013533+0000","last_active":"2024-02-19T15:28:55.998048+0000","last_peered":"2024-02-19T15:28:55.998048+0000","last_clean":"2024-02-19T15:28:55.998048+0000","last_became_active":"2024-02-19T15:28:55.013257+0000","last_became_peered":"2024-02-19T15:28:55.013257+0000","last_unstale":"2024-02-19T15:28:55.998048+0000","last_undegraded":"2024-02-19T15:28:55.998048+0000","last_fullsized":"2024-02-19T15:28:55.998048+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:09:50.600613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.996797+0000","last_change":"2024-02-19T15:28:55.002804+0000","last_active":"2024-02-19T15:28:55.996797+0000","last_peered":"2024-02-19T15:28:55.996797+0000","last_clean":"2024-02-19T15:28:55.996797+0000","last_became_active":"2024-02-19T15:28:55.002525+0000","last_became_peered":"2024-02-19T15:28:55.002525+0000","last_unstale":"2024-02-19T15:28:55.996797+0000","last_undegraded":"2024-02-19T15:28:55.996797+0000","last_fullsized":"2024-02-19T15:28:55.996797+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:22.956669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.036459+0000","last_change":"2024-02-19T15:28:55.033867+0000","last_active":"2024-02-19T15:28:55.036459+0000","last_peered":"2024-02-19T15:28:55.036459+0000","last_clean":"2024-02-19T15:28:55.036459+0000","last_became_active":"2024-02-19T15:28:55.033537+0000","last_became_peered":"2024-02-19T15:28:55.033537+0000","last_unstale":"2024-02-19T15:28:55.036459+0000","last_undegraded":"2024-02-19T15:28:55.036459+0000","last_fullsized":"2024-02-19T15:28:55.036459+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:00:25.674979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:56.196162+0000","last_change":"2024-02-19T15:28:55.025350+0000","last_active":"2024-02-19T15:28:56.196162+0000","last_peered":"2024-02-19T15:28:56.196162+0000","last_clean":"2024-02-19T15:28:56.196162+0000","last_became_active":"2024-02-19T15:28:55.024881+0000","last_became_peered":"2024-02-19T15:28:55.024881+0000","last_unstale":"2024-02-19T15:28:56.196162+0000","last_undegraded":"2024-02-19T15:28:56.196162+0000","last_fullsized":"2024-02-19T15:28:56.196162+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:52:43.326080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.037341+0000","last_change":"2024-02-19T15:28:55.035357+0000","last_active":"2024-02-19T15:28:55.037341+0000","last_peered":"2024-02-19T15:28:55.037341+0000","last_clean":"2024-02-19T15:28:55.037341+0000","last_became_active":"2024-02-19T15:28:55.034624+0000","last_became_peered":"2024-02-19T15:28:55.034624+0000","last_unstale":"2024-02-19T15:28:55.037341+0000","last_undegraded":"2024-02-19T15:28:55.037341+0000","last_fullsized":"2024-02-19T15:28:55.037341+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:15.873342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.031375+0000","last_change":"2024-02-19T15:28:55.028884+0000","last_active":"2024-02-19T15:28:55.031375+0000","last_peered":"2024-02-19T15:28:55.031375+0000","last_clean":"2024-02-19T15:28:55.031375+0000","last_became_active":"2024-02-19T15:28:55.028534+0000","last_became_peered":"2024-02-19T15:28:55.028534+0000","last_unstale":"2024-02-19T15:28:55.031375+0000","last_undegraded":"2024-02-19T15:28:55.031375+0000","last_fullsized":"2024-02-19T15:28:55.031375+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:22:40.379622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.037386+0000","last_change":"2024-02-19T15:28:55.035973+0000","last_active":"2024-02-19T15:28:55.037386+0000","last_peered":"2024-02-19T15:28:55.037386+0000","last_clean":"2024-02-19T15:28:55.037386+0000","last_became_active":"2024-02-19T15:28:55.035656+0000","last_became_peered":"2024-02-19T15:28:55.035656+0000","last_unstale":"2024-02-19T15:28:55.037386+0000","last_undegraded":"2024-02-19T15:28:55.037386+0000","last_fullsized":"2024-02-19T15:28:55.037386+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:49:00.412895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.024217+0000","last_change":"2024-02-19T15:28:55.020555+0000","last_active":"2024-02-19T15:28:55.024217+0000","last_peered":"2024-02-19T15:28:55.024217+0000","last_clean":"2024-02-19T15:28:55.024217+0000","last_became_active":"2024-02-19T15:28:55.020202+0000","last_became_peered":"2024-02-19T15:28:55.020202+0000","last_unstale":"2024-02-19T15:28:55.024217+0000","last_undegraded":"2024-02-19T15:28:55.024217+0000","last_fullsized":"2024-02-19T15:28:55.024217+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:00:23.721397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.025840+0000","last_change":"2024-02-19T15:28:55.023485+0000","last_active":"2024-02-19T15:28:55.025840+0000","last_peered":"2024-02-19T15:28:55.025840+0000","last_clean":"2024-02-19T15:28:55.025840+0000","last_became_active":"2024-02-19T15:28:55.023160+0000","last_became_peered":"2024-02-19T15:28:55.023160+0000","last_unstale":"2024-02-19T15:28:55.025840+0000","last_undegraded":"2024-02-19T15:28:55.025840+0000","last_fullsized":"2024-02-19T15:28:55.025840+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:19.307747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:28:53.983641+0000","last_change":"2024-02-19T15:28:53.983641+0000","last_active":"2024-02-19T15:28:53.983641+0000","last_peered":"2024-02-19T15:28:53.983641+0000","last_clean":"2024-02-19T15:28:53.983641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:28:53.983641+0000","last_undegraded":"2024-02-19T15:28:53.983641+0000","last_fullsized":"2024-02-19T15:28:53.983641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.031118+0000","last_change":"2024-02-19T15:28:55.028726+0000","last_active":"2024-02-19T15:28:55.031118+0000","last_peered":"2024-02-19T15:28:55.031118+0000","last_clean":"2024-02-19T15:28:55.031118+0000","last_became_active":"2024-02-19T15:28:55.028450+0000","last_became_peered":"2024-02-19T15:28:55.028450+0000","last_unstale":"2024-02-19T15:28:55.031118+0000","last_undegraded":"2024-02-19T15:28:55.031118+0000","last_fullsized":"2024-02-19T15:28:55.031118+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:31:18.921611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.998343+0000","last_change":"2024-02-19T15:28:55.016980+0000","last_active":"2024-02-19T15:28:55.998343+0000","last_peered":"2024-02-19T15:28:55.998343+0000","last_clean":"2024-02-19T15:28:55.998343+0000","last_became_active":"2024-02-19T15:28:55.016767+0000","last_became_peered":"2024-02-19T15:28:55.016767+0000","last_unstale":"2024-02-19T15:28:55.998343+0000","last_undegraded":"2024-02-19T15:28:55.998343+0000","last_fullsized":"2024-02-19T15:28:55.998343+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:32.057879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.030714+0000","last_change":"2024-02-19T15:28:55.020698+0000","last_active":"2024-02-19T15:28:55.030714+0000","last_peered":"2024-02-19T15:28:55.030714+0000","last_clean":"2024-02-19T15:28:55.030714+0000","last_became_active":"2024-02-19T15:28:55.020444+0000","last_became_peered":"2024-02-19T15:28:55.020444+0000","last_unstale":"2024-02-19T15:28:55.030714+0000","last_undegraded":"2024-02-19T15:28:55.030714+0000","last_fullsized":"2024-02-19T15:28:55.030714+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:02:55.430828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.996973+0000","last_change":"2024-02-19T15:28:55.010867+0000","last_active":"2024-02-19T15:28:55.996973+0000","last_peered":"2024-02-19T15:28:55.996973+0000","last_clean":"2024-02-19T15:28:55.996973+0000","last_became_active":"2024-02-19T15:28:55.010605+0000","last_became_peered":"2024-02-19T15:28:55.010605+0000","last_unstale":"2024-02-19T15:28:55.996973+0000","last_undegraded":"2024-02-19T15:28:55.996973+0000","last_fullsized":"2024-02-19T15:28:55.996973+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:41:07.665818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.030693+0000","last_change":"2024-02-19T15:28:55.028834+0000","last_active":"2024-02-19T15:28:55.030693+0000","last_peered":"2024-02-19T15:28:55.030693+0000","last_clean":"2024-02-19T15:28:55.030693+0000","last_became_active":"2024-02-19T15:28:55.028547+0000","last_became_peered":"2024-02-19T15:28:55.028547+0000","last_unstale":"2024-02-19T15:28:55.030693+0000","last_undegraded":"2024-02-19T15:28:55.030693+0000","last_fullsized":"2024-02-19T15:28:55.030693+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:24.706275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997214+0000","last_change":"2024-02-19T15:28:55.015157+0000","last_active":"2024-02-19T15:28:55.997214+0000","last_peered":"2024-02-19T15:28:55.997214+0000","last_clean":"2024-02-19T15:28:55.997214+0000","last_became_active":"2024-02-19T15:28:55.014874+0000","last_became_peered":"2024-02-19T15:28:55.014874+0000","last_unstale":"2024-02-19T15:28:55.997214+0000","last_undegraded":"2024-02-19T15:28:55.997214+0000","last_fullsized":"2024-02-19T15:28:55.997214+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:59:22.763599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997079+0000","last_change":"2024-02-19T15:28:55.003282+0000","last_active":"2024-02-19T15:28:55.997079+0000","last_peered":"2024-02-19T15:28:55.997079+0000","last_clean":"2024-02-19T15:28:55.997079+0000","last_became_active":"2024-02-19T15:28:55.002930+0000","last_became_peered":"2024-02-19T15:28:55.002930+0000","last_unstale":"2024-02-19T15:28:55.997079+0000","last_undegraded":"2024-02-19T15:28:55.997079+0000","last_fullsized":"2024-02-19T15:28:55.997079+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:26:58.415308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.030541+0000","last_change":"2024-02-19T15:28:55.029082+0000","last_active":"2024-02-19T15:28:55.030541+0000","last_peered":"2024-02-19T15:28:55.030541+0000","last_clean":"2024-02-19T15:28:55.030541+0000","last_became_active":"2024-02-19T15:28:55.027609+0000","last_became_peered":"2024-02-19T15:28:55.027609+0000","last_unstale":"2024-02-19T15:28:55.030541+0000","last_undegraded":"2024-02-19T15:28:55.030541+0000","last_fullsized":"2024-02-19T15:28:55.030541+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:48:40.828864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:56.195760+0000","last_change":"2024-02-19T15:28:55.034276+0000","last_active":"2024-02-19T15:28:56.195760+0000","last_peered":"2024-02-19T15:28:56.195760+0000","last_clean":"2024-02-19T15:28:56.195760+0000","last_became_active":"2024-02-19T15:28:55.033935+0000","last_became_peered":"2024-02-19T15:28:55.033935+0000","last_unstale":"2024-02-19T15:28:56.195760+0000","last_undegraded":"2024-02-19T15:28:56.195760+0000","last_fullsized":"2024-02-19T15:28:56.195760+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:19:18.262726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.037364+0000","last_change":"2024-02-19T15:28:55.035854+0000","last_active":"2024-02-19T15:28:55.037364+0000","last_peered":"2024-02-19T15:28:55.037364+0000","last_clean":"2024-02-19T15:28:55.037364+0000","last_became_active":"2024-02-19T15:28:55.034504+0000","last_became_peered":"2024-02-19T15:28:55.034504+0000","last_unstale":"2024-02-19T15:28:55.037364+0000","last_undegraded":"2024-02-19T15:28:55.037364+0000","last_fullsized":"2024-02-19T15:28:55.037364+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:41:45.635707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"124'4","reported_seq":16,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.029963+0000","last_change":"2024-02-19T15:28:54.017441+0000","last_active":"2024-02-19T15:28:55.029963+0000","last_peered":"2024-02-19T15:28:55.029963+0000","last_clean":"2024-02-19T15:28:55.029963+0000","last_became_active":"2024-02-19T15:28:54.017178+0000","last_became_peered":"2024-02-19T15:28:54.017178+0000","last_unstale":"2024-02-19T15:28:55.029963+0000","last_undegraded":"2024-02-19T15:28:55.029963+0000","last_fullsized":"2024-02-19T15:28:55.029963+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:00:24.672075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:28:53.983641+0000","last_change":"2024-02-19T15:28:53.983641+0000","last_active":"2024-02-19T15:28:53.983641+0000","last_peered":"2024-02-19T15:28:53.983641+0000","last_clean":"2024-02-19T15:28:53.983641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:28:53.983641+0000","last_undegraded":"2024-02-19T15:28:53.983641+0000","last_fullsized":"2024-02-19T15:28:53.983641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1e","version":"124'1","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.022958+0000","last_change":"2024-02-19T15:28:54.019905+0000","last_active":"2024-02-19T15:28:55.022958+0000","last_peered":"2024-02-19T15:28:55.022958+0000","last_clean":"2024-02-19T15:28:55.022958+0000","last_became_active":"2024-02-19T15:28:54.019651+0000","last_became_peered":"2024-02-19T15:28:54.019651+0000","last_unstale":"2024-02-19T15:28:55.022958+0000","last_undegraded":"2024-02-19T15:28:55.022958+0000","last_fullsized":"2024-02-19T15:28:55.022958+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:16:40.743331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.030507+0000","last_change":"2024-02-19T15:28:55.027266+0000","last_active":"2024-02-19T15:28:55.030507+0000","last_peered":"2024-02-19T15:28:55.030507+0000","last_clean":"2024-02-19T15:28:55.030507+0000","last_became_active":"2024-02-19T15:28:55.026242+0000","last_became_peered":"2024-02-19T15:28:55.026242+0000","last_unstale":"2024-02-19T15:28:55.030507+0000","last_undegraded":"2024-02-19T15:28:55.030507+0000","last_fullsized":"2024-02-19T15:28:55.030507+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:00:22.232665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"124'2","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.031992+0000","last_change":"2024-02-19T15:28:54.015943+0000","last_active":"2024-02-19T15:28:55.031992+0000","last_peered":"2024-02-19T15:28:55.031992+0000","last_clean":"2024-02-19T15:28:55.031992+0000","last_became_active":"2024-02-19T15:28:54.015679+0000","last_became_peered":"2024-02-19T15:28:54.015679+0000","last_unstale":"2024-02-19T15:28:55.031992+0000","last_undegraded":"2024-02-19T15:28:55.031992+0000","last_fullsized":"2024-02-19T15:28:55.031992+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:09:28.603113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.029786+0000","last_change":"2024-02-19T15:28:55.026500+0000","last_active":"2024-02-19T15:28:55.029786+0000","last_peered":"2024-02-19T15:28:55.029786+0000","last_clean":"2024-02-19T15:28:55.029786+0000","last_became_active":"2024-02-19T15:28:55.026152+0000","last_became_peered":"2024-02-19T15:28:55.026152+0000","last_unstale":"2024-02-19T15:28:55.029786+0000","last_undegraded":"2024-02-19T15:28:55.029786+0000","last_fullsized":"2024-02-19T15:28:55.029786+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:42:31.827963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.998268+0000","last_change":"2024-02-19T15:28:54.010287+0000","last_active":"2024-02-19T15:28:55.998268+0000","last_peered":"2024-02-19T15:28:55.998268+0000","last_clean":"2024-02-19T15:28:55.998268+0000","last_became_active":"2024-02-19T15:28:54.009959+0000","last_became_peered":"2024-02-19T15:28:54.009959+0000","last_unstale":"2024-02-19T15:28:55.998268+0000","last_undegraded":"2024-02-19T15:28:55.998268+0000","last_fullsized":"2024-02-19T15:28:55.998268+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:41:03.496395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.037295+0000","last_change":"2024-02-19T15:28:55.035147+0000","last_active":"2024-02-19T15:28:55.037295+0000","last_peered":"2024-02-19T15:28:55.037295+0000","last_clean":"2024-02-19T15:28:55.037295+0000","last_became_active":"2024-02-19T15:28:55.034641+0000","last_became_peered":"2024-02-19T15:28:55.034641+0000","last_unstale":"2024-02-19T15:28:55.037295+0000","last_undegraded":"2024-02-19T15:28:55.037295+0000","last_fullsized":"2024-02-19T15:28:55.037295+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:00:23.721397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"124'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.998017+0000","last_change":"2024-02-19T15:28:54.013352+0000","last_active":"2024-02-19T15:28:55.998017+0000","last_peered":"2024-02-19T15:28:55.998017+0000","last_clean":"2024-02-19T15:28:55.998017+0000","last_became_active":"2024-02-19T15:28:54.013001+0000","last_became_peered":"2024-02-19T15:28:54.013001+0000","last_unstale":"2024-02-19T15:28:55.998017+0000","last_undegraded":"2024-02-19T15:28:55.998017+0000","last_fullsized":"2024-02-19T15:28:55.998017+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:29:12.356834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:56.195848+0000","last_change":"2024-02-19T15:28:55.023488+0000","last_active":"2024-02-19T15:28:56.195848+0000","last_peered":"2024-02-19T15:28:56.195848+0000","last_clean":"2024-02-19T15:28:56.195848+0000","last_became_active":"2024-02-19T15:28:55.023195+0000","last_became_peered":"2024-02-19T15:28:55.023195+0000","last_unstale":"2024-02-19T15:28:56.195848+0000","last_undegraded":"2024-02-19T15:28:56.195848+0000","last_fullsized":"2024-02-19T15:28:56.195848+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:21.882175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"124'1","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.035459+0000","last_change":"2024-02-19T15:28:54.012384+0000","last_active":"2024-02-19T15:28:55.035459+0000","last_peered":"2024-02-19T15:28:55.035459+0000","last_clean":"2024-02-19T15:28:55.035459+0000","last_became_active":"2024-02-19T15:28:54.010948+0000","last_became_peered":"2024-02-19T15:28:54.010948+0000","last_unstale":"2024-02-19T15:28:55.035459+0000","last_undegraded":"2024-02-19T15:28:55.035459+0000","last_fullsized":"2024-02-19T15:28:55.035459+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:52:42.323176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:28:53.983641+0000","last_change":"2024-02-19T15:28:53.983641+0000","last_active":"2024-02-19T15:28:53.983641+0000","last_peered":"2024-02-19T15:28:53.983641+0000","last_clean":"2024-02-19T15:28:53.983641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:28:53.983641+0000","last_undegraded":"2024-02-19T15:28:53.983641+0000","last_fullsized":"2024-02-19T15:28:53.983641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:56.195655+0000","last_change":"2024-02-19T15:28:54.008537+0000","last_active":"2024-02-19T15:28:56.195655+0000","last_peered":"2024-02-19T15:28:56.195655+0000","last_clean":"2024-02-19T15:28:56.195655+0000","last_became_active":"2024-02-19T15:28:54.007669+0000","last_became_peered":"2024-02-19T15:28:54.007669+0000","last_unstale":"2024-02-19T15:28:56.195655+0000","last_undegraded":"2024-02-19T15:28:56.195655+0000","last_fullsized":"2024-02-19T15:28:56.195655+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:23.703371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997865+0000","last_change":"2024-02-19T15:28:55.031153+0000","last_active":"2024-02-19T15:28:55.997865+0000","last_peered":"2024-02-19T15:28:55.997865+0000","last_clean":"2024-02-19T15:28:55.997865+0000","last_became_active":"2024-02-19T15:28:55.030410+0000","last_became_peered":"2024-02-19T15:28:55.030410+0000","last_unstale":"2024-02-19T15:28:55.997865+0000","last_undegraded":"2024-02-19T15:28:55.997865+0000","last_fullsized":"2024-02-19T15:28:55.997865+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:05:32.136340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997945+0000","last_change":"2024-02-19T15:28:54.008237+0000","last_active":"2024-02-19T15:28:55.997945+0000","last_peered":"2024-02-19T15:28:55.997945+0000","last_clean":"2024-02-19T15:28:55.997945+0000","last_became_active":"2024-02-19T15:28:54.007956+0000","last_became_peered":"2024-02-19T15:28:54.007956+0000","last_unstale":"2024-02-19T15:28:55.997945+0000","last_undegraded":"2024-02-19T15:28:55.997945+0000","last_fullsized":"2024-02-19T15:28:55.997945+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:12:21.539862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997282+0000","last_change":"2024-02-19T15:28:55.025586+0000","last_active":"2024-02-19T15:28:55.997282+0000","last_peered":"2024-02-19T15:28:55.997282+0000","last_clean":"2024-02-19T15:28:55.997282+0000","last_became_active":"2024-02-19T15:28:55.025327+0000","last_became_peered":"2024-02-19T15:28:55.025327+0000","last_unstale":"2024-02-19T15:28:55.997282+0000","last_undegraded":"2024-02-19T15:28:55.997282+0000","last_fullsized":"2024-02-19T15:28:55.997282+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:59:43.922225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.002123+0000","last_change":"2024-02-19T15:28:54.012138+0000","last_active":"2024-02-19T15:28:55.002123+0000","last_peered":"2024-02-19T15:28:55.002123+0000","last_clean":"2024-02-19T15:28:55.002123+0000","last_became_active":"2024-02-19T15:28:54.011858+0000","last_became_peered":"2024-02-19T15:28:54.011858+0000","last_unstale":"2024-02-19T15:28:55.002123+0000","last_undegraded":"2024-02-19T15:28:55.002123+0000","last_fullsized":"2024-02-19T15:28:55.002123+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:41:03.496395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.030639+0000","last_change":"2024-02-19T15:28:55.028215+0000","last_active":"2024-02-19T15:28:55.030639+0000","last_peered":"2024-02-19T15:28:55.030639+0000","last_clean":"2024-02-19T15:28:55.030639+0000","last_became_active":"2024-02-19T15:28:55.028012+0000","last_became_peered":"2024-02-19T15:28:55.028012+0000","last_unstale":"2024-02-19T15:28:55.030639+0000","last_undegraded":"2024-02-19T15:28:55.030639+0000","last_fullsized":"2024-02-19T15:28:55.030639+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:43:09.938262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.013178+0000","last_change":"2024-02-19T15:28:54.017288+0000","last_active":"2024-02-19T15:28:55.013178+0000","last_peered":"2024-02-19T15:28:55.013178+0000","last_clean":"2024-02-19T15:28:55.013178+0000","last_became_active":"2024-02-19T15:28:54.017012+0000","last_became_peered":"2024-02-19T15:28:54.017012+0000","last_unstale":"2024-02-19T15:28:55.013178+0000","last_undegraded":"2024-02-19T15:28:55.013178+0000","last_fullsized":"2024-02-19T15:28:55.013178+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:33:21.121261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.035499+0000","last_change":"2024-02-19T15:28:55.034148+0000","last_active":"2024-02-19T15:28:55.035499+0000","last_peered":"2024-02-19T15:28:55.035499+0000","last_clean":"2024-02-19T15:28:55.035499+0000","last_became_active":"2024-02-19T15:28:55.033835+0000","last_became_peered":"2024-02-19T15:28:55.033835+0000","last_unstale":"2024-02-19T15:28:55.035499+0000","last_undegraded":"2024-02-19T15:28:55.035499+0000","last_fullsized":"2024-02-19T15:28:55.035499+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:36:13.025443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"124'3","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.036017+0000","last_change":"2024-02-19T15:28:54.017365+0000","last_active":"2024-02-19T15:28:55.036017+0000","last_peered":"2024-02-19T15:28:55.036017+0000","last_clean":"2024-02-19T15:28:55.036017+0000","last_became_active":"2024-02-19T15:28:54.017028+0000","last_became_peered":"2024-02-19T15:28:54.017028+0000","last_unstale":"2024-02-19T15:28:55.036017+0000","last_undegraded":"2024-02-19T15:28:55.036017+0000","last_fullsized":"2024-02-19T15:28:55.036017+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:52:44.486731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.026393+0000","last_change":"2024-02-19T15:28:55.020689+0000","last_active":"2024-02-19T15:28:55.026393+0000","last_peered":"2024-02-19T15:28:55.026393+0000","last_clean":"2024-02-19T15:28:55.026393+0000","last_became_active":"2024-02-19T15:28:55.020369+0000","last_became_peered":"2024-02-19T15:28:55.020369+0000","last_unstale":"2024-02-19T15:28:55.026393+0000","last_undegraded":"2024-02-19T15:28:55.026393+0000","last_fullsized":"2024-02-19T15:28:55.026393+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:41:45.635707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:28:52.980737+0000","last_change":"2024-02-19T15:28:52.980737+0000","last_active":"2024-02-19T15:28:52.980737+0000","last_peered":"2024-02-19T15:28:52.980737+0000","last_clean":"2024-02-19T15:28:52.980737+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:28:52.980737+0000","last_undegraded":"2024-02-19T15:28:52.980737+0000","last_fullsized":"2024-02-19T15:28:52.980737+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997370+0000","last_change":"2024-02-19T15:28:55.025895+0000","last_active":"2024-02-19T15:28:55.997370+0000","last_peered":"2024-02-19T15:28:55.997370+0000","last_clean":"2024-02-19T15:28:55.997370+0000","last_became_active":"2024-02-19T15:28:55.025602+0000","last_became_peered":"2024-02-19T15:28:55.025602+0000","last_unstale":"2024-02-19T15:28:55.997370+0000","last_undegraded":"2024-02-19T15:28:55.997370+0000","last_fullsized":"2024-02-19T15:28:55.997370+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:12:11.330108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997194+0000","last_change":"2024-02-19T15:28:54.015919+0000","last_active":"2024-02-19T15:28:55.997194+0000","last_peered":"2024-02-19T15:28:55.997194+0000","last_clean":"2024-02-19T15:28:55.997194+0000","last_became_active":"2024-02-19T15:28:54.015028+0000","last_became_peered":"2024-02-19T15:28:54.015028+0000","last_unstale":"2024-02-19T15:28:55.997194+0000","last_undegraded":"2024-02-19T15:28:55.997194+0000","last_fullsized":"2024-02-19T15:28:55.997194+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:51:24.273929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997839+0000","last_change":"2024-02-19T15:28:55.008077+0000","last_active":"2024-02-19T15:28:55.997839+0000","last_peered":"2024-02-19T15:28:55.997839+0000","last_clean":"2024-02-19T15:28:55.997839+0000","last_became_active":"2024-02-19T15:28:55.005108+0000","last_became_peered":"2024-02-19T15:28:55.005108+0000","last_unstale":"2024-02-19T15:28:55.997839+0000","last_undegraded":"2024-02-19T15:28:55.997839+0000","last_fullsized":"2024-02-19T15:28:55.997839+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:04:44.776418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997962+0000","last_change":"2024-02-19T15:28:54.009514+0000","last_active":"2024-02-19T15:28:55.997962+0000","last_peered":"2024-02-19T15:28:55.997962+0000","last_clean":"2024-02-19T15:28:55.997962+0000","last_became_active":"2024-02-19T15:28:54.009225+0000","last_became_peered":"2024-02-19T15:28:54.009225+0000","last_unstale":"2024-02-19T15:28:55.997962+0000","last_undegraded":"2024-02-19T15:28:55.997962+0000","last_fullsized":"2024-02-19T15:28:55.997962+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:32:03.071994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:28:53.983641+0000","last_change":"2024-02-19T15:28:53.983641+0000","last_active":"2024-02-19T15:28:53.983641+0000","last_peered":"2024-02-19T15:28:53.983641+0000","last_clean":"2024-02-19T15:28:53.983641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:28:53.983641+0000","last_undegraded":"2024-02-19T15:28:53.983641+0000","last_fullsized":"2024-02-19T15:28:53.983641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:28:52.980737+0000","last_change":"2024-02-19T15:28:52.980737+0000","last_active":"2024-02-19T15:28:52.980737+0000","last_peered":"2024-02-19T15:28:52.980737+0000","last_clean":"2024-02-19T15:28:52.980737+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:28:52.980737+0000","last_undegraded":"2024-02-19T15:28:52.980737+0000","last_fullsized":"2024-02-19T15:28:52.980737+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:28:53.983641+0000","last_change":"2024-02-19T15:28:53.983641+0000","last_active":"2024-02-19T15:28:53.983641+0000","last_peered":"2024-02-19T15:28:53.983641+0000","last_clean":"2024-02-19T15:28:53.983641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:28:53.983641+0000","last_undegraded":"2024-02-19T15:28:53.983641+0000","last_fullsized":"2024-02-19T15:28:53.983641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.f","version":"124'1","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.027768+0000","last_change":"2024-02-19T15:28:54.013010+0000","last_active":"2024-02-19T15:28:55.027768+0000","last_peered":"2024-02-19T15:28:55.027768+0000","last_clean":"2024-02-19T15:28:55.027768+0000","last_became_active":"2024-02-19T15:28:54.012003+0000","last_became_peered":"2024-02-19T15:28:54.012003+0000","last_unstale":"2024-02-19T15:28:55.027768+0000","last_undegraded":"2024-02-19T15:28:55.027768+0000","last_fullsized":"2024-02-19T15:28:55.027768+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:26:23.224887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.030454+0000","last_change":"2024-02-19T15:28:55.025608+0000","last_active":"2024-02-19T15:28:55.030454+0000","last_peered":"2024-02-19T15:28:55.030454+0000","last_clean":"2024-02-19T15:28:55.030454+0000","last_became_active":"2024-02-19T15:28:55.025335+0000","last_became_peered":"2024-02-19T15:28:55.025335+0000","last_unstale":"2024-02-19T15:28:55.030454+0000","last_undegraded":"2024-02-19T15:28:55.030454+0000","last_fullsized":"2024-02-19T15:28:55.030454+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:17:30.675159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"124'2","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.025318+0000","last_change":"2024-02-19T15:28:54.009778+0000","last_active":"2024-02-19T15:28:55.025318+0000","last_peered":"2024-02-19T15:28:55.025318+0000","last_clean":"2024-02-19T15:28:55.025318+0000","last_became_active":"2024-02-19T15:28:54.008046+0000","last_became_peered":"2024-02-19T15:28:54.008046+0000","last_unstale":"2024-02-19T15:28:55.025318+0000","last_undegraded":"2024-02-19T15:28:55.025318+0000","last_fullsized":"2024-02-19T15:28:55.025318+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:37:10.679182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.029547+0000","last_change":"2024-02-19T15:28:55.028307+0000","last_active":"2024-02-19T15:28:55.029547+0000","last_peered":"2024-02-19T15:28:55.029547+0000","last_clean":"2024-02-19T15:28:55.029547+0000","last_became_active":"2024-02-19T15:28:55.028027+0000","last_became_peered":"2024-02-19T15:28:55.028027+0000","last_unstale":"2024-02-19T15:28:55.029547+0000","last_undegraded":"2024-02-19T15:28:55.029547+0000","last_fullsized":"2024-02-19T15:28:55.029547+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:36:32.215403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"124'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:56.196027+0000","last_change":"2024-02-19T15:28:54.012595+0000","last_active":"2024-02-19T15:28:56.196027+0000","last_peered":"2024-02-19T15:28:56.196027+0000","last_clean":"2024-02-19T15:28:56.196027+0000","last_became_active":"2024-02-19T15:28:54.012267+0000","last_became_peered":"2024-02-19T15:28:54.012267+0000","last_unstale":"2024-02-19T15:28:56.196027+0000","last_undegraded":"2024-02-19T15:28:56.196027+0000","last_fullsized":"2024-02-19T15:28:56.196027+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:43:08.935358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.998420+0000","last_change":"2024-02-19T15:28:55.030666+0000","last_active":"2024-02-19T15:28:55.998420+0000","last_peered":"2024-02-19T15:28:55.998420+0000","last_clean":"2024-02-19T15:28:55.998420+0000","last_became_active":"2024-02-19T15:28:55.030379+0000","last_became_peered":"2024-02-19T15:28:55.030379+0000","last_unstale":"2024-02-19T15:28:55.998420+0000","last_undegraded":"2024-02-19T15:28:55.998420+0000","last_fullsized":"2024-02-19T15:28:55.998420+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:21.882175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.001878+0000","last_change":"2024-02-19T15:28:54.019286+0000","last_active":"2024-02-19T15:28:55.001878+0000","last_peered":"2024-02-19T15:28:55.001878+0000","last_clean":"2024-02-19T15:28:55.001878+0000","last_became_active":"2024-02-19T15:28:54.018926+0000","last_became_peered":"2024-02-19T15:28:54.018926+0000","last_unstale":"2024-02-19T15:28:55.001878+0000","last_undegraded":"2024-02-19T15:28:55.001878+0000","last_fullsized":"2024-02-19T15:28:55.001878+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:32:03.071994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997461+0000","last_change":"2024-02-19T15:28:55.011735+0000","last_active":"2024-02-19T15:28:55.997461+0000","last_peered":"2024-02-19T15:28:55.997461+0000","last_clean":"2024-02-19T15:28:55.997461+0000","last_became_active":"2024-02-19T15:28:55.011408+0000","last_became_peered":"2024-02-19T15:28:55.011408+0000","last_unstale":"2024-02-19T15:28:55.997461+0000","last_undegraded":"2024-02-19T15:28:55.997461+0000","last_fullsized":"2024-02-19T15:28:55.997461+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:44:43.807957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"124'2","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.024755+0000","last_change":"2024-02-19T15:28:54.014397+0000","last_active":"2024-02-19T15:28:55.024755+0000","last_peered":"2024-02-19T15:28:55.024755+0000","last_clean":"2024-02-19T15:28:55.024755+0000","last_became_active":"2024-02-19T15:28:54.013650+0000","last_became_peered":"2024-02-19T15:28:54.013650+0000","last_unstale":"2024-02-19T15:28:55.024755+0000","last_undegraded":"2024-02-19T15:28:55.024755+0000","last_fullsized":"2024-02-19T15:28:55.024755+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:48.551468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:56.195987+0000","last_change":"2024-02-19T15:28:55.019576+0000","last_active":"2024-02-19T15:28:56.195987+0000","last_peered":"2024-02-19T15:28:56.195987+0000","last_clean":"2024-02-19T15:28:56.195987+0000","last_became_active":"2024-02-19T15:28:55.019006+0000","last_became_peered":"2024-02-19T15:28:55.019006+0000","last_unstale":"2024-02-19T15:28:56.195987+0000","last_undegraded":"2024-02-19T15:28:56.195987+0000","last_fullsized":"2024-02-19T15:28:56.195987+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:54:31.300972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.012249+0000","last_change":"2024-02-19T15:28:54.016895+0000","last_active":"2024-02-19T15:28:55.012249+0000","last_peered":"2024-02-19T15:28:55.012249+0000","last_clean":"2024-02-19T15:28:55.012249+0000","last_became_active":"2024-02-19T15:28:54.016590+0000","last_became_peered":"2024-02-19T15:28:54.016590+0000","last_unstale":"2024-02-19T15:28:55.012249+0000","last_undegraded":"2024-02-19T15:28:55.012249+0000","last_fullsized":"2024-02-19T15:28:55.012249+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:27:57.216088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997845+0000","last_change":"2024-02-19T15:28:55.024467+0000","last_active":"2024-02-19T15:28:55.997845+0000","last_peered":"2024-02-19T15:28:55.997845+0000","last_clean":"2024-02-19T15:28:55.997845+0000","last_became_active":"2024-02-19T15:28:55.024130+0000","last_became_peered":"2024-02-19T15:28:55.024130+0000","last_unstale":"2024-02-19T15:28:55.997845+0000","last_undegraded":"2024-02-19T15:28:55.997845+0000","last_fullsized":"2024-02-19T15:28:55.997845+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:11:46.168966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.012646+0000","last_change":"2024-02-19T15:28:54.011800+0000","last_active":"2024-02-19T15:28:55.012646+0000","last_peered":"2024-02-19T15:28:55.012646+0000","last_clean":"2024-02-19T15:28:55.012646+0000","last_became_active":"2024-02-19T15:28:54.011499+0000","last_became_peered":"2024-02-19T15:28:54.011499+0000","last_unstale":"2024-02-19T15:28:55.012646+0000","last_undegraded":"2024-02-19T15:28:55.012646+0000","last_fullsized":"2024-02-19T15:28:55.012646+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:37:58.857193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.025990+0000","last_change":"2024-02-19T15:28:55.022411+0000","last_active":"2024-02-19T15:28:55.025990+0000","last_peered":"2024-02-19T15:28:55.025990+0000","last_clean":"2024-02-19T15:28:55.025990+0000","last_became_active":"2024-02-19T15:28:55.022007+0000","last_became_peered":"2024-02-19T15:28:55.022007+0000","last_unstale":"2024-02-19T15:28:55.025990+0000","last_undegraded":"2024-02-19T15:28:55.025990+0000","last_fullsized":"2024-02-19T15:28:55.025990+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:49:00.412895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.018810+0000","last_change":"2024-02-19T15:28:54.009536+0000","last_active":"2024-02-19T15:28:55.018810+0000","last_peered":"2024-02-19T15:28:55.018810+0000","last_clean":"2024-02-19T15:28:55.018810+0000","last_became_active":"2024-02-19T15:28:54.009226+0000","last_became_peered":"2024-02-19T15:28:54.009226+0000","last_unstale":"2024-02-19T15:28:55.018810+0000","last_undegraded":"2024-02-19T15:28:55.018810+0000","last_fullsized":"2024-02-19T15:28:55.018810+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:28:21.679679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:56.196180+0000","last_change":"2024-02-19T15:28:55.025560+0000","last_active":"2024-02-19T15:28:56.196180+0000","last_peered":"2024-02-19T15:28:56.196180+0000","last_clean":"2024-02-19T15:28:56.196180+0000","last_became_active":"2024-02-19T15:28:55.025201+0000","last_became_peered":"2024-02-19T15:28:55.025201+0000","last_unstale":"2024-02-19T15:28:56.196180+0000","last_undegraded":"2024-02-19T15:28:56.196180+0000","last_fullsized":"2024-02-19T15:28:56.196180+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:27:58.218992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997601+0000","last_change":"2024-02-19T15:28:54.014553+0000","last_active":"2024-02-19T15:28:55.997601+0000","last_peered":"2024-02-19T15:28:55.997601+0000","last_clean":"2024-02-19T15:28:55.997601+0000","last_became_active":"2024-02-19T15:28:54.014103+0000","last_became_peered":"2024-02-19T15:28:54.014103+0000","last_unstale":"2024-02-19T15:28:55.997601+0000","last_undegraded":"2024-02-19T15:28:55.997601+0000","last_fullsized":"2024-02-19T15:28:55.997601+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:10:19.992342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997331+0000","last_change":"2024-02-19T15:28:55.013057+0000","last_active":"2024-02-19T15:28:55.997331+0000","last_peered":"2024-02-19T15:28:55.997331+0000","last_clean":"2024-02-19T15:28:55.997331+0000","last_became_active":"2024-02-19T15:28:55.012566+0000","last_became_peered":"2024-02-19T15:28:55.012566+0000","last_unstale":"2024-02-19T15:28:55.997331+0000","last_undegraded":"2024-02-19T15:28:55.997331+0000","last_fullsized":"2024-02-19T15:28:55.997331+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:25:39.345932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.011366+0000","last_change":"2024-02-19T15:28:54.015239+0000","last_active":"2024-02-19T15:28:55.011366+0000","last_peered":"2024-02-19T15:28:55.011366+0000","last_clean":"2024-02-19T15:28:55.011366+0000","last_became_active":"2024-02-19T15:28:54.014910+0000","last_became_peered":"2024-02-19T15:28:54.014910+0000","last_unstale":"2024-02-19T15:28:55.011366+0000","last_undegraded":"2024-02-19T15:28:55.011366+0000","last_fullsized":"2024-02-19T15:28:55.011366+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:40:27.107863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.998291+0000","last_change":"2024-02-19T15:28:55.015539+0000","last_active":"2024-02-19T15:28:55.998291+0000","last_peered":"2024-02-19T15:28:55.998291+0000","last_clean":"2024-02-19T15:28:55.998291+0000","last_became_active":"2024-02-19T15:28:55.015311+0000","last_became_peered":"2024-02-19T15:28:55.015311+0000","last_unstale":"2024-02-19T15:28:55.998291+0000","last_undegraded":"2024-02-19T15:28:55.998291+0000","last_fullsized":"2024-02-19T15:28:55.998291+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:36:00.916278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.998301+0000","last_change":"2024-02-19T15:28:54.006923+0000","last_active":"2024-02-19T15:28:55.998301+0000","last_peered":"2024-02-19T15:28:55.998301+0000","last_clean":"2024-02-19T15:28:55.998301+0000","last_became_active":"2024-02-19T15:28:54.006565+0000","last_became_peered":"2024-02-19T15:28:54.006565+0000","last_unstale":"2024-02-19T15:28:55.998301+0000","last_undegraded":"2024-02-19T15:28:55.998301+0000","last_fullsized":"2024-02-19T15:28:55.998301+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:09:31.778802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:56.195989+0000","last_change":"2024-02-19T15:28:55.020806+0000","last_active":"2024-02-19T15:28:56.195989+0000","last_peered":"2024-02-19T15:28:56.195989+0000","last_clean":"2024-02-19T15:28:56.195989+0000","last_became_active":"2024-02-19T15:28:55.020492+0000","last_became_peered":"2024-02-19T15:28:55.020492+0000","last_unstale":"2024-02-19T15:28:56.195989+0000","last_undegraded":"2024-02-19T15:28:56.195989+0000","last_fullsized":"2024-02-19T15:28:56.195989+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:34.966857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.019728+0000","last_change":"2024-02-19T15:28:54.012342+0000","last_active":"2024-02-19T15:28:55.019728+0000","last_peered":"2024-02-19T15:28:55.019728+0000","last_clean":"2024-02-19T15:28:55.019728+0000","last_became_active":"2024-02-19T15:28:54.011912+0000","last_became_peered":"2024-02-19T15:28:54.011912+0000","last_unstale":"2024-02-19T15:28:55.019728+0000","last_undegraded":"2024-02-19T15:28:55.019728+0000","last_fullsized":"2024-02-19T15:28:55.019728+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:08:02.941863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.029783+0000","last_change":"2024-02-19T15:28:55.026791+0000","last_active":"2024-02-19T15:28:55.029783+0000","last_peered":"2024-02-19T15:28:55.029783+0000","last_clean":"2024-02-19T15:28:55.029783+0000","last_became_active":"2024-02-19T15:28:55.026245+0000","last_became_peered":"2024-02-19T15:28:55.026245+0000","last_unstale":"2024-02-19T15:28:55.029783+0000","last_undegraded":"2024-02-19T15:28:55.029783+0000","last_fullsized":"2024-02-19T15:28:55.029783+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:24:16.358341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.998010+0000","last_change":"2024-02-19T15:28:54.008333+0000","last_active":"2024-02-19T15:28:55.998010+0000","last_peered":"2024-02-19T15:28:55.998010+0000","last_clean":"2024-02-19T15:28:55.998010+0000","last_became_active":"2024-02-19T15:28:54.007566+0000","last_became_peered":"2024-02-19T15:28:54.007566+0000","last_unstale":"2024-02-19T15:28:55.998010+0000","last_undegraded":"2024-02-19T15:28:55.998010+0000","last_fullsized":"2024-02-19T15:28:55.998010+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:48.551468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.024992+0000","last_change":"2024-02-19T15:28:55.019576+0000","last_active":"2024-02-19T15:28:55.024992+0000","last_peered":"2024-02-19T15:28:55.024992+0000","last_clean":"2024-02-19T15:28:55.024992+0000","last_became_active":"2024-02-19T15:28:55.018374+0000","last_became_peered":"2024-02-19T15:28:55.018374+0000","last_unstale":"2024-02-19T15:28:55.024992+0000","last_undegraded":"2024-02-19T15:28:55.024992+0000","last_fullsized":"2024-02-19T15:28:55.024992+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:10:13.944369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.011078+0000","last_change":"2024-02-19T15:28:54.019151+0000","last_active":"2024-02-19T15:28:55.011078+0000","last_peered":"2024-02-19T15:28:55.011078+0000","last_clean":"2024-02-19T15:28:55.011078+0000","last_became_active":"2024-02-19T15:28:54.018771+0000","last_became_peered":"2024-02-19T15:28:54.018771+0000","last_unstale":"2024-02-19T15:28:55.011078+0000","last_undegraded":"2024-02-19T15:28:55.011078+0000","last_fullsized":"2024-02-19T15:28:55.011078+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:36:31.212498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.034069+0000","last_change":"2024-02-19T15:28:55.033043+0000","last_active":"2024-02-19T15:28:55.034069+0000","last_peered":"2024-02-19T15:28:55.034069+0000","last_clean":"2024-02-19T15:28:55.034069+0000","last_became_active":"2024-02-19T15:28:55.021889+0000","last_became_peered":"2024-02-19T15:28:55.021889+0000","last_unstale":"2024-02-19T15:28:55.034069+0000","last_undegraded":"2024-02-19T15:28:55.034069+0000","last_fullsized":"2024-02-19T15:28:55.034069+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:40:28.110767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.019362+0000","last_change":"2024-02-19T15:28:54.016335+0000","last_active":"2024-02-19T15:28:55.019362+0000","last_peered":"2024-02-19T15:28:55.019362+0000","last_clean":"2024-02-19T15:28:55.019362+0000","last_became_active":"2024-02-19T15:28:54.016042+0000","last_became_peered":"2024-02-19T15:28:54.016042+0000","last_unstale":"2024-02-19T15:28:55.019362+0000","last_undegraded":"2024-02-19T15:28:55.019362+0000","last_fullsized":"2024-02-19T15:28:55.019362+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:28:16.592184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997882+0000","last_change":"2024-02-19T15:28:55.015900+0000","last_active":"2024-02-19T15:28:55.997882+0000","last_peered":"2024-02-19T15:28:55.997882+0000","last_clean":"2024-02-19T15:28:55.997882+0000","last_became_active":"2024-02-19T15:28:55.015612+0000","last_became_peered":"2024-02-19T15:28:55.015612+0000","last_unstale":"2024-02-19T15:28:55.997882+0000","last_undegraded":"2024-02-19T15:28:55.997882+0000","last_fullsized":"2024-02-19T15:28:55.997882+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:35:42.119295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.018819+0000","last_change":"2024-02-19T15:28:54.013670+0000","last_active":"2024-02-19T15:28:55.018819+0000","last_peered":"2024-02-19T15:28:55.018819+0000","last_clean":"2024-02-19T15:28:55.018819+0000","last_became_active":"2024-02-19T15:28:54.008802+0000","last_became_peered":"2024-02-19T15:28:54.008802+0000","last_unstale":"2024-02-19T15:28:55.018819+0000","last_undegraded":"2024-02-19T15:28:55.018819+0000","last_fullsized":"2024-02-19T15:28:55.018819+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:14:20.149906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.030404+0000","last_change":"2024-02-19T15:28:55.028047+0000","last_active":"2024-02-19T15:28:55.030404+0000","last_peered":"2024-02-19T15:28:55.030404+0000","last_clean":"2024-02-19T15:28:55.030404+0000","last_became_active":"2024-02-19T15:28:55.027744+0000","last_became_peered":"2024-02-19T15:28:55.027744+0000","last_unstale":"2024-02-19T15:28:55.030404+0000","last_undegraded":"2024-02-19T15:28:55.030404+0000","last_fullsized":"2024-02-19T15:28:55.030404+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:59:14.276335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"124'2","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.027465+0000","last_change":"2024-02-19T15:28:54.017971+0000","last_active":"2024-02-19T15:28:55.027465+0000","last_peered":"2024-02-19T15:28:55.027465+0000","last_clean":"2024-02-19T15:28:55.027465+0000","last_became_active":"2024-02-19T15:28:54.017571+0000","last_became_peered":"2024-02-19T15:28:54.017571+0000","last_unstale":"2024-02-19T15:28:55.027465+0000","last_undegraded":"2024-02-19T15:28:55.027465+0000","last_fullsized":"2024-02-19T15:28:55.027465+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:02:54.427924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":354,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.005738+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:28:55.005738+0000","last_peered":"2024-02-19T15:28:55.005738+0000","last_clean":"2024-02-19T15:28:55.005738+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:28:55.005738+0000","last_undegraded":"2024-02-19T15:28:55.005738+0000","last_fullsized":"2024-02-19T15:28:55.005738+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":110,"acting":110,"num_store_stats":7},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":798,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":3208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":60,"acting":60,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509527,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36824,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335016,"statfs":{"total":96636764160,"available":96599056384,"internally_reserved":0,"allocated":1507328,"data_stored":652444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.434,"5min":0.373,"15min":0.373},"max":{"1min":0.81499999999999995,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.624,"15min":0.73399999999999999},"min":{"1min":0.45400000000000001,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":0.80500000000000005,"5min":2.8100000000000001,"15min":4.6029999999999998},"last":0.80500000000000005}]},{"osd":1,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.64600000000000002,"15min":0.65200000000000002},"min":{"1min":0.47199999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.91000000000000003,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.67500000000000004,"15min":0.65900000000000003},"min":{"1min":0.50700000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93000000000000005,"5min":2.774,"15min":2.774},"last":0.84999999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.39100000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.91000000000000003,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70399999999999996,"15min":0.68999999999999995},"min":{"1min":0.53300000000000003,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.95399999999999996,"5min":1.321,"15min":1.321},"last":0.78900000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.83699999999999997,"15min":0.72799999999999998},"min":{"1min":0.51100000000000001,"5min":0.46600000000000003,"15min":0.438},"max":{"1min":0.97899999999999998,"5min":14.045999999999999,"15min":14.045999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68200000000000005,"15min":0.66400000000000003},"min":{"1min":0.54000000000000004,"5min":0.433,"15min":0.433},"max":{"1min":0.85999999999999999,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.76400000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.67900000000000005,"15min":0.625},"min":{"1min":0.44800000000000001,"5min":0.32400000000000001,"15min":0.309},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.495},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.71999999999999997,"15min":0.63400000000000001},"min":{"1min":0.52700000000000002,"5min":0.35599999999999998,"15min":0.34799999999999998},"max":{"1min":1.026,"5min":2.8500000000000001,"15min":2.8500000000000001},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.69999999999999996,"15min":0.628},"min":{"1min":0.505,"5min":0.38200000000000001,"15min":0.32700000000000001},"max":{"1min":1.1830000000000001,"5min":1.363,"15min":1.363},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.75900000000000001,"15min":0.69499999999999995},"min":{"1min":0.54700000000000004,"5min":0.38,"15min":0.38},"max":{"1min":1.0589999999999999,"5min":1.411,"15min":1.411},"last":0.56999999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79200000000000004,"15min":0.69199999999999995},"min":{"1min":0.63700000000000001,"5min":0.42799999999999999,"15min":0.39600000000000002},"max":{"1min":1.0820000000000001,"5min":3.0539999999999998,"15min":3.0539999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.78500000000000003,"15min":0.70299999999999996},"min":{"1min":0.65900000000000003,"5min":0.439,"15min":0.34499999999999997},"max":{"1min":1.2190000000000001,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.70399999999999996}]}]},{"osd":7,"up_from":14,"seq":60129542232,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37452,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334388,"statfs":{"total":96636764160,"available":96598413312,"internally_reserved":0,"allocated":2162688,"data_stored":1263929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.89200000000000002,"15min":0.68799999999999994},"min":{"1min":0.41199999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.82299999999999995,"5min":15.127000000000001,"15min":15.127000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.70899999999999996,"15min":0.69099999999999995},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.93899999999999995,"5min":6.0890000000000004,"15min":6.0890000000000004},"last":0.46400000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.94699999999999995,"15min":0.70499999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.94999999999999996,"5min":15.196999999999999,"15min":15.196999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.81499999999999995,"15min":0.75800000000000001},"min":{"1min":0.44800000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.99299999999999999,"5min":10.933999999999999,"15min":10.933999999999999},"last":0.80600000000000005}]},{"osd":2,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.80000000000000004,"15min":0.68700000000000006},"min":{"1min":0.46500000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":0.874,"5min":10.84,"15min":10.84},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71099999999999997,"15min":0.69899999999999995},"min":{"1min":0.48499999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.91100000000000003,"5min":2.3170000000000002,"15min":2.508},"last":0.71899999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.86199999999999999,"15min":0.72599999999999998},"min":{"1min":0.49199999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.91500000000000004,"5min":17.094000000000001,"15min":17.094000000000001},"last":0.753},{"interface":"front","average":{"1min":0.748,"5min":0.82999999999999996,"15min":0.73499999999999999},"min":{"1min":0.54500000000000004,"5min":0.46300000000000002,"15min":0.42799999999999999},"max":{"1min":0.97499999999999998,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.59899999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.55400000000000005},"min":{"1min":0.34999999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.97099999999999997,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.97999999999999998,"15min":0.70199999999999996},"min":{"1min":0.37,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96599999999999997,"5min":26.625,"15min":26.625},"last":0.83099999999999996}]},{"osd":5,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":1.091,"15min":0.72899999999999998},"min":{"1min":0.441,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0409999999999999,"5min":31.844999999999999,"15min":31.844999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.90100000000000002,"15min":0.71299999999999997},"min":{"1min":0.54400000000000004,"5min":0.38,"15min":0.38},"max":{"1min":0.995,"5min":16.776,"15min":16.776},"last":0.86099999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.73899999999999999,"15min":0.63600000000000001},"min":{"1min":0.44800000000000001,"5min":0.375,"15min":0.36099999999999999},"max":{"1min":1.0820000000000001,"5min":1.958,"15min":1.958},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.79700000000000004,"15min":0.69899999999999995},"min":{"1min":0.51200000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0800000000000001,"5min":3.5099999999999998,"15min":3.5099999999999998},"last":0.85099999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542233,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36860,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334980,"statfs":{"total":96636764160,"available":96599019520,"internally_reserved":0,"allocated":1552384,"data_stored":671423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.69199999999999995,"15min":0.65100000000000002},"min":{"1min":0.375,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.94899999999999995,"5min":5.7789999999999999,"15min":5.7789999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.748,"15min":0.64900000000000002},"min":{"1min":0.40999999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.91700000000000004,"5min":12.359999999999999,"15min":12.359999999999999},"last":0.63900000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.77600000000000002,"15min":0.68200000000000005},"min":{"1min":0.49099999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98599999999999999,"5min":10.938000000000001,"15min":10.938000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.82999999999999996,"15min":0.72599999999999998},"min":{"1min":0.46700000000000003,"5min":0.45800000000000002,"15min":0.42599999999999999},"max":{"1min":0.95099999999999996,"5min":10.531000000000001,"15min":10.531000000000001},"last":0.60799999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.81299999999999994,"15min":0.72599999999999998},"min":{"1min":0.496,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92200000000000004,"5min":12.925000000000001,"15min":12.925000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.755,"15min":0.64900000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0029999999999999,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.73699999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.97999999999999998,"15min":0.75800000000000001},"min":{"1min":0.51200000000000001,"5min":0.49399999999999999,"15min":0.45800000000000002},"max":{"1min":1.0680000000000001,"5min":26.161999999999999,"15min":26.161999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.97499999999999998,"15min":0.76200000000000001},"min":{"1min":0.49199999999999999,"5min":0.44500000000000001,"15min":0.41399999999999998},"max":{"1min":0.86399999999999999,"5min":27.934999999999999,"15min":27.934999999999999},"last":0.66000000000000003}]},{"osd":4,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.81799999999999995,"15min":0.64000000000000001},"min":{"1min":0.45500000000000002,"5min":0.35499999999999998,"15min":0.33200000000000002},"max":{"1min":0.86299999999999999,"5min":11.73,"15min":11.73},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.80700000000000005,"15min":0.70299999999999996},"min":{"1min":0.39200000000000002,"5min":0.308,"15min":0.308},"max":{"1min":1.0169999999999999,"5min":10.731,"15min":10.731},"last":0.67200000000000004}]},{"osd":6,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.79300000000000004,"15min":0.68500000000000005},"min":{"1min":0.42899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0649999999999999,"5min":6.5229999999999997,"15min":6.5229999999999997},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.94699999999999995,"15min":0.79100000000000004},"min":{"1min":0.53000000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.091,"5min":10.432,"15min":10.432},"last":0.78600000000000003}]},{"osd":7,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.80800000000000005,"15min":0.70499999999999996},"min":{"1min":0.40300000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.099,"5min":11.986000000000001,"15min":11.986000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.84999999999999998,"15min":0.73899999999999999},"min":{"1min":0.52200000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98399999999999999,"5min":8.9770000000000003,"15min":8.9770000000000003},"last":0.76800000000000002}]}]},{"osd":4,"up_from":14,"seq":60129542231,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36856,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334984,"statfs":{"total":96636764160,"available":96599023616,"internally_reserved":0,"allocated":1548288,"data_stored":671381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.4279999999999999,"15min":0.87},"min":{"1min":0.35599999999999998,"5min":0.34399999999999997,"15min":0.33300000000000002},"max":{"1min":11.925000000000001,"5min":76.117000000000004,"15min":76.117000000000004},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.65200000000000002,"15min":0.61399999999999999},"min":{"1min":0.39100000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.95899999999999996,"5min":3.2709999999999999,"15min":3.2709999999999999},"last":0.64900000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.534,"15min":0.91700000000000004},"min":{"1min":0.443,"5min":0.437,"15min":0.39100000000000001},"max":{"1min":11.837999999999999,"5min":80.513000000000005,"15min":80.513000000000005},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.3080000000000001,"5min":0.84399999999999997,"15min":0.69999999999999996},"min":{"1min":0.41999999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":13.089,"5min":13.089,"15min":13.089},"last":0.68700000000000006}]},{"osd":2,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":0.90700000000000003,"15min":0.745},"min":{"1min":0.47399999999999998,"5min":0.436,"15min":0.436},"max":{"1min":13.255000000000001,"5min":14.362,"15min":14.362},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.587,"15min":0.97799999999999998},"min":{"1min":0.42499999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":12.372999999999999,"5min":84.093000000000004,"15min":84.093000000000004},"last":0.84199999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.6339999999999999,"15min":0.99099999999999999},"min":{"1min":0.53800000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":12.319000000000001,"5min":84.140000000000001,"15min":84.140000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66100000000000003,"15min":0.65400000000000003},"min":{"1min":0.438,"5min":0.42799999999999999,"15min":0.40500000000000003},"max":{"1min":1.0249999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.55500000000000005}]},{"osd":5,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":0.91100000000000003,"15min":0.69799999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.36499999999999999},"max":{"1min":13.292999999999999,"5min":14.318,"15min":14.318},"last":0.753},{"interface":"front","average":{"1min":1.371,"5min":1.5669999999999999,"15min":0.96899999999999997},"min":{"1min":0.60899999999999999,"5min":0.40000000000000002,"15min":0.36599999999999999},"max":{"1min":12.821,"5min":76.164000000000001,"15min":76.164000000000001},"last":0.57599999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:28:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":0.79400000000000004,"15min":0.64000000000000001},"min":{"1min":0.40300000000000002,"5min":0.33500000000000002,"15min":0.32700000000000001},"max":{"1min":15.196,"5min":15.196,"15min":15.196},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.393,"15min":0.89100000000000001},"min":{"1min":0.55400000000000005,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":13.634,"5min":60.151000000000003,"15min":60.151000000000003},"last":1.0169999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":1.006,"15min":0.77200000000000002},"min":{"1min":0.52700000000000002,"5min":0.40699999999999997,"15min":0.38400000000000001},"max":{"1min":1.6080000000000001,"5min":29.036000000000001,"15min":29.036000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":0.95599999999999996,"15min":0.80000000000000004},"min":{"1min":0.55500000000000005,"5min":0.46999999999999997,"15min":0.443},"max":{"1min":13.519,"5min":13.519,"15min":13.519},"last":0.81799999999999995}]}]},{"osd":3,"up_from":14,"seq":60129542231,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41020,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330820,"statfs":{"total":96636764160,"available":96594759680,"internally_reserved":0,"allocated":1552384,"data_stored":671471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.41599999999999998,"15min":0.443},"min":{"1min":0.25700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.68400000000000005,"5min":0.82699999999999996,"15min":0.97299999999999998},"last":0.75},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.44400000000000001,"15min":0.61399999999999999},"min":{"1min":0.27500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52400000000000002,"5min":1.0840000000000001,"15min":4.2450000000000001},"last":0.33300000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.443,"15min":0.44500000000000001},"min":{"1min":0.27800000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.76900000000000002,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.432},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.49299999999999999,"15min":0.67500000000000004},"min":{"1min":0.34399999999999997,"5min":0.31,"15min":0.31},"max":{"1min":0.73199999999999998,"5min":0.85099999999999998,"15min":4.0220000000000002},"last":0.37}]},{"osd":2,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.53800000000000003,"15min":0.60199999999999998},"min":{"1min":0.38700000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72899999999999998,"5min":1.24,"15min":1.24},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52300000000000002,"15min":0.57999999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0169999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.47199999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.746,"15min":0.69299999999999995},"min":{"1min":0.621,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.73799999999999999,"15min":0.83999999999999997},"min":{"1min":0.48699999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":4.2039999999999997},"last":0.54600000000000004}]},{"osd":5,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.75,"15min":0.83799999999999997},"min":{"1min":0.51400000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.252,"5min":1.252,"15min":4.1589999999999998},"last":0.63},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.749,"15min":0.75},"min":{"1min":0.502,"5min":0.46999999999999997,"15min":0.46800000000000003},"max":{"1min":1.081,"5min":1.081,"15min":1.3680000000000001},"last":0.67900000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.77300000000000002,"15min":0.873},"min":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.052,"5min":1.3180000000000001,"15min":4.0940000000000003},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.77100000000000002,"15min":0.80900000000000005},"min":{"1min":0.55300000000000005,"5min":0.51800000000000002,"15min":0.502},"max":{"1min":1.0569999999999999,"5min":1.206,"15min":1.9550000000000001},"last":0.61499999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.76200000000000001,"15min":0.77900000000000003},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.504},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.6579999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.77500000000000002,"15min":0.89800000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.0169999999999999,"5min":1.1919999999999999,"15min":3.9660000000000002},"last":0.70699999999999996}]}]},{"osd":2,"up_from":14,"seq":60129542232,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36860,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334980,"statfs":{"total":96636764160,"available":96599019520,"internally_reserved":0,"allocated":1552384,"data_stored":671927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.53600000000000003,"15min":0.65300000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.57499999999999996,"5min":15.42,"15min":15.42},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.50700000000000001,"15min":0.81599999999999995},"min":{"1min":0.26500000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.72899999999999998,"5min":6.7000000000000002,"15min":8.2129999999999992},"last":0.46999999999999997}]},{"osd":1,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.55100000000000005,"15min":0.82899999999999996},"min":{"1min":0.27500000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.61099999999999999,"5min":9.5269999999999992,"15min":9.5269999999999992},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.73699999999999999,"15min":0.93799999999999994},"min":{"1min":0.39800000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.755,"5min":22.954999999999998,"15min":22.954999999999998},"last":0.56200000000000006}]},{"osd":3,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.55000000000000004,"15min":0.74299999999999999},"min":{"1min":0.33100000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81399999999999995,"5min":3.7799999999999998,"15min":4.7679999999999998},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.628,"15min":0.78400000000000003},"min":{"1min":0.44700000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.73499999999999999,"5min":7.734,"15min":7.734},"last":0.58999999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.83199999999999996,"15min":0.876},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.04,"5min":15.369,"15min":15.369},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73899999999999999,"15min":0.86299999999999999},"min":{"1min":0.58999999999999997,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":3.6819999999999999,"15min":4.3440000000000003},"last":0.97599999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.80500000000000005,"15min":0.86699999999999999},"min":{"1min":0.57799999999999996,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0269999999999999,"5min":7.6520000000000001,"15min":7.6520000000000001},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.90500000000000003,"15min":0.94699999999999995},"min":{"1min":0.66400000000000003,"5min":0.496,"15min":0.47399999999999998},"max":{"1min":1.161,"5min":15.391,"15min":15.391},"last":1.0720000000000001}]},{"osd":6,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.877,"15min":0.91700000000000004},"min":{"1min":0.55500000000000005,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.22,"5min":15.337,"15min":15.337},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77000000000000002,"15min":0.92300000000000004},"min":{"1min":0.61699999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0580000000000001,"5min":3.7450000000000001,"15min":4.4320000000000004},"last":1.127}]},{"osd":7,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.81799999999999995,"15min":0.95199999999999996},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.52600000000000002},"max":{"1min":1.1850000000000001,"5min":3.71,"15min":4.4829999999999997},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.82699999999999996,"15min":0.92300000000000004},"min":{"1min":0.58099999999999996,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.079,"5min":7.6929999999999996,"15min":7.6929999999999996},"last":1.01}]}]},{"osd":1,"up_from":14,"seq":60129542233,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41024,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330816,"statfs":{"total":96636764160,"available":96594755584,"internally_reserved":0,"allocated":1556480,"data_stored":677611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.39500000000000002,"15min":0.42499999999999999},"min":{"1min":0.29199999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.438,"15min":0.46600000000000003},"min":{"1min":0.33200000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.315}]},{"osd":2,"last update":"Mon Feb 19 15:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.434,"15min":0.44700000000000001},"min":{"1min":0.34200000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.48099999999999998,"15min":0.52400000000000002},"min":{"1min":0.38400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.33400000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.57699999999999996,"15min":0.629},"min":{"1min":0.42599999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.365},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.54100000000000004,"15min":0.61899999999999999},"min":{"1min":0.48299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81100000000000005,"5min":1.0069999999999999,"15min":1.4350000000000001},"last":0.47599999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.70299999999999996,"15min":0.629},"min":{"1min":0.49199999999999999,"5min":0.434,"15min":0.377},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.72399999999999998,"15min":0.68999999999999995},"min":{"1min":0.54300000000000004,"5min":0.435,"15min":0.435},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.64200000000000002}]},{"osd":5,"last update":"Mon Feb 19 15:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.72299999999999998,"15min":0.70199999999999996},"min":{"1min":0.48599999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.2849999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.748,"5min":0.72299999999999998,"15min":0.69099999999999995},"min":{"1min":0.51600000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.77400000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.754,"15min":0.73599999999999999},"min":{"1min":0.57899999999999996,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3460000000000001},"last":0.747},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.77500000000000002,"15min":0.75600000000000001},"min":{"1min":0.65400000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4710000000000001},"last":0.67100000000000004}]},{"osd":7,"last update":"Mon Feb 19 15:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.77700000000000002,"15min":0.78900000000000003},"min":{"1min":0.57799999999999996,"5min":0.496,"15min":0.496},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.4930000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.79000000000000004,"15min":0.75600000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":0.81899999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542231,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41592,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330248,"statfs":{"total":96636764160,"available":96594173952,"internally_reserved":0,"allocated":2138112,"data_stored":1261679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:28:00 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.39100000000000001,"15min":0.58899999999999997},"min":{"1min":0.23499999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.52600000000000002,"5min":0.745,"15min":5.0199999999999996},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45000000000000001,"15min":0.47899999999999998},"min":{"1min":0.26200000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.61399999999999999,"5min":0.77300000000000002,"15min":1.2050000000000001},"last":0.60899999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:28:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53400000000000003,"15min":0.74399999999999999},"min":{"1min":0.39000000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.64000000000000001,"5min":0.82399999999999995,"15min":5.6669999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.497,"15min":0.70499999999999996},"min":{"1min":0.40600000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69899999999999995,"5min":0.85899999999999999,"15min":5.4649999999999999},"last":0.504}]},{"osd":3,"last update":"Mon Feb 19 15:28:00 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.46400000000000002,"15min":0.64400000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.81799999999999995,"5min":0.93799999999999994,"15min":4.9539999999999997},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.504,"15min":0.59599999999999997},"min":{"1min":0.36899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76200000000000001,"5min":1.052,"15min":1.345},"last":0.64700000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:28:00 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.73199999999999998,"15min":0.73799999999999999},"min":{"1min":0.505,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.97899999999999998,"5min":1.091,"15min":1.095},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.74399999999999999,"15min":0.89200000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.99199999999999999,"5min":1.085,"15min":4.7880000000000003},"last":0.627}]},{"osd":5,"last update":"Mon Feb 19 15:28:00 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.70499999999999996,"15min":0.73999999999999999},"min":{"1min":0.47299999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.95299999999999996,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.69499999999999995,"15min":0.82699999999999996},"min":{"1min":0.5,"5min":0.5,"15min":0.46100000000000002},"max":{"1min":0.89500000000000002,"5min":1.169,"15min":4.8810000000000002},"last":0.67400000000000004}]},{"osd":6,"last update":"Mon Feb 19 15:28:00 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71499999999999997,"15min":0.88},"min":{"1min":0.498,"5min":0.439,"15min":0.439},"max":{"1min":0.96199999999999997,"5min":1.1200000000000001,"15min":4.8460000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71299999999999997,"15min":0.749},"min":{"1min":0.58199999999999996,"5min":0.50800000000000001,"15min":0.49399999999999999},"max":{"1min":0.93100000000000005,"5min":1.109,"15min":1.109},"last":0.68799999999999994}]},{"osd":7,"last update":"Mon Feb 19 15:28:00 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.90700000000000003},"min":{"1min":0.52800000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.93000000000000005,"5min":1.2450000000000001,"15min":4.9809999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.72999999999999998,"15min":0.90600000000000003},"min":{"1min":0.55200000000000005,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.91800000000000004,"5min":1.149,"15min":4.915},"last":0.74199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:28:59.174 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:28:59.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.354+0000 7f32b4a00640 1 Processor -- start 2024-02-19T15:28:59.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.354+0000 7f32b4a00640 1 -- start start 2024-02-19T15:28:59.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.354+0000 7f32b4a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f32b0162830 0x7f32b015e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:59.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.354+0000 7f32b4a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b00acf30 con 0x7f32b015dd10 2024-02-19T15:28:59.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.354+0000 7f32b4a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b00cea90 con 0x7f32b006b8c0 2024-02-19T15:28:59.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.354+0000 7f32b4a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b00ad180 con 0x7f32b0162830 2024-02-19T15:28:59.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.354+0000 7f32ade00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f32b0162830 0x7f32b015e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:59.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.354+0000 7f32ae800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f32b015dd10 0x7f32b00adf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41168/0 (socket says 172.21.15.47:41168) 2024-02-19T15:28:59.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.354+0000 7f32ae800640 1 -- 172.21.15.47:0/996185508 learned_addr learned my addr 172.21.15.47:0/996185508 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:28:59.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.354+0000 7f32ade00640 1 -- 172.21.15.47:0/996185508 >> v1:172.21.15.116:6789/0 conn(0x7f32b006b8c0 legacy=0x7f32b00cdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:59.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.354+0000 7f32ade00640 1 -- 172.21.15.47:0/996185508 >> v1:172.21.15.47:6789/0 conn(0x7f32b015dd10 legacy=0x7f32b00adf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:59.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.354+0000 7f32ad400640 1 -- 172.21.15.47:0/996185508 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438274940 0 0) 0x7f32b00acf30 con 0x7f32b015dd10 2024-02-19T15:28:59.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.354+0000 7f32ade00640 1 -- 172.21.15.47:0/996185508 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b00ad460 con 0x7f32b0162830 2024-02-19T15:28:59.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.358+0000 7f32ade00640 1 --2- 172.21.15.47:0/996185508 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32b0162830 0x7f32b015e0f0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f329c0097c0 tx=0x7f329c01c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f263edf333afd1f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:59.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.358+0000 7f32ad400640 1 -- 172.21.15.47:0/996185508 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c0ac640 con 0x7f32b0162830 2024-02-19T15:28:59.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.358+0000 7f32ad400640 1 -- 172.21.15.47:0/996185508 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329c0ac7a0 con 0x7f32b0162830 2024-02-19T15:28:59.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.358+0000 7f32ad400640 1 -- 172.21.15.47:0/996185508 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c0acb30 con 0x7f32b0162830 2024-02-19T15:28:59.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.358+0000 7f32b4a00640 1 -- 172.21.15.47:0/996185508 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32b0162830 msgr2=0x7f32b015e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:59.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.358+0000 7f32b4a00640 1 --2- 172.21.15.47:0/996185508 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32b0162830 0x7f32b015e0f0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f329c0097c0 tx=0x7f329c01c310 comp rx=0 tx=0).stop 2024-02-19T15:28:59.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.358+0000 7f32b4a00640 1 -- 172.21.15.47:0/996185508 shutdown_connections 2024-02-19T15:28:59.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.358+0000 7f32b4a00640 1 --2- 172.21.15.47:0/996185508 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32b0162830 0x7f32b015e0f0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.358+0000 7f32b4a00640 1 -- 172.21.15.47:0/996185508 >> 172.21.15.47:0/996185508 conn(0x7f32b00d0080 msgr2=0x7f32b00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:59.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.358+0000 7f32b4a00640 1 -- 172.21.15.47:0/996185508 shutdown_connections 2024-02-19T15:28:59.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.358+0000 7f32b4a00640 1 -- 172.21.15.47:0/996185508 wait complete. 2024-02-19T15:28:59.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32b4a00640 1 Processor -- start 2024-02-19T15:28:59.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32b4a00640 1 -- start start 2024-02-19T15:28:59.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32b4a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32b006b8c0 0x7f32b009b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:59.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32b4a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32b015dd10 0x7f32b009c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:59.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32b4a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32b0162830 0x7f32b015cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:59.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32b4a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b009c590 con 0x7f32b0162830 2024-02-19T15:28:59.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32b4a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b009c7c0 con 0x7f32b006b8c0 2024-02-19T15:28:59.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32b4a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b009caa0 con 0x7f32b015dd10 2024-02-19T15:28:59.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32af200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32b0162830 0x7f32b015cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:59.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32af200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32b0162830 0x7f32b015cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51134/0 (socket says 172.21.15.47:51134) 2024-02-19T15:28:59.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32af200640 1 -- 172.21.15.47:0/916340526 learned_addr learned my addr 172.21.15.47:0/916340526 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:59.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32ae800640 1 --2- 172.21.15.47:0/916340526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32b006b8c0 0x7f32b009b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:59.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32ade00640 1 --2- 172.21.15.47:0/916340526 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32b015dd10 0x7f32b009c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:59.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32af200640 1 -- 172.21.15.47:0/916340526 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32b015dd10 msgr2=0x7f32b009c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:59.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32af200640 1 --2- 172.21.15.47:0/916340526 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32b015dd10 0x7f32b009c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32af200640 1 -- 172.21.15.47:0/916340526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32b006b8c0 msgr2=0x7f32b009b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:59.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32af200640 1 --2- 172.21.15.47:0/916340526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32b006b8c0 0x7f32b009b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32af200640 1 -- 172.21.15.47:0/916340526 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329c009420 con 0x7f32b0162830 2024-02-19T15:28:59.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32af200640 1 --2- 172.21.15.47:0/916340526 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32b0162830 0x7f32b015cb20 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f32a800b970 tx=0x7f32a800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:59.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32a6a00640 1 -- 172.21.15.47:0/916340526 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a800cb40 con 0x7f32b0162830 2024-02-19T15:28:59.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32a6a00640 1 -- 172.21.15.47:0/916340526 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a800cca0 con 0x7f32b0162830 2024-02-19T15:28:59.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32a6a00640 1 -- 172.21.15.47:0/916340526 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a8014680 con 0x7f32b0162830 2024-02-19T15:28:59.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.362+0000 7f32b4a00640 1 -- 172.21.15.47:0/916340526 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b009cd80 con 0x7f32b0162830 2024-02-19T15:28:59.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.366+0000 7f32b4a00640 1 -- 172.21.15.47:0/916340526 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b009d340 con 0x7f32b0162830 2024-02-19T15:28:59.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.366+0000 7f32b4a00640 1 -- 172.21.15.47:0/916340526 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b009d530 con 0x7f32b0162830 2024-02-19T15:28:59.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.366+0000 7f32a6a00640 1 -- 172.21.15.47:0/916340526 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f32a801d3b0 con 0x7f32b0162830 2024-02-19T15:28:59.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.366+0000 7f32a6a00640 1 --2- 172.21.15.47:0/916340526 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3290066e40 0x7f3290069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:59.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.366+0000 7f32a6a00640 1 -- 172.21.15.47:0/916340526 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32a8084870 con 0x7f32b0162830 2024-02-19T15:28:59.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.366+0000 7f32ae800640 1 --2- 172.21.15.47:0/916340526 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3290066e40 0x7f3290069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:59.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.366+0000 7f32ae800640 1 --2- 172.21.15.47:0/916340526 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3290066e40 0x7f3290069300 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f32a0007fd0 tx=0x7f32a0007d60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:59.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.370+0000 7f32a6a00640 1 -- 172.21.15.47:0/916340526 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f32a8051670 con 0x7f32b0162830 2024-02-19T15:28:59.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.530+0000 7f32b4a00640 1 -- 172.21.15.47:0/916340526 --> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f32b00b5d00 con 0x7f3290066e40 2024-02-19T15:28:59.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.538+0000 7f32a6a00640 1 -- 172.21.15.47:0/916340526 <== mgr.4103 v2:172.21.15.116:6832/2412897884 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289150 (secure 0 0 0) 0x7f32b00b5d00 con 0x7f3290066e40 2024-02-19T15:28:59.540 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:28:59.542 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-02-19T15:28:59.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.542+0000 7f32b4a00640 1 -- 172.21.15.47:0/916340526 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3290066e40 msgr2=0x7f3290069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:59.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.542+0000 7f32b4a00640 1 --2- 172.21.15.47:0/916340526 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3290066e40 0x7f3290069300 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f32a0007fd0 tx=0x7f32a0007d60 comp rx=0 tx=0).stop 2024-02-19T15:28:59.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.542+0000 7f32b4a00640 1 -- 172.21.15.47:0/916340526 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32b0162830 msgr2=0x7f32b015cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:59.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.542+0000 7f32b4a00640 1 --2- 172.21.15.47:0/916340526 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32b0162830 0x7f32b015cb20 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f32a800b970 tx=0x7f32a800be30 comp rx=0 tx=0).stop 2024-02-19T15:28:59.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.542+0000 7f32b4a00640 1 -- 172.21.15.47:0/916340526 shutdown_connections 2024-02-19T15:28:59.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.542+0000 7f32b4a00640 1 --2- 172.21.15.47:0/916340526 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3290066e40 0x7f3290069300 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.542+0000 7f32b4a00640 1 --2- 172.21.15.47:0/916340526 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32b0162830 0x7f32b015cb20 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.542+0000 7f32b4a00640 1 --2- 172.21.15.47:0/916340526 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32b015dd10 0x7f32b009c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.542+0000 7f32b4a00640 1 --2- 172.21.15.47:0/916340526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32b006b8c0 0x7f32b009b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.542+0000 7f32b4a00640 1 -- 172.21.15.47:0/916340526 >> 172.21.15.47:0/916340526 conn(0x7f32b00d0080 msgr2=0x7f32b00b4c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:59.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.542+0000 7f32b4a00640 1 -- 172.21.15.47:0/916340526 shutdown_connections 2024-02-19T15:28:59.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.542+0000 7f32b4a00640 1 -- 172.21.15.47:0/916340526 wait complete. 2024-02-19T15:28:59.561 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-02-19T15:28:59.246867+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":117,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":308540,"kb_used_data":13312,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754666180,"statfs":{"total":773094113280,"available":772778168320,"internally_reserved":0,"allocated":13631488,"data_stored":6568644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":302173746},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001441"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997573+0000","last_change":"2024-02-19T15:28:55.012834+0000","last_active":"2024-02-19T15:28:55.997573+0000","last_peered":"2024-02-19T15:28:55.997573+0000","last_clean":"2024-02-19T15:28:55.997573+0000","last_became_active":"2024-02-19T15:28:55.012200+0000","last_became_peered":"2024-02-19T15:28:55.012200+0000","last_unstale":"2024-02-19T15:28:55.997573+0000","last_undegraded":"2024-02-19T15:28:55.997573+0000","last_fullsized":"2024-02-19T15:28:55.997573+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:22:03.698617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.998347+0000","last_change":"2024-02-19T15:28:55.011248+0000","last_active":"2024-02-19T15:28:55.998347+0000","last_peered":"2024-02-19T15:28:55.998347+0000","last_clean":"2024-02-19T15:28:55.998347+0000","last_became_active":"2024-02-19T15:28:55.010739+0000","last_became_peered":"2024-02-19T15:28:55.010739+0000","last_unstale":"2024-02-19T15:28:55.998347+0000","last_undegraded":"2024-02-19T15:28:55.998347+0000","last_fullsized":"2024-02-19T15:28:55.998347+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:06:35.962884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997704+0000","last_change":"2024-02-19T15:28:55.024721+0000","last_active":"2024-02-19T15:28:55.997704+0000","last_peered":"2024-02-19T15:28:55.997704+0000","last_clean":"2024-02-19T15:28:55.997704+0000","last_became_active":"2024-02-19T15:28:55.024450+0000","last_became_peered":"2024-02-19T15:28:55.024450+0000","last_unstale":"2024-02-19T15:28:55.997704+0000","last_undegraded":"2024-02-19T15:28:55.997704+0000","last_fullsized":"2024-02-19T15:28:55.997704+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:56:36.059710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.996941+0000","last_change":"2024-02-19T15:28:55.016650+0000","last_active":"2024-02-19T15:28:55.996941+0000","last_peered":"2024-02-19T15:28:55.996941+0000","last_clean":"2024-02-19T15:28:55.996941+0000","last_became_active":"2024-02-19T15:28:55.016324+0000","last_became_peered":"2024-02-19T15:28:55.016324+0000","last_unstale":"2024-02-19T15:28:55.996941+0000","last_undegraded":"2024-02-19T15:28:55.996941+0000","last_fullsized":"2024-02-19T15:28:55.996941+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:55:03.720594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.024398+0000","last_change":"2024-02-19T15:28:55.020595+0000","last_active":"2024-02-19T15:28:55.024398+0000","last_peered":"2024-02-19T15:28:55.024398+0000","last_clean":"2024-02-19T15:28:55.024398+0000","last_became_active":"2024-02-19T15:28:55.020240+0000","last_became_peered":"2024-02-19T15:28:55.020240+0000","last_unstale":"2024-02-19T15:28:55.024398+0000","last_undegraded":"2024-02-19T15:28:55.024398+0000","last_fullsized":"2024-02-19T15:28:55.024398+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:15.873342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:56.195924+0000","last_change":"2024-02-19T15:28:55.020804+0000","last_active":"2024-02-19T15:28:56.195924+0000","last_peered":"2024-02-19T15:28:56.195924+0000","last_clean":"2024-02-19T15:28:56.195924+0000","last_became_active":"2024-02-19T15:28:55.019838+0000","last_became_peered":"2024-02-19T15:28:55.019838+0000","last_unstale":"2024-02-19T15:28:56.195924+0000","last_undegraded":"2024-02-19T15:28:56.195924+0000","last_fullsized":"2024-02-19T15:28:56.195924+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:59:45.677859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:56.195670+0000","last_change":"2024-02-19T15:28:55.016093+0000","last_active":"2024-02-19T15:28:56.195670+0000","last_peered":"2024-02-19T15:28:56.195670+0000","last_clean":"2024-02-19T15:28:56.195670+0000","last_became_active":"2024-02-19T15:28:55.015790+0000","last_became_peered":"2024-02-19T15:28:55.015790+0000","last_unstale":"2024-02-19T15:28:56.195670+0000","last_undegraded":"2024-02-19T15:28:56.195670+0000","last_fullsized":"2024-02-19T15:28:56.195670+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:06:35.962884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997696+0000","last_change":"2024-02-19T15:28:55.022019+0000","last_active":"2024-02-19T15:28:55.997696+0000","last_peered":"2024-02-19T15:28:55.997696+0000","last_clean":"2024-02-19T15:28:55.997696+0000","last_became_active":"2024-02-19T15:28:55.021354+0000","last_became_peered":"2024-02-19T15:28:55.021354+0000","last_unstale":"2024-02-19T15:28:55.997696+0000","last_undegraded":"2024-02-19T15:28:55.997696+0000","last_fullsized":"2024-02-19T15:28:55.997696+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:25:51.251513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997570+0000","last_change":"2024-02-19T15:28:55.010536+0000","last_active":"2024-02-19T15:28:55.997570+0000","last_peered":"2024-02-19T15:28:55.997570+0000","last_clean":"2024-02-19T15:28:55.997570+0000","last_became_active":"2024-02-19T15:28:55.010285+0000","last_became_peered":"2024-02-19T15:28:55.010285+0000","last_unstale":"2024-02-19T15:28:55.997570+0000","last_undegraded":"2024-02-19T15:28:55.997570+0000","last_fullsized":"2024-02-19T15:28:55.997570+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:31:21.089542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.998391+0000","last_change":"2024-02-19T15:28:55.011009+0000","last_active":"2024-02-19T15:28:55.998391+0000","last_peered":"2024-02-19T15:28:55.998391+0000","last_clean":"2024-02-19T15:28:55.998391+0000","last_became_active":"2024-02-19T15:28:55.010328+0000","last_became_peered":"2024-02-19T15:28:55.010328+0000","last_unstale":"2024-02-19T15:28:55.998391+0000","last_undegraded":"2024-02-19T15:28:55.998391+0000","last_fullsized":"2024-02-19T15:28:55.998391+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:19:18.262726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.026272+0000","last_change":"2024-02-19T15:28:55.025024+0000","last_active":"2024-02-19T15:28:55.026272+0000","last_peered":"2024-02-19T15:28:55.026272+0000","last_clean":"2024-02-19T15:28:55.026272+0000","last_became_active":"2024-02-19T15:28:55.024729+0000","last_became_peered":"2024-02-19T15:28:55.024729+0000","last_unstale":"2024-02-19T15:28:55.026272+0000","last_undegraded":"2024-02-19T15:28:55.026272+0000","last_fullsized":"2024-02-19T15:28:55.026272+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:41:57.334120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997207+0000","last_change":"2024-02-19T15:28:55.008318+0000","last_active":"2024-02-19T15:28:55.997207+0000","last_peered":"2024-02-19T15:28:55.997207+0000","last_clean":"2024-02-19T15:28:55.997207+0000","last_became_active":"2024-02-19T15:28:55.007452+0000","last_became_peered":"2024-02-19T15:28:55.007452+0000","last_unstale":"2024-02-19T15:28:55.997207+0000","last_undegraded":"2024-02-19T15:28:55.997207+0000","last_fullsized":"2024-02-19T15:28:55.997207+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:54:46.772229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997044+0000","last_change":"2024-02-19T15:28:55.010569+0000","last_active":"2024-02-19T15:28:55.997044+0000","last_peered":"2024-02-19T15:28:55.997044+0000","last_clean":"2024-02-19T15:28:55.997044+0000","last_became_active":"2024-02-19T15:28:55.010227+0000","last_became_peered":"2024-02-19T15:28:55.010227+0000","last_unstale":"2024-02-19T15:28:55.997044+0000","last_undegraded":"2024-02-19T15:28:55.997044+0000","last_fullsized":"2024-02-19T15:28:55.997044+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:59:51.326518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.998048+0000","last_change":"2024-02-19T15:28:55.013533+0000","last_active":"2024-02-19T15:28:55.998048+0000","last_peered":"2024-02-19T15:28:55.998048+0000","last_clean":"2024-02-19T15:28:55.998048+0000","last_became_active":"2024-02-19T15:28:55.013257+0000","last_became_peered":"2024-02-19T15:28:55.013257+0000","last_unstale":"2024-02-19T15:28:55.998048+0000","last_undegraded":"2024-02-19T15:28:55.998048+0000","last_fullsized":"2024-02-19T15:28:55.998048+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:09:50.600613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.996797+0000","last_change":"2024-02-19T15:28:55.002804+0000","last_active":"2024-02-19T15:28:55.996797+0000","last_peered":"2024-02-19T15:28:55.996797+0000","last_clean":"2024-02-19T15:28:55.996797+0000","last_became_active":"2024-02-19T15:28:55.002525+0000","last_became_peered":"2024-02-19T15:28:55.002525+0000","last_unstale":"2024-02-19T15:28:55.996797+0000","last_undegraded":"2024-02-19T15:28:55.996797+0000","last_fullsized":"2024-02-19T15:28:55.996797+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:22.956669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.036459+0000","last_change":"2024-02-19T15:28:55.033867+0000","last_active":"2024-02-19T15:28:55.036459+0000","last_peered":"2024-02-19T15:28:55.036459+0000","last_clean":"2024-02-19T15:28:55.036459+0000","last_became_active":"2024-02-19T15:28:55.033537+0000","last_became_peered":"2024-02-19T15:28:55.033537+0000","last_unstale":"2024-02-19T15:28:55.036459+0000","last_undegraded":"2024-02-19T15:28:55.036459+0000","last_fullsized":"2024-02-19T15:28:55.036459+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:00:25.674979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:56.196162+0000","last_change":"2024-02-19T15:28:55.025350+0000","last_active":"2024-02-19T15:28:56.196162+0000","last_peered":"2024-02-19T15:28:56.196162+0000","last_clean":"2024-02-19T15:28:56.196162+0000","last_became_active":"2024-02-19T15:28:55.024881+0000","last_became_peered":"2024-02-19T15:28:55.024881+0000","last_unstale":"2024-02-19T15:28:56.196162+0000","last_undegraded":"2024-02-19T15:28:56.196162+0000","last_fullsized":"2024-02-19T15:28:56.196162+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:52:43.326080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.037341+0000","last_change":"2024-02-19T15:28:55.035357+0000","last_active":"2024-02-19T15:28:55.037341+0000","last_peered":"2024-02-19T15:28:55.037341+0000","last_clean":"2024-02-19T15:28:55.037341+0000","last_became_active":"2024-02-19T15:28:55.034624+0000","last_became_peered":"2024-02-19T15:28:55.034624+0000","last_unstale":"2024-02-19T15:28:55.037341+0000","last_undegraded":"2024-02-19T15:28:55.037341+0000","last_fullsized":"2024-02-19T15:28:55.037341+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:15.873342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.031375+0000","last_change":"2024-02-19T15:28:55.028884+0000","last_active":"2024-02-19T15:28:55.031375+0000","last_peered":"2024-02-19T15:28:55.031375+0000","last_clean":"2024-02-19T15:28:55.031375+0000","last_became_active":"2024-02-19T15:28:55.028534+0000","last_became_peered":"2024-02-19T15:28:55.028534+0000","last_unstale":"2024-02-19T15:28:55.031375+0000","last_undegraded":"2024-02-19T15:28:55.031375+0000","last_fullsized":"2024-02-19T15:28:55.031375+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:22:40.379622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.037386+0000","last_change":"2024-02-19T15:28:55.035973+0000","last_active":"2024-02-19T15:28:55.037386+0000","last_peered":"2024-02-19T15:28:55.037386+0000","last_clean":"2024-02-19T15:28:55.037386+0000","last_became_active":"2024-02-19T15:28:55.035656+0000","last_became_peered":"2024-02-19T15:28:55.035656+0000","last_unstale":"2024-02-19T15:28:55.037386+0000","last_undegraded":"2024-02-19T15:28:55.037386+0000","last_fullsized":"2024-02-19T15:28:55.037386+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:49:00.412895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.024217+0000","last_change":"2024-02-19T15:28:55.020555+0000","last_active":"2024-02-19T15:28:55.024217+0000","last_peered":"2024-02-19T15:28:55.024217+0000","last_clean":"2024-02-19T15:28:55.024217+0000","last_became_active":"2024-02-19T15:28:55.020202+0000","last_became_peered":"2024-02-19T15:28:55.020202+0000","last_unstale":"2024-02-19T15:28:55.024217+0000","last_undegraded":"2024-02-19T15:28:55.024217+0000","last_fullsized":"2024-02-19T15:28:55.024217+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:00:23.721397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.025840+0000","last_change":"2024-02-19T15:28:55.023485+0000","last_active":"2024-02-19T15:28:55.025840+0000","last_peered":"2024-02-19T15:28:55.025840+0000","last_clean":"2024-02-19T15:28:55.025840+0000","last_became_active":"2024-02-19T15:28:55.023160+0000","last_became_peered":"2024-02-19T15:28:55.023160+0000","last_unstale":"2024-02-19T15:28:55.025840+0000","last_undegraded":"2024-02-19T15:28:55.025840+0000","last_fullsized":"2024-02-19T15:28:55.025840+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:19.307747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997381+0000","last_change":"2024-02-19T15:28:55.006679+0000","last_active":"2024-02-19T15:28:55.997381+0000","last_peered":"2024-02-19T15:28:55.997381+0000","last_clean":"2024-02-19T15:28:55.997381+0000","last_became_active":"2024-02-19T15:28:55.006408+0000","last_became_peered":"2024-02-19T15:28:55.006408+0000","last_unstale":"2024-02-19T15:28:55.997381+0000","last_undegraded":"2024-02-19T15:28:55.997381+0000","last_fullsized":"2024-02-19T15:28:55.997381+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:00:12.817900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.031118+0000","last_change":"2024-02-19T15:28:55.028726+0000","last_active":"2024-02-19T15:28:55.031118+0000","last_peered":"2024-02-19T15:28:55.031118+0000","last_clean":"2024-02-19T15:28:55.031118+0000","last_became_active":"2024-02-19T15:28:55.028450+0000","last_became_peered":"2024-02-19T15:28:55.028450+0000","last_unstale":"2024-02-19T15:28:55.031118+0000","last_undegraded":"2024-02-19T15:28:55.031118+0000","last_fullsized":"2024-02-19T15:28:55.031118+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:31:18.921611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.998343+0000","last_change":"2024-02-19T15:28:55.016980+0000","last_active":"2024-02-19T15:28:55.998343+0000","last_peered":"2024-02-19T15:28:55.998343+0000","last_clean":"2024-02-19T15:28:55.998343+0000","last_became_active":"2024-02-19T15:28:55.016767+0000","last_became_peered":"2024-02-19T15:28:55.016767+0000","last_unstale":"2024-02-19T15:28:55.998343+0000","last_undegraded":"2024-02-19T15:28:55.998343+0000","last_fullsized":"2024-02-19T15:28:55.998343+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:32.057879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.030714+0000","last_change":"2024-02-19T15:28:55.020698+0000","last_active":"2024-02-19T15:28:55.030714+0000","last_peered":"2024-02-19T15:28:55.030714+0000","last_clean":"2024-02-19T15:28:55.030714+0000","last_became_active":"2024-02-19T15:28:55.020444+0000","last_became_peered":"2024-02-19T15:28:55.020444+0000","last_unstale":"2024-02-19T15:28:55.030714+0000","last_undegraded":"2024-02-19T15:28:55.030714+0000","last_fullsized":"2024-02-19T15:28:55.030714+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:02:55.430828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.996973+0000","last_change":"2024-02-19T15:28:55.010867+0000","last_active":"2024-02-19T15:28:55.996973+0000","last_peered":"2024-02-19T15:28:55.996973+0000","last_clean":"2024-02-19T15:28:55.996973+0000","last_became_active":"2024-02-19T15:28:55.010605+0000","last_became_peered":"2024-02-19T15:28:55.010605+0000","last_unstale":"2024-02-19T15:28:55.996973+0000","last_undegraded":"2024-02-19T15:28:55.996973+0000","last_fullsized":"2024-02-19T15:28:55.996973+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:41:07.665818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.030693+0000","last_change":"2024-02-19T15:28:55.028834+0000","last_active":"2024-02-19T15:28:55.030693+0000","last_peered":"2024-02-19T15:28:55.030693+0000","last_clean":"2024-02-19T15:28:55.030693+0000","last_became_active":"2024-02-19T15:28:55.028547+0000","last_became_peered":"2024-02-19T15:28:55.028547+0000","last_unstale":"2024-02-19T15:28:55.030693+0000","last_undegraded":"2024-02-19T15:28:55.030693+0000","last_fullsized":"2024-02-19T15:28:55.030693+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:24.706275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997214+0000","last_change":"2024-02-19T15:28:55.015157+0000","last_active":"2024-02-19T15:28:55.997214+0000","last_peered":"2024-02-19T15:28:55.997214+0000","last_clean":"2024-02-19T15:28:55.997214+0000","last_became_active":"2024-02-19T15:28:55.014874+0000","last_became_peered":"2024-02-19T15:28:55.014874+0000","last_unstale":"2024-02-19T15:28:55.997214+0000","last_undegraded":"2024-02-19T15:28:55.997214+0000","last_fullsized":"2024-02-19T15:28:55.997214+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:59:22.763599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997079+0000","last_change":"2024-02-19T15:28:55.003282+0000","last_active":"2024-02-19T15:28:55.997079+0000","last_peered":"2024-02-19T15:28:55.997079+0000","last_clean":"2024-02-19T15:28:55.997079+0000","last_became_active":"2024-02-19T15:28:55.002930+0000","last_became_peered":"2024-02-19T15:28:55.002930+0000","last_unstale":"2024-02-19T15:28:55.997079+0000","last_undegraded":"2024-02-19T15:28:55.997079+0000","last_fullsized":"2024-02-19T15:28:55.997079+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:26:58.415308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.030541+0000","last_change":"2024-02-19T15:28:55.029082+0000","last_active":"2024-02-19T15:28:55.030541+0000","last_peered":"2024-02-19T15:28:55.030541+0000","last_clean":"2024-02-19T15:28:55.030541+0000","last_became_active":"2024-02-19T15:28:55.027609+0000","last_became_peered":"2024-02-19T15:28:55.027609+0000","last_unstale":"2024-02-19T15:28:55.030541+0000","last_undegraded":"2024-02-19T15:28:55.030541+0000","last_fullsized":"2024-02-19T15:28:55.030541+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:48:40.828864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:56.195760+0000","last_change":"2024-02-19T15:28:55.034276+0000","last_active":"2024-02-19T15:28:56.195760+0000","last_peered":"2024-02-19T15:28:56.195760+0000","last_clean":"2024-02-19T15:28:56.195760+0000","last_became_active":"2024-02-19T15:28:55.033935+0000","last_became_peered":"2024-02-19T15:28:55.033935+0000","last_unstale":"2024-02-19T15:28:56.195760+0000","last_undegraded":"2024-02-19T15:28:56.195760+0000","last_fullsized":"2024-02-19T15:28:56.195760+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:19:18.262726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.037364+0000","last_change":"2024-02-19T15:28:55.035854+0000","last_active":"2024-02-19T15:28:55.037364+0000","last_peered":"2024-02-19T15:28:55.037364+0000","last_clean":"2024-02-19T15:28:55.037364+0000","last_became_active":"2024-02-19T15:28:55.034504+0000","last_became_peered":"2024-02-19T15:28:55.034504+0000","last_unstale":"2024-02-19T15:28:55.037364+0000","last_undegraded":"2024-02-19T15:28:55.037364+0000","last_fullsized":"2024-02-19T15:28:55.037364+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:41:45.635707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"124'4","reported_seq":16,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.029963+0000","last_change":"2024-02-19T15:28:54.017441+0000","last_active":"2024-02-19T15:28:55.029963+0000","last_peered":"2024-02-19T15:28:55.029963+0000","last_clean":"2024-02-19T15:28:55.029963+0000","last_became_active":"2024-02-19T15:28:54.017178+0000","last_became_peered":"2024-02-19T15:28:54.017178+0000","last_unstale":"2024-02-19T15:28:55.029963+0000","last_undegraded":"2024-02-19T15:28:55.029963+0000","last_fullsized":"2024-02-19T15:28:55.029963+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:00:24.672075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997672+0000","last_change":"2024-02-19T15:28:55.017965+0000","last_active":"2024-02-19T15:28:55.997672+0000","last_peered":"2024-02-19T15:28:55.997672+0000","last_clean":"2024-02-19T15:28:55.997672+0000","last_became_active":"2024-02-19T15:28:55.017506+0000","last_became_peered":"2024-02-19T15:28:55.017506+0000","last_unstale":"2024-02-19T15:28:55.997672+0000","last_undegraded":"2024-02-19T15:28:55.997672+0000","last_fullsized":"2024-02-19T15:28:55.997672+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:02:52.662668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"124'1","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.022958+0000","last_change":"2024-02-19T15:28:54.019905+0000","last_active":"2024-02-19T15:28:55.022958+0000","last_peered":"2024-02-19T15:28:55.022958+0000","last_clean":"2024-02-19T15:28:55.022958+0000","last_became_active":"2024-02-19T15:28:54.019651+0000","last_became_peered":"2024-02-19T15:28:54.019651+0000","last_unstale":"2024-02-19T15:28:55.022958+0000","last_undegraded":"2024-02-19T15:28:55.022958+0000","last_fullsized":"2024-02-19T15:28:55.022958+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:16:40.743331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.030507+0000","last_change":"2024-02-19T15:28:55.027266+0000","last_active":"2024-02-19T15:28:55.030507+0000","last_peered":"2024-02-19T15:28:55.030507+0000","last_clean":"2024-02-19T15:28:55.030507+0000","last_became_active":"2024-02-19T15:28:55.026242+0000","last_became_peered":"2024-02-19T15:28:55.026242+0000","last_unstale":"2024-02-19T15:28:55.030507+0000","last_undegraded":"2024-02-19T15:28:55.030507+0000","last_fullsized":"2024-02-19T15:28:55.030507+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:00:22.232665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"124'2","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.031992+0000","last_change":"2024-02-19T15:28:54.015943+0000","last_active":"2024-02-19T15:28:55.031992+0000","last_peered":"2024-02-19T15:28:55.031992+0000","last_clean":"2024-02-19T15:28:55.031992+0000","last_became_active":"2024-02-19T15:28:54.015679+0000","last_became_peered":"2024-02-19T15:28:54.015679+0000","last_unstale":"2024-02-19T15:28:55.031992+0000","last_undegraded":"2024-02-19T15:28:55.031992+0000","last_fullsized":"2024-02-19T15:28:55.031992+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:09:28.603113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.029786+0000","last_change":"2024-02-19T15:28:55.026500+0000","last_active":"2024-02-19T15:28:55.029786+0000","last_peered":"2024-02-19T15:28:55.029786+0000","last_clean":"2024-02-19T15:28:55.029786+0000","last_became_active":"2024-02-19T15:28:55.026152+0000","last_became_peered":"2024-02-19T15:28:55.026152+0000","last_unstale":"2024-02-19T15:28:55.029786+0000","last_undegraded":"2024-02-19T15:28:55.029786+0000","last_fullsized":"2024-02-19T15:28:55.029786+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:42:31.827963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.998268+0000","last_change":"2024-02-19T15:28:54.010287+0000","last_active":"2024-02-19T15:28:55.998268+0000","last_peered":"2024-02-19T15:28:55.998268+0000","last_clean":"2024-02-19T15:28:55.998268+0000","last_became_active":"2024-02-19T15:28:54.009959+0000","last_became_peered":"2024-02-19T15:28:54.009959+0000","last_unstale":"2024-02-19T15:28:55.998268+0000","last_undegraded":"2024-02-19T15:28:55.998268+0000","last_fullsized":"2024-02-19T15:28:55.998268+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:41:03.496395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.037295+0000","last_change":"2024-02-19T15:28:55.035147+0000","last_active":"2024-02-19T15:28:55.037295+0000","last_peered":"2024-02-19T15:28:55.037295+0000","last_clean":"2024-02-19T15:28:55.037295+0000","last_became_active":"2024-02-19T15:28:55.034641+0000","last_became_peered":"2024-02-19T15:28:55.034641+0000","last_unstale":"2024-02-19T15:28:55.037295+0000","last_undegraded":"2024-02-19T15:28:55.037295+0000","last_fullsized":"2024-02-19T15:28:55.037295+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:00:23.721397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"124'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.998017+0000","last_change":"2024-02-19T15:28:54.013352+0000","last_active":"2024-02-19T15:28:55.998017+0000","last_peered":"2024-02-19T15:28:55.998017+0000","last_clean":"2024-02-19T15:28:55.998017+0000","last_became_active":"2024-02-19T15:28:54.013001+0000","last_became_peered":"2024-02-19T15:28:54.013001+0000","last_unstale":"2024-02-19T15:28:55.998017+0000","last_undegraded":"2024-02-19T15:28:55.998017+0000","last_fullsized":"2024-02-19T15:28:55.998017+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:29:12.356834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:56.195848+0000","last_change":"2024-02-19T15:28:55.023488+0000","last_active":"2024-02-19T15:28:56.195848+0000","last_peered":"2024-02-19T15:28:56.195848+0000","last_clean":"2024-02-19T15:28:56.195848+0000","last_became_active":"2024-02-19T15:28:55.023195+0000","last_became_peered":"2024-02-19T15:28:55.023195+0000","last_unstale":"2024-02-19T15:28:56.195848+0000","last_undegraded":"2024-02-19T15:28:56.195848+0000","last_fullsized":"2024-02-19T15:28:56.195848+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:21.882175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"124'1","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.035459+0000","last_change":"2024-02-19T15:28:54.012384+0000","last_active":"2024-02-19T15:28:55.035459+0000","last_peered":"2024-02-19T15:28:55.035459+0000","last_clean":"2024-02-19T15:28:55.035459+0000","last_became_active":"2024-02-19T15:28:54.010948+0000","last_became_peered":"2024-02-19T15:28:54.010948+0000","last_unstale":"2024-02-19T15:28:55.035459+0000","last_undegraded":"2024-02-19T15:28:55.035459+0000","last_fullsized":"2024-02-19T15:28:55.035459+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:52:42.323176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997519+0000","last_change":"2024-02-19T15:28:55.022212+0000","last_active":"2024-02-19T15:28:55.997519+0000","last_peered":"2024-02-19T15:28:55.997519+0000","last_clean":"2024-02-19T15:28:55.997519+0000","last_became_active":"2024-02-19T15:28:55.021649+0000","last_became_peered":"2024-02-19T15:28:55.021649+0000","last_unstale":"2024-02-19T15:28:55.997519+0000","last_undegraded":"2024-02-19T15:28:55.997519+0000","last_fullsized":"2024-02-19T15:28:55.997519+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:48:37.030289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:56.195655+0000","last_change":"2024-02-19T15:28:54.008537+0000","last_active":"2024-02-19T15:28:56.195655+0000","last_peered":"2024-02-19T15:28:56.195655+0000","last_clean":"2024-02-19T15:28:56.195655+0000","last_became_active":"2024-02-19T15:28:54.007669+0000","last_became_peered":"2024-02-19T15:28:54.007669+0000","last_unstale":"2024-02-19T15:28:56.195655+0000","last_undegraded":"2024-02-19T15:28:56.195655+0000","last_fullsized":"2024-02-19T15:28:56.195655+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:23.703371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997865+0000","last_change":"2024-02-19T15:28:55.031153+0000","last_active":"2024-02-19T15:28:55.997865+0000","last_peered":"2024-02-19T15:28:55.997865+0000","last_clean":"2024-02-19T15:28:55.997865+0000","last_became_active":"2024-02-19T15:28:55.030410+0000","last_became_peered":"2024-02-19T15:28:55.030410+0000","last_unstale":"2024-02-19T15:28:55.997865+0000","last_undegraded":"2024-02-19T15:28:55.997865+0000","last_fullsized":"2024-02-19T15:28:55.997865+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:05:32.136340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997945+0000","last_change":"2024-02-19T15:28:54.008237+0000","last_active":"2024-02-19T15:28:55.997945+0000","last_peered":"2024-02-19T15:28:55.997945+0000","last_clean":"2024-02-19T15:28:55.997945+0000","last_became_active":"2024-02-19T15:28:54.007956+0000","last_became_peered":"2024-02-19T15:28:54.007956+0000","last_unstale":"2024-02-19T15:28:55.997945+0000","last_undegraded":"2024-02-19T15:28:55.997945+0000","last_fullsized":"2024-02-19T15:28:55.997945+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:12:21.539862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997282+0000","last_change":"2024-02-19T15:28:55.025586+0000","last_active":"2024-02-19T15:28:55.997282+0000","last_peered":"2024-02-19T15:28:55.997282+0000","last_clean":"2024-02-19T15:28:55.997282+0000","last_became_active":"2024-02-19T15:28:55.025327+0000","last_became_peered":"2024-02-19T15:28:55.025327+0000","last_unstale":"2024-02-19T15:28:55.997282+0000","last_undegraded":"2024-02-19T15:28:55.997282+0000","last_fullsized":"2024-02-19T15:28:55.997282+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:59:43.922225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.002123+0000","last_change":"2024-02-19T15:28:54.012138+0000","last_active":"2024-02-19T15:28:55.002123+0000","last_peered":"2024-02-19T15:28:55.002123+0000","last_clean":"2024-02-19T15:28:55.002123+0000","last_became_active":"2024-02-19T15:28:54.011858+0000","last_became_peered":"2024-02-19T15:28:54.011858+0000","last_unstale":"2024-02-19T15:28:55.002123+0000","last_undegraded":"2024-02-19T15:28:55.002123+0000","last_fullsized":"2024-02-19T15:28:55.002123+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:41:03.496395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.030639+0000","last_change":"2024-02-19T15:28:55.028215+0000","last_active":"2024-02-19T15:28:55.030639+0000","last_peered":"2024-02-19T15:28:55.030639+0000","last_clean":"2024-02-19T15:28:55.030639+0000","last_became_active":"2024-02-19T15:28:55.028012+0000","last_became_peered":"2024-02-19T15:28:55.028012+0000","last_unstale":"2024-02-19T15:28:55.030639+0000","last_undegraded":"2024-02-19T15:28:55.030639+0000","last_fullsized":"2024-02-19T15:28:55.030639+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:43:09.938262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.013178+0000","last_change":"2024-02-19T15:28:54.017288+0000","last_active":"2024-02-19T15:28:55.013178+0000","last_peered":"2024-02-19T15:28:55.013178+0000","last_clean":"2024-02-19T15:28:55.013178+0000","last_became_active":"2024-02-19T15:28:54.017012+0000","last_became_peered":"2024-02-19T15:28:54.017012+0000","last_unstale":"2024-02-19T15:28:55.013178+0000","last_undegraded":"2024-02-19T15:28:55.013178+0000","last_fullsized":"2024-02-19T15:28:55.013178+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:33:21.121261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.035499+0000","last_change":"2024-02-19T15:28:55.034148+0000","last_active":"2024-02-19T15:28:55.035499+0000","last_peered":"2024-02-19T15:28:55.035499+0000","last_clean":"2024-02-19T15:28:55.035499+0000","last_became_active":"2024-02-19T15:28:55.033835+0000","last_became_peered":"2024-02-19T15:28:55.033835+0000","last_unstale":"2024-02-19T15:28:55.035499+0000","last_undegraded":"2024-02-19T15:28:55.035499+0000","last_fullsized":"2024-02-19T15:28:55.035499+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:36:13.025443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"124'3","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.036017+0000","last_change":"2024-02-19T15:28:54.017365+0000","last_active":"2024-02-19T15:28:55.036017+0000","last_peered":"2024-02-19T15:28:55.036017+0000","last_clean":"2024-02-19T15:28:55.036017+0000","last_became_active":"2024-02-19T15:28:54.017028+0000","last_became_peered":"2024-02-19T15:28:54.017028+0000","last_unstale":"2024-02-19T15:28:55.036017+0000","last_undegraded":"2024-02-19T15:28:55.036017+0000","last_fullsized":"2024-02-19T15:28:55.036017+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:52:44.486731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.026393+0000","last_change":"2024-02-19T15:28:55.020689+0000","last_active":"2024-02-19T15:28:55.026393+0000","last_peered":"2024-02-19T15:28:55.026393+0000","last_clean":"2024-02-19T15:28:55.026393+0000","last_became_active":"2024-02-19T15:28:55.020369+0000","last_became_peered":"2024-02-19T15:28:55.020369+0000","last_unstale":"2024-02-19T15:28:55.026393+0000","last_undegraded":"2024-02-19T15:28:55.026393+0000","last_fullsized":"2024-02-19T15:28:55.026393+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:41:45.635707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"124'6","reported_seq":20,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997343+0000","last_change":"2024-02-19T15:28:54.009480+0000","last_active":"2024-02-19T15:28:55.997343+0000","last_peered":"2024-02-19T15:28:55.997343+0000","last_clean":"2024-02-19T15:28:55.997343+0000","last_became_active":"2024-02-19T15:28:54.009170+0000","last_became_peered":"2024-02-19T15:28:54.009170+0000","last_unstale":"2024-02-19T15:28:55.997343+0000","last_undegraded":"2024-02-19T15:28:55.997343+0000","last_fullsized":"2024-02-19T15:28:55.997343+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:13:18.324075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997370+0000","last_change":"2024-02-19T15:28:55.025895+0000","last_active":"2024-02-19T15:28:55.997370+0000","last_peered":"2024-02-19T15:28:55.997370+0000","last_clean":"2024-02-19T15:28:55.997370+0000","last_became_active":"2024-02-19T15:28:55.025602+0000","last_became_peered":"2024-02-19T15:28:55.025602+0000","last_unstale":"2024-02-19T15:28:55.997370+0000","last_undegraded":"2024-02-19T15:28:55.997370+0000","last_fullsized":"2024-02-19T15:28:55.997370+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:12:11.330108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997194+0000","last_change":"2024-02-19T15:28:54.015919+0000","last_active":"2024-02-19T15:28:55.997194+0000","last_peered":"2024-02-19T15:28:55.997194+0000","last_clean":"2024-02-19T15:28:55.997194+0000","last_became_active":"2024-02-19T15:28:54.015028+0000","last_became_peered":"2024-02-19T15:28:54.015028+0000","last_unstale":"2024-02-19T15:28:55.997194+0000","last_undegraded":"2024-02-19T15:28:55.997194+0000","last_fullsized":"2024-02-19T15:28:55.997194+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:51:24.273929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997839+0000","last_change":"2024-02-19T15:28:55.008077+0000","last_active":"2024-02-19T15:28:55.997839+0000","last_peered":"2024-02-19T15:28:55.997839+0000","last_clean":"2024-02-19T15:28:55.997839+0000","last_became_active":"2024-02-19T15:28:55.005108+0000","last_became_peered":"2024-02-19T15:28:55.005108+0000","last_unstale":"2024-02-19T15:28:55.997839+0000","last_undegraded":"2024-02-19T15:28:55.997839+0000","last_fullsized":"2024-02-19T15:28:55.997839+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:04:44.776418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997962+0000","last_change":"2024-02-19T15:28:54.009514+0000","last_active":"2024-02-19T15:28:55.997962+0000","last_peered":"2024-02-19T15:28:55.997962+0000","last_clean":"2024-02-19T15:28:55.997962+0000","last_became_active":"2024-02-19T15:28:54.009225+0000","last_became_peered":"2024-02-19T15:28:54.009225+0000","last_unstale":"2024-02-19T15:28:55.997962+0000","last_undegraded":"2024-02-19T15:28:55.997962+0000","last_fullsized":"2024-02-19T15:28:55.997962+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:32:03.071994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997117+0000","last_change":"2024-02-19T15:28:55.007546+0000","last_active":"2024-02-19T15:28:55.997117+0000","last_peered":"2024-02-19T15:28:55.997117+0000","last_clean":"2024-02-19T15:28:55.997117+0000","last_became_active":"2024-02-19T15:28:55.007245+0000","last_became_peered":"2024-02-19T15:28:55.007245+0000","last_unstale":"2024-02-19T15:28:55.997117+0000","last_undegraded":"2024-02-19T15:28:55.997117+0000","last_fullsized":"2024-02-19T15:28:55.997117+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:00:37.046088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"124'8","reported_seq":22,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997717+0000","last_change":"2024-02-19T15:28:54.006791+0000","last_active":"2024-02-19T15:28:55.997717+0000","last_peered":"2024-02-19T15:28:55.997717+0000","last_clean":"2024-02-19T15:28:55.997717+0000","last_became_active":"2024-02-19T15:28:54.006500+0000","last_became_peered":"2024-02-19T15:28:54.006500+0000","last_unstale":"2024-02-19T15:28:55.997717+0000","last_undegraded":"2024-02-19T15:28:55.997717+0000","last_fullsized":"2024-02-19T15:28:55.997717+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:06:51.585324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997413+0000","last_change":"2024-02-19T15:28:55.019436+0000","last_active":"2024-02-19T15:28:55.997413+0000","last_peered":"2024-02-19T15:28:55.997413+0000","last_clean":"2024-02-19T15:28:55.997413+0000","last_became_active":"2024-02-19T15:28:55.019135+0000","last_became_peered":"2024-02-19T15:28:55.019135+0000","last_unstale":"2024-02-19T15:28:55.997413+0000","last_undegraded":"2024-02-19T15:28:55.997413+0000","last_fullsized":"2024-02-19T15:28:55.997413+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:53:54.585217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"124'1","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.027768+0000","last_change":"2024-02-19T15:28:54.013010+0000","last_active":"2024-02-19T15:28:55.027768+0000","last_peered":"2024-02-19T15:28:55.027768+0000","last_clean":"2024-02-19T15:28:55.027768+0000","last_became_active":"2024-02-19T15:28:54.012003+0000","last_became_peered":"2024-02-19T15:28:54.012003+0000","last_unstale":"2024-02-19T15:28:55.027768+0000","last_undegraded":"2024-02-19T15:28:55.027768+0000","last_fullsized":"2024-02-19T15:28:55.027768+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:26:23.224887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.030454+0000","last_change":"2024-02-19T15:28:55.025608+0000","last_active":"2024-02-19T15:28:55.030454+0000","last_peered":"2024-02-19T15:28:55.030454+0000","last_clean":"2024-02-19T15:28:55.030454+0000","last_became_active":"2024-02-19T15:28:55.025335+0000","last_became_peered":"2024-02-19T15:28:55.025335+0000","last_unstale":"2024-02-19T15:28:55.030454+0000","last_undegraded":"2024-02-19T15:28:55.030454+0000","last_fullsized":"2024-02-19T15:28:55.030454+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:17:30.675159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"124'2","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.025318+0000","last_change":"2024-02-19T15:28:54.009778+0000","last_active":"2024-02-19T15:28:55.025318+0000","last_peered":"2024-02-19T15:28:55.025318+0000","last_clean":"2024-02-19T15:28:55.025318+0000","last_became_active":"2024-02-19T15:28:54.008046+0000","last_became_peered":"2024-02-19T15:28:54.008046+0000","last_unstale":"2024-02-19T15:28:55.025318+0000","last_undegraded":"2024-02-19T15:28:55.025318+0000","last_fullsized":"2024-02-19T15:28:55.025318+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:37:10.679182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.029547+0000","last_change":"2024-02-19T15:28:55.028307+0000","last_active":"2024-02-19T15:28:55.029547+0000","last_peered":"2024-02-19T15:28:55.029547+0000","last_clean":"2024-02-19T15:28:55.029547+0000","last_became_active":"2024-02-19T15:28:55.028027+0000","last_became_peered":"2024-02-19T15:28:55.028027+0000","last_unstale":"2024-02-19T15:28:55.029547+0000","last_undegraded":"2024-02-19T15:28:55.029547+0000","last_fullsized":"2024-02-19T15:28:55.029547+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:36:32.215403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"124'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:56.196027+0000","last_change":"2024-02-19T15:28:54.012595+0000","last_active":"2024-02-19T15:28:56.196027+0000","last_peered":"2024-02-19T15:28:56.196027+0000","last_clean":"2024-02-19T15:28:56.196027+0000","last_became_active":"2024-02-19T15:28:54.012267+0000","last_became_peered":"2024-02-19T15:28:54.012267+0000","last_unstale":"2024-02-19T15:28:56.196027+0000","last_undegraded":"2024-02-19T15:28:56.196027+0000","last_fullsized":"2024-02-19T15:28:56.196027+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:43:08.935358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.998420+0000","last_change":"2024-02-19T15:28:55.030666+0000","last_active":"2024-02-19T15:28:55.998420+0000","last_peered":"2024-02-19T15:28:55.998420+0000","last_clean":"2024-02-19T15:28:55.998420+0000","last_became_active":"2024-02-19T15:28:55.030379+0000","last_became_peered":"2024-02-19T15:28:55.030379+0000","last_unstale":"2024-02-19T15:28:55.998420+0000","last_undegraded":"2024-02-19T15:28:55.998420+0000","last_fullsized":"2024-02-19T15:28:55.998420+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:21.882175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.001878+0000","last_change":"2024-02-19T15:28:54.019286+0000","last_active":"2024-02-19T15:28:55.001878+0000","last_peered":"2024-02-19T15:28:55.001878+0000","last_clean":"2024-02-19T15:28:55.001878+0000","last_became_active":"2024-02-19T15:28:54.018926+0000","last_became_peered":"2024-02-19T15:28:54.018926+0000","last_unstale":"2024-02-19T15:28:55.001878+0000","last_undegraded":"2024-02-19T15:28:55.001878+0000","last_fullsized":"2024-02-19T15:28:55.001878+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:32:03.071994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997461+0000","last_change":"2024-02-19T15:28:55.011735+0000","last_active":"2024-02-19T15:28:55.997461+0000","last_peered":"2024-02-19T15:28:55.997461+0000","last_clean":"2024-02-19T15:28:55.997461+0000","last_became_active":"2024-02-19T15:28:55.011408+0000","last_became_peered":"2024-02-19T15:28:55.011408+0000","last_unstale":"2024-02-19T15:28:55.997461+0000","last_undegraded":"2024-02-19T15:28:55.997461+0000","last_fullsized":"2024-02-19T15:28:55.997461+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:44:43.807957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"124'2","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.024755+0000","last_change":"2024-02-19T15:28:54.014397+0000","last_active":"2024-02-19T15:28:55.024755+0000","last_peered":"2024-02-19T15:28:55.024755+0000","last_clean":"2024-02-19T15:28:55.024755+0000","last_became_active":"2024-02-19T15:28:54.013650+0000","last_became_peered":"2024-02-19T15:28:54.013650+0000","last_unstale":"2024-02-19T15:28:55.024755+0000","last_undegraded":"2024-02-19T15:28:55.024755+0000","last_fullsized":"2024-02-19T15:28:55.024755+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:48.551468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:56.195987+0000","last_change":"2024-02-19T15:28:55.019576+0000","last_active":"2024-02-19T15:28:56.195987+0000","last_peered":"2024-02-19T15:28:56.195987+0000","last_clean":"2024-02-19T15:28:56.195987+0000","last_became_active":"2024-02-19T15:28:55.019006+0000","last_became_peered":"2024-02-19T15:28:55.019006+0000","last_unstale":"2024-02-19T15:28:56.195987+0000","last_undegraded":"2024-02-19T15:28:56.195987+0000","last_fullsized":"2024-02-19T15:28:56.195987+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:54:31.300972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.012249+0000","last_change":"2024-02-19T15:28:54.016895+0000","last_active":"2024-02-19T15:28:55.012249+0000","last_peered":"2024-02-19T15:28:55.012249+0000","last_clean":"2024-02-19T15:28:55.012249+0000","last_became_active":"2024-02-19T15:28:54.016590+0000","last_became_peered":"2024-02-19T15:28:54.016590+0000","last_unstale":"2024-02-19T15:28:55.012249+0000","last_undegraded":"2024-02-19T15:28:55.012249+0000","last_fullsized":"2024-02-19T15:28:55.012249+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:27:57.216088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997845+0000","last_change":"2024-02-19T15:28:55.024467+0000","last_active":"2024-02-19T15:28:55.997845+0000","last_peered":"2024-02-19T15:28:55.997845+0000","last_clean":"2024-02-19T15:28:55.997845+0000","last_became_active":"2024-02-19T15:28:55.024130+0000","last_became_peered":"2024-02-19T15:28:55.024130+0000","last_unstale":"2024-02-19T15:28:55.997845+0000","last_undegraded":"2024-02-19T15:28:55.997845+0000","last_fullsized":"2024-02-19T15:28:55.997845+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:11:46.168966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.012646+0000","last_change":"2024-02-19T15:28:54.011800+0000","last_active":"2024-02-19T15:28:55.012646+0000","last_peered":"2024-02-19T15:28:55.012646+0000","last_clean":"2024-02-19T15:28:55.012646+0000","last_became_active":"2024-02-19T15:28:54.011499+0000","last_became_peered":"2024-02-19T15:28:54.011499+0000","last_unstale":"2024-02-19T15:28:55.012646+0000","last_undegraded":"2024-02-19T15:28:55.012646+0000","last_fullsized":"2024-02-19T15:28:55.012646+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:37:58.857193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.025990+0000","last_change":"2024-02-19T15:28:55.022411+0000","last_active":"2024-02-19T15:28:55.025990+0000","last_peered":"2024-02-19T15:28:55.025990+0000","last_clean":"2024-02-19T15:28:55.025990+0000","last_became_active":"2024-02-19T15:28:55.022007+0000","last_became_peered":"2024-02-19T15:28:55.022007+0000","last_unstale":"2024-02-19T15:28:55.025990+0000","last_undegraded":"2024-02-19T15:28:55.025990+0000","last_fullsized":"2024-02-19T15:28:55.025990+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:49:00.412895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.018810+0000","last_change":"2024-02-19T15:28:54.009536+0000","last_active":"2024-02-19T15:28:55.018810+0000","last_peered":"2024-02-19T15:28:55.018810+0000","last_clean":"2024-02-19T15:28:55.018810+0000","last_became_active":"2024-02-19T15:28:54.009226+0000","last_became_peered":"2024-02-19T15:28:54.009226+0000","last_unstale":"2024-02-19T15:28:55.018810+0000","last_undegraded":"2024-02-19T15:28:55.018810+0000","last_fullsized":"2024-02-19T15:28:55.018810+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:28:21.679679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:56.196180+0000","last_change":"2024-02-19T15:28:55.025560+0000","last_active":"2024-02-19T15:28:56.196180+0000","last_peered":"2024-02-19T15:28:56.196180+0000","last_clean":"2024-02-19T15:28:56.196180+0000","last_became_active":"2024-02-19T15:28:55.025201+0000","last_became_peered":"2024-02-19T15:28:55.025201+0000","last_unstale":"2024-02-19T15:28:56.196180+0000","last_undegraded":"2024-02-19T15:28:56.196180+0000","last_fullsized":"2024-02-19T15:28:56.196180+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:27:58.218992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997601+0000","last_change":"2024-02-19T15:28:54.014553+0000","last_active":"2024-02-19T15:28:55.997601+0000","last_peered":"2024-02-19T15:28:55.997601+0000","last_clean":"2024-02-19T15:28:55.997601+0000","last_became_active":"2024-02-19T15:28:54.014103+0000","last_became_peered":"2024-02-19T15:28:54.014103+0000","last_unstale":"2024-02-19T15:28:55.997601+0000","last_undegraded":"2024-02-19T15:28:55.997601+0000","last_fullsized":"2024-02-19T15:28:55.997601+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:10:19.992342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997331+0000","last_change":"2024-02-19T15:28:55.013057+0000","last_active":"2024-02-19T15:28:55.997331+0000","last_peered":"2024-02-19T15:28:55.997331+0000","last_clean":"2024-02-19T15:28:55.997331+0000","last_became_active":"2024-02-19T15:28:55.012566+0000","last_became_peered":"2024-02-19T15:28:55.012566+0000","last_unstale":"2024-02-19T15:28:55.997331+0000","last_undegraded":"2024-02-19T15:28:55.997331+0000","last_fullsized":"2024-02-19T15:28:55.997331+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T03:25:39.345932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.011366+0000","last_change":"2024-02-19T15:28:54.015239+0000","last_active":"2024-02-19T15:28:55.011366+0000","last_peered":"2024-02-19T15:28:55.011366+0000","last_clean":"2024-02-19T15:28:55.011366+0000","last_became_active":"2024-02-19T15:28:54.014910+0000","last_became_peered":"2024-02-19T15:28:54.014910+0000","last_unstale":"2024-02-19T15:28:55.011366+0000","last_undegraded":"2024-02-19T15:28:55.011366+0000","last_fullsized":"2024-02-19T15:28:55.011366+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:40:27.107863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.998291+0000","last_change":"2024-02-19T15:28:55.015539+0000","last_active":"2024-02-19T15:28:55.998291+0000","last_peered":"2024-02-19T15:28:55.998291+0000","last_clean":"2024-02-19T15:28:55.998291+0000","last_became_active":"2024-02-19T15:28:55.015311+0000","last_became_peered":"2024-02-19T15:28:55.015311+0000","last_unstale":"2024-02-19T15:28:55.998291+0000","last_undegraded":"2024-02-19T15:28:55.998291+0000","last_fullsized":"2024-02-19T15:28:55.998291+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:36:00.916278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.998301+0000","last_change":"2024-02-19T15:28:54.006923+0000","last_active":"2024-02-19T15:28:55.998301+0000","last_peered":"2024-02-19T15:28:55.998301+0000","last_clean":"2024-02-19T15:28:55.998301+0000","last_became_active":"2024-02-19T15:28:54.006565+0000","last_became_peered":"2024-02-19T15:28:54.006565+0000","last_unstale":"2024-02-19T15:28:55.998301+0000","last_undegraded":"2024-02-19T15:28:55.998301+0000","last_fullsized":"2024-02-19T15:28:55.998301+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:09:31.778802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:56.195989+0000","last_change":"2024-02-19T15:28:55.020806+0000","last_active":"2024-02-19T15:28:56.195989+0000","last_peered":"2024-02-19T15:28:56.195989+0000","last_clean":"2024-02-19T15:28:56.195989+0000","last_became_active":"2024-02-19T15:28:55.020492+0000","last_became_peered":"2024-02-19T15:28:55.020492+0000","last_unstale":"2024-02-19T15:28:56.195989+0000","last_undegraded":"2024-02-19T15:28:56.195989+0000","last_fullsized":"2024-02-19T15:28:56.195989+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:34.966857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.019728+0000","last_change":"2024-02-19T15:28:54.012342+0000","last_active":"2024-02-19T15:28:55.019728+0000","last_peered":"2024-02-19T15:28:55.019728+0000","last_clean":"2024-02-19T15:28:55.019728+0000","last_became_active":"2024-02-19T15:28:54.011912+0000","last_became_peered":"2024-02-19T15:28:54.011912+0000","last_unstale":"2024-02-19T15:28:55.019728+0000","last_undegraded":"2024-02-19T15:28:55.019728+0000","last_fullsized":"2024-02-19T15:28:55.019728+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:08:02.941863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.029783+0000","last_change":"2024-02-19T15:28:55.026791+0000","last_active":"2024-02-19T15:28:55.029783+0000","last_peered":"2024-02-19T15:28:55.029783+0000","last_clean":"2024-02-19T15:28:55.029783+0000","last_became_active":"2024-02-19T15:28:55.026245+0000","last_became_peered":"2024-02-19T15:28:55.026245+0000","last_unstale":"2024-02-19T15:28:55.029783+0000","last_undegraded":"2024-02-19T15:28:55.029783+0000","last_fullsized":"2024-02-19T15:28:55.029783+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:24:16.358341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.998010+0000","last_change":"2024-02-19T15:28:54.008333+0000","last_active":"2024-02-19T15:28:55.998010+0000","last_peered":"2024-02-19T15:28:55.998010+0000","last_clean":"2024-02-19T15:28:55.998010+0000","last_became_active":"2024-02-19T15:28:54.007566+0000","last_became_peered":"2024-02-19T15:28:54.007566+0000","last_unstale":"2024-02-19T15:28:55.998010+0000","last_undegraded":"2024-02-19T15:28:55.998010+0000","last_fullsized":"2024-02-19T15:28:55.998010+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:48.551468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.024992+0000","last_change":"2024-02-19T15:28:55.019576+0000","last_active":"2024-02-19T15:28:55.024992+0000","last_peered":"2024-02-19T15:28:55.024992+0000","last_clean":"2024-02-19T15:28:55.024992+0000","last_became_active":"2024-02-19T15:28:55.018374+0000","last_became_peered":"2024-02-19T15:28:55.018374+0000","last_unstale":"2024-02-19T15:28:55.024992+0000","last_undegraded":"2024-02-19T15:28:55.024992+0000","last_fullsized":"2024-02-19T15:28:55.024992+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:10:13.944369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.011078+0000","last_change":"2024-02-19T15:28:54.019151+0000","last_active":"2024-02-19T15:28:55.011078+0000","last_peered":"2024-02-19T15:28:55.011078+0000","last_clean":"2024-02-19T15:28:55.011078+0000","last_became_active":"2024-02-19T15:28:54.018771+0000","last_became_peered":"2024-02-19T15:28:54.018771+0000","last_unstale":"2024-02-19T15:28:55.011078+0000","last_undegraded":"2024-02-19T15:28:55.011078+0000","last_fullsized":"2024-02-19T15:28:55.011078+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:36:31.212498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.034069+0000","last_change":"2024-02-19T15:28:55.033043+0000","last_active":"2024-02-19T15:28:55.034069+0000","last_peered":"2024-02-19T15:28:55.034069+0000","last_clean":"2024-02-19T15:28:55.034069+0000","last_became_active":"2024-02-19T15:28:55.021889+0000","last_became_peered":"2024-02-19T15:28:55.021889+0000","last_unstale":"2024-02-19T15:28:55.034069+0000","last_undegraded":"2024-02-19T15:28:55.034069+0000","last_fullsized":"2024-02-19T15:28:55.034069+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:40:28.110767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.019362+0000","last_change":"2024-02-19T15:28:54.016335+0000","last_active":"2024-02-19T15:28:55.019362+0000","last_peered":"2024-02-19T15:28:55.019362+0000","last_clean":"2024-02-19T15:28:55.019362+0000","last_became_active":"2024-02-19T15:28:54.016042+0000","last_became_peered":"2024-02-19T15:28:54.016042+0000","last_unstale":"2024-02-19T15:28:55.019362+0000","last_undegraded":"2024-02-19T15:28:55.019362+0000","last_fullsized":"2024-02-19T15:28:55.019362+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:28:16.592184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.997882+0000","last_change":"2024-02-19T15:28:55.015900+0000","last_active":"2024-02-19T15:28:55.997882+0000","last_peered":"2024-02-19T15:28:55.997882+0000","last_clean":"2024-02-19T15:28:55.997882+0000","last_became_active":"2024-02-19T15:28:55.015612+0000","last_became_peered":"2024-02-19T15:28:55.015612+0000","last_unstale":"2024-02-19T15:28:55.997882+0000","last_undegraded":"2024-02-19T15:28:55.997882+0000","last_fullsized":"2024-02-19T15:28:55.997882+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:35:42.119295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.018819+0000","last_change":"2024-02-19T15:28:54.013670+0000","last_active":"2024-02-19T15:28:55.018819+0000","last_peered":"2024-02-19T15:28:55.018819+0000","last_clean":"2024-02-19T15:28:55.018819+0000","last_became_active":"2024-02-19T15:28:54.008802+0000","last_became_peered":"2024-02-19T15:28:54.008802+0000","last_unstale":"2024-02-19T15:28:55.018819+0000","last_undegraded":"2024-02-19T15:28:55.018819+0000","last_fullsized":"2024-02-19T15:28:55.018819+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:14:20.149906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.030404+0000","last_change":"2024-02-19T15:28:55.028047+0000","last_active":"2024-02-19T15:28:55.030404+0000","last_peered":"2024-02-19T15:28:55.030404+0000","last_clean":"2024-02-19T15:28:55.030404+0000","last_became_active":"2024-02-19T15:28:55.027744+0000","last_became_peered":"2024-02-19T15:28:55.027744+0000","last_unstale":"2024-02-19T15:28:55.030404+0000","last_undegraded":"2024-02-19T15:28:55.030404+0000","last_fullsized":"2024-02-19T15:28:55.030404+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:53.983641+0000","last_clean_scrub_stamp":"2024-02-19T15:28:53.983641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:59:14.276335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"124'2","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.027465+0000","last_change":"2024-02-19T15:28:54.017971+0000","last_active":"2024-02-19T15:28:55.027465+0000","last_peered":"2024-02-19T15:28:55.027465+0000","last_clean":"2024-02-19T15:28:55.027465+0000","last_became_active":"2024-02-19T15:28:54.017571+0000","last_became_peered":"2024-02-19T15:28:54.017571+0000","last_unstale":"2024-02-19T15:28:55.027465+0000","last_undegraded":"2024-02-19T15:28:55.027465+0000","last_fullsized":"2024-02-19T15:28:55.027465+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:28:52.980737+0000","last_clean_scrub_stamp":"2024-02-19T15:28:52.980737+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:02:54.427924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":354,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T15:28:55.005738+0000","last_change":"2024-02-19T15:21:53.337753+0000","last_active":"2024-02-19T15:28:55.005738+0000","last_peered":"2024-02-19T15:28:55.005738+0000","last_clean":"2024-02-19T15:28:55.005738+0000","last_became_active":"2024-02-19T15:21:53.336224+0000","last_became_peered":"2024-02-19T15:21:53.336224+0000","last_unstale":"2024-02-19T15:28:55.005738+0000","last_undegraded":"2024-02-19T15:28:55.005738+0000","last_fullsized":"2024-02-19T15:28:55.005738+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:21:51.710216+0000","last_clean_scrub_stamp":"2024-02-19T15:21:51.710216+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:07.041170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509528,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36876,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334964,"statfs":{"total":96636764160,"available":96599003136,"internally_reserved":0,"allocated":1568768,"data_stored":679223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.434,"5min":0.373,"15min":0.373},"max":{"1min":0.81499999999999995,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.624,"15min":0.73399999999999999},"min":{"1min":0.45400000000000001,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":0.80500000000000005,"5min":2.8100000000000001,"15min":4.6029999999999998},"last":0.72999999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.64600000000000002,"15min":0.65200000000000002},"min":{"1min":0.47199999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.91000000000000003,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.373},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.67500000000000004,"15min":0.65900000000000003},"min":{"1min":0.50700000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93000000000000005,"5min":2.774,"15min":2.774},"last":0.67500000000000004}]},{"osd":2,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.39100000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.91000000000000003,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70399999999999996,"15min":0.68999999999999995},"min":{"1min":0.53300000000000003,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.95399999999999996,"5min":1.321,"15min":1.321},"last":0.60999999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.83699999999999997,"15min":0.72799999999999998},"min":{"1min":0.51100000000000001,"5min":0.46600000000000003,"15min":0.438},"max":{"1min":0.97899999999999998,"5min":14.045999999999999,"15min":14.045999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68200000000000005,"15min":0.66400000000000003},"min":{"1min":0.54000000000000004,"5min":0.433,"15min":0.433},"max":{"1min":0.85999999999999999,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.5}]},{"osd":4,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.67900000000000005,"15min":0.625},"min":{"1min":0.44800000000000001,"5min":0.32400000000000001,"15min":0.309},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.71999999999999997,"15min":0.63400000000000001},"min":{"1min":0.52700000000000002,"5min":0.35599999999999998,"15min":0.34799999999999998},"max":{"1min":1.026,"5min":2.8500000000000001,"15min":2.8500000000000001},"last":0.83499999999999996}]},{"osd":5,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.69999999999999996,"15min":0.628},"min":{"1min":0.505,"5min":0.38200000000000001,"15min":0.32700000000000001},"max":{"1min":1.1830000000000001,"5min":1.363,"15min":1.363},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.75900000000000001,"15min":0.69499999999999995},"min":{"1min":0.54700000000000004,"5min":0.38,"15min":0.38},"max":{"1min":1.0589999999999999,"5min":1.411,"15min":1.411},"last":0.69899999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79200000000000004,"15min":0.69199999999999995},"min":{"1min":0.63700000000000001,"5min":0.42799999999999999,"15min":0.39600000000000002},"max":{"1min":1.0820000000000001,"5min":3.0539999999999998,"15min":3.0539999999999998},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.78500000000000003,"15min":0.70299999999999996},"min":{"1min":0.65900000000000003,"5min":0.439,"15min":0.34499999999999997},"max":{"1min":1.2190000000000001,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.78400000000000003}]}]},{"osd":7,"up_from":14,"seq":60129542232,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37452,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334388,"statfs":{"total":96636764160,"available":96598413312,"internally_reserved":0,"allocated":2162688,"data_stored":1263929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.89200000000000002,"15min":0.68799999999999994},"min":{"1min":0.41199999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.82299999999999995,"5min":15.127000000000001,"15min":15.127000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.70899999999999996,"15min":0.69099999999999995},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.93899999999999995,"5min":6.0890000000000004,"15min":6.0890000000000004},"last":0.46400000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.94699999999999995,"15min":0.70499999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.94999999999999996,"5min":15.196999999999999,"15min":15.196999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.81499999999999995,"15min":0.75800000000000001},"min":{"1min":0.44800000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.99299999999999999,"5min":10.933999999999999,"15min":10.933999999999999},"last":0.80600000000000005}]},{"osd":2,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.80000000000000004,"15min":0.68700000000000006},"min":{"1min":0.46500000000000002,"5min":0.45700000000000002,"15min":0.42799999999999999},"max":{"1min":0.874,"5min":10.84,"15min":10.84},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71099999999999997,"15min":0.69899999999999995},"min":{"1min":0.48499999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.91100000000000003,"5min":2.3170000000000002,"15min":2.508},"last":0.71899999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.86199999999999999,"15min":0.72599999999999998},"min":{"1min":0.49199999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.91500000000000004,"5min":17.094000000000001,"15min":17.094000000000001},"last":0.753},{"interface":"front","average":{"1min":0.748,"5min":0.82999999999999996,"15min":0.73499999999999999},"min":{"1min":0.54500000000000004,"5min":0.46300000000000002,"15min":0.42799999999999999},"max":{"1min":0.97499999999999998,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.59899999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.55400000000000005},"min":{"1min":0.34999999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.97099999999999997,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.97999999999999998,"15min":0.70199999999999996},"min":{"1min":0.37,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96599999999999997,"5min":26.625,"15min":26.625},"last":0.83099999999999996}]},{"osd":5,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":1.091,"15min":0.72899999999999998},"min":{"1min":0.441,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0409999999999999,"5min":31.844999999999999,"15min":31.844999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.90100000000000002,"15min":0.71299999999999997},"min":{"1min":0.54400000000000004,"5min":0.38,"15min":0.38},"max":{"1min":0.995,"5min":16.776,"15min":16.776},"last":0.86099999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.73899999999999999,"15min":0.63600000000000001},"min":{"1min":0.44800000000000001,"5min":0.375,"15min":0.36099999999999999},"max":{"1min":1.0820000000000001,"5min":1.958,"15min":1.958},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.79700000000000004,"15min":0.69899999999999995},"min":{"1min":0.51200000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0800000000000001,"5min":3.5099999999999998,"15min":3.5099999999999998},"last":0.85099999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542233,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36860,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334980,"statfs":{"total":96636764160,"available":96599019520,"internally_reserved":0,"allocated":1552384,"data_stored":671423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.69199999999999995,"15min":0.65100000000000002},"min":{"1min":0.375,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.94899999999999995,"5min":5.7789999999999999,"15min":5.7789999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.748,"15min":0.64900000000000002},"min":{"1min":0.40999999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.91700000000000004,"5min":12.359999999999999,"15min":12.359999999999999},"last":0.63900000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.77600000000000002,"15min":0.68200000000000005},"min":{"1min":0.49099999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98599999999999999,"5min":10.938000000000001,"15min":10.938000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.82999999999999996,"15min":0.72599999999999998},"min":{"1min":0.46700000000000003,"5min":0.45800000000000002,"15min":0.42599999999999999},"max":{"1min":0.95099999999999996,"5min":10.531000000000001,"15min":10.531000000000001},"last":0.60799999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.81299999999999994,"15min":0.72599999999999998},"min":{"1min":0.496,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92200000000000004,"5min":12.925000000000001,"15min":12.925000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.755,"15min":0.64900000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0029999999999999,"5min":12.401999999999999,"15min":12.401999999999999},"last":0.73699999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.97999999999999998,"15min":0.75800000000000001},"min":{"1min":0.51200000000000001,"5min":0.49399999999999999,"15min":0.45800000000000002},"max":{"1min":1.0680000000000001,"5min":26.161999999999999,"15min":26.161999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.97499999999999998,"15min":0.76200000000000001},"min":{"1min":0.49199999999999999,"5min":0.44500000000000001,"15min":0.41399999999999998},"max":{"1min":0.86399999999999999,"5min":27.934999999999999,"15min":27.934999999999999},"last":0.66000000000000003}]},{"osd":4,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.81799999999999995,"15min":0.64000000000000001},"min":{"1min":0.45500000000000002,"5min":0.35499999999999998,"15min":0.33200000000000002},"max":{"1min":0.86299999999999999,"5min":11.73,"15min":11.73},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.80700000000000005,"15min":0.70299999999999996},"min":{"1min":0.39200000000000002,"5min":0.308,"15min":0.308},"max":{"1min":1.0169999999999999,"5min":10.731,"15min":10.731},"last":0.67200000000000004}]},{"osd":6,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.79300000000000004,"15min":0.68500000000000005},"min":{"1min":0.42899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0649999999999999,"5min":6.5229999999999997,"15min":6.5229999999999997},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.94699999999999995,"15min":0.79100000000000004},"min":{"1min":0.53000000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.091,"5min":10.432,"15min":10.432},"last":0.78600000000000003}]},{"osd":7,"last update":"Mon Feb 19 15:28:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.80800000000000005,"15min":0.70499999999999996},"min":{"1min":0.40300000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.099,"5min":11.986000000000001,"15min":11.986000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.84999999999999998,"15min":0.73899999999999999},"min":{"1min":0.52200000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98399999999999999,"5min":8.9770000000000003,"15min":8.9770000000000003},"last":0.76800000000000002}]}]},{"osd":4,"up_from":14,"seq":60129542231,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36856,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334984,"statfs":{"total":96636764160,"available":96599023616,"internally_reserved":0,"allocated":1548288,"data_stored":671381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.4279999999999999,"15min":0.87},"min":{"1min":0.35599999999999998,"5min":0.34399999999999997,"15min":0.33300000000000002},"max":{"1min":11.925000000000001,"5min":76.117000000000004,"15min":76.117000000000004},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.65200000000000002,"15min":0.61399999999999999},"min":{"1min":0.39100000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.95899999999999996,"5min":3.2709999999999999,"15min":3.2709999999999999},"last":0.64900000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.534,"15min":0.91700000000000004},"min":{"1min":0.443,"5min":0.437,"15min":0.39100000000000001},"max":{"1min":11.837999999999999,"5min":80.513000000000005,"15min":80.513000000000005},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.3080000000000001,"5min":0.84399999999999997,"15min":0.69999999999999996},"min":{"1min":0.41999999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":13.089,"5min":13.089,"15min":13.089},"last":0.68700000000000006}]},{"osd":2,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":0.90700000000000003,"15min":0.745},"min":{"1min":0.47399999999999998,"5min":0.436,"15min":0.436},"max":{"1min":13.255000000000001,"5min":14.362,"15min":14.362},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.587,"15min":0.97799999999999998},"min":{"1min":0.42499999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":12.372999999999999,"5min":84.093000000000004,"15min":84.093000000000004},"last":0.84199999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.6339999999999999,"15min":0.99099999999999999},"min":{"1min":0.53800000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":12.319000000000001,"5min":84.140000000000001,"15min":84.140000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66100000000000003,"15min":0.65400000000000003},"min":{"1min":0.438,"5min":0.42799999999999999,"15min":0.40500000000000003},"max":{"1min":1.0249999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.55500000000000005}]},{"osd":5,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":0.91100000000000003,"15min":0.69799999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.36499999999999999},"max":{"1min":13.292999999999999,"5min":14.318,"15min":14.318},"last":0.753},{"interface":"front","average":{"1min":1.371,"5min":1.5669999999999999,"15min":0.96899999999999997},"min":{"1min":0.60899999999999999,"5min":0.40000000000000002,"15min":0.36599999999999999},"max":{"1min":12.821,"5min":76.164000000000001,"15min":76.164000000000001},"last":0.57599999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:28:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":0.79400000000000004,"15min":0.64000000000000001},"min":{"1min":0.40300000000000002,"5min":0.33500000000000002,"15min":0.32700000000000001},"max":{"1min":15.196,"5min":15.196,"15min":15.196},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.393,"15min":0.89100000000000001},"min":{"1min":0.55400000000000005,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":13.634,"5min":60.151000000000003,"15min":60.151000000000003},"last":1.0169999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":1.006,"15min":0.77200000000000002},"min":{"1min":0.52700000000000002,"5min":0.40699999999999997,"15min":0.38400000000000001},"max":{"1min":1.6080000000000001,"5min":29.036000000000001,"15min":29.036000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":0.95599999999999996,"15min":0.80000000000000004},"min":{"1min":0.55500000000000005,"5min":0.46999999999999997,"15min":0.443},"max":{"1min":13.519,"5min":13.519,"15min":13.519},"last":0.81799999999999995}]}]},{"osd":3,"up_from":14,"seq":60129542231,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41020,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330820,"statfs":{"total":96636764160,"available":96594759680,"internally_reserved":0,"allocated":1552384,"data_stored":671471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.41599999999999998,"15min":0.443},"min":{"1min":0.25700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.68400000000000005,"5min":0.82699999999999996,"15min":0.97299999999999998},"last":0.75},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.44400000000000001,"15min":0.61399999999999999},"min":{"1min":0.27500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52400000000000002,"5min":1.0840000000000001,"15min":4.2450000000000001},"last":0.33300000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.443,"15min":0.44500000000000001},"min":{"1min":0.27800000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.76900000000000002,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.432},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.49299999999999999,"15min":0.67500000000000004},"min":{"1min":0.34399999999999997,"5min":0.31,"15min":0.31},"max":{"1min":0.73199999999999998,"5min":0.85099999999999998,"15min":4.0220000000000002},"last":0.37}]},{"osd":2,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.53800000000000003,"15min":0.60199999999999998},"min":{"1min":0.38700000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72899999999999998,"5min":1.24,"15min":1.24},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52300000000000002,"15min":0.57999999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0169999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.47199999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.746,"15min":0.69299999999999995},"min":{"1min":0.621,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.73799999999999999,"15min":0.83999999999999997},"min":{"1min":0.48699999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":4.2039999999999997},"last":0.54600000000000004}]},{"osd":5,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.75,"15min":0.83799999999999997},"min":{"1min":0.51400000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.252,"5min":1.252,"15min":4.1589999999999998},"last":0.63},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.749,"15min":0.75},"min":{"1min":0.502,"5min":0.46999999999999997,"15min":0.46800000000000003},"max":{"1min":1.081,"5min":1.081,"15min":1.3680000000000001},"last":0.67900000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.77300000000000002,"15min":0.873},"min":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.052,"5min":1.3180000000000001,"15min":4.0940000000000003},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.77100000000000002,"15min":0.80900000000000005},"min":{"1min":0.55300000000000005,"5min":0.51800000000000002,"15min":0.502},"max":{"1min":1.0569999999999999,"5min":1.206,"15min":1.9550000000000001},"last":0.61499999999999999}]},{"osd":7,"last update":"Mon Feb 19 15:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.76200000000000001,"15min":0.77900000000000003},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.504},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.6579999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.77500000000000002,"15min":0.89800000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.0169999999999999,"5min":1.1919999999999999,"15min":3.9660000000000002},"last":0.70699999999999996}]}]},{"osd":2,"up_from":14,"seq":60129542232,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36860,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334980,"statfs":{"total":96636764160,"available":96599019520,"internally_reserved":0,"allocated":1552384,"data_stored":671927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.53600000000000003,"15min":0.65300000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.57499999999999996,"5min":15.42,"15min":15.42},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.50700000000000001,"15min":0.81599999999999995},"min":{"1min":0.26500000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.72899999999999998,"5min":6.7000000000000002,"15min":8.2129999999999992},"last":0.46999999999999997}]},{"osd":1,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.55100000000000005,"15min":0.82899999999999996},"min":{"1min":0.27500000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.61099999999999999,"5min":9.5269999999999992,"15min":9.5269999999999992},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.73699999999999999,"15min":0.93799999999999994},"min":{"1min":0.39800000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.755,"5min":22.954999999999998,"15min":22.954999999999998},"last":0.56200000000000006}]},{"osd":3,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.55000000000000004,"15min":0.74299999999999999},"min":{"1min":0.33100000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81399999999999995,"5min":3.7799999999999998,"15min":4.7679999999999998},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.628,"15min":0.78400000000000003},"min":{"1min":0.44700000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.73499999999999999,"5min":7.734,"15min":7.734},"last":0.58999999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.83199999999999996,"15min":0.876},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.04,"5min":15.369,"15min":15.369},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73899999999999999,"15min":0.86299999999999999},"min":{"1min":0.58999999999999997,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":3.6819999999999999,"15min":4.3440000000000003},"last":0.97599999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.80500000000000005,"15min":0.86699999999999999},"min":{"1min":0.57799999999999996,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0269999999999999,"5min":7.6520000000000001,"15min":7.6520000000000001},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.90500000000000003,"15min":0.94699999999999995},"min":{"1min":0.66400000000000003,"5min":0.496,"15min":0.47399999999999998},"max":{"1min":1.161,"5min":15.391,"15min":15.391},"last":1.0720000000000001}]},{"osd":6,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.877,"15min":0.91700000000000004},"min":{"1min":0.55500000000000005,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.22,"5min":15.337,"15min":15.337},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77000000000000002,"15min":0.92300000000000004},"min":{"1min":0.61699999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0580000000000001,"5min":3.7450000000000001,"15min":4.4320000000000004},"last":1.127}]},{"osd":7,"last update":"Mon Feb 19 15:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.81799999999999995,"15min":0.95199999999999996},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.52600000000000002},"max":{"1min":1.1850000000000001,"5min":3.71,"15min":4.4829999999999997},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.82699999999999996,"15min":0.92300000000000004},"min":{"1min":0.58099999999999996,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.079,"5min":7.6929999999999996,"15min":7.6929999999999996},"last":1.01}]}]},{"osd":1,"up_from":14,"seq":60129542233,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41024,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330816,"statfs":{"total":96636764160,"available":96594755584,"internally_reserved":0,"allocated":1556480,"data_stored":677611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.39500000000000002,"15min":0.42499999999999999},"min":{"1min":0.29199999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.438,"15min":0.46600000000000003},"min":{"1min":0.33200000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.315}]},{"osd":2,"last update":"Mon Feb 19 15:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.434,"15min":0.44700000000000001},"min":{"1min":0.34200000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.48099999999999998,"15min":0.52400000000000002},"min":{"1min":0.38400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.33400000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.57699999999999996,"15min":0.629},"min":{"1min":0.42599999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.365},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.54100000000000004,"15min":0.61899999999999999},"min":{"1min":0.48299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81100000000000005,"5min":1.0069999999999999,"15min":1.4350000000000001},"last":0.47599999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.70299999999999996,"15min":0.629},"min":{"1min":0.49199999999999999,"5min":0.434,"15min":0.377},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.72399999999999998,"15min":0.68999999999999995},"min":{"1min":0.54300000000000004,"5min":0.435,"15min":0.435},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.64200000000000002}]},{"osd":5,"last update":"Mon Feb 19 15:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.72299999999999998,"15min":0.70199999999999996},"min":{"1min":0.48599999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.2849999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.748,"5min":0.72299999999999998,"15min":0.69099999999999995},"min":{"1min":0.51600000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.77400000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.754,"15min":0.73599999999999999},"min":{"1min":0.57899999999999996,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3460000000000001},"last":0.747},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.77500000000000002,"15min":0.75600000000000001},"min":{"1min":0.65400000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4710000000000001},"last":0.67100000000000004}]},{"osd":7,"last update":"Mon Feb 19 15:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.77700000000000002,"15min":0.78900000000000003},"min":{"1min":0.57799999999999996,"5min":0.496,"15min":0.496},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.4930000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.79000000000000004,"15min":0.75600000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":0.81899999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542231,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41592,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330248,"statfs":{"total":96636764160,"available":96594173952,"internally_reserved":0,"allocated":2138112,"data_stored":1261679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:28:00 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.39100000000000001,"15min":0.58899999999999997},"min":{"1min":0.23499999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.52600000000000002,"5min":0.745,"15min":5.0199999999999996},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45000000000000001,"15min":0.47899999999999998},"min":{"1min":0.26200000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.61399999999999999,"5min":0.77300000000000002,"15min":1.2050000000000001},"last":0.60899999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:28:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53400000000000003,"15min":0.74399999999999999},"min":{"1min":0.39000000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.64000000000000001,"5min":0.82399999999999995,"15min":5.6669999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.497,"15min":0.70499999999999996},"min":{"1min":0.40600000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69899999999999995,"5min":0.85899999999999999,"15min":5.4649999999999999},"last":0.504}]},{"osd":3,"last update":"Mon Feb 19 15:28:00 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.46400000000000002,"15min":0.64400000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.81799999999999995,"5min":0.93799999999999994,"15min":4.9539999999999997},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.504,"15min":0.59599999999999997},"min":{"1min":0.36899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76200000000000001,"5min":1.052,"15min":1.345},"last":0.64700000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:28:00 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.73199999999999998,"15min":0.73799999999999999},"min":{"1min":0.505,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.97899999999999998,"5min":1.091,"15min":1.095},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.74399999999999999,"15min":0.89200000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.99199999999999999,"5min":1.085,"15min":4.7880000000000003},"last":0.627}]},{"osd":5,"last update":"Mon Feb 19 15:28:00 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.70499999999999996,"15min":0.73999999999999999},"min":{"1min":0.47299999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.95299999999999996,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.69499999999999995,"15min":0.82699999999999996},"min":{"1min":0.5,"5min":0.5,"15min":0.46100000000000002},"max":{"1min":0.89500000000000002,"5min":1.169,"15min":4.8810000000000002},"last":0.67400000000000004}]},{"osd":6,"last update":"Mon Feb 19 15:28:00 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71499999999999997,"15min":0.88},"min":{"1min":0.498,"5min":0.439,"15min":0.439},"max":{"1min":0.96199999999999997,"5min":1.1200000000000001,"15min":4.8460000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71299999999999997,"15min":0.749},"min":{"1min":0.58199999999999996,"5min":0.50800000000000001,"15min":0.49399999999999999},"max":{"1min":0.93100000000000005,"5min":1.109,"15min":1.109},"last":0.68799999999999994}]},{"osd":7,"last update":"Mon Feb 19 15:28:00 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.90700000000000003},"min":{"1min":0.52800000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.93000000000000005,"5min":1.2450000000000001,"15min":4.9809999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.72999999999999998,"15min":0.90600000000000003},"min":{"1min":0.55200000000000005,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.91800000000000004,"5min":1.149,"15min":4.915},"last":0.74199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:28:59.565 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T15:28:59.565 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-19T15:28:59.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.750+0000 7fa1b3e00640 1 Processor -- start 2024-02-19T15:28:59.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.750+0000 7fa1b3e00640 1 -- start start 2024-02-19T15:28:59.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.754+0000 7fa1b3e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1ac0ceb20 0x7fa1ac0d2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:59.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.754+0000 7fa1b3e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa1ac15d860 0x7fa1ac0d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:59.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.754+0000 7fa1b3e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ac0cc690 con 0x7fa1ac0d0630 2024-02-19T15:28:59.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.754+0000 7fa1b3e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ac0cfca0 con 0x7fa1ac15d860 2024-02-19T15:28:59.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.754+0000 7fa1b3e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac0d0ac0 con 0x7fa1ac0ceb20 2024-02-19T15:28:59.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.754+0000 7fa1b0a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa1ac15d860 0x7fa1ac0d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:59.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.754+0000 7fa1b1400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1ac0ceb20 0x7fa1ac0d2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:59.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.754+0000 7fa1b0a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa1ac15d860 0x7fa1ac0d00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:57388/0 (socket says 172.21.15.47:57388) 2024-02-19T15:28:59.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.754+0000 7fa1b0a00640 1 -- 172.21.15.47:0/1138536237 learned_addr learned my addr 172.21.15.47:0/1138536237 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:59.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.754+0000 7fa1b0a00640 1 -- 172.21.15.47:0/1138536237 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1ac0ceb20 msgr2=0x7fa1ac0d2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:28:59.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.754+0000 7fa1b0a00640 1 --2- 172.21.15.47:0/1138536237 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1ac0ceb20 0x7fa1ac0d2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.754+0000 7fa1b0a00640 1 -- 172.21.15.47:0/1138536237 >> v1:172.21.15.116:6789/0 conn(0x7fa1ac0d0630 legacy=0x7fa1ac0ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:59.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.754+0000 7fa1b0a00640 1 -- 172.21.15.47:0/1138536237 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac0cc950 con 0x7fa1ac15d860 2024-02-19T15:28:59.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.758+0000 7fa1b0a00640 1 --2- 172.21.15.47:0/1138536237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1ac15d860 0x7fa1ac0d00f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fa198009870 tx=0x7fa19801c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd87640c315872c7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:59.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.758+0000 7fa1abe00640 1 -- 172.21.15.47:0/1138536237 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1980ab600 con 0x7fa1ac15d860 2024-02-19T15:28:59.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.758+0000 7fa1abe00640 1 -- 172.21.15.47:0/1138536237 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1980a7050 con 0x7fa1ac15d860 2024-02-19T15:28:59.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.758+0000 7fa1abe00640 1 -- 172.21.15.47:0/1138536237 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1980ab980 con 0x7fa1ac15d860 2024-02-19T15:28:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.758+0000 7fa1b3e00640 1 -- 172.21.15.47:0/1138536237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1ac15d860 msgr2=0x7fa1ac0d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.758+0000 7fa1b3e00640 1 --2- 172.21.15.47:0/1138536237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1ac15d860 0x7fa1ac0d00f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fa198009870 tx=0x7fa19801c4d0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.758+0000 7fa1b3e00640 1 -- 172.21.15.47:0/1138536237 shutdown_connections 2024-02-19T15:28:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.758+0000 7fa1b3e00640 1 --2- 172.21.15.47:0/1138536237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1ac15d860 0x7fa1ac0d00f0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.758+0000 7fa1b3e00640 1 --2- 172.21.15.47:0/1138536237 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1ac0ceb20 0x7fa1ac0d2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.758+0000 7fa1b3e00640 1 -- 172.21.15.47:0/1138536237 >> 172.21.15.47:0/1138536237 conn(0x7fa1ac09aed0 msgr2=0x7fa1ac0d1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:28:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.758+0000 7fa1b3e00640 1 -- 172.21.15.47:0/1138536237 shutdown_connections 2024-02-19T15:28:59.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.758+0000 7fa1b3e00640 1 -- 172.21.15.47:0/1138536237 wait complete. 2024-02-19T15:28:59.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.758+0000 7fa1b3e00640 1 Processor -- start 2024-02-19T15:28:59.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.758+0000 7fa1b3e00640 1 -- start start 2024-02-19T15:28:59.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1b3e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1ac0ceb20 0x7fa1ac157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:59.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1b3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1ac0d0630 0x7fa1ac158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:59.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1b3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1ac15d860 0x7fa1ac0c6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:59.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1b3e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac0b64c0 con 0x7fa1ac15d860 2024-02-19T15:28:59.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1b3e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac0b67a0 con 0x7fa1ac0d0630 2024-02-19T15:28:59.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1b3e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac0b6a80 con 0x7fa1ac0ceb20 2024-02-19T15:28:59.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1b1400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1ac0ceb20 0x7fa1ac157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:59.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1b1400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1ac0ceb20 0x7fa1ac157e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:36522/0 (socket says 172.21.15.47:36522) 2024-02-19T15:28:59.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1b1400640 1 -- 172.21.15.47:0/3506953526 learned_addr learned my addr 172.21.15.47:0/3506953526 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:28:59.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1b0a00640 1 --2- 172.21.15.47:0/3506953526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1ac0d0630 0x7fa1ac158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:59.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1b1e00640 1 --2- 172.21.15.47:0/3506953526 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1ac15d860 0x7fa1ac0c6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:59.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1b1400640 1 -- 172.21.15.47:0/3506953526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1ac0d0630 msgr2=0x7fa1ac158370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:59.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1b1400640 1 --2- 172.21.15.47:0/3506953526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1ac0d0630 0x7fa1ac158370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1b1400640 1 -- 172.21.15.47:0/3506953526 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1ac15d860 msgr2=0x7fa1ac0c6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:59.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1b1400640 1 --2- 172.21.15.47:0/3506953526 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1ac15d860 0x7fa1ac0c6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1b1400640 1 -- 172.21.15.47:0/3506953526 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1980094d0 con 0x7fa1ac0ceb20 2024-02-19T15:28:59.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1b1400640 1 --2- 172.21.15.47:0/3506953526 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1ac0ceb20 0x7fa1ac157e30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa1a000e930 tx=0x7fa1a000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:59.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1a9600640 1 -- 172.21.15.47:0/3506953526 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0009690 con 0x7fa1ac0ceb20 2024-02-19T15:28:59.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1a9600640 1 -- 172.21.15.47:0/3506953526 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a000ca00 con 0x7fa1ac0ceb20 2024-02-19T15:28:59.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1b3e00640 1 -- 172.21.15.47:0/3506953526 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ac0cda80 con 0x7fa1ac0ceb20 2024-02-19T15:28:59.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1a9600640 1 -- 172.21.15.47:0/3506953526 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a000cd70 con 0x7fa1ac0ceb20 2024-02-19T15:28:59.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1b3e00640 1 -- 172.21.15.47:0/3506953526 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ac0b7040 con 0x7fa1ac0ceb20 2024-02-19T15:28:59.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1b3e00640 1 -- 172.21.15.47:0/3506953526 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ac075360 con 0x7fa1ac0ceb20 2024-02-19T15:28:59.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.762+0000 7fa1a9600640 1 -- 172.21.15.47:0/3506953526 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa1a0013470 con 0x7fa1ac0ceb20 2024-02-19T15:28:59.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.766+0000 7fa1a9600640 1 --2- 172.21.15.47:0/3506953526 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa18c066e40 0x7fa18c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:28:59.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.766+0000 7fa1a9600640 1 -- 172.21.15.47:0/3506953526 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1a0085370 con 0x7fa1ac0ceb20 2024-02-19T15:28:59.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.766+0000 7fa1b0a00640 1 --2- 172.21.15.47:0/3506953526 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa18c066e40 0x7fa18c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:28:59.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.766+0000 7fa1b0a00640 1 --2- 172.21.15.47:0/3506953526 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa18c066e40 0x7fa18c069300 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fa1980ad000 tx=0x7fa1980a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:28:59.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.770+0000 7fa1a9600640 1 -- 172.21.15.47:0/3506953526 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa1a0052340 con 0x7fa1ac0ceb20 2024-02-19T15:28:59.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.990+0000 7fa1b3e00640 1 -- 172.21.15.47:0/3506953526 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa1ac0d2220 con 0x7fa1ac0ceb20 2024-02-19T15:28:59.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.994+0000 7fa1a9600640 1 -- 172.21.15.47:0/3506953526 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7fa1a00170b0 con 0x7fa1ac0ceb20 2024-02-19T15:28:59.996 INFO:teuthology.orchestra.run.smithi047.stdout:[client.0] 2024-02-19T15:28:59.996 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDjcdNlb7eRNRAAk6ZLGfjhcdlR9GoXFCd9eA== 2024-02-19T15:28:59.996 INFO:teuthology.orchestra.run.smithi047.stdout: caps mds = "allow" 2024-02-19T15:28:59.996 INFO:teuthology.orchestra.run.smithi047.stdout: caps mon = "allow r" 2024-02-19T15:28:59.997 INFO:teuthology.orchestra.run.smithi047.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-19T15:28:59.998 INFO:teuthology.orchestra.run.smithi047.stderr:updated caps for client.0 2024-02-19T15:28:59.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.998+0000 7fa1b3e00640 1 -- 172.21.15.47:0/3506953526 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa18c066e40 msgr2=0x7fa18c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:59.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.998+0000 7fa1b3e00640 1 --2- 172.21.15.47:0/3506953526 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa18c066e40 0x7fa18c069300 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fa1980ad000 tx=0x7fa1980a6000 comp rx=0 tx=0).stop 2024-02-19T15:28:59.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.998+0000 7fa1b3e00640 1 -- 172.21.15.47:0/3506953526 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1ac0ceb20 msgr2=0x7fa1ac157e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:28:59.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.998+0000 7fa1b3e00640 1 --2- 172.21.15.47:0/3506953526 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1ac0ceb20 0x7fa1ac157e30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa1a000e930 tx=0x7fa1a000edf0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.998+0000 7fa1b3e00640 1 -- 172.21.15.47:0/3506953526 shutdown_connections 2024-02-19T15:28:59.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.998+0000 7fa1b3e00640 1 --2- 172.21.15.47:0/3506953526 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa18c066e40 0x7fa18c069300 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.998+0000 7fa1b3e00640 1 --2- 172.21.15.47:0/3506953526 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1ac15d860 0x7fa1ac0c6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:28:59.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.998+0000 7fa1b3e00640 1 --2- 172.21.15.47:0/3506953526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1ac0d0630 0x7fa1ac158370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.998+0000 7fa1b3e00640 1 --2- 172.21.15.47:0/3506953526 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa1ac0ceb20 0x7fa1ac157e30 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.998+0000 7fa1b3e00640 1 -- 172.21.15.47:0/3506953526 >> 172.21.15.47:0/3506953526 conn(0x7fa1ac09aed0 msgr2=0x7fa1ac0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:00.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.998+0000 7fa1b3e00640 1 -- 172.21.15.47:0/3506953526 shutdown_connections 2024-02-19T15:29:00.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:28:59.998+0000 7fa1b3e00640 1 -- 172.21.15.47:0/3506953526 wait complete. 2024-02-19T15:29:00.017 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:00.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.158+0000 7f359cc00640 1 Processor -- start 2024-02-19T15:29:00.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.158+0000 7f359cc00640 1 -- start start 2024-02-19T15:29:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.162+0000 7f359cc00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f359806d650 0x7f35980d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.162+0000 7f359cc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35980ced10 0x7f359806bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.162+0000 7f359cc00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35980cce30 con 0x7f35980cded0 2024-02-19T15:29:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.162+0000 7f359cc00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35980cd110 con 0x7f359806d650 2024-02-19T15:29:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.162+0000 7f359cc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35980cd3f0 con 0x7f35980ced10 2024-02-19T15:29:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.162+0000 7f3595e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f359806d650 0x7f35980d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.162+0000 7f3595e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f359806d650 0x7f35980d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51154/0 (socket says 172.21.15.47:51154) 2024-02-19T15:29:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.162+0000 7f3595e00640 1 -- 172.21.15.47:0/2563792551 learned_addr learned my addr 172.21.15.47:0/2563792551 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.162+0000 7f3596800640 1 --2- 172.21.15.47:0/2563792551 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35980ced10 0x7f359806bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.162+0000 7f3595e00640 1 -- 172.21.15.47:0/2563792551 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35980ced10 msgr2=0x7f359806bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.162+0000 7f3595e00640 1 --2- 172.21.15.47:0/2563792551 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35980ced10 0x7f359806bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.162+0000 7f3595e00640 1 -- 172.21.15.47:0/2563792551 >> v1:172.21.15.116:6789/0 conn(0x7f35980cded0 legacy=0x7f35980ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.162+0000 7f3595e00640 1 -- 172.21.15.47:0/2563792551 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35980cd6d0 con 0x7f359806d650 2024-02-19T15:29:00.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f3595e00640 1 --2- 172.21.15.47:0/2563792551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f359806d650 0x7f35980d08b0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f3588009e50 tx=0x7f358801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc6ec56a670e02e2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:00.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f3595400640 1 -- 172.21.15.47:0/2563792551 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35880a5850 con 0x7f359806d650 2024-02-19T15:29:00.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f3595400640 1 -- 172.21.15.47:0/2563792551 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35880a59b0 con 0x7f359806d650 2024-02-19T15:29:00.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f3595400640 1 -- 172.21.15.47:0/2563792551 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35880a5d40 con 0x7f359806d650 2024-02-19T15:29:00.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f359cc00640 1 -- 172.21.15.47:0/2563792551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f359806d650 msgr2=0x7f35980d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:00.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f359cc00640 1 --2- 172.21.15.47:0/2563792551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f359806d650 0x7f35980d08b0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f3588009e50 tx=0x7f358801c970 comp rx=0 tx=0).stop 2024-02-19T15:29:00.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f359cc00640 1 -- 172.21.15.47:0/2563792551 shutdown_connections 2024-02-19T15:29:00.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f359cc00640 1 --2- 172.21.15.47:0/2563792551 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35980ced10 0x7f359806bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f359cc00640 1 --2- 172.21.15.47:0/2563792551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f359806d650 0x7f35980d08b0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f359cc00640 1 -- 172.21.15.47:0/2563792551 >> 172.21.15.47:0/2563792551 conn(0x7f35980d1bf0 msgr2=0x7f35980d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:00.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f359cc00640 1 -- 172.21.15.47:0/2563792551 shutdown_connections 2024-02-19T15:29:00.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f359cc00640 1 -- 172.21.15.47:0/2563792551 wait complete. 2024-02-19T15:29:00.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f359cc00640 1 Processor -- start 2024-02-19T15:29:00.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f359cc00640 1 -- start start 2024-02-19T15:29:00.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f359cc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f359806d650 0x7f35980c7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f359cc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35980cded0 0x7f35980c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f359cc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35980ced10 0x7f35980c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f359cc00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3598158680 con 0x7f359806d650 2024-02-19T15:29:00.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f359cc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3598159930 con 0x7f35980cded0 2024-02-19T15:29:00.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f359cc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3598158890 con 0x7f35980ced10 2024-02-19T15:29:00.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f3595e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f359806d650 0x7f35980c7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f3595e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f359806d650 0x7f35980c7ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51170/0 (socket says 172.21.15.47:51170) 2024-02-19T15:29:00.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.166+0000 7f3595e00640 1 -- 172.21.15.47:0/4219370081 learned_addr learned my addr 172.21.15.47:0/4219370081 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:00.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.170+0000 7f358d400640 1 --2- 172.21.15.47:0/4219370081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35980cded0 0x7f35980c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.170+0000 7f3596800640 1 --2- 172.21.15.47:0/4219370081 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35980ced10 0x7f35980c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.170+0000 7f3595e00640 1 -- 172.21.15.47:0/4219370081 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35980ced10 msgr2=0x7f35980c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:00.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.170+0000 7f3595e00640 1 --2- 172.21.15.47:0/4219370081 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35980ced10 0x7f35980c9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.170+0000 7f3595e00640 1 -- 172.21.15.47:0/4219370081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35980cded0 msgr2=0x7f35980c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:00.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.170+0000 7f3595e00640 1 --2- 172.21.15.47:0/4219370081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35980cded0 0x7f35980c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.170+0000 7f3595e00640 1 -- 172.21.15.47:0/4219370081 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3588009b30 con 0x7f359806d650 2024-02-19T15:29:00.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.170+0000 7f358d400640 1 --2- 172.21.15.47:0/4219370081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35980cded0 0x7f35980c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:29:00.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.170+0000 7f3596800640 1 --2- 172.21.15.47:0/4219370081 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35980ced10 0x7f35980c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:29:00.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.170+0000 7f3595e00640 1 --2- 172.21.15.47:0/4219370081 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f359806d650 0x7f35980c7ea0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f358801a5a0 tx=0x7f3588019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:00.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.170+0000 7f358ea00640 1 -- 172.21.15.47:0/4219370081 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35880b8670 con 0x7f359806d650 2024-02-19T15:29:00.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.170+0000 7f358ea00640 1 -- 172.21.15.47:0/4219370081 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35880b87d0 con 0x7f359806d650 2024-02-19T15:29:00.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.170+0000 7f359cc00640 1 -- 172.21.15.47:0/4219370081 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3598158b70 con 0x7f359806d650 2024-02-19T15:29:00.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.170+0000 7f358ea00640 1 -- 172.21.15.47:0/4219370081 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35880b8b80 con 0x7f359806d650 2024-02-19T15:29:00.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.170+0000 7f359cc00640 1 -- 172.21.15.47:0/4219370081 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3598159130 con 0x7f359806d650 2024-02-19T15:29:00.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.170+0000 7f359cc00640 1 -- 172.21.15.47:0/4219370081 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3598075360 con 0x7f359806d650 2024-02-19T15:29:00.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.170+0000 7f358ea00640 1 -- 172.21.15.47:0/4219370081 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f35880b8ce0 con 0x7f359806d650 2024-02-19T15:29:00.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.174+0000 7f358ea00640 1 --2- 172.21.15.47:0/4219370081 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3574066e40 0x7f3574069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.174+0000 7f358ea00640 1 -- 172.21.15.47:0/4219370081 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35881201d0 con 0x7f359806d650 2024-02-19T15:29:00.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.174+0000 7f358d400640 1 --2- 172.21.15.47:0/4219370081 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3574066e40 0x7f3574069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.174+0000 7f358d400640 1 --2- 172.21.15.47:0/4219370081 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3574066e40 0x7f3574069300 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f359806fe10 tx=0x7f3580009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:00.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.178+0000 7f358ea00640 1 -- 172.21.15.47:0/4219370081 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f35880ecfb0 con 0x7f359806d650 2024-02-19T15:29:00.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.374+0000 7f359cc00640 1 -- 172.21.15.47:0/4219370081 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35980cf580 con 0x7f359806d650 2024-02-19T15:29:00.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.374+0000 7f358ea00640 1 -- 172.21.15.47:0/4219370081 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+4943 (secure 0 0 0) 0x7f35880f0670 con 0x7f359806d650 2024-02-19T15:29:00.375 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:00.375 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":91,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7028,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/809721153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":809721153},{"type":"v1","addr":"172.21.15.47:6834","nonce":809721153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7034,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1553163168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1553163168},{"type":"v1","addr":"172.21.15.116:6836","nonce":1553163168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":7037,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:28:57.400659+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8361},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:00.375 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 91 2024-02-19T15:29:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.378+0000 7f359cc00640 1 -- 172.21.15.47:0/4219370081 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3574066e40 msgr2=0x7f3574069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.378+0000 7f359cc00640 1 --2- 172.21.15.47:0/4219370081 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3574066e40 0x7f3574069300 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f359806fe10 tx=0x7f3580009040 comp rx=0 tx=0).stop 2024-02-19T15:29:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.378+0000 7f359cc00640 1 -- 172.21.15.47:0/4219370081 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f359806d650 msgr2=0x7f35980c7ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.378+0000 7f359cc00640 1 --2- 172.21.15.47:0/4219370081 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f359806d650 0x7f35980c7ea0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f358801a5a0 tx=0x7f3588019710 comp rx=0 tx=0).stop 2024-02-19T15:29:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.378+0000 7f359cc00640 1 -- 172.21.15.47:0/4219370081 shutdown_connections 2024-02-19T15:29:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.378+0000 7f359cc00640 1 --2- 172.21.15.47:0/4219370081 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3574066e40 0x7f3574069300 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.378+0000 7f359cc00640 1 --2- 172.21.15.47:0/4219370081 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35980ced10 0x7f35980c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.378+0000 7f359cc00640 1 --2- 172.21.15.47:0/4219370081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35980cded0 0x7f35980c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.378+0000 7f359cc00640 1 --2- 172.21.15.47:0/4219370081 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f359806d650 0x7f35980c7ea0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.378+0000 7f359cc00640 1 -- 172.21.15.47:0/4219370081 >> 172.21.15.47:0/4219370081 conn(0x7f35980d1bf0 msgr2=0x7f359815dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.378+0000 7f359cc00640 1 -- 172.21.15.47:0/4219370081 shutdown_connections 2024-02-19T15:29:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.378+0000 7f359cc00640 1 -- 172.21.15.47:0/4219370081 wait complete. 2024-02-19T15:29:00.801 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 91, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:28:57.400659+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8361}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:00.801 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T15:29:00.801 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-19T15:29:00.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.942+0000 7f69af800640 1 Processor -- start 2024-02-19T15:29:00.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.942+0000 7f69af800640 1 -- start start 2024-02-19T15:29:00.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.942+0000 7f69af800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69a806d650 0x7f69a80d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.942+0000 7f69af800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a80cce30 con 0x7f69a80ced10 2024-02-19T15:29:00.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.942+0000 7f69af800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a80cd110 con 0x7f69a80cded0 2024-02-19T15:29:00.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.942+0000 7f69af800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a80cd3f0 con 0x7f69a806d650 2024-02-19T15:29:00.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.946+0000 7f69ad800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f69a80ced10 0x7f69a806bd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41176/0 (socket says 172.21.15.47:41176) 2024-02-19T15:29:00.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.946+0000 7f69ad800640 1 -- 172.21.15.47:0/351536377 learned_addr learned my addr 172.21.15.47:0/351536377 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:29:00.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.946+0000 7f69ace00640 1 --2- 172.21.15.47:0/351536377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69a806d650 0x7f69a80d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.946+0000 7f69a7e00640 1 -- 172.21.15.47:0/351536377 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018493505 0 0) 0x7f69a80cce30 con 0x7f69a80ced10 2024-02-19T15:29:00.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.946+0000 7f69a7e00640 1 -- 172.21.15.47:0/351536377 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f698c003530 con 0x7f69a80ced10 2024-02-19T15:29:00.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.946+0000 7f69a7e00640 1 -- 172.21.15.47:0/351536377 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180481674 0 0) 0x7f69a80cd110 con 0x7f69a80cded0 2024-02-19T15:29:00.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.946+0000 7f69a7e00640 1 -- 172.21.15.47:0/351536377 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69a80cce30 con 0x7f69a80cded0 2024-02-19T15:29:00.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.946+0000 7f69ace00640 1 -- 172.21.15.47:0/351536377 >> v1:172.21.15.116:6789/0 conn(0x7f69a80cded0 legacy=0x7f69a80ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:00.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.946+0000 7f69ace00640 1 -- 172.21.15.47:0/351536377 >> v1:172.21.15.47:6789/0 conn(0x7f69a80ced10 legacy=0x7f69a806bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:00.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.946+0000 7f69ace00640 1 -- 172.21.15.47:0/351536377 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a80cd6d0 con 0x7f69a806d650 2024-02-19T15:29:00.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.946+0000 7f69ace00640 1 --2- 172.21.15.47:0/351536377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69a806d650 0x7f69a80d08b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f69980097f0 tx=0x7f699801c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba930b52ad4f8b90 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:00.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69a7e00640 1 -- 172.21.15.47:0/351536377 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69980ab640 con 0x7f69a806d650 2024-02-19T15:29:00.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69a7e00640 1 -- 172.21.15.47:0/351536377 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69980a7070 con 0x7f69a806d650 2024-02-19T15:29:00.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69a7e00640 1 -- 172.21.15.47:0/351536377 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69980ab9c0 con 0x7f69a806d650 2024-02-19T15:29:00.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69af800640 1 -- 172.21.15.47:0/351536377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69a806d650 msgr2=0x7f69a80d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:00.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69af800640 1 --2- 172.21.15.47:0/351536377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69a806d650 0x7f69a80d08b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f69980097f0 tx=0x7f699801c420 comp rx=0 tx=0).stop 2024-02-19T15:29:00.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69af800640 1 -- 172.21.15.47:0/351536377 shutdown_connections 2024-02-19T15:29:00.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69af800640 1 --2- 172.21.15.47:0/351536377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69a806d650 0x7f69a80d08b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69af800640 1 -- 172.21.15.47:0/351536377 >> 172.21.15.47:0/351536377 conn(0x7f69a80d1bf0 msgr2=0x7f69a80d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:00.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69af800640 1 -- 172.21.15.47:0/351536377 shutdown_connections 2024-02-19T15:29:00.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69af800640 1 -- 172.21.15.47:0/351536377 wait complete. 2024-02-19T15:29:00.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69af800640 1 Processor -- start 2024-02-19T15:29:00.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69af800640 1 -- start start 2024-02-19T15:29:00.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69af800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69a806d650 0x7f69a80cb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69af800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80cded0 0x7f69a81592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69af800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69a80ced10 0x7f69a80b5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69af800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a815c890 con 0x7f69a806d650 2024-02-19T15:29:00.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69af800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a80cb6c0 con 0x7f69a80cded0 2024-02-19T15:29:00.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69af800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a80cc190 con 0x7f69a80ced10 2024-02-19T15:29:00.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69ace00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69a806d650 0x7f69a80cb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69ace00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69a806d650 0x7f69a80cb180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51178/0 (socket says 172.21.15.47:51178) 2024-02-19T15:29:00.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69ace00640 1 -- 172.21.15.47:0/4069801541 learned_addr learned my addr 172.21.15.47:0/4069801541 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:00.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f69ad800640 1 --2- 172.21.15.47:0/4069801541 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69a80ced10 0x7f69a80b5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.950+0000 7f699fe00640 1 --2- 172.21.15.47:0/4069801541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80cded0 0x7f69a81592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.954+0000 7f69ace00640 1 -- 172.21.15.47:0/4069801541 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69a80ced10 msgr2=0x7f69a80b5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:00.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.954+0000 7f69ace00640 1 --2- 172.21.15.47:0/4069801541 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69a80ced10 0x7f69a80b5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.954+0000 7f69ace00640 1 -- 172.21.15.47:0/4069801541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80cded0 msgr2=0x7f69a81592f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:00.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.954+0000 7f69ace00640 1 --2- 172.21.15.47:0/4069801541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80cded0 0x7f69a81592f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.954+0000 7f69ace00640 1 -- 172.21.15.47:0/4069801541 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6998009420 con 0x7f69a806d650 2024-02-19T15:29:00.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.954+0000 7f69ace00640 1 --2- 172.21.15.47:0/4069801541 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69a806d650 0x7f69a80cb180 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f6998019030 tx=0x7f699801ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:00.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.954+0000 7f69a5600640 1 -- 172.21.15.47:0/4069801541 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69980abe10 con 0x7f69a806d650 2024-02-19T15:29:00.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.954+0000 7f69a5600640 1 -- 172.21.15.47:0/4069801541 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69980bf410 con 0x7f69a806d650 2024-02-19T15:29:00.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.954+0000 7f69a5600640 1 -- 172.21.15.47:0/4069801541 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69980bf7c0 con 0x7f69a806d650 2024-02-19T15:29:00.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.954+0000 7f69af800640 1 -- 172.21.15.47:0/4069801541 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a815de70 con 0x7f69a806d650 2024-02-19T15:29:00.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.954+0000 7f69af800640 1 -- 172.21.15.47:0/4069801541 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a80cbbd0 con 0x7f69a806d650 2024-02-19T15:29:00.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.954+0000 7f69a5600640 1 -- 172.21.15.47:0/4069801541 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f69980b5400 con 0x7f69a806d650 2024-02-19T15:29:00.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.954+0000 7f69a5600640 1 --2- 172.21.15.47:0/4069801541 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6984066e40 0x7f6984069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.954+0000 7f69a5600640 1 -- 172.21.15.47:0/4069801541 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f699811eff0 con 0x7f69a806d650 2024-02-19T15:29:00.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f699fe00640 1 --2- 172.21.15.47:0/4069801541 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6984066e40 0x7f6984069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69b2c72000 1 Processor -- start 2024-02-19T15:29:00.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69b2c72000 1 -- start start 2024-02-19T15:29:00.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f699fe00640 1 --2- 172.21.15.47:0/4069801541 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6984066e40 0x7f6984069300 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f69a815c620 tx=0x7f6990009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:00.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69b2c72000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd73d920 0x562afd73dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69b2c72000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73e240 0x562afd6f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69b2c72000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 0x562afd6f4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69b2c72000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x562afd5bbd00 con 0x562afd6f3f30 2024-02-19T15:29:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69b2c72000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562afd54d430 con 0x562afd73d920 2024-02-19T15:29:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69b2c72000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x562afd54a960 con 0x562afd73e240 2024-02-19T15:29:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69ad800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 0x562afd6f4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69ad800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 0x562afd6f4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51188/0 (socket says 172.21.15.47:51188) 2024-02-19T15:29:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69ad800640 1 -- 172.21.15.47:0/525223869 learned_addr learned my addr 172.21.15.47:0/525223869 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69ace00640 1 --2- 172.21.15.47:0/525223869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd73d920 0x562afd73dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f699fe00640 1 --2- 172.21.15.47:0/525223869 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73e240 0x562afd6f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69ad800640 1 -- 172.21.15.47:0/525223869 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73e240 msgr2=0x562afd6f39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69ad800640 1 --2- 172.21.15.47:0/525223869 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73e240 0x562afd6f39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69ad800640 1 -- 172.21.15.47:0/525223869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd73d920 msgr2=0x562afd73dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69ad800640 1 --2- 172.21.15.47:0/525223869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd73d920 0x562afd73dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69ad800640 1 -- 172.21.15.47:0/525223869 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562afd23c890 con 0x562afd6f3f30 2024-02-19T15:29:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69ace00640 1 --2- 172.21.15.47:0/525223869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd73d920 0x562afd73dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:29:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69ad800640 1 --2- 172.21.15.47:0/525223869 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 0x562afd6f4310 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f69a00021f0 tx=0x7f69a00074d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d93abf07dabfbfa0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:00.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f699d600640 1 -- 172.21.15.47:0/525223869 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a0001ea0 con 0x562afd6f3f30 2024-02-19T15:29:00.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f699d600640 1 -- 172.21.15.47:0/525223869 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a0011580 con 0x562afd6f3f30 2024-02-19T15:29:00.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f699d600640 1 -- 172.21.15.47:0/525223869 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a0011910 con 0x562afd6f3f30 2024-02-19T15:29:00.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69b2c72000 1 -- 172.21.15.47:0/525223869 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 msgr2=0x562afd6f4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:00.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69b2c72000 1 --2- 172.21.15.47:0/525223869 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 0x562afd6f4310 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f69a00021f0 tx=0x7f69a00074d0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69b2c72000 1 -- 172.21.15.47:0/525223869 shutdown_connections 2024-02-19T15:29:00.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69b2c72000 1 --2- 172.21.15.47:0/525223869 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 0x562afd6f4310 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69b2c72000 1 --2- 172.21.15.47:0/525223869 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73e240 0x562afd6f39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69b2c72000 1 --2- 172.21.15.47:0/525223869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd73d920 0x562afd73dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69b2c72000 1 -- 172.21.15.47:0/525223869 >> 172.21.15.47:0/525223869 conn(0x562afd6fa2a0 msgr2=0x562afd6f22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:00.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69b2c72000 1 -- 172.21.15.47:0/525223869 shutdown_connections 2024-02-19T15:29:00.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.958+0000 7f69b2c72000 1 -- 172.21.15.47:0/525223869 wait complete. 2024-02-19T15:29:00.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69b2c72000 1 -- start start 2024-02-19T15:29:00.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69b2c72000 1 Processor -- start 2024-02-19T15:29:00.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69b2c72000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 0x562afd6f8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69b2c72000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73d920 0x562afd6f8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69b2c72000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd73e240 0x562afd73e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69b2c72000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x562afd546140 con 0x562afd6f3f30 2024-02-19T15:29:00.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69b2c72000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562afd4e0630 con 0x562afd73e240 2024-02-19T15:29:00.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69b2c72000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x562afd4e02d0 con 0x562afd73d920 2024-02-19T15:29:00.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69b2c72000 10 client.0 _ll_register_callbacks cb 0x562afd5f61c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:29:00.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69ace00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 0x562afd6f8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69ace00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 0x562afd6f8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51190/0 (socket says 172.21.15.47:51190) 2024-02-19T15:29:00.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69ace00640 1 -- 172.21.15.47:0/2109345013 learned_addr learned my addr 172.21.15.47:0/2109345013 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:00.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f699fe00640 1 --2- 172.21.15.47:0/2109345013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73d920 0x562afd6f8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69ace00640 1 -- 172.21.15.47:0/2109345013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73d920 msgr2=0x562afd6f8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:00.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69ad800640 1 --2- 172.21.15.47:0/2109345013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd73e240 0x562afd73e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69ace00640 1 --2- 172.21.15.47:0/2109345013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73d920 0x562afd6f8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69ace00640 1 -- 172.21.15.47:0/2109345013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd73e240 msgr2=0x562afd73e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:00.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69ace00640 1 --2- 172.21.15.47:0/2109345013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd73e240 0x562afd73e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69ace00640 1 -- 172.21.15.47:0/2109345013 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f698c003530 con 0x562afd6f3f30 2024-02-19T15:29:00.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f699fe00640 1 --2- 172.21.15.47:0/2109345013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73d920 0x562afd6f8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:29:00.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69ad800640 1 --2- 172.21.15.47:0/2109345013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd73e240 0x562afd73e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:29:00.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69ace00640 1 --2- 172.21.15.47:0/2109345013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 0x562afd6f8530 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f69980ec080 tx=0x7f69980ef640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e119dae74c9a980 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:00.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f6996000640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:29:00.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f6996000640 1 -- 172.21.15.47:0/2109345013 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69980f0ea0 con 0x562afd6f3f30 2024-02-19T15:29:00.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f6996000640 1 -- 172.21.15.47:0/2109345013 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69980f0080 con 0x562afd6f3f30 2024-02-19T15:29:00.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69b2c72000 1 -- 172.21.15.47:0/2109345013 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562afd4f9850 con 0x562afd6f3f30 2024-02-19T15:29:00.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f6996000640 1 -- 172.21.15.47:0/2109345013 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69980f7df0 con 0x562afd6f3f30 2024-02-19T15:29:00.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69b2c72000 10 client.8496 fetch_fsmap learned FSMap version 91 2024-02-19T15:29:00.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69b2c72000 1 -- 172.21.15.47:0/2109345013 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=2+}) v3 -- 0x562afd505e70 con 0x562afd6f3f30 2024-02-19T15:29:00.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f6996000640 1 -- 172.21.15.47:0/2109345013 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562afd4f9850 con 0x562afd6f3f30 2024-02-19T15:29:00.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f6996000640 1 -- 172.21.15.47:0/2109345013 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 91) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f69980c8020 con 0x562afd6f3f30 2024-02-19T15:29:00.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69b2c72000 10 client.8496 fetch_fsmap finished waiting for FSMap version 91 2024-02-19T15:29:00.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69b2c72000 10 client.8496 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-19T15:29:00.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69b2c72000 20 client.8496 populate_metadata read hostname 'smithi047' 2024-02-19T15:29:00.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69b2c72000 1 --2- 172.21.15.47:0/2109345013 >> [v2:172.21.15.47:6833/4036766514,v1:172.21.15.47:6835/4036766514] conn(0x562afd73ed70 0x562afd73f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69b2c72000 4 client.8496 mds_command: new command op to 8361 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T15:29:00.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f69b2c72000 1 -- 172.21.15.47:0/2109345013 --> [v2:172.21.15.47:6833/4036766514,v1:172.21.15.47:6835/4036766514] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562afd542f20 con 0x562afd73ed70 2024-02-19T15:29:00.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f699fe00640 1 --2- 172.21.15.47:0/2109345013 >> [v2:172.21.15.47:6833/4036766514,v1:172.21.15.47:6835/4036766514] conn(0x562afd73ed70 0x562afd73f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f6996000640 10 client.8496 ms_handle_connect on v2:172.21.15.47:6833/4036766514 2024-02-19T15:29:00.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.962+0000 7f699fe00640 1 --2- 172.21.15.47:0/2109345013 >> [v2:172.21.15.47:6833/4036766514,v1:172.21.15.47:6835/4036766514] conn(0x562afd73ed70 0x562afd73f150 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f1cda88822bc8fe server_cookie=3aa63b088443296a in_seq=0 out_seq=0 2024-02-19T15:29:00.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.966+0000 7f6996000640 1 -- 172.21.15.47:0/2109345013 <== mds.0 v2:172.21.15.47:6833/4036766514 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562afd542f20 con 0x562afd73ed70 2024-02-19T15:29:00.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.966+0000 7f6996000640 10 client.8496 handle_command_reply: tid=0 2024-02-19T15:29:00.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.966+0000 7f69b2c72000 1 client.8496 shutdown 2024-02-19T15:29:00.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.966+0000 7f69b2c72000 1 -- 172.21.15.47:0/2109345013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 msgr2=0x562afd6f8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:00.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.966+0000 7f69b2c72000 1 --2- 172.21.15.47:0/2109345013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 0x562afd6f8530 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f69980ec080 tx=0x7f69980ef640 comp rx=0 tx=0).stop 2024-02-19T15:29:00.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.966+0000 7f69b2c72000 1 -- 172.21.15.47:0/2109345013 shutdown_connections 2024-02-19T15:29:00.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.966+0000 7f69b2c72000 1 --2- 172.21.15.47:0/2109345013 >> [v2:172.21.15.47:6833/4036766514,v1:172.21.15.47:6835/4036766514] conn(0x562afd73ed70 0x562afd73f150 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.966+0000 7f69b2c72000 1 --2- 172.21.15.47:0/2109345013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd73e240 0x562afd73e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.966+0000 7f69b2c72000 1 --2- 172.21.15.47:0/2109345013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73d920 0x562afd6f8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.966+0000 7f6996000640 0 client.8496 ms_handle_reset on v2:172.21.15.47:6833/4036766514 2024-02-19T15:29:00.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.966+0000 7f69b2c72000 1 --2- 172.21.15.47:0/2109345013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 0x562afd6f8530 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.966+0000 7f69b2c72000 1 -- 172.21.15.47:0/2109345013 >> 172.21.15.47:0/2109345013 conn(0x562afd6fa2a0 msgr2=0x562afd6f29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:00.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.966+0000 7f69b2c72000 1 -- 172.21.15.47:0/2109345013 shutdown_connections 2024-02-19T15:29:00.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.966+0000 7f69b2c72000 1 -- 172.21.15.47:0/2109345013 wait complete. 2024-02-19T15:29:00.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.966+0000 7f69b2c72000 20 client.8496 trim_cache size 0 max 16384 2024-02-19T15:29:00.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.982+0000 7f69b2c72000 1 Processor -- start 2024-02-19T15:29:00.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.982+0000 7f69b2c72000 1 -- start start 2024-02-19T15:29:00.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.982+0000 7f69b2c72000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd6f3f30 0x562afd79e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.982+0000 7f69b2c72000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd73d920 0x562afd79eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.982+0000 7f69b2c72000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73e240 0x562afd79f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.982+0000 7f69b2c72000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x562afd1fd940 con 0x562afd73d920 2024-02-19T15:29:00.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.982+0000 7f69b2c72000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562afd4de660 con 0x562afd6f3f30 2024-02-19T15:29:00.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.982+0000 7f69b2c72000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x562afd73cf10 con 0x562afd73e240 2024-02-19T15:29:00.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.982+0000 7f699fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd73d920 0x562afd79eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.982+0000 7f699fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd73d920 0x562afd79eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51204/0 (socket says 172.21.15.47:51204) 2024-02-19T15:29:00.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.982+0000 7f699fe00640 1 -- 172.21.15.47:0/4083516575 learned_addr learned my addr 172.21.15.47:0/4083516575 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:00.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.982+0000 7f69ad800640 1 --2- 172.21.15.47:0/4083516575 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73e240 0x562afd79f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.982+0000 7f69ace00640 1 --2- 172.21.15.47:0/4083516575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd6f3f30 0x562afd79e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.982+0000 7f699fe00640 1 -- 172.21.15.47:0/4083516575 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73e240 msgr2=0x562afd79f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:00.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.982+0000 7f699fe00640 1 --2- 172.21.15.47:0/4083516575 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73e240 0x562afd79f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.982+0000 7f699fe00640 1 -- 172.21.15.47:0/4083516575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd6f3f30 msgr2=0x562afd79e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:00.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.982+0000 7f699fe00640 1 --2- 172.21.15.47:0/4083516575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd6f3f30 0x562afd79e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.982+0000 7f699fe00640 1 -- 172.21.15.47:0/4083516575 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562afd4ebe10 con 0x562afd73d920 2024-02-19T15:29:00.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.982+0000 7f69ad800640 1 --2- 172.21.15.47:0/4083516575 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73e240 0x562afd79f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:29:00.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.982+0000 7f69ace00640 1 --2- 172.21.15.47:0/4083516575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd6f3f30 0x562afd79e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:29:00.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f699fe00640 1 --2- 172.21.15.47:0/4083516575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd73d920 0x562afd79eac0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x562afd51bb00 tx=0x7f6990012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ae836d25b673e1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:00.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f699cc00640 1 -- 172.21.15.47:0/4083516575 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69900013c0 con 0x562afd73d920 2024-02-19T15:29:00.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f699cc00640 1 -- 172.21.15.47:0/4083516575 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6990027d20 con 0x562afd73d920 2024-02-19T15:29:00.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f699cc00640 1 -- 172.21.15.47:0/4083516575 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6990027350 con 0x562afd73d920 2024-02-19T15:29:00.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69b2c72000 1 -- 172.21.15.47:0/4083516575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd73d920 msgr2=0x562afd79eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:00.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69b2c72000 1 --2- 172.21.15.47:0/4083516575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd73d920 0x562afd79eac0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x562afd51bb00 tx=0x7f6990012000 comp rx=0 tx=0).stop 2024-02-19T15:29:00.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69b2c72000 1 -- 172.21.15.47:0/4083516575 shutdown_connections 2024-02-19T15:29:00.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69b2c72000 1 --2- 172.21.15.47:0/4083516575 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73e240 0x562afd79f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69b2c72000 1 --2- 172.21.15.47:0/4083516575 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd73d920 0x562afd79eac0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69b2c72000 1 --2- 172.21.15.47:0/4083516575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd6f3f30 0x562afd79e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69b2c72000 1 -- 172.21.15.47:0/4083516575 >> 172.21.15.47:0/4083516575 conn(0x562afd6fa2a0 msgr2=0x562afd7500a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:00.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69b2c72000 1 -- 172.21.15.47:0/4083516575 shutdown_connections 2024-02-19T15:29:00.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69b2c72000 1 -- 172.21.15.47:0/4083516575 wait complete. 2024-02-19T15:29:00.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69b2c72000 1 -- start start 2024-02-19T15:29:00.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69b2c72000 1 Processor -- start 2024-02-19T15:29:00.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69b2c72000 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 0x562afd786d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69b2c72000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd73d920 0x562afd7876c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69b2c72000 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73e240 0x562afd787c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69b2c72000 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x562afd50b2b0 con 0x562afd6f3f30 2024-02-19T15:29:00.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69b2c72000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562afd54a140 con 0x562afd73d920 2024-02-19T15:29:00.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69b2c72000 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x562afd54ec80 con 0x562afd73e240 2024-02-19T15:29:00.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69b2c72000 10 client.0 _ll_register_callbacks cb 0x7f69a0001900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:29:00.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69ace00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 0x562afd786d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69ace00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 0x562afd786d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51208/0 (socket says 172.21.15.47:51208) 2024-02-19T15:29:00.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69ace00640 1 -- 172.21.15.47:0/160669584 learned_addr learned my addr 172.21.15.47:0/160669584 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:00.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69ad800640 1 --2- 172.21.15.47:0/160669584 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73e240 0x562afd787c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f699fe00640 1 --2- 172.21.15.47:0/160669584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd73d920 0x562afd7876c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69ace00640 1 -- 172.21.15.47:0/160669584 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73e240 msgr2=0x562afd787c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:00.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69ace00640 1 --2- 172.21.15.47:0/160669584 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73e240 0x562afd787c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69ace00640 1 -- 172.21.15.47:0/160669584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd73d920 msgr2=0x562afd7876c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:00.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69ace00640 1 --2- 172.21.15.47:0/160669584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd73d920 0x562afd7876c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.986+0000 7f69ace00640 1 -- 172.21.15.47:0/160669584 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6990019040 con 0x562afd6f3f30 2024-02-19T15:29:00.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69ace00640 1 --2- 172.21.15.47:0/160669584 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 0x562afd786d70 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f699801c990 tx=0x7f699811d420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67cc87122a3ef3eb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:00.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f6996000640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-02-19T15:29:00.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f6996000640 1 -- 172.21.15.47:0/160669584 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69980f0ea0 con 0x562afd6f3f30 2024-02-19T15:29:00.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f6996000640 1 -- 172.21.15.47:0/160669584 <== mon.0 v2:172.21.15.47:3300/0 2 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69980ef070 con 0x562afd6f3f30 2024-02-19T15:29:00.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 1 -- 172.21.15.47:0/160669584 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562afd74a500 con 0x562afd6f3f30 2024-02-19T15:29:00.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f6996000640 1 -- 172.21.15.47:0/160669584 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69980ef230 con 0x562afd6f3f30 2024-02-19T15:29:00.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 10 client.8502 fetch_fsmap learned FSMap version 91 2024-02-19T15:29:00.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 1 -- 172.21.15.47:0/160669584 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=2+}) v3 -- 0x562afd23ccc0 con 0x562afd6f3f30 2024-02-19T15:29:00.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f6996000640 1 -- 172.21.15.47:0/160669584 <== mon.0 v2:172.21.15.47:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562afd74a500 con 0x562afd6f3f30 2024-02-19T15:29:00.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f6996000640 1 -- 172.21.15.47:0/160669584 <== mon.0 v2:172.21.15.47:3300/0 5 ==== fsmap(e 91) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f69980c8040 con 0x562afd6f3f30 2024-02-19T15:29:00.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 10 client.8502 fetch_fsmap finished waiting for FSMap version 91 2024-02-19T15:29:00.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 10 client.8502 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-19T15:29:00.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 20 client.8502 populate_metadata read hostname 'smithi047' 2024-02-19T15:29:00.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 1 --2- 172.21.15.47:0/160669584 >> [v2:172.21.15.47:6833/4036766514,v1:172.21.15.47:6835/4036766514] conn(0x562afd788140 0x562afd788520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:00.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 4 client.8502 mds_command: new command op to 8361 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T15:29:00.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 1 -- 172.21.15.47:0/160669584 --> [v2:172.21.15.47:6833/4036766514,v1:172.21.15.47:6835/4036766514] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562afd4cbe80 con 0x562afd788140 2024-02-19T15:29:00.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f699fe00640 1 --2- 172.21.15.47:0/160669584 >> [v2:172.21.15.47:6833/4036766514,v1:172.21.15.47:6835/4036766514] conn(0x562afd788140 0x562afd788520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:00.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f699fe00640 1 --2- 172.21.15.47:0/160669584 >> [v2:172.21.15.47:6833/4036766514,v1:172.21.15.47:6835/4036766514] conn(0x562afd788140 0x562afd788520 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=409d2178158d5d90 server_cookie=56adc22d7898c929 in_seq=0 out_seq=0 2024-02-19T15:29:00.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f6996000640 10 client.8502 ms_handle_connect on v2:172.21.15.47:6833/4036766514 2024-02-19T15:29:00.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f6996000640 1 -- 172.21.15.47:0/160669584 <== mds.0 v2:172.21.15.47:6833/4036766514 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x562afd4cbe80 con 0x562afd788140 2024-02-19T15:29:00.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f6996000640 10 client.8502 handle_command_reply: tid=0 2024-02-19T15:29:00.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 1 client.8502 shutdown 2024-02-19T15:29:00.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 1 -- 172.21.15.47:0/160669584 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 msgr2=0x562afd786d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:00.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 1 --2- 172.21.15.47:0/160669584 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 0x562afd786d70 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f699801c990 tx=0x7f699811d420 comp rx=0 tx=0).stop 2024-02-19T15:29:00.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 1 -- 172.21.15.47:0/160669584 shutdown_connections 2024-02-19T15:29:00.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 1 --2- 172.21.15.47:0/160669584 >> [v2:172.21.15.47:6833/4036766514,v1:172.21.15.47:6835/4036766514] conn(0x562afd788140 0x562afd788520 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 1 --2- 172.21.15.47:0/160669584 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x562afd73e240 0x562afd787c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 1 --2- 172.21.15.47:0/160669584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562afd73d920 0x562afd7876c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f6996000640 0 client.8502 ms_handle_reset on v2:172.21.15.47:6833/4036766514 2024-02-19T15:29:00.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 1 --2- 172.21.15.47:0/160669584 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x562afd6f3f30 0x562afd786d70 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 1 -- 172.21.15.47:0/160669584 >> 172.21.15.47:0/160669584 conn(0x562afd6fa2a0 msgr2=0x562afd7507b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:00.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 1 -- 172.21.15.47:0/160669584 shutdown_connections 2024-02-19T15:29:00.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 1 -- 172.21.15.47:0/160669584 wait complete. 2024-02-19T15:29:00.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69b2c72000 20 client.8502 trim_cache size 0 max 16384 2024-02-19T15:29:00.995 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-02-19T15:29:00.995 INFO:teuthology.orchestra.run.smithi047.stdout: "cluster_fsid": "ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e", 2024-02-19T15:29:00.995 INFO:teuthology.orchestra.run.smithi047.stdout: "whoami": 0, 2024-02-19T15:29:00.995 INFO:teuthology.orchestra.run.smithi047.stdout: "id": 8361, 2024-02-19T15:29:00.995 INFO:teuthology.orchestra.run.smithi047.stdout: "want_state": "up:active", 2024-02-19T15:29:00.995 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "up:active", 2024-02-19T15:29:00.995 INFO:teuthology.orchestra.run.smithi047.stdout: "fs_name": "cephfs", 2024-02-19T15:29:00.995 INFO:teuthology.orchestra.run.smithi047.stdout: "rank_uptime": 5.9992435869999996, 2024-02-19T15:29:00.996 INFO:teuthology.orchestra.run.smithi047.stdout: "mdsmap_epoch": 91, 2024-02-19T15:29:00.996 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch": 125, 2024-02-19T15:29:00.996 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap_epoch_barrier": 125, 2024-02-19T15:29:00.996 INFO:teuthology.orchestra.run.smithi047.stdout: "uptime": 10.831821992 2024-02-19T15:29:00.996 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-02-19T15:29:00.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69af800640 1 -- 172.21.15.47:0/4069801541 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6984066e40 msgr2=0x7f6984069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:00.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69af800640 1 --2- 172.21.15.47:0/4069801541 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6984066e40 0x7f6984069300 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f69a815c620 tx=0x7f6990009040 comp rx=0 tx=0).stop 2024-02-19T15:29:00.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69af800640 1 -- 172.21.15.47:0/4069801541 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69a806d650 msgr2=0x7f69a80cb180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:00.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69af800640 1 --2- 172.21.15.47:0/4069801541 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69a806d650 0x7f69a80cb180 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f6998019030 tx=0x7f699801ac60 comp rx=0 tx=0).stop 2024-02-19T15:29:00.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69af800640 1 -- 172.21.15.47:0/4069801541 shutdown_connections 2024-02-19T15:29:00.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69af800640 1 --2- 172.21.15.47:0/4069801541 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6984066e40 0x7f6984069300 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69af800640 1 --2- 172.21.15.47:0/4069801541 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69a80ced10 0x7f69a80b5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69af800640 1 --2- 172.21.15.47:0/4069801541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80cded0 0x7f69a81592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69af800640 1 --2- 172.21.15.47:0/4069801541 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f69a806d650 0x7f69a80cb180 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:00.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69af800640 1 -- 172.21.15.47:0/4069801541 >> 172.21.15.47:0/4069801541 conn(0x7f69a80d1bf0 msgr2=0x7f69a80d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:00.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69af800640 1 -- 172.21.15.47:0/4069801541 shutdown_connections 2024-02-19T15:29:00.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:00.990+0000 7f69af800640 1 -- 172.21.15.47:0/4069801541 wait complete. 2024-02-19T15:29:01.010 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-02-19T15:29:01.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.214+0000 7fccfd200640 1 Processor -- start 2024-02-19T15:29:01.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.214+0000 7fccfd200640 1 -- start start 2024-02-19T15:29:01.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.218+0000 7fccfd200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccf8145590 0x7fccf80727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:01.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.218+0000 7fccfd200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fccf80cea80 0x7fccf806c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:01.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.218+0000 7fccfd200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf815d280 con 0x7fccf815c390 2024-02-19T15:29:01.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.218+0000 7fccfd200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf8070ff0 con 0x7fccf80cea80 2024-02-19T15:29:01.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.218+0000 7fccfd200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf80709e0 con 0x7fccf8145590 2024-02-19T15:29:01.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.218+0000 7fccefe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fccf80cea80 0x7fccf806c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:01.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.218+0000 7fccf5e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccf8145590 0x7fccf80727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:01.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.218+0000 7fccefe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fccf80cea80 0x7fccf806c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:57462/0 (socket says 172.21.15.47:57462) 2024-02-19T15:29:01.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.218+0000 7fccefe00640 1 -- 172.21.15.47:0/2109946898 learned_addr learned my addr 172.21.15.47:0/2109946898 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:01.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.218+0000 7fccf5e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccf8145590 0x7fccf80727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:36622/0 (socket says 172.21.15.47:36622) 2024-02-19T15:29:01.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.218+0000 7fccefe00640 1 -- 172.21.15.47:0/2109946898 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccf8145590 msgr2=0x7fccf80727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:01.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.218+0000 7fccefe00640 1 --2- 172.21.15.47:0/2109946898 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccf8145590 0x7fccf80727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:01.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.218+0000 7fccefe00640 1 -- 172.21.15.47:0/2109946898 >> v1:172.21.15.116:6789/0 conn(0x7fccf815c390 legacy=0x7fccf815c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:01.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.218+0000 7fccefe00640 1 -- 172.21.15.47:0/2109946898 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf806cee0 con 0x7fccf80cea80 2024-02-19T15:29:01.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.218+0000 7fccf5e00640 1 --2- 172.21.15.47:0/2109946898 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccf8145590 0x7fccf80727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:29:01.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.222+0000 7fccefe00640 1 --2- 172.21.15.47:0/2109946898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf80cea80 0x7fccf806c9a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fcce00098a0 tx=0x7fcce001c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fee27b9e12bdaee7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:01.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.222+0000 7fccf5400640 1 -- 172.21.15.47:0/2109946898 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce00a7070 con 0x7fccf80cea80 2024-02-19T15:29:01.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.222+0000 7fccf5400640 1 -- 172.21.15.47:0/2109946898 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce00ab430 con 0x7fccf80cea80 2024-02-19T15:29:01.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.222+0000 7fccf5400640 1 -- 172.21.15.47:0/2109946898 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce00ab7c0 con 0x7fccf80cea80 2024-02-19T15:29:01.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.222+0000 7fccfd200640 1 -- 172.21.15.47:0/2109946898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf80cea80 msgr2=0x7fccf806c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:01.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.222+0000 7fccfd200640 1 --2- 172.21.15.47:0/2109946898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf80cea80 0x7fccf806c9a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fcce00098a0 tx=0x7fcce001c540 comp rx=0 tx=0).stop 2024-02-19T15:29:01.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.222+0000 7fccfd200640 1 -- 172.21.15.47:0/2109946898 shutdown_connections 2024-02-19T15:29:01.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.222+0000 7fccfd200640 1 --2- 172.21.15.47:0/2109946898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf80cea80 0x7fccf806c9a0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:01.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.222+0000 7fccfd200640 1 --2- 172.21.15.47:0/2109946898 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccf8145590 0x7fccf80727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:01.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.222+0000 7fccfd200640 1 -- 172.21.15.47:0/2109946898 >> 172.21.15.47:0/2109946898 conn(0x7fccf80b16a0 msgr2=0x7fccf80b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:01.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.222+0000 7fccfd200640 1 -- 172.21.15.47:0/2109946898 shutdown_connections 2024-02-19T15:29:01.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.222+0000 7fccfd200640 1 -- 172.21.15.47:0/2109946898 wait complete. 2024-02-19T15:29:01.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.222+0000 7fccfd200640 1 Processor -- start 2024-02-19T15:29:01.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.222+0000 7fccfd200640 1 -- start start 2024-02-19T15:29:01.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccfd200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf80cea80 0x7fccf8136f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:01.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccfd200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccf8145590 0x7fccf8141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:01.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccfd200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccf815c390 0x7fccf8159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:01.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccfd200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf81425e0 con 0x7fccf815c390 2024-02-19T15:29:01.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccfd200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf81428c0 con 0x7fccf80cea80 2024-02-19T15:29:01.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccfd200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf8137490 con 0x7fccf8145590 2024-02-19T15:29:01.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccf6800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccf815c390 0x7fccf8159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:01.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccf6800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccf815c390 0x7fccf8159a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51218/0 (socket says 172.21.15.47:51218) 2024-02-19T15:29:01.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccf6800640 1 -- 172.21.15.47:0/357535130 learned_addr learned my addr 172.21.15.47:0/357535130 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:01.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccf5e00640 1 --2- 172.21.15.47:0/357535130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf80cea80 0x7fccf8136f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:01.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccefe00640 1 --2- 172.21.15.47:0/357535130 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccf8145590 0x7fccf8141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:01.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccf6800640 1 -- 172.21.15.47:0/357535130 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccf8145590 msgr2=0x7fccf8141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:01.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccf6800640 1 --2- 172.21.15.47:0/357535130 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccf8145590 0x7fccf8141310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:01.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccf6800640 1 -- 172.21.15.47:0/357535130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf80cea80 msgr2=0x7fccf8136f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:01.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccf6800640 1 --2- 172.21.15.47:0/357535130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf80cea80 0x7fccf8136f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:01.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccf6800640 1 -- 172.21.15.47:0/357535130 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce8009420 con 0x7fccf815c390 2024-02-19T15:29:01.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccf6800640 1 --2- 172.21.15.47:0/357535130 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccf815c390 0x7fccf8159a80 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fccf000c760 tx=0x7fccf000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:01.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccee000640 1 -- 172.21.15.47:0/357535130 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf0003b80 con 0x7fccf815c390 2024-02-19T15:29:01.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccee000640 1 -- 172.21.15.47:0/357535130 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf0003ce0 con 0x7fccf815c390 2024-02-19T15:29:01.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccee000640 1 -- 172.21.15.47:0/357535130 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf000aa30 con 0x7fccf815c390 2024-02-19T15:29:01.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccfd200640 1 -- 172.21.15.47:0/357535130 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce00094d0 con 0x7fccf815c390 2024-02-19T15:29:01.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccfd200640 1 -- 172.21.15.47:0/357535130 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf8137a10 con 0x7fccf815c390 2024-02-19T15:29:01.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccfd200640 1 -- 172.21.15.47:0/357535130 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf8142a50 con 0x7fccf815c390 2024-02-19T15:29:01.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.226+0000 7fccee000640 1 -- 172.21.15.47:0/357535130 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fccf000ab90 con 0x7fccf815c390 2024-02-19T15:29:01.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.230+0000 7fccee000640 1 --2- 172.21.15.47:0/357535130 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fccd4066e40 0x7fccd4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:01.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.230+0000 7fccee000640 1 -- 172.21.15.47:0/357535130 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fccf0084660 con 0x7fccf815c390 2024-02-19T15:29:01.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.230+0000 7fccf5e00640 1 --2- 172.21.15.47:0/357535130 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fccd4066e40 0x7fccd4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:01.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.230+0000 7fccf5e00640 1 --2- 172.21.15.47:0/357535130 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fccd4066e40 0x7fccd4069300 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fccf813ce30 tx=0x7fcce8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:01.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.234+0000 7fccee000640 1 -- 172.21.15.47:0/357535130 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fccf0051500 con 0x7fccf815c390 2024-02-19T15:29:01.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.430+0000 7fccfd200640 1 -- 172.21.15.47:0/357535130 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fccf8075360 con 0x7fccf815c390 2024-02-19T15:29:01.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.430+0000 7fccee000640 1 -- 172.21.15.47:0/357535130 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v91) v1 ==== 53+0+73 (secure 0 0 0) 0x7fccf0054bc0 con 0x7fccf815c390 2024-02-19T15:29:01.432 INFO:teuthology.orchestra.run.smithi047.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-02-19T15:29:01.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.434+0000 7fccfd200640 1 -- 172.21.15.47:0/357535130 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fccd4066e40 msgr2=0x7fccd4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:01.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.434+0000 7fccfd200640 1 --2- 172.21.15.47:0/357535130 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fccd4066e40 0x7fccd4069300 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fccf813ce30 tx=0x7fcce8009190 comp rx=0 tx=0).stop 2024-02-19T15:29:01.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.434+0000 7fccfd200640 1 -- 172.21.15.47:0/357535130 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccf815c390 msgr2=0x7fccf8159a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:01.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.434+0000 7fccfd200640 1 --2- 172.21.15.47:0/357535130 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccf815c390 0x7fccf8159a80 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fccf000c760 tx=0x7fccf000cc20 comp rx=0 tx=0).stop 2024-02-19T15:29:01.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.434+0000 7fccfd200640 1 -- 172.21.15.47:0/357535130 shutdown_connections 2024-02-19T15:29:01.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.434+0000 7fccfd200640 1 --2- 172.21.15.47:0/357535130 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fccd4066e40 0x7fccd4069300 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:01.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.434+0000 7fccfd200640 1 --2- 172.21.15.47:0/357535130 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccf815c390 0x7fccf8159a80 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:01.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.434+0000 7fccfd200640 1 --2- 172.21.15.47:0/357535130 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccf8145590 0x7fccf8141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:01.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.434+0000 7fccfd200640 1 --2- 172.21.15.47:0/357535130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf80cea80 0x7fccf8136f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:01.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.434+0000 7fccfd200640 1 -- 172.21.15.47:0/357535130 >> 172.21.15.47:0/357535130 conn(0x7fccf80b16a0 msgr2=0x7fccf80cdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:01.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.434+0000 7fccfd200640 1 -- 172.21.15.47:0/357535130 shutdown_connections 2024-02-19T15:29:01.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:01.434+0000 7fccfd200640 1 -- 172.21.15.47:0/357535130 wait complete. 2024-02-19T15:29:01.460 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-02-19T15:29:01.461 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T15:29:01.461 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi047.front.sepia.ceph.com 2024-02-19T15:29:01.461 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-19T15:29:01.461 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-19T15:29:01.461 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-02-19T15:29:01.461 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-02-19T15:29:01.461 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T15:29:01.461 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:29:01.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:29:01.461 DEBUG:teuthology.orchestra.run.smithi047:> ip addr 2024-02-19T15:29:01.472 INFO:teuthology.orchestra.run.smithi047.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T15:29:01.472 INFO:teuthology.orchestra.run.smithi047.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T15:29:01.472 INFO:teuthology.orchestra.run.smithi047.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T15:29:01.472 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:29:01.472 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 ::1/128 scope host 2024-02-19T15:29:01.472 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:29:01.472 INFO:teuthology.orchestra.run.smithi047.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:29:01.473 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:78 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:29:01.473 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f0 2024-02-19T15:29:01.473 INFO:teuthology.orchestra.run.smithi047.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T15:29:01.473 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8e brd ff:ff:ff:ff:ff:ff 2024-02-19T15:29:01.473 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f0 2024-02-19T15:29:01.473 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.21.15.47/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-02-19T15:29:01.473 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft 42073sec preferred_lft 42073sec 2024-02-19T15:29:01.473 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::ec4:7aff:febd:148e/64 scope link 2024-02-19T15:29:01.473 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:29:01.473 INFO:teuthology.orchestra.run.smithi047.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:29:01.473 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:6b:ff:79 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:29:01.473 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp7s0f1 2024-02-19T15:29:01.473 INFO:teuthology.orchestra.run.smithi047.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-02-19T15:29:01.473 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0c:c4:7a:bd:14:8f brd ff:ff:ff:ff:ff:ff 2024-02-19T15:29:01.474 INFO:teuthology.orchestra.run.smithi047.stdout: altname enp1s0f1 2024-02-19T15:29:01.474 INFO:teuthology.orchestra.run.smithi047.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-02-19T15:29:01.474 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 02:42:47:cf:b1:62 brd ff:ff:ff:ff:ff:ff 2024-02-19T15:29:01.474 INFO:teuthology.orchestra.run.smithi047.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-02-19T15:29:01.474 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:29:01.474 INFO:teuthology.orchestra.run.smithi047.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-02-19T15:29:01.474 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-02-19T15:29:01.474 INFO:teuthology.orchestra.run.smithi047.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-02-19T15:29:01.474 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:29:01.474 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-02-19T15:29:01.474 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:29:01.474 INFO:teuthology.orchestra.run.smithi047.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:29:01.474 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-19T15:29:01.474 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-02-19T15:29:01.474 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:29:01.475 INFO:teuthology.orchestra.run.smithi047.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-02-19T15:29:01.475 INFO:teuthology.orchestra.run.smithi047.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-02-19T15:29:01.475 INFO:teuthology.orchestra.run.smithi047.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-02-19T15:29:01.475 INFO:teuthology.orchestra.run.smithi047.stdout: valid_lft forever preferred_lft forever 2024-02-19T15:29:01.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:29:01.475 DEBUG:teuthology.orchestra.run.smithi047:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T15:29:01.539 INFO:teuthology.orchestra.run.smithi047.stdout:1 2024-02-19T15:29:01.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:29:01.541 DEBUG:teuthology.orchestra.run.smithi047:> route 2024-02-19T15:29:01.593 INFO:teuthology.orchestra.run.smithi047.stdout:Kernel IP routing table 2024-02-19T15:29:01.593 INFO:teuthology.orchestra.run.smithi047.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T15:29:01.593 INFO:teuthology.orchestra.run.smithi047.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-02-19T15:29:01.593 INFO:teuthology.orchestra.run.smithi047.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-02-19T15:29:01.593 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-02-19T15:29:01.593 INFO:teuthology.orchestra.run.smithi047.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T15:29:01.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:29:01.593 DEBUG:teuthology.orchestra.run.smithi047:> (cd / && exec bash -c ' 2024-02-19T15:29:01.593 DEBUG:teuthology.orchestra.run.smithi047:> set -e 2024-02-19T15:29:01.593 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-02-19T15:29:01.594 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-02-19T15:29:01.594 DEBUG:teuthology.orchestra.run.smithi047:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-02-19T15:29:01.594 DEBUG:teuthology.orchestra.run.smithi047:> ') 2024-02-19T15:29:01.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:29:01.685 DEBUG:teuthology.orchestra.run.smithi047:> ip netns list 2024-02-19T15:29:01.697 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-02-19T15:29:01.697 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-02-19T15:29:01.697 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:29:01.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:29:01.698 DEBUG:teuthology.orchestra.run.smithi047:> sudo ip link set brx.0 up 2024-02-19T15:29:01.759 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:29:01.759 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-19T15:29:01.808 INFO:teuthology.orchestra.run.smithi047.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-19T15:29:01.808 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T15:29:01.808 DEBUG:teuthology.orchestra.run.smithi047:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:29:01.859 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-19T15:29:01.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:29:01.859 DEBUG:teuthology.orchestra.run.smithi047:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:29:01.948 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:29:01.948 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:29:01.948 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:29:01.963 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:29:01.963 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:29:01.963 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:29:01.963 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:29:01.963 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:29:01.973 INFO:teuthology.orchestra.run.smithi047.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T15:29:01.973 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T15:29:01.973 INFO:teuthology.orchestra.run.smithi047.stdout:invalid new device string format 2024-02-19T15:29:01.973 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:3300,172.21.15.116:3300,172.21.15.116:3301" 2024-02-19T15:29:01.973 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T15:29:01.973 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: resolved to: "172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790" 2024-02-19T15:29:01.973 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.47:6789,172.21.15.116:6789,172.21.15.116:6790:/ 2024-02-19T15:29:01.973 INFO:teuthology.orchestra.run.smithi047.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e" will pass to kernel 2024-02-19T15:29:01.978 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T15:29:01.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:29:01.978 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-19T15:29:01.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:29:01.998 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T15:29:02.060 INFO:teuthology.orchestra.run.smithi047.stdout:client8508 2024-02-19T15:29:02.061 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:29:02.061 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:29:02.116 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-02-19T15:29:02.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.306+0000 7f0222400640 1 Processor -- start 2024-02-19T15:29:02.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.306+0000 7f0222400640 1 -- start start 2024-02-19T15:29:02.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.306+0000 7f0222400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f021c0d2650 0x7f021c06c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:02.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.306+0000 7f0222400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f021c15e280 0x7f021c06c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:02.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.306+0000 7f0222400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f021c0cc620 con 0x7f021c0ce3b0 2024-02-19T15:29:02.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.306+0000 7f0222400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f021c0d12e0 con 0x7f021c15e280 2024-02-19T15:29:02.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.306+0000 7f0222400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f021c0d1940 con 0x7f021c0d2650 2024-02-19T15:29:02.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.306+0000 7f021b400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f021c0ce3b0 0x7f021c0ce790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41186/0 (socket says 172.21.15.47:41186) 2024-02-19T15:29:02.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.306+0000 7f021b400640 1 -- 172.21.15.47:0/33695299 learned_addr learned my addr 172.21.15.47:0/33695299 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:29:02.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.306+0000 7f021be00640 1 --2- 172.21.15.47:0/33695299 >> v2:172.21.15.116:3300/0 conn(0x7f021c15e280 0x7f021c06c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:02.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.306+0000 7f0213e00640 1 --2- 172.21.15.47:0/33695299 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f021c0d2650 0x7f021c06c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:02.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.306+0000 7f021aa00640 1 -- 172.21.15.47:0/33695299 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052607040 0 0) 0x7f021c0cc620 con 0x7f021c0ce3b0 2024-02-19T15:29:02.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.306+0000 7f021aa00640 1 -- 172.21.15.47:0/33695299 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0200003530 con 0x7f021c0ce3b0 2024-02-19T15:29:02.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.306+0000 7f021be00640 1 -- 172.21.15.47:0/33695299 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f021c0d2650 msgr2=0x7f021c06c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:02.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.306+0000 7f021be00640 1 --2- 172.21.15.47:0/33695299 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f021c0d2650 0x7f021c06c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:02.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.306+0000 7f021be00640 1 -- 172.21.15.47:0/33695299 >> v1:172.21.15.47:6789/0 conn(0x7f021c0ce3b0 legacy=0x7f021c0ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:02.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.306+0000 7f021be00640 1 -- 172.21.15.47:0/33695299 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021c0d1000 con 0x7f021c15e280 2024-02-19T15:29:02.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.310+0000 7f021be00640 1 --2- 172.21.15.47:0/33695299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f021c15e280 0x7f021c06c710 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0214009d90 tx=0x7f021401c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f9f8c3c68e23323 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:02.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.310+0000 7f021aa00640 1 -- 172.21.15.47:0/33695299 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02140a5820 con 0x7f021c15e280 2024-02-19T15:29:02.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.310+0000 7f021aa00640 1 -- 172.21.15.47:0/33695299 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02140a5980 con 0x7f021c15e280 2024-02-19T15:29:02.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.310+0000 7f021aa00640 1 -- 172.21.15.47:0/33695299 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02140a5d10 con 0x7f021c15e280 2024-02-19T15:29:02.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.310+0000 7f0222400640 1 -- 172.21.15.47:0/33695299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f021c15e280 msgr2=0x7f021c06c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:02.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.310+0000 7f0222400640 1 --2- 172.21.15.47:0/33695299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f021c15e280 0x7f021c06c710 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0214009d90 tx=0x7f021401c850 comp rx=0 tx=0).stop 2024-02-19T15:29:02.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.310+0000 7f0222400640 1 -- 172.21.15.47:0/33695299 shutdown_connections 2024-02-19T15:29:02.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.310+0000 7f0222400640 1 --2- 172.21.15.47:0/33695299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f021c15e280 0x7f021c06c710 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:02.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.310+0000 7f0222400640 1 --2- 172.21.15.47:0/33695299 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f021c0d2650 0x7f021c06c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:02.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.310+0000 7f0222400640 1 -- 172.21.15.47:0/33695299 >> 172.21.15.47:0/33695299 conn(0x7f021c0d1b30 msgr2=0x7f021c0d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:02.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.310+0000 7f0222400640 1 -- 172.21.15.47:0/33695299 shutdown_connections 2024-02-19T15:29:02.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.310+0000 7f0222400640 1 -- 172.21.15.47:0/33695299 wait complete. 2024-02-19T15:29:02.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f0222400640 1 Processor -- start 2024-02-19T15:29:02.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f0222400640 1 -- start start 2024-02-19T15:29:02.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f0222400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f021c0ce3b0 0x7f021c150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:02.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f0222400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f021c0d2650 0x7f021c1550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:02.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f0222400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f021c15e280 0x7f021c160bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:02.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f0222400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c155620 con 0x7f021c15e280 2024-02-19T15:29:02.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f0222400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c15bf20 con 0x7f021c0ce3b0 2024-02-19T15:29:02.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f0222400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f021c15bbf0 con 0x7f021c0d2650 2024-02-19T15:29:02.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f021be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f021c15e280 0x7f021c160bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:02.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f021be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f021c15e280 0x7f021c160bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51222/0 (socket says 172.21.15.47:51222) 2024-02-19T15:29:02.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f021be00640 1 -- 172.21.15.47:0/3224978542 learned_addr learned my addr 172.21.15.47:0/3224978542 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:02.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f0213e00640 1 --2- 172.21.15.47:0/3224978542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f021c0d2650 0x7f021c1550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:02.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f021be00640 1 -- 172.21.15.47:0/3224978542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f021c0d2650 msgr2=0x7f021c1550e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:02.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f021be00640 1 --2- 172.21.15.47:0/3224978542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f021c0d2650 0x7f021c1550e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:02.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f021b400640 1 --2- 172.21.15.47:0/3224978542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f021c0ce3b0 0x7f021c150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:02.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f021be00640 1 -- 172.21.15.47:0/3224978542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f021c0ce3b0 msgr2=0x7f021c150140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:02.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f021be00640 1 --2- 172.21.15.47:0/3224978542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f021c0ce3b0 0x7f021c150140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:02.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f021be00640 1 -- 172.21.15.47:0/3224978542 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0214009a70 con 0x7f021c15e280 2024-02-19T15:29:02.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f0213e00640 1 --2- 172.21.15.47:0/3224978542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f021c0d2650 0x7f021c1550e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:29:02.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f021b400640 1 --2- 172.21.15.47:0/3224978542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f021c0ce3b0 0x7f021c150140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:29:02.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f021be00640 1 --2- 172.21.15.47:0/3224978542 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f021c15e280 0x7f021c160bb0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f02140b0040 tx=0x7f02140195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:02.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f0213400640 1 -- 172.21.15.47:0/3224978542 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02140ad640 con 0x7f021c15e280 2024-02-19T15:29:02.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f0213400640 1 -- 172.21.15.47:0/3224978542 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02140ad7a0 con 0x7f021c15e280 2024-02-19T15:29:02.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f0213400640 1 -- 172.21.15.47:0/3224978542 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02140adb50 con 0x7f021c15e280 2024-02-19T15:29:02.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f0222400640 1 -- 172.21.15.47:0/3224978542 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f021c155850 con 0x7f021c15e280 2024-02-19T15:29:02.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.314+0000 7f0222400640 1 -- 172.21.15.47:0/3224978542 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f021c155e10 con 0x7f021c15e280 2024-02-19T15:29:02.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.318+0000 7f0222400640 1 -- 172.21.15.47:0/3224978542 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f021c15c0b0 con 0x7f021c15e280 2024-02-19T15:29:02.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.318+0000 7f0213400640 1 -- 172.21.15.47:0/3224978542 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f02140b4050 con 0x7f021c15e280 2024-02-19T15:29:02.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.318+0000 7f0213400640 1 --2- 172.21.15.47:0/3224978542 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f01f8066e40 0x7f01f8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:02.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.318+0000 7f0213400640 1 -- 172.21.15.47:0/3224978542 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f021411fc30 con 0x7f021c15e280 2024-02-19T15:29:02.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.318+0000 7f021b400640 1 --2- 172.21.15.47:0/3224978542 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f01f8066e40 0x7f01f8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:02.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.318+0000 7f021b400640 1 --2- 172.21.15.47:0/3224978542 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f01f8066e40 0x7f01f8069300 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f020c005890 tx=0x7f020c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:02.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.322+0000 7f0213400640 1 -- 172.21.15.47:0/3224978542 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f02140ec8f0 con 0x7f021c15e280 2024-02-19T15:29:02.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:02.522+0000 7f0222400640 1 -- 172.21.15.47:0/3224978542 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f021c075360 con 0x7f021c15e280 2024-02-19T15:29:03.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.034+0000 7f0213400640 1 -- 172.21.15.47:0/3224978542 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v92) v1 ==== 105+0+0 (secure 0 0 0) 0x7f02140b20b0 con 0x7f021c15e280 2024-02-19T15:29:03.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.038+0000 7f0222400640 1 -- 172.21.15.47:0/3224978542 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f01f8066e40 msgr2=0x7f01f8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:03.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.038+0000 7f0222400640 1 --2- 172.21.15.47:0/3224978542 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f01f8066e40 0x7f01f8069300 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f020c005890 tx=0x7f020c009040 comp rx=0 tx=0).stop 2024-02-19T15:29:03.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.038+0000 7f0222400640 1 -- 172.21.15.47:0/3224978542 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f021c15e280 msgr2=0x7f021c160bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:03.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.038+0000 7f0222400640 1 --2- 172.21.15.47:0/3224978542 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f021c15e280 0x7f021c160bb0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f02140b0040 tx=0x7f02140195f0 comp rx=0 tx=0).stop 2024-02-19T15:29:03.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.038+0000 7f0222400640 1 -- 172.21.15.47:0/3224978542 shutdown_connections 2024-02-19T15:29:03.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.038+0000 7f0222400640 1 --2- 172.21.15.47:0/3224978542 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f01f8066e40 0x7f01f8069300 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:03.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.038+0000 7f0222400640 1 --2- 172.21.15.47:0/3224978542 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f021c15e280 0x7f021c160bb0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:03.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.038+0000 7f0222400640 1 --2- 172.21.15.47:0/3224978542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f021c0d2650 0x7f021c1550e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:03.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.038+0000 7f0222400640 1 --2- 172.21.15.47:0/3224978542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f021c0ce3b0 0x7f021c150140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:03.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.038+0000 7f0222400640 1 -- 172.21.15.47:0/3224978542 >> 172.21.15.47:0/3224978542 conn(0x7f021c0d1b30 msgr2=0x7f021c0c5900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:03.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.038+0000 7f0222400640 1 -- 172.21.15.47:0/3224978542 shutdown_connections 2024-02-19T15:29:03.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.038+0000 7f0222400640 1 -- 172.21.15.47:0/3224978542 wait complete. 2024-02-19T15:29:03.061 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:03.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.202+0000 7fcc01a00640 1 Processor -- start 2024-02-19T15:29:03.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.202+0000 7fcc01a00640 1 -- start start 2024-02-19T15:29:03.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.202+0000 7fcc01a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcbfc0b57b0 0x7fcbfc0ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:03.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.202+0000 7fcc01a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbfc161aa0 con 0x7fcbfc1581f0 2024-02-19T15:29:03.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.202+0000 7fcc01a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbfc15f310 con 0x7fcbfc1585d0 2024-02-19T15:29:03.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.202+0000 7fcc01a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbfc162400 con 0x7fcbfc0b57b0 2024-02-19T15:29:03.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.202+0000 7fcbf3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcbfc0b57b0 0x7fcbfc0ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:03.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.202+0000 7fcbf3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcbfc0b57b0 0x7fcbfc0ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51236/0 (socket says 172.21.15.47:51236) 2024-02-19T15:29:03.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.202+0000 7fcbf3e00640 1 -- 172.21.15.47:0/1018043688 learned_addr learned my addr 172.21.15.47:0/1018043688 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:03.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.202+0000 7fcbf3e00640 1 -- 172.21.15.47:0/1018043688 >> v1:172.21.15.116:6789/0 conn(0x7fcbfc1585d0 legacy=0x7fcbfc0addf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:03.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.202+0000 7fcbf3e00640 1 -- 172.21.15.47:0/1018043688 >> v1:172.21.15.47:6789/0 conn(0x7fcbfc1581f0 legacy=0x7fcbfc0b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:03.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.202+0000 7fcbf3e00640 1 -- 172.21.15.47:0/1018043688 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbfc162820 con 0x7fcbfc0b57b0 2024-02-19T15:29:03.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.206+0000 7fcbf3e00640 1 --2- 172.21.15.47:0/1018043688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbfc0b57b0 0x7fcbfc0ad8b0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4009870 tx=0x7fcbe401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83de5bdd674af7b0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:03.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.206+0000 7fcbfb400640 1 -- 172.21.15.47:0/1018043688 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe40ac620 con 0x7fcbfc0b57b0 2024-02-19T15:29:03.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.206+0000 7fcbfb400640 1 -- 172.21.15.47:0/1018043688 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe40ac780 con 0x7fcbfc0b57b0 2024-02-19T15:29:03.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.206+0000 7fcbfb400640 1 -- 172.21.15.47:0/1018043688 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe40acb10 con 0x7fcbfc0b57b0 2024-02-19T15:29:03.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.206+0000 7fcc01a00640 1 -- 172.21.15.47:0/1018043688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbfc0b57b0 msgr2=0x7fcbfc0ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:03.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.206+0000 7fcc01a00640 1 --2- 172.21.15.47:0/1018043688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbfc0b57b0 0x7fcbfc0ad8b0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4009870 tx=0x7fcbe401c4d0 comp rx=0 tx=0).stop 2024-02-19T15:29:03.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.206+0000 7fcc01a00640 1 -- 172.21.15.47:0/1018043688 shutdown_connections 2024-02-19T15:29:03.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.206+0000 7fcc01a00640 1 --2- 172.21.15.47:0/1018043688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbfc0b57b0 0x7fcbfc0ad8b0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:03.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.206+0000 7fcc01a00640 1 -- 172.21.15.47:0/1018043688 >> 172.21.15.47:0/1018043688 conn(0x7fcbfc0b24f0 msgr2=0x7fcbfc0b2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:03.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.206+0000 7fcc01a00640 1 -- 172.21.15.47:0/1018043688 shutdown_connections 2024-02-19T15:29:03.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.206+0000 7fcc01a00640 1 -- 172.21.15.47:0/1018043688 wait complete. 2024-02-19T15:29:03.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcc01a00640 1 Processor -- start 2024-02-19T15:29:03.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcc01a00640 1 -- start start 2024-02-19T15:29:03.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcc01a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbfc0b57b0 0x7fcbfc14e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:03.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcc01a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbfc1581f0 0x7fcbfc152760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:03.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcc01a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcbfc1585d0 0x7fcbfc14ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:03.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcc01a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbfc14eaa0 con 0x7fcbfc0b57b0 2024-02-19T15:29:03.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcc01a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbfc14ed80 con 0x7fcbfc1581f0 2024-02-19T15:29:03.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcc01a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbfc14f060 con 0x7fcbfc1585d0 2024-02-19T15:29:03.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcbfbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbfc0b57b0 0x7fcbfc14e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:03.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcbfbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbfc0b57b0 0x7fcbfc14e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51252/0 (socket says 172.21.15.47:51252) 2024-02-19T15:29:03.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcbfbe00640 1 -- 172.21.15.47:0/170535935 learned_addr learned my addr 172.21.15.47:0/170535935 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:03.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcbf3e00640 1 --2- 172.21.15.47:0/170535935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbfc1581f0 0x7fcbfc152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:03.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcc01000640 1 --2- 172.21.15.47:0/170535935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcbfc1585d0 0x7fcbfc14ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:03.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcbfbe00640 1 -- 172.21.15.47:0/170535935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcbfc1585d0 msgr2=0x7fcbfc14ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:03.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcbfbe00640 1 --2- 172.21.15.47:0/170535935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcbfc1585d0 0x7fcbfc14ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:03.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcbfbe00640 1 -- 172.21.15.47:0/170535935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbfc1581f0 msgr2=0x7fcbfc152760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:03.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcbfbe00640 1 --2- 172.21.15.47:0/170535935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbfc1581f0 0x7fcbfc152760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:03.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcbfbe00640 1 -- 172.21.15.47:0/170535935 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe40094d0 con 0x7fcbfc0b57b0 2024-02-19T15:29:03.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcbf3e00640 1 --2- 172.21.15.47:0/170535935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbfc1581f0 0x7fcbfc152760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:29:03.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcbfbe00640 1 --2- 172.21.15.47:0/170535935 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbfc0b57b0 0x7fcbfc14e4c0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fcbec00c3c0 tx=0x7fcbec00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:03.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcbf8c00640 1 -- 172.21.15.47:0/170535935 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec0052d0 con 0x7fcbfc0b57b0 2024-02-19T15:29:03.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcbf8c00640 1 -- 172.21.15.47:0/170535935 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbec005430 con 0x7fcbfc0b57b0 2024-02-19T15:29:03.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcbf8c00640 1 -- 172.21.15.47:0/170535935 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec003840 con 0x7fcbfc0b57b0 2024-02-19T15:29:03.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcc01a00640 1 -- 172.21.15.47:0/170535935 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbfc14f340 con 0x7fcbfc0b57b0 2024-02-19T15:29:03.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.210+0000 7fcc01a00640 1 -- 172.21.15.47:0/170535935 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbfc152ca0 con 0x7fcbfc0b57b0 2024-02-19T15:29:03.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.214+0000 7fcc01a00640 1 -- 172.21.15.47:0/170535935 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbfc075360 con 0x7fcbfc0b57b0 2024-02-19T15:29:03.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.214+0000 7fcbf8c00640 1 -- 172.21.15.47:0/170535935 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fcbec0039a0 con 0x7fcbfc0b57b0 2024-02-19T15:29:03.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.214+0000 7fcbf8c00640 1 --2- 172.21.15.47:0/170535935 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcbd8066e40 0x7fcbd8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:03.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.214+0000 7fcbf8c00640 1 -- 172.21.15.47:0/170535935 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcbec083fd0 con 0x7fcbfc0b57b0 2024-02-19T15:29:03.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.214+0000 7fcbf3e00640 1 --2- 172.21.15.47:0/170535935 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcbd8066e40 0x7fcbd8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:03.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.214+0000 7fcbf3e00640 1 --2- 172.21.15.47:0/170535935 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcbd8066e40 0x7fcbd8069300 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fcbe40ae000 tx=0x7fcbe4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:03.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.222+0000 7fcbf8c00640 1 -- 172.21.15.47:0/170535935 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fcbec050dc0 con 0x7fcbfc0b57b0 2024-02-19T15:29:03.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.418+0000 7fcc01a00640 1 -- 172.21.15.47:0/170535935 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbfc0b41d0 con 0x7fcbfc0b57b0 2024-02-19T15:29:03.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.418+0000 7fcbf8c00640 1 -- 172.21.15.47:0/170535935 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+4959 (secure 0 0 0) 0x7fcbec054480 con 0x7fcbfc0b57b0 2024-02-19T15:29:03.420 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:03.420 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":93,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7028,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/809721153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":809721153},{"type":"v1","addr":"172.21.15.47:6834","nonce":809721153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7034,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1553163168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1553163168},{"type":"v1","addr":"172.21.15.116:6836","nonce":1553163168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:03.040328+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:creating","state_seq":1,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:03.421 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 93 2024-02-19T15:29:03.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.422+0000 7fcc01a00640 1 -- 172.21.15.47:0/170535935 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcbd8066e40 msgr2=0x7fcbd8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:03.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.422+0000 7fcc01a00640 1 --2- 172.21.15.47:0/170535935 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcbd8066e40 0x7fcbd8069300 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fcbe40ae000 tx=0x7fcbe4008040 comp rx=0 tx=0).stop 2024-02-19T15:29:03.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.422+0000 7fcc01a00640 1 -- 172.21.15.47:0/170535935 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbfc0b57b0 msgr2=0x7fcbfc14e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:03.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.422+0000 7fcc01a00640 1 --2- 172.21.15.47:0/170535935 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbfc0b57b0 0x7fcbfc14e4c0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fcbec00c3c0 tx=0x7fcbec00c880 comp rx=0 tx=0).stop 2024-02-19T15:29:03.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.422+0000 7fcc01a00640 1 -- 172.21.15.47:0/170535935 shutdown_connections 2024-02-19T15:29:03.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.422+0000 7fcc01a00640 1 --2- 172.21.15.47:0/170535935 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcbd8066e40 0x7fcbd8069300 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:03.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.422+0000 7fcc01a00640 1 --2- 172.21.15.47:0/170535935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcbfc1585d0 0x7fcbfc14ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:03.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.422+0000 7fcc01a00640 1 --2- 172.21.15.47:0/170535935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbfc1581f0 0x7fcbfc152760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:03.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.422+0000 7fcc01a00640 1 --2- 172.21.15.47:0/170535935 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcbfc0b57b0 0x7fcbfc14e4c0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:03.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.422+0000 7fcc01a00640 1 -- 172.21.15.47:0/170535935 >> 172.21.15.47:0/170535935 conn(0x7fcbfc0b24f0 msgr2=0x7fcbfc15df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:03.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.422+0000 7fcc01a00640 1 -- 172.21.15.47:0/170535935 shutdown_connections 2024-02-19T15:29:03.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:03.422+0000 7fcc01a00640 1 -- 172.21.15.47:0/170535935 wait complete. 2024-02-19T15:29:03.441 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-02-19T15:29:05.946 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:29:05.950 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:29:08.442 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:08.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.582+0000 7f79c6200640 1 Processor -- start 2024-02-19T15:29:08.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.582+0000 7f79c6200640 1 -- start start 2024-02-19T15:29:08.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.582+0000 7f79c6200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79c01476e0 0x7f79c00ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:08.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.582+0000 7f79c6200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f79c00b33c0 0x7f79c0071950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:08.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.582+0000 7f79c6200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c00af1b0 con 0x7f79c006c9a0 2024-02-19T15:29:08.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.582+0000 7f79c6200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79c0071530 con 0x7f79c00b33c0 2024-02-19T15:29:08.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.582+0000 7f79c6200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c0070980 con 0x7f79c01476e0 2024-02-19T15:29:08.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.586+0000 7f79bfe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f79c00b33c0 0x7f79c0071950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:08.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.586+0000 7f79bf400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79c01476e0 0x7f79c00ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:08.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.586+0000 7f79b6a00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f79c006c9a0 0x7f79c00b2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:52652/0 (socket says 172.21.15.47:52652) 2024-02-19T15:29:08.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.586+0000 7f79b6a00640 1 -- 172.21.15.47:0/82080875 learned_addr learned my addr 172.21.15.47:0/82080875 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:29:08.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.586+0000 7f79bea00640 1 -- 172.21.15.47:0/82080875 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335637947 0 0) 0x7f79c00af1b0 con 0x7f79c006c9a0 2024-02-19T15:29:08.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.586+0000 7f79bf400640 1 -- 172.21.15.47:0/82080875 >> v2:172.21.15.116:3300/0 conn(0x7f79c00b33c0 msgr2=0x7f79c0071950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:08.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.586+0000 7f79bf400640 1 --2- 172.21.15.47:0/82080875 >> v2:172.21.15.116:3300/0 conn(0x7f79c00b33c0 0x7f79c0071950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:08.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.586+0000 7f79bf400640 1 -- 172.21.15.47:0/82080875 >> v1:172.21.15.116:6789/0 conn(0x7f79c006c9a0 legacy=0x7f79c00b2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:08.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.586+0000 7f79bf400640 1 -- 172.21.15.47:0/82080875 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c0070b90 con 0x7f79c01476e0 2024-02-19T15:29:08.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.586+0000 7f79bf400640 1 --2- 172.21.15.47:0/82080875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79c01476e0 0x7f79c00ae730 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f79b0009d90 tx=0x7f79b001c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc0698dbba2391a7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:08.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.586+0000 7f79bea00640 1 -- 172.21.15.47:0/82080875 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b00a5a00 con 0x7f79c01476e0 2024-02-19T15:29:08.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.586+0000 7f79bea00640 1 -- 172.21.15.47:0/82080875 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b00a5b60 con 0x7f79c01476e0 2024-02-19T15:29:08.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.586+0000 7f79bea00640 1 -- 172.21.15.47:0/82080875 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b00ad440 con 0x7f79c01476e0 2024-02-19T15:29:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79c6200640 1 -- 172.21.15.47:0/82080875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79c01476e0 msgr2=0x7f79c00ae730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79c6200640 1 --2- 172.21.15.47:0/82080875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79c01476e0 0x7f79c00ae730 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f79b0009d90 tx=0x7f79b001c940 comp rx=0 tx=0).stop 2024-02-19T15:29:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79c6200640 1 -- 172.21.15.47:0/82080875 shutdown_connections 2024-02-19T15:29:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79c6200640 1 --2- 172.21.15.47:0/82080875 >> v2:172.21.15.116:3300/0 conn(0x7f79c00b33c0 0x7f79c0071950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79c6200640 1 --2- 172.21.15.47:0/82080875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79c01476e0 0x7f79c00ae730 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79c6200640 1 -- 172.21.15.47:0/82080875 >> 172.21.15.47:0/82080875 conn(0x7f79c00b16a0 msgr2=0x7f79c00b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79c6200640 1 -- 172.21.15.47:0/82080875 shutdown_connections 2024-02-19T15:29:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79c6200640 1 -- 172.21.15.47:0/82080875 wait complete. 2024-02-19T15:29:08.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79c6200640 1 Processor -- start 2024-02-19T15:29:08.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79c6200640 1 -- start start 2024-02-19T15:29:08.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79c6200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79c006c9a0 0x7f79c013f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:08.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79c6200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79c00b33c0 0x7f79c0160c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:08.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79c6200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c01476e0 0x7f79c015d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:08.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79c6200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c015e6f0 con 0x7f79c00b33c0 2024-02-19T15:29:08.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79c6200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c015e9d0 con 0x7f79c01476e0 2024-02-19T15:29:08.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79c6200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c015ecb0 con 0x7f79c006c9a0 2024-02-19T15:29:08.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79b6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79c00b33c0 0x7f79c0160c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:08.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79b6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79c00b33c0 0x7f79c0160c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49178/0 (socket says 172.21.15.47:49178) 2024-02-19T15:29:08.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79b6a00640 1 -- 172.21.15.47:0/525746802 learned_addr learned my addr 172.21.15.47:0/525746802 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:08.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79bf400640 1 --2- 172.21.15.47:0/525746802 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79c006c9a0 0x7f79c013f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:08.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79bfe00640 1 --2- 172.21.15.47:0/525746802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c01476e0 0x7f79c015d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:08.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79b6a00640 1 -- 172.21.15.47:0/525746802 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79c006c9a0 msgr2=0x7f79c013f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:08.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79b6a00640 1 --2- 172.21.15.47:0/525746802 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79c006c9a0 0x7f79c013f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:08.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79b6a00640 1 -- 172.21.15.47:0/525746802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c01476e0 msgr2=0x7f79c015d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:08.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79b6a00640 1 --2- 172.21.15.47:0/525746802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c01476e0 0x7f79c015d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:08.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.590+0000 7f79b6a00640 1 -- 172.21.15.47:0/525746802 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b0009a70 con 0x7f79c00b33c0 2024-02-19T15:29:08.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.594+0000 7f79b6a00640 1 --2- 172.21.15.47:0/525746802 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79c00b33c0 0x7f79c0160c90 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f79a800c450 tx=0x7f79a800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:08.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.594+0000 7f79b7e00640 1 -- 172.21.15.47:0/525746802 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a80074e0 con 0x7f79c00b33c0 2024-02-19T15:29:08.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.594+0000 7f79b7e00640 1 -- 172.21.15.47:0/525746802 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a8007640 con 0x7f79c00b33c0 2024-02-19T15:29:08.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.594+0000 7f79c6200640 1 -- 172.21.15.47:0/525746802 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79c015ef90 con 0x7f79c00b33c0 2024-02-19T15:29:08.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.594+0000 7f79b7e00640 1 -- 172.21.15.47:0/525746802 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a80049f0 con 0x7f79c00b33c0 2024-02-19T15:29:08.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.594+0000 7f79c6200640 1 -- 172.21.15.47:0/525746802 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79c015f790 con 0x7f79c00b33c0 2024-02-19T15:29:08.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.594+0000 7f79c6200640 1 -- 172.21.15.47:0/525746802 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79c015e440 con 0x7f79c00b33c0 2024-02-19T15:29:08.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.594+0000 7f79b7e00640 1 -- 172.21.15.47:0/525746802 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f79a8004b50 con 0x7f79c00b33c0 2024-02-19T15:29:08.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.594+0000 7f79b7e00640 1 --2- 172.21.15.47:0/525746802 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f799c066e40 0x7f799c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:08.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.594+0000 7f79b7e00640 1 -- 172.21.15.47:0/525746802 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f79a8083f20 con 0x7f79c00b33c0 2024-02-19T15:29:08.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.594+0000 7f79bf400640 1 --2- 172.21.15.47:0/525746802 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f799c066e40 0x7f799c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:08.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.598+0000 7f79bf400640 1 --2- 172.21.15.47:0/525746802 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f799c066e40 0x7f799c069300 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f79b00af000 tx=0x7f79b00a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:08.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.606+0000 7f79b7e00640 1 -- 172.21.15.47:0/525746802 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f79a8050dc0 con 0x7f79c00b33c0 2024-02-19T15:29:08.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.802+0000 7f79c6200640 1 -- 172.21.15.47:0/525746802 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79c0075360 con 0x7f79c00b33c0 2024-02-19T15:29:08.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.806+0000 7f79b7e00640 1 -- 172.21.15.47:0/525746802 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4957 (secure 0 0 0) 0x7f79a8018020 con 0x7f79c00b33c0 2024-02-19T15:29:08.805 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:08.805 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":94,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7028,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/809721153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":809721153},{"type":"v1","addr":"172.21.15.47:6834","nonce":809721153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7034,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1553163168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1553163168},{"type":"v1","addr":"172.21.15.116:6836","nonce":1553163168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:03.058284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:08.805 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 94 2024-02-19T15:29:08.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.806+0000 7f79c6200640 1 -- 172.21.15.47:0/525746802 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f799c066e40 msgr2=0x7f799c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:08.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.806+0000 7f79c6200640 1 --2- 172.21.15.47:0/525746802 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f799c066e40 0x7f799c069300 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f79b00af000 tx=0x7f79b00a7070 comp rx=0 tx=0).stop 2024-02-19T15:29:08.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.806+0000 7f79c6200640 1 -- 172.21.15.47:0/525746802 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79c00b33c0 msgr2=0x7f79c0160c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:08.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.806+0000 7f79c6200640 1 --2- 172.21.15.47:0/525746802 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79c00b33c0 0x7f79c0160c90 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f79a800c450 tx=0x7f79a800c910 comp rx=0 tx=0).stop 2024-02-19T15:29:08.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.806+0000 7f79c6200640 1 -- 172.21.15.47:0/525746802 shutdown_connections 2024-02-19T15:29:08.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.806+0000 7f79c6200640 1 --2- 172.21.15.47:0/525746802 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f799c066e40 0x7f799c069300 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:08.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.806+0000 7f79c6200640 1 --2- 172.21.15.47:0/525746802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c01476e0 0x7f79c015d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:08.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.806+0000 7f79c6200640 1 --2- 172.21.15.47:0/525746802 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79c00b33c0 0x7f79c0160c90 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:08.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.806+0000 7f79c6200640 1 --2- 172.21.15.47:0/525746802 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79c006c9a0 0x7f79c013f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:08.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.806+0000 7f79c6200640 1 -- 172.21.15.47:0/525746802 >> 172.21.15.47:0/525746802 conn(0x7f79c00b16a0 msgr2=0x7f79c00b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:08.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.806+0000 7f79c6200640 1 -- 172.21.15.47:0/525746802 shutdown_connections 2024-02-19T15:29:08.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.806+0000 7f79c6200640 1 -- 172.21.15.47:0/525746802 wait complete. 2024-02-19T15:29:08.826 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:08.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.966+0000 7fe039200640 1 Processor -- start 2024-02-19T15:29:08.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.966+0000 7fe039200640 1 -- start start 2024-02-19T15:29:08.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.966+0000 7fe039200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe0340cded0 0x7fe0340ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:08.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.966+0000 7fe039200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe0340ceb40 0x7fe03406bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:08.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.966+0000 7fe039200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0340cce30 con 0x7fe03406d650 2024-02-19T15:29:08.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.966+0000 7fe039200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0340cefd0 con 0x7fe0340ceb40 2024-02-19T15:29:08.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.966+0000 7fe039200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0340cd110 con 0x7fe0340cded0 2024-02-19T15:29:08.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.966+0000 7fe02be00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe0340cded0 0x7fe0340ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:08.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.966+0000 7fe032800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe0340ceb40 0x7fe03406bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:08.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.966+0000 7fe031e00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe03406d650 0x7fe0340d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:52656/0 (socket says 172.21.15.47:52656) 2024-02-19T15:29:08.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.966+0000 7fe031e00640 1 -- 172.21.15.47:0/4035104007 learned_addr learned my addr 172.21.15.47:0/4035104007 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:29:08.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.966+0000 7fe031400640 1 -- 172.21.15.47:0/4035104007 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284279886 0 0) 0x7fe0340cce30 con 0x7fe03406d650 2024-02-19T15:29:08.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.966+0000 7fe02be00640 1 -- 172.21.15.47:0/4035104007 >> v2:172.21.15.47:3300/0 conn(0x7fe0340ceb40 msgr2=0x7fe03406bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:08.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.966+0000 7fe02be00640 1 --2- 172.21.15.47:0/4035104007 >> v2:172.21.15.47:3300/0 conn(0x7fe0340ceb40 0x7fe03406bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:08.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.966+0000 7fe02be00640 1 -- 172.21.15.47:0/4035104007 >> v1:172.21.15.116:6789/0 conn(0x7fe03406d650 legacy=0x7fe0340d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:08.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.966+0000 7fe02be00640 1 -- 172.21.15.47:0/4035104007 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0340cd3f0 con 0x7fe0340cded0 2024-02-19T15:29:08.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.970+0000 7fe02be00640 1 --2- 172.21.15.47:0/4035104007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0340cded0 0x7fe0340ce600 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe03406b950 tx=0x7fe01c01c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e3ab4078719ec0a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:08.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.970+0000 7fe031400640 1 -- 172.21.15.47:0/4035104007 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c0ac620 con 0x7fe0340cded0 2024-02-19T15:29:08.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.970+0000 7fe031400640 1 -- 172.21.15.47:0/4035104007 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01c0ac780 con 0x7fe0340cded0 2024-02-19T15:29:08.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.970+0000 7fe031400640 1 -- 172.21.15.47:0/4035104007 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c0acb10 con 0x7fe0340cded0 2024-02-19T15:29:08.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.970+0000 7fe039200640 1 -- 172.21.15.47:0/4035104007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0340cded0 msgr2=0x7fe0340ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:08.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.970+0000 7fe039200640 1 --2- 172.21.15.47:0/4035104007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0340cded0 0x7fe0340ce600 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe03406b950 tx=0x7fe01c01c3c0 comp rx=0 tx=0).stop 2024-02-19T15:29:08.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.970+0000 7fe039200640 1 -- 172.21.15.47:0/4035104007 shutdown_connections 2024-02-19T15:29:08.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.970+0000 7fe039200640 1 --2- 172.21.15.47:0/4035104007 >> v2:172.21.15.47:3300/0 conn(0x7fe0340ceb40 0x7fe03406bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:08.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.970+0000 7fe039200640 1 --2- 172.21.15.47:0/4035104007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0340cded0 0x7fe0340ce600 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:08.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.970+0000 7fe039200640 1 -- 172.21.15.47:0/4035104007 >> 172.21.15.47:0/4035104007 conn(0x7fe0340d1bf0 msgr2=0x7fe0340d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:08.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.970+0000 7fe039200640 1 -- 172.21.15.47:0/4035104007 shutdown_connections 2024-02-19T15:29:08.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.970+0000 7fe039200640 1 -- 172.21.15.47:0/4035104007 wait complete. 2024-02-19T15:29:08.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe039200640 1 Processor -- start 2024-02-19T15:29:08.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe039200640 1 -- start start 2024-02-19T15:29:08.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe039200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe03406d650 0x7fe0340ae860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:08.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe039200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0340cded0 0x7fe0340c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:08.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe039200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0340ceb40 0x7fe0341fa480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:08.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe039200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0340af160 con 0x7fe0340ceb40 2024-02-19T15:29:08.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe039200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03415d560 con 0x7fe0340cded0 2024-02-19T15:29:08.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe039200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe03415d7b0 con 0x7fe03406d650 2024-02-19T15:29:08.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe032800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0340ceb40 0x7fe0341fa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:08.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe032800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0340ceb40 0x7fe0341fa480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49186/0 (socket says 172.21.15.47:49186) 2024-02-19T15:29:08.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe032800640 1 -- 172.21.15.47:0/499162707 learned_addr learned my addr 172.21.15.47:0/499162707 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:08.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe02be00640 1 --2- 172.21.15.47:0/499162707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0340cded0 0x7fe0340c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:08.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe031e00640 1 --2- 172.21.15.47:0/499162707 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe03406d650 0x7fe0340ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:08.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe032800640 1 -- 172.21.15.47:0/499162707 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe03406d650 msgr2=0x7fe0340ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:08.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe032800640 1 --2- 172.21.15.47:0/499162707 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe03406d650 0x7fe0340ae860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:08.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe032800640 1 -- 172.21.15.47:0/499162707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0340cded0 msgr2=0x7fe0340c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:08.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe032800640 1 --2- 172.21.15.47:0/499162707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0340cded0 0x7fe0340c7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:08.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe032800640 1 -- 172.21.15.47:0/499162707 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01c009420 con 0x7fe0340ceb40 2024-02-19T15:29:08.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe032800640 1 --2- 172.21.15.47:0/499162707 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0340ceb40 0x7fe0341fa480 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fe02c00efc0 tx=0x7fe02c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:08.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe02a000640 1 -- 172.21.15.47:0/499162707 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c014480 con 0x7fe0340ceb40 2024-02-19T15:29:08.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe02a000640 1 -- 172.21.15.47:0/499162707 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02c00f040 con 0x7fe0340ceb40 2024-02-19T15:29:08.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe039200640 1 -- 172.21.15.47:0/499162707 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe03415da90 con 0x7fe0340ceb40 2024-02-19T15:29:08.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe02a000640 1 -- 172.21.15.47:0/499162707 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c014820 con 0x7fe0340ceb40 2024-02-19T15:29:08.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.974+0000 7fe039200640 1 -- 172.21.15.47:0/499162707 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0340aeda0 con 0x7fe0340ceb40 2024-02-19T15:29:08.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.978+0000 7fe039200640 1 -- 172.21.15.47:0/499162707 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe034075360 con 0x7fe0340ceb40 2024-02-19T15:29:08.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.978+0000 7fe02a000640 1 -- 172.21.15.47:0/499162707 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe02c014980 con 0x7fe0340ceb40 2024-02-19T15:29:08.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.978+0000 7fe02a000640 1 --2- 172.21.15.47:0/499162707 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe010066e40 0x7fe010069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:08.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.978+0000 7fe02a000640 1 -- 172.21.15.47:0/499162707 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe02c085740 con 0x7fe0340ceb40 2024-02-19T15:29:08.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.978+0000 7fe031e00640 1 --2- 172.21.15.47:0/499162707 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe010066e40 0x7fe010069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:08.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.978+0000 7fe031e00640 1 --2- 172.21.15.47:0/499162707 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe010066e40 0x7fe010069300 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fe024000f70 tx=0x7fe024007df0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:08.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:08.982+0000 7fe02a000640 1 -- 172.21.15.47:0/499162707 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe02c052670 con 0x7fe0340ceb40 2024-02-19T15:29:09.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.238+0000 7fe039200640 1 -- 172.21.15.47:0/499162707 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0340cf580 con 0x7fe0340ceb40 2024-02-19T15:29:09.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.238+0000 7fe02a000640 1 -- 172.21.15.47:0/499162707 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4957 (secure 0 0 0) 0x7fe02c055d30 con 0x7fe0340ceb40 2024-02-19T15:29:09.237 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:09.237 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":94,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7028,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6834/809721153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":809721153},{"type":"v1","addr":"172.21.15.47:6834","nonce":809721153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7034,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/1553163168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1553163168},{"type":"v1","addr":"172.21.15.116:6836","nonce":1553163168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:03.058284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:09.238 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 94 2024-02-19T15:29:09.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.238+0000 7fe039200640 1 -- 172.21.15.47:0/499162707 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe010066e40 msgr2=0x7fe010069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:09.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.238+0000 7fe039200640 1 --2- 172.21.15.47:0/499162707 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe010066e40 0x7fe010069300 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fe024000f70 tx=0x7fe024007df0 comp rx=0 tx=0).stop 2024-02-19T15:29:09.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.238+0000 7fe039200640 1 -- 172.21.15.47:0/499162707 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0340ceb40 msgr2=0x7fe0341fa480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:09.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.238+0000 7fe039200640 1 --2- 172.21.15.47:0/499162707 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0340ceb40 0x7fe0341fa480 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fe02c00efc0 tx=0x7fe02c00c490 comp rx=0 tx=0).stop 2024-02-19T15:29:09.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.238+0000 7fe039200640 1 -- 172.21.15.47:0/499162707 shutdown_connections 2024-02-19T15:29:09.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.238+0000 7fe039200640 1 --2- 172.21.15.47:0/499162707 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe010066e40 0x7fe010069300 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:09.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.238+0000 7fe039200640 1 --2- 172.21.15.47:0/499162707 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0340ceb40 0x7fe0341fa480 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:09.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.238+0000 7fe039200640 1 --2- 172.21.15.47:0/499162707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0340cded0 0x7fe0340c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:09.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.238+0000 7fe039200640 1 --2- 172.21.15.47:0/499162707 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe03406d650 0x7fe0340ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:09.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.238+0000 7fe039200640 1 -- 172.21.15.47:0/499162707 >> 172.21.15.47:0/499162707 conn(0x7fe0340d1bf0 msgr2=0x7fe034160dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:09.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.238+0000 7fe039200640 1 -- 172.21.15.47:0/499162707 shutdown_connections 2024-02-19T15:29:09.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.238+0000 7fe039200640 1 -- 172.21.15.47:0/499162707 wait complete. 2024-02-19T15:29:09.259 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are a and b 2024-02-19T15:29:09.259 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T15:29:09.259 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:29:09.274 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T15:29:09.274 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-02-19T15:29:09.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.454+0000 7fed3cc00640 1 Processor -- start 2024-02-19T15:29:09.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.458+0000 7fed3cc00640 1 -- start start 2024-02-19T15:29:09.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.458+0000 7fed3cc00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fed381613a0 0x7fed380adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:09.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.458+0000 7fed3cc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed3806fdf0 0x7fed3806e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:09.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.458+0000 7fed3cc00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fed3806e5f0 0x7fed3806ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:09.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.458+0000 7fed3cc00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed38161780 con 0x7fed381613a0 2024-02-19T15:29:09.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.458+0000 7fed3cc00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed380b01a0 con 0x7fed3806e5f0 2024-02-19T15:29:09.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.458+0000 7fed3cc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed380ae3c0 con 0x7fed3806fdf0 2024-02-19T15:29:09.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.458+0000 7fed35e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fed381613a0 0x7fed380adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:09.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.458+0000 7fed35e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fed381613a0 0x7fed380adbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49200/0 (socket says 172.21.15.47:49200) 2024-02-19T15:29:09.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.458+0000 7fed35e00640 1 -- 172.21.15.47:0/2321011036 learned_addr learned my addr 172.21.15.47:0/2321011036 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:09.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.458+0000 7fed36800640 1 --2- 172.21.15.47:0/2321011036 >> v2:172.21.15.116:3300/0 conn(0x7fed3806e5f0 0x7fed3806ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:09.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.458+0000 7fed2fe00640 1 --2- 172.21.15.47:0/2321011036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed3806fdf0 0x7fed3806e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:09.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.458+0000 7fed35e00640 1 -- 172.21.15.47:0/2321011036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed3806fdf0 msgr2=0x7fed3806e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:09.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.458+0000 7fed35e00640 1 --2- 172.21.15.47:0/2321011036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed3806fdf0 0x7fed3806e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:09.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.458+0000 7fed35e00640 1 -- 172.21.15.47:0/2321011036 >> v2:172.21.15.116:3300/0 conn(0x7fed3806e5f0 msgr2=0x7fed3806ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:09.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.458+0000 7fed35e00640 1 --2- 172.21.15.47:0/2321011036 >> v2:172.21.15.116:3300/0 conn(0x7fed3806e5f0 0x7fed3806ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:09.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.458+0000 7fed35e00640 1 -- 172.21.15.47:0/2321011036 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed380ae6a0 con 0x7fed381613a0 2024-02-19T15:29:09.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.462+0000 7fed35e00640 1 --2- 172.21.15.47:0/2321011036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed381613a0 0x7fed380adbb0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fed380677b0 tx=0x7fed2801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6424bdcbbfbfae4a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:09.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.462+0000 7fed35400640 1 -- 172.21.15.47:0/2321011036 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed280ad660 con 0x7fed381613a0 2024-02-19T15:29:09.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.462+0000 7fed35400640 1 -- 172.21.15.47:0/2321011036 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed280ad7c0 con 0x7fed381613a0 2024-02-19T15:29:09.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.462+0000 7fed35400640 1 -- 172.21.15.47:0/2321011036 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed280adb50 con 0x7fed381613a0 2024-02-19T15:29:09.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.462+0000 7fed3cc00640 1 -- 172.21.15.47:0/2321011036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed381613a0 msgr2=0x7fed380adbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:09.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.462+0000 7fed3cc00640 1 --2- 172.21.15.47:0/2321011036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed381613a0 0x7fed380adbb0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fed380677b0 tx=0x7fed2801c490 comp rx=0 tx=0).stop 2024-02-19T15:29:09.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed3cc00640 1 -- 172.21.15.47:0/2321011036 shutdown_connections 2024-02-19T15:29:09.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed3cc00640 1 --2- 172.21.15.47:0/2321011036 >> v2:172.21.15.116:3300/0 conn(0x7fed3806e5f0 0x7fed3806ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:09.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed3cc00640 1 --2- 172.21.15.47:0/2321011036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed3806fdf0 0x7fed3806e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:09.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed3cc00640 1 --2- 172.21.15.47:0/2321011036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed381613a0 0x7fed380adbb0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:09.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed3cc00640 1 -- 172.21.15.47:0/2321011036 >> 172.21.15.47:0/2321011036 conn(0x7fed380d0ad0 msgr2=0x7fed380d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:09.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed3cc00640 1 -- 172.21.15.47:0/2321011036 shutdown_connections 2024-02-19T15:29:09.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed3cc00640 1 -- 172.21.15.47:0/2321011036 wait complete. 2024-02-19T15:29:09.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed3cc00640 1 Processor -- start 2024-02-19T15:29:09.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed3cc00640 1 -- start start 2024-02-19T15:29:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed3cc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed3806e5f0 0x7fed38147330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed3cc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed3806fdf0 0x7fed381569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed3cc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed381613a0 0x7fed3815d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed3cc00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3815fbf0 con 0x7fed381613a0 2024-02-19T15:29:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed3cc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed38155ec0 con 0x7fed3806e5f0 2024-02-19T15:29:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed3cc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed381561a0 con 0x7fed3806fdf0 2024-02-19T15:29:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed36800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed381613a0 0x7fed3815d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed36800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed381613a0 0x7fed3815d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49204/0 (socket says 172.21.15.47:49204) 2024-02-19T15:29:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed36800640 1 -- 172.21.15.47:0/2927770387 learned_addr learned my addr 172.21.15.47:0/2927770387 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed2fe00640 1 --2- 172.21.15.47:0/2927770387 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed3806fdf0 0x7fed381569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed35e00640 1 --2- 172.21.15.47:0/2927770387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed3806e5f0 0x7fed38147330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:09.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed36800640 1 -- 172.21.15.47:0/2927770387 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed3806fdf0 msgr2=0x7fed381569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:09.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed36800640 1 --2- 172.21.15.47:0/2927770387 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed3806fdf0 0x7fed381569a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:09.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed36800640 1 -- 172.21.15.47:0/2927770387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed3806e5f0 msgr2=0x7fed38147330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:09.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed36800640 1 --2- 172.21.15.47:0/2927770387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed3806e5f0 0x7fed38147330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:09.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed36800640 1 -- 172.21.15.47:0/2927770387 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed280094d0 con 0x7fed381613a0 2024-02-19T15:29:09.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed36800640 1 --2- 172.21.15.47:0/2927770387 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed381613a0 0x7fed3815d800 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fed3000efc0 tx=0x7fed3000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:09.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed2e000640 1 -- 172.21.15.47:0/2927770387 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed30014480 con 0x7fed381613a0 2024-02-19T15:29:09.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed2e000640 1 -- 172.21.15.47:0/2927770387 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3000f040 con 0x7fed381613a0 2024-02-19T15:29:09.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed3cc00640 1 -- 172.21.15.47:0/2927770387 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed38156480 con 0x7fed381613a0 2024-02-19T15:29:09.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.466+0000 7fed2e000640 1 -- 172.21.15.47:0/2927770387 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed30014820 con 0x7fed381613a0 2024-02-19T15:29:09.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.470+0000 7fed3cc00640 1 -- 172.21.15.47:0/2927770387 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed3815dd40 con 0x7fed381613a0 2024-02-19T15:29:09.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.470+0000 7fed3cc00640 1 -- 172.21.15.47:0/2927770387 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed38156670 con 0x7fed381613a0 2024-02-19T15:29:09.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.470+0000 7fed2e000640 1 -- 172.21.15.47:0/2927770387 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fed30014980 con 0x7fed381613a0 2024-02-19T15:29:09.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.470+0000 7fed2e000640 1 --2- 172.21.15.47:0/2927770387 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fed14066e70 0x7fed14069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:09.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.470+0000 7fed2e000640 1 -- 172.21.15.47:0/2927770387 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed30085520 con 0x7fed381613a0 2024-02-19T15:29:09.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.470+0000 7fed35e00640 1 --2- 172.21.15.47:0/2927770387 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fed14066e70 0x7fed14069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:09.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.474+0000 7fed35e00640 1 --2- 172.21.15.47:0/2927770387 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fed14066e70 0x7fed14069330 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fed280af000 tx=0x7fed280a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:09.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.474+0000 7fed2e000640 1 -- 172.21.15.47:0/2927770387 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fed30052320 con 0x7fed381613a0 2024-02-19T15:29:09.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:09.698+0000 7fed3cc00640 1 -- 172.21.15.47:0/2927770387 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7fed38075360 con 0x7fed381613a0 2024-02-19T15:29:10.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.058+0000 7fed2e000640 1 -- 172.21.15.47:0/2927770387 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7028 v95) v1 ==== 95+0+0 (secure 0 0 0) 0x7fed300559e0 con 0x7fed381613a0 2024-02-19T15:29:10.058 INFO:teuthology.orchestra.run.smithi047.stderr:failed mds gid 7028 2024-02-19T15:29:10.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.058+0000 7fed3cc00640 1 -- 172.21.15.47:0/2927770387 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fed14066e70 msgr2=0x7fed14069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:10.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.058+0000 7fed3cc00640 1 --2- 172.21.15.47:0/2927770387 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fed14066e70 0x7fed14069330 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fed280af000 tx=0x7fed280a7000 comp rx=0 tx=0).stop 2024-02-19T15:29:10.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.058+0000 7fed3cc00640 1 -- 172.21.15.47:0/2927770387 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed381613a0 msgr2=0x7fed3815d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:10.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.058+0000 7fed3cc00640 1 --2- 172.21.15.47:0/2927770387 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed381613a0 0x7fed3815d800 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fed3000efc0 tx=0x7fed3000c490 comp rx=0 tx=0).stop 2024-02-19T15:29:10.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.058+0000 7fed3cc00640 1 -- 172.21.15.47:0/2927770387 shutdown_connections 2024-02-19T15:29:10.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.058+0000 7fed3cc00640 1 --2- 172.21.15.47:0/2927770387 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fed14066e70 0x7fed14069330 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:10.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.058+0000 7fed3cc00640 1 --2- 172.21.15.47:0/2927770387 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed381613a0 0x7fed3815d800 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:10.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.058+0000 7fed3cc00640 1 --2- 172.21.15.47:0/2927770387 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed3806fdf0 0x7fed381569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:10.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.058+0000 7fed3cc00640 1 --2- 172.21.15.47:0/2927770387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed3806e5f0 0x7fed38147330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:10.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.058+0000 7fed3cc00640 1 -- 172.21.15.47:0/2927770387 >> 172.21.15.47:0/2927770387 conn(0x7fed380d0ad0 msgr2=0x7fed380b3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:10.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.058+0000 7fed3cc00640 1 -- 172.21.15.47:0/2927770387 shutdown_connections 2024-02-19T15:29:10.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.058+0000 7fed3cc00640 1 -- 172.21.15.47:0/2927770387 wait complete. 2024-02-19T15:29:10.081 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T15:29:10.081 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:29:10.096 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T15:29:10.097 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-02-19T15:29:10.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.242+0000 7ff143c00640 1 Processor -- start 2024-02-19T15:29:10.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.242+0000 7ff143c00640 1 -- start start 2024-02-19T15:29:10.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.242+0000 7ff143c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff13c0cded0 0x7ff13c0ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:10.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.242+0000 7ff143c00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13c0cce30 con 0x7ff13c06d650 2024-02-19T15:29:10.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.242+0000 7ff143c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13c0cd110 con 0x7ff13c0ceb40 2024-02-19T15:29:10.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.242+0000 7ff143c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff13c0cefd0 con 0x7ff13c0cded0 2024-02-19T15:29:10.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.242+0000 7ff133e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff13c0cded0 0x7ff13c0ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:10.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.242+0000 7ff141200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff13c06d650 0x7ff13c0d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34528/0 (socket says 172.21.15.47:34528) 2024-02-19T15:29:10.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.242+0000 7ff141200640 1 -- 172.21.15.47:0/1116101560 learned_addr learned my addr 172.21.15.47:0/1116101560 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:29:10.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.242+0000 7ff13be00640 1 -- 172.21.15.47:0/1116101560 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092328589 0 0) 0x7ff13c0cce30 con 0x7ff13c06d650 2024-02-19T15:29:10.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.242+0000 7ff133e00640 1 -- 172.21.15.47:0/1116101560 >> v1:172.21.15.116:6789/0 conn(0x7ff13c0ceb40 legacy=0x7ff13c06bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:10.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.242+0000 7ff133e00640 1 -- 172.21.15.47:0/1116101560 >> v1:172.21.15.47:6789/0 conn(0x7ff13c06d650 legacy=0x7ff13c0d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:10.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.242+0000 7ff133e00640 1 -- 172.21.15.47:0/1116101560 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13c0cd3f0 con 0x7ff13c0cded0 2024-02-19T15:29:10.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.246+0000 7ff133e00640 1 --2- 172.21.15.47:0/1116101560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0cded0 0x7ff13c0ce600 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7ff124009740 tx=0x7ff12401c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff818163a0a96b21 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:10.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.246+0000 7ff13be00640 1 -- 172.21.15.47:0/1116101560 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1240a4ea0 con 0x7ff13c0cded0 2024-02-19T15:29:10.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.246+0000 7ff13be00640 1 -- 172.21.15.47:0/1116101560 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1240a7050 con 0x7ff13c0cded0 2024-02-19T15:29:10.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.246+0000 7ff13be00640 1 -- 172.21.15.47:0/1116101560 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1240ab820 con 0x7ff13c0cded0 2024-02-19T15:29:10.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.246+0000 7ff143c00640 1 -- 172.21.15.47:0/1116101560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0cded0 msgr2=0x7ff13c0ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:10.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.246+0000 7ff143c00640 1 --2- 172.21.15.47:0/1116101560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0cded0 0x7ff13c0ce600 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7ff124009740 tx=0x7ff12401c210 comp rx=0 tx=0).stop 2024-02-19T15:29:10.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.246+0000 7ff143c00640 1 -- 172.21.15.47:0/1116101560 shutdown_connections 2024-02-19T15:29:10.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.246+0000 7ff143c00640 1 --2- 172.21.15.47:0/1116101560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0cded0 0x7ff13c0ce600 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:10.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.246+0000 7ff143c00640 1 -- 172.21.15.47:0/1116101560 >> 172.21.15.47:0/1116101560 conn(0x7ff13c0d1bf0 msgr2=0x7ff13c0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:10.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.246+0000 7ff143c00640 1 -- 172.21.15.47:0/1116101560 shutdown_connections 2024-02-19T15:29:10.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.246+0000 7ff143c00640 1 -- 172.21.15.47:0/1116101560 wait complete. 2024-02-19T15:29:10.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff143c00640 1 Processor -- start 2024-02-19T15:29:10.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff143c00640 1 -- start start 2024-02-19T15:29:10.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff143c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff13c06d650 0x7ff13c15e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:10.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff143c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0cded0 0x7ff13c15b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:10.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff143c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c0ceb40 0x7ff13c15f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:10.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff143c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c0c8cc0 con 0x7ff13c0cded0 2024-02-19T15:29:10.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff143c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c0c9560 con 0x7ff13c0ceb40 2024-02-19T15:29:10.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff143c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c15f2d0 con 0x7ff13c06d650 2024-02-19T15:29:10.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff133e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0cded0 0x7ff13c15b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:10.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff133e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0cded0 0x7ff13c15b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49224/0 (socket says 172.21.15.47:49224) 2024-02-19T15:29:10.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff133e00640 1 -- 172.21.15.47:0/623040820 learned_addr learned my addr 172.21.15.47:0/623040820 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:10.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff141c00640 1 --2- 172.21.15.47:0/623040820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c0ceb40 0x7ff13c15f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:10.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff141200640 1 --2- 172.21.15.47:0/623040820 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff13c06d650 0x7ff13c15e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:10.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff133e00640 1 -- 172.21.15.47:0/623040820 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff13c06d650 msgr2=0x7ff13c15e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:10.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff133e00640 1 --2- 172.21.15.47:0/623040820 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff13c06d650 0x7ff13c15e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:10.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff133e00640 1 -- 172.21.15.47:0/623040820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c0ceb40 msgr2=0x7ff13c15f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:10.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff133e00640 1 --2- 172.21.15.47:0/623040820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c0ceb40 0x7ff13c15f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:10.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff133e00640 1 -- 172.21.15.47:0/623040820 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff124009420 con 0x7ff13c0cded0 2024-02-19T15:29:10.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff133e00640 1 --2- 172.21.15.47:0/623040820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0cded0 0x7ff13c15b8d0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7ff124009740 tx=0x7ff124018fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:10.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff139600640 1 -- 172.21.15.47:0/623040820 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1240b4610 con 0x7ff13c0cded0 2024-02-19T15:29:10.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff139600640 1 -- 172.21.15.47:0/623040820 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1240abe90 con 0x7ff13c0cded0 2024-02-19T15:29:10.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff139600640 1 -- 172.21.15.47:0/623040820 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1240b49d0 con 0x7ff13c0cded0 2024-02-19T15:29:10.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.250+0000 7ff143c00640 1 -- 172.21.15.47:0/623040820 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff13c0c8f10 con 0x7ff13c0cded0 2024-02-19T15:29:10.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.254+0000 7ff143c00640 1 -- 172.21.15.47:0/623040820 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13c15ef10 con 0x7ff13c0cded0 2024-02-19T15:29:10.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.254+0000 7ff143c00640 1 -- 172.21.15.47:0/623040820 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13c0c96f0 con 0x7ff13c0cded0 2024-02-19T15:29:10.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.254+0000 7ff139600640 1 -- 172.21.15.47:0/623040820 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff1240b2040 con 0x7ff13c0cded0 2024-02-19T15:29:10.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.254+0000 7ff139600640 1 --2- 172.21.15.47:0/623040820 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff118066e40 0x7ff118069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:10.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.254+0000 7ff139600640 1 -- 172.21.15.47:0/623040820 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff12411e9c0 con 0x7ff13c0cded0 2024-02-19T15:29:10.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.254+0000 7ff141200640 1 --2- 172.21.15.47:0/623040820 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff118066e40 0x7ff118069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:10.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.254+0000 7ff141200640 1 --2- 172.21.15.47:0/623040820 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff118066e40 0x7ff118069300 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff12c000f70 tx=0x7ff12c007220 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:10.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.258+0000 7ff139600640 1 -- 172.21.15.47:0/623040820 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff1240eb850 con 0x7ff13c0cded0 2024-02-19T15:29:10.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:10.482+0000 7ff143c00640 1 -- 172.21.15.47:0/623040820 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7ff13c075360 con 0x7ff13c0cded0 2024-02-19T15:29:11.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.058+0000 7ff139600640 1 -- 172.21.15.47:0/623040820 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 7034 v96) v1 ==== 95+0+0 (secure 0 0 0) 0x7ff1240c2090 con 0x7ff13c0cded0 2024-02-19T15:29:11.059 INFO:teuthology.orchestra.run.smithi047.stderr:failed mds gid 7034 2024-02-19T15:29:11.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.062+0000 7ff143c00640 1 -- 172.21.15.47:0/623040820 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff118066e40 msgr2=0x7ff118069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:11.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.062+0000 7ff143c00640 1 --2- 172.21.15.47:0/623040820 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff118066e40 0x7ff118069300 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff12c000f70 tx=0x7ff12c007220 comp rx=0 tx=0).stop 2024-02-19T15:29:11.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.062+0000 7ff143c00640 1 -- 172.21.15.47:0/623040820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0cded0 msgr2=0x7ff13c15b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:11.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.062+0000 7ff143c00640 1 --2- 172.21.15.47:0/623040820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0cded0 0x7ff13c15b8d0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7ff124009740 tx=0x7ff124018fb0 comp rx=0 tx=0).stop 2024-02-19T15:29:11.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.062+0000 7ff143c00640 1 -- 172.21.15.47:0/623040820 shutdown_connections 2024-02-19T15:29:11.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.062+0000 7ff143c00640 1 --2- 172.21.15.47:0/623040820 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff118066e40 0x7ff118069300 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:11.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.062+0000 7ff143c00640 1 --2- 172.21.15.47:0/623040820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c0ceb40 0x7ff13c15f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:11.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.062+0000 7ff143c00640 1 --2- 172.21.15.47:0/623040820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c0cded0 0x7ff13c15b8d0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:11.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.062+0000 7ff143c00640 1 --2- 172.21.15.47:0/623040820 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff13c06d650 0x7ff13c15e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:11.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.062+0000 7ff143c00640 1 -- 172.21.15.47:0/623040820 >> 172.21.15.47:0/623040820 conn(0x7ff13c0d1bf0 msgr2=0x7ff13c06bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:11.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.062+0000 7ff143c00640 1 -- 172.21.15.47:0/623040820 shutdown_connections 2024-02-19T15:29:11.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.062+0000 7ff143c00640 1 -- 172.21.15.47:0/623040820 wait complete. 2024-02-19T15:29:11.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:29:11.086 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-02-19T15:29:11.086 DEBUG:teuthology.orchestra.run.smithi047:> mkdir pin 2024-02-19T15:29:11.086 DEBUG:teuthology.orchestra.run.smithi047:> touch pin/placeholder 2024-02-19T15:29:11.086 DEBUG:teuthology.orchestra.run.smithi047:> ') 2024-02-19T15:29:11.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:29:11.107 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 pin) 2024-02-19T15:29:11.163 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:11.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.346+0000 7f7a27a00640 1 Processor -- start 2024-02-19T15:29:11.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.346+0000 7f7a27a00640 1 -- start start 2024-02-19T15:29:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.346+0000 7f7a27a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7a2015c390 0x7f7a200b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.350+0000 7f7a27a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a200cf920 0x7f7a200d0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.350+0000 7f7a27a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a200cda30 con 0x7f7a200ceca0 2024-02-19T15:29:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.350+0000 7f7a27a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a200d2ae0 con 0x7f7a2015c390 2024-02-19T15:29:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.350+0000 7f7a27a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a200cfdb0 con 0x7f7a200cf920 2024-02-19T15:29:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.350+0000 7f7a17e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a200cf920 0x7f7a200d0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.350+0000 7f7a25000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7a2015c390 0x7f7a200b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.350+0000 7f7a17e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a200cf920 0x7f7a200d0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:47018/0 (socket says 172.21.15.47:47018) 2024-02-19T15:29:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.350+0000 7f7a17e00640 1 -- 172.21.15.47:0/3461419221 learned_addr learned my addr 172.21.15.47:0/3461419221 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.350+0000 7f7a25000640 1 -- 172.21.15.47:0/3461419221 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a200cf920 msgr2=0x7f7a200d0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.350+0000 7f7a25000640 1 --2- 172.21.15.47:0/3461419221 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a200cf920 0x7f7a200d0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:11.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.350+0000 7f7a25000640 1 -- 172.21.15.47:0/3461419221 >> v1:172.21.15.116:6789/0 conn(0x7f7a200ceca0 legacy=0x7f7a200b14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:11.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.350+0000 7f7a25000640 1 -- 172.21.15.47:0/3461419221 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a200d2800 con 0x7f7a2015c390 2024-02-19T15:29:11.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.350+0000 7f7a17e00640 1 --2- 172.21.15.47:0/3461419221 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a200cf920 0x7f7a200d0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:29:11.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.350+0000 7f7a25000640 1 --2- 172.21.15.47:0/3461419221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a2015c390 0x7f7a200b2c10 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7a100097c0 tx=0x7f7a1001c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4df02137e86a98bb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:11.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.350+0000 7f7a1fe00640 1 -- 172.21.15.47:0/3461419221 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a100ab600 con 0x7f7a2015c390 2024-02-19T15:29:11.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a1fe00640 1 -- 172.21.15.47:0/3461419221 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a100a7050 con 0x7f7a2015c390 2024-02-19T15:29:11.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a1fe00640 1 -- 172.21.15.47:0/3461419221 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a100ab980 con 0x7f7a2015c390 2024-02-19T15:29:11.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a27a00640 1 -- 172.21.15.47:0/3461419221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a2015c390 msgr2=0x7f7a200b2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:11.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a27a00640 1 --2- 172.21.15.47:0/3461419221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a2015c390 0x7f7a200b2c10 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7a100097c0 tx=0x7f7a1001c3e0 comp rx=0 tx=0).stop 2024-02-19T15:29:11.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a27a00640 1 -- 172.21.15.47:0/3461419221 shutdown_connections 2024-02-19T15:29:11.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a27a00640 1 --2- 172.21.15.47:0/3461419221 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a200cf920 0x7f7a200d0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:11.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a27a00640 1 --2- 172.21.15.47:0/3461419221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a2015c390 0x7f7a200b2c10 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:11.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a27a00640 1 -- 172.21.15.47:0/3461419221 >> 172.21.15.47:0/3461419221 conn(0x7f7a200d1b40 msgr2=0x7f7a200d1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:11.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a27a00640 1 -- 172.21.15.47:0/3461419221 shutdown_connections 2024-02-19T15:29:11.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a27a00640 1 -- 172.21.15.47:0/3461419221 wait complete. 2024-02-19T15:29:11.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a27a00640 1 Processor -- start 2024-02-19T15:29:11.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a27a00640 1 -- start start 2024-02-19T15:29:11.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a27a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a200ceca0 0x7f7a20147f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:11.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a27a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a2015c390 0x7f7a200b4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:11.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a27a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a200cb340 0x7f7a200c5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:11.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a27a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a200b5370 con 0x7f7a2015c390 2024-02-19T15:29:11.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a27a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2014e3c0 con 0x7f7a200ceca0 2024-02-19T15:29:11.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a27a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a200b5560 con 0x7f7a200cb340 2024-02-19T15:29:11.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a17e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a2015c390 0x7f7a200b4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:11.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a17e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a2015c390 0x7f7a200b4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49234/0 (socket says 172.21.15.47:49234) 2024-02-19T15:29:11.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a17e00640 1 -- 172.21.15.47:0/2533456690 learned_addr learned my addr 172.21.15.47:0/2533456690 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:11.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a25a00640 1 --2- 172.21.15.47:0/2533456690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a200cb340 0x7f7a200c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:11.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.354+0000 7f7a25000640 1 --2- 172.21.15.47:0/2533456690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a200ceca0 0x7f7a20147f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:11.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.358+0000 7f7a17e00640 1 -- 172.21.15.47:0/2533456690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a200cb340 msgr2=0x7f7a200c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:11.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.358+0000 7f7a17e00640 1 --2- 172.21.15.47:0/2533456690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a200cb340 0x7f7a200c5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:11.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.358+0000 7f7a17e00640 1 -- 172.21.15.47:0/2533456690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a200ceca0 msgr2=0x7f7a20147f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:11.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.358+0000 7f7a17e00640 1 --2- 172.21.15.47:0/2533456690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a200ceca0 0x7f7a20147f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:11.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.358+0000 7f7a17e00640 1 -- 172.21.15.47:0/2533456690 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a080094d0 con 0x7f7a2015c390 2024-02-19T15:29:11.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.358+0000 7f7a25a00640 1 --2- 172.21.15.47:0/2533456690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a200cb340 0x7f7a200c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:29:11.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.358+0000 7f7a17e00640 1 --2- 172.21.15.47:0/2533456690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a2015c390 0x7f7a200b4e30 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f7a0800e9d0 tx=0x7f7a0800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:11.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.358+0000 7f7a1d600640 1 -- 172.21.15.47:0/2533456690 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0800ca60 con 0x7f7a2015c390 2024-02-19T15:29:11.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.358+0000 7f7a1d600640 1 -- 172.21.15.47:0/2533456690 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0800cbc0 con 0x7f7a2015c390 2024-02-19T15:29:11.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.358+0000 7f7a1d600640 1 -- 172.21.15.47:0/2533456690 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a08013470 con 0x7f7a2015c390 2024-02-19T15:29:11.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.358+0000 7f7a27a00640 1 -- 172.21.15.47:0/2533456690 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a10009420 con 0x7f7a2015c390 2024-02-19T15:29:11.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.358+0000 7f7a27a00640 1 -- 172.21.15.47:0/2533456690 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a200b5b20 con 0x7f7a2015c390 2024-02-19T15:29:11.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.358+0000 7f7a27a00640 1 -- 172.21.15.47:0/2533456690 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a20075360 con 0x7f7a2015c390 2024-02-19T15:29:11.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.358+0000 7f7a1d600640 1 -- 172.21.15.47:0/2533456690 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7a080135d0 con 0x7f7a2015c390 2024-02-19T15:29:11.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.358+0000 7f7a1d600640 1 --2- 172.21.15.47:0/2533456690 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f79fc066e40 0x7f79fc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:11.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.358+0000 7f7a1d600640 1 -- 172.21.15.47:0/2533456690 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a08084d40 con 0x7f7a2015c390 2024-02-19T15:29:11.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.362+0000 7f7a25000640 1 --2- 172.21.15.47:0/2533456690 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f79fc066e40 0x7f79fc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:11.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.362+0000 7f7a25000640 1 --2- 172.21.15.47:0/2533456690 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f79fc066e40 0x7f79fc069300 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f7a100ad000 tx=0x7f7a100a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:11.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.362+0000 7f7a1d600640 1 -- 172.21.15.47:0/2533456690 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7a08051b50 con 0x7f7a2015c390 2024-02-19T15:29:11.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.562+0000 7f7a27a00640 1 -- 172.21.15.47:0/2533456690 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a200cff00 con 0x7f7a2015c390 2024-02-19T15:29:11.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.566+0000 7f7a1d600640 1 -- 172.21.15.47:0/2533456690 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3376 (secure 0 0 0) 0x7f7a08017d40 con 0x7f7a2015c390 2024-02-19T15:29:11.565 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:11.565 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":96,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:03.058284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:11.565 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 96 2024-02-19T15:29:11.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.566+0000 7f7a27a00640 1 -- 172.21.15.47:0/2533456690 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f79fc066e40 msgr2=0x7f79fc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:11.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.566+0000 7f7a27a00640 1 --2- 172.21.15.47:0/2533456690 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f79fc066e40 0x7f79fc069300 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f7a100ad000 tx=0x7f7a100a6000 comp rx=0 tx=0).stop 2024-02-19T15:29:11.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.566+0000 7f7a27a00640 1 -- 172.21.15.47:0/2533456690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a2015c390 msgr2=0x7f7a200b4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:11.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.566+0000 7f7a27a00640 1 --2- 172.21.15.47:0/2533456690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a2015c390 0x7f7a200b4e30 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f7a0800e9d0 tx=0x7f7a0800ee90 comp rx=0 tx=0).stop 2024-02-19T15:29:11.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.566+0000 7f7a27a00640 1 -- 172.21.15.47:0/2533456690 shutdown_connections 2024-02-19T15:29:11.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.566+0000 7f7a27a00640 1 --2- 172.21.15.47:0/2533456690 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f79fc066e40 0x7f79fc069300 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:11.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.566+0000 7f7a27a00640 1 --2- 172.21.15.47:0/2533456690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a200cb340 0x7f7a200c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:11.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.566+0000 7f7a27a00640 1 --2- 172.21.15.47:0/2533456690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a2015c390 0x7f7a200b4e30 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:11.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.566+0000 7f7a27a00640 1 --2- 172.21.15.47:0/2533456690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a200ceca0 0x7f7a20147f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:11.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.566+0000 7f7a27a00640 1 -- 172.21.15.47:0/2533456690 >> 172.21.15.47:0/2533456690 conn(0x7f7a200d1b40 msgr2=0x7f7a201672d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:11.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.566+0000 7f7a27a00640 1 -- 172.21.15.47:0/2533456690 shutdown_connections 2024-02-19T15:29:11.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:11.566+0000 7f7a27a00640 1 -- 172.21.15.47:0/2533456690 wait complete. 2024-02-19T15:29:11.586 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-02-19T15:29:11.739 INFO:teuthology.orchestra.run.smithi116.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-02-19T15:29:11.740 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-02-19T15:29:11.740 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-02-19T15:29:13.740 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.882+0000 7f3684a00640 1 Processor -- start 2024-02-19T15:29:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.882+0000 7f3684a00640 1 -- start start 2024-02-19T15:29:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.882+0000 7f3684a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f368015b1a0 0x7f368015b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.882+0000 7f3684a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36800718c0 con 0x7f368006c9a0 2024-02-19T15:29:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.882+0000 7f3684a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3680071ba0 con 0x7f3680071350 2024-02-19T15:29:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.882+0000 7f3684a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3680071e80 con 0x7f368015b1a0 2024-02-19T15:29:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.882+0000 7f367de00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f368015b1a0 0x7f368015b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.882+0000 7f367e800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f368006c9a0 0x7f368015cde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34082/0 (socket says 172.21.15.47:34082) 2024-02-19T15:29:13.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.882+0000 7f367e800640 1 -- 172.21.15.47:0/2702118833 learned_addr learned my addr 172.21.15.47:0/2702118833 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:29:13.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.886+0000 7f367d400640 1 -- 172.21.15.47:0/2702118833 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489381632 0 0) 0x7f36800718c0 con 0x7f368006c9a0 2024-02-19T15:29:13.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.886+0000 7f367d400640 1 -- 172.21.15.47:0/2702118833 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3664005740 con 0x7f368006c9a0 2024-02-19T15:29:13.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.886+0000 7f367d400640 1 -- 172.21.15.47:0/2702118833 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815685484 0 0) 0x7f3680071ba0 con 0x7f3680071350 2024-02-19T15:29:13.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.886+0000 7f367de00640 1 -- 172.21.15.47:0/2702118833 >> v1:172.21.15.116:6789/0 conn(0x7f3680071350 legacy=0x7f368015c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:13.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.886+0000 7f367de00640 1 -- 172.21.15.47:0/2702118833 >> v1:172.21.15.47:6789/0 conn(0x7f368006c9a0 legacy=0x7f368015cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:13.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.886+0000 7f367de00640 1 -- 172.21.15.47:0/2702118833 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3680072160 con 0x7f368015b1a0 2024-02-19T15:29:13.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.886+0000 7f367de00640 1 --2- 172.21.15.47:0/2702118833 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f368015b1a0 0x7f368015b580 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f36700097f0 tx=0x7f367001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bae25af7d9fe0093 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:13.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.886+0000 7f367d400640 1 -- 172.21.15.47:0/2702118833 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36700ab600 con 0x7f368015b1a0 2024-02-19T15:29:13.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.886+0000 7f367d400640 1 -- 172.21.15.47:0/2702118833 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36700a7050 con 0x7f368015b1a0 2024-02-19T15:29:13.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.886+0000 7f367d400640 1 -- 172.21.15.47:0/2702118833 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36700ab980 con 0x7f368015b1a0 2024-02-19T15:29:13.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.886+0000 7f3684a00640 1 -- 172.21.15.47:0/2702118833 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f368015b1a0 msgr2=0x7f368015b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:13.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.886+0000 7f3684a00640 1 --2- 172.21.15.47:0/2702118833 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f368015b1a0 0x7f368015b580 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f36700097f0 tx=0x7f367001c4a0 comp rx=0 tx=0).stop 2024-02-19T15:29:13.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.886+0000 7f3684a00640 1 -- 172.21.15.47:0/2702118833 shutdown_connections 2024-02-19T15:29:13.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.886+0000 7f3684a00640 1 --2- 172.21.15.47:0/2702118833 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f368015b1a0 0x7f368015b580 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:13.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.886+0000 7f3684a00640 1 -- 172.21.15.47:0/2702118833 >> 172.21.15.47:0/2702118833 conn(0x7f36800b16a0 msgr2=0x7f36800b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:13.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.886+0000 7f3684a00640 1 -- 172.21.15.47:0/2702118833 shutdown_connections 2024-02-19T15:29:13.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.886+0000 7f3684a00640 1 -- 172.21.15.47:0/2702118833 wait complete. 2024-02-19T15:29:13.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.890+0000 7f3684a00640 1 Processor -- start 2024-02-19T15:29:13.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.890+0000 7f3684a00640 1 -- start start 2024-02-19T15:29:13.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f3684a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f368006c9a0 0x7f3680138120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:13.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f3684a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3680071350 0x7f368014b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:13.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f3684a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f368015b1a0 0x7f36800cdf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:13.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f3684a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368014e150 con 0x7f368006c9a0 2024-02-19T15:29:13.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f3684a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368014d540 con 0x7f368015b1a0 2024-02-19T15:29:13.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f3684a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f368014d820 con 0x7f3680071350 2024-02-19T15:29:13.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f367de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f368006c9a0 0x7f3680138120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:13.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f367de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f368006c9a0 0x7f3680138120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39614/0 (socket says 172.21.15.47:39614) 2024-02-19T15:29:13.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f367de00640 1 -- 172.21.15.47:0/4180998945 learned_addr learned my addr 172.21.15.47:0/4180998945 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:13.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f367e800640 1 --2- 172.21.15.47:0/4180998945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f368015b1a0 0x7f36800cdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:13.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f3677e00640 1 --2- 172.21.15.47:0/4180998945 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3680071350 0x7f368014b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:13.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f367de00640 1 -- 172.21.15.47:0/4180998945 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3680071350 msgr2=0x7f368014b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:13.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f367de00640 1 --2- 172.21.15.47:0/4180998945 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3680071350 0x7f368014b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:13.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f367de00640 1 -- 172.21.15.47:0/4180998945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f368015b1a0 msgr2=0x7f36800cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:13.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f367de00640 1 --2- 172.21.15.47:0/4180998945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f368015b1a0 0x7f36800cdf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:13.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f367de00640 1 -- 172.21.15.47:0/4180998945 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3670009420 con 0x7f368006c9a0 2024-02-19T15:29:13.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f367de00640 1 --2- 172.21.15.47:0/4180998945 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f368006c9a0 0x7f3680138120 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f367001c980 tx=0x7f3670019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:13.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f3676000640 1 -- 172.21.15.47:0/4180998945 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36700bb070 con 0x7f368006c9a0 2024-02-19T15:29:13.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f3676000640 1 -- 172.21.15.47:0/4180998945 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36700abd20 con 0x7f368006c9a0 2024-02-19T15:29:13.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f3684a00640 1 -- 172.21.15.47:0/4180998945 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f368014db00 con 0x7f368006c9a0 2024-02-19T15:29:13.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f3676000640 1 -- 172.21.15.47:0/4180998945 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36700b4650 con 0x7f368006c9a0 2024-02-19T15:29:13.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f3684a00640 1 -- 172.21.15.47:0/4180998945 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f368014d1a0 con 0x7f368006c9a0 2024-02-19T15:29:13.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f3684a00640 1 -- 172.21.15.47:0/4180998945 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3680075360 con 0x7f368006c9a0 2024-02-19T15:29:13.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.894+0000 7f3676000640 1 -- 172.21.15.47:0/4180998945 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f36700b2040 con 0x7f368006c9a0 2024-02-19T15:29:13.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.898+0000 7f3676000640 1 --2- 172.21.15.47:0/4180998945 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f365c066e40 0x7f365c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:13.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.898+0000 7f3676000640 1 -- 172.21.15.47:0/4180998945 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f367011de80 con 0x7f368006c9a0 2024-02-19T15:29:13.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.898+0000 7f3677e00640 1 --2- 172.21.15.47:0/4180998945 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f365c066e40 0x7f365c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:13.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.898+0000 7f3677e00640 1 --2- 172.21.15.47:0/4180998945 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f365c066e40 0x7f365c069300 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f368014b380 tx=0x7f3668009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:13.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:13.902+0000 7f3676000640 1 -- 172.21.15.47:0/4180998945 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f36700eac70 con 0x7f368006c9a0 2024-02-19T15:29:14.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:14.178+0000 7f3684a00640 1 -- 172.21.15.47:0/4180998945 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36801509c0 con 0x7f368006c9a0 2024-02-19T15:29:14.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:14.178+0000 7f3676000640 1 -- 172.21.15.47:0/4180998945 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3376 (secure 0 0 0) 0x7f36700b0d40 con 0x7f368006c9a0 2024-02-19T15:29:14.178 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:14.178 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":96,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:03.058284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:14.178 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 96 2024-02-19T15:29:14.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:14.178+0000 7f3684a00640 1 -- 172.21.15.47:0/4180998945 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f365c066e40 msgr2=0x7f365c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:14.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:14.178+0000 7f3684a00640 1 --2- 172.21.15.47:0/4180998945 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f365c066e40 0x7f365c069300 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f368014b380 tx=0x7f3668009040 comp rx=0 tx=0).stop 2024-02-19T15:29:14.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:14.178+0000 7f3684a00640 1 -- 172.21.15.47:0/4180998945 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f368006c9a0 msgr2=0x7f3680138120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:14.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:14.178+0000 7f3684a00640 1 --2- 172.21.15.47:0/4180998945 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f368006c9a0 0x7f3680138120 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f367001c980 tx=0x7f3670019240 comp rx=0 tx=0).stop 2024-02-19T15:29:14.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:14.178+0000 7f3684a00640 1 -- 172.21.15.47:0/4180998945 shutdown_connections 2024-02-19T15:29:14.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:14.178+0000 7f3684a00640 1 --2- 172.21.15.47:0/4180998945 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f365c066e40 0x7f365c069300 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:14.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:14.178+0000 7f3684a00640 1 --2- 172.21.15.47:0/4180998945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f368015b1a0 0x7f36800cdf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:14.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:14.178+0000 7f3684a00640 1 --2- 172.21.15.47:0/4180998945 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3680071350 0x7f368014b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:14.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:14.178+0000 7f3684a00640 1 --2- 172.21.15.47:0/4180998945 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f368006c9a0 0x7f3680138120 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:14.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:14.178+0000 7f3684a00640 1 -- 172.21.15.47:0/4180998945 >> 172.21.15.47:0/4180998945 conn(0x7f36800b16a0 msgr2=0x7f3680070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:14.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:14.178+0000 7f3684a00640 1 -- 172.21.15.47:0/4180998945 shutdown_connections 2024-02-19T15:29:14.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:14.178+0000 7f3684a00640 1 -- 172.21.15.47:0/4180998945 wait complete. 2024-02-19T15:29:14.199 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-02-19T15:29:14.359 INFO:teuthology.orchestra.run.smithi116.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-02-19T15:29:14.360 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-02-19T15:29:14.360 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-02-19T15:29:16.361 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:16.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.502+0000 7f58c3e00640 1 Processor -- start 2024-02-19T15:29:16.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.502+0000 7f58c3e00640 1 -- start start 2024-02-19T15:29:16.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.502+0000 7f58c3e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f58c40717e0 0x7f58c40b1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:16.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.502+0000 7f58c3e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c40b1a50 con 0x7f58c415ba70 2024-02-19T15:29:16.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.502+0000 7f58c3e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c40b1d30 con 0x7f58c4071400 2024-02-19T15:29:16.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.502+0000 7f58c3e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58c40b2010 con 0x7f58c40717e0 2024-02-19T15:29:16.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.502+0000 7f58c3400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f58c415ba70 0x7f58c415a4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34094/0 (socket says 172.21.15.47:34094) 2024-02-19T15:29:16.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.502+0000 7f58c3400640 1 -- 172.21.15.47:0/2499177131 learned_addr learned my addr 172.21.15.47:0/2499177131 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:29:16.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.502+0000 7f58c2000640 1 --2- 172.21.15.47:0/2499177131 >> v2:172.21.15.116:3300/0 conn(0x7f58c40717e0 0x7f58c40b1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:16.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.502+0000 7f58c1600640 1 -- 172.21.15.47:0/2499177131 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953811172 0 0) 0x7f58c40b1a50 con 0x7f58c415ba70 2024-02-19T15:29:16.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.502+0000 7f58c1600640 1 -- 172.21.15.47:0/2499177131 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58a8003530 con 0x7f58c415ba70 2024-02-19T15:29:16.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.502+0000 7f58c1600640 1 -- 172.21.15.47:0/2499177131 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53313575 0 0) 0x7f58c40b1d30 con 0x7f58c4071400 2024-02-19T15:29:16.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.502+0000 7f58c1600640 1 -- 172.21.15.47:0/2499177131 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c40b1a50 con 0x7f58c4071400 2024-02-19T15:29:16.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.502+0000 7f58c2000640 1 -- 172.21.15.47:0/2499177131 >> v1:172.21.15.116:6789/0 conn(0x7f58c4071400 legacy=0x7f58c4070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:16.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.502+0000 7f58c2000640 1 -- 172.21.15.47:0/2499177131 >> v1:172.21.15.47:6789/0 conn(0x7f58c415ba70 legacy=0x7f58c415a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:16.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.502+0000 7f58c2000640 1 -- 172.21.15.47:0/2499177131 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c415b270 con 0x7f58c40717e0 2024-02-19T15:29:16.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.506+0000 7f58c2000640 1 --2- 172.21.15.47:0/2499177131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c40717e0 0x7f58c40b1400 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f58b00097c0 tx=0x7f58b001c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc4617c06a6dc12b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:16.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.506+0000 7f58c1600640 1 -- 172.21.15.47:0/2499177131 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b00ad680 con 0x7f58c40717e0 2024-02-19T15:29:16.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.506+0000 7f58c1600640 1 -- 172.21.15.47:0/2499177131 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b00ad7e0 con 0x7f58c40717e0 2024-02-19T15:29:16.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.506+0000 7f58c1600640 1 -- 172.21.15.47:0/2499177131 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b00adb50 con 0x7f58c40717e0 2024-02-19T15:29:16.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.506+0000 7f58c3e00640 1 -- 172.21.15.47:0/2499177131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c40717e0 msgr2=0x7f58c40b1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:16.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.506+0000 7f58c3e00640 1 --2- 172.21.15.47:0/2499177131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c40717e0 0x7f58c40b1400 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f58b00097c0 tx=0x7f58b001c470 comp rx=0 tx=0).stop 2024-02-19T15:29:16.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.506+0000 7f58c3e00640 1 -- 172.21.15.47:0/2499177131 shutdown_connections 2024-02-19T15:29:16.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.506+0000 7f58c3e00640 1 --2- 172.21.15.47:0/2499177131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c40717e0 0x7f58c40b1400 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:16.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.506+0000 7f58c3e00640 1 -- 172.21.15.47:0/2499177131 >> 172.21.15.47:0/2499177131 conn(0x7f58c40cf7c0 msgr2=0x7f58c40cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:16.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.506+0000 7f58c3e00640 1 -- 172.21.15.47:0/2499177131 shutdown_connections 2024-02-19T15:29:16.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.506+0000 7f58c3e00640 1 -- 172.21.15.47:0/2499177131 wait complete. 2024-02-19T15:29:16.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c3e00640 1 Processor -- start 2024-02-19T15:29:16.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c3e00640 1 -- start start 2024-02-19T15:29:16.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c4071400 0x7f58c41566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:16.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58c40717e0 0x7f58c4154600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:16.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c3e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58c415ba70 0x7f58c4150080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:16.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c3e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c40ced50 con 0x7f58c40717e0 2024-02-19T15:29:16.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c3e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c40ceff0 con 0x7f58c4071400 2024-02-19T15:29:16.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c3e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c41540e0 con 0x7f58c415ba70 2024-02-19T15:29:16.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c2000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58c40717e0 0x7f58c4154600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:16.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c2a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c4071400 0x7f58c41566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:16.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c2000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58c40717e0 0x7f58c4154600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39626/0 (socket says 172.21.15.47:39626) 2024-02-19T15:29:16.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c2000640 1 -- 172.21.15.47:0/53467625 learned_addr learned my addr 172.21.15.47:0/53467625 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:16.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c3400640 1 --2- 172.21.15.47:0/53467625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58c415ba70 0x7f58c4150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:16.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c2a00640 1 -- 172.21.15.47:0/53467625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58c415ba70 msgr2=0x7f58c4150080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:16.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c2a00640 1 --2- 172.21.15.47:0/53467625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58c415ba70 0x7f58c4150080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:16.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c2a00640 1 -- 172.21.15.47:0/53467625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58c40717e0 msgr2=0x7f58c4154600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:16.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c2a00640 1 --2- 172.21.15.47:0/53467625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58c40717e0 0x7f58c4154600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:16.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c2a00640 1 -- 172.21.15.47:0/53467625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b0009420 con 0x7f58c4071400 2024-02-19T15:29:16.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c3400640 1 --2- 172.21.15.47:0/53467625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58c415ba70 0x7f58c4150080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:29:16.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c2000640 1 --2- 172.21.15.47:0/53467625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58c40717e0 0x7f58c4154600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:29:16.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c2a00640 1 --2- 172.21.15.47:0/53467625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c4071400 0x7f58c41566d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f58b400c3c0 tx=0x7f58b400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:16.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58baa00640 1 -- 172.21.15.47:0/53467625 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b40052d0 con 0x7f58c4071400 2024-02-19T15:29:16.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58baa00640 1 -- 172.21.15.47:0/53467625 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b4005450 con 0x7f58c4071400 2024-02-19T15:29:16.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58baa00640 1 -- 172.21.15.47:0/53467625 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b4003840 con 0x7f58c4071400 2024-02-19T15:29:16.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c3e00640 1 -- 172.21.15.47:0/53467625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c4157d30 con 0x7f58c4071400 2024-02-19T15:29:16.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.510+0000 7f58c3e00640 1 -- 172.21.15.47:0/53467625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c4156c10 con 0x7f58c4071400 2024-02-19T15:29:16.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.514+0000 7f58c3e00640 1 -- 172.21.15.47:0/53467625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58c4153f40 con 0x7f58c4071400 2024-02-19T15:29:16.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.514+0000 7f58baa00640 1 -- 172.21.15.47:0/53467625 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f58b40039a0 con 0x7f58c4071400 2024-02-19T15:29:16.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.514+0000 7f58baa00640 1 --2- 172.21.15.47:0/53467625 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f58a4066e70 0x7f58a4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:16.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.514+0000 7f58baa00640 1 -- 172.21.15.47:0/53467625 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58b4083ee0 con 0x7f58c4071400 2024-02-19T15:29:16.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.514+0000 7f58c2000640 1 --2- 172.21.15.47:0/53467625 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f58a4066e70 0x7f58a4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:16.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.514+0000 7f58c2000640 1 --2- 172.21.15.47:0/53467625 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f58a4066e70 0x7f58a4069330 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f58b001c4b0 tx=0x7f58b00a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:16.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.518+0000 7f58baa00640 1 -- 172.21.15.47:0/53467625 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f58c4153f40 con 0x7f58c4071400 2024-02-19T15:29:16.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.714+0000 7f58c3e00640 1 -- 172.21.15.47:0/53467625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58c4153f40 con 0x7f58c4071400 2024-02-19T15:29:16.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.714+0000 7f58baa00640 1 -- 172.21.15.47:0/53467625 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3376 (secure 0 0 0) 0x7f58c4153f40 con 0x7f58c4071400 2024-02-19T15:29:16.714 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:16.714 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":96,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:03.058284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:16.715 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 96 2024-02-19T15:29:16.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.718+0000 7f58c3e00640 1 -- 172.21.15.47:0/53467625 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f58a4066e70 msgr2=0x7f58a4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:16.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.718+0000 7f58c3e00640 1 --2- 172.21.15.47:0/53467625 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f58a4066e70 0x7f58a4069330 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f58b001c4b0 tx=0x7f58b00a7000 comp rx=0 tx=0).stop 2024-02-19T15:29:16.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.718+0000 7f58c3e00640 1 -- 172.21.15.47:0/53467625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c4071400 msgr2=0x7f58c41566d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:16.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.718+0000 7f58c3e00640 1 --2- 172.21.15.47:0/53467625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c4071400 0x7f58c41566d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f58b400c3c0 tx=0x7f58b400c880 comp rx=0 tx=0).stop 2024-02-19T15:29:16.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.718+0000 7f58c3e00640 1 -- 172.21.15.47:0/53467625 shutdown_connections 2024-02-19T15:29:16.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.718+0000 7f58c3e00640 1 --2- 172.21.15.47:0/53467625 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f58a4066e70 0x7f58a4069330 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:16.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.718+0000 7f58c3e00640 1 --2- 172.21.15.47:0/53467625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58c415ba70 0x7f58c4150080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:16.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.718+0000 7f58c3e00640 1 --2- 172.21.15.47:0/53467625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58c40717e0 0x7f58c4154600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:16.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.718+0000 7f58c3e00640 1 --2- 172.21.15.47:0/53467625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c4071400 0x7f58c41566d0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:16.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.718+0000 7f58c3e00640 1 -- 172.21.15.47:0/53467625 >> 172.21.15.47:0/53467625 conn(0x7f58c40cf7c0 msgr2=0x7f58c4151dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:16.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.718+0000 7f58c3e00640 1 -- 172.21.15.47:0/53467625 shutdown_connections 2024-02-19T15:29:16.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:16.718+0000 7f58c3e00640 1 -- 172.21.15.47:0/53467625 wait complete. 2024-02-19T15:29:16.747 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-02-19T15:29:16.900 INFO:teuthology.orchestra.run.smithi116.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-02-19T15:29:16.900 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-02-19T15:29:16.900 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-02-19T15:29:18.901 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:19.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.042+0000 7fdbd7200640 1 Processor -- start 2024-02-19T15:29:19.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.042+0000 7fdbd7200640 1 -- start start 2024-02-19T15:29:19.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.046+0000 7fdbd7200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdbd00cded0 0x7fdbd00ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:19.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.046+0000 7fdbd7200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd00ceb40 0x7fdbd006bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:19.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.046+0000 7fdbd7200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd00cce30 con 0x7fdbd006d650 2024-02-19T15:29:19.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.046+0000 7fdbd7200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbd00cefd0 con 0x7fdbd00cded0 2024-02-19T15:29:19.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.046+0000 7fdbd7200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd00cd110 con 0x7fdbd00ceb40 2024-02-19T15:29:19.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.046+0000 7fdbcf400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdbd00cded0 0x7fdbd00ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:19.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.046+0000 7fdbcf400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fdbd00cded0 0x7fdbd00ce600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39642/0 (socket says 172.21.15.47:39642) 2024-02-19T15:29:19.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.046+0000 7fdbd5200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd00ceb40 0x7fdbd006bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:19.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.046+0000 7fdbcfe00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdbd006d650 0x7fdbd00d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:49900/0 (socket says 172.21.15.47:49900) 2024-02-19T15:29:19.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.046+0000 7fdbcf400640 1 -- 172.21.15.47:0/3319331143 learned_addr learned my addr 172.21.15.47:0/3319331143 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:19.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.046+0000 7fdbcf400640 1 -- 172.21.15.47:0/3319331143 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd00ceb40 msgr2=0x7fdbd006bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:19.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.046+0000 7fdbcf400640 1 --2- 172.21.15.47:0/3319331143 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd00ceb40 0x7fdbd006bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:19.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.046+0000 7fdbcf400640 1 -- 172.21.15.47:0/3319331143 >> v1:172.21.15.116:6789/0 conn(0x7fdbd006d650 legacy=0x7fdbd00d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:19.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.046+0000 7fdbcf400640 1 -- 172.21.15.47:0/3319331143 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd00cd3f0 con 0x7fdbd00cded0 2024-02-19T15:29:19.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.046+0000 7fdbcea00640 1 -- 172.21.15.47:0/3319331143 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012476230 0 0) 0x7fdbd00cce30 con 0x7fdbd006d650 2024-02-19T15:29:19.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.050+0000 7fdbcf400640 1 --2- 172.21.15.47:0/3319331143 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd00cded0 0x7fdbd00ce600 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc009870 tx=0x7fdbbc01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c946205110d2295c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:19.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.050+0000 7fdbcea00640 1 -- 172.21.15.47:0/3319331143 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc0a6040 con 0x7fdbd00cded0 2024-02-19T15:29:19.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.050+0000 7fdbcea00640 1 -- 172.21.15.47:0/3319331143 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbbc0aa600 con 0x7fdbd00cded0 2024-02-19T15:29:19.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.050+0000 7fdbcea00640 1 -- 172.21.15.47:0/3319331143 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc0aa990 con 0x7fdbd00cded0 2024-02-19T15:29:19.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.050+0000 7fdbd7200640 1 -- 172.21.15.47:0/3319331143 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd00cded0 msgr2=0x7fdbd00ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:19.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.050+0000 7fdbd7200640 1 --2- 172.21.15.47:0/3319331143 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd00cded0 0x7fdbd00ce600 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc009870 tx=0x7fdbbc01c4a0 comp rx=0 tx=0).stop 2024-02-19T15:29:19.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.050+0000 7fdbd7200640 1 -- 172.21.15.47:0/3319331143 shutdown_connections 2024-02-19T15:29:19.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.050+0000 7fdbd7200640 1 --2- 172.21.15.47:0/3319331143 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd00ceb40 0x7fdbd006bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:19.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.050+0000 7fdbd7200640 1 --2- 172.21.15.47:0/3319331143 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd00cded0 0x7fdbd00ce600 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:19.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.050+0000 7fdbd7200640 1 -- 172.21.15.47:0/3319331143 >> 172.21.15.47:0/3319331143 conn(0x7fdbd00d1bf0 msgr2=0x7fdbd00d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:19.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.050+0000 7fdbd7200640 1 -- 172.21.15.47:0/3319331143 shutdown_connections 2024-02-19T15:29:19.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.050+0000 7fdbd7200640 1 -- 172.21.15.47:0/3319331143 wait complete. 2024-02-19T15:29:19.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.050+0000 7fdbd7200640 1 Processor -- start 2024-02-19T15:29:19.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.050+0000 7fdbd7200640 1 -- start start 2024-02-19T15:29:19.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbd7200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd006d650 0x7fdbd009efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:19.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbd7200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd00cded0 0x7fdbd00b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:19.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbd7200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd00ceb40 0x7fdbd00b62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:19.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbd7200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd0148eb0 con 0x7fdbd00cded0 2024-02-19T15:29:19.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbd7200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd0149190 con 0x7fdbd00ceb40 2024-02-19T15:29:19.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbd7200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd0149470 con 0x7fdbd006d650 2024-02-19T15:29:19.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbcf400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd00cded0 0x7fdbd00b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:19.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbcf400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd00cded0 0x7fdbd00b5da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39658/0 (socket says 172.21.15.47:39658) 2024-02-19T15:29:19.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbcf400640 1 -- 172.21.15.47:0/1098882297 learned_addr learned my addr 172.21.15.47:0/1098882297 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:19.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbcfe00640 1 --2- 172.21.15.47:0/1098882297 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd006d650 0x7fdbd009efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:19.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbd5200640 1 --2- 172.21.15.47:0/1098882297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd00ceb40 0x7fdbd00b62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:19.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbcf400640 1 -- 172.21.15.47:0/1098882297 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd006d650 msgr2=0x7fdbd009efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:19.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbcf400640 1 --2- 172.21.15.47:0/1098882297 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd006d650 0x7fdbd009efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:19.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbcf400640 1 -- 172.21.15.47:0/1098882297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd00ceb40 msgr2=0x7fdbd00b62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:19.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbcf400640 1 --2- 172.21.15.47:0/1098882297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd00ceb40 0x7fdbd00b62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:19.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbcf400640 1 -- 172.21.15.47:0/1098882297 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbbc0094d0 con 0x7fdbd00cded0 2024-02-19T15:29:19.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbcf400640 1 --2- 172.21.15.47:0/1098882297 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd00cded0 0x7fdbd00b5da0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc01ab50 tx=0x7fdbbc01ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:19.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbc7e00640 1 -- 172.21.15.47:0/1098882297 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc0b3600 con 0x7fdbd00cded0 2024-02-19T15:29:19.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbc7e00640 1 -- 172.21.15.47:0/1098882297 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbbc0b3780 con 0x7fdbd00cded0 2024-02-19T15:29:19.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbc7e00640 1 -- 172.21.15.47:0/1098882297 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc0b3b30 con 0x7fdbd00cded0 2024-02-19T15:29:19.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbd7200640 1 -- 172.21.15.47:0/1098882297 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd0149750 con 0x7fdbd00cded0 2024-02-19T15:29:19.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbd7200640 1 -- 172.21.15.47:0/1098882297 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd0148ad0 con 0x7fdbd00cded0 2024-02-19T15:29:19.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbd7200640 1 -- 172.21.15.47:0/1098882297 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbd0075360 con 0x7fdbd00cded0 2024-02-19T15:29:19.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.054+0000 7fdbc7e00640 1 -- 172.21.15.47:0/1098882297 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fdbbc0b3c90 con 0x7fdbd00cded0 2024-02-19T15:29:19.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.058+0000 7fdbc7e00640 1 --2- 172.21.15.47:0/1098882297 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbb0066e40 0x7fdbb0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:19.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.058+0000 7fdbc7e00640 1 -- 172.21.15.47:0/1098882297 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdbbc11eb90 con 0x7fdbd00cded0 2024-02-19T15:29:19.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.058+0000 7fdbcfe00640 1 --2- 172.21.15.47:0/1098882297 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbb0066e40 0x7fdbb0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:19.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.058+0000 7fdbcfe00640 1 --2- 172.21.15.47:0/1098882297 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbb0066e40 0x7fdbb0069300 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0000f70 tx=0x7fdbc0007df0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:19.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.062+0000 7fdbc7e00640 1 -- 172.21.15.47:0/1098882297 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fdbbc0eb990 con 0x7fdbd00cded0 2024-02-19T15:29:19.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.270+0000 7fdbd7200640 1 -- 172.21.15.47:0/1098882297 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbd0149c00 con 0x7fdbd00cded0 2024-02-19T15:29:19.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.270+0000 7fdbc7e00640 1 -- 172.21.15.47:0/1098882297 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3376 (secure 0 0 0) 0x7fdbbc0ef050 con 0x7fdbd00cded0 2024-02-19T15:29:19.270 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:19.270 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":96,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:03.058284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:19.270 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 96 2024-02-19T15:29:19.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.270+0000 7fdbd7200640 1 -- 172.21.15.47:0/1098882297 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbb0066e40 msgr2=0x7fdbb0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:19.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.270+0000 7fdbd7200640 1 --2- 172.21.15.47:0/1098882297 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbb0066e40 0x7fdbb0069300 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0000f70 tx=0x7fdbc0007df0 comp rx=0 tx=0).stop 2024-02-19T15:29:19.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.270+0000 7fdbd7200640 1 -- 172.21.15.47:0/1098882297 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd00cded0 msgr2=0x7fdbd00b5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:19.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.270+0000 7fdbd7200640 1 --2- 172.21.15.47:0/1098882297 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd00cded0 0x7fdbd00b5da0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc01ab50 tx=0x7fdbbc01ace0 comp rx=0 tx=0).stop 2024-02-19T15:29:19.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.270+0000 7fdbd7200640 1 -- 172.21.15.47:0/1098882297 shutdown_connections 2024-02-19T15:29:19.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.270+0000 7fdbd7200640 1 --2- 172.21.15.47:0/1098882297 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdbb0066e40 0x7fdbb0069300 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:19.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.270+0000 7fdbd7200640 1 --2- 172.21.15.47:0/1098882297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbd00ceb40 0x7fdbd00b62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:19.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.270+0000 7fdbd7200640 1 --2- 172.21.15.47:0/1098882297 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdbd00cded0 0x7fdbd00b5da0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:19.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.270+0000 7fdbd7200640 1 --2- 172.21.15.47:0/1098882297 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdbd006d650 0x7fdbd009efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:19.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.270+0000 7fdbd7200640 1 -- 172.21.15.47:0/1098882297 >> 172.21.15.47:0/1098882297 conn(0x7fdbd00d1bf0 msgr2=0x7fdbd00d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:19.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.270+0000 7fdbd7200640 1 -- 172.21.15.47:0/1098882297 shutdown_connections 2024-02-19T15:29:19.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:19.270+0000 7fdbd7200640 1 -- 172.21.15.47:0/1098882297 wait complete. 2024-02-19T15:29:19.293 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-02-19T15:29:19.446 INFO:teuthology.orchestra.run.smithi116.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-02-19T15:29:19.447 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-02-19T15:29:19.447 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-02-19T15:29:21.447 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:21.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.586+0000 7f13bb200640 1 Processor -- start 2024-02-19T15:29:21.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.586+0000 7f13bb200640 1 -- start start 2024-02-19T15:29:21.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.590+0000 7f13bb200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b415e180 0x7f13b40b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:21.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.590+0000 7f13bb200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f13b406c1d0 0x7f13b406c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:21.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.590+0000 7f13bb200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b40b2270 con 0x7f13b406cdf0 2024-02-19T15:29:21.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.590+0000 7f13bb200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13b40af880 con 0x7f13b406c1d0 2024-02-19T15:29:21.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.590+0000 7f13bb200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b406d280 con 0x7f13b415e180 2024-02-19T15:29:21.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.590+0000 7f13abe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f13b406c1d0 0x7f13b406c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:21.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.590+0000 7f13b3e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b415e180 0x7f13b40b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:21.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.590+0000 7f13abe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f13b406c1d0 0x7f13b406c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39672/0 (socket says 172.21.15.47:39672) 2024-02-19T15:29:21.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.590+0000 7f13abe00640 1 -- 172.21.15.47:0/79240158 learned_addr learned my addr 172.21.15.47:0/79240158 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:21.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.590+0000 7f13b3e00640 1 -- 172.21.15.47:0/79240158 >> v2:172.21.15.47:3300/0 conn(0x7f13b406c1d0 msgr2=0x7f13b406c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:21.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.590+0000 7f13b3e00640 1 --2- 172.21.15.47:0/79240158 >> v2:172.21.15.47:3300/0 conn(0x7f13b406c1d0 0x7f13b406c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:21.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.590+0000 7f13b3e00640 1 -- 172.21.15.47:0/79240158 >> v1:172.21.15.116:6789/0 conn(0x7f13b406cdf0 legacy=0x7f13b40cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:21.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.590+0000 7f13b3e00640 1 -- 172.21.15.47:0/79240158 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b40b2550 con 0x7f13b415e180 2024-02-19T15:29:21.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.590+0000 7f13abe00640 1 --2- 172.21.15.47:0/79240158 >> v2:172.21.15.47:3300/0 conn(0x7f13b406c1d0 0x7f13b406c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:29:21.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.594+0000 7f13b3e00640 1 --2- 172.21.15.47:0/79240158 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b415e180 0x7f13b40b1c20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f13a40097c0 tx=0x7f13a401c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b99e4a71a7dca68b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:21.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.594+0000 7f13b3400640 1 -- 172.21.15.47:0/79240158 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a40ab670 con 0x7f13b415e180 2024-02-19T15:29:21.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.594+0000 7f13b3400640 1 -- 172.21.15.47:0/79240158 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a40ab7d0 con 0x7f13b415e180 2024-02-19T15:29:21.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.594+0000 7f13b3400640 1 -- 172.21.15.47:0/79240158 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a40abb60 con 0x7f13b415e180 2024-02-19T15:29:21.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.594+0000 7f13bb200640 1 -- 172.21.15.47:0/79240158 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b415e180 msgr2=0x7f13b40b1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:21.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.594+0000 7f13bb200640 1 --2- 172.21.15.47:0/79240158 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b415e180 0x7f13b40b1c20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f13a40097c0 tx=0x7f13a401c470 comp rx=0 tx=0).stop 2024-02-19T15:29:21.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.594+0000 7f13bb200640 1 -- 172.21.15.47:0/79240158 shutdown_connections 2024-02-19T15:29:21.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.594+0000 7f13bb200640 1 --2- 172.21.15.47:0/79240158 >> v2:172.21.15.47:3300/0 conn(0x7f13b406c1d0 0x7f13b406c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:21.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.594+0000 7f13bb200640 1 --2- 172.21.15.47:0/79240158 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b415e180 0x7f13b40b1c20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:21.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.594+0000 7f13bb200640 1 -- 172.21.15.47:0/79240158 >> 172.21.15.47:0/79240158 conn(0x7f13b40d1ba0 msgr2=0x7f13b40d1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:21.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.594+0000 7f13bb200640 1 -- 172.21.15.47:0/79240158 shutdown_connections 2024-02-19T15:29:21.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.594+0000 7f13bb200640 1 -- 172.21.15.47:0/79240158 wait complete. 2024-02-19T15:29:21.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.594+0000 7f13bb200640 1 Processor -- start 2024-02-19T15:29:21.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.594+0000 7f13bb200640 1 -- start start 2024-02-19T15:29:21.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13bb200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b406c1d0 0x7f13b4156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:21.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13bb200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b406cdf0 0x7f13b40b6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:21.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13bb200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f13b415e180 0x7f13b40c64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:21.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13bb200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b40b7850 con 0x7f13b415e180 2024-02-19T15:29:21.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13bb200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b4156860 con 0x7f13b406cdf0 2024-02-19T15:29:21.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13bb200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b40b6c40 con 0x7f13b406c1d0 2024-02-19T15:29:21.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13b3e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b406c1d0 0x7f13b4156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:21.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13abe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b406cdf0 0x7f13b40b6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:21.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13b9200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f13b415e180 0x7f13b40c64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:21.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13abe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b406cdf0 0x7f13b40b6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:51880/0 (socket says 172.21.15.47:51880) 2024-02-19T15:29:21.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13b3e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b406c1d0 0x7f13b4156320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:53240/0 (socket says 172.21.15.47:53240) 2024-02-19T15:29:21.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13abe00640 1 -- 172.21.15.47:0/3447064941 learned_addr learned my addr 172.21.15.47:0/3447064941 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:21.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13abe00640 1 -- 172.21.15.47:0/3447064941 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b406c1d0 msgr2=0x7f13b4156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:21.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13abe00640 1 --2- 172.21.15.47:0/3447064941 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b406c1d0 0x7f13b4156320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:21.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13abe00640 1 -- 172.21.15.47:0/3447064941 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f13b415e180 msgr2=0x7f13b40c64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:21.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13abe00640 1 --2- 172.21.15.47:0/3447064941 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f13b415e180 0x7f13b40c64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:21.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13abe00640 1 -- 172.21.15.47:0/3447064941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139c0094d0 con 0x7f13b406cdf0 2024-02-19T15:29:21.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13abe00640 1 --2- 172.21.15.47:0/3447064941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b406cdf0 0x7f13b40b6410 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f139c00e930 tx=0x7f139c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:21.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13b0c00640 1 -- 172.21.15.47:0/3447064941 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c00ca00 con 0x7f13b406cdf0 2024-02-19T15:29:21.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13b0c00640 1 -- 172.21.15.47:0/3447064941 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f139c00cb60 con 0x7f13b406cdf0 2024-02-19T15:29:21.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13b0c00640 1 -- 172.21.15.47:0/3447064941 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c013470 con 0x7f13b406cdf0 2024-02-19T15:29:21.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13bb200640 1 -- 172.21.15.47:0/3447064941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a4009420 con 0x7f13b406cdf0 2024-02-19T15:29:21.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13bb200640 1 -- 172.21.15.47:0/3447064941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b40b7200 con 0x7f13b406cdf0 2024-02-19T15:29:21.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.598+0000 7f13bb200640 1 -- 172.21.15.47:0/3447064941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b4068500 con 0x7f13b406cdf0 2024-02-19T15:29:21.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.602+0000 7f13b0c00640 1 -- 172.21.15.47:0/3447064941 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f139c0135d0 con 0x7f13b406cdf0 2024-02-19T15:29:21.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.602+0000 7f13b0c00640 1 --2- 172.21.15.47:0/3447064941 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1390066e40 0x7f1390069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:21.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.602+0000 7f13b0c00640 1 -- 172.21.15.47:0/3447064941 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f139c084bd0 con 0x7f13b406cdf0 2024-02-19T15:29:21.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.602+0000 7f13b3e00640 1 --2- 172.21.15.47:0/3447064941 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1390066e40 0x7f1390069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:21.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.602+0000 7f13b3e00640 1 --2- 172.21.15.47:0/3447064941 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1390066e40 0x7f1390069300 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f13a40af000 tx=0x7f13a40b6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:21.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.606+0000 7f13b0c00640 1 -- 172.21.15.47:0/3447064941 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f139c051a60 con 0x7f13b406cdf0 2024-02-19T15:29:21.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.806+0000 7f13bb200640 1 -- 172.21.15.47:0/3447064941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13b4075360 con 0x7f13b406cdf0 2024-02-19T15:29:21.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.806+0000 7f13b0c00640 1 -- 172.21.15.47:0/3447064941 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3377 (secure 0 0 0) 0x7f139c017d50 con 0x7f13b406cdf0 2024-02-19T15:29:21.805 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:21.805 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:20.165277+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:21.805 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 97 2024-02-19T15:29:21.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.806+0000 7f13bb200640 1 -- 172.21.15.47:0/3447064941 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1390066e40 msgr2=0x7f1390069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:21.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.806+0000 7f13bb200640 1 --2- 172.21.15.47:0/3447064941 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1390066e40 0x7f1390069300 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f13a40af000 tx=0x7f13a40b6040 comp rx=0 tx=0).stop 2024-02-19T15:29:21.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.806+0000 7f13bb200640 1 -- 172.21.15.47:0/3447064941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b406cdf0 msgr2=0x7f13b40b6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:21.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.806+0000 7f13bb200640 1 --2- 172.21.15.47:0/3447064941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b406cdf0 0x7f13b40b6410 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f139c00e930 tx=0x7f139c00edf0 comp rx=0 tx=0).stop 2024-02-19T15:29:21.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.806+0000 7f13bb200640 1 -- 172.21.15.47:0/3447064941 shutdown_connections 2024-02-19T15:29:21.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.806+0000 7f13bb200640 1 --2- 172.21.15.47:0/3447064941 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1390066e40 0x7f1390069300 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:21.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.806+0000 7f13bb200640 1 --2- 172.21.15.47:0/3447064941 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f13b415e180 0x7f13b40c64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:21.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.806+0000 7f13bb200640 1 --2- 172.21.15.47:0/3447064941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b406cdf0 0x7f13b40b6410 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:21.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.806+0000 7f13bb200640 1 --2- 172.21.15.47:0/3447064941 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b406c1d0 0x7f13b4156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:21.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.806+0000 7f13bb200640 1 -- 172.21.15.47:0/3447064941 >> 172.21.15.47:0/3447064941 conn(0x7f13b40d1ba0 msgr2=0x7f13b406ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:21.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.806+0000 7f13bb200640 1 -- 172.21.15.47:0/3447064941 shutdown_connections 2024-02-19T15:29:21.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:21.806+0000 7f13bb200640 1 -- 172.21.15.47:0/3447064941 wait complete. 2024-02-19T15:29:21.827 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-02-19T15:29:21.983 INFO:teuthology.orchestra.run.smithi116.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"8","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/pin","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"3","version":"3","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-02-19T15:29:22.038 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "8" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/pin", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "3", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-02-19T15:29:22.038 INFO:tasks.cephfs.cephfs_test_case:[('/pin', 1)] =?= [('/pin', 1)] 2024-02-19T15:29:22.038 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:29:22.039 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:29:22.048 DEBUG:teuthology.orchestra.run.smithi047:> stdin-killer -- python3 -c ' 2024-02-19T15:29:22.048 DEBUG:teuthology.orchestra.run.smithi047:> import fcntl 2024-02-19T15:29:22.048 DEBUG:teuthology.orchestra.run.smithi047:> import sys 2024-02-19T15:29:22.049 DEBUG:teuthology.orchestra.run.smithi047:> import time 2024-02-19T15:29:22.049 DEBUG:teuthology.orchestra.run.smithi047:> import os 2024-02-19T15:29:22.049 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:29:22.049 DEBUG:teuthology.orchestra.run.smithi047:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-02-19T15:29:22.049 DEBUG:teuthology.orchestra.run.smithi047:> 2024-02-19T15:29:22.049 DEBUG:teuthology.orchestra.run.smithi047:> os.mkdir("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked") 2024-02-19T15:29:22.049 DEBUG:teuthology.orchestra.run.smithi047:> fd = os.open("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked", os.O_RDONLY) 2024-02-19T15:29:22.049 DEBUG:teuthology.orchestra.run.smithi047:> while True: 2024-02-19T15:29:22.049 DEBUG:teuthology.orchestra.run.smithi047:> print("open_dir_background: keeping dir open", file=sys.stderr) 2024-02-19T15:29:22.049 DEBUG:teuthology.orchestra.run.smithi047:> try: 2024-02-19T15:29:22.049 DEBUG:teuthology.orchestra.run.smithi047:> if os.read(0, 4096) == b"": 2024-02-19T15:29:22.049 DEBUG:teuthology.orchestra.run.smithi047:> break 2024-02-19T15:29:22.049 DEBUG:teuthology.orchestra.run.smithi047:> except BlockingIOError: 2024-02-19T15:29:22.049 DEBUG:teuthology.orchestra.run.smithi047:> pass 2024-02-19T15:29:22.049 DEBUG:teuthology.orchestra.run.smithi047:> time.sleep(2) 2024-02-19T15:29:22.049 DEBUG:teuthology.orchestra.run.smithi047:> ' 2024-02-19T15:29:22.094 DEBUG:teuthology.orchestra.run.smithi047:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-02-19T15:29:22.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:29:22.104 INFO:teuthology.orchestra.run.smithi047.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked': No such file or directory 2024-02-19T15:29:22.161 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:29:23.105 DEBUG:teuthology.orchestra.run.smithi047:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-02-19T15:29:23.114 INFO:teuthology.orchestra.run.smithi047.stdout: File: /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-02-19T15:29:23.115 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-02-19T15:29:23.115 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 2bh/43d Inode: 2199023255552 Links: 2 2024-02-19T15:29:23.115 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0775/drwxrwxr-x) Uid: ( 1000/ ubuntu) Gid: ( 1271/ ubuntu) 2024-02-19T15:29:23.115 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-02-19 15:29:22.158369235 +0000 2024-02-19T15:29:23.115 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-02-19 15:29:22.158369235 +0000 2024-02-19T15:29:23.115 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-02-19 15:29:22.158369235 +0000 2024-02-19T15:29:23.115 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-02-19 15:29:22.158369235 +0000 2024-02-19T15:29:23.115 INFO:tasks.cephfs.mount:File pin/to-be-unlinked became visible from 0 after 1s 2024-02-19T15:29:23.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:29:23.115 DEBUG:teuthology.orchestra.run.smithi047:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir pin/to-be-unlinked) 2024-02-19T15:29:23.165 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache num_strays 2024-02-19T15:29:23.353 INFO:teuthology.orchestra.run.smithi116.stdout:{"mds_cache":{"num_strays":1}} 2024-02-19T15:29:23.353 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-02-19T15:29:23.354 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_created 2024-02-19T15:29:23.512 INFO:teuthology.orchestra.run.smithi116.stdout:{"mds_cache":{"strays_created":1}} 2024-02-19T15:29:23.512 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-02-19T15:29:23.513 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_enqueued 2024-02-19T15:29:23.667 INFO:teuthology.orchestra.run.smithi116.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-02-19T15:29:23.667 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-02-19T15:29:23.667 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2024-02-19T15:29:23.833 INFO:teuthology.orchestra.run.smithi116.stdout:{"message":"","return_code":0} 2024-02-19T15:29:23.833 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-02-19T15:29:23.833 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json cache drop 2024-02-19T15:29:23.990 INFO:teuthology.orchestra.run.smithi116.stdout:{"client_recall":{"return_code":0,"message":"(0) Success","recalled":0},"flush_journal":{"return_code":0,"message":""},"trim_cache":{"trimmed":0},"cache":{"pool":{"items":321,"bytes":64735}},"duration":0.0062503460000000004} 2024-02-19T15:29:23.990 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cache": { "pool": { "bytes": 64735, "items": 321 } }, "client_recall": { "message": "(0) Success", "recalled": 0, "return_code": 0 }, "duration": 0.006250346, "flush_journal": { "message": "", "return_code": 0 }, "trim_cache": { "trimmed": 0 } } 2024-02-19T15:29:23.990 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-02-19T15:29:24.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.138+0000 7fd753400640 1 Processor -- start 2024-02-19T15:29:24.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.138+0000 7fd753400640 1 -- start start 2024-02-19T15:29:24.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.138+0000 7fd753400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd74c0d2650 0x7fd74c06c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:24.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.138+0000 7fd753400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd74c15e280 0x7fd74c06c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:24.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.138+0000 7fd753400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd74c0cc620 con 0x7fd74c0ce3b0 2024-02-19T15:29:24.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.138+0000 7fd753400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd74c0d12e0 con 0x7fd74c0d2650 2024-02-19T15:29:24.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.138+0000 7fd753400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c0d1940 con 0x7fd74c15e280 2024-02-19T15:29:24.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.138+0000 7fd750a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd74c0ce3b0 0x7fd74c0ce790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42700/0 (socket says 172.21.15.47:42700) 2024-02-19T15:29:24.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.138+0000 7fd750a00640 1 -- 172.21.15.47:0/1886927590 learned_addr learned my addr 172.21.15.47:0/1886927590 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:29:24.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.138+0000 7fd751400640 1 --2- 172.21.15.47:0/1886927590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd74c15e280 0x7fd74c06c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:24.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.138+0000 7fd74be00640 1 --2- 172.21.15.47:0/1886927590 >> v2:172.21.15.116:3300/0 conn(0x7fd74c0d2650 0x7fd74c06c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:24.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.138+0000 7fd74b400640 1 -- 172.21.15.47:0/1886927590 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839580492 0 0) 0x7fd74c0cc620 con 0x7fd74c0ce3b0 2024-02-19T15:29:24.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.138+0000 7fd74b400640 1 -- 172.21.15.47:0/1886927590 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd734003530 con 0x7fd74c0ce3b0 2024-02-19T15:29:24.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.138+0000 7fd751400640 1 -- 172.21.15.47:0/1886927590 >> v2:172.21.15.116:3300/0 conn(0x7fd74c0d2650 msgr2=0x7fd74c06c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:24.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.138+0000 7fd751400640 1 --2- 172.21.15.47:0/1886927590 >> v2:172.21.15.116:3300/0 conn(0x7fd74c0d2650 0x7fd74c06c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:24.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.138+0000 7fd751400640 1 -- 172.21.15.47:0/1886927590 >> v1:172.21.15.47:6789/0 conn(0x7fd74c0ce3b0 legacy=0x7fd74c0ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:24.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.138+0000 7fd751400640 1 -- 172.21.15.47:0/1886927590 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74c0d1000 con 0x7fd74c15e280 2024-02-19T15:29:24.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.142+0000 7fd751400640 1 --2- 172.21.15.47:0/1886927590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd74c15e280 0x7fd74c06c710 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd744009d90 tx=0x7fd74401c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96863494afb5d804 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:24.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.142+0000 7fd74b400640 1 -- 172.21.15.47:0/1886927590 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7440a5820 con 0x7fd74c15e280 2024-02-19T15:29:24.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.142+0000 7fd74b400640 1 -- 172.21.15.47:0/1886927590 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7440a5980 con 0x7fd74c15e280 2024-02-19T15:29:24.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.142+0000 7fd74b400640 1 -- 172.21.15.47:0/1886927590 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7440a5d10 con 0x7fd74c15e280 2024-02-19T15:29:24.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.142+0000 7fd753400640 1 -- 172.21.15.47:0/1886927590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd74c15e280 msgr2=0x7fd74c06c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:24.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.142+0000 7fd753400640 1 --2- 172.21.15.47:0/1886927590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd74c15e280 0x7fd74c06c710 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd744009d90 tx=0x7fd74401c850 comp rx=0 tx=0).stop 2024-02-19T15:29:24.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.142+0000 7fd753400640 1 -- 172.21.15.47:0/1886927590 shutdown_connections 2024-02-19T15:29:24.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.142+0000 7fd753400640 1 --2- 172.21.15.47:0/1886927590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd74c15e280 0x7fd74c06c710 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:24.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.142+0000 7fd753400640 1 --2- 172.21.15.47:0/1886927590 >> v2:172.21.15.116:3300/0 conn(0x7fd74c0d2650 0x7fd74c06c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:24.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.142+0000 7fd753400640 1 -- 172.21.15.47:0/1886927590 >> 172.21.15.47:0/1886927590 conn(0x7fd74c0d1b30 msgr2=0x7fd74c0d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:24.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.142+0000 7fd753400640 1 -- 172.21.15.47:0/1886927590 shutdown_connections 2024-02-19T15:29:24.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.142+0000 7fd753400640 1 -- 172.21.15.47:0/1886927590 wait complete. 2024-02-19T15:29:24.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd753400640 1 Processor -- start 2024-02-19T15:29:24.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd753400640 1 -- start start 2024-02-19T15:29:24.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd753400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd74c0ce3b0 0x7fd74c0ca050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:24.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd753400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd74c0d2650 0x7fd74c160020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:24.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd753400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd74c15e280 0x7fd74c158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:24.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd753400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c0ca650 con 0x7fd74c0ce3b0 2024-02-19T15:29:24.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd753400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c0ca930 con 0x7fd74c0d2650 2024-02-19T15:29:24.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd753400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c0cac10 con 0x7fd74c15e280 2024-02-19T15:29:24.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd750a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd74c0ce3b0 0x7fd74c0ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:24.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd751400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd74c15e280 0x7fd74c158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:24.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd74be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd74c0d2650 0x7fd74c160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:24.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd751400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd74c15e280 0x7fd74c158d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:51586/0 (socket says 172.21.15.47:51586) 2024-02-19T15:29:24.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd751400640 1 -- 172.21.15.47:0/1670266342 learned_addr learned my addr 172.21.15.47:0/1670266342 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:24.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd750a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd74c0ce3b0 0x7fd74c0ca050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45940/0 (socket says 172.21.15.47:45940) 2024-02-19T15:29:24.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd751400640 1 -- 172.21.15.47:0/1670266342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd74c0d2650 msgr2=0x7fd74c160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:24.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd751400640 1 --2- 172.21.15.47:0/1670266342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd74c0d2650 0x7fd74c160020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:24.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd751400640 1 -- 172.21.15.47:0/1670266342 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd74c0ce3b0 msgr2=0x7fd74c0ca050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:24.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd751400640 1 --2- 172.21.15.47:0/1670266342 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd74c0ce3b0 0x7fd74c0ca050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:24.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd751400640 1 -- 172.21.15.47:0/1670266342 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd744009a70 con 0x7fd74c15e280 2024-02-19T15:29:24.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd751400640 1 --2- 172.21.15.47:0/1670266342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd74c15e280 0x7fd74c158d20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd74401af00 tx=0x7fd7440195f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:24.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd748c00640 1 -- 172.21.15.47:0/1670266342 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7440b9650 con 0x7fd74c15e280 2024-02-19T15:29:24.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd748c00640 1 -- 172.21.15.47:0/1670266342 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7440b97b0 con 0x7fd74c15e280 2024-02-19T15:29:24.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd748c00640 1 -- 172.21.15.47:0/1670266342 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7440b9b20 con 0x7fd74c15e280 2024-02-19T15:29:24.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd753400640 1 -- 172.21.15.47:0/1670266342 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd74c0caef0 con 0x7fd74c15e280 2024-02-19T15:29:24.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd753400640 1 -- 172.21.15.47:0/1670266342 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd74c160560 con 0x7fd74c15e280 2024-02-19T15:29:24.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.146+0000 7fd753400640 1 -- 172.21.15.47:0/1670266342 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd74c075360 con 0x7fd74c15e280 2024-02-19T15:29:24.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.150+0000 7fd748c00640 1 -- 172.21.15.47:0/1670266342 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd74c0cdcd0 con 0x7fd74c15e280 2024-02-19T15:29:24.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.150+0000 7fd748c00640 1 --2- 172.21.15.47:0/1670266342 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd72c066e40 0x7fd72c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:24.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.150+0000 7fd748c00640 1 -- 172.21.15.47:0/1670266342 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd74411f8d0 con 0x7fd74c15e280 2024-02-19T15:29:24.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.150+0000 7fd750a00640 1 --2- 172.21.15.47:0/1670266342 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd72c066e40 0x7fd72c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:24.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.150+0000 7fd750a00640 1 --2- 172.21.15.47:0/1670266342 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd72c066e40 0x7fd72c069300 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fd740007fd0 tx=0x7fd740009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:24.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.154+0000 7fd748c00640 1 -- 172.21.15.47:0/1670266342 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd7440ec6d0 con 0x7fd74c15e280 2024-02-19T15:29:24.163 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:29:24.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.350+0000 7fd753400640 1 -- 172.21.15.47:0/1670266342 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7fd74c0cf1e0 con 0x7fd74c15e280 2024-02-19T15:29:24.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.882+0000 7fd748c00640 1 -- 172.21.15.47:0/1670266342 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v98) v1 ==== 105+0+0 (secure 0 0 0) 0x7fd7440efd90 con 0x7fd74c15e280 2024-02-19T15:29:24.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.886+0000 7fd753400640 1 -- 172.21.15.47:0/1670266342 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd72c066e40 msgr2=0x7fd72c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:24.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.886+0000 7fd753400640 1 --2- 172.21.15.47:0/1670266342 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd72c066e40 0x7fd72c069300 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fd740007fd0 tx=0x7fd740009040 comp rx=0 tx=0).stop 2024-02-19T15:29:24.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.886+0000 7fd753400640 1 -- 172.21.15.47:0/1670266342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd74c15e280 msgr2=0x7fd74c158d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:24.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.886+0000 7fd753400640 1 --2- 172.21.15.47:0/1670266342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd74c15e280 0x7fd74c158d20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd74401af00 tx=0x7fd7440195f0 comp rx=0 tx=0).stop 2024-02-19T15:29:24.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.886+0000 7fd753400640 1 -- 172.21.15.47:0/1670266342 shutdown_connections 2024-02-19T15:29:24.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.886+0000 7fd753400640 1 --2- 172.21.15.47:0/1670266342 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd72c066e40 0x7fd72c069300 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:24.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.886+0000 7fd753400640 1 --2- 172.21.15.47:0/1670266342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd74c15e280 0x7fd74c158d20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:24.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.886+0000 7fd753400640 1 --2- 172.21.15.47:0/1670266342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd74c0d2650 0x7fd74c160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:24.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.886+0000 7fd753400640 1 --2- 172.21.15.47:0/1670266342 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd74c0ce3b0 0x7fd74c0ca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:24.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.886+0000 7fd753400640 1 -- 172.21.15.47:0/1670266342 >> 172.21.15.47:0/1670266342 conn(0x7fd74c0d1b30 msgr2=0x7fd74c06bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:24.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.886+0000 7fd753400640 1 -- 172.21.15.47:0/1670266342 shutdown_connections 2024-02-19T15:29:24.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:24.886+0000 7fd753400640 1 -- 172.21.15.47:0/1670266342 wait complete. 2024-02-19T15:29:24.905 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:25.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.046+0000 7f6d08e00640 1 Processor -- start 2024-02-19T15:29:25.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.046+0000 7f6d08e00640 1 -- start start 2024-02-19T15:29:25.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.046+0000 7f6d08e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6d04071350 0x7f6d04165e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:25.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.046+0000 7f6d08e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d040718c0 con 0x7f6d0415d6e0 2024-02-19T15:29:25.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.046+0000 7f6d08e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d04071ba0 con 0x7f6d0415b1a0 2024-02-19T15:29:25.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.046+0000 7f6d08e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d04071e80 con 0x7f6d04071350 2024-02-19T15:29:25.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.046+0000 7f6d02800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6d04071350 0x7f6d04165e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:25.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.046+0000 7f6d02800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6d04071350 0x7f6d04165e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45948/0 (socket says 172.21.15.47:45948) 2024-02-19T15:29:25.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.046+0000 7f6d02800640 1 -- 172.21.15.47:0/1646235716 learned_addr learned my addr 172.21.15.47:0/1646235716 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:25.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.046+0000 7f6cf9400640 1 --1- 172.21.15.47:0/1646235716 >> v1:172.21.15.47:6789/0 conn(0x7f6d0415d6e0 0x7f6d04162740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42708/0 (socket says 172.21.15.47:42708) 2024-02-19T15:29:25.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.046+0000 7f6d02800640 1 -- 172.21.15.47:0/1646235716 >> v1:172.21.15.116:6789/0 conn(0x7f6d0415b1a0 legacy=0x7f6d0415b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:25.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.046+0000 7f6d02800640 1 -- 172.21.15.47:0/1646235716 >> v1:172.21.15.47:6789/0 conn(0x7f6d0415d6e0 legacy=0x7f6d04162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:25.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.046+0000 7f6d02800640 1 -- 172.21.15.47:0/1646235716 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d04072160 con 0x7f6d04071350 2024-02-19T15:29:25.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.046+0000 7f6d01400640 1 -- 172.21.15.47:0/1646235716 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190574350 0 0) 0x7f6d040718c0 con 0x7f6d0415d6e0 2024-02-19T15:29:25.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.050+0000 7f6d02800640 1 --2- 172.21.15.47:0/1646235716 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d04071350 0x7f6d04165e70 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc0098a0 tx=0x7f6cfc01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6379dcb29d2ad161 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:25.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.050+0000 7f6d01400640 1 -- 172.21.15.47:0/1646235716 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc0a9620 con 0x7f6d04071350 2024-02-19T15:29:25.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.050+0000 7f6d01400640 1 -- 172.21.15.47:0/1646235716 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cfc0a9780 con 0x7f6d04071350 2024-02-19T15:29:25.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.050+0000 7f6d01400640 1 -- 172.21.15.47:0/1646235716 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc0a9b10 con 0x7f6d04071350 2024-02-19T15:29:25.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.050+0000 7f6d08e00640 1 -- 172.21.15.47:0/1646235716 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d04071350 msgr2=0x7f6d04165e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:25.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.050+0000 7f6d08e00640 1 --2- 172.21.15.47:0/1646235716 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d04071350 0x7f6d04165e70 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc0098a0 tx=0x7f6cfc01c540 comp rx=0 tx=0).stop 2024-02-19T15:29:25.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.050+0000 7f6d08e00640 1 -- 172.21.15.47:0/1646235716 shutdown_connections 2024-02-19T15:29:25.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.050+0000 7f6d08e00640 1 --2- 172.21.15.47:0/1646235716 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d04071350 0x7f6d04165e70 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:25.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.050+0000 7f6d08e00640 1 -- 172.21.15.47:0/1646235716 >> 172.21.15.47:0/1646235716 conn(0x7f6d040b16a0 msgr2=0x7f6d040b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:25.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.050+0000 7f6d08e00640 1 -- 172.21.15.47:0/1646235716 shutdown_connections 2024-02-19T15:29:25.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.050+0000 7f6d08e00640 1 -- 172.21.15.47:0/1646235716 wait complete. 2024-02-19T15:29:25.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.054+0000 7f6d08e00640 1 Processor -- start 2024-02-19T15:29:25.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.054+0000 7f6d08e00640 1 -- start start 2024-02-19T15:29:25.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.054+0000 7f6d08e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d04071350 0x7f6d041f9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:25.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.054+0000 7f6d08e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0415b1a0 0x7f6d041574e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:25.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.054+0000 7f6d08e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d0415d6e0 0x7f6d04157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:25.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.054+0000 7f6d08e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d04158340 con 0x7f6d04071350 2024-02-19T15:29:25.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.054+0000 7f6d08e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d040b31a0 con 0x7f6d0415d6e0 2024-02-19T15:29:25.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.054+0000 7f6d08e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d040b3480 con 0x7f6d0415b1a0 2024-02-19T15:29:25.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.054+0000 7f6d01e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d04071350 0x7f6d041f9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:25.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.054+0000 7f6d01e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d04071350 0x7f6d041f9740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45952/0 (socket says 172.21.15.47:45952) 2024-02-19T15:29:25.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.054+0000 7f6d01e00640 1 -- 172.21.15.47:0/1183383374 learned_addr learned my addr 172.21.15.47:0/1183383374 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:25.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.054+0000 7f6cf9400640 1 --2- 172.21.15.47:0/1183383374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0415b1a0 0x7f6d041574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:25.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.054+0000 7f6d02800640 1 --2- 172.21.15.47:0/1183383374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d0415d6e0 0x7f6d04157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:25.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.054+0000 7f6d01e00640 1 -- 172.21.15.47:0/1183383374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0415b1a0 msgr2=0x7f6d041574e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:25.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.054+0000 7f6d01e00640 1 --2- 172.21.15.47:0/1183383374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0415b1a0 0x7f6d041574e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:25.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.054+0000 7f6d01e00640 1 -- 172.21.15.47:0/1183383374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d0415d6e0 msgr2=0x7f6d04157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:25.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.054+0000 7f6d01e00640 1 --2- 172.21.15.47:0/1183383374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d0415d6e0 0x7f6d04157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:25.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.058+0000 7f6d01e00640 1 -- 172.21.15.47:0/1183383374 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cfc0094d0 con 0x7f6d04071350 2024-02-19T15:29:25.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.058+0000 7f6d01e00640 1 --2- 172.21.15.47:0/1183383374 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d04071350 0x7f6d041f9740 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400c500 tx=0x7f6cf400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:25.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.058+0000 7f6cfaa00640 1 -- 172.21.15.47:0/1183383374 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4005220 con 0x7f6d04071350 2024-02-19T15:29:25.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.058+0000 7f6cfaa00640 1 -- 172.21.15.47:0/1183383374 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf4005380 con 0x7f6d04071350 2024-02-19T15:29:25.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.058+0000 7f6d08e00640 1 -- 172.21.15.47:0/1183383374 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d040b3760 con 0x7f6d04071350 2024-02-19T15:29:25.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.058+0000 7f6cfaa00640 1 -- 172.21.15.47:0/1183383374 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4007270 con 0x7f6d04071350 2024-02-19T15:29:25.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.058+0000 7f6d08e00640 1 -- 172.21.15.47:0/1183383374 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d040b3d20 con 0x7f6d04071350 2024-02-19T15:29:25.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.058+0000 7f6d08e00640 1 -- 172.21.15.47:0/1183383374 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d04075360 con 0x7f6d04071350 2024-02-19T15:29:25.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.058+0000 7f6cfaa00640 1 -- 172.21.15.47:0/1183383374 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6cf40073d0 con 0x7f6d04071350 2024-02-19T15:29:25.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.062+0000 7f6cfaa00640 1 --2- 172.21.15.47:0/1183383374 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ce0066e40 0x7f6ce0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:25.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.062+0000 7f6cfaa00640 1 -- 172.21.15.47:0/1183383374 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6cf40844d0 con 0x7f6d04071350 2024-02-19T15:29:25.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.062+0000 7f6cf9400640 1 --2- 172.21.15.47:0/1183383374 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ce0066e40 0x7f6ce0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:25.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.062+0000 7f6cf9400640 1 --2- 172.21.15.47:0/1183383374 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ce0066e40 0x7f6ce0069300 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f6cec007fd0 tx=0x7f6cec007f20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:25.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.062+0000 7f6cfaa00640 1 -- 172.21.15.47:0/1183383374 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6cf4051230 con 0x7f6d04071350 2024-02-19T15:29:25.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.258+0000 7f6d08e00640 1 -- 172.21.15.47:0/1183383374 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d041f6a90 con 0x7f6d04071350 2024-02-19T15:29:25.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.258+0000 7f6cfaa00640 1 -- 172.21.15.47:0/1183383374 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3379 (secure 0 0 0) 0x7f6d041f6a90 con 0x7f6d04071350 2024-02-19T15:29:25.260 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:25.260 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:24.882579+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:25.261 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 98 2024-02-19T15:29:25.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.262+0000 7f6d08e00640 1 -- 172.21.15.47:0/1183383374 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ce0066e40 msgr2=0x7f6ce0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:25.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.262+0000 7f6d08e00640 1 --2- 172.21.15.47:0/1183383374 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ce0066e40 0x7f6ce0069300 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f6cec007fd0 tx=0x7f6cec007f20 comp rx=0 tx=0).stop 2024-02-19T15:29:25.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.262+0000 7f6d08e00640 1 -- 172.21.15.47:0/1183383374 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d04071350 msgr2=0x7f6d041f9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:25.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.262+0000 7f6d08e00640 1 --2- 172.21.15.47:0/1183383374 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d04071350 0x7f6d041f9740 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400c500 tx=0x7f6cf400c9c0 comp rx=0 tx=0).stop 2024-02-19T15:29:25.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.262+0000 7f6d08e00640 1 -- 172.21.15.47:0/1183383374 shutdown_connections 2024-02-19T15:29:25.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.262+0000 7f6d08e00640 1 --2- 172.21.15.47:0/1183383374 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ce0066e40 0x7f6ce0069300 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:25.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.262+0000 7f6d08e00640 1 --2- 172.21.15.47:0/1183383374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d0415d6e0 0x7f6d04157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:25.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.262+0000 7f6d08e00640 1 --2- 172.21.15.47:0/1183383374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0415b1a0 0x7f6d041574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:25.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.262+0000 7f6d08e00640 1 --2- 172.21.15.47:0/1183383374 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d04071350 0x7f6d041f9740 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:25.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.262+0000 7f6d08e00640 1 -- 172.21.15.47:0/1183383374 >> 172.21.15.47:0/1183383374 conn(0x7f6d040b16a0 msgr2=0x7f6d040b0ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:25.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.262+0000 7f6d08e00640 1 -- 172.21.15.47:0/1183383374 shutdown_connections 2024-02-19T15:29:25.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:25.262+0000 7f6d08e00640 1 -- 172.21.15.47:0/1183383374 wait complete. 2024-02-19T15:29:25.483 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:24.882579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:25.484 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:26.164 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:29:26.485 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:26.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.626+0000 7f667d200640 1 Processor -- start 2024-02-19T15:29:26.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.626+0000 7f667d200640 1 -- start start 2024-02-19T15:29:26.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.626+0000 7f667d200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f667815bb90 0x7f66780cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:26.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.626+0000 7f667d200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66780d2150 0x7f66780cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:26.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.626+0000 7f667d200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66780b3300 con 0x7f667815bf70 2024-02-19T15:29:26.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.626+0000 7f667d200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66780d0c60 con 0x7f667815bb90 2024-02-19T15:29:26.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.626+0000 7f667d200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66780d0980 con 0x7f66780d2150 2024-02-19T15:29:26.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.626+0000 7f6677400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f667815bb90 0x7f66780cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:26.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.626+0000 7f6677400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f667815bb90 0x7f66780cd4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45956/0 (socket says 172.21.15.47:45956) 2024-02-19T15:29:26.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.626+0000 7f6677400640 1 -- 172.21.15.47:0/242842740 learned_addr learned my addr 172.21.15.47:0/242842740 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:26.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.626+0000 7f6677e00640 1 --2- 172.21.15.47:0/242842740 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66780d2150 0x7f66780cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:26.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.630+0000 7f6677400640 1 -- 172.21.15.47:0/242842740 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66780d2150 msgr2=0x7f66780cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:26.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.630+0000 7f6677400640 1 --2- 172.21.15.47:0/242842740 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66780d2150 0x7f66780cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:26.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.630+0000 7f6677400640 1 -- 172.21.15.47:0/242842740 >> v1:172.21.15.116:6789/0 conn(0x7f667815bf70 legacy=0x7f66780d1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:26.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.630+0000 7f6677400640 1 -- 172.21.15.47:0/242842740 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66780cc9d0 con 0x7f667815bb90 2024-02-19T15:29:26.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.630+0000 7f6677400640 1 --2- 172.21.15.47:0/242842740 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f667815bb90 0x7f66780cd4b0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f6668009870 tx=0x7f666801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9608e641ec9b3f5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:26.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.630+0000 7f6676a00640 1 -- 172.21.15.47:0/242842740 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66680ab620 con 0x7f667815bb90 2024-02-19T15:29:26.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.630+0000 7f6676a00640 1 -- 172.21.15.47:0/242842740 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66680ab780 con 0x7f667815bb90 2024-02-19T15:29:26.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.630+0000 7f6676a00640 1 -- 172.21.15.47:0/242842740 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66680abb10 con 0x7f667815bb90 2024-02-19T15:29:26.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.630+0000 7f667d200640 1 -- 172.21.15.47:0/242842740 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f667815bb90 msgr2=0x7f66780cd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:26.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.630+0000 7f667d200640 1 --2- 172.21.15.47:0/242842740 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f667815bb90 0x7f66780cd4b0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f6668009870 tx=0x7f666801c460 comp rx=0 tx=0).stop 2024-02-19T15:29:26.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.630+0000 7f667d200640 1 -- 172.21.15.47:0/242842740 shutdown_connections 2024-02-19T15:29:26.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.630+0000 7f667d200640 1 --2- 172.21.15.47:0/242842740 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66780d2150 0x7f66780cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:26.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.630+0000 7f667d200640 1 --2- 172.21.15.47:0/242842740 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f667815bb90 0x7f66780cd4b0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:26.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.630+0000 7f667d200640 1 -- 172.21.15.47:0/242842740 >> 172.21.15.47:0/242842740 conn(0x7f667809a9d0 msgr2=0x7f667809ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:26.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.630+0000 7f667d200640 1 -- 172.21.15.47:0/242842740 shutdown_connections 2024-02-19T15:29:26.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.630+0000 7f667d200640 1 -- 172.21.15.47:0/242842740 wait complete. 2024-02-19T15:29:26.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.634+0000 7f667d200640 1 Processor -- start 2024-02-19T15:29:26.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.634+0000 7f667d200640 1 -- start start 2024-02-19T15:29:26.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.634+0000 7f667d200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66780d2150 0x7f6678148af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:26.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.634+0000 7f667d200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f667815bb90 0x7f6678159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:26.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.634+0000 7f667d200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f667815bf70 0x7f66780c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:26.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.634+0000 7f667d200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66780c7f00 con 0x7f66780d2150 2024-02-19T15:29:26.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.634+0000 7f667d200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66780c7050 con 0x7f667815bf70 2024-02-19T15:29:26.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.634+0000 7f667d200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66780c7280 con 0x7f667815bb90 2024-02-19T15:29:26.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.634+0000 7f6677400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66780d2150 0x7f6678148af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:26.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.634+0000 7f6677400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66780d2150 0x7f6678148af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45970/0 (socket says 172.21.15.47:45970) 2024-02-19T15:29:26.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.634+0000 7f6677400640 1 -- 172.21.15.47:0/2099284454 learned_addr learned my addr 172.21.15.47:0/2099284454 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:26.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.634+0000 7f666fe00640 1 --2- 172.21.15.47:0/2099284454 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f667815bb90 0x7f6678159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:26.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.634+0000 7f6677e00640 1 --2- 172.21.15.47:0/2099284454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f667815bf70 0x7f66780c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:26.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.634+0000 7f6677400640 1 -- 172.21.15.47:0/2099284454 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f667815bb90 msgr2=0x7f6678159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:26.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.634+0000 7f6677400640 1 --2- 172.21.15.47:0/2099284454 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f667815bb90 0x7f6678159d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:26.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.634+0000 7f6677400640 1 -- 172.21.15.47:0/2099284454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f667815bf70 msgr2=0x7f66780c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:26.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.634+0000 7f6677400640 1 --2- 172.21.15.47:0/2099284454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f667815bf70 0x7f66780c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:26.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.634+0000 7f6677400640 1 -- 172.21.15.47:0/2099284454 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66680094d0 con 0x7f66780d2150 2024-02-19T15:29:26.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.634+0000 7f6677e00640 1 --2- 172.21.15.47:0/2099284454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f667815bf70 0x7f66780c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:29:26.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.634+0000 7f6677400640 1 --2- 172.21.15.47:0/2099284454 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66780d2150 0x7f6678148af0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f666801cd00 tx=0x7f666801aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:26.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.638+0000 7f666f400640 1 -- 172.21.15.47:0/2099284454 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66680be3f0 con 0x7f66780d2150 2024-02-19T15:29:26.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.638+0000 7f666f400640 1 -- 172.21.15.47:0/2099284454 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66680be550 con 0x7f66780d2150 2024-02-19T15:29:26.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.638+0000 7f666f400640 1 -- 172.21.15.47:0/2099284454 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66680be900 con 0x7f66780d2150 2024-02-19T15:29:26.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.638+0000 7f667d200640 1 -- 172.21.15.47:0/2099284454 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66780c7560 con 0x7f66780d2150 2024-02-19T15:29:26.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.638+0000 7f667d200640 1 -- 172.21.15.47:0/2099284454 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66780c7b20 con 0x7f66780d2150 2024-02-19T15:29:26.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.638+0000 7f667d200640 1 -- 172.21.15.47:0/2099284454 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f667809b1a0 con 0x7f66780d2150 2024-02-19T15:29:26.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.638+0000 7f666f400640 1 -- 172.21.15.47:0/2099284454 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f66680b43f0 con 0x7f66780d2150 2024-02-19T15:29:26.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.638+0000 7f666f400640 1 --2- 172.21.15.47:0/2099284454 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6654066e70 0x7f6654069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:26.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.638+0000 7f666f400640 1 -- 172.21.15.47:0/2099284454 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f666811f270 con 0x7f66780d2150 2024-02-19T15:29:26.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.638+0000 7f666fe00640 1 --2- 172.21.15.47:0/2099284454 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6654066e70 0x7f6654069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:26.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.642+0000 7f666fe00640 1 --2- 172.21.15.47:0/2099284454 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6654066e70 0x7f6654069330 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f667815a960 tx=0x7f6660007e40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:26.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.642+0000 7f666f400640 1 -- 172.21.15.47:0/2099284454 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f66680ec070 con 0x7f66780d2150 2024-02-19T15:29:26.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.842+0000 7f667d200640 1 -- 172.21.15.47:0/2099284454 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6678075360 con 0x7f66780d2150 2024-02-19T15:29:26.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.842+0000 7f666f400640 1 -- 172.21.15.47:0/2099284454 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3379 (secure 0 0 0) 0x7f66680ef730 con 0x7f66780d2150 2024-02-19T15:29:26.841 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:26.842 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:24.882579+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:26.842 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 98 2024-02-19T15:29:26.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.842+0000 7f667d200640 1 -- 172.21.15.47:0/2099284454 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6654066e70 msgr2=0x7f6654069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:26.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.842+0000 7f667d200640 1 --2- 172.21.15.47:0/2099284454 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6654066e70 0x7f6654069330 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f667815a960 tx=0x7f6660007e40 comp rx=0 tx=0).stop 2024-02-19T15:29:26.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.842+0000 7f667d200640 1 -- 172.21.15.47:0/2099284454 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66780d2150 msgr2=0x7f6678148af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:26.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.842+0000 7f667d200640 1 --2- 172.21.15.47:0/2099284454 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66780d2150 0x7f6678148af0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f666801cd00 tx=0x7f666801aca0 comp rx=0 tx=0).stop 2024-02-19T15:29:26.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.842+0000 7f667d200640 1 -- 172.21.15.47:0/2099284454 shutdown_connections 2024-02-19T15:29:26.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.842+0000 7f667d200640 1 --2- 172.21.15.47:0/2099284454 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6654066e70 0x7f6654069330 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:26.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.842+0000 7f667d200640 1 --2- 172.21.15.47:0/2099284454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f667815bf70 0x7f66780c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:26.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.842+0000 7f667d200640 1 --2- 172.21.15.47:0/2099284454 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f667815bb90 0x7f6678159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:26.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.842+0000 7f667d200640 1 --2- 172.21.15.47:0/2099284454 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66780d2150 0x7f6678148af0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:26.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.842+0000 7f667d200640 1 -- 172.21.15.47:0/2099284454 >> 172.21.15.47:0/2099284454 conn(0x7f667809a9d0 msgr2=0x7f66780d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:26.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.842+0000 7f667d200640 1 -- 172.21.15.47:0/2099284454 shutdown_connections 2024-02-19T15:29:26.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:26.842+0000 7f667d200640 1 -- 172.21.15.47:0/2099284454 wait complete. 2024-02-19T15:29:27.065 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:24.882579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:27.065 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:28.066 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:28.166 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:29:28.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.206+0000 7f1b67200640 1 Processor -- start 2024-02-19T15:29:28.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.206+0000 7f1b67200640 1 -- start start 2024-02-19T15:29:28.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.210+0000 7f1b67200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b6815dbb0 0x7f1b680726a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:28.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.210+0000 7f1b67200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1b680b1b10 0x7f1b68070ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:28.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.210+0000 7f1b67200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b680caf40 con 0x7f1b680cb3d0 2024-02-19T15:29:28.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.210+0000 7f1b67200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b681625f0 con 0x7f1b680b1b10 2024-02-19T15:29:28.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.210+0000 7f1b67200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b680ca610 con 0x7f1b6815dbb0 2024-02-19T15:29:28.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.210+0000 7f1b65e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b6815dbb0 0x7f1b680726a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:28.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.210+0000 7f1b66800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1b680b1b10 0x7f1b68070ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:28.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.210+0000 7f1b5d400640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1b680cb3d0 0x7f1b680b1400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:48380/0 (socket says 172.21.15.47:48380) 2024-02-19T15:29:28.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.210+0000 7f1b5d400640 1 -- 172.21.15.47:0/1193754125 learned_addr learned my addr 172.21.15.47:0/1193754125 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:29:28.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.210+0000 7f1b65400640 1 -- 172.21.15.47:0/1193754125 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637650672 0 0) 0x7f1b680caf40 con 0x7f1b680cb3d0 2024-02-19T15:29:28.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.210+0000 7f1b65400640 1 -- 172.21.15.47:0/1193754125 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b4c003530 con 0x7f1b680cb3d0 2024-02-19T15:29:28.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.210+0000 7f1b66800640 1 -- 172.21.15.47:0/1193754125 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b6815dbb0 msgr2=0x7f1b680726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:28.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.210+0000 7f1b66800640 1 --2- 172.21.15.47:0/1193754125 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b6815dbb0 0x7f1b680726a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:28.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.210+0000 7f1b66800640 1 -- 172.21.15.47:0/1193754125 >> v1:172.21.15.116:6789/0 conn(0x7f1b680cb3d0 legacy=0x7f1b680b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:28.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.210+0000 7f1b66800640 1 -- 172.21.15.47:0/1193754125 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b680ca8f0 con 0x7f1b680b1b10 2024-02-19T15:29:28.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.210+0000 7f1b65e00640 1 --2- 172.21.15.47:0/1193754125 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b6815dbb0 0x7f1b680726a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:29:28.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.214+0000 7f1b66800640 1 --2- 172.21.15.47:0/1193754125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b680b1b10 0x7f1b68070ec0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f1b60009d90 tx=0x7f1b6001c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b396fd3a2d4e165 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:28.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.214+0000 7f1b65400640 1 -- 172.21.15.47:0/1193754125 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b600a5930 con 0x7f1b680b1b10 2024-02-19T15:29:28.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.214+0000 7f1b65400640 1 -- 172.21.15.47:0/1193754125 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b600a5a90 con 0x7f1b680b1b10 2024-02-19T15:29:28.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.214+0000 7f1b65400640 1 -- 172.21.15.47:0/1193754125 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b600a5e20 con 0x7f1b680b1b10 2024-02-19T15:29:28.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.214+0000 7f1b67200640 1 -- 172.21.15.47:0/1193754125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b680b1b10 msgr2=0x7f1b68070ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:28.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.214+0000 7f1b67200640 1 --2- 172.21.15.47:0/1193754125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b680b1b10 0x7f1b68070ec0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f1b60009d90 tx=0x7f1b6001c940 comp rx=0 tx=0).stop 2024-02-19T15:29:28.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.214+0000 7f1b67200640 1 -- 172.21.15.47:0/1193754125 shutdown_connections 2024-02-19T15:29:28.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.214+0000 7f1b67200640 1 --2- 172.21.15.47:0/1193754125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b680b1b10 0x7f1b68070ec0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:28.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.214+0000 7f1b67200640 1 --2- 172.21.15.47:0/1193754125 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b6815dbb0 0x7f1b680726a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:28.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.214+0000 7f1b67200640 1 -- 172.21.15.47:0/1193754125 >> 172.21.15.47:0/1193754125 conn(0x7f1b680cf7c0 msgr2=0x7f1b680cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:28.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.214+0000 7f1b67200640 1 -- 172.21.15.47:0/1193754125 shutdown_connections 2024-02-19T15:29:28.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.214+0000 7f1b67200640 1 -- 172.21.15.47:0/1193754125 wait complete. 2024-02-19T15:29:28.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.214+0000 7f1b67200640 1 Processor -- start 2024-02-19T15:29:28.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b67200640 1 -- start start 2024-02-19T15:29:28.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b67200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b680b1b10 0x7f1b6814c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:28.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b67200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b680cb3d0 0x7f1b6814ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:28.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b67200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b6815dbb0 0x7f1b681562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:28.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b67200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6814d840 con 0x7f1b680b1b10 2024-02-19T15:29:28.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b67200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6814db20 con 0x7f1b680cb3d0 2024-02-19T15:29:28.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b67200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6814de00 con 0x7f1b6815dbb0 2024-02-19T15:29:28.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b65e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b680b1b10 0x7f1b6814c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:28.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b65e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b680b1b10 0x7f1b6814c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45986/0 (socket says 172.21.15.47:45986) 2024-02-19T15:29:28.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b65e00640 1 -- 172.21.15.47:0/2094782264 learned_addr learned my addr 172.21.15.47:0/2094782264 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:28.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b5d400640 1 --2- 172.21.15.47:0/2094782264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b680cb3d0 0x7f1b6814ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:28.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b66800640 1 --2- 172.21.15.47:0/2094782264 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b6815dbb0 0x7f1b681562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b65e00640 1 -- 172.21.15.47:0/2094782264 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b6815dbb0 msgr2=0x7f1b681562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b65e00640 1 --2- 172.21.15.47:0/2094782264 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b6815dbb0 0x7f1b681562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b65e00640 1 -- 172.21.15.47:0/2094782264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b680cb3d0 msgr2=0x7f1b6814ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b65e00640 1 --2- 172.21.15.47:0/2094782264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b680cb3d0 0x7f1b6814ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b65e00640 1 -- 172.21.15.47:0/2094782264 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b58009420 con 0x7f1b680b1b10 2024-02-19T15:29:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b66800640 1 --2- 172.21.15.47:0/2094782264 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b6815dbb0 0x7f1b681562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:29:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b65e00640 1 --2- 172.21.15.47:0/2094782264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b680b1b10 0x7f1b6814c880 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800ebe0 tx=0x7f1b5800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b5ea00640 1 -- 172.21.15.47:0/2094782264 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5800eee0 con 0x7f1b680b1b10 2024-02-19T15:29:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b5ea00640 1 -- 172.21.15.47:0/2094782264 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5800cc30 con 0x7f1b680b1b10 2024-02-19T15:29:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b5ea00640 1 -- 172.21.15.47:0/2094782264 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b58013470 con 0x7f1b680b1b10 2024-02-19T15:29:28.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b67200640 1 -- 172.21.15.47:0/2094782264 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b60009a70 con 0x7f1b680b1b10 2024-02-19T15:29:28.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b67200640 1 -- 172.21.15.47:0/2094782264 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b6806c870 con 0x7f1b680b1b10 2024-02-19T15:29:28.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.218+0000 7f1b67200640 1 -- 172.21.15.47:0/2094782264 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b68161460 con 0x7f1b680b1b10 2024-02-19T15:29:28.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.222+0000 7f1b5ea00640 1 -- 172.21.15.47:0/2094782264 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1b580135d0 con 0x7f1b680b1b10 2024-02-19T15:29:28.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.222+0000 7f1b5ea00640 1 --2- 172.21.15.47:0/2094782264 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1b44066e40 0x7f1b44069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:28.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.222+0000 7f1b5ea00640 1 -- 172.21.15.47:0/2094782264 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b58084d40 con 0x7f1b680b1b10 2024-02-19T15:29:28.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.222+0000 7f1b5d400640 1 --2- 172.21.15.47:0/2094782264 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1b44066e40 0x7f1b44069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:28.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.222+0000 7f1b5d400640 1 --2- 172.21.15.47:0/2094782264 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1b44066e40 0x7f1b44069300 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f1b50007fd0 tx=0x7f1b50009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:28.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.226+0000 7f1b5ea00640 1 -- 172.21.15.47:0/2094782264 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1b58051b40 con 0x7f1b680b1b10 2024-02-19T15:29:28.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.450+0000 7f1b67200640 1 -- 172.21.15.47:0/2094782264 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b68075360 con 0x7f1b680b1b10 2024-02-19T15:29:28.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.450+0000 7f1b5ea00640 1 -- 172.21.15.47:0/2094782264 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3379 (secure 0 0 0) 0x7f1b58017d50 con 0x7f1b680b1b10 2024-02-19T15:29:28.451 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:28.451 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:24.882579+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:28.451 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 98 2024-02-19T15:29:28.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.454+0000 7f1b67200640 1 -- 172.21.15.47:0/2094782264 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1b44066e40 msgr2=0x7f1b44069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:28.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.454+0000 7f1b67200640 1 --2- 172.21.15.47:0/2094782264 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1b44066e40 0x7f1b44069300 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f1b50007fd0 tx=0x7f1b50009040 comp rx=0 tx=0).stop 2024-02-19T15:29:28.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.454+0000 7f1b67200640 1 -- 172.21.15.47:0/2094782264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b680b1b10 msgr2=0x7f1b6814c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:28.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.454+0000 7f1b67200640 1 --2- 172.21.15.47:0/2094782264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b680b1b10 0x7f1b6814c880 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800ebe0 tx=0x7f1b5800c520 comp rx=0 tx=0).stop 2024-02-19T15:29:28.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.454+0000 7f1b67200640 1 -- 172.21.15.47:0/2094782264 shutdown_connections 2024-02-19T15:29:28.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.454+0000 7f1b67200640 1 --2- 172.21.15.47:0/2094782264 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1b44066e40 0x7f1b44069300 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:28.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.454+0000 7f1b67200640 1 --2- 172.21.15.47:0/2094782264 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b6815dbb0 0x7f1b681562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:28.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.454+0000 7f1b67200640 1 --2- 172.21.15.47:0/2094782264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b680cb3d0 0x7f1b6814ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:28.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.454+0000 7f1b67200640 1 --2- 172.21.15.47:0/2094782264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b680b1b10 0x7f1b6814c880 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:28.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.454+0000 7f1b67200640 1 -- 172.21.15.47:0/2094782264 >> 172.21.15.47:0/2094782264 conn(0x7f1b680cf7c0 msgr2=0x7f1b680cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:28.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.454+0000 7f1b67200640 1 -- 172.21.15.47:0/2094782264 shutdown_connections 2024-02-19T15:29:28.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:28.454+0000 7f1b67200640 1 -- 172.21.15.47:0/2094782264 wait complete. 2024-02-19T15:29:28.673 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:24.882579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:28.673 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:29.674 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:29.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.814+0000 7f962c600640 1 Processor -- start 2024-02-19T15:29:29.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.814+0000 7f962c600640 1 -- start start 2024-02-19T15:29:29.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.814+0000 7f962c600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96240b2fc0 0x7f96240d0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:29.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.814+0000 7f962c600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96240d10a0 con 0x7f96240cef90 2024-02-19T15:29:29.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.814+0000 7f962c600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96240b33a0 con 0x7f96240ceb00 2024-02-19T15:29:29.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.814+0000 7f962c600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96240cc2f0 con 0x7f96240b2fc0 2024-02-19T15:29:29.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.818+0000 7f9629200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f96240cef90 0x7f96240ce3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42718/0 (socket says 172.21.15.47:42718) 2024-02-19T15:29:29.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.818+0000 7f9629200640 1 -- 172.21.15.47:0/3511246439 learned_addr learned my addr 172.21.15.47:0/3511246439 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:29:29.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.818+0000 7f9629c00640 1 --2- 172.21.15.47:0/3511246439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96240b2fc0 0x7f96240d0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:29.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.818+0000 7f9623e00640 1 -- 172.21.15.47:0/3511246439 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215705123 0 0) 0x7f96240d10a0 con 0x7f96240cef90 2024-02-19T15:29:29.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.818+0000 7f9623e00640 1 -- 172.21.15.47:0/3511246439 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f960c003530 con 0x7f96240cef90 2024-02-19T15:29:29.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.818+0000 7f9623e00640 1 -- 172.21.15.47:0/3511246439 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229831041 0 0) 0x7f96240b33a0 con 0x7f96240ceb00 2024-02-19T15:29:29.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.818+0000 7f9623e00640 1 -- 172.21.15.47:0/3511246439 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96240d10a0 con 0x7f96240ceb00 2024-02-19T15:29:29.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.818+0000 7f9629c00640 1 -- 172.21.15.47:0/3511246439 >> v1:172.21.15.116:6789/0 conn(0x7f96240ceb00 legacy=0x7f96240cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:29.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.818+0000 7f9629c00640 1 -- 172.21.15.47:0/3511246439 >> v1:172.21.15.47:6789/0 conn(0x7f96240cef90 legacy=0x7f96240ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:29.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.818+0000 7f9629c00640 1 -- 172.21.15.47:0/3511246439 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96240cc5d0 con 0x7f96240b2fc0 2024-02-19T15:29:29.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.818+0000 7f9629c00640 1 --2- 172.21.15.47:0/3511246439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96240b2fc0 0x7f96240d0770 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f96180097f0 tx=0x7f961801c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25befdc3a512daa server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:29.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.818+0000 7f9623e00640 1 -- 172.21.15.47:0/3511246439 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96180ab620 con 0x7f96240b2fc0 2024-02-19T15:29:29.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f9623e00640 1 -- 172.21.15.47:0/3511246439 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96180ab780 con 0x7f96240b2fc0 2024-02-19T15:29:29.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f9623e00640 1 -- 172.21.15.47:0/3511246439 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96180abaf0 con 0x7f96240b2fc0 2024-02-19T15:29:29.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f962c600640 1 -- 172.21.15.47:0/3511246439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96240b2fc0 msgr2=0x7f96240d0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:29.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f962c600640 1 --2- 172.21.15.47:0/3511246439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96240b2fc0 0x7f96240d0770 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f96180097f0 tx=0x7f961801c420 comp rx=0 tx=0).stop 2024-02-19T15:29:29.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f962c600640 1 -- 172.21.15.47:0/3511246439 shutdown_connections 2024-02-19T15:29:29.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f962c600640 1 --2- 172.21.15.47:0/3511246439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96240b2fc0 0x7f96240d0770 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:29.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f962c600640 1 -- 172.21.15.47:0/3511246439 >> 172.21.15.47:0/3511246439 conn(0x7f96240d1b70 msgr2=0x7f96240d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:29.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f962c600640 1 -- 172.21.15.47:0/3511246439 shutdown_connections 2024-02-19T15:29:29.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f962c600640 1 -- 172.21.15.47:0/3511246439 wait complete. 2024-02-19T15:29:29.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f962c600640 1 Processor -- start 2024-02-19T15:29:29.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f962c600640 1 -- start start 2024-02-19T15:29:29.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f962c600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96240b2fc0 0x7f9624150100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:29.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f962c600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96240ceb00 0x7f962414d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:29.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f962c600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96240cef90 0x7f96240ca490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:29.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f962c600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96240cb9f0 con 0x7f96240ceb00 2024-02-19T15:29:29.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f962c600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962414d9d0 con 0x7f96240cef90 2024-02-19T15:29:29.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f962c600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96240c83c0 con 0x7f96240b2fc0 2024-02-19T15:29:29.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f9629200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96240ceb00 0x7f962414d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:29.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f9629200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96240ceb00 0x7f962414d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:45994/0 (socket says 172.21.15.47:45994) 2024-02-19T15:29:29.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f9629200640 1 -- 172.21.15.47:0/1770249795 learned_addr learned my addr 172.21.15.47:0/1770249795 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:29.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f9629c00640 1 --2- 172.21.15.47:0/1770249795 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96240b2fc0 0x7f9624150100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:29.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.822+0000 7f962a600640 1 --2- 172.21.15.47:0/1770249795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96240cef90 0x7f96240ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:29.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.826+0000 7f9629200640 1 -- 172.21.15.47:0/1770249795 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96240b2fc0 msgr2=0x7f9624150100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:29.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.826+0000 7f9629200640 1 --2- 172.21.15.47:0/1770249795 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96240b2fc0 0x7f9624150100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:29.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.826+0000 7f9629200640 1 -- 172.21.15.47:0/1770249795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96240cef90 msgr2=0x7f96240ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:29.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.826+0000 7f9629200640 1 --2- 172.21.15.47:0/1770249795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96240cef90 0x7f96240ca490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:29.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.826+0000 7f9629200640 1 -- 172.21.15.47:0/1770249795 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9618009420 con 0x7f96240ceb00 2024-02-19T15:29:29.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.826+0000 7f962a600640 1 --2- 172.21.15.47:0/1770249795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96240cef90 0x7f96240ca490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:29:29.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.826+0000 7f9629200640 1 --2- 172.21.15.47:0/1770249795 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96240ceb00 0x7f962414d490 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f9610005890 tx=0x7f961000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:29.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.826+0000 7f9621600640 1 -- 172.21.15.47:0/1770249795 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9610005cf0 con 0x7f96240ceb00 2024-02-19T15:29:29.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.826+0000 7f9621600640 1 -- 172.21.15.47:0/1770249795 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9610005e50 con 0x7f96240ceb00 2024-02-19T15:29:29.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.826+0000 7f962c600640 1 -- 172.21.15.47:0/1770249795 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96240cb3a0 con 0x7f96240ceb00 2024-02-19T15:29:29.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.826+0000 7f9621600640 1 -- 172.21.15.47:0/1770249795 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96100072b0 con 0x7f96240ceb00 2024-02-19T15:29:29.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.826+0000 7f962c600640 1 -- 172.21.15.47:0/1770249795 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96240c7eb0 con 0x7f96240ceb00 2024-02-19T15:29:29.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.826+0000 7f962c600640 1 -- 172.21.15.47:0/1770249795 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96240c8190 con 0x7f96240ceb00 2024-02-19T15:29:29.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.826+0000 7f9621600640 1 -- 172.21.15.47:0/1770249795 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9610007410 con 0x7f96240ceb00 2024-02-19T15:29:29.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.826+0000 7f9621600640 1 --2- 172.21.15.47:0/1770249795 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9604066e40 0x7f9604069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:29.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.826+0000 7f9621600640 1 -- 172.21.15.47:0/1770249795 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9610083fc0 con 0x7f96240ceb00 2024-02-19T15:29:29.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.830+0000 7f9629c00640 1 --2- 172.21.15.47:0/1770249795 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9604066e40 0x7f9604069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:29.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.830+0000 7f9629c00640 1 --2- 172.21.15.47:0/1770249795 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9604066e40 0x7f9604069300 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f96180af000 tx=0x7f96180a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:29.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:29.830+0000 7f9621600640 1 -- 172.21.15.47:0/1770249795 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f96100b61c0 con 0x7f96240ceb00 2024-02-19T15:29:30.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:30.026+0000 7f962c600640 1 -- 172.21.15.47:0/1770249795 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9624075360 con 0x7f96240ceb00 2024-02-19T15:29:30.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:30.026+0000 7f9621600640 1 -- 172.21.15.47:0/1770249795 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3379 (secure 0 0 0) 0x7f9610054990 con 0x7f96240ceb00 2024-02-19T15:29:30.028 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:30.028 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:24.882579+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:30.028 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 98 2024-02-19T15:29:30.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:30.030+0000 7f962c600640 1 -- 172.21.15.47:0/1770249795 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9604066e40 msgr2=0x7f9604069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:30.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:30.030+0000 7f962c600640 1 --2- 172.21.15.47:0/1770249795 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9604066e40 0x7f9604069300 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f96180af000 tx=0x7f96180a7000 comp rx=0 tx=0).stop 2024-02-19T15:29:30.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:30.030+0000 7f962c600640 1 -- 172.21.15.47:0/1770249795 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96240ceb00 msgr2=0x7f962414d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:30.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:30.030+0000 7f962c600640 1 --2- 172.21.15.47:0/1770249795 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96240ceb00 0x7f962414d490 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f9610005890 tx=0x7f961000c960 comp rx=0 tx=0).stop 2024-02-19T15:29:30.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:30.030+0000 7f962c600640 1 -- 172.21.15.47:0/1770249795 shutdown_connections 2024-02-19T15:29:30.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:30.030+0000 7f962c600640 1 --2- 172.21.15.47:0/1770249795 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9604066e40 0x7f9604069300 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:30.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:30.030+0000 7f962c600640 1 --2- 172.21.15.47:0/1770249795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96240cef90 0x7f96240ca490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:30.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:30.030+0000 7f962c600640 1 --2- 172.21.15.47:0/1770249795 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96240ceb00 0x7f962414d490 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:30.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:30.030+0000 7f962c600640 1 --2- 172.21.15.47:0/1770249795 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96240b2fc0 0x7f9624150100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:30.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:30.030+0000 7f962c600640 1 -- 172.21.15.47:0/1770249795 >> 172.21.15.47:0/1770249795 conn(0x7f96240d1b70 msgr2=0x7f96240702b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:30.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:30.030+0000 7f962c600640 1 -- 172.21.15.47:0/1770249795 shutdown_connections 2024-02-19T15:29:30.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:30.030+0000 7f962c600640 1 -- 172.21.15.47:0/1770249795 wait complete. 2024-02-19T15:29:30.168 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:29:30.251 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:24.882579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:30.251 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:31.251 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:31.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.390+0000 7f901c400640 1 Processor -- start 2024-02-19T15:29:31.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.390+0000 7f901c400640 1 -- start start 2024-02-19T15:29:31.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.394+0000 7f901c400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f901415c0d0 0x7f90140cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:31.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.394+0000 7f901c400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f90140d0540 0x7f90140b1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:31.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.394+0000 7f901c400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90140b0fe0 con 0x7f90140cd180 2024-02-19T15:29:31.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.394+0000 7f901c400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f901415c4b0 con 0x7f90140d0540 2024-02-19T15:29:31.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.394+0000 7f901c400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90140b1880 con 0x7f901415c0d0 2024-02-19T15:29:31.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.394+0000 7f9019000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f90140cd180 0x7f90140cfe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42726/0 (socket says 172.21.15.47:42726) 2024-02-19T15:29:31.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.394+0000 7f9019000640 1 -- 172.21.15.47:0/1378617999 learned_addr learned my addr 172.21.15.47:0/1378617999 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:29:31.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.394+0000 7f901a400640 1 --2- 172.21.15.47:0/1378617999 >> v2:172.21.15.47:3300/0 conn(0x7f90140d0540 0x7f90140b1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:31.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.394+0000 7f9019a00640 1 --2- 172.21.15.47:0/1378617999 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f901415c0d0 0x7f90140cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:31.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.394+0000 7f901a400640 1 -- 172.21.15.47:0/1378617999 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f901415c0d0 msgr2=0x7f90140cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:31.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.394+0000 7f901a400640 1 --2- 172.21.15.47:0/1378617999 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f901415c0d0 0x7f90140cf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:31.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.394+0000 7f901a400640 1 -- 172.21.15.47:0/1378617999 >> v1:172.21.15.47:6789/0 conn(0x7f90140cd180 legacy=0x7f90140cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:31.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.394+0000 7f901a400640 1 -- 172.21.15.47:0/1378617999 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90140cd8f0 con 0x7f90140d0540 2024-02-19T15:29:31.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.394+0000 7f9019a00640 1 --2- 172.21.15.47:0/1378617999 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f901415c0d0 0x7f90140cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:29:31.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.398+0000 7f901a400640 1 --2- 172.21.15.47:0/1378617999 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90140d0540 0x7f90140b1d10 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f900c009d90 tx=0x7f900c01c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1b60043e0c15113 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:31.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.398+0000 7f9013e00640 1 -- 172.21.15.47:0/1378617999 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c0a5a00 con 0x7f90140d0540 2024-02-19T15:29:31.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.398+0000 7f9013e00640 1 -- 172.21.15.47:0/1378617999 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900c0a5b60 con 0x7f90140d0540 2024-02-19T15:29:31.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.398+0000 7f9013e00640 1 -- 172.21.15.47:0/1378617999 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c0ad440 con 0x7f90140d0540 2024-02-19T15:29:31.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.398+0000 7f901c400640 1 -- 172.21.15.47:0/1378617999 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90140d0540 msgr2=0x7f90140b1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:31.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.398+0000 7f901c400640 1 --2- 172.21.15.47:0/1378617999 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90140d0540 0x7f90140b1d10 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f900c009d90 tx=0x7f900c01c940 comp rx=0 tx=0).stop 2024-02-19T15:29:31.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.398+0000 7f901c400640 1 -- 172.21.15.47:0/1378617999 shutdown_connections 2024-02-19T15:29:31.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.398+0000 7f901c400640 1 --2- 172.21.15.47:0/1378617999 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90140d0540 0x7f90140b1d10 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:31.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.398+0000 7f901c400640 1 --2- 172.21.15.47:0/1378617999 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f901415c0d0 0x7f90140cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:31.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.398+0000 7f901c400640 1 -- 172.21.15.47:0/1378617999 >> 172.21.15.47:0/1378617999 conn(0x7f90140d1ba0 msgr2=0x7f90140d1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:31.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.398+0000 7f901c400640 1 -- 172.21.15.47:0/1378617999 shutdown_connections 2024-02-19T15:29:31.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.398+0000 7f901c400640 1 -- 172.21.15.47:0/1378617999 wait complete. 2024-02-19T15:29:31.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f901c400640 1 Processor -- start 2024-02-19T15:29:31.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f901c400640 1 -- start start 2024-02-19T15:29:31.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f901c400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90140cd180 0x7f9014158df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:31.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f901c400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90140d0540 0x7f9014164860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:31.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f901c400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f901415c0d0 0x7f901414a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:31.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f901c400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901414b250 con 0x7f90140cd180 2024-02-19T15:29:31.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f901c400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9014164f60 con 0x7f901415c0d0 2024-02-19T15:29:31.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f901c400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901406c170 con 0x7f90140d0540 2024-02-19T15:29:31.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f9019a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90140cd180 0x7f9014158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:31.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f9019a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90140cd180 0x7f9014158df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:46004/0 (socket says 172.21.15.47:46004) 2024-02-19T15:29:31.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f9019a00640 1 -- 172.21.15.47:0/1684990976 learned_addr learned my addr 172.21.15.47:0/1684990976 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:31.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f9019000640 1 --2- 172.21.15.47:0/1684990976 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90140d0540 0x7f9014164860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:31.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f901a400640 1 --2- 172.21.15.47:0/1684990976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f901415c0d0 0x7f901414a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:31.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f9019a00640 1 -- 172.21.15.47:0/1684990976 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90140d0540 msgr2=0x7f9014164860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:31.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f9019a00640 1 --2- 172.21.15.47:0/1684990976 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90140d0540 0x7f9014164860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:31.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f9019a00640 1 -- 172.21.15.47:0/1684990976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f901415c0d0 msgr2=0x7f901414a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:31.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f9019a00640 1 --2- 172.21.15.47:0/1684990976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f901415c0d0 0x7f901414a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:31.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f9019a00640 1 -- 172.21.15.47:0/1684990976 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900c009a70 con 0x7f90140cd180 2024-02-19T15:29:31.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f901a400640 1 --2- 172.21.15.47:0/1684990976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f901415c0d0 0x7f901414a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:29:31.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f9019000640 1 --2- 172.21.15.47:0/1684990976 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90140d0540 0x7f9014164860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:29:31.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f9019a00640 1 --2- 172.21.15.47:0/1684990976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90140cd180 0x7f9014158df0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f900800d800 tx=0x7f900800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:31.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f9011600640 1 -- 172.21.15.47:0/1684990976 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90080096c0 con 0x7f90140cd180 2024-02-19T15:29:31.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f9011600640 1 -- 172.21.15.47:0/1684990976 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9008009820 con 0x7f90140cd180 2024-02-19T15:29:31.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f9011600640 1 -- 172.21.15.47:0/1684990976 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9008009bd0 con 0x7f90140cd180 2024-02-19T15:29:31.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f901c400640 1 -- 172.21.15.47:0/1684990976 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f901406c450 con 0x7f90140cd180 2024-02-19T15:29:31.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f901c400640 1 -- 172.21.15.47:0/1684990976 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f901406ca10 con 0x7f90140cd180 2024-02-19T15:29:31.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.402+0000 7f901c400640 1 -- 172.21.15.47:0/1684990976 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9014068500 con 0x7f90140cd180 2024-02-19T15:29:31.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.406+0000 7f9011600640 1 -- 172.21.15.47:0/1684990976 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9008009d30 con 0x7f90140cd180 2024-02-19T15:29:31.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.406+0000 7f9011600640 1 --2- 172.21.15.47:0/1684990976 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ff4066e70 0x7f8ff4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:31.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.406+0000 7f9011600640 1 -- 172.21.15.47:0/1684990976 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9008085180 con 0x7f90140cd180 2024-02-19T15:29:31.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.406+0000 7f9019000640 1 --2- 172.21.15.47:0/1684990976 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ff4066e70 0x7f8ff4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:31.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.406+0000 7f9019000640 1 --2- 172.21.15.47:0/1684990976 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ff4066e70 0x7f8ff4069330 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9000007fd0 tx=0x7f9000009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:31.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.410+0000 7f9011600640 1 -- 172.21.15.47:0/1684990976 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f9008051f60 con 0x7f90140cd180 2024-02-19T15:29:31.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.606+0000 7f901c400640 1 -- 172.21.15.47:0/1684990976 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9014075360 con 0x7f90140cd180 2024-02-19T15:29:31.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.606+0000 7f9011600640 1 -- 172.21.15.47:0/1684990976 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3379 (secure 0 0 0) 0x7f9008055620 con 0x7f90140cd180 2024-02-19T15:29:31.607 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:31.607 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:24.882579+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:31.607 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 98 2024-02-19T15:29:31.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.610+0000 7f901c400640 1 -- 172.21.15.47:0/1684990976 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ff4066e70 msgr2=0x7f8ff4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:31.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.610+0000 7f901c400640 1 --2- 172.21.15.47:0/1684990976 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ff4066e70 0x7f8ff4069330 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9000007fd0 tx=0x7f9000009040 comp rx=0 tx=0).stop 2024-02-19T15:29:31.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.610+0000 7f901c400640 1 -- 172.21.15.47:0/1684990976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90140cd180 msgr2=0x7f9014158df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:31.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.610+0000 7f901c400640 1 --2- 172.21.15.47:0/1684990976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90140cd180 0x7f9014158df0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f900800d800 tx=0x7f900800dcc0 comp rx=0 tx=0).stop 2024-02-19T15:29:31.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.610+0000 7f901c400640 1 -- 172.21.15.47:0/1684990976 shutdown_connections 2024-02-19T15:29:31.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.610+0000 7f901c400640 1 --2- 172.21.15.47:0/1684990976 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ff4066e70 0x7f8ff4069330 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:31.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.610+0000 7f901c400640 1 --2- 172.21.15.47:0/1684990976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f901415c0d0 0x7f901414a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:31.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.610+0000 7f901c400640 1 --2- 172.21.15.47:0/1684990976 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90140d0540 0x7f9014164860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:31.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.610+0000 7f901c400640 1 --2- 172.21.15.47:0/1684990976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90140cd180 0x7f9014158df0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:31.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.610+0000 7f901c400640 1 -- 172.21.15.47:0/1684990976 >> 172.21.15.47:0/1684990976 conn(0x7f90140d1ba0 msgr2=0x7f90140aebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:31.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.610+0000 7f901c400640 1 -- 172.21.15.47:0/1684990976 shutdown_connections 2024-02-19T15:29:31.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:31.610+0000 7f901c400640 1 -- 172.21.15.47:0/1684990976 wait complete. 2024-02-19T15:29:31.829 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:24.882579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:31.829 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:32.170 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:29:32.830 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:32.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.970+0000 7fc9e6c00640 1 Processor -- start 2024-02-19T15:29:32.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.970+0000 7fc9e6c00640 1 -- start start 2024-02-19T15:29:32.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.974+0000 7fc9e6c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc9e00b2fc0 0x7fc9e00d0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:32.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.974+0000 7fc9e6c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc9e00cef90 0x7fc9e00ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:32.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.974+0000 7fc9e6c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9e00ce930 0x7fc9e006bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:32.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.974+0000 7fc9e6c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9e00cedc0 con 0x7fc9e00b2fc0 2024-02-19T15:29:32.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.974+0000 7fc9e6c00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9e00d10a0 con 0x7fc9e00cef90 2024-02-19T15:29:32.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.974+0000 7fc9e6c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e00b33a0 con 0x7fc9e00ce930 2024-02-19T15:29:32.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.974+0000 7fc9dfe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc9e00b2fc0 0x7fc9e00d0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:32.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.974+0000 7fc9dfe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc9e00b2fc0 0x7fc9e00d0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:46008/0 (socket says 172.21.15.47:46008) 2024-02-19T15:29:32.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.974+0000 7fc9dfe00640 1 -- 172.21.15.47:0/1896751249 learned_addr learned my addr 172.21.15.47:0/1896751249 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:32.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.974+0000 7fc9d7400640 1 --2- 172.21.15.47:0/1896751249 >> v2:172.21.15.116:3300/0 conn(0x7fc9e00cef90 0x7fc9e00ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:32.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.974+0000 7fc9dfe00640 1 -- 172.21.15.47:0/1896751249 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9e00ce930 msgr2=0x7fc9e006bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:32.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.974+0000 7fc9e4c00640 1 --2- 172.21.15.47:0/1896751249 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9e00ce930 0x7fc9e006bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:32.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.974+0000 7fc9dfe00640 1 --2- 172.21.15.47:0/1896751249 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9e00ce930 0x7fc9e006bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:32.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.974+0000 7fc9dfe00640 1 -- 172.21.15.47:0/1896751249 >> v2:172.21.15.116:3300/0 conn(0x7fc9e00cef90 msgr2=0x7fc9e00ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:32.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.974+0000 7fc9dfe00640 1 --2- 172.21.15.47:0/1896751249 >> v2:172.21.15.116:3300/0 conn(0x7fc9e00cef90 0x7fc9e00ce3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:32.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.974+0000 7fc9dfe00640 1 -- 172.21.15.47:0/1896751249 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e00cc2f0 con 0x7fc9e00b2fc0 2024-02-19T15:29:32.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.974+0000 7fc9d7400640 1 --2- 172.21.15.47:0/1896751249 >> v2:172.21.15.116:3300/0 conn(0x7fc9e00cef90 0x7fc9e00ce3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:29:32.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.974+0000 7fc9e4c00640 1 --2- 172.21.15.47:0/1896751249 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9e00ce930 0x7fc9e006bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:29:32.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.974+0000 7fc9dfe00640 1 --2- 172.21.15.47:0/1896751249 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9e00b2fc0 0x7fc9e00d0770 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fc9e009fbb0 tx=0x7fc9d001c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2878740d566d4a4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:32.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9df400640 1 -- 172.21.15.47:0/1896751249 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d00a6500 con 0x7fc9e00b2fc0 2024-02-19T15:29:32.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9df400640 1 -- 172.21.15.47:0/1896751249 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d00a6660 con 0x7fc9e00b2fc0 2024-02-19T15:29:32.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9df400640 1 -- 172.21.15.47:0/1896751249 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d00a69f0 con 0x7fc9e00b2fc0 2024-02-19T15:29:32.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9e6c00640 1 -- 172.21.15.47:0/1896751249 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9e00b2fc0 msgr2=0x7fc9e00d0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:32.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9e6c00640 1 --2- 172.21.15.47:0/1896751249 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9e00b2fc0 0x7fc9e00d0770 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fc9e009fbb0 tx=0x7fc9d001c670 comp rx=0 tx=0).stop 2024-02-19T15:29:32.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9e6c00640 1 -- 172.21.15.47:0/1896751249 shutdown_connections 2024-02-19T15:29:32.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9e6c00640 1 --2- 172.21.15.47:0/1896751249 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9e00ce930 0x7fc9e006bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:32.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9e6c00640 1 --2- 172.21.15.47:0/1896751249 >> v2:172.21.15.116:3300/0 conn(0x7fc9e00cef90 0x7fc9e00ce3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:32.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9e6c00640 1 --2- 172.21.15.47:0/1896751249 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9e00b2fc0 0x7fc9e00d0770 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:32.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9e6c00640 1 -- 172.21.15.47:0/1896751249 >> 172.21.15.47:0/1896751249 conn(0x7fc9e00d1b70 msgr2=0x7fc9e00d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:32.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9e6c00640 1 -- 172.21.15.47:0/1896751249 shutdown_connections 2024-02-19T15:29:32.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9e6c00640 1 -- 172.21.15.47:0/1896751249 wait complete. 2024-02-19T15:29:32.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9e6c00640 1 Processor -- start 2024-02-19T15:29:32.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9e6c00640 1 -- start start 2024-02-19T15:29:32.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9e6c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9e00b2fc0 0x7fc9e0157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:32.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9e6c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9e00ce930 0x7fc9e014cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:32.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9e6c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9e00cef90 0x7fc9e0156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:32.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9e6c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e014d0a0 con 0x7fc9e00cef90 2024-02-19T15:29:32.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9e6c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e0156b60 con 0x7fc9e00ce930 2024-02-19T15:29:32.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9e6c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e0159950 con 0x7fc9e00b2fc0 2024-02-19T15:29:32.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9e4c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9e00cef90 0x7fc9e0156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:32.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9e4c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9e00cef90 0x7fc9e0156380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:46016/0 (socket says 172.21.15.47:46016) 2024-02-19T15:29:32.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9e4c00640 1 -- 172.21.15.47:0/2915554219 learned_addr learned my addr 172.21.15.47:0/2915554219 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:32.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9dfe00640 1 --2- 172.21.15.47:0/2915554219 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9e00b2fc0 0x7fc9e0157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:32.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.978+0000 7fc9d7400640 1 --2- 172.21.15.47:0/2915554219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9e00ce930 0x7fc9e014cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:32.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.982+0000 7fc9e4c00640 1 -- 172.21.15.47:0/2915554219 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9e00b2fc0 msgr2=0x7fc9e0157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:32.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.982+0000 7fc9e4c00640 1 --2- 172.21.15.47:0/2915554219 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9e00b2fc0 0x7fc9e0157060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:32.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.982+0000 7fc9e4c00640 1 -- 172.21.15.47:0/2915554219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9e00ce930 msgr2=0x7fc9e014cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:32.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.982+0000 7fc9e4c00640 1 --2- 172.21.15.47:0/2915554219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9e00ce930 0x7fc9e014cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:32.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.982+0000 7fc9e4c00640 1 -- 172.21.15.47:0/2915554219 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d0009b30 con 0x7fc9e00cef90 2024-02-19T15:29:32.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.982+0000 7fc9e4c00640 1 --2- 172.21.15.47:0/2915554219 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9e00cef90 0x7fc9e0156380 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fc9d800c570 tx=0x7fc9d800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:32.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.982+0000 7fc9dcc00640 1 -- 172.21.15.47:0/2915554219 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8004440 con 0x7fc9e00cef90 2024-02-19T15:29:32.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.982+0000 7fc9dcc00640 1 -- 172.21.15.47:0/2915554219 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d8003620 con 0x7fc9e00cef90 2024-02-19T15:29:32.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.982+0000 7fc9dcc00640 1 -- 172.21.15.47:0/2915554219 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d80039d0 con 0x7fc9e00cef90 2024-02-19T15:29:32.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.982+0000 7fc9e6c00640 1 -- 172.21.15.47:0/2915554219 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9e01568c0 con 0x7fc9e00cef90 2024-02-19T15:29:32.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.982+0000 7fc9e6c00640 1 -- 172.21.15.47:0/2915554219 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9e0159ed0 con 0x7fc9e00cef90 2024-02-19T15:29:32.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.982+0000 7fc9e6c00640 1 -- 172.21.15.47:0/2915554219 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9e0075360 con 0x7fc9e00cef90 2024-02-19T15:29:32.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.982+0000 7fc9dcc00640 1 -- 172.21.15.47:0/2915554219 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc9d8003b30 con 0x7fc9e00cef90 2024-02-19T15:29:32.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.982+0000 7fc9dcc00640 1 --2- 172.21.15.47:0/2915554219 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc9bc066e70 0x7fc9bc069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:32.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.982+0000 7fc9dcc00640 1 -- 172.21.15.47:0/2915554219 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9d8084b50 con 0x7fc9e00cef90 2024-02-19T15:29:32.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.986+0000 7fc9dfe00640 1 --2- 172.21.15.47:0/2915554219 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc9bc066e70 0x7fc9bc069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:32.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.986+0000 7fc9dfe00640 1 --2- 172.21.15.47:0/2915554219 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc9bc066e70 0x7fc9bc069330 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fc9d00ae000 tx=0x7fc9d00b2040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:32.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:32.986+0000 7fc9dcc00640 1 -- 172.21.15.47:0/2915554219 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc9d8051950 con 0x7fc9e00cef90 2024-02-19T15:29:33.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:33.182+0000 7fc9e6c00640 1 -- 172.21.15.47:0/2915554219 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9e00b24c0 con 0x7fc9e00cef90 2024-02-19T15:29:33.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:33.182+0000 7fc9dcc00640 1 -- 172.21.15.47:0/2915554219 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3379 (secure 0 0 0) 0x7fc9d8015090 con 0x7fc9e00cef90 2024-02-19T15:29:33.183 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:33.183 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:24.882579+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:33.183 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 98 2024-02-19T15:29:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:33.186+0000 7fc9e6c00640 1 -- 172.21.15.47:0/2915554219 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc9bc066e70 msgr2=0x7fc9bc069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:33.186+0000 7fc9e6c00640 1 --2- 172.21.15.47:0/2915554219 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc9bc066e70 0x7fc9bc069330 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fc9d00ae000 tx=0x7fc9d00b2040 comp rx=0 tx=0).stop 2024-02-19T15:29:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:33.186+0000 7fc9e6c00640 1 -- 172.21.15.47:0/2915554219 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9e00cef90 msgr2=0x7fc9e0156380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:33.186+0000 7fc9e6c00640 1 --2- 172.21.15.47:0/2915554219 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9e00cef90 0x7fc9e0156380 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fc9d800c570 tx=0x7fc9d800ca30 comp rx=0 tx=0).stop 2024-02-19T15:29:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:33.186+0000 7fc9e6c00640 1 -- 172.21.15.47:0/2915554219 shutdown_connections 2024-02-19T15:29:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:33.186+0000 7fc9e6c00640 1 --2- 172.21.15.47:0/2915554219 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc9bc066e70 0x7fc9bc069330 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:33.186+0000 7fc9e6c00640 1 --2- 172.21.15.47:0/2915554219 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9e00cef90 0x7fc9e0156380 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:33.186+0000 7fc9e6c00640 1 --2- 172.21.15.47:0/2915554219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9e00ce930 0x7fc9e014cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:33.186+0000 7fc9e6c00640 1 --2- 172.21.15.47:0/2915554219 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9e00b2fc0 0x7fc9e0157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:33.186+0000 7fc9e6c00640 1 -- 172.21.15.47:0/2915554219 >> 172.21.15.47:0/2915554219 conn(0x7fc9e00d1b70 msgr2=0x7fc9e0157a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:33.186+0000 7fc9e6c00640 1 -- 172.21.15.47:0/2915554219 shutdown_connections 2024-02-19T15:29:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:33.186+0000 7fc9e6c00640 1 -- 172.21.15.47:0/2915554219 wait complete. 2024-02-19T15:29:33.407 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:24.882579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:33.407 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:34.172 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:29:34.408 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:34.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.550+0000 7ffb85600640 1 Processor -- start 2024-02-19T15:29:34.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.550+0000 7ffb85600640 1 -- start start 2024-02-19T15:29:34.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.550+0000 7ffb85600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb801476e0 0x7ffb800ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:34.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.550+0000 7ffb85600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb800af1b0 con 0x7ffb800b33c0 2024-02-19T15:29:34.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.550+0000 7ffb85600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb80070b90 con 0x7ffb8006c9a0 2024-02-19T15:29:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.550+0000 7ffb85600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb80071530 con 0x7ffb801476e0 2024-02-19T15:29:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.550+0000 7ffb7f400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ffb800b33c0 0x7ffb80071950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39116/0 (socket says 172.21.15.47:39116) 2024-02-19T15:29:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.550+0000 7ffb7f400640 1 -- 172.21.15.47:0/311288680 learned_addr learned my addr 172.21.15.47:0/311288680 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:29:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.550+0000 7ffb7ea00640 1 --2- 172.21.15.47:0/311288680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb801476e0 0x7ffb800ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.550+0000 7ffb7e000640 1 -- 172.21.15.47:0/311288680 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740176634 0 0) 0x7ffb800af1b0 con 0x7ffb800b33c0 2024-02-19T15:29:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.550+0000 7ffb7e000640 1 -- 172.21.15.47:0/311288680 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb64003530 con 0x7ffb800b33c0 2024-02-19T15:29:34.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.550+0000 7ffb7e000640 1 -- 172.21.15.47:0/311288680 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185952742 0 0) 0x7ffb80070b90 con 0x7ffb8006c9a0 2024-02-19T15:29:34.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.550+0000 7ffb7e000640 1 -- 172.21.15.47:0/311288680 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb800af1b0 con 0x7ffb8006c9a0 2024-02-19T15:29:34.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.550+0000 7ffb7ea00640 1 -- 172.21.15.47:0/311288680 >> v1:172.21.15.116:6789/0 conn(0x7ffb8006c9a0 legacy=0x7ffb800b2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:34.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.550+0000 7ffb7ea00640 1 -- 172.21.15.47:0/311288680 >> v1:172.21.15.47:6789/0 conn(0x7ffb800b33c0 legacy=0x7ffb80071950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:34.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.550+0000 7ffb7ea00640 1 -- 172.21.15.47:0/311288680 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb80070e70 con 0x7ffb801476e0 2024-02-19T15:29:34.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.554+0000 7ffb7ea00640 1 --2- 172.21.15.47:0/311288680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb801476e0 0x7ffb800ae730 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ffb700097f0 tx=0x7ffb7001c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8db5846ad96a455 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:34.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.554+0000 7ffb7e000640 1 -- 172.21.15.47:0/311288680 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb700ad660 con 0x7ffb801476e0 2024-02-19T15:29:34.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.554+0000 7ffb7e000640 1 -- 172.21.15.47:0/311288680 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb700ad7c0 con 0x7ffb801476e0 2024-02-19T15:29:34.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.554+0000 7ffb7e000640 1 -- 172.21.15.47:0/311288680 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb700adb50 con 0x7ffb801476e0 2024-02-19T15:29:34.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.554+0000 7ffb85600640 1 -- 172.21.15.47:0/311288680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb801476e0 msgr2=0x7ffb800ae730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:34.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.554+0000 7ffb85600640 1 --2- 172.21.15.47:0/311288680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb801476e0 0x7ffb800ae730 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ffb700097f0 tx=0x7ffb7001c4e0 comp rx=0 tx=0).stop 2024-02-19T15:29:34.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.554+0000 7ffb85600640 1 -- 172.21.15.47:0/311288680 shutdown_connections 2024-02-19T15:29:34.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.554+0000 7ffb85600640 1 --2- 172.21.15.47:0/311288680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb801476e0 0x7ffb800ae730 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:34.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.554+0000 7ffb85600640 1 -- 172.21.15.47:0/311288680 >> 172.21.15.47:0/311288680 conn(0x7ffb800b16a0 msgr2=0x7ffb800b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:34.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.554+0000 7ffb85600640 1 -- 172.21.15.47:0/311288680 shutdown_connections 2024-02-19T15:29:34.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.554+0000 7ffb85600640 1 -- 172.21.15.47:0/311288680 wait complete. 2024-02-19T15:29:34.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.558+0000 7ffb85600640 1 Processor -- start 2024-02-19T15:29:34.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.558+0000 7ffb85600640 1 -- start start 2024-02-19T15:29:34.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.558+0000 7ffb85600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb8006c9a0 0x7ffb801590e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:34.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.558+0000 7ffb85600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb800b33c0 0x7ffb8013d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:34.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.558+0000 7ffb85600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb801476e0 0x7ffb800b2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:34.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.558+0000 7ffb85600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8013daa0 con 0x7ffb800b33c0 2024-02-19T15:29:34.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.558+0000 7ffb85600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8013dd80 con 0x7ffb801476e0 2024-02-19T15:29:34.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.558+0000 7ffb85600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8013e060 con 0x7ffb8006c9a0 2024-02-19T15:29:34.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.562+0000 7ffb76000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb800b33c0 0x7ffb8013d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:34.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.562+0000 7ffb76000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb800b33c0 0x7ffb8013d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55996/0 (socket says 172.21.15.47:55996) 2024-02-19T15:29:34.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.562+0000 7ffb76000640 1 -- 172.21.15.47:0/3273189013 learned_addr learned my addr 172.21.15.47:0/3273189013 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:34.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.562+0000 7ffb7f400640 1 --2- 172.21.15.47:0/3273189013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb801476e0 0x7ffb800b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:34.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.562+0000 7ffb7ea00640 1 --2- 172.21.15.47:0/3273189013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb8006c9a0 0x7ffb801590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:34.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.562+0000 7ffb76000640 1 -- 172.21.15.47:0/3273189013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb8006c9a0 msgr2=0x7ffb801590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:34.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.562+0000 7ffb76000640 1 --2- 172.21.15.47:0/3273189013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb8006c9a0 0x7ffb801590e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:34.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.562+0000 7ffb76000640 1 -- 172.21.15.47:0/3273189013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb801476e0 msgr2=0x7ffb800b2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:34.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.562+0000 7ffb76000640 1 --2- 172.21.15.47:0/3273189013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb801476e0 0x7ffb800b2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:34.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.562+0000 7ffb76000640 1 -- 172.21.15.47:0/3273189013 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb70009420 con 0x7ffb800b33c0 2024-02-19T15:29:34.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.562+0000 7ffb76000640 1 --2- 172.21.15.47:0/3273189013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb800b33c0 0x7ffb8013d130 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7ffb6800c3c0 tx=0x7ffb6800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:34.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.562+0000 7ffb77400640 1 -- 172.21.15.47:0/3273189013 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb680052d0 con 0x7ffb800b33c0 2024-02-19T15:29:34.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.562+0000 7ffb77400640 1 -- 172.21.15.47:0/3273189013 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb68005430 con 0x7ffb800b33c0 2024-02-19T15:29:34.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.562+0000 7ffb77400640 1 -- 172.21.15.47:0/3273189013 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb68003840 con 0x7ffb800b33c0 2024-02-19T15:29:34.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.562+0000 7ffb85600640 1 -- 172.21.15.47:0/3273189013 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb8013e340 con 0x7ffb800b33c0 2024-02-19T15:29:34.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.562+0000 7ffb85600640 1 -- 172.21.15.47:0/3273189013 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb8015f9a0 con 0x7ffb800b33c0 2024-02-19T15:29:34.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.562+0000 7ffb85600640 1 -- 172.21.15.47:0/3273189013 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb8015fbc0 con 0x7ffb800b33c0 2024-02-19T15:29:34.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.562+0000 7ffb77400640 1 -- 172.21.15.47:0/3273189013 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ffb680039a0 con 0x7ffb800b33c0 2024-02-19T15:29:34.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.566+0000 7ffb77400640 1 --2- 172.21.15.47:0/3273189013 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffb5c066e70 0x7ffb5c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:34.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.566+0000 7ffb77400640 1 -- 172.21.15.47:0/3273189013 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb680840d0 con 0x7ffb800b33c0 2024-02-19T15:29:34.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.566+0000 7ffb7ea00640 1 --2- 172.21.15.47:0/3273189013 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffb5c066e70 0x7ffb5c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:34.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.566+0000 7ffb7ea00640 1 --2- 172.21.15.47:0/3273189013 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffb5c066e70 0x7ffb5c069330 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7ffb700af000 tx=0x7ffb700a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:34.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.570+0000 7ffb77400640 1 -- 172.21.15.47:0/3273189013 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ffb68050d90 con 0x7ffb800b33c0 2024-02-19T15:29:34.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.766+0000 7ffb85600640 1 -- 172.21.15.47:0/3273189013 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb8013e4d0 con 0x7ffb800b33c0 2024-02-19T15:29:34.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.766+0000 7ffb77400640 1 -- 172.21.15.47:0/3273189013 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3379 (secure 0 0 0) 0x7ffb68054450 con 0x7ffb800b33c0 2024-02-19T15:29:34.765 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:34.765 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:24.882579+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:34.766 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 98 2024-02-19T15:29:34.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.766+0000 7ffb85600640 1 -- 172.21.15.47:0/3273189013 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffb5c066e70 msgr2=0x7ffb5c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:34.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.766+0000 7ffb85600640 1 --2- 172.21.15.47:0/3273189013 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffb5c066e70 0x7ffb5c069330 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7ffb700af000 tx=0x7ffb700a7000 comp rx=0 tx=0).stop 2024-02-19T15:29:34.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.766+0000 7ffb85600640 1 -- 172.21.15.47:0/3273189013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb800b33c0 msgr2=0x7ffb8013d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:34.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.766+0000 7ffb85600640 1 --2- 172.21.15.47:0/3273189013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb800b33c0 0x7ffb8013d130 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7ffb6800c3c0 tx=0x7ffb6800c880 comp rx=0 tx=0).stop 2024-02-19T15:29:34.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.766+0000 7ffb85600640 1 -- 172.21.15.47:0/3273189013 shutdown_connections 2024-02-19T15:29:34.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.766+0000 7ffb85600640 1 --2- 172.21.15.47:0/3273189013 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ffb5c066e70 0x7ffb5c069330 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:34.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.766+0000 7ffb85600640 1 --2- 172.21.15.47:0/3273189013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb801476e0 0x7ffb800b2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:34.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.766+0000 7ffb85600640 1 --2- 172.21.15.47:0/3273189013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb800b33c0 0x7ffb8013d130 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:34.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.766+0000 7ffb85600640 1 --2- 172.21.15.47:0/3273189013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb8006c9a0 0x7ffb801590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:34.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.766+0000 7ffb85600640 1 -- 172.21.15.47:0/3273189013 >> 172.21.15.47:0/3273189013 conn(0x7ffb800b16a0 msgr2=0x7ffb801427d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:34.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.766+0000 7ffb85600640 1 -- 172.21.15.47:0/3273189013 shutdown_connections 2024-02-19T15:29:34.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:34.766+0000 7ffb85600640 1 -- 172.21.15.47:0/3273189013 wait complete. 2024-02-19T15:29:34.992 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:24.882579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:34.993 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:35.994 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:36.001 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:29:36.042 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:29:36.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.134+0000 7f9bd1000640 1 Processor -- start 2024-02-19T15:29:36.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.134+0000 7f9bd1000640 1 -- start start 2024-02-19T15:29:36.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.134+0000 7f9bd1000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9bcc15d660 0x7f9bcc06d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:36.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.134+0000 7f9bd1000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bcc06ddf0 con 0x7f9bcc15aef0 2024-02-19T15:29:36.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.134+0000 7f9bd1000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bcc06e0d0 con 0x7f9bcc06ef50 2024-02-19T15:29:36.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.134+0000 7f9bd1000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bcc15c350 con 0x7f9bcc15d660 2024-02-19T15:29:36.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.134+0000 7f9bc9400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9bcc15d660 0x7f9bcc06d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:36.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.138+0000 7f9bca800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f9bcc15aef0 0x7f9bcc1646f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39128/0 (socket says 172.21.15.47:39128) 2024-02-19T15:29:36.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.138+0000 7f9bca800640 1 -- 172.21.15.47:0/2146662759 learned_addr learned my addr 172.21.15.47:0/2146662759 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:29:36.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.138+0000 7f9bc9400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9bcc15d660 0x7f9bcc06d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56002/0 (socket says 172.21.15.47:56002) 2024-02-19T15:29:36.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.138+0000 7f9bc9400640 1 -- 172.21.15.47:0/2146662759 >> v1:172.21.15.116:6789/0 conn(0x7f9bcc06ef50 legacy=0x7f9bcc06f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:36.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.138+0000 7f9bc9400640 1 -- 172.21.15.47:0/2146662759 >> v1:172.21.15.47:6789/0 conn(0x7f9bcc15aef0 legacy=0x7f9bcc1646f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:36.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.138+0000 7f9bc9400640 1 -- 172.21.15.47:0/2146662759 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc06e3b0 con 0x7f9bcc15d660 2024-02-19T15:29:36.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.138+0000 7f9bc9400640 1 --2- 172.21.15.47:0/2146662759 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bcc15d660 0x7f9bcc06d6a0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f9bb800ba60 tx=0x7f9bb801eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47e14fd612fce630 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:36.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.138+0000 7f9bc8a00640 1 -- 172.21.15.47:0/2146662759 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb80a7a30 con 0x7f9bcc15d660 2024-02-19T15:29:36.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.138+0000 7f9bc8a00640 1 -- 172.21.15.47:0/2146662759 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb80a7b90 con 0x7f9bcc15d660 2024-02-19T15:29:36.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.138+0000 7f9bc8a00640 1 -- 172.21.15.47:0/2146662759 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb80af440 con 0x7f9bcc15d660 2024-02-19T15:29:36.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bd1000640 1 -- 172.21.15.47:0/2146662759 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bcc15d660 msgr2=0x7f9bcc06d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:36.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bd1000640 1 --2- 172.21.15.47:0/2146662759 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bcc15d660 0x7f9bcc06d6a0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f9bb800ba60 tx=0x7f9bb801eb70 comp rx=0 tx=0).stop 2024-02-19T15:29:36.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bd1000640 1 -- 172.21.15.47:0/2146662759 shutdown_connections 2024-02-19T15:29:36.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bd1000640 1 --2- 172.21.15.47:0/2146662759 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bcc15d660 0x7f9bcc06d6a0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:36.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bd1000640 1 -- 172.21.15.47:0/2146662759 >> 172.21.15.47:0/2146662759 conn(0x7f9bcc0d0ad0 msgr2=0x7f9bcc0d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:36.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bd1000640 1 -- 172.21.15.47:0/2146662759 shutdown_connections 2024-02-19T15:29:36.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bd1000640 1 -- 172.21.15.47:0/2146662759 wait complete. 2024-02-19T15:29:36.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bd1000640 1 Processor -- start 2024-02-19T15:29:36.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bd1000640 1 -- start start 2024-02-19T15:29:36.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bd1000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bcc06ef50 0x7f9bcc146980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:36.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bd1000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9bcc15aef0 0x7f9bcc0b5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:36.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bd1000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bcc15d660 0x7f9bcc13a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:36.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bd1000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc15a370 con 0x7f9bcc06ef50 2024-02-19T15:29:36.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bd1000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc0b3a90 con 0x7f9bcc15d660 2024-02-19T15:29:36.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bd1000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc0b57a0 con 0x7f9bcc15aef0 2024-02-19T15:29:36.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bc9e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bcc06ef50 0x7f9bcc146980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:36.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bc9e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bcc06ef50 0x7f9bcc146980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56018/0 (socket says 172.21.15.47:56018) 2024-02-19T15:29:36.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bc9e00640 1 -- 172.21.15.47:0/2988133301 learned_addr learned my addr 172.21.15.47:0/2988133301 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:36.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bca800640 1 --2- 172.21.15.47:0/2988133301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bcc15d660 0x7f9bcc13a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:36.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bc9400640 1 --2- 172.21.15.47:0/2988133301 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9bcc15aef0 0x7f9bcc0b5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:36.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bc9e00640 1 -- 172.21.15.47:0/2988133301 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9bcc15aef0 msgr2=0x7f9bcc0b5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:36.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bc9e00640 1 --2- 172.21.15.47:0/2988133301 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9bcc15aef0 0x7f9bcc0b5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:36.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bc9e00640 1 -- 172.21.15.47:0/2988133301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bcc15d660 msgr2=0x7f9bcc13a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:36.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bc9e00640 1 --2- 172.21.15.47:0/2988133301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bcc15d660 0x7f9bcc13a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:36.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bc9e00640 1 -- 172.21.15.47:0/2988133301 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb800b6c0 con 0x7f9bcc06ef50 2024-02-19T15:29:36.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bca800640 1 --2- 172.21.15.47:0/2988133301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bcc15d660 0x7f9bcc13a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:29:36.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.142+0000 7f9bc9e00640 1 --2- 172.21.15.47:0/2988133301 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bcc06ef50 0x7f9bcc146980 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc00c500 tx=0x7f9bbc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:36.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.146+0000 7f9bc2000640 1 -- 172.21.15.47:0/2988133301 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc005220 con 0x7f9bcc06ef50 2024-02-19T15:29:36.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.146+0000 7f9bc2000640 1 -- 172.21.15.47:0/2988133301 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bbc005380 con 0x7f9bcc06ef50 2024-02-19T15:29:36.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.146+0000 7f9bc2000640 1 -- 172.21.15.47:0/2988133301 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc007270 con 0x7f9bcc06ef50 2024-02-19T15:29:36.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.146+0000 7f9bd1000640 1 -- 172.21.15.47:0/2988133301 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bcc159db0 con 0x7f9bcc06ef50 2024-02-19T15:29:36.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.146+0000 7f9bd1000640 1 -- 172.21.15.47:0/2988133301 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bcc0b3720 con 0x7f9bcc06ef50 2024-02-19T15:29:36.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.146+0000 7f9bd1000640 1 -- 172.21.15.47:0/2988133301 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bcc0b3580 con 0x7f9bcc06ef50 2024-02-19T15:29:36.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.146+0000 7f9bc2000640 1 -- 172.21.15.47:0/2988133301 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9bbc0073d0 con 0x7f9bcc06ef50 2024-02-19T15:29:36.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.146+0000 7f9bc2000640 1 --2- 172.21.15.47:0/2988133301 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9bac066e40 0x7f9bac069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:36.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.146+0000 7f9bc2000640 1 -- 172.21.15.47:0/2988133301 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9bbc084250 con 0x7f9bcc06ef50 2024-02-19T15:29:36.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.146+0000 7f9bc9400640 1 --2- 172.21.15.47:0/2988133301 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9bac066e40 0x7f9bac069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:36.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.150+0000 7f9bc9400640 1 --2- 172.21.15.47:0/2988133301 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9bac066e40 0x7f9bac069300 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f9bb80b1000 tx=0x7f9bb8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:36.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.150+0000 7f9bc2000640 1 -- 172.21.15.47:0/2988133301 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f9bcc0b3580 con 0x7f9bcc06ef50 2024-02-19T15:29:36.174 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:29:36.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.350+0000 7f9bd1000640 1 -- 172.21.15.47:0/2988133301 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bcc075360 con 0x7f9bcc06ef50 2024-02-19T15:29:36.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.350+0000 7f9bc2000640 1 -- 172.21.15.47:0/2988133301 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3379 (secure 0 0 0) 0x7f9bbc051050 con 0x7f9bcc06ef50 2024-02-19T15:29:36.349 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:36.349 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:24.882579+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:36.349 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 98 2024-02-19T15:29:36.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.350+0000 7f9bd1000640 1 -- 172.21.15.47:0/2988133301 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9bac066e40 msgr2=0x7f9bac069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:36.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.350+0000 7f9bd1000640 1 --2- 172.21.15.47:0/2988133301 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9bac066e40 0x7f9bac069300 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f9bb80b1000 tx=0x7f9bb8008040 comp rx=0 tx=0).stop 2024-02-19T15:29:36.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.350+0000 7f9bd1000640 1 -- 172.21.15.47:0/2988133301 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bcc06ef50 msgr2=0x7f9bcc146980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:36.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.350+0000 7f9bd1000640 1 --2- 172.21.15.47:0/2988133301 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bcc06ef50 0x7f9bcc146980 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc00c500 tx=0x7f9bbc00c9c0 comp rx=0 tx=0).stop 2024-02-19T15:29:36.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.350+0000 7f9bd1000640 1 -- 172.21.15.47:0/2988133301 shutdown_connections 2024-02-19T15:29:36.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.350+0000 7f9bd1000640 1 --2- 172.21.15.47:0/2988133301 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9bac066e40 0x7f9bac069300 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:36.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.350+0000 7f9bd1000640 1 --2- 172.21.15.47:0/2988133301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bcc15d660 0x7f9bcc13a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:36.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.350+0000 7f9bd1000640 1 --2- 172.21.15.47:0/2988133301 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9bcc15aef0 0x7f9bcc0b5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:36.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.350+0000 7f9bd1000640 1 --2- 172.21.15.47:0/2988133301 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bcc06ef50 0x7f9bcc146980 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:36.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.350+0000 7f9bd1000640 1 -- 172.21.15.47:0/2988133301 >> 172.21.15.47:0/2988133301 conn(0x7f9bcc0d0ad0 msgr2=0x7f9bcc15da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:36.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.350+0000 7f9bd1000640 1 -- 172.21.15.47:0/2988133301 shutdown_connections 2024-02-19T15:29:36.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:36.350+0000 7f9bd1000640 1 -- 172.21.15.47:0/2988133301 wait complete. 2024-02-19T15:29:36.571 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:24.882579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:36.571 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:37.572 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:37.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.714+0000 7f6ee9e00640 1 Processor -- start 2024-02-19T15:29:37.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.714+0000 7f6ee9e00640 1 -- start start 2024-02-19T15:29:37.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.714+0000 7f6ee9e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee415f180 0x7f6ee40cca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:37.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.714+0000 7f6ee9e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6ee4067100 0x7f6ee40ccf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:37.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.714+0000 7f6ee9e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee406bf20 con 0x7f6ee40ae2f0 2024-02-19T15:29:37.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.714+0000 7f6ee9e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee406c520 con 0x7f6ee4067100 2024-02-19T15:29:37.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.714+0000 7f6ee9e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee40cfd80 con 0x7f6ee415f180 2024-02-19T15:29:37.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.714+0000 7f6ee3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6ee4067100 0x7f6ee40ccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:37.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.714+0000 7f6ee9400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6ee40ae2f0 0x7f6ee415f990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39132/0 (socket says 172.21.15.47:39132) 2024-02-19T15:29:37.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.714+0000 7f6ee9400640 1 -- 172.21.15.47:0/390917536 learned_addr learned my addr 172.21.15.47:0/390917536 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:29:37.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.714+0000 7f6ee8a00640 1 --2- 172.21.15.47:0/390917536 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee415f180 0x7f6ee40cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:37.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.714+0000 7f6ee3e00640 1 -- 172.21.15.47:0/390917536 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee415f180 msgr2=0x7f6ee40cca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:37.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.714+0000 7f6ee3400640 1 -- 172.21.15.47:0/390917536 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826462049 0 0) 0x7f6ee406bf20 con 0x7f6ee40ae2f0 2024-02-19T15:29:37.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.714+0000 7f6ee3e00640 1 --2- 172.21.15.47:0/390917536 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee415f180 0x7f6ee40cca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:37.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.714+0000 7f6ee3e00640 1 -- 172.21.15.47:0/390917536 >> v1:172.21.15.47:6789/0 conn(0x7f6ee40ae2f0 legacy=0x7f6ee415f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:37.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.714+0000 7f6ee3e00640 1 -- 172.21.15.47:0/390917536 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee406c7c0 con 0x7f6ee4067100 2024-02-19T15:29:37.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.718+0000 7f6ee3e00640 1 --2- 172.21.15.47:0/390917536 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ee4067100 0x7f6ee40ccf90 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f6ed00097c0 tx=0x7f6ed001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd75915a6d74a8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:37.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.718+0000 7f6ee3400640 1 -- 172.21.15.47:0/390917536 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed00ab600 con 0x7f6ee4067100 2024-02-19T15:29:37.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.718+0000 7f6ee3400640 1 -- 172.21.15.47:0/390917536 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed00a7050 con 0x7f6ee4067100 2024-02-19T15:29:37.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.718+0000 7f6ee3400640 1 -- 172.21.15.47:0/390917536 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed00ab980 con 0x7f6ee4067100 2024-02-19T15:29:37.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.718+0000 7f6ee9e00640 1 -- 172.21.15.47:0/390917536 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ee4067100 msgr2=0x7f6ee40ccf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:37.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.718+0000 7f6ee9e00640 1 --2- 172.21.15.47:0/390917536 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ee4067100 0x7f6ee40ccf90 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f6ed00097c0 tx=0x7f6ed001c410 comp rx=0 tx=0).stop 2024-02-19T15:29:37.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.718+0000 7f6ee9e00640 1 -- 172.21.15.47:0/390917536 shutdown_connections 2024-02-19T15:29:37.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.718+0000 7f6ee9e00640 1 --2- 172.21.15.47:0/390917536 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ee4067100 0x7f6ee40ccf90 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:37.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.718+0000 7f6ee9e00640 1 --2- 172.21.15.47:0/390917536 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee415f180 0x7f6ee40cca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:37.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.718+0000 7f6ee9e00640 1 -- 172.21.15.47:0/390917536 >> 172.21.15.47:0/390917536 conn(0x7f6ee40b1be0 msgr2=0x7f6ee40b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:37.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.718+0000 7f6ee9e00640 1 -- 172.21.15.47:0/390917536 shutdown_connections 2024-02-19T15:29:37.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.718+0000 7f6ee9e00640 1 -- 172.21.15.47:0/390917536 wait complete. 2024-02-19T15:29:37.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee9e00640 1 Processor -- start 2024-02-19T15:29:37.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee9e00640 1 -- start start 2024-02-19T15:29:37.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee9e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee4067100 0x7f6ee40c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:37.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee9e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ee40ae2f0 0x7f6ee41569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:37.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee9e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ee415f180 0x7f6ee40c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:37.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee9e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee40b4bc0 con 0x7f6ee415f180 2024-02-19T15:29:37.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee9e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee40c9610 con 0x7f6ee40ae2f0 2024-02-19T15:29:37.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee9e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee40c98f0 con 0x7f6ee4067100 2024-02-19T15:29:37.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee9400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ee415f180 0x7f6ee40c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:37.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee9400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ee415f180 0x7f6ee40c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56038/0 (socket says 172.21.15.47:56038) 2024-02-19T15:29:37.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee9400640 1 -- 172.21.15.47:0/2592735538 learned_addr learned my addr 172.21.15.47:0/2592735538 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:37.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee8a00640 1 --2- 172.21.15.47:0/2592735538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee4067100 0x7f6ee40c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:37.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee3e00640 1 --2- 172.21.15.47:0/2592735538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ee40ae2f0 0x7f6ee41569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:37.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee9400640 1 -- 172.21.15.47:0/2592735538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee4067100 msgr2=0x7f6ee40c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:37.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee9400640 1 --2- 172.21.15.47:0/2592735538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee4067100 0x7f6ee40c9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:37.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee9400640 1 -- 172.21.15.47:0/2592735538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ee40ae2f0 msgr2=0x7f6ee41569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:37.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee9400640 1 --2- 172.21.15.47:0/2592735538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ee40ae2f0 0x7f6ee41569a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:37.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee9400640 1 -- 172.21.15.47:0/2592735538 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed0009420 con 0x7f6ee415f180 2024-02-19T15:29:37.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee9400640 1 --2- 172.21.15.47:0/2592735538 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ee415f180 0x7f6ee40c74a0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f6edc00c7a0 tx=0x7f6edc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:37.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee0c00640 1 -- 172.21.15.47:0/2592735538 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6edc00a640 con 0x7f6ee415f180 2024-02-19T15:29:37.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee0c00640 1 -- 172.21.15.47:0/2592735538 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6edc00a7a0 con 0x7f6ee415f180 2024-02-19T15:29:37.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee0c00640 1 -- 172.21.15.47:0/2592735538 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6edc00abd0 con 0x7f6ee415f180 2024-02-19T15:29:37.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee9e00640 1 -- 172.21.15.47:0/2592735538 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee409aa70 con 0x7f6ee415f180 2024-02-19T15:29:37.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.722+0000 7f6ee9e00640 1 -- 172.21.15.47:0/2592735538 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee40c79e0 con 0x7f6ee415f180 2024-02-19T15:29:37.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.726+0000 7f6ee9e00640 1 -- 172.21.15.47:0/2592735538 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee4075360 con 0x7f6ee415f180 2024-02-19T15:29:37.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.726+0000 7f6ee0c00640 1 -- 172.21.15.47:0/2592735538 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6edc00ad30 con 0x7f6ee415f180 2024-02-19T15:29:37.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.726+0000 7f6ee0c00640 1 --2- 172.21.15.47:0/2592735538 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ec4066e40 0x7f6ec4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:37.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.726+0000 7f6ee0c00640 1 -- 172.21.15.47:0/2592735538 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6edc083f20 con 0x7f6ee415f180 2024-02-19T15:29:37.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.726+0000 7f6ee8a00640 1 --2- 172.21.15.47:0/2592735538 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ec4066e40 0x7f6ec4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:37.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.726+0000 7f6ee8a00640 1 --2- 172.21.15.47:0/2592735538 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ec4066e40 0x7f6ec4069300 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f6ee40b4fd0 tx=0x7f6ed8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:37.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.730+0000 7f6ee0c00640 1 -- 172.21.15.47:0/2592735538 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6edc050db0 con 0x7f6ee415f180 2024-02-19T15:29:37.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.922+0000 7f6ee9e00640 1 -- 172.21.15.47:0/2592735538 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ee409af20 con 0x7f6ee415f180 2024-02-19T15:29:37.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.926+0000 7f6ee0c00640 1 -- 172.21.15.47:0/2592735538 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3379 (secure 0 0 0) 0x7f6edc018070 con 0x7f6ee415f180 2024-02-19T15:29:37.925 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:37.925 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:24.882579+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:37.925 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 98 2024-02-19T15:29:37.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.926+0000 7f6ee9e00640 1 -- 172.21.15.47:0/2592735538 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ec4066e40 msgr2=0x7f6ec4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:37.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.926+0000 7f6ee9e00640 1 --2- 172.21.15.47:0/2592735538 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ec4066e40 0x7f6ec4069300 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f6ee40b4fd0 tx=0x7f6ed8009190 comp rx=0 tx=0).stop 2024-02-19T15:29:37.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.926+0000 7f6ee9e00640 1 -- 172.21.15.47:0/2592735538 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ee415f180 msgr2=0x7f6ee40c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:37.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.926+0000 7f6ee9e00640 1 --2- 172.21.15.47:0/2592735538 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ee415f180 0x7f6ee40c74a0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f6edc00c7a0 tx=0x7f6edc00cc60 comp rx=0 tx=0).stop 2024-02-19T15:29:37.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.926+0000 7f6ee9e00640 1 -- 172.21.15.47:0/2592735538 shutdown_connections 2024-02-19T15:29:37.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.926+0000 7f6ee9e00640 1 --2- 172.21.15.47:0/2592735538 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ec4066e40 0x7f6ec4069300 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:37.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.926+0000 7f6ee9e00640 1 --2- 172.21.15.47:0/2592735538 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ee415f180 0x7f6ee40c74a0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:37.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.926+0000 7f6ee9e00640 1 --2- 172.21.15.47:0/2592735538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ee40ae2f0 0x7f6ee41569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:37.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.926+0000 7f6ee9e00640 1 --2- 172.21.15.47:0/2592735538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee4067100 0x7f6ee40c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:37.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.926+0000 7f6ee9e00640 1 -- 172.21.15.47:0/2592735538 >> 172.21.15.47:0/2592735538 conn(0x7f6ee40b1be0 msgr2=0x7f6ee415d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:37.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.926+0000 7f6ee9e00640 1 -- 172.21.15.47:0/2592735538 shutdown_connections 2024-02-19T15:29:37.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:37.926+0000 7f6ee9e00640 1 -- 172.21.15.47:0/2592735538 wait complete. 2024-02-19T15:29:38.147 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:24.882579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:38.147 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:38.176 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:29:39.148 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:39.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.322+0000 7f3868c00640 1 Processor -- start 2024-02-19T15:29:39.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.322+0000 7f3868c00640 1 -- start start 2024-02-19T15:29:39.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.322+0000 7f3868c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3864145590 0x7f38640727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:39.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.322+0000 7f3868c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f38640cea80 0x7f386406c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:39.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.322+0000 7f3868c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f386415d280 con 0x7f386415c390 2024-02-19T15:29:39.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.322+0000 7f3868c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3864070ff0 con 0x7f3864145590 2024-02-19T15:29:39.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.322+0000 7f3868c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38640709e0 con 0x7f38640cea80 2024-02-19T15:29:39.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.322+0000 7f3861e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3864145590 0x7f38640727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:39.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.322+0000 7f3861e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3864145590 0x7f38640727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56042/0 (socket says 172.21.15.47:56042) 2024-02-19T15:29:39.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.322+0000 7f3861e00640 1 -- 172.21.15.47:0/1392394913 learned_addr learned my addr 172.21.15.47:0/1392394913 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:39.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.322+0000 7f385be00640 1 --2- 172.21.15.47:0/1392394913 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f38640cea80 0x7f386406c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:39.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.322+0000 7f3861e00640 1 -- 172.21.15.47:0/1392394913 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f38640cea80 msgr2=0x7f386406c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:39.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.322+0000 7f3861e00640 1 --2- 172.21.15.47:0/1392394913 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f38640cea80 0x7f386406c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:39.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.322+0000 7f3861e00640 1 -- 172.21.15.47:0/1392394913 >> v1:172.21.15.116:6789/0 conn(0x7f386415c390 legacy=0x7f386415c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:39.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.322+0000 7f3861e00640 1 -- 172.21.15.47:0/1392394913 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f386406cee0 con 0x7f3864145590 2024-02-19T15:29:39.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.326+0000 7f3861e00640 1 --2- 172.21.15.47:0/1392394913 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3864145590 0x7f38640727b0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f38540098a0 tx=0x7f385401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc556c46e71d1405 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:39.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.326+0000 7f3861400640 1 -- 172.21.15.47:0/1392394913 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38540ab650 con 0x7f3864145590 2024-02-19T15:29:39.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.326+0000 7f3861400640 1 -- 172.21.15.47:0/1392394913 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38540ab7b0 con 0x7f3864145590 2024-02-19T15:29:39.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.326+0000 7f3861400640 1 -- 172.21.15.47:0/1392394913 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38540abb40 con 0x7f3864145590 2024-02-19T15:29:39.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.326+0000 7f3868c00640 1 -- 172.21.15.47:0/1392394913 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3864145590 msgr2=0x7f38640727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:39.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.326+0000 7f3868c00640 1 --2- 172.21.15.47:0/1392394913 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3864145590 0x7f38640727b0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f38540098a0 tx=0x7f385401c540 comp rx=0 tx=0).stop 2024-02-19T15:29:39.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.326+0000 7f3868c00640 1 -- 172.21.15.47:0/1392394913 shutdown_connections 2024-02-19T15:29:39.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.326+0000 7f3868c00640 1 --2- 172.21.15.47:0/1392394913 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f38640cea80 0x7f386406c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:39.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.326+0000 7f3868c00640 1 --2- 172.21.15.47:0/1392394913 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3864145590 0x7f38640727b0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:39.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.326+0000 7f3868c00640 1 -- 172.21.15.47:0/1392394913 >> 172.21.15.47:0/1392394913 conn(0x7f38640b16a0 msgr2=0x7f38640b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:39.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.326+0000 7f3868c00640 1 -- 172.21.15.47:0/1392394913 shutdown_connections 2024-02-19T15:29:39.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.326+0000 7f3868c00640 1 -- 172.21.15.47:0/1392394913 wait complete. 2024-02-19T15:29:39.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.330+0000 7f3868c00640 1 Processor -- start 2024-02-19T15:29:39.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.330+0000 7f3868c00640 1 -- start start 2024-02-19T15:29:39.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.330+0000 7f3868c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38640cea80 0x7f3864141280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:39.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.330+0000 7f3868c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3864145590 0x7f38640b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:39.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.330+0000 7f3868c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f386415c390 0x7f38640adae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:39.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.330+0000 7f3868c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3864140e60 con 0x7f38640cea80 2024-02-19T15:29:39.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.330+0000 7f3868c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38641401e0 con 0x7f3864145590 2024-02-19T15:29:39.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.330+0000 7f3868c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38641404c0 con 0x7f386415c390 2024-02-19T15:29:39.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.330+0000 7f3861e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38640cea80 0x7f3864141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:39.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.330+0000 7f3861e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38640cea80 0x7f3864141280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56046/0 (socket says 172.21.15.47:56046) 2024-02-19T15:29:39.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.330+0000 7f3861e00640 1 -- 172.21.15.47:0/3277082443 learned_addr learned my addr 172.21.15.47:0/3277082443 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:39.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.330+0000 7f385be00640 1 --2- 172.21.15.47:0/3277082443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3864145590 0x7f38640b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:39.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.330+0000 7f3862800640 1 --2- 172.21.15.47:0/3277082443 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f386415c390 0x7f38640adae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:39.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.334+0000 7f3861e00640 1 -- 172.21.15.47:0/3277082443 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f386415c390 msgr2=0x7f38640adae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:39.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.334+0000 7f3861e00640 1 --2- 172.21.15.47:0/3277082443 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f386415c390 0x7f38640adae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:39.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.334+0000 7f3861e00640 1 -- 172.21.15.47:0/3277082443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3864145590 msgr2=0x7f38640b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:39.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.334+0000 7f3861e00640 1 --2- 172.21.15.47:0/3277082443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3864145590 0x7f38640b40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:39.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.334+0000 7f3861e00640 1 -- 172.21.15.47:0/3277082443 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38540094d0 con 0x7f38640cea80 2024-02-19T15:29:39.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.334+0000 7f3861e00640 1 --2- 172.21.15.47:0/3277082443 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38640cea80 0x7f3864141280 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f38540ad040 tx=0x7f385401ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:39.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.334+0000 7f385a000640 1 -- 172.21.15.47:0/3277082443 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38540b4800 con 0x7f38640cea80 2024-02-19T15:29:39.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.334+0000 7f385a000640 1 -- 172.21.15.47:0/3277082443 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38540b4980 con 0x7f38640cea80 2024-02-19T15:29:39.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.334+0000 7f3868c00640 1 -- 172.21.15.47:0/3277082443 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38641407a0 con 0x7f38640cea80 2024-02-19T15:29:39.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.334+0000 7f385a000640 1 -- 172.21.15.47:0/3277082443 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38540b4d30 con 0x7f38640cea80 2024-02-19T15:29:39.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.334+0000 7f3868c00640 1 -- 172.21.15.47:0/3277082443 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38640b4610 con 0x7f38640cea80 2024-02-19T15:29:39.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.334+0000 7f3868c00640 1 -- 172.21.15.47:0/3277082443 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38641410d0 con 0x7f38640cea80 2024-02-19T15:29:39.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.334+0000 7f385a000640 1 -- 172.21.15.47:0/3277082443 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f38540c7390 con 0x7f38640cea80 2024-02-19T15:29:39.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.338+0000 7f385a000640 1 --2- 172.21.15.47:0/3277082443 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3840066e70 0x7f3840069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:39.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.338+0000 7f385a000640 1 -- 172.21.15.47:0/3277082443 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f385411f400 con 0x7f38640cea80 2024-02-19T15:29:39.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.338+0000 7f385be00640 1 --2- 172.21.15.47:0/3277082443 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3840066e70 0x7f3840069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:39.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.338+0000 7f385be00640 1 --2- 172.21.15.47:0/3277082443 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3840066e70 0x7f3840069330 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f38640b4c80 tx=0x7f384c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:39.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.342+0000 7f385a000640 1 -- 172.21.15.47:0/3277082443 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f38540ec3c0 con 0x7f38640cea80 2024-02-19T15:29:39.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.594+0000 7f3868c00640 1 -- 172.21.15.47:0/3277082443 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3864075360 con 0x7f38640cea80 2024-02-19T15:29:39.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.594+0000 7f385a000640 1 -- 172.21.15.47:0/3277082443 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3379 (secure 0 0 0) 0x7f38540efa80 con 0x7f38640cea80 2024-02-19T15:29:39.594 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:39.595 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:24.882579+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:39.595 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 98 2024-02-19T15:29:39.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.598+0000 7f3868c00640 1 -- 172.21.15.47:0/3277082443 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3840066e70 msgr2=0x7f3840069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:39.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.598+0000 7f3868c00640 1 --2- 172.21.15.47:0/3277082443 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3840066e70 0x7f3840069330 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f38640b4c80 tx=0x7f384c008040 comp rx=0 tx=0).stop 2024-02-19T15:29:39.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.598+0000 7f3868c00640 1 -- 172.21.15.47:0/3277082443 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38640cea80 msgr2=0x7f3864141280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:39.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.598+0000 7f3868c00640 1 --2- 172.21.15.47:0/3277082443 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38640cea80 0x7f3864141280 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f38540ad040 tx=0x7f385401ad80 comp rx=0 tx=0).stop 2024-02-19T15:29:39.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.598+0000 7f3868c00640 1 -- 172.21.15.47:0/3277082443 shutdown_connections 2024-02-19T15:29:39.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.598+0000 7f3868c00640 1 --2- 172.21.15.47:0/3277082443 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3840066e70 0x7f3840069330 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:39.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.598+0000 7f3868c00640 1 --2- 172.21.15.47:0/3277082443 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f386415c390 0x7f38640adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:39.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.598+0000 7f3868c00640 1 --2- 172.21.15.47:0/3277082443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3864145590 0x7f38640b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:39.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.598+0000 7f3868c00640 1 --2- 172.21.15.47:0/3277082443 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38640cea80 0x7f3864141280 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:39.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.598+0000 7f3868c00640 1 -- 172.21.15.47:0/3277082443 >> 172.21.15.47:0/3277082443 conn(0x7f38640b16a0 msgr2=0x7f38640b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:39.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.598+0000 7f3868c00640 1 -- 172.21.15.47:0/3277082443 shutdown_connections 2024-02-19T15:29:39.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:39.598+0000 7f3868c00640 1 -- 172.21.15.47:0/3277082443 wait complete. 2024-02-19T15:29:39.818 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:24.882579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:39.819 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:40.178 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:29:40.819 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:40.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.958+0000 7f829c000640 1 Processor -- start 2024-02-19T15:29:40.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.958+0000 7f829c000640 1 -- start start 2024-02-19T15:29:40.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.962+0000 7f829c000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f829415bcd0 0x7f82940cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:40.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.962+0000 7f829c000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82940cc700 con 0x7f82940cedb0 2024-02-19T15:29:40.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.962+0000 7f829c000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82940cf300 con 0x7f82940d0360 2024-02-19T15:29:40.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.962+0000 7f829c000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82940cf5e0 con 0x7f829415bcd0 2024-02-19T15:29:40.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.962+0000 7f8299600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f829415bcd0 0x7f82940cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:40.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.962+0000 7f829a000640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f82940d0360 0x7f829415d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:42106/0 (socket says 172.21.15.47:42106) 2024-02-19T15:29:40.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.962+0000 7f829a000640 1 -- 172.21.15.47:0/1025802942 learned_addr learned my addr 172.21.15.47:0/1025802942 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:29:40.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.962+0000 7f8298c00640 1 --1- 172.21.15.47:0/1025802942 >> v1:172.21.15.47:6789/0 conn(0x7f82940cedb0 0x7f82940cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39144/0 (socket says 172.21.15.47:39144) 2024-02-19T15:29:40.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.962+0000 7f8299600640 1 -- 172.21.15.47:0/1025802942 >> v1:172.21.15.116:6789/0 conn(0x7f82940d0360 legacy=0x7f829415d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:40.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.962+0000 7f8299600640 1 -- 172.21.15.47:0/1025802942 >> v1:172.21.15.47:6789/0 conn(0x7f82940cedb0 legacy=0x7f82940cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:40.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.962+0000 7f8299600640 1 -- 172.21.15.47:0/1025802942 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82940cc9a0 con 0x7f829415bcd0 2024-02-19T15:29:40.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.962+0000 7f8293e00640 1 -- 172.21.15.47:0/1025802942 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772244239 0 0) 0x7f82940cc700 con 0x7f82940cedb0 2024-02-19T15:29:40.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.966+0000 7f8299600640 1 --2- 172.21.15.47:0/1025802942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f829415bcd0 0x7f82940cd480 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f82880097c0 tx=0x7f828801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c5cab05be4df0e4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:40.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.966+0000 7f8293e00640 1 -- 172.21.15.47:0/1025802942 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82880ab670 con 0x7f829415bcd0 2024-02-19T15:29:40.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.966+0000 7f8293e00640 1 -- 172.21.15.47:0/1025802942 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82880ab7d0 con 0x7f829415bcd0 2024-02-19T15:29:40.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.966+0000 7f8293e00640 1 -- 172.21.15.47:0/1025802942 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82880abb60 con 0x7f829415bcd0 2024-02-19T15:29:40.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.966+0000 7f829c000640 1 -- 172.21.15.47:0/1025802942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f829415bcd0 msgr2=0x7f82940cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:40.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.966+0000 7f829c000640 1 --2- 172.21.15.47:0/1025802942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f829415bcd0 0x7f82940cd480 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f82880097c0 tx=0x7f828801c470 comp rx=0 tx=0).stop 2024-02-19T15:29:40.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.966+0000 7f829c000640 1 -- 172.21.15.47:0/1025802942 shutdown_connections 2024-02-19T15:29:40.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.966+0000 7f829c000640 1 --2- 172.21.15.47:0/1025802942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f829415bcd0 0x7f82940cd480 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:40.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.966+0000 7f829c000640 1 -- 172.21.15.47:0/1025802942 >> 172.21.15.47:0/1025802942 conn(0x7f82940d1bf0 msgr2=0x7f82940d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:40.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.966+0000 7f829c000640 1 -- 172.21.15.47:0/1025802942 shutdown_connections 2024-02-19T15:29:40.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.966+0000 7f829c000640 1 -- 172.21.15.47:0/1025802942 wait complete. 2024-02-19T15:29:40.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.970+0000 7f829c000640 1 Processor -- start 2024-02-19T15:29:40.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.970+0000 7f829c000640 1 -- start start 2024-02-19T15:29:40.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.970+0000 7f829c000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82940cedb0 0x7f829414d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:40.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f829c000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82940d0360 0x7f8294155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:40.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f829c000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f829415bcd0 0x7f82940b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:40.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f829c000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82940b7b40 con 0x7f82940cedb0 2024-02-19T15:29:40.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f829c000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829414dd20 con 0x7f82940d0360 2024-02-19T15:29:40.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f829c000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829414e000 con 0x7f829415bcd0 2024-02-19T15:29:40.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f8299600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82940cedb0 0x7f829414d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:40.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f8299600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82940cedb0 0x7f829414d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56060/0 (socket says 172.21.15.47:56060) 2024-02-19T15:29:40.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f8299600640 1 -- 172.21.15.47:0/4171422048 learned_addr learned my addr 172.21.15.47:0/4171422048 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:40.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f8298c00640 1 --2- 172.21.15.47:0/4171422048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82940d0360 0x7f8294155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:40.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f829a000640 1 --2- 172.21.15.47:0/4171422048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f829415bcd0 0x7f82940b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:40.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f8299600640 1 -- 172.21.15.47:0/4171422048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f829415bcd0 msgr2=0x7f82940b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:40.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f8299600640 1 --2- 172.21.15.47:0/4171422048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f829415bcd0 0x7f82940b4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:40.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f8299600640 1 -- 172.21.15.47:0/4171422048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82940d0360 msgr2=0x7f8294155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:40.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f8299600640 1 --2- 172.21.15.47:0/4171422048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82940d0360 0x7f8294155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:40.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f8299600640 1 -- 172.21.15.47:0/4171422048 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8288009420 con 0x7f82940cedb0 2024-02-19T15:29:40.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f8298c00640 1 --2- 172.21.15.47:0/4171422048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82940d0360 0x7f8294155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:29:40.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f829a000640 1 --2- 172.21.15.47:0/4171422048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f829415bcd0 0x7f82940b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:29:40.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f8299600640 1 --2- 172.21.15.47:0/4171422048 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82940cedb0 0x7f829414d6f0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f82880ad040 tx=0x7f828801acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:40.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f8291600640 1 -- 172.21.15.47:0/4171422048 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82880b5810 con 0x7f82940cedb0 2024-02-19T15:29:40.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f8291600640 1 -- 172.21.15.47:0/4171422048 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82880b5990 con 0x7f82940cedb0 2024-02-19T15:29:40.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f8291600640 1 -- 172.21.15.47:0/4171422048 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82880b5d40 con 0x7f82940cedb0 2024-02-19T15:29:40.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f829c000640 1 -- 172.21.15.47:0/4171422048 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82940cdb40 con 0x7f82940cedb0 2024-02-19T15:29:40.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f829c000640 1 -- 172.21.15.47:0/4171422048 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f829414e5c0 con 0x7f82940cedb0 2024-02-19T15:29:40.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f829c000640 1 -- 172.21.15.47:0/4171422048 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8294075360 con 0x7f82940cedb0 2024-02-19T15:29:40.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.974+0000 7f8291600640 1 -- 172.21.15.47:0/4171422048 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f82880b4390 con 0x7f82940cedb0 2024-02-19T15:29:40.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.978+0000 7f8291600640 1 --2- 172.21.15.47:0/4171422048 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8274066e40 0x7f8274069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:40.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.978+0000 7f8291600640 1 -- 172.21.15.47:0/4171422048 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f828811f670 con 0x7f82940cedb0 2024-02-19T15:29:40.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.978+0000 7f8298c00640 1 --2- 172.21.15.47:0/4171422048 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8274066e40 0x7f8274069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:40.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.978+0000 7f8298c00640 1 --2- 172.21.15.47:0/4171422048 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8274066e40 0x7f8274069300 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f829414cc90 tx=0x7f8280007dd0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:40.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:40.982+0000 7f8291600640 1 -- 172.21.15.47:0/4171422048 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f82880ec4e0 con 0x7f82940cedb0 2024-02-19T15:29:41.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:41.178+0000 7f829c000640 1 -- 172.21.15.47:0/4171422048 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f829406c860 con 0x7f82940cedb0 2024-02-19T15:29:41.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:41.178+0000 7f8291600640 1 -- 172.21.15.47:0/4171422048 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3379 (secure 0 0 0) 0x7f82880efba0 con 0x7f82940cedb0 2024-02-19T15:29:41.178 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:41.178 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:24.882579+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:41.178 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 98 2024-02-19T15:29:41.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:41.178+0000 7f829c000640 1 -- 172.21.15.47:0/4171422048 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8274066e40 msgr2=0x7f8274069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:41.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:41.178+0000 7f829c000640 1 --2- 172.21.15.47:0/4171422048 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8274066e40 0x7f8274069300 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f829414cc90 tx=0x7f8280007dd0 comp rx=0 tx=0).stop 2024-02-19T15:29:41.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:41.178+0000 7f829c000640 1 -- 172.21.15.47:0/4171422048 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82940cedb0 msgr2=0x7f829414d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:41.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:41.178+0000 7f829c000640 1 --2- 172.21.15.47:0/4171422048 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82940cedb0 0x7f829414d6f0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f82880ad040 tx=0x7f828801acb0 comp rx=0 tx=0).stop 2024-02-19T15:29:41.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:41.178+0000 7f829c000640 1 -- 172.21.15.47:0/4171422048 shutdown_connections 2024-02-19T15:29:41.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:41.178+0000 7f829c000640 1 --2- 172.21.15.47:0/4171422048 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8274066e40 0x7f8274069300 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:41.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:41.178+0000 7f829c000640 1 --2- 172.21.15.47:0/4171422048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f829415bcd0 0x7f82940b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:41.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:41.178+0000 7f829c000640 1 --2- 172.21.15.47:0/4171422048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82940d0360 0x7f8294155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:41.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:41.178+0000 7f829c000640 1 --2- 172.21.15.47:0/4171422048 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82940cedb0 0x7f829414d6f0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:41.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:41.178+0000 7f829c000640 1 -- 172.21.15.47:0/4171422048 >> 172.21.15.47:0/4171422048 conn(0x7f82940d1bf0 msgr2=0x7f829406d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:41.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:41.178+0000 7f829c000640 1 -- 172.21.15.47:0/4171422048 shutdown_connections 2024-02-19T15:29:41.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:41.178+0000 7f829c000640 1 -- 172.21.15.47:0/4171422048 wait complete. 2024-02-19T15:29:41.401 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:24.882579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:41.401 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:42.180 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:29:42.402 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:42.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.542+0000 7fcc08e00640 1 Processor -- start 2024-02-19T15:29:42.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.542+0000 7fcc08e00640 1 -- start start 2024-02-19T15:29:42.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.546+0000 7fcc08e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcc0415f620 0x7fcc040b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:42.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.546+0000 7fcc08e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc040b2330 0x7fcc0406ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:42.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.546+0000 7fcc08e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0406b5d0 con 0x7fcc041614e0 2024-02-19T15:29:42.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.546+0000 7fcc08e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc040b1c00 con 0x7fcc0415f620 2024-02-19T15:29:42.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.546+0000 7fcc08e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc040b27e0 con 0x7fcc040b2330 2024-02-19T15:29:42.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.546+0000 7fcbfbe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcc0415f620 0x7fcc040b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:42.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.546+0000 7fcbfbe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcc0415f620 0x7fcc040b1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56070/0 (socket says 172.21.15.47:56070) 2024-02-19T15:29:42.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.546+0000 7fcbfbe00640 1 -- 172.21.15.47:0/1949094927 learned_addr learned my addr 172.21.15.47:0/1949094927 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:42.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.546+0000 7fcc01e00640 1 --1- 172.21.15.47:0/1949094927 >> v1:172.21.15.47:6789/0 conn(0x7fcc041614e0 0x7fcc0415eee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39146/0 (socket says 172.21.15.47:39146) 2024-02-19T15:29:42.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.546+0000 7fcc02800640 1 --2- 172.21.15.47:0/1949094927 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc040b2330 0x7fcc0406ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:42.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.546+0000 7fcbfbe00640 1 -- 172.21.15.47:0/1949094927 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc040b2330 msgr2=0x7fcc0406ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:42.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.546+0000 7fcbfbe00640 1 --2- 172.21.15.47:0/1949094927 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc040b2330 0x7fcc0406ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:42.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.546+0000 7fcbfbe00640 1 -- 172.21.15.47:0/1949094927 >> v1:172.21.15.47:6789/0 conn(0x7fcc041614e0 legacy=0x7fcc0415eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:42.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.546+0000 7fcbfbe00640 1 -- 172.21.15.47:0/1949094927 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc040b1640 con 0x7fcc0415f620 2024-02-19T15:29:42.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.546+0000 7fcc01400640 1 -- 172.21.15.47:0/1949094927 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1304599826 0 0) 0x7fcc0406b5d0 con 0x7fcc041614e0 2024-02-19T15:29:42.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.550+0000 7fcbfbe00640 1 --2- 172.21.15.47:0/1949094927 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc0415f620 0x7fcc040b1df0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fcbec009870 tx=0x7fcbec01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d2acc9b55131705 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:42.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.550+0000 7fcc01400640 1 -- 172.21.15.47:0/1949094927 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec0a9620 con 0x7fcc0415f620 2024-02-19T15:29:42.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.550+0000 7fcc01400640 1 -- 172.21.15.47:0/1949094927 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbec0a9780 con 0x7fcc0415f620 2024-02-19T15:29:42.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.550+0000 7fcc01400640 1 -- 172.21.15.47:0/1949094927 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec0a9b10 con 0x7fcc0415f620 2024-02-19T15:29:42.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.550+0000 7fcc08e00640 1 -- 172.21.15.47:0/1949094927 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc0415f620 msgr2=0x7fcc040b1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:42.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.550+0000 7fcc08e00640 1 --2- 172.21.15.47:0/1949094927 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc0415f620 0x7fcc040b1df0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fcbec009870 tx=0x7fcbec01c560 comp rx=0 tx=0).stop 2024-02-19T15:29:42.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.550+0000 7fcc08e00640 1 -- 172.21.15.47:0/1949094927 shutdown_connections 2024-02-19T15:29:42.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.550+0000 7fcc08e00640 1 --2- 172.21.15.47:0/1949094927 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc040b2330 0x7fcc0406ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:42.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.550+0000 7fcc08e00640 1 --2- 172.21.15.47:0/1949094927 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc0415f620 0x7fcc040b1df0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:42.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.550+0000 7fcc08e00640 1 -- 172.21.15.47:0/1949094927 >> 172.21.15.47:0/1949094927 conn(0x7fcc040b4fc0 msgr2=0x7fcc040b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:42.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.550+0000 7fcc08e00640 1 -- 172.21.15.47:0/1949094927 shutdown_connections 2024-02-19T15:29:42.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.550+0000 7fcc08e00640 1 -- 172.21.15.47:0/1949094927 wait complete. 2024-02-19T15:29:42.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.550+0000 7fcc08e00640 1 Processor -- start 2024-02-19T15:29:42.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.550+0000 7fcc08e00640 1 -- start start 2024-02-19T15:29:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.550+0000 7fcc08e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc040b2330 0x7fcc041588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.550+0000 7fcc08e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc0415f620 0x7fcc04158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.550+0000 7fcc08e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc041614e0 0x7fcc040d04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.550+0000 7fcc08e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc040cf710 con 0x7fcc041614e0 2024-02-19T15:29:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.550+0000 7fcc08e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc040cf9f0 con 0x7fcc0415f620 2024-02-19T15:29:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.550+0000 7fcc08e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc040cfcd0 con 0x7fcc040b2330 2024-02-19T15:29:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.554+0000 7fcc02800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc041614e0 0x7fcc040d04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.554+0000 7fcc02800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc041614e0 0x7fcc040d04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56086/0 (socket says 172.21.15.47:56086) 2024-02-19T15:29:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.554+0000 7fcc02800640 1 -- 172.21.15.47:0/543576027 learned_addr learned my addr 172.21.15.47:0/543576027 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.554+0000 7fcc01e00640 1 --2- 172.21.15.47:0/543576027 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc040b2330 0x7fcc041588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:42.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.554+0000 7fcbfbe00640 1 --2- 172.21.15.47:0/543576027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc0415f620 0x7fcc04158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:42.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.554+0000 7fcc02800640 1 -- 172.21.15.47:0/543576027 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc040b2330 msgr2=0x7fcc041588d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:42.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.554+0000 7fcc02800640 1 --2- 172.21.15.47:0/543576027 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc040b2330 0x7fcc041588d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:42.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.554+0000 7fcc02800640 1 -- 172.21.15.47:0/543576027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc0415f620 msgr2=0x7fcc04158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:42.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.554+0000 7fcc02800640 1 --2- 172.21.15.47:0/543576027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc0415f620 0x7fcc04158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:42.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.554+0000 7fcc02800640 1 -- 172.21.15.47:0/543576027 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbec0094d0 con 0x7fcc041614e0 2024-02-19T15:29:42.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.554+0000 7fcc02800640 1 --2- 172.21.15.47:0/543576027 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc041614e0 0x7fcc040d04d0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc007e40 tx=0x7fcbfc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:42.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.554+0000 7fcbfa000640 1 -- 172.21.15.47:0/543576027 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc014620 con 0x7fcc041614e0 2024-02-19T15:29:42.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.554+0000 7fcbfa000640 1 -- 172.21.15.47:0/543576027 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbfc00ee90 con 0x7fcc041614e0 2024-02-19T15:29:42.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.554+0000 7fcc08e00640 1 -- 172.21.15.47:0/543576027 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc040cffb0 con 0x7fcc041614e0 2024-02-19T15:29:42.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.554+0000 7fcbfa000640 1 -- 172.21.15.47:0/543576027 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc0149e0 con 0x7fcc041614e0 2024-02-19T15:29:42.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.554+0000 7fcc08e00640 1 -- 172.21.15.47:0/543576027 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc041585b0 con 0x7fcc041614e0 2024-02-19T15:29:42.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.554+0000 7fcc08e00640 1 -- 172.21.15.47:0/543576027 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc040d01f0 con 0x7fcc041614e0 2024-02-19T15:29:42.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.554+0000 7fcbfa000640 1 -- 172.21.15.47:0/543576027 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fcbfc014b40 con 0x7fcc041614e0 2024-02-19T15:29:42.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.558+0000 7fcbfa000640 1 --2- 172.21.15.47:0/543576027 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcbe0066e70 0x7fcbe0069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:42.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.558+0000 7fcbfa000640 1 -- 172.21.15.47:0/543576027 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcbfc085300 con 0x7fcc041614e0 2024-02-19T15:29:42.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.558+0000 7fcc01e00640 1 --2- 172.21.15.47:0/543576027 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcbe0066e70 0x7fcbe0069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:42.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.558+0000 7fcc01e00640 1 --2- 172.21.15.47:0/543576027 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcbe0066e70 0x7fcbe0069330 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4000f70 tx=0x7fcbf4007df0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:42.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.562+0000 7fcbfa000640 1 -- 172.21.15.47:0/543576027 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fcc040d01f0 con 0x7fcc041614e0 2024-02-19T15:29:42.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.754+0000 7fcc08e00640 1 -- 172.21.15.47:0/543576027 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc040d01f0 con 0x7fcc041614e0 2024-02-19T15:29:42.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.754+0000 7fcbfa000640 1 -- 172.21.15.47:0/543576027 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3379 (secure 0 0 0) 0x7fcc040d01f0 con 0x7fcc041614e0 2024-02-19T15:29:42.754 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:42.754 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:24.882579+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:42.754 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 98 2024-02-19T15:29:42.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.758+0000 7fcc08e00640 1 -- 172.21.15.47:0/543576027 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcbe0066e70 msgr2=0x7fcbe0069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:42.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.758+0000 7fcc08e00640 1 --2- 172.21.15.47:0/543576027 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcbe0066e70 0x7fcbe0069330 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4000f70 tx=0x7fcbf4007df0 comp rx=0 tx=0).stop 2024-02-19T15:29:42.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.758+0000 7fcc08e00640 1 -- 172.21.15.47:0/543576027 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc041614e0 msgr2=0x7fcc040d04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:42.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.758+0000 7fcc08e00640 1 --2- 172.21.15.47:0/543576027 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc041614e0 0x7fcc040d04d0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc007e40 tx=0x7fcbfc00e4b0 comp rx=0 tx=0).stop 2024-02-19T15:29:42.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.758+0000 7fcc08e00640 1 -- 172.21.15.47:0/543576027 shutdown_connections 2024-02-19T15:29:42.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.758+0000 7fcc08e00640 1 --2- 172.21.15.47:0/543576027 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcbe0066e70 0x7fcbe0069330 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:42.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.758+0000 7fcc08e00640 1 --2- 172.21.15.47:0/543576027 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc041614e0 0x7fcc040d04d0 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:42.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.758+0000 7fcc08e00640 1 --2- 172.21.15.47:0/543576027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc0415f620 0x7fcc04158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:42.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.758+0000 7fcc08e00640 1 --2- 172.21.15.47:0/543576027 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc040b2330 0x7fcc041588d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:42.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.758+0000 7fcc08e00640 1 -- 172.21.15.47:0/543576027 >> 172.21.15.47:0/543576027 conn(0x7fcc040b4fc0 msgr2=0x7fcc04160900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:42.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.758+0000 7fcc08e00640 1 -- 172.21.15.47:0/543576027 shutdown_connections 2024-02-19T15:29:42.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:42.758+0000 7fcc08e00640 1 -- 172.21.15.47:0/543576027 wait complete. 2024-02-19T15:29:42.977 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:24.882579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:42.977 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:43.979 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:44.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.118+0000 7f7ee3a00640 1 Processor -- start 2024-02-19T15:29:44.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.118+0000 7f7ee3a00640 1 -- start start 2024-02-19T15:29:44.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.122+0000 7f7ee3a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7edc0ceb70 0x7f7edc0d0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:44.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.122+0000 7f7ee3a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7edc0d26b0 0x7f7edc0b5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:44.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.122+0000 7f7ee3a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7edc0cdae0 con 0x7f7edc15be90 2024-02-19T15:29:44.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.122+0000 7f7ee3a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7edc0d0af0 con 0x7f7edc0ceb70 2024-02-19T15:29:44.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.122+0000 7f7ee3a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7edc0d0810 con 0x7f7edc0d26b0 2024-02-19T15:29:44.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.122+0000 7f7ed3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7edc0ceb70 0x7f7edc0d0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:44.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.122+0000 7f7ed3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7edc0ceb70 0x7f7edc0d0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43576/0 (socket says 172.21.15.47:43576) 2024-02-19T15:29:44.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.122+0000 7f7ed3e00640 1 -- 172.21.15.47:0/1158050862 learned_addr learned my addr 172.21.15.47:0/1158050862 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:44.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.122+0000 7f7ee1a00640 1 --2- 172.21.15.47:0/1158050862 >> v2:172.21.15.116:3300/0 conn(0x7f7edc0d26b0 0x7f7edc0b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:44.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.122+0000 7f7ed3e00640 1 -- 172.21.15.47:0/1158050862 >> v2:172.21.15.116:3300/0 conn(0x7f7edc0d26b0 msgr2=0x7f7edc0b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:44.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.122+0000 7f7ed3e00640 1 --2- 172.21.15.47:0/1158050862 >> v2:172.21.15.116:3300/0 conn(0x7f7edc0d26b0 0x7f7edc0b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:44.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.122+0000 7f7ed3e00640 1 -- 172.21.15.47:0/1158050862 >> v1:172.21.15.116:6789/0 conn(0x7f7edc15be90 legacy=0x7f7edc0ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:44.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.122+0000 7f7ed3e00640 1 -- 172.21.15.47:0/1158050862 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7edc0b4a80 con 0x7f7edc0ceb70 2024-02-19T15:29:44.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.126+0000 7f7ed3e00640 1 --2- 172.21.15.47:0/1158050862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7edc0ceb70 0x7f7edc0d0170 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f7edc0c53b0 tx=0x7f7ec401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d93c3a869d5f6f58 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:44.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.126+0000 7f7edbe00640 1 -- 172.21.15.47:0/1158050862 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec40aa610 con 0x7f7edc0ceb70 2024-02-19T15:29:44.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.126+0000 7f7edbe00640 1 -- 172.21.15.47:0/1158050862 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec40a6040 con 0x7f7edc0ceb70 2024-02-19T15:29:44.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.126+0000 7f7edbe00640 1 -- 172.21.15.47:0/1158050862 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec40aa990 con 0x7f7edc0ceb70 2024-02-19T15:29:44.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.126+0000 7f7ee3a00640 1 -- 172.21.15.47:0/1158050862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7edc0ceb70 msgr2=0x7f7edc0d0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:44.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.126+0000 7f7ee3a00640 1 --2- 172.21.15.47:0/1158050862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7edc0ceb70 0x7f7edc0d0170 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f7edc0c53b0 tx=0x7f7ec401c4b0 comp rx=0 tx=0).stop 2024-02-19T15:29:44.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.126+0000 7f7ee3a00640 1 -- 172.21.15.47:0/1158050862 shutdown_connections 2024-02-19T15:29:44.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.126+0000 7f7ee3a00640 1 --2- 172.21.15.47:0/1158050862 >> v2:172.21.15.116:3300/0 conn(0x7f7edc0d26b0 0x7f7edc0b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:44.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.126+0000 7f7ee3a00640 1 --2- 172.21.15.47:0/1158050862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7edc0ceb70 0x7f7edc0d0170 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:44.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.126+0000 7f7ee3a00640 1 -- 172.21.15.47:0/1158050862 >> 172.21.15.47:0/1158050862 conn(0x7f7edc0d1b90 msgr2=0x7f7edc0d1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:44.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.126+0000 7f7ee3a00640 1 -- 172.21.15.47:0/1158050862 shutdown_connections 2024-02-19T15:29:44.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.126+0000 7f7ee3a00640 1 -- 172.21.15.47:0/1158050862 wait complete. 2024-02-19T15:29:44.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.126+0000 7f7ee3a00640 1 Processor -- start 2024-02-19T15:29:44.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.126+0000 7f7ee3a00640 1 -- start start 2024-02-19T15:29:44.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.126+0000 7f7ee3a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7edc0ceb70 0x7f7edc14b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:44.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.126+0000 7f7ee3a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7edc0d26b0 0x7f7edc06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:44.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ee3a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7edc15be90 0x7f7edc15acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:44.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ee3a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc06d9b0 con 0x7f7edc0ceb70 2024-02-19T15:29:44.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ee3a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc14b710 con 0x7f7edc0d26b0 2024-02-19T15:29:44.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ee3a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc15a830 con 0x7f7edc15be90 2024-02-19T15:29:44.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ee1000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7edc0ceb70 0x7f7edc14b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:44.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ee1000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7edc0ceb70 0x7f7edc14b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43588/0 (socket says 172.21.15.47:43588) 2024-02-19T15:29:44.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ee1000640 1 -- 172.21.15.47:0/2407912113 learned_addr learned my addr 172.21.15.47:0/2407912113 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:44.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ed3e00640 1 --2- 172.21.15.47:0/2407912113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7edc0d26b0 0x7f7edc06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:44.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ee1a00640 1 --2- 172.21.15.47:0/2407912113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7edc15be90 0x7f7edc15acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:44.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ee1000640 1 -- 172.21.15.47:0/2407912113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7edc15be90 msgr2=0x7f7edc15acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:44.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ee1000640 1 --2- 172.21.15.47:0/2407912113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7edc15be90 0x7f7edc15acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:44.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ee1000640 1 -- 172.21.15.47:0/2407912113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7edc0d26b0 msgr2=0x7f7edc06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:44.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ee1000640 1 --2- 172.21.15.47:0/2407912113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7edc0d26b0 0x7f7edc06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:44.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ee1000640 1 -- 172.21.15.47:0/2407912113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec40094d0 con 0x7f7edc0ceb70 2024-02-19T15:29:44.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ee1a00640 1 --2- 172.21.15.47:0/2407912113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7edc15be90 0x7f7edc15acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:29:44.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ee1000640 1 --2- 172.21.15.47:0/2407912113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7edc0ceb70 0x7f7edc14b1d0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc00b440 tx=0x7f7ecc00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:44.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ed9600640 1 -- 172.21.15.47:0/2407912113 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc011890 con 0x7f7edc0ceb70 2024-02-19T15:29:44.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ed9600640 1 -- 172.21.15.47:0/2407912113 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ecc0119f0 con 0x7f7edc0ceb70 2024-02-19T15:29:44.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ed9600640 1 -- 172.21.15.47:0/2407912113 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc011da0 con 0x7f7edc0ceb70 2024-02-19T15:29:44.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ee3a00640 1 -- 172.21.15.47:0/2407912113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7edc06cf10 con 0x7f7edc0ceb70 2024-02-19T15:29:44.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ee3a00640 1 -- 172.21.15.47:0/2407912113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7edc06d6d0 con 0x7f7edc0ceb70 2024-02-19T15:29:44.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ee3a00640 1 -- 172.21.15.47:0/2407912113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7edc068500 con 0x7f7edc0ceb70 2024-02-19T15:29:44.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.130+0000 7f7ed9600640 1 -- 172.21.15.47:0/2407912113 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7ecc01a400 con 0x7f7edc0ceb70 2024-02-19T15:29:44.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.134+0000 7f7ed9600640 1 --2- 172.21.15.47:0/2407912113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7eb8066e40 0x7f7eb8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:44.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.134+0000 7f7ed9600640 1 -- 172.21.15.47:0/2407912113 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ecc084a40 con 0x7f7edc0ceb70 2024-02-19T15:29:44.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.134+0000 7f7ed3e00640 1 --2- 172.21.15.47:0/2407912113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7eb8066e40 0x7f7eb8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:44.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.134+0000 7f7ed3e00640 1 --2- 172.21.15.47:0/2407912113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7eb8066e40 0x7f7eb8069300 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f7ec40ae000 tx=0x7f7ec40b5040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:44.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.138+0000 7f7ed9600640 1 -- 172.21.15.47:0/2407912113 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7ecc051820 con 0x7f7edc0ceb70 2024-02-19T15:29:44.180 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:29:44.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.334+0000 7f7ee3a00640 1 -- 172.21.15.47:0/2407912113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7edc0cd2c0 con 0x7f7edc0ceb70 2024-02-19T15:29:44.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.334+0000 7f7ed9600640 1 -- 172.21.15.47:0/2407912113 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3379 (secure 0 0 0) 0x7f7ecc015090 con 0x7f7edc0ceb70 2024-02-19T15:29:44.333 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:44.334 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:24.882579+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:44.334 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 98 2024-02-19T15:29:44.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.334+0000 7f7ee3a00640 1 -- 172.21.15.47:0/2407912113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7eb8066e40 msgr2=0x7f7eb8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:44.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.334+0000 7f7ee3a00640 1 --2- 172.21.15.47:0/2407912113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7eb8066e40 0x7f7eb8069300 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f7ec40ae000 tx=0x7f7ec40b5040 comp rx=0 tx=0).stop 2024-02-19T15:29:44.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.334+0000 7f7ee3a00640 1 -- 172.21.15.47:0/2407912113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7edc0ceb70 msgr2=0x7f7edc14b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:44.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.334+0000 7f7ee3a00640 1 --2- 172.21.15.47:0/2407912113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7edc0ceb70 0x7f7edc14b1d0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc00b440 tx=0x7f7ecc00b900 comp rx=0 tx=0).stop 2024-02-19T15:29:44.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.334+0000 7f7ee3a00640 1 -- 172.21.15.47:0/2407912113 shutdown_connections 2024-02-19T15:29:44.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.334+0000 7f7ee3a00640 1 --2- 172.21.15.47:0/2407912113 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7eb8066e40 0x7f7eb8069300 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:44.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.334+0000 7f7ee3a00640 1 --2- 172.21.15.47:0/2407912113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7edc15be90 0x7f7edc15acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:44.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.334+0000 7f7ee3a00640 1 --2- 172.21.15.47:0/2407912113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7edc0d26b0 0x7f7edc06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:44.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.334+0000 7f7ee3a00640 1 --2- 172.21.15.47:0/2407912113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7edc0ceb70 0x7f7edc14b1d0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:44.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.334+0000 7f7ee3a00640 1 -- 172.21.15.47:0/2407912113 >> 172.21.15.47:0/2407912113 conn(0x7f7edc0d1b90 msgr2=0x7f7edc1547c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:44.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.334+0000 7f7ee3a00640 1 -- 172.21.15.47:0/2407912113 shutdown_connections 2024-02-19T15:29:44.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:44.334+0000 7f7ee3a00640 1 -- 172.21.15.47:0/2407912113 wait complete. 2024-02-19T15:29:44.556 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:24.882579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:44.556 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:45.557 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:45.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.698+0000 7f57f5a00640 1 Processor -- start 2024-02-19T15:29:45.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.698+0000 7f57f5a00640 1 -- start start 2024-02-19T15:29:45.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.698+0000 7f57f5a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f57f006d650 0x7f57f00d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:45.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.698+0000 7f57f5a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57f00ced10 0x7f57f006bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:45.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.698+0000 7f57f5a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f00cce30 con 0x7f57f00cded0 2024-02-19T15:29:45.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.698+0000 7f57f5a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57f00cd110 con 0x7f57f006d650 2024-02-19T15:29:45.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.698+0000 7f57f5a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f00cd3f0 con 0x7f57f00ced10 2024-02-19T15:29:45.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.698+0000 7f57eea00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f57f006d650 0x7f57f00d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:45.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.698+0000 7f57ef400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57f00ced10 0x7f57f006bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:45.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.698+0000 7f57eea00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f57f006d650 0x7f57f00d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:41946/0 (socket says 172.21.15.47:41946) 2024-02-19T15:29:45.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.698+0000 7f57eea00640 1 -- 172.21.15.47:0/1902644918 learned_addr learned my addr 172.21.15.47:0/1902644918 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:45.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.698+0000 7f57eea00640 1 -- 172.21.15.47:0/1902644918 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57f00ced10 msgr2=0x7f57f006bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:45.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.698+0000 7f57eea00640 1 --2- 172.21.15.47:0/1902644918 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57f00ced10 0x7f57f006bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:45.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.698+0000 7f57eea00640 1 -- 172.21.15.47:0/1902644918 >> v1:172.21.15.116:6789/0 conn(0x7f57f00cded0 legacy=0x7f57f00ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:45.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.698+0000 7f57eea00640 1 -- 172.21.15.47:0/1902644918 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f00cd6d0 con 0x7f57f006d650 2024-02-19T15:29:45.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.702+0000 7f57eea00640 1 --2- 172.21.15.47:0/1902644918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57f006d650 0x7f57f00d08b0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f57e0009870 tx=0x7f57e001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1f0c1662291dd32 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.702+0000 7f57ee000640 1 -- 172.21.15.47:0/1902644918 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e00aa620 con 0x7f57f006d650 2024-02-19T15:29:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.702+0000 7f57ee000640 1 -- 172.21.15.47:0/1902644918 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e00aa780 con 0x7f57f006d650 2024-02-19T15:29:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.702+0000 7f57ee000640 1 -- 172.21.15.47:0/1902644918 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e00aab10 con 0x7f57f006d650 2024-02-19T15:29:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.702+0000 7f57f5a00640 1 -- 172.21.15.47:0/1902644918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57f006d650 msgr2=0x7f57f00d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.702+0000 7f57f5a00640 1 --2- 172.21.15.47:0/1902644918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57f006d650 0x7f57f00d08b0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f57e0009870 tx=0x7f57e001c4a0 comp rx=0 tx=0).stop 2024-02-19T15:29:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.702+0000 7f57f5a00640 1 -- 172.21.15.47:0/1902644918 shutdown_connections 2024-02-19T15:29:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.702+0000 7f57f5a00640 1 --2- 172.21.15.47:0/1902644918 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57f00ced10 0x7f57f006bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.702+0000 7f57f5a00640 1 --2- 172.21.15.47:0/1902644918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57f006d650 0x7f57f00d08b0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.702+0000 7f57f5a00640 1 -- 172.21.15.47:0/1902644918 >> 172.21.15.47:0/1902644918 conn(0x7f57f00d1bf0 msgr2=0x7f57f00d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.702+0000 7f57f5a00640 1 -- 172.21.15.47:0/1902644918 shutdown_connections 2024-02-19T15:29:45.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.702+0000 7f57f5a00640 1 -- 172.21.15.47:0/1902644918 wait complete. 2024-02-19T15:29:45.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57f5a00640 1 Processor -- start 2024-02-19T15:29:45.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57f5a00640 1 -- start start 2024-02-19T15:29:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57f5a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57f006d650 0x7f57f00c7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57f5a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57f00cded0 0x7f57f00c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57f5a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57f00ced10 0x7f57f00b7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57f5a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f00c6d20 con 0x7f57f00ced10 2024-02-19T15:29:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57f5a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f0161040 con 0x7f57f006d650 2024-02-19T15:29:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57f5a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f01612e0 con 0x7f57f00cded0 2024-02-19T15:29:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57ef400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57f00ced10 0x7f57f00b7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57ef400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57f00ced10 0x7f57f00b7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43594/0 (socket says 172.21.15.47:43594) 2024-02-19T15:29:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57ef400640 1 -- 172.21.15.47:0/1668427142 learned_addr learned my addr 172.21.15.47:0/1668427142 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57eea00640 1 --2- 172.21.15.47:0/1668427142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57f006d650 0x7f57f00c7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57e6000640 1 --2- 172.21.15.47:0/1668427142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57f00cded0 0x7f57f00c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57ef400640 1 -- 172.21.15.47:0/1668427142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57f00cded0 msgr2=0x7f57f00c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57ef400640 1 --2- 172.21.15.47:0/1668427142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57f00cded0 0x7f57f00c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57ef400640 1 -- 172.21.15.47:0/1668427142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57f006d650 msgr2=0x7f57f00c7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57ef400640 1 --2- 172.21.15.47:0/1668427142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57f006d650 0x7f57f00c7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57ef400640 1 -- 172.21.15.47:0/1668427142 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e00094d0 con 0x7f57f00ced10 2024-02-19T15:29:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57e6000640 1 --2- 172.21.15.47:0/1668427142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57f00cded0 0x7f57f00c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:29:45.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57ef400640 1 --2- 172.21.15.47:0/1668427142 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57f00ced10 0x7f57f00b7380 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f57e800efc0 tx=0x7f57e800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:45.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57e7400640 1 -- 172.21.15.47:0/1668427142 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e8014480 con 0x7f57f00ced10 2024-02-19T15:29:45.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57e7400640 1 -- 172.21.15.47:0/1668427142 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e800f040 con 0x7f57f00ced10 2024-02-19T15:29:45.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57f5a00640 1 -- 172.21.15.47:0/1668427142 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f01615c0 con 0x7f57f00ced10 2024-02-19T15:29:45.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57e7400640 1 -- 172.21.15.47:0/1668427142 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e8014820 con 0x7f57f00ced10 2024-02-19T15:29:45.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.706+0000 7f57f5a00640 1 -- 172.21.15.47:0/1668427142 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f0159e50 con 0x7f57f00ced10 2024-02-19T15:29:45.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.710+0000 7f57f5a00640 1 -- 172.21.15.47:0/1668427142 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f00c6e60 con 0x7f57f00ced10 2024-02-19T15:29:45.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.710+0000 7f57e7400640 1 -- 172.21.15.47:0/1668427142 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f57e8014980 con 0x7f57f00ced10 2024-02-19T15:29:45.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.710+0000 7f57e7400640 1 --2- 172.21.15.47:0/1668427142 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f57cc066e40 0x7f57cc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:45.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.710+0000 7f57e7400640 1 -- 172.21.15.47:0/1668427142 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f57e8085560 con 0x7f57f00ced10 2024-02-19T15:29:45.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.710+0000 7f57eea00640 1 --2- 172.21.15.47:0/1668427142 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f57cc066e40 0x7f57cc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:45.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.710+0000 7f57eea00640 1 --2- 172.21.15.47:0/1668427142 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f57cc066e40 0x7f57cc069300 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f57e00ae000 tx=0x7f57e00b5040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:45.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.714+0000 7f57e7400640 1 -- 172.21.15.47:0/1668427142 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f57e8052340 con 0x7f57f00ced10 2024-02-19T15:29:45.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.914+0000 7f57f5a00640 1 -- 172.21.15.47:0/1668427142 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57f0075360 con 0x7f57f00ced10 2024-02-19T15:29:45.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.914+0000 7f57e7400640 1 -- 172.21.15.47:0/1668427142 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3379 (secure 0 0 0) 0x7f57e8018d50 con 0x7f57f00ced10 2024-02-19T15:29:45.913 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:45.913 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:24.882579+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:45.913 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 98 2024-02-19T15:29:45.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.914+0000 7f57f5a00640 1 -- 172.21.15.47:0/1668427142 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f57cc066e40 msgr2=0x7f57cc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:45.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.914+0000 7f57f5a00640 1 --2- 172.21.15.47:0/1668427142 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f57cc066e40 0x7f57cc069300 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f57e00ae000 tx=0x7f57e00b5040 comp rx=0 tx=0).stop 2024-02-19T15:29:45.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.914+0000 7f57f5a00640 1 -- 172.21.15.47:0/1668427142 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57f00ced10 msgr2=0x7f57f00b7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:45.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.914+0000 7f57f5a00640 1 --2- 172.21.15.47:0/1668427142 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57f00ced10 0x7f57f00b7380 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f57e800efc0 tx=0x7f57e800c490 comp rx=0 tx=0).stop 2024-02-19T15:29:45.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.914+0000 7f57f5a00640 1 -- 172.21.15.47:0/1668427142 shutdown_connections 2024-02-19T15:29:45.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.914+0000 7f57f5a00640 1 --2- 172.21.15.47:0/1668427142 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f57cc066e40 0x7f57cc069300 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:45.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.914+0000 7f57f5a00640 1 --2- 172.21.15.47:0/1668427142 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57f00ced10 0x7f57f00b7380 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:45.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.914+0000 7f57f5a00640 1 --2- 172.21.15.47:0/1668427142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57f00cded0 0x7f57f00c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:45.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.914+0000 7f57f5a00640 1 --2- 172.21.15.47:0/1668427142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57f006d650 0x7f57f00c7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:45.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.914+0000 7f57f5a00640 1 -- 172.21.15.47:0/1668427142 >> 172.21.15.47:0/1668427142 conn(0x7f57f00d1bf0 msgr2=0x7f57f015dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:45.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.914+0000 7f57f5a00640 1 -- 172.21.15.47:0/1668427142 shutdown_connections 2024-02-19T15:29:45.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:45.914+0000 7f57f5a00640 1 -- 172.21.15.47:0/1668427142 wait complete. 2024-02-19T15:29:46.136 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:24.882579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:46.136 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:46.180 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:29:47.137 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:47.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.282+0000 7fb8cde00640 1 Processor -- start 2024-02-19T15:29:47.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.282+0000 7fb8cde00640 1 -- start start 2024-02-19T15:29:47.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.282+0000 7fb8cde00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb8c8145590 0x7fb8c80727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:47.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.282+0000 7fb8cde00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb8c80cea80 0x7fb8c806c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:47.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.282+0000 7fb8cde00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c815d280 con 0x7fb8c815c390 2024-02-19T15:29:47.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.282+0000 7fb8cde00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8c8070ff0 con 0x7fb8c8145590 2024-02-19T15:29:47.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.282+0000 7fb8cde00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c80709e0 con 0x7fb8c80cea80 2024-02-19T15:29:47.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.282+0000 7fb8c6a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb8c8145590 0x7fb8c80727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:47.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.282+0000 7fb8c6a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb8c8145590 0x7fb8c80727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43600/0 (socket says 172.21.15.47:43600) 2024-02-19T15:29:47.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.282+0000 7fb8c6a00640 1 -- 172.21.15.47:0/2849360720 learned_addr learned my addr 172.21.15.47:0/2849360720 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:47.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.282+0000 7fb8bfe00640 1 --2- 172.21.15.47:0/2849360720 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb8c80cea80 0x7fb8c806c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:47.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.282+0000 7fb8c6a00640 1 -- 172.21.15.47:0/2849360720 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb8c80cea80 msgr2=0x7fb8c806c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:47.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.282+0000 7fb8c6a00640 1 --2- 172.21.15.47:0/2849360720 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb8c80cea80 0x7fb8c806c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:47.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.282+0000 7fb8c6a00640 1 -- 172.21.15.47:0/2849360720 >> v1:172.21.15.116:6789/0 conn(0x7fb8c815c390 legacy=0x7fb8c815c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:47.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.282+0000 7fb8c6a00640 1 -- 172.21.15.47:0/2849360720 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c806cee0 con 0x7fb8c8145590 2024-02-19T15:29:47.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.286+0000 7fb8c6a00640 1 --2- 172.21.15.47:0/2849360720 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8c8145590 0x7fb8c80727b0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fb8b80098a0 tx=0x7fb8b801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e4d71bf8399993e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:47.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.286+0000 7fb8c6000640 1 -- 172.21.15.47:0/2849360720 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b80ab650 con 0x7fb8c8145590 2024-02-19T15:29:47.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.286+0000 7fb8c6000640 1 -- 172.21.15.47:0/2849360720 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b80ab7b0 con 0x7fb8c8145590 2024-02-19T15:29:47.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.286+0000 7fb8c6000640 1 -- 172.21.15.47:0/2849360720 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b80abb40 con 0x7fb8c8145590 2024-02-19T15:29:47.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.286+0000 7fb8cde00640 1 -- 172.21.15.47:0/2849360720 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8c8145590 msgr2=0x7fb8c80727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:47.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.286+0000 7fb8cde00640 1 --2- 172.21.15.47:0/2849360720 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8c8145590 0x7fb8c80727b0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fb8b80098a0 tx=0x7fb8b801c540 comp rx=0 tx=0).stop 2024-02-19T15:29:47.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.286+0000 7fb8cde00640 1 -- 172.21.15.47:0/2849360720 shutdown_connections 2024-02-19T15:29:47.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.286+0000 7fb8cde00640 1 --2- 172.21.15.47:0/2849360720 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb8c80cea80 0x7fb8c806c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:47.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.286+0000 7fb8cde00640 1 --2- 172.21.15.47:0/2849360720 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8c8145590 0x7fb8c80727b0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:47.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.286+0000 7fb8cde00640 1 -- 172.21.15.47:0/2849360720 >> 172.21.15.47:0/2849360720 conn(0x7fb8c80b16a0 msgr2=0x7fb8c80b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:47.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.286+0000 7fb8cde00640 1 -- 172.21.15.47:0/2849360720 shutdown_connections 2024-02-19T15:29:47.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.286+0000 7fb8cde00640 1 -- 172.21.15.47:0/2849360720 wait complete. 2024-02-19T15:29:47.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8cde00640 1 Processor -- start 2024-02-19T15:29:47.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8cde00640 1 -- start start 2024-02-19T15:29:47.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8cde00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8c80cea80 0x7fb8c815adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:47.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8cde00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8c8145590 0x7fb8c8134140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:47.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8cde00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb8c815c390 0x7fb8c80b2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:47.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8cde00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c815b910 con 0x7fb8c80cea80 2024-02-19T15:29:47.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8cde00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c80b3230 con 0x7fb8c8145590 2024-02-19T15:29:47.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8cde00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c80b3510 con 0x7fb8c815c390 2024-02-19T15:29:47.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8c6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8c80cea80 0x7fb8c815adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:47.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8c6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8c80cea80 0x7fb8c815adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43616/0 (socket says 172.21.15.47:43616) 2024-02-19T15:29:47.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8c6a00640 1 -- 172.21.15.47:0/3379738360 learned_addr learned my addr 172.21.15.47:0/3379738360 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:47.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8bfe00640 1 --2- 172.21.15.47:0/3379738360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8c8145590 0x7fb8c8134140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:47.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8c7400640 1 --2- 172.21.15.47:0/3379738360 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb8c815c390 0x7fb8c80b2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:47.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8c6a00640 1 -- 172.21.15.47:0/3379738360 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb8c815c390 msgr2=0x7fb8c80b2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:47.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8c6a00640 1 --2- 172.21.15.47:0/3379738360 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb8c815c390 0x7fb8c80b2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:47.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8c6a00640 1 -- 172.21.15.47:0/3379738360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8c8145590 msgr2=0x7fb8c8134140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:47.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8c6a00640 1 --2- 172.21.15.47:0/3379738360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8c8145590 0x7fb8c8134140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:47.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8c6a00640 1 -- 172.21.15.47:0/3379738360 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b80094d0 con 0x7fb8c80cea80 2024-02-19T15:29:47.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8c6a00640 1 --2- 172.21.15.47:0/3379738360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8c80cea80 0x7fb8c815adc0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fb8b80ad040 tx=0x7fb8b801ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:47.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8bf400640 1 -- 172.21.15.47:0/3379738360 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b80b4800 con 0x7fb8c80cea80 2024-02-19T15:29:47.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8bf400640 1 -- 172.21.15.47:0/3379738360 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b80b4980 con 0x7fb8c80cea80 2024-02-19T15:29:47.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8cde00640 1 -- 172.21.15.47:0/3379738360 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c8138490 con 0x7fb8c80cea80 2024-02-19T15:29:47.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8bf400640 1 -- 172.21.15.47:0/3379738360 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b80b4d30 con 0x7fb8c80cea80 2024-02-19T15:29:47.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.290+0000 7fb8cde00640 1 -- 172.21.15.47:0/3379738360 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c80b37f0 con 0x7fb8c80cea80 2024-02-19T15:29:47.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.294+0000 7fb8cde00640 1 -- 172.21.15.47:0/3379738360 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c8075360 con 0x7fb8c80cea80 2024-02-19T15:29:47.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.294+0000 7fb8bf400640 1 -- 172.21.15.47:0/3379738360 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb8b80c7390 con 0x7fb8c80cea80 2024-02-19T15:29:47.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.294+0000 7fb8bf400640 1 --2- 172.21.15.47:0/3379738360 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb8a4066e70 0x7fb8a4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:47.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.294+0000 7fb8bf400640 1 -- 172.21.15.47:0/3379738360 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb8b811f500 con 0x7fb8c80cea80 2024-02-19T15:29:47.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.294+0000 7fb8bfe00640 1 --2- 172.21.15.47:0/3379738360 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb8a4066e70 0x7fb8a4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:47.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.298+0000 7fb8bfe00640 1 --2- 172.21.15.47:0/3379738360 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb8a4066e70 0x7fb8a4069330 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb8c81f9d40 tx=0x7fb8b0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:47.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.298+0000 7fb8bf400640 1 -- 172.21.15.47:0/3379738360 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb8b80ec3e0 con 0x7fb8c80cea80 2024-02-19T15:29:47.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.498+0000 7fb8cde00640 1 -- 172.21.15.47:0/3379738360 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8c813d170 con 0x7fb8c80cea80 2024-02-19T15:29:47.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.498+0000 7fb8bf400640 1 -- 172.21.15.47:0/3379738360 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3379 (secure 0 0 0) 0x7fb8b80efaa0 con 0x7fb8c80cea80 2024-02-19T15:29:47.497 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:47.497 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:24.882579+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:47.497 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 98 2024-02-19T15:29:47.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.498+0000 7fb8cde00640 1 -- 172.21.15.47:0/3379738360 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb8a4066e70 msgr2=0x7fb8a4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:47.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.498+0000 7fb8cde00640 1 --2- 172.21.15.47:0/3379738360 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb8a4066e70 0x7fb8a4069330 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb8c81f9d40 tx=0x7fb8b0008040 comp rx=0 tx=0).stop 2024-02-19T15:29:47.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.498+0000 7fb8cde00640 1 -- 172.21.15.47:0/3379738360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8c80cea80 msgr2=0x7fb8c815adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:47.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.498+0000 7fb8cde00640 1 --2- 172.21.15.47:0/3379738360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8c80cea80 0x7fb8c815adc0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fb8b80ad040 tx=0x7fb8b801ad80 comp rx=0 tx=0).stop 2024-02-19T15:29:47.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.498+0000 7fb8cde00640 1 -- 172.21.15.47:0/3379738360 shutdown_connections 2024-02-19T15:29:47.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.498+0000 7fb8cde00640 1 --2- 172.21.15.47:0/3379738360 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb8a4066e70 0x7fb8a4069330 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:47.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.498+0000 7fb8cde00640 1 --2- 172.21.15.47:0/3379738360 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb8c815c390 0x7fb8c80b2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:47.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.498+0000 7fb8cde00640 1 --2- 172.21.15.47:0/3379738360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8c8145590 0x7fb8c8134140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:47.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.498+0000 7fb8cde00640 1 --2- 172.21.15.47:0/3379738360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8c80cea80 0x7fb8c815adc0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:47.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.498+0000 7fb8cde00640 1 -- 172.21.15.47:0/3379738360 >> 172.21.15.47:0/3379738360 conn(0x7fb8c80b16a0 msgr2=0x7fb8c80b0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:47.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.498+0000 7fb8cde00640 1 -- 172.21.15.47:0/3379738360 shutdown_connections 2024-02-19T15:29:47.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:47.498+0000 7fb8cde00640 1 -- 172.21.15.47:0/3379738360 wait complete. 2024-02-19T15:29:47.718 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:24.882579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:47.719 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:48.180 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:29:48.719 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:48.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.862+0000 7fd8b8a00640 1 Processor -- start 2024-02-19T15:29:48.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.862+0000 7fd8b8a00640 1 -- start start 2024-02-19T15:29:48.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.862+0000 7fd8b8a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8b406d650 0x7fd8b40d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:48.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.862+0000 7fd8b8a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd8b40ced10 0x7fd8b406bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:48.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.862+0000 7fd8b8a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b40cce30 con 0x7fd8b40cded0 2024-02-19T15:29:48.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.862+0000 7fd8b8a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b40cd110 con 0x7fd8b40ced10 2024-02-19T15:29:48.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.862+0000 7fd8b8a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b40cd3f0 con 0x7fd8b406d650 2024-02-19T15:29:48.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.862+0000 7fd8abe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd8b40cded0 0x7fd8b40ce600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41150/0 (socket says 172.21.15.47:41150) 2024-02-19T15:29:48.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.862+0000 7fd8abe00640 1 -- 172.21.15.47:0/2594682231 learned_addr learned my addr 172.21.15.47:0/2594682231 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:29:48.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.862+0000 7fd8b2800640 1 --2- 172.21.15.47:0/2594682231 >> v2:172.21.15.47:3300/0 conn(0x7fd8b40ced10 0x7fd8b406bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:48.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.862+0000 7fd8b1e00640 1 --2- 172.21.15.47:0/2594682231 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8b406d650 0x7fd8b40d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:48.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.866+0000 7fd8b1400640 1 -- 172.21.15.47:0/2594682231 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832518700 0 0) 0x7fd8b40cce30 con 0x7fd8b40cded0 2024-02-19T15:29:48.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.866+0000 7fd8b1400640 1 -- 172.21.15.47:0/2594682231 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd898003530 con 0x7fd8b40cded0 2024-02-19T15:29:48.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.866+0000 7fd8b2800640 1 -- 172.21.15.47:0/2594682231 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8b406d650 msgr2=0x7fd8b40d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:48.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.866+0000 7fd8b2800640 1 --2- 172.21.15.47:0/2594682231 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8b406d650 0x7fd8b40d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:48.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.866+0000 7fd8b2800640 1 -- 172.21.15.47:0/2594682231 >> v1:172.21.15.47:6789/0 conn(0x7fd8b40cded0 legacy=0x7fd8b40ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:48.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.866+0000 7fd8b2800640 1 -- 172.21.15.47:0/2594682231 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b40cd6d0 con 0x7fd8b40ced10 2024-02-19T15:29:48.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.866+0000 7fd8b2800640 1 --2- 172.21.15.47:0/2594682231 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8b40ced10 0x7fd8b406bd60 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac009d90 tx=0x7fd8ac01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c034bd230c0aeac3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:48.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.866+0000 7fd8b1400640 1 -- 172.21.15.47:0/2594682231 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac0a5990 con 0x7fd8b40ced10 2024-02-19T15:29:48.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.866+0000 7fd8b1400640 1 -- 172.21.15.47:0/2594682231 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ac0a5af0 con 0x7fd8b40ced10 2024-02-19T15:29:48.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.866+0000 7fd8b1400640 1 -- 172.21.15.47:0/2594682231 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac0a5e80 con 0x7fd8b40ced10 2024-02-19T15:29:48.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.866+0000 7fd8b8a00640 1 -- 172.21.15.47:0/2594682231 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8b40ced10 msgr2=0x7fd8b406bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:48.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.866+0000 7fd8b8a00640 1 --2- 172.21.15.47:0/2594682231 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8b40ced10 0x7fd8b406bd60 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac009d90 tx=0x7fd8ac01c8b0 comp rx=0 tx=0).stop 2024-02-19T15:29:48.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.866+0000 7fd8b8a00640 1 -- 172.21.15.47:0/2594682231 shutdown_connections 2024-02-19T15:29:48.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.866+0000 7fd8b8a00640 1 --2- 172.21.15.47:0/2594682231 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8b40ced10 0x7fd8b406bd60 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:48.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.866+0000 7fd8b8a00640 1 --2- 172.21.15.47:0/2594682231 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8b406d650 0x7fd8b40d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:48.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.866+0000 7fd8b8a00640 1 -- 172.21.15.47:0/2594682231 >> 172.21.15.47:0/2594682231 conn(0x7fd8b40d1bf0 msgr2=0x7fd8b40d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:48.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.870+0000 7fd8b8a00640 1 -- 172.21.15.47:0/2594682231 shutdown_connections 2024-02-19T15:29:48.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.870+0000 7fd8b8a00640 1 -- 172.21.15.47:0/2594682231 wait complete. 2024-02-19T15:29:48.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.870+0000 7fd8b8a00640 1 Processor -- start 2024-02-19T15:29:48.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.870+0000 7fd8b8a00640 1 -- start start 2024-02-19T15:29:48.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8b8a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8b406d650 0x7fd8b4147b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:48.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8b8a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8b40cded0 0x7fd8b41589d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:48.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8b8a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8b40ced10 0x7fd8b4160160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:48.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8b8a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b40cae10 con 0x7fd8b40ced10 2024-02-19T15:29:48.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8b8a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b40ca200 con 0x7fd8b406d650 2024-02-19T15:29:48.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8b8a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b40ca4e0 con 0x7fd8b40cded0 2024-02-19T15:29:48.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8b2800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8b40ced10 0x7fd8b4160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:48.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8b2800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8b40ced10 0x7fd8b4160160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43634/0 (socket says 172.21.15.47:43634) 2024-02-19T15:29:48.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8b2800640 1 -- 172.21.15.47:0/3371566778 learned_addr learned my addr 172.21.15.47:0/3371566778 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:48.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8b1e00640 1 --2- 172.21.15.47:0/3371566778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8b406d650 0x7fd8b4147b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:48.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8abe00640 1 --2- 172.21.15.47:0/3371566778 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8b40cded0 0x7fd8b41589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:48.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8b2800640 1 -- 172.21.15.47:0/3371566778 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8b40cded0 msgr2=0x7fd8b41589d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:48.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8b2800640 1 --2- 172.21.15.47:0/3371566778 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8b40cded0 0x7fd8b41589d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:48.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8b2800640 1 -- 172.21.15.47:0/3371566778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8b406d650 msgr2=0x7fd8b4147b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:48.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8b2800640 1 --2- 172.21.15.47:0/3371566778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8b406d650 0x7fd8b4147b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:48.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8b2800640 1 -- 172.21.15.47:0/3371566778 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ac009a70 con 0x7fd8b40ced10 2024-02-19T15:29:48.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8b2800640 1 --2- 172.21.15.47:0/3371566778 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8b40ced10 0x7fd8b4160160 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac0b0040 tx=0x7fd8ac019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:48.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8aa000640 1 -- 172.21.15.47:0/3371566778 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac0adce0 con 0x7fd8b40ced10 2024-02-19T15:29:48.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8aa000640 1 -- 172.21.15.47:0/3371566778 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ac0ade40 con 0x7fd8b40ced10 2024-02-19T15:29:48.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8aa000640 1 -- 172.21.15.47:0/3371566778 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac0b9690 con 0x7fd8b40ced10 2024-02-19T15:29:48.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8b8a00640 1 -- 172.21.15.47:0/3371566778 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b40ca7c0 con 0x7fd8b40ced10 2024-02-19T15:29:48.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8b8a00640 1 -- 172.21.15.47:0/3371566778 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b40c9eb0 con 0x7fd8b40ced10 2024-02-19T15:29:48.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.874+0000 7fd8b8a00640 1 -- 172.21.15.47:0/3371566778 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b40caf80 con 0x7fd8b40ced10 2024-02-19T15:29:48.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.878+0000 7fd8aa000640 1 -- 172.21.15.47:0/3371566778 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd8ac0b97f0 con 0x7fd8b40ced10 2024-02-19T15:29:48.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.878+0000 7fd8aa000640 1 --2- 172.21.15.47:0/3371566778 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd890066e40 0x7fd890069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:48.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.878+0000 7fd8aa000640 1 -- 172.21.15.47:0/3371566778 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd8ac120220 con 0x7fd8b40ced10 2024-02-19T15:29:48.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.878+0000 7fd8b1e00640 1 --2- 172.21.15.47:0/3371566778 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd890066e40 0x7fd890069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:48.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.878+0000 7fd8b1e00640 1 --2- 172.21.15.47:0/3371566778 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd890066e40 0x7fd890069300 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4009ac0 tx=0x7fd8a4009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:48.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:48.886+0000 7fd8aa000640 1 -- 172.21.15.47:0/3371566778 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd8ac0ed100 con 0x7fd8b40ced10 2024-02-19T15:29:49.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:49.086+0000 7fd8b8a00640 1 -- 172.21.15.47:0/3371566778 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8b4075360 con 0x7fd8b40ced10 2024-02-19T15:29:49.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:49.086+0000 7fd8aa000640 1 -- 172.21.15.47:0/3371566778 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3379 (secure 0 0 0) 0x7fd8ac009450 con 0x7fd8b40ced10 2024-02-19T15:29:49.085 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:49.085 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:24.882579+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:49.085 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 98 2024-02-19T15:29:49.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:49.086+0000 7fd8b8a00640 1 -- 172.21.15.47:0/3371566778 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd890066e40 msgr2=0x7fd890069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:49.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:49.086+0000 7fd8b8a00640 1 --2- 172.21.15.47:0/3371566778 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd890066e40 0x7fd890069300 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4009ac0 tx=0x7fd8a4009190 comp rx=0 tx=0).stop 2024-02-19T15:29:49.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:49.086+0000 7fd8b8a00640 1 -- 172.21.15.47:0/3371566778 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8b40ced10 msgr2=0x7fd8b4160160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:49.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:49.086+0000 7fd8b8a00640 1 --2- 172.21.15.47:0/3371566778 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8b40ced10 0x7fd8b4160160 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac0b0040 tx=0x7fd8ac019650 comp rx=0 tx=0).stop 2024-02-19T15:29:49.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:49.086+0000 7fd8b8a00640 1 -- 172.21.15.47:0/3371566778 shutdown_connections 2024-02-19T15:29:49.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:49.086+0000 7fd8b8a00640 1 --2- 172.21.15.47:0/3371566778 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd890066e40 0x7fd890069300 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:49.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:49.086+0000 7fd8b8a00640 1 --2- 172.21.15.47:0/3371566778 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8b40ced10 0x7fd8b4160160 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:49.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:49.086+0000 7fd8b8a00640 1 --2- 172.21.15.47:0/3371566778 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd8b40cded0 0x7fd8b41589d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:49.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:49.086+0000 7fd8b8a00640 1 --2- 172.21.15.47:0/3371566778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8b406d650 0x7fd8b4147b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:49.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:49.086+0000 7fd8b8a00640 1 -- 172.21.15.47:0/3371566778 >> 172.21.15.47:0/3371566778 conn(0x7fd8b40d1bf0 msgr2=0x7fd8b40ce5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:49.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:49.086+0000 7fd8b8a00640 1 -- 172.21.15.47:0/3371566778 shutdown_connections 2024-02-19T15:29:49.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:49.086+0000 7fd8b8a00640 1 -- 172.21.15.47:0/3371566778 wait complete. 2024-02-19T15:29:49.309 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:24.882579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:49.309 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:50.180 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:29:50.309 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:50.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.450+0000 7f1fbfe00640 1 Processor -- start 2024-02-19T15:29:50.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.450+0000 7f1fbfe00640 1 -- start start 2024-02-19T15:29:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.450+0000 7f1fbfe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1fb815e0e0 0x7f1fb80d25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.450+0000 7f1fbfe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1fb815e4c0 0x7f1fb80d07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.450+0000 7f1fbfe00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb80d00d0 con 0x7f1fb80d4970 2024-02-19T15:29:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.450+0000 7f1fbfe00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fb80d2e60 con 0x7f1fb815e0e0 2024-02-19T15:29:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.450+0000 7f1fbfe00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fb80ce960 con 0x7f1fb815e4c0 2024-02-19T15:29:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.450+0000 7f1fbd400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1fb815e0e0 0x7f1fb80d25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.450+0000 7f1fbd400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1fb815e0e0 0x7f1fb80d25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43644/0 (socket says 172.21.15.47:43644) 2024-02-19T15:29:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.450+0000 7f1fbca00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1fb80d4970 0x7f1fb80cf9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41152/0 (socket says 172.21.15.47:41152) 2024-02-19T15:29:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.450+0000 7f1fbd400640 1 -- 172.21.15.47:0/3683177348 learned_addr learned my addr 172.21.15.47:0/3683177348 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.454+0000 7f1fbde00640 1 --2- 172.21.15.47:0/3683177348 >> v2:172.21.15.116:3300/0 conn(0x7f1fb815e4c0 0x7f1fb80d07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.454+0000 7f1fbd400640 1 -- 172.21.15.47:0/3683177348 >> v2:172.21.15.116:3300/0 conn(0x7f1fb815e4c0 msgr2=0x7f1fb80d07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.454+0000 7f1fbd400640 1 --2- 172.21.15.47:0/3683177348 >> v2:172.21.15.116:3300/0 conn(0x7f1fb815e4c0 0x7f1fb80d07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.454+0000 7f1fbd400640 1 -- 172.21.15.47:0/3683177348 >> v1:172.21.15.47:6789/0 conn(0x7f1fb80d4970 legacy=0x7f1fb80cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.454+0000 7f1fbd400640 1 -- 172.21.15.47:0/3683177348 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb80cec00 con 0x7f1fb815e0e0 2024-02-19T15:29:50.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.454+0000 7f1fbd400640 1 --2- 172.21.15.47:0/3683177348 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1fb815e0e0 0x7f1fb80d25d0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f1fb80d0780 tx=0x7f1fac01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=148169fba6a4abe2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:50.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.454+0000 7f1fb7e00640 1 -- 172.21.15.47:0/3683177348 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fac0a8a10 con 0x7f1fb815e0e0 2024-02-19T15:29:50.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.454+0000 7f1fb7e00640 1 -- 172.21.15.47:0/3683177348 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fac0a8b70 con 0x7f1fb815e0e0 2024-02-19T15:29:50.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.454+0000 7f1fb7e00640 1 -- 172.21.15.47:0/3683177348 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fac0b0440 con 0x7f1fb815e0e0 2024-02-19T15:29:50.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.454+0000 7f1fbfe00640 1 -- 172.21.15.47:0/3683177348 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1fb815e0e0 msgr2=0x7f1fb80d25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:50.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.454+0000 7f1fbfe00640 1 --2- 172.21.15.47:0/3683177348 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1fb815e0e0 0x7f1fb80d25d0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f1fb80d0780 tx=0x7f1fac01ed10 comp rx=0 tx=0).stop 2024-02-19T15:29:50.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.458+0000 7f1fbfe00640 1 -- 172.21.15.47:0/3683177348 shutdown_connections 2024-02-19T15:29:50.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.458+0000 7f1fbfe00640 1 --2- 172.21.15.47:0/3683177348 >> v2:172.21.15.116:3300/0 conn(0x7f1fb815e4c0 0x7f1fb80d07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:50.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.458+0000 7f1fbfe00640 1 --2- 172.21.15.47:0/3683177348 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1fb815e0e0 0x7f1fb80d25d0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:50.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.458+0000 7f1fbfe00640 1 -- 172.21.15.47:0/3683177348 >> 172.21.15.47:0/3683177348 conn(0x7f1fb80d3de0 msgr2=0x7f1fb80d41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:50.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.458+0000 7f1fbfe00640 1 -- 172.21.15.47:0/3683177348 shutdown_connections 2024-02-19T15:29:50.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.458+0000 7f1fbfe00640 1 -- 172.21.15.47:0/3683177348 wait complete. 2024-02-19T15:29:50.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.458+0000 7f1fbfe00640 1 Processor -- start 2024-02-19T15:29:50.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.458+0000 7f1fbfe00640 1 -- start start 2024-02-19T15:29:50.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.458+0000 7f1fbfe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb80d4970 0x7f1fb815c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:50.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.458+0000 7f1fbfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1fb815e0e0 0x7f1fb81f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:50.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.458+0000 7f1fbfe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1fb815e4c0 0x7f1fb80b92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:50.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.458+0000 7f1fbfe00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb809ec70 con 0x7f1fb815e0e0 2024-02-19T15:29:50.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.458+0000 7f1fbfe00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb815c6c0 con 0x7f1fb80d4970 2024-02-19T15:29:50.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.458+0000 7f1fbfe00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb80b6b10 con 0x7f1fb815e4c0 2024-02-19T15:29:50.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.458+0000 7f1fbca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1fb815e0e0 0x7f1fb81f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:50.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.458+0000 7f1fbca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1fb815e0e0 0x7f1fb81f3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43650/0 (socket says 172.21.15.47:43650) 2024-02-19T15:29:50.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.458+0000 7f1fbca00640 1 -- 172.21.15.47:0/1438221675 learned_addr learned my addr 172.21.15.47:0/1438221675 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:50.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.458+0000 7f1fbd400640 1 --2- 172.21.15.47:0/1438221675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb80d4970 0x7f1fb815c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:50.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.458+0000 7f1fbde00640 1 --2- 172.21.15.47:0/1438221675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1fb815e4c0 0x7f1fb80b92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:50.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.462+0000 7f1fbca00640 1 -- 172.21.15.47:0/1438221675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1fb815e4c0 msgr2=0x7f1fb80b92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:50.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.462+0000 7f1fbca00640 1 --2- 172.21.15.47:0/1438221675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1fb815e4c0 0x7f1fb80b92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:50.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.462+0000 7f1fbca00640 1 -- 172.21.15.47:0/1438221675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb80d4970 msgr2=0x7f1fb815c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:50.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.462+0000 7f1fbca00640 1 --2- 172.21.15.47:0/1438221675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb80d4970 0x7f1fb815c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:50.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.462+0000 7f1fbca00640 1 -- 172.21.15.47:0/1438221675 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fac0094d0 con 0x7f1fb815e0e0 2024-02-19T15:29:50.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.462+0000 7f1fbd400640 1 --2- 172.21.15.47:0/1438221675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb80d4970 0x7f1fb815c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:29:50.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.462+0000 7f1fbca00640 1 --2- 172.21.15.47:0/1438221675 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1fb815e0e0 0x7f1fb81f3b30 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f1fa400c370 tx=0x7f1fa400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:50.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.462+0000 7f1fb5600640 1 -- 172.21.15.47:0/1438221675 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa40074e0 con 0x7f1fb815e0e0 2024-02-19T15:29:50.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.462+0000 7f1fb5600640 1 -- 172.21.15.47:0/1438221675 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa4007640 con 0x7f1fb815e0e0 2024-02-19T15:29:50.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.462+0000 7f1fb5600640 1 -- 172.21.15.47:0/1438221675 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4004a30 con 0x7f1fb815e0e0 2024-02-19T15:29:50.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.462+0000 7f1fbfe00640 1 -- 172.21.15.47:0/1438221675 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb80b6df0 con 0x7f1fb815e0e0 2024-02-19T15:29:50.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.462+0000 7f1fbfe00640 1 -- 172.21.15.47:0/1438221675 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb80b73b0 con 0x7f1fb815e0e0 2024-02-19T15:29:50.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.462+0000 7f1fbfe00640 1 -- 172.21.15.47:0/1438221675 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb8075360 con 0x7f1fb815e0e0 2024-02-19T15:29:50.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.462+0000 7f1fb5600640 1 -- 172.21.15.47:0/1438221675 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1fa4004b90 con 0x7f1fb815e0e0 2024-02-19T15:29:50.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.462+0000 7f1fb5600640 1 --2- 172.21.15.47:0/1438221675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1f98066e40 0x7f1f98069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:50.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.462+0000 7f1fb5600640 1 -- 172.21.15.47:0/1438221675 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1fa40843a0 con 0x7f1fb815e0e0 2024-02-19T15:29:50.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.466+0000 7f1fbd400640 1 --2- 172.21.15.47:0/1438221675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1f98066e40 0x7f1f98069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:50.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.466+0000 7f1fbd400640 1 --2- 172.21.15.47:0/1438221675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1f98066e40 0x7f1f98069300 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f1fac0b2000 tx=0x7f1fac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:50.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.470+0000 7f1fb5600640 1 -- 172.21.15.47:0/1438221675 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1fa40511e0 con 0x7f1fb815e0e0 2024-02-19T15:29:50.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.674+0000 7f1fbfe00640 1 -- 172.21.15.47:0/1438221675 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fb80ce110 con 0x7f1fb815e0e0 2024-02-19T15:29:50.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.674+0000 7f1fb5600640 1 -- 172.21.15.47:0/1438221675 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3379 (secure 0 0 0) 0x7f1fa40548a0 con 0x7f1fb815e0e0 2024-02-19T15:29:50.675 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:50.675 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:24.882579+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:50.676 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 98 2024-02-19T15:29:50.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.678+0000 7f1fbfe00640 1 -- 172.21.15.47:0/1438221675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1f98066e40 msgr2=0x7f1f98069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:50.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.678+0000 7f1fbfe00640 1 --2- 172.21.15.47:0/1438221675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1f98066e40 0x7f1f98069300 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f1fac0b2000 tx=0x7f1fac008040 comp rx=0 tx=0).stop 2024-02-19T15:29:50.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.678+0000 7f1fbfe00640 1 -- 172.21.15.47:0/1438221675 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1fb815e0e0 msgr2=0x7f1fb81f3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:50.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.678+0000 7f1fbfe00640 1 --2- 172.21.15.47:0/1438221675 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1fb815e0e0 0x7f1fb81f3b30 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f1fa400c370 tx=0x7f1fa400c830 comp rx=0 tx=0).stop 2024-02-19T15:29:50.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.678+0000 7f1fbfe00640 1 -- 172.21.15.47:0/1438221675 shutdown_connections 2024-02-19T15:29:50.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.678+0000 7f1fbfe00640 1 --2- 172.21.15.47:0/1438221675 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1f98066e40 0x7f1f98069300 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:50.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.678+0000 7f1fbfe00640 1 --2- 172.21.15.47:0/1438221675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1fb815e4c0 0x7f1fb80b92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:50.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.678+0000 7f1fbfe00640 1 --2- 172.21.15.47:0/1438221675 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1fb815e0e0 0x7f1fb81f3b30 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:50.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.678+0000 7f1fbfe00640 1 --2- 172.21.15.47:0/1438221675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb80d4970 0x7f1fb815c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:50.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.678+0000 7f1fbfe00640 1 -- 172.21.15.47:0/1438221675 >> 172.21.15.47:0/1438221675 conn(0x7f1fb80d3de0 msgr2=0x7f1fb80c85b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:50.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.678+0000 7f1fbfe00640 1 -- 172.21.15.47:0/1438221675 shutdown_connections 2024-02-19T15:29:50.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:50.678+0000 7f1fbfe00640 1 -- 172.21.15.47:0/1438221675 wait complete. 2024-02-19T15:29:50.897 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:24.882579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:50.898 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:51.898 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:52.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.038+0000 7ff57e600640 1 Processor -- start 2024-02-19T15:29:52.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.038+0000 7ff57e600640 1 -- start start 2024-02-19T15:29:52.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.038+0000 7ff57e600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff57806d650 0x7ff5780d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:52.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.038+0000 7ff57e600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5780cded0 0x7ff5780ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:52.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.038+0000 7ff57e600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5780cce30 con 0x7ff5780ceb40 2024-02-19T15:29:52.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.038+0000 7ff57e600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5780cefd0 con 0x7ff57806d650 2024-02-19T15:29:52.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.038+0000 7ff57e600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5780cd110 con 0x7ff5780cded0 2024-02-19T15:29:52.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.038+0000 7ff577400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff57806d650 0x7ff5780d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:52.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.038+0000 7ff577400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff57806d650 0x7ff5780d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43652/0 (socket says 172.21.15.47:43652) 2024-02-19T15:29:52.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.038+0000 7ff577400640 1 -- 172.21.15.47:0/1226003528 learned_addr learned my addr 172.21.15.47:0/1226003528 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:52.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.038+0000 7ff56fe00640 1 --2- 172.21.15.47:0/1226003528 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5780cded0 0x7ff5780ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:52.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.042+0000 7ff577400640 1 -- 172.21.15.47:0/1226003528 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5780cded0 msgr2=0x7ff5780ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:52.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.042+0000 7ff577400640 1 --2- 172.21.15.47:0/1226003528 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5780cded0 0x7ff5780ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:52.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.042+0000 7ff577400640 1 -- 172.21.15.47:0/1226003528 >> v1:172.21.15.116:6789/0 conn(0x7ff5780ceb40 legacy=0x7ff578161d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:52.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.042+0000 7ff577400640 1 -- 172.21.15.47:0/1226003528 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5780cd3f0 con 0x7ff57806d650 2024-02-19T15:29:52.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.042+0000 7ff577400640 1 --2- 172.21.15.47:0/1226003528 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff57806d650 0x7ff5780d08b0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7ff568009870 tx=0x7ff56801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ae186ab8d3317e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:52.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.042+0000 7ff576a00640 1 -- 172.21.15.47:0/1226003528 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5680ac620 con 0x7ff57806d650 2024-02-19T15:29:52.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.042+0000 7ff576a00640 1 -- 172.21.15.47:0/1226003528 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5680ac780 con 0x7ff57806d650 2024-02-19T15:29:52.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.042+0000 7ff576a00640 1 -- 172.21.15.47:0/1226003528 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5680acb10 con 0x7ff57806d650 2024-02-19T15:29:52.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff57e600640 1 -- 172.21.15.47:0/1226003528 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff57806d650 msgr2=0x7ff5780d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:52.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff57e600640 1 --2- 172.21.15.47:0/1226003528 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff57806d650 0x7ff5780d08b0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7ff568009870 tx=0x7ff56801c4a0 comp rx=0 tx=0).stop 2024-02-19T15:29:52.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff57e600640 1 -- 172.21.15.47:0/1226003528 shutdown_connections 2024-02-19T15:29:52.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff57e600640 1 --2- 172.21.15.47:0/1226003528 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5780cded0 0x7ff5780ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:52.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff57e600640 1 --2- 172.21.15.47:0/1226003528 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff57806d650 0x7ff5780d08b0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:52.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff57e600640 1 -- 172.21.15.47:0/1226003528 >> 172.21.15.47:0/1226003528 conn(0x7ff5780d1bf0 msgr2=0x7ff5780d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:52.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff57e600640 1 -- 172.21.15.47:0/1226003528 shutdown_connections 2024-02-19T15:29:52.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff57e600640 1 -- 172.21.15.47:0/1226003528 wait complete. 2024-02-19T15:29:52.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff57e600640 1 Processor -- start 2024-02-19T15:29:52.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff57e600640 1 -- start start 2024-02-19T15:29:52.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff57e600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff57806d650 0x7ff5780c8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:52.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff57e600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5780cded0 0x7ff5780c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:52.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff57e600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5780ceb40 0x7ff5780cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:52.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff57e600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5780c9200 con 0x7ff57806d650 2024-02-19T15:29:52.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff57e600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5780ca040 con 0x7ff5780ceb40 2024-02-19T15:29:52.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff57e600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5780c9430 con 0x7ff5780cded0 2024-02-19T15:29:52.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff577400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff57806d650 0x7ff5780c8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:52.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff577400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff57806d650 0x7ff5780c8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43668/0 (socket says 172.21.15.47:43668) 2024-02-19T15:29:52.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff577400640 1 -- 172.21.15.47:0/591581921 learned_addr learned my addr 172.21.15.47:0/591581921 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:52.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff56fe00640 1 --2- 172.21.15.47:0/591581921 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5780cded0 0x7ff5780c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:52.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff577e00640 1 --2- 172.21.15.47:0/591581921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5780ceb40 0x7ff5780cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:52.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff577400640 1 -- 172.21.15.47:0/591581921 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5780cded0 msgr2=0x7ff5780c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:52.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff577400640 1 --2- 172.21.15.47:0/591581921 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5780cded0 0x7ff5780c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:52.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff577400640 1 -- 172.21.15.47:0/591581921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5780ceb40 msgr2=0x7ff5780cb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:52.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff577400640 1 --2- 172.21.15.47:0/591581921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5780ceb40 0x7ff5780cb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:52.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff577400640 1 -- 172.21.15.47:0/591581921 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5680094d0 con 0x7ff57806d650 2024-02-19T15:29:52.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.046+0000 7ff56fe00640 1 --2- 172.21.15.47:0/591581921 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5780cded0 0x7ff5780c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:29:52.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.050+0000 7ff577400640 1 --2- 172.21.15.47:0/591581921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff57806d650 0x7ff5780c8700 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7ff5680af040 tx=0x7ff568019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:52.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.050+0000 7ff56f400640 1 -- 172.21.15.47:0/591581921 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5680b7470 con 0x7ff57806d650 2024-02-19T15:29:52.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.050+0000 7ff56f400640 1 -- 172.21.15.47:0/591581921 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5680b75d0 con 0x7ff57806d650 2024-02-19T15:29:52.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.050+0000 7ff56f400640 1 -- 172.21.15.47:0/591581921 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5680b7980 con 0x7ff57806d650 2024-02-19T15:29:52.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.050+0000 7ff57e600640 1 -- 172.21.15.47:0/591581921 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5780c9710 con 0x7ff57806d650 2024-02-19T15:29:52.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.050+0000 7ff57e600640 1 -- 172.21.15.47:0/591581921 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5780c9cd0 con 0x7ff57806d650 2024-02-19T15:29:52.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.050+0000 7ff57e600640 1 -- 172.21.15.47:0/591581921 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff578075360 con 0x7ff57806d650 2024-02-19T15:29:52.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.050+0000 7ff56f400640 1 -- 172.21.15.47:0/591581921 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff5680b3070 con 0x7ff57806d650 2024-02-19T15:29:52.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.050+0000 7ff56f400640 1 --2- 172.21.15.47:0/591581921 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff554066e70 0x7ff554069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:52.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.050+0000 7ff56f400640 1 -- 172.21.15.47:0/591581921 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff56811ef10 con 0x7ff57806d650 2024-02-19T15:29:52.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.050+0000 7ff56fe00640 1 --2- 172.21.15.47:0/591581921 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff554066e70 0x7ff554069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:52.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.054+0000 7ff56fe00640 1 --2- 172.21.15.47:0/591581921 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff554066e70 0x7ff554069330 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7ff578168f20 tx=0x7ff560008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:52.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.054+0000 7ff56f400640 1 -- 172.21.15.47:0/591581921 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff5680ebd50 con 0x7ff57806d650 2024-02-19T15:29:52.180 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:29:52.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.250+0000 7ff57e600640 1 -- 172.21.15.47:0/591581921 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5780cf580 con 0x7ff57806d650 2024-02-19T15:29:52.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.250+0000 7ff56f400640 1 -- 172.21.15.47:0/591581921 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3379 (secure 0 0 0) 0x7ff5680b1090 con 0x7ff57806d650 2024-02-19T15:29:52.251 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:52.251 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:24.882579+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:52.251 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 98 2024-02-19T15:29:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.254+0000 7ff57e600640 1 -- 172.21.15.47:0/591581921 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff554066e70 msgr2=0x7ff554069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.254+0000 7ff57e600640 1 --2- 172.21.15.47:0/591581921 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff554066e70 0x7ff554069330 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7ff578168f20 tx=0x7ff560008040 comp rx=0 tx=0).stop 2024-02-19T15:29:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.254+0000 7ff57e600640 1 -- 172.21.15.47:0/591581921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff57806d650 msgr2=0x7ff5780c8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.254+0000 7ff57e600640 1 --2- 172.21.15.47:0/591581921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff57806d650 0x7ff5780c8700 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7ff5680af040 tx=0x7ff568019240 comp rx=0 tx=0).stop 2024-02-19T15:29:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.254+0000 7ff57e600640 1 -- 172.21.15.47:0/591581921 shutdown_connections 2024-02-19T15:29:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.254+0000 7ff57e600640 1 --2- 172.21.15.47:0/591581921 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff554066e70 0x7ff554069330 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.254+0000 7ff57e600640 1 --2- 172.21.15.47:0/591581921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5780ceb40 0x7ff5780cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.254+0000 7ff57e600640 1 --2- 172.21.15.47:0/591581921 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5780cded0 0x7ff5780c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.254+0000 7ff57e600640 1 --2- 172.21.15.47:0/591581921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff57806d650 0x7ff5780c8700 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.254+0000 7ff57e600640 1 -- 172.21.15.47:0/591581921 >> 172.21.15.47:0/591581921 conn(0x7ff5780d1bf0 msgr2=0x7ff578070130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.254+0000 7ff57e600640 1 -- 172.21.15.47:0/591581921 shutdown_connections 2024-02-19T15:29:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:52.254+0000 7ff57e600640 1 -- 172.21.15.47:0/591581921 wait complete. 2024-02-19T15:29:52.474 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:24.882579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:52.474 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:53.475 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:53.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.614+0000 7f17af400640 1 Processor -- start 2024-02-19T15:29:53.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.614+0000 7f17af400640 1 -- start start 2024-02-19T15:29:53.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.618+0000 7f17af400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f17a806d650 0x7f17a80d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:53.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.618+0000 7f17af400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f17a80ced10 0x7f17a806bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:53.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.618+0000 7f17af400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a80cce30 con 0x7f17a80cded0 2024-02-19T15:29:53.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.618+0000 7f17af400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a80cd110 con 0x7f17a806d650 2024-02-19T15:29:53.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.618+0000 7f17af400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a80cd3f0 con 0x7f17a80ced10 2024-02-19T15:29:53.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.618+0000 7f17aca00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f17a806d650 0x7f17a80d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:53.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.618+0000 7f17aca00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f17a806d650 0x7f17a80d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:38280/0 (socket says 172.21.15.47:38280) 2024-02-19T15:29:53.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.618+0000 7f17aca00640 1 -- 172.21.15.47:0/200994607 learned_addr learned my addr 172.21.15.47:0/200994607 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:53.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.618+0000 7f17ad400640 1 --2- 172.21.15.47:0/200994607 >> v2:172.21.15.116:3300/0 conn(0x7f17a80ced10 0x7f17a806bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:53.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.618+0000 7f17aca00640 1 -- 172.21.15.47:0/200994607 >> v2:172.21.15.116:3300/0 conn(0x7f17a80ced10 msgr2=0x7f17a806bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:53.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.618+0000 7f17aca00640 1 --2- 172.21.15.47:0/200994607 >> v2:172.21.15.116:3300/0 conn(0x7f17a80ced10 0x7f17a806bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:53.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.618+0000 7f17aca00640 1 -- 172.21.15.47:0/200994607 >> v1:172.21.15.116:6789/0 conn(0x7f17a80cded0 legacy=0x7f17a80ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:53.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.618+0000 7f17aca00640 1 -- 172.21.15.47:0/200994607 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a80cd6d0 con 0x7f17a806d650 2024-02-19T15:29:53.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.622+0000 7f17aca00640 1 --2- 172.21.15.47:0/200994607 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17a806d650 0x7f17a80d08b0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f179c009870 tx=0x7f179c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cc09ba6df54d903 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:53.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.622+0000 7f17a7400640 1 -- 172.21.15.47:0/200994607 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c0ab680 con 0x7f17a806d650 2024-02-19T15:29:53.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.622+0000 7f17a7400640 1 -- 172.21.15.47:0/200994607 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f179c0ab7e0 con 0x7f17a806d650 2024-02-19T15:29:53.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.622+0000 7f17a7400640 1 -- 172.21.15.47:0/200994607 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c0abb70 con 0x7f17a806d650 2024-02-19T15:29:53.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.622+0000 7f17af400640 1 -- 172.21.15.47:0/200994607 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17a806d650 msgr2=0x7f17a80d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:53.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.622+0000 7f17af400640 1 --2- 172.21.15.47:0/200994607 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17a806d650 0x7f17a80d08b0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f179c009870 tx=0x7f179c01c460 comp rx=0 tx=0).stop 2024-02-19T15:29:53.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.622+0000 7f17af400640 1 -- 172.21.15.47:0/200994607 shutdown_connections 2024-02-19T15:29:53.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.622+0000 7f17af400640 1 --2- 172.21.15.47:0/200994607 >> v2:172.21.15.116:3300/0 conn(0x7f17a80ced10 0x7f17a806bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:53.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.622+0000 7f17af400640 1 --2- 172.21.15.47:0/200994607 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17a806d650 0x7f17a80d08b0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:53.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.622+0000 7f17af400640 1 -- 172.21.15.47:0/200994607 >> 172.21.15.47:0/200994607 conn(0x7f17a80d1bf0 msgr2=0x7f17a80d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:53.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.622+0000 7f17af400640 1 -- 172.21.15.47:0/200994607 shutdown_connections 2024-02-19T15:29:53.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.622+0000 7f17af400640 1 -- 172.21.15.47:0/200994607 wait complete. 2024-02-19T15:29:53.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.622+0000 7f17af400640 1 Processor -- start 2024-02-19T15:29:53.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.622+0000 7f17af400640 1 -- start start 2024-02-19T15:29:53.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17af400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a806d650 0x7f17a815e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:53.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17af400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17a80cded0 0x7f17a81fa3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:53.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17af400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17a80ced10 0x7f17a81fa8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:53.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17af400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a815cd40 con 0x7f17a80cded0 2024-02-19T15:29:53.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17af400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a815dc60 con 0x7f17a806d650 2024-02-19T15:29:53.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17af400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a815c4a0 con 0x7f17a80ced10 2024-02-19T15:29:53.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17aca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a806d650 0x7f17a815e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:53.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17a7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17a80cded0 0x7f17a81fa3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:53.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17aca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a806d650 0x7f17a815e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:54862/0 (socket says 172.21.15.47:54862) 2024-02-19T15:29:53.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17aca00640 1 -- 172.21.15.47:0/436421455 learned_addr learned my addr 172.21.15.47:0/436421455 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:53.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17ad400640 1 --2- 172.21.15.47:0/436421455 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17a80ced10 0x7f17a81fa8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:53.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17a7e00640 1 -- 172.21.15.47:0/436421455 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17a80ced10 msgr2=0x7f17a81fa8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:53.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17a7e00640 1 --2- 172.21.15.47:0/436421455 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17a80ced10 0x7f17a81fa8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:53.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17a7e00640 1 -- 172.21.15.47:0/436421455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a806d650 msgr2=0x7f17a815e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:53.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17a7e00640 1 --2- 172.21.15.47:0/436421455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a806d650 0x7f17a815e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:53.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17a7e00640 1 -- 172.21.15.47:0/436421455 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179c0094d0 con 0x7f17a80cded0 2024-02-19T15:29:53.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17ad400640 1 --2- 172.21.15.47:0/436421455 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17a80ced10 0x7f17a81fa8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:29:53.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17a7e00640 1 --2- 172.21.15.47:0/436421455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17a80cded0 0x7f17a81fa3a0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f179400c390 tx=0x7f179400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:53.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17a4c00640 1 -- 172.21.15.47:0/436421455 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17940052d0 con 0x7f17a80cded0 2024-02-19T15:29:53.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17a4c00640 1 -- 172.21.15.47:0/436421455 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1794005430 con 0x7f17a80cded0 2024-02-19T15:29:53.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17af400640 1 -- 172.21.15.47:0/436421455 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a815c780 con 0x7f17a80cded0 2024-02-19T15:29:53.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17a4c00640 1 -- 172.21.15.47:0/436421455 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1794003840 con 0x7f17a80cded0 2024-02-19T15:29:53.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17af400640 1 -- 172.21.15.47:0/436421455 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a815d8f0 con 0x7f17a80cded0 2024-02-19T15:29:53.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17af400640 1 -- 172.21.15.47:0/436421455 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a80c5720 con 0x7f17a80cded0 2024-02-19T15:29:53.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.626+0000 7f17a4c00640 1 -- 172.21.15.47:0/436421455 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f17940039a0 con 0x7f17a80cded0 2024-02-19T15:29:53.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.630+0000 7f17a4c00640 1 --2- 172.21.15.47:0/436421455 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1788066e40 0x7f1788069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:53.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.630+0000 7f17a4c00640 1 -- 172.21.15.47:0/436421455 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1794083f60 con 0x7f17a80cded0 2024-02-19T15:29:53.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.630+0000 7f17aca00640 1 --2- 172.21.15.47:0/436421455 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1788066e40 0x7f1788069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:53.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.630+0000 7f17aca00640 1 --2- 172.21.15.47:0/436421455 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1788066e40 0x7f1788069300 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f179c01c4a0 tx=0x7f179c0b6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:53.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.634+0000 7f17a4c00640 1 -- 172.21.15.47:0/436421455 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f17a80c5720 con 0x7f17a80cded0 2024-02-19T15:29:53.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.826+0000 7f17af400640 1 -- 172.21.15.47:0/436421455 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17a80c5720 con 0x7f17a80cded0 2024-02-19T15:29:53.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.826+0000 7f17a4c00640 1 -- 172.21.15.47:0/436421455 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3379 (secure 0 0 0) 0x7f17a80c5720 con 0x7f17a80cded0 2024-02-19T15:29:53.828 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:53.828 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:24.882579+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:53.828 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 98 2024-02-19T15:29:53.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.830+0000 7f17af400640 1 -- 172.21.15.47:0/436421455 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1788066e40 msgr2=0x7f1788069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:53.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.830+0000 7f17af400640 1 --2- 172.21.15.47:0/436421455 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1788066e40 0x7f1788069300 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f179c01c4a0 tx=0x7f179c0b6040 comp rx=0 tx=0).stop 2024-02-19T15:29:53.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.830+0000 7f17af400640 1 -- 172.21.15.47:0/436421455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17a80cded0 msgr2=0x7f17a81fa3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:53.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.830+0000 7f17af400640 1 --2- 172.21.15.47:0/436421455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17a80cded0 0x7f17a81fa3a0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f179400c390 tx=0x7f179400c850 comp rx=0 tx=0).stop 2024-02-19T15:29:53.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.830+0000 7f17af400640 1 -- 172.21.15.47:0/436421455 shutdown_connections 2024-02-19T15:29:53.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.830+0000 7f17af400640 1 --2- 172.21.15.47:0/436421455 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1788066e40 0x7f1788069300 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:53.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.830+0000 7f17af400640 1 --2- 172.21.15.47:0/436421455 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17a80ced10 0x7f17a81fa8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:53.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.830+0000 7f17af400640 1 --2- 172.21.15.47:0/436421455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17a80cded0 0x7f17a81fa3a0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:53.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.830+0000 7f17af400640 1 --2- 172.21.15.47:0/436421455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a806d650 0x7f17a815e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:53.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.830+0000 7f17af400640 1 -- 172.21.15.47:0/436421455 >> 172.21.15.47:0/436421455 conn(0x7f17a80d1bf0 msgr2=0x7f17a80d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:53.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.830+0000 7f17af400640 1 -- 172.21.15.47:0/436421455 shutdown_connections 2024-02-19T15:29:53.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:53.830+0000 7f17af400640 1 -- 172.21.15.47:0/436421455 wait complete. 2024-02-19T15:29:54.050 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:24.882579+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:54.051 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:54.182 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:29:55.052 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:55.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.194+0000 7f4ca3800640 1 Processor -- start 2024-02-19T15:29:55.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.194+0000 7f4ca3800640 1 -- start start 2024-02-19T15:29:55.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.194+0000 7f4ca3800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0d0540 0x7f4c9c0cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:55.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.194+0000 7f4ca3800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4c9c0cd180 0x7f4c9c0b1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:55.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.194+0000 7f4ca3800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c9c0b0fe0 con 0x7f4c9c15c0d0 2024-02-19T15:29:55.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.194+0000 7f4ca3800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c9c15c4b0 con 0x7f4c9c0cd180 2024-02-19T15:29:55.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.194+0000 7f4ca3800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c0b1880 con 0x7f4c9c0d0540 2024-02-19T15:29:55.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.194+0000 7f4ca0e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f4c9c15c0d0 0x7f4c9c0cfe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36072/0 (socket says 172.21.15.47:36072) 2024-02-19T15:29:55.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.194+0000 7f4ca0e00640 1 -- 172.21.15.47:0/3184735903 learned_addr learned my addr 172.21.15.47:0/3184735903 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:29:55.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.194+0000 7f4ca1800640 1 --2- 172.21.15.47:0/3184735903 >> v2:172.21.15.116:3300/0 conn(0x7f4c9c0cd180 0x7f4c9c0b1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:55.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.194+0000 7f4c9be00640 1 --2- 172.21.15.47:0/3184735903 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0d0540 0x7f4c9c0cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:55.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4c9b400640 1 -- 172.21.15.47:0/3184735903 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3454652983 0 0) 0x7f4c9c0b0fe0 con 0x7f4c9c15c0d0 2024-02-19T15:29:55.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4c9b400640 1 -- 172.21.15.47:0/3184735903 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c84003530 con 0x7f4c9c15c0d0 2024-02-19T15:29:55.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4c9b400640 1 -- 172.21.15.47:0/3184735903 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3674830717 0 0) 0x7f4c84003530 con 0x7f4c9c15c0d0 2024-02-19T15:29:55.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4c9b400640 1 -- 172.21.15.47:0/3184735903 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0d0540 msgr2=0x7f4c9c0cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:55.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4c9b400640 1 --2- 172.21.15.47:0/3184735903 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0d0540 0x7f4c9c0cf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:55.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4c9b400640 1 -- 172.21.15.47:0/3184735903 >> v2:172.21.15.116:3300/0 conn(0x7f4c9c0cd180 msgr2=0x7f4c9c0b1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:55.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4c9b400640 1 --2- 172.21.15.47:0/3184735903 >> v2:172.21.15.116:3300/0 conn(0x7f4c9c0cd180 0x7f4c9c0b1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:55.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4c9b400640 1 -- 172.21.15.47:0/3184735903 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9c0cd8f0 con 0x7f4c9c15c0d0 2024-02-19T15:29:55.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4c9b400640 1 -- 172.21.15.47:0/3184735903 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f4c90001680 con 0x7f4c9c15c0d0 2024-02-19T15:29:55.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4c9b400640 1 -- 172.21.15.47:0/3184735903 >> v1:172.21.15.47:6789/0 conn(0x7f4c9c15c0d0 legacy=0x7f4c9c0cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:55.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4c9b400640 1 --2- 172.21.15.47:0/3184735903 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0d0540 0x7f4c84003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:55.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4c9b400640 1 --2- 172.21.15.47:0/3184735903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c84003f50 0x7f4c840047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:55.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4c9b400640 1 --2- 172.21.15.47:0/3184735903 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c84004d10 0x7f4c84005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:55.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4c9b400640 1 -- 172.21.15.47:0/3184735903 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c15c4b0 con 0x7f4c84004d10 2024-02-19T15:29:55.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4c9b400640 1 -- 172.21.15.47:0/3184735903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c0b1880 con 0x7f4c84003f50 2024-02-19T15:29:55.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4c9b400640 1 -- 172.21.15.47:0/3184735903 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8400b990 con 0x7f4c9c0d0540 2024-02-19T15:29:55.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4ca1800640 1 --2- 172.21.15.47:0/3184735903 >> v2:172.21.15.116:3300/0 conn(0x7f4c9c0cd180 0x7f4c9c0b1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:29:55.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4ca0e00640 1 --2- 172.21.15.47:0/3184735903 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0d0540 0x7f4c84003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:55.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4ca1800640 1 --2- 172.21.15.47:0/3184735903 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c84004d10 0x7f4c84005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:55.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4c9be00640 1 --2- 172.21.15.47:0/3184735903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c84003f50 0x7f4c840047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:55.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4ca1800640 1 -- 172.21.15.47:0/3184735903 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0d0540 msgr2=0x7f4c84003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:55.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4ca1800640 1 --2- 172.21.15.47:0/3184735903 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0d0540 0x7f4c84003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:55.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4ca1800640 1 -- 172.21.15.47:0/3184735903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c84003f50 msgr2=0x7f4c840047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:55.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4ca1800640 1 --2- 172.21.15.47:0/3184735903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c84003f50 0x7f4c840047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:55.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4ca1800640 1 -- 172.21.15.47:0/3184735903 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c84003530 con 0x7f4c84004d10 2024-02-19T15:29:55.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.198+0000 7f4ca0e00640 1 --2- 172.21.15.47:0/3184735903 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0d0540 0x7f4c84003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:29:55.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.202+0000 7f4ca1800640 1 --2- 172.21.15.47:0/3184735903 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c84004d10 0x7f4c84005190 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f4c9400e380 tx=0x7f4c9401ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93bcd54555ba99ad server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:55.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.202+0000 7f4c9b400640 1 -- 172.21.15.47:0/3184735903 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c940a5e50 con 0x7f4c84004d10 2024-02-19T15:29:55.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.202+0000 7f4c9b400640 1 -- 172.21.15.47:0/3184735903 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9400f040 con 0x7f4c84004d10 2024-02-19T15:29:55.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.202+0000 7f4c9b400640 1 -- 172.21.15.47:0/3184735903 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c94003ba0 con 0x7f4c84004d10 2024-02-19T15:29:55.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.202+0000 7f4ca3800640 1 -- 172.21.15.47:0/3184735903 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c84004d10 msgr2=0x7f4c84005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:55.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.202+0000 7f4ca3800640 1 --2- 172.21.15.47:0/3184735903 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c84004d10 0x7f4c84005190 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f4c9400e380 tx=0x7f4c9401ce40 comp rx=0 tx=0).stop 2024-02-19T15:29:55.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.202+0000 7f4ca1800640 1 -- 172.21.15.47:0/3184735903 reap_dead start 2024-02-19T15:29:55.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.202+0000 7f4ca3800640 1 -- 172.21.15.47:0/3184735903 shutdown_connections 2024-02-19T15:29:55.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.202+0000 7f4ca3800640 1 -- 172.21.15.47:0/3184735903 >> 172.21.15.47:0/3184735903 conn(0x7f4c9c0d1ba0 msgr2=0x7f4c9c0d1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:55.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.202+0000 7f4ca3800640 1 -- 172.21.15.47:0/3184735903 shutdown_connections 2024-02-19T15:29:55.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.202+0000 7f4ca3800640 1 -- 172.21.15.47:0/3184735903 wait complete. 2024-02-19T15:29:55.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4ca3800640 1 Processor -- start 2024-02-19T15:29:55.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4ca3800640 1 -- start start 2024-02-19T15:29:55.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4ca3800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c151340 0x7f4c9c0cb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:55.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4ca3800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c9c0cb3a0 0x7f4c9c06d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:55.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4ca3800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c9c0b5500 0x7f4c9c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:55.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4ca3800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c0b7990 con 0x7f4c9c0b5500 2024-02-19T15:29:55.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4ca3800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c0b6ff0 con 0x7f4c9c0cb3a0 2024-02-19T15:29:55.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4ca3800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c0b72d0 con 0x7f4c9c151340 2024-02-19T15:29:55.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4ca1800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c9c0b5500 0x7f4c9c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:55.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4ca1800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c9c0b5500 0x7f4c9c06d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:38294/0 (socket says 172.21.15.47:38294) 2024-02-19T15:29:55.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4ca1800640 1 -- 172.21.15.47:0/1724789790 learned_addr learned my addr 172.21.15.47:0/1724789790 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:55.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4ca0e00640 1 --2- 172.21.15.47:0/1724789790 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c151340 0x7f4c9c0cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:55.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4c9be00640 1 --2- 172.21.15.47:0/1724789790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c9c0cb3a0 0x7f4c9c06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:55.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4ca1800640 1 -- 172.21.15.47:0/1724789790 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c151340 msgr2=0x7f4c9c0cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:55.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4ca1800640 1 --2- 172.21.15.47:0/1724789790 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c151340 0x7f4c9c0cb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:55.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4ca1800640 1 -- 172.21.15.47:0/1724789790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c9c0cb3a0 msgr2=0x7f4c9c06d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:55.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4ca1800640 1 --2- 172.21.15.47:0/1724789790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c9c0cb3a0 0x7f4c9c06d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:55.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4ca1800640 1 -- 172.21.15.47:0/1724789790 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9c0b75b0 con 0x7f4c9c0b5500 2024-02-19T15:29:55.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4ca1800640 1 --2- 172.21.15.47:0/1724789790 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c9c0b5500 0x7f4c9c06d680 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f4c9401d6e0 tx=0x7f4c94009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:55.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4c98c00640 1 -- 172.21.15.47:0/1724789790 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c940039a0 con 0x7f4c9c0b5500 2024-02-19T15:29:55.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4c98c00640 1 -- 172.21.15.47:0/1724789790 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c94003b00 con 0x7f4c9c0b5500 2024-02-19T15:29:55.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4c98c00640 1 -- 172.21.15.47:0/1724789790 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c94003e70 con 0x7f4c9c0b5500 2024-02-19T15:29:55.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4ca3800640 1 -- 172.21.15.47:0/1724789790 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c9c0b6c10 con 0x7f4c9c0b5500 2024-02-19T15:29:55.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.206+0000 7f4ca3800640 1 -- 172.21.15.47:0/1724789790 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c9c1518f0 con 0x7f4c9c0b5500 2024-02-19T15:29:55.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.210+0000 7f4ca3800640 1 -- 172.21.15.47:0/1724789790 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c9c068500 con 0x7f4c9c0b5500 2024-02-19T15:29:55.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.210+0000 7f4c98c00640 1 -- 172.21.15.47:0/1724789790 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4c94004020 con 0x7f4c9c0b5500 2024-02-19T15:29:55.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.210+0000 7f4c98c00640 1 --2- 172.21.15.47:0/1724789790 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4c7c066e40 0x7f4c7c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:55.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.210+0000 7f4c98c00640 1 -- 172.21.15.47:0/1724789790 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4c9411fa10 con 0x7f4c9c0b5500 2024-02-19T15:29:55.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.210+0000 7f4ca0e00640 1 --2- 172.21.15.47:0/1724789790 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4c7c066e40 0x7f4c7c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:55.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.210+0000 7f4ca0e00640 1 --2- 172.21.15.47:0/1724789790 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4c7c066e40 0x7f4c7c069300 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f4c90004850 tx=0x7f4c90009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:55.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.214+0000 7f4c98c00640 1 -- 172.21.15.47:0/1724789790 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f4c940ec7b0 con 0x7f4c9c0b5500 2024-02-19T15:29:55.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.410+0000 7f4ca3800640 1 -- 172.21.15.47:0/1724789790 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c9c0b7ad0 con 0x7f4c9c0b5500 2024-02-19T15:29:55.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.410+0000 7f4c98c00640 1 -- 172.21.15.47:0/1724789790 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f4c940bd020 con 0x7f4c9c0b5500 2024-02-19T15:29:55.412 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:55.412 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:55.412 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:29:55.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.414+0000 7f4ca3800640 1 -- 172.21.15.47:0/1724789790 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4c7c066e40 msgr2=0x7f4c7c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:55.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.414+0000 7f4ca3800640 1 --2- 172.21.15.47:0/1724789790 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4c7c066e40 0x7f4c7c069300 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f4c90004850 tx=0x7f4c90009040 comp rx=0 tx=0).stop 2024-02-19T15:29:55.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.414+0000 7f4ca3800640 1 -- 172.21.15.47:0/1724789790 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c9c0b5500 msgr2=0x7f4c9c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:55.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.414+0000 7f4ca3800640 1 --2- 172.21.15.47:0/1724789790 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c9c0b5500 0x7f4c9c06d680 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f4c9401d6e0 tx=0x7f4c94009750 comp rx=0 tx=0).stop 2024-02-19T15:29:55.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.414+0000 7f4ca3800640 1 -- 172.21.15.47:0/1724789790 shutdown_connections 2024-02-19T15:29:55.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.414+0000 7f4ca3800640 1 --2- 172.21.15.47:0/1724789790 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4c7c066e40 0x7f4c7c069300 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:55.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.414+0000 7f4ca3800640 1 --2- 172.21.15.47:0/1724789790 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c9c0b5500 0x7f4c9c06d680 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:55.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.414+0000 7f4ca3800640 1 --2- 172.21.15.47:0/1724789790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c9c0cb3a0 0x7f4c9c06d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:55.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.414+0000 7f4ca3800640 1 --2- 172.21.15.47:0/1724789790 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c151340 0x7f4c9c0cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:55.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.414+0000 7f4ca3800640 1 -- 172.21.15.47:0/1724789790 >> 172.21.15.47:0/1724789790 conn(0x7f4c9c0d1ba0 msgr2=0x7f4c9c0ce1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:55.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.414+0000 7f4ca3800640 1 -- 172.21.15.47:0/1724789790 shutdown_connections 2024-02-19T15:29:55.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:55.414+0000 7f4ca3800640 1 -- 172.21.15.47:0/1724789790 wait complete. 2024-02-19T15:29:55.634 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:55.635 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:56.184 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:29:56.635 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:56.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.774+0000 7fd10d000640 1 Processor -- start 2024-02-19T15:29:56.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.778+0000 7fd10d000640 1 -- start start 2024-02-19T15:29:56.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.778+0000 7fd10d000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd1080717e0 0x7fd1080b1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:56.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.778+0000 7fd10d000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd10815ba70 0x7fd10815a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:56.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.778+0000 7fd10d000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1080b1a50 con 0x7fd108071400 2024-02-19T15:29:56.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.778+0000 7fd10d000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1080b2010 con 0x7fd1080717e0 2024-02-19T15:29:56.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.778+0000 7fd10d000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1080b1d30 con 0x7fd10815ba70 2024-02-19T15:29:56.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.778+0000 7fd0fea00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd1080717e0 0x7fd1080b1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:56.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.778+0000 7fd0fea00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd1080717e0 0x7fd1080b1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:38300/0 (socket says 172.21.15.47:38300) 2024-02-19T15:29:56.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.778+0000 7fd0fea00640 1 -- 172.21.15.47:0/2675663620 learned_addr learned my addr 172.21.15.47:0/2675663620 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:56.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.778+0000 7fd107400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd108071400 0x7fd108070990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36076/0 (socket says 172.21.15.47:36076) 2024-02-19T15:29:56.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.778+0000 7fd107e00640 1 --2- 172.21.15.47:0/2675663620 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd10815ba70 0x7fd10815a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:56.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.778+0000 7fd0fea00640 1 -- 172.21.15.47:0/2675663620 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd10815ba70 msgr2=0x7fd10815a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:56.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.778+0000 7fd0fea00640 1 --2- 172.21.15.47:0/2675663620 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd10815ba70 0x7fd10815a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:56.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.778+0000 7fd0fea00640 1 -- 172.21.15.47:0/2675663620 >> v1:172.21.15.47:6789/0 conn(0x7fd108071400 legacy=0x7fd108070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:56.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.778+0000 7fd0fea00640 1 -- 172.21.15.47:0/2675663620 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd10815b270 con 0x7fd1080717e0 2024-02-19T15:29:56.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.778+0000 7fd106a00640 1 -- 172.21.15.47:0/2675663620 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419843389 0 0) 0x7fd1080b1a50 con 0x7fd108071400 2024-02-19T15:29:56.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.782+0000 7fd0fea00640 1 --2- 172.21.15.47:0/2675663620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1080717e0 0x7fd1080b1400 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fd0f000bb10 tx=0x7fd0f001ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc43bd6c5586b94a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:56.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.782+0000 7fd106a00640 1 -- 172.21.15.47:0/2675663620 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f00a7cf0 con 0x7fd1080717e0 2024-02-19T15:29:56.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.782+0000 7fd106a00640 1 -- 172.21.15.47:0/2675663620 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f00a7e50 con 0x7fd1080717e0 2024-02-19T15:29:56.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.782+0000 7fd106a00640 1 -- 172.21.15.47:0/2675663620 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f00af660 con 0x7fd1080717e0 2024-02-19T15:29:56.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.782+0000 7fd10d000640 1 -- 172.21.15.47:0/2675663620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1080717e0 msgr2=0x7fd1080b1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:56.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.782+0000 7fd10d000640 1 --2- 172.21.15.47:0/2675663620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1080717e0 0x7fd1080b1400 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fd0f000bb10 tx=0x7fd0f001ec00 comp rx=0 tx=0).stop 2024-02-19T15:29:56.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.782+0000 7fd10d000640 1 -- 172.21.15.47:0/2675663620 shutdown_connections 2024-02-19T15:29:56.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.782+0000 7fd10d000640 1 --2- 172.21.15.47:0/2675663620 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd10815ba70 0x7fd10815a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:56.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.782+0000 7fd10d000640 1 --2- 172.21.15.47:0/2675663620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1080717e0 0x7fd1080b1400 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:56.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.782+0000 7fd10d000640 1 -- 172.21.15.47:0/2675663620 >> 172.21.15.47:0/2675663620 conn(0x7fd1080cf7c0 msgr2=0x7fd1080cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:56.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.782+0000 7fd10d000640 1 -- 172.21.15.47:0/2675663620 shutdown_connections 2024-02-19T15:29:56.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.782+0000 7fd10d000640 1 -- 172.21.15.47:0/2675663620 wait complete. 2024-02-19T15:29:56.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.782+0000 7fd10d000640 1 Processor -- start 2024-02-19T15:29:56.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.782+0000 7fd10d000640 1 -- start start 2024-02-19T15:29:56.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd10d000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd108071400 0x7fd1080b53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:56.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd10d000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1080717e0 0x7fd10814ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:56.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd10d000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd10815ba70 0x7fd10813fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:56.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd10d000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10814f4b0 con 0x7fd10815ba70 2024-02-19T15:29:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd10d000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1080b4f60 con 0x7fd1080717e0 2024-02-19T15:29:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd10d000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd10813cd20 con 0x7fd108071400 2024-02-19T15:29:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd107e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd10815ba70 0x7fd10813fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd107e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd10815ba70 0x7fd10813fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:38312/0 (socket says 172.21.15.47:38312) 2024-02-19T15:29:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd107e00640 1 -- 172.21.15.47:0/2977253505 learned_addr learned my addr 172.21.15.47:0/2977253505 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd107400640 1 --2- 172.21.15.47:0/2977253505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd108071400 0x7fd1080b53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd0fea00640 1 --2- 172.21.15.47:0/2977253505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1080717e0 0x7fd10814ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd107e00640 1 -- 172.21.15.47:0/2977253505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd108071400 msgr2=0x7fd1080b53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd107e00640 1 --2- 172.21.15.47:0/2977253505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd108071400 0x7fd1080b53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd107e00640 1 -- 172.21.15.47:0/2977253505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1080717e0 msgr2=0x7fd10814ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd107e00640 1 --2- 172.21.15.47:0/2977253505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1080717e0 0x7fd10814ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd107e00640 1 -- 172.21.15.47:0/2977253505 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f000b770 con 0x7fd10815ba70 2024-02-19T15:29:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd107e00640 1 --2- 172.21.15.47:0/2977253505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd10815ba70 0x7fd10813fa40 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fd10000d560 tx=0x7fd10000da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd0ffe00640 1 -- 172.21.15.47:0/2977253505 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1000125f0 con 0x7fd10815ba70 2024-02-19T15:29:56.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd0ffe00640 1 -- 172.21.15.47:0/2977253505 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd100012770 con 0x7fd10815ba70 2024-02-19T15:29:56.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd0ffe00640 1 -- 172.21.15.47:0/2977253505 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd100012b20 con 0x7fd10815ba70 2024-02-19T15:29:56.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd10d000640 1 -- 172.21.15.47:0/2977253505 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd10813ff80 con 0x7fd10815ba70 2024-02-19T15:29:56.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd10d000640 1 -- 172.21.15.47:0/2977253505 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1080b4630 con 0x7fd10815ba70 2024-02-19T15:29:56.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd10d000640 1 -- 172.21.15.47:0/2977253505 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1080b50f0 con 0x7fd10815ba70 2024-02-19T15:29:56.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.786+0000 7fd0ffe00640 1 -- 172.21.15.47:0/2977253505 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd100012c80 con 0x7fd10815ba70 2024-02-19T15:29:56.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.790+0000 7fd0ffe00640 1 --2- 172.21.15.47:0/2977253505 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd0e4066e40 0x7fd0e4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:56.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.790+0000 7fd0ffe00640 1 -- 172.21.15.47:0/2977253505 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd100084f40 con 0x7fd10815ba70 2024-02-19T15:29:56.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.790+0000 7fd107400640 1 --2- 172.21.15.47:0/2977253505 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd0e4066e40 0x7fd0e4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:56.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.790+0000 7fd107400640 1 --2- 172.21.15.47:0/2977253505 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd0e4066e40 0x7fd0e4069300 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8007fd0 tx=0x7fd0f8007f20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:56.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.794+0000 7fd0ffe00640 1 -- 172.21.15.47:0/2977253505 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd100051cf0 con 0x7fd10815ba70 2024-02-19T15:29:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.986+0000 7fd10d000640 1 -- 172.21.15.47:0/2977253505 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd108075360 con 0x7fd10815ba70 2024-02-19T15:29:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.990+0000 7fd0ffe00640 1 -- 172.21.15.47:0/2977253505 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7fd1000553b0 con 0x7fd10815ba70 2024-02-19T15:29:56.988 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:56.988 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:56.989 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:29:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.990+0000 7fd10d000640 1 -- 172.21.15.47:0/2977253505 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd0e4066e40 msgr2=0x7fd0e4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.990+0000 7fd10d000640 1 --2- 172.21.15.47:0/2977253505 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd0e4066e40 0x7fd0e4069300 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8007fd0 tx=0x7fd0f8007f20 comp rx=0 tx=0).stop 2024-02-19T15:29:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.990+0000 7fd10d000640 1 -- 172.21.15.47:0/2977253505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd10815ba70 msgr2=0x7fd10813fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.990+0000 7fd10d000640 1 --2- 172.21.15.47:0/2977253505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd10815ba70 0x7fd10813fa40 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fd10000d560 tx=0x7fd10000da20 comp rx=0 tx=0).stop 2024-02-19T15:29:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.990+0000 7fd10d000640 1 -- 172.21.15.47:0/2977253505 shutdown_connections 2024-02-19T15:29:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.990+0000 7fd10d000640 1 --2- 172.21.15.47:0/2977253505 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd0e4066e40 0x7fd0e4069300 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.990+0000 7fd10d000640 1 --2- 172.21.15.47:0/2977253505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd10815ba70 0x7fd10813fa40 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.990+0000 7fd10d000640 1 --2- 172.21.15.47:0/2977253505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1080717e0 0x7fd10814ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.990+0000 7fd10d000640 1 --2- 172.21.15.47:0/2977253505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd108071400 0x7fd1080b53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.990+0000 7fd10d000640 1 -- 172.21.15.47:0/2977253505 >> 172.21.15.47:0/2977253505 conn(0x7fd1080cf7c0 msgr2=0x7fd10806df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.990+0000 7fd10d000640 1 -- 172.21.15.47:0/2977253505 shutdown_connections 2024-02-19T15:29:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:56.990+0000 7fd10d000640 1 -- 172.21.15.47:0/2977253505 wait complete. 2024-02-19T15:29:57.212 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:57.212 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:58.187 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:29:58.213 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:58.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f6090e00640 1 Processor -- start 2024-02-19T15:29:58.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f6090e00640 1 -- start start 2024-02-19T15:29:58.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f6090e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f608c0d20f0 0x7f608c06d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:58.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f6090e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608c06c3e0 con 0x7f608c06d120 2024-02-19T15:29:58.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f6090e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608c06c680 con 0x7f608c15dd70 2024-02-19T15:29:58.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f6090e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f608c06daf0 con 0x7f608c0d20f0 2024-02-19T15:29:58.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f6083e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f608c06d120 0x7f608c0d19e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36090/0 (socket says 172.21.15.47:36090) 2024-02-19T15:29:58.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f6083e00640 1 -- 172.21.15.47:0/3147023472 learned_addr learned my addr 172.21.15.47:0/3147023472 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:29:58.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f608be00640 1 --2- 172.21.15.47:0/3147023472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f608c0d20f0 0x7f608c06d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:58.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f608aa00640 1 -- 172.21.15.47:0/3147023472 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649269187 0 0) 0x7f608c06c3e0 con 0x7f608c06d120 2024-02-19T15:29:58.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f608aa00640 1 -- 172.21.15.47:0/3147023472 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6070003530 con 0x7f608c06d120 2024-02-19T15:29:58.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f608aa00640 1 -- 172.21.15.47:0/3147023472 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3788352295 0 0) 0x7f6070003530 con 0x7f608c06d120 2024-02-19T15:29:58.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f608aa00640 1 -- 172.21.15.47:0/3147023472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f608c0d20f0 msgr2=0x7f608c06d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:58.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f608aa00640 1 --2- 172.21.15.47:0/3147023472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f608c0d20f0 0x7f608c06d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:58.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f608aa00640 1 -- 172.21.15.47:0/3147023472 >> v1:172.21.15.116:6789/0 conn(0x7f608c15dd70 legacy=0x7f608c0ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:58.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f608aa00640 1 -- 172.21.15.47:0/3147023472 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f608c15f110 con 0x7f608c06d120 2024-02-19T15:29:58.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f608aa00640 1 -- 172.21.15.47:0/3147023472 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f6074001680 con 0x7f608c06d120 2024-02-19T15:29:58.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f608aa00640 1 -- 172.21.15.47:0/3147023472 >> v1:172.21.15.47:6789/0 conn(0x7f608c06d120 legacy=0x7f608c0d19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:58.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f608aa00640 1 --2- 172.21.15.47:0/3147023472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60700039b0 0x7f6070003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:58.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f608aa00640 1 --2- 172.21.15.47:0/3147023472 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6070004360 0x7f6070004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:58.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.358+0000 7f608aa00640 1 --2- 172.21.15.47:0/3147023472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f60700050d0 0x7f607000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:58.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.362+0000 7f608aa00640 1 -- 172.21.15.47:0/3147023472 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607000bf80 con 0x7f6070004360 2024-02-19T15:29:58.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.362+0000 7f608aa00640 1 -- 172.21.15.47:0/3147023472 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607000c0f0 con 0x7f60700039b0 2024-02-19T15:29:58.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.362+0000 7f608aa00640 1 -- 172.21.15.47:0/3147023472 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f607000c2b0 con 0x7f60700050d0 2024-02-19T15:29:58.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.362+0000 7f608be00640 1 --2- 172.21.15.47:0/3147023472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f608c0d20f0 0x7f608c06d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:29:58.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.362+0000 7f6083e00640 1 --2- 172.21.15.47:0/3147023472 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6070004360 0x7f6070004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:58.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.362+0000 7f608b400640 1 --2- 172.21.15.47:0/3147023472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60700039b0 0x7f6070003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:58.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.362+0000 7f608be00640 1 --2- 172.21.15.47:0/3147023472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f60700050d0 0x7f607000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:58.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.362+0000 7f6083e00640 1 -- 172.21.15.47:0/3147023472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f60700050d0 msgr2=0x7f607000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:58.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.362+0000 7f6083e00640 1 --2- 172.21.15.47:0/3147023472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f60700050d0 0x7f607000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:58.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.362+0000 7f6083e00640 1 -- 172.21.15.47:0/3147023472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60700039b0 msgr2=0x7f6070003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:58.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.362+0000 7f6083e00640 1 --2- 172.21.15.47:0/3147023472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60700039b0 0x7f6070003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:58.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.362+0000 7f6083e00640 1 -- 172.21.15.47:0/3147023472 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6070003530 con 0x7f6070004360 2024-02-19T15:29:58.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.366+0000 7f6083e00640 1 --2- 172.21.15.47:0/3147023472 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6070004360 0x7f6070004830 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f607400c190 tx=0x7f607401cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13b9c006c0f56ce9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:58.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.366+0000 7f608aa00640 1 -- 172.21.15.47:0/3147023472 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6074005ae0 con 0x7f6070004360 2024-02-19T15:29:58.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.366+0000 7f608aa00640 1 -- 172.21.15.47:0/3147023472 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6074005c60 con 0x7f6070004360 2024-02-19T15:29:58.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.366+0000 7f608aa00640 1 -- 172.21.15.47:0/3147023472 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6074007070 con 0x7f6070004360 2024-02-19T15:29:58.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.366+0000 7f6090e00640 1 -- 172.21.15.47:0/3147023472 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6070004360 msgr2=0x7f6070004830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:58.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.366+0000 7f6090e00640 1 --2- 172.21.15.47:0/3147023472 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6070004360 0x7f6070004830 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f607400c190 tx=0x7f607401cbe0 comp rx=0 tx=0).stop 2024-02-19T15:29:58.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.366+0000 7f608be00640 1 -- 172.21.15.47:0/3147023472 reap_dead start 2024-02-19T15:29:58.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.366+0000 7f6090e00640 1 -- 172.21.15.47:0/3147023472 shutdown_connections 2024-02-19T15:29:58.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.366+0000 7f6090e00640 1 -- 172.21.15.47:0/3147023472 >> 172.21.15.47:0/3147023472 conn(0x7f608c09a9d0 msgr2=0x7f608c09ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:58.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.366+0000 7f6090e00640 1 -- 172.21.15.47:0/3147023472 shutdown_connections 2024-02-19T15:29:58.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.366+0000 7f6090e00640 1 -- 172.21.15.47:0/3147023472 wait complete. 2024-02-19T15:29:58.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f6090e00640 1 Processor -- start 2024-02-19T15:29:58.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f6090e00640 1 -- start start 2024-02-19T15:29:58.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f6090e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6070004360 0x7f608c1541c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:58.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f6090e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f608c14f840 0x7f608c0b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:58.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f6090e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f608c14fca0 0x7f608c1f70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:58.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f6090e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608c0b6050 con 0x7f6070004360 2024-02-19T15:29:58.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f6090e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608c0b5d70 con 0x7f608c14f840 2024-02-19T15:29:58.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f6090e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f608c151380 con 0x7f608c14fca0 2024-02-19T15:29:58.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f608b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6070004360 0x7f608c1541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:58.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f608b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6070004360 0x7f608c1541c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:38328/0 (socket says 172.21.15.47:38328) 2024-02-19T15:29:58.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f608b400640 1 -- 172.21.15.47:0/10053411 learned_addr learned my addr 172.21.15.47:0/10053411 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:58.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f608be00640 1 --2- 172.21.15.47:0/10053411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f608c14fca0 0x7f608c1f70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:58.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f6083e00640 1 --2- 172.21.15.47:0/10053411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f608c14f840 0x7f608c0b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:58.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f608b400640 1 -- 172.21.15.47:0/10053411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f608c14fca0 msgr2=0x7f608c1f70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:58.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f608b400640 1 --2- 172.21.15.47:0/10053411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f608c14fca0 0x7f608c1f70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:58.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f608b400640 1 -- 172.21.15.47:0/10053411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f608c14f840 msgr2=0x7f608c0b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:58.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f608b400640 1 --2- 172.21.15.47:0/10053411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f608c14f840 0x7f608c0b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:58.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f608b400640 1 -- 172.21.15.47:0/10053411 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f608c151660 con 0x7f6070004360 2024-02-19T15:29:58.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f608b400640 1 --2- 172.21.15.47:0/10053411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6070004360 0x7f608c1541c0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f607c00f7f0 tx=0x7f607c00fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:58.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f6083400640 1 -- 172.21.15.47:0/10053411 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607c00fea0 con 0x7f6070004360 2024-02-19T15:29:58.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f6083400640 1 -- 172.21.15.47:0/10053411 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607c0052e0 con 0x7f6070004360 2024-02-19T15:29:58.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f6083400640 1 -- 172.21.15.47:0/10053411 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607c00d970 con 0x7f6070004360 2024-02-19T15:29:58.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f6090e00640 1 -- 172.21.15.47:0/10053411 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f608c151940 con 0x7f6070004360 2024-02-19T15:29:58.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f6090e00640 1 -- 172.21.15.47:0/10053411 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f608c0b5830 con 0x7f6070004360 2024-02-19T15:29:58.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.370+0000 7f6090e00640 1 -- 172.21.15.47:0/10053411 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f608c1511e0 con 0x7f6070004360 2024-02-19T15:29:58.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.374+0000 7f6083400640 1 -- 172.21.15.47:0/10053411 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f607c00dad0 con 0x7f6070004360 2024-02-19T15:29:58.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.374+0000 7f6083400640 1 --2- 172.21.15.47:0/10053411 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6068066e70 0x7f6068069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:58.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.374+0000 7f6083400640 1 -- 172.21.15.47:0/10053411 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f607c084a70 con 0x7f6070004360 2024-02-19T15:29:58.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.374+0000 7f6083e00640 1 --2- 172.21.15.47:0/10053411 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6068066e70 0x7f6068069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:58.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.374+0000 7f6083e00640 1 --2- 172.21.15.47:0/10053411 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6068066e70 0x7f6068069330 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f60740ab000 tx=0x7f60740a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:58.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.382+0000 7f6083400640 1 -- 172.21.15.47:0/10053411 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f608c1511e0 con 0x7f6070004360 2024-02-19T15:29:58.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.578+0000 7f6090e00640 1 -- 172.21.15.47:0/10053411 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f608c1511e0 con 0x7f6070004360 2024-02-19T15:29:58.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.578+0000 7f6083400640 1 -- 172.21.15.47:0/10053411 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f608c1511e0 con 0x7f6070004360 2024-02-19T15:29:58.577 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:29:58.577 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:29:58.578 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:29:58.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.578+0000 7f6090e00640 1 -- 172.21.15.47:0/10053411 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6068066e70 msgr2=0x7f6068069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:58.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.578+0000 7f6090e00640 1 --2- 172.21.15.47:0/10053411 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6068066e70 0x7f6068069330 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f60740ab000 tx=0x7f60740a7070 comp rx=0 tx=0).stop 2024-02-19T15:29:58.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.578+0000 7f6090e00640 1 -- 172.21.15.47:0/10053411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6070004360 msgr2=0x7f608c1541c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:58.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.578+0000 7f6090e00640 1 --2- 172.21.15.47:0/10053411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6070004360 0x7f608c1541c0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f607c00f7f0 tx=0x7f607c00fcb0 comp rx=0 tx=0).stop 2024-02-19T15:29:58.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.578+0000 7f6090e00640 1 -- 172.21.15.47:0/10053411 shutdown_connections 2024-02-19T15:29:58.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.578+0000 7f6090e00640 1 --2- 172.21.15.47:0/10053411 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6068066e70 0x7f6068069330 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:58.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.578+0000 7f6090e00640 1 --2- 172.21.15.47:0/10053411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f608c14fca0 0x7f608c1f70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:58.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.578+0000 7f6090e00640 1 --2- 172.21.15.47:0/10053411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f608c14f840 0x7f608c0b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:58.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.578+0000 7f6090e00640 1 --2- 172.21.15.47:0/10053411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6070004360 0x7f608c1541c0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:58.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.578+0000 7f6090e00640 1 -- 172.21.15.47:0/10053411 >> 172.21.15.47:0/10053411 conn(0x7f608c09a9d0 msgr2=0x7f608c0b2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:58.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.578+0000 7f6090e00640 1 -- 172.21.15.47:0/10053411 shutdown_connections 2024-02-19T15:29:58.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:58.578+0000 7f6090e00640 1 -- 172.21.15.47:0/10053411 wait complete. 2024-02-19T15:29:58.800 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:29:58.801 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:29:59.801 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:29:59.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.946+0000 7fed3be00640 1 Processor -- start 2024-02-19T15:29:59.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.946+0000 7fed3be00640 1 -- start start 2024-02-19T15:29:59.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.946+0000 7fed3be00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed3415e250 0x7fed340d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:59.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.946+0000 7fed3be00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fed340cfb30 0x7fed340d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:59.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.946+0000 7fed3be00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed340d4970 con 0x7fed340d0e70 2024-02-19T15:29:59.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.946+0000 7fed3be00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed340d2eb0 con 0x7fed340cfb30 2024-02-19T15:29:59.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.946+0000 7fed3be00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed340b3240 con 0x7fed3415e250 2024-02-19T15:29:59.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.946+0000 7fed39e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fed340d0e70 0x7fed340ce940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36096/0 (socket says 172.21.15.47:36096) 2024-02-19T15:29:59.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.946+0000 7fed39e00640 1 -- 172.21.15.47:0/2367710111 learned_addr learned my addr 172.21.15.47:0/2367710111 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:29:59.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.946+0000 7fed39400640 1 --2- 172.21.15.47:0/2367710111 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed3415e250 0x7fed340d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:59.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.946+0000 7fed38a00640 1 --2- 172.21.15.47:0/2367710111 >> v2:172.21.15.116:3300/0 conn(0x7fed340cfb30 0x7fed340d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:59.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.946+0000 7fed33e00640 1 -- 172.21.15.47:0/2367710111 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720905727 0 0) 0x7fed340d4970 con 0x7fed340d0e70 2024-02-19T15:29:59.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.946+0000 7fed33e00640 1 -- 172.21.15.47:0/2367710111 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed1c003530 con 0x7fed340d0e70 2024-02-19T15:29:59.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.946+0000 7fed33e00640 1 -- 172.21.15.47:0/2367710111 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1265201833 0 0) 0x7fed1c003530 con 0x7fed340d0e70 2024-02-19T15:29:59.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.946+0000 7fed39400640 1 -- 172.21.15.47:0/2367710111 >> v2:172.21.15.116:3300/0 conn(0x7fed340cfb30 msgr2=0x7fed340d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:29:59.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.946+0000 7fed39400640 1 --2- 172.21.15.47:0/2367710111 >> v2:172.21.15.116:3300/0 conn(0x7fed340cfb30 0x7fed340d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:59.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.946+0000 7fed39400640 1 -- 172.21.15.47:0/2367710111 >> v1:172.21.15.47:6789/0 conn(0x7fed340d0e70 legacy=0x7fed340ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:59.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.946+0000 7fed39400640 1 -- 172.21.15.47:0/2367710111 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3415e630 con 0x7fed3415e250 2024-02-19T15:29:59.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.946+0000 7fed38a00640 1 --2- 172.21.15.47:0/2367710111 >> v2:172.21.15.116:3300/0 conn(0x7fed340cfb30 0x7fed340d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:29:59.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.950+0000 7fed39400640 1 --2- 172.21.15.47:0/2367710111 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed3415e250 0x7fed340d2430 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fed280097c0 tx=0x7fed2801c270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c153c647c0c479a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:59.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.950+0000 7fed33e00640 1 -- 172.21.15.47:0/2367710111 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2c001a00 con 0x7fed3415e250 2024-02-19T15:29:59.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.950+0000 7fed33e00640 1 -- 172.21.15.47:0/2367710111 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed280a7050 con 0x7fed3415e250 2024-02-19T15:29:59.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.950+0000 7fed33e00640 1 -- 172.21.15.47:0/2367710111 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed280ab620 con 0x7fed3415e250 2024-02-19T15:29:59.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.950+0000 7fed3be00640 1 -- 172.21.15.47:0/2367710111 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed3415e250 msgr2=0x7fed340d2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:59.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.950+0000 7fed3be00640 1 --2- 172.21.15.47:0/2367710111 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed3415e250 0x7fed340d2430 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fed280097c0 tx=0x7fed2801c270 comp rx=0 tx=0).stop 2024-02-19T15:29:59.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.950+0000 7fed3be00640 1 -- 172.21.15.47:0/2367710111 shutdown_connections 2024-02-19T15:29:59.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.950+0000 7fed3be00640 1 --2- 172.21.15.47:0/2367710111 >> v2:172.21.15.116:3300/0 conn(0x7fed340cfb30 0x7fed340d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:59.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.950+0000 7fed3be00640 1 --2- 172.21.15.47:0/2367710111 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed3415e250 0x7fed340d2430 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:59.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.950+0000 7fed3be00640 1 -- 172.21.15.47:0/2367710111 >> 172.21.15.47:0/2367710111 conn(0x7fed340d3e50 msgr2=0x7fed340d4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:29:59.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.950+0000 7fed3be00640 1 -- 172.21.15.47:0/2367710111 shutdown_connections 2024-02-19T15:29:59.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.950+0000 7fed3be00640 1 -- 172.21.15.47:0/2367710111 wait complete. 2024-02-19T15:29:59.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed3be00640 1 Processor -- start 2024-02-19T15:29:59.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed3be00640 1 -- start start 2024-02-19T15:29:59.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed3be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed340cfb30 0x7fed340cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:59.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed3be00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed340d0e70 0x7fed340cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:59.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed3be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed3415e250 0x7fed341559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:59.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed3be00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed340cb630 con 0x7fed3415e250 2024-02-19T15:29:59.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed3be00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed340cda90 con 0x7fed340cfb30 2024-02-19T15:29:59.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed3be00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed34154ca0 con 0x7fed340d0e70 2024-02-19T15:29:59.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed39e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed3415e250 0x7fed341559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:59.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed38a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed340d0e70 0x7fed340cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:59.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed39400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed340cfb30 0x7fed340cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:59.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed38a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed340d0e70 0x7fed340cd550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:51450/0 (socket says 172.21.15.47:51450) 2024-02-19T15:29:59.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed38a00640 1 -- 172.21.15.47:0/2045506219 learned_addr learned my addr 172.21.15.47:0/2045506219 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:29:59.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed38a00640 1 -- 172.21.15.47:0/2045506219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed340cfb30 msgr2=0x7fed340cb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:59.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed38a00640 1 --2- 172.21.15.47:0/2045506219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed340cfb30 0x7fed340cb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:59.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed38a00640 1 -- 172.21.15.47:0/2045506219 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed3415e250 msgr2=0x7fed341559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:29:59.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed38a00640 1 --2- 172.21.15.47:0/2045506219 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed3415e250 0x7fed341559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:29:59.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed38a00640 1 -- 172.21.15.47:0/2045506219 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed20009420 con 0x7fed340d0e70 2024-02-19T15:29:59.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed38a00640 1 --2- 172.21.15.47:0/2045506219 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed340d0e70 0x7fed340cd550 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fed2000ebe0 tx=0x7fed2000c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:59.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed31600640 1 -- 172.21.15.47:0/2045506219 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2000eee0 con 0x7fed340d0e70 2024-02-19T15:29:59.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed31600640 1 -- 172.21.15.47:0/2045506219 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed2000cbd0 con 0x7fed340d0e70 2024-02-19T15:29:59.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed31600640 1 -- 172.21.15.47:0/2045506219 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed20013470 con 0x7fed340d0e70 2024-02-19T15:29:59.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed3be00640 1 -- 172.21.15.47:0/2045506219 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed28009420 con 0x7fed340d0e70 2024-02-19T15:29:59.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.954+0000 7fed3be00640 1 -- 172.21.15.47:0/2045506219 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed34155540 con 0x7fed340d0e70 2024-02-19T15:29:59.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.958+0000 7fed3be00640 1 -- 172.21.15.47:0/2045506219 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed34068500 con 0x7fed340d0e70 2024-02-19T15:29:59.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.958+0000 7fed31600640 1 -- 172.21.15.47:0/2045506219 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fed200135d0 con 0x7fed340d0e70 2024-02-19T15:29:59.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.958+0000 7fed31600640 1 --2- 172.21.15.47:0/2045506219 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fed14066e40 0x7fed14069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:29:59.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.958+0000 7fed31600640 1 -- 172.21.15.47:0/2045506219 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fed20084cd0 con 0x7fed340d0e70 2024-02-19T15:29:59.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.958+0000 7fed39400640 1 --2- 172.21.15.47:0/2045506219 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fed14066e40 0x7fed14069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:29:59.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.958+0000 7fed39400640 1 --2- 172.21.15.47:0/2045506219 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fed14066e40 0x7fed14069300 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fed280ad000 tx=0x7fed280a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:29:59.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:29:59.962+0000 7fed31600640 1 -- 172.21.15.47:0/2045506219 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fed20051a90 con 0x7fed340d0e70 2024-02-19T15:30:00.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:00.162+0000 7fed3be00640 1 -- 172.21.15.47:0/2045506219 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed340b6510 con 0x7fed340d0e70 2024-02-19T15:30:00.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:00.162+0000 7fed31600640 1 -- 172.21.15.47:0/2045506219 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7fed20021d80 con 0x7fed340d0e70 2024-02-19T15:30:00.161 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:00.161 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:00.161 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:00.162+0000 7fed3be00640 1 -- 172.21.15.47:0/2045506219 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fed14066e40 msgr2=0x7fed14069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:00.162+0000 7fed3be00640 1 --2- 172.21.15.47:0/2045506219 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fed14066e40 0x7fed14069300 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fed280ad000 tx=0x7fed280a6000 comp rx=0 tx=0).stop 2024-02-19T15:30:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:00.162+0000 7fed3be00640 1 -- 172.21.15.47:0/2045506219 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed340d0e70 msgr2=0x7fed340cd550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:00.162+0000 7fed3be00640 1 --2- 172.21.15.47:0/2045506219 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed340d0e70 0x7fed340cd550 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fed2000ebe0 tx=0x7fed2000c520 comp rx=0 tx=0).stop 2024-02-19T15:30:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:00.166+0000 7fed3be00640 1 -- 172.21.15.47:0/2045506219 shutdown_connections 2024-02-19T15:30:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:00.166+0000 7fed3be00640 1 --2- 172.21.15.47:0/2045506219 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fed14066e40 0x7fed14069300 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:00.166+0000 7fed3be00640 1 --2- 172.21.15.47:0/2045506219 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed3415e250 0x7fed341559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:00.166+0000 7fed3be00640 1 --2- 172.21.15.47:0/2045506219 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed340d0e70 0x7fed340cd550 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:00.166+0000 7fed3be00640 1 --2- 172.21.15.47:0/2045506219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed340cfb30 0x7fed340cb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:00.166+0000 7fed3be00640 1 -- 172.21.15.47:0/2045506219 >> 172.21.15.47:0/2045506219 conn(0x7fed340d3e50 msgr2=0x7fed340b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:00.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:00.166+0000 7fed3be00640 1 -- 172.21.15.47:0/2045506219 shutdown_connections 2024-02-19T15:30:00.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:00.166+0000 7fed3be00640 1 -- 172.21.15.47:0/2045506219 wait complete. 2024-02-19T15:30:00.188 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:00.385 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:00.385 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:01.386 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:01.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.526+0000 7f8f0b400640 1 Processor -- start 2024-02-19T15:30:01.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.526+0000 7f8f0b400640 1 -- start start 2024-02-19T15:30:01.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.530+0000 7f8f0b400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8f0406b5c0 0x7f8f04166ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:01.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.530+0000 7f8f0b400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f04145a80 con 0x7f8f041456a0 2024-02-19T15:30:01.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.530+0000 7f8f0b400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f040d1880 con 0x7f8f040cf670 2024-02-19T15:30:01.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.530+0000 7f8f0b400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f040d03c0 con 0x7f8f0406b5c0 2024-02-19T15:30:01.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.530+0000 7f8f08a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8f041456a0 0x7f8f040cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36108/0 (socket says 172.21.15.47:36108) 2024-02-19T15:30:01.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.530+0000 7f8f08a00640 1 -- 172.21.15.47:0/3573392345 learned_addr learned my addr 172.21.15.47:0/3573392345 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:01.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.530+0000 7f8f09400640 1 --2- 172.21.15.47:0/3573392345 >> v2:172.21.15.116:3300/0 conn(0x7f8f0406b5c0 0x7f8f04166ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:01.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.530+0000 7f8f03e00640 1 -- 172.21.15.47:0/3573392345 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485381167 0 0) 0x7f8f04145a80 con 0x7f8f041456a0 2024-02-19T15:30:01.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.530+0000 7f8f03e00640 1 -- 172.21.15.47:0/3573392345 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee8003530 con 0x7f8f041456a0 2024-02-19T15:30:01.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.530+0000 7f8f09400640 1 -- 172.21.15.47:0/3573392345 >> v1:172.21.15.116:6789/0 conn(0x7f8f040cf670 legacy=0x7f8f0406aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:01.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.530+0000 7f8f09400640 1 -- 172.21.15.47:0/3573392345 >> v1:172.21.15.47:6789/0 conn(0x7f8f041456a0 legacy=0x7f8f040cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:01.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.530+0000 7f8f09400640 1 -- 172.21.15.47:0/3573392345 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f040cfb20 con 0x7f8f0406b5c0 2024-02-19T15:30:01.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.534+0000 7f8f09400640 1 --2- 172.21.15.47:0/3573392345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0406b5c0 0x7f8f04166ba0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f8efc009d90 tx=0x7f8efc01c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b00d6a57c157c29 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:01.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.534+0000 7f8f03e00640 1 -- 172.21.15.47:0/3573392345 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc0a5920 con 0x7f8f0406b5c0 2024-02-19T15:30:01.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.534+0000 7f8f03e00640 1 -- 172.21.15.47:0/3573392345 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8efc0a5a80 con 0x7f8f0406b5c0 2024-02-19T15:30:01.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.534+0000 7f8f03e00640 1 -- 172.21.15.47:0/3573392345 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc0a5e10 con 0x7f8f0406b5c0 2024-02-19T15:30:01.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.534+0000 7f8f0b400640 1 -- 172.21.15.47:0/3573392345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0406b5c0 msgr2=0x7f8f04166ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:01.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.534+0000 7f8f0b400640 1 --2- 172.21.15.47:0/3573392345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0406b5c0 0x7f8f04166ba0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f8efc009d90 tx=0x7f8efc01c970 comp rx=0 tx=0).stop 2024-02-19T15:30:01.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.534+0000 7f8f0b400640 1 -- 172.21.15.47:0/3573392345 shutdown_connections 2024-02-19T15:30:01.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.534+0000 7f8f0b400640 1 --2- 172.21.15.47:0/3573392345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0406b5c0 0x7f8f04166ba0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:01.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.534+0000 7f8f0b400640 1 -- 172.21.15.47:0/3573392345 >> 172.21.15.47:0/3573392345 conn(0x7f8f040d0ad0 msgr2=0x7f8f040d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:01.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.534+0000 7f8f0b400640 1 -- 172.21.15.47:0/3573392345 shutdown_connections 2024-02-19T15:30:01.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.534+0000 7f8f0b400640 1 -- 172.21.15.47:0/3573392345 wait complete. 2024-02-19T15:30:01.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.534+0000 7f8f0b400640 1 Processor -- start 2024-02-19T15:30:01.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.534+0000 7f8f0b400640 1 -- start start 2024-02-19T15:30:01.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8f0b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0406b5c0 0x7f8f0413c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:01.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8f0b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f040cf670 0x7f8f040cb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:01.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8f0b400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f041456a0 0x7f8f0413b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:01.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8f0b400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f040cc9e0 con 0x7f8f040cf670 2024-02-19T15:30:01.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8f0b400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f04140040 con 0x7f8f0406b5c0 2024-02-19T15:30:01.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8f0b400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f040cc140 con 0x7f8f041456a0 2024-02-19T15:30:01.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8efbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f040cf670 0x7f8f040cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:01.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8efbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f040cf670 0x7f8f040cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:38348/0 (socket says 172.21.15.47:38348) 2024-02-19T15:30:01.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8efbe00640 1 -- 172.21.15.47:0/2656740912 learned_addr learned my addr 172.21.15.47:0/2656740912 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:01.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8f09400640 1 --2- 172.21.15.47:0/2656740912 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f041456a0 0x7f8f0413b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:01.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8f08a00640 1 --2- 172.21.15.47:0/2656740912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0406b5c0 0x7f8f0413c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:01.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8efbe00640 1 -- 172.21.15.47:0/2656740912 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f041456a0 msgr2=0x7f8f0413b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:01.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8efbe00640 1 --2- 172.21.15.47:0/2656740912 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f041456a0 0x7f8f0413b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:01.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8efbe00640 1 -- 172.21.15.47:0/2656740912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0406b5c0 msgr2=0x7f8f0413c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:01.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8efbe00640 1 --2- 172.21.15.47:0/2656740912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0406b5c0 0x7f8f0413c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:01.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8efbe00640 1 -- 172.21.15.47:0/2656740912 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc009a70 con 0x7f8f040cf670 2024-02-19T15:30:01.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8efbe00640 1 --2- 172.21.15.47:0/2656740912 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f040cf670 0x7f8f040cb720 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f8eec00c430 tx=0x7f8eec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:01.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8f01600640 1 -- 172.21.15.47:0/2656740912 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec0052d0 con 0x7f8f040cf670 2024-02-19T15:30:01.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8f01600640 1 -- 172.21.15.47:0/2656740912 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eec005430 con 0x7f8f040cf670 2024-02-19T15:30:01.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8f0b400640 1 -- 172.21.15.47:0/2656740912 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f040cc420 con 0x7f8f040cf670 2024-02-19T15:30:01.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8f01600640 1 -- 172.21.15.47:0/2656740912 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec007260 con 0x7f8f040cf670 2024-02-19T15:30:01.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8f0b400640 1 -- 172.21.15.47:0/2656740912 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0415db20 con 0x7f8f040cf670 2024-02-19T15:30:01.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.538+0000 7f8f0b400640 1 -- 172.21.15.47:0/2656740912 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f0413fe10 con 0x7f8f040cf670 2024-02-19T15:30:01.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.542+0000 7f8f01600640 1 -- 172.21.15.47:0/2656740912 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8eec0073c0 con 0x7f8f040cf670 2024-02-19T15:30:01.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.542+0000 7f8f01600640 1 --2- 172.21.15.47:0/2656740912 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ee0066e40 0x7f8ee0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:01.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.542+0000 7f8f01600640 1 -- 172.21.15.47:0/2656740912 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8eec083ef0 con 0x7f8f040cf670 2024-02-19T15:30:01.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.542+0000 7f8f08a00640 1 --2- 172.21.15.47:0/2656740912 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ee0066e40 0x7f8ee0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:01.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.542+0000 7f8f08a00640 1 --2- 172.21.15.47:0/2656740912 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ee0066e40 0x7f8ee0069300 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f8ef40011f0 tx=0x7f8ef4007d40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:01.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.546+0000 7f8f01600640 1 -- 172.21.15.47:0/2656740912 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8f0413fe10 con 0x7f8f040cf670 2024-02-19T15:30:01.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.738+0000 7f8f0b400640 1 -- 172.21.15.47:0/2656740912 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f0413fe10 con 0x7f8f040cf670 2024-02-19T15:30:01.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.742+0000 7f8f01600640 1 -- 172.21.15.47:0/2656740912 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f8f0413fe10 con 0x7f8f040cf670 2024-02-19T15:30:01.740 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:01.740 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:01.740 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:01.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.742+0000 7f8f0b400640 1 -- 172.21.15.47:0/2656740912 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ee0066e40 msgr2=0x7f8ee0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:01.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.742+0000 7f8f0b400640 1 --2- 172.21.15.47:0/2656740912 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ee0066e40 0x7f8ee0069300 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f8ef40011f0 tx=0x7f8ef4007d40 comp rx=0 tx=0).stop 2024-02-19T15:30:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.742+0000 7f8f0b400640 1 -- 172.21.15.47:0/2656740912 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f040cf670 msgr2=0x7f8f040cb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.742+0000 7f8f0b400640 1 --2- 172.21.15.47:0/2656740912 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f040cf670 0x7f8f040cb720 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f8eec00c430 tx=0x7f8eec00c8f0 comp rx=0 tx=0).stop 2024-02-19T15:30:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.742+0000 7f8f0b400640 1 -- 172.21.15.47:0/2656740912 shutdown_connections 2024-02-19T15:30:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.742+0000 7f8f0b400640 1 --2- 172.21.15.47:0/2656740912 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8ee0066e40 0x7f8ee0069300 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.742+0000 7f8f0b400640 1 --2- 172.21.15.47:0/2656740912 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f041456a0 0x7f8f0413b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.742+0000 7f8f0b400640 1 --2- 172.21.15.47:0/2656740912 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f040cf670 0x7f8f040cb720 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.742+0000 7f8f0b400640 1 --2- 172.21.15.47:0/2656740912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0406b5c0 0x7f8f0413c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.742+0000 7f8f0b400640 1 -- 172.21.15.47:0/2656740912 >> 172.21.15.47:0/2656740912 conn(0x7f8f040d0ad0 msgr2=0x7f8f040b1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.742+0000 7f8f0b400640 1 -- 172.21.15.47:0/2656740912 shutdown_connections 2024-02-19T15:30:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:01.742+0000 7f8f0b400640 1 -- 172.21.15.47:0/2656740912 wait complete. 2024-02-19T15:30:01.962 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:01.963 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:02.190 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:02.963 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:03.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.102+0000 7fae74600640 1 Processor -- start 2024-02-19T15:30:03.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.102+0000 7fae74600640 1 -- start start 2024-02-19T15:30:03.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.106+0000 7fae74600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae6c1613a0 0x7fae6c0adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:03.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.106+0000 7fae74600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fae6c06fdf0 0x7fae6c06e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:03.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.106+0000 7fae74600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae6c161780 con 0x7fae6c06e5f0 2024-02-19T15:30:03.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.106+0000 7fae74600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae6c0b01a0 con 0x7fae6c06fdf0 2024-02-19T15:30:03.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.106+0000 7fae74600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae6c0ae3c0 con 0x7fae6c1613a0 2024-02-19T15:30:03.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.106+0000 7fae71200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fae6c06fdf0 0x7fae6c06e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:03.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.106+0000 7fae71200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fae6c06fdf0 0x7fae6c06e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:38360/0 (socket says 172.21.15.47:38360) 2024-02-19T15:30:03.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.106+0000 7fae71200640 1 -- 172.21.15.47:0/1882468411 learned_addr learned my addr 172.21.15.47:0/1882468411 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:03.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.106+0000 7fae71c00640 1 --2- 172.21.15.47:0/1882468411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae6c1613a0 0x7fae6c0adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:03.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.106+0000 7fae71200640 1 -- 172.21.15.47:0/1882468411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae6c1613a0 msgr2=0x7fae6c0adbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:03.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.106+0000 7fae71200640 1 --2- 172.21.15.47:0/1882468411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae6c1613a0 0x7fae6c0adbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:03.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.106+0000 7fae71200640 1 -- 172.21.15.47:0/1882468411 >> v1:172.21.15.47:6789/0 conn(0x7fae6c06e5f0 legacy=0x7fae6c161a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:03.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.106+0000 7fae71200640 1 -- 172.21.15.47:0/1882468411 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae6c0ae6a0 con 0x7fae6c06fdf0 2024-02-19T15:30:03.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.110+0000 7fae71200640 1 --2- 172.21.15.47:0/1882468411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae6c06fdf0 0x7fae6c06e0b0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fae58009870 tx=0x7fae5801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253cc75bdd961c71 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:03.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.110+0000 7fae6be00640 1 -- 172.21.15.47:0/1882468411 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae580ab600 con 0x7fae6c06fdf0 2024-02-19T15:30:03.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.110+0000 7fae6be00640 1 -- 172.21.15.47:0/1882468411 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae580a7050 con 0x7fae6c06fdf0 2024-02-19T15:30:03.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.110+0000 7fae6be00640 1 -- 172.21.15.47:0/1882468411 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae580ab980 con 0x7fae6c06fdf0 2024-02-19T15:30:03.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.110+0000 7fae74600640 1 -- 172.21.15.47:0/1882468411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae6c06fdf0 msgr2=0x7fae6c06e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:03.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.110+0000 7fae74600640 1 --2- 172.21.15.47:0/1882468411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae6c06fdf0 0x7fae6c06e0b0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fae58009870 tx=0x7fae5801c4d0 comp rx=0 tx=0).stop 2024-02-19T15:30:03.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.110+0000 7fae74600640 1 -- 172.21.15.47:0/1882468411 shutdown_connections 2024-02-19T15:30:03.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.110+0000 7fae74600640 1 --2- 172.21.15.47:0/1882468411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae6c06fdf0 0x7fae6c06e0b0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:03.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.110+0000 7fae74600640 1 --2- 172.21.15.47:0/1882468411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae6c1613a0 0x7fae6c0adbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:03.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.110+0000 7fae74600640 1 -- 172.21.15.47:0/1882468411 >> 172.21.15.47:0/1882468411 conn(0x7fae6c0d0ad0 msgr2=0x7fae6c0d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:03.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.110+0000 7fae74600640 1 -- 172.21.15.47:0/1882468411 shutdown_connections 2024-02-19T15:30:03.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.110+0000 7fae74600640 1 -- 172.21.15.47:0/1882468411 wait complete. 2024-02-19T15:30:03.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.110+0000 7fae74600640 1 Processor -- start 2024-02-19T15:30:03.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.110+0000 7fae74600640 1 -- start start 2024-02-19T15:30:03.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae74600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae6c06e5f0 0x7fae6c151750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:03.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae74600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae6c06fdf0 0x7fae6c150210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:03.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae74600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae6c1613a0 0x7fae6c14f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:03.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae74600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6c150a60 con 0x7fae6c06fdf0 2024-02-19T15:30:03.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae74600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6c150d40 con 0x7fae6c1613a0 2024-02-19T15:30:03.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae74600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae6c151020 con 0x7fae6c06e5f0 2024-02-19T15:30:03.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae71200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae6c06fdf0 0x7fae6c150210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:03.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae71200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae6c06fdf0 0x7fae6c150210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:38376/0 (socket says 172.21.15.47:38376) 2024-02-19T15:30:03.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae71200640 1 -- 172.21.15.47:0/1512723104 learned_addr learned my addr 172.21.15.47:0/1512723104 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:03.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae71c00640 1 --2- 172.21.15.47:0/1512723104 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae6c06e5f0 0x7fae6c151750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:03.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae72600640 1 --2- 172.21.15.47:0/1512723104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae6c1613a0 0x7fae6c14f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:03.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae71200640 1 -- 172.21.15.47:0/1512723104 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae6c06e5f0 msgr2=0x7fae6c151750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:03.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae71200640 1 --2- 172.21.15.47:0/1512723104 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae6c06e5f0 0x7fae6c151750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:03.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae71200640 1 -- 172.21.15.47:0/1512723104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae6c1613a0 msgr2=0x7fae6c14f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:03.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae71200640 1 --2- 172.21.15.47:0/1512723104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae6c1613a0 0x7fae6c14f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:03.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae71200640 1 -- 172.21.15.47:0/1512723104 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae580094d0 con 0x7fae6c06fdf0 2024-02-19T15:30:03.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae72600640 1 --2- 172.21.15.47:0/1512723104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae6c1613a0 0x7fae6c14f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:30:03.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae71200640 1 --2- 172.21.15.47:0/1512723104 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae6c06fdf0 0x7fae6c150210 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fae580ae040 tx=0x7fae58019270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:03.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae69600640 1 -- 172.21.15.47:0/1512723104 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae580b4420 con 0x7fae6c06fdf0 2024-02-19T15:30:03.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae69600640 1 -- 172.21.15.47:0/1512723104 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae580b4580 con 0x7fae6c06fdf0 2024-02-19T15:30:03.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae69600640 1 -- 172.21.15.47:0/1512723104 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae580b4930 con 0x7fae6c06fdf0 2024-02-19T15:30:03.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae74600640 1 -- 172.21.15.47:0/1512723104 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae6c151300 con 0x7fae6c06fdf0 2024-02-19T15:30:03.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae74600640 1 -- 172.21.15.47:0/1512723104 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae6c14ff30 con 0x7fae6c06fdf0 2024-02-19T15:30:03.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.114+0000 7fae74600640 1 -- 172.21.15.47:0/1512723104 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae6c075360 con 0x7fae6c06fdf0 2024-02-19T15:30:03.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.118+0000 7fae69600640 1 -- 172.21.15.47:0/1512723104 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fae580b2070 con 0x7fae6c06fdf0 2024-02-19T15:30:03.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.118+0000 7fae69600640 1 --2- 172.21.15.47:0/1512723104 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fae4c066e40 0x7fae4c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:03.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.118+0000 7fae69600640 1 -- 172.21.15.47:0/1512723104 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fae5811dcd0 con 0x7fae6c06fdf0 2024-02-19T15:30:03.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.118+0000 7fae71c00640 1 --2- 172.21.15.47:0/1512723104 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fae4c066e40 0x7fae4c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:03.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.118+0000 7fae71c00640 1 --2- 172.21.15.47:0/1512723104 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fae4c066e40 0x7fae4c069300 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fae6c153690 tx=0x7fae60008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:03.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.122+0000 7fae69600640 1 -- 172.21.15.47:0/1512723104 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fae580eab20 con 0x7fae6c06fdf0 2024-02-19T15:30:03.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.346+0000 7fae74600640 1 -- 172.21.15.47:0/1512723104 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae6c0d15f0 con 0x7fae6c06fdf0 2024-02-19T15:30:03.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.346+0000 7fae69600640 1 -- 172.21.15.47:0/1512723104 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7fae580b0d80 con 0x7fae6c06fdf0 2024-02-19T15:30:03.346 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:03.346 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:03.346 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:03.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.350+0000 7fae74600640 1 -- 172.21.15.47:0/1512723104 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fae4c066e40 msgr2=0x7fae4c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:03.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.350+0000 7fae74600640 1 --2- 172.21.15.47:0/1512723104 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fae4c066e40 0x7fae4c069300 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fae6c153690 tx=0x7fae60008040 comp rx=0 tx=0).stop 2024-02-19T15:30:03.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.350+0000 7fae74600640 1 -- 172.21.15.47:0/1512723104 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae6c06fdf0 msgr2=0x7fae6c150210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:03.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.350+0000 7fae74600640 1 --2- 172.21.15.47:0/1512723104 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae6c06fdf0 0x7fae6c150210 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fae580ae040 tx=0x7fae58019270 comp rx=0 tx=0).stop 2024-02-19T15:30:03.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.350+0000 7fae74600640 1 -- 172.21.15.47:0/1512723104 shutdown_connections 2024-02-19T15:30:03.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.350+0000 7fae74600640 1 --2- 172.21.15.47:0/1512723104 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fae4c066e40 0x7fae4c069300 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:03.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.350+0000 7fae74600640 1 --2- 172.21.15.47:0/1512723104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae6c1613a0 0x7fae6c14f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:03.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.350+0000 7fae74600640 1 --2- 172.21.15.47:0/1512723104 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae6c06fdf0 0x7fae6c150210 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:03.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.350+0000 7fae74600640 1 --2- 172.21.15.47:0/1512723104 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae6c06e5f0 0x7fae6c151750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:03.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.350+0000 7fae74600640 1 -- 172.21.15.47:0/1512723104 >> 172.21.15.47:0/1512723104 conn(0x7fae6c0d0ad0 msgr2=0x7fae6c0b0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:03.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.350+0000 7fae74600640 1 -- 172.21.15.47:0/1512723104 shutdown_connections 2024-02-19T15:30:03.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:03.350+0000 7fae74600640 1 -- 172.21.15.47:0/1512723104 wait complete. 2024-02-19T15:30:03.576 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:03.576 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:04.192 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:04.577 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:04.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.717+0000 7f686b800640 1 Processor -- start 2024-02-19T15:30:04.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.717+0000 7f686b800640 1 -- start start 2024-02-19T15:30:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.717+0000 7f686b800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f68640cea80 0x7f68640727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.717+0000 7f686b800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f686415c390 0x7f686415c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.717+0000 7f686b800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686415d280 con 0x7f6864145590 2024-02-19T15:30:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.717+0000 7f686b800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6864070ff0 con 0x7f68640cea80 2024-02-19T15:30:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.717+0000 7f686b800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f686415cd40 con 0x7f686415c390 2024-02-19T15:30:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.717+0000 7f6863e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f68640cea80 0x7f68640727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.717+0000 7f6863e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f68640cea80 0x7f68640727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56376/0 (socket says 172.21.15.47:56376) 2024-02-19T15:30:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.717+0000 7f6863e00640 1 -- 172.21.15.47:0/3941676001 learned_addr learned my addr 172.21.15.47:0/3941676001 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.717+0000 7f6869800640 1 --2- 172.21.15.47:0/3941676001 >> v2:172.21.15.116:3300/0 conn(0x7f686415c390 0x7f686415c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.717+0000 7f6863e00640 1 -- 172.21.15.47:0/3941676001 >> v2:172.21.15.116:3300/0 conn(0x7f686415c390 msgr2=0x7f686415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.717+0000 7f6863e00640 1 --2- 172.21.15.47:0/3941676001 >> v2:172.21.15.116:3300/0 conn(0x7f686415c390 0x7f686415c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.717+0000 7f6863e00640 1 -- 172.21.15.47:0/3941676001 >> v1:172.21.15.47:6789/0 conn(0x7f6864145590 legacy=0x7f686406c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.717+0000 7f6863e00640 1 -- 172.21.15.47:0/3941676001 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68640709e0 con 0x7f68640cea80 2024-02-19T15:30:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.717+0000 7f6869800640 1 --2- 172.21.15.47:0/3941676001 >> v2:172.21.15.116:3300/0 conn(0x7f686415c390 0x7f686415c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:30:04.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.721+0000 7f6863e00640 1 --2- 172.21.15.47:0/3941676001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68640cea80 0x7f68640727b0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f6864072380 tx=0x7f685001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f91cd005c0e3e55 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:04.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.721+0000 7f6863400640 1 -- 172.21.15.47:0/3941676001 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68500ab610 con 0x7f68640cea80 2024-02-19T15:30:04.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.721+0000 7f6863400640 1 -- 172.21.15.47:0/3941676001 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68500ab770 con 0x7f68640cea80 2024-02-19T15:30:04.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.721+0000 7f6863400640 1 -- 172.21.15.47:0/3941676001 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68500abb00 con 0x7f68640cea80 2024-02-19T15:30:04.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.721+0000 7f686b800640 1 -- 172.21.15.47:0/3941676001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68640cea80 msgr2=0x7f68640727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:04.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.721+0000 7f686b800640 1 --2- 172.21.15.47:0/3941676001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68640cea80 0x7f68640727b0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f6864072380 tx=0x7f685001c520 comp rx=0 tx=0).stop 2024-02-19T15:30:04.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.721+0000 7f686b800640 1 -- 172.21.15.47:0/3941676001 shutdown_connections 2024-02-19T15:30:04.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.721+0000 7f686b800640 1 --2- 172.21.15.47:0/3941676001 >> v2:172.21.15.116:3300/0 conn(0x7f686415c390 0x7f686415c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:04.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.721+0000 7f686b800640 1 --2- 172.21.15.47:0/3941676001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68640cea80 0x7f68640727b0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:04.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.721+0000 7f686b800640 1 -- 172.21.15.47:0/3941676001 >> 172.21.15.47:0/3941676001 conn(0x7f68640b16a0 msgr2=0x7f68640b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:04.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.721+0000 7f686b800640 1 -- 172.21.15.47:0/3941676001 shutdown_connections 2024-02-19T15:30:04.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.721+0000 7f686b800640 1 -- 172.21.15.47:0/3941676001 wait complete. 2024-02-19T15:30:04.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f686b800640 1 Processor -- start 2024-02-19T15:30:04.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f686b800640 1 -- start start 2024-02-19T15:30:04.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f686b800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68640cea80 0x7f686413c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:04.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f686b800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6864145590 0x7f68641439b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:04.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f686b800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f686415c390 0x7f68640ad840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:04.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f686b800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686415be80 con 0x7f68640cea80 2024-02-19T15:30:04.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f686b800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6864142910 con 0x7f6864145590 2024-02-19T15:30:04.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f686b800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6864142bf0 con 0x7f686415c390 2024-02-19T15:30:04.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f6868e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68640cea80 0x7f686413c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f6868e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68640cea80 0x7f686413c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56380/0 (socket says 172.21.15.47:56380) 2024-02-19T15:30:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f6868e00640 1 -- 172.21.15.47:0/3078224359 learned_addr learned my addr 172.21.15.47:0/3078224359 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f6868e00640 1 -- 172.21.15.47:0/3078224359 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f686415c390 msgr2=0x7f68640ad840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f6868e00640 1 --2- 172.21.15.47:0/3078224359 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f686415c390 0x7f68640ad840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f6868e00640 1 -- 172.21.15.47:0/3078224359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6864145590 msgr2=0x7f68641439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f6868e00640 1 --2- 172.21.15.47:0/3078224359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6864145590 0x7f68641439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f6868e00640 1 -- 172.21.15.47:0/3078224359 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68500094d0 con 0x7f68640cea80 2024-02-19T15:30:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f6868e00640 1 --2- 172.21.15.47:0/3078224359 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68640cea80 0x7f686413c300 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f685800c420 tx=0x7f685800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f6860c00640 1 -- 172.21.15.47:0/3078224359 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68580052d0 con 0x7f68640cea80 2024-02-19T15:30:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f6860c00640 1 -- 172.21.15.47:0/3078224359 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6858005430 con 0x7f68640cea80 2024-02-19T15:30:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f6860c00640 1 -- 172.21.15.47:0/3078224359 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6858007240 con 0x7f68640cea80 2024-02-19T15:30:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f686b800640 1 -- 172.21.15.47:0/3078224359 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6864142ed0 con 0x7f68640cea80 2024-02-19T15:30:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f686b800640 1 -- 172.21.15.47:0/3078224359 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6864143490 con 0x7f68640cea80 2024-02-19T15:30:04.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.725+0000 7f686b800640 1 -- 172.21.15.47:0/3078224359 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6864143680 con 0x7f68640cea80 2024-02-19T15:30:04.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.729+0000 7f6860c00640 1 -- 172.21.15.47:0/3078224359 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f68580073a0 con 0x7f68640cea80 2024-02-19T15:30:04.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.729+0000 7f6860c00640 1 --2- 172.21.15.47:0/3078224359 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6844066e40 0x7f6844069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:04.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.729+0000 7f6860c00640 1 -- 172.21.15.47:0/3078224359 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6858083ed0 con 0x7f68640cea80 2024-02-19T15:30:04.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.729+0000 7f6863e00640 1 --2- 172.21.15.47:0/3078224359 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6844066e40 0x7f6844069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:04.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.729+0000 7f6863e00640 1 --2- 172.21.15.47:0/3078224359 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6844066e40 0x7f6844069300 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f68500af000 tx=0x7f6850008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:04.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.733+0000 7f6860c00640 1 -- 172.21.15.47:0/3078224359 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6858050c70 con 0x7f68640cea80 2024-02-19T15:30:04.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.929+0000 7f686b800640 1 -- 172.21.15.47:0/3078224359 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6864075360 con 0x7f68640cea80 2024-02-19T15:30:04.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.929+0000 7f6860c00640 1 -- 172.21.15.47:0/3078224359 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f6858054330 con 0x7f68640cea80 2024-02-19T15:30:04.930 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:04.930 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:04.930 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:04.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.933+0000 7f686b800640 1 -- 172.21.15.47:0/3078224359 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6844066e40 msgr2=0x7f6844069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:04.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.933+0000 7f686b800640 1 --2- 172.21.15.47:0/3078224359 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6844066e40 0x7f6844069300 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f68500af000 tx=0x7f6850008040 comp rx=0 tx=0).stop 2024-02-19T15:30:04.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.933+0000 7f686b800640 1 -- 172.21.15.47:0/3078224359 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68640cea80 msgr2=0x7f686413c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:04.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.933+0000 7f686b800640 1 --2- 172.21.15.47:0/3078224359 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68640cea80 0x7f686413c300 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f685800c420 tx=0x7f685800c8e0 comp rx=0 tx=0).stop 2024-02-19T15:30:04.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.933+0000 7f686b800640 1 -- 172.21.15.47:0/3078224359 shutdown_connections 2024-02-19T15:30:04.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.933+0000 7f686b800640 1 --2- 172.21.15.47:0/3078224359 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6844066e40 0x7f6844069300 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:04.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.933+0000 7f686b800640 1 --2- 172.21.15.47:0/3078224359 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f686415c390 0x7f68640ad840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:04.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.933+0000 7f686b800640 1 --2- 172.21.15.47:0/3078224359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6864145590 0x7f68641439b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:04.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.933+0000 7f686b800640 1 --2- 172.21.15.47:0/3078224359 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68640cea80 0x7f686413c300 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:04.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.933+0000 7f686b800640 1 -- 172.21.15.47:0/3078224359 >> 172.21.15.47:0/3078224359 conn(0x7f68640b16a0 msgr2=0x7f68640b0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:04.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.933+0000 7f686b800640 1 -- 172.21.15.47:0/3078224359 shutdown_connections 2024-02-19T15:30:04.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:04.933+0000 7f686b800640 1 -- 172.21.15.47:0/3078224359 wait complete. 2024-02-19T15:30:05.153 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:05.153 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:06.091 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:30:06.095 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:30:06.154 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:06.193 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:06.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.293+0000 7f52ab800640 1 Processor -- start 2024-02-19T15:30:06.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.293+0000 7f52ab800640 1 -- start start 2024-02-19T15:30:06.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.293+0000 7f52ab800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f52a415e180 0x7f52a40b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:06.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.293+0000 7f52ab800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f52a406c1d0 0x7f52a406c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.293+0000 7f52ab800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52a406cdf0 0x7f52a40cfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.293+0000 7f52ab800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52a406d280 con 0x7f52a415e180 2024-02-19T15:30:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.293+0000 7f52ab800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52a40b2270 con 0x7f52a406c1d0 2024-02-19T15:30:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.293+0000 7f52ab800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a40b2550 con 0x7f52a406cdf0 2024-02-19T15:30:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.293+0000 7f52a8e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f52a415e180 0x7f52a40b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.293+0000 7f52a8e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f52a415e180 0x7f52a40b1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56386/0 (socket says 172.21.15.47:56386) 2024-02-19T15:30:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.293+0000 7f52a8e00640 1 -- 172.21.15.47:0/4110789991 learned_addr learned my addr 172.21.15.47:0/4110789991 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.293+0000 7f529be00640 1 --2- 172.21.15.47:0/4110789991 >> v2:172.21.15.116:3300/0 conn(0x7f52a406c1d0 0x7f52a406c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.293+0000 7f52a9800640 1 --2- 172.21.15.47:0/4110789991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52a406cdf0 0x7f52a40cfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.293+0000 7f52a8e00640 1 -- 172.21.15.47:0/4110789991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52a406cdf0 msgr2=0x7f52a40cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.293+0000 7f52a8e00640 1 --2- 172.21.15.47:0/4110789991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52a406cdf0 0x7f52a40cfda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.293+0000 7f52a8e00640 1 -- 172.21.15.47:0/4110789991 >> v2:172.21.15.116:3300/0 conn(0x7f52a406c1d0 msgr2=0x7f52a406c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.293+0000 7f52a8e00640 1 --2- 172.21.15.47:0/4110789991 >> v2:172.21.15.116:3300/0 conn(0x7f52a406c1d0 0x7f52a406c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:06.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.293+0000 7f52a8e00640 1 -- 172.21.15.47:0/4110789991 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a40b2830 con 0x7f52a415e180 2024-02-19T15:30:06.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.293+0000 7f529be00640 1 --2- 172.21.15.47:0/4110789991 >> v2:172.21.15.116:3300/0 conn(0x7f52a406c1d0 0x7f52a406c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:30:06.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.297+0000 7f52a8e00640 1 --2- 172.21.15.47:0/4110789991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52a415e180 0x7f52a40b1c20 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f52a4071ea0 tx=0x7f529401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a76f95a47fd727fd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:06.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.297+0000 7f52a3e00640 1 -- 172.21.15.47:0/4110789991 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52940aa610 con 0x7f52a415e180 2024-02-19T15:30:06.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.297+0000 7f52a3e00640 1 -- 172.21.15.47:0/4110789991 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52940a6040 con 0x7f52a415e180 2024-02-19T15:30:06.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.297+0000 7f52a3e00640 1 -- 172.21.15.47:0/4110789991 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52940aa990 con 0x7f52a415e180 2024-02-19T15:30:06.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.297+0000 7f52ab800640 1 -- 172.21.15.47:0/4110789991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52a415e180 msgr2=0x7f52a40b1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:06.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.297+0000 7f52ab800640 1 --2- 172.21.15.47:0/4110789991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52a415e180 0x7f52a40b1c20 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f52a4071ea0 tx=0x7f529401c4f0 comp rx=0 tx=0).stop 2024-02-19T15:30:06.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.297+0000 7f52ab800640 1 -- 172.21.15.47:0/4110789991 shutdown_connections 2024-02-19T15:30:06.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.297+0000 7f52ab800640 1 --2- 172.21.15.47:0/4110789991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52a406cdf0 0x7f52a40cfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:06.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.297+0000 7f52ab800640 1 --2- 172.21.15.47:0/4110789991 >> v2:172.21.15.116:3300/0 conn(0x7f52a406c1d0 0x7f52a406c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:06.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.297+0000 7f52ab800640 1 --2- 172.21.15.47:0/4110789991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52a415e180 0x7f52a40b1c20 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:06.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.297+0000 7f52ab800640 1 -- 172.21.15.47:0/4110789991 >> 172.21.15.47:0/4110789991 conn(0x7f52a40d1ba0 msgr2=0x7f52a40d1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:06.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.297+0000 7f52ab800640 1 -- 172.21.15.47:0/4110789991 shutdown_connections 2024-02-19T15:30:06.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.297+0000 7f52ab800640 1 -- 172.21.15.47:0/4110789991 wait complete. 2024-02-19T15:30:06.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f52ab800640 1 Processor -- start 2024-02-19T15:30:06.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f52ab800640 1 -- start start 2024-02-19T15:30:06.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f52ab800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52a406c1d0 0x7f52a414b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:06.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f52ab800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52a406cdf0 0x7f52a41556b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:06.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f52ab800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52a415e180 0x7f52a4155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:06.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f52ab800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a414be60 con 0x7f52a406cdf0 2024-02-19T15:30:06.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f52ab800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a40b5ce0 con 0x7f52a406c1d0 2024-02-19T15:30:06.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f52ab800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a40b7420 con 0x7f52a415e180 2024-02-19T15:30:06.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f529be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52a406cdf0 0x7f52a41556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:06.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f529be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52a406cdf0 0x7f52a41556b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56392/0 (socket says 172.21.15.47:56392) 2024-02-19T15:30:06.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f529be00640 1 -- 172.21.15.47:0/1207723706 learned_addr learned my addr 172.21.15.47:0/1207723706 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:06.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f52a8e00640 1 --2- 172.21.15.47:0/1207723706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52a406c1d0 0x7f52a414b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:06.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f52a9800640 1 --2- 172.21.15.47:0/1207723706 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52a415e180 0x7f52a4155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:06.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f529be00640 1 -- 172.21.15.47:0/1207723706 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52a415e180 msgr2=0x7f52a4155bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:06.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f529be00640 1 --2- 172.21.15.47:0/1207723706 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52a415e180 0x7f52a4155bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:06.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f529be00640 1 -- 172.21.15.47:0/1207723706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52a406c1d0 msgr2=0x7f52a414b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:06.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f529be00640 1 --2- 172.21.15.47:0/1207723706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52a406c1d0 0x7f52a414b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:06.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f529be00640 1 -- 172.21.15.47:0/1207723706 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52940094d0 con 0x7f52a406cdf0 2024-02-19T15:30:06.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f52a9800640 1 --2- 172.21.15.47:0/1207723706 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52a415e180 0x7f52a4155bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:30:06.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f529be00640 1 --2- 172.21.15.47:0/1207723706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52a406cdf0 0x7f52a41556b0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f528c00d800 tx=0x7f528c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:06.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f52a1600640 1 -- 172.21.15.47:0/1207723706 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c004030 con 0x7f52a406cdf0 2024-02-19T15:30:06.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f52a1600640 1 -- 172.21.15.47:0/1207723706 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528c004190 con 0x7f52a406cdf0 2024-02-19T15:30:06.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f52a1600640 1 -- 172.21.15.47:0/1207723706 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c004520 con 0x7f52a406cdf0 2024-02-19T15:30:06.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.301+0000 7f52ab800640 1 -- 172.21.15.47:0/1207723706 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a414cf10 con 0x7f52a406cdf0 2024-02-19T15:30:06.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.305+0000 7f52ab800640 1 -- 172.21.15.47:0/1207723706 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a40b5970 con 0x7f52a406cdf0 2024-02-19T15:30:06.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.305+0000 7f52ab800640 1 -- 172.21.15.47:0/1207723706 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a4068500 con 0x7f52a406cdf0 2024-02-19T15:30:06.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.305+0000 7f52a1600640 1 -- 172.21.15.47:0/1207723706 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f528c0096a0 con 0x7f52a406cdf0 2024-02-19T15:30:06.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.305+0000 7f52a1600640 1 --2- 172.21.15.47:0/1207723706 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5280066e40 0x7f5280069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:06.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.305+0000 7f52a1600640 1 -- 172.21.15.47:0/1207723706 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f528c085790 con 0x7f52a406cdf0 2024-02-19T15:30:06.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.309+0000 7f52a8e00640 1 --2- 172.21.15.47:0/1207723706 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5280066e40 0x7f5280069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:06.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.309+0000 7f52a8e00640 1 --2- 172.21.15.47:0/1207723706 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5280066e40 0x7f5280069300 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f52940ae000 tx=0x7f52940b5040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:06.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.309+0000 7f52a1600640 1 -- 172.21.15.47:0/1207723706 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f528c052520 con 0x7f52a406cdf0 2024-02-19T15:30:06.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.513+0000 7f52ab800640 1 -- 172.21.15.47:0/1207723706 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52a4075360 con 0x7f52a406cdf0 2024-02-19T15:30:06.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.513+0000 7f52a1600640 1 -- 172.21.15.47:0/1207723706 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f528c018d80 con 0x7f52a406cdf0 2024-02-19T15:30:06.514 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:06.514 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:06.514 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:06.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.517+0000 7f52ab800640 1 -- 172.21.15.47:0/1207723706 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5280066e40 msgr2=0x7f5280069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:06.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.517+0000 7f52ab800640 1 --2- 172.21.15.47:0/1207723706 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5280066e40 0x7f5280069300 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f52940ae000 tx=0x7f52940b5040 comp rx=0 tx=0).stop 2024-02-19T15:30:06.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.517+0000 7f52ab800640 1 -- 172.21.15.47:0/1207723706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52a406cdf0 msgr2=0x7f52a41556b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:06.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.517+0000 7f52ab800640 1 --2- 172.21.15.47:0/1207723706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52a406cdf0 0x7f52a41556b0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f528c00d800 tx=0x7f528c00dcc0 comp rx=0 tx=0).stop 2024-02-19T15:30:06.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.517+0000 7f52ab800640 1 -- 172.21.15.47:0/1207723706 shutdown_connections 2024-02-19T15:30:06.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.517+0000 7f52ab800640 1 --2- 172.21.15.47:0/1207723706 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5280066e40 0x7f5280069300 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:06.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.517+0000 7f52ab800640 1 --2- 172.21.15.47:0/1207723706 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52a415e180 0x7f52a4155bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:06.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.517+0000 7f52ab800640 1 --2- 172.21.15.47:0/1207723706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52a406cdf0 0x7f52a41556b0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:06.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.517+0000 7f52ab800640 1 --2- 172.21.15.47:0/1207723706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52a406c1d0 0x7f52a414b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:06.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.517+0000 7f52ab800640 1 -- 172.21.15.47:0/1207723706 >> 172.21.15.47:0/1207723706 conn(0x7f52a40d1ba0 msgr2=0x7f52a4156ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:06.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.517+0000 7f52ab800640 1 -- 172.21.15.47:0/1207723706 shutdown_connections 2024-02-19T15:30:06.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:06.517+0000 7f52ab800640 1 -- 172.21.15.47:0/1207723706 wait complete. 2024-02-19T15:30:06.745 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:06.746 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:07.746 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:07.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.885+0000 7f175bc00640 1 Processor -- start 2024-02-19T15:30:07.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.885+0000 7f175bc00640 1 -- start start 2024-02-19T15:30:07.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.885+0000 7f175bc00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f175415d6e0 0x7f1754162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:07.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.885+0000 7f175bc00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1754071350 0x7f1754164c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:07.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.885+0000 7f175bc00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17540718c0 con 0x7f175415b1a0 2024-02-19T15:30:07.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.885+0000 7f175bc00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1754071ba0 con 0x7f1754071350 2024-02-19T15:30:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.885+0000 7f175bc00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1754071e80 con 0x7f175415d6e0 2024-02-19T15:30:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.885+0000 7f1759c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1754071350 0x7f1754164c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.885+0000 7f1759c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1754071350 0x7f1754164c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56398/0 (socket says 172.21.15.47:56398) 2024-02-19T15:30:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.885+0000 7f1759c00640 1 -- 172.21.15.47:0/2804100108 learned_addr learned my addr 172.21.15.47:0/2804100108 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.885+0000 7f1753e00640 1 --2- 172.21.15.47:0/2804100108 >> v2:172.21.15.116:3300/0 conn(0x7f175415d6e0 0x7f1754162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.889+0000 7f1759c00640 1 -- 172.21.15.47:0/2804100108 >> v2:172.21.15.116:3300/0 conn(0x7f175415d6e0 msgr2=0x7f1754162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.889+0000 7f1759c00640 1 --2- 172.21.15.47:0/2804100108 >> v2:172.21.15.116:3300/0 conn(0x7f175415d6e0 0x7f1754162740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.889+0000 7f1759c00640 1 -- 172.21.15.47:0/2804100108 >> v1:172.21.15.116:6789/0 conn(0x7f175415b1a0 legacy=0x7f175415b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.889+0000 7f1759c00640 1 -- 172.21.15.47:0/2804100108 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1754072160 con 0x7f1754071350 2024-02-19T15:30:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.889+0000 7f1753e00640 1 --2- 172.21.15.47:0/2804100108 >> v2:172.21.15.116:3300/0 conn(0x7f175415d6e0 0x7f1754162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:30:07.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.889+0000 7f1759c00640 1 --2- 172.21.15.47:0/2804100108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1754071350 0x7f1754164c90 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f174c009e50 tx=0x7f174c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e24a1bc755ca3bf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:07.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.889+0000 7f1753400640 1 -- 172.21.15.47:0/2804100108 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c0a58e0 con 0x7f1754071350 2024-02-19T15:30:07.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.889+0000 7f1753400640 1 -- 172.21.15.47:0/2804100108 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174c0a5a40 con 0x7f1754071350 2024-02-19T15:30:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.889+0000 7f1753400640 1 -- 172.21.15.47:0/2804100108 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c0a5dd0 con 0x7f1754071350 2024-02-19T15:30:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.889+0000 7f175bc00640 1 -- 172.21.15.47:0/2804100108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1754071350 msgr2=0x7f1754164c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.889+0000 7f175bc00640 1 --2- 172.21.15.47:0/2804100108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1754071350 0x7f1754164c90 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f174c009e50 tx=0x7f174c01c9f0 comp rx=0 tx=0).stop 2024-02-19T15:30:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.889+0000 7f175bc00640 1 -- 172.21.15.47:0/2804100108 shutdown_connections 2024-02-19T15:30:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.889+0000 7f175bc00640 1 --2- 172.21.15.47:0/2804100108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1754071350 0x7f1754164c90 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.889+0000 7f175bc00640 1 --2- 172.21.15.47:0/2804100108 >> v2:172.21.15.116:3300/0 conn(0x7f175415d6e0 0x7f1754162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.889+0000 7f175bc00640 1 -- 172.21.15.47:0/2804100108 >> 172.21.15.47:0/2804100108 conn(0x7f17540b16a0 msgr2=0x7f17540b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.889+0000 7f175bc00640 1 -- 172.21.15.47:0/2804100108 shutdown_connections 2024-02-19T15:30:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.889+0000 7f175bc00640 1 -- 172.21.15.47:0/2804100108 wait complete. 2024-02-19T15:30:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f175bc00640 1 Processor -- start 2024-02-19T15:30:07.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f175bc00640 1 -- start start 2024-02-19T15:30:07.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f175bc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1754071350 0x7f1754163f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:07.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f175bc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f175415b1a0 0x7f17540aea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f175bc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f175415d6e0 0x7f17541fa280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f175bc00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17541f7190 con 0x7f175415d6e0 2024-02-19T15:30:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f175bc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17541f7470 con 0x7f175415b1a0 2024-02-19T15:30:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f175bc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17541f7750 con 0x7f1754071350 2024-02-19T15:30:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f1759c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f175415d6e0 0x7f17541fa280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f1759c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f175415d6e0 0x7f17541fa280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56406/0 (socket says 172.21.15.47:56406) 2024-02-19T15:30:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f1759c00640 1 -- 172.21.15.47:0/2230882340 learned_addr learned my addr 172.21.15.47:0/2230882340 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f1753e00640 1 --2- 172.21.15.47:0/2230882340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f175415b1a0 0x7f17540aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f1759200640 1 --2- 172.21.15.47:0/2230882340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1754071350 0x7f1754163f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f1759c00640 1 -- 172.21.15.47:0/2230882340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1754071350 msgr2=0x7f1754163f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f1759c00640 1 --2- 172.21.15.47:0/2230882340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1754071350 0x7f1754163f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f1759c00640 1 -- 172.21.15.47:0/2230882340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f175415b1a0 msgr2=0x7f17540aea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f1759c00640 1 --2- 172.21.15.47:0/2230882340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f175415b1a0 0x7f17540aea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f1759c00640 1 -- 172.21.15.47:0/2230882340 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174c009b30 con 0x7f175415d6e0 2024-02-19T15:30:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f1759c00640 1 --2- 172.21.15.47:0/2230882340 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f175415d6e0 0x7f17541fa280 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f174c019600 tx=0x7f174c019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:07.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f1750c00640 1 -- 172.21.15.47:0/2230882340 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c0b8880 con 0x7f175415d6e0 2024-02-19T15:30:07.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f1750c00640 1 -- 172.21.15.47:0/2230882340 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174c0b8a00 con 0x7f175415d6e0 2024-02-19T15:30:07.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f175bc00640 1 -- 172.21.15.47:0/2230882340 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17541f7a30 con 0x7f175415d6e0 2024-02-19T15:30:07.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f1750c00640 1 -- 172.21.15.47:0/2230882340 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c0b8db0 con 0x7f175415d6e0 2024-02-19T15:30:07.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.893+0000 7f175bc00640 1 -- 172.21.15.47:0/2230882340 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17540aefc0 con 0x7f175415d6e0 2024-02-19T15:30:07.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.897+0000 7f175bc00640 1 -- 172.21.15.47:0/2230882340 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1754075360 con 0x7f175415d6e0 2024-02-19T15:30:07.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.897+0000 7f1750c00640 1 -- 172.21.15.47:0/2230882340 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f174c0c3460 con 0x7f175415d6e0 2024-02-19T15:30:07.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.897+0000 7f1750c00640 1 --2- 172.21.15.47:0/2230882340 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1734066e40 0x7f1734069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:07.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.897+0000 7f1750c00640 1 -- 172.21.15.47:0/2230882340 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f174c120130 con 0x7f175415d6e0 2024-02-19T15:30:07.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.897+0000 7f1759200640 1 --2- 172.21.15.47:0/2230882340 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1734066e40 0x7f1734069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:07.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.897+0000 7f1759200640 1 --2- 172.21.15.47:0/2230882340 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1734066e40 0x7f1734069300 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f1748001810 tx=0x7f1748009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:07.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:07.901+0000 7f1750c00640 1 -- 172.21.15.47:0/2230882340 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f174c0ecef0 con 0x7f175415d6e0 2024-02-19T15:30:08.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:08.097+0000 7f175bc00640 1 -- 172.21.15.47:0/2230882340 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17541f7890 con 0x7f175415d6e0 2024-02-19T15:30:08.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:08.097+0000 7f1750c00640 1 -- 172.21.15.47:0/2230882340 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f17541f7890 con 0x7f175415d6e0 2024-02-19T15:30:08.098 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:08.099 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:08.099 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:08.101+0000 7f175bc00640 1 -- 172.21.15.47:0/2230882340 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1734066e40 msgr2=0x7f1734069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:08.101+0000 7f175bc00640 1 --2- 172.21.15.47:0/2230882340 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1734066e40 0x7f1734069300 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f1748001810 tx=0x7f1748009040 comp rx=0 tx=0).stop 2024-02-19T15:30:08.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:08.101+0000 7f175bc00640 1 -- 172.21.15.47:0/2230882340 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f175415d6e0 msgr2=0x7f17541fa280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:08.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:08.101+0000 7f175bc00640 1 --2- 172.21.15.47:0/2230882340 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f175415d6e0 0x7f17541fa280 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f174c019600 tx=0x7f174c019790 comp rx=0 tx=0).stop 2024-02-19T15:30:08.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:08.101+0000 7f175bc00640 1 -- 172.21.15.47:0/2230882340 shutdown_connections 2024-02-19T15:30:08.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:08.101+0000 7f175bc00640 1 --2- 172.21.15.47:0/2230882340 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1734066e40 0x7f1734069300 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:08.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:08.101+0000 7f175bc00640 1 --2- 172.21.15.47:0/2230882340 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f175415d6e0 0x7f17541fa280 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:08.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:08.101+0000 7f175bc00640 1 --2- 172.21.15.47:0/2230882340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f175415b1a0 0x7f17540aea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:08.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:08.101+0000 7f175bc00640 1 --2- 172.21.15.47:0/2230882340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1754071350 0x7f1754163f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:08.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:08.101+0000 7f175bc00640 1 -- 172.21.15.47:0/2230882340 >> 172.21.15.47:0/2230882340 conn(0x7f17540b16a0 msgr2=0x7f175415a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:08.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:08.101+0000 7f175bc00640 1 -- 172.21.15.47:0/2230882340 shutdown_connections 2024-02-19T15:30:08.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:08.101+0000 7f175bc00640 1 -- 172.21.15.47:0/2230882340 wait complete. 2024-02-19T15:30:08.195 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:08.322 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:08.322 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:09.323 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:09.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.461+0000 7fd575800640 1 Processor -- start 2024-02-19T15:30:09.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.461+0000 7fd575800640 1 -- start start 2024-02-19T15:30:09.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd575800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd5700adbb0 0x7fd570168da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:09.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd575800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd570161780 con 0x7fd5701613a0 2024-02-19T15:30:09.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd575800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5700b01a0 con 0x7fd57006fdf0 2024-02-19T15:30:09.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd575800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5700ae3c0 con 0x7fd5700adbb0 2024-02-19T15:30:09.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56ea00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd5701613a0 0x7fd57006e0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58816/0 (socket says 172.21.15.47:58816) 2024-02-19T15:30:09.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56ea00640 1 -- 172.21.15.47:0/1706363567 learned_addr learned my addr 172.21.15.47:0/1706363567 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56f400640 1 --2- 172.21.15.47:0/1706363567 >> v2:172.21.15.116:3300/0 conn(0x7fd5700adbb0 0x7fd570168da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56e000640 1 -- 172.21.15.47:0/1706363567 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806617155 0 0) 0x7fd570161780 con 0x7fd5701613a0 2024-02-19T15:30:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56e000640 1 -- 172.21.15.47:0/1706363567 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd554003530 con 0x7fd5701613a0 2024-02-19T15:30:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56e000640 1 -- 172.21.15.47:0/1706363567 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840005278 0 0) 0x7fd5700b01a0 con 0x7fd57006fdf0 2024-02-19T15:30:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56e000640 1 -- 172.21.15.47:0/1706363567 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd570161780 con 0x7fd57006fdf0 2024-02-19T15:30:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56e000640 1 -- 172.21.15.47:0/1706363567 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2297311814 0 0) 0x7fd554003530 con 0x7fd5701613a0 2024-02-19T15:30:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56e000640 1 -- 172.21.15.47:0/1706363567 >> v2:172.21.15.116:3300/0 conn(0x7fd5700adbb0 msgr2=0x7fd570168da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56e000640 1 --2- 172.21.15.47:0/1706363567 >> v2:172.21.15.116:3300/0 conn(0x7fd5700adbb0 0x7fd570168da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56e000640 1 -- 172.21.15.47:0/1706363567 >> v1:172.21.15.116:6789/0 conn(0x7fd57006fdf0 legacy=0x7fd57006e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56e000640 1 -- 172.21.15.47:0/1706363567 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5700ae6a0 con 0x7fd5701613a0 2024-02-19T15:30:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56e000640 1 -- 172.21.15.47:0/1706363567 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7fd560001680 con 0x7fd5701613a0 2024-02-19T15:30:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56e000640 1 -- 172.21.15.47:0/1706363567 >> v1:172.21.15.47:6789/0 conn(0x7fd5701613a0 legacy=0x7fd57006e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56e000640 1 --2- 172.21.15.47:0/1706363567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd554003b10 0x7fd554003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:09.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56e000640 1 --2- 172.21.15.47:0/1706363567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5540047d0 0x7fd554004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:09.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56e000640 1 --2- 172.21.15.47:0/1706363567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd554005190 0x7fd55400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:09.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56e000640 1 -- 172.21.15.47:0/1706363567 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5700ae3c0 con 0x7fd5540047d0 2024-02-19T15:30:09.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56e000640 1 -- 172.21.15.47:0/1706363567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55400c0c0 con 0x7fd554003b10 2024-02-19T15:30:09.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56e000640 1 -- 172.21.15.47:0/1706363567 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55400c280 con 0x7fd554005190 2024-02-19T15:30:09.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd566000640 1 --2- 172.21.15.47:0/1706363567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5540047d0 0x7fd554004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:09.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56f400640 1 --2- 172.21.15.47:0/1706363567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd554005190 0x7fd55400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:09.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56ea00640 1 --2- 172.21.15.47:0/1706363567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd554003b10 0x7fd554003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:09.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd566000640 1 -- 172.21.15.47:0/1706363567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd554005190 msgr2=0x7fd55400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:09.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd566000640 1 --2- 172.21.15.47:0/1706363567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd554005190 0x7fd55400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:09.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd566000640 1 -- 172.21.15.47:0/1706363567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd554003b10 msgr2=0x7fd554003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:09.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd566000640 1 --2- 172.21.15.47:0/1706363567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd554003b10 0x7fd554003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:09.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd566000640 1 -- 172.21.15.47:0/1706363567 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd554003530 con 0x7fd5540047d0 2024-02-19T15:30:09.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.465+0000 7fd56f400640 1 -- 172.21.15.47:0/1706363567 reap_dead start 2024-02-19T15:30:09.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.469+0000 7fd566000640 1 --2- 172.21.15.47:0/1706363567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5540047d0 0x7fd554004c50 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fd570070280 tx=0x7fd55801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea82d738be0f7a3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:09.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.469+0000 7fd56e000640 1 -- 172.21.15.47:0/1706363567 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5580052d0 con 0x7fd5540047d0 2024-02-19T15:30:09.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.469+0000 7fd56e000640 1 -- 172.21.15.47:0/1706363567 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5580a7050 con 0x7fd5540047d0 2024-02-19T15:30:09.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.469+0000 7fd56e000640 1 -- 172.21.15.47:0/1706363567 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd558003840 con 0x7fd5540047d0 2024-02-19T15:30:09.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.469+0000 7fd575800640 1 -- 172.21.15.47:0/1706363567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5540047d0 msgr2=0x7fd554004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:09.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.469+0000 7fd575800640 1 --2- 172.21.15.47:0/1706363567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5540047d0 0x7fd554004c50 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fd570070280 tx=0x7fd55801c500 comp rx=0 tx=0).stop 2024-02-19T15:30:09.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.469+0000 7fd575800640 1 -- 172.21.15.47:0/1706363567 shutdown_connections 2024-02-19T15:30:09.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.469+0000 7fd575800640 1 --2- 172.21.15.47:0/1706363567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5540047d0 0x7fd554004c50 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:09.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.469+0000 7fd575800640 1 -- 172.21.15.47:0/1706363567 >> 172.21.15.47:0/1706363567 conn(0x7fd5700d0ad0 msgr2=0x7fd5700d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:09.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.469+0000 7fd575800640 1 -- 172.21.15.47:0/1706363567 shutdown_connections 2024-02-19T15:30:09.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.469+0000 7fd575800640 1 -- 172.21.15.47:0/1706363567 wait complete. 2024-02-19T15:30:09.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd575800640 1 Processor -- start 2024-02-19T15:30:09.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd575800640 1 -- start start 2024-02-19T15:30:09.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd575800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5540047d0 0x7fd570150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:09.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd575800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd57015c710 0x7fd5701558b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:09.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd575800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd57015dc40 0x7fd57015ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:09.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd575800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57015cba0 con 0x7fd5540047d0 2024-02-19T15:30:09.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd575800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57015ce80 con 0x7fd57015dc40 2024-02-19T15:30:09.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd575800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd57015d160 con 0x7fd57015c710 2024-02-19T15:30:09.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd56ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5540047d0 0x7fd570150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:09.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd56ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5540047d0 0x7fd570150f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56420/0 (socket says 172.21.15.47:56420) 2024-02-19T15:30:09.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd56ea00640 1 -- 172.21.15.47:0/2724830227 learned_addr learned my addr 172.21.15.47:0/2724830227 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:09.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd566000640 1 --2- 172.21.15.47:0/2724830227 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd57015c710 0x7fd5701558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:09.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd56f400640 1 --2- 172.21.15.47:0/2724830227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd57015dc40 0x7fd57015ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:09.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd56ea00640 1 -- 172.21.15.47:0/2724830227 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd57015c710 msgr2=0x7fd5701558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:09.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd56ea00640 1 --2- 172.21.15.47:0/2724830227 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd57015c710 0x7fd5701558b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:09.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd56ea00640 1 -- 172.21.15.47:0/2724830227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd57015dc40 msgr2=0x7fd57015ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:09.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd56ea00640 1 --2- 172.21.15.47:0/2724830227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd57015dc40 0x7fd57015ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:09.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd56ea00640 1 -- 172.21.15.47:0/2724830227 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57015d440 con 0x7fd5540047d0 2024-02-19T15:30:09.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd56ea00640 1 --2- 172.21.15.47:0/2724830227 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5540047d0 0x7fd570150f70 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fd560004f00 tx=0x7fd560007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:09.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd567400640 1 -- 172.21.15.47:0/2724830227 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5600010c0 con 0x7fd5540047d0 2024-02-19T15:30:09.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd567400640 1 -- 172.21.15.47:0/2724830227 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56000f7b0 con 0x7fd5540047d0 2024-02-19T15:30:09.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd575800640 1 -- 172.21.15.47:0/2724830227 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd57015d720 con 0x7fd5540047d0 2024-02-19T15:30:09.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd567400640 1 -- 172.21.15.47:0/2724830227 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56000fb40 con 0x7fd5540047d0 2024-02-19T15:30:09.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd575800640 1 -- 172.21.15.47:0/2724830227 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd570155df0 con 0x7fd5540047d0 2024-02-19T15:30:09.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.473+0000 7fd575800640 1 -- 172.21.15.47:0/2724830227 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd570155fe0 con 0x7fd5540047d0 2024-02-19T15:30:09.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.477+0000 7fd567400640 1 -- 172.21.15.47:0/2724830227 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd56000fca0 con 0x7fd5540047d0 2024-02-19T15:30:09.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.477+0000 7fd567400640 1 --2- 172.21.15.47:0/2724830227 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd54c066e40 0x7fd54c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:09.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.477+0000 7fd567400640 1 -- 172.21.15.47:0/2724830227 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd560084aa0 con 0x7fd5540047d0 2024-02-19T15:30:09.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.477+0000 7fd566000640 1 --2- 172.21.15.47:0/2724830227 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd54c066e40 0x7fd54c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:09.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.477+0000 7fd566000640 1 --2- 172.21.15.47:0/2724830227 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd54c066e40 0x7fd54c069300 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fd5580ac000 tx=0x7fd5580a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:09.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.481+0000 7fd567400640 1 -- 172.21.15.47:0/2724830227 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd560051860 con 0x7fd5540047d0 2024-02-19T15:30:09.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.809+0000 7fd575800640 1 -- 172.21.15.47:0/2724830227 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd570075360 con 0x7fd5540047d0 2024-02-19T15:30:09.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.809+0000 7fd567400640 1 -- 172.21.15.47:0/2724830227 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7fd560054f20 con 0x7fd5540047d0 2024-02-19T15:30:09.809 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:09.809 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:09.809 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:09.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.809+0000 7fd575800640 1 -- 172.21.15.47:0/2724830227 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd54c066e40 msgr2=0x7fd54c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:09.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.809+0000 7fd575800640 1 --2- 172.21.15.47:0/2724830227 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd54c066e40 0x7fd54c069300 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fd5580ac000 tx=0x7fd5580a6000 comp rx=0 tx=0).stop 2024-02-19T15:30:09.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.809+0000 7fd575800640 1 -- 172.21.15.47:0/2724830227 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5540047d0 msgr2=0x7fd570150f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:09.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.809+0000 7fd575800640 1 --2- 172.21.15.47:0/2724830227 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5540047d0 0x7fd570150f70 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fd560004f00 tx=0x7fd560007430 comp rx=0 tx=0).stop 2024-02-19T15:30:09.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.809+0000 7fd575800640 1 -- 172.21.15.47:0/2724830227 shutdown_connections 2024-02-19T15:30:09.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.809+0000 7fd575800640 1 --2- 172.21.15.47:0/2724830227 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd54c066e40 0x7fd54c069300 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:09.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.809+0000 7fd575800640 1 --2- 172.21.15.47:0/2724830227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd57015dc40 0x7fd57015ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:09.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.809+0000 7fd575800640 1 --2- 172.21.15.47:0/2724830227 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd57015c710 0x7fd5701558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:09.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.809+0000 7fd575800640 1 --2- 172.21.15.47:0/2724830227 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5540047d0 0x7fd570150f70 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:09.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.809+0000 7fd575800640 1 -- 172.21.15.47:0/2724830227 >> 172.21.15.47:0/2724830227 conn(0x7fd5700d0ad0 msgr2=0x7fd570156320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:09.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.809+0000 7fd575800640 1 -- 172.21.15.47:0/2724830227 shutdown_connections 2024-02-19T15:30:09.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:09.809+0000 7fd575800640 1 -- 172.21.15.47:0/2724830227 wait complete. 2024-02-19T15:30:10.031 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:10.031 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:10.197 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:11.033 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:11.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.173+0000 7f6e39000640 1 Processor -- start 2024-02-19T15:30:11.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.173+0000 7f6e39000640 1 -- start start 2024-02-19T15:30:11.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.173+0000 7f6e39000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e34145590 0x7f6e340727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:11.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.173+0000 7f6e39000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6e340cea80 0x7f6e3406c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:11.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.173+0000 7f6e39000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6e3415c390 0x7f6e3415c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:11.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.173+0000 7f6e39000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e3415d280 con 0x7f6e340cea80 2024-02-19T15:30:11.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.173+0000 7f6e39000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e34070ff0 con 0x7f6e3415c390 2024-02-19T15:30:11.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.173+0000 7f6e39000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3415cd60 con 0x7f6e34145590 2024-02-19T15:30:11.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.173+0000 7f6e29400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6e340cea80 0x7f6e3406c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:11.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.173+0000 7f6e29400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6e340cea80 0x7f6e3406c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56434/0 (socket says 172.21.15.47:56434) 2024-02-19T15:30:11.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.173+0000 7f6e29400640 1 -- 172.21.15.47:0/3220806260 learned_addr learned my addr 172.21.15.47:0/3220806260 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:11.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.173+0000 7f6e32800640 1 --2- 172.21.15.47:0/3220806260 >> v2:172.21.15.116:3300/0 conn(0x7f6e3415c390 0x7f6e3415c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:11.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.173+0000 7f6e31e00640 1 --2- 172.21.15.47:0/3220806260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e34145590 0x7f6e340727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:11.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.177+0000 7f6e29400640 1 -- 172.21.15.47:0/3220806260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e34145590 msgr2=0x7f6e340727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:11.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.177+0000 7f6e29400640 1 --2- 172.21.15.47:0/3220806260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e34145590 0x7f6e340727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:11.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.177+0000 7f6e29400640 1 -- 172.21.15.47:0/3220806260 >> v2:172.21.15.116:3300/0 conn(0x7f6e3415c390 msgr2=0x7f6e3415c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:11.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.177+0000 7f6e29400640 1 --2- 172.21.15.47:0/3220806260 >> v2:172.21.15.116:3300/0 conn(0x7f6e3415c390 0x7f6e3415c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:11.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.177+0000 7f6e29400640 1 -- 172.21.15.47:0/3220806260 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e340709e0 con 0x7f6e340cea80 2024-02-19T15:30:11.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.181+0000 7f6e29400640 1 --2- 172.21.15.47:0/3220806260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e340cea80 0x7f6e3406c9a0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c009870 tx=0x7f6e1c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ee328f88ba35f71 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:11.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.181+0000 7f6e31400640 1 -- 172.21.15.47:0/3220806260 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c0ac630 con 0x7f6e340cea80 2024-02-19T15:30:11.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.181+0000 7f6e31400640 1 -- 172.21.15.47:0/3220806260 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e1c0ac790 con 0x7f6e340cea80 2024-02-19T15:30:11.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.181+0000 7f6e31400640 1 -- 172.21.15.47:0/3220806260 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c0acb20 con 0x7f6e340cea80 2024-02-19T15:30:11.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.181+0000 7f6e39000640 1 -- 172.21.15.47:0/3220806260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e340cea80 msgr2=0x7f6e3406c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:11.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.181+0000 7f6e39000640 1 --2- 172.21.15.47:0/3220806260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e340cea80 0x7f6e3406c9a0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c009870 tx=0x7f6e1c01c520 comp rx=0 tx=0).stop 2024-02-19T15:30:11.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.181+0000 7f6e39000640 1 -- 172.21.15.47:0/3220806260 shutdown_connections 2024-02-19T15:30:11.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.181+0000 7f6e39000640 1 --2- 172.21.15.47:0/3220806260 >> v2:172.21.15.116:3300/0 conn(0x7f6e3415c390 0x7f6e3415c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:11.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.181+0000 7f6e39000640 1 --2- 172.21.15.47:0/3220806260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e340cea80 0x7f6e3406c9a0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:11.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.181+0000 7f6e39000640 1 --2- 172.21.15.47:0/3220806260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e34145590 0x7f6e340727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:11.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.181+0000 7f6e39000640 1 -- 172.21.15.47:0/3220806260 >> 172.21.15.47:0/3220806260 conn(0x7f6e340b16a0 msgr2=0x7f6e340b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:11.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.181+0000 7f6e39000640 1 -- 172.21.15.47:0/3220806260 shutdown_connections 2024-02-19T15:30:11.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.181+0000 7f6e39000640 1 -- 172.21.15.47:0/3220806260 wait complete. 2024-02-19T15:30:11.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.181+0000 7f6e39000640 1 Processor -- start 2024-02-19T15:30:11.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.181+0000 7f6e39000640 1 -- start start 2024-02-19T15:30:11.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e39000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e340cea80 0x7f6e341364c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:11.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e39000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e34145590 0x7f6e341381e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:11.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e39000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e3415c390 0x7f6e340b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:11.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e39000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e34136a00 con 0x7f6e340cea80 2024-02-19T15:30:11.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e39000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e34137140 con 0x7f6e3415c390 2024-02-19T15:30:11.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e39000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e34137420 con 0x7f6e34145590 2024-02-19T15:30:11.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e31e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e340cea80 0x7f6e341364c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:11.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e31e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e340cea80 0x7f6e341364c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56442/0 (socket says 172.21.15.47:56442) 2024-02-19T15:30:11.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e31e00640 1 -- 172.21.15.47:0/1177387967 learned_addr learned my addr 172.21.15.47:0/1177387967 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:11.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e29400640 1 --2- 172.21.15.47:0/1177387967 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e34145590 0x7f6e341381e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:11.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e32800640 1 --2- 172.21.15.47:0/1177387967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e3415c390 0x7f6e340b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:11.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e31e00640 1 -- 172.21.15.47:0/1177387967 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e34145590 msgr2=0x7f6e341381e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:11.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e31e00640 1 --2- 172.21.15.47:0/1177387967 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e34145590 0x7f6e341381e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:11.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e31e00640 1 -- 172.21.15.47:0/1177387967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e3415c390 msgr2=0x7f6e340b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:11.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e31e00640 1 --2- 172.21.15.47:0/1177387967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e3415c390 0x7f6e340b35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:11.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e31e00640 1 -- 172.21.15.47:0/1177387967 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c0094d0 con 0x7f6e340cea80 2024-02-19T15:30:11.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e29400640 1 --2- 172.21.15.47:0/1177387967 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e34145590 0x7f6e341381e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:30:11.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e32800640 1 --2- 172.21.15.47:0/1177387967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e3415c390 0x7f6e340b35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:30:11.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e31e00640 1 --2- 172.21.15.47:0/1177387967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e340cea80 0x7f6e341364c0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f6e2400e8b0 tx=0x7f6e2400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:11.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e2aa00640 1 -- 172.21.15.47:0/1177387967 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2400c940 con 0x7f6e340cea80 2024-02-19T15:30:11.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e2aa00640 1 -- 172.21.15.47:0/1177387967 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e2400caa0 con 0x7f6e340cea80 2024-02-19T15:30:11.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e39000640 1 -- 172.21.15.47:0/1177387967 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e34137700 con 0x7f6e340cea80 2024-02-19T15:30:11.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e2aa00640 1 -- 172.21.15.47:0/1177387967 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2400ce30 con 0x7f6e340cea80 2024-02-19T15:30:11.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e39000640 1 -- 172.21.15.47:0/1177387967 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e34137cc0 con 0x7f6e340cea80 2024-02-19T15:30:11.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.185+0000 7f6e39000640 1 -- 172.21.15.47:0/1177387967 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e340ae080 con 0x7f6e340cea80 2024-02-19T15:30:11.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.189+0000 7f6e2aa00640 1 -- 172.21.15.47:0/1177387967 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6e24013470 con 0x7f6e340cea80 2024-02-19T15:30:11.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.189+0000 7f6e2aa00640 1 --2- 172.21.15.47:0/1177387967 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6e10066e40 0x7f6e10069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:11.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.189+0000 7f6e2aa00640 1 -- 172.21.15.47:0/1177387967 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6e24084d80 con 0x7f6e340cea80 2024-02-19T15:30:11.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.189+0000 7f6e29400640 1 --2- 172.21.15.47:0/1177387967 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6e10066e40 0x7f6e10069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:11.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.189+0000 7f6e29400640 1 --2- 172.21.15.47:0/1177387967 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6e10066e40 0x7f6e10069300 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c0ae000 tx=0x7f6e1c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.193+0000 7f6e2aa00640 1 -- 172.21.15.47:0/1177387967 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6e24051a10 con 0x7f6e340cea80 2024-02-19T15:30:11.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.493+0000 7f6e39000640 1 -- 172.21.15.47:0/1177387967 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e34075360 con 0x7f6e340cea80 2024-02-19T15:30:11.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.493+0000 7f6e2aa00640 1 -- 172.21.15.47:0/1177387967 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f6e24017d80 con 0x7f6e340cea80 2024-02-19T15:30:11.495 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:11.495 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:11.495 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:11.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.497+0000 7f6e39000640 1 -- 172.21.15.47:0/1177387967 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6e10066e40 msgr2=0x7f6e10069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:11.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.497+0000 7f6e39000640 1 --2- 172.21.15.47:0/1177387967 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6e10066e40 0x7f6e10069300 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c0ae000 tx=0x7f6e1c008040 comp rx=0 tx=0).stop 2024-02-19T15:30:11.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.497+0000 7f6e39000640 1 -- 172.21.15.47:0/1177387967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e340cea80 msgr2=0x7f6e341364c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:11.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.497+0000 7f6e39000640 1 --2- 172.21.15.47:0/1177387967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e340cea80 0x7f6e341364c0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f6e2400e8b0 tx=0x7f6e2400ed70 comp rx=0 tx=0).stop 2024-02-19T15:30:11.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.497+0000 7f6e39000640 1 -- 172.21.15.47:0/1177387967 shutdown_connections 2024-02-19T15:30:11.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.497+0000 7f6e39000640 1 --2- 172.21.15.47:0/1177387967 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6e10066e40 0x7f6e10069300 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:11.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.497+0000 7f6e39000640 1 --2- 172.21.15.47:0/1177387967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e3415c390 0x7f6e340b35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:11.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.497+0000 7f6e39000640 1 --2- 172.21.15.47:0/1177387967 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6e34145590 0x7f6e341381e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:11.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.497+0000 7f6e39000640 1 --2- 172.21.15.47:0/1177387967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e340cea80 0x7f6e341364c0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:11.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.497+0000 7f6e39000640 1 -- 172.21.15.47:0/1177387967 >> 172.21.15.47:0/1177387967 conn(0x7f6e340b16a0 msgr2=0x7f6e34072610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:11.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.497+0000 7f6e39000640 1 -- 172.21.15.47:0/1177387967 shutdown_connections 2024-02-19T15:30:11.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:11.497+0000 7f6e39000640 1 -- 172.21.15.47:0/1177387967 wait complete. 2024-02-19T15:30:11.728 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:11.728 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:12.199 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:12.729 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:12.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.869+0000 7f25b1400640 1 Processor -- start 2024-02-19T15:30:12.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.869+0000 7f25b1400640 1 -- start start 2024-02-19T15:30:12.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.869+0000 7f25b1400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25ac0cd9b0 0x7f25ac0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:12.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.869+0000 7f25b1400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac070c50 con 0x7f25ac06cf30 2024-02-19T15:30:12.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.869+0000 7f25b1400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac06d4f0 con 0x7f25ac152f00 2024-02-19T15:30:12.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.869+0000 7f25b1400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac06d7d0 con 0x7f25ac0cd9b0 2024-02-19T15:30:12.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.869+0000 7f25a1e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f25ac06cf30 0x7f25ac071960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58822/0 (socket says 172.21.15.47:58822) 2024-02-19T15:30:12.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.869+0000 7f25a1e00640 1 -- 172.21.15.47:0/933264835 learned_addr learned my addr 172.21.15.47:0/933264835 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:12.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.869+0000 7f25ab200640 1 --2- 172.21.15.47:0/933264835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25ac0cd9b0 0x7f25ac0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:12.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.869+0000 7f25a9e00640 1 -- 172.21.15.47:0/933264835 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650194345 0 0) 0x7f25ac070c50 con 0x7f25ac06cf30 2024-02-19T15:30:12.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.869+0000 7f25a9e00640 1 -- 172.21.15.47:0/933264835 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2590003530 con 0x7f25ac06cf30 2024-02-19T15:30:12.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.869+0000 7f25a9e00640 1 -- 172.21.15.47:0/933264835 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932097975 0 0) 0x7f25ac06d4f0 con 0x7f25ac152f00 2024-02-19T15:30:12.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.869+0000 7f25a9e00640 1 -- 172.21.15.47:0/933264835 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25ac070c50 con 0x7f25ac152f00 2024-02-19T15:30:12.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.869+0000 7f25ab200640 1 -- 172.21.15.47:0/933264835 >> v1:172.21.15.116:6789/0 conn(0x7f25ac152f00 legacy=0x7f25ac15cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:12.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.869+0000 7f25ab200640 1 -- 172.21.15.47:0/933264835 >> v1:172.21.15.47:6789/0 conn(0x7f25ac06cf30 legacy=0x7f25ac071960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:12.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.869+0000 7f25ab200640 1 -- 172.21.15.47:0/933264835 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac070e80 con 0x7f25ac0cd9b0 2024-02-19T15:30:12.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.873+0000 7f25ab200640 1 --2- 172.21.15.47:0/933264835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25ac0cd9b0 0x7f25ac0cde20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f25a4009820 tx=0x7f25a401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba8199363adff9c1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:12.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.873+0000 7f25a9e00640 1 -- 172.21.15.47:0/933264835 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a40ab600 con 0x7f25ac0cd9b0 2024-02-19T15:30:12.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.873+0000 7f25a9e00640 1 -- 172.21.15.47:0/933264835 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a40ab760 con 0x7f25ac0cd9b0 2024-02-19T15:30:12.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.873+0000 7f25a9e00640 1 -- 172.21.15.47:0/933264835 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a40abaf0 con 0x7f25ac0cd9b0 2024-02-19T15:30:12.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.873+0000 7f25b1400640 1 -- 172.21.15.47:0/933264835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25ac0cd9b0 msgr2=0x7f25ac0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:12.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.873+0000 7f25b1400640 1 --2- 172.21.15.47:0/933264835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25ac0cd9b0 0x7f25ac0cde20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f25a4009820 tx=0x7f25a401c400 comp rx=0 tx=0).stop 2024-02-19T15:30:12.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.873+0000 7f25b1400640 1 -- 172.21.15.47:0/933264835 shutdown_connections 2024-02-19T15:30:12.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.873+0000 7f25b1400640 1 --2- 172.21.15.47:0/933264835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25ac0cd9b0 0x7f25ac0cde20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:12.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.873+0000 7f25b1400640 1 -- 172.21.15.47:0/933264835 >> 172.21.15.47:0/933264835 conn(0x7f25ac0cfde0 msgr2=0x7f25ac0d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:12.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.873+0000 7f25b1400640 1 -- 172.21.15.47:0/933264835 shutdown_connections 2024-02-19T15:30:12.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.873+0000 7f25b1400640 1 -- 172.21.15.47:0/933264835 wait complete. 2024-02-19T15:30:12.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.877+0000 7f25b1400640 1 Processor -- start 2024-02-19T15:30:12.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.877+0000 7f25b1400640 1 -- start start 2024-02-19T15:30:12.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.877+0000 7f25b1400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ac06cf30 0x7f25ac14ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:12.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.877+0000 7f25b1400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25ac0cd9b0 0x7f25ac155d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:12.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.877+0000 7f25b1400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25ac152f00 0x7f25ac147ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:12.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.877+0000 7f25b1400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac14f230 con 0x7f25ac06cf30 2024-02-19T15:30:12.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.877+0000 7f25b1400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac1510b0 con 0x7f25ac152f00 2024-02-19T15:30:12.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.877+0000 7f25b1400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac151350 con 0x7f25ac0cd9b0 2024-02-19T15:30:12.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.877+0000 7f25aa800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ac06cf30 0x7f25ac14ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:12.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.877+0000 7f25aa800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ac06cf30 0x7f25ac14ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56454/0 (socket says 172.21.15.47:56454) 2024-02-19T15:30:12.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.877+0000 7f25aa800640 1 -- 172.21.15.47:0/2263946560 learned_addr learned my addr 172.21.15.47:0/2263946560 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:12.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.877+0000 7f25ab200640 1 --2- 172.21.15.47:0/2263946560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25ac152f00 0x7f25ac147ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:12.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.877+0000 7f25a1e00640 1 --2- 172.21.15.47:0/2263946560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25ac0cd9b0 0x7f25ac155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:12.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.877+0000 7f25aa800640 1 -- 172.21.15.47:0/2263946560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25ac0cd9b0 msgr2=0x7f25ac155d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:12.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.877+0000 7f25aa800640 1 --2- 172.21.15.47:0/2263946560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25ac0cd9b0 0x7f25ac155d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:12.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.877+0000 7f25aa800640 1 -- 172.21.15.47:0/2263946560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25ac152f00 msgr2=0x7f25ac147ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:12.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.877+0000 7f25aa800640 1 --2- 172.21.15.47:0/2263946560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25ac152f00 0x7f25ac147ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:12.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.877+0000 7f25aa800640 1 -- 172.21.15.47:0/2263946560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a4009420 con 0x7f25ac06cf30 2024-02-19T15:30:12.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.877+0000 7f25aa800640 1 --2- 172.21.15.47:0/2263946560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ac06cf30 0x7f25ac14ebc0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f259c00c430 tx=0x7f259c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:12.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.881+0000 7f25a3400640 1 -- 172.21.15.47:0/2263946560 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c0052d0 con 0x7f25ac06cf30 2024-02-19T15:30:12.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.881+0000 7f25a3400640 1 -- 172.21.15.47:0/2263946560 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259c005450 con 0x7f25ac06cf30 2024-02-19T15:30:12.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.881+0000 7f25b1400640 1 -- 172.21.15.47:0/2263946560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ac151630 con 0x7f25ac06cf30 2024-02-19T15:30:12.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.881+0000 7f25a3400640 1 -- 172.21.15.47:0/2263946560 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c003840 con 0x7f25ac06cf30 2024-02-19T15:30:12.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.881+0000 7f25b1400640 1 -- 172.21.15.47:0/2263946560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ac148400 con 0x7f25ac06cf30 2024-02-19T15:30:12.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.881+0000 7f25b1400640 1 -- 172.21.15.47:0/2263946560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ac075360 con 0x7f25ac06cf30 2024-02-19T15:30:12.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.881+0000 7f25a3400640 1 -- 172.21.15.47:0/2263946560 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f259c0039a0 con 0x7f25ac06cf30 2024-02-19T15:30:12.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.881+0000 7f25a3400640 1 --2- 172.21.15.47:0/2263946560 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2588066e40 0x7f2588069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:12.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.881+0000 7f25a3400640 1 -- 172.21.15.47:0/2263946560 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f259c083fe0 con 0x7f25ac06cf30 2024-02-19T15:30:12.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.881+0000 7f25a1e00640 1 --2- 172.21.15.47:0/2263946560 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2588066e40 0x7f2588069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:12.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.885+0000 7f25a1e00640 1 --2- 172.21.15.47:0/2263946560 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2588066e40 0x7f2588069300 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f25ac0cd3e0 tx=0x7f2594001aa0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:12.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:12.885+0000 7f25a3400640 1 -- 172.21.15.47:0/2263946560 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f259c050e30 con 0x7f25ac06cf30 2024-02-19T15:30:13.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:13.085+0000 7f25b1400640 1 -- 172.21.15.47:0/2263946560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25ac14c8d0 con 0x7f25ac06cf30 2024-02-19T15:30:13.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:13.085+0000 7f25a3400640 1 -- 172.21.15.47:0/2263946560 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f259c017d80 con 0x7f25ac06cf30 2024-02-19T15:30:13.084 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:13.085 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:13.085 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:13.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:13.085+0000 7f25b1400640 1 -- 172.21.15.47:0/2263946560 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2588066e40 msgr2=0x7f2588069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:13.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:13.085+0000 7f25b1400640 1 --2- 172.21.15.47:0/2263946560 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2588066e40 0x7f2588069300 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f25ac0cd3e0 tx=0x7f2594001aa0 comp rx=0 tx=0).stop 2024-02-19T15:30:13.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:13.085+0000 7f25b1400640 1 -- 172.21.15.47:0/2263946560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ac06cf30 msgr2=0x7f25ac14ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:13.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:13.085+0000 7f25b1400640 1 --2- 172.21.15.47:0/2263946560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ac06cf30 0x7f25ac14ebc0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f259c00c430 tx=0x7f259c00c8f0 comp rx=0 tx=0).stop 2024-02-19T15:30:13.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:13.085+0000 7f25b1400640 1 -- 172.21.15.47:0/2263946560 shutdown_connections 2024-02-19T15:30:13.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:13.085+0000 7f25b1400640 1 --2- 172.21.15.47:0/2263946560 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2588066e40 0x7f2588069300 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:13.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:13.085+0000 7f25b1400640 1 --2- 172.21.15.47:0/2263946560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25ac152f00 0x7f25ac147ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:13.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:13.085+0000 7f25b1400640 1 --2- 172.21.15.47:0/2263946560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25ac0cd9b0 0x7f25ac155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:13.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:13.085+0000 7f25b1400640 1 --2- 172.21.15.47:0/2263946560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25ac06cf30 0x7f25ac14ebc0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:13.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:13.085+0000 7f25b1400640 1 -- 172.21.15.47:0/2263946560 >> 172.21.15.47:0/2263946560 conn(0x7f25ac0cfde0 msgr2=0x7f25ac14d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:13.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:13.085+0000 7f25b1400640 1 -- 172.21.15.47:0/2263946560 shutdown_connections 2024-02-19T15:30:13.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:13.085+0000 7f25b1400640 1 -- 172.21.15.47:0/2263946560 wait complete. 2024-02-19T15:30:13.307 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:13.308 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:14.201 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:14.309 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:14.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.453+0000 7faf58a00640 1 Processor -- start 2024-02-19T15:30:14.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.453+0000 7faf58a00640 1 -- start start 2024-02-19T15:30:14.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.453+0000 7faf58a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7faf5415dd70 0x7faf540ae860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:14.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.453+0000 7faf58a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf540aeda0 0x7faf5406d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:14.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.453+0000 7faf58a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf5406dab0 con 0x7faf5406d660 2024-02-19T15:30:14.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.453+0000 7faf58a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf5415f3f0 con 0x7faf5415dd70 2024-02-19T15:30:14.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.453+0000 7faf58a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf5406c3e0 con 0x7faf540aeda0 2024-02-19T15:30:14.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.453+0000 7faf4be00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf540aeda0 0x7faf5406d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:14.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.453+0000 7faf53400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7faf5415dd70 0x7faf540ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:14.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.453+0000 7faf4be00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf540aeda0 0x7faf5406d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:60762/0 (socket says 172.21.15.47:60762) 2024-02-19T15:30:14.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.453+0000 7faf4be00640 1 -- 172.21.15.47:0/1667477353 learned_addr learned my addr 172.21.15.47:0/1667477353 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:14.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.453+0000 7faf53400640 1 -- 172.21.15.47:0/1667477353 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf540aeda0 msgr2=0x7faf5406d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:14.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.453+0000 7faf53400640 1 --2- 172.21.15.47:0/1667477353 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf540aeda0 0x7faf5406d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:14.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.453+0000 7faf53400640 1 -- 172.21.15.47:0/1667477353 >> v1:172.21.15.116:6789/0 conn(0x7faf5406d660 legacy=0x7faf540d19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:14.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.453+0000 7faf53400640 1 -- 172.21.15.47:0/1667477353 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf5406c680 con 0x7faf5415dd70 2024-02-19T15:30:14.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.457+0000 7faf53400640 1 --2- 172.21.15.47:0/1667477353 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf5415dd70 0x7faf540ae860 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7faf440097c0 tx=0x7faf4401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39b8d96fc6b3e8e2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:14.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.457+0000 7faf52a00640 1 -- 172.21.15.47:0/1667477353 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf440ac640 con 0x7faf5415dd70 2024-02-19T15:30:14.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.457+0000 7faf52a00640 1 -- 172.21.15.47:0/1667477353 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf440ac7a0 con 0x7faf5415dd70 2024-02-19T15:30:14.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.457+0000 7faf52a00640 1 -- 172.21.15.47:0/1667477353 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf440acb30 con 0x7faf5415dd70 2024-02-19T15:30:14.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.457+0000 7faf58a00640 1 -- 172.21.15.47:0/1667477353 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf5415dd70 msgr2=0x7faf540ae860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:14.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.457+0000 7faf58a00640 1 --2- 172.21.15.47:0/1667477353 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf5415dd70 0x7faf540ae860 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7faf440097c0 tx=0x7faf4401c3e0 comp rx=0 tx=0).stop 2024-02-19T15:30:14.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.457+0000 7faf58a00640 1 -- 172.21.15.47:0/1667477353 shutdown_connections 2024-02-19T15:30:14.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.457+0000 7faf58a00640 1 --2- 172.21.15.47:0/1667477353 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf540aeda0 0x7faf5406d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:14.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.457+0000 7faf58a00640 1 --2- 172.21.15.47:0/1667477353 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf5415dd70 0x7faf540ae860 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:14.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.457+0000 7faf58a00640 1 -- 172.21.15.47:0/1667477353 >> 172.21.15.47:0/1667477353 conn(0x7faf5409a9d0 msgr2=0x7faf5409ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:14.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.457+0000 7faf58a00640 1 -- 172.21.15.47:0/1667477353 shutdown_connections 2024-02-19T15:30:14.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.457+0000 7faf58a00640 1 -- 172.21.15.47:0/1667477353 wait complete. 2024-02-19T15:30:14.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.461+0000 7faf58a00640 1 Processor -- start 2024-02-19T15:30:14.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.461+0000 7faf58a00640 1 -- start start 2024-02-19T15:30:14.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.461+0000 7faf58a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf5406d660 0x7faf5409b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:14.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.461+0000 7faf58a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf540aeda0 0x7faf54153e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:14.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.461+0000 7faf58a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf5415dd70 0x7faf54159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:14.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.461+0000 7faf58a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf54155550 con 0x7faf5415dd70 2024-02-19T15:30:14.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.461+0000 7faf58a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf541546a0 con 0x7faf540aeda0 2024-02-19T15:30:14.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.461+0000 7faf58a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf541559e0 con 0x7faf5406d660 2024-02-19T15:30:14.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.461+0000 7faf53e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf5415dd70 0x7faf54159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:14.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.461+0000 7faf4be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf540aeda0 0x7faf54153e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:14.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.461+0000 7faf53e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf5415dd70 0x7faf54159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60606/0 (socket says 172.21.15.47:60606) 2024-02-19T15:30:14.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.461+0000 7faf53e00640 1 -- 172.21.15.47:0/595387488 learned_addr learned my addr 172.21.15.47:0/595387488 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:14.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.461+0000 7faf53400640 1 --2- 172.21.15.47:0/595387488 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf5406d660 0x7faf5409b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:14.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.461+0000 7faf53e00640 1 -- 172.21.15.47:0/595387488 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf5406d660 msgr2=0x7faf5409b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:14.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.461+0000 7faf53e00640 1 --2- 172.21.15.47:0/595387488 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf5406d660 0x7faf5409b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:14.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.461+0000 7faf53e00640 1 -- 172.21.15.47:0/595387488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf540aeda0 msgr2=0x7faf54153e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:14.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.461+0000 7faf53e00640 1 --2- 172.21.15.47:0/595387488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf540aeda0 0x7faf54153e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:14.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.461+0000 7faf53e00640 1 -- 172.21.15.47:0/595387488 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf44009420 con 0x7faf5415dd70 2024-02-19T15:30:14.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.461+0000 7faf53e00640 1 --2- 172.21.15.47:0/595387488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf5415dd70 0x7faf54159b50 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7faf4c00c770 tx=0x7faf4c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:14.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.461+0000 7faf4b400640 1 -- 172.21.15.47:0/595387488 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c003b80 con 0x7faf5415dd70 2024-02-19T15:30:14.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.465+0000 7faf4b400640 1 -- 172.21.15.47:0/595387488 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf4c003ce0 con 0x7faf5415dd70 2024-02-19T15:30:14.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.465+0000 7faf4b400640 1 -- 172.21.15.47:0/595387488 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c00aa90 con 0x7faf5415dd70 2024-02-19T15:30:14.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.465+0000 7faf58a00640 1 -- 172.21.15.47:0/595387488 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf54154940 con 0x7faf5415dd70 2024-02-19T15:30:14.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.465+0000 7faf58a00640 1 -- 172.21.15.47:0/595387488 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf54154f00 con 0x7faf5415dd70 2024-02-19T15:30:14.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.465+0000 7faf58a00640 1 -- 172.21.15.47:0/595387488 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf541556c0 con 0x7faf5415dd70 2024-02-19T15:30:14.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.465+0000 7faf4b400640 1 -- 172.21.15.47:0/595387488 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7faf4c00abf0 con 0x7faf5415dd70 2024-02-19T15:30:14.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.465+0000 7faf4b400640 1 --2- 172.21.15.47:0/595387488 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faf30066e40 0x7faf30069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:14.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.465+0000 7faf4b400640 1 -- 172.21.15.47:0/595387488 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7faf4c0847a0 con 0x7faf5415dd70 2024-02-19T15:30:14.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.465+0000 7faf53400640 1 --2- 172.21.15.47:0/595387488 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faf30066e40 0x7faf30069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:14.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.469+0000 7faf53400640 1 --2- 172.21.15.47:0/595387488 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faf30066e40 0x7faf30069300 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7faf440ae000 tx=0x7faf440a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:14.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.473+0000 7faf4b400640 1 -- 172.21.15.47:0/595387488 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7faf4c051600 con 0x7faf5415dd70 2024-02-19T15:30:14.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.673+0000 7faf58a00640 1 -- 172.21.15.47:0/595387488 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf54075360 con 0x7faf5415dd70 2024-02-19T15:30:14.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.673+0000 7faf4b400640 1 -- 172.21.15.47:0/595387488 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7faf4c054cc0 con 0x7faf5415dd70 2024-02-19T15:30:14.672 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:14.673 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:14.673 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:14.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.673+0000 7faf58a00640 1 -- 172.21.15.47:0/595387488 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faf30066e40 msgr2=0x7faf30069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:14.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.673+0000 7faf58a00640 1 --2- 172.21.15.47:0/595387488 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faf30066e40 0x7faf30069300 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7faf440ae000 tx=0x7faf440a7000 comp rx=0 tx=0).stop 2024-02-19T15:30:14.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.673+0000 7faf58a00640 1 -- 172.21.15.47:0/595387488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf5415dd70 msgr2=0x7faf54159b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:14.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.673+0000 7faf58a00640 1 --2- 172.21.15.47:0/595387488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf5415dd70 0x7faf54159b50 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7faf4c00c770 tx=0x7faf4c00cc30 comp rx=0 tx=0).stop 2024-02-19T15:30:14.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.673+0000 7faf58a00640 1 -- 172.21.15.47:0/595387488 shutdown_connections 2024-02-19T15:30:14.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.673+0000 7faf58a00640 1 --2- 172.21.15.47:0/595387488 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faf30066e40 0x7faf30069300 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:14.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.673+0000 7faf58a00640 1 --2- 172.21.15.47:0/595387488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf5415dd70 0x7faf54159b50 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:14.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.673+0000 7faf58a00640 1 --2- 172.21.15.47:0/595387488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf540aeda0 0x7faf54153e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:14.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.673+0000 7faf58a00640 1 --2- 172.21.15.47:0/595387488 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf5406d660 0x7faf5409b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:14.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.673+0000 7faf58a00640 1 -- 172.21.15.47:0/595387488 >> 172.21.15.47:0/595387488 conn(0x7faf5409a9d0 msgr2=0x7faf540cfb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:14.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.673+0000 7faf58a00640 1 -- 172.21.15.47:0/595387488 shutdown_connections 2024-02-19T15:30:14.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:14.673+0000 7faf58a00640 1 -- 172.21.15.47:0/595387488 wait complete. 2024-02-19T15:30:14.895 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:14.895 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:15.896 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:16.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.037+0000 7fc169400640 1 Processor -- start 2024-02-19T15:30:16.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.037+0000 7fc169400640 1 -- start start 2024-02-19T15:30:16.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.037+0000 7fc169400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc16415dd70 0x7fc1640ae860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:16.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.037+0000 7fc169400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc1640aeda0 0x7fc16406d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:16.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.037+0000 7fc169400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc16406dab0 con 0x7fc16406d660 2024-02-19T15:30:16.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.037+0000 7fc169400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc16415f3f0 con 0x7fc1640aeda0 2024-02-19T15:30:16.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.037+0000 7fc169400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc16406c3e0 con 0x7fc16415dd70 2024-02-19T15:30:16.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.037+0000 7fc15be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc1640aeda0 0x7fc16406d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:16.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.037+0000 7fc15be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc1640aeda0 0x7fc16406d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60622/0 (socket says 172.21.15.47:60622) 2024-02-19T15:30:16.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.037+0000 7fc15be00640 1 -- 172.21.15.47:0/3488594683 learned_addr learned my addr 172.21.15.47:0/3488594683 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:16.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.037+0000 7fc163e00640 1 --2- 172.21.15.47:0/3488594683 >> v2:172.21.15.116:3300/0 conn(0x7fc16415dd70 0x7fc1640ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:16.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.037+0000 7fc15be00640 1 -- 172.21.15.47:0/3488594683 >> v2:172.21.15.116:3300/0 conn(0x7fc16415dd70 msgr2=0x7fc1640ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:16.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.037+0000 7fc15be00640 1 --2- 172.21.15.47:0/3488594683 >> v2:172.21.15.116:3300/0 conn(0x7fc16415dd70 0x7fc1640ae860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:16.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.037+0000 7fc15be00640 1 -- 172.21.15.47:0/3488594683 >> v1:172.21.15.116:6789/0 conn(0x7fc16406d660 legacy=0x7fc1640d19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:16.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.037+0000 7fc15be00640 1 -- 172.21.15.47:0/3488594683 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16406c680 con 0x7fc1640aeda0 2024-02-19T15:30:16.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.041+0000 7fc15be00640 1 --2- 172.21.15.47:0/3488594683 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1640aeda0 0x7fc16406d120 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fc16406e880 tx=0x7fc14c01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=561a9630fe8bdc15 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:16.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.041+0000 7fc163400640 1 -- 172.21.15.47:0/3488594683 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c0ab620 con 0x7fc1640aeda0 2024-02-19T15:30:16.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.041+0000 7fc163400640 1 -- 172.21.15.47:0/3488594683 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14c0ab780 con 0x7fc1640aeda0 2024-02-19T15:30:16.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.041+0000 7fc163400640 1 -- 172.21.15.47:0/3488594683 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c0abb10 con 0x7fc1640aeda0 2024-02-19T15:30:16.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.041+0000 7fc169400640 1 -- 172.21.15.47:0/3488594683 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1640aeda0 msgr2=0x7fc16406d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:16.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.041+0000 7fc169400640 1 --2- 172.21.15.47:0/3488594683 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1640aeda0 0x7fc16406d120 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fc16406e880 tx=0x7fc14c01c420 comp rx=0 tx=0).stop 2024-02-19T15:30:16.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.041+0000 7fc169400640 1 -- 172.21.15.47:0/3488594683 shutdown_connections 2024-02-19T15:30:16.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.041+0000 7fc169400640 1 --2- 172.21.15.47:0/3488594683 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1640aeda0 0x7fc16406d120 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:16.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.041+0000 7fc169400640 1 --2- 172.21.15.47:0/3488594683 >> v2:172.21.15.116:3300/0 conn(0x7fc16415dd70 0x7fc1640ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:16.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.041+0000 7fc169400640 1 -- 172.21.15.47:0/3488594683 >> 172.21.15.47:0/3488594683 conn(0x7fc16409a9d0 msgr2=0x7fc16409ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:16.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.041+0000 7fc169400640 1 -- 172.21.15.47:0/3488594683 shutdown_connections 2024-02-19T15:30:16.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.041+0000 7fc169400640 1 -- 172.21.15.47:0/3488594683 wait complete. 2024-02-19T15:30:16.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc169400640 1 Processor -- start 2024-02-19T15:30:16.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc169400640 1 -- start start 2024-02-19T15:30:16.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc169400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc16406d660 0x7fc16415ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:16.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc169400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1640aeda0 0x7fc16415c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:16.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc169400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc16415dd70 0x7fc16415c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:16.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc169400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1640b4610 con 0x7fc16415dd70 2024-02-19T15:30:16.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc169400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1640b48f0 con 0x7fc1640aeda0 2024-02-19T15:30:16.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc169400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1640b4bd0 con 0x7fc16406d660 2024-02-19T15:30:16.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc168a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc16415dd70 0x7fc16415c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:16.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc168a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc16415dd70 0x7fc16415c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60636/0 (socket says 172.21.15.47:60636) 2024-02-19T15:30:16.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc168a00640 1 -- 172.21.15.47:0/310174511 learned_addr learned my addr 172.21.15.47:0/310174511 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:16.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc163e00640 1 --2- 172.21.15.47:0/310174511 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc16406d660 0x7fc16415ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:16.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc15be00640 1 --2- 172.21.15.47:0/310174511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1640aeda0 0x7fc16415c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:16.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc168a00640 1 -- 172.21.15.47:0/310174511 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc16406d660 msgr2=0x7fc16415ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:16.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc168a00640 1 --2- 172.21.15.47:0/310174511 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc16406d660 0x7fc16415ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:16.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc168a00640 1 -- 172.21.15.47:0/310174511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1640aeda0 msgr2=0x7fc16415c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:16.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc168a00640 1 --2- 172.21.15.47:0/310174511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1640aeda0 0x7fc16415c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:16.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc168a00640 1 -- 172.21.15.47:0/310174511 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14c0094d0 con 0x7fc16415dd70 2024-02-19T15:30:16.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc168a00640 1 --2- 172.21.15.47:0/310174511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc16415dd70 0x7fc16415c550 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fc15c00c780 tx=0x7fc15c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:16.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc160c00640 1 -- 172.21.15.47:0/310174511 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c004b70 con 0x7fc16415dd70 2024-02-19T15:30:16.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc160c00640 1 -- 172.21.15.47:0/310174511 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15c00ce90 con 0x7fc16415dd70 2024-02-19T15:30:16.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc160c00640 1 -- 172.21.15.47:0/310174511 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c00a850 con 0x7fc16415dd70 2024-02-19T15:30:16.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc169400640 1 -- 172.21.15.47:0/310174511 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1640b4eb0 con 0x7fc16415dd70 2024-02-19T15:30:16.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.045+0000 7fc169400640 1 -- 172.21.15.47:0/310174511 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc16415d3c0 con 0x7fc16415dd70 2024-02-19T15:30:16.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.049+0000 7fc169400640 1 -- 172.21.15.47:0/310174511 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc164075360 con 0x7fc16415dd70 2024-02-19T15:30:16.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.049+0000 7fc160c00640 1 -- 172.21.15.47:0/310174511 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc15c00a9b0 con 0x7fc16415dd70 2024-02-19T15:30:16.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.049+0000 7fc160c00640 1 --2- 172.21.15.47:0/310174511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc140066e40 0x7fc140069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:16.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.049+0000 7fc160c00640 1 -- 172.21.15.47:0/310174511 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc15c084b60 con 0x7fc16415dd70 2024-02-19T15:30:16.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.049+0000 7fc163e00640 1 --2- 172.21.15.47:0/310174511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc140066e40 0x7fc140069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:16.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.049+0000 7fc163e00640 1 --2- 172.21.15.47:0/310174511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc140066e40 0x7fc140069300 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fc1540079c0 tx=0x7fc154008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:16.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.057+0000 7fc160c00640 1 -- 172.21.15.47:0/310174511 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc15c051920 con 0x7fc16415dd70 2024-02-19T15:30:16.202 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:16.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.257+0000 7fc169400640 1 -- 172.21.15.47:0/310174511 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1640b5360 con 0x7fc16415dd70 2024-02-19T15:30:16.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.257+0000 7fc160c00640 1 -- 172.21.15.47:0/310174511 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7fc15c054fe0 con 0x7fc16415dd70 2024-02-19T15:30:16.259 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:16.259 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:16.259 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:16.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.261+0000 7fc169400640 1 -- 172.21.15.47:0/310174511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc140066e40 msgr2=0x7fc140069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:16.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.261+0000 7fc169400640 1 --2- 172.21.15.47:0/310174511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc140066e40 0x7fc140069300 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fc1540079c0 tx=0x7fc154008040 comp rx=0 tx=0).stop 2024-02-19T15:30:16.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.261+0000 7fc169400640 1 -- 172.21.15.47:0/310174511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc16415dd70 msgr2=0x7fc16415c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:16.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.261+0000 7fc169400640 1 --2- 172.21.15.47:0/310174511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc16415dd70 0x7fc16415c550 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fc15c00c780 tx=0x7fc15c00cc40 comp rx=0 tx=0).stop 2024-02-19T15:30:16.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.261+0000 7fc169400640 1 -- 172.21.15.47:0/310174511 shutdown_connections 2024-02-19T15:30:16.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.261+0000 7fc169400640 1 --2- 172.21.15.47:0/310174511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc140066e40 0x7fc140069300 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:16.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.261+0000 7fc169400640 1 --2- 172.21.15.47:0/310174511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc16415dd70 0x7fc16415c550 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:16.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.261+0000 7fc169400640 1 --2- 172.21.15.47:0/310174511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1640aeda0 0x7fc16415c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:16.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.261+0000 7fc169400640 1 --2- 172.21.15.47:0/310174511 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc16406d660 0x7fc16415ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:16.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.261+0000 7fc169400640 1 -- 172.21.15.47:0/310174511 >> 172.21.15.47:0/310174511 conn(0x7fc16409a9d0 msgr2=0x7fc16415a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:16.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.261+0000 7fc169400640 1 -- 172.21.15.47:0/310174511 shutdown_connections 2024-02-19T15:30:16.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:16.261+0000 7fc169400640 1 -- 172.21.15.47:0/310174511 wait complete. 2024-02-19T15:30:16.481 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:16.481 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:17.482 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:17.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.621+0000 7f0d80000640 1 Processor -- start 2024-02-19T15:30:17.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.621+0000 7f0d80000640 1 -- start start 2024-02-19T15:30:17.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.625+0000 7f0d80000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d780cf920 0x7f0d780b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:17.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.625+0000 7f0d80000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0d780d0830 0x7f0d780cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:17.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.625+0000 7f0d80000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d780d2800 con 0x7f0d7815c390 2024-02-19T15:30:17.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.625+0000 7f0d80000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d780d2ae0 con 0x7f0d780d0830 2024-02-19T15:30:17.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.625+0000 7f0d80000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d780cfdb0 con 0x7f0d780cf920 2024-02-19T15:30:17.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.625+0000 7f0d7cc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d780cf920 0x7f0d780b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:17.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.625+0000 7f0d7d600640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0d7815c390 0x7f0d780d0120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:41188/0 (socket says 172.21.15.47:41188) 2024-02-19T15:30:17.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.625+0000 7f0d7d600640 1 -- 172.21.15.47:0/414579165 learned_addr learned my addr 172.21.15.47:0/414579165 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:17.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.625+0000 7f0d7e000640 1 --2- 172.21.15.47:0/414579165 >> v2:172.21.15.116:3300/0 conn(0x7f0d780d0830 0x7f0d780cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:17.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.625+0000 7f0d77e00640 1 -- 172.21.15.47:0/414579165 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939047578 0 0) 0x7f0d780d2800 con 0x7f0d7815c390 2024-02-19T15:30:17.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.625+0000 7f0d7cc00640 1 -- 172.21.15.47:0/414579165 >> v2:172.21.15.116:3300/0 conn(0x7f0d780d0830 msgr2=0x7f0d780cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:17.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.625+0000 7f0d7cc00640 1 --2- 172.21.15.47:0/414579165 >> v2:172.21.15.116:3300/0 conn(0x7f0d780d0830 0x7f0d780cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:17.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.625+0000 7f0d7cc00640 1 -- 172.21.15.47:0/414579165 >> v1:172.21.15.116:6789/0 conn(0x7f0d7815c390 legacy=0x7f0d780d0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:17.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.625+0000 7f0d7cc00640 1 -- 172.21.15.47:0/414579165 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d780b2650 con 0x7f0d780cf920 2024-02-19T15:30:17.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.629+0000 7f0d7cc00640 1 --2- 172.21.15.47:0/414579165 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d780cf920 0x7f0d780b2c10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0d640097c0 tx=0x7f0d6401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42227754c108065c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:17.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.629+0000 7f0d77e00640 1 -- 172.21.15.47:0/414579165 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d640ab600 con 0x7f0d780cf920 2024-02-19T15:30:17.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.629+0000 7f0d77e00640 1 -- 172.21.15.47:0/414579165 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d640ab760 con 0x7f0d780cf920 2024-02-19T15:30:17.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.629+0000 7f0d77e00640 1 -- 172.21.15.47:0/414579165 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d640abaf0 con 0x7f0d780cf920 2024-02-19T15:30:17.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.629+0000 7f0d80000640 1 -- 172.21.15.47:0/414579165 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d780cf920 msgr2=0x7f0d780b2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:17.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.629+0000 7f0d80000640 1 --2- 172.21.15.47:0/414579165 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d780cf920 0x7f0d780b2c10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0d640097c0 tx=0x7f0d6401c3e0 comp rx=0 tx=0).stop 2024-02-19T15:30:17.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.629+0000 7f0d80000640 1 -- 172.21.15.47:0/414579165 shutdown_connections 2024-02-19T15:30:17.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.629+0000 7f0d80000640 1 --2- 172.21.15.47:0/414579165 >> v2:172.21.15.116:3300/0 conn(0x7f0d780d0830 0x7f0d780cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:17.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.629+0000 7f0d80000640 1 --2- 172.21.15.47:0/414579165 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d780cf920 0x7f0d780b2c10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:17.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.629+0000 7f0d80000640 1 -- 172.21.15.47:0/414579165 >> 172.21.15.47:0/414579165 conn(0x7f0d780d1b40 msgr2=0x7f0d780d1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:17.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.629+0000 7f0d80000640 1 -- 172.21.15.47:0/414579165 shutdown_connections 2024-02-19T15:30:17.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.629+0000 7f0d80000640 1 -- 172.21.15.47:0/414579165 wait complete. 2024-02-19T15:30:17.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.629+0000 7f0d80000640 1 Processor -- start 2024-02-19T15:30:17.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.629+0000 7f0d80000640 1 -- start start 2024-02-19T15:30:17.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d80000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d780d0830 0x7f0d7806c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:17.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d80000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d7815c390 0x7f0d781663b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:17.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d80000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d78153680 0x7f0d78153b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:17.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d80000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d78154eb0 con 0x7f0d780d0830 2024-02-19T15:30:17.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d80000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d781543c0 con 0x7f0d7815c390 2024-02-19T15:30:17.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d80000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d781668f0 con 0x7f0d78153680 2024-02-19T15:30:17.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d7d600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d780d0830 0x7f0d7806c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:17.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d7d600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d780d0830 0x7f0d7806c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60640/0 (socket says 172.21.15.47:60640) 2024-02-19T15:30:17.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d7d600640 1 -- 172.21.15.47:0/2172267572 learned_addr learned my addr 172.21.15.47:0/2172267572 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:17.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d7cc00640 1 --2- 172.21.15.47:0/2172267572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d7815c390 0x7f0d781663b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:17.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d7e000640 1 --2- 172.21.15.47:0/2172267572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d78153680 0x7f0d78153b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:17.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d7d600640 1 -- 172.21.15.47:0/2172267572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d78153680 msgr2=0x7f0d78153b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:17.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d7d600640 1 --2- 172.21.15.47:0/2172267572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d78153680 0x7f0d78153b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:17.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d7d600640 1 -- 172.21.15.47:0/2172267572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d7815c390 msgr2=0x7f0d781663b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:17.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d7d600640 1 --2- 172.21.15.47:0/2172267572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d7815c390 0x7f0d781663b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:17.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d7d600640 1 -- 172.21.15.47:0/2172267572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d64009420 con 0x7f0d780d0830 2024-02-19T15:30:17.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d7d600640 1 --2- 172.21.15.47:0/2172267572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d780d0830 0x7f0d7806c000 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c00c4d0 tx=0x7f0d6c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:17.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d75600640 1 -- 172.21.15.47:0/2172267572 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c0074e0 con 0x7f0d780d0830 2024-02-19T15:30:17.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d75600640 1 -- 172.21.15.47:0/2172267572 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6c007640 con 0x7f0d780d0830 2024-02-19T15:30:17.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d80000640 1 -- 172.21.15.47:0/2172267572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d78154610 con 0x7f0d780d0830 2024-02-19T15:30:17.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d75600640 1 -- 172.21.15.47:0/2172267572 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c004a30 con 0x7f0d780d0830 2024-02-19T15:30:17.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d80000640 1 -- 172.21.15.47:0/2172267572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d78154bd0 con 0x7f0d780d0830 2024-02-19T15:30:17.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d80000640 1 -- 172.21.15.47:0/2172267572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d78075360 con 0x7f0d780d0830 2024-02-19T15:30:17.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.633+0000 7f0d75600640 1 -- 172.21.15.47:0/2172267572 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0d6c004b90 con 0x7f0d780d0830 2024-02-19T15:30:17.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.637+0000 7f0d75600640 1 --2- 172.21.15.47:0/2172267572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d58066e40 0x7f0d58069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:17.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.637+0000 7f0d75600640 1 -- 172.21.15.47:0/2172267572 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0d6c083d60 con 0x7f0d780d0830 2024-02-19T15:30:17.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.637+0000 7f0d7cc00640 1 --2- 172.21.15.47:0/2172267572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d58066e40 0x7f0d58069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:17.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.637+0000 7f0d7cc00640 1 --2- 172.21.15.47:0/2172267572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d58066e40 0x7f0d58069300 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f0d640af000 tx=0x7f0d640a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:17.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.641+0000 7f0d75600640 1 -- 172.21.15.47:0/2172267572 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0d6c050aa0 con 0x7f0d780d0830 2024-02-19T15:30:17.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.837+0000 7f0d80000640 1 -- 172.21.15.47:0/2172267572 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d780cff00 con 0x7f0d780d0830 2024-02-19T15:30:17.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.837+0000 7f0d75600640 1 -- 172.21.15.47:0/2172267572 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f0d6c018020 con 0x7f0d780d0830 2024-02-19T15:30:17.837 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:17.838 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:17.838 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:17.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.837+0000 7f0d80000640 1 -- 172.21.15.47:0/2172267572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d58066e40 msgr2=0x7f0d58069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:17.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.837+0000 7f0d80000640 1 --2- 172.21.15.47:0/2172267572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d58066e40 0x7f0d58069300 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f0d640af000 tx=0x7f0d640a7000 comp rx=0 tx=0).stop 2024-02-19T15:30:17.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.837+0000 7f0d80000640 1 -- 172.21.15.47:0/2172267572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d780d0830 msgr2=0x7f0d7806c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:17.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.837+0000 7f0d80000640 1 --2- 172.21.15.47:0/2172267572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d780d0830 0x7f0d7806c000 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c00c4d0 tx=0x7f0d6c00c990 comp rx=0 tx=0).stop 2024-02-19T15:30:17.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.837+0000 7f0d80000640 1 -- 172.21.15.47:0/2172267572 shutdown_connections 2024-02-19T15:30:17.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.837+0000 7f0d80000640 1 --2- 172.21.15.47:0/2172267572 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d58066e40 0x7f0d58069300 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:17.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.837+0000 7f0d80000640 1 --2- 172.21.15.47:0/2172267572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d78153680 0x7f0d78153b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:17.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.837+0000 7f0d80000640 1 --2- 172.21.15.47:0/2172267572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d7815c390 0x7f0d781663b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:17.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.837+0000 7f0d80000640 1 --2- 172.21.15.47:0/2172267572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d780d0830 0x7f0d7806c000 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:17.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.837+0000 7f0d80000640 1 -- 172.21.15.47:0/2172267572 >> 172.21.15.47:0/2172267572 conn(0x7f0d780d1b40 msgr2=0x7f0d780d0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:17.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.841+0000 7f0d80000640 1 -- 172.21.15.47:0/2172267572 shutdown_connections 2024-02-19T15:30:17.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:17.841+0000 7f0d80000640 1 -- 172.21.15.47:0/2172267572 wait complete. 2024-02-19T15:30:18.060 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:18.060 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:18.205 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:19.061 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:19.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.201+0000 7fc9fe400640 1 Processor -- start 2024-02-19T15:30:19.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.201+0000 7fc9fe400640 1 -- start start 2024-02-19T15:30:19.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.205+0000 7fc9fe400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9f815bcd0 0x7fc9f80cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:19.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.205+0000 7fc9fe400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f80cc700 con 0x7fc9f80d0360 2024-02-19T15:30:19.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.205+0000 7fc9fe400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f80cf300 con 0x7fc9f80cedb0 2024-02-19T15:30:19.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.205+0000 7fc9fe400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f80cf5e0 con 0x7fc9f815bcd0 2024-02-19T15:30:19.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.205+0000 7fc9f7e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc9f80d0360 0x7fc9f815d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37312/0 (socket says 172.21.15.47:37312) 2024-02-19T15:30:19.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.205+0000 7fc9f7e00640 1 -- 172.21.15.47:0/52433803 learned_addr learned my addr 172.21.15.47:0/52433803 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:19.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.205+0000 7fc9f7400640 1 --2- 172.21.15.47:0/52433803 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9f815bcd0 0x7fc9f80cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:19.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.205+0000 7fc9f6a00640 1 -- 172.21.15.47:0/52433803 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732921752 0 0) 0x7fc9f80cc700 con 0x7fc9f80d0360 2024-02-19T15:30:19.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.205+0000 7fc9f6a00640 1 -- 172.21.15.47:0/52433803 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9dc003530 con 0x7fc9f80d0360 2024-02-19T15:30:19.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.205+0000 7fc9f6a00640 1 -- 172.21.15.47:0/52433803 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081769774 0 0) 0x7fc9f80cf300 con 0x7fc9f80cedb0 2024-02-19T15:30:19.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.205+0000 7fc9f7400640 1 -- 172.21.15.47:0/52433803 >> v1:172.21.15.116:6789/0 conn(0x7fc9f80cedb0 legacy=0x7fc9f80cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:19.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.205+0000 7fc9f7400640 1 -- 172.21.15.47:0/52433803 >> v1:172.21.15.47:6789/0 conn(0x7fc9f80d0360 legacy=0x7fc9f815d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:19.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.205+0000 7fc9f7400640 1 -- 172.21.15.47:0/52433803 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f80cc9a0 con 0x7fc9f815bcd0 2024-02-19T15:30:19.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.209+0000 7fc9f7400640 1 --2- 172.21.15.47:0/52433803 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9f815bcd0 0x7fc9f80cd480 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc9e80097f0 tx=0x7fc9e801c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35f7ae2c57b9d905 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:19.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.209+0000 7fc9f6a00640 1 -- 172.21.15.47:0/52433803 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e80aa610 con 0x7fc9f815bcd0 2024-02-19T15:30:19.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.209+0000 7fc9f6a00640 1 -- 172.21.15.47:0/52433803 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e80a6040 con 0x7fc9f815bcd0 2024-02-19T15:30:19.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.209+0000 7fc9f6a00640 1 -- 172.21.15.47:0/52433803 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e80aa990 con 0x7fc9f815bcd0 2024-02-19T15:30:19.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.209+0000 7fc9fe400640 1 -- 172.21.15.47:0/52433803 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9f815bcd0 msgr2=0x7fc9f80cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:19.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.209+0000 7fc9fe400640 1 --2- 172.21.15.47:0/52433803 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9f815bcd0 0x7fc9f80cd480 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc9e80097f0 tx=0x7fc9e801c4b0 comp rx=0 tx=0).stop 2024-02-19T15:30:19.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.209+0000 7fc9fe400640 1 -- 172.21.15.47:0/52433803 shutdown_connections 2024-02-19T15:30:19.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.209+0000 7fc9fe400640 1 --2- 172.21.15.47:0/52433803 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9f815bcd0 0x7fc9f80cd480 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:19.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.209+0000 7fc9fe400640 1 -- 172.21.15.47:0/52433803 >> 172.21.15.47:0/52433803 conn(0x7fc9f80d1bf0 msgr2=0x7fc9f80d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:19.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.209+0000 7fc9fe400640 1 -- 172.21.15.47:0/52433803 shutdown_connections 2024-02-19T15:30:19.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.209+0000 7fc9fe400640 1 -- 172.21.15.47:0/52433803 wait complete. 2024-02-19T15:30:19.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.209+0000 7fc9fe400640 1 Processor -- start 2024-02-19T15:30:19.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.209+0000 7fc9fe400640 1 -- start start 2024-02-19T15:30:19.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9fe400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9f80cedb0 0x7fc9f8158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:19.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9fe400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9f80d0360 0x7fc9f80c5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:19.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9fe400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9f815bcd0 0x7fc9f8154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:19.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9fe400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f80b61f0 con 0x7fc9f80d0360 2024-02-19T15:30:19.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9fe400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f80b55e0 con 0x7fc9f815bcd0 2024-02-19T15:30:19.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9fe400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f80b58c0 con 0x7fc9f80cedb0 2024-02-19T15:30:19.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9eea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9f80d0360 0x7fc9f80c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:19.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9eea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9f80d0360 0x7fc9f80c5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60650/0 (socket says 172.21.15.47:60650) 2024-02-19T15:30:19.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9f7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9f815bcd0 0x7fc9f8154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:19.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9eea00640 1 -- 172.21.15.47:0/3981693627 learned_addr learned my addr 172.21.15.47:0/3981693627 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:19.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9f7400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9f80cedb0 0x7fc9f8158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:19.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9eea00640 1 -- 172.21.15.47:0/3981693627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9f80cedb0 msgr2=0x7fc9f8158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:19.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9eea00640 1 --2- 172.21.15.47:0/3981693627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9f80cedb0 0x7fc9f8158440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:19.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9eea00640 1 -- 172.21.15.47:0/3981693627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9f815bcd0 msgr2=0x7fc9f8154880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:19.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9eea00640 1 --2- 172.21.15.47:0/3981693627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9f815bcd0 0x7fc9f8154880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:19.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9eea00640 1 -- 172.21.15.47:0/3981693627 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e8009420 con 0x7fc9f80d0360 2024-02-19T15:30:19.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9eea00640 1 --2- 172.21.15.47:0/3981693627 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9f80d0360 0x7fc9f80c5280 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0001ec0 tx=0x7fc9e000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:19.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9efe00640 1 -- 172.21.15.47:0/3981693627 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e00054e0 con 0x7fc9f80d0360 2024-02-19T15:30:19.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9efe00640 1 -- 172.21.15.47:0/3981693627 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e0003600 con 0x7fc9f80d0360 2024-02-19T15:30:19.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9efe00640 1 -- 172.21.15.47:0/3981693627 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e00039b0 con 0x7fc9f80d0360 2024-02-19T15:30:19.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9fe400640 1 -- 172.21.15.47:0/3981693627 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f80cdb40 con 0x7fc9f80d0360 2024-02-19T15:30:19.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9fe400640 1 -- 172.21.15.47:0/3981693627 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f80b5e80 con 0x7fc9f80d0360 2024-02-19T15:30:19.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.213+0000 7fc9fe400640 1 -- 172.21.15.47:0/3981693627 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f8075360 con 0x7fc9f80d0360 2024-02-19T15:30:19.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.217+0000 7fc9efe00640 1 -- 172.21.15.47:0/3981693627 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc9e0003b10 con 0x7fc9f80d0360 2024-02-19T15:30:19.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.217+0000 7fc9efe00640 1 --2- 172.21.15.47:0/3981693627 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc9d4066e40 0x7fc9d4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:19.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.217+0000 7fc9efe00640 1 -- 172.21.15.47:0/3981693627 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc9e0084090 con 0x7fc9f80d0360 2024-02-19T15:30:19.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.217+0000 7fc9f7400640 1 --2- 172.21.15.47:0/3981693627 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc9d4066e40 0x7fc9d4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:19.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.217+0000 7fc9f7400640 1 --2- 172.21.15.47:0/3981693627 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc9d4066e40 0x7fc9d4069300 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc9e80ae000 tx=0x7fc9e80a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:19.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.221+0000 7fc9efe00640 1 -- 172.21.15.47:0/3981693627 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc9e0050e50 con 0x7fc9f80d0360 2024-02-19T15:30:19.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.417+0000 7fc9fe400640 1 -- 172.21.15.47:0/3981693627 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9f80d1590 con 0x7fc9f80d0360 2024-02-19T15:30:19.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.417+0000 7fc9efe00640 1 -- 172.21.15.47:0/3981693627 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7fc9e0017090 con 0x7fc9f80d0360 2024-02-19T15:30:19.418 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:19.418 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:19.418 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.421+0000 7fc9fe400640 1 -- 172.21.15.47:0/3981693627 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc9d4066e40 msgr2=0x7fc9d4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.421+0000 7fc9fe400640 1 --2- 172.21.15.47:0/3981693627 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc9d4066e40 0x7fc9d4069300 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc9e80ae000 tx=0x7fc9e80a7000 comp rx=0 tx=0).stop 2024-02-19T15:30:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.421+0000 7fc9fe400640 1 -- 172.21.15.47:0/3981693627 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9f80d0360 msgr2=0x7fc9f80c5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.421+0000 7fc9fe400640 1 --2- 172.21.15.47:0/3981693627 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9f80d0360 0x7fc9f80c5280 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0001ec0 tx=0x7fc9e000c8e0 comp rx=0 tx=0).stop 2024-02-19T15:30:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.421+0000 7fc9fe400640 1 -- 172.21.15.47:0/3981693627 shutdown_connections 2024-02-19T15:30:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.421+0000 7fc9fe400640 1 --2- 172.21.15.47:0/3981693627 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc9d4066e40 0x7fc9d4069300 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.421+0000 7fc9fe400640 1 --2- 172.21.15.47:0/3981693627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9f815bcd0 0x7fc9f8154880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.421+0000 7fc9fe400640 1 --2- 172.21.15.47:0/3981693627 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc9f80d0360 0x7fc9f80c5280 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.421+0000 7fc9fe400640 1 --2- 172.21.15.47:0/3981693627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9f80cedb0 0x7fc9f8158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.421+0000 7fc9fe400640 1 -- 172.21.15.47:0/3981693627 >> 172.21.15.47:0/3981693627 conn(0x7fc9f80d1bf0 msgr2=0x7fc9f815c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:19.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.421+0000 7fc9fe400640 1 -- 172.21.15.47:0/3981693627 shutdown_connections 2024-02-19T15:30:19.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:19.421+0000 7fc9fe400640 1 -- 172.21.15.47:0/3981693627 wait complete. 2024-02-19T15:30:19.649 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:19.649 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:20.205 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:20.649 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:20.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.789+0000 7f765a600640 1 Processor -- start 2024-02-19T15:30:20.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.789+0000 7f765a600640 1 -- start start 2024-02-19T15:30:20.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.789+0000 7f765a600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f765415bcd0 0x7f76540cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:20.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.789+0000 7f765a600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f76540cedb0 0x7f76540cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:20.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.789+0000 7f765a600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76540cc700 con 0x7f76540d2710 2024-02-19T15:30:20.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.789+0000 7f765a600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76540cf5e0 con 0x7f76540cedb0 2024-02-19T15:30:20.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.789+0000 7f765a600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76540cf300 con 0x7f765415bcd0 2024-02-19T15:30:20.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.789+0000 7f7653e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f76540d2710 0x7f765415d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37326/0 (socket says 172.21.15.47:37326) 2024-02-19T15:30:20.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.789+0000 7f7653e00640 1 -- 172.21.15.47:0/1573939529 learned_addr learned my addr 172.21.15.47:0/1573939529 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:20.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.793+0000 7f764aa00640 1 --2- 172.21.15.47:0/1573939529 >> v2:172.21.15.47:3300/0 conn(0x7f76540cedb0 0x7f76540cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:20.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.793+0000 7f7653400640 1 --2- 172.21.15.47:0/1573939529 >> v2:172.21.15.116:3300/0 conn(0x7f765415bcd0 0x7f76540cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:20.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.793+0000 7f7652a00640 1 -- 172.21.15.47:0/1573939529 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440621576 0 0) 0x7f76540cc700 con 0x7f76540d2710 2024-02-19T15:30:20.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.793+0000 7f7652a00640 1 -- 172.21.15.47:0/1573939529 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7638003530 con 0x7f76540d2710 2024-02-19T15:30:20.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.793+0000 7f764aa00640 1 -- 172.21.15.47:0/1573939529 >> v2:172.21.15.116:3300/0 conn(0x7f765415bcd0 msgr2=0x7f76540cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:20.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.793+0000 7f764aa00640 1 --2- 172.21.15.47:0/1573939529 >> v2:172.21.15.116:3300/0 conn(0x7f765415bcd0 0x7f76540cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:20.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.793+0000 7f764aa00640 1 -- 172.21.15.47:0/1573939529 >> v1:172.21.15.47:6789/0 conn(0x7f76540d2710 legacy=0x7f765415d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:20.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.793+0000 7f764aa00640 1 -- 172.21.15.47:0/1573939529 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76540cc9a0 con 0x7f76540cedb0 2024-02-19T15:30:20.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.793+0000 7f764aa00640 1 --2- 172.21.15.47:0/1573939529 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76540cedb0 0x7f76540cfc50 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f7654071a60 tx=0x7f763c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad9752291ccb7a1a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:20.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.793+0000 7f7652a00640 1 -- 172.21.15.47:0/1573939529 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c0ad660 con 0x7f76540cedb0 2024-02-19T15:30:20.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.793+0000 7f7652a00640 1 -- 172.21.15.47:0/1573939529 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763c0ad7c0 con 0x7f76540cedb0 2024-02-19T15:30:20.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.793+0000 7f7652a00640 1 -- 172.21.15.47:0/1573939529 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c0adb50 con 0x7f76540cedb0 2024-02-19T15:30:20.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.797+0000 7f765a600640 1 -- 172.21.15.47:0/1573939529 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76540cedb0 msgr2=0x7f76540cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:20.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.797+0000 7f765a600640 1 --2- 172.21.15.47:0/1573939529 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76540cedb0 0x7f76540cfc50 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f7654071a60 tx=0x7f763c01c450 comp rx=0 tx=0).stop 2024-02-19T15:30:20.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.797+0000 7f765a600640 1 -- 172.21.15.47:0/1573939529 shutdown_connections 2024-02-19T15:30:20.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.797+0000 7f765a600640 1 --2- 172.21.15.47:0/1573939529 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76540cedb0 0x7f76540cfc50 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:20.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.797+0000 7f765a600640 1 --2- 172.21.15.47:0/1573939529 >> v2:172.21.15.116:3300/0 conn(0x7f765415bcd0 0x7f76540cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:20.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.797+0000 7f765a600640 1 -- 172.21.15.47:0/1573939529 >> 172.21.15.47:0/1573939529 conn(0x7f76540d1bf0 msgr2=0x7f76540d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:20.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.797+0000 7f765a600640 1 -- 172.21.15.47:0/1573939529 shutdown_connections 2024-02-19T15:30:20.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.797+0000 7f765a600640 1 -- 172.21.15.47:0/1573939529 wait complete. 2024-02-19T15:30:20.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.797+0000 7f765a600640 1 Processor -- start 2024-02-19T15:30:20.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.797+0000 7f765a600640 1 -- start start 2024-02-19T15:30:20.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f765a600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76540cedb0 0x7f76541524a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:20.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f765a600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76540d2710 0x7f765415a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:20.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f765a600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765415bcd0 0x7f765406fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:20.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f765a600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76540c8e60 con 0x7f76540cedb0 2024-02-19T15:30:20.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f765a600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76540c9100 con 0x7f76540d2710 2024-02-19T15:30:20.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f765a600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76540c93e0 con 0x7f765415bcd0 2024-02-19T15:30:20.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f7653400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76540cedb0 0x7f76541524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:20.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f7653400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76540cedb0 0x7f76541524a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60668/0 (socket says 172.21.15.47:60668) 2024-02-19T15:30:20.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f7653400640 1 -- 172.21.15.47:0/793475077 learned_addr learned my addr 172.21.15.47:0/793475077 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:20.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f7653e00640 1 --2- 172.21.15.47:0/793475077 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765415bcd0 0x7f765406fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:20.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f764aa00640 1 --2- 172.21.15.47:0/793475077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76540d2710 0x7f765415a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:20.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f7653400640 1 -- 172.21.15.47:0/793475077 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765415bcd0 msgr2=0x7f765406fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:20.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f7653400640 1 --2- 172.21.15.47:0/793475077 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765415bcd0 0x7f765406fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:20.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f7653400640 1 -- 172.21.15.47:0/793475077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76540d2710 msgr2=0x7f765415a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:20.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f7653400640 1 --2- 172.21.15.47:0/793475077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76540d2710 0x7f765415a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:20.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f7653400640 1 -- 172.21.15.47:0/793475077 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f763c009420 con 0x7f76540cedb0 2024-02-19T15:30:20.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f764aa00640 1 --2- 172.21.15.47:0/793475077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76540d2710 0x7f765415a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:30:20.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f7653e00640 1 --2- 172.21.15.47:0/793475077 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765415bcd0 0x7f765406fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:30:20.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f7653400640 1 --2- 172.21.15.47:0/793475077 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76540cedb0 0x7f76541524a0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f764400e930 tx=0x7f764400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:20.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f764be00640 1 -- 172.21.15.47:0/793475077 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644009690 con 0x7f76540cedb0 2024-02-19T15:30:20.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f764be00640 1 -- 172.21.15.47:0/793475077 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764400ca00 con 0x7f76540cedb0 2024-02-19T15:30:20.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f764be00640 1 -- 172.21.15.47:0/793475077 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764400cd70 con 0x7f76540cedb0 2024-02-19T15:30:20.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f765a600640 1 -- 172.21.15.47:0/793475077 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76540cdb40 con 0x7f76540cedb0 2024-02-19T15:30:20.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f765a600640 1 -- 172.21.15.47:0/793475077 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76540c99a0 con 0x7f76540cedb0 2024-02-19T15:30:20.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.801+0000 7f765a600640 1 -- 172.21.15.47:0/793475077 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7654075360 con 0x7f76540cedb0 2024-02-19T15:30:20.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.805+0000 7f764be00640 1 -- 172.21.15.47:0/793475077 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7644013470 con 0x7f76540cedb0 2024-02-19T15:30:20.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.805+0000 7f764be00640 1 --2- 172.21.15.47:0/793475077 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7630066e40 0x7f7630069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:20.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.805+0000 7f764be00640 1 -- 172.21.15.47:0/793475077 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7644084d60 con 0x7f76540cedb0 2024-02-19T15:30:20.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.805+0000 7f764aa00640 1 --2- 172.21.15.47:0/793475077 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7630066e40 0x7f7630069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:20.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.805+0000 7f764aa00640 1 --2- 172.21.15.47:0/793475077 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7630066e40 0x7f7630069300 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f763c0af000 tx=0x7f763c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:20.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:20.809+0000 7f764be00640 1 -- 172.21.15.47:0/793475077 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7644051bc0 con 0x7f76540cedb0 2024-02-19T15:30:21.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:21.025+0000 7f765a600640 1 -- 172.21.15.47:0/793475077 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76540b3330 con 0x7f76540cedb0 2024-02-19T15:30:21.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:21.025+0000 7f764be00640 1 -- 172.21.15.47:0/793475077 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f7644021d80 con 0x7f76540cedb0 2024-02-19T15:30:21.024 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:21.024 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:21.024 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:21.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:21.025+0000 7f765a600640 1 -- 172.21.15.47:0/793475077 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7630066e40 msgr2=0x7f7630069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:21.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:21.025+0000 7f765a600640 1 --2- 172.21.15.47:0/793475077 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7630066e40 0x7f7630069300 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f763c0af000 tx=0x7f763c008040 comp rx=0 tx=0).stop 2024-02-19T15:30:21.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:21.025+0000 7f765a600640 1 -- 172.21.15.47:0/793475077 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76540cedb0 msgr2=0x7f76541524a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:21.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:21.025+0000 7f765a600640 1 --2- 172.21.15.47:0/793475077 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76540cedb0 0x7f76541524a0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f764400e930 tx=0x7f764400edf0 comp rx=0 tx=0).stop 2024-02-19T15:30:21.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:21.025+0000 7f765a600640 1 -- 172.21.15.47:0/793475077 shutdown_connections 2024-02-19T15:30:21.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:21.025+0000 7f765a600640 1 --2- 172.21.15.47:0/793475077 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7630066e40 0x7f7630069300 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:21.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:21.025+0000 7f765a600640 1 --2- 172.21.15.47:0/793475077 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765415bcd0 0x7f765406fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:21.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:21.025+0000 7f765a600640 1 --2- 172.21.15.47:0/793475077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76540d2710 0x7f765415a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:21.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:21.025+0000 7f765a600640 1 --2- 172.21.15.47:0/793475077 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f76540cedb0 0x7f76541524a0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:21.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:21.025+0000 7f765a600640 1 -- 172.21.15.47:0/793475077 >> 172.21.15.47:0/793475077 conn(0x7f76540d1bf0 msgr2=0x7f765415b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:21.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:21.025+0000 7f765a600640 1 -- 172.21.15.47:0/793475077 shutdown_connections 2024-02-19T15:30:21.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:21.025+0000 7f765a600640 1 -- 172.21.15.47:0/793475077 wait complete. 2024-02-19T15:30:21.251 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:21.251 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:22.207 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:22.252 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:22.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.393+0000 7f177fa00640 1 Processor -- start 2024-02-19T15:30:22.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.393+0000 7f177fa00640 1 -- start start 2024-02-19T15:30:22.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.397+0000 7f177fa00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f177815c0d0 0x7f17780cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:22.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.397+0000 7f177fa00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17780d0540 0x7f17780b1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:22.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.397+0000 7f177fa00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17780b0fe0 con 0x7f17780cd180 2024-02-19T15:30:22.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.397+0000 7f177fa00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f177815c4b0 con 0x7f177815c0d0 2024-02-19T15:30:22.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.397+0000 7f177fa00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17780b1880 con 0x7f17780d0540 2024-02-19T15:30:22.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.397+0000 7f177d000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f177815c0d0 0x7f17780cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:22.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.397+0000 7f177da00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17780d0540 0x7f17780b1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:22.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.397+0000 7f1777e00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f17780cd180 0x7f17780cfe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:41204/0 (socket says 172.21.15.47:41204) 2024-02-19T15:30:22.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.397+0000 7f1777e00640 1 -- 172.21.15.47:0/1591241212 learned_addr learned my addr 172.21.15.47:0/1591241212 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:22.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.397+0000 7f177da00640 1 -- 172.21.15.47:0/1591241212 >> v2:172.21.15.116:3300/0 conn(0x7f177815c0d0 msgr2=0x7f17780cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:22.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.397+0000 7f177da00640 1 --2- 172.21.15.47:0/1591241212 >> v2:172.21.15.116:3300/0 conn(0x7f177815c0d0 0x7f17780cf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:22.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.397+0000 7f177da00640 1 -- 172.21.15.47:0/1591241212 >> v1:172.21.15.116:6789/0 conn(0x7f17780cd180 legacy=0x7f17780cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:22.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.397+0000 7f177da00640 1 -- 172.21.15.47:0/1591241212 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17780cd8f0 con 0x7f17780d0540 2024-02-19T15:30:22.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.397+0000 7f177d000640 1 --2- 172.21.15.47:0/1591241212 >> v2:172.21.15.116:3300/0 conn(0x7f177815c0d0 0x7f17780cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:30:22.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.401+0000 7f177da00640 1 --2- 172.21.15.47:0/1591241212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17780d0540 0x7f17780b1d10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1770009d90 tx=0x7f177001c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e50b00105639f1b3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:22.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.401+0000 7f1777400640 1 -- 172.21.15.47:0/1591241212 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17700a5a00 con 0x7f17780d0540 2024-02-19T15:30:22.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.401+0000 7f1777400640 1 -- 172.21.15.47:0/1591241212 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17700a5b60 con 0x7f17780d0540 2024-02-19T15:30:22.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.401+0000 7f1777400640 1 -- 172.21.15.47:0/1591241212 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17700ad440 con 0x7f17780d0540 2024-02-19T15:30:22.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.401+0000 7f177fa00640 1 -- 172.21.15.47:0/1591241212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17780d0540 msgr2=0x7f17780b1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:22.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.401+0000 7f177fa00640 1 --2- 172.21.15.47:0/1591241212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17780d0540 0x7f17780b1d10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1770009d90 tx=0x7f177001c940 comp rx=0 tx=0).stop 2024-02-19T15:30:22.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.401+0000 7f177fa00640 1 -- 172.21.15.47:0/1591241212 shutdown_connections 2024-02-19T15:30:22.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.401+0000 7f177fa00640 1 --2- 172.21.15.47:0/1591241212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17780d0540 0x7f17780b1d10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:22.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.401+0000 7f177fa00640 1 --2- 172.21.15.47:0/1591241212 >> v2:172.21.15.116:3300/0 conn(0x7f177815c0d0 0x7f17780cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:22.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.401+0000 7f177fa00640 1 -- 172.21.15.47:0/1591241212 >> 172.21.15.47:0/1591241212 conn(0x7f17780d1ba0 msgr2=0x7f17780d1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:22.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.401+0000 7f177fa00640 1 -- 172.21.15.47:0/1591241212 shutdown_connections 2024-02-19T15:30:22.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.401+0000 7f177fa00640 1 -- 172.21.15.47:0/1591241212 wait complete. 2024-02-19T15:30:22.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.401+0000 7f177fa00640 1 Processor -- start 2024-02-19T15:30:22.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.401+0000 7f177fa00640 1 -- start start 2024-02-19T15:30:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.401+0000 7f177fa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17780cd180 0x7f177815c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.401+0000 7f177fa00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17780d0540 0x7f177806fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.401+0000 7f177fa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f177815c0d0 0x7f1778155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.401+0000 7f177fa00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177815da50 con 0x7f17780cd180 2024-02-19T15:30:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.401+0000 7f177fa00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177815ce40 con 0x7f177815c0d0 2024-02-19T15:30:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.401+0000 7f177fa00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f177815d120 con 0x7f17780d0540 2024-02-19T15:30:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.405+0000 7f177d000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17780cd180 0x7f177815c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.405+0000 7f177d000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17780cd180 0x7f177815c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60678/0 (socket says 172.21.15.47:60678) 2024-02-19T15:30:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.405+0000 7f177d000640 1 -- 172.21.15.47:0/2094505631 learned_addr learned my addr 172.21.15.47:0/2094505631 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.405+0000 7f1777e00640 1 --2- 172.21.15.47:0/2094505631 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17780d0540 0x7f177806fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.405+0000 7f177da00640 1 --2- 172.21.15.47:0/2094505631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f177815c0d0 0x7f1778155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.405+0000 7f177d000640 1 -- 172.21.15.47:0/2094505631 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17780d0540 msgr2=0x7f177806fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.405+0000 7f177d000640 1 --2- 172.21.15.47:0/2094505631 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17780d0540 0x7f177806fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.405+0000 7f177d000640 1 -- 172.21.15.47:0/2094505631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f177815c0d0 msgr2=0x7f1778155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.405+0000 7f177d000640 1 --2- 172.21.15.47:0/2094505631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f177815c0d0 0x7f1778155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.405+0000 7f177d000640 1 -- 172.21.15.47:0/2094505631 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c009420 con 0x7f17780cd180 2024-02-19T15:30:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.405+0000 7f177d000640 1 --2- 172.21.15.47:0/2094505631 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17780cd180 0x7f177815c5f0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f176c00e980 tx=0x7f176c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.405+0000 7f1774c00640 1 -- 172.21.15.47:0/2094505631 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176c00ca00 con 0x7f17780cd180 2024-02-19T15:30:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.405+0000 7f1774c00640 1 -- 172.21.15.47:0/2094505631 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f176c00cb60 con 0x7f17780cd180 2024-02-19T15:30:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.405+0000 7f1774c00640 1 -- 172.21.15.47:0/2094505631 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176c013470 con 0x7f17780cd180 2024-02-19T15:30:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.405+0000 7f177fa00640 1 -- 172.21.15.47:0/2094505631 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1770009a70 con 0x7f17780cd180 2024-02-19T15:30:22.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.405+0000 7f177fa00640 1 -- 172.21.15.47:0/2094505631 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f177815d6e0 con 0x7f17780cd180 2024-02-19T15:30:22.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.405+0000 7f177fa00640 1 -- 172.21.15.47:0/2094505631 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1778068500 con 0x7f17780cd180 2024-02-19T15:30:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.405+0000 7f1774c00640 1 -- 172.21.15.47:0/2094505631 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f176c0135d0 con 0x7f17780cd180 2024-02-19T15:30:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.409+0000 7f1774c00640 1 --2- 172.21.15.47:0/2094505631 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1758066e40 0x7f1758069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.409+0000 7f1774c00640 1 -- 172.21.15.47:0/2094505631 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f176c084cd0 con 0x7f17780cd180 2024-02-19T15:30:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.409+0000 7f1777e00640 1 --2- 172.21.15.47:0/2094505631 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1758066e40 0x7f1758069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.409+0000 7f1777e00640 1 --2- 172.21.15.47:0/2094505631 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1758066e40 0x7f1758069300 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f17780c8a00 tx=0x7f1764007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:22.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.413+0000 7f1774c00640 1 -- 172.21.15.47:0/2094505631 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f176c051b20 con 0x7f17780cd180 2024-02-19T15:30:22.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.605+0000 7f177fa00640 1 -- 172.21.15.47:0/2094505631 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1778075360 con 0x7f17780cd180 2024-02-19T15:30:22.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.605+0000 7f1774c00640 1 -- 172.21.15.47:0/2094505631 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f176c021d80 con 0x7f17780cd180 2024-02-19T15:30:22.608 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:22.608 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:22.608 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:22.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.609+0000 7f177fa00640 1 -- 172.21.15.47:0/2094505631 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1758066e40 msgr2=0x7f1758069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:22.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.609+0000 7f177fa00640 1 --2- 172.21.15.47:0/2094505631 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1758066e40 0x7f1758069300 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f17780c8a00 tx=0x7f1764007e90 comp rx=0 tx=0).stop 2024-02-19T15:30:22.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.609+0000 7f177fa00640 1 -- 172.21.15.47:0/2094505631 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17780cd180 msgr2=0x7f177815c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:22.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.609+0000 7f177fa00640 1 --2- 172.21.15.47:0/2094505631 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17780cd180 0x7f177815c5f0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f176c00e980 tx=0x7f176c00ee40 comp rx=0 tx=0).stop 2024-02-19T15:30:22.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.609+0000 7f177fa00640 1 -- 172.21.15.47:0/2094505631 shutdown_connections 2024-02-19T15:30:22.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.609+0000 7f177fa00640 1 --2- 172.21.15.47:0/2094505631 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1758066e40 0x7f1758069300 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:22.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.609+0000 7f177fa00640 1 --2- 172.21.15.47:0/2094505631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f177815c0d0 0x7f1778155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:22.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.609+0000 7f177fa00640 1 --2- 172.21.15.47:0/2094505631 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17780d0540 0x7f177806fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:22.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.609+0000 7f177fa00640 1 --2- 172.21.15.47:0/2094505631 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17780cd180 0x7f177815c5f0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:22.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.609+0000 7f177fa00640 1 -- 172.21.15.47:0/2094505631 >> 172.21.15.47:0/2094505631 conn(0x7f17780d1ba0 msgr2=0x7f17780cec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:22.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.609+0000 7f177fa00640 1 -- 172.21.15.47:0/2094505631 shutdown_connections 2024-02-19T15:30:22.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:22.609+0000 7f177fa00640 1 -- 172.21.15.47:0/2094505631 wait complete. 2024-02-19T15:30:22.830 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:22.830 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:23.831 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:23.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.973+0000 7f8c5ca00640 1 Processor -- start 2024-02-19T15:30:23.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.973+0000 7f8c5ca00640 1 -- start start 2024-02-19T15:30:23.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.973+0000 7f8c5ca00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c581456a0 0x7f8c580cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:23.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.973+0000 7f8c5ca00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8c580cf4a0 0x7f8c5806aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:23.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.973+0000 7f8c5ca00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c5806b850 con 0x7f8c5806b3f0 2024-02-19T15:30:23.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.973+0000 7f8c5ca00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c580d03c0 con 0x7f8c580cf4a0 2024-02-19T15:30:23.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.973+0000 7f8c5ca00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c580cf900 con 0x7f8c581456a0 2024-02-19T15:30:23.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.973+0000 7f8c55400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8c580cf4a0 0x7f8c5806aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:23.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.973+0000 7f8c55e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c581456a0 0x7f8c580cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:23.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.973+0000 7f8c55400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8c580cf4a0 0x7f8c5806aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:50546/0 (socket says 172.21.15.47:50546) 2024-02-19T15:30:23.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.973+0000 7f8c56800640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8c5806b3f0 0x7f8c5815f830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:35786/0 (socket says 172.21.15.47:35786) 2024-02-19T15:30:23.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.973+0000 7f8c55400640 1 -- 172.21.15.47:0/3292711318 learned_addr learned my addr 172.21.15.47:0/3292711318 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:23.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.973+0000 7f8c55400640 1 -- 172.21.15.47:0/3292711318 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c581456a0 msgr2=0x7f8c580cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:23.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.973+0000 7f8c55400640 1 --2- 172.21.15.47:0/3292711318 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c581456a0 0x7f8c580cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:23.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.973+0000 7f8c55400640 1 -- 172.21.15.47:0/3292711318 >> v1:172.21.15.116:6789/0 conn(0x7f8c5806b3f0 legacy=0x7f8c5815f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:23.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.973+0000 7f8c55400640 1 -- 172.21.15.47:0/3292711318 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c58145a80 con 0x7f8c580cf4a0 2024-02-19T15:30:23.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.973+0000 7f8c55e00640 1 --2- 172.21.15.47:0/3292711318 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c581456a0 0x7f8c580cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:30:23.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.977+0000 7f8c55400640 1 --2- 172.21.15.47:0/3292711318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c580cf4a0 0x7f8c5806aeb0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f8c440098a0 tx=0x7f8c4401c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90c53a5e45e39adc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:23.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.977+0000 7f8c54a00640 1 -- 172.21.15.47:0/3292711318 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c440ab600 con 0x7f8c580cf4a0 2024-02-19T15:30:23.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.977+0000 7f8c54a00640 1 -- 172.21.15.47:0/3292711318 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c440a7050 con 0x7f8c580cf4a0 2024-02-19T15:30:23.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.977+0000 7f8c54a00640 1 -- 172.21.15.47:0/3292711318 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c440ab980 con 0x7f8c580cf4a0 2024-02-19T15:30:23.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.977+0000 7f8c5ca00640 1 -- 172.21.15.47:0/3292711318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c580cf4a0 msgr2=0x7f8c5806aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:23.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.977+0000 7f8c5ca00640 1 --2- 172.21.15.47:0/3292711318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c580cf4a0 0x7f8c5806aeb0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f8c440098a0 tx=0x7f8c4401c540 comp rx=0 tx=0).stop 2024-02-19T15:30:23.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.977+0000 7f8c5ca00640 1 -- 172.21.15.47:0/3292711318 shutdown_connections 2024-02-19T15:30:23.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.977+0000 7f8c5ca00640 1 --2- 172.21.15.47:0/3292711318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c580cf4a0 0x7f8c5806aeb0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:23.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.977+0000 7f8c5ca00640 1 --2- 172.21.15.47:0/3292711318 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c581456a0 0x7f8c580cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:23.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.977+0000 7f8c5ca00640 1 -- 172.21.15.47:0/3292711318 >> 172.21.15.47:0/3292711318 conn(0x7f8c580d0ad0 msgr2=0x7f8c580d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:23.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.977+0000 7f8c5ca00640 1 -- 172.21.15.47:0/3292711318 shutdown_connections 2024-02-19T15:30:23.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.977+0000 7f8c5ca00640 1 -- 172.21.15.47:0/3292711318 wait complete. 2024-02-19T15:30:23.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.977+0000 7f8c5ca00640 1 Processor -- start 2024-02-19T15:30:23.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c5ca00640 1 -- start start 2024-02-19T15:30:23.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c5ca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c5806b3f0 0x7f8c58140b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:23.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c5ca00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c580cf4a0 0x7f8c5815abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:23.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c5ca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c581456a0 0x7f8c5815da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:23.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c5ca00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c581410c0 con 0x7f8c5806b3f0 2024-02-19T15:30:23.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c5ca00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c580b3ef0 con 0x7f8c581456a0 2024-02-19T15:30:23.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c5ca00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c58140130 con 0x7f8c580cf4a0 2024-02-19T15:30:23.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c55e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c5806b3f0 0x7f8c58140b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:23.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c55e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c5806b3f0 0x7f8c58140b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52282/0 (socket says 172.21.15.47:52282) 2024-02-19T15:30:23.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c55e00640 1 -- 172.21.15.47:0/2669187661 learned_addr learned my addr 172.21.15.47:0/2669187661 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:23.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c56800640 1 --2- 172.21.15.47:0/2669187661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c581456a0 0x7f8c5815da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:23.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c55400640 1 --2- 172.21.15.47:0/2669187661 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c580cf4a0 0x7f8c5815abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:23.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c55e00640 1 -- 172.21.15.47:0/2669187661 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c580cf4a0 msgr2=0x7f8c5815abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:23.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c55e00640 1 --2- 172.21.15.47:0/2669187661 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c580cf4a0 0x7f8c5815abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:23.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c55e00640 1 -- 172.21.15.47:0/2669187661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c581456a0 msgr2=0x7f8c5815da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:23.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c55e00640 1 --2- 172.21.15.47:0/2669187661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c581456a0 0x7f8c5815da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:23.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c55e00640 1 -- 172.21.15.47:0/2669187661 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c48009420 con 0x7f8c5806b3f0 2024-02-19T15:30:23.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c56800640 1 --2- 172.21.15.47:0/2669187661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c581456a0 0x7f8c5815da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:30:23.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c55e00640 1 --2- 172.21.15.47:0/2669187661 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c5806b3f0 0x7f8c58140b80 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f8c4800e980 tx=0x7f8c4800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:23.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c4e000640 1 -- 172.21.15.47:0/2669187661 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4800ca00 con 0x7f8c5806b3f0 2024-02-19T15:30:23.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c4e000640 1 -- 172.21.15.47:0/2669187661 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c4800cb60 con 0x7f8c5806b3f0 2024-02-19T15:30:23.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c4e000640 1 -- 172.21.15.47:0/2669187661 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c48013470 con 0x7f8c5806b3f0 2024-02-19T15:30:23.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c5ca00640 1 -- 172.21.15.47:0/2669187661 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c440094d0 con 0x7f8c5806b3f0 2024-02-19T15:30:23.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c5ca00640 1 -- 172.21.15.47:0/2669187661 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c580b4420 con 0x7f8c5806b3f0 2024-02-19T15:30:23.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.981+0000 7f8c5ca00640 1 -- 172.21.15.47:0/2669187661 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c58075360 con 0x7f8c5806b3f0 2024-02-19T15:30:23.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.985+0000 7f8c4e000640 1 -- 172.21.15.47:0/2669187661 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8c480135d0 con 0x7f8c5806b3f0 2024-02-19T15:30:23.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.985+0000 7f8c4e000640 1 --2- 172.21.15.47:0/2669187661 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8c38066e40 0x7f8c38069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:23.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.985+0000 7f8c4e000640 1 -- 172.21.15.47:0/2669187661 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8c48084c60 con 0x7f8c5806b3f0 2024-02-19T15:30:23.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.985+0000 7f8c55400640 1 --2- 172.21.15.47:0/2669187661 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8c38066e40 0x7f8c38069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:23.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.985+0000 7f8c55400640 1 --2- 172.21.15.47:0/2669187661 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8c38066e40 0x7f8c38069300 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f8c58067cb0 tx=0x7f8c44008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:23.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:23.989+0000 7f8c4e000640 1 -- 172.21.15.47:0/2669187661 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8c48051ac0 con 0x7f8c5806b3f0 2024-02-19T15:30:24.207 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:24.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:24.301+0000 7f8c5ca00640 1 -- 172.21.15.47:0/2669187661 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c58072b90 con 0x7f8c5806b3f0 2024-02-19T15:30:24.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:24.301+0000 7f8c4e000640 1 -- 172.21.15.47:0/2669187661 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f8c48021d80 con 0x7f8c5806b3f0 2024-02-19T15:30:24.300 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:24.301 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:24.301 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:24.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:24.301+0000 7f8c5ca00640 1 -- 172.21.15.47:0/2669187661 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8c38066e40 msgr2=0x7f8c38069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:24.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:24.301+0000 7f8c5ca00640 1 --2- 172.21.15.47:0/2669187661 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8c38066e40 0x7f8c38069300 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f8c58067cb0 tx=0x7f8c44008040 comp rx=0 tx=0).stop 2024-02-19T15:30:24.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:24.301+0000 7f8c5ca00640 1 -- 172.21.15.47:0/2669187661 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c5806b3f0 msgr2=0x7f8c58140b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:24.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:24.301+0000 7f8c5ca00640 1 --2- 172.21.15.47:0/2669187661 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c5806b3f0 0x7f8c58140b80 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f8c4800e980 tx=0x7f8c4800ee40 comp rx=0 tx=0).stop 2024-02-19T15:30:24.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:24.301+0000 7f8c5ca00640 1 -- 172.21.15.47:0/2669187661 shutdown_connections 2024-02-19T15:30:24.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:24.301+0000 7f8c5ca00640 1 --2- 172.21.15.47:0/2669187661 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8c38066e40 0x7f8c38069300 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:24.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:24.301+0000 7f8c5ca00640 1 --2- 172.21.15.47:0/2669187661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c581456a0 0x7f8c5815da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:24.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:24.301+0000 7f8c5ca00640 1 --2- 172.21.15.47:0/2669187661 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c580cf4a0 0x7f8c5815abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:24.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:24.301+0000 7f8c5ca00640 1 --2- 172.21.15.47:0/2669187661 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c5806b3f0 0x7f8c58140b80 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:24.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:24.301+0000 7f8c5ca00640 1 -- 172.21.15.47:0/2669187661 >> 172.21.15.47:0/2669187661 conn(0x7f8c580d0ad0 msgr2=0x7f8c580ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:24.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:24.301+0000 7f8c5ca00640 1 -- 172.21.15.47:0/2669187661 shutdown_connections 2024-02-19T15:30:24.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:24.301+0000 7f8c5ca00640 1 -- 172.21.15.47:0/2669187661 wait complete. 2024-02-19T15:30:24.523 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:24.523 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:25.525 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:25.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.665+0000 7f636ae00640 1 Processor -- start 2024-02-19T15:30:25.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.665+0000 7f636ae00640 1 -- start start 2024-02-19T15:30:25.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.665+0000 7f636ae00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f636406d650 0x7f63640d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:25.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.665+0000 7f636ae00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f63640cded0 0x7f63640ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:25.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.665+0000 7f636ae00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63640cce30 con 0x7f63640ceb40 2024-02-19T15:30:25.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.665+0000 7f636ae00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63640cefd0 con 0x7f63640cded0 2024-02-19T15:30:25.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.665+0000 7f636ae00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63640cd110 con 0x7f636406d650 2024-02-19T15:30:25.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.665+0000 7f635be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f63640cded0 0x7f63640ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:25.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.665+0000 7f635be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f63640cded0 0x7f63640ce600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52294/0 (socket says 172.21.15.47:52294) 2024-02-19T15:30:25.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.665+0000 7f635be00640 1 -- 172.21.15.47:0/4087047505 learned_addr learned my addr 172.21.15.47:0/4087047505 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:25.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.665+0000 7f6363e00640 1 --2- 172.21.15.47:0/4087047505 >> v2:172.21.15.116:3300/0 conn(0x7f636406d650 0x7f63640d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:25.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.665+0000 7f635be00640 1 -- 172.21.15.47:0/4087047505 >> v2:172.21.15.116:3300/0 conn(0x7f636406d650 msgr2=0x7f63640d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:25.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.665+0000 7f635be00640 1 --2- 172.21.15.47:0/4087047505 >> v2:172.21.15.116:3300/0 conn(0x7f636406d650 0x7f63640d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:25.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.665+0000 7f635be00640 1 -- 172.21.15.47:0/4087047505 >> v1:172.21.15.116:6789/0 conn(0x7f63640ceb40 legacy=0x7f6364161d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:25.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.665+0000 7f635be00640 1 -- 172.21.15.47:0/4087047505 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63640cd3f0 con 0x7f63640cded0 2024-02-19T15:30:25.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.665+0000 7f6363e00640 1 --2- 172.21.15.47:0/4087047505 >> v2:172.21.15.116:3300/0 conn(0x7f636406d650 0x7f63640d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:30:25.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.669+0000 7f635be00640 1 --2- 172.21.15.47:0/4087047505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63640cded0 0x7f63640ce600 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f6364072160 tx=0x7f634c01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1833de878a0a1fb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:25.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.669+0000 7f6363400640 1 -- 172.21.15.47:0/4087047505 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c0ab600 con 0x7f63640cded0 2024-02-19T15:30:25.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.669+0000 7f6363400640 1 -- 172.21.15.47:0/4087047505 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f634c0a7050 con 0x7f63640cded0 2024-02-19T15:30:25.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.669+0000 7f6363400640 1 -- 172.21.15.47:0/4087047505 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c0ab980 con 0x7f63640cded0 2024-02-19T15:30:25.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.669+0000 7f636ae00640 1 -- 172.21.15.47:0/4087047505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63640cded0 msgr2=0x7f63640ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:25.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.669+0000 7f636ae00640 1 --2- 172.21.15.47:0/4087047505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63640cded0 0x7f63640ce600 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f6364072160 tx=0x7f634c01c420 comp rx=0 tx=0).stop 2024-02-19T15:30:25.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.669+0000 7f636ae00640 1 -- 172.21.15.47:0/4087047505 shutdown_connections 2024-02-19T15:30:25.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.669+0000 7f636ae00640 1 --2- 172.21.15.47:0/4087047505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63640cded0 0x7f63640ce600 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:25.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.669+0000 7f636ae00640 1 --2- 172.21.15.47:0/4087047505 >> v2:172.21.15.116:3300/0 conn(0x7f636406d650 0x7f63640d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:25.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.669+0000 7f636ae00640 1 -- 172.21.15.47:0/4087047505 >> 172.21.15.47:0/4087047505 conn(0x7f63640d1bf0 msgr2=0x7f63640d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:25.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.669+0000 7f636ae00640 1 -- 172.21.15.47:0/4087047505 shutdown_connections 2024-02-19T15:30:25.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.669+0000 7f636ae00640 1 -- 172.21.15.47:0/4087047505 wait complete. 2024-02-19T15:30:25.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f636ae00640 1 Processor -- start 2024-02-19T15:30:25.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f636ae00640 1 -- start start 2024-02-19T15:30:25.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f636ae00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f636406d650 0x7f63640c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:25.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f636ae00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63640cded0 0x7f63640c5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:25.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f636ae00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63640ceb40 0x7f6364160100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:25.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f636ae00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63640c7f40 con 0x7f63640cded0 2024-02-19T15:30:25.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f636ae00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636415aef0 con 0x7f636406d650 2024-02-19T15:30:25.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f636ae00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f636406f680 con 0x7f63640ceb40 2024-02-19T15:30:25.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f635be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63640cded0 0x7f63640c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:25.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f635be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63640cded0 0x7f63640c5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52308/0 (socket says 172.21.15.47:52308) 2024-02-19T15:30:25.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f635be00640 1 -- 172.21.15.47:0/3811070108 learned_addr learned my addr 172.21.15.47:0/3811070108 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:25.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f6363e00640 1 --2- 172.21.15.47:0/3811070108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f636406d650 0x7f63640c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:25.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f6368e00640 1 --2- 172.21.15.47:0/3811070108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63640ceb40 0x7f6364160100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:25.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f635be00640 1 -- 172.21.15.47:0/3811070108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63640ceb40 msgr2=0x7f6364160100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:25.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f635be00640 1 --2- 172.21.15.47:0/3811070108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63640ceb40 0x7f6364160100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:25.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f635be00640 1 -- 172.21.15.47:0/3811070108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f636406d650 msgr2=0x7f63640c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:25.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f635be00640 1 --2- 172.21.15.47:0/3811070108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f636406d650 0x7f63640c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:25.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f635be00640 1 -- 172.21.15.47:0/3811070108 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634c0094d0 con 0x7f63640cded0 2024-02-19T15:30:25.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f6363e00640 1 --2- 172.21.15.47:0/3811070108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f636406d650 0x7f63640c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:30:25.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f6368e00640 1 --2- 172.21.15.47:0/3811070108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63640ceb40 0x7f6364160100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:30:25.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f635be00640 1 --2- 172.21.15.47:0/3811070108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63640cded0 0x7f63640c5720 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f634c01a050 tx=0x7f634c0191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:25.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f6360c00640 1 -- 172.21.15.47:0/3811070108 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c0b4620 con 0x7f63640cded0 2024-02-19T15:30:25.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f6360c00640 1 -- 172.21.15.47:0/3811070108 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f634c0b47a0 con 0x7f63640cded0 2024-02-19T15:30:25.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f6360c00640 1 -- 172.21.15.47:0/3811070108 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c0b4b50 con 0x7f63640cded0 2024-02-19T15:30:25.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f636ae00640 1 -- 172.21.15.47:0/3811070108 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63640c5c60 con 0x7f63640cded0 2024-02-19T15:30:25.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.673+0000 7f636ae00640 1 -- 172.21.15.47:0/3811070108 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f636406f8c0 con 0x7f63640cded0 2024-02-19T15:30:25.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.677+0000 7f636ae00640 1 -- 172.21.15.47:0/3811070108 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6364075360 con 0x7f63640cded0 2024-02-19T15:30:25.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.677+0000 7f6360c00640 1 -- 172.21.15.47:0/3811070108 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f634c0b2070 con 0x7f63640cded0 2024-02-19T15:30:25.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.677+0000 7f6360c00640 1 --2- 172.21.15.47:0/3811070108 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6340066e40 0x7f6340069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:25.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.677+0000 7f6360c00640 1 -- 172.21.15.47:0/3811070108 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f634c11e420 con 0x7f63640cded0 2024-02-19T15:30:25.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.677+0000 7f6363e00640 1 --2- 172.21.15.47:0/3811070108 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6340066e40 0x7f6340069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:25.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.677+0000 7f6363e00640 1 --2- 172.21.15.47:0/3811070108 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6340066e40 0x7f6340069300 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f636415dcf0 tx=0x7f6354008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:25.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:25.681+0000 7f6360c00640 1 -- 172.21.15.47:0/3811070108 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f634c0eb1c0 con 0x7f63640cded0 2024-02-19T15:30:26.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:26.037+0000 7f636ae00640 1 -- 172.21.15.47:0/3811070108 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63640cf580 con 0x7f63640cded0 2024-02-19T15:30:26.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:26.037+0000 7f6360c00640 1 -- 172.21.15.47:0/3811070108 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f634c0b9d80 con 0x7f63640cded0 2024-02-19T15:30:26.036 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:26.036 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:26.036 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:26.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:26.037+0000 7f636ae00640 1 -- 172.21.15.47:0/3811070108 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6340066e40 msgr2=0x7f6340069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:26.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:26.037+0000 7f636ae00640 1 --2- 172.21.15.47:0/3811070108 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6340066e40 0x7f6340069300 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f636415dcf0 tx=0x7f6354008040 comp rx=0 tx=0).stop 2024-02-19T15:30:26.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:26.037+0000 7f636ae00640 1 -- 172.21.15.47:0/3811070108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63640cded0 msgr2=0x7f63640c5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:26.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:26.037+0000 7f636ae00640 1 --2- 172.21.15.47:0/3811070108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63640cded0 0x7f63640c5720 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f634c01a050 tx=0x7f634c0191c0 comp rx=0 tx=0).stop 2024-02-19T15:30:26.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:26.037+0000 7f636ae00640 1 -- 172.21.15.47:0/3811070108 shutdown_connections 2024-02-19T15:30:26.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:26.037+0000 7f636ae00640 1 --2- 172.21.15.47:0/3811070108 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6340066e40 0x7f6340069300 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:26.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:26.037+0000 7f636ae00640 1 --2- 172.21.15.47:0/3811070108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63640ceb40 0x7f6364160100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:26.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:26.037+0000 7f636ae00640 1 --2- 172.21.15.47:0/3811070108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63640cded0 0x7f63640c5720 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:26.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:26.037+0000 7f636ae00640 1 --2- 172.21.15.47:0/3811070108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f636406d650 0x7f63640c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:26.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:26.037+0000 7f636ae00640 1 -- 172.21.15.47:0/3811070108 >> 172.21.15.47:0/3811070108 conn(0x7f63640d1bf0 msgr2=0x7f636415e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:26.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:26.037+0000 7f636ae00640 1 -- 172.21.15.47:0/3811070108 shutdown_connections 2024-02-19T15:30:26.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:26.037+0000 7f636ae00640 1 -- 172.21.15.47:0/3811070108 wait complete. 2024-02-19T15:30:26.209 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:26.259 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:26.259 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:27.260 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:27.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.401+0000 7f010e800640 1 Processor -- start 2024-02-19T15:30:27.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.401+0000 7f010e800640 1 -- start start 2024-02-19T15:30:27.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.401+0000 7f010e800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0108145590 0x7f01080727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:27.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.401+0000 7f010e800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f010815d280 con 0x7f01080cea80 2024-02-19T15:30:27.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.401+0000 7f010e800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0108070ff0 con 0x7f010815c390 2024-02-19T15:30:27.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.401+0000 7f010e800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01080709e0 con 0x7f0108145590 2024-02-19T15:30:27.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.401+0000 7f0107400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0108145590 0x7f01080727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:27.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.401+0000 7f00ffe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f01080cea80 0x7f010806c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41840/0 (socket says 172.21.15.47:41840) 2024-02-19T15:30:27.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.401+0000 7f00ffe00640 1 -- 172.21.15.47:0/68093226 learned_addr learned my addr 172.21.15.47:0/68093226 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:27.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.401+0000 7f0107400640 1 -- 172.21.15.47:0/68093226 >> v1:172.21.15.116:6789/0 conn(0x7f010815c390 legacy=0x7f010815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:27.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.401+0000 7f0107400640 1 -- 172.21.15.47:0/68093226 >> v1:172.21.15.47:6789/0 conn(0x7f01080cea80 legacy=0x7f010806c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:27.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.401+0000 7f0107400640 1 -- 172.21.15.47:0/68093226 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01080cf020 con 0x7f0108145590 2024-02-19T15:30:27.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.405+0000 7f0107400640 1 --2- 172.21.15.47:0/68093226 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0108145590 0x7f01080727b0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f00f80097c0 tx=0x7f00f801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd1ec3ba6910d92f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:27.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.405+0000 7f0106a00640 1 -- 172.21.15.47:0/68093226 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f80ac410 con 0x7f0108145590 2024-02-19T15:30:27.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.405+0000 7f0106a00640 1 -- 172.21.15.47:0/68093226 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f80ac570 con 0x7f0108145590 2024-02-19T15:30:27.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.405+0000 7f0106a00640 1 -- 172.21.15.47:0/68093226 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f80ac900 con 0x7f0108145590 2024-02-19T15:30:27.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.405+0000 7f010e800640 1 -- 172.21.15.47:0/68093226 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0108145590 msgr2=0x7f01080727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:27.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.405+0000 7f010e800640 1 --2- 172.21.15.47:0/68093226 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0108145590 0x7f01080727b0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f00f80097c0 tx=0x7f00f801c4a0 comp rx=0 tx=0).stop 2024-02-19T15:30:27.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.405+0000 7f010e800640 1 -- 172.21.15.47:0/68093226 shutdown_connections 2024-02-19T15:30:27.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.405+0000 7f010e800640 1 --2- 172.21.15.47:0/68093226 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0108145590 0x7f01080727b0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:27.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.405+0000 7f010e800640 1 -- 172.21.15.47:0/68093226 >> 172.21.15.47:0/68093226 conn(0x7f01080b16a0 msgr2=0x7f01080b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:27.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.405+0000 7f010e800640 1 -- 172.21.15.47:0/68093226 shutdown_connections 2024-02-19T15:30:27.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.405+0000 7f010e800640 1 -- 172.21.15.47:0/68093226 wait complete. 2024-02-19T15:30:27.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f010e800640 1 Processor -- start 2024-02-19T15:30:27.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f010e800640 1 -- start start 2024-02-19T15:30:27.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f010e800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01080cea80 0x7f010815a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:27.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f010e800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0108145590 0x7f010813efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:27.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f010e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f010815c390 0x7f010813f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:27.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f010e800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01080c58f0 con 0x7f0108145590 2024-02-19T15:30:27.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f010e800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0108141d80 con 0x7f010815c390 2024-02-19T15:30:27.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f010e800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0108142060 con 0x7f01080cea80 2024-02-19T15:30:27.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f00ffe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0108145590 0x7f010813efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:27.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f00ffe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0108145590 0x7f010813efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52330/0 (socket says 172.21.15.47:52330) 2024-02-19T15:30:27.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f00ffe00640 1 -- 172.21.15.47:0/3663390363 learned_addr learned my addr 172.21.15.47:0/3663390363 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:27.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f0107400640 1 --2- 172.21.15.47:0/3663390363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01080cea80 0x7f010815a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:27.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f0107e00640 1 --2- 172.21.15.47:0/3663390363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f010815c390 0x7f010813f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:27.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f00ffe00640 1 -- 172.21.15.47:0/3663390363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01080cea80 msgr2=0x7f010815a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:27.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f00ffe00640 1 --2- 172.21.15.47:0/3663390363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01080cea80 0x7f010815a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:27.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f00ffe00640 1 -- 172.21.15.47:0/3663390363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f010815c390 msgr2=0x7f010813f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:27.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f00ffe00640 1 --2- 172.21.15.47:0/3663390363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f010815c390 0x7f010813f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:27.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f00ffe00640 1 -- 172.21.15.47:0/3663390363 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f8009420 con 0x7f0108145590 2024-02-19T15:30:27.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f00ffe00640 1 --2- 172.21.15.47:0/3663390363 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0108145590 0x7f010813efc0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f00f000c500 tx=0x7f00f000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:27.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f00ff400640 1 -- 172.21.15.47:0/3663390363 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f0005cf0 con 0x7f0108145590 2024-02-19T15:30:27.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f00ff400640 1 -- 172.21.15.47:0/3663390363 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f0005e50 con 0x7f0108145590 2024-02-19T15:30:27.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f00ff400640 1 -- 172.21.15.47:0/3663390363 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f00072f0 con 0x7f0108145590 2024-02-19T15:30:27.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f010e800640 1 -- 172.21.15.47:0/3663390363 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0108142340 con 0x7f0108145590 2024-02-19T15:30:27.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f010e800640 1 -- 172.21.15.47:0/3663390363 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f010815ac50 con 0x7f0108145590 2024-02-19T15:30:27.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.409+0000 7f010e800640 1 -- 172.21.15.47:0/3663390363 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01080c5750 con 0x7f0108145590 2024-02-19T15:30:27.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.413+0000 7f00ff400640 1 -- 172.21.15.47:0/3663390363 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f00f0007450 con 0x7f0108145590 2024-02-19T15:30:27.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.413+0000 7f00ff400640 1 --2- 172.21.15.47:0/3663390363 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f00e4066e40 0x7f00e4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:27.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.413+0000 7f00ff400640 1 -- 172.21.15.47:0/3663390363 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f00f0083f30 con 0x7f0108145590 2024-02-19T15:30:27.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.413+0000 7f0107400640 1 --2- 172.21.15.47:0/3663390363 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f00e4066e40 0x7f00e4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:27.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.413+0000 7f0107400640 1 --2- 172.21.15.47:0/3663390363 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f00e4066e40 0x7f00e4069300 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f00f80ae000 tx=0x7f00f8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:27.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.417+0000 7f00ff400640 1 -- 172.21.15.47:0/3663390363 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f01080c5750 con 0x7f0108145590 2024-02-19T15:30:27.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.609+0000 7f010e800640 1 -- 172.21.15.47:0/3663390363 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01080c5750 con 0x7f0108145590 2024-02-19T15:30:27.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.613+0000 7f00ff400640 1 -- 172.21.15.47:0/3663390363 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f00f0017d80 con 0x7f0108145590 2024-02-19T15:30:27.612 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:27.612 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:27.612 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:27.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.613+0000 7f010e800640 1 -- 172.21.15.47:0/3663390363 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f00e4066e40 msgr2=0x7f00e4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:27.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.613+0000 7f010e800640 1 --2- 172.21.15.47:0/3663390363 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f00e4066e40 0x7f00e4069300 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f00f80ae000 tx=0x7f00f8008040 comp rx=0 tx=0).stop 2024-02-19T15:30:27.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.613+0000 7f010e800640 1 -- 172.21.15.47:0/3663390363 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0108145590 msgr2=0x7f010813efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:27.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.613+0000 7f010e800640 1 --2- 172.21.15.47:0/3663390363 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0108145590 0x7f010813efc0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f00f000c500 tx=0x7f00f000c9c0 comp rx=0 tx=0).stop 2024-02-19T15:30:27.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.613+0000 7f010e800640 1 -- 172.21.15.47:0/3663390363 shutdown_connections 2024-02-19T15:30:27.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.613+0000 7f010e800640 1 --2- 172.21.15.47:0/3663390363 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f00e4066e40 0x7f00e4069300 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:27.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.613+0000 7f010e800640 1 --2- 172.21.15.47:0/3663390363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f010815c390 0x7f010813f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:27.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.613+0000 7f010e800640 1 --2- 172.21.15.47:0/3663390363 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0108145590 0x7f010813efc0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:27.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.613+0000 7f010e800640 1 --2- 172.21.15.47:0/3663390363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01080cea80 0x7f010815a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:27.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.613+0000 7f010e800640 1 -- 172.21.15.47:0/3663390363 >> 172.21.15.47:0/3663390363 conn(0x7f01080b16a0 msgr2=0x7f01080ad840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:27.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.613+0000 7f010e800640 1 -- 172.21.15.47:0/3663390363 shutdown_connections 2024-02-19T15:30:27.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:27.613+0000 7f010e800640 1 -- 172.21.15.47:0/3663390363 wait complete. 2024-02-19T15:30:27.835 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:27.835 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:28.211 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:28.836 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:28.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.977+0000 7f7a61600640 1 Processor -- start 2024-02-19T15:30:28.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.977+0000 7f7a61600640 1 -- start start 2024-02-19T15:30:28.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.977+0000 7f7a61600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a5c06b330 0x7f7a5c0b4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:28.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.977+0000 7f7a61600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5c06b7c0 con 0x7f7a5c06aeb0 2024-02-19T15:30:28.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.977+0000 7f7a61600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5c071a00 con 0x7f7a5c145740 2024-02-19T15:30:28.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.977+0000 7f7a61600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5c071ce0 con 0x7f7a5c06b330 2024-02-19T15:30:28.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.977+0000 7f7a53e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f7a5c06aeb0 0x7f7a5c0b3a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41844/0 (socket says 172.21.15.47:41844) 2024-02-19T15:30:28.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.977+0000 7f7a53e00640 1 -- 172.21.15.47:0/1311484352 learned_addr learned my addr 172.21.15.47:0/1311484352 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:28.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.977+0000 7f7a5b400640 1 --2- 172.21.15.47:0/1311484352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a5c06b330 0x7f7a5c0b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:28.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.977+0000 7f7a5a000640 1 -- 172.21.15.47:0/1311484352 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226653002 0 0) 0x7f7a5c06b7c0 con 0x7f7a5c06aeb0 2024-02-19T15:30:28.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.977+0000 7f7a5a000640 1 -- 172.21.15.47:0/1311484352 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a40003530 con 0x7f7a5c06aeb0 2024-02-19T15:30:28.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.977+0000 7f7a5b400640 1 -- 172.21.15.47:0/1311484352 >> v1:172.21.15.116:6789/0 conn(0x7f7a5c145740 legacy=0x7f7a5c0cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:28.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.977+0000 7f7a5a000640 1 -- 172.21.15.47:0/1311484352 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708454480 0 0) 0x7f7a5c071a00 con 0x7f7a5c145740 2024-02-19T15:30:28.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.977+0000 7f7a5b400640 1 -- 172.21.15.47:0/1311484352 >> v1:172.21.15.47:6789/0 conn(0x7f7a5c06aeb0 legacy=0x7f7a5c0b3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:28.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.977+0000 7f7a5b400640 1 -- 172.21.15.47:0/1311484352 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5c071fc0 con 0x7f7a5c06b330 2024-02-19T15:30:28.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.981+0000 7f7a5b400640 1 --2- 172.21.15.47:0/1311484352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a5c06b330 0x7f7a5c0b4130 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7a54009dc0 tx=0x7f7a5401c840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c310e9280a4f57b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:28.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.981+0000 7f7a5a000640 1 -- 172.21.15.47:0/1311484352 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a540a5900 con 0x7f7a5c06b330 2024-02-19T15:30:28.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.981+0000 7f7a5a000640 1 -- 172.21.15.47:0/1311484352 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a540a5a60 con 0x7f7a5c06b330 2024-02-19T15:30:28.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.981+0000 7f7a5a000640 1 -- 172.21.15.47:0/1311484352 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a540a5df0 con 0x7f7a5c06b330 2024-02-19T15:30:28.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.981+0000 7f7a61600640 1 -- 172.21.15.47:0/1311484352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a5c06b330 msgr2=0x7f7a5c0b4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:28.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.981+0000 7f7a61600640 1 --2- 172.21.15.47:0/1311484352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a5c06b330 0x7f7a5c0b4130 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7a54009dc0 tx=0x7f7a5401c840 comp rx=0 tx=0).stop 2024-02-19T15:30:28.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.981+0000 7f7a61600640 1 -- 172.21.15.47:0/1311484352 shutdown_connections 2024-02-19T15:30:28.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.981+0000 7f7a61600640 1 --2- 172.21.15.47:0/1311484352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a5c06b330 0x7f7a5c0b4130 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:28.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.981+0000 7f7a61600640 1 -- 172.21.15.47:0/1311484352 >> 172.21.15.47:0/1311484352 conn(0x7f7a5c0d0a50 msgr2=0x7f7a5c0d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:28.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.981+0000 7f7a61600640 1 -- 172.21.15.47:0/1311484352 shutdown_connections 2024-02-19T15:30:28.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.981+0000 7f7a61600640 1 -- 172.21.15.47:0/1311484352 wait complete. 2024-02-19T15:30:28.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a61600640 1 Processor -- start 2024-02-19T15:30:28.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a61600640 1 -- start start 2024-02-19T15:30:28.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a61600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a5c06aeb0 0x7f7a5c0ca000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:28.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a61600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a5c06b330 0x7f7a5c157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:28.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a61600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a5c145740 0x7f7a5c158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:28.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a61600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5c13b750 con 0x7f7a5c06b330 2024-02-19T15:30:28.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a61600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5c06ef60 con 0x7f7a5c06aeb0 2024-02-19T15:30:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a61600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5c06f240 con 0x7f7a5c145740 2024-02-19T15:30:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a53e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a5c06b330 0x7f7a5c157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a53e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a5c06b330 0x7f7a5c157fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52336/0 (socket says 172.21.15.47:52336) 2024-02-19T15:30:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a53e00640 1 -- 172.21.15.47:0/1124927487 learned_addr learned my addr 172.21.15.47:0/1124927487 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a5aa00640 1 --2- 172.21.15.47:0/1124927487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a5c06aeb0 0x7f7a5c0ca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a5b400640 1 --2- 172.21.15.47:0/1124927487 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a5c145740 0x7f7a5c158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a53e00640 1 -- 172.21.15.47:0/1124927487 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a5c145740 msgr2=0x7f7a5c158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a53e00640 1 --2- 172.21.15.47:0/1124927487 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a5c145740 0x7f7a5c158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a53e00640 1 -- 172.21.15.47:0/1124927487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a5c06aeb0 msgr2=0x7f7a5c0ca000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a53e00640 1 --2- 172.21.15.47:0/1124927487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a5c06aeb0 0x7f7a5c0ca000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a53e00640 1 -- 172.21.15.47:0/1124927487 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a54009a70 con 0x7f7a5c06b330 2024-02-19T15:30:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a53e00640 1 --2- 172.21.15.47:0/1124927487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a5c06b330 0x7f7a5c157fc0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f7a4400c450 tx=0x7f7a4400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:28.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a52a00640 1 -- 172.21.15.47:0/1124927487 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a44005cf0 con 0x7f7a5c06b330 2024-02-19T15:30:28.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a52a00640 1 -- 172.21.15.47:0/1124927487 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a44005e50 con 0x7f7a5c06b330 2024-02-19T15:30:28.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a61600640 1 -- 172.21.15.47:0/1124927487 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a5c06f520 con 0x7f7a5c06b330 2024-02-19T15:30:28.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a52a00640 1 -- 172.21.15.47:0/1124927487 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a44007290 con 0x7f7a5c06b330 2024-02-19T15:30:28.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a61600640 1 -- 172.21.15.47:0/1124927487 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a5c0ca710 con 0x7f7a5c06b330 2024-02-19T15:30:28.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.985+0000 7f7a61600640 1 -- 172.21.15.47:0/1124927487 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a5c06f760 con 0x7f7a5c06b330 2024-02-19T15:30:28.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.989+0000 7f7a52a00640 1 -- 172.21.15.47:0/1124927487 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7a440073f0 con 0x7f7a5c06b330 2024-02-19T15:30:28.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.989+0000 7f7a52a00640 1 --2- 172.21.15.47:0/1124927487 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a38066e40 0x7f7a38069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:28.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.989+0000 7f7a52a00640 1 -- 172.21.15.47:0/1124927487 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7a44083fe0 con 0x7f7a5c06b330 2024-02-19T15:30:28.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.989+0000 7f7a5aa00640 1 --2- 172.21.15.47:0/1124927487 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a38066e40 0x7f7a38069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:28.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.989+0000 7f7a5aa00640 1 --2- 172.21.15.47:0/1124927487 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a38066e40 0x7f7a38069300 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c13e030 tx=0x7f7a4c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:28.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:28.997+0000 7f7a52a00640 1 -- 172.21.15.47:0/1124927487 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7a5c06f760 con 0x7f7a5c06b330 2024-02-19T15:30:29.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:29.197+0000 7f7a61600640 1 -- 172.21.15.47:0/1124927487 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a5c13b890 con 0x7f7a5c06b330 2024-02-19T15:30:29.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:29.197+0000 7f7a52a00640 1 -- 172.21.15.47:0/1124927487 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f7a5c13b890 con 0x7f7a5c06b330 2024-02-19T15:30:29.197 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:29.197 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:29.198 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:29.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:29.197+0000 7f7a61600640 1 -- 172.21.15.47:0/1124927487 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a38066e40 msgr2=0x7f7a38069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:29.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:29.197+0000 7f7a61600640 1 --2- 172.21.15.47:0/1124927487 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a38066e40 0x7f7a38069300 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c13e030 tx=0x7f7a4c009040 comp rx=0 tx=0).stop 2024-02-19T15:30:29.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:29.197+0000 7f7a61600640 1 -- 172.21.15.47:0/1124927487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a5c06b330 msgr2=0x7f7a5c157fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:29.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:29.197+0000 7f7a61600640 1 --2- 172.21.15.47:0/1124927487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a5c06b330 0x7f7a5c157fc0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f7a4400c450 tx=0x7f7a4400c910 comp rx=0 tx=0).stop 2024-02-19T15:30:29.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:29.197+0000 7f7a61600640 1 -- 172.21.15.47:0/1124927487 shutdown_connections 2024-02-19T15:30:29.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:29.197+0000 7f7a61600640 1 --2- 172.21.15.47:0/1124927487 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7a38066e40 0x7f7a38069300 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:29.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:29.197+0000 7f7a61600640 1 --2- 172.21.15.47:0/1124927487 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a5c145740 0x7f7a5c158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:29.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:29.197+0000 7f7a61600640 1 --2- 172.21.15.47:0/1124927487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a5c06b330 0x7f7a5c157fc0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:29.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:29.197+0000 7f7a61600640 1 --2- 172.21.15.47:0/1124927487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a5c06aeb0 0x7f7a5c0ca000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:29.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:29.197+0000 7f7a61600640 1 -- 172.21.15.47:0/1124927487 >> 172.21.15.47:0/1124927487 conn(0x7f7a5c0d0a50 msgr2=0x7f7a5c0d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:29.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:29.201+0000 7f7a61600640 1 -- 172.21.15.47:0/1124927487 shutdown_connections 2024-02-19T15:30:29.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:29.201+0000 7f7a61600640 1 -- 172.21.15.47:0/1124927487 wait complete. 2024-02-19T15:30:29.421 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:29.421 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:30.213 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:30.422 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:30.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.565+0000 7f0ada000640 1 Processor -- start 2024-02-19T15:30:30.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.565+0000 7f0ada000640 1 -- start start 2024-02-19T15:30:30.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.565+0000 7f0ada000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0ad40acec0 0x7f0ad406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:30.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.565+0000 7f0ada000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0ad40cdf90 0x7f0ad40ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:30.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.565+0000 7f0ada000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad406e290 con 0x7f0ad406e470 2024-02-19T15:30:30.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.565+0000 7f0ada000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad409e010 con 0x7f0ad40cdf90 2024-02-19T15:30:30.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.565+0000 7f0ada000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad409dd30 con 0x7f0ad40acec0 2024-02-19T15:30:30.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.565+0000 7f0ad9600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0ad40cdf90 0x7f0ad40ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:30.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.565+0000 7f0ad9600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0ad40cdf90 0x7f0ad40ce410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52346/0 (socket says 172.21.15.47:52346) 2024-02-19T15:30:30.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.565+0000 7f0ad9600640 1 -- 172.21.15.47:0/1317476336 learned_addr learned my addr 172.21.15.47:0/1317476336 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:30.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.565+0000 7f0ad8c00640 1 --2- 172.21.15.47:0/1317476336 >> v2:172.21.15.116:3300/0 conn(0x7f0ad40acec0 0x7f0ad406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:30.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.565+0000 7f0ad9600640 1 -- 172.21.15.47:0/1317476336 >> v2:172.21.15.116:3300/0 conn(0x7f0ad40acec0 msgr2=0x7f0ad406dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:30.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.565+0000 7f0ad9600640 1 --2- 172.21.15.47:0/1317476336 >> v2:172.21.15.116:3300/0 conn(0x7f0ad40acec0 0x7f0ad406dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:30.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.565+0000 7f0ad9600640 1 -- 172.21.15.47:0/1317476336 >> v1:172.21.15.116:6789/0 conn(0x7f0ad406e470 legacy=0x7f0ad40ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:30.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.565+0000 7f0ad9600640 1 -- 172.21.15.47:0/1317476336 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad406cf90 con 0x7f0ad40cdf90 2024-02-19T15:30:30.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.569+0000 7f0ad9600640 1 --2- 172.21.15.47:0/1317476336 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad40cdf90 0x7f0ad40ce410 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f0ad40736c0 tx=0x7f0acc01c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a9965fecf94f04 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:30.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.569+0000 7f0ad3e00640 1 -- 172.21.15.47:0/1317476336 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc0a5780 con 0x7f0ad40cdf90 2024-02-19T15:30:30.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.569+0000 7f0ad3e00640 1 -- 172.21.15.47:0/1317476336 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0acc0a58e0 con 0x7f0ad40cdf90 2024-02-19T15:30:30.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.569+0000 7f0ad3e00640 1 -- 172.21.15.47:0/1317476336 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc0a5c70 con 0x7f0ad40cdf90 2024-02-19T15:30:30.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.569+0000 7f0ada000640 1 -- 172.21.15.47:0/1317476336 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad40cdf90 msgr2=0x7f0ad40ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:30.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.569+0000 7f0ada000640 1 --2- 172.21.15.47:0/1317476336 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad40cdf90 0x7f0ad40ce410 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f0ad40736c0 tx=0x7f0acc01c9c0 comp rx=0 tx=0).stop 2024-02-19T15:30:30.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.569+0000 7f0ada000640 1 -- 172.21.15.47:0/1317476336 shutdown_connections 2024-02-19T15:30:30.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.569+0000 7f0ada000640 1 --2- 172.21.15.47:0/1317476336 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad40cdf90 0x7f0ad40ce410 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:30.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.569+0000 7f0ada000640 1 --2- 172.21.15.47:0/1317476336 >> v2:172.21.15.116:3300/0 conn(0x7f0ad40acec0 0x7f0ad406dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:30.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.569+0000 7f0ada000640 1 -- 172.21.15.47:0/1317476336 >> 172.21.15.47:0/1317476336 conn(0x7f0ad40b1be0 msgr2=0x7f0ad40b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:30.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.569+0000 7f0ada000640 1 -- 172.21.15.47:0/1317476336 shutdown_connections 2024-02-19T15:30:30.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.569+0000 7f0ada000640 1 -- 172.21.15.47:0/1317476336 wait complete. 2024-02-19T15:30:30.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.569+0000 7f0ada000640 1 Processor -- start 2024-02-19T15:30:30.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.569+0000 7f0ada000640 1 -- start start 2024-02-19T15:30:30.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0ada000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ad406e470 0x7f0ad40c7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:30.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0ada000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad40acec0 0x7f0ad415d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:30.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0ada000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad40cdf90 0x7f0ad40c5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:30.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0ada000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad40c4e50 con 0x7f0ad40acec0 2024-02-19T15:30:30.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0ada000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad40c6b50 con 0x7f0ad40cdf90 2024-02-19T15:30:30.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0ada000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad40c77d0 con 0x7f0ad406e470 2024-02-19T15:30:30.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0acbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad40acec0 0x7f0ad415d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:30.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0acbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad40acec0 0x7f0ad415d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52360/0 (socket says 172.21.15.47:52360) 2024-02-19T15:30:30.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0acbe00640 1 -- 172.21.15.47:0/1763282403 learned_addr learned my addr 172.21.15.47:0/1763282403 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:30.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0ad9600640 1 --2- 172.21.15.47:0/1763282403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad40cdf90 0x7f0ad40c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:30.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0ad8c00640 1 --2- 172.21.15.47:0/1763282403 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ad406e470 0x7f0ad40c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:30.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0acbe00640 1 -- 172.21.15.47:0/1763282403 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ad406e470 msgr2=0x7f0ad40c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:30.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0acbe00640 1 --2- 172.21.15.47:0/1763282403 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ad406e470 0x7f0ad40c7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:30.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0acbe00640 1 -- 172.21.15.47:0/1763282403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad40cdf90 msgr2=0x7f0ad40c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:30.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0acbe00640 1 --2- 172.21.15.47:0/1763282403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad40cdf90 0x7f0ad40c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:30.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0acbe00640 1 -- 172.21.15.47:0/1763282403 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0acc009b30 con 0x7f0ad40acec0 2024-02-19T15:30:30.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0acbe00640 1 --2- 172.21.15.47:0/1763282403 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad40acec0 0x7f0ad415d7f0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f0abc00c340 tx=0x7f0abc00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:30.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0ad1600640 1 -- 172.21.15.47:0/1763282403 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0abc0052d0 con 0x7f0ad40acec0 2024-02-19T15:30:30.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0ad1600640 1 -- 172.21.15.47:0/1763282403 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0abc005430 con 0x7f0ad40acec0 2024-02-19T15:30:30.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0ad1600640 1 -- 172.21.15.47:0/1763282403 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0abc007260 con 0x7f0ad40acec0 2024-02-19T15:30:30.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0ada000640 1 -- 172.21.15.47:0/1763282403 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad40c7ab0 con 0x7f0ad40acec0 2024-02-19T15:30:30.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0ada000640 1 -- 172.21.15.47:0/1763282403 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad40c7d90 con 0x7f0ad40acec0 2024-02-19T15:30:30.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.573+0000 7f0ada000640 1 -- 172.21.15.47:0/1763282403 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad4075360 con 0x7f0ad40acec0 2024-02-19T15:30:30.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.577+0000 7f0ad1600640 1 -- 172.21.15.47:0/1763282403 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0abc0073c0 con 0x7f0ad40acec0 2024-02-19T15:30:30.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.577+0000 7f0ad1600640 1 --2- 172.21.15.47:0/1763282403 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0ab0066e40 0x7f0ab0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:30.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.577+0000 7f0ad1600640 1 -- 172.21.15.47:0/1763282403 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0abc083d50 con 0x7f0ad40acec0 2024-02-19T15:30:30.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.577+0000 7f0ad8c00640 1 --2- 172.21.15.47:0/1763282403 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0ab0066e40 0x7f0ab0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:30.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.577+0000 7f0ad8c00640 1 --2- 172.21.15.47:0/1763282403 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0ab0066e40 0x7f0ab0069300 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4009b60 tx=0x7f0ac4009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:30.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.585+0000 7f0ad1600640 1 -- 172.21.15.47:0/1763282403 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0abc050ba0 con 0x7f0ad40acec0 2024-02-19T15:30:30.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.777+0000 7f0ada000640 1 -- 172.21.15.47:0/1763282403 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ad40b1190 con 0x7f0ad40acec0 2024-02-19T15:30:30.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.781+0000 7f0ad1600640 1 -- 172.21.15.47:0/1763282403 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f0abc054260 con 0x7f0ad40acec0 2024-02-19T15:30:30.780 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:30.781 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:30.782 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:30.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.781+0000 7f0ada000640 1 -- 172.21.15.47:0/1763282403 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0ab0066e40 msgr2=0x7f0ab0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:30.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.781+0000 7f0ada000640 1 --2- 172.21.15.47:0/1763282403 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0ab0066e40 0x7f0ab0069300 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4009b60 tx=0x7f0ac4009190 comp rx=0 tx=0).stop 2024-02-19T15:30:30.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.781+0000 7f0ada000640 1 -- 172.21.15.47:0/1763282403 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad40acec0 msgr2=0x7f0ad415d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:30.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.781+0000 7f0ada000640 1 --2- 172.21.15.47:0/1763282403 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad40acec0 0x7f0ad415d7f0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f0abc00c340 tx=0x7f0abc00c800 comp rx=0 tx=0).stop 2024-02-19T15:30:30.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.781+0000 7f0ada000640 1 -- 172.21.15.47:0/1763282403 shutdown_connections 2024-02-19T15:30:30.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.781+0000 7f0ada000640 1 --2- 172.21.15.47:0/1763282403 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0ab0066e40 0x7f0ab0069300 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:30.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.781+0000 7f0ada000640 1 --2- 172.21.15.47:0/1763282403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad40cdf90 0x7f0ad40c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:30.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.781+0000 7f0ada000640 1 --2- 172.21.15.47:0/1763282403 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad40acec0 0x7f0ad415d7f0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:30.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.781+0000 7f0ada000640 1 --2- 172.21.15.47:0/1763282403 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ad406e470 0x7f0ad40c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:30.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.781+0000 7f0ada000640 1 -- 172.21.15.47:0/1763282403 >> 172.21.15.47:0/1763282403 conn(0x7f0ad40b1be0 msgr2=0x7f0ad415f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:30.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.781+0000 7f0ada000640 1 -- 172.21.15.47:0/1763282403 shutdown_connections 2024-02-19T15:30:30.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:30.781+0000 7f0ada000640 1 -- 172.21.15.47:0/1763282403 wait complete. 2024-02-19T15:30:31.003 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:31.004 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:32.005 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:32.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.145+0000 7efedc600640 1 Processor -- start 2024-02-19T15:30:32.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.145+0000 7efedc600640 1 -- start start 2024-02-19T15:30:32.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.145+0000 7efedc600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efed415e270 0x7efed40d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:32.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.145+0000 7efedc600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efed40d1760 0x7efed40ce320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:32.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.145+0000 7efedc600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed40b3240 con 0x7efed40cff60 2024-02-19T15:30:32.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.145+0000 7efedc600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed40b3520 con 0x7efed415e270 2024-02-19T15:30:32.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.145+0000 7efedc600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed406cb50 con 0x7efed40d1760 2024-02-19T15:30:32.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.145+0000 7efed9200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efed415e270 0x7efed40d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:32.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.149+0000 7efed9c00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7efed40cff60 0x7efed40d0340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41856/0 (socket says 172.21.15.47:41856) 2024-02-19T15:30:32.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.149+0000 7efed9c00640 1 -- 172.21.15.47:0/2326273776 learned_addr learned my addr 172.21.15.47:0/2326273776 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:32.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.149+0000 7efeda600640 1 --2- 172.21.15.47:0/2326273776 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efed40d1760 0x7efed40ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:32.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.149+0000 7efed9200640 1 -- 172.21.15.47:0/2326273776 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efed40d1760 msgr2=0x7efed40ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:32.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.149+0000 7efed9200640 1 --2- 172.21.15.47:0/2326273776 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efed40d1760 0x7efed40ce320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:32.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.149+0000 7efed9200640 1 -- 172.21.15.47:0/2326273776 >> v1:172.21.15.47:6789/0 conn(0x7efed40cff60 legacy=0x7efed40d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:32.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.149+0000 7efed3e00640 1 -- 172.21.15.47:0/2326273776 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1458548475 0 0) 0x7efed40b3240 con 0x7efed40cff60 2024-02-19T15:30:32.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.149+0000 7efed9200640 1 -- 172.21.15.47:0/2326273776 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed406ce30 con 0x7efed415e270 2024-02-19T15:30:32.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.149+0000 7efed9200640 1 --2- 172.21.15.47:0/2326273776 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415e270 0x7efed40d0dc0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7efec00097c0 tx=0x7efec001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ef0771d7e78f59a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:32.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.149+0000 7efed3e00640 1 -- 172.21.15.47:0/2326273776 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec00a7070 con 0x7efed415e270 2024-02-19T15:30:32.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.149+0000 7efed3e00640 1 -- 172.21.15.47:0/2326273776 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec00ab430 con 0x7efed415e270 2024-02-19T15:30:32.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.149+0000 7efed3e00640 1 -- 172.21.15.47:0/2326273776 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec00ab7c0 con 0x7efed415e270 2024-02-19T15:30:32.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efedc600640 1 -- 172.21.15.47:0/2326273776 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415e270 msgr2=0x7efed40d0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:32.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efedc600640 1 --2- 172.21.15.47:0/2326273776 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415e270 0x7efed40d0dc0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7efec00097c0 tx=0x7efec001c410 comp rx=0 tx=0).stop 2024-02-19T15:30:32.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efedc600640 1 -- 172.21.15.47:0/2326273776 shutdown_connections 2024-02-19T15:30:32.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efedc600640 1 --2- 172.21.15.47:0/2326273776 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efed40d1760 0x7efed40ce320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:32.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efedc600640 1 --2- 172.21.15.47:0/2326273776 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415e270 0x7efed40d0dc0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:32.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efedc600640 1 -- 172.21.15.47:0/2326273776 >> 172.21.15.47:0/2326273776 conn(0x7efed40d1b90 msgr2=0x7efed40d1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:32.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efedc600640 1 -- 172.21.15.47:0/2326273776 shutdown_connections 2024-02-19T15:30:32.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efedc600640 1 -- 172.21.15.47:0/2326273776 wait complete. 2024-02-19T15:30:32.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efedc600640 1 Processor -- start 2024-02-19T15:30:32.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efedc600640 1 -- start start 2024-02-19T15:30:32.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efedc600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efed40cff60 0x7efed415f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:32.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efedc600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efed40d1760 0x7efed414fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:32.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efedc600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415e270 0x7efed4157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:32.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efedc600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed415f960 con 0x7efed415e270 2024-02-19T15:30:32.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efedc600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed415fc40 con 0x7efed40d1760 2024-02-19T15:30:32.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efedc600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed415ff20 con 0x7efed40cff60 2024-02-19T15:30:32.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efeda600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415e270 0x7efed4157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:32.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efeda600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415e270 0x7efed4157e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52372/0 (socket says 172.21.15.47:52372) 2024-02-19T15:30:32.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efeda600640 1 -- 172.21.15.47:0/1587203667 learned_addr learned my addr 172.21.15.47:0/1587203667 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:32.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efed9c00640 1 --2- 172.21.15.47:0/1587203667 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efed40cff60 0x7efed415f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:32.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efed9200640 1 --2- 172.21.15.47:0/1587203667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efed40d1760 0x7efed414fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:32.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efeda600640 1 -- 172.21.15.47:0/1587203667 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efed40cff60 msgr2=0x7efed415f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:32.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efeda600640 1 --2- 172.21.15.47:0/1587203667 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efed40cff60 0x7efed415f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:32.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efeda600640 1 -- 172.21.15.47:0/1587203667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efed40d1760 msgr2=0x7efed414fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:32.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efeda600640 1 --2- 172.21.15.47:0/1587203667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efed40d1760 0x7efed414fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:32.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.153+0000 7efeda600640 1 -- 172.21.15.47:0/1587203667 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec0009420 con 0x7efed415e270 2024-02-19T15:30:32.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.157+0000 7efeda600640 1 --2- 172.21.15.47:0/1587203667 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415e270 0x7efed4157e50 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7efecc007de0 tx=0x7efecc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:32.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.157+0000 7efed1600640 1 -- 172.21.15.47:0/1587203667 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efecc014620 con 0x7efed415e270 2024-02-19T15:30:32.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.157+0000 7efed1600640 1 -- 172.21.15.47:0/1587203667 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efecc00ee90 con 0x7efed415e270 2024-02-19T15:30:32.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.157+0000 7efedc600640 1 -- 172.21.15.47:0/1587203667 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed4151850 con 0x7efed415e270 2024-02-19T15:30:32.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.157+0000 7efed1600640 1 -- 172.21.15.47:0/1587203667 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efecc0149e0 con 0x7efed415e270 2024-02-19T15:30:32.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.157+0000 7efedc600640 1 -- 172.21.15.47:0/1587203667 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed415f5c0 con 0x7efed415e270 2024-02-19T15:30:32.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.157+0000 7efedc600640 1 -- 172.21.15.47:0/1587203667 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed4068500 con 0x7efed415e270 2024-02-19T15:30:32.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.157+0000 7efed1600640 1 -- 172.21.15.47:0/1587203667 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efecc014b40 con 0x7efed415e270 2024-02-19T15:30:32.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.157+0000 7efed1600640 1 --2- 172.21.15.47:0/1587203667 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efeb4066e40 0x7efeb4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:32.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.157+0000 7efed1600640 1 -- 172.21.15.47:0/1587203667 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efecc085570 con 0x7efed415e270 2024-02-19T15:30:32.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.157+0000 7efed9c00640 1 --2- 172.21.15.47:0/1587203667 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efeb4066e40 0x7efeb4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:32.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.161+0000 7efed9c00640 1 --2- 172.21.15.47:0/1587203667 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efeb4066e40 0x7efeb4069300 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7efec8000f70 tx=0x7efec8007b60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:32.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.161+0000 7efed1600640 1 -- 172.21.15.47:0/1587203667 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7efecc052280 con 0x7efed415e270 2024-02-19T15:30:32.215 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:32.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.573+0000 7efedc600640 1 -- 172.21.15.47:0/1587203667 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efed406c840 con 0x7efed415e270 2024-02-19T15:30:32.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.573+0000 7efed1600640 1 -- 172.21.15.47:0/1587203667 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7efecc055940 con 0x7efed415e270 2024-02-19T15:30:32.574 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:32.574 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:32.574 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:32.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.577+0000 7efedc600640 1 -- 172.21.15.47:0/1587203667 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efeb4066e40 msgr2=0x7efeb4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:32.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.577+0000 7efedc600640 1 --2- 172.21.15.47:0/1587203667 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efeb4066e40 0x7efeb4069300 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7efec8000f70 tx=0x7efec8007b60 comp rx=0 tx=0).stop 2024-02-19T15:30:32.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.577+0000 7efedc600640 1 -- 172.21.15.47:0/1587203667 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415e270 msgr2=0x7efed4157e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:32.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.577+0000 7efedc600640 1 --2- 172.21.15.47:0/1587203667 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415e270 0x7efed4157e50 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7efecc007de0 tx=0x7efecc00e4b0 comp rx=0 tx=0).stop 2024-02-19T15:30:32.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.577+0000 7efedc600640 1 -- 172.21.15.47:0/1587203667 shutdown_connections 2024-02-19T15:30:32.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.577+0000 7efedc600640 1 --2- 172.21.15.47:0/1587203667 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efeb4066e40 0x7efeb4069300 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:32.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.577+0000 7efedc600640 1 --2- 172.21.15.47:0/1587203667 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efed415e270 0x7efed4157e50 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:32.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.577+0000 7efedc600640 1 --2- 172.21.15.47:0/1587203667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efed40d1760 0x7efed414fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:32.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.577+0000 7efedc600640 1 --2- 172.21.15.47:0/1587203667 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efed40cff60 0x7efed415f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:32.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.577+0000 7efedc600640 1 -- 172.21.15.47:0/1587203667 >> 172.21.15.47:0/1587203667 conn(0x7efed40d1b90 msgr2=0x7efed41591e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:32.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.577+0000 7efedc600640 1 -- 172.21.15.47:0/1587203667 shutdown_connections 2024-02-19T15:30:32.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:32.577+0000 7efedc600640 1 -- 172.21.15.47:0/1587203667 wait complete. 2024-02-19T15:30:32.798 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:32.798 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:33.799 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:33.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.941+0000 7f86fe200640 1 Processor -- start 2024-02-19T15:30:33.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.941+0000 7f86fe200640 1 -- start start 2024-02-19T15:30:33.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.941+0000 7f86fe200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86f815d6e0 0x7f86f8162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:33.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.941+0000 7f86fe200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f86f8071350 0x7f86f8164c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:33.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.941+0000 7f86fe200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f80718c0 con 0x7f86f815b1a0 2024-02-19T15:30:33.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.941+0000 7f86fe200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86f8071ba0 con 0x7f86f8071350 2024-02-19T15:30:33.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.941+0000 7f86fe200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f8071e80 con 0x7f86f815d6e0 2024-02-19T15:30:33.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.941+0000 7f86f7400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f86f815b1a0 0x7f86f815b580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51480/0 (socket says 172.21.15.47:51480) 2024-02-19T15:30:33.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.941+0000 7f86f7400640 1 -- 172.21.15.47:0/371723883 learned_addr learned my addr 172.21.15.47:0/371723883 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:33.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.941+0000 7f86f7e00640 1 --2- 172.21.15.47:0/371723883 >> v2:172.21.15.47:3300/0 conn(0x7f86f8071350 0x7f86f8164c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:33.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.941+0000 7f86f6a00640 1 --2- 172.21.15.47:0/371723883 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86f815d6e0 0x7f86f8162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:33.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.941+0000 7f86f6000640 1 -- 172.21.15.47:0/371723883 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763317346 0 0) 0x7f86f80718c0 con 0x7f86f815b1a0 2024-02-19T15:30:33.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.941+0000 7f86f6000640 1 -- 172.21.15.47:0/371723883 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86dc005740 con 0x7f86f815b1a0 2024-02-19T15:30:33.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.941+0000 7f86f7e00640 1 -- 172.21.15.47:0/371723883 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86f815d6e0 msgr2=0x7f86f8162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:33.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.941+0000 7f86f7e00640 1 --2- 172.21.15.47:0/371723883 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86f815d6e0 0x7f86f8162740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:33.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.941+0000 7f86f7e00640 1 -- 172.21.15.47:0/371723883 >> v1:172.21.15.47:6789/0 conn(0x7f86f815b1a0 legacy=0x7f86f815b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:33.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.941+0000 7f86f7e00640 1 -- 172.21.15.47:0/371723883 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f8072160 con 0x7f86f8071350 2024-02-19T15:30:33.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.941+0000 7f86f6a00640 1 --2- 172.21.15.47:0/371723883 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86f815d6e0 0x7f86f8162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:30:33.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.945+0000 7f86f7e00640 1 --2- 172.21.15.47:0/371723883 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f86f8071350 0x7f86f8164c90 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f86f0009d90 tx=0x7f86f001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e789dec87adebf18 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:33.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.945+0000 7f86f6000640 1 -- 172.21.15.47:0/371723883 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f00a5950 con 0x7f86f8071350 2024-02-19T15:30:33.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.945+0000 7f86f6000640 1 -- 172.21.15.47:0/371723883 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f00a5ab0 con 0x7f86f8071350 2024-02-19T15:30:33.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.945+0000 7f86f6000640 1 -- 172.21.15.47:0/371723883 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f00a5e40 con 0x7f86f8071350 2024-02-19T15:30:33.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.945+0000 7f86fe200640 1 -- 172.21.15.47:0/371723883 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f86f8071350 msgr2=0x7f86f8164c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:33.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.945+0000 7f86fe200640 1 --2- 172.21.15.47:0/371723883 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f86f8071350 0x7f86f8164c90 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f86f0009d90 tx=0x7f86f001c970 comp rx=0 tx=0).stop 2024-02-19T15:30:33.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.945+0000 7f86fe200640 1 -- 172.21.15.47:0/371723883 shutdown_connections 2024-02-19T15:30:33.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.945+0000 7f86fe200640 1 --2- 172.21.15.47:0/371723883 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f86f8071350 0x7f86f8164c90 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:33.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.945+0000 7f86fe200640 1 --2- 172.21.15.47:0/371723883 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86f815d6e0 0x7f86f8162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:33.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.945+0000 7f86fe200640 1 -- 172.21.15.47:0/371723883 >> 172.21.15.47:0/371723883 conn(0x7f86f80b16a0 msgr2=0x7f86f80b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:33.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.945+0000 7f86fe200640 1 -- 172.21.15.47:0/371723883 shutdown_connections 2024-02-19T15:30:33.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.945+0000 7f86fe200640 1 -- 172.21.15.47:0/371723883 wait complete. 2024-02-19T15:30:33.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.945+0000 7f86fe200640 1 Processor -- start 2024-02-19T15:30:33.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86fe200640 1 -- start start 2024-02-19T15:30:33.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86fe200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86f8071350 0x7f86f815f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:33.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86fe200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f86f815b1a0 0x7f86f8154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:33.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86fe200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86f815d6e0 0x7f86f8159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:33.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86fe200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f8157390 con 0x7f86f815b1a0 2024-02-19T15:30:33.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86fe200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f8165040 con 0x7f86f8071350 2024-02-19T15:30:33.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86fe200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f8156a30 con 0x7f86f815d6e0 2024-02-19T15:30:33.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86f6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f86f815b1a0 0x7f86f8154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:33.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86f6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f86f815b1a0 0x7f86f8154480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57178/0 (socket says 172.21.15.47:57178) 2024-02-19T15:30:33.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86f6a00640 1 -- 172.21.15.47:0/1207311505 learned_addr learned my addr 172.21.15.47:0/1207311505 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:33.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86f7e00640 1 --2- 172.21.15.47:0/1207311505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86f815d6e0 0x7f86f8159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:33.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86f7400640 1 --2- 172.21.15.47:0/1207311505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86f8071350 0x7f86f815f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:33.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86f6a00640 1 -- 172.21.15.47:0/1207311505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86f815d6e0 msgr2=0x7f86f8159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:33.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86f6a00640 1 --2- 172.21.15.47:0/1207311505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86f815d6e0 0x7f86f8159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:33.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86f6a00640 1 -- 172.21.15.47:0/1207311505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86f8071350 msgr2=0x7f86f815f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:33.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86f6a00640 1 --2- 172.21.15.47:0/1207311505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86f8071350 0x7f86f815f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:33.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86f6a00640 1 -- 172.21.15.47:0/1207311505 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f0009a70 con 0x7f86f815b1a0 2024-02-19T15:30:33.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86f6a00640 1 --2- 172.21.15.47:0/1207311505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f86f815b1a0 0x7f86f8154480 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f86e400e810 tx=0x7f86e400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:33.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86ef400640 1 -- 172.21.15.47:0/1207311505 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e400c8d0 con 0x7f86f815b1a0 2024-02-19T15:30:33.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86ef400640 1 -- 172.21.15.47:0/1207311505 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86e400ca30 con 0x7f86f815b1a0 2024-02-19T15:30:33.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86ef400640 1 -- 172.21.15.47:0/1207311505 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e400cdc0 con 0x7f86f815b1a0 2024-02-19T15:30:33.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86fe200640 1 -- 172.21.15.47:0/1207311505 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f8156cd0 con 0x7f86f815b1a0 2024-02-19T15:30:33.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.949+0000 7f86fe200640 1 -- 172.21.15.47:0/1207311505 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f815f740 con 0x7f86f815b1a0 2024-02-19T15:30:33.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.953+0000 7f86fe200640 1 -- 172.21.15.47:0/1207311505 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f81574d0 con 0x7f86f815b1a0 2024-02-19T15:30:33.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.953+0000 7f86ef400640 1 -- 172.21.15.47:0/1207311505 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f86e4013410 con 0x7f86f815b1a0 2024-02-19T15:30:33.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.953+0000 7f86ef400640 1 --2- 172.21.15.47:0/1207311505 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f86d8066e40 0x7f86d8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:33.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.953+0000 7f86ef400640 1 -- 172.21.15.47:0/1207311505 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f86e4084880 con 0x7f86f815b1a0 2024-02-19T15:30:33.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.953+0000 7f86f7400640 1 --2- 172.21.15.47:0/1207311505 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f86d8066e40 0x7f86d8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:33.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.953+0000 7f86f7400640 1 --2- 172.21.15.47:0/1207311505 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f86d8066e40 0x7f86d8069300 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f86e8007fd0 tx=0x7f86e8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:33.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:33.957+0000 7f86ef400640 1 -- 172.21.15.47:0/1207311505 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f86f81574d0 con 0x7f86f815b1a0 2024-02-19T15:30:34.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:34.157+0000 7f86fe200640 1 -- 172.21.15.47:0/1207311505 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86f81574d0 con 0x7f86f815b1a0 2024-02-19T15:30:34.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:34.157+0000 7f86ef400640 1 -- 172.21.15.47:0/1207311505 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f86e4021d80 con 0x7f86f815b1a0 2024-02-19T15:30:34.159 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:34.159 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:34.159 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:34.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:34.161+0000 7f86fe200640 1 -- 172.21.15.47:0/1207311505 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f86d8066e40 msgr2=0x7f86d8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:34.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:34.161+0000 7f86fe200640 1 --2- 172.21.15.47:0/1207311505 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f86d8066e40 0x7f86d8069300 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f86e8007fd0 tx=0x7f86e8009040 comp rx=0 tx=0).stop 2024-02-19T15:30:34.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:34.161+0000 7f86fe200640 1 -- 172.21.15.47:0/1207311505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f86f815b1a0 msgr2=0x7f86f8154480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:34.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:34.161+0000 7f86fe200640 1 --2- 172.21.15.47:0/1207311505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f86f815b1a0 0x7f86f8154480 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f86e400e810 tx=0x7f86e400ecd0 comp rx=0 tx=0).stop 2024-02-19T15:30:34.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:34.161+0000 7f86fe200640 1 -- 172.21.15.47:0/1207311505 shutdown_connections 2024-02-19T15:30:34.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:34.161+0000 7f86fe200640 1 --2- 172.21.15.47:0/1207311505 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f86d8066e40 0x7f86d8069300 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:34.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:34.161+0000 7f86fe200640 1 --2- 172.21.15.47:0/1207311505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86f815d6e0 0x7f86f8159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:34.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:34.161+0000 7f86fe200640 1 --2- 172.21.15.47:0/1207311505 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f86f815b1a0 0x7f86f8154480 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:34.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:34.161+0000 7f86fe200640 1 --2- 172.21.15.47:0/1207311505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86f8071350 0x7f86f815f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:34.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:34.161+0000 7f86fe200640 1 -- 172.21.15.47:0/1207311505 >> 172.21.15.47:0/1207311505 conn(0x7f86f80b16a0 msgr2=0x7f86f80af680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:34.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:34.161+0000 7f86fe200640 1 -- 172.21.15.47:0/1207311505 shutdown_connections 2024-02-19T15:30:34.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:34.161+0000 7f86fe200640 1 -- 172.21.15.47:0/1207311505 wait complete. 2024-02-19T15:30:34.217 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:34.382 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:34.382 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:35.383 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:35.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.521+0000 7fc356800640 1 Processor -- start 2024-02-19T15:30:35.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.521+0000 7fc356800640 1 -- start start 2024-02-19T15:30:35.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.525+0000 7fc356800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc35006ef50 0x7fc35006d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:35.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.525+0000 7fc356800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc35015d660 0x7fc3501646f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:35.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.525+0000 7fc356800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35006ddf0 con 0x7fc35006f330 2024-02-19T15:30:35.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.525+0000 7fc356800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc35015c350 con 0x7fc35015d660 2024-02-19T15:30:35.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.525+0000 7fc356800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35006dbe0 con 0x7fc35006ef50 2024-02-19T15:30:35.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.525+0000 7fc34f400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc35006ef50 0x7fc35006d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:35.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.525+0000 7fc34fe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc35015d660 0x7fc3501646f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:35.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.525+0000 7fc34f400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc35006ef50 0x7fc35006d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:52438/0 (socket says 172.21.15.47:52438) 2024-02-19T15:30:35.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.525+0000 7fc34f400640 1 -- 172.21.15.47:0/1704761057 learned_addr learned my addr 172.21.15.47:0/1704761057 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:35.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.525+0000 7fc34f400640 1 -- 172.21.15.47:0/1704761057 >> v2:172.21.15.116:3300/0 conn(0x7fc35015d660 msgr2=0x7fc3501646f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:35.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.525+0000 7fc34f400640 1 --2- 172.21.15.47:0/1704761057 >> v2:172.21.15.116:3300/0 conn(0x7fc35015d660 0x7fc3501646f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:35.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.525+0000 7fc34f400640 1 -- 172.21.15.47:0/1704761057 >> v1:172.21.15.116:6789/0 conn(0x7fc35006f330 legacy=0x7fc35015aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:35.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.525+0000 7fc34f400640 1 -- 172.21.15.47:0/1704761057 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35006e0d0 con 0x7fc35006ef50 2024-02-19T15:30:35.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.525+0000 7fc34e000640 1 -- 172.21.15.47:0/1704761057 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435631214 0 0) 0x7fc35006ddf0 con 0x7fc35006f330 2024-02-19T15:30:35.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.529+0000 7fc34f400640 1 --2- 172.21.15.47:0/1704761057 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc35006ef50 0x7fc35006d6a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc340009870 tx=0x7fc34001c530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c17a672b905e54 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:35.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.529+0000 7fc34e000640 1 -- 172.21.15.47:0/1704761057 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3400a95f0 con 0x7fc35006ef50 2024-02-19T15:30:35.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.529+0000 7fc34e000640 1 -- 172.21.15.47:0/1704761057 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3400a9750 con 0x7fc35006ef50 2024-02-19T15:30:35.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.529+0000 7fc34e000640 1 -- 172.21.15.47:0/1704761057 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3400a9ae0 con 0x7fc35006ef50 2024-02-19T15:30:35.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.529+0000 7fc356800640 1 -- 172.21.15.47:0/1704761057 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc35006ef50 msgr2=0x7fc35006d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:35.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.529+0000 7fc356800640 1 --2- 172.21.15.47:0/1704761057 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc35006ef50 0x7fc35006d6a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc340009870 tx=0x7fc34001c530 comp rx=0 tx=0).stop 2024-02-19T15:30:35.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.529+0000 7fc356800640 1 -- 172.21.15.47:0/1704761057 shutdown_connections 2024-02-19T15:30:35.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.529+0000 7fc356800640 1 --2- 172.21.15.47:0/1704761057 >> v2:172.21.15.116:3300/0 conn(0x7fc35015d660 0x7fc3501646f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:35.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.529+0000 7fc356800640 1 --2- 172.21.15.47:0/1704761057 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc35006ef50 0x7fc35006d6a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:35.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.529+0000 7fc356800640 1 -- 172.21.15.47:0/1704761057 >> 172.21.15.47:0/1704761057 conn(0x7fc3500d0ad0 msgr2=0x7fc3500d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:35.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.529+0000 7fc356800640 1 -- 172.21.15.47:0/1704761057 shutdown_connections 2024-02-19T15:30:35.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.529+0000 7fc356800640 1 -- 172.21.15.47:0/1704761057 wait complete. 2024-02-19T15:30:35.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.529+0000 7fc356800640 1 Processor -- start 2024-02-19T15:30:35.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.529+0000 7fc356800640 1 -- start start 2024-02-19T15:30:35.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc356800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc35006ef50 0x7fc3500b3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:35.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc356800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc35006f330 0x7fc35014f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:35.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc356800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc35015d660 0x7fc3500b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:35.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc356800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc350142d50 con 0x7fc35006ef50 2024-02-19T15:30:35.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc356800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35014fc30 con 0x7fc35006f330 2024-02-19T15:30:35.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc356800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35014ff10 con 0x7fc35015d660 2024-02-19T15:30:35.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc34f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc35006ef50 0x7fc3500b3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:35.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc34f400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc35006ef50 0x7fc3500b3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57190/0 (socket says 172.21.15.47:57190) 2024-02-19T15:30:35.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc34f400640 1 -- 172.21.15.47:0/2590721000 learned_addr learned my addr 172.21.15.47:0/2590721000 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:35.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc34ea00640 1 --2- 172.21.15.47:0/2590721000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc35006f330 0x7fc35014f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:35.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc34fe00640 1 --2- 172.21.15.47:0/2590721000 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc35015d660 0x7fc3500b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:35.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc34f400640 1 -- 172.21.15.47:0/2590721000 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc35015d660 msgr2=0x7fc3500b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:35.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc34f400640 1 --2- 172.21.15.47:0/2590721000 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc35015d660 0x7fc3500b05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:35.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc34f400640 1 -- 172.21.15.47:0/2590721000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc35006f330 msgr2=0x7fc35014f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:35.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc34f400640 1 --2- 172.21.15.47:0/2590721000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc35006f330 0x7fc35014f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:35.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc34f400640 1 -- 172.21.15.47:0/2590721000 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3400094d0 con 0x7fc35006ef50 2024-02-19T15:30:35.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc34ea00640 1 --2- 172.21.15.47:0/2590721000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc35006f330 0x7fc35014f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:30:35.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc34fe00640 1 --2- 172.21.15.47:0/2590721000 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc35015d660 0x7fc3500b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:30:35.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc34f400640 1 --2- 172.21.15.47:0/2590721000 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc35006ef50 0x7fc3500b3bb0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fc34001abe0 tx=0x7fc3400192d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:35.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc347400640 1 -- 172.21.15.47:0/2590721000 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3400b1800 con 0x7fc35006ef50 2024-02-19T15:30:35.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc347400640 1 -- 172.21.15.47:0/2590721000 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3400b1980 con 0x7fc35006ef50 2024-02-19T15:30:35.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc356800640 1 -- 172.21.15.47:0/2590721000 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc350142f60 con 0x7fc35006ef50 2024-02-19T15:30:35.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc347400640 1 -- 172.21.15.47:0/2590721000 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3400b1d30 con 0x7fc35006ef50 2024-02-19T15:30:35.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc356800640 1 -- 172.21.15.47:0/2590721000 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35014f900 con 0x7fc35006ef50 2024-02-19T15:30:35.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc356800640 1 -- 172.21.15.47:0/2590721000 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc350143480 con 0x7fc35006ef50 2024-02-19T15:30:35.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.533+0000 7fc347400640 1 -- 172.21.15.47:0/2590721000 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc3400c43f0 con 0x7fc35006ef50 2024-02-19T15:30:35.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.537+0000 7fc347400640 1 --2- 172.21.15.47:0/2590721000 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc330066e40 0x7fc330069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:35.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.537+0000 7fc347400640 1 -- 172.21.15.47:0/2590721000 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc34011f4c0 con 0x7fc35006ef50 2024-02-19T15:30:35.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.537+0000 7fc34ea00640 1 --2- 172.21.15.47:0/2590721000 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc330066e40 0x7fc330069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:35.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.537+0000 7fc34ea00640 1 --2- 172.21.15.47:0/2590721000 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc330066e40 0x7fc330069300 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fc3501f9fa0 tx=0x7fc33c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:35.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.541+0000 7fc347400640 1 -- 172.21.15.47:0/2590721000 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc350143480 con 0x7fc35006ef50 2024-02-19T15:30:35.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.733+0000 7fc356800640 1 -- 172.21.15.47:0/2590721000 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc350143480 con 0x7fc35006ef50 2024-02-19T15:30:35.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.733+0000 7fc347400640 1 -- 172.21.15.47:0/2590721000 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7fc350143480 con 0x7fc35006ef50 2024-02-19T15:30:35.734 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:35.735 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:35.735 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:35.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.737+0000 7fc356800640 1 -- 172.21.15.47:0/2590721000 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc330066e40 msgr2=0x7fc330069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:35.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.737+0000 7fc356800640 1 --2- 172.21.15.47:0/2590721000 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc330066e40 0x7fc330069300 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fc3501f9fa0 tx=0x7fc33c009040 comp rx=0 tx=0).stop 2024-02-19T15:30:35.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.737+0000 7fc356800640 1 -- 172.21.15.47:0/2590721000 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc35006ef50 msgr2=0x7fc3500b3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:35.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.737+0000 7fc356800640 1 --2- 172.21.15.47:0/2590721000 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc35006ef50 0x7fc3500b3bb0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fc34001abe0 tx=0x7fc3400192d0 comp rx=0 tx=0).stop 2024-02-19T15:30:35.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.737+0000 7fc356800640 1 -- 172.21.15.47:0/2590721000 shutdown_connections 2024-02-19T15:30:35.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.737+0000 7fc356800640 1 --2- 172.21.15.47:0/2590721000 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc330066e40 0x7fc330069300 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:35.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.737+0000 7fc356800640 1 --2- 172.21.15.47:0/2590721000 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc35015d660 0x7fc3500b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:35.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.737+0000 7fc356800640 1 --2- 172.21.15.47:0/2590721000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc35006f330 0x7fc35014f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:35.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.737+0000 7fc356800640 1 --2- 172.21.15.47:0/2590721000 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc35006ef50 0x7fc3500b3bb0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:35.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.737+0000 7fc356800640 1 -- 172.21.15.47:0/2590721000 >> 172.21.15.47:0/2590721000 conn(0x7fc3500d0ad0 msgr2=0x7fc3500ce580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:35.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.737+0000 7fc356800640 1 -- 172.21.15.47:0/2590721000 shutdown_connections 2024-02-19T15:30:35.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:35.737+0000 7fc356800640 1 -- 172.21.15.47:0/2590721000 wait complete. 2024-02-19T15:30:35.959 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:35.959 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:36.150 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:30:36.153 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:30:36.219 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:36.960 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:37.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.101+0000 7f7430600640 1 Processor -- start 2024-02-19T15:30:37.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.101+0000 7f7430600640 1 -- start start 2024-02-19T15:30:37.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.101+0000 7f7430600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f742815e180 0x7f74280b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:37.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.101+0000 7f7430600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f742806c1d0 0x7f742806c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:37.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.101+0000 7f7430600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74280b2270 con 0x7f742806cdf0 2024-02-19T15:30:37.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.101+0000 7f7430600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74280af880 con 0x7f742815e180 2024-02-19T15:30:37.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.101+0000 7f7430600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f742806d280 con 0x7f742806c1d0 2024-02-19T15:30:37.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.101+0000 7f742e600640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f742806cdf0 0x7f74280cfda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51486/0 (socket says 172.21.15.47:51486) 2024-02-19T15:30:37.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.101+0000 7f742e600640 1 -- 172.21.15.47:0/1901543483 learned_addr learned my addr 172.21.15.47:0/1901543483 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:37.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f742d200640 1 --2- 172.21.15.47:0/1901543483 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f742806c1d0 0x7f742806c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:37.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f742dc00640 1 --2- 172.21.15.47:0/1901543483 >> v2:172.21.15.116:3300/0 conn(0x7f742815e180 0x7f74280b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:37.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f7427e00640 1 -- 172.21.15.47:0/1901543483 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1179845386 0 0) 0x7f74280b2270 con 0x7f742806cdf0 2024-02-19T15:30:37.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f7427e00640 1 -- 172.21.15.47:0/1901543483 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7410003530 con 0x7f742806cdf0 2024-02-19T15:30:37.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f7427e00640 1 -- 172.21.15.47:0/1901543483 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1862739037 0 0) 0x7f7410003530 con 0x7f742806cdf0 2024-02-19T15:30:37.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f7427e00640 1 -- 172.21.15.47:0/1901543483 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f742806c1d0 msgr2=0x7f742806c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:37.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f7427e00640 1 --2- 172.21.15.47:0/1901543483 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f742806c1d0 0x7f742806c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:37.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f7427e00640 1 -- 172.21.15.47:0/1901543483 >> v2:172.21.15.116:3300/0 conn(0x7f742815e180 msgr2=0x7f74280b1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:37.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f7427e00640 1 --2- 172.21.15.47:0/1901543483 >> v2:172.21.15.116:3300/0 conn(0x7f742815e180 0x7f74280b1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:37.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f7427e00640 1 -- 172.21.15.47:0/1901543483 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74280b2550 con 0x7f742806cdf0 2024-02-19T15:30:37.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f7427e00640 1 -- 172.21.15.47:0/1901543483 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f7420001a00 con 0x7f742806cdf0 2024-02-19T15:30:37.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f742d200640 1 --2- 172.21.15.47:0/1901543483 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f742806c1d0 0x7f742806c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:30:37.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f7427e00640 1 -- 172.21.15.47:0/1901543483 >> v1:172.21.15.47:6789/0 conn(0x7f742806cdf0 legacy=0x7f74280cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:37.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f7427e00640 1 --2- 172.21.15.47:0/1901543483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7410003980 0x7f7410003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:37.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f7427e00640 1 --2- 172.21.15.47:0/1901543483 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f742806c1d0 0x7f74100047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:37.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f7427e00640 1 --2- 172.21.15.47:0/1901543483 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7410004d10 0x7f7410005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:37.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f7427e00640 1 -- 172.21.15.47:0/1901543483 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74280af880 con 0x7f7410004d10 2024-02-19T15:30:37.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f7427e00640 1 -- 172.21.15.47:0/1901543483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742806d280 con 0x7f7410003980 2024-02-19T15:30:37.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f7427e00640 1 -- 172.21.15.47:0/1901543483 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741000b990 con 0x7f742806c1d0 2024-02-19T15:30:37.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f742e600640 1 --2- 172.21.15.47:0/1901543483 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7410004d10 0x7f7410005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:37.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f742dc00640 1 --2- 172.21.15.47:0/1901543483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7410003980 0x7f7410003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:37.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f742d200640 1 --2- 172.21.15.47:0/1901543483 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f742806c1d0 0x7f74100047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:37.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f742dc00640 1 -- 172.21.15.47:0/1901543483 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f742806c1d0 msgr2=0x7f74100047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:37.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f742dc00640 1 --2- 172.21.15.47:0/1901543483 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f742806c1d0 0x7f74100047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:37.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f742dc00640 1 -- 172.21.15.47:0/1901543483 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7410004d10 msgr2=0x7f7410005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:37.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f742dc00640 1 --2- 172.21.15.47:0/1901543483 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7410004d10 0x7f7410005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:37.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f742dc00640 1 -- 172.21.15.47:0/1901543483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7414009420 con 0x7f7410003980 2024-02-19T15:30:37.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.105+0000 7f742d200640 1 --2- 172.21.15.47:0/1901543483 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f742806c1d0 0x7f74100047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:30:37.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.109+0000 7f742dc00640 1 --2- 172.21.15.47:0/1901543483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7410003980 0x7f7410003da0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f741c009ae0 tx=0x7f741c01cab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7fbeea0b9f27984 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:37.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.109+0000 7f7427e00640 1 -- 172.21.15.47:0/1901543483 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c0a56a0 con 0x7f7410003980 2024-02-19T15:30:37.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.109+0000 7f7427e00640 1 -- 172.21.15.47:0/1901543483 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741c0a5800 con 0x7f7410003980 2024-02-19T15:30:37.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.109+0000 7f7427e00640 1 -- 172.21.15.47:0/1901543483 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c0a5bb0 con 0x7f7410003980 2024-02-19T15:30:37.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.109+0000 7f7430600640 1 -- 172.21.15.47:0/1901543483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7410003980 msgr2=0x7f7410003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:37.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.109+0000 7f7430600640 1 --2- 172.21.15.47:0/1901543483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7410003980 0x7f7410003da0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f741c009ae0 tx=0x7f741c01cab0 comp rx=0 tx=0).stop 2024-02-19T15:30:37.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.109+0000 7f742e600640 1 -- 172.21.15.47:0/1901543483 reap_dead start 2024-02-19T15:30:37.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.109+0000 7f7430600640 1 -- 172.21.15.47:0/1901543483 shutdown_connections 2024-02-19T15:30:37.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.109+0000 7f7430600640 1 -- 172.21.15.47:0/1901543483 >> 172.21.15.47:0/1901543483 conn(0x7f74280d1ba0 msgr2=0x7f74280d1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:37.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.109+0000 7f7430600640 1 -- 172.21.15.47:0/1901543483 shutdown_connections 2024-02-19T15:30:37.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.109+0000 7f7430600640 1 -- 172.21.15.47:0/1901543483 wait complete. 2024-02-19T15:30:37.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.113+0000 7f7430600640 1 Processor -- start 2024-02-19T15:30:37.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.113+0000 7f7430600640 1 -- start start 2024-02-19T15:30:37.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.113+0000 7f7430600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7410003980 0x7f7428161210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:37.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.113+0000 7f7430600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f742815eee0 0x7f742815b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:37.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.113+0000 7f7430600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f742815b810 0x7f74281f6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:37.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.113+0000 7f7430600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742815aba0 con 0x7f742815eee0 2024-02-19T15:30:37.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.113+0000 7f7430600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742815bc90 con 0x7f7410003980 2024-02-19T15:30:37.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.113+0000 7f7430600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74281f36a0 con 0x7f742815b810 2024-02-19T15:30:37.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.113+0000 7f742d200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f742815eee0 0x7f742815b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:37.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.113+0000 7f742d200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f742815eee0 0x7f742815b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57208/0 (socket says 172.21.15.47:57208) 2024-02-19T15:30:37.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.113+0000 7f742d200640 1 -- 172.21.15.47:0/851658861 learned_addr learned my addr 172.21.15.47:0/851658861 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:37.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.113+0000 7f742e600640 1 --2- 172.21.15.47:0/851658861 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f742815b810 0x7f74281f6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:37.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.113+0000 7f742dc00640 1 --2- 172.21.15.47:0/851658861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7410003980 0x7f7428161210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:37.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.113+0000 7f742d200640 1 -- 172.21.15.47:0/851658861 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f742815b810 msgr2=0x7f74281f6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:37.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.113+0000 7f742d200640 1 --2- 172.21.15.47:0/851658861 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f742815b810 0x7f74281f6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:37.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.113+0000 7f742d200640 1 -- 172.21.15.47:0/851658861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7410003980 msgr2=0x7f7428161210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:37.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.113+0000 7f742d200640 1 --2- 172.21.15.47:0/851658861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7410003980 0x7f7428161210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:37.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.113+0000 7f742d200640 1 -- 172.21.15.47:0/851658861 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74281f3980 con 0x7f742815eee0 2024-02-19T15:30:37.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.113+0000 7f742dc00640 1 --2- 172.21.15.47:0/851658861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7410003980 0x7f7428161210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:30:37.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.113+0000 7f742d200640 1 --2- 172.21.15.47:0/851658861 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f742815eee0 0x7f742815b2d0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f7414011cd0 tx=0x7f741400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:37.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.117+0000 7f7425600640 1 -- 172.21.15.47:0/851658861 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7414014440 con 0x7f742815eee0 2024-02-19T15:30:37.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.117+0000 7f7425600640 1 -- 172.21.15.47:0/851658861 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74140145a0 con 0x7f742815eee0 2024-02-19T15:30:37.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.117+0000 7f7425600640 1 -- 172.21.15.47:0/851658861 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7414014950 con 0x7f742815eee0 2024-02-19T15:30:37.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.117+0000 7f7430600640 1 -- 172.21.15.47:0/851658861 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74281f3c60 con 0x7f742815eee0 2024-02-19T15:30:37.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.117+0000 7f7430600640 1 -- 172.21.15.47:0/851658861 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74281f4220 con 0x7f742815eee0 2024-02-19T15:30:37.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.117+0000 7f7430600640 1 -- 172.21.15.47:0/851658861 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7428068500 con 0x7f742815eee0 2024-02-19T15:30:37.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.117+0000 7f7425600640 1 -- 172.21.15.47:0/851658861 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7414014ab0 con 0x7f742815eee0 2024-02-19T15:30:37.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.117+0000 7f7425600640 1 --2- 172.21.15.47:0/851658861 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7408066e70 0x7f7408069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:37.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.117+0000 7f7425600640 1 -- 172.21.15.47:0/851658861 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7414085780 con 0x7f742815eee0 2024-02-19T15:30:37.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.117+0000 7f742dc00640 1 --2- 172.21.15.47:0/851658861 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7408066e70 0x7f7408069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:37.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.121+0000 7f742dc00640 1 --2- 172.21.15.47:0/851658861 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7408066e70 0x7f7408069330 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f741c0af000 tx=0x7f741c0046d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:37.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.121+0000 7f7425600640 1 -- 172.21.15.47:0/851658861 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7414052490 con 0x7f742815eee0 2024-02-19T15:30:37.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.325+0000 7f7430600640 1 -- 172.21.15.47:0/851658861 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7428075360 con 0x7f742815eee0 2024-02-19T15:30:37.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.325+0000 7f7425600640 1 -- 172.21.15.47:0/851658861 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f7414055b50 con 0x7f742815eee0 2024-02-19T15:30:37.326 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:37.326 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:37.326 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:37.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.329+0000 7f7430600640 1 -- 172.21.15.47:0/851658861 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7408066e70 msgr2=0x7f7408069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:37.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.329+0000 7f7430600640 1 --2- 172.21.15.47:0/851658861 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7408066e70 0x7f7408069330 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f741c0af000 tx=0x7f741c0046d0 comp rx=0 tx=0).stop 2024-02-19T15:30:37.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.329+0000 7f7430600640 1 -- 172.21.15.47:0/851658861 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f742815eee0 msgr2=0x7f742815b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:37.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.329+0000 7f7430600640 1 --2- 172.21.15.47:0/851658861 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f742815eee0 0x7f742815b2d0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f7414011cd0 tx=0x7f741400f610 comp rx=0 tx=0).stop 2024-02-19T15:30:37.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.329+0000 7f7430600640 1 -- 172.21.15.47:0/851658861 shutdown_connections 2024-02-19T15:30:37.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.329+0000 7f7430600640 1 --2- 172.21.15.47:0/851658861 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7408066e70 0x7f7408069330 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:37.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.329+0000 7f7430600640 1 --2- 172.21.15.47:0/851658861 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f742815b810 0x7f74281f6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:37.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.329+0000 7f7430600640 1 --2- 172.21.15.47:0/851658861 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f742815eee0 0x7f742815b2d0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:37.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.329+0000 7f7430600640 1 --2- 172.21.15.47:0/851658861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7410003980 0x7f7428161210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:37.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.329+0000 7f7430600640 1 -- 172.21.15.47:0/851658861 >> 172.21.15.47:0/851658861 conn(0x7f74280d1ba0 msgr2=0x7f74280cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:37.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.329+0000 7f7430600640 1 -- 172.21.15.47:0/851658861 shutdown_connections 2024-02-19T15:30:37.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:37.329+0000 7f7430600640 1 -- 172.21.15.47:0/851658861 wait complete. 2024-02-19T15:30:37.550 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:37.550 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:38.221 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:38.551 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:38.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.689+0000 7f7522a00640 1 Processor -- start 2024-02-19T15:30:38.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.693+0000 7f7522a00640 1 -- start start 2024-02-19T15:30:38.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.693+0000 7f7522a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f751c15ba70 0x7f751c06dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:38.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.693+0000 7f7522a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f751c0b1ee0 0x7f751c0720c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:38.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.693+0000 7f7522a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f751c06e6a0 con 0x7f751c0cf000 2024-02-19T15:30:38.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.693+0000 7f7522a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f751c06e980 con 0x7f751c0b1ee0 2024-02-19T15:30:38.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.693+0000 7f7522a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f751c0b1400 con 0x7f751c15ba70 2024-02-19T15:30:38.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.693+0000 7f7520c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f751c0b1ee0 0x7f751c0720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:38.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.693+0000 7f7520c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f751c0b1ee0 0x7f751c0720c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57214/0 (socket says 172.21.15.47:57214) 2024-02-19T15:30:38.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.693+0000 7f7520c00640 1 -- 172.21.15.47:0/147516216 learned_addr learned my addr 172.21.15.47:0/147516216 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:38.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.693+0000 7f7521600640 1 --2- 172.21.15.47:0/147516216 >> v2:172.21.15.116:3300/0 conn(0x7f751c15ba70 0x7f751c06dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:38.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.693+0000 7f7520c00640 1 -- 172.21.15.47:0/147516216 >> v2:172.21.15.116:3300/0 conn(0x7f751c15ba70 msgr2=0x7f751c06dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:38.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.693+0000 7f7520c00640 1 --2- 172.21.15.47:0/147516216 >> v2:172.21.15.116:3300/0 conn(0x7f751c15ba70 0x7f751c06dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:38.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.693+0000 7f7520c00640 1 -- 172.21.15.47:0/147516216 >> v1:172.21.15.47:6789/0 conn(0x7f751c0cf000 legacy=0x7f751c15f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:38.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.693+0000 7f7520c00640 1 -- 172.21.15.47:0/147516216 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751c0b16e0 con 0x7f751c0b1ee0 2024-02-19T15:30:38.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.693+0000 7f751be00640 1 -- 172.21.15.47:0/147516216 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767287097 0 0) 0x7f751c06e6a0 con 0x7f751c0cf000 2024-02-19T15:30:38.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.697+0000 7f7520c00640 1 --2- 172.21.15.47:0/147516216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f751c0b1ee0 0x7f751c0720c0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f751c0b24f0 tx=0x7f750801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d3c58bc94dc2657 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:38.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.697+0000 7f751be00640 1 -- 172.21.15.47:0/147516216 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75080ab600 con 0x7f751c0b1ee0 2024-02-19T15:30:38.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.697+0000 7f751be00640 1 -- 172.21.15.47:0/147516216 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75080a7050 con 0x7f751c0b1ee0 2024-02-19T15:30:38.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.697+0000 7f751be00640 1 -- 172.21.15.47:0/147516216 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75080ab980 con 0x7f751c0b1ee0 2024-02-19T15:30:38.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.697+0000 7f7522a00640 1 -- 172.21.15.47:0/147516216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f751c0b1ee0 msgr2=0x7f751c0720c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:38.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.697+0000 7f7522a00640 1 --2- 172.21.15.47:0/147516216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f751c0b1ee0 0x7f751c0720c0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f751c0b24f0 tx=0x7f750801c460 comp rx=0 tx=0).stop 2024-02-19T15:30:38.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.697+0000 7f7522a00640 1 -- 172.21.15.47:0/147516216 shutdown_connections 2024-02-19T15:30:38.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.697+0000 7f7522a00640 1 --2- 172.21.15.47:0/147516216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f751c0b1ee0 0x7f751c0720c0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:38.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.697+0000 7f7522a00640 1 --2- 172.21.15.47:0/147516216 >> v2:172.21.15.116:3300/0 conn(0x7f751c15ba70 0x7f751c06dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:38.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.697+0000 7f7522a00640 1 -- 172.21.15.47:0/147516216 >> 172.21.15.47:0/147516216 conn(0x7f751c0cf7c0 msgr2=0x7f751c0cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:38.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.697+0000 7f7522a00640 1 -- 172.21.15.47:0/147516216 shutdown_connections 2024-02-19T15:30:38.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.697+0000 7f7522a00640 1 -- 172.21.15.47:0/147516216 wait complete. 2024-02-19T15:30:38.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.697+0000 7f7522a00640 1 Processor -- start 2024-02-19T15:30:38.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.697+0000 7f7522a00640 1 -- start start 2024-02-19T15:30:38.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7522a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f751c0b1ee0 0x7f751c154dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:38.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7522a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c0cf000 0x7f751c0c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:38.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7522a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c15ba70 0x7f751c0b3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:38.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7522a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751c1505f0 con 0x7f751c0b1ee0 2024-02-19T15:30:38.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7522a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751c14bda0 con 0x7f751c0cf000 2024-02-19T15:30:38.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7522a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f751c14fec0 con 0x7f751c15ba70 2024-02-19T15:30:38.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7521600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f751c0b1ee0 0x7f751c154dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:38.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7521600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f751c0b1ee0 0x7f751c154dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57218/0 (socket says 172.21.15.47:57218) 2024-02-19T15:30:38.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7521600640 1 -- 172.21.15.47:0/3316024996 learned_addr learned my addr 172.21.15.47:0/3316024996 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:38.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7520c00640 1 --2- 172.21.15.47:0/3316024996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c0cf000 0x7f751c0c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:38.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7522000640 1 --2- 172.21.15.47:0/3316024996 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c15ba70 0x7f751c0b3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:38.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7521600640 1 -- 172.21.15.47:0/3316024996 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c15ba70 msgr2=0x7f751c0b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:38.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7521600640 1 --2- 172.21.15.47:0/3316024996 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c15ba70 0x7f751c0b3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:38.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7521600640 1 -- 172.21.15.47:0/3316024996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c0cf000 msgr2=0x7f751c0c8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:38.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7521600640 1 --2- 172.21.15.47:0/3316024996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c0cf000 0x7f751c0c8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:38.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7521600640 1 -- 172.21.15.47:0/3316024996 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75080094d0 con 0x7f751c0b1ee0 2024-02-19T15:30:38.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7520c00640 1 --2- 172.21.15.47:0/3316024996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c0cf000 0x7f751c0c8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:30:38.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7522000640 1 --2- 172.21.15.47:0/3316024996 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c15ba70 0x7f751c0b3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:30:38.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7521600640 1 --2- 172.21.15.47:0/3316024996 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f751c0b1ee0 0x7f751c154dd0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f751000d5f0 tx=0x7f751000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:38.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7519600640 1 -- 172.21.15.47:0/3316024996 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75100125f0 con 0x7f751c0b1ee0 2024-02-19T15:30:38.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7519600640 1 -- 172.21.15.47:0/3316024996 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7510012750 con 0x7f751c0b1ee0 2024-02-19T15:30:38.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7519600640 1 -- 172.21.15.47:0/3316024996 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7510012ae0 con 0x7f751c0b1ee0 2024-02-19T15:30:38.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7522a00640 1 -- 172.21.15.47:0/3316024996 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f751c14e7b0 con 0x7f751c0b1ee0 2024-02-19T15:30:38.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7522a00640 1 -- 172.21.15.47:0/3316024996 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f751c151e30 con 0x7f751c0b1ee0 2024-02-19T15:30:38.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7522a00640 1 -- 172.21.15.47:0/3316024996 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f751c075360 con 0x7f751c0b1ee0 2024-02-19T15:30:38.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.701+0000 7f7519600640 1 -- 172.21.15.47:0/3316024996 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7510012c40 con 0x7f751c0b1ee0 2024-02-19T15:30:38.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.705+0000 7f7519600640 1 --2- 172.21.15.47:0/3316024996 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f74fc066e40 0x7f74fc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:38.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.705+0000 7f7519600640 1 -- 172.21.15.47:0/3316024996 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7510084f20 con 0x7f751c0b1ee0 2024-02-19T15:30:38.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.705+0000 7f7520c00640 1 --2- 172.21.15.47:0/3316024996 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f74fc066e40 0x7f74fc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:38.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.705+0000 7f7520c00640 1 --2- 172.21.15.47:0/3316024996 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f74fc066e40 0x7f74fc069300 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f75080ad000 tx=0x7f7508008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:38.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.709+0000 7f7519600640 1 -- 172.21.15.47:0/3316024996 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7510051cc0 con 0x7f751c0b1ee0 2024-02-19T15:30:38.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.905+0000 7f7522a00640 1 -- 172.21.15.47:0/3316024996 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f751c0ce8d0 con 0x7f751c0b1ee0 2024-02-19T15:30:38.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.905+0000 7f7519600640 1 -- 172.21.15.47:0/3316024996 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f7510007380 con 0x7f751c0b1ee0 2024-02-19T15:30:38.904 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:38.905 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:38.905 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:38.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.905+0000 7f7522a00640 1 -- 172.21.15.47:0/3316024996 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f74fc066e40 msgr2=0x7f74fc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:38.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.905+0000 7f7522a00640 1 --2- 172.21.15.47:0/3316024996 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f74fc066e40 0x7f74fc069300 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f75080ad000 tx=0x7f7508008040 comp rx=0 tx=0).stop 2024-02-19T15:30:38.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.905+0000 7f7522a00640 1 -- 172.21.15.47:0/3316024996 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f751c0b1ee0 msgr2=0x7f751c154dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:38.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.905+0000 7f7522a00640 1 --2- 172.21.15.47:0/3316024996 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f751c0b1ee0 0x7f751c154dd0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f751000d5f0 tx=0x7f751000dab0 comp rx=0 tx=0).stop 2024-02-19T15:30:38.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.905+0000 7f7522a00640 1 -- 172.21.15.47:0/3316024996 shutdown_connections 2024-02-19T15:30:38.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.905+0000 7f7522a00640 1 --2- 172.21.15.47:0/3316024996 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f74fc066e40 0x7f74fc069300 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:38.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.905+0000 7f7522a00640 1 --2- 172.21.15.47:0/3316024996 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c15ba70 0x7f751c0b3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:38.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.905+0000 7f7522a00640 1 --2- 172.21.15.47:0/3316024996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c0cf000 0x7f751c0c8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:38.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.905+0000 7f7522a00640 1 --2- 172.21.15.47:0/3316024996 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f751c0b1ee0 0x7f751c154dd0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:38.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.905+0000 7f7522a00640 1 -- 172.21.15.47:0/3316024996 >> 172.21.15.47:0/3316024996 conn(0x7f751c0cf7c0 msgr2=0x7f751c0706b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:38.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.905+0000 7f7522a00640 1 -- 172.21.15.47:0/3316024996 shutdown_connections 2024-02-19T15:30:38.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:38.905+0000 7f7522a00640 1 -- 172.21.15.47:0/3316024996 wait complete. 2024-02-19T15:30:39.127 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:39.127 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:40.128 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:40.223 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:40.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.269+0000 7f57d2800640 1 Processor -- start 2024-02-19T15:30:40.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.269+0000 7f57d2800640 1 -- start start 2024-02-19T15:30:40.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.269+0000 7f57d2800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f57cc15bcd0 0x7f57cc0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:40.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.269+0000 7f57d2800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f57cc0cedb0 0x7f57cc0cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:40.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.269+0000 7f57d2800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57cc0cc700 con 0x7f57cc0d2710 2024-02-19T15:30:40.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.269+0000 7f57d2800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57cc0cf5e0 con 0x7f57cc15bcd0 2024-02-19T15:30:40.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.269+0000 7f57d2800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57cc0cf300 con 0x7f57cc0cedb0 2024-02-19T15:30:40.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.269+0000 7f57cb400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f57cc15bcd0 0x7f57cc0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:40.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.269+0000 7f57cb400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f57cc15bcd0 0x7f57cc0cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57226/0 (socket says 172.21.15.47:57226) 2024-02-19T15:30:40.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.269+0000 7f57cb400640 1 -- 172.21.15.47:0/4067979582 learned_addr learned my addr 172.21.15.47:0/4067979582 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:40.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.269+0000 7f57cbe00640 1 --1- 172.21.15.47:0/4067979582 >> v1:172.21.15.47:6789/0 conn(0x7f57cc0d2710 0x7f57cc15d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51500/0 (socket says 172.21.15.47:51500) 2024-02-19T15:30:40.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.269+0000 7f57c3e00640 1 --2- 172.21.15.47:0/4067979582 >> v2:172.21.15.116:3300/0 conn(0x7f57cc0cedb0 0x7f57cc0cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:40.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.269+0000 7f57cb400640 1 -- 172.21.15.47:0/4067979582 >> v2:172.21.15.116:3300/0 conn(0x7f57cc0cedb0 msgr2=0x7f57cc0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:40.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.269+0000 7f57cb400640 1 --2- 172.21.15.47:0/4067979582 >> v2:172.21.15.116:3300/0 conn(0x7f57cc0cedb0 0x7f57cc0cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:40.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.269+0000 7f57cb400640 1 -- 172.21.15.47:0/4067979582 >> v1:172.21.15.47:6789/0 conn(0x7f57cc0d2710 legacy=0x7f57cc15d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:40.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.269+0000 7f57cb400640 1 -- 172.21.15.47:0/4067979582 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57cc0cc9a0 con 0x7f57cc15bcd0 2024-02-19T15:30:40.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.273+0000 7f57cb400640 1 --2- 172.21.15.47:0/4067979582 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57cc15bcd0 0x7f57cc0cd480 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f57cc0ce100 tx=0x7f57bc01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a30ae17fb29fe0c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:40.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.273+0000 7f57caa00640 1 -- 172.21.15.47:0/4067979582 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc0ab600 con 0x7f57cc15bcd0 2024-02-19T15:30:40.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.273+0000 7f57caa00640 1 -- 172.21.15.47:0/4067979582 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57bc0ab760 con 0x7f57cc15bcd0 2024-02-19T15:30:40.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.273+0000 7f57caa00640 1 -- 172.21.15.47:0/4067979582 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc0abaf0 con 0x7f57cc15bcd0 2024-02-19T15:30:40.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.273+0000 7f57d2800640 1 -- 172.21.15.47:0/4067979582 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57cc15bcd0 msgr2=0x7f57cc0cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:40.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.273+0000 7f57d2800640 1 --2- 172.21.15.47:0/4067979582 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57cc15bcd0 0x7f57cc0cd480 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f57cc0ce100 tx=0x7f57bc01c4f0 comp rx=0 tx=0).stop 2024-02-19T15:30:40.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.273+0000 7f57d2800640 1 -- 172.21.15.47:0/4067979582 shutdown_connections 2024-02-19T15:30:40.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.273+0000 7f57d2800640 1 --2- 172.21.15.47:0/4067979582 >> v2:172.21.15.116:3300/0 conn(0x7f57cc0cedb0 0x7f57cc0cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:40.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.273+0000 7f57d2800640 1 --2- 172.21.15.47:0/4067979582 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57cc15bcd0 0x7f57cc0cd480 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:40.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.273+0000 7f57d2800640 1 -- 172.21.15.47:0/4067979582 >> 172.21.15.47:0/4067979582 conn(0x7f57cc0d1bf0 msgr2=0x7f57cc0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:40.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.273+0000 7f57d2800640 1 -- 172.21.15.47:0/4067979582 shutdown_connections 2024-02-19T15:30:40.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.273+0000 7f57d2800640 1 -- 172.21.15.47:0/4067979582 wait complete. 2024-02-19T15:30:40.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57d2800640 1 Processor -- start 2024-02-19T15:30:40.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57d2800640 1 -- start start 2024-02-19T15:30:40.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57d2800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc0cedb0 0x7f57cc0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:40.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57d2800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57cc0d2710 0x7f57cc1536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:40.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57d2800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57cc15bcd0 0x7f57cc159ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:40.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57d2800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc159ac0 con 0x7f57cc15bcd0 2024-02-19T15:30:40.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57d2800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc0c7a80 con 0x7f57cc0cedb0 2024-02-19T15:30:40.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57d2800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc153be0 con 0x7f57cc0d2710 2024-02-19T15:30:40.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57cbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57cc15bcd0 0x7f57cc159ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:40.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57cbe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57cc15bcd0 0x7f57cc159ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57234/0 (socket says 172.21.15.47:57234) 2024-02-19T15:30:40.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57cbe00640 1 -- 172.21.15.47:0/919698627 learned_addr learned my addr 172.21.15.47:0/919698627 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:40.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57cb400640 1 --2- 172.21.15.47:0/919698627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc0cedb0 0x7f57cc0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:40.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57c3e00640 1 --2- 172.21.15.47:0/919698627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57cc0d2710 0x7f57cc1536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:40.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57cbe00640 1 -- 172.21.15.47:0/919698627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57cc0d2710 msgr2=0x7f57cc1536a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:40.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57cbe00640 1 --2- 172.21.15.47:0/919698627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57cc0d2710 0x7f57cc1536a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:40.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57cbe00640 1 -- 172.21.15.47:0/919698627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc0cedb0 msgr2=0x7f57cc0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:40.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57cbe00640 1 --2- 172.21.15.47:0/919698627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc0cedb0 0x7f57cc0c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:40.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57cbe00640 1 -- 172.21.15.47:0/919698627 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57bc0094d0 con 0x7f57cc15bcd0 2024-02-19T15:30:40.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57cb400640 1 --2- 172.21.15.47:0/919698627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc0cedb0 0x7f57cc0c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:30:40.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57c3e00640 1 --2- 172.21.15.47:0/919698627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57cc0d2710 0x7f57cc1536a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:30:40.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57cbe00640 1 --2- 172.21.15.47:0/919698627 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57cc15bcd0 0x7f57cc159ee0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f57c400c730 tx=0x7f57c400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:40.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57c3400640 1 -- 172.21.15.47:0/919698627 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c400ce40 con 0x7f57cc15bcd0 2024-02-19T15:30:40.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57c3400640 1 -- 172.21.15.47:0/919698627 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c400a640 con 0x7f57cc15bcd0 2024-02-19T15:30:40.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57d2800640 1 -- 172.21.15.47:0/919698627 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57cc0cdb40 con 0x7f57cc15bcd0 2024-02-19T15:30:40.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57c3400640 1 -- 172.21.15.47:0/919698627 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c400a9b0 con 0x7f57cc15bcd0 2024-02-19T15:30:40.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.277+0000 7f57d2800640 1 -- 172.21.15.47:0/919698627 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57cc1591b0 con 0x7f57cc15bcd0 2024-02-19T15:30:40.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.281+0000 7f57d2800640 1 -- 172.21.15.47:0/919698627 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57cc159c00 con 0x7f57cc15bcd0 2024-02-19T15:30:40.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.281+0000 7f57c3400640 1 -- 172.21.15.47:0/919698627 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f57c400ab10 con 0x7f57cc15bcd0 2024-02-19T15:30:40.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.281+0000 7f57c3400640 1 --2- 172.21.15.47:0/919698627 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f57a8066e40 0x7f57a8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:40.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.281+0000 7f57c3400640 1 -- 172.21.15.47:0/919698627 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f57c4084110 con 0x7f57cc15bcd0 2024-02-19T15:30:40.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.281+0000 7f57cb400640 1 --2- 172.21.15.47:0/919698627 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f57a8066e40 0x7f57a8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:40.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.285+0000 7f57cb400640 1 --2- 172.21.15.47:0/919698627 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f57a8066e40 0x7f57a8069300 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f57bc0af000 tx=0x7f57bc0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:40.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.289+0000 7f57c3400640 1 -- 172.21.15.47:0/919698627 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f57cc159c00 con 0x7f57cc15bcd0 2024-02-19T15:30:40.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.481+0000 7f57d2800640 1 -- 172.21.15.47:0/919698627 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57cc159c00 con 0x7f57cc15bcd0 2024-02-19T15:30:40.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.485+0000 7f57c3400640 1 -- 172.21.15.47:0/919698627 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f57c4017d80 con 0x7f57cc15bcd0 2024-02-19T15:30:40.484 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:40.484 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:40.484 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:40.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.485+0000 7f57d2800640 1 -- 172.21.15.47:0/919698627 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f57a8066e40 msgr2=0x7f57a8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:40.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.485+0000 7f57d2800640 1 --2- 172.21.15.47:0/919698627 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f57a8066e40 0x7f57a8069300 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f57bc0af000 tx=0x7f57bc0a7000 comp rx=0 tx=0).stop 2024-02-19T15:30:40.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.485+0000 7f57d2800640 1 -- 172.21.15.47:0/919698627 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57cc15bcd0 msgr2=0x7f57cc159ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:40.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.485+0000 7f57d2800640 1 --2- 172.21.15.47:0/919698627 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57cc15bcd0 0x7f57cc159ee0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f57c400c730 tx=0x7f57c400cbf0 comp rx=0 tx=0).stop 2024-02-19T15:30:40.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.485+0000 7f57d2800640 1 -- 172.21.15.47:0/919698627 shutdown_connections 2024-02-19T15:30:40.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.485+0000 7f57d2800640 1 --2- 172.21.15.47:0/919698627 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f57a8066e40 0x7f57a8069300 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:40.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.485+0000 7f57d2800640 1 --2- 172.21.15.47:0/919698627 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57cc15bcd0 0x7f57cc159ee0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:40.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.485+0000 7f57d2800640 1 --2- 172.21.15.47:0/919698627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57cc0d2710 0x7f57cc1536a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:40.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.485+0000 7f57d2800640 1 --2- 172.21.15.47:0/919698627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc0cedb0 0x7f57cc0c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:40.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.485+0000 7f57d2800640 1 -- 172.21.15.47:0/919698627 >> 172.21.15.47:0/919698627 conn(0x7f57cc0d1bf0 msgr2=0x7f57cc158910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:40.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.485+0000 7f57d2800640 1 -- 172.21.15.47:0/919698627 shutdown_connections 2024-02-19T15:30:40.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:40.485+0000 7f57d2800640 1 -- 172.21.15.47:0/919698627 wait complete. 2024-02-19T15:30:40.706 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:40.706 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:41.707 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:41.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.849+0000 7f3d0ea00640 1 Processor -- start 2024-02-19T15:30:41.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.849+0000 7f3d0ea00640 1 -- start start 2024-02-19T15:30:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.849+0000 7f3d0ea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d080cea80 0x7f3d080727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.849+0000 7f3d0ea00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3d0815c390 0x7f3d0815c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.849+0000 7f3d0ea00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d0815d280 con 0x7f3d08145590 2024-02-19T15:30:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.849+0000 7f3d0ea00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d08070ff0 con 0x7f3d0815c390 2024-02-19T15:30:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.849+0000 7f3d0ea00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0815cd60 con 0x7f3d080cea80 2024-02-19T15:30:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.849+0000 7f3d07e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f3d08145590 0x7f3d0806c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51512/0 (socket says 172.21.15.47:51512) 2024-02-19T15:30:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.849+0000 7f3d07e00640 1 -- 172.21.15.47:0/1312567405 learned_addr learned my addr 172.21.15.47:0/1312567405 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:41.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.849+0000 7f3d0ca00640 1 --2- 172.21.15.47:0/1312567405 >> v2:172.21.15.47:3300/0 conn(0x7f3d0815c390 0x7f3d0815c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:41.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.849+0000 7f3cff400640 1 --2- 172.21.15.47:0/1312567405 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d080cea80 0x7f3d080727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:41.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.849+0000 7f3d07400640 1 -- 172.21.15.47:0/1312567405 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025617647 0 0) 0x7f3d0815d280 con 0x7f3d08145590 2024-02-19T15:30:41.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.849+0000 7f3d07400640 1 -- 172.21.15.47:0/1312567405 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cec003530 con 0x7f3d08145590 2024-02-19T15:30:41.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.849+0000 7f3d0ca00640 1 -- 172.21.15.47:0/1312567405 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d080cea80 msgr2=0x7f3d080727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:41.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.849+0000 7f3d0ca00640 1 --2- 172.21.15.47:0/1312567405 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d080cea80 0x7f3d080727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:41.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.849+0000 7f3d0ca00640 1 -- 172.21.15.47:0/1312567405 >> v1:172.21.15.47:6789/0 conn(0x7f3d08145590 legacy=0x7f3d0806c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:41.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.849+0000 7f3d0ca00640 1 -- 172.21.15.47:0/1312567405 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d080709e0 con 0x7f3d0815c390 2024-02-19T15:30:41.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.853+0000 7f3d0ca00640 1 --2- 172.21.15.47:0/1312567405 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d0815c390 0x7f3d0815c820 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f3d000097c0 tx=0x7f3d0001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74f96f6af41fb629 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:41.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.853+0000 7f3d07400640 1 -- 172.21.15.47:0/1312567405 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d000ac410 con 0x7f3d0815c390 2024-02-19T15:30:41.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.853+0000 7f3d07400640 1 -- 172.21.15.47:0/1312567405 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d000ac570 con 0x7f3d0815c390 2024-02-19T15:30:41.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.853+0000 7f3d07400640 1 -- 172.21.15.47:0/1312567405 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d000ac900 con 0x7f3d0815c390 2024-02-19T15:30:41.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.853+0000 7f3d0ea00640 1 -- 172.21.15.47:0/1312567405 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d0815c390 msgr2=0x7f3d0815c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:41.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.853+0000 7f3d0ea00640 1 --2- 172.21.15.47:0/1312567405 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d0815c390 0x7f3d0815c820 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f3d000097c0 tx=0x7f3d0001c4a0 comp rx=0 tx=0).stop 2024-02-19T15:30:41.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.853+0000 7f3d0ea00640 1 -- 172.21.15.47:0/1312567405 shutdown_connections 2024-02-19T15:30:41.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.853+0000 7f3d0ea00640 1 --2- 172.21.15.47:0/1312567405 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d0815c390 0x7f3d0815c820 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:41.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.853+0000 7f3d0ea00640 1 --2- 172.21.15.47:0/1312567405 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d080cea80 0x7f3d080727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:41.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.853+0000 7f3d0ea00640 1 -- 172.21.15.47:0/1312567405 >> 172.21.15.47:0/1312567405 conn(0x7f3d080b16a0 msgr2=0x7f3d080b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:41.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.853+0000 7f3d0ea00640 1 -- 172.21.15.47:0/1312567405 shutdown_connections 2024-02-19T15:30:41.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.853+0000 7f3d0ea00640 1 -- 172.21.15.47:0/1312567405 wait complete. 2024-02-19T15:30:41.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d0ea00640 1 Processor -- start 2024-02-19T15:30:41.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d0ea00640 1 -- start start 2024-02-19T15:30:41.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d0ea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d080cea80 0x7f3d080a9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:41.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d0ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d08145590 0x7f3d0813a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:41.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d0ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d0815c390 0x7f3d081590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:41.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d0ea00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d080a9ed0 con 0x7f3d0815c390 2024-02-19T15:30:41.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d0ea00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d080aa170 con 0x7f3d08145590 2024-02-19T15:30:41.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d0ea00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d080aa450 con 0x7f3d080cea80 2024-02-19T15:30:41.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d0ca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d0815c390 0x7f3d081590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:41.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d0ca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d0815c390 0x7f3d081590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57246/0 (socket says 172.21.15.47:57246) 2024-02-19T15:30:41.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d0ca00640 1 -- 172.21.15.47:0/814431366 learned_addr learned my addr 172.21.15.47:0/814431366 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:41.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3cff400640 1 --2- 172.21.15.47:0/814431366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d08145590 0x7f3d0813a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:41.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d07e00640 1 --2- 172.21.15.47:0/814431366 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d080cea80 0x7f3d080a9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:41.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d0ca00640 1 -- 172.21.15.47:0/814431366 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d080cea80 msgr2=0x7f3d080a9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:41.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d0ca00640 1 --2- 172.21.15.47:0/814431366 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d080cea80 0x7f3d080a9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:41.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d0ca00640 1 -- 172.21.15.47:0/814431366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d08145590 msgr2=0x7f3d0813a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:41.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d0ca00640 1 --2- 172.21.15.47:0/814431366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d08145590 0x7f3d0813a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:41.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d0ca00640 1 -- 172.21.15.47:0/814431366 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d00009420 con 0x7f3d0815c390 2024-02-19T15:30:41.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d0ca00640 1 --2- 172.21.15.47:0/814431366 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d0815c390 0x7f3d081590a0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f3d0001a0d0 tx=0x7f3d00019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:41.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d04c00640 1 -- 172.21.15.47:0/814431366 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00006e20 con 0x7f3d0815c390 2024-02-19T15:30:41.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d04c00640 1 -- 172.21.15.47:0/814431366 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d000b5410 con 0x7f3d0815c390 2024-02-19T15:30:41.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d04c00640 1 -- 172.21.15.47:0/814431366 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d000b57c0 con 0x7f3d0815c390 2024-02-19T15:30:41.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d0ea00640 1 -- 172.21.15.47:0/814431366 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d08141120 con 0x7f3d0815c390 2024-02-19T15:30:41.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.857+0000 7f3d0ea00640 1 -- 172.21.15.47:0/814431366 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d081597b0 con 0x7f3d0815c390 2024-02-19T15:30:41.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.861+0000 7f3d0ea00640 1 -- 172.21.15.47:0/814431366 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d08075360 con 0x7f3d0815c390 2024-02-19T15:30:41.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.861+0000 7f3d04c00640 1 -- 172.21.15.47:0/814431366 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3d000b3040 con 0x7f3d0815c390 2024-02-19T15:30:41.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.861+0000 7f3d04c00640 1 --2- 172.21.15.47:0/814431366 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ce4066e40 0x7f3ce4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:41.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.861+0000 7f3d04c00640 1 -- 172.21.15.47:0/814431366 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3d0011e8a0 con 0x7f3d0815c390 2024-02-19T15:30:41.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.861+0000 7f3d07e00640 1 --2- 172.21.15.47:0/814431366 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ce4066e40 0x7f3ce4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:41.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.861+0000 7f3d07e00640 1 --2- 172.21.15.47:0/814431366 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ce4066e40 0x7f3ce4069300 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8005c10 tx=0x7f3cf8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:41.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:41.865+0000 7f3d04c00640 1 -- 172.21.15.47:0/814431366 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f3d000eb640 con 0x7f3d0815c390 2024-02-19T15:30:42.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:42.061+0000 7f3d0ea00640 1 -- 172.21.15.47:0/814431366 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d080b0600 con 0x7f3d0815c390 2024-02-19T15:30:42.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:42.061+0000 7f3d04c00640 1 -- 172.21.15.47:0/814431366 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f3d000b1d80 con 0x7f3d0815c390 2024-02-19T15:30:42.063 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:42.063 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:42.063 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:42.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:42.065+0000 7f3d0ea00640 1 -- 172.21.15.47:0/814431366 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ce4066e40 msgr2=0x7f3ce4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:42.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:42.065+0000 7f3d0ea00640 1 --2- 172.21.15.47:0/814431366 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ce4066e40 0x7f3ce4069300 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8005c10 tx=0x7f3cf8009040 comp rx=0 tx=0).stop 2024-02-19T15:30:42.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:42.065+0000 7f3d0ea00640 1 -- 172.21.15.47:0/814431366 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d0815c390 msgr2=0x7f3d081590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:42.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:42.065+0000 7f3d0ea00640 1 --2- 172.21.15.47:0/814431366 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d0815c390 0x7f3d081590a0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f3d0001a0d0 tx=0x7f3d00019240 comp rx=0 tx=0).stop 2024-02-19T15:30:42.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:42.065+0000 7f3d0ea00640 1 -- 172.21.15.47:0/814431366 shutdown_connections 2024-02-19T15:30:42.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:42.065+0000 7f3d0ea00640 1 --2- 172.21.15.47:0/814431366 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3ce4066e40 0x7f3ce4069300 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:42.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:42.065+0000 7f3d0ea00640 1 --2- 172.21.15.47:0/814431366 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d0815c390 0x7f3d081590a0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:42.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:42.065+0000 7f3d0ea00640 1 --2- 172.21.15.47:0/814431366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d08145590 0x7f3d0813a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:42.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:42.065+0000 7f3d0ea00640 1 --2- 172.21.15.47:0/814431366 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d080cea80 0x7f3d080a9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:42.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:42.065+0000 7f3d0ea00640 1 -- 172.21.15.47:0/814431366 >> 172.21.15.47:0/814431366 conn(0x7f3d080b16a0 msgr2=0x7f3d080af510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:42.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:42.065+0000 7f3d0ea00640 1 -- 172.21.15.47:0/814431366 shutdown_connections 2024-02-19T15:30:42.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:42.065+0000 7f3d0ea00640 1 -- 172.21.15.47:0/814431366 wait complete. 2024-02-19T15:30:42.223 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:42.286 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:42.286 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:43.287 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:43.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.429+0000 7f840e000640 1 Processor -- start 2024-02-19T15:30:43.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.429+0000 7f840e000640 1 -- start start 2024-02-19T15:30:43.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.429+0000 7f840e000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f84080b2500 0x7f840806ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:43.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.429+0000 7f840e000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840806b5d0 con 0x7f84081614e0 2024-02-19T15:30:43.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.429+0000 7f840e000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84080b1640 con 0x7f840815f620 2024-02-19T15:30:43.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.429+0000 7f840e000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84080b1c00 con 0x7f84080b2500 2024-02-19T15:30:43.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.429+0000 7f8407400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f84081614e0 0x7f840815eee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52964/0 (socket says 172.21.15.47:52964) 2024-02-19T15:30:43.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.429+0000 7f8407400640 1 -- 172.21.15.47:0/3145850046 learned_addr learned my addr 172.21.15.47:0/3145850046 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:43.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.433+0000 7f8407e00640 1 --2- 172.21.15.47:0/3145850046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f84080b2500 0x7f840806ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:43.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.433+0000 7f8406a00640 1 -- 172.21.15.47:0/3145850046 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081086375 0 0) 0x7f840806b5d0 con 0x7f84081614e0 2024-02-19T15:30:43.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.433+0000 7f8406a00640 1 -- 172.21.15.47:0/3145850046 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83ec003530 con 0x7f84081614e0 2024-02-19T15:30:43.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.433+0000 7f8406a00640 1 -- 172.21.15.47:0/3145850046 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397621534 0 0) 0x7f84080b1640 con 0x7f840815f620 2024-02-19T15:30:43.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.433+0000 7f8406a00640 1 -- 172.21.15.47:0/3145850046 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f840806b5d0 con 0x7f840815f620 2024-02-19T15:30:43.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.433+0000 7f8407e00640 1 -- 172.21.15.47:0/3145850046 >> v1:172.21.15.116:6789/0 conn(0x7f840815f620 legacy=0x7f84080b1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:43.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.433+0000 7f8407e00640 1 -- 172.21.15.47:0/3145850046 >> v1:172.21.15.47:6789/0 conn(0x7f84081614e0 legacy=0x7f840815eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:43.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.433+0000 7f8407e00640 1 -- 172.21.15.47:0/3145850046 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84080b1920 con 0x7f84080b2500 2024-02-19T15:30:43.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.433+0000 7f8407e00640 1 --2- 172.21.15.47:0/3145850046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f84080b2500 0x7f840806ba60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8400009dc0 tx=0x7f840001c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bef47ea04f8ec8a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:43.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.433+0000 7f8406a00640 1 -- 172.21.15.47:0/3145850046 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84000a5960 con 0x7f84080b2500 2024-02-19T15:30:43.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.433+0000 7f8406a00640 1 -- 172.21.15.47:0/3145850046 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84000a5ac0 con 0x7f84080b2500 2024-02-19T15:30:43.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.433+0000 7f8406a00640 1 -- 172.21.15.47:0/3145850046 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84000a5e50 con 0x7f84080b2500 2024-02-19T15:30:43.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f840e000640 1 -- 172.21.15.47:0/3145850046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f84080b2500 msgr2=0x7f840806ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:43.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f840e000640 1 --2- 172.21.15.47:0/3145850046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f84080b2500 0x7f840806ba60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8400009dc0 tx=0x7f840001c990 comp rx=0 tx=0).stop 2024-02-19T15:30:43.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f840e000640 1 -- 172.21.15.47:0/3145850046 shutdown_connections 2024-02-19T15:30:43.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f840e000640 1 --2- 172.21.15.47:0/3145850046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f84080b2500 0x7f840806ba60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:43.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f840e000640 1 -- 172.21.15.47:0/3145850046 >> 172.21.15.47:0/3145850046 conn(0x7f84080b4fc0 msgr2=0x7f84080b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:43.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f840e000640 1 -- 172.21.15.47:0/3145850046 shutdown_connections 2024-02-19T15:30:43.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f840e000640 1 -- 172.21.15.47:0/3145850046 wait complete. 2024-02-19T15:30:43.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f840e000640 1 Processor -- start 2024-02-19T15:30:43.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f840e000640 1 -- start start 2024-02-19T15:30:43.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f840e000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84080b2500 0x7f84080cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:43.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f840e000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f840815f620 0x7f8408157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:43.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f840e000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84081614e0 0x7f84081579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:43.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f840e000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8408157f00 con 0x7f84081614e0 2024-02-19T15:30:43.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f840e000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840809d700 con 0x7f84080b2500 2024-02-19T15:30:43.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f840e000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f840809d9e0 con 0x7f840815f620 2024-02-19T15:30:43.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f8407e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84081614e0 0x7f84081579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:43.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f8407e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84081614e0 0x7f84081579c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43782/0 (socket says 172.21.15.47:43782) 2024-02-19T15:30:43.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f8407e00640 1 -- 172.21.15.47:0/2734673454 learned_addr learned my addr 172.21.15.47:0/2734673454 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:43.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f83ffe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f840815f620 0x7f8408157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:43.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f8407400640 1 --2- 172.21.15.47:0/2734673454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84080b2500 0x7f84080cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:43.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f8407e00640 1 -- 172.21.15.47:0/2734673454 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f840815f620 msgr2=0x7f8408157480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:43.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f8407e00640 1 --2- 172.21.15.47:0/2734673454 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f840815f620 0x7f8408157480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:43.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f8407e00640 1 -- 172.21.15.47:0/2734673454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84080b2500 msgr2=0x7f84080cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:43.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.437+0000 7f8407e00640 1 --2- 172.21.15.47:0/2734673454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84080b2500 0x7f84080cb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:43.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.441+0000 7f8407e00640 1 -- 172.21.15.47:0/2734673454 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8400009a70 con 0x7f84081614e0 2024-02-19T15:30:43.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.441+0000 7f8407e00640 1 --2- 172.21.15.47:0/2734673454 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84081614e0 0x7f84081579c0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f84000195a0 tx=0x7f8400019730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:43.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.441+0000 7f83ff400640 1 -- 172.21.15.47:0/2734673454 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84000adcc0 con 0x7f84081614e0 2024-02-19T15:30:43.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.441+0000 7f83ff400640 1 -- 172.21.15.47:0/2734673454 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84000ade20 con 0x7f84081614e0 2024-02-19T15:30:43.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.441+0000 7f840e000640 1 -- 172.21.15.47:0/2734673454 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f840809dcc0 con 0x7f84081614e0 2024-02-19T15:30:43.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.441+0000 7f83ff400640 1 -- 172.21.15.47:0/2734673454 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84000b7650 con 0x7f84081614e0 2024-02-19T15:30:43.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.441+0000 7f840e000640 1 -- 172.21.15.47:0/2734673454 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84080cbc80 con 0x7f84081614e0 2024-02-19T15:30:43.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.441+0000 7f840e000640 1 -- 172.21.15.47:0/2734673454 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8408068320 con 0x7f84081614e0 2024-02-19T15:30:43.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.441+0000 7f83ff400640 1 -- 172.21.15.47:0/2734673454 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f84000b4050 con 0x7f84081614e0 2024-02-19T15:30:43.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.445+0000 7f83ff400640 1 --2- 172.21.15.47:0/2734673454 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f83e4066e40 0x7f83e4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:43.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.445+0000 7f83ff400640 1 -- 172.21.15.47:0/2734673454 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f840011f9c0 con 0x7f84081614e0 2024-02-19T15:30:43.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.445+0000 7f8407400640 1 --2- 172.21.15.47:0/2734673454 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f83e4066e40 0x7f83e4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:43.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.445+0000 7f8407400640 1 --2- 172.21.15.47:0/2734673454 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f83e4066e40 0x7f83e4069300 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f83f8007fd0 tx=0x7f83f8007d60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:43.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.449+0000 7f83ff400640 1 -- 172.21.15.47:0/2734673454 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f84000ec840 con 0x7f84081614e0 2024-02-19T15:30:43.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.641+0000 7f840e000640 1 -- 172.21.15.47:0/2734673454 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8408075360 con 0x7f84081614e0 2024-02-19T15:30:43.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.645+0000 7f83ff400640 1 -- 172.21.15.47:0/2734673454 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f84000b2020 con 0x7f84081614e0 2024-02-19T15:30:43.644 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:43.644 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:43.644 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:43.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.645+0000 7f840e000640 1 -- 172.21.15.47:0/2734673454 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f83e4066e40 msgr2=0x7f83e4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:43.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.645+0000 7f840e000640 1 --2- 172.21.15.47:0/2734673454 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f83e4066e40 0x7f83e4069300 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f83f8007fd0 tx=0x7f83f8007d60 comp rx=0 tx=0).stop 2024-02-19T15:30:43.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.645+0000 7f840e000640 1 -- 172.21.15.47:0/2734673454 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84081614e0 msgr2=0x7f84081579c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:43.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.645+0000 7f840e000640 1 --2- 172.21.15.47:0/2734673454 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84081614e0 0x7f84081579c0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f84000195a0 tx=0x7f8400019730 comp rx=0 tx=0).stop 2024-02-19T15:30:43.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.645+0000 7f840e000640 1 -- 172.21.15.47:0/2734673454 shutdown_connections 2024-02-19T15:30:43.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.645+0000 7f840e000640 1 --2- 172.21.15.47:0/2734673454 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f83e4066e40 0x7f83e4069300 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:43.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.645+0000 7f840e000640 1 --2- 172.21.15.47:0/2734673454 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84081614e0 0x7f84081579c0 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:43.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.645+0000 7f840e000640 1 --2- 172.21.15.47:0/2734673454 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f840815f620 0x7f8408157480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:43.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.645+0000 7f840e000640 1 --2- 172.21.15.47:0/2734673454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84080b2500 0x7f84080cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:43.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.645+0000 7f840e000640 1 -- 172.21.15.47:0/2734673454 >> 172.21.15.47:0/2734673454 conn(0x7f84080b4fc0 msgr2=0x7f84080af150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:43.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.645+0000 7f840e000640 1 -- 172.21.15.47:0/2734673454 shutdown_connections 2024-02-19T15:30:43.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:43.645+0000 7f840e000640 1 -- 172.21.15.47:0/2734673454 wait complete. 2024-02-19T15:30:43.867 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:43.867 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:44.225 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:44.867 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:45.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdc3200640 1 Processor -- start 2024-02-19T15:30:45.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdc3200640 1 -- start start 2024-02-19T15:30:45.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdc3200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdbc06d650 0x7efdbc0d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:45.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdc3200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efdbc0cded0 0x7efdbc0ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:45.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdc3200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdbc0cce30 con 0x7efdbc0ceb40 2024-02-19T15:30:45.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdc3200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdbc0cefd0 con 0x7efdbc0cded0 2024-02-19T15:30:45.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdc3200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc0cd110 con 0x7efdbc06d650 2024-02-19T15:30:45.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdc1200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7efdbc0ceb40 0x7efdbc161d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52976/0 (socket says 172.21.15.47:52976) 2024-02-19T15:30:45.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdc1200640 1 -- 172.21.15.47:0/3259139862 learned_addr learned my addr 172.21.15.47:0/3259139862 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:45.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdb3e00640 1 --2- 172.21.15.47:0/3259139862 >> v2:172.21.15.116:3300/0 conn(0x7efdbc0cded0 0x7efdbc0ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:45.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdbbe00640 1 --2- 172.21.15.47:0/3259139862 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdbc06d650 0x7efdbc0d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:45.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdbb400640 1 -- 172.21.15.47:0/3259139862 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679237285 0 0) 0x7efdbc0cce30 con 0x7efdbc0ceb40 2024-02-19T15:30:45.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdbb400640 1 -- 172.21.15.47:0/3259139862 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda0003530 con 0x7efdbc0ceb40 2024-02-19T15:30:45.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdbb400640 1 -- 172.21.15.47:0/3259139862 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1573966317 0 0) 0x7efda0003530 con 0x7efdbc0ceb40 2024-02-19T15:30:45.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdbb400640 1 -- 172.21.15.47:0/3259139862 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdbc06d650 msgr2=0x7efdbc0d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:45.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdbb400640 1 --2- 172.21.15.47:0/3259139862 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdbc06d650 0x7efdbc0d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:45.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdbb400640 1 -- 172.21.15.47:0/3259139862 >> v2:172.21.15.116:3300/0 conn(0x7efdbc0cded0 msgr2=0x7efdbc0ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:45.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdbb400640 1 --2- 172.21.15.47:0/3259139862 >> v2:172.21.15.116:3300/0 conn(0x7efdbc0cded0 0x7efdbc0ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:45.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdbb400640 1 -- 172.21.15.47:0/3259139862 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdbc0cd3f0 con 0x7efdbc0ceb40 2024-02-19T15:30:45.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdbb400640 1 -- 172.21.15.47:0/3259139862 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7efdb4001a00 con 0x7efdbc0ceb40 2024-02-19T15:30:45.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.009+0000 7efdbb400640 1 -- 172.21.15.47:0/3259139862 >> v1:172.21.15.47:6789/0 conn(0x7efdbc0ceb40 legacy=0x7efdbc161d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:45.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.013+0000 7efdbb400640 1 --2- 172.21.15.47:0/3259139862 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdbc06d650 0x7efda0003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:45.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.013+0000 7efdbb400640 1 --2- 172.21.15.47:0/3259139862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efda0003f00 0x7efda00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:45.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.013+0000 7efdbb400640 1 --2- 172.21.15.47:0/3259139862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efda0004d10 0x7efda0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:45.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.013+0000 7efdbb400640 1 -- 172.21.15.47:0/3259139862 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc0cefd0 con 0x7efda0004d10 2024-02-19T15:30:45.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.013+0000 7efdbb400640 1 -- 172.21.15.47:0/3259139862 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc0cd110 con 0x7efda0003f00 2024-02-19T15:30:45.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.013+0000 7efdbb400640 1 -- 172.21.15.47:0/3259139862 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda000b9e0 con 0x7efdbc06d650 2024-02-19T15:30:45.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.013+0000 7efdc1200640 1 --2- 172.21.15.47:0/3259139862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efda0004d10 0x7efda0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:45.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.013+0000 7efdb3e00640 1 --2- 172.21.15.47:0/3259139862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efda0003f00 0x7efda00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:45.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.013+0000 7efdbbe00640 1 --2- 172.21.15.47:0/3259139862 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdbc06d650 0x7efda0003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:45.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.013+0000 7efdc1200640 1 -- 172.21.15.47:0/3259139862 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdbc06d650 msgr2=0x7efda0003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:45.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.013+0000 7efdc1200640 1 --2- 172.21.15.47:0/3259139862 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdbc06d650 0x7efda0003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:45.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.013+0000 7efdc1200640 1 -- 172.21.15.47:0/3259139862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efda0003f00 msgr2=0x7efda00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:45.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.013+0000 7efdc1200640 1 --2- 172.21.15.47:0/3259139862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efda0003f00 0x7efda00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:45.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.013+0000 7efdc1200640 1 -- 172.21.15.47:0/3259139862 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda0003530 con 0x7efda0004d10 2024-02-19T15:30:45.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.013+0000 7efdb3e00640 1 --2- 172.21.15.47:0/3259139862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efda0003f00 0x7efda00047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:30:45.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.013+0000 7efdc1200640 1 --2- 172.21.15.47:0/3259139862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efda0004d10 0x7efda0005190 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7efdb400a890 tx=0x7efdb401fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3972ab7548e36eb1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:45.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.013+0000 7efdbb400640 1 -- 172.21.15.47:0/3259139862 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb40aa710 con 0x7efda0004d10 2024-02-19T15:30:45.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.013+0000 7efdbb400640 1 -- 172.21.15.47:0/3259139862 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb40aa870 con 0x7efda0004d10 2024-02-19T15:30:45.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.013+0000 7efdbb400640 1 -- 172.21.15.47:0/3259139862 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb40aaca0 con 0x7efda0004d10 2024-02-19T15:30:45.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc3200640 1 -- 172.21.15.47:0/3259139862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efda0004d10 msgr2=0x7efda0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:45.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc3200640 1 --2- 172.21.15.47:0/3259139862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efda0004d10 0x7efda0005190 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7efdb400a890 tx=0x7efdb401fae0 comp rx=0 tx=0).stop 2024-02-19T15:30:45.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc1200640 1 -- 172.21.15.47:0/3259139862 reap_dead start 2024-02-19T15:30:45.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc3200640 1 -- 172.21.15.47:0/3259139862 shutdown_connections 2024-02-19T15:30:45.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc3200640 1 -- 172.21.15.47:0/3259139862 >> 172.21.15.47:0/3259139862 conn(0x7efdbc0d1bf0 msgr2=0x7efdbc0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:45.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc3200640 1 -- 172.21.15.47:0/3259139862 shutdown_connections 2024-02-19T15:30:45.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc3200640 1 -- 172.21.15.47:0/3259139862 wait complete. 2024-02-19T15:30:45.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc3200640 1 Processor -- start 2024-02-19T15:30:45.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc3200640 1 -- start start 2024-02-19T15:30:45.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc3200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efda0004d10 0x7efdbc15fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:45.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc3200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdbc0c8cc0 0x7efdbc0ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:45.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc3200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdbc0c9140 0x7efdbc09f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:45.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc3200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc0c96a0 con 0x7efdbc0c9140 2024-02-19T15:30:45.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc3200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc0c9980 con 0x7efdbc0c8cc0 2024-02-19T15:30:45.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc3200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc0c9c60 con 0x7efda0004d10 2024-02-19T15:30:45.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc1200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdbc0c9140 0x7efdbc09f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:45.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc1200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdbc0c9140 0x7efdbc09f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43804/0 (socket says 172.21.15.47:43804) 2024-02-19T15:30:45.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc1200640 1 -- 172.21.15.47:0/2038247407 learned_addr learned my addr 172.21.15.47:0/2038247407 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:45.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdb3e00640 1 --2- 172.21.15.47:0/2038247407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdbc0c8cc0 0x7efdbc0ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:45.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdbbe00640 1 --2- 172.21.15.47:0/2038247407 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efda0004d10 0x7efdbc15fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:45.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc1200640 1 -- 172.21.15.47:0/2038247407 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efda0004d10 msgr2=0x7efdbc15fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:45.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc1200640 1 --2- 172.21.15.47:0/2038247407 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efda0004d10 0x7efdbc15fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:45.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc1200640 1 -- 172.21.15.47:0/2038247407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdbc0c8cc0 msgr2=0x7efdbc0ca3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:45.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc1200640 1 --2- 172.21.15.47:0/2038247407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdbc0c8cc0 0x7efdbc0ca3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:45.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.017+0000 7efdc1200640 1 -- 172.21.15.47:0/2038247407 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdbc09e890 con 0x7efdbc0c9140 2024-02-19T15:30:45.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.021+0000 7efdc1200640 1 --2- 172.21.15.47:0/2038247407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdbc0c9140 0x7efdbc09f090 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7efdb400a890 tx=0x7efda0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:45.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.021+0000 7efdb8c00640 1 -- 172.21.15.47:0/2038247407 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb40b8070 con 0x7efdbc0c9140 2024-02-19T15:30:45.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.021+0000 7efdb8c00640 1 -- 172.21.15.47:0/2038247407 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb4001fe0 con 0x7efdbc0c9140 2024-02-19T15:30:45.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.021+0000 7efdb8c00640 1 -- 172.21.15.47:0/2038247407 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb40aa510 con 0x7efdbc0c9140 2024-02-19T15:30:45.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.021+0000 7efdc3200640 1 -- 172.21.15.47:0/2038247407 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdbc09eb70 con 0x7efdbc0c9140 2024-02-19T15:30:45.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.021+0000 7efdc3200640 1 -- 172.21.15.47:0/2038247407 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdbc160290 con 0x7efdbc0c9140 2024-02-19T15:30:45.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.021+0000 7efdc3200640 1 -- 172.21.15.47:0/2038247407 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdbc09e6f0 con 0x7efdbc0c9140 2024-02-19T15:30:45.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.021+0000 7efdb8c00640 1 -- 172.21.15.47:0/2038247407 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efdb40017c0 con 0x7efdbc0c9140 2024-02-19T15:30:45.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.025+0000 7efdb8c00640 1 --2- 172.21.15.47:0/2038247407 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd98066e40 0x7efd98069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:45.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.025+0000 7efdb8c00640 1 -- 172.21.15.47:0/2038247407 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efdb4121200 con 0x7efdbc0c9140 2024-02-19T15:30:45.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.025+0000 7efdbbe00640 1 --2- 172.21.15.47:0/2038247407 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd98066e40 0x7efd98069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:45.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.025+0000 7efdbbe00640 1 --2- 172.21.15.47:0/2038247407 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd98066e40 0x7efd98069300 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7efdbc1f6cc0 tx=0x7efdac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:45.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.029+0000 7efdb8c00640 1 -- 172.21.15.47:0/2038247407 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7efdbc09e6f0 con 0x7efdbc0c9140 2024-02-19T15:30:45.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.225+0000 7efdc3200640 1 -- 172.21.15.47:0/2038247407 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdbc09e6f0 con 0x7efdbc0c9140 2024-02-19T15:30:45.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.225+0000 7efdb8c00640 1 -- 172.21.15.47:0/2038247407 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7efdbc09e6f0 con 0x7efdbc0c9140 2024-02-19T15:30:45.226 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:45.226 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:45.227 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:45.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.229+0000 7efdc3200640 1 -- 172.21.15.47:0/2038247407 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd98066e40 msgr2=0x7efd98069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:45.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.229+0000 7efdc3200640 1 --2- 172.21.15.47:0/2038247407 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd98066e40 0x7efd98069300 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7efdbc1f6cc0 tx=0x7efdac008040 comp rx=0 tx=0).stop 2024-02-19T15:30:45.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.229+0000 7efdc3200640 1 -- 172.21.15.47:0/2038247407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdbc0c9140 msgr2=0x7efdbc09f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:45.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.229+0000 7efdc3200640 1 --2- 172.21.15.47:0/2038247407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdbc0c9140 0x7efdbc09f090 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7efdb400a890 tx=0x7efda0005800 comp rx=0 tx=0).stop 2024-02-19T15:30:45.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.229+0000 7efdc3200640 1 -- 172.21.15.47:0/2038247407 shutdown_connections 2024-02-19T15:30:45.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.229+0000 7efdc3200640 1 --2- 172.21.15.47:0/2038247407 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efd98066e40 0x7efd98069300 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:45.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.229+0000 7efdc3200640 1 --2- 172.21.15.47:0/2038247407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efdbc0c9140 0x7efdbc09f090 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:45.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.229+0000 7efdc3200640 1 --2- 172.21.15.47:0/2038247407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdbc0c8cc0 0x7efdbc0ca3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:45.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.229+0000 7efdc3200640 1 --2- 172.21.15.47:0/2038247407 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efda0004d10 0x7efdbc15fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:45.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.229+0000 7efdc3200640 1 -- 172.21.15.47:0/2038247407 >> 172.21.15.47:0/2038247407 conn(0x7efdbc0d1bf0 msgr2=0x7efdbc0b7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:45.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.229+0000 7efdc3200640 1 -- 172.21.15.47:0/2038247407 shutdown_connections 2024-02-19T15:30:45.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:45.229+0000 7efdc3200640 1 -- 172.21.15.47:0/2038247407 wait complete. 2024-02-19T15:30:45.449 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:45.450 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:46.227 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:46.451 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:46.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.589+0000 7fc729a00640 1 Processor -- start 2024-02-19T15:30:46.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.593+0000 7fc729a00640 1 -- start start 2024-02-19T15:30:46.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.593+0000 7fc729a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7240cded0 0x7fc7240ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:46.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.593+0000 7fc729a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7240cce30 con 0x7fc7240ceb40 2024-02-19T15:30:46.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.593+0000 7fc729a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7240cd110 con 0x7fc72406d650 2024-02-19T15:30:46.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.593+0000 7fc729a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7240cefd0 con 0x7fc7240cded0 2024-02-19T15:30:46.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.593+0000 7fc723400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc7240ceb40 0x7fc72406bd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52982/0 (socket says 172.21.15.47:52982) 2024-02-19T15:30:46.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.593+0000 7fc723400640 1 -- 172.21.15.47:0/1293443974 learned_addr learned my addr 172.21.15.47:0/1293443974 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:46.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.593+0000 7fc71a000640 1 --2- 172.21.15.47:0/1293443974 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7240cded0 0x7fc7240ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:46.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.593+0000 7fc722000640 1 -- 172.21.15.47:0/1293443974 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346507715 0 0) 0x7fc7240cce30 con 0x7fc7240ceb40 2024-02-19T15:30:46.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.593+0000 7fc722000640 1 -- 172.21.15.47:0/1293443974 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc708003530 con 0x7fc7240ceb40 2024-02-19T15:30:46.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.593+0000 7fc722000640 1 -- 172.21.15.47:0/1293443974 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217091148 0 0) 0x7fc7240cd110 con 0x7fc72406d650 2024-02-19T15:30:46.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.593+0000 7fc722000640 1 -- 172.21.15.47:0/1293443974 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7240cce30 con 0x7fc72406d650 2024-02-19T15:30:46.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.593+0000 7fc71a000640 1 -- 172.21.15.47:0/1293443974 >> v1:172.21.15.116:6789/0 conn(0x7fc72406d650 legacy=0x7fc7240d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:46.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.593+0000 7fc71a000640 1 -- 172.21.15.47:0/1293443974 >> v1:172.21.15.47:6789/0 conn(0x7fc7240ceb40 legacy=0x7fc72406bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:46.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.593+0000 7fc71a000640 1 -- 172.21.15.47:0/1293443974 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7240cd3f0 con 0x7fc7240cded0 2024-02-19T15:30:46.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.597+0000 7fc71a000640 1 --2- 172.21.15.47:0/1293443974 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7240cded0 0x7fc7240ce600 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0097f0 tx=0x7fc70c01c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=decc81460d47e6e8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:46.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.597+0000 7fc722000640 1 -- 172.21.15.47:0/1293443974 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c0ab620 con 0x7fc7240cded0 2024-02-19T15:30:46.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.597+0000 7fc722000640 1 -- 172.21.15.47:0/1293443974 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70c0ab780 con 0x7fc7240cded0 2024-02-19T15:30:46.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.597+0000 7fc722000640 1 -- 172.21.15.47:0/1293443974 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c0abb10 con 0x7fc7240cded0 2024-02-19T15:30:46.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.597+0000 7fc729a00640 1 -- 172.21.15.47:0/1293443974 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7240cded0 msgr2=0x7fc7240ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:46.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.597+0000 7fc729a00640 1 --2- 172.21.15.47:0/1293443974 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7240cded0 0x7fc7240ce600 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0097f0 tx=0x7fc70c01c420 comp rx=0 tx=0).stop 2024-02-19T15:30:46.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.597+0000 7fc729a00640 1 -- 172.21.15.47:0/1293443974 shutdown_connections 2024-02-19T15:30:46.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.597+0000 7fc729a00640 1 --2- 172.21.15.47:0/1293443974 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7240cded0 0x7fc7240ce600 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:46.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.597+0000 7fc729a00640 1 -- 172.21.15.47:0/1293443974 >> 172.21.15.47:0/1293443974 conn(0x7fc7240d1bf0 msgr2=0x7fc7240d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:46.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.597+0000 7fc729a00640 1 -- 172.21.15.47:0/1293443974 shutdown_connections 2024-02-19T15:30:46.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.597+0000 7fc729a00640 1 -- 172.21.15.47:0/1293443974 wait complete. 2024-02-19T15:30:46.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.597+0000 7fc729a00640 1 Processor -- start 2024-02-19T15:30:46.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.597+0000 7fc729a00640 1 -- start start 2024-02-19T15:30:46.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc729a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc72406d650 0x7fc724159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:46.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc729a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7240cded0 0x7fc724159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:46.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc729a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7240ceb40 0x7fc72406f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:46.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc729a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72415fe40 con 0x7fc7240cded0 2024-02-19T15:30:46.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc729a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72415f1c0 con 0x7fc72406d650 2024-02-19T15:30:46.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc729a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72415f4a0 con 0x7fc7240ceb40 2024-02-19T15:30:46.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc71a000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7240cded0 0x7fc724159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:46.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc71a000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7240cded0 0x7fc724159ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43818/0 (socket says 172.21.15.47:43818) 2024-02-19T15:30:46.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc71a000640 1 -- 172.21.15.47:0/390546207 learned_addr learned my addr 172.21.15.47:0/390546207 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:46.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc723400640 1 --2- 172.21.15.47:0/390546207 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7240ceb40 0x7fc72406f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:46.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc722a00640 1 --2- 172.21.15.47:0/390546207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc72406d650 0x7fc724159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:46.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc71a000640 1 -- 172.21.15.47:0/390546207 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7240ceb40 msgr2=0x7fc72406f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:46.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc71a000640 1 --2- 172.21.15.47:0/390546207 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7240ceb40 0x7fc72406f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:46.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc71a000640 1 -- 172.21.15.47:0/390546207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc72406d650 msgr2=0x7fc724159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:46.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc71a000640 1 --2- 172.21.15.47:0/390546207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc72406d650 0x7fc724159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:46.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc71a000640 1 -- 172.21.15.47:0/390546207 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70c009420 con 0x7fc7240cded0 2024-02-19T15:30:46.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc71a000640 1 --2- 172.21.15.47:0/390546207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7240cded0 0x7fc724159ef0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0ad040 tx=0x7fc70c01ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:46.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc71b400640 1 -- 172.21.15.47:0/390546207 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c0abdc0 con 0x7fc7240cded0 2024-02-19T15:30:46.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc71b400640 1 -- 172.21.15.47:0/390546207 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70c0bf410 con 0x7fc7240cded0 2024-02-19T15:30:46.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc729a00640 1 -- 172.21.15.47:0/390546207 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72415f780 con 0x7fc7240cded0 2024-02-19T15:30:46.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc71b400640 1 -- 172.21.15.47:0/390546207 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c0bf7c0 con 0x7fc7240cded0 2024-02-19T15:30:46.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc729a00640 1 -- 172.21.15.47:0/390546207 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc724159b80 con 0x7fc7240cded0 2024-02-19T15:30:46.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.601+0000 7fc729a00640 1 -- 172.21.15.47:0/390546207 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc72415ff80 con 0x7fc7240cded0 2024-02-19T15:30:46.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.605+0000 7fc71b400640 1 -- 172.21.15.47:0/390546207 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc70c0bf920 con 0x7fc7240cded0 2024-02-19T15:30:46.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.605+0000 7fc71b400640 1 --2- 172.21.15.47:0/390546207 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc700066e40 0x7fc700069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:46.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.605+0000 7fc71b400640 1 -- 172.21.15.47:0/390546207 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc70c11f220 con 0x7fc7240cded0 2024-02-19T15:30:46.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.605+0000 7fc722a00640 1 --2- 172.21.15.47:0/390546207 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc700066e40 0x7fc700069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:46.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.605+0000 7fc722a00640 1 --2- 172.21.15.47:0/390546207 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc700066e40 0x7fc700069300 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc7140011f0 tx=0x7fc714009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:46.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.613+0000 7fc71b400640 1 -- 172.21.15.47:0/390546207 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc72415ff80 con 0x7fc7240cded0 2024-02-19T15:30:46.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.813+0000 7fc729a00640 1 -- 172.21.15.47:0/390546207 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc72415ff80 con 0x7fc7240cded0 2024-02-19T15:30:46.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.813+0000 7fc71b400640 1 -- 172.21.15.47:0/390546207 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7fc72415ff80 con 0x7fc7240cded0 2024-02-19T15:30:46.813 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:46.814 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:46.814 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:46.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.817+0000 7fc729a00640 1 -- 172.21.15.47:0/390546207 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc700066e40 msgr2=0x7fc700069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:46.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.817+0000 7fc729a00640 1 --2- 172.21.15.47:0/390546207 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc700066e40 0x7fc700069300 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc7140011f0 tx=0x7fc714009040 comp rx=0 tx=0).stop 2024-02-19T15:30:46.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.817+0000 7fc729a00640 1 -- 172.21.15.47:0/390546207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7240cded0 msgr2=0x7fc724159ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:46.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.817+0000 7fc729a00640 1 --2- 172.21.15.47:0/390546207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7240cded0 0x7fc724159ef0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0ad040 tx=0x7fc70c01ac60 comp rx=0 tx=0).stop 2024-02-19T15:30:46.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.817+0000 7fc729a00640 1 -- 172.21.15.47:0/390546207 shutdown_connections 2024-02-19T15:30:46.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.817+0000 7fc729a00640 1 --2- 172.21.15.47:0/390546207 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc700066e40 0x7fc700069300 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:46.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.817+0000 7fc729a00640 1 --2- 172.21.15.47:0/390546207 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc7240ceb40 0x7fc72406f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:46.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.817+0000 7fc729a00640 1 --2- 172.21.15.47:0/390546207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7240cded0 0x7fc724159ef0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:46.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.817+0000 7fc729a00640 1 --2- 172.21.15.47:0/390546207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc72406d650 0x7fc724159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:46.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.817+0000 7fc729a00640 1 -- 172.21.15.47:0/390546207 >> 172.21.15.47:0/390546207 conn(0x7fc7240d1bf0 msgr2=0x7fc72415d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:46.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.817+0000 7fc729a00640 1 -- 172.21.15.47:0/390546207 shutdown_connections 2024-02-19T15:30:46.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:46.817+0000 7fc729a00640 1 -- 172.21.15.47:0/390546207 wait complete. 2024-02-19T15:30:47.036 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:47.036 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:48.038 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:48.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.177+0000 7f4a46800640 1 Processor -- start 2024-02-19T15:30:48.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.177+0000 7f4a46800640 1 -- start start 2024-02-19T15:30:48.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.181+0000 7f4a46800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a400cded0 0x7f4a400ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:48.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.181+0000 7f4a46800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4a400ceb40 0x7f4a4006bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:48.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.181+0000 7f4a46800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a400cce30 con 0x7f4a4006d650 2024-02-19T15:30:48.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.181+0000 7f4a46800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a400cefd0 con 0x7f4a400ceb40 2024-02-19T15:30:48.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.181+0000 7f4a46800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a400cd110 con 0x7f4a400cded0 2024-02-19T15:30:48.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.181+0000 7f4a3f400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f4a4006d650 0x7f4a400d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52984/0 (socket says 172.21.15.47:52984) 2024-02-19T15:30:48.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.181+0000 7f4a3f400640 1 -- 172.21.15.47:0/2701045091 learned_addr learned my addr 172.21.15.47:0/2701045091 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:48.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.181+0000 7f4a37e00640 1 --2- 172.21.15.47:0/2701045091 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a400cded0 0x7f4a400ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:48.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.181+0000 7f4a3fe00640 1 --2- 172.21.15.47:0/2701045091 >> v2:172.21.15.116:3300/0 conn(0x7f4a400ceb40 0x7f4a4006bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:48.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.181+0000 7f4a3ea00640 1 -- 172.21.15.47:0/2701045091 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885034895 0 0) 0x7f4a400cce30 con 0x7f4a4006d650 2024-02-19T15:30:48.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.181+0000 7f4a3ea00640 1 -- 172.21.15.47:0/2701045091 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a24003530 con 0x7f4a4006d650 2024-02-19T15:30:48.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.181+0000 7f4a37e00640 1 -- 172.21.15.47:0/2701045091 >> v2:172.21.15.116:3300/0 conn(0x7f4a400ceb40 msgr2=0x7f4a4006bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:48.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.181+0000 7f4a37e00640 1 --2- 172.21.15.47:0/2701045091 >> v2:172.21.15.116:3300/0 conn(0x7f4a400ceb40 0x7f4a4006bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:48.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.181+0000 7f4a37e00640 1 -- 172.21.15.47:0/2701045091 >> v1:172.21.15.47:6789/0 conn(0x7f4a4006d650 legacy=0x7f4a400d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:48.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.181+0000 7f4a37e00640 1 -- 172.21.15.47:0/2701045091 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a400cd3f0 con 0x7f4a400cded0 2024-02-19T15:30:48.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.181+0000 7f4a3fe00640 1 --2- 172.21.15.47:0/2701045091 >> v2:172.21.15.116:3300/0 conn(0x7f4a400ceb40 0x7f4a4006bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:30:48.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.185+0000 7f4a37e00640 1 --2- 172.21.15.47:0/2701045091 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a400cded0 0x7f4a400ce600 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4a280097c0 tx=0x7f4a2801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68e16739a8fe043 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:48.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.185+0000 7f4a3ea00640 1 -- 172.21.15.47:0/2701045091 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a280ac640 con 0x7f4a400cded0 2024-02-19T15:30:48.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.185+0000 7f4a3ea00640 1 -- 172.21.15.47:0/2701045091 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a280ac7a0 con 0x7f4a400cded0 2024-02-19T15:30:48.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.185+0000 7f4a3ea00640 1 -- 172.21.15.47:0/2701045091 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a280acb30 con 0x7f4a400cded0 2024-02-19T15:30:48.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.185+0000 7f4a46800640 1 -- 172.21.15.47:0/2701045091 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a400cded0 msgr2=0x7f4a400ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:48.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.185+0000 7f4a46800640 1 --2- 172.21.15.47:0/2701045091 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a400cded0 0x7f4a400ce600 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4a280097c0 tx=0x7f4a2801c3e0 comp rx=0 tx=0).stop 2024-02-19T15:30:48.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.185+0000 7f4a46800640 1 -- 172.21.15.47:0/2701045091 shutdown_connections 2024-02-19T15:30:48.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.185+0000 7f4a46800640 1 --2- 172.21.15.47:0/2701045091 >> v2:172.21.15.116:3300/0 conn(0x7f4a400ceb40 0x7f4a4006bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:48.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.185+0000 7f4a46800640 1 --2- 172.21.15.47:0/2701045091 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a400cded0 0x7f4a400ce600 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:48.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.185+0000 7f4a46800640 1 -- 172.21.15.47:0/2701045091 >> 172.21.15.47:0/2701045091 conn(0x7f4a400d1bf0 msgr2=0x7f4a400d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:48.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.185+0000 7f4a46800640 1 -- 172.21.15.47:0/2701045091 shutdown_connections 2024-02-19T15:30:48.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.185+0000 7f4a46800640 1 -- 172.21.15.47:0/2701045091 wait complete. 2024-02-19T15:30:48.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.185+0000 7f4a46800640 1 Processor -- start 2024-02-19T15:30:48.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.185+0000 7f4a46800640 1 -- start start 2024-02-19T15:30:48.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a46800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a4006d650 0x7f4a4015d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:48.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a46800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4a400cded0 0x7f4a400c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:48.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a46800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a400ceb40 0x7f4a4009b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:48.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a46800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4015da70 con 0x7f4a400cded0 2024-02-19T15:30:48.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a46800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a400c9b60 con 0x7f4a400ceb40 2024-02-19T15:30:48.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a46800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a400c95a0 con 0x7f4a4006d650 2024-02-19T15:30:48.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a37e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4a400cded0 0x7f4a400c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:48.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a37e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4a400cded0 0x7f4a400c7e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43820/0 (socket says 172.21.15.47:43820) 2024-02-19T15:30:48.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a37e00640 1 -- 172.21.15.47:0/4255350254 learned_addr learned my addr 172.21.15.47:0/4255350254 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:48.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a3fe00640 1 --2- 172.21.15.47:0/4255350254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a400ceb40 0x7f4a4009b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:48.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a3f400640 1 --2- 172.21.15.47:0/4255350254 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a4006d650 0x7f4a4015d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:48.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a37e00640 1 -- 172.21.15.47:0/4255350254 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a4006d650 msgr2=0x7f4a4015d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:48.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a37e00640 1 --2- 172.21.15.47:0/4255350254 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a4006d650 0x7f4a4015d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:48.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a37e00640 1 -- 172.21.15.47:0/4255350254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a400ceb40 msgr2=0x7f4a4009b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:48.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a37e00640 1 --2- 172.21.15.47:0/4255350254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a400ceb40 0x7f4a4009b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:48.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a37e00640 1 -- 172.21.15.47:0/4255350254 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a38009a70 con 0x7f4a400cded0 2024-02-19T15:30:48.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a37e00640 1 --2- 172.21.15.47:0/4255350254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4a400cded0 0x7f4a400c7e60 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f4a2801aa90 tx=0x7f4a2801ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:48.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a37400640 1 -- 172.21.15.47:0/4255350254 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a280b7650 con 0x7f4a400cded0 2024-02-19T15:30:48.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a37400640 1 -- 172.21.15.47:0/4255350254 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a280b77b0 con 0x7f4a400cded0 2024-02-19T15:30:48.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a37400640 1 -- 172.21.15.47:0/4255350254 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a280b7b60 con 0x7f4a400cded0 2024-02-19T15:30:48.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a46800640 1 -- 172.21.15.47:0/4255350254 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a28009420 con 0x7f4a400cded0 2024-02-19T15:30:48.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a46800640 1 -- 172.21.15.47:0/4255350254 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a400c7660 con 0x7f4a400cded0 2024-02-19T15:30:48.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.189+0000 7f4a46800640 1 -- 172.21.15.47:0/4255350254 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a40075360 con 0x7f4a400cded0 2024-02-19T15:30:48.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.193+0000 7f4a37400640 1 -- 172.21.15.47:0/4255350254 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4a280b3070 con 0x7f4a400cded0 2024-02-19T15:30:48.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.193+0000 7f4a37400640 1 --2- 172.21.15.47:0/4255350254 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4a1c066e40 0x7f4a1c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:48.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.193+0000 7f4a37400640 1 -- 172.21.15.47:0/4255350254 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4a2811f4b0 con 0x7f4a400cded0 2024-02-19T15:30:48.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.193+0000 7f4a3f400640 1 --2- 172.21.15.47:0/4255350254 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4a1c066e40 0x7f4a1c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:48.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.193+0000 7f4a3f400640 1 --2- 172.21.15.47:0/4255350254 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4a1c066e40 0x7f4a1c069300 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f4a30005890 tx=0x7f4a30009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:48.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.197+0000 7f4a37400640 1 -- 172.21.15.47:0/4255350254 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f4a280ec370 con 0x7f4a400cded0 2024-02-19T15:30:48.229 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:48.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.505+0000 7f4a46800640 1 -- 172.21.15.47:0/4255350254 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a400708f0 con 0x7f4a400cded0 2024-02-19T15:30:48.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.505+0000 7f4a37400640 1 -- 172.21.15.47:0/4255350254 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f4a280b1d80 con 0x7f4a400cded0 2024-02-19T15:30:48.505 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:48.505 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:48.505 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:48.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.509+0000 7f4a46800640 1 -- 172.21.15.47:0/4255350254 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4a1c066e40 msgr2=0x7f4a1c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:48.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.509+0000 7f4a46800640 1 --2- 172.21.15.47:0/4255350254 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4a1c066e40 0x7f4a1c069300 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f4a30005890 tx=0x7f4a30009040 comp rx=0 tx=0).stop 2024-02-19T15:30:48.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.509+0000 7f4a46800640 1 -- 172.21.15.47:0/4255350254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4a400cded0 msgr2=0x7f4a400c7e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:48.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.509+0000 7f4a46800640 1 --2- 172.21.15.47:0/4255350254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4a400cded0 0x7f4a400c7e60 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f4a2801aa90 tx=0x7f4a2801ac20 comp rx=0 tx=0).stop 2024-02-19T15:30:48.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.509+0000 7f4a46800640 1 -- 172.21.15.47:0/4255350254 shutdown_connections 2024-02-19T15:30:48.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.509+0000 7f4a46800640 1 --2- 172.21.15.47:0/4255350254 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4a1c066e40 0x7f4a1c069300 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:48.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.509+0000 7f4a46800640 1 --2- 172.21.15.47:0/4255350254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a400ceb40 0x7f4a4009b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:48.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.509+0000 7f4a46800640 1 --2- 172.21.15.47:0/4255350254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4a400cded0 0x7f4a400c7e60 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:48.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.509+0000 7f4a46800640 1 --2- 172.21.15.47:0/4255350254 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a4006d650 0x7f4a4015d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:48.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.509+0000 7f4a46800640 1 -- 172.21.15.47:0/4255350254 >> 172.21.15.47:0/4255350254 conn(0x7f4a400d1bf0 msgr2=0x7f4a400b5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:48.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.509+0000 7f4a46800640 1 -- 172.21.15.47:0/4255350254 shutdown_connections 2024-02-19T15:30:48.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:48.509+0000 7f4a46800640 1 -- 172.21.15.47:0/4255350254 wait complete. 2024-02-19T15:30:48.728 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:48.728 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:49.729 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:49.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.869+0000 7f54b0000640 1 Processor -- start 2024-02-19T15:30:49.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.869+0000 7f54b0000640 1 -- start start 2024-02-19T15:30:49.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.869+0000 7f54b0000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54a80b3a20 0x7f54a80cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:49.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.869+0000 7f54b0000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a806d780 con 0x7f54a80ccc50 2024-02-19T15:30:49.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.869+0000 7f54b0000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a80cd0e0 con 0x7f54a80cee30 2024-02-19T15:30:49.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.869+0000 7f54b0000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a806d9d0 con 0x7f54a80b3a20 2024-02-19T15:30:49.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.869+0000 7f54ae000640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f54a80ccc50 0x7f54a8071a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52988/0 (socket says 172.21.15.47:52988) 2024-02-19T15:30:49.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.869+0000 7f54ae000640 1 -- 172.21.15.47:0/2296916113 learned_addr learned my addr 172.21.15.47:0/2296916113 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:49.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.869+0000 7f54acc00640 1 --2- 172.21.15.47:0/2296916113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54a80b3a20 0x7f54a80cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:49.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.869+0000 7f54a7e00640 1 -- 172.21.15.47:0/2296916113 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316059255 0 0) 0x7f54a806d780 con 0x7f54a80ccc50 2024-02-19T15:30:49.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.869+0000 7f54a7e00640 1 -- 172.21.15.47:0/2296916113 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5490003530 con 0x7f54a80ccc50 2024-02-19T15:30:49.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.869+0000 7f54a7e00640 1 -- 172.21.15.47:0/2296916113 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591830686 0 0) 0x7f54a80cd0e0 con 0x7f54a80cee30 2024-02-19T15:30:49.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.869+0000 7f54a7e00640 1 -- 172.21.15.47:0/2296916113 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54a806d780 con 0x7f54a80cee30 2024-02-19T15:30:49.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.869+0000 7f54acc00640 1 -- 172.21.15.47:0/2296916113 >> v1:172.21.15.116:6789/0 conn(0x7f54a80cee30 legacy=0x7f54a80cf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:49.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.869+0000 7f54acc00640 1 -- 172.21.15.47:0/2296916113 >> v1:172.21.15.47:6789/0 conn(0x7f54a80ccc50 legacy=0x7f54a8071a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:49.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.869+0000 7f54acc00640 1 -- 172.21.15.47:0/2296916113 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a806dcb0 con 0x7f54a80b3a20 2024-02-19T15:30:49.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.873+0000 7f54acc00640 1 --2- 172.21.15.47:0/2296916113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54a80b3a20 0x7f54a80cc710 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5494009820 tx=0x7f549401c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6845aa1fca5e13c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:49.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.873+0000 7f54a7e00640 1 -- 172.21.15.47:0/2296916113 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54940ac640 con 0x7f54a80b3a20 2024-02-19T15:30:49.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.873+0000 7f54a7e00640 1 -- 172.21.15.47:0/2296916113 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54940ac7a0 con 0x7f54a80b3a20 2024-02-19T15:30:49.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.873+0000 7f54a7e00640 1 -- 172.21.15.47:0/2296916113 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54940acb30 con 0x7f54a80b3a20 2024-02-19T15:30:49.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.873+0000 7f54b0000640 1 -- 172.21.15.47:0/2296916113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54a80b3a20 msgr2=0x7f54a80cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:49.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.873+0000 7f54b0000640 1 --2- 172.21.15.47:0/2296916113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54a80b3a20 0x7f54a80cc710 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5494009820 tx=0x7f549401c4c0 comp rx=0 tx=0).stop 2024-02-19T15:30:49.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.873+0000 7f54b0000640 1 -- 172.21.15.47:0/2296916113 shutdown_connections 2024-02-19T15:30:49.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.873+0000 7f54b0000640 1 --2- 172.21.15.47:0/2296916113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54a80b3a20 0x7f54a80cc710 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:49.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.873+0000 7f54b0000640 1 -- 172.21.15.47:0/2296916113 >> 172.21.15.47:0/2296916113 conn(0x7f54a80d0a50 msgr2=0x7f54a80d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:49.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.873+0000 7f54b0000640 1 -- 172.21.15.47:0/2296916113 shutdown_connections 2024-02-19T15:30:49.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.873+0000 7f54b0000640 1 -- 172.21.15.47:0/2296916113 wait complete. 2024-02-19T15:30:49.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54b0000640 1 Processor -- start 2024-02-19T15:30:49.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54b0000640 1 -- start start 2024-02-19T15:30:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54b0000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54a80b3a20 0x7f54a8142770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54b0000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54a80ccc50 0x7f54a813c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54b0000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54a80cee30 0x7f54a8140950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54b0000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a81416d0 con 0x7f54a80b3a20 2024-02-19T15:30:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54b0000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a81419b0 con 0x7f54a80cee30 2024-02-19T15:30:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54b0000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a8141c90 con 0x7f54a80ccc50 2024-02-19T15:30:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54ad600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54a80b3a20 0x7f54a8142770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54ad600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54a80b3a20 0x7f54a8142770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43822/0 (socket says 172.21.15.47:43822) 2024-02-19T15:30:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54ad600640 1 -- 172.21.15.47:0/2854201402 learned_addr learned my addr 172.21.15.47:0/2854201402 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:49.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54acc00640 1 --2- 172.21.15.47:0/2854201402 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54a80ccc50 0x7f54a813c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:49.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54ae000640 1 --2- 172.21.15.47:0/2854201402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54a80cee30 0x7f54a8140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:49.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54ad600640 1 -- 172.21.15.47:0/2854201402 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54a80ccc50 msgr2=0x7f54a813c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:49.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54ad600640 1 --2- 172.21.15.47:0/2854201402 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54a80ccc50 0x7f54a813c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:49.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54ad600640 1 -- 172.21.15.47:0/2854201402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54a80cee30 msgr2=0x7f54a8140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:49.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54ad600640 1 --2- 172.21.15.47:0/2854201402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54a80cee30 0x7f54a8140950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:49.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54ad600640 1 -- 172.21.15.47:0/2854201402 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5494009420 con 0x7f54a80b3a20 2024-02-19T15:30:49.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54ad600640 1 --2- 172.21.15.47:0/2854201402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54a80b3a20 0x7f54a8142770 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f549c00c430 tx=0x7f549c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:49.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54a5600640 1 -- 172.21.15.47:0/2854201402 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c0052d0 con 0x7f54a80b3a20 2024-02-19T15:30:49.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54a5600640 1 -- 172.21.15.47:0/2854201402 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549c005450 con 0x7f54a80b3a20 2024-02-19T15:30:49.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54a5600640 1 -- 172.21.15.47:0/2854201402 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c003840 con 0x7f54a80b3a20 2024-02-19T15:30:49.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54b0000640 1 -- 172.21.15.47:0/2854201402 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a8141f70 con 0x7f54a80b3a20 2024-02-19T15:30:49.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.877+0000 7f54b0000640 1 -- 172.21.15.47:0/2854201402 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a813c5a0 con 0x7f54a80b3a20 2024-02-19T15:30:49.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.881+0000 7f54b0000640 1 -- 172.21.15.47:0/2854201402 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a8075360 con 0x7f54a80b3a20 2024-02-19T15:30:49.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.881+0000 7f54a5600640 1 -- 172.21.15.47:0/2854201402 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f549c0039a0 con 0x7f54a80b3a20 2024-02-19T15:30:49.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.881+0000 7f54a5600640 1 --2- 172.21.15.47:0/2854201402 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5488066e40 0x7f5488069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:49.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.881+0000 7f54a5600640 1 -- 172.21.15.47:0/2854201402 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f549c084030 con 0x7f54a80b3a20 2024-02-19T15:30:49.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.881+0000 7f54acc00640 1 --2- 172.21.15.47:0/2854201402 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5488066e40 0x7f5488069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:49.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.881+0000 7f54acc00640 1 --2- 172.21.15.47:0/2854201402 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5488066e40 0x7f5488069300 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f54940ae000 tx=0x7f5494008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:49.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:49.885+0000 7f54a5600640 1 -- 172.21.15.47:0/2854201402 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f549c0b61c0 con 0x7f54a80b3a20 2024-02-19T15:30:50.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:50.085+0000 7f54b0000640 1 -- 172.21.15.47:0/2854201402 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54a80b24c0 con 0x7f54a80b3a20 2024-02-19T15:30:50.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:50.085+0000 7f54a5600640 1 -- 172.21.15.47:0/2854201402 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f549c0549b0 con 0x7f54a80b3a20 2024-02-19T15:30:50.085 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:50.085 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:50.085 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:50.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:50.085+0000 7f54b0000640 1 -- 172.21.15.47:0/2854201402 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5488066e40 msgr2=0x7f5488069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:50.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:50.085+0000 7f54b0000640 1 --2- 172.21.15.47:0/2854201402 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5488066e40 0x7f5488069300 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f54940ae000 tx=0x7f5494008040 comp rx=0 tx=0).stop 2024-02-19T15:30:50.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:50.085+0000 7f54b0000640 1 -- 172.21.15.47:0/2854201402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54a80b3a20 msgr2=0x7f54a8142770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:50.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:50.085+0000 7f54b0000640 1 --2- 172.21.15.47:0/2854201402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54a80b3a20 0x7f54a8142770 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f549c00c430 tx=0x7f549c00c8f0 comp rx=0 tx=0).stop 2024-02-19T15:30:50.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:50.085+0000 7f54b0000640 1 -- 172.21.15.47:0/2854201402 shutdown_connections 2024-02-19T15:30:50.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:50.085+0000 7f54b0000640 1 --2- 172.21.15.47:0/2854201402 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5488066e40 0x7f5488069300 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:50.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:50.085+0000 7f54b0000640 1 --2- 172.21.15.47:0/2854201402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54a80cee30 0x7f54a8140950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:50.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:50.085+0000 7f54b0000640 1 --2- 172.21.15.47:0/2854201402 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54a80ccc50 0x7f54a813c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:50.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:50.085+0000 7f54b0000640 1 --2- 172.21.15.47:0/2854201402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54a80b3a20 0x7f54a8142770 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:50.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:50.085+0000 7f54b0000640 1 -- 172.21.15.47:0/2854201402 >> 172.21.15.47:0/2854201402 conn(0x7f54a80d0a50 msgr2=0x7f54a8142f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:50.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:50.085+0000 7f54b0000640 1 -- 172.21.15.47:0/2854201402 shutdown_connections 2024-02-19T15:30:50.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:50.085+0000 7f54b0000640 1 -- 172.21.15.47:0/2854201402 wait complete. 2024-02-19T15:30:50.231 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:50.307 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:50.308 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:51.309 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:51.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.449+0000 7fef7a600640 1 Processor -- start 2024-02-19T15:30:51.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.449+0000 7fef7a600640 1 -- start start 2024-02-19T15:30:51.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.449+0000 7fef7a600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef7415e180 0x7fef740b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:51.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.449+0000 7fef7a600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fef7406cfc0 0x7fef740ae870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:51.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.449+0000 7fef7a600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef740b2270 con 0x7fef7406c1d0 2024-02-19T15:30:51.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.449+0000 7fef7a600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef740b2550 con 0x7fef7406cfc0 2024-02-19T15:30:51.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.449+0000 7fef7a600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef740b2830 con 0x7fef7415e180 2024-02-19T15:30:51.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.449+0000 7fef73e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fef7406cfc0 0x7fef740ae870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:51.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.449+0000 7fef6aa00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fef7406c1d0 0x7fef7406c8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52992/0 (socket says 172.21.15.47:52992) 2024-02-19T15:30:51.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.449+0000 7fef6aa00640 1 -- 172.21.15.47:0/2723695726 learned_addr learned my addr 172.21.15.47:0/2723695726 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:51.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.449+0000 7fef73400640 1 --2- 172.21.15.47:0/2723695726 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef7415e180 0x7fef740b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:51.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.449+0000 7fef73e00640 1 -- 172.21.15.47:0/2723695726 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef7415e180 msgr2=0x7fef740b1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:51.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.449+0000 7fef73e00640 1 --2- 172.21.15.47:0/2723695726 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef7415e180 0x7fef740b1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:51.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.449+0000 7fef73e00640 1 -- 172.21.15.47:0/2723695726 >> v1:172.21.15.47:6789/0 conn(0x7fef7406c1d0 legacy=0x7fef7406c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:51.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.453+0000 7fef73e00640 1 -- 172.21.15.47:0/2723695726 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef740b2b10 con 0x7fef7406cfc0 2024-02-19T15:30:51.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.453+0000 7fef72a00640 1 -- 172.21.15.47:0/2723695726 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540215566 0 0) 0x7fef740b2270 con 0x7fef7406c1d0 2024-02-19T15:30:51.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.457+0000 7fef73e00640 1 --2- 172.21.15.47:0/2723695726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef7406cfc0 0x7fef740ae870 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fef6c0097c0 tx=0x7fef6c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb99971872f44ad server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:51.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.457+0000 7fef72a00640 1 -- 172.21.15.47:0/2723695726 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6c0ab610 con 0x7fef7406cfc0 2024-02-19T15:30:51.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.457+0000 7fef72a00640 1 -- 172.21.15.47:0/2723695726 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6c0ab770 con 0x7fef7406cfc0 2024-02-19T15:30:51.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.457+0000 7fef72a00640 1 -- 172.21.15.47:0/2723695726 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6c0abb00 con 0x7fef7406cfc0 2024-02-19T15:30:51.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.457+0000 7fef7a600640 1 -- 172.21.15.47:0/2723695726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef7406cfc0 msgr2=0x7fef740ae870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:51.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.457+0000 7fef7a600640 1 --2- 172.21.15.47:0/2723695726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef7406cfc0 0x7fef740ae870 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fef6c0097c0 tx=0x7fef6c01c470 comp rx=0 tx=0).stop 2024-02-19T15:30:51.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.457+0000 7fef7a600640 1 -- 172.21.15.47:0/2723695726 shutdown_connections 2024-02-19T15:30:51.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.457+0000 7fef7a600640 1 --2- 172.21.15.47:0/2723695726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef7406cfc0 0x7fef740ae870 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:51.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.457+0000 7fef7a600640 1 --2- 172.21.15.47:0/2723695726 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef7415e180 0x7fef740b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:51.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.457+0000 7fef7a600640 1 -- 172.21.15.47:0/2723695726 >> 172.21.15.47:0/2723695726 conn(0x7fef740d1ba0 msgr2=0x7fef740d1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:51.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.457+0000 7fef7a600640 1 -- 172.21.15.47:0/2723695726 shutdown_connections 2024-02-19T15:30:51.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.457+0000 7fef7a600640 1 -- 172.21.15.47:0/2723695726 wait complete. 2024-02-19T15:30:51.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.457+0000 7fef7a600640 1 Processor -- start 2024-02-19T15:30:51.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.457+0000 7fef7a600640 1 -- start start 2024-02-19T15:30:51.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef7a600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef7406c1d0 0x7fef7415cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:51.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef7a600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef7406cfc0 0x7fef7414c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:51.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef7a600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef7415e180 0x7fef74155580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:51.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef7a600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7415ec50 con 0x7fef7415e180 2024-02-19T15:30:51.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef7a600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7415d710 con 0x7fef7406c1d0 2024-02-19T15:30:51.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef7a600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef7415e920 con 0x7fef7406cfc0 2024-02-19T15:30:51.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef73e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef7415e180 0x7fef74155580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:51.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef73e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef7415e180 0x7fef74155580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43840/0 (socket says 172.21.15.47:43840) 2024-02-19T15:30:51.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef73e00640 1 -- 172.21.15.47:0/630944440 learned_addr learned my addr 172.21.15.47:0/630944440 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:51.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef73400640 1 --2- 172.21.15.47:0/630944440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef7406c1d0 0x7fef7415cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:51.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef6aa00640 1 --2- 172.21.15.47:0/630944440 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef7406cfc0 0x7fef7414c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:51.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef73e00640 1 -- 172.21.15.47:0/630944440 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef7406cfc0 msgr2=0x7fef7414c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:51.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef73e00640 1 --2- 172.21.15.47:0/630944440 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef7406cfc0 0x7fef7414c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:51.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef73e00640 1 -- 172.21.15.47:0/630944440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef7406c1d0 msgr2=0x7fef7415cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:51.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef73e00640 1 --2- 172.21.15.47:0/630944440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef7406c1d0 0x7fef7415cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:51.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef73e00640 1 -- 172.21.15.47:0/630944440 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef6c009420 con 0x7fef7415e180 2024-02-19T15:30:51.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef73400640 1 --2- 172.21.15.47:0/630944440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef7406c1d0 0x7fef7415cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:30:51.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef73e00640 1 --2- 172.21.15.47:0/630944440 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef7415e180 0x7fef74155580 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fef6c0ad040 tx=0x7fef6c01acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:51.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef6be00640 1 -- 172.21.15.47:0/630944440 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6c0bf610 con 0x7fef7415e180 2024-02-19T15:30:51.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef6be00640 1 -- 172.21.15.47:0/630944440 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6c0bf770 con 0x7fef7415e180 2024-02-19T15:30:51.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef6be00640 1 -- 172.21.15.47:0/630944440 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6c0bfb20 con 0x7fef7415e180 2024-02-19T15:30:51.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef7a600640 1 -- 172.21.15.47:0/630944440 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef7415d150 con 0x7fef7415e180 2024-02-19T15:30:51.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef7a600640 1 -- 172.21.15.47:0/630944440 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef7414ce70 con 0x7fef7415e180 2024-02-19T15:30:51.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.461+0000 7fef7a600640 1 -- 172.21.15.47:0/630944440 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef74068500 con 0x7fef7415e180 2024-02-19T15:30:51.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.465+0000 7fef6be00640 1 -- 172.21.15.47:0/630944440 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fef6c0bfc80 con 0x7fef7415e180 2024-02-19T15:30:51.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.465+0000 7fef6be00640 1 --2- 172.21.15.47:0/630944440 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef50066e70 0x7fef50069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:51.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.465+0000 7fef6be00640 1 -- 172.21.15.47:0/630944440 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fef6c11f800 con 0x7fef7415e180 2024-02-19T15:30:51.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.465+0000 7fef73400640 1 --2- 172.21.15.47:0/630944440 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef50066e70 0x7fef50069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:51.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.465+0000 7fef73400640 1 --2- 172.21.15.47:0/630944440 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef50066e70 0x7fef50069330 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fef64003fd0 tx=0x7fef64003a00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:51.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.469+0000 7fef6be00640 1 -- 172.21.15.47:0/630944440 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fef6c0ec760 con 0x7fef7415e180 2024-02-19T15:30:51.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.665+0000 7fef7a600640 1 -- 172.21.15.47:0/630944440 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef74075360 con 0x7fef7415e180 2024-02-19T15:30:51.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.665+0000 7fef6be00640 1 -- 172.21.15.47:0/630944440 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7fef6c0efe20 con 0x7fef7415e180 2024-02-19T15:30:51.666 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:51.666 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:51.666 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:51.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.669+0000 7fef7a600640 1 -- 172.21.15.47:0/630944440 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef50066e70 msgr2=0x7fef50069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:51.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.669+0000 7fef7a600640 1 --2- 172.21.15.47:0/630944440 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef50066e70 0x7fef50069330 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fef64003fd0 tx=0x7fef64003a00 comp rx=0 tx=0).stop 2024-02-19T15:30:51.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.669+0000 7fef7a600640 1 -- 172.21.15.47:0/630944440 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef7415e180 msgr2=0x7fef74155580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:51.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.669+0000 7fef7a600640 1 --2- 172.21.15.47:0/630944440 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef7415e180 0x7fef74155580 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fef6c0ad040 tx=0x7fef6c01acb0 comp rx=0 tx=0).stop 2024-02-19T15:30:51.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.669+0000 7fef7a600640 1 -- 172.21.15.47:0/630944440 shutdown_connections 2024-02-19T15:30:51.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.669+0000 7fef7a600640 1 --2- 172.21.15.47:0/630944440 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fef50066e70 0x7fef50069330 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:51.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.669+0000 7fef7a600640 1 --2- 172.21.15.47:0/630944440 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef7415e180 0x7fef74155580 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:51.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.669+0000 7fef7a600640 1 --2- 172.21.15.47:0/630944440 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef7406cfc0 0x7fef7414c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:51.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.669+0000 7fef7a600640 1 --2- 172.21.15.47:0/630944440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef7406c1d0 0x7fef7415cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:51.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.669+0000 7fef7a600640 1 -- 172.21.15.47:0/630944440 >> 172.21.15.47:0/630944440 conn(0x7fef740d1ba0 msgr2=0x7fef74157130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:51.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.669+0000 7fef7a600640 1 -- 172.21.15.47:0/630944440 shutdown_connections 2024-02-19T15:30:51.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:51.669+0000 7fef7a600640 1 -- 172.21.15.47:0/630944440 wait complete. 2024-02-19T15:30:51.889 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:51.889 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:52.233 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:52.890 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:53.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.029+0000 7fa70ac00640 1 Processor -- start 2024-02-19T15:30:53.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.029+0000 7fa70ac00640 1 -- start start 2024-02-19T15:30:53.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.033+0000 7fa70ac00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7041614e0 0x7fa70415f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:53.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.033+0000 7fa70ac00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa70415eee0 0x7fa7040b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:53.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.033+0000 7fa70ac00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa7040b2330 0x7fa70406ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:53.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.033+0000 7fa70ac00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7040b1c00 con 0x7fa70415eee0 2024-02-19T15:30:53.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.033+0000 7fa70ac00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa70406b5d0 con 0x7fa7040b2330 2024-02-19T15:30:53.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.033+0000 7fa70ac00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7040b2790 con 0x7fa7041614e0 2024-02-19T15:30:53.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.033+0000 7fa6fb400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa70415eee0 0x7fa7040b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:53.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.033+0000 7fa6fb400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa70415eee0 0x7fa7040b1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43852/0 (socket says 172.21.15.47:43852) 2024-02-19T15:30:53.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.033+0000 7fa6fb400640 1 -- 172.21.15.47:0/304384885 learned_addr learned my addr 172.21.15.47:0/304384885 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:53.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.033+0000 7fa708c00640 1 --2- 172.21.15.47:0/304384885 >> v2:172.21.15.116:3300/0 conn(0x7fa7040b2330 0x7fa70406ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:53.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.033+0000 7fa703e00640 1 --2- 172.21.15.47:0/304384885 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7041614e0 0x7fa70415f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:53.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.033+0000 7fa6fb400640 1 -- 172.21.15.47:0/304384885 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7041614e0 msgr2=0x7fa70415f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:53.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.033+0000 7fa6fb400640 1 --2- 172.21.15.47:0/304384885 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7041614e0 0x7fa70415f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:53.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.033+0000 7fa6fb400640 1 -- 172.21.15.47:0/304384885 >> v2:172.21.15.116:3300/0 conn(0x7fa7040b2330 msgr2=0x7fa70406ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:53.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.033+0000 7fa6fb400640 1 --2- 172.21.15.47:0/304384885 >> v2:172.21.15.116:3300/0 conn(0x7fa7040b2330 0x7fa70406ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:53.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.033+0000 7fa6fb400640 1 -- 172.21.15.47:0/304384885 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7040b1640 con 0x7fa70415eee0 2024-02-19T15:30:53.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.033+0000 7fa708c00640 1 --2- 172.21.15.47:0/304384885 >> v2:172.21.15.116:3300/0 conn(0x7fa7040b2330 0x7fa70406ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:30:53.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.037+0000 7fa6fb400640 1 --2- 172.21.15.47:0/304384885 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa70415eee0 0x7fa7040b1df0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fa70415c050 tx=0x7fa6ec01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c82b3bf7fc7e01 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:53.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.037+0000 7fa703400640 1 -- 172.21.15.47:0/304384885 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec0aa640 con 0x7fa70415eee0 2024-02-19T15:30:53.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.037+0000 7fa703400640 1 -- 172.21.15.47:0/304384885 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ec0aa7a0 con 0x7fa70415eee0 2024-02-19T15:30:53.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.037+0000 7fa703400640 1 -- 172.21.15.47:0/304384885 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec0aab30 con 0x7fa70415eee0 2024-02-19T15:30:53.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.037+0000 7fa70ac00640 1 -- 172.21.15.47:0/304384885 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa70415eee0 msgr2=0x7fa7040b1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:53.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.037+0000 7fa70ac00640 1 --2- 172.21.15.47:0/304384885 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa70415eee0 0x7fa7040b1df0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fa70415c050 tx=0x7fa6ec01c520 comp rx=0 tx=0).stop 2024-02-19T15:30:53.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.037+0000 7fa70ac00640 1 -- 172.21.15.47:0/304384885 shutdown_connections 2024-02-19T15:30:53.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.037+0000 7fa70ac00640 1 --2- 172.21.15.47:0/304384885 >> v2:172.21.15.116:3300/0 conn(0x7fa7040b2330 0x7fa70406ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:53.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.037+0000 7fa70ac00640 1 --2- 172.21.15.47:0/304384885 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa70415eee0 0x7fa7040b1df0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:53.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.037+0000 7fa70ac00640 1 --2- 172.21.15.47:0/304384885 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7041614e0 0x7fa70415f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:53.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.037+0000 7fa70ac00640 1 -- 172.21.15.47:0/304384885 >> 172.21.15.47:0/304384885 conn(0x7fa7040b4fc0 msgr2=0x7fa7040b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:53.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.037+0000 7fa70ac00640 1 -- 172.21.15.47:0/304384885 shutdown_connections 2024-02-19T15:30:53.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.037+0000 7fa70ac00640 1 -- 172.21.15.47:0/304384885 wait complete. 2024-02-19T15:30:53.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.037+0000 7fa70ac00640 1 Processor -- start 2024-02-19T15:30:53.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.037+0000 7fa70ac00640 1 -- start start 2024-02-19T15:30:53.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa70ac00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7040b2330 0x7fa704158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:53.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa70ac00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70415eee0 0x7fa70415f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:53.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa70ac00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7041614e0 0x7fa70409b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:53.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa70ac00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7041601a0 con 0x7fa7041614e0 2024-02-19T15:30:53.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa70ac00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70409aed0 con 0x7fa70415eee0 2024-02-19T15:30:53.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa70ac00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa70415fec0 con 0x7fa7040b2330 2024-02-19T15:30:53.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa708c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7041614e0 0x7fa70409b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa708c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7041614e0 0x7fa70409b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43856/0 (socket says 172.21.15.47:43856) 2024-02-19T15:30:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa708c00640 1 -- 172.21.15.47:0/1171817220 learned_addr learned my addr 172.21.15.47:0/1171817220 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa6fb400640 1 --2- 172.21.15.47:0/1171817220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70415eee0 0x7fa70415f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa703e00640 1 --2- 172.21.15.47:0/1171817220 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7040b2330 0x7fa704158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa708c00640 1 -- 172.21.15.47:0/1171817220 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7040b2330 msgr2=0x7fa704158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa708c00640 1 --2- 172.21.15.47:0/1171817220 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7040b2330 0x7fa704158850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa708c00640 1 -- 172.21.15.47:0/1171817220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70415eee0 msgr2=0x7fa70415f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa708c00640 1 --2- 172.21.15.47:0/1171817220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70415eee0 0x7fa70415f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa708c00640 1 -- 172.21.15.47:0/1171817220 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ec0094d0 con 0x7fa7041614e0 2024-02-19T15:30:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa708c00640 1 --2- 172.21.15.47:0/1171817220 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7041614e0 0x7fa70409b1e0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc007e10 tx=0x7fa6fc00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:53.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa700c00640 1 -- 172.21.15.47:0/1171817220 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc003ba0 con 0x7fa7041614e0 2024-02-19T15:30:53.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa700c00640 1 -- 172.21.15.47:0/1171817220 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6fc00f040 con 0x7fa7041614e0 2024-02-19T15:30:53.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa70ac00640 1 -- 172.21.15.47:0/1171817220 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa70409a3f0 con 0x7fa7041614e0 2024-02-19T15:30:53.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa700c00640 1 -- 172.21.15.47:0/1171817220 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc00de80 con 0x7fa7041614e0 2024-02-19T15:30:53.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa70ac00640 1 -- 172.21.15.47:0/1171817220 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa70409a9b0 con 0x7fa7041614e0 2024-02-19T15:30:53.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa70ac00640 1 -- 172.21.15.47:0/1171817220 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa70409a250 con 0x7fa7041614e0 2024-02-19T15:30:53.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.041+0000 7fa700c00640 1 -- 172.21.15.47:0/1171817220 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa6fc004030 con 0x7fa7041614e0 2024-02-19T15:30:53.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.045+0000 7fa700c00640 1 --2- 172.21.15.47:0/1171817220 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa6e0066e40 0x7fa6e0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:53.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.045+0000 7fa700c00640 1 -- 172.21.15.47:0/1171817220 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa6fc084d10 con 0x7fa7041614e0 2024-02-19T15:30:53.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.045+0000 7fa703e00640 1 --2- 172.21.15.47:0/1171817220 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa6e0066e40 0x7fa6e0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:53.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.045+0000 7fa703e00640 1 --2- 172.21.15.47:0/1171817220 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa6e0066e40 0x7fa6e0069300 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fa6f40079c0 tx=0x7fa6f4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:53.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.049+0000 7fa700c00640 1 -- 172.21.15.47:0/1171817220 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa70409a250 con 0x7fa7041614e0 2024-02-19T15:30:53.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.249+0000 7fa70ac00640 1 -- 172.21.15.47:0/1171817220 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa70409a250 con 0x7fa7041614e0 2024-02-19T15:30:53.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.253+0000 7fa700c00640 1 -- 172.21.15.47:0/1171817220 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7fa70409a250 con 0x7fa7041614e0 2024-02-19T15:30:53.251 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:53.251 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:53.252 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:53.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.253+0000 7fa70ac00640 1 -- 172.21.15.47:0/1171817220 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa6e0066e40 msgr2=0x7fa6e0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:53.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.253+0000 7fa70ac00640 1 --2- 172.21.15.47:0/1171817220 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa6e0066e40 0x7fa6e0069300 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fa6f40079c0 tx=0x7fa6f4008040 comp rx=0 tx=0).stop 2024-02-19T15:30:53.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.253+0000 7fa70ac00640 1 -- 172.21.15.47:0/1171817220 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7041614e0 msgr2=0x7fa70409b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:53.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.253+0000 7fa70ac00640 1 --2- 172.21.15.47:0/1171817220 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7041614e0 0x7fa70409b1e0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc007e10 tx=0x7fa6fc00d4a0 comp rx=0 tx=0).stop 2024-02-19T15:30:53.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.253+0000 7fa70ac00640 1 -- 172.21.15.47:0/1171817220 shutdown_connections 2024-02-19T15:30:53.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.253+0000 7fa70ac00640 1 --2- 172.21.15.47:0/1171817220 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa6e0066e40 0x7fa6e0069300 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:53.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.253+0000 7fa70ac00640 1 --2- 172.21.15.47:0/1171817220 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7041614e0 0x7fa70409b1e0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:53.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.253+0000 7fa70ac00640 1 --2- 172.21.15.47:0/1171817220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70415eee0 0x7fa70415f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:53.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.253+0000 7fa70ac00640 1 --2- 172.21.15.47:0/1171817220 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7040b2330 0x7fa704158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:53.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.253+0000 7fa70ac00640 1 -- 172.21.15.47:0/1171817220 >> 172.21.15.47:0/1171817220 conn(0x7fa7040b4fc0 msgr2=0x7fa70406e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:53.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.253+0000 7fa70ac00640 1 -- 172.21.15.47:0/1171817220 shutdown_connections 2024-02-19T15:30:53.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:53.253+0000 7fa70ac00640 1 -- 172.21.15.47:0/1171817220 wait complete. 2024-02-19T15:30:53.474 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:53.474 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:54.235 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:54.475 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:54.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.617+0000 7f5a72e00640 1 Processor -- start 2024-02-19T15:30:54.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.617+0000 7f5a72e00640 1 -- start start 2024-02-19T15:30:54.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.617+0000 7f5a72e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5a6c15b1a0 0x7f5a6c15b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:54.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.617+0000 7f5a72e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6c0718c0 con 0x7f5a6c071350 2024-02-19T15:30:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.617+0000 7f5a72e00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6c071ba0 con 0x7f5a6c06c9a0 2024-02-19T15:30:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.617+0000 7f5a72e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a6c071e80 con 0x7f5a6c15b1a0 2024-02-19T15:30:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.617+0000 7f5a63e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5a6c071350 0x7f5a6c15c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35572/0 (socket says 172.21.15.47:35572) 2024-02-19T15:30:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.617+0000 7f5a63e00640 1 -- 172.21.15.47:0/130653277 learned_addr learned my addr 172.21.15.47:0/130653277 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.617+0000 7f5a6be00640 1 --2- 172.21.15.47:0/130653277 >> v2:172.21.15.116:3300/0 conn(0x7f5a6c15b1a0 0x7f5a6c15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.617+0000 7f5a6b400640 1 -- 172.21.15.47:0/130653277 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959127960 0 0) 0x7f5a6c0718c0 con 0x7f5a6c071350 2024-02-19T15:30:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.617+0000 7f5a6b400640 1 -- 172.21.15.47:0/130653277 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a50003530 con 0x7f5a6c071350 2024-02-19T15:30:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.617+0000 7f5a6b400640 1 -- 172.21.15.47:0/130653277 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625468098 0 0) 0x7f5a6c071ba0 con 0x7f5a6c06c9a0 2024-02-19T15:30:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.617+0000 7f5a6b400640 1 -- 172.21.15.47:0/130653277 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a6c0718c0 con 0x7f5a6c06c9a0 2024-02-19T15:30:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.617+0000 7f5a6be00640 1 -- 172.21.15.47:0/130653277 >> v1:172.21.15.116:6789/0 conn(0x7f5a6c06c9a0 legacy=0x7f5a6c15cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:54.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.617+0000 7f5a6be00640 1 -- 172.21.15.47:0/130653277 >> v1:172.21.15.47:6789/0 conn(0x7f5a6c071350 legacy=0x7f5a6c15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:54.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.617+0000 7f5a6b400640 1 -- 172.21.15.47:0/130653277 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2429707504 0 0) 0x7f5a50003530 con 0x7f5a6c071350 2024-02-19T15:30:54.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.617+0000 7f5a6be00640 1 -- 172.21.15.47:0/130653277 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6c072160 con 0x7f5a6c15b1a0 2024-02-19T15:30:54.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.621+0000 7f5a6be00640 1 --2- 172.21.15.47:0/130653277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a6c15b1a0 0x7f5a6c15b580 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c0097f0 tx=0x7f5a5c01c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53be287f03b20e35 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:54.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.621+0000 7f5a6b400640 1 -- 172.21.15.47:0/130653277 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a54001680 con 0x7f5a6c15b1a0 2024-02-19T15:30:54.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.621+0000 7f5a6b400640 1 -- 172.21.15.47:0/130653277 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a5c0ac440 con 0x7f5a6c15b1a0 2024-02-19T15:30:54.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.621+0000 7f5a6b400640 1 -- 172.21.15.47:0/130653277 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c0ac7d0 con 0x7f5a6c15b1a0 2024-02-19T15:30:54.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.621+0000 7f5a72e00640 1 -- 172.21.15.47:0/130653277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a6c15b1a0 msgr2=0x7f5a6c15b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:54.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.621+0000 7f5a72e00640 1 --2- 172.21.15.47:0/130653277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a6c15b1a0 0x7f5a6c15b580 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c0097f0 tx=0x7f5a5c01c3c0 comp rx=0 tx=0).stop 2024-02-19T15:30:54.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.621+0000 7f5a72e00640 1 -- 172.21.15.47:0/130653277 shutdown_connections 2024-02-19T15:30:54.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.621+0000 7f5a72e00640 1 --2- 172.21.15.47:0/130653277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a6c15b1a0 0x7f5a6c15b580 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:54.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.621+0000 7f5a72e00640 1 -- 172.21.15.47:0/130653277 >> 172.21.15.47:0/130653277 conn(0x7f5a6c0b16a0 msgr2=0x7f5a6c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:54.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.621+0000 7f5a72e00640 1 -- 172.21.15.47:0/130653277 shutdown_connections 2024-02-19T15:30:54.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.621+0000 7f5a72e00640 1 -- 172.21.15.47:0/130653277 wait complete. 2024-02-19T15:30:54.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a72e00640 1 Processor -- start 2024-02-19T15:30:54.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a72e00640 1 -- start start 2024-02-19T15:30:54.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a72e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a6c06c9a0 0x7f5a6c1576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:54.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a72e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a6c071350 0x7f5a6c15c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:54.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a72e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a6c15b1a0 0x7f5a6c15cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:54.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a72e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c1fb450 con 0x7f5a6c06c9a0 2024-02-19T15:30:54.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a72e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c157c20 con 0x7f5a6c15b1a0 2024-02-19T15:30:54.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a72e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c1f7e90 con 0x7f5a6c071350 2024-02-19T15:30:54.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a6be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a6c06c9a0 0x7f5a6c1576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:54.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a6be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a6c06c9a0 0x7f5a6c1576e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:47744/0 (socket says 172.21.15.47:47744) 2024-02-19T15:30:54.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a6be00640 1 -- 172.21.15.47:0/1900410101 learned_addr learned my addr 172.21.15.47:0/1900410101 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:54.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a70e00640 1 --2- 172.21.15.47:0/1900410101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a6c15b1a0 0x7f5a6c15cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:54.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a63e00640 1 --2- 172.21.15.47:0/1900410101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a6c071350 0x7f5a6c15c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:54.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a6be00640 1 -- 172.21.15.47:0/1900410101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a6c071350 msgr2=0x7f5a6c15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:54.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a6be00640 1 --2- 172.21.15.47:0/1900410101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a6c071350 0x7f5a6c15c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:54.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a6be00640 1 -- 172.21.15.47:0/1900410101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a6c15b1a0 msgr2=0x7f5a6c15cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:54.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a6be00640 1 --2- 172.21.15.47:0/1900410101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a6c15b1a0 0x7f5a6c15cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:54.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a6be00640 1 -- 172.21.15.47:0/1900410101 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c009420 con 0x7f5a6c06c9a0 2024-02-19T15:30:54.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a6be00640 1 --2- 172.21.15.47:0/1900410101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a6c06c9a0 0x7f5a6c1576e0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c0097f0 tx=0x7f5a5c019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:54.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a68c00640 1 -- 172.21.15.47:0/1900410101 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c0bc070 con 0x7f5a6c06c9a0 2024-02-19T15:30:54.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a68c00640 1 -- 172.21.15.47:0/1900410101 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a5c004020 con 0x7f5a6c06c9a0 2024-02-19T15:30:54.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a72e00640 1 -- 172.21.15.47:0/1900410101 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6c1f6e60 con 0x7f5a6c06c9a0 2024-02-19T15:30:54.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a68c00640 1 -- 172.21.15.47:0/1900410101 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c0acea0 con 0x7f5a6c06c9a0 2024-02-19T15:30:54.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a72e00640 1 -- 172.21.15.47:0/1900410101 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6c1f73b0 con 0x7f5a6c06c9a0 2024-02-19T15:30:54.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.625+0000 7f5a72e00640 1 -- 172.21.15.47:0/1900410101 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6c075360 con 0x7f5a6c06c9a0 2024-02-19T15:30:54.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.629+0000 7f5a68c00640 1 -- 172.21.15.47:0/1900410101 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5a5c0b3040 con 0x7f5a6c06c9a0 2024-02-19T15:30:54.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.629+0000 7f5a68c00640 1 --2- 172.21.15.47:0/1900410101 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5a48066e40 0x7f5a48069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:54.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.629+0000 7f5a68c00640 1 -- 172.21.15.47:0/1900410101 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5a5c11e4a0 con 0x7f5a6c06c9a0 2024-02-19T15:30:54.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.629+0000 7f5a63e00640 1 --2- 172.21.15.47:0/1900410101 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5a48066e40 0x7f5a48069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:54.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.629+0000 7f5a63e00640 1 --2- 172.21.15.47:0/1900410101 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5a48066e40 0x7f5a48069300 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c15d2c0 tx=0x7f5a54009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:54.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.633+0000 7f5a68c00640 1 -- 172.21.15.47:0/1900410101 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f5a5c0eb130 con 0x7f5a6c06c9a0 2024-02-19T15:30:54.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.829+0000 7f5a72e00640 1 -- 172.21.15.47:0/1900410101 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a6c0cf660 con 0x7f5a6c06c9a0 2024-02-19T15:30:54.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.829+0000 7f5a68c00640 1 -- 172.21.15.47:0/1900410101 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f5a5c0b1d80 con 0x7f5a6c06c9a0 2024-02-19T15:30:54.829 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:54.829 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:54.829 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:54.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.833+0000 7f5a72e00640 1 -- 172.21.15.47:0/1900410101 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5a48066e40 msgr2=0x7f5a48069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:54.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.833+0000 7f5a72e00640 1 --2- 172.21.15.47:0/1900410101 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5a48066e40 0x7f5a48069300 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c15d2c0 tx=0x7f5a54009040 comp rx=0 tx=0).stop 2024-02-19T15:30:54.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.833+0000 7f5a72e00640 1 -- 172.21.15.47:0/1900410101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a6c06c9a0 msgr2=0x7f5a6c1576e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:54.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.833+0000 7f5a72e00640 1 --2- 172.21.15.47:0/1900410101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a6c06c9a0 0x7f5a6c1576e0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c0097f0 tx=0x7f5a5c019160 comp rx=0 tx=0).stop 2024-02-19T15:30:54.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.833+0000 7f5a72e00640 1 -- 172.21.15.47:0/1900410101 shutdown_connections 2024-02-19T15:30:54.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.833+0000 7f5a72e00640 1 --2- 172.21.15.47:0/1900410101 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5a48066e40 0x7f5a48069300 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:54.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.833+0000 7f5a72e00640 1 --2- 172.21.15.47:0/1900410101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a6c15b1a0 0x7f5a6c15cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:54.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.833+0000 7f5a72e00640 1 --2- 172.21.15.47:0/1900410101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a6c071350 0x7f5a6c15c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:54.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.833+0000 7f5a72e00640 1 --2- 172.21.15.47:0/1900410101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5a6c06c9a0 0x7f5a6c1576e0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:54.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.833+0000 7f5a72e00640 1 -- 172.21.15.47:0/1900410101 >> 172.21.15.47:0/1900410101 conn(0x7f5a6c0b16a0 msgr2=0x7f5a6c0af5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:54.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.833+0000 7f5a72e00640 1 -- 172.21.15.47:0/1900410101 shutdown_connections 2024-02-19T15:30:54.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:54.833+0000 7f5a72e00640 1 -- 172.21.15.47:0/1900410101 wait complete. 2024-02-19T15:30:55.061 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:55.061 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:56.062 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:56.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.201+0000 7fd96ec00640 1 Processor -- start 2024-02-19T15:30:56.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.201+0000 7fd96ec00640 1 -- start start 2024-02-19T15:30:56.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.205+0000 7fd96ec00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd96815bcd0 0x7fd9680cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:56.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.205+0000 7fd96ec00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd9680cedb0 0x7fd9680cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:56.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.205+0000 7fd96ec00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9680cc700 con 0x7fd9680d2710 2024-02-19T15:30:56.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.205+0000 7fd96ec00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9680cf5e0 con 0x7fd96815bcd0 2024-02-19T15:30:56.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.205+0000 7fd96ec00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9680cf300 con 0x7fd9680cedb0 2024-02-19T15:30:56.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.205+0000 7fd967e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd96815bcd0 0x7fd9680cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:56.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.205+0000 7fd967e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd96815bcd0 0x7fd9680cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:47758/0 (socket says 172.21.15.47:47758) 2024-02-19T15:30:56.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.205+0000 7fd967e00640 1 -- 172.21.15.47:0/2863633086 learned_addr learned my addr 172.21.15.47:0/2863633086 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:56.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.205+0000 7fd967400640 1 --2- 172.21.15.47:0/2863633086 >> v2:172.21.15.116:3300/0 conn(0x7fd9680cedb0 0x7fd9680cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:56.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.205+0000 7fd967e00640 1 -- 172.21.15.47:0/2863633086 >> v2:172.21.15.116:3300/0 conn(0x7fd9680cedb0 msgr2=0x7fd9680cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:56.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.205+0000 7fd967e00640 1 --2- 172.21.15.47:0/2863633086 >> v2:172.21.15.116:3300/0 conn(0x7fd9680cedb0 0x7fd9680cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:56.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.205+0000 7fd967e00640 1 -- 172.21.15.47:0/2863633086 >> v1:172.21.15.116:6789/0 conn(0x7fd9680d2710 legacy=0x7fd96815d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:56.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.205+0000 7fd967e00640 1 -- 172.21.15.47:0/2863633086 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9680cc9a0 con 0x7fd96815bcd0 2024-02-19T15:30:56.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.209+0000 7fd967e00640 1 --2- 172.21.15.47:0/2863633086 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd96815bcd0 0x7fd9680cd480 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fd9680ce100 tx=0x7fd95801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfd49ea580a8026c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:56.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.209+0000 7fd966a00640 1 -- 172.21.15.47:0/2863633086 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9580ab600 con 0x7fd96815bcd0 2024-02-19T15:30:56.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.209+0000 7fd966a00640 1 -- 172.21.15.47:0/2863633086 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9580ab760 con 0x7fd96815bcd0 2024-02-19T15:30:56.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.209+0000 7fd966a00640 1 -- 172.21.15.47:0/2863633086 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9580abaf0 con 0x7fd96815bcd0 2024-02-19T15:30:56.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.209+0000 7fd96ec00640 1 -- 172.21.15.47:0/2863633086 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd96815bcd0 msgr2=0x7fd9680cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:56.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.209+0000 7fd96ec00640 1 --2- 172.21.15.47:0/2863633086 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd96815bcd0 0x7fd9680cd480 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fd9680ce100 tx=0x7fd95801c4f0 comp rx=0 tx=0).stop 2024-02-19T15:30:56.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.209+0000 7fd96ec00640 1 -- 172.21.15.47:0/2863633086 shutdown_connections 2024-02-19T15:30:56.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.209+0000 7fd96ec00640 1 --2- 172.21.15.47:0/2863633086 >> v2:172.21.15.116:3300/0 conn(0x7fd9680cedb0 0x7fd9680cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:56.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.209+0000 7fd96ec00640 1 --2- 172.21.15.47:0/2863633086 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd96815bcd0 0x7fd9680cd480 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:56.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.209+0000 7fd96ec00640 1 -- 172.21.15.47:0/2863633086 >> 172.21.15.47:0/2863633086 conn(0x7fd9680d1bf0 msgr2=0x7fd9680d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:56.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.209+0000 7fd96ec00640 1 -- 172.21.15.47:0/2863633086 shutdown_connections 2024-02-19T15:30:56.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.209+0000 7fd96ec00640 1 -- 172.21.15.47:0/2863633086 wait complete. 2024-02-19T15:30:56.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd96ec00640 1 Processor -- start 2024-02-19T15:30:56.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd96ec00640 1 -- start start 2024-02-19T15:30:56.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd96ec00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9680cedb0 0x7fd9680cab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:56.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd96ec00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9680d2710 0x7fd96806bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:56.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd96ec00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd96815bcd0 0x7fd9680cb600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:56.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd96ec00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96806d4d0 con 0x7fd9680d2710 2024-02-19T15:30:56.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd96ec00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96806c850 con 0x7fd9680cedb0 2024-02-19T15:30:56.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd96ec00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96806cb30 con 0x7fd96815bcd0 2024-02-19T15:30:56.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd967400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9680d2710 0x7fd96806bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:56.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd967400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9680d2710 0x7fd96806bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:47770/0 (socket says 172.21.15.47:47770) 2024-02-19T15:30:56.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd967400640 1 -- 172.21.15.47:0/792079369 learned_addr learned my addr 172.21.15.47:0/792079369 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:56.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd967e00640 1 --2- 172.21.15.47:0/792079369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9680cedb0 0x7fd9680cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:56.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd96cc00640 1 --2- 172.21.15.47:0/792079369 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd96815bcd0 0x7fd9680cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:56.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd967400640 1 -- 172.21.15.47:0/792079369 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd96815bcd0 msgr2=0x7fd9680cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:56.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd967400640 1 --2- 172.21.15.47:0/792079369 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd96815bcd0 0x7fd9680cb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:56.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd967400640 1 -- 172.21.15.47:0/792079369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9680cedb0 msgr2=0x7fd9680cab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:56.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd967400640 1 --2- 172.21.15.47:0/792079369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9680cedb0 0x7fd9680cab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:56.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd967400640 1 -- 172.21.15.47:0/792079369 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9580094d0 con 0x7fd9680d2710 2024-02-19T15:30:56.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd96cc00640 1 --2- 172.21.15.47:0/792079369 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd96815bcd0 0x7fd9680cb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:30:56.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd967400640 1 --2- 172.21.15.47:0/792079369 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9680d2710 0x7fd96806bfc0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fd95400e870 tx=0x7fd95400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:56.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd95fe00640 1 -- 172.21.15.47:0/792079369 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd954009510 con 0x7fd9680d2710 2024-02-19T15:30:56.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd95fe00640 1 -- 172.21.15.47:0/792079369 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95400eec0 con 0x7fd9680d2710 2024-02-19T15:30:56.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd96ec00640 1 -- 172.21.15.47:0/792079369 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9680cdb40 con 0x7fd9680d2710 2024-02-19T15:30:56.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd95fe00640 1 -- 172.21.15.47:0/792079369 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95400cb40 con 0x7fd9680d2710 2024-02-19T15:30:56.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.213+0000 7fd96ec00640 1 -- 172.21.15.47:0/792079369 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96806d0f0 con 0x7fd9680d2710 2024-02-19T15:30:56.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.217+0000 7fd96ec00640 1 -- 172.21.15.47:0/792079369 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd968075360 con 0x7fd9680d2710 2024-02-19T15:30:56.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.217+0000 7fd95fe00640 1 -- 172.21.15.47:0/792079369 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fd95400ccc0 con 0x7fd9680d2710 2024-02-19T15:30:56.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.217+0000 7fd95fe00640 1 --2- 172.21.15.47:0/792079369 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd948066e40 0x7fd948069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:56.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.217+0000 7fd95fe00640 1 -- 172.21.15.47:0/792079369 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd954084ac0 con 0x7fd9680d2710 2024-02-19T15:30:56.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.217+0000 7fd967e00640 1 --2- 172.21.15.47:0/792079369 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd948066e40 0x7fd948069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:56.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.217+0000 7fd967e00640 1 --2- 172.21.15.47:0/792079369 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd948066e40 0x7fd948069300 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd9580af000 tx=0x7fd9580b6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:56.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.221+0000 7fd95fe00640 1 -- 172.21.15.47:0/792079369 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fd954051910 con 0x7fd9680d2710 2024-02-19T15:30:56.234 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:56.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.417+0000 7fd96ec00640 1 -- 172.21.15.47:0/792079369 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9680d1590 con 0x7fd9680d2710 2024-02-19T15:30:56.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.417+0000 7fd95fe00640 1 -- 172.21.15.47:0/792079369 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7fd954021d80 con 0x7fd9680d2710 2024-02-19T15:30:56.417 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:56.417 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:56.417 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:56.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.417+0000 7fd96ec00640 1 -- 172.21.15.47:0/792079369 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd948066e40 msgr2=0x7fd948069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:56.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.417+0000 7fd96ec00640 1 --2- 172.21.15.47:0/792079369 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd948066e40 0x7fd948069300 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd9580af000 tx=0x7fd9580b6040 comp rx=0 tx=0).stop 2024-02-19T15:30:56.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.417+0000 7fd96ec00640 1 -- 172.21.15.47:0/792079369 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9680d2710 msgr2=0x7fd96806bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:56.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.417+0000 7fd96ec00640 1 --2- 172.21.15.47:0/792079369 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9680d2710 0x7fd96806bfc0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fd95400e870 tx=0x7fd95400ec70 comp rx=0 tx=0).stop 2024-02-19T15:30:56.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.417+0000 7fd96ec00640 1 -- 172.21.15.47:0/792079369 shutdown_connections 2024-02-19T15:30:56.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.417+0000 7fd96ec00640 1 --2- 172.21.15.47:0/792079369 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fd948066e40 0x7fd948069300 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:56.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.417+0000 7fd96ec00640 1 --2- 172.21.15.47:0/792079369 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd96815bcd0 0x7fd9680cb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:56.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.417+0000 7fd96ec00640 1 --2- 172.21.15.47:0/792079369 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9680d2710 0x7fd96806bfc0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:56.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.417+0000 7fd96ec00640 1 --2- 172.21.15.47:0/792079369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9680cedb0 0x7fd9680cab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:56.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.417+0000 7fd96ec00640 1 -- 172.21.15.47:0/792079369 >> 172.21.15.47:0/792079369 conn(0x7fd9680d1bf0 msgr2=0x7fd968158910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:56.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.421+0000 7fd96ec00640 1 -- 172.21.15.47:0/792079369 shutdown_connections 2024-02-19T15:30:56.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:56.421+0000 7fd96ec00640 1 -- 172.21.15.47:0/792079369 wait complete. 2024-02-19T15:30:56.639 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:56.639 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:57.640 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:57.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.781+0000 7ff1b3a00640 1 Processor -- start 2024-02-19T15:30:57.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.781+0000 7ff1b3a00640 1 -- start start 2024-02-19T15:30:57.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.781+0000 7ff1b3a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff1ac145590 0x7ff1ac0727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:57.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.781+0000 7ff1b3a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff1ac15c390 0x7ff1ac15c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:57.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.781+0000 7ff1b3a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ac15d280 con 0x7ff1ac0cea80 2024-02-19T15:30:57.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.781+0000 7ff1b3a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1ac070ff0 con 0x7ff1ac145590 2024-02-19T15:30:57.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.781+0000 7ff1b3a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1ac15cd40 con 0x7ff1ac15c390 2024-02-19T15:30:57.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.781+0000 7ff1b1000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff1ac145590 0x7ff1ac0727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:57.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.781+0000 7ff1abe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff1ac0cea80 0x7ff1ac06c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35578/0 (socket says 172.21.15.47:35578) 2024-02-19T15:30:57.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.781+0000 7ff1abe00640 1 -- 172.21.15.47:0/387863290 learned_addr learned my addr 172.21.15.47:0/387863290 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:57.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.781+0000 7ff1b1a00640 1 --2- 172.21.15.47:0/387863290 >> v2:172.21.15.116:3300/0 conn(0x7ff1ac15c390 0x7ff1ac15c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:57.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.781+0000 7ff1b1000640 1 -- 172.21.15.47:0/387863290 >> v2:172.21.15.116:3300/0 conn(0x7ff1ac15c390 msgr2=0x7ff1ac15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:57.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.781+0000 7ff1b1000640 1 --2- 172.21.15.47:0/387863290 >> v2:172.21.15.116:3300/0 conn(0x7ff1ac15c390 0x7ff1ac15c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:57.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.781+0000 7ff1b1000640 1 -- 172.21.15.47:0/387863290 >> v1:172.21.15.47:6789/0 conn(0x7ff1ac0cea80 legacy=0x7ff1ac06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:57.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.781+0000 7ff1ab400640 1 -- 172.21.15.47:0/387863290 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759896763 0 0) 0x7ff1ac15d280 con 0x7ff1ac0cea80 2024-02-19T15:30:57.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.781+0000 7ff1b1000640 1 -- 172.21.15.47:0/387863290 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ac0709e0 con 0x7ff1ac145590 2024-02-19T15:30:57.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.789+0000 7ff1b1000640 1 --2- 172.21.15.47:0/387863290 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ac145590 0x7ff1ac0727b0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac072380 tx=0x7ff1a001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb810d7f3d73ac6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:57.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.789+0000 7ff1ab400640 1 -- 172.21.15.47:0/387863290 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a00ab600 con 0x7ff1ac145590 2024-02-19T15:30:57.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.789+0000 7ff1ab400640 1 -- 172.21.15.47:0/387863290 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a00a7050 con 0x7ff1ac145590 2024-02-19T15:30:57.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.789+0000 7ff1ab400640 1 -- 172.21.15.47:0/387863290 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a00ab980 con 0x7ff1ac145590 2024-02-19T15:30:57.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.789+0000 7ff1b3a00640 1 -- 172.21.15.47:0/387863290 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ac145590 msgr2=0x7ff1ac0727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:57.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.789+0000 7ff1b3a00640 1 --2- 172.21.15.47:0/387863290 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ac145590 0x7ff1ac0727b0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac072380 tx=0x7ff1a001c480 comp rx=0 tx=0).stop 2024-02-19T15:30:57.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.789+0000 7ff1b3a00640 1 -- 172.21.15.47:0/387863290 shutdown_connections 2024-02-19T15:30:57.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.789+0000 7ff1b3a00640 1 --2- 172.21.15.47:0/387863290 >> v2:172.21.15.116:3300/0 conn(0x7ff1ac15c390 0x7ff1ac15c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:57.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.789+0000 7ff1b3a00640 1 --2- 172.21.15.47:0/387863290 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ac145590 0x7ff1ac0727b0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:57.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.789+0000 7ff1b3a00640 1 -- 172.21.15.47:0/387863290 >> 172.21.15.47:0/387863290 conn(0x7ff1ac0b16a0 msgr2=0x7ff1ac0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:57.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.789+0000 7ff1b3a00640 1 -- 172.21.15.47:0/387863290 shutdown_connections 2024-02-19T15:30:57.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.789+0000 7ff1b3a00640 1 -- 172.21.15.47:0/387863290 wait complete. 2024-02-19T15:30:57.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.789+0000 7ff1b3a00640 1 Processor -- start 2024-02-19T15:30:57.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.789+0000 7ff1b3a00640 1 -- start start 2024-02-19T15:30:57.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1b3a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1ac0cea80 0x7ff1ac0c4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:57.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1b3a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1ac145590 0x7ff1ac0b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:57.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1b3a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ac15c390 0x7ff1ac13e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:57.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1b3a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac13eee0 con 0x7ff1ac15c390 2024-02-19T15:30:57.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1b3a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac13d4f0 con 0x7ff1ac145590 2024-02-19T15:30:57.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1b3a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac13ec00 con 0x7ff1ac0cea80 2024-02-19T15:30:57.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1b1a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ac15c390 0x7ff1ac13e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:57.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1b1a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ac15c390 0x7ff1ac13e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:47784/0 (socket says 172.21.15.47:47784) 2024-02-19T15:30:57.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1b1a00640 1 -- 172.21.15.47:0/1448356767 learned_addr learned my addr 172.21.15.47:0/1448356767 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:57.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1abe00640 1 --2- 172.21.15.47:0/1448356767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1ac145590 0x7ff1ac0b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:57.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1b1000640 1 --2- 172.21.15.47:0/1448356767 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1ac0cea80 0x7ff1ac0c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:57.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1b1a00640 1 -- 172.21.15.47:0/1448356767 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1ac0cea80 msgr2=0x7ff1ac0c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:57.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1b1a00640 1 --2- 172.21.15.47:0/1448356767 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1ac0cea80 0x7ff1ac0c4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:57.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1b1a00640 1 -- 172.21.15.47:0/1448356767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1ac145590 msgr2=0x7ff1ac0b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:57.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1b1a00640 1 --2- 172.21.15.47:0/1448356767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1ac145590 0x7ff1ac0b3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:57.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1b1a00640 1 -- 172.21.15.47:0/1448356767 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a0009420 con 0x7ff1ac15c390 2024-02-19T15:30:57.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1b1a00640 1 --2- 172.21.15.47:0/1448356767 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ac15c390 0x7ff1ac13e020 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4007e60 tx=0x7ff1a400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:57.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1a8c00640 1 -- 172.21.15.47:0/1448356767 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4014420 con 0x7ff1ac15c390 2024-02-19T15:30:57.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1a8c00640 1 -- 172.21.15.47:0/1448356767 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a40145a0 con 0x7ff1ac15c390 2024-02-19T15:30:57.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1a8c00640 1 -- 172.21.15.47:0/1448356767 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4014950 con 0x7ff1ac15c390 2024-02-19T15:30:57.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1b3a00640 1 -- 172.21.15.47:0/1448356767 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1ac142290 con 0x7ff1ac15c390 2024-02-19T15:30:57.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1b3a00640 1 -- 172.21.15.47:0/1448356767 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1ac142850 con 0x7ff1ac15c390 2024-02-19T15:30:57.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.793+0000 7ff1b3a00640 1 -- 172.21.15.47:0/1448356767 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1ac075360 con 0x7ff1ac15c390 2024-02-19T15:30:57.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.797+0000 7ff1a8c00640 1 -- 172.21.15.47:0/1448356767 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff1a4014ab0 con 0x7ff1ac15c390 2024-02-19T15:30:57.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.797+0000 7ff1a8c00640 1 --2- 172.21.15.47:0/1448356767 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff18c066e40 0x7ff18c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:57.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.797+0000 7ff1a8c00640 1 -- 172.21.15.47:0/1448356767 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff1a4085210 con 0x7ff1ac15c390 2024-02-19T15:30:57.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.797+0000 7ff1b1000640 1 --2- 172.21.15.47:0/1448356767 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff18c066e40 0x7ff18c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:57.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.797+0000 7ff1b1000640 1 --2- 172.21.15.47:0/1448356767 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff18c066e40 0x7ff18c069300 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff1a00ad000 tx=0x7ff1a00a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:57.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.801+0000 7ff1a8c00640 1 -- 172.21.15.47:0/1448356767 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff1a4052060 con 0x7ff1ac15c390 2024-02-19T15:30:57.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.997+0000 7ff1b3a00640 1 -- 172.21.15.47:0/1448356767 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1ac0cf630 con 0x7ff1ac15c390 2024-02-19T15:30:57.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.997+0000 7ff1a8c00640 1 -- 172.21.15.47:0/1448356767 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7ff1a4055720 con 0x7ff1ac15c390 2024-02-19T15:30:57.996 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:57.997 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:57.997 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:57.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.997+0000 7ff1b3a00640 1 -- 172.21.15.47:0/1448356767 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff18c066e40 msgr2=0x7ff18c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:57.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.997+0000 7ff1b3a00640 1 --2- 172.21.15.47:0/1448356767 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff18c066e40 0x7ff18c069300 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff1a00ad000 tx=0x7ff1a00a6000 comp rx=0 tx=0).stop 2024-02-19T15:30:57.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.997+0000 7ff1b3a00640 1 -- 172.21.15.47:0/1448356767 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ac15c390 msgr2=0x7ff1ac13e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:57.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.997+0000 7ff1b3a00640 1 --2- 172.21.15.47:0/1448356767 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ac15c390 0x7ff1ac13e020 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4007e60 tx=0x7ff1a400e4b0 comp rx=0 tx=0).stop 2024-02-19T15:30:57.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.997+0000 7ff1b3a00640 1 -- 172.21.15.47:0/1448356767 shutdown_connections 2024-02-19T15:30:57.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.997+0000 7ff1b3a00640 1 --2- 172.21.15.47:0/1448356767 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff18c066e40 0x7ff18c069300 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:57.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.997+0000 7ff1b3a00640 1 --2- 172.21.15.47:0/1448356767 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff1ac15c390 0x7ff1ac13e020 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:57.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.997+0000 7ff1b3a00640 1 --2- 172.21.15.47:0/1448356767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1ac145590 0x7ff1ac0b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:57.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.997+0000 7ff1b3a00640 1 --2- 172.21.15.47:0/1448356767 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff1ac0cea80 0x7ff1ac0c4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:57.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.997+0000 7ff1b3a00640 1 -- 172.21.15.47:0/1448356767 >> 172.21.15.47:0/1448356767 conn(0x7ff1ac0b16a0 msgr2=0x7ff1ac15b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:57.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.997+0000 7ff1b3a00640 1 -- 172.21.15.47:0/1448356767 shutdown_connections 2024-02-19T15:30:57.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:57.997+0000 7ff1b3a00640 1 -- 172.21.15.47:0/1448356767 wait complete. 2024-02-19T15:30:58.220 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:58.220 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:30:58.237 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:30:59.221 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:30:59.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.361+0000 7f6f4fe00640 1 Processor -- start 2024-02-19T15:30:59.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.361+0000 7f6f4fe00640 1 -- start start 2024-02-19T15:30:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.365+0000 7f6f4fe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6f5006d120 0x7f6f5006d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.365+0000 7f6f4fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f500d19e0 0x7f6f500d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.365+0000 7f6f4fe00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f500d2370 con 0x7f6f5015dd70 2024-02-19T15:30:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.365+0000 7f6f4fe00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f5006daf0 con 0x7f6f5006d120 2024-02-19T15:30:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.365+0000 7f6f4fe00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5015f3f0 con 0x7f6f500d19e0 2024-02-19T15:30:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.365+0000 7f6f4ea00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6f5015dd70 0x7f6f500ae860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:35594/0 (socket says 172.21.15.47:35594) 2024-02-19T15:30:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.365+0000 7f6f4ea00640 1 -- 172.21.15.47:0/1683264961 learned_addr learned my addr 172.21.15.47:0/1683264961 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:30:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.365+0000 7f6f4f400640 1 --2- 172.21.15.47:0/1683264961 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f500d19e0 0x7f6f500d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.365+0000 7f6f46000640 1 --2- 172.21.15.47:0/1683264961 >> v2:172.21.15.116:3300/0 conn(0x7f6f5006d120 0x7f6f5006d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:59.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.365+0000 7f6f4e000640 1 -- 172.21.15.47:0/1683264961 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1758117110 0 0) 0x7f6f400014a0 con 0x7f6f5015dd70 2024-02-19T15:30:59.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.365+0000 7f6f4e000640 1 -- 172.21.15.47:0/1683264961 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f34003530 con 0x7f6f5015dd70 2024-02-19T15:30:59.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.365+0000 7f6f4f400640 1 -- 172.21.15.47:0/1683264961 >> v2:172.21.15.116:3300/0 conn(0x7f6f5006d120 msgr2=0x7f6f5006d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:30:59.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.365+0000 7f6f4f400640 1 --2- 172.21.15.47:0/1683264961 >> v2:172.21.15.116:3300/0 conn(0x7f6f5006d120 0x7f6f5006d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:59.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.365+0000 7f6f4f400640 1 -- 172.21.15.47:0/1683264961 >> v1:172.21.15.47:6789/0 conn(0x7f6f5015dd70 legacy=0x7f6f500ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:59.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.365+0000 7f6f4f400640 1 -- 172.21.15.47:0/1683264961 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5006c3e0 con 0x7f6f500d19e0 2024-02-19T15:30:59.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.369+0000 7f6f4f400640 1 --2- 172.21.15.47:0/1683264961 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f500d19e0 0x7f6f500d1e30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6f480097c0 tx=0x7f6f4801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=158f5580142f8ec4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:59.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.369+0000 7f6f4e000640 1 -- 172.21.15.47:0/1683264961 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f480ab600 con 0x7f6f500d19e0 2024-02-19T15:30:59.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.369+0000 7f6f4e000640 1 -- 172.21.15.47:0/1683264961 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f480a7050 con 0x7f6f500d19e0 2024-02-19T15:30:59.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.369+0000 7f6f4e000640 1 -- 172.21.15.47:0/1683264961 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f480ab980 con 0x7f6f500d19e0 2024-02-19T15:30:59.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.373+0000 7f6f4fe00640 1 -- 172.21.15.47:0/1683264961 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f500d19e0 msgr2=0x7f6f500d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:59.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.373+0000 7f6f4fe00640 1 --2- 172.21.15.47:0/1683264961 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f500d19e0 0x7f6f500d1e30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6f480097c0 tx=0x7f6f4801c410 comp rx=0 tx=0).stop 2024-02-19T15:30:59.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.373+0000 7f6f4fe00640 1 -- 172.21.15.47:0/1683264961 shutdown_connections 2024-02-19T15:30:59.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.373+0000 7f6f4fe00640 1 --2- 172.21.15.47:0/1683264961 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f500d19e0 0x7f6f500d1e30 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:59.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.373+0000 7f6f4fe00640 1 --2- 172.21.15.47:0/1683264961 >> v2:172.21.15.116:3300/0 conn(0x7f6f5006d120 0x7f6f5006d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:59.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.373+0000 7f6f4fe00640 1 -- 172.21.15.47:0/1683264961 >> 172.21.15.47:0/1683264961 conn(0x7f6f5009a9d0 msgr2=0x7f6f5009ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:59.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.373+0000 7f6f4fe00640 1 -- 172.21.15.47:0/1683264961 shutdown_connections 2024-02-19T15:30:59.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.373+0000 7f6f4fe00640 1 -- 172.21.15.47:0/1683264961 wait complete. 2024-02-19T15:30:59.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.373+0000 7f6f4fe00640 1 Processor -- start 2024-02-19T15:30:59.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.373+0000 7f6f4fe00640 1 -- start start 2024-02-19T15:30:59.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f4fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f5006d120 0x7f6f5015b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:59.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f4fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f500d19e0 0x7f6f50153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:59.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f4fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f5015dd70 0x7f6f500af740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:59.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f4fe00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5015d260 con 0x7f6f500d19e0 2024-02-19T15:30:59.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f4fe00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5015c1c0 con 0x7f6f5015dd70 2024-02-19T15:30:59.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f4fe00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5015c4a0 con 0x7f6f5006d120 2024-02-19T15:30:59.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f46000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f500d19e0 0x7f6f50153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:59.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f46000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f500d19e0 0x7f6f50153e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:47800/0 (socket says 172.21.15.47:47800) 2024-02-19T15:30:59.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f46000640 1 -- 172.21.15.47:0/4142923464 learned_addr learned my addr 172.21.15.47:0/4142923464 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:30:59.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f4ea00640 1 --2- 172.21.15.47:0/4142923464 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f5006d120 0x7f6f5015b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:59.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f4f400640 1 --2- 172.21.15.47:0/4142923464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f5015dd70 0x7f6f500af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:59.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f46000640 1 -- 172.21.15.47:0/4142923464 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f5006d120 msgr2=0x7f6f5015b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:59.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f46000640 1 --2- 172.21.15.47:0/4142923464 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f5006d120 0x7f6f5015b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:59.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f46000640 1 -- 172.21.15.47:0/4142923464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f5015dd70 msgr2=0x7f6f500af740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:59.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f46000640 1 --2- 172.21.15.47:0/4142923464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f5015dd70 0x7f6f500af740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:59.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f46000640 1 -- 172.21.15.47:0/4142923464 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f48009420 con 0x7f6f500d19e0 2024-02-19T15:30:59.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f46000640 1 --2- 172.21.15.47:0/4142923464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f500d19e0 0x7f6f50153e50 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f6f3800e930 tx=0x7f6f3800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:59.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f47400640 1 -- 172.21.15.47:0/4142923464 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f38009690 con 0x7f6f500d19e0 2024-02-19T15:30:59.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f47400640 1 -- 172.21.15.47:0/4142923464 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3800ca00 con 0x7f6f500d19e0 2024-02-19T15:30:59.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f47400640 1 -- 172.21.15.47:0/4142923464 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3800cd70 con 0x7f6f500d19e0 2024-02-19T15:30:59.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f4fe00640 1 -- 172.21.15.47:0/4142923464 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f5015c780 con 0x7f6f500d19e0 2024-02-19T15:30:59.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f4fe00640 1 -- 172.21.15.47:0/4142923464 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f5015cd40 con 0x7f6f500d19e0 2024-02-19T15:30:59.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f4fe00640 1 -- 172.21.15.47:0/4142923464 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f50075360 con 0x7f6f500d19e0 2024-02-19T15:30:59.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.377+0000 7f6f47400640 1 -- 172.21.15.47:0/4142923464 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6f38013470 con 0x7f6f500d19e0 2024-02-19T15:30:59.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.381+0000 7f6f47400640 1 --2- 172.21.15.47:0/4142923464 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6f2c066e40 0x7f6f2c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:30:59.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.381+0000 7f6f47400640 1 -- 172.21.15.47:0/4142923464 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6f38084da0 con 0x7f6f500d19e0 2024-02-19T15:30:59.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.381+0000 7f6f4ea00640 1 --2- 172.21.15.47:0/4142923464 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6f2c066e40 0x7f6f2c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:30:59.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.381+0000 7f6f4ea00640 1 --2- 172.21.15.47:0/4142923464 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6f2c066e40 0x7f6f2c069300 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6f40005c10 tx=0x7f6f40009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:30:59.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.385+0000 7f6f47400640 1 -- 172.21.15.47:0/4142923464 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6f38051b70 con 0x7f6f500d19e0 2024-02-19T15:30:59.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.581+0000 7f6f4fe00640 1 -- 172.21.15.47:0/4142923464 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f500d27c0 con 0x7f6f500d19e0 2024-02-19T15:30:59.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.581+0000 7f6f47400640 1 -- 172.21.15.47:0/4142923464 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f6f38021d80 con 0x7f6f500d19e0 2024-02-19T15:30:59.579 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:30:59.579 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:30:59.580 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:30:59.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.581+0000 7f6f4fe00640 1 -- 172.21.15.47:0/4142923464 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6f2c066e40 msgr2=0x7f6f2c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:59.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.581+0000 7f6f4fe00640 1 --2- 172.21.15.47:0/4142923464 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6f2c066e40 0x7f6f2c069300 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6f40005c10 tx=0x7f6f40009040 comp rx=0 tx=0).stop 2024-02-19T15:30:59.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.581+0000 7f6f4fe00640 1 -- 172.21.15.47:0/4142923464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f500d19e0 msgr2=0x7f6f50153e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:30:59.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.581+0000 7f6f4fe00640 1 --2- 172.21.15.47:0/4142923464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f500d19e0 0x7f6f50153e50 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f6f3800e930 tx=0x7f6f3800edf0 comp rx=0 tx=0).stop 2024-02-19T15:30:59.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.581+0000 7f6f4fe00640 1 -- 172.21.15.47:0/4142923464 shutdown_connections 2024-02-19T15:30:59.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.581+0000 7f6f4fe00640 1 --2- 172.21.15.47:0/4142923464 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6f2c066e40 0x7f6f2c069300 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:59.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.581+0000 7f6f4fe00640 1 --2- 172.21.15.47:0/4142923464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f5015dd70 0x7f6f500af740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:59.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.581+0000 7f6f4fe00640 1 --2- 172.21.15.47:0/4142923464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f500d19e0 0x7f6f50153e50 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:59.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.581+0000 7f6f4fe00640 1 --2- 172.21.15.47:0/4142923464 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f5006d120 0x7f6f5015b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:30:59.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.581+0000 7f6f4fe00640 1 -- 172.21.15.47:0/4142923464 >> 172.21.15.47:0/4142923464 conn(0x7f6f5009a9d0 msgr2=0x7f6f5015a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:30:59.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.581+0000 7f6f4fe00640 1 -- 172.21.15.47:0/4142923464 shutdown_connections 2024-02-19T15:30:59.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:30:59.581+0000 7f6f4fe00640 1 -- 172.21.15.47:0/4142923464 wait complete. 2024-02-19T15:30:59.809 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:30:59.809 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:00.239 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:00.809 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:00.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.949+0000 7f1c3fc00640 1 Processor -- start 2024-02-19T15:31:00.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.949+0000 7f1c3fc00640 1 -- start start 2024-02-19T15:31:00.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.949+0000 7f1c3fc00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1c3815bef0 0x7f1c380b2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:00.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.949+0000 7f1c3fc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c380b5d90 0x7f1c380d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:00.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.949+0000 7f1c3fc00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1c380d0df0 0x7f1c3806b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:00.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.949+0000 7f1c3fc00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c380b2d40 con 0x7f1c3815bef0 2024-02-19T15:31:00.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.949+0000 7f1c3fc00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c380d00f0 con 0x7f1c380d0df0 2024-02-19T15:31:00.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.949+0000 7f1c3fc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c380af690 con 0x7f1c380b5d90 2024-02-19T15:31:00.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.953+0000 7f1c3d200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1c3815bef0 0x7f1c380b2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:00.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.953+0000 7f1c3d200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1c3815bef0 0x7f1c380b2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:47808/0 (socket says 172.21.15.47:47808) 2024-02-19T15:31:00.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.953+0000 7f1c3d200640 1 -- 172.21.15.47:0/137850568 learned_addr learned my addr 172.21.15.47:0/137850568 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:00.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.953+0000 7f1c3dc00640 1 --2- 172.21.15.47:0/137850568 >> v2:172.21.15.116:3300/0 conn(0x7f1c380d0df0 0x7f1c3806b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:00.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.953+0000 7f1c2fe00640 1 --2- 172.21.15.47:0/137850568 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c380b5d90 0x7f1c380d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:00.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.953+0000 7f1c3d200640 1 -- 172.21.15.47:0/137850568 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c380b5d90 msgr2=0x7f1c380d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:00.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.953+0000 7f1c3d200640 1 --2- 172.21.15.47:0/137850568 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c380b5d90 0x7f1c380d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:00.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.953+0000 7f1c3d200640 1 -- 172.21.15.47:0/137850568 >> v2:172.21.15.116:3300/0 conn(0x7f1c380d0df0 msgr2=0x7f1c3806b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:00.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.953+0000 7f1c3d200640 1 --2- 172.21.15.47:0/137850568 >> v2:172.21.15.116:3300/0 conn(0x7f1c380d0df0 0x7f1c3806b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:00.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.953+0000 7f1c3d200640 1 -- 172.21.15.47:0/137850568 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c380af970 con 0x7f1c3815bef0 2024-02-19T15:31:00.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.953+0000 7f1c3d200640 1 --2- 172.21.15.47:0/137850568 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c3815bef0 0x7f1c380b2800 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f1c28009870 tx=0x7f1c2801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=293a1c29a7360e15 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:00.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.953+0000 7f1c37e00640 1 -- 172.21.15.47:0/137850568 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c280ab600 con 0x7f1c3815bef0 2024-02-19T15:31:00.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.953+0000 7f1c37e00640 1 -- 172.21.15.47:0/137850568 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c280a7050 con 0x7f1c3815bef0 2024-02-19T15:31:00.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.953+0000 7f1c37e00640 1 -- 172.21.15.47:0/137850568 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c280ab980 con 0x7f1c3815bef0 2024-02-19T15:31:00.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3fc00640 1 -- 172.21.15.47:0/137850568 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c3815bef0 msgr2=0x7f1c380b2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:00.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3fc00640 1 --2- 172.21.15.47:0/137850568 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c3815bef0 0x7f1c380b2800 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f1c28009870 tx=0x7f1c2801c460 comp rx=0 tx=0).stop 2024-02-19T15:31:00.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3fc00640 1 -- 172.21.15.47:0/137850568 shutdown_connections 2024-02-19T15:31:00.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3fc00640 1 --2- 172.21.15.47:0/137850568 >> v2:172.21.15.116:3300/0 conn(0x7f1c380d0df0 0x7f1c3806b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:00.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3fc00640 1 --2- 172.21.15.47:0/137850568 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c380b5d90 0x7f1c380d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:00.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3fc00640 1 --2- 172.21.15.47:0/137850568 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c3815bef0 0x7f1c380b2800 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:00.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3fc00640 1 -- 172.21.15.47:0/137850568 >> 172.21.15.47:0/137850568 conn(0x7f1c380b4fc0 msgr2=0x7f1c380b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:00.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3fc00640 1 -- 172.21.15.47:0/137850568 shutdown_connections 2024-02-19T15:31:00.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3fc00640 1 -- 172.21.15.47:0/137850568 wait complete. 2024-02-19T15:31:00.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3fc00640 1 Processor -- start 2024-02-19T15:31:00.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3fc00640 1 -- start start 2024-02-19T15:31:00.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3fc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c380b5d90 0x7f1c3814a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:00.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3fc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c380d0df0 0x7f1c3814f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3fc00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c3815bef0 0x7f1c38158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3fc00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c381575e0 con 0x7f1c380d0df0 2024-02-19T15:31:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3fc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c381578a0 con 0x7f1c380b5d90 2024-02-19T15:31:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3fc00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c38157b80 con 0x7f1c3815bef0 2024-02-19T15:31:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c2fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c380d0df0 0x7f1c3814f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c2fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c380d0df0 0x7f1c3814f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:47822/0 (socket says 172.21.15.47:47822) 2024-02-19T15:31:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c2fe00640 1 -- 172.21.15.47:0/2298627260 learned_addr learned my addr 172.21.15.47:0/2298627260 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3dc00640 1 --2- 172.21.15.47:0/2298627260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c3815bef0 0x7f1c38158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3d200640 1 --2- 172.21.15.47:0/2298627260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c380b5d90 0x7f1c3814a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c2fe00640 1 -- 172.21.15.47:0/2298627260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c3815bef0 msgr2=0x7f1c38158660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c2fe00640 1 --2- 172.21.15.47:0/2298627260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c3815bef0 0x7f1c38158660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c2fe00640 1 -- 172.21.15.47:0/2298627260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c380b5d90 msgr2=0x7f1c3814a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c2fe00640 1 --2- 172.21.15.47:0/2298627260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c380b5d90 0x7f1c3814a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c2fe00640 1 -- 172.21.15.47:0/2298627260 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c280094d0 con 0x7f1c380d0df0 2024-02-19T15:31:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3d200640 1 --2- 172.21.15.47:0/2298627260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c380b5d90 0x7f1c3814a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:31:00.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.957+0000 7f1c3dc00640 1 --2- 172.21.15.47:0/2298627260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c3815bef0 0x7f1c38158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:31:00.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.961+0000 7f1c2fe00640 1 --2- 172.21.15.47:0/2298627260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c380d0df0 0x7f1c3814f330 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000d800 tx=0x7f1c2000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:00.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.961+0000 7f1c35600640 1 -- 172.21.15.47:0/2298627260 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c200096a0 con 0x7f1c380d0df0 2024-02-19T15:31:00.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.961+0000 7f1c35600640 1 -- 172.21.15.47:0/2298627260 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c20009800 con 0x7f1c380d0df0 2024-02-19T15:31:00.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.961+0000 7f1c3fc00640 1 -- 172.21.15.47:0/2298627260 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c38157e60 con 0x7f1c380d0df0 2024-02-19T15:31:00.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.961+0000 7f1c35600640 1 -- 172.21.15.47:0/2298627260 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c20009b70 con 0x7f1c380d0df0 2024-02-19T15:31:00.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.961+0000 7f1c3fc00640 1 -- 172.21.15.47:0/2298627260 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c3814a9b0 con 0x7f1c380d0df0 2024-02-19T15:31:00.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.961+0000 7f1c3fc00640 1 -- 172.21.15.47:0/2298627260 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c38075360 con 0x7f1c380d0df0 2024-02-19T15:31:00.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.961+0000 7f1c35600640 1 -- 172.21.15.47:0/2298627260 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1c20009cd0 con 0x7f1c380d0df0 2024-02-19T15:31:00.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.961+0000 7f1c35600640 1 --2- 172.21.15.47:0/2298627260 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1c14066e40 0x7f1c14069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:00.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.961+0000 7f1c35600640 1 -- 172.21.15.47:0/2298627260 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1c20085290 con 0x7f1c380d0df0 2024-02-19T15:31:00.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.961+0000 7f1c3d200640 1 --2- 172.21.15.47:0/2298627260 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1c14066e40 0x7f1c14069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:00.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.965+0000 7f1c3d200640 1 --2- 172.21.15.47:0/2298627260 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1c14066e40 0x7f1c14069300 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f1c280ad000 tx=0x7f1c280a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:00.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:00.973+0000 7f1c35600640 1 -- 172.21.15.47:0/2298627260 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1c20051fa0 con 0x7f1c380d0df0 2024-02-19T15:31:01.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:01.165+0000 7f1c3fc00640 1 -- 172.21.15.47:0/2298627260 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c38074650 con 0x7f1c380d0df0 2024-02-19T15:31:01.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:01.165+0000 7f1c35600640 1 -- 172.21.15.47:0/2298627260 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f1c20055660 con 0x7f1c380d0df0 2024-02-19T15:31:01.167 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:01.167 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:01.167 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:31:01.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:01.169+0000 7f1c3fc00640 1 -- 172.21.15.47:0/2298627260 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1c14066e40 msgr2=0x7f1c14069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:01.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:01.169+0000 7f1c3fc00640 1 --2- 172.21.15.47:0/2298627260 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1c14066e40 0x7f1c14069300 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f1c280ad000 tx=0x7f1c280a6000 comp rx=0 tx=0).stop 2024-02-19T15:31:01.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:01.169+0000 7f1c3fc00640 1 -- 172.21.15.47:0/2298627260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c380d0df0 msgr2=0x7f1c3814f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:01.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:01.169+0000 7f1c3fc00640 1 --2- 172.21.15.47:0/2298627260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c380d0df0 0x7f1c3814f330 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000d800 tx=0x7f1c2000dcc0 comp rx=0 tx=0).stop 2024-02-19T15:31:01.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:01.169+0000 7f1c3fc00640 1 -- 172.21.15.47:0/2298627260 shutdown_connections 2024-02-19T15:31:01.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:01.169+0000 7f1c3fc00640 1 --2- 172.21.15.47:0/2298627260 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1c14066e40 0x7f1c14069300 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:01.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:01.169+0000 7f1c3fc00640 1 --2- 172.21.15.47:0/2298627260 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c3815bef0 0x7f1c38158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:01.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:01.169+0000 7f1c3fc00640 1 --2- 172.21.15.47:0/2298627260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c380d0df0 0x7f1c3814f330 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:01.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:01.169+0000 7f1c3fc00640 1 --2- 172.21.15.47:0/2298627260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c380b5d90 0x7f1c3814a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:01.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:01.169+0000 7f1c3fc00640 1 -- 172.21.15.47:0/2298627260 >> 172.21.15.47:0/2298627260 conn(0x7f1c380b4fc0 msgr2=0x7f1c38165600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:01.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:01.169+0000 7f1c3fc00640 1 -- 172.21.15.47:0/2298627260 shutdown_connections 2024-02-19T15:31:01.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:01.169+0000 7f1c3fc00640 1 -- 172.21.15.47:0/2298627260 wait complete. 2024-02-19T15:31:01.397 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:01.397 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:02.241 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:02.398 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:02.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.537+0000 7fbc1f200640 1 Processor -- start 2024-02-19T15:31:02.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.537+0000 7fbc1f200640 1 -- start start 2024-02-19T15:31:02.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.541+0000 7fbc1f200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc18145590 0x7fbc180727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:02.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.541+0000 7fbc1f200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbc180cea80 0x7fbc1806c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:02.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.541+0000 7fbc1f200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc1815d280 con 0x7fbc1815c390 2024-02-19T15:31:02.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.541+0000 7fbc1f200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc18070ff0 con 0x7fbc180cea80 2024-02-19T15:31:02.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.541+0000 7fbc1f200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc180709e0 con 0x7fbc18145590 2024-02-19T15:31:02.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.541+0000 7fbc0fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbc180cea80 0x7fbc1806c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:02.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.541+0000 7fbc0fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbc180cea80 0x7fbc1806c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:47826/0 (socket says 172.21.15.47:47826) 2024-02-19T15:31:02.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.541+0000 7fbc0fe00640 1 -- 172.21.15.47:0/3769017082 learned_addr learned my addr 172.21.15.47:0/3769017082 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:02.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.541+0000 7fbc17e00640 1 --2- 172.21.15.47:0/3769017082 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc18145590 0x7fbc180727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:02.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.541+0000 7fbc0fe00640 1 -- 172.21.15.47:0/3769017082 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc18145590 msgr2=0x7fbc180727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:02.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.541+0000 7fbc0fe00640 1 --2- 172.21.15.47:0/3769017082 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc18145590 0x7fbc180727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:02.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.541+0000 7fbc0fe00640 1 -- 172.21.15.47:0/3769017082 >> v1:172.21.15.116:6789/0 conn(0x7fbc1815c390 legacy=0x7fbc1815c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:02.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.541+0000 7fbc0fe00640 1 -- 172.21.15.47:0/3769017082 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc1806cee0 con 0x7fbc180cea80 2024-02-19T15:31:02.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.541+0000 7fbc17e00640 1 --2- 172.21.15.47:0/3769017082 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc18145590 0x7fbc180727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:31:02.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.545+0000 7fbc0fe00640 1 --2- 172.21.15.47:0/3769017082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc180cea80 0x7fbc1806c9a0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fbc000098a0 tx=0x7fbc0001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af1de5d16b4163e0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:02.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.545+0000 7fbc17400640 1 -- 172.21.15.47:0/3769017082 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc000a7070 con 0x7fbc180cea80 2024-02-19T15:31:02.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.545+0000 7fbc17400640 1 -- 172.21.15.47:0/3769017082 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc000ab430 con 0x7fbc180cea80 2024-02-19T15:31:02.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.545+0000 7fbc17400640 1 -- 172.21.15.47:0/3769017082 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc000ab7c0 con 0x7fbc180cea80 2024-02-19T15:31:02.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.545+0000 7fbc1f200640 1 -- 172.21.15.47:0/3769017082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc180cea80 msgr2=0x7fbc1806c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:02.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.545+0000 7fbc1f200640 1 --2- 172.21.15.47:0/3769017082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc180cea80 0x7fbc1806c9a0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fbc000098a0 tx=0x7fbc0001c540 comp rx=0 tx=0).stop 2024-02-19T15:31:02.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.545+0000 7fbc1f200640 1 -- 172.21.15.47:0/3769017082 shutdown_connections 2024-02-19T15:31:02.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.545+0000 7fbc1f200640 1 --2- 172.21.15.47:0/3769017082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc180cea80 0x7fbc1806c9a0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:02.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.545+0000 7fbc1f200640 1 --2- 172.21.15.47:0/3769017082 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc18145590 0x7fbc180727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:02.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.545+0000 7fbc1f200640 1 -- 172.21.15.47:0/3769017082 >> 172.21.15.47:0/3769017082 conn(0x7fbc180b16a0 msgr2=0x7fbc180b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:02.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.545+0000 7fbc1f200640 1 -- 172.21.15.47:0/3769017082 shutdown_connections 2024-02-19T15:31:02.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.545+0000 7fbc1f200640 1 -- 172.21.15.47:0/3769017082 wait complete. 2024-02-19T15:31:02.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.549+0000 7fbc1f200640 1 Processor -- start 2024-02-19T15:31:02.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.549+0000 7fbc1f200640 1 -- start start 2024-02-19T15:31:02.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.549+0000 7fbc1f200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc180cea80 0x7fbc180b4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:02.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.549+0000 7fbc1f200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc18145590 0x7fbc1813f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:02.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc1f200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc1815c390 0x7fbc1815add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:02.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc1f200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc181439b0 con 0x7fbc1815c390 2024-02-19T15:31:02.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc1f200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc18142da0 con 0x7fbc180cea80 2024-02-19T15:31:02.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc1f200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc18143080 con 0x7fbc18145590 2024-02-19T15:31:02.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc1d200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc1815c390 0x7fbc1815add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:02.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc1d200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc1815c390 0x7fbc1815add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:47836/0 (socket says 172.21.15.47:47836) 2024-02-19T15:31:02.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc17e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc180cea80 0x7fbc180b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:02.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc1d200640 1 -- 172.21.15.47:0/2954547799 learned_addr learned my addr 172.21.15.47:0/2954547799 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:02.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc0fe00640 1 --2- 172.21.15.47:0/2954547799 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc18145590 0x7fbc1813f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:02.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc1d200640 1 -- 172.21.15.47:0/2954547799 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc18145590 msgr2=0x7fbc1813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:02.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc1d200640 1 --2- 172.21.15.47:0/2954547799 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc18145590 0x7fbc1813f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:02.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc1d200640 1 -- 172.21.15.47:0/2954547799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc180cea80 msgr2=0x7fbc180b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:02.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc1d200640 1 --2- 172.21.15.47:0/2954547799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc180cea80 0x7fbc180b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:02.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc1d200640 1 -- 172.21.15.47:0/2954547799 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc000094d0 con 0x7fbc1815c390 2024-02-19T15:31:02.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc1d200640 1 --2- 172.21.15.47:0/2954547799 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc1815c390 0x7fbc1815add0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fbc1000c730 tx=0x7fbc1000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:02.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc14c00640 1 -- 172.21.15.47:0/2954547799 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc10003b80 con 0x7fbc1815c390 2024-02-19T15:31:02.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc14c00640 1 -- 172.21.15.47:0/2954547799 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc10003ce0 con 0x7fbc1815c390 2024-02-19T15:31:02.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc14c00640 1 -- 172.21.15.47:0/2954547799 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1000aa30 con 0x7fbc1815c390 2024-02-19T15:31:02.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc1f200640 1 -- 172.21.15.47:0/2954547799 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc18143360 con 0x7fbc1815c390 2024-02-19T15:31:02.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc1f200640 1 -- 172.21.15.47:0/2954547799 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc1815bcd0 con 0x7fbc1815c390 2024-02-19T15:31:02.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc1f200640 1 -- 172.21.15.47:0/2954547799 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc18143b20 con 0x7fbc1815c390 2024-02-19T15:31:02.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.553+0000 7fbc14c00640 1 -- 172.21.15.47:0/2954547799 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fbc1000ab90 con 0x7fbc1815c390 2024-02-19T15:31:02.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.557+0000 7fbc14c00640 1 --2- 172.21.15.47:0/2954547799 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbbf4066e70 0x7fbbf4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:02.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.557+0000 7fbc14c00640 1 -- 172.21.15.47:0/2954547799 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbc10084730 con 0x7fbc1815c390 2024-02-19T15:31:02.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.557+0000 7fbc17e00640 1 --2- 172.21.15.47:0/2954547799 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbbf4066e70 0x7fbbf4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:02.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.557+0000 7fbc17e00640 1 --2- 172.21.15.47:0/2954547799 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbbf4066e70 0x7fbbf4069330 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fbc1813fb00 tx=0x7fbc08008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:02.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.561+0000 7fbc14c00640 1 -- 172.21.15.47:0/2954547799 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fbc100514f0 con 0x7fbc1815c390 2024-02-19T15:31:02.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.753+0000 7fbc1f200640 1 -- 172.21.15.47:0/2954547799 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc18075360 con 0x7fbc1815c390 2024-02-19T15:31:02.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.757+0000 7fbc14c00640 1 -- 172.21.15.47:0/2954547799 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7fbc10054bb0 con 0x7fbc1815c390 2024-02-19T15:31:02.755 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:02.755 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:02.756 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:31:02.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.757+0000 7fbc1f200640 1 -- 172.21.15.47:0/2954547799 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbbf4066e70 msgr2=0x7fbbf4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:02.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.757+0000 7fbc1f200640 1 --2- 172.21.15.47:0/2954547799 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbbf4066e70 0x7fbbf4069330 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fbc1813fb00 tx=0x7fbc08008040 comp rx=0 tx=0).stop 2024-02-19T15:31:02.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.757+0000 7fbc1f200640 1 -- 172.21.15.47:0/2954547799 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc1815c390 msgr2=0x7fbc1815add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:02.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.757+0000 7fbc1f200640 1 --2- 172.21.15.47:0/2954547799 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc1815c390 0x7fbc1815add0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fbc1000c730 tx=0x7fbc1000cbf0 comp rx=0 tx=0).stop 2024-02-19T15:31:02.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.757+0000 7fbc1f200640 1 -- 172.21.15.47:0/2954547799 shutdown_connections 2024-02-19T15:31:02.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.757+0000 7fbc1f200640 1 --2- 172.21.15.47:0/2954547799 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fbbf4066e70 0x7fbbf4069330 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:02.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.757+0000 7fbc1f200640 1 --2- 172.21.15.47:0/2954547799 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc1815c390 0x7fbc1815add0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:02.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.757+0000 7fbc1f200640 1 --2- 172.21.15.47:0/2954547799 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc18145590 0x7fbc1813f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:02.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.757+0000 7fbc1f200640 1 --2- 172.21.15.47:0/2954547799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc180cea80 0x7fbc180b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:02.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.757+0000 7fbc1f200640 1 -- 172.21.15.47:0/2954547799 >> 172.21.15.47:0/2954547799 conn(0x7fbc180b16a0 msgr2=0x7fbc180ae250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:02.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.757+0000 7fbc1f200640 1 -- 172.21.15.47:0/2954547799 shutdown_connections 2024-02-19T15:31:02.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:02.757+0000 7fbc1f200640 1 -- 172.21.15.47:0/2954547799 wait complete. 2024-02-19T15:31:02.978 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:02.978 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:03.979 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:04.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.121+0000 7f21e1000640 1 Processor -- start 2024-02-19T15:31:04.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.121+0000 7f21e1000640 1 -- start start 2024-02-19T15:31:04.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.121+0000 7f21e1000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21dc06fdf0 0x7f21dc0adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:04.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.121+0000 7f21e1000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f21dc06e7c0 0x7f21dc06aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:04.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.121+0000 7f21e1000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21dc161780 con 0x7f21dc1613a0 2024-02-19T15:31:04.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.121+0000 7f21e1000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21dc0b01a0 con 0x7f21dc06e7c0 2024-02-19T15:31:04.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.121+0000 7f21e1000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc0ae3c0 con 0x7f21dc06fdf0 2024-02-19T15:31:04.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.121+0000 7f21da800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f21dc06e7c0 0x7f21dc06aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:04.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.121+0000 7f21d9400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21dc06fdf0 0x7f21dc0adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:04.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.121+0000 7f21d9e00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f21dc1613a0 0x7f21dc06e0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:45006/0 (socket says 172.21.15.47:45006) 2024-02-19T15:31:04.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.121+0000 7f21da800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f21dc06e7c0 0x7f21dc06aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:48134/0 (socket says 172.21.15.47:48134) 2024-02-19T15:31:04.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.121+0000 7f21d9e00640 1 -- 172.21.15.47:0/1951322275 learned_addr learned my addr 172.21.15.47:0/1951322275 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:31:04.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.121+0000 7f21da800640 1 -- 172.21.15.47:0/1951322275 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21dc06fdf0 msgr2=0x7f21dc0adbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:04.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.121+0000 7f21da800640 1 --2- 172.21.15.47:0/1951322275 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21dc06fdf0 0x7f21dc0adbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:04.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.121+0000 7f21da800640 1 -- 172.21.15.47:0/1951322275 >> v1:172.21.15.116:6789/0 conn(0x7f21dc1613a0 legacy=0x7f21dc06e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:04.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.121+0000 7f21da800640 1 -- 172.21.15.47:0/1951322275 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21dc0ae6a0 con 0x7f21dc06e7c0 2024-02-19T15:31:04.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.121+0000 7f21d8a00640 1 -- 172.21.15.47:0/1951322275 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691642945 0 0) 0x7f21dc161780 con 0x7f21dc1613a0 2024-02-19T15:31:04.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.125+0000 7f21da800640 1 --2- 172.21.15.47:0/1951322275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21dc06e7c0 0x7f21dc06aeb0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f21d4009d90 tx=0x7f21d401c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c57b9dcfd7802789 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:04.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.125+0000 7f21d8a00640 1 -- 172.21.15.47:0/1951322275 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d40a58b0 con 0x7f21dc06e7c0 2024-02-19T15:31:04.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.125+0000 7f21d8a00640 1 -- 172.21.15.47:0/1951322275 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d40a5a10 con 0x7f21dc06e7c0 2024-02-19T15:31:04.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.125+0000 7f21d8a00640 1 -- 172.21.15.47:0/1951322275 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d40a5da0 con 0x7f21dc06e7c0 2024-02-19T15:31:04.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.125+0000 7f21e1000640 1 -- 172.21.15.47:0/1951322275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21dc06e7c0 msgr2=0x7f21dc06aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:04.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.125+0000 7f21e1000640 1 --2- 172.21.15.47:0/1951322275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21dc06e7c0 0x7f21dc06aeb0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f21d4009d90 tx=0x7f21d401c8e0 comp rx=0 tx=0).stop 2024-02-19T15:31:04.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.125+0000 7f21e1000640 1 -- 172.21.15.47:0/1951322275 shutdown_connections 2024-02-19T15:31:04.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.125+0000 7f21e1000640 1 --2- 172.21.15.47:0/1951322275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21dc06e7c0 0x7f21dc06aeb0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:04.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.125+0000 7f21e1000640 1 --2- 172.21.15.47:0/1951322275 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21dc06fdf0 0x7f21dc0adbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:04.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.125+0000 7f21e1000640 1 -- 172.21.15.47:0/1951322275 >> 172.21.15.47:0/1951322275 conn(0x7f21dc0d0ad0 msgr2=0x7f21dc0d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:04.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.125+0000 7f21e1000640 1 -- 172.21.15.47:0/1951322275 shutdown_connections 2024-02-19T15:31:04.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.125+0000 7f21e1000640 1 -- 172.21.15.47:0/1951322275 wait complete. 2024-02-19T15:31:04.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21e1000640 1 Processor -- start 2024-02-19T15:31:04.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21e1000640 1 -- start start 2024-02-19T15:31:04.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21e1000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21dc06e7c0 0x7f21dc15a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:04.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21e1000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21dc06fdf0 0x7f21dc156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:04.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21e1000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21dc1613a0 0x7f21dc1569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:04.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21e1000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc152ee0 con 0x7f21dc1613a0 2024-02-19T15:31:04.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21e1000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc159d90 con 0x7f21dc06fdf0 2024-02-19T15:31:04.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21e1000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc15a070 con 0x7f21dc06e7c0 2024-02-19T15:31:04.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21da800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21dc1613a0 0x7f21dc1569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:04.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21da800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21dc1613a0 0x7f21dc1569a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57846/0 (socket says 172.21.15.47:57846) 2024-02-19T15:31:04.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21da800640 1 -- 172.21.15.47:0/4238360487 learned_addr learned my addr 172.21.15.47:0/4238360487 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:04.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21d9e00640 1 --2- 172.21.15.47:0/4238360487 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21dc06e7c0 0x7f21dc15a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:04.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21d9400640 1 --2- 172.21.15.47:0/4238360487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21dc06fdf0 0x7f21dc156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:04.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21da800640 1 -- 172.21.15.47:0/4238360487 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21dc06e7c0 msgr2=0x7f21dc15a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:04.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21da800640 1 --2- 172.21.15.47:0/4238360487 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21dc06e7c0 0x7f21dc15a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:04.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21da800640 1 -- 172.21.15.47:0/4238360487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21dc06fdf0 msgr2=0x7f21dc156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:04.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21da800640 1 --2- 172.21.15.47:0/4238360487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21dc06fdf0 0x7f21dc156460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:04.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21da800640 1 -- 172.21.15.47:0/4238360487 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d4009a70 con 0x7f21dc1613a0 2024-02-19T15:31:04.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21da800640 1 --2- 172.21.15.47:0/4238360487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21dc1613a0 0x7f21dc1569a0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f21d401af90 tx=0x7f21d4019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:04.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21d2000640 1 -- 172.21.15.47:0/4238360487 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d40b9860 con 0x7f21dc1613a0 2024-02-19T15:31:04.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21d2000640 1 -- 172.21.15.47:0/4238360487 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d40b99c0 con 0x7f21dc1613a0 2024-02-19T15:31:04.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21d2000640 1 -- 172.21.15.47:0/4238360487 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d40b9d30 con 0x7f21dc1613a0 2024-02-19T15:31:04.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21e1000640 1 -- 172.21.15.47:0/4238360487 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21dc15a350 con 0x7f21dc1613a0 2024-02-19T15:31:04.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21e1000640 1 -- 172.21.15.47:0/4238360487 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21dc15c790 con 0x7f21dc1613a0 2024-02-19T15:31:04.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.129+0000 7f21e1000640 1 -- 172.21.15.47:0/4238360487 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21dc075360 con 0x7f21dc1613a0 2024-02-19T15:31:04.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.133+0000 7f21d2000640 1 -- 172.21.15.47:0/4238360487 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f21d40c4460 con 0x7f21dc1613a0 2024-02-19T15:31:04.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.133+0000 7f21d2000640 1 --2- 172.21.15.47:0/4238360487 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f21bc066e40 0x7f21bc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:04.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.133+0000 7f21d2000640 1 -- 172.21.15.47:0/4238360487 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f21d411fd90 con 0x7f21dc1613a0 2024-02-19T15:31:04.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.133+0000 7f21d9e00640 1 --2- 172.21.15.47:0/4238360487 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f21bc066e40 0x7f21bc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:04.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.133+0000 7f21d9e00640 1 --2- 172.21.15.47:0/4238360487 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f21bc066e40 0x7f21bc069300 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f21cc000f70 tx=0x7f21cc007df0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:04.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.137+0000 7f21d2000640 1 -- 172.21.15.47:0/4238360487 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f21d40ecbe0 con 0x7f21dc1613a0 2024-02-19T15:31:04.243 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:04.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.329+0000 7f21e1000640 1 -- 172.21.15.47:0/4238360487 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21dc153300 con 0x7f21dc1613a0 2024-02-19T15:31:04.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.329+0000 7f21d2000640 1 -- 172.21.15.47:0/4238360487 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f21d40f02a0 con 0x7f21dc1613a0 2024-02-19T15:31:04.330 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:04.330 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:04.330 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:31:04.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.333+0000 7f21e1000640 1 -- 172.21.15.47:0/4238360487 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f21bc066e40 msgr2=0x7f21bc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:04.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.333+0000 7f21e1000640 1 --2- 172.21.15.47:0/4238360487 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f21bc066e40 0x7f21bc069300 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f21cc000f70 tx=0x7f21cc007df0 comp rx=0 tx=0).stop 2024-02-19T15:31:04.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.333+0000 7f21e1000640 1 -- 172.21.15.47:0/4238360487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21dc1613a0 msgr2=0x7f21dc1569a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:04.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.333+0000 7f21e1000640 1 --2- 172.21.15.47:0/4238360487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21dc1613a0 0x7f21dc1569a0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f21d401af90 tx=0x7f21d4019680 comp rx=0 tx=0).stop 2024-02-19T15:31:04.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.333+0000 7f21e1000640 1 -- 172.21.15.47:0/4238360487 shutdown_connections 2024-02-19T15:31:04.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.333+0000 7f21e1000640 1 --2- 172.21.15.47:0/4238360487 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f21bc066e40 0x7f21bc069300 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:04.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.333+0000 7f21e1000640 1 --2- 172.21.15.47:0/4238360487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21dc1613a0 0x7f21dc1569a0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:04.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.333+0000 7f21e1000640 1 --2- 172.21.15.47:0/4238360487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21dc06fdf0 0x7f21dc156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:04.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.333+0000 7f21e1000640 1 --2- 172.21.15.47:0/4238360487 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21dc06e7c0 0x7f21dc15a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:04.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.333+0000 7f21e1000640 1 -- 172.21.15.47:0/4238360487 >> 172.21.15.47:0/4238360487 conn(0x7f21dc0d0ad0 msgr2=0x7f21dc0b02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:04.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.333+0000 7f21e1000640 1 -- 172.21.15.47:0/4238360487 shutdown_connections 2024-02-19T15:31:04.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:04.333+0000 7f21e1000640 1 -- 172.21.15.47:0/4238360487 wait complete. 2024-02-19T15:31:04.553 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:04.553 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:05.554 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:05.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.693+0000 7f27ede00640 1 Processor -- start 2024-02-19T15:31:05.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.693+0000 7f27ede00640 1 -- start start 2024-02-19T15:31:05.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27ede00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27e815bcd0 0x7f27e80cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:05.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27ede00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f27e80cedb0 0x7f27e80cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:05.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27ede00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e80cc700 con 0x7f27e80d2710 2024-02-19T15:31:05.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27ede00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e80cf5e0 con 0x7f27e80cedb0 2024-02-19T15:31:05.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27ede00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e80cf300 con 0x7f27e815bcd0 2024-02-19T15:31:05.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e7400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f27e80d2710 0x7f27e815d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42588/0 (socket says 172.21.15.47:42588) 2024-02-19T15:31:05.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e7400640 1 -- 172.21.15.47:0/3628103023 learned_addr learned my addr 172.21.15.47:0/3628103023 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:31:05.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27dfe00640 1 --2- 172.21.15.47:0/3628103023 >> v2:172.21.15.116:3300/0 conn(0x7f27e80cedb0 0x7f27e80cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:05.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e6a00640 1 --2- 172.21.15.47:0/3628103023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27e815bcd0 0x7f27e80cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:05.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e6000640 1 -- 172.21.15.47:0/3628103023 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 103415104 0 0) 0x7f27e80cc700 con 0x7f27e80d2710 2024-02-19T15:31:05.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e6000640 1 -- 172.21.15.47:0/3628103023 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27cc003530 con 0x7f27e80d2710 2024-02-19T15:31:05.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e6000640 1 -- 172.21.15.47:0/3628103023 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 761056456 0 0) 0x7f27cc003530 con 0x7f27e80d2710 2024-02-19T15:31:05.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e6000640 1 -- 172.21.15.47:0/3628103023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27e815bcd0 msgr2=0x7f27e80cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:05.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e6000640 1 --2- 172.21.15.47:0/3628103023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27e815bcd0 0x7f27e80cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:05.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e6000640 1 -- 172.21.15.47:0/3628103023 >> v2:172.21.15.116:3300/0 conn(0x7f27e80cedb0 msgr2=0x7f27e80cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:05.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e6000640 1 --2- 172.21.15.47:0/3628103023 >> v2:172.21.15.116:3300/0 conn(0x7f27e80cedb0 0x7f27e80cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:05.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e6000640 1 -- 172.21.15.47:0/3628103023 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e80cc9a0 con 0x7f27e80d2710 2024-02-19T15:31:05.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e6000640 1 -- 172.21.15.47:0/3628103023 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f27e0001a00 con 0x7f27e80d2710 2024-02-19T15:31:05.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27dfe00640 1 --2- 172.21.15.47:0/3628103023 >> v2:172.21.15.116:3300/0 conn(0x7f27e80cedb0 0x7f27e80cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:31:05.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e6000640 1 -- 172.21.15.47:0/3628103023 >> v1:172.21.15.47:6789/0 conn(0x7f27e80d2710 legacy=0x7f27e815d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:05.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e6000640 1 --2- 172.21.15.47:0/3628103023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27cc003980 0x7f27cc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:05.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e6000640 1 --2- 172.21.15.47:0/3628103023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27e815bcd0 0x7f27cc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:05.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e6000640 1 --2- 172.21.15.47:0/3628103023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27cc004d10 0x7f27cc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:05.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e6000640 1 -- 172.21.15.47:0/3628103023 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e80cf5e0 con 0x7f27cc004d10 2024-02-19T15:31:05.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e6000640 1 -- 172.21.15.47:0/3628103023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e80cf300 con 0x7f27cc003980 2024-02-19T15:31:05.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e6000640 1 -- 172.21.15.47:0/3628103023 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc00b990 con 0x7f27e815bcd0 2024-02-19T15:31:05.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e7400640 1 --2- 172.21.15.47:0/3628103023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27cc004d10 0x7f27cc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:05.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e6a00640 1 --2- 172.21.15.47:0/3628103023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27cc003980 0x7f27cc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:05.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27dfe00640 1 --2- 172.21.15.47:0/3628103023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27e815bcd0 0x7f27cc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:05.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e7400640 1 -- 172.21.15.47:0/3628103023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27e815bcd0 msgr2=0x7f27cc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:05.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e7400640 1 --2- 172.21.15.47:0/3628103023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27e815bcd0 0x7f27cc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:05.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e7400640 1 -- 172.21.15.47:0/3628103023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27cc003980 msgr2=0x7f27cc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:05.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e7400640 1 --2- 172.21.15.47:0/3628103023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27cc003980 0x7f27cc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:05.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.697+0000 7f27e7400640 1 -- 172.21.15.47:0/3628103023 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc003530 con 0x7f27cc004d10 2024-02-19T15:31:05.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.701+0000 7f27e7400640 1 --2- 172.21.15.47:0/3628103023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27cc004d10 0x7f27cc005190 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f27e000a860 tx=0x7f27e001d380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=216697f9a76d402e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:05.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.701+0000 7f27e6000640 1 -- 172.21.15.47:0/3628103023 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e00ad630 con 0x7f27cc004d10 2024-02-19T15:31:05.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.701+0000 7f27e6000640 1 -- 172.21.15.47:0/3628103023 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e00a8070 con 0x7f27cc004d10 2024-02-19T15:31:05.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.701+0000 7f27e6000640 1 -- 172.21.15.47:0/3628103023 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e00ad9f0 con 0x7f27cc004d10 2024-02-19T15:31:05.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.701+0000 7f27ede00640 1 -- 172.21.15.47:0/3628103023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27cc004d10 msgr2=0x7f27cc005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:05.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.701+0000 7f27ede00640 1 --2- 172.21.15.47:0/3628103023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27cc004d10 0x7f27cc005190 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f27e000a860 tx=0x7f27e001d380 comp rx=0 tx=0).stop 2024-02-19T15:31:05.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.701+0000 7f27e7400640 1 -- 172.21.15.47:0/3628103023 reap_dead start 2024-02-19T15:31:05.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.701+0000 7f27ede00640 1 -- 172.21.15.47:0/3628103023 shutdown_connections 2024-02-19T15:31:05.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.701+0000 7f27ede00640 1 -- 172.21.15.47:0/3628103023 >> 172.21.15.47:0/3628103023 conn(0x7f27e80d1bf0 msgr2=0x7f27e80d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:05.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.701+0000 7f27ede00640 1 -- 172.21.15.47:0/3628103023 shutdown_connections 2024-02-19T15:31:05.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.701+0000 7f27ede00640 1 -- 172.21.15.47:0/3628103023 wait complete. 2024-02-19T15:31:05.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.701+0000 7f27ede00640 1 Processor -- start 2024-02-19T15:31:05.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.701+0000 7f27ede00640 1 -- start start 2024-02-19T15:31:05.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27ede00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27cc004d10 0x7f27e8154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:05.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27ede00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27e80b6f10 0x7f27e80c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:05.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27ede00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27e80c8270 0x7f27e814bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:05.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27ede00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e80b5e70 con 0x7f27e80b6f10 2024-02-19T15:31:05.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27ede00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e80b6150 con 0x7f27cc004d10 2024-02-19T15:31:05.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27ede00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e80b6430 con 0x7f27e80c8270 2024-02-19T15:31:05.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27dfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27e80b6f10 0x7f27e80c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:05.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27dfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27e80b6f10 0x7f27e80c7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57868/0 (socket says 172.21.15.47:57868) 2024-02-19T15:31:05.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27dfe00640 1 -- 172.21.15.47:0/2332541012 learned_addr learned my addr 172.21.15.47:0/2332541012 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:05.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27e6a00640 1 --2- 172.21.15.47:0/2332541012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27cc004d10 0x7f27e8154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:05.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27e7400640 1 --2- 172.21.15.47:0/2332541012 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27e80c8270 0x7f27e814bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:05.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27dfe00640 1 -- 172.21.15.47:0/2332541012 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27e80c8270 msgr2=0x7f27e814bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:05.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27dfe00640 1 --2- 172.21.15.47:0/2332541012 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27e80c8270 0x7f27e814bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:05.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27dfe00640 1 -- 172.21.15.47:0/2332541012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27cc004d10 msgr2=0x7f27e8154e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:05.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27dfe00640 1 --2- 172.21.15.47:0/2332541012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27cc004d10 0x7f27e8154e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:05.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27dfe00640 1 -- 172.21.15.47:0/2332541012 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e80b6710 con 0x7f27e80b6f10 2024-02-19T15:31:05.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27dfe00640 1 --2- 172.21.15.47:0/2332541012 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27e80b6f10 0x7f27e80c7d30 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f27d0011cd0 tx=0x7f27d000f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:05.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27df400640 1 -- 172.21.15.47:0/2332541012 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d0011d00 con 0x7f27e80b6f10 2024-02-19T15:31:05.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27df400640 1 -- 172.21.15.47:0/2332541012 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d0011e60 con 0x7f27e80b6f10 2024-02-19T15:31:05.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27df400640 1 -- 172.21.15.47:0/2332541012 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d0004240 con 0x7f27e80b6f10 2024-02-19T15:31:05.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27ede00640 1 -- 172.21.15.47:0/2332541012 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e80b69f0 con 0x7f27e80b6f10 2024-02-19T15:31:05.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27ede00640 1 -- 172.21.15.47:0/2332541012 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e8155360 con 0x7f27e80b6f10 2024-02-19T15:31:05.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.705+0000 7f27ede00640 1 -- 172.21.15.47:0/2332541012 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e80b6b80 con 0x7f27e80b6f10 2024-02-19T15:31:05.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.709+0000 7f27df400640 1 -- 172.21.15.47:0/2332541012 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f27d00043a0 con 0x7f27e80b6f10 2024-02-19T15:31:05.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.709+0000 7f27df400640 1 --2- 172.21.15.47:0/2332541012 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f27c4066e70 0x7f27c4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:05.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.709+0000 7f27df400640 1 -- 172.21.15.47:0/2332541012 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f27d0085010 con 0x7f27e80b6f10 2024-02-19T15:31:05.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.709+0000 7f27e6a00640 1 --2- 172.21.15.47:0/2332541012 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f27c4066e70 0x7f27c4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:05.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.709+0000 7f27e6a00640 1 --2- 172.21.15.47:0/2332541012 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f27c4066e70 0x7f27c4069330 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f27d8009be0 tx=0x7f27d8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:05.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.713+0000 7f27df400640 1 -- 172.21.15.47:0/2332541012 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f27d0051f00 con 0x7f27e80b6f10 2024-02-19T15:31:05.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.925+0000 7f27ede00640 1 -- 172.21.15.47:0/2332541012 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27e8075360 con 0x7f27e80b6f10 2024-02-19T15:31:05.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.925+0000 7f27df400640 1 -- 172.21.15.47:0/2332541012 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f27d0022090 con 0x7f27e80b6f10 2024-02-19T15:31:05.925 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:05.925 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:05.925 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:31:05.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.929+0000 7f27ede00640 1 -- 172.21.15.47:0/2332541012 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f27c4066e70 msgr2=0x7f27c4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:05.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.929+0000 7f27ede00640 1 --2- 172.21.15.47:0/2332541012 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f27c4066e70 0x7f27c4069330 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f27d8009be0 tx=0x7f27d8009190 comp rx=0 tx=0).stop 2024-02-19T15:31:05.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.929+0000 7f27ede00640 1 -- 172.21.15.47:0/2332541012 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27e80b6f10 msgr2=0x7f27e80c7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:05.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.929+0000 7f27ede00640 1 --2- 172.21.15.47:0/2332541012 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27e80b6f10 0x7f27e80c7d30 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f27d0011cd0 tx=0x7f27d000f610 comp rx=0 tx=0).stop 2024-02-19T15:31:05.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.929+0000 7f27ede00640 1 -- 172.21.15.47:0/2332541012 shutdown_connections 2024-02-19T15:31:05.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.929+0000 7f27ede00640 1 --2- 172.21.15.47:0/2332541012 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f27c4066e70 0x7f27c4069330 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:05.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.929+0000 7f27ede00640 1 --2- 172.21.15.47:0/2332541012 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27e80c8270 0x7f27e814bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:05.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.929+0000 7f27ede00640 1 --2- 172.21.15.47:0/2332541012 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27e80b6f10 0x7f27e80c7d30 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:05.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.929+0000 7f27ede00640 1 --2- 172.21.15.47:0/2332541012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27cc004d10 0x7f27e8154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:05.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.929+0000 7f27ede00640 1 -- 172.21.15.47:0/2332541012 >> 172.21.15.47:0/2332541012 conn(0x7f27e80d1bf0 msgr2=0x7f27e806fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:05.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.929+0000 7f27ede00640 1 -- 172.21.15.47:0/2332541012 shutdown_connections 2024-02-19T15:31:05.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:05.929+0000 7f27ede00640 1 -- 172.21.15.47:0/2332541012 wait complete. 2024-02-19T15:31:06.148 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:06.148 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:06.220 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:31:06.223 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:31:06.245 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:07.149 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:07.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.289+0000 7f8ddea00640 1 Processor -- start 2024-02-19T15:31:07.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.289+0000 7f8ddea00640 1 -- start start 2024-02-19T15:31:07.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.289+0000 7f8ddea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd815e180 0x7f8dd80b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:07.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.289+0000 7f8ddea00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8dd806c1d0 0x7f8dd806c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:07.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.289+0000 7f8ddea00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8dd806cdf0 0x7f8dd80cfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:07.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.289+0000 7f8ddea00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd806d280 con 0x7f8dd806cdf0 2024-02-19T15:31:07.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.289+0000 7f8ddea00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd80b2270 con 0x7f8dd806c1d0 2024-02-19T15:31:07.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.289+0000 7f8ddea00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd80b2550 con 0x7f8dd815e180 2024-02-19T15:31:07.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.289+0000 7f8ddca00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8dd806cdf0 0x7f8dd80cfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:07.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.289+0000 7f8ddca00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8dd806cdf0 0x7f8dd80cfda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57884/0 (socket says 172.21.15.47:57884) 2024-02-19T15:31:07.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.289+0000 7f8ddca00640 1 -- 172.21.15.47:0/4194703318 learned_addr learned my addr 172.21.15.47:0/4194703318 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:07.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.293+0000 7f8dd7e00640 1 --2- 172.21.15.47:0/4194703318 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd815e180 0x7f8dd80b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:07.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.293+0000 7f8dcf400640 1 --2- 172.21.15.47:0/4194703318 >> v2:172.21.15.116:3300/0 conn(0x7f8dd806c1d0 0x7f8dd806c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:07.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.293+0000 7f8ddca00640 1 -- 172.21.15.47:0/4194703318 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd815e180 msgr2=0x7f8dd80b1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:07.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.293+0000 7f8ddca00640 1 --2- 172.21.15.47:0/4194703318 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd815e180 0x7f8dd80b1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:07.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.293+0000 7f8ddca00640 1 -- 172.21.15.47:0/4194703318 >> v2:172.21.15.116:3300/0 conn(0x7f8dd806c1d0 msgr2=0x7f8dd806c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:07.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.293+0000 7f8ddca00640 1 --2- 172.21.15.47:0/4194703318 >> v2:172.21.15.116:3300/0 conn(0x7f8dd806c1d0 0x7f8dd806c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:07.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.293+0000 7f8ddca00640 1 -- 172.21.15.47:0/4194703318 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd80b2830 con 0x7f8dd806cdf0 2024-02-19T15:31:07.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.293+0000 7f8dd7e00640 1 --2- 172.21.15.47:0/4194703318 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd815e180 0x7f8dd80b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:31:07.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.293+0000 7f8ddca00640 1 --2- 172.21.15.47:0/4194703318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dd806cdf0 0x7f8dd80cfda0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8071ed0 tx=0x7f8dd001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c3d467cc4da886 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:07.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.293+0000 7f8dd7400640 1 -- 172.21.15.47:0/4194703318 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd00aa610 con 0x7f8dd806cdf0 2024-02-19T15:31:07.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.293+0000 7f8dd7400640 1 -- 172.21.15.47:0/4194703318 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd00a6040 con 0x7f8dd806cdf0 2024-02-19T15:31:07.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.293+0000 7f8dd7400640 1 -- 172.21.15.47:0/4194703318 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd00aa990 con 0x7f8dd806cdf0 2024-02-19T15:31:07.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.293+0000 7f8ddea00640 1 -- 172.21.15.47:0/4194703318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dd806cdf0 msgr2=0x7f8dd80cfda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:07.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.293+0000 7f8ddea00640 1 --2- 172.21.15.47:0/4194703318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dd806cdf0 0x7f8dd80cfda0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8071ed0 tx=0x7f8dd001c4f0 comp rx=0 tx=0).stop 2024-02-19T15:31:07.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddea00640 1 -- 172.21.15.47:0/4194703318 shutdown_connections 2024-02-19T15:31:07.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddea00640 1 --2- 172.21.15.47:0/4194703318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dd806cdf0 0x7f8dd80cfda0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:07.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddea00640 1 --2- 172.21.15.47:0/4194703318 >> v2:172.21.15.116:3300/0 conn(0x7f8dd806c1d0 0x7f8dd806c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:07.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddea00640 1 --2- 172.21.15.47:0/4194703318 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd815e180 0x7f8dd80b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:07.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddea00640 1 -- 172.21.15.47:0/4194703318 >> 172.21.15.47:0/4194703318 conn(0x7f8dd80d1ba0 msgr2=0x7f8dd80d1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:07.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddea00640 1 -- 172.21.15.47:0/4194703318 shutdown_connections 2024-02-19T15:31:07.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddea00640 1 -- 172.21.15.47:0/4194703318 wait complete. 2024-02-19T15:31:07.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddea00640 1 Processor -- start 2024-02-19T15:31:07.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddea00640 1 -- start start 2024-02-19T15:31:07.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd806c1d0 0x7f8dd8156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:07.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dd806cdf0 0x7f8dd815c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:07.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dd815e180 0x7f8dd8156c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:07.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddea00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd80c75c0 con 0x7f8dd815e180 2024-02-19T15:31:07.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddea00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd80c6940 con 0x7f8dd806cdf0 2024-02-19T15:31:07.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddea00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd80c6c20 con 0x7f8dd806c1d0 2024-02-19T15:31:07.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dd815e180 0x7f8dd8156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:07.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dd815e180 0x7f8dd8156c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57888/0 (socket says 172.21.15.47:57888) 2024-02-19T15:31:07.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddca00640 1 -- 172.21.15.47:0/4072097689 learned_addr learned my addr 172.21.15.47:0/4072097689 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:07.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8dcf400640 1 --2- 172.21.15.47:0/4072097689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dd806cdf0 0x7f8dd815c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:07.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8dd7e00640 1 --2- 172.21.15.47:0/4072097689 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd806c1d0 0x7f8dd8156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:07.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddca00640 1 -- 172.21.15.47:0/4072097689 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd806c1d0 msgr2=0x7f8dd8156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:07.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddca00640 1 --2- 172.21.15.47:0/4072097689 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd806c1d0 0x7f8dd8156170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:07.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddca00640 1 -- 172.21.15.47:0/4072097689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dd806cdf0 msgr2=0x7f8dd815c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:07.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddca00640 1 --2- 172.21.15.47:0/4072097689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dd806cdf0 0x7f8dd815c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:07.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddca00640 1 -- 172.21.15.47:0/4072097689 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd00094d0 con 0x7f8dd815e180 2024-02-19T15:31:07.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8ddca00640 1 --2- 172.21.15.47:0/4072097689 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dd815e180 0x7f8dd8156c60 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f8dd00ac040 tx=0x7f8dd001ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:07.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.297+0000 7f8dd4c00640 1 -- 172.21.15.47:0/4072097689 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd00bd5f0 con 0x7f8dd815e180 2024-02-19T15:31:07.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.301+0000 7f8dd4c00640 1 -- 172.21.15.47:0/4072097689 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd00aae90 con 0x7f8dd815e180 2024-02-19T15:31:07.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.301+0000 7f8ddea00640 1 -- 172.21.15.47:0/4072097689 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd80c6f00 con 0x7f8dd815e180 2024-02-19T15:31:07.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.301+0000 7f8dd4c00640 1 -- 172.21.15.47:0/4072097689 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd00bd9b0 con 0x7f8dd815e180 2024-02-19T15:31:07.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.301+0000 7f8ddea00640 1 -- 172.21.15.47:0/4072097689 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd81566b0 con 0x7f8dd815e180 2024-02-19T15:31:07.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.301+0000 7f8ddea00640 1 -- 172.21.15.47:0/4072097689 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd8068500 con 0x7f8dd815e180 2024-02-19T15:31:07.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.301+0000 7f8dd4c00640 1 -- 172.21.15.47:0/4072097689 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8dd00b33f0 con 0x7f8dd815e180 2024-02-19T15:31:07.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.301+0000 7f8dd4c00640 1 --2- 172.21.15.47:0/4072097689 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8db4066e40 0x7f8db4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:07.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.301+0000 7f8dd4c00640 1 -- 172.21.15.47:0/4072097689 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8dd011f3f0 con 0x7f8dd815e180 2024-02-19T15:31:07.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.301+0000 7f8dd7e00640 1 --2- 172.21.15.47:0/4072097689 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8db4066e40 0x7f8db4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:07.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.301+0000 7f8dd7e00640 1 --2- 172.21.15.47:0/4072097689 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8db4066e40 0x7f8db4069300 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8003b90 tx=0x7f8dc8009890 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:07.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.305+0000 7f8dd4c00640 1 -- 172.21.15.47:0/4072097689 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8dd00ec240 con 0x7f8dd815e180 2024-02-19T15:31:07.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.521+0000 7f8ddea00640 1 -- 172.21.15.47:0/4072097689 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8dd80c7700 con 0x7f8dd815e180 2024-02-19T15:31:07.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.525+0000 7f8dd4c00640 1 -- 172.21.15.47:0/4072097689 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f8dd80c7700 con 0x7f8dd815e180 2024-02-19T15:31:07.523 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:07.523 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:07.524 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:31:07.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.525+0000 7f8ddea00640 1 -- 172.21.15.47:0/4072097689 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8db4066e40 msgr2=0x7f8db4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:07.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.525+0000 7f8ddea00640 1 --2- 172.21.15.47:0/4072097689 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8db4066e40 0x7f8db4069300 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8003b90 tx=0x7f8dc8009890 comp rx=0 tx=0).stop 2024-02-19T15:31:07.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.525+0000 7f8ddea00640 1 -- 172.21.15.47:0/4072097689 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dd815e180 msgr2=0x7f8dd8156c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:07.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.525+0000 7f8ddea00640 1 --2- 172.21.15.47:0/4072097689 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dd815e180 0x7f8dd8156c60 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f8dd00ac040 tx=0x7f8dd001ad30 comp rx=0 tx=0).stop 2024-02-19T15:31:07.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.525+0000 7f8ddea00640 1 -- 172.21.15.47:0/4072097689 shutdown_connections 2024-02-19T15:31:07.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.525+0000 7f8ddea00640 1 --2- 172.21.15.47:0/4072097689 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8db4066e40 0x7f8db4069300 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:07.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.525+0000 7f8ddea00640 1 --2- 172.21.15.47:0/4072097689 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dd815e180 0x7f8dd8156c60 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:07.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.525+0000 7f8ddea00640 1 --2- 172.21.15.47:0/4072097689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dd806cdf0 0x7f8dd815c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:07.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.525+0000 7f8ddea00640 1 --2- 172.21.15.47:0/4072097689 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dd806c1d0 0x7f8dd8156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:07.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.525+0000 7f8ddea00640 1 -- 172.21.15.47:0/4072097689 >> 172.21.15.47:0/4072097689 conn(0x7f8dd80d1ba0 msgr2=0x7f8dd8160fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:07.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.525+0000 7f8ddea00640 1 -- 172.21.15.47:0/4072097689 shutdown_connections 2024-02-19T15:31:07.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:07.525+0000 7f8ddea00640 1 -- 172.21.15.47:0/4072097689 wait complete. 2024-02-19T15:31:07.745 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:07.746 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:08.247 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:08.746 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:08.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff41e200640 1 Processor -- start 2024-02-19T15:31:08.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff41e200640 1 -- start start 2024-02-19T15:31:08.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff41e200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff4180cea80 0x7ff4180727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:08.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff41e200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41815d280 con 0x7ff418145590 2024-02-19T15:31:08.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff41e200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff418070ff0 con 0x7ff41815c390 2024-02-19T15:31:08.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff41e200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4180709e0 con 0x7ff4180cea80 2024-02-19T15:31:08.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff417400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff418145590 0x7ff41806c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42592/0 (socket says 172.21.15.47:42592) 2024-02-19T15:31:08.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff417400640 1 -- 172.21.15.47:0/3235375100 learned_addr learned my addr 172.21.15.47:0/3235375100 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:31:08.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff40fe00640 1 --2- 172.21.15.47:0/3235375100 >> v2:172.21.15.116:3300/0 conn(0x7ff4180cea80 0x7ff4180727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:08.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff416a00640 1 -- 172.21.15.47:0/3235375100 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527306747 0 0) 0x7ff41815d280 con 0x7ff418145590 2024-02-19T15:31:08.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff416a00640 1 -- 172.21.15.47:0/3235375100 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3fc003530 con 0x7ff418145590 2024-02-19T15:31:08.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff416a00640 1 -- 172.21.15.47:0/3235375100 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2658559018 0 0) 0x7ff3fc003530 con 0x7ff418145590 2024-02-19T15:31:08.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff416a00640 1 -- 172.21.15.47:0/3235375100 >> v2:172.21.15.116:3300/0 conn(0x7ff4180cea80 msgr2=0x7ff4180727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:08.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff416a00640 1 --2- 172.21.15.47:0/3235375100 >> v2:172.21.15.116:3300/0 conn(0x7ff4180cea80 0x7ff4180727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:08.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff416a00640 1 -- 172.21.15.47:0/3235375100 >> v1:172.21.15.116:6789/0 conn(0x7ff41815c390 legacy=0x7ff41815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:08.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff416a00640 1 -- 172.21.15.47:0/3235375100 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4180cf020 con 0x7ff418145590 2024-02-19T15:31:08.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff416a00640 1 -- 172.21.15.47:0/3235375100 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7ff408001680 con 0x7ff418145590 2024-02-19T15:31:08.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff40fe00640 1 --2- 172.21.15.47:0/3235375100 >> v2:172.21.15.116:3300/0 conn(0x7ff4180cea80 0x7ff4180727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:31:08.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff416a00640 1 -- 172.21.15.47:0/3235375100 >> v1:172.21.15.47:6789/0 conn(0x7ff418145590 legacy=0x7ff41806c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:08.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff416a00640 1 --2- 172.21.15.47:0/3235375100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3fc0039d0 0x7ff3fc003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:08.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff416a00640 1 --2- 172.21.15.47:0/3235375100 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff3fc0047d0 0x7ff3fc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:08.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff416a00640 1 --2- 172.21.15.47:0/3235375100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3fc0051e0 0x7ff3fc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:08.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff416a00640 1 -- 172.21.15.47:0/3235375100 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4180709e0 con 0x7ff3fc0051e0 2024-02-19T15:31:08.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff416a00640 1 -- 172.21.15.47:0/3235375100 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc00c130 con 0x7ff3fc0039d0 2024-02-19T15:31:08.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff416a00640 1 -- 172.21.15.47:0/3235375100 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc00c2f0 con 0x7ff3fc0047d0 2024-02-19T15:31:08.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff417e00640 1 --2- 172.21.15.47:0/3235375100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3fc0051e0 0x7ff3fc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:08.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff417400640 1 --2- 172.21.15.47:0/3235375100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3fc0039d0 0x7ff3fc003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:08.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.889+0000 7ff40fe00640 1 --2- 172.21.15.47:0/3235375100 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff3fc0047d0 0x7ff3fc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:08.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.893+0000 7ff417e00640 1 -- 172.21.15.47:0/3235375100 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff3fc0047d0 msgr2=0x7ff3fc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:08.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.893+0000 7ff417e00640 1 --2- 172.21.15.47:0/3235375100 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff3fc0047d0 0x7ff3fc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:08.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.893+0000 7ff417e00640 1 -- 172.21.15.47:0/3235375100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3fc0039d0 msgr2=0x7ff3fc003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:08.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.893+0000 7ff417e00640 1 --2- 172.21.15.47:0/3235375100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3fc0039d0 0x7ff3fc003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:08.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.893+0000 7ff417e00640 1 -- 172.21.15.47:0/3235375100 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc003530 con 0x7ff3fc0051e0 2024-02-19T15:31:08.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.893+0000 7ff417e00640 1 -- 172.21.15.47:0/3235375100 reap_dead start 2024-02-19T15:31:08.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.893+0000 7ff417e00640 1 --2- 172.21.15.47:0/3235375100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3fc0051e0 0x7ff3fc00b990 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7ff410001e70 tx=0x7ff41001c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bbf5f2cd306de4e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:08.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.893+0000 7ff416a00640 1 -- 172.21.15.47:0/3235375100 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff410003b80 con 0x7ff3fc0051e0 2024-02-19T15:31:08.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.893+0000 7ff416a00640 1 -- 172.21.15.47:0/3235375100 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff410003d00 con 0x7ff3fc0051e0 2024-02-19T15:31:08.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.893+0000 7ff416a00640 1 -- 172.21.15.47:0/3235375100 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff410005260 con 0x7ff3fc0051e0 2024-02-19T15:31:08.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff41e200640 1 -- 172.21.15.47:0/3235375100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3fc0051e0 msgr2=0x7ff3fc00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:08.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff41e200640 1 --2- 172.21.15.47:0/3235375100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3fc0051e0 0x7ff3fc00b990 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7ff410001e70 tx=0x7ff41001c780 comp rx=0 tx=0).stop 2024-02-19T15:31:08.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff41e200640 1 -- 172.21.15.47:0/3235375100 shutdown_connections 2024-02-19T15:31:08.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff41e200640 1 --2- 172.21.15.47:0/3235375100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3fc0051e0 0x7ff3fc00b990 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:08.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff41e200640 1 -- 172.21.15.47:0/3235375100 >> 172.21.15.47:0/3235375100 conn(0x7ff4180b16a0 msgr2=0x7ff4180b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:08.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff41e200640 1 -- 172.21.15.47:0/3235375100 shutdown_connections 2024-02-19T15:31:08.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff41e200640 1 -- 172.21.15.47:0/3235375100 wait complete. 2024-02-19T15:31:08.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff41e200640 1 Processor -- start 2024-02-19T15:31:08.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff41e200640 1 -- start start 2024-02-19T15:31:08.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff41e200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3fc0051e0 0x7ff4180c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:08.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff41e200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff418140740 0x7ff418143320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:08.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff41e200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4180c7f50 0x7ff4180c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:08.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff41e200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4180c62d0 con 0x7ff3fc0051e0 2024-02-19T15:31:08.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff41e200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4180c8910 con 0x7ff418140740 2024-02-19T15:31:08.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff41e200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff418140bc0 con 0x7ff4180c7f50 2024-02-19T15:31:08.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff417400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3fc0051e0 0x7ff4180c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:08.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff417400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3fc0051e0 0x7ff4180c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57906/0 (socket says 172.21.15.47:57906) 2024-02-19T15:31:08.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff417400640 1 -- 172.21.15.47:0/646896445 learned_addr learned my addr 172.21.15.47:0/646896445 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:08.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff40fe00640 1 --2- 172.21.15.47:0/646896445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff418140740 0x7ff418143320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:08.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff417e00640 1 --2- 172.21.15.47:0/646896445 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4180c7f50 0x7ff4180c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:08.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff417400640 1 -- 172.21.15.47:0/646896445 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4180c7f50 msgr2=0x7ff4180c83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:08.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff417400640 1 --2- 172.21.15.47:0/646896445 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4180c7f50 0x7ff4180c83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:08.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff417400640 1 -- 172.21.15.47:0/646896445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff418140740 msgr2=0x7ff418143320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:08.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff417400640 1 --2- 172.21.15.47:0/646896445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff418140740 0x7ff418143320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:08.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff417400640 1 -- 172.21.15.47:0/646896445 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4180c9f40 con 0x7ff3fc0051e0 2024-02-19T15:31:08.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff40fe00640 1 --2- 172.21.15.47:0/646896445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff418140740 0x7ff418143320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:31:08.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff417e00640 1 --2- 172.21.15.47:0/646896445 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4180c7f50 0x7ff4180c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:31:08.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.897+0000 7ff417400640 1 --2- 172.21.15.47:0/646896445 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3fc0051e0 0x7ff4180c5770 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7ff408004f00 tx=0x7ff408007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:08.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.901+0000 7ff40f400640 1 -- 172.21.15.47:0/646896445 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40800f750 con 0x7ff3fc0051e0 2024-02-19T15:31:08.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.901+0000 7ff40f400640 1 -- 172.21.15.47:0/646896445 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40800f8d0 con 0x7ff3fc0051e0 2024-02-19T15:31:08.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.901+0000 7ff41e200640 1 -- 172.21.15.47:0/646896445 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4180c5f30 con 0x7ff3fc0051e0 2024-02-19T15:31:08.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.901+0000 7ff40f400640 1 -- 172.21.15.47:0/646896445 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40800fc80 con 0x7ff3fc0051e0 2024-02-19T15:31:08.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.901+0000 7ff41e200640 1 -- 172.21.15.47:0/646896445 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff418143e50 con 0x7ff3fc0051e0 2024-02-19T15:31:08.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.901+0000 7ff41e200640 1 -- 172.21.15.47:0/646896445 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4180c6410 con 0x7ff3fc0051e0 2024-02-19T15:31:08.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.901+0000 7ff40f400640 1 -- 172.21.15.47:0/646896445 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7ff40801d460 con 0x7ff3fc0051e0 2024-02-19T15:31:08.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.901+0000 7ff40f400640 1 --2- 172.21.15.47:0/646896445 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff3f4066e70 0x7ff3f4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:08.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.901+0000 7ff40f400640 1 -- 172.21.15.47:0/646896445 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff408084a40 con 0x7ff3fc0051e0 2024-02-19T15:31:08.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.901+0000 7ff40fe00640 1 --2- 172.21.15.47:0/646896445 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff3f4066e70 0x7ff3f4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:08.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.905+0000 7ff40fe00640 1 --2- 172.21.15.47:0/646896445 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff3f4066e70 0x7ff3f4069330 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ff418144650 tx=0x7ff400008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:08.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:08.905+0000 7ff40f400640 1 -- 172.21.15.47:0/646896445 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7ff4180c6410 con 0x7ff3fc0051e0 2024-02-19T15:31:09.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:09.105+0000 7ff41e200640 1 -- 172.21.15.47:0/646896445 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4180c6410 con 0x7ff3fc0051e0 2024-02-19T15:31:09.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:09.105+0000 7ff40f400640 1 -- 172.21.15.47:0/646896445 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7ff408086e60 con 0x7ff3fc0051e0 2024-02-19T15:31:09.104 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:09.104 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:09.104 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:31:09.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:09.105+0000 7ff41e200640 1 -- 172.21.15.47:0/646896445 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff3f4066e70 msgr2=0x7ff3f4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:09.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:09.105+0000 7ff41e200640 1 --2- 172.21.15.47:0/646896445 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff3f4066e70 0x7ff3f4069330 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ff418144650 tx=0x7ff400008040 comp rx=0 tx=0).stop 2024-02-19T15:31:09.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:09.105+0000 7ff41e200640 1 -- 172.21.15.47:0/646896445 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3fc0051e0 msgr2=0x7ff4180c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:09.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:09.105+0000 7ff41e200640 1 --2- 172.21.15.47:0/646896445 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3fc0051e0 0x7ff4180c5770 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7ff408004f00 tx=0x7ff408007430 comp rx=0 tx=0).stop 2024-02-19T15:31:09.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:09.105+0000 7ff41e200640 1 -- 172.21.15.47:0/646896445 shutdown_connections 2024-02-19T15:31:09.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:09.105+0000 7ff41e200640 1 --2- 172.21.15.47:0/646896445 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7ff3f4066e70 0x7ff3f4069330 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:09.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:09.105+0000 7ff41e200640 1 --2- 172.21.15.47:0/646896445 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4180c7f50 0x7ff4180c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:09.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:09.105+0000 7ff41e200640 1 --2- 172.21.15.47:0/646896445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff418140740 0x7ff418143320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:09.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:09.105+0000 7ff41e200640 1 --2- 172.21.15.47:0/646896445 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3fc0051e0 0x7ff4180c5770 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:09.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:09.105+0000 7ff41e200640 1 -- 172.21.15.47:0/646896445 >> 172.21.15.47:0/646896445 conn(0x7ff4180b16a0 msgr2=0x7ff4180aeca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:09.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:09.105+0000 7ff41e200640 1 -- 172.21.15.47:0/646896445 shutdown_connections 2024-02-19T15:31:09.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:09.105+0000 7ff41e200640 1 -- 172.21.15.47:0/646896445 wait complete. 2024-02-19T15:31:09.328 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:09.328 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:10.248 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:10.330 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:10.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.469+0000 7f013b200640 1 Processor -- start 2024-02-19T15:31:10.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.469+0000 7f013b200640 1 -- start start 2024-02-19T15:31:10.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.473+0000 7f013b200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f013c15dd20 0x7f013c0ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:10.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.473+0000 7f013b200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f013c15e100 0x7f013c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:10.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.473+0000 7f013b200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f013c0ce6e0 con 0x7f013c0cdf90 2024-02-19T15:31:10.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.473+0000 7f013b200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f013c0acec0 con 0x7f013c15e100 2024-02-19T15:31:10.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.473+0000 7f013b200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f013c0ce420 con 0x7f013c15dd20 2024-02-19T15:31:10.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.473+0000 7f0133e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f013c15e100 0x7f013c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:10.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.473+0000 7f0133e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f013c15e100 0x7f013c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57920/0 (socket says 172.21.15.47:57920) 2024-02-19T15:31:10.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.473+0000 7f0133e00640 1 -- 172.21.15.47:0/592949495 learned_addr learned my addr 172.21.15.47:0/592949495 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:10.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.473+0000 7f0139e00640 1 --2- 172.21.15.47:0/592949495 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f013c15dd20 0x7f013c0ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:10.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.473+0000 7f0133e00640 1 -- 172.21.15.47:0/592949495 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f013c15dd20 msgr2=0x7f013c0ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:10.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.473+0000 7f0133e00640 1 --2- 172.21.15.47:0/592949495 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f013c15dd20 0x7f013c0ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:10.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.473+0000 7f0133e00640 1 -- 172.21.15.47:0/592949495 >> v1:172.21.15.116:6789/0 conn(0x7f013c0cdf90 legacy=0x7f013c0cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:10.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.473+0000 7f0133e00640 1 -- 172.21.15.47:0/592949495 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f013c06c300 con 0x7f013c15e100 2024-02-19T15:31:10.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.477+0000 7f0133e00640 1 --2- 172.21.15.47:0/592949495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f013c15e100 0x7f013c06b9a0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f0124009870 tx=0x7f012401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=821156cf78719771 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:10.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.477+0000 7f0139400640 1 -- 172.21.15.47:0/592949495 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01240ab610 con 0x7f013c15e100 2024-02-19T15:31:10.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.477+0000 7f0139400640 1 -- 172.21.15.47:0/592949495 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01240ab770 con 0x7f013c15e100 2024-02-19T15:31:10.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.477+0000 7f0139400640 1 -- 172.21.15.47:0/592949495 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01240abb00 con 0x7f013c15e100 2024-02-19T15:31:10.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.477+0000 7f013b200640 1 -- 172.21.15.47:0/592949495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f013c15e100 msgr2=0x7f013c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:10.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.477+0000 7f013b200640 1 --2- 172.21.15.47:0/592949495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f013c15e100 0x7f013c06b9a0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f0124009870 tx=0x7f012401c410 comp rx=0 tx=0).stop 2024-02-19T15:31:10.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.477+0000 7f013b200640 1 -- 172.21.15.47:0/592949495 shutdown_connections 2024-02-19T15:31:10.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.477+0000 7f013b200640 1 --2- 172.21.15.47:0/592949495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f013c15e100 0x7f013c06b9a0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:10.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.477+0000 7f013b200640 1 --2- 172.21.15.47:0/592949495 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f013c15dd20 0x7f013c0ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:10.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.477+0000 7f013b200640 1 -- 172.21.15.47:0/592949495 >> 172.21.15.47:0/592949495 conn(0x7f013c0b1be0 msgr2=0x7f013c0b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:10.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.477+0000 7f013b200640 1 -- 172.21.15.47:0/592949495 shutdown_connections 2024-02-19T15:31:10.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.477+0000 7f013b200640 1 -- 172.21.15.47:0/592949495 wait complete. 2024-02-19T15:31:10.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.477+0000 7f013b200640 1 Processor -- start 2024-02-19T15:31:10.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.477+0000 7f013b200640 1 -- start start 2024-02-19T15:31:10.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.477+0000 7f013b200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f013c0cdf90 0x7f013c15c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:10.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.477+0000 7f013b200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f013c15dd20 0x7f013c15a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:10.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.477+0000 7f013b200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f013c15e100 0x7f013c15ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:10.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.477+0000 7f013b200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013c0c7620 con 0x7f013c15dd20 2024-02-19T15:31:10.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.477+0000 7f013b200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013c15b050 con 0x7f013c0cdf90 2024-02-19T15:31:10.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.477+0000 7f013b200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f013c0c8ac0 con 0x7f013c15e100 2024-02-19T15:31:10.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.481+0000 7f0133e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f013c15dd20 0x7f013c15a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:10.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.481+0000 7f0133e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f013c15dd20 0x7f013c15a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57926/0 (socket says 172.21.15.47:57926) 2024-02-19T15:31:10.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.481+0000 7f0133e00640 1 -- 172.21.15.47:0/3085570295 learned_addr learned my addr 172.21.15.47:0/3085570295 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:10.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.481+0000 7f0139e00640 1 --2- 172.21.15.47:0/3085570295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f013c0cdf90 0x7f013c15c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:10.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.481+0000 7f013a800640 1 --2- 172.21.15.47:0/3085570295 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f013c15e100 0x7f013c15ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:10.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.481+0000 7f0133e00640 1 -- 172.21.15.47:0/3085570295 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f013c15e100 msgr2=0x7f013c15ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:10.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.481+0000 7f0133e00640 1 --2- 172.21.15.47:0/3085570295 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f013c15e100 0x7f013c15ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:10.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.481+0000 7f0133e00640 1 -- 172.21.15.47:0/3085570295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f013c0cdf90 msgr2=0x7f013c15c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:10.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.481+0000 7f0133e00640 1 --2- 172.21.15.47:0/3085570295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f013c0cdf90 0x7f013c15c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:10.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.481+0000 7f0133e00640 1 -- 172.21.15.47:0/3085570295 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01240094d0 con 0x7f013c15dd20 2024-02-19T15:31:10.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.481+0000 7f0133e00640 1 --2- 172.21.15.47:0/3085570295 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f013c15dd20 0x7f013c15a5d0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f012401a040 tx=0x7f012401ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:10.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.481+0000 7f0132000640 1 -- 172.21.15.47:0/3085570295 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01240b4800 con 0x7f013c15dd20 2024-02-19T15:31:10.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.481+0000 7f0132000640 1 -- 172.21.15.47:0/3085570295 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01240b4960 con 0x7f013c15dd20 2024-02-19T15:31:10.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.481+0000 7f0132000640 1 -- 172.21.15.47:0/3085570295 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01240b4d10 con 0x7f013c15dd20 2024-02-19T15:31:10.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.481+0000 7f013b200640 1 -- 172.21.15.47:0/3085570295 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f013c09b8c0 con 0x7f013c15dd20 2024-02-19T15:31:10.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.481+0000 7f013b200640 1 -- 172.21.15.47:0/3085570295 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f013c09be80 con 0x7f013c15dd20 2024-02-19T15:31:10.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.481+0000 7f013b200640 1 -- 172.21.15.47:0/3085570295 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f013c0c8c50 con 0x7f013c15dd20 2024-02-19T15:31:10.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.481+0000 7f0132000640 1 -- 172.21.15.47:0/3085570295 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f01240c7440 con 0x7f013c15dd20 2024-02-19T15:31:10.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.485+0000 7f0132000640 1 --2- 172.21.15.47:0/3085570295 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0118066e40 0x7f0118069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:10.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.485+0000 7f0132000640 1 -- 172.21.15.47:0/3085570295 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f012411f4f0 con 0x7f013c15dd20 2024-02-19T15:31:10.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.485+0000 7f0139e00640 1 --2- 172.21.15.47:0/3085570295 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0118066e40 0x7f0118069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:10.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.485+0000 7f0139e00640 1 --2- 172.21.15.47:0/3085570295 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0118066e40 0x7f0118069300 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f013c168c00 tx=0x7f012c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:10.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.489+0000 7f0132000640 1 -- 172.21.15.47:0/3085570295 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f013c0c8c50 con 0x7f013c15dd20 2024-02-19T15:31:10.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.685+0000 7f013b200640 1 -- 172.21.15.47:0/3085570295 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f013c0c8c00 con 0x7f013c15dd20 2024-02-19T15:31:10.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.685+0000 7f0132000640 1 -- 172.21.15.47:0/3085570295 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f01240ec340 con 0x7f013c15dd20 2024-02-19T15:31:10.685 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:10.685 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:10.685 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:31:10.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.689+0000 7f013b200640 1 -- 172.21.15.47:0/3085570295 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0118066e40 msgr2=0x7f0118069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:10.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.689+0000 7f013b200640 1 --2- 172.21.15.47:0/3085570295 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0118066e40 0x7f0118069300 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f013c168c00 tx=0x7f012c009190 comp rx=0 tx=0).stop 2024-02-19T15:31:10.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.689+0000 7f013b200640 1 -- 172.21.15.47:0/3085570295 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f013c15dd20 msgr2=0x7f013c15a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:10.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.689+0000 7f013b200640 1 --2- 172.21.15.47:0/3085570295 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f013c15dd20 0x7f013c15a5d0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f012401a040 tx=0x7f012401ac50 comp rx=0 tx=0).stop 2024-02-19T15:31:10.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.689+0000 7f013b200640 1 -- 172.21.15.47:0/3085570295 shutdown_connections 2024-02-19T15:31:10.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.689+0000 7f013b200640 1 --2- 172.21.15.47:0/3085570295 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0118066e40 0x7f0118069300 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:10.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.689+0000 7f013b200640 1 --2- 172.21.15.47:0/3085570295 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f013c15e100 0x7f013c15ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:10.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.689+0000 7f013b200640 1 --2- 172.21.15.47:0/3085570295 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f013c15dd20 0x7f013c15a5d0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:10.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.689+0000 7f013b200640 1 --2- 172.21.15.47:0/3085570295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f013c0cdf90 0x7f013c15c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:10.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.689+0000 7f013b200640 1 -- 172.21.15.47:0/3085570295 >> 172.21.15.47:0/3085570295 conn(0x7f013c0b1be0 msgr2=0x7f013c06b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:10.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.689+0000 7f013b200640 1 -- 172.21.15.47:0/3085570295 shutdown_connections 2024-02-19T15:31:10.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:10.689+0000 7f013b200640 1 -- 172.21.15.47:0/3085570295 wait complete. 2024-02-19T15:31:10.907 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:10.908 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:11.909 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:12.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.049+0000 7faba2200640 1 Processor -- start 2024-02-19T15:31:12.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.049+0000 7faba2200640 1 -- start start 2024-02-19T15:31:12.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.053+0000 7faba2200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab9c0cea80 0x7fab9c0727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:12.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.053+0000 7faba2200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fab9c15c390 0x7fab9c15c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:12.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.053+0000 7faba2200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab9c15d280 con 0x7fab9c145590 2024-02-19T15:31:12.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.053+0000 7faba2200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab9c070ff0 con 0x7fab9c15c390 2024-02-19T15:31:12.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.053+0000 7faba2200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c15cd60 con 0x7fab9c0cea80 2024-02-19T15:31:12.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.053+0000 7fab9be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fab9c15c390 0x7fab9c15c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:12.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.053+0000 7fab9be00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fab9c15c390 0x7fab9c15c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57940/0 (socket says 172.21.15.47:57940) 2024-02-19T15:31:12.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.053+0000 7fab9be00640 1 -- 172.21.15.47:0/4056962761 learned_addr learned my addr 172.21.15.47:0/4056962761 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:12.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.053+0000 7fab9b400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fab9c145590 0x7fab9c06c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42608/0 (socket says 172.21.15.47:42608) 2024-02-19T15:31:12.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.053+0000 7fab9aa00640 1 --2- 172.21.15.47:0/4056962761 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab9c0cea80 0x7fab9c0727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:12.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.053+0000 7fab9be00640 1 -- 172.21.15.47:0/4056962761 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab9c0cea80 msgr2=0x7fab9c0727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:12.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.053+0000 7fab9be00640 1 --2- 172.21.15.47:0/4056962761 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab9c0cea80 0x7fab9c0727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:12.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.053+0000 7fab9be00640 1 -- 172.21.15.47:0/4056962761 >> v1:172.21.15.47:6789/0 conn(0x7fab9c145590 legacy=0x7fab9c06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:12.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.053+0000 7fab9be00640 1 -- 172.21.15.47:0/4056962761 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab9c0709e0 con 0x7fab9c15c390 2024-02-19T15:31:12.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.053+0000 7fab9be00640 1 --2- 172.21.15.47:0/4056962761 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab9c15c390 0x7fab9c15c820 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fab94009e50 tx=0x7fab9401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e293dc6028868e4c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:12.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7fab9a000640 1 -- 172.21.15.47:0/4056962761 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab940a5810 con 0x7fab9c15c390 2024-02-19T15:31:12.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7fab9a000640 1 -- 172.21.15.47:0/4056962761 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab940a5970 con 0x7fab9c15c390 2024-02-19T15:31:12.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7fab9a000640 1 -- 172.21.15.47:0/4056962761 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab940a5d00 con 0x7fab9c15c390 2024-02-19T15:31:12.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7faba2200640 1 -- 172.21.15.47:0/4056962761 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab9c15c390 msgr2=0x7fab9c15c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:12.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7faba2200640 1 --2- 172.21.15.47:0/4056962761 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab9c15c390 0x7fab9c15c820 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fab94009e50 tx=0x7fab9401ca30 comp rx=0 tx=0).stop 2024-02-19T15:31:12.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7faba2200640 1 -- 172.21.15.47:0/4056962761 shutdown_connections 2024-02-19T15:31:12.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7faba2200640 1 --2- 172.21.15.47:0/4056962761 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab9c15c390 0x7fab9c15c820 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:12.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7faba2200640 1 --2- 172.21.15.47:0/4056962761 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab9c0cea80 0x7fab9c0727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:12.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7faba2200640 1 -- 172.21.15.47:0/4056962761 >> 172.21.15.47:0/4056962761 conn(0x7fab9c0b16a0 msgr2=0x7fab9c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:12.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7faba2200640 1 -- 172.21.15.47:0/4056962761 shutdown_connections 2024-02-19T15:31:12.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7faba2200640 1 -- 172.21.15.47:0/4056962761 wait complete. 2024-02-19T15:31:12.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7faba2200640 1 Processor -- start 2024-02-19T15:31:12.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7faba2200640 1 -- start start 2024-02-19T15:31:12.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7faba2200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab9c0cea80 0x7fab9c06d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:12.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7faba2200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab9c145590 0x7fab9c0af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:12.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7faba2200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab9c15c390 0x7fab9c15b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:12.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7faba2200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c1438e0 con 0x7fab9c145590 2024-02-19T15:31:12.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7faba2200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c142a30 con 0x7fab9c0cea80 2024-02-19T15:31:12.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7faba2200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c142c60 con 0x7fab9c15c390 2024-02-19T15:31:12.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7fab9aa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab9c145590 0x7fab9c0af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:12.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7fab9aa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab9c145590 0x7fab9c0af9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57952/0 (socket says 172.21.15.47:57952) 2024-02-19T15:31:12.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7fab9aa00640 1 -- 172.21.15.47:0/429236117 learned_addr learned my addr 172.21.15.47:0/429236117 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:12.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7fab9be00640 1 --2- 172.21.15.47:0/429236117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab9c15c390 0x7fab9c15b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:12.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.057+0000 7fab9b400640 1 --2- 172.21.15.47:0/429236117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab9c0cea80 0x7fab9c06d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:12.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.061+0000 7fab9aa00640 1 -- 172.21.15.47:0/429236117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab9c15c390 msgr2=0x7fab9c15b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:12.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.061+0000 7fab9aa00640 1 --2- 172.21.15.47:0/429236117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab9c15c390 0x7fab9c15b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:12.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.061+0000 7fab9aa00640 1 -- 172.21.15.47:0/429236117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab9c0cea80 msgr2=0x7fab9c06d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:12.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.061+0000 7fab9aa00640 1 --2- 172.21.15.47:0/429236117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab9c0cea80 0x7fab9c06d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:12.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.061+0000 7fab9aa00640 1 -- 172.21.15.47:0/429236117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab94009b30 con 0x7fab9c145590 2024-02-19T15:31:12.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.061+0000 7fab9be00640 1 --2- 172.21.15.47:0/429236117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab9c15c390 0x7fab9c15b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:31:12.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.061+0000 7fab9aa00640 1 --2- 172.21.15.47:0/429236117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab9c145590 0x7fab9c0af9b0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fab8800d800 tx=0x7fab8800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:12.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.061+0000 7fab93400640 1 -- 172.21.15.47:0/429236117 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8800b6c0 con 0x7fab9c145590 2024-02-19T15:31:12.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.061+0000 7fab93400640 1 -- 172.21.15.47:0/429236117 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab8800b820 con 0x7fab9c145590 2024-02-19T15:31:12.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.061+0000 7fab93400640 1 -- 172.21.15.47:0/429236117 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8800bb90 con 0x7fab9c145590 2024-02-19T15:31:12.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.061+0000 7faba2200640 1 -- 172.21.15.47:0/429236117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab9c142f40 con 0x7fab9c145590 2024-02-19T15:31:12.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.061+0000 7faba2200640 1 -- 172.21.15.47:0/429236117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab9c143500 con 0x7fab9c145590 2024-02-19T15:31:12.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.061+0000 7faba2200640 1 -- 172.21.15.47:0/429236117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab9c075360 con 0x7fab9c145590 2024-02-19T15:31:12.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.061+0000 7fab93400640 1 -- 172.21.15.47:0/429236117 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fab8800bcf0 con 0x7fab9c145590 2024-02-19T15:31:12.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.065+0000 7fab93400640 1 --2- 172.21.15.47:0/429236117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fab7c066e40 0x7fab7c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:12.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.065+0000 7fab93400640 1 -- 172.21.15.47:0/429236117 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fab88084fe0 con 0x7fab9c145590 2024-02-19T15:31:12.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.065+0000 7fab9b400640 1 --2- 172.21.15.47:0/429236117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fab7c066e40 0x7fab7c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:12.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.065+0000 7fab9b400640 1 --2- 172.21.15.47:0/429236117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fab7c066e40 0x7fab7c069300 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fab9c13a490 tx=0x7fab8c007cf0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:12.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.069+0000 7fab93400640 1 -- 172.21.15.47:0/429236117 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fab88051da0 con 0x7fab9c145590 2024-02-19T15:31:12.250 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:12.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.281+0000 7faba2200640 1 -- 172.21.15.47:0/429236117 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab9c06d230 con 0x7fab9c145590 2024-02-19T15:31:12.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.285+0000 7fab93400640 1 -- 172.21.15.47:0/429236117 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7fab88055460 con 0x7fab9c145590 2024-02-19T15:31:12.283 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:12.283 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:12.283 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:31:12.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.285+0000 7faba2200640 1 -- 172.21.15.47:0/429236117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fab7c066e40 msgr2=0x7fab7c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:12.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.285+0000 7faba2200640 1 --2- 172.21.15.47:0/429236117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fab7c066e40 0x7fab7c069300 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fab9c13a490 tx=0x7fab8c007cf0 comp rx=0 tx=0).stop 2024-02-19T15:31:12.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.285+0000 7faba2200640 1 -- 172.21.15.47:0/429236117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab9c145590 msgr2=0x7fab9c0af9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:12.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.285+0000 7faba2200640 1 --2- 172.21.15.47:0/429236117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab9c145590 0x7fab9c0af9b0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fab8800d800 tx=0x7fab8800dcc0 comp rx=0 tx=0).stop 2024-02-19T15:31:12.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.285+0000 7faba2200640 1 -- 172.21.15.47:0/429236117 shutdown_connections 2024-02-19T15:31:12.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.285+0000 7faba2200640 1 --2- 172.21.15.47:0/429236117 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fab7c066e40 0x7fab7c069300 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:12.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.285+0000 7faba2200640 1 --2- 172.21.15.47:0/429236117 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fab9c15c390 0x7fab9c15b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:12.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.285+0000 7faba2200640 1 --2- 172.21.15.47:0/429236117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab9c145590 0x7fab9c0af9b0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:12.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.285+0000 7faba2200640 1 --2- 172.21.15.47:0/429236117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab9c0cea80 0x7fab9c06d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:12.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.285+0000 7faba2200640 1 -- 172.21.15.47:0/429236117 >> 172.21.15.47:0/429236117 conn(0x7fab9c0b16a0 msgr2=0x7fab9c141a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:12.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.285+0000 7faba2200640 1 -- 172.21.15.47:0/429236117 shutdown_connections 2024-02-19T15:31:12.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:12.285+0000 7faba2200640 1 -- 172.21.15.47:0/429236117 wait complete. 2024-02-19T15:31:12.505 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:12.506 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:13.506 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:13.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.649+0000 7fdad3200640 1 Processor -- start 2024-02-19T15:31:13.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.649+0000 7fdad3200640 1 -- start start 2024-02-19T15:31:13.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.649+0000 7fdad3200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdad415dd20 0x7fdad40ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:13.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.649+0000 7fdad3200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdad40cd020 0x7fdad406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:13.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.649+0000 7fdad3200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad40ce6e0 con 0x7fdad415e100 2024-02-19T15:31:13.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.649+0000 7fdad3200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdad40acec0 con 0x7fdad40cd020 2024-02-19T15:31:13.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.649+0000 7fdad3200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad406c300 con 0x7fdad415dd20 2024-02-19T15:31:13.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.649+0000 7fdad1e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdad415dd20 0x7fdad40ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:13.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.649+0000 7fdad2800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdad40cd020 0x7fdad406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:13.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.649+0000 7fdacbe00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdad415e100 0x7fdad40cc910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:51650/0 (socket says 172.21.15.47:51650) 2024-02-19T15:31:13.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.649+0000 7fdad1e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdad415dd20 0x7fdad40ca6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:33126/0 (socket says 172.21.15.47:33126) 2024-02-19T15:31:13.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.649+0000 7fdacbe00640 1 -- 172.21.15.47:0/1520808936 learned_addr learned my addr 172.21.15.47:0/1520808936 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:31:13.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.649+0000 7fdad1e00640 1 -- 172.21.15.47:0/1520808936 >> v2:172.21.15.116:3300/0 conn(0x7fdad40cd020 msgr2=0x7fdad406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:13.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.649+0000 7fdad1e00640 1 --2- 172.21.15.47:0/1520808936 >> v2:172.21.15.116:3300/0 conn(0x7fdad40cd020 0x7fdad406b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:13.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.649+0000 7fdad1e00640 1 -- 172.21.15.47:0/1520808936 >> v1:172.21.15.116:6789/0 conn(0x7fdad415e100 legacy=0x7fdad40cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:13.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.649+0000 7fdad1e00640 1 -- 172.21.15.47:0/1520808936 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad406c5e0 con 0x7fdad415dd20 2024-02-19T15:31:13.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.649+0000 7fdad1400640 1 -- 172.21.15.47:0/1520808936 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403700909 0 0) 0x7fdad40ce6e0 con 0x7fdad415e100 2024-02-19T15:31:13.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.653+0000 7fdad1e00640 1 --2- 172.21.15.47:0/1520808936 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdad415dd20 0x7fdad40ca6e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdac40097c0 tx=0x7fdac401c350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b5dbe35217022c1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:13.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.653+0000 7fdad1400640 1 -- 172.21.15.47:0/1520808936 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac40ab600 con 0x7fdad415dd20 2024-02-19T15:31:13.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.653+0000 7fdad1400640 1 -- 172.21.15.47:0/1520808936 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac40a7050 con 0x7fdad415dd20 2024-02-19T15:31:13.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.653+0000 7fdad1400640 1 -- 172.21.15.47:0/1520808936 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac40ab980 con 0x7fdad415dd20 2024-02-19T15:31:13.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.653+0000 7fdad3200640 1 -- 172.21.15.47:0/1520808936 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdad415dd20 msgr2=0x7fdad40ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:13.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.653+0000 7fdad3200640 1 --2- 172.21.15.47:0/1520808936 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdad415dd20 0x7fdad40ca6e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdac40097c0 tx=0x7fdac401c350 comp rx=0 tx=0).stop 2024-02-19T15:31:13.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.653+0000 7fdad3200640 1 -- 172.21.15.47:0/1520808936 shutdown_connections 2024-02-19T15:31:13.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.653+0000 7fdad3200640 1 --2- 172.21.15.47:0/1520808936 >> v2:172.21.15.116:3300/0 conn(0x7fdad40cd020 0x7fdad406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:13.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.653+0000 7fdad3200640 1 --2- 172.21.15.47:0/1520808936 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdad415dd20 0x7fdad40ca6e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:13.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.653+0000 7fdad3200640 1 -- 172.21.15.47:0/1520808936 >> 172.21.15.47:0/1520808936 conn(0x7fdad40b1be0 msgr2=0x7fdad40b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:13.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.653+0000 7fdad3200640 1 -- 172.21.15.47:0/1520808936 shutdown_connections 2024-02-19T15:31:13.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.653+0000 7fdad3200640 1 -- 172.21.15.47:0/1520808936 wait complete. 2024-02-19T15:31:13.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdad3200640 1 Processor -- start 2024-02-19T15:31:13.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdad3200640 1 -- start start 2024-02-19T15:31:13.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdad3200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdad40cd020 0x7fdad406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:13.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdad3200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad415dd20 0x7fdad41552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:13.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdad3200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdad415e100 0x7fdad40c8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:13.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdad3200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad4155b80 con 0x7fdad40cd020 2024-02-19T15:31:13.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdad3200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad415ae80 con 0x7fdad415dd20 2024-02-19T15:31:13.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdad3200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad415b160 con 0x7fdad415e100 2024-02-19T15:31:13.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdad1e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdad40cd020 0x7fdad406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:13.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdad1e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdad40cd020 0x7fdad406ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42574/0 (socket says 172.21.15.47:42574) 2024-02-19T15:31:13.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdad1e00640 1 -- 172.21.15.47:0/2755044931 learned_addr learned my addr 172.21.15.47:0/2755044931 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:13.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdad2800640 1 --2- 172.21.15.47:0/2755044931 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdad415e100 0x7fdad40c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:13.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdacbe00640 1 --2- 172.21.15.47:0/2755044931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad415dd20 0x7fdad41552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:13.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdad1e00640 1 -- 172.21.15.47:0/2755044931 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdad415e100 msgr2=0x7fdad40c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:13.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdad1e00640 1 --2- 172.21.15.47:0/2755044931 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdad415e100 0x7fdad40c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:13.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdad1e00640 1 -- 172.21.15.47:0/2755044931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad415dd20 msgr2=0x7fdad41552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:13.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdad1e00640 1 --2- 172.21.15.47:0/2755044931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad415dd20 0x7fdad41552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:13.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdad1e00640 1 -- 172.21.15.47:0/2755044931 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac4009420 con 0x7fdad40cd020 2024-02-19T15:31:13.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdad1e00640 1 --2- 172.21.15.47:0/2755044931 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdad40cd020 0x7fdad406ea10 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fdac40ae040 tx=0x7fdac401ab90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:13.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdaca000640 1 -- 172.21.15.47:0/2755044931 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac4004020 con 0x7fdad40cd020 2024-02-19T15:31:13.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdaca000640 1 -- 172.21.15.47:0/2755044931 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac4004180 con 0x7fdad40cd020 2024-02-19T15:31:13.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdaca000640 1 -- 172.21.15.47:0/2755044931 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac4004510 con 0x7fdad40cd020 2024-02-19T15:31:13.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdad3200640 1 -- 172.21.15.47:0/2755044931 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdad4155dd0 con 0x7fdad40cd020 2024-02-19T15:31:13.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.657+0000 7fdad3200640 1 -- 172.21.15.47:0/2755044931 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdad41557f0 con 0x7fdad40cd020 2024-02-19T15:31:13.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.661+0000 7fdad3200640 1 -- 172.21.15.47:0/2755044931 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdad4075360 con 0x7fdad40cd020 2024-02-19T15:31:13.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.661+0000 7fdaca000640 1 -- 172.21.15.47:0/2755044931 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fdac4003620 con 0x7fdad40cd020 2024-02-19T15:31:13.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.661+0000 7fdaca000640 1 --2- 172.21.15.47:0/2755044931 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdab0066e40 0x7fdab0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:13.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.661+0000 7fdaca000640 1 -- 172.21.15.47:0/2755044931 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdac411eee0 con 0x7fdad40cd020 2024-02-19T15:31:13.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.661+0000 7fdacbe00640 1 --2- 172.21.15.47:0/2755044931 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdab0066e40 0x7fdab0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:13.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.661+0000 7fdacbe00640 1 --2- 172.21.15.47:0/2755044931 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdab0066e40 0x7fdab0069300 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fdad40cc960 tx=0x7fdabc007d60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:13.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.669+0000 7fdaca000640 1 -- 172.21.15.47:0/2755044931 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fdac40ebdd0 con 0x7fdad40cd020 2024-02-19T15:31:13.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.865+0000 7fdad3200640 1 -- 172.21.15.47:0/2755044931 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdad4066e40 con 0x7fdad40cd020 2024-02-19T15:31:13.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.865+0000 7fdaca000640 1 -- 172.21.15.47:0/2755044931 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7fdac40bb020 con 0x7fdad40cd020 2024-02-19T15:31:13.864 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:13.864 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:13.864 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:31:13.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.865+0000 7fdad3200640 1 -- 172.21.15.47:0/2755044931 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdab0066e40 msgr2=0x7fdab0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:13.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.865+0000 7fdad3200640 1 --2- 172.21.15.47:0/2755044931 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdab0066e40 0x7fdab0069300 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fdad40cc960 tx=0x7fdabc007d60 comp rx=0 tx=0).stop 2024-02-19T15:31:13.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.865+0000 7fdad3200640 1 -- 172.21.15.47:0/2755044931 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdad40cd020 msgr2=0x7fdad406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:13.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.865+0000 7fdad3200640 1 --2- 172.21.15.47:0/2755044931 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdad40cd020 0x7fdad406ea10 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fdac40ae040 tx=0x7fdac401ab90 comp rx=0 tx=0).stop 2024-02-19T15:31:13.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.865+0000 7fdad3200640 1 -- 172.21.15.47:0/2755044931 shutdown_connections 2024-02-19T15:31:13.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.865+0000 7fdad3200640 1 --2- 172.21.15.47:0/2755044931 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdab0066e40 0x7fdab0069300 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:13.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.865+0000 7fdad3200640 1 --2- 172.21.15.47:0/2755044931 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdad415e100 0x7fdad40c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:13.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.865+0000 7fdad3200640 1 --2- 172.21.15.47:0/2755044931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad415dd20 0x7fdad41552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:13.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.865+0000 7fdad3200640 1 --2- 172.21.15.47:0/2755044931 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdad40cd020 0x7fdad406ea10 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:13.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.865+0000 7fdad3200640 1 -- 172.21.15.47:0/2755044931 >> 172.21.15.47:0/2755044931 conn(0x7fdad40b1be0 msgr2=0x7fdad40caf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:13.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.865+0000 7fdad3200640 1 -- 172.21.15.47:0/2755044931 shutdown_connections 2024-02-19T15:31:13.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:13.865+0000 7fdad3200640 1 -- 172.21.15.47:0/2755044931 wait complete. 2024-02-19T15:31:14.087 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:14.088 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:14.252 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:15.089 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:15.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.229+0000 7fa64de00640 1 Processor -- start 2024-02-19T15:31:15.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.233+0000 7fa64de00640 1 -- start start 2024-02-19T15:31:15.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.233+0000 7fa64de00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa64806d120 0x7fa64806d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:15.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.233+0000 7fa64de00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64806c3e0 con 0x7fa6480d19e0 2024-02-19T15:31:15.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.233+0000 7fa64de00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64806c680 con 0x7fa64815dd70 2024-02-19T15:31:15.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.233+0000 7fa64de00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa64806daf0 con 0x7fa64806d120 2024-02-19T15:31:15.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.233+0000 7fa647e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa64806d120 0x7fa64806d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:15.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.233+0000 7fa647e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa64806d120 0x7fa64806d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42590/0 (socket says 172.21.15.47:42590) 2024-02-19T15:31:15.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.233+0000 7fa64d400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa6480d19e0 0x7fa64815e150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:40878/0 (socket says 172.21.15.47:40878) 2024-02-19T15:31:15.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.233+0000 7fa647e00640 1 -- 172.21.15.47:0/2928676534 learned_addr learned my addr 172.21.15.47:0/2928676534 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:15.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.233+0000 7fa647e00640 1 -- 172.21.15.47:0/2928676534 >> v1:172.21.15.116:6789/0 conn(0x7fa64815dd70 legacy=0x7fa6480ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:15.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.233+0000 7fa647e00640 1 -- 172.21.15.47:0/2928676534 >> v1:172.21.15.47:6789/0 conn(0x7fa6480d19e0 legacy=0x7fa64815e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:15.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.233+0000 7fa647e00640 1 -- 172.21.15.47:0/2928676534 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64815f110 con 0x7fa64806d120 2024-02-19T15:31:15.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.237+0000 7fa647e00640 1 --2- 172.21.15.47:0/2928676534 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64806d120 0x7fa64806d5b0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fa634009870 tx=0x7fa63401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37c687a5b6cde47f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:15.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.237+0000 7fa647400640 1 -- 172.21.15.47:0/2928676534 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6340ad680 con 0x7fa64806d120 2024-02-19T15:31:15.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.237+0000 7fa647400640 1 -- 172.21.15.47:0/2928676534 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6340ad7e0 con 0x7fa64806d120 2024-02-19T15:31:15.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.237+0000 7fa647400640 1 -- 172.21.15.47:0/2928676534 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6340adb70 con 0x7fa64806d120 2024-02-19T15:31:15.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.237+0000 7fa64de00640 1 -- 172.21.15.47:0/2928676534 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64806d120 msgr2=0x7fa64806d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:15.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.237+0000 7fa64de00640 1 --2- 172.21.15.47:0/2928676534 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64806d120 0x7fa64806d5b0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fa634009870 tx=0x7fa63401c490 comp rx=0 tx=0).stop 2024-02-19T15:31:15.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.237+0000 7fa64de00640 1 -- 172.21.15.47:0/2928676534 shutdown_connections 2024-02-19T15:31:15.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.237+0000 7fa64de00640 1 --2- 172.21.15.47:0/2928676534 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64806d120 0x7fa64806d5b0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:15.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.237+0000 7fa64de00640 1 -- 172.21.15.47:0/2928676534 >> 172.21.15.47:0/2928676534 conn(0x7fa64809a9d0 msgr2=0x7fa64809ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:15.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.237+0000 7fa64de00640 1 -- 172.21.15.47:0/2928676534 shutdown_connections 2024-02-19T15:31:15.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.237+0000 7fa64de00640 1 -- 172.21.15.47:0/2928676534 wait complete. 2024-02-19T15:31:15.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa64de00640 1 Processor -- start 2024-02-19T15:31:15.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa64de00640 1 -- start start 2024-02-19T15:31:15.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa64de00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa64806d120 0x7fa648151060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:15.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa64de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6480d19e0 0x7fa648155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:15.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa64de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64815dd70 0x7fa648156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:15.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa64de00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6481515a0 con 0x7fa64815dd70 2024-02-19T15:31:15.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa64de00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa648160230 con 0x7fa6480d19e0 2024-02-19T15:31:15.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa64de00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa648160510 con 0x7fa64806d120 2024-02-19T15:31:15.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa64d400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64815dd70 0x7fa648156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:15.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa64d400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64815dd70 0x7fa648156040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42604/0 (socket says 172.21.15.47:42604) 2024-02-19T15:31:15.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa64d400640 1 -- 172.21.15.47:0/3357926169 learned_addr learned my addr 172.21.15.47:0/3357926169 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:15.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa64ca00640 1 --2- 172.21.15.47:0/3357926169 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa64806d120 0x7fa648151060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:15.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa647e00640 1 --2- 172.21.15.47:0/3357926169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6480d19e0 0x7fa648155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:15.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa64d400640 1 -- 172.21.15.47:0/3357926169 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa64806d120 msgr2=0x7fa648151060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:15.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa64d400640 1 --2- 172.21.15.47:0/3357926169 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa64806d120 0x7fa648151060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:15.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa64d400640 1 -- 172.21.15.47:0/3357926169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6480d19e0 msgr2=0x7fa648155b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:15.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa64d400640 1 --2- 172.21.15.47:0/3357926169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6480d19e0 0x7fa648155b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:15.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa64d400640 1 -- 172.21.15.47:0/3357926169 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6340094d0 con 0x7fa64815dd70 2024-02-19T15:31:15.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa64d400640 1 --2- 172.21.15.47:0/3357926169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64815dd70 0x7fa648156040 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fa64000c7a0 tx=0x7fa64000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:15.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa644c00640 1 -- 172.21.15.47:0/3357926169 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64000a640 con 0x7fa64815dd70 2024-02-19T15:31:15.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa644c00640 1 -- 172.21.15.47:0/3357926169 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64000a7a0 con 0x7fa64815dd70 2024-02-19T15:31:15.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa644c00640 1 -- 172.21.15.47:0/3357926169 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64000ab10 con 0x7fa64815dd70 2024-02-19T15:31:15.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa64de00640 1 -- 172.21.15.47:0/3357926169 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6480d2380 con 0x7fa64815dd70 2024-02-19T15:31:15.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.241+0000 7fa64de00640 1 -- 172.21.15.47:0/3357926169 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa648160ad0 con 0x7fa64815dd70 2024-02-19T15:31:15.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.245+0000 7fa64de00640 1 -- 172.21.15.47:0/3357926169 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa648075360 con 0x7fa64815dd70 2024-02-19T15:31:15.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.245+0000 7fa644c00640 1 -- 172.21.15.47:0/3357926169 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa64000ac70 con 0x7fa64815dd70 2024-02-19T15:31:15.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.245+0000 7fa644c00640 1 --2- 172.21.15.47:0/3357926169 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa628066e40 0x7fa628069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:15.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.245+0000 7fa644c00640 1 -- 172.21.15.47:0/3357926169 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa6400842a0 con 0x7fa64815dd70 2024-02-19T15:31:15.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.245+0000 7fa64ca00640 1 --2- 172.21.15.47:0/3357926169 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa628066e40 0x7fa628069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:15.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.245+0000 7fa64ca00640 1 --2- 172.21.15.47:0/3357926169 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa628066e40 0x7fa628069300 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa63c0098e0 tx=0x7fa63c009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:15.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.249+0000 7fa644c00640 1 -- 172.21.15.47:0/3357926169 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa640051060 con 0x7fa64815dd70 2024-02-19T15:31:15.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.445+0000 7fa64de00640 1 -- 172.21.15.47:0/3357926169 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6480b0440 con 0x7fa64815dd70 2024-02-19T15:31:15.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.445+0000 7fa644c00640 1 -- 172.21.15.47:0/3357926169 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7fa640017d80 con 0x7fa64815dd70 2024-02-19T15:31:15.445 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:15.445 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:15.445 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:31:15.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.449+0000 7fa64de00640 1 -- 172.21.15.47:0/3357926169 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa628066e40 msgr2=0x7fa628069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:15.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.449+0000 7fa64de00640 1 --2- 172.21.15.47:0/3357926169 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa628066e40 0x7fa628069300 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa63c0098e0 tx=0x7fa63c009060 comp rx=0 tx=0).stop 2024-02-19T15:31:15.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.449+0000 7fa64de00640 1 -- 172.21.15.47:0/3357926169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64815dd70 msgr2=0x7fa648156040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:15.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.449+0000 7fa64de00640 1 --2- 172.21.15.47:0/3357926169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64815dd70 0x7fa648156040 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fa64000c7a0 tx=0x7fa64000cc60 comp rx=0 tx=0).stop 2024-02-19T15:31:15.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.449+0000 7fa64de00640 1 -- 172.21.15.47:0/3357926169 shutdown_connections 2024-02-19T15:31:15.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.449+0000 7fa64de00640 1 --2- 172.21.15.47:0/3357926169 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa628066e40 0x7fa628069300 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:15.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.449+0000 7fa64de00640 1 --2- 172.21.15.47:0/3357926169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64815dd70 0x7fa648156040 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:15.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.449+0000 7fa64de00640 1 --2- 172.21.15.47:0/3357926169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6480d19e0 0x7fa648155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:15.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.449+0000 7fa64de00640 1 --2- 172.21.15.47:0/3357926169 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa64806d120 0x7fa648151060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:15.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.449+0000 7fa64de00640 1 -- 172.21.15.47:0/3357926169 >> 172.21.15.47:0/3357926169 conn(0x7fa64809a9d0 msgr2=0x7fa64809adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:15.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.449+0000 7fa64de00640 1 -- 172.21.15.47:0/3357926169 shutdown_connections 2024-02-19T15:31:15.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:15.449+0000 7fa64de00640 1 -- 172.21.15.47:0/3357926169 wait complete. 2024-02-19T15:31:15.668 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:15.668 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:16.254 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:16.669 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:16.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.809+0000 7f4b37c00640 1 Processor -- start 2024-02-19T15:31:16.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.809+0000 7f4b37c00640 1 -- start start 2024-02-19T15:31:16.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.813+0000 7f4b37c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b300ce3c0 0x7f4b300ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:16.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.813+0000 7f4b37c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4b3015e570 0x7f4b3006c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:16.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.813+0000 7f4b37c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4b300b2e20 0x7f4b3006c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:16.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.813+0000 7f4b37c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b300d2ad0 con 0x7f4b300b2e20 2024-02-19T15:31:16.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.813+0000 7f4b37c00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b300cc190 con 0x7f4b3015e570 2024-02-19T15:31:16.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.813+0000 7f4b37c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b300cece0 con 0x7f4b300ce3c0 2024-02-19T15:31:16.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.813+0000 7f4b35c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4b300b2e20 0x7f4b3006c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:16.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.813+0000 7f4b35c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4b300b2e20 0x7f4b3006c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42612/0 (socket says 172.21.15.47:42612) 2024-02-19T15:31:16.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.813+0000 7f4b35c00640 1 -- 172.21.15.47:0/1732355784 learned_addr learned my addr 172.21.15.47:0/1732355784 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:16.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.813+0000 7f4b35200640 1 --2- 172.21.15.47:0/1732355784 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b300ce3c0 0x7f4b300ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:16.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.813+0000 7f4b27e00640 1 --2- 172.21.15.47:0/1732355784 >> v2:172.21.15.116:3300/0 conn(0x7f4b3015e570 0x7f4b3006c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:16.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.813+0000 7f4b35c00640 1 -- 172.21.15.47:0/1732355784 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b300ce3c0 msgr2=0x7f4b300ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:16.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.813+0000 7f4b35c00640 1 --2- 172.21.15.47:0/1732355784 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b300ce3c0 0x7f4b300ce7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:16.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.813+0000 7f4b35c00640 1 -- 172.21.15.47:0/1732355784 >> v2:172.21.15.116:3300/0 conn(0x7f4b3015e570 msgr2=0x7f4b3006c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:16.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.813+0000 7f4b35c00640 1 --2- 172.21.15.47:0/1732355784 >> v2:172.21.15.116:3300/0 conn(0x7f4b3015e570 0x7f4b3006c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:16.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.813+0000 7f4b35c00640 1 -- 172.21.15.47:0/1732355784 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b300d0ac0 con 0x7f4b300b2e20 2024-02-19T15:31:16.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.817+0000 7f4b35c00640 1 --2- 172.21.15.47:0/1732355784 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b300b2e20 0x7f4b3006c830 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f4b28009e50 tx=0x7f4b2801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=479d2ea0a24cebf3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:16.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.817+0000 7f4b2fe00640 1 -- 172.21.15.47:0/1732355784 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b280a5670 con 0x7f4b300b2e20 2024-02-19T15:31:16.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.817+0000 7f4b2fe00640 1 -- 172.21.15.47:0/1732355784 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b280a57d0 con 0x7f4b300b2e20 2024-02-19T15:31:16.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.817+0000 7f4b2fe00640 1 -- 172.21.15.47:0/1732355784 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b280a5b60 con 0x7f4b300b2e20 2024-02-19T15:31:16.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.817+0000 7f4b37c00640 1 -- 172.21.15.47:0/1732355784 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b300b2e20 msgr2=0x7f4b3006c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:16.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.817+0000 7f4b37c00640 1 --2- 172.21.15.47:0/1732355784 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b300b2e20 0x7f4b3006c830 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f4b28009e50 tx=0x7f4b2801c9f0 comp rx=0 tx=0).stop 2024-02-19T15:31:16.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.817+0000 7f4b37c00640 1 -- 172.21.15.47:0/1732355784 shutdown_connections 2024-02-19T15:31:16.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.817+0000 7f4b37c00640 1 --2- 172.21.15.47:0/1732355784 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b300b2e20 0x7f4b3006c830 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:16.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.817+0000 7f4b37c00640 1 --2- 172.21.15.47:0/1732355784 >> v2:172.21.15.116:3300/0 conn(0x7f4b3015e570 0x7f4b3006c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:16.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.817+0000 7f4b37c00640 1 --2- 172.21.15.47:0/1732355784 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b300ce3c0 0x7f4b300ce7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:16.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.817+0000 7f4b37c00640 1 -- 172.21.15.47:0/1732355784 >> 172.21.15.47:0/1732355784 conn(0x7f4b300d1b40 msgr2=0x7f4b300d1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:16.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.817+0000 7f4b37c00640 1 -- 172.21.15.47:0/1732355784 shutdown_connections 2024-02-19T15:31:16.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.817+0000 7f4b37c00640 1 -- 172.21.15.47:0/1732355784 wait complete. 2024-02-19T15:31:16.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.817+0000 7f4b37c00640 1 Processor -- start 2024-02-19T15:31:16.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.817+0000 7f4b37c00640 1 -- start start 2024-02-19T15:31:16.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b37c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b300b2e20 0x7f4b301520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:16.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b37c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b300ce3c0 0x7f4b30153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:16.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b37c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b3015e570 0x7f4b30153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:16.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b37c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3015a280 con 0x7f4b300ce3c0 2024-02-19T15:31:16.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b37c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b30159420 con 0x7f4b300b2e20 2024-02-19T15:31:16.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b37c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b301525f0 con 0x7f4b3015e570 2024-02-19T15:31:16.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b27e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b300ce3c0 0x7f4b30153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:16.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b27e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b300ce3c0 0x7f4b30153970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42624/0 (socket says 172.21.15.47:42624) 2024-02-19T15:31:16.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b27e00640 1 -- 172.21.15.47:0/1604760579 learned_addr learned my addr 172.21.15.47:0/1604760579 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:16.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b35c00640 1 --2- 172.21.15.47:0/1604760579 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b3015e570 0x7f4b30153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:16.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b35200640 1 --2- 172.21.15.47:0/1604760579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b300b2e20 0x7f4b301520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:16.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b27e00640 1 -- 172.21.15.47:0/1604760579 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b3015e570 msgr2=0x7f4b30153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:16.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b27e00640 1 --2- 172.21.15.47:0/1604760579 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b3015e570 0x7f4b30153eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:16.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b27e00640 1 -- 172.21.15.47:0/1604760579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b300b2e20 msgr2=0x7f4b301520b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:16.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b27e00640 1 --2- 172.21.15.47:0/1604760579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b300b2e20 0x7f4b301520b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:16.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b27e00640 1 -- 172.21.15.47:0/1604760579 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b28009b30 con 0x7f4b300ce3c0 2024-02-19T15:31:16.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b27e00640 1 --2- 172.21.15.47:0/1604760579 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b300ce3c0 0x7f4b30153970 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800e840 tx=0x7f4b1800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:16.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b2d600640 1 -- 172.21.15.47:0/1604760579 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18009510 con 0x7f4b300ce3c0 2024-02-19T15:31:16.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b2d600640 1 -- 172.21.15.47:0/1604760579 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1800ee90 con 0x7f4b300ce3c0 2024-02-19T15:31:16.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b37c00640 1 -- 172.21.15.47:0/1604760579 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b30159670 con 0x7f4b300ce3c0 2024-02-19T15:31:16.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b2d600640 1 -- 172.21.15.47:0/1604760579 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1800cb80 con 0x7f4b300ce3c0 2024-02-19T15:31:16.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b37c00640 1 -- 172.21.15.47:0/1604760579 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b30159c30 con 0x7f4b300ce3c0 2024-02-19T15:31:16.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b37c00640 1 -- 172.21.15.47:0/1604760579 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b3015e0c0 con 0x7f4b300ce3c0 2024-02-19T15:31:16.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.821+0000 7f4b2d600640 1 -- 172.21.15.47:0/1604760579 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4b1800cce0 con 0x7f4b300ce3c0 2024-02-19T15:31:16.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.825+0000 7f4b2d600640 1 --2- 172.21.15.47:0/1604760579 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4b0c066e40 0x7f4b0c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:16.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.825+0000 7f4b2d600640 1 -- 172.21.15.47:0/1604760579 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4b18084aa0 con 0x7f4b300ce3c0 2024-02-19T15:31:16.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.825+0000 7f4b35200640 1 --2- 172.21.15.47:0/1604760579 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4b0c066e40 0x7f4b0c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:16.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.825+0000 7f4b35200640 1 --2- 172.21.15.47:0/1604760579 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4b0c066e40 0x7f4b0c069300 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f4b20009ae0 tx=0x7f4b20009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:16.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:16.833+0000 7f4b2d600640 1 -- 172.21.15.47:0/1604760579 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f4b180518f0 con 0x7f4b300ce3c0 2024-02-19T15:31:17.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:17.025+0000 7f4b37c00640 1 -- 172.21.15.47:0/1604760579 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b300cfdd0 con 0x7f4b300ce3c0 2024-02-19T15:31:17.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:17.029+0000 7f4b2d600640 1 -- 172.21.15.47:0/1604760579 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f4b18021d80 con 0x7f4b300ce3c0 2024-02-19T15:31:17.027 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:17.027 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:17.027 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:31:17.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:17.029+0000 7f4b37c00640 1 -- 172.21.15.47:0/1604760579 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4b0c066e40 msgr2=0x7f4b0c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:17.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:17.029+0000 7f4b37c00640 1 --2- 172.21.15.47:0/1604760579 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4b0c066e40 0x7f4b0c069300 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f4b20009ae0 tx=0x7f4b20009190 comp rx=0 tx=0).stop 2024-02-19T15:31:17.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:17.029+0000 7f4b37c00640 1 -- 172.21.15.47:0/1604760579 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b300ce3c0 msgr2=0x7f4b30153970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:17.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:17.029+0000 7f4b37c00640 1 --2- 172.21.15.47:0/1604760579 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b300ce3c0 0x7f4b30153970 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800e840 tx=0x7f4b1800ec40 comp rx=0 tx=0).stop 2024-02-19T15:31:17.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:17.029+0000 7f4b37c00640 1 -- 172.21.15.47:0/1604760579 shutdown_connections 2024-02-19T15:31:17.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:17.029+0000 7f4b37c00640 1 --2- 172.21.15.47:0/1604760579 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4b0c066e40 0x7f4b0c069300 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:17.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:17.029+0000 7f4b37c00640 1 --2- 172.21.15.47:0/1604760579 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b3015e570 0x7f4b30153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:17.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:17.029+0000 7f4b37c00640 1 --2- 172.21.15.47:0/1604760579 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b300ce3c0 0x7f4b30153970 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:17.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:17.029+0000 7f4b37c00640 1 --2- 172.21.15.47:0/1604760579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b300b2e20 0x7f4b301520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:17.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:17.029+0000 7f4b37c00640 1 -- 172.21.15.47:0/1604760579 >> 172.21.15.47:0/1604760579 conn(0x7f4b300d1b40 msgr2=0x7f4b30070480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:17.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:17.029+0000 7f4b37c00640 1 -- 172.21.15.47:0/1604760579 shutdown_connections 2024-02-19T15:31:17.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:17.029+0000 7f4b37c00640 1 -- 172.21.15.47:0/1604760579 wait complete. 2024-02-19T15:31:17.250 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:17.250 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:18.251 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:18.255 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:18.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.393+0000 7f94ada00640 1 Processor -- start 2024-02-19T15:31:18.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.393+0000 7f94ada00640 1 -- start start 2024-02-19T15:31:18.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.393+0000 7f94ada00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f94a81456a0 0x7f94a80cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:18.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.393+0000 7f94ada00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94a80cf4a0 0x7f94a806aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:18.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.393+0000 7f94ada00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a806b850 con 0x7f94a806b3f0 2024-02-19T15:31:18.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.393+0000 7f94ada00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94a80d03c0 con 0x7f94a81456a0 2024-02-19T15:31:18.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.393+0000 7f94ada00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a80cf900 con 0x7f94a80cf4a0 2024-02-19T15:31:18.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.393+0000 7f94a6a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f94a81456a0 0x7f94a80cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:18.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.393+0000 7f949e000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94a80cf4a0 0x7f94a806aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:18.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.393+0000 7f94a6a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f94a81456a0 0x7f94a80cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42636/0 (socket says 172.21.15.47:42636) 2024-02-19T15:31:18.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.393+0000 7f949e000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94a80cf4a0 0x7f94a806aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:33162/0 (socket says 172.21.15.47:33162) 2024-02-19T15:31:18.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.393+0000 7f94a6a00640 1 -- 172.21.15.47:0/2796830066 learned_addr learned my addr 172.21.15.47:0/2796830066 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:18.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.393+0000 7f94a7400640 1 --1- 172.21.15.47:0/2796830066 >> v1:172.21.15.116:6789/0 conn(0x7f94a806b3f0 0x7f94a815f830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:51670/0 (socket says 172.21.15.47:51670) 2024-02-19T15:31:18.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.397+0000 7f94a6a00640 1 -- 172.21.15.47:0/2796830066 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94a80cf4a0 msgr2=0x7f94a806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:18.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.397+0000 7f94a6a00640 1 --2- 172.21.15.47:0/2796830066 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94a80cf4a0 0x7f94a806aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:18.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.397+0000 7f94a6a00640 1 -- 172.21.15.47:0/2796830066 >> v1:172.21.15.116:6789/0 conn(0x7f94a806b3f0 legacy=0x7f94a815f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:18.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.397+0000 7f94a6000640 1 -- 172.21.15.47:0/2796830066 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040953592 0 0) 0x7f94a806b850 con 0x7f94a806b3f0 2024-02-19T15:31:18.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.397+0000 7f94a6a00640 1 -- 172.21.15.47:0/2796830066 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a8145a80 con 0x7f94a81456a0 2024-02-19T15:31:18.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.401+0000 7f94a6a00640 1 --2- 172.21.15.47:0/2796830066 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94a81456a0 0x7f94a80cef60 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f94980098a0 tx=0x7f949801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5453320df2aaff24 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:18.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.401+0000 7f94a6000640 1 -- 172.21.15.47:0/2796830066 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94980ab600 con 0x7f94a81456a0 2024-02-19T15:31:18.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.401+0000 7f94a6000640 1 -- 172.21.15.47:0/2796830066 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94980a7050 con 0x7f94a81456a0 2024-02-19T15:31:18.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.401+0000 7f94a6000640 1 -- 172.21.15.47:0/2796830066 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94980ab980 con 0x7f94a81456a0 2024-02-19T15:31:18.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.401+0000 7f94ada00640 1 -- 172.21.15.47:0/2796830066 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94a81456a0 msgr2=0x7f94a80cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:18.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.401+0000 7f94ada00640 1 --2- 172.21.15.47:0/2796830066 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94a81456a0 0x7f94a80cef60 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f94980098a0 tx=0x7f949801c540 comp rx=0 tx=0).stop 2024-02-19T15:31:18.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.401+0000 7f94ada00640 1 -- 172.21.15.47:0/2796830066 shutdown_connections 2024-02-19T15:31:18.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.401+0000 7f94ada00640 1 --2- 172.21.15.47:0/2796830066 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94a80cf4a0 0x7f94a806aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:18.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.401+0000 7f94ada00640 1 --2- 172.21.15.47:0/2796830066 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94a81456a0 0x7f94a80cef60 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:18.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.401+0000 7f94ada00640 1 -- 172.21.15.47:0/2796830066 >> 172.21.15.47:0/2796830066 conn(0x7f94a80d0ad0 msgr2=0x7f94a80d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:18.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.401+0000 7f94ada00640 1 -- 172.21.15.47:0/2796830066 shutdown_connections 2024-02-19T15:31:18.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.401+0000 7f94ada00640 1 -- 172.21.15.47:0/2796830066 wait complete. 2024-02-19T15:31:18.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.401+0000 7f94ada00640 1 Processor -- start 2024-02-19T15:31:18.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.401+0000 7f94ada00640 1 -- start start 2024-02-19T15:31:18.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f94ada00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94a806b3f0 0x7f94a8156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:18.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f94ada00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94a80cf4a0 0x7f94a80b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:18.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f94ada00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a81456a0 0x7f94a813ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:18.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f94ada00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a8157dc0 con 0x7f94a806b3f0 2024-02-19T15:31:18.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f94ada00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a81571b0 con 0x7f94a81456a0 2024-02-19T15:31:18.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f94ada00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a8157490 con 0x7f94a80cf4a0 2024-02-19T15:31:18.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f94a6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94a806b3f0 0x7f94a8156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:18.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f94a6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94a806b3f0 0x7f94a8156960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42640/0 (socket says 172.21.15.47:42640) 2024-02-19T15:31:18.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f94a6a00640 1 -- 172.21.15.47:0/3936396592 learned_addr learned my addr 172.21.15.47:0/3936396592 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:18.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f949e000640 1 --2- 172.21.15.47:0/3936396592 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94a80cf4a0 0x7f94a80b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:18.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f94a7400640 1 --2- 172.21.15.47:0/3936396592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a81456a0 0x7f94a813ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:18.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f94a6a00640 1 -- 172.21.15.47:0/3936396592 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94a80cf4a0 msgr2=0x7f94a80b4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:18.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f94a6a00640 1 --2- 172.21.15.47:0/3936396592 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94a80cf4a0 0x7f94a80b4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:18.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f94a6a00640 1 -- 172.21.15.47:0/3936396592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a81456a0 msgr2=0x7f94a813ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:18.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f94a6a00640 1 --2- 172.21.15.47:0/3936396592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a81456a0 0x7f94a813ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:18.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f94a6a00640 1 -- 172.21.15.47:0/3936396592 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94980094d0 con 0x7f94a806b3f0 2024-02-19T15:31:18.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f949e000640 1 --2- 172.21.15.47:0/3936396592 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94a80cf4a0 0x7f94a80b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:31:18.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f94a6a00640 1 --2- 172.21.15.47:0/3936396592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94a806b3f0 0x7f94a8156960 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f949801abf0 tx=0x7f94980192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:18.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f949f400640 1 -- 172.21.15.47:0/3936396592 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94980b6440 con 0x7f94a806b3f0 2024-02-19T15:31:18.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f949f400640 1 -- 172.21.15.47:0/3936396592 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94980b65a0 con 0x7f94a806b3f0 2024-02-19T15:31:18.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f949f400640 1 -- 172.21.15.47:0/3936396592 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94980b6930 con 0x7f94a806b3f0 2024-02-19T15:31:18.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f94ada00640 1 -- 172.21.15.47:0/3936396592 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a8157770 con 0x7f94a806b3f0 2024-02-19T15:31:18.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f94ada00640 1 -- 172.21.15.47:0/3936396592 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a8156ea0 con 0x7f94a806b3f0 2024-02-19T15:31:18.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.405+0000 7f94ada00640 1 -- 172.21.15.47:0/3936396592 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a8075360 con 0x7f94a806b3f0 2024-02-19T15:31:18.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.409+0000 7f949f400640 1 -- 172.21.15.47:0/3936396592 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f94980b2040 con 0x7f94a806b3f0 2024-02-19T15:31:18.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.409+0000 7f949f400640 1 --2- 172.21.15.47:0/3936396592 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9484066e40 0x7f9484069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:18.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.409+0000 7f949f400640 1 -- 172.21.15.47:0/3936396592 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f949811e090 con 0x7f94a806b3f0 2024-02-19T15:31:18.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.409+0000 7f949e000640 1 --2- 172.21.15.47:0/3936396592 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9484066e40 0x7f9484069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:18.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.409+0000 7f949e000640 1 --2- 172.21.15.47:0/3936396592 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9484066e40 0x7f9484069300 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f94a8152f30 tx=0x7f949000b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:18.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.413+0000 7f949f400640 1 -- 172.21.15.47:0/3936396592 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f94980eada0 con 0x7f94a806b3f0 2024-02-19T15:31:18.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.609+0000 7f94ada00640 1 -- 172.21.15.47:0/3936396592 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94a8072b90 con 0x7f94a806b3f0 2024-02-19T15:31:18.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.609+0000 7f949f400640 1 -- 172.21.15.47:0/3936396592 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f94980bbd80 con 0x7f94a806b3f0 2024-02-19T15:31:18.610 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:18.610 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:18.611 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:31:18.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.613+0000 7f94ada00640 1 -- 172.21.15.47:0/3936396592 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9484066e40 msgr2=0x7f9484069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:18.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.613+0000 7f94ada00640 1 --2- 172.21.15.47:0/3936396592 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9484066e40 0x7f9484069300 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f94a8152f30 tx=0x7f949000b040 comp rx=0 tx=0).stop 2024-02-19T15:31:18.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.613+0000 7f94ada00640 1 -- 172.21.15.47:0/3936396592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94a806b3f0 msgr2=0x7f94a8156960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:18.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.613+0000 7f94ada00640 1 --2- 172.21.15.47:0/3936396592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94a806b3f0 0x7f94a8156960 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f949801abf0 tx=0x7f94980192e0 comp rx=0 tx=0).stop 2024-02-19T15:31:18.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.613+0000 7f94ada00640 1 -- 172.21.15.47:0/3936396592 shutdown_connections 2024-02-19T15:31:18.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.613+0000 7f94ada00640 1 --2- 172.21.15.47:0/3936396592 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9484066e40 0x7f9484069300 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:18.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.613+0000 7f94ada00640 1 --2- 172.21.15.47:0/3936396592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a81456a0 0x7f94a813ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:18.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.613+0000 7f94ada00640 1 --2- 172.21.15.47:0/3936396592 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94a80cf4a0 0x7f94a80b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:18.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.613+0000 7f94ada00640 1 --2- 172.21.15.47:0/3936396592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94a806b3f0 0x7f94a8156960 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:18.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.613+0000 7f94ada00640 1 -- 172.21.15.47:0/3936396592 >> 172.21.15.47:0/3936396592 conn(0x7f94a80d0ad0 msgr2=0x7f94a80cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:18.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.613+0000 7f94ada00640 1 -- 172.21.15.47:0/3936396592 shutdown_connections 2024-02-19T15:31:18.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:18.613+0000 7f94ada00640 1 -- 172.21.15.47:0/3936396592 wait complete. 2024-02-19T15:31:18.833 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:18.833 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:19.834 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:19.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.973+0000 7fc4b9a00640 1 Processor -- start 2024-02-19T15:31:19.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.977+0000 7fc4b9a00640 1 -- start start 2024-02-19T15:31:19.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.977+0000 7fc4b9a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc4b415dc90 0x7fc4b40cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:19.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.977+0000 7fc4b9a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4b406b5c0 0x7fc4b40ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:19.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.977+0000 7fc4b9a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b40ce9e0 con 0x7fc4b415e070 2024-02-19T15:31:19.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.977+0000 7fc4b9a00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4b4072ae0 con 0x7fc4b415dc90 2024-02-19T15:31:19.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.977+0000 7fc4b9a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b4161a10 con 0x7fc4b406b5c0 2024-02-19T15:31:19.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.977+0000 7fc4b3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc4b415dc90 0x7fc4b40cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:19.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.977+0000 7fc4b3e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc4b415dc90 0x7fc4b40cfc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42654/0 (socket says 172.21.15.47:42654) 2024-02-19T15:31:19.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.977+0000 7fc4abe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc4b415e070 0x7fc4b406aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:40892/0 (socket says 172.21.15.47:40892) 2024-02-19T15:31:19.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.977+0000 7fc4b3e00640 1 -- 172.21.15.47:0/1070719535 learned_addr learned my addr 172.21.15.47:0/1070719535 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:19.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.977+0000 7fc4b9000640 1 --2- 172.21.15.47:0/1070719535 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4b406b5c0 0x7fc4b40ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:19.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.977+0000 7fc4b3e00640 1 -- 172.21.15.47:0/1070719535 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4b406b5c0 msgr2=0x7fc4b40ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:19.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.977+0000 7fc4b3e00640 1 --2- 172.21.15.47:0/1070719535 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4b406b5c0 0x7fc4b40ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:19.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.977+0000 7fc4b3e00640 1 -- 172.21.15.47:0/1070719535 >> v1:172.21.15.47:6789/0 conn(0x7fc4b415e070 legacy=0x7fc4b406aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:19.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.977+0000 7fc4b3e00640 1 -- 172.21.15.47:0/1070719535 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b4161cf0 con 0x7fc4b415dc90 2024-02-19T15:31:19.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.981+0000 7fc4b3e00640 1 --2- 172.21.15.47:0/1070719535 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4b415dc90 0x7fc4b40cfc10 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4009870 tx=0x7fc4a401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74aa3781539d0fd9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:19.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.981+0000 7fc4b3400640 1 -- 172.21.15.47:0/1070719535 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a40a8a50 con 0x7fc4b415dc90 2024-02-19T15:31:19.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.981+0000 7fc4b3400640 1 -- 172.21.15.47:0/1070719535 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a40a8bb0 con 0x7fc4b415dc90 2024-02-19T15:31:19.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.981+0000 7fc4b3400640 1 -- 172.21.15.47:0/1070719535 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a40b0440 con 0x7fc4b415dc90 2024-02-19T15:31:19.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.981+0000 7fc4b9a00640 1 -- 172.21.15.47:0/1070719535 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4b415dc90 msgr2=0x7fc4b40cfc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:19.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.981+0000 7fc4b9a00640 1 --2- 172.21.15.47:0/1070719535 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4b415dc90 0x7fc4b40cfc10 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4009870 tx=0x7fc4a401ed50 comp rx=0 tx=0).stop 2024-02-19T15:31:19.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.981+0000 7fc4b9a00640 1 -- 172.21.15.47:0/1070719535 shutdown_connections 2024-02-19T15:31:19.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.981+0000 7fc4b9a00640 1 --2- 172.21.15.47:0/1070719535 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4b406b5c0 0x7fc4b40ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:19.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.981+0000 7fc4b9a00640 1 --2- 172.21.15.47:0/1070719535 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4b415dc90 0x7fc4b40cfc10 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:19.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.981+0000 7fc4b9a00640 1 -- 172.21.15.47:0/1070719535 >> 172.21.15.47:0/1070719535 conn(0x7fc4b40d08d0 msgr2=0x7fc4b40d0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:19.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.981+0000 7fc4b9a00640 1 -- 172.21.15.47:0/1070719535 shutdown_connections 2024-02-19T15:31:19.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.981+0000 7fc4b9a00640 1 -- 172.21.15.47:0/1070719535 wait complete. 2024-02-19T15:31:19.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.981+0000 7fc4b9a00640 1 Processor -- start 2024-02-19T15:31:19.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.981+0000 7fc4b9a00640 1 -- start start 2024-02-19T15:31:19.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b9a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4b406b5c0 0x7fc4b40b4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:19.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b9a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4b415dc90 0x7fc4b41f2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:19.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b9a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4b415e070 0x7fc4b40b5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:19.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b9a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b4149160 con 0x7fc4b406b5c0 2024-02-19T15:31:19.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b9a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b4149ce0 con 0x7fc4b415e070 2024-02-19T15:31:19.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b9a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b41493b0 con 0x7fc4b415dc90 2024-02-19T15:31:19.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4b406b5c0 0x7fc4b40b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:19.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4b406b5c0 0x7fc4b40b4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42656/0 (socket says 172.21.15.47:42656) 2024-02-19T15:31:19.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b3e00640 1 -- 172.21.15.47:0/1482964646 learned_addr learned my addr 172.21.15.47:0/1482964646 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:19.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b9000640 1 --2- 172.21.15.47:0/1482964646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4b415e070 0x7fc4b40b5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:19.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4abe00640 1 --2- 172.21.15.47:0/1482964646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4b415dc90 0x7fc4b41f2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:19.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b3e00640 1 -- 172.21.15.47:0/1482964646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4b415dc90 msgr2=0x7fc4b41f2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:19.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b3e00640 1 --2- 172.21.15.47:0/1482964646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4b415dc90 0x7fc4b41f2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:19.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b3e00640 1 -- 172.21.15.47:0/1482964646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4b415e070 msgr2=0x7fc4b40b5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:19.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b3e00640 1 --2- 172.21.15.47:0/1482964646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4b415e070 0x7fc4b40b5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:19.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b3e00640 1 -- 172.21.15.47:0/1482964646 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a40094d0 con 0x7fc4b406b5c0 2024-02-19T15:31:19.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b3e00640 1 --2- 172.21.15.47:0/1482964646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4b406b5c0 0x7fc4b40b4000 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fc4a401d400 tx=0x7fc4a401baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:19.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b0c00640 1 -- 172.21.15.47:0/1482964646 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a40ba600 con 0x7fc4b406b5c0 2024-02-19T15:31:19.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b0c00640 1 -- 172.21.15.47:0/1482964646 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a40ba780 con 0x7fc4b406b5c0 2024-02-19T15:31:19.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b0c00640 1 -- 172.21.15.47:0/1482964646 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a40bab30 con 0x7fc4b406b5c0 2024-02-19T15:31:19.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b9a00640 1 -- 172.21.15.47:0/1482964646 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b4149690 con 0x7fc4b406b5c0 2024-02-19T15:31:19.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b9a00640 1 -- 172.21.15.47:0/1482964646 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b40b4540 con 0x7fc4b406b5c0 2024-02-19T15:31:19.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.985+0000 7fc4b9a00640 1 -- 172.21.15.47:0/1482964646 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4b4075360 con 0x7fc4b406b5c0 2024-02-19T15:31:19.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.989+0000 7fc4b0c00640 1 -- 172.21.15.47:0/1482964646 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fc4a40bac90 con 0x7fc4b406b5c0 2024-02-19T15:31:19.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.989+0000 7fc4b0c00640 1 --2- 172.21.15.47:0/1482964646 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc490066e90 0x7fc490069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:19.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.989+0000 7fc4b0c00640 1 -- 172.21.15.47:0/1482964646 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc4a4122fe0 con 0x7fc4b406b5c0 2024-02-19T15:31:19.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.989+0000 7fc4abe00640 1 --2- 172.21.15.47:0/1482964646 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc490066e90 0x7fc490069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:19.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.989+0000 7fc4abe00640 1 --2- 172.21.15.47:0/1482964646 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc490066e90 0x7fc490069350 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fc4b414b370 tx=0x7fc49c007cf0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:19.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:19.997+0000 7fc4b0c00640 1 -- 172.21.15.47:0/1482964646 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fc4a40efed0 con 0x7fc4b406b5c0 2024-02-19T15:31:20.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:20.193+0000 7fc4b9a00640 1 -- 172.21.15.47:0/1482964646 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4b40d1560 con 0x7fc4b406b5c0 2024-02-19T15:31:20.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:20.193+0000 7fc4b0c00640 1 -- 172.21.15.47:0/1482964646 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7fc4a40b5d80 con 0x7fc4b406b5c0 2024-02-19T15:31:20.194 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:20.194 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:20.194 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:31:20.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:20.197+0000 7fc4b9a00640 1 -- 172.21.15.47:0/1482964646 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc490066e90 msgr2=0x7fc490069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:20.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:20.197+0000 7fc4b9a00640 1 --2- 172.21.15.47:0/1482964646 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc490066e90 0x7fc490069350 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fc4b414b370 tx=0x7fc49c007cf0 comp rx=0 tx=0).stop 2024-02-19T15:31:20.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:20.197+0000 7fc4b9a00640 1 -- 172.21.15.47:0/1482964646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4b406b5c0 msgr2=0x7fc4b40b4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:20.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:20.197+0000 7fc4b9a00640 1 --2- 172.21.15.47:0/1482964646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4b406b5c0 0x7fc4b40b4000 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fc4a401d400 tx=0x7fc4a401baf0 comp rx=0 tx=0).stop 2024-02-19T15:31:20.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:20.197+0000 7fc4b9a00640 1 -- 172.21.15.47:0/1482964646 shutdown_connections 2024-02-19T15:31:20.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:20.197+0000 7fc4b9a00640 1 --2- 172.21.15.47:0/1482964646 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fc490066e90 0x7fc490069350 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:20.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:20.197+0000 7fc4b9a00640 1 --2- 172.21.15.47:0/1482964646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4b415e070 0x7fc4b40b5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:20.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:20.197+0000 7fc4b9a00640 1 --2- 172.21.15.47:0/1482964646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4b415dc90 0x7fc4b41f2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:20.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:20.197+0000 7fc4b9a00640 1 --2- 172.21.15.47:0/1482964646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4b406b5c0 0x7fc4b40b4000 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:20.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:20.197+0000 7fc4b9a00640 1 -- 172.21.15.47:0/1482964646 >> 172.21.15.47:0/1482964646 conn(0x7fc4b40d08d0 msgr2=0x7fc4b40ca470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:20.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:20.197+0000 7fc4b9a00640 1 -- 172.21.15.47:0/1482964646 shutdown_connections 2024-02-19T15:31:20.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:20.197+0000 7fc4b9a00640 1 -- 172.21.15.47:0/1482964646 wait complete. 2024-02-19T15:31:20.256 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:20.417 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:20.417 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:21.417 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:21.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.557+0000 7f4efe000640 1 Processor -- start 2024-02-19T15:31:21.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.557+0000 7f4efe000640 1 -- start start 2024-02-19T15:31:21.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.561+0000 7f4efe000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4ef81613a0 0x7f4ef80adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:21.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.561+0000 7f4efe000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef8161780 con 0x7f4ef806fdf0 2024-02-19T15:31:21.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.561+0000 7f4efe000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef80b01a0 con 0x7f4ef806e7c0 2024-02-19T15:31:21.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.561+0000 7f4efe000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef80ae3c0 con 0x7f4ef81613a0 2024-02-19T15:31:21.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.561+0000 7f4ef7400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4ef81613a0 0x7f4ef80adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:21.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.561+0000 7f4eefe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f4ef806fdf0 0x7f4ef806e0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:40896/0 (socket says 172.21.15.47:40896) 2024-02-19T15:31:21.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.561+0000 7f4eefe00640 1 -- 172.21.15.47:0/934195073 learned_addr learned my addr 172.21.15.47:0/934195073 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:31:21.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.561+0000 7f4ef7400640 1 -- 172.21.15.47:0/934195073 >> v1:172.21.15.116:6789/0 conn(0x7f4ef806e7c0 legacy=0x7f4ef806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:21.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.561+0000 7f4ef7400640 1 -- 172.21.15.47:0/934195073 >> v1:172.21.15.47:6789/0 conn(0x7f4ef806fdf0 legacy=0x7f4ef806e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:21.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.561+0000 7f4ef7400640 1 -- 172.21.15.47:0/934195073 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef80ae6a0 con 0x7f4ef81613a0 2024-02-19T15:31:21.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.565+0000 7f4ef7400640 1 --2- 172.21.15.47:0/934195073 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ef81613a0 0x7f4ef80adbb0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f4ee80097c0 tx=0x7f4ee801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=639e0712a94e3146 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:21.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.565+0000 7f4ef6a00640 1 -- 172.21.15.47:0/934195073 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee80ab600 con 0x7f4ef81613a0 2024-02-19T15:31:21.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.565+0000 7f4ef6a00640 1 -- 172.21.15.47:0/934195073 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee80a7050 con 0x7f4ef81613a0 2024-02-19T15:31:21.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.565+0000 7f4ef6a00640 1 -- 172.21.15.47:0/934195073 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee80ab980 con 0x7f4ef81613a0 2024-02-19T15:31:21.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.565+0000 7f4efe000640 1 -- 172.21.15.47:0/934195073 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ef81613a0 msgr2=0x7f4ef80adbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:21.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.565+0000 7f4efe000640 1 --2- 172.21.15.47:0/934195073 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ef81613a0 0x7f4ef80adbb0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f4ee80097c0 tx=0x7f4ee801c3d0 comp rx=0 tx=0).stop 2024-02-19T15:31:21.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.565+0000 7f4efe000640 1 -- 172.21.15.47:0/934195073 shutdown_connections 2024-02-19T15:31:21.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.565+0000 7f4efe000640 1 --2- 172.21.15.47:0/934195073 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ef81613a0 0x7f4ef80adbb0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:21.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.565+0000 7f4efe000640 1 -- 172.21.15.47:0/934195073 >> 172.21.15.47:0/934195073 conn(0x7f4ef80d0ad0 msgr2=0x7f4ef80d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:21.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.565+0000 7f4efe000640 1 -- 172.21.15.47:0/934195073 shutdown_connections 2024-02-19T15:31:21.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.565+0000 7f4efe000640 1 -- 172.21.15.47:0/934195073 wait complete. 2024-02-19T15:31:21.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.565+0000 7f4efe000640 1 Processor -- start 2024-02-19T15:31:21.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.565+0000 7f4efe000640 1 -- start start 2024-02-19T15:31:21.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.565+0000 7f4efe000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef806e7c0 0x7f4ef81560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:21.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4efe000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ef806fdf0 0x7f4ef80d04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:21.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4efe000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ef81613a0 0x7f4ef80b0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:21.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4efe000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef814f670 con 0x7f4ef81613a0 2024-02-19T15:31:21.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4efe000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef8154420 con 0x7f4ef806e7c0 2024-02-19T15:31:21.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4efe000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef8156780 con 0x7f4ef806fdf0 2024-02-19T15:31:21.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4ef7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ef81613a0 0x7f4ef80b0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:21.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4ef7e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ef81613a0 0x7f4ef80b0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42670/0 (socket says 172.21.15.47:42670) 2024-02-19T15:31:21.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4ef7e00640 1 -- 172.21.15.47:0/797945367 learned_addr learned my addr 172.21.15.47:0/797945367 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:21.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4ef7400640 1 --2- 172.21.15.47:0/797945367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef806e7c0 0x7f4ef81560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:21.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4eefe00640 1 --2- 172.21.15.47:0/797945367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ef806fdf0 0x7f4ef80d04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:21.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4ef7e00640 1 -- 172.21.15.47:0/797945367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ef806fdf0 msgr2=0x7f4ef80d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:21.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4ef7e00640 1 --2- 172.21.15.47:0/797945367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ef806fdf0 0x7f4ef80d04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:21.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4ef7e00640 1 -- 172.21.15.47:0/797945367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef806e7c0 msgr2=0x7f4ef81560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:21.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4ef7e00640 1 --2- 172.21.15.47:0/797945367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef806e7c0 0x7f4ef81560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:21.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4ef7e00640 1 -- 172.21.15.47:0/797945367 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee8009420 con 0x7f4ef81613a0 2024-02-19T15:31:21.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4ef7e00640 1 --2- 172.21.15.47:0/797945367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ef81613a0 0x7f4ef80b0620 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f4ef000b970 tx=0x7f4ef000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:21.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4eef400640 1 -- 172.21.15.47:0/797945367 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef000cb40 con 0x7f4ef81613a0 2024-02-19T15:31:21.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4eef400640 1 -- 172.21.15.47:0/797945367 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef000cca0 con 0x7f4ef81613a0 2024-02-19T15:31:21.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4efe000640 1 -- 172.21.15.47:0/797945367 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef8156a60 con 0x7f4ef81613a0 2024-02-19T15:31:21.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4eef400640 1 -- 172.21.15.47:0/797945367 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef0014680 con 0x7f4ef81613a0 2024-02-19T15:31:21.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4efe000640 1 -- 172.21.15.47:0/797945367 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef8154950 con 0x7f4ef81613a0 2024-02-19T15:31:21.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4efe000640 1 -- 172.21.15.47:0/797945367 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef8075360 con 0x7f4ef81613a0 2024-02-19T15:31:21.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.569+0000 7f4eef400640 1 -- 172.21.15.47:0/797945367 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f4ef001d3b0 con 0x7f4ef81613a0 2024-02-19T15:31:21.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.573+0000 7f4eef400640 1 --2- 172.21.15.47:0/797945367 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4ed4066e40 0x7f4ed4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:21.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.573+0000 7f4eef400640 1 -- 172.21.15.47:0/797945367 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4ef0084730 con 0x7f4ef81613a0 2024-02-19T15:31:21.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.573+0000 7f4ef7400640 1 --2- 172.21.15.47:0/797945367 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4ed4066e40 0x7f4ed4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:21.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.573+0000 7f4ef7400640 1 --2- 172.21.15.47:0/797945367 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4ed4066e40 0x7f4ed4069300 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f4ee80ad000 tx=0x7f4ee80a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:21.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.577+0000 7f4eef400640 1 -- 172.21.15.47:0/797945367 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f4ef0051620 con 0x7f4ef81613a0 2024-02-19T15:31:21.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.773+0000 7f4efe000640 1 -- 172.21.15.47:0/797945367 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ef80d15f0 con 0x7f4ef81613a0 2024-02-19T15:31:21.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.773+0000 7f4eef400640 1 -- 172.21.15.47:0/797945367 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f4ef0019d80 con 0x7f4ef81613a0 2024-02-19T15:31:21.772 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:21.772 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:21.772 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:31:21.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.773+0000 7f4efe000640 1 -- 172.21.15.47:0/797945367 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4ed4066e40 msgr2=0x7f4ed4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:21.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.773+0000 7f4efe000640 1 --2- 172.21.15.47:0/797945367 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4ed4066e40 0x7f4ed4069300 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f4ee80ad000 tx=0x7f4ee80a6000 comp rx=0 tx=0).stop 2024-02-19T15:31:21.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.773+0000 7f4efe000640 1 -- 172.21.15.47:0/797945367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ef81613a0 msgr2=0x7f4ef80b0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:21.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.773+0000 7f4efe000640 1 --2- 172.21.15.47:0/797945367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ef81613a0 0x7f4ef80b0620 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f4ef000b970 tx=0x7f4ef000be30 comp rx=0 tx=0).stop 2024-02-19T15:31:21.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.773+0000 7f4efe000640 1 -- 172.21.15.47:0/797945367 shutdown_connections 2024-02-19T15:31:21.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.773+0000 7f4efe000640 1 --2- 172.21.15.47:0/797945367 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4ed4066e40 0x7f4ed4069300 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:21.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.773+0000 7f4efe000640 1 --2- 172.21.15.47:0/797945367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ef81613a0 0x7f4ef80b0620 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:21.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.773+0000 7f4efe000640 1 --2- 172.21.15.47:0/797945367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ef806fdf0 0x7f4ef80d04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:21.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.773+0000 7f4efe000640 1 --2- 172.21.15.47:0/797945367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef806e7c0 0x7f4ef81560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:21.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.773+0000 7f4efe000640 1 -- 172.21.15.47:0/797945367 >> 172.21.15.47:0/797945367 conn(0x7f4ef80d0ad0 msgr2=0x7f4ef80b3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:21.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.773+0000 7f4efe000640 1 -- 172.21.15.47:0/797945367 shutdown_connections 2024-02-19T15:31:21.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:21.773+0000 7f4efe000640 1 -- 172.21.15.47:0/797945367 wait complete. 2024-02-19T15:31:22.000 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:22.001 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:22.259 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:23.001 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:23.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.141+0000 7f60a5200640 1 Processor -- start 2024-02-19T15:31:23.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.141+0000 7f60a5200640 1 -- start start 2024-02-19T15:31:23.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.145+0000 7f60a5200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f60a00cedb0 0x7f60a00cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:23.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.145+0000 7f60a5200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a00cc700 con 0x7f60a015bcd0 2024-02-19T15:31:23.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.145+0000 7f60a5200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a00cf300 con 0x7f60a00d2710 2024-02-19T15:31:23.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.145+0000 7f60a5200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60a00cf5e0 con 0x7f60a00cedb0 2024-02-19T15:31:23.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.145+0000 7f6097e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f60a00cedb0 0x7f60a00cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:23.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.145+0000 7f6097e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f60a00cedb0 0x7f60a00cfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42682/0 (socket says 172.21.15.47:42682) 2024-02-19T15:31:23.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.145+0000 7f6097e00640 1 -- 172.21.15.47:0/1036385032 learned_addr learned my addr 172.21.15.47:0/1036385032 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:23.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.145+0000 7f6097e00640 1 -- 172.21.15.47:0/1036385032 >> v1:172.21.15.116:6789/0 conn(0x7f60a00d2710 legacy=0x7f60a015d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:23.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.145+0000 7f6097e00640 1 -- 172.21.15.47:0/1036385032 >> v1:172.21.15.47:6789/0 conn(0x7f60a015bcd0 legacy=0x7f60a00cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:23.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.145+0000 7f6097e00640 1 -- 172.21.15.47:0/1036385032 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a00cc9a0 con 0x7f60a00cedb0 2024-02-19T15:31:23.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.145+0000 7f609d400640 1 -- 172.21.15.47:0/1036385032 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679940646 0 0) 0x7f60a00cc700 con 0x7f60a015bcd0 2024-02-19T15:31:23.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.149+0000 7f6097e00640 1 --2- 172.21.15.47:0/1036385032 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60a00cedb0 0x7f60a00cfc50 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f6088009870 tx=0x7f608801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfdab8f6e1e2ae62 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:23.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.149+0000 7f609d400640 1 -- 172.21.15.47:0/1036385032 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60880ab600 con 0x7f60a00cedb0 2024-02-19T15:31:23.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.149+0000 7f609d400640 1 -- 172.21.15.47:0/1036385032 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60880ab760 con 0x7f60a00cedb0 2024-02-19T15:31:23.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.149+0000 7f609d400640 1 -- 172.21.15.47:0/1036385032 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60880abaf0 con 0x7f60a00cedb0 2024-02-19T15:31:23.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.149+0000 7f60a5200640 1 -- 172.21.15.47:0/1036385032 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60a00cedb0 msgr2=0x7f60a00cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:23.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.149+0000 7f60a5200640 1 --2- 172.21.15.47:0/1036385032 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60a00cedb0 0x7f60a00cfc50 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f6088009870 tx=0x7f608801c4f0 comp rx=0 tx=0).stop 2024-02-19T15:31:23.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.149+0000 7f60a5200640 1 -- 172.21.15.47:0/1036385032 shutdown_connections 2024-02-19T15:31:23.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.149+0000 7f60a5200640 1 --2- 172.21.15.47:0/1036385032 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60a00cedb0 0x7f60a00cfc50 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:23.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.149+0000 7f60a5200640 1 -- 172.21.15.47:0/1036385032 >> 172.21.15.47:0/1036385032 conn(0x7f60a00d1bf0 msgr2=0x7f60a00d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:23.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.149+0000 7f60a5200640 1 -- 172.21.15.47:0/1036385032 shutdown_connections 2024-02-19T15:31:23.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.149+0000 7f60a5200640 1 -- 172.21.15.47:0/1036385032 wait complete. 2024-02-19T15:31:23.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f60a5200640 1 Processor -- start 2024-02-19T15:31:23.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f60a5200640 1 -- start start 2024-02-19T15:31:23.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f60a5200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60a00cedb0 0x7f60a0152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:23.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f60a5200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60a00d2710 0x7f60a00c9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:23.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f60a5200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f60a015bcd0 0x7f60a01528b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:23.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f60a5200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a00cdb90 con 0x7f60a00d2710 2024-02-19T15:31:23.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f60a5200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a0155af0 con 0x7f60a00cedb0 2024-02-19T15:31:23.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f60a5200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a00c9520 con 0x7f60a015bcd0 2024-02-19T15:31:23.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f6097e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60a00d2710 0x7f60a00c9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:23.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f6097e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60a00d2710 0x7f60a00c9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42684/0 (socket says 172.21.15.47:42684) 2024-02-19T15:31:23.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f6097e00640 1 -- 172.21.15.47:0/2672421451 learned_addr learned my addr 172.21.15.47:0/2672421451 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:23.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f609de00640 1 --2- 172.21.15.47:0/2672421451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60a00cedb0 0x7f60a0152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:23.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f609e800640 1 --2- 172.21.15.47:0/2672421451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f60a015bcd0 0x7f60a01528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:23.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f6097e00640 1 -- 172.21.15.47:0/2672421451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f60a015bcd0 msgr2=0x7f60a01528b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:23.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f6097e00640 1 --2- 172.21.15.47:0/2672421451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f60a015bcd0 0x7f60a01528b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:23.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f6097e00640 1 -- 172.21.15.47:0/2672421451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60a00cedb0 msgr2=0x7f60a0152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:23.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f6097e00640 1 --2- 172.21.15.47:0/2672421451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60a00cedb0 0x7f60a0152180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:23.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f6097e00640 1 -- 172.21.15.47:0/2672421451 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60880094d0 con 0x7f60a00d2710 2024-02-19T15:31:23.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f6097e00640 1 --2- 172.21.15.47:0/2672421451 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60a00d2710 0x7f60a00c9940 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f60880ad040 tx=0x7f608801ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:23.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f6096000640 1 -- 172.21.15.47:0/2672421451 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60880b4800 con 0x7f60a00d2710 2024-02-19T15:31:23.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f6096000640 1 -- 172.21.15.47:0/2672421451 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60880b4980 con 0x7f60a00d2710 2024-02-19T15:31:23.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f6096000640 1 -- 172.21.15.47:0/2672421451 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60880b4d30 con 0x7f60a00d2710 2024-02-19T15:31:23.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f60a5200640 1 -- 172.21.15.47:0/2672421451 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60a015d240 con 0x7f60a00d2710 2024-02-19T15:31:23.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.153+0000 7f60a5200640 1 -- 172.21.15.47:0/2672421451 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a0153570 con 0x7f60a00d2710 2024-02-19T15:31:23.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.157+0000 7f60a5200640 1 -- 172.21.15.47:0/2672421451 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60a00c9660 con 0x7f60a00d2710 2024-02-19T15:31:23.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.157+0000 7f6096000640 1 -- 172.21.15.47:0/2672421451 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f60880b3380 con 0x7f60a00d2710 2024-02-19T15:31:23.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.157+0000 7f6096000640 1 --2- 172.21.15.47:0/2672421451 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f607c066e70 0x7f607c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:23.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.157+0000 7f6096000640 1 -- 172.21.15.47:0/2672421451 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f608811efe0 con 0x7f60a00d2710 2024-02-19T15:31:23.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.157+0000 7f609de00640 1 --2- 172.21.15.47:0/2672421451 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f607c066e70 0x7f607c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:23.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.161+0000 7f609de00640 1 --2- 172.21.15.47:0/2672421451 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f607c066e70 0x7f607c069330 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f6090000f70 tx=0x7f6090009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:23.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.161+0000 7f6096000640 1 -- 172.21.15.47:0/2672421451 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f60a00c9660 con 0x7f60a00d2710 2024-02-19T15:31:23.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.357+0000 7f60a5200640 1 -- 172.21.15.47:0/2672421451 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60a00c9660 con 0x7f60a00d2710 2024-02-19T15:31:23.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.357+0000 7f6096000640 1 -- 172.21.15.47:0/2672421451 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f60a00c9660 con 0x7f60a00d2710 2024-02-19T15:31:23.358 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:23.358 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:23.358 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:31:23.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.361+0000 7f60a5200640 1 -- 172.21.15.47:0/2672421451 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f607c066e70 msgr2=0x7f607c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:23.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.361+0000 7f60a5200640 1 --2- 172.21.15.47:0/2672421451 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f607c066e70 0x7f607c069330 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f6090000f70 tx=0x7f6090009040 comp rx=0 tx=0).stop 2024-02-19T15:31:23.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.361+0000 7f60a5200640 1 -- 172.21.15.47:0/2672421451 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60a00d2710 msgr2=0x7f60a00c9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:23.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.361+0000 7f60a5200640 1 --2- 172.21.15.47:0/2672421451 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60a00d2710 0x7f60a00c9940 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f60880ad040 tx=0x7f608801ad30 comp rx=0 tx=0).stop 2024-02-19T15:31:23.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.361+0000 7f60a5200640 1 -- 172.21.15.47:0/2672421451 shutdown_connections 2024-02-19T15:31:23.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.361+0000 7f60a5200640 1 --2- 172.21.15.47:0/2672421451 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f607c066e70 0x7f607c069330 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:23.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.361+0000 7f60a5200640 1 --2- 172.21.15.47:0/2672421451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f60a015bcd0 0x7f60a01528b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:23.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.361+0000 7f60a5200640 1 --2- 172.21.15.47:0/2672421451 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60a00d2710 0x7f60a00c9940 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:23.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.361+0000 7f60a5200640 1 --2- 172.21.15.47:0/2672421451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60a00cedb0 0x7f60a0152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:23.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.361+0000 7f60a5200640 1 -- 172.21.15.47:0/2672421451 >> 172.21.15.47:0/2672421451 conn(0x7f60a00d1bf0 msgr2=0x7f60a00d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:23.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.361+0000 7f60a5200640 1 -- 172.21.15.47:0/2672421451 shutdown_connections 2024-02-19T15:31:23.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:23.361+0000 7f60a5200640 1 -- 172.21.15.47:0/2672421451 wait complete. 2024-02-19T15:31:23.581 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:23.581 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:24.260 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:24.583 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:24.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.725+0000 7f611ea00640 1 Processor -- start 2024-02-19T15:31:24.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.725+0000 7f611ea00640 1 -- start start 2024-02-19T15:31:24.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.725+0000 7f611ea00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f611815ba70 0x7f611806dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:24.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.725+0000 7f611ea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61180727d0 0x7f611815be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:24.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.725+0000 7f611ea00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f611806e6a0 con 0x7f61180b1ee0 2024-02-19T15:31:24.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.725+0000 7f611ea00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f611806e980 con 0x7f611815ba70 2024-02-19T15:31:24.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.725+0000 7f611ea00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61180b1400 con 0x7f61180727d0 2024-02-19T15:31:24.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.725+0000 7f611d600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f611815ba70 0x7f611806dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:24.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.725+0000 7f611cc00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f61180b1ee0 0x7f61180720c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45814/0 (socket says 172.21.15.47:45814) 2024-02-19T15:31:24.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.725+0000 7f611cc00640 1 -- 172.21.15.47:0/1549339764 learned_addr learned my addr 172.21.15.47:0/1549339764 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:31:24.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.725+0000 7f611e000640 1 --2- 172.21.15.47:0/1549339764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61180727d0 0x7f611815be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:24.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.725+0000 7f611d600640 1 -- 172.21.15.47:0/1549339764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61180727d0 msgr2=0x7f611815be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:24.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.725+0000 7f611d600640 1 --2- 172.21.15.47:0/1549339764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61180727d0 0x7f611815be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:24.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.725+0000 7f611d600640 1 -- 172.21.15.47:0/1549339764 >> v1:172.21.15.47:6789/0 conn(0x7f61180b1ee0 legacy=0x7f61180720c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:24.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.725+0000 7f611d600640 1 -- 172.21.15.47:0/1549339764 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61180b16e0 con 0x7f611815ba70 2024-02-19T15:31:24.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.725+0000 7f6117e00640 1 -- 172.21.15.47:0/1549339764 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457182747 0 0) 0x7f611806e6a0 con 0x7f61180b1ee0 2024-02-19T15:31:24.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.725+0000 7f611e000640 1 --2- 172.21.15.47:0/1549339764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61180727d0 0x7f611815be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:31:24.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.729+0000 7f611d600640 1 --2- 172.21.15.47:0/1549339764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f611815ba70 0x7f611806dd90 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f610c0097c0 tx=0x7f610c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd4941df340e81e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:24.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.729+0000 7f6117e00640 1 -- 172.21.15.47:0/1549339764 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c0ab600 con 0x7f611815ba70 2024-02-19T15:31:24.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.729+0000 7f6117e00640 1 -- 172.21.15.47:0/1549339764 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f610c0ab760 con 0x7f611815ba70 2024-02-19T15:31:24.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.729+0000 7f6117e00640 1 -- 172.21.15.47:0/1549339764 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c0abaf0 con 0x7f611815ba70 2024-02-19T15:31:24.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.729+0000 7f611ea00640 1 -- 172.21.15.47:0/1549339764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f611815ba70 msgr2=0x7f611806dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:24.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.729+0000 7f611ea00640 1 --2- 172.21.15.47:0/1549339764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f611815ba70 0x7f611806dd90 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f610c0097c0 tx=0x7f610c01c3e0 comp rx=0 tx=0).stop 2024-02-19T15:31:24.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.729+0000 7f611ea00640 1 -- 172.21.15.47:0/1549339764 shutdown_connections 2024-02-19T15:31:24.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.729+0000 7f611ea00640 1 --2- 172.21.15.47:0/1549339764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61180727d0 0x7f611815be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:24.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.729+0000 7f611ea00640 1 --2- 172.21.15.47:0/1549339764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f611815ba70 0x7f611806dd90 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:24.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.729+0000 7f611ea00640 1 -- 172.21.15.47:0/1549339764 >> 172.21.15.47:0/1549339764 conn(0x7f61180cf7c0 msgr2=0x7f61180cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:24.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.729+0000 7f611ea00640 1 -- 172.21.15.47:0/1549339764 shutdown_connections 2024-02-19T15:31:24.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.729+0000 7f611ea00640 1 -- 172.21.15.47:0/1549339764 wait complete. 2024-02-19T15:31:24.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611ea00640 1 Processor -- start 2024-02-19T15:31:24.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611ea00640 1 -- start start 2024-02-19T15:31:24.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61180727d0 0x7f61180b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:24.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611ea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61180b1ee0 0x7f611814abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:24.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f611815ba70 0x7f61180c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:24.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611ea00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61180b3630 con 0x7f61180727d0 2024-02-19T15:31:24.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611ea00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61180b2a20 con 0x7f611815ba70 2024-02-19T15:31:24.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611ea00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61180b2d00 con 0x7f61180b1ee0 2024-02-19T15:31:24.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611d600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61180727d0 0x7f61180b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:24.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611d600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61180727d0 0x7f61180b3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36498/0 (socket says 172.21.15.47:36498) 2024-02-19T15:31:24.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611d600640 1 -- 172.21.15.47:0/2218704546 learned_addr learned my addr 172.21.15.47:0/2218704546 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:24.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611cc00640 1 --2- 172.21.15.47:0/2218704546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61180b1ee0 0x7f611814abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:24.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611d600640 1 -- 172.21.15.47:0/2218704546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61180b1ee0 msgr2=0x7f611814abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:24.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611d600640 1 --2- 172.21.15.47:0/2218704546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61180b1ee0 0x7f611814abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:24.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611e000640 1 --2- 172.21.15.47:0/2218704546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f611815ba70 0x7f61180c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:24.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611d600640 1 -- 172.21.15.47:0/2218704546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f611815ba70 msgr2=0x7f61180c8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:24.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611d600640 1 --2- 172.21.15.47:0/2218704546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f611815ba70 0x7f61180c8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:24.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611d600640 1 -- 172.21.15.47:0/2218704546 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610c009420 con 0x7f61180727d0 2024-02-19T15:31:24.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611e000640 1 --2- 172.21.15.47:0/2218704546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f611815ba70 0x7f61180c8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:31:24.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611d600640 1 --2- 172.21.15.47:0/2218704546 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61180727d0 0x7f61180b3e10 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f610c0ad040 tx=0x7f610c01ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:24.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f6115600640 1 -- 172.21.15.47:0/2218704546 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c0b5810 con 0x7f61180727d0 2024-02-19T15:31:24.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f6115600640 1 -- 172.21.15.47:0/2218704546 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f610c0b5990 con 0x7f61180727d0 2024-02-19T15:31:24.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f6115600640 1 -- 172.21.15.47:0/2218704546 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c0b5d40 con 0x7f61180727d0 2024-02-19T15:31:24.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.733+0000 7f611ea00640 1 -- 172.21.15.47:0/2218704546 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61180b2fe0 con 0x7f61180727d0 2024-02-19T15:31:24.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.737+0000 7f611ea00640 1 -- 172.21.15.47:0/2218704546 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61180b4350 con 0x7f61180727d0 2024-02-19T15:31:24.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.737+0000 7f611ea00640 1 -- 172.21.15.47:0/2218704546 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61180b2e40 con 0x7f61180727d0 2024-02-19T15:31:24.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.737+0000 7f6115600640 1 -- 172.21.15.47:0/2218704546 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f610c0c93f0 con 0x7f61180727d0 2024-02-19T15:31:24.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.737+0000 7f6115600640 1 --2- 172.21.15.47:0/2218704546 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f60f8066e70 0x7f60f8069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:24.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.737+0000 7f6115600640 1 -- 172.21.15.47:0/2218704546 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f610c11f6a0 con 0x7f61180727d0 2024-02-19T15:31:24.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.737+0000 7f611cc00640 1 --2- 172.21.15.47:0/2218704546 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f60f8066e70 0x7f60f8069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:24.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.741+0000 7f611cc00640 1 --2- 172.21.15.47:0/2218704546 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f60f8066e70 0x7f60f8069330 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f61180aec80 tx=0x7f6104007b60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:24.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.741+0000 7f6115600640 1 -- 172.21.15.47:0/2218704546 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f61180b2e40 con 0x7f61180727d0 2024-02-19T15:31:24.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.941+0000 7f611ea00640 1 -- 172.21.15.47:0/2218704546 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61180b2e40 con 0x7f61180727d0 2024-02-19T15:31:24.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.941+0000 7f6115600640 1 -- 172.21.15.47:0/2218704546 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3427 (secure 0 0 0) 0x7f610c0c4d80 con 0x7f61180727d0 2024-02-19T15:31:24.942 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:24.943 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:29:54.888177+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:24.943 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 99 2024-02-19T15:31:24.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.945+0000 7f611ea00640 1 -- 172.21.15.47:0/2218704546 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f60f8066e70 msgr2=0x7f60f8069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:24.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.945+0000 7f611ea00640 1 --2- 172.21.15.47:0/2218704546 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f60f8066e70 0x7f60f8069330 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f61180aec80 tx=0x7f6104007b60 comp rx=0 tx=0).stop 2024-02-19T15:31:24.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.945+0000 7f611ea00640 1 -- 172.21.15.47:0/2218704546 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61180727d0 msgr2=0x7f61180b3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:24.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.945+0000 7f611ea00640 1 --2- 172.21.15.47:0/2218704546 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61180727d0 0x7f61180b3e10 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f610c0ad040 tx=0x7f610c01ac20 comp rx=0 tx=0).stop 2024-02-19T15:31:24.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.945+0000 7f611ea00640 1 -- 172.21.15.47:0/2218704546 shutdown_connections 2024-02-19T15:31:24.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.945+0000 7f611ea00640 1 --2- 172.21.15.47:0/2218704546 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f60f8066e70 0x7f60f8069330 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:24.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.945+0000 7f611ea00640 1 --2- 172.21.15.47:0/2218704546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f611815ba70 0x7f61180c8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:24.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.945+0000 7f611ea00640 1 --2- 172.21.15.47:0/2218704546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61180b1ee0 0x7f611814abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:24.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.945+0000 7f611ea00640 1 --2- 172.21.15.47:0/2218704546 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61180727d0 0x7f61180b3e10 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:24.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.945+0000 7f611ea00640 1 -- 172.21.15.47:0/2218704546 >> 172.21.15.47:0/2218704546 conn(0x7f61180cf7c0 msgr2=0x7f611815e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:24.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.945+0000 7f611ea00640 1 -- 172.21.15.47:0/2218704546 shutdown_connections 2024-02-19T15:31:24.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:24.945+0000 7f611ea00640 1 -- 172.21.15.47:0/2218704546 wait complete. 2024-02-19T15:31:25.166 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:29:54.888177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:25.166 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:26.167 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:26.259 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:26.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f705fe00640 1 Processor -- start 2024-02-19T15:31:26.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f705fe00640 1 -- start start 2024-02-19T15:31:26.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f705fe00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f705815d6e0 0x7f7058162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:26.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f705fe00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70580718c0 con 0x7f7058071350 2024-02-19T15:31:26.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f705fe00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7058071ba0 con 0x7f705815b1a0 2024-02-19T15:31:26.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f705fe00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071e80 con 0x7f705815d6e0 2024-02-19T15:31:26.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f705de00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f7058071350 0x7f7058164c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45830/0 (socket says 172.21.15.47:45830) 2024-02-19T15:31:26.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f705de00640 1 -- 172.21.15.47:0/1011679578 learned_addr learned my addr 172.21.15.47:0/1011679578 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:31:26.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f705ca00640 1 --2- 172.21.15.47:0/1011679578 >> v2:172.21.15.116:3300/0 conn(0x7f705815d6e0 0x7f7058162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:26.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f7057e00640 1 -- 172.21.15.47:0/1011679578 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807543707 0 0) 0x7f70580718c0 con 0x7f7058071350 2024-02-19T15:31:26.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f7057e00640 1 -- 172.21.15.47:0/1011679578 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7040003530 con 0x7f7058071350 2024-02-19T15:31:26.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f7057e00640 1 -- 172.21.15.47:0/1011679578 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2309901613 0 0) 0x7f7040003530 con 0x7f7058071350 2024-02-19T15:31:26.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f7057e00640 1 -- 172.21.15.47:0/1011679578 >> v2:172.21.15.116:3300/0 conn(0x7f705815d6e0 msgr2=0x7f7058162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:26.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f7057e00640 1 --2- 172.21.15.47:0/1011679578 >> v2:172.21.15.116:3300/0 conn(0x7f705815d6e0 0x7f7058162740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:26.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f7057e00640 1 -- 172.21.15.47:0/1011679578 >> v1:172.21.15.116:6789/0 conn(0x7f705815b1a0 legacy=0x7f705815b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:26.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f7057e00640 1 -- 172.21.15.47:0/1011679578 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7058072160 con 0x7f7058071350 2024-02-19T15:31:26.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f7057e00640 1 -- 172.21.15.47:0/1011679578 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f7050001a00 con 0x7f7058071350 2024-02-19T15:31:26.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f705ca00640 1 --2- 172.21.15.47:0/1011679578 >> v2:172.21.15.116:3300/0 conn(0x7f705815d6e0 0x7f7058162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:31:26.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f7057e00640 1 -- 172.21.15.47:0/1011679578 >> v1:172.21.15.47:6789/0 conn(0x7f7058071350 legacy=0x7f7058164c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:26.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f7057e00640 1 --2- 172.21.15.47:0/1011679578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70400039b0 0x7f7040003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:26.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f7057e00640 1 --2- 172.21.15.47:0/1011679578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70400047d0 0x7f7040004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:26.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f7057e00640 1 --2- 172.21.15.47:0/1011679578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7040005190 0x7f704000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:26.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f7057e00640 1 -- 172.21.15.47:0/1011679578 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7058071e80 con 0x7f7040005190 2024-02-19T15:31:26.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f7057e00640 1 -- 172.21.15.47:0/1011679578 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704000bfe0 con 0x7f70400039b0 2024-02-19T15:31:26.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.309+0000 7f7057e00640 1 -- 172.21.15.47:0/1011679578 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704000c1a0 con 0x7f70400047d0 2024-02-19T15:31:26.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.313+0000 7f705d400640 1 --2- 172.21.15.47:0/1011679578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70400039b0 0x7f7040003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:26.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.313+0000 7f705ca00640 1 --2- 172.21.15.47:0/1011679578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70400047d0 0x7f7040004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:26.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.313+0000 7f705de00640 1 --2- 172.21.15.47:0/1011679578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7040005190 0x7f704000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:26.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.313+0000 7f705d400640 1 -- 172.21.15.47:0/1011679578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70400047d0 msgr2=0x7f7040004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:26.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.313+0000 7f705d400640 1 --2- 172.21.15.47:0/1011679578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70400047d0 0x7f7040004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:26.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.313+0000 7f705d400640 1 -- 172.21.15.47:0/1011679578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7040005190 msgr2=0x7f704000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:26.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.313+0000 7f705d400640 1 --2- 172.21.15.47:0/1011679578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7040005190 0x7f704000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:26.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.313+0000 7f705d400640 1 -- 172.21.15.47:0/1011679578 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7040003530 con 0x7f70400039b0 2024-02-19T15:31:26.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.313+0000 7f705de00640 1 -- 172.21.15.47:0/1011679578 reap_dead start 2024-02-19T15:31:26.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.313+0000 7f705d400640 1 --2- 172.21.15.47:0/1011679578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70400039b0 0x7f7040003d90 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f704c001860 tx=0x7f704c020910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1db387107402dc63 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:26.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.313+0000 7f7057e00640 1 -- 172.21.15.47:0/1011679578 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c0aa710 con 0x7f70400039b0 2024-02-19T15:31:26.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f7057e00640 1 -- 172.21.15.47:0/1011679578 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704c0aa870 con 0x7f70400039b0 2024-02-19T15:31:26.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f7057e00640 1 -- 172.21.15.47:0/1011679578 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c0aac60 con 0x7f70400039b0 2024-02-19T15:31:26.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f705fe00640 1 -- 172.21.15.47:0/1011679578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70400039b0 msgr2=0x7f7040003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:26.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f705fe00640 1 --2- 172.21.15.47:0/1011679578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70400039b0 0x7f7040003d90 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f704c001860 tx=0x7f704c020910 comp rx=0 tx=0).stop 2024-02-19T15:31:26.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f705fe00640 1 -- 172.21.15.47:0/1011679578 shutdown_connections 2024-02-19T15:31:26.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f705fe00640 1 --2- 172.21.15.47:0/1011679578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70400039b0 0x7f7040003d90 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:26.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f705fe00640 1 -- 172.21.15.47:0/1011679578 >> 172.21.15.47:0/1011679578 conn(0x7f70580b16a0 msgr2=0x7f70580b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:26.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f705fe00640 1 -- 172.21.15.47:0/1011679578 shutdown_connections 2024-02-19T15:31:26.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f705fe00640 1 -- 172.21.15.47:0/1011679578 wait complete. 2024-02-19T15:31:26.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f705fe00640 1 Processor -- start 2024-02-19T15:31:26.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f705fe00640 1 -- start start 2024-02-19T15:31:26.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f705fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70400039b0 0x7f7058153be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f705fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70581f03b0 0x7f70580ced40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f705fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70581f0830 0x7f70580cf280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f705fe00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7058154120 con 0x7f70581f03b0 2024-02-19T15:31:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f705fe00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70581562e0 con 0x7f70400039b0 2024-02-19T15:31:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f705fe00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70581565c0 con 0x7f70581f0830 2024-02-19T15:31:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f705ca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70581f03b0 0x7f70580ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f705ca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70581f03b0 0x7f70580ced40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36516/0 (socket says 172.21.15.47:36516) 2024-02-19T15:31:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f705d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70400039b0 0x7f7058153be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f705ca00640 1 -- 172.21.15.47:0/485351243 learned_addr learned my addr 172.21.15.47:0/485351243 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.317+0000 7f705de00640 1 --2- 172.21.15.47:0/485351243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70581f0830 0x7f70580cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.321+0000 7f705ca00640 1 -- 172.21.15.47:0/485351243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70581f0830 msgr2=0x7f70580cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.321+0000 7f705ca00640 1 --2- 172.21.15.47:0/485351243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70581f0830 0x7f70580cf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.321+0000 7f705ca00640 1 -- 172.21.15.47:0/485351243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70400039b0 msgr2=0x7f7058153be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.321+0000 7f705ca00640 1 --2- 172.21.15.47:0/485351243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70400039b0 0x7f7058153be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:26.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.321+0000 7f705ca00640 1 -- 172.21.15.47:0/485351243 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70581568a0 con 0x7f70581f03b0 2024-02-19T15:31:26.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.321+0000 7f705ca00640 1 --2- 172.21.15.47:0/485351243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70581f03b0 0x7f70580ced40 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f7044011ca0 tx=0x7f704400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:26.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.321+0000 7f7055600640 1 -- 172.21.15.47:0/485351243 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7044011cd0 con 0x7f70581f03b0 2024-02-19T15:31:26.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.321+0000 7f7055600640 1 -- 172.21.15.47:0/485351243 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7044011e30 con 0x7f70581f03b0 2024-02-19T15:31:26.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.321+0000 7f7055600640 1 -- 172.21.15.47:0/485351243 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7044014640 con 0x7f70581f03b0 2024-02-19T15:31:26.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.321+0000 7f705fe00640 1 -- 172.21.15.47:0/485351243 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7058156b80 con 0x7f70581f03b0 2024-02-19T15:31:26.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.321+0000 7f705fe00640 1 -- 172.21.15.47:0/485351243 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7058157140 con 0x7f70581f03b0 2024-02-19T15:31:26.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.321+0000 7f705fe00640 1 -- 172.21.15.47:0/485351243 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70581572d0 con 0x7f70581f03b0 2024-02-19T15:31:26.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.321+0000 7f7055600640 1 -- 172.21.15.47:0/485351243 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f70440147a0 con 0x7f70581f03b0 2024-02-19T15:31:26.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.321+0000 7f7055600640 1 --2- 172.21.15.47:0/485351243 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7038066e70 0x7f7038069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:26.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.321+0000 7f7055600640 1 -- 172.21.15.47:0/485351243 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7044085270 con 0x7f70581f03b0 2024-02-19T15:31:26.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.325+0000 7f705d400640 1 --2- 172.21.15.47:0/485351243 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7038066e70 0x7f7038069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:26.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.325+0000 7f705d400640 1 --2- 172.21.15.47:0/485351243 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7038066e70 0x7f7038069330 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f704c0b3000 tx=0x7f704c0ab000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:26.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.325+0000 7f7055600640 1 -- 172.21.15.47:0/485351243 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7044052010 con 0x7f70581f03b0 2024-02-19T15:31:26.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.521+0000 7f705fe00640 1 -- 172.21.15.47:0/485351243 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7058075360 con 0x7f70581f03b0 2024-02-19T15:31:26.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.521+0000 7f7055600640 1 -- 172.21.15.47:0/485351243 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f7044022d80 con 0x7f70581f03b0 2024-02-19T15:31:26.522 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:26.522 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:26.522 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:26.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.525+0000 7f705fe00640 1 -- 172.21.15.47:0/485351243 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7038066e70 msgr2=0x7f7038069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:26.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.525+0000 7f705fe00640 1 --2- 172.21.15.47:0/485351243 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7038066e70 0x7f7038069330 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f704c0b3000 tx=0x7f704c0ab000 comp rx=0 tx=0).stop 2024-02-19T15:31:26.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.525+0000 7f705fe00640 1 -- 172.21.15.47:0/485351243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70581f03b0 msgr2=0x7f70580ced40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:26.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.525+0000 7f705fe00640 1 --2- 172.21.15.47:0/485351243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70581f03b0 0x7f70580ced40 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f7044011ca0 tx=0x7f704400f610 comp rx=0 tx=0).stop 2024-02-19T15:31:26.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.525+0000 7f705fe00640 1 -- 172.21.15.47:0/485351243 shutdown_connections 2024-02-19T15:31:26.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.525+0000 7f705fe00640 1 --2- 172.21.15.47:0/485351243 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7038066e70 0x7f7038069330 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:26.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.525+0000 7f705fe00640 1 --2- 172.21.15.47:0/485351243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70581f0830 0x7f70580cf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:26.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.525+0000 7f705fe00640 1 --2- 172.21.15.47:0/485351243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70581f03b0 0x7f70580ced40 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:26.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.525+0000 7f705fe00640 1 --2- 172.21.15.47:0/485351243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70400039b0 0x7f7058153be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:26.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.525+0000 7f705fe00640 1 -- 172.21.15.47:0/485351243 >> 172.21.15.47:0/485351243 conn(0x7f70580b16a0 msgr2=0x7f7058154960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:26.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.525+0000 7f705fe00640 1 -- 172.21.15.47:0/485351243 shutdown_connections 2024-02-19T15:31:26.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:26.525+0000 7f705fe00640 1 -- 172.21.15.47:0/485351243 wait complete. 2024-02-19T15:31:26.745 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:26.745 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:27.746 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:27.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.885+0000 7f12f7c00640 1 Processor -- start 2024-02-19T15:31:27.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.885+0000 7f12f7c00640 1 -- start start 2024-02-19T15:31:27.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.889+0000 7f12f7c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f00ceb70 0x7f12f00d0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:27.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.889+0000 7f12f7c00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f00cdae0 con 0x7f12f015be90 2024-02-19T15:31:27.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.889+0000 7f12f7c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f00d0af0 con 0x7f12f00d26b0 2024-02-19T15:31:27.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.889+0000 7f12f7c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f00d0810 con 0x7f12f00ceb70 2024-02-19T15:31:27.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.889+0000 7f12f5200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f12f015be90 0x7f12f00ce460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45844/0 (socket says 172.21.15.47:45844) 2024-02-19T15:31:27.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.889+0000 7f12f5200640 1 -- 172.21.15.47:0/231095283 learned_addr learned my addr 172.21.15.47:0/231095283 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:31:27.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.889+0000 7f12efe00640 1 --2- 172.21.15.47:0/231095283 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f00ceb70 0x7f12f00d0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:27.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.889+0000 7f12ef400640 1 -- 172.21.15.47:0/231095283 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213355123 0 0) 0x7f12f00cdae0 con 0x7f12f015be90 2024-02-19T15:31:27.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.889+0000 7f12ef400640 1 -- 172.21.15.47:0/231095283 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12d8003530 con 0x7f12f015be90 2024-02-19T15:31:27.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.889+0000 7f12efe00640 1 -- 172.21.15.47:0/231095283 >> v1:172.21.15.116:6789/0 conn(0x7f12f00d26b0 legacy=0x7f12f00b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:27.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.889+0000 7f12ef400640 1 -- 172.21.15.47:0/231095283 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801193655 0 0) 0x7f12f00d0af0 con 0x7f12f00d26b0 2024-02-19T15:31:27.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.889+0000 7f12efe00640 1 -- 172.21.15.47:0/231095283 >> v1:172.21.15.47:6789/0 conn(0x7f12f015be90 legacy=0x7f12f00ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:27.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.889+0000 7f12efe00640 1 -- 172.21.15.47:0/231095283 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f00b4a80 con 0x7f12f00ceb70 2024-02-19T15:31:27.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.893+0000 7f12efe00640 1 --2- 172.21.15.47:0/231095283 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f00ceb70 0x7f12f00d0170 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f12dc0097f0 tx=0x7f12dc01c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac2906a442449530 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:27.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.893+0000 7f12ef400640 1 -- 172.21.15.47:0/231095283 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc0aa610 con 0x7f12f00ceb70 2024-02-19T15:31:27.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.893+0000 7f12ef400640 1 -- 172.21.15.47:0/231095283 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12dc0a6040 con 0x7f12f00ceb70 2024-02-19T15:31:27.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.893+0000 7f12ef400640 1 -- 172.21.15.47:0/231095283 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc0aa990 con 0x7f12f00ceb70 2024-02-19T15:31:27.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.893+0000 7f12f7c00640 1 -- 172.21.15.47:0/231095283 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f00ceb70 msgr2=0x7f12f00d0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:27.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.893+0000 7f12f7c00640 1 --2- 172.21.15.47:0/231095283 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f00ceb70 0x7f12f00d0170 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f12dc0097f0 tx=0x7f12dc01c4b0 comp rx=0 tx=0).stop 2024-02-19T15:31:27.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.893+0000 7f12f7c00640 1 -- 172.21.15.47:0/231095283 shutdown_connections 2024-02-19T15:31:27.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.893+0000 7f12f7c00640 1 --2- 172.21.15.47:0/231095283 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f00ceb70 0x7f12f00d0170 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:27.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.893+0000 7f12f7c00640 1 -- 172.21.15.47:0/231095283 >> 172.21.15.47:0/231095283 conn(0x7f12f00d1b90 msgr2=0x7f12f00d1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:27.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.893+0000 7f12f7c00640 1 -- 172.21.15.47:0/231095283 shutdown_connections 2024-02-19T15:31:27.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.893+0000 7f12f7c00640 1 -- 172.21.15.47:0/231095283 wait complete. 2024-02-19T15:31:27.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.893+0000 7f12f7c00640 1 Processor -- start 2024-02-19T15:31:27.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.893+0000 7f12f7c00640 1 -- start start 2024-02-19T15:31:27.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.893+0000 7f12f7c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f00ceb70 0x7f12f015ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:27.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.893+0000 7f12f7c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12f00d26b0 0x7f12f00b6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:27.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.893+0000 7f12f7c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f015be90 0x7f12f0140730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:27.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.893+0000 7f12f7c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f00c6fd0 con 0x7f12f00d26b0 2024-02-19T15:31:27.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.893+0000 7f12f7c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f00c72b0 con 0x7f12f00ceb70 2024-02-19T15:31:27.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.893+0000 7f12f7c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f00c7590 con 0x7f12f015be90 2024-02-19T15:31:27.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.897+0000 7f12efe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12f00d26b0 0x7f12f00b6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:27.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.897+0000 7f12efe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12f00d26b0 0x7f12f00b6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36530/0 (socket says 172.21.15.47:36530) 2024-02-19T15:31:27.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.897+0000 7f12efe00640 1 -- 172.21.15.47:0/457229696 learned_addr learned my addr 172.21.15.47:0/457229696 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:27.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.897+0000 7f12f5200640 1 --2- 172.21.15.47:0/457229696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f00ceb70 0x7f12f015ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:27.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.897+0000 7f12f5c00640 1 --2- 172.21.15.47:0/457229696 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f015be90 0x7f12f0140730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:27.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.897+0000 7f12efe00640 1 -- 172.21.15.47:0/457229696 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f015be90 msgr2=0x7f12f0140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:27.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.897+0000 7f12efe00640 1 --2- 172.21.15.47:0/457229696 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f015be90 0x7f12f0140730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:27.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.897+0000 7f12efe00640 1 -- 172.21.15.47:0/457229696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f00ceb70 msgr2=0x7f12f015ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:27.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.897+0000 7f12efe00640 1 --2- 172.21.15.47:0/457229696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f00ceb70 0x7f12f015ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:27.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.897+0000 7f12efe00640 1 -- 172.21.15.47:0/457229696 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12dc009420 con 0x7f12f00d26b0 2024-02-19T15:31:27.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.897+0000 7f12efe00640 1 --2- 172.21.15.47:0/457229696 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12f00d26b0 0x7f12f00b6c50 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f12dc0ac040 tx=0x7f12dc01acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:27.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.897+0000 7f12ecc00640 1 -- 172.21.15.47:0/457229696 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc0b4400 con 0x7f12f00d26b0 2024-02-19T15:31:27.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.897+0000 7f12ecc00640 1 -- 172.21.15.47:0/457229696 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12dc0b4560 con 0x7f12f00d26b0 2024-02-19T15:31:27.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.897+0000 7f12f7c00640 1 -- 172.21.15.47:0/457229696 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f00c7870 con 0x7f12f00d26b0 2024-02-19T15:31:27.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.897+0000 7f12ecc00640 1 -- 172.21.15.47:0/457229696 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc0b4910 con 0x7f12f00d26b0 2024-02-19T15:31:27.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.897+0000 7f12f7c00640 1 -- 172.21.15.47:0/457229696 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f015b040 con 0x7f12f00d26b0 2024-02-19T15:31:27.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.897+0000 7f12f7c00640 1 -- 172.21.15.47:0/457229696 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f0068500 con 0x7f12f00d26b0 2024-02-19T15:31:27.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.897+0000 7f12ecc00640 1 -- 172.21.15.47:0/457229696 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f12dc0b4a70 con 0x7f12f00d26b0 2024-02-19T15:31:27.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.901+0000 7f12ecc00640 1 --2- 172.21.15.47:0/457229696 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f12d0066e40 0x7f12d0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:27.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.901+0000 7f12ecc00640 1 -- 172.21.15.47:0/457229696 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f12dc11efb0 con 0x7f12f00d26b0 2024-02-19T15:31:27.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.901+0000 7f12f5200640 1 --2- 172.21.15.47:0/457229696 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f12d0066e40 0x7f12d0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:27.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.901+0000 7f12f5200640 1 --2- 172.21.15.47:0/457229696 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f12d0066e40 0x7f12d0069300 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f12e4007fd0 tx=0x7f12e4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:27.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:27.905+0000 7f12ecc00640 1 -- 172.21.15.47:0/457229696 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f12dc0ebd70 con 0x7f12f00d26b0 2024-02-19T15:31:28.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:28.097+0000 7f12f7c00640 1 -- 172.21.15.47:0/457229696 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12f00d1170 con 0x7f12f00d26b0 2024-02-19T15:31:28.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:28.097+0000 7f12ecc00640 1 -- 172.21.15.47:0/457229696 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f12dc0c3070 con 0x7f12f00d26b0 2024-02-19T15:31:28.098 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:28.098 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:28.098 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:28.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:28.101+0000 7f12f7c00640 1 -- 172.21.15.47:0/457229696 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f12d0066e40 msgr2=0x7f12d0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:28.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:28.101+0000 7f12f7c00640 1 --2- 172.21.15.47:0/457229696 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f12d0066e40 0x7f12d0069300 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f12e4007fd0 tx=0x7f12e4009040 comp rx=0 tx=0).stop 2024-02-19T15:31:28.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:28.101+0000 7f12f7c00640 1 -- 172.21.15.47:0/457229696 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12f00d26b0 msgr2=0x7f12f00b6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:28.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:28.101+0000 7f12f7c00640 1 --2- 172.21.15.47:0/457229696 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12f00d26b0 0x7f12f00b6c50 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f12dc0ac040 tx=0x7f12dc01acf0 comp rx=0 tx=0).stop 2024-02-19T15:31:28.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:28.101+0000 7f12f7c00640 1 -- 172.21.15.47:0/457229696 shutdown_connections 2024-02-19T15:31:28.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:28.101+0000 7f12f7c00640 1 --2- 172.21.15.47:0/457229696 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f12d0066e40 0x7f12d0069300 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:28.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:28.101+0000 7f12f7c00640 1 --2- 172.21.15.47:0/457229696 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f015be90 0x7f12f0140730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:28.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:28.101+0000 7f12f7c00640 1 --2- 172.21.15.47:0/457229696 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12f00d26b0 0x7f12f00b6c50 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:28.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:28.101+0000 7f12f7c00640 1 --2- 172.21.15.47:0/457229696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f00ceb70 0x7f12f015ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:28.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:28.101+0000 7f12f7c00640 1 -- 172.21.15.47:0/457229696 >> 172.21.15.47:0/457229696 conn(0x7f12f00d1b90 msgr2=0x7f12f00cffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:28.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:28.101+0000 7f12f7c00640 1 -- 172.21.15.47:0/457229696 shutdown_connections 2024-02-19T15:31:28.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:28.101+0000 7f12f7c00640 1 -- 172.21.15.47:0/457229696 wait complete. 2024-02-19T15:31:28.260 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:28.320 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:28.320 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:29.321 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:29.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.461+0000 7f3d93400640 1 Processor -- start 2024-02-19T15:31:29.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.465+0000 7f3d93400640 1 -- start start 2024-02-19T15:31:29.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.465+0000 7f3d93400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3d8c06d650 0x7f3d8c0d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:29.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.465+0000 7f3d93400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d8c0cce30 con 0x7f3d8c0ced10 2024-02-19T15:31:29.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.465+0000 7f3d93400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d8c0cd110 con 0x7f3d8c0cded0 2024-02-19T15:31:29.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.465+0000 7f3d93400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d8c0cd3f0 con 0x7f3d8c06d650 2024-02-19T15:31:29.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.465+0000 7f3d90a00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3d8c06d650 0x7f3d8c0d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:29.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.465+0000 7f3d91400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f3d8c0ced10 0x7f3d8c06bd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45856/0 (socket says 172.21.15.47:45856) 2024-02-19T15:31:29.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.465+0000 7f3d91400640 1 -- 172.21.15.47:0/2641857590 learned_addr learned my addr 172.21.15.47:0/2641857590 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:31:29.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.465+0000 7f3d90a00640 1 -- 172.21.15.47:0/2641857590 >> v1:172.21.15.116:6789/0 conn(0x7f3d8c0cded0 legacy=0x7f3d8c0ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:29.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.465+0000 7f3d90a00640 1 -- 172.21.15.47:0/2641857590 >> v1:172.21.15.47:6789/0 conn(0x7f3d8c0ced10 legacy=0x7f3d8c06bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:29.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.465+0000 7f3d90a00640 1 -- 172.21.15.47:0/2641857590 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8c0cd6d0 con 0x7f3d8c06d650 2024-02-19T15:31:29.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.465+0000 7f3d8b400640 1 -- 172.21.15.47:0/2641857590 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621665 0 0) 0x7f3d8c0cce30 con 0x7f3d8c0ced10 2024-02-19T15:31:29.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.469+0000 7f3d90a00640 1 --2- 172.21.15.47:0/2641857590 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d8c06d650 0x7f3d8c0d08b0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f3d800097f0 tx=0x7f3d8001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8270f5228219d4d1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:29.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.469+0000 7f3d8b400640 1 -- 172.21.15.47:0/2641857590 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d800ab600 con 0x7f3d8c06d650 2024-02-19T15:31:29.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.469+0000 7f3d8b400640 1 -- 172.21.15.47:0/2641857590 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d800ab760 con 0x7f3d8c06d650 2024-02-19T15:31:29.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.469+0000 7f3d8b400640 1 -- 172.21.15.47:0/2641857590 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d800abaf0 con 0x7f3d8c06d650 2024-02-19T15:31:29.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.469+0000 7f3d93400640 1 -- 172.21.15.47:0/2641857590 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d8c06d650 msgr2=0x7f3d8c0d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:29.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.469+0000 7f3d93400640 1 --2- 172.21.15.47:0/2641857590 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d8c06d650 0x7f3d8c0d08b0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f3d800097f0 tx=0x7f3d8001c3e0 comp rx=0 tx=0).stop 2024-02-19T15:31:29.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.469+0000 7f3d93400640 1 -- 172.21.15.47:0/2641857590 shutdown_connections 2024-02-19T15:31:29.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.469+0000 7f3d93400640 1 --2- 172.21.15.47:0/2641857590 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d8c06d650 0x7f3d8c0d08b0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:29.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.469+0000 7f3d93400640 1 -- 172.21.15.47:0/2641857590 >> 172.21.15.47:0/2641857590 conn(0x7f3d8c0d1bf0 msgr2=0x7f3d8c0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:29.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.469+0000 7f3d93400640 1 -- 172.21.15.47:0/2641857590 shutdown_connections 2024-02-19T15:31:29.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.469+0000 7f3d93400640 1 -- 172.21.15.47:0/2641857590 wait complete. 2024-02-19T15:31:29.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.473+0000 7f3d93400640 1 Processor -- start 2024-02-19T15:31:29.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.473+0000 7f3d93400640 1 -- start start 2024-02-19T15:31:29.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d93400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d8c06d650 0x7f3d8c0c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:29.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d93400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d8c0cded0 0x7f3d8c0c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:29.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d93400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d8c0ced10 0x7f3d8c14b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:29.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d93400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c0c7d20 con 0x7f3d8c0ced10 2024-02-19T15:31:29.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d93400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c0c9fc0 con 0x7f3d8c0cded0 2024-02-19T15:31:29.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d93400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c0c7110 con 0x7f3d8c06d650 2024-02-19T15:31:29.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d91400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d8c0ced10 0x7f3d8c14b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:29.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d91400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d8c0ced10 0x7f3d8c14b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36542/0 (socket says 172.21.15.47:36542) 2024-02-19T15:31:29.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d91400640 1 -- 172.21.15.47:0/2147981019 learned_addr learned my addr 172.21.15.47:0/2147981019 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:29.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d90a00640 1 --2- 172.21.15.47:0/2147981019 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d8c06d650 0x7f3d8c0c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:29.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d8be00640 1 --2- 172.21.15.47:0/2147981019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d8c0cded0 0x7f3d8c0c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:29.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d91400640 1 -- 172.21.15.47:0/2147981019 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d8c06d650 msgr2=0x7f3d8c0c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:29.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d91400640 1 --2- 172.21.15.47:0/2147981019 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d8c06d650 0x7f3d8c0c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:29.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d91400640 1 -- 172.21.15.47:0/2147981019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d8c0cded0 msgr2=0x7f3d8c0c9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:29.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d91400640 1 --2- 172.21.15.47:0/2147981019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d8c0cded0 0x7f3d8c0c9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:29.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d91400640 1 -- 172.21.15.47:0/2147981019 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d80009420 con 0x7f3d8c0ced10 2024-02-19T15:31:29.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d91400640 1 --2- 172.21.15.47:0/2147981019 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d8c0ced10 0x7f3d8c14b010 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f3d8400c7a0 tx=0x7f3d8400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:29.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d88c00640 1 -- 172.21.15.47:0/2147981019 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8400a640 con 0x7f3d8c0ced10 2024-02-19T15:31:29.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d88c00640 1 -- 172.21.15.47:0/2147981019 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8400a7a0 con 0x7f3d8c0ced10 2024-02-19T15:31:29.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d88c00640 1 -- 172.21.15.47:0/2147981019 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8400abd0 con 0x7f3d8c0ced10 2024-02-19T15:31:29.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d93400640 1 -- 172.21.15.47:0/2147981019 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d8c0c73f0 con 0x7f3d8c0ced10 2024-02-19T15:31:29.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d93400640 1 -- 172.21.15.47:0/2147981019 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d8c0c79b0 con 0x7f3d8c0ced10 2024-02-19T15:31:29.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d93400640 1 -- 172.21.15.47:0/2147981019 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d8c0c7e90 con 0x7f3d8c0ced10 2024-02-19T15:31:29.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.477+0000 7f3d88c00640 1 -- 172.21.15.47:0/2147981019 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3d8400ad30 con 0x7f3d8c0ced10 2024-02-19T15:31:29.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.481+0000 7f3d88c00640 1 --2- 172.21.15.47:0/2147981019 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3d6c066e40 0x7f3d6c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:29.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.481+0000 7f3d88c00640 1 -- 172.21.15.47:0/2147981019 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3d84083f50 con 0x7f3d8c0ced10 2024-02-19T15:31:29.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.481+0000 7f3d90a00640 1 --2- 172.21.15.47:0/2147981019 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3d6c066e40 0x7f3d6c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:29.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.481+0000 7f3d90a00640 1 --2- 172.21.15.47:0/2147981019 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3d6c066e40 0x7f3d6c069300 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f3d800af000 tx=0x7f3d800a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:29.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.485+0000 7f3d88c00640 1 -- 172.21.15.47:0/2147981019 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f3d84050db0 con 0x7f3d8c0ced10 2024-02-19T15:31:29.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.681+0000 7f3d93400640 1 -- 172.21.15.47:0/2147981019 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d8c075360 con 0x7f3d8c0ced10 2024-02-19T15:31:29.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.685+0000 7f3d88c00640 1 -- 172.21.15.47:0/2147981019 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f3d84018070 con 0x7f3d8c0ced10 2024-02-19T15:31:29.682 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:29.683 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:29.683 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:29.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.685+0000 7f3d93400640 1 -- 172.21.15.47:0/2147981019 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3d6c066e40 msgr2=0x7f3d6c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:29.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.685+0000 7f3d93400640 1 --2- 172.21.15.47:0/2147981019 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3d6c066e40 0x7f3d6c069300 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f3d800af000 tx=0x7f3d800a7000 comp rx=0 tx=0).stop 2024-02-19T15:31:29.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.685+0000 7f3d93400640 1 -- 172.21.15.47:0/2147981019 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d8c0ced10 msgr2=0x7f3d8c14b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:29.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.685+0000 7f3d93400640 1 --2- 172.21.15.47:0/2147981019 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d8c0ced10 0x7f3d8c14b010 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f3d8400c7a0 tx=0x7f3d8400cc60 comp rx=0 tx=0).stop 2024-02-19T15:31:29.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.685+0000 7f3d93400640 1 -- 172.21.15.47:0/2147981019 shutdown_connections 2024-02-19T15:31:29.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.685+0000 7f3d93400640 1 --2- 172.21.15.47:0/2147981019 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3d6c066e40 0x7f3d6c069300 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:29.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.685+0000 7f3d93400640 1 --2- 172.21.15.47:0/2147981019 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d8c0ced10 0x7f3d8c14b010 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:29.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.685+0000 7f3d93400640 1 --2- 172.21.15.47:0/2147981019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d8c0cded0 0x7f3d8c0c9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:29.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.685+0000 7f3d93400640 1 --2- 172.21.15.47:0/2147981019 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d8c06d650 0x7f3d8c0c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:29.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.685+0000 7f3d93400640 1 -- 172.21.15.47:0/2147981019 >> 172.21.15.47:0/2147981019 conn(0x7f3d8c0d1bf0 msgr2=0x7f3d8c0d05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:29.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.685+0000 7f3d93400640 1 -- 172.21.15.47:0/2147981019 shutdown_connections 2024-02-19T15:31:29.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:29.685+0000 7f3d93400640 1 -- 172.21.15.47:0/2147981019 wait complete. 2024-02-19T15:31:29.905 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:29.905 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:30.262 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:30.906 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:31.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.045+0000 7f961a000640 1 Processor -- start 2024-02-19T15:31:31.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.045+0000 7f961a000640 1 -- start start 2024-02-19T15:31:31.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.049+0000 7f961a000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f96140cef90 0x7f96140ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:31.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.049+0000 7f961a000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f96140ce930 0x7f96140cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:31.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.049+0000 7f961a000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96140ced80 con 0x7f96140b2fc0 2024-02-19T15:31:31.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.049+0000 7f961a000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96140d0e80 con 0x7f96140cef90 2024-02-19T15:31:31.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.049+0000 7f961a000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96140d10a0 con 0x7f96140ce930 2024-02-19T15:31:31.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.049+0000 7f960aa00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f96140cef90 0x7f96140ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:31.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.049+0000 7f960aa00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f96140cef90 0x7f96140ce3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36552/0 (socket says 172.21.15.47:36552) 2024-02-19T15:31:31.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.049+0000 7f960aa00640 1 -- 172.21.15.47:0/1950039807 learned_addr learned my addr 172.21.15.47:0/1950039807 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:31.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.049+0000 7f9613e00640 1 --2- 172.21.15.47:0/1950039807 >> v2:172.21.15.116:3300/0 conn(0x7f96140ce930 0x7f96140cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:31.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.049+0000 7f960aa00640 1 -- 172.21.15.47:0/1950039807 >> v2:172.21.15.116:3300/0 conn(0x7f96140ce930 msgr2=0x7f96140cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:31.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.049+0000 7f960aa00640 1 --2- 172.21.15.47:0/1950039807 >> v2:172.21.15.116:3300/0 conn(0x7f96140ce930 0x7f96140cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:31.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.049+0000 7f960aa00640 1 -- 172.21.15.47:0/1950039807 >> v1:172.21.15.116:6789/0 conn(0x7f96140b2fc0 legacy=0x7f96140d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:31.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.049+0000 7f960aa00640 1 -- 172.21.15.47:0/1950039807 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96140b33a0 con 0x7f96140cef90 2024-02-19T15:31:31.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.053+0000 7f960aa00640 1 --2- 172.21.15.47:0/1950039807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96140cef90 0x7f96140ce3f0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f96140729e0 tx=0x7f95fc01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6809d6e51c079f5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.053+0000 7f9612a00640 1 -- 172.21.15.47:0/1950039807 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95fc0ab660 con 0x7f96140cef90 2024-02-19T15:31:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.053+0000 7f9612a00640 1 -- 172.21.15.47:0/1950039807 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95fc0ab7c0 con 0x7f96140cef90 2024-02-19T15:31:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.053+0000 7f9612a00640 1 -- 172.21.15.47:0/1950039807 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95fc0abb50 con 0x7f96140cef90 2024-02-19T15:31:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.053+0000 7f961a000640 1 -- 172.21.15.47:0/1950039807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96140cef90 msgr2=0x7f96140ce3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.053+0000 7f961a000640 1 --2- 172.21.15.47:0/1950039807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96140cef90 0x7f96140ce3f0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f96140729e0 tx=0x7f95fc01c460 comp rx=0 tx=0).stop 2024-02-19T15:31:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.053+0000 7f961a000640 1 -- 172.21.15.47:0/1950039807 shutdown_connections 2024-02-19T15:31:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.053+0000 7f961a000640 1 --2- 172.21.15.47:0/1950039807 >> v2:172.21.15.116:3300/0 conn(0x7f96140ce930 0x7f96140cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.053+0000 7f961a000640 1 --2- 172.21.15.47:0/1950039807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96140cef90 0x7f96140ce3f0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.053+0000 7f961a000640 1 -- 172.21.15.47:0/1950039807 >> 172.21.15.47:0/1950039807 conn(0x7f96140d1b70 msgr2=0x7f96140d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.053+0000 7f961a000640 1 -- 172.21.15.47:0/1950039807 shutdown_connections 2024-02-19T15:31:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.053+0000 7f961a000640 1 -- 172.21.15.47:0/1950039807 wait complete. 2024-02-19T15:31:31.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.053+0000 7f961a000640 1 Processor -- start 2024-02-19T15:31:31.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.053+0000 7f961a000640 1 -- start start 2024-02-19T15:31:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.053+0000 7f961a000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96140b2fc0 0x7f961414c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.053+0000 7f961a000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96140ce930 0x7f96140c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.053+0000 7f961a000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96140cef90 0x7f96140c5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.053+0000 7f961a000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96140c67c0 con 0x7f96140ce930 2024-02-19T15:31:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.053+0000 7f961a000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9614160c40 con 0x7f96140b2fc0 2024-02-19T15:31:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.053+0000 7f961a000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9614160f20 con 0x7f96140cef90 2024-02-19T15:31:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.057+0000 7f960aa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96140ce930 0x7f96140c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.057+0000 7f960aa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96140ce930 0x7f96140c58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36562/0 (socket says 172.21.15.47:36562) 2024-02-19T15:31:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.057+0000 7f960aa00640 1 -- 172.21.15.47:0/226177680 learned_addr learned my addr 172.21.15.47:0/226177680 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.057+0000 7f9613400640 1 --2- 172.21.15.47:0/226177680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96140b2fc0 0x7f961414c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:31.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.057+0000 7f9613e00640 1 --2- 172.21.15.47:0/226177680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96140cef90 0x7f96140c5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:31.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.057+0000 7f960aa00640 1 -- 172.21.15.47:0/226177680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96140cef90 msgr2=0x7f96140c5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:31.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.057+0000 7f960aa00640 1 --2- 172.21.15.47:0/226177680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96140cef90 0x7f96140c5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:31.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.057+0000 7f960aa00640 1 -- 172.21.15.47:0/226177680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96140b2fc0 msgr2=0x7f961414c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:31.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.057+0000 7f960aa00640 1 --2- 172.21.15.47:0/226177680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96140b2fc0 0x7f961414c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:31.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.057+0000 7f960aa00640 1 -- 172.21.15.47:0/226177680 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95fc0094d0 con 0x7f96140ce930 2024-02-19T15:31:31.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.057+0000 7f960aa00640 1 --2- 172.21.15.47:0/226177680 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96140ce930 0x7f96140c58b0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f95fc019070 tx=0x7f95fc01aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:31.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.057+0000 7f960be00640 1 -- 172.21.15.47:0/226177680 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95fc0b4800 con 0x7f96140ce930 2024-02-19T15:31:31.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.057+0000 7f960be00640 1 -- 172.21.15.47:0/226177680 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95fc0b4980 con 0x7f96140ce930 2024-02-19T15:31:31.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.057+0000 7f960be00640 1 -- 172.21.15.47:0/226177680 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95fc0b4d30 con 0x7f96140ce930 2024-02-19T15:31:31.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.057+0000 7f961a000640 1 -- 172.21.15.47:0/226177680 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9614161200 con 0x7f96140ce930 2024-02-19T15:31:31.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.057+0000 7f961a000640 1 -- 172.21.15.47:0/226177680 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f961414cc10 con 0x7f96140ce930 2024-02-19T15:31:31.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.057+0000 7f961a000640 1 -- 172.21.15.47:0/226177680 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9614075360 con 0x7f96140ce930 2024-02-19T15:31:31.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.057+0000 7f960be00640 1 -- 172.21.15.47:0/226177680 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f95fc0a9070 con 0x7f96140ce930 2024-02-19T15:31:31.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.061+0000 7f960be00640 1 --2- 172.21.15.47:0/226177680 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f95f0066e70 0x7f95f0069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:31.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.061+0000 7f960be00640 1 -- 172.21.15.47:0/226177680 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f95fc11f300 con 0x7f96140ce930 2024-02-19T15:31:31.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.061+0000 7f9613400640 1 --2- 172.21.15.47:0/226177680 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f95f0066e70 0x7f95f0069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:31.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.061+0000 7f9613400640 1 --2- 172.21.15.47:0/226177680 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f95f0066e70 0x7f95f0069330 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f9604001ea0 tx=0x7f9604009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:31.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.065+0000 7f960be00640 1 -- 172.21.15.47:0/226177680 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f95fc0ec310 con 0x7f96140ce930 2024-02-19T15:31:31.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.261+0000 7f961a000640 1 -- 172.21.15.47:0/226177680 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f961406cf70 con 0x7f96140ce930 2024-02-19T15:31:31.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.265+0000 7f960be00640 1 -- 172.21.15.47:0/226177680 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f95fc0ef9d0 con 0x7f96140ce930 2024-02-19T15:31:31.263 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:31.263 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:31.263 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:31.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.265+0000 7f961a000640 1 -- 172.21.15.47:0/226177680 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f95f0066e70 msgr2=0x7f95f0069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:31.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.265+0000 7f961a000640 1 --2- 172.21.15.47:0/226177680 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f95f0066e70 0x7f95f0069330 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f9604001ea0 tx=0x7f9604009040 comp rx=0 tx=0).stop 2024-02-19T15:31:31.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.265+0000 7f961a000640 1 -- 172.21.15.47:0/226177680 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96140ce930 msgr2=0x7f96140c58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:31.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.265+0000 7f961a000640 1 --2- 172.21.15.47:0/226177680 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96140ce930 0x7f96140c58b0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f95fc019070 tx=0x7f95fc01aca0 comp rx=0 tx=0).stop 2024-02-19T15:31:31.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.265+0000 7f961a000640 1 -- 172.21.15.47:0/226177680 shutdown_connections 2024-02-19T15:31:31.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.265+0000 7f961a000640 1 --2- 172.21.15.47:0/226177680 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f95f0066e70 0x7f95f0069330 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:31.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.265+0000 7f961a000640 1 --2- 172.21.15.47:0/226177680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96140cef90 0x7f96140c5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:31.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.265+0000 7f961a000640 1 --2- 172.21.15.47:0/226177680 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96140ce930 0x7f96140c58b0 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:31.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.265+0000 7f961a000640 1 --2- 172.21.15.47:0/226177680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96140b2fc0 0x7f961414c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:31.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.265+0000 7f961a000640 1 -- 172.21.15.47:0/226177680 >> 172.21.15.47:0/226177680 conn(0x7f96140d1b70 msgr2=0x7f9614070350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:31.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.265+0000 7f961a000640 1 -- 172.21.15.47:0/226177680 shutdown_connections 2024-02-19T15:31:31.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:31.265+0000 7f961a000640 1 -- 172.21.15.47:0/226177680 wait complete. 2024-02-19T15:31:31.485 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:31.485 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:32.264 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:32.486 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:32.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.625+0000 7f865b200640 1 Processor -- start 2024-02-19T15:31:32.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.629+0000 7f865b200640 1 -- start start 2024-02-19T15:31:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.629+0000 7f865b200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f865c15dc90 0x7f865c0cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.629+0000 7f865b200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f865c161a10 con 0x7f865c15e070 2024-02-19T15:31:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.629+0000 7f865b200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f865c161cf0 con 0x7f865c06b5c0 2024-02-19T15:31:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.629+0000 7f865b200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f865c161fd0 con 0x7f865c15dc90 2024-02-19T15:31:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.629+0000 7f8659e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f865c15dc90 0x7f865c0cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.629+0000 7f8659e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f865c15dc90 0x7f865c0cfc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36572/0 (socket says 172.21.15.47:36572) 2024-02-19T15:31:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.629+0000 7f8659e00640 1 -- 172.21.15.47:0/3683346635 learned_addr learned my addr 172.21.15.47:0/3683346635 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.629+0000 7f8653e00640 1 --1- 172.21.15.47:0/3683346635 >> v1:172.21.15.47:6789/0 conn(0x7f865c15e070 0x7f865c06aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45858/0 (socket says 172.21.15.47:45858) 2024-02-19T15:31:32.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.629+0000 7f8659e00640 1 -- 172.21.15.47:0/3683346635 >> v1:172.21.15.116:6789/0 conn(0x7f865c06b5c0 legacy=0x7f865c071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:32.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.629+0000 7f8659e00640 1 -- 172.21.15.47:0/3683346635 >> v1:172.21.15.47:6789/0 conn(0x7f865c15e070 legacy=0x7f865c06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:32.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.629+0000 7f8659e00640 1 -- 172.21.15.47:0/3683346635 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865c1622b0 con 0x7f865c15dc90 2024-02-19T15:31:32.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.629+0000 7f8659400640 1 -- 172.21.15.47:0/3683346635 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029567081 0 0) 0x7f865c161a10 con 0x7f865c15e070 2024-02-19T15:31:32.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.633+0000 7f8659e00640 1 --2- 172.21.15.47:0/3683346635 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f865c15dc90 0x7f865c0cfc10 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f864c009870 tx=0x7f864c01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea43d6bc6e95c7c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:32.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.633+0000 7f8659400640 1 -- 172.21.15.47:0/3683346635 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c0a8a50 con 0x7f865c15dc90 2024-02-19T15:31:32.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.633+0000 7f8659400640 1 -- 172.21.15.47:0/3683346635 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864c0a8bb0 con 0x7f865c15dc90 2024-02-19T15:31:32.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.633+0000 7f8659400640 1 -- 172.21.15.47:0/3683346635 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c0b0440 con 0x7f865c15dc90 2024-02-19T15:31:32.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.633+0000 7f865b200640 1 -- 172.21.15.47:0/3683346635 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f865c15dc90 msgr2=0x7f865c0cfc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:32.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.633+0000 7f865b200640 1 --2- 172.21.15.47:0/3683346635 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f865c15dc90 0x7f865c0cfc10 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f864c009870 tx=0x7f864c01ed50 comp rx=0 tx=0).stop 2024-02-19T15:31:32.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.633+0000 7f865b200640 1 -- 172.21.15.47:0/3683346635 shutdown_connections 2024-02-19T15:31:32.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.633+0000 7f865b200640 1 --2- 172.21.15.47:0/3683346635 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f865c15dc90 0x7f865c0cfc10 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:32.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.633+0000 7f865b200640 1 -- 172.21.15.47:0/3683346635 >> 172.21.15.47:0/3683346635 conn(0x7f865c0d08d0 msgr2=0x7f865c0d0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:32.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.633+0000 7f865b200640 1 -- 172.21.15.47:0/3683346635 shutdown_connections 2024-02-19T15:31:32.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.633+0000 7f865b200640 1 -- 172.21.15.47:0/3683346635 wait complete. 2024-02-19T15:31:32.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f865b200640 1 Processor -- start 2024-02-19T15:31:32.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f865b200640 1 -- start start 2024-02-19T15:31:32.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f865b200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f865c06b5c0 0x7f865c06eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f865b200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f865c15dc90 0x7f865c0b4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f865b200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f865c15e070 0x7f865c0b4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f865b200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865c06ea10 con 0x7f865c15e070 2024-02-19T15:31:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f865b200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865c06de00 con 0x7f865c15dc90 2024-02-19T15:31:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f865b200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f865c06e0e0 con 0x7f865c06b5c0 2024-02-19T15:31:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f865a800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f865c15e070 0x7f865c0b4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f8659e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f865c06b5c0 0x7f865c06eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f8653e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f865c15dc90 0x7f865c0b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f8659e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f865c06b5c0 0x7f865c06eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:38660/0 (socket says 172.21.15.47:38660) 2024-02-19T15:31:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f8659e00640 1 -- 172.21.15.47:0/1826056620 learned_addr learned my addr 172.21.15.47:0/1826056620 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f8659e00640 1 -- 172.21.15.47:0/1826056620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f865c15dc90 msgr2=0x7f865c0b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f8659e00640 1 --2- 172.21.15.47:0/1826056620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f865c15dc90 0x7f865c0b4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f8659e00640 1 -- 172.21.15.47:0/1826056620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f865c15e070 msgr2=0x7f865c0b4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f8659e00640 1 --2- 172.21.15.47:0/1826056620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f865c15e070 0x7f865c0b4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f8659e00640 1 -- 172.21.15.47:0/1826056620 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c0094d0 con 0x7f865c06b5c0 2024-02-19T15:31:32.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f865a800640 1 --2- 172.21.15.47:0/1826056620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f865c15e070 0x7f865c0b4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:31:32.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f8659e00640 1 --2- 172.21.15.47:0/1826056620 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f865c06b5c0 0x7f865c06eea0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f864c0b3040 tx=0x7f864c01baf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:32.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f8652000640 1 -- 172.21.15.47:0/1826056620 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c0b0d20 con 0x7f865c06b5c0 2024-02-19T15:31:32.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f8652000640 1 -- 172.21.15.47:0/1826056620 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864c0b0e80 con 0x7f865c06b5c0 2024-02-19T15:31:32.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f8652000640 1 -- 172.21.15.47:0/1826056620 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c004240 con 0x7f865c06b5c0 2024-02-19T15:31:32.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.637+0000 7f865b200640 1 -- 172.21.15.47:0/1826056620 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f865c06e3c0 con 0x7f865c06b5c0 2024-02-19T15:31:32.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.641+0000 7f865b200640 1 -- 172.21.15.47:0/1826056620 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f865c149ca0 con 0x7f865c06b5c0 2024-02-19T15:31:32.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.641+0000 7f865b200640 1 -- 172.21.15.47:0/1826056620 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f865c149e90 con 0x7f865c06b5c0 2024-02-19T15:31:32.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.641+0000 7f8652000640 1 -- 172.21.15.47:0/1826056620 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f864c0b7040 con 0x7f865c06b5c0 2024-02-19T15:31:32.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.645+0000 7f8652000640 1 --2- 172.21.15.47:0/1826056620 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8638066e40 0x7f8638069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:32.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.645+0000 7f8652000640 1 -- 172.21.15.47:0/1826056620 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f864c121b70 con 0x7f865c06b5c0 2024-02-19T15:31:32.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.645+0000 7f8653e00640 1 --2- 172.21.15.47:0/1826056620 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8638066e40 0x7f8638069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:32.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.645+0000 7f8653e00640 1 --2- 172.21.15.47:0/1826056620 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8638066e40 0x7f8638069300 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f865c1f6fb0 tx=0x7f8644007240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:32.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:32.645+0000 7f8652000640 1 -- 172.21.15.47:0/1826056620 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f864c0ee930 con 0x7f865c06b5c0 2024-02-19T15:31:33.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:33.005+0000 7f865b200640 1 -- 172.21.15.47:0/1826056620 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f865c075360 con 0x7f865c06b5c0 2024-02-19T15:31:33.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:33.005+0000 7f8652000640 1 -- 172.21.15.47:0/1826056620 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f864c0b5070 con 0x7f865c06b5c0 2024-02-19T15:31:33.005 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:33.005 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:33.006 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:33.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:33.009+0000 7f865b200640 1 -- 172.21.15.47:0/1826056620 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8638066e40 msgr2=0x7f8638069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:33.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:33.009+0000 7f865b200640 1 --2- 172.21.15.47:0/1826056620 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8638066e40 0x7f8638069300 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f865c1f6fb0 tx=0x7f8644007240 comp rx=0 tx=0).stop 2024-02-19T15:31:33.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:33.009+0000 7f865b200640 1 -- 172.21.15.47:0/1826056620 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f865c06b5c0 msgr2=0x7f865c06eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:33.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:33.009+0000 7f865b200640 1 --2- 172.21.15.47:0/1826056620 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f865c06b5c0 0x7f865c06eea0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f864c0b3040 tx=0x7f864c01baf0 comp rx=0 tx=0).stop 2024-02-19T15:31:33.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:33.009+0000 7f865b200640 1 -- 172.21.15.47:0/1826056620 shutdown_connections 2024-02-19T15:31:33.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:33.009+0000 7f865b200640 1 --2- 172.21.15.47:0/1826056620 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8638066e40 0x7f8638069300 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:33.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:33.009+0000 7f865b200640 1 --2- 172.21.15.47:0/1826056620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f865c15e070 0x7f865c0b4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:33.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:33.009+0000 7f865b200640 1 --2- 172.21.15.47:0/1826056620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f865c15dc90 0x7f865c0b4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:33.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:33.009+0000 7f865b200640 1 --2- 172.21.15.47:0/1826056620 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f865c06b5c0 0x7f865c06eea0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:33.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:33.009+0000 7f865b200640 1 -- 172.21.15.47:0/1826056620 >> 172.21.15.47:0/1826056620 conn(0x7f865c0d08d0 msgr2=0x7f865c0b02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:33.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:33.009+0000 7f865b200640 1 -- 172.21.15.47:0/1826056620 shutdown_connections 2024-02-19T15:31:33.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:33.009+0000 7f865b200640 1 -- 172.21.15.47:0/1826056620 wait complete. 2024-02-19T15:31:33.228 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:33.228 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:34.229 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:34.266 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:34.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.373+0000 7f8613200640 1 Processor -- start 2024-02-19T15:31:34.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.373+0000 7f8613200640 1 -- start start 2024-02-19T15:31:34.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.373+0000 7f8613200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f860c145590 0x7f860c0727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:34.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.373+0000 7f8613200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f860c0cea80 0x7f860c06c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:34.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.373+0000 7f8613200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f860c15c390 0x7f860c15c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:34.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.373+0000 7f8613200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860c15d280 con 0x7f860c15c390 2024-02-19T15:31:34.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.373+0000 7f8613200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860c070ff0 con 0x7f860c145590 2024-02-19T15:31:34.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.373+0000 7f8613200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860c15cd60 con 0x7f860c0cea80 2024-02-19T15:31:34.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.373+0000 7f8611200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f860c15c390 0x7f860c15c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:34.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.373+0000 7f8611200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f860c15c390 0x7f860c15c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:35644/0 (socket says 172.21.15.47:35644) 2024-02-19T15:31:34.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.373+0000 7f8611200640 1 -- 172.21.15.47:0/4008562585 learned_addr learned my addr 172.21.15.47:0/4008562585 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:34.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.373+0000 7f8603e00640 1 --2- 172.21.15.47:0/4008562585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f860c0cea80 0x7f860c06c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:34.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.373+0000 7f860be00640 1 --2- 172.21.15.47:0/4008562585 >> v2:172.21.15.116:3300/0 conn(0x7f860c145590 0x7f860c0727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:34.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.373+0000 7f8611200640 1 -- 172.21.15.47:0/4008562585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f860c0cea80 msgr2=0x7f860c06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:34.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.373+0000 7f8611200640 1 --2- 172.21.15.47:0/4008562585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f860c0cea80 0x7f860c06c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:34.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.373+0000 7f8611200640 1 -- 172.21.15.47:0/4008562585 >> v2:172.21.15.116:3300/0 conn(0x7f860c145590 msgr2=0x7f860c0727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:34.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.373+0000 7f8611200640 1 --2- 172.21.15.47:0/4008562585 >> v2:172.21.15.116:3300/0 conn(0x7f860c145590 0x7f860c0727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:34.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.373+0000 7f8611200640 1 -- 172.21.15.47:0/4008562585 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860c0709e0 con 0x7f860c15c390 2024-02-19T15:31:34.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.377+0000 7f8611200640 1 --2- 172.21.15.47:0/4008562585 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f860c15c390 0x7f860c15c820 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f8604009e50 tx=0x7f860401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83e501d0505bfbe server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:34.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.377+0000 7f860b400640 1 -- 172.21.15.47:0/4008562585 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86040a5780 con 0x7f860c15c390 2024-02-19T15:31:34.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.377+0000 7f860b400640 1 -- 172.21.15.47:0/4008562585 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86040a58e0 con 0x7f860c15c390 2024-02-19T15:31:34.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.377+0000 7f860b400640 1 -- 172.21.15.47:0/4008562585 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86040a5c70 con 0x7f860c15c390 2024-02-19T15:31:34.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.377+0000 7f8613200640 1 -- 172.21.15.47:0/4008562585 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f860c15c390 msgr2=0x7f860c15c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:34.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.377+0000 7f8613200640 1 --2- 172.21.15.47:0/4008562585 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f860c15c390 0x7f860c15c820 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f8604009e50 tx=0x7f860401c9f0 comp rx=0 tx=0).stop 2024-02-19T15:31:34.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.377+0000 7f8613200640 1 -- 172.21.15.47:0/4008562585 shutdown_connections 2024-02-19T15:31:34.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.377+0000 7f8613200640 1 --2- 172.21.15.47:0/4008562585 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f860c15c390 0x7f860c15c820 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:34.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.377+0000 7f8613200640 1 --2- 172.21.15.47:0/4008562585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f860c0cea80 0x7f860c06c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:34.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.377+0000 7f8613200640 1 --2- 172.21.15.47:0/4008562585 >> v2:172.21.15.116:3300/0 conn(0x7f860c145590 0x7f860c0727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:34.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.377+0000 7f8613200640 1 -- 172.21.15.47:0/4008562585 >> 172.21.15.47:0/4008562585 conn(0x7f860c0b16a0 msgr2=0x7f860c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:34.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.377+0000 7f8613200640 1 -- 172.21.15.47:0/4008562585 shutdown_connections 2024-02-19T15:31:34.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.377+0000 7f8613200640 1 -- 172.21.15.47:0/4008562585 wait complete. 2024-02-19T15:31:34.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8613200640 1 Processor -- start 2024-02-19T15:31:34.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8613200640 1 -- start start 2024-02-19T15:31:34.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8613200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f860c0cea80 0x7f860c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:34.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8613200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f860c145590 0x7f860c15b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:34.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8613200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f860c15c390 0x7f860c15b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:34.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8613200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c0c53d0 con 0x7f860c145590 2024-02-19T15:31:34.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8613200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c0b35c0 con 0x7f860c0cea80 2024-02-19T15:31:34.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8613200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860c0c4a30 con 0x7f860c15c390 2024-02-19T15:31:34.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8603e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f860c145590 0x7f860c15b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:34.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8603e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f860c145590 0x7f860c15b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:35650/0 (socket says 172.21.15.47:35650) 2024-02-19T15:31:34.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8603e00640 1 -- 172.21.15.47:0/586616971 learned_addr learned my addr 172.21.15.47:0/586616971 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:34.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8611200640 1 --2- 172.21.15.47:0/586616971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f860c15c390 0x7f860c15b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:34.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f860be00640 1 --2- 172.21.15.47:0/586616971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f860c0cea80 0x7f860c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:34.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8603e00640 1 -- 172.21.15.47:0/586616971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f860c15c390 msgr2=0x7f860c15b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:34.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8603e00640 1 --2- 172.21.15.47:0/586616971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f860c15c390 0x7f860c15b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:34.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8603e00640 1 -- 172.21.15.47:0/586616971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f860c0cea80 msgr2=0x7f860c06d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:34.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8603e00640 1 --2- 172.21.15.47:0/586616971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f860c0cea80 0x7f860c06d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:34.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8603e00640 1 -- 172.21.15.47:0/586616971 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8604009b30 con 0x7f860c145590 2024-02-19T15:31:34.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8603e00640 1 --2- 172.21.15.47:0/586616971 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f860c145590 0x7f860c15b280 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f85f400d800 tx=0x7f85f400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:34.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8608c00640 1 -- 172.21.15.47:0/586616971 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f40096a0 con 0x7f860c145590 2024-02-19T15:31:34.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8608c00640 1 -- 172.21.15.47:0/586616971 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f4009800 con 0x7f860c145590 2024-02-19T15:31:34.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8613200640 1 -- 172.21.15.47:0/586616971 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860c0c4d10 con 0x7f860c145590 2024-02-19T15:31:34.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8608c00640 1 -- 172.21.15.47:0/586616971 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f4009b70 con 0x7f860c145590 2024-02-19T15:31:34.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8613200640 1 -- 172.21.15.47:0/586616971 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f860c06d8a0 con 0x7f860c145590 2024-02-19T15:31:34.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.381+0000 7f8613200640 1 -- 172.21.15.47:0/586616971 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860c0cf2e0 con 0x7f860c145590 2024-02-19T15:31:34.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.385+0000 7f8608c00640 1 -- 172.21.15.47:0/586616971 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f85f4009cd0 con 0x7f860c145590 2024-02-19T15:31:34.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.385+0000 7f8608c00640 1 --2- 172.21.15.47:0/586616971 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f85e8066e40 0x7f85e8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:34.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.385+0000 7f8608c00640 1 -- 172.21.15.47:0/586616971 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f85f4085100 con 0x7f860c145590 2024-02-19T15:31:34.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.385+0000 7f860be00640 1 --2- 172.21.15.47:0/586616971 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f85e8066e40 0x7f85e8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:34.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.385+0000 7f860be00640 1 --2- 172.21.15.47:0/586616971 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f85e8066e40 0x7f85e8069300 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f860c0b1170 tx=0x7f85fc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:34.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.389+0000 7f8608c00640 1 -- 172.21.15.47:0/586616971 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f860c0cf2e0 con 0x7f860c145590 2024-02-19T15:31:34.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.585+0000 7f8613200640 1 -- 172.21.15.47:0/586616971 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f860c0cf2e0 con 0x7f860c145590 2024-02-19T15:31:34.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.585+0000 7f8608c00640 1 -- 172.21.15.47:0/586616971 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f860c0cf2e0 con 0x7f860c145590 2024-02-19T15:31:34.584 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:34.584 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:34.584 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:34.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.585+0000 7f8613200640 1 -- 172.21.15.47:0/586616971 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f85e8066e40 msgr2=0x7f85e8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:34.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.585+0000 7f8613200640 1 --2- 172.21.15.47:0/586616971 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f85e8066e40 0x7f85e8069300 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f860c0b1170 tx=0x7f85fc008040 comp rx=0 tx=0).stop 2024-02-19T15:31:34.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.585+0000 7f8613200640 1 -- 172.21.15.47:0/586616971 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f860c145590 msgr2=0x7f860c15b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:34.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.585+0000 7f8613200640 1 --2- 172.21.15.47:0/586616971 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f860c145590 0x7f860c15b280 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f85f400d800 tx=0x7f85f400dcc0 comp rx=0 tx=0).stop 2024-02-19T15:31:34.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.585+0000 7f8613200640 1 -- 172.21.15.47:0/586616971 shutdown_connections 2024-02-19T15:31:34.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.585+0000 7f8613200640 1 --2- 172.21.15.47:0/586616971 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f85e8066e40 0x7f85e8069300 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:34.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.585+0000 7f8613200640 1 --2- 172.21.15.47:0/586616971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f860c15c390 0x7f860c15b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:34.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.585+0000 7f8613200640 1 --2- 172.21.15.47:0/586616971 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f860c145590 0x7f860c15b280 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:34.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.585+0000 7f8613200640 1 --2- 172.21.15.47:0/586616971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f860c0cea80 0x7f860c06d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:34.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.585+0000 7f8613200640 1 -- 172.21.15.47:0/586616971 >> 172.21.15.47:0/586616971 conn(0x7f860c0b16a0 msgr2=0x7f860c144860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:34.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.585+0000 7f8613200640 1 -- 172.21.15.47:0/586616971 shutdown_connections 2024-02-19T15:31:34.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:34.585+0000 7f8613200640 1 -- 172.21.15.47:0/586616971 wait complete. 2024-02-19T15:31:34.806 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:34.807 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:35.807 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:35.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.949+0000 7faccda00640 1 Processor -- start 2024-02-19T15:31:35.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.949+0000 7faccda00640 1 -- start start 2024-02-19T15:31:35.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.949+0000 7faccda00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7facc80cea80 0x7facc80727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:35.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.949+0000 7faccda00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc815d280 con 0x7facc815c390 2024-02-19T15:31:35.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.949+0000 7faccda00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc8070ff0 con 0x7facc8145590 2024-02-19T15:31:35.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.949+0000 7faccda00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc80709e0 con 0x7facc80cea80 2024-02-19T15:31:35.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.949+0000 7facc7400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7facc815c390 0x7facc815c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45074/0 (socket says 172.21.15.47:45074) 2024-02-19T15:31:35.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.949+0000 7facc7400640 1 -- 172.21.15.47:0/669373160 learned_addr learned my addr 172.21.15.47:0/669373160 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:31:35.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.949+0000 7facbe000640 1 --2- 172.21.15.47:0/669373160 >> v2:172.21.15.116:3300/0 conn(0x7facc80cea80 0x7facc80727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:35.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.953+0000 7facc6000640 1 -- 172.21.15.47:0/669373160 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692101063 0 0) 0x7facc815d280 con 0x7facc815c390 2024-02-19T15:31:35.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.953+0000 7facc6000640 1 -- 172.21.15.47:0/669373160 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facac003530 con 0x7facc815c390 2024-02-19T15:31:35.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.953+0000 7facc6000640 1 -- 172.21.15.47:0/669373160 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508331632 0 0) 0x7facc8070ff0 con 0x7facc8145590 2024-02-19T15:31:35.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.953+0000 7facc6000640 1 -- 172.21.15.47:0/669373160 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facc815d280 con 0x7facc8145590 2024-02-19T15:31:35.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.953+0000 7facbe000640 1 -- 172.21.15.47:0/669373160 >> v1:172.21.15.116:6789/0 conn(0x7facc8145590 legacy=0x7facc806c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:35.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.953+0000 7facbe000640 1 -- 172.21.15.47:0/669373160 >> v1:172.21.15.47:6789/0 conn(0x7facc815c390 legacy=0x7facc815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:35.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.953+0000 7facbe000640 1 -- 172.21.15.47:0/669373160 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc80cf020 con 0x7facc80cea80 2024-02-19T15:31:35.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.953+0000 7facbe000640 1 --2- 172.21.15.47:0/669373160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc80cea80 0x7facc80727b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7facb00097c0 tx=0x7facb001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a13b3a19e9dc8a71 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:35.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.953+0000 7facc6000640 1 -- 172.21.15.47:0/669373160 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb00ac410 con 0x7facc80cea80 2024-02-19T15:31:35.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.953+0000 7facc6000640 1 -- 172.21.15.47:0/669373160 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb00ac570 con 0x7facc80cea80 2024-02-19T15:31:35.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.953+0000 7facc6000640 1 -- 172.21.15.47:0/669373160 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb00ac8e0 con 0x7facc80cea80 2024-02-19T15:31:35.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.957+0000 7faccda00640 1 -- 172.21.15.47:0/669373160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc80cea80 msgr2=0x7facc80727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:35.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.957+0000 7faccda00640 1 --2- 172.21.15.47:0/669373160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc80cea80 0x7facc80727b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7facb00097c0 tx=0x7facb001c4a0 comp rx=0 tx=0).stop 2024-02-19T15:31:35.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.957+0000 7faccda00640 1 -- 172.21.15.47:0/669373160 shutdown_connections 2024-02-19T15:31:35.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.957+0000 7faccda00640 1 --2- 172.21.15.47:0/669373160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc80cea80 0x7facc80727b0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:35.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.957+0000 7faccda00640 1 -- 172.21.15.47:0/669373160 >> 172.21.15.47:0/669373160 conn(0x7facc80b16a0 msgr2=0x7facc80b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:35.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.957+0000 7faccda00640 1 -- 172.21.15.47:0/669373160 shutdown_connections 2024-02-19T15:31:35.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.957+0000 7faccda00640 1 -- 172.21.15.47:0/669373160 wait complete. 2024-02-19T15:31:35.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.957+0000 7faccda00640 1 Processor -- start 2024-02-19T15:31:35.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.957+0000 7faccda00640 1 -- start start 2024-02-19T15:31:35.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.957+0000 7faccda00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7facc80cea80 0x7facc813d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:35.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.957+0000 7faccda00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7facc8145590 0x7facc8156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:35.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.957+0000 7faccda00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc815c390 0x7facc80ad870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:35.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.957+0000 7faccda00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc8158130 con 0x7facc80cea80 2024-02-19T15:31:35.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.957+0000 7faccda00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc813d670 con 0x7facc815c390 2024-02-19T15:31:35.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.957+0000 7faccda00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc8155d40 con 0x7facc8145590 2024-02-19T15:31:35.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.957+0000 7facc6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7facc80cea80 0x7facc813d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:35.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.957+0000 7facc6a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7facc80cea80 0x7facc813d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:35658/0 (socket says 172.21.15.47:35658) 2024-02-19T15:31:35.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.957+0000 7facc6a00640 1 -- 172.21.15.47:0/3547713891 learned_addr learned my addr 172.21.15.47:0/3547713891 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:35.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.957+0000 7facc7400640 1 --2- 172.21.15.47:0/3547713891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc815c390 0x7facc80ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:35.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.957+0000 7facbe000640 1 --2- 172.21.15.47:0/3547713891 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7facc8145590 0x7facc8156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:35.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.961+0000 7facc6a00640 1 -- 172.21.15.47:0/3547713891 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7facc8145590 msgr2=0x7facc8156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:35.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.961+0000 7facc6a00640 1 --2- 172.21.15.47:0/3547713891 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7facc8145590 0x7facc8156b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:35.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.961+0000 7facc6a00640 1 -- 172.21.15.47:0/3547713891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc815c390 msgr2=0x7facc80ad870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:35.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.961+0000 7facc6a00640 1 --2- 172.21.15.47:0/3547713891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc815c390 0x7facc80ad870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:35.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.961+0000 7facc6a00640 1 -- 172.21.15.47:0/3547713891 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb0009420 con 0x7facc80cea80 2024-02-19T15:31:35.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.961+0000 7facc7400640 1 --2- 172.21.15.47:0/3547713891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc815c390 0x7facc80ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:31:35.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.961+0000 7facc6a00640 1 --2- 172.21.15.47:0/3547713891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7facc80cea80 0x7facc813d130 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7facb800c490 tx=0x7facb800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:35.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.961+0000 7facbf400640 1 -- 172.21.15.47:0/3547713891 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb80052d0 con 0x7facc80cea80 2024-02-19T15:31:35.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.961+0000 7facbf400640 1 -- 172.21.15.47:0/3547713891 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb8005430 con 0x7facc80cea80 2024-02-19T15:31:35.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.961+0000 7faccda00640 1 -- 172.21.15.47:0/3547713891 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc8156020 con 0x7facc80cea80 2024-02-19T15:31:35.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.961+0000 7facbf400640 1 -- 172.21.15.47:0/3547713891 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb8012870 con 0x7facc80cea80 2024-02-19T15:31:35.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.961+0000 7faccda00640 1 -- 172.21.15.47:0/3547713891 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc81565e0 con 0x7facc80cea80 2024-02-19T15:31:35.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.961+0000 7faccda00640 1 -- 172.21.15.47:0/3547713891 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc8075360 con 0x7facc80cea80 2024-02-19T15:31:35.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.961+0000 7facbf400640 1 -- 172.21.15.47:0/3547713891 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7facb80129d0 con 0x7facc80cea80 2024-02-19T15:31:35.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.961+0000 7facbf400640 1 --2- 172.21.15.47:0/3547713891 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faca4066e40 0x7faca4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:35.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.965+0000 7facbf400640 1 -- 172.21.15.47:0/3547713891 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7facb8084630 con 0x7facc80cea80 2024-02-19T15:31:35.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.965+0000 7facbe000640 1 --2- 172.21.15.47:0/3547713891 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faca4066e40 0x7faca4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:35.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.965+0000 7facbe000640 1 --2- 172.21.15.47:0/3547713891 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faca4066e40 0x7faca4069300 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7facb00ae000 tx=0x7facb00a6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:35.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:35.969+0000 7facbf400640 1 -- 172.21.15.47:0/3547713891 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7facb8051340 con 0x7facc80cea80 2024-02-19T15:31:36.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:36.165+0000 7faccda00640 1 -- 172.21.15.47:0/3547713891 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facc806f1f0 con 0x7facc80cea80 2024-02-19T15:31:36.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:36.165+0000 7facbf400640 1 -- 172.21.15.47:0/3547713891 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7facb8054a00 con 0x7facc80cea80 2024-02-19T15:31:36.164 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:36.164 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:36.164 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:36.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:36.165+0000 7faccda00640 1 -- 172.21.15.47:0/3547713891 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faca4066e40 msgr2=0x7faca4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:36.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:36.165+0000 7faccda00640 1 --2- 172.21.15.47:0/3547713891 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faca4066e40 0x7faca4069300 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7facb00ae000 tx=0x7facb00a6040 comp rx=0 tx=0).stop 2024-02-19T15:31:36.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:36.165+0000 7faccda00640 1 -- 172.21.15.47:0/3547713891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7facc80cea80 msgr2=0x7facc813d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:36.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:36.165+0000 7faccda00640 1 --2- 172.21.15.47:0/3547713891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7facc80cea80 0x7facc813d130 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7facb800c490 tx=0x7facb800c950 comp rx=0 tx=0).stop 2024-02-19T15:31:36.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:36.165+0000 7faccda00640 1 -- 172.21.15.47:0/3547713891 shutdown_connections 2024-02-19T15:31:36.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:36.165+0000 7faccda00640 1 --2- 172.21.15.47:0/3547713891 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7faca4066e40 0x7faca4069300 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:36.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:36.165+0000 7faccda00640 1 --2- 172.21.15.47:0/3547713891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc815c390 0x7facc80ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:36.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:36.165+0000 7faccda00640 1 --2- 172.21.15.47:0/3547713891 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7facc8145590 0x7facc8156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:36.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:36.165+0000 7faccda00640 1 --2- 172.21.15.47:0/3547713891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7facc80cea80 0x7facc813d130 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:36.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:36.165+0000 7faccda00640 1 -- 172.21.15.47:0/3547713891 >> 172.21.15.47:0/3547713891 conn(0x7facc80b16a0 msgr2=0x7facc806db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:36.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:36.165+0000 7faccda00640 1 -- 172.21.15.47:0/3547713891 shutdown_connections 2024-02-19T15:31:36.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:36.165+0000 7faccda00640 1 -- 172.21.15.47:0/3547713891 wait complete. 2024-02-19T15:31:36.268 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:36.275 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:31:36.279 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:31:36.396 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:36.396 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:37.396 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:37.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.537+0000 7f668ec00640 1 Processor -- start 2024-02-19T15:31:37.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.537+0000 7f668ec00640 1 -- start start 2024-02-19T15:31:37.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.537+0000 7f668ec00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f66881476e0 0x7f66880ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:37.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.537+0000 7f668ec00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f668806c9a0 0x7f66880b2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:37.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.537+0000 7f668ec00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66880aec70 0x7f66880b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:37.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.537+0000 7f668ec00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6688071530 con 0x7f66881476e0 2024-02-19T15:31:37.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.537+0000 7f668ec00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6688070980 con 0x7f668806c9a0 2024-02-19T15:31:37.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.537+0000 7f668ec00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66880af1b0 con 0x7f66880aec70 2024-02-19T15:31:37.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.537+0000 7f6687e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f66881476e0 0x7f66880ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:37.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.537+0000 7f6687e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f66881476e0 0x7f66880ae730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:35662/0 (socket says 172.21.15.47:35662) 2024-02-19T15:31:37.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.537+0000 7f6687e00640 1 -- 172.21.15.47:0/1306188888 learned_addr learned my addr 172.21.15.47:0/1306188888 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:37.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.541+0000 7f6687400640 1 --2- 172.21.15.47:0/1306188888 >> v2:172.21.15.116:3300/0 conn(0x7f668806c9a0 0x7f66880b2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:37.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.541+0000 7f668cc00640 1 --2- 172.21.15.47:0/1306188888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66880aec70 0x7f66880b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:37.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.541+0000 7f6687e00640 1 -- 172.21.15.47:0/1306188888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66880aec70 msgr2=0x7f66880b31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:37.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.541+0000 7f6687e00640 1 --2- 172.21.15.47:0/1306188888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66880aec70 0x7f66880b31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:37.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.541+0000 7f6687e00640 1 -- 172.21.15.47:0/1306188888 >> v2:172.21.15.116:3300/0 conn(0x7f668806c9a0 msgr2=0x7f66880b2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:37.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.541+0000 7f6687e00640 1 --2- 172.21.15.47:0/1306188888 >> v2:172.21.15.116:3300/0 conn(0x7f668806c9a0 0x7f66880b2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:37.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.541+0000 7f6687e00640 1 -- 172.21.15.47:0/1306188888 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6688070b90 con 0x7f66881476e0 2024-02-19T15:31:37.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.541+0000 7f668cc00640 1 --2- 172.21.15.47:0/1306188888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66880aec70 0x7f66880b31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:31:37.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.541+0000 7f6687e00640 1 --2- 172.21.15.47:0/1306188888 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66881476e0 0x7f66880ae730 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f6688070720 tx=0x7f667801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c52a36bb98bf3e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:37.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.541+0000 7f6686a00640 1 -- 172.21.15.47:0/1306188888 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66780aa610 con 0x7f66881476e0 2024-02-19T15:31:37.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.541+0000 7f6686a00640 1 -- 172.21.15.47:0/1306188888 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66780a6040 con 0x7f66881476e0 2024-02-19T15:31:37.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.541+0000 7f6686a00640 1 -- 172.21.15.47:0/1306188888 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66780aa990 con 0x7f66881476e0 2024-02-19T15:31:37.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.541+0000 7f668ec00640 1 -- 172.21.15.47:0/1306188888 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66881476e0 msgr2=0x7f66880ae730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:37.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.541+0000 7f668ec00640 1 --2- 172.21.15.47:0/1306188888 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66881476e0 0x7f66880ae730 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f6688070720 tx=0x7f667801c4f0 comp rx=0 tx=0).stop 2024-02-19T15:31:37.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.541+0000 7f668ec00640 1 -- 172.21.15.47:0/1306188888 shutdown_connections 2024-02-19T15:31:37.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.541+0000 7f668ec00640 1 --2- 172.21.15.47:0/1306188888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66880aec70 0x7f66880b31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:37.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.541+0000 7f668ec00640 1 --2- 172.21.15.47:0/1306188888 >> v2:172.21.15.116:3300/0 conn(0x7f668806c9a0 0x7f66880b2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:37.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.541+0000 7f668ec00640 1 --2- 172.21.15.47:0/1306188888 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66881476e0 0x7f66880ae730 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:37.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.541+0000 7f668ec00640 1 -- 172.21.15.47:0/1306188888 >> 172.21.15.47:0/1306188888 conn(0x7f66880b16a0 msgr2=0x7f66880b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:37.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f668ec00640 1 -- 172.21.15.47:0/1306188888 shutdown_connections 2024-02-19T15:31:37.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f668ec00640 1 -- 172.21.15.47:0/1306188888 wait complete. 2024-02-19T15:31:37.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f668ec00640 1 Processor -- start 2024-02-19T15:31:37.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f668ec00640 1 -- start start 2024-02-19T15:31:37.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f668ec00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f668806c9a0 0x7f6688160990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:37.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f668ec00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66880aec70 0x7f668815d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:37.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f668ec00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66881476e0 0x7f66880cde80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:37.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f668ec00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668815d6d0 con 0x7f66881476e0 2024-02-19T15:31:37.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f668ec00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668815f5f0 con 0x7f66880aec70 2024-02-19T15:31:37.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f668ec00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f668815e550 con 0x7f668806c9a0 2024-02-19T15:31:37.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f668cc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66881476e0 0x7f66880cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:37.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f668cc00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66881476e0 0x7f66880cde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:35666/0 (socket says 172.21.15.47:35666) 2024-02-19T15:31:37.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f668cc00640 1 -- 172.21.15.47:0/18809711 learned_addr learned my addr 172.21.15.47:0/18809711 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:37.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f6687400640 1 --2- 172.21.15.47:0/18809711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66880aec70 0x7f668815d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:37.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f6687e00640 1 --2- 172.21.15.47:0/18809711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f668806c9a0 0x7f6688160990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:37.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f668cc00640 1 -- 172.21.15.47:0/18809711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f668806c9a0 msgr2=0x7f6688160990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:37.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f668cc00640 1 --2- 172.21.15.47:0/18809711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f668806c9a0 0x7f6688160990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:37.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f668cc00640 1 -- 172.21.15.47:0/18809711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66880aec70 msgr2=0x7f668815d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:37.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f668cc00640 1 --2- 172.21.15.47:0/18809711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66880aec70 0x7f668815d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:37.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f668cc00640 1 -- 172.21.15.47:0/18809711 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66780094d0 con 0x7f66881476e0 2024-02-19T15:31:37.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f668cc00640 1 --2- 172.21.15.47:0/18809711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66881476e0 0x7f66880cde80 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f668000ce70 tx=0x7f668000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:37.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f667fe00640 1 -- 172.21.15.47:0/18809711 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6680003ba0 con 0x7f66881476e0 2024-02-19T15:31:37.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f667fe00640 1 -- 172.21.15.47:0/18809711 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6680003d20 con 0x7f66881476e0 2024-02-19T15:31:37.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f667fe00640 1 -- 172.21.15.47:0/18809711 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668000d880 con 0x7f66881476e0 2024-02-19T15:31:37.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.545+0000 7f668ec00640 1 -- 172.21.15.47:0/18809711 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f668815e830 con 0x7f66881476e0 2024-02-19T15:31:37.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.549+0000 7f668ec00640 1 -- 172.21.15.47:0/18809711 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f668815edf0 con 0x7f66881476e0 2024-02-19T15:31:37.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.549+0000 7f668ec00640 1 -- 172.21.15.47:0/18809711 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6688075360 con 0x7f66881476e0 2024-02-19T15:31:37.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.549+0000 7f667fe00640 1 -- 172.21.15.47:0/18809711 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f668000d9e0 con 0x7f66881476e0 2024-02-19T15:31:37.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.549+0000 7f667fe00640 1 --2- 172.21.15.47:0/18809711 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6668066e40 0x7f6668069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:37.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.549+0000 7f667fe00640 1 -- 172.21.15.47:0/18809711 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6680085330 con 0x7f66881476e0 2024-02-19T15:31:37.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.549+0000 7f6687e00640 1 --2- 172.21.15.47:0/18809711 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6668066e40 0x7f6668069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:37.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.549+0000 7f6687e00640 1 --2- 172.21.15.47:0/18809711 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6668066e40 0x7f6668069300 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f66780ae000 tx=0x7f66780b5040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:37.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.557+0000 7f667fe00640 1 -- 172.21.15.47:0/18809711 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6680052110 con 0x7f66881476e0 2024-02-19T15:31:37.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.753+0000 7f668ec00640 1 -- 172.21.15.47:0/18809711 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66880cf3f0 con 0x7f66881476e0 2024-02-19T15:31:37.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.753+0000 7f667fe00640 1 -- 172.21.15.47:0/18809711 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f66800557d0 con 0x7f66881476e0 2024-02-19T15:31:37.753 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:37.753 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:37.753 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:37.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.757+0000 7f668ec00640 1 -- 172.21.15.47:0/18809711 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6668066e40 msgr2=0x7f6668069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:37.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.757+0000 7f668ec00640 1 --2- 172.21.15.47:0/18809711 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6668066e40 0x7f6668069300 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f66780ae000 tx=0x7f66780b5040 comp rx=0 tx=0).stop 2024-02-19T15:31:37.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.757+0000 7f668ec00640 1 -- 172.21.15.47:0/18809711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66881476e0 msgr2=0x7f66880cde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:37.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.757+0000 7f668ec00640 1 --2- 172.21.15.47:0/18809711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66881476e0 0x7f66880cde80 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f668000ce70 tx=0x7f668000a710 comp rx=0 tx=0).stop 2024-02-19T15:31:37.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.757+0000 7f668ec00640 1 -- 172.21.15.47:0/18809711 shutdown_connections 2024-02-19T15:31:37.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.757+0000 7f668ec00640 1 --2- 172.21.15.47:0/18809711 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6668066e40 0x7f6668069300 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:37.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.757+0000 7f668ec00640 1 --2- 172.21.15.47:0/18809711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66881476e0 0x7f66880cde80 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:37.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.757+0000 7f668ec00640 1 --2- 172.21.15.47:0/18809711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66880aec70 0x7f668815d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:37.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.757+0000 7f668ec00640 1 --2- 172.21.15.47:0/18809711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f668806c9a0 0x7f6688160990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:37.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.757+0000 7f668ec00640 1 -- 172.21.15.47:0/18809711 >> 172.21.15.47:0/18809711 conn(0x7f66880b16a0 msgr2=0x7f66880afde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:37.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.757+0000 7f668ec00640 1 -- 172.21.15.47:0/18809711 shutdown_connections 2024-02-19T15:31:37.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:37.757+0000 7f668ec00640 1 -- 172.21.15.47:0/18809711 wait complete. 2024-02-19T15:31:37.975 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:37.975 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:38.270 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:38.976 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:39.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.117+0000 7f497a600640 1 Processor -- start 2024-02-19T15:31:39.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.117+0000 7f497a600640 1 -- start start 2024-02-19T15:31:39.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.121+0000 7f497a600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f497415df50 0x7f49740d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:39.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.121+0000 7f497a600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49740d2da0 con 0x7f49740d0f90 2024-02-19T15:31:39.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.121+0000 7f497a600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49740b6df0 con 0x7f49740d4a70 2024-02-19T15:31:39.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.121+0000 7f497a600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49740b70d0 con 0x7f497415df50 2024-02-19T15:31:39.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.121+0000 7f4973400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f497415df50 0x7f49740d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:39.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.121+0000 7f4973e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f49740d0f90 0x7f4974167d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45098/0 (socket says 172.21.15.47:45098) 2024-02-19T15:31:39.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.121+0000 7f4973400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f497415df50 0x7f49740d2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:35678/0 (socket says 172.21.15.47:35678) 2024-02-19T15:31:39.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.121+0000 7f4973e00640 1 -- 172.21.15.47:0/353428891 learned_addr learned my addr 172.21.15.47:0/353428891 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:31:39.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.121+0000 7f4973400640 1 -- 172.21.15.47:0/353428891 >> v1:172.21.15.116:6789/0 conn(0x7f49740d4a70 legacy=0x7f49740d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:39.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.121+0000 7f4973400640 1 -- 172.21.15.47:0/353428891 >> v1:172.21.15.47:6789/0 conn(0x7f49740d0f90 legacy=0x7f4974167d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:39.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.121+0000 7f4973400640 1 -- 172.21.15.47:0/353428891 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49740b73b0 con 0x7f497415df50 2024-02-19T15:31:39.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.125+0000 7f4973400640 1 --2- 172.21.15.47:0/353428891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f497415df50 0x7f49740d2450 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f49640097c0 tx=0x7f496401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f282a070f23c0f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:39.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.125+0000 7f4972000640 1 -- 172.21.15.47:0/353428891 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49640ab600 con 0x7f497415df50 2024-02-19T15:31:39.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.125+0000 7f4972000640 1 -- 172.21.15.47:0/353428891 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49640a7050 con 0x7f497415df50 2024-02-19T15:31:39.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.125+0000 7f4972000640 1 -- 172.21.15.47:0/353428891 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49640ab980 con 0x7f497415df50 2024-02-19T15:31:39.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.125+0000 7f497a600640 1 -- 172.21.15.47:0/353428891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f497415df50 msgr2=0x7f49740d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:39.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.125+0000 7f497a600640 1 --2- 172.21.15.47:0/353428891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f497415df50 0x7f49740d2450 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f49640097c0 tx=0x7f496401c430 comp rx=0 tx=0).stop 2024-02-19T15:31:39.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.125+0000 7f497a600640 1 -- 172.21.15.47:0/353428891 shutdown_connections 2024-02-19T15:31:39.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.125+0000 7f497a600640 1 --2- 172.21.15.47:0/353428891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f497415df50 0x7f49740d2450 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:39.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.125+0000 7f497a600640 1 -- 172.21.15.47:0/353428891 >> 172.21.15.47:0/353428891 conn(0x7f49740d3e70 msgr2=0x7f49740d4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:39.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.125+0000 7f497a600640 1 -- 172.21.15.47:0/353428891 shutdown_connections 2024-02-19T15:31:39.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.125+0000 7f497a600640 1 -- 172.21.15.47:0/353428891 wait complete. 2024-02-19T15:31:39.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.125+0000 7f497a600640 1 Processor -- start 2024-02-19T15:31:39.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.125+0000 7f497a600640 1 -- start start 2024-02-19T15:31:39.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.125+0000 7f497a600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49740d0f90 0x7f4974154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:39.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f497a600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49740d4a70 0x7f4974155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:39.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f497a600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f497415df50 0x7f4974156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:39.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f497a600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4974157670 con 0x7f49740d4a70 2024-02-19T15:31:39.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f497a600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4974156d10 con 0x7f49740d0f90 2024-02-19T15:31:39.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f497a600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4974156fb0 con 0x7f497415df50 2024-02-19T15:31:39.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f4972a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49740d4a70 0x7f4974155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:39.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f4973400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49740d0f90 0x7f4974154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:39.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f4973400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49740d0f90 0x7f4974154a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:52570/0 (socket says 172.21.15.47:52570) 2024-02-19T15:31:39.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f4973400640 1 -- 172.21.15.47:0/3614458201 learned_addr learned my addr 172.21.15.47:0/3614458201 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:39.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f4973e00640 1 --2- 172.21.15.47:0/3614458201 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f497415df50 0x7f4974156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:39.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f4973400640 1 -- 172.21.15.47:0/3614458201 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f497415df50 msgr2=0x7f4974156080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:39.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f4973400640 1 --2- 172.21.15.47:0/3614458201 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f497415df50 0x7f4974156080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:39.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f4973400640 1 -- 172.21.15.47:0/3614458201 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49740d4a70 msgr2=0x7f4974155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:39.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f4973400640 1 --2- 172.21.15.47:0/3614458201 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49740d4a70 0x7f4974155b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:39.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f4973400640 1 -- 172.21.15.47:0/3614458201 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4964009420 con 0x7f49740d0f90 2024-02-19T15:31:39.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f4972a00640 1 --2- 172.21.15.47:0/3614458201 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49740d4a70 0x7f4974155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:31:39.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f4973400640 1 --2- 172.21.15.47:0/3614458201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49740d0f90 0x7f4974154a70 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4964019040 tx=0x7f496401ac70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:39.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f496b400640 1 -- 172.21.15.47:0/3614458201 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49640b4870 con 0x7f49740d0f90 2024-02-19T15:31:39.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f496b400640 1 -- 172.21.15.47:0/3614458201 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49640b49f0 con 0x7f49740d0f90 2024-02-19T15:31:39.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f497a600640 1 -- 172.21.15.47:0/3614458201 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4974157290 con 0x7f49740d0f90 2024-02-19T15:31:39.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f496b400640 1 -- 172.21.15.47:0/3614458201 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49640b4da0 con 0x7f49740d0f90 2024-02-19T15:31:39.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f497a600640 1 -- 172.21.15.47:0/3614458201 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4974154fb0 con 0x7f49740d0f90 2024-02-19T15:31:39.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.129+0000 7f497a600640 1 -- 172.21.15.47:0/3614458201 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49741577b0 con 0x7f49740d0f90 2024-02-19T15:31:39.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.133+0000 7f496b400640 1 -- 172.21.15.47:0/3614458201 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f49640b2070 con 0x7f49740d0f90 2024-02-19T15:31:39.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.133+0000 7f496b400640 1 --2- 172.21.15.47:0/3614458201 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4954066e40 0x7f4954069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:39.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.133+0000 7f496b400640 1 -- 172.21.15.47:0/3614458201 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f496411f740 con 0x7f49740d0f90 2024-02-19T15:31:39.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.133+0000 7f4972a00640 1 --2- 172.21.15.47:0/3614458201 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4954066e40 0x7f4954069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:39.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.133+0000 7f4972a00640 1 --2- 172.21.15.47:0/3614458201 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4954066e40 0x7f4954069300 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f49741596f0 tx=0x7f4960007e40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:39.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.137+0000 7f496b400640 1 -- 172.21.15.47:0/3614458201 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f49640ec610 con 0x7f49740d0f90 2024-02-19T15:31:39.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.329+0000 7f497a600640 1 -- 172.21.15.47:0/3614458201 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4974075360 con 0x7f49740d0f90 2024-02-19T15:31:39.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.329+0000 7f496b400640 1 -- 172.21.15.47:0/3614458201 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f49640b0090 con 0x7f49740d0f90 2024-02-19T15:31:39.330 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:39.330 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:39.331 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:39.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.333+0000 7f497a600640 1 -- 172.21.15.47:0/3614458201 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4954066e40 msgr2=0x7f4954069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:39.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.333+0000 7f497a600640 1 --2- 172.21.15.47:0/3614458201 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4954066e40 0x7f4954069300 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f49741596f0 tx=0x7f4960007e40 comp rx=0 tx=0).stop 2024-02-19T15:31:39.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.333+0000 7f497a600640 1 -- 172.21.15.47:0/3614458201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49740d0f90 msgr2=0x7f4974154a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:39.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.333+0000 7f497a600640 1 --2- 172.21.15.47:0/3614458201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49740d0f90 0x7f4974154a70 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4964019040 tx=0x7f496401ac70 comp rx=0 tx=0).stop 2024-02-19T15:31:39.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.333+0000 7f497a600640 1 -- 172.21.15.47:0/3614458201 shutdown_connections 2024-02-19T15:31:39.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.333+0000 7f497a600640 1 --2- 172.21.15.47:0/3614458201 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4954066e40 0x7f4954069300 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:39.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.333+0000 7f497a600640 1 --2- 172.21.15.47:0/3614458201 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f497415df50 0x7f4974156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:39.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.333+0000 7f497a600640 1 --2- 172.21.15.47:0/3614458201 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49740d4a70 0x7f4974155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:39.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.333+0000 7f497a600640 1 --2- 172.21.15.47:0/3614458201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49740d0f90 0x7f4974154a70 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:39.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.333+0000 7f497a600640 1 -- 172.21.15.47:0/3614458201 >> 172.21.15.47:0/3614458201 conn(0x7f49740d3e70 msgr2=0x7f497406caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:39.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.333+0000 7f497a600640 1 -- 172.21.15.47:0/3614458201 shutdown_connections 2024-02-19T15:31:39.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:39.333+0000 7f497a600640 1 -- 172.21.15.47:0/3614458201 wait complete. 2024-02-19T15:31:39.563 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:39.563 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:40.270 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:40.564 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:40.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.705+0000 7fa9b5800640 1 Processor -- start 2024-02-19T15:31:40.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.705+0000 7fa9b5800640 1 -- start start 2024-02-19T15:31:40.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.705+0000 7fa9b5800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b00d0360 0x7fa9b015d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:40.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.705+0000 7fa9b5800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b00cc700 con 0x7fa9b015bcd0 2024-02-19T15:31:40.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.705+0000 7fa9b5800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b00cf300 con 0x7fa9b00cedb0 2024-02-19T15:31:40.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.705+0000 7fa9b5800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b00cf5e0 con 0x7fa9b00d0360 2024-02-19T15:31:40.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.709+0000 7fa9aea00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa9b015bcd0 0x7fa9b00cd480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45110/0 (socket says 172.21.15.47:45110) 2024-02-19T15:31:40.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.709+0000 7fa9aea00640 1 -- 172.21.15.47:0/2605448572 learned_addr learned my addr 172.21.15.47:0/2605448572 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:31:40.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.709+0000 7fa9af400640 1 --2- 172.21.15.47:0/2605448572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b00d0360 0x7fa9b015d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:40.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.709+0000 7fa9ae000640 1 -- 172.21.15.47:0/2605448572 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3738846188 0 0) 0x7fa9b00cc700 con 0x7fa9b015bcd0 2024-02-19T15:31:40.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.709+0000 7fa9ae000640 1 -- 172.21.15.47:0/2605448572 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa994003530 con 0x7fa9b015bcd0 2024-02-19T15:31:40.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.709+0000 7fa9ae000640 1 -- 172.21.15.47:0/2605448572 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985178323 0 0) 0x7fa9b00cf300 con 0x7fa9b00cedb0 2024-02-19T15:31:40.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.709+0000 7fa9ae000640 1 -- 172.21.15.47:0/2605448572 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b00cc700 con 0x7fa9b00cedb0 2024-02-19T15:31:40.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.709+0000 7fa9af400640 1 -- 172.21.15.47:0/2605448572 >> v1:172.21.15.116:6789/0 conn(0x7fa9b00cedb0 legacy=0x7fa9b00cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:40.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.709+0000 7fa9af400640 1 -- 172.21.15.47:0/2605448572 >> v1:172.21.15.47:6789/0 conn(0x7fa9b015bcd0 legacy=0x7fa9b00cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:40.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.709+0000 7fa9af400640 1 -- 172.21.15.47:0/2605448572 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b00cc9a0 con 0x7fa9b00d0360 2024-02-19T15:31:40.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.709+0000 7fa9af400640 1 --2- 172.21.15.47:0/2605448572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b00d0360 0x7fa9b015d4f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8009dc0 tx=0x7fa9a801c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f637fb5651cfb933 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:40.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.709+0000 7fa9ae000640 1 -- 172.21.15.47:0/2605448572 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a80a5a20 con 0x7fa9b00d0360 2024-02-19T15:31:40.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.713+0000 7fa9ae000640 1 -- 172.21.15.47:0/2605448572 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a80a5b80 con 0x7fa9b00d0360 2024-02-19T15:31:40.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.713+0000 7fa9ae000640 1 -- 172.21.15.47:0/2605448572 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a80ad440 con 0x7fa9b00d0360 2024-02-19T15:31:40.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.713+0000 7fa9b5800640 1 -- 172.21.15.47:0/2605448572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b00d0360 msgr2=0x7fa9b015d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:40.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.713+0000 7fa9b5800640 1 --2- 172.21.15.47:0/2605448572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b00d0360 0x7fa9b015d4f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8009dc0 tx=0x7fa9a801c960 comp rx=0 tx=0).stop 2024-02-19T15:31:40.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.713+0000 7fa9b5800640 1 -- 172.21.15.47:0/2605448572 shutdown_connections 2024-02-19T15:31:40.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.713+0000 7fa9b5800640 1 --2- 172.21.15.47:0/2605448572 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b00d0360 0x7fa9b015d4f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:40.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.713+0000 7fa9b5800640 1 -- 172.21.15.47:0/2605448572 >> 172.21.15.47:0/2605448572 conn(0x7fa9b00d1bf0 msgr2=0x7fa9b00d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:40.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.713+0000 7fa9b5800640 1 -- 172.21.15.47:0/2605448572 shutdown_connections 2024-02-19T15:31:40.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.713+0000 7fa9b5800640 1 -- 172.21.15.47:0/2605448572 wait complete. 2024-02-19T15:31:40.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.713+0000 7fa9b5800640 1 Processor -- start 2024-02-19T15:31:40.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.713+0000 7fa9b5800640 1 -- start start 2024-02-19T15:31:40.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9b5800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00cedb0 0x7fa9b0153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:40.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9b5800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9b00d0360 0x7fa9b00cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:40.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9b5800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b015bcd0 0x7fa9b00cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:40.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9b5800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b00cdb90 con 0x7fa9b00cedb0 2024-02-19T15:31:40.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9b5800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b01549d0 con 0x7fa9b00d0360 2024-02-19T15:31:40.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9b5800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b00c5660 con 0x7fa9b015bcd0 2024-02-19T15:31:40.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9aea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00cedb0 0x7fa9b0153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:40.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9aea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00cedb0 0x7fa9b0153ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:35694/0 (socket says 172.21.15.47:35694) 2024-02-19T15:31:40.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9aea00640 1 -- 172.21.15.47:0/3524848228 learned_addr learned my addr 172.21.15.47:0/3524848228 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:40.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9a7e00640 1 --2- 172.21.15.47:0/3524848228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9b00d0360 0x7fa9b00cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:40.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9af400640 1 --2- 172.21.15.47:0/3524848228 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b015bcd0 0x7fa9b00cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:40.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9aea00640 1 -- 172.21.15.47:0/3524848228 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b015bcd0 msgr2=0x7fa9b00cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:40.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9aea00640 1 --2- 172.21.15.47:0/3524848228 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b015bcd0 0x7fa9b00cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:40.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9aea00640 1 -- 172.21.15.47:0/3524848228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9b00d0360 msgr2=0x7fa9b00cb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:40.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9aea00640 1 --2- 172.21.15.47:0/3524848228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9b00d0360 0x7fa9b00cb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:40.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9aea00640 1 -- 172.21.15.47:0/3524848228 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a8009a70 con 0x7fa9b00cedb0 2024-02-19T15:31:40.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9aea00640 1 --2- 172.21.15.47:0/3524848228 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00cedb0 0x7fa9b0153ee0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000c4d0 tx=0x7fa9a000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:40.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9a6a00640 1 -- 172.21.15.47:0/3524848228 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0005cf0 con 0x7fa9b00cedb0 2024-02-19T15:31:40.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9a6a00640 1 -- 172.21.15.47:0/3524848228 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a0005e50 con 0x7fa9b00cedb0 2024-02-19T15:31:40.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9b5800640 1 -- 172.21.15.47:0/3524848228 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b015d240 con 0x7fa9b00cedb0 2024-02-19T15:31:40.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9a6a00640 1 -- 172.21.15.47:0/3524848228 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a00072b0 con 0x7fa9b00cedb0 2024-02-19T15:31:40.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9b5800640 1 -- 172.21.15.47:0/3524848228 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b00c5c20 con 0x7fa9b00cedb0 2024-02-19T15:31:40.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.717+0000 7fa9b5800640 1 -- 172.21.15.47:0/3524848228 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b0075360 con 0x7fa9b00cedb0 2024-02-19T15:31:40.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.721+0000 7fa9a6a00640 1 -- 172.21.15.47:0/3524848228 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fa9a0007410 con 0x7fa9b00cedb0 2024-02-19T15:31:40.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.721+0000 7fa9a6a00640 1 --2- 172.21.15.47:0/3524848228 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa98c066e40 0x7fa98c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:40.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.721+0000 7fa9a6a00640 1 -- 172.21.15.47:0/3524848228 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa9a0083fe0 con 0x7fa9b00cedb0 2024-02-19T15:31:40.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.721+0000 7fa9a7e00640 1 --2- 172.21.15.47:0/3524848228 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa98c066e40 0x7fa98c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:40.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.721+0000 7fa9a7e00640 1 --2- 172.21.15.47:0/3524848228 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa98c066e40 0x7fa98c069300 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fa9980011f0 tx=0x7fa998009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:40.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.725+0000 7fa9a6a00640 1 -- 172.21.15.47:0/3524848228 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fa9a0050e70 con 0x7fa9b00cedb0 2024-02-19T15:31:40.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.921+0000 7fa9b5800640 1 -- 172.21.15.47:0/3524848228 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9b0069020 con 0x7fa9b00cedb0 2024-02-19T15:31:40.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.921+0000 7fa9a6a00640 1 -- 172.21.15.47:0/3524848228 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7fa9a0017090 con 0x7fa9b00cedb0 2024-02-19T15:31:40.922 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:40.922 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:40.922 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:40.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.925+0000 7fa9b5800640 1 -- 172.21.15.47:0/3524848228 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa98c066e40 msgr2=0x7fa98c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:40.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.925+0000 7fa9b5800640 1 --2- 172.21.15.47:0/3524848228 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa98c066e40 0x7fa98c069300 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fa9980011f0 tx=0x7fa998009040 comp rx=0 tx=0).stop 2024-02-19T15:31:40.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.925+0000 7fa9b5800640 1 -- 172.21.15.47:0/3524848228 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00cedb0 msgr2=0x7fa9b0153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:40.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.925+0000 7fa9b5800640 1 --2- 172.21.15.47:0/3524848228 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00cedb0 0x7fa9b0153ee0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000c4d0 tx=0x7fa9a000c990 comp rx=0 tx=0).stop 2024-02-19T15:31:40.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.925+0000 7fa9b5800640 1 -- 172.21.15.47:0/3524848228 shutdown_connections 2024-02-19T15:31:40.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.925+0000 7fa9b5800640 1 --2- 172.21.15.47:0/3524848228 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fa98c066e40 0x7fa98c069300 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:40.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.925+0000 7fa9b5800640 1 --2- 172.21.15.47:0/3524848228 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa9b015bcd0 0x7fa9b00cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:40.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.925+0000 7fa9b5800640 1 --2- 172.21.15.47:0/3524848228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9b00d0360 0x7fa9b00cb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:40.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.925+0000 7fa9b5800640 1 --2- 172.21.15.47:0/3524848228 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9b00cedb0 0x7fa9b0153ee0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:40.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.925+0000 7fa9b5800640 1 -- 172.21.15.47:0/3524848228 >> 172.21.15.47:0/3524848228 conn(0x7fa9b00d1bf0 msgr2=0x7fa9b015c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:40.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.925+0000 7fa9b5800640 1 -- 172.21.15.47:0/3524848228 shutdown_connections 2024-02-19T15:31:40.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:40.925+0000 7fa9b5800640 1 -- 172.21.15.47:0/3524848228 wait complete. 2024-02-19T15:31:41.145 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:41.145 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:42.146 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:42.272 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:42.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.289+0000 7fe9f8000640 1 Processor -- start 2024-02-19T15:31:42.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.289+0000 7fe9f8000640 1 -- start start 2024-02-19T15:31:42.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.289+0000 7fe9f8000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe9f015bcd0 0x7fe9f00cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:42.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.289+0000 7fe9f8000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9f00d0360 0x7fe9f015d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:42.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.289+0000 7fe9f8000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f00cc700 con 0x7fe9f00cedb0 2024-02-19T15:31:42.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.289+0000 7fe9f8000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9f00cf5e0 con 0x7fe9f015bcd0 2024-02-19T15:31:42.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.289+0000 7fe9f8000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f00cf300 con 0x7fe9f00d0360 2024-02-19T15:31:42.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.289+0000 7fe9f5600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe9f015bcd0 0x7fe9f00cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:42.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.289+0000 7fe9f5600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe9f015bcd0 0x7fe9f00cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:35706/0 (socket says 172.21.15.47:35706) 2024-02-19T15:31:42.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.289+0000 7fe9f5600640 1 -- 172.21.15.47:0/2992312040 learned_addr learned my addr 172.21.15.47:0/2992312040 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:42.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.289+0000 7fe9f6000640 1 --2- 172.21.15.47:0/2992312040 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9f00d0360 0x7fe9f015d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:42.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.289+0000 7fe9f5600640 1 -- 172.21.15.47:0/2992312040 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9f00d0360 msgr2=0x7fe9f015d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:42.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.289+0000 7fe9f5600640 1 --2- 172.21.15.47:0/2992312040 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9f00d0360 0x7fe9f015d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:42.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.289+0000 7fe9f5600640 1 -- 172.21.15.47:0/2992312040 >> v1:172.21.15.47:6789/0 conn(0x7fe9f00cedb0 legacy=0x7fe9f00cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:42.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.289+0000 7fe9f5600640 1 -- 172.21.15.47:0/2992312040 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f00cc9a0 con 0x7fe9f015bcd0 2024-02-19T15:31:42.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.293+0000 7fe9f5600640 1 --2- 172.21.15.47:0/2992312040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9f015bcd0 0x7fe9f00cd480 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4009870 tx=0x7fe9e401c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c903d93793478f39 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:42.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.293+0000 7fe9efe00640 1 -- 172.21.15.47:0/2992312040 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e40a95f0 con 0x7fe9f015bcd0 2024-02-19T15:31:42.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.293+0000 7fe9efe00640 1 -- 172.21.15.47:0/2992312040 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e40a9750 con 0x7fe9f015bcd0 2024-02-19T15:31:42.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.293+0000 7fe9efe00640 1 -- 172.21.15.47:0/2992312040 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e40a9ae0 con 0x7fe9f015bcd0 2024-02-19T15:31:42.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.293+0000 7fe9f8000640 1 -- 172.21.15.47:0/2992312040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9f015bcd0 msgr2=0x7fe9f00cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:42.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.293+0000 7fe9f8000640 1 --2- 172.21.15.47:0/2992312040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9f015bcd0 0x7fe9f00cd480 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4009870 tx=0x7fe9e401c530 comp rx=0 tx=0).stop 2024-02-19T15:31:42.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.293+0000 7fe9f8000640 1 -- 172.21.15.47:0/2992312040 shutdown_connections 2024-02-19T15:31:42.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.293+0000 7fe9f8000640 1 --2- 172.21.15.47:0/2992312040 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9f00d0360 0x7fe9f015d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:42.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.293+0000 7fe9f8000640 1 --2- 172.21.15.47:0/2992312040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9f015bcd0 0x7fe9f00cd480 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:42.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.293+0000 7fe9f8000640 1 -- 172.21.15.47:0/2992312040 >> 172.21.15.47:0/2992312040 conn(0x7fe9f00d1bf0 msgr2=0x7fe9f00d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:42.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.293+0000 7fe9f8000640 1 -- 172.21.15.47:0/2992312040 shutdown_connections 2024-02-19T15:31:42.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.293+0000 7fe9f8000640 1 -- 172.21.15.47:0/2992312040 wait complete. 2024-02-19T15:31:42.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.293+0000 7fe9f8000640 1 Processor -- start 2024-02-19T15:31:42.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f8000640 1 -- start start 2024-02-19T15:31:42.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f8000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9f00cedb0 0x7fe9f00c5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:42.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f8000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9f00d0360 0x7fe9f00c69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:42.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f8000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9f015bcd0 0x7fe9f00c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:42.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f8000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f00c82c0 con 0x7fe9f015bcd0 2024-02-19T15:31:42.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f8000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f0159b00 con 0x7fe9f00d0360 2024-02-19T15:31:42.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f8000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f00c5980 con 0x7fe9f00cedb0 2024-02-19T15:31:42.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f6000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9f015bcd0 0x7fe9f00c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:42.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f6000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9f015bcd0 0x7fe9f00c6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:35708/0 (socket says 172.21.15.47:35708) 2024-02-19T15:31:42.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f6000640 1 -- 172.21.15.47:0/2201199279 learned_addr learned my addr 172.21.15.47:0/2201199279 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:42.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f5600640 1 --2- 172.21.15.47:0/2201199279 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9f00cedb0 0x7fe9f00c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:42.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f4c00640 1 --2- 172.21.15.47:0/2201199279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9f00d0360 0x7fe9f00c69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:42.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f6000640 1 -- 172.21.15.47:0/2201199279 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9f00cedb0 msgr2=0x7fe9f00c5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:42.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f6000640 1 --2- 172.21.15.47:0/2201199279 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9f00cedb0 0x7fe9f00c5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:42.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f6000640 1 -- 172.21.15.47:0/2201199279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9f00d0360 msgr2=0x7fe9f00c69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:42.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f6000640 1 --2- 172.21.15.47:0/2201199279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9f00d0360 0x7fe9f00c69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:42.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f6000640 1 -- 172.21.15.47:0/2201199279 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e40094d0 con 0x7fe9f015bcd0 2024-02-19T15:31:42.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f5600640 1 --2- 172.21.15.47:0/2201199279 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9f00cedb0 0x7fe9f00c5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:31:42.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f6000640 1 --2- 172.21.15.47:0/2201199279 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9f015bcd0 0x7fe9f00c6ef0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8007d40 tx=0x7fe9e800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:42.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9ed600640 1 -- 172.21.15.47:0/2201199279 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e8014420 con 0x7fe9f015bcd0 2024-02-19T15:31:42.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9ed600640 1 -- 172.21.15.47:0/2201199279 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e80145a0 con 0x7fe9f015bcd0 2024-02-19T15:31:42.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9ed600640 1 -- 172.21.15.47:0/2201199279 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e8014950 con 0x7fe9f015bcd0 2024-02-19T15:31:42.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f8000640 1 -- 172.21.15.47:0/2201199279 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f00cdb40 con 0x7fe9f015bcd0 2024-02-19T15:31:42.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f8000640 1 -- 172.21.15.47:0/2201199279 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f00c5f40 con 0x7fe9f015bcd0 2024-02-19T15:31:42.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.297+0000 7fe9f8000640 1 -- 172.21.15.47:0/2201199279 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f0075360 con 0x7fe9f015bcd0 2024-02-19T15:31:42.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.301+0000 7fe9ed600640 1 -- 172.21.15.47:0/2201199279 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe9e8014ab0 con 0x7fe9f015bcd0 2024-02-19T15:31:42.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.301+0000 7fe9ed600640 1 --2- 172.21.15.47:0/2201199279 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe9d0066e40 0x7fe9d0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:42.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.301+0000 7fe9ed600640 1 -- 172.21.15.47:0/2201199279 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe9e8085350 con 0x7fe9f015bcd0 2024-02-19T15:31:42.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.301+0000 7fe9f5600640 1 --2- 172.21.15.47:0/2201199279 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe9d0066e40 0x7fe9d0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:42.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.301+0000 7fe9f5600640 1 --2- 172.21.15.47:0/2201199279 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe9d0066e40 0x7fe9d0069300 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fe9e40ae000 tx=0x7fe9e40b3040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:42.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.309+0000 7fe9ed600640 1 -- 172.21.15.47:0/2201199279 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe9e8052100 con 0x7fe9f015bcd0 2024-02-19T15:31:42.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.505+0000 7fe9f8000640 1 -- 172.21.15.47:0/2201199279 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9f00b30c0 con 0x7fe9f015bcd0 2024-02-19T15:31:42.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.505+0000 7fe9ed600640 1 -- 172.21.15.47:0/2201199279 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7fe9e80557c0 con 0x7fe9f015bcd0 2024-02-19T15:31:42.505 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:42.505 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:42.505 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:42.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.509+0000 7fe9f8000640 1 -- 172.21.15.47:0/2201199279 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe9d0066e40 msgr2=0x7fe9d0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:42.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.509+0000 7fe9f8000640 1 --2- 172.21.15.47:0/2201199279 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe9d0066e40 0x7fe9d0069300 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fe9e40ae000 tx=0x7fe9e40b3040 comp rx=0 tx=0).stop 2024-02-19T15:31:42.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.509+0000 7fe9f8000640 1 -- 172.21.15.47:0/2201199279 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9f015bcd0 msgr2=0x7fe9f00c6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:42.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.509+0000 7fe9f8000640 1 --2- 172.21.15.47:0/2201199279 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9f015bcd0 0x7fe9f00c6ef0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8007d40 tx=0x7fe9e800e4b0 comp rx=0 tx=0).stop 2024-02-19T15:31:42.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.509+0000 7fe9f8000640 1 -- 172.21.15.47:0/2201199279 shutdown_connections 2024-02-19T15:31:42.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.509+0000 7fe9f8000640 1 --2- 172.21.15.47:0/2201199279 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe9d0066e40 0x7fe9d0069300 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:42.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.509+0000 7fe9f8000640 1 --2- 172.21.15.47:0/2201199279 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9f015bcd0 0x7fe9f00c6ef0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:42.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.509+0000 7fe9f8000640 1 --2- 172.21.15.47:0/2201199279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9f00d0360 0x7fe9f00c69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:42.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.509+0000 7fe9f8000640 1 --2- 172.21.15.47:0/2201199279 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9f00cedb0 0x7fe9f00c5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:42.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.509+0000 7fe9f8000640 1 -- 172.21.15.47:0/2201199279 >> 172.21.15.47:0/2201199279 conn(0x7fe9f00d1bf0 msgr2=0x7fe9f015b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:42.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.509+0000 7fe9f8000640 1 -- 172.21.15.47:0/2201199279 shutdown_connections 2024-02-19T15:31:42.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:42.509+0000 7fe9f8000640 1 -- 172.21.15.47:0/2201199279 wait complete. 2024-02-19T15:31:42.729 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:42.729 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:43.730 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:43.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.869+0000 7f8c8ac00640 1 Processor -- start 2024-02-19T15:31:43.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.869+0000 7f8c8ac00640 1 -- start start 2024-02-19T15:31:43.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.873+0000 7f8c8ac00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8c8415bcd0 0x7f8c840cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:43.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.873+0000 7f8c8ac00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c840cc700 con 0x7f8c840d0360 2024-02-19T15:31:43.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.873+0000 7f8c8ac00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c840cf300 con 0x7f8c840cedb0 2024-02-19T15:31:43.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.873+0000 7f8c8ac00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c840cf5e0 con 0x7f8c8415bcd0 2024-02-19T15:31:43.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.873+0000 7f8c83e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8c8415bcd0 0x7f8c840cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:43.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.873+0000 7f8c88c00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8c840d0360 0x7f8c8415d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51334/0 (socket says 172.21.15.47:51334) 2024-02-19T15:31:43.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.873+0000 7f8c88c00640 1 -- 172.21.15.47:0/734235219 learned_addr learned my addr 172.21.15.47:0/734235219 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:31:43.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.873+0000 7f8c83e00640 1 -- 172.21.15.47:0/734235219 >> v1:172.21.15.116:6789/0 conn(0x7f8c840cedb0 legacy=0x7f8c840cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:43.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.873+0000 7f8c83e00640 1 -- 172.21.15.47:0/734235219 >> v1:172.21.15.47:6789/0 conn(0x7f8c840d0360 legacy=0x7f8c8415d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:43.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.873+0000 7f8c83e00640 1 -- 172.21.15.47:0/734235219 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c840cc9a0 con 0x7f8c8415bcd0 2024-02-19T15:31:43.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.873+0000 7f8c83400640 1 -- 172.21.15.47:0/734235219 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732099005 0 0) 0x7f8c840cc700 con 0x7f8c840d0360 2024-02-19T15:31:43.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.877+0000 7f8c83e00640 1 --2- 172.21.15.47:0/734235219 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c8415bcd0 0x7f8c840cd480 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f8c740097c0 tx=0x7f8c7401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad5d9c5d8415ca8c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:43.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.877+0000 7f8c83400640 1 -- 172.21.15.47:0/734235219 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c740ab670 con 0x7f8c8415bcd0 2024-02-19T15:31:43.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.877+0000 7f8c83400640 1 -- 172.21.15.47:0/734235219 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c740ab7d0 con 0x7f8c8415bcd0 2024-02-19T15:31:43.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.877+0000 7f8c83400640 1 -- 172.21.15.47:0/734235219 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c740abb60 con 0x7f8c8415bcd0 2024-02-19T15:31:43.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.877+0000 7f8c8ac00640 1 -- 172.21.15.47:0/734235219 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c8415bcd0 msgr2=0x7f8c840cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:43.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.877+0000 7f8c8ac00640 1 --2- 172.21.15.47:0/734235219 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c8415bcd0 0x7f8c840cd480 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f8c740097c0 tx=0x7f8c7401c470 comp rx=0 tx=0).stop 2024-02-19T15:31:43.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.877+0000 7f8c8ac00640 1 -- 172.21.15.47:0/734235219 shutdown_connections 2024-02-19T15:31:43.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.877+0000 7f8c8ac00640 1 --2- 172.21.15.47:0/734235219 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c8415bcd0 0x7f8c840cd480 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:43.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.877+0000 7f8c8ac00640 1 -- 172.21.15.47:0/734235219 >> 172.21.15.47:0/734235219 conn(0x7f8c840d1bf0 msgr2=0x7f8c840d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:43.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.877+0000 7f8c8ac00640 1 -- 172.21.15.47:0/734235219 shutdown_connections 2024-02-19T15:31:43.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.877+0000 7f8c8ac00640 1 -- 172.21.15.47:0/734235219 wait complete. 2024-02-19T15:31:43.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.877+0000 7f8c8ac00640 1 Processor -- start 2024-02-19T15:31:43.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.877+0000 7f8c8ac00640 1 -- start start 2024-02-19T15:31:43.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c8ac00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c840cedb0 0x7f8c84146fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:43.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c8ac00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c840d0360 0x7f8c840ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:43.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c8ac00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c8415bcd0 0x7f8c840cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:43.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c8ac00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c840cb820 con 0x7f8c840cedb0 2024-02-19T15:31:43.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c8ac00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c840cac30 con 0x7f8c8415bcd0 2024-02-19T15:31:43.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c8ac00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c84147510 con 0x7f8c840d0360 2024-02-19T15:31:43.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c83e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c840cedb0 0x7f8c84146fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:43.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c83e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c840cedb0 0x7f8c84146fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48386/0 (socket says 172.21.15.47:48386) 2024-02-19T15:31:43.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c83e00640 1 -- 172.21.15.47:0/3523937511 learned_addr learned my addr 172.21.15.47:0/3523937511 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:43.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c88c00640 1 --2- 172.21.15.47:0/3523937511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c8415bcd0 0x7f8c840cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:43.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c7b400640 1 --2- 172.21.15.47:0/3523937511 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c840d0360 0x7f8c840ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:43.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c83e00640 1 -- 172.21.15.47:0/3523937511 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c840d0360 msgr2=0x7f8c840ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:43.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c83e00640 1 --2- 172.21.15.47:0/3523937511 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c840d0360 0x7f8c840ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:43.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c83e00640 1 -- 172.21.15.47:0/3523937511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c8415bcd0 msgr2=0x7f8c840cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:43.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c83e00640 1 --2- 172.21.15.47:0/3523937511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c8415bcd0 0x7f8c840cdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:43.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c83e00640 1 -- 172.21.15.47:0/3523937511 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c74009420 con 0x7f8c840cedb0 2024-02-19T15:31:43.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c83e00640 1 --2- 172.21.15.47:0/3523937511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c840cedb0 0x7f8c84146fd0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f8c740ad040 tx=0x7f8c7401acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:43.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c80c00640 1 -- 172.21.15.47:0/3523937511 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c740b5810 con 0x7f8c840cedb0 2024-02-19T15:31:43.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c80c00640 1 -- 172.21.15.47:0/3523937511 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c740b5990 con 0x7f8c840cedb0 2024-02-19T15:31:43.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c80c00640 1 -- 172.21.15.47:0/3523937511 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c740b5d40 con 0x7f8c840cedb0 2024-02-19T15:31:43.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c8ac00640 1 -- 172.21.15.47:0/3523937511 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c840cdb40 con 0x7f8c840cedb0 2024-02-19T15:31:43.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c8ac00640 1 -- 172.21.15.47:0/3523937511 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c840cb160 con 0x7f8c840cedb0 2024-02-19T15:31:43.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c8ac00640 1 -- 172.21.15.47:0/3523937511 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c84075360 con 0x7f8c840cedb0 2024-02-19T15:31:43.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.881+0000 7f8c80c00640 1 -- 172.21.15.47:0/3523937511 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8c740bf410 con 0x7f8c840cedb0 2024-02-19T15:31:43.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.885+0000 7f8c80c00640 1 --2- 172.21.15.47:0/3523937511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8c60066e40 0x7f8c60069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:43.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.885+0000 7f8c80c00640 1 -- 172.21.15.47:0/3523937511 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8c7411f6e0 con 0x7f8c840cedb0 2024-02-19T15:31:43.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.885+0000 7f8c7b400640 1 --2- 172.21.15.47:0/3523937511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8c60066e40 0x7f8c60069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:43.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.885+0000 7f8c7b400640 1 --2- 172.21.15.47:0/3523937511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8c60066e40 0x7f8c60069300 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f8c8415b6c0 tx=0x7f8c6c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:43.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:43.889+0000 7f8c80c00640 1 -- 172.21.15.47:0/3523937511 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f8c740ec400 con 0x7f8c840cedb0 2024-02-19T15:31:44.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:44.081+0000 7f8c8ac00640 1 -- 172.21.15.47:0/3523937511 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c8406ef50 con 0x7f8c840cedb0 2024-02-19T15:31:44.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:44.081+0000 7f8c80c00640 1 -- 172.21.15.47:0/3523937511 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f8c740efac0 con 0x7f8c840cedb0 2024-02-19T15:31:44.082 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:44.082 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:44.083 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:44.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:44.085+0000 7f8c8ac00640 1 -- 172.21.15.47:0/3523937511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8c60066e40 msgr2=0x7f8c60069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:44.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:44.085+0000 7f8c8ac00640 1 --2- 172.21.15.47:0/3523937511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8c60066e40 0x7f8c60069300 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f8c8415b6c0 tx=0x7f8c6c009040 comp rx=0 tx=0).stop 2024-02-19T15:31:44.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:44.085+0000 7f8c8ac00640 1 -- 172.21.15.47:0/3523937511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c840cedb0 msgr2=0x7f8c84146fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:44.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:44.085+0000 7f8c8ac00640 1 --2- 172.21.15.47:0/3523937511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c840cedb0 0x7f8c84146fd0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f8c740ad040 tx=0x7f8c7401acb0 comp rx=0 tx=0).stop 2024-02-19T15:31:44.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:44.085+0000 7f8c8ac00640 1 -- 172.21.15.47:0/3523937511 shutdown_connections 2024-02-19T15:31:44.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:44.085+0000 7f8c8ac00640 1 --2- 172.21.15.47:0/3523937511 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8c60066e40 0x7f8c60069300 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:44.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:44.085+0000 7f8c8ac00640 1 --2- 172.21.15.47:0/3523937511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c8415bcd0 0x7f8c840cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:44.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:44.085+0000 7f8c8ac00640 1 --2- 172.21.15.47:0/3523937511 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c840d0360 0x7f8c840ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:44.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:44.085+0000 7f8c8ac00640 1 --2- 172.21.15.47:0/3523937511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c840cedb0 0x7f8c84146fd0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:44.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:44.085+0000 7f8c8ac00640 1 -- 172.21.15.47:0/3523937511 >> 172.21.15.47:0/3523937511 conn(0x7f8c840d1bf0 msgr2=0x7f8c840d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:44.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:44.085+0000 7f8c8ac00640 1 -- 172.21.15.47:0/3523937511 shutdown_connections 2024-02-19T15:31:44.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:44.085+0000 7f8c8ac00640 1 -- 172.21.15.47:0/3523937511 wait complete. 2024-02-19T15:31:44.274 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:44.304 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:44.304 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:45.305 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:45.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.445+0000 7f0d47000640 1 Processor -- start 2024-02-19T15:31:45.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.445+0000 7f0d47000640 1 -- start start 2024-02-19T15:31:45.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.449+0000 7f0d47000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0d4006cfc0 0x7f0d400b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:45.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.449+0000 7f0d47000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4006c660 con 0x7f0d4015e180 2024-02-19T15:31:45.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.449+0000 7f0d47000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d400b2270 con 0x7f0d4006c1d0 2024-02-19T15:31:45.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.449+0000 7f0d47000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d400b2550 con 0x7f0d4006cfc0 2024-02-19T15:31:45.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.449+0000 7f0d37e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0d4006cfc0 0x7f0d400b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:45.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.449+0000 7f0d3fe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0d4015e180 0x7f0d4006c8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51338/0 (socket says 172.21.15.47:51338) 2024-02-19T15:31:45.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.449+0000 7f0d3fe00640 1 -- 172.21.15.47:0/4280928891 learned_addr learned my addr 172.21.15.47:0/4280928891 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:31:45.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.449+0000 7f0d37e00640 1 -- 172.21.15.47:0/4280928891 >> v1:172.21.15.116:6789/0 conn(0x7f0d4006c1d0 legacy=0x7f0d400ae870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:45.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.449+0000 7f0d37e00640 1 -- 172.21.15.47:0/4280928891 >> v1:172.21.15.47:6789/0 conn(0x7f0d4015e180 legacy=0x7f0d4006c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:45.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.449+0000 7f0d37e00640 1 -- 172.21.15.47:0/4280928891 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d400b2830 con 0x7f0d4006cfc0 2024-02-19T15:31:45.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.449+0000 7f0d3f400640 1 -- 172.21.15.47:0/4280928891 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894579275 0 0) 0x7f0d4006c660 con 0x7f0d4015e180 2024-02-19T15:31:45.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.453+0000 7f0d37e00640 1 --2- 172.21.15.47:0/4280928891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d4006cfc0 0x7f0d400b1c20 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f0d280097c0 tx=0x7f0d2801c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921505237bb70826 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:45.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.453+0000 7f0d3f400640 1 -- 172.21.15.47:0/4280928891 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d280a7070 con 0x7f0d4006cfc0 2024-02-19T15:31:45.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.453+0000 7f0d3f400640 1 -- 172.21.15.47:0/4280928891 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d280ab430 con 0x7f0d4006cfc0 2024-02-19T15:31:45.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.453+0000 7f0d3f400640 1 -- 172.21.15.47:0/4280928891 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d280ab7c0 con 0x7f0d4006cfc0 2024-02-19T15:31:45.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.453+0000 7f0d47000640 1 -- 172.21.15.47:0/4280928891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d4006cfc0 msgr2=0x7f0d400b1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:45.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.453+0000 7f0d47000640 1 --2- 172.21.15.47:0/4280928891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d4006cfc0 0x7f0d400b1c20 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f0d280097c0 tx=0x7f0d2801c430 comp rx=0 tx=0).stop 2024-02-19T15:31:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.453+0000 7f0d47000640 1 -- 172.21.15.47:0/4280928891 shutdown_connections 2024-02-19T15:31:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.453+0000 7f0d47000640 1 --2- 172.21.15.47:0/4280928891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d4006cfc0 0x7f0d400b1c20 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.453+0000 7f0d47000640 1 -- 172.21.15.47:0/4280928891 >> 172.21.15.47:0/4280928891 conn(0x7f0d400d1ba0 msgr2=0x7f0d400d1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.453+0000 7f0d47000640 1 -- 172.21.15.47:0/4280928891 shutdown_connections 2024-02-19T15:31:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.453+0000 7f0d47000640 1 -- 172.21.15.47:0/4280928891 wait complete. 2024-02-19T15:31:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.453+0000 7f0d47000640 1 Processor -- start 2024-02-19T15:31:45.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.453+0000 7f0d47000640 1 -- start start 2024-02-19T15:31:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d47000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d4006c1d0 0x7f0d400b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d47000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d4006cfc0 0x7f0d400b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d47000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d4015e180 0x7f0d400c7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d47000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d400b7ac0 con 0x7f0d4006c1d0 2024-02-19T15:31:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d47000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d400b51d0 con 0x7f0d4006cfc0 2024-02-19T15:31:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d47000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d400b77e0 con 0x7f0d4015e180 2024-02-19T15:31:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d3fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d4006c1d0 0x7f0d400b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d3fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d4006c1d0 0x7f0d400b7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48392/0 (socket says 172.21.15.47:48392) 2024-02-19T15:31:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d3fe00640 1 -- 172.21.15.47:0/3578615587 learned_addr learned my addr 172.21.15.47:0/3578615587 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d37e00640 1 --2- 172.21.15.47:0/3578615587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d4006cfc0 0x7f0d400b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d45000640 1 --2- 172.21.15.47:0/3578615587 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d4015e180 0x7f0d400c7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d3fe00640 1 -- 172.21.15.47:0/3578615587 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d4015e180 msgr2=0x7f0d400c7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:45.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d3fe00640 1 --2- 172.21.15.47:0/3578615587 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d4015e180 0x7f0d400c7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:45.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d3fe00640 1 -- 172.21.15.47:0/3578615587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d4006cfc0 msgr2=0x7f0d400b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:45.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d3fe00640 1 --2- 172.21.15.47:0/3578615587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d4006cfc0 0x7f0d400b4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:45.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d3fe00640 1 -- 172.21.15.47:0/3578615587 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d28009420 con 0x7f0d4006c1d0 2024-02-19T15:31:45.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d45000640 1 --2- 172.21.15.47:0/3578615587 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d4015e180 0x7f0d400c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:31:45.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d3fe00640 1 --2- 172.21.15.47:0/3578615587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d4006c1d0 0x7f0d400b7150 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f0d300011f0 tx=0x7f0d3000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:45.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d3cc00640 1 -- 172.21.15.47:0/3578615587 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d300074e0 con 0x7f0d4006c1d0 2024-02-19T15:31:45.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d3cc00640 1 -- 172.21.15.47:0/3578615587 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d30007640 con 0x7f0d4006c1d0 2024-02-19T15:31:45.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d47000640 1 -- 172.21.15.47:0/3578615587 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d4014f580 con 0x7f0d4006c1d0 2024-02-19T15:31:45.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d3cc00640 1 -- 172.21.15.47:0/3578615587 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d30004a10 con 0x7f0d4006c1d0 2024-02-19T15:31:45.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d47000640 1 -- 172.21.15.47:0/3578615587 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d4014fb40 con 0x7f0d4006c1d0 2024-02-19T15:31:45.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.457+0000 7f0d47000640 1 -- 172.21.15.47:0/3578615587 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d40068500 con 0x7f0d4006c1d0 2024-02-19T15:31:45.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.461+0000 7f0d3cc00640 1 -- 172.21.15.47:0/3578615587 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0d30004b70 con 0x7f0d4006c1d0 2024-02-19T15:31:45.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.461+0000 7f0d3cc00640 1 --2- 172.21.15.47:0/3578615587 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d1c066e40 0x7f0d1c069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:45.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.461+0000 7f0d3cc00640 1 -- 172.21.15.47:0/3578615587 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0d30083fb0 con 0x7f0d4006c1d0 2024-02-19T15:31:45.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.461+0000 7f0d37e00640 1 --2- 172.21.15.47:0/3578615587 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d1c066e40 0x7f0d1c069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:45.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.461+0000 7f0d37e00640 1 --2- 172.21.15.47:0/3578615587 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d1c066e40 0x7f0d1c069300 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f0d280af000 tx=0x7f0d280a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:45.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.465+0000 7f0d3cc00640 1 -- 172.21.15.47:0/3578615587 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0d30050d70 con 0x7f0d4006c1d0 2024-02-19T15:31:45.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.665+0000 7f0d47000640 1 -- 172.21.15.47:0/3578615587 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d4014fd30 con 0x7f0d4006c1d0 2024-02-19T15:31:45.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.665+0000 7f0d3cc00640 1 -- 172.21.15.47:0/3578615587 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f0d30018020 con 0x7f0d4006c1d0 2024-02-19T15:31:45.664 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:45.664 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:45.664 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:45.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.665+0000 7f0d47000640 1 -- 172.21.15.47:0/3578615587 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d1c066e40 msgr2=0x7f0d1c069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:45.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.665+0000 7f0d47000640 1 --2- 172.21.15.47:0/3578615587 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d1c066e40 0x7f0d1c069300 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f0d280af000 tx=0x7f0d280a7000 comp rx=0 tx=0).stop 2024-02-19T15:31:45.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.665+0000 7f0d47000640 1 -- 172.21.15.47:0/3578615587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d4006c1d0 msgr2=0x7f0d400b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:45.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.665+0000 7f0d47000640 1 --2- 172.21.15.47:0/3578615587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d4006c1d0 0x7f0d400b7150 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f0d300011f0 tx=0x7f0d3000c9b0 comp rx=0 tx=0).stop 2024-02-19T15:31:45.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.665+0000 7f0d47000640 1 -- 172.21.15.47:0/3578615587 shutdown_connections 2024-02-19T15:31:45.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.665+0000 7f0d47000640 1 --2- 172.21.15.47:0/3578615587 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0d1c066e40 0x7f0d1c069300 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:45.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.665+0000 7f0d47000640 1 --2- 172.21.15.47:0/3578615587 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d4015e180 0x7f0d400c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:45.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.665+0000 7f0d47000640 1 --2- 172.21.15.47:0/3578615587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d4006cfc0 0x7f0d400b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:45.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.665+0000 7f0d47000640 1 --2- 172.21.15.47:0/3578615587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0d4006c1d0 0x7f0d400b7150 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:45.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.665+0000 7f0d47000640 1 -- 172.21.15.47:0/3578615587 >> 172.21.15.47:0/3578615587 conn(0x7f0d400d1ba0 msgr2=0x7f0d400d1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:45.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.665+0000 7f0d47000640 1 -- 172.21.15.47:0/3578615587 shutdown_connections 2024-02-19T15:31:45.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:45.665+0000 7f0d47000640 1 -- 172.21.15.47:0/3578615587 wait complete. 2024-02-19T15:31:45.886 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:45.886 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:46.276 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:46.887 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:47.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.025+0000 7fb0ece00640 1 Processor -- start 2024-02-19T15:31:47.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.029+0000 7fb0ece00640 1 -- start start 2024-02-19T15:31:47.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.029+0000 7fb0ece00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb0e806d650 0x7fb0e80d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:47.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.029+0000 7fb0ece00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb0e80ced10 0x7fb0e806bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:47.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.029+0000 7fb0ece00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e80cce30 con 0x7fb0e80cded0 2024-02-19T15:31:47.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.029+0000 7fb0ece00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e80cd110 con 0x7fb0e80ced10 2024-02-19T15:31:47.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.029+0000 7fb0ece00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e80cd3f0 con 0x7fb0e806d650 2024-02-19T15:31:47.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.029+0000 7fb0e6800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb0e80ced10 0x7fb0e806bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:47.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.029+0000 7fb0e5e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb0e806d650 0x7fb0e80d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:47.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.029+0000 7fb0e6800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb0e80ced10 0x7fb0e806bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:38620/0 (socket says 172.21.15.47:38620) 2024-02-19T15:31:47.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.029+0000 7fb0e5e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb0e806d650 0x7fb0e80d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:41484/0 (socket says 172.21.15.47:41484) 2024-02-19T15:31:47.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.029+0000 7fb0e6800640 1 -- 172.21.15.47:0/4145781527 learned_addr learned my addr 172.21.15.47:0/4145781527 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:47.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.029+0000 7fb0e6800640 1 -- 172.21.15.47:0/4145781527 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb0e806d650 msgr2=0x7fb0e80d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:47.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.029+0000 7fb0e6800640 1 --2- 172.21.15.47:0/4145781527 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb0e806d650 0x7fb0e80d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:47.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.029+0000 7fb0e6800640 1 -- 172.21.15.47:0/4145781527 >> v1:172.21.15.116:6789/0 conn(0x7fb0e80cded0 legacy=0x7fb0e80ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:47.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.029+0000 7fb0e6800640 1 -- 172.21.15.47:0/4145781527 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e80cd6d0 con 0x7fb0e80ced10 2024-02-19T15:31:47.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.029+0000 7fb0e5e00640 1 --2- 172.21.15.47:0/4145781527 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb0e806d650 0x7fb0e80d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:31:47.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.033+0000 7fb0e6800640 1 --2- 172.21.15.47:0/4145781527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0e80ced10 0x7fb0e806bd60 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0009e50 tx=0x7fb0e001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a459953e594693b1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:47.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.033+0000 7fb0e5400640 1 -- 172.21.15.47:0/4145781527 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e00a5850 con 0x7fb0e80ced10 2024-02-19T15:31:47.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.033+0000 7fb0e5400640 1 -- 172.21.15.47:0/4145781527 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e00a59b0 con 0x7fb0e80ced10 2024-02-19T15:31:47.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.033+0000 7fb0e5400640 1 -- 172.21.15.47:0/4145781527 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e00a5d40 con 0x7fb0e80ced10 2024-02-19T15:31:47.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.033+0000 7fb0ece00640 1 -- 172.21.15.47:0/4145781527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0e80ced10 msgr2=0x7fb0e806bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:47.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.033+0000 7fb0ece00640 1 --2- 172.21.15.47:0/4145781527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0e80ced10 0x7fb0e806bd60 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0009e50 tx=0x7fb0e001c970 comp rx=0 tx=0).stop 2024-02-19T15:31:47.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.033+0000 7fb0ece00640 1 -- 172.21.15.47:0/4145781527 shutdown_connections 2024-02-19T15:31:47.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.033+0000 7fb0ece00640 1 --2- 172.21.15.47:0/4145781527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0e80ced10 0x7fb0e806bd60 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:47.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.033+0000 7fb0ece00640 1 --2- 172.21.15.47:0/4145781527 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb0e806d650 0x7fb0e80d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:47.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.033+0000 7fb0ece00640 1 -- 172.21.15.47:0/4145781527 >> 172.21.15.47:0/4145781527 conn(0x7fb0e80d1bf0 msgr2=0x7fb0e80d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:47.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.033+0000 7fb0ece00640 1 -- 172.21.15.47:0/4145781527 shutdown_connections 2024-02-19T15:31:47.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.033+0000 7fb0ece00640 1 -- 172.21.15.47:0/4145781527 wait complete. 2024-02-19T15:31:47.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0ece00640 1 Processor -- start 2024-02-19T15:31:47.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0ece00640 1 -- start start 2024-02-19T15:31:47.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0ece00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0e806d650 0x7fb0e80cbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:47.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0ece00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0e80cded0 0x7fb0e815c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:47.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0ece00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb0e80ced10 0x7fb0e814abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:47.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0ece00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e80c9570 con 0x7fb0e80cded0 2024-02-19T15:31:47.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0ece00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e80c7f30 con 0x7fb0e806d650 2024-02-19T15:31:47.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0ece00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e80c8210 con 0x7fb0e80ced10 2024-02-19T15:31:47.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0dfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0e80cded0 0x7fb0e815c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:47.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0e5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0e806d650 0x7fb0e80cbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:47.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0dfe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0e80cded0 0x7fb0e815c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48398/0 (socket says 172.21.15.47:48398) 2024-02-19T15:31:47.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0dfe00640 1 -- 172.21.15.47:0/2988841939 learned_addr learned my addr 172.21.15.47:0/2988841939 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:47.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0e6800640 1 --2- 172.21.15.47:0/2988841939 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb0e80ced10 0x7fb0e814abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:47.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0e5e00640 1 -- 172.21.15.47:0/2988841939 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb0e80ced10 msgr2=0x7fb0e814abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:47.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0e5e00640 1 --2- 172.21.15.47:0/2988841939 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb0e80ced10 0x7fb0e814abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:47.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0e5e00640 1 -- 172.21.15.47:0/2988841939 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0e80cded0 msgr2=0x7fb0e815c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:47.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0e5e00640 1 --2- 172.21.15.47:0/2988841939 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0e80cded0 0x7fb0e815c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:47.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0e5e00640 1 -- 172.21.15.47:0/2988841939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d8009420 con 0x7fb0e806d650 2024-02-19T15:31:47.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0dfe00640 1 --2- 172.21.15.47:0/2988841939 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0e80cded0 0x7fb0e815c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:31:47.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0e5e00640 1 --2- 172.21.15.47:0/2988841939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0e806d650 0x7fb0e80cbd10 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb0d800e8e0 tx=0x7fb0d800eda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:47.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0de000640 1 -- 172.21.15.47:0/2988841939 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d800c8e0 con 0x7fb0e806d650 2024-02-19T15:31:47.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0de000640 1 -- 172.21.15.47:0/2988841939 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d800ca40 con 0x7fb0e806d650 2024-02-19T15:31:47.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0ece00640 1 -- 172.21.15.47:0/2988841939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e0009b30 con 0x7fb0e806d650 2024-02-19T15:31:47.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0de000640 1 -- 172.21.15.47:0/2988841939 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d800cdf0 con 0x7fb0e806d650 2024-02-19T15:31:47.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.037+0000 7fb0ece00640 1 -- 172.21.15.47:0/2988841939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e80cc250 con 0x7fb0e806d650 2024-02-19T15:31:47.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.041+0000 7fb0ece00640 1 -- 172.21.15.47:0/2988841939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e8075360 con 0x7fb0e806d650 2024-02-19T15:31:47.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.041+0000 7fb0de000640 1 -- 172.21.15.47:0/2988841939 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb0d80059a0 con 0x7fb0e806d650 2024-02-19T15:31:47.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.041+0000 7fb0de000640 1 --2- 172.21.15.47:0/2988841939 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb0c4066e40 0x7fb0c4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:47.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.041+0000 7fb0de000640 1 -- 172.21.15.47:0/2988841939 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb0d8084c50 con 0x7fb0e806d650 2024-02-19T15:31:47.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.041+0000 7fb0dfe00640 1 --2- 172.21.15.47:0/2988841939 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb0c4066e40 0x7fb0c4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:47.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.045+0000 7fb0dfe00640 1 --2- 172.21.15.47:0/2988841939 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb0c4066e40 0x7fb0c4069300 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fb0e814ed20 tx=0x7fb0d0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:47.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.045+0000 7fb0de000640 1 -- 172.21.15.47:0/2988841939 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb0d8051b20 con 0x7fb0e806d650 2024-02-19T15:31:47.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.241+0000 7fb0ece00640 1 -- 172.21.15.47:0/2988841939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0e80cf580 con 0x7fb0e806d650 2024-02-19T15:31:47.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.241+0000 7fb0de000640 1 -- 172.21.15.47:0/2988841939 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7fb0d8021090 con 0x7fb0e806d650 2024-02-19T15:31:47.241 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:47.241 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:47.241 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:47.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.245+0000 7fb0ece00640 1 -- 172.21.15.47:0/2988841939 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb0c4066e40 msgr2=0x7fb0c4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:47.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.245+0000 7fb0ece00640 1 --2- 172.21.15.47:0/2988841939 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb0c4066e40 0x7fb0c4069300 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fb0e814ed20 tx=0x7fb0d0009040 comp rx=0 tx=0).stop 2024-02-19T15:31:47.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.245+0000 7fb0ece00640 1 -- 172.21.15.47:0/2988841939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0e806d650 msgr2=0x7fb0e80cbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:47.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.245+0000 7fb0ece00640 1 --2- 172.21.15.47:0/2988841939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0e806d650 0x7fb0e80cbd10 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb0d800e8e0 tx=0x7fb0d800eda0 comp rx=0 tx=0).stop 2024-02-19T15:31:47.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.245+0000 7fb0ece00640 1 -- 172.21.15.47:0/2988841939 shutdown_connections 2024-02-19T15:31:47.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.245+0000 7fb0ece00640 1 --2- 172.21.15.47:0/2988841939 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb0c4066e40 0x7fb0c4069300 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:47.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.245+0000 7fb0ece00640 1 --2- 172.21.15.47:0/2988841939 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb0e80ced10 0x7fb0e814abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:47.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.245+0000 7fb0ece00640 1 --2- 172.21.15.47:0/2988841939 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0e80cded0 0x7fb0e815c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:47.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.245+0000 7fb0ece00640 1 --2- 172.21.15.47:0/2988841939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0e806d650 0x7fb0e80cbd10 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:47.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.245+0000 7fb0ece00640 1 -- 172.21.15.47:0/2988841939 >> 172.21.15.47:0/2988841939 conn(0x7fb0e80d1bf0 msgr2=0x7fb0e80b5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:47.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.245+0000 7fb0ece00640 1 -- 172.21.15.47:0/2988841939 shutdown_connections 2024-02-19T15:31:47.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:47.245+0000 7fb0ece00640 1 -- 172.21.15.47:0/2988841939 wait complete. 2024-02-19T15:31:47.464 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:47.464 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:48.278 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:48.465 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:48.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.605+0000 7eff41200640 1 Processor -- start 2024-02-19T15:31:48.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.605+0000 7eff41200640 1 -- start start 2024-02-19T15:31:48.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.605+0000 7eff41200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff3c0cf670 0x7eff3c06aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:48.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff41200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff3c145a80 con 0x7eff3c06b3f0 2024-02-19T15:31:48.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff41200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff3c0d1880 con 0x7eff3c1456a0 2024-02-19T15:31:48.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff41200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c0d03c0 con 0x7eff3c0cf670 2024-02-19T15:31:48.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff3a800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7eff3c06b3f0 0x7eff3c06dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51352/0 (socket says 172.21.15.47:51352) 2024-02-19T15:31:48.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff3a800640 1 -- 172.21.15.47:0/232032652 learned_addr learned my addr 172.21.15.47:0/232032652 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:31:48.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff33e00640 1 --2- 172.21.15.47:0/232032652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff3c0cf670 0x7eff3c06aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:48.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff39400640 1 -- 172.21.15.47:0/232032652 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1662508084 0 0) 0x7eff3c145a80 con 0x7eff3c06b3f0 2024-02-19T15:31:48.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff39400640 1 -- 172.21.15.47:0/232032652 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff20003530 con 0x7eff3c06b3f0 2024-02-19T15:31:48.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff39400640 1 -- 172.21.15.47:0/232032652 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384421019 0 0) 0x7eff3c0d1880 con 0x7eff3c1456a0 2024-02-19T15:31:48.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff39400640 1 -- 172.21.15.47:0/232032652 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff3c145a80 con 0x7eff3c1456a0 2024-02-19T15:31:48.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff39400640 1 -- 172.21.15.47:0/232032652 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3896250764 0 0) 0x7eff20003530 con 0x7eff3c06b3f0 2024-02-19T15:31:48.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff39400640 1 -- 172.21.15.47:0/232032652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff3c0cf670 msgr2=0x7eff3c06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:48.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff39400640 1 --2- 172.21.15.47:0/232032652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff3c0cf670 0x7eff3c06aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:48.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff39400640 1 -- 172.21.15.47:0/232032652 >> v1:172.21.15.116:6789/0 conn(0x7eff3c1456a0 legacy=0x7eff3c0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:48.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff39400640 1 -- 172.21.15.47:0/232032652 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3c0cfb20 con 0x7eff3c06b3f0 2024-02-19T15:31:48.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff39400640 1 -- 172.21.15.47:0/232032652 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7eff34001a00 con 0x7eff3c06b3f0 2024-02-19T15:31:48.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff39400640 1 -- 172.21.15.47:0/232032652 >> v1:172.21.15.47:6789/0 conn(0x7eff3c06b3f0 legacy=0x7eff3c06dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:48.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff39400640 1 --2- 172.21.15.47:0/232032652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff3c0cf670 0x7eff20003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:48.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff39400640 1 --2- 172.21.15.47:0/232032652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff200040a0 0x7eff200047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:48.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff39400640 1 --2- 172.21.15.47:0/232032652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff20004d10 0x7eff20005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:48.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff39400640 1 -- 172.21.15.47:0/232032652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c0d03c0 con 0x7eff200040a0 2024-02-19T15:31:48.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff39400640 1 -- 172.21.15.47:0/232032652 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2000ba70 con 0x7eff20004d10 2024-02-19T15:31:48.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff39400640 1 -- 172.21.15.47:0/232032652 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff2000bc30 con 0x7eff3c0cf670 2024-02-19T15:31:48.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff33e00640 1 --2- 172.21.15.47:0/232032652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff200040a0 0x7eff200047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:48.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff39e00640 1 --2- 172.21.15.47:0/232032652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff3c0cf670 0x7eff20003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:48.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff3a800640 1 --2- 172.21.15.47:0/232032652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff20004d10 0x7eff20005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:48.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff33e00640 1 -- 172.21.15.47:0/232032652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff3c0cf670 msgr2=0x7eff20003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:48.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff33e00640 1 --2- 172.21.15.47:0/232032652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff3c0cf670 0x7eff20003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:48.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff33e00640 1 -- 172.21.15.47:0/232032652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff20004d10 msgr2=0x7eff20005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:48.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff33e00640 1 --2- 172.21.15.47:0/232032652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff20004d10 0x7eff20005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:48.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff33e00640 1 -- 172.21.15.47:0/232032652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff20003530 con 0x7eff200040a0 2024-02-19T15:31:48.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff3a800640 1 --2- 172.21.15.47:0/232032652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff20004d10 0x7eff20005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:31:48.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.609+0000 7eff39e00640 1 --2- 172.21.15.47:0/232032652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff3c0cf670 0x7eff20003b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:31:48.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.613+0000 7eff33e00640 1 --2- 172.21.15.47:0/232032652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff200040a0 0x7eff200047d0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7eff24009ab0 tx=0x7eff2401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ef29fc919eac89 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:48.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.613+0000 7eff39400640 1 -- 172.21.15.47:0/232032652 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff240a5640 con 0x7eff200040a0 2024-02-19T15:31:48.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.613+0000 7eff39400640 1 -- 172.21.15.47:0/232032652 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff240a57a0 con 0x7eff200040a0 2024-02-19T15:31:48.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.613+0000 7eff39400640 1 -- 172.21.15.47:0/232032652 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff240a5b50 con 0x7eff200040a0 2024-02-19T15:31:48.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.613+0000 7eff41200640 1 -- 172.21.15.47:0/232032652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff200040a0 msgr2=0x7eff200047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:48.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.613+0000 7eff41200640 1 --2- 172.21.15.47:0/232032652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff200040a0 0x7eff200047d0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7eff24009ab0 tx=0x7eff2401ca50 comp rx=0 tx=0).stop 2024-02-19T15:31:48.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.613+0000 7eff3a800640 1 -- 172.21.15.47:0/232032652 reap_dead start 2024-02-19T15:31:48.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.613+0000 7eff41200640 1 -- 172.21.15.47:0/232032652 shutdown_connections 2024-02-19T15:31:48.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.613+0000 7eff41200640 1 -- 172.21.15.47:0/232032652 >> 172.21.15.47:0/232032652 conn(0x7eff3c0d0ad0 msgr2=0x7eff3c0d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:48.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.613+0000 7eff41200640 1 -- 172.21.15.47:0/232032652 shutdown_connections 2024-02-19T15:31:48.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.613+0000 7eff41200640 1 -- 172.21.15.47:0/232032652 wait complete. 2024-02-19T15:31:48.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.613+0000 7eff41200640 1 Processor -- start 2024-02-19T15:31:48.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.613+0000 7eff41200640 1 -- start start 2024-02-19T15:31:48.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff41200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff200040a0 0x7eff3c143710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:48.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff41200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff3c1e22c0 0x7eff3c1db0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:48.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff41200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c1df1d0 0x7eff3c1db5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:48.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff41200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c0b5a80 con 0x7eff3c1e22c0 2024-02-19T15:31:48.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff41200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c0b49e0 con 0x7eff3c1df1d0 2024-02-19T15:31:48.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff41200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c0b4cc0 con 0x7eff200040a0 2024-02-19T15:31:48.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff33e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff3c1e22c0 0x7eff3c1db0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:48.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff33e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff3c1e22c0 0x7eff3c1db0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48428/0 (socket says 172.21.15.47:48428) 2024-02-19T15:31:48.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff33e00640 1 -- 172.21.15.47:0/468526815 learned_addr learned my addr 172.21.15.47:0/468526815 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:48.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff39e00640 1 --2- 172.21.15.47:0/468526815 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff200040a0 0x7eff3c143710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:48.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff3a800640 1 --2- 172.21.15.47:0/468526815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c1df1d0 0x7eff3c1db5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:48.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff33e00640 1 -- 172.21.15.47:0/468526815 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff200040a0 msgr2=0x7eff3c143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:48.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff33e00640 1 --2- 172.21.15.47:0/468526815 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff200040a0 0x7eff3c143710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:48.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff33e00640 1 -- 172.21.15.47:0/468526815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c1df1d0 msgr2=0x7eff3c1db5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:48.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff33e00640 1 --2- 172.21.15.47:0/468526815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c1df1d0 0x7eff3c1db5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:48.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff33e00640 1 -- 172.21.15.47:0/468526815 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3c0b4fa0 con 0x7eff3c1e22c0 2024-02-19T15:31:48.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff33e00640 1 --2- 172.21.15.47:0/468526815 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff3c1e22c0 0x7eff3c1db0a0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7eff24009ab0 tx=0x7eff20005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:48.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff32000640 1 -- 172.21.15.47:0/468526815 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff24004220 con 0x7eff3c1e22c0 2024-02-19T15:31:48.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff32000640 1 -- 172.21.15.47:0/468526815 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff24004380 con 0x7eff3c1e22c0 2024-02-19T15:31:48.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff41200640 1 -- 172.21.15.47:0/468526815 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff3c0b5280 con 0x7eff3c1e22c0 2024-02-19T15:31:48.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff32000640 1 -- 172.21.15.47:0/468526815 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff24003820 con 0x7eff3c1e22c0 2024-02-19T15:31:48.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff41200640 1 -- 172.21.15.47:0/468526815 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff3c0b4620 con 0x7eff3c1e22c0 2024-02-19T15:31:48.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff41200640 1 -- 172.21.15.47:0/468526815 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff3c075360 con 0x7eff3c1e22c0 2024-02-19T15:31:48.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.617+0000 7eff32000640 1 -- 172.21.15.47:0/468526815 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7eff240b4040 con 0x7eff3c1e22c0 2024-02-19T15:31:48.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.621+0000 7eff32000640 1 --2- 172.21.15.47:0/468526815 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7eff18066e70 0x7eff18069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:48.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.621+0000 7eff32000640 1 -- 172.21.15.47:0/468526815 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7eff2411ecb0 con 0x7eff3c1e22c0 2024-02-19T15:31:48.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.621+0000 7eff39e00640 1 --2- 172.21.15.47:0/468526815 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7eff18066e70 0x7eff18069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:48.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.621+0000 7eff39e00640 1 --2- 172.21.15.47:0/468526815 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7eff18066e70 0x7eff18069330 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7eff2c000c10 tx=0x7eff2c007220 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:48.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.625+0000 7eff32000640 1 -- 172.21.15.47:0/468526815 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7eff240ebcc0 con 0x7eff3c1e22c0 2024-02-19T15:31:48.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.817+0000 7eff41200640 1 -- 172.21.15.47:0/468526815 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff3c072b90 con 0x7eff3c1e22c0 2024-02-19T15:31:48.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.817+0000 7eff32000640 1 -- 172.21.15.47:0/468526815 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7eff240b2d80 con 0x7eff3c1e22c0 2024-02-19T15:31:48.818 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:48.818 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:48.819 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:48.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.821+0000 7eff41200640 1 -- 172.21.15.47:0/468526815 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7eff18066e70 msgr2=0x7eff18069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:48.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.821+0000 7eff41200640 1 --2- 172.21.15.47:0/468526815 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7eff18066e70 0x7eff18069330 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7eff2c000c10 tx=0x7eff2c007220 comp rx=0 tx=0).stop 2024-02-19T15:31:48.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.821+0000 7eff41200640 1 -- 172.21.15.47:0/468526815 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff3c1e22c0 msgr2=0x7eff3c1db0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:48.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.821+0000 7eff41200640 1 --2- 172.21.15.47:0/468526815 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff3c1e22c0 0x7eff3c1db0a0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7eff24009ab0 tx=0x7eff20005800 comp rx=0 tx=0).stop 2024-02-19T15:31:48.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.821+0000 7eff41200640 1 -- 172.21.15.47:0/468526815 shutdown_connections 2024-02-19T15:31:48.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.821+0000 7eff41200640 1 --2- 172.21.15.47:0/468526815 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7eff18066e70 0x7eff18069330 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:48.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.821+0000 7eff41200640 1 --2- 172.21.15.47:0/468526815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c1df1d0 0x7eff3c1db5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:48.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.821+0000 7eff41200640 1 --2- 172.21.15.47:0/468526815 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff3c1e22c0 0x7eff3c1db0a0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:48.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.821+0000 7eff41200640 1 --2- 172.21.15.47:0/468526815 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff200040a0 0x7eff3c143710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:48.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.821+0000 7eff41200640 1 -- 172.21.15.47:0/468526815 >> 172.21.15.47:0/468526815 conn(0x7eff3c0d0ad0 msgr2=0x7eff3c06f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:48.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.821+0000 7eff41200640 1 -- 172.21.15.47:0/468526815 shutdown_connections 2024-02-19T15:31:48.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:48.821+0000 7eff41200640 1 -- 172.21.15.47:0/468526815 wait complete. 2024-02-19T15:31:49.041 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:49.041 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:50.042 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:50.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.185+0000 7f235a800640 1 Processor -- start 2024-02-19T15:31:50.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.185+0000 7f235a800640 1 -- start start 2024-02-19T15:31:50.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.185+0000 7f235a800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2354145590 0x7f23540727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:50.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.185+0000 7f235a800640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f23540cea80 0x7f235406c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:50.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.185+0000 7f235a800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f235415c390 0x7f235415c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:50.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.185+0000 7f235a800640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f235415d280 con 0x7f235415c390 2024-02-19T15:31:50.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.185+0000 7f235a800640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2354070ff0 con 0x7f23540cea80 2024-02-19T15:31:50.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.185+0000 7f235a800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f235415cd60 con 0x7f2354145590 2024-02-19T15:31:50.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.185+0000 7f2353e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f235415c390 0x7f235415c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:50.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.185+0000 7f2352a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f23540cea80 0x7f235406c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:50.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.185+0000 7f2353400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2354145590 0x7f23540727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:50.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.185+0000 7f2353e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f235415c390 0x7f235415c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48432/0 (socket says 172.21.15.47:48432) 2024-02-19T15:31:50.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.185+0000 7f2353e00640 1 -- 172.21.15.47:0/191986463 learned_addr learned my addr 172.21.15.47:0/191986463 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:50.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.185+0000 7f2353e00640 1 -- 172.21.15.47:0/191986463 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2354145590 msgr2=0x7f23540727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:50.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.185+0000 7f2353e00640 1 --2- 172.21.15.47:0/191986463 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2354145590 0x7f23540727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:50.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.185+0000 7f2353e00640 1 -- 172.21.15.47:0/191986463 >> v2:172.21.15.116:3300/0 conn(0x7f23540cea80 msgr2=0x7f235406c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:50.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.185+0000 7f2353e00640 1 --2- 172.21.15.47:0/191986463 >> v2:172.21.15.116:3300/0 conn(0x7f23540cea80 0x7f235406c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:50.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.185+0000 7f2353e00640 1 -- 172.21.15.47:0/191986463 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23540709e0 con 0x7f235415c390 2024-02-19T15:31:50.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.185+0000 7f2352a00640 1 --2- 172.21.15.47:0/191986463 >> v2:172.21.15.116:3300/0 conn(0x7f23540cea80 0x7f235406c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:31:50.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.193+0000 7f2353e00640 1 --2- 172.21.15.47:0/191986463 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f235415c390 0x7f235415c820 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f234c009e50 tx=0x7f234c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36ffd88851c32cac server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:50.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.193+0000 7f2352000640 1 -- 172.21.15.47:0/191986463 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f234c0a5780 con 0x7f235415c390 2024-02-19T15:31:50.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.193+0000 7f2352000640 1 -- 172.21.15.47:0/191986463 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f234c0a58e0 con 0x7f235415c390 2024-02-19T15:31:50.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.193+0000 7f2352000640 1 -- 172.21.15.47:0/191986463 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f234c0a5c70 con 0x7f235415c390 2024-02-19T15:31:50.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.193+0000 7f235a800640 1 -- 172.21.15.47:0/191986463 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f235415c390 msgr2=0x7f235415c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:50.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.193+0000 7f235a800640 1 --2- 172.21.15.47:0/191986463 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f235415c390 0x7f235415c820 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f234c009e50 tx=0x7f234c01c9f0 comp rx=0 tx=0).stop 2024-02-19T15:31:50.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.193+0000 7f235a800640 1 -- 172.21.15.47:0/191986463 shutdown_connections 2024-02-19T15:31:50.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.193+0000 7f235a800640 1 --2- 172.21.15.47:0/191986463 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f235415c390 0x7f235415c820 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:50.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.193+0000 7f235a800640 1 --2- 172.21.15.47:0/191986463 >> v2:172.21.15.116:3300/0 conn(0x7f23540cea80 0x7f235406c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:50.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.193+0000 7f235a800640 1 --2- 172.21.15.47:0/191986463 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2354145590 0x7f23540727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:50.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.193+0000 7f235a800640 1 -- 172.21.15.47:0/191986463 >> 172.21.15.47:0/191986463 conn(0x7f23540b16a0 msgr2=0x7f23540b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:50.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.193+0000 7f235a800640 1 -- 172.21.15.47:0/191986463 shutdown_connections 2024-02-19T15:31:50.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.193+0000 7f235a800640 1 -- 172.21.15.47:0/191986463 wait complete. 2024-02-19T15:31:50.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.193+0000 7f235a800640 1 Processor -- start 2024-02-19T15:31:50.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.193+0000 7f235a800640 1 -- start start 2024-02-19T15:31:50.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f235a800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23540cea80 0x7f2354158220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:50.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f235a800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2354145590 0x7f2354134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:50.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f235a800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f235415c390 0x7f235413c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:50.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f235a800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23541f6520 con 0x7f2354145590 2024-02-19T15:31:50.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f235a800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23540b38d0 con 0x7f23540cea80 2024-02-19T15:31:50.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f235a800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2354158760 con 0x7f235415c390 2024-02-19T15:31:50.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f2352a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2354145590 0x7f2354134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:50.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f2352a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2354145590 0x7f2354134eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48442/0 (socket says 172.21.15.47:48442) 2024-02-19T15:31:50.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f2352a00640 1 -- 172.21.15.47:0/975329560 learned_addr learned my addr 172.21.15.47:0/975329560 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:50.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f2353e00640 1 --2- 172.21.15.47:0/975329560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f235415c390 0x7f235413c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:50.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f2353400640 1 --2- 172.21.15.47:0/975329560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23540cea80 0x7f2354158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:50.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f2352a00640 1 -- 172.21.15.47:0/975329560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f235415c390 msgr2=0x7f235413c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:50.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f2352a00640 1 --2- 172.21.15.47:0/975329560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f235415c390 0x7f235413c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:50.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f2352a00640 1 -- 172.21.15.47:0/975329560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23540cea80 msgr2=0x7f2354158220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:50.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f2352a00640 1 --2- 172.21.15.47:0/975329560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23540cea80 0x7f2354158220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:50.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f2352a00640 1 -- 172.21.15.47:0/975329560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2340009420 con 0x7f2354145590 2024-02-19T15:31:50.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f2352a00640 1 --2- 172.21.15.47:0/975329560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2354145590 0x7f2354134eb0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f234000e9b0 tx=0x7f234000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:50.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f234b400640 1 -- 172.21.15.47:0/975329560 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2340009700 con 0x7f2354145590 2024-02-19T15:31:50.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f234b400640 1 -- 172.21.15.47:0/975329560 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f234000ca00 con 0x7f2354145590 2024-02-19T15:31:50.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f234b400640 1 -- 172.21.15.47:0/975329560 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f234000cd90 con 0x7f2354145590 2024-02-19T15:31:50.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f235a800640 1 -- 172.21.15.47:0/975329560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f234c009b30 con 0x7f2354145590 2024-02-19T15:31:50.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f235a800640 1 -- 172.21.15.47:0/975329560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23540b3210 con 0x7f2354145590 2024-02-19T15:31:50.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f235a800640 1 -- 172.21.15.47:0/975329560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23540b3a10 con 0x7f2354145590 2024-02-19T15:31:50.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.197+0000 7f234b400640 1 -- 172.21.15.47:0/975329560 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2340013470 con 0x7f2354145590 2024-02-19T15:31:50.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.201+0000 7f234b400640 1 --2- 172.21.15.47:0/975329560 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2334066e40 0x7f2334069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:50.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.201+0000 7f234b400640 1 -- 172.21.15.47:0/975329560 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2340084ca0 con 0x7f2354145590 2024-02-19T15:31:50.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.201+0000 7f2353400640 1 --2- 172.21.15.47:0/975329560 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2334066e40 0x7f2334069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:50.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.201+0000 7f2353400640 1 --2- 172.21.15.47:0/975329560 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2334066e40 0x7f2334069300 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f2344009b60 tx=0x7f2344009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:50.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.205+0000 7f234b400640 1 -- 172.21.15.47:0/975329560 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f23540b3a10 con 0x7f2354145590 2024-02-19T15:31:50.280 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:50.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.401+0000 7f235a800640 1 -- 172.21.15.47:0/975329560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23540b3a10 con 0x7f2354145590 2024-02-19T15:31:50.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.405+0000 7f234b400640 1 -- 172.21.15.47:0/975329560 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f2340021d80 con 0x7f2354145590 2024-02-19T15:31:50.402 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:50.402 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:50.403 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:50.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.405+0000 7f235a800640 1 -- 172.21.15.47:0/975329560 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2334066e40 msgr2=0x7f2334069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:50.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.405+0000 7f235a800640 1 --2- 172.21.15.47:0/975329560 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2334066e40 0x7f2334069300 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f2344009b60 tx=0x7f2344009190 comp rx=0 tx=0).stop 2024-02-19T15:31:50.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.405+0000 7f235a800640 1 -- 172.21.15.47:0/975329560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2354145590 msgr2=0x7f2354134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:50.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.405+0000 7f235a800640 1 --2- 172.21.15.47:0/975329560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2354145590 0x7f2354134eb0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f234000e9b0 tx=0x7f234000ee70 comp rx=0 tx=0).stop 2024-02-19T15:31:50.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.405+0000 7f235a800640 1 -- 172.21.15.47:0/975329560 shutdown_connections 2024-02-19T15:31:50.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.405+0000 7f235a800640 1 --2- 172.21.15.47:0/975329560 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2334066e40 0x7f2334069300 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:50.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.405+0000 7f235a800640 1 --2- 172.21.15.47:0/975329560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f235415c390 0x7f235413c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:50.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.405+0000 7f235a800640 1 --2- 172.21.15.47:0/975329560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2354145590 0x7f2354134eb0 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:50.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.405+0000 7f235a800640 1 --2- 172.21.15.47:0/975329560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23540cea80 0x7f2354158220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:50.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.405+0000 7f235a800640 1 -- 172.21.15.47:0/975329560 >> 172.21.15.47:0/975329560 conn(0x7f23540b16a0 msgr2=0x7f23540af510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:50.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.405+0000 7f235a800640 1 -- 172.21.15.47:0/975329560 shutdown_connections 2024-02-19T15:31:50.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:50.405+0000 7f235a800640 1 -- 172.21.15.47:0/975329560 wait complete. 2024-02-19T15:31:50.626 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:50.626 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:51.627 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:51.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.769+0000 7f038e000640 1 Processor -- start 2024-02-19T15:31:51.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.769+0000 7f038e000640 1 -- start start 2024-02-19T15:31:51.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.769+0000 7f038e000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f03880cea80 0x7f03880727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:51.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.769+0000 7f038e000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f038815d280 con 0x7f038815c390 2024-02-19T15:31:51.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.769+0000 7f038e000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0388070ff0 con 0x7f0388145590 2024-02-19T15:31:51.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.769+0000 7f038e000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03880709e0 con 0x7f03880cea80 2024-02-19T15:31:51.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.769+0000 7f0387e00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f038815c390 0x7f038815c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51364/0 (socket says 172.21.15.47:51364) 2024-02-19T15:31:51.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.769+0000 7f0387e00640 1 -- 172.21.15.47:0/3912968986 learned_addr learned my addr 172.21.15.47:0/3912968986 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:31:51.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.769+0000 7f037fe00640 1 --2- 172.21.15.47:0/3912968986 >> v2:172.21.15.116:3300/0 conn(0x7f03880cea80 0x7f03880727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:51.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.769+0000 7f0386a00640 1 -- 172.21.15.47:0/3912968986 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496627012 0 0) 0x7f038815d280 con 0x7f038815c390 2024-02-19T15:31:51.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.769+0000 7f0386a00640 1 -- 172.21.15.47:0/3912968986 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f036c003530 con 0x7f038815c390 2024-02-19T15:31:51.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.769+0000 7f037fe00640 1 -- 172.21.15.47:0/3912968986 >> v1:172.21.15.116:6789/0 conn(0x7f0388145590 legacy=0x7f038806c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:51.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.769+0000 7f037fe00640 1 -- 172.21.15.47:0/3912968986 >> v1:172.21.15.47:6789/0 conn(0x7f038815c390 legacy=0x7f038815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:51.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.769+0000 7f037fe00640 1 -- 172.21.15.47:0/3912968986 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03880cf020 con 0x7f03880cea80 2024-02-19T15:31:51.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.773+0000 7f037fe00640 1 --2- 172.21.15.47:0/3912968986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03880cea80 0x7f03880727b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f03700097c0 tx=0x7f037001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=414c3b7be27aadc6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:51.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.773+0000 7f0386a00640 1 -- 172.21.15.47:0/3912968986 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03700ac410 con 0x7f03880cea80 2024-02-19T15:31:51.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.773+0000 7f0386a00640 1 -- 172.21.15.47:0/3912968986 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03700ac570 con 0x7f03880cea80 2024-02-19T15:31:51.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.773+0000 7f0386a00640 1 -- 172.21.15.47:0/3912968986 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03700ac900 con 0x7f03880cea80 2024-02-19T15:31:51.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.773+0000 7f038e000640 1 -- 172.21.15.47:0/3912968986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03880cea80 msgr2=0x7f03880727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:51.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.773+0000 7f038e000640 1 --2- 172.21.15.47:0/3912968986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03880cea80 0x7f03880727b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f03700097c0 tx=0x7f037001c4a0 comp rx=0 tx=0).stop 2024-02-19T15:31:51.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.773+0000 7f038e000640 1 -- 172.21.15.47:0/3912968986 shutdown_connections 2024-02-19T15:31:51.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.773+0000 7f038e000640 1 --2- 172.21.15.47:0/3912968986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03880cea80 0x7f03880727b0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:51.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.773+0000 7f038e000640 1 -- 172.21.15.47:0/3912968986 >> 172.21.15.47:0/3912968986 conn(0x7f03880b16a0 msgr2=0x7f03880b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:51.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.773+0000 7f038e000640 1 -- 172.21.15.47:0/3912968986 shutdown_connections 2024-02-19T15:31:51.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.773+0000 7f038e000640 1 -- 172.21.15.47:0/3912968986 wait complete. 2024-02-19T15:31:51.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f038e000640 1 Processor -- start 2024-02-19T15:31:51.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f038e000640 1 -- start start 2024-02-19T15:31:51.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f038e000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03880cea80 0x7f038813aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:51.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f038e000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0388145590 0x7f03880b3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:51.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f038e000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f038815c390 0x7f0388157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:51.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f038e000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038813af60 con 0x7f038815c390 2024-02-19T15:31:51.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f038e000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03880b4af0 con 0x7f03880cea80 2024-02-19T15:31:51.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f038e000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03880b4810 con 0x7f0388145590 2024-02-19T15:31:51.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f0387e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f038815c390 0x7f0388157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:51.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f0387e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f038815c390 0x7f0388157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:48444/0 (socket says 172.21.15.47:48444) 2024-02-19T15:31:51.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f0387e00640 1 -- 172.21.15.47:0/2301292084 learned_addr learned my addr 172.21.15.47:0/2301292084 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:51.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f0387400640 1 --2- 172.21.15.47:0/2301292084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03880cea80 0x7f038813aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:51.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f037fe00640 1 --2- 172.21.15.47:0/2301292084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0388145590 0x7f03880b3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:51.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f0387e00640 1 -- 172.21.15.47:0/2301292084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0388145590 msgr2=0x7f03880b3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:51.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f0387e00640 1 --2- 172.21.15.47:0/2301292084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0388145590 0x7f03880b3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:51.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f0387e00640 1 -- 172.21.15.47:0/2301292084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03880cea80 msgr2=0x7f038813aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:51.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f0387e00640 1 --2- 172.21.15.47:0/2301292084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03880cea80 0x7f038813aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:51.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f0387e00640 1 -- 172.21.15.47:0/2301292084 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0370009420 con 0x7f038815c390 2024-02-19T15:31:51.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f0387e00640 1 --2- 172.21.15.47:0/2301292084 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f038815c390 0x7f0388157ce0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f038000caa0 tx=0x7f038000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:51.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f037f400640 1 -- 172.21.15.47:0/2301292084 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038000ac00 con 0x7f038815c390 2024-02-19T15:31:51.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f037f400640 1 -- 172.21.15.47:0/2301292084 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f038000ad60 con 0x7f038815c390 2024-02-19T15:31:51.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f037f400640 1 -- 172.21.15.47:0/2301292084 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0380012620 con 0x7f038815c390 2024-02-19T15:31:51.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f038e000640 1 -- 172.21.15.47:0/2301292084 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03880ad970 con 0x7f038815c390 2024-02-19T15:31:51.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f038e000640 1 -- 172.21.15.47:0/2301292084 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03880adf30 con 0x7f038815c390 2024-02-19T15:31:51.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.777+0000 7f038e000640 1 -- 172.21.15.47:0/2301292084 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03880ae120 con 0x7f038815c390 2024-02-19T15:31:51.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.781+0000 7f037f400640 1 -- 172.21.15.47:0/2301292084 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0380007070 con 0x7f038815c390 2024-02-19T15:31:51.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.781+0000 7f037f400640 1 --2- 172.21.15.47:0/2301292084 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0364066e40 0x7f0364069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:51.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.781+0000 7f037f400640 1 -- 172.21.15.47:0/2301292084 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0380084980 con 0x7f038815c390 2024-02-19T15:31:51.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.781+0000 7f0387400640 1 --2- 172.21.15.47:0/2301292084 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0364066e40 0x7f0364069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:51.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.781+0000 7f0387400640 1 --2- 172.21.15.47:0/2301292084 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0364066e40 0x7f0364069300 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f0378007fd0 tx=0x7f0378009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:51.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.785+0000 7f037f400640 1 -- 172.21.15.47:0/2301292084 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f03800517d0 con 0x7f038815c390 2024-02-19T15:31:51.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.981+0000 7f038e000640 1 -- 172.21.15.47:0/2301292084 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03881392e0 con 0x7f038815c390 2024-02-19T15:31:51.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.981+0000 7f037f400640 1 -- 172.21.15.47:0/2301292084 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f0380017020 con 0x7f038815c390 2024-02-19T15:31:51.982 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:51.982 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:51.982 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:51.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.985+0000 7f038e000640 1 -- 172.21.15.47:0/2301292084 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0364066e40 msgr2=0x7f0364069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:51.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.985+0000 7f038e000640 1 --2- 172.21.15.47:0/2301292084 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0364066e40 0x7f0364069300 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f0378007fd0 tx=0x7f0378009040 comp rx=0 tx=0).stop 2024-02-19T15:31:51.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.985+0000 7f038e000640 1 -- 172.21.15.47:0/2301292084 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f038815c390 msgr2=0x7f0388157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:51.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.985+0000 7f038e000640 1 --2- 172.21.15.47:0/2301292084 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f038815c390 0x7f0388157ce0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f038000caa0 tx=0x7f038000cf60 comp rx=0 tx=0).stop 2024-02-19T15:31:51.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.985+0000 7f038e000640 1 -- 172.21.15.47:0/2301292084 shutdown_connections 2024-02-19T15:31:51.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.985+0000 7f038e000640 1 --2- 172.21.15.47:0/2301292084 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0364066e40 0x7f0364069300 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:51.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.985+0000 7f038e000640 1 --2- 172.21.15.47:0/2301292084 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f038815c390 0x7f0388157ce0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:51.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.985+0000 7f038e000640 1 --2- 172.21.15.47:0/2301292084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0388145590 0x7f03880b3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:51.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.985+0000 7f038e000640 1 --2- 172.21.15.47:0/2301292084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03880cea80 0x7f038813aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:51.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.985+0000 7f038e000640 1 -- 172.21.15.47:0/2301292084 >> 172.21.15.47:0/2301292084 conn(0x7f03880b16a0 msgr2=0x7f03880b02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:51.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.985+0000 7f038e000640 1 -- 172.21.15.47:0/2301292084 shutdown_connections 2024-02-19T15:31:51.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:51.985+0000 7f038e000640 1 -- 172.21.15.47:0/2301292084 wait complete. 2024-02-19T15:31:52.205 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:52.205 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:52.282 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:53.206 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:53.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.349+0000 7f28b5a00640 1 Processor -- start 2024-02-19T15:31:53.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.349+0000 7f28b5a00640 1 -- start start 2024-02-19T15:31:53.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.353+0000 7f28b5a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28b00cedb0 0x7f28b00cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:53.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.353+0000 7f28b5a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b00cc700 con 0x7f28b015bcd0 2024-02-19T15:31:53.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.353+0000 7f28b5a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b00cf300 con 0x7f28b00d2710 2024-02-19T15:31:53.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.353+0000 7f28b5a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b00cf5e0 con 0x7f28b00cedb0 2024-02-19T15:31:53.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.353+0000 7f28aea00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f28b015bcd0 0x7f28b00cd480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36232/0 (socket says 172.21.15.47:36232) 2024-02-19T15:31:53.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.353+0000 7f28aea00640 1 -- 172.21.15.47:0/3026314375 learned_addr learned my addr 172.21.15.47:0/3026314375 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:31:53.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.353+0000 7f28a6000640 1 --2- 172.21.15.47:0/3026314375 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28b00cedb0 0x7f28b00cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:53.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.353+0000 7f28ae000640 1 -- 172.21.15.47:0/3026314375 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507844936 0 0) 0x7f28b00cc700 con 0x7f28b015bcd0 2024-02-19T15:31:53.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.353+0000 7f28ae000640 1 -- 172.21.15.47:0/3026314375 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2894003530 con 0x7f28b015bcd0 2024-02-19T15:31:53.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.353+0000 7f28ae000640 1 -- 172.21.15.47:0/3026314375 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163913108 0 0) 0x7f28b00cf300 con 0x7f28b00d2710 2024-02-19T15:31:53.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.353+0000 7f28ae000640 1 -- 172.21.15.47:0/3026314375 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28b00cc700 con 0x7f28b00d2710 2024-02-19T15:31:53.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.353+0000 7f28a6000640 1 -- 172.21.15.47:0/3026314375 >> v1:172.21.15.116:6789/0 conn(0x7f28b00d2710 legacy=0x7f28b015d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:53.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.353+0000 7f28a6000640 1 -- 172.21.15.47:0/3026314375 >> v1:172.21.15.47:6789/0 conn(0x7f28b015bcd0 legacy=0x7f28b00cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:53.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.353+0000 7f28ae000640 1 -- 172.21.15.47:0/3026314375 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1067415096 0 0) 0x7f2894003530 con 0x7f28b015bcd0 2024-02-19T15:31:53.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.353+0000 7f28a6000640 1 -- 172.21.15.47:0/3026314375 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b00cc9a0 con 0x7f28b00cedb0 2024-02-19T15:31:53.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.357+0000 7f28a6000640 1 --2- 172.21.15.47:0/3026314375 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28b00cedb0 0x7f28b00cfc50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f28980097f0 tx=0x7f289801c3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7ca102486ae21c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:53.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.357+0000 7f28ae000640 1 -- 172.21.15.47:0/3026314375 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a0001680 con 0x7f28b00cedb0 2024-02-19T15:31:53.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.357+0000 7f28ae000640 1 -- 172.21.15.47:0/3026314375 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28980a7050 con 0x7f28b00cedb0 2024-02-19T15:31:53.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.357+0000 7f28ae000640 1 -- 172.21.15.47:0/3026314375 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28980ab620 con 0x7f28b00cedb0 2024-02-19T15:31:53.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.357+0000 7f28b5a00640 1 -- 172.21.15.47:0/3026314375 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28b00cedb0 msgr2=0x7f28b00cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:53.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.357+0000 7f28b5a00640 1 --2- 172.21.15.47:0/3026314375 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28b00cedb0 0x7f28b00cfc50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f28980097f0 tx=0x7f289801c3d0 comp rx=0 tx=0).stop 2024-02-19T15:31:53.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.357+0000 7f28b5a00640 1 -- 172.21.15.47:0/3026314375 shutdown_connections 2024-02-19T15:31:53.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.357+0000 7f28b5a00640 1 --2- 172.21.15.47:0/3026314375 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28b00cedb0 0x7f28b00cfc50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:53.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.357+0000 7f28b5a00640 1 -- 172.21.15.47:0/3026314375 >> 172.21.15.47:0/3026314375 conn(0x7f28b00d1bf0 msgr2=0x7f28b00d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:53.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.357+0000 7f28b5a00640 1 -- 172.21.15.47:0/3026314375 shutdown_connections 2024-02-19T15:31:53.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.357+0000 7f28b5a00640 1 -- 172.21.15.47:0/3026314375 wait complete. 2024-02-19T15:31:53.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.357+0000 7f28b5a00640 1 Processor -- start 2024-02-19T15:31:53.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.357+0000 7f28b5a00640 1 -- start start 2024-02-19T15:31:53.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.357+0000 7f28b5a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28b00cedb0 0x7f28b0159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:53.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.357+0000 7f28b5a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28b00d2710 0x7f28b00b5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:53.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.357+0000 7f28b5a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28b015bcd0 0x7f28b0156610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:53.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.357+0000 7f28b5a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b00cdb90 con 0x7f28b00cedb0 2024-02-19T15:31:53.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.357+0000 7f28b5a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b00b69e0 con 0x7f28b00d2710 2024-02-19T15:31:53.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.357+0000 7f28b5a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b00b6700 con 0x7f28b015bcd0 2024-02-19T15:31:53.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.357+0000 7f28aea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28b00cedb0 0x7f28b0159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:53.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28aea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28b00cedb0 0x7f28b0159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43822/0 (socket says 172.21.15.47:43822) 2024-02-19T15:31:53.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28aea00640 1 -- 172.21.15.47:0/2963774084 learned_addr learned my addr 172.21.15.47:0/2963774084 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:53.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28af400640 1 --2- 172.21.15.47:0/2963774084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28b015bcd0 0x7f28b0156610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:53.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28aea00640 1 -- 172.21.15.47:0/2963774084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28b015bcd0 msgr2=0x7f28b0156610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:53.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28a6000640 1 --2- 172.21.15.47:0/2963774084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28b00d2710 0x7f28b00b5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:53.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28aea00640 1 --2- 172.21.15.47:0/2963774084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28b015bcd0 0x7f28b0156610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:53.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28aea00640 1 -- 172.21.15.47:0/2963774084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28b00d2710 msgr2=0x7f28b00b5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:53.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28aea00640 1 --2- 172.21.15.47:0/2963774084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28b00d2710 0x7f28b00b5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:53.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28aea00640 1 -- 172.21.15.47:0/2963774084 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2898009420 con 0x7f28b00cedb0 2024-02-19T15:31:53.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28af400640 1 --2- 172.21.15.47:0/2963774084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28b015bcd0 0x7f28b0156610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:31:53.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28a6000640 1 --2- 172.21.15.47:0/2963774084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28b00d2710 0x7f28b00b5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:31:53.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28aea00640 1 --2- 172.21.15.47:0/2963774084 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28b00cedb0 0x7f28b0159740 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f28a000c750 tx=0x7f28a000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:53.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28a7400640 1 -- 172.21.15.47:0/2963774084 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a0005af0 con 0x7f28b00cedb0 2024-02-19T15:31:53.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28a7400640 1 -- 172.21.15.47:0/2963774084 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a0005c50 con 0x7f28b00cedb0 2024-02-19T15:31:53.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28b5a00640 1 -- 172.21.15.47:0/2963774084 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b015d240 con 0x7f28b00cedb0 2024-02-19T15:31:53.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28a7400640 1 -- 172.21.15.47:0/2963774084 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a0007070 con 0x7f28b00cedb0 2024-02-19T15:31:53.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28b5a00640 1 -- 172.21.15.47:0/2963774084 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b0158f60 con 0x7f28b00cedb0 2024-02-19T15:31:53.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28b5a00640 1 -- 172.21.15.47:0/2963774084 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b0158ae0 con 0x7f28b00cedb0 2024-02-19T15:31:53.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28a7400640 1 -- 172.21.15.47:0/2963774084 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f28a00071d0 con 0x7f28b00cedb0 2024-02-19T15:31:53.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28a7400640 1 --2- 172.21.15.47:0/2963774084 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f288c066e70 0x7f288c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:53.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.361+0000 7f28a7400640 1 -- 172.21.15.47:0/2963774084 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f28a0084160 con 0x7f28b00cedb0 2024-02-19T15:31:53.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.365+0000 7f28a6000640 1 --2- 172.21.15.47:0/2963774084 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f288c066e70 0x7f288c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:53.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.365+0000 7f28a6000640 1 --2- 172.21.15.47:0/2963774084 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f288c066e70 0x7f288c069330 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f28980ad000 tx=0x7f2898008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:53.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.369+0000 7f28a7400640 1 -- 172.21.15.47:0/2963774084 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f28b0158ae0 con 0x7f28b00cedb0 2024-02-19T15:31:53.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.565+0000 7f28b5a00640 1 -- 172.21.15.47:0/2963774084 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28b0158ae0 con 0x7f28b00cedb0 2024-02-19T15:31:53.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.565+0000 7f28a7400640 1 -- 172.21.15.47:0/2963774084 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f28a0017d80 con 0x7f28b00cedb0 2024-02-19T15:31:53.565 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:53.565 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:53.565 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:53.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.569+0000 7f28b5a00640 1 -- 172.21.15.47:0/2963774084 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f288c066e70 msgr2=0x7f288c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:53.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.569+0000 7f28b5a00640 1 --2- 172.21.15.47:0/2963774084 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f288c066e70 0x7f288c069330 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f28980ad000 tx=0x7f2898008040 comp rx=0 tx=0).stop 2024-02-19T15:31:53.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.569+0000 7f28b5a00640 1 -- 172.21.15.47:0/2963774084 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28b00cedb0 msgr2=0x7f28b0159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:53.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.569+0000 7f28b5a00640 1 --2- 172.21.15.47:0/2963774084 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28b00cedb0 0x7f28b0159740 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f28a000c750 tx=0x7f28a000cc10 comp rx=0 tx=0).stop 2024-02-19T15:31:53.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.569+0000 7f28b5a00640 1 -- 172.21.15.47:0/2963774084 shutdown_connections 2024-02-19T15:31:53.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.569+0000 7f28b5a00640 1 --2- 172.21.15.47:0/2963774084 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f288c066e70 0x7f288c069330 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:53.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.569+0000 7f28b5a00640 1 --2- 172.21.15.47:0/2963774084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28b015bcd0 0x7f28b0156610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:53.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.569+0000 7f28b5a00640 1 --2- 172.21.15.47:0/2963774084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28b00d2710 0x7f28b00b5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:53.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.569+0000 7f28b5a00640 1 --2- 172.21.15.47:0/2963774084 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f28b00cedb0 0x7f28b0159740 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:53.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.569+0000 7f28b5a00640 1 -- 172.21.15.47:0/2963774084 >> 172.21.15.47:0/2963774084 conn(0x7f28b00d1bf0 msgr2=0x7f28b006c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:53.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.569+0000 7f28b5a00640 1 -- 172.21.15.47:0/2963774084 shutdown_connections 2024-02-19T15:31:53.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:53.569+0000 7f28b5a00640 1 -- 172.21.15.47:0/2963774084 wait complete. 2024-02-19T15:31:53.796 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:53.797 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:54.284 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:54.798 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:54.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.941+0000 7fdce2a00640 1 Processor -- start 2024-02-19T15:31:54.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.941+0000 7fdce2a00640 1 -- start start 2024-02-19T15:31:54.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.941+0000 7fdce2a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdcdc06b330 0x7fdcdc0b4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:54.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.941+0000 7fdce2a00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcdc06b7c0 con 0x7fdcdc06aeb0 2024-02-19T15:31:54.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.941+0000 7fdce2a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcdc071a00 con 0x7fdcdc145740 2024-02-19T15:31:54.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.941+0000 7fdce2a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcdc071ce0 con 0x7fdcdc06b330 2024-02-19T15:31:54.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.941+0000 7fdcdb400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fdcdc06aeb0 0x7fdcdc0b3a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36236/0 (socket says 172.21.15.47:36236) 2024-02-19T15:31:54.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.941+0000 7fdcdb400640 1 -- 172.21.15.47:0/1938450889 learned_addr learned my addr 172.21.15.47:0/1938450889 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:31:54.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.941+0000 7fdce0a00640 1 --2- 172.21.15.47:0/1938450889 >> v2:172.21.15.116:3300/0 conn(0x7fdcdc06b330 0x7fdcdc0b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:54.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.941+0000 7fdcdaa00640 1 -- 172.21.15.47:0/1938450889 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958117124 0 0) 0x7fdcdc06b7c0 con 0x7fdcdc06aeb0 2024-02-19T15:31:54.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.941+0000 7fdcdaa00640 1 -- 172.21.15.47:0/1938450889 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc0003530 con 0x7fdcdc06aeb0 2024-02-19T15:31:54.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.941+0000 7fdce0a00640 1 -- 172.21.15.47:0/1938450889 >> v1:172.21.15.116:6789/0 conn(0x7fdcdc145740 legacy=0x7fdcdc0cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:54.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.941+0000 7fdce0a00640 1 -- 172.21.15.47:0/1938450889 >> v1:172.21.15.47:6789/0 conn(0x7fdcdc06aeb0 legacy=0x7fdcdc0b3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:54.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.941+0000 7fdce0a00640 1 -- 172.21.15.47:0/1938450889 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcdc071fc0 con 0x7fdcdc06b330 2024-02-19T15:31:54.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.945+0000 7fdce0a00640 1 --2- 172.21.15.47:0/1938450889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcdc06b330 0x7fdcdc0b4130 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4009d90 tx=0x7fdcd401c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=658cdb27eede81f9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:54.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.945+0000 7fdcdaa00640 1 -- 172.21.15.47:0/1938450889 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd40a58e0 con 0x7fdcdc06b330 2024-02-19T15:31:54.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.945+0000 7fdcdaa00640 1 -- 172.21.15.47:0/1938450889 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd40a5a40 con 0x7fdcdc06b330 2024-02-19T15:31:54.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.945+0000 7fdcdaa00640 1 -- 172.21.15.47:0/1938450889 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd40a5dd0 con 0x7fdcdc06b330 2024-02-19T15:31:54.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.945+0000 7fdce2a00640 1 -- 172.21.15.47:0/1938450889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcdc06b330 msgr2=0x7fdcdc0b4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:54.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.945+0000 7fdce2a00640 1 --2- 172.21.15.47:0/1938450889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcdc06b330 0x7fdcdc0b4130 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4009d90 tx=0x7fdcd401c820 comp rx=0 tx=0).stop 2024-02-19T15:31:54.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.945+0000 7fdce2a00640 1 -- 172.21.15.47:0/1938450889 shutdown_connections 2024-02-19T15:31:54.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.945+0000 7fdce2a00640 1 --2- 172.21.15.47:0/1938450889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcdc06b330 0x7fdcdc0b4130 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:54.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.945+0000 7fdce2a00640 1 -- 172.21.15.47:0/1938450889 >> 172.21.15.47:0/1938450889 conn(0x7fdcdc0d0a50 msgr2=0x7fdcdc0d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:54.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.945+0000 7fdce2a00640 1 -- 172.21.15.47:0/1938450889 shutdown_connections 2024-02-19T15:31:54.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.945+0000 7fdce2a00640 1 -- 172.21.15.47:0/1938450889 wait complete. 2024-02-19T15:31:54.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.949+0000 7fdce2a00640 1 Processor -- start 2024-02-19T15:31:54.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.949+0000 7fdce2a00640 1 -- start start 2024-02-19T15:31:54.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.949+0000 7fdce2a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcdc06aeb0 0x7fdcdc15b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:54.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.949+0000 7fdce2a00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdcdc06b330 0x7fdcdc13d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:54.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.949+0000 7fdce2a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdcdc145740 0x7fdcdc13dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:54.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.949+0000 7fdce2a00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc13f730 con 0x7fdcdc145740 2024-02-19T15:31:54.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.949+0000 7fdce2a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc13fa10 con 0x7fdcdc06aeb0 2024-02-19T15:31:54.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.949+0000 7fdce2a00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc13fcf0 con 0x7fdcdc06b330 2024-02-19T15:31:54.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.949+0000 7fdce0a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdcdc145740 0x7fdcdc13dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:54.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.949+0000 7fdce0a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdcdc145740 0x7fdcdc13dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43830/0 (socket says 172.21.15.47:43830) 2024-02-19T15:31:54.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.949+0000 7fdce0a00640 1 -- 172.21.15.47:0/2069356219 learned_addr learned my addr 172.21.15.47:0/2069356219 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:54.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.949+0000 7fdcdb400640 1 --2- 172.21.15.47:0/2069356219 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdcdc06b330 0x7fdcdc13d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:54.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.949+0000 7fdcdbe00640 1 --2- 172.21.15.47:0/2069356219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcdc06aeb0 0x7fdcdc15b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:54.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.953+0000 7fdce0a00640 1 -- 172.21.15.47:0/2069356219 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdcdc06b330 msgr2=0x7fdcdc13d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:54.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.953+0000 7fdce0a00640 1 --2- 172.21.15.47:0/2069356219 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdcdc06b330 0x7fdcdc13d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:54.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.953+0000 7fdce0a00640 1 -- 172.21.15.47:0/2069356219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcdc06aeb0 msgr2=0x7fdcdc15b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:54.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.953+0000 7fdce0a00640 1 --2- 172.21.15.47:0/2069356219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcdc06aeb0 0x7fdcdc15b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:54.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.953+0000 7fdce0a00640 1 -- 172.21.15.47:0/2069356219 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd4009a70 con 0x7fdcdc145740 2024-02-19T15:31:54.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.953+0000 7fdce0a00640 1 --2- 172.21.15.47:0/2069356219 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdcdc145740 0x7fdcdc13dde0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fdcd40b0040 tx=0x7fdcd40195c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:54.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.953+0000 7fdcd3e00640 1 -- 172.21.15.47:0/2069356219 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd40b9860 con 0x7fdcdc145740 2024-02-19T15:31:54.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.953+0000 7fdcd3e00640 1 -- 172.21.15.47:0/2069356219 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd40b99c0 con 0x7fdcdc145740 2024-02-19T15:31:54.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.953+0000 7fdce2a00640 1 -- 172.21.15.47:0/2069356219 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcdc13ffd0 con 0x7fdcdc145740 2024-02-19T15:31:54.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.953+0000 7fdcd3e00640 1 -- 172.21.15.47:0/2069356219 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd40b9d50 con 0x7fdcdc145740 2024-02-19T15:31:54.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.953+0000 7fdce2a00640 1 -- 172.21.15.47:0/2069356219 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcdc15b9e0 con 0x7fdcdc145740 2024-02-19T15:31:54.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.953+0000 7fdce2a00640 1 -- 172.21.15.47:0/2069356219 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcdc075360 con 0x7fdcdc145740 2024-02-19T15:31:54.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.953+0000 7fdcd3e00640 1 -- 172.21.15.47:0/2069356219 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fdcd40ad440 con 0x7fdcdc145740 2024-02-19T15:31:54.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.953+0000 7fdcd3e00640 1 --2- 172.21.15.47:0/2069356219 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdcbc066e40 0x7fdcbc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:54.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.953+0000 7fdcd3e00640 1 -- 172.21.15.47:0/2069356219 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdcd41200f0 con 0x7fdcdc145740 2024-02-19T15:31:54.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.957+0000 7fdcdbe00640 1 --2- 172.21.15.47:0/2069356219 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdcbc066e40 0x7fdcbc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:54.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.957+0000 7fdcdbe00640 1 --2- 172.21.15.47:0/2069356219 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdcbc066e40 0x7fdcbc069300 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fdccc007fd0 tx=0x7fdccc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:54.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:54.957+0000 7fdcd3e00640 1 -- 172.21.15.47:0/2069356219 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fdcd40ecfc0 con 0x7fdcdc145740 2024-02-19T15:31:55.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:55.157+0000 7fdce2a00640 1 -- 172.21.15.47:0/2069356219 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcdc136240 con 0x7fdcdc145740 2024-02-19T15:31:55.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:55.161+0000 7fdcd3e00640 1 -- 172.21.15.47:0/2069356219 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7fdcd40bdd80 con 0x7fdcdc145740 2024-02-19T15:31:55.159 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:55.159 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:55.159 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:55.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:55.161+0000 7fdce2a00640 1 -- 172.21.15.47:0/2069356219 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdcbc066e40 msgr2=0x7fdcbc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:55.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:55.161+0000 7fdce2a00640 1 --2- 172.21.15.47:0/2069356219 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdcbc066e40 0x7fdcbc069300 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fdccc007fd0 tx=0x7fdccc009040 comp rx=0 tx=0).stop 2024-02-19T15:31:55.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:55.161+0000 7fdce2a00640 1 -- 172.21.15.47:0/2069356219 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdcdc145740 msgr2=0x7fdcdc13dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:55.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:55.161+0000 7fdce2a00640 1 --2- 172.21.15.47:0/2069356219 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdcdc145740 0x7fdcdc13dde0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fdcd40b0040 tx=0x7fdcd40195c0 comp rx=0 tx=0).stop 2024-02-19T15:31:55.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:55.161+0000 7fdce2a00640 1 -- 172.21.15.47:0/2069356219 shutdown_connections 2024-02-19T15:31:55.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:55.161+0000 7fdce2a00640 1 --2- 172.21.15.47:0/2069356219 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fdcbc066e40 0x7fdcbc069300 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:55.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:55.161+0000 7fdce2a00640 1 --2- 172.21.15.47:0/2069356219 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdcdc145740 0x7fdcdc13dde0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:55.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:55.161+0000 7fdce2a00640 1 --2- 172.21.15.47:0/2069356219 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdcdc06b330 0x7fdcdc13d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:55.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:55.161+0000 7fdce2a00640 1 --2- 172.21.15.47:0/2069356219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcdc06aeb0 0x7fdcdc15b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:55.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:55.161+0000 7fdce2a00640 1 -- 172.21.15.47:0/2069356219 >> 172.21.15.47:0/2069356219 conn(0x7fdcdc0d0a50 msgr2=0x7fdcdc0cf0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:55.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:55.161+0000 7fdce2a00640 1 -- 172.21.15.47:0/2069356219 shutdown_connections 2024-02-19T15:31:55.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:55.161+0000 7fdce2a00640 1 -- 172.21.15.47:0/2069356219 wait complete. 2024-02-19T15:31:55.381 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:55.381 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:56.287 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:56.381 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:56.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.520+0000 7f89e4e00640 1 Processor -- start 2024-02-19T15:31:56.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.520+0000 7f89e4e00640 1 -- start start 2024-02-19T15:31:56.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.524+0000 7f89e4e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89e00d19e0 0x7f89e00d1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:56.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.524+0000 7f89e4e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f89e00cec90 0x7f89e00cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:56.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.524+0000 7f89e4e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e00d2300 con 0x7f89e00cf0f0 2024-02-19T15:31:56.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.524+0000 7f89e4e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89e00ce280 con 0x7f89e00cec90 2024-02-19T15:31:56.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.524+0000 7f89e4e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e015cd40 con 0x7f89e00d19e0 2024-02-19T15:31:56.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.524+0000 7f89dfe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f89e00cf0f0 0x7f89e00cca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36242/0 (socket says 172.21.15.47:36242) 2024-02-19T15:31:56.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.524+0000 7f89dfe00640 1 -- 172.21.15.47:0/1645408778 learned_addr learned my addr 172.21.15.47:0/1645408778 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:31:56.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.524+0000 7f89d7e00640 1 --2- 172.21.15.47:0/1645408778 >> v2:172.21.15.116:3300/0 conn(0x7f89e00cec90 0x7f89e00cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:56.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.524+0000 7f89df400640 1 --2- 172.21.15.47:0/1645408778 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89e00d19e0 0x7f89e00d1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:56.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.524+0000 7f89dea00640 1 -- 172.21.15.47:0/1645408778 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024442489 0 0) 0x7f89e00d2300 con 0x7f89e00cf0f0 2024-02-19T15:31:56.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.524+0000 7f89dea00640 1 -- 172.21.15.47:0/1645408778 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89c4003530 con 0x7f89e00cf0f0 2024-02-19T15:31:56.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.524+0000 7f89d7e00640 1 -- 172.21.15.47:0/1645408778 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89e00d19e0 msgr2=0x7f89e00d1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:56.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.524+0000 7f89d7e00640 1 --2- 172.21.15.47:0/1645408778 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89e00d19e0 0x7f89e00d1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:56.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.524+0000 7f89d7e00640 1 -- 172.21.15.47:0/1645408778 >> v1:172.21.15.47:6789/0 conn(0x7f89e00cf0f0 legacy=0x7f89e00cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:56.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.524+0000 7f89d7e00640 1 -- 172.21.15.47:0/1645408778 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e015d020 con 0x7f89e00cec90 2024-02-19T15:31:56.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.528+0000 7f89d7e00640 1 --2- 172.21.15.47:0/1645408778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89e00cec90 0x7f89e00cc4c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f89c80097c0 tx=0x7f89c801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98ff216447d87d08 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.528+0000 7f89dea00640 1 -- 172.21.15.47:0/1645408778 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c80ad640 con 0x7f89e00cec90 2024-02-19T15:31:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.528+0000 7f89dea00640 1 -- 172.21.15.47:0/1645408778 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c80ad7a0 con 0x7f89e00cec90 2024-02-19T15:31:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.528+0000 7f89dea00640 1 -- 172.21.15.47:0/1645408778 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c80adb30 con 0x7f89e00cec90 2024-02-19T15:31:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.528+0000 7f89e4e00640 1 -- 172.21.15.47:0/1645408778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89e00cec90 msgr2=0x7f89e00cc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.528+0000 7f89e4e00640 1 --2- 172.21.15.47:0/1645408778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89e00cec90 0x7f89e00cc4c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f89c80097c0 tx=0x7f89c801c4a0 comp rx=0 tx=0).stop 2024-02-19T15:31:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.528+0000 7f89e4e00640 1 -- 172.21.15.47:0/1645408778 shutdown_connections 2024-02-19T15:31:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.528+0000 7f89e4e00640 1 --2- 172.21.15.47:0/1645408778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89e00cec90 0x7f89e00cc4c0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.528+0000 7f89e4e00640 1 --2- 172.21.15.47:0/1645408778 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89e00d19e0 0x7f89e00d1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.528+0000 7f89e4e00640 1 -- 172.21.15.47:0/1645408778 >> 172.21.15.47:0/1645408778 conn(0x7f89e009a9d0 msgr2=0x7f89e009ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.528+0000 7f89e4e00640 1 -- 172.21.15.47:0/1645408778 shutdown_connections 2024-02-19T15:31:56.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.528+0000 7f89e4e00640 1 -- 172.21.15.47:0/1645408778 wait complete. 2024-02-19T15:31:56.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.528+0000 7f89e4e00640 1 Processor -- start 2024-02-19T15:31:56.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.528+0000 7f89e4e00640 1 -- start start 2024-02-19T15:31:56.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89e4e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89e00cec90 0x7f89e00cb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:56.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89e4e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89e00d19e0 0x7f89e006c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:56.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89e4e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89e0153b70 0x7f89e0154020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:56.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89e4e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e0154560 con 0x7f89e00cec90 2024-02-19T15:31:56.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89e4e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e01597d0 con 0x7f89e00d19e0 2024-02-19T15:31:56.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89e4e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e0159ab0 con 0x7f89e0153b70 2024-02-19T15:31:56.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89df400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89e00cec90 0x7f89e00cb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:56.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89df400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89e00cec90 0x7f89e00cb520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43836/0 (socket says 172.21.15.47:43836) 2024-02-19T15:31:56.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89df400640 1 -- 172.21.15.47:0/1189681565 learned_addr learned my addr 172.21.15.47:0/1189681565 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:56.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89d7e00640 1 --2- 172.21.15.47:0/1189681565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89e00d19e0 0x7f89e006c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:56.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89dfe00640 1 --2- 172.21.15.47:0/1189681565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89e0153b70 0x7f89e0154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:56.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89df400640 1 -- 172.21.15.47:0/1189681565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89e0153b70 msgr2=0x7f89e0154020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:56.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89df400640 1 --2- 172.21.15.47:0/1189681565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89e0153b70 0x7f89e0154020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:56.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89df400640 1 -- 172.21.15.47:0/1189681565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89e00d19e0 msgr2=0x7f89e006c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:56.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89df400640 1 --2- 172.21.15.47:0/1189681565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89e00d19e0 0x7f89e006c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:56.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89df400640 1 -- 172.21.15.47:0/1189681565 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c8009420 con 0x7f89e00cec90 2024-02-19T15:31:56.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89d7e00640 1 --2- 172.21.15.47:0/1189681565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89e00d19e0 0x7f89e006c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:31:56.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89df400640 1 --2- 172.21.15.47:0/1189681565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89e00cec90 0x7f89e00cb520 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f89d000e930 tx=0x7f89d000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:56.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89d7400640 1 -- 172.21.15.47:0/1189681565 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d0009690 con 0x7f89e00cec90 2024-02-19T15:31:56.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89d7400640 1 -- 172.21.15.47:0/1189681565 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d000ca00 con 0x7f89e00cec90 2024-02-19T15:31:56.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89d7400640 1 -- 172.21.15.47:0/1189681565 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d000cd70 con 0x7f89e00cec90 2024-02-19T15:31:56.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89e4e00640 1 -- 172.21.15.47:0/1189681565 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e0159d90 con 0x7f89e00cec90 2024-02-19T15:31:56.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89e4e00640 1 -- 172.21.15.47:0/1189681565 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e015a350 con 0x7f89e00cec90 2024-02-19T15:31:56.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89e4e00640 1 -- 172.21.15.47:0/1189681565 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e015a4e0 con 0x7f89e00cec90 2024-02-19T15:31:56.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.532+0000 7f89d7400640 1 -- 172.21.15.47:0/1189681565 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f89d0013470 con 0x7f89e00cec90 2024-02-19T15:31:56.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.536+0000 7f89d7400640 1 --2- 172.21.15.47:0/1189681565 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f89bc066e40 0x7f89bc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:56.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.536+0000 7f89d7400640 1 -- 172.21.15.47:0/1189681565 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f89d0084e30 con 0x7f89e00cec90 2024-02-19T15:31:56.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.536+0000 7f89d7e00640 1 --2- 172.21.15.47:0/1189681565 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f89bc066e40 0x7f89bc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:56.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.536+0000 7f89d7e00640 1 --2- 172.21.15.47:0/1189681565 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f89bc066e40 0x7f89bc069300 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f89c80af000 tx=0x7f89c8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:56.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.540+0000 7f89d7400640 1 -- 172.21.15.47:0/1189681565 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f89d0051b70 con 0x7f89e00cec90 2024-02-19T15:31:56.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.944+0000 7f89e4e00640 1 -- 172.21.15.47:0/1189681565 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89e0157d30 con 0x7f89e00cec90 2024-02-19T15:31:56.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.944+0000 7f89d7400640 1 -- 172.21.15.47:0/1189681565 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f89d0021090 con 0x7f89e00cec90 2024-02-19T15:31:56.944 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:56.944 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:56.944 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:56.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.948+0000 7f89e4e00640 1 -- 172.21.15.47:0/1189681565 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f89bc066e40 msgr2=0x7f89bc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:56.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.948+0000 7f89e4e00640 1 --2- 172.21.15.47:0/1189681565 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f89bc066e40 0x7f89bc069300 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f89c80af000 tx=0x7f89c8008040 comp rx=0 tx=0).stop 2024-02-19T15:31:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.948+0000 7f89e4e00640 1 -- 172.21.15.47:0/1189681565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89e00cec90 msgr2=0x7f89e00cb520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.948+0000 7f89e4e00640 1 --2- 172.21.15.47:0/1189681565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89e00cec90 0x7f89e00cb520 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f89d000e930 tx=0x7f89d000edf0 comp rx=0 tx=0).stop 2024-02-19T15:31:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.948+0000 7f89e4e00640 1 -- 172.21.15.47:0/1189681565 shutdown_connections 2024-02-19T15:31:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.948+0000 7f89e4e00640 1 --2- 172.21.15.47:0/1189681565 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f89bc066e40 0x7f89bc069300 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.948+0000 7f89e4e00640 1 --2- 172.21.15.47:0/1189681565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f89e0153b70 0x7f89e0154020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.948+0000 7f89e4e00640 1 --2- 172.21.15.47:0/1189681565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89e00d19e0 0x7f89e006c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.948+0000 7f89e4e00640 1 --2- 172.21.15.47:0/1189681565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89e00cec90 0x7f89e00cb520 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.948+0000 7f89e4e00640 1 -- 172.21.15.47:0/1189681565 >> 172.21.15.47:0/1189681565 conn(0x7f89e009a9d0 msgr2=0x7f89e009adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.948+0000 7f89e4e00640 1 -- 172.21.15.47:0/1189681565 shutdown_connections 2024-02-19T15:31:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:56.948+0000 7f89e4e00640 1 -- 172.21.15.47:0/1189681565 wait complete. 2024-02-19T15:31:57.175 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:57.175 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:58.176 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:58.288 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:31:58.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.316+0000 7f85b2600640 1 Processor -- start 2024-02-19T15:31:58.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.316+0000 7f85b2600640 1 -- start start 2024-02-19T15:31:58.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.316+0000 7f85b2600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f85ac06d650 0x7f85ac0d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:58.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.316+0000 7f85b2600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85ac0cded0 0x7f85ac0ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:58.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.316+0000 7f85b2600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ac0cce30 con 0x7f85ac0ceb40 2024-02-19T15:31:58.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.316+0000 7f85b2600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ac0cefd0 con 0x7f85ac06d650 2024-02-19T15:31:58.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.316+0000 7f85b2600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac0cd110 con 0x7f85ac0cded0 2024-02-19T15:31:58.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.320+0000 7f85ab400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f85ac06d650 0x7f85ac0d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:58.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.320+0000 7f85ab400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f85ac06d650 0x7f85ac0d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43838/0 (socket says 172.21.15.47:43838) 2024-02-19T15:31:58.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.320+0000 7f85ab400640 1 -- 172.21.15.47:0/2570166268 learned_addr learned my addr 172.21.15.47:0/2570166268 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:58.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.320+0000 7f85a3e00640 1 --2- 172.21.15.47:0/2570166268 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85ac0cded0 0x7f85ac0ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:58.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.320+0000 7f85ab400640 1 -- 172.21.15.47:0/2570166268 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85ac0cded0 msgr2=0x7f85ac0ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:58.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.320+0000 7f85ab400640 1 --2- 172.21.15.47:0/2570166268 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85ac0cded0 0x7f85ac0ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:58.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.320+0000 7f85ab400640 1 -- 172.21.15.47:0/2570166268 >> v1:172.21.15.116:6789/0 conn(0x7f85ac0ceb40 legacy=0x7f85ac161d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:58.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.320+0000 7f85ab400640 1 -- 172.21.15.47:0/2570166268 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac0cd3f0 con 0x7f85ac06d650 2024-02-19T15:31:58.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.320+0000 7f85a3e00640 1 --2- 172.21.15.47:0/2570166268 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85ac0cded0 0x7f85ac0ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:31:58.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.320+0000 7f85ab400640 1 --2- 172.21.15.47:0/2570166268 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85ac06d650 0x7f85ac0d08b0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f859c009870 tx=0x7f859c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19efcd0fb540d2e2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:58.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.320+0000 7f85aaa00640 1 -- 172.21.15.47:0/2570166268 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c0ac620 con 0x7f85ac06d650 2024-02-19T15:31:58.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.320+0000 7f85aaa00640 1 -- 172.21.15.47:0/2570166268 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f859c0ac780 con 0x7f85ac06d650 2024-02-19T15:31:58.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.320+0000 7f85aaa00640 1 -- 172.21.15.47:0/2570166268 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c0acb10 con 0x7f85ac06d650 2024-02-19T15:31:58.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85b2600640 1 -- 172.21.15.47:0/2570166268 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85ac06d650 msgr2=0x7f85ac0d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:58.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85b2600640 1 --2- 172.21.15.47:0/2570166268 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85ac06d650 0x7f85ac0d08b0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f859c009870 tx=0x7f859c01c4a0 comp rx=0 tx=0).stop 2024-02-19T15:31:58.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85b2600640 1 -- 172.21.15.47:0/2570166268 shutdown_connections 2024-02-19T15:31:58.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85b2600640 1 --2- 172.21.15.47:0/2570166268 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85ac0cded0 0x7f85ac0ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:58.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85b2600640 1 --2- 172.21.15.47:0/2570166268 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85ac06d650 0x7f85ac0d08b0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:58.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85b2600640 1 -- 172.21.15.47:0/2570166268 >> 172.21.15.47:0/2570166268 conn(0x7f85ac0d1bf0 msgr2=0x7f85ac0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:58.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85b2600640 1 -- 172.21.15.47:0/2570166268 shutdown_connections 2024-02-19T15:31:58.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85b2600640 1 -- 172.21.15.47:0/2570166268 wait complete. 2024-02-19T15:31:58.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85b2600640 1 Processor -- start 2024-02-19T15:31:58.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85b2600640 1 -- start start 2024-02-19T15:31:58.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85b2600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac06d650 0x7f85ac0c8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:58.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85b2600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85ac0cded0 0x7f85ac0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:58.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85b2600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85ac0ceb40 0x7f85ac0cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:58.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85b2600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac0c9200 con 0x7f85ac0cded0 2024-02-19T15:31:58.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85b2600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac0ca040 con 0x7f85ac06d650 2024-02-19T15:31:58.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85b2600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac0c9430 con 0x7f85ac0ceb40 2024-02-19T15:31:58.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85a3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85ac0cded0 0x7f85ac0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:58.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85a3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85ac0cded0 0x7f85ac0c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43840/0 (socket says 172.21.15.47:43840) 2024-02-19T15:31:58.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85a3e00640 1 -- 172.21.15.47:0/1168970367 learned_addr learned my addr 172.21.15.47:0/1168970367 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:58.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85ab400640 1 --2- 172.21.15.47:0/1168970367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac06d650 0x7f85ac0c8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:58.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85abe00640 1 --2- 172.21.15.47:0/1168970367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85ac0ceb40 0x7f85ac0cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:58.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85a3e00640 1 -- 172.21.15.47:0/1168970367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85ac0ceb40 msgr2=0x7f85ac0cb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:58.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85a3e00640 1 --2- 172.21.15.47:0/1168970367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85ac0ceb40 0x7f85ac0cb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:58.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85a3e00640 1 -- 172.21.15.47:0/1168970367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac06d650 msgr2=0x7f85ac0c8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:58.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85a3e00640 1 --2- 172.21.15.47:0/1168970367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac06d650 0x7f85ac0c8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:58.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85a3e00640 1 -- 172.21.15.47:0/1168970367 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f859c0094d0 con 0x7f85ac0cded0 2024-02-19T15:31:58.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.324+0000 7f85abe00640 1 --2- 172.21.15.47:0/1168970367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85ac0ceb40 0x7f85ac0cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:31:58.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.328+0000 7f85a3e00640 1 --2- 172.21.15.47:0/1168970367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85ac0cded0 0x7f85ac0c8cc0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f859400d800 tx=0x7f859400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:58.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.328+0000 7f85a3400640 1 -- 172.21.15.47:0/1168970367 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85940096c0 con 0x7f85ac0cded0 2024-02-19T15:31:58.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.328+0000 7f85a3400640 1 -- 172.21.15.47:0/1168970367 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8594009820 con 0x7f85ac0cded0 2024-02-19T15:31:58.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.328+0000 7f85a3400640 1 -- 172.21.15.47:0/1168970367 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8594009bd0 con 0x7f85ac0cded0 2024-02-19T15:31:58.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.328+0000 7f85b2600640 1 -- 172.21.15.47:0/1168970367 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ac0c9710 con 0x7f85ac0cded0 2024-02-19T15:31:58.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.328+0000 7f85b2600640 1 -- 172.21.15.47:0/1168970367 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ac0c9cd0 con 0x7f85ac0cded0 2024-02-19T15:31:58.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.328+0000 7f85b2600640 1 -- 172.21.15.47:0/1168970367 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ac0ca1f0 con 0x7f85ac0cded0 2024-02-19T15:31:58.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.328+0000 7f85a3400640 1 -- 172.21.15.47:0/1168970367 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f8594009d30 con 0x7f85ac0cded0 2024-02-19T15:31:58.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.328+0000 7f85a3400640 1 --2- 172.21.15.47:0/1168970367 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8588066e70 0x7f8588069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:58.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.328+0000 7f85a3400640 1 -- 172.21.15.47:0/1168970367 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f85940850b0 con 0x7f85ac0cded0 2024-02-19T15:31:58.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.332+0000 7f85ab400640 1 --2- 172.21.15.47:0/1168970367 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8588066e70 0x7f8588069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:58.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.332+0000 7f85ab400640 1 --2- 172.21.15.47:0/1168970367 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8588066e70 0x7f8588069330 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f859c0ae000 tx=0x7f859c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:58.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.332+0000 7f85a3400640 1 -- 172.21.15.47:0/1168970367 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f85ac0ca1f0 con 0x7f85ac0cded0 2024-02-19T15:31:58.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.600+0000 7f85b2600640 1 -- 172.21.15.47:0/1168970367 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85ac0ca1b0 con 0x7f85ac0cded0 2024-02-19T15:31:58.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.600+0000 7f85a3400640 1 -- 172.21.15.47:0/1168970367 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f8594051e50 con 0x7f85ac0cded0 2024-02-19T15:31:58.599 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:31:58.600 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:31:58.600 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:31:58.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.600+0000 7f85b2600640 1 -- 172.21.15.47:0/1168970367 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8588066e70 msgr2=0x7f8588069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:58.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.600+0000 7f85b2600640 1 --2- 172.21.15.47:0/1168970367 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8588066e70 0x7f8588069330 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f859c0ae000 tx=0x7f859c008040 comp rx=0 tx=0).stop 2024-02-19T15:31:58.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.600+0000 7f85b2600640 1 -- 172.21.15.47:0/1168970367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85ac0cded0 msgr2=0x7f85ac0c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:58.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.600+0000 7f85b2600640 1 --2- 172.21.15.47:0/1168970367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85ac0cded0 0x7f85ac0c8cc0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f859400d800 tx=0x7f859400dcc0 comp rx=0 tx=0).stop 2024-02-19T15:31:58.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.600+0000 7f85b2600640 1 -- 172.21.15.47:0/1168970367 shutdown_connections 2024-02-19T15:31:58.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.600+0000 7f85b2600640 1 --2- 172.21.15.47:0/1168970367 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8588066e70 0x7f8588069330 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:58.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.600+0000 7f85b2600640 1 --2- 172.21.15.47:0/1168970367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85ac0ceb40 0x7f85ac0cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:58.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.600+0000 7f85b2600640 1 --2- 172.21.15.47:0/1168970367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85ac0cded0 0x7f85ac0c8cc0 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:58.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.600+0000 7f85b2600640 1 --2- 172.21.15.47:0/1168970367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac06d650 0x7f85ac0c8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:58.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.600+0000 7f85b2600640 1 -- 172.21.15.47:0/1168970367 >> 172.21.15.47:0/1168970367 conn(0x7f85ac0d1bf0 msgr2=0x7f85ac070130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:58.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.600+0000 7f85b2600640 1 -- 172.21.15.47:0/1168970367 shutdown_connections 2024-02-19T15:31:58.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:58.600+0000 7f85b2600640 1 -- 172.21.15.47:0/1168970367 wait complete. 2024-02-19T15:31:58.822 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:31:58.822 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:31:59.823 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:31:59.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.964+0000 7f6acba00640 1 Processor -- start 2024-02-19T15:31:59.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.964+0000 7f6acba00640 1 -- start start 2024-02-19T15:31:59.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.968+0000 7f6acba00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6ac406d650 0x7f6ac40d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:59.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.968+0000 7f6acba00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ac40cded0 0x7f6ac40ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:59.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.968+0000 7f6acba00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac40cce30 con 0x7f6ac40ceb40 2024-02-19T15:31:59.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.968+0000 7f6acba00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ac40cefd0 con 0x7f6ac406d650 2024-02-19T15:31:59.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.968+0000 7f6acba00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac40cd110 con 0x7f6ac40cded0 2024-02-19T15:31:59.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.968+0000 7f6ac9000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6ac406d650 0x7f6ac40d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:59.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.968+0000 7f6ac9a00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6ac40ceb40 0x7f6ac4161d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36256/0 (socket says 172.21.15.47:36256) 2024-02-19T15:31:59.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.968+0000 7f6ac9a00640 1 -- 172.21.15.47:0/905134036 learned_addr learned my addr 172.21.15.47:0/905134036 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:31:59.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.968+0000 7f6ac3e00640 1 --2- 172.21.15.47:0/905134036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ac40cded0 0x7f6ac40ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:59.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.968+0000 7f6ac3400640 1 -- 172.21.15.47:0/905134036 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225801489 0 0) 0x7f6ac40cce30 con 0x7f6ac40ceb40 2024-02-19T15:31:59.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.968+0000 7f6ac9000640 1 -- 172.21.15.47:0/905134036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ac40cded0 msgr2=0x7f6ac40ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:31:59.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.968+0000 7f6ac9000640 1 --2- 172.21.15.47:0/905134036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ac40cded0 0x7f6ac40ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:59.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.968+0000 7f6ac9000640 1 -- 172.21.15.47:0/905134036 >> v1:172.21.15.47:6789/0 conn(0x7f6ac40ceb40 legacy=0x7f6ac4161d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:59.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.968+0000 7f6ac9000640 1 -- 172.21.15.47:0/905134036 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac40cd3f0 con 0x7f6ac406d650 2024-02-19T15:31:59.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6ac9000640 1 --2- 172.21.15.47:0/905134036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ac406d650 0x7f6ac40d08b0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f6ab80097c0 tx=0x7f6ab801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39ecedb47c20d264 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:59.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6ac3400640 1 -- 172.21.15.47:0/905134036 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab80ab600 con 0x7f6ac406d650 2024-02-19T15:31:59.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6ac3400640 1 -- 172.21.15.47:0/905134036 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab80a7050 con 0x7f6ac406d650 2024-02-19T15:31:59.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6ac3400640 1 -- 172.21.15.47:0/905134036 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab80ab980 con 0x7f6ac406d650 2024-02-19T15:31:59.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6acba00640 1 -- 172.21.15.47:0/905134036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ac406d650 msgr2=0x7f6ac40d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:59.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6acba00640 1 --2- 172.21.15.47:0/905134036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ac406d650 0x7f6ac40d08b0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f6ab80097c0 tx=0x7f6ab801c3e0 comp rx=0 tx=0).stop 2024-02-19T15:31:59.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6acba00640 1 -- 172.21.15.47:0/905134036 shutdown_connections 2024-02-19T15:31:59.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6acba00640 1 --2- 172.21.15.47:0/905134036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ac40cded0 0x7f6ac40ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:59.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6acba00640 1 --2- 172.21.15.47:0/905134036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ac406d650 0x7f6ac40d08b0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:59.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6acba00640 1 -- 172.21.15.47:0/905134036 >> 172.21.15.47:0/905134036 conn(0x7f6ac40d1bf0 msgr2=0x7f6ac40d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:31:59.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6acba00640 1 -- 172.21.15.47:0/905134036 shutdown_connections 2024-02-19T15:31:59.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6acba00640 1 -- 172.21.15.47:0/905134036 wait complete. 2024-02-19T15:31:59.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6acba00640 1 Processor -- start 2024-02-19T15:31:59.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6acba00640 1 -- start start 2024-02-19T15:31:59.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6acba00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ac406d650 0x7f6ac40ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:59.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6acba00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ac40cded0 0x7f6ac415ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:59.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6acba00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ac40ceb40 0x7f6ac415b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:59.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6acba00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac415b910 con 0x7f6ac406d650 2024-02-19T15:31:59.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6acba00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac415d370 con 0x7f6ac40ceb40 2024-02-19T15:31:59.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6acba00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac415d0d0 con 0x7f6ac40cded0 2024-02-19T15:31:59.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6ac9000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ac406d650 0x7f6ac40ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:59.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6ac9000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ac406d650 0x7f6ac40ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43852/0 (socket says 172.21.15.47:43852) 2024-02-19T15:31:59.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.972+0000 7f6ac9000640 1 -- 172.21.15.47:0/2316279346 learned_addr learned my addr 172.21.15.47:0/2316279346 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:31:59.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.976+0000 7f6ac3e00640 1 --2- 172.21.15.47:0/2316279346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ac40cded0 0x7f6ac415ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:59.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.976+0000 7f6ac9000640 1 -- 172.21.15.47:0/2316279346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ac40cded0 msgr2=0x7f6ac415ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:59.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.976+0000 7f6ac9000640 1 --2- 172.21.15.47:0/2316279346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ac40cded0 0x7f6ac415ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:59.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.976+0000 7f6ac9a00640 1 --2- 172.21.15.47:0/2316279346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ac40ceb40 0x7f6ac415b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:59.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.976+0000 7f6ac9000640 1 -- 172.21.15.47:0/2316279346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ac40ceb40 msgr2=0x7f6ac415b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:31:59.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.976+0000 7f6ac9000640 1 --2- 172.21.15.47:0/2316279346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ac40ceb40 0x7f6ac415b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:31:59.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.976+0000 7f6ac9000640 1 -- 172.21.15.47:0/2316279346 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab8009420 con 0x7f6ac406d650 2024-02-19T15:31:59.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.976+0000 7f6ac9a00640 1 --2- 172.21.15.47:0/2316279346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ac40ceb40 0x7f6ac415b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:31:59.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.976+0000 7f6ac9000640 1 --2- 172.21.15.47:0/2316279346 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ac406d650 0x7f6ac40ca2a0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f6ab80ae040 tx=0x7f6ab801ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:59.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.976+0000 7f6ac0c00640 1 -- 172.21.15.47:0/2316279346 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab80b7650 con 0x7f6ac406d650 2024-02-19T15:31:59.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.976+0000 7f6ac0c00640 1 -- 172.21.15.47:0/2316279346 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab80abe90 con 0x7f6ac406d650 2024-02-19T15:31:59.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.976+0000 7f6ac0c00640 1 -- 172.21.15.47:0/2316279346 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab80b7a10 con 0x7f6ac406d650 2024-02-19T15:31:59.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.976+0000 7f6acba00640 1 -- 172.21.15.47:0/2316279346 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac40c9800 con 0x7f6ac406d650 2024-02-19T15:31:59.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.976+0000 7f6acba00640 1 -- 172.21.15.47:0/2316279346 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac40c9d80 con 0x7f6ac406d650 2024-02-19T15:31:59.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.976+0000 7f6acba00640 1 -- 172.21.15.47:0/2316279346 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac4075360 con 0x7f6ac406d650 2024-02-19T15:31:59.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.976+0000 7f6ac0c00640 1 -- 172.21.15.47:0/2316279346 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6ab80b7b70 con 0x7f6ac406d650 2024-02-19T15:31:59.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.976+0000 7f6ac0c00640 1 --2- 172.21.15.47:0/2316279346 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6aa4066e70 0x7f6aa4069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:31:59.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.976+0000 7f6ac0c00640 1 -- 172.21.15.47:0/2316279346 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6ab811f2d0 con 0x7f6ac406d650 2024-02-19T15:31:59.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.980+0000 7f6ac3e00640 1 --2- 172.21.15.47:0/2316279346 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6aa4066e70 0x7f6aa4069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:31:59.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.980+0000 7f6ac3e00640 1 --2- 172.21.15.47:0/2316279346 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6aa4066e70 0x7f6aa4069330 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f6ac409f460 tx=0x7f6ab0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:31:59.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:31:59.980+0000 7f6ac0c00640 1 -- 172.21.15.47:0/2316279346 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6ab80ec070 con 0x7f6ac406d650 2024-02-19T15:32:00.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:00.176+0000 7f6acba00640 1 -- 172.21.15.47:0/2316279346 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ac40d2490 con 0x7f6ac406d650 2024-02-19T15:32:00.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:00.180+0000 7f6ac0c00640 1 -- 172.21.15.47:0/2316279346 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f6ab80bbd80 con 0x7f6ac406d650 2024-02-19T15:32:00.179 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:00.179 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:00.179 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:00.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:00.180+0000 7f6acba00640 1 -- 172.21.15.47:0/2316279346 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6aa4066e70 msgr2=0x7f6aa4069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:00.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:00.180+0000 7f6acba00640 1 --2- 172.21.15.47:0/2316279346 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6aa4066e70 0x7f6aa4069330 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f6ac409f460 tx=0x7f6ab0008040 comp rx=0 tx=0).stop 2024-02-19T15:32:00.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:00.180+0000 7f6acba00640 1 -- 172.21.15.47:0/2316279346 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ac406d650 msgr2=0x7f6ac40ca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:00.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:00.180+0000 7f6acba00640 1 --2- 172.21.15.47:0/2316279346 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ac406d650 0x7f6ac40ca2a0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f6ab80ae040 tx=0x7f6ab801ac20 comp rx=0 tx=0).stop 2024-02-19T15:32:00.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:00.180+0000 7f6acba00640 1 -- 172.21.15.47:0/2316279346 shutdown_connections 2024-02-19T15:32:00.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:00.180+0000 7f6acba00640 1 --2- 172.21.15.47:0/2316279346 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6aa4066e70 0x7f6aa4069330 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:00.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:00.180+0000 7f6acba00640 1 --2- 172.21.15.47:0/2316279346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ac40ceb40 0x7f6ac415b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:00.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:00.180+0000 7f6acba00640 1 --2- 172.21.15.47:0/2316279346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ac40cded0 0x7f6ac415ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:00.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:00.180+0000 7f6acba00640 1 --2- 172.21.15.47:0/2316279346 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ac406d650 0x7f6ac40ca2a0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:00.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:00.180+0000 7f6acba00640 1 -- 172.21.15.47:0/2316279346 >> 172.21.15.47:0/2316279346 conn(0x7f6ac40d1bf0 msgr2=0x7f6ac40d06d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:00.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:00.180+0000 7f6acba00640 1 -- 172.21.15.47:0/2316279346 shutdown_connections 2024-02-19T15:32:00.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:00.180+0000 7f6acba00640 1 -- 172.21.15.47:0/2316279346 wait complete. 2024-02-19T15:32:00.290 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:32:00.402 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:00.402 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:01.403 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:01.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.544+0000 7feaf6200640 1 Processor -- start 2024-02-19T15:32:01.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.544+0000 7feaf6200640 1 -- start start 2024-02-19T15:32:01.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.548+0000 7feaf6200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7feaf015bcd0 0x7feaf00cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:01.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.548+0000 7feaf6200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf00cc700 con 0x7feaf00d0360 2024-02-19T15:32:01.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.548+0000 7feaf6200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf00cf300 con 0x7feaf00cedb0 2024-02-19T15:32:01.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.548+0000 7feaf6200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaf00cf5e0 con 0x7feaf015bcd0 2024-02-19T15:32:01.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.548+0000 7feaef400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7feaf015bcd0 0x7feaf00cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:01.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.548+0000 7feaefe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7feaf00d0360 0x7feaf015d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36268/0 (socket says 172.21.15.47:36268) 2024-02-19T15:32:01.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.548+0000 7feaefe00640 1 -- 172.21.15.47:0/2297858648 learned_addr learned my addr 172.21.15.47:0/2297858648 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:32:01.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.548+0000 7feaef400640 1 -- 172.21.15.47:0/2297858648 >> v1:172.21.15.116:6789/0 conn(0x7feaf00cedb0 legacy=0x7feaf00cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:01.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.548+0000 7feaef400640 1 -- 172.21.15.47:0/2297858648 >> v1:172.21.15.47:6789/0 conn(0x7feaf00d0360 legacy=0x7feaf015d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:01.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.548+0000 7feaef400640 1 -- 172.21.15.47:0/2297858648 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf00cc9a0 con 0x7feaf015bcd0 2024-02-19T15:32:01.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.548+0000 7feaeea00640 1 -- 172.21.15.47:0/2297858648 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522391778 0 0) 0x7feaf00cc700 con 0x7feaf00d0360 2024-02-19T15:32:01.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.552+0000 7feaef400640 1 --2- 172.21.15.47:0/2297858648 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf015bcd0 0x7feaf00cd480 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7feae00097c0 tx=0x7feae001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76078fbefcc00b86 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:01.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.552+0000 7feaeea00640 1 -- 172.21.15.47:0/2297858648 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae00ab670 con 0x7feaf015bcd0 2024-02-19T15:32:01.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.552+0000 7feaeea00640 1 -- 172.21.15.47:0/2297858648 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae00ab7d0 con 0x7feaf015bcd0 2024-02-19T15:32:01.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.552+0000 7feaeea00640 1 -- 172.21.15.47:0/2297858648 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae00abb60 con 0x7feaf015bcd0 2024-02-19T15:32:01.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.552+0000 7feaf6200640 1 -- 172.21.15.47:0/2297858648 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf015bcd0 msgr2=0x7feaf00cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:01.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.552+0000 7feaf6200640 1 --2- 172.21.15.47:0/2297858648 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf015bcd0 0x7feaf00cd480 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7feae00097c0 tx=0x7feae001c470 comp rx=0 tx=0).stop 2024-02-19T15:32:01.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.552+0000 7feaf6200640 1 -- 172.21.15.47:0/2297858648 shutdown_connections 2024-02-19T15:32:01.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.552+0000 7feaf6200640 1 --2- 172.21.15.47:0/2297858648 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf015bcd0 0x7feaf00cd480 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:01.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.552+0000 7feaf6200640 1 -- 172.21.15.47:0/2297858648 >> 172.21.15.47:0/2297858648 conn(0x7feaf00d1bf0 msgr2=0x7feaf00d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:01.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.552+0000 7feaf6200640 1 -- 172.21.15.47:0/2297858648 shutdown_connections 2024-02-19T15:32:01.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.552+0000 7feaf6200640 1 -- 172.21.15.47:0/2297858648 wait complete. 2024-02-19T15:32:01.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feaf6200640 1 Processor -- start 2024-02-19T15:32:01.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feaf6200640 1 -- start start 2024-02-19T15:32:01.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feaf6200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00cedb0 0x7feaf0146fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:01.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feaf6200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaf00d0360 0x7feaf00ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:01.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feaf6200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feaf015bcd0 0x7feaf00cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:01.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feaf6200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf00cb820 con 0x7feaf00cedb0 2024-02-19T15:32:01.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feaf6200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf00cac30 con 0x7feaf00d0360 2024-02-19T15:32:01.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feaf6200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf0147510 con 0x7feaf015bcd0 2024-02-19T15:32:01.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feaef400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00cedb0 0x7feaf0146fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:01.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feaef400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00cedb0 0x7feaf0146fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43864/0 (socket says 172.21.15.47:43864) 2024-02-19T15:32:01.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feaef400640 1 -- 172.21.15.47:0/4009595982 learned_addr learned my addr 172.21.15.47:0/4009595982 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:01.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feae7e00640 1 --2- 172.21.15.47:0/4009595982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaf00d0360 0x7feaf00ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:01.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feaefe00640 1 --2- 172.21.15.47:0/4009595982 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feaf015bcd0 0x7feaf00cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:01.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feaef400640 1 -- 172.21.15.47:0/4009595982 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feaf015bcd0 msgr2=0x7feaf00cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:01.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feaef400640 1 --2- 172.21.15.47:0/4009595982 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feaf015bcd0 0x7feaf00cdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:01.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feaef400640 1 -- 172.21.15.47:0/4009595982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaf00d0360 msgr2=0x7feaf00ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:01.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feaef400640 1 --2- 172.21.15.47:0/4009595982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaf00d0360 0x7feaf00ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:01.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feaef400640 1 -- 172.21.15.47:0/4009595982 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae0009420 con 0x7feaf00cedb0 2024-02-19T15:32:01.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feaef400640 1 --2- 172.21.15.47:0/4009595982 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00cedb0 0x7feaf0146fd0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7feae00ad040 tx=0x7feae001acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:01.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feae7400640 1 -- 172.21.15.47:0/4009595982 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae00b5810 con 0x7feaf00cedb0 2024-02-19T15:32:01.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feae7400640 1 -- 172.21.15.47:0/4009595982 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae00b5990 con 0x7feaf00cedb0 2024-02-19T15:32:01.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feaf6200640 1 -- 172.21.15.47:0/4009595982 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaf00cdb40 con 0x7feaf00cedb0 2024-02-19T15:32:01.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feae7400640 1 -- 172.21.15.47:0/4009595982 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae00b5d40 con 0x7feaf00cedb0 2024-02-19T15:32:01.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.556+0000 7feaf6200640 1 -- 172.21.15.47:0/4009595982 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaf00cb160 con 0x7feaf00cedb0 2024-02-19T15:32:01.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.560+0000 7feaf6200640 1 -- 172.21.15.47:0/4009595982 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaf0075360 con 0x7feaf00cedb0 2024-02-19T15:32:01.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.560+0000 7feae7400640 1 -- 172.21.15.47:0/4009595982 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7feae00bf410 con 0x7feaf00cedb0 2024-02-19T15:32:01.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.560+0000 7feae7400640 1 --2- 172.21.15.47:0/4009595982 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feacc066e40 0x7feacc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:01.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.560+0000 7feae7400640 1 -- 172.21.15.47:0/4009595982 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7feae011fd80 con 0x7feaf00cedb0 2024-02-19T15:32:01.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.560+0000 7feae7e00640 1 --2- 172.21.15.47:0/4009595982 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feacc066e40 0x7feacc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:01.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.560+0000 7feae7e00640 1 --2- 172.21.15.47:0/4009595982 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feacc066e40 0x7feacc069300 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7feaf015b6c0 tx=0x7fead8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:01.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.568+0000 7feae7400640 1 -- 172.21.15.47:0/4009595982 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7feae00ecbd0 con 0x7feaf00cedb0 2024-02-19T15:32:01.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.764+0000 7feaf6200640 1 -- 172.21.15.47:0/4009595982 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feaf006ef50 con 0x7feaf00cedb0 2024-02-19T15:32:01.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.764+0000 7feae7400640 1 -- 172.21.15.47:0/4009595982 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7feae00f0290 con 0x7feaf00cedb0 2024-02-19T15:32:01.764 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:01.764 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:01.764 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.764+0000 7feaf6200640 1 -- 172.21.15.47:0/4009595982 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feacc066e40 msgr2=0x7feacc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.764+0000 7feaf6200640 1 --2- 172.21.15.47:0/4009595982 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feacc066e40 0x7feacc069300 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7feaf015b6c0 tx=0x7fead8009040 comp rx=0 tx=0).stop 2024-02-19T15:32:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.764+0000 7feaf6200640 1 -- 172.21.15.47:0/4009595982 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00cedb0 msgr2=0x7feaf0146fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.764+0000 7feaf6200640 1 --2- 172.21.15.47:0/4009595982 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00cedb0 0x7feaf0146fd0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7feae00ad040 tx=0x7feae001acb0 comp rx=0 tx=0).stop 2024-02-19T15:32:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.764+0000 7feaf6200640 1 -- 172.21.15.47:0/4009595982 shutdown_connections 2024-02-19T15:32:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.764+0000 7feaf6200640 1 --2- 172.21.15.47:0/4009595982 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7feacc066e40 0x7feacc069300 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.764+0000 7feaf6200640 1 --2- 172.21.15.47:0/4009595982 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feaf015bcd0 0x7feaf00cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.764+0000 7feaf6200640 1 --2- 172.21.15.47:0/4009595982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaf00d0360 0x7feaf00ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.764+0000 7feaf6200640 1 --2- 172.21.15.47:0/4009595982 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feaf00cedb0 0x7feaf0146fd0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:01.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.764+0000 7feaf6200640 1 -- 172.21.15.47:0/4009595982 >> 172.21.15.47:0/4009595982 conn(0x7feaf00d1bf0 msgr2=0x7feaf00d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:01.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.764+0000 7feaf6200640 1 -- 172.21.15.47:0/4009595982 shutdown_connections 2024-02-19T15:32:01.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:01.768+0000 7feaf6200640 1 -- 172.21.15.47:0/4009595982 wait complete. 2024-02-19T15:32:01.987 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:01.987 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:02.292 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:32:02.987 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:03.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.128+0000 7f7e7b400640 1 Processor -- start 2024-02-19T15:32:03.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.128+0000 7f7e7b400640 1 -- start start 2024-02-19T15:32:03.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.128+0000 7f7e7b400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e7415b1a0 0x7f7e7415b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:03.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.128+0000 7f7e7b400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7e74071350 0x7f7e7415bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:03.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.128+0000 7f7e7b400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7e7406c9a0 0x7f7e7415c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:03.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.128+0000 7f7e7b400640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7415c000 con 0x7f7e74071350 2024-02-19T15:32:03.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.128+0000 7f7e7b400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e740718c0 con 0x7f7e7406c9a0 2024-02-19T15:32:03.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.128+0000 7f7e7b400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e74071ba0 con 0x7f7e7415b1a0 2024-02-19T15:32:03.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.128+0000 7f7e73e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7e74071350 0x7f7e7415bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:03.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.128+0000 7f7e73e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7e74071350 0x7f7e7415bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43874/0 (socket says 172.21.15.47:43874) 2024-02-19T15:32:03.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.128+0000 7f7e73e00640 1 -- 172.21.15.47:0/3184222237 learned_addr learned my addr 172.21.15.47:0/3184222237 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:03.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.128+0000 7f7e78a00640 1 --2- 172.21.15.47:0/3184222237 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e7415b1a0 0x7f7e7415b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:03.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.128+0000 7f7e79400640 1 --2- 172.21.15.47:0/3184222237 >> v2:172.21.15.116:3300/0 conn(0x7f7e7406c9a0 0x7f7e7415c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:03.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.128+0000 7f7e73e00640 1 -- 172.21.15.47:0/3184222237 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e7415b1a0 msgr2=0x7f7e7415b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:03.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.128+0000 7f7e73e00640 1 --2- 172.21.15.47:0/3184222237 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e7415b1a0 0x7f7e7415b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:03.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.128+0000 7f7e73e00640 1 -- 172.21.15.47:0/3184222237 >> v2:172.21.15.116:3300/0 conn(0x7f7e7406c9a0 msgr2=0x7f7e7415c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:03.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.128+0000 7f7e73e00640 1 --2- 172.21.15.47:0/3184222237 >> v2:172.21.15.116:3300/0 conn(0x7f7e7406c9a0 0x7f7e7415c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:03.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.128+0000 7f7e73e00640 1 -- 172.21.15.47:0/3184222237 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e74071e80 con 0x7f7e74071350 2024-02-19T15:32:03.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.128+0000 7f7e79400640 1 --2- 172.21.15.47:0/3184222237 >> v2:172.21.15.116:3300/0 conn(0x7f7e7406c9a0 0x7f7e7415c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:32:03.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.132+0000 7f7e73e00640 1 --2- 172.21.15.47:0/3184222237 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e74071350 0x7f7e7415bac0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f7e74072440 tx=0x7f7e6001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=574e7581e2d0ddbb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:03.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.132+0000 7f7e73400640 1 -- 172.21.15.47:0/3184222237 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e600aa430 con 0x7f7e74071350 2024-02-19T15:32:03.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.132+0000 7f7e73400640 1 -- 172.21.15.47:0/3184222237 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e600aa590 con 0x7f7e74071350 2024-02-19T15:32:03.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.132+0000 7f7e73400640 1 -- 172.21.15.47:0/3184222237 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e600aa920 con 0x7f7e74071350 2024-02-19T15:32:03.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.132+0000 7f7e7b400640 1 -- 172.21.15.47:0/3184222237 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e74071350 msgr2=0x7f7e7415bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:03.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.132+0000 7f7e7b400640 1 --2- 172.21.15.47:0/3184222237 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e74071350 0x7f7e7415bac0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f7e74072440 tx=0x7f7e6001c4f0 comp rx=0 tx=0).stop 2024-02-19T15:32:03.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.136+0000 7f7e7b400640 1 -- 172.21.15.47:0/3184222237 shutdown_connections 2024-02-19T15:32:03.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.136+0000 7f7e7b400640 1 --2- 172.21.15.47:0/3184222237 >> v2:172.21.15.116:3300/0 conn(0x7f7e7406c9a0 0x7f7e7415c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:03.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.136+0000 7f7e7b400640 1 --2- 172.21.15.47:0/3184222237 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e74071350 0x7f7e7415bac0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:03.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.136+0000 7f7e7b400640 1 --2- 172.21.15.47:0/3184222237 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e7415b1a0 0x7f7e7415b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:03.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.136+0000 7f7e7b400640 1 -- 172.21.15.47:0/3184222237 >> 172.21.15.47:0/3184222237 conn(0x7f7e740b16a0 msgr2=0x7f7e740b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:03.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.136+0000 7f7e7b400640 1 -- 172.21.15.47:0/3184222237 shutdown_connections 2024-02-19T15:32:03.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.136+0000 7f7e7b400640 1 -- 172.21.15.47:0/3184222237 wait complete. 2024-02-19T15:32:03.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.136+0000 7f7e7b400640 1 Processor -- start 2024-02-19T15:32:03.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.136+0000 7f7e7b400640 1 -- start start 2024-02-19T15:32:03.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e7b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e7406c9a0 0x7f7e74140340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:03.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e7b400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e74071350 0x7f7e74154020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:03.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e7b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e7415b1a0 0x7f7e74154560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:03.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e7b400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e741f7a40 con 0x7f7e7406c9a0 2024-02-19T15:32:03.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e7b400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7414f000 con 0x7f7e7415b1a0 2024-02-19T15:32:03.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e7b400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e741f7c90 con 0x7f7e74071350 2024-02-19T15:32:03.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e78a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e7406c9a0 0x7f7e74140340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:03.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e78a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e7406c9a0 0x7f7e74140340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43876/0 (socket says 172.21.15.47:43876) 2024-02-19T15:32:03.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e78a00640 1 -- 172.21.15.47:0/3903618057 learned_addr learned my addr 172.21.15.47:0/3903618057 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:03.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e73e00640 1 --2- 172.21.15.47:0/3903618057 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e74071350 0x7f7e74154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:03.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e79400640 1 --2- 172.21.15.47:0/3903618057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e7415b1a0 0x7f7e74154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:03.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e78a00640 1 -- 172.21.15.47:0/3903618057 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e74071350 msgr2=0x7f7e74154020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:03.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e78a00640 1 --2- 172.21.15.47:0/3903618057 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e74071350 0x7f7e74154020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:03.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e78a00640 1 -- 172.21.15.47:0/3903618057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e7415b1a0 msgr2=0x7f7e74154560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:03.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e78a00640 1 --2- 172.21.15.47:0/3903618057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e7415b1a0 0x7f7e74154560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:03.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e78a00640 1 -- 172.21.15.47:0/3903618057 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e600094d0 con 0x7f7e7406c9a0 2024-02-19T15:32:03.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e78a00640 1 --2- 172.21.15.47:0/3903618057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e7406c9a0 0x7f7e74140340 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800e930 tx=0x7f7e6800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:03.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e70c00640 1 -- 172.21.15.47:0/3903618057 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e68009690 con 0x7f7e7406c9a0 2024-02-19T15:32:03.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e70c00640 1 -- 172.21.15.47:0/3903618057 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6800ca00 con 0x7f7e7406c9a0 2024-02-19T15:32:03.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e70c00640 1 -- 172.21.15.47:0/3903618057 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6800cd70 con 0x7f7e7406c9a0 2024-02-19T15:32:03.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e7b400640 1 -- 172.21.15.47:0/3903618057 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e741f7f70 con 0x7f7e7406c9a0 2024-02-19T15:32:03.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e7b400640 1 -- 172.21.15.47:0/3903618057 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e74140880 con 0x7f7e7406c9a0 2024-02-19T15:32:03.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e7b400640 1 -- 172.21.15.47:0/3903618057 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e740cbee0 con 0x7f7e7406c9a0 2024-02-19T15:32:03.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.140+0000 7f7e70c00640 1 -- 172.21.15.47:0/3903618057 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f7e68013470 con 0x7f7e7406c9a0 2024-02-19T15:32:03.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.144+0000 7f7e70c00640 1 --2- 172.21.15.47:0/3903618057 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7e54066e40 0x7f7e54069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:03.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.144+0000 7f7e70c00640 1 -- 172.21.15.47:0/3903618057 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7e68084d10 con 0x7f7e7406c9a0 2024-02-19T15:32:03.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.144+0000 7f7e73e00640 1 --2- 172.21.15.47:0/3903618057 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7e54066e40 0x7f7e54069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:03.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.144+0000 7f7e73e00640 1 --2- 172.21.15.47:0/3903618057 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7e54066e40 0x7f7e54069300 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f7e600ae000 tx=0x7f7e600a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:03.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.148+0000 7f7e70c00640 1 -- 172.21.15.47:0/3903618057 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f7e6808a020 con 0x7f7e7406c9a0 2024-02-19T15:32:03.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.344+0000 7f7e7b400640 1 -- 172.21.15.47:0/3903618057 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e74075360 con 0x7f7e7406c9a0 2024-02-19T15:32:03.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.344+0000 7f7e70c00640 1 -- 172.21.15.47:0/3903618057 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f7e68021d80 con 0x7f7e7406c9a0 2024-02-19T15:32:03.343 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:03.343 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:03.343 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:03.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.344+0000 7f7e7b400640 1 -- 172.21.15.47:0/3903618057 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7e54066e40 msgr2=0x7f7e54069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:03.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.344+0000 7f7e7b400640 1 --2- 172.21.15.47:0/3903618057 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7e54066e40 0x7f7e54069300 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f7e600ae000 tx=0x7f7e600a7000 comp rx=0 tx=0).stop 2024-02-19T15:32:03.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.344+0000 7f7e7b400640 1 -- 172.21.15.47:0/3903618057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e7406c9a0 msgr2=0x7f7e74140340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:03.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.344+0000 7f7e7b400640 1 --2- 172.21.15.47:0/3903618057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e7406c9a0 0x7f7e74140340 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800e930 tx=0x7f7e6800edf0 comp rx=0 tx=0).stop 2024-02-19T15:32:03.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.344+0000 7f7e7b400640 1 -- 172.21.15.47:0/3903618057 shutdown_connections 2024-02-19T15:32:03.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.344+0000 7f7e7b400640 1 --2- 172.21.15.47:0/3903618057 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f7e54066e40 0x7f7e54069300 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:03.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.344+0000 7f7e7b400640 1 --2- 172.21.15.47:0/3903618057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e7415b1a0 0x7f7e74154560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:03.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.344+0000 7f7e7b400640 1 --2- 172.21.15.47:0/3903618057 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e74071350 0x7f7e74154020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:03.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.344+0000 7f7e7b400640 1 --2- 172.21.15.47:0/3903618057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e7406c9a0 0x7f7e74140340 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:03.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.344+0000 7f7e7b400640 1 -- 172.21.15.47:0/3903618057 >> 172.21.15.47:0/3903618057 conn(0x7f7e740b16a0 msgr2=0x7f7e7415b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:03.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.344+0000 7f7e7b400640 1 -- 172.21.15.47:0/3903618057 shutdown_connections 2024-02-19T15:32:03.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:03.344+0000 7f7e7b400640 1 -- 172.21.15.47:0/3903618057 wait complete. 2024-02-19T15:32:03.566 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:03.567 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:04.294 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:32:04.567 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:04.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.708+0000 7f0fa0000640 1 Processor -- start 2024-02-19T15:32:04.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.708+0000 7f0fa0000640 1 -- start start 2024-02-19T15:32:04.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.708+0000 7f0fa0000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0f9815b1a0 0x7f0f9815b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:04.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.708+0000 7f0fa0000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f980718c0 con 0x7f0f98071350 2024-02-19T15:32:04.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.708+0000 7f0fa0000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f98071ba0 con 0x7f0f9806c9a0 2024-02-19T15:32:04.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.708+0000 7f0fa0000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98071e80 con 0x7f0f9815b1a0 2024-02-19T15:32:04.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.708+0000 7f0f9d600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0f9815b1a0 0x7f0f9815b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:04.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.708+0000 7f0f9cc00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0f98071350 0x7f0f9815c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37916/0 (socket says 172.21.15.47:37916) 2024-02-19T15:32:04.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.708+0000 7f0f9cc00640 1 -- 172.21.15.47:0/3694749235 learned_addr learned my addr 172.21.15.47:0/3694749235 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:32:04.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.708+0000 7f0f9d600640 1 -- 172.21.15.47:0/3694749235 >> v1:172.21.15.116:6789/0 conn(0x7f0f9806c9a0 legacy=0x7f0f9815cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:04.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.708+0000 7f0f9d600640 1 -- 172.21.15.47:0/3694749235 >> v1:172.21.15.47:6789/0 conn(0x7f0f98071350 legacy=0x7f0f9815c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:04.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.708+0000 7f0f9d600640 1 -- 172.21.15.47:0/3694749235 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f98072160 con 0x7f0f9815b1a0 2024-02-19T15:32:04.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.712+0000 7f0f9d600640 1 --2- 172.21.15.47:0/3694749235 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f9815b1a0 0x7f0f9815b580 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0097f0 tx=0x7f0f8c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=534968eca3eab5e4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:04.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.712+0000 7f0f97e00640 1 -- 172.21.15.47:0/3694749235 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c0a7070 con 0x7f0f9815b1a0 2024-02-19T15:32:04.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.712+0000 7f0f97e00640 1 -- 172.21.15.47:0/3694749235 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8c0ab430 con 0x7f0f9815b1a0 2024-02-19T15:32:04.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.712+0000 7f0f97e00640 1 -- 172.21.15.47:0/3694749235 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c0ab7c0 con 0x7f0f9815b1a0 2024-02-19T15:32:04.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.712+0000 7f0fa0000640 1 -- 172.21.15.47:0/3694749235 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f9815b1a0 msgr2=0x7f0f9815b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:04.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.712+0000 7f0fa0000640 1 --2- 172.21.15.47:0/3694749235 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f9815b1a0 0x7f0f9815b580 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0097f0 tx=0x7f0f8c01c4a0 comp rx=0 tx=0).stop 2024-02-19T15:32:04.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.712+0000 7f0fa0000640 1 -- 172.21.15.47:0/3694749235 shutdown_connections 2024-02-19T15:32:04.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.712+0000 7f0fa0000640 1 --2- 172.21.15.47:0/3694749235 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f9815b1a0 0x7f0f9815b580 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:04.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.712+0000 7f0fa0000640 1 -- 172.21.15.47:0/3694749235 >> 172.21.15.47:0/3694749235 conn(0x7f0f980b16a0 msgr2=0x7f0f980b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:04.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.712+0000 7f0fa0000640 1 -- 172.21.15.47:0/3694749235 shutdown_connections 2024-02-19T15:32:04.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.712+0000 7f0fa0000640 1 -- 172.21.15.47:0/3694749235 wait complete. 2024-02-19T15:32:04.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.712+0000 7f0fa0000640 1 Processor -- start 2024-02-19T15:32:04.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0fa0000640 1 -- start start 2024-02-19T15:32:04.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0fa0000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f9806c9a0 0x7f0f980ce380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:04.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0fa0000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f98071350 0x7f0f9814d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:04.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0fa0000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f9815b1a0 0x7f0f980b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:04.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0fa0000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980ae250 con 0x7f0f9815b1a0 2024-02-19T15:32:04.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0fa0000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980ae530 con 0x7f0f9806c9a0 2024-02-19T15:32:04.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0fa0000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980ae810 con 0x7f0f98071350 2024-02-19T15:32:04.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0f9e000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f9815b1a0 0x7f0f980b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:04.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0f9e000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f9815b1a0 0x7f0f980b3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:41650/0 (socket says 172.21.15.47:41650) 2024-02-19T15:32:04.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0f9e000640 1 -- 172.21.15.47:0/3434981082 learned_addr learned my addr 172.21.15.47:0/3434981082 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:04.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0f9d600640 1 --2- 172.21.15.47:0/3434981082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f9806c9a0 0x7f0f980ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:04.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0f9cc00640 1 --2- 172.21.15.47:0/3434981082 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f98071350 0x7f0f9814d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:04.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0f9e000640 1 -- 172.21.15.47:0/3434981082 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f98071350 msgr2=0x7f0f9814d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:04.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0f9e000640 1 --2- 172.21.15.47:0/3434981082 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f98071350 0x7f0f9814d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:04.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0f9e000640 1 -- 172.21.15.47:0/3434981082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f9806c9a0 msgr2=0x7f0f980ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:04.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0f9e000640 1 --2- 172.21.15.47:0/3434981082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f9806c9a0 0x7f0f980ce380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:04.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0f9e000640 1 -- 172.21.15.47:0/3434981082 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8c009420 con 0x7f0f9815b1a0 2024-02-19T15:32:04.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0f9e000640 1 --2- 172.21.15.47:0/3434981082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f9815b1a0 0x7f0f980b3d60 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f0f9000c730 tx=0x7f0f9000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:04.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0f95600640 1 -- 172.21.15.47:0/3434981082 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f90003b80 con 0x7f0f9815b1a0 2024-02-19T15:32:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0f95600640 1 -- 172.21.15.47:0/3434981082 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f90003ce0 con 0x7f0f9815b1a0 2024-02-19T15:32:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0f95600640 1 -- 172.21.15.47:0/3434981082 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9000aa30 con 0x7f0f9815b1a0 2024-02-19T15:32:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0fa0000640 1 -- 172.21.15.47:0/3434981082 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f980aeaf0 con 0x7f0f9815b1a0 2024-02-19T15:32:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0fa0000640 1 -- 172.21.15.47:0/3434981082 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f980af0b0 con 0x7f0f9815b1a0 2024-02-19T15:32:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.716+0000 7f0fa0000640 1 -- 172.21.15.47:0/3434981082 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f981f76f0 con 0x7f0f9815b1a0 2024-02-19T15:32:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.720+0000 7f0f95600640 1 -- 172.21.15.47:0/3434981082 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f0f9000ab90 con 0x7f0f9815b1a0 2024-02-19T15:32:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.720+0000 7f0f95600640 1 --2- 172.21.15.47:0/3434981082 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0f78066e40 0x7f0f78069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.720+0000 7f0f95600640 1 -- 172.21.15.47:0/3434981082 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0f90084710 con 0x7f0f9815b1a0 2024-02-19T15:32:04.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.720+0000 7f0f9d600640 1 --2- 172.21.15.47:0/3434981082 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0f78066e40 0x7f0f78069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:04.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.720+0000 7f0f9d600640 1 --2- 172.21.15.47:0/3434981082 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0f78066e40 0x7f0f78069300 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0af000 tx=0x7f0f8c0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:04.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.724+0000 7f0f95600640 1 -- 172.21.15.47:0/3434981082 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f0f90051570 con 0x7f0f9815b1a0 2024-02-19T15:32:04.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.920+0000 7f0fa0000640 1 -- 172.21.15.47:0/3434981082 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f980b0820 con 0x7f0f9815b1a0 2024-02-19T15:32:04.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.920+0000 7f0f95600640 1 -- 172.21.15.47:0/3434981082 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f0f90054c30 con 0x7f0f9815b1a0 2024-02-19T15:32:04.919 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:04.919 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:04.920 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:04.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.920+0000 7f0fa0000640 1 -- 172.21.15.47:0/3434981082 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0f78066e40 msgr2=0x7f0f78069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:04.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.920+0000 7f0fa0000640 1 --2- 172.21.15.47:0/3434981082 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0f78066e40 0x7f0f78069300 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0af000 tx=0x7f0f8c0a7000 comp rx=0 tx=0).stop 2024-02-19T15:32:04.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.920+0000 7f0fa0000640 1 -- 172.21.15.47:0/3434981082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f9815b1a0 msgr2=0x7f0f980b3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:04.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.920+0000 7f0fa0000640 1 --2- 172.21.15.47:0/3434981082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f9815b1a0 0x7f0f980b3d60 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f0f9000c730 tx=0x7f0f9000cbf0 comp rx=0 tx=0).stop 2024-02-19T15:32:04.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.920+0000 7f0fa0000640 1 -- 172.21.15.47:0/3434981082 shutdown_connections 2024-02-19T15:32:04.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.920+0000 7f0fa0000640 1 --2- 172.21.15.47:0/3434981082 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f0f78066e40 0x7f0f78069300 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:04.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.920+0000 7f0fa0000640 1 --2- 172.21.15.47:0/3434981082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f9815b1a0 0x7f0f980b3d60 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:04.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.920+0000 7f0fa0000640 1 --2- 172.21.15.47:0/3434981082 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f98071350 0x7f0f9814d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:04.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.920+0000 7f0fa0000640 1 --2- 172.21.15.47:0/3434981082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f9806c9a0 0x7f0f980ce380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:04.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.920+0000 7f0fa0000640 1 -- 172.21.15.47:0/3434981082 >> 172.21.15.47:0/3434981082 conn(0x7f0f980b16a0 msgr2=0x7f0f980b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:04.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.920+0000 7f0fa0000640 1 -- 172.21.15.47:0/3434981082 shutdown_connections 2024-02-19T15:32:04.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:04.920+0000 7f0fa0000640 1 -- 172.21.15.47:0/3434981082 wait complete. 2024-02-19T15:32:05.144 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:05.144 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:06.145 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:06.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.284+0000 7fad97400640 1 Processor -- start 2024-02-19T15:32:06.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.284+0000 7fad97400640 1 -- start start 2024-02-19T15:32:06.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.288+0000 7fad97400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fad9006cfc0 0x7fad900b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:06.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.288+0000 7fad97400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad9006c660 con 0x7fad9006c1d0 2024-02-19T15:32:06.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.288+0000 7fad97400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad900b2270 con 0x7fad9015e180 2024-02-19T15:32:06.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.288+0000 7fad97400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad900b2550 con 0x7fad9006cfc0 2024-02-19T15:32:06.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.288+0000 7fad95400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fad9006c1d0 0x7fad900ae870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37920/0 (socket says 172.21.15.47:37920) 2024-02-19T15:32:06.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.288+0000 7fad95400640 1 -- 172.21.15.47:0/3730089969 learned_addr learned my addr 172.21.15.47:0/3730089969 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:32:06.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.288+0000 7fad8fe00640 1 --2- 172.21.15.47:0/3730089969 >> v2:172.21.15.116:3300/0 conn(0x7fad9006cfc0 0x7fad900b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:06.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.288+0000 7fad8f400640 1 -- 172.21.15.47:0/3730089969 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176641317 0 0) 0x7fad9006c660 con 0x7fad9006c1d0 2024-02-19T15:32:06.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.288+0000 7fad8f400640 1 -- 172.21.15.47:0/3730089969 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad78003530 con 0x7fad9006c1d0 2024-02-19T15:32:06.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.288+0000 7fad8fe00640 1 -- 172.21.15.47:0/3730089969 >> v1:172.21.15.116:6789/0 conn(0x7fad9015e180 legacy=0x7fad9006c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:06.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.288+0000 7fad8fe00640 1 -- 172.21.15.47:0/3730089969 >> v1:172.21.15.47:6789/0 conn(0x7fad9006c1d0 legacy=0x7fad900ae870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:06.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.288+0000 7fad8fe00640 1 -- 172.21.15.47:0/3730089969 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad900b2830 con 0x7fad9006cfc0 2024-02-19T15:32:06.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.292+0000 7fad8fe00640 1 --2- 172.21.15.47:0/3730089969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad9006cfc0 0x7fad900b1c20 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fad7c0097c0 tx=0x7fad7c01c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf60a5cf66cc81b0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:06.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.292+0000 7fad8f400640 1 -- 172.21.15.47:0/3730089969 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c0ab670 con 0x7fad9006cfc0 2024-02-19T15:32:06.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.292+0000 7fad8f400640 1 -- 172.21.15.47:0/3730089969 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7c0ab7d0 con 0x7fad9006cfc0 2024-02-19T15:32:06.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.292+0000 7fad8f400640 1 -- 172.21.15.47:0/3730089969 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c0abb60 con 0x7fad9006cfc0 2024-02-19T15:32:06.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.292+0000 7fad97400640 1 -- 172.21.15.47:0/3730089969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad9006cfc0 msgr2=0x7fad900b1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:06.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.292+0000 7fad97400640 1 --2- 172.21.15.47:0/3730089969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad9006cfc0 0x7fad900b1c20 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fad7c0097c0 tx=0x7fad7c01c470 comp rx=0 tx=0).stop 2024-02-19T15:32:06.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.292+0000 7fad97400640 1 -- 172.21.15.47:0/3730089969 shutdown_connections 2024-02-19T15:32:06.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.292+0000 7fad97400640 1 --2- 172.21.15.47:0/3730089969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad9006cfc0 0x7fad900b1c20 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:06.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.292+0000 7fad97400640 1 -- 172.21.15.47:0/3730089969 >> 172.21.15.47:0/3730089969 conn(0x7fad900d1ba0 msgr2=0x7fad900d1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:06.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.292+0000 7fad97400640 1 -- 172.21.15.47:0/3730089969 shutdown_connections 2024-02-19T15:32:06.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.292+0000 7fad97400640 1 -- 172.21.15.47:0/3730089969 wait complete. 2024-02-19T15:32:06.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.292+0000 7fad97400640 1 Processor -- start 2024-02-19T15:32:06.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad97400640 1 -- start start 2024-02-19T15:32:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad97400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad9006c1d0 0x7fad9014d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad97400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad9006cfc0 0x7fad9014df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad97400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad9015e180 0x7fad900b64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad97400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad900c6ea0 con 0x7fad9006c1d0 2024-02-19T15:32:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad97400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad900c7180 con 0x7fad9015e180 2024-02-19T15:32:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad97400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad900c7460 con 0x7fad9006cfc0 2024-02-19T15:32:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad94a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad9006c1d0 0x7fad9014d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad94a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad9006c1d0 0x7fad9014d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:41662/0 (socket says 172.21.15.47:41662) 2024-02-19T15:32:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad94a00640 1 -- 172.21.15.47:0/3909032736 learned_addr learned my addr 172.21.15.47:0/3909032736 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad8fe00640 1 --2- 172.21.15.47:0/3909032736 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad9006cfc0 0x7fad9014df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:06.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad95400640 1 --2- 172.21.15.47:0/3909032736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad9015e180 0x7fad900b64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:06.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad94a00640 1 -- 172.21.15.47:0/3909032736 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad9006cfc0 msgr2=0x7fad9014df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:06.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad94a00640 1 --2- 172.21.15.47:0/3909032736 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad9006cfc0 0x7fad9014df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:06.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad94a00640 1 -- 172.21.15.47:0/3909032736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad9015e180 msgr2=0x7fad900b64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:06.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad94a00640 1 --2- 172.21.15.47:0/3909032736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad9015e180 0x7fad900b64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:06.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad94a00640 1 -- 172.21.15.47:0/3909032736 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7c009420 con 0x7fad9006c1d0 2024-02-19T15:32:06.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad94a00640 1 --2- 172.21.15.47:0/3909032736 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad9006c1d0 0x7fad9014d6c0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fad8400c500 tx=0x7fad8400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:06.297 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:32:06.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad8cc00640 1 -- 172.21.15.47:0/3909032736 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad84005220 con 0x7fad9006c1d0 2024-02-19T15:32:06.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad8cc00640 1 -- 172.21.15.47:0/3909032736 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad84005380 con 0x7fad9006c1d0 2024-02-19T15:32:06.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad8cc00640 1 -- 172.21.15.47:0/3909032736 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad84007270 con 0x7fad9006c1d0 2024-02-19T15:32:06.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad97400640 1 -- 172.21.15.47:0/3909032736 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad9015e930 con 0x7fad9006c1d0 2024-02-19T15:32:06.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad97400640 1 -- 172.21.15.47:0/3909032736 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad9014dc00 con 0x7fad9006c1d0 2024-02-19T15:32:06.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.296+0000 7fad97400640 1 -- 172.21.15.47:0/3909032736 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad90068500 con 0x7fad9006c1d0 2024-02-19T15:32:06.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.300+0000 7fad8cc00640 1 -- 172.21.15.47:0/3909032736 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fad840073d0 con 0x7fad9006c1d0 2024-02-19T15:32:06.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.300+0000 7fad8cc00640 1 --2- 172.21.15.47:0/3909032736 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fad70066e40 0x7fad70069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:06.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.300+0000 7fad8cc00640 1 -- 172.21.15.47:0/3909032736 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fad84084280 con 0x7fad9006c1d0 2024-02-19T15:32:06.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.300+0000 7fad8fe00640 1 --2- 172.21.15.47:0/3909032736 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fad70066e40 0x7fad70069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:06.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.300+0000 7fad8fe00640 1 --2- 172.21.15.47:0/3909032736 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fad70066e40 0x7fad70069300 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fad7c0af000 tx=0x7fad7c0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:06.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.304+0000 7fad8cc00640 1 -- 172.21.15.47:0/3909032736 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fad84051050 con 0x7fad9006c1d0 2024-02-19T15:32:06.347 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:32:06.351 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:32:06.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.524+0000 7fad97400640 1 -- 172.21.15.47:0/3909032736 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad90075360 con 0x7fad9006c1d0 2024-02-19T15:32:06.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.524+0000 7fad8cc00640 1 -- 172.21.15.47:0/3909032736 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7fad84054710 con 0x7fad9006c1d0 2024-02-19T15:32:06.525 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:06.525 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:06.525 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:06.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.528+0000 7fad97400640 1 -- 172.21.15.47:0/3909032736 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fad70066e40 msgr2=0x7fad70069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:06.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.528+0000 7fad97400640 1 --2- 172.21.15.47:0/3909032736 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fad70066e40 0x7fad70069300 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fad7c0af000 tx=0x7fad7c0a7000 comp rx=0 tx=0).stop 2024-02-19T15:32:06.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.528+0000 7fad97400640 1 -- 172.21.15.47:0/3909032736 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad9006c1d0 msgr2=0x7fad9014d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:06.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.528+0000 7fad97400640 1 --2- 172.21.15.47:0/3909032736 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad9006c1d0 0x7fad9014d6c0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fad8400c500 tx=0x7fad8400c9c0 comp rx=0 tx=0).stop 2024-02-19T15:32:06.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.528+0000 7fad97400640 1 -- 172.21.15.47:0/3909032736 shutdown_connections 2024-02-19T15:32:06.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.528+0000 7fad97400640 1 --2- 172.21.15.47:0/3909032736 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fad70066e40 0x7fad70069300 secure :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fad7c0af000 tx=0x7fad7c0a7000 comp rx=0 tx=0).stop 2024-02-19T15:32:06.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.528+0000 7fad97400640 1 --2- 172.21.15.47:0/3909032736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad9015e180 0x7fad900b64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:06.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.528+0000 7fad97400640 1 --2- 172.21.15.47:0/3909032736 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad9006cfc0 0x7fad9014df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:06.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.528+0000 7fad97400640 1 --2- 172.21.15.47:0/3909032736 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad9006c1d0 0x7fad9014d6c0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:06.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.528+0000 7fad97400640 1 -- 172.21.15.47:0/3909032736 >> 172.21.15.47:0/3909032736 conn(0x7fad900d1ba0 msgr2=0x7fad9006f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:06.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.528+0000 7fad97400640 1 -- 172.21.15.47:0/3909032736 shutdown_connections 2024-02-19T15:32:06.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:06.528+0000 7fad97400640 1 -- 172.21.15.47:0/3909032736 wait complete. 2024-02-19T15:32:06.748 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:06.748 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:07.749 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:07.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.888+0000 7fcc3fe00640 1 Processor -- start 2024-02-19T15:32:07.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.888+0000 7fcc3fe00640 1 -- start start 2024-02-19T15:32:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.892+0000 7fcc3fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc380cded0 0x7fcc380ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.892+0000 7fcc3fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcc380ceb40 0x7fcc3806bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.892+0000 7fcc3fe00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc380cce30 con 0x7fcc3806d650 2024-02-19T15:32:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.892+0000 7fcc3fe00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc380cefd0 con 0x7fcc380ceb40 2024-02-19T15:32:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.892+0000 7fcc3fe00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc380cd110 con 0x7fcc380cded0 2024-02-19T15:32:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.892+0000 7fcc3de00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcc380ceb40 0x7fcc3806bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.892+0000 7fcc3de00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcc380ceb40 0x7fcc3806bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:41676/0 (socket says 172.21.15.47:41676) 2024-02-19T15:32:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.892+0000 7fcc3de00640 1 -- 172.21.15.47:0/1448195620 learned_addr learned my addr 172.21.15.47:0/1448195620 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.892+0000 7fcc3ca00640 1 --2- 172.21.15.47:0/1448195620 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc380cded0 0x7fcc380ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.892+0000 7fcc3de00640 1 -- 172.21.15.47:0/1448195620 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc380cded0 msgr2=0x7fcc380ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.892+0000 7fcc3de00640 1 --2- 172.21.15.47:0/1448195620 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc380cded0 0x7fcc380ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.892+0000 7fcc3de00640 1 -- 172.21.15.47:0/1448195620 >> v1:172.21.15.116:6789/0 conn(0x7fcc3806d650 legacy=0x7fcc380d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.892+0000 7fcc3de00640 1 -- 172.21.15.47:0/1448195620 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc380cd3f0 con 0x7fcc380ceb40 2024-02-19T15:32:07.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3de00640 1 --2- 172.21.15.47:0/1448195620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc380ceb40 0x7fcc3806bd60 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fcc30009e50 tx=0x7fcc3001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93024c00ab909bfb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:07.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc37e00640 1 -- 172.21.15.47:0/1448195620 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc300a5830 con 0x7fcc380ceb40 2024-02-19T15:32:07.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc37e00640 1 -- 172.21.15.47:0/1448195620 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc300a5990 con 0x7fcc380ceb40 2024-02-19T15:32:07.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc37e00640 1 -- 172.21.15.47:0/1448195620 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc300a5d20 con 0x7fcc380ceb40 2024-02-19T15:32:07.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3fe00640 1 -- 172.21.15.47:0/1448195620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc380ceb40 msgr2=0x7fcc3806bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:07.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3fe00640 1 --2- 172.21.15.47:0/1448195620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc380ceb40 0x7fcc3806bd60 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fcc30009e50 tx=0x7fcc3001c970 comp rx=0 tx=0).stop 2024-02-19T15:32:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3fe00640 1 -- 172.21.15.47:0/1448195620 shutdown_connections 2024-02-19T15:32:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3fe00640 1 --2- 172.21.15.47:0/1448195620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc380ceb40 0x7fcc3806bd60 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3fe00640 1 --2- 172.21.15.47:0/1448195620 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc380cded0 0x7fcc380ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3fe00640 1 -- 172.21.15.47:0/1448195620 >> 172.21.15.47:0/1448195620 conn(0x7fcc380d1bf0 msgr2=0x7fcc380d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:07.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3fe00640 1 -- 172.21.15.47:0/1448195620 shutdown_connections 2024-02-19T15:32:07.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3fe00640 1 -- 172.21.15.47:0/1448195620 wait complete. 2024-02-19T15:32:07.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3fe00640 1 Processor -- start 2024-02-19T15:32:07.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3fe00640 1 -- start start 2024-02-19T15:32:07.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc3806d650 0x7fcc3815a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:07.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3fe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc380cded0 0x7fcc381594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:07.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3fe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc380ceb40 0x7fcc381601f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:07.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3fe00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3815b600 con 0x7fcc380cded0 2024-02-19T15:32:07.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3fe00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3815b070 con 0x7fcc3806d650 2024-02-19T15:32:07.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3fe00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3815ad90 con 0x7fcc380ceb40 2024-02-19T15:32:07.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3ca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc380cded0 0x7fcc381594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:07.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3ca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc380cded0 0x7fcc381594a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:41682/0 (socket says 172.21.15.47:41682) 2024-02-19T15:32:07.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3ca00640 1 -- 172.21.15.47:0/3163375263 learned_addr learned my addr 172.21.15.47:0/3163375263 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:07.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.896+0000 7fcc3d400640 1 --2- 172.21.15.47:0/3163375263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc3806d650 0x7fcc3815a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:07.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.900+0000 7fcc3de00640 1 --2- 172.21.15.47:0/3163375263 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc380ceb40 0x7fcc381601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:07.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.900+0000 7fcc3ca00640 1 -- 172.21.15.47:0/3163375263 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc380ceb40 msgr2=0x7fcc381601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:07.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.900+0000 7fcc3ca00640 1 --2- 172.21.15.47:0/3163375263 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc380ceb40 0x7fcc381601f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:07.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.900+0000 7fcc3ca00640 1 -- 172.21.15.47:0/3163375263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc3806d650 msgr2=0x7fcc3815a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:07.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.900+0000 7fcc3ca00640 1 --2- 172.21.15.47:0/3163375263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc3806d650 0x7fcc3815a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:07.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.900+0000 7fcc3ca00640 1 -- 172.21.15.47:0/3163375263 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc30009b30 con 0x7fcc380cded0 2024-02-19T15:32:07.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.900+0000 7fcc3ca00640 1 --2- 172.21.15.47:0/3163375263 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc380cded0 0x7fcc381594a0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400e870 tx=0x7fcc2400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:07.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.900+0000 7fcc35600640 1 -- 172.21.15.47:0/3163375263 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc24009510 con 0x7fcc380cded0 2024-02-19T15:32:07.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.900+0000 7fcc35600640 1 -- 172.21.15.47:0/3163375263 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2400eec0 con 0x7fcc380cded0 2024-02-19T15:32:07.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.900+0000 7fcc3fe00640 1 -- 172.21.15.47:0/3163375263 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3815f710 con 0x7fcc380cded0 2024-02-19T15:32:07.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.900+0000 7fcc35600640 1 -- 172.21.15.47:0/3163375263 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2400cb40 con 0x7fcc380cded0 2024-02-19T15:32:07.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.900+0000 7fcc3fe00640 1 -- 172.21.15.47:0/3163375263 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3815fcd0 con 0x7fcc380cded0 2024-02-19T15:32:07.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.900+0000 7fcc3fe00640 1 -- 172.21.15.47:0/3163375263 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc38075360 con 0x7fcc380cded0 2024-02-19T15:32:07.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.900+0000 7fcc35600640 1 -- 172.21.15.47:0/3163375263 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fcc2400cca0 con 0x7fcc380cded0 2024-02-19T15:32:07.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.900+0000 7fcc35600640 1 --2- 172.21.15.47:0/3163375263 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcc18066e70 0x7fcc18069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:07.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.900+0000 7fcc35600640 1 -- 172.21.15.47:0/3163375263 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcc24084ab0 con 0x7fcc380cded0 2024-02-19T15:32:07.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.904+0000 7fcc3d400640 1 --2- 172.21.15.47:0/3163375263 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcc18066e70 0x7fcc18069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:07.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.904+0000 7fcc3d400640 1 --2- 172.21.15.47:0/3163375263 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcc18066e70 0x7fcc18069330 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fcc380cbdc0 tx=0x7fcc2c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:07.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:07.904+0000 7fcc35600640 1 -- 172.21.15.47:0/3163375263 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fcc24051900 con 0x7fcc380cded0 2024-02-19T15:32:08.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:08.104+0000 7fcc3fe00640 1 -- 172.21.15.47:0/3163375263 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc381616f0 con 0x7fcc380cded0 2024-02-19T15:32:08.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:08.104+0000 7fcc35600640 1 -- 172.21.15.47:0/3163375263 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7fcc24021d80 con 0x7fcc380cded0 2024-02-19T15:32:08.106 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:08.106 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:08.106 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:08.108+0000 7fcc3fe00640 1 -- 172.21.15.47:0/3163375263 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcc18066e70 msgr2=0x7fcc18069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:08.108+0000 7fcc3fe00640 1 --2- 172.21.15.47:0/3163375263 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcc18066e70 0x7fcc18069330 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fcc380cbdc0 tx=0x7fcc2c009040 comp rx=0 tx=0).stop 2024-02-19T15:32:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:08.108+0000 7fcc3fe00640 1 -- 172.21.15.47:0/3163375263 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc380cded0 msgr2=0x7fcc381594a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:08.108+0000 7fcc3fe00640 1 --2- 172.21.15.47:0/3163375263 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc380cded0 0x7fcc381594a0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400e870 tx=0x7fcc2400ec70 comp rx=0 tx=0).stop 2024-02-19T15:32:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:08.108+0000 7fcc3fe00640 1 -- 172.21.15.47:0/3163375263 shutdown_connections 2024-02-19T15:32:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:08.108+0000 7fcc3fe00640 1 --2- 172.21.15.47:0/3163375263 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fcc18066e70 0x7fcc18069330 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:08.108+0000 7fcc3fe00640 1 --2- 172.21.15.47:0/3163375263 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc380ceb40 0x7fcc381601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:08.108+0000 7fcc3fe00640 1 --2- 172.21.15.47:0/3163375263 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc380cded0 0x7fcc381594a0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:08.108+0000 7fcc3fe00640 1 --2- 172.21.15.47:0/3163375263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc3806d650 0x7fcc3815a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:08.108+0000 7fcc3fe00640 1 -- 172.21.15.47:0/3163375263 >> 172.21.15.47:0/3163375263 conn(0x7fcc380d1bf0 msgr2=0x7fcc380cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:08.108+0000 7fcc3fe00640 1 -- 172.21.15.47:0/3163375263 shutdown_connections 2024-02-19T15:32:08.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:08.108+0000 7fcc3fe00640 1 -- 172.21.15.47:0/3163375263 wait complete. 2024-02-19T15:32:08.298 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:32:08.338 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:08.338 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:09.339 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:09.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.480+0000 7efc10200640 1 Processor -- start 2024-02-19T15:32:09.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.480+0000 7efc10200640 1 -- start start 2024-02-19T15:32:09.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.484+0000 7efc10200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc081613a0 0x7efc080adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:09.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.484+0000 7efc10200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efc0806e7c0 0x7efc0806aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:09.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.484+0000 7efc10200640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc08161780 con 0x7efc0806fdf0 2024-02-19T15:32:09.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.484+0000 7efc10200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc080b01a0 con 0x7efc0806e7c0 2024-02-19T15:32:09.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.484+0000 7efc10200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc080ae3c0 con 0x7efc081613a0 2024-02-19T15:32:09.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.484+0000 7efc0ce00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7efc0806fdf0 0x7efc0806e0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37926/0 (socket says 172.21.15.47:37926) 2024-02-19T15:32:09.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.484+0000 7efc0ce00640 1 -- 172.21.15.47:0/708045830 learned_addr learned my addr 172.21.15.47:0/708045830 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:32:09.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.484+0000 7efc0e200640 1 --2- 172.21.15.47:0/708045830 >> v2:172.21.15.116:3300/0 conn(0x7efc0806e7c0 0x7efc0806aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:09.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.484+0000 7efc0d800640 1 --2- 172.21.15.47:0/708045830 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc081613a0 0x7efc080adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:09.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.484+0000 7efc07e00640 1 -- 172.21.15.47:0/708045830 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483275127 0 0) 0x7efc08161780 con 0x7efc0806fdf0 2024-02-19T15:32:09.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.484+0000 7efc07e00640 1 -- 172.21.15.47:0/708045830 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbf0003530 con 0x7efc0806fdf0 2024-02-19T15:32:09.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.484+0000 7efc0d800640 1 -- 172.21.15.47:0/708045830 >> v2:172.21.15.116:3300/0 conn(0x7efc0806e7c0 msgr2=0x7efc0806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:09.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.484+0000 7efc0d800640 1 --2- 172.21.15.47:0/708045830 >> v2:172.21.15.116:3300/0 conn(0x7efc0806e7c0 0x7efc0806aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:09.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.484+0000 7efc0d800640 1 -- 172.21.15.47:0/708045830 >> v1:172.21.15.47:6789/0 conn(0x7efc0806fdf0 legacy=0x7efc0806e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:09.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.484+0000 7efc0d800640 1 -- 172.21.15.47:0/708045830 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc080ae6a0 con 0x7efc081613a0 2024-02-19T15:32:09.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.484+0000 7efc0e200640 1 --2- 172.21.15.47:0/708045830 >> v2:172.21.15.116:3300/0 conn(0x7efc0806e7c0 0x7efc0806aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:32:09.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.488+0000 7efc0d800640 1 --2- 172.21.15.47:0/708045830 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc081613a0 0x7efc080adbb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0097c0 tx=0x7efbfc01c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3d1b5e6cab79602 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:09.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.488+0000 7efc07e00640 1 -- 172.21.15.47:0/708045830 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc0ab600 con 0x7efc081613a0 2024-02-19T15:32:09.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.488+0000 7efc07e00640 1 -- 172.21.15.47:0/708045830 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbfc0a7050 con 0x7efc081613a0 2024-02-19T15:32:09.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.488+0000 7efc07e00640 1 -- 172.21.15.47:0/708045830 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc0ab980 con 0x7efc081613a0 2024-02-19T15:32:09.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.488+0000 7efc10200640 1 -- 172.21.15.47:0/708045830 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc081613a0 msgr2=0x7efc080adbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:09.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.488+0000 7efc10200640 1 --2- 172.21.15.47:0/708045830 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc081613a0 0x7efc080adbb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0097c0 tx=0x7efbfc01c410 comp rx=0 tx=0).stop 2024-02-19T15:32:09.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.488+0000 7efc10200640 1 -- 172.21.15.47:0/708045830 shutdown_connections 2024-02-19T15:32:09.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.488+0000 7efc10200640 1 --2- 172.21.15.47:0/708045830 >> v2:172.21.15.116:3300/0 conn(0x7efc0806e7c0 0x7efc0806aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:09.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.488+0000 7efc10200640 1 --2- 172.21.15.47:0/708045830 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc081613a0 0x7efc080adbb0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:09.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.488+0000 7efc10200640 1 -- 172.21.15.47:0/708045830 >> 172.21.15.47:0/708045830 conn(0x7efc080d0ad0 msgr2=0x7efc080d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:09.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.488+0000 7efc10200640 1 -- 172.21.15.47:0/708045830 shutdown_connections 2024-02-19T15:32:09.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.488+0000 7efc10200640 1 -- 172.21.15.47:0/708045830 wait complete. 2024-02-19T15:32:09.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.488+0000 7efc10200640 1 Processor -- start 2024-02-19T15:32:09.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc10200640 1 -- start start 2024-02-19T15:32:09.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc10200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc0806e7c0 0x7efc08152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:09.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc10200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc0806fdf0 0x7efc08153ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:09.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc10200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc081613a0 0x7efc081567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:09.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc10200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc08153690 con 0x7efc081613a0 2024-02-19T15:32:09.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc10200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc08152150 con 0x7efc0806fdf0 2024-02-19T15:32:09.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc10200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc08152a10 con 0x7efc0806e7c0 2024-02-19T15:32:09.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc0e200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc081613a0 0x7efc081567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:09.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc0e200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc081613a0 0x7efc081567f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:41698/0 (socket says 172.21.15.47:41698) 2024-02-19T15:32:09.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc0e200640 1 -- 172.21.15.47:0/1754765817 learned_addr learned my addr 172.21.15.47:0/1754765817 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:09.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc0ce00640 1 --2- 172.21.15.47:0/1754765817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc0806fdf0 0x7efc08153ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:09.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc0d800640 1 --2- 172.21.15.47:0/1754765817 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc0806e7c0 0x7efc08152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:09.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc0e200640 1 -- 172.21.15.47:0/1754765817 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc0806e7c0 msgr2=0x7efc08152310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:09.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc0e200640 1 --2- 172.21.15.47:0/1754765817 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc0806e7c0 0x7efc08152310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:09.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc0e200640 1 -- 172.21.15.47:0/1754765817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc0806fdf0 msgr2=0x7efc08153ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:09.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc0e200640 1 --2- 172.21.15.47:0/1754765817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc0806fdf0 0x7efc08153ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:09.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc0e200640 1 -- 172.21.15.47:0/1754765817 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc00009a70 con 0x7efc081613a0 2024-02-19T15:32:09.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc0e200640 1 --2- 172.21.15.47:0/1754765817 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc081613a0 0x7efc081567f0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7efc0000efc0 tx=0x7efc00009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:09.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc05600640 1 -- 172.21.15.47:0/1754765817 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc00009d70 con 0x7efc081613a0 2024-02-19T15:32:09.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc05600640 1 -- 172.21.15.47:0/1754765817 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0000f040 con 0x7efc081613a0 2024-02-19T15:32:09.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc05600640 1 -- 172.21.15.47:0/1754765817 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc000039a0 con 0x7efc081613a0 2024-02-19T15:32:09.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc10200640 1 -- 172.21.15.47:0/1754765817 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbfc009420 con 0x7efc081613a0 2024-02-19T15:32:09.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc10200640 1 -- 172.21.15.47:0/1754765817 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc08152fd0 con 0x7efc081613a0 2024-02-19T15:32:09.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.492+0000 7efc10200640 1 -- 172.21.15.47:0/1754765817 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc08075360 con 0x7efc081613a0 2024-02-19T15:32:09.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.496+0000 7efc05600640 1 -- 172.21.15.47:0/1754765817 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efc00014480 con 0x7efc081613a0 2024-02-19T15:32:09.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.496+0000 7efc05600640 1 --2- 172.21.15.47:0/1754765817 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efbe8066e40 0x7efbe8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:09.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.496+0000 7efc05600640 1 -- 172.21.15.47:0/1754765817 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efc000855b0 con 0x7efc081613a0 2024-02-19T15:32:09.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.496+0000 7efc0d800640 1 --2- 172.21.15.47:0/1754765817 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efbe8066e40 0x7efbe8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:09.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.496+0000 7efc0d800640 1 --2- 172.21.15.47:0/1754765817 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efbe8066e40 0x7efbe8069300 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0ad000 tx=0x7efbfc0a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:09.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.504+0000 7efc05600640 1 -- 172.21.15.47:0/1754765817 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7efc00052410 con 0x7efc081613a0 2024-02-19T15:32:09.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.704+0000 7efc10200640 1 -- 172.21.15.47:0/1754765817 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc080d15f0 con 0x7efc081613a0 2024-02-19T15:32:09.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.704+0000 7efc05600640 1 -- 172.21.15.47:0/1754765817 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7efc00018d80 con 0x7efc081613a0 2024-02-19T15:32:09.703 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:09.703 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:09.703 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:09.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.704+0000 7efc10200640 1 -- 172.21.15.47:0/1754765817 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efbe8066e40 msgr2=0x7efbe8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:09.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.704+0000 7efc10200640 1 --2- 172.21.15.47:0/1754765817 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efbe8066e40 0x7efbe8069300 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0ad000 tx=0x7efbfc0a6000 comp rx=0 tx=0).stop 2024-02-19T15:32:09.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.704+0000 7efc10200640 1 -- 172.21.15.47:0/1754765817 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc081613a0 msgr2=0x7efc081567f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:09.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.704+0000 7efc10200640 1 --2- 172.21.15.47:0/1754765817 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc081613a0 0x7efc081567f0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7efc0000efc0 tx=0x7efc00009f90 comp rx=0 tx=0).stop 2024-02-19T15:32:09.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.704+0000 7efc10200640 1 -- 172.21.15.47:0/1754765817 shutdown_connections 2024-02-19T15:32:09.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.704+0000 7efc10200640 1 --2- 172.21.15.47:0/1754765817 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efbe8066e40 0x7efbe8069300 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:09.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.704+0000 7efc10200640 1 --2- 172.21.15.47:0/1754765817 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc081613a0 0x7efc081567f0 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:09.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.704+0000 7efc10200640 1 --2- 172.21.15.47:0/1754765817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc0806fdf0 0x7efc08153ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:09.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.704+0000 7efc10200640 1 --2- 172.21.15.47:0/1754765817 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc0806e7c0 0x7efc08152310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:09.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.704+0000 7efc10200640 1 -- 172.21.15.47:0/1754765817 >> 172.21.15.47:0/1754765817 conn(0x7efc080d0ad0 msgr2=0x7efc080b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:09.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.704+0000 7efc10200640 1 -- 172.21.15.47:0/1754765817 shutdown_connections 2024-02-19T15:32:09.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:09.704+0000 7efc10200640 1 -- 172.21.15.47:0/1754765817 wait complete. 2024-02-19T15:32:09.927 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:09.927 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:10.300 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:32:10.928 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:11.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.068+0000 7fe071e00640 1 Processor -- start 2024-02-19T15:32:11.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.068+0000 7fe071e00640 1 -- start start 2024-02-19T15:32:11.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.072+0000 7fe071e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe06c0d19e0 0x7fe06c0d1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:11.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.072+0000 7fe071e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe06c0ccbd0 0x7fe06c15be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:11.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.072+0000 7fe071e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe06c0d2300 con 0x7fe06c0cec90 2024-02-19T15:32:11.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.072+0000 7fe071e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe06c0ce280 con 0x7fe06c0ccbd0 2024-02-19T15:32:11.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.072+0000 7fe071e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe06c15cd40 con 0x7fe06c0d19e0 2024-02-19T15:32:11.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.072+0000 7fe071400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe06c0ccbd0 0x7fe06c15be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:11.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.072+0000 7fe071400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe06c0ccbd0 0x7fe06c15be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:41704/0 (socket says 172.21.15.47:41704) 2024-02-19T15:32:11.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.072+0000 7fe071400640 1 -- 172.21.15.47:0/2865327685 learned_addr learned my addr 172.21.15.47:0/2865327685 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:11.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.072+0000 7fe063e00640 1 --1- 172.21.15.47:0/2865327685 >> v1:172.21.15.47:6789/0 conn(0x7fe06c0cec90 0x7fe06c0cc4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37932/0 (socket says 172.21.15.47:37932) 2024-02-19T15:32:11.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.072+0000 7fe070a00640 1 --2- 172.21.15.47:0/2865327685 >> v2:172.21.15.116:3300/0 conn(0x7fe06c0d19e0 0x7fe06c0d1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:11.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.072+0000 7fe071400640 1 -- 172.21.15.47:0/2865327685 >> v2:172.21.15.116:3300/0 conn(0x7fe06c0d19e0 msgr2=0x7fe06c0d1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:11.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.072+0000 7fe071400640 1 --2- 172.21.15.47:0/2865327685 >> v2:172.21.15.116:3300/0 conn(0x7fe06c0d19e0 0x7fe06c0d1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:11.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.072+0000 7fe071400640 1 -- 172.21.15.47:0/2865327685 >> v1:172.21.15.47:6789/0 conn(0x7fe06c0cec90 legacy=0x7fe06c0cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:11.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.072+0000 7fe071400640 1 -- 172.21.15.47:0/2865327685 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06c15d020 con 0x7fe06c0ccbd0 2024-02-19T15:32:11.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.076+0000 7fe071400640 1 --2- 172.21.15.47:0/2865327685 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe06c0ccbd0 0x7fe06c15be50 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fe06c0d19b0 tx=0x7fe06401c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c77c134a284191 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:11.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.076+0000 7fe06be00640 1 -- 172.21.15.47:0/2865327685 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0640a5780 con 0x7fe06c0ccbd0 2024-02-19T15:32:11.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.076+0000 7fe06be00640 1 -- 172.21.15.47:0/2865327685 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0640a58e0 con 0x7fe06c0ccbd0 2024-02-19T15:32:11.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.076+0000 7fe06be00640 1 -- 172.21.15.47:0/2865327685 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0640a5c70 con 0x7fe06c0ccbd0 2024-02-19T15:32:11.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.076+0000 7fe071e00640 1 -- 172.21.15.47:0/2865327685 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe06c0ccbd0 msgr2=0x7fe06c15be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:11.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.076+0000 7fe071e00640 1 --2- 172.21.15.47:0/2865327685 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe06c0ccbd0 0x7fe06c15be50 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fe06c0d19b0 tx=0x7fe06401c9c0 comp rx=0 tx=0).stop 2024-02-19T15:32:11.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.076+0000 7fe071e00640 1 -- 172.21.15.47:0/2865327685 shutdown_connections 2024-02-19T15:32:11.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.076+0000 7fe071e00640 1 --2- 172.21.15.47:0/2865327685 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe06c0ccbd0 0x7fe06c15be50 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:11.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.076+0000 7fe071e00640 1 --2- 172.21.15.47:0/2865327685 >> v2:172.21.15.116:3300/0 conn(0x7fe06c0d19e0 0x7fe06c0d1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:11.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.076+0000 7fe071e00640 1 -- 172.21.15.47:0/2865327685 >> 172.21.15.47:0/2865327685 conn(0x7fe06c09a9d0 msgr2=0x7fe06c09ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:11.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.076+0000 7fe071e00640 1 -- 172.21.15.47:0/2865327685 shutdown_connections 2024-02-19T15:32:11.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.076+0000 7fe071e00640 1 -- 172.21.15.47:0/2865327685 wait complete. 2024-02-19T15:32:11.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.076+0000 7fe071e00640 1 Processor -- start 2024-02-19T15:32:11.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.076+0000 7fe071e00640 1 -- start start 2024-02-19T15:32:11.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe071e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe06c0ccbd0 0x7fe06c152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:11.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe071e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe06c0cec90 0x7fe06c0b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:11.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe071e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe06c0d19e0 0x7fe06c1545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:11.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe071e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c0b4980 con 0x7fe06c0ccbd0 2024-02-19T15:32:11.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe071e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c0b4c60 con 0x7fe06c0cec90 2024-02-19T15:32:11.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe071e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c0b4f40 con 0x7fe06c0d19e0 2024-02-19T15:32:11.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe070a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe06c0ccbd0 0x7fe06c152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:11.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe070a00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe06c0ccbd0 0x7fe06c152c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:41706/0 (socket says 172.21.15.47:41706) 2024-02-19T15:32:11.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe070a00640 1 -- 172.21.15.47:0/2758859030 learned_addr learned my addr 172.21.15.47:0/2758859030 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:11.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe063e00640 1 --2- 172.21.15.47:0/2758859030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe06c0cec90 0x7fe06c0b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:11.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe071400640 1 --2- 172.21.15.47:0/2758859030 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe06c0d19e0 0x7fe06c1545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:11.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe070a00640 1 -- 172.21.15.47:0/2758859030 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe06c0d19e0 msgr2=0x7fe06c1545d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:11.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe070a00640 1 --2- 172.21.15.47:0/2758859030 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe06c0d19e0 0x7fe06c1545d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:11.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe070a00640 1 -- 172.21.15.47:0/2758859030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe06c0cec90 msgr2=0x7fe06c0b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:11.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe070a00640 1 --2- 172.21.15.47:0/2758859030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe06c0cec90 0x7fe06c0b5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:11.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe070a00640 1 -- 172.21.15.47:0/2758859030 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe064009b30 con 0x7fe06c0ccbd0 2024-02-19T15:32:11.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe070a00640 1 --2- 172.21.15.47:0/2758859030 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe06c0ccbd0 0x7fe06c152c50 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00d800 tx=0x7fe05c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:11.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe069600640 1 -- 172.21.15.47:0/2758859030 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c00b6c0 con 0x7fe06c0ccbd0 2024-02-19T15:32:11.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe069600640 1 -- 172.21.15.47:0/2758859030 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05c00b820 con 0x7fe06c0ccbd0 2024-02-19T15:32:11.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe069600640 1 -- 172.21.15.47:0/2758859030 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c00bb90 con 0x7fe06c0ccbd0 2024-02-19T15:32:11.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe071e00640 1 -- 172.21.15.47:0/2758859030 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe06c0b5220 con 0x7fe06c0ccbd0 2024-02-19T15:32:11.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe071e00640 1 -- 172.21.15.47:0/2758859030 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe06c0b4610 con 0x7fe06c0ccbd0 2024-02-19T15:32:11.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe071e00640 1 -- 172.21.15.47:0/2758859030 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe06c075360 con 0x7fe06c0ccbd0 2024-02-19T15:32:11.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.080+0000 7fe069600640 1 -- 172.21.15.47:0/2758859030 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe05c00bcf0 con 0x7fe06c0ccbd0 2024-02-19T15:32:11.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.084+0000 7fe069600640 1 --2- 172.21.15.47:0/2758859030 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe048066e40 0x7fe048069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:11.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.084+0000 7fe069600640 1 -- 172.21.15.47:0/2758859030 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe05c084fe0 con 0x7fe06c0ccbd0 2024-02-19T15:32:11.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.084+0000 7fe063e00640 1 --2- 172.21.15.47:0/2758859030 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe048066e40 0x7fe048069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:11.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.084+0000 7fe063e00640 1 --2- 172.21.15.47:0/2758859030 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe048066e40 0x7fe048069300 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fe06c1f8400 tx=0x7fe054007cf0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:11.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.092+0000 7fe069600640 1 -- 172.21.15.47:0/2758859030 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe05c051da0 con 0x7fe06c0ccbd0 2024-02-19T15:32:11.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.288+0000 7fe071e00640 1 -- 172.21.15.47:0/2758859030 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe06c0d0890 con 0x7fe06c0ccbd0 2024-02-19T15:32:11.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.288+0000 7fe069600640 1 -- 172.21.15.47:0/2758859030 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7fe05c055460 con 0x7fe06c0ccbd0 2024-02-19T15:32:11.288 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:11.288 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:11.289 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:11.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.292+0000 7fe071e00640 1 -- 172.21.15.47:0/2758859030 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe048066e40 msgr2=0x7fe048069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:11.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.292+0000 7fe071e00640 1 --2- 172.21.15.47:0/2758859030 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe048066e40 0x7fe048069300 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fe06c1f8400 tx=0x7fe054007cf0 comp rx=0 tx=0).stop 2024-02-19T15:32:11.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.292+0000 7fe071e00640 1 -- 172.21.15.47:0/2758859030 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe06c0ccbd0 msgr2=0x7fe06c152c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:11.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.292+0000 7fe071e00640 1 --2- 172.21.15.47:0/2758859030 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe06c0ccbd0 0x7fe06c152c50 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00d800 tx=0x7fe05c00dcc0 comp rx=0 tx=0).stop 2024-02-19T15:32:11.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.292+0000 7fe071e00640 1 -- 172.21.15.47:0/2758859030 shutdown_connections 2024-02-19T15:32:11.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.292+0000 7fe071e00640 1 --2- 172.21.15.47:0/2758859030 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe048066e40 0x7fe048069300 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:11.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.292+0000 7fe071e00640 1 --2- 172.21.15.47:0/2758859030 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe06c0d19e0 0x7fe06c1545d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:11.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.292+0000 7fe071e00640 1 --2- 172.21.15.47:0/2758859030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe06c0cec90 0x7fe06c0b5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:11.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.292+0000 7fe071e00640 1 --2- 172.21.15.47:0/2758859030 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe06c0ccbd0 0x7fe06c152c50 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:11.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.292+0000 7fe071e00640 1 -- 172.21.15.47:0/2758859030 >> 172.21.15.47:0/2758859030 conn(0x7fe06c09a9d0 msgr2=0x7fe06c0c5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:11.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.292+0000 7fe071e00640 1 -- 172.21.15.47:0/2758859030 shutdown_connections 2024-02-19T15:32:11.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:11.292+0000 7fe071e00640 1 -- 172.21.15.47:0/2758859030 wait complete. 2024-02-19T15:32:11.511 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:11.511 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:12.302 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:32:12.512 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:12.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.652+0000 7f9e1b200640 1 Processor -- start 2024-02-19T15:32:12.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.652+0000 7f9e1b200640 1 -- start start 2024-02-19T15:32:12.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.656+0000 7f9e1b200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e140cded0 0x7f9e140ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:12.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.656+0000 7f9e1b200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9e140ceb40 0x7f9e1406bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:12.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.656+0000 7f9e1b200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e140cce30 con 0x7f9e1406d650 2024-02-19T15:32:12.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.656+0000 7f9e1b200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e140cefd0 con 0x7f9e140ceb40 2024-02-19T15:32:12.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.656+0000 7f9e1b200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e140cd110 con 0x7f9e140cded0 2024-02-19T15:32:12.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.656+0000 7f9e19200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9e140ceb40 0x7f9e1406bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:12.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.656+0000 7f9e0be00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e140cded0 0x7f9e140ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:12.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.656+0000 7f9e0be00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e140cded0 0x7f9e140ce600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:40494/0 (socket says 172.21.15.47:40494) 2024-02-19T15:32:12.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.656+0000 7f9e13e00640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9e1406d650 0x7f9e140d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.47:54304/0 (socket says 172.21.15.47:54304) 2024-02-19T15:32:12.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.656+0000 7f9e0be00640 1 -- 172.21.15.47:0/492692690 learned_addr learned my addr 172.21.15.47:0/492692690 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:12.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.656+0000 7f9e0be00640 1 -- 172.21.15.47:0/492692690 >> v2:172.21.15.116:3300/0 conn(0x7f9e140ceb40 msgr2=0x7f9e1406bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:12.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.656+0000 7f9e0be00640 1 --2- 172.21.15.47:0/492692690 >> v2:172.21.15.116:3300/0 conn(0x7f9e140ceb40 0x7f9e1406bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:12.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.656+0000 7f9e0be00640 1 -- 172.21.15.47:0/492692690 >> v1:172.21.15.116:6789/0 conn(0x7f9e1406d650 legacy=0x7f9e140d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:12.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.656+0000 7f9e0be00640 1 -- 172.21.15.47:0/492692690 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e140cd3f0 con 0x7f9e140cded0 2024-02-19T15:32:12.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.656+0000 7f9e19200640 1 --2- 172.21.15.47:0/492692690 >> v2:172.21.15.116:3300/0 conn(0x7f9e140ceb40 0x7f9e1406bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:32:12.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.660+0000 7f9e0be00640 1 --2- 172.21.15.47:0/492692690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e140cded0 0x7f9e140ce600 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc009870 tx=0x7f9dfc01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7de098f75134a768 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:12.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.660+0000 7f9e13400640 1 -- 172.21.15.47:0/492692690 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc0a6040 con 0x7f9e140cded0 2024-02-19T15:32:12.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.660+0000 7f9e13400640 1 -- 172.21.15.47:0/492692690 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dfc0aa600 con 0x7f9e140cded0 2024-02-19T15:32:12.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.660+0000 7f9e13400640 1 -- 172.21.15.47:0/492692690 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc0aa990 con 0x7f9e140cded0 2024-02-19T15:32:12.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.660+0000 7f9e1b200640 1 -- 172.21.15.47:0/492692690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e140cded0 msgr2=0x7f9e140ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:12.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.660+0000 7f9e1b200640 1 --2- 172.21.15.47:0/492692690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e140cded0 0x7f9e140ce600 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc009870 tx=0x7f9dfc01c4a0 comp rx=0 tx=0).stop 2024-02-19T15:32:12.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.660+0000 7f9e1b200640 1 -- 172.21.15.47:0/492692690 shutdown_connections 2024-02-19T15:32:12.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.660+0000 7f9e1b200640 1 --2- 172.21.15.47:0/492692690 >> v2:172.21.15.116:3300/0 conn(0x7f9e140ceb40 0x7f9e1406bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:12.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.660+0000 7f9e1b200640 1 --2- 172.21.15.47:0/492692690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e140cded0 0x7f9e140ce600 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:12.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.660+0000 7f9e1b200640 1 -- 172.21.15.47:0/492692690 >> 172.21.15.47:0/492692690 conn(0x7f9e140d1bf0 msgr2=0x7f9e140d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:12.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.660+0000 7f9e1b200640 1 -- 172.21.15.47:0/492692690 shutdown_connections 2024-02-19T15:32:12.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.660+0000 7f9e1b200640 1 -- 172.21.15.47:0/492692690 wait complete. 2024-02-19T15:32:12.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.660+0000 7f9e1b200640 1 Processor -- start 2024-02-19T15:32:12.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.660+0000 7f9e1b200640 1 -- start start 2024-02-19T15:32:12.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e1b200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9e1406d650 0x7f9e140cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:12.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e1b200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e140cded0 0x7f9e14157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:12.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e1b200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e140ceb40 0x7f9e1415a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:12.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e1b200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e14159660 con 0x7f9e1406d650 2024-02-19T15:32:12.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e1b200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e14157ea0 con 0x7f9e140ceb40 2024-02-19T15:32:12.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e1b200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e14158180 con 0x7f9e140cded0 2024-02-19T15:32:12.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e13e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9e1406d650 0x7f9e140cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:12.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e13e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9e1406d650 0x7f9e140cbd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:41712/0 (socket says 172.21.15.47:41712) 2024-02-19T15:32:12.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e13e00640 1 -- 172.21.15.47:0/32174261 learned_addr learned my addr 172.21.15.47:0/32174261 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:12.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e0be00640 1 --2- 172.21.15.47:0/32174261 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e140cded0 0x7f9e14157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:12.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e19200640 1 --2- 172.21.15.47:0/32174261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e140ceb40 0x7f9e1415a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:12.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e13e00640 1 -- 172.21.15.47:0/32174261 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e140cded0 msgr2=0x7f9e14157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:12.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e13e00640 1 --2- 172.21.15.47:0/32174261 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e140cded0 0x7f9e14157160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:12.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e13e00640 1 -- 172.21.15.47:0/32174261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e140ceb40 msgr2=0x7f9e1415a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:12.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e13e00640 1 --2- 172.21.15.47:0/32174261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e140ceb40 0x7f9e1415a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:12.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e13e00640 1 -- 172.21.15.47:0/32174261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0c009a70 con 0x7f9e1406d650 2024-02-19T15:32:12.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e13e00640 1 --2- 172.21.15.47:0/32174261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9e1406d650 0x7f9e140cbd70 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400c370 tx=0x7f9e0400c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:12.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e10c00640 1 -- 172.21.15.47:0/32174261 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e040074e0 con 0x7f9e1406d650 2024-02-19T15:32:12.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e10c00640 1 -- 172.21.15.47:0/32174261 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e04007640 con 0x7f9e1406d650 2024-02-19T15:32:12.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e10c00640 1 -- 172.21.15.47:0/32174261 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e04004a30 con 0x7f9e1406d650 2024-02-19T15:32:12.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e1b200640 1 -- 172.21.15.47:0/32174261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dfc0094d0 con 0x7f9e1406d650 2024-02-19T15:32:12.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e1b200640 1 -- 172.21.15.47:0/32174261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e14158740 con 0x7f9e1406d650 2024-02-19T15:32:12.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.664+0000 7f9e1b200640 1 -- 172.21.15.47:0/32174261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e14075360 con 0x7f9e1406d650 2024-02-19T15:32:12.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.668+0000 7f9e10c00640 1 -- 172.21.15.47:0/32174261 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9e04004b90 con 0x7f9e1406d650 2024-02-19T15:32:12.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.668+0000 7f9e10c00640 1 --2- 172.21.15.47:0/32174261 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9df0066e40 0x7f9df0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:12.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.668+0000 7f9e10c00640 1 -- 172.21.15.47:0/32174261 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9e040842e0 con 0x7f9e1406d650 2024-02-19T15:32:12.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.668+0000 7f9e0be00640 1 --2- 172.21.15.47:0/32174261 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9df0066e40 0x7f9df0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:12.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.668+0000 7f9e0be00640 1 --2- 172.21.15.47:0/32174261 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9df0066e40 0x7f9df0069300 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc0ae000 tx=0x7f9dfc0b5040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:12.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.676+0000 7f9e10c00640 1 -- 172.21.15.47:0/32174261 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f9e04051080 con 0x7f9e1406d650 2024-02-19T15:32:12.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.872+0000 7f9e1b200640 1 -- 172.21.15.47:0/32174261 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e140708f0 con 0x7f9e1406d650 2024-02-19T15:32:12.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.872+0000 7f9e10c00640 1 -- 172.21.15.47:0/32174261 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f9e04054740 con 0x7f9e1406d650 2024-02-19T15:32:12.874 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:12.874 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:12.874 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:12.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.876+0000 7f9e1b200640 1 -- 172.21.15.47:0/32174261 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9df0066e40 msgr2=0x7f9df0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:12.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.876+0000 7f9e1b200640 1 --2- 172.21.15.47:0/32174261 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9df0066e40 0x7f9df0069300 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc0ae000 tx=0x7f9dfc0b5040 comp rx=0 tx=0).stop 2024-02-19T15:32:12.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.876+0000 7f9e1b200640 1 -- 172.21.15.47:0/32174261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9e1406d650 msgr2=0x7f9e140cbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:12.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.876+0000 7f9e1b200640 1 --2- 172.21.15.47:0/32174261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9e1406d650 0x7f9e140cbd70 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400c370 tx=0x7f9e0400c770 comp rx=0 tx=0).stop 2024-02-19T15:32:12.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.876+0000 7f9e1b200640 1 -- 172.21.15.47:0/32174261 shutdown_connections 2024-02-19T15:32:12.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.876+0000 7f9e1b200640 1 --2- 172.21.15.47:0/32174261 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9df0066e40 0x7f9df0069300 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:12.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.876+0000 7f9e1b200640 1 --2- 172.21.15.47:0/32174261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e140ceb40 0x7f9e1415a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:12.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.876+0000 7f9e1b200640 1 --2- 172.21.15.47:0/32174261 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e140cded0 0x7f9e14157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:12.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.876+0000 7f9e1b200640 1 --2- 172.21.15.47:0/32174261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9e1406d650 0x7f9e140cbd70 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:12.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.876+0000 7f9e1b200640 1 -- 172.21.15.47:0/32174261 >> 172.21.15.47:0/32174261 conn(0x7f9e140d1bf0 msgr2=0x7f9e140d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:12.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.876+0000 7f9e1b200640 1 -- 172.21.15.47:0/32174261 shutdown_connections 2024-02-19T15:32:12.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:12.876+0000 7f9e1b200640 1 -- 172.21.15.47:0/32174261 wait complete. 2024-02-19T15:32:13.098 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:13.098 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:14.099 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:14.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.240+0000 7fb822800640 1 Processor -- start 2024-02-19T15:32:14.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.240+0000 7fb822800640 1 -- start start 2024-02-19T15:32:14.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.240+0000 7fb822800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb81c0cea80 0x7fb81c0727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:14.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.240+0000 7fb822800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb81c15d280 con 0x7fb81c15c390 2024-02-19T15:32:14.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.240+0000 7fb822800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb81c070ff0 con 0x7fb81c145590 2024-02-19T15:32:14.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.240+0000 7fb822800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c0709e0 con 0x7fb81c0cea80 2024-02-19T15:32:14.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.240+0000 7fb81be00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb81c15c390 0x7fb81c15c820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51726/0 (socket says 172.21.15.47:51726) 2024-02-19T15:32:14.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.240+0000 7fb81be00640 1 -- 172.21.15.47:0/339377416 learned_addr learned my addr 172.21.15.47:0/339377416 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:32:14.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.240+0000 7fb812a00640 1 --2- 172.21.15.47:0/339377416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb81c0cea80 0x7fb81c0727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:14.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.240+0000 7fb81aa00640 1 -- 172.21.15.47:0/339377416 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087898611 0 0) 0x7fb81c15d280 con 0x7fb81c15c390 2024-02-19T15:32:14.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.240+0000 7fb81aa00640 1 -- 172.21.15.47:0/339377416 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb800003530 con 0x7fb81c15c390 2024-02-19T15:32:14.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.240+0000 7fb81aa00640 1 -- 172.21.15.47:0/339377416 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549047662 0 0) 0x7fb81c070ff0 con 0x7fb81c145590 2024-02-19T15:32:14.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.240+0000 7fb81aa00640 1 -- 172.21.15.47:0/339377416 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb81c15d280 con 0x7fb81c145590 2024-02-19T15:32:14.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.240+0000 7fb812a00640 1 -- 172.21.15.47:0/339377416 >> v1:172.21.15.116:6789/0 conn(0x7fb81c145590 legacy=0x7fb81c06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:14.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.240+0000 7fb812a00640 1 -- 172.21.15.47:0/339377416 >> v1:172.21.15.47:6789/0 conn(0x7fb81c15c390 legacy=0x7fb81c15c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:14.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.240+0000 7fb812a00640 1 -- 172.21.15.47:0/339377416 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81c0cf020 con 0x7fb81c0cea80 2024-02-19T15:32:14.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.244+0000 7fb812a00640 1 --2- 172.21.15.47:0/339377416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb81c0cea80 0x7fb81c0727b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb8040097f0 tx=0x7fb80401c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6132d1be9a06c883 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:14.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.244+0000 7fb81aa00640 1 -- 172.21.15.47:0/339377416 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8040ab5f0 con 0x7fb81c0cea80 2024-02-19T15:32:14.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.244+0000 7fb81aa00640 1 -- 172.21.15.47:0/339377416 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8040ab750 con 0x7fb81c0cea80 2024-02-19T15:32:14.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.244+0000 7fb81aa00640 1 -- 172.21.15.47:0/339377416 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8040abae0 con 0x7fb81c0cea80 2024-02-19T15:32:14.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.244+0000 7fb822800640 1 -- 172.21.15.47:0/339377416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb81c0cea80 msgr2=0x7fb81c0727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:14.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.244+0000 7fb822800640 1 --2- 172.21.15.47:0/339377416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb81c0cea80 0x7fb81c0727b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb8040097f0 tx=0x7fb80401c4e0 comp rx=0 tx=0).stop 2024-02-19T15:32:14.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.244+0000 7fb822800640 1 -- 172.21.15.47:0/339377416 shutdown_connections 2024-02-19T15:32:14.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.244+0000 7fb822800640 1 --2- 172.21.15.47:0/339377416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb81c0cea80 0x7fb81c0727b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:14.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.244+0000 7fb822800640 1 -- 172.21.15.47:0/339377416 >> 172.21.15.47:0/339377416 conn(0x7fb81c0b16a0 msgr2=0x7fb81c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:14.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.244+0000 7fb822800640 1 -- 172.21.15.47:0/339377416 shutdown_connections 2024-02-19T15:32:14.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.244+0000 7fb822800640 1 -- 172.21.15.47:0/339377416 wait complete. 2024-02-19T15:32:14.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb822800640 1 Processor -- start 2024-02-19T15:32:14.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb822800640 1 -- start start 2024-02-19T15:32:14.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb822800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb81c0cea80 0x7fb81c0c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:14.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb822800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81c145590 0x7fb81c0b3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:14.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb822800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb81c15c390 0x7fb81c0c6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:14.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb822800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c0c59d0 con 0x7fb81c15c390 2024-02-19T15:32:14.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb822800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c0c6fc0 con 0x7fb81c145590 2024-02-19T15:32:14.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb822800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c0c49b0 con 0x7fb81c0cea80 2024-02-19T15:32:14.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb81be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb81c15c390 0x7fb81c0c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:14.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb81be00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb81c15c390 0x7fb81c0c6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49764/0 (socket says 172.21.15.47:49764) 2024-02-19T15:32:14.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb81be00640 1 -- 172.21.15.47:0/4165106044 learned_addr learned my addr 172.21.15.47:0/4165106044 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:14.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb812a00640 1 --2- 172.21.15.47:0/4165106044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81c145590 0x7fb81c0b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:14.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb81b400640 1 --2- 172.21.15.47:0/4165106044 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb81c0cea80 0x7fb81c0c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:14.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb81be00640 1 -- 172.21.15.47:0/4165106044 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb81c0cea80 msgr2=0x7fb81c0c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:14.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb81be00640 1 --2- 172.21.15.47:0/4165106044 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb81c0cea80 0x7fb81c0c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:14.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb81be00640 1 -- 172.21.15.47:0/4165106044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81c145590 msgr2=0x7fb81c0b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:14.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb81be00640 1 --2- 172.21.15.47:0/4165106044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81c145590 0x7fb81c0b3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:14.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb81be00640 1 -- 172.21.15.47:0/4165106044 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb804009420 con 0x7fb81c15c390 2024-02-19T15:32:14.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb81be00640 1 --2- 172.21.15.47:0/4165106044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb81c15c390 0x7fb81c0c6540 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fb81400caf0 tx=0x7fb81400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:14.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb813e00640 1 -- 172.21.15.47:0/4165106044 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81400aca0 con 0x7fb81c15c390 2024-02-19T15:32:14.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb813e00640 1 -- 172.21.15.47:0/4165106044 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81400ae00 con 0x7fb81c15c390 2024-02-19T15:32:14.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb813e00640 1 -- 172.21.15.47:0/4165106044 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb814012620 con 0x7fb81c15c390 2024-02-19T15:32:14.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb822800640 1 -- 172.21.15.47:0/4165106044 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb81c0c4c90 con 0x7fb81c15c390 2024-02-19T15:32:14.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.248+0000 7fb822800640 1 -- 172.21.15.47:0/4165106044 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81c0c72a0 con 0x7fb81c15c390 2024-02-19T15:32:14.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.252+0000 7fb822800640 1 -- 172.21.15.47:0/4165106044 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81c075360 con 0x7fb81c15c390 2024-02-19T15:32:14.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.252+0000 7fb813e00640 1 -- 172.21.15.47:0/4165106044 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fb814007070 con 0x7fb81c15c390 2024-02-19T15:32:14.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.252+0000 7fb813e00640 1 --2- 172.21.15.47:0/4165106044 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb7f8066e40 0x7fb7f8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:14.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.252+0000 7fb813e00640 1 -- 172.21.15.47:0/4165106044 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb814084a30 con 0x7fb81c15c390 2024-02-19T15:32:14.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.252+0000 7fb81b400640 1 --2- 172.21.15.47:0/4165106044 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb7f8066e40 0x7fb7f8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:14.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.252+0000 7fb81b400640 1 --2- 172.21.15.47:0/4165106044 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb7f8066e40 0x7fb7f8069300 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb80c0011f0 tx=0x7fb80c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:14.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.256+0000 7fb813e00640 1 -- 172.21.15.47:0/4165106044 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fb8140517f0 con 0x7fb81c15c390 2024-02-19T15:32:14.304 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:32:14.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.496+0000 7fb822800640 1 -- 172.21.15.47:0/4165106044 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb81c0c5140 con 0x7fb81c15c390 2024-02-19T15:32:14.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.496+0000 7fb813e00640 1 -- 172.21.15.47:0/4165106044 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7fb814017020 con 0x7fb81c15c390 2024-02-19T15:32:14.496 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:14.496 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:14.496 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:14.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.496+0000 7fb822800640 1 -- 172.21.15.47:0/4165106044 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb7f8066e40 msgr2=0x7fb7f8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:14.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.496+0000 7fb822800640 1 --2- 172.21.15.47:0/4165106044 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb7f8066e40 0x7fb7f8069300 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb80c0011f0 tx=0x7fb80c009040 comp rx=0 tx=0).stop 2024-02-19T15:32:14.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.496+0000 7fb822800640 1 -- 172.21.15.47:0/4165106044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb81c15c390 msgr2=0x7fb81c0c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:14.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.496+0000 7fb822800640 1 --2- 172.21.15.47:0/4165106044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb81c15c390 0x7fb81c0c6540 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fb81400caf0 tx=0x7fb81400a060 comp rx=0 tx=0).stop 2024-02-19T15:32:14.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.496+0000 7fb822800640 1 -- 172.21.15.47:0/4165106044 shutdown_connections 2024-02-19T15:32:14.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.496+0000 7fb822800640 1 --2- 172.21.15.47:0/4165106044 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fb7f8066e40 0x7fb7f8069300 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:14.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.496+0000 7fb822800640 1 --2- 172.21.15.47:0/4165106044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb81c15c390 0x7fb81c0c6540 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:14.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.496+0000 7fb822800640 1 --2- 172.21.15.47:0/4165106044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81c145590 0x7fb81c0b3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:14.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.496+0000 7fb822800640 1 --2- 172.21.15.47:0/4165106044 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb81c0cea80 0x7fb81c0c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:14.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.496+0000 7fb822800640 1 -- 172.21.15.47:0/4165106044 >> 172.21.15.47:0/4165106044 conn(0x7fb81c0b16a0 msgr2=0x7fb81c0af6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:14.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.500+0000 7fb822800640 1 -- 172.21.15.47:0/4165106044 shutdown_connections 2024-02-19T15:32:14.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:14.500+0000 7fb822800640 1 -- 172.21.15.47:0/4165106044 wait complete. 2024-02-19T15:32:14.726 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:14.726 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:15.727 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:15.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.868+0000 7fec65000640 1 Processor -- start 2024-02-19T15:32:15.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.868+0000 7fec65000640 1 -- start start 2024-02-19T15:32:15.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.868+0000 7fec65000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec601476e0 0x7fec600ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:15.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.868+0000 7fec65000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fec600b33c0 0x7fec60071950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:15.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.868+0000 7fec65000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec600af1b0 con 0x7fec6006c9a0 2024-02-19T15:32:15.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.868+0000 7fec65000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec60071530 con 0x7fec600b33c0 2024-02-19T15:32:15.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.868+0000 7fec65000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec60070980 con 0x7fec601476e0 2024-02-19T15:32:15.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.868+0000 7fec5e800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fec600b33c0 0x7fec60071950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:15.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.868+0000 7fec5e800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fec600b33c0 0x7fec60071950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49776/0 (socket says 172.21.15.47:49776) 2024-02-19T15:32:15.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.868+0000 7fec5e800640 1 -- 172.21.15.47:0/182047252 learned_addr learned my addr 172.21.15.47:0/182047252 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:15.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.868+0000 7fec5de00640 1 --2- 172.21.15.47:0/182047252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec601476e0 0x7fec600ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:15.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.868+0000 7fec5e800640 1 -- 172.21.15.47:0/182047252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec601476e0 msgr2=0x7fec600ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:15.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.868+0000 7fec5e800640 1 --2- 172.21.15.47:0/182047252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec601476e0 0x7fec600ae730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:15.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.868+0000 7fec5e800640 1 -- 172.21.15.47:0/182047252 >> v1:172.21.15.47:6789/0 conn(0x7fec6006c9a0 legacy=0x7fec600b2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:15.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.868+0000 7fec5e800640 1 -- 172.21.15.47:0/182047252 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec60070b90 con 0x7fec600b33c0 2024-02-19T15:32:15.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.872+0000 7fec5e800640 1 --2- 172.21.15.47:0/182047252 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec600b33c0 0x7fec60071950 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fec58009e50 tx=0x7fec5801ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df185548f0b0776e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:15.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.872+0000 7fec5d400640 1 -- 172.21.15.47:0/182047252 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec580a58d0 con 0x7fec600b33c0 2024-02-19T15:32:15.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.872+0000 7fec5d400640 1 -- 172.21.15.47:0/182047252 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec580a5a30 con 0x7fec600b33c0 2024-02-19T15:32:15.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.872+0000 7fec5d400640 1 -- 172.21.15.47:0/182047252 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec580a5dc0 con 0x7fec600b33c0 2024-02-19T15:32:15.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.872+0000 7fec65000640 1 -- 172.21.15.47:0/182047252 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec600b33c0 msgr2=0x7fec60071950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:15.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.872+0000 7fec65000640 1 --2- 172.21.15.47:0/182047252 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec600b33c0 0x7fec60071950 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fec58009e50 tx=0x7fec5801ca00 comp rx=0 tx=0).stop 2024-02-19T15:32:15.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.872+0000 7fec65000640 1 -- 172.21.15.47:0/182047252 shutdown_connections 2024-02-19T15:32:15.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.872+0000 7fec65000640 1 --2- 172.21.15.47:0/182047252 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec600b33c0 0x7fec60071950 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:15.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.872+0000 7fec65000640 1 --2- 172.21.15.47:0/182047252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec601476e0 0x7fec600ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:15.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.872+0000 7fec65000640 1 -- 172.21.15.47:0/182047252 >> 172.21.15.47:0/182047252 conn(0x7fec600b16a0 msgr2=0x7fec600b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:15.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.872+0000 7fec65000640 1 -- 172.21.15.47:0/182047252 shutdown_connections 2024-02-19T15:32:15.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.872+0000 7fec65000640 1 -- 172.21.15.47:0/182047252 wait complete. 2024-02-19T15:32:15.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.872+0000 7fec65000640 1 Processor -- start 2024-02-19T15:32:15.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec65000640 1 -- start start 2024-02-19T15:32:15.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec65000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec6006c9a0 0x7fec60142080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:15.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec65000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec600b33c0 0x7fec6015d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:15.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec65000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec601476e0 0x7fec6013f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:15.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec65000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec600ce370 con 0x7fec6006c9a0 2024-02-19T15:32:15.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec65000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec60143f40 con 0x7fec601476e0 2024-02-19T15:32:15.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec65000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec601425c0 con 0x7fec600b33c0 2024-02-19T15:32:15.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec5de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec6006c9a0 0x7fec60142080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:15.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec5de00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec6006c9a0 0x7fec60142080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49792/0 (socket says 172.21.15.47:49792) 2024-02-19T15:32:15.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec5de00640 1 -- 172.21.15.47:0/2031792664 learned_addr learned my addr 172.21.15.47:0/2031792664 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:15.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec57e00640 1 --2- 172.21.15.47:0/2031792664 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec600b33c0 0x7fec6015d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:15.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec5e800640 1 --2- 172.21.15.47:0/2031792664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec601476e0 0x7fec6013f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:15.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec5de00640 1 -- 172.21.15.47:0/2031792664 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec600b33c0 msgr2=0x7fec6015d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:15.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec5de00640 1 --2- 172.21.15.47:0/2031792664 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec600b33c0 0x7fec6015d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:15.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec5de00640 1 -- 172.21.15.47:0/2031792664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec601476e0 msgr2=0x7fec6013f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:15.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec5de00640 1 --2- 172.21.15.47:0/2031792664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec601476e0 0x7fec6013f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:15.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec5de00640 1 -- 172.21.15.47:0/2031792664 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec58009b30 con 0x7fec6006c9a0 2024-02-19T15:32:15.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec5de00640 1 --2- 172.21.15.47:0/2031792664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec6006c9a0 0x7fec60142080 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fec5000d800 tx=0x7fec5000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:15.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec56000640 1 -- 172.21.15.47:0/2031792664 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5000b6c0 con 0x7fec6006c9a0 2024-02-19T15:32:15.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec56000640 1 -- 172.21.15.47:0/2031792664 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5000b820 con 0x7fec6006c9a0 2024-02-19T15:32:15.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec56000640 1 -- 172.21.15.47:0/2031792664 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5000bb90 con 0x7fec6006c9a0 2024-02-19T15:32:15.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec65000640 1 -- 172.21.15.47:0/2031792664 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec60142ea0 con 0x7fec6006c9a0 2024-02-19T15:32:15.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec65000640 1 -- 172.21.15.47:0/2031792664 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec60143460 con 0x7fec6006c9a0 2024-02-19T15:32:15.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.876+0000 7fec65000640 1 -- 172.21.15.47:0/2031792664 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec60075360 con 0x7fec6006c9a0 2024-02-19T15:32:15.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.880+0000 7fec56000640 1 -- 172.21.15.47:0/2031792664 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fec5000bcf0 con 0x7fec6006c9a0 2024-02-19T15:32:15.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.880+0000 7fec56000640 1 --2- 172.21.15.47:0/2031792664 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fec3c066e70 0x7fec3c069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:15.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.880+0000 7fec56000640 1 -- 172.21.15.47:0/2031792664 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fec50084ff0 con 0x7fec6006c9a0 2024-02-19T15:32:15.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.880+0000 7fec57e00640 1 --2- 172.21.15.47:0/2031792664 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fec3c066e70 0x7fec3c069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:15.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.880+0000 7fec57e00640 1 --2- 172.21.15.47:0/2031792664 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fec3c066e70 0x7fec3c069330 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fec601401f0 tx=0x7fec48009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:15.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:15.884+0000 7fec56000640 1 -- 172.21.15.47:0/2031792664 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fec50051d60 con 0x7fec6006c9a0 2024-02-19T15:32:16.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:16.148+0000 7fec65000640 1 -- 172.21.15.47:0/2031792664 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec600cf3f0 con 0x7fec6006c9a0 2024-02-19T15:32:16.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:16.148+0000 7fec56000640 1 -- 172.21.15.47:0/2031792664 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7fec50011340 con 0x7fec6006c9a0 2024-02-19T15:32:16.149 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:16.150 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:16.150 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:16.152+0000 7fec65000640 1 -- 172.21.15.47:0/2031792664 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fec3c066e70 msgr2=0x7fec3c069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:16.152+0000 7fec65000640 1 --2- 172.21.15.47:0/2031792664 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fec3c066e70 0x7fec3c069330 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fec601401f0 tx=0x7fec48009040 comp rx=0 tx=0).stop 2024-02-19T15:32:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:16.152+0000 7fec65000640 1 -- 172.21.15.47:0/2031792664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec6006c9a0 msgr2=0x7fec60142080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:16.152+0000 7fec65000640 1 --2- 172.21.15.47:0/2031792664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec6006c9a0 0x7fec60142080 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fec5000d800 tx=0x7fec5000dcc0 comp rx=0 tx=0).stop 2024-02-19T15:32:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:16.152+0000 7fec65000640 1 -- 172.21.15.47:0/2031792664 shutdown_connections 2024-02-19T15:32:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:16.152+0000 7fec65000640 1 --2- 172.21.15.47:0/2031792664 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fec3c066e70 0x7fec3c069330 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:16.152+0000 7fec65000640 1 --2- 172.21.15.47:0/2031792664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec601476e0 0x7fec6013f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:16.152+0000 7fec65000640 1 --2- 172.21.15.47:0/2031792664 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec600b33c0 0x7fec6015d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:16.152+0000 7fec65000640 1 --2- 172.21.15.47:0/2031792664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec6006c9a0 0x7fec60142080 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:16.152+0000 7fec65000640 1 -- 172.21.15.47:0/2031792664 >> 172.21.15.47:0/2031792664 conn(0x7fec600b16a0 msgr2=0x7fec60140be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:16.152+0000 7fec65000640 1 -- 172.21.15.47:0/2031792664 shutdown_connections 2024-02-19T15:32:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:16.152+0000 7fec65000640 1 -- 172.21.15.47:0/2031792664 wait complete. 2024-02-19T15:32:16.306 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:32:16.372 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:16.373 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:17.374 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:17.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.516+0000 7f36f7c00640 1 Processor -- start 2024-02-19T15:32:17.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.516+0000 7f36f7c00640 1 -- start start 2024-02-19T15:32:17.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.516+0000 7f36f7c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f006aeb0 0x7f36f006b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:17.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.516+0000 7f36f7c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f36f00b3a20 0x7f36f00b3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:17.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.516+0000 7f36f7c00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f0071a00 con 0x7f36f0145740 2024-02-19T15:32:17.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.516+0000 7f36f7c00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36f006b870 con 0x7f36f00b3a20 2024-02-19T15:32:17.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.516+0000 7f36f7c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f0071ce0 con 0x7f36f006aeb0 2024-02-19T15:32:17.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.516+0000 7f36efe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f006aeb0 0x7f36f006b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:17.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.516+0000 7f36f5c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f36f00b3a20 0x7f36f00b3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:17.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.516+0000 7f36efe00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f006aeb0 0x7f36f006b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.47:49474/0 (socket says 172.21.15.47:49474) 2024-02-19T15:32:17.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.516+0000 7f36efe00640 1 -- 172.21.15.47:0/49013691 learned_addr learned my addr 172.21.15.47:0/49013691 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:17.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.516+0000 7f36efe00640 1 -- 172.21.15.47:0/49013691 >> v2:172.21.15.116:3300/0 conn(0x7f36f00b3a20 msgr2=0x7f36f00b3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:17.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.516+0000 7f36efe00640 1 --2- 172.21.15.47:0/49013691 >> v2:172.21.15.116:3300/0 conn(0x7f36f00b3a20 0x7f36f00b3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:17.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.516+0000 7f36efe00640 1 -- 172.21.15.47:0/49013691 >> v1:172.21.15.116:6789/0 conn(0x7f36f0145740 legacy=0x7f36f00cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:17.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.516+0000 7f36ef400640 1 -- 172.21.15.47:0/49013691 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769230426 0 0) 0x7f36f0071a00 con 0x7f36f0145740 2024-02-19T15:32:17.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.516+0000 7f36efe00640 1 -- 172.21.15.47:0/49013691 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f0071fc0 con 0x7f36f006aeb0 2024-02-19T15:32:17.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.520+0000 7f36efe00640 1 --2- 172.21.15.47:0/49013691 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f006aeb0 0x7f36f006b330 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009870 tx=0x7f36dc01c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47ae13cecc75b550 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:17.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.520+0000 7f36ef400640 1 -- 172.21.15.47:0/49013691 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36dc0ab610 con 0x7f36f006aeb0 2024-02-19T15:32:17.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.520+0000 7f36ef400640 1 -- 172.21.15.47:0/49013691 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36dc0ab770 con 0x7f36f006aeb0 2024-02-19T15:32:17.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.520+0000 7f36ef400640 1 -- 172.21.15.47:0/49013691 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36dc0abb00 con 0x7f36f006aeb0 2024-02-19T15:32:17.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.520+0000 7f36f7c00640 1 -- 172.21.15.47:0/49013691 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f006aeb0 msgr2=0x7f36f006b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:17.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.520+0000 7f36f7c00640 1 --2- 172.21.15.47:0/49013691 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f006aeb0 0x7f36f006b330 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009870 tx=0x7f36dc01c410 comp rx=0 tx=0).stop 2024-02-19T15:32:17.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.520+0000 7f36f7c00640 1 -- 172.21.15.47:0/49013691 shutdown_connections 2024-02-19T15:32:17.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.520+0000 7f36f7c00640 1 --2- 172.21.15.47:0/49013691 >> v2:172.21.15.116:3300/0 conn(0x7f36f00b3a20 0x7f36f00b3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:17.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.520+0000 7f36f7c00640 1 --2- 172.21.15.47:0/49013691 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f006aeb0 0x7f36f006b330 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:17.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.520+0000 7f36f7c00640 1 -- 172.21.15.47:0/49013691 >> 172.21.15.47:0/49013691 conn(0x7f36f00d0a50 msgr2=0x7f36f00d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:17.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.520+0000 7f36f7c00640 1 -- 172.21.15.47:0/49013691 shutdown_connections 2024-02-19T15:32:17.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.520+0000 7f36f7c00640 1 -- 172.21.15.47:0/49013691 wait complete. 2024-02-19T15:32:17.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36f7c00640 1 Processor -- start 2024-02-19T15:32:17.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36f7c00640 1 -- start start 2024-02-19T15:32:17.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36f7c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f006aeb0 0x7f36f015b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:17.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36f7c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f36f00b3a20 0x7f36f00c9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:17.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36f7c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0145740 0x7f36f00ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:17.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36f7c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f006d780 con 0x7f36f00b3a20 2024-02-19T15:32:17.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36f7c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f006e020 con 0x7f36f006aeb0 2024-02-19T15:32:17.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36f7c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f015e1a0 con 0x7f36f0145740 2024-02-19T15:32:17.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36efe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f36f00b3a20 0x7f36f00c9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:17.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36f5200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f006aeb0 0x7f36f015b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:17.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36efe00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f36f00b3a20 0x7f36f00c9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49798/0 (socket says 172.21.15.47:49798) 2024-02-19T15:32:17.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36efe00640 1 -- 172.21.15.47:0/2628454669 learned_addr learned my addr 172.21.15.47:0/2628454669 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:17.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36f5c00640 1 --2- 172.21.15.47:0/2628454669 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0145740 0x7f36f00ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:17.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36efe00640 1 -- 172.21.15.47:0/2628454669 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0145740 msgr2=0x7f36f00ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:17.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36efe00640 1 --2- 172.21.15.47:0/2628454669 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0145740 0x7f36f00ca2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:17.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36efe00640 1 -- 172.21.15.47:0/2628454669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f006aeb0 msgr2=0x7f36f015b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:17.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36efe00640 1 --2- 172.21.15.47:0/2628454669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f006aeb0 0x7f36f015b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:17.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36efe00640 1 -- 172.21.15.47:0/2628454669 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36dc0094d0 con 0x7f36f00b3a20 2024-02-19T15:32:17.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36f5c00640 1 --2- 172.21.15.47:0/2628454669 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0145740 0x7f36f00ca2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:32:17.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36efe00640 1 --2- 172.21.15.47:0/2628454669 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f36f00b3a20 0x7f36f00c9da0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f36dc0097c0 tx=0x7f36dc01ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:17.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36ecc00640 1 -- 172.21.15.47:0/2628454669 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36dc0b4800 con 0x7f36f00b3a20 2024-02-19T15:32:17.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36ecc00640 1 -- 172.21.15.47:0/2628454669 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36dc0b4960 con 0x7f36f00b3a20 2024-02-19T15:32:17.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36ecc00640 1 -- 172.21.15.47:0/2628454669 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36dc0b4d10 con 0x7f36f00b3a20 2024-02-19T15:32:17.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36f7c00640 1 -- 172.21.15.47:0/2628454669 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f006d9d0 con 0x7f36f00b3a20 2024-02-19T15:32:17.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.524+0000 7f36f7c00640 1 -- 172.21.15.47:0/2628454669 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36f015d430 con 0x7f36f00b3a20 2024-02-19T15:32:17.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.528+0000 7f36f7c00640 1 -- 172.21.15.47:0/2628454669 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f006e190 con 0x7f36f00b3a20 2024-02-19T15:32:17.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.528+0000 7f36ecc00640 1 -- 172.21.15.47:0/2628454669 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f36dc0c7440 con 0x7f36f00b3a20 2024-02-19T15:32:17.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.528+0000 7f36ecc00640 1 --2- 172.21.15.47:0/2628454669 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f36d0066e40 0x7f36d0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:17.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.528+0000 7f36ecc00640 1 -- 172.21.15.47:0/2628454669 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f36dc11f470 con 0x7f36f00b3a20 2024-02-19T15:32:17.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.528+0000 7f36f5200640 1 --2- 172.21.15.47:0/2628454669 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f36d0066e40 0x7f36d0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:17.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.528+0000 7f36f5200640 1 --2- 172.21.15.47:0/2628454669 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f36d0066e40 0x7f36d0069300 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f36e4007fd0 tx=0x7f36e4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:17.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.536+0000 7f36ecc00640 1 -- 172.21.15.47:0/2628454669 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f36dc0ec360 con 0x7f36f00b3a20 2024-02-19T15:32:17.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.728+0000 7f36f7c00640 1 -- 172.21.15.47:0/2628454669 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36f0075360 con 0x7f36f00b3a20 2024-02-19T15:32:17.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.732+0000 7f36ecc00640 1 -- 172.21.15.47:0/2628454669 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f36dc0ab5d0 con 0x7f36f00b3a20 2024-02-19T15:32:17.730 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:17.730 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:17.731 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:17.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.732+0000 7f36f7c00640 1 -- 172.21.15.47:0/2628454669 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f36d0066e40 msgr2=0x7f36d0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:17.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.732+0000 7f36f7c00640 1 --2- 172.21.15.47:0/2628454669 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f36d0066e40 0x7f36d0069300 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f36e4007fd0 tx=0x7f36e4009040 comp rx=0 tx=0).stop 2024-02-19T15:32:17.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.732+0000 7f36f7c00640 1 -- 172.21.15.47:0/2628454669 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f36f00b3a20 msgr2=0x7f36f00c9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:17.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.732+0000 7f36f7c00640 1 --2- 172.21.15.47:0/2628454669 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f36f00b3a20 0x7f36f00c9da0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f36dc0097c0 tx=0x7f36dc01ac50 comp rx=0 tx=0).stop 2024-02-19T15:32:17.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.732+0000 7f36f7c00640 1 -- 172.21.15.47:0/2628454669 shutdown_connections 2024-02-19T15:32:17.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.732+0000 7f36f7c00640 1 --2- 172.21.15.47:0/2628454669 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f36d0066e40 0x7f36d0069300 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:17.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.732+0000 7f36f7c00640 1 --2- 172.21.15.47:0/2628454669 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0145740 0x7f36f00ca2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:17.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.732+0000 7f36f7c00640 1 --2- 172.21.15.47:0/2628454669 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f36f00b3a20 0x7f36f00c9da0 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:17.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.732+0000 7f36f7c00640 1 --2- 172.21.15.47:0/2628454669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f006aeb0 0x7f36f015b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:17.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.732+0000 7f36f7c00640 1 -- 172.21.15.47:0/2628454669 >> 172.21.15.47:0/2628454669 conn(0x7f36f00d0a50 msgr2=0x7f36f0143d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:17.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.732+0000 7f36f7c00640 1 -- 172.21.15.47:0/2628454669 shutdown_connections 2024-02-19T15:32:17.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:17.732+0000 7f36f7c00640 1 -- 172.21.15.47:0/2628454669 wait complete. 2024-02-19T15:32:17.952 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:17.953 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:18.308 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:32:18.953 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:19.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.092+0000 7f9121e00640 1 Processor -- start 2024-02-19T15:32:19.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.092+0000 7f9121e00640 1 -- start start 2024-02-19T15:32:19.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.096+0000 7f9121e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f911c0cff60 0x7f911c0d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.096+0000 7f9121e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f911c15e270 0x7f911c0d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.096+0000 7f9121e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f911c0d1760 0x7f911c0ce320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.096+0000 7f9121e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f911c0b3520 con 0x7f911c0cff60 2024-02-19T15:32:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.096+0000 7f9121e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f911c0b3240 con 0x7f911c15e270 2024-02-19T15:32:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.096+0000 7f9121e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f911c06cb50 con 0x7f911c0d1760 2024-02-19T15:32:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.096+0000 7f911aa00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f911c0cff60 0x7f911c0d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.096+0000 7f911aa00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f911c0cff60 0x7f911c0d0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49806/0 (socket says 172.21.15.47:49806) 2024-02-19T15:32:19.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.096+0000 7f911aa00640 1 -- 172.21.15.47:0/3537323639 learned_addr learned my addr 172.21.15.47:0/3537323639 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:19.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.096+0000 7f911b400640 1 --2- 172.21.15.47:0/3537323639 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f911c0d1760 0x7f911c0ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:19.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.096+0000 7f9112000640 1 --2- 172.21.15.47:0/3537323639 >> v2:172.21.15.116:3300/0 conn(0x7f911c15e270 0x7f911c0d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:19.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.096+0000 7f911aa00640 1 -- 172.21.15.47:0/3537323639 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f911c0d1760 msgr2=0x7f911c0ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:19.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.096+0000 7f911aa00640 1 --2- 172.21.15.47:0/3537323639 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f911c0d1760 0x7f911c0ce320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:19.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.096+0000 7f911aa00640 1 -- 172.21.15.47:0/3537323639 >> v2:172.21.15.116:3300/0 conn(0x7f911c15e270 msgr2=0x7f911c0d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:19.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.096+0000 7f911aa00640 1 --2- 172.21.15.47:0/3537323639 >> v2:172.21.15.116:3300/0 conn(0x7f911c15e270 0x7f911c0d0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:19.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.096+0000 7f911aa00640 1 -- 172.21.15.47:0/3537323639 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911c06ce30 con 0x7f911c0cff60 2024-02-19T15:32:19.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.096+0000 7f911b400640 1 --2- 172.21.15.47:0/3537323639 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f911c0d1760 0x7f911c0ce320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:32:19.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.096+0000 7f9112000640 1 --2- 172.21.15.47:0/3537323639 >> v2:172.21.15.116:3300/0 conn(0x7f911c15e270 0x7f911c0d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:32:19.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.100+0000 7f911aa00640 1 --2- 172.21.15.47:0/3537323639 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f911c0cff60 0x7f911c0d0dc0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f911c072f90 tx=0x7f910c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6ee0abc30515f6a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:19.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.100+0000 7f911a000640 1 -- 172.21.15.47:0/3537323639 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910c0ac640 con 0x7f911c0cff60 2024-02-19T15:32:19.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.100+0000 7f911a000640 1 -- 172.21.15.47:0/3537323639 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f910c0ac7a0 con 0x7f911c0cff60 2024-02-19T15:32:19.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.100+0000 7f911a000640 1 -- 172.21.15.47:0/3537323639 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910c0acb30 con 0x7f911c0cff60 2024-02-19T15:32:19.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.100+0000 7f9121e00640 1 -- 172.21.15.47:0/3537323639 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f911c0cff60 msgr2=0x7f911c0d0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:19.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.100+0000 7f9121e00640 1 --2- 172.21.15.47:0/3537323639 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f911c0cff60 0x7f911c0d0dc0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f911c072f90 tx=0x7f910c01c490 comp rx=0 tx=0).stop 2024-02-19T15:32:19.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.100+0000 7f9121e00640 1 -- 172.21.15.47:0/3537323639 shutdown_connections 2024-02-19T15:32:19.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.100+0000 7f9121e00640 1 --2- 172.21.15.47:0/3537323639 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f911c0d1760 0x7f911c0ce320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:19.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.100+0000 7f9121e00640 1 --2- 172.21.15.47:0/3537323639 >> v2:172.21.15.116:3300/0 conn(0x7f911c15e270 0x7f911c0d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:19.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.100+0000 7f9121e00640 1 --2- 172.21.15.47:0/3537323639 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f911c0cff60 0x7f911c0d0dc0 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:19.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.100+0000 7f9121e00640 1 -- 172.21.15.47:0/3537323639 >> 172.21.15.47:0/3537323639 conn(0x7f911c0d1b90 msgr2=0x7f911c0d1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:19.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.100+0000 7f9121e00640 1 -- 172.21.15.47:0/3537323639 shutdown_connections 2024-02-19T15:32:19.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.100+0000 7f9121e00640 1 -- 172.21.15.47:0/3537323639 wait complete. 2024-02-19T15:32:19.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9121e00640 1 Processor -- start 2024-02-19T15:32:19.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9121e00640 1 -- start start 2024-02-19T15:32:19.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9121e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f911c0cff60 0x7f911c150e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:19.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9121e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f911c0d1760 0x7f911c06ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:19.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9121e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f911c15e270 0x7f911c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:19.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9121e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911c1607c0 con 0x7f911c0d1760 2024-02-19T15:32:19.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9121e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911c1603a0 con 0x7f911c0cff60 2024-02-19T15:32:19.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9121e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f911c15f720 con 0x7f911c15e270 2024-02-19T15:32:19.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9112000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f911c0d1760 0x7f911c06ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:19.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9112000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f911c0d1760 0x7f911c06ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49818/0 (socket says 172.21.15.47:49818) 2024-02-19T15:32:19.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9112000640 1 -- 172.21.15.47:0/856092611 learned_addr learned my addr 172.21.15.47:0/856092611 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:19.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f911aa00640 1 --2- 172.21.15.47:0/856092611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f911c0cff60 0x7f911c150e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:19.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f911b400640 1 --2- 172.21.15.47:0/856092611 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f911c15e270 0x7f911c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:19.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9112000640 1 -- 172.21.15.47:0/856092611 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f911c15e270 msgr2=0x7f911c06f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:19.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9112000640 1 --2- 172.21.15.47:0/856092611 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f911c15e270 0x7f911c06f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:19.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9112000640 1 -- 172.21.15.47:0/856092611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f911c0cff60 msgr2=0x7f911c150e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:19.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9112000640 1 --2- 172.21.15.47:0/856092611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f911c0cff60 0x7f911c150e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:19.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9112000640 1 -- 172.21.15.47:0/856092611 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910c0094d0 con 0x7f911c0d1760 2024-02-19T15:32:19.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f911aa00640 1 --2- 172.21.15.47:0/856092611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f911c0cff60 0x7f911c150e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:32:19.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9112000640 1 --2- 172.21.15.47:0/856092611 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f911c0d1760 0x7f911c06ec60 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f910400c7f0 tx=0x7f910400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:19.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9113e00640 1 -- 172.21.15.47:0/856092611 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9104007820 con 0x7f911c0d1760 2024-02-19T15:32:19.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9113e00640 1 -- 172.21.15.47:0/856092611 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9104007980 con 0x7f911c0d1760 2024-02-19T15:32:19.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9113e00640 1 -- 172.21.15.47:0/856092611 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9104007d30 con 0x7f911c0d1760 2024-02-19T15:32:19.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9121e00640 1 -- 172.21.15.47:0/856092611 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f911c15fa00 con 0x7f911c0d1760 2024-02-19T15:32:19.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.104+0000 7f9121e00640 1 -- 172.21.15.47:0/856092611 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f911c15ffc0 con 0x7f911c0d1760 2024-02-19T15:32:19.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.108+0000 7f9121e00640 1 -- 172.21.15.47:0/856092611 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f911c068500 con 0x7f911c0d1760 2024-02-19T15:32:19.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.108+0000 7f9113e00640 1 -- 172.21.15.47:0/856092611 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f91040113f0 con 0x7f911c0d1760 2024-02-19T15:32:19.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.108+0000 7f9113e00640 1 --2- 172.21.15.47:0/856092611 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f90f8066e40 0x7f90f8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:19.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.108+0000 7f9113e00640 1 -- 172.21.15.47:0/856092611 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9104084e60 con 0x7f911c0d1760 2024-02-19T15:32:19.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.108+0000 7f911aa00640 1 --2- 172.21.15.47:0/856092611 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f90f8066e40 0x7f90f8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:19.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.112+0000 7f911aa00640 1 --2- 172.21.15.47:0/856092611 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f90f8066e40 0x7f90f8069300 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f910c0ae000 tx=0x7f910c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:19.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.120+0000 7f9113e00640 1 -- 172.21.15.47:0/856092611 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f9104051b70 con 0x7f911c0d1760 2024-02-19T15:32:19.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.316+0000 7f9121e00640 1 -- 172.21.15.47:0/856092611 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f911c06c840 con 0x7f911c0d1760 2024-02-19T15:32:19.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.316+0000 7f9113e00640 1 -- 172.21.15.47:0/856092611 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f910401f090 con 0x7f911c0d1760 2024-02-19T15:32:19.315 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:19.315 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:19.315 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:19.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.316+0000 7f9121e00640 1 -- 172.21.15.47:0/856092611 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f90f8066e40 msgr2=0x7f90f8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:19.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.316+0000 7f9121e00640 1 --2- 172.21.15.47:0/856092611 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f90f8066e40 0x7f90f8069300 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f910c0ae000 tx=0x7f910c008040 comp rx=0 tx=0).stop 2024-02-19T15:32:19.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.316+0000 7f9121e00640 1 -- 172.21.15.47:0/856092611 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f911c0d1760 msgr2=0x7f911c06ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:19.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.316+0000 7f9121e00640 1 --2- 172.21.15.47:0/856092611 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f911c0d1760 0x7f911c06ec60 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f910400c7f0 tx=0x7f910400ccb0 comp rx=0 tx=0).stop 2024-02-19T15:32:19.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.316+0000 7f9121e00640 1 -- 172.21.15.47:0/856092611 shutdown_connections 2024-02-19T15:32:19.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.316+0000 7f9121e00640 1 --2- 172.21.15.47:0/856092611 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f90f8066e40 0x7f90f8069300 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:19.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.316+0000 7f9121e00640 1 --2- 172.21.15.47:0/856092611 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f911c15e270 0x7f911c06f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:19.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.316+0000 7f9121e00640 1 --2- 172.21.15.47:0/856092611 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f911c0d1760 0x7f911c06ec60 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:19.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.316+0000 7f9121e00640 1 --2- 172.21.15.47:0/856092611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f911c0cff60 0x7f911c150e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:19.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.316+0000 7f9121e00640 1 -- 172.21.15.47:0/856092611 >> 172.21.15.47:0/856092611 conn(0x7f911c0d1b90 msgr2=0x7f911c0703f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:19.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.316+0000 7f9121e00640 1 -- 172.21.15.47:0/856092611 shutdown_connections 2024-02-19T15:32:19.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:19.316+0000 7f9121e00640 1 -- 172.21.15.47:0/856092611 wait complete. 2024-02-19T15:32:19.538 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:19.539 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:20.310 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:32:20.540 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:20.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.680+0000 7f5e41800640 1 Processor -- start 2024-02-19T15:32:20.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.680+0000 7f5e41800640 1 -- start start 2024-02-19T15:32:20.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.680+0000 7f5e41800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e3c0cedb0 0x7f5e3c0cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:20.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.680+0000 7f5e41800640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3c0cc700 con 0x7f5e3c15bcd0 2024-02-19T15:32:20.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.680+0000 7f5e41800640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3c0cf300 con 0x7f5e3c0d2710 2024-02-19T15:32:20.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.680+0000 7f5e41800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c0cf5e0 con 0x7f5e3c0cedb0 2024-02-19T15:32:20.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e3aa00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5e3c15bcd0 0x7f5e3c0cd480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51748/0 (socket says 172.21.15.47:51748) 2024-02-19T15:32:20.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e3aa00640 1 -- 172.21.15.47:0/1290589188 learned_addr learned my addr 172.21.15.47:0/1290589188 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:32:20.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e33e00640 1 --2- 172.21.15.47:0/1290589188 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e3c0cedb0 0x7f5e3c0cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:20.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e3a000640 1 -- 172.21.15.47:0/1290589188 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2548191717 0 0) 0x7f5e3c0cc700 con 0x7f5e3c15bcd0 2024-02-19T15:32:20.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e3a000640 1 -- 172.21.15.47:0/1290589188 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e20003530 con 0x7f5e3c15bcd0 2024-02-19T15:32:20.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e3a000640 1 -- 172.21.15.47:0/1290589188 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2366319290 0 0) 0x7f5e20003530 con 0x7f5e3c15bcd0 2024-02-19T15:32:20.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e3a000640 1 -- 172.21.15.47:0/1290589188 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e3c0cedb0 msgr2=0x7f5e3c0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:20.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e3a000640 1 --2- 172.21.15.47:0/1290589188 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e3c0cedb0 0x7f5e3c0cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:20.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e3a000640 1 -- 172.21.15.47:0/1290589188 >> v1:172.21.15.116:6789/0 conn(0x7f5e3c0d2710 legacy=0x7f5e3c15d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:20.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e3a000640 1 -- 172.21.15.47:0/1290589188 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c0cc9a0 con 0x7f5e3c15bcd0 2024-02-19T15:32:20.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e3a000640 1 -- 172.21.15.47:0/1290589188 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f5e2c001680 con 0x7f5e3c15bcd0 2024-02-19T15:32:20.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e3a000640 1 -- 172.21.15.47:0/1290589188 >> v1:172.21.15.47:6789/0 conn(0x7f5e3c15bcd0 legacy=0x7f5e3c0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:20.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e3a000640 1 --2- 172.21.15.47:0/1290589188 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e3c0cedb0 0x7f5e200039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:20.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e3a000640 1 --2- 172.21.15.47:0/1290589188 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e20003ef0 0x7f5e200047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:20.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e3a000640 1 --2- 172.21.15.47:0/1290589188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e20004d10 0x7f5e20005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:20.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e3a000640 1 -- 172.21.15.47:0/1290589188 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c0cf5e0 con 0x7f5e20003ef0 2024-02-19T15:32:20.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e3a000640 1 -- 172.21.15.47:0/1290589188 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2000b9f0 con 0x7f5e20004d10 2024-02-19T15:32:20.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e3a000640 1 -- 172.21.15.47:0/1290589188 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2000bbb0 con 0x7f5e3c0cedb0 2024-02-19T15:32:20.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e33e00640 1 --2- 172.21.15.47:0/1290589188 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e20003ef0 0x7f5e200047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:20.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e3aa00640 1 --2- 172.21.15.47:0/1290589188 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e3c0cedb0 0x7f5e200039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:20.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e3b400640 1 --2- 172.21.15.47:0/1290589188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e20004d10 0x7f5e20005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:20.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e33e00640 1 -- 172.21.15.47:0/1290589188 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e3c0cedb0 msgr2=0x7f5e200039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:20.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e33e00640 1 --2- 172.21.15.47:0/1290589188 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e3c0cedb0 0x7f5e200039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:20.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e33e00640 1 -- 172.21.15.47:0/1290589188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e20004d10 msgr2=0x7f5e20005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:20.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e33e00640 1 --2- 172.21.15.47:0/1290589188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e20004d10 0x7f5e20005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:20.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.684+0000 7f5e33e00640 1 -- 172.21.15.47:0/1290589188 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e20003530 con 0x7f5e20003ef0 2024-02-19T15:32:20.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.688+0000 7f5e33e00640 1 --2- 172.21.15.47:0/1290589188 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e20003ef0 0x7f5e200047d0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009ae0 tx=0x7f5e2401c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=314daa96febf4a85 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:20.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.688+0000 7f5e3a000640 1 -- 172.21.15.47:0/1290589188 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e240a55a0 con 0x7f5e20003ef0 2024-02-19T15:32:20.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.688+0000 7f5e3a000640 1 -- 172.21.15.47:0/1290589188 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e240a5700 con 0x7f5e20003ef0 2024-02-19T15:32:20.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.688+0000 7f5e3a000640 1 -- 172.21.15.47:0/1290589188 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e240a5ab0 con 0x7f5e20003ef0 2024-02-19T15:32:20.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.688+0000 7f5e41800640 1 -- 172.21.15.47:0/1290589188 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e20003ef0 msgr2=0x7f5e200047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:20.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.688+0000 7f5e41800640 1 --2- 172.21.15.47:0/1290589188 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e20003ef0 0x7f5e200047d0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009ae0 tx=0x7f5e2401c9b0 comp rx=0 tx=0).stop 2024-02-19T15:32:20.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.688+0000 7f5e3b400640 1 -- 172.21.15.47:0/1290589188 reap_dead start 2024-02-19T15:32:20.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.688+0000 7f5e41800640 1 -- 172.21.15.47:0/1290589188 shutdown_connections 2024-02-19T15:32:20.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.688+0000 7f5e41800640 1 -- 172.21.15.47:0/1290589188 >> 172.21.15.47:0/1290589188 conn(0x7f5e3c0d1bf0 msgr2=0x7f5e3c0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:20.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.688+0000 7f5e41800640 1 -- 172.21.15.47:0/1290589188 shutdown_connections 2024-02-19T15:32:20.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.688+0000 7f5e41800640 1 -- 172.21.15.47:0/1290589188 wait complete. 2024-02-19T15:32:20.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e41800640 1 Processor -- start 2024-02-19T15:32:20.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e41800640 1 -- start start 2024-02-19T15:32:20.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e41800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e20003ef0 0x7f5e3c14ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:20.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e41800640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e3c0c5280 0x7f5e3c0b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:20.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e41800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e3c0c5700 0x7f5e3c0b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:20.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e41800640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c0cdb90 con 0x7f5e3c0c5700 2024-02-19T15:32:20.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e41800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c1f2160 con 0x7f5e20003ef0 2024-02-19T15:32:20.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e41800640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c1f1e80 con 0x7f5e3c0c5280 2024-02-19T15:32:20.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e3b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e3c0c5700 0x7f5e3c0b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:20.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e3b400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e3c0c5700 0x7f5e3c0b07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49824/0 (socket says 172.21.15.47:49824) 2024-02-19T15:32:20.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e3b400640 1 -- 172.21.15.47:0/3669954353 learned_addr learned my addr 172.21.15.47:0/3669954353 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:20.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e3aa00640 1 --2- 172.21.15.47:0/3669954353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e20003ef0 0x7f5e3c14ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:20.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e33e00640 1 --2- 172.21.15.47:0/3669954353 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e3c0c5280 0x7f5e3c0b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:20.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e3b400640 1 -- 172.21.15.47:0/3669954353 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e3c0c5280 msgr2=0x7f5e3c0b0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:20.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e3b400640 1 --2- 172.21.15.47:0/3669954353 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e3c0c5280 0x7f5e3c0b0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:20.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e3b400640 1 -- 172.21.15.47:0/3669954353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e20003ef0 msgr2=0x7f5e3c14ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:20.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e3b400640 1 --2- 172.21.15.47:0/3669954353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e20003ef0 0x7f5e3c14ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:20.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e3b400640 1 -- 172.21.15.47:0/3669954353 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c0ae9e0 con 0x7f5e3c0c5700 2024-02-19T15:32:20.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e33e00640 1 --2- 172.21.15.47:0/3669954353 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e3c0c5280 0x7f5e3c0b0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:32:20.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e3b400640 1 --2- 172.21.15.47:0/3669954353 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e3c0c5700 0x7f5e3c0b07c0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c0cf210 tx=0x7f5e34011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:20.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e32a00640 1 -- 172.21.15.47:0/3669954353 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e34006e30 con 0x7f5e3c0c5700 2024-02-19T15:32:20.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e32a00640 1 -- 172.21.15.47:0/3669954353 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e34003980 con 0x7f5e3c0c5700 2024-02-19T15:32:20.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e41800640 1 -- 172.21.15.47:0/3669954353 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3c0aecc0 con 0x7f5e3c0c5700 2024-02-19T15:32:20.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e32a00640 1 -- 172.21.15.47:0/3669954353 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e34003d30 con 0x7f5e3c0c5700 2024-02-19T15:32:20.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.692+0000 7f5e41800640 1 -- 172.21.15.47:0/3669954353 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3c150510 con 0x7f5e3c0c5700 2024-02-19T15:32:20.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.696+0000 7f5e41800640 1 -- 172.21.15.47:0/3669954353 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3c075360 con 0x7f5e3c0c5700 2024-02-19T15:32:20.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.696+0000 7f5e32a00640 1 -- 172.21.15.47:0/3669954353 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f5e34005110 con 0x7f5e3c0c5700 2024-02-19T15:32:20.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.696+0000 7f5e32a00640 1 --2- 172.21.15.47:0/3669954353 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5e18066e40 0x7f5e18069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:20.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.696+0000 7f5e32a00640 1 -- 172.21.15.47:0/3669954353 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5e34087460 con 0x7f5e3c0c5700 2024-02-19T15:32:20.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.696+0000 7f5e3aa00640 1 --2- 172.21.15.47:0/3669954353 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5e18066e40 0x7f5e18069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:20.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.700+0000 7f5e3aa00640 1 --2- 172.21.15.47:0/3669954353 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5e18066e40 0x7f5e18069300 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c001cc0 tx=0x7f5e2c001c30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:20.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.704+0000 7f5e32a00640 1 -- 172.21.15.47:0/3669954353 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f5e34054200 con 0x7f5e3c0c5700 2024-02-19T15:32:20.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.900+0000 7f5e41800640 1 -- 172.21.15.47:0/3669954353 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e3c0af170 con 0x7f5e3c0c5700 2024-02-19T15:32:20.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.900+0000 7f5e32a00640 1 -- 172.21.15.47:0/3669954353 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f5e3401bd80 con 0x7f5e3c0c5700 2024-02-19T15:32:20.899 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:20.899 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:20.899 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:20.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.900+0000 7f5e41800640 1 -- 172.21.15.47:0/3669954353 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5e18066e40 msgr2=0x7f5e18069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:20.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.900+0000 7f5e41800640 1 --2- 172.21.15.47:0/3669954353 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5e18066e40 0x7f5e18069300 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c001cc0 tx=0x7f5e2c001c30 comp rx=0 tx=0).stop 2024-02-19T15:32:20.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.900+0000 7f5e41800640 1 -- 172.21.15.47:0/3669954353 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e3c0c5700 msgr2=0x7f5e3c0b07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:20.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.900+0000 7f5e41800640 1 --2- 172.21.15.47:0/3669954353 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e3c0c5700 0x7f5e3c0b07c0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c0cf210 tx=0x7f5e34011f30 comp rx=0 tx=0).stop 2024-02-19T15:32:20.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.900+0000 7f5e41800640 1 -- 172.21.15.47:0/3669954353 shutdown_connections 2024-02-19T15:32:20.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.900+0000 7f5e41800640 1 --2- 172.21.15.47:0/3669954353 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f5e18066e40 0x7f5e18069300 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:20.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.900+0000 7f5e41800640 1 --2- 172.21.15.47:0/3669954353 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e3c0c5700 0x7f5e3c0b07c0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:20.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.900+0000 7f5e41800640 1 --2- 172.21.15.47:0/3669954353 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e3c0c5280 0x7f5e3c0b0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:20.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.900+0000 7f5e41800640 1 --2- 172.21.15.47:0/3669954353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e20003ef0 0x7f5e3c14ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:20.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.900+0000 7f5e41800640 1 -- 172.21.15.47:0/3669954353 >> 172.21.15.47:0/3669954353 conn(0x7f5e3c0d1bf0 msgr2=0x7f5e3c0b48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:20.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.900+0000 7f5e41800640 1 -- 172.21.15.47:0/3669954353 shutdown_connections 2024-02-19T15:32:20.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:20.900+0000 7f5e41800640 1 -- 172.21.15.47:0/3669954353 wait complete. 2024-02-19T15:32:21.121 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:21.121 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:22.123 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:22.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.264+0000 7f85a0e00640 1 Processor -- start 2024-02-19T15:32:22.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.264+0000 7f85a0e00640 1 -- start start 2024-02-19T15:32:22.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.264+0000 7f85a0e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f859c0cddc0 0x7f859c06c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:22.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.264+0000 7f85a0e00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f859c0cfe00 0x7f859c0ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:22.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.264+0000 7f85a0e00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f859c06d200 con 0x7f859c0ce6e0 2024-02-19T15:32:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.264+0000 7f85a0e00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f859c06c780 con 0x7f859c0cddc0 2024-02-19T15:32:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.264+0000 7f85a0e00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f859c0d1180 con 0x7f859c0cfe00 2024-02-19T15:32:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.264+0000 7f8599e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f859c0cddc0 0x7f859c06c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.264+0000 7f8599e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f859c0cddc0 0x7f859c06c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49832/0 (socket says 172.21.15.47:49832) 2024-02-19T15:32:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.264+0000 7f8599e00640 1 -- 172.21.15.47:0/2418657972 learned_addr learned my addr 172.21.15.47:0/2418657972 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.264+0000 7f859a800640 1 --1- 172.21.15.47:0/2418657972 >> v1:172.21.15.47:6789/0 conn(0x7f859c0ce6e0 0x7f859c15d7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51752/0 (socket says 172.21.15.47:51752) 2024-02-19T15:32:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.264+0000 7f8593e00640 1 --2- 172.21.15.47:0/2418657972 >> v2:172.21.15.116:3300/0 conn(0x7f859c0cfe00 0x7f859c0ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.264+0000 7f8599e00640 1 -- 172.21.15.47:0/2418657972 >> v2:172.21.15.116:3300/0 conn(0x7f859c0cfe00 msgr2=0x7f859c0ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:22.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.264+0000 7f8599400640 1 -- 172.21.15.47:0/2418657972 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566103907 0 0) 0x7f85940014a0 con 0x7f859c0ce6e0 2024-02-19T15:32:22.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.264+0000 7f8599e00640 1 --2- 172.21.15.47:0/2418657972 >> v2:172.21.15.116:3300/0 conn(0x7f859c0cfe00 0x7f859c0ce1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:22.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.264+0000 7f8599e00640 1 -- 172.21.15.47:0/2418657972 >> v1:172.21.15.47:6789/0 conn(0x7f859c0ce6e0 legacy=0x7f859c15d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:22.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.264+0000 7f8599e00640 1 -- 172.21.15.47:0/2418657972 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f859c0d0600 con 0x7f859c0cddc0 2024-02-19T15:32:22.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.268+0000 7f8599e00640 1 --2- 172.21.15.47:0/2418657972 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f859c0cddc0 0x7f859c06c240 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f859c071d90 tx=0x7f858c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e65f538362a212 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:22.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.268+0000 7f8599400640 1 -- 172.21.15.47:0/2418657972 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858c0ac630 con 0x7f859c0cddc0 2024-02-19T15:32:22.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.268+0000 7f8599400640 1 -- 172.21.15.47:0/2418657972 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f858c0ac790 con 0x7f859c0cddc0 2024-02-19T15:32:22.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.268+0000 7f8599400640 1 -- 172.21.15.47:0/2418657972 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858c0acb20 con 0x7f859c0cddc0 2024-02-19T15:32:22.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.268+0000 7f85a0e00640 1 -- 172.21.15.47:0/2418657972 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f859c0cddc0 msgr2=0x7f859c06c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:22.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.268+0000 7f85a0e00640 1 --2- 172.21.15.47:0/2418657972 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f859c0cddc0 0x7f859c06c240 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f859c071d90 tx=0x7f858c01c4f0 comp rx=0 tx=0).stop 2024-02-19T15:32:22.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.268+0000 7f85a0e00640 1 -- 172.21.15.47:0/2418657972 shutdown_connections 2024-02-19T15:32:22.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.268+0000 7f85a0e00640 1 --2- 172.21.15.47:0/2418657972 >> v2:172.21.15.116:3300/0 conn(0x7f859c0cfe00 0x7f859c0ce1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:22.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.268+0000 7f85a0e00640 1 --2- 172.21.15.47:0/2418657972 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f859c0cddc0 0x7f859c06c240 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:22.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.268+0000 7f85a0e00640 1 -- 172.21.15.47:0/2418657972 >> 172.21.15.47:0/2418657972 conn(0x7f859c0d1b30 msgr2=0x7f859c0d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:22.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.268+0000 7f85a0e00640 1 -- 172.21.15.47:0/2418657972 shutdown_connections 2024-02-19T15:32:22.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.268+0000 7f85a0e00640 1 -- 172.21.15.47:0/2418657972 wait complete. 2024-02-19T15:32:22.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f85a0e00640 1 Processor -- start 2024-02-19T15:32:22.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f85a0e00640 1 -- start start 2024-02-19T15:32:22.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f85a0e00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f859c0cddc0 0x7f859c155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:22.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f85a0e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f859c0ce6e0 0x7f859c156da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:22.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f85a0e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f859c0cfe00 0x7f859c14dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:22.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f85a0e00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859c155990 con 0x7f859c0cfe00 2024-02-19T15:32:22.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f85a0e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859c155360 con 0x7f859c0ce6e0 2024-02-19T15:32:22.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f85a0e00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f859c0b7380 con 0x7f859c0cddc0 2024-02-19T15:32:22.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f859a800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f859c0cfe00 0x7f859c14dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:22.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f859a800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f859c0cfe00 0x7f859c14dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49838/0 (socket says 172.21.15.47:49838) 2024-02-19T15:32:22.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f859a800640 1 -- 172.21.15.47:0/1458170862 learned_addr learned my addr 172.21.15.47:0/1458170862 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:22.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f8599e00640 1 --2- 172.21.15.47:0/1458170862 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f859c0cddc0 0x7f859c155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:22.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f8593e00640 1 --2- 172.21.15.47:0/1458170862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f859c0ce6e0 0x7f859c156da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:22.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f859a800640 1 -- 172.21.15.47:0/1458170862 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f859c0cddc0 msgr2=0x7f859c155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:22.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f859a800640 1 --2- 172.21.15.47:0/1458170862 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f859c0cddc0 0x7f859c155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:22.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f859a800640 1 -- 172.21.15.47:0/1458170862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f859c0ce6e0 msgr2=0x7f859c156da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:22.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f859a800640 1 --2- 172.21.15.47:0/1458170862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f859c0ce6e0 0x7f859c156da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:22.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f859a800640 1 -- 172.21.15.47:0/1458170862 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858c0094d0 con 0x7f859c0cfe00 2024-02-19T15:32:22.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f859a800640 1 --2- 172.21.15.47:0/1458170862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f859c0cfe00 0x7f859c14dc90 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f859400c7d0 tx=0x7f859400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:22.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f8592000640 1 -- 172.21.15.47:0/1458170862 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859400a640 con 0x7f859c0cfe00 2024-02-19T15:32:22.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f8592000640 1 -- 172.21.15.47:0/1458170862 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f859400a7a0 con 0x7f859c0cfe00 2024-02-19T15:32:22.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f8592000640 1 -- 172.21.15.47:0/1458170862 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859400ab50 con 0x7f859c0cfe00 2024-02-19T15:32:22.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f85a0e00640 1 -- 172.21.15.47:0/1458170862 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f859c0b7660 con 0x7f859c0cfe00 2024-02-19T15:32:22.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.272+0000 7f85a0e00640 1 -- 172.21.15.47:0/1458170862 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f859c1562f0 con 0x7f859c0cfe00 2024-02-19T15:32:22.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.276+0000 7f85a0e00640 1 -- 172.21.15.47:0/1458170862 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f859c155ad0 con 0x7f859c0cfe00 2024-02-19T15:32:22.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.276+0000 7f8592000640 1 -- 172.21.15.47:0/1458170862 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f859400acb0 con 0x7f859c0cfe00 2024-02-19T15:32:22.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.276+0000 7f8592000640 1 --2- 172.21.15.47:0/1458170862 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8578066e40 0x7f8578069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:22.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.276+0000 7f8592000640 1 -- 172.21.15.47:0/1458170862 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8594083ce0 con 0x7f859c0cfe00 2024-02-19T15:32:22.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.276+0000 7f8599e00640 1 --2- 172.21.15.47:0/1458170862 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8578066e40 0x7f8578069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:22.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.276+0000 7f8599e00640 1 --2- 172.21.15.47:0/1458170862 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8578066e40 0x7f8578069300 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f858c0ae000 tx=0x7f858c0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:22.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.280+0000 7f8592000640 1 -- 172.21.15.47:0/1458170862 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f859c155ad0 con 0x7f859c0cfe00 2024-02-19T15:32:22.309 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:32:22.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.476+0000 7f85a0e00640 1 -- 172.21.15.47:0/1458170862 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f859c06d200 con 0x7f859c0cfe00 2024-02-19T15:32:22.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.480+0000 7f8592000640 1 -- 172.21.15.47:0/1458170862 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f859c06d200 con 0x7f859c0cfe00 2024-02-19T15:32:22.478 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:22.478 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:22.478 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:22.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.480+0000 7f85a0e00640 1 -- 172.21.15.47:0/1458170862 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8578066e40 msgr2=0x7f8578069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:22.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.480+0000 7f85a0e00640 1 --2- 172.21.15.47:0/1458170862 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8578066e40 0x7f8578069300 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f858c0ae000 tx=0x7f858c0a7000 comp rx=0 tx=0).stop 2024-02-19T15:32:22.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.480+0000 7f85a0e00640 1 -- 172.21.15.47:0/1458170862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f859c0cfe00 msgr2=0x7f859c14dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:22.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.480+0000 7f85a0e00640 1 --2- 172.21.15.47:0/1458170862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f859c0cfe00 0x7f859c14dc90 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f859400c7d0 tx=0x7f859400cc90 comp rx=0 tx=0).stop 2024-02-19T15:32:22.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.480+0000 7f85a0e00640 1 -- 172.21.15.47:0/1458170862 shutdown_connections 2024-02-19T15:32:22.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.480+0000 7f85a0e00640 1 --2- 172.21.15.47:0/1458170862 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f8578066e40 0x7f8578069300 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:22.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.480+0000 7f85a0e00640 1 --2- 172.21.15.47:0/1458170862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f859c0cfe00 0x7f859c14dc90 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:22.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.480+0000 7f85a0e00640 1 --2- 172.21.15.47:0/1458170862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f859c0ce6e0 0x7f859c156da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:22.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.480+0000 7f85a0e00640 1 --2- 172.21.15.47:0/1458170862 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f859c0cddc0 0x7f859c155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:22.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.480+0000 7f85a0e00640 1 -- 172.21.15.47:0/1458170862 >> 172.21.15.47:0/1458170862 conn(0x7f859c0d1b30 msgr2=0x7f859c06c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:22.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.480+0000 7f85a0e00640 1 -- 172.21.15.47:0/1458170862 shutdown_connections 2024-02-19T15:32:22.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:22.480+0000 7f85a0e00640 1 -- 172.21.15.47:0/1458170862 wait complete. 2024-02-19T15:32:22.702 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:22.702 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:23.703 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:23.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcf02400640 1 Processor -- start 2024-02-19T15:32:23.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcf02400640 1 -- start start 2024-02-19T15:32:23.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcf02400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcefc1613a0 0x7fcefc0adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:23.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcf02400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcefc06fdf0 0x7fcefc06e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:23.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcf02400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcefc161780 con 0x7fcefc06e5f0 2024-02-19T15:32:23.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcf02400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcefc0b01a0 con 0x7fcefc06fdf0 2024-02-19T15:32:23.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcf02400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0ae3c0 con 0x7fcefc1613a0 2024-02-19T15:32:23.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcefbe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fcefc06e5f0 0x7fcefc161a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54834/0 (socket says 172.21.15.47:54834) 2024-02-19T15:32:23.845 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcefbe00640 1 -- 172.21.15.47:0/2501604101 learned_addr learned my addr 172.21.15.47:0/2501604101 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:32:23.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcefb400640 1 --2- 172.21.15.47:0/2501604101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcefc1613a0 0x7fcefc0adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:23.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcef3e00640 1 --2- 172.21.15.47:0/2501604101 >> v2:172.21.15.116:3300/0 conn(0x7fcefc06fdf0 0x7fcefc06e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:23.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcefaa00640 1 -- 172.21.15.47:0/2501604101 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279731521 0 0) 0x7fcefc161780 con 0x7fcefc06e5f0 2024-02-19T15:32:23.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcefaa00640 1 -- 172.21.15.47:0/2501604101 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee0003530 con 0x7fcefc06e5f0 2024-02-19T15:32:23.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcefaa00640 1 -- 172.21.15.47:0/2501604101 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1413313895 0 0) 0x7fcee0003530 con 0x7fcefc06e5f0 2024-02-19T15:32:23.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcefaa00640 1 -- 172.21.15.47:0/2501604101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcefc1613a0 msgr2=0x7fcefc0adbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:23.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcefaa00640 1 --2- 172.21.15.47:0/2501604101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcefc1613a0 0x7fcefc0adbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:23.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcefaa00640 1 -- 172.21.15.47:0/2501604101 >> v2:172.21.15.116:3300/0 conn(0x7fcefc06fdf0 msgr2=0x7fcefc06e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:23.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcefaa00640 1 --2- 172.21.15.47:0/2501604101 >> v2:172.21.15.116:3300/0 conn(0x7fcefc06fdf0 0x7fcefc06e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:23.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcefaa00640 1 -- 172.21.15.47:0/2501604101 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc0ae6a0 con 0x7fcefc06e5f0 2024-02-19T15:32:23.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcefaa00640 1 -- 172.21.15.47:0/2501604101 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7fcef4001a00 con 0x7fcefc06e5f0 2024-02-19T15:32:23.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcefaa00640 1 -- 172.21.15.47:0/2501604101 >> v1:172.21.15.47:6789/0 conn(0x7fcefc06e5f0 legacy=0x7fcefc161a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:23.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcefaa00640 1 --2- 172.21.15.47:0/2501604101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcefc1613a0 0x7fcee0003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:23.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcefaa00640 1 --2- 172.21.15.47:0/2501604101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcee0003f00 0x7fcee00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:23.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcefaa00640 1 --2- 172.21.15.47:0/2501604101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcee0004d10 0x7fcee0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:23.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcefaa00640 1 -- 172.21.15.47:0/2501604101 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0b01a0 con 0x7fcee0003f00 2024-02-19T15:32:23.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcefaa00640 1 -- 172.21.15.47:0/2501604101 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0ae3c0 con 0x7fcee0004d10 2024-02-19T15:32:23.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.844+0000 7fcefaa00640 1 -- 172.21.15.47:0/2501604101 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee000b9e0 con 0x7fcefc1613a0 2024-02-19T15:32:23.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.848+0000 7fcef3e00640 1 --2- 172.21.15.47:0/2501604101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcee0003f00 0x7fcee00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:23.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.848+0000 7fcefb400640 1 --2- 172.21.15.47:0/2501604101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcefc1613a0 0x7fcee0003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:23.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.848+0000 7fcefbe00640 1 --2- 172.21.15.47:0/2501604101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcee0004d10 0x7fcee0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:23.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.848+0000 7fcefb400640 1 -- 172.21.15.47:0/2501604101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcee0004d10 msgr2=0x7fcee0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:23.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.848+0000 7fcefb400640 1 --2- 172.21.15.47:0/2501604101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcee0004d10 0x7fcee0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:23.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.848+0000 7fcefb400640 1 -- 172.21.15.47:0/2501604101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcee0003f00 msgr2=0x7fcee00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:23.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.848+0000 7fcefb400640 1 --2- 172.21.15.47:0/2501604101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcee0003f00 0x7fcee00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:23.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.848+0000 7fcefb400640 1 -- 172.21.15.47:0/2501604101 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee0003530 con 0x7fcefc1613a0 2024-02-19T15:32:23.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.848+0000 7fcef3e00640 1 --2- 172.21.15.47:0/2501604101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcee0003f00 0x7fcee00047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:32:23.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.848+0000 7fcefb400640 1 --2- 172.21.15.47:0/2501604101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcefc1613a0 0x7fcee0003980 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fceec009b10 tx=0x7fceec01cab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8fde1efc4039843 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:23.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcefaa00640 1 -- 172.21.15.47:0/2501604101 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec0a5510 con 0x7fcefc1613a0 2024-02-19T15:32:23.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcefaa00640 1 -- 172.21.15.47:0/2501604101 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceec0a5670 con 0x7fcefc1613a0 2024-02-19T15:32:23.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcefaa00640 1 -- 172.21.15.47:0/2501604101 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec0a5a20 con 0x7fcefc1613a0 2024-02-19T15:32:23.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcf02400640 1 -- 172.21.15.47:0/2501604101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcefc1613a0 msgr2=0x7fcee0003980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:23.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcf02400640 1 --2- 172.21.15.47:0/2501604101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcefc1613a0 0x7fcee0003980 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fceec009b10 tx=0x7fceec01cab0 comp rx=0 tx=0).stop 2024-02-19T15:32:23.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcefbe00640 1 -- 172.21.15.47:0/2501604101 reap_dead start 2024-02-19T15:32:23.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcf02400640 1 -- 172.21.15.47:0/2501604101 shutdown_connections 2024-02-19T15:32:23.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcf02400640 1 -- 172.21.15.47:0/2501604101 >> 172.21.15.47:0/2501604101 conn(0x7fcefc0d0ad0 msgr2=0x7fcefc0d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:23.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcf02400640 1 -- 172.21.15.47:0/2501604101 shutdown_connections 2024-02-19T15:32:23.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcf02400640 1 -- 172.21.15.47:0/2501604101 wait complete. 2024-02-19T15:32:23.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcf02400640 1 Processor -- start 2024-02-19T15:32:23.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcf02400640 1 -- start start 2024-02-19T15:32:23.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcf02400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcefc1613a0 0x7fcefc147770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:23.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcf02400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcefc15d890 0x7fcefc0ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:23.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcf02400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcefc15e200 0x7fcefc15e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:23.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcf02400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0ce330 con 0x7fcefc15d890 2024-02-19T15:32:23.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcf02400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc15ebc0 con 0x7fcefc15e200 2024-02-19T15:32:23.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcf02400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0cd6b0 con 0x7fcefc1613a0 2024-02-19T15:32:23.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcef3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcefc15d890 0x7fcefc0ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:23.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcef3e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcefc15d890 0x7fcefc0ccd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56414/0 (socket says 172.21.15.47:56414) 2024-02-19T15:32:23.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.852+0000 7fcef3e00640 1 -- 172.21.15.47:0/1397660299 learned_addr learned my addr 172.21.15.47:0/1397660299 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:23.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.856+0000 7fcefb400640 1 --2- 172.21.15.47:0/1397660299 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcefc1613a0 0x7fcefc147770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:23.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.856+0000 7fcefbe00640 1 --2- 172.21.15.47:0/1397660299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcefc15e200 0x7fcefc15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:23.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.856+0000 7fcef3e00640 1 -- 172.21.15.47:0/1397660299 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcefc1613a0 msgr2=0x7fcefc147770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:23.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.856+0000 7fcef3e00640 1 --2- 172.21.15.47:0/1397660299 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcefc1613a0 0x7fcefc147770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:23.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.856+0000 7fcef3e00640 1 -- 172.21.15.47:0/1397660299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcefc15e200 msgr2=0x7fcefc15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:23.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.856+0000 7fcef3e00640 1 --2- 172.21.15.47:0/1397660299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcefc15e200 0x7fcefc15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:23.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.856+0000 7fcef3e00640 1 -- 172.21.15.47:0/1397660299 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc0cd990 con 0x7fcefc15d890 2024-02-19T15:32:23.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.856+0000 7fcef3e00640 1 --2- 172.21.15.47:0/1397660299 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcefc15d890 0x7fcefc0ccd40 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fcee4011cd0 tx=0x7fcee400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:23.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.856+0000 7fcef3400640 1 -- 172.21.15.47:0/1397660299 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee4011d00 con 0x7fcefc15d890 2024-02-19T15:32:23.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.856+0000 7fcef3400640 1 -- 172.21.15.47:0/1397660299 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee4011e60 con 0x7fcefc15d890 2024-02-19T15:32:23.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.856+0000 7fcef3400640 1 -- 172.21.15.47:0/1397660299 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee4004240 con 0x7fcefc15d890 2024-02-19T15:32:23.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.856+0000 7fcf02400640 1 -- 172.21.15.47:0/1397660299 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcefc0cdc70 con 0x7fcefc15d890 2024-02-19T15:32:23.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.856+0000 7fcf02400640 1 -- 172.21.15.47:0/1397660299 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcefc147cb0 con 0x7fcefc15d890 2024-02-19T15:32:23.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.856+0000 7fcf02400640 1 -- 172.21.15.47:0/1397660299 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcefc075360 con 0x7fcefc15d890 2024-02-19T15:32:23.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.856+0000 7fcef3400640 1 -- 172.21.15.47:0/1397660299 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fcee40043a0 con 0x7fcefc15d890 2024-02-19T15:32:23.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.860+0000 7fcef3400640 1 --2- 172.21.15.47:0/1397660299 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fced8066e40 0x7fced8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:23.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.860+0000 7fcef3400640 1 -- 172.21.15.47:0/1397660299 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcee4085060 con 0x7fcefc15d890 2024-02-19T15:32:23.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.860+0000 7fcefb400640 1 --2- 172.21.15.47:0/1397660299 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fced8066e40 0x7fced8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:23.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.860+0000 7fcefb400640 1 --2- 172.21.15.47:0/1397660299 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fced8066e40 0x7fced8069300 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fceec0af000 tx=0x7fceec0046d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:23.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:23.868+0000 7fcef3400640 1 -- 172.21.15.47:0/1397660299 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fcee40b71c0 con 0x7fcefc15d890 2024-02-19T15:32:24.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:24.076+0000 7fcf02400640 1 -- 172.21.15.47:0/1397660299 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcefc0d15f0 con 0x7fcefc15d890 2024-02-19T15:32:24.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:24.076+0000 7fcef3400640 1 -- 172.21.15.47:0/1397660299 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7fcee4022020 con 0x7fcefc15d890 2024-02-19T15:32:24.074 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:24.075 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:24.075 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:24.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:24.076+0000 7fcf02400640 1 -- 172.21.15.47:0/1397660299 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fced8066e40 msgr2=0x7fced8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:24.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:24.076+0000 7fcf02400640 1 --2- 172.21.15.47:0/1397660299 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fced8066e40 0x7fced8069300 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fceec0af000 tx=0x7fceec0046d0 comp rx=0 tx=0).stop 2024-02-19T15:32:24.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:24.076+0000 7fcf02400640 1 -- 172.21.15.47:0/1397660299 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcefc15d890 msgr2=0x7fcefc0ccd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:24.076+0000 7fcf02400640 1 --2- 172.21.15.47:0/1397660299 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcefc15d890 0x7fcefc0ccd40 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fcee4011cd0 tx=0x7fcee400f610 comp rx=0 tx=0).stop 2024-02-19T15:32:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:24.076+0000 7fcf02400640 1 -- 172.21.15.47:0/1397660299 shutdown_connections 2024-02-19T15:32:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:24.076+0000 7fcf02400640 1 --2- 172.21.15.47:0/1397660299 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fced8066e40 0x7fced8069300 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:24.076+0000 7fcf02400640 1 --2- 172.21.15.47:0/1397660299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcefc15e200 0x7fcefc15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:24.076+0000 7fcf02400640 1 --2- 172.21.15.47:0/1397660299 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcefc15d890 0x7fcefc0ccd40 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:24.076+0000 7fcf02400640 1 --2- 172.21.15.47:0/1397660299 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcefc1613a0 0x7fcefc147770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:24.076+0000 7fcf02400640 1 -- 172.21.15.47:0/1397660299 >> 172.21.15.47:0/1397660299 conn(0x7fcefc0d0ad0 msgr2=0x7fcefc06f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:24.076+0000 7fcf02400640 1 -- 172.21.15.47:0/1397660299 shutdown_connections 2024-02-19T15:32:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:24.076+0000 7fcf02400640 1 -- 172.21.15.47:0/1397660299 wait complete. 2024-02-19T15:32:24.297 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:24.297 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:24.310 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:32:25.298 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:25.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.440+0000 7efc80200640 1 Processor -- start 2024-02-19T15:32:25.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.440+0000 7efc80200640 1 -- start start 2024-02-19T15:32:25.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.440+0000 7efc80200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efc7806d650 0x7efc780d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:25.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.440+0000 7efc80200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efc780ced10 0x7efc7806bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:25.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.440+0000 7efc80200640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc780cce30 con 0x7efc780cded0 2024-02-19T15:32:25.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.444+0000 7efc80200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc780cd110 con 0x7efc7806d650 2024-02-19T15:32:25.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.444+0000 7efc80200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc780cd3f0 con 0x7efc780ced10 2024-02-19T15:32:25.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.444+0000 7efc7d800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efc7806d650 0x7efc780d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:25.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.444+0000 7efc7d800640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efc7806d650 0x7efc780d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56418/0 (socket says 172.21.15.47:56418) 2024-02-19T15:32:25.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.444+0000 7efc7d800640 1 -- 172.21.15.47:0/3853993191 learned_addr learned my addr 172.21.15.47:0/3853993191 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:25.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.444+0000 7efc7e200640 1 --2- 172.21.15.47:0/3853993191 >> v2:172.21.15.116:3300/0 conn(0x7efc780ced10 0x7efc7806bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:25.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.444+0000 7efc7d800640 1 -- 172.21.15.47:0/3853993191 >> v2:172.21.15.116:3300/0 conn(0x7efc780ced10 msgr2=0x7efc7806bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:25.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.444+0000 7efc7d800640 1 --2- 172.21.15.47:0/3853993191 >> v2:172.21.15.116:3300/0 conn(0x7efc780ced10 0x7efc7806bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:25.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.444+0000 7efc7d800640 1 -- 172.21.15.47:0/3853993191 >> v1:172.21.15.116:6789/0 conn(0x7efc780cded0 legacy=0x7efc780ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:25.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.444+0000 7efc7d800640 1 -- 172.21.15.47:0/3853993191 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc780cd6d0 con 0x7efc7806d650 2024-02-19T15:32:25.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.444+0000 7efc7d800640 1 --2- 172.21.15.47:0/3853993191 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc7806d650 0x7efc780d08b0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7efc6c009870 tx=0x7efc6c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5480addc462b7484 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:25.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.444+0000 7efc77e00640 1 -- 172.21.15.47:0/3853993191 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c0ab680 con 0x7efc7806d650 2024-02-19T15:32:25.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.444+0000 7efc77e00640 1 -- 172.21.15.47:0/3853993191 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6c0ab7e0 con 0x7efc7806d650 2024-02-19T15:32:25.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.444+0000 7efc77e00640 1 -- 172.21.15.47:0/3853993191 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c0abb70 con 0x7efc7806d650 2024-02-19T15:32:25.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.448+0000 7efc80200640 1 -- 172.21.15.47:0/3853993191 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc7806d650 msgr2=0x7efc780d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:25.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.448+0000 7efc80200640 1 --2- 172.21.15.47:0/3853993191 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc7806d650 0x7efc780d08b0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7efc6c009870 tx=0x7efc6c01c460 comp rx=0 tx=0).stop 2024-02-19T15:32:25.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.448+0000 7efc80200640 1 -- 172.21.15.47:0/3853993191 shutdown_connections 2024-02-19T15:32:25.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.448+0000 7efc80200640 1 --2- 172.21.15.47:0/3853993191 >> v2:172.21.15.116:3300/0 conn(0x7efc780ced10 0x7efc7806bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:25.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.448+0000 7efc80200640 1 --2- 172.21.15.47:0/3853993191 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc7806d650 0x7efc780d08b0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:25.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.448+0000 7efc80200640 1 -- 172.21.15.47:0/3853993191 >> 172.21.15.47:0/3853993191 conn(0x7efc780d1bf0 msgr2=0x7efc780d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:25.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.448+0000 7efc80200640 1 -- 172.21.15.47:0/3853993191 shutdown_connections 2024-02-19T15:32:25.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.448+0000 7efc80200640 1 -- 172.21.15.47:0/3853993191 wait complete. 2024-02-19T15:32:25.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.448+0000 7efc80200640 1 Processor -- start 2024-02-19T15:32:25.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.448+0000 7efc80200640 1 -- start start 2024-02-19T15:32:25.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc80200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc7806d650 0x7efc780c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:25.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc80200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc780cded0 0x7efc780c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:25.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc80200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc780ced10 0x7efc78146c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:25.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc80200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc780cbaf0 con 0x7efc7806d650 2024-02-19T15:32:25.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc80200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc780c9200 con 0x7efc780ced10 2024-02-19T15:32:25.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc80200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc780cb190 con 0x7efc780cded0 2024-02-19T15:32:25.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc7d800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc7806d650 0x7efc780c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:25.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc7d800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc7806d650 0x7efc780c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56422/0 (socket says 172.21.15.47:56422) 2024-02-19T15:32:25.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc7d800640 1 -- 172.21.15.47:0/1574822951 learned_addr learned my addr 172.21.15.47:0/1574822951 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:25.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc7e200640 1 --2- 172.21.15.47:0/1574822951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc780ced10 0x7efc78146c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:25.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc7ce00640 1 --2- 172.21.15.47:0/1574822951 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc780cded0 0x7efc780c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:25.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc7d800640 1 -- 172.21.15.47:0/1574822951 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc780cded0 msgr2=0x7efc780c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:25.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc7d800640 1 --2- 172.21.15.47:0/1574822951 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc780cded0 0x7efc780c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:25.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc7d800640 1 -- 172.21.15.47:0/1574822951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc780ced10 msgr2=0x7efc78146c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:25.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc7d800640 1 --2- 172.21.15.47:0/1574822951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc780ced10 0x7efc78146c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:25.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc7d800640 1 -- 172.21.15.47:0/1574822951 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6c0094d0 con 0x7efc7806d650 2024-02-19T15:32:25.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc7d800640 1 --2- 172.21.15.47:0/1574822951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc7806d650 0x7efc780c8cc0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7efc6c0ad040 tx=0x7efc6c01aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:25.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc75600640 1 -- 172.21.15.47:0/1574822951 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c0ab680 con 0x7efc7806d650 2024-02-19T15:32:25.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc75600640 1 -- 172.21.15.47:0/1574822951 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6c0b4820 con 0x7efc7806d650 2024-02-19T15:32:25.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc75600640 1 -- 172.21.15.47:0/1574822951 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c0b4bd0 con 0x7efc7806d650 2024-02-19T15:32:25.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc80200640 1 -- 172.21.15.47:0/1574822951 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc780cb430 con 0x7efc7806d650 2024-02-19T15:32:25.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc80200640 1 -- 172.21.15.47:0/1574822951 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7815e8a0 con 0x7efc7806d650 2024-02-19T15:32:25.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.452+0000 7efc80200640 1 -- 172.21.15.47:0/1574822951 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc78075360 con 0x7efc7806d650 2024-02-19T15:32:25.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.456+0000 7efc75600640 1 -- 172.21.15.47:0/1574822951 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7efc6c0b4d30 con 0x7efc7806d650 2024-02-19T15:32:25.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.456+0000 7efc75600640 1 --2- 172.21.15.47:0/1574822951 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc58066e40 0x7efc58069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:25.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.456+0000 7efc75600640 1 -- 172.21.15.47:0/1574822951 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efc6c11f6d0 con 0x7efc7806d650 2024-02-19T15:32:25.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.456+0000 7efc7ce00640 1 --2- 172.21.15.47:0/1574822951 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc58066e40 0x7efc58069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:25.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.456+0000 7efc7ce00640 1 --2- 172.21.15.47:0/1574822951 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc58066e40 0x7efc58069300 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7efc780c8670 tx=0x7efc64009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:25.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.460+0000 7efc75600640 1 -- 172.21.15.47:0/1574822951 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7efc6c0ec490 con 0x7efc7806d650 2024-02-19T15:32:25.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.656+0000 7efc80200640 1 -- 172.21.15.47:0/1574822951 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc780cf580 con 0x7efc7806d650 2024-02-19T15:32:25.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.656+0000 7efc75600640 1 -- 172.21.15.47:0/1574822951 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7efc6c0efb50 con 0x7efc7806d650 2024-02-19T15:32:25.656 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:25.656 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:25.657 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:25.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.660+0000 7efc80200640 1 -- 172.21.15.47:0/1574822951 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc58066e40 msgr2=0x7efc58069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:25.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.660+0000 7efc80200640 1 --2- 172.21.15.47:0/1574822951 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc58066e40 0x7efc58069300 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7efc780c8670 tx=0x7efc64009040 comp rx=0 tx=0).stop 2024-02-19T15:32:25.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.660+0000 7efc80200640 1 -- 172.21.15.47:0/1574822951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc7806d650 msgr2=0x7efc780c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:25.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.660+0000 7efc80200640 1 --2- 172.21.15.47:0/1574822951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc7806d650 0x7efc780c8cc0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7efc6c0ad040 tx=0x7efc6c01aca0 comp rx=0 tx=0).stop 2024-02-19T15:32:25.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.660+0000 7efc80200640 1 -- 172.21.15.47:0/1574822951 shutdown_connections 2024-02-19T15:32:25.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.660+0000 7efc80200640 1 --2- 172.21.15.47:0/1574822951 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7efc58066e40 0x7efc58069300 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:25.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.660+0000 7efc80200640 1 --2- 172.21.15.47:0/1574822951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc780ced10 0x7efc78146c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:25.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.660+0000 7efc80200640 1 --2- 172.21.15.47:0/1574822951 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc780cded0 0x7efc780c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:25.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.660+0000 7efc80200640 1 --2- 172.21.15.47:0/1574822951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc7806d650 0x7efc780c8cc0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:25.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.660+0000 7efc80200640 1 -- 172.21.15.47:0/1574822951 >> 172.21.15.47:0/1574822951 conn(0x7efc780d1bf0 msgr2=0x7efc780d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:25.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.660+0000 7efc80200640 1 -- 172.21.15.47:0/1574822951 shutdown_connections 2024-02-19T15:32:25.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:25.660+0000 7efc80200640 1 -- 172.21.15.47:0/1574822951 wait complete. 2024-02-19T15:32:25.878 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:25.879 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:26.312 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:32:26.879 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:27.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.020+0000 7f0926c00640 1 Processor -- start 2024-02-19T15:32:27.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.020+0000 7f0926c00640 1 -- start start 2024-02-19T15:32:27.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.020+0000 7f0926c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f09200b2fc0 0x7f09200d0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:27.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.020+0000 7f0926c00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f09200cef90 0x7f09200ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:27.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.020+0000 7f0926c00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09200ced80 con 0x7f09200ce930 2024-02-19T15:32:27.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.020+0000 7f0926c00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09200d10a0 con 0x7f09200cef90 2024-02-19T15:32:27.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.020+0000 7f0926c00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09200b33a0 con 0x7f09200b2fc0 2024-02-19T15:32:27.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.020+0000 7f0917e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f09200cef90 0x7f09200ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:27.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.020+0000 7f0924c00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f09200ce930 0x7f092006bfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54846/0 (socket says 172.21.15.47:54846) 2024-02-19T15:32:27.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.020+0000 7f0924c00640 1 -- 172.21.15.47:0/2543781866 learned_addr learned my addr 172.21.15.47:0/2543781866 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:32:27.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.020+0000 7f091fe00640 1 --2- 172.21.15.47:0/2543781866 >> v2:172.21.15.116:3300/0 conn(0x7f09200b2fc0 0x7f09200d0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:27.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.024+0000 7f0917e00640 1 -- 172.21.15.47:0/2543781866 >> v2:172.21.15.116:3300/0 conn(0x7f09200b2fc0 msgr2=0x7f09200d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:27.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.024+0000 7f0917e00640 1 --2- 172.21.15.47:0/2543781866 >> v2:172.21.15.116:3300/0 conn(0x7f09200b2fc0 0x7f09200d0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:27.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.024+0000 7f0917e00640 1 -- 172.21.15.47:0/2543781866 >> v1:172.21.15.47:6789/0 conn(0x7f09200ce930 legacy=0x7f092006bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:27.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.024+0000 7f0917e00640 1 -- 172.21.15.47:0/2543781866 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09200cc2f0 con 0x7f09200cef90 2024-02-19T15:32:27.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.024+0000 7f091f400640 1 -- 172.21.15.47:0/2543781866 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144279564 0 0) 0x7f09180014a0 con 0x7f09200ce930 2024-02-19T15:32:27.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.024+0000 7f0917e00640 1 --2- 172.21.15.47:0/2543781866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09200cef90 0x7f09200ce3f0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f09200d2890 tx=0x7f090801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be4fca8557b5ab7e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:27.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.024+0000 7f091f400640 1 -- 172.21.15.47:0/2543781866 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09080ac640 con 0x7f09200cef90 2024-02-19T15:32:27.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.024+0000 7f091f400640 1 -- 172.21.15.47:0/2543781866 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09080ac7a0 con 0x7f09200cef90 2024-02-19T15:32:27.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.024+0000 7f091f400640 1 -- 172.21.15.47:0/2543781866 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09080acb30 con 0x7f09200cef90 2024-02-19T15:32:27.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.024+0000 7f0926c00640 1 -- 172.21.15.47:0/2543781866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09200cef90 msgr2=0x7f09200ce3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:27.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.024+0000 7f0926c00640 1 --2- 172.21.15.47:0/2543781866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09200cef90 0x7f09200ce3f0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f09200d2890 tx=0x7f090801c3f0 comp rx=0 tx=0).stop 2024-02-19T15:32:27.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.024+0000 7f0926c00640 1 -- 172.21.15.47:0/2543781866 shutdown_connections 2024-02-19T15:32:27.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.024+0000 7f0926c00640 1 --2- 172.21.15.47:0/2543781866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09200cef90 0x7f09200ce3f0 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:27.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.024+0000 7f0926c00640 1 --2- 172.21.15.47:0/2543781866 >> v2:172.21.15.116:3300/0 conn(0x7f09200b2fc0 0x7f09200d0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:27.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.024+0000 7f0926c00640 1 -- 172.21.15.47:0/2543781866 >> 172.21.15.47:0/2543781866 conn(0x7f09200d1b70 msgr2=0x7f09200d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:27.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.024+0000 7f0926c00640 1 -- 172.21.15.47:0/2543781866 shutdown_connections 2024-02-19T15:32:27.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.024+0000 7f0926c00640 1 -- 172.21.15.47:0/2543781866 wait complete. 2024-02-19T15:32:27.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f0926c00640 1 Processor -- start 2024-02-19T15:32:27.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f0926c00640 1 -- start start 2024-02-19T15:32:27.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f0926c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09200b2fc0 0x7f0920147030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:27.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f0926c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09200ce930 0x7f0920154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:27.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f0926c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09200cef90 0x7f092006fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:27.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f0926c00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092015fa20 con 0x7f09200cef90 2024-02-19T15:32:27.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f0926c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0920156180 con 0x7f09200b2fc0 2024-02-19T15:32:27.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f0926c00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09201558e0 con 0x7f09200ce930 2024-02-19T15:32:27.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f0924c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09200cef90 0x7f092006fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:27.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f0924c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09200cef90 0x7f092006fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56434/0 (socket says 172.21.15.47:56434) 2024-02-19T15:32:27.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f0924c00640 1 -- 172.21.15.47:0/3433014969 learned_addr learned my addr 172.21.15.47:0/3433014969 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:27.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f091fe00640 1 --2- 172.21.15.47:0/3433014969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09200b2fc0 0x7f0920147030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:27.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f0917e00640 1 --2- 172.21.15.47:0/3433014969 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09200ce930 0x7f0920154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:27.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f0924c00640 1 -- 172.21.15.47:0/3433014969 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09200ce930 msgr2=0x7f0920154e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:27.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f0924c00640 1 --2- 172.21.15.47:0/3433014969 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09200ce930 0x7f0920154e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:27.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f0924c00640 1 -- 172.21.15.47:0/3433014969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09200b2fc0 msgr2=0x7f0920147030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:27.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f0924c00640 1 --2- 172.21.15.47:0/3433014969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09200b2fc0 0x7f0920147030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:27.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f0924c00640 1 -- 172.21.15.47:0/3433014969 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0908009420 con 0x7f09200cef90 2024-02-19T15:32:27.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f0917e00640 1 --2- 172.21.15.47:0/3433014969 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09200ce930 0x7f0920154e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:32:27.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f0924c00640 1 --2- 172.21.15.47:0/3433014969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09200cef90 0x7f092006fbe0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f091800c700 tx=0x7f091800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:27.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f091cc00640 1 -- 172.21.15.47:0/3433014969 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091800ce10 con 0x7f09200cef90 2024-02-19T15:32:27.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f091cc00640 1 -- 172.21.15.47:0/3433014969 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f091800a640 con 0x7f09200cef90 2024-02-19T15:32:27.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.028+0000 7f091cc00640 1 -- 172.21.15.47:0/3433014969 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091800aa70 con 0x7f09200cef90 2024-02-19T15:32:27.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.032+0000 7f0926c00640 1 -- 172.21.15.47:0/3433014969 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0920155bc0 con 0x7f09200cef90 2024-02-19T15:32:27.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.032+0000 7f0926c00640 1 -- 172.21.15.47:0/3433014969 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09201565b0 con 0x7f09200cef90 2024-02-19T15:32:27.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.032+0000 7f0926c00640 1 -- 172.21.15.47:0/3433014969 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f092015fb60 con 0x7f09200cef90 2024-02-19T15:32:27.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.032+0000 7f091cc00640 1 -- 172.21.15.47:0/3433014969 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f091800abd0 con 0x7f09200cef90 2024-02-19T15:32:27.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.032+0000 7f091cc00640 1 --2- 172.21.15.47:0/3433014969 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f08fc066e40 0x7f08fc069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:27.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.032+0000 7f091cc00640 1 -- 172.21.15.47:0/3433014969 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0918083d20 con 0x7f09200cef90 2024-02-19T15:32:27.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.032+0000 7f091fe00640 1 --2- 172.21.15.47:0/3433014969 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f08fc066e40 0x7f08fc069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:27.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.032+0000 7f091fe00640 1 --2- 172.21.15.47:0/3433014969 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f08fc066e40 0x7f08fc069300 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f0910009ac0 tx=0x7f0910009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:27.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.036+0000 7f091cc00640 1 -- 172.21.15.47:0/3433014969 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f092015fb60 con 0x7f09200cef90 2024-02-19T15:32:27.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.232+0000 7f0926c00640 1 -- 172.21.15.47:0/3433014969 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09200ced80 con 0x7f09200cef90 2024-02-19T15:32:27.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.232+0000 7f091cc00640 1 -- 172.21.15.47:0/3433014969 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f09200ced80 con 0x7f09200cef90 2024-02-19T15:32:27.233 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:27.233 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:27.233 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:27.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.236+0000 7f0926c00640 1 -- 172.21.15.47:0/3433014969 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f08fc066e40 msgr2=0x7f08fc069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:27.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.236+0000 7f0926c00640 1 --2- 172.21.15.47:0/3433014969 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f08fc066e40 0x7f08fc069300 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f0910009ac0 tx=0x7f0910009190 comp rx=0 tx=0).stop 2024-02-19T15:32:27.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.236+0000 7f0926c00640 1 -- 172.21.15.47:0/3433014969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09200cef90 msgr2=0x7f092006fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:27.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.236+0000 7f0926c00640 1 --2- 172.21.15.47:0/3433014969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09200cef90 0x7f092006fbe0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f091800c700 tx=0x7f091800cbc0 comp rx=0 tx=0).stop 2024-02-19T15:32:27.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.236+0000 7f0926c00640 1 -- 172.21.15.47:0/3433014969 shutdown_connections 2024-02-19T15:32:27.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.236+0000 7f0926c00640 1 --2- 172.21.15.47:0/3433014969 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f08fc066e40 0x7f08fc069300 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:27.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.236+0000 7f0926c00640 1 --2- 172.21.15.47:0/3433014969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09200cef90 0x7f092006fbe0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:27.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.236+0000 7f0926c00640 1 --2- 172.21.15.47:0/3433014969 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09200ce930 0x7f0920154e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:27.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.236+0000 7f0926c00640 1 --2- 172.21.15.47:0/3433014969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09200b2fc0 0x7f0920147030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:27.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.236+0000 7f0926c00640 1 -- 172.21.15.47:0/3433014969 >> 172.21.15.47:0/3433014969 conn(0x7f09200d1b70 msgr2=0x7f092015e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:27.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.236+0000 7f0926c00640 1 -- 172.21.15.47:0/3433014969 shutdown_connections 2024-02-19T15:32:27.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:27.236+0000 7f0926c00640 1 -- 172.21.15.47:0/3433014969 wait complete. 2024-02-19T15:32:27.455 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:27.455 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:28.313 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:32:28.456 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:28.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.596+0000 7f9068000640 1 Processor -- start 2024-02-19T15:32:28.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.596+0000 7f9068000640 1 -- start start 2024-02-19T15:32:28.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.600+0000 7f9068000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f906015bcd0 0x7f90600cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:28.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.600+0000 7f9068000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90600d0360 0x7f906015d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:28.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.600+0000 7f9068000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90600cc700 con 0x7f90600cedb0 2024-02-19T15:32:28.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.600+0000 7f9068000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90600cf5e0 con 0x7f906015bcd0 2024-02-19T15:32:28.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.600+0000 7f9068000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90600cf300 con 0x7f90600d0360 2024-02-19T15:32:28.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.600+0000 7f9065600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f906015bcd0 0x7f90600cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:28.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.600+0000 7f9064c00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f90600cedb0 0x7f90600cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54860/0 (socket says 172.21.15.47:54860) 2024-02-19T15:32:28.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.600+0000 7f9064c00640 1 -- 172.21.15.47:0/3033517101 learned_addr learned my addr 172.21.15.47:0/3033517101 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:32:28.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.600+0000 7f9066000640 1 --2- 172.21.15.47:0/3033517101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90600d0360 0x7f906015d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:28.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.600+0000 7f9065600640 1 -- 172.21.15.47:0/3033517101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90600d0360 msgr2=0x7f906015d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:28.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.600+0000 7f9065600640 1 --2- 172.21.15.47:0/3033517101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90600d0360 0x7f906015d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:28.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.600+0000 7f9065600640 1 -- 172.21.15.47:0/3033517101 >> v1:172.21.15.47:6789/0 conn(0x7f90600cedb0 legacy=0x7f90600cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:28.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.600+0000 7f9065600640 1 -- 172.21.15.47:0/3033517101 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90600cc9a0 con 0x7f906015bcd0 2024-02-19T15:32:28.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.600+0000 7f905fe00640 1 -- 172.21.15.47:0/3033517101 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359686742 0 0) 0x7f90600cc700 con 0x7f90600cedb0 2024-02-19T15:32:28.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.600+0000 7f9065600640 1 --2- 172.21.15.47:0/3033517101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f906015bcd0 0x7f90600cd480 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f90540097c0 tx=0x7f905401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcdaff1c27dc22a6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:28.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.604+0000 7f905fe00640 1 -- 172.21.15.47:0/3033517101 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90540ab670 con 0x7f906015bcd0 2024-02-19T15:32:28.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.604+0000 7f905fe00640 1 -- 172.21.15.47:0/3033517101 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90540ab7d0 con 0x7f906015bcd0 2024-02-19T15:32:28.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.604+0000 7f905fe00640 1 -- 172.21.15.47:0/3033517101 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90540abb60 con 0x7f906015bcd0 2024-02-19T15:32:28.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.604+0000 7f9068000640 1 -- 172.21.15.47:0/3033517101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f906015bcd0 msgr2=0x7f90600cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:28.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.604+0000 7f9068000640 1 --2- 172.21.15.47:0/3033517101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f906015bcd0 0x7f90600cd480 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f90540097c0 tx=0x7f905401c470 comp rx=0 tx=0).stop 2024-02-19T15:32:28.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.604+0000 7f9068000640 1 -- 172.21.15.47:0/3033517101 shutdown_connections 2024-02-19T15:32:28.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.604+0000 7f9068000640 1 --2- 172.21.15.47:0/3033517101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90600d0360 0x7f906015d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:28.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.604+0000 7f9068000640 1 --2- 172.21.15.47:0/3033517101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f906015bcd0 0x7f90600cd480 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:28.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.604+0000 7f9068000640 1 -- 172.21.15.47:0/3033517101 >> 172.21.15.47:0/3033517101 conn(0x7f90600d1bf0 msgr2=0x7f90600d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:28.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.604+0000 7f9068000640 1 -- 172.21.15.47:0/3033517101 shutdown_connections 2024-02-19T15:32:28.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.604+0000 7f9068000640 1 -- 172.21.15.47:0/3033517101 wait complete. 2024-02-19T15:32:28.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.604+0000 7f9068000640 1 Processor -- start 2024-02-19T15:32:28.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.604+0000 7f9068000640 1 -- start start 2024-02-19T15:32:28.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.604+0000 7f9068000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90600cedb0 0x7f9060151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:28.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.604+0000 7f9068000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90600d0360 0x7f90600cb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:28.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.604+0000 7f9068000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f906015bcd0 0x7f90600c7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:28.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.604+0000 7f9068000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90600cbbb0 con 0x7f906015bcd0 2024-02-19T15:32:28.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.604+0000 7f9068000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9060152420 con 0x7f90600d0360 2024-02-19T15:32:28.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.604+0000 7f9068000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9060152700 con 0x7f90600cedb0 2024-02-19T15:32:28.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.604+0000 7f9066000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f906015bcd0 0x7f90600c7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:28.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.608+0000 7f9066000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f906015bcd0 0x7f90600c7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56444/0 (socket says 172.21.15.47:56444) 2024-02-19T15:32:28.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.608+0000 7f9066000640 1 -- 172.21.15.47:0/2616563200 learned_addr learned my addr 172.21.15.47:0/2616563200 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:28.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.608+0000 7f9065600640 1 --2- 172.21.15.47:0/2616563200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90600cedb0 0x7f9060151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:28.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.608+0000 7f9064c00640 1 --2- 172.21.15.47:0/2616563200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90600d0360 0x7f90600cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:28.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.608+0000 7f9066000640 1 -- 172.21.15.47:0/2616563200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90600cedb0 msgr2=0x7f9060151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:28.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.608+0000 7f9066000640 1 --2- 172.21.15.47:0/2616563200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90600cedb0 0x7f9060151d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:28.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.608+0000 7f9066000640 1 -- 172.21.15.47:0/2616563200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90600d0360 msgr2=0x7f90600cb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:28.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.608+0000 7f9066000640 1 --2- 172.21.15.47:0/2616563200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90600d0360 0x7f90600cb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:28.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.608+0000 7f9066000640 1 -- 172.21.15.47:0/2616563200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9054009420 con 0x7f906015bcd0 2024-02-19T15:32:28.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.608+0000 7f9065600640 1 --2- 172.21.15.47:0/2616563200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90600cedb0 0x7f9060151d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:32:28.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.608+0000 7f9066000640 1 --2- 172.21.15.47:0/2616563200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f906015bcd0 0x7f90600c7ac0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f905800ef90 tx=0x7f905800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:28.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.608+0000 7f905d600640 1 -- 172.21.15.47:0/2616563200 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9058009b20 con 0x7f906015bcd0 2024-02-19T15:32:28.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.608+0000 7f905d600640 1 -- 172.21.15.47:0/2616563200 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905800f040 con 0x7f906015bcd0 2024-02-19T15:32:28.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.608+0000 7f9068000640 1 -- 172.21.15.47:0/2616563200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90600cdb40 con 0x7f906015bcd0 2024-02-19T15:32:28.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.608+0000 7f905d600640 1 -- 172.21.15.47:0/2616563200 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90580146c0 con 0x7f906015bcd0 2024-02-19T15:32:28.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.608+0000 7f9068000640 1 -- 172.21.15.47:0/2616563200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9060152cc0 con 0x7f906015bcd0 2024-02-19T15:32:28.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.608+0000 7f9068000640 1 -- 172.21.15.47:0/2616563200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f906006fc70 con 0x7f906015bcd0 2024-02-19T15:32:28.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.608+0000 7f905d600640 1 -- 172.21.15.47:0/2616563200 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9058014820 con 0x7f906015bcd0 2024-02-19T15:32:28.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.608+0000 7f905d600640 1 --2- 172.21.15.47:0/2616563200 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9040066e40 0x7f9040069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:28.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.612+0000 7f905d600640 1 -- 172.21.15.47:0/2616563200 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f90580857e0 con 0x7f906015bcd0 2024-02-19T15:32:28.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.612+0000 7f9065600640 1 --2- 172.21.15.47:0/2616563200 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9040066e40 0x7f9040069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:28.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.612+0000 7f9065600640 1 --2- 172.21.15.47:0/2616563200 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9040066e40 0x7f9040069300 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f90540af000 tx=0x7f90540a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:28.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.612+0000 7f905d600640 1 -- 172.21.15.47:0/2616563200 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f9058052580 con 0x7f906015bcd0 2024-02-19T15:32:28.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.812+0000 7f9068000640 1 -- 172.21.15.47:0/2616563200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9060075360 con 0x7f906015bcd0 2024-02-19T15:32:28.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.816+0000 7f905d600640 1 -- 172.21.15.47:0/2616563200 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f9058018090 con 0x7f906015bcd0 2024-02-19T15:32:28.814 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:28.814 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:28.815 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:28.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.816+0000 7f9068000640 1 -- 172.21.15.47:0/2616563200 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9040066e40 msgr2=0x7f9040069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:28.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.816+0000 7f9068000640 1 --2- 172.21.15.47:0/2616563200 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9040066e40 0x7f9040069300 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f90540af000 tx=0x7f90540a7000 comp rx=0 tx=0).stop 2024-02-19T15:32:28.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.816+0000 7f9068000640 1 -- 172.21.15.47:0/2616563200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f906015bcd0 msgr2=0x7f90600c7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:28.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.816+0000 7f9068000640 1 --2- 172.21.15.47:0/2616563200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f906015bcd0 0x7f90600c7ac0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f905800ef90 tx=0x7f905800c490 comp rx=0 tx=0).stop 2024-02-19T15:32:28.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.816+0000 7f9068000640 1 -- 172.21.15.47:0/2616563200 shutdown_connections 2024-02-19T15:32:28.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.816+0000 7f9068000640 1 --2- 172.21.15.47:0/2616563200 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9040066e40 0x7f9040069300 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:28.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.816+0000 7f9068000640 1 --2- 172.21.15.47:0/2616563200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f906015bcd0 0x7f90600c7ac0 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:28.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.816+0000 7f9068000640 1 --2- 172.21.15.47:0/2616563200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90600d0360 0x7f90600cb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:28.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.816+0000 7f9068000640 1 --2- 172.21.15.47:0/2616563200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90600cedb0 0x7f9060151d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:28.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.816+0000 7f9068000640 1 -- 172.21.15.47:0/2616563200 >> 172.21.15.47:0/2616563200 conn(0x7f90600d1bf0 msgr2=0x7f90600cfc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:28.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.816+0000 7f9068000640 1 -- 172.21.15.47:0/2616563200 shutdown_connections 2024-02-19T15:32:28.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:28.816+0000 7f9068000640 1 -- 172.21.15.47:0/2616563200 wait complete. 2024-02-19T15:32:29.035 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:29.035 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:30.037 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:30.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1ba1400640 1 Processor -- start 2024-02-19T15:32:30.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1ba1400640 1 -- start start 2024-02-19T15:32:30.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1ba1400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1b9c0cea80 0x7f1b9c0727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:30.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1ba1400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b9c15d280 con 0x7f1b9c145590 2024-02-19T15:32:30.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1ba1400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b9c070ff0 con 0x7f1b9c15c390 2024-02-19T15:32:30.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1ba1400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b9c0709e0 con 0x7f1b9c0cea80 2024-02-19T15:32:30.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b9a800640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1b9c145590 0x7f1b9c06c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54866/0 (socket says 172.21.15.47:54866) 2024-02-19T15:32:30.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b9a800640 1 -- 172.21.15.47:0/3303595645 learned_addr learned my addr 172.21.15.47:0/3303595645 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:32:30.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b91e00640 1 --2- 172.21.15.47:0/3303595645 >> v2:172.21.15.116:3300/0 conn(0x7f1b9c0cea80 0x7f1b9c0727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:30.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b99e00640 1 -- 172.21.15.47:0/3303595645 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336083717 0 0) 0x7f1b9c15d280 con 0x7f1b9c145590 2024-02-19T15:32:30.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b99e00640 1 -- 172.21.15.47:0/3303595645 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b80003530 con 0x7f1b9c145590 2024-02-19T15:32:30.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b99e00640 1 -- 172.21.15.47:0/3303595645 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656440653 0 0) 0x7f1b9c070ff0 con 0x7f1b9c15c390 2024-02-19T15:32:30.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b99e00640 1 -- 172.21.15.47:0/3303595645 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b9c15d280 con 0x7f1b9c15c390 2024-02-19T15:32:30.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b99e00640 1 -- 172.21.15.47:0/3303595645 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 941277166 0 0) 0x7f1b80003530 con 0x7f1b9c145590 2024-02-19T15:32:30.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b99e00640 1 -- 172.21.15.47:0/3303595645 >> v2:172.21.15.116:3300/0 conn(0x7f1b9c0cea80 msgr2=0x7f1b9c0727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:30.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b99e00640 1 --2- 172.21.15.47:0/3303595645 >> v2:172.21.15.116:3300/0 conn(0x7f1b9c0cea80 0x7f1b9c0727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:30.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b99e00640 1 -- 172.21.15.47:0/3303595645 >> v1:172.21.15.116:6789/0 conn(0x7f1b9c15c390 legacy=0x7f1b9c15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:30.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b99e00640 1 -- 172.21.15.47:0/3303595645 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9c0cf020 con 0x7f1b9c145590 2024-02-19T15:32:30.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b99e00640 1 -- 172.21.15.47:0/3303595645 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3928399118 0 0) 0x7f1b8c001680 con 0x7f1b9c145590 2024-02-19T15:32:30.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b91e00640 1 --2- 172.21.15.47:0/3303595645 >> v2:172.21.15.116:3300/0 conn(0x7f1b9c0cea80 0x7f1b9c0727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:32:30.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b99e00640 1 -- 172.21.15.47:0/3303595645 >> v1:172.21.15.47:6789/0 conn(0x7f1b9c145590 legacy=0x7f1b9c06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:30.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b99e00640 1 --2- 172.21.15.47:0/3303595645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b80003b10 0x7f1b80003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:30.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b99e00640 1 --2- 172.21.15.47:0/3303595645 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b800047d0 0x7f1b80004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:30.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b99e00640 1 --2- 172.21.15.47:0/3303595645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b80005190 0x7f1b8000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:30.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b99e00640 1 -- 172.21.15.47:0/3303595645 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c0709e0 con 0x7f1b80003b10 2024-02-19T15:32:30.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b99e00640 1 -- 172.21.15.47:0/3303595645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8000c010 con 0x7f1b80005190 2024-02-19T15:32:30.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b99e00640 1 -- 172.21.15.47:0/3303595645 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8000c1d0 con 0x7f1b800047d0 2024-02-19T15:32:30.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.180+0000 7f1b9a800640 1 --2- 172.21.15.47:0/3303595645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b80003b10 0x7f1b80003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:30.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.184+0000 7f1b91e00640 1 --2- 172.21.15.47:0/3303595645 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b800047d0 0x7f1b80004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:30.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.184+0000 7f1b9b200640 1 --2- 172.21.15.47:0/3303595645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b80005190 0x7f1b8000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:30.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.184+0000 7f1b9a800640 1 -- 172.21.15.47:0/3303595645 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b800047d0 msgr2=0x7f1b80004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:30.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.184+0000 7f1b9a800640 1 --2- 172.21.15.47:0/3303595645 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b800047d0 0x7f1b80004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:30.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.184+0000 7f1b9a800640 1 -- 172.21.15.47:0/3303595645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b80005190 msgr2=0x7f1b8000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:30.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.184+0000 7f1b9a800640 1 --2- 172.21.15.47:0/3303595645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b80005190 0x7f1b8000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:30.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.184+0000 7f1b9a800640 1 -- 172.21.15.47:0/3303595645 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b80003530 con 0x7f1b80003b10 2024-02-19T15:32:30.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.184+0000 7f1b9b200640 1 -- 172.21.15.47:0/3303595645 reap_dead start 2024-02-19T15:32:30.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.184+0000 7f1b9a800640 1 --2- 172.21.15.47:0/3303595645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b80003b10 0x7f1b80003f10 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c00c160 tx=0x7f1b8c01cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3339384516888df server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:30.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.184+0000 7f1b99e00640 1 -- 172.21.15.47:0/3303595645 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c0058e0 con 0x7f1b80003b10 2024-02-19T15:32:30.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.184+0000 7f1b99e00640 1 -- 172.21.15.47:0/3303595645 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8c005a60 con 0x7f1b80003b10 2024-02-19T15:32:30.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.184+0000 7f1b99e00640 1 -- 172.21.15.47:0/3303595645 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c005e50 con 0x7f1b80003b10 2024-02-19T15:32:30.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.184+0000 7f1ba1400640 1 -- 172.21.15.47:0/3303595645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b80003b10 msgr2=0x7f1b80003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:30.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.184+0000 7f1ba1400640 1 --2- 172.21.15.47:0/3303595645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b80003b10 0x7f1b80003f10 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c00c160 tx=0x7f1b8c01cbe0 comp rx=0 tx=0).stop 2024-02-19T15:32:30.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.184+0000 7f1ba1400640 1 -- 172.21.15.47:0/3303595645 shutdown_connections 2024-02-19T15:32:30.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.184+0000 7f1ba1400640 1 --2- 172.21.15.47:0/3303595645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b80003b10 0x7f1b80003f10 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:30.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.184+0000 7f1ba1400640 1 -- 172.21.15.47:0/3303595645 >> 172.21.15.47:0/3303595645 conn(0x7f1b9c0b16a0 msgr2=0x7f1b9c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:30.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.184+0000 7f1ba1400640 1 -- 172.21.15.47:0/3303595645 shutdown_connections 2024-02-19T15:32:30.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.188+0000 7f1ba1400640 1 -- 172.21.15.47:0/3303595645 wait complete. 2024-02-19T15:32:30.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.188+0000 7f1ba1400640 1 Processor -- start 2024-02-19T15:32:30.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.188+0000 7f1ba1400640 1 -- start start 2024-02-19T15:32:30.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.188+0000 7f1ba1400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b80003b10 0x7f1b9c1f4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:30.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.188+0000 7f1ba1400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b9c0aea70 0x7f1b9c15e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:30.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.188+0000 7f1ba1400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b9c0aeef0 0x7f1b9c15e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:30.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.188+0000 7f1ba1400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c1f5140 con 0x7f1b80003b10 2024-02-19T15:32:30.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.188+0000 7f1ba1400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c1f53e0 con 0x7f1b9c0aeef0 2024-02-19T15:32:30.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.188+0000 7f1ba1400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c1f56c0 con 0x7f1b9c0aea70 2024-02-19T15:32:30.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.188+0000 7f1b9a800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b80003b10 0x7f1b9c1f4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:30.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.188+0000 7f1b9a800640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b80003b10 0x7f1b9c1f4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56468/0 (socket says 172.21.15.47:56468) 2024-02-19T15:32:30.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.188+0000 7f1b9a800640 1 -- 172.21.15.47:0/1449701238 learned_addr learned my addr 172.21.15.47:0/1449701238 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:30.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.188+0000 7f1b91e00640 1 --2- 172.21.15.47:0/1449701238 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b9c0aea70 0x7f1b9c15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:30.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.188+0000 7f1b9b200640 1 --2- 172.21.15.47:0/1449701238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b9c0aeef0 0x7f1b9c15e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:30.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.188+0000 7f1b9a800640 1 -- 172.21.15.47:0/1449701238 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b9c0aea70 msgr2=0x7f1b9c15e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:30.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.188+0000 7f1b9a800640 1 --2- 172.21.15.47:0/1449701238 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b9c0aea70 0x7f1b9c15e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:30.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.188+0000 7f1b9a800640 1 -- 172.21.15.47:0/1449701238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b9c0aeef0 msgr2=0x7f1b9c15e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:30.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.188+0000 7f1b9a800640 1 --2- 172.21.15.47:0/1449701238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b9c0aeef0 0x7f1b9c15e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:30.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.188+0000 7f1b9a800640 1 -- 172.21.15.47:0/1449701238 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9c1f59a0 con 0x7f1b80003b10 2024-02-19T15:32:30.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.188+0000 7f1b9a800640 1 --2- 172.21.15.47:0/1449701238 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b80003b10 0x7f1b9c1f4c00 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0ab040 tx=0x7f1b80005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:30.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.192+0000 7f1b93400640 1 -- 172.21.15.47:0/1449701238 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c001680 con 0x7f1b80003b10 2024-02-19T15:32:30.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.192+0000 7f1b93400640 1 -- 172.21.15.47:0/1449701238 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8c0047f0 con 0x7f1b80003b10 2024-02-19T15:32:30.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.192+0000 7f1b93400640 1 -- 172.21.15.47:0/1449701238 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c004ba0 con 0x7f1b80003b10 2024-02-19T15:32:30.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.192+0000 7f1ba1400640 1 -- 172.21.15.47:0/1449701238 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b9c1f5c80 con 0x7f1b80003b10 2024-02-19T15:32:30.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.192+0000 7f1ba1400640 1 -- 172.21.15.47:0/1449701238 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b9c15eee0 con 0x7f1b80003b10 2024-02-19T15:32:30.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.192+0000 7f1ba1400640 1 -- 172.21.15.47:0/1449701238 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b9c075360 con 0x7f1b80003b10 2024-02-19T15:32:30.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.192+0000 7f1b93400640 1 -- 172.21.15.47:0/1449701238 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f1b8c004d00 con 0x7f1b80003b10 2024-02-19T15:32:30.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.192+0000 7f1b93400640 1 --2- 172.21.15.47:0/1449701238 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1b78066e70 0x7f1b78069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:30.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.192+0000 7f1b93400640 1 -- 172.21.15.47:0/1449701238 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1b8c11e330 con 0x7f1b80003b10 2024-02-19T15:32:30.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.196+0000 7f1b91e00640 1 --2- 172.21.15.47:0/1449701238 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1b78066e70 0x7f1b78069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:30.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.196+0000 7f1b91e00640 1 --2- 172.21.15.47:0/1449701238 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1b78066e70 0x7f1b78069330 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f1b84009fd0 tx=0x7f1b8400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:30.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.196+0000 7f1b93400640 1 -- 172.21.15.47:0/1449701238 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f1b8c0eb0d0 con 0x7f1b80003b10 2024-02-19T15:32:30.315 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:32:30.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.392+0000 7f1ba1400640 1 -- 172.21.15.47:0/1449701238 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b9c06dcb0 con 0x7f1b80003b10 2024-02-19T15:32:30.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.392+0000 7f1b93400640 1 -- 172.21.15.47:0/1449701238 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f1b8c0ee790 con 0x7f1b80003b10 2024-02-19T15:32:30.393 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:30.393 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:30.394 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:30.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.396+0000 7f1ba1400640 1 -- 172.21.15.47:0/1449701238 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1b78066e70 msgr2=0x7f1b78069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:30.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.396+0000 7f1ba1400640 1 --2- 172.21.15.47:0/1449701238 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1b78066e70 0x7f1b78069330 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f1b84009fd0 tx=0x7f1b8400b040 comp rx=0 tx=0).stop 2024-02-19T15:32:30.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.396+0000 7f1ba1400640 1 -- 172.21.15.47:0/1449701238 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b80003b10 msgr2=0x7f1b9c1f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:30.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.396+0000 7f1ba1400640 1 --2- 172.21.15.47:0/1449701238 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b80003b10 0x7f1b9c1f4c00 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0ab040 tx=0x7f1b80005800 comp rx=0 tx=0).stop 2024-02-19T15:32:30.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.396+0000 7f1ba1400640 1 -- 172.21.15.47:0/1449701238 shutdown_connections 2024-02-19T15:32:30.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.396+0000 7f1ba1400640 1 --2- 172.21.15.47:0/1449701238 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f1b78066e70 0x7f1b78069330 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:30.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.396+0000 7f1ba1400640 1 --2- 172.21.15.47:0/1449701238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b9c0aeef0 0x7f1b9c15e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:30.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.396+0000 7f1ba1400640 1 --2- 172.21.15.47:0/1449701238 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1b9c0aea70 0x7f1b9c15e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:30.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.396+0000 7f1ba1400640 1 --2- 172.21.15.47:0/1449701238 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b80003b10 0x7f1b9c1f4c00 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:30.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.396+0000 7f1ba1400640 1 -- 172.21.15.47:0/1449701238 >> 172.21.15.47:0/1449701238 conn(0x7f1b9c0b16a0 msgr2=0x7f1b9c141750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:30.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.396+0000 7f1ba1400640 1 -- 172.21.15.47:0/1449701238 shutdown_connections 2024-02-19T15:32:30.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:30.396+0000 7f1ba1400640 1 -- 172.21.15.47:0/1449701238 wait complete. 2024-02-19T15:32:30.617 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:30.617 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:31.618 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:31.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.760+0000 7f75d6000640 1 Processor -- start 2024-02-19T15:32:31.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.760+0000 7f75d6000640 1 -- start start 2024-02-19T15:32:31.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.760+0000 7f75d6000640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f75d015e1b0 0x7f75d006d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:31.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.760+0000 7f75d6000640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d00b1b10 con 0x7f75d00d2150 2024-02-19T15:32:31.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.760+0000 7f75d6000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d00cea30 con 0x7f75d015ddd0 2024-02-19T15:32:31.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.760+0000 7f75d6000640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75d00ced10 con 0x7f75d015e1b0 2024-02-19T15:32:31.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.760+0000 7f75cfe00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f75d00d2150 0x7f75d00b1fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54882/0 (socket says 172.21.15.47:54882) 2024-02-19T15:32:31.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.760+0000 7f75cfe00640 1 -- 172.21.15.47:0/2796984674 learned_addr learned my addr 172.21.15.47:0/2796984674 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:32:31.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.760+0000 7f75d5600640 1 --2- 172.21.15.47:0/2796984674 >> v2:172.21.15.47:3300/0 conn(0x7f75d015e1b0 0x7f75d006d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:31.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.760+0000 7f75cf400640 1 -- 172.21.15.47:0/2796984674 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853090919 0 0) 0x7f75d00b1b10 con 0x7f75d00d2150 2024-02-19T15:32:31.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.760+0000 7f75cf400640 1 -- 172.21.15.47:0/2796984674 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b8003530 con 0x7f75d00d2150 2024-02-19T15:32:31.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.760+0000 7f75d5600640 1 -- 172.21.15.47:0/2796984674 >> v1:172.21.15.116:6789/0 conn(0x7f75d015ddd0 legacy=0x7f75d00d1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:31.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.760+0000 7f75d5600640 1 -- 172.21.15.47:0/2796984674 >> v1:172.21.15.47:6789/0 conn(0x7f75d00d2150 legacy=0x7f75d00b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:31.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.760+0000 7f75d5600640 1 -- 172.21.15.47:0/2796984674 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d00ceff0 con 0x7f75d015e1b0 2024-02-19T15:32:31.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.764+0000 7f75d5600640 1 --2- 172.21.15.47:0/2796984674 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75d015e1b0 0x7f75d006d590 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f75c8009d90 tx=0x7f75c801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12410f4911a11a7a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:31.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.764+0000 7f75cf400640 1 -- 172.21.15.47:0/2796984674 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c80a59e0 con 0x7f75d015e1b0 2024-02-19T15:32:31.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.764+0000 7f75cf400640 1 -- 172.21.15.47:0/2796984674 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c80a5b40 con 0x7f75d015e1b0 2024-02-19T15:32:31.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.764+0000 7f75cf400640 1 -- 172.21.15.47:0/2796984674 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c80ad440 con 0x7f75d015e1b0 2024-02-19T15:32:31.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.764+0000 7f75d6000640 1 -- 172.21.15.47:0/2796984674 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75d015e1b0 msgr2=0x7f75d006d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:31.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.764+0000 7f75d6000640 1 --2- 172.21.15.47:0/2796984674 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75d015e1b0 0x7f75d006d590 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f75c8009d90 tx=0x7f75c801c940 comp rx=0 tx=0).stop 2024-02-19T15:32:31.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.764+0000 7f75d6000640 1 -- 172.21.15.47:0/2796984674 shutdown_connections 2024-02-19T15:32:31.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.764+0000 7f75d6000640 1 --2- 172.21.15.47:0/2796984674 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75d015e1b0 0x7f75d006d590 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:31.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.764+0000 7f75d6000640 1 -- 172.21.15.47:0/2796984674 >> 172.21.15.47:0/2796984674 conn(0x7f75d009a9d0 msgr2=0x7f75d009ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:31.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.764+0000 7f75d6000640 1 -- 172.21.15.47:0/2796984674 shutdown_connections 2024-02-19T15:32:31.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.764+0000 7f75d6000640 1 -- 172.21.15.47:0/2796984674 wait complete. 2024-02-19T15:32:31.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.764+0000 7f75d6000640 1 Processor -- start 2024-02-19T15:32:31.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.764+0000 7f75d6000640 1 -- start start 2024-02-19T15:32:31.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75d6000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75d00d2150 0x7f75d0150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:31.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75d6000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d015ddd0 0x7f75d0161040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:31.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75d6000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75d015e1b0 0x7f75d006f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:31.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75d6000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d00b6f50 con 0x7f75d015e1b0 2024-02-19T15:32:31.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75d6000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d00b7510 con 0x7f75d00d2150 2024-02-19T15:32:31.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75d6000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d0161580 con 0x7f75d015ddd0 2024-02-19T15:32:31.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75d5600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75d015e1b0 0x7f75d006f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:31.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75d5600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75d015e1b0 0x7f75d006f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56490/0 (socket says 172.21.15.47:56490) 2024-02-19T15:32:31.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75d5600640 1 -- 172.21.15.47:0/1085604551 learned_addr learned my addr 172.21.15.47:0/1085604551 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:31.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75d4c00640 1 --2- 172.21.15.47:0/1085604551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75d00d2150 0x7f75d0150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:31.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75cfe00640 1 --2- 172.21.15.47:0/1085604551 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d015ddd0 0x7f75d0161040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:31.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75d5600640 1 -- 172.21.15.47:0/1085604551 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d015ddd0 msgr2=0x7f75d0161040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:31.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75d5600640 1 --2- 172.21.15.47:0/1085604551 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d015ddd0 0x7f75d0161040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:31.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75d5600640 1 -- 172.21.15.47:0/1085604551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75d00d2150 msgr2=0x7f75d0150a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:31.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75d5600640 1 --2- 172.21.15.47:0/1085604551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75d00d2150 0x7f75d0150a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:31.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75d5600640 1 -- 172.21.15.47:0/1085604551 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c8009a70 con 0x7f75d015e1b0 2024-02-19T15:32:31.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75d5600640 1 --2- 172.21.15.47:0/1085604551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75d015e1b0 0x7f75d006f8a0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f75c8009d90 tx=0x7f75c80196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:31.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75ccc00640 1 -- 172.21.15.47:0/1085604551 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c80add90 con 0x7f75d015e1b0 2024-02-19T15:32:31.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75ccc00640 1 -- 172.21.15.47:0/1085604551 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c8009450 con 0x7f75d015e1b0 2024-02-19T15:32:31.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75ccc00640 1 -- 172.21.15.47:0/1085604551 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c80b9690 con 0x7f75d015e1b0 2024-02-19T15:32:31.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75d6000640 1 -- 172.21.15.47:0/1085604551 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d015e7a0 con 0x7f75d015e1b0 2024-02-19T15:32:31.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75d6000640 1 -- 172.21.15.47:0/1085604551 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d00b71a0 con 0x7f75d015e1b0 2024-02-19T15:32:31.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.768+0000 7f75d6000640 1 -- 172.21.15.47:0/1085604551 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d0075360 con 0x7f75d015e1b0 2024-02-19T15:32:31.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.772+0000 7f75ccc00640 1 -- 172.21.15.47:0/1085604551 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f75c80b97f0 con 0x7f75d015e1b0 2024-02-19T15:32:31.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.772+0000 7f75ccc00640 1 --2- 172.21.15.47:0/1085604551 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f75b0066e40 0x7f75b0069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:31.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.772+0000 7f75ccc00640 1 -- 172.21.15.47:0/1085604551 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f75c8120090 con 0x7f75d015e1b0 2024-02-19T15:32:31.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.772+0000 7f75d4c00640 1 --2- 172.21.15.47:0/1085604551 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f75b0066e40 0x7f75b0069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:31.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.772+0000 7f75d4c00640 1 --2- 172.21.15.47:0/1085604551 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f75b0066e40 0x7f75b0069300 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f75c4007fd0 tx=0x7f75c4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:31.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.780+0000 7f75ccc00640 1 -- 172.21.15.47:0/1085604551 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f75c80ecee0 con 0x7f75d015e1b0 2024-02-19T15:32:31.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.976+0000 7f75d6000640 1 -- 172.21.15.47:0/1085604551 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75d00d2920 con 0x7f75d015e1b0 2024-02-19T15:32:31.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.976+0000 7f75ccc00640 1 -- 172.21.15.47:0/1085604551 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f75c80bdd80 con 0x7f75d015e1b0 2024-02-19T15:32:31.976 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:31.976 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:31.976 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:31.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.980+0000 7f75d6000640 1 -- 172.21.15.47:0/1085604551 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f75b0066e40 msgr2=0x7f75b0069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:31.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.980+0000 7f75d6000640 1 --2- 172.21.15.47:0/1085604551 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f75b0066e40 0x7f75b0069300 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f75c4007fd0 tx=0x7f75c4009040 comp rx=0 tx=0).stop 2024-02-19T15:32:31.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.980+0000 7f75d6000640 1 -- 172.21.15.47:0/1085604551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75d015e1b0 msgr2=0x7f75d006f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:31.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.980+0000 7f75d6000640 1 --2- 172.21.15.47:0/1085604551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75d015e1b0 0x7f75d006f8a0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f75c8009d90 tx=0x7f75c80196e0 comp rx=0 tx=0).stop 2024-02-19T15:32:31.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.980+0000 7f75d6000640 1 -- 172.21.15.47:0/1085604551 shutdown_connections 2024-02-19T15:32:31.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.980+0000 7f75d6000640 1 --2- 172.21.15.47:0/1085604551 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f75b0066e40 0x7f75b0069300 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:31.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.980+0000 7f75d6000640 1 --2- 172.21.15.47:0/1085604551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75d015e1b0 0x7f75d006f8a0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:31.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.980+0000 7f75d6000640 1 --2- 172.21.15.47:0/1085604551 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d015ddd0 0x7f75d0161040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:31.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.980+0000 7f75d6000640 1 --2- 172.21.15.47:0/1085604551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75d00d2150 0x7f75d0150a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:31.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.980+0000 7f75d6000640 1 -- 172.21.15.47:0/1085604551 >> 172.21.15.47:0/1085604551 conn(0x7f75d009a9d0 msgr2=0x7f75d006d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:31.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.980+0000 7f75d6000640 1 -- 172.21.15.47:0/1085604551 shutdown_connections 2024-02-19T15:32:31.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:31.980+0000 7f75d6000640 1 -- 172.21.15.47:0/1085604551 wait complete. 2024-02-19T15:32:32.198 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:32.198 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:32.318 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:32:33.200 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:33.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.340+0000 7fda49400640 1 Processor -- start 2024-02-19T15:32:33.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.340+0000 7fda49400640 1 -- start start 2024-02-19T15:32:33.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.340+0000 7fda49400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fda440cedb0 0x7fda440cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:33.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.340+0000 7fda49400640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda440cc700 con 0x7fda440d2710 2024-02-19T15:32:33.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.344+0000 7fda49400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda440cf300 con 0x7fda4415bcd0 2024-02-19T15:32:33.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.344+0000 7fda49400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda440cf5e0 con 0x7fda440cedb0 2024-02-19T15:32:33.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.344+0000 7fda43200640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fda440d2710 0x7fda4415d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60592/0 (socket says 172.21.15.47:60592) 2024-02-19T15:32:33.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.344+0000 7fda43200640 1 -- 172.21.15.47:0/3898790066 learned_addr learned my addr 172.21.15.47:0/3898790066 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:32:33.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.344+0000 7fda39e00640 1 --2- 172.21.15.47:0/3898790066 >> v2:172.21.15.116:3300/0 conn(0x7fda440cedb0 0x7fda440cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:33.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.344+0000 7fda41e00640 1 -- 172.21.15.47:0/3898790066 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471584862 0 0) 0x7fda440cc700 con 0x7fda440d2710 2024-02-19T15:32:33.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.344+0000 7fda41e00640 1 -- 172.21.15.47:0/3898790066 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda28003530 con 0x7fda440d2710 2024-02-19T15:32:33.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.344+0000 7fda39e00640 1 -- 172.21.15.47:0/3898790066 >> v1:172.21.15.116:6789/0 conn(0x7fda4415bcd0 legacy=0x7fda440cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:33.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.344+0000 7fda39e00640 1 -- 172.21.15.47:0/3898790066 >> v1:172.21.15.47:6789/0 conn(0x7fda440d2710 legacy=0x7fda4415d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:33.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.344+0000 7fda39e00640 1 -- 172.21.15.47:0/3898790066 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda440cc9a0 con 0x7fda440cedb0 2024-02-19T15:32:33.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda39e00640 1 --2- 172.21.15.47:0/3898790066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda440cedb0 0x7fda440cfc50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fda2c0097c0 tx=0x7fda2c01c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ba2bcaec39d135 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:33.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda41e00640 1 -- 172.21.15.47:0/3898790066 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c0ad680 con 0x7fda440cedb0 2024-02-19T15:32:33.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda41e00640 1 -- 172.21.15.47:0/3898790066 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2c0ad7e0 con 0x7fda440cedb0 2024-02-19T15:32:33.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda41e00640 1 -- 172.21.15.47:0/3898790066 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c0adb70 con 0x7fda440cedb0 2024-02-19T15:32:33.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda49400640 1 -- 172.21.15.47:0/3898790066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda440cedb0 msgr2=0x7fda440cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:33.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda49400640 1 --2- 172.21.15.47:0/3898790066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda440cedb0 0x7fda440cfc50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fda2c0097c0 tx=0x7fda2c01c470 comp rx=0 tx=0).stop 2024-02-19T15:32:33.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda49400640 1 -- 172.21.15.47:0/3898790066 shutdown_connections 2024-02-19T15:32:33.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda49400640 1 --2- 172.21.15.47:0/3898790066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda440cedb0 0x7fda440cfc50 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:33.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda49400640 1 -- 172.21.15.47:0/3898790066 >> 172.21.15.47:0/3898790066 conn(0x7fda440d1bf0 msgr2=0x7fda440d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:33.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda49400640 1 -- 172.21.15.47:0/3898790066 shutdown_connections 2024-02-19T15:32:33.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda49400640 1 -- 172.21.15.47:0/3898790066 wait complete. 2024-02-19T15:32:33.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda49400640 1 Processor -- start 2024-02-19T15:32:33.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda49400640 1 -- start start 2024-02-19T15:32:33.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda49400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda440cedb0 0x7fda44152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:33.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda49400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda440d2710 0x7fda440c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:33.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda49400640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda4415bcd0 0x7fda440c5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:33.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda49400640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda440cdb90 con 0x7fda440d2710 2024-02-19T15:32:33.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda49400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda441531f0 con 0x7fda440cedb0 2024-02-19T15:32:33.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda49400640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda440c9710 con 0x7fda4415bcd0 2024-02-19T15:32:33.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda39e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda440d2710 0x7fda440c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:33.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda39e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda440d2710 0x7fda440c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55686/0 (socket says 172.21.15.47:55686) 2024-02-19T15:32:33.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda39e00640 1 -- 172.21.15.47:0/2552398373 learned_addr learned my addr 172.21.15.47:0/2552398373 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:33.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.348+0000 7fda42800640 1 --2- 172.21.15.47:0/2552398373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda440cedb0 0x7fda44152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:33.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.352+0000 7fda43200640 1 --2- 172.21.15.47:0/2552398373 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda4415bcd0 0x7fda440c5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:33.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.352+0000 7fda39e00640 1 -- 172.21.15.47:0/2552398373 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda4415bcd0 msgr2=0x7fda440c5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:33.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.352+0000 7fda39e00640 1 --2- 172.21.15.47:0/2552398373 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda4415bcd0 0x7fda440c5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:33.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.352+0000 7fda39e00640 1 -- 172.21.15.47:0/2552398373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda440cedb0 msgr2=0x7fda44152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:33.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.352+0000 7fda39e00640 1 --2- 172.21.15.47:0/2552398373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda440cedb0 0x7fda44152cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:33.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.352+0000 7fda39e00640 1 -- 172.21.15.47:0/2552398373 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2c009420 con 0x7fda440d2710 2024-02-19T15:32:33.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.352+0000 7fda39e00640 1 --2- 172.21.15.47:0/2552398373 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda440d2710 0x7fda440c8cc0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fda2c01a0a0 tx=0x7fda2c01acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:33.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.352+0000 7fda3b400640 1 -- 172.21.15.47:0/2552398373 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c0add60 con 0x7fda440d2710 2024-02-19T15:32:33.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.352+0000 7fda3b400640 1 -- 172.21.15.47:0/2552398373 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2c0b8450 con 0x7fda440d2710 2024-02-19T15:32:33.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.352+0000 7fda3b400640 1 -- 172.21.15.47:0/2552398373 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c0b8800 con 0x7fda440d2710 2024-02-19T15:32:33.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.352+0000 7fda49400640 1 -- 172.21.15.47:0/2552398373 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda4415d240 con 0x7fda440d2710 2024-02-19T15:32:33.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.352+0000 7fda49400640 1 -- 172.21.15.47:0/2552398373 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda440c9cd0 con 0x7fda440d2710 2024-02-19T15:32:33.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.352+0000 7fda49400640 1 -- 172.21.15.47:0/2552398373 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda44075360 con 0x7fda440d2710 2024-02-19T15:32:33.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.352+0000 7fda3b400640 1 -- 172.21.15.47:0/2552398373 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fda2c0b4070 con 0x7fda440d2710 2024-02-19T15:32:33.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.356+0000 7fda3b400640 1 --2- 172.21.15.47:0/2552398373 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fda20066e40 0x7fda20069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:33.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.356+0000 7fda3b400640 1 -- 172.21.15.47:0/2552398373 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fda2c11efa0 con 0x7fda440d2710 2024-02-19T15:32:33.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.356+0000 7fda42800640 1 --2- 172.21.15.47:0/2552398373 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fda20066e40 0x7fda20069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:33.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.356+0000 7fda42800640 1 --2- 172.21.15.47:0/2552398373 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fda20066e40 0x7fda20069300 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fda4415b8d0 tx=0x7fda34009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:33.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.360+0000 7fda3b400640 1 -- 172.21.15.47:0/2552398373 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fda2c0ebe80 con 0x7fda440d2710 2024-02-19T15:32:33.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.576+0000 7fda49400640 1 -- 172.21.15.47:0/2552398373 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda44157b80 con 0x7fda440d2710 2024-02-19T15:32:33.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.576+0000 7fda3b400640 1 -- 172.21.15.47:0/2552398373 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7fda2c0b2d80 con 0x7fda440d2710 2024-02-19T15:32:33.575 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:33.575 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:33.575 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:33.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.576+0000 7fda49400640 1 -- 172.21.15.47:0/2552398373 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fda20066e40 msgr2=0x7fda20069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:33.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.576+0000 7fda49400640 1 --2- 172.21.15.47:0/2552398373 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fda20066e40 0x7fda20069300 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fda4415b8d0 tx=0x7fda34009040 comp rx=0 tx=0).stop 2024-02-19T15:32:33.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.576+0000 7fda49400640 1 -- 172.21.15.47:0/2552398373 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda440d2710 msgr2=0x7fda440c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:33.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.576+0000 7fda49400640 1 --2- 172.21.15.47:0/2552398373 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda440d2710 0x7fda440c8cc0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fda2c01a0a0 tx=0x7fda2c01acb0 comp rx=0 tx=0).stop 2024-02-19T15:32:33.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.576+0000 7fda49400640 1 -- 172.21.15.47:0/2552398373 shutdown_connections 2024-02-19T15:32:33.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.576+0000 7fda49400640 1 --2- 172.21.15.47:0/2552398373 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fda20066e40 0x7fda20069300 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:33.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.576+0000 7fda49400640 1 --2- 172.21.15.47:0/2552398373 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda4415bcd0 0x7fda440c5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:33.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.576+0000 7fda49400640 1 --2- 172.21.15.47:0/2552398373 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda440d2710 0x7fda440c8cc0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:33.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.576+0000 7fda49400640 1 --2- 172.21.15.47:0/2552398373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda440cedb0 0x7fda44152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:33.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.576+0000 7fda49400640 1 -- 172.21.15.47:0/2552398373 >> 172.21.15.47:0/2552398373 conn(0x7fda440d1bf0 msgr2=0x7fda44156b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:33.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.576+0000 7fda49400640 1 -- 172.21.15.47:0/2552398373 shutdown_connections 2024-02-19T15:32:33.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:33.576+0000 7fda49400640 1 -- 172.21.15.47:0/2552398373 wait complete. 2024-02-19T15:32:33.799 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:33.799 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:34.320 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:32:34.800 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:34.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.944+0000 7f9f0c000640 1 Processor -- start 2024-02-19T15:32:34.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.944+0000 7f9f0c000640 1 -- start start 2024-02-19T15:32:34.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.944+0000 7f9f0c000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9f040cef90 0x7f9f040ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:34.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.944+0000 7f9f0c000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f040ce930 0x7f9f040cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:34.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.944+0000 7f9f0c000640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f040d10a0 con 0x7f9f040b2fc0 2024-02-19T15:32:34.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.944+0000 7f9f0c000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f040cedc0 con 0x7f9f040cef90 2024-02-19T15:32:34.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.944+0000 7f9f0c000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f040d0e80 con 0x7f9f040ce930 2024-02-19T15:32:34.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.944+0000 7f9f08c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9f040cef90 0x7f9f040ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:34.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.944+0000 7f9f0a000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f040ce930 0x7f9f040cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:34.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.944+0000 7f9f08c00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9f040cef90 0x7f9f040ce3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:32828/0 (socket says 172.21.15.47:32828) 2024-02-19T15:32:34.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.944+0000 7f9f08c00640 1 -- 172.21.15.47:0/1547194248 learned_addr learned my addr 172.21.15.47:0/1547194248 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:34.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.944+0000 7f9f08c00640 1 -- 172.21.15.47:0/1547194248 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f040ce930 msgr2=0x7f9f040cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:34.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.944+0000 7f9f08c00640 1 --2- 172.21.15.47:0/1547194248 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f040ce930 0x7f9f040cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:34.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.944+0000 7f9f08c00640 1 -- 172.21.15.47:0/1547194248 >> v1:172.21.15.116:6789/0 conn(0x7f9f040b2fc0 legacy=0x7f9f040d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:34.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.944+0000 7f9f08c00640 1 -- 172.21.15.47:0/1547194248 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f040b33a0 con 0x7f9f040cef90 2024-02-19T15:32:34.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.948+0000 7f9f08c00640 1 --2- 172.21.15.47:0/1547194248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f040cef90 0x7f9f040ce3f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0009870 tx=0x7f9ef001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4383003b8d26126 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:34.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.948+0000 7f9f03e00640 1 -- 172.21.15.47:0/1547194248 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef00a6040 con 0x7f9f040cef90 2024-02-19T15:32:34.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.948+0000 7f9f03e00640 1 -- 172.21.15.47:0/1547194248 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef00aa600 con 0x7f9f040cef90 2024-02-19T15:32:34.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.948+0000 7f9f03e00640 1 -- 172.21.15.47:0/1547194248 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef00aa990 con 0x7f9f040cef90 2024-02-19T15:32:34.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.948+0000 7f9f0c000640 1 -- 172.21.15.47:0/1547194248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f040cef90 msgr2=0x7f9f040ce3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:34.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.948+0000 7f9f0c000640 1 --2- 172.21.15.47:0/1547194248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f040cef90 0x7f9f040ce3f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0009870 tx=0x7f9ef001c4a0 comp rx=0 tx=0).stop 2024-02-19T15:32:34.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.948+0000 7f9f0c000640 1 -- 172.21.15.47:0/1547194248 shutdown_connections 2024-02-19T15:32:34.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.948+0000 7f9f0c000640 1 --2- 172.21.15.47:0/1547194248 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f040ce930 0x7f9f040cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:34.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.948+0000 7f9f0c000640 1 --2- 172.21.15.47:0/1547194248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f040cef90 0x7f9f040ce3f0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:34.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.948+0000 7f9f0c000640 1 -- 172.21.15.47:0/1547194248 >> 172.21.15.47:0/1547194248 conn(0x7f9f040d1b70 msgr2=0x7f9f040d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:34.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.948+0000 7f9f0c000640 1 -- 172.21.15.47:0/1547194248 shutdown_connections 2024-02-19T15:32:34.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.948+0000 7f9f0c000640 1 -- 172.21.15.47:0/1547194248 wait complete. 2024-02-19T15:32:34.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f0c000640 1 Processor -- start 2024-02-19T15:32:34.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f0c000640 1 -- start start 2024-02-19T15:32:34.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f0c000640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f040b2fc0 0x7f9f040c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:34.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f0c000640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f040ce930 0x7f9f04160d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:34.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f0c000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f040cef90 0x7f9f0414f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:34.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f0c000640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f04161610 con 0x7f9f040b2fc0 2024-02-19T15:32:34.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f0c000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f04161b40 con 0x7f9f040cef90 2024-02-19T15:32:34.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f0c000640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f040c8300 con 0x7f9f040ce930 2024-02-19T15:32:34.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f09600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f040b2fc0 0x7f9f040c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:34.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f08c00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f040ce930 0x7f9f04160d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:34.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f09600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f040b2fc0 0x7f9f040c58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55694/0 (socket says 172.21.15.47:55694) 2024-02-19T15:32:34.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f0a000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f040cef90 0x7f9f0414f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:34.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f09600640 1 -- 172.21.15.47:0/3607488273 learned_addr learned my addr 172.21.15.47:0/3607488273 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:34.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f08c00640 1 -- 172.21.15.47:0/3607488273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f040cef90 msgr2=0x7f9f0414f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:34.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f08c00640 1 --2- 172.21.15.47:0/3607488273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f040cef90 0x7f9f0414f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:34.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f08c00640 1 -- 172.21.15.47:0/3607488273 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f040b2fc0 msgr2=0x7f9f040c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:34.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f08c00640 1 --2- 172.21.15.47:0/3607488273 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f040b2fc0 0x7f9f040c58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:34.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f08c00640 1 -- 172.21.15.47:0/3607488273 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef00094d0 con 0x7f9f040ce930 2024-02-19T15:32:34.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f09600640 1 --2- 172.21.15.47:0/3607488273 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f040b2fc0 0x7f9f040c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:32:34.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f08c00640 1 --2- 172.21.15.47:0/3607488273 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f040ce930 0x7f9f04160d00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9ef00ab000 tx=0x7f9ef0019240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:34.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f01600640 1 -- 172.21.15.47:0/3607488273 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef00b3800 con 0x7f9f040ce930 2024-02-19T15:32:34.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f01600640 1 -- 172.21.15.47:0/3607488273 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef00b3960 con 0x7f9f040ce930 2024-02-19T15:32:34.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f01600640 1 -- 172.21.15.47:0/3607488273 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef00b3d10 con 0x7f9f040ce930 2024-02-19T15:32:34.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f0c000640 1 -- 172.21.15.47:0/3607488273 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f04161860 con 0x7f9f040ce930 2024-02-19T15:32:34.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.952+0000 7f9f0c000640 1 -- 172.21.15.47:0/3607488273 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f040c7f20 con 0x7f9f040ce930 2024-02-19T15:32:34.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.956+0000 7f9f0c000640 1 -- 172.21.15.47:0/3607488273 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f040b0500 con 0x7f9f040ce930 2024-02-19T15:32:34.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.956+0000 7f9f01600640 1 -- 172.21.15.47:0/3607488273 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f9ef00c6440 con 0x7f9f040ce930 2024-02-19T15:32:34.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.956+0000 7f9f01600640 1 --2- 172.21.15.47:0/3607488273 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9ee4066e40 0x7f9ee4069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:34.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.956+0000 7f9f01600640 1 -- 172.21.15.47:0/3607488273 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9ef011fa20 con 0x7f9f040ce930 2024-02-19T15:32:34.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.956+0000 7f9f09600640 1 --2- 172.21.15.47:0/3607488273 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9ee4066e40 0x7f9ee4069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:34.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.956+0000 7f9f09600640 1 --2- 172.21.15.47:0/3607488273 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9ee4066e40 0x7f9ee4069300 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8001c00 tx=0x7f9ef8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:34.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:34.960+0000 7f9f01600640 1 -- 172.21.15.47:0/3607488273 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f9ef00ec8f0 con 0x7f9f040ce930 2024-02-19T15:32:35.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:35.156+0000 7f9f0c000640 1 -- 172.21.15.47:0/3607488273 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f04075360 con 0x7f9f040ce930 2024-02-19T15:32:35.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:35.156+0000 7f9f01600640 1 -- 172.21.15.47:0/3607488273 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f9ef00effb0 con 0x7f9f040ce930 2024-02-19T15:32:35.156 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:35.156 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:35.156 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:35.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:35.160+0000 7f9f0c000640 1 -- 172.21.15.47:0/3607488273 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9ee4066e40 msgr2=0x7f9ee4069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:35.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:35.160+0000 7f9f0c000640 1 --2- 172.21.15.47:0/3607488273 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9ee4066e40 0x7f9ee4069300 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8001c00 tx=0x7f9ef8009040 comp rx=0 tx=0).stop 2024-02-19T15:32:35.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:35.160+0000 7f9f0c000640 1 -- 172.21.15.47:0/3607488273 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f040ce930 msgr2=0x7f9f04160d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:35.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:35.160+0000 7f9f0c000640 1 --2- 172.21.15.47:0/3607488273 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f040ce930 0x7f9f04160d00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9ef00ab000 tx=0x7f9ef0019240 comp rx=0 tx=0).stop 2024-02-19T15:32:35.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:35.160+0000 7f9f0c000640 1 -- 172.21.15.47:0/3607488273 shutdown_connections 2024-02-19T15:32:35.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:35.160+0000 7f9f0c000640 1 --2- 172.21.15.47:0/3607488273 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9ee4066e40 0x7f9ee4069300 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:35.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:35.160+0000 7f9f0c000640 1 --2- 172.21.15.47:0/3607488273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f040cef90 0x7f9f0414f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:35.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:35.160+0000 7f9f0c000640 1 --2- 172.21.15.47:0/3607488273 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f040ce930 0x7f9f04160d00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:35.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:35.160+0000 7f9f0c000640 1 --2- 172.21.15.47:0/3607488273 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f040b2fc0 0x7f9f040c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:35.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:35.160+0000 7f9f0c000640 1 -- 172.21.15.47:0/3607488273 >> 172.21.15.47:0/3607488273 conn(0x7f9f040d1b70 msgr2=0x7f9f040d1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:35.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:35.160+0000 7f9f0c000640 1 -- 172.21.15.47:0/3607488273 shutdown_connections 2024-02-19T15:32:35.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:35.160+0000 7f9f0c000640 1 -- 172.21.15.47:0/3607488273 wait complete. 2024-02-19T15:32:35.379 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:35.379 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:36.321 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:32:36.380 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:32:36.431 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:32:36.435 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:32:36.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.524+0000 7f905b600640 1 Processor -- start 2024-02-19T15:32:36.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.524+0000 7f905b600640 1 -- start start 2024-02-19T15:32:36.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.524+0000 7f905b600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f905415f620 0x7f90540b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:36.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.524+0000 7f905b600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90540b2330 0x7f905406ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:36.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.524+0000 7f905b600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f905406b5d0 con 0x7f90541614e0 2024-02-19T15:32:36.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.524+0000 7f905b600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90540b1c00 con 0x7f905415f620 2024-02-19T15:32:36.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.524+0000 7f905b600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90540b27e0 con 0x7f90540b2330 2024-02-19T15:32:36.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.524+0000 7f9058c00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f90541614e0 0x7f905415eee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60602/0 (socket says 172.21.15.47:60602) 2024-02-19T15:32:36.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.524+0000 7f9058c00640 1 -- 172.21.15.47:0/2017624279 learned_addr learned my addr 172.21.15.47:0/2017624279 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:32:36.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.524+0000 7f9059600640 1 --2- 172.21.15.47:0/2017624279 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90540b2330 0x7f905406ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:36.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.524+0000 7f9053e00640 1 --2- 172.21.15.47:0/2017624279 >> v2:172.21.15.116:3300/0 conn(0x7f905415f620 0x7f90540b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:36.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.524+0000 7f9053400640 1 -- 172.21.15.47:0/2017624279 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066932378 0 0) 0x7f905406b5d0 con 0x7f90541614e0 2024-02-19T15:32:36.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.524+0000 7f9053400640 1 -- 172.21.15.47:0/2017624279 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f903c003530 con 0x7f90541614e0 2024-02-19T15:32:36.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.524+0000 7f9053400640 1 -- 172.21.15.47:0/2017624279 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3458107874 0 0) 0x7f903c003530 con 0x7f90541614e0 2024-02-19T15:32:36.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.524+0000 7f9059600640 1 -- 172.21.15.47:0/2017624279 >> v2:172.21.15.116:3300/0 conn(0x7f905415f620 msgr2=0x7f90540b1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:36.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.524+0000 7f9059600640 1 --2- 172.21.15.47:0/2017624279 >> v2:172.21.15.116:3300/0 conn(0x7f905415f620 0x7f90540b1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:36.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.524+0000 7f9059600640 1 -- 172.21.15.47:0/2017624279 >> v1:172.21.15.47:6789/0 conn(0x7f90541614e0 legacy=0x7f905415eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:36.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.524+0000 7f9059600640 1 -- 172.21.15.47:0/2017624279 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90540b1640 con 0x7f90540b2330 2024-02-19T15:32:36.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.528+0000 7f9059600640 1 --2- 172.21.15.47:0/2017624279 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90540b2330 0x7f905406ba60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f904c009d90 tx=0x7f904c01c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de5f201a74f04bbf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:36.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.528+0000 7f9053400640 1 -- 172.21.15.47:0/2017624279 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9048001680 con 0x7f90540b2330 2024-02-19T15:32:36.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.528+0000 7f9053400640 1 -- 172.21.15.47:0/2017624279 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f904c0a55a0 con 0x7f90540b2330 2024-02-19T15:32:36.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.528+0000 7f9053400640 1 -- 172.21.15.47:0/2017624279 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904c0a5910 con 0x7f90540b2330 2024-02-19T15:32:36.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.528+0000 7f905b600640 1 -- 172.21.15.47:0/2017624279 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90540b2330 msgr2=0x7f905406ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:36.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.528+0000 7f905b600640 1 --2- 172.21.15.47:0/2017624279 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90540b2330 0x7f905406ba60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f904c009d90 tx=0x7f904c01c890 comp rx=0 tx=0).stop 2024-02-19T15:32:36.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.528+0000 7f905b600640 1 -- 172.21.15.47:0/2017624279 shutdown_connections 2024-02-19T15:32:36.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.528+0000 7f905b600640 1 --2- 172.21.15.47:0/2017624279 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90540b2330 0x7f905406ba60 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:36.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.528+0000 7f905b600640 1 --2- 172.21.15.47:0/2017624279 >> v2:172.21.15.116:3300/0 conn(0x7f905415f620 0x7f90540b1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:36.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.528+0000 7f905b600640 1 -- 172.21.15.47:0/2017624279 >> 172.21.15.47:0/2017624279 conn(0x7f90540b4fc0 msgr2=0x7f90540b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:36.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.528+0000 7f905b600640 1 -- 172.21.15.47:0/2017624279 shutdown_connections 2024-02-19T15:32:36.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.528+0000 7f905b600640 1 -- 172.21.15.47:0/2017624279 wait complete. 2024-02-19T15:32:36.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f905b600640 1 Processor -- start 2024-02-19T15:32:36.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f905b600640 1 -- start start 2024-02-19T15:32:36.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f905b600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90540b2330 0x7f90541580e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:36.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f905b600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f905415f620 0x7f90540d0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:36.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f905b600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90541614e0 0x7f90541605b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:36.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f905b600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90540cee40 con 0x7f90541614e0 2024-02-19T15:32:36.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f905b600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90540ce270 con 0x7f90540b2330 2024-02-19T15:32:36.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f905b600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90540ce510 con 0x7f905415f620 2024-02-19T15:32:36.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f9059600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90541614e0 0x7f90541605b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:36.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f9058c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90540b2330 0x7f90541580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:36.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f9059600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90541614e0 0x7f90541605b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55704/0 (socket says 172.21.15.47:55704) 2024-02-19T15:32:36.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f9058c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90540b2330 0x7f90541580e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.47:32850/0 (socket says 172.21.15.47:32850) 2024-02-19T15:32:36.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f9059600640 1 -- 172.21.15.47:0/3964820453 learned_addr learned my addr 172.21.15.47:0/3964820453 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:36.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f9053e00640 1 --2- 172.21.15.47:0/3964820453 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f905415f620 0x7f90540d0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:36.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f9059600640 1 -- 172.21.15.47:0/3964820453 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f905415f620 msgr2=0x7f90540d0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:36.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f9059600640 1 --2- 172.21.15.47:0/3964820453 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f905415f620 0x7f90540d0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:36.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f9059600640 1 -- 172.21.15.47:0/3964820453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90540b2330 msgr2=0x7f90541580e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:36.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f9059600640 1 --2- 172.21.15.47:0/3964820453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90540b2330 0x7f90541580e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:36.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f9059600640 1 -- 172.21.15.47:0/3964820453 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904c009a70 con 0x7f90541614e0 2024-02-19T15:32:36.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f9059600640 1 --2- 172.21.15.47:0/3964820453 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90541614e0 0x7f90541605b0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f904c0b0040 tx=0x7f904c019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:36.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f9050c00640 1 -- 172.21.15.47:0/3964820453 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904c0ad850 con 0x7f90541614e0 2024-02-19T15:32:36.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f9050c00640 1 -- 172.21.15.47:0/3964820453 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f904c0ad9b0 con 0x7f90541614e0 2024-02-19T15:32:36.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f905b600640 1 -- 172.21.15.47:0/3964820453 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90540ce7f0 con 0x7f90541614e0 2024-02-19T15:32:36.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f9050c00640 1 -- 172.21.15.47:0/3964820453 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904c0add80 con 0x7f90541614e0 2024-02-19T15:32:36.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.532+0000 7f905b600640 1 -- 172.21.15.47:0/3964820453 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9054158620 con 0x7f90541614e0 2024-02-19T15:32:36.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.536+0000 7f905b600640 1 -- 172.21.15.47:0/3964820453 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90540cefb0 con 0x7f90541614e0 2024-02-19T15:32:36.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.536+0000 7f9050c00640 1 -- 172.21.15.47:0/3964820453 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f904c0b4050 con 0x7f90541614e0 2024-02-19T15:32:36.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.540+0000 7f9050c00640 1 --2- 172.21.15.47:0/3964820453 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9034066e70 0x7f9034069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:36.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.540+0000 7f9050c00640 1 -- 172.21.15.47:0/3964820453 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f904c11f830 con 0x7f90541614e0 2024-02-19T15:32:36.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.540+0000 7f9058c00640 1 --2- 172.21.15.47:0/3964820453 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9034066e70 0x7f9034069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:36.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.540+0000 7f9058c00640 1 --2- 172.21.15.47:0/3964820453 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9034066e70 0x7f9034069330 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9048001ec0 tx=0x7f90480016e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:36.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.540+0000 7f9050c00640 1 -- 172.21.15.47:0/3964820453 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f904c0ec730 con 0x7f90541614e0 2024-02-19T15:32:36.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.736+0000 7f905b600640 1 -- 172.21.15.47:0/3964820453 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9054075360 con 0x7f90541614e0 2024-02-19T15:32:36.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.736+0000 7f9050c00640 1 -- 172.21.15.47:0/3964820453 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3428 (secure 0 0 0) 0x7f904c0b2d80 con 0x7f90541614e0 2024-02-19T15:32:36.736 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:36.736 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:28:54.409299+0000","modified":"2024-02-19T15:31:25.167152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8361,"mds_1":7037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7037":{"gid":7037,"name":"b","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.116:6837/4110008874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":4110008874},{"type":"v1","addr":"172.21.15.116:6837","nonce":4110008874}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:29:54.888176+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":3,"addr":"172.21.15.47:6835/4036766514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6833","nonce":4036766514},{"type":"v1","addr":"172.21.15.47:6835","nonce":4036766514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T15:32:36.737 INFO:teuthology.orchestra.run.smithi047.stderr:dumped fsmap epoch 100 2024-02-19T15:32:36.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.740+0000 7f905b600640 1 -- 172.21.15.47:0/3964820453 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9034066e70 msgr2=0x7f9034069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:36.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.740+0000 7f905b600640 1 --2- 172.21.15.47:0/3964820453 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9034066e70 0x7f9034069330 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9048001ec0 tx=0x7f90480016e0 comp rx=0 tx=0).stop 2024-02-19T15:32:36.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.740+0000 7f905b600640 1 -- 172.21.15.47:0/3964820453 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90541614e0 msgr2=0x7f90541605b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:36.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.740+0000 7f905b600640 1 --2- 172.21.15.47:0/3964820453 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90541614e0 0x7f90541605b0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f904c0b0040 tx=0x7f904c019630 comp rx=0 tx=0).stop 2024-02-19T15:32:36.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.740+0000 7f905b600640 1 -- 172.21.15.47:0/3964820453 shutdown_connections 2024-02-19T15:32:36.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.740+0000 7f905b600640 1 --2- 172.21.15.47:0/3964820453 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f9034066e70 0x7f9034069330 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:36.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.740+0000 7f905b600640 1 --2- 172.21.15.47:0/3964820453 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90541614e0 0x7f90541605b0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:36.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.740+0000 7f905b600640 1 --2- 172.21.15.47:0/3964820453 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f905415f620 0x7f90540d0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:36.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.740+0000 7f905b600640 1 --2- 172.21.15.47:0/3964820453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90540b2330 0x7f90541580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:36.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.740+0000 7f905b600640 1 -- 172.21.15.47:0/3964820453 >> 172.21.15.47:0/3964820453 conn(0x7f90540b4fc0 msgr2=0x7f905415e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:36.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.740+0000 7f905b600640 1 -- 172.21.15.47:0/3964820453 shutdown_connections 2024-02-19T15:32:36.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:36.740+0000 7f905b600640 1 -- 172.21.15.47:0/3964820453 wait complete. 2024-02-19T15:32:36.959 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:28:54.409299+0000', 'modified': '2024-02-19T15:31:25.167152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8361, 'mds_1': 7037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7037': {'gid': 7037, 'name': 'b', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.116:6837/4110008874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6835', 'nonce': 4110008874}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 4110008874}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:29:54.888176+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.47:6835/4036766514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6833', 'nonce': 4036766514}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4036766514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:32:36.959 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7037:up:stopping 2024-02-19T15:32:37.960 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 100, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T15:28:54.409299+0000", "damaged": [], "data_pools": [ 19 ], "enabled": true, "epoch": 100, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7037": { "addr": "172.21.15.116:6837/4110008874", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6835", "nonce": 4110008874, "type": "v2" }, { "addr": "172.21.15.116:6837", "nonce": 4110008874, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7037, "incarnation": 93, "join_fscid": -1, "laggy_since": "2024-02-19T15:29:54.888176+0000", "name": "b", "rank": 1, "state": "up:stopping", "state_seq": 5 }, "gid_8361": { "addr": "172.21.15.47:6835/4036766514", "addrs": { "addrvec": [ { "addr": "172.21.15.47:6833", "nonce": 4036766514, "type": "v2" }, { "addr": "172.21.15.47:6835", "nonce": 4036766514, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8361, "incarnation": 87, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 18, "modified": "2024-02-19T15:31:25.167152+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8361, "mds_1": 7037 } } } ], "standbys": [] } 2024-02-19T15:32:37.961 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T15:32:37.961 DEBUG:teuthology.orchestra.run.smithi047:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:32:38.048 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:32:38.118 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T15:32:38.119 INFO:tasks.cephfs.mount:Terminating background process 2024-02-19T15:32:38.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-02-19T15:32:38.323 INFO:teuthology.orchestra.run.smithi047.stderr:open_dir_background: keeping dir open 2024-02-19T15:32:38.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-02-19T15:32:38.334 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:32:38.334 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:32:38.385 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-02-19T15:32:38.385 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:32:38.437 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:32:38.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.616+0000 7f478ec00640 1 Processor -- start 2024-02-19T15:32:38.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.620+0000 7f478ec00640 1 -- start start 2024-02-19T15:32:38.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.620+0000 7f478ec00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f478815f590 0x7f47880d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:38.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.620+0000 7f478ec00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f47880b0a80 0x7f47880b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:38.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.620+0000 7f478ec00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47880b1a10 0x7f47880ae860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:38.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.620+0000 7f478ec00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47880cc530 con 0x7f47880b0a80 2024-02-19T15:32:38.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.620+0000 7f478ec00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47880b0280 con 0x7f478815f590 2024-02-19T15:32:38.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.620+0000 7f478ec00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47880b1e40 con 0x7f47880b1a10 2024-02-19T15:32:38.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.620+0000 7f477fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f47880b0a80 0x7f47880b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:38.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.620+0000 7f477fe00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f47880b0a80 0x7f47880b14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55714/0 (socket says 172.21.15.47:55714) 2024-02-19T15:32:38.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.620+0000 7f477fe00640 1 -- 172.21.15.47:0/1520553756 learned_addr learned my addr 172.21.15.47:0/1520553756 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:38.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.620+0000 7f478cc00640 1 --2- 172.21.15.47:0/1520553756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47880b1a10 0x7f47880ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:38.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.620+0000 7f4787e00640 1 --2- 172.21.15.47:0/1520553756 >> v2:172.21.15.116:3300/0 conn(0x7f478815f590 0x7f47880d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:38.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.620+0000 7f477fe00640 1 -- 172.21.15.47:0/1520553756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47880b1a10 msgr2=0x7f47880ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:38.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.620+0000 7f477fe00640 1 --2- 172.21.15.47:0/1520553756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47880b1a10 0x7f47880ae860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:38.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.620+0000 7f477fe00640 1 -- 172.21.15.47:0/1520553756 >> v2:172.21.15.116:3300/0 conn(0x7f478815f590 msgr2=0x7f47880d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:38.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.620+0000 7f477fe00640 1 --2- 172.21.15.47:0/1520553756 >> v2:172.21.15.116:3300/0 conn(0x7f478815f590 0x7f47880d2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:38.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.620+0000 7f477fe00640 1 -- 172.21.15.47:0/1520553756 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47880b04d0 con 0x7f47880b0a80 2024-02-19T15:32:38.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.624+0000 7f477fe00640 1 --2- 172.21.15.47:0/1520553756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47880b0a80 0x7f47880b14d0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f478806be80 tx=0x7f477001c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75329effc800b676 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:38.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.624+0000 7f4787400640 1 -- 172.21.15.47:0/1520553756 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47700ab600 con 0x7f47880b0a80 2024-02-19T15:32:38.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.624+0000 7f4787400640 1 -- 172.21.15.47:0/1520553756 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47700a7050 con 0x7f47880b0a80 2024-02-19T15:32:38.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.624+0000 7f4787400640 1 -- 172.21.15.47:0/1520553756 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47700ab980 con 0x7f47880b0a80 2024-02-19T15:32:38.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.624+0000 7f478ec00640 1 -- 172.21.15.47:0/1520553756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47880b0a80 msgr2=0x7f47880b14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:38.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.624+0000 7f478ec00640 1 --2- 172.21.15.47:0/1520553756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47880b0a80 0x7f47880b14d0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f478806be80 tx=0x7f477001c230 comp rx=0 tx=0).stop 2024-02-19T15:32:38.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.624+0000 7f478ec00640 1 -- 172.21.15.47:0/1520553756 shutdown_connections 2024-02-19T15:32:38.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.624+0000 7f478ec00640 1 --2- 172.21.15.47:0/1520553756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47880b1a10 0x7f47880ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:38.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.624+0000 7f478ec00640 1 --2- 172.21.15.47:0/1520553756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47880b0a80 0x7f47880b14d0 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:38.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.624+0000 7f478ec00640 1 --2- 172.21.15.47:0/1520553756 >> v2:172.21.15.116:3300/0 conn(0x7f478815f590 0x7f47880d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:38.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.624+0000 7f478ec00640 1 -- 172.21.15.47:0/1520553756 >> 172.21.15.47:0/1520553756 conn(0x7f47880d1b70 msgr2=0x7f47880d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:38.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.624+0000 7f478ec00640 1 -- 172.21.15.47:0/1520553756 shutdown_connections 2024-02-19T15:32:38.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.624+0000 7f478ec00640 1 -- 172.21.15.47:0/1520553756 wait complete. 2024-02-19T15:32:38.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.624+0000 7f478ec00640 1 Processor -- start 2024-02-19T15:32:38.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.624+0000 7f478ec00640 1 -- start start 2024-02-19T15:32:38.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f478ec00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47880b0a80 0x7f47881f6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:38.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f478ec00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47880b1a10 0x7f478815bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:38.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f478ec00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f478815f590 0x7f47880c9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:38.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f478ec00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47881f6ae0 con 0x7f47880b0a80 2024-02-19T15:32:38.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f478ec00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47880ca320 con 0x7f478815f590 2024-02-19T15:32:38.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f478ec00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47881f6480 con 0x7f47880b1a10 2024-02-19T15:32:38.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f4787e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47880b0a80 0x7f47881f6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:38.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f4787e00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47880b0a80 0x7f47881f6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55728/0 (socket says 172.21.15.47:55728) 2024-02-19T15:32:38.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f4787e00640 1 -- 172.21.15.47:0/3984190248 learned_addr learned my addr 172.21.15.47:0/3984190248 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:38.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f478cc00640 1 --2- 172.21.15.47:0/3984190248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f478815f590 0x7f47880c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:38.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f477fe00640 1 --2- 172.21.15.47:0/3984190248 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47880b1a10 0x7f478815bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:38.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f4787e00640 1 -- 172.21.15.47:0/3984190248 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47880b1a10 msgr2=0x7f478815bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:38.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f4787e00640 1 --2- 172.21.15.47:0/3984190248 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47880b1a10 0x7f478815bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:38.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f4787e00640 1 -- 172.21.15.47:0/3984190248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f478815f590 msgr2=0x7f47880c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:38.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f4787e00640 1 --2- 172.21.15.47:0/3984190248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f478815f590 0x7f47880c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:38.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f4787e00640 1 -- 172.21.15.47:0/3984190248 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47700094d0 con 0x7f47880b0a80 2024-02-19T15:32:38.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f4787e00640 1 --2- 172.21.15.47:0/3984190248 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47880b0a80 0x7f47881f6f00 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f477800e870 tx=0x7f477800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:38.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f4784c00640 1 -- 172.21.15.47:0/3984190248 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4778009510 con 0x7f47880b0a80 2024-02-19T15:32:38.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f4784c00640 1 -- 172.21.15.47:0/3984190248 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477800eec0 con 0x7f47880b0a80 2024-02-19T15:32:38.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f478ec00640 1 -- 172.21.15.47:0/3984190248 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47880c9d60 con 0x7f47880b0a80 2024-02-19T15:32:38.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f4784c00640 1 -- 172.21.15.47:0/3984190248 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477800cb40 con 0x7f47880b0a80 2024-02-19T15:32:38.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f478ec00640 1 -- 172.21.15.47:0/3984190248 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47881f6700 con 0x7f47880b0a80 2024-02-19T15:32:38.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.628+0000 7f478ec00640 1 -- 172.21.15.47:0/3984190248 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4788075360 con 0x7f47880b0a80 2024-02-19T15:32:38.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.632+0000 7f4784c00640 1 -- 172.21.15.47:0/3984190248 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f477800cca0 con 0x7f47880b0a80 2024-02-19T15:32:38.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.632+0000 7f4784c00640 1 --2- 172.21.15.47:0/3984190248 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4764066e40 0x7f4764069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:38.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.632+0000 7f4784c00640 1 -- 172.21.15.47:0/3984190248 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4778084ba0 con 0x7f47880b0a80 2024-02-19T15:32:38.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.632+0000 7f477fe00640 1 --2- 172.21.15.47:0/3984190248 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4764066e40 0x7f4764069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:38.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.632+0000 7f477fe00640 1 --2- 172.21.15.47:0/3984190248 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4764066e40 0x7f4764069300 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f47700ad000 tx=0x7f4770008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:38.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.636+0000 7f4784c00640 1 -- 172.21.15.47:0/3984190248 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f4778051960 con 0x7f47880b0a80 2024-02-19T15:32:38.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.824+0000 7f478ec00640 1 -- 172.21.15.47:0/3984190248 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4788074890 con 0x7f47880b0a80 2024-02-19T15:32:38.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.824+0000 7f4784c00640 1 -- 172.21.15.47:0/3984190248 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16733 (secure 0 0 0) 0x7f4778017090 con 0x7f47880b0a80 2024-02-19T15:32:38.825 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-02-19T15:32:38.825 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":129,"fsid":"ffb36f1b-9699-4f22-aca2-c5ca3bc53f7e","created":"2024-02-19T15:21:40.035369+0000","modified":"2024-02-19T15:29:48.179901+0000","last_up_change":"2024-02-19T15:21:51.710216+0000","last_in_change":"2024-02-19T15:21:42.552244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T15:21:51.209941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:28:52.814262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-02-19T15:28:53.381287+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"98fb351e-d895-42c6-8746-924f0c4d0776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6809","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6811","nonce":1968337531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6814","nonce":1968337531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6821","nonce":1968337531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":1968337531},{"type":"v1","addr":"172.21.15.47:6818","nonce":1968337531}]},"public_addr":"172.21.15.47:6811/1968337531","cluster_addr":"172.21.15.47:6814/1968337531","heartbeat_back_addr":"172.21.15.47:6821/1968337531","heartbeat_front_addr":"172.21.15.47:6818/1968337531","state":["exists","up"]},{"osd":1,"uuid":"5ffa7095-5c4c-4975-b8c7-a3d9c6613018","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6801","nonce":1591346632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6803","nonce":1591346632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6807","nonce":1591346632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":1591346632},{"type":"v1","addr":"172.21.15.47:6805","nonce":1591346632}]},"public_addr":"172.21.15.47:6801/1591346632","cluster_addr":"172.21.15.47:6803/1591346632","heartbeat_back_addr":"172.21.15.47:6807/1591346632","heartbeat_front_addr":"172.21.15.47:6805/1591346632","state":["exists","up"]},{"osd":2,"uuid":"dd82bd48-437f-45d7-9633-c8afd0ace368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6810","nonce":2881016873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6813","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6815","nonce":2881016873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6823","nonce":2881016873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6817","nonce":2881016873},{"type":"v1","addr":"172.21.15.47:6819","nonce":2881016873}]},"public_addr":"172.21.15.47:6810/2881016873","cluster_addr":"172.21.15.47:6815/2881016873","heartbeat_back_addr":"172.21.15.47:6823/2881016873","heartbeat_front_addr":"172.21.15.47:6819/2881016873","state":["exists","up"]},{"osd":3,"uuid":"07f72bca-9e3d-4b97-8a28-f7c711c1620a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6825","nonce":4149334728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6827","nonce":4149334728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6831","nonce":4149334728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":4149334728},{"type":"v1","addr":"172.21.15.47:6829","nonce":4149334728}]},"public_addr":"172.21.15.47:6825/4149334728","cluster_addr":"172.21.15.47:6827/4149334728","heartbeat_back_addr":"172.21.15.47:6831/4149334728","heartbeat_front_addr":"172.21.15.47:6829/4149334728","state":["exists","up"]},{"osd":4,"uuid":"9e9b38e5-8b24-4be2-b072-d46ae037c3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6812","nonce":3366344088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6816","nonce":3366344088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6823","nonce":3366344088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3366344088},{"type":"v1","addr":"172.21.15.116:6820","nonce":3366344088}]},"public_addr":"172.21.15.116:6812/3366344088","cluster_addr":"172.21.15.116:6816/3366344088","heartbeat_back_addr":"172.21.15.116:6823/3366344088","heartbeat_front_addr":"172.21.15.116:6820/3366344088","state":["exists","up"]},{"osd":5,"uuid":"f1585559-4b9e-4272-85d4-b8a5ecb0c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6810","nonce":3478542733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6813","nonce":3478542733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6819","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6821","nonce":3478542733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6815","nonce":3478542733},{"type":"v1","addr":"172.21.15.116:6817","nonce":3478542733}]},"public_addr":"172.21.15.116:6810/3478542733","cluster_addr":"172.21.15.116:6813/3478542733","heartbeat_back_addr":"172.21.15.116:6821/3478542733","heartbeat_front_addr":"172.21.15.116:6817/3478542733","state":["exists","up"]},{"osd":6,"uuid":"10295c58-e6c4-494b-bc8f-5c6d085d7fc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6825","nonce":2921645124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6827","nonce":2921645124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6831","nonce":2921645124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2921645124},{"type":"v1","addr":"172.21.15.116:6829","nonce":2921645124}]},"public_addr":"172.21.15.116:6825/2921645124","cluster_addr":"172.21.15.116:6827/2921645124","heartbeat_back_addr":"172.21.15.116:6831/2921645124","heartbeat_front_addr":"172.21.15.116:6829/2921645124","state":["exists","up"]},{"osd":7,"uuid":"a472a117-2914-48bb-a15d-e07c9349d453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6801","nonce":3212743377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6803","nonce":3212743377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6807","nonce":3212743377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3212743377},{"type":"v1","addr":"172.21.15.116:6805","nonce":3212743377}]},"public_addr":"172.21.15.116:6801/3212743377","cluster_addr":"172.21.15.116:6803/3212743377","heartbeat_back_addr":"172.21.15.116:6807/3212743377","heartbeat_front_addr":"172.21.15.116:6805/3212743377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.257719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.049141+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.168619+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.138353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.116489+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.156674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.212305+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T15:21:48.015427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.22","mappings":[{"from":2,"to":0}]},{"pgid":"19.26","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:32:38.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.828+0000 7f478ec00640 1 -- 172.21.15.47:0/3984190248 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4764066e40 msgr2=0x7f4764069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:38.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.828+0000 7f478ec00640 1 --2- 172.21.15.47:0/3984190248 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4764066e40 0x7f4764069300 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f47700ad000 tx=0x7f4770008040 comp rx=0 tx=0).stop 2024-02-19T15:32:38.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.828+0000 7f478ec00640 1 -- 172.21.15.47:0/3984190248 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47880b0a80 msgr2=0x7f47881f6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:38.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.828+0000 7f478ec00640 1 --2- 172.21.15.47:0/3984190248 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47880b0a80 0x7f47881f6f00 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f477800e870 tx=0x7f477800ec70 comp rx=0 tx=0).stop 2024-02-19T15:32:38.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.828+0000 7f478ec00640 1 -- 172.21.15.47:0/3984190248 shutdown_connections 2024-02-19T15:32:38.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.828+0000 7f478ec00640 1 --2- 172.21.15.47:0/3984190248 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f4764066e40 0x7f4764069300 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:38.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.828+0000 7f478ec00640 1 --2- 172.21.15.47:0/3984190248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f478815f590 0x7f47880c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:38.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.828+0000 7f478ec00640 1 --2- 172.21.15.47:0/3984190248 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47880b1a10 0x7f478815bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:38.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.828+0000 7f478ec00640 1 --2- 172.21.15.47:0/3984190248 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47880b0a80 0x7f47881f6f00 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:38.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.828+0000 7f478ec00640 1 -- 172.21.15.47:0/3984190248 >> 172.21.15.47:0/3984190248 conn(0x7f47880d1b70 msgr2=0x7f478806d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:38.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.828+0000 7f478ec00640 1 -- 172.21.15.47:0/3984190248 shutdown_connections 2024-02-19T15:32:38.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.828+0000 7f478ec00640 1 -- 172.21.15.47:0/3984190248 wait complete. 2024-02-19T15:32:38.846 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T15:21:51.209941+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:28:52.814262+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '124', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:28:53.381287+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:32:38.847 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:32:38.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.992+0000 7f6cbf200640 1 Processor -- start 2024-02-19T15:32:38.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.992+0000 7f6cbf200640 1 -- start start 2024-02-19T15:32:38.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.992+0000 7f6cbf200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cc015e160 0x7f6cc0066cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:38.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.992+0000 7f6cbf200640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6cc00998c0 0x7f6cc00b0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:38.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.992+0000 7f6cbf200640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6cc00b0e50 0x7f6cc00ca690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:38.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.992+0000 7f6cbf200640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cc00afaa0 con 0x7f6cc00998c0 2024-02-19T15:32:38.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.992+0000 7f6cbf200640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cc00cd2d0 con 0x7f6cc00b0e50 2024-02-19T15:32:38.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.992+0000 7f6cbf200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc00cd520 con 0x7f6cc015e160 2024-02-19T15:32:38.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.992+0000 7f6cbd400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6cc00998c0 0x7f6cc00b0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:38.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.992+0000 7f6cbd400640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6cc00998c0 0x7f6cc00b0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55736/0 (socket says 172.21.15.47:55736) 2024-02-19T15:32:38.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.992+0000 7f6cbd400640 1 -- 172.21.15.47:0/1002778547 learned_addr learned my addr 172.21.15.47:0/1002778547 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:38.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.992+0000 7f6cbde00640 1 --2- 172.21.15.47:0/1002778547 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cc015e160 0x7f6cc0066cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:38.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.992+0000 7f6cbe800640 1 --2- 172.21.15.47:0/1002778547 >> v2:172.21.15.116:3300/0 conn(0x7f6cc00b0e50 0x7f6cc00ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:38.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.992+0000 7f6cbd400640 1 -- 172.21.15.47:0/1002778547 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cc015e160 msgr2=0x7f6cc0066cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:38.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.992+0000 7f6cbd400640 1 --2- 172.21.15.47:0/1002778547 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cc015e160 0x7f6cc0066cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:38.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.992+0000 7f6cbd400640 1 -- 172.21.15.47:0/1002778547 >> v2:172.21.15.116:3300/0 conn(0x7f6cc00b0e50 msgr2=0x7f6cc00ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:38.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.992+0000 7f6cbd400640 1 --2- 172.21.15.47:0/1002778547 >> v2:172.21.15.116:3300/0 conn(0x7f6cc00b0e50 0x7f6cc00ca690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:38.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.992+0000 7f6cbd400640 1 -- 172.21.15.47:0/1002778547 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc00afe30 con 0x7f6cc00998c0 2024-02-19T15:32:38.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.996+0000 7f6cbd400640 1 --2- 172.21.15.47:0/1002778547 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cc00998c0 0x7f6cc00b0910 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0099d00 tx=0x7f6cac01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b15ee104d2a022d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:38.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.996+0000 7f6cbca00640 1 -- 172.21.15.47:0/1002778547 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cac0a89d0 con 0x7f6cc00998c0 2024-02-19T15:32:38.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.996+0000 7f6cbca00640 1 -- 172.21.15.47:0/1002778547 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cac0a8b30 con 0x7f6cc00998c0 2024-02-19T15:32:38.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.996+0000 7f6cbca00640 1 -- 172.21.15.47:0/1002778547 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cac0b0440 con 0x7f6cc00998c0 2024-02-19T15:32:38.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.996+0000 7f6cbf200640 1 -- 172.21.15.47:0/1002778547 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cc00998c0 msgr2=0x7f6cc00b0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:38.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.996+0000 7f6cbf200640 1 --2- 172.21.15.47:0/1002778547 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cc00998c0 0x7f6cc00b0910 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0099d00 tx=0x7f6cac01ed10 comp rx=0 tx=0).stop 2024-02-19T15:32:38.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.996+0000 7f6cbf200640 1 -- 172.21.15.47:0/1002778547 shutdown_connections 2024-02-19T15:32:38.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.996+0000 7f6cbf200640 1 --2- 172.21.15.47:0/1002778547 >> v2:172.21.15.116:3300/0 conn(0x7f6cc00b0e50 0x7f6cc00ca690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:38.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.996+0000 7f6cbf200640 1 --2- 172.21.15.47:0/1002778547 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cc00998c0 0x7f6cc00b0910 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:38.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.996+0000 7f6cbf200640 1 --2- 172.21.15.47:0/1002778547 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cc015e160 0x7f6cc0066cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:38.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.996+0000 7f6cbf200640 1 -- 172.21.15.47:0/1002778547 >> 172.21.15.47:0/1002778547 conn(0x7f6cc0072380 msgr2=0x7f6cc0072790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:38.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.996+0000 7f6cbf200640 1 -- 172.21.15.47:0/1002778547 shutdown_connections 2024-02-19T15:32:38.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:38.996+0000 7f6cbf200640 1 -- 172.21.15.47:0/1002778547 wait complete. 2024-02-19T15:32:38.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.000+0000 7f6cbf200640 1 Processor -- start 2024-02-19T15:32:38.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.000+0000 7f6cbf200640 1 -- start start 2024-02-19T15:32:39.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.000+0000 7f6cbf200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc00998c0 0x7f6cc00c4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:39.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.000+0000 7f6cbf200640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cc015e160 0x7f6cc00d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:39.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.000+0000 7f6cbf200640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cc00c65d0 0x7f6cc00cf980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:39.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.000+0000 7f6cbf200640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc00c61b0 con 0x7f6cc015e160 2024-02-19T15:32:39.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.000+0000 7f6cbf200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc0155c90 con 0x7f6cc00998c0 2024-02-19T15:32:39.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.000+0000 7f6cbf200640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc00d0f60 con 0x7f6cc00c65d0 2024-02-19T15:32:39.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.000+0000 7f6cbd400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cc015e160 0x7f6cc00d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:39.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.000+0000 7f6cbd400640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cc015e160 0x7f6cc00d0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55748/0 (socket says 172.21.15.47:55748) 2024-02-19T15:32:39.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.000+0000 7f6cbd400640 1 -- 172.21.15.47:0/731426382 learned_addr learned my addr 172.21.15.47:0/731426382 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:39.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.000+0000 7f6cbde00640 1 --2- 172.21.15.47:0/731426382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc00998c0 0x7f6cc00c4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:39.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.000+0000 7f6cbe800640 1 --2- 172.21.15.47:0/731426382 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cc00c65d0 0x7f6cc00cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:39.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.000+0000 7f6cbd400640 1 -- 172.21.15.47:0/731426382 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cc00c65d0 msgr2=0x7f6cc00cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:39.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.000+0000 7f6cbd400640 1 --2- 172.21.15.47:0/731426382 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cc00c65d0 0x7f6cc00cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.000+0000 7f6cbd400640 1 -- 172.21.15.47:0/731426382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc00998c0 msgr2=0x7f6cc00c4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:39.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.000+0000 7f6cbd400640 1 --2- 172.21.15.47:0/731426382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc00998c0 0x7f6cc00c4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.000+0000 7f6cbd400640 1 -- 172.21.15.47:0/731426382 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cac0094d0 con 0x7f6cc015e160 2024-02-19T15:32:39.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.004+0000 7f6cbd400640 1 --2- 172.21.15.47:0/731426382 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cc015e160 0x7f6cc00d0a20 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f6cac01d3c0 tx=0x7f6cac01bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:39.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.004+0000 7f6cb6000640 1 -- 172.21.15.47:0/731426382 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cac0ba600 con 0x7f6cc015e160 2024-02-19T15:32:39.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.004+0000 7f6cb6000640 1 -- 172.21.15.47:0/731426382 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cac0ba760 con 0x7f6cc015e160 2024-02-19T15:32:39.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.004+0000 7f6cb6000640 1 -- 172.21.15.47:0/731426382 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cac0bab10 con 0x7f6cc015e160 2024-02-19T15:32:39.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.004+0000 7f6cbf200640 1 -- 172.21.15.47:0/731426382 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc00c5530 con 0x7f6cc015e160 2024-02-19T15:32:39.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.004+0000 7f6cbf200640 1 -- 172.21.15.47:0/731426382 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc00c5af0 con 0x7f6cc015e160 2024-02-19T15:32:39.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.004+0000 7f6cbf200640 1 -- 172.21.15.47:0/731426382 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cc004f260 con 0x7f6cc015e160 2024-02-19T15:32:39.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.004+0000 7f6cb6000640 1 -- 172.21.15.47:0/731426382 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f6cac0bac70 con 0x7f6cc015e160 2024-02-19T15:32:39.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.004+0000 7f6cb6000640 1 --2- 172.21.15.47:0/731426382 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ca0066e70 0x7f6ca0069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:39.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.004+0000 7f6cb6000640 1 -- 172.21.15.47:0/731426382 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6cac122ee0 con 0x7f6cc015e160 2024-02-19T15:32:39.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.008+0000 7f6cbde00640 1 --2- 172.21.15.47:0/731426382 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ca0066e70 0x7f6ca0069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:39.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.008+0000 7f6cbde00640 1 --2- 172.21.15.47:0/731426382 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ca0066e70 0x7f6ca0069330 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6cc009e2a0 tx=0x7f6cb0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:39.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.008+0000 7f6cb6000640 1 -- 172.21.15.47:0/731426382 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f6cac0efd30 con 0x7f6cc015e160 2024-02-19T15:32:39.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.180+0000 7f6cbf200640 1 -- 172.21.15.47:0/731426382 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6cc00ccfb0 con 0x7f6cc015e160 2024-02-19T15:32:39.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.180+0000 7f6cb6000640 1 -- 172.21.15.47:0/731426382 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6cac0f33f0 con 0x7f6cc015e160 2024-02-19T15:32:39.180 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-02-19T15:32:39.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.184+0000 7f6cbf200640 1 -- 172.21.15.47:0/731426382 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ca0066e70 msgr2=0x7f6ca0069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:39.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.184+0000 7f6cbf200640 1 --2- 172.21.15.47:0/731426382 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ca0066e70 0x7f6ca0069330 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6cc009e2a0 tx=0x7f6cb0008040 comp rx=0 tx=0).stop 2024-02-19T15:32:39.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.184+0000 7f6cbf200640 1 -- 172.21.15.47:0/731426382 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cc015e160 msgr2=0x7f6cc00d0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:39.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.184+0000 7f6cbf200640 1 --2- 172.21.15.47:0/731426382 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cc015e160 0x7f6cc00d0a20 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f6cac01d3c0 tx=0x7f6cac01bab0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.184+0000 7f6cbf200640 1 -- 172.21.15.47:0/731426382 shutdown_connections 2024-02-19T15:32:39.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.184+0000 7f6cbf200640 1 --2- 172.21.15.47:0/731426382 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f6ca0066e70 0x7f6ca0069330 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.184+0000 7f6cbf200640 1 --2- 172.21.15.47:0/731426382 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cc00c65d0 0x7f6cc00cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.184+0000 7f6cbf200640 1 --2- 172.21.15.47:0/731426382 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6cc015e160 0x7f6cc00d0a20 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.184+0000 7f6cbf200640 1 --2- 172.21.15.47:0/731426382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc00998c0 0x7f6cc00c4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.184+0000 7f6cbf200640 1 -- 172.21.15.47:0/731426382 >> 172.21.15.47:0/731426382 conn(0x7f6cc0072380 msgr2=0x7f6cc00cc6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:39.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.184+0000 7f6cbf200640 1 -- 172.21.15.47:0/731426382 shutdown_connections 2024-02-19T15:32:39.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.184+0000 7f6cbf200640 1 -- 172.21.15.47:0/731426382 wait complete. 2024-02-19T15:32:39.205 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:32:39.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.348+0000 7f217ea00640 1 Processor -- start 2024-02-19T15:32:39.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.348+0000 7f217ea00640 1 -- start start 2024-02-19T15:32:39.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.348+0000 7f217ea00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f21780ce3b0 0x7f21780ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:39.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.348+0000 7f217ea00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f217815e280 0x7f21780cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:39.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.348+0000 7f217ea00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21780cc620 con 0x7f21780d2650 2024-02-19T15:32:39.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.348+0000 7f217ea00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21780d12e0 con 0x7f21780ce3b0 2024-02-19T15:32:39.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.348+0000 7f217ea00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21780d1940 con 0x7f217815e280 2024-02-19T15:32:39.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.352+0000 7f2177e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f21780ce3b0 0x7f21780ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:39.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.352+0000 7f2177e00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f21780ce3b0 0x7f21780ce790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55762/0 (socket says 172.21.15.47:55762) 2024-02-19T15:32:39.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.352+0000 7f2177e00640 1 -- 172.21.15.47:0/3409748889 learned_addr learned my addr 172.21.15.47:0/3409748889 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:39.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.352+0000 7f217ca00640 1 --2- 172.21.15.47:0/3409748889 >> v2:172.21.15.116:3300/0 conn(0x7f217815e280 0x7f21780cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:39.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.352+0000 7f2177e00640 1 -- 172.21.15.47:0/3409748889 >> v2:172.21.15.116:3300/0 conn(0x7f217815e280 msgr2=0x7f21780cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:39.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.352+0000 7f2177e00640 1 --2- 172.21.15.47:0/3409748889 >> v2:172.21.15.116:3300/0 conn(0x7f217815e280 0x7f21780cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.352+0000 7f2177e00640 1 -- 172.21.15.47:0/3409748889 >> v1:172.21.15.116:6789/0 conn(0x7f21780d2650 legacy=0x7f21780ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:39.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.352+0000 7f2177e00640 1 -- 172.21.15.47:0/3409748889 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21780cecd0 con 0x7f21780ce3b0 2024-02-19T15:32:39.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.352+0000 7f2177e00640 1 --2- 172.21.15.47:0/3409748889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21780ce3b0 0x7f21780ce790 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f2178067b60 tx=0x7f216801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3b38831d099b9f9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:39.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.352+0000 7f2177400640 1 -- 172.21.15.47:0/3409748889 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21680ab600 con 0x7f21780ce3b0 2024-02-19T15:32:39.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.352+0000 7f2177400640 1 -- 172.21.15.47:0/3409748889 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21680a7050 con 0x7f21780ce3b0 2024-02-19T15:32:39.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.352+0000 7f2177400640 1 -- 172.21.15.47:0/3409748889 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21680ab980 con 0x7f21780ce3b0 2024-02-19T15:32:39.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ea00640 1 -- 172.21.15.47:0/3409748889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21780ce3b0 msgr2=0x7f21780ce790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:39.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ea00640 1 --2- 172.21.15.47:0/3409748889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21780ce3b0 0x7f21780ce790 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f2178067b60 tx=0x7f216801c400 comp rx=0 tx=0).stop 2024-02-19T15:32:39.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ea00640 1 -- 172.21.15.47:0/3409748889 shutdown_connections 2024-02-19T15:32:39.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ea00640 1 --2- 172.21.15.47:0/3409748889 >> v2:172.21.15.116:3300/0 conn(0x7f217815e280 0x7f21780cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ea00640 1 --2- 172.21.15.47:0/3409748889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21780ce3b0 0x7f21780ce790 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ea00640 1 -- 172.21.15.47:0/3409748889 >> 172.21.15.47:0/3409748889 conn(0x7f21780d1b30 msgr2=0x7f21780d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:39.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ea00640 1 -- 172.21.15.47:0/3409748889 shutdown_connections 2024-02-19T15:32:39.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ea00640 1 -- 172.21.15.47:0/3409748889 wait complete. 2024-02-19T15:32:39.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ea00640 1 Processor -- start 2024-02-19T15:32:39.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ea00640 1 -- start start 2024-02-19T15:32:39.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21780ce3b0 0x7f2178154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:39.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ea00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21780d2650 0x7f217814f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:39.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ea00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f217815e280 0x7f21780cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:39.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ea00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217814f760 con 0x7f217815e280 2024-02-19T15:32:39.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ea00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21780cbb70 con 0x7f21780ce3b0 2024-02-19T15:32:39.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ea00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2178154c70 con 0x7f21780d2650 2024-02-19T15:32:39.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f217815e280 0x7f21780cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:39.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ca00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f217815e280 0x7f21780cb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55764/0 (socket says 172.21.15.47:55764) 2024-02-19T15:32:39.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ca00640 1 -- 172.21.15.47:0/1738898517 learned_addr learned my addr 172.21.15.47:0/1738898517 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:39.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f2177e00640 1 --2- 172.21.15.47:0/1738898517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21780ce3b0 0x7f2178154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:39.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f216f400640 1 --2- 172.21.15.47:0/1738898517 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21780d2650 0x7f217814f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:39.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ca00640 1 -- 172.21.15.47:0/1738898517 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21780d2650 msgr2=0x7f217814f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:39.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ca00640 1 --2- 172.21.15.47:0/1738898517 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21780d2650 0x7f217814f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ca00640 1 -- 172.21.15.47:0/1738898517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21780ce3b0 msgr2=0x7f2178154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:39.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ca00640 1 --2- 172.21.15.47:0/1738898517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21780ce3b0 0x7f2178154590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f217ca00640 1 -- 172.21.15.47:0/1738898517 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21680094d0 con 0x7f217815e280 2024-02-19T15:32:39.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.356+0000 7f216f400640 1 --2- 172.21.15.47:0/1738898517 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21780d2650 0x7f217814f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:32:39.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.360+0000 7f217ca00640 1 --2- 172.21.15.47:0/1738898517 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f217815e280 0x7f21780cb630 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f217000efc0 tx=0x7f217000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:39.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.360+0000 7f2174c00640 1 -- 172.21.15.47:0/1738898517 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2170009ba0 con 0x7f217815e280 2024-02-19T15:32:39.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.360+0000 7f2174c00640 1 -- 172.21.15.47:0/1738898517 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f217000f040 con 0x7f217815e280 2024-02-19T15:32:39.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.360+0000 7f217ea00640 1 -- 172.21.15.47:0/1738898517 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2178154f50 con 0x7f217815e280 2024-02-19T15:32:39.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.360+0000 7f2174c00640 1 -- 172.21.15.47:0/1738898517 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21700146c0 con 0x7f217815e280 2024-02-19T15:32:39.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.360+0000 7f217ea00640 1 -- 172.21.15.47:0/1738898517 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2178155510 con 0x7f217815e280 2024-02-19T15:32:39.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.360+0000 7f217ea00640 1 -- 172.21.15.47:0/1738898517 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2178075360 con 0x7f217815e280 2024-02-19T15:32:39.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.360+0000 7f2174c00640 1 -- 172.21.15.47:0/1738898517 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f2170014820 con 0x7f217815e280 2024-02-19T15:32:39.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.360+0000 7f2174c00640 1 --2- 172.21.15.47:0/1738898517 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2154066e40 0x7f2154069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:39.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.360+0000 7f2174c00640 1 -- 172.21.15.47:0/1738898517 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2170085900 con 0x7f217815e280 2024-02-19T15:32:39.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.360+0000 7f2177e00640 1 --2- 172.21.15.47:0/1738898517 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2154066e40 0x7f2154069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:39.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.364+0000 7f2177e00640 1 --2- 172.21.15.47:0/1738898517 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2154066e40 0x7f2154069300 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f21680ad000 tx=0x7f21680a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:39.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.364+0000 7f2174c00640 1 -- 172.21.15.47:0/1738898517 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f2170052580 con 0x7f217815e280 2024-02-19T15:32:39.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.536+0000 7f217ea00640 1 -- 172.21.15.47:0/1738898517 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f21780cdf80 con 0x7f217815e280 2024-02-19T15:32:39.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.540+0000 7f2174c00640 1 -- 172.21.15.47:0/1738898517 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v129) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2170018020 con 0x7f217815e280 2024-02-19T15:32:39.538 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 32 2024-02-19T15:32:39.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.540+0000 7f217ea00640 1 -- 172.21.15.47:0/1738898517 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2154066e40 msgr2=0x7f2154069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:39.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.540+0000 7f217ea00640 1 --2- 172.21.15.47:0/1738898517 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2154066e40 0x7f2154069300 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f21680ad000 tx=0x7f21680a6000 comp rx=0 tx=0).stop 2024-02-19T15:32:39.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.540+0000 7f217ea00640 1 -- 172.21.15.47:0/1738898517 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f217815e280 msgr2=0x7f21780cb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:39.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.540+0000 7f217ea00640 1 --2- 172.21.15.47:0/1738898517 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f217815e280 0x7f21780cb630 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f217000efc0 tx=0x7f217000c490 comp rx=0 tx=0).stop 2024-02-19T15:32:39.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.540+0000 7f217ea00640 1 -- 172.21.15.47:0/1738898517 shutdown_connections 2024-02-19T15:32:39.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.540+0000 7f217ea00640 1 --2- 172.21.15.47:0/1738898517 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f2154066e40 0x7f2154069300 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.540+0000 7f217ea00640 1 --2- 172.21.15.47:0/1738898517 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f217815e280 0x7f21780cb630 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.540+0000 7f217ea00640 1 --2- 172.21.15.47:0/1738898517 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21780d2650 0x7f217814f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.540+0000 7f217ea00640 1 --2- 172.21.15.47:0/1738898517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21780ce3b0 0x7f2178154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.540+0000 7f217ea00640 1 -- 172.21.15.47:0/1738898517 >> 172.21.15.47:0/1738898517 conn(0x7f21780d1b30 msgr2=0x7f21780c5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:39.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.540+0000 7f217ea00640 1 -- 172.21.15.47:0/1738898517 shutdown_connections 2024-02-19T15:32:39.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.540+0000 7f217ea00640 1 -- 172.21.15.47:0/1738898517 wait complete. 2024-02-19T15:32:39.561 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:32:39.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.704+0000 7fe9f3600640 1 Processor -- start 2024-02-19T15:32:39.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.704+0000 7fe9f3600640 1 -- start start 2024-02-19T15:32:39.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.704+0000 7fe9f3600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe9ec0d1900 0x7fe9ec06f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:39.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.704+0000 7fe9f3600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe9ec15f890 0x7fe9ec15fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:39.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.704+0000 7fe9f3600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ec15cf80 con 0x7fe9ec06aeb0 2024-02-19T15:32:39.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.704+0000 7fe9f3600640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9ec15c3b0 con 0x7fe9ec15f890 2024-02-19T15:32:39.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.704+0000 7fe9f3600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9ec15c650 con 0x7fe9ec0d1900 2024-02-19T15:32:39.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.708+0000 7fe9f1600640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe9ec15f890 0x7fe9ec15fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:39.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.708+0000 7fe9f0c00640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fe9ec06aeb0 0x7fe9ec06b290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60618/0 (socket says 172.21.15.47:60618) 2024-02-19T15:32:39.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.708+0000 7fe9f0c00640 1 -- 172.21.15.47:0/108195903 learned_addr learned my addr 172.21.15.47:0/108195903 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:32:39.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.708+0000 7fe9e3e00640 1 --2- 172.21.15.47:0/108195903 >> v2:172.21.15.116:3300/0 conn(0x7fe9ec0d1900 0x7fe9ec06f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:39.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.708+0000 7fe9f1600640 1 -- 172.21.15.47:0/108195903 >> v2:172.21.15.116:3300/0 conn(0x7fe9ec0d1900 msgr2=0x7fe9ec06f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:32:39.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.708+0000 7fe9f1600640 1 --2- 172.21.15.47:0/108195903 >> v2:172.21.15.116:3300/0 conn(0x7fe9ec0d1900 0x7fe9ec06f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.708+0000 7fe9f1600640 1 -- 172.21.15.47:0/108195903 >> v1:172.21.15.47:6789/0 conn(0x7fe9ec06aeb0 legacy=0x7fe9ec06b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:39.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.708+0000 7fe9f1600640 1 -- 172.21.15.47:0/108195903 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ec15c930 con 0x7fe9ec15f890 2024-02-19T15:32:39.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.708+0000 7fe9ebe00640 1 -- 172.21.15.47:0/108195903 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682984306 0 0) 0x7fe9ec15cf80 con 0x7fe9ec06aeb0 2024-02-19T15:32:39.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.708+0000 7fe9f1600640 1 --2- 172.21.15.47:0/108195903 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9ec15f890 0x7fe9ec15fd00 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4009d90 tx=0x7fe9e401c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ad7e1123acedf3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.708+0000 7fe9ebe00640 1 -- 172.21.15.47:0/108195903 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e40a5840 con 0x7fe9ec15f890 2024-02-19T15:32:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.708+0000 7fe9ebe00640 1 -- 172.21.15.47:0/108195903 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e40a59a0 con 0x7fe9ec15f890 2024-02-19T15:32:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.708+0000 7fe9ebe00640 1 -- 172.21.15.47:0/108195903 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e40a5d30 con 0x7fe9ec15f890 2024-02-19T15:32:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f3600640 1 -- 172.21.15.47:0/108195903 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9ec15f890 msgr2=0x7fe9ec15fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f3600640 1 --2- 172.21.15.47:0/108195903 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9ec15f890 0x7fe9ec15fd00 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4009d90 tx=0x7fe9e401c870 comp rx=0 tx=0).stop 2024-02-19T15:32:39.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f3600640 1 -- 172.21.15.47:0/108195903 shutdown_connections 2024-02-19T15:32:39.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f3600640 1 --2- 172.21.15.47:0/108195903 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9ec15f890 0x7fe9ec15fd00 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f3600640 1 --2- 172.21.15.47:0/108195903 >> v2:172.21.15.116:3300/0 conn(0x7fe9ec0d1900 0x7fe9ec06f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f3600640 1 -- 172.21.15.47:0/108195903 >> 172.21.15.47:0/108195903 conn(0x7fe9ec0d0d30 msgr2=0x7fe9ec0d1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:39.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f3600640 1 -- 172.21.15.47:0/108195903 shutdown_connections 2024-02-19T15:32:39.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f3600640 1 -- 172.21.15.47:0/108195903 wait complete. 2024-02-19T15:32:39.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f3600640 1 Processor -- start 2024-02-19T15:32:39.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f3600640 1 -- start start 2024-02-19T15:32:39.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f3600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9ec06aeb0 0x7fe9ec14bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:39.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f3600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9ec0d1900 0x7fe9ec0cf520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:39.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f3600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9ec15f890 0x7fe9ec0c4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:39.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f3600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec157650 con 0x7fe9ec06aeb0 2024-02-19T15:32:39.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f3600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec0d06e0 con 0x7fe9ec0d1900 2024-02-19T15:32:39.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f3600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec14c510 con 0x7fe9ec15f890 2024-02-19T15:32:39.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f0c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9ec06aeb0 0x7fe9ec14bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:39.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f0c00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9ec06aeb0 0x7fe9ec14bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55790/0 (socket says 172.21.15.47:55790) 2024-02-19T15:32:39.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f0c00640 1 -- 172.21.15.47:0/2897205574 learned_addr learned my addr 172.21.15.47:0/2897205574 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:39.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9e3e00640 1 --2- 172.21.15.47:0/2897205574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9ec0d1900 0x7fe9ec0cf520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:39.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f1600640 1 --2- 172.21.15.47:0/2897205574 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9ec15f890 0x7fe9ec0c4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:39.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f0c00640 1 -- 172.21.15.47:0/2897205574 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9ec15f890 msgr2=0x7fe9ec0c4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f0c00640 1 --2- 172.21.15.47:0/2897205574 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9ec15f890 0x7fe9ec0c4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f0c00640 1 -- 172.21.15.47:0/2897205574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9ec0d1900 msgr2=0x7fe9ec0cf520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f0c00640 1 --2- 172.21.15.47:0/2897205574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9ec0d1900 0x7fe9ec0cf520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f0c00640 1 -- 172.21.15.47:0/2897205574 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e4009a70 con 0x7fe9ec06aeb0 2024-02-19T15:32:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f1600640 1 --2- 172.21.15.47:0/2897205574 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9ec15f890 0x7fe9ec0c4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:32:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.712+0000 7fe9f0c00640 1 --2- 172.21.15.47:0/2897205574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9ec06aeb0 0x7fe9ec14bfd0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc00c4b0 tx=0x7fe9dc00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.716+0000 7fe9e9600640 1 -- 172.21.15.47:0/2897205574 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc0074e0 con 0x7fe9ec06aeb0 2024-02-19T15:32:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.716+0000 7fe9e9600640 1 -- 172.21.15.47:0/2897205574 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9dc007660 con 0x7fe9ec06aeb0 2024-02-19T15:32:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.716+0000 7fe9e9600640 1 -- 172.21.15.47:0/2897205574 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc004a30 con 0x7fe9ec06aeb0 2024-02-19T15:32:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.716+0000 7fe9f3600640 1 -- 172.21.15.47:0/2897205574 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ec1578a0 con 0x7fe9ec06aeb0 2024-02-19T15:32:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.716+0000 7fe9f3600640 1 -- 172.21.15.47:0/2897205574 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9ec157c30 con 0x7fe9ec06aeb0 2024-02-19T15:32:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.716+0000 7fe9f3600640 1 -- 172.21.15.47:0/2897205574 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9ec075360 con 0x7fe9ec06aeb0 2024-02-19T15:32:39.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.716+0000 7fe9e9600640 1 -- 172.21.15.47:0/2897205574 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7fe9dc004b90 con 0x7fe9ec06aeb0 2024-02-19T15:32:39.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.716+0000 7fe9e9600640 1 --2- 172.21.15.47:0/2897205574 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe9c8066e40 0x7fe9c8069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:39.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.716+0000 7fe9e9600640 1 -- 172.21.15.47:0/2897205574 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe9dc083f60 con 0x7fe9ec06aeb0 2024-02-19T15:32:39.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.716+0000 7fe9e3e00640 1 --2- 172.21.15.47:0/2897205574 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe9c8066e40 0x7fe9c8069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:39.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.720+0000 7fe9e3e00640 1 --2- 172.21.15.47:0/2897205574 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe9c8066e40 0x7fe9c8069300 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec15b700 tx=0x7fe9d4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:39.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.720+0000 7fe9e9600640 1 -- 172.21.15.47:0/2897205574 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7fe9dc054850 con 0x7fe9ec06aeb0 2024-02-19T15:32:39.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.884+0000 7fe9f3600640 1 -- 172.21.15.47:0/2897205574 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe9ec072bc0 con 0x7fe9ec06aeb0 2024-02-19T15:32:39.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.884+0000 7fe9e9600640 1 -- 172.21.15.47:0/2897205574 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v129) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe9dc018020 con 0x7fe9ec06aeb0 2024-02-19T15:32:39.885 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 64 2024-02-19T15:32:39.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.888+0000 7fe9f3600640 1 -- 172.21.15.47:0/2897205574 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe9c8066e40 msgr2=0x7fe9c8069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:39.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.888+0000 7fe9f3600640 1 --2- 172.21.15.47:0/2897205574 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe9c8066e40 0x7fe9c8069300 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec15b700 tx=0x7fe9d4008040 comp rx=0 tx=0).stop 2024-02-19T15:32:39.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.888+0000 7fe9f3600640 1 -- 172.21.15.47:0/2897205574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9ec06aeb0 msgr2=0x7fe9ec14bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:39.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.888+0000 7fe9f3600640 1 --2- 172.21.15.47:0/2897205574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9ec06aeb0 0x7fe9ec14bfd0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc00c4b0 tx=0x7fe9dc00c970 comp rx=0 tx=0).stop 2024-02-19T15:32:39.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.888+0000 7fe9f3600640 1 -- 172.21.15.47:0/2897205574 shutdown_connections 2024-02-19T15:32:39.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.888+0000 7fe9f3600640 1 --2- 172.21.15.47:0/2897205574 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7fe9c8066e40 0x7fe9c8069300 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.888+0000 7fe9f3600640 1 --2- 172.21.15.47:0/2897205574 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9ec15f890 0x7fe9ec0c4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.888+0000 7fe9f3600640 1 --2- 172.21.15.47:0/2897205574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9ec0d1900 0x7fe9ec0cf520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.888+0000 7fe9f3600640 1 --2- 172.21.15.47:0/2897205574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9ec06aeb0 0x7fe9ec14bfd0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:39.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.888+0000 7fe9f3600640 1 -- 172.21.15.47:0/2897205574 >> 172.21.15.47:0/2897205574 conn(0x7fe9ec0d0d30 msgr2=0x7fe9ec0b1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:39.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.888+0000 7fe9f3600640 1 -- 172.21.15.47:0/2897205574 shutdown_connections 2024-02-19T15:32:39.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:39.888+0000 7fe9f3600640 1 -- 172.21.15.47:0/2897205574 wait complete. 2024-02-19T15:32:39.906 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-19T15:32:40.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.048+0000 7f3e91600640 1 Processor -- start 2024-02-19T15:32:40.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.048+0000 7f3e91600640 1 -- start start 2024-02-19T15:32:40.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.048+0000 7f3e91600640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3e8c15bcd0 0x7f3e8c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:40.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.048+0000 7f3e91600640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8c0cc700 con 0x7f3e8c0d0360 2024-02-19T15:32:40.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.048+0000 7f3e91600640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8c0cf300 con 0x7f3e8c0cedb0 2024-02-19T15:32:40.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.048+0000 7f3e91600640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e8c0cf5e0 con 0x7f3e8c15bcd0 2024-02-19T15:32:40.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.048+0000 7f3e8b400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f3e8c0d0360 0x7f3e8c15d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60626/0 (socket says 172.21.15.47:60626) 2024-02-19T15:32:40.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.048+0000 7f3e8b400640 1 -- 172.21.15.47:0/3505920742 learned_addr learned my addr 172.21.15.47:0/3505920742 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-02-19T15:32:40.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.048+0000 7f3e8aa00640 1 --2- 172.21.15.47:0/3505920742 >> v2:172.21.15.116:3300/0 conn(0x7f3e8c15bcd0 0x7f3e8c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:40.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.048+0000 7f3e8a000640 1 -- 172.21.15.47:0/3505920742 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 752098071 0 0) 0x7f3e8c0cc700 con 0x7f3e8c0d0360 2024-02-19T15:32:40.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.052+0000 7f3e8a000640 1 -- 172.21.15.47:0/3505920742 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e70003530 con 0x7f3e8c0d0360 2024-02-19T15:32:40.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.052+0000 7f3e8a000640 1 -- 172.21.15.47:0/3505920742 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2755457482 0 0) 0x7f3e70003530 con 0x7f3e8c0d0360 2024-02-19T15:32:40.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.052+0000 7f3e8aa00640 1 -- 172.21.15.47:0/3505920742 >> v1:172.21.15.116:6789/0 conn(0x7f3e8c0cedb0 legacy=0x7f3e8c0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:40.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.052+0000 7f3e8aa00640 1 -- 172.21.15.47:0/3505920742 >> v1:172.21.15.47:6789/0 conn(0x7f3e8c0d0360 legacy=0x7f3e8c15d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:40.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.052+0000 7f3e8aa00640 1 -- 172.21.15.47:0/3505920742 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8c0cc9a0 con 0x7f3e8c15bcd0 2024-02-19T15:32:40.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.052+0000 7f3e8aa00640 1 --2- 172.21.15.47:0/3505920742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e8c15bcd0 0x7f3e8c0cd480 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0097c0 tx=0x7f3e7c01c270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc7768dcd530d984 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.052+0000 7f3e8a000640 1 -- 172.21.15.47:0/3505920742 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e84001a00 con 0x7f3e8c15bcd0 2024-02-19T15:32:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.052+0000 7f3e8a000640 1 -- 172.21.15.47:0/3505920742 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e7c0a7050 con 0x7f3e8c15bcd0 2024-02-19T15:32:40.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.052+0000 7f3e8a000640 1 -- 172.21.15.47:0/3505920742 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c0ab620 con 0x7f3e8c15bcd0 2024-02-19T15:32:40.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e91600640 1 -- 172.21.15.47:0/3505920742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e8c15bcd0 msgr2=0x7f3e8c0cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:40.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e91600640 1 --2- 172.21.15.47:0/3505920742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e8c15bcd0 0x7f3e8c0cd480 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0097c0 tx=0x7f3e7c01c270 comp rx=0 tx=0).stop 2024-02-19T15:32:40.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e91600640 1 -- 172.21.15.47:0/3505920742 shutdown_connections 2024-02-19T15:32:40.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e91600640 1 --2- 172.21.15.47:0/3505920742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e8c15bcd0 0x7f3e8c0cd480 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:40.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e91600640 1 -- 172.21.15.47:0/3505920742 >> 172.21.15.47:0/3505920742 conn(0x7f3e8c0d1bf0 msgr2=0x7f3e8c0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:40.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e91600640 1 -- 172.21.15.47:0/3505920742 shutdown_connections 2024-02-19T15:32:40.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e91600640 1 -- 172.21.15.47:0/3505920742 wait complete. 2024-02-19T15:32:40.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e91600640 1 Processor -- start 2024-02-19T15:32:40.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e91600640 1 -- start start 2024-02-19T15:32:40.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e91600640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e8c0cedb0 0x7f3e8c0cb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:40.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e91600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e8c0d0360 0x7f3e8c0c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:40.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e91600640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e8c15bcd0 0x7f3e8c0c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:40.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e91600640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c06c5d0 con 0x7f3e8c0cedb0 2024-02-19T15:32:40.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e91600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c06bfc0 con 0x7f3e8c0d0360 2024-02-19T15:32:40.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e91600640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c0c9940 con 0x7f3e8c15bcd0 2024-02-19T15:32:40.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e8aa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e8c0cedb0 0x7f3e8c0cb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:40.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e8aa00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e8c0cedb0 0x7f3e8c0cb510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:55800/0 (socket says 172.21.15.47:55800) 2024-02-19T15:32:40.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e8aa00640 1 -- 172.21.15.47:0/3922891441 learned_addr learned my addr 172.21.15.47:0/3922891441 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-02-19T15:32:40.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e83e00640 1 --2- 172.21.15.47:0/3922891441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e8c0d0360 0x7f3e8c0c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:40.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e8b400640 1 --2- 172.21.15.47:0/3922891441 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e8c15bcd0 0x7f3e8c0c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:40.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e8aa00640 1 -- 172.21.15.47:0/3922891441 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e8c15bcd0 msgr2=0x7f3e8c0c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:40.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e8aa00640 1 --2- 172.21.15.47:0/3922891441 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e8c15bcd0 0x7f3e8c0c72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:40.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e8aa00640 1 -- 172.21.15.47:0/3922891441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e8c0d0360 msgr2=0x7f3e8c0c6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:40.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e8aa00640 1 --2- 172.21.15.47:0/3922891441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e8c0d0360 0x7f3e8c0c6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:40.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.056+0000 7f3e8aa00640 1 -- 172.21.15.47:0/3922891441 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e7c009420 con 0x7f3e8c0cedb0 2024-02-19T15:32:40.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.060+0000 7f3e8aa00640 1 --2- 172.21.15.47:0/3922891441 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e8c0cedb0 0x7f3e8c0cb510 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0097c0 tx=0x7f3e7c01aab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:40.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.060+0000 7f3e82a00640 1 -- 172.21.15.47:0/3922891441 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c004220 con 0x7f3e8c0cedb0 2024-02-19T15:32:40.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.060+0000 7f3e82a00640 1 -- 172.21.15.47:0/3922891441 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e7c004380 con 0x7f3e8c0cedb0 2024-02-19T15:32:40.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.060+0000 7f3e82a00640 1 -- 172.21.15.47:0/3922891441 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c0b78a0 con 0x7f3e8c0cedb0 2024-02-19T15:32:40.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.060+0000 7f3e91600640 1 -- 172.21.15.47:0/3922891441 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e8c0c9c20 con 0x7f3e8c0cedb0 2024-02-19T15:32:40.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.060+0000 7f3e91600640 1 -- 172.21.15.47:0/3922891441 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e8c0c95f0 con 0x7f3e8c0cedb0 2024-02-19T15:32:40.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.060+0000 7f3e91600640 1 -- 172.21.15.47:0/3922891441 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e8c0b6a80 con 0x7f3e8c0cedb0 2024-02-19T15:32:40.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.060+0000 7f3e82a00640 1 -- 172.21.15.47:0/3922891441 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 86409+0+0 (secure 0 0 0) 0x7f3e7c0b7a00 con 0x7f3e8c0cedb0 2024-02-19T15:32:40.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.060+0000 7f3e82a00640 1 --2- 172.21.15.47:0/3922891441 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3e68066e40 0x7f3e68069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:32:40.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.060+0000 7f3e82a00640 1 -- 172.21.15.47:0/3922891441 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3e7c11e730 con 0x7f3e8c0cedb0 2024-02-19T15:32:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.060+0000 7f3e83e00640 1 --2- 172.21.15.47:0/3922891441 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3e68066e40 0x7f3e68069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:32:40.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.064+0000 7f3e83e00640 1 --2- 172.21.15.47:0/3922891441 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3e68066e40 0x7f3e68069300 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c0b30b0 tx=0x7f3e74009930 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:32:40.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.072+0000 7f3e82a00640 1 -- 172.21.15.47:0/3922891441 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176549 (secure 0 0 0) 0x7f3e7c0eb4f0 con 0x7f3e8c0cedb0 2024-02-19T15:32:40.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.312+0000 7f3e91600640 1 -- 172.21.15.47:0/3922891441 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3e8c0c9de0 con 0x7f3e8c0cedb0 2024-02-19T15:32:40.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.312+0000 7f3e82a00640 1 -- 172.21.15.47:0/3922891441 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v129) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3e7c0b7cb0 con 0x7f3e8c0cedb0 2024-02-19T15:32:40.313 INFO:teuthology.orchestra.run.smithi047.stderr:listed 0 entries 2024-02-19T15:32:40.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.316+0000 7f3e91600640 1 -- 172.21.15.47:0/3922891441 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3e68066e40 msgr2=0x7f3e68069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:40.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.316+0000 7f3e91600640 1 --2- 172.21.15.47:0/3922891441 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3e68066e40 0x7f3e68069300 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c0b30b0 tx=0x7f3e74009930 comp rx=0 tx=0).stop 2024-02-19T15:32:40.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.316+0000 7f3e91600640 1 -- 172.21.15.47:0/3922891441 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e8c0cedb0 msgr2=0x7f3e8c0cb510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:32:40.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.316+0000 7f3e91600640 1 --2- 172.21.15.47:0/3922891441 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e8c0cedb0 0x7f3e8c0cb510 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0097c0 tx=0x7f3e7c01aab0 comp rx=0 tx=0).stop 2024-02-19T15:32:40.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.316+0000 7f3e91600640 1 -- 172.21.15.47:0/3922891441 shutdown_connections 2024-02-19T15:32:40.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.316+0000 7f3e91600640 1 --2- 172.21.15.47:0/3922891441 >> [v2:172.21.15.116:6832/2412897884,v1:172.21.15.116:6833/2412897884] conn(0x7f3e68066e40 0x7f3e68069300 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:40.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.316+0000 7f3e91600640 1 --2- 172.21.15.47:0/3922891441 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e8c15bcd0 0x7f3e8c0c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:40.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.316+0000 7f3e91600640 1 --2- 172.21.15.47:0/3922891441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e8c0d0360 0x7f3e8c0c6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:40.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.316+0000 7f3e91600640 1 --2- 172.21.15.47:0/3922891441 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e8c0cedb0 0x7f3e8c0cb510 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:32:40.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.316+0000 7f3e91600640 1 -- 172.21.15.47:0/3922891441 >> 172.21.15.47:0/3922891441 conn(0x7f3e8c0d1bf0 msgr2=0x7f3e8c15af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:32:40.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.316+0000 7f3e91600640 1 -- 172.21.15.47:0/3922891441 shutdown_connections 2024-02-19T15:32:40.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-02-19T15:32:40.316+0000 7f3e91600640 1 -- 172.21.15.47:0/3922891441 wait complete. 2024-02-19T15:32:40.344 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-02-19T15:32:40.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:32:40.344 DEBUG:teuthology.orchestra.run.smithi047:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-02-19T15:33:06.485 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:33:06.489 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:33:36.540 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:33:36.543 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:34:06.603 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:34:06.608 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:34:36.668 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:34:36.672 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:35:06.756 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:35:06.759 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:35:36.813 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:35:36.817 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:36:06.877 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:36:06.881 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:36:36.944 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:36:36.948 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:37:07.004 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:37:07.007 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:37:37.085 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:37:37.089 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:37:40.348 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout 2024-02-19T15:37:40.414 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() socket.timeout 2024-02-19T15:38:07.140 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:38:07.143 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:38:37.196 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:38:37.199 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:39:07.252 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:39:07.256 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:39:37.307 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:39:37.311 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:40:07.361 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:40:07.365 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:40:37.420 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:40:37.423 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:41:07.507 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:41:07.511 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:41:37.567 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:41:37.570 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:42:07.630 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:42:07.634 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:42:37.711 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:42:37.715 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:43:07.776 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:43:07.779 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:43:37.877 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:43:37.880 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:44:07.928 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:44:07.931 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:44:37.984 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:44:37.989 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:45:08.040 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:45:08.044 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:45:38.103 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:45:38.107 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:46:08.169 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:46:08.173 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:46:38.231 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:46:38.235 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:47:08.288 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:47:08.291 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:47:38.360 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:47:38.364 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:48:08.416 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:48:08.419 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:48:38.492 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:48:38.496 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:49:08.551 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:49:08.555 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:49:38.607 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:49:38.611 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:50:08.671 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:50:08.675 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:50:38.727 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:50:38.730 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:51:08.798 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:51:08.803 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:51:38.862 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:51:38.866 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:52:08.918 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:52:08.921 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:52:38.999 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:52:39.004 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:53:09.063 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:53:09.067 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:53:39.121 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:53:39.124 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:54:09.183 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:54:09.186 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:54:39.239 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:54:39.243 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:55:09.302 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:55:09.305 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:55:39.358 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:55:39.362 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:56:09.462 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:56:09.465 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:56:39.519 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:56:39.523 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:57:09.574 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:57:09.578 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:57:39.646 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:57:39.650 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:58:09.711 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:58:09.715 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:58:39.767 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:58:39.771 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:59:09.861 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:59:09.865 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:59:39.913 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:59:39.916 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:00:09.968 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:00:09.971 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:00:40.048 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:00:40.052 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:01:10.117 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:01:10.121 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:01:40.176 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:01:40.179 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:02:10.241 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:02:10.244 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:02:40.322 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:02:40.326 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:03:10.372 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:03:10.376 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:03:40.432 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:03:40.436 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:04:10.497 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:04:10.501 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:04:40.560 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:04:40.565 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:05:10.649 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:05:10.653 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:05:40.705 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:05:40.709 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:06:10.770 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:06:10.773 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:06:40.833 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:06:40.837 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:07:10.897 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:07:10.901 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:07:41.011 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:07:41.016 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:08:11.074 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:08:11.079 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:08:41.136 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:08:41.140 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:09:11.194 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:09:11.197 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:09:41.246 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:09:41.250 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:10:11.298 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:10:11.302 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:10:41.354 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:10:41.358 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:11:11.424 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:11:11.427 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:11:41.482 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:11:41.485 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:12:11.537 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:12:11.540 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:12:41.610 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:12:41.614 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:13:11.666 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:13:11.670 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:13:41.738 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:13:41.741 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:14:11.810 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:14:11.813 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:14:41.908 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:14:41.913 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:15:11.966 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:15:11.970 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:15:42.028 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:15:42.032 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:16:12.104 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:16:12.108 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:16:42.174 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:16:42.178 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:17:12.263 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:17:12.267 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:17:42.344 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:17:42.348 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:18:12.415 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:18:12.421 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:18:42.480 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:18:42.484 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:19:12.547 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:19:12.551 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:19:42.609 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:19:42.613 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:20:12.672 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:20:12.676 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:20:42.737 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:20:42.741 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:21:12.799 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:21:12.803 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:21:42.900 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:21:42.905 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:22:12.953 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:22:12.956 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:22:43.092 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:22:43.096 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:23:13.148 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:23:13.153 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:23:43.238 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:23:43.243 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:24:13.301 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:24:13.307 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:24:43.371 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:24:43.375 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:25:13.428 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:25:13.434 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:25:43.493 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:25:43.498 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:26:13.565 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:26:13.569 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:26:43.622 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:26:43.627 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:27:13.768 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:27:13.771 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:27:43.826 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:27:43.830 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:28:13.921 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:28:13.924 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:28:44.060 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:28:44.063 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:29:14.112 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:29:14.125 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:29:40.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:29:40.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:29:40.977018+0000) 2024-02-19T16:29:41.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:29:41.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:29:41.977271+0000) 2024-02-19T16:29:42.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:29:42.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:29:42.977538+0000) 2024-02-19T16:29:43.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:29:43.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:29:43.977793+0000) 2024-02-19T16:29:44.257 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:29:44.260 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:29:44.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:29:44.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:29:44.978066+0000) 2024-02-19T16:29:45.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:29:45.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:29:45.978337+0000) 2024-02-19T16:29:46.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:29:46.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:29:46.978624+0000) 2024-02-19T16:29:47.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:29:47.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:29:47.978889+0000) 2024-02-19T16:29:48.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:29:48.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:29:48.979044+0000) 2024-02-19T16:29:49.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:29:49.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:29:49.979204+0000) 2024-02-19T16:29:50.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:29:50.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:29:50.979449+0000) 2024-02-19T16:29:51.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:29:51.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:29:51.979708+0000) 2024-02-19T16:29:52.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:29:52.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:29:52.979909+0000) 2024-02-19T16:29:53.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:29:53.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:29:53.980143+0000) 2024-02-19T16:29:54.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:29:54.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:29:54.980274+0000) 2024-02-19T16:29:55.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:29:55.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:29:55.980439+0000) 2024-02-19T16:29:56.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:29:56.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:29:56.980638+0000) 2024-02-19T16:29:57.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:29:57.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:29:57.980866+0000) 2024-02-19T16:29:58.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:29:58.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:29:58.981141+0000) 2024-02-19T16:29:59.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:29:59.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:29:59.981427+0000) 2024-02-19T16:30:00.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:00.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:00.981677+0000) 2024-02-19T16:30:01.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:01.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:01.981842+0000) 2024-02-19T16:30:02.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:02.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:02.982077+0000) 2024-02-19T16:30:03.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:03.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:03.982290+0000) 2024-02-19T16:30:04.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:04.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:04.982493+0000) 2024-02-19T16:30:05.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:05.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:05.982789+0000) 2024-02-19T16:30:06.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:06.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:06.982969+0000) 2024-02-19T16:30:07.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:07.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:07.983156+0000) 2024-02-19T16:30:08.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:08.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:08.983305+0000) 2024-02-19T16:30:09.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:09.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:09.983551+0000) 2024-02-19T16:30:10.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:10.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:10.983803+0000) 2024-02-19T16:30:11.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:11.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:11.983997+0000) 2024-02-19T16:30:12.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:12.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:12.984228+0000) 2024-02-19T16:30:13.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:13.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:13.984452+0000) 2024-02-19T16:30:14.377 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:30:14.380 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:30:14.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:14.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:14.984701+0000) 2024-02-19T16:30:15.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:15.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:15.984852+0000) 2024-02-19T16:30:16.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:16.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:16.985046+0000) 2024-02-19T16:30:17.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:17.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:17.985290+0000) 2024-02-19T16:30:18.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:18.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:18.985473+0000) 2024-02-19T16:30:19.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:19.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:19.985597+0000) 2024-02-19T16:30:20.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:20.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:20.985846+0000) 2024-02-19T16:30:21.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:21.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:21.986022+0000) 2024-02-19T16:30:22.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:22.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:22.986259+0000) 2024-02-19T16:30:23.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:23.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:23.986508+0000) 2024-02-19T16:30:24.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:24.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:24.986765+0000) 2024-02-19T16:30:25.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:25.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:25.986906+0000) 2024-02-19T16:30:26.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:26.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:26.987024+0000) 2024-02-19T16:30:27.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:27.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:27.987217+0000) 2024-02-19T16:30:28.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:28.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:28.987350+0000) 2024-02-19T16:30:29.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:29.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:29.987544+0000) 2024-02-19T16:30:30.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:30.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:30.987742+0000) 2024-02-19T16:30:31.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:31.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:31.988011+0000) 2024-02-19T16:30:32.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:32.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:32.988270+0000) 2024-02-19T16:30:33.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:33.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:33.988504+0000) 2024-02-19T16:30:34.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:34.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:34.988779+0000) 2024-02-19T16:30:35.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:35.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:35.988982+0000) 2024-02-19T16:30:36.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:36.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:36.989182+0000) 2024-02-19T16:30:37.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:37.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:37.989374+0000) 2024-02-19T16:30:38.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:38.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:38.989576+0000) 2024-02-19T16:30:39.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:39.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:39.989757+0000) 2024-02-19T16:30:40.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:40.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:40.989997+0000) 2024-02-19T16:30:41.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:41.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:41.990252+0000) 2024-02-19T16:30:42.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:42.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:42.990507+0000) 2024-02-19T16:30:43.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:43.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:43.990798+0000) 2024-02-19T16:30:44.482 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:30:44.485 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:30:44.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:44.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:44.991059+0000) 2024-02-19T16:30:45.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:45.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:45.991304+0000) 2024-02-19T16:30:46.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:46.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:46.991545+0000) 2024-02-19T16:30:47.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:47.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:47.991709+0000) 2024-02-19T16:30:48.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:48.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:48.991980+0000) 2024-02-19T16:30:49.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:49.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:49.992210+0000) 2024-02-19T16:30:50.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:50.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:50.992387+0000) 2024-02-19T16:30:51.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:51.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:51.992608+0000) 2024-02-19T16:30:52.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:52.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:52.992824+0000) 2024-02-19T16:30:53.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:53.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:53.993064+0000) 2024-02-19T16:30:54.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:54.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:54.993226+0000) 2024-02-19T16:30:55.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:55.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:55.993426+0000) 2024-02-19T16:30:56.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:56.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:56.993608+0000) 2024-02-19T16:30:57.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:57.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:57.993810+0000) 2024-02-19T16:30:58.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:58.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:58.994054+0000) 2024-02-19T16:30:59.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:30:59.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:30:59.994250+0000) 2024-02-19T16:31:00.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:00.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:00.994528+0000) 2024-02-19T16:31:01.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:01.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:01.994857+0000) 2024-02-19T16:31:02.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:02.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:02.995024+0000) 2024-02-19T16:31:03.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:03.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:03.995267+0000) 2024-02-19T16:31:04.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:04.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:04.995447+0000) 2024-02-19T16:31:05.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:05.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:05.995707+0000) 2024-02-19T16:31:06.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:06.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:06.995936+0000) 2024-02-19T16:31:07.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:07.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:07.996208+0000) 2024-02-19T16:31:08.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:08.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:08.996492+0000) 2024-02-19T16:31:09.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:09.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:09.996645+0000) 2024-02-19T16:31:10.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:10.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:10.996882+0000) 2024-02-19T16:31:11.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:11.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:11.997053+0000) 2024-02-19T16:31:12.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:12.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:12.997296+0000) 2024-02-19T16:31:13.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:13.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:13.997548+0000) 2024-02-19T16:31:14.596 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:31:14.599 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:31:14.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:14.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:14.997807+0000) 2024-02-19T16:31:15.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:15.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:15.998058+0000) 2024-02-19T16:31:16.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:16.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:16.998252+0000) 2024-02-19T16:31:17.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:17.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:17.998530+0000) 2024-02-19T16:31:18.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:18.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:18.998838+0000) 2024-02-19T16:31:19.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:19.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:19.999113+0000) 2024-02-19T16:31:20.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:20.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:20.999380+0000) 2024-02-19T16:31:21.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:21.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:21.999598+0000) 2024-02-19T16:31:22.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:22.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:22.999856+0000) 2024-02-19T16:31:23.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:23.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:24.000054+0000) 2024-02-19T16:31:24.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:24.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:25.000249+0000) 2024-02-19T16:31:26.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:25.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:26.000386+0000) 2024-02-19T16:31:27.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:26.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:27.000579+0000) 2024-02-19T16:31:28.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:27.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:28.000837+0000) 2024-02-19T16:31:29.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:28.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:29.001049+0000) 2024-02-19T16:31:30.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:29.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:30.001242+0000) 2024-02-19T16:31:31.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:30.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:31.001364+0000) 2024-02-19T16:31:32.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:31.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:32.001572+0000) 2024-02-19T16:31:33.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:32.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:33.001778+0000) 2024-02-19T16:31:34.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:33.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:34.002022+0000) 2024-02-19T16:31:35.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:34.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:35.002271+0000) 2024-02-19T16:31:36.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:35.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:36.002517+0000) 2024-02-19T16:31:37.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:37.000+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:37.002691+0000) 2024-02-19T16:31:38.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:38.000+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:38.002884+0000) 2024-02-19T16:31:39.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:39.000+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:39.003077+0000) 2024-02-19T16:31:40.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:40.000+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:40.003233+0000) 2024-02-19T16:31:41.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:41.000+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:41.003444+0000) 2024-02-19T16:31:42.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:42.000+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:42.003762+0000) 2024-02-19T16:31:43.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:43.000+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:43.004016+0000) 2024-02-19T16:31:44.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:44.000+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:44.004275+0000) 2024-02-19T16:31:44.726 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:31:44.729 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:31:45.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:45.000+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:45.004574+0000) 2024-02-19T16:31:46.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:46.000+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:46.004865+0000) 2024-02-19T16:31:47.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:47.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:47.005147+0000) 2024-02-19T16:31:48.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:48.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:48.005423+0000) 2024-02-19T16:31:49.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:49.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:49.005707+0000) 2024-02-19T16:31:50.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:50.000+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:50.005901+0000) 2024-02-19T16:31:51.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:51.000+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:51.006160+0000) 2024-02-19T16:31:52.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:52.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:52.006414+0000) 2024-02-19T16:31:53.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:53.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:53.006702+0000) 2024-02-19T16:31:54.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:54.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:54.006997+0000) 2024-02-19T16:31:55.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:55.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:55.007234+0000) 2024-02-19T16:31:56.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:56.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:56.007516+0000) 2024-02-19T16:31:57.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:57.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:57.007833+0000) 2024-02-19T16:31:58.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:58.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:58.008094+0000) 2024-02-19T16:31:59.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:31:59.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:31:59.008364+0000) 2024-02-19T16:32:00.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:00.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:00.008606+0000) 2024-02-19T16:32:01.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:01.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:01.008860+0000) 2024-02-19T16:32:02.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:02.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:02.009094+0000) 2024-02-19T16:32:03.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:03.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:03.009293+0000) 2024-02-19T16:32:04.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:04.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:04.009497+0000) 2024-02-19T16:32:05.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:05.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:05.009779+0000) 2024-02-19T16:32:06.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:06.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:06.010070+0000) 2024-02-19T16:32:07.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:07.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:07.010349+0000) 2024-02-19T16:32:08.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:08.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:08.010571+0000) 2024-02-19T16:32:09.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:09.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:09.010801+0000) 2024-02-19T16:32:10.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:10.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:10.010993+0000) 2024-02-19T16:32:11.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:11.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:11.011249+0000) 2024-02-19T16:32:12.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:12.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:12.011535+0000) 2024-02-19T16:32:13.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:13.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:13.011822+0000) 2024-02-19T16:32:14.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:14.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:14.012111+0000) 2024-02-19T16:32:14.873 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:32:14.877 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:32:15.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:15.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:15.012322+0000) 2024-02-19T16:32:16.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:16.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:16.012548+0000) 2024-02-19T16:32:17.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:17.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:17.012851+0000) 2024-02-19T16:32:18.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:18.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:18.013133+0000) 2024-02-19T16:32:19.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:19.011+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:19.013359+0000) 2024-02-19T16:32:20.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:20.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:20.013610+0000) 2024-02-19T16:32:21.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:21.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:21.013814+0000) 2024-02-19T16:32:22.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:22.011+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:22.014024+0000) 2024-02-19T16:32:23.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:23.011+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:23.014204+0000) 2024-02-19T16:32:24.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:24.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:24.014410+0000) 2024-02-19T16:32:25.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:25.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:25.014613+0000) 2024-02-19T16:32:26.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:26.011+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:26.014886+0000) 2024-02-19T16:32:27.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:27.011+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:27.015085+0000) 2024-02-19T16:32:28.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:28.011+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:28.015240+0000) 2024-02-19T16:32:29.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:29.011+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:29.015441+0000) 2024-02-19T16:32:30.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:30.011+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:30.015717+0000) 2024-02-19T16:32:31.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:31.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:31.015942+0000) 2024-02-19T16:32:32.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:32.011+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:32.016107+0000) 2024-02-19T16:32:33.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:33.011+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:33.016379+0000) 2024-02-19T16:32:34.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:34.011+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:34.016676+0000) 2024-02-19T16:32:35.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:35.011+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:35.016916+0000) 2024-02-19T16:32:36.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:36.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:36.017182+0000) 2024-02-19T16:32:37.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:37.011+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:37.017436+0000) 2024-02-19T16:32:38.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:38.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:38.017727+0000) 2024-02-19T16:32:39.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:39.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:39.017931+0000) 2024-02-19T16:32:40.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:40.011+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:40.018241+0000) 2024-02-19T16:32:41.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:41.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:41.018469+0000) 2024-02-19T16:32:42.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:42.011+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:42.018644+0000) 2024-02-19T16:32:43.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:43.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:43.018939+0000) 2024-02-19T16:32:44.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:44.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:44.019192+0000) 2024-02-19T16:32:44.945 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:32:44.949 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:32:45.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:45.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:45.019463+0000) 2024-02-19T16:32:46.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:46.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:46.019714+0000) 2024-02-19T16:32:47.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:47.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:47.019930+0000) 2024-02-19T16:32:48.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:48.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:48.020188+0000) 2024-02-19T16:32:49.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:49.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:49.020403+0000) 2024-02-19T16:32:50.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:50.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:50.020594+0000) 2024-02-19T16:32:51.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:51.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:51.020780+0000) 2024-02-19T16:32:52.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:52.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:52.021062+0000) 2024-02-19T16:32:53.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:53.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:53.021303+0000) 2024-02-19T16:32:54.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:54.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:54.021538+0000) 2024-02-19T16:32:55.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:55.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:55.021787+0000) 2024-02-19T16:32:56.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:56.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:56.021981+0000) 2024-02-19T16:32:57.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:57.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:57.022238+0000) 2024-02-19T16:32:58.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:58.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:58.022447+0000) 2024-02-19T16:32:59.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:32:59.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:32:59.022624+0000) 2024-02-19T16:33:00.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:00.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:00.022895+0000) 2024-02-19T16:33:01.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:01.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:01.023140+0000) 2024-02-19T16:33:02.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:02.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:02.023393+0000) 2024-02-19T16:33:03.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:03.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:03.023646+0000) 2024-02-19T16:33:04.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:04.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:04.023907+0000) 2024-02-19T16:33:05.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:05.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:05.024152+0000) 2024-02-19T16:33:06.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:06.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:06.024390+0000) 2024-02-19T16:33:07.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:07.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:07.024626+0000) 2024-02-19T16:33:08.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:08.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:08.024815+0000) 2024-02-19T16:33:09.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:09.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:09.025056+0000) 2024-02-19T16:33:10.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:10.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:10.025304+0000) 2024-02-19T16:33:11.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:11.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:11.025546+0000) 2024-02-19T16:33:12.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:12.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:12.025759+0000) 2024-02-19T16:33:13.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:13.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:13.025954+0000) 2024-02-19T16:33:14.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:14.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:14.026191+0000) 2024-02-19T16:33:15.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:15.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:15.026438+0000) 2024-02-19T16:33:15.084 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:33:15.087 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:33:16.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:16.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:16.026680+0000) 2024-02-19T16:33:17.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:17.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:17.026867+0000) 2024-02-19T16:33:18.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:18.026+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:18.027065+0000) 2024-02-19T16:33:19.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:19.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:19.027258+0000) 2024-02-19T16:33:20.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:20.026+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:20.027512+0000) 2024-02-19T16:33:21.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:21.026+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:21.027750+0000) 2024-02-19T16:33:22.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:22.026+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:22.028030+0000) 2024-02-19T16:33:23.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:23.026+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:23.028273+0000) 2024-02-19T16:33:24.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:24.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:24.028501+0000) 2024-02-19T16:33:25.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:25.026+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:25.028776+0000) 2024-02-19T16:33:26.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:26.026+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:26.029053+0000) 2024-02-19T16:33:27.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:27.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:27.029308+0000) 2024-02-19T16:33:28.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:28.026+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:28.029514+0000) 2024-02-19T16:33:29.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:29.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:29.029752+0000) 2024-02-19T16:33:30.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:30.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:30.029985+0000) 2024-02-19T16:33:31.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:31.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:31.030257+0000) 2024-02-19T16:33:32.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:32.026+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:32.030566+0000) 2024-02-19T16:33:33.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:33.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:33.030853+0000) 2024-02-19T16:33:34.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:34.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:34.031088+0000) 2024-02-19T16:33:35.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:35.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:35.031313+0000) 2024-02-19T16:33:36.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:36.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:36.031495+0000) 2024-02-19T16:33:37.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:37.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:37.031698+0000) 2024-02-19T16:33:38.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:38.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:38.031869+0000) 2024-02-19T16:33:39.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:39.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:39.032126+0000) 2024-02-19T16:33:40.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:40.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:40.032319+0000) 2024-02-19T16:33:41.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:41.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:41.032570+0000) 2024-02-19T16:33:42.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:42.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:42.032806+0000) 2024-02-19T16:33:43.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:43.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:43.033076+0000) 2024-02-19T16:33:44.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:44.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:44.033365+0000) 2024-02-19T16:33:45.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:45.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:45.033658+0000) 2024-02-19T16:33:45.156 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:33:45.160 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:33:46.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:46.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:46.033865+0000) 2024-02-19T16:33:47.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:47.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:47.034124+0000) 2024-02-19T16:33:48.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:48.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:48.034418+0000) 2024-02-19T16:33:49.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:49.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:49.034755+0000) 2024-02-19T16:33:50.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:50.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:50.034973+0000) 2024-02-19T16:33:51.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:51.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:51.035247+0000) 2024-02-19T16:33:52.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:52.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:52.035570+0000) 2024-02-19T16:33:53.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:53.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:53.035820+0000) 2024-02-19T16:33:54.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:54.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:54.036081+0000) 2024-02-19T16:33:55.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:55.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:55.036333+0000) 2024-02-19T16:33:56.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:56.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:56.036513+0000) 2024-02-19T16:33:57.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:57.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:57.036754+0000) 2024-02-19T16:33:58.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:58.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:58.036942+0000) 2024-02-19T16:33:59.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:33:59.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:33:59.037201+0000) 2024-02-19T16:34:00.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:00.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:00.037527+0000) 2024-02-19T16:34:01.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:01.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:01.037764+0000) 2024-02-19T16:34:02.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:02.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:02.038022+0000) 2024-02-19T16:34:03.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:03.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:03.038294+0000) 2024-02-19T16:34:04.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:04.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:04.038607+0000) 2024-02-19T16:34:05.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:05.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:05.038998+0000) 2024-02-19T16:34:06.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:06.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:06.039285+0000) 2024-02-19T16:34:07.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:07.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:07.039556+0000) 2024-02-19T16:34:08.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:08.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:08.039830+0000) 2024-02-19T16:34:09.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:09.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:09.040088+0000) 2024-02-19T16:34:10.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:10.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:10.040295+0000) 2024-02-19T16:34:11.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:11.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:11.040576+0000) 2024-02-19T16:34:12.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:12.037+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:12.040872+0000) 2024-02-19T16:34:13.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:13.037+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:13.041077+0000) 2024-02-19T16:34:14.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:14.037+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:14.041339+0000) 2024-02-19T16:34:15.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:15.037+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:15.041630+0000) 2024-02-19T16:34:15.279 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:34:15.282 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:34:16.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:16.037+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:16.041805+0000) 2024-02-19T16:34:17.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:17.041+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:17.042080+0000) 2024-02-19T16:34:18.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:18.037+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:18.042361+0000) 2024-02-19T16:34:19.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:19.037+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:19.042577+0000) 2024-02-19T16:34:20.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:20.041+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:20.042831+0000) 2024-02-19T16:34:21.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:21.041+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:21.043057+0000) 2024-02-19T16:34:22.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:22.041+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:22.043334+0000) 2024-02-19T16:34:23.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:23.041+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:23.043531+0000) 2024-02-19T16:34:24.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:24.041+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:24.043790+0000) 2024-02-19T16:34:25.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:25.041+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:25.044081+0000) 2024-02-19T16:34:26.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:26.041+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:26.044376+0000) 2024-02-19T16:34:27.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:27.041+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:27.044702+0000) 2024-02-19T16:34:28.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:28.041+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:28.044947+0000) 2024-02-19T16:34:29.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:29.041+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:29.045217+0000) 2024-02-19T16:34:30.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:30.041+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:30.045513+0000) 2024-02-19T16:34:31.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:31.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:31.045797+0000) 2024-02-19T16:34:32.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:32.041+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:32.046017+0000) 2024-02-19T16:34:33.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:33.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:33.046275+0000) 2024-02-19T16:34:34.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:34.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:34.046495+0000) 2024-02-19T16:34:35.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:35.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:35.046794+0000) 2024-02-19T16:34:36.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:36.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:36.047086+0000) 2024-02-19T16:34:37.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:37.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:37.047393+0000) 2024-02-19T16:34:38.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:38.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:38.047590+0000) 2024-02-19T16:34:39.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:39.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:39.047763+0000) 2024-02-19T16:34:40.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:40.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:40.047962+0000) 2024-02-19T16:34:41.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:41.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:41.048250+0000) 2024-02-19T16:34:42.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:42.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:42.048533+0000) 2024-02-19T16:34:43.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:43.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:43.048817+0000) 2024-02-19T16:34:44.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:44.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:44.049105+0000) 2024-02-19T16:34:45.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:45.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:45.049367+0000) 2024-02-19T16:34:45.409 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:34:45.412 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:34:46.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:46.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:46.049628+0000) 2024-02-19T16:34:47.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:47.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:47.049837+0000) 2024-02-19T16:34:48.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:48.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:48.050036+0000) 2024-02-19T16:34:49.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:49.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:49.050240+0000) 2024-02-19T16:34:50.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:50.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:50.050420+0000) 2024-02-19T16:34:51.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:51.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:51.050764+0000) 2024-02-19T16:34:52.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:52.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:52.050930+0000) 2024-02-19T16:34:53.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:53.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:53.051243+0000) 2024-02-19T16:34:54.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:54.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:54.051474+0000) 2024-02-19T16:34:55.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:55.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:55.051738+0000) 2024-02-19T16:34:56.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:56.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:56.051993+0000) 2024-02-19T16:34:57.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:57.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:57.052240+0000) 2024-02-19T16:34:58.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:58.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:58.052451+0000) 2024-02-19T16:34:59.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:34:59.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:34:59.052757+0000) 2024-02-19T16:35:00.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:00.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:00.052973+0000) 2024-02-19T16:35:01.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:01.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:01.053151+0000) 2024-02-19T16:35:02.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:02.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:02.053447+0000) 2024-02-19T16:35:03.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:03.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:03.053732+0000) 2024-02-19T16:35:04.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:04.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:04.053994+0000) 2024-02-19T16:35:05.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:05.048+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:05.054278+0000) 2024-02-19T16:35:06.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:06.048+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:06.054533+0000) 2024-02-19T16:35:07.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:07.052+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:07.054781+0000) 2024-02-19T16:35:08.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:08.052+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:08.055037+0000) 2024-02-19T16:35:09.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:09.052+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:09.055278+0000) 2024-02-19T16:35:10.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:10.052+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:10.055454+0000) 2024-02-19T16:35:11.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:11.052+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:11.055696+0000) 2024-02-19T16:35:12.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:12.052+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:12.055871+0000) 2024-02-19T16:35:13.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:13.052+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:13.056108+0000) 2024-02-19T16:35:14.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:14.052+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:14.056366+0000) 2024-02-19T16:35:15.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:15.052+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:15.056611+0000) 2024-02-19T16:35:15.481 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:35:15.485 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:35:16.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:16.052+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:16.056798+0000) 2024-02-19T16:35:17.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:17.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:17.057035+0000) 2024-02-19T16:35:18.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:18.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:18.057217+0000) 2024-02-19T16:35:19.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:19.052+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:19.057477+0000) 2024-02-19T16:35:20.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:20.052+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:20.057730+0000) 2024-02-19T16:35:21.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:21.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:21.058000+0000) 2024-02-19T16:35:22.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:22.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:22.058238+0000) 2024-02-19T16:35:23.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:23.052+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:23.058470+0000) 2024-02-19T16:35:24.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:24.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:24.058784+0000) 2024-02-19T16:35:25.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:25.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:25.058969+0000) 2024-02-19T16:35:26.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:26.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:26.059215+0000) 2024-02-19T16:35:27.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:27.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:27.059456+0000) 2024-02-19T16:35:28.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:28.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:28.059657+0000) 2024-02-19T16:35:29.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:29.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:29.059898+0000) 2024-02-19T16:35:30.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:30.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:30.060085+0000) 2024-02-19T16:35:31.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:31.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:31.060323+0000) 2024-02-19T16:35:32.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:32.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:32.060543+0000) 2024-02-19T16:35:33.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:33.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:33.060846+0000) 2024-02-19T16:35:34.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:34.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:34.061152+0000) 2024-02-19T16:35:35.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:35.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:35.061438+0000) 2024-02-19T16:35:36.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:36.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:36.061727+0000) 2024-02-19T16:35:37.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:37.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:37.061980+0000) 2024-02-19T16:35:38.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:38.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:38.062240+0000) 2024-02-19T16:35:39.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:39.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:39.062532+0000) 2024-02-19T16:35:40.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:40.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:40.062754+0000) 2024-02-19T16:35:41.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:41.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:41.062948+0000) 2024-02-19T16:35:42.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:42.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:42.063202+0000) 2024-02-19T16:35:43.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:43.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:43.063482+0000) 2024-02-19T16:35:44.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:44.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:44.063758+0000) 2024-02-19T16:35:45.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:45.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:45.064067+0000) 2024-02-19T16:35:45.545 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:35:45.549 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:35:46.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:46.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:46.064305+0000) 2024-02-19T16:35:47.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:47.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:47.064598+0000) 2024-02-19T16:35:48.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:48.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:48.064844+0000) 2024-02-19T16:35:49.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:49.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:49.065061+0000) 2024-02-19T16:35:50.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:50.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:50.065228+0000) 2024-02-19T16:35:51.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:51.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:51.065497+0000) 2024-02-19T16:35:52.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:52.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:52.065767+0000) 2024-02-19T16:35:53.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:53.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:53.065986+0000) 2024-02-19T16:35:54.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:54.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:54.066164+0000) 2024-02-19T16:35:55.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:55.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:55.066449+0000) 2024-02-19T16:35:56.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:56.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:56.066672+0000) 2024-02-19T16:35:57.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:57.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:57.066827+0000) 2024-02-19T16:35:58.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:58.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:58.067088+0000) 2024-02-19T16:35:59.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:35:59.063+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:35:59.067230+0000) 2024-02-19T16:36:00.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:00.063+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:00.067435+0000) 2024-02-19T16:36:01.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:01.063+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:01.067713+0000) 2024-02-19T16:36:02.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:02.063+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:02.067946+0000) 2024-02-19T16:36:03.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:03.067+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:03.068240+0000) 2024-02-19T16:36:04.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:04.067+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:04.068521+0000) 2024-02-19T16:36:05.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:05.067+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:05.068833+0000) 2024-02-19T16:36:06.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:06.063+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:06.069115+0000) 2024-02-19T16:36:07.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:07.063+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:07.069401+0000) 2024-02-19T16:36:08.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:08.063+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:08.069608+0000) 2024-02-19T16:36:09.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:09.063+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:09.069830+0000) 2024-02-19T16:36:10.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:10.063+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:10.070119+0000) 2024-02-19T16:36:11.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:11.063+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:11.070404+0000) 2024-02-19T16:36:12.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:12.067+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:12.070696+0000) 2024-02-19T16:36:13.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:13.067+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:13.070995+0000) 2024-02-19T16:36:14.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:14.067+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:14.071204+0000) 2024-02-19T16:36:15.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:15.067+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:15.071445+0000) 2024-02-19T16:36:15.676 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:36:15.679 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:36:16.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:16.067+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:16.071746+0000) 2024-02-19T16:36:17.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:17.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:17.072036+0000) 2024-02-19T16:36:18.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:18.067+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:18.072340+0000) 2024-02-19T16:36:19.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:19.067+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:19.072653+0000) 2024-02-19T16:36:20.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:20.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:20.072907+0000) 2024-02-19T16:36:21.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:21.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:21.073182+0000) 2024-02-19T16:36:22.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:22.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:22.073437+0000) 2024-02-19T16:36:23.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:23.067+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:23.073680+0000) 2024-02-19T16:36:24.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:24.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:24.073915+0000) 2024-02-19T16:36:25.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:25.067+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:25.074174+0000) 2024-02-19T16:36:26.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:26.067+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:26.074424+0000) 2024-02-19T16:36:27.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:27.067+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:27.074604+0000) 2024-02-19T16:36:28.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:28.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:28.074898+0000) 2024-02-19T16:36:29.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:29.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:29.075148+0000) 2024-02-19T16:36:30.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:30.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:30.075354+0000) 2024-02-19T16:36:31.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:31.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:31.075535+0000) 2024-02-19T16:36:32.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:32.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:32.075774+0000) 2024-02-19T16:36:33.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:33.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:33.076063+0000) 2024-02-19T16:36:34.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:34.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:34.076304+0000) 2024-02-19T16:36:35.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:35.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:35.076588+0000) 2024-02-19T16:36:36.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:36.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:36.076893+0000) 2024-02-19T16:36:37.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:37.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:37.077155+0000) 2024-02-19T16:36:38.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:38.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:38.077381+0000) 2024-02-19T16:36:39.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:39.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:39.077552+0000) 2024-02-19T16:36:40.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:40.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:40.077863+0000) 2024-02-19T16:36:41.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:41.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:41.078079+0000) 2024-02-19T16:36:42.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:42.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:42.078348+0000) 2024-02-19T16:36:43.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:43.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:43.078553+0000) 2024-02-19T16:36:44.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:44.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:44.078871+0000) 2024-02-19T16:36:45.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:45.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:45.079177+0000) 2024-02-19T16:36:45.807 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:36:45.812 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:36:46.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:46.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:46.079332+0000) 2024-02-19T16:36:47.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:47.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:47.079506+0000) 2024-02-19T16:36:48.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:48.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:48.079788+0000) 2024-02-19T16:36:49.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:49.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:49.080093+0000) 2024-02-19T16:36:50.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:50.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:50.080283+0000) 2024-02-19T16:36:51.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:51.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:51.080598+0000) 2024-02-19T16:36:52.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:52.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:52.080865+0000) 2024-02-19T16:36:53.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:53.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:53.081090+0000) 2024-02-19T16:36:54.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:54.078+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:54.081393+0000) 2024-02-19T16:36:55.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:55.078+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:55.081607+0000) 2024-02-19T16:36:56.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:56.078+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:56.081870+0000) 2024-02-19T16:36:57.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:57.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:57.082108+0000) 2024-02-19T16:36:58.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:58.078+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:58.082375+0000) 2024-02-19T16:36:59.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:36:59.078+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:36:59.082598+0000) 2024-02-19T16:37:00.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:00.078+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:00.082852+0000) 2024-02-19T16:37:01.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:01.078+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:01.082984+0000) 2024-02-19T16:37:02.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:02.078+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:02.083129+0000) 2024-02-19T16:37:03.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:03.078+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:03.083279+0000) 2024-02-19T16:37:04.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:04.082+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:04.083520+0000) 2024-02-19T16:37:05.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:05.082+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:05.083734+0000) 2024-02-19T16:37:06.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:06.078+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:06.083913+0000) 2024-02-19T16:37:07.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:07.082+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:07.084124+0000) 2024-02-19T16:37:08.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:08.082+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:08.084369+0000) 2024-02-19T16:37:09.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:09.082+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:09.084538+0000) 2024-02-19T16:37:10.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:10.082+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:10.084691+0000) 2024-02-19T16:37:11.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:11.082+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:11.084866+0000) 2024-02-19T16:37:12.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:12.082+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:12.085172+0000) 2024-02-19T16:37:13.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:13.082+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:13.085442+0000) 2024-02-19T16:37:14.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:14.082+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:14.085694+0000) 2024-02-19T16:37:15.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:15.082+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:15.085953+0000) 2024-02-19T16:37:15.921 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:37:15.924 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:37:16.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:16.082+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:16.086188+0000) 2024-02-19T16:37:17.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:17.082+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:17.086326+0000) 2024-02-19T16:37:18.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:18.082+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:18.086526+0000) 2024-02-19T16:37:19.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:19.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:19.086797+0000) 2024-02-19T16:37:20.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:20.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:20.086984+0000) 2024-02-19T16:37:21.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:21.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:21.087248+0000) 2024-02-19T16:37:22.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:22.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:22.087476+0000) 2024-02-19T16:37:23.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:23.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:23.087617+0000) 2024-02-19T16:37:24.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:24.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:24.087792+0000) 2024-02-19T16:37:25.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:25.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:25.088037+0000) 2024-02-19T16:37:26.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:26.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:26.088232+0000) 2024-02-19T16:37:27.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:27.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:27.088491+0000) 2024-02-19T16:37:28.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:28.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:28.088723+0000) 2024-02-19T16:37:29.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:29.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:29.088898+0000) 2024-02-19T16:37:30.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:30.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:30.089137+0000) 2024-02-19T16:37:31.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:31.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:31.089386+0000) 2024-02-19T16:37:32.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:32.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:32.089583+0000) 2024-02-19T16:37:33.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:33.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:33.089761+0000) 2024-02-19T16:37:34.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:34.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:34.089925+0000) 2024-02-19T16:37:35.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:35.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:35.090168+0000) 2024-02-19T16:37:36.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:36.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:36.090407+0000) 2024-02-19T16:37:37.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:37.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:37.090663+0000) 2024-02-19T16:37:38.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:38.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:38.090852+0000) 2024-02-19T16:37:39.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:39.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:39.091008+0000) 2024-02-19T16:37:40.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:40.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:40.091146+0000) 2024-02-19T16:37:41.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:41.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:41.091338+0000) 2024-02-19T16:37:42.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:42.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:42.091587+0000) 2024-02-19T16:37:43.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:43.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:43.091790+0000) 2024-02-19T16:37:44.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:44.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:44.092000+0000) 2024-02-19T16:37:45.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:45.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:45.092123+0000) 2024-02-19T16:37:46.001 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:37:46.004 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:37:46.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:46.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:46.092374+0000) 2024-02-19T16:37:47.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:47.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:47.092619+0000) 2024-02-19T16:37:48.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:48.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:48.092795+0000) 2024-02-19T16:37:49.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:49.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:49.092974+0000) 2024-02-19T16:37:50.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:50.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:50.093246+0000) 2024-02-19T16:37:51.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:51.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:51.093493+0000) 2024-02-19T16:37:52.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:52.089+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:52.093730+0000) 2024-02-19T16:37:53.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:53.089+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:53.093929+0000) 2024-02-19T16:37:54.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:54.089+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:54.094169+0000) 2024-02-19T16:37:55.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:55.089+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:55.094423+0000) 2024-02-19T16:37:56.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:56.089+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:56.094629+0000) 2024-02-19T16:37:57.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:57.093+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:57.094866+0000) 2024-02-19T16:37:58.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:58.093+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:58.095010+0000) 2024-02-19T16:37:59.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:37:59.093+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:37:59.095166+0000) 2024-02-19T16:38:00.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:00.093+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:00.095338+0000) 2024-02-19T16:38:01.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:01.093+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:01.095575+0000) 2024-02-19T16:38:02.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:02.093+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:02.095823+0000) 2024-02-19T16:38:03.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:03.093+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:03.096026+0000) 2024-02-19T16:38:04.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:04.093+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:04.096261+0000) 2024-02-19T16:38:05.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:05.093+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:05.096399+0000) 2024-02-19T16:38:06.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:06.093+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:06.096577+0000) 2024-02-19T16:38:07.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:07.093+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:07.096729+0000) 2024-02-19T16:38:08.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:08.093+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:08.096964+0000) 2024-02-19T16:38:09.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:09.093+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:09.097209+0000) 2024-02-19T16:38:10.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:10.093+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:10.097382+0000) 2024-02-19T16:38:11.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:11.093+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:11.097559+0000) 2024-02-19T16:38:12.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:12.093+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:12.097781+0000) 2024-02-19T16:38:13.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:13.093+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:13.097996+0000) 2024-02-19T16:38:14.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:14.093+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:14.098233+0000) 2024-02-19T16:38:15.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:15.093+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:15.098501+0000) 2024-02-19T16:38:16.053 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:38:16.057 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:38:16.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:16.097+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:16.098834+0000) 2024-02-19T16:38:17.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:17.097+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:17.099087+0000) 2024-02-19T16:38:18.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:18.097+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:18.099323+0000) 2024-02-19T16:38:19.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:19.097+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:19.099581+0000) 2024-02-19T16:38:20.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:20.097+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:20.099847+0000) 2024-02-19T16:38:21.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:21.097+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:21.100079+0000) 2024-02-19T16:38:22.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:22.097+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:22.100315+0000) 2024-02-19T16:38:23.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:23.097+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:23.100464+0000) 2024-02-19T16:38:24.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:24.097+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:24.100716+0000) 2024-02-19T16:38:25.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:25.097+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:25.100966+0000) 2024-02-19T16:38:26.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:26.097+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:26.101141+0000) 2024-02-19T16:38:27.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:27.097+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:27.101402+0000) 2024-02-19T16:38:28.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:28.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:28.101642+0000) 2024-02-19T16:38:29.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:29.097+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:29.101875+0000) 2024-02-19T16:38:30.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:30.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:30.102139+0000) 2024-02-19T16:38:31.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:31.097+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:31.102276+0000) 2024-02-19T16:38:32.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:32.097+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:32.102519+0000) 2024-02-19T16:38:33.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:33.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:33.102823+0000) 2024-02-19T16:38:34.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:34.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:34.103045+0000) 2024-02-19T16:38:35.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:35.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:35.103255+0000) 2024-02-19T16:38:36.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:36.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:36.103447+0000) 2024-02-19T16:38:37.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:37.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:37.103644+0000) 2024-02-19T16:38:38.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:38.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:38.103925+0000) 2024-02-19T16:38:39.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:39.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:39.104071+0000) 2024-02-19T16:38:40.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:40.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:40.104225+0000) 2024-02-19T16:38:41.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:41.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:41.104460+0000) 2024-02-19T16:38:42.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:42.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:42.104718+0000) 2024-02-19T16:38:43.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:43.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:43.104978+0000) 2024-02-19T16:38:44.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:44.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:44.105245+0000) 2024-02-19T16:38:45.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:45.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:45.105451+0000) 2024-02-19T16:38:46.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:46.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:46.105650+0000) 2024-02-19T16:38:46.112 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:38:46.115 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:38:47.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:47.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:47.105860+0000) 2024-02-19T16:38:48.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:48.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:48.106029+0000) 2024-02-19T16:38:49.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:49.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:49.106283+0000) 2024-02-19T16:38:50.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:50.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:50.106516+0000) 2024-02-19T16:38:51.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:51.104+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:51.106788+0000) 2024-02-19T16:38:52.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:52.104+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:52.106966+0000) 2024-02-19T16:38:53.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:53.104+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:53.107155+0000) 2024-02-19T16:38:54.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:54.104+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:54.107341+0000) 2024-02-19T16:38:55.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:55.104+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:55.107578+0000) 2024-02-19T16:38:56.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:56.104+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:56.107811+0000) 2024-02-19T16:38:57.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:57.104+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:57.107987+0000) 2024-02-19T16:38:58.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:58.104+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:58.108154+0000) 2024-02-19T16:38:59.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:38:59.104+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:38:59.108391+0000) 2024-02-19T16:39:00.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:00.104+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:00.108629+0000) 2024-02-19T16:39:01.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:01.104+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:01.108869+0000) 2024-02-19T16:39:02.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:02.108+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:02.109107+0000) 2024-02-19T16:39:03.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:03.104+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:03.109347+0000) 2024-02-19T16:39:04.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:04.104+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:04.109601+0000) 2024-02-19T16:39:05.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:05.108+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:05.109801+0000) 2024-02-19T16:39:06.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:06.104+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:06.110052+0000) 2024-02-19T16:39:07.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:07.104+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:07.110255+0000) 2024-02-19T16:39:08.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:08.104+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:08.110507+0000) 2024-02-19T16:39:09.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:09.108+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:09.110830+0000) 2024-02-19T16:39:10.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:10.108+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:10.110998+0000) 2024-02-19T16:39:11.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:11.108+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:11.111186+0000) 2024-02-19T16:39:12.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:12.108+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:12.111394+0000) 2024-02-19T16:39:13.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:13.108+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:13.111627+0000) 2024-02-19T16:39:14.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:14.108+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:14.111853+0000) 2024-02-19T16:39:15.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:15.108+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:15.112137+0000) 2024-02-19T16:39:16.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:16.108+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:16.112448+0000) 2024-02-19T16:39:16.164 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:39:16.168 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:39:17.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:17.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:17.112712+0000) 2024-02-19T16:39:18.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:18.108+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:18.112854+0000) 2024-02-19T16:39:19.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:19.108+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:19.113077+0000) 2024-02-19T16:39:20.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:20.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:20.113333+0000) 2024-02-19T16:39:21.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:21.108+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:21.113550+0000) 2024-02-19T16:39:22.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:22.108+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:22.113768+0000) 2024-02-19T16:39:23.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:23.108+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:23.114025+0000) 2024-02-19T16:39:24.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:24.108+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:24.114276+0000) 2024-02-19T16:39:25.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:25.108+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:25.114586+0000) 2024-02-19T16:39:26.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:26.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:26.114785+0000) 2024-02-19T16:39:27.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:27.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:27.114965+0000) 2024-02-19T16:39:28.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:28.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:28.115247+0000) 2024-02-19T16:39:29.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:29.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:29.115520+0000) 2024-02-19T16:39:30.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:30.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:30.115730+0000) 2024-02-19T16:39:31.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:31.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:31.115923+0000) 2024-02-19T16:39:32.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:32.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:32.116159+0000) 2024-02-19T16:39:33.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:33.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:33.116423+0000) 2024-02-19T16:39:34.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:34.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:34.116570+0000) 2024-02-19T16:39:35.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:35.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:35.116794+0000) 2024-02-19T16:39:36.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:36.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:36.117042+0000) 2024-02-19T16:39:37.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:37.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:37.117248+0000) 2024-02-19T16:39:38.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:38.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:38.117499+0000) 2024-02-19T16:39:39.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:39.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:39.117750+0000) 2024-02-19T16:39:40.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:40.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:40.118024+0000) 2024-02-19T16:39:41.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:41.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:41.118275+0000) 2024-02-19T16:39:42.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:42.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:42.118423+0000) 2024-02-19T16:39:43.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:43.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:43.118598+0000) 2024-02-19T16:39:44.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:44.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:44.118864+0000) 2024-02-19T16:39:45.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:45.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:45.119111+0000) 2024-02-19T16:39:46.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:46.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:46.119354+0000) 2024-02-19T16:39:46.298 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:39:46.301 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:39:47.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:47.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:47.119589+0000) 2024-02-19T16:39:48.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:48.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:48.119818+0000) 2024-02-19T16:39:49.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:49.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:49.120070+0000) 2024-02-19T16:39:50.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:50.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:50.120219+0000) 2024-02-19T16:39:51.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:51.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:51.120375+0000) 2024-02-19T16:39:52.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:52.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:52.120614+0000) 2024-02-19T16:39:53.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:53.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:53.120819+0000) 2024-02-19T16:39:54.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:54.119+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:54.121000+0000) 2024-02-19T16:39:55.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:55.119+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:55.121251+0000) 2024-02-19T16:39:56.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:56.119+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:56.121450+0000) 2024-02-19T16:39:57.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:57.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:57.121712+0000) 2024-02-19T16:39:58.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:58.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:58.121962+0000) 2024-02-19T16:39:59.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:39:59.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:39:59.122255+0000) 2024-02-19T16:40:00.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:00.119+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:00.122508+0000) 2024-02-19T16:40:01.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:01.119+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:01.122785+0000) 2024-02-19T16:40:02.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:02.119+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:02.122974+0000) 2024-02-19T16:40:03.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:03.119+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:03.123209+0000) 2024-02-19T16:40:04.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:04.119+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:04.123470+0000) 2024-02-19T16:40:05.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:05.119+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:05.123768+0000) 2024-02-19T16:40:06.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:06.119+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:06.124012+0000) 2024-02-19T16:40:07.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:07.119+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:07.124199+0000) 2024-02-19T16:40:08.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:08.119+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:08.124397+0000) 2024-02-19T16:40:09.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:09.123+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:09.124659+0000) 2024-02-19T16:40:10.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:10.123+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:10.124837+0000) 2024-02-19T16:40:11.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:11.123+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:11.125070+0000) 2024-02-19T16:40:12.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:12.123+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:12.125328+0000) 2024-02-19T16:40:13.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:13.123+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:13.125642+0000) 2024-02-19T16:40:14.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:14.119+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:14.125905+0000) 2024-02-19T16:40:15.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:15.119+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:15.126139+0000) 2024-02-19T16:40:16.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:16.119+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:16.126400+0000) 2024-02-19T16:40:16.413 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:40:16.416 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:40:17.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:17.123+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:17.126663+0000) 2024-02-19T16:40:18.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:18.123+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:18.126888+0000) 2024-02-19T16:40:19.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:19.123+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:19.127080+0000) 2024-02-19T16:40:20.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:20.123+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:20.127303+0000) 2024-02-19T16:40:21.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:21.123+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:21.127571+0000) 2024-02-19T16:40:22.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:22.123+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:22.127796+0000) 2024-02-19T16:40:23.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:23.123+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:23.127987+0000) 2024-02-19T16:40:24.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:24.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:24.128202+0000) 2024-02-19T16:40:25.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:25.123+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:25.128469+0000) 2024-02-19T16:40:26.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:26.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:26.128703+0000) 2024-02-19T16:40:27.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:27.123+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:27.128934+0000) 2024-02-19T16:40:28.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:28.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:28.129148+0000) 2024-02-19T16:40:29.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:29.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:29.129392+0000) 2024-02-19T16:40:30.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:30.123+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:30.129579+0000) 2024-02-19T16:40:31.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:31.123+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:31.129796+0000) 2024-02-19T16:40:32.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:32.123+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:32.130034+0000) 2024-02-19T16:40:33.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:33.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:33.130289+0000) 2024-02-19T16:40:34.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:34.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:34.130576+0000) 2024-02-19T16:40:35.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:35.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:35.130771+0000) 2024-02-19T16:40:36.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:36.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:36.130899+0000) 2024-02-19T16:40:37.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:37.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:37.131128+0000) 2024-02-19T16:40:38.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:38.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:38.131382+0000) 2024-02-19T16:40:39.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:39.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:39.131588+0000) 2024-02-19T16:40:40.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:40.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:40.131869+0000) 2024-02-19T16:40:41.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:41.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:41.132051+0000) 2024-02-19T16:40:42.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:42.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:42.132285+0000) 2024-02-19T16:40:43.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:43.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:43.132562+0000) 2024-02-19T16:40:44.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:44.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:44.132865+0000) 2024-02-19T16:40:45.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:45.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:45.133162+0000) 2024-02-19T16:40:46.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:46.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:46.133355+0000) 2024-02-19T16:40:46.468 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:40:46.471 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:40:47.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:47.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:47.133549+0000) 2024-02-19T16:40:48.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:48.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:48.133804+0000) 2024-02-19T16:40:49.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:49.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:49.134059+0000) 2024-02-19T16:40:50.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:50.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:50.134332+0000) 2024-02-19T16:40:51.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:51.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:51.134521+0000) 2024-02-19T16:40:52.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:52.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:52.134688+0000) 2024-02-19T16:40:53.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:53.134+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:53.134939+0000) 2024-02-19T16:40:54.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:54.134+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:54.135115+0000) 2024-02-19T16:40:55.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:55.134+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:55.135351+0000) 2024-02-19T16:40:56.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:56.134+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:56.135554+0000) 2024-02-19T16:40:57.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:57.134+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:57.135731+0000) 2024-02-19T16:40:58.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:58.134+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:58.135975+0000) 2024-02-19T16:40:59.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:40:59.134+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:40:59.136202+0000) 2024-02-19T16:41:00.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:00.134+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:00.136393+0000) 2024-02-19T16:41:01.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:01.134+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:01.136681+0000) 2024-02-19T16:41:02.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:02.134+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:02.136921+0000) 2024-02-19T16:41:03.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:03.134+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:03.137122+0000) 2024-02-19T16:41:04.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:04.134+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:04.137286+0000) 2024-02-19T16:41:05.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:05.134+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:05.137549+0000) 2024-02-19T16:41:06.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:06.134+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:06.137724+0000) 2024-02-19T16:41:07.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:07.134+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:07.137914+0000) 2024-02-19T16:41:08.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:08.134+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:08.138103+0000) 2024-02-19T16:41:09.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:09.134+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:09.138363+0000) 2024-02-19T16:41:10.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:10.134+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:10.138599+0000) 2024-02-19T16:41:11.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:11.138+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:11.138884+0000) 2024-02-19T16:41:12.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:12.138+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:12.139060+0000) 2024-02-19T16:41:13.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:13.138+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:13.139255+0000) 2024-02-19T16:41:14.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:14.138+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:14.139475+0000) 2024-02-19T16:41:15.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:15.138+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:15.139697+0000) 2024-02-19T16:41:16.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:16.138+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:16.139942+0000) 2024-02-19T16:41:16.589 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:41:16.593 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:41:17.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:17.138+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:17.140154+0000) 2024-02-19T16:41:18.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:18.138+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:18.140402+0000) 2024-02-19T16:41:19.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:19.138+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:19.140633+0000) 2024-02-19T16:41:20.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:20.138+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:20.140881+0000) 2024-02-19T16:41:21.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:21.138+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:21.141084+0000) 2024-02-19T16:41:22.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:22.138+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:22.141375+0000) 2024-02-19T16:41:23.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:23.138+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:23.141662+0000) 2024-02-19T16:41:24.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:24.138+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:24.141962+0000) 2024-02-19T16:41:25.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:25.138+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:25.142248+0000) 2024-02-19T16:41:26.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:26.138+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:26.142503+0000) 2024-02-19T16:41:27.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:27.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:27.142829+0000) 2024-02-19T16:41:28.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:28.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:28.143146+0000) 2024-02-19T16:41:29.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:29.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:29.143441+0000) 2024-02-19T16:41:30.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:30.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:30.143724+0000) 2024-02-19T16:41:31.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:31.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:31.143919+0000) 2024-02-19T16:41:32.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:32.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:32.144153+0000) 2024-02-19T16:41:33.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:33.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:33.144331+0000) 2024-02-19T16:41:34.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:34.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:34.144586+0000) 2024-02-19T16:41:35.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:35.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:35.144789+0000) 2024-02-19T16:41:36.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:36.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:36.145066+0000) 2024-02-19T16:41:37.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:37.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:37.145354+0000) 2024-02-19T16:41:38.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:38.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:38.145649+0000) 2024-02-19T16:41:39.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:39.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:39.145872+0000) 2024-02-19T16:41:40.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:40.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:40.146120+0000) 2024-02-19T16:41:41.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:41.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:41.146393+0000) 2024-02-19T16:41:42.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:42.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:42.146695+0000) 2024-02-19T16:41:43.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:43.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:43.146918+0000) 2024-02-19T16:41:44.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:44.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:44.147207+0000) 2024-02-19T16:41:45.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:45.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:45.147417+0000) 2024-02-19T16:41:46.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:46.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:46.147676+0000) 2024-02-19T16:41:46.704 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:41:46.707 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:41:47.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:47.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:47.147975+0000) 2024-02-19T16:41:48.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:48.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:48.148156+0000) 2024-02-19T16:41:49.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:49.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:49.148380+0000) 2024-02-19T16:41:50.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:50.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:50.148604+0000) 2024-02-19T16:41:51.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:51.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:51.148739+0000) 2024-02-19T16:41:52.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:52.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:52.148992+0000) 2024-02-19T16:41:53.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:53.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:53.149278+0000) 2024-02-19T16:41:54.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:54.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:54.149578+0000) 2024-02-19T16:41:55.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:55.149+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:55.149909+0000) 2024-02-19T16:41:56.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:56.149+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:56.150230+0000) 2024-02-19T16:41:57.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:57.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:57.150525+0000) 2024-02-19T16:41:58.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:58.149+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:58.150854+0000) 2024-02-19T16:41:59.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:41:59.149+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:41:59.151054+0000) 2024-02-19T16:42:00.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:00.149+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:00.151227+0000) 2024-02-19T16:42:01.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:01.149+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:01.151514+0000) 2024-02-19T16:42:02.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:02.149+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:02.151729+0000) 2024-02-19T16:42:03.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:03.149+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:03.152023+0000) 2024-02-19T16:42:04.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:04.149+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:04.152307+0000) 2024-02-19T16:42:05.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:05.149+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:05.152601+0000) 2024-02-19T16:42:06.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:06.149+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:06.152772+0000) 2024-02-19T16:42:07.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:07.149+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:07.152974+0000) 2024-02-19T16:42:08.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:08.149+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:08.153230+0000) 2024-02-19T16:42:09.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:09.149+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:09.153392+0000) 2024-02-19T16:42:10.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:10.153+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:10.153678+0000) 2024-02-19T16:42:11.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:11.149+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:11.153957+0000) 2024-02-19T16:42:12.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:12.149+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:12.154148+0000) 2024-02-19T16:42:13.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:13.149+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:13.154359+0000) 2024-02-19T16:42:14.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:14.149+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:14.154634+0000) 2024-02-19T16:42:15.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:15.153+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:15.154845+0000) 2024-02-19T16:42:16.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:16.153+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:16.155061+0000) 2024-02-19T16:42:16.826 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:42:16.830 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:42:17.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:17.153+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:17.155347+0000) 2024-02-19T16:42:18.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:18.153+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:18.155508+0000) 2024-02-19T16:42:19.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:19.153+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:19.155795+0000) 2024-02-19T16:42:20.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:20.153+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:20.156047+0000) 2024-02-19T16:42:21.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:21.153+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:21.156306+0000) 2024-02-19T16:42:22.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:22.153+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:22.156551+0000) 2024-02-19T16:42:23.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:23.153+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:23.156699+0000) 2024-02-19T16:42:24.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:24.153+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:24.156956+0000) 2024-02-19T16:42:25.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:25.153+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:25.157188+0000) 2024-02-19T16:42:26.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:26.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:26.157440+0000) 2024-02-19T16:42:27.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:27.153+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:27.157701+0000) 2024-02-19T16:42:28.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:28.153+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:28.157980+0000) 2024-02-19T16:42:29.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:29.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:29.158179+0000) 2024-02-19T16:42:30.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:30.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:30.158404+0000) 2024-02-19T16:42:31.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:31.153+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:31.158688+0000) 2024-02-19T16:42:32.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:32.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:32.159039+0000) 2024-02-19T16:42:33.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:33.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:33.159263+0000) 2024-02-19T16:42:34.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:34.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:34.159514+0000) 2024-02-19T16:42:35.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:35.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:35.159836+0000) 2024-02-19T16:42:36.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:36.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:36.160138+0000) 2024-02-19T16:42:37.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:37.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:37.160364+0000) 2024-02-19T16:42:38.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:38.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:38.160660+0000) 2024-02-19T16:42:39.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:39.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:39.160939+0000) 2024-02-19T16:42:40.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:40.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:40.161214+0000) 2024-02-19T16:42:41.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:41.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:41.161498+0000) 2024-02-19T16:42:42.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:42.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:42.161726+0000) 2024-02-19T16:42:43.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:43.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:43.162011+0000) 2024-02-19T16:42:44.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:44.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:44.162310+0000) 2024-02-19T16:42:45.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:45.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:45.162606+0000) 2024-02-19T16:42:46.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:46.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:46.162798+0000) 2024-02-19T16:42:46.907 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:42:46.912 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:42:47.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:47.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:47.162930+0000) 2024-02-19T16:42:48.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:48.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:48.163160+0000) 2024-02-19T16:42:49.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:49.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:49.163382+0000) 2024-02-19T16:42:50.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:50.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:50.163550+0000) 2024-02-19T16:42:51.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:51.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:51.163747+0000) 2024-02-19T16:42:52.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:52.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:52.163997+0000) 2024-02-19T16:42:53.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:53.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:53.164274+0000) 2024-02-19T16:42:54.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:54.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:54.164576+0000) 2024-02-19T16:42:55.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:55.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:55.164773+0000) 2024-02-19T16:42:56.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:56.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:56.164998+0000) 2024-02-19T16:42:57.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:57.164+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:57.165176+0000) 2024-02-19T16:42:58.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:58.164+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:58.165407+0000) 2024-02-19T16:42:59.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:42:59.164+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:42:59.165658+0000) 2024-02-19T16:43:00.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:00.164+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:00.165859+0000) 2024-02-19T16:43:01.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:01.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:01.166094+0000) 2024-02-19T16:43:02.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:02.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:02.166325+0000) 2024-02-19T16:43:03.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:03.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:03.166569+0000) 2024-02-19T16:43:04.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:04.164+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:04.166850+0000) 2024-02-19T16:43:05.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:05.164+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:05.166998+0000) 2024-02-19T16:43:06.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:06.164+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:06.167119+0000) 2024-02-19T16:43:07.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:07.164+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:07.167359+0000) 2024-02-19T16:43:08.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:08.164+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:08.167596+0000) 2024-02-19T16:43:09.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:09.164+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:09.167849+0000) 2024-02-19T16:43:10.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:10.164+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:10.168105+0000) 2024-02-19T16:43:11.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:11.164+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:11.168238+0000) 2024-02-19T16:43:12.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:12.164+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:12.168368+0000) 2024-02-19T16:43:13.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:13.164+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:13.168597+0000) 2024-02-19T16:43:14.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:14.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:14.168797+0000) 2024-02-19T16:43:15.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:15.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:15.169054+0000) 2024-02-19T16:43:16.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:16.164+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:16.169279+0000) 2024-02-19T16:43:17.021 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:43:17.025 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:43:17.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:17.164+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:17.169410+0000) 2024-02-19T16:43:18.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:18.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:18.169664+0000) 2024-02-19T16:43:19.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:19.164+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:19.169809+0000) 2024-02-19T16:43:20.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:20.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:20.170042+0000) 2024-02-19T16:43:21.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:21.164+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:21.170215+0000) 2024-02-19T16:43:22.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:22.164+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:22.170434+0000) 2024-02-19T16:43:23.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:23.164+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:23.170633+0000) 2024-02-19T16:43:24.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:24.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:24.170878+0000) 2024-02-19T16:43:25.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:25.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:25.171134+0000) 2024-02-19T16:43:26.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:26.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:26.171379+0000) 2024-02-19T16:43:27.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:27.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:27.171557+0000) 2024-02-19T16:43:28.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:28.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:28.171753+0000) 2024-02-19T16:43:29.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:29.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:29.171958+0000) 2024-02-19T16:43:30.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:30.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:30.172219+0000) 2024-02-19T16:43:31.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:31.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:31.172422+0000) 2024-02-19T16:43:32.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:32.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:32.172688+0000) 2024-02-19T16:43:33.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:33.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:33.172934+0000) 2024-02-19T16:43:34.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:34.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:34.173110+0000) 2024-02-19T16:43:35.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:35.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:35.173261+0000) 2024-02-19T16:43:36.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:36.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:36.173542+0000) 2024-02-19T16:43:37.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:37.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:37.173841+0000) 2024-02-19T16:43:38.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:38.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:38.174131+0000) 2024-02-19T16:43:39.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:39.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:39.174427+0000) 2024-02-19T16:43:40.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:40.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:40.174658+0000) 2024-02-19T16:43:41.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:41.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:41.174898+0000) 2024-02-19T16:43:42.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:42.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:42.175188+0000) 2024-02-19T16:43:43.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:43.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:43.175350+0000) 2024-02-19T16:43:44.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:44.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:44.175541+0000) 2024-02-19T16:43:45.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:45.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:45.175840+0000) 2024-02-19T16:43:46.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:46.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:46.176062+0000) 2024-02-19T16:43:47.151 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:43:47.155 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:43:47.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:47.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:47.176283+0000) 2024-02-19T16:43:48.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:48.175+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:48.176572+0000) 2024-02-19T16:43:49.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:49.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:49.176871+0000) 2024-02-19T16:43:50.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:50.175+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:50.177184+0000) 2024-02-19T16:43:51.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:51.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:51.177423+0000) 2024-02-19T16:43:52.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:52.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:52.177624+0000) 2024-02-19T16:43:53.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:53.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:53.177879+0000) 2024-02-19T16:43:54.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:54.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:54.178168+0000) 2024-02-19T16:43:55.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:55.175+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:55.178458+0000) 2024-02-19T16:43:56.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:56.175+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:56.178762+0000) 2024-02-19T16:43:57.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:57.175+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:57.179051+0000) 2024-02-19T16:43:58.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:58.175+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:58.179366+0000) 2024-02-19T16:43:59.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:43:59.175+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:43:59.179562+0000) 2024-02-19T16:44:00.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:00.175+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:00.179836+0000) 2024-02-19T16:44:01.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:01.175+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:01.180119+0000) 2024-02-19T16:44:02.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:02.179+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:02.180362+0000) 2024-02-19T16:44:03.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:03.179+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:03.180648+0000) 2024-02-19T16:44:04.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:04.179+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:04.180925+0000) 2024-02-19T16:44:05.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:05.179+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:05.181259+0000) 2024-02-19T16:44:06.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:06.175+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:06.181544+0000) 2024-02-19T16:44:07.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:07.175+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:07.181717+0000) 2024-02-19T16:44:08.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:08.179+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:08.181973+0000) 2024-02-19T16:44:09.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:09.179+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:09.182222+0000) 2024-02-19T16:44:10.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:10.179+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:10.182494+0000) 2024-02-19T16:44:11.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:11.179+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:11.182746+0000) 2024-02-19T16:44:12.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:12.179+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:12.182928+0000) 2024-02-19T16:44:13.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:13.179+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:13.183166+0000) 2024-02-19T16:44:14.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:14.179+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:14.183306+0000) 2024-02-19T16:44:15.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:15.179+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:15.183510+0000) 2024-02-19T16:44:16.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:16.179+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:16.183761+0000) 2024-02-19T16:44:17.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:17.179+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:17.183994+0000) 2024-02-19T16:44:17.273 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:44:17.276 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:44:18.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:18.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:18.184234+0000) 2024-02-19T16:44:19.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:19.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:19.184461+0000) 2024-02-19T16:44:20.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:20.179+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:20.184702+0000) 2024-02-19T16:44:21.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:21.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:21.184947+0000) 2024-02-19T16:44:22.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:22.179+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:22.185182+0000) 2024-02-19T16:44:23.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:23.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:23.185390+0000) 2024-02-19T16:44:24.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:24.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:24.185599+0000) 2024-02-19T16:44:25.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:25.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:25.185882+0000) 2024-02-19T16:44:26.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:26.179+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:26.186116+0000) 2024-02-19T16:44:27.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:27.179+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:27.186352+0000) 2024-02-19T16:44:28.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:28.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:28.186518+0000) 2024-02-19T16:44:29.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:29.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:29.186860+0000) 2024-02-19T16:44:30.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:30.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:30.187050+0000) 2024-02-19T16:44:31.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:31.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:31.187191+0000) 2024-02-19T16:44:32.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:32.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:32.187472+0000) 2024-02-19T16:44:33.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:33.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:33.187729+0000) 2024-02-19T16:44:34.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:34.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:34.188021+0000) 2024-02-19T16:44:35.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:35.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:35.188361+0000) 2024-02-19T16:44:36.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:36.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:36.188581+0000) 2024-02-19T16:44:37.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:37.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:37.188758+0000) 2024-02-19T16:44:38.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:38.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:38.188994+0000) 2024-02-19T16:44:39.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:39.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:39.189136+0000) 2024-02-19T16:44:40.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:40.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:40.189428+0000) 2024-02-19T16:44:41.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:41.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:41.189713+0000) 2024-02-19T16:44:42.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:42.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:42.190007+0000) 2024-02-19T16:44:43.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:43.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:43.190325+0000) 2024-02-19T16:44:44.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:44.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:44.190574+0000) 2024-02-19T16:44:45.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:45.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:45.190797+0000) 2024-02-19T16:44:46.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:46.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:46.190956+0000) 2024-02-19T16:44:47.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:47.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:47.191140+0000) 2024-02-19T16:44:47.379 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:44:47.383 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:44:48.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:48.190+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:48.191313+0000) 2024-02-19T16:44:49.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:49.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:49.191509+0000) 2024-02-19T16:44:50.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:50.190+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:50.191769+0000) 2024-02-19T16:44:51.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:51.190+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:51.192011+0000) 2024-02-19T16:44:52.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:52.190+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:52.192288+0000) 2024-02-19T16:44:53.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:53.190+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:53.192545+0000) 2024-02-19T16:44:54.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:54.190+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:54.192835+0000) 2024-02-19T16:44:55.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:55.190+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:55.193159+0000) 2024-02-19T16:44:56.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:56.190+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:56.193479+0000) 2024-02-19T16:44:57.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:57.190+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:57.193709+0000) 2024-02-19T16:44:58.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:58.190+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:58.193910+0000) 2024-02-19T16:44:59.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:44:59.190+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:44:59.194167+0000) 2024-02-19T16:45:00.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:00.190+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:00.194448+0000) 2024-02-19T16:45:01.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:01.190+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:01.194674+0000) 2024-02-19T16:45:02.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:02.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:02.194916+0000) 2024-02-19T16:45:03.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:03.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:03.195158+0000) 2024-02-19T16:45:04.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:04.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:04.195403+0000) 2024-02-19T16:45:05.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:05.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:05.195691+0000) 2024-02-19T16:45:06.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:06.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:06.195905+0000) 2024-02-19T16:45:07.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:07.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:07.196171+0000) 2024-02-19T16:45:08.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:08.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:08.196309+0000) 2024-02-19T16:45:09.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:09.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:09.196500+0000) 2024-02-19T16:45:10.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:10.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:10.196730+0000) 2024-02-19T16:45:11.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:11.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:11.196956+0000) 2024-02-19T16:45:12.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:12.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:12.197181+0000) 2024-02-19T16:45:13.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:13.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:13.197458+0000) 2024-02-19T16:45:14.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:14.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:14.197662+0000) 2024-02-19T16:45:15.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:15.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:15.197956+0000) 2024-02-19T16:45:16.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:16.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:16.198240+0000) 2024-02-19T16:45:17.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:17.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:17.198438+0000) 2024-02-19T16:45:17.459 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:45:17.463 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:45:18.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:18.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:18.198591+0000) 2024-02-19T16:45:19.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:19.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:19.198847+0000) 2024-02-19T16:45:20.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:20.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:20.199005+0000) 2024-02-19T16:45:21.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:21.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:21.199190+0000) 2024-02-19T16:45:22.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:22.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:22.199407+0000) 2024-02-19T16:45:23.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:23.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:23.199617+0000) 2024-02-19T16:45:24.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:24.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:24.199852+0000) 2024-02-19T16:45:25.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:25.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:25.200129+0000) 2024-02-19T16:45:26.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:26.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:26.200280+0000) 2024-02-19T16:45:27.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:27.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:27.200498+0000) 2024-02-19T16:45:28.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:28.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:28.200737+0000) 2024-02-19T16:45:29.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:29.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:29.200973+0000) 2024-02-19T16:45:30.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:30.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:30.201221+0000) 2024-02-19T16:45:31.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:31.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:31.201375+0000) 2024-02-19T16:45:32.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:32.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:32.201638+0000) 2024-02-19T16:45:33.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:33.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:33.201982+0000) 2024-02-19T16:45:34.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:34.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:34.202247+0000) 2024-02-19T16:45:35.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:35.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:35.202534+0000) 2024-02-19T16:45:36.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:36.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:36.202762+0000) 2024-02-19T16:45:37.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:37.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:37.202989+0000) 2024-02-19T16:45:38.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:38.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:38.203182+0000) 2024-02-19T16:45:39.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:39.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:39.203409+0000) 2024-02-19T16:45:40.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:40.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:40.203646+0000) 2024-02-19T16:45:41.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:41.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:41.203836+0000) 2024-02-19T16:45:42.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:42.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:42.204008+0000) 2024-02-19T16:45:43.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:43.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:43.204237+0000) 2024-02-19T16:45:44.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:44.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:44.204422+0000) 2024-02-19T16:45:45.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:45.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:45.204605+0000) 2024-02-19T16:45:46.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:46.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:46.204846+0000) 2024-02-19T16:45:47.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:47.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:47.205022+0000) 2024-02-19T16:45:47.741 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:45:47.745 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:45:48.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:48.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:48.205270+0000) 2024-02-19T16:45:49.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:49.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:49.205512+0000) 2024-02-19T16:45:50.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:50.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:50.205687+0000) 2024-02-19T16:45:51.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:51.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:51.205925+0000) 2024-02-19T16:45:52.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:52.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:52.206163+0000) 2024-02-19T16:45:53.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:53.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:53.206397+0000) 2024-02-19T16:45:54.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:54.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:54.206652+0000) 2024-02-19T16:45:55.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:55.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:55.206900+0000) 2024-02-19T16:45:56.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:56.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:56.207079+0000) 2024-02-19T16:45:57.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:57.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:57.207343+0000) 2024-02-19T16:45:58.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:58.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:58.207573+0000) 2024-02-19T16:45:59.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:45:59.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:45:59.207810+0000) 2024-02-19T16:46:00.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:00.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:00.208060+0000) 2024-02-19T16:46:01.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:01.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:01.208256+0000) 2024-02-19T16:46:02.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:02.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:02.208498+0000) 2024-02-19T16:46:03.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:03.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:03.208681+0000) 2024-02-19T16:46:04.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:04.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:04.208864+0000) 2024-02-19T16:46:05.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:05.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:05.209071+0000) 2024-02-19T16:46:06.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:06.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:06.209326+0000) 2024-02-19T16:46:07.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:07.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:07.209567+0000) 2024-02-19T16:46:08.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:08.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:08.209758+0000) 2024-02-19T16:46:09.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:09.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:09.209965+0000) 2024-02-19T16:46:10.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:10.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:10.210187+0000) 2024-02-19T16:46:11.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:11.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:11.210410+0000) 2024-02-19T16:46:12.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:12.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:12.210643+0000) 2024-02-19T16:46:13.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:13.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:13.210836+0000) 2024-02-19T16:46:14.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:14.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:14.211029+0000) 2024-02-19T16:46:15.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:15.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:15.211228+0000) 2024-02-19T16:46:16.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:16.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:16.211367+0000) 2024-02-19T16:46:17.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:17.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:17.211610+0000) 2024-02-19T16:46:17.813 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:46:17.816 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:46:18.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:18.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:18.211749+0000) 2024-02-19T16:46:19.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:19.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:19.211992+0000) 2024-02-19T16:46:20.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:20.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:20.212231+0000) 2024-02-19T16:46:21.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:21.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:21.212393+0000) 2024-02-19T16:46:22.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:22.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:22.212605+0000) 2024-02-19T16:46:23.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:23.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:23.212848+0000) 2024-02-19T16:46:24.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:24.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:24.213044+0000) 2024-02-19T16:46:25.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:25.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:25.213289+0000) 2024-02-19T16:46:26.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:26.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:26.213529+0000) 2024-02-19T16:46:27.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:27.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:27.213729+0000) 2024-02-19T16:46:28.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:28.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:28.213972+0000) 2024-02-19T16:46:29.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:29.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:29.214213+0000) 2024-02-19T16:46:30.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:30.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:30.214451+0000) 2024-02-19T16:46:31.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:31.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:31.214743+0000) 2024-02-19T16:46:32.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:32.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:32.214948+0000) 2024-02-19T16:46:33.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:33.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:33.215197+0000) 2024-02-19T16:46:34.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:34.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:34.215384+0000) 2024-02-19T16:46:35.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:35.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:35.215643+0000) 2024-02-19T16:46:36.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:36.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:36.215895+0000) 2024-02-19T16:46:37.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:37.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:37.216099+0000) 2024-02-19T16:46:38.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:38.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:38.216279+0000) 2024-02-19T16:46:39.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:39.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:39.216550+0000) 2024-02-19T16:46:40.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:40.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:40.216781+0000) 2024-02-19T16:46:41.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:41.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:41.217019+0000) 2024-02-19T16:46:42.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:42.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:42.217220+0000) 2024-02-19T16:46:43.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:43.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:43.217452+0000) 2024-02-19T16:46:44.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:44.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:44.217694+0000) 2024-02-19T16:46:45.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:45.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:45.217963+0000) 2024-02-19T16:46:46.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:46.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:46.218163+0000) 2024-02-19T16:46:47.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:47.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:47.218434+0000) 2024-02-19T16:46:47.918 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:46:47.922 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:46:48.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:48.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:48.218638+0000) 2024-02-19T16:46:49.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:49.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:49.218905+0000) 2024-02-19T16:46:50.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:50.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:50.219034+0000) 2024-02-19T16:46:51.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:51.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:51.219215+0000) 2024-02-19T16:46:52.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:52.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:52.219464+0000) 2024-02-19T16:46:53.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:53.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:53.219706+0000) 2024-02-19T16:46:54.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:54.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:54.219950+0000) 2024-02-19T16:46:55.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:55.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:55.220195+0000) 2024-02-19T16:46:56.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:56.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:56.220470+0000) 2024-02-19T16:46:57.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:57.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:57.220707+0000) 2024-02-19T16:46:58.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:58.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:58.220948+0000) 2024-02-19T16:46:59.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:46:59.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:46:59.221185+0000) 2024-02-19T16:47:00.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:00.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:00.221430+0000) 2024-02-19T16:47:01.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:01.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:01.221665+0000) 2024-02-19T16:47:02.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:02.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:02.221800+0000) 2024-02-19T16:47:03.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:03.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:03.222060+0000) 2024-02-19T16:47:04.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:04.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:04.222279+0000) 2024-02-19T16:47:05.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:05.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:05.222526+0000) 2024-02-19T16:47:06.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:06.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:06.222753+0000) 2024-02-19T16:47:07.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:07.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:07.222921+0000) 2024-02-19T16:47:08.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:08.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:08.223093+0000) 2024-02-19T16:47:09.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:09.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:09.223310+0000) 2024-02-19T16:47:10.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:10.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:10.223482+0000) 2024-02-19T16:47:11.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:11.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:11.223657+0000) 2024-02-19T16:47:12.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:12.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:12.223872+0000) 2024-02-19T16:47:13.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:13.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:13.224069+0000) 2024-02-19T16:47:14.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:14.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:14.224297+0000) 2024-02-19T16:47:15.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:15.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:15.224547+0000) 2024-02-19T16:47:16.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:16.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:16.224793+0000) 2024-02-19T16:47:17.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:17.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:17.225024+0000) 2024-02-19T16:47:18.051 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:47:18.056 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:47:18.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:18.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:18.225285+0000) 2024-02-19T16:47:19.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:19.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:19.225555+0000) 2024-02-19T16:47:20.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:20.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:20.225767+0000) 2024-02-19T16:47:21.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:21.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:21.225968+0000) 2024-02-19T16:47:22.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:22.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:22.226161+0000) 2024-02-19T16:47:23.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:23.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:23.226348+0000) 2024-02-19T16:47:24.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:24.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:24.226583+0000) 2024-02-19T16:47:25.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:25.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:25.226761+0000) 2024-02-19T16:47:26.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:26.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:26.226936+0000) 2024-02-19T16:47:27.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:27.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:27.227124+0000) 2024-02-19T16:47:28.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:28.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:28.227376+0000) 2024-02-19T16:47:29.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:29.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:29.227618+0000) 2024-02-19T16:47:30.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:30.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:30.227869+0000) 2024-02-19T16:47:31.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:31.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:31.228044+0000) 2024-02-19T16:47:32.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:32.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:32.228265+0000) 2024-02-19T16:47:33.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:33.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:33.228482+0000) 2024-02-19T16:47:34.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:34.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:34.228684+0000) 2024-02-19T16:47:35.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:35.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:35.228940+0000) 2024-02-19T16:47:36.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:36.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:36.229196+0000) 2024-02-19T16:47:37.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:37.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:37.229401+0000) 2024-02-19T16:47:38.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:38.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:38.229657+0000) 2024-02-19T16:47:39.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:39.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:39.229897+0000) 2024-02-19T16:47:40.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:40.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:40.230126+0000) 2024-02-19T16:47:41.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:41.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:41.230364+0000) 2024-02-19T16:47:42.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:42.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:42.230572+0000) 2024-02-19T16:47:43.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:43.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:43.230827+0000) 2024-02-19T16:47:44.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:44.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:44.231062+0000) 2024-02-19T16:47:45.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:45.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:45.231327+0000) 2024-02-19T16:47:46.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:46.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:46.231585+0000) 2024-02-19T16:47:47.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:47.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:47.231818+0000) 2024-02-19T16:47:48.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:48.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:48.232078+0000) 2024-02-19T16:47:48.390 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:47:48.394 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:47:49.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:49.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:49.232328+0000) 2024-02-19T16:47:50.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:50.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:50.232568+0000) 2024-02-19T16:47:51.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:51.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:51.232826+0000) 2024-02-19T16:47:52.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:52.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:52.233026+0000) 2024-02-19T16:47:53.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:53.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:53.233278+0000) 2024-02-19T16:47:54.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:54.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:54.233519+0000) 2024-02-19T16:47:55.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:55.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:55.233765+0000) 2024-02-19T16:47:56.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:56.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:56.233979+0000) 2024-02-19T16:47:57.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:57.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:57.234165+0000) 2024-02-19T16:47:58.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:58.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:58.234435+0000) 2024-02-19T16:47:59.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:47:59.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:47:59.234629+0000) 2024-02-19T16:48:00.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:00.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:00.234849+0000) 2024-02-19T16:48:01.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:01.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:01.235076+0000) 2024-02-19T16:48:02.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:02.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:02.235235+0000) 2024-02-19T16:48:03.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:03.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:03.235461+0000) 2024-02-19T16:48:04.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:04.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:04.235713+0000) 2024-02-19T16:48:05.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:05.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:05.235883+0000) 2024-02-19T16:48:06.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:06.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:06.236152+0000) 2024-02-19T16:48:07.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:07.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:07.236389+0000) 2024-02-19T16:48:08.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:08.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:08.236622+0000) 2024-02-19T16:48:09.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:09.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:09.236812+0000) 2024-02-19T16:48:10.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:10.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:10.236993+0000) 2024-02-19T16:48:11.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:11.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:11.237228+0000) 2024-02-19T16:48:12.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:12.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:12.237428+0000) 2024-02-19T16:48:13.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:13.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:13.237657+0000) 2024-02-19T16:48:14.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:14.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:14.237863+0000) 2024-02-19T16:48:15.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:15.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:15.238091+0000) 2024-02-19T16:48:16.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:16.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:16.238345+0000) 2024-02-19T16:48:17.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:17.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:17.238575+0000) 2024-02-19T16:48:18.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:18.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:18.238828+0000) 2024-02-19T16:48:18.504 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:48:18.507 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:48:19.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:19.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:19.238975+0000) 2024-02-19T16:48:20.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:20.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:20.239152+0000) 2024-02-19T16:48:21.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:21.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:21.239332+0000) 2024-02-19T16:48:22.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:22.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:22.239575+0000) 2024-02-19T16:48:23.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:23.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:23.239775+0000) 2024-02-19T16:48:24.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:24.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:24.240041+0000) 2024-02-19T16:48:25.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:25.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:25.240298+0000) 2024-02-19T16:48:26.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:26.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:26.240569+0000) 2024-02-19T16:48:27.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:27.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:27.240850+0000) 2024-02-19T16:48:28.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:28.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:28.241109+0000) 2024-02-19T16:48:29.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:29.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:29.241293+0000) 2024-02-19T16:48:30.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:30.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:30.241463+0000) 2024-02-19T16:48:31.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:31.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:31.241701+0000) 2024-02-19T16:48:32.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:32.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:32.241939+0000) 2024-02-19T16:48:33.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:33.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:33.242115+0000) 2024-02-19T16:48:34.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:34.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:34.242362+0000) 2024-02-19T16:48:35.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:35.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:35.242613+0000) 2024-02-19T16:48:36.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:36.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:36.242818+0000) 2024-02-19T16:48:37.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:37.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:37.242993+0000) 2024-02-19T16:48:38.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:38.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:38.243198+0000) 2024-02-19T16:48:39.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:39.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:39.243398+0000) 2024-02-19T16:48:40.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:40.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:40.243656+0000) 2024-02-19T16:48:41.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:41.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:41.243856+0000) 2024-02-19T16:48:42.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:42.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:42.244102+0000) 2024-02-19T16:48:43.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:43.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:43.244290+0000) 2024-02-19T16:48:44.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:44.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:44.244530+0000) 2024-02-19T16:48:45.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:45.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:45.244847+0000) 2024-02-19T16:48:46.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:46.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:46.245085+0000) 2024-02-19T16:48:47.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:47.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:47.245350+0000) 2024-02-19T16:48:48.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:48.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:48.245582+0000) 2024-02-19T16:48:48.642 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:48:48.646 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:48:49.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:49.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:49.245769+0000) 2024-02-19T16:48:50.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:50.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:50.245943+0000) 2024-02-19T16:48:51.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:51.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:51.246181+0000) 2024-02-19T16:48:52.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:52.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:52.246469+0000) 2024-02-19T16:48:53.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:53.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:53.246691+0000) 2024-02-19T16:48:54.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:54.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:54.246948+0000) 2024-02-19T16:48:55.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:55.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:55.247185+0000) 2024-02-19T16:48:56.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:56.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:56.247390+0000) 2024-02-19T16:48:57.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:57.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:57.247652+0000) 2024-02-19T16:48:58.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:58.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:58.247935+0000) 2024-02-19T16:48:59.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:48:59.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:48:59.248215+0000) 2024-02-19T16:49:00.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:00.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:00.248479+0000) 2024-02-19T16:49:01.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:01.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:01.248730+0000) 2024-02-19T16:49:02.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:02.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:02.248991+0000) 2024-02-19T16:49:03.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:03.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:03.249206+0000) 2024-02-19T16:49:04.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:04.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:04.249451+0000) 2024-02-19T16:49:05.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:05.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:05.249713+0000) 2024-02-19T16:49:06.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:06.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:06.249956+0000) 2024-02-19T16:49:07.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:07.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:07.250245+0000) 2024-02-19T16:49:08.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:08.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:08.250536+0000) 2024-02-19T16:49:09.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:09.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:09.250746+0000) 2024-02-19T16:49:10.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:10.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:10.250879+0000) 2024-02-19T16:49:11.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:11.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:11.251053+0000) 2024-02-19T16:49:12.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:12.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:12.251257+0000) 2024-02-19T16:49:13.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:13.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:13.251436+0000) 2024-02-19T16:49:14.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:14.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:14.251721+0000) 2024-02-19T16:49:15.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:15.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:15.251952+0000) 2024-02-19T16:49:16.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:16.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:16.252146+0000) 2024-02-19T16:49:17.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:17.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:17.252387+0000) 2024-02-19T16:49:18.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:18.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:18.252674+0000) 2024-02-19T16:49:18.764 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:49:18.767 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:49:19.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:19.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:19.252928+0000) 2024-02-19T16:49:20.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:20.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:20.253205+0000) 2024-02-19T16:49:21.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:21.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:21.253440+0000) 2024-02-19T16:49:22.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:22.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:22.253643+0000) 2024-02-19T16:49:23.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:23.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:23.253891+0000) 2024-02-19T16:49:24.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:24.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:24.254121+0000) 2024-02-19T16:49:25.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:25.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:25.254381+0000) 2024-02-19T16:49:26.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:26.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:26.254617+0000) 2024-02-19T16:49:27.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:27.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:27.254850+0000) 2024-02-19T16:49:28.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:28.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:28.255038+0000) 2024-02-19T16:49:29.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:29.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:29.255271+0000) 2024-02-19T16:49:30.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:30.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:30.255526+0000) 2024-02-19T16:49:31.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:31.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:31.255725+0000) 2024-02-19T16:49:32.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:32.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:32.255949+0000) 2024-02-19T16:49:33.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:33.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:33.256217+0000) 2024-02-19T16:49:34.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:34.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:34.256461+0000) 2024-02-19T16:49:35.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:35.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:35.256732+0000) 2024-02-19T16:49:36.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:36.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:36.256976+0000) 2024-02-19T16:49:37.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:37.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:37.257155+0000) 2024-02-19T16:49:38.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:38.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:38.257360+0000) 2024-02-19T16:49:39.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:39.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:39.257606+0000) 2024-02-19T16:49:40.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:40.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:40.257812+0000) 2024-02-19T16:49:41.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:41.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:41.257997+0000) 2024-02-19T16:49:42.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:42.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:42.258215+0000) 2024-02-19T16:49:43.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:43.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:43.258443+0000) 2024-02-19T16:49:44.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:44.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:44.258685+0000) 2024-02-19T16:49:45.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:45.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:45.258934+0000) 2024-02-19T16:49:46.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:46.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:46.259177+0000) 2024-02-19T16:49:47.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:47.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:47.259434+0000) 2024-02-19T16:49:48.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:48.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:48.259684+0000) 2024-02-19T16:49:48.904 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:49:48.909 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:49:49.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:49.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:49.259834+0000) 2024-02-19T16:49:50.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:50.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:50.260070+0000) 2024-02-19T16:49:51.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:51.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:51.260265+0000) 2024-02-19T16:49:52.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:52.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:52.260442+0000) 2024-02-19T16:49:53.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:53.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:53.260691+0000) 2024-02-19T16:49:54.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:54.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:54.260887+0000) 2024-02-19T16:49:55.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:55.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:55.261177+0000) 2024-02-19T16:49:56.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:56.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:56.261367+0000) 2024-02-19T16:49:57.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:57.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:57.261649+0000) 2024-02-19T16:49:58.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:58.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:58.261880+0000) 2024-02-19T16:49:59.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:49:59.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:49:59.262128+0000) 2024-02-19T16:50:00.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:00.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:00.262328+0000) 2024-02-19T16:50:01.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:01.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:01.262582+0000) 2024-02-19T16:50:02.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:02.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:02.262827+0000) 2024-02-19T16:50:03.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:03.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:03.263013+0000) 2024-02-19T16:50:04.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:04.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:04.263187+0000) 2024-02-19T16:50:05.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:05.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:05.263447+0000) 2024-02-19T16:50:06.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:06.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:06.263728+0000) 2024-02-19T16:50:07.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:07.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:07.263953+0000) 2024-02-19T16:50:08.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:08.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:08.264195+0000) 2024-02-19T16:50:09.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:09.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:09.264447+0000) 2024-02-19T16:50:10.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:10.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:10.264690+0000) 2024-02-19T16:50:11.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:11.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:11.264871+0000) 2024-02-19T16:50:12.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:12.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:12.265145+0000) 2024-02-19T16:50:13.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:13.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:13.265308+0000) 2024-02-19T16:50:14.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:14.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:14.265555+0000) 2024-02-19T16:50:15.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:15.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:15.265752+0000) 2024-02-19T16:50:16.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:16.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:16.266029+0000) 2024-02-19T16:50:17.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:17.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:17.266264+0000) 2024-02-19T16:50:18.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:18.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:18.266498+0000) 2024-02-19T16:50:19.017 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:50:19.020 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:50:19.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:19.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:19.266807+0000) 2024-02-19T16:50:20.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:20.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:20.267019+0000) 2024-02-19T16:50:21.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:21.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:21.267277+0000) 2024-02-19T16:50:22.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:22.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:22.267518+0000) 2024-02-19T16:50:23.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:23.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:23.267787+0000) 2024-02-19T16:50:24.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:24.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:24.268036+0000) 2024-02-19T16:50:25.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:25.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:25.268223+0000) 2024-02-19T16:50:26.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:26.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:26.268460+0000) 2024-02-19T16:50:27.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:27.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:27.268748+0000) 2024-02-19T16:50:28.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:28.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:28.268986+0000) 2024-02-19T16:50:29.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:29.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:29.269247+0000) 2024-02-19T16:50:30.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:30.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:30.269484+0000) 2024-02-19T16:50:31.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:31.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:31.269754+0000) 2024-02-19T16:50:32.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:32.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:32.270010+0000) 2024-02-19T16:50:33.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:33.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:33.270284+0000) 2024-02-19T16:50:34.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:34.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:34.270521+0000) 2024-02-19T16:50:35.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:35.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:35.270846+0000) 2024-02-19T16:50:36.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:36.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:36.271043+0000) 2024-02-19T16:50:37.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:37.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:37.271301+0000) 2024-02-19T16:50:38.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:38.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:38.271586+0000) 2024-02-19T16:50:39.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:39.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:39.271844+0000) 2024-02-19T16:50:40.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:40.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:40.272028+0000) 2024-02-19T16:50:41.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:41.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:41.272273+0000) 2024-02-19T16:50:42.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:42.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:42.272449+0000) 2024-02-19T16:50:43.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:43.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:43.272656+0000) 2024-02-19T16:50:44.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:44.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:44.272910+0000) 2024-02-19T16:50:45.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:45.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:45.273181+0000) 2024-02-19T16:50:46.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:46.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:46.273415+0000) 2024-02-19T16:50:47.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:47.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:47.273686+0000) 2024-02-19T16:50:48.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:48.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:48.273880+0000) 2024-02-19T16:50:49.123 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:50:49.127 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:50:49.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:49.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:49.274136+0000) 2024-02-19T16:50:50.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:50.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:50.274393+0000) 2024-02-19T16:50:51.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:51.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:51.274582+0000) 2024-02-19T16:50:52.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:52.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:52.274860+0000) 2024-02-19T16:50:53.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:53.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:53.275095+0000) 2024-02-19T16:50:54.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:54.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:54.275316+0000) 2024-02-19T16:50:55.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:55.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:55.275599+0000) 2024-02-19T16:50:56.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:56.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:56.275850+0000) 2024-02-19T16:50:57.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:57.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:57.276085+0000) 2024-02-19T16:50:58.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:58.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:58.276314+0000) 2024-02-19T16:50:59.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:50:59.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:50:59.276491+0000) 2024-02-19T16:51:00.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:00.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:00.276728+0000) 2024-02-19T16:51:01.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:01.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:01.277036+0000) 2024-02-19T16:51:02.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:02.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:02.277335+0000) 2024-02-19T16:51:03.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:03.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:03.277646+0000) 2024-02-19T16:51:04.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:04.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:04.277891+0000) 2024-02-19T16:51:05.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:05.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:05.278140+0000) 2024-02-19T16:51:06.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:06.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:06.278403+0000) 2024-02-19T16:51:07.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:07.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:07.278623+0000) 2024-02-19T16:51:08.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:08.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:08.278863+0000) 2024-02-19T16:51:09.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:09.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:09.279135+0000) 2024-02-19T16:51:10.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:10.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:10.279368+0000) 2024-02-19T16:51:11.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:11.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:11.279503+0000) 2024-02-19T16:51:12.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:12.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:12.279741+0000) 2024-02-19T16:51:13.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:13.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:13.279975+0000) 2024-02-19T16:51:14.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:14.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:14.280107+0000) 2024-02-19T16:51:15.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:15.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:15.280367+0000) 2024-02-19T16:51:16.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:16.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:16.280533+0000) 2024-02-19T16:51:17.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:17.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:17.280758+0000) 2024-02-19T16:51:18.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:18.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:18.280931+0000) 2024-02-19T16:51:19.192 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:51:19.198 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:51:19.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:19.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:19.281195+0000) 2024-02-19T16:51:20.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:20.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:20.281395+0000) 2024-02-19T16:51:21.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:21.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:21.281610+0000) 2024-02-19T16:51:22.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:22.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:22.281829+0000) 2024-02-19T16:51:23.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:23.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:23.282092+0000) 2024-02-19T16:51:24.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:24.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:24.282307+0000) 2024-02-19T16:51:25.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:25.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:25.282533+0000) 2024-02-19T16:51:26.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:26.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:26.282697+0000) 2024-02-19T16:51:27.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:27.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:27.282923+0000) 2024-02-19T16:51:28.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:28.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:28.283071+0000) 2024-02-19T16:51:29.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:29.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:29.283196+0000) 2024-02-19T16:51:30.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:30.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:30.283428+0000) 2024-02-19T16:51:31.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:31.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:31.283663+0000) 2024-02-19T16:51:32.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:32.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:32.283858+0000) 2024-02-19T16:51:33.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:33.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:33.284022+0000) 2024-02-19T16:51:34.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:34.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:34.284250+0000) 2024-02-19T16:51:35.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:35.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:35.284539+0000) 2024-02-19T16:51:36.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:36.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:36.284732+0000) 2024-02-19T16:51:37.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:37.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:37.284958+0000) 2024-02-19T16:51:38.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:38.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:38.285204+0000) 2024-02-19T16:51:39.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:39.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:39.285344+0000) 2024-02-19T16:51:40.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:40.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:40.285544+0000) 2024-02-19T16:51:41.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:41.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:41.285804+0000) 2024-02-19T16:51:42.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:42.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:42.285984+0000) 2024-02-19T16:51:43.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:43.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:43.286223+0000) 2024-02-19T16:51:44.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:44.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:44.286450+0000) 2024-02-19T16:51:45.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:45.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:45.286760+0000) 2024-02-19T16:51:46.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:46.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:46.286933+0000) 2024-02-19T16:51:47.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:47.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:47.287133+0000) 2024-02-19T16:51:48.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:48.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:48.287313+0000) 2024-02-19T16:51:49.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:49.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:49.287449+0000) 2024-02-19T16:51:49.301 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:51:49.304 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:51:50.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:50.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:50.287651+0000) 2024-02-19T16:51:51.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:51.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:51.287930+0000) 2024-02-19T16:51:52.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:52.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:52.288190+0000) 2024-02-19T16:51:53.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:53.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:53.288435+0000) 2024-02-19T16:51:54.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:54.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:54.288687+0000) 2024-02-19T16:51:55.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:55.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:55.288988+0000) 2024-02-19T16:51:56.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:56.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:56.289264+0000) 2024-02-19T16:51:57.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:57.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:57.289546+0000) 2024-02-19T16:51:58.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:58.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:58.289826+0000) 2024-02-19T16:51:59.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:51:59.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:51:59.290142+0000) 2024-02-19T16:52:00.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:00.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:00.290394+0000) 2024-02-19T16:52:01.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:01.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:01.290595+0000) 2024-02-19T16:52:02.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:02.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:02.290804+0000) 2024-02-19T16:52:03.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:03.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:03.291040+0000) 2024-02-19T16:52:04.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:04.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:04.291260+0000) 2024-02-19T16:52:05.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:05.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:05.291455+0000) 2024-02-19T16:52:06.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:06.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:06.291663+0000) 2024-02-19T16:52:07.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:07.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:07.291960+0000) 2024-02-19T16:52:08.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:08.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:08.292240+0000) 2024-02-19T16:52:09.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:09.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:09.292559+0000) 2024-02-19T16:52:10.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:10.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:10.292819+0000) 2024-02-19T16:52:11.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:11.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:11.293118+0000) 2024-02-19T16:52:12.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:12.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:12.293399+0000) 2024-02-19T16:52:13.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:13.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:13.293712+0000) 2024-02-19T16:52:14.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:14.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:14.293934+0000) 2024-02-19T16:52:15.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:15.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:15.294242+0000) 2024-02-19T16:52:16.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:16.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:16.294540+0000) 2024-02-19T16:52:17.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:17.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:17.294849+0000) 2024-02-19T16:52:18.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:18.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:18.295004+0000) 2024-02-19T16:52:19.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:19.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:19.295241+0000) 2024-02-19T16:52:19.361 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:52:19.364 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:52:20.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:20.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:20.295442+0000) 2024-02-19T16:52:21.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:21.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:21.295674+0000) 2024-02-19T16:52:22.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:22.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:22.295930+0000) 2024-02-19T16:52:23.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:23.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:23.296193+0000) 2024-02-19T16:52:24.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:24.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:24.296399+0000) 2024-02-19T16:52:25.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:25.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:25.296682+0000) 2024-02-19T16:52:26.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:26.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:26.296983+0000) 2024-02-19T16:52:27.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:27.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:27.297278+0000) 2024-02-19T16:52:28.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:28.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:28.297477+0000) 2024-02-19T16:52:29.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:29.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:29.297740+0000) 2024-02-19T16:52:30.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:30.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:30.298033+0000) 2024-02-19T16:52:31.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:31.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:31.298322+0000) 2024-02-19T16:52:32.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:32.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:32.298521+0000) 2024-02-19T16:52:33.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:33.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:33.298843+0000) 2024-02-19T16:52:34.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:34.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:34.298996+0000) 2024-02-19T16:52:35.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:35.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:35.299319+0000) 2024-02-19T16:52:36.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:36.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:36.299605+0000) 2024-02-19T16:52:37.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:37.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:37.299788+0000) 2024-02-19T16:52:38.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:38.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:38.299969+0000) 2024-02-19T16:52:39.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:39.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:39.300162+0000) 2024-02-19T16:52:40.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:40.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:40.300409+0000) 2024-02-19T16:52:41.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:41.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:41.300654+0000) 2024-02-19T16:52:42.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:42.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:42.300926+0000) 2024-02-19T16:52:43.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:43.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:43.301195+0000) 2024-02-19T16:52:44.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:44.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:44.301437+0000) 2024-02-19T16:52:45.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:45.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:45.301651+0000) 2024-02-19T16:52:46.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:46.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:46.301898+0000) 2024-02-19T16:52:47.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:47.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:47.302145+0000) 2024-02-19T16:52:48.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:48.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:48.302365+0000) 2024-02-19T16:52:49.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:49.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:49.302546+0000) 2024-02-19T16:52:49.680 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:52:49.685 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:52:50.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:50.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:50.302811+0000) 2024-02-19T16:52:51.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:51.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:51.302945+0000) 2024-02-19T16:52:52.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:52.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:52.303146+0000) 2024-02-19T16:52:53.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:53.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:53.303338+0000) 2024-02-19T16:52:54.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:54.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:54.303592+0000) 2024-02-19T16:52:55.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:55.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:55.303857+0000) 2024-02-19T16:52:56.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:56.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:56.304093+0000) 2024-02-19T16:52:57.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:57.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:57.304272+0000) 2024-02-19T16:52:58.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:58.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:58.304523+0000) 2024-02-19T16:52:59.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:52:59.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:52:59.304777+0000) 2024-02-19T16:53:00.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:00.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:00.304973+0000) 2024-02-19T16:53:01.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:01.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:01.305212+0000) 2024-02-19T16:53:02.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:02.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:02.305457+0000) 2024-02-19T16:53:03.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:03.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:03.305708+0000) 2024-02-19T16:53:04.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:04.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:04.305913+0000) 2024-02-19T16:53:05.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:05.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:05.306130+0000) 2024-02-19T16:53:06.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:06.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:06.306375+0000) 2024-02-19T16:53:07.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:07.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:07.306645+0000) 2024-02-19T16:53:08.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:08.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:08.306901+0000) 2024-02-19T16:53:09.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:09.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:09.307151+0000) 2024-02-19T16:53:10.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:10.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:10.307434+0000) 2024-02-19T16:53:11.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:11.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:11.307674+0000) 2024-02-19T16:53:12.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:12.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:12.307957+0000) 2024-02-19T16:53:13.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:13.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:13.308171+0000) 2024-02-19T16:53:14.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:14.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:14.308404+0000) 2024-02-19T16:53:15.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:15.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:15.308602+0000) 2024-02-19T16:53:16.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:16.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:16.308837+0000) 2024-02-19T16:53:17.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:17.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:17.309068+0000) 2024-02-19T16:53:18.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:18.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:18.309354+0000) 2024-02-19T16:53:19.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:19.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:19.309640+0000) 2024-02-19T16:53:19.777 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:53:19.781 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:53:20.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:20.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:20.309895+0000) 2024-02-19T16:53:21.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:21.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:21.310178+0000) 2024-02-19T16:53:22.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:22.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:22.310378+0000) 2024-02-19T16:53:23.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:23.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:23.310578+0000) 2024-02-19T16:53:24.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:24.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:24.310920+0000) 2024-02-19T16:53:25.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:25.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:25.311228+0000) 2024-02-19T16:53:26.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:26.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:26.311528+0000) 2024-02-19T16:53:27.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:27.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:27.311714+0000) 2024-02-19T16:53:28.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:28.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:28.311985+0000) 2024-02-19T16:53:29.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:29.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:29.312244+0000) 2024-02-19T16:53:30.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:30.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:30.312443+0000) 2024-02-19T16:53:31.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:31.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:31.312655+0000) 2024-02-19T16:53:32.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:32.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:32.312866+0000) 2024-02-19T16:53:33.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:33.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:33.313093+0000) 2024-02-19T16:53:34.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:34.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:34.313293+0000) 2024-02-19T16:53:35.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:35.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:35.313556+0000) 2024-02-19T16:53:36.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:36.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:36.313762+0000) 2024-02-19T16:53:37.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:37.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:37.313999+0000) 2024-02-19T16:53:38.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:38.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:38.314157+0000) 2024-02-19T16:53:39.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:39.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:39.314319+0000) 2024-02-19T16:53:40.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:40.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:40.314581+0000) 2024-02-19T16:53:41.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:41.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:41.314881+0000) 2024-02-19T16:53:42.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:42.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:42.315109+0000) 2024-02-19T16:53:43.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:43.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:43.315260+0000) 2024-02-19T16:53:44.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:44.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:44.315485+0000) 2024-02-19T16:53:45.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:45.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:45.315730+0000) 2024-02-19T16:53:46.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:46.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:46.315987+0000) 2024-02-19T16:53:47.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:47.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:47.316269+0000) 2024-02-19T16:53:48.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:48.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:48.316454+0000) 2024-02-19T16:53:49.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:49.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:49.316706+0000) 2024-02-19T16:53:49.908 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:53:49.911 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:53:50.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:50.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:50.316976+0000) 2024-02-19T16:53:51.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:51.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:51.317170+0000) 2024-02-19T16:53:52.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:52.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:52.317399+0000) 2024-02-19T16:53:53.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:53.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:53.317647+0000) 2024-02-19T16:53:54.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:54.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:54.317854+0000) 2024-02-19T16:53:55.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:55.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:55.318096+0000) 2024-02-19T16:53:56.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:56.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:56.318383+0000) 2024-02-19T16:53:57.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:57.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:57.318631+0000) 2024-02-19T16:53:58.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:58.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:58.318897+0000) 2024-02-19T16:53:59.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:53:59.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:53:59.319143+0000) 2024-02-19T16:54:00.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:00.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:00.319287+0000) 2024-02-19T16:54:01.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:01.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:01.319488+0000) 2024-02-19T16:54:02.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:02.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:02.319742+0000) 2024-02-19T16:54:03.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:03.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:03.319942+0000) 2024-02-19T16:54:04.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:04.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:04.320185+0000) 2024-02-19T16:54:05.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:05.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:05.320385+0000) 2024-02-19T16:54:06.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:06.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:06.320627+0000) 2024-02-19T16:54:07.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:07.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:07.320860+0000) 2024-02-19T16:54:08.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:08.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:08.321127+0000) 2024-02-19T16:54:09.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:09.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:09.321321+0000) 2024-02-19T16:54:10.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:10.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:10.321517+0000) 2024-02-19T16:54:11.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:11.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:11.321733+0000) 2024-02-19T16:54:12.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:12.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:12.321954+0000) 2024-02-19T16:54:13.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:13.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:13.322208+0000) 2024-02-19T16:54:14.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:14.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:14.322458+0000) 2024-02-19T16:54:15.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:15.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:15.322749+0000) 2024-02-19T16:54:16.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:16.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:16.322942+0000) 2024-02-19T16:54:17.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:17.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:17.323105+0000) 2024-02-19T16:54:18.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:18.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:18.323346+0000) 2024-02-19T16:54:19.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:19.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:19.323530+0000) 2024-02-19T16:54:19.997 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:54:20.000 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:54:20.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:20.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:20.323712+0000) 2024-02-19T16:54:21.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:21.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:21.323922+0000) 2024-02-19T16:54:22.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:22.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:22.324152+0000) 2024-02-19T16:54:23.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:23.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:23.324420+0000) 2024-02-19T16:54:24.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:24.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:24.324607+0000) 2024-02-19T16:54:25.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:25.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:25.324834+0000) 2024-02-19T16:54:26.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:26.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:26.325085+0000) 2024-02-19T16:54:27.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:27.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:27.325321+0000) 2024-02-19T16:54:28.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:28.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:28.325556+0000) 2024-02-19T16:54:29.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:29.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:29.325793+0000) 2024-02-19T16:54:30.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:30.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:30.326033+0000) 2024-02-19T16:54:31.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:31.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:31.326278+0000) 2024-02-19T16:54:32.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:32.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:32.326526+0000) 2024-02-19T16:54:33.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:33.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:33.326838+0000) 2024-02-19T16:54:34.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:34.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:34.327083+0000) 2024-02-19T16:54:35.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:35.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:35.327324+0000) 2024-02-19T16:54:36.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:36.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:36.327571+0000) 2024-02-19T16:54:37.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:37.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:37.327812+0000) 2024-02-19T16:54:38.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:38.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:38.327990+0000) 2024-02-19T16:54:39.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:39.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:39.328181+0000) 2024-02-19T16:54:40.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:40.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:40.328345+0000) 2024-02-19T16:54:41.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:41.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:41.328577+0000) 2024-02-19T16:54:42.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:42.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:42.328835+0000) 2024-02-19T16:54:43.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:43.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:43.329064+0000) 2024-02-19T16:54:44.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:44.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:44.329302+0000) 2024-02-19T16:54:45.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:45.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:45.329545+0000) 2024-02-19T16:54:46.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:46.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:46.329803+0000) 2024-02-19T16:54:47.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:47.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:47.330022+0000) 2024-02-19T16:54:48.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:48.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:48.330353+0000) 2024-02-19T16:54:49.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:49.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:49.330633+0000) 2024-02-19T16:54:50.128 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:54:50.132 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:54:50.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:50.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:50.330907+0000) 2024-02-19T16:54:51.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:51.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:51.331183+0000) 2024-02-19T16:54:52.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:52.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:52.331428+0000) 2024-02-19T16:54:53.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:53.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:53.331628+0000) 2024-02-19T16:54:54.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:54.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:54.331855+0000) 2024-02-19T16:54:55.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:55.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:55.332077+0000) 2024-02-19T16:54:56.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:56.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:56.332231+0000) 2024-02-19T16:54:57.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:57.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:57.332495+0000) 2024-02-19T16:54:58.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:58.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:58.332779+0000) 2024-02-19T16:54:59.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:54:59.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:54:59.333067+0000) 2024-02-19T16:55:00.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:00.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:00.333318+0000) 2024-02-19T16:55:01.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:01.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:01.333512+0000) 2024-02-19T16:55:02.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:02.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:02.333737+0000) 2024-02-19T16:55:03.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:03.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:03.333993+0000) 2024-02-19T16:55:04.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:04.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:04.334251+0000) 2024-02-19T16:55:05.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:05.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:05.334540+0000) 2024-02-19T16:55:06.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:06.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:06.334822+0000) 2024-02-19T16:55:07.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:07.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:07.334982+0000) 2024-02-19T16:55:08.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:08.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:08.335214+0000) 2024-02-19T16:55:09.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:09.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:09.335453+0000) 2024-02-19T16:55:10.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:10.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:10.335649+0000) 2024-02-19T16:55:11.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:11.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:11.335838+0000) 2024-02-19T16:55:12.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:12.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:12.336086+0000) 2024-02-19T16:55:13.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:13.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:13.336331+0000) 2024-02-19T16:55:14.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:14.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:14.336578+0000) 2024-02-19T16:55:15.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:15.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:15.336798+0000) 2024-02-19T16:55:16.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:16.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:16.337014+0000) 2024-02-19T16:55:17.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:17.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:17.337245+0000) 2024-02-19T16:55:18.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:18.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:18.337415+0000) 2024-02-19T16:55:19.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:19.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:19.337641+0000) 2024-02-19T16:55:20.267 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:55:20.271 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:55:20.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:20.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:20.337907+0000) 2024-02-19T16:55:21.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:21.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:21.338127+0000) 2024-02-19T16:55:22.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:22.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:22.338328+0000) 2024-02-19T16:55:23.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:23.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:23.338556+0000) 2024-02-19T16:55:24.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:24.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:24.338823+0000) 2024-02-19T16:55:25.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:25.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:25.339103+0000) 2024-02-19T16:55:26.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:26.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:26.339350+0000) 2024-02-19T16:55:27.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:27.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:27.339592+0000) 2024-02-19T16:55:28.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:28.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:28.339830+0000) 2024-02-19T16:55:29.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:29.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:29.340084+0000) 2024-02-19T16:55:30.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:30.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:30.340336+0000) 2024-02-19T16:55:31.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:31.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:31.340547+0000) 2024-02-19T16:55:32.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:32.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:32.340795+0000) 2024-02-19T16:55:33.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:33.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:33.340983+0000) 2024-02-19T16:55:34.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:34.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:34.341175+0000) 2024-02-19T16:55:35.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:35.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:35.341348+0000) 2024-02-19T16:55:36.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:36.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:36.341548+0000) 2024-02-19T16:55:37.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:37.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:37.341753+0000) 2024-02-19T16:55:38.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:38.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:38.341947+0000) 2024-02-19T16:55:39.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:39.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:39.342196+0000) 2024-02-19T16:55:40.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:40.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:40.342450+0000) 2024-02-19T16:55:41.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:41.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:41.342798+0000) 2024-02-19T16:55:42.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:42.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:42.343002+0000) 2024-02-19T16:55:43.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:43.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:43.343274+0000) 2024-02-19T16:55:44.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:44.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:44.343535+0000) 2024-02-19T16:55:45.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:45.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:45.343766+0000) 2024-02-19T16:55:46.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:46.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:46.343972+0000) 2024-02-19T16:55:47.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:47.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:47.344156+0000) 2024-02-19T16:55:48.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:48.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:48.344402+0000) 2024-02-19T16:55:49.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:49.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:49.344685+0000) 2024-02-19T16:55:50.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:50.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:50.344944+0000) 2024-02-19T16:55:50.398 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:55:50.401 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:55:51.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:51.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:51.345179+0000) 2024-02-19T16:55:52.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:52.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:52.345320+0000) 2024-02-19T16:55:53.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:53.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:53.345584+0000) 2024-02-19T16:55:54.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:54.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:54.345869+0000) 2024-02-19T16:55:55.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:55.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:55.346129+0000) 2024-02-19T16:55:56.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:56.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:56.346368+0000) 2024-02-19T16:55:57.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:57.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:57.346618+0000) 2024-02-19T16:55:58.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:58.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:58.346919+0000) 2024-02-19T16:55:59.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:55:59.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:55:59.347085+0000) 2024-02-19T16:56:00.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:00.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:00.347318+0000) 2024-02-19T16:56:01.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:01.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:01.347480+0000) 2024-02-19T16:56:02.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:02.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:02.347715+0000) 2024-02-19T16:56:03.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:03.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:03.347970+0000) 2024-02-19T16:56:04.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:04.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:04.348219+0000) 2024-02-19T16:56:05.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:05.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:05.348483+0000) 2024-02-19T16:56:06.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:06.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:06.348745+0000) 2024-02-19T16:56:07.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:07.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:07.348993+0000) 2024-02-19T16:56:08.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:08.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:08.349259+0000) 2024-02-19T16:56:09.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:09.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:09.349535+0000) 2024-02-19T16:56:10.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:10.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:10.349738+0000) 2024-02-19T16:56:11.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:11.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:11.349936+0000) 2024-02-19T16:56:12.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:12.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:12.350168+0000) 2024-02-19T16:56:13.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:13.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:13.350411+0000) 2024-02-19T16:56:14.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:14.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:14.350577+0000) 2024-02-19T16:56:15.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:15.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:15.350881+0000) 2024-02-19T16:56:16.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:16.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:16.351136+0000) 2024-02-19T16:56:17.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:17.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:17.351308+0000) 2024-02-19T16:56:18.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:18.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:18.351582+0000) 2024-02-19T16:56:19.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:19.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:19.351836+0000) 2024-02-19T16:56:20.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:20.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:20.352089+0000) 2024-02-19T16:56:20.464 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:56:20.467 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:56:21.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:21.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:21.352264+0000) 2024-02-19T16:56:22.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:22.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:22.352465+0000) 2024-02-19T16:56:23.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:23.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:23.352671+0000) 2024-02-19T16:56:24.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:24.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:24.352947+0000) 2024-02-19T16:56:25.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:25.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:25.353194+0000) 2024-02-19T16:56:26.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:26.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:26.353450+0000) 2024-02-19T16:56:27.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:27.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:27.353699+0000) 2024-02-19T16:56:28.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:28.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:28.353954+0000) 2024-02-19T16:56:29.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:29.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:29.354269+0000) 2024-02-19T16:56:30.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:30.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:30.354469+0000) 2024-02-19T16:56:31.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:31.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:31.354601+0000) 2024-02-19T16:56:32.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:32.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:32.354818+0000) 2024-02-19T16:56:33.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:33.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:33.355048+0000) 2024-02-19T16:56:34.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:34.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:34.355194+0000) 2024-02-19T16:56:35.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:35.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:35.355460+0000) 2024-02-19T16:56:36.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:36.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:36.355713+0000) 2024-02-19T16:56:37.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:37.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:37.355927+0000) 2024-02-19T16:56:38.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:38.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:38.356167+0000) 2024-02-19T16:56:39.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:39.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:39.356440+0000) 2024-02-19T16:56:40.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:40.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:40.356678+0000) 2024-02-19T16:56:41.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:41.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:41.356948+0000) 2024-02-19T16:56:42.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:42.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:42.357148+0000) 2024-02-19T16:56:43.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:43.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:43.357426+0000) 2024-02-19T16:56:44.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:44.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:44.357672+0000) 2024-02-19T16:56:45.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:45.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:45.357908+0000) 2024-02-19T16:56:46.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:46.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:46.358078+0000) 2024-02-19T16:56:47.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:47.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:47.358261+0000) 2024-02-19T16:56:48.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:48.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:48.358457+0000) 2024-02-19T16:56:49.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:49.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:49.358743+0000) 2024-02-19T16:56:50.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:50.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:50.358987+0000) 2024-02-19T16:56:50.584 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:56:50.588 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:56:51.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:51.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:51.359182+0000) 2024-02-19T16:56:52.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:52.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:52.359357+0000) 2024-02-19T16:56:53.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:53.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:53.359600+0000) 2024-02-19T16:56:54.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:54.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:54.359865+0000) 2024-02-19T16:56:55.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:55.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:55.360129+0000) 2024-02-19T16:56:56.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:56.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:56.360353+0000) 2024-02-19T16:56:57.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:57.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:57.360582+0000) 2024-02-19T16:56:58.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:58.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:58.360838+0000) 2024-02-19T16:56:59.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:56:59.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:56:59.361141+0000) 2024-02-19T16:57:00.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:00.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:00.361420+0000) 2024-02-19T16:57:01.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:01.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:01.361616+0000) 2024-02-19T16:57:02.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:02.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:02.361786+0000) 2024-02-19T16:57:03.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:03.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:03.362041+0000) 2024-02-19T16:57:04.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:04.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:04.362267+0000) 2024-02-19T16:57:05.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:05.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:05.362548+0000) 2024-02-19T16:57:06.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:06.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:06.362781+0000) 2024-02-19T16:57:07.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:07.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:07.363022+0000) 2024-02-19T16:57:08.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:08.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:08.363223+0000) 2024-02-19T16:57:09.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:09.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:09.363495+0000) 2024-02-19T16:57:10.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:10.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:10.363768+0000) 2024-02-19T16:57:11.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:11.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:11.364008+0000) 2024-02-19T16:57:12.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:12.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:12.364248+0000) 2024-02-19T16:57:13.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:13.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:13.364465+0000) 2024-02-19T16:57:14.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:14.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:14.364724+0000) 2024-02-19T16:57:15.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:15.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:15.364961+0000) 2024-02-19T16:57:16.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:16.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:16.365146+0000) 2024-02-19T16:57:17.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:17.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:17.365417+0000) 2024-02-19T16:57:18.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:18.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:18.365656+0000) 2024-02-19T16:57:19.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:19.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:19.365869+0000) 2024-02-19T16:57:20.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:20.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:20.366071+0000) 2024-02-19T16:57:20.715 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:57:20.718 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:57:21.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:21.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:21.366242+0000) 2024-02-19T16:57:22.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:22.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:22.366512+0000) 2024-02-19T16:57:23.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:23.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:23.366830+0000) 2024-02-19T16:57:24.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:24.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:24.367016+0000) 2024-02-19T16:57:25.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:25.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:25.367196+0000) 2024-02-19T16:57:26.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:26.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:26.367427+0000) 2024-02-19T16:57:27.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:27.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:27.367698+0000) 2024-02-19T16:57:28.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:28.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:28.367933+0000) 2024-02-19T16:57:29.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:29.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:29.368191+0000) 2024-02-19T16:57:30.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:30.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:30.368417+0000) 2024-02-19T16:57:31.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:31.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:31.368672+0000) 2024-02-19T16:57:32.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:32.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:32.368922+0000) 2024-02-19T16:57:33.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:33.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:33.369181+0000) 2024-02-19T16:57:34.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:34.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:34.369435+0000) 2024-02-19T16:57:35.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:35.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:35.369696+0000) 2024-02-19T16:57:36.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:36.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:36.369864+0000) 2024-02-19T16:57:37.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:37.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:37.370093+0000) 2024-02-19T16:57:38.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:38.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:38.370283+0000) 2024-02-19T16:57:39.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:39.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:39.370439+0000) 2024-02-19T16:57:40.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:40.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:40.370585+0000) 2024-02-19T16:57:41.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:41.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:41.370730+0000) 2024-02-19T16:57:42.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:42.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:42.370977+0000) 2024-02-19T16:57:43.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:43.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:43.371235+0000) 2024-02-19T16:57:44.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:44.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:44.371481+0000) 2024-02-19T16:57:45.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:45.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:45.371703+0000) 2024-02-19T16:57:46.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:46.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:46.371942+0000) 2024-02-19T16:57:47.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:47.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:47.372199+0000) 2024-02-19T16:57:48.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:48.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:48.372389+0000) 2024-02-19T16:57:49.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:49.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:49.372604+0000) 2024-02-19T16:57:50.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:50.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:50.372824+0000) 2024-02-19T16:57:50.787 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:57:50.791 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:57:51.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:51.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:51.373074+0000) 2024-02-19T16:57:52.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:52.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:52.373347+0000) 2024-02-19T16:57:53.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:53.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:53.373591+0000) 2024-02-19T16:57:54.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:54.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:54.373831+0000) 2024-02-19T16:57:55.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:55.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:55.374068+0000) 2024-02-19T16:57:56.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:56.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:56.374276+0000) 2024-02-19T16:57:57.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:57.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:57.374468+0000) 2024-02-19T16:57:58.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:58.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:58.374735+0000) 2024-02-19T16:57:59.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:57:59.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:57:59.374913+0000) 2024-02-19T16:58:00.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:00.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:00.375152+0000) 2024-02-19T16:58:01.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:01.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:01.375345+0000) 2024-02-19T16:58:02.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:02.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:02.375594+0000) 2024-02-19T16:58:03.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:03.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:03.375784+0000) 2024-02-19T16:58:04.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:04.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:04.375993+0000) 2024-02-19T16:58:05.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:05.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:05.376238+0000) 2024-02-19T16:58:06.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:06.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:06.376419+0000) 2024-02-19T16:58:07.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:07.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:07.376655+0000) 2024-02-19T16:58:08.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:08.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:08.376885+0000) 2024-02-19T16:58:09.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:09.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:09.377039+0000) 2024-02-19T16:58:10.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:10.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:10.377282+0000) 2024-02-19T16:58:11.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:11.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:11.377453+0000) 2024-02-19T16:58:12.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:12.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:12.377688+0000) 2024-02-19T16:58:13.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:13.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:13.377939+0000) 2024-02-19T16:58:14.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:14.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:14.378137+0000) 2024-02-19T16:58:15.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:15.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:15.378400+0000) 2024-02-19T16:58:16.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:16.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:16.378643+0000) 2024-02-19T16:58:17.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:17.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:17.378840+0000) 2024-02-19T16:58:18.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:18.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:18.379063+0000) 2024-02-19T16:58:19.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:19.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:19.379305+0000) 2024-02-19T16:58:20.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:20.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:20.379564+0000) 2024-02-19T16:58:20.847 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:58:20.851 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:58:21.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:21.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:21.379776+0000) 2024-02-19T16:58:22.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:22.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:22.380007+0000) 2024-02-19T16:58:23.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:23.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:23.380251+0000) 2024-02-19T16:58:24.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:24.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:24.380495+0000) 2024-02-19T16:58:25.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:25.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:25.380776+0000) 2024-02-19T16:58:26.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:26.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:26.381023+0000) 2024-02-19T16:58:27.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:27.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:27.381227+0000) 2024-02-19T16:58:28.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:28.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:28.381444+0000) 2024-02-19T16:58:29.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:29.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:29.381687+0000) 2024-02-19T16:58:30.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:30.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:30.381926+0000) 2024-02-19T16:58:31.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:31.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:31.382198+0000) 2024-02-19T16:58:32.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:32.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:32.382368+0000) 2024-02-19T16:58:33.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:33.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:33.382618+0000) 2024-02-19T16:58:34.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:34.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:34.382930+0000) 2024-02-19T16:58:35.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:35.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:35.383178+0000) 2024-02-19T16:58:36.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:36.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:36.383436+0000) 2024-02-19T16:58:37.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:37.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:37.383699+0000) 2024-02-19T16:58:38.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:38.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:38.383912+0000) 2024-02-19T16:58:39.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:39.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:39.384190+0000) 2024-02-19T16:58:40.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:40.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:40.384343+0000) 2024-02-19T16:58:41.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:41.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:41.384586+0000) 2024-02-19T16:58:42.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:42.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:42.384842+0000) 2024-02-19T16:58:43.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:43.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:43.385077+0000) 2024-02-19T16:58:44.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:44.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:44.385257+0000) 2024-02-19T16:58:45.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:45.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:45.385499+0000) 2024-02-19T16:58:46.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:46.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:46.385676+0000) 2024-02-19T16:58:47.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:47.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:47.385909+0000) 2024-02-19T16:58:48.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:48.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:48.386088+0000) 2024-02-19T16:58:49.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:49.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:49.386350+0000) 2024-02-19T16:58:50.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:50.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:50.386524+0000) 2024-02-19T16:58:50.902 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:58:50.905 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:58:51.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:51.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:51.386807+0000) 2024-02-19T16:58:52.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:52.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:52.386981+0000) 2024-02-19T16:58:53.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:53.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:53.387205+0000) 2024-02-19T16:58:54.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:54.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:54.387475+0000) 2024-02-19T16:58:55.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:55.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:55.387723+0000) 2024-02-19T16:58:56.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:56.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:56.387974+0000) 2024-02-19T16:58:57.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:57.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:57.388251+0000) 2024-02-19T16:58:58.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:58.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:58.388520+0000) 2024-02-19T16:58:59.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:58:59.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:58:59.388724+0000) 2024-02-19T16:59:00.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:00.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:00.388994+0000) 2024-02-19T16:59:01.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:01.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:01.389244+0000) 2024-02-19T16:59:02.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:02.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:02.389512+0000) 2024-02-19T16:59:03.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:03.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:03.389713+0000) 2024-02-19T16:59:04.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:04.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:04.389981+0000) 2024-02-19T16:59:05.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:05.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:05.390235+0000) 2024-02-19T16:59:06.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:06.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:06.390515+0000) 2024-02-19T16:59:07.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:07.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:07.390795+0000) 2024-02-19T16:59:08.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:08.388+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:08.391069+0000) 2024-02-19T16:59:09.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:09.388+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:09.391309+0000) 2024-02-19T16:59:10.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:10.388+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:10.391510+0000) 2024-02-19T16:59:11.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:11.388+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:11.391699+0000) 2024-02-19T16:59:12.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:12.388+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:12.391969+0000) 2024-02-19T16:59:13.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:13.388+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:13.392204+0000) 2024-02-19T16:59:14.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:14.388+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:14.392440+0000) 2024-02-19T16:59:15.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:15.388+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:15.392692+0000) 2024-02-19T16:59:16.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:16.388+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:16.392926+0000) 2024-02-19T16:59:17.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:17.388+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:17.393179+0000) 2024-02-19T16:59:18.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:18.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:18.393455+0000) 2024-02-19T16:59:19.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:19.388+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:19.393699+0000) 2024-02-19T16:59:20.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:20.388+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:20.393970+0000) 2024-02-19T16:59:21.017 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:59:21.020 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:59:21.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:21.388+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:21.394210+0000) 2024-02-19T16:59:22.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:22.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:22.394448+0000) 2024-02-19T16:59:23.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:23.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:23.394739+0000) 2024-02-19T16:59:24.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:24.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:24.394911+0000) 2024-02-19T16:59:25.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:25.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:25.395112+0000) 2024-02-19T16:59:26.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:26.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:26.395269+0000) 2024-02-19T16:59:27.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:27.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:27.395416+0000) 2024-02-19T16:59:28.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:28.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:28.395642+0000) 2024-02-19T16:59:29.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:29.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:29.395835+0000) 2024-02-19T16:59:30.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:30.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:30.396051+0000) 2024-02-19T16:59:31.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:31.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:31.396343+0000) 2024-02-19T16:59:32.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:32.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:32.396612+0000) 2024-02-19T16:59:33.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:33.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:33.396849+0000) 2024-02-19T16:59:34.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:34.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:34.397051+0000) 2024-02-19T16:59:35.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:35.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:35.397304+0000) 2024-02-19T16:59:36.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:36.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:36.397556+0000) 2024-02-19T16:59:37.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:37.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:37.397767+0000) 2024-02-19T16:59:38.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:38.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:38.398055+0000) 2024-02-19T16:59:39.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:39.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:39.398297+0000) 2024-02-19T16:59:40.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:40.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:40.398561+0000) 2024-02-19T16:59:41.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:41.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:41.398791+0000) 2024-02-19T16:59:42.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:42.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:42.398973+0000) 2024-02-19T16:59:43.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:43.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:43.399216+0000) 2024-02-19T16:59:44.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:44.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:44.399405+0000) 2024-02-19T16:59:45.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:45.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:45.399654+0000) 2024-02-19T16:59:46.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:46.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:46.399913+0000) 2024-02-19T16:59:47.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:47.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:47.400111+0000) 2024-02-19T16:59:48.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:48.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:48.400334+0000) 2024-02-19T16:59:49.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:49.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:49.400553+0000) 2024-02-19T16:59:50.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:50.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:50.400791+0000) 2024-02-19T16:59:51.129 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:59:51.133 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T16:59:51.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:51.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:51.401030+0000) 2024-02-19T16:59:52.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:52.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:52.401209+0000) 2024-02-19T16:59:53.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:53.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:53.401486+0000) 2024-02-19T16:59:54.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:54.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:54.401729+0000) 2024-02-19T16:59:55.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:55.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:55.401991+0000) 2024-02-19T16:59:56.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:56.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:56.402242+0000) 2024-02-19T16:59:57.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:57.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:57.402547+0000) 2024-02-19T16:59:58.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:58.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:58.402753+0000) 2024-02-19T16:59:59.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T16:59:59.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T15:59:59.402980+0000) 2024-02-19T17:00:00.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:00.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:00.403132+0000) 2024-02-19T17:00:01.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:01.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:01.403305+0000) 2024-02-19T17:00:02.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:02.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:02.403550+0000) 2024-02-19T17:00:03.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:03.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:03.403791+0000) 2024-02-19T17:00:04.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:04.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:04.404035+0000) 2024-02-19T17:00:05.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:05.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:05.404295+0000) 2024-02-19T17:00:06.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:06.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:06.404547+0000) 2024-02-19T17:00:07.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:07.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:07.404684+0000) 2024-02-19T17:00:08.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:08.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:08.404896+0000) 2024-02-19T17:00:09.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:09.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:09.405083+0000) 2024-02-19T17:00:10.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:10.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:10.405332+0000) 2024-02-19T17:00:11.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:11.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:11.405503+0000) 2024-02-19T17:00:12.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:12.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:12.405686+0000) 2024-02-19T17:00:13.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:13.399+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:13.405813+0000) 2024-02-19T17:00:14.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:14.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:14.406017+0000) 2024-02-19T17:00:15.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:15.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:15.406220+0000) 2024-02-19T17:00:16.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:16.399+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:16.406398+0000) 2024-02-19T17:00:17.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:17.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:17.406594+0000) 2024-02-19T17:00:18.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:18.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:18.406829+0000) 2024-02-19T17:00:19.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:19.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:19.407074+0000) 2024-02-19T17:00:20.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:20.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:20.407265+0000) 2024-02-19T17:00:21.251 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:00:21.255 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:00:21.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:21.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:21.407483+0000) 2024-02-19T17:00:22.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:22.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:22.407737+0000) 2024-02-19T17:00:23.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:23.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:23.407962+0000) 2024-02-19T17:00:24.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:24.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:24.408203+0000) 2024-02-19T17:00:25.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:25.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:25.408419+0000) 2024-02-19T17:00:26.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:26.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:26.408609+0000) 2024-02-19T17:00:27.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:27.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:27.408775+0000) 2024-02-19T17:00:28.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:28.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:28.409004+0000) 2024-02-19T17:00:29.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:29.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:29.409260+0000) 2024-02-19T17:00:30.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:30.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:30.409519+0000) 2024-02-19T17:00:31.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:31.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:31.409710+0000) 2024-02-19T17:00:32.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:32.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:32.409904+0000) 2024-02-19T17:00:33.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:33.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:33.410136+0000) 2024-02-19T17:00:34.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:34.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:34.410378+0000) 2024-02-19T17:00:35.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:35.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:35.410579+0000) 2024-02-19T17:00:36.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:36.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:36.410862+0000) 2024-02-19T17:00:37.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:37.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:37.411126+0000) 2024-02-19T17:00:38.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:38.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:38.411371+0000) 2024-02-19T17:00:39.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:39.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:39.411610+0000) 2024-02-19T17:00:40.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:40.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:40.411881+0000) 2024-02-19T17:00:41.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:41.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:41.412133+0000) 2024-02-19T17:00:42.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:42.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:42.412331+0000) 2024-02-19T17:00:43.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:43.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:43.412566+0000) 2024-02-19T17:00:44.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:44.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:44.412817+0000) 2024-02-19T17:00:45.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:45.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:45.413063+0000) 2024-02-19T17:00:46.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:46.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:46.413298+0000) 2024-02-19T17:00:47.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:47.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:47.413529+0000) 2024-02-19T17:00:48.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:48.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:48.413716+0000) 2024-02-19T17:00:49.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:49.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:49.413972+0000) 2024-02-19T17:00:50.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:50.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:50.414213+0000) 2024-02-19T17:00:51.322 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:00:51.326 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:00:51.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:51.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:51.414499+0000) 2024-02-19T17:00:52.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:52.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:52.414703+0000) 2024-02-19T17:00:53.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:53.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:53.414925+0000) 2024-02-19T17:00:54.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:54.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:54.415100+0000) 2024-02-19T17:00:55.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:55.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:55.415374+0000) 2024-02-19T17:00:56.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:56.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:56.415599+0000) 2024-02-19T17:00:57.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:57.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:57.415855+0000) 2024-02-19T17:00:58.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:58.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:58.416060+0000) 2024-02-19T17:00:59.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:00:59.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:00:59.416345+0000) 2024-02-19T17:01:00.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:00.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:00.416551+0000) 2024-02-19T17:01:01.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:01.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:01.416805+0000) 2024-02-19T17:01:02.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:02.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:02.417057+0000) 2024-02-19T17:01:03.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:03.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:03.417331+0000) 2024-02-19T17:01:04.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:04.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:04.417506+0000) 2024-02-19T17:01:05.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:05.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:05.417750+0000) 2024-02-19T17:01:06.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:06.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:06.418024+0000) 2024-02-19T17:01:07.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:07.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:07.418265+0000) 2024-02-19T17:01:08.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:08.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:08.418470+0000) 2024-02-19T17:01:09.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:09.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:09.418739+0000) 2024-02-19T17:01:10.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:10.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:10.418963+0000) 2024-02-19T17:01:11.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:11.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:11.419230+0000) 2024-02-19T17:01:12.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:12.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:12.419431+0000) 2024-02-19T17:01:13.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:13.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:13.419610+0000) 2024-02-19T17:01:14.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:14.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:14.419855+0000) 2024-02-19T17:01:15.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:15.418+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:15.420106+0000) 2024-02-19T17:01:16.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:16.418+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:16.420285+0000) 2024-02-19T17:01:17.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:17.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:17.420539+0000) 2024-02-19T17:01:18.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:18.418+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:18.420773+0000) 2024-02-19T17:01:19.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:19.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:19.421015+0000) 2024-02-19T17:01:20.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:20.418+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:20.421190+0000) 2024-02-19T17:01:21.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:21.418+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:21.421472+0000) 2024-02-19T17:01:21.452 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:01:21.455 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:01:22.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:22.418+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:22.421720+0000) 2024-02-19T17:01:23.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:23.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:23.421962+0000) 2024-02-19T17:01:24.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:24.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:24.422144+0000) 2024-02-19T17:01:25.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:25.418+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:25.422326+0000) 2024-02-19T17:01:26.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:26.418+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:26.422505+0000) 2024-02-19T17:01:27.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:27.418+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:27.422821+0000) 2024-02-19T17:01:28.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:28.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:28.423028+0000) 2024-02-19T17:01:29.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:29.418+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:29.423188+0000) 2024-02-19T17:01:30.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:30.418+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:30.423445+0000) 2024-02-19T17:01:31.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:31.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:31.423694+0000) 2024-02-19T17:01:32.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:32.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:32.423945+0000) 2024-02-19T17:01:33.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:33.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:33.424192+0000) 2024-02-19T17:01:34.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:34.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:34.424431+0000) 2024-02-19T17:01:35.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:35.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:35.424723+0000) 2024-02-19T17:01:36.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:36.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:36.425015+0000) 2024-02-19T17:01:37.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:37.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:37.425234+0000) 2024-02-19T17:01:38.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:38.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:38.425408+0000) 2024-02-19T17:01:39.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:39.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:39.425617+0000) 2024-02-19T17:01:40.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:40.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:40.425867+0000) 2024-02-19T17:01:41.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:41.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:41.426127+0000) 2024-02-19T17:01:42.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:42.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:42.426313+0000) 2024-02-19T17:01:43.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:43.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:43.426610+0000) 2024-02-19T17:01:44.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:44.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:44.426805+0000) 2024-02-19T17:01:45.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:45.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:45.427115+0000) 2024-02-19T17:01:46.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:46.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:46.427381+0000) 2024-02-19T17:01:47.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:47.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:47.427679+0000) 2024-02-19T17:01:48.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:48.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:48.427961+0000) 2024-02-19T17:01:49.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:49.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:49.428240+0000) 2024-02-19T17:01:50.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:50.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:50.428548+0000) 2024-02-19T17:01:51.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:51.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:51.428825+0000) 2024-02-19T17:01:51.524 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:01:51.527 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:01:52.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:52.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:52.429022+0000) 2024-02-19T17:01:53.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:53.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:53.429282+0000) 2024-02-19T17:01:54.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:54.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:54.429539+0000) 2024-02-19T17:01:55.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:55.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:55.429787+0000) 2024-02-19T17:01:56.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:56.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:56.430083+0000) 2024-02-19T17:01:57.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:57.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:57.430367+0000) 2024-02-19T17:01:58.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:58.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:58.430566+0000) 2024-02-19T17:01:59.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:01:59.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:01:59.430754+0000) 2024-02-19T17:02:00.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:00.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:00.430978+0000) 2024-02-19T17:02:01.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:01.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:01.431187+0000) 2024-02-19T17:02:02.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:02.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:02.431428+0000) 2024-02-19T17:02:03.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:03.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:03.431609+0000) 2024-02-19T17:02:04.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:04.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:04.431864+0000) 2024-02-19T17:02:05.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:05.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:05.432115+0000) 2024-02-19T17:02:06.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:06.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:06.432395+0000) 2024-02-19T17:02:07.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:07.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:07.432584+0000) 2024-02-19T17:02:08.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:08.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:08.432762+0000) 2024-02-19T17:02:09.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:09.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:09.433037+0000) 2024-02-19T17:02:10.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:10.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:10.433259+0000) 2024-02-19T17:02:11.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:11.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:11.433438+0000) 2024-02-19T17:02:12.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:12.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:12.433615+0000) 2024-02-19T17:02:13.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:13.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:13.433868+0000) 2024-02-19T17:02:14.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:14.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:14.434093+0000) 2024-02-19T17:02:15.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:15.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:15.434397+0000) 2024-02-19T17:02:16.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:16.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:16.434691+0000) 2024-02-19T17:02:17.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:17.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:17.434905+0000) 2024-02-19T17:02:18.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:18.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:18.435097+0000) 2024-02-19T17:02:19.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:19.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:19.435334+0000) 2024-02-19T17:02:20.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:20.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:20.435604+0000) 2024-02-19T17:02:21.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:21.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:21.435860+0000) 2024-02-19T17:02:21.653 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:02:21.657 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:02:22.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:22.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:22.436067+0000) 2024-02-19T17:02:23.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:23.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:23.436326+0000) 2024-02-19T17:02:24.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:24.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:24.436557+0000) 2024-02-19T17:02:25.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:25.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:25.436801+0000) 2024-02-19T17:02:26.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:26.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:26.437029+0000) 2024-02-19T17:02:27.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:27.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:27.437318+0000) 2024-02-19T17:02:28.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:28.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:28.437516+0000) 2024-02-19T17:02:29.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:29.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:29.437770+0000) 2024-02-19T17:02:30.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:30.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:30.438052+0000) 2024-02-19T17:02:31.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:31.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:31.438259+0000) 2024-02-19T17:02:32.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:32.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:32.438516+0000) 2024-02-19T17:02:33.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:33.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:33.438769+0000) 2024-02-19T17:02:34.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:34.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:34.439033+0000) 2024-02-19T17:02:35.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:35.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:35.439267+0000) 2024-02-19T17:02:36.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:36.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:36.439395+0000) 2024-02-19T17:02:37.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:37.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:37.439532+0000) 2024-02-19T17:02:38.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:38.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:38.439809+0000) 2024-02-19T17:02:39.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:39.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:39.440099+0000) 2024-02-19T17:02:40.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:40.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:40.440290+0000) 2024-02-19T17:02:41.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:41.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:41.440541+0000) 2024-02-19T17:02:42.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:42.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:42.440772+0000) 2024-02-19T17:02:43.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:43.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:43.441010+0000) 2024-02-19T17:02:44.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:44.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:44.441188+0000) 2024-02-19T17:02:45.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:45.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:45.441393+0000) 2024-02-19T17:02:46.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:46.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:46.441574+0000) 2024-02-19T17:02:47.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:47.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:47.441766+0000) 2024-02-19T17:02:48.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:48.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:48.441997+0000) 2024-02-19T17:02:49.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:49.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:49.442189+0000) 2024-02-19T17:02:50.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:50.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:50.442428+0000) 2024-02-19T17:02:51.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:51.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:51.442666+0000) 2024-02-19T17:02:51.750 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:02:51.755 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:02:52.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:52.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:52.442955+0000) 2024-02-19T17:02:53.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:53.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:53.443198+0000) 2024-02-19T17:02:54.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:54.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:54.443413+0000) 2024-02-19T17:02:55.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:55.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:55.443701+0000) 2024-02-19T17:02:56.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:56.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:56.443954+0000) 2024-02-19T17:02:57.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:57.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:57.444205+0000) 2024-02-19T17:02:58.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:58.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:58.444445+0000) 2024-02-19T17:02:59.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:02:59.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:02:59.444599+0000) 2024-02-19T17:03:00.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:00.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:00.444848+0000) 2024-02-19T17:03:01.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:01.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:01.445081+0000) 2024-02-19T17:03:02.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:02.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:02.445314+0000) 2024-02-19T17:03:03.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:03.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:03.445544+0000) 2024-02-19T17:03:04.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:04.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:04.445801+0000) 2024-02-19T17:03:05.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:05.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:05.446054+0000) 2024-02-19T17:03:06.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:06.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:06.446302+0000) 2024-02-19T17:03:07.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:07.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:07.446554+0000) 2024-02-19T17:03:08.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:08.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:08.446831+0000) 2024-02-19T17:03:09.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:09.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:09.447026+0000) 2024-02-19T17:03:10.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:10.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:10.447218+0000) 2024-02-19T17:03:11.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:11.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:11.447472+0000) 2024-02-19T17:03:12.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:12.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:12.447692+0000) 2024-02-19T17:03:13.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:13.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:13.447938+0000) 2024-02-19T17:03:14.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:14.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:14.448121+0000) 2024-02-19T17:03:15.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:15.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:15.448332+0000) 2024-02-19T17:03:16.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:16.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:16.448508+0000) 2024-02-19T17:03:17.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:17.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:17.448686+0000) 2024-02-19T17:03:18.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:18.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:18.448943+0000) 2024-02-19T17:03:19.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:19.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:19.449214+0000) 2024-02-19T17:03:20.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:20.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:20.449530+0000) 2024-02-19T17:03:21.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:21.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:21.449788+0000) 2024-02-19T17:03:21.872 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:03:21.875 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:03:22.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:22.444+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:22.450019+0000) 2024-02-19T17:03:23.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:23.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:23.450280+0000) 2024-02-19T17:03:24.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:24.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:24.450518+0000) 2024-02-19T17:03:25.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:25.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:25.450762+0000) 2024-02-19T17:03:26.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:26.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:26.450952+0000) 2024-02-19T17:03:27.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:27.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:27.451195+0000) 2024-02-19T17:03:28.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:28.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:28.451374+0000) 2024-02-19T17:03:29.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:29.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:29.451567+0000) 2024-02-19T17:03:30.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:30.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:30.451847+0000) 2024-02-19T17:03:31.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:31.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:31.452086+0000) 2024-02-19T17:03:32.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:32.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:32.452258+0000) 2024-02-19T17:03:33.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:33.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:33.452437+0000) 2024-02-19T17:03:34.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:34.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:34.452680+0000) 2024-02-19T17:03:35.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:35.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:35.452941+0000) 2024-02-19T17:03:36.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:36.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:36.453215+0000) 2024-02-19T17:03:37.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:37.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:37.453447+0000) 2024-02-19T17:03:38.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:38.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:38.453649+0000) 2024-02-19T17:03:39.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:39.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:39.453905+0000) 2024-02-19T17:03:40.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:40.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:40.454143+0000) 2024-02-19T17:03:41.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:41.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:41.454331+0000) 2024-02-19T17:03:42.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:42.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:42.454532+0000) 2024-02-19T17:03:43.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:43.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:43.454802+0000) 2024-02-19T17:03:44.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:44.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:44.455023+0000) 2024-02-19T17:03:45.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:45.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:45.455220+0000) 2024-02-19T17:03:46.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:46.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:46.455391+0000) 2024-02-19T17:03:47.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:47.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:47.455663+0000) 2024-02-19T17:03:48.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:48.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:48.455888+0000) 2024-02-19T17:03:49.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:49.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:49.456147+0000) 2024-02-19T17:03:50.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:50.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:50.456368+0000) 2024-02-19T17:03:51.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:51.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:51.456570+0000) 2024-02-19T17:03:52.003 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:03:52.006 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:03:52.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:52.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:52.456860+0000) 2024-02-19T17:03:53.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:53.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:53.457142+0000) 2024-02-19T17:03:54.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:54.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:54.457418+0000) 2024-02-19T17:03:55.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:55.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:55.457688+0000) 2024-02-19T17:03:56.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:56.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:56.457883+0000) 2024-02-19T17:03:57.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:57.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:57.458137+0000) 2024-02-19T17:03:58.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:58.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:58.458458+0000) 2024-02-19T17:03:59.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:03:59.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:03:59.458809+0000) 2024-02-19T17:04:00.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:00.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:00.458971+0000) 2024-02-19T17:04:01.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:01.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:01.459188+0000) 2024-02-19T17:04:02.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:02.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:02.459414+0000) 2024-02-19T17:04:03.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:03.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:03.459722+0000) 2024-02-19T17:04:04.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:04.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:04.459999+0000) 2024-02-19T17:04:05.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:05.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:05.460295+0000) 2024-02-19T17:04:06.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:06.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:06.460511+0000) 2024-02-19T17:04:07.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:07.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:07.460748+0000) 2024-02-19T17:04:08.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:08.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:08.461009+0000) 2024-02-19T17:04:09.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:09.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:09.461298+0000) 2024-02-19T17:04:10.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:10.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:10.461577+0000) 2024-02-19T17:04:11.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:11.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:11.461776+0000) 2024-02-19T17:04:12.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:12.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:12.462037+0000) 2024-02-19T17:04:13.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:13.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:13.462338+0000) 2024-02-19T17:04:14.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:14.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:14.462636+0000) 2024-02-19T17:04:15.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:15.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:15.462937+0000) 2024-02-19T17:04:16.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:16.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:16.463195+0000) 2024-02-19T17:04:17.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:17.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:17.463503+0000) 2024-02-19T17:04:18.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:18.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:18.463785+0000) 2024-02-19T17:04:19.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:19.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:19.464170+0000) 2024-02-19T17:04:20.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:20.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:20.464447+0000) 2024-02-19T17:04:21.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:21.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:21.464705+0000) 2024-02-19T17:04:22.090 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:04:22.094 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:04:22.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:22.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:22.464943+0000) 2024-02-19T17:04:23.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:23.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:23.465181+0000) 2024-02-19T17:04:24.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:24.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:24.465384+0000) 2024-02-19T17:04:25.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:25.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:25.465667+0000) 2024-02-19T17:04:26.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:26.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:26.465889+0000) 2024-02-19T17:04:27.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:27.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:27.466066+0000) 2024-02-19T17:04:28.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:28.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:28.466283+0000) 2024-02-19T17:04:29.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:29.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:29.466474+0000) 2024-02-19T17:04:30.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:30.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:30.466723+0000) 2024-02-19T17:04:31.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:31.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:31.466976+0000) 2024-02-19T17:04:32.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:32.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:32.467295+0000) 2024-02-19T17:04:33.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:33.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:33.467620+0000) 2024-02-19T17:04:34.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:34.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:34.467898+0000) 2024-02-19T17:04:35.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:35.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:35.468188+0000) 2024-02-19T17:04:36.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:36.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:36.468475+0000) 2024-02-19T17:04:37.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:37.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:37.468708+0000) 2024-02-19T17:04:38.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:38.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:38.468998+0000) 2024-02-19T17:04:39.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:39.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:39.469279+0000) 2024-02-19T17:04:40.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:40.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:40.469567+0000) 2024-02-19T17:04:41.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:41.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:41.469871+0000) 2024-02-19T17:04:42.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:42.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:42.470115+0000) 2024-02-19T17:04:43.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:43.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:43.470348+0000) 2024-02-19T17:04:44.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:44.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:44.470631+0000) 2024-02-19T17:04:45.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:45.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:45.470920+0000) 2024-02-19T17:04:46.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:46.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:46.471084+0000) 2024-02-19T17:04:47.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:47.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:47.471255+0000) 2024-02-19T17:04:48.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:48.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:48.471458+0000) 2024-02-19T17:04:49.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:49.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:49.471640+0000) 2024-02-19T17:04:50.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:50.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:50.471849+0000) 2024-02-19T17:04:51.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:51.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:51.472102+0000) 2024-02-19T17:04:52.229 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:04:52.233 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:04:52.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:52.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:52.472396+0000) 2024-02-19T17:04:53.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:53.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:53.472622+0000) 2024-02-19T17:04:54.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:54.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:54.472763+0000) 2024-02-19T17:04:55.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:55.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:55.473028+0000) 2024-02-19T17:04:56.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:56.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:56.473323+0000) 2024-02-19T17:04:57.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:57.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:57.473552+0000) 2024-02-19T17:04:58.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:58.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:58.473826+0000) 2024-02-19T17:04:59.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:04:59.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:04:59.474028+0000) 2024-02-19T17:05:00.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:00.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:00.474298+0000) 2024-02-19T17:05:01.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:01.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:01.474537+0000) 2024-02-19T17:05:02.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:02.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:02.474771+0000) 2024-02-19T17:05:03.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:03.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:03.475045+0000) 2024-02-19T17:05:04.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:04.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:04.475343+0000) 2024-02-19T17:05:05.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:05.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:05.475576+0000) 2024-02-19T17:05:06.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:06.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:06.475834+0000) 2024-02-19T17:05:07.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:07.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:07.476003+0000) 2024-02-19T17:05:08.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:08.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:08.476266+0000) 2024-02-19T17:05:09.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:09.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:09.476529+0000) 2024-02-19T17:05:10.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:10.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:10.476770+0000) 2024-02-19T17:05:11.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:11.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:11.477007+0000) 2024-02-19T17:05:12.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:12.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:12.477205+0000) 2024-02-19T17:05:13.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:13.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:13.477487+0000) 2024-02-19T17:05:14.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:14.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:14.477751+0000) 2024-02-19T17:05:15.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:15.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:15.478016+0000) 2024-02-19T17:05:16.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:16.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:16.478209+0000) 2024-02-19T17:05:17.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:17.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:17.478406+0000) 2024-02-19T17:05:18.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:18.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:18.478608+0000) 2024-02-19T17:05:19.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:19.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:19.478870+0000) 2024-02-19T17:05:20.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:20.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:20.479110+0000) 2024-02-19T17:05:21.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:21.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:21.479318+0000) 2024-02-19T17:05:22.368 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:05:22.371 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:05:22.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:22.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:22.479511+0000) 2024-02-19T17:05:23.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:23.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:23.479778+0000) 2024-02-19T17:05:24.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:24.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:24.480052+0000) 2024-02-19T17:05:25.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:25.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:25.480296+0000) 2024-02-19T17:05:26.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:26.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:26.480495+0000) 2024-02-19T17:05:27.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:27.474+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:27.480727+0000) 2024-02-19T17:05:28.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:28.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:28.480961+0000) 2024-02-19T17:05:29.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:29.474+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:29.481212+0000) 2024-02-19T17:05:30.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:30.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:30.481415+0000) 2024-02-19T17:05:31.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:31.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:31.481660+0000) 2024-02-19T17:05:32.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:32.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:32.481880+0000) 2024-02-19T17:05:33.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:33.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:33.482061+0000) 2024-02-19T17:05:34.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:34.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:34.482242+0000) 2024-02-19T17:05:35.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:35.474+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:35.482490+0000) 2024-02-19T17:05:36.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:36.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:36.482695+0000) 2024-02-19T17:05:37.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:37.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:37.482857+0000) 2024-02-19T17:05:38.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:38.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:38.482996+0000) 2024-02-19T17:05:39.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:39.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:39.483189+0000) 2024-02-19T17:05:40.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:40.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:40.483411+0000) 2024-02-19T17:05:41.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:41.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:41.483673+0000) 2024-02-19T17:05:42.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:42.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:42.483911+0000) 2024-02-19T17:05:43.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:43.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:43.484139+0000) 2024-02-19T17:05:44.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:44.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:44.484375+0000) 2024-02-19T17:05:45.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:45.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:45.484618+0000) 2024-02-19T17:05:46.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:46.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:46.484877+0000) 2024-02-19T17:05:47.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:47.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:47.485069+0000) 2024-02-19T17:05:48.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:48.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:48.485342+0000) 2024-02-19T17:05:49.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:49.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:49.485588+0000) 2024-02-19T17:05:50.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:50.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:50.485825+0000) 2024-02-19T17:05:51.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:51.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:51.486066+0000) 2024-02-19T17:05:52.424 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:05:52.427 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:05:52.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:52.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:52.486308+0000) 2024-02-19T17:05:53.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:53.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:53.486538+0000) 2024-02-19T17:05:54.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:54.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:54.486786+0000) 2024-02-19T17:05:55.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:55.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:55.487017+0000) 2024-02-19T17:05:56.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:56.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:56.487241+0000) 2024-02-19T17:05:57.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:57.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:57.487477+0000) 2024-02-19T17:05:58.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:58.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:58.487632+0000) 2024-02-19T17:05:59.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:05:59.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:05:59.487793+0000) 2024-02-19T17:06:00.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:00.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:00.488034+0000) 2024-02-19T17:06:01.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:01.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:01.488289+0000) 2024-02-19T17:06:02.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:02.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:02.488492+0000) 2024-02-19T17:06:03.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:03.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:03.488728+0000) 2024-02-19T17:06:04.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:04.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:04.488884+0000) 2024-02-19T17:06:05.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:05.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:05.489113+0000) 2024-02-19T17:06:06.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:06.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:06.489374+0000) 2024-02-19T17:06:07.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:07.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:07.489625+0000) 2024-02-19T17:06:08.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:08.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:08.489865+0000) 2024-02-19T17:06:09.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:09.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:09.490037+0000) 2024-02-19T17:06:10.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:10.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:10.490242+0000) 2024-02-19T17:06:11.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:11.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:11.490483+0000) 2024-02-19T17:06:12.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:12.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:12.490755+0000) 2024-02-19T17:06:13.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:13.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:13.491000+0000) 2024-02-19T17:06:14.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:14.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:14.491242+0000) 2024-02-19T17:06:15.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:15.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:15.491493+0000) 2024-02-19T17:06:16.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:16.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:16.491722+0000) 2024-02-19T17:06:17.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:17.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:17.491964+0000) 2024-02-19T17:06:18.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:18.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:18.492225+0000) 2024-02-19T17:06:19.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:19.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:19.492495+0000) 2024-02-19T17:06:20.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:20.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:20.492738+0000) 2024-02-19T17:06:21.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:21.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:21.492970+0000) 2024-02-19T17:06:22.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:22.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:22.493213+0000) 2024-02-19T17:06:22.487 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:06:22.491 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:06:23.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:23.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:23.493450+0000) 2024-02-19T17:06:24.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:24.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:24.493703+0000) 2024-02-19T17:06:25.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:25.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:25.493909+0000) 2024-02-19T17:06:26.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:26.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:26.494162+0000) 2024-02-19T17:06:27.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:27.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:27.494411+0000) 2024-02-19T17:06:28.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:28.489+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:28.494604+0000) 2024-02-19T17:06:29.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:29.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:29.494849+0000) 2024-02-19T17:06:30.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:30.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:30.495062+0000) 2024-02-19T17:06:31.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:31.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:31.495317+0000) 2024-02-19T17:06:32.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:32.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:32.495573+0000) 2024-02-19T17:06:33.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:33.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:33.495821+0000) 2024-02-19T17:06:34.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:34.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:34.496052+0000) 2024-02-19T17:06:35.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:35.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:35.496318+0000) 2024-02-19T17:06:36.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:36.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:36.496592+0000) 2024-02-19T17:06:37.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:37.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:37.496831+0000) 2024-02-19T17:06:38.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:38.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:38.497088+0000) 2024-02-19T17:06:39.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:39.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:39.497356+0000) 2024-02-19T17:06:40.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:40.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:40.497585+0000) 2024-02-19T17:06:41.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:41.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:41.497830+0000) 2024-02-19T17:06:42.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:42.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:42.498055+0000) 2024-02-19T17:06:43.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:43.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:43.498296+0000) 2024-02-19T17:06:44.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:44.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:44.498557+0000) 2024-02-19T17:06:45.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:45.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:45.498783+0000) 2024-02-19T17:06:46.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:46.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:46.499009+0000) 2024-02-19T17:06:47.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:47.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:47.499198+0000) 2024-02-19T17:06:48.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:48.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:48.499392+0000) 2024-02-19T17:06:49.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:49.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:49.499607+0000) 2024-02-19T17:06:50.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:50.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:50.499865+0000) 2024-02-19T17:06:51.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:51.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:51.500143+0000) 2024-02-19T17:06:52.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:52.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:52.500273+0000) 2024-02-19T17:06:52.617 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:06:52.621 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:06:53.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:53.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:53.500470+0000) 2024-02-19T17:06:54.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:54.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:54.500724+0000) 2024-02-19T17:06:55.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:55.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:55.500922+0000) 2024-02-19T17:06:56.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:56.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:56.501182+0000) 2024-02-19T17:06:57.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:57.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:57.501435+0000) 2024-02-19T17:06:58.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:58.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:58.501672+0000) 2024-02-19T17:06:59.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:06:59.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:06:59.501915+0000) 2024-02-19T17:07:00.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:00.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:00.502080+0000) 2024-02-19T17:07:01.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:01.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:01.502347+0000) 2024-02-19T17:07:02.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:02.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:02.502585+0000) 2024-02-19T17:07:03.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:03.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:03.502815+0000) 2024-02-19T17:07:04.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:04.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:04.503009+0000) 2024-02-19T17:07:05.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:05.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:05.503260+0000) 2024-02-19T17:07:06.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:06.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:06.503502+0000) 2024-02-19T17:07:07.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:07.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:07.503752+0000) 2024-02-19T17:07:08.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:08.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:08.504029+0000) 2024-02-19T17:07:09.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:09.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:09.504265+0000) 2024-02-19T17:07:10.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:10.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:10.504502+0000) 2024-02-19T17:07:11.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:11.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:11.504663+0000) 2024-02-19T17:07:12.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:12.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:12.504831+0000) 2024-02-19T17:07:13.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:13.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:13.505071+0000) 2024-02-19T17:07:14.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:14.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:14.505317+0000) 2024-02-19T17:07:15.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:15.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:15.505557+0000) 2024-02-19T17:07:16.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:16.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:16.505826+0000) 2024-02-19T17:07:17.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:17.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:17.506085+0000) 2024-02-19T17:07:18.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:18.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:18.506324+0000) 2024-02-19T17:07:19.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:19.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:19.506559+0000) 2024-02-19T17:07:20.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:20.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:20.506826+0000) 2024-02-19T17:07:21.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:21.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:21.507067+0000) 2024-02-19T17:07:22.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:22.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:22.507268+0000) 2024-02-19T17:07:22.731 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:07:22.734 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:07:23.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:23.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:23.507533+0000) 2024-02-19T17:07:24.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:24.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:24.507773+0000) 2024-02-19T17:07:25.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:25.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:25.507956+0000) 2024-02-19T17:07:26.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:26.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:26.508175+0000) 2024-02-19T17:07:27.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:27.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:27.508353+0000) 2024-02-19T17:07:28.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:28.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:28.508595+0000) 2024-02-19T17:07:29.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:29.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:29.508827+0000) 2024-02-19T17:07:30.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:30.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:30.509070+0000) 2024-02-19T17:07:31.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:31.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:31.509259+0000) 2024-02-19T17:07:32.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:32.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:32.509432+0000) 2024-02-19T17:07:33.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:33.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:33.509623+0000) 2024-02-19T17:07:34.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:34.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:34.509881+0000) 2024-02-19T17:07:35.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:35.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:35.510113+0000) 2024-02-19T17:07:36.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:36.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:36.510285+0000) 2024-02-19T17:07:37.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:37.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:37.510523+0000) 2024-02-19T17:07:38.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:38.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:38.510805+0000) 2024-02-19T17:07:39.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:39.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:39.511044+0000) 2024-02-19T17:07:40.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:40.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:40.511306+0000) 2024-02-19T17:07:41.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:41.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:41.511550+0000) 2024-02-19T17:07:42.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:42.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:42.511769+0000) 2024-02-19T17:07:43.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:43.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:43.512043+0000) 2024-02-19T17:07:44.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:44.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:44.512215+0000) 2024-02-19T17:07:45.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:45.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:45.512399+0000) 2024-02-19T17:07:46.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:46.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:46.512668+0000) 2024-02-19T17:07:47.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:47.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:47.512909+0000) 2024-02-19T17:07:48.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:48.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:48.513155+0000) 2024-02-19T17:07:49.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:49.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:49.513404+0000) 2024-02-19T17:07:50.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:50.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:50.513581+0000) 2024-02-19T17:07:51.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:51.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:51.513841+0000) 2024-02-19T17:07:52.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:52.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:52.514118+0000) 2024-02-19T17:07:52.804 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:07:52.807 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:07:53.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:53.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:53.514361+0000) 2024-02-19T17:07:54.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:54.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:54.514560+0000) 2024-02-19T17:07:55.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:55.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:55.514810+0000) 2024-02-19T17:07:56.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:56.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:56.515085+0000) 2024-02-19T17:07:57.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:57.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:57.515260+0000) 2024-02-19T17:07:58.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:58.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:58.515524+0000) 2024-02-19T17:07:59.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:07:59.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:07:59.515725+0000) 2024-02-19T17:08:00.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:00.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:00.515883+0000) 2024-02-19T17:08:01.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:01.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:01.516146+0000) 2024-02-19T17:08:02.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:02.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:02.516330+0000) 2024-02-19T17:08:03.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:03.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:03.516518+0000) 2024-02-19T17:08:04.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:04.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:04.516706+0000) 2024-02-19T17:08:05.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:05.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:05.516904+0000) 2024-02-19T17:08:06.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:06.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:06.517152+0000) 2024-02-19T17:08:07.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:07.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:07.517391+0000) 2024-02-19T17:08:08.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:08.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:08.517659+0000) 2024-02-19T17:08:09.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:09.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:09.517924+0000) 2024-02-19T17:08:10.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:10.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:10.518149+0000) 2024-02-19T17:08:11.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:11.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:11.518376+0000) 2024-02-19T17:08:12.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:12.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:12.518611+0000) 2024-02-19T17:08:13.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:13.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:13.518802+0000) 2024-02-19T17:08:14.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:14.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:14.519034+0000) 2024-02-19T17:08:15.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:15.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:15.519282+0000) 2024-02-19T17:08:16.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:16.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:16.519536+0000) 2024-02-19T17:08:17.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:17.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:17.519759+0000) 2024-02-19T17:08:18.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:18.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:18.519999+0000) 2024-02-19T17:08:19.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:19.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:19.520250+0000) 2024-02-19T17:08:20.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:20.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:20.520413+0000) 2024-02-19T17:08:21.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:21.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:21.520631+0000) 2024-02-19T17:08:22.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:22.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:22.520863+0000) 2024-02-19T17:08:22.858 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:08:22.862 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:08:23.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:23.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:23.521114+0000) 2024-02-19T17:08:24.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:24.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:24.521293+0000) 2024-02-19T17:08:25.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:25.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:25.521516+0000) 2024-02-19T17:08:26.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:26.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:26.521718+0000) 2024-02-19T17:08:27.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:27.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:27.521943+0000) 2024-02-19T17:08:28.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:28.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:28.522162+0000) 2024-02-19T17:08:29.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:29.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:29.522409+0000) 2024-02-19T17:08:30.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:30.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:30.522585+0000) 2024-02-19T17:08:31.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:31.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:31.522842+0000) 2024-02-19T17:08:32.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:32.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:32.523028+0000) 2024-02-19T17:08:33.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:33.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:33.523186+0000) 2024-02-19T17:08:34.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:34.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:34.523363+0000) 2024-02-19T17:08:35.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:35.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:35.523614+0000) 2024-02-19T17:08:36.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:36.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:36.523819+0000) 2024-02-19T17:08:37.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:37.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:37.524059+0000) 2024-02-19T17:08:38.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:38.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:38.524295+0000) 2024-02-19T17:08:39.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:39.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:39.524496+0000) 2024-02-19T17:08:40.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:40.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:40.524762+0000) 2024-02-19T17:08:41.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:41.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:41.525003+0000) 2024-02-19T17:08:42.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:42.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:42.525212+0000) 2024-02-19T17:08:43.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:43.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:43.525489+0000) 2024-02-19T17:08:44.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:44.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:44.525770+0000) 2024-02-19T17:08:45.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:45.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:45.526054+0000) 2024-02-19T17:08:46.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:46.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:46.526303+0000) 2024-02-19T17:08:47.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:47.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:47.526543+0000) 2024-02-19T17:08:48.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:48.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:48.526822+0000) 2024-02-19T17:08:49.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:49.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:49.527065+0000) 2024-02-19T17:08:50.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:50.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:50.527340+0000) 2024-02-19T17:08:51.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:51.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:51.527515+0000) 2024-02-19T17:08:52.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:52.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:52.527780+0000) 2024-02-19T17:08:52.997 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:08:53.000 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:08:53.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:53.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:53.527952+0000) 2024-02-19T17:08:54.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:54.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:54.528158+0000) 2024-02-19T17:08:55.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:55.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:55.528437+0000) 2024-02-19T17:08:56.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:56.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:56.528670+0000) 2024-02-19T17:08:57.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:57.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:57.528918+0000) 2024-02-19T17:08:58.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:58.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:58.529157+0000) 2024-02-19T17:08:59.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:08:59.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:08:59.529425+0000) 2024-02-19T17:09:00.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:00.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:00.529610+0000) 2024-02-19T17:09:01.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:01.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:01.529842+0000) 2024-02-19T17:09:02.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:02.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:02.530105+0000) 2024-02-19T17:09:03.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:03.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:03.530346+0000) 2024-02-19T17:09:04.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:04.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:04.530589+0000) 2024-02-19T17:09:05.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:05.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:05.530826+0000) 2024-02-19T17:09:06.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:06.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:06.531084+0000) 2024-02-19T17:09:07.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:07.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:07.531343+0000) 2024-02-19T17:09:08.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:08.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:08.531597+0000) 2024-02-19T17:09:09.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:09.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:09.531850+0000) 2024-02-19T17:09:10.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:10.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:10.532042+0000) 2024-02-19T17:09:11.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:11.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:11.532304+0000) 2024-02-19T17:09:12.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:12.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:12.532539+0000) 2024-02-19T17:09:13.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:13.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:13.532787+0000) 2024-02-19T17:09:14.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:14.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:14.532958+0000) 2024-02-19T17:09:15.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:15.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:15.533144+0000) 2024-02-19T17:09:16.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:16.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:16.533302+0000) 2024-02-19T17:09:17.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:17.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:17.533543+0000) 2024-02-19T17:09:18.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:18.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:18.533796+0000) 2024-02-19T17:09:19.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:19.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:19.534006+0000) 2024-02-19T17:09:20.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:20.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:20.534262+0000) 2024-02-19T17:09:21.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:21.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:21.534534+0000) 2024-02-19T17:09:22.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:22.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:22.534807+0000) 2024-02-19T17:09:23.111 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:09:23.115 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:09:23.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:23.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:23.535053+0000) 2024-02-19T17:09:24.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:24.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:24.535291+0000) 2024-02-19T17:09:25.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:25.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:25.535528+0000) 2024-02-19T17:09:26.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:26.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:26.535780+0000) 2024-02-19T17:09:27.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:27.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:27.536027+0000) 2024-02-19T17:09:28.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:28.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:28.536263+0000) 2024-02-19T17:09:29.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:29.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:29.536450+0000) 2024-02-19T17:09:30.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:30.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:30.536691+0000) 2024-02-19T17:09:31.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:31.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:31.536940+0000) 2024-02-19T17:09:32.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:32.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:32.537176+0000) 2024-02-19T17:09:33.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:33.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:33.537439+0000) 2024-02-19T17:09:34.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:34.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:34.537645+0000) 2024-02-19T17:09:35.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:35.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:35.537900+0000) 2024-02-19T17:09:36.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:36.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:36.538152+0000) 2024-02-19T17:09:37.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:37.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:37.538390+0000) 2024-02-19T17:09:38.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:38.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:38.538628+0000) 2024-02-19T17:09:39.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:39.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:39.538844+0000) 2024-02-19T17:09:40.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:40.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:40.539088+0000) 2024-02-19T17:09:41.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:41.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:41.539333+0000) 2024-02-19T17:09:42.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:42.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:42.539524+0000) 2024-02-19T17:09:43.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:43.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:43.539742+0000) 2024-02-19T17:09:44.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:44.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:44.540003+0000) 2024-02-19T17:09:45.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:45.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:45.540210+0000) 2024-02-19T17:09:46.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:46.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:46.540471+0000) 2024-02-19T17:09:47.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:47.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:47.540707+0000) 2024-02-19T17:09:48.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:48.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:48.540953+0000) 2024-02-19T17:09:49.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:49.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:49.541203+0000) 2024-02-19T17:09:50.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:50.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:50.541466+0000) 2024-02-19T17:09:51.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:51.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:51.541700+0000) 2024-02-19T17:09:52.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:52.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:52.541955+0000) 2024-02-19T17:09:53.223 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:09:53.227 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:09:53.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:53.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:53.542197+0000) 2024-02-19T17:09:54.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:54.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:54.542377+0000) 2024-02-19T17:09:55.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:55.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:55.542581+0000) 2024-02-19T17:09:56.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:56.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:56.542845+0000) 2024-02-19T17:09:57.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:57.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:57.543060+0000) 2024-02-19T17:09:58.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:58.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:58.543321+0000) 2024-02-19T17:09:59.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:09:59.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:09:59.543569+0000) 2024-02-19T17:10:00.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:00.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:00.543819+0000) 2024-02-19T17:10:01.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:01.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:01.544059+0000) 2024-02-19T17:10:02.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:02.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:02.544288+0000) 2024-02-19T17:10:03.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:03.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:03.544464+0000) 2024-02-19T17:10:04.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:04.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:04.544696+0000) 2024-02-19T17:10:05.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:05.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:05.544956+0000) 2024-02-19T17:10:06.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:06.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:06.545192+0000) 2024-02-19T17:10:07.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:07.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:07.545367+0000) 2024-02-19T17:10:08.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:08.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:08.545621+0000) 2024-02-19T17:10:09.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:09.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:09.545791+0000) 2024-02-19T17:10:10.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:10.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:10.546068+0000) 2024-02-19T17:10:11.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:11.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:11.546322+0000) 2024-02-19T17:10:12.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:12.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:12.546567+0000) 2024-02-19T17:10:13.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:13.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:13.546865+0000) 2024-02-19T17:10:14.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:14.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:14.547138+0000) 2024-02-19T17:10:15.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:15.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:15.547384+0000) 2024-02-19T17:10:16.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:16.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:16.547647+0000) 2024-02-19T17:10:17.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:17.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:17.547921+0000) 2024-02-19T17:10:18.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:18.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:18.548168+0000) 2024-02-19T17:10:19.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:19.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:19.548350+0000) 2024-02-19T17:10:20.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:20.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:20.548551+0000) 2024-02-19T17:10:21.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:21.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:21.548850+0000) 2024-02-19T17:10:22.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:22.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:22.549124+0000) 2024-02-19T17:10:23.279 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:10:23.283 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:10:23.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:23.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:23.549302+0000) 2024-02-19T17:10:24.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:24.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:24.549453+0000) 2024-02-19T17:10:25.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:25.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:25.549619+0000) 2024-02-19T17:10:26.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:26.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:26.549816+0000) 2024-02-19T17:10:27.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:27.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:27.550044+0000) 2024-02-19T17:10:28.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:28.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:28.550314+0000) 2024-02-19T17:10:29.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:29.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:29.550511+0000) 2024-02-19T17:10:30.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:30.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:30.550808+0000) 2024-02-19T17:10:31.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:31.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:31.550950+0000) 2024-02-19T17:10:32.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:32.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:32.551169+0000) 2024-02-19T17:10:33.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:33.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:33.551420+0000) 2024-02-19T17:10:34.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:34.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:34.551669+0000) 2024-02-19T17:10:35.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:35.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:35.551839+0000) 2024-02-19T17:10:36.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:36.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:36.552081+0000) 2024-02-19T17:10:37.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:37.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:37.552356+0000) 2024-02-19T17:10:38.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:38.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:38.552613+0000) 2024-02-19T17:10:39.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:39.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:39.552838+0000) 2024-02-19T17:10:40.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:40.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:40.553026+0000) 2024-02-19T17:10:41.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:41.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:41.553210+0000) 2024-02-19T17:10:42.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:42.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:42.553420+0000) 2024-02-19T17:10:43.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:43.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:43.553649+0000) 2024-02-19T17:10:44.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:44.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:44.553923+0000) 2024-02-19T17:10:45.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:45.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:45.554181+0000) 2024-02-19T17:10:46.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:46.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:46.554334+0000) 2024-02-19T17:10:47.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:47.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:47.554551+0000) 2024-02-19T17:10:48.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:48.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:48.554820+0000) 2024-02-19T17:10:49.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:49.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:49.555051+0000) 2024-02-19T17:10:50.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:50.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:50.555322+0000) 2024-02-19T17:10:51.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:51.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:51.555520+0000) 2024-02-19T17:10:52.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:52.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:52.555742+0000) 2024-02-19T17:10:53.384 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:10:53.387 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:10:53.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:53.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:53.555982+0000) 2024-02-19T17:10:54.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:54.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:54.556216+0000) 2024-02-19T17:10:55.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:55.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:55.556458+0000) 2024-02-19T17:10:56.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:56.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:56.556709+0000) 2024-02-19T17:10:57.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:57.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:57.556975+0000) 2024-02-19T17:10:58.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:58.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:58.557093+0000) 2024-02-19T17:10:59.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:10:59.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:10:59.557250+0000) 2024-02-19T17:11:00.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:00.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:00.557487+0000) 2024-02-19T17:11:01.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:01.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:01.557646+0000) 2024-02-19T17:11:02.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:02.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:02.557818+0000) 2024-02-19T17:11:03.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:03.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:03.558040+0000) 2024-02-19T17:11:04.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:04.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:04.558278+0000) 2024-02-19T17:11:05.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:05.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:05.558527+0000) 2024-02-19T17:11:06.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:06.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:06.558801+0000) 2024-02-19T17:11:07.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:07.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:07.558982+0000) 2024-02-19T17:11:08.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:08.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:08.559158+0000) 2024-02-19T17:11:09.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:09.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:09.559455+0000) 2024-02-19T17:11:10.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:10.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:10.559715+0000) 2024-02-19T17:11:11.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:11.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:11.559968+0000) 2024-02-19T17:11:12.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:12.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:12.560168+0000) 2024-02-19T17:11:13.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:13.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:13.560357+0000) 2024-02-19T17:11:14.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:14.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:14.560520+0000) 2024-02-19T17:11:15.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:15.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:15.560742+0000) 2024-02-19T17:11:16.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:16.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:16.560930+0000) 2024-02-19T17:11:17.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:17.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:17.561154+0000) 2024-02-19T17:11:18.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:18.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:18.561386+0000) 2024-02-19T17:11:19.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:19.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:19.561627+0000) 2024-02-19T17:11:20.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:20.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:20.561917+0000) 2024-02-19T17:11:21.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:21.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:21.562104+0000) 2024-02-19T17:11:22.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:22.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:22.562324+0000) 2024-02-19T17:11:23.506 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:11:23.510 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:11:23.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:23.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:23.562562+0000) 2024-02-19T17:11:24.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:24.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:24.562868+0000) 2024-02-19T17:11:25.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:25.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:25.563038+0000) 2024-02-19T17:11:26.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:26.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:26.563247+0000) 2024-02-19T17:11:27.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:27.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:27.563480+0000) 2024-02-19T17:11:28.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:28.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:28.563679+0000) 2024-02-19T17:11:29.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:29.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:29.563895+0000) 2024-02-19T17:11:30.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:30.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:30.564129+0000) 2024-02-19T17:11:31.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:31.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:31.564369+0000) 2024-02-19T17:11:32.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:32.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:32.564631+0000) 2024-02-19T17:11:33.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:33.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:33.564879+0000) 2024-02-19T17:11:34.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:34.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:34.565084+0000) 2024-02-19T17:11:35.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:35.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:35.565310+0000) 2024-02-19T17:11:36.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:36.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:36.565485+0000) 2024-02-19T17:11:37.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:37.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:37.565644+0000) 2024-02-19T17:11:38.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:38.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:38.565883+0000) 2024-02-19T17:11:39.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:39.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:39.566062+0000) 2024-02-19T17:11:40.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:40.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:40.566296+0000) 2024-02-19T17:11:41.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:41.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:41.566517+0000) 2024-02-19T17:11:42.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:42.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:42.566800+0000) 2024-02-19T17:11:43.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:43.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:43.566994+0000) 2024-02-19T17:11:44.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:44.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:44.567171+0000) 2024-02-19T17:11:45.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:45.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:45.567421+0000) 2024-02-19T17:11:46.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:46.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:46.567681+0000) 2024-02-19T17:11:47.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:47.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:47.567916+0000) 2024-02-19T17:11:48.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:48.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:48.568148+0000) 2024-02-19T17:11:49.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:49.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:49.568371+0000) 2024-02-19T17:11:50.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:50.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:50.568542+0000) 2024-02-19T17:11:51.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:51.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:51.568779+0000) 2024-02-19T17:11:52.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:52.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:52.568964+0000) 2024-02-19T17:11:53.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:53.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:53.569201+0000) 2024-02-19T17:11:53.586 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:11:53.591 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:11:54.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:54.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:54.569410+0000) 2024-02-19T17:11:55.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:55.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:55.569579+0000) 2024-02-19T17:11:56.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:56.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:56.569831+0000) 2024-02-19T17:11:57.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:57.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:57.570044+0000) 2024-02-19T17:11:58.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:58.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:58.570224+0000) 2024-02-19T17:11:59.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:11:59.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:11:59.570467+0000) 2024-02-19T17:12:00.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:00.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:00.570735+0000) 2024-02-19T17:12:01.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:01.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:01.570995+0000) 2024-02-19T17:12:02.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:02.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:02.571227+0000) 2024-02-19T17:12:03.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:03.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:03.571457+0000) 2024-02-19T17:12:04.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:04.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:04.571702+0000) 2024-02-19T17:12:05.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:05.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:05.571928+0000) 2024-02-19T17:12:06.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:06.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:06.572183+0000) 2024-02-19T17:12:07.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:07.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:07.572441+0000) 2024-02-19T17:12:08.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:08.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:08.572664+0000) 2024-02-19T17:12:09.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:09.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:09.572935+0000) 2024-02-19T17:12:10.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:10.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:10.573178+0000) 2024-02-19T17:12:11.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:11.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:11.573425+0000) 2024-02-19T17:12:12.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:12.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:12.573666+0000) 2024-02-19T17:12:13.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:13.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:13.573877+0000) 2024-02-19T17:12:14.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:14.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:14.574106+0000) 2024-02-19T17:12:15.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:15.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:15.574293+0000) 2024-02-19T17:12:16.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:16.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:16.574524+0000) 2024-02-19T17:12:17.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:17.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:17.574802+0000) 2024-02-19T17:12:18.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:18.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:18.574975+0000) 2024-02-19T17:12:19.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:19.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:19.575229+0000) 2024-02-19T17:12:20.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:20.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:20.575421+0000) 2024-02-19T17:12:21.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:21.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:21.575669+0000) 2024-02-19T17:12:22.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:22.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:22.575798+0000) 2024-02-19T17:12:23.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:23.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:23.576034+0000) 2024-02-19T17:12:23.700 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:12:23.704 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:12:24.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:24.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:24.576276+0000) 2024-02-19T17:12:25.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:25.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:25.576521+0000) 2024-02-19T17:12:26.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:26.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:26.576755+0000) 2024-02-19T17:12:27.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:27.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:27.576987+0000) 2024-02-19T17:12:28.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:28.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:28.577174+0000) 2024-02-19T17:12:29.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:29.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:29.577439+0000) 2024-02-19T17:12:30.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:30.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:30.577600+0000) 2024-02-19T17:12:31.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:31.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:31.577786+0000) 2024-02-19T17:12:32.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:32.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:32.577943+0000) 2024-02-19T17:12:33.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:33.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:33.578160+0000) 2024-02-19T17:12:34.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:34.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:34.578394+0000) 2024-02-19T17:12:35.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:35.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:35.578608+0000) 2024-02-19T17:12:36.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:36.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:36.578833+0000) 2024-02-19T17:12:37.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:37.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:37.579008+0000) 2024-02-19T17:12:38.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:38.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:38.579213+0000) 2024-02-19T17:12:39.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:39.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:39.579484+0000) 2024-02-19T17:12:40.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:40.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:40.579729+0000) 2024-02-19T17:12:41.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:41.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:41.579987+0000) 2024-02-19T17:12:42.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:42.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:42.580260+0000) 2024-02-19T17:12:43.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:43.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:43.580526+0000) 2024-02-19T17:12:44.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:44.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:44.580792+0000) 2024-02-19T17:12:45.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:45.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:45.581057+0000) 2024-02-19T17:12:46.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:46.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:46.581261+0000) 2024-02-19T17:12:47.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:47.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:47.581414+0000) 2024-02-19T17:12:48.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:48.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:48.581637+0000) 2024-02-19T17:12:49.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:49.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:49.581811+0000) 2024-02-19T17:12:50.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:50.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:50.582042+0000) 2024-02-19T17:12:51.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:51.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:51.582342+0000) 2024-02-19T17:12:52.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:52.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:52.582606+0000) 2024-02-19T17:12:53.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:53.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:53.582924+0000) 2024-02-19T17:12:53.763 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:12:53.767 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:12:54.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:54.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:54.583066+0000) 2024-02-19T17:12:55.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:55.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:55.583202+0000) 2024-02-19T17:12:56.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:56.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:56.583418+0000) 2024-02-19T17:12:57.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:57.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:57.583661+0000) 2024-02-19T17:12:58.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:58.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:58.583830+0000) 2024-02-19T17:12:59.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:12:59.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:12:59.584084+0000) 2024-02-19T17:13:00.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:00.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:00.584356+0000) 2024-02-19T17:13:01.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:01.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:01.584618+0000) 2024-02-19T17:13:02.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:02.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:02.584809+0000) 2024-02-19T17:13:03.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:03.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:03.584987+0000) 2024-02-19T17:13:04.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:04.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:04.585229+0000) 2024-02-19T17:13:05.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:05.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:05.585482+0000) 2024-02-19T17:13:06.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:06.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:06.585726+0000) 2024-02-19T17:13:07.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:07.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:07.585954+0000) 2024-02-19T17:13:08.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:08.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:08.586139+0000) 2024-02-19T17:13:09.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:09.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:09.586363+0000) 2024-02-19T17:13:10.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:10.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:10.586577+0000) 2024-02-19T17:13:11.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:11.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:11.586823+0000) 2024-02-19T17:13:12.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:12.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:12.587005+0000) 2024-02-19T17:13:13.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:13.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:13.587248+0000) 2024-02-19T17:13:14.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:14.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:14.587427+0000) 2024-02-19T17:13:15.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:15.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:15.587655+0000) 2024-02-19T17:13:16.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:16.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:16.587889+0000) 2024-02-19T17:13:17.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:17.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:17.588082+0000) 2024-02-19T17:13:18.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:18.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:18.588305+0000) 2024-02-19T17:13:19.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:19.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:19.588573+0000) 2024-02-19T17:13:20.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:20.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:20.588860+0000) 2024-02-19T17:13:21.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:21.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:21.589090+0000) 2024-02-19T17:13:22.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:22.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:22.589255+0000) 2024-02-19T17:13:23.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:23.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:23.589502+0000) 2024-02-19T17:13:23.902 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:13:23.905 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:13:24.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:24.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:24.589666+0000) 2024-02-19T17:13:25.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:25.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:25.589926+0000) 2024-02-19T17:13:26.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:26.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:26.590194+0000) 2024-02-19T17:13:27.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:27.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:27.590333+0000) 2024-02-19T17:13:28.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:28.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:28.590525+0000) 2024-02-19T17:13:29.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:29.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:29.590830+0000) 2024-02-19T17:13:30.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:30.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:30.591074+0000) 2024-02-19T17:13:31.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:31.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:31.591311+0000) 2024-02-19T17:13:32.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:32.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:32.591563+0000) 2024-02-19T17:13:33.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:33.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:33.591855+0000) 2024-02-19T17:13:34.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:34.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:34.592113+0000) 2024-02-19T17:13:35.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:35.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:35.592381+0000) 2024-02-19T17:13:36.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:36.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:36.592613+0000) 2024-02-19T17:13:37.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:37.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:37.592868+0000) 2024-02-19T17:13:38.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:38.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:38.593110+0000) 2024-02-19T17:13:39.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:39.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:39.593353+0000) 2024-02-19T17:13:40.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:40.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:40.593613+0000) 2024-02-19T17:13:41.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:41.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:41.593866+0000) 2024-02-19T17:13:42.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:42.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:42.594116+0000) 2024-02-19T17:13:43.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:43.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:43.594398+0000) 2024-02-19T17:13:44.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:44.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:44.594664+0000) 2024-02-19T17:13:45.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:45.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:45.594961+0000) 2024-02-19T17:13:46.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:46.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:46.595228+0000) 2024-02-19T17:13:47.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:47.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:47.595472+0000) 2024-02-19T17:13:48.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:48.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:48.595718+0000) 2024-02-19T17:13:49.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:49.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:49.596067+0000) 2024-02-19T17:13:50.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:50.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:50.596288+0000) 2024-02-19T17:13:51.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:51.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:51.596535+0000) 2024-02-19T17:13:52.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:52.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:52.596791+0000) 2024-02-19T17:13:53.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:53.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:53.597016+0000) 2024-02-19T17:13:54.032 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:13:54.036 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:13:54.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:54.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:54.597269+0000) 2024-02-19T17:13:55.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:55.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:55.597533+0000) 2024-02-19T17:13:56.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:56.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:56.597801+0000) 2024-02-19T17:13:57.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:57.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:57.598050+0000) 2024-02-19T17:13:58.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:58.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:58.598303+0000) 2024-02-19T17:13:59.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:13:59.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:13:59.598568+0000) 2024-02-19T17:14:00.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:00.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:00.598822+0000) 2024-02-19T17:14:01.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:01.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:01.599109+0000) 2024-02-19T17:14:02.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:02.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:02.599314+0000) 2024-02-19T17:14:03.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:03.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:03.599548+0000) 2024-02-19T17:14:04.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:04.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:04.599801+0000) 2024-02-19T17:14:05.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:05.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:05.600073+0000) 2024-02-19T17:14:06.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:06.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:06.600266+0000) 2024-02-19T17:14:07.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:07.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:07.600504+0000) 2024-02-19T17:14:08.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:08.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:08.600753+0000) 2024-02-19T17:14:09.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:09.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:09.600943+0000) 2024-02-19T17:14:10.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:10.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:10.601110+0000) 2024-02-19T17:14:11.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:11.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:11.601278+0000) 2024-02-19T17:14:12.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:12.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:12.601477+0000) 2024-02-19T17:14:13.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:13.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:13.601720+0000) 2024-02-19T17:14:14.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:14.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:14.601975+0000) 2024-02-19T17:14:15.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:15.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:15.602241+0000) 2024-02-19T17:14:16.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:16.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:16.602481+0000) 2024-02-19T17:14:17.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:17.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:17.602798+0000) 2024-02-19T17:14:18.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:18.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:18.603007+0000) 2024-02-19T17:14:19.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:19.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:19.603246+0000) 2024-02-19T17:14:20.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:20.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:20.603512+0000) 2024-02-19T17:14:21.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:21.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:21.603762+0000) 2024-02-19T17:14:22.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:22.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:22.603973+0000) 2024-02-19T17:14:23.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:23.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:23.604204+0000) 2024-02-19T17:14:24.172 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:14:24.177 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:14:24.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:24.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:24.604494+0000) 2024-02-19T17:14:25.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:25.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:25.604767+0000) 2024-02-19T17:14:26.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:26.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:26.605001+0000) 2024-02-19T17:14:27.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:27.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:27.605236+0000) 2024-02-19T17:14:28.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:28.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:28.605442+0000) 2024-02-19T17:14:29.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:29.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:29.605586+0000) 2024-02-19T17:14:30.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:30.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:30.605822+0000) 2024-02-19T17:14:31.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:31.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:31.606073+0000) 2024-02-19T17:14:32.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:32.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:32.606290+0000) 2024-02-19T17:14:33.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:33.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:33.606524+0000) 2024-02-19T17:14:34.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:34.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:34.606754+0000) 2024-02-19T17:14:35.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:35.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:35.606961+0000) 2024-02-19T17:14:36.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:36.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:36.607194+0000) 2024-02-19T17:14:37.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:37.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:37.607514+0000) 2024-02-19T17:14:38.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:38.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:38.607781+0000) 2024-02-19T17:14:39.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:39.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:39.608045+0000) 2024-02-19T17:14:40.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:40.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:40.608282+0000) 2024-02-19T17:14:41.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:41.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:41.608524+0000) 2024-02-19T17:14:42.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:42.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:42.608773+0000) 2024-02-19T17:14:43.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:43.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:43.609027+0000) 2024-02-19T17:14:44.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:44.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:44.609245+0000) 2024-02-19T17:14:45.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:45.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:45.609484+0000) 2024-02-19T17:14:46.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:46.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:46.609695+0000) 2024-02-19T17:14:47.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:47.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:47.609853+0000) 2024-02-19T17:14:48.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:48.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:48.610101+0000) 2024-02-19T17:14:49.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:49.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:49.610379+0000) 2024-02-19T17:14:50.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:50.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:50.610629+0000) 2024-02-19T17:14:51.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:51.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:51.610852+0000) 2024-02-19T17:14:52.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:52.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:52.611118+0000) 2024-02-19T17:14:53.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:53.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:53.611373+0000) 2024-02-19T17:14:54.419 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:14:54.423 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:14:54.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:54.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:54.611588+0000) 2024-02-19T17:14:55.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:55.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:55.611839+0000) 2024-02-19T17:14:56.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:56.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:56.612072+0000) 2024-02-19T17:14:57.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:57.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:57.612325+0000) 2024-02-19T17:14:58.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:58.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:58.612583+0000) 2024-02-19T17:14:59.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:14:59.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:14:59.612797+0000) 2024-02-19T17:15:00.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:00.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:00.612968+0000) 2024-02-19T17:15:01.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:01.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:01.613173+0000) 2024-02-19T17:15:02.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:02.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:02.613408+0000) 2024-02-19T17:15:03.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:03.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:03.613623+0000) 2024-02-19T17:15:04.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:04.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:04.613855+0000) 2024-02-19T17:15:05.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:05.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:05.614064+0000) 2024-02-19T17:15:06.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:06.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:06.614301+0000) 2024-02-19T17:15:07.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:07.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:07.614482+0000) 2024-02-19T17:15:08.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:08.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:08.614673+0000) 2024-02-19T17:15:09.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:09.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:09.614934+0000) 2024-02-19T17:15:10.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:10.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:10.615206+0000) 2024-02-19T17:15:11.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:11.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:11.615412+0000) 2024-02-19T17:15:12.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:12.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:12.615665+0000) 2024-02-19T17:15:13.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:13.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:13.615890+0000) 2024-02-19T17:15:14.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:14.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:14.616126+0000) 2024-02-19T17:15:15.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:15.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:15.616393+0000) 2024-02-19T17:15:16.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:16.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:16.616580+0000) 2024-02-19T17:15:17.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:17.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:17.616776+0000) 2024-02-19T17:15:18.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:18.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:18.617020+0000) 2024-02-19T17:15:19.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:19.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:19.617206+0000) 2024-02-19T17:15:20.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:20.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:20.617367+0000) 2024-02-19T17:15:21.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:21.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:21.617605+0000) 2024-02-19T17:15:22.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:22.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:22.617854+0000) 2024-02-19T17:15:23.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:23.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:23.618108+0000) 2024-02-19T17:15:24.550 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:15:24.553 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:15:24.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:24.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:24.618271+0000) 2024-02-19T17:15:25.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:25.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:25.618473+0000) 2024-02-19T17:15:26.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:26.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:26.618743+0000) 2024-02-19T17:15:27.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:27.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:27.618984+0000) 2024-02-19T17:15:28.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:28.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:28.619167+0000) 2024-02-19T17:15:29.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:29.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:29.619317+0000) 2024-02-19T17:15:30.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:30.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:30.619570+0000) 2024-02-19T17:15:31.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:31.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:31.619857+0000) 2024-02-19T17:15:32.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:32.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:32.620086+0000) 2024-02-19T17:15:33.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:33.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:33.620309+0000) 2024-02-19T17:15:34.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:34.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:34.620565+0000) 2024-02-19T17:15:35.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:35.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:35.620826+0000) 2024-02-19T17:15:36.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:36.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:36.621098+0000) 2024-02-19T17:15:37.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:37.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:37.621263+0000) 2024-02-19T17:15:38.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:38.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:38.621474+0000) 2024-02-19T17:15:39.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:39.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:39.621695+0000) 2024-02-19T17:15:40.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:40.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:40.621916+0000) 2024-02-19T17:15:41.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:41.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:41.622078+0000) 2024-02-19T17:15:42.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:42.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:42.622307+0000) 2024-02-19T17:15:43.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:43.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:43.622588+0000) 2024-02-19T17:15:44.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:44.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:44.622860+0000) 2024-02-19T17:15:45.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:45.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:45.623043+0000) 2024-02-19T17:15:46.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:46.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:46.623268+0000) 2024-02-19T17:15:47.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:47.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:47.623458+0000) 2024-02-19T17:15:48.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:48.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:48.623609+0000) 2024-02-19T17:15:49.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:49.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:49.623769+0000) 2024-02-19T17:15:50.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:50.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:50.624004+0000) 2024-02-19T17:15:51.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:51.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:51.624203+0000) 2024-02-19T17:15:52.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:52.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:52.624384+0000) 2024-02-19T17:15:53.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:53.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:53.624607+0000) 2024-02-19T17:15:54.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:54.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:54.624851+0000) 2024-02-19T17:15:54.680 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:15:54.683 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:15:55.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:55.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:55.625054+0000) 2024-02-19T17:15:56.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:56.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:56.625241+0000) 2024-02-19T17:15:57.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:57.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:57.625460+0000) 2024-02-19T17:15:58.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:58.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:58.625719+0000) 2024-02-19T17:15:59.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:15:59.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:15:59.625909+0000) 2024-02-19T17:16:00.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:00.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:00.626098+0000) 2024-02-19T17:16:01.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:01.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:01.626221+0000) 2024-02-19T17:16:02.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:02.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:02.626441+0000) 2024-02-19T17:16:03.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:03.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:03.626638+0000) 2024-02-19T17:16:04.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:04.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:04.626838+0000) 2024-02-19T17:16:05.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:05.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:05.627011+0000) 2024-02-19T17:16:06.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:06.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:06.627187+0000) 2024-02-19T17:16:07.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:07.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:07.627383+0000) 2024-02-19T17:16:08.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:08.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:08.627629+0000) 2024-02-19T17:16:09.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:09.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:09.627788+0000) 2024-02-19T17:16:10.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:10.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:10.628049+0000) 2024-02-19T17:16:11.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:11.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:11.628240+0000) 2024-02-19T17:16:12.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:12.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:12.628498+0000) 2024-02-19T17:16:13.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:13.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:13.628735+0000) 2024-02-19T17:16:14.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:14.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:14.628972+0000) 2024-02-19T17:16:15.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:15.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:15.629211+0000) 2024-02-19T17:16:16.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:16.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:16.629411+0000) 2024-02-19T17:16:17.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:17.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:17.629667+0000) 2024-02-19T17:16:18.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:18.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:18.629875+0000) 2024-02-19T17:16:19.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:19.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:19.630098+0000) 2024-02-19T17:16:20.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:20.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:20.630289+0000) 2024-02-19T17:16:21.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:21.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:21.630548+0000) 2024-02-19T17:16:22.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:22.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:22.630754+0000) 2024-02-19T17:16:23.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:23.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:23.630975+0000) 2024-02-19T17:16:24.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:24.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:24.631195+0000) 2024-02-19T17:16:24.756 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:16:24.760 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:16:25.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:25.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:25.631476+0000) 2024-02-19T17:16:26.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:26.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:26.631650+0000) 2024-02-19T17:16:27.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:27.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:27.631889+0000) 2024-02-19T17:16:28.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:28.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:28.632137+0000) 2024-02-19T17:16:29.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:29.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:29.632377+0000) 2024-02-19T17:16:30.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:30.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:30.632633+0000) 2024-02-19T17:16:31.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:31.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:31.632839+0000) 2024-02-19T17:16:32.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:32.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:32.633020+0000) 2024-02-19T17:16:33.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:33.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:33.633241+0000) 2024-02-19T17:16:34.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:34.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:34.633476+0000) 2024-02-19T17:16:35.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:35.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:35.633700+0000) 2024-02-19T17:16:36.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:36.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:36.633936+0000) 2024-02-19T17:16:37.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:37.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:37.634174+0000) 2024-02-19T17:16:38.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:38.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:38.634367+0000) 2024-02-19T17:16:39.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:39.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:39.634626+0000) 2024-02-19T17:16:40.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:40.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:40.634883+0000) 2024-02-19T17:16:41.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:41.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:41.635069+0000) 2024-02-19T17:16:42.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:42.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:42.635255+0000) 2024-02-19T17:16:43.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:43.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:43.635478+0000) 2024-02-19T17:16:44.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:44.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:44.635673+0000) 2024-02-19T17:16:45.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:45.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:45.635895+0000) 2024-02-19T17:16:46.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:46.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:46.636097+0000) 2024-02-19T17:16:47.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:47.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:47.636336+0000) 2024-02-19T17:16:48.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:48.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:48.636587+0000) 2024-02-19T17:16:49.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:49.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:49.636845+0000) 2024-02-19T17:16:50.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:50.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:50.637013+0000) 2024-02-19T17:16:51.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:51.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:51.637174+0000) 2024-02-19T17:16:52.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:52.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:52.637411+0000) 2024-02-19T17:16:53.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:53.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:53.637561+0000) 2024-02-19T17:16:54.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:54.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:54.637783+0000) 2024-02-19T17:16:54.889 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:16:54.893 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:16:55.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:55.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:55.638016+0000) 2024-02-19T17:16:56.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:56.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:56.638250+0000) 2024-02-19T17:16:57.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:57.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:57.638489+0000) 2024-02-19T17:16:58.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:58.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:58.638702+0000) 2024-02-19T17:16:59.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:16:59.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:16:59.638898+0000) 2024-02-19T17:17:00.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:00.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:00.639087+0000) 2024-02-19T17:17:01.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:01.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:01.639274+0000) 2024-02-19T17:17:02.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:02.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:02.639529+0000) 2024-02-19T17:17:03.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:03.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:03.639792+0000) 2024-02-19T17:17:04.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:04.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:04.639930+0000) 2024-02-19T17:17:05.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:05.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:05.640174+0000) 2024-02-19T17:17:06.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:06.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:06.640382+0000) 2024-02-19T17:17:07.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:07.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:07.640524+0000) 2024-02-19T17:17:08.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:08.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:08.640780+0000) 2024-02-19T17:17:09.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:09.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:09.640923+0000) 2024-02-19T17:17:10.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:10.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:10.641170+0000) 2024-02-19T17:17:11.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:11.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:11.641422+0000) 2024-02-19T17:17:12.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:12.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:12.641625+0000) 2024-02-19T17:17:13.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:13.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:13.641818+0000) 2024-02-19T17:17:14.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:14.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:14.641976+0000) 2024-02-19T17:17:15.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:15.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:15.642236+0000) 2024-02-19T17:17:16.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:16.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:16.642484+0000) 2024-02-19T17:17:17.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:17.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:17.642693+0000) 2024-02-19T17:17:18.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:18.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:18.642886+0000) 2024-02-19T17:17:19.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:19.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:19.643149+0000) 2024-02-19T17:17:20.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:20.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:20.643392+0000) 2024-02-19T17:17:21.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:21.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:21.643659+0000) 2024-02-19T17:17:22.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:22.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:22.643826+0000) 2024-02-19T17:17:23.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:23.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:23.644025+0000) 2024-02-19T17:17:24.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:24.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:24.644313+0000) 2024-02-19T17:17:25.029 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:17:25.032 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:17:25.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:25.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:25.644579+0000) 2024-02-19T17:17:26.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:26.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:26.644855+0000) 2024-02-19T17:17:27.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:27.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:27.645054+0000) 2024-02-19T17:17:28.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:28.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:28.645344+0000) 2024-02-19T17:17:29.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:29.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:29.645655+0000) 2024-02-19T17:17:30.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:30.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:30.645920+0000) 2024-02-19T17:17:31.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:31.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:31.646182+0000) 2024-02-19T17:17:32.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:32.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:32.646414+0000) 2024-02-19T17:17:33.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:33.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:33.646610+0000) 2024-02-19T17:17:34.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:34.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:34.646845+0000) 2024-02-19T17:17:35.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:35.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:35.647127+0000) 2024-02-19T17:17:36.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:36.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:36.647373+0000) 2024-02-19T17:17:37.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:37.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:37.647543+0000) 2024-02-19T17:17:38.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:38.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:38.647701+0000) 2024-02-19T17:17:39.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:39.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:39.647896+0000) 2024-02-19T17:17:40.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:40.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:40.648156+0000) 2024-02-19T17:17:41.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:41.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:41.648397+0000) 2024-02-19T17:17:42.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:42.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:42.648600+0000) 2024-02-19T17:17:43.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:43.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:43.648796+0000) 2024-02-19T17:17:44.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:44.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:44.649066+0000) 2024-02-19T17:17:45.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:45.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:45.649293+0000) 2024-02-19T17:17:46.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:46.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:46.649521+0000) 2024-02-19T17:17:47.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:47.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:47.649762+0000) 2024-02-19T17:17:48.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:48.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:48.649941+0000) 2024-02-19T17:17:49.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:49.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:49.650134+0000) 2024-02-19T17:17:50.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:50.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:50.650415+0000) 2024-02-19T17:17:51.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:51.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:51.650667+0000) 2024-02-19T17:17:52.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:52.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:52.650908+0000) 2024-02-19T17:17:53.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:53.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:53.651068+0000) 2024-02-19T17:17:54.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:54.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:54.651213+0000) 2024-02-19T17:17:55.109 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:17:55.112 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:17:55.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:55.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:55.651482+0000) 2024-02-19T17:17:56.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:56.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:56.651743+0000) 2024-02-19T17:17:57.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:57.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:57.651942+0000) 2024-02-19T17:17:58.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:58.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:58.652180+0000) 2024-02-19T17:17:59.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:17:59.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:17:59.652454+0000) 2024-02-19T17:18:00.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:00.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:00.652690+0000) 2024-02-19T17:18:01.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:01.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:01.652876+0000) 2024-02-19T17:18:02.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:02.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:02.653102+0000) 2024-02-19T17:18:03.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:03.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:03.653344+0000) 2024-02-19T17:18:04.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:04.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:04.653606+0000) 2024-02-19T17:18:05.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:05.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:05.653905+0000) 2024-02-19T17:18:06.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:06.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:06.654046+0000) 2024-02-19T17:18:07.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:07.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:07.654253+0000) 2024-02-19T17:18:08.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:08.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:08.654492+0000) 2024-02-19T17:18:09.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:09.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:09.654797+0000) 2024-02-19T17:18:10.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:10.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:10.654956+0000) 2024-02-19T17:18:11.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:11.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:11.655108+0000) 2024-02-19T17:18:12.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:12.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:12.655306+0000) 2024-02-19T17:18:13.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:13.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:13.655550+0000) 2024-02-19T17:18:14.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:14.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:14.655816+0000) 2024-02-19T17:18:15.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:15.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:15.656016+0000) 2024-02-19T17:18:16.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:16.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:16.656251+0000) 2024-02-19T17:18:17.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:17.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:17.656449+0000) 2024-02-19T17:18:18.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:18.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:18.656688+0000) 2024-02-19T17:18:19.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:19.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:19.656910+0000) 2024-02-19T17:18:20.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:20.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:20.657097+0000) 2024-02-19T17:18:21.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:21.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:21.657289+0000) 2024-02-19T17:18:22.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:22.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:22.657467+0000) 2024-02-19T17:18:23.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:23.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:23.657715+0000) 2024-02-19T17:18:24.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:24.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:24.657949+0000) 2024-02-19T17:18:25.249 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:18:25.252 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:18:25.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:25.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:25.658142+0000) 2024-02-19T17:18:26.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:26.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:26.658348+0000) 2024-02-19T17:18:27.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:27.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:27.658518+0000) 2024-02-19T17:18:28.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:28.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:28.658805+0000) 2024-02-19T17:18:29.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:29.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:29.658928+0000) 2024-02-19T17:18:30.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:30.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:30.659070+0000) 2024-02-19T17:18:31.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:31.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:31.659213+0000) 2024-02-19T17:18:32.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:32.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:32.659442+0000) 2024-02-19T17:18:33.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:33.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:33.659729+0000) 2024-02-19T17:18:34.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:34.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:34.660001+0000) 2024-02-19T17:18:35.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:35.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:35.660218+0000) 2024-02-19T17:18:36.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:36.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:36.660461+0000) 2024-02-19T17:18:37.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:37.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:37.660682+0000) 2024-02-19T17:18:38.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:38.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:38.660914+0000) 2024-02-19T17:18:39.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:39.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:39.661184+0000) 2024-02-19T17:18:40.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:40.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:40.661376+0000) 2024-02-19T17:18:41.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:41.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:41.661565+0000) 2024-02-19T17:18:42.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:42.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:42.661833+0000) 2024-02-19T17:18:43.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:43.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:43.662154+0000) 2024-02-19T17:18:44.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:44.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:44.662409+0000) 2024-02-19T17:18:45.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:45.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:45.662781+0000) 2024-02-19T17:18:46.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:46.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:46.663008+0000) 2024-02-19T17:18:47.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:47.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:47.663209+0000) 2024-02-19T17:18:48.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:48.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:48.663427+0000) 2024-02-19T17:18:49.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:49.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:49.663686+0000) 2024-02-19T17:18:50.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:50.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:50.663858+0000) 2024-02-19T17:18:51.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:51.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:51.664135+0000) 2024-02-19T17:18:52.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:52.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:52.664442+0000) 2024-02-19T17:18:53.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:53.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:53.664740+0000) 2024-02-19T17:18:54.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:54.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:54.664923+0000) 2024-02-19T17:18:55.370 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:18:55.373 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:18:55.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:55.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:55.665210+0000) 2024-02-19T17:18:56.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:56.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:56.665513+0000) 2024-02-19T17:18:57.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:57.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:57.665765+0000) 2024-02-19T17:18:58.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:58.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:58.666030+0000) 2024-02-19T17:18:59.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:18:59.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:18:59.666325+0000) 2024-02-19T17:19:00.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:00.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:00.666502+0000) 2024-02-19T17:19:01.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:01.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:01.666826+0000) 2024-02-19T17:19:02.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:02.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:02.667018+0000) 2024-02-19T17:19:03.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:03.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:03.667186+0000) 2024-02-19T17:19:04.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:04.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:04.667419+0000) 2024-02-19T17:19:05.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:05.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:05.667711+0000) 2024-02-19T17:19:06.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:06.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:06.668042+0000) 2024-02-19T17:19:07.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:07.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:07.668297+0000) 2024-02-19T17:19:08.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:08.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:08.668505+0000) 2024-02-19T17:19:09.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:09.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:09.668804+0000) 2024-02-19T17:19:10.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:10.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:10.669044+0000) 2024-02-19T17:19:11.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:11.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:11.669355+0000) 2024-02-19T17:19:12.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:12.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:12.669660+0000) 2024-02-19T17:19:13.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:13.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:13.669864+0000) 2024-02-19T17:19:14.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:14.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:14.670115+0000) 2024-02-19T17:19:15.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:15.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:15.670349+0000) 2024-02-19T17:19:16.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:16.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:16.670626+0000) 2024-02-19T17:19:17.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:17.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:17.670837+0000) 2024-02-19T17:19:18.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:18.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:18.671020+0000) 2024-02-19T17:19:19.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:19.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:19.671281+0000) 2024-02-19T17:19:20.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:20.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:20.671523+0000) 2024-02-19T17:19:21.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:21.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:21.671791+0000) 2024-02-19T17:19:22.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:22.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:22.672023+0000) 2024-02-19T17:19:23.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:23.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:23.672295+0000) 2024-02-19T17:19:24.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:24.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:24.672494+0000) 2024-02-19T17:19:25.452 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:19:25.455 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:19:25.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:25.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:25.672713+0000) 2024-02-19T17:19:26.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:26.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:26.672983+0000) 2024-02-19T17:19:27.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:27.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:27.673263+0000) 2024-02-19T17:19:28.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:28.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:28.673564+0000) 2024-02-19T17:19:29.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:29.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:29.673849+0000) 2024-02-19T17:19:30.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:30.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:30.674140+0000) 2024-02-19T17:19:31.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:31.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:31.674341+0000) 2024-02-19T17:19:32.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:32.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:32.674595+0000) 2024-02-19T17:19:33.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:33.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:33.674867+0000) 2024-02-19T17:19:34.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:34.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:34.675185+0000) 2024-02-19T17:19:35.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:35.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:35.675480+0000) 2024-02-19T17:19:36.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:36.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:36.675749+0000) 2024-02-19T17:19:37.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:37.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:37.675999+0000) 2024-02-19T17:19:38.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:38.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:38.676253+0000) 2024-02-19T17:19:39.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:39.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:39.676532+0000) 2024-02-19T17:19:40.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:40.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:40.676742+0000) 2024-02-19T17:19:41.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:41.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:41.676910+0000) 2024-02-19T17:19:42.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:42.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:42.677194+0000) 2024-02-19T17:19:43.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:43.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:43.677466+0000) 2024-02-19T17:19:44.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:44.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:44.677713+0000) 2024-02-19T17:19:45.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:45.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:45.677969+0000) 2024-02-19T17:19:46.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:46.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:46.678215+0000) 2024-02-19T17:19:47.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:47.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:47.678497+0000) 2024-02-19T17:19:48.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:48.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:48.678824+0000) 2024-02-19T17:19:49.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:49.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:49.679073+0000) 2024-02-19T17:19:50.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:50.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:50.679328+0000) 2024-02-19T17:19:51.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:51.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:51.679584+0000) 2024-02-19T17:19:52.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:52.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:52.679899+0000) 2024-02-19T17:19:53.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:53.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:53.680088+0000) 2024-02-19T17:19:54.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:54.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:54.680297+0000) 2024-02-19T17:19:55.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:55.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:55.680505+0000) 2024-02-19T17:19:55.730 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:19:55.736 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:19:56.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:56.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:56.680730+0000) 2024-02-19T17:19:57.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:57.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:57.680904+0000) 2024-02-19T17:19:58.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:58.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:58.681066+0000) 2024-02-19T17:19:59.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:19:59.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:19:59.681269+0000) 2024-02-19T17:20:00.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:00.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:00.681511+0000) 2024-02-19T17:20:01.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:01.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:01.681748+0000) 2024-02-19T17:20:02.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:02.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:02.681958+0000) 2024-02-19T17:20:03.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:03.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:03.682112+0000) 2024-02-19T17:20:04.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:04.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:04.682234+0000) 2024-02-19T17:20:05.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:05.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:05.682414+0000) 2024-02-19T17:20:06.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:06.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:06.682597+0000) 2024-02-19T17:20:07.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:07.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:07.682809+0000) 2024-02-19T17:20:08.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:08.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:08.682952+0000) 2024-02-19T17:20:09.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:09.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:09.683101+0000) 2024-02-19T17:20:10.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:10.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:10.683290+0000) 2024-02-19T17:20:11.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:11.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:11.683533+0000) 2024-02-19T17:20:12.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:12.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:12.683754+0000) 2024-02-19T17:20:13.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:13.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:13.683988+0000) 2024-02-19T17:20:14.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:14.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:14.684214+0000) 2024-02-19T17:20:15.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:15.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:15.684445+0000) 2024-02-19T17:20:16.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:16.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:16.684652+0000) 2024-02-19T17:20:17.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:17.684+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:17.684791+0000) 2024-02-19T17:20:18.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:18.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:18.685016+0000) 2024-02-19T17:20:19.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:19.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:19.685241+0000) 2024-02-19T17:20:20.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:20.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:20.685479+0000) 2024-02-19T17:20:21.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:21.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:21.685716+0000) 2024-02-19T17:20:22.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:22.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:22.685898+0000) 2024-02-19T17:20:23.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:23.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:23.686120+0000) 2024-02-19T17:20:24.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:24.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:24.686363+0000) 2024-02-19T17:20:25.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:25.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:25.686579+0000) 2024-02-19T17:20:25.789 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:20:25.793 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:20:26.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:26.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:26.686831+0000) 2024-02-19T17:20:27.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:27.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:27.686961+0000) 2024-02-19T17:20:28.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:28.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:28.687081+0000) 2024-02-19T17:20:29.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:29.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:29.687298+0000) 2024-02-19T17:20:30.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:30.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:30.687488+0000) 2024-02-19T17:20:31.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:31.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:31.687682+0000) 2024-02-19T17:20:32.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:32.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:32.687880+0000) 2024-02-19T17:20:33.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:33.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:33.688124+0000) 2024-02-19T17:20:34.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:34.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:34.688294+0000) 2024-02-19T17:20:35.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:35.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:35.688550+0000) 2024-02-19T17:20:36.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:36.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:36.688742+0000) 2024-02-19T17:20:37.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:37.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:37.688927+0000) 2024-02-19T17:20:38.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:38.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:38.689165+0000) 2024-02-19T17:20:39.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:39.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:39.689427+0000) 2024-02-19T17:20:40.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:40.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:40.689643+0000) 2024-02-19T17:20:41.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:41.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:41.689871+0000) 2024-02-19T17:20:42.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:42.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:42.690079+0000) 2024-02-19T17:20:43.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:43.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:43.690292+0000) 2024-02-19T17:20:44.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:44.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:44.690517+0000) 2024-02-19T17:20:45.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:45.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:45.690832+0000) 2024-02-19T17:20:46.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:46.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:46.691016+0000) 2024-02-19T17:20:47.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:47.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:47.691189+0000) 2024-02-19T17:20:48.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:48.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:48.691444+0000) 2024-02-19T17:20:49.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:49.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:49.691702+0000) 2024-02-19T17:20:50.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:50.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:50.691945+0000) 2024-02-19T17:20:51.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:51.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:51.692170+0000) 2024-02-19T17:20:52.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:52.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:52.692374+0000) 2024-02-19T17:20:53.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:53.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:53.692616+0000) 2024-02-19T17:20:54.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:54.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:54.692818+0000) 2024-02-19T17:20:55.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:55.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:55.693030+0000) 2024-02-19T17:20:55.925 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:20:55.928 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:20:56.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:56.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:56.693280+0000) 2024-02-19T17:20:57.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:57.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:57.693450+0000) 2024-02-19T17:20:58.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:58.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:58.693692+0000) 2024-02-19T17:20:59.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:20:59.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:20:59.693951+0000) 2024-02-19T17:21:00.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:00.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:00.694186+0000) 2024-02-19T17:21:01.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:01.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:01.694465+0000) 2024-02-19T17:21:02.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:02.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:02.694696+0000) 2024-02-19T17:21:03.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:03.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:03.694920+0000) 2024-02-19T17:21:04.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:04.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:04.695134+0000) 2024-02-19T17:21:05.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:05.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:05.695338+0000) 2024-02-19T17:21:06.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:06.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:06.695559+0000) 2024-02-19T17:21:07.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:07.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:07.695802+0000) 2024-02-19T17:21:08.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:08.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:08.696053+0000) 2024-02-19T17:21:09.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:09.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:09.696290+0000) 2024-02-19T17:21:10.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:10.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:10.696523+0000) 2024-02-19T17:21:11.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:11.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:11.696773+0000) 2024-02-19T17:21:12.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:12.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:12.696972+0000) 2024-02-19T17:21:13.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:13.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:13.697192+0000) 2024-02-19T17:21:14.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:14.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:14.697426+0000) 2024-02-19T17:21:15.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:15.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:15.697654+0000) 2024-02-19T17:21:16.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:16.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:16.697910+0000) 2024-02-19T17:21:17.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:17.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:17.698102+0000) 2024-02-19T17:21:18.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:18.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:18.698336+0000) 2024-02-19T17:21:19.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:19.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:19.698616+0000) 2024-02-19T17:21:20.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:20.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:20.698895+0000) 2024-02-19T17:21:21.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:21.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:21.699095+0000) 2024-02-19T17:21:22.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:22.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:22.699230+0000) 2024-02-19T17:21:23.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:23.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:23.699467+0000) 2024-02-19T17:21:24.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:24.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:24.699718+0000) 2024-02-19T17:21:25.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:25.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:25.699928+0000) 2024-02-19T17:21:26.001 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:21:26.004 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:21:26.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:26.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:26.700170+0000) 2024-02-19T17:21:27.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:27.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:27.700402+0000) 2024-02-19T17:21:28.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:28.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:28.700627+0000) 2024-02-19T17:21:29.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:29.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:29.700786+0000) 2024-02-19T17:21:30.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:30.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:30.701027+0000) 2024-02-19T17:21:31.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:31.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:31.701199+0000) 2024-02-19T17:21:32.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:32.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:32.701411+0000) 2024-02-19T17:21:33.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:33.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:33.701649+0000) 2024-02-19T17:21:34.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:34.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:34.701897+0000) 2024-02-19T17:21:35.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:35.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:35.702048+0000) 2024-02-19T17:21:36.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:36.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:36.702211+0000) 2024-02-19T17:21:37.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:37.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:37.702455+0000) 2024-02-19T17:21:38.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:38.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:38.702642+0000) 2024-02-19T17:21:39.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:39.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:39.702888+0000) 2024-02-19T17:21:40.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:40.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:40.703066+0000) 2024-02-19T17:21:41.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:41.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:41.703214+0000) 2024-02-19T17:21:42.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:42.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:42.703404+0000) 2024-02-19T17:21:43.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:43.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:43.703616+0000) 2024-02-19T17:21:44.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:44.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:44.703763+0000) 2024-02-19T17:21:45.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:45.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:45.703960+0000) 2024-02-19T17:21:46.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:46.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:46.704144+0000) 2024-02-19T17:21:47.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:47.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:47.704385+0000) 2024-02-19T17:21:48.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:48.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:48.704579+0000) 2024-02-19T17:21:49.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:49.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:49.704803+0000) 2024-02-19T17:21:50.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:50.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:50.705041+0000) 2024-02-19T17:21:51.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:51.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:51.705271+0000) 2024-02-19T17:21:52.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:52.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:52.705505+0000) 2024-02-19T17:21:53.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:53.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:53.705730+0000) 2024-02-19T17:21:54.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:54.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:54.705994+0000) 2024-02-19T17:21:55.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:55.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:55.706200+0000) 2024-02-19T17:21:56.119 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:21:56.122 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:21:56.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:56.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:56.706427+0000) 2024-02-19T17:21:57.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:57.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:57.706620+0000) 2024-02-19T17:21:58.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:58.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:58.706851+0000) 2024-02-19T17:21:59.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:21:59.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:21:59.707089+0000) 2024-02-19T17:22:00.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:00.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:00.707285+0000) 2024-02-19T17:22:01.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:01.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:01.707541+0000) 2024-02-19T17:22:02.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:02.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:02.707790+0000) 2024-02-19T17:22:03.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:03.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:03.708057+0000) 2024-02-19T17:22:04.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:04.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:04.708301+0000) 2024-02-19T17:22:05.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:05.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:05.708547+0000) 2024-02-19T17:22:06.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:06.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:06.708794+0000) 2024-02-19T17:22:07.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:07.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:07.708977+0000) 2024-02-19T17:22:08.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:08.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:08.709240+0000) 2024-02-19T17:22:09.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:09.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:09.709439+0000) 2024-02-19T17:22:10.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:10.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:10.709671+0000) 2024-02-19T17:22:11.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:11.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:11.709868+0000) 2024-02-19T17:22:12.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:12.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:12.710089+0000) 2024-02-19T17:22:13.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:13.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:13.710288+0000) 2024-02-19T17:22:14.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:14.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:14.710520+0000) 2024-02-19T17:22:15.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:15.710+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:15.710837+0000) 2024-02-19T17:22:16.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:16.710+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:16.711090+0000) 2024-02-19T17:22:17.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:17.710+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:17.711343+0000) 2024-02-19T17:22:18.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:18.710+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:18.711589+0000) 2024-02-19T17:22:19.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:19.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:19.711810+0000) 2024-02-19T17:22:20.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:20.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:20.712083+0000) 2024-02-19T17:22:21.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:21.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:21.712403+0000) 2024-02-19T17:22:22.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:22.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:22.712674+0000) 2024-02-19T17:22:23.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:23.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:23.712911+0000) 2024-02-19T17:22:24.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:24.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:24.713147+0000) 2024-02-19T17:22:25.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:25.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:25.713409+0000) 2024-02-19T17:22:26.174 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:22:26.177 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:22:26.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:26.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:26.713673+0000) 2024-02-19T17:22:27.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:27.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:27.713947+0000) 2024-02-19T17:22:28.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:28.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:28.714171+0000) 2024-02-19T17:22:29.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:29.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:29.714400+0000) 2024-02-19T17:22:30.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:30.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:30.714644+0000) 2024-02-19T17:22:31.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:31.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:31.714911+0000) 2024-02-19T17:22:32.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:32.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:32.715135+0000) 2024-02-19T17:22:33.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:33.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:33.715367+0000) 2024-02-19T17:22:34.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:34.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:34.715566+0000) 2024-02-19T17:22:35.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:35.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:35.715788+0000) 2024-02-19T17:22:36.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:36.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:36.716034+0000) 2024-02-19T17:22:37.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:37.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:37.716288+0000) 2024-02-19T17:22:38.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:38.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:38.716528+0000) 2024-02-19T17:22:39.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:39.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:39.716759+0000) 2024-02-19T17:22:40.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:40.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:40.716945+0000) 2024-02-19T17:22:41.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:41.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:41.717161+0000) 2024-02-19T17:22:42.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:42.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:42.717308+0000) 2024-02-19T17:22:43.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:43.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:43.717486+0000) 2024-02-19T17:22:44.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:44.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:44.717709+0000) 2024-02-19T17:22:45.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:45.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:45.718000+0000) 2024-02-19T17:22:46.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:46.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:46.718195+0000) 2024-02-19T17:22:47.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:47.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:47.718381+0000) 2024-02-19T17:22:48.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:48.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:48.718601+0000) 2024-02-19T17:22:49.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:49.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:49.718862+0000) 2024-02-19T17:22:50.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:50.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:50.719054+0000) 2024-02-19T17:22:51.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:51.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:51.719300+0000) 2024-02-19T17:22:52.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:52.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:52.719468+0000) 2024-02-19T17:22:53.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:53.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:53.719684+0000) 2024-02-19T17:22:54.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:54.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:54.719917+0000) 2024-02-19T17:22:55.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:55.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:55.720165+0000) 2024-02-19T17:22:56.254 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:22:56.259 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:22:56.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:56.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:56.720412+0000) 2024-02-19T17:22:57.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:57.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:57.720618+0000) 2024-02-19T17:22:58.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:58.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:58.720858+0000) 2024-02-19T17:22:59.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:22:59.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:22:59.720993+0000) 2024-02-19T17:23:00.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:00.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:00.721239+0000) 2024-02-19T17:23:01.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:01.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:01.721512+0000) 2024-02-19T17:23:02.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:02.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:02.721748+0000) 2024-02-19T17:23:03.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:03.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:03.721998+0000) 2024-02-19T17:23:04.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:04.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:04.722176+0000) 2024-02-19T17:23:05.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:05.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:05.722423+0000) 2024-02-19T17:23:06.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:06.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:06.722658+0000) 2024-02-19T17:23:07.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:07.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:07.722865+0000) 2024-02-19T17:23:08.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:08.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:08.723037+0000) 2024-02-19T17:23:09.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:09.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:09.723256+0000) 2024-02-19T17:23:10.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:10.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:10.723514+0000) 2024-02-19T17:23:11.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:11.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:11.723761+0000) 2024-02-19T17:23:12.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:12.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:12.723935+0000) 2024-02-19T17:23:13.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:13.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:13.724204+0000) 2024-02-19T17:23:14.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:14.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:14.724448+0000) 2024-02-19T17:23:15.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:15.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:15.724697+0000) 2024-02-19T17:23:16.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:16.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:16.724973+0000) 2024-02-19T17:23:17.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:17.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:17.725174+0000) 2024-02-19T17:23:18.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:18.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:18.725429+0000) 2024-02-19T17:23:19.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:19.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:19.725669+0000) 2024-02-19T17:23:20.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:20.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:20.725904+0000) 2024-02-19T17:23:21.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:21.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:21.726187+0000) 2024-02-19T17:23:22.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:22.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:22.726460+0000) 2024-02-19T17:23:23.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:23.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:23.726690+0000) 2024-02-19T17:23:24.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:24.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:24.726880+0000) 2024-02-19T17:23:25.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:25.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:25.727103+0000) 2024-02-19T17:23:26.310 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:23:26.314 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:23:26.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:26.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:26.727371+0000) 2024-02-19T17:23:27.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:27.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:27.727564+0000) 2024-02-19T17:23:28.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:28.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:28.727811+0000) 2024-02-19T17:23:29.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:29.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:29.727998+0000) 2024-02-19T17:23:30.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:30.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:30.728243+0000) 2024-02-19T17:23:31.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:31.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:31.728496+0000) 2024-02-19T17:23:32.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:32.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:32.728726+0000) 2024-02-19T17:23:33.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:33.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:33.728929+0000) 2024-02-19T17:23:34.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:34.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:34.729220+0000) 2024-02-19T17:23:35.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:35.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:35.729503+0000) 2024-02-19T17:23:36.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:36.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:36.729718+0000) 2024-02-19T17:23:37.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:37.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:37.729940+0000) 2024-02-19T17:23:38.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:38.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:38.730187+0000) 2024-02-19T17:23:39.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:39.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:39.730428+0000) 2024-02-19T17:23:40.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:40.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:40.730668+0000) 2024-02-19T17:23:41.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:41.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:41.730907+0000) 2024-02-19T17:23:42.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:42.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:42.731138+0000) 2024-02-19T17:23:43.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:43.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:43.731332+0000) 2024-02-19T17:23:44.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:44.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:44.731568+0000) 2024-02-19T17:23:45.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:45.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:45.731795+0000) 2024-02-19T17:23:46.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:46.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:46.732043+0000) 2024-02-19T17:23:47.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:47.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:47.732305+0000) 2024-02-19T17:23:48.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:48.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:48.732576+0000) 2024-02-19T17:23:49.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:49.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:49.732842+0000) 2024-02-19T17:23:50.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:50.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:50.733097+0000) 2024-02-19T17:23:51.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:51.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:51.733343+0000) 2024-02-19T17:23:52.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:52.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:52.733583+0000) 2024-02-19T17:23:53.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:53.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:53.733829+0000) 2024-02-19T17:23:54.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:54.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:54.734060+0000) 2024-02-19T17:23:55.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:55.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:55.734304+0000) 2024-02-19T17:23:56.490 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:23:56.494 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:23:56.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:56.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:56.734542+0000) 2024-02-19T17:23:57.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:57.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:57.734840+0000) 2024-02-19T17:23:58.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:58.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:58.735121+0000) 2024-02-19T17:23:59.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:23:59.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:23:59.735342+0000) 2024-02-19T17:24:00.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:00.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:00.735598+0000) 2024-02-19T17:24:01.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:01.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:01.735847+0000) 2024-02-19T17:24:02.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:02.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:02.736109+0000) 2024-02-19T17:24:03.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:03.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:03.736309+0000) 2024-02-19T17:24:04.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:04.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:04.736482+0000) 2024-02-19T17:24:05.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:05.736+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:05.736736+0000) 2024-02-19T17:24:06.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:06.736+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:06.736921+0000) 2024-02-19T17:24:07.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:07.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:07.737130+0000) 2024-02-19T17:24:08.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:08.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:08.737468+0000) 2024-02-19T17:24:09.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:09.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:09.737665+0000) 2024-02-19T17:24:10.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:10.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:10.737921+0000) 2024-02-19T17:24:11.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:11.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:11.738203+0000) 2024-02-19T17:24:12.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:12.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:12.738445+0000) 2024-02-19T17:24:13.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:13.736+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:13.738689+0000) 2024-02-19T17:24:14.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:14.736+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:14.738901+0000) 2024-02-19T17:24:15.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:15.736+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:15.739053+0000) 2024-02-19T17:24:16.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:16.736+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:16.739338+0000) 2024-02-19T17:24:17.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:17.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:17.739605+0000) 2024-02-19T17:24:18.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:18.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:18.739857+0000) 2024-02-19T17:24:19.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:19.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:19.740125+0000) 2024-02-19T17:24:20.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:20.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:20.740385+0000) 2024-02-19T17:24:21.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:21.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:21.740623+0000) 2024-02-19T17:24:22.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:22.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:22.740866+0000) 2024-02-19T17:24:23.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:23.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:23.741108+0000) 2024-02-19T17:24:24.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:24.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:24.741383+0000) 2024-02-19T17:24:25.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:25.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:25.741650+0000) 2024-02-19T17:24:26.628 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:24:26.633 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:24:26.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:26.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:26.741888+0000) 2024-02-19T17:24:27.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:27.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:27.742111+0000) 2024-02-19T17:24:28.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:28.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:28.742336+0000) 2024-02-19T17:24:29.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:29.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:29.742577+0000) 2024-02-19T17:24:30.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:30.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:30.742820+0000) 2024-02-19T17:24:31.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:31.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:31.742975+0000) 2024-02-19T17:24:32.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:32.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:32.743175+0000) 2024-02-19T17:24:33.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:33.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:33.743419+0000) 2024-02-19T17:24:34.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:34.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:34.743693+0000) 2024-02-19T17:24:35.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:35.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:35.743879+0000) 2024-02-19T17:24:36.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:36.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:36.744129+0000) 2024-02-19T17:24:37.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:37.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:37.744377+0000) 2024-02-19T17:24:38.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:38.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:38.744644+0000) 2024-02-19T17:24:39.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:39.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:39.744890+0000) 2024-02-19T17:24:40.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:40.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:40.745120+0000) 2024-02-19T17:24:41.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:41.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:41.745376+0000) 2024-02-19T17:24:42.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:42.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:42.745636+0000) 2024-02-19T17:24:43.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:43.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:43.745909+0000) 2024-02-19T17:24:44.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:44.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:44.746154+0000) 2024-02-19T17:24:45.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:45.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:45.746417+0000) 2024-02-19T17:24:46.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:46.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:46.746610+0000) 2024-02-19T17:24:47.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:47.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:47.746872+0000) 2024-02-19T17:24:48.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:48.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:48.747024+0000) 2024-02-19T17:24:49.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:49.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:49.747155+0000) 2024-02-19T17:24:50.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:50.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:50.747381+0000) 2024-02-19T17:24:51.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:51.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:51.747625+0000) 2024-02-19T17:24:52.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:52.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:52.747874+0000) 2024-02-19T17:24:53.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:53.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:53.748117+0000) 2024-02-19T17:24:54.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:54.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:54.748340+0000) 2024-02-19T17:24:55.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:55.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:55.748613+0000) 2024-02-19T17:24:56.683 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:24:56.687 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:24:56.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:56.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:56.748902+0000) 2024-02-19T17:24:57.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:57.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:57.749133+0000) 2024-02-19T17:24:58.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:58.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:58.749305+0000) 2024-02-19T17:24:59.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:24:59.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:24:59.749499+0000) 2024-02-19T17:25:00.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:00.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:00.749699+0000) 2024-02-19T17:25:01.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:01.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:01.749926+0000) 2024-02-19T17:25:02.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:02.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:02.750204+0000) 2024-02-19T17:25:03.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:03.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:03.750449+0000) 2024-02-19T17:25:04.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:04.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:04.750748+0000) 2024-02-19T17:25:05.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:05.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:05.751002+0000) 2024-02-19T17:25:06.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:06.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:06.751215+0000) 2024-02-19T17:25:07.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:07.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:07.751488+0000) 2024-02-19T17:25:08.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:08.751+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:08.751663+0000) 2024-02-19T17:25:09.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:09.751+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:09.751928+0000) 2024-02-19T17:25:10.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:10.751+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:10.752181+0000) 2024-02-19T17:25:11.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:11.751+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:11.752458+0000) 2024-02-19T17:25:12.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:12.751+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:12.752663+0000) 2024-02-19T17:25:13.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:13.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:13.752887+0000) 2024-02-19T17:25:14.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:14.751+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:14.753061+0000) 2024-02-19T17:25:15.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:15.751+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:15.753254+0000) 2024-02-19T17:25:16.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:16.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:16.753485+0000) 2024-02-19T17:25:17.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:17.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:17.753737+0000) 2024-02-19T17:25:18.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:18.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:18.753970+0000) 2024-02-19T17:25:19.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:19.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:19.754144+0000) 2024-02-19T17:25:20.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:20.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:20.754369+0000) 2024-02-19T17:25:21.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:21.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:21.754640+0000) 2024-02-19T17:25:22.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:22.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:22.754880+0000) 2024-02-19T17:25:23.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:23.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:23.755123+0000) 2024-02-19T17:25:24.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:24.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:24.755379+0000) 2024-02-19T17:25:25.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:25.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:25.755622+0000) 2024-02-19T17:25:26.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:26.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:26.755854+0000) 2024-02-19T17:25:26.823 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:25:26.827 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:25:27.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:27.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:27.756083+0000) 2024-02-19T17:25:28.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:28.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:28.756322+0000) 2024-02-19T17:25:29.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:29.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:29.756580+0000) 2024-02-19T17:25:30.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:30.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:30.756768+0000) 2024-02-19T17:25:31.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:31.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:31.756978+0000) 2024-02-19T17:25:32.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:32.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:32.757226+0000) 2024-02-19T17:25:33.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:33.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:33.757486+0000) 2024-02-19T17:25:34.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:34.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:34.757743+0000) 2024-02-19T17:25:35.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:35.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:35.757947+0000) 2024-02-19T17:25:36.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:36.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:36.758211+0000) 2024-02-19T17:25:37.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:37.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:37.758474+0000) 2024-02-19T17:25:38.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:38.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:38.758743+0000) 2024-02-19T17:25:39.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:39.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:39.758972+0000) 2024-02-19T17:25:40.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:40.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:40.759168+0000) 2024-02-19T17:25:41.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:41.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:41.759360+0000) 2024-02-19T17:25:42.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:42.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:42.759606+0000) 2024-02-19T17:25:43.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:43.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:43.759861+0000) 2024-02-19T17:25:44.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:44.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:44.760170+0000) 2024-02-19T17:25:45.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:45.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:45.760438+0000) 2024-02-19T17:25:46.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:46.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:46.760673+0000) 2024-02-19T17:25:47.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:47.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:47.760851+0000) 2024-02-19T17:25:48.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:48.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:48.761044+0000) 2024-02-19T17:25:49.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:49.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:49.761255+0000) 2024-02-19T17:25:50.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:50.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:50.761457+0000) 2024-02-19T17:25:51.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:51.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:51.761605+0000) 2024-02-19T17:25:52.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:52.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:52.761823+0000) 2024-02-19T17:25:53.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:53.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:53.762071+0000) 2024-02-19T17:25:54.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:54.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:54.762354+0000) 2024-02-19T17:25:55.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:55.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:55.762602+0000) 2024-02-19T17:25:56.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:56.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:56.762817+0000) 2024-02-19T17:25:56.880 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:25:56.884 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:25:57.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:57.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:57.763061+0000) 2024-02-19T17:25:58.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:58.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:58.763268+0000) 2024-02-19T17:25:59.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:25:59.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:25:59.763512+0000) 2024-02-19T17:26:00.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:00.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:00.763707+0000) 2024-02-19T17:26:01.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:01.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:01.763933+0000) 2024-02-19T17:26:02.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:02.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:02.764164+0000) 2024-02-19T17:26:03.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:03.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:03.764386+0000) 2024-02-19T17:26:04.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:04.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:04.764576+0000) 2024-02-19T17:26:05.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:05.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:05.764812+0000) 2024-02-19T17:26:06.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:06.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:06.765058+0000) 2024-02-19T17:26:07.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:07.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:07.765284+0000) 2024-02-19T17:26:08.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:08.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:08.765484+0000) 2024-02-19T17:26:09.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:09.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:09.765678+0000) 2024-02-19T17:26:10.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:10.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:10.765904+0000) 2024-02-19T17:26:11.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:11.766+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:11.766154+0000) 2024-02-19T17:26:12.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:12.766+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:12.766328+0000) 2024-02-19T17:26:13.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:13.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:13.766545+0000) 2024-02-19T17:26:14.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:14.766+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:14.766817+0000) 2024-02-19T17:26:15.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:15.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:15.767051+0000) 2024-02-19T17:26:16.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:16.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:16.767305+0000) 2024-02-19T17:26:17.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:17.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:17.767541+0000) 2024-02-19T17:26:18.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:18.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:18.767808+0000) 2024-02-19T17:26:19.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:19.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:19.768059+0000) 2024-02-19T17:26:20.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:20.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:20.768302+0000) 2024-02-19T17:26:21.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:21.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:21.768538+0000) 2024-02-19T17:26:22.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:22.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:22.768714+0000) 2024-02-19T17:26:23.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:23.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:23.768950+0000) 2024-02-19T17:26:24.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:24.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:24.769199+0000) 2024-02-19T17:26:25.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:25.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:25.769426+0000) 2024-02-19T17:26:26.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:26.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:26.769621+0000) 2024-02-19T17:26:27.001 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:26:27.005 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:26:27.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:27.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:27.769842+0000) 2024-02-19T17:26:28.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:28.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:28.769999+0000) 2024-02-19T17:26:29.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:29.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:29.770257+0000) 2024-02-19T17:26:30.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:30.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:30.770527+0000) 2024-02-19T17:26:31.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:31.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:31.770801+0000) 2024-02-19T17:26:32.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:32.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:32.771036+0000) 2024-02-19T17:26:33.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:33.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:33.771237+0000) 2024-02-19T17:26:34.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:34.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:34.771460+0000) 2024-02-19T17:26:35.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:35.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:35.771723+0000) 2024-02-19T17:26:36.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:36.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:36.771924+0000) 2024-02-19T17:26:37.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:37.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:37.772145+0000) 2024-02-19T17:26:38.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:38.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:38.772394+0000) 2024-02-19T17:26:39.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:39.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:39.772646+0000) 2024-02-19T17:26:40.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:40.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:40.772886+0000) 2024-02-19T17:26:41.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:41.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:41.773108+0000) 2024-02-19T17:26:42.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:42.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:42.773288+0000) 2024-02-19T17:26:43.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:43.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:43.773544+0000) 2024-02-19T17:26:44.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:44.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:44.773713+0000) 2024-02-19T17:26:45.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:45.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:45.773977+0000) 2024-02-19T17:26:46.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:46.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:46.774199+0000) 2024-02-19T17:26:47.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:47.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:47.774400+0000) 2024-02-19T17:26:48.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:48.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:48.774659+0000) 2024-02-19T17:26:49.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:49.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:49.774862+0000) 2024-02-19T17:26:50.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:50.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:50.775106+0000) 2024-02-19T17:26:51.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:51.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:51.775343+0000) 2024-02-19T17:26:52.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:52.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:52.775551+0000) 2024-02-19T17:26:53.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:53.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:53.775768+0000) 2024-02-19T17:26:54.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:54.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:54.775908+0000) 2024-02-19T17:26:55.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:55.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:55.776138+0000) 2024-02-19T17:26:56.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:56.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:56.776314+0000) 2024-02-19T17:26:57.106 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:26:57.111 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:26:57.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:57.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:57.776553+0000) 2024-02-19T17:26:58.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:58.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:58.776721+0000) 2024-02-19T17:26:59.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:26:59.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:26:59.776920+0000) 2024-02-19T17:27:00.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:00.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:00.777123+0000) 2024-02-19T17:27:01.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:01.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:01.777379+0000) 2024-02-19T17:27:02.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:02.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:02.777634+0000) 2024-02-19T17:27:03.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:03.777+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:03.777913+0000) 2024-02-19T17:27:04.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:04.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:04.778110+0000) 2024-02-19T17:27:05.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:05.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:05.778307+0000) 2024-02-19T17:27:06.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:06.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:06.778501+0000) 2024-02-19T17:27:07.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:07.777+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:07.778798+0000) 2024-02-19T17:27:08.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:08.777+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:08.778957+0000) 2024-02-19T17:27:09.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:09.777+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:09.779209+0000) 2024-02-19T17:27:10.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:10.777+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:10.779451+0000) 2024-02-19T17:27:11.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:11.777+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:11.779636+0000) 2024-02-19T17:27:12.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:12.777+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:12.779804+0000) 2024-02-19T17:27:13.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:13.777+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:13.780038+0000) 2024-02-19T17:27:14.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:14.777+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:14.780304+0000) 2024-02-19T17:27:15.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:15.777+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:15.780536+0000) 2024-02-19T17:27:16.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:16.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:16.780740+0000) 2024-02-19T17:27:17.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:17.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:17.780930+0000) 2024-02-19T17:27:18.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:18.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:18.781108+0000) 2024-02-19T17:27:19.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:19.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:19.781355+0000) 2024-02-19T17:27:20.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:20.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:20.781517+0000) 2024-02-19T17:27:21.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:21.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:21.781727+0000) 2024-02-19T17:27:22.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:22.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:22.781946+0000) 2024-02-19T17:27:23.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:23.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:23.782202+0000) 2024-02-19T17:27:24.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:24.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:24.782396+0000) 2024-02-19T17:27:25.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:25.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:25.782666+0000) 2024-02-19T17:27:26.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:26.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:26.782986+0000) 2024-02-19T17:27:27.245 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:27:27.249 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:27:27.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:27.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:27.783271+0000) 2024-02-19T17:27:28.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:28.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:28.783552+0000) 2024-02-19T17:27:29.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:29.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:29.783793+0000) 2024-02-19T17:27:30.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:30.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:30.784055+0000) 2024-02-19T17:27:31.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:31.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:31.784303+0000) 2024-02-19T17:27:32.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:32.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:32.784476+0000) 2024-02-19T17:27:33.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:33.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:33.784737+0000) 2024-02-19T17:27:34.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:34.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:34.785042+0000) 2024-02-19T17:27:35.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:35.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:35.785302+0000) 2024-02-19T17:27:36.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:36.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:36.785468+0000) 2024-02-19T17:27:37.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:37.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:37.785686+0000) 2024-02-19T17:27:38.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:38.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:38.785876+0000) 2024-02-19T17:27:39.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:39.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:39.786101+0000) 2024-02-19T17:27:40.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:40.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:40.786274+0000) 2024-02-19T17:27:41.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:41.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:41.786525+0000) 2024-02-19T17:27:42.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:42.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:42.786814+0000) 2024-02-19T17:27:43.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:43.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:43.786953+0000) 2024-02-19T17:27:44.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:44.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:44.787150+0000) 2024-02-19T17:27:45.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:45.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:45.787302+0000) 2024-02-19T17:27:46.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:46.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:46.787515+0000) 2024-02-19T17:27:47.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:47.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:47.787775+0000) 2024-02-19T17:27:48.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:48.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:48.788017+0000) 2024-02-19T17:27:49.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:49.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:49.788214+0000) 2024-02-19T17:27:50.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:50.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:50.788472+0000) 2024-02-19T17:27:51.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:51.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:51.788703+0000) 2024-02-19T17:27:52.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:52.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:52.788877+0000) 2024-02-19T17:27:53.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:53.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:53.789078+0000) 2024-02-19T17:27:54.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:54.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:54.789342+0000) 2024-02-19T17:27:55.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:55.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:55.789645+0000) 2024-02-19T17:27:56.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:56.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:56.789923+0000) 2024-02-19T17:27:57.316 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:27:57.320 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:27:57.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:57.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:57.790173+0000) 2024-02-19T17:27:58.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:58.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:58.790412+0000) 2024-02-19T17:27:59.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:27:59.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:27:59.790674+0000) 2024-02-19T17:28:00.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:00.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:00.790954+0000) 2024-02-19T17:28:01.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:01.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:01.791218+0000) 2024-02-19T17:28:02.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:02.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:02.791433+0000) 2024-02-19T17:28:03.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:03.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:03.791714+0000) 2024-02-19T17:28:04.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:04.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:04.792002+0000) 2024-02-19T17:28:05.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:05.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:05.792307+0000) 2024-02-19T17:28:06.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:06.792+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:06.792503+0000) 2024-02-19T17:28:07.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:07.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:07.792774+0000) 2024-02-19T17:28:08.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:08.792+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:08.793079+0000) 2024-02-19T17:28:09.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:09.792+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:09.793254+0000) 2024-02-19T17:28:10.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:10.792+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:10.793465+0000) 2024-02-19T17:28:11.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:11.792+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:11.793728+0000) 2024-02-19T17:28:12.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:12.792+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:12.793956+0000) 2024-02-19T17:28:13.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:13.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:13.794152+0000) 2024-02-19T17:28:14.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:14.792+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:14.794403+0000) 2024-02-19T17:28:15.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:15.792+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:15.794700+0000) 2024-02-19T17:28:16.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:16.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:16.794931+0000) 2024-02-19T17:28:17.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:17.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:17.795149+0000) 2024-02-19T17:28:18.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:18.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:18.795422+0000) 2024-02-19T17:28:19.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:19.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:19.795644+0000) 2024-02-19T17:28:20.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:20.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:20.795919+0000) 2024-02-19T17:28:21.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:21.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:21.796215+0000) 2024-02-19T17:28:22.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:22.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:22.796410+0000) 2024-02-19T17:28:23.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:23.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:23.796674+0000) 2024-02-19T17:28:24.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:24.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:24.796986+0000) 2024-02-19T17:28:25.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:25.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:25.797128+0000) 2024-02-19T17:28:26.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:26.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:26.797394+0000) 2024-02-19T17:28:27.372 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:28:27.375 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:28:27.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:27.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:27.797640+0000) 2024-02-19T17:28:28.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:28.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:28.797852+0000) 2024-02-19T17:28:29.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:29.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:29.798127+0000) 2024-02-19T17:28:30.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:30.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:30.798402+0000) 2024-02-19T17:28:31.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:31.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:31.798694+0000) 2024-02-19T17:28:32.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:32.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:32.799050+0000) 2024-02-19T17:28:33.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:33.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:33.799358+0000) 2024-02-19T17:28:34.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:34.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:34.799612+0000) 2024-02-19T17:28:35.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:35.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:35.799822+0000) 2024-02-19T17:28:36.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:36.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:36.800088+0000) 2024-02-19T17:28:37.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:37.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:37.800321+0000) 2024-02-19T17:28:38.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:38.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:38.800610+0000) 2024-02-19T17:28:39.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:39.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:39.800912+0000) 2024-02-19T17:28:40.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:40.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:40.801135+0000) 2024-02-19T17:28:41.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:41.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:41.801308+0000) 2024-02-19T17:28:42.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:42.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:42.801583+0000) 2024-02-19T17:28:43.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:43.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:43.801923+0000) 2024-02-19T17:28:44.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:44.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:44.802169+0000) 2024-02-19T17:28:45.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:45.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:45.802454+0000) 2024-02-19T17:28:46.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:46.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:46.802756+0000) 2024-02-19T17:28:47.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:47.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:47.803040+0000) 2024-02-19T17:28:48.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:48.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:48.803201+0000) 2024-02-19T17:28:49.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:49.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:49.803450+0000) 2024-02-19T17:28:50.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:50.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:50.803745+0000) 2024-02-19T17:28:51.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:51.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:51.803992+0000) 2024-02-19T17:28:52.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:52.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:52.804190+0000) 2024-02-19T17:28:53.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:53.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:53.804459+0000) 2024-02-19T17:28:54.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:54.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:54.804711+0000) 2024-02-19T17:28:55.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:55.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:55.804987+0000) 2024-02-19T17:28:56.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:56.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:56.805230+0000) 2024-02-19T17:28:57.486 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:28:57.490 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:28:57.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:57.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:57.805530+0000) 2024-02-19T17:28:58.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:58.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:58.805747+0000) 2024-02-19T17:28:59.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:28:59.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:28:59.805984+0000) 2024-02-19T17:29:00.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:00.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:00.806246+0000) 2024-02-19T17:29:01.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:01.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:01.806443+0000) 2024-02-19T17:29:02.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:02.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:02.806753+0000) 2024-02-19T17:29:03.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:03.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:03.807020+0000) 2024-02-19T17:29:04.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:04.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:04.807200+0000) 2024-02-19T17:29:05.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:05.807+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:05.807465+0000) 2024-02-19T17:29:06.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:06.807+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:06.807707+0000) 2024-02-19T17:29:07.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:07.807+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:07.807905+0000) 2024-02-19T17:29:08.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:08.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:08.808171+0000) 2024-02-19T17:29:09.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:09.807+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:09.808401+0000) 2024-02-19T17:29:10.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:10.807+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:10.808594+0000) 2024-02-19T17:29:11.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:11.807+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:11.808862+0000) 2024-02-19T17:29:12.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:12.807+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:12.809081+0000) 2024-02-19T17:29:13.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:13.807+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:13.809316+0000) 2024-02-19T17:29:14.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:14.807+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:14.809569+0000) 2024-02-19T17:29:15.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:15.807+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:15.809755+0000) 2024-02-19T17:29:16.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:16.807+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:16.809966+0000) 2024-02-19T17:29:17.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:17.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:17.810213+0000) 2024-02-19T17:29:18.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:18.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:18.810463+0000) 2024-02-19T17:29:19.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:19.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:19.810822+0000) 2024-02-19T17:29:20.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:20.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:20.811097+0000) 2024-02-19T17:29:21.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:21.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:21.811301+0000) 2024-02-19T17:29:22.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:22.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:22.811533+0000) 2024-02-19T17:29:23.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:23.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:23.811859+0000) 2024-02-19T17:29:24.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:24.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:24.812151+0000) 2024-02-19T17:29:25.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:25.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:25.812452+0000) 2024-02-19T17:29:26.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:26.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:26.812681+0000) 2024-02-19T17:29:27.607 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:29:27.611 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:29:27.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:27.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:27.812956+0000) 2024-02-19T17:29:28.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:28.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:28.813235+0000) 2024-02-19T17:29:29.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:29.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:29.813536+0000) 2024-02-19T17:29:30.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:30.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:30.813786+0000) 2024-02-19T17:29:31.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:31.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:31.814076+0000) 2024-02-19T17:29:32.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:32.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:32.814375+0000) 2024-02-19T17:29:33.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:33.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:33.814653+0000) 2024-02-19T17:29:34.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:34.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:34.814855+0000) 2024-02-19T17:29:35.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:35.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:35.815093+0000) 2024-02-19T17:29:36.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:36.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:36.815306+0000) 2024-02-19T17:29:37.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:37.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:37.815603+0000) 2024-02-19T17:29:38.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:38.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:38.815900+0000) 2024-02-19T17:29:39.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:39.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:39.816190+0000) 2024-02-19T17:29:40.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:40.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:40.816440+0000) 2024-02-19T17:29:41.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:41.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:41.816708+0000) 2024-02-19T17:29:42.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:42.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:42.816960+0000) 2024-02-19T17:29:43.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:43.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:43.817164+0000) 2024-02-19T17:29:44.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:44.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:44.817417+0000) 2024-02-19T17:29:45.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:45.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:45.817713+0000) 2024-02-19T17:29:46.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:46.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:46.818006+0000) 2024-02-19T17:29:47.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:47.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:47.818340+0000) 2024-02-19T17:29:48.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:48.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:48.818535+0000) 2024-02-19T17:29:49.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:49.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:49.818843+0000) 2024-02-19T17:29:50.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:50.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:50.819036+0000) 2024-02-19T17:29:51.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:51.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:51.819258+0000) 2024-02-19T17:29:52.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:52.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:52.819524+0000) 2024-02-19T17:29:53.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:53.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:53.819791+0000) 2024-02-19T17:29:54.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:54.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:54.820013+0000) 2024-02-19T17:29:55.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:55.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:55.820217+0000) 2024-02-19T17:29:56.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:56.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:56.820477+0000) 2024-02-19T17:29:57.730 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:29:57.734 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:29:57.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:57.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:57.820706+0000) 2024-02-19T17:29:58.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:58.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:58.820993+0000) 2024-02-19T17:29:59.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:29:59.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:29:59.821282+0000) 2024-02-19T17:30:00.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:00.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:00.821489+0000) 2024-02-19T17:30:01.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:01.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:01.821783+0000) 2024-02-19T17:30:02.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:02.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:02.821979+0000) 2024-02-19T17:30:03.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:03.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:03.822250+0000) 2024-02-19T17:30:04.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:04.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:04.822530+0000) 2024-02-19T17:30:05.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:05.822+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:05.822890+0000) 2024-02-19T17:30:06.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:06.822+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:06.823148+0000) 2024-02-19T17:30:07.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:07.822+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:07.823356+0000) 2024-02-19T17:30:08.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:08.822+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:08.823636+0000) 2024-02-19T17:30:09.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:09.822+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:09.823875+0000) 2024-02-19T17:30:10.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:10.822+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:10.824142+0000) 2024-02-19T17:30:11.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:11.822+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:11.824389+0000) 2024-02-19T17:30:12.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:12.822+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:12.824520+0000) 2024-02-19T17:30:13.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:13.822+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:13.824713+0000) 2024-02-19T17:30:14.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:14.822+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:14.824893+0000) 2024-02-19T17:30:15.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:15.822+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:15.825138+0000) 2024-02-19T17:30:16.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:16.822+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:16.825378+0000) 2024-02-19T17:30:17.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:17.822+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:17.825617+0000) 2024-02-19T17:30:18.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:18.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:18.825832+0000) 2024-02-19T17:30:19.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:19.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:19.826097+0000) 2024-02-19T17:30:20.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:20.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:20.826334+0000) 2024-02-19T17:30:21.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:21.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:21.826579+0000) 2024-02-19T17:30:22.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:22.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:22.826866+0000) 2024-02-19T17:30:23.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:23.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:23.827056+0000) 2024-02-19T17:30:24.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:24.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:24.827245+0000) 2024-02-19T17:30:25.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:25.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:25.827471+0000) 2024-02-19T17:30:26.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:26.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:26.827642+0000) 2024-02-19T17:30:27.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:27.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:27.827849+0000) 2024-02-19T17:30:27.852 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:30:27.855 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:30:28.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:28.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:28.828071+0000) 2024-02-19T17:30:29.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:29.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:29.828336+0000) 2024-02-19T17:30:30.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:30.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:30.828561+0000) 2024-02-19T17:30:31.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:31.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:31.828732+0000) 2024-02-19T17:30:32.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:32.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:32.828991+0000) 2024-02-19T17:30:33.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:33.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:33.829258+0000) 2024-02-19T17:30:34.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:34.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:34.829482+0000) 2024-02-19T17:30:35.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:35.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:35.829737+0000) 2024-02-19T17:30:36.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:36.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:36.829970+0000) 2024-02-19T17:30:37.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:37.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:37.830179+0000) 2024-02-19T17:30:38.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:38.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:38.830414+0000) 2024-02-19T17:30:39.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:39.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:39.830621+0000) 2024-02-19T17:30:40.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:40.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:40.830860+0000) 2024-02-19T17:30:41.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:41.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:41.831104+0000) 2024-02-19T17:30:42.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:42.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:42.831366+0000) 2024-02-19T17:30:43.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:43.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:43.831631+0000) 2024-02-19T17:30:44.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:44.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:44.831820+0000) 2024-02-19T17:30:45.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:45.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:45.832064+0000) 2024-02-19T17:30:46.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:46.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:46.832199+0000) 2024-02-19T17:30:47.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:47.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:47.832444+0000) 2024-02-19T17:30:48.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:48.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:48.832620+0000) 2024-02-19T17:30:49.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:49.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:49.832868+0000) 2024-02-19T17:30:50.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:50.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:50.833117+0000) 2024-02-19T17:30:51.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:51.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:51.833318+0000) 2024-02-19T17:30:52.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:52.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:52.833526+0000) 2024-02-19T17:30:53.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:53.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:53.833753+0000) 2024-02-19T17:30:54.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:54.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:54.833941+0000) 2024-02-19T17:30:55.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:55.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:55.834187+0000) 2024-02-19T17:30:56.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:56.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:56.834335+0000) 2024-02-19T17:30:57.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:57.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:57.834587+0000) 2024-02-19T17:30:57.939 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:30:57.943 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:30:58.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:58.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:58.834879+0000) 2024-02-19T17:30:59.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:30:59.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:30:59.835182+0000) 2024-02-19T17:31:00.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:00.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:00.835359+0000) 2024-02-19T17:31:01.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:01.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:01.835609+0000) 2024-02-19T17:31:02.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:02.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:02.835862+0000) 2024-02-19T17:31:03.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:03.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:03.836096+0000) 2024-02-19T17:31:04.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:04.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:04.836331+0000) 2024-02-19T17:31:05.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:05.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:05.836570+0000) 2024-02-19T17:31:06.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:06.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:06.836830+0000) 2024-02-19T17:31:07.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:07.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:07.837075+0000) 2024-02-19T17:31:08.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:08.837+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:08.837278+0000) 2024-02-19T17:31:09.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:09.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:09.837519+0000) 2024-02-19T17:31:10.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:10.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:10.837750+0000) 2024-02-19T17:31:11.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:11.837+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:11.837990+0000) 2024-02-19T17:31:12.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:12.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:12.838127+0000) 2024-02-19T17:31:13.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:13.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:13.838363+0000) 2024-02-19T17:31:14.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:14.837+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:14.838598+0000) 2024-02-19T17:31:15.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:15.837+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:15.838851+0000) 2024-02-19T17:31:16.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:16.837+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:16.839070+0000) 2024-02-19T17:31:17.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:17.837+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:17.839285+0000) 2024-02-19T17:31:18.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:18.837+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:18.839488+0000) 2024-02-19T17:31:19.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:19.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:19.839728+0000) 2024-02-19T17:31:20.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:20.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:20.839961+0000) 2024-02-19T17:31:21.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:21.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:21.840204+0000) 2024-02-19T17:31:22.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:22.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:22.840390+0000) 2024-02-19T17:31:23.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:23.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:23.840579+0000) 2024-02-19T17:31:24.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:24.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:24.840775+0000) 2024-02-19T17:31:25.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:25.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:25.841045+0000) 2024-02-19T17:31:26.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:26.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:26.841271+0000) 2024-02-19T17:31:27.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:27.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:27.841505+0000) 2024-02-19T17:31:28.063 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:31:28.067 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:31:28.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:28.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:28.841686+0000) 2024-02-19T17:31:29.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:29.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:29.841961+0000) 2024-02-19T17:31:30.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:30.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:30.842147+0000) 2024-02-19T17:31:31.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:31.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:31.842388+0000) 2024-02-19T17:31:32.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:32.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:32.842577+0000) 2024-02-19T17:31:33.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:33.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:33.842825+0000) 2024-02-19T17:31:34.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:34.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:34.843065+0000) 2024-02-19T17:31:35.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:35.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:35.843327+0000) 2024-02-19T17:31:36.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:36.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:36.843505+0000) 2024-02-19T17:31:37.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:37.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:37.843730+0000) 2024-02-19T17:31:38.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:38.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:38.843991+0000) 2024-02-19T17:31:39.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:39.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:39.844248+0000) 2024-02-19T17:31:40.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:40.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:40.844484+0000) 2024-02-19T17:31:41.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:41.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:41.844728+0000) 2024-02-19T17:31:42.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:42.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:42.845000+0000) 2024-02-19T17:31:43.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:43.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:43.845176+0000) 2024-02-19T17:31:44.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:44.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:44.845351+0000) 2024-02-19T17:31:45.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:45.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:45.845620+0000) 2024-02-19T17:31:46.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:46.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:46.845894+0000) 2024-02-19T17:31:47.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:47.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:47.846088+0000) 2024-02-19T17:31:48.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:48.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:48.846309+0000) 2024-02-19T17:31:49.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:49.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:49.846511+0000) 2024-02-19T17:31:50.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:50.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:50.846791+0000) 2024-02-19T17:31:51.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:51.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:51.847016+0000) 2024-02-19T17:31:52.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:52.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:52.847358+0000) 2024-02-19T17:31:53.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:53.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:53.847631+0000) 2024-02-19T17:31:54.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:54.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:54.847869+0000) 2024-02-19T17:31:55.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:55.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:55.848100+0000) 2024-02-19T17:31:56.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:56.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:56.848324+0000) 2024-02-19T17:31:57.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:57.848+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:57.848550+0000) 2024-02-19T17:31:58.175 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:31:58.178 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:31:58.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:58.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:58.848798+0000) 2024-02-19T17:31:59.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:31:59.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:31:59.849013+0000) 2024-02-19T17:32:00.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:00.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:00.849175+0000) 2024-02-19T17:32:01.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:01.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:01.849341+0000) 2024-02-19T17:32:02.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:02.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:02.849589+0000) 2024-02-19T17:32:03.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:03.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:03.849848+0000) 2024-02-19T17:32:04.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:04.848+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:04.850101+0000) 2024-02-19T17:32:05.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:05.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:05.850314+0000) 2024-02-19T17:32:06.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:06.848+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:06.850574+0000) 2024-02-19T17:32:07.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:07.848+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:07.850842+0000) 2024-02-19T17:32:08.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:08.848+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:08.851101+0000) 2024-02-19T17:32:09.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:09.848+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:09.851298+0000) 2024-02-19T17:32:10.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:10.848+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:10.851558+0000) 2024-02-19T17:32:11.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:11.848+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:11.851831+0000) 2024-02-19T17:32:12.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:12.848+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:12.852081+0000) 2024-02-19T17:32:13.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:13.852+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:13.852281+0000) 2024-02-19T17:32:14.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:14.852+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:14.852516+0000) 2024-02-19T17:32:15.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:15.848+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:15.852808+0000) 2024-02-19T17:32:16.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:16.848+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:16.853069+0000) 2024-02-19T17:32:17.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:17.848+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:17.853246+0000) 2024-02-19T17:32:18.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:18.852+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:18.853528+0000) 2024-02-19T17:32:19.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:19.852+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:19.853779+0000) 2024-02-19T17:32:20.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:20.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:20.854019+0000) 2024-02-19T17:32:21.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:21.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:21.854263+0000) 2024-02-19T17:32:22.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:22.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:22.854496+0000) 2024-02-19T17:32:23.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:23.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:23.854789+0000) 2024-02-19T17:32:24.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:24.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:24.855014+0000) 2024-02-19T17:32:25.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:25.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:25.855265+0000) 2024-02-19T17:32:26.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:26.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:26.855500+0000) 2024-02-19T17:32:27.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:27.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:27.855759+0000) 2024-02-19T17:32:28.290 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:32:28.293 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:32:28.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:28.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:28.856010+0000) 2024-02-19T17:32:29.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:29.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:29.856278+0000) 2024-02-19T17:32:30.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:30.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:30.856440+0000) 2024-02-19T17:32:31.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:31.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:31.856669+0000) 2024-02-19T17:32:32.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:32.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:32.856897+0000) 2024-02-19T17:32:33.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:33.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:33.857118+0000) 2024-02-19T17:32:34.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:34.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:34.857357+0000) 2024-02-19T17:32:35.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:35.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:35.857637+0000) 2024-02-19T17:32:36.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:36.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:36.857817+0000) 2024-02-19T17:32:37.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:37.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:37.858049+0000) 2024-02-19T17:32:38.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:38.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:38.858298+0000) 2024-02-19T17:32:39.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:39.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:39.858562+0000) 2024-02-19T17:32:40.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:40.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:40.858776+0000) 2024-02-19T17:32:41.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:41.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:41.858998+0000) 2024-02-19T17:32:42.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:42.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:42.859218+0000) 2024-02-19T17:32:43.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:43.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:43.859448+0000) 2024-02-19T17:32:44.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:44.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:44.859634+0000) 2024-02-19T17:32:45.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:45.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:45.859855+0000) 2024-02-19T17:32:46.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:46.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:46.860018+0000) 2024-02-19T17:32:47.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:47.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:47.860186+0000) 2024-02-19T17:32:48.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:48.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:48.860345+0000) 2024-02-19T17:32:49.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:49.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:49.860545+0000) 2024-02-19T17:32:50.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:50.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:50.860779+0000) 2024-02-19T17:32:51.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:51.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:51.861025+0000) 2024-02-19T17:32:52.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:52.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:52.861193+0000) 2024-02-19T17:32:53.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:53.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:53.861471+0000) 2024-02-19T17:32:54.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:54.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:54.861650+0000) 2024-02-19T17:32:55.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:55.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:55.861890+0000) 2024-02-19T17:32:56.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:56.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:56.862125+0000) 2024-02-19T17:32:57.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:57.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:57.862367+0000) 2024-02-19T17:32:58.371 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:32:58.374 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:32:58.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:58.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:58.862568+0000) 2024-02-19T17:32:59.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:32:59.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:32:59.862844+0000) 2024-02-19T17:33:00.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:00.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:00.863043+0000) 2024-02-19T17:33:01.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:01.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:01.863226+0000) 2024-02-19T17:33:02.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:02.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:02.863447+0000) 2024-02-19T17:33:03.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:03.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:03.863631+0000) 2024-02-19T17:33:04.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:04.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:04.863859+0000) 2024-02-19T17:33:05.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:05.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:05.864120+0000) 2024-02-19T17:33:06.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:06.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:06.864424+0000) 2024-02-19T17:33:07.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:07.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:07.864612+0000) 2024-02-19T17:33:08.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:08.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:08.864885+0000) 2024-02-19T17:33:09.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:09.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:09.865163+0000) 2024-02-19T17:33:10.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:10.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:10.865416+0000) 2024-02-19T17:33:11.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:11.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:11.865695+0000) 2024-02-19T17:33:12.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:12.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:12.865899+0000) 2024-02-19T17:33:13.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:13.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:13.866150+0000) 2024-02-19T17:33:14.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:14.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:14.866416+0000) 2024-02-19T17:33:15.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:15.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:15.866639+0000) 2024-02-19T17:33:16.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:16.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:16.866874+0000) 2024-02-19T17:33:17.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:17.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:17.867123+0000) 2024-02-19T17:33:18.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:18.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:18.867358+0000) 2024-02-19T17:33:19.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:19.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:19.867584+0000) 2024-02-19T17:33:20.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:20.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:20.867777+0000) 2024-02-19T17:33:21.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:21.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:21.867959+0000) 2024-02-19T17:33:22.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:22.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:22.868203+0000) 2024-02-19T17:33:23.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:23.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:23.868458+0000) 2024-02-19T17:33:24.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:24.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:24.868707+0000) 2024-02-19T17:33:25.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:25.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:25.868904+0000) 2024-02-19T17:33:26.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:26.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:26.869133+0000) 2024-02-19T17:33:27.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:27.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:27.869382+0000) 2024-02-19T17:33:28.431 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:33:28.435 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:33:28.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:28.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:28.869634+0000) 2024-02-19T17:33:29.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:29.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:29.869854+0000) 2024-02-19T17:33:30.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:30.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:30.870091+0000) 2024-02-19T17:33:31.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:31.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:31.870321+0000) 2024-02-19T17:33:32.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:32.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:32.870550+0000) 2024-02-19T17:33:33.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:33.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:33.870828+0000) 2024-02-19T17:33:34.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:34.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:34.871010+0000) 2024-02-19T17:33:35.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:35.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:35.871263+0000) 2024-02-19T17:33:36.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:36.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:36.871432+0000) 2024-02-19T17:33:37.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:37.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:37.871700+0000) 2024-02-19T17:33:38.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:38.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:38.871864+0000) 2024-02-19T17:33:39.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:39.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:39.872146+0000) 2024-02-19T17:33:40.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:40.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:40.872322+0000) 2024-02-19T17:33:41.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:41.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:41.872584+0000) 2024-02-19T17:33:42.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:42.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:42.872802+0000) 2024-02-19T17:33:43.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:43.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:43.873050+0000) 2024-02-19T17:33:44.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:44.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:44.873297+0000) 2024-02-19T17:33:45.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:45.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:45.873569+0000) 2024-02-19T17:33:46.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:46.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:46.873817+0000) 2024-02-19T17:33:47.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:47.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:47.874052+0000) 2024-02-19T17:33:48.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:48.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:48.874225+0000) 2024-02-19T17:33:49.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:49.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:49.874452+0000) 2024-02-19T17:33:50.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:50.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:50.874625+0000) 2024-02-19T17:33:51.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:51.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:51.874914+0000) 2024-02-19T17:33:52.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:52.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:52.875146+0000) 2024-02-19T17:33:53.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:53.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:53.875400+0000) 2024-02-19T17:33:54.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:54.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:54.875572+0000) 2024-02-19T17:33:55.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:55.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:55.875822+0000) 2024-02-19T17:33:56.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:56.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:56.876042+0000) 2024-02-19T17:33:57.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:57.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:57.876285+0000) 2024-02-19T17:33:58.547 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:33:58.550 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:33:58.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:58.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:58.876481+0000) 2024-02-19T17:33:59.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:33:59.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:33:59.876731+0000) 2024-02-19T17:34:00.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:00.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:00.876973+0000) 2024-02-19T17:34:01.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:01.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:01.877216+0000) 2024-02-19T17:34:02.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:02.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:02.877415+0000) 2024-02-19T17:34:03.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:03.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:03.877667+0000) 2024-02-19T17:34:04.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:04.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:04.877866+0000) 2024-02-19T17:34:05.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:05.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:05.878092+0000) 2024-02-19T17:34:06.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:06.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:06.878326+0000) 2024-02-19T17:34:07.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:07.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:07.878577+0000) 2024-02-19T17:34:08.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:08.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:08.878823+0000) 2024-02-19T17:34:09.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:09.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:09.878970+0000) 2024-02-19T17:34:10.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:10.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:10.879085+0000) 2024-02-19T17:34:11.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:11.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:11.879287+0000) 2024-02-19T17:34:12.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:12.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:12.879525+0000) 2024-02-19T17:34:13.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:13.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:13.879797+0000) 2024-02-19T17:34:14.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:14.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:14.879966+0000) 2024-02-19T17:34:15.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:15.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:15.880220+0000) 2024-02-19T17:34:16.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:16.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:16.880437+0000) 2024-02-19T17:34:17.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:17.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:17.880683+0000) 2024-02-19T17:34:18.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:18.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:18.880862+0000) 2024-02-19T17:34:19.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:19.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:19.881113+0000) 2024-02-19T17:34:20.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:20.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:20.881309+0000) 2024-02-19T17:34:21.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:21.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:21.881554+0000) 2024-02-19T17:34:22.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:22.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:22.881749+0000) 2024-02-19T17:34:23.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:23.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:23.881994+0000) 2024-02-19T17:34:24.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:24.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:24.882249+0000) 2024-02-19T17:34:25.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:25.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:25.882510+0000) 2024-02-19T17:34:26.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:26.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:26.882693+0000) 2024-02-19T17:34:27.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:27.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:27.882941+0000) 2024-02-19T17:34:28.668 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:34:28.672 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:34:28.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:28.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:28.883146+0000) 2024-02-19T17:34:29.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:29.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:29.883398+0000) 2024-02-19T17:34:30.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:30.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:30.883593+0000) 2024-02-19T17:34:31.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:31.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:31.883856+0000) 2024-02-19T17:34:32.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:32.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:32.884100+0000) 2024-02-19T17:34:33.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:33.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:33.884421+0000) 2024-02-19T17:34:34.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:34.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:34.884700+0000) 2024-02-19T17:34:35.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:35.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:35.884923+0000) 2024-02-19T17:34:36.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:36.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:36.885080+0000) 2024-02-19T17:34:37.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:37.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:37.885328+0000) 2024-02-19T17:34:38.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:38.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:38.885465+0000) 2024-02-19T17:34:39.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:39.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:39.885703+0000) 2024-02-19T17:34:40.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:40.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:40.885877+0000) 2024-02-19T17:34:41.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:41.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:41.886038+0000) 2024-02-19T17:34:42.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:42.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:42.886184+0000) 2024-02-19T17:34:43.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:43.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:43.886405+0000) 2024-02-19T17:34:44.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:44.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:44.886598+0000) 2024-02-19T17:34:45.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:45.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:45.886862+0000) 2024-02-19T17:34:46.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:46.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:46.887118+0000) 2024-02-19T17:34:47.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:47.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:47.887262+0000) 2024-02-19T17:34:48.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:48.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:48.887518+0000) 2024-02-19T17:34:49.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:49.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:49.887780+0000) 2024-02-19T17:34:50.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:50.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:50.888023+0000) 2024-02-19T17:34:51.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:51.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:51.888275+0000) 2024-02-19T17:34:52.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:52.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:52.888440+0000) 2024-02-19T17:34:53.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:53.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:53.888678+0000) 2024-02-19T17:34:54.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:54.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:54.888900+0000) 2024-02-19T17:34:55.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:55.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:55.889178+0000) 2024-02-19T17:34:56.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:56.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:56.889401+0000) 2024-02-19T17:34:57.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:57.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:57.889547+0000) 2024-02-19T17:34:58.808 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:34:58.812 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:34:58.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:58.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:58.889772+0000) 2024-02-19T17:34:59.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:34:59.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:34:59.890038+0000) 2024-02-19T17:35:00.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:00.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:00.890262+0000) 2024-02-19T17:35:01.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:01.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:01.890563+0000) 2024-02-19T17:35:02.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:02.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:02.890779+0000) 2024-02-19T17:35:03.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:03.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:03.890943+0000) 2024-02-19T17:35:04.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:04.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:04.891107+0000) 2024-02-19T17:35:05.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:05.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:05.891255+0000) 2024-02-19T17:35:06.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:06.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:06.891465+0000) 2024-02-19T17:35:07.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:07.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:07.891711+0000) 2024-02-19T17:35:08.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:08.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:08.891948+0000) 2024-02-19T17:35:09.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:09.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:09.892246+0000) 2024-02-19T17:35:10.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:10.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:10.892480+0000) 2024-02-19T17:35:11.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:11.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:11.892817+0000) 2024-02-19T17:35:12.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:12.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:12.892974+0000) 2024-02-19T17:35:13.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:13.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:13.893236+0000) 2024-02-19T17:35:14.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:14.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:14.893443+0000) 2024-02-19T17:35:15.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:15.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:15.893752+0000) 2024-02-19T17:35:16.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:16.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:16.893979+0000) 2024-02-19T17:35:17.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:17.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:17.894219+0000) 2024-02-19T17:35:18.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:18.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:18.894350+0000) 2024-02-19T17:35:19.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:19.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:19.894606+0000) 2024-02-19T17:35:20.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:20.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:20.894896+0000) 2024-02-19T17:35:21.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:21.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:21.895074+0000) 2024-02-19T17:35:22.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:22.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:22.895207+0000) 2024-02-19T17:35:23.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:23.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:23.895467+0000) 2024-02-19T17:35:24.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:24.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:24.895718+0000) 2024-02-19T17:35:25.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:25.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:25.895971+0000) 2024-02-19T17:35:26.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:26.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:26.896203+0000) 2024-02-19T17:35:27.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:27.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:27.896445+0000) 2024-02-19T17:35:28.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:28.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:28.896708+0000) 2024-02-19T17:35:28.955 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:35:28.958 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:35:29.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:29.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:29.896921+0000) 2024-02-19T17:35:30.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:30.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:30.897158+0000) 2024-02-19T17:35:31.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:31.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:31.897333+0000) 2024-02-19T17:35:32.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:32.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:32.897541+0000) 2024-02-19T17:35:33.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:33.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:33.897726+0000) 2024-02-19T17:35:34.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:34.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:34.897882+0000) 2024-02-19T17:35:35.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:35.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:35.898122+0000) 2024-02-19T17:35:36.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:36.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:36.898315+0000) 2024-02-19T17:35:37.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:37.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:37.898593+0000) 2024-02-19T17:35:38.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:38.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:38.898807+0000) 2024-02-19T17:35:39.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:39.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:39.898986+0000) 2024-02-19T17:35:40.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:40.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:40.899162+0000) 2024-02-19T17:35:41.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:41.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:41.899347+0000) 2024-02-19T17:35:42.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:42.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:42.899521+0000) 2024-02-19T17:35:43.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:43.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:43.899806+0000) 2024-02-19T17:35:44.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:44.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:44.900038+0000) 2024-02-19T17:35:45.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:45.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:45.900219+0000) 2024-02-19T17:35:46.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:46.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:46.900411+0000) 2024-02-19T17:35:47.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:47.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:47.900651+0000) 2024-02-19T17:35:48.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:48.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:48.900864+0000) 2024-02-19T17:35:49.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:49.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:49.901044+0000) 2024-02-19T17:35:50.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:50.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:50.901291+0000) 2024-02-19T17:35:51.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:51.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:51.901462+0000) 2024-02-19T17:35:52.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:52.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:52.901696+0000) 2024-02-19T17:35:53.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:53.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:53.901941+0000) 2024-02-19T17:35:54.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:54.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:54.902204+0000) 2024-02-19T17:35:55.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:55.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:55.902483+0000) 2024-02-19T17:35:56.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:56.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:56.902743+0000) 2024-02-19T17:35:57.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:57.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:57.902886+0000) 2024-02-19T17:35:58.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:58.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:58.903121+0000) 2024-02-19T17:35:59.078 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:35:59.082 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:35:59.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:35:59.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:35:59.903317+0000) 2024-02-19T17:36:00.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:00.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:00.903543+0000) 2024-02-19T17:36:01.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:01.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:01.903791+0000) 2024-02-19T17:36:02.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:02.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:02.904052+0000) 2024-02-19T17:36:03.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:03.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:03.904309+0000) 2024-02-19T17:36:04.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:04.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:04.904527+0000) 2024-02-19T17:36:05.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:05.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:05.904782+0000) 2024-02-19T17:36:06.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:06.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:06.905000+0000) 2024-02-19T17:36:07.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:07.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:07.905264+0000) 2024-02-19T17:36:08.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:08.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:08.905517+0000) 2024-02-19T17:36:09.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:09.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:09.905722+0000) 2024-02-19T17:36:10.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:10.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:10.905965+0000) 2024-02-19T17:36:11.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:11.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:11.906238+0000) 2024-02-19T17:36:12.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:12.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:12.906393+0000) 2024-02-19T17:36:13.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:13.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:13.906697+0000) 2024-02-19T17:36:14.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:14.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:14.906928+0000) 2024-02-19T17:36:15.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:15.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:15.907125+0000) 2024-02-19T17:36:16.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:16.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:16.907373+0000) 2024-02-19T17:36:17.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:17.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:17.907611+0000) 2024-02-19T17:36:18.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:18.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:18.907804+0000) 2024-02-19T17:36:19.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:19.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:19.907968+0000) 2024-02-19T17:36:20.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:20.908+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:20.908207+0000) 2024-02-19T17:36:21.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:21.908+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:21.908458+0000) 2024-02-19T17:36:22.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:22.908+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:22.908684+0000) 2024-02-19T17:36:23.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:23.908+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:23.908968+0000) 2024-02-19T17:36:24.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:24.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:24.909199+0000) 2024-02-19T17:36:25.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:25.908+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:25.909411+0000) 2024-02-19T17:36:26.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:26.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:26.909586+0000) 2024-02-19T17:36:27.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:27.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:27.909823+0000) 2024-02-19T17:36:28.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:28.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:28.910064+0000) 2024-02-19T17:36:29.192 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:36:29.195 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:36:29.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:29.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:29.910243+0000) 2024-02-19T17:36:30.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:30.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:30.910390+0000) 2024-02-19T17:36:31.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:31.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:31.910649+0000) 2024-02-19T17:36:32.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:32.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:32.910894+0000) 2024-02-19T17:36:33.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:33.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:33.911138+0000) 2024-02-19T17:36:34.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:34.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:34.911403+0000) 2024-02-19T17:36:35.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:35.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:35.911673+0000) 2024-02-19T17:36:36.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:36.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:36.911897+0000) 2024-02-19T17:36:37.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:37.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:37.912113+0000) 2024-02-19T17:36:38.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:38.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:38.912302+0000) 2024-02-19T17:36:39.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:39.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:39.912502+0000) 2024-02-19T17:36:40.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:40.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:40.912680+0000) 2024-02-19T17:36:41.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:41.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:41.912956+0000) 2024-02-19T17:36:42.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:42.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:42.913165+0000) 2024-02-19T17:36:43.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:43.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:43.913345+0000) 2024-02-19T17:36:44.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:44.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:44.913581+0000) 2024-02-19T17:36:45.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:45.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:45.913839+0000) 2024-02-19T17:36:46.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:46.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:46.914029+0000) 2024-02-19T17:36:47.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:47.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:47.914237+0000) 2024-02-19T17:36:48.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:48.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:48.914502+0000) 2024-02-19T17:36:49.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:49.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:49.914738+0000) 2024-02-19T17:36:50.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:50.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:50.914937+0000) 2024-02-19T17:36:51.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:51.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:51.915115+0000) 2024-02-19T17:36:52.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:52.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:52.915356+0000) 2024-02-19T17:36:53.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:53.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:53.915601+0000) 2024-02-19T17:36:54.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:54.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:54.915853+0000) 2024-02-19T17:36:55.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:55.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:55.916104+0000) 2024-02-19T17:36:56.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:56.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:56.916340+0000) 2024-02-19T17:36:57.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:57.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:57.916630+0000) 2024-02-19T17:36:58.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:58.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:58.916885+0000) 2024-02-19T17:36:59.314 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:36:59.317 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:36:59.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:36:59.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:36:59.917139+0000) 2024-02-19T17:37:00.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:00.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:00.917377+0000) 2024-02-19T17:37:01.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:01.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:01.917683+0000) 2024-02-19T17:37:02.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:02.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:02.917925+0000) 2024-02-19T17:37:03.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:03.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:03.918169+0000) 2024-02-19T17:37:04.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:04.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:04.918417+0000) 2024-02-19T17:37:05.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:05.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:05.918673+0000) 2024-02-19T17:37:06.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:06.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:06.918929+0000) 2024-02-19T17:37:07.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:07.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:07.919169+0000) 2024-02-19T17:37:08.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:08.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:08.919355+0000) 2024-02-19T17:37:09.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:09.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:09.919586+0000) 2024-02-19T17:37:10.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:10.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:10.919821+0000) 2024-02-19T17:37:11.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:11.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:11.920090+0000) 2024-02-19T17:37:12.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:12.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:12.920333+0000) 2024-02-19T17:37:13.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:13.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:13.920579+0000) 2024-02-19T17:37:14.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:14.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:14.920809+0000) 2024-02-19T17:37:15.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:15.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:15.921095+0000) 2024-02-19T17:37:16.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:16.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:16.921347+0000) 2024-02-19T17:37:17.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:17.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:17.921615+0000) 2024-02-19T17:37:18.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:18.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:18.921838+0000) 2024-02-19T17:37:19.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:19.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:19.922087+0000) 2024-02-19T17:37:20.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:20.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:20.922286+0000) 2024-02-19T17:37:21.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:21.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:21.922529+0000) 2024-02-19T17:37:22.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:22.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:22.922701+0000) 2024-02-19T17:37:23.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:23.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:23.922959+0000) 2024-02-19T17:37:24.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:24.923+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:24.923166+0000) 2024-02-19T17:37:25.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:25.923+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:25.923360+0000) 2024-02-19T17:37:26.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:26.923+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:26.923592+0000) 2024-02-19T17:37:27.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:27.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:27.923851+0000) 2024-02-19T17:37:28.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:28.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:28.924115+0000) 2024-02-19T17:37:29.435 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:37:29.439 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:37:29.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:29.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:29.924370+0000) 2024-02-19T17:37:30.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:30.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:30.924563+0000) 2024-02-19T17:37:31.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:31.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:31.924786+0000) 2024-02-19T17:37:32.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:32.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:32.925005+0000) 2024-02-19T17:37:33.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:33.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:33.925254+0000) 2024-02-19T17:37:34.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:34.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:34.925496+0000) 2024-02-19T17:37:35.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:35.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:35.925778+0000) 2024-02-19T17:37:36.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:36.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:36.925944+0000) 2024-02-19T17:37:37.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:37.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:37.926126+0000) 2024-02-19T17:37:38.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:38.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:38.926374+0000) 2024-02-19T17:37:39.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:39.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:39.926574+0000) 2024-02-19T17:37:40.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:40.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:40.926837+0000) 2024-02-19T17:37:41.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:41.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:41.927012+0000) 2024-02-19T17:37:42.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:42.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:42.927233+0000) 2024-02-19T17:37:43.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:43.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:43.927411+0000) 2024-02-19T17:37:44.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:44.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:44.927685+0000) 2024-02-19T17:37:45.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:45.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:45.927951+0000) 2024-02-19T17:37:46.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:46.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:46.928153+0000) 2024-02-19T17:37:47.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:47.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:47.928333+0000) 2024-02-19T17:37:48.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:48.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:48.928555+0000) 2024-02-19T17:37:49.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:49.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:49.928802+0000) 2024-02-19T17:37:50.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:50.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:50.929071+0000) 2024-02-19T17:37:51.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:51.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:51.929346+0000) 2024-02-19T17:37:52.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:52.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:52.929578+0000) 2024-02-19T17:37:53.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:53.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:53.929792+0000) 2024-02-19T17:37:54.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:54.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:54.929986+0000) 2024-02-19T17:37:55.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:55.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:55.930270+0000) 2024-02-19T17:37:56.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:56.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:56.930500+0000) 2024-02-19T17:37:57.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:57.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:57.930653+0000) 2024-02-19T17:37:58.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:58.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:58.930884+0000) 2024-02-19T17:37:59.516 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:37:59.519 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:37:59.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:37:59.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:37:59.931126+0000) 2024-02-19T17:38:00.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:00.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:00.931301+0000) 2024-02-19T17:38:01.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:01.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:01.931502+0000) 2024-02-19T17:38:02.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:02.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:02.931756+0000) 2024-02-19T17:38:03.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:03.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:03.932001+0000) 2024-02-19T17:38:04.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:04.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:04.932235+0000) 2024-02-19T17:38:05.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:05.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:05.932513+0000) 2024-02-19T17:38:06.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:06.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:06.932739+0000) 2024-02-19T17:38:07.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:07.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:07.932978+0000) 2024-02-19T17:38:08.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:08.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:08.933165+0000) 2024-02-19T17:38:09.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:09.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:09.933408+0000) 2024-02-19T17:38:10.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:10.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:10.933582+0000) 2024-02-19T17:38:11.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:11.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:11.933758+0000) 2024-02-19T17:38:12.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:12.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:12.934029+0000) 2024-02-19T17:38:13.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:13.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:13.934258+0000) 2024-02-19T17:38:14.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:14.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:14.934431+0000) 2024-02-19T17:38:15.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:15.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:15.934631+0000) 2024-02-19T17:38:16.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:16.934+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:16.934832+0000) 2024-02-19T17:38:17.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:17.934+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:17.935009+0000) 2024-02-19T17:38:18.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:18.934+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:18.935245+0000) 2024-02-19T17:38:19.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:19.934+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:19.935507+0000) 2024-02-19T17:38:20.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:20.934+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:20.935741+0000) 2024-02-19T17:38:21.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:21.934+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:21.935981+0000) 2024-02-19T17:38:22.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:22.934+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:22.936152+0000) 2024-02-19T17:38:23.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:23.934+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:23.936334+0000) 2024-02-19T17:38:24.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:24.934+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:24.936494+0000) 2024-02-19T17:38:25.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:25.934+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:25.936704+0000) 2024-02-19T17:38:26.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:26.934+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:26.936961+0000) 2024-02-19T17:38:27.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:27.934+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:27.937195+0000) 2024-02-19T17:38:28.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:28.934+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:28.937423+0000) 2024-02-19T17:38:29.600 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:38:29.604 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:38:29.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:29.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:29.937630+0000) 2024-02-19T17:38:30.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:30.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:30.937887+0000) 2024-02-19T17:38:31.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:31.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:31.938166+0000) 2024-02-19T17:38:32.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:32.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:32.938386+0000) 2024-02-19T17:38:33.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:33.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:33.938556+0000) 2024-02-19T17:38:34.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:34.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:34.938843+0000) 2024-02-19T17:38:35.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:35.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:35.939029+0000) 2024-02-19T17:38:36.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:36.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:36.939266+0000) 2024-02-19T17:38:37.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:37.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:37.939434+0000) 2024-02-19T17:38:38.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:38.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:38.939682+0000) 2024-02-19T17:38:39.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:39.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:39.939943+0000) 2024-02-19T17:38:40.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:40.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:40.940111+0000) 2024-02-19T17:38:41.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:41.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:41.940334+0000) 2024-02-19T17:38:42.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:42.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:42.940573+0000) 2024-02-19T17:38:43.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:43.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:43.940811+0000) 2024-02-19T17:38:44.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:44.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:44.941048+0000) 2024-02-19T17:38:45.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:45.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:45.941299+0000) 2024-02-19T17:38:46.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:46.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:46.941552+0000) 2024-02-19T17:38:47.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:47.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:47.941749+0000) 2024-02-19T17:38:48.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:48.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:48.941995+0000) 2024-02-19T17:38:49.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:49.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:49.942238+0000) 2024-02-19T17:38:50.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:50.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:50.942490+0000) 2024-02-19T17:38:51.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:51.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:51.942629+0000) 2024-02-19T17:38:52.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:52.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:52.942868+0000) 2024-02-19T17:38:53.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:53.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:53.943009+0000) 2024-02-19T17:38:54.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:54.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:54.943164+0000) 2024-02-19T17:38:55.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:55.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:55.943399+0000) 2024-02-19T17:38:56.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:56.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:56.943635+0000) 2024-02-19T17:38:57.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:57.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:57.943777+0000) 2024-02-19T17:38:58.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:58.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:58.943986+0000) 2024-02-19T17:38:59.728 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:38:59.732 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:38:59.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:38:59.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:38:59.944145+0000) 2024-02-19T17:39:00.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:00.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:00.944408+0000) 2024-02-19T17:39:01.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:01.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:01.944597+0000) 2024-02-19T17:39:02.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:02.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:02.944828+0000) 2024-02-19T17:39:03.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:03.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:03.944959+0000) 2024-02-19T17:39:04.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:04.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:04.945160+0000) 2024-02-19T17:39:05.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:05.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:05.945346+0000) 2024-02-19T17:39:06.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:06.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:06.945575+0000) 2024-02-19T17:39:07.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:07.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:07.945792+0000) 2024-02-19T17:39:08.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:08.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:08.945971+0000) 2024-02-19T17:39:09.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:09.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:09.946218+0000) 2024-02-19T17:39:10.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:10.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:10.946451+0000) 2024-02-19T17:39:11.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:11.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:11.946740+0000) 2024-02-19T17:39:12.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:12.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:12.946933+0000) 2024-02-19T17:39:13.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:13.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:13.947108+0000) 2024-02-19T17:39:14.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:14.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:14.947357+0000) 2024-02-19T17:39:15.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:15.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:15.947628+0000) 2024-02-19T17:39:16.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:16.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:16.947826+0000) 2024-02-19T17:39:17.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:17.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:17.948083+0000) 2024-02-19T17:39:18.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:18.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:18.948321+0000) 2024-02-19T17:39:19.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:19.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:19.948567+0000) 2024-02-19T17:39:20.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:20.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:20.948824+0000) 2024-02-19T17:39:21.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:21.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:21.949080+0000) 2024-02-19T17:39:22.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:22.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:22.949280+0000) 2024-02-19T17:39:23.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:23.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:23.949506+0000) 2024-02-19T17:39:24.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:24.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:24.949743+0000) 2024-02-19T17:39:25.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:25.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:25.949957+0000) 2024-02-19T17:39:26.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:26.949+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:26.950188+0000) 2024-02-19T17:39:27.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:27.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:27.950346+0000) 2024-02-19T17:39:28.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:28.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:28.950611+0000) 2024-02-19T17:39:29.841 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:39:29.845 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:39:29.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:29.949+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:29.950872+0000) 2024-02-19T17:39:30.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:30.949+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:30.951010+0000) 2024-02-19T17:39:31.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:31.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:31.951136+0000) 2024-02-19T17:39:32.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:32.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:32.951327+0000) 2024-02-19T17:39:33.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:33.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:33.951518+0000) 2024-02-19T17:39:34.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:34.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:34.951720+0000) 2024-02-19T17:39:35.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:35.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:35.951998+0000) 2024-02-19T17:39:36.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:36.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:36.952293+0000) 2024-02-19T17:39:37.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:37.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:37.952526+0000) 2024-02-19T17:39:38.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:38.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:38.952747+0000) 2024-02-19T17:39:39.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:39.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:39.953006+0000) 2024-02-19T17:39:40.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:40.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:40.953239+0000) 2024-02-19T17:39:41.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:41.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:41.953478+0000) 2024-02-19T17:39:42.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:42.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:42.953714+0000) 2024-02-19T17:39:43.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:43.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:43.953934+0000) 2024-02-19T17:39:44.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:44.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:44.954196+0000) 2024-02-19T17:39:45.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:45.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:45.954349+0000) 2024-02-19T17:39:46.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:46.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:46.954585+0000) 2024-02-19T17:39:47.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:47.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:47.954794+0000) 2024-02-19T17:39:48.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:48.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:48.955039+0000) 2024-02-19T17:39:49.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:49.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:49.955257+0000) 2024-02-19T17:39:50.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:50.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:50.955444+0000) 2024-02-19T17:39:51.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:51.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:51.955672+0000) 2024-02-19T17:39:52.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:52.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:52.955846+0000) 2024-02-19T17:39:53.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:53.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:53.956032+0000) 2024-02-19T17:39:54.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:54.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:54.956283+0000) 2024-02-19T17:39:55.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:55.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:55.956558+0000) 2024-02-19T17:39:56.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:56.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:56.956741+0000) 2024-02-19T17:39:57.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:57.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:57.956954+0000) 2024-02-19T17:39:58.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:58.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:58.957206+0000) 2024-02-19T17:39:59.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:39:59.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:39:59.957386+0000) 2024-02-19T17:39:59.963 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:39:59.968 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:40:00.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:00.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:00.957639+0000) 2024-02-19T17:40:01.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:01.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:01.957917+0000) 2024-02-19T17:40:02.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:02.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:02.958192+0000) 2024-02-19T17:40:03.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:03.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:03.958472+0000) 2024-02-19T17:40:04.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:04.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:04.958658+0000) 2024-02-19T17:40:05.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:05.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:05.958856+0000) 2024-02-19T17:40:06.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:06.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:06.959054+0000) 2024-02-19T17:40:07.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:07.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:07.959221+0000) 2024-02-19T17:40:08.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:08.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:08.959335+0000) 2024-02-19T17:40:09.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:09.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:09.959579+0000) 2024-02-19T17:40:10.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:10.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:10.959806+0000) 2024-02-19T17:40:11.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:11.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:11.959977+0000) 2024-02-19T17:40:12.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:12.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:12.960251+0000) 2024-02-19T17:40:13.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:13.960+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:13.960443+0000) 2024-02-19T17:40:14.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:14.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:14.960643+0000) 2024-02-19T17:40:15.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:15.960+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:15.960823+0000) 2024-02-19T17:40:16.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:16.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:16.960983+0000) 2024-02-19T17:40:17.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:17.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:17.961201+0000) 2024-02-19T17:40:18.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:18.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:18.961432+0000) 2024-02-19T17:40:19.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:19.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:19.961660+0000) 2024-02-19T17:40:20.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:20.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:20.961787+0000) 2024-02-19T17:40:21.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:21.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:21.961959+0000) 2024-02-19T17:40:22.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:22.960+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:22.962200+0000) 2024-02-19T17:40:23.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:23.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:23.962427+0000) 2024-02-19T17:40:24.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:24.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:24.962597+0000) 2024-02-19T17:40:25.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:25.960+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:25.962839+0000) 2024-02-19T17:40:26.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:26.960+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:26.963075+0000) 2024-02-19T17:40:27.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:27.960+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:27.963301+0000) 2024-02-19T17:40:28.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:28.960+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:28.963522+0000) 2024-02-19T17:40:29.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:29.960+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:29.963843+0000) 2024-02-19T17:40:30.044 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:40:30.048 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:40:30.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:30.960+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:30.964087+0000) 2024-02-19T17:40:31.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:31.964+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:31.964340+0000) 2024-02-19T17:40:32.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:32.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:32.964580+0000) 2024-02-19T17:40:33.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:33.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:33.964780+0000) 2024-02-19T17:40:34.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:34.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:34.965003+0000) 2024-02-19T17:40:35.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:35.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:35.965251+0000) 2024-02-19T17:40:36.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:36.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:36.965470+0000) 2024-02-19T17:40:37.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:37.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:37.965639+0000) 2024-02-19T17:40:38.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:38.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:38.965872+0000) 2024-02-19T17:40:39.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:39.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:39.966050+0000) 2024-02-19T17:40:40.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:40.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:40.966259+0000) 2024-02-19T17:40:41.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:41.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:41.966517+0000) 2024-02-19T17:40:42.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:42.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:42.966810+0000) 2024-02-19T17:40:43.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:43.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:43.967005+0000) 2024-02-19T17:40:44.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:44.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:44.967244+0000) 2024-02-19T17:40:45.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:45.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:45.967406+0000) 2024-02-19T17:40:46.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:46.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:46.967682+0000) 2024-02-19T17:40:47.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:47.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:47.967844+0000) 2024-02-19T17:40:48.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:48.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:48.968100+0000) 2024-02-19T17:40:49.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:49.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:49.968335+0000) 2024-02-19T17:40:50.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:50.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:50.968546+0000) 2024-02-19T17:40:51.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:51.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:51.968803+0000) 2024-02-19T17:40:52.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:52.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:52.968979+0000) 2024-02-19T17:40:53.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:53.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:53.969131+0000) 2024-02-19T17:40:54.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:54.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:54.969342+0000) 2024-02-19T17:40:55.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:55.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:55.969524+0000) 2024-02-19T17:40:56.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:56.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:56.969775+0000) 2024-02-19T17:40:57.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:57.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:57.969934+0000) 2024-02-19T17:40:58.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:58.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:58.970105+0000) 2024-02-19T17:40:59.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:40:59.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:40:59.970331+0000) 2024-02-19T17:41:00.175 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:41:00.178 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:41:00.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:00.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:00.970516+0000) 2024-02-19T17:41:01.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:01.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:01.970701+0000) 2024-02-19T17:41:02.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:02.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:02.970970+0000) 2024-02-19T17:41:03.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:03.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:03.971165+0000) 2024-02-19T17:41:04.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:04.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:04.971365+0000) 2024-02-19T17:41:05.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:05.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:05.971611+0000) 2024-02-19T17:41:06.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:06.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:06.971833+0000) 2024-02-19T17:41:07.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:07.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:07.972092+0000) 2024-02-19T17:41:08.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:08.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:08.972326+0000) 2024-02-19T17:41:09.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:09.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:09.972574+0000) 2024-02-19T17:41:10.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:10.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:10.972763+0000) 2024-02-19T17:41:11.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:11.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:11.972929+0000) 2024-02-19T17:41:12.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:12.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:12.973161+0000) 2024-02-19T17:41:13.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:13.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:13.973336+0000) 2024-02-19T17:41:14.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:14.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:14.973579+0000) 2024-02-19T17:41:15.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:15.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:15.973809+0000) 2024-02-19T17:41:16.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:16.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:16.974012+0000) 2024-02-19T17:41:17.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:17.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:17.974222+0000) 2024-02-19T17:41:18.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:18.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:18.974443+0000) 2024-02-19T17:41:19.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:19.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:19.974697+0000) 2024-02-19T17:41:20.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:20.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:20.974949+0000) 2024-02-19T17:41:21.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:21.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:21.975176+0000) 2024-02-19T17:41:22.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:22.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:22.975400+0000) 2024-02-19T17:41:23.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:23.975+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:23.975639+0000) 2024-02-19T17:41:24.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:24.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:24.975881+0000) 2024-02-19T17:41:25.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:25.975+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:25.976135+0000) 2024-02-19T17:41:26.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:26.975+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:26.976386+0000) 2024-02-19T17:41:27.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:27.975+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:27.976574+0000) 2024-02-19T17:41:28.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:28.975+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:28.976807+0000) 2024-02-19T17:41:29.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:29.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:29.976979+0000) 2024-02-19T17:41:30.288 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:41:30.292 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:41:30.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:30.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:30.977164+0000) 2024-02-19T17:41:31.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:31.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:31.977351+0000) 2024-02-19T17:41:32.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:32.975+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:32.977584+0000) 2024-02-19T17:41:33.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:33.975+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:33.977842+0000) 2024-02-19T17:41:34.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:34.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:34.978032+0000) 2024-02-19T17:41:35.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:35.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:35.978276+0000) 2024-02-19T17:41:36.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:36.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:36.978516+0000) 2024-02-19T17:41:37.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:37.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:37.978797+0000) 2024-02-19T17:41:38.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:38.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:38.978982+0000) 2024-02-19T17:41:39.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:39.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:39.979201+0000) 2024-02-19T17:41:40.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:40.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:40.979384+0000) 2024-02-19T17:41:41.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:41.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:41.979578+0000) 2024-02-19T17:41:42.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:42.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:42.979821+0000) 2024-02-19T17:41:43.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:43.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:43.980018+0000) 2024-02-19T17:41:44.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:44.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:44.980258+0000) 2024-02-19T17:41:45.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:45.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:45.980504+0000) 2024-02-19T17:41:46.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:46.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:46.980749+0000) 2024-02-19T17:41:47.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:47.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:47.980927+0000) 2024-02-19T17:41:48.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:48.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:48.981121+0000) 2024-02-19T17:41:49.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:49.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:49.981356+0000) 2024-02-19T17:41:50.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:50.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:50.981555+0000) 2024-02-19T17:41:51.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:51.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:51.981765+0000) 2024-02-19T17:41:52.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:52.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:52.981982+0000) 2024-02-19T17:41:53.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:53.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:53.982238+0000) 2024-02-19T17:41:54.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:54.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:54.982458+0000) 2024-02-19T17:41:55.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:55.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:55.982755+0000) 2024-02-19T17:41:56.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:56.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:56.982906+0000) 2024-02-19T17:41:57.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:57.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:57.983046+0000) 2024-02-19T17:41:58.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:58.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:58.983217+0000) 2024-02-19T17:41:59.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:41:59.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:41:59.983413+0000) 2024-02-19T17:42:00.410 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:42:00.414 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:42:00.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:00.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:00.983659+0000) 2024-02-19T17:42:01.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:01.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:01.983902+0000) 2024-02-19T17:42:02.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:02.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:02.984130+0000) 2024-02-19T17:42:03.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:03.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:03.984323+0000) 2024-02-19T17:42:04.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:04.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:04.984574+0000) 2024-02-19T17:42:05.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:05.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:05.984791+0000) 2024-02-19T17:42:06.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:06.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:06.985048+0000) 2024-02-19T17:42:07.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:07.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:07.985300+0000) 2024-02-19T17:42:08.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:08.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:08.985457+0000) 2024-02-19T17:42:09.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:09.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:09.985659+0000) 2024-02-19T17:42:10.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:10.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:10.985878+0000) 2024-02-19T17:42:11.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:11.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:11.986067+0000) 2024-02-19T17:42:12.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:12.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:12.986306+0000) 2024-02-19T17:42:13.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:13.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:13.986558+0000) 2024-02-19T17:42:14.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:14.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:14.986776+0000) 2024-02-19T17:42:15.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:15.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:15.987012+0000) 2024-02-19T17:42:16.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:16.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:16.987221+0000) 2024-02-19T17:42:17.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:17.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:17.987488+0000) 2024-02-19T17:42:18.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:18.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:18.987719+0000) 2024-02-19T17:42:19.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:19.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:19.987906+0000) 2024-02-19T17:42:20.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:20.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:20.988064+0000) 2024-02-19T17:42:21.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:21.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:21.988322+0000) 2024-02-19T17:42:22.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:22.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:22.988550+0000) 2024-02-19T17:42:23.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:23.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:23.988809+0000) 2024-02-19T17:42:24.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:24.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:24.989047+0000) 2024-02-19T17:42:25.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:25.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:25.989301+0000) 2024-02-19T17:42:26.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:26.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:26.989535+0000) 2024-02-19T17:42:27.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:27.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:27.989790+0000) 2024-02-19T17:42:28.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:28.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:28.990029+0000) 2024-02-19T17:42:29.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:29.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:29.990269+0000) 2024-02-19T17:42:30.524 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:42:30.527 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:42:30.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:30.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:30.990466+0000) 2024-02-19T17:42:31.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:31.990+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:31.990741+0000) 2024-02-19T17:42:32.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:32.990+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:32.990883+0000) 2024-02-19T17:42:33.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:33.990+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:33.991080+0000) 2024-02-19T17:42:34.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:34.990+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:34.991248+0000) 2024-02-19T17:42:35.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:35.990+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:35.991472+0000) 2024-02-19T17:42:36.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:36.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:36.991629+0000) 2024-02-19T17:42:37.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:37.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:37.991889+0000) 2024-02-19T17:42:38.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:38.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:38.992064+0000) 2024-02-19T17:42:39.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:39.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:39.992187+0000) 2024-02-19T17:42:40.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:40.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:40.992411+0000) 2024-02-19T17:42:41.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:41.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:41.992586+0000) 2024-02-19T17:42:42.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:42.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:42.992844+0000) 2024-02-19T17:42:43.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:43.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:43.993082+0000) 2024-02-19T17:42:44.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:44.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:44.993263+0000) 2024-02-19T17:42:45.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:45.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:45.993505+0000) 2024-02-19T17:42:46.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:46.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:46.993754+0000) 2024-02-19T17:42:47.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:47.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:47.993961+0000) 2024-02-19T17:42:48.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:48.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:48.994169+0000) 2024-02-19T17:42:49.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:49.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:49.994426+0000) 2024-02-19T17:42:50.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:50.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:50.994667+0000) 2024-02-19T17:42:51.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:51.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:51.994884+0000) 2024-02-19T17:42:52.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:52.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:52.995019+0000) 2024-02-19T17:42:53.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:53.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:53.995189+0000) 2024-02-19T17:42:54.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:54.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:54.995367+0000) 2024-02-19T17:42:55.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:55.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:55.995626+0000) 2024-02-19T17:42:56.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:56.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:56.995854+0000) 2024-02-19T17:42:57.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:57.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:57.996089+0000) 2024-02-19T17:42:58.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:58.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:58.996327+0000) 2024-02-19T17:42:59.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:42:59.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:42:59.996569+0000) 2024-02-19T17:43:00.596 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:43:00.599 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:43:00.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:00.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:00.996807+0000) 2024-02-19T17:43:01.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:01.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:01.997074+0000) 2024-02-19T17:43:02.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:02.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:02.997274+0000) 2024-02-19T17:43:03.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:03.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:03.997437+0000) 2024-02-19T17:43:04.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:04.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:04.997589+0000) 2024-02-19T17:43:05.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:05.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:05.997832+0000) 2024-02-19T17:43:06.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:06.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:06.998018+0000) 2024-02-19T17:43:07.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:07.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:07.998243+0000) 2024-02-19T17:43:08.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:08.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:08.998478+0000) 2024-02-19T17:43:09.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:09.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:09.998852+0000) 2024-02-19T17:43:10.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:10.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:10.999071+0000) 2024-02-19T17:43:11.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:11.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:11.999193+0000) 2024-02-19T17:43:12.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:12.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:12.999439+0000) 2024-02-19T17:43:13.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:13.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:13.999631+0000) 2024-02-19T17:43:14.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:14.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:14.999869+0000) 2024-02-19T17:43:15.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:15.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:16.000167+0000) 2024-02-19T17:43:16.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:16.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:17.000297+0000) 2024-02-19T17:43:17.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:17.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:18.000472+0000) 2024-02-19T17:43:18.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:18.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:19.000642+0000) 2024-02-19T17:43:19.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:19.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:20.000863+0000) 2024-02-19T17:43:20.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:20.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:21.001112+0000) 2024-02-19T17:43:21.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:21.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:22.001337+0000) 2024-02-19T17:43:22.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:22.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:23.001557+0000) 2024-02-19T17:43:23.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:23.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:24.001798+0000) 2024-02-19T17:43:24.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:24.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:25.002029+0000) 2024-02-19T17:43:25.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:25.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:26.002251+0000) 2024-02-19T17:43:26.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:26.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:27.002473+0000) 2024-02-19T17:43:27.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:28.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:28.002747+0000) 2024-02-19T17:43:28.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:29.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:29.002908+0000) 2024-02-19T17:43:29.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:30.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:30.003050+0000) 2024-02-19T17:43:30.719 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:43:30.722 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:43:30.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:31.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:31.003254+0000) 2024-02-19T17:43:31.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:32.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:32.003493+0000) 2024-02-19T17:43:32.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:33.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:33.003733+0000) 2024-02-19T17:43:33.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:34.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:34.003975+0000) 2024-02-19T17:43:34.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:35.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:35.004249+0000) 2024-02-19T17:43:35.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:36.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:36.004464+0000) 2024-02-19T17:43:36.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:37.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:37.004619+0000) 2024-02-19T17:43:37.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:38.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:38.004881+0000) 2024-02-19T17:43:38.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:39.000+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:39.005108+0000) 2024-02-19T17:43:39.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:40.000+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:40.005345+0000) 2024-02-19T17:43:40.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:41.000+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:41.005541+0000) 2024-02-19T17:43:41.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:42.000+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:42.005765+0000) 2024-02-19T17:43:42.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:43.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:43.005939+0000) 2024-02-19T17:43:43.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:44.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:44.006201+0000) 2024-02-19T17:43:44.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:45.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:45.006369+0000) 2024-02-19T17:43:45.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:46.000+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:46.006617+0000) 2024-02-19T17:43:46.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:47.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:47.006835+0000) 2024-02-19T17:43:47.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:48.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:48.007012+0000) 2024-02-19T17:43:48.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:49.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:49.007190+0000) 2024-02-19T17:43:49.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:50.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:50.007385+0000) 2024-02-19T17:43:50.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:51.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:51.007610+0000) 2024-02-19T17:43:51.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:52.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:52.007868+0000) 2024-02-19T17:43:52.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:53.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:53.008138+0000) 2024-02-19T17:43:53.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:54.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:54.008322+0000) 2024-02-19T17:43:54.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:55.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:55.008538+0000) 2024-02-19T17:43:55.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:56.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:56.008820+0000) 2024-02-19T17:43:56.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:57.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:57.009016+0000) 2024-02-19T17:43:57.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:58.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:58.009247+0000) 2024-02-19T17:43:58.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:43:59.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:43:59.009470+0000) 2024-02-19T17:43:59.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:00.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:00.009678+0000) 2024-02-19T17:44:00.799 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:44:00.803 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:44:00.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:01.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:01.009895+0000) 2024-02-19T17:44:01.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:02.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:02.010093+0000) 2024-02-19T17:44:02.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:03.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:03.010262+0000) 2024-02-19T17:44:03.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:04.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:04.010479+0000) 2024-02-19T17:44:04.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:05.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:05.010788+0000) 2024-02-19T17:44:05.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:06.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:06.011012+0000) 2024-02-19T17:44:06.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:07.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:07.011235+0000) 2024-02-19T17:44:07.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:08.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:08.011479+0000) 2024-02-19T17:44:08.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:09.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:09.011655+0000) 2024-02-19T17:44:09.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:10.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:10.011879+0000) 2024-02-19T17:44:10.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:11.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:11.012117+0000) 2024-02-19T17:44:11.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:12.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:12.012372+0000) 2024-02-19T17:44:12.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:13.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:13.012617+0000) 2024-02-19T17:44:13.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:14.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:14.012853+0000) 2024-02-19T17:44:14.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:15.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:15.013087+0000) 2024-02-19T17:44:15.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:16.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:16.013316+0000) 2024-02-19T17:44:16.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:17.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:17.013447+0000) 2024-02-19T17:44:17.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:18.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:18.013695+0000) 2024-02-19T17:44:18.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:19.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:19.013930+0000) 2024-02-19T17:44:19.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:20.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:20.014170+0000) 2024-02-19T17:44:20.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:21.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:21.014401+0000) 2024-02-19T17:44:21.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:22.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:22.014609+0000) 2024-02-19T17:44:22.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:23.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:23.014848+0000) 2024-02-19T17:44:23.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:24.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:24.015046+0000) 2024-02-19T17:44:24.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:25.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:25.015262+0000) 2024-02-19T17:44:25.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:26.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:26.015503+0000) 2024-02-19T17:44:26.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:27.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:27.015742+0000) 2024-02-19T17:44:27.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:28.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:28.015914+0000) 2024-02-19T17:44:28.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:29.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:29.016185+0000) 2024-02-19T17:44:29.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:30.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:30.016401+0000) 2024-02-19T17:44:30.939 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:44:30.943 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:44:30.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:31.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:31.016608+0000) 2024-02-19T17:44:31.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:32.016+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:32.016810+0000) 2024-02-19T17:44:32.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:33.016+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:33.017049+0000) 2024-02-19T17:44:33.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:34.016+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:34.017300+0000) 2024-02-19T17:44:34.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:35.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:35.017524+0000) 2024-02-19T17:44:35.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:36.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:36.017778+0000) 2024-02-19T17:44:36.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:37.016+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:37.018026+0000) 2024-02-19T17:44:37.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:38.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:38.018204+0000) 2024-02-19T17:44:38.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:39.016+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:39.018441+0000) 2024-02-19T17:44:39.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:40.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:40.018674+0000) 2024-02-19T17:44:40.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:41.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:41.018874+0000) 2024-02-19T17:44:41.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:42.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:42.019068+0000) 2024-02-19T17:44:42.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:43.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:43.019260+0000) 2024-02-19T17:44:43.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:44.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:44.019520+0000) 2024-02-19T17:44:44.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:45.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:45.019758+0000) 2024-02-19T17:44:45.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:46.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:46.020062+0000) 2024-02-19T17:44:46.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:47.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:47.020319+0000) 2024-02-19T17:44:47.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:48.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:48.020569+0000) 2024-02-19T17:44:48.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:49.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:49.020773+0000) 2024-02-19T17:44:49.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:50.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:50.020965+0000) 2024-02-19T17:44:50.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:51.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:51.021222+0000) 2024-02-19T17:44:51.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:52.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:52.021479+0000) 2024-02-19T17:44:52.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:53.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:53.021713+0000) 2024-02-19T17:44:53.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:54.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:54.021904+0000) 2024-02-19T17:44:54.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:55.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:55.022138+0000) 2024-02-19T17:44:55.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:56.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:56.022403+0000) 2024-02-19T17:44:56.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:57.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:57.022637+0000) 2024-02-19T17:44:57.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:58.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:58.022917+0000) 2024-02-19T17:44:58.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:44:59.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:44:59.023073+0000) 2024-02-19T17:44:59.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:00.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:00.023194+0000) 2024-02-19T17:45:00.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:01.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:01.023435+0000) 2024-02-19T17:45:01.261 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:45:01.264 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:45:01.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:02.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:02.023703+0000) 2024-02-19T17:45:02.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:03.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:03.023936+0000) 2024-02-19T17:45:03.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:04.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:04.024138+0000) 2024-02-19T17:45:05.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:05.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:05.024310+0000) 2024-02-19T17:45:06.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:06.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:06.024555+0000) 2024-02-19T17:45:07.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:07.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:07.024756+0000) 2024-02-19T17:45:08.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:08.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:08.025014+0000) 2024-02-19T17:45:09.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:09.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:09.025246+0000) 2024-02-19T17:45:10.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:10.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:10.025436+0000) 2024-02-19T17:45:11.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:11.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:11.025695+0000) 2024-02-19T17:45:12.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:12.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:12.025972+0000) 2024-02-19T17:45:13.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:13.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:13.026169+0000) 2024-02-19T17:45:14.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:14.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:14.026416+0000) 2024-02-19T17:45:15.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:15.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:15.026592+0000) 2024-02-19T17:45:16.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:16.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:16.026860+0000) 2024-02-19T17:45:17.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:17.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:17.027095+0000) 2024-02-19T17:45:18.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:18.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:18.027251+0000) 2024-02-19T17:45:19.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:19.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:19.027453+0000) 2024-02-19T17:45:20.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:20.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:20.027711+0000) 2024-02-19T17:45:21.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:21.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:21.027995+0000) 2024-02-19T17:45:22.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:22.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:22.028289+0000) 2024-02-19T17:45:23.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:23.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:23.028529+0000) 2024-02-19T17:45:24.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:24.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:24.028783+0000) 2024-02-19T17:45:25.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:25.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:25.029013+0000) 2024-02-19T17:45:26.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:26.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:26.029245+0000) 2024-02-19T17:45:27.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:27.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:27.029504+0000) 2024-02-19T17:45:28.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:28.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:28.029677+0000) 2024-02-19T17:45:29.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:29.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:29.029918+0000) 2024-02-19T17:45:30.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:30.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:30.030170+0000) 2024-02-19T17:45:31.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:31.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:31.030400+0000) 2024-02-19T17:45:31.325 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:45:31.328 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:45:32.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:32.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:32.030643+0000) 2024-02-19T17:45:33.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:33.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:33.030883+0000) 2024-02-19T17:45:34.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:34.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:34.031045+0000) 2024-02-19T17:45:35.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:35.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:35.031286+0000) 2024-02-19T17:45:36.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:36.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:36.031558+0000) 2024-02-19T17:45:37.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:37.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:37.031739+0000) 2024-02-19T17:45:38.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:38.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:38.032016+0000) 2024-02-19T17:45:39.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:39.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:39.032262+0000) 2024-02-19T17:45:40.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:40.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:40.032507+0000) 2024-02-19T17:45:41.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:41.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:41.032731+0000) 2024-02-19T17:45:42.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:42.026+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:42.032949+0000) 2024-02-19T17:45:43.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:43.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:43.033192+0000) 2024-02-19T17:45:44.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:44.026+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:44.033423+0000) 2024-02-19T17:45:45.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:45.026+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:45.033715+0000) 2024-02-19T17:45:46.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:46.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:46.033987+0000) 2024-02-19T17:45:47.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:47.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:47.034240+0000) 2024-02-19T17:45:48.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:48.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:48.034423+0000) 2024-02-19T17:45:49.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:49.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:49.034631+0000) 2024-02-19T17:45:50.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:50.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:50.034819+0000) 2024-02-19T17:45:51.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:51.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:51.034994+0000) 2024-02-19T17:45:52.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:52.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:52.035270+0000) 2024-02-19T17:45:53.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:53.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:53.035521+0000) 2024-02-19T17:45:54.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:54.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:54.035720+0000) 2024-02-19T17:45:55.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:55.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:55.035961+0000) 2024-02-19T17:45:56.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:56.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:56.036152+0000) 2024-02-19T17:45:57.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:57.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:57.036392+0000) 2024-02-19T17:45:58.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:58.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:58.036636+0000) 2024-02-19T17:45:59.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:45:59.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:45:59.036883+0000) 2024-02-19T17:46:00.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:00.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:00.037128+0000) 2024-02-19T17:46:01.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:01.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:01.037387+0000) 2024-02-19T17:46:01.455 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:46:01.458 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:46:02.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:02.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:02.037658+0000) 2024-02-19T17:46:03.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:03.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:03.037884+0000) 2024-02-19T17:46:04.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:04.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:04.038127+0000) 2024-02-19T17:46:05.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:05.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:05.038323+0000) 2024-02-19T17:46:06.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:06.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:06.038617+0000) 2024-02-19T17:46:07.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:07.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:07.038875+0000) 2024-02-19T17:46:08.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:08.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:08.039063+0000) 2024-02-19T17:46:09.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:09.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:09.039312+0000) 2024-02-19T17:46:10.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:10.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:10.039555+0000) 2024-02-19T17:46:11.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:11.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:11.039809+0000) 2024-02-19T17:46:12.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:12.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:12.040011+0000) 2024-02-19T17:46:13.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:13.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:13.040239+0000) 2024-02-19T17:46:14.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:14.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:14.040486+0000) 2024-02-19T17:46:15.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:15.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:15.040695+0000) 2024-02-19T17:46:16.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:16.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:16.040959+0000) 2024-02-19T17:46:17.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:17.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:17.041193+0000) 2024-02-19T17:46:18.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:18.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:18.041376+0000) 2024-02-19T17:46:19.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:19.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:19.041573+0000) 2024-02-19T17:46:20.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:20.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:20.041825+0000) 2024-02-19T17:46:21.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:21.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:21.042054+0000) 2024-02-19T17:46:22.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:22.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:22.042287+0000) 2024-02-19T17:46:23.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:23.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:23.042421+0000) 2024-02-19T17:46:24.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:24.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:24.042661+0000) 2024-02-19T17:46:25.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:25.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:25.042895+0000) 2024-02-19T17:46:26.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:26.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:26.043147+0000) 2024-02-19T17:46:27.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:27.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:27.043336+0000) 2024-02-19T17:46:28.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:28.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:28.043584+0000) 2024-02-19T17:46:29.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:29.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:29.043821+0000) 2024-02-19T17:46:30.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:30.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:30.044042+0000) 2024-02-19T17:46:31.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:31.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:31.044256+0000) 2024-02-19T17:46:31.587 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:46:31.590 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:46:32.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:32.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:32.044464+0000) 2024-02-19T17:46:33.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:33.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:33.044723+0000) 2024-02-19T17:46:34.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:34.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:34.044951+0000) 2024-02-19T17:46:35.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:35.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:35.045191+0000) 2024-02-19T17:46:36.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:36.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:36.045416+0000) 2024-02-19T17:46:37.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:37.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:37.045639+0000) 2024-02-19T17:46:38.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:38.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:38.045893+0000) 2024-02-19T17:46:39.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:39.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:39.046118+0000) 2024-02-19T17:46:40.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:40.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:40.046371+0000) 2024-02-19T17:46:41.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:41.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:41.046523+0000) 2024-02-19T17:46:42.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:42.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:42.046691+0000) 2024-02-19T17:46:43.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:43.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:43.046847+0000) 2024-02-19T17:46:44.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:44.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:44.046981+0000) 2024-02-19T17:46:45.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:45.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:45.047233+0000) 2024-02-19T17:46:46.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:46.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:46.047503+0000) 2024-02-19T17:46:47.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:47.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:47.047676+0000) 2024-02-19T17:46:48.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:48.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:48.047871+0000) 2024-02-19T17:46:49.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:49.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:49.048055+0000) 2024-02-19T17:46:50.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:50.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:50.048194+0000) 2024-02-19T17:46:51.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:51.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:51.048413+0000) 2024-02-19T17:46:52.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:52.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:52.048650+0000) 2024-02-19T17:46:53.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:53.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:53.048886+0000) 2024-02-19T17:46:54.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:54.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:54.049115+0000) 2024-02-19T17:46:55.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:55.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:55.049270+0000) 2024-02-19T17:46:56.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:56.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:56.049446+0000) 2024-02-19T17:46:57.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:57.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:57.049693+0000) 2024-02-19T17:46:58.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:58.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:58.049891+0000) 2024-02-19T17:46:59.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:46:59.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:46:59.050088+0000) 2024-02-19T17:47:00.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:00.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:00.050330+0000) 2024-02-19T17:47:01.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:01.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:01.050570+0000) 2024-02-19T17:47:01.725 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:47:01.729 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:47:02.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:02.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:02.050847+0000) 2024-02-19T17:47:03.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:03.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:03.051060+0000) 2024-02-19T17:47:04.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:04.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:04.051323+0000) 2024-02-19T17:47:05.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:05.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:05.051496+0000) 2024-02-19T17:47:06.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:06.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:06.051772+0000) 2024-02-19T17:47:07.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:07.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:07.052001+0000) 2024-02-19T17:47:08.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:08.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:08.052235+0000) 2024-02-19T17:47:09.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:09.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:09.052368+0000) 2024-02-19T17:47:10.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:10.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:10.052623+0000) 2024-02-19T17:47:11.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:11.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:11.052844+0000) 2024-02-19T17:47:12.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:12.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:12.053081+0000) 2024-02-19T17:47:13.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:13.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:13.053314+0000) 2024-02-19T17:47:14.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:14.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:14.053624+0000) 2024-02-19T17:47:15.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:15.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:15.053875+0000) 2024-02-19T17:47:16.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:16.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:16.054117+0000) 2024-02-19T17:47:17.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:17.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:17.054343+0000) 2024-02-19T17:47:18.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:18.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:18.054557+0000) 2024-02-19T17:47:19.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:19.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:19.054786+0000) 2024-02-19T17:47:20.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:20.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:20.054981+0000) 2024-02-19T17:47:21.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:21.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:21.055150+0000) 2024-02-19T17:47:22.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:22.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:22.055326+0000) 2024-02-19T17:47:23.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:23.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:23.055511+0000) 2024-02-19T17:47:24.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:24.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:24.055734+0000) 2024-02-19T17:47:25.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:25.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:25.055926+0000) 2024-02-19T17:47:26.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:26.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:26.056146+0000) 2024-02-19T17:47:27.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:27.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:27.056324+0000) 2024-02-19T17:47:28.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:28.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:28.056552+0000) 2024-02-19T17:47:29.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:29.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:29.056711+0000) 2024-02-19T17:47:30.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:30.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:30.056936+0000) 2024-02-19T17:47:31.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:31.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:31.057137+0000) 2024-02-19T17:47:31.831 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:47:31.834 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:47:32.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:32.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:32.057367+0000) 2024-02-19T17:47:33.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:33.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:33.057536+0000) 2024-02-19T17:47:34.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:34.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:34.057721+0000) 2024-02-19T17:47:35.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:35.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:35.057934+0000) 2024-02-19T17:47:36.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:36.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:36.058120+0000) 2024-02-19T17:47:37.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:37.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:37.058280+0000) 2024-02-19T17:47:38.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:38.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:38.058508+0000) 2024-02-19T17:47:39.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:39.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:39.058697+0000) 2024-02-19T17:47:40.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:40.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:40.058875+0000) 2024-02-19T17:47:41.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:41.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:41.059043+0000) 2024-02-19T17:47:42.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:42.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:42.059279+0000) 2024-02-19T17:47:43.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:43.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:43.059519+0000) 2024-02-19T17:47:44.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:44.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:44.059758+0000) 2024-02-19T17:47:45.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:45.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:45.059932+0000) 2024-02-19T17:47:46.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:46.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:46.060187+0000) 2024-02-19T17:47:47.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:47.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:47.060410+0000) 2024-02-19T17:47:48.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:48.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:48.060664+0000) 2024-02-19T17:47:49.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:49.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:49.060896+0000) 2024-02-19T17:47:50.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:50.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:50.061038+0000) 2024-02-19T17:47:51.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:51.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:51.061237+0000) 2024-02-19T17:47:52.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:52.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:52.061483+0000) 2024-02-19T17:47:53.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:53.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:53.061725+0000) 2024-02-19T17:47:54.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:54.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:54.061971+0000) 2024-02-19T17:47:55.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:55.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:55.062117+0000) 2024-02-19T17:47:56.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:56.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:56.062339+0000) 2024-02-19T17:47:57.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:57.056+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:57.062560+0000) 2024-02-19T17:47:58.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:58.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:58.062824+0000) 2024-02-19T17:47:59.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:47:59.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:47:59.063003+0000) 2024-02-19T17:48:00.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:00.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:00.063146+0000) 2024-02-19T17:48:01.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:01.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:01.063399+0000) 2024-02-19T17:48:02.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:02.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:02.063660+0000) 2024-02-19T17:48:02.145 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:48:02.148 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:48:03.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:03.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:03.063900+0000) 2024-02-19T17:48:04.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:04.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:04.064143+0000) 2024-02-19T17:48:05.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:05.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:05.064327+0000) 2024-02-19T17:48:06.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:06.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:06.064577+0000) 2024-02-19T17:48:07.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:07.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:07.064826+0000) 2024-02-19T17:48:08.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:08.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:08.065063+0000) 2024-02-19T17:48:09.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:09.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:09.065238+0000) 2024-02-19T17:48:10.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:10.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:10.065399+0000) 2024-02-19T17:48:11.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:11.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:11.065565+0000) 2024-02-19T17:48:12.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:12.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:12.065809+0000) 2024-02-19T17:48:13.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:13.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:13.065997+0000) 2024-02-19T17:48:14.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:14.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:14.066273+0000) 2024-02-19T17:48:15.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:15.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:15.066507+0000) 2024-02-19T17:48:16.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:16.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:16.066808+0000) 2024-02-19T17:48:17.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:17.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:17.066993+0000) 2024-02-19T17:48:18.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:18.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:18.067125+0000) 2024-02-19T17:48:19.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:19.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:19.067352+0000) 2024-02-19T17:48:20.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:20.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:20.067588+0000) 2024-02-19T17:48:21.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:21.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:21.067815+0000) 2024-02-19T17:48:22.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:22.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:22.068058+0000) 2024-02-19T17:48:23.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:23.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:23.068229+0000) 2024-02-19T17:48:24.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:24.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:24.068485+0000) 2024-02-19T17:48:25.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:25.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:25.068736+0000) 2024-02-19T17:48:26.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:26.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:26.068958+0000) 2024-02-19T17:48:27.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:27.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:27.069100+0000) 2024-02-19T17:48:28.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:28.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:28.069292+0000) 2024-02-19T17:48:29.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:29.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:29.069566+0000) 2024-02-19T17:48:30.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:30.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:30.069805+0000) 2024-02-19T17:48:31.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:31.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:31.070016+0000) 2024-02-19T17:48:32.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:32.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:32.070214+0000) 2024-02-19T17:48:32.199 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:48:32.203 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:48:33.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:33.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:33.070439+0000) 2024-02-19T17:48:34.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:34.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:34.070681+0000) 2024-02-19T17:48:35.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:35.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:35.070949+0000) 2024-02-19T17:48:36.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:36.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:36.071145+0000) 2024-02-19T17:48:37.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:37.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:37.071279+0000) 2024-02-19T17:48:38.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:38.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:38.071536+0000) 2024-02-19T17:48:39.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:39.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:39.071772+0000) 2024-02-19T17:48:40.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:40.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:40.072047+0000) 2024-02-19T17:48:41.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:41.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:41.072237+0000) 2024-02-19T17:48:42.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:42.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:42.072432+0000) 2024-02-19T17:48:43.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:43.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:43.072618+0000) 2024-02-19T17:48:44.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:44.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:44.072830+0000) 2024-02-19T17:48:45.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:45.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:45.073052+0000) 2024-02-19T17:48:46.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:46.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:46.073284+0000) 2024-02-19T17:48:47.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:47.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:47.073520+0000) 2024-02-19T17:48:48.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:48.067+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:48.073755+0000) 2024-02-19T17:48:49.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:49.067+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:49.073986+0000) 2024-02-19T17:48:50.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:50.067+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:50.074185+0000) 2024-02-19T17:48:51.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:51.067+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:51.074416+0000) 2024-02-19T17:48:52.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:52.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:52.074681+0000) 2024-02-19T17:48:53.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:53.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:53.074870+0000) 2024-02-19T17:48:54.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:54.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:54.075036+0000) 2024-02-19T17:48:55.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:55.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:55.075279+0000) 2024-02-19T17:48:56.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:56.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:56.075491+0000) 2024-02-19T17:48:57.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:57.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:57.075620+0000) 2024-02-19T17:48:58.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:58.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:58.075805+0000) 2024-02-19T17:48:59.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:48:59.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:48:59.076052+0000) 2024-02-19T17:49:00.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:00.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:00.076256+0000) 2024-02-19T17:49:01.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:01.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:01.076477+0000) 2024-02-19T17:49:02.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:02.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:02.076688+0000) 2024-02-19T17:49:02.330 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:49:02.334 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:49:03.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:03.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:03.076944+0000) 2024-02-19T17:49:04.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:04.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:04.077189+0000) 2024-02-19T17:49:05.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:05.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:05.077365+0000) 2024-02-19T17:49:06.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:06.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:06.077612+0000) 2024-02-19T17:49:07.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:07.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:07.077838+0000) 2024-02-19T17:49:08.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:08.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:08.078142+0000) 2024-02-19T17:49:09.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:09.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:09.078376+0000) 2024-02-19T17:49:10.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:10.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:10.078632+0000) 2024-02-19T17:49:11.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:11.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:11.078900+0000) 2024-02-19T17:49:12.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:12.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:12.079039+0000) 2024-02-19T17:49:13.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:13.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:13.079264+0000) 2024-02-19T17:49:14.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:14.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:14.079519+0000) 2024-02-19T17:49:15.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:15.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:15.079768+0000) 2024-02-19T17:49:16.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:16.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:16.080016+0000) 2024-02-19T17:49:17.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:17.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:17.080261+0000) 2024-02-19T17:49:18.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:18.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:18.080500+0000) 2024-02-19T17:49:19.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:19.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:19.080745+0000) 2024-02-19T17:49:20.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:20.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:20.080990+0000) 2024-02-19T17:49:21.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:21.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:21.081226+0000) 2024-02-19T17:49:22.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:22.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:22.081484+0000) 2024-02-19T17:49:23.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:23.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:23.081736+0000) 2024-02-19T17:49:24.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:24.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:24.081932+0000) 2024-02-19T17:49:25.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:25.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:25.082129+0000) 2024-02-19T17:49:26.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:26.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:26.082409+0000) 2024-02-19T17:49:27.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:27.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:27.082593+0000) 2024-02-19T17:49:28.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:28.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:28.082835+0000) 2024-02-19T17:49:29.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:29.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:29.083028+0000) 2024-02-19T17:49:30.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:30.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:30.083272+0000) 2024-02-19T17:49:31.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:31.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:31.083448+0000) 2024-02-19T17:49:32.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:32.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:32.083642+0000) 2024-02-19T17:49:32.453 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:49:32.458 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:49:33.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:33.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:33.083902+0000) 2024-02-19T17:49:34.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:34.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:34.084103+0000) 2024-02-19T17:49:35.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:35.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:35.084331+0000) 2024-02-19T17:49:36.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:36.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:36.084612+0000) 2024-02-19T17:49:37.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:37.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:37.084854+0000) 2024-02-19T17:49:38.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:38.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:38.085104+0000) 2024-02-19T17:49:39.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:39.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:39.085336+0000) 2024-02-19T17:49:40.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:40.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:40.085581+0000) 2024-02-19T17:49:41.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:41.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:41.085844+0000) 2024-02-19T17:49:42.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:42.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:42.086080+0000) 2024-02-19T17:49:43.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:43.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:43.086278+0000) 2024-02-19T17:49:44.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:44.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:44.086525+0000) 2024-02-19T17:49:45.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:45.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:45.086800+0000) 2024-02-19T17:49:46.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:46.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:46.087106+0000) 2024-02-19T17:49:47.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:47.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:47.087370+0000) 2024-02-19T17:49:48.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:48.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:48.087605+0000) 2024-02-19T17:49:49.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:49.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:49.087847+0000) 2024-02-19T17:49:50.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:50.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:50.088031+0000) 2024-02-19T17:49:51.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:51.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:51.088218+0000) 2024-02-19T17:49:52.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:52.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:52.088459+0000) 2024-02-19T17:49:53.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:53.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:53.088663+0000) 2024-02-19T17:49:54.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:54.082+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:54.088906+0000) 2024-02-19T17:49:55.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:55.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:55.089165+0000) 2024-02-19T17:49:56.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:56.082+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:56.089410+0000) 2024-02-19T17:49:57.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:57.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:57.089664+0000) 2024-02-19T17:49:58.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:58.082+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:58.089846+0000) 2024-02-19T17:49:59.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:49:59.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:49:59.090095+0000) 2024-02-19T17:50:00.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:00.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:00.090340+0000) 2024-02-19T17:50:01.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:01.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:01.090625+0000) 2024-02-19T17:50:02.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:02.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:02.090813+0000) 2024-02-19T17:50:02.575 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:50:02.578 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:50:03.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:03.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:03.090989+0000) 2024-02-19T17:50:04.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:04.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:04.091112+0000) 2024-02-19T17:50:05.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:05.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:05.091353+0000) 2024-02-19T17:50:06.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:06.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:06.091630+0000) 2024-02-19T17:50:07.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:07.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:07.091802+0000) 2024-02-19T17:50:08.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:08.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:08.091995+0000) 2024-02-19T17:50:09.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:09.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:09.092191+0000) 2024-02-19T17:50:10.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:10.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:10.092394+0000) 2024-02-19T17:50:11.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:11.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:11.092622+0000) 2024-02-19T17:50:12.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:12.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:12.092799+0000) 2024-02-19T17:50:13.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:13.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:13.092996+0000) 2024-02-19T17:50:14.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:14.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:14.093216+0000) 2024-02-19T17:50:15.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:15.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:15.093399+0000) 2024-02-19T17:50:16.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:16.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:16.093642+0000) 2024-02-19T17:50:17.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:17.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:17.093882+0000) 2024-02-19T17:50:18.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:18.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:18.094063+0000) 2024-02-19T17:50:19.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:19.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:19.094345+0000) 2024-02-19T17:50:20.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:20.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:20.094592+0000) 2024-02-19T17:50:21.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:21.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:21.094804+0000) 2024-02-19T17:50:22.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:22.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:22.094928+0000) 2024-02-19T17:50:23.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:23.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:23.095118+0000) 2024-02-19T17:50:24.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:24.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:24.095298+0000) 2024-02-19T17:50:25.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:25.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:25.095528+0000) 2024-02-19T17:50:26.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:26.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:26.095782+0000) 2024-02-19T17:50:27.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:27.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:27.096031+0000) 2024-02-19T17:50:28.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:28.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:28.096279+0000) 2024-02-19T17:50:29.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:29.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:29.096522+0000) 2024-02-19T17:50:30.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:30.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:30.096696+0000) 2024-02-19T17:50:31.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:31.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:31.096888+0000) 2024-02-19T17:50:32.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:32.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:32.097065+0000) 2024-02-19T17:50:32.631 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:50:32.640 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:50:33.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:33.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:33.097323+0000) 2024-02-19T17:50:34.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:34.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:34.097589+0000) 2024-02-19T17:50:35.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:35.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:35.097820+0000) 2024-02-19T17:50:36.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:36.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:36.098060+0000) 2024-02-19T17:50:37.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:37.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:37.098289+0000) 2024-02-19T17:50:38.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:38.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:38.098559+0000) 2024-02-19T17:50:39.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:39.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:39.098786+0000) 2024-02-19T17:50:40.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:40.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:40.098960+0000) 2024-02-19T17:50:41.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:41.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:41.099122+0000) 2024-02-19T17:50:42.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:42.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:42.099318+0000) 2024-02-19T17:50:43.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:43.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:43.099572+0000) 2024-02-19T17:50:44.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:44.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:44.099823+0000) 2024-02-19T17:50:45.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:45.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:45.100069+0000) 2024-02-19T17:50:46.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:46.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:46.100257+0000) 2024-02-19T17:50:47.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:47.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:47.100437+0000) 2024-02-19T17:50:48.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:48.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:48.100633+0000) 2024-02-19T17:50:49.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:49.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:49.100929+0000) 2024-02-19T17:50:50.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:50.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:50.101134+0000) 2024-02-19T17:50:51.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:51.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:51.101369+0000) 2024-02-19T17:50:52.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:52.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:52.101629+0000) 2024-02-19T17:50:53.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:53.097+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:53.101917+0000) 2024-02-19T17:50:54.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:54.097+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:54.102102+0000) 2024-02-19T17:50:55.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:55.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:55.102353+0000) 2024-02-19T17:50:56.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:56.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:56.102572+0000) 2024-02-19T17:50:57.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:57.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:57.102812+0000) 2024-02-19T17:50:58.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:58.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:58.103054+0000) 2024-02-19T17:50:59.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:50:59.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:50:59.103288+0000) 2024-02-19T17:51:00.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:00.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:00.103494+0000) 2024-02-19T17:51:01.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:01.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:01.103727+0000) 2024-02-19T17:51:02.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:02.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:02.103992+0000) 2024-02-19T17:51:02.790 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:51:02.796 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:51:03.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:03.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:03.104230+0000) 2024-02-19T17:51:04.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:04.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:04.104636+0000) 2024-02-19T17:51:05.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:05.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:05.104868+0000) 2024-02-19T17:51:06.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:06.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:06.105124+0000) 2024-02-19T17:51:07.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:07.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:07.105355+0000) 2024-02-19T17:51:08.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:08.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:08.105605+0000) 2024-02-19T17:51:09.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:09.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:09.105862+0000) 2024-02-19T17:51:10.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:10.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:10.106051+0000) 2024-02-19T17:51:11.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:11.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:11.106306+0000) 2024-02-19T17:51:12.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:12.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:12.106545+0000) 2024-02-19T17:51:13.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:13.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:13.106829+0000) 2024-02-19T17:51:14.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:14.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:14.107026+0000) 2024-02-19T17:51:15.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:15.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:15.107219+0000) 2024-02-19T17:51:16.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:16.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:16.107387+0000) 2024-02-19T17:51:17.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:17.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:17.107525+0000) 2024-02-19T17:51:18.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:18.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:18.107727+0000) 2024-02-19T17:51:19.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:19.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:19.107924+0000) 2024-02-19T17:51:20.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:20.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:20.108133+0000) 2024-02-19T17:51:21.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:21.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:21.108335+0000) 2024-02-19T17:51:22.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:22.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:22.108588+0000) 2024-02-19T17:51:23.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:23.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:23.108829+0000) 2024-02-19T17:51:24.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:24.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:24.109018+0000) 2024-02-19T17:51:25.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:25.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:25.109186+0000) 2024-02-19T17:51:26.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:26.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:26.109412+0000) 2024-02-19T17:51:27.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:27.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:27.109649+0000) 2024-02-19T17:51:28.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:28.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:28.109898+0000) 2024-02-19T17:51:29.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:29.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:29.110140+0000) 2024-02-19T17:51:30.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:30.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:30.110321+0000) 2024-02-19T17:51:31.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:31.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:31.110589+0000) 2024-02-19T17:51:32.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:32.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:32.110774+0000) 2024-02-19T17:51:32.867 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:51:32.870 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:51:33.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:33.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:33.111020+0000) 2024-02-19T17:51:34.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:34.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:34.111255+0000) 2024-02-19T17:51:35.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:35.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:35.111512+0000) 2024-02-19T17:51:36.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:36.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:36.111770+0000) 2024-02-19T17:51:37.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:37.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:37.112026+0000) 2024-02-19T17:51:38.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:38.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:38.112264+0000) 2024-02-19T17:51:39.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:39.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:39.112458+0000) 2024-02-19T17:51:40.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:40.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:40.112705+0000) 2024-02-19T17:51:41.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:41.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:41.112947+0000) 2024-02-19T17:51:42.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:42.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:42.113186+0000) 2024-02-19T17:51:43.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:43.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:43.113440+0000) 2024-02-19T17:51:44.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:44.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:44.113701+0000) 2024-02-19T17:51:45.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:45.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:45.113872+0000) 2024-02-19T17:51:46.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:46.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:46.114125+0000) 2024-02-19T17:51:47.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:47.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:47.114364+0000) 2024-02-19T17:51:48.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:48.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:48.114581+0000) 2024-02-19T17:51:49.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:49.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:49.114864+0000) 2024-02-19T17:51:50.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:50.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:50.115139+0000) 2024-02-19T17:51:51.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:51.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:51.115373+0000) 2024-02-19T17:51:52.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:52.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:52.115610+0000) 2024-02-19T17:51:53.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:53.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:53.115837+0000) 2024-02-19T17:51:54.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:54.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:54.116095+0000) 2024-02-19T17:51:55.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:55.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:55.116303+0000) 2024-02-19T17:51:56.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:56.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:56.116570+0000) 2024-02-19T17:51:57.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:57.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:57.116803+0000) 2024-02-19T17:51:58.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:58.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:58.117049+0000) 2024-02-19T17:51:59.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:51:59.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:51:59.117266+0000) 2024-02-19T17:52:00.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:00.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:00.117514+0000) 2024-02-19T17:52:01.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:01.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:01.117715+0000) 2024-02-19T17:52:02.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:02.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:02.117917+0000) 2024-02-19T17:52:02.989 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:52:02.992 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:52:03.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:03.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:03.118155+0000) 2024-02-19T17:52:04.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:04.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:04.118394+0000) 2024-02-19T17:52:05.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:05.112+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:05.118628+0000) 2024-02-19T17:52:06.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:06.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:06.118891+0000) 2024-02-19T17:52:07.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:07.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:07.119090+0000) 2024-02-19T17:52:08.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:08.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:08.119280+0000) 2024-02-19T17:52:09.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:09.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:09.119512+0000) 2024-02-19T17:52:10.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:10.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:10.119768+0000) 2024-02-19T17:52:11.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:11.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:11.120017+0000) 2024-02-19T17:52:12.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:12.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:12.120239+0000) 2024-02-19T17:52:13.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:13.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:13.120474+0000) 2024-02-19T17:52:14.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:14.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:14.120652+0000) 2024-02-19T17:52:15.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:15.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:15.120879+0000) 2024-02-19T17:52:16.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:16.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:16.121142+0000) 2024-02-19T17:52:17.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:17.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:17.121402+0000) 2024-02-19T17:52:18.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:18.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:18.121638+0000) 2024-02-19T17:52:19.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:19.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:19.121839+0000) 2024-02-19T17:52:20.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:20.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:20.122090+0000) 2024-02-19T17:52:21.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:21.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:21.122285+0000) 2024-02-19T17:52:22.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:22.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:22.122512+0000) 2024-02-19T17:52:23.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:23.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:23.122789+0000) 2024-02-19T17:52:24.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:24.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:24.123016+0000) 2024-02-19T17:52:25.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:25.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:25.123281+0000) 2024-02-19T17:52:26.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:26.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:26.123580+0000) 2024-02-19T17:52:27.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:27.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:27.123802+0000) 2024-02-19T17:52:28.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:28.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:28.124079+0000) 2024-02-19T17:52:29.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:29.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:29.124289+0000) 2024-02-19T17:52:30.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:30.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:30.124453+0000) 2024-02-19T17:52:31.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:31.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:31.124609+0000) 2024-02-19T17:52:32.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:32.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:32.124771+0000) 2024-02-19T17:52:33.045 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:52:33.048 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:52:33.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:33.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:33.125004+0000) 2024-02-19T17:52:34.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:34.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:34.125219+0000) 2024-02-19T17:52:35.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:35.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:35.125435+0000) 2024-02-19T17:52:36.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:36.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:36.125612+0000) 2024-02-19T17:52:37.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:37.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:37.125886+0000) 2024-02-19T17:52:38.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:38.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:38.126122+0000) 2024-02-19T17:52:39.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:39.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:39.126341+0000) 2024-02-19T17:52:40.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:40.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:40.126524+0000) 2024-02-19T17:52:41.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:41.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:41.126811+0000) 2024-02-19T17:52:42.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:42.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:42.127038+0000) 2024-02-19T17:52:43.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:43.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:43.127175+0000) 2024-02-19T17:52:44.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:44.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:44.127401+0000) 2024-02-19T17:52:45.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:45.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:45.127626+0000) 2024-02-19T17:52:46.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:46.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:46.127833+0000) 2024-02-19T17:52:47.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:47.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:47.128015+0000) 2024-02-19T17:52:48.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:48.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:48.128245+0000) 2024-02-19T17:52:49.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:49.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:49.128456+0000) 2024-02-19T17:52:50.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:50.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:50.128629+0000) 2024-02-19T17:52:51.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:51.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:51.128873+0000) 2024-02-19T17:52:52.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:52.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:52.129114+0000) 2024-02-19T17:52:53.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:53.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:53.129395+0000) 2024-02-19T17:52:54.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:54.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:54.129648+0000) 2024-02-19T17:52:55.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:55.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:55.129871+0000) 2024-02-19T17:52:56.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:56.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:56.130076+0000) 2024-02-19T17:52:57.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:57.123+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:57.130257+0000) 2024-02-19T17:52:58.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:58.123+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:58.130485+0000) 2024-02-19T17:52:59.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:52:59.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:52:59.130747+0000) 2024-02-19T17:53:00.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:00.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:00.130946+0000) 2024-02-19T17:53:01.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:01.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:01.131156+0000) 2024-02-19T17:53:02.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:02.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:02.131417+0000) 2024-02-19T17:53:03.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:03.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:03.131685+0000) 2024-02-19T17:53:03.392 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:53:03.397 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:53:04.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:04.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:04.131932+0000) 2024-02-19T17:53:05.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:05.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:05.132167+0000) 2024-02-19T17:53:06.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:06.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:06.132373+0000) 2024-02-19T17:53:07.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:07.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:07.132564+0000) 2024-02-19T17:53:08.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:08.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:08.132807+0000) 2024-02-19T17:53:09.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:09.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:09.133079+0000) 2024-02-19T17:53:10.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:10.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:10.133284+0000) 2024-02-19T17:53:11.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:11.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:11.133482+0000) 2024-02-19T17:53:12.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:12.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:12.133735+0000) 2024-02-19T17:53:13.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:13.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:13.133941+0000) 2024-02-19T17:53:14.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:14.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:14.134122+0000) 2024-02-19T17:53:15.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:15.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:15.134262+0000) 2024-02-19T17:53:16.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:16.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:16.134483+0000) 2024-02-19T17:53:17.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:17.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:17.134654+0000) 2024-02-19T17:53:18.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:18.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:18.134898+0000) 2024-02-19T17:53:19.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:19.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:19.135050+0000) 2024-02-19T17:53:20.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:20.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:20.135305+0000) 2024-02-19T17:53:21.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:21.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:21.135478+0000) 2024-02-19T17:53:22.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:22.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:22.135707+0000) 2024-02-19T17:53:23.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:23.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:23.135977+0000) 2024-02-19T17:53:24.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:24.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:24.136215+0000) 2024-02-19T17:53:25.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:25.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:25.136330+0000) 2024-02-19T17:53:26.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:26.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:26.136557+0000) 2024-02-19T17:53:27.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:27.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:27.136793+0000) 2024-02-19T17:53:28.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:28.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:28.137065+0000) 2024-02-19T17:53:29.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:29.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:29.137258+0000) 2024-02-19T17:53:30.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:30.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:30.137421+0000) 2024-02-19T17:53:31.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:31.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:31.137677+0000) 2024-02-19T17:53:32.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:32.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:32.137902+0000) 2024-02-19T17:53:33.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:33.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:33.138098+0000) 2024-02-19T17:53:33.449 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:53:33.460 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:53:34.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:34.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:34.138360+0000) 2024-02-19T17:53:35.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:35.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:35.138632+0000) 2024-02-19T17:53:36.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:36.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:36.138880+0000) 2024-02-19T17:53:37.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:37.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:37.139130+0000) 2024-02-19T17:53:38.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:38.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:38.139366+0000) 2024-02-19T17:53:39.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:39.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:39.139601+0000) 2024-02-19T17:53:40.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:40.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:40.139776+0000) 2024-02-19T17:53:41.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:41.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:41.139978+0000) 2024-02-19T17:53:42.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:42.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:42.140225+0000) 2024-02-19T17:53:43.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:43.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:43.140424+0000) 2024-02-19T17:53:44.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:44.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:44.140673+0000) 2024-02-19T17:53:45.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:45.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:45.140902+0000) 2024-02-19T17:53:46.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:46.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:46.141161+0000) 2024-02-19T17:53:47.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:47.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:47.141389+0000) 2024-02-19T17:53:48.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:48.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:48.141583+0000) 2024-02-19T17:53:49.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:49.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:49.141815+0000) 2024-02-19T17:53:50.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:50.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:50.141983+0000) 2024-02-19T17:53:51.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:51.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:51.142160+0000) 2024-02-19T17:53:52.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:52.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:52.142402+0000) 2024-02-19T17:53:53.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:53.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:53.142595+0000) 2024-02-19T17:53:54.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:54.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:54.142775+0000) 2024-02-19T17:53:55.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:55.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:55.142959+0000) 2024-02-19T17:53:56.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:56.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:56.143212+0000) 2024-02-19T17:53:57.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:57.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:57.143477+0000) 2024-02-19T17:53:58.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:58.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:58.143719+0000) 2024-02-19T17:53:59.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:53:59.138+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:53:59.143947+0000) 2024-02-19T17:54:00.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:00.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:00.144125+0000) 2024-02-19T17:54:01.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:01.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:01.144369+0000) 2024-02-19T17:54:02.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:02.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:02.144614+0000) 2024-02-19T17:54:03.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:03.138+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:03.144859+0000) 2024-02-19T17:54:03.578 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:54:03.582 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:54:04.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:04.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:04.145046+0000) 2024-02-19T17:54:05.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:05.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:05.145285+0000) 2024-02-19T17:54:06.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:06.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:06.145511+0000) 2024-02-19T17:54:07.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:07.138+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:07.145745+0000) 2024-02-19T17:54:08.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:08.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:08.145922+0000) 2024-02-19T17:54:09.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:09.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:09.146193+0000) 2024-02-19T17:54:10.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:10.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:10.146436+0000) 2024-02-19T17:54:11.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:11.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:11.146629+0000) 2024-02-19T17:54:12.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:12.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:12.146874+0000) 2024-02-19T17:54:13.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:13.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:13.147124+0000) 2024-02-19T17:54:14.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:14.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:14.147374+0000) 2024-02-19T17:54:15.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:15.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:15.147548+0000) 2024-02-19T17:54:16.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:16.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:16.147786+0000) 2024-02-19T17:54:17.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:17.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:17.148059+0000) 2024-02-19T17:54:18.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:18.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:18.148306+0000) 2024-02-19T17:54:19.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:19.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:19.148483+0000) 2024-02-19T17:54:20.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:20.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:20.148724+0000) 2024-02-19T17:54:21.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:21.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:21.148974+0000) 2024-02-19T17:54:22.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:22.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:22.149220+0000) 2024-02-19T17:54:23.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:23.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:23.149422+0000) 2024-02-19T17:54:24.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:24.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:24.149659+0000) 2024-02-19T17:54:25.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:25.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:25.149901+0000) 2024-02-19T17:54:26.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:26.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:26.150151+0000) 2024-02-19T17:54:27.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:27.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:27.150323+0000) 2024-02-19T17:54:28.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:28.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:28.150572+0000) 2024-02-19T17:54:29.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:29.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:29.150784+0000) 2024-02-19T17:54:30.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:30.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:30.150957+0000) 2024-02-19T17:54:31.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:31.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:31.151130+0000) 2024-02-19T17:54:32.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:32.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:32.151387+0000) 2024-02-19T17:54:33.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:33.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:33.151623+0000) 2024-02-19T17:54:33.708 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:54:33.711 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:54:34.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:34.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:34.151879+0000) 2024-02-19T17:54:35.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:35.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:35.152131+0000) 2024-02-19T17:54:36.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:36.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:36.152313+0000) 2024-02-19T17:54:37.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:37.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:37.152558+0000) 2024-02-19T17:54:38.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:38.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:38.152808+0000) 2024-02-19T17:54:39.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:39.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:39.153046+0000) 2024-02-19T17:54:40.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:40.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:40.153271+0000) 2024-02-19T17:54:41.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:41.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:41.153515+0000) 2024-02-19T17:54:42.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:42.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:42.153719+0000) 2024-02-19T17:54:43.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:43.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:43.153899+0000) 2024-02-19T17:54:44.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:44.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:44.154134+0000) 2024-02-19T17:54:45.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:45.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:45.154380+0000) 2024-02-19T17:54:46.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:46.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:46.154587+0000) 2024-02-19T17:54:47.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:47.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:47.154865+0000) 2024-02-19T17:54:48.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:48.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:48.155115+0000) 2024-02-19T17:54:49.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:49.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:49.155304+0000) 2024-02-19T17:54:50.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:50.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:50.155509+0000) 2024-02-19T17:54:51.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:51.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:51.155754+0000) 2024-02-19T17:54:52.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:52.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:52.155987+0000) 2024-02-19T17:54:53.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:53.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:53.156246+0000) 2024-02-19T17:54:54.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:54.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:54.156537+0000) 2024-02-19T17:54:55.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:55.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:55.156772+0000) 2024-02-19T17:54:56.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:56.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:56.157031+0000) 2024-02-19T17:54:57.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:57.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:57.157281+0000) 2024-02-19T17:54:58.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:58.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:58.157526+0000) 2024-02-19T17:54:59.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:54:59.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:54:59.157753+0000) 2024-02-19T17:55:00.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:00.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:00.158000+0000) 2024-02-19T17:55:01.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:01.153+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:01.158260+0000) 2024-02-19T17:55:02.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:02.153+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:02.158521+0000) 2024-02-19T17:55:03.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:03.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:03.158819+0000) 2024-02-19T17:55:03.767 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:55:03.772 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:55:04.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:04.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:04.159003+0000) 2024-02-19T17:55:05.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:05.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:05.159269+0000) 2024-02-19T17:55:06.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:06.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:06.159516+0000) 2024-02-19T17:55:07.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:07.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:07.159761+0000) 2024-02-19T17:55:08.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:08.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:08.160007+0000) 2024-02-19T17:55:09.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:09.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:09.160208+0000) 2024-02-19T17:55:10.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:10.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:10.160442+0000) 2024-02-19T17:55:11.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:11.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:11.160611+0000) 2024-02-19T17:55:12.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:12.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:12.160863+0000) 2024-02-19T17:55:13.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:13.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:13.161098+0000) 2024-02-19T17:55:14.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:14.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:14.161347+0000) 2024-02-19T17:55:15.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:15.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:15.161550+0000) 2024-02-19T17:55:16.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:16.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:16.161830+0000) 2024-02-19T17:55:17.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:17.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:17.162058+0000) 2024-02-19T17:55:18.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:18.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:18.162298+0000) 2024-02-19T17:55:19.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:19.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:19.162483+0000) 2024-02-19T17:55:20.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:20.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:20.162702+0000) 2024-02-19T17:55:21.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:21.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:21.162925+0000) 2024-02-19T17:55:22.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:22.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:22.163159+0000) 2024-02-19T17:55:23.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:23.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:23.163345+0000) 2024-02-19T17:55:24.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:24.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:24.163574+0000) 2024-02-19T17:55:25.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:25.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:25.163751+0000) 2024-02-19T17:55:26.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:26.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:26.164033+0000) 2024-02-19T17:55:27.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:27.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:27.164228+0000) 2024-02-19T17:55:28.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:28.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:28.164504+0000) 2024-02-19T17:55:29.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:29.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:29.164687+0000) 2024-02-19T17:55:30.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:30.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:30.164938+0000) 2024-02-19T17:55:31.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:31.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:31.165177+0000) 2024-02-19T17:55:32.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:32.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:32.165348+0000) 2024-02-19T17:55:33.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:33.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:33.165589+0000) 2024-02-19T17:55:33.829 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:55:33.832 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:55:34.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:34.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:34.165776+0000) 2024-02-19T17:55:35.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:35.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:35.166037+0000) 2024-02-19T17:55:36.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:36.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:36.166238+0000) 2024-02-19T17:55:37.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:37.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:37.166473+0000) 2024-02-19T17:55:38.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:38.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:38.166802+0000) 2024-02-19T17:55:39.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:39.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:39.166959+0000) 2024-02-19T17:55:40.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:40.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:40.167147+0000) 2024-02-19T17:55:41.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:41.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:41.167400+0000) 2024-02-19T17:55:42.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:42.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:42.167575+0000) 2024-02-19T17:55:43.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:43.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:43.167830+0000) 2024-02-19T17:55:44.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:44.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:44.168109+0000) 2024-02-19T17:55:45.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:45.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:45.168396+0000) 2024-02-19T17:55:46.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:46.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:46.168650+0000) 2024-02-19T17:55:47.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:47.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:47.168844+0000) 2024-02-19T17:55:48.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:48.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:48.168989+0000) 2024-02-19T17:55:49.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:49.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:49.169244+0000) 2024-02-19T17:55:50.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:50.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:50.169471+0000) 2024-02-19T17:55:51.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:51.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:51.169698+0000) 2024-02-19T17:55:52.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:52.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:52.169956+0000) 2024-02-19T17:55:53.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:53.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:53.170140+0000) 2024-02-19T17:55:54.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:54.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:54.170395+0000) 2024-02-19T17:55:55.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:55.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:55.170591+0000) 2024-02-19T17:55:56.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:56.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:56.170878+0000) 2024-02-19T17:55:57.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:57.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:57.171056+0000) 2024-02-19T17:55:58.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:58.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:58.171296+0000) 2024-02-19T17:55:59.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:55:59.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:55:59.171524+0000) 2024-02-19T17:56:00.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:00.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:00.171774+0000) 2024-02-19T17:56:01.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:01.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:01.172028+0000) 2024-02-19T17:56:02.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:02.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:02.172229+0000) 2024-02-19T17:56:03.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:03.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:03.172415+0000) 2024-02-19T17:56:03.907 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:56:03.910 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:56:04.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:04.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:04.172719+0000) 2024-02-19T17:56:05.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:05.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:05.172956+0000) 2024-02-19T17:56:06.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:06.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:06.173164+0000) 2024-02-19T17:56:07.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:07.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:07.173431+0000) 2024-02-19T17:56:08.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:08.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:08.173655+0000) 2024-02-19T17:56:09.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:09.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:09.173905+0000) 2024-02-19T17:56:10.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:10.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:10.174154+0000) 2024-02-19T17:56:11.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:11.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:11.174386+0000) 2024-02-19T17:56:12.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:12.168+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:12.174559+0000) 2024-02-19T17:56:13.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:13.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:13.174760+0000) 2024-02-19T17:56:14.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:14.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:14.174950+0000) 2024-02-19T17:56:15.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:15.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:15.175142+0000) 2024-02-19T17:56:16.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:16.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:16.175372+0000) 2024-02-19T17:56:17.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:17.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:17.175599+0000) 2024-02-19T17:56:18.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:18.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:18.175848+0000) 2024-02-19T17:56:19.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:19.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:19.176042+0000) 2024-02-19T17:56:20.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:20.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:20.176290+0000) 2024-02-19T17:56:21.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:21.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:21.176515+0000) 2024-02-19T17:56:22.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:22.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:22.176757+0000) 2024-02-19T17:56:23.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:23.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:23.176995+0000) 2024-02-19T17:56:24.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:24.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:24.177252+0000) 2024-02-19T17:56:25.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:25.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:25.177516+0000) 2024-02-19T17:56:26.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:26.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:26.177702+0000) 2024-02-19T17:56:27.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:27.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:27.177901+0000) 2024-02-19T17:56:28.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:28.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:28.178080+0000) 2024-02-19T17:56:29.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:29.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:29.178255+0000) 2024-02-19T17:56:30.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:30.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:30.178479+0000) 2024-02-19T17:56:31.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:31.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:31.178739+0000) 2024-02-19T17:56:32.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:32.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:32.178929+0000) 2024-02-19T17:56:33.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:33.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:33.179125+0000) 2024-02-19T17:56:34.037 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:56:34.041 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:56:34.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:34.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:34.179397+0000) 2024-02-19T17:56:35.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:35.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:35.179533+0000) 2024-02-19T17:56:36.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:36.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:36.179795+0000) 2024-02-19T17:56:37.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:37.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:37.180048+0000) 2024-02-19T17:56:38.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:38.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:38.180294+0000) 2024-02-19T17:56:39.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:39.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:39.180493+0000) 2024-02-19T17:56:40.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:40.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:40.180738+0000) 2024-02-19T17:56:41.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:41.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:41.180983+0000) 2024-02-19T17:56:42.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:42.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:42.181199+0000) 2024-02-19T17:56:43.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:43.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:43.181477+0000) 2024-02-19T17:56:44.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:44.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:44.181716+0000) 2024-02-19T17:56:45.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:45.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:45.181886+0000) 2024-02-19T17:56:46.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:46.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:46.182163+0000) 2024-02-19T17:56:47.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:47.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:47.182407+0000) 2024-02-19T17:56:48.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:48.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:48.182652+0000) 2024-02-19T17:56:49.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:49.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:49.182848+0000) 2024-02-19T17:56:50.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:50.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:50.183122+0000) 2024-02-19T17:56:51.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:51.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:51.183314+0000) 2024-02-19T17:56:52.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:52.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:52.183561+0000) 2024-02-19T17:56:53.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:53.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:53.183802+0000) 2024-02-19T17:56:54.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:54.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:54.184002+0000) 2024-02-19T17:56:55.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:55.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:55.184243+0000) 2024-02-19T17:56:56.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:56.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:56.184504+0000) 2024-02-19T17:56:57.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:57.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:57.184703+0000) 2024-02-19T17:56:58.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:58.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:58.184979+0000) 2024-02-19T17:56:59.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:56:59.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:56:59.185217+0000) 2024-02-19T17:57:00.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:00.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:00.185430+0000) 2024-02-19T17:57:01.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:01.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:01.185665+0000) 2024-02-19T17:57:02.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:02.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:02.185873+0000) 2024-02-19T17:57:03.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:03.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:03.186064+0000) 2024-02-19T17:57:04.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:04.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:04.186330+0000) 2024-02-19T17:57:04.167 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:57:04.172 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:57:05.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:05.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:05.186564+0000) 2024-02-19T17:57:06.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:06.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:06.186785+0000) 2024-02-19T17:57:07.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:07.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:07.186965+0000) 2024-02-19T17:57:08.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:08.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:08.187222+0000) 2024-02-19T17:57:09.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:09.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:09.187455+0000) 2024-02-19T17:57:10.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:10.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:10.187693+0000) 2024-02-19T17:57:11.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:11.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:11.187889+0000) 2024-02-19T17:57:12.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:12.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:12.188138+0000) 2024-02-19T17:57:13.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:13.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:13.188400+0000) 2024-02-19T17:57:14.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:14.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:14.188593+0000) 2024-02-19T17:57:15.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:15.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:15.188837+0000) 2024-02-19T17:57:16.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:16.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:16.189021+0000) 2024-02-19T17:57:17.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:17.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:17.189217+0000) 2024-02-19T17:57:18.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:18.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:18.189420+0000) 2024-02-19T17:57:19.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:19.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:19.189689+0000) 2024-02-19T17:57:20.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:20.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:20.189877+0000) 2024-02-19T17:57:21.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:21.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:21.190119+0000) 2024-02-19T17:57:22.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:22.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:22.190383+0000) 2024-02-19T17:57:23.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:23.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:23.190624+0000) 2024-02-19T17:57:24.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:24.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:24.190820+0000) 2024-02-19T17:57:25.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:25.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:25.191058+0000) 2024-02-19T17:57:26.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:26.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:26.191317+0000) 2024-02-19T17:57:27.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:27.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:27.191558+0000) 2024-02-19T17:57:28.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:28.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:28.191820+0000) 2024-02-19T17:57:29.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:29.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:29.191991+0000) 2024-02-19T17:57:30.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:30.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:30.192248+0000) 2024-02-19T17:57:31.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:31.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:31.192489+0000) 2024-02-19T17:57:32.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:32.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:32.192738+0000) 2024-02-19T17:57:33.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:33.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:33.192980+0000) 2024-02-19T17:57:34.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:34.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:34.193184+0000) 2024-02-19T17:57:34.264 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:57:34.267 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:57:35.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:35.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:35.193435+0000) 2024-02-19T17:57:36.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:36.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:36.193600+0000) 2024-02-19T17:57:37.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:37.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:37.193760+0000) 2024-02-19T17:57:38.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:38.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:38.193948+0000) 2024-02-19T17:57:39.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:39.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:39.194189+0000) 2024-02-19T17:57:40.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:40.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:40.194453+0000) 2024-02-19T17:57:41.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:41.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:41.194592+0000) 2024-02-19T17:57:42.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:42.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:42.194773+0000) 2024-02-19T17:57:43.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:43.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:43.194938+0000) 2024-02-19T17:57:44.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:44.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:44.195174+0000) 2024-02-19T17:57:45.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:45.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:45.195409+0000) 2024-02-19T17:57:46.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:46.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:46.195617+0000) 2024-02-19T17:57:47.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:47.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:47.195871+0000) 2024-02-19T17:57:48.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:48.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:48.196135+0000) 2024-02-19T17:57:49.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:49.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:49.196418+0000) 2024-02-19T17:57:50.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:50.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:50.196591+0000) 2024-02-19T17:57:51.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:51.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:51.196768+0000) 2024-02-19T17:57:52.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:52.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:52.196972+0000) 2024-02-19T17:57:53.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:53.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:53.197179+0000) 2024-02-19T17:57:54.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:54.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:54.197423+0000) 2024-02-19T17:57:55.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:55.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:55.197678+0000) 2024-02-19T17:57:56.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:56.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:56.197996+0000) 2024-02-19T17:57:57.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:57.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:57.198323+0000) 2024-02-19T17:57:58.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:58.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:58.198614+0000) 2024-02-19T17:57:59.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:57:59.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:57:59.198895+0000) 2024-02-19T17:58:00.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:00.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:00.199104+0000) 2024-02-19T17:58:01.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:01.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:01.199309+0000) 2024-02-19T17:58:02.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:02.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:02.199541+0000) 2024-02-19T17:58:03.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:03.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:03.199841+0000) 2024-02-19T17:58:04.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:04.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:04.200134+0000) 2024-02-19T17:58:04.346 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:58:04.349 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:58:05.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:05.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:05.200416+0000) 2024-02-19T17:58:06.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:06.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:06.200678+0000) 2024-02-19T17:58:07.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:07.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:07.200875+0000) 2024-02-19T17:58:08.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:08.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:08.201139+0000) 2024-02-19T17:58:09.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:09.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:09.201429+0000) 2024-02-19T17:58:10.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:10.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:10.201741+0000) 2024-02-19T17:58:11.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:11.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:11.202009+0000) 2024-02-19T17:58:12.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:12.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:12.202291+0000) 2024-02-19T17:58:13.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:13.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:13.202582+0000) 2024-02-19T17:58:14.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:14.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:14.202803+0000) 2024-02-19T17:58:15.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:15.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:15.203073+0000) 2024-02-19T17:58:16.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:16.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:16.203321+0000) 2024-02-19T17:58:17.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:17.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:17.203627+0000) 2024-02-19T17:58:18.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:18.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:18.203917+0000) 2024-02-19T17:58:19.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:19.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:19.204152+0000) 2024-02-19T17:58:20.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:20.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:20.204332+0000) 2024-02-19T17:58:21.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:21.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:21.204506+0000) 2024-02-19T17:58:22.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:22.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:22.204806+0000) 2024-02-19T17:58:23.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:23.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:23.205022+0000) 2024-02-19T17:58:24.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:24.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:24.205248+0000) 2024-02-19T17:58:25.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:25.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:25.205411+0000) 2024-02-19T17:58:26.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:26.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:26.205668+0000) 2024-02-19T17:58:27.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:27.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:27.205931+0000) 2024-02-19T17:58:28.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:28.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:28.206114+0000) 2024-02-19T17:58:29.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:29.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:29.206359+0000) 2024-02-19T17:58:30.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:30.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:30.206577+0000) 2024-02-19T17:58:31.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:31.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:31.206755+0000) 2024-02-19T17:58:32.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:32.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:32.206945+0000) 2024-02-19T17:58:33.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:33.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:33.207229+0000) 2024-02-19T17:58:34.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:34.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:34.207398+0000) 2024-02-19T17:58:34.493 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:58:34.496 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:58:35.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:35.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:35.207525+0000) 2024-02-19T17:58:36.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:36.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:36.207756+0000) 2024-02-19T17:58:37.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:37.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:37.207978+0000) 2024-02-19T17:58:38.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:38.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:38.208252+0000) 2024-02-19T17:58:39.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:39.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:39.208432+0000) 2024-02-19T17:58:40.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:40.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:40.208692+0000) 2024-02-19T17:58:41.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:41.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:41.208976+0000) 2024-02-19T17:58:42.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:42.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:42.209227+0000) 2024-02-19T17:58:43.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:43.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:43.209487+0000) 2024-02-19T17:58:44.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:44.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:44.209661+0000) 2024-02-19T17:58:45.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:45.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:45.209783+0000) 2024-02-19T17:58:46.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:46.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:46.210072+0000) 2024-02-19T17:58:47.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:47.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:47.210313+0000) 2024-02-19T17:58:48.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:48.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:48.210548+0000) 2024-02-19T17:58:49.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:49.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:49.210815+0000) 2024-02-19T17:58:50.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:50.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:50.211011+0000) 2024-02-19T17:58:51.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:51.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:51.211186+0000) 2024-02-19T17:58:52.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:52.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:52.211394+0000) 2024-02-19T17:58:53.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:53.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:53.211678+0000) 2024-02-19T17:58:54.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:54.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:54.211868+0000) 2024-02-19T17:58:55.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:55.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:55.212016+0000) 2024-02-19T17:58:56.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:56.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:56.212292+0000) 2024-02-19T17:58:57.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:57.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:57.212512+0000) 2024-02-19T17:58:58.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:58.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:58.212753+0000) 2024-02-19T17:58:59.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:58:59.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:58:59.213022+0000) 2024-02-19T17:59:00.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:00.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:00.213227+0000) 2024-02-19T17:59:01.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:01.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:01.213422+0000) 2024-02-19T17:59:02.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:02.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:02.213633+0000) 2024-02-19T17:59:03.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:03.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:03.213772+0000) 2024-02-19T17:59:04.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:04.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:04.213953+0000) 2024-02-19T17:59:04.622 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:59:04.626 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:59:05.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:05.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:05.214195+0000) 2024-02-19T17:59:06.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:06.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:06.214438+0000) 2024-02-19T17:59:07.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:07.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:07.214671+0000) 2024-02-19T17:59:08.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:08.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:08.214899+0000) 2024-02-19T17:59:09.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:09.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:09.215077+0000) 2024-02-19T17:59:10.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:10.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:10.215222+0000) 2024-02-19T17:59:11.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:11.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:11.215408+0000) 2024-02-19T17:59:12.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:12.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:12.215538+0000) 2024-02-19T17:59:13.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:13.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:13.215695+0000) 2024-02-19T17:59:14.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:14.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:14.215902+0000) 2024-02-19T17:59:15.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:15.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:15.216158+0000) 2024-02-19T17:59:16.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:16.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:16.216423+0000) 2024-02-19T17:59:17.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:17.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:17.216690+0000) 2024-02-19T17:59:18.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:18.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:18.216928+0000) 2024-02-19T17:59:19.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:19.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:19.217107+0000) 2024-02-19T17:59:20.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:20.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:20.217314+0000) 2024-02-19T17:59:21.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:21.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:21.217557+0000) 2024-02-19T17:59:22.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:22.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:22.217814+0000) 2024-02-19T17:59:23.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:23.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:23.218014+0000) 2024-02-19T17:59:24.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:24.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:24.218198+0000) 2024-02-19T17:59:25.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:25.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:25.218363+0000) 2024-02-19T17:59:26.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:26.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:26.218679+0000) 2024-02-19T17:59:27.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:27.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:27.218847+0000) 2024-02-19T17:59:28.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:28.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:28.219012+0000) 2024-02-19T17:59:29.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:29.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:29.219205+0000) 2024-02-19T17:59:30.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:30.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:30.219384+0000) 2024-02-19T17:59:31.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:31.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:31.219627+0000) 2024-02-19T17:59:32.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:32.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:32.219879+0000) 2024-02-19T17:59:33.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:33.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:33.220094+0000) 2024-02-19T17:59:34.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:34.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:34.220340+0000) 2024-02-19T17:59:34.753 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:59:34.756 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T17:59:35.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:35.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:35.220574+0000) 2024-02-19T17:59:36.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:36.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:36.220827+0000) 2024-02-19T17:59:37.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:37.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:37.221064+0000) 2024-02-19T17:59:38.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:38.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:38.221291+0000) 2024-02-19T17:59:39.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:39.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:39.221571+0000) 2024-02-19T17:59:40.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:40.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:40.221846+0000) 2024-02-19T17:59:41.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:41.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:41.222106+0000) 2024-02-19T17:59:42.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:42.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:42.222370+0000) 2024-02-19T17:59:43.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:43.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:43.222605+0000) 2024-02-19T17:59:44.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:44.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:44.222842+0000) 2024-02-19T17:59:45.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:45.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:45.223045+0000) 2024-02-19T17:59:46.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:46.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:46.223294+0000) 2024-02-19T17:59:47.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:47.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:47.223542+0000) 2024-02-19T17:59:48.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:48.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:48.223782+0000) 2024-02-19T17:59:49.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:49.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:49.223981+0000) 2024-02-19T17:59:50.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:50.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:50.224210+0000) 2024-02-19T17:59:51.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:51.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:51.224409+0000) 2024-02-19T17:59:52.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:52.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:52.224585+0000) 2024-02-19T17:59:53.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:53.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:53.224762+0000) 2024-02-19T17:59:54.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:54.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:54.225035+0000) 2024-02-19T17:59:55.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:55.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:55.225239+0000) 2024-02-19T17:59:56.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:56.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:56.225423+0000) 2024-02-19T17:59:57.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:57.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:57.225662+0000) 2024-02-19T17:59:58.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:58.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:58.225851+0000) 2024-02-19T17:59:59.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T17:59:59.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T16:59:59.226263+0000) 2024-02-19T18:00:00.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:00.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:00.226512+0000) 2024-02-19T18:00:01.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:01.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:01.226800+0000) 2024-02-19T18:00:02.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:02.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:02.227031+0000) 2024-02-19T18:00:03.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:03.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:03.227175+0000) 2024-02-19T18:00:04.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:04.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:04.227310+0000) 2024-02-19T18:00:04.883 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:00:04.887 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:00:05.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:05.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:05.227504+0000) 2024-02-19T18:00:06.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:06.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:06.227859+0000) 2024-02-19T18:00:07.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:07.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:07.228144+0000) 2024-02-19T18:00:08.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:08.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:08.228425+0000) 2024-02-19T18:00:09.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:09.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:09.228714+0000) 2024-02-19T18:00:10.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:10.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:10.228926+0000) 2024-02-19T18:00:11.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:11.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:11.229227+0000) 2024-02-19T18:00:12.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:12.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:12.229426+0000) 2024-02-19T18:00:13.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:13.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:13.229710+0000) 2024-02-19T18:00:14.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:14.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:14.230004+0000) 2024-02-19T18:00:15.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:15.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:15.230218+0000) 2024-02-19T18:00:16.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:16.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:16.230457+0000) 2024-02-19T18:00:17.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:17.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:17.230674+0000) 2024-02-19T18:00:18.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:18.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:18.230946+0000) 2024-02-19T18:00:19.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:19.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:19.231264+0000) 2024-02-19T18:00:20.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:20.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:20.231580+0000) 2024-02-19T18:00:21.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:21.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:21.231751+0000) 2024-02-19T18:00:22.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:22.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:22.231937+0000) 2024-02-19T18:00:23.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:23.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:23.232163+0000) 2024-02-19T18:00:24.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:24.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:24.232405+0000) 2024-02-19T18:00:25.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:25.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:25.232643+0000) 2024-02-19T18:00:26.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:26.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:26.232892+0000) 2024-02-19T18:00:27.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:27.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:27.233088+0000) 2024-02-19T18:00:28.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:28.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:28.233331+0000) 2024-02-19T18:00:29.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:29.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:29.233540+0000) 2024-02-19T18:00:30.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:30.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:30.233694+0000) 2024-02-19T18:00:31.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:31.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:31.233916+0000) 2024-02-19T18:00:32.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:32.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:32.234149+0000) 2024-02-19T18:00:33.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:33.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:33.234384+0000) 2024-02-19T18:00:34.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:34.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:34.234621+0000) 2024-02-19T18:00:34.974 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:00:34.977 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:00:35.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:35.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:35.234884+0000) 2024-02-19T18:00:36.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:36.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:36.235144+0000) 2024-02-19T18:00:37.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:37.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:37.235270+0000) 2024-02-19T18:00:38.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:38.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:38.235459+0000) 2024-02-19T18:00:39.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:39.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:39.235701+0000) 2024-02-19T18:00:40.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:40.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:40.235946+0000) 2024-02-19T18:00:41.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:41.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:41.236183+0000) 2024-02-19T18:00:42.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:42.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:42.236424+0000) 2024-02-19T18:00:43.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:43.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:43.236620+0000) 2024-02-19T18:00:44.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:44.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:44.236852+0000) 2024-02-19T18:00:45.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:45.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:45.237037+0000) 2024-02-19T18:00:46.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:46.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:46.237188+0000) 2024-02-19T18:00:47.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:47.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:47.237383+0000) 2024-02-19T18:00:48.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:48.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:48.237647+0000) 2024-02-19T18:00:49.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:49.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:49.237841+0000) 2024-02-19T18:00:50.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:50.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:50.238051+0000) 2024-02-19T18:00:51.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:51.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:51.238291+0000) 2024-02-19T18:00:52.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:52.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:52.238528+0000) 2024-02-19T18:00:53.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:53.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:53.238694+0000) 2024-02-19T18:00:54.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:54.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:54.238849+0000) 2024-02-19T18:00:55.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:55.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:55.239080+0000) 2024-02-19T18:00:56.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:56.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:56.239394+0000) 2024-02-19T18:00:57.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:57.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:57.239637+0000) 2024-02-19T18:00:58.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:58.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:58.239809+0000) 2024-02-19T18:00:59.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:00:59.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:00:59.239990+0000) 2024-02-19T18:01:00.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:00.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:00.240252+0000) 2024-02-19T18:01:01.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:01.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:01.240487+0000) 2024-02-19T18:01:02.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:02.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:02.240728+0000) 2024-02-19T18:01:03.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:03.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:03.240890+0000) 2024-02-19T18:01:04.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:04.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:04.241139+0000) 2024-02-19T18:01:05.121 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:01:05.126 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:01:05.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:05.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:05.241401+0000) 2024-02-19T18:01:06.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:06.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:06.241684+0000) 2024-02-19T18:01:07.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:07.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:07.241860+0000) 2024-02-19T18:01:08.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:08.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:08.242085+0000) 2024-02-19T18:01:09.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:09.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:09.242334+0000) 2024-02-19T18:01:10.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:10.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:10.242544+0000) 2024-02-19T18:01:11.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:11.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:11.242777+0000) 2024-02-19T18:01:12.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:12.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:12.243021+0000) 2024-02-19T18:01:13.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:13.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:13.243270+0000) 2024-02-19T18:01:14.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:14.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:14.243457+0000) 2024-02-19T18:01:15.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:15.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:15.243708+0000) 2024-02-19T18:01:16.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:16.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:16.243954+0000) 2024-02-19T18:01:17.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:17.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:17.244124+0000) 2024-02-19T18:01:18.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:18.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:18.244322+0000) 2024-02-19T18:01:19.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:19.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:19.244566+0000) 2024-02-19T18:01:20.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:20.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:20.244752+0000) 2024-02-19T18:01:21.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:21.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:21.245028+0000) 2024-02-19T18:01:22.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:22.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:22.245238+0000) 2024-02-19T18:01:23.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:23.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:23.245480+0000) 2024-02-19T18:01:24.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:24.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:24.245713+0000) 2024-02-19T18:01:25.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:25.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:25.245895+0000) 2024-02-19T18:01:26.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:26.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:26.246074+0000) 2024-02-19T18:01:27.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:27.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:27.246300+0000) 2024-02-19T18:01:28.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:28.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:28.246544+0000) 2024-02-19T18:01:29.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:29.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:29.246780+0000) 2024-02-19T18:01:30.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:30.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:30.246918+0000) 2024-02-19T18:01:31.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:31.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:31.247162+0000) 2024-02-19T18:01:32.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:32.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:32.247399+0000) 2024-02-19T18:01:33.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:33.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:33.247594+0000) 2024-02-19T18:01:34.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:34.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:34.247814+0000) 2024-02-19T18:01:35.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:35.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:35.248073+0000) 2024-02-19T18:01:35.251 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:01:35.255 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:01:36.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:36.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:36.248337+0000) 2024-02-19T18:01:37.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:37.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:37.248588+0000) 2024-02-19T18:01:38.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:38.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:38.248843+0000) 2024-02-19T18:01:39.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:39.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:39.249041+0000) 2024-02-19T18:01:40.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:40.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:40.249280+0000) 2024-02-19T18:01:41.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:41.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:41.249516+0000) 2024-02-19T18:01:42.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:42.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:42.249702+0000) 2024-02-19T18:01:43.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:43.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:43.249924+0000) 2024-02-19T18:01:44.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:44.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:44.250169+0000) 2024-02-19T18:01:45.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:45.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:45.250407+0000) 2024-02-19T18:01:46.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:46.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:46.250659+0000) 2024-02-19T18:01:47.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:47.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:47.250971+0000) 2024-02-19T18:01:48.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:48.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:48.251210+0000) 2024-02-19T18:01:49.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:49.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:49.251448+0000) 2024-02-19T18:01:50.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:50.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:50.251576+0000) 2024-02-19T18:01:51.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:51.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:51.251746+0000) 2024-02-19T18:01:52.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:52.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:52.252021+0000) 2024-02-19T18:01:53.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:53.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:53.252259+0000) 2024-02-19T18:01:54.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:54.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:54.252538+0000) 2024-02-19T18:01:55.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:55.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:55.252790+0000) 2024-02-19T18:01:56.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:56.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:56.253041+0000) 2024-02-19T18:01:57.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:57.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:57.253239+0000) 2024-02-19T18:01:58.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:58.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:58.253372+0000) 2024-02-19T18:01:59.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:01:59.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:01:59.253618+0000) 2024-02-19T18:02:00.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:00.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:00.253857+0000) 2024-02-19T18:02:01.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:01.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:01.254132+0000) 2024-02-19T18:02:02.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:02.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:02.254359+0000) 2024-02-19T18:02:03.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:03.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:03.254533+0000) 2024-02-19T18:02:04.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:04.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:04.254753+0000) 2024-02-19T18:02:05.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:05.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:05.254943+0000) 2024-02-19T18:02:05.347 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:02:05.351 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:02:06.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:06.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:06.255232+0000) 2024-02-19T18:02:07.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:07.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:07.255484+0000) 2024-02-19T18:02:08.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:08.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:08.255729+0000) 2024-02-19T18:02:09.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:09.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:09.255960+0000) 2024-02-19T18:02:10.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:10.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:10.256224+0000) 2024-02-19T18:02:11.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:11.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:11.256490+0000) 2024-02-19T18:02:12.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:12.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:12.256698+0000) 2024-02-19T18:02:13.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:13.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:13.256895+0000) 2024-02-19T18:02:14.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:14.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:14.257157+0000) 2024-02-19T18:02:15.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:15.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:15.257417+0000) 2024-02-19T18:02:16.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:16.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:16.257629+0000) 2024-02-19T18:02:17.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:17.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:17.257804+0000) 2024-02-19T18:02:18.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:18.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:18.258048+0000) 2024-02-19T18:02:19.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:19.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:19.258230+0000) 2024-02-19T18:02:20.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:20.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:20.258460+0000) 2024-02-19T18:02:21.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:21.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:21.258624+0000) 2024-02-19T18:02:22.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:22.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:22.258869+0000) 2024-02-19T18:02:23.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:23.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:23.259064+0000) 2024-02-19T18:02:24.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:24.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:24.259305+0000) 2024-02-19T18:02:25.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:25.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:25.259546+0000) 2024-02-19T18:02:26.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:26.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:26.259689+0000) 2024-02-19T18:02:27.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:27.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:27.259922+0000) 2024-02-19T18:02:28.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:28.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:28.260164+0000) 2024-02-19T18:02:29.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:29.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:29.260411+0000) 2024-02-19T18:02:30.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:30.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:30.260674+0000) 2024-02-19T18:02:31.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:31.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:31.260898+0000) 2024-02-19T18:02:32.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:32.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:32.261103+0000) 2024-02-19T18:02:33.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:33.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:33.261300+0000) 2024-02-19T18:02:34.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:34.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:34.261514+0000) 2024-02-19T18:02:35.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:35.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:35.261751+0000) 2024-02-19T18:02:35.403 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:02:35.407 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:02:36.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:36.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:36.261996+0000) 2024-02-19T18:02:37.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:37.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:37.262236+0000) 2024-02-19T18:02:38.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:38.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:38.262421+0000) 2024-02-19T18:02:39.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:39.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:39.262702+0000) 2024-02-19T18:02:40.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:40.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:40.262936+0000) 2024-02-19T18:02:41.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:41.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:41.263152+0000) 2024-02-19T18:02:42.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:42.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:42.263386+0000) 2024-02-19T18:02:43.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:43.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:43.263641+0000) 2024-02-19T18:02:44.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:44.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:44.263886+0000) 2024-02-19T18:02:45.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:45.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:45.264119+0000) 2024-02-19T18:02:46.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:46.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:46.264353+0000) 2024-02-19T18:02:47.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:47.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:47.264582+0000) 2024-02-19T18:02:48.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:48.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:48.264821+0000) 2024-02-19T18:02:49.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:49.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:49.265029+0000) 2024-02-19T18:02:50.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:50.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:50.265221+0000) 2024-02-19T18:02:51.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:51.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:51.265463+0000) 2024-02-19T18:02:52.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:52.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:52.265678+0000) 2024-02-19T18:02:53.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:53.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:53.265874+0000) 2024-02-19T18:02:54.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:54.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:54.266150+0000) 2024-02-19T18:02:55.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:55.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:55.266285+0000) 2024-02-19T18:02:56.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:56.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:56.266494+0000) 2024-02-19T18:02:57.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:57.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:57.266799+0000) 2024-02-19T18:02:58.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:58.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:58.266955+0000) 2024-02-19T18:02:59.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:02:59.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:02:59.267204+0000) 2024-02-19T18:03:00.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:00.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:00.267403+0000) 2024-02-19T18:03:01.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:01.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:01.267605+0000) 2024-02-19T18:03:02.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:02.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:02.267862+0000) 2024-02-19T18:03:03.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:03.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:03.268064+0000) 2024-02-19T18:03:04.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:04.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:04.268291+0000) 2024-02-19T18:03:05.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:05.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:05.268549+0000) 2024-02-19T18:03:05.468 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:03:05.473 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:03:06.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:06.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:06.268803+0000) 2024-02-19T18:03:07.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:07.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:07.269023+0000) 2024-02-19T18:03:08.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:08.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:08.269232+0000) 2024-02-19T18:03:09.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:09.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:09.269435+0000) 2024-02-19T18:03:10.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:10.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:10.269663+0000) 2024-02-19T18:03:11.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:11.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:11.269871+0000) 2024-02-19T18:03:12.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:12.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:12.270070+0000) 2024-02-19T18:03:13.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:13.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:13.270273+0000) 2024-02-19T18:03:14.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:14.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:14.270458+0000) 2024-02-19T18:03:15.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:15.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:15.270756+0000) 2024-02-19T18:03:16.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:16.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:16.271028+0000) 2024-02-19T18:03:17.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:17.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:17.271261+0000) 2024-02-19T18:03:18.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:18.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:18.271507+0000) 2024-02-19T18:03:19.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:19.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:19.271758+0000) 2024-02-19T18:03:20.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:20.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:20.272033+0000) 2024-02-19T18:03:21.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:21.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:21.272203+0000) 2024-02-19T18:03:22.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:22.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:22.272438+0000) 2024-02-19T18:03:23.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:23.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:23.272637+0000) 2024-02-19T18:03:24.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:24.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:24.272818+0000) 2024-02-19T18:03:25.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:25.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:25.273052+0000) 2024-02-19T18:03:26.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:26.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:26.273246+0000) 2024-02-19T18:03:27.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:27.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:27.273439+0000) 2024-02-19T18:03:28.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:28.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:28.273678+0000) 2024-02-19T18:03:29.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:29.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:29.273941+0000) 2024-02-19T18:03:30.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:30.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:30.274183+0000) 2024-02-19T18:03:31.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:31.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:31.274431+0000) 2024-02-19T18:03:32.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:32.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:32.274665+0000) 2024-02-19T18:03:33.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:33.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:33.274908+0000) 2024-02-19T18:03:34.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:34.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:34.275103+0000) 2024-02-19T18:03:35.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:35.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:35.275307+0000) 2024-02-19T18:03:35.528 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:03:35.531 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:03:36.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:36.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:36.275548+0000) 2024-02-19T18:03:37.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:37.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:37.275735+0000) 2024-02-19T18:03:38.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:38.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:38.275988+0000) 2024-02-19T18:03:39.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:39.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:39.276194+0000) 2024-02-19T18:03:40.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:40.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:40.276436+0000) 2024-02-19T18:03:41.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:41.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:41.276570+0000) 2024-02-19T18:03:42.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:42.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:42.276767+0000) 2024-02-19T18:03:43.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:43.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:43.277011+0000) 2024-02-19T18:03:44.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:44.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:44.277252+0000) 2024-02-19T18:03:45.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:45.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:45.277497+0000) 2024-02-19T18:03:46.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:46.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:46.277749+0000) 2024-02-19T18:03:47.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:47.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:47.277927+0000) 2024-02-19T18:03:48.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:48.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:48.278188+0000) 2024-02-19T18:03:49.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:49.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:49.278443+0000) 2024-02-19T18:03:50.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:50.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:50.278784+0000) 2024-02-19T18:03:51.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:51.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:51.278944+0000) 2024-02-19T18:03:52.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:52.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:52.279150+0000) 2024-02-19T18:03:53.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:53.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:53.279331+0000) 2024-02-19T18:03:54.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:54.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:54.279589+0000) 2024-02-19T18:03:55.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:55.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:55.279799+0000) 2024-02-19T18:03:56.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:56.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:56.280018+0000) 2024-02-19T18:03:57.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:57.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:57.280200+0000) 2024-02-19T18:03:58.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:58.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:58.280373+0000) 2024-02-19T18:03:59.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:03:59.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:03:59.280610+0000) 2024-02-19T18:04:00.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:00.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:00.280852+0000) 2024-02-19T18:04:01.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:01.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:01.281101+0000) 2024-02-19T18:04:02.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:02.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:02.281299+0000) 2024-02-19T18:04:03.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:03.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:03.281557+0000) 2024-02-19T18:04:04.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:04.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:04.281797+0000) 2024-02-19T18:04:05.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:05.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:05.282009+0000) 2024-02-19T18:04:05.653 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:04:05.658 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:04:06.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:06.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:06.282254+0000) 2024-02-19T18:04:07.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:07.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:07.282478+0000) 2024-02-19T18:04:08.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:08.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:08.282679+0000) 2024-02-19T18:04:09.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:09.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:09.282939+0000) 2024-02-19T18:04:10.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:10.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:10.283120+0000) 2024-02-19T18:04:11.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:11.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:11.283258+0000) 2024-02-19T18:04:12.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:12.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:12.283454+0000) 2024-02-19T18:04:13.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:13.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:13.283600+0000) 2024-02-19T18:04:14.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:14.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:14.283795+0000) 2024-02-19T18:04:15.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:15.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:15.283980+0000) 2024-02-19T18:04:16.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:16.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:16.284195+0000) 2024-02-19T18:04:17.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:17.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:17.284375+0000) 2024-02-19T18:04:18.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:18.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:18.284627+0000) 2024-02-19T18:04:19.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:19.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:19.284882+0000) 2024-02-19T18:04:20.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:20.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:20.285143+0000) 2024-02-19T18:04:21.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:21.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:21.285390+0000) 2024-02-19T18:04:22.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:22.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:22.285659+0000) 2024-02-19T18:04:23.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:23.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:23.285905+0000) 2024-02-19T18:04:24.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:24.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:24.286127+0000) 2024-02-19T18:04:25.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:25.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:25.286328+0000) 2024-02-19T18:04:26.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:26.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:26.286497+0000) 2024-02-19T18:04:27.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:27.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:27.286748+0000) 2024-02-19T18:04:28.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:28.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:28.287032+0000) 2024-02-19T18:04:29.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:29.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:29.287267+0000) 2024-02-19T18:04:30.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:30.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:30.287447+0000) 2024-02-19T18:04:31.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:31.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:31.287700+0000) 2024-02-19T18:04:32.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:32.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:32.287945+0000) 2024-02-19T18:04:33.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:33.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:33.288177+0000) 2024-02-19T18:04:34.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:34.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:34.288418+0000) 2024-02-19T18:04:35.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:35.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:35.288654+0000) 2024-02-19T18:04:35.791 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:04:35.795 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:04:36.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:36.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:36.288903+0000) 2024-02-19T18:04:37.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:37.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:37.289071+0000) 2024-02-19T18:04:38.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:38.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:38.289271+0000) 2024-02-19T18:04:39.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:39.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:39.289557+0000) 2024-02-19T18:04:40.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:40.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:40.289789+0000) 2024-02-19T18:04:41.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:41.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:41.290094+0000) 2024-02-19T18:04:42.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:42.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:42.290327+0000) 2024-02-19T18:04:43.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:43.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:43.290560+0000) 2024-02-19T18:04:44.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:44.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:44.290841+0000) 2024-02-19T18:04:45.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:45.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:45.291037+0000) 2024-02-19T18:04:46.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:46.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:46.291207+0000) 2024-02-19T18:04:47.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:47.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:47.291451+0000) 2024-02-19T18:04:48.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:48.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:48.291653+0000) 2024-02-19T18:04:49.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:49.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:49.291874+0000) 2024-02-19T18:04:50.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:50.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:50.292039+0000) 2024-02-19T18:04:51.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:51.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:51.292295+0000) 2024-02-19T18:04:52.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:52.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:52.292492+0000) 2024-02-19T18:04:53.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:53.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:53.292735+0000) 2024-02-19T18:04:54.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:54.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:54.292908+0000) 2024-02-19T18:04:55.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:55.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:55.293067+0000) 2024-02-19T18:04:56.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:56.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:56.293323+0000) 2024-02-19T18:04:57.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:57.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:57.293510+0000) 2024-02-19T18:04:58.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:58.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:58.293752+0000) 2024-02-19T18:04:59.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:04:59.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:04:59.293986+0000) 2024-02-19T18:05:00.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:00.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:00.294212+0000) 2024-02-19T18:05:01.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:01.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:01.294456+0000) 2024-02-19T18:05:02.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:02.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:02.294749+0000) 2024-02-19T18:05:03.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:03.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:03.294989+0000) 2024-02-19T18:05:04.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:04.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:04.295218+0000) 2024-02-19T18:05:05.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:05.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:05.295463+0000) 2024-02-19T18:05:05.905 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:05:05.909 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:05:06.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:06.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:06.295675+0000) 2024-02-19T18:05:07.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:07.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:07.295912+0000) 2024-02-19T18:05:08.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:08.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:08.296131+0000) 2024-02-19T18:05:09.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:09.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:09.296308+0000) 2024-02-19T18:05:10.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:10.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:10.296545+0000) 2024-02-19T18:05:11.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:11.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:11.296748+0000) 2024-02-19T18:05:12.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:12.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:12.296996+0000) 2024-02-19T18:05:13.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:13.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:13.297237+0000) 2024-02-19T18:05:14.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:14.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:14.297412+0000) 2024-02-19T18:05:15.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:15.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:15.297650+0000) 2024-02-19T18:05:16.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:16.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:16.297924+0000) 2024-02-19T18:05:17.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:17.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:17.298161+0000) 2024-02-19T18:05:18.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:18.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:18.298406+0000) 2024-02-19T18:05:19.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:19.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:19.298609+0000) 2024-02-19T18:05:20.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:20.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:20.298849+0000) 2024-02-19T18:05:21.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:21.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:21.298977+0000) 2024-02-19T18:05:22.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:22.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:22.299194+0000) 2024-02-19T18:05:23.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:23.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:23.299434+0000) 2024-02-19T18:05:24.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:24.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:24.299646+0000) 2024-02-19T18:05:25.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:25.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:25.299882+0000) 2024-02-19T18:05:26.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:26.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:26.300101+0000) 2024-02-19T18:05:27.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:27.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:27.300301+0000) 2024-02-19T18:05:28.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:28.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:28.300559+0000) 2024-02-19T18:05:29.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:29.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:29.300868+0000) 2024-02-19T18:05:30.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:30.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:30.301115+0000) 2024-02-19T18:05:31.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:31.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:31.301300+0000) 2024-02-19T18:05:32.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:32.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:32.301532+0000) 2024-02-19T18:05:33.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:33.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:33.301788+0000) 2024-02-19T18:05:34.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:34.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:34.302131+0000) 2024-02-19T18:05:35.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:35.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:35.302325+0000) 2024-02-19T18:05:36.028 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:05:36.032 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:05:36.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:36.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:36.302635+0000) 2024-02-19T18:05:37.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:37.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:37.302840+0000) 2024-02-19T18:05:38.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:38.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:38.303100+0000) 2024-02-19T18:05:39.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:39.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:39.303397+0000) 2024-02-19T18:05:40.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:40.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:40.303676+0000) 2024-02-19T18:05:41.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:41.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:41.303978+0000) 2024-02-19T18:05:42.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:42.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:42.304200+0000) 2024-02-19T18:05:43.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:43.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:43.304466+0000) 2024-02-19T18:05:44.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:44.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:44.304777+0000) 2024-02-19T18:05:45.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:45.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:45.304997+0000) 2024-02-19T18:05:46.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:46.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:46.305271+0000) 2024-02-19T18:05:47.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:47.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:47.305568+0000) 2024-02-19T18:05:48.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:48.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:48.305827+0000) 2024-02-19T18:05:49.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:49.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:49.306053+0000) 2024-02-19T18:05:50.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:50.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:50.306231+0000) 2024-02-19T18:05:51.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:51.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:51.306454+0000) 2024-02-19T18:05:52.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:52.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:52.306632+0000) 2024-02-19T18:05:53.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:53.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:53.306852+0000) 2024-02-19T18:05:54.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:54.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:54.307135+0000) 2024-02-19T18:05:55.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:55.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:55.307456+0000) 2024-02-19T18:05:56.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:56.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:56.307758+0000) 2024-02-19T18:05:57.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:57.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:57.308046+0000) 2024-02-19T18:05:58.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:58.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:58.308246+0000) 2024-02-19T18:05:59.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:05:59.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:05:59.308503+0000) 2024-02-19T18:06:00.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:00.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:00.308749+0000) 2024-02-19T18:06:01.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:01.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:01.308972+0000) 2024-02-19T18:06:02.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:02.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:02.309151+0000) 2024-02-19T18:06:03.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:03.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:03.309379+0000) 2024-02-19T18:06:04.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:04.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:04.309641+0000) 2024-02-19T18:06:05.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:05.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:05.309795+0000) 2024-02-19T18:06:06.125 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:06:06.129 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:06:06.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:06.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:06.310007+0000) 2024-02-19T18:06:07.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:07.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:07.310205+0000) 2024-02-19T18:06:08.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:08.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:08.310463+0000) 2024-02-19T18:06:09.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:09.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:09.310660+0000) 2024-02-19T18:06:10.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:10.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:10.310919+0000) 2024-02-19T18:06:11.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:11.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:11.311128+0000) 2024-02-19T18:06:12.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:12.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:12.311387+0000) 2024-02-19T18:06:13.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:13.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:13.311640+0000) 2024-02-19T18:06:14.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:14.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:14.311890+0000) 2024-02-19T18:06:15.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:15.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:15.312173+0000) 2024-02-19T18:06:16.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:16.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:16.312471+0000) 2024-02-19T18:06:17.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:17.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:17.312744+0000) 2024-02-19T18:06:18.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:18.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:18.313028+0000) 2024-02-19T18:06:19.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:19.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:19.313249+0000) 2024-02-19T18:06:20.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:20.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:20.313529+0000) 2024-02-19T18:06:21.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:21.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:21.313687+0000) 2024-02-19T18:06:22.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:22.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:22.313863+0000) 2024-02-19T18:06:23.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:23.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:23.314078+0000) 2024-02-19T18:06:24.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:24.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:24.314348+0000) 2024-02-19T18:06:25.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:25.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:25.314535+0000) 2024-02-19T18:06:26.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:26.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:26.314789+0000) 2024-02-19T18:06:27.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:27.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:27.314962+0000) 2024-02-19T18:06:28.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:28.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:28.315155+0000) 2024-02-19T18:06:29.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:29.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:29.315360+0000) 2024-02-19T18:06:30.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:30.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:30.315657+0000) 2024-02-19T18:06:31.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:31.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:31.315926+0000) 2024-02-19T18:06:32.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:32.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:32.316163+0000) 2024-02-19T18:06:33.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:33.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:33.316432+0000) 2024-02-19T18:06:34.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:34.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:34.316715+0000) 2024-02-19T18:06:35.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:35.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:35.317001+0000) 2024-02-19T18:06:36.257 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:06:36.260 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:06:36.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:36.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:36.317236+0000) 2024-02-19T18:06:37.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:37.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:37.317502+0000) 2024-02-19T18:06:38.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:38.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:38.317788+0000) 2024-02-19T18:06:39.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:39.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:39.318092+0000) 2024-02-19T18:06:40.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:40.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:40.318348+0000) 2024-02-19T18:06:41.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:41.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:41.318525+0000) 2024-02-19T18:06:42.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:42.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:42.318837+0000) 2024-02-19T18:06:43.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:43.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:43.318997+0000) 2024-02-19T18:06:44.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:44.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:44.319130+0000) 2024-02-19T18:06:45.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:45.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:45.319338+0000) 2024-02-19T18:06:46.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:46.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:46.319515+0000) 2024-02-19T18:06:47.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:47.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:47.319759+0000) 2024-02-19T18:06:48.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:48.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:48.319999+0000) 2024-02-19T18:06:49.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:49.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:49.320226+0000) 2024-02-19T18:06:50.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:50.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:50.320460+0000) 2024-02-19T18:06:51.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:51.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:51.320698+0000) 2024-02-19T18:06:52.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:52.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:52.320855+0000) 2024-02-19T18:06:53.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:53.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:53.321095+0000) 2024-02-19T18:06:54.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:54.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:54.321304+0000) 2024-02-19T18:06:55.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:55.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:55.321587+0000) 2024-02-19T18:06:56.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:56.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:56.321841+0000) 2024-02-19T18:06:57.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:57.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:57.322015+0000) 2024-02-19T18:06:58.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:58.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:58.322257+0000) 2024-02-19T18:06:59.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:06:59.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:06:59.322429+0000) 2024-02-19T18:07:00.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:00.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:00.322649+0000) 2024-02-19T18:07:01.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:01.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:01.322899+0000) 2024-02-19T18:07:02.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:02.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:02.323156+0000) 2024-02-19T18:07:03.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:03.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:03.323284+0000) 2024-02-19T18:07:04.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:04.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:04.323526+0000) 2024-02-19T18:07:05.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:05.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:05.323767+0000) 2024-02-19T18:07:06.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:06.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:06.323981+0000) 2024-02-19T18:07:06.324 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:07:06.328 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:07:07.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:07.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:07.324198+0000) 2024-02-19T18:07:08.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:08.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:08.324468+0000) 2024-02-19T18:07:09.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:09.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:09.324700+0000) 2024-02-19T18:07:10.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:10.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:10.325045+0000) 2024-02-19T18:07:11.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:11.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:11.325275+0000) 2024-02-19T18:07:12.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:12.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:12.325513+0000) 2024-02-19T18:07:13.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:13.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:13.325745+0000) 2024-02-19T18:07:14.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:14.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:14.325975+0000) 2024-02-19T18:07:15.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:15.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:15.326161+0000) 2024-02-19T18:07:16.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:16.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:16.326387+0000) 2024-02-19T18:07:17.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:17.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:17.326593+0000) 2024-02-19T18:07:18.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:18.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:18.326842+0000) 2024-02-19T18:07:19.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:19.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:19.327076+0000) 2024-02-19T18:07:20.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:20.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:20.327258+0000) 2024-02-19T18:07:21.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:21.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:21.327535+0000) 2024-02-19T18:07:22.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:22.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:22.327752+0000) 2024-02-19T18:07:23.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:23.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:23.327986+0000) 2024-02-19T18:07:24.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:24.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:24.328157+0000) 2024-02-19T18:07:25.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:25.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:25.328391+0000) 2024-02-19T18:07:26.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:26.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:26.328654+0000) 2024-02-19T18:07:27.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:27.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:27.328849+0000) 2024-02-19T18:07:28.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:28.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:28.328996+0000) 2024-02-19T18:07:29.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:29.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:29.329220+0000) 2024-02-19T18:07:30.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:30.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:30.329396+0000) 2024-02-19T18:07:31.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:31.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:31.329637+0000) 2024-02-19T18:07:32.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:32.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:32.329826+0000) 2024-02-19T18:07:33.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:33.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:33.330087+0000) 2024-02-19T18:07:34.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:34.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:34.330259+0000) 2024-02-19T18:07:35.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:35.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:35.330505+0000) 2024-02-19T18:07:36.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:36.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:36.330739+0000) 2024-02-19T18:07:36.395 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:07:36.398 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:07:37.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:37.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:37.330985+0000) 2024-02-19T18:07:38.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:38.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:38.331246+0000) 2024-02-19T18:07:39.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:39.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:39.331492+0000) 2024-02-19T18:07:40.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:40.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:40.331757+0000) 2024-02-19T18:07:41.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:41.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:41.332004+0000) 2024-02-19T18:07:42.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:42.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:42.332244+0000) 2024-02-19T18:07:43.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:43.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:43.332447+0000) 2024-02-19T18:07:44.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:44.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:44.332708+0000) 2024-02-19T18:07:45.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:45.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:45.332966+0000) 2024-02-19T18:07:46.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:46.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:46.333130+0000) 2024-02-19T18:07:47.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:47.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:47.333385+0000) 2024-02-19T18:07:48.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:48.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:48.333626+0000) 2024-02-19T18:07:49.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:49.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:49.333884+0000) 2024-02-19T18:07:50.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:50.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:50.334165+0000) 2024-02-19T18:07:51.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:51.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:51.334436+0000) 2024-02-19T18:07:52.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:52.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:52.334656+0000) 2024-02-19T18:07:53.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:53.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:53.334900+0000) 2024-02-19T18:07:54.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:54.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:54.335111+0000) 2024-02-19T18:07:55.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:55.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:55.335323+0000) 2024-02-19T18:07:56.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:56.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:56.335580+0000) 2024-02-19T18:07:57.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:57.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:57.335845+0000) 2024-02-19T18:07:58.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:58.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:58.336106+0000) 2024-02-19T18:07:59.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:07:59.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:07:59.336391+0000) 2024-02-19T18:08:00.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:00.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:00.336541+0000) 2024-02-19T18:08:01.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:01.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:01.336703+0000) 2024-02-19T18:08:02.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:02.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:02.336938+0000) 2024-02-19T18:08:03.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:03.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:03.337185+0000) 2024-02-19T18:08:04.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:04.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:04.337466+0000) 2024-02-19T18:08:05.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:05.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:05.337723+0000) 2024-02-19T18:08:06.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:06.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:06.337961+0000) 2024-02-19T18:08:06.469 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:08:06.473 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:08:07.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:07.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:07.338186+0000) 2024-02-19T18:08:08.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:08.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:08.338467+0000) 2024-02-19T18:08:09.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:09.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:09.338741+0000) 2024-02-19T18:08:10.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:10.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:10.338944+0000) 2024-02-19T18:08:11.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:11.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:11.339146+0000) 2024-02-19T18:08:12.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:12.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:12.339325+0000) 2024-02-19T18:08:13.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:13.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:13.339595+0000) 2024-02-19T18:08:14.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:14.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:14.339786+0000) 2024-02-19T18:08:15.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:15.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:15.339965+0000) 2024-02-19T18:08:16.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:16.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:16.340162+0000) 2024-02-19T18:08:17.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:17.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:17.340346+0000) 2024-02-19T18:08:18.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:18.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:18.340567+0000) 2024-02-19T18:08:19.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:19.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:19.340719+0000) 2024-02-19T18:08:20.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:20.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:20.340966+0000) 2024-02-19T18:08:21.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:21.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:21.341251+0000) 2024-02-19T18:08:22.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:22.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:22.341491+0000) 2024-02-19T18:08:23.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:23.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:23.341711+0000) 2024-02-19T18:08:24.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:24.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:24.341979+0000) 2024-02-19T18:08:25.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:25.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:25.342234+0000) 2024-02-19T18:08:26.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:26.336+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:26.342514+0000) 2024-02-19T18:08:27.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:27.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:27.342818+0000) 2024-02-19T18:08:28.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:28.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:28.342963+0000) 2024-02-19T18:08:29.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:29.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:29.343189+0000) 2024-02-19T18:08:30.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:30.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:30.343393+0000) 2024-02-19T18:08:31.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:31.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:31.343654+0000) 2024-02-19T18:08:32.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:32.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:32.343877+0000) 2024-02-19T18:08:33.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:33.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:33.343990+0000) 2024-02-19T18:08:34.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:34.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:34.344218+0000) 2024-02-19T18:08:35.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:35.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:35.344444+0000) 2024-02-19T18:08:36.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:36.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:36.344731+0000) 2024-02-19T18:08:36.535 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:08:36.539 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:08:37.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:37.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:37.344995+0000) 2024-02-19T18:08:38.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:38.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:38.345249+0000) 2024-02-19T18:08:39.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:39.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:39.345501+0000) 2024-02-19T18:08:40.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:40.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:40.345769+0000) 2024-02-19T18:08:41.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:41.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:41.346024+0000) 2024-02-19T18:08:42.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:42.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:42.346260+0000) 2024-02-19T18:08:43.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:43.340+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:43.346443+0000) 2024-02-19T18:08:44.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:44.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:44.346692+0000) 2024-02-19T18:08:45.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:45.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:45.346895+0000) 2024-02-19T18:08:46.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:46.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:46.347106+0000) 2024-02-19T18:08:47.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:47.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:47.347295+0000) 2024-02-19T18:08:48.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:48.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:48.347548+0000) 2024-02-19T18:08:49.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:49.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:49.347801+0000) 2024-02-19T18:08:50.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:50.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:50.348026+0000) 2024-02-19T18:08:51.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:51.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:51.348270+0000) 2024-02-19T18:08:52.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:52.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:52.348475+0000) 2024-02-19T18:08:53.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:53.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:53.348714+0000) 2024-02-19T18:08:54.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:54.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:54.348888+0000) 2024-02-19T18:08:55.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:55.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:55.349088+0000) 2024-02-19T18:08:56.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:56.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:56.349359+0000) 2024-02-19T18:08:57.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:57.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:57.349708+0000) 2024-02-19T18:08:58.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:58.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:58.349960+0000) 2024-02-19T18:08:59.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:08:59.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:08:59.350176+0000) 2024-02-19T18:09:00.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:00.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:00.350419+0000) 2024-02-19T18:09:01.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:01.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:01.350693+0000) 2024-02-19T18:09:02.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:02.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:02.350932+0000) 2024-02-19T18:09:03.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:03.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:03.351120+0000) 2024-02-19T18:09:04.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:04.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:04.351314+0000) 2024-02-19T18:09:05.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:05.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:05.351465+0000) 2024-02-19T18:09:06.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:06.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:06.351695+0000) 2024-02-19T18:09:06.650 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:09:06.653 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:09:07.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:07.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:07.351896+0000) 2024-02-19T18:09:08.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:08.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:08.352142+0000) 2024-02-19T18:09:09.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:09.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:09.352385+0000) 2024-02-19T18:09:10.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:10.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:10.352626+0000) 2024-02-19T18:09:11.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:11.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:11.352866+0000) 2024-02-19T18:09:12.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:12.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:12.353109+0000) 2024-02-19T18:09:13.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:13.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:13.353355+0000) 2024-02-19T18:09:14.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:14.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:14.353613+0000) 2024-02-19T18:09:15.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:15.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:15.353792+0000) 2024-02-19T18:09:16.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:16.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:16.354016+0000) 2024-02-19T18:09:17.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:17.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:17.354251+0000) 2024-02-19T18:09:18.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:18.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:18.354517+0000) 2024-02-19T18:09:19.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:19.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:19.354686+0000) 2024-02-19T18:09:20.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:20.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:20.354833+0000) 2024-02-19T18:09:21.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:21.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:21.355012+0000) 2024-02-19T18:09:22.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:22.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:22.355227+0000) 2024-02-19T18:09:23.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:23.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:23.355473+0000) 2024-02-19T18:09:24.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:24.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:24.355717+0000) 2024-02-19T18:09:25.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:25.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:25.355923+0000) 2024-02-19T18:09:26.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:26.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:26.356117+0000) 2024-02-19T18:09:27.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:27.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:27.356363+0000) 2024-02-19T18:09:28.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:28.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:28.356559+0000) 2024-02-19T18:09:29.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:29.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:29.356797+0000) 2024-02-19T18:09:30.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:30.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:30.357042+0000) 2024-02-19T18:09:31.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:31.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:31.357219+0000) 2024-02-19T18:09:32.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:32.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:32.357409+0000) 2024-02-19T18:09:33.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:33.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:33.357595+0000) 2024-02-19T18:09:34.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:34.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:34.357853+0000) 2024-02-19T18:09:35.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:35.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:35.358014+0000) 2024-02-19T18:09:36.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:36.351+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:36.358265+0000) 2024-02-19T18:09:36.782 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:09:36.787 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:09:37.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:37.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:37.358502+0000) 2024-02-19T18:09:38.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:38.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:38.358788+0000) 2024-02-19T18:09:39.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:39.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:39.358972+0000) 2024-02-19T18:09:40.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:40.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:40.359124+0000) 2024-02-19T18:09:41.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:41.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:41.359352+0000) 2024-02-19T18:09:42.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:42.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:42.359604+0000) 2024-02-19T18:09:43.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:43.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:43.359823+0000) 2024-02-19T18:09:44.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:44.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:44.360091+0000) 2024-02-19T18:09:45.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:45.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:45.360321+0000) 2024-02-19T18:09:46.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:46.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:46.360589+0000) 2024-02-19T18:09:47.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:47.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:47.360845+0000) 2024-02-19T18:09:48.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:48.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:48.361036+0000) 2024-02-19T18:09:49.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:49.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:49.361269+0000) 2024-02-19T18:09:50.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:50.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:50.361544+0000) 2024-02-19T18:09:51.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:51.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:51.361820+0000) 2024-02-19T18:09:52.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:52.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:52.362054+0000) 2024-02-19T18:09:53.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:53.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:53.362262+0000) 2024-02-19T18:09:54.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:54.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:54.362539+0000) 2024-02-19T18:09:55.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:55.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:55.362809+0000) 2024-02-19T18:09:56.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:56.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:56.363015+0000) 2024-02-19T18:09:57.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:57.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:57.363251+0000) 2024-02-19T18:09:58.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:58.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:58.363489+0000) 2024-02-19T18:09:59.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:09:59.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:09:59.363734+0000) 2024-02-19T18:10:00.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:00.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:00.364008+0000) 2024-02-19T18:10:01.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:01.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:01.364237+0000) 2024-02-19T18:10:02.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:02.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:02.364463+0000) 2024-02-19T18:10:03.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:03.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:03.364693+0000) 2024-02-19T18:10:04.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:04.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:04.364959+0000) 2024-02-19T18:10:05.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:05.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:05.365154+0000) 2024-02-19T18:10:06.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:06.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:06.365342+0000) 2024-02-19T18:10:06.905 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:10:06.909 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:10:07.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:07.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:07.365582+0000) 2024-02-19T18:10:08.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:08.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:08.365798+0000) 2024-02-19T18:10:09.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:09.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:09.365965+0000) 2024-02-19T18:10:10.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:10.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:10.366236+0000) 2024-02-19T18:10:11.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:11.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:11.366475+0000) 2024-02-19T18:10:12.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:12.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:12.366759+0000) 2024-02-19T18:10:13.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:13.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:13.366979+0000) 2024-02-19T18:10:14.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:14.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:14.367252+0000) 2024-02-19T18:10:15.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:15.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:15.367472+0000) 2024-02-19T18:10:16.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:16.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:16.367682+0000) 2024-02-19T18:10:17.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:17.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:17.367885+0000) 2024-02-19T18:10:18.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:18.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:18.368097+0000) 2024-02-19T18:10:19.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:19.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:19.368338+0000) 2024-02-19T18:10:20.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:20.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:20.368543+0000) 2024-02-19T18:10:21.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:21.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:21.368738+0000) 2024-02-19T18:10:22.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:22.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:22.368963+0000) 2024-02-19T18:10:23.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:23.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:23.369191+0000) 2024-02-19T18:10:24.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:24.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:24.369326+0000) 2024-02-19T18:10:25.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:25.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:25.369553+0000) 2024-02-19T18:10:26.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:26.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:26.369751+0000) 2024-02-19T18:10:27.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:27.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:27.369989+0000) 2024-02-19T18:10:28.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:28.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:28.370155+0000) 2024-02-19T18:10:29.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:29.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:29.370336+0000) 2024-02-19T18:10:30.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:30.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:30.370547+0000) 2024-02-19T18:10:31.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:31.366+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:31.370816+0000) 2024-02-19T18:10:32.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:32.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:32.371066+0000) 2024-02-19T18:10:33.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:33.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:33.371208+0000) 2024-02-19T18:10:34.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:34.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:34.371434+0000) 2024-02-19T18:10:35.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:35.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:35.371627+0000) 2024-02-19T18:10:36.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:36.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:36.371906+0000) 2024-02-19T18:10:37.021 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:10:37.024 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:10:37.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:37.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:37.372162+0000) 2024-02-19T18:10:38.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:38.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:38.372470+0000) 2024-02-19T18:10:39.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:39.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:39.372691+0000) 2024-02-19T18:10:40.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:40.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:40.372942+0000) 2024-02-19T18:10:41.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:41.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:41.373177+0000) 2024-02-19T18:10:42.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:42.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:42.373369+0000) 2024-02-19T18:10:43.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:43.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:43.373565+0000) 2024-02-19T18:10:44.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:44.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:44.373803+0000) 2024-02-19T18:10:45.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:45.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:45.373968+0000) 2024-02-19T18:10:46.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:46.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:46.374214+0000) 2024-02-19T18:10:47.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:47.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:47.374422+0000) 2024-02-19T18:10:48.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:48.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:48.374667+0000) 2024-02-19T18:10:49.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:49.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:49.374933+0000) 2024-02-19T18:10:50.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:50.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:50.375101+0000) 2024-02-19T18:10:51.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:51.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:51.375347+0000) 2024-02-19T18:10:52.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:52.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:52.375502+0000) 2024-02-19T18:10:53.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:53.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:53.375751+0000) 2024-02-19T18:10:54.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:54.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:54.376007+0000) 2024-02-19T18:10:55.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:55.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:55.376198+0000) 2024-02-19T18:10:56.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:56.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:56.376347+0000) 2024-02-19T18:10:57.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:57.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:57.376592+0000) 2024-02-19T18:10:58.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:58.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:58.376854+0000) 2024-02-19T18:10:59.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:10:59.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:10:59.377080+0000) 2024-02-19T18:11:00.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:00.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:00.377267+0000) 2024-02-19T18:11:01.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:01.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:01.377492+0000) 2024-02-19T18:11:02.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:02.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:02.377749+0000) 2024-02-19T18:11:03.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:03.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:03.377914+0000) 2024-02-19T18:11:04.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:04.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:04.378150+0000) 2024-02-19T18:11:05.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:05.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:05.378328+0000) 2024-02-19T18:11:06.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:06.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:06.378565+0000) 2024-02-19T18:11:07.074 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:11:07.078 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:11:07.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:07.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:07.378844+0000) 2024-02-19T18:11:08.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:08.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:08.379061+0000) 2024-02-19T18:11:09.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:09.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:09.379330+0000) 2024-02-19T18:11:10.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:10.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:10.379588+0000) 2024-02-19T18:11:11.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:11.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:11.379814+0000) 2024-02-19T18:11:12.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:12.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:12.380057+0000) 2024-02-19T18:11:13.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:13.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:13.380222+0000) 2024-02-19T18:11:14.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:14.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:14.380470+0000) 2024-02-19T18:11:15.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:15.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:15.380703+0000) 2024-02-19T18:11:16.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:16.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:16.380944+0000) 2024-02-19T18:11:17.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:17.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:17.381216+0000) 2024-02-19T18:11:18.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:18.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:18.381429+0000) 2024-02-19T18:11:19.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:19.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:19.381663+0000) 2024-02-19T18:11:20.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:20.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:20.381870+0000) 2024-02-19T18:11:21.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:21.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:21.382148+0000) 2024-02-19T18:11:22.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:22.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:22.382400+0000) 2024-02-19T18:11:23.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:23.377+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:23.382594+0000) 2024-02-19T18:11:24.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:24.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:24.382842+0000) 2024-02-19T18:11:25.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:25.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:25.383091+0000) 2024-02-19T18:11:26.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:26.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:26.383246+0000) 2024-02-19T18:11:27.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:27.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:27.383449+0000) 2024-02-19T18:11:28.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:28.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:28.383665+0000) 2024-02-19T18:11:29.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:29.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:29.383851+0000) 2024-02-19T18:11:30.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:30.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:30.384119+0000) 2024-02-19T18:11:31.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:31.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:31.384391+0000) 2024-02-19T18:11:32.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:32.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:32.384603+0000) 2024-02-19T18:11:33.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:33.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:33.384841+0000) 2024-02-19T18:11:34.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:34.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:34.385016+0000) 2024-02-19T18:11:35.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:35.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:35.385233+0000) 2024-02-19T18:11:36.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:36.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:36.385420+0000) 2024-02-19T18:11:37.193 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:11:37.196 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:11:37.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:37.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:37.385664+0000) 2024-02-19T18:11:38.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:38.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:38.385903+0000) 2024-02-19T18:11:39.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:39.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:39.386083+0000) 2024-02-19T18:11:40.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:40.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:40.386225+0000) 2024-02-19T18:11:41.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:41.381+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:41.386448+0000) 2024-02-19T18:11:42.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:42.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:42.386648+0000) 2024-02-19T18:11:43.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:43.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:43.386803+0000) 2024-02-19T18:11:44.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:44.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:44.386965+0000) 2024-02-19T18:11:45.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:45.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:45.387206+0000) 2024-02-19T18:11:46.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:46.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:46.387460+0000) 2024-02-19T18:11:47.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:47.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:47.387698+0000) 2024-02-19T18:11:48.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:48.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:48.387938+0000) 2024-02-19T18:11:49.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:49.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:49.388166+0000) 2024-02-19T18:11:50.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:50.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:50.388421+0000) 2024-02-19T18:11:51.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:51.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:51.388652+0000) 2024-02-19T18:11:52.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:52.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:52.388912+0000) 2024-02-19T18:11:53.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:53.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:53.389101+0000) 2024-02-19T18:11:54.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:54.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:54.389329+0000) 2024-02-19T18:11:55.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:55.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:55.389592+0000) 2024-02-19T18:11:56.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:56.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:56.389843+0000) 2024-02-19T18:11:57.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:57.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:57.390071+0000) 2024-02-19T18:11:58.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:58.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:58.390333+0000) 2024-02-19T18:11:59.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:11:59.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:11:59.390536+0000) 2024-02-19T18:12:00.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:00.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:00.390858+0000) 2024-02-19T18:12:01.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:01.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:01.391089+0000) 2024-02-19T18:12:02.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:02.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:02.391300+0000) 2024-02-19T18:12:03.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:03.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:03.391489+0000) 2024-02-19T18:12:04.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:04.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:04.391662+0000) 2024-02-19T18:12:05.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:05.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:05.391916+0000) 2024-02-19T18:12:06.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:06.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:06.392177+0000) 2024-02-19T18:12:07.324 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:12:07.327 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:12:07.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:07.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:07.392428+0000) 2024-02-19T18:12:08.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:08.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:08.392608+0000) 2024-02-19T18:12:09.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:09.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:09.392798+0000) 2024-02-19T18:12:10.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:10.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:10.392995+0000) 2024-02-19T18:12:11.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:11.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:11.393255+0000) 2024-02-19T18:12:12.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:12.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:12.393512+0000) 2024-02-19T18:12:13.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:13.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:13.393708+0000) 2024-02-19T18:12:14.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:14.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:14.393953+0000) 2024-02-19T18:12:15.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:15.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:15.394151+0000) 2024-02-19T18:12:16.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:16.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:16.394392+0000) 2024-02-19T18:12:17.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:17.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:17.394580+0000) 2024-02-19T18:12:18.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:18.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:18.394822+0000) 2024-02-19T18:12:19.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:19.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:19.395035+0000) 2024-02-19T18:12:20.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:20.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:20.395287+0000) 2024-02-19T18:12:21.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:21.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:21.395514+0000) 2024-02-19T18:12:22.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:22.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:22.395770+0000) 2024-02-19T18:12:23.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:23.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:23.395946+0000) 2024-02-19T18:12:24.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:24.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:24.396174+0000) 2024-02-19T18:12:25.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:25.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:25.396415+0000) 2024-02-19T18:12:26.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:26.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:26.396630+0000) 2024-02-19T18:12:27.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:27.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:27.396864+0000) 2024-02-19T18:12:28.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:28.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:28.397104+0000) 2024-02-19T18:12:29.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:29.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:29.397335+0000) 2024-02-19T18:12:30.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:30.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:30.397573+0000) 2024-02-19T18:12:31.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:31.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:31.397722+0000) 2024-02-19T18:12:32.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:32.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:32.397973+0000) 2024-02-19T18:12:33.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:33.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:33.398221+0000) 2024-02-19T18:12:34.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:34.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:34.398486+0000) 2024-02-19T18:12:35.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:35.392+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:35.398608+0000) 2024-02-19T18:12:36.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:36.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:36.398869+0000) 2024-02-19T18:12:37.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:37.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:37.399117+0000) 2024-02-19T18:12:37.423 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:12:37.427 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:12:38.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:38.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:38.399372+0000) 2024-02-19T18:12:39.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:39.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:39.399511+0000) 2024-02-19T18:12:40.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:40.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:40.399726+0000) 2024-02-19T18:12:41.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:41.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:41.399953+0000) 2024-02-19T18:12:42.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:42.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:42.400180+0000) 2024-02-19T18:12:43.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:43.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:43.400411+0000) 2024-02-19T18:12:44.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:44.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:44.400662+0000) 2024-02-19T18:12:45.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:45.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:45.400923+0000) 2024-02-19T18:12:46.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:46.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:46.401110+0000) 2024-02-19T18:12:47.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:47.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:47.401284+0000) 2024-02-19T18:12:48.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:48.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:48.401540+0000) 2024-02-19T18:12:49.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:49.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:49.401772+0000) 2024-02-19T18:12:50.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:50.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:50.401944+0000) 2024-02-19T18:12:51.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:51.396+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:51.402186+0000) 2024-02-19T18:12:52.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:52.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:52.402360+0000) 2024-02-19T18:12:53.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:53.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:53.402559+0000) 2024-02-19T18:12:54.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:54.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:54.402779+0000) 2024-02-19T18:12:55.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:55.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:55.402906+0000) 2024-02-19T18:12:56.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:56.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:56.403143+0000) 2024-02-19T18:12:57.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:57.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:57.403385+0000) 2024-02-19T18:12:58.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:58.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:58.403615+0000) 2024-02-19T18:12:59.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:12:59.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:12:59.403843+0000) 2024-02-19T18:13:00.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:00.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:00.404078+0000) 2024-02-19T18:13:01.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:01.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:01.404257+0000) 2024-02-19T18:13:02.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:02.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:02.404490+0000) 2024-02-19T18:13:03.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:03.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:03.404626+0000) 2024-02-19T18:13:04.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:04.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:04.404874+0000) 2024-02-19T18:13:05.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:05.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:05.405111+0000) 2024-02-19T18:13:06.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:06.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:06.405366+0000) 2024-02-19T18:13:07.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:07.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:07.405593+0000) 2024-02-19T18:13:07.505 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:13:07.509 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:13:08.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:08.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:08.405882+0000) 2024-02-19T18:13:09.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:09.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:09.406115+0000) 2024-02-19T18:13:10.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:10.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:10.406370+0000) 2024-02-19T18:13:11.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:11.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:11.406610+0000) 2024-02-19T18:13:12.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:12.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:12.406862+0000) 2024-02-19T18:13:13.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:13.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:13.407188+0000) 2024-02-19T18:13:14.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:14.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:14.407380+0000) 2024-02-19T18:13:15.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:15.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:15.407613+0000) 2024-02-19T18:13:16.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:16.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:16.407860+0000) 2024-02-19T18:13:17.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:17.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:17.408143+0000) 2024-02-19T18:13:18.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:18.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:18.408355+0000) 2024-02-19T18:13:19.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:19.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:19.408488+0000) 2024-02-19T18:13:20.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:20.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:20.408723+0000) 2024-02-19T18:13:21.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:21.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:21.408898+0000) 2024-02-19T18:13:22.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:22.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:22.409149+0000) 2024-02-19T18:13:23.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:23.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:23.409415+0000) 2024-02-19T18:13:24.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:24.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:24.409683+0000) 2024-02-19T18:13:25.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:25.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:25.409918+0000) 2024-02-19T18:13:26.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:26.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:26.410138+0000) 2024-02-19T18:13:27.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:27.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:27.410313+0000) 2024-02-19T18:13:28.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:28.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:28.410539+0000) 2024-02-19T18:13:29.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:29.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:29.410802+0000) 2024-02-19T18:13:30.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:30.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:30.411040+0000) 2024-02-19T18:13:31.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:31.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:31.411319+0000) 2024-02-19T18:13:32.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:32.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:32.411578+0000) 2024-02-19T18:13:33.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:33.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:33.411813+0000) 2024-02-19T18:13:34.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:34.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:34.412015+0000) 2024-02-19T18:13:35.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:35.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:35.412155+0000) 2024-02-19T18:13:36.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:36.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:36.412333+0000) 2024-02-19T18:13:37.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:37.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:37.412574+0000) 2024-02-19T18:13:37.560 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:13:37.564 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:13:38.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:38.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:38.412809+0000) 2024-02-19T18:13:39.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:39.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:39.413052+0000) 2024-02-19T18:13:40.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:40.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:40.413308+0000) 2024-02-19T18:13:41.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:41.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:41.413566+0000) 2024-02-19T18:13:42.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:42.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:42.413696+0000) 2024-02-19T18:13:43.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:43.407+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:43.413834+0000) 2024-02-19T18:13:44.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:44.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:44.414046+0000) 2024-02-19T18:13:45.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:45.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:45.414298+0000) 2024-02-19T18:13:46.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:46.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:46.414564+0000) 2024-02-19T18:13:47.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:47.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:47.414784+0000) 2024-02-19T18:13:48.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:48.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:48.415010+0000) 2024-02-19T18:13:49.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:49.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:49.415239+0000) 2024-02-19T18:13:50.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:50.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:50.415404+0000) 2024-02-19T18:13:51.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:51.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:51.415570+0000) 2024-02-19T18:13:52.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:52.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:52.415795+0000) 2024-02-19T18:13:53.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:53.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:53.416026+0000) 2024-02-19T18:13:54.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:54.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:54.416198+0000) 2024-02-19T18:13:55.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:55.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:55.416365+0000) 2024-02-19T18:13:56.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:56.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:56.416565+0000) 2024-02-19T18:13:57.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:57.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:57.416737+0000) 2024-02-19T18:13:58.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:58.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:58.416981+0000) 2024-02-19T18:13:59.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:13:59.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:13:59.417214+0000) 2024-02-19T18:14:00.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:00.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:00.417430+0000) 2024-02-19T18:14:01.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:01.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:01.417656+0000) 2024-02-19T18:14:02.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:02.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:02.417899+0000) 2024-02-19T18:14:03.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:03.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:03.418097+0000) 2024-02-19T18:14:04.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:04.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:04.418341+0000) 2024-02-19T18:14:05.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:05.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:05.418596+0000) 2024-02-19T18:14:06.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:06.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:06.418880+0000) 2024-02-19T18:14:07.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:07.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:07.419085+0000) 2024-02-19T18:14:07.675 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:14:07.678 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:14:08.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:08.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:08.419337+0000) 2024-02-19T18:14:09.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:09.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:09.419575+0000) 2024-02-19T18:14:10.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:10.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:10.419814+0000) 2024-02-19T18:14:11.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:11.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:11.420071+0000) 2024-02-19T18:14:12.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:12.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:12.420300+0000) 2024-02-19T18:14:13.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:13.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:13.420556+0000) 2024-02-19T18:14:14.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:14.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:14.420815+0000) 2024-02-19T18:14:15.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:15.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:15.420940+0000) 2024-02-19T18:14:16.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:16.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:16.421199+0000) 2024-02-19T18:14:17.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:17.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:17.421434+0000) 2024-02-19T18:14:18.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:18.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:18.421724+0000) 2024-02-19T18:14:19.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:19.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:19.421890+0000) 2024-02-19T18:14:20.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:20.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:20.422066+0000) 2024-02-19T18:14:21.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:21.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:21.422253+0000) 2024-02-19T18:14:22.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:22.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:22.422451+0000) 2024-02-19T18:14:23.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:23.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:23.422703+0000) 2024-02-19T18:14:24.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:24.418+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:24.422981+0000) 2024-02-19T18:14:25.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:25.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:25.423190+0000) 2024-02-19T18:14:26.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:26.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:26.423438+0000) 2024-02-19T18:14:27.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:27.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:27.423621+0000) 2024-02-19T18:14:28.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:28.418+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:28.423861+0000) 2024-02-19T18:14:29.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:29.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:29.424088+0000) 2024-02-19T18:14:30.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:30.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:30.424372+0000) 2024-02-19T18:14:31.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:31.418+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:31.424545+0000) 2024-02-19T18:14:32.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:32.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:32.424774+0000) 2024-02-19T18:14:33.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:33.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:33.424953+0000) 2024-02-19T18:14:34.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:34.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:34.425195+0000) 2024-02-19T18:14:35.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:35.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:35.425424+0000) 2024-02-19T18:14:36.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:36.418+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:36.425629+0000) 2024-02-19T18:14:37.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:37.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:37.425876+0000) 2024-02-19T18:14:37.740 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:14:37.744 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:14:38.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:38.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:38.426134+0000) 2024-02-19T18:14:39.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:39.422+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:39.426377+0000) 2024-02-19T18:14:40.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:40.418+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:40.426550+0000) 2024-02-19T18:14:41.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:41.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:41.426774+0000) 2024-02-19T18:14:42.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:42.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:42.426950+0000) 2024-02-19T18:14:43.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:43.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:43.427083+0000) 2024-02-19T18:14:44.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:44.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:44.427241+0000) 2024-02-19T18:14:45.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:45.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:45.427423+0000) 2024-02-19T18:14:46.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:46.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:46.427710+0000) 2024-02-19T18:14:47.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:47.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:47.427965+0000) 2024-02-19T18:14:48.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:48.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:48.428144+0000) 2024-02-19T18:14:49.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:49.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:49.428391+0000) 2024-02-19T18:14:50.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:50.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:50.428596+0000) 2024-02-19T18:14:51.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:51.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:51.428864+0000) 2024-02-19T18:14:52.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:52.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:52.429036+0000) 2024-02-19T18:14:53.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:53.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:53.429214+0000) 2024-02-19T18:14:54.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:54.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:54.429364+0000) 2024-02-19T18:14:55.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:55.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:55.429589+0000) 2024-02-19T18:14:56.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:56.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:56.429842+0000) 2024-02-19T18:14:57.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:57.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:57.430017+0000) 2024-02-19T18:14:58.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:58.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:58.430259+0000) 2024-02-19T18:14:59.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:14:59.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:14:59.430492+0000) 2024-02-19T18:15:00.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:00.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:00.430802+0000) 2024-02-19T18:15:01.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:01.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:01.431074+0000) 2024-02-19T18:15:02.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:02.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:02.431228+0000) 2024-02-19T18:15:03.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:03.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:03.431344+0000) 2024-02-19T18:15:04.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:04.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:04.431590+0000) 2024-02-19T18:15:05.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:05.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:05.431811+0000) 2024-02-19T18:15:06.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:06.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:06.432064+0000) 2024-02-19T18:15:07.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:07.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:07.432294+0000) 2024-02-19T18:15:07.804 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:15:07.807 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:15:08.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:08.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:08.432535+0000) 2024-02-19T18:15:09.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:09.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:09.432739+0000) 2024-02-19T18:15:10.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:10.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:10.432897+0000) 2024-02-19T18:15:11.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:11.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:11.433029+0000) 2024-02-19T18:15:12.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:12.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:12.433253+0000) 2024-02-19T18:15:13.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:13.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:13.433429+0000) 2024-02-19T18:15:14.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:14.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:14.433635+0000) 2024-02-19T18:15:15.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:15.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:15.433826+0000) 2024-02-19T18:15:16.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:16.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:16.434040+0000) 2024-02-19T18:15:17.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:17.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:17.434250+0000) 2024-02-19T18:15:18.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:18.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:18.434438+0000) 2024-02-19T18:15:19.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:19.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:19.434621+0000) 2024-02-19T18:15:20.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:20.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:20.434863+0000) 2024-02-19T18:15:21.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:21.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:21.435037+0000) 2024-02-19T18:15:22.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:22.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:22.435310+0000) 2024-02-19T18:15:23.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:23.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:23.435502+0000) 2024-02-19T18:15:24.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:24.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:24.435704+0000) 2024-02-19T18:15:25.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:25.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:25.435877+0000) 2024-02-19T18:15:26.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:26.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:26.436064+0000) 2024-02-19T18:15:27.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:27.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:27.436295+0000) 2024-02-19T18:15:28.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:28.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:28.436491+0000) 2024-02-19T18:15:29.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:29.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:29.436740+0000) 2024-02-19T18:15:30.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:30.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:30.436946+0000) 2024-02-19T18:15:31.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:31.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:31.437186+0000) 2024-02-19T18:15:32.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:32.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:32.437419+0000) 2024-02-19T18:15:33.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:33.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:33.437622+0000) 2024-02-19T18:15:34.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:34.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:34.437818+0000) 2024-02-19T18:15:35.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:35.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:35.438042+0000) 2024-02-19T18:15:36.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:36.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:36.438318+0000) 2024-02-19T18:15:37.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:37.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:37.438512+0000) 2024-02-19T18:15:38.045 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:15:38.048 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:15:38.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:38.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:38.438789+0000) 2024-02-19T18:15:39.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:39.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:39.438956+0000) 2024-02-19T18:15:40.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:40.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:40.439204+0000) 2024-02-19T18:15:41.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:41.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:41.439443+0000) 2024-02-19T18:15:42.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:42.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:42.439577+0000) 2024-02-19T18:15:43.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:43.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:43.439821+0000) 2024-02-19T18:15:44.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:44.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:44.440004+0000) 2024-02-19T18:15:45.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:45.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:45.440198+0000) 2024-02-19T18:15:46.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:46.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:46.440394+0000) 2024-02-19T18:15:47.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:47.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:47.440620+0000) 2024-02-19T18:15:48.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:48.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:48.440827+0000) 2024-02-19T18:15:49.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:49.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:49.441066+0000) 2024-02-19T18:15:50.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:50.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:50.441250+0000) 2024-02-19T18:15:51.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:51.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:51.441506+0000) 2024-02-19T18:15:52.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:52.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:52.441745+0000) 2024-02-19T18:15:53.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:53.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:53.442047+0000) 2024-02-19T18:15:54.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:54.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:54.442311+0000) 2024-02-19T18:15:55.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:55.437+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:55.442542+0000) 2024-02-19T18:15:56.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:56.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:56.442847+0000) 2024-02-19T18:15:57.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:57.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:57.442998+0000) 2024-02-19T18:15:58.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:58.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:58.443273+0000) 2024-02-19T18:15:59.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:15:59.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:15:59.443506+0000) 2024-02-19T18:16:00.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:00.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:00.443684+0000) 2024-02-19T18:16:01.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:01.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:01.443924+0000) 2024-02-19T18:16:02.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:02.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:02.444117+0000) 2024-02-19T18:16:03.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:03.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:03.444345+0000) 2024-02-19T18:16:04.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:04.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:04.444586+0000) 2024-02-19T18:16:05.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:05.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:05.444830+0000) 2024-02-19T18:16:06.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:06.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:06.445082+0000) 2024-02-19T18:16:07.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:07.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:07.445326+0000) 2024-02-19T18:16:08.209 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:16:08.214 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:16:08.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:08.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:08.445587+0000) 2024-02-19T18:16:09.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:09.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:09.445751+0000) 2024-02-19T18:16:10.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:10.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:10.445991+0000) 2024-02-19T18:16:11.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:11.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:11.446243+0000) 2024-02-19T18:16:12.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:12.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:12.446461+0000) 2024-02-19T18:16:13.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:13.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:13.446683+0000) 2024-02-19T18:16:14.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:14.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:14.446916+0000) 2024-02-19T18:16:15.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:15.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:15.447150+0000) 2024-02-19T18:16:16.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:16.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:16.447395+0000) 2024-02-19T18:16:17.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:17.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:17.447627+0000) 2024-02-19T18:16:18.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:18.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:18.447869+0000) 2024-02-19T18:16:19.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:19.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:19.448098+0000) 2024-02-19T18:16:20.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:20.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:20.448270+0000) 2024-02-19T18:16:21.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:21.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:21.448554+0000) 2024-02-19T18:16:22.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:22.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:22.448789+0000) 2024-02-19T18:16:23.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:23.444+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:23.449032+0000) 2024-02-19T18:16:24.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:24.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:24.449267+0000) 2024-02-19T18:16:25.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:25.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:25.449504+0000) 2024-02-19T18:16:26.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:26.444+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:26.449666+0000) 2024-02-19T18:16:27.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:27.444+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:27.449850+0000) 2024-02-19T18:16:28.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:28.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:28.450093+0000) 2024-02-19T18:16:29.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:29.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:29.450276+0000) 2024-02-19T18:16:30.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:30.444+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:30.450543+0000) 2024-02-19T18:16:31.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:31.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:31.450796+0000) 2024-02-19T18:16:32.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:32.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:32.450976+0000) 2024-02-19T18:16:33.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:33.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:33.451173+0000) 2024-02-19T18:16:34.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:34.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:34.451419+0000) 2024-02-19T18:16:35.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:35.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:35.451623+0000) 2024-02-19T18:16:36.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:36.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:36.451835+0000) 2024-02-19T18:16:37.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:37.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:37.452032+0000) 2024-02-19T18:16:38.332 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:16:38.335 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:16:38.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:38.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:38.452290+0000) 2024-02-19T18:16:39.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:39.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:39.452520+0000) 2024-02-19T18:16:40.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:40.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:40.452761+0000) 2024-02-19T18:16:41.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:41.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:41.452998+0000) 2024-02-19T18:16:42.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:42.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:42.453166+0000) 2024-02-19T18:16:43.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:43.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:43.453337+0000) 2024-02-19T18:16:44.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:44.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:44.453510+0000) 2024-02-19T18:16:45.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:45.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:45.453683+0000) 2024-02-19T18:16:46.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:46.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:46.453943+0000) 2024-02-19T18:16:47.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:47.448+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:47.454141+0000) 2024-02-19T18:16:48.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:48.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:48.454384+0000) 2024-02-19T18:16:49.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:49.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:49.454651+0000) 2024-02-19T18:16:50.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:50.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:50.454900+0000) 2024-02-19T18:16:51.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:51.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:51.455156+0000) 2024-02-19T18:16:52.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:52.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:52.455334+0000) 2024-02-19T18:16:53.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:53.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:53.455571+0000) 2024-02-19T18:16:54.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:54.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:54.455759+0000) 2024-02-19T18:16:55.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:55.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:55.455998+0000) 2024-02-19T18:16:56.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:56.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:56.456184+0000) 2024-02-19T18:16:57.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:57.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:57.456445+0000) 2024-02-19T18:16:58.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:58.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:58.456656+0000) 2024-02-19T18:16:59.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:16:59.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:16:59.456808+0000) 2024-02-19T18:17:00.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:00.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:00.457009+0000) 2024-02-19T18:17:01.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:01.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:01.457238+0000) 2024-02-19T18:17:02.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:02.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:02.457484+0000) 2024-02-19T18:17:03.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:03.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:03.457714+0000) 2024-02-19T18:17:04.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:04.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:04.457965+0000) 2024-02-19T18:17:05.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:05.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:05.458217+0000) 2024-02-19T18:17:06.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:06.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:06.458487+0000) 2024-02-19T18:17:07.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:07.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:07.458791+0000) 2024-02-19T18:17:08.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:08.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:08.458929+0000) 2024-02-19T18:17:08.463 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:17:08.466 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:17:09.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:09.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:09.459089+0000) 2024-02-19T18:17:10.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:10.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:10.459315+0000) 2024-02-19T18:17:11.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:11.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:11.459588+0000) 2024-02-19T18:17:12.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:12.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:12.459787+0000) 2024-02-19T18:17:13.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:13.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:13.459938+0000) 2024-02-19T18:17:14.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:14.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:14.460125+0000) 2024-02-19T18:17:15.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:15.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:15.460339+0000) 2024-02-19T18:17:16.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:16.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:16.460617+0000) 2024-02-19T18:17:17.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:17.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:17.460873+0000) 2024-02-19T18:17:18.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:18.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:18.461032+0000) 2024-02-19T18:17:19.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:19.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:19.461268+0000) 2024-02-19T18:17:20.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:20.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:20.461472+0000) 2024-02-19T18:17:21.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:21.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:21.461660+0000) 2024-02-19T18:17:22.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:22.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:22.461887+0000) 2024-02-19T18:17:23.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:23.455+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:23.462112+0000) 2024-02-19T18:17:24.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:24.455+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:24.462354+0000) 2024-02-19T18:17:25.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:25.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:25.462581+0000) 2024-02-19T18:17:26.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:26.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:26.462814+0000) 2024-02-19T18:17:27.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:27.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:27.463055+0000) 2024-02-19T18:17:28.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:28.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:28.463217+0000) 2024-02-19T18:17:29.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:29.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:29.463403+0000) 2024-02-19T18:17:30.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:30.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:30.463569+0000) 2024-02-19T18:17:31.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:31.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:31.463802+0000) 2024-02-19T18:17:32.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:32.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:32.464049+0000) 2024-02-19T18:17:33.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:33.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:33.464310+0000) 2024-02-19T18:17:34.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:34.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:34.464533+0000) 2024-02-19T18:17:35.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:35.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:35.464685+0000) 2024-02-19T18:17:36.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:36.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:36.464876+0000) 2024-02-19T18:17:37.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:37.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:37.465102+0000) 2024-02-19T18:17:38.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:38.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:38.465361+0000) 2024-02-19T18:17:38.569 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:17:38.573 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:17:39.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:39.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:39.465628+0000) 2024-02-19T18:17:40.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:40.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:40.465870+0000) 2024-02-19T18:17:41.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:41.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:41.466088+0000) 2024-02-19T18:17:42.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:42.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:42.466321+0000) 2024-02-19T18:17:43.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:43.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:43.466570+0000) 2024-02-19T18:17:44.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:44.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:44.466812+0000) 2024-02-19T18:17:45.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:45.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:45.467046+0000) 2024-02-19T18:17:46.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:46.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:46.467212+0000) 2024-02-19T18:17:47.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:47.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:47.467449+0000) 2024-02-19T18:17:48.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:48.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:48.467622+0000) 2024-02-19T18:17:49.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:49.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:49.467820+0000) 2024-02-19T18:17:50.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:50.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:50.468044+0000) 2024-02-19T18:17:51.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:51.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:51.468228+0000) 2024-02-19T18:17:52.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:52.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:52.468414+0000) 2024-02-19T18:17:53.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:53.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:53.468628+0000) 2024-02-19T18:17:54.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:54.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:54.468866+0000) 2024-02-19T18:17:55.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:55.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:55.469058+0000) 2024-02-19T18:17:56.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:56.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:56.469237+0000) 2024-02-19T18:17:57.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:57.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:57.469483+0000) 2024-02-19T18:17:58.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:58.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:58.469752+0000) 2024-02-19T18:17:59.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:17:59.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:17:59.469980+0000) 2024-02-19T18:18:00.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:00.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:00.470238+0000) 2024-02-19T18:18:01.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:01.463+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:01.470412+0000) 2024-02-19T18:18:02.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:02.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:02.470658+0000) 2024-02-19T18:18:03.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:03.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:03.470873+0000) 2024-02-19T18:18:04.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:04.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:04.471057+0000) 2024-02-19T18:18:05.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:05.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:05.471293+0000) 2024-02-19T18:18:06.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:06.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:06.471491+0000) 2024-02-19T18:18:07.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:07.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:07.471738+0000) 2024-02-19T18:18:08.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:08.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:08.471922+0000) 2024-02-19T18:18:08.642 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:18:08.646 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:18:09.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:09.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:09.472103+0000) 2024-02-19T18:18:10.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:10.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:10.472327+0000) 2024-02-19T18:18:11.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:11.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:11.472483+0000) 2024-02-19T18:18:12.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:12.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:12.472724+0000) 2024-02-19T18:18:13.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:13.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:13.472935+0000) 2024-02-19T18:18:14.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:14.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:14.473159+0000) 2024-02-19T18:18:15.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:15.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:15.473347+0000) 2024-02-19T18:18:16.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:16.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:16.473539+0000) 2024-02-19T18:18:17.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:17.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:17.473763+0000) 2024-02-19T18:18:18.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:18.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:18.473975+0000) 2024-02-19T18:18:19.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:19.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:19.474159+0000) 2024-02-19T18:18:20.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:20.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:20.474409+0000) 2024-02-19T18:18:21.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:21.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:21.474632+0000) 2024-02-19T18:18:22.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:22.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:22.474826+0000) 2024-02-19T18:18:23.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:23.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:23.474970+0000) 2024-02-19T18:18:24.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:24.474+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:24.475195+0000) 2024-02-19T18:18:25.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:25.474+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:25.475424+0000) 2024-02-19T18:18:26.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:26.474+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:26.475634+0000) 2024-02-19T18:18:27.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:27.474+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:27.475900+0000) 2024-02-19T18:18:28.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:28.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:28.476147+0000) 2024-02-19T18:18:29.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:29.474+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:29.476405+0000) 2024-02-19T18:18:30.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:30.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:30.476639+0000) 2024-02-19T18:18:31.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:31.474+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:31.476909+0000) 2024-02-19T18:18:32.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:32.474+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:32.477154+0000) 2024-02-19T18:18:33.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:33.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:33.477306+0000) 2024-02-19T18:18:34.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:34.474+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:34.477468+0000) 2024-02-19T18:18:35.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:35.474+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:35.477677+0000) 2024-02-19T18:18:36.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:36.474+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:36.477940+0000) 2024-02-19T18:18:37.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:37.474+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:37.478226+0000) 2024-02-19T18:18:38.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:38.474+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:38.478396+0000) 2024-02-19T18:18:38.698 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:18:38.701 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:18:39.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:39.474+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:39.478581+0000) 2024-02-19T18:18:40.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:40.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:40.478808+0000) 2024-02-19T18:18:41.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:41.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:41.479007+0000) 2024-02-19T18:18:42.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:42.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:42.479203+0000) 2024-02-19T18:18:43.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:43.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:43.479358+0000) 2024-02-19T18:18:44.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:44.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:44.479598+0000) 2024-02-19T18:18:45.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:45.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:45.479805+0000) 2024-02-19T18:18:46.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:46.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:46.480065+0000) 2024-02-19T18:18:47.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:47.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:47.480331+0000) 2024-02-19T18:18:48.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:48.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:48.480537+0000) 2024-02-19T18:18:49.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:49.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:49.480783+0000) 2024-02-19T18:18:50.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:50.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:50.480966+0000) 2024-02-19T18:18:51.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:51.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:51.481186+0000) 2024-02-19T18:18:52.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:52.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:52.481338+0000) 2024-02-19T18:18:53.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:53.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:53.481545+0000) 2024-02-19T18:18:54.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:54.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:54.481788+0000) 2024-02-19T18:18:55.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:55.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:55.481984+0000) 2024-02-19T18:18:56.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:56.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:56.482189+0000) 2024-02-19T18:18:57.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:57.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:57.482406+0000) 2024-02-19T18:18:58.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:58.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:58.482644+0000) 2024-02-19T18:18:59.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:18:59.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:18:59.482867+0000) 2024-02-19T18:19:00.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:00.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:00.483143+0000) 2024-02-19T18:19:01.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:01.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:01.483383+0000) 2024-02-19T18:19:02.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:02.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:02.483648+0000) 2024-02-19T18:19:03.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:03.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:03.483861+0000) 2024-02-19T18:19:04.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:04.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:04.484046+0000) 2024-02-19T18:19:05.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:05.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:05.484249+0000) 2024-02-19T18:19:06.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:06.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:06.484537+0000) 2024-02-19T18:19:07.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:07.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:07.484802+0000) 2024-02-19T18:19:08.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:08.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:08.485037+0000) 2024-02-19T18:19:08.754 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:19:08.758 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:19:09.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:09.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:09.485205+0000) 2024-02-19T18:19:10.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:10.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:10.485377+0000) 2024-02-19T18:19:11.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:11.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:11.485548+0000) 2024-02-19T18:19:12.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:12.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:12.485749+0000) 2024-02-19T18:19:13.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:13.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:13.485982+0000) 2024-02-19T18:19:14.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:14.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:14.486219+0000) 2024-02-19T18:19:15.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:15.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:15.486438+0000) 2024-02-19T18:19:16.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:16.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:16.486752+0000) 2024-02-19T18:19:17.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:17.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:17.486946+0000) 2024-02-19T18:19:18.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:18.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:18.487163+0000) 2024-02-19T18:19:19.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:19.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:19.487385+0000) 2024-02-19T18:19:20.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:20.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:20.487636+0000) 2024-02-19T18:19:21.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:21.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:21.487841+0000) 2024-02-19T18:19:22.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:22.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:22.488062+0000) 2024-02-19T18:19:23.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:23.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:23.488319+0000) 2024-02-19T18:19:24.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:24.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:24.488500+0000) 2024-02-19T18:19:25.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:25.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:25.488677+0000) 2024-02-19T18:19:26.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:26.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:26.488936+0000) 2024-02-19T18:19:27.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:27.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:27.489134+0000) 2024-02-19T18:19:28.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:28.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:28.489252+0000) 2024-02-19T18:19:29.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:29.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:29.489476+0000) 2024-02-19T18:19:30.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:30.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:30.489705+0000) 2024-02-19T18:19:31.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:31.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:31.489955+0000) 2024-02-19T18:19:32.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:32.489+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:32.490204+0000) 2024-02-19T18:19:33.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:33.489+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:33.490449+0000) 2024-02-19T18:19:34.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:34.489+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:34.490686+0000) 2024-02-19T18:19:35.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:35.489+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:35.490928+0000) 2024-02-19T18:19:36.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:36.489+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:36.491184+0000) 2024-02-19T18:19:37.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:37.489+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:37.491395+0000) 2024-02-19T18:19:38.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:38.489+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:38.491650+0000) 2024-02-19T18:19:38.860 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:19:38.863 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:19:39.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:39.489+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:39.491880+0000) 2024-02-19T18:19:40.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:40.489+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:40.492134+0000) 2024-02-19T18:19:41.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:41.489+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:41.492365+0000) 2024-02-19T18:19:42.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:42.489+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:42.492543+0000) 2024-02-19T18:19:43.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:43.489+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:43.492821+0000) 2024-02-19T18:19:44.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:44.489+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:44.493089+0000) 2024-02-19T18:19:45.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:45.489+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:45.493320+0000) 2024-02-19T18:19:46.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:46.489+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:46.493565+0000) 2024-02-19T18:19:47.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:47.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:47.493818+0000) 2024-02-19T18:19:48.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:48.489+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:48.494074+0000) 2024-02-19T18:19:49.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:49.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:49.494316+0000) 2024-02-19T18:19:50.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:50.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:50.494550+0000) 2024-02-19T18:19:51.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:51.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:51.494828+0000) 2024-02-19T18:19:52.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:52.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:52.495077+0000) 2024-02-19T18:19:53.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:53.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:53.495317+0000) 2024-02-19T18:19:54.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:54.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:54.495593+0000) 2024-02-19T18:19:55.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:55.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:55.495823+0000) 2024-02-19T18:19:56.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:56.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:56.496005+0000) 2024-02-19T18:19:57.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:57.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:57.496252+0000) 2024-02-19T18:19:58.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:58.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:58.496503+0000) 2024-02-19T18:19:59.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:19:59.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:19:59.496776+0000) 2024-02-19T18:20:00.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:00.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:00.497012+0000) 2024-02-19T18:20:01.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:01.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:01.497211+0000) 2024-02-19T18:20:02.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:02.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:02.497456+0000) 2024-02-19T18:20:03.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:03.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:03.497724+0000) 2024-02-19T18:20:04.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:04.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:04.497965+0000) 2024-02-19T18:20:05.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:05.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:05.498220+0000) 2024-02-19T18:20:06.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:06.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:06.498462+0000) 2024-02-19T18:20:07.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:07.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:07.498782+0000) 2024-02-19T18:20:08.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:08.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:08.499025+0000) 2024-02-19T18:20:08.991 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:20:08.995 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:20:09.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:09.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:09.499224+0000) 2024-02-19T18:20:10.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:10.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:10.499472+0000) 2024-02-19T18:20:11.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:11.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:11.499739+0000) 2024-02-19T18:20:12.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:12.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:12.499908+0000) 2024-02-19T18:20:13.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:13.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:13.500159+0000) 2024-02-19T18:20:14.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:14.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:14.500334+0000) 2024-02-19T18:20:15.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:15.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:15.500562+0000) 2024-02-19T18:20:16.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:16.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:16.500827+0000) 2024-02-19T18:20:17.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:17.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:17.501078+0000) 2024-02-19T18:20:18.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:18.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:18.501315+0000) 2024-02-19T18:20:19.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:19.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:19.501585+0000) 2024-02-19T18:20:20.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:20.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:20.501777+0000) 2024-02-19T18:20:21.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:21.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:21.502017+0000) 2024-02-19T18:20:22.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:22.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:22.502258+0000) 2024-02-19T18:20:23.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:23.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:23.502493+0000) 2024-02-19T18:20:24.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:24.500+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:24.502779+0000) 2024-02-19T18:20:25.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:25.500+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:25.503005+0000) 2024-02-19T18:20:26.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:26.500+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:26.503254+0000) 2024-02-19T18:20:27.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:27.500+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:27.503512+0000) 2024-02-19T18:20:28.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:28.500+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:28.503767+0000) 2024-02-19T18:20:29.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:29.500+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:29.504000+0000) 2024-02-19T18:20:30.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:30.500+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:30.504213+0000) 2024-02-19T18:20:31.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:31.500+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:31.504460+0000) 2024-02-19T18:20:32.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:32.500+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:32.504699+0000) 2024-02-19T18:20:33.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:33.500+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:33.504936+0000) 2024-02-19T18:20:34.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:34.500+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:34.505166+0000) 2024-02-19T18:20:35.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:35.500+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:35.505403+0000) 2024-02-19T18:20:36.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:36.500+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:36.505651+0000) 2024-02-19T18:20:37.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:37.500+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:37.505857+0000) 2024-02-19T18:20:38.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:38.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:38.506126+0000) 2024-02-19T18:20:39.114 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:20:39.119 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:20:39.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:39.500+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:39.506355+0000) 2024-02-19T18:20:40.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:40.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:40.506605+0000) 2024-02-19T18:20:41.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:41.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:41.506814+0000) 2024-02-19T18:20:42.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:42.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:42.507013+0000) 2024-02-19T18:20:43.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:43.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:43.507191+0000) 2024-02-19T18:20:44.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:44.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:44.507389+0000) 2024-02-19T18:20:45.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:45.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:45.507638+0000) 2024-02-19T18:20:46.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:46.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:46.507915+0000) 2024-02-19T18:20:47.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:47.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:47.508136+0000) 2024-02-19T18:20:48.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:48.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:48.508341+0000) 2024-02-19T18:20:49.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:49.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:49.508546+0000) 2024-02-19T18:20:50.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:50.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:50.508758+0000) 2024-02-19T18:20:51.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:51.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:51.509038+0000) 2024-02-19T18:20:52.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:52.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:52.509247+0000) 2024-02-19T18:20:53.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:53.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:53.509489+0000) 2024-02-19T18:20:54.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:54.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:54.509743+0000) 2024-02-19T18:20:55.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:55.504+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:55.509993+0000) 2024-02-19T18:20:56.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:56.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:56.510253+0000) 2024-02-19T18:20:57.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:57.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:57.510487+0000) 2024-02-19T18:20:58.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:58.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:58.510695+0000) 2024-02-19T18:20:59.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:20:59.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:20:59.510987+0000) 2024-02-19T18:21:00.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:00.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:00.511173+0000) 2024-02-19T18:21:01.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:01.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:01.511306+0000) 2024-02-19T18:21:02.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:02.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:02.511572+0000) 2024-02-19T18:21:03.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:03.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:03.511773+0000) 2024-02-19T18:21:04.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:04.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:04.511990+0000) 2024-02-19T18:21:05.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:05.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:05.512236+0000) 2024-02-19T18:21:06.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:06.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:06.512490+0000) 2024-02-19T18:21:07.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:07.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:07.512730+0000) 2024-02-19T18:21:08.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:08.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:08.512976+0000) 2024-02-19T18:21:09.261 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:21:09.265 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:21:09.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:09.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:09.513220+0000) 2024-02-19T18:21:10.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:10.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:10.513466+0000) 2024-02-19T18:21:11.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:11.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:11.513664+0000) 2024-02-19T18:21:12.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:12.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:12.513908+0000) 2024-02-19T18:21:13.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:13.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:13.514106+0000) 2024-02-19T18:21:14.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:14.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:14.514377+0000) 2024-02-19T18:21:15.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:15.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:15.514632+0000) 2024-02-19T18:21:16.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:16.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:16.514878+0000) 2024-02-19T18:21:17.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:17.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:17.515008+0000) 2024-02-19T18:21:18.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:18.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:18.515279+0000) 2024-02-19T18:21:19.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:19.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:19.515486+0000) 2024-02-19T18:21:20.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:20.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:20.515762+0000) 2024-02-19T18:21:21.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:21.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:21.515972+0000) 2024-02-19T18:21:22.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:22.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:22.516214+0000) 2024-02-19T18:21:23.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:23.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:23.516461+0000) 2024-02-19T18:21:24.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:24.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:24.516747+0000) 2024-02-19T18:21:25.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:25.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:25.516959+0000) 2024-02-19T18:21:26.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:26.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:26.517212+0000) 2024-02-19T18:21:27.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:27.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:27.517466+0000) 2024-02-19T18:21:28.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:28.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:28.517657+0000) 2024-02-19T18:21:29.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:29.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:29.517918+0000) 2024-02-19T18:21:30.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:30.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:30.518156+0000) 2024-02-19T18:21:31.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:31.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:31.518418+0000) 2024-02-19T18:21:32.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:32.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:32.518551+0000) 2024-02-19T18:21:33.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:33.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:33.518854+0000) 2024-02-19T18:21:34.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:34.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:34.519058+0000) 2024-02-19T18:21:35.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:35.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:35.519262+0000) 2024-02-19T18:21:36.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:36.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:36.519459+0000) 2024-02-19T18:21:37.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:37.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:37.519703+0000) 2024-02-19T18:21:38.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:38.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:38.519954+0000) 2024-02-19T18:21:39.376 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:21:39.380 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:21:39.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:39.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:39.520250+0000) 2024-02-19T18:21:40.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:40.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:40.520456+0000) 2024-02-19T18:21:41.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:41.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:41.520657+0000) 2024-02-19T18:21:42.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:42.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:42.520918+0000) 2024-02-19T18:21:43.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:43.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:43.521179+0000) 2024-02-19T18:21:44.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:44.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:44.521484+0000) 2024-02-19T18:21:45.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:45.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:45.521768+0000) 2024-02-19T18:21:46.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:46.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:46.521984+0000) 2024-02-19T18:21:47.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:47.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:47.522145+0000) 2024-02-19T18:21:48.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:48.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:48.522349+0000) 2024-02-19T18:21:49.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:49.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:49.522598+0000) 2024-02-19T18:21:50.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:50.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:50.522882+0000) 2024-02-19T18:21:51.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:51.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:51.523072+0000) 2024-02-19T18:21:52.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:52.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:52.523308+0000) 2024-02-19T18:21:53.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:53.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:53.523521+0000) 2024-02-19T18:21:54.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:54.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:54.523730+0000) 2024-02-19T18:21:55.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:55.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:55.523956+0000) 2024-02-19T18:21:56.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:56.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:56.524205+0000) 2024-02-19T18:21:57.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:57.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:57.524424+0000) 2024-02-19T18:21:58.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:58.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:58.524666+0000) 2024-02-19T18:21:59.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:21:59.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:21:59.524912+0000) 2024-02-19T18:22:00.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:00.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:00.525125+0000) 2024-02-19T18:22:01.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:01.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:01.525320+0000) 2024-02-19T18:22:02.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:02.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:02.525590+0000) 2024-02-19T18:22:03.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:03.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:03.525818+0000) 2024-02-19T18:22:04.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:04.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:04.526088+0000) 2024-02-19T18:22:05.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:05.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:05.526354+0000) 2024-02-19T18:22:06.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:06.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:06.526539+0000) 2024-02-19T18:22:07.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:07.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:07.526751+0000) 2024-02-19T18:22:08.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:08.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:08.526945+0000) 2024-02-19T18:22:09.515 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:22:09.519 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:22:09.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:09.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:09.527126+0000) 2024-02-19T18:22:10.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:10.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:10.527388+0000) 2024-02-19T18:22:11.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:11.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:11.527637+0000) 2024-02-19T18:22:12.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:12.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:12.527810+0000) 2024-02-19T18:22:13.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:13.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:13.527989+0000) 2024-02-19T18:22:14.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:14.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:14.528188+0000) 2024-02-19T18:22:15.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:15.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:15.528446+0000) 2024-02-19T18:22:16.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:16.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:16.528626+0000) 2024-02-19T18:22:17.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:17.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:17.528864+0000) 2024-02-19T18:22:18.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:18.522+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:18.529037+0000) 2024-02-19T18:22:19.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:19.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:19.529275+0000) 2024-02-19T18:22:20.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:20.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:20.529515+0000) 2024-02-19T18:22:21.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:21.522+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:21.529743+0000) 2024-02-19T18:22:22.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:22.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:22.529971+0000) 2024-02-19T18:22:23.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:23.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:23.530209+0000) 2024-02-19T18:22:24.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:24.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:24.530452+0000) 2024-02-19T18:22:25.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:25.522+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:25.530646+0000) 2024-02-19T18:22:26.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:26.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:26.530900+0000) 2024-02-19T18:22:27.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:27.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:27.531123+0000) 2024-02-19T18:22:28.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:28.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:28.531362+0000) 2024-02-19T18:22:29.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:29.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:29.531611+0000) 2024-02-19T18:22:30.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:30.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:30.531869+0000) 2024-02-19T18:22:31.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:31.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:31.532043+0000) 2024-02-19T18:22:32.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:32.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:32.532291+0000) 2024-02-19T18:22:33.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:33.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:33.532508+0000) 2024-02-19T18:22:34.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:34.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:34.532769+0000) 2024-02-19T18:22:35.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:35.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:35.533031+0000) 2024-02-19T18:22:36.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:36.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:36.533286+0000) 2024-02-19T18:22:37.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:37.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:37.533521+0000) 2024-02-19T18:22:38.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:38.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:38.533760+0000) 2024-02-19T18:22:39.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:39.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:39.533991+0000) 2024-02-19T18:22:39.789 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:22:39.792 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:22:40.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:40.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:40.534224+0000) 2024-02-19T18:22:41.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:41.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:41.534448+0000) 2024-02-19T18:22:42.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:42.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:42.534682+0000) 2024-02-19T18:22:43.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:43.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:43.534970+0000) 2024-02-19T18:22:44.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:44.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:44.535285+0000) 2024-02-19T18:22:45.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:45.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:45.535551+0000) 2024-02-19T18:22:46.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:46.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:46.535740+0000) 2024-02-19T18:22:47.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:47.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:47.535985+0000) 2024-02-19T18:22:48.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:48.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:48.536203+0000) 2024-02-19T18:22:49.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:49.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:49.536453+0000) 2024-02-19T18:22:50.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:50.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:50.536647+0000) 2024-02-19T18:22:51.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:51.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:51.536912+0000) 2024-02-19T18:22:52.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:52.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:52.537147+0000) 2024-02-19T18:22:53.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:53.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:53.537385+0000) 2024-02-19T18:22:54.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:54.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:54.537584+0000) 2024-02-19T18:22:55.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:55.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:55.537846+0000) 2024-02-19T18:22:56.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:56.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:56.538095+0000) 2024-02-19T18:22:57.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:57.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:57.538266+0000) 2024-02-19T18:22:58.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:58.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:58.538519+0000) 2024-02-19T18:22:59.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:22:59.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:22:59.538779+0000) 2024-02-19T18:23:00.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:00.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:00.538913+0000) 2024-02-19T18:23:01.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:01.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:01.539094+0000) 2024-02-19T18:23:02.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:02.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:02.539320+0000) 2024-02-19T18:23:03.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:03.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:03.539567+0000) 2024-02-19T18:23:04.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:04.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:04.539768+0000) 2024-02-19T18:23:05.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:05.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:05.539994+0000) 2024-02-19T18:23:06.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:06.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:06.540262+0000) 2024-02-19T18:23:07.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:07.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:07.540499+0000) 2024-02-19T18:23:08.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:08.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:08.540710+0000) 2024-02-19T18:23:09.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:09.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:09.540955+0000) 2024-02-19T18:23:09.867 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:23:09.871 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:23:10.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:10.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:10.541190+0000) 2024-02-19T18:23:11.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:11.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:11.541431+0000) 2024-02-19T18:23:12.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:12.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:12.541707+0000) 2024-02-19T18:23:13.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:13.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:13.541907+0000) 2024-02-19T18:23:14.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:14.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:14.542180+0000) 2024-02-19T18:23:15.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:15.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:15.542419+0000) 2024-02-19T18:23:16.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:16.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:16.542683+0000) 2024-02-19T18:23:17.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:17.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:17.542916+0000) 2024-02-19T18:23:18.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:18.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:18.543084+0000) 2024-02-19T18:23:19.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:19.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:19.543280+0000) 2024-02-19T18:23:20.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:20.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:20.543551+0000) 2024-02-19T18:23:21.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:21.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:21.543802+0000) 2024-02-19T18:23:22.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:22.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:22.544040+0000) 2024-02-19T18:23:23.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:23.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:23.544234+0000) 2024-02-19T18:23:24.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:24.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:24.544475+0000) 2024-02-19T18:23:25.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:25.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:25.544713+0000) 2024-02-19T18:23:26.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:26.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:26.544970+0000) 2024-02-19T18:23:27.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:27.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:27.545133+0000) 2024-02-19T18:23:28.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:28.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:28.545377+0000) 2024-02-19T18:23:29.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:29.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:29.545613+0000) 2024-02-19T18:23:30.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:30.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:30.545853+0000) 2024-02-19T18:23:31.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:31.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:31.546093+0000) 2024-02-19T18:23:32.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:32.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:32.546315+0000) 2024-02-19T18:23:33.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:33.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:33.546553+0000) 2024-02-19T18:23:34.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:34.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:34.546756+0000) 2024-02-19T18:23:35.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:35.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:35.546965+0000) 2024-02-19T18:23:36.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:36.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:36.547169+0000) 2024-02-19T18:23:37.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:37.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:37.547349+0000) 2024-02-19T18:23:38.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:38.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:38.547592+0000) 2024-02-19T18:23:39.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:39.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:39.547855+0000) 2024-02-19T18:23:39.929 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:23:39.932 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:23:40.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:40.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:40.548097+0000) 2024-02-19T18:23:41.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:41.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:41.548327+0000) 2024-02-19T18:23:42.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:42.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:42.548588+0000) 2024-02-19T18:23:43.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:43.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:43.548810+0000) 2024-02-19T18:23:44.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:44.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:44.549063+0000) 2024-02-19T18:23:45.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:45.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:45.549329+0000) 2024-02-19T18:23:46.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:46.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:46.549575+0000) 2024-02-19T18:23:47.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:47.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:47.549840+0000) 2024-02-19T18:23:48.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:48.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:48.550052+0000) 2024-02-19T18:23:49.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:49.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:49.550258+0000) 2024-02-19T18:23:50.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:50.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:50.550461+0000) 2024-02-19T18:23:51.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:51.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:51.550700+0000) 2024-02-19T18:23:52.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:52.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:52.550929+0000) 2024-02-19T18:23:53.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:53.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:53.551082+0000) 2024-02-19T18:23:54.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:54.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:54.551356+0000) 2024-02-19T18:23:55.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:55.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:55.551556+0000) 2024-02-19T18:23:56.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:56.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:56.551721+0000) 2024-02-19T18:23:57.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:57.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:57.551996+0000) 2024-02-19T18:23:58.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:58.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:58.552168+0000) 2024-02-19T18:23:59.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:23:59.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:23:59.552391+0000) 2024-02-19T18:24:00.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:00.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:00.552652+0000) 2024-02-19T18:24:01.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:01.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:01.552933+0000) 2024-02-19T18:24:02.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:02.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:02.553189+0000) 2024-02-19T18:24:03.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:03.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:03.553418+0000) 2024-02-19T18:24:04.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:04.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:04.553653+0000) 2024-02-19T18:24:05.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:05.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:05.553881+0000) 2024-02-19T18:24:06.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:06.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:06.554061+0000) 2024-02-19T18:24:07.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:07.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:07.554240+0000) 2024-02-19T18:24:08.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:08.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:08.554487+0000) 2024-02-19T18:24:09.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:09.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:09.554701+0000) 2024-02-19T18:24:10.054 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:24:10.058 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:24:10.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:10.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:10.554958+0000) 2024-02-19T18:24:11.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:11.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:11.555221+0000) 2024-02-19T18:24:12.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:12.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:12.555451+0000) 2024-02-19T18:24:13.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:13.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:13.555591+0000) 2024-02-19T18:24:14.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:14.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:14.555832+0000) 2024-02-19T18:24:15.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:15.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:15.556104+0000) 2024-02-19T18:24:16.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:16.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:16.556353+0000) 2024-02-19T18:24:17.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:17.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:17.556551+0000) 2024-02-19T18:24:18.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:18.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:18.556800+0000) 2024-02-19T18:24:19.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:19.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:19.557049+0000) 2024-02-19T18:24:20.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:20.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:20.557305+0000) 2024-02-19T18:24:21.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:21.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:21.557561+0000) 2024-02-19T18:24:22.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:22.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:22.557782+0000) 2024-02-19T18:24:23.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:23.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:23.558027+0000) 2024-02-19T18:24:24.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:24.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:24.558223+0000) 2024-02-19T18:24:25.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:25.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:25.558424+0000) 2024-02-19T18:24:26.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:26.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:26.558598+0000) 2024-02-19T18:24:27.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:27.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:27.558843+0000) 2024-02-19T18:24:28.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:28.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:28.559021+0000) 2024-02-19T18:24:29.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:29.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:29.559203+0000) 2024-02-19T18:24:30.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:30.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:30.559522+0000) 2024-02-19T18:24:31.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:31.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:31.559786+0000) 2024-02-19T18:24:32.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:32.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:32.560041+0000) 2024-02-19T18:24:33.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:33.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:33.560293+0000) 2024-02-19T18:24:34.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:34.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:34.560563+0000) 2024-02-19T18:24:35.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:35.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:35.560734+0000) 2024-02-19T18:24:36.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:36.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:36.560908+0000) 2024-02-19T18:24:37.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:37.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:37.561148+0000) 2024-02-19T18:24:38.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:38.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:38.561404+0000) 2024-02-19T18:24:39.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:39.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:39.561666+0000) 2024-02-19T18:24:40.177 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:24:40.181 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:24:40.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:40.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:40.561904+0000) 2024-02-19T18:24:41.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:41.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:41.562141+0000) 2024-02-19T18:24:42.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:42.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:42.562375+0000) 2024-02-19T18:24:43.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:43.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:43.562643+0000) 2024-02-19T18:24:44.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:44.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:44.562855+0000) 2024-02-19T18:24:45.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:45.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:45.563090+0000) 2024-02-19T18:24:46.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:46.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:46.563310+0000) 2024-02-19T18:24:47.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:47.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:47.563579+0000) 2024-02-19T18:24:48.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:48.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:48.563761+0000) 2024-02-19T18:24:49.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:49.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:49.563997+0000) 2024-02-19T18:24:50.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:50.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:50.564197+0000) 2024-02-19T18:24:51.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:51.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:51.564457+0000) 2024-02-19T18:24:52.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:52.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:52.564710+0000) 2024-02-19T18:24:53.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:53.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:53.564969+0000) 2024-02-19T18:24:54.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:54.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:54.565260+0000) 2024-02-19T18:24:55.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:55.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:55.565447+0000) 2024-02-19T18:24:56.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:56.560+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:56.565721+0000) 2024-02-19T18:24:57.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:57.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:57.565896+0000) 2024-02-19T18:24:58.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:58.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:58.566155+0000) 2024-02-19T18:24:59.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:24:59.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:24:59.566323+0000) 2024-02-19T18:25:00.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:00.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:00.566595+0000) 2024-02-19T18:25:01.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:01.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:01.566803+0000) 2024-02-19T18:25:02.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:02.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:02.567036+0000) 2024-02-19T18:25:03.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:03.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:03.567254+0000) 2024-02-19T18:25:04.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:04.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:04.567510+0000) 2024-02-19T18:25:05.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:05.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:05.567773+0000) 2024-02-19T18:25:06.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:06.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:06.568066+0000) 2024-02-19T18:25:07.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:07.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:07.568267+0000) 2024-02-19T18:25:08.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:08.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:08.568536+0000) 2024-02-19T18:25:09.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:09.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:09.568769+0000) 2024-02-19T18:25:10.325 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:25:10.329 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:25:10.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:10.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:10.569012+0000) 2024-02-19T18:25:11.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:11.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:11.569284+0000) 2024-02-19T18:25:12.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:12.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:12.569485+0000) 2024-02-19T18:25:13.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:13.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:13.569744+0000) 2024-02-19T18:25:14.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:14.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:14.570032+0000) 2024-02-19T18:25:15.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:15.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:15.570333+0000) 2024-02-19T18:25:16.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:16.563+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:16.570638+0000) 2024-02-19T18:25:17.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:17.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:17.570962+0000) 2024-02-19T18:25:18.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:18.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:18.571208+0000) 2024-02-19T18:25:19.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:19.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:19.571386+0000) 2024-02-19T18:25:20.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:20.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:20.571638+0000) 2024-02-19T18:25:21.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:21.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:21.571845+0000) 2024-02-19T18:25:22.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:22.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:22.572101+0000) 2024-02-19T18:25:23.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:23.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:23.572349+0000) 2024-02-19T18:25:24.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:24.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:24.572630+0000) 2024-02-19T18:25:25.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:25.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:25.572903+0000) 2024-02-19T18:25:26.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:26.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:26.573198+0000) 2024-02-19T18:25:27.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:27.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:27.573432+0000) 2024-02-19T18:25:28.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:28.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:28.573630+0000) 2024-02-19T18:25:29.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:29.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:29.573893+0000) 2024-02-19T18:25:30.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:30.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:30.574152+0000) 2024-02-19T18:25:31.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:31.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:31.574462+0000) 2024-02-19T18:25:32.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:32.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:32.574791+0000) 2024-02-19T18:25:33.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:33.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:33.575108+0000) 2024-02-19T18:25:34.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:34.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:34.575343+0000) 2024-02-19T18:25:35.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:35.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:35.575545+0000) 2024-02-19T18:25:36.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:36.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:36.575758+0000) 2024-02-19T18:25:37.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:37.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:37.575955+0000) 2024-02-19T18:25:38.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:38.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:38.576212+0000) 2024-02-19T18:25:39.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:39.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:39.576501+0000) 2024-02-19T18:25:40.498 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:25:40.503 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:25:40.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:40.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:40.576766+0000) 2024-02-19T18:25:41.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:41.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:41.577023+0000) 2024-02-19T18:25:42.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:42.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:42.577207+0000) 2024-02-19T18:25:43.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:43.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:43.577450+0000) 2024-02-19T18:25:44.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:44.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:44.577699+0000) 2024-02-19T18:25:45.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:45.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:45.577941+0000) 2024-02-19T18:25:46.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:46.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:46.578186+0000) 2024-02-19T18:25:47.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:47.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:47.578421+0000) 2024-02-19T18:25:48.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:48.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:48.578660+0000) 2024-02-19T18:25:49.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:49.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:49.578866+0000) 2024-02-19T18:25:50.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:50.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:50.579109+0000) 2024-02-19T18:25:51.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:51.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:51.579300+0000) 2024-02-19T18:25:52.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:52.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:52.579536+0000) 2024-02-19T18:25:53.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:53.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:53.579708+0000) 2024-02-19T18:25:54.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:54.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:54.579915+0000) 2024-02-19T18:25:55.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:55.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:55.580136+0000) 2024-02-19T18:25:56.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:56.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:56.580407+0000) 2024-02-19T18:25:57.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:57.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:57.580629+0000) 2024-02-19T18:25:58.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:58.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:58.580854+0000) 2024-02-19T18:25:59.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:25:59.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:25:59.581054+0000) 2024-02-19T18:26:00.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:00.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:00.581232+0000) 2024-02-19T18:26:01.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:01.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:01.581479+0000) 2024-02-19T18:26:02.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:02.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:02.581733+0000) 2024-02-19T18:26:03.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:03.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:03.581935+0000) 2024-02-19T18:26:04.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:04.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:04.582176+0000) 2024-02-19T18:26:05.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:05.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:05.582410+0000) 2024-02-19T18:26:06.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:06.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:06.582623+0000) 2024-02-19T18:26:07.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:07.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:07.582833+0000) 2024-02-19T18:26:08.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:08.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:08.583010+0000) 2024-02-19T18:26:09.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:09.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:09.583187+0000) 2024-02-19T18:26:10.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:10.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:10.583428+0000) 2024-02-19T18:26:10.644 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:26:10.647 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:26:11.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:11.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:11.583693+0000) 2024-02-19T18:26:12.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:12.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:12.583938+0000) 2024-02-19T18:26:13.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:13.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:13.584166+0000) 2024-02-19T18:26:14.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:14.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:14.584402+0000) 2024-02-19T18:26:15.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:15.578+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:15.584559+0000) 2024-02-19T18:26:16.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:16.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:16.584746+0000) 2024-02-19T18:26:17.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:17.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:17.584876+0000) 2024-02-19T18:26:18.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:18.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:18.585069+0000) 2024-02-19T18:26:19.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:19.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:19.585279+0000) 2024-02-19T18:26:20.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:20.578+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:20.585519+0000) 2024-02-19T18:26:21.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:21.578+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:21.585753+0000) 2024-02-19T18:26:22.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:22.578+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:22.585952+0000) 2024-02-19T18:26:23.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:23.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:23.586205+0000) 2024-02-19T18:26:24.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:24.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:24.586386+0000) 2024-02-19T18:26:25.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:25.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:25.586653+0000) 2024-02-19T18:26:26.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:26.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:26.586919+0000) 2024-02-19T18:26:27.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:27.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:27.587109+0000) 2024-02-19T18:26:28.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:28.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:28.587294+0000) 2024-02-19T18:26:29.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:29.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:29.587520+0000) 2024-02-19T18:26:30.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:30.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:30.587807+0000) 2024-02-19T18:26:31.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:31.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:31.587999+0000) 2024-02-19T18:26:32.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:32.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:32.588214+0000) 2024-02-19T18:26:33.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:33.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:33.588456+0000) 2024-02-19T18:26:34.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:34.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:34.588631+0000) 2024-02-19T18:26:35.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:35.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:35.588815+0000) 2024-02-19T18:26:36.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:36.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:36.589018+0000) 2024-02-19T18:26:37.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:37.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:37.589193+0000) 2024-02-19T18:26:38.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:38.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:38.589436+0000) 2024-02-19T18:26:39.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:39.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:39.589692+0000) 2024-02-19T18:26:40.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:40.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:40.589971+0000) 2024-02-19T18:26:40.736 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:26:40.739 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:26:41.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:41.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:41.590209+0000) 2024-02-19T18:26:42.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:42.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:42.590449+0000) 2024-02-19T18:26:43.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:43.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:43.590686+0000) 2024-02-19T18:26:44.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:44.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:44.590929+0000) 2024-02-19T18:26:45.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:45.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:45.591185+0000) 2024-02-19T18:26:46.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:46.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:46.591448+0000) 2024-02-19T18:26:47.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:47.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:47.591641+0000) 2024-02-19T18:26:48.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:48.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:48.591803+0000) 2024-02-19T18:26:49.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:49.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:49.592033+0000) 2024-02-19T18:26:50.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:50.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:50.592273+0000) 2024-02-19T18:26:51.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:51.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:51.592514+0000) 2024-02-19T18:26:52.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:52.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:52.592774+0000) 2024-02-19T18:26:53.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:53.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:53.593024+0000) 2024-02-19T18:26:54.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:54.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:54.593272+0000) 2024-02-19T18:26:55.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:55.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:55.593526+0000) 2024-02-19T18:26:56.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:56.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:56.593816+0000) 2024-02-19T18:26:57.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:57.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:57.594101+0000) 2024-02-19T18:26:58.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:58.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:58.594365+0000) 2024-02-19T18:26:59.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:26:59.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:26:59.594617+0000) 2024-02-19T18:27:00.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:00.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:00.594833+0000) 2024-02-19T18:27:01.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:01.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:01.595100+0000) 2024-02-19T18:27:02.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:02.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:02.595288+0000) 2024-02-19T18:27:03.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:03.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:03.595429+0000) 2024-02-19T18:27:04.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:04.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:04.595659+0000) 2024-02-19T18:27:05.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:05.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:05.595985+0000) 2024-02-19T18:27:06.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:06.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:06.596202+0000) 2024-02-19T18:27:07.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:07.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:07.596427+0000) 2024-02-19T18:27:08.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:08.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:08.596612+0000) 2024-02-19T18:27:09.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:09.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:09.596797+0000) 2024-02-19T18:27:10.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:10.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:10.596962+0000) 2024-02-19T18:27:10.873 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:27:10.876 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:27:11.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:11.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:11.597149+0000) 2024-02-19T18:27:12.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:12.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:12.597354+0000) 2024-02-19T18:27:13.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:13.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:13.597553+0000) 2024-02-19T18:27:14.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:14.593+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:14.597799+0000) 2024-02-19T18:27:15.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:15.593+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:15.598051+0000) 2024-02-19T18:27:16.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:16.593+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:16.598345+0000) 2024-02-19T18:27:17.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:17.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:17.598584+0000) 2024-02-19T18:27:18.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:18.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:18.598799+0000) 2024-02-19T18:27:19.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:19.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:19.598962+0000) 2024-02-19T18:27:20.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:20.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:20.599204+0000) 2024-02-19T18:27:21.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:21.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:21.599464+0000) 2024-02-19T18:27:22.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:22.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:22.599678+0000) 2024-02-19T18:27:23.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:23.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:23.599909+0000) 2024-02-19T18:27:24.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:24.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:24.600113+0000) 2024-02-19T18:27:25.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:25.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:25.600233+0000) 2024-02-19T18:27:26.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:26.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:26.600478+0000) 2024-02-19T18:27:27.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:27.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:27.600669+0000) 2024-02-19T18:27:28.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:28.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:28.600937+0000) 2024-02-19T18:27:29.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:29.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:29.601145+0000) 2024-02-19T18:27:30.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:30.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:30.601384+0000) 2024-02-19T18:27:31.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:31.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:31.601654+0000) 2024-02-19T18:27:32.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:32.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:32.601852+0000) 2024-02-19T18:27:33.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:33.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:33.602100+0000) 2024-02-19T18:27:34.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:34.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:34.602291+0000) 2024-02-19T18:27:35.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:35.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:35.602464+0000) 2024-02-19T18:27:36.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:36.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:36.602739+0000) 2024-02-19T18:27:37.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:37.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:37.602934+0000) 2024-02-19T18:27:38.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:38.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:38.603188+0000) 2024-02-19T18:27:39.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:39.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:39.603399+0000) 2024-02-19T18:27:40.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:40.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:40.603678+0000) 2024-02-19T18:27:41.004 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:27:41.008 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:27:41.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:41.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:41.603850+0000) 2024-02-19T18:27:42.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:42.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:42.604079+0000) 2024-02-19T18:27:43.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:43.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:43.604352+0000) 2024-02-19T18:27:44.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:44.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:44.604611+0000) 2024-02-19T18:27:45.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:45.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:45.604867+0000) 2024-02-19T18:27:46.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:46.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:46.605121+0000) 2024-02-19T18:27:47.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:47.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:47.605395+0000) 2024-02-19T18:27:48.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:48.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:48.605619+0000) 2024-02-19T18:27:49.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:49.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:49.605874+0000) 2024-02-19T18:27:50.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:50.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:50.606116+0000) 2024-02-19T18:27:51.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:51.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:51.606383+0000) 2024-02-19T18:27:52.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:52.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:52.606613+0000) 2024-02-19T18:27:53.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:53.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:53.606803+0000) 2024-02-19T18:27:54.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:54.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:54.607000+0000) 2024-02-19T18:27:55.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:55.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:55.607134+0000) 2024-02-19T18:27:56.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:56.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:56.607376+0000) 2024-02-19T18:27:57.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:57.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:57.607624+0000) 2024-02-19T18:27:58.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:58.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:58.607889+0000) 2024-02-19T18:27:59.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:27:59.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:27:59.608098+0000) 2024-02-19T18:28:00.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:00.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:00.608347+0000) 2024-02-19T18:28:01.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:01.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:01.608583+0000) 2024-02-19T18:28:02.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:02.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:02.608862+0000) 2024-02-19T18:28:03.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:03.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:03.609132+0000) 2024-02-19T18:28:04.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:04.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:04.609371+0000) 2024-02-19T18:28:05.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:05.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:05.609610+0000) 2024-02-19T18:28:06.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:06.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:06.609880+0000) 2024-02-19T18:28:07.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:07.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:07.610049+0000) 2024-02-19T18:28:08.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:08.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:08.610226+0000) 2024-02-19T18:28:09.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:09.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:09.610422+0000) 2024-02-19T18:28:10.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:10.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:10.610622+0000) 2024-02-19T18:28:11.085 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:28:11.089 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:28:11.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:11.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:11.610855+0000) 2024-02-19T18:28:12.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:12.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:12.611079+0000) 2024-02-19T18:28:13.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:13.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:13.611329+0000) 2024-02-19T18:28:14.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:14.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:14.611571+0000) 2024-02-19T18:28:15.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:15.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:15.611824+0000) 2024-02-19T18:28:16.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:16.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:16.612041+0000) 2024-02-19T18:28:17.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:17.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:17.612303+0000) 2024-02-19T18:28:18.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:18.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:18.612580+0000) 2024-02-19T18:28:19.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:19.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:19.612823+0000) 2024-02-19T18:28:20.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:20.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:20.613077+0000) 2024-02-19T18:28:21.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:21.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:21.613336+0000) 2024-02-19T18:28:22.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:22.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:22.613524+0000) 2024-02-19T18:28:23.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:23.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:23.613694+0000) 2024-02-19T18:28:24.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:24.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:24.613936+0000) 2024-02-19T18:28:25.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:25.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:25.614170+0000) 2024-02-19T18:28:26.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:26.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:26.614356+0000) 2024-02-19T18:28:27.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:27.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:27.614596+0000) 2024-02-19T18:28:28.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:28.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:28.614829+0000) 2024-02-19T18:28:29.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:29.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:29.614953+0000) 2024-02-19T18:28:30.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:30.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:30.615216+0000) 2024-02-19T18:28:31.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:31.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:31.615423+0000) 2024-02-19T18:28:32.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:32.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:32.615661+0000) 2024-02-19T18:28:33.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:33.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:33.615850+0000) 2024-02-19T18:28:34.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:34.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:34.616134+0000) 2024-02-19T18:28:35.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:35.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:35.616378+0000) 2024-02-19T18:28:36.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:36.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:36.616636+0000) 2024-02-19T18:28:37.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:37.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:37.616904+0000) 2024-02-19T18:28:38.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:38.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:38.617111+0000) 2024-02-19T18:28:39.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:39.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:39.617323+0000) 2024-02-19T18:28:40.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:40.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:40.617565+0000) 2024-02-19T18:28:41.216 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:28:41.221 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:28:41.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:41.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:41.617818+0000) 2024-02-19T18:28:42.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:42.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:42.617992+0000) 2024-02-19T18:28:43.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:43.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:43.618243+0000) 2024-02-19T18:28:44.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:44.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:44.618490+0000) 2024-02-19T18:28:45.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:45.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:45.618772+0000) 2024-02-19T18:28:46.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:46.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:46.618964+0000) 2024-02-19T18:28:47.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:47.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:47.619193+0000) 2024-02-19T18:28:48.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:48.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:48.619466+0000) 2024-02-19T18:28:49.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:49.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:49.619669+0000) 2024-02-19T18:28:50.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:50.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:50.619933+0000) 2024-02-19T18:28:51.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:51.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:51.620189+0000) 2024-02-19T18:28:52.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:52.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:52.620443+0000) 2024-02-19T18:28:53.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:53.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:53.620696+0000) 2024-02-19T18:28:54.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:54.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:54.620939+0000) 2024-02-19T18:28:55.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:55.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:55.621195+0000) 2024-02-19T18:28:56.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:56.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:56.621430+0000) 2024-02-19T18:28:57.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:57.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:57.621640+0000) 2024-02-19T18:28:58.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:58.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:58.621776+0000) 2024-02-19T18:28:59.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:28:59.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:28:59.622001+0000) 2024-02-19T18:29:00.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:00.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:00.622255+0000) 2024-02-19T18:29:01.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:01.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:01.622513+0000) 2024-02-19T18:29:02.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:02.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:02.622740+0000) 2024-02-19T18:29:03.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:03.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:03.622973+0000) 2024-02-19T18:29:04.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:04.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:04.623144+0000) 2024-02-19T18:29:05.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:05.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:05.623310+0000) 2024-02-19T18:29:06.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:06.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:06.623543+0000) 2024-02-19T18:29:07.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:07.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:07.623792+0000) 2024-02-19T18:29:08.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:08.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:08.624035+0000) 2024-02-19T18:29:09.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:09.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:09.624279+0000) 2024-02-19T18:29:10.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:10.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:10.624543+0000) 2024-02-19T18:29:11.362 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:29:11.367 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:29:11.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:11.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:11.624786+0000) 2024-02-19T18:29:12.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:12.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:12.625031+0000) 2024-02-19T18:29:13.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:13.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:13.625282+0000) 2024-02-19T18:29:14.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:14.619+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:14.625463+0000) 2024-02-19T18:29:15.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:15.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:15.625646+0000) 2024-02-19T18:29:16.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:16.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:16.625876+0000) 2024-02-19T18:29:17.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:17.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:17.626144+0000) 2024-02-19T18:29:18.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:18.619+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:18.626413+0000) 2024-02-19T18:29:19.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:19.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:19.626691+0000) 2024-02-19T18:29:20.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:20.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:20.626925+0000) 2024-02-19T18:29:21.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:21.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:21.627099+0000) 2024-02-19T18:29:22.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:22.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:22.627264+0000) 2024-02-19T18:29:23.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:23.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:23.627444+0000) 2024-02-19T18:29:24.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:24.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:24.627662+0000) 2024-02-19T18:29:25.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:25.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:25.627911+0000) 2024-02-19T18:29:26.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:26.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:26.628147+0000) 2024-02-19T18:29:27.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:27.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:27.628389+0000) 2024-02-19T18:29:28.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:28.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:28.628577+0000) 2024-02-19T18:29:29.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:29.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:29.628855+0000) 2024-02-19T18:29:30.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:30.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:30.629135+0000) 2024-02-19T18:29:31.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:31.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:31.629387+0000) 2024-02-19T18:29:32.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:32.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:32.629646+0000) 2024-02-19T18:29:33.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:33.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:33.629912+0000) 2024-02-19T18:29:34.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:34.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:34.630182+0000) 2024-02-19T18:29:35.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:35.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:35.630450+0000) 2024-02-19T18:29:36.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:36.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:36.630746+0000) 2024-02-19T18:29:37.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:37.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:37.631001+0000) 2024-02-19T18:29:38.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:38.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:38.631246+0000) 2024-02-19T18:29:39.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:39.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:39.631492+0000) 2024-02-19T18:29:40.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:40.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:40.631775+0000) 2024-02-19T18:29:41.502 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:29:41.505 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:29:41.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:41.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:41.632014+0000) 2024-02-19T18:29:42.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:42.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:42.632247+0000) 2024-02-19T18:29:43.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:43.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:43.632501+0000) 2024-02-19T18:29:44.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:44.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:44.632694+0000) 2024-02-19T18:29:45.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:45.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:45.632911+0000) 2024-02-19T18:29:46.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:46.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:46.633063+0000) 2024-02-19T18:29:47.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:47.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:47.633244+0000) 2024-02-19T18:29:48.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:48.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:48.633483+0000) 2024-02-19T18:29:49.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:49.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:49.633702+0000) 2024-02-19T18:29:50.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:50.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:50.633901+0000) 2024-02-19T18:29:51.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:51.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:51.634105+0000) 2024-02-19T18:29:52.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:52.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:52.634364+0000) 2024-02-19T18:29:53.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:53.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:53.634531+0000) 2024-02-19T18:29:54.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:54.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:54.634812+0000) 2024-02-19T18:29:55.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:55.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:55.635039+0000) 2024-02-19T18:29:56.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:56.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:56.635290+0000) 2024-02-19T18:29:57.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:57.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:57.635526+0000) 2024-02-19T18:29:58.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:58.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:58.635722+0000) 2024-02-19T18:29:59.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:29:59.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:29:59.635929+0000) 2024-02-19T18:30:00.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:00.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:00.636166+0000) 2024-02-19T18:30:01.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:01.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:01.636440+0000) 2024-02-19T18:30:02.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:02.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:02.636648+0000) 2024-02-19T18:30:03.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:03.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:03.636886+0000) 2024-02-19T18:30:04.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:04.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:04.637148+0000) 2024-02-19T18:30:05.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:05.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:05.637407+0000) 2024-02-19T18:30:06.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:06.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:06.637680+0000) 2024-02-19T18:30:07.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:07.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:07.637919+0000) 2024-02-19T18:30:08.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:08.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:08.638090+0000) 2024-02-19T18:30:09.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:09.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:09.638273+0000) 2024-02-19T18:30:10.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:10.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:10.638465+0000) 2024-02-19T18:30:11.591 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:30:11.596 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:30:11.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:11.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:11.638785+0000) 2024-02-19T18:30:12.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:12.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:12.639028+0000) 2024-02-19T18:30:13.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:13.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:13.639218+0000) 2024-02-19T18:30:14.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:14.634+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:14.639405+0000) 2024-02-19T18:30:15.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:15.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:15.639645+0000) 2024-02-19T18:30:16.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:16.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:16.639927+0000) 2024-02-19T18:30:17.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:17.634+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:17.640158+0000) 2024-02-19T18:30:18.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:18.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:18.640419+0000) 2024-02-19T18:30:19.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:19.634+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:19.640658+0000) 2024-02-19T18:30:20.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:20.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:20.640908+0000) 2024-02-19T18:30:21.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:21.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:21.641178+0000) 2024-02-19T18:30:22.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:22.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:22.641425+0000) 2024-02-19T18:30:23.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:23.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:23.641645+0000) 2024-02-19T18:30:24.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:24.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:24.641891+0000) 2024-02-19T18:30:25.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:25.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:25.642144+0000) 2024-02-19T18:30:26.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:26.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:26.642328+0000) 2024-02-19T18:30:27.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:27.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:27.642502+0000) 2024-02-19T18:30:28.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:28.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:28.642661+0000) 2024-02-19T18:30:29.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:29.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:29.642852+0000) 2024-02-19T18:30:30.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:30.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:30.643090+0000) 2024-02-19T18:30:31.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:31.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:31.643320+0000) 2024-02-19T18:30:32.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:32.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:32.643582+0000) 2024-02-19T18:30:33.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:33.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:33.643789+0000) 2024-02-19T18:30:34.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:34.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:34.643948+0000) 2024-02-19T18:30:35.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:35.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:35.644163+0000) 2024-02-19T18:30:36.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:36.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:36.644434+0000) 2024-02-19T18:30:37.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:37.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:37.644687+0000) 2024-02-19T18:30:38.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:38.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:38.644901+0000) 2024-02-19T18:30:39.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:39.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:39.645147+0000) 2024-02-19T18:30:40.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:40.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:40.645414+0000) 2024-02-19T18:30:41.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:41.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:41.645624+0000) 2024-02-19T18:30:41.731 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:30:41.736 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:30:42.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:42.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:42.645899+0000) 2024-02-19T18:30:43.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:43.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:43.646179+0000) 2024-02-19T18:30:44.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:44.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:44.646396+0000) 2024-02-19T18:30:45.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:45.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:45.646572+0000) 2024-02-19T18:30:46.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:46.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:46.646773+0000) 2024-02-19T18:30:47.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:47.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:47.646936+0000) 2024-02-19T18:30:48.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:48.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:48.647138+0000) 2024-02-19T18:30:49.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:49.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:49.647400+0000) 2024-02-19T18:30:50.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:50.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:50.647576+0000) 2024-02-19T18:30:51.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:51.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:51.647770+0000) 2024-02-19T18:30:52.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:52.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:52.648014+0000) 2024-02-19T18:30:53.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:53.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:53.648188+0000) 2024-02-19T18:30:54.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:54.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:54.648369+0000) 2024-02-19T18:30:55.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:55.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:55.648617+0000) 2024-02-19T18:30:56.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:56.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:56.648801+0000) 2024-02-19T18:30:57.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:57.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:57.649025+0000) 2024-02-19T18:30:58.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:58.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:58.649232+0000) 2024-02-19T18:30:59.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:30:59.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:30:59.649491+0000) 2024-02-19T18:31:00.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:00.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:00.649699+0000) 2024-02-19T18:31:01.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:01.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:01.649977+0000) 2024-02-19T18:31:02.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:02.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:02.650219+0000) 2024-02-19T18:31:03.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:03.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:03.650460+0000) 2024-02-19T18:31:04.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:04.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:04.650651+0000) 2024-02-19T18:31:05.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:05.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:05.650927+0000) 2024-02-19T18:31:06.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:06.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:06.651087+0000) 2024-02-19T18:31:07.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:07.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:07.651343+0000) 2024-02-19T18:31:08.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:08.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:08.651532+0000) 2024-02-19T18:31:09.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:09.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:09.651767+0000) 2024-02-19T18:31:10.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:10.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:10.651977+0000) 2024-02-19T18:31:11.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:11.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:11.652241+0000) 2024-02-19T18:31:11.792 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:31:11.796 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:31:12.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:12.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:12.652478+0000) 2024-02-19T18:31:13.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:13.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:13.652730+0000) 2024-02-19T18:31:14.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:14.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:14.652937+0000) 2024-02-19T18:31:15.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:15.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:15.653171+0000) 2024-02-19T18:31:16.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:16.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:16.653422+0000) 2024-02-19T18:31:17.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:17.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:17.653669+0000) 2024-02-19T18:31:18.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:18.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:18.653909+0000) 2024-02-19T18:31:19.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:19.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:19.654104+0000) 2024-02-19T18:31:20.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:20.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:20.654340+0000) 2024-02-19T18:31:21.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:21.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:21.654575+0000) 2024-02-19T18:31:22.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:22.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:22.654883+0000) 2024-02-19T18:31:23.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:23.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:23.655038+0000) 2024-02-19T18:31:24.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:24.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:24.655234+0000) 2024-02-19T18:31:25.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:25.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:25.655389+0000) 2024-02-19T18:31:26.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:26.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:26.655664+0000) 2024-02-19T18:31:27.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:27.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:27.655914+0000) 2024-02-19T18:31:28.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:28.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:28.656090+0000) 2024-02-19T18:31:29.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:29.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:29.656319+0000) 2024-02-19T18:31:30.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:30.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:30.656557+0000) 2024-02-19T18:31:31.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:31.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:31.656811+0000) 2024-02-19T18:31:32.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:32.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:32.657045+0000) 2024-02-19T18:31:33.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:33.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:33.657213+0000) 2024-02-19T18:31:34.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:34.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:34.657397+0000) 2024-02-19T18:31:35.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:35.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:35.657629+0000) 2024-02-19T18:31:36.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:36.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:36.657903+0000) 2024-02-19T18:31:37.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:37.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:37.658143+0000) 2024-02-19T18:31:38.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:38.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:38.658404+0000) 2024-02-19T18:31:39.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:39.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:39.658575+0000) 2024-02-19T18:31:40.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:40.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:40.658750+0000) 2024-02-19T18:31:41.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:41.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:41.658928+0000) 2024-02-19T18:31:41.900 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:31:41.903 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:31:42.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:42.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:42.659172+0000) 2024-02-19T18:31:43.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:43.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:43.659378+0000) 2024-02-19T18:31:44.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:44.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:44.659570+0000) 2024-02-19T18:31:45.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:45.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:45.659828+0000) 2024-02-19T18:31:46.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:46.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:46.660065+0000) 2024-02-19T18:31:47.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:47.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:47.660311+0000) 2024-02-19T18:31:48.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:48.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:48.660450+0000) 2024-02-19T18:31:49.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:49.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:49.660707+0000) 2024-02-19T18:31:50.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:50.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:50.660880+0000) 2024-02-19T18:31:51.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:51.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:51.661074+0000) 2024-02-19T18:31:52.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:52.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:52.661337+0000) 2024-02-19T18:31:53.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:53.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:53.661525+0000) 2024-02-19T18:31:54.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:54.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:54.661738+0000) 2024-02-19T18:31:55.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:55.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:55.661979+0000) 2024-02-19T18:31:56.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:56.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:56.662246+0000) 2024-02-19T18:31:57.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:57.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:57.662515+0000) 2024-02-19T18:31:58.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:58.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:58.662749+0000) 2024-02-19T18:31:59.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:31:59.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:31:59.662945+0000) 2024-02-19T18:32:00.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:00.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:00.663062+0000) 2024-02-19T18:32:01.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:01.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:01.663232+0000) 2024-02-19T18:32:02.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:02.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:02.663463+0000) 2024-02-19T18:32:03.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:03.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:03.663713+0000) 2024-02-19T18:32:04.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:04.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:04.663955+0000) 2024-02-19T18:32:05.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:05.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:05.664189+0000) 2024-02-19T18:32:06.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:06.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:06.664447+0000) 2024-02-19T18:32:07.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:07.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:07.664683+0000) 2024-02-19T18:32:08.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:08.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:08.664939+0000) 2024-02-19T18:32:09.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:09.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:09.665188+0000) 2024-02-19T18:32:10.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:10.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:10.665466+0000) 2024-02-19T18:32:11.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:11.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:11.665717+0000) 2024-02-19T18:32:12.030 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:32:12.034 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:32:12.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:12.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:12.665915+0000) 2024-02-19T18:32:13.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:13.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:13.666083+0000) 2024-02-19T18:32:14.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:14.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:14.666325+0000) 2024-02-19T18:32:15.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:15.660+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:15.666525+0000) 2024-02-19T18:32:16.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:16.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:16.666787+0000) 2024-02-19T18:32:17.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:17.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:17.666994+0000) 2024-02-19T18:32:18.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:18.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:18.667120+0000) 2024-02-19T18:32:19.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:19.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:19.667294+0000) 2024-02-19T18:32:20.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:20.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:20.667496+0000) 2024-02-19T18:32:21.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:21.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:21.667657+0000) 2024-02-19T18:32:22.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:22.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:22.667835+0000) 2024-02-19T18:32:23.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:23.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:23.668068+0000) 2024-02-19T18:32:24.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:24.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:24.668315+0000) 2024-02-19T18:32:25.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:25.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:25.668455+0000) 2024-02-19T18:32:26.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:26.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:26.668696+0000) 2024-02-19T18:32:27.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:27.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:27.668914+0000) 2024-02-19T18:32:28.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:28.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:28.669108+0000) 2024-02-19T18:32:29.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:29.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:29.669364+0000) 2024-02-19T18:32:30.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:30.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:30.669602+0000) 2024-02-19T18:32:31.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:31.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:31.669774+0000) 2024-02-19T18:32:32.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:32.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:32.669912+0000) 2024-02-19T18:32:33.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:33.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:33.670168+0000) 2024-02-19T18:32:34.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:34.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:34.670416+0000) 2024-02-19T18:32:35.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:35.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:35.670543+0000) 2024-02-19T18:32:36.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:36.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:36.670801+0000) 2024-02-19T18:32:37.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:37.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:37.670959+0000) 2024-02-19T18:32:38.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:38.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:38.671132+0000) 2024-02-19T18:32:39.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:39.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:39.671291+0000) 2024-02-19T18:32:40.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:40.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:40.671558+0000) 2024-02-19T18:32:41.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:41.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:41.671728+0000) 2024-02-19T18:32:42.153 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:32:42.156 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:32:42.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:42.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:42.671971+0000) 2024-02-19T18:32:43.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:43.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:43.672163+0000) 2024-02-19T18:32:44.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:44.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:44.672317+0000) 2024-02-19T18:32:45.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:45.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:45.672558+0000) 2024-02-19T18:32:46.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:46.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:46.672756+0000) 2024-02-19T18:32:47.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:47.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:47.672999+0000) 2024-02-19T18:32:48.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:48.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:48.673174+0000) 2024-02-19T18:32:49.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:49.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:49.673423+0000) 2024-02-19T18:32:50.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:50.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:50.673695+0000) 2024-02-19T18:32:51.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:51.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:51.673852+0000) 2024-02-19T18:32:52.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:52.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:52.674102+0000) 2024-02-19T18:32:53.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:53.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:53.674337+0000) 2024-02-19T18:32:54.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:54.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:54.674555+0000) 2024-02-19T18:32:55.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:55.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:55.674768+0000) 2024-02-19T18:32:56.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:56.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:56.674916+0000) 2024-02-19T18:32:57.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:57.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:57.675120+0000) 2024-02-19T18:32:58.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:58.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:58.675332+0000) 2024-02-19T18:32:59.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:32:59.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:32:59.675583+0000) 2024-02-19T18:33:00.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:00.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:00.675771+0000) 2024-02-19T18:33:01.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:01.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:01.676050+0000) 2024-02-19T18:33:02.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:02.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:02.676290+0000) 2024-02-19T18:33:03.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:03.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:03.676512+0000) 2024-02-19T18:33:04.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:04.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:04.676756+0000) 2024-02-19T18:33:05.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:05.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:05.676996+0000) 2024-02-19T18:33:06.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:06.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:06.677257+0000) 2024-02-19T18:33:07.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:07.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:07.677496+0000) 2024-02-19T18:33:08.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:08.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:08.677690+0000) 2024-02-19T18:33:09.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:09.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:09.677934+0000) 2024-02-19T18:33:10.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:10.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:10.678219+0000) 2024-02-19T18:33:11.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:11.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:11.678397+0000) 2024-02-19T18:33:12.234 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:33:12.237 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:33:12.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:12.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:12.678572+0000) 2024-02-19T18:33:13.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:13.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:13.678803+0000) 2024-02-19T18:33:14.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:14.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:14.679045+0000) 2024-02-19T18:33:15.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:15.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:15.679235+0000) 2024-02-19T18:33:16.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:16.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:16.679506+0000) 2024-02-19T18:33:17.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:17.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:17.679701+0000) 2024-02-19T18:33:18.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:18.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:18.679957+0000) 2024-02-19T18:33:19.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:19.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:19.680192+0000) 2024-02-19T18:33:20.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:20.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:20.680389+0000) 2024-02-19T18:33:21.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:21.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:21.680582+0000) 2024-02-19T18:33:22.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:22.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:22.680819+0000) 2024-02-19T18:33:23.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:23.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:23.681019+0000) 2024-02-19T18:33:24.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:24.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:24.681268+0000) 2024-02-19T18:33:25.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:25.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:25.681510+0000) 2024-02-19T18:33:26.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:26.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:26.681703+0000) 2024-02-19T18:33:27.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:27.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:27.681933+0000) 2024-02-19T18:33:28.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:28.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:28.682111+0000) 2024-02-19T18:33:29.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:29.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:29.682303+0000) 2024-02-19T18:33:30.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:30.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:30.682543+0000) 2024-02-19T18:33:31.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:31.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:31.682815+0000) 2024-02-19T18:33:32.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:32.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:32.682955+0000) 2024-02-19T18:33:33.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:33.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:33.683228+0000) 2024-02-19T18:33:34.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:34.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:34.683403+0000) 2024-02-19T18:33:35.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:35.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:35.683652+0000) 2024-02-19T18:33:36.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:36.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:36.683935+0000) 2024-02-19T18:33:37.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:37.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:37.684175+0000) 2024-02-19T18:33:38.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:38.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:38.684406+0000) 2024-02-19T18:33:39.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:39.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:39.684604+0000) 2024-02-19T18:33:40.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:40.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:40.684806+0000) 2024-02-19T18:33:41.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:41.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:41.684975+0000) 2024-02-19T18:33:42.357 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:33:42.361 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:33:42.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:42.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:42.685126+0000) 2024-02-19T18:33:43.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:43.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:43.685395+0000) 2024-02-19T18:33:44.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:44.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:44.685621+0000) 2024-02-19T18:33:45.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:45.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:45.685894+0000) 2024-02-19T18:33:46.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:46.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:46.686082+0000) 2024-02-19T18:33:47.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:47.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:47.686214+0000) 2024-02-19T18:33:48.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:48.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:48.686438+0000) 2024-02-19T18:33:49.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:49.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:49.686582+0000) 2024-02-19T18:33:50.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:50.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:50.686844+0000) 2024-02-19T18:33:51.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:51.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:51.687088+0000) 2024-02-19T18:33:52.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:52.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:52.687327+0000) 2024-02-19T18:33:53.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:53.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:53.687566+0000) 2024-02-19T18:33:54.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:54.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:54.687742+0000) 2024-02-19T18:33:55.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:55.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:55.687981+0000) 2024-02-19T18:33:56.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:56.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:56.688226+0000) 2024-02-19T18:33:57.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:57.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:57.688463+0000) 2024-02-19T18:33:58.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:58.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:58.688640+0000) 2024-02-19T18:33:59.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:33:59.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:33:59.688915+0000) 2024-02-19T18:34:00.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:00.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:00.689205+0000) 2024-02-19T18:34:01.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:01.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:01.689530+0000) 2024-02-19T18:34:02.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:02.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:02.689737+0000) 2024-02-19T18:34:03.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:03.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:03.690047+0000) 2024-02-19T18:34:04.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:04.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:04.690279+0000) 2024-02-19T18:34:05.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:05.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:05.690531+0000) 2024-02-19T18:34:06.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:06.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:06.690870+0000) 2024-02-19T18:34:07.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:07.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:07.691093+0000) 2024-02-19T18:34:08.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:08.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:08.691389+0000) 2024-02-19T18:34:09.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:09.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:09.691693+0000) 2024-02-19T18:34:10.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:10.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:10.691948+0000) 2024-02-19T18:34:11.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:11.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:11.692114+0000) 2024-02-19T18:34:12.480 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:34:12.483 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:34:12.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:12.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:12.692296+0000) 2024-02-19T18:34:13.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:13.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:13.692513+0000) 2024-02-19T18:34:14.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:14.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:14.692722+0000) 2024-02-19T18:34:15.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:15.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:15.692883+0000) 2024-02-19T18:34:16.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:16.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:16.693132+0000) 2024-02-19T18:34:17.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:17.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:17.693310+0000) 2024-02-19T18:34:18.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:18.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:18.693550+0000) 2024-02-19T18:34:19.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:19.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:19.693722+0000) 2024-02-19T18:34:20.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:20.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:20.693982+0000) 2024-02-19T18:34:21.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:21.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:21.694196+0000) 2024-02-19T18:34:22.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:22.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:22.694464+0000) 2024-02-19T18:34:23.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:23.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:23.694698+0000) 2024-02-19T18:34:24.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:24.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:24.694885+0000) 2024-02-19T18:34:25.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:25.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:25.695145+0000) 2024-02-19T18:34:26.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:26.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:26.695349+0000) 2024-02-19T18:34:27.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:27.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:27.695620+0000) 2024-02-19T18:34:28.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:28.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:28.695854+0000) 2024-02-19T18:34:29.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:29.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:29.696053+0000) 2024-02-19T18:34:30.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:30.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:30.696332+0000) 2024-02-19T18:34:31.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:31.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:31.696588+0000) 2024-02-19T18:34:32.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:32.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:32.696852+0000) 2024-02-19T18:34:33.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:33.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:33.697148+0000) 2024-02-19T18:34:34.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:34.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:34.697366+0000) 2024-02-19T18:34:35.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:35.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:35.697610+0000) 2024-02-19T18:34:36.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:36.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:36.697926+0000) 2024-02-19T18:34:37.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:37.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:37.698154+0000) 2024-02-19T18:34:38.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:38.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:38.698377+0000) 2024-02-19T18:34:39.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:39.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:39.698632+0000) 2024-02-19T18:34:40.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:40.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:40.698839+0000) 2024-02-19T18:34:41.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:41.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:41.699129+0000) 2024-02-19T18:34:42.603 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:34:42.606 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:34:42.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:42.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:42.699314+0000) 2024-02-19T18:34:43.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:43.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:43.699514+0000) 2024-02-19T18:34:44.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:44.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:44.699771+0000) 2024-02-19T18:34:45.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:45.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:45.699971+0000) 2024-02-19T18:34:46.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:46.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:46.700231+0000) 2024-02-19T18:34:47.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:47.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:47.700510+0000) 2024-02-19T18:34:48.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:48.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:48.700816+0000) 2024-02-19T18:34:49.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:49.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:49.701144+0000) 2024-02-19T18:34:50.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:50.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:50.701399+0000) 2024-02-19T18:34:51.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:51.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:51.701667+0000) 2024-02-19T18:34:52.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:52.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:52.701954+0000) 2024-02-19T18:34:53.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:53.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:53.702200+0000) 2024-02-19T18:34:54.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:54.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:54.702450+0000) 2024-02-19T18:34:55.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:55.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:55.702777+0000) 2024-02-19T18:34:56.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:56.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:56.702956+0000) 2024-02-19T18:34:57.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:57.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:57.703098+0000) 2024-02-19T18:34:58.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:58.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:58.703271+0000) 2024-02-19T18:34:59.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:34:59.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:34:59.703535+0000) 2024-02-19T18:35:00.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:00.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:00.703786+0000) 2024-02-19T18:35:01.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:01.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:01.704048+0000) 2024-02-19T18:35:02.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:02.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:02.704330+0000) 2024-02-19T18:35:03.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:03.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:03.704633+0000) 2024-02-19T18:35:04.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:04.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:04.704845+0000) 2024-02-19T18:35:05.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:05.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:05.705051+0000) 2024-02-19T18:35:06.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:06.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:06.705254+0000) 2024-02-19T18:35:07.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:07.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:07.705524+0000) 2024-02-19T18:35:08.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:08.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:08.705783+0000) 2024-02-19T18:35:09.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:09.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:09.706030+0000) 2024-02-19T18:35:10.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:10.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:10.706265+0000) 2024-02-19T18:35:11.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:11.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:11.706530+0000) 2024-02-19T18:35:12.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:12.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:12.706701+0000) 2024-02-19T18:35:12.741 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:35:12.745 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:35:13.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:13.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:13.706923+0000) 2024-02-19T18:35:14.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:14.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:14.707145+0000) 2024-02-19T18:35:15.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:15.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:15.707358+0000) 2024-02-19T18:35:16.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:16.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:16.707683+0000) 2024-02-19T18:35:17.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:17.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:17.707959+0000) 2024-02-19T18:35:18.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:18.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:18.708239+0000) 2024-02-19T18:35:19.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:19.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:19.708483+0000) 2024-02-19T18:35:20.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:20.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:20.708660+0000) 2024-02-19T18:35:21.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:21.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:21.708859+0000) 2024-02-19T18:35:22.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:22.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:22.709154+0000) 2024-02-19T18:35:23.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:23.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:23.709354+0000) 2024-02-19T18:35:24.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:24.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:24.709488+0000) 2024-02-19T18:35:25.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:25.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:25.709766+0000) 2024-02-19T18:35:26.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:26.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:26.709987+0000) 2024-02-19T18:35:27.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:27.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:27.710168+0000) 2024-02-19T18:35:28.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:28.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:28.710342+0000) 2024-02-19T18:35:29.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:29.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:29.710599+0000) 2024-02-19T18:35:30.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:30.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:30.710865+0000) 2024-02-19T18:35:31.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:31.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:31.711161+0000) 2024-02-19T18:35:32.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:32.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:32.711399+0000) 2024-02-19T18:35:33.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:33.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:33.711635+0000) 2024-02-19T18:35:34.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:34.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:34.711871+0000) 2024-02-19T18:35:35.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:35.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:35.712089+0000) 2024-02-19T18:35:36.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:36.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:36.712398+0000) 2024-02-19T18:35:37.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:37.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:37.712728+0000) 2024-02-19T18:35:38.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:38.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:38.712960+0000) 2024-02-19T18:35:39.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:39.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:39.713216+0000) 2024-02-19T18:35:40.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:40.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:40.713502+0000) 2024-02-19T18:35:41.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:41.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:41.713705+0000) 2024-02-19T18:35:42.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:42.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:42.713952+0000) 2024-02-19T18:35:43.040 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:35:43.043 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:35:43.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:43.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:43.714156+0000) 2024-02-19T18:35:44.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:44.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:44.714418+0000) 2024-02-19T18:35:45.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:45.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:45.714772+0000) 2024-02-19T18:35:46.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:46.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:46.714977+0000) 2024-02-19T18:35:47.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:47.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:47.715132+0000) 2024-02-19T18:35:48.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:48.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:48.715267+0000) 2024-02-19T18:35:49.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:49.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:49.715483+0000) 2024-02-19T18:35:50.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:50.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:50.715803+0000) 2024-02-19T18:35:51.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:51.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:51.716086+0000) 2024-02-19T18:35:52.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:52.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:52.716280+0000) 2024-02-19T18:35:53.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:53.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:53.716463+0000) 2024-02-19T18:35:54.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:54.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:54.716718+0000) 2024-02-19T18:35:55.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:55.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:55.717020+0000) 2024-02-19T18:35:56.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:56.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:56.717226+0000) 2024-02-19T18:35:57.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:57.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:57.717451+0000) 2024-02-19T18:35:58.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:58.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:58.717670+0000) 2024-02-19T18:35:59.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:35:59.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:35:59.717848+0000) 2024-02-19T18:36:00.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:00.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:00.718076+0000) 2024-02-19T18:36:01.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:01.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:01.718328+0000) 2024-02-19T18:36:02.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:02.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:02.718526+0000) 2024-02-19T18:36:03.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:03.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:03.718816+0000) 2024-02-19T18:36:04.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:04.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:04.719058+0000) 2024-02-19T18:36:05.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:05.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:05.719293+0000) 2024-02-19T18:36:06.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:06.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:06.719516+0000) 2024-02-19T18:36:07.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:07.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:07.719777+0000) 2024-02-19T18:36:08.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:08.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:08.719954+0000) 2024-02-19T18:36:09.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:09.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:09.720169+0000) 2024-02-19T18:36:10.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:10.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:10.720444+0000) 2024-02-19T18:36:11.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:11.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:11.720620+0000) 2024-02-19T18:36:12.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:12.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:12.720849+0000) 2024-02-19T18:36:13.096 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:36:13.100 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:36:13.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:13.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:13.721096+0000) 2024-02-19T18:36:14.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:14.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:14.721279+0000) 2024-02-19T18:36:15.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:15.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:15.721487+0000) 2024-02-19T18:36:16.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:16.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:16.721669+0000) 2024-02-19T18:36:17.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:17.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:17.721896+0000) 2024-02-19T18:36:18.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:18.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:18.722142+0000) 2024-02-19T18:36:19.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:19.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:19.722319+0000) 2024-02-19T18:36:20.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:20.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:20.722521+0000) 2024-02-19T18:36:21.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:21.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:21.722802+0000) 2024-02-19T18:36:22.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:22.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:22.723010+0000) 2024-02-19T18:36:23.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:23.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:23.723250+0000) 2024-02-19T18:36:24.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:24.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:24.723481+0000) 2024-02-19T18:36:25.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:25.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:25.723734+0000) 2024-02-19T18:36:26.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:26.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:26.723988+0000) 2024-02-19T18:36:27.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:27.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:27.724191+0000) 2024-02-19T18:36:28.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:28.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:28.724463+0000) 2024-02-19T18:36:29.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:29.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:29.724695+0000) 2024-02-19T18:36:30.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:30.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:30.724935+0000) 2024-02-19T18:36:31.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:31.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:31.725190+0000) 2024-02-19T18:36:32.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:32.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:32.725359+0000) 2024-02-19T18:36:33.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:33.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:33.725603+0000) 2024-02-19T18:36:34.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:34.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:34.725844+0000) 2024-02-19T18:36:35.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:35.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:35.726090+0000) 2024-02-19T18:36:36.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:36.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:36.726362+0000) 2024-02-19T18:36:37.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:37.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:37.726562+0000) 2024-02-19T18:36:38.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:38.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:38.726802+0000) 2024-02-19T18:36:39.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:39.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:39.727049+0000) 2024-02-19T18:36:40.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:40.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:40.727201+0000) 2024-02-19T18:36:41.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:41.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:41.727450+0000) 2024-02-19T18:36:42.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:42.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:42.727672+0000) 2024-02-19T18:36:43.218 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:36:43.221 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:36:43.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:43.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:43.727914+0000) 2024-02-19T18:36:44.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:44.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:44.728155+0000) 2024-02-19T18:36:45.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:45.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:45.728349+0000) 2024-02-19T18:36:46.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:46.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:46.728601+0000) 2024-02-19T18:36:47.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:47.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:47.728831+0000) 2024-02-19T18:36:48.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:48.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:48.729072+0000) 2024-02-19T18:36:49.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:49.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:49.729341+0000) 2024-02-19T18:36:50.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:50.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:50.729599+0000) 2024-02-19T18:36:51.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:51.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:51.729842+0000) 2024-02-19T18:36:52.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:52.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:52.730083+0000) 2024-02-19T18:36:53.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:53.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:53.730324+0000) 2024-02-19T18:36:54.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:54.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:54.730560+0000) 2024-02-19T18:36:55.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:55.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:55.730778+0000) 2024-02-19T18:36:56.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:56.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:56.730932+0000) 2024-02-19T18:36:57.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:57.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:57.731124+0000) 2024-02-19T18:36:58.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:58.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:58.731403+0000) 2024-02-19T18:36:59.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:36:59.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:36:59.731614+0000) 2024-02-19T18:37:00.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:00.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:00.731855+0000) 2024-02-19T18:37:01.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:01.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:01.732031+0000) 2024-02-19T18:37:02.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:02.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:02.732276+0000) 2024-02-19T18:37:03.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:03.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:03.732515+0000) 2024-02-19T18:37:04.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:04.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:04.732756+0000) 2024-02-19T18:37:05.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:05.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:05.732923+0000) 2024-02-19T18:37:06.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:06.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:06.733209+0000) 2024-02-19T18:37:07.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:07.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:07.733474+0000) 2024-02-19T18:37:08.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:08.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:08.733726+0000) 2024-02-19T18:37:09.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:09.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:09.733963+0000) 2024-02-19T18:37:10.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:10.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:10.734208+0000) 2024-02-19T18:37:11.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:11.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:11.734410+0000) 2024-02-19T18:37:12.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:12.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:12.734674+0000) 2024-02-19T18:37:13.331 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:37:13.335 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:37:13.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:13.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:13.734928+0000) 2024-02-19T18:37:14.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:14.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:14.735171+0000) 2024-02-19T18:37:15.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:15.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:15.735417+0000) 2024-02-19T18:37:16.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:16.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:16.735585+0000) 2024-02-19T18:37:17.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:17.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:17.735800+0000) 2024-02-19T18:37:18.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:18.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:18.736007+0000) 2024-02-19T18:37:19.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:19.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:19.736286+0000) 2024-02-19T18:37:20.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:20.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:20.736479+0000) 2024-02-19T18:37:21.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:21.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:21.736749+0000) 2024-02-19T18:37:22.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:22.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:22.736980+0000) 2024-02-19T18:37:23.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:23.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:23.737217+0000) 2024-02-19T18:37:24.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:24.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:24.737455+0000) 2024-02-19T18:37:25.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:25.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:25.737681+0000) 2024-02-19T18:37:26.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:26.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:26.737905+0000) 2024-02-19T18:37:27.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:27.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:27.738130+0000) 2024-02-19T18:37:28.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:28.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:28.738331+0000) 2024-02-19T18:37:29.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:29.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:29.738604+0000) 2024-02-19T18:37:30.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:30.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:30.738827+0000) 2024-02-19T18:37:31.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:31.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:31.738968+0000) 2024-02-19T18:37:32.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:32.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:32.739174+0000) 2024-02-19T18:37:33.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:33.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:33.739336+0000) 2024-02-19T18:37:34.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:34.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:34.739492+0000) 2024-02-19T18:37:35.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:35.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:35.739729+0000) 2024-02-19T18:37:36.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:36.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:36.739982+0000) 2024-02-19T18:37:37.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:37.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:37.740291+0000) 2024-02-19T18:37:38.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:38.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:38.740494+0000) 2024-02-19T18:37:39.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:39.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:39.740808+0000) 2024-02-19T18:37:40.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:40.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:40.741098+0000) 2024-02-19T18:37:41.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:41.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:41.741322+0000) 2024-02-19T18:37:42.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:42.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:42.741490+0000) 2024-02-19T18:37:43.453 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:37:43.456 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:37:43.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:43.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:43.741750+0000) 2024-02-19T18:37:44.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:44.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:44.742059+0000) 2024-02-19T18:37:45.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:45.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:45.742369+0000) 2024-02-19T18:37:46.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:46.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:46.742670+0000) 2024-02-19T18:37:47.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:47.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:47.742873+0000) 2024-02-19T18:37:48.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:48.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:48.743006+0000) 2024-02-19T18:37:49.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:49.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:49.743160+0000) 2024-02-19T18:37:50.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:50.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:50.743422+0000) 2024-02-19T18:37:51.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:51.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:51.743723+0000) 2024-02-19T18:37:52.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:52.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:52.744031+0000) 2024-02-19T18:37:53.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:53.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:53.744312+0000) 2024-02-19T18:37:54.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:54.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:54.744548+0000) 2024-02-19T18:37:55.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:55.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:55.744722+0000) 2024-02-19T18:37:56.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:56.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:56.744984+0000) 2024-02-19T18:37:57.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:57.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:57.745277+0000) 2024-02-19T18:37:58.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:58.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:58.745453+0000) 2024-02-19T18:37:59.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:37:59.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:37:59.745751+0000) 2024-02-19T18:38:00.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:00.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:00.745979+0000) 2024-02-19T18:38:01.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:01.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:01.746279+0000) 2024-02-19T18:38:02.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:02.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:02.746510+0000) 2024-02-19T18:38:03.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:03.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:03.746835+0000) 2024-02-19T18:38:04.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:04.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:04.747114+0000) 2024-02-19T18:38:05.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:05.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:05.747394+0000) 2024-02-19T18:38:06.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:06.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:06.747687+0000) 2024-02-19T18:38:07.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:07.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:07.747940+0000) 2024-02-19T18:38:08.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:08.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:08.748137+0000) 2024-02-19T18:38:09.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:09.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:09.748404+0000) 2024-02-19T18:38:10.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:10.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:10.748703+0000) 2024-02-19T18:38:11.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:11.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:11.748964+0000) 2024-02-19T18:38:12.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:12.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:12.749264+0000) 2024-02-19T18:38:13.634 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:38:13.637 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:38:13.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:13.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:13.749575+0000) 2024-02-19T18:38:14.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:14.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:14.749819+0000) 2024-02-19T18:38:15.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:15.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:15.750075+0000) 2024-02-19T18:38:16.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:16.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:16.750406+0000) 2024-02-19T18:38:17.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:17.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:17.750650+0000) 2024-02-19T18:38:18.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:18.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:18.750846+0000) 2024-02-19T18:38:19.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:19.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:19.751061+0000) 2024-02-19T18:38:20.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:20.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:20.751279+0000) 2024-02-19T18:38:21.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:21.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:21.751582+0000) 2024-02-19T18:38:22.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:22.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:22.751808+0000) 2024-02-19T18:38:23.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:23.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:23.751986+0000) 2024-02-19T18:38:24.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:24.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:24.752193+0000) 2024-02-19T18:38:25.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:25.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:25.752437+0000) 2024-02-19T18:38:26.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:26.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:26.752759+0000) 2024-02-19T18:38:27.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:27.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:27.752953+0000) 2024-02-19T18:38:28.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:28.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:28.753129+0000) 2024-02-19T18:38:29.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:29.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:29.753353+0000) 2024-02-19T18:38:30.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:30.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:30.753622+0000) 2024-02-19T18:38:31.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:31.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:31.753913+0000) 2024-02-19T18:38:32.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:32.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:32.754145+0000) 2024-02-19T18:38:33.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:33.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:33.754394+0000) 2024-02-19T18:38:34.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:34.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:34.754677+0000) 2024-02-19T18:38:35.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:35.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:35.754972+0000) 2024-02-19T18:38:36.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:36.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:36.755209+0000) 2024-02-19T18:38:37.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:37.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:37.755502+0000) 2024-02-19T18:38:38.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:38.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:38.755723+0000) 2024-02-19T18:38:39.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:39.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:39.755967+0000) 2024-02-19T18:38:40.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:40.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:40.756140+0000) 2024-02-19T18:38:41.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:41.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:41.756432+0000) 2024-02-19T18:38:42.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:42.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:42.756642+0000) 2024-02-19T18:38:43.698 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:38:43.702 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:38:43.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:43.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:43.756905+0000) 2024-02-19T18:38:44.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:44.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:44.757206+0000) 2024-02-19T18:38:45.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:45.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:45.757494+0000) 2024-02-19T18:38:46.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:46.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:46.757713+0000) 2024-02-19T18:38:47.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:47.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:47.757902+0000) 2024-02-19T18:38:48.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:48.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:48.758100+0000) 2024-02-19T18:38:49.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:49.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:49.758292+0000) 2024-02-19T18:38:50.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:50.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:50.758493+0000) 2024-02-19T18:38:51.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:51.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:51.758693+0000) 2024-02-19T18:38:52.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:52.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:52.758879+0000) 2024-02-19T18:38:53.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:53.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:53.759063+0000) 2024-02-19T18:38:54.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:54.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:54.759238+0000) 2024-02-19T18:38:55.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:55.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:55.759447+0000) 2024-02-19T18:38:56.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:56.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:56.759769+0000) 2024-02-19T18:38:57.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:57.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:57.759979+0000) 2024-02-19T18:38:58.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:58.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:58.760158+0000) 2024-02-19T18:38:59.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:38:59.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:38:59.760418+0000) 2024-02-19T18:39:00.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:00.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:00.760603+0000) 2024-02-19T18:39:01.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:01.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:01.760883+0000) 2024-02-19T18:39:02.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:02.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:02.761119+0000) 2024-02-19T18:39:03.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:03.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:03.761374+0000) 2024-02-19T18:39:04.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:04.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:04.761545+0000) 2024-02-19T18:39:05.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:05.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:05.761786+0000) 2024-02-19T18:39:06.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:06.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:06.761994+0000) 2024-02-19T18:39:07.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:07.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:07.762247+0000) 2024-02-19T18:39:08.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:08.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:08.762491+0000) 2024-02-19T18:39:09.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:09.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:09.762767+0000) 2024-02-19T18:39:10.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:10.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:10.763061+0000) 2024-02-19T18:39:11.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:11.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:11.763262+0000) 2024-02-19T18:39:12.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:12.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:12.763463+0000) 2024-02-19T18:39:13.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:13.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:13.763781+0000) 2024-02-19T18:39:13.829 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:39:13.833 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:39:14.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:14.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:14.763997+0000) 2024-02-19T18:39:15.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:15.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:15.764257+0000) 2024-02-19T18:39:16.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:16.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:16.764596+0000) 2024-02-19T18:39:17.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:17.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:17.764847+0000) 2024-02-19T18:39:18.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:18.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:18.765102+0000) 2024-02-19T18:39:19.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:19.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:19.765381+0000) 2024-02-19T18:39:20.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:20.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:20.765671+0000) 2024-02-19T18:39:21.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:21.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:21.765974+0000) 2024-02-19T18:39:22.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:22.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:22.766175+0000) 2024-02-19T18:39:23.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:23.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:23.766443+0000) 2024-02-19T18:39:24.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:24.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:24.766637+0000) 2024-02-19T18:39:25.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:25.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:25.766926+0000) 2024-02-19T18:39:26.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:26.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:26.767164+0000) 2024-02-19T18:39:27.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:27.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:27.767440+0000) 2024-02-19T18:39:28.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:28.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:28.767675+0000) 2024-02-19T18:39:29.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:29.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:29.767961+0000) 2024-02-19T18:39:30.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:30.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:30.768186+0000) 2024-02-19T18:39:31.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:31.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:31.768455+0000) 2024-02-19T18:39:32.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:32.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:32.768644+0000) 2024-02-19T18:39:33.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:33.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:33.768919+0000) 2024-02-19T18:39:34.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:34.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:34.769234+0000) 2024-02-19T18:39:35.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:35.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:35.769510+0000) 2024-02-19T18:39:36.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:36.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:36.769830+0000) 2024-02-19T18:39:37.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:37.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:37.770024+0000) 2024-02-19T18:39:38.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:38.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:38.770285+0000) 2024-02-19T18:39:39.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:39.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:39.770572+0000) 2024-02-19T18:39:40.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:40.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:40.770860+0000) 2024-02-19T18:39:41.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:41.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:41.771089+0000) 2024-02-19T18:39:42.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:42.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:42.771340+0000) 2024-02-19T18:39:43.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:43.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:43.771548+0000) 2024-02-19T18:39:43.883 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:39:43.887 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:39:44.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:44.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:44.771818+0000) 2024-02-19T18:39:45.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:45.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:45.772115+0000) 2024-02-19T18:39:46.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:46.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:46.772435+0000) 2024-02-19T18:39:47.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:47.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:47.772651+0000) 2024-02-19T18:39:48.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:48.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:48.772944+0000) 2024-02-19T18:39:49.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:49.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:49.773222+0000) 2024-02-19T18:39:50.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:50.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:50.773479+0000) 2024-02-19T18:39:51.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:51.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:51.773707+0000) 2024-02-19T18:39:52.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:52.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:52.774019+0000) 2024-02-19T18:39:53.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:53.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:53.774209+0000) 2024-02-19T18:39:54.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:54.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:54.774468+0000) 2024-02-19T18:39:55.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:55.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:55.774798+0000) 2024-02-19T18:39:56.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:56.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:56.775040+0000) 2024-02-19T18:39:57.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:57.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:57.775239+0000) 2024-02-19T18:39:58.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:58.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:58.775575+0000) 2024-02-19T18:39:59.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:39:59.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:39:59.775804+0000) 2024-02-19T18:40:00.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:00.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:00.776030+0000) 2024-02-19T18:40:01.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:01.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:01.776249+0000) 2024-02-19T18:40:02.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:02.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:02.776529+0000) 2024-02-19T18:40:03.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:03.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:03.776723+0000) 2024-02-19T18:40:04.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:04.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:04.776991+0000) 2024-02-19T18:40:05.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:05.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:05.777189+0000) 2024-02-19T18:40:06.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:06.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:06.777427+0000) 2024-02-19T18:40:07.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:07.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:07.777716+0000) 2024-02-19T18:40:08.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:08.777+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:08.777998+0000) 2024-02-19T18:40:09.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:09.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:09.778226+0000) 2024-02-19T18:40:10.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:10.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:10.778363+0000) 2024-02-19T18:40:11.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:11.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:11.778625+0000) 2024-02-19T18:40:12.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:12.777+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:12.778849+0000) 2024-02-19T18:40:13.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:13.777+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:13.778996+0000) 2024-02-19T18:40:14.008 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:40:14.011 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:40:14.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:14.777+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:14.779137+0000) 2024-02-19T18:40:15.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:15.777+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:15.779376+0000) 2024-02-19T18:40:16.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:16.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:16.779622+0000) 2024-02-19T18:40:17.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:17.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:17.779903+0000) 2024-02-19T18:40:18.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:18.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:18.780179+0000) 2024-02-19T18:40:19.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:19.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:19.780501+0000) 2024-02-19T18:40:20.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:20.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:20.780732+0000) 2024-02-19T18:40:21.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:21.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:21.780918+0000) 2024-02-19T18:40:22.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:22.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:22.781104+0000) 2024-02-19T18:40:23.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:23.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:23.781306+0000) 2024-02-19T18:40:24.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:24.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:24.781566+0000) 2024-02-19T18:40:25.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:25.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:25.781720+0000) 2024-02-19T18:40:26.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:26.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:26.781868+0000) 2024-02-19T18:40:27.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:27.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:27.782067+0000) 2024-02-19T18:40:28.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:28.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:28.782329+0000) 2024-02-19T18:40:29.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:29.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:29.782557+0000) 2024-02-19T18:40:30.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:30.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:30.782875+0000) 2024-02-19T18:40:31.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:31.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:31.783114+0000) 2024-02-19T18:40:32.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:32.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:32.783278+0000) 2024-02-19T18:40:33.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:33.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:33.783446+0000) 2024-02-19T18:40:34.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:34.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:34.783698+0000) 2024-02-19T18:40:35.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:35.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:35.783976+0000) 2024-02-19T18:40:36.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:36.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:36.784184+0000) 2024-02-19T18:40:37.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:37.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:37.784421+0000) 2024-02-19T18:40:38.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:38.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:38.784675+0000) 2024-02-19T18:40:39.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:39.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:39.784926+0000) 2024-02-19T18:40:40.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:40.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:40.785120+0000) 2024-02-19T18:40:41.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:41.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:41.785331+0000) 2024-02-19T18:40:42.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:42.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:42.785524+0000) 2024-02-19T18:40:43.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:43.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:43.785818+0000) 2024-02-19T18:40:44.071 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:40:44.074 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:40:44.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:44.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:44.786010+0000) 2024-02-19T18:40:45.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:45.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:45.786314+0000) 2024-02-19T18:40:46.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:46.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:46.786619+0000) 2024-02-19T18:40:47.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:47.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:47.786900+0000) 2024-02-19T18:40:48.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:48.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:48.787160+0000) 2024-02-19T18:40:49.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:49.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:49.787441+0000) 2024-02-19T18:40:50.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:50.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:50.787731+0000) 2024-02-19T18:40:51.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:51.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:51.788043+0000) 2024-02-19T18:40:52.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:52.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:52.788428+0000) 2024-02-19T18:40:53.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:53.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:53.788708+0000) 2024-02-19T18:40:54.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:54.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:54.788925+0000) 2024-02-19T18:40:55.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:55.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:55.789132+0000) 2024-02-19T18:40:56.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:56.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:56.789432+0000) 2024-02-19T18:40:57.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:57.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:57.789755+0000) 2024-02-19T18:40:58.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:58.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:58.790042+0000) 2024-02-19T18:40:59.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:40:59.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:40:59.790242+0000) 2024-02-19T18:41:00.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:00.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:00.790525+0000) 2024-02-19T18:41:01.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:01.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:01.790839+0000) 2024-02-19T18:41:02.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:02.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:02.791108+0000) 2024-02-19T18:41:03.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:03.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:03.791334+0000) 2024-02-19T18:41:04.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:04.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:04.791559+0000) 2024-02-19T18:41:05.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:05.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:05.791733+0000) 2024-02-19T18:41:06.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:06.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:06.792040+0000) 2024-02-19T18:41:07.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:07.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:07.792303+0000) 2024-02-19T18:41:08.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:08.792+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:08.792592+0000) 2024-02-19T18:41:09.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:09.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:09.792862+0000) 2024-02-19T18:41:10.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:10.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:10.793036+0000) 2024-02-19T18:41:11.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:11.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:11.793280+0000) 2024-02-19T18:41:12.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:12.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:12.793509+0000) 2024-02-19T18:41:13.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:13.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:13.793752+0000) 2024-02-19T18:41:14.186 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:41:14.190 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:41:14.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:14.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:14.793989+0000) 2024-02-19T18:41:15.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:15.792+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:15.794206+0000) 2024-02-19T18:41:16.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:16.792+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:16.794418+0000) 2024-02-19T18:41:17.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:17.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:17.794585+0000) 2024-02-19T18:41:18.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:18.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:18.794854+0000) 2024-02-19T18:41:19.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:19.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:19.795086+0000) 2024-02-19T18:41:20.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:20.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:20.795285+0000) 2024-02-19T18:41:21.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:21.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:21.795559+0000) 2024-02-19T18:41:22.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:22.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:22.795798+0000) 2024-02-19T18:41:23.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:23.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:23.796037+0000) 2024-02-19T18:41:24.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:24.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:24.796297+0000) 2024-02-19T18:41:25.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:25.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:25.796557+0000) 2024-02-19T18:41:26.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:26.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:26.796794+0000) 2024-02-19T18:41:27.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:27.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:27.797024+0000) 2024-02-19T18:41:28.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:28.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:28.797229+0000) 2024-02-19T18:41:29.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:29.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:29.797445+0000) 2024-02-19T18:41:30.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:30.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:30.797684+0000) 2024-02-19T18:41:31.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:31.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:31.797934+0000) 2024-02-19T18:41:32.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:32.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:32.798205+0000) 2024-02-19T18:41:33.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:33.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:33.798418+0000) 2024-02-19T18:41:34.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:34.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:34.798668+0000) 2024-02-19T18:41:35.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:35.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:35.798842+0000) 2024-02-19T18:41:36.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:36.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:36.799064+0000) 2024-02-19T18:41:37.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:37.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:37.799263+0000) 2024-02-19T18:41:38.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:38.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:38.799466+0000) 2024-02-19T18:41:39.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:39.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:39.799648+0000) 2024-02-19T18:41:40.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:40.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:40.799898+0000) 2024-02-19T18:41:41.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:41.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:41.800144+0000) 2024-02-19T18:41:42.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:42.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:42.800384+0000) 2024-02-19T18:41:43.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:43.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:43.800585+0000) 2024-02-19T18:41:44.312 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:41:44.316 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:41:44.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:44.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:44.800841+0000) 2024-02-19T18:41:45.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:45.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:45.801028+0000) 2024-02-19T18:41:46.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:46.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:46.801241+0000) 2024-02-19T18:41:47.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:47.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:47.801415+0000) 2024-02-19T18:41:48.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:48.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:48.801662+0000) 2024-02-19T18:41:49.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:49.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:49.801900+0000) 2024-02-19T18:41:50.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:50.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:50.802172+0000) 2024-02-19T18:41:51.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:51.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:51.802427+0000) 2024-02-19T18:41:52.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:52.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:52.802662+0000) 2024-02-19T18:41:53.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:53.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:53.802908+0000) 2024-02-19T18:41:54.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:54.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:54.803163+0000) 2024-02-19T18:41:55.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:55.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:55.803423+0000) 2024-02-19T18:41:56.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:56.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:56.803694+0000) 2024-02-19T18:41:57.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:57.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:57.803884+0000) 2024-02-19T18:41:58.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:58.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:58.804168+0000) 2024-02-19T18:41:59.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:41:59.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:41:59.804375+0000) 2024-02-19T18:42:00.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:00.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:00.804578+0000) 2024-02-19T18:42:01.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:01.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:01.804823+0000) 2024-02-19T18:42:02.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:02.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:02.805036+0000) 2024-02-19T18:42:03.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:03.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:03.805217+0000) 2024-02-19T18:42:04.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:04.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:04.805466+0000) 2024-02-19T18:42:05.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:05.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:05.805705+0000) 2024-02-19T18:42:06.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:06.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:06.805894+0000) 2024-02-19T18:42:07.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:07.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:07.806076+0000) 2024-02-19T18:42:08.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:08.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:08.806260+0000) 2024-02-19T18:42:09.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:09.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:09.806484+0000) 2024-02-19T18:42:10.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:10.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:10.806685+0000) 2024-02-19T18:42:11.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:11.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:11.806910+0000) 2024-02-19T18:42:12.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:12.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:12.807124+0000) 2024-02-19T18:42:13.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:13.807+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:13.807386+0000) 2024-02-19T18:42:14.430 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:42:14.435 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:42:14.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:14.807+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:14.807636+0000) 2024-02-19T18:42:15.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:15.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:15.807895+0000) 2024-02-19T18:42:16.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:16.807+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:16.808092+0000) 2024-02-19T18:42:17.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:17.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:17.808378+0000) 2024-02-19T18:42:18.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:18.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:18.808597+0000) 2024-02-19T18:42:19.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:19.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:19.808797+0000) 2024-02-19T18:42:20.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:20.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:20.809010+0000) 2024-02-19T18:42:21.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:21.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:21.809255+0000) 2024-02-19T18:42:22.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:22.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:22.809512+0000) 2024-02-19T18:42:23.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:23.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:23.809825+0000) 2024-02-19T18:42:24.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:24.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:24.810090+0000) 2024-02-19T18:42:25.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:25.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:25.810377+0000) 2024-02-19T18:42:26.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:26.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:26.810670+0000) 2024-02-19T18:42:27.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:27.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:27.810966+0000) 2024-02-19T18:42:28.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:28.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:28.811140+0000) 2024-02-19T18:42:29.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:29.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:29.811349+0000) 2024-02-19T18:42:30.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:30.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:30.811593+0000) 2024-02-19T18:42:31.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:31.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:31.811778+0000) 2024-02-19T18:42:32.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:32.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:32.812072+0000) 2024-02-19T18:42:33.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:33.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:33.812347+0000) 2024-02-19T18:42:34.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:34.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:34.812548+0000) 2024-02-19T18:42:35.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:35.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:35.812827+0000) 2024-02-19T18:42:36.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:36.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:36.813053+0000) 2024-02-19T18:42:37.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:37.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:37.813319+0000) 2024-02-19T18:42:38.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:38.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:38.813577+0000) 2024-02-19T18:42:39.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:39.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:39.813734+0000) 2024-02-19T18:42:40.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:40.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:40.813908+0000) 2024-02-19T18:42:41.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:41.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:41.814195+0000) 2024-02-19T18:42:42.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:42.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:42.814376+0000) 2024-02-19T18:42:43.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:43.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:43.814548+0000) 2024-02-19T18:42:44.570 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:42:44.574 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:42:44.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:44.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:44.814855+0000) 2024-02-19T18:42:45.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:45.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:45.815018+0000) 2024-02-19T18:42:46.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:46.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:46.815261+0000) 2024-02-19T18:42:47.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:47.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:47.815486+0000) 2024-02-19T18:42:48.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:48.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:48.815666+0000) 2024-02-19T18:42:49.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:49.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:49.815916+0000) 2024-02-19T18:42:50.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:50.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:50.816158+0000) 2024-02-19T18:42:51.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:51.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:51.816395+0000) 2024-02-19T18:42:52.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:52.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:52.816630+0000) 2024-02-19T18:42:53.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:53.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:53.816868+0000) 2024-02-19T18:42:54.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:54.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:54.817115+0000) 2024-02-19T18:42:55.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:55.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:55.817338+0000) 2024-02-19T18:42:56.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:56.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:56.817604+0000) 2024-02-19T18:42:57.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:57.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:57.817833+0000) 2024-02-19T18:42:58.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:58.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:58.818109+0000) 2024-02-19T18:42:59.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:42:59.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:42:59.818347+0000) 2024-02-19T18:43:00.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:00.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:00.818587+0000) 2024-02-19T18:43:01.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:01.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:01.818773+0000) 2024-02-19T18:43:02.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:02.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:02.819023+0000) 2024-02-19T18:43:03.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:03.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:03.819261+0000) 2024-02-19T18:43:04.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:04.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:04.819503+0000) 2024-02-19T18:43:05.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:05.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:05.819767+0000) 2024-02-19T18:43:06.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:06.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:06.820013+0000) 2024-02-19T18:43:07.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:07.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:07.820169+0000) 2024-02-19T18:43:08.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:08.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:08.820356+0000) 2024-02-19T18:43:09.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:09.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:09.820630+0000) 2024-02-19T18:43:10.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:10.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:10.820894+0000) 2024-02-19T18:43:11.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:11.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:11.821146+0000) 2024-02-19T18:43:12.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:12.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:12.821313+0000) 2024-02-19T18:43:13.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:13.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:13.821518+0000) 2024-02-19T18:43:14.642 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:43:14.645 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:43:14.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:14.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:14.821772+0000) 2024-02-19T18:43:15.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:15.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:15.822004+0000) 2024-02-19T18:43:16.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:16.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:16.822245+0000) 2024-02-19T18:43:17.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:17.822+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:17.822496+0000) 2024-02-19T18:43:18.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:18.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:18.822783+0000) 2024-02-19T18:43:19.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:19.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:19.823029+0000) 2024-02-19T18:43:20.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:20.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:20.823202+0000) 2024-02-19T18:43:21.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:21.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:21.823459+0000) 2024-02-19T18:43:22.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:22.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:22.823715+0000) 2024-02-19T18:43:23.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:23.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:23.823950+0000) 2024-02-19T18:43:24.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:24.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:24.824196+0000) 2024-02-19T18:43:25.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:25.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:25.824437+0000) 2024-02-19T18:43:26.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:26.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:26.824639+0000) 2024-02-19T18:43:27.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:27.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:27.824875+0000) 2024-02-19T18:43:28.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:28.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:28.825111+0000) 2024-02-19T18:43:29.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:29.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:29.825286+0000) 2024-02-19T18:43:30.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:30.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:30.825531+0000) 2024-02-19T18:43:31.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:31.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:31.825710+0000) 2024-02-19T18:43:32.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:32.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:32.825911+0000) 2024-02-19T18:43:33.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:33.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:33.826162+0000) 2024-02-19T18:43:34.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:34.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:34.826361+0000) 2024-02-19T18:43:35.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:35.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:35.826555+0000) 2024-02-19T18:43:36.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:36.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:36.826808+0000) 2024-02-19T18:43:37.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:37.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:37.827062+0000) 2024-02-19T18:43:38.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:38.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:38.827294+0000) 2024-02-19T18:43:39.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:39.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:39.827532+0000) 2024-02-19T18:43:40.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:40.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:40.827775+0000) 2024-02-19T18:43:41.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:41.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:41.827985+0000) 2024-02-19T18:43:42.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:42.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:42.828214+0000) 2024-02-19T18:43:43.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:43.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:43.828489+0000) 2024-02-19T18:43:44.721 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:43:44.725 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:43:44.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:44.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:44.828729+0000) 2024-02-19T18:43:45.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:45.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:45.829001+0000) 2024-02-19T18:43:46.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:46.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:46.829208+0000) 2024-02-19T18:43:47.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:47.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:47.829386+0000) 2024-02-19T18:43:48.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:48.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:48.829589+0000) 2024-02-19T18:43:49.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:49.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:49.829768+0000) 2024-02-19T18:43:50.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:50.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:50.829968+0000) 2024-02-19T18:43:51.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:51.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:51.830247+0000) 2024-02-19T18:43:52.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:52.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:52.830482+0000) 2024-02-19T18:43:53.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:53.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:53.830697+0000) 2024-02-19T18:43:54.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:54.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:54.830895+0000) 2024-02-19T18:43:55.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:55.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:55.831075+0000) 2024-02-19T18:43:56.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:56.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:56.831283+0000) 2024-02-19T18:43:57.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:57.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:57.831527+0000) 2024-02-19T18:43:58.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:58.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:58.831778+0000) 2024-02-19T18:43:59.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:43:59.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:43:59.832039+0000) 2024-02-19T18:44:00.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:00.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:00.832299+0000) 2024-02-19T18:44:01.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:01.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:01.832533+0000) 2024-02-19T18:44:02.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:02.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:02.832757+0000) 2024-02-19T18:44:03.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:03.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:03.832973+0000) 2024-02-19T18:44:04.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:04.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:04.833187+0000) 2024-02-19T18:44:05.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:05.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:05.833428+0000) 2024-02-19T18:44:06.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:06.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:06.833678+0000) 2024-02-19T18:44:07.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:07.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:07.833907+0000) 2024-02-19T18:44:08.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:08.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:08.834173+0000) 2024-02-19T18:44:09.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:09.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:09.834369+0000) 2024-02-19T18:44:10.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:10.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:10.834564+0000) 2024-02-19T18:44:11.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:11.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:11.834814+0000) 2024-02-19T18:44:12.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:12.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:12.835036+0000) 2024-02-19T18:44:13.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:13.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:13.835288+0000) 2024-02-19T18:44:14.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:14.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:14.835574+0000) 2024-02-19T18:44:14.858 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:44:14.862 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:44:15.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:15.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:15.835761+0000) 2024-02-19T18:44:16.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:16.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:16.835975+0000) 2024-02-19T18:44:17.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:17.833+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:17.836189+0000) 2024-02-19T18:44:18.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:18.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:18.836436+0000) 2024-02-19T18:44:19.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:19.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:19.836606+0000) 2024-02-19T18:44:20.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:20.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:20.836850+0000) 2024-02-19T18:44:21.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:21.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:21.837048+0000) 2024-02-19T18:44:22.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:22.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:22.837291+0000) 2024-02-19T18:44:23.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:23.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:23.837531+0000) 2024-02-19T18:44:24.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:24.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:24.837737+0000) 2024-02-19T18:44:25.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:25.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:25.837996+0000) 2024-02-19T18:44:26.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:26.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:26.838278+0000) 2024-02-19T18:44:27.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:27.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:27.838449+0000) 2024-02-19T18:44:28.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:28.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:28.838753+0000) 2024-02-19T18:44:29.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:29.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:29.838944+0000) 2024-02-19T18:44:30.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:30.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:30.839190+0000) 2024-02-19T18:44:31.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:31.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:31.839379+0000) 2024-02-19T18:44:32.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:32.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:32.839558+0000) 2024-02-19T18:44:33.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:33.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:33.839796+0000) 2024-02-19T18:44:34.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:34.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:34.840047+0000) 2024-02-19T18:44:35.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:35.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:35.840178+0000) 2024-02-19T18:44:36.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:36.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:36.840430+0000) 2024-02-19T18:44:37.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:37.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:37.840664+0000) 2024-02-19T18:44:38.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:38.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:38.840895+0000) 2024-02-19T18:44:39.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:39.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:39.841148+0000) 2024-02-19T18:44:40.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:40.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:40.841344+0000) 2024-02-19T18:44:41.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:41.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:41.841483+0000) 2024-02-19T18:44:42.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:42.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:42.841746+0000) 2024-02-19T18:44:43.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:43.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:43.841980+0000) 2024-02-19T18:44:44.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:44.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:44.842216+0000) 2024-02-19T18:44:44.983 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:44:44.987 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:44:45.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:45.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:45.842483+0000) 2024-02-19T18:44:46.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:46.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:46.842765+0000) 2024-02-19T18:44:47.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:47.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:47.842952+0000) 2024-02-19T18:44:48.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:48.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:48.843179+0000) 2024-02-19T18:44:49.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:49.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:49.843422+0000) 2024-02-19T18:44:50.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:50.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:50.843673+0000) 2024-02-19T18:44:51.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:51.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:51.843930+0000) 2024-02-19T18:44:52.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:52.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:52.844159+0000) 2024-02-19T18:44:53.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:53.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:53.844414+0000) 2024-02-19T18:44:54.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:54.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:54.844670+0000) 2024-02-19T18:44:55.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:55.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:55.844911+0000) 2024-02-19T18:44:56.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:56.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:56.845150+0000) 2024-02-19T18:44:57.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:57.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:57.845347+0000) 2024-02-19T18:44:58.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:58.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:58.845592+0000) 2024-02-19T18:44:59.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:44:59.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:44:59.845838+0000) 2024-02-19T18:45:00.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:00.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:00.846035+0000) 2024-02-19T18:45:01.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:01.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:01.846268+0000) 2024-02-19T18:45:02.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:02.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:02.846505+0000) 2024-02-19T18:45:03.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:03.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:03.846686+0000) 2024-02-19T18:45:04.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:04.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:04.846958+0000) 2024-02-19T18:45:05.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:05.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:05.847212+0000) 2024-02-19T18:45:06.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:06.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:06.847469+0000) 2024-02-19T18:45:07.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:07.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:07.847710+0000) 2024-02-19T18:45:08.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:08.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:08.847995+0000) 2024-02-19T18:45:09.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:09.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:09.848242+0000) 2024-02-19T18:45:10.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:10.848+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:10.848517+0000) 2024-02-19T18:45:11.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:11.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:11.848760+0000) 2024-02-19T18:45:12.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:12.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:12.848944+0000) 2024-02-19T18:45:13.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:13.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:13.849177+0000) 2024-02-19T18:45:14.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:14.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:14.849429+0000) 2024-02-19T18:45:15.108 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:45:15.113 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:45:15.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:15.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:15.849664+0000) 2024-02-19T18:45:16.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:16.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:16.849912+0000) 2024-02-19T18:45:17.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:17.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:17.850172+0000) 2024-02-19T18:45:18.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:18.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:18.850449+0000) 2024-02-19T18:45:19.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:19.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:19.850671+0000) 2024-02-19T18:45:20.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:20.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:20.850881+0000) 2024-02-19T18:45:21.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:21.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:21.851072+0000) 2024-02-19T18:45:22.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:22.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:22.851218+0000) 2024-02-19T18:45:23.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:23.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:23.851364+0000) 2024-02-19T18:45:24.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:24.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:24.851609+0000) 2024-02-19T18:45:25.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:25.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:25.851864+0000) 2024-02-19T18:45:26.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:26.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:26.852121+0000) 2024-02-19T18:45:27.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:27.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:27.852385+0000) 2024-02-19T18:45:28.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:28.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:28.852575+0000) 2024-02-19T18:45:29.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:29.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:29.852838+0000) 2024-02-19T18:45:30.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:30.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:30.853073+0000) 2024-02-19T18:45:31.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:31.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:31.853241+0000) 2024-02-19T18:45:32.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:32.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:32.853506+0000) 2024-02-19T18:45:33.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:33.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:33.853777+0000) 2024-02-19T18:45:34.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:34.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:34.853959+0000) 2024-02-19T18:45:35.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:35.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:35.854200+0000) 2024-02-19T18:45:36.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:36.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:36.854467+0000) 2024-02-19T18:45:37.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:37.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:37.854774+0000) 2024-02-19T18:45:38.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:38.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:38.854942+0000) 2024-02-19T18:45:39.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:39.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:39.855107+0000) 2024-02-19T18:45:40.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:40.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:40.855390+0000) 2024-02-19T18:45:41.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:41.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:41.855639+0000) 2024-02-19T18:45:42.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:42.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:42.855820+0000) 2024-02-19T18:45:43.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:43.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:43.856075+0000) 2024-02-19T18:45:44.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:44.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:44.856273+0000) 2024-02-19T18:45:45.240 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:45:45.244 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:45:45.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:45.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:45.856504+0000) 2024-02-19T18:45:46.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:46.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:46.856761+0000) 2024-02-19T18:45:47.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:47.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:47.856979+0000) 2024-02-19T18:45:48.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:48.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:48.857222+0000) 2024-02-19T18:45:49.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:49.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:49.857468+0000) 2024-02-19T18:45:50.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:50.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:50.857719+0000) 2024-02-19T18:45:51.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:51.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:51.857965+0000) 2024-02-19T18:45:52.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:52.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:52.858204+0000) 2024-02-19T18:45:53.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:53.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:53.858368+0000) 2024-02-19T18:45:54.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:54.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:54.858608+0000) 2024-02-19T18:45:55.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:55.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:55.858884+0000) 2024-02-19T18:45:56.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:56.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:56.859137+0000) 2024-02-19T18:45:57.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:57.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:57.859335+0000) 2024-02-19T18:45:58.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:58.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:58.859579+0000) 2024-02-19T18:45:59.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:45:59.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:45:59.859833+0000) 2024-02-19T18:46:00.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:00.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:00.860001+0000) 2024-02-19T18:46:01.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:01.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:01.860194+0000) 2024-02-19T18:46:02.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:02.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:02.860395+0000) 2024-02-19T18:46:03.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:03.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:03.860596+0000) 2024-02-19T18:46:04.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:04.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:04.860846+0000) 2024-02-19T18:46:05.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:05.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:05.861063+0000) 2024-02-19T18:46:06.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:06.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:06.861327+0000) 2024-02-19T18:46:07.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:07.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:07.861568+0000) 2024-02-19T18:46:08.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:08.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:08.861813+0000) 2024-02-19T18:46:09.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:09.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:09.862074+0000) 2024-02-19T18:46:10.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:10.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:10.862329+0000) 2024-02-19T18:46:11.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:11.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:11.862566+0000) 2024-02-19T18:46:12.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:12.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:12.862818+0000) 2024-02-19T18:46:13.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:13.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:13.863058+0000) 2024-02-19T18:46:14.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:14.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:14.863240+0000) 2024-02-19T18:46:15.385 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:46:15.396 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:46:15.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:15.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:15.863515+0000) 2024-02-19T18:46:16.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:16.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:16.863765+0000) 2024-02-19T18:46:17.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:17.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:17.863948+0000) 2024-02-19T18:46:18.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:18.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:18.864169+0000) 2024-02-19T18:46:19.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:19.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:19.864413+0000) 2024-02-19T18:46:20.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:20.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:20.864651+0000) 2024-02-19T18:46:21.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:21.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:21.864852+0000) 2024-02-19T18:46:22.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:22.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:22.865105+0000) 2024-02-19T18:46:23.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:23.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:23.865305+0000) 2024-02-19T18:46:24.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:24.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:24.865525+0000) 2024-02-19T18:46:25.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:25.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:25.865755+0000) 2024-02-19T18:46:26.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:26.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:26.866018+0000) 2024-02-19T18:46:27.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:27.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:27.866281+0000) 2024-02-19T18:46:28.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:28.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:28.866561+0000) 2024-02-19T18:46:29.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:29.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:29.866808+0000) 2024-02-19T18:46:30.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:30.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:30.866965+0000) 2024-02-19T18:46:31.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:31.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:31.867194+0000) 2024-02-19T18:46:32.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:32.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:32.867428+0000) 2024-02-19T18:46:33.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:33.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:33.867663+0000) 2024-02-19T18:46:34.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:34.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:34.867916+0000) 2024-02-19T18:46:35.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:35.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:35.868174+0000) 2024-02-19T18:46:36.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:36.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:36.868423+0000) 2024-02-19T18:46:37.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:37.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:37.868597+0000) 2024-02-19T18:46:38.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:38.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:38.868781+0000) 2024-02-19T18:46:39.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:39.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:39.869038+0000) 2024-02-19T18:46:40.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:40.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:40.869284+0000) 2024-02-19T18:46:41.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:41.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:41.869537+0000) 2024-02-19T18:46:42.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:42.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:42.869721+0000) 2024-02-19T18:46:43.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:43.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:43.869913+0000) 2024-02-19T18:46:44.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:44.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:44.870175+0000) 2024-02-19T18:46:45.507 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:46:45.512 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:46:45.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:45.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:45.870422+0000) 2024-02-19T18:46:46.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:46.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:46.870692+0000) 2024-02-19T18:46:47.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:47.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:47.870914+0000) 2024-02-19T18:46:48.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:48.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:48.871137+0000) 2024-02-19T18:46:49.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:49.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:49.871325+0000) 2024-02-19T18:46:50.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:50.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:50.871585+0000) 2024-02-19T18:46:51.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:51.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:51.871838+0000) 2024-02-19T18:46:52.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:52.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:52.872103+0000) 2024-02-19T18:46:53.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:53.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:53.872335+0000) 2024-02-19T18:46:54.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:54.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:54.872666+0000) 2024-02-19T18:46:55.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:55.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:55.872909+0000) 2024-02-19T18:46:56.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:56.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:56.873158+0000) 2024-02-19T18:46:57.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:57.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:57.873348+0000) 2024-02-19T18:46:58.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:58.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:58.873602+0000) 2024-02-19T18:46:59.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:46:59.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:46:59.873849+0000) 2024-02-19T18:47:00.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:00.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:00.874096+0000) 2024-02-19T18:47:01.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:01.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:01.874341+0000) 2024-02-19T18:47:02.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:02.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:02.874578+0000) 2024-02-19T18:47:03.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:03.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:03.874778+0000) 2024-02-19T18:47:04.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:04.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:04.874978+0000) 2024-02-19T18:47:05.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:05.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:05.875176+0000) 2024-02-19T18:47:06.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:06.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:06.875396+0000) 2024-02-19T18:47:07.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:07.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:07.875589+0000) 2024-02-19T18:47:08.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:08.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:08.875870+0000) 2024-02-19T18:47:09.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:09.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:09.876122+0000) 2024-02-19T18:47:10.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:10.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:10.876321+0000) 2024-02-19T18:47:11.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:11.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:11.876563+0000) 2024-02-19T18:47:12.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:12.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:12.876791+0000) 2024-02-19T18:47:13.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:13.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:13.877037+0000) 2024-02-19T18:47:14.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:14.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:14.877232+0000) 2024-02-19T18:47:15.647 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:47:15.650 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:47:15.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:15.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:15.877450+0000) 2024-02-19T18:47:16.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:16.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:16.877673+0000) 2024-02-19T18:47:17.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:17.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:17.877887+0000) 2024-02-19T18:47:18.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:18.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:18.878135+0000) 2024-02-19T18:47:19.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:19.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:19.878412+0000) 2024-02-19T18:47:20.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:20.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:20.878652+0000) 2024-02-19T18:47:21.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:21.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:21.878858+0000) 2024-02-19T18:47:22.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:22.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:22.879105+0000) 2024-02-19T18:47:23.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:23.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:23.879298+0000) 2024-02-19T18:47:24.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:24.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:24.879501+0000) 2024-02-19T18:47:25.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:25.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:25.879660+0000) 2024-02-19T18:47:26.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:26.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:26.879851+0000) 2024-02-19T18:47:27.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:27.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:27.880094+0000) 2024-02-19T18:47:28.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:28.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:28.880333+0000) 2024-02-19T18:47:29.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:29.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:29.880570+0000) 2024-02-19T18:47:30.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:30.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:30.880835+0000) 2024-02-19T18:47:31.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:31.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:31.881077+0000) 2024-02-19T18:47:32.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:32.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:32.881294+0000) 2024-02-19T18:47:33.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:33.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:33.881501+0000) 2024-02-19T18:47:34.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:34.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:34.881791+0000) 2024-02-19T18:47:35.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:35.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:35.882034+0000) 2024-02-19T18:47:36.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:36.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:36.882286+0000) 2024-02-19T18:47:37.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:37.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:37.882457+0000) 2024-02-19T18:47:38.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:38.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:38.882735+0000) 2024-02-19T18:47:39.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:39.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:39.882943+0000) 2024-02-19T18:47:40.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:40.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:40.883172+0000) 2024-02-19T18:47:41.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:41.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:41.883372+0000) 2024-02-19T18:47:42.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:42.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:42.883650+0000) 2024-02-19T18:47:43.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:43.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:43.883896+0000) 2024-02-19T18:47:44.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:44.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:44.884152+0000) 2024-02-19T18:47:45.768 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:47:45.771 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:47:45.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:45.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:45.884414+0000) 2024-02-19T18:47:46.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:46.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:46.884687+0000) 2024-02-19T18:47:47.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:47.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:47.884897+0000) 2024-02-19T18:47:48.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:48.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:48.885199+0000) 2024-02-19T18:47:49.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:49.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:49.885419+0000) 2024-02-19T18:47:50.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:50.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:50.885675+0000) 2024-02-19T18:47:51.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:51.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:51.885931+0000) 2024-02-19T18:47:52.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:52.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:52.886143+0000) 2024-02-19T18:47:53.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:53.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:53.886362+0000) 2024-02-19T18:47:54.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:54.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:54.886610+0000) 2024-02-19T18:47:55.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:55.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:55.886860+0000) 2024-02-19T18:47:56.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:56.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:56.887042+0000) 2024-02-19T18:47:57.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:57.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:57.887241+0000) 2024-02-19T18:47:58.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:58.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:58.887489+0000) 2024-02-19T18:47:59.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:47:59.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:47:59.887711+0000) 2024-02-19T18:48:00.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:00.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:00.887869+0000) 2024-02-19T18:48:01.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:01.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:01.888096+0000) 2024-02-19T18:48:02.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:02.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:02.888313+0000) 2024-02-19T18:48:03.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:03.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:03.888544+0000) 2024-02-19T18:48:04.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:04.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:04.888802+0000) 2024-02-19T18:48:05.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:05.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:05.889040+0000) 2024-02-19T18:48:06.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:06.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:06.889290+0000) 2024-02-19T18:48:07.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:07.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:07.889491+0000) 2024-02-19T18:48:08.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:08.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:08.889734+0000) 2024-02-19T18:48:09.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:09.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:09.889949+0000) 2024-02-19T18:48:10.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:10.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:10.890211+0000) 2024-02-19T18:48:11.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:11.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:11.890409+0000) 2024-02-19T18:48:12.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:12.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:12.890607+0000) 2024-02-19T18:48:13.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:13.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:13.890880+0000) 2024-02-19T18:48:14.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:14.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:14.891076+0000) 2024-02-19T18:48:15.849 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:48:15.853 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:48:15.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:15.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:15.891278+0000) 2024-02-19T18:48:16.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:16.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:16.891528+0000) 2024-02-19T18:48:17.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:17.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:17.891760+0000) 2024-02-19T18:48:18.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:18.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:18.892030+0000) 2024-02-19T18:48:19.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:19.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:19.892286+0000) 2024-02-19T18:48:20.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:20.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:20.892529+0000) 2024-02-19T18:48:21.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:21.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:21.892799+0000) 2024-02-19T18:48:22.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:22.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:22.893006+0000) 2024-02-19T18:48:23.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:23.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:23.893259+0000) 2024-02-19T18:48:24.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:24.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:24.893458+0000) 2024-02-19T18:48:25.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:25.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:25.893708+0000) 2024-02-19T18:48:26.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:26.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:26.893971+0000) 2024-02-19T18:48:27.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:27.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:27.894201+0000) 2024-02-19T18:48:28.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:28.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:28.894384+0000) 2024-02-19T18:48:29.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:29.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:29.894586+0000) 2024-02-19T18:48:30.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:30.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:30.894831+0000) 2024-02-19T18:48:31.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:31.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:31.895126+0000) 2024-02-19T18:48:32.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:32.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:32.895343+0000) 2024-02-19T18:48:33.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:33.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:33.895580+0000) 2024-02-19T18:48:34.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:34.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:34.895764+0000) 2024-02-19T18:48:35.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:35.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:35.896010+0000) 2024-02-19T18:48:36.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:36.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:36.896238+0000) 2024-02-19T18:48:37.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:37.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:37.896461+0000) 2024-02-19T18:48:38.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:38.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:38.896714+0000) 2024-02-19T18:48:39.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:39.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:39.896897+0000) 2024-02-19T18:48:40.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:40.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:40.897099+0000) 2024-02-19T18:48:41.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:41.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:41.897349+0000) 2024-02-19T18:48:42.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:42.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:42.897597+0000) 2024-02-19T18:48:43.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:43.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:43.897865+0000) 2024-02-19T18:48:44.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:44.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:44.898092+0000) 2024-02-19T18:48:45.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:45.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:45.898348+0000) 2024-02-19T18:48:45.989 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:48:45.994 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:48:46.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:46.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:46.898607+0000) 2024-02-19T18:48:47.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:47.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:47.898845+0000) 2024-02-19T18:48:48.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:48.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:48.899036+0000) 2024-02-19T18:48:49.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:49.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:49.899239+0000) 2024-02-19T18:48:50.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:50.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:50.899460+0000) 2024-02-19T18:48:51.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:51.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:51.899728+0000) 2024-02-19T18:48:52.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:52.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:52.899909+0000) 2024-02-19T18:48:53.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:53.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:53.900165+0000) 2024-02-19T18:48:54.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:54.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:54.900350+0000) 2024-02-19T18:48:55.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:55.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:55.900602+0000) 2024-02-19T18:48:56.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:56.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:56.900874+0000) 2024-02-19T18:48:57.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:57.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:57.901057+0000) 2024-02-19T18:48:58.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:58.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:58.901259+0000) 2024-02-19T18:48:59.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:48:59.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:48:59.901455+0000) 2024-02-19T18:49:00.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:00.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:00.901739+0000) 2024-02-19T18:49:01.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:01.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:01.901998+0000) 2024-02-19T18:49:02.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:02.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:02.902237+0000) 2024-02-19T18:49:03.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:03.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:03.902460+0000) 2024-02-19T18:49:04.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:04.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:04.902631+0000) 2024-02-19T18:49:05.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:05.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:05.902928+0000) 2024-02-19T18:49:06.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:06.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:06.903172+0000) 2024-02-19T18:49:07.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:07.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:07.903376+0000) 2024-02-19T18:49:08.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:08.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:08.903647+0000) 2024-02-19T18:49:09.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:09.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:09.903896+0000) 2024-02-19T18:49:10.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:10.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:10.904113+0000) 2024-02-19T18:49:11.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:11.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:11.904376+0000) 2024-02-19T18:49:12.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:12.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:12.904628+0000) 2024-02-19T18:49:13.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:13.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:13.904910+0000) 2024-02-19T18:49:14.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:14.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:14.905097+0000) 2024-02-19T18:49:15.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:15.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:15.905305+0000) 2024-02-19T18:49:16.049 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:49:16.052 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:49:16.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:16.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:16.905570+0000) 2024-02-19T18:49:17.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:17.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:17.905816+0000) 2024-02-19T18:49:18.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:18.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:18.906013+0000) 2024-02-19T18:49:19.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:19.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:19.906259+0000) 2024-02-19T18:49:20.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:20.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:20.906489+0000) 2024-02-19T18:49:21.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:21.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:21.906680+0000) 2024-02-19T18:49:22.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:22.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:22.906883+0000) 2024-02-19T18:49:23.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:23.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:23.907143+0000) 2024-02-19T18:49:24.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:24.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:24.907340+0000) 2024-02-19T18:49:25.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:25.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:25.907591+0000) 2024-02-19T18:49:26.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:26.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:26.907861+0000) 2024-02-19T18:49:27.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:27.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:27.908112+0000) 2024-02-19T18:49:28.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:28.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:28.908372+0000) 2024-02-19T18:49:29.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:29.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:29.908565+0000) 2024-02-19T18:49:30.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:30.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:30.908840+0000) 2024-02-19T18:49:31.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:31.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:31.909085+0000) 2024-02-19T18:49:32.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:32.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:32.909274+0000) 2024-02-19T18:49:33.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:33.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:33.909527+0000) 2024-02-19T18:49:34.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:34.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:34.909683+0000) 2024-02-19T18:49:35.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:35.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:35.909847+0000) 2024-02-19T18:49:36.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:36.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:36.910120+0000) 2024-02-19T18:49:37.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:37.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:37.910343+0000) 2024-02-19T18:49:38.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:38.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:38.910585+0000) 2024-02-19T18:49:39.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:39.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:39.910878+0000) 2024-02-19T18:49:40.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:40.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:40.911047+0000) 2024-02-19T18:49:41.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:41.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:41.911195+0000) 2024-02-19T18:49:42.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:42.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:42.911424+0000) 2024-02-19T18:49:43.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:43.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:43.911695+0000) 2024-02-19T18:49:44.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:44.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:44.911918+0000) 2024-02-19T18:49:45.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:45.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:45.912181+0000) 2024-02-19T18:49:46.183 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:49:46.186 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:49:46.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:46.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:46.912429+0000) 2024-02-19T18:49:47.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:47.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:47.912667+0000) 2024-02-19T18:49:48.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:48.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:48.912884+0000) 2024-02-19T18:49:49.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:49.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:49.913118+0000) 2024-02-19T18:49:50.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:50.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:50.913357+0000) 2024-02-19T18:49:51.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:51.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:51.913599+0000) 2024-02-19T18:49:52.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:52.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:52.913867+0000) 2024-02-19T18:49:53.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:53.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:53.914118+0000) 2024-02-19T18:49:54.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:54.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:54.914370+0000) 2024-02-19T18:49:55.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:55.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:55.914616+0000) 2024-02-19T18:49:56.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:56.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:56.914903+0000) 2024-02-19T18:49:57.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:57.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:57.915157+0000) 2024-02-19T18:49:58.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:58.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:58.915414+0000) 2024-02-19T18:49:59.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:49:59.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:49:59.915668+0000) 2024-02-19T18:50:00.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:00.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:00.915955+0000) 2024-02-19T18:50:01.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:01.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:01.916194+0000) 2024-02-19T18:50:02.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:02.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:02.916439+0000) 2024-02-19T18:50:03.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:03.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:03.916667+0000) 2024-02-19T18:50:04.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:04.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:04.916833+0000) 2024-02-19T18:50:05.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:05.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:05.917007+0000) 2024-02-19T18:50:06.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:06.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:06.917273+0000) 2024-02-19T18:50:07.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:07.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:07.917561+0000) 2024-02-19T18:50:08.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:08.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:08.917794+0000) 2024-02-19T18:50:09.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:09.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:09.917979+0000) 2024-02-19T18:50:10.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:10.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:10.918215+0000) 2024-02-19T18:50:11.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:11.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:11.918507+0000) 2024-02-19T18:50:12.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:12.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:12.918755+0000) 2024-02-19T18:50:13.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:13.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:13.918971+0000) 2024-02-19T18:50:14.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:14.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:14.919203+0000) 2024-02-19T18:50:15.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:15.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:15.919406+0000) 2024-02-19T18:50:16.238 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:50:16.241 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:50:16.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:16.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:16.919654+0000) 2024-02-19T18:50:17.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:17.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:17.919899+0000) 2024-02-19T18:50:18.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:18.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:18.920183+0000) 2024-02-19T18:50:19.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:19.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:19.920381+0000) 2024-02-19T18:50:20.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:20.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:20.920563+0000) 2024-02-19T18:50:21.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:21.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:21.920766+0000) 2024-02-19T18:50:22.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:22.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:22.921005+0000) 2024-02-19T18:50:23.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:23.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:23.921202+0000) 2024-02-19T18:50:24.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:24.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:24.921435+0000) 2024-02-19T18:50:25.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:25.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:25.921692+0000) 2024-02-19T18:50:26.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:26.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:26.921952+0000) 2024-02-19T18:50:27.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:27.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:27.922210+0000) 2024-02-19T18:50:28.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:28.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:28.922440+0000) 2024-02-19T18:50:29.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:29.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:29.922701+0000) 2024-02-19T18:50:30.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:30.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:30.922938+0000) 2024-02-19T18:50:31.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:31.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:31.923188+0000) 2024-02-19T18:50:32.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:32.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:32.923425+0000) 2024-02-19T18:50:33.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:33.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:33.923667+0000) 2024-02-19T18:50:34.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:34.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:34.923872+0000) 2024-02-19T18:50:35.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:35.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:35.924104+0000) 2024-02-19T18:50:36.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:36.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:36.924308+0000) 2024-02-19T18:50:37.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:37.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:37.924496+0000) 2024-02-19T18:50:38.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:38.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:38.924748+0000) 2024-02-19T18:50:39.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:39.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:39.924938+0000) 2024-02-19T18:50:40.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:40.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:40.925120+0000) 2024-02-19T18:50:41.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:41.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:41.925365+0000) 2024-02-19T18:50:42.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:42.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:42.925559+0000) 2024-02-19T18:50:43.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:43.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:43.925793+0000) 2024-02-19T18:50:44.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:44.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:44.926018+0000) 2024-02-19T18:50:45.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:45.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:45.926270+0000) 2024-02-19T18:50:46.370 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:50:46.373 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:50:46.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:46.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:46.926573+0000) 2024-02-19T18:50:47.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:47.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:47.926828+0000) 2024-02-19T18:50:48.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:48.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:48.927096+0000) 2024-02-19T18:50:49.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:49.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:49.927421+0000) 2024-02-19T18:50:50.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:50.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:50.927634+0000) 2024-02-19T18:50:51.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:51.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:51.927866+0000) 2024-02-19T18:50:52.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:52.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:52.928137+0000) 2024-02-19T18:50:53.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:53.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:53.928405+0000) 2024-02-19T18:50:54.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:54.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:54.928690+0000) 2024-02-19T18:50:55.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:55.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:55.928898+0000) 2024-02-19T18:50:56.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:56.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:56.929084+0000) 2024-02-19T18:50:57.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:57.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:57.929338+0000) 2024-02-19T18:50:58.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:58.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:58.929593+0000) 2024-02-19T18:50:59.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:50:59.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:50:59.929823+0000) 2024-02-19T18:51:00.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:00.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:00.930028+0000) 2024-02-19T18:51:01.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:01.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:01.930212+0000) 2024-02-19T18:51:02.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:02.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:02.930439+0000) 2024-02-19T18:51:03.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:03.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:03.930675+0000) 2024-02-19T18:51:04.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:04.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:04.930936+0000) 2024-02-19T18:51:05.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:05.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:05.931133+0000) 2024-02-19T18:51:06.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:06.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:06.931394+0000) 2024-02-19T18:51:07.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:07.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:07.931630+0000) 2024-02-19T18:51:08.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:08.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:08.931883+0000) 2024-02-19T18:51:09.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:09.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:09.932124+0000) 2024-02-19T18:51:10.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:10.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:10.932446+0000) 2024-02-19T18:51:11.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:11.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:11.932653+0000) 2024-02-19T18:51:12.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:12.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:12.932886+0000) 2024-02-19T18:51:13.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:13.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:13.933131+0000) 2024-02-19T18:51:14.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:14.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:14.933329+0000) 2024-02-19T18:51:15.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:15.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:15.933579+0000) 2024-02-19T18:51:16.526 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:51:16.530 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:51:16.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:16.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:16.933853+0000) 2024-02-19T18:51:17.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:17.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:17.934104+0000) 2024-02-19T18:51:18.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:18.934+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:18.934301+0000) 2024-02-19T18:51:19.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:19.934+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:19.934492+0000) 2024-02-19T18:51:20.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:20.934+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:20.934773+0000) 2024-02-19T18:51:21.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:21.934+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:21.935046+0000) 2024-02-19T18:51:22.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:22.934+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:22.935292+0000) 2024-02-19T18:51:23.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:23.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:23.935550+0000) 2024-02-19T18:51:24.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:24.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:24.935805+0000) 2024-02-19T18:51:25.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:25.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:25.936064+0000) 2024-02-19T18:51:26.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:26.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:26.936315+0000) 2024-02-19T18:51:27.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:27.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:27.936541+0000) 2024-02-19T18:51:28.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:28.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:28.936799+0000) 2024-02-19T18:51:29.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:29.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:29.937002+0000) 2024-02-19T18:51:30.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:30.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:30.937260+0000) 2024-02-19T18:51:31.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:31.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:31.937511+0000) 2024-02-19T18:51:32.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:32.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:32.937757+0000) 2024-02-19T18:51:33.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:33.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:33.937944+0000) 2024-02-19T18:51:34.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:34.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:34.938149+0000) 2024-02-19T18:51:35.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:35.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:35.938393+0000) 2024-02-19T18:51:36.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:36.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:36.938663+0000) 2024-02-19T18:51:37.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:37.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:37.938912+0000) 2024-02-19T18:51:38.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:38.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:38.939049+0000) 2024-02-19T18:51:39.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:39.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:39.939236+0000) 2024-02-19T18:51:40.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:40.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:40.939430+0000) 2024-02-19T18:51:41.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:41.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:41.939670+0000) 2024-02-19T18:51:42.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:42.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:42.939921+0000) 2024-02-19T18:51:43.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:43.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:43.940162+0000) 2024-02-19T18:51:44.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:44.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:44.940411+0000) 2024-02-19T18:51:45.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:45.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:45.940631+0000) 2024-02-19T18:51:46.672 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:51:46.676 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:51:46.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:46.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:46.940874+0000) 2024-02-19T18:51:47.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:47.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:47.941149+0000) 2024-02-19T18:51:48.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:48.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:48.941353+0000) 2024-02-19T18:51:49.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:49.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:49.941596+0000) 2024-02-19T18:51:50.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:50.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:50.941821+0000) 2024-02-19T18:51:51.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:51.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:51.942049+0000) 2024-02-19T18:51:52.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:52.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:52.942253+0000) 2024-02-19T18:51:53.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:53.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:53.942431+0000) 2024-02-19T18:51:54.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:54.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:54.942668+0000) 2024-02-19T18:51:55.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:55.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:55.942939+0000) 2024-02-19T18:51:56.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:56.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:56.943162+0000) 2024-02-19T18:51:57.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:57.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:57.943400+0000) 2024-02-19T18:51:58.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:58.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:58.943679+0000) 2024-02-19T18:51:59.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:51:59.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:51:59.943936+0000) 2024-02-19T18:52:00.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:00.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:00.944187+0000) 2024-02-19T18:52:01.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:01.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:01.944425+0000) 2024-02-19T18:52:02.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:02.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:02.944619+0000) 2024-02-19T18:52:03.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:03.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:03.944818+0000) 2024-02-19T18:52:04.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:04.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:04.945097+0000) 2024-02-19T18:52:05.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:05.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:05.945267+0000) 2024-02-19T18:52:06.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:06.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:06.945489+0000) 2024-02-19T18:52:07.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:07.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:07.945742+0000) 2024-02-19T18:52:08.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:08.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:08.945967+0000) 2024-02-19T18:52:09.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:09.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:09.946153+0000) 2024-02-19T18:52:10.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:10.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:10.946416+0000) 2024-02-19T18:52:11.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:11.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:11.946675+0000) 2024-02-19T18:52:12.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:12.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:12.946939+0000) 2024-02-19T18:52:13.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:13.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:13.947157+0000) 2024-02-19T18:52:14.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:14.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:14.947348+0000) 2024-02-19T18:52:15.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:15.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:15.947601+0000) 2024-02-19T18:52:16.728 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:52:16.731 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:52:16.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:16.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:16.947846+0000) 2024-02-19T18:52:17.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:17.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:17.948082+0000) 2024-02-19T18:52:18.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:18.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:18.948255+0000) 2024-02-19T18:52:19.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:19.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:19.948489+0000) 2024-02-19T18:52:20.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:20.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:20.948750+0000) 2024-02-19T18:52:21.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:21.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:21.948996+0000) 2024-02-19T18:52:22.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:22.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:22.949264+0000) 2024-02-19T18:52:23.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:23.945+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:23.949469+0000) 2024-02-19T18:52:24.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:24.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:24.949679+0000) 2024-02-19T18:52:25.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:25.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:25.949937+0000) 2024-02-19T18:52:26.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:26.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:26.950153+0000) 2024-02-19T18:52:27.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:27.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:27.950404+0000) 2024-02-19T18:52:28.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:28.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:28.950588+0000) 2024-02-19T18:52:29.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:29.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:29.950828+0000) 2024-02-19T18:52:30.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:30.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:30.951032+0000) 2024-02-19T18:52:31.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:31.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:31.951281+0000) 2024-02-19T18:52:32.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:32.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:32.951495+0000) 2024-02-19T18:52:33.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:33.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:33.951680+0000) 2024-02-19T18:52:34.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:34.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:34.951914+0000) 2024-02-19T18:52:35.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:35.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:35.952049+0000) 2024-02-19T18:52:36.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:36.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:36.952311+0000) 2024-02-19T18:52:37.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:37.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:37.952551+0000) 2024-02-19T18:52:38.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:38.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:38.952800+0000) 2024-02-19T18:52:39.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:39.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:39.953009+0000) 2024-02-19T18:52:40.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:40.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:40.953144+0000) 2024-02-19T18:52:41.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:41.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:41.953409+0000) 2024-02-19T18:52:42.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:42.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:42.953599+0000) 2024-02-19T18:52:43.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:43.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:43.953794+0000) 2024-02-19T18:52:44.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:44.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:44.954026+0000) 2024-02-19T18:52:45.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:45.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:45.954249+0000) 2024-02-19T18:52:46.892 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:52:46.895 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:52:46.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:46.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:46.954512+0000) 2024-02-19T18:52:47.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:47.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:47.954808+0000) 2024-02-19T18:52:48.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:48.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:48.954956+0000) 2024-02-19T18:52:49.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:49.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:49.955155+0000) 2024-02-19T18:52:50.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:50.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:50.955376+0000) 2024-02-19T18:52:51.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:51.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:51.955634+0000) 2024-02-19T18:52:52.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:52.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:52.955821+0000) 2024-02-19T18:52:53.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:53.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:53.955986+0000) 2024-02-19T18:52:54.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:54.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:54.956222+0000) 2024-02-19T18:52:55.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:55.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:55.956429+0000) 2024-02-19T18:52:56.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:56.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:56.956669+0000) 2024-02-19T18:52:57.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:57.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:57.956929+0000) 2024-02-19T18:52:58.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:58.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:58.957185+0000) 2024-02-19T18:52:59.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:52:59.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:52:59.957439+0000) 2024-02-19T18:53:00.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:00.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:00.957641+0000) 2024-02-19T18:53:01.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:01.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:01.957857+0000) 2024-02-19T18:53:02.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:02.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:02.958107+0000) 2024-02-19T18:53:03.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:03.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:03.958344+0000) 2024-02-19T18:53:04.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:04.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:04.958578+0000) 2024-02-19T18:53:05.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:05.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:05.958855+0000) 2024-02-19T18:53:06.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:06.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:06.959064+0000) 2024-02-19T18:53:07.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:07.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:07.959263+0000) 2024-02-19T18:53:08.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:08.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:08.959447+0000) 2024-02-19T18:53:09.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:09.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:09.959697+0000) 2024-02-19T18:53:10.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:10.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:10.959912+0000) 2024-02-19T18:53:11.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:11.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:11.960106+0000) 2024-02-19T18:53:12.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:12.960+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:12.960352+0000) 2024-02-19T18:53:13.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:13.960+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:13.960636+0000) 2024-02-19T18:53:14.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:14.960+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:14.960830+0000) 2024-02-19T18:53:15.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:15.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:15.961064+0000) 2024-02-19T18:53:16.972 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:53:16.975 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:53:16.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:16.960+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:16.961337+0000) 2024-02-19T18:53:17.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:17.960+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:17.961532+0000) 2024-02-19T18:53:18.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:18.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:18.961719+0000) 2024-02-19T18:53:19.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:19.960+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:19.961921+0000) 2024-02-19T18:53:20.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:20.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:20.962150+0000) 2024-02-19T18:53:21.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:21.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:21.962419+0000) 2024-02-19T18:53:22.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:22.960+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:22.962699+0000) 2024-02-19T18:53:23.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:23.960+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:23.962908+0000) 2024-02-19T18:53:24.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:24.960+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:24.963103+0000) 2024-02-19T18:53:25.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:25.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:25.963293+0000) 2024-02-19T18:53:26.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:26.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:26.963444+0000) 2024-02-19T18:53:27.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:27.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:27.963697+0000) 2024-02-19T18:53:28.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:28.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:28.963937+0000) 2024-02-19T18:53:29.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:29.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:29.964116+0000) 2024-02-19T18:53:30.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:30.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:30.964316+0000) 2024-02-19T18:53:31.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:31.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:31.964574+0000) 2024-02-19T18:53:32.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:32.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:32.964830+0000) 2024-02-19T18:53:33.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:33.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:33.965060+0000) 2024-02-19T18:53:34.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:34.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:34.965301+0000) 2024-02-19T18:53:35.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:35.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:35.965553+0000) 2024-02-19T18:53:36.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:36.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:36.965818+0000) 2024-02-19T18:53:37.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:37.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:37.966030+0000) 2024-02-19T18:53:38.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:38.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:38.966279+0000) 2024-02-19T18:53:39.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:39.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:39.966535+0000) 2024-02-19T18:53:40.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:40.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:40.966837+0000) 2024-02-19T18:53:41.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:41.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:41.967068+0000) 2024-02-19T18:53:42.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:42.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:42.967286+0000) 2024-02-19T18:53:43.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:43.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:43.967521+0000) 2024-02-19T18:53:44.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:44.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:44.967771+0000) 2024-02-19T18:53:45.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:45.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:45.968022+0000) 2024-02-19T18:53:46.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:46.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:46.968268+0000) 2024-02-19T18:53:47.034 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:53:47.038 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:53:47.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:47.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:47.968527+0000) 2024-02-19T18:53:48.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:48.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:48.968807+0000) 2024-02-19T18:53:49.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:49.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:49.968998+0000) 2024-02-19T18:53:50.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:50.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:50.969161+0000) 2024-02-19T18:53:51.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:51.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:51.969360+0000) 2024-02-19T18:53:52.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:52.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:52.969519+0000) 2024-02-19T18:53:53.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:53.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:53.969724+0000) 2024-02-19T18:53:54.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:54.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:54.969967+0000) 2024-02-19T18:53:55.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:55.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:55.970196+0000) 2024-02-19T18:53:56.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:56.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:56.970470+0000) 2024-02-19T18:53:57.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:57.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:57.970773+0000) 2024-02-19T18:53:58.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:58.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:58.971046+0000) 2024-02-19T18:53:59.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:53:59.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:53:59.971305+0000) 2024-02-19T18:54:00.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:00.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:00.971531+0000) 2024-02-19T18:54:01.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:01.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:01.971796+0000) 2024-02-19T18:54:02.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:02.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:02.972042+0000) 2024-02-19T18:54:03.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:03.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:03.972306+0000) 2024-02-19T18:54:04.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:04.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:04.972477+0000) 2024-02-19T18:54:05.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:05.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:05.972710+0000) 2024-02-19T18:54:06.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:06.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:06.972882+0000) 2024-02-19T18:54:07.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:07.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:07.973038+0000) 2024-02-19T18:54:08.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:08.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:08.973166+0000) 2024-02-19T18:54:09.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:09.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:09.973416+0000) 2024-02-19T18:54:10.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:10.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:10.973682+0000) 2024-02-19T18:54:11.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:11.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:11.973935+0000) 2024-02-19T18:54:12.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:12.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:12.974182+0000) 2024-02-19T18:54:13.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:13.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:13.974449+0000) 2024-02-19T18:54:14.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:14.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:14.974770+0000) 2024-02-19T18:54:15.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:15.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:15.974995+0000) 2024-02-19T18:54:16.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:16.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:16.975213+0000) 2024-02-19T18:54:17.151 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:54:17.154 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:54:17.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:17.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:17.975403+0000) 2024-02-19T18:54:18.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:18.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:18.975673+0000) 2024-02-19T18:54:19.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:19.975+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:19.975944+0000) 2024-02-19T18:54:20.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:20.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:20.976200+0000) 2024-02-19T18:54:21.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:21.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:21.976459+0000) 2024-02-19T18:54:22.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:22.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:22.976637+0000) 2024-02-19T18:54:23.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:23.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:23.976888+0000) 2024-02-19T18:54:24.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:24.971+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:24.977099+0000) 2024-02-19T18:54:25.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:25.975+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:25.977291+0000) 2024-02-19T18:54:26.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:26.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:26.977561+0000) 2024-02-19T18:54:27.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:27.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:27.977792+0000) 2024-02-19T18:54:28.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:28.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:28.978040+0000) 2024-02-19T18:54:29.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:29.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:29.978260+0000) 2024-02-19T18:54:30.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:30.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:30.978453+0000) 2024-02-19T18:54:31.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:31.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:31.978683+0000) 2024-02-19T18:54:32.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:32.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:32.978882+0000) 2024-02-19T18:54:33.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:33.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:33.979050+0000) 2024-02-19T18:54:34.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:34.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:34.979290+0000) 2024-02-19T18:54:35.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:35.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:35.979500+0000) 2024-02-19T18:54:36.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:36.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:36.979735+0000) 2024-02-19T18:54:37.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:37.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:37.979982+0000) 2024-02-19T18:54:38.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:38.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:38.980255+0000) 2024-02-19T18:54:39.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:39.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:39.980525+0000) 2024-02-19T18:54:40.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:40.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:40.980849+0000) 2024-02-19T18:54:41.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:41.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:41.981157+0000) 2024-02-19T18:54:42.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:42.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:42.981450+0000) 2024-02-19T18:54:43.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:43.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:43.981641+0000) 2024-02-19T18:54:44.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:44.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:44.981957+0000) 2024-02-19T18:54:45.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:45.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:45.982172+0000) 2024-02-19T18:54:46.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:46.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:46.982458+0000) 2024-02-19T18:54:47.273 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:54:47.278 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:54:47.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:47.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:47.982813+0000) 2024-02-19T18:54:48.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:48.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:48.983098+0000) 2024-02-19T18:54:49.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:49.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:49.983396+0000) 2024-02-19T18:54:50.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:50.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:50.983699+0000) 2024-02-19T18:54:51.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:51.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:51.983978+0000) 2024-02-19T18:54:52.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:52.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:52.984189+0000) 2024-02-19T18:54:53.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:53.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:53.984441+0000) 2024-02-19T18:54:54.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:54.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:54.984727+0000) 2024-02-19T18:54:56.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:55.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:55.985009+0000) 2024-02-19T18:54:57.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:56.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:56.985315+0000) 2024-02-19T18:54:58.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:57.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:57.985521+0000) 2024-02-19T18:54:59.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:58.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:58.985746+0000) 2024-02-19T18:55:00.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:54:59.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:54:59.986003+0000) 2024-02-19T18:55:01.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:00.982+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:00.986264+0000) 2024-02-19T18:55:02.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:01.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:01.986500+0000) 2024-02-19T18:55:03.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:02.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:02.986845+0000) 2024-02-19T18:55:04.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:03.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:03.987128+0000) 2024-02-19T18:55:05.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:04.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:04.987357+0000) 2024-02-19T18:55:06.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:05.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:05.987630+0000) 2024-02-19T18:55:07.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:06.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:06.987932+0000) 2024-02-19T18:55:08.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:07.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:07.988250+0000) 2024-02-19T18:55:09.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:08.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:08.988526+0000) 2024-02-19T18:55:10.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:09.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:09.988757+0000) 2024-02-19T18:55:11.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:10.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:10.988940+0000) 2024-02-19T18:55:12.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:11.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:11.989205+0000) 2024-02-19T18:55:13.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:12.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:12.989463+0000) 2024-02-19T18:55:14.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:13.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:13.989683+0000) 2024-02-19T18:55:15.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:14.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:14.989891+0000) 2024-02-19T18:55:16.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:15.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:15.990167+0000) 2024-02-19T18:55:17.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:16.990+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:16.990406+0000) 2024-02-19T18:55:17.378 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:55:17.382 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:55:18.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:17.986+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:17.990650+0000) 2024-02-19T18:55:19.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:18.990+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:18.990839+0000) 2024-02-19T18:55:20.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:19.990+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:19.991042+0000) 2024-02-19T18:55:21.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:20.990+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:20.991254+0000) 2024-02-19T18:55:22.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:21.990+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:21.991434+0000) 2024-02-19T18:55:23.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:22.990+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:22.991642+0000) 2024-02-19T18:55:24.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:23.990+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:23.991929+0000) 2024-02-19T18:55:25.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:24.990+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:24.992242+0000) 2024-02-19T18:55:26.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:25.990+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:25.992483+0000) 2024-02-19T18:55:27.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:26.990+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:26.992834+0000) 2024-02-19T18:55:28.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:27.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:27.993048+0000) 2024-02-19T18:55:29.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:28.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:28.993292+0000) 2024-02-19T18:55:30.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:29.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:29.993530+0000) 2024-02-19T18:55:31.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:30.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:30.993732+0000) 2024-02-19T18:55:32.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:31.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:31.993962+0000) 2024-02-19T18:55:33.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:32.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:32.994213+0000) 2024-02-19T18:55:34.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:33.989+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:33.994534+0000) 2024-02-19T18:55:35.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:34.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:34.994795+0000) 2024-02-19T18:55:36.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:35.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:35.995000+0000) 2024-02-19T18:55:37.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:36.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:36.995170+0000) 2024-02-19T18:55:38.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:37.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:37.995308+0000) 2024-02-19T18:55:39.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:38.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:38.995588+0000) 2024-02-19T18:55:40.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:39.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:39.995883+0000) 2024-02-19T18:55:41.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:40.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:40.996181+0000) 2024-02-19T18:55:42.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:41.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:41.996380+0000) 2024-02-19T18:55:43.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:42.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:42.996620+0000) 2024-02-19T18:55:44.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:43.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:43.996827+0000) 2024-02-19T18:55:45.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:44.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:44.997039+0000) 2024-02-19T18:55:46.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:45.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:45.997260+0000) 2024-02-19T18:55:47.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:46.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:46.997459+0000) 2024-02-19T18:55:47.434 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:55:47.437 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:55:48.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:47.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:47.997679+0000) 2024-02-19T18:55:49.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:48.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:48.997971+0000) 2024-02-19T18:55:50.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:49.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:49.998257+0000) 2024-02-19T18:55:51.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:50.993+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:50.998552+0000) 2024-02-19T18:55:52.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:51.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:51.998767+0000) 2024-02-19T18:55:53.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:52.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:52.999021+0000) 2024-02-19T18:55:54.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:53.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:53.999295+0000) 2024-02-19T18:55:55.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:54.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:54.999588+0000) 2024-02-19T18:55:56.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:55.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:55.999899+0000) 2024-02-19T18:55:57.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:56.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:57.000177+0000) 2024-02-19T18:55:58.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:57.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:58.000445+0000) 2024-02-19T18:55:59.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:58.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:55:59.000742+0000) 2024-02-19T18:56:00.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:55:59.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:00.001087+0000) 2024-02-19T18:56:01.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:00.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:01.001282+0000) 2024-02-19T18:56:02.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:01.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:02.001553+0000) 2024-02-19T18:56:03.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:02.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:03.001768+0000) 2024-02-19T18:56:04.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:03.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:04.002043+0000) 2024-02-19T18:56:05.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:04.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:05.002285+0000) 2024-02-19T18:56:06.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:05.997+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:06.002561+0000) 2024-02-19T18:56:07.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:07.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:07.002845+0000) 2024-02-19T18:56:08.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:08.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:08.003090+0000) 2024-02-19T18:56:09.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:09.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:09.003323+0000) 2024-02-19T18:56:10.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:10.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:10.003526+0000) 2024-02-19T18:56:11.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:11.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:11.003789+0000) 2024-02-19T18:56:12.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:12.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:12.003998+0000) 2024-02-19T18:56:13.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:13.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:13.004209+0000) 2024-02-19T18:56:14.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:14.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:14.004354+0000) 2024-02-19T18:56:15.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:15.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:15.004620+0000) 2024-02-19T18:56:16.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:16.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:16.004907+0000) 2024-02-19T18:56:17.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:17.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:17.005257+0000) 2024-02-19T18:56:17.560 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:56:17.563 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:56:18.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:18.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:18.005530+0000) 2024-02-19T18:56:19.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:19.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:19.005752+0000) 2024-02-19T18:56:20.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:20.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:20.005969+0000) 2024-02-19T18:56:21.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:21.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:21.006187+0000) 2024-02-19T18:56:22.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:22.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:22.006448+0000) 2024-02-19T18:56:23.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:23.001+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:23.006658+0000) 2024-02-19T18:56:24.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:24.005+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:24.006900+0000) 2024-02-19T18:56:25.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:25.005+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:25.007092+0000) 2024-02-19T18:56:26.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:26.005+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:26.007381+0000) 2024-02-19T18:56:27.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:27.005+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:27.007682+0000) 2024-02-19T18:56:28.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:28.005+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:28.007887+0000) 2024-02-19T18:56:29.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:29.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:29.008173+0000) 2024-02-19T18:56:30.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:30.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:30.008394+0000) 2024-02-19T18:56:31.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:31.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:31.008627+0000) 2024-02-19T18:56:32.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:32.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:32.008846+0000) 2024-02-19T18:56:33.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:33.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:33.009122+0000) 2024-02-19T18:56:34.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:34.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:34.009414+0000) 2024-02-19T18:56:35.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:35.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:35.009705+0000) 2024-02-19T18:56:36.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:36.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:36.009942+0000) 2024-02-19T18:56:37.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:37.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:37.010157+0000) 2024-02-19T18:56:38.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:38.004+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:38.010413+0000) 2024-02-19T18:56:39.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:39.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:39.010725+0000) 2024-02-19T18:56:40.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:40.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:40.010911+0000) 2024-02-19T18:56:41.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:41.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:41.011147+0000) 2024-02-19T18:56:42.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:42.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:42.011426+0000) 2024-02-19T18:56:43.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:43.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:43.011673+0000) 2024-02-19T18:56:44.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:44.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:44.011919+0000) 2024-02-19T18:56:45.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:45.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:45.012148+0000) 2024-02-19T18:56:46.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:46.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:46.012417+0000) 2024-02-19T18:56:47.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:47.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:47.012614+0000) 2024-02-19T18:56:47.687 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:56:47.691 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:56:48.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:48.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:48.012827+0000) 2024-02-19T18:56:49.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:49.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:49.013044+0000) 2024-02-19T18:56:50.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:50.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:50.013242+0000) 2024-02-19T18:56:51.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:51.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:51.013400+0000) 2024-02-19T18:56:52.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:52.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:52.013539+0000) 2024-02-19T18:56:53.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:53.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:53.013704+0000) 2024-02-19T18:56:54.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:54.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:54.013932+0000) 2024-02-19T18:56:55.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:55.008+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:55.014180+0000) 2024-02-19T18:56:56.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:56.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:56.014378+0000) 2024-02-19T18:56:57.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:57.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:57.014547+0000) 2024-02-19T18:56:58.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:58.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:58.014763+0000) 2024-02-19T18:56:59.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:56:59.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:56:59.014925+0000) 2024-02-19T18:57:00.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:00.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:00.015130+0000) 2024-02-19T18:57:01.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:01.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:01.015320+0000) 2024-02-19T18:57:02.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:02.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:02.015556+0000) 2024-02-19T18:57:03.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:03.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:03.015776+0000) 2024-02-19T18:57:04.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:04.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:04.016011+0000) 2024-02-19T18:57:05.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:05.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:05.016261+0000) 2024-02-19T18:57:06.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:06.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:06.016505+0000) 2024-02-19T18:57:07.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:07.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:07.016750+0000) 2024-02-19T18:57:08.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:08.016+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:08.016941+0000) 2024-02-19T18:57:09.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:09.016+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:09.017203+0000) 2024-02-19T18:57:10.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:10.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:10.017444+0000) 2024-02-19T18:57:11.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:11.016+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:11.017657+0000) 2024-02-19T18:57:12.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:12.016+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:12.017853+0000) 2024-02-19T18:57:13.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:13.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:13.018119+0000) 2024-02-19T18:57:14.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:14.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:14.018351+0000) 2024-02-19T18:57:15.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:15.012+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:15.018524+0000) 2024-02-19T18:57:16.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:16.016+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:16.018818+0000) 2024-02-19T18:57:17.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:17.016+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:17.019067+0000) 2024-02-19T18:57:17.801 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:57:17.804 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:57:18.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:18.016+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:18.019322+0000) 2024-02-19T18:57:19.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:19.016+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:19.019633+0000) 2024-02-19T18:57:20.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:20.016+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:20.019877+0000) 2024-02-19T18:57:21.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:21.016+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:21.020108+0000) 2024-02-19T18:57:22.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:22.020+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:22.020389+0000) 2024-02-19T18:57:23.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:23.016+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:23.020618+0000) 2024-02-19T18:57:24.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:24.016+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:24.020873+0000) 2024-02-19T18:57:25.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:25.020+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:25.021116+0000) 2024-02-19T18:57:26.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:26.020+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:26.021391+0000) 2024-02-19T18:57:27.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:27.016+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:27.021662+0000) 2024-02-19T18:57:28.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:28.016+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:28.021895+0000) 2024-02-19T18:57:29.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:29.020+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:29.022074+0000) 2024-02-19T18:57:30.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:30.015+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:30.022312+0000) 2024-02-19T18:57:31.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:31.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:31.022577+0000) 2024-02-19T18:57:32.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:32.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:32.022800+0000) 2024-02-19T18:57:33.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:33.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:33.022990+0000) 2024-02-19T18:57:34.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:34.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:34.023222+0000) 2024-02-19T18:57:35.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:35.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:35.023419+0000) 2024-02-19T18:57:36.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:36.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:36.023649+0000) 2024-02-19T18:57:37.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:37.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:37.023899+0000) 2024-02-19T18:57:38.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:38.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:38.024178+0000) 2024-02-19T18:57:39.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:39.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:39.024423+0000) 2024-02-19T18:57:40.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:40.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:40.024627+0000) 2024-02-19T18:57:41.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:41.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:41.024893+0000) 2024-02-19T18:57:42.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:42.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:42.025126+0000) 2024-02-19T18:57:43.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:43.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:43.025293+0000) 2024-02-19T18:57:44.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:44.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:44.025523+0000) 2024-02-19T18:57:45.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:45.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:45.025739+0000) 2024-02-19T18:57:46.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:46.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:46.025915+0000) 2024-02-19T18:57:47.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:47.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:47.026133+0000) 2024-02-19T18:57:47.940 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:57:47.943 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:57:48.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:48.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:48.026333+0000) 2024-02-19T18:57:49.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:49.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:49.026509+0000) 2024-02-19T18:57:50.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:50.019+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:50.026673+0000) 2024-02-19T18:57:51.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:51.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:51.026943+0000) 2024-02-19T18:57:52.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:52.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:52.027173+0000) 2024-02-19T18:57:53.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:53.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:53.027401+0000) 2024-02-19T18:57:54.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:54.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:54.027560+0000) 2024-02-19T18:57:55.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:55.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:55.027736+0000) 2024-02-19T18:57:56.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:56.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:56.027858+0000) 2024-02-19T18:57:57.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:57.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:57.028101+0000) 2024-02-19T18:57:58.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:58.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:58.028334+0000) 2024-02-19T18:57:59.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:57:59.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:57:59.028515+0000) 2024-02-19T18:58:00.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:00.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:00.028700+0000) 2024-02-19T18:58:01.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:01.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:01.028926+0000) 2024-02-19T18:58:02.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:02.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:02.029241+0000) 2024-02-19T18:58:03.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:03.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:03.029435+0000) 2024-02-19T18:58:04.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:04.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:04.029648+0000) 2024-02-19T18:58:05.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:05.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:05.029884+0000) 2024-02-19T18:58:06.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:06.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:06.030111+0000) 2024-02-19T18:58:07.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:07.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:07.030393+0000) 2024-02-19T18:58:08.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:08.023+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:08.030562+0000) 2024-02-19T18:58:09.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:09.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:09.030823+0000) 2024-02-19T18:58:10.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:10.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:10.031053+0000) 2024-02-19T18:58:11.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:11.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:11.031258+0000) 2024-02-19T18:58:12.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:12.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:12.031475+0000) 2024-02-19T18:58:13.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:13.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:13.031673+0000) 2024-02-19T18:58:14.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:14.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:14.031898+0000) 2024-02-19T18:58:15.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:15.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:15.032168+0000) 2024-02-19T18:58:16.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:16.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:16.032433+0000) 2024-02-19T18:58:17.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:17.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:17.032633+0000) 2024-02-19T18:58:18.021 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:58:18.024 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:58:18.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:18.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:18.032881+0000) 2024-02-19T18:58:19.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:19.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:19.033085+0000) 2024-02-19T18:58:20.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:20.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:20.033336+0000) 2024-02-19T18:58:21.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:21.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:21.033525+0000) 2024-02-19T18:58:22.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:22.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:22.033758+0000) 2024-02-19T18:58:23.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:23.027+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:23.033941+0000) 2024-02-19T18:58:24.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:24.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:24.034179+0000) 2024-02-19T18:58:25.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:25.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:25.034405+0000) 2024-02-19T18:58:26.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:26.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:26.034657+0000) 2024-02-19T18:58:27.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:27.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:27.034794+0000) 2024-02-19T18:58:28.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:28.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:28.035003+0000) 2024-02-19T18:58:29.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:29.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:29.035246+0000) 2024-02-19T18:58:30.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:30.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:30.035464+0000) 2024-02-19T18:58:31.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:31.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:31.035655+0000) 2024-02-19T18:58:32.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:32.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:32.035882+0000) 2024-02-19T18:58:33.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:33.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:33.036127+0000) 2024-02-19T18:58:34.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:34.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:34.036329+0000) 2024-02-19T18:58:35.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:35.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:35.036574+0000) 2024-02-19T18:58:36.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:36.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:36.036776+0000) 2024-02-19T18:58:37.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:37.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:37.037041+0000) 2024-02-19T18:58:38.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:38.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:38.037246+0000) 2024-02-19T18:58:39.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:39.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:39.037460+0000) 2024-02-19T18:58:40.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:40.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:40.037721+0000) 2024-02-19T18:58:41.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:41.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:41.037897+0000) 2024-02-19T18:58:42.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:42.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:42.038033+0000) 2024-02-19T18:58:43.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:43.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:43.038271+0000) 2024-02-19T18:58:44.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:44.030+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:44.038490+0000) 2024-02-19T18:58:45.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:45.034+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:45.038680+0000) 2024-02-19T18:58:46.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:46.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:46.038886+0000) 2024-02-19T18:58:47.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:47.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:47.039093+0000) 2024-02-19T18:58:48.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:48.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:48.039266+0000) 2024-02-19T18:58:48.076 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:58:48.080 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:58:49.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:49.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:49.039479+0000) 2024-02-19T18:58:50.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:50.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:50.039668+0000) 2024-02-19T18:58:51.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:51.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:51.039895+0000) 2024-02-19T18:58:52.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:52.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:52.040134+0000) 2024-02-19T18:58:53.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:53.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:53.040371+0000) 2024-02-19T18:58:54.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:54.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:54.040569+0000) 2024-02-19T18:58:55.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:55.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:55.040764+0000) 2024-02-19T18:58:56.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:56.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:56.041003+0000) 2024-02-19T18:58:57.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:57.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:57.041259+0000) 2024-02-19T18:58:58.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:58.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:58.041496+0000) 2024-02-19T18:58:59.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:58:59.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:58:59.041744+0000) 2024-02-19T18:59:00.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:00.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:00.041987+0000) 2024-02-19T18:59:01.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:01.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:01.042214+0000) 2024-02-19T18:59:02.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:02.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:02.042431+0000) 2024-02-19T18:59:03.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:03.038+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:03.042604+0000) 2024-02-19T18:59:04.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:04.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:04.042836+0000) 2024-02-19T18:59:05.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:05.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:05.043111+0000) 2024-02-19T18:59:06.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:06.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:06.043314+0000) 2024-02-19T18:59:07.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:07.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:07.043571+0000) 2024-02-19T18:59:08.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:08.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:08.043816+0000) 2024-02-19T18:59:09.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:09.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:09.044008+0000) 2024-02-19T18:59:10.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:10.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:10.044254+0000) 2024-02-19T18:59:11.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:11.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:11.044498+0000) 2024-02-19T18:59:12.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:12.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:12.044743+0000) 2024-02-19T18:59:13.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:13.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:13.044977+0000) 2024-02-19T18:59:14.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:14.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:14.045220+0000) 2024-02-19T18:59:15.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:15.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:15.045487+0000) 2024-02-19T18:59:16.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:16.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:16.045719+0000) 2024-02-19T18:59:17.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:17.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:17.045870+0000) 2024-02-19T18:59:18.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:18.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:18.046075+0000) 2024-02-19T18:59:18.165 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:59:18.168 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:59:19.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:19.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:19.046321+0000) 2024-02-19T18:59:20.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:20.042+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:20.046575+0000) 2024-02-19T18:59:21.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:21.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:21.046839+0000) 2024-02-19T18:59:22.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:22.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:22.047085+0000) 2024-02-19T18:59:23.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:23.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:23.047342+0000) 2024-02-19T18:59:24.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:24.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:24.047580+0000) 2024-02-19T18:59:25.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:25.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:25.047774+0000) 2024-02-19T18:59:26.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:26.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:26.047982+0000) 2024-02-19T18:59:27.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:27.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:27.048192+0000) 2024-02-19T18:59:28.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:28.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:28.048423+0000) 2024-02-19T18:59:29.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:29.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:29.048658+0000) 2024-02-19T18:59:30.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:30.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:30.048860+0000) 2024-02-19T18:59:31.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:31.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:31.049069+0000) 2024-02-19T18:59:32.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:32.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:32.049251+0000) 2024-02-19T18:59:33.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:33.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:33.049497+0000) 2024-02-19T18:59:34.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:34.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:34.049757+0000) 2024-02-19T18:59:35.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:35.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:35.049928+0000) 2024-02-19T18:59:36.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:36.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:36.050165+0000) 2024-02-19T18:59:37.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:37.045+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:37.050434+0000) 2024-02-19T18:59:38.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:38.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:38.050655+0000) 2024-02-19T18:59:39.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:39.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:39.050823+0000) 2024-02-19T18:59:40.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:40.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:40.051006+0000) 2024-02-19T18:59:41.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:41.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:41.051277+0000) 2024-02-19T18:59:42.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:42.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:42.051532+0000) 2024-02-19T18:59:43.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:43.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:43.051779+0000) 2024-02-19T18:59:44.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:44.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:44.052027+0000) 2024-02-19T18:59:45.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:45.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:45.052267+0000) 2024-02-19T18:59:46.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:46.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:46.052471+0000) 2024-02-19T18:59:47.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:47.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:47.052659+0000) 2024-02-19T18:59:48.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:48.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:48.052857+0000) 2024-02-19T18:59:48.295 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:59:48.299 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T18:59:49.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:49.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:49.053063+0000) 2024-02-19T18:59:50.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:50.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:50.053247+0000) 2024-02-19T18:59:51.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:51.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:51.053464+0000) 2024-02-19T18:59:52.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:52.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:52.053657+0000) 2024-02-19T18:59:53.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:53.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:53.053893+0000) 2024-02-19T18:59:54.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:54.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:54.054112+0000) 2024-02-19T18:59:55.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:55.049+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:55.054288+0000) 2024-02-19T18:59:56.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:56.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:56.054554+0000) 2024-02-19T18:59:57.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:57.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:57.054828+0000) 2024-02-19T18:59:58.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:58.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:58.055065+0000) 2024-02-19T18:59:59.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T18:59:59.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T17:59:59.055344+0000) 2024-02-19T19:00:00.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:00.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:00.055587+0000) 2024-02-19T19:00:01.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:01.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:01.055845+0000) 2024-02-19T19:00:02.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:02.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:02.056050+0000) 2024-02-19T19:00:03.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:03.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:03.056243+0000) 2024-02-19T19:00:04.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:04.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:04.056481+0000) 2024-02-19T19:00:05.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:05.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:05.056659+0000) 2024-02-19T19:00:06.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:06.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:06.056927+0000) 2024-02-19T19:00:07.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:07.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:07.057220+0000) 2024-02-19T19:00:08.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:08.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:08.057484+0000) 2024-02-19T19:00:09.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:09.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:09.057756+0000) 2024-02-19T19:00:10.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:10.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:10.057989+0000) 2024-02-19T19:00:11.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:11.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:11.058200+0000) 2024-02-19T19:00:12.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:12.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:12.058439+0000) 2024-02-19T19:00:13.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:13.053+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:13.058681+0000) 2024-02-19T19:00:14.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:14.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:14.058886+0000) 2024-02-19T19:00:15.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:15.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:15.059152+0000) 2024-02-19T19:00:16.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:16.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:16.059404+0000) 2024-02-19T19:00:17.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:17.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:17.059678+0000) 2024-02-19T19:00:18.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:18.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:18.059878+0000) 2024-02-19T19:00:18.426 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:00:18.430 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:00:19.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:19.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:19.060089+0000) 2024-02-19T19:00:20.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:20.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:20.060369+0000) 2024-02-19T19:00:21.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:21.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:21.060629+0000) 2024-02-19T19:00:22.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:22.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:22.060908+0000) 2024-02-19T19:00:23.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:23.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:23.061120+0000) 2024-02-19T19:00:24.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:24.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:24.061369+0000) 2024-02-19T19:00:25.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:25.057+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:25.061544+0000) 2024-02-19T19:00:26.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:26.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:26.061799+0000) 2024-02-19T19:00:27.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:27.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:27.062090+0000) 2024-02-19T19:00:28.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:28.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:28.062350+0000) 2024-02-19T19:00:29.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:29.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:29.062611+0000) 2024-02-19T19:00:30.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:30.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:30.062876+0000) 2024-02-19T19:00:31.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:31.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:31.063094+0000) 2024-02-19T19:00:32.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:32.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:32.063332+0000) 2024-02-19T19:00:33.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:33.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:33.063585+0000) 2024-02-19T19:00:34.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:34.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:34.063827+0000) 2024-02-19T19:00:35.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:35.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:35.064066+0000) 2024-02-19T19:00:36.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:36.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:36.064268+0000) 2024-02-19T19:00:37.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:37.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:37.064424+0000) 2024-02-19T19:00:38.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:38.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:38.064593+0000) 2024-02-19T19:00:39.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:39.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:39.064907+0000) 2024-02-19T19:00:40.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:40.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:40.065152+0000) 2024-02-19T19:00:41.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:41.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:41.065399+0000) 2024-02-19T19:00:42.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:42.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:42.065667+0000) 2024-02-19T19:00:43.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:43.060+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:43.065892+0000) 2024-02-19T19:00:44.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:44.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:44.066150+0000) 2024-02-19T19:00:45.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:45.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:45.066298+0000) 2024-02-19T19:00:46.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:46.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:46.066533+0000) 2024-02-19T19:00:47.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:47.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:47.066790+0000) 2024-02-19T19:00:48.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:48.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:48.067020+0000) 2024-02-19T19:00:48.558 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:00:48.561 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:00:49.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:49.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:49.067170+0000) 2024-02-19T19:00:50.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:50.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:50.067299+0000) 2024-02-19T19:00:51.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:51.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:51.067545+0000) 2024-02-19T19:00:52.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:52.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:52.067792+0000) 2024-02-19T19:00:53.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:53.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:53.068027+0000) 2024-02-19T19:00:54.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:54.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:54.068303+0000) 2024-02-19T19:00:55.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:55.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:55.068545+0000) 2024-02-19T19:00:56.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:56.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:56.068795+0000) 2024-02-19T19:00:57.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:57.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:57.069050+0000) 2024-02-19T19:00:58.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:58.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:58.069204+0000) 2024-02-19T19:00:59.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:00:59.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:00:59.069469+0000) 2024-02-19T19:01:00.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:00.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:00.069720+0000) 2024-02-19T19:01:01.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:01.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:01.069949+0000) 2024-02-19T19:01:02.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:02.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:02.070192+0000) 2024-02-19T19:01:03.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:03.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:03.070397+0000) 2024-02-19T19:01:04.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:04.064+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:04.070654+0000) 2024-02-19T19:01:05.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:05.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:05.070927+0000) 2024-02-19T19:01:06.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:06.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:06.071123+0000) 2024-02-19T19:01:07.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:07.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:07.071313+0000) 2024-02-19T19:01:08.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:08.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:08.071556+0000) 2024-02-19T19:01:09.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:09.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:09.071761+0000) 2024-02-19T19:01:10.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:10.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:10.072030+0000) 2024-02-19T19:01:11.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:11.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:11.072297+0000) 2024-02-19T19:01:12.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:12.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:12.072586+0000) 2024-02-19T19:01:13.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:13.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:13.072852+0000) 2024-02-19T19:01:14.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:14.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:14.073095+0000) 2024-02-19T19:01:15.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:15.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:15.073373+0000) 2024-02-19T19:01:16.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:16.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:16.073627+0000) 2024-02-19T19:01:17.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:17.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:17.073854+0000) 2024-02-19T19:01:18.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:18.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:18.074095+0000) 2024-02-19T19:01:18.679 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:01:18.683 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:01:19.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:19.068+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:19.074276+0000) 2024-02-19T19:01:20.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:20.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:20.074561+0000) 2024-02-19T19:01:21.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:21.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:21.074800+0000) 2024-02-19T19:01:22.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:22.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:22.075014+0000) 2024-02-19T19:01:23.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:23.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:23.075244+0000) 2024-02-19T19:01:24.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:24.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:24.075502+0000) 2024-02-19T19:01:25.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:25.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:25.075775+0000) 2024-02-19T19:01:26.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:26.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:26.076085+0000) 2024-02-19T19:01:27.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:27.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:27.076321+0000) 2024-02-19T19:01:28.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:28.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:28.076586+0000) 2024-02-19T19:01:29.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:29.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:29.076834+0000) 2024-02-19T19:01:30.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:30.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:30.077021+0000) 2024-02-19T19:01:31.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:31.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:31.077237+0000) 2024-02-19T19:01:32.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:32.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:32.077445+0000) 2024-02-19T19:01:33.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:33.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:33.077687+0000) 2024-02-19T19:01:34.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:34.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:34.077944+0000) 2024-02-19T19:01:35.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:35.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:35.078289+0000) 2024-02-19T19:01:36.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:36.071+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:36.078569+0000) 2024-02-19T19:01:37.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:37.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:37.078849+0000) 2024-02-19T19:01:38.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:38.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:38.079074+0000) 2024-02-19T19:01:39.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:39.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:39.079254+0000) 2024-02-19T19:01:40.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:40.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:40.079511+0000) 2024-02-19T19:01:41.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:41.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:41.079741+0000) 2024-02-19T19:01:42.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:42.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:42.080006+0000) 2024-02-19T19:01:43.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:43.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:43.080302+0000) 2024-02-19T19:01:44.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:44.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:44.080590+0000) 2024-02-19T19:01:45.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:45.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:45.080908+0000) 2024-02-19T19:01:46.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:46.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:46.081196+0000) 2024-02-19T19:01:47.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:47.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:47.081444+0000) 2024-02-19T19:01:48.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:48.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:48.081685+0000) 2024-02-19T19:01:48.801 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:01:48.805 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:01:49.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:49.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:49.081938+0000) 2024-02-19T19:01:50.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:50.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:50.082117+0000) 2024-02-19T19:01:51.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:51.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:51.082376+0000) 2024-02-19T19:01:52.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:52.075+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:52.082565+0000) 2024-02-19T19:01:53.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:53.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:53.082765+0000) 2024-02-19T19:01:54.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:54.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:54.082966+0000) 2024-02-19T19:01:55.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:55.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:55.083159+0000) 2024-02-19T19:01:56.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:56.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:56.083425+0000) 2024-02-19T19:01:57.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:57.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:57.083626+0000) 2024-02-19T19:01:58.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:58.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:58.083871+0000) 2024-02-19T19:01:59.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:01:59.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:01:59.084111+0000) 2024-02-19T19:02:00.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:00.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:00.084311+0000) 2024-02-19T19:02:01.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:01.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:01.084558+0000) 2024-02-19T19:02:02.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:02.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:02.084803+0000) 2024-02-19T19:02:03.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:03.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:03.085037+0000) 2024-02-19T19:02:04.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:04.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:04.085242+0000) 2024-02-19T19:02:05.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:05.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:05.085483+0000) 2024-02-19T19:02:06.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:06.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:06.085724+0000) 2024-02-19T19:02:07.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:07.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:07.085971+0000) 2024-02-19T19:02:08.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:08.079+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:08.086223+0000) 2024-02-19T19:02:09.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:09.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:09.086480+0000) 2024-02-19T19:02:10.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:10.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:10.086671+0000) 2024-02-19T19:02:11.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:11.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:11.086945+0000) 2024-02-19T19:02:12.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:12.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:12.087138+0000) 2024-02-19T19:02:13.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:13.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:13.087337+0000) 2024-02-19T19:02:14.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:14.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:14.087567+0000) 2024-02-19T19:02:15.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:15.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:15.087800+0000) 2024-02-19T19:02:16.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:16.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:16.088063+0000) 2024-02-19T19:02:17.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:17.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:17.088257+0000) 2024-02-19T19:02:18.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:18.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:18.088490+0000) 2024-02-19T19:02:18.932 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:02:18.936 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:02:19.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:19.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:19.088742+0000) 2024-02-19T19:02:20.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:20.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:20.088919+0000) 2024-02-19T19:02:21.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:21.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:21.089143+0000) 2024-02-19T19:02:22.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:22.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:22.089349+0000) 2024-02-19T19:02:23.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:23.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:23.089534+0000) 2024-02-19T19:02:24.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:24.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:24.089776+0000) 2024-02-19T19:02:25.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:25.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:25.090032+0000) 2024-02-19T19:02:26.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:26.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:26.090212+0000) 2024-02-19T19:02:27.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:27.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:27.090394+0000) 2024-02-19T19:02:28.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:28.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:28.090613+0000) 2024-02-19T19:02:29.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:29.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:29.090892+0000) 2024-02-19T19:02:30.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:30.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:30.091131+0000) 2024-02-19T19:02:31.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:31.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:31.091328+0000) 2024-02-19T19:02:32.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:32.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:32.091573+0000) 2024-02-19T19:02:33.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:33.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:33.091824+0000) 2024-02-19T19:02:34.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:34.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:34.091978+0000) 2024-02-19T19:02:35.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:35.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:35.092234+0000) 2024-02-19T19:02:36.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:36.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:36.092407+0000) 2024-02-19T19:02:37.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:37.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:37.092666+0000) 2024-02-19T19:02:38.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:38.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:38.092906+0000) 2024-02-19T19:02:39.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:39.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:39.093082+0000) 2024-02-19T19:02:40.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:40.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:40.093298+0000) 2024-02-19T19:02:41.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:41.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:41.093473+0000) 2024-02-19T19:02:42.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:42.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:42.093705+0000) 2024-02-19T19:02:43.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:43.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:43.093871+0000) 2024-02-19T19:02:44.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:44.086+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:44.094073+0000) 2024-02-19T19:02:45.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:45.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:45.094237+0000) 2024-02-19T19:02:46.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:46.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:46.094520+0000) 2024-02-19T19:02:47.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:47.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:47.094800+0000) 2024-02-19T19:02:48.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:48.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:48.095001+0000) 2024-02-19T19:02:49.080 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:02:49.084 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:02:49.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:49.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:49.095173+0000) 2024-02-19T19:02:50.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:50.090+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:50.095388+0000) 2024-02-19T19:02:51.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:51.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:51.095641+0000) 2024-02-19T19:02:52.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:52.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:52.095814+0000) 2024-02-19T19:02:53.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:53.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:53.096026+0000) 2024-02-19T19:02:54.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:54.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:54.096206+0000) 2024-02-19T19:02:55.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:55.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:55.096480+0000) 2024-02-19T19:02:56.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:56.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:56.096652+0000) 2024-02-19T19:02:57.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:57.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:57.096877+0000) 2024-02-19T19:02:58.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:58.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:58.097153+0000) 2024-02-19T19:02:59.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:02:59.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:02:59.097420+0000) 2024-02-19T19:03:00.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:00.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:00.097596+0000) 2024-02-19T19:03:01.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:01.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:01.097831+0000) 2024-02-19T19:03:02.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:02.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:02.098066+0000) 2024-02-19T19:03:03.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:03.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:03.098273+0000) 2024-02-19T19:03:04.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:04.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:04.098479+0000) 2024-02-19T19:03:05.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:05.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:05.098696+0000) 2024-02-19T19:03:06.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:06.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:06.098964+0000) 2024-02-19T19:03:07.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:07.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:07.099181+0000) 2024-02-19T19:03:08.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:08.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:08.099427+0000) 2024-02-19T19:03:09.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:09.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:09.099637+0000) 2024-02-19T19:03:10.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:10.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:10.099839+0000) 2024-02-19T19:03:11.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:11.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:11.100088+0000) 2024-02-19T19:03:12.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:12.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:12.100265+0000) 2024-02-19T19:03:13.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:13.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:13.100506+0000) 2024-02-19T19:03:14.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:14.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:14.100759+0000) 2024-02-19T19:03:15.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:15.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:15.100997+0000) 2024-02-19T19:03:16.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:16.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:16.101232+0000) 2024-02-19T19:03:17.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:17.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:17.101460+0000) 2024-02-19T19:03:18.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:18.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:18.101654+0000) 2024-02-19T19:03:19.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:19.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:19.101890+0000) 2024-02-19T19:03:19.152 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:03:19.156 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:03:20.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:20.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:20.102150+0000) 2024-02-19T19:03:21.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:21.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:21.102390+0000) 2024-02-19T19:03:22.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:22.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:22.102594+0000) 2024-02-19T19:03:23.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:23.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:23.102838+0000) 2024-02-19T19:03:24.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:24.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:24.102964+0000) 2024-02-19T19:03:25.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:25.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:25.103187+0000) 2024-02-19T19:03:26.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:26.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:26.103357+0000) 2024-02-19T19:03:27.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:27.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:27.103631+0000) 2024-02-19T19:03:28.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:28.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:28.103835+0000) 2024-02-19T19:03:29.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:29.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:29.104048+0000) 2024-02-19T19:03:30.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:30.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:30.104202+0000) 2024-02-19T19:03:31.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:31.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:31.104464+0000) 2024-02-19T19:03:32.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:32.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:32.104660+0000) 2024-02-19T19:03:33.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:33.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:33.104872+0000) 2024-02-19T19:03:34.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:34.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:34.105077+0000) 2024-02-19T19:03:35.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:35.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:35.105332+0000) 2024-02-19T19:03:36.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:36.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:36.105565+0000) 2024-02-19T19:03:37.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:37.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:37.105778+0000) 2024-02-19T19:03:38.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:38.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:38.105974+0000) 2024-02-19T19:03:39.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:39.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:39.106210+0000) 2024-02-19T19:03:40.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:40.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:40.106396+0000) 2024-02-19T19:03:41.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:41.101+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:41.106632+0000) 2024-02-19T19:03:42.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:42.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:42.106832+0000) 2024-02-19T19:03:43.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:43.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:43.106999+0000) 2024-02-19T19:03:44.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:44.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:44.107177+0000) 2024-02-19T19:03:45.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:45.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:45.107433+0000) 2024-02-19T19:03:46.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:46.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:46.107677+0000) 2024-02-19T19:03:47.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:47.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:47.107889+0000) 2024-02-19T19:03:48.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:48.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:48.108085+0000) 2024-02-19T19:03:49.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:49.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:49.108345+0000) 2024-02-19T19:03:49.258 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:03:49.261 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:03:50.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:50.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:50.108539+0000) 2024-02-19T19:03:51.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:51.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:51.108779+0000) 2024-02-19T19:03:52.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:52.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:52.109025+0000) 2024-02-19T19:03:53.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:53.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:53.109281+0000) 2024-02-19T19:03:54.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:54.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:54.109515+0000) 2024-02-19T19:03:55.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:55.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:55.109723+0000) 2024-02-19T19:03:56.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:56.105+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:56.110018+0000) 2024-02-19T19:03:57.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:57.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:57.110280+0000) 2024-02-19T19:03:58.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:58.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:58.110453+0000) 2024-02-19T19:03:59.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:03:59.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:03:59.110615+0000) 2024-02-19T19:04:00.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:00.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:00.110896+0000) 2024-02-19T19:04:01.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:01.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:01.111136+0000) 2024-02-19T19:04:02.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:02.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:02.111339+0000) 2024-02-19T19:04:03.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:03.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:03.111579+0000) 2024-02-19T19:04:04.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:04.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:04.111831+0000) 2024-02-19T19:04:05.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:05.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:05.112006+0000) 2024-02-19T19:04:06.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:06.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:06.112217+0000) 2024-02-19T19:04:07.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:07.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:07.112466+0000) 2024-02-19T19:04:08.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:08.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:08.112687+0000) 2024-02-19T19:04:09.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:09.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:09.112950+0000) 2024-02-19T19:04:10.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:10.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:10.113193+0000) 2024-02-19T19:04:11.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:11.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:11.113435+0000) 2024-02-19T19:04:12.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:12.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:12.113653+0000) 2024-02-19T19:04:13.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:13.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:13.113896+0000) 2024-02-19T19:04:14.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:14.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:14.114093+0000) 2024-02-19T19:04:15.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:15.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:15.114274+0000) 2024-02-19T19:04:16.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:16.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:16.114534+0000) 2024-02-19T19:04:17.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:17.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:17.114829+0000) 2024-02-19T19:04:18.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:18.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:18.115028+0000) 2024-02-19T19:04:19.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:19.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:19.115275+0000) 2024-02-19T19:04:19.397 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:04:19.400 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:04:20.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:20.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:20.115530+0000) 2024-02-19T19:04:21.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:21.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:21.115765+0000) 2024-02-19T19:04:22.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:22.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:22.115940+0000) 2024-02-19T19:04:23.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:23.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:23.116172+0000) 2024-02-19T19:04:24.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:24.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:24.116382+0000) 2024-02-19T19:04:25.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:25.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:25.116623+0000) 2024-02-19T19:04:26.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:26.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:26.116819+0000) 2024-02-19T19:04:27.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:27.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:27.117093+0000) 2024-02-19T19:04:28.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:28.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:28.117329+0000) 2024-02-19T19:04:29.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:29.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:29.117549+0000) 2024-02-19T19:04:30.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:30.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:30.117767+0000) 2024-02-19T19:04:31.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:31.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:31.118055+0000) 2024-02-19T19:04:32.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:32.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:32.118312+0000) 2024-02-19T19:04:33.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:33.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:33.118487+0000) 2024-02-19T19:04:34.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:34.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:34.118675+0000) 2024-02-19T19:04:35.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:35.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:35.118840+0000) 2024-02-19T19:04:36.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:36.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:36.119036+0000) 2024-02-19T19:04:37.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:37.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:37.119191+0000) 2024-02-19T19:04:38.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:38.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:38.119437+0000) 2024-02-19T19:04:39.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:39.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:39.119630+0000) 2024-02-19T19:04:40.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:40.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:40.119898+0000) 2024-02-19T19:04:41.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:41.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:41.120072+0000) 2024-02-19T19:04:42.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:42.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:42.120329+0000) 2024-02-19T19:04:43.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:43.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:43.120565+0000) 2024-02-19T19:04:44.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:44.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:44.120813+0000) 2024-02-19T19:04:45.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:45.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:45.121086+0000) 2024-02-19T19:04:46.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:46.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:46.121290+0000) 2024-02-19T19:04:47.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:47.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:47.121548+0000) 2024-02-19T19:04:48.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:48.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:48.121800+0000) 2024-02-19T19:04:49.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:49.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:49.122041+0000) 2024-02-19T19:04:49.459 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:04:49.464 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:04:50.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:50.116+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:50.122216+0000) 2024-02-19T19:04:51.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:51.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:51.122463+0000) 2024-02-19T19:04:52.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:52.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:52.122739+0000) 2024-02-19T19:04:53.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:53.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:53.122919+0000) 2024-02-19T19:04:54.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:54.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:54.123100+0000) 2024-02-19T19:04:55.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:55.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:55.123309+0000) 2024-02-19T19:04:56.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:56.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:56.123559+0000) 2024-02-19T19:04:57.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:57.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:57.123788+0000) 2024-02-19T19:04:58.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:58.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:58.124085+0000) 2024-02-19T19:04:59.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:04:59.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:04:59.124313+0000) 2024-02-19T19:05:00.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:00.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:00.124583+0000) 2024-02-19T19:05:01.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:01.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:01.124727+0000) 2024-02-19T19:05:02.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:02.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:02.124984+0000) 2024-02-19T19:05:03.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:03.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:03.125235+0000) 2024-02-19T19:05:04.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:04.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:04.125500+0000) 2024-02-19T19:05:05.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:05.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:05.125743+0000) 2024-02-19T19:05:06.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:06.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:06.125990+0000) 2024-02-19T19:05:07.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:07.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:07.126200+0000) 2024-02-19T19:05:08.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:08.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:08.126419+0000) 2024-02-19T19:05:09.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:09.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:09.126628+0000) 2024-02-19T19:05:10.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:10.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:10.126885+0000) 2024-02-19T19:05:11.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:11.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:11.127114+0000) 2024-02-19T19:05:12.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:12.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:12.127393+0000) 2024-02-19T19:05:13.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:13.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:13.127692+0000) 2024-02-19T19:05:14.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:14.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:14.127989+0000) 2024-02-19T19:05:15.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:15.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:15.128258+0000) 2024-02-19T19:05:16.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:16.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:16.128556+0000) 2024-02-19T19:05:17.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:17.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:17.128862+0000) 2024-02-19T19:05:18.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:18.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:18.129110+0000) 2024-02-19T19:05:19.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:19.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:19.129289+0000) 2024-02-19T19:05:19.583 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:05:19.586 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:05:20.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:20.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:20.129497+0000) 2024-02-19T19:05:21.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:21.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:21.129703+0000) 2024-02-19T19:05:22.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:22.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:22.130006+0000) 2024-02-19T19:05:23.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:23.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:23.130264+0000) 2024-02-19T19:05:24.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:24.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:24.130505+0000) 2024-02-19T19:05:25.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:25.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:25.130701+0000) 2024-02-19T19:05:26.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:26.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:26.130864+0000) 2024-02-19T19:05:27.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:27.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:27.131053+0000) 2024-02-19T19:05:28.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:28.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:28.131297+0000) 2024-02-19T19:05:29.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:29.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:29.131477+0000) 2024-02-19T19:05:30.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:30.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:30.131683+0000) 2024-02-19T19:05:31.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:31.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:31.131905+0000) 2024-02-19T19:05:32.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:32.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:32.132160+0000) 2024-02-19T19:05:33.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:33.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:33.132413+0000) 2024-02-19T19:05:34.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:34.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:34.132680+0000) 2024-02-19T19:05:35.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:35.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:35.132885+0000) 2024-02-19T19:05:36.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:36.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:36.133149+0000) 2024-02-19T19:05:37.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:37.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:37.133410+0000) 2024-02-19T19:05:38.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:38.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:38.133647+0000) 2024-02-19T19:05:39.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:39.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:39.133915+0000) 2024-02-19T19:05:40.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:40.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:40.134161+0000) 2024-02-19T19:05:41.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:41.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:41.134403+0000) 2024-02-19T19:05:42.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:42.127+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:42.134648+0000) 2024-02-19T19:05:43.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:43.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:43.134886+0000) 2024-02-19T19:05:44.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:44.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:44.135073+0000) 2024-02-19T19:05:45.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:45.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:45.135310+0000) 2024-02-19T19:05:46.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:46.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:46.135555+0000) 2024-02-19T19:05:47.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:47.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:47.135779+0000) 2024-02-19T19:05:48.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:48.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:48.136014+0000) 2024-02-19T19:05:49.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:49.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:49.136191+0000) 2024-02-19T19:05:49.714 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:05:49.718 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:05:50.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:50.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:50.136394+0000) 2024-02-19T19:05:51.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:51.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:51.136639+0000) 2024-02-19T19:05:52.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:52.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:52.136903+0000) 2024-02-19T19:05:53.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:53.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:53.137120+0000) 2024-02-19T19:05:54.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:54.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:54.137368+0000) 2024-02-19T19:05:55.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:55.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:55.137613+0000) 2024-02-19T19:05:56.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:56.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:56.137852+0000) 2024-02-19T19:05:57.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:57.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:57.138083+0000) 2024-02-19T19:05:58.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:58.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:58.138289+0000) 2024-02-19T19:05:59.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:05:59.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:05:59.138428+0000) 2024-02-19T19:06:00.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:00.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:00.138690+0000) 2024-02-19T19:06:01.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:01.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:01.138967+0000) 2024-02-19T19:06:02.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:02.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:02.139218+0000) 2024-02-19T19:06:03.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:03.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:03.139475+0000) 2024-02-19T19:06:04.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:04.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:04.139715+0000) 2024-02-19T19:06:05.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:05.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:05.139959+0000) 2024-02-19T19:06:06.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:06.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:06.140182+0000) 2024-02-19T19:06:07.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:07.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:07.140456+0000) 2024-02-19T19:06:08.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:08.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:08.140679+0000) 2024-02-19T19:06:09.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:09.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:09.140885+0000) 2024-02-19T19:06:10.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:10.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:10.141165+0000) 2024-02-19T19:06:11.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:11.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:11.141440+0000) 2024-02-19T19:06:12.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:12.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:12.141580+0000) 2024-02-19T19:06:13.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:13.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:13.141830+0000) 2024-02-19T19:06:14.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:14.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:14.142098+0000) 2024-02-19T19:06:15.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:15.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:15.142411+0000) 2024-02-19T19:06:16.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:16.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:16.142646+0000) 2024-02-19T19:06:17.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:17.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:17.142791+0000) 2024-02-19T19:06:18.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:18.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:18.143038+0000) 2024-02-19T19:06:19.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:19.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:19.143278+0000) 2024-02-19T19:06:19.861 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:06:19.865 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:06:20.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:20.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:20.143422+0000) 2024-02-19T19:06:21.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:21.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:21.143706+0000) 2024-02-19T19:06:22.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:22.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:22.143995+0000) 2024-02-19T19:06:23.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:23.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:23.144262+0000) 2024-02-19T19:06:24.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:24.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:24.144552+0000) 2024-02-19T19:06:25.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:25.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:25.144838+0000) 2024-02-19T19:06:26.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:26.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:26.145134+0000) 2024-02-19T19:06:27.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:27.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:27.145464+0000) 2024-02-19T19:06:28.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:28.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:28.145744+0000) 2024-02-19T19:06:29.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:29.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:29.145922+0000) 2024-02-19T19:06:30.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:30.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:30.146159+0000) 2024-02-19T19:06:31.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:31.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:31.146359+0000) 2024-02-19T19:06:32.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:32.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:32.146626+0000) 2024-02-19T19:06:33.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:33.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:33.146932+0000) 2024-02-19T19:06:34.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:34.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:34.147075+0000) 2024-02-19T19:06:35.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:35.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:35.147360+0000) 2024-02-19T19:06:36.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:36.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:36.147600+0000) 2024-02-19T19:06:37.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:37.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:37.147910+0000) 2024-02-19T19:06:38.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:38.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:38.148165+0000) 2024-02-19T19:06:39.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:39.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:39.148382+0000) 2024-02-19T19:06:40.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:40.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:40.148667+0000) 2024-02-19T19:06:41.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:41.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:41.148945+0000) 2024-02-19T19:06:42.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:42.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:42.149233+0000) 2024-02-19T19:06:43.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:43.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:43.149527+0000) 2024-02-19T19:06:44.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:44.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:44.149826+0000) 2024-02-19T19:06:45.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:45.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:45.150059+0000) 2024-02-19T19:06:46.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:46.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:46.150349+0000) 2024-02-19T19:06:47.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:47.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:47.150658+0000) 2024-02-19T19:06:48.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:48.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:48.150929+0000) 2024-02-19T19:06:49.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:49.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:49.151119+0000) 2024-02-19T19:06:49.992 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:06:49.995 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:06:50.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:50.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:50.151272+0000) 2024-02-19T19:06:51.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:51.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:51.151515+0000) 2024-02-19T19:06:52.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:52.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:52.151743+0000) 2024-02-19T19:06:53.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:53.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:53.152007+0000) 2024-02-19T19:06:54.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:54.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:54.152307+0000) 2024-02-19T19:06:55.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:55.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:55.152595+0000) 2024-02-19T19:06:56.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:56.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:56.152834+0000) 2024-02-19T19:06:57.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:57.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:57.153109+0000) 2024-02-19T19:06:58.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:58.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:58.153358+0000) 2024-02-19T19:06:59.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:06:59.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:06:59.153610+0000) 2024-02-19T19:07:00.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:00.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:00.153871+0000) 2024-02-19T19:07:01.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:01.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:01.154047+0000) 2024-02-19T19:07:02.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:02.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:02.154296+0000) 2024-02-19T19:07:03.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:03.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:03.154552+0000) 2024-02-19T19:07:04.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:04.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:04.154818+0000) 2024-02-19T19:07:05.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:05.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:05.154971+0000) 2024-02-19T19:07:06.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:06.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:06.155172+0000) 2024-02-19T19:07:07.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:07.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:07.155443+0000) 2024-02-19T19:07:08.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:08.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:08.155692+0000) 2024-02-19T19:07:09.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:09.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:09.155945+0000) 2024-02-19T19:07:10.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:10.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:10.156170+0000) 2024-02-19T19:07:11.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:11.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:11.156418+0000) 2024-02-19T19:07:12.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:12.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:12.156665+0000) 2024-02-19T19:07:13.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:13.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:13.156904+0000) 2024-02-19T19:07:14.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:14.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:14.157155+0000) 2024-02-19T19:07:15.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:15.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:15.157429+0000) 2024-02-19T19:07:16.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:16.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:16.157674+0000) 2024-02-19T19:07:17.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:17.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:17.157945+0000) 2024-02-19T19:07:18.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:18.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:18.158200+0000) 2024-02-19T19:07:19.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:19.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:19.158372+0000) 2024-02-19T19:07:20.114 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:07:20.117 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:07:20.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:20.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:20.158560+0000) 2024-02-19T19:07:21.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:21.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:21.158874+0000) 2024-02-19T19:07:22.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:22.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:22.159065+0000) 2024-02-19T19:07:23.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:23.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:23.159267+0000) 2024-02-19T19:07:24.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:24.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:24.159516+0000) 2024-02-19T19:07:25.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:25.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:25.159751+0000) 2024-02-19T19:07:26.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:26.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:26.160009+0000) 2024-02-19T19:07:27.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:27.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:27.160265+0000) 2024-02-19T19:07:28.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:28.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:28.160440+0000) 2024-02-19T19:07:29.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:29.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:29.160671+0000) 2024-02-19T19:07:30.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:30.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:30.160950+0000) 2024-02-19T19:07:31.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:31.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:31.161195+0000) 2024-02-19T19:07:32.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:32.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:32.161436+0000) 2024-02-19T19:07:33.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:33.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:33.161723+0000) 2024-02-19T19:07:34.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:34.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:34.161988+0000) 2024-02-19T19:07:35.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:35.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:35.162249+0000) 2024-02-19T19:07:36.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:36.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:36.162478+0000) 2024-02-19T19:07:37.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:37.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:37.162664+0000) 2024-02-19T19:07:38.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:38.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:38.162909+0000) 2024-02-19T19:07:39.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:39.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:39.163157+0000) 2024-02-19T19:07:40.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:40.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:40.163300+0000) 2024-02-19T19:07:41.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:41.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:41.163502+0000) 2024-02-19T19:07:42.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:42.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:42.163742+0000) 2024-02-19T19:07:43.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:43.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:43.163985+0000) 2024-02-19T19:07:44.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:44.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:44.164170+0000) 2024-02-19T19:07:45.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:45.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:45.164370+0000) 2024-02-19T19:07:46.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:46.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:46.164615+0000) 2024-02-19T19:07:47.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:47.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:47.164873+0000) 2024-02-19T19:07:48.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:48.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:48.165124+0000) 2024-02-19T19:07:49.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:49.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:49.165348+0000) 2024-02-19T19:07:50.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:50.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:50.165564+0000) 2024-02-19T19:07:50.254 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:07:50.257 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:07:51.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:51.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:51.165813+0000) 2024-02-19T19:07:52.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:52.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:52.166018+0000) 2024-02-19T19:07:53.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:53.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:53.166215+0000) 2024-02-19T19:07:54.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:54.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:54.166464+0000) 2024-02-19T19:07:55.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:55.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:55.166692+0000) 2024-02-19T19:07:56.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:56.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:56.166890+0000) 2024-02-19T19:07:57.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:57.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:57.167081+0000) 2024-02-19T19:07:58.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:58.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:58.167324+0000) 2024-02-19T19:07:59.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:07:59.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:07:59.167609+0000) 2024-02-19T19:08:00.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:00.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:00.167855+0000) 2024-02-19T19:08:01.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:01.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:01.168094+0000) 2024-02-19T19:08:02.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:02.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:02.168340+0000) 2024-02-19T19:08:03.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:03.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:03.168546+0000) 2024-02-19T19:08:04.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:04.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:04.168729+0000) 2024-02-19T19:08:05.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:05.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:05.169014+0000) 2024-02-19T19:08:06.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:06.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:06.169221+0000) 2024-02-19T19:08:07.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:07.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:07.169535+0000) 2024-02-19T19:08:08.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:08.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:08.169743+0000) 2024-02-19T19:08:09.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:09.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:09.169950+0000) 2024-02-19T19:08:10.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:10.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:10.170179+0000) 2024-02-19T19:08:11.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:11.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:11.170379+0000) 2024-02-19T19:08:12.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:12.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:12.170578+0000) 2024-02-19T19:08:13.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:13.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:13.170801+0000) 2024-02-19T19:08:14.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:14.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:14.171060+0000) 2024-02-19T19:08:15.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:15.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:15.171203+0000) 2024-02-19T19:08:16.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:16.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:16.171450+0000) 2024-02-19T19:08:17.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:17.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:17.171635+0000) 2024-02-19T19:08:18.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:18.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:18.171888+0000) 2024-02-19T19:08:19.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:19.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:19.172129+0000) 2024-02-19T19:08:20.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:20.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:20.172374+0000) 2024-02-19T19:08:20.600 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:08:20.604 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:08:21.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:21.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:21.172553+0000) 2024-02-19T19:08:22.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:22.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:22.172803+0000) 2024-02-19T19:08:23.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:23.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:23.173039+0000) 2024-02-19T19:08:24.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:24.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:24.173303+0000) 2024-02-19T19:08:25.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:25.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:25.173432+0000) 2024-02-19T19:08:26.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:26.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:26.173679+0000) 2024-02-19T19:08:27.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:27.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:27.173887+0000) 2024-02-19T19:08:28.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:28.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:28.174164+0000) 2024-02-19T19:08:29.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:29.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:29.174455+0000) 2024-02-19T19:08:30.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:30.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:30.174697+0000) 2024-02-19T19:08:31.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:31.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:31.174983+0000) 2024-02-19T19:08:32.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:32.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:32.175175+0000) 2024-02-19T19:08:33.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:33.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:33.175345+0000) 2024-02-19T19:08:34.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:34.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:34.175629+0000) 2024-02-19T19:08:35.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:35.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:35.175864+0000) 2024-02-19T19:08:36.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:36.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:36.176089+0000) 2024-02-19T19:08:37.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:37.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:37.176370+0000) 2024-02-19T19:08:38.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:38.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:38.176609+0000) 2024-02-19T19:08:39.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:39.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:39.176816+0000) 2024-02-19T19:08:40.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:40.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:40.177084+0000) 2024-02-19T19:08:41.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:41.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:41.177290+0000) 2024-02-19T19:08:42.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:42.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:42.177539+0000) 2024-02-19T19:08:43.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:43.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:43.177793+0000) 2024-02-19T19:08:44.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:44.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:44.178030+0000) 2024-02-19T19:08:45.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:45.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:45.178282+0000) 2024-02-19T19:08:46.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:46.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:46.178525+0000) 2024-02-19T19:08:47.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:47.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:47.178782+0000) 2024-02-19T19:08:48.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:48.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:48.178931+0000) 2024-02-19T19:08:49.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:49.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:49.179061+0000) 2024-02-19T19:08:50.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:50.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:50.179257+0000) 2024-02-19T19:08:50.664 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:08:50.667 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:08:51.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:51.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:51.179517+0000) 2024-02-19T19:08:52.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:52.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:52.179758+0000) 2024-02-19T19:08:53.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:53.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:53.179994+0000) 2024-02-19T19:08:54.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:54.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:54.180213+0000) 2024-02-19T19:08:55.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:55.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:55.180407+0000) 2024-02-19T19:08:56.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:56.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:56.180665+0000) 2024-02-19T19:08:57.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:57.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:57.180933+0000) 2024-02-19T19:08:58.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:58.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:58.181186+0000) 2024-02-19T19:08:59.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:08:59.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:08:59.181417+0000) 2024-02-19T19:09:00.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:00.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:00.181632+0000) 2024-02-19T19:09:01.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:01.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:01.181871+0000) 2024-02-19T19:09:02.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:02.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:02.182152+0000) 2024-02-19T19:09:03.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:03.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:03.182390+0000) 2024-02-19T19:09:04.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:04.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:04.182598+0000) 2024-02-19T19:09:05.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:05.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:05.182787+0000) 2024-02-19T19:09:06.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:06.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:06.182950+0000) 2024-02-19T19:09:07.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:07.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:07.183083+0000) 2024-02-19T19:09:08.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:08.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:08.183227+0000) 2024-02-19T19:09:09.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:09.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:09.183410+0000) 2024-02-19T19:09:10.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:10.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:10.183620+0000) 2024-02-19T19:09:11.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:11.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:11.183827+0000) 2024-02-19T19:09:12.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:12.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:12.184077+0000) 2024-02-19T19:09:13.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:13.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:13.184334+0000) 2024-02-19T19:09:14.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:14.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:14.184476+0000) 2024-02-19T19:09:15.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:15.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:15.184740+0000) 2024-02-19T19:09:16.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:16.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:16.184985+0000) 2024-02-19T19:09:17.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:17.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:17.185230+0000) 2024-02-19T19:09:18.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:18.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:18.185487+0000) 2024-02-19T19:09:19.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:19.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:19.185820+0000) 2024-02-19T19:09:20.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:20.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:20.186099+0000) 2024-02-19T19:09:20.778 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:09:20.782 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:09:21.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:21.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:21.186349+0000) 2024-02-19T19:09:22.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:22.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:22.186588+0000) 2024-02-19T19:09:23.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:23.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:23.186852+0000) 2024-02-19T19:09:24.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:24.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:24.187099+0000) 2024-02-19T19:09:25.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:25.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:25.187341+0000) 2024-02-19T19:09:26.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:26.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:26.187587+0000) 2024-02-19T19:09:27.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:27.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:27.187849+0000) 2024-02-19T19:09:28.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:28.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:28.188127+0000) 2024-02-19T19:09:29.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:29.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:29.188338+0000) 2024-02-19T19:09:30.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:30.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:30.188605+0000) 2024-02-19T19:09:31.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:31.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:31.188830+0000) 2024-02-19T19:09:32.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:32.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:32.189062+0000) 2024-02-19T19:09:33.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:33.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:33.189306+0000) 2024-02-19T19:09:34.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:34.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:34.189547+0000) 2024-02-19T19:09:35.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:35.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:35.189795+0000) 2024-02-19T19:09:36.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:36.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:36.189944+0000) 2024-02-19T19:09:37.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:37.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:37.190206+0000) 2024-02-19T19:09:38.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:38.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:38.190385+0000) 2024-02-19T19:09:39.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:39.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:39.190662+0000) 2024-02-19T19:09:40.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:40.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:40.190844+0000) 2024-02-19T19:09:41.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:41.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:41.190993+0000) 2024-02-19T19:09:42.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:42.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:42.191157+0000) 2024-02-19T19:09:43.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:43.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:43.191401+0000) 2024-02-19T19:09:44.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:44.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:44.191646+0000) 2024-02-19T19:09:45.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:45.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:45.191926+0000) 2024-02-19T19:09:46.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:46.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:46.192132+0000) 2024-02-19T19:09:47.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:47.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:47.192279+0000) 2024-02-19T19:09:48.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:48.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:48.192453+0000) 2024-02-19T19:09:49.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:49.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:49.192699+0000) 2024-02-19T19:09:50.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:50.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:50.192896+0000) 2024-02-19T19:09:50.885 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:09:50.888 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:09:51.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:51.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:51.193115+0000) 2024-02-19T19:09:52.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:52.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:52.193362+0000) 2024-02-19T19:09:53.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:53.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:53.193568+0000) 2024-02-19T19:09:54.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:54.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:54.193771+0000) 2024-02-19T19:09:55.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:55.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:55.194008+0000) 2024-02-19T19:09:56.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:56.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:56.194178+0000) 2024-02-19T19:09:57.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:57.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:57.194432+0000) 2024-02-19T19:09:58.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:58.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:58.194664+0000) 2024-02-19T19:09:59.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:09:59.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:09:59.194911+0000) 2024-02-19T19:10:00.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:00.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:00.195089+0000) 2024-02-19T19:10:01.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:01.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:01.195239+0000) 2024-02-19T19:10:02.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:02.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:02.195480+0000) 2024-02-19T19:10:03.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:03.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:03.195731+0000) 2024-02-19T19:10:04.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:04.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:04.195959+0000) 2024-02-19T19:10:05.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:05.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:05.196203+0000) 2024-02-19T19:10:06.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:06.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:06.196470+0000) 2024-02-19T19:10:07.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:07.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:07.196758+0000) 2024-02-19T19:10:08.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:08.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:08.197004+0000) 2024-02-19T19:10:09.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:09.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:09.197237+0000) 2024-02-19T19:10:10.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:10.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:10.197486+0000) 2024-02-19T19:10:11.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:11.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:11.197686+0000) 2024-02-19T19:10:12.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:12.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:12.197905+0000) 2024-02-19T19:10:13.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:13.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:13.198173+0000) 2024-02-19T19:10:14.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:14.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:14.198449+0000) 2024-02-19T19:10:15.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:15.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:15.198695+0000) 2024-02-19T19:10:16.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:16.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:16.198931+0000) 2024-02-19T19:10:17.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:17.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:17.199186+0000) 2024-02-19T19:10:18.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:18.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:18.199367+0000) 2024-02-19T19:10:19.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:19.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:19.199617+0000) 2024-02-19T19:10:20.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:20.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:20.199816+0000) 2024-02-19T19:10:21.006 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:10:21.009 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:10:21.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:21.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:21.200043+0000) 2024-02-19T19:10:22.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:22.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:22.200215+0000) 2024-02-19T19:10:23.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:23.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:23.200458+0000) 2024-02-19T19:10:24.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:24.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:24.200695+0000) 2024-02-19T19:10:25.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:25.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:25.200876+0000) 2024-02-19T19:10:26.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:26.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:26.201120+0000) 2024-02-19T19:10:27.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:27.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:27.201365+0000) 2024-02-19T19:10:28.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:28.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:28.201552+0000) 2024-02-19T19:10:29.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:29.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:29.201797+0000) 2024-02-19T19:10:30.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:30.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:30.201973+0000) 2024-02-19T19:10:31.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:31.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:31.202223+0000) 2024-02-19T19:10:32.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:32.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:32.202464+0000) 2024-02-19T19:10:33.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:33.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:33.202700+0000) 2024-02-19T19:10:34.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:34.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:34.203003+0000) 2024-02-19T19:10:35.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:35.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:35.203212+0000) 2024-02-19T19:10:36.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:36.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:36.203451+0000) 2024-02-19T19:10:37.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:37.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:37.203710+0000) 2024-02-19T19:10:38.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:38.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:38.203953+0000) 2024-02-19T19:10:39.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:39.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:39.204201+0000) 2024-02-19T19:10:40.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:40.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:40.204436+0000) 2024-02-19T19:10:41.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:41.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:41.204672+0000) 2024-02-19T19:10:42.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:42.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:42.204916+0000) 2024-02-19T19:10:43.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:43.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:43.205114+0000) 2024-02-19T19:10:44.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:44.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:44.205362+0000) 2024-02-19T19:10:45.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:45.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:45.205565+0000) 2024-02-19T19:10:46.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:46.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:46.205834+0000) 2024-02-19T19:10:47.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:47.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:47.206090+0000) 2024-02-19T19:10:48.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:48.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:48.206335+0000) 2024-02-19T19:10:49.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:49.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:49.206513+0000) 2024-02-19T19:10:50.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:50.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:50.206815+0000) 2024-02-19T19:10:51.121 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:10:51.125 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:10:51.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:51.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:51.206984+0000) 2024-02-19T19:10:52.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:52.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:52.207163+0000) 2024-02-19T19:10:53.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:53.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:53.207356+0000) 2024-02-19T19:10:54.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:54.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:54.207605+0000) 2024-02-19T19:10:55.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:55.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:55.207849+0000) 2024-02-19T19:10:56.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:56.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:56.208067+0000) 2024-02-19T19:10:57.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:57.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:57.208343+0000) 2024-02-19T19:10:58.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:58.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:58.208578+0000) 2024-02-19T19:10:59.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:10:59.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:10:59.208784+0000) 2024-02-19T19:11:00.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:00.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:00.209046+0000) 2024-02-19T19:11:01.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:01.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:01.209277+0000) 2024-02-19T19:11:02.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:02.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:02.209551+0000) 2024-02-19T19:11:03.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:03.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:03.209832+0000) 2024-02-19T19:11:04.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:04.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:04.210117+0000) 2024-02-19T19:11:05.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:05.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:05.210320+0000) 2024-02-19T19:11:06.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:06.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:06.210664+0000) 2024-02-19T19:11:07.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:07.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:07.210957+0000) 2024-02-19T19:11:08.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:08.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:08.211249+0000) 2024-02-19T19:11:09.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:09.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:09.211549+0000) 2024-02-19T19:11:10.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:10.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:10.211782+0000) 2024-02-19T19:11:11.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:11.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:11.211961+0000) 2024-02-19T19:11:12.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:12.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:12.212225+0000) 2024-02-19T19:11:13.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:13.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:13.212430+0000) 2024-02-19T19:11:14.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:14.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:14.212703+0000) 2024-02-19T19:11:15.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:15.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:15.212990+0000) 2024-02-19T19:11:16.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:16.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:16.213283+0000) 2024-02-19T19:11:17.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:17.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:17.213565+0000) 2024-02-19T19:11:18.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:18.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:18.213826+0000) 2024-02-19T19:11:19.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:19.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:19.214117+0000) 2024-02-19T19:11:20.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:20.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:20.214387+0000) 2024-02-19T19:11:21.210 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:11:21.213 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:11:21.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:21.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:21.214560+0000) 2024-02-19T19:11:22.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:22.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:22.214821+0000) 2024-02-19T19:11:23.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:23.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:23.215101+0000) 2024-02-19T19:11:24.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:24.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:24.215337+0000) 2024-02-19T19:11:25.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:25.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:25.215578+0000) 2024-02-19T19:11:26.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:26.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:26.215803+0000) 2024-02-19T19:11:27.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:27.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:27.216064+0000) 2024-02-19T19:11:28.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:28.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:28.216327+0000) 2024-02-19T19:11:29.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:29.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:29.216596+0000) 2024-02-19T19:11:30.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:30.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:30.216844+0000) 2024-02-19T19:11:31.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:31.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:31.217091+0000) 2024-02-19T19:11:32.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:32.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:32.217321+0000) 2024-02-19T19:11:33.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:33.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:33.217583+0000) 2024-02-19T19:11:34.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:34.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:34.217818+0000) 2024-02-19T19:11:35.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:35.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:35.218047+0000) 2024-02-19T19:11:36.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:36.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:36.218300+0000) 2024-02-19T19:11:37.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:37.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:37.218550+0000) 2024-02-19T19:11:38.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:38.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:38.218855+0000) 2024-02-19T19:11:39.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:39.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:39.219050+0000) 2024-02-19T19:11:40.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:40.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:40.219271+0000) 2024-02-19T19:11:41.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:41.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:41.219524+0000) 2024-02-19T19:11:42.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:42.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:42.219775+0000) 2024-02-19T19:11:43.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:43.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:43.220028+0000) 2024-02-19T19:11:44.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:44.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:44.220259+0000) 2024-02-19T19:11:45.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:45.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:45.220523+0000) 2024-02-19T19:11:46.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:46.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:46.220771+0000) 2024-02-19T19:11:47.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:47.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:47.221013+0000) 2024-02-19T19:11:48.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:48.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:48.221289+0000) 2024-02-19T19:11:49.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:49.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:49.221436+0000) 2024-02-19T19:11:50.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:50.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:50.221664+0000) 2024-02-19T19:11:51.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:51.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:51.221884+0000) 2024-02-19T19:11:51.332 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:11:51.337 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:11:52.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:52.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:52.222094+0000) 2024-02-19T19:11:53.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:53.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:53.222334+0000) 2024-02-19T19:11:54.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:54.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:54.222585+0000) 2024-02-19T19:11:55.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:55.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:55.222824+0000) 2024-02-19T19:11:56.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:56.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:56.223068+0000) 2024-02-19T19:11:57.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:57.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:57.223316+0000) 2024-02-19T19:11:58.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:58.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:58.223525+0000) 2024-02-19T19:11:59.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:11:59.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:11:59.223774+0000) 2024-02-19T19:12:00.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:00.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:00.224023+0000) 2024-02-19T19:12:01.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:01.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:01.224256+0000) 2024-02-19T19:12:02.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:02.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:02.224482+0000) 2024-02-19T19:12:03.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:03.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:03.224730+0000) 2024-02-19T19:12:04.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:04.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:04.224925+0000) 2024-02-19T19:12:05.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:05.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:05.225175+0000) 2024-02-19T19:12:06.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:06.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:06.225391+0000) 2024-02-19T19:12:07.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:07.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:07.225670+0000) 2024-02-19T19:12:08.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:08.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:08.225870+0000) 2024-02-19T19:12:09.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:09.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:09.226125+0000) 2024-02-19T19:12:10.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:10.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:10.226355+0000) 2024-02-19T19:12:11.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:11.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:11.226615+0000) 2024-02-19T19:12:12.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:12.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:12.226861+0000) 2024-02-19T19:12:13.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:13.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:13.227141+0000) 2024-02-19T19:12:14.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:14.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:14.227393+0000) 2024-02-19T19:12:15.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:15.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:15.227672+0000) 2024-02-19T19:12:16.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:16.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:16.227899+0000) 2024-02-19T19:12:17.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:17.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:17.228158+0000) 2024-02-19T19:12:18.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:18.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:18.228402+0000) 2024-02-19T19:12:19.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:19.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:19.228576+0000) 2024-02-19T19:12:20.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:20.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:20.228805+0000) 2024-02-19T19:12:21.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:21.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:21.229057+0000) 2024-02-19T19:12:21.455 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:12:21.458 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:12:22.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:22.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:22.229290+0000) 2024-02-19T19:12:23.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:23.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:23.229509+0000) 2024-02-19T19:12:24.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:24.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:24.229713+0000) 2024-02-19T19:12:25.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:25.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:25.229979+0000) 2024-02-19T19:12:26.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:26.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:26.230173+0000) 2024-02-19T19:12:27.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:27.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:27.230396+0000) 2024-02-19T19:12:28.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:28.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:28.230588+0000) 2024-02-19T19:12:29.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:29.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:29.230814+0000) 2024-02-19T19:12:30.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:30.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:30.231088+0000) 2024-02-19T19:12:31.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:31.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:31.231334+0000) 2024-02-19T19:12:32.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:32.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:32.231575+0000) 2024-02-19T19:12:33.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:33.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:33.231812+0000) 2024-02-19T19:12:34.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:34.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:34.232037+0000) 2024-02-19T19:12:35.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:35.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:35.232313+0000) 2024-02-19T19:12:36.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:36.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:36.232491+0000) 2024-02-19T19:12:37.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:37.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:37.232741+0000) 2024-02-19T19:12:38.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:38.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:38.232923+0000) 2024-02-19T19:12:39.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:39.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:39.233178+0000) 2024-02-19T19:12:40.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:40.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:40.233385+0000) 2024-02-19T19:12:41.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:41.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:41.233569+0000) 2024-02-19T19:12:42.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:42.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:42.233797+0000) 2024-02-19T19:12:43.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:43.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:43.234034+0000) 2024-02-19T19:12:44.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:44.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:44.234268+0000) 2024-02-19T19:12:45.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:45.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:45.234468+0000) 2024-02-19T19:12:46.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:46.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:46.234699+0000) 2024-02-19T19:12:47.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:47.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:47.235083+0000) 2024-02-19T19:12:48.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:48.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:48.235269+0000) 2024-02-19T19:12:49.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:49.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:49.235448+0000) 2024-02-19T19:12:50.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:50.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:50.235665+0000) 2024-02-19T19:12:51.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:51.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:51.235890+0000) 2024-02-19T19:12:51.586 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:12:51.589 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:12:52.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:52.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:52.236125+0000) 2024-02-19T19:12:53.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:53.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:53.236362+0000) 2024-02-19T19:12:54.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:54.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:54.236574+0000) 2024-02-19T19:12:55.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:55.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:55.236796+0000) 2024-02-19T19:12:56.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:56.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:56.237059+0000) 2024-02-19T19:12:57.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:57.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:57.237304+0000) 2024-02-19T19:12:58.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:58.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:58.237560+0000) 2024-02-19T19:12:59.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:12:59.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:12:59.237813+0000) 2024-02-19T19:13:00.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:00.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:00.238094+0000) 2024-02-19T19:13:01.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:01.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:01.238350+0000) 2024-02-19T19:13:02.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:02.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:02.238572+0000) 2024-02-19T19:13:03.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:03.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:03.238818+0000) 2024-02-19T19:13:04.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:04.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:04.239064+0000) 2024-02-19T19:13:05.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:05.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:05.239315+0000) 2024-02-19T19:13:06.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:06.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:06.239529+0000) 2024-02-19T19:13:07.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:07.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:07.239760+0000) 2024-02-19T19:13:08.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:08.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:08.239987+0000) 2024-02-19T19:13:09.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:09.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:09.240224+0000) 2024-02-19T19:13:10.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:10.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:10.240430+0000) 2024-02-19T19:13:11.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:11.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:11.240693+0000) 2024-02-19T19:13:12.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:12.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:12.240942+0000) 2024-02-19T19:13:13.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:13.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:13.241184+0000) 2024-02-19T19:13:14.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:14.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:14.241376+0000) 2024-02-19T19:13:15.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:15.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:15.241605+0000) 2024-02-19T19:13:16.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:16.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:16.241850+0000) 2024-02-19T19:13:17.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:17.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:17.242098+0000) 2024-02-19T19:13:18.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:18.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:18.242343+0000) 2024-02-19T19:13:19.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:19.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:19.242526+0000) 2024-02-19T19:13:20.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:20.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:20.242736+0000) 2024-02-19T19:13:21.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:21.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:21.242866+0000) 2024-02-19T19:13:21.917 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:13:21.920 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:13:22.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:22.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:22.243072+0000) 2024-02-19T19:13:23.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:23.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:23.243268+0000) 2024-02-19T19:13:24.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:24.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:24.243491+0000) 2024-02-19T19:13:25.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:25.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:25.243747+0000) 2024-02-19T19:13:26.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:26.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:26.243989+0000) 2024-02-19T19:13:27.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:27.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:27.244254+0000) 2024-02-19T19:13:28.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:28.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:28.244514+0000) 2024-02-19T19:13:29.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:29.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:29.244774+0000) 2024-02-19T19:13:30.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:30.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:30.245039+0000) 2024-02-19T19:13:31.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:31.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:31.245272+0000) 2024-02-19T19:13:32.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:32.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:32.245529+0000) 2024-02-19T19:13:33.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:33.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:33.245789+0000) 2024-02-19T19:13:34.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:34.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:34.246034+0000) 2024-02-19T19:13:35.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:35.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:35.246334+0000) 2024-02-19T19:13:36.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:36.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:36.246542+0000) 2024-02-19T19:13:37.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:37.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:37.246772+0000) 2024-02-19T19:13:38.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:38.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:38.246937+0000) 2024-02-19T19:13:39.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:39.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:39.247087+0000) 2024-02-19T19:13:40.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:40.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:40.247330+0000) 2024-02-19T19:13:41.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:41.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:41.247562+0000) 2024-02-19T19:13:42.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:42.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:42.247748+0000) 2024-02-19T19:13:43.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:43.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:43.247977+0000) 2024-02-19T19:13:44.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:44.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:44.248199+0000) 2024-02-19T19:13:45.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:45.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:45.248446+0000) 2024-02-19T19:13:46.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:46.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:46.248698+0000) 2024-02-19T19:13:47.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:47.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:47.248940+0000) 2024-02-19T19:13:48.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:48.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:48.249185+0000) 2024-02-19T19:13:49.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:49.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:49.249413+0000) 2024-02-19T19:13:50.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:50.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:50.249666+0000) 2024-02-19T19:13:51.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:51.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:51.249907+0000) 2024-02-19T19:13:51.972 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:13:51.976 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:13:52.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:52.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:52.250093+0000) 2024-02-19T19:13:53.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:53.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:53.250325+0000) 2024-02-19T19:13:54.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:54.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:54.250522+0000) 2024-02-19T19:13:55.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:55.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:55.250787+0000) 2024-02-19T19:13:56.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:56.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:56.250989+0000) 2024-02-19T19:13:57.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:57.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:57.251195+0000) 2024-02-19T19:13:58.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:58.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:58.251429+0000) 2024-02-19T19:13:59.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:13:59.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:13:59.251688+0000) 2024-02-19T19:14:00.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:00.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:00.251954+0000) 2024-02-19T19:14:01.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:01.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:01.252176+0000) 2024-02-19T19:14:02.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:02.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:02.252361+0000) 2024-02-19T19:14:03.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:03.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:03.252583+0000) 2024-02-19T19:14:04.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:04.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:04.252825+0000) 2024-02-19T19:14:05.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:05.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:05.253066+0000) 2024-02-19T19:14:06.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:06.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:06.253242+0000) 2024-02-19T19:14:07.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:07.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:07.253390+0000) 2024-02-19T19:14:08.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:08.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:08.253556+0000) 2024-02-19T19:14:09.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:09.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:09.253810+0000) 2024-02-19T19:14:10.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:10.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:10.254011+0000) 2024-02-19T19:14:11.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:11.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:11.254191+0000) 2024-02-19T19:14:12.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:12.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:12.254424+0000) 2024-02-19T19:14:13.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:13.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:13.254613+0000) 2024-02-19T19:14:14.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:14.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:14.254862+0000) 2024-02-19T19:14:15.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:15.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:15.255024+0000) 2024-02-19T19:14:16.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:16.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:16.255202+0000) 2024-02-19T19:14:17.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:17.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:17.255462+0000) 2024-02-19T19:14:18.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:18.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:18.255689+0000) 2024-02-19T19:14:19.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:19.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:19.255922+0000) 2024-02-19T19:14:20.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:20.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:20.256154+0000) 2024-02-19T19:14:21.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:21.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:21.256391+0000) 2024-02-19T19:14:22.095 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:14:22.099 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:14:22.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:22.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:22.256666+0000) 2024-02-19T19:14:23.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:23.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:23.256897+0000) 2024-02-19T19:14:24.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:24.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:24.257171+0000) 2024-02-19T19:14:25.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:25.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:25.257394+0000) 2024-02-19T19:14:26.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:26.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:26.257601+0000) 2024-02-19T19:14:27.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:27.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:27.257888+0000) 2024-02-19T19:14:28.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:28.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:28.258114+0000) 2024-02-19T19:14:29.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:29.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:29.258429+0000) 2024-02-19T19:14:30.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:30.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:30.258659+0000) 2024-02-19T19:14:31.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:31.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:31.258904+0000) 2024-02-19T19:14:32.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:32.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:32.259152+0000) 2024-02-19T19:14:33.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:33.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:33.259401+0000) 2024-02-19T19:14:34.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:34.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:34.259600+0000) 2024-02-19T19:14:35.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:35.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:35.259836+0000) 2024-02-19T19:14:36.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:36.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:36.260022+0000) 2024-02-19T19:14:37.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:37.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:37.260326+0000) 2024-02-19T19:14:38.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:38.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:38.260539+0000) 2024-02-19T19:14:39.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:39.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:39.260732+0000) 2024-02-19T19:14:40.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:40.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:40.260961+0000) 2024-02-19T19:14:41.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:41.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:41.261096+0000) 2024-02-19T19:14:42.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:42.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:42.261353+0000) 2024-02-19T19:14:43.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:43.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:43.261587+0000) 2024-02-19T19:14:44.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:44.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:44.261829+0000) 2024-02-19T19:14:45.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:45.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:45.262081+0000) 2024-02-19T19:14:46.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:46.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:46.262325+0000) 2024-02-19T19:14:47.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:47.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:47.262575+0000) 2024-02-19T19:14:48.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:48.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:48.262834+0000) 2024-02-19T19:14:49.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:49.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:49.263080+0000) 2024-02-19T19:14:50.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:50.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:50.263318+0000) 2024-02-19T19:14:51.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:51.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:51.263568+0000) 2024-02-19T19:14:52.234 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:14:52.237 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:14:52.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:52.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:52.263838+0000) 2024-02-19T19:14:53.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:53.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:53.264115+0000) 2024-02-19T19:14:54.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:54.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:54.264386+0000) 2024-02-19T19:14:55.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:55.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:55.264593+0000) 2024-02-19T19:14:56.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:56.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:56.264799+0000) 2024-02-19T19:14:57.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:57.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:57.265050+0000) 2024-02-19T19:14:58.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:58.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:58.265254+0000) 2024-02-19T19:14:59.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:14:59.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:14:59.265510+0000) 2024-02-19T19:15:00.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:00.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:00.265756+0000) 2024-02-19T19:15:01.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:01.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:01.265989+0000) 2024-02-19T19:15:02.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:02.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:02.266187+0000) 2024-02-19T19:15:03.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:03.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:03.266436+0000) 2024-02-19T19:15:04.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:04.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:04.266698+0000) 2024-02-19T19:15:05.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:05.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:05.266922+0000) 2024-02-19T19:15:06.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:06.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:06.267069+0000) 2024-02-19T19:15:07.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:07.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:07.267296+0000) 2024-02-19T19:15:08.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:08.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:08.267533+0000) 2024-02-19T19:15:09.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:09.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:09.267759+0000) 2024-02-19T19:15:10.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:10.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:10.267976+0000) 2024-02-19T19:15:11.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:11.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:11.268181+0000) 2024-02-19T19:15:12.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:12.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:12.268425+0000) 2024-02-19T19:15:13.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:13.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:13.268655+0000) 2024-02-19T19:15:14.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:14.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:14.268895+0000) 2024-02-19T19:15:15.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:15.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:15.269129+0000) 2024-02-19T19:15:16.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:16.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:16.269304+0000) 2024-02-19T19:15:17.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:17.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:17.269499+0000) 2024-02-19T19:15:18.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:18.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:18.269719+0000) 2024-02-19T19:15:19.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:19.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:19.269981+0000) 2024-02-19T19:15:20.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:20.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:20.270248+0000) 2024-02-19T19:15:21.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:21.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:21.270465+0000) 2024-02-19T19:15:22.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:22.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:22.270753+0000) 2024-02-19T19:15:22.349 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:15:22.352 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:15:23.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:23.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:23.270949+0000) 2024-02-19T19:15:24.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:24.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:24.271157+0000) 2024-02-19T19:15:25.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:25.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:25.271400+0000) 2024-02-19T19:15:26.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:26.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:26.271639+0000) 2024-02-19T19:15:27.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:27.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:27.271850+0000) 2024-02-19T19:15:28.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:28.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:28.272108+0000) 2024-02-19T19:15:29.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:29.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:29.272354+0000) 2024-02-19T19:15:30.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:30.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:30.272593+0000) 2024-02-19T19:15:31.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:31.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:31.272773+0000) 2024-02-19T19:15:32.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:32.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:32.273024+0000) 2024-02-19T19:15:33.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:33.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:33.273232+0000) 2024-02-19T19:15:34.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:34.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:34.273462+0000) 2024-02-19T19:15:35.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:35.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:35.273698+0000) 2024-02-19T19:15:36.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:36.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:36.273886+0000) 2024-02-19T19:15:37.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:37.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:37.274114+0000) 2024-02-19T19:15:38.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:38.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:38.274333+0000) 2024-02-19T19:15:39.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:39.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:39.274578+0000) 2024-02-19T19:15:40.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:40.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:40.274787+0000) 2024-02-19T19:15:41.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:41.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:41.275019+0000) 2024-02-19T19:15:42.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:42.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:42.275252+0000) 2024-02-19T19:15:43.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:43.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:43.275487+0000) 2024-02-19T19:15:44.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:44.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:44.275732+0000) 2024-02-19T19:15:45.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:45.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:45.275952+0000) 2024-02-19T19:15:46.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:46.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:46.276153+0000) 2024-02-19T19:15:47.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:47.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:47.276416+0000) 2024-02-19T19:15:48.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:48.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:48.276595+0000) 2024-02-19T19:15:49.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:49.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:49.276838+0000) 2024-02-19T19:15:50.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:50.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:50.277041+0000) 2024-02-19T19:15:51.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:51.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:51.277286+0000) 2024-02-19T19:15:52.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:52.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:52.277462+0000) 2024-02-19T19:15:52.414 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:15:52.417 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:15:53.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:53.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:53.277741+0000) 2024-02-19T19:15:54.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:54.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:54.278010+0000) 2024-02-19T19:15:55.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:55.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:55.278275+0000) 2024-02-19T19:15:56.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:56.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:56.278562+0000) 2024-02-19T19:15:57.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:57.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:57.278869+0000) 2024-02-19T19:15:58.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:58.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:58.279166+0000) 2024-02-19T19:15:59.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:15:59.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:15:59.279412+0000) 2024-02-19T19:16:00.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:00.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:00.279675+0000) 2024-02-19T19:16:01.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:01.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:01.279940+0000) 2024-02-19T19:16:02.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:02.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:02.280230+0000) 2024-02-19T19:16:03.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:03.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:03.280517+0000) 2024-02-19T19:16:04.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:04.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:04.280806+0000) 2024-02-19T19:16:05.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:05.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:05.280981+0000) 2024-02-19T19:16:06.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:06.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:06.281141+0000) 2024-02-19T19:16:07.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:07.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:07.281413+0000) 2024-02-19T19:16:08.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:08.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:08.281721+0000) 2024-02-19T19:16:09.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:09.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:09.281970+0000) 2024-02-19T19:16:10.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:10.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:10.282180+0000) 2024-02-19T19:16:11.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:11.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:11.282432+0000) 2024-02-19T19:16:12.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:12.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:12.282689+0000) 2024-02-19T19:16:13.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:13.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:13.283035+0000) 2024-02-19T19:16:14.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:14.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:14.283331+0000) 2024-02-19T19:16:15.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:15.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:15.283671+0000) 2024-02-19T19:16:16.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:16.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:16.283999+0000) 2024-02-19T19:16:17.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:17.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:17.284328+0000) 2024-02-19T19:16:18.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:18.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:18.284577+0000) 2024-02-19T19:16:19.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:19.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:19.284758+0000) 2024-02-19T19:16:20.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:20.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:20.285031+0000) 2024-02-19T19:16:21.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:21.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:21.285321+0000) 2024-02-19T19:16:22.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:22.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:22.285600+0000) 2024-02-19T19:16:22.498 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:16:22.502 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:16:23.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:23.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:23.285890+0000) 2024-02-19T19:16:24.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:24.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:24.286176+0000) 2024-02-19T19:16:25.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:25.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:25.286459+0000) 2024-02-19T19:16:26.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:26.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:26.286780+0000) 2024-02-19T19:16:27.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:27.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:27.287040+0000) 2024-02-19T19:16:28.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:28.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:28.287261+0000) 2024-02-19T19:16:29.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:29.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:29.287560+0000) 2024-02-19T19:16:30.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:30.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:30.287839+0000) 2024-02-19T19:16:31.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:31.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:31.288092+0000) 2024-02-19T19:16:32.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:32.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:32.288365+0000) 2024-02-19T19:16:33.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:33.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:33.288624+0000) 2024-02-19T19:16:34.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:34.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:34.288828+0000) 2024-02-19T19:16:35.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:35.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:35.289089+0000) 2024-02-19T19:16:36.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:36.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:36.289368+0000) 2024-02-19T19:16:37.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:37.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:37.289697+0000) 2024-02-19T19:16:38.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:38.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:38.290013+0000) 2024-02-19T19:16:39.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:39.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:39.290285+0000) 2024-02-19T19:16:40.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:40.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:40.290592+0000) 2024-02-19T19:16:41.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:41.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:41.290890+0000) 2024-02-19T19:16:42.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:42.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:42.291176+0000) 2024-02-19T19:16:43.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:43.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:43.291400+0000) 2024-02-19T19:16:44.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:44.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:44.291589+0000) 2024-02-19T19:16:45.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:45.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:45.291840+0000) 2024-02-19T19:16:46.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:46.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:46.292091+0000) 2024-02-19T19:16:47.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:47.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:47.292379+0000) 2024-02-19T19:16:48.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:48.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:48.292627+0000) 2024-02-19T19:16:49.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:49.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:49.292856+0000) 2024-02-19T19:16:50.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:50.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:50.293160+0000) 2024-02-19T19:16:51.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:51.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:51.293456+0000) 2024-02-19T19:16:52.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:52.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:52.293742+0000) 2024-02-19T19:16:52.574 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:16:52.577 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:16:53.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:53.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:53.293939+0000) 2024-02-19T19:16:54.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:54.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:54.294205+0000) 2024-02-19T19:16:55.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:55.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:55.294375+0000) 2024-02-19T19:16:56.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:56.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:56.294586+0000) 2024-02-19T19:16:57.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:57.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:57.294873+0000) 2024-02-19T19:16:58.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:58.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:58.295089+0000) 2024-02-19T19:16:59.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:16:59.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:16:59.295361+0000) 2024-02-19T19:17:00.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:00.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:00.295659+0000) 2024-02-19T19:17:01.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:01.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:01.295927+0000) 2024-02-19T19:17:02.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:02.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:02.296199+0000) 2024-02-19T19:17:03.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:03.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:03.296514+0000) 2024-02-19T19:17:04.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:04.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:04.296756+0000) 2024-02-19T19:17:05.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:05.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:05.297008+0000) 2024-02-19T19:17:06.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:06.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:06.297245+0000) 2024-02-19T19:17:07.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:07.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:07.297522+0000) 2024-02-19T19:17:08.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:08.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:08.297846+0000) 2024-02-19T19:17:09.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:09.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:09.298138+0000) 2024-02-19T19:17:10.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:10.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:10.298427+0000) 2024-02-19T19:17:11.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:11.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:11.298597+0000) 2024-02-19T19:17:12.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:12.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:12.298904+0000) 2024-02-19T19:17:13.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:13.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:13.299203+0000) 2024-02-19T19:17:14.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:14.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:14.299436+0000) 2024-02-19T19:17:15.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:15.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:15.299617+0000) 2024-02-19T19:17:16.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:16.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:16.299765+0000) 2024-02-19T19:17:17.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:17.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:17.300003+0000) 2024-02-19T19:17:18.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:18.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:18.300232+0000) 2024-02-19T19:17:19.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:19.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:19.300489+0000) 2024-02-19T19:17:20.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:20.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:20.300727+0000) 2024-02-19T19:17:21.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:21.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:21.300909+0000) 2024-02-19T19:17:22.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:22.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:22.301151+0000) 2024-02-19T19:17:22.633 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:17:22.636 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:17:23.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:23.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:23.301402+0000) 2024-02-19T19:17:24.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:24.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:24.301643+0000) 2024-02-19T19:17:25.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:25.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:25.301833+0000) 2024-02-19T19:17:26.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:26.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:26.302045+0000) 2024-02-19T19:17:27.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:27.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:27.302332+0000) 2024-02-19T19:17:28.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:28.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:28.302554+0000) 2024-02-19T19:17:29.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:29.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:29.302802+0000) 2024-02-19T19:17:30.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:30.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:30.303049+0000) 2024-02-19T19:17:31.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:31.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:31.303245+0000) 2024-02-19T19:17:32.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:32.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:32.303456+0000) 2024-02-19T19:17:33.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:33.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:33.303716+0000) 2024-02-19T19:17:34.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:34.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:34.303964+0000) 2024-02-19T19:17:35.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:35.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:35.304236+0000) 2024-02-19T19:17:36.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:36.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:36.304413+0000) 2024-02-19T19:17:37.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:37.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:37.304684+0000) 2024-02-19T19:17:38.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:38.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:38.304880+0000) 2024-02-19T19:17:39.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:39.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:39.305135+0000) 2024-02-19T19:17:40.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:40.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:40.305378+0000) 2024-02-19T19:17:41.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:41.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:41.305620+0000) 2024-02-19T19:17:42.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:42.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:42.305861+0000) 2024-02-19T19:17:43.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:43.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:43.306072+0000) 2024-02-19T19:17:44.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:44.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:44.306363+0000) 2024-02-19T19:17:45.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:45.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:45.306659+0000) 2024-02-19T19:17:46.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:46.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:46.306943+0000) 2024-02-19T19:17:47.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:47.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:47.307211+0000) 2024-02-19T19:17:48.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:48.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:48.307482+0000) 2024-02-19T19:17:49.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:49.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:49.307794+0000) 2024-02-19T19:17:50.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:50.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:50.308046+0000) 2024-02-19T19:17:51.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:51.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:51.308372+0000) 2024-02-19T19:17:52.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:52.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:52.308668+0000) 2024-02-19T19:17:52.738 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:17:52.741 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:17:53.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:53.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:53.308896+0000) 2024-02-19T19:17:54.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:54.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:54.309228+0000) 2024-02-19T19:17:55.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:55.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:55.309505+0000) 2024-02-19T19:17:56.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:56.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:56.309768+0000) 2024-02-19T19:17:57.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:57.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:57.310101+0000) 2024-02-19T19:17:58.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:58.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:58.310337+0000) 2024-02-19T19:17:59.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:17:59.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:17:59.310564+0000) 2024-02-19T19:18:00.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:00.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:00.310854+0000) 2024-02-19T19:18:01.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:01.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:01.311047+0000) 2024-02-19T19:18:02.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:02.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:02.311321+0000) 2024-02-19T19:18:03.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:03.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:03.311589+0000) 2024-02-19T19:18:04.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:04.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:04.311787+0000) 2024-02-19T19:18:05.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:05.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:05.312079+0000) 2024-02-19T19:18:06.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:06.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:06.312280+0000) 2024-02-19T19:18:07.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:07.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:07.312546+0000) 2024-02-19T19:18:08.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:08.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:08.312702+0000) 2024-02-19T19:18:09.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:09.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:09.312959+0000) 2024-02-19T19:18:10.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:10.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:10.313180+0000) 2024-02-19T19:18:11.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:11.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:11.313442+0000) 2024-02-19T19:18:12.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:12.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:12.313754+0000) 2024-02-19T19:18:13.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:13.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:13.313995+0000) 2024-02-19T19:18:14.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:14.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:14.314196+0000) 2024-02-19T19:18:15.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:15.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:15.314454+0000) 2024-02-19T19:18:16.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:16.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:16.314749+0000) 2024-02-19T19:18:17.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:17.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:17.315016+0000) 2024-02-19T19:18:18.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:18.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:18.315226+0000) 2024-02-19T19:18:19.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:19.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:19.315483+0000) 2024-02-19T19:18:20.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:20.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:20.315801+0000) 2024-02-19T19:18:21.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:21.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:21.316035+0000) 2024-02-19T19:18:22.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:22.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:22.316208+0000) 2024-02-19T19:18:22.817 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:18:22.820 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:18:23.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:23.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:23.316469+0000) 2024-02-19T19:18:24.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:24.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:24.316697+0000) 2024-02-19T19:18:25.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:25.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:25.316879+0000) 2024-02-19T19:18:26.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:26.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:26.317142+0000) 2024-02-19T19:18:27.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:27.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:27.317447+0000) 2024-02-19T19:18:28.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:28.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:28.317644+0000) 2024-02-19T19:18:29.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:29.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:29.317892+0000) 2024-02-19T19:18:30.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:30.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:30.318156+0000) 2024-02-19T19:18:31.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:31.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:31.318458+0000) 2024-02-19T19:18:32.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:32.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:32.318765+0000) 2024-02-19T19:18:33.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:33.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:33.319014+0000) 2024-02-19T19:18:34.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:34.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:34.319317+0000) 2024-02-19T19:18:35.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:35.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:35.319624+0000) 2024-02-19T19:18:36.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:36.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:36.319945+0000) 2024-02-19T19:18:37.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:37.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:37.320160+0000) 2024-02-19T19:18:38.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:38.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:38.320422+0000) 2024-02-19T19:18:39.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:39.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:39.320628+0000) 2024-02-19T19:18:40.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:40.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:40.320854+0000) 2024-02-19T19:18:41.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:41.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:41.321153+0000) 2024-02-19T19:18:42.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:42.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:42.321382+0000) 2024-02-19T19:18:43.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:43.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:43.321646+0000) 2024-02-19T19:18:44.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:44.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:44.321885+0000) 2024-02-19T19:18:45.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:45.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:45.322137+0000) 2024-02-19T19:18:46.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:46.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:46.322382+0000) 2024-02-19T19:18:47.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:47.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:47.322631+0000) 2024-02-19T19:18:48.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:48.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:48.322922+0000) 2024-02-19T19:18:49.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:49.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:49.323099+0000) 2024-02-19T19:18:50.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:50.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:50.323390+0000) 2024-02-19T19:18:51.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:51.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:51.323648+0000) 2024-02-19T19:18:52.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:52.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:52.323896+0000) 2024-02-19T19:18:52.869 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:18:52.872 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:18:53.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:53.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:53.324189+0000) 2024-02-19T19:18:54.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:54.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:54.324386+0000) 2024-02-19T19:18:55.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:55.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:55.324551+0000) 2024-02-19T19:18:56.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:56.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:56.324818+0000) 2024-02-19T19:18:57.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:57.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:57.325099+0000) 2024-02-19T19:18:58.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:58.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:58.325370+0000) 2024-02-19T19:18:59.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:18:59.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:18:59.325506+0000) 2024-02-19T19:19:00.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:00.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:00.325768+0000) 2024-02-19T19:19:01.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:01.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:01.326013+0000) 2024-02-19T19:19:02.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:02.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:02.326255+0000) 2024-02-19T19:19:03.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:03.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:03.326415+0000) 2024-02-19T19:19:04.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:04.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:04.326596+0000) 2024-02-19T19:19:05.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:05.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:05.326878+0000) 2024-02-19T19:19:06.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:06.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:06.327031+0000) 2024-02-19T19:19:07.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:07.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:07.327218+0000) 2024-02-19T19:19:08.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:08.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:08.327420+0000) 2024-02-19T19:19:09.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:09.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:09.327634+0000) 2024-02-19T19:19:10.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:10.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:10.327881+0000) 2024-02-19T19:19:11.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:11.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:11.328021+0000) 2024-02-19T19:19:12.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:12.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:12.328200+0000) 2024-02-19T19:19:13.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:13.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:13.328480+0000) 2024-02-19T19:19:14.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:14.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:14.328716+0000) 2024-02-19T19:19:15.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:15.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:15.328887+0000) 2024-02-19T19:19:16.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:16.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:16.329136+0000) 2024-02-19T19:19:17.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:17.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:17.329383+0000) 2024-02-19T19:19:18.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:18.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:18.329617+0000) 2024-02-19T19:19:19.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:19.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:19.329779+0000) 2024-02-19T19:19:20.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:20.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:20.330020+0000) 2024-02-19T19:19:21.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:21.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:21.330216+0000) 2024-02-19T19:19:22.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:22.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:22.330473+0000) 2024-02-19T19:19:23.003 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:19:23.007 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:19:23.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:23.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:23.330750+0000) 2024-02-19T19:19:24.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:24.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:24.330953+0000) 2024-02-19T19:19:25.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:25.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:25.331152+0000) 2024-02-19T19:19:26.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:26.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:26.331352+0000) 2024-02-19T19:19:27.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:27.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:27.331553+0000) 2024-02-19T19:19:28.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:28.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:28.331800+0000) 2024-02-19T19:19:29.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:29.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:29.332050+0000) 2024-02-19T19:19:30.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:30.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:30.332263+0000) 2024-02-19T19:19:31.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:31.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:31.332502+0000) 2024-02-19T19:19:32.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:32.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:32.332757+0000) 2024-02-19T19:19:33.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:33.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:33.333009+0000) 2024-02-19T19:19:34.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:34.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:34.333205+0000) 2024-02-19T19:19:35.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:35.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:35.333348+0000) 2024-02-19T19:19:36.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:36.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:36.333573+0000) 2024-02-19T19:19:37.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:37.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:37.333804+0000) 2024-02-19T19:19:38.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:38.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:38.334012+0000) 2024-02-19T19:19:39.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:39.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:39.334250+0000) 2024-02-19T19:19:40.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:40.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:40.334455+0000) 2024-02-19T19:19:41.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:41.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:41.334679+0000) 2024-02-19T19:19:42.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:42.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:42.334906+0000) 2024-02-19T19:19:43.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:43.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:43.335146+0000) 2024-02-19T19:19:44.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:44.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:44.335411+0000) 2024-02-19T19:19:45.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:45.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:45.335633+0000) 2024-02-19T19:19:46.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:46.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:46.335766+0000) 2024-02-19T19:19:47.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:47.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:47.336015+0000) 2024-02-19T19:19:48.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:48.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:48.336248+0000) 2024-02-19T19:19:49.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:49.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:49.336509+0000) 2024-02-19T19:19:50.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:50.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:50.336781+0000) 2024-02-19T19:19:51.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:51.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:51.336959+0000) 2024-02-19T19:19:52.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:52.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:52.337141+0000) 2024-02-19T19:19:53.124 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:19:53.127 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:19:53.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:53.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:53.337328+0000) 2024-02-19T19:19:54.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:54.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:54.337499+0000) 2024-02-19T19:19:55.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:55.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:55.337725+0000) 2024-02-19T19:19:56.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:56.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:56.337959+0000) 2024-02-19T19:19:57.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:57.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:57.338224+0000) 2024-02-19T19:19:58.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:58.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:58.338425+0000) 2024-02-19T19:19:59.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:19:59.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:19:59.338601+0000) 2024-02-19T19:20:00.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:00.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:00.338864+0000) 2024-02-19T19:20:01.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:01.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:01.339007+0000) 2024-02-19T19:20:02.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:02.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:02.339262+0000) 2024-02-19T19:20:03.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:03.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:03.339519+0000) 2024-02-19T19:20:04.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:04.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:04.339772+0000) 2024-02-19T19:20:05.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:05.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:05.339963+0000) 2024-02-19T19:20:06.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:06.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:06.340199+0000) 2024-02-19T19:20:07.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:07.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:07.340429+0000) 2024-02-19T19:20:08.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:08.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:08.340689+0000) 2024-02-19T19:20:09.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:09.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:09.340967+0000) 2024-02-19T19:20:10.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:10.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:10.341115+0000) 2024-02-19T19:20:11.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:11.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:11.341352+0000) 2024-02-19T19:20:12.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:12.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:12.341592+0000) 2024-02-19T19:20:13.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:13.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:13.341864+0000) 2024-02-19T19:20:14.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:14.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:14.342148+0000) 2024-02-19T19:20:15.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:15.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:15.342372+0000) 2024-02-19T19:20:16.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:16.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:16.342564+0000) 2024-02-19T19:20:17.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:17.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:17.342789+0000) 2024-02-19T19:20:18.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:18.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:18.343038+0000) 2024-02-19T19:20:19.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:19.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:19.343314+0000) 2024-02-19T19:20:20.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:20.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:20.343589+0000) 2024-02-19T19:20:21.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:21.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:21.343853+0000) 2024-02-19T19:20:22.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:22.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:22.344120+0000) 2024-02-19T19:20:23.244 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:20:23.249 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:20:23.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:23.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:23.344349+0000) 2024-02-19T19:20:24.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:24.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:24.344617+0000) 2024-02-19T19:20:25.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:25.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:25.344849+0000) 2024-02-19T19:20:26.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:26.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:26.345079+0000) 2024-02-19T19:20:27.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:27.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:27.345243+0000) 2024-02-19T19:20:28.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:28.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:28.345460+0000) 2024-02-19T19:20:29.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:29.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:29.345713+0000) 2024-02-19T19:20:30.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:30.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:30.345897+0000) 2024-02-19T19:20:31.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:31.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:31.346217+0000) 2024-02-19T19:20:32.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:32.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:32.346478+0000) 2024-02-19T19:20:33.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:33.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:33.346668+0000) 2024-02-19T19:20:34.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:34.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:34.346915+0000) 2024-02-19T19:20:35.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:35.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:35.347109+0000) 2024-02-19T19:20:36.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:36.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:36.347388+0000) 2024-02-19T19:20:37.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:37.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:37.347591+0000) 2024-02-19T19:20:38.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:38.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:38.347835+0000) 2024-02-19T19:20:39.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:39.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:39.348070+0000) 2024-02-19T19:20:40.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:40.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:40.348310+0000) 2024-02-19T19:20:41.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:41.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:41.348546+0000) 2024-02-19T19:20:42.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:42.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:42.348788+0000) 2024-02-19T19:20:43.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:43.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:43.348984+0000) 2024-02-19T19:20:44.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:44.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:44.349246+0000) 2024-02-19T19:20:45.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:45.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:45.349509+0000) 2024-02-19T19:20:46.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:46.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:46.349755+0000) 2024-02-19T19:20:47.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:47.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:47.350005+0000) 2024-02-19T19:20:48.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:48.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:48.350242+0000) 2024-02-19T19:20:49.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:49.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:49.350436+0000) 2024-02-19T19:20:50.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:50.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:50.350636+0000) 2024-02-19T19:20:51.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:51.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:51.350890+0000) 2024-02-19T19:20:52.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:52.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:52.351130+0000) 2024-02-19T19:20:53.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:53.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:53.351366+0000) 2024-02-19T19:20:53.383 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:20:53.386 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:20:54.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:54.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:54.351552+0000) 2024-02-19T19:20:55.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:55.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:55.351794+0000) 2024-02-19T19:20:56.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:56.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:56.352000+0000) 2024-02-19T19:20:57.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:57.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:57.352256+0000) 2024-02-19T19:20:58.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:58.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:58.352567+0000) 2024-02-19T19:20:59.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:20:59.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:20:59.352772+0000) 2024-02-19T19:21:00.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:00.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:00.352932+0000) 2024-02-19T19:21:01.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:01.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:01.353199+0000) 2024-02-19T19:21:02.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:02.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:02.353420+0000) 2024-02-19T19:21:03.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:03.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:03.353590+0000) 2024-02-19T19:21:04.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:04.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:04.353898+0000) 2024-02-19T19:21:05.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:05.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:05.354186+0000) 2024-02-19T19:21:06.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:06.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:06.354492+0000) 2024-02-19T19:21:07.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:07.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:07.354839+0000) 2024-02-19T19:21:08.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:08.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:08.355127+0000) 2024-02-19T19:21:09.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:09.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:09.355347+0000) 2024-02-19T19:21:10.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:10.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:10.355487+0000) 2024-02-19T19:21:11.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:11.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:11.355770+0000) 2024-02-19T19:21:12.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:12.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:12.356068+0000) 2024-02-19T19:21:13.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:13.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:13.356351+0000) 2024-02-19T19:21:14.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:14.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:14.356555+0000) 2024-02-19T19:21:15.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:15.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:15.356776+0000) 2024-02-19T19:21:16.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:16.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:16.357046+0000) 2024-02-19T19:21:17.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:17.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:17.357350+0000) 2024-02-19T19:21:18.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:18.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:18.357516+0000) 2024-02-19T19:21:19.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:19.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:19.357731+0000) 2024-02-19T19:21:20.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:20.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:20.358004+0000) 2024-02-19T19:21:21.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:21.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:21.358267+0000) 2024-02-19T19:21:22.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:22.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:22.358463+0000) 2024-02-19T19:21:23.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:23.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:23.358737+0000) 2024-02-19T19:21:23.437 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:21:23.441 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:21:24.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:24.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:24.358978+0000) 2024-02-19T19:21:25.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:25.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:25.359251+0000) 2024-02-19T19:21:26.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:26.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:26.359422+0000) 2024-02-19T19:21:27.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:27.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:27.359724+0000) 2024-02-19T19:21:28.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:28.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:28.360001+0000) 2024-02-19T19:21:29.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:29.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:29.360263+0000) 2024-02-19T19:21:30.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:30.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:30.360530+0000) 2024-02-19T19:21:31.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:31.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:31.360829+0000) 2024-02-19T19:21:32.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:32.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:32.361064+0000) 2024-02-19T19:21:33.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:33.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:33.361278+0000) 2024-02-19T19:21:34.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:34.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:34.361453+0000) 2024-02-19T19:21:35.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:35.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:35.361643+0000) 2024-02-19T19:21:36.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:36.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:36.361860+0000) 2024-02-19T19:21:37.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:37.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:37.362165+0000) 2024-02-19T19:21:38.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:38.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:38.362455+0000) 2024-02-19T19:21:39.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:39.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:39.362685+0000) 2024-02-19T19:21:40.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:40.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:40.362965+0000) 2024-02-19T19:21:41.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:41.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:41.363177+0000) 2024-02-19T19:21:42.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:42.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:42.363319+0000) 2024-02-19T19:21:43.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:43.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:43.363543+0000) 2024-02-19T19:21:44.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:44.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:44.363733+0000) 2024-02-19T19:21:45.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:45.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:45.363945+0000) 2024-02-19T19:21:46.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:46.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:46.364221+0000) 2024-02-19T19:21:47.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:47.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:47.364510+0000) 2024-02-19T19:21:48.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:48.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:48.364736+0000) 2024-02-19T19:21:49.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:49.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:49.364989+0000) 2024-02-19T19:21:50.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:50.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:50.365282+0000) 2024-02-19T19:21:51.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:51.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:51.365547+0000) 2024-02-19T19:21:52.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:52.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:52.365796+0000) 2024-02-19T19:21:53.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:53.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:53.366018+0000) 2024-02-19T19:21:53.501 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:21:53.504 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:21:54.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:54.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:54.366264+0000) 2024-02-19T19:21:55.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:55.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:55.366544+0000) 2024-02-19T19:21:56.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:56.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:56.366820+0000) 2024-02-19T19:21:57.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:57.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:57.367083+0000) 2024-02-19T19:21:58.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:58.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:58.367275+0000) 2024-02-19T19:21:59.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:21:59.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:21:59.367501+0000) 2024-02-19T19:22:00.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:00.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:00.367754+0000) 2024-02-19T19:22:01.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:01.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:01.367961+0000) 2024-02-19T19:22:02.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:02.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:02.368203+0000) 2024-02-19T19:22:03.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:03.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:03.368459+0000) 2024-02-19T19:22:04.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:04.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:04.368693+0000) 2024-02-19T19:22:05.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:05.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:05.368886+0000) 2024-02-19T19:22:06.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:06.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:06.369130+0000) 2024-02-19T19:22:07.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:07.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:07.369408+0000) 2024-02-19T19:22:08.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:08.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:08.369579+0000) 2024-02-19T19:22:09.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:09.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:09.369824+0000) 2024-02-19T19:22:10.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:10.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:10.370068+0000) 2024-02-19T19:22:11.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:11.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:11.370316+0000) 2024-02-19T19:22:12.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:12.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:12.370535+0000) 2024-02-19T19:22:13.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:13.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:13.370786+0000) 2024-02-19T19:22:14.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:14.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:14.370922+0000) 2024-02-19T19:22:15.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:15.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:15.371139+0000) 2024-02-19T19:22:16.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:16.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:16.371387+0000) 2024-02-19T19:22:17.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:17.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:17.371634+0000) 2024-02-19T19:22:18.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:18.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:18.371865+0000) 2024-02-19T19:22:19.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:19.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:19.372112+0000) 2024-02-19T19:22:20.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:20.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:20.372319+0000) 2024-02-19T19:22:21.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:21.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:21.372569+0000) 2024-02-19T19:22:22.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:22.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:22.372823+0000) 2024-02-19T19:22:23.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:23.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:23.373009+0000) 2024-02-19T19:22:23.624 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:22:23.627 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:22:24.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:24.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:24.373257+0000) 2024-02-19T19:22:25.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:25.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:25.373508+0000) 2024-02-19T19:22:26.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:26.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:26.373663+0000) 2024-02-19T19:22:27.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:27.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:27.373916+0000) 2024-02-19T19:22:28.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:28.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:28.374105+0000) 2024-02-19T19:22:29.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:29.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:29.374354+0000) 2024-02-19T19:22:30.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:30.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:30.374627+0000) 2024-02-19T19:22:31.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:31.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:31.374856+0000) 2024-02-19T19:22:32.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:32.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:32.374997+0000) 2024-02-19T19:22:33.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:33.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:33.375136+0000) 2024-02-19T19:22:34.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:34.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:34.375320+0000) 2024-02-19T19:22:35.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:35.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:35.375584+0000) 2024-02-19T19:22:36.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:36.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:36.375813+0000) 2024-02-19T19:22:37.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:37.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:37.376099+0000) 2024-02-19T19:22:38.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:38.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:38.376301+0000) 2024-02-19T19:22:39.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:39.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:39.376480+0000) 2024-02-19T19:22:40.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:40.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:40.376773+0000) 2024-02-19T19:22:41.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:41.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:41.377014+0000) 2024-02-19T19:22:42.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:42.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:42.377292+0000) 2024-02-19T19:22:43.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:43.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:43.377546+0000) 2024-02-19T19:22:44.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:44.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:44.377808+0000) 2024-02-19T19:22:45.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:45.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:45.377994+0000) 2024-02-19T19:22:46.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:46.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:46.378215+0000) 2024-02-19T19:22:47.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:47.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:47.378431+0000) 2024-02-19T19:22:48.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:48.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:48.378611+0000) 2024-02-19T19:22:49.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:49.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:49.378792+0000) 2024-02-19T19:22:50.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:50.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:50.379002+0000) 2024-02-19T19:22:51.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:51.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:51.379263+0000) 2024-02-19T19:22:52.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:52.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:52.379490+0000) 2024-02-19T19:22:53.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:53.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:53.379662+0000) 2024-02-19T19:22:53.762 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:22:53.765 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:22:54.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:54.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:54.379922+0000) 2024-02-19T19:22:55.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:55.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:55.380218+0000) 2024-02-19T19:22:56.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:56.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:56.380519+0000) 2024-02-19T19:22:57.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:57.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:57.380737+0000) 2024-02-19T19:22:58.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:58.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:58.381010+0000) 2024-02-19T19:22:59.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:22:59.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:22:59.381307+0000) 2024-02-19T19:23:00.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:00.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:00.381541+0000) 2024-02-19T19:23:01.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:01.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:01.381800+0000) 2024-02-19T19:23:02.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:02.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:02.382059+0000) 2024-02-19T19:23:03.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:03.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:03.382351+0000) 2024-02-19T19:23:04.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:04.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:04.382592+0000) 2024-02-19T19:23:05.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:05.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:05.382855+0000) 2024-02-19T19:23:06.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:06.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:06.383200+0000) 2024-02-19T19:23:07.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:07.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:07.383518+0000) 2024-02-19T19:23:08.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:08.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:08.383802+0000) 2024-02-19T19:23:09.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:09.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:09.384080+0000) 2024-02-19T19:23:10.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:10.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:10.384333+0000) 2024-02-19T19:23:11.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:11.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:11.384617+0000) 2024-02-19T19:23:12.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:12.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:12.384907+0000) 2024-02-19T19:23:13.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:13.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:13.385178+0000) 2024-02-19T19:23:14.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:14.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:14.385479+0000) 2024-02-19T19:23:15.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:15.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:15.385691+0000) 2024-02-19T19:23:16.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:16.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:16.385952+0000) 2024-02-19T19:23:17.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:17.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:17.386244+0000) 2024-02-19T19:23:18.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:18.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:18.386421+0000) 2024-02-19T19:23:19.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:19.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:19.386677+0000) 2024-02-19T19:23:20.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:20.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:20.386946+0000) 2024-02-19T19:23:21.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:21.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:21.387199+0000) 2024-02-19T19:23:22.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:22.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:22.387471+0000) 2024-02-19T19:23:23.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:23.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:23.387774+0000) 2024-02-19T19:23:23.842 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:23:23.845 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:23:24.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:24.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:24.388001+0000) 2024-02-19T19:23:25.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:25.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:25.388221+0000) 2024-02-19T19:23:26.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:26.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:26.388423+0000) 2024-02-19T19:23:27.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:27.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:27.388618+0000) 2024-02-19T19:23:28.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:28.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:28.388854+0000) 2024-02-19T19:23:29.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:29.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:29.389119+0000) 2024-02-19T19:23:30.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:30.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:30.389364+0000) 2024-02-19T19:23:31.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:31.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:31.389601+0000) 2024-02-19T19:23:32.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:32.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:32.389848+0000) 2024-02-19T19:23:33.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:33.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:33.390088+0000) 2024-02-19T19:23:34.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:34.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:34.390322+0000) 2024-02-19T19:23:35.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:35.385+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:35.390503+0000) 2024-02-19T19:23:36.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:36.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:36.390813+0000) 2024-02-19T19:23:37.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:37.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:37.391004+0000) 2024-02-19T19:23:38.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:38.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:38.391206+0000) 2024-02-19T19:23:39.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:39.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:39.391377+0000) 2024-02-19T19:23:40.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:40.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:40.391612+0000) 2024-02-19T19:23:41.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:41.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:41.391818+0000) 2024-02-19T19:23:42.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:42.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:42.392070+0000) 2024-02-19T19:23:43.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:43.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:43.392283+0000) 2024-02-19T19:23:44.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:44.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:44.392458+0000) 2024-02-19T19:23:45.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:45.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:45.392644+0000) 2024-02-19T19:23:46.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:46.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:46.392889+0000) 2024-02-19T19:23:47.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:47.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:47.393086+0000) 2024-02-19T19:23:48.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:48.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:48.393331+0000) 2024-02-19T19:23:49.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:49.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:49.393583+0000) 2024-02-19T19:23:50.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:50.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:50.393838+0000) 2024-02-19T19:23:51.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:51.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:51.394046+0000) 2024-02-19T19:23:52.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:52.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:52.394318+0000) 2024-02-19T19:23:53.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:53.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:53.394581+0000) 2024-02-19T19:23:53.898 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:23:53.901 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:23:54.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:54.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:54.394785+0000) 2024-02-19T19:23:55.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:55.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:55.394940+0000) 2024-02-19T19:23:56.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:56.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:56.395158+0000) 2024-02-19T19:23:57.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:57.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:57.395434+0000) 2024-02-19T19:23:58.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:58.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:58.395674+0000) 2024-02-19T19:23:59.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:23:59.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:23:59.395981+0000) 2024-02-19T19:24:00.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:00.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:00.396220+0000) 2024-02-19T19:24:01.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:01.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:01.396395+0000) 2024-02-19T19:24:02.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:02.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:02.396639+0000) 2024-02-19T19:24:03.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:03.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:03.396827+0000) 2024-02-19T19:24:04.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:04.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:04.397090+0000) 2024-02-19T19:24:05.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:05.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:05.397298+0000) 2024-02-19T19:24:06.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:06.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:06.397502+0000) 2024-02-19T19:24:07.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:07.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:07.397753+0000) 2024-02-19T19:24:08.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:08.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:08.397930+0000) 2024-02-19T19:24:09.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:09.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:09.398195+0000) 2024-02-19T19:24:10.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:10.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:10.398451+0000) 2024-02-19T19:24:11.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:11.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:11.398761+0000) 2024-02-19T19:24:12.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:12.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:12.399012+0000) 2024-02-19T19:24:13.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:13.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:13.399248+0000) 2024-02-19T19:24:14.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:14.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:14.399466+0000) 2024-02-19T19:24:15.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:15.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:15.399718+0000) 2024-02-19T19:24:16.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:16.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:16.399976+0000) 2024-02-19T19:24:17.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:17.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:17.400233+0000) 2024-02-19T19:24:18.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:18.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:18.400445+0000) 2024-02-19T19:24:19.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:19.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:19.400704+0000) 2024-02-19T19:24:20.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:20.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:20.400971+0000) 2024-02-19T19:24:21.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:21.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:21.401256+0000) 2024-02-19T19:24:22.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:22.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:22.401546+0000) 2024-02-19T19:24:23.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:23.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:23.401850+0000) 2024-02-19T19:24:24.020 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:24:24.023 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:24:24.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:24.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:24.402024+0000) 2024-02-19T19:24:25.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:25.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:25.402292+0000) 2024-02-19T19:24:26.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:26.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:26.402574+0000) 2024-02-19T19:24:27.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:27.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:27.402809+0000) 2024-02-19T19:24:28.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:28.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:28.403091+0000) 2024-02-19T19:24:29.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:29.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:29.403409+0000) 2024-02-19T19:24:30.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:30.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:30.403691+0000) 2024-02-19T19:24:31.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:31.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:31.403983+0000) 2024-02-19T19:24:32.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:32.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:32.404239+0000) 2024-02-19T19:24:33.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:33.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:33.404522+0000) 2024-02-19T19:24:34.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:34.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:34.404776+0000) 2024-02-19T19:24:35.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:35.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:35.404910+0000) 2024-02-19T19:24:36.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:36.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:36.405188+0000) 2024-02-19T19:24:37.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:37.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:37.405477+0000) 2024-02-19T19:24:38.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:38.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:38.405680+0000) 2024-02-19T19:24:39.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:39.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:39.405883+0000) 2024-02-19T19:24:40.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:40.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:40.406176+0000) 2024-02-19T19:24:41.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:41.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:41.406461+0000) 2024-02-19T19:24:42.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:42.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:42.406659+0000) 2024-02-19T19:24:43.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:43.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:43.406950+0000) 2024-02-19T19:24:44.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:44.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:44.407166+0000) 2024-02-19T19:24:45.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:45.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:45.407375+0000) 2024-02-19T19:24:46.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:46.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:46.407554+0000) 2024-02-19T19:24:47.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:47.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:47.407772+0000) 2024-02-19T19:24:48.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:48.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:48.407979+0000) 2024-02-19T19:24:49.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:49.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:49.408205+0000) 2024-02-19T19:24:50.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:50.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:50.408343+0000) 2024-02-19T19:24:51.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:51.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:51.408630+0000) 2024-02-19T19:24:52.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:52.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:52.408916+0000) 2024-02-19T19:24:53.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:53.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:53.409193+0000) 2024-02-19T19:24:54.148 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:24:54.152 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:24:54.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:54.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:54.409514+0000) 2024-02-19T19:24:55.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:55.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:55.409803+0000) 2024-02-19T19:24:56.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:56.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:56.410042+0000) 2024-02-19T19:24:57.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:57.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:57.410304+0000) 2024-02-19T19:24:58.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:58.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:58.410545+0000) 2024-02-19T19:24:59.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:24:59.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:24:59.410815+0000) 2024-02-19T19:25:00.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:00.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:00.411116+0000) 2024-02-19T19:25:01.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:01.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:01.411372+0000) 2024-02-19T19:25:02.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:02.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:02.411578+0000) 2024-02-19T19:25:03.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:03.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:03.411841+0000) 2024-02-19T19:25:04.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:04.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:04.412029+0000) 2024-02-19T19:25:05.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:05.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:05.412264+0000) 2024-02-19T19:25:06.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:06.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:06.412468+0000) 2024-02-19T19:25:07.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:07.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:07.412734+0000) 2024-02-19T19:25:08.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:08.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:08.412932+0000) 2024-02-19T19:25:09.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:09.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:09.413138+0000) 2024-02-19T19:25:10.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:10.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:10.413399+0000) 2024-02-19T19:25:11.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:11.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:11.413584+0000) 2024-02-19T19:25:12.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:12.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:12.413833+0000) 2024-02-19T19:25:13.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:13.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:13.414099+0000) 2024-02-19T19:25:14.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:14.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:14.414333+0000) 2024-02-19T19:25:15.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:15.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:15.414554+0000) 2024-02-19T19:25:16.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:16.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:16.414832+0000) 2024-02-19T19:25:17.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:17.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:17.414983+0000) 2024-02-19T19:25:18.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:18.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:18.415254+0000) 2024-02-19T19:25:19.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:19.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:19.415451+0000) 2024-02-19T19:25:20.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:20.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:20.415694+0000) 2024-02-19T19:25:21.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:21.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:21.415913+0000) 2024-02-19T19:25:22.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:22.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:22.416160+0000) 2024-02-19T19:25:23.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:23.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:23.416349+0000) 2024-02-19T19:25:24.287 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:25:24.290 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:25:24.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:24.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:24.416601+0000) 2024-02-19T19:25:25.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:25.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:25.416831+0000) 2024-02-19T19:25:26.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:26.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:26.417038+0000) 2024-02-19T19:25:27.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:27.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:27.417237+0000) 2024-02-19T19:25:28.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:28.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:28.417464+0000) 2024-02-19T19:25:29.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:29.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:29.417696+0000) 2024-02-19T19:25:30.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:30.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:30.417893+0000) 2024-02-19T19:25:31.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:31.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:31.418150+0000) 2024-02-19T19:25:32.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:32.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:32.418426+0000) 2024-02-19T19:25:33.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:33.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:33.418667+0000) 2024-02-19T19:25:34.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:34.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:34.418907+0000) 2024-02-19T19:25:35.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:35.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:35.419081+0000) 2024-02-19T19:25:36.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:36.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:36.419289+0000) 2024-02-19T19:25:37.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:37.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:37.419511+0000) 2024-02-19T19:25:38.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:38.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:38.419781+0000) 2024-02-19T19:25:39.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:39.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:39.420029+0000) 2024-02-19T19:25:40.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:40.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:40.420278+0000) 2024-02-19T19:25:41.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:41.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:41.420521+0000) 2024-02-19T19:25:42.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:42.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:42.420769+0000) 2024-02-19T19:25:43.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:43.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:43.421017+0000) 2024-02-19T19:25:44.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:44.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:44.421257+0000) 2024-02-19T19:25:45.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:45.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:45.421436+0000) 2024-02-19T19:25:46.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:46.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:46.421615+0000) 2024-02-19T19:25:47.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:47.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:47.421879+0000) 2024-02-19T19:25:48.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:48.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:48.422069+0000) 2024-02-19T19:25:49.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:49.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:49.422246+0000) 2024-02-19T19:25:50.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:50.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:50.422470+0000) 2024-02-19T19:25:51.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:51.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:51.422672+0000) 2024-02-19T19:25:52.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:52.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:52.422913+0000) 2024-02-19T19:25:53.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:53.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:53.423112+0000) 2024-02-19T19:25:54.347 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:25:54.351 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:25:54.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:54.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:54.423328+0000) 2024-02-19T19:25:55.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:55.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:55.423527+0000) 2024-02-19T19:25:56.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:56.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:56.423793+0000) 2024-02-19T19:25:57.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:57.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:57.424094+0000) 2024-02-19T19:25:58.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:58.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:58.424407+0000) 2024-02-19T19:25:59.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:25:59.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:25:59.424598+0000) 2024-02-19T19:26:00.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:00.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:00.424880+0000) 2024-02-19T19:26:01.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:01.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:01.425166+0000) 2024-02-19T19:26:02.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:02.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:02.425398+0000) 2024-02-19T19:26:03.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:03.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:03.425682+0000) 2024-02-19T19:26:04.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:04.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:04.425961+0000) 2024-02-19T19:26:05.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:05.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:05.426257+0000) 2024-02-19T19:26:06.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:06.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:06.426511+0000) 2024-02-19T19:26:07.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:07.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:07.426828+0000) 2024-02-19T19:26:08.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:08.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:08.427109+0000) 2024-02-19T19:26:09.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:09.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:09.427390+0000) 2024-02-19T19:26:10.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:10.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:10.427640+0000) 2024-02-19T19:26:11.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:11.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:11.427868+0000) 2024-02-19T19:26:12.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:12.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:12.428159+0000) 2024-02-19T19:26:13.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:13.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:13.428381+0000) 2024-02-19T19:26:14.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:14.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:14.428584+0000) 2024-02-19T19:26:15.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:15.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:15.428846+0000) 2024-02-19T19:26:16.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:16.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:16.429154+0000) 2024-02-19T19:26:17.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:17.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:17.429378+0000) 2024-02-19T19:26:18.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:18.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:18.429554+0000) 2024-02-19T19:26:19.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:19.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:19.429763+0000) 2024-02-19T19:26:20.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:20.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:20.430037+0000) 2024-02-19T19:26:21.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:21.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:21.430295+0000) 2024-02-19T19:26:22.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:22.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:22.430557+0000) 2024-02-19T19:26:23.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:23.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:23.430882+0000) 2024-02-19T19:26:24.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:24.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:24.431162+0000) 2024-02-19T19:26:24.481 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:26:24.485 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:26:25.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:25.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:25.431426+0000) 2024-02-19T19:26:26.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:26.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:26.431649+0000) 2024-02-19T19:26:27.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:27.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:27.431925+0000) 2024-02-19T19:26:28.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:28.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:28.432060+0000) 2024-02-19T19:26:29.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:29.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:29.432295+0000) 2024-02-19T19:26:30.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:30.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:30.432538+0000) 2024-02-19T19:26:31.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:31.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:31.432801+0000) 2024-02-19T19:26:32.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:32.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:32.433062+0000) 2024-02-19T19:26:33.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:33.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:33.433321+0000) 2024-02-19T19:26:34.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:34.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:34.433570+0000) 2024-02-19T19:26:35.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:35.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:35.433851+0000) 2024-02-19T19:26:36.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:36.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:36.434062+0000) 2024-02-19T19:26:37.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:37.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:37.434332+0000) 2024-02-19T19:26:38.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:38.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:38.434571+0000) 2024-02-19T19:26:39.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:39.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:39.434868+0000) 2024-02-19T19:26:40.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:40.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:40.435127+0000) 2024-02-19T19:26:41.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:41.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:41.435266+0000) 2024-02-19T19:26:42.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:42.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:42.435504+0000) 2024-02-19T19:26:43.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:43.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:43.435779+0000) 2024-02-19T19:26:44.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:44.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:44.436004+0000) 2024-02-19T19:26:45.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:45.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:45.436232+0000) 2024-02-19T19:26:46.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:46.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:46.436531+0000) 2024-02-19T19:26:47.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:47.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:47.436746+0000) 2024-02-19T19:26:48.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:48.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:48.436914+0000) 2024-02-19T19:26:49.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:49.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:49.437130+0000) 2024-02-19T19:26:50.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:50.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:50.437399+0000) 2024-02-19T19:26:51.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:51.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:51.437644+0000) 2024-02-19T19:26:52.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:52.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:52.437883+0000) 2024-02-19T19:26:53.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:53.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:53.438054+0000) 2024-02-19T19:26:54.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:54.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:54.438292+0000) 2024-02-19T19:26:54.559 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:26:54.562 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:26:55.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:55.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:55.438531+0000) 2024-02-19T19:26:56.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:56.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:56.438760+0000) 2024-02-19T19:26:57.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:57.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:57.438972+0000) 2024-02-19T19:26:58.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:58.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:58.439227+0000) 2024-02-19T19:26:59.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:26:59.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:26:59.439490+0000) 2024-02-19T19:27:00.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:00.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:00.439690+0000) 2024-02-19T19:27:01.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:01.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:01.439902+0000) 2024-02-19T19:27:02.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:02.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:02.440082+0000) 2024-02-19T19:27:03.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:03.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:03.440331+0000) 2024-02-19T19:27:04.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:04.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:04.440541+0000) 2024-02-19T19:27:05.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:05.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:05.440757+0000) 2024-02-19T19:27:06.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:06.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:06.440940+0000) 2024-02-19T19:27:07.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:07.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:07.441158+0000) 2024-02-19T19:27:08.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:08.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:08.441366+0000) 2024-02-19T19:27:09.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:09.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:09.441570+0000) 2024-02-19T19:27:10.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:10.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:10.441855+0000) 2024-02-19T19:27:11.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:11.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:11.442080+0000) 2024-02-19T19:27:12.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:12.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:12.442368+0000) 2024-02-19T19:27:13.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:13.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:13.442611+0000) 2024-02-19T19:27:14.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:14.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:14.442802+0000) 2024-02-19T19:27:15.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:15.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:15.443032+0000) 2024-02-19T19:27:16.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:16.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:16.443218+0000) 2024-02-19T19:27:17.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:17.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:17.443405+0000) 2024-02-19T19:27:18.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:18.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:18.443655+0000) 2024-02-19T19:27:19.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:19.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:19.443903+0000) 2024-02-19T19:27:20.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:20.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:20.444094+0000) 2024-02-19T19:27:21.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:21.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:21.444283+0000) 2024-02-19T19:27:22.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:22.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:22.444510+0000) 2024-02-19T19:27:23.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:23.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:23.444770+0000) 2024-02-19T19:27:24.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:24.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:24.445006+0000) 2024-02-19T19:27:24.616 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:27:24.619 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:27:25.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:25.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:25.445221+0000) 2024-02-19T19:27:26.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:26.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:26.445389+0000) 2024-02-19T19:27:27.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:27.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:27.445625+0000) 2024-02-19T19:27:28.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:28.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:28.445883+0000) 2024-02-19T19:27:29.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:29.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:29.446135+0000) 2024-02-19T19:27:30.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:30.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:30.446374+0000) 2024-02-19T19:27:31.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:31.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:31.446618+0000) 2024-02-19T19:27:32.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:32.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:32.446809+0000) 2024-02-19T19:27:33.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:33.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:33.447051+0000) 2024-02-19T19:27:34.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:34.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:34.447333+0000) 2024-02-19T19:27:35.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:35.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:35.447556+0000) 2024-02-19T19:27:36.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:36.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:36.447703+0000) 2024-02-19T19:27:37.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:37.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:37.447883+0000) 2024-02-19T19:27:38.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:38.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:38.448133+0000) 2024-02-19T19:27:39.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:39.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:39.448375+0000) 2024-02-19T19:27:40.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:40.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:40.448542+0000) 2024-02-19T19:27:41.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:41.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:41.448754+0000) 2024-02-19T19:27:42.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:42.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:42.448986+0000) 2024-02-19T19:27:43.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:43.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:43.449274+0000) 2024-02-19T19:27:44.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:44.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:44.449535+0000) 2024-02-19T19:27:45.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:45.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:45.449768+0000) 2024-02-19T19:27:46.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:46.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:46.450026+0000) 2024-02-19T19:27:47.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:47.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:47.450293+0000) 2024-02-19T19:27:48.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:48.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:48.450520+0000) 2024-02-19T19:27:49.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:49.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:49.450828+0000) 2024-02-19T19:27:50.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:50.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:50.451073+0000) 2024-02-19T19:27:51.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:51.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:51.451257+0000) 2024-02-19T19:27:52.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:52.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:52.451535+0000) 2024-02-19T19:27:53.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:53.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:53.451779+0000) 2024-02-19T19:27:54.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:54.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:54.451971+0000) 2024-02-19T19:27:54.747 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:27:54.751 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:27:55.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:55.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:55.452241+0000) 2024-02-19T19:27:56.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:56.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:56.452506+0000) 2024-02-19T19:27:57.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:57.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:57.452721+0000) 2024-02-19T19:27:58.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:58.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:58.452978+0000) 2024-02-19T19:27:59.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:27:59.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:27:59.453230+0000) 2024-02-19T19:28:00.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:00.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:00.453469+0000) 2024-02-19T19:28:01.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:01.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:01.453646+0000) 2024-02-19T19:28:02.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:02.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:02.453892+0000) 2024-02-19T19:28:03.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:03.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:03.454140+0000) 2024-02-19T19:28:04.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:04.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:04.454370+0000) 2024-02-19T19:28:05.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:05.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:05.454509+0000) 2024-02-19T19:28:06.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:06.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:06.454699+0000) 2024-02-19T19:28:07.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:07.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:07.454934+0000) 2024-02-19T19:28:08.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:08.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:08.455103+0000) 2024-02-19T19:28:09.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:09.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:09.455297+0000) 2024-02-19T19:28:10.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:10.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:10.455545+0000) 2024-02-19T19:28:11.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:11.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:11.455783+0000) 2024-02-19T19:28:12.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:12.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:12.456036+0000) 2024-02-19T19:28:13.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:13.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:13.456286+0000) 2024-02-19T19:28:14.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:14.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:14.456520+0000) 2024-02-19T19:28:15.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:15.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:15.456737+0000) 2024-02-19T19:28:16.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:16.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:16.456967+0000) 2024-02-19T19:28:17.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:17.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:17.457179+0000) 2024-02-19T19:28:18.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:18.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:18.457420+0000) 2024-02-19T19:28:19.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:19.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:19.457652+0000) 2024-02-19T19:28:20.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:20.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:20.457849+0000) 2024-02-19T19:28:21.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:21.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:21.458146+0000) 2024-02-19T19:28:22.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:22.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:22.458340+0000) 2024-02-19T19:28:23.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:23.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:23.458586+0000) 2024-02-19T19:28:24.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:24.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:24.458871+0000) 2024-02-19T19:28:24.819 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:28:24.823 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:28:25.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:25.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:25.459020+0000) 2024-02-19T19:28:26.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:26.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:26.459179+0000) 2024-02-19T19:28:27.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:27.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:27.459365+0000) 2024-02-19T19:28:28.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:28.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:28.459536+0000) 2024-02-19T19:28:29.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:29.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:29.459797+0000) 2024-02-19T19:28:30.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:30.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:30.459936+0000) 2024-02-19T19:28:31.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:31.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:31.460184+0000) 2024-02-19T19:28:32.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:32.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:32.460421+0000) 2024-02-19T19:28:33.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:33.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:33.460609+0000) 2024-02-19T19:28:34.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:34.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:34.460789+0000) 2024-02-19T19:28:35.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:35.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:35.460989+0000) 2024-02-19T19:28:36.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:36.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:36.461237+0000) 2024-02-19T19:28:37.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:37.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:37.461490+0000) 2024-02-19T19:28:38.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:38.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:38.461697+0000) 2024-02-19T19:28:39.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:39.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:39.461972+0000) 2024-02-19T19:28:40.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:40.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:40.462213+0000) 2024-02-19T19:28:41.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:41.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:41.462458+0000) 2024-02-19T19:28:42.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:42.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:42.462645+0000) 2024-02-19T19:28:43.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:43.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:43.462935+0000) 2024-02-19T19:28:44.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:44.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:44.463123+0000) 2024-02-19T19:28:45.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:45.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:45.463383+0000) 2024-02-19T19:28:46.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:46.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:46.463658+0000) 2024-02-19T19:28:47.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:47.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:47.463847+0000) 2024-02-19T19:28:48.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:48.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:48.464101+0000) 2024-02-19T19:28:49.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:49.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:49.464379+0000) 2024-02-19T19:28:50.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:50.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:50.464616+0000) 2024-02-19T19:28:51.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:51.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:51.464892+0000) 2024-02-19T19:28:52.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:52.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:52.465163+0000) 2024-02-19T19:28:53.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:53.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:53.465430+0000) 2024-02-19T19:28:54.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:54.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:54.465680+0000) 2024-02-19T19:28:54.982 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:28:54.986 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:28:55.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:55.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:55.465889+0000) 2024-02-19T19:28:56.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:56.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:56.466152+0000) 2024-02-19T19:28:57.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:57.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:57.466397+0000) 2024-02-19T19:28:58.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:58.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:58.466547+0000) 2024-02-19T19:28:59.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:28:59.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:28:59.466828+0000) 2024-02-19T19:29:00.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:00.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:00.466949+0000) 2024-02-19T19:29:01.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:01.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:01.467097+0000) 2024-02-19T19:29:02.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:02.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:02.467344+0000) 2024-02-19T19:29:03.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:03.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:03.467594+0000) 2024-02-19T19:29:04.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:04.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:04.467833+0000) 2024-02-19T19:29:05.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:05.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:05.468070+0000) 2024-02-19T19:29:06.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:06.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:06.468245+0000) 2024-02-19T19:29:07.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:07.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:07.468481+0000) 2024-02-19T19:29:08.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:08.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:08.468668+0000) 2024-02-19T19:29:09.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:09.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:09.468853+0000) 2024-02-19T19:29:10.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:10.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:10.469104+0000) 2024-02-19T19:29:11.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:11.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:11.469356+0000) 2024-02-19T19:29:12.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:12.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:12.469610+0000) 2024-02-19T19:29:13.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:13.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:13.469840+0000) 2024-02-19T19:29:14.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:14.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:14.470096+0000) 2024-02-19T19:29:15.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:15.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:15.470337+0000) 2024-02-19T19:29:16.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:16.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:16.470585+0000) 2024-02-19T19:29:17.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:17.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:17.470772+0000) 2024-02-19T19:29:18.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:18.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:18.470995+0000) 2024-02-19T19:29:19.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:19.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:19.471189+0000) 2024-02-19T19:29:20.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:20.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:20.471469+0000) 2024-02-19T19:29:21.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:21.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:21.471717+0000) 2024-02-19T19:29:22.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:22.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:22.471899+0000) 2024-02-19T19:29:23.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:23.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:23.472127+0000) 2024-02-19T19:29:24.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:24.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:24.472321+0000) 2024-02-19T19:29:25.030 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:29:25.034 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:29:25.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:25.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:25.472562+0000) 2024-02-19T19:29:26.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:26.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:26.472810+0000) 2024-02-19T19:29:27.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:27.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:27.473109+0000) 2024-02-19T19:29:28.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:28.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:28.473319+0000) 2024-02-19T19:29:29.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:29.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:29.473559+0000) 2024-02-19T19:29:30.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:30.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:30.473736+0000) 2024-02-19T19:29:31.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:31.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:31.473933+0000) 2024-02-19T19:29:32.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:32.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:32.474181+0000) 2024-02-19T19:29:33.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:33.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:33.474348+0000) 2024-02-19T19:29:34.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:34.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:34.474528+0000) 2024-02-19T19:29:35.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:35.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:35.474788+0000) 2024-02-19T19:29:36.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:36.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:36.474917+0000) 2024-02-19T19:29:37.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:37.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:37.475121+0000) 2024-02-19T19:29:38.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:38.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:38.475359+0000) 2024-02-19T19:29:39.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:39.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:39.475595+0000) 2024-02-19T19:29:40.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:40.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:40.475850+0000) 2024-02-19T19:29:41.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:41.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:41.476093+0000) 2024-02-19T19:29:42.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:42.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:42.476334+0000) 2024-02-19T19:29:43.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:43.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:43.476573+0000) 2024-02-19T19:29:44.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:44.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:44.476832+0000) 2024-02-19T19:29:45.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:45.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:45.477083+0000) 2024-02-19T19:29:46.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:46.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:46.477320+0000) 2024-02-19T19:29:47.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:47.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:47.477503+0000) 2024-02-19T19:29:48.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:48.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:48.477682+0000) 2024-02-19T19:29:49.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:49.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:49.477874+0000) 2024-02-19T19:29:50.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:50.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:50.478130+0000) 2024-02-19T19:29:51.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:51.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:51.478410+0000) 2024-02-19T19:29:52.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:52.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:52.478692+0000) 2024-02-19T19:29:53.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:53.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:53.478928+0000) 2024-02-19T19:29:54.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:54.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:54.479210+0000) 2024-02-19T19:29:55.086 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:29:55.089 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:29:55.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:55.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:55.479467+0000) 2024-02-19T19:29:56.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:56.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:56.479662+0000) 2024-02-19T19:29:57.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:57.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:57.479913+0000) 2024-02-19T19:29:58.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:58.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:58.480185+0000) 2024-02-19T19:29:59.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:29:59.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:29:59.480361+0000) 2024-02-19T19:30:00.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:00.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:00.480554+0000) 2024-02-19T19:30:01.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:01.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:01.480727+0000) 2024-02-19T19:30:02.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:02.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:02.480937+0000) 2024-02-19T19:30:03.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:03.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:03.481139+0000) 2024-02-19T19:30:04.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:04.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:04.481378+0000) 2024-02-19T19:30:05.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:05.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:05.481557+0000) 2024-02-19T19:30:06.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:06.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:06.481762+0000) 2024-02-19T19:30:07.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:07.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:07.482043+0000) 2024-02-19T19:30:08.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:08.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:08.482308+0000) 2024-02-19T19:30:09.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:09.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:09.482583+0000) 2024-02-19T19:30:10.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:10.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:10.482760+0000) 2024-02-19T19:30:11.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:11.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:11.482991+0000) 2024-02-19T19:30:12.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:12.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:12.483273+0000) 2024-02-19T19:30:13.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:13.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:13.483503+0000) 2024-02-19T19:30:14.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:14.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:14.483746+0000) 2024-02-19T19:30:15.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:15.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:15.483957+0000) 2024-02-19T19:30:16.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:16.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:16.484221+0000) 2024-02-19T19:30:17.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:17.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:17.484488+0000) 2024-02-19T19:30:18.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:18.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:18.484668+0000) 2024-02-19T19:30:19.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:19.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:19.484857+0000) 2024-02-19T19:30:20.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:20.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:20.485102+0000) 2024-02-19T19:30:21.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:21.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:21.485298+0000) 2024-02-19T19:30:22.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:22.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:22.485551+0000) 2024-02-19T19:30:23.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:23.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:23.485773+0000) 2024-02-19T19:30:24.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:24.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:24.485982+0000) 2024-02-19T19:30:25.223 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:30:25.227 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:30:25.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:25.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:25.486188+0000) 2024-02-19T19:30:26.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:26.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:26.486440+0000) 2024-02-19T19:30:27.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:27.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:27.486698+0000) 2024-02-19T19:30:28.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:28.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:28.486987+0000) 2024-02-19T19:30:29.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:29.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:29.487179+0000) 2024-02-19T19:30:30.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:30.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:30.487456+0000) 2024-02-19T19:30:31.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:31.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:31.487641+0000) 2024-02-19T19:30:32.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:32.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:32.487923+0000) 2024-02-19T19:30:33.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:33.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:33.488143+0000) 2024-02-19T19:30:34.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:34.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:34.488331+0000) 2024-02-19T19:30:35.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:35.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:35.488569+0000) 2024-02-19T19:30:36.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:36.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:36.488826+0000) 2024-02-19T19:30:37.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:37.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:37.489080+0000) 2024-02-19T19:30:38.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:38.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:38.489318+0000) 2024-02-19T19:30:39.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:39.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:39.489566+0000) 2024-02-19T19:30:40.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:40.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:40.489771+0000) 2024-02-19T19:30:41.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:41.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:41.490021+0000) 2024-02-19T19:30:42.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:42.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:42.490260+0000) 2024-02-19T19:30:43.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:43.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:43.490444+0000) 2024-02-19T19:30:44.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:44.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:44.490670+0000) 2024-02-19T19:30:45.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:45.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:45.490910+0000) 2024-02-19T19:30:46.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:46.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:46.491145+0000) 2024-02-19T19:30:47.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:47.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:47.491396+0000) 2024-02-19T19:30:48.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:48.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:48.491619+0000) 2024-02-19T19:30:49.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:49.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:49.491887+0000) 2024-02-19T19:30:50.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:50.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:50.492123+0000) 2024-02-19T19:30:51.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:51.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:51.492360+0000) 2024-02-19T19:30:52.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:52.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:52.492545+0000) 2024-02-19T19:30:53.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:53.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:53.492786+0000) 2024-02-19T19:30:54.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:54.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:54.493060+0000) 2024-02-19T19:30:55.353 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:30:55.356 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:30:55.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:55.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:55.493326+0000) 2024-02-19T19:30:56.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:56.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:56.493502+0000) 2024-02-19T19:30:57.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:57.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:57.493728+0000) 2024-02-19T19:30:58.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:58.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:58.493986+0000) 2024-02-19T19:30:59.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:30:59.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:30:59.494217+0000) 2024-02-19T19:31:00.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:00.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:00.494455+0000) 2024-02-19T19:31:01.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:01.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:01.494744+0000) 2024-02-19T19:31:02.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:02.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:02.494927+0000) 2024-02-19T19:31:03.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:03.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:03.495131+0000) 2024-02-19T19:31:04.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:04.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:04.495404+0000) 2024-02-19T19:31:05.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:05.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:05.495664+0000) 2024-02-19T19:31:06.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:06.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:06.495846+0000) 2024-02-19T19:31:07.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:07.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:07.496069+0000) 2024-02-19T19:31:08.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:08.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:08.496212+0000) 2024-02-19T19:31:09.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:09.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:09.496449+0000) 2024-02-19T19:31:10.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:10.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:10.496633+0000) 2024-02-19T19:31:11.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:11.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:11.496811+0000) 2024-02-19T19:31:12.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:12.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:12.497002+0000) 2024-02-19T19:31:13.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:13.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:13.497245+0000) 2024-02-19T19:31:14.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:14.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:14.497455+0000) 2024-02-19T19:31:15.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:15.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:15.497729+0000) 2024-02-19T19:31:16.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:16.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:16.498021+0000) 2024-02-19T19:31:17.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:17.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:17.498269+0000) 2024-02-19T19:31:18.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:18.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:18.498468+0000) 2024-02-19T19:31:19.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:19.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:19.498743+0000) 2024-02-19T19:31:20.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:20.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:20.498965+0000) 2024-02-19T19:31:21.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:21.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:21.499124+0000) 2024-02-19T19:31:22.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:22.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:22.499301+0000) 2024-02-19T19:31:23.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:23.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:23.499491+0000) 2024-02-19T19:31:24.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:24.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:24.499770+0000) 2024-02-19T19:31:25.491 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:31:25.495 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:31:25.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:25.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:25.499940+0000) 2024-02-19T19:31:26.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:26.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:26.500114+0000) 2024-02-19T19:31:27.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:27.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:27.500303+0000) 2024-02-19T19:31:28.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:28.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:28.500468+0000) 2024-02-19T19:31:29.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:29.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:29.500665+0000) 2024-02-19T19:31:30.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:30.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:30.500885+0000) 2024-02-19T19:31:31.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:31.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:31.501164+0000) 2024-02-19T19:31:32.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:32.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:32.501411+0000) 2024-02-19T19:31:33.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:33.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:33.501599+0000) 2024-02-19T19:31:34.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:34.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:34.501771+0000) 2024-02-19T19:31:35.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:35.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:35.501981+0000) 2024-02-19T19:31:36.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:36.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:36.502239+0000) 2024-02-19T19:31:37.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:37.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:37.502483+0000) 2024-02-19T19:31:38.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:38.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:38.502662+0000) 2024-02-19T19:31:39.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:39.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:39.502808+0000) 2024-02-19T19:31:40.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:40.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:40.503010+0000) 2024-02-19T19:31:41.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:41.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:41.503169+0000) 2024-02-19T19:31:42.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:42.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:42.503309+0000) 2024-02-19T19:31:43.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:43.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:43.503596+0000) 2024-02-19T19:31:44.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:44.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:44.503737+0000) 2024-02-19T19:31:45.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:45.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:45.503921+0000) 2024-02-19T19:31:46.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:46.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:46.504056+0000) 2024-02-19T19:31:47.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:47.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:47.504194+0000) 2024-02-19T19:31:48.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:48.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:48.504321+0000) 2024-02-19T19:31:49.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:49.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:49.504475+0000) 2024-02-19T19:31:50.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:50.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:50.504609+0000) 2024-02-19T19:31:51.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:51.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:51.504754+0000) 2024-02-19T19:31:52.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:52.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:52.504888+0000) 2024-02-19T19:31:53.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:53.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:53.505009+0000) 2024-02-19T19:31:54.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:54.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:54.505163+0000) 2024-02-19T19:31:55.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:55.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:55.505406+0000) 2024-02-19T19:31:55.567 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:31:55.571 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:31:56.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:56.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:56.505641+0000) 2024-02-19T19:31:57.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:57.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:57.505884+0000) 2024-02-19T19:31:58.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:58.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:58.506130+0000) 2024-02-19T19:31:59.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:31:59.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:31:59.506379+0000) 2024-02-19T19:32:00.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:00.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:00.506570+0000) 2024-02-19T19:32:01.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:01.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:01.506850+0000) 2024-02-19T19:32:02.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:02.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:02.507038+0000) 2024-02-19T19:32:03.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:03.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:03.507241+0000) 2024-02-19T19:32:04.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:04.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:04.507425+0000) 2024-02-19T19:32:05.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:05.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:05.507615+0000) 2024-02-19T19:32:06.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:06.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:06.507852+0000) 2024-02-19T19:32:07.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:07.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:07.508101+0000) 2024-02-19T19:32:08.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:08.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:08.508330+0000) 2024-02-19T19:32:09.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:09.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:09.508521+0000) 2024-02-19T19:32:10.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:10.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:10.508722+0000) 2024-02-19T19:32:11.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:11.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:11.508898+0000) 2024-02-19T19:32:12.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:12.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:12.509128+0000) 2024-02-19T19:32:13.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:13.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:13.509358+0000) 2024-02-19T19:32:14.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:14.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:14.509599+0000) 2024-02-19T19:32:15.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:15.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:15.509841+0000) 2024-02-19T19:32:16.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:16.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:16.510091+0000) 2024-02-19T19:32:17.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:17.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:17.510308+0000) 2024-02-19T19:32:18.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:18.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:18.510553+0000) 2024-02-19T19:32:19.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:19.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:19.510786+0000) 2024-02-19T19:32:20.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:20.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:20.510970+0000) 2024-02-19T19:32:21.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:21.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:21.511167+0000) 2024-02-19T19:32:22.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:22.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:22.511387+0000) 2024-02-19T19:32:23.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:23.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:23.511644+0000) 2024-02-19T19:32:24.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:24.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:24.511858+0000) 2024-02-19T19:32:25.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:25.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:25.512094+0000) 2024-02-19T19:32:25.687 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:32:25.691 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:32:26.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:26.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:26.512279+0000) 2024-02-19T19:32:27.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:27.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:27.512530+0000) 2024-02-19T19:32:28.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:28.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:28.512765+0000) 2024-02-19T19:32:29.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:29.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:29.512942+0000) 2024-02-19T19:32:30.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:30.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:30.513184+0000) 2024-02-19T19:32:31.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:31.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:31.513460+0000) 2024-02-19T19:32:32.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:32.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:32.513675+0000) 2024-02-19T19:32:33.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:33.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:33.513872+0000) 2024-02-19T19:32:34.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:34.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:34.514125+0000) 2024-02-19T19:32:35.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:35.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:35.514402+0000) 2024-02-19T19:32:36.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:36.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:36.514658+0000) 2024-02-19T19:32:37.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:37.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:37.514939+0000) 2024-02-19T19:32:38.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:38.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:38.515130+0000) 2024-02-19T19:32:39.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:39.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:39.515389+0000) 2024-02-19T19:32:40.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:40.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:40.515567+0000) 2024-02-19T19:32:41.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:41.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:41.515835+0000) 2024-02-19T19:32:42.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:42.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:42.516068+0000) 2024-02-19T19:32:43.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:43.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:43.516242+0000) 2024-02-19T19:32:44.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:44.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:44.516490+0000) 2024-02-19T19:32:45.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:45.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:45.516750+0000) 2024-02-19T19:32:46.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:46.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:46.516941+0000) 2024-02-19T19:32:47.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:47.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:47.517183+0000) 2024-02-19T19:32:48.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:48.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:48.517375+0000) 2024-02-19T19:32:49.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:49.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:49.517566+0000) 2024-02-19T19:32:50.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:50.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:50.517794+0000) 2024-02-19T19:32:51.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:51.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:51.517992+0000) 2024-02-19T19:32:52.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:52.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:52.518230+0000) 2024-02-19T19:32:53.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:53.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:53.518473+0000) 2024-02-19T19:32:54.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:54.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:54.518752+0000) 2024-02-19T19:32:55.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:55.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:55.518937+0000) 2024-02-19T19:32:55.833 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:32:55.836 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:32:56.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:56.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:56.519214+0000) 2024-02-19T19:32:57.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:57.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:57.519496+0000) 2024-02-19T19:32:58.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:58.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:58.519680+0000) 2024-02-19T19:32:59.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:32:59.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:32:59.519957+0000) 2024-02-19T19:33:00.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:00.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:00.520176+0000) 2024-02-19T19:33:01.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:01.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:01.520416+0000) 2024-02-19T19:33:02.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:02.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:02.520662+0000) 2024-02-19T19:33:03.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:03.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:03.520929+0000) 2024-02-19T19:33:04.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:04.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:04.521174+0000) 2024-02-19T19:33:05.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:05.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:05.521341+0000) 2024-02-19T19:33:06.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:06.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:06.521615+0000) 2024-02-19T19:33:07.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:07.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:07.521829+0000) 2024-02-19T19:33:08.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:08.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:08.522053+0000) 2024-02-19T19:33:09.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:09.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:09.522291+0000) 2024-02-19T19:33:10.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:10.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:10.522532+0000) 2024-02-19T19:33:11.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:11.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:11.522827+0000) 2024-02-19T19:33:12.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:12.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:12.523074+0000) 2024-02-19T19:33:13.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:13.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:13.523330+0000) 2024-02-19T19:33:14.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:14.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:14.523582+0000) 2024-02-19T19:33:15.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:15.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:15.523845+0000) 2024-02-19T19:33:16.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:16.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:16.524125+0000) 2024-02-19T19:33:17.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:17.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:17.524391+0000) 2024-02-19T19:33:18.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:18.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:18.524634+0000) 2024-02-19T19:33:19.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:19.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:19.524806+0000) 2024-02-19T19:33:20.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:20.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:20.525047+0000) 2024-02-19T19:33:21.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:21.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:21.525263+0000) 2024-02-19T19:33:22.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:22.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:22.525416+0000) 2024-02-19T19:33:23.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:23.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:23.525656+0000) 2024-02-19T19:33:24.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:24.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:24.525905+0000) 2024-02-19T19:33:25.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:25.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:25.526120+0000) 2024-02-19T19:33:25.904 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:33:25.908 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:33:26.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:26.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:26.526350+0000) 2024-02-19T19:33:27.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:27.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:27.526614+0000) 2024-02-19T19:33:28.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:28.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:28.526836+0000) 2024-02-19T19:33:29.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:29.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:29.527018+0000) 2024-02-19T19:33:30.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:30.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:30.527221+0000) 2024-02-19T19:33:31.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:31.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:31.527414+0000) 2024-02-19T19:33:32.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:32.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:32.527669+0000) 2024-02-19T19:33:33.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:33.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:33.527865+0000) 2024-02-19T19:33:34.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:34.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:34.528087+0000) 2024-02-19T19:33:35.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:35.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:35.528289+0000) 2024-02-19T19:33:36.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:36.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:36.528471+0000) 2024-02-19T19:33:37.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:37.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:37.528724+0000) 2024-02-19T19:33:38.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:38.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:38.528918+0000) 2024-02-19T19:33:39.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:39.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:39.529175+0000) 2024-02-19T19:33:40.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:40.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:40.529369+0000) 2024-02-19T19:33:41.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:41.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:41.529615+0000) 2024-02-19T19:33:42.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:42.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:42.529898+0000) 2024-02-19T19:33:43.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:43.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:43.530148+0000) 2024-02-19T19:33:44.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:44.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:44.530315+0000) 2024-02-19T19:33:45.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:45.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:45.530583+0000) 2024-02-19T19:33:46.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:46.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:46.530841+0000) 2024-02-19T19:33:47.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:47.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:47.530996+0000) 2024-02-19T19:33:48.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:48.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:48.531188+0000) 2024-02-19T19:33:49.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:49.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:49.531385+0000) 2024-02-19T19:33:50.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:50.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:50.531638+0000) 2024-02-19T19:33:51.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:51.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:51.531896+0000) 2024-02-19T19:33:52.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:52.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:52.532152+0000) 2024-02-19T19:33:53.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:53.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:53.532416+0000) 2024-02-19T19:33:54.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:54.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:54.532678+0000) 2024-02-19T19:33:55.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:55.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:55.532857+0000) 2024-02-19T19:33:56.068 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:33:56.071 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:33:56.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:56.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:56.533042+0000) 2024-02-19T19:33:57.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:57.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:57.533302+0000) 2024-02-19T19:33:58.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:58.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:58.533541+0000) 2024-02-19T19:33:59.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:33:59.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:33:59.533788+0000) 2024-02-19T19:34:00.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:00.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:00.533980+0000) 2024-02-19T19:34:01.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:01.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:01.534230+0000) 2024-02-19T19:34:02.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:02.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:02.534476+0000) 2024-02-19T19:34:03.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:03.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:03.534645+0000) 2024-02-19T19:34:04.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:04.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:04.534900+0000) 2024-02-19T19:34:05.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:05.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:05.535139+0000) 2024-02-19T19:34:06.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:06.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:06.535391+0000) 2024-02-19T19:34:07.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:07.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:07.535605+0000) 2024-02-19T19:34:08.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:08.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:08.535799+0000) 2024-02-19T19:34:09.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:09.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:09.535985+0000) 2024-02-19T19:34:10.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:10.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:10.536230+0000) 2024-02-19T19:34:11.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:11.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:11.536416+0000) 2024-02-19T19:34:12.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:12.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:12.536597+0000) 2024-02-19T19:34:13.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:13.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:13.536866+0000) 2024-02-19T19:34:14.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:14.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:14.537055+0000) 2024-02-19T19:34:15.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:15.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:15.537258+0000) 2024-02-19T19:34:16.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:16.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:16.537497+0000) 2024-02-19T19:34:17.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:17.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:17.537761+0000) 2024-02-19T19:34:18.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:18.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:18.537946+0000) 2024-02-19T19:34:19.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:19.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:19.538124+0000) 2024-02-19T19:34:20.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:20.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:20.538278+0000) 2024-02-19T19:34:21.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:21.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:21.538459+0000) 2024-02-19T19:34:22.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:22.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:22.538701+0000) 2024-02-19T19:34:23.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:23.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:23.538946+0000) 2024-02-19T19:34:24.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:24.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:24.539092+0000) 2024-02-19T19:34:25.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:25.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:25.539274+0000) 2024-02-19T19:34:26.141 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:34:26.144 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:34:26.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:26.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:26.539538+0000) 2024-02-19T19:34:27.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:27.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:27.539788+0000) 2024-02-19T19:34:28.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:28.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:28.539997+0000) 2024-02-19T19:34:29.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:29.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:29.540258+0000) 2024-02-19T19:34:30.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:30.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:30.540435+0000) 2024-02-19T19:34:31.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:31.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:31.540664+0000) 2024-02-19T19:34:32.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:32.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:32.540917+0000) 2024-02-19T19:34:33.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:33.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:33.541166+0000) 2024-02-19T19:34:34.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:34.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:34.541388+0000) 2024-02-19T19:34:35.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:35.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:35.541601+0000) 2024-02-19T19:34:36.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:36.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:36.541843+0000) 2024-02-19T19:34:37.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:37.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:37.542103+0000) 2024-02-19T19:34:38.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:38.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:38.542278+0000) 2024-02-19T19:34:39.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:39.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:39.542529+0000) 2024-02-19T19:34:40.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:40.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:40.542832+0000) 2024-02-19T19:34:41.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:41.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:41.543023+0000) 2024-02-19T19:34:42.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:42.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:42.543210+0000) 2024-02-19T19:34:43.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:43.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:43.543448+0000) 2024-02-19T19:34:44.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:44.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:44.543687+0000) 2024-02-19T19:34:45.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:45.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:45.543927+0000) 2024-02-19T19:34:46.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:46.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:46.544142+0000) 2024-02-19T19:34:47.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:47.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:47.544354+0000) 2024-02-19T19:34:48.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:48.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:48.544625+0000) 2024-02-19T19:34:49.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:49.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:49.544830+0000) 2024-02-19T19:34:50.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:50.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:50.545080+0000) 2024-02-19T19:34:51.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:51.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:51.545251+0000) 2024-02-19T19:34:52.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:52.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:52.545499+0000) 2024-02-19T19:34:53.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:53.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:53.545747+0000) 2024-02-19T19:34:54.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:54.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:54.545900+0000) 2024-02-19T19:34:55.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:55.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:55.546129+0000) 2024-02-19T19:34:56.270 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:34:56.274 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:34:56.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:56.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:56.546381+0000) 2024-02-19T19:34:57.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:57.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:57.546623+0000) 2024-02-19T19:34:58.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:58.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:58.546859+0000) 2024-02-19T19:34:59.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:34:59.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:34:59.547069+0000) 2024-02-19T19:35:00.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:00.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:00.547269+0000) 2024-02-19T19:35:01.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:01.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:01.547408+0000) 2024-02-19T19:35:02.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:02.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:02.547675+0000) 2024-02-19T19:35:03.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:03.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:03.547923+0000) 2024-02-19T19:35:04.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:04.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:04.548194+0000) 2024-02-19T19:35:05.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:05.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:05.548428+0000) 2024-02-19T19:35:06.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:06.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:06.548701+0000) 2024-02-19T19:35:07.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:07.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:07.548865+0000) 2024-02-19T19:35:08.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:08.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:08.549049+0000) 2024-02-19T19:35:09.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:09.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:09.549310+0000) 2024-02-19T19:35:10.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:10.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:10.549566+0000) 2024-02-19T19:35:11.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:11.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:11.549792+0000) 2024-02-19T19:35:12.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:12.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:12.550053+0000) 2024-02-19T19:35:13.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:13.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:13.550224+0000) 2024-02-19T19:35:14.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:14.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:14.550415+0000) 2024-02-19T19:35:15.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:15.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:15.550637+0000) 2024-02-19T19:35:16.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:16.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:16.550873+0000) 2024-02-19T19:35:17.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:17.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:17.551132+0000) 2024-02-19T19:35:18.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:18.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:18.551386+0000) 2024-02-19T19:35:19.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:19.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:19.551588+0000) 2024-02-19T19:35:20.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:20.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:20.551837+0000) 2024-02-19T19:35:21.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:21.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:21.552110+0000) 2024-02-19T19:35:22.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:22.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:22.552357+0000) 2024-02-19T19:35:23.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:23.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:23.552597+0000) 2024-02-19T19:35:24.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:24.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:24.552764+0000) 2024-02-19T19:35:25.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:25.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:25.553012+0000) 2024-02-19T19:35:26.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:26.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:26.553197+0000) 2024-02-19T19:35:26.560 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:35:26.563 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:35:27.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:27.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:27.553482+0000) 2024-02-19T19:35:28.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:28.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:28.553743+0000) 2024-02-19T19:35:29.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:29.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:29.553995+0000) 2024-02-19T19:35:30.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:30.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:30.554212+0000) 2024-02-19T19:35:31.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:31.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:31.554442+0000) 2024-02-19T19:35:32.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:32.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:32.554687+0000) 2024-02-19T19:35:33.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:33.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:33.554983+0000) 2024-02-19T19:35:34.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:34.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:34.555171+0000) 2024-02-19T19:35:35.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:35.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:35.555437+0000) 2024-02-19T19:35:36.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:36.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:36.555657+0000) 2024-02-19T19:35:37.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:37.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:37.555901+0000) 2024-02-19T19:35:38.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:38.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:38.556074+0000) 2024-02-19T19:35:39.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:39.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:39.556329+0000) 2024-02-19T19:35:40.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:40.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:40.556575+0000) 2024-02-19T19:35:41.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:41.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:41.556846+0000) 2024-02-19T19:35:42.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:42.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:42.557038+0000) 2024-02-19T19:35:43.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:43.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:43.557263+0000) 2024-02-19T19:35:44.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:44.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:44.557503+0000) 2024-02-19T19:35:45.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:45.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:45.557740+0000) 2024-02-19T19:35:46.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:46.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:46.557989+0000) 2024-02-19T19:35:47.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:47.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:47.558271+0000) 2024-02-19T19:35:48.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:48.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:48.558465+0000) 2024-02-19T19:35:49.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:49.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:49.558656+0000) 2024-02-19T19:35:50.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:50.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:50.558857+0000) 2024-02-19T19:35:51.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:51.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:51.559117+0000) 2024-02-19T19:35:52.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:52.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:52.559322+0000) 2024-02-19T19:35:53.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:53.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:53.559597+0000) 2024-02-19T19:35:54.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:54.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:54.559843+0000) 2024-02-19T19:35:55.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:55.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:55.560108+0000) 2024-02-19T19:35:56.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:56.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:56.560344+0000) 2024-02-19T19:35:56.681 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:35:56.685 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:35:57.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:57.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:57.560566+0000) 2024-02-19T19:35:58.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:58.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:58.560812+0000) 2024-02-19T19:35:59.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:35:59.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:35:59.561040+0000) 2024-02-19T19:36:00.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:00.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:00.561283+0000) 2024-02-19T19:36:01.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:01.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:01.561548+0000) 2024-02-19T19:36:02.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:02.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:02.561755+0000) 2024-02-19T19:36:03.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:03.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:03.561999+0000) 2024-02-19T19:36:04.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:04.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:04.562248+0000) 2024-02-19T19:36:05.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:05.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:05.562483+0000) 2024-02-19T19:36:06.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:06.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:06.562809+0000) 2024-02-19T19:36:07.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:07.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:07.563050+0000) 2024-02-19T19:36:08.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:08.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:08.563220+0000) 2024-02-19T19:36:09.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:09.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:09.563415+0000) 2024-02-19T19:36:10.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:10.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:10.563651+0000) 2024-02-19T19:36:11.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:11.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:11.563888+0000) 2024-02-19T19:36:12.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:12.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:12.564110+0000) 2024-02-19T19:36:13.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:13.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:13.564352+0000) 2024-02-19T19:36:14.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:14.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:14.564575+0000) 2024-02-19T19:36:15.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:15.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:15.564826+0000) 2024-02-19T19:36:16.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:16.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:16.565068+0000) 2024-02-19T19:36:17.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:17.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:17.565318+0000) 2024-02-19T19:36:18.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:18.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:18.565555+0000) 2024-02-19T19:36:19.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:19.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:19.565786+0000) 2024-02-19T19:36:20.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:20.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:20.565976+0000) 2024-02-19T19:36:21.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:21.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:21.566217+0000) 2024-02-19T19:36:22.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:22.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:22.566443+0000) 2024-02-19T19:36:23.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:23.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:23.566640+0000) 2024-02-19T19:36:24.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:24.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:24.566800+0000) 2024-02-19T19:36:25.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:25.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:25.566977+0000) 2024-02-19T19:36:26.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:26.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:26.567199+0000) 2024-02-19T19:36:26.812 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:36:26.816 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:36:27.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:27.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:27.567465+0000) 2024-02-19T19:36:28.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:28.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:28.567718+0000) 2024-02-19T19:36:29.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:29.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:29.567947+0000) 2024-02-19T19:36:30.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:30.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:30.568193+0000) 2024-02-19T19:36:31.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:31.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:31.568373+0000) 2024-02-19T19:36:32.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:32.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:32.568574+0000) 2024-02-19T19:36:33.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:33.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:33.568787+0000) 2024-02-19T19:36:34.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:34.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:34.568949+0000) 2024-02-19T19:36:35.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:35.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:35.569180+0000) 2024-02-19T19:36:36.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:36.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:36.569449+0000) 2024-02-19T19:36:37.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:37.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:37.569690+0000) 2024-02-19T19:36:38.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:38.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:38.569931+0000) 2024-02-19T19:36:39.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:39.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:39.570178+0000) 2024-02-19T19:36:40.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:40.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:40.570371+0000) 2024-02-19T19:36:41.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:41.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:41.570591+0000) 2024-02-19T19:36:42.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:42.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:42.570776+0000) 2024-02-19T19:36:43.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:43.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:43.570972+0000) 2024-02-19T19:36:44.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:44.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:44.571161+0000) 2024-02-19T19:36:45.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:45.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:45.571393+0000) 2024-02-19T19:36:46.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:46.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:46.571646+0000) 2024-02-19T19:36:47.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:47.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:47.571909+0000) 2024-02-19T19:36:48.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:48.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:48.572157+0000) 2024-02-19T19:36:49.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:49.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:49.572352+0000) 2024-02-19T19:36:50.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:50.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:50.572620+0000) 2024-02-19T19:36:51.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:51.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:51.572847+0000) 2024-02-19T19:36:52.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:52.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:52.573109+0000) 2024-02-19T19:36:53.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:53.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:53.573370+0000) 2024-02-19T19:36:54.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:54.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:54.573544+0000) 2024-02-19T19:36:55.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:55.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:55.573767+0000) 2024-02-19T19:36:56.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:56.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:56.573962+0000) 2024-02-19T19:36:56.942 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:36:56.946 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:36:57.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:57.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:57.574155+0000) 2024-02-19T19:36:58.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:58.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:58.574323+0000) 2024-02-19T19:36:59.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:36:59.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:36:59.574505+0000) 2024-02-19T19:37:00.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:00.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:00.574810+0000) 2024-02-19T19:37:01.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:01.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:01.574929+0000) 2024-02-19T19:37:02.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:02.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:02.575098+0000) 2024-02-19T19:37:03.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:03.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:03.575355+0000) 2024-02-19T19:37:04.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:04.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:04.575632+0000) 2024-02-19T19:37:05.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:05.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:05.575817+0000) 2024-02-19T19:37:06.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:06.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:06.576010+0000) 2024-02-19T19:37:07.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:07.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:07.576229+0000) 2024-02-19T19:37:08.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:08.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:08.576463+0000) 2024-02-19T19:37:09.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:09.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:09.576711+0000) 2024-02-19T19:37:10.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:10.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:10.576990+0000) 2024-02-19T19:37:11.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:11.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:11.577201+0000) 2024-02-19T19:37:12.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:12.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:12.577391+0000) 2024-02-19T19:37:13.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:13.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:13.577579+0000) 2024-02-19T19:37:14.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:14.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:14.577851+0000) 2024-02-19T19:37:15.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:15.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:15.578041+0000) 2024-02-19T19:37:16.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:16.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:16.578240+0000) 2024-02-19T19:37:17.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:17.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:17.578388+0000) 2024-02-19T19:37:18.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:18.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:18.578649+0000) 2024-02-19T19:37:19.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:19.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:19.578895+0000) 2024-02-19T19:37:20.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:20.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:20.579077+0000) 2024-02-19T19:37:21.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:21.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:21.579209+0000) 2024-02-19T19:37:22.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:22.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:22.579467+0000) 2024-02-19T19:37:23.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:23.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:23.579650+0000) 2024-02-19T19:37:24.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:24.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:24.579835+0000) 2024-02-19T19:37:25.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:25.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:25.580057+0000) 2024-02-19T19:37:26.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:26.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:26.580290+0000) 2024-02-19T19:37:27.073 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:37:27.076 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:37:27.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:27.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:27.580491+0000) 2024-02-19T19:37:28.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:28.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:28.580742+0000) 2024-02-19T19:37:29.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:29.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:29.580926+0000) 2024-02-19T19:37:30.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:30.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:30.581159+0000) 2024-02-19T19:37:31.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:31.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:31.581418+0000) 2024-02-19T19:37:32.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:32.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:32.581663+0000) 2024-02-19T19:37:33.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:33.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:33.581896+0000) 2024-02-19T19:37:34.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:34.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:34.582171+0000) 2024-02-19T19:37:35.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:35.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:35.582357+0000) 2024-02-19T19:37:36.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:36.575+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:36.582588+0000) 2024-02-19T19:37:37.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:37.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:37.582768+0000) 2024-02-19T19:37:38.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:38.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:38.582953+0000) 2024-02-19T19:37:39.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:39.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:39.583190+0000) 2024-02-19T19:37:40.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:40.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:40.583338+0000) 2024-02-19T19:37:41.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:41.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:41.583561+0000) 2024-02-19T19:37:42.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:42.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:42.583752+0000) 2024-02-19T19:37:43.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:43.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:43.583959+0000) 2024-02-19T19:37:44.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:44.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:44.584212+0000) 2024-02-19T19:37:45.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:45.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:45.584451+0000) 2024-02-19T19:37:46.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:46.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:46.584657+0000) 2024-02-19T19:37:47.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:47.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:47.584842+0000) 2024-02-19T19:37:48.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:48.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:48.585050+0000) 2024-02-19T19:37:49.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:49.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:49.585243+0000) 2024-02-19T19:37:50.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:50.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:50.585470+0000) 2024-02-19T19:37:51.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:51.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:51.585705+0000) 2024-02-19T19:37:52.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:52.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:52.585962+0000) 2024-02-19T19:37:53.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:53.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:53.586192+0000) 2024-02-19T19:37:54.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:54.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:54.586430+0000) 2024-02-19T19:37:55.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:55.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:55.586627+0000) 2024-02-19T19:37:56.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:56.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:56.586923+0000) 2024-02-19T19:37:57.220 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:37:57.223 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:37:57.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:57.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:57.587170+0000) 2024-02-19T19:37:58.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:58.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:58.587424+0000) 2024-02-19T19:37:59.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:37:59.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:37:59.587622+0000) 2024-02-19T19:38:00.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:00.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:00.587820+0000) 2024-02-19T19:38:01.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:01.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:01.588046+0000) 2024-02-19T19:38:02.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:02.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:02.588293+0000) 2024-02-19T19:38:03.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:03.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:03.588545+0000) 2024-02-19T19:38:04.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:04.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:04.588813+0000) 2024-02-19T19:38:05.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:05.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:05.589069+0000) 2024-02-19T19:38:06.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:06.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:06.589321+0000) 2024-02-19T19:38:07.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:07.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:07.589569+0000) 2024-02-19T19:38:08.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:08.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:08.589711+0000) 2024-02-19T19:38:09.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:09.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:09.589912+0000) 2024-02-19T19:38:10.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:10.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:10.590155+0000) 2024-02-19T19:38:11.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:11.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:11.590399+0000) 2024-02-19T19:38:12.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:12.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:12.590643+0000) 2024-02-19T19:38:13.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:13.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:13.590836+0000) 2024-02-19T19:38:14.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:14.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:14.590985+0000) 2024-02-19T19:38:15.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:15.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:15.591120+0000) 2024-02-19T19:38:16.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:16.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:16.591327+0000) 2024-02-19T19:38:17.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:17.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:17.591581+0000) 2024-02-19T19:38:18.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:18.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:18.591833+0000) 2024-02-19T19:38:19.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:19.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:19.592112+0000) 2024-02-19T19:38:20.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:20.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:20.592373+0000) 2024-02-19T19:38:21.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:21.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:21.592633+0000) 2024-02-19T19:38:22.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:22.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:22.592843+0000) 2024-02-19T19:38:23.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:23.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:23.593007+0000) 2024-02-19T19:38:24.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:24.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:24.593290+0000) 2024-02-19T19:38:25.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:25.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:25.593551+0000) 2024-02-19T19:38:26.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:26.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:26.593808+0000) 2024-02-19T19:38:27.300 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:38:27.303 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:38:27.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:27.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:27.594122+0000) 2024-02-19T19:38:28.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:28.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:28.594354+0000) 2024-02-19T19:38:29.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:29.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:29.594549+0000) 2024-02-19T19:38:30.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:30.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:30.594820+0000) 2024-02-19T19:38:31.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:31.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:31.595008+0000) 2024-02-19T19:38:32.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:32.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:32.595214+0000) 2024-02-19T19:38:33.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:33.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:33.595396+0000) 2024-02-19T19:38:34.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:34.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:34.595625+0000) 2024-02-19T19:38:35.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:35.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:35.595893+0000) 2024-02-19T19:38:36.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:36.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:36.596164+0000) 2024-02-19T19:38:37.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:37.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:37.596348+0000) 2024-02-19T19:38:38.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:38.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:38.596601+0000) 2024-02-19T19:38:39.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:39.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:39.596913+0000) 2024-02-19T19:38:40.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:40.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:40.597181+0000) 2024-02-19T19:38:41.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:41.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:41.597390+0000) 2024-02-19T19:38:42.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:42.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:42.597599+0000) 2024-02-19T19:38:43.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:43.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:43.597822+0000) 2024-02-19T19:38:44.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:44.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:44.598014+0000) 2024-02-19T19:38:45.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:45.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:45.598277+0000) 2024-02-19T19:38:46.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:46.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:46.598544+0000) 2024-02-19T19:38:47.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:47.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:47.598820+0000) 2024-02-19T19:38:48.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:48.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:48.598953+0000) 2024-02-19T19:38:49.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:49.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:49.599078+0000) 2024-02-19T19:38:50.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:50.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:50.599328+0000) 2024-02-19T19:38:51.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:51.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:51.599573+0000) 2024-02-19T19:38:52.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:52.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:52.599805+0000) 2024-02-19T19:38:53.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:53.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:53.600095+0000) 2024-02-19T19:38:54.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:54.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:54.600327+0000) 2024-02-19T19:38:55.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:55.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:55.600569+0000) 2024-02-19T19:38:56.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:56.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:56.600796+0000) 2024-02-19T19:38:57.359 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:38:57.363 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:38:57.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:57.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:57.600968+0000) 2024-02-19T19:38:58.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:58.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:58.601240+0000) 2024-02-19T19:38:59.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:38:59.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:38:59.601490+0000) 2024-02-19T19:39:00.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:00.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:00.601765+0000) 2024-02-19T19:39:01.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:01.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:01.601960+0000) 2024-02-19T19:39:02.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:02.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:02.602199+0000) 2024-02-19T19:39:03.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:03.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:03.602439+0000) 2024-02-19T19:39:04.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:04.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:04.602644+0000) 2024-02-19T19:39:05.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:05.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:05.602889+0000) 2024-02-19T19:39:06.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:06.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:06.603079+0000) 2024-02-19T19:39:07.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:07.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:07.603269+0000) 2024-02-19T19:39:08.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:08.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:08.603577+0000) 2024-02-19T19:39:09.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:09.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:09.603760+0000) 2024-02-19T19:39:10.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:10.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:10.604064+0000) 2024-02-19T19:39:11.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:11.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:11.604268+0000) 2024-02-19T19:39:12.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:12.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:12.604507+0000) 2024-02-19T19:39:13.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:13.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:13.604748+0000) 2024-02-19T19:39:14.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:14.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:14.604912+0000) 2024-02-19T19:39:15.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:15.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:15.605180+0000) 2024-02-19T19:39:16.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:16.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:16.605452+0000) 2024-02-19T19:39:17.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:17.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:17.605769+0000) 2024-02-19T19:39:18.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:18.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:18.606011+0000) 2024-02-19T19:39:19.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:19.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:19.606232+0000) 2024-02-19T19:39:20.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:20.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:20.606493+0000) 2024-02-19T19:39:21.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:21.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:21.606820+0000) 2024-02-19T19:39:22.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:22.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:22.607076+0000) 2024-02-19T19:39:23.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:23.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:23.607299+0000) 2024-02-19T19:39:24.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:24.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:24.607498+0000) 2024-02-19T19:39:25.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:25.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:25.607736+0000) 2024-02-19T19:39:26.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:26.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:26.607979+0000) 2024-02-19T19:39:27.503 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:39:27.506 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:39:27.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:27.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:27.608216+0000) 2024-02-19T19:39:28.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:28.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:28.608470+0000) 2024-02-19T19:39:29.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:29.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:29.608710+0000) 2024-02-19T19:39:30.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:30.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:30.608940+0000) 2024-02-19T19:39:31.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:31.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:31.609129+0000) 2024-02-19T19:39:32.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:32.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:32.609400+0000) 2024-02-19T19:39:33.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:33.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:33.609693+0000) 2024-02-19T19:39:34.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:34.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:34.609922+0000) 2024-02-19T19:39:35.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:35.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:35.610052+0000) 2024-02-19T19:39:36.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:36.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:36.610212+0000) 2024-02-19T19:39:37.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:37.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:37.610382+0000) 2024-02-19T19:39:38.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:38.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:38.610664+0000) 2024-02-19T19:39:39.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:39.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:39.610922+0000) 2024-02-19T19:39:40.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:40.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:40.611174+0000) 2024-02-19T19:39:41.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:41.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:41.611424+0000) 2024-02-19T19:39:42.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:42.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:42.611664+0000) 2024-02-19T19:39:43.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:43.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:43.611857+0000) 2024-02-19T19:39:44.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:44.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:44.612088+0000) 2024-02-19T19:39:45.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:45.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:45.612327+0000) 2024-02-19T19:39:46.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:46.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:46.612606+0000) 2024-02-19T19:39:47.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:47.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:47.612807+0000) 2024-02-19T19:39:48.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:48.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:48.613079+0000) 2024-02-19T19:39:49.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:49.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:49.613330+0000) 2024-02-19T19:39:50.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:50.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:50.613597+0000) 2024-02-19T19:39:51.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:51.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:51.613838+0000) 2024-02-19T19:39:52.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:52.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:52.613991+0000) 2024-02-19T19:39:53.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:53.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:53.614186+0000) 2024-02-19T19:39:54.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:54.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:54.614370+0000) 2024-02-19T19:39:55.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:55.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:55.614574+0000) 2024-02-19T19:39:56.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:56.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:56.614822+0000) 2024-02-19T19:39:57.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:57.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:57.615033+0000) 2024-02-19T19:39:57.641 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:39:57.644 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:39:58.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:58.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:58.615290+0000) 2024-02-19T19:39:59.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:39:59.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:39:59.615546+0000) 2024-02-19T19:40:00.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:00.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:00.615804+0000) 2024-02-19T19:40:01.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:01.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:01.616004+0000) 2024-02-19T19:40:02.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:02.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:02.616222+0000) 2024-02-19T19:40:03.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:03.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:03.616429+0000) 2024-02-19T19:40:04.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:04.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:04.616677+0000) 2024-02-19T19:40:05.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:05.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:05.616848+0000) 2024-02-19T19:40:06.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:06.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:06.617051+0000) 2024-02-19T19:40:07.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:07.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:07.617289+0000) 2024-02-19T19:40:08.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:08.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:08.617524+0000) 2024-02-19T19:40:09.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:09.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:09.617766+0000) 2024-02-19T19:40:10.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:10.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:10.618014+0000) 2024-02-19T19:40:11.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:11.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:11.618257+0000) 2024-02-19T19:40:12.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:12.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:12.618504+0000) 2024-02-19T19:40:13.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:13.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:13.618825+0000) 2024-02-19T19:40:14.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:14.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:14.619007+0000) 2024-02-19T19:40:15.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:15.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:15.619192+0000) 2024-02-19T19:40:16.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:16.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:16.619331+0000) 2024-02-19T19:40:17.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:17.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:17.619518+0000) 2024-02-19T19:40:18.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:18.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:18.619791+0000) 2024-02-19T19:40:19.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:19.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:19.620011+0000) 2024-02-19T19:40:20.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:20.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:20.620271+0000) 2024-02-19T19:40:21.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:21.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:21.620466+0000) 2024-02-19T19:40:22.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:22.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:22.620703+0000) 2024-02-19T19:40:23.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:23.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:23.620905+0000) 2024-02-19T19:40:24.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:24.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:24.621151+0000) 2024-02-19T19:40:25.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:25.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:25.621391+0000) 2024-02-19T19:40:26.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:26.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:26.621651+0000) 2024-02-19T19:40:27.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:27.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:27.621916+0000) 2024-02-19T19:40:27.746 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:40:27.749 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:40:28.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:28.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:28.622171+0000) 2024-02-19T19:40:29.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:29.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:29.622373+0000) 2024-02-19T19:40:30.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:30.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:30.622538+0000) 2024-02-19T19:40:31.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:31.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:31.622849+0000) 2024-02-19T19:40:32.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:32.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:32.623115+0000) 2024-02-19T19:40:33.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:33.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:33.623350+0000) 2024-02-19T19:40:34.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:34.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:34.623614+0000) 2024-02-19T19:40:35.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:35.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:35.623871+0000) 2024-02-19T19:40:36.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:36.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:36.624129+0000) 2024-02-19T19:40:37.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:37.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:37.624372+0000) 2024-02-19T19:40:38.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:38.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:38.624601+0000) 2024-02-19T19:40:39.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:39.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:39.624771+0000) 2024-02-19T19:40:40.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:40.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:40.625036+0000) 2024-02-19T19:40:41.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:41.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:41.625273+0000) 2024-02-19T19:40:42.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:42.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:42.625462+0000) 2024-02-19T19:40:43.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:43.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:43.625703+0000) 2024-02-19T19:40:44.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:44.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:44.625918+0000) 2024-02-19T19:40:45.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:45.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:45.626108+0000) 2024-02-19T19:40:46.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:46.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:46.626355+0000) 2024-02-19T19:40:47.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:47.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:47.626536+0000) 2024-02-19T19:40:48.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:48.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:48.626769+0000) 2024-02-19T19:40:49.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:49.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:49.627019+0000) 2024-02-19T19:40:50.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:50.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:50.627193+0000) 2024-02-19T19:40:51.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:51.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:51.627477+0000) 2024-02-19T19:40:52.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:52.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:52.627754+0000) 2024-02-19T19:40:53.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:53.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:53.628020+0000) 2024-02-19T19:40:54.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:54.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:54.628301+0000) 2024-02-19T19:40:55.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:55.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:55.628503+0000) 2024-02-19T19:40:56.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:56.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:56.628684+0000) 2024-02-19T19:40:57.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:57.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:57.628918+0000) 2024-02-19T19:40:57.877 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:40:57.881 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:40:58.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:58.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:58.629055+0000) 2024-02-19T19:40:59.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:40:59.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:40:59.629222+0000) 2024-02-19T19:41:00.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:00.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:00.629493+0000) 2024-02-19T19:41:01.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:01.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:01.629723+0000) 2024-02-19T19:41:02.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:02.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:02.629982+0000) 2024-02-19T19:41:03.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:03.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:03.630273+0000) 2024-02-19T19:41:04.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:04.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:04.630577+0000) 2024-02-19T19:41:05.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:05.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:05.630863+0000) 2024-02-19T19:41:06.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:06.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:06.631119+0000) 2024-02-19T19:41:07.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:07.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:07.631426+0000) 2024-02-19T19:41:08.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:08.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:08.631692+0000) 2024-02-19T19:41:09.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:09.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:09.631895+0000) 2024-02-19T19:41:10.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:10.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:10.632146+0000) 2024-02-19T19:41:11.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:11.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:11.632313+0000) 2024-02-19T19:41:12.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:12.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:12.632514+0000) 2024-02-19T19:41:13.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:13.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:13.632783+0000) 2024-02-19T19:41:14.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:14.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:14.633080+0000) 2024-02-19T19:41:15.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:15.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:15.633308+0000) 2024-02-19T19:41:16.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:16.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:16.633547+0000) 2024-02-19T19:41:17.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:17.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:17.633812+0000) 2024-02-19T19:41:18.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:18.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:18.634036+0000) 2024-02-19T19:41:19.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:19.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:19.634272+0000) 2024-02-19T19:41:20.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:20.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:20.634509+0000) 2024-02-19T19:41:21.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:21.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:21.634764+0000) 2024-02-19T19:41:22.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:22.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:22.634917+0000) 2024-02-19T19:41:23.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:23.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:23.635158+0000) 2024-02-19T19:41:24.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:24.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:24.635405+0000) 2024-02-19T19:41:25.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:25.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:25.635642+0000) 2024-02-19T19:41:26.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:26.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:26.635852+0000) 2024-02-19T19:41:27.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:27.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:27.635999+0000) 2024-02-19T19:41:27.983 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:41:27.987 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:41:28.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:28.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:28.636217+0000) 2024-02-19T19:41:29.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:29.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:29.636407+0000) 2024-02-19T19:41:30.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:30.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:30.636650+0000) 2024-02-19T19:41:31.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:31.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:31.636889+0000) 2024-02-19T19:41:32.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:32.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:32.637127+0000) 2024-02-19T19:41:33.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:33.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:33.637322+0000) 2024-02-19T19:41:34.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:34.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:34.637504+0000) 2024-02-19T19:41:35.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:35.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:35.637698+0000) 2024-02-19T19:41:36.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:36.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:36.637930+0000) 2024-02-19T19:41:37.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:37.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:37.638177+0000) 2024-02-19T19:41:38.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:38.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:38.638423+0000) 2024-02-19T19:41:39.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:39.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:39.638703+0000) 2024-02-19T19:41:40.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:40.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:40.638854+0000) 2024-02-19T19:41:41.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:41.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:41.639069+0000) 2024-02-19T19:41:42.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:42.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:42.639274+0000) 2024-02-19T19:41:43.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:43.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:43.639546+0000) 2024-02-19T19:41:44.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:44.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:44.639846+0000) 2024-02-19T19:41:45.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:45.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:45.640123+0000) 2024-02-19T19:41:46.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:46.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:46.640330+0000) 2024-02-19T19:41:47.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:47.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:47.640615+0000) 2024-02-19T19:41:48.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:48.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:48.640928+0000) 2024-02-19T19:41:49.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:49.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:49.641230+0000) 2024-02-19T19:41:50.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:50.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:50.641452+0000) 2024-02-19T19:41:51.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:51.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:51.641609+0000) 2024-02-19T19:41:52.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:52.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:52.641887+0000) 2024-02-19T19:41:53.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:53.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:53.642108+0000) 2024-02-19T19:41:54.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:54.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:54.642376+0000) 2024-02-19T19:41:55.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:55.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:55.642604+0000) 2024-02-19T19:41:56.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:56.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:56.642834+0000) 2024-02-19T19:41:57.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:57.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:57.643128+0000) 2024-02-19T19:41:58.088 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:41:58.092 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:41:58.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:58.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:58.643386+0000) 2024-02-19T19:41:59.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:41:59.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:41:59.643573+0000) 2024-02-19T19:42:00.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:00.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:00.643797+0000) 2024-02-19T19:42:01.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:01.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:01.643993+0000) 2024-02-19T19:42:02.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:02.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:02.644171+0000) 2024-02-19T19:42:03.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:03.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:03.644445+0000) 2024-02-19T19:42:04.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:04.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:04.644628+0000) 2024-02-19T19:42:05.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:05.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:05.644855+0000) 2024-02-19T19:42:06.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:06.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:06.645134+0000) 2024-02-19T19:42:07.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:07.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:07.645446+0000) 2024-02-19T19:42:08.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:08.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:08.645670+0000) 2024-02-19T19:42:09.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:09.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:09.645877+0000) 2024-02-19T19:42:10.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:10.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:10.646122+0000) 2024-02-19T19:42:11.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:11.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:11.646290+0000) 2024-02-19T19:42:12.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:12.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:12.646532+0000) 2024-02-19T19:42:13.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:13.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:13.646771+0000) 2024-02-19T19:42:14.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:14.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:14.646949+0000) 2024-02-19T19:42:15.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:15.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:15.647172+0000) 2024-02-19T19:42:16.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:16.647+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:16.647419+0000) 2024-02-19T19:42:17.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:17.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:17.647625+0000) 2024-02-19T19:42:18.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:18.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:18.647764+0000) 2024-02-19T19:42:19.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:19.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:19.647963+0000) 2024-02-19T19:42:20.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:20.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:20.648198+0000) 2024-02-19T19:42:21.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:21.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:21.648385+0000) 2024-02-19T19:42:22.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:22.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:22.648644+0000) 2024-02-19T19:42:23.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:23.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:23.648897+0000) 2024-02-19T19:42:24.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:24.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:24.649130+0000) 2024-02-19T19:42:25.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:25.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:25.649377+0000) 2024-02-19T19:42:26.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:26.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:26.649616+0000) 2024-02-19T19:42:27.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:27.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:27.649853+0000) 2024-02-19T19:42:28.209 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:42:28.212 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:42:28.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:28.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:28.650089+0000) 2024-02-19T19:42:29.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:29.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:29.650317+0000) 2024-02-19T19:42:30.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:30.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:30.650459+0000) 2024-02-19T19:42:31.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:31.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:31.650646+0000) 2024-02-19T19:42:32.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:32.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:32.650859+0000) 2024-02-19T19:42:33.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:33.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:33.651028+0000) 2024-02-19T19:42:34.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:34.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:34.651229+0000) 2024-02-19T19:42:35.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:35.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:35.651423+0000) 2024-02-19T19:42:36.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:36.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:36.651634+0000) 2024-02-19T19:42:37.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:37.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:37.651843+0000) 2024-02-19T19:42:38.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:38.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:38.652117+0000) 2024-02-19T19:42:39.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:39.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:39.652297+0000) 2024-02-19T19:42:40.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:40.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:40.652434+0000) 2024-02-19T19:42:41.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:41.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:41.652658+0000) 2024-02-19T19:42:42.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:42.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:42.652860+0000) 2024-02-19T19:42:43.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:43.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:43.653099+0000) 2024-02-19T19:42:44.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:44.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:44.653375+0000) 2024-02-19T19:42:45.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:45.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:45.653611+0000) 2024-02-19T19:42:46.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:46.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:46.653871+0000) 2024-02-19T19:42:47.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:47.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:47.654069+0000) 2024-02-19T19:42:48.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:48.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:48.654313+0000) 2024-02-19T19:42:49.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:49.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:49.654559+0000) 2024-02-19T19:42:50.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:50.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:50.654844+0000) 2024-02-19T19:42:51.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:51.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:51.654976+0000) 2024-02-19T19:42:52.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:52.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:52.655114+0000) 2024-02-19T19:42:53.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:53.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:53.655338+0000) 2024-02-19T19:42:54.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:54.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:54.655541+0000) 2024-02-19T19:42:55.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:55.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:55.655861+0000) 2024-02-19T19:42:56.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:56.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:56.656106+0000) 2024-02-19T19:42:57.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:57.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:57.656352+0000) 2024-02-19T19:42:58.348 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:42:58.352 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:42:58.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:58.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:58.656598+0000) 2024-02-19T19:42:59.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:42:59.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:42:59.656774+0000) 2024-02-19T19:43:00.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:00.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:00.657052+0000) 2024-02-19T19:43:01.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:01.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:01.657316+0000) 2024-02-19T19:43:02.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:02.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:02.657571+0000) 2024-02-19T19:43:03.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:03.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:03.657698+0000) 2024-02-19T19:43:04.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:04.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:04.657917+0000) 2024-02-19T19:43:05.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:05.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:05.658155+0000) 2024-02-19T19:43:06.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:06.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:06.658333+0000) 2024-02-19T19:43:07.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:07.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:07.658573+0000) 2024-02-19T19:43:08.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:08.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:08.658820+0000) 2024-02-19T19:43:09.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:09.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:09.658984+0000) 2024-02-19T19:43:10.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:10.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:10.659149+0000) 2024-02-19T19:43:11.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:11.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:11.659339+0000) 2024-02-19T19:43:12.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:12.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:12.659501+0000) 2024-02-19T19:43:13.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:13.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:13.659747+0000) 2024-02-19T19:43:14.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:14.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:14.660036+0000) 2024-02-19T19:43:15.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:15.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:15.660278+0000) 2024-02-19T19:43:16.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:16.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:16.660493+0000) 2024-02-19T19:43:17.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:17.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:17.660705+0000) 2024-02-19T19:43:18.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:18.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:18.660969+0000) 2024-02-19T19:43:19.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:19.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:19.661272+0000) 2024-02-19T19:43:20.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:20.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:20.661515+0000) 2024-02-19T19:43:21.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:21.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:21.661698+0000) 2024-02-19T19:43:22.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:22.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:22.661956+0000) 2024-02-19T19:43:23.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:23.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:23.662190+0000) 2024-02-19T19:43:24.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:24.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:24.662369+0000) 2024-02-19T19:43:25.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:25.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:25.662571+0000) 2024-02-19T19:43:26.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:26.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:26.662692+0000) 2024-02-19T19:43:27.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:27.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:27.662922+0000) 2024-02-19T19:43:28.420 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:43:28.423 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:43:28.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:28.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:28.663169+0000) 2024-02-19T19:43:29.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:29.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:29.663413+0000) 2024-02-19T19:43:30.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:30.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:30.663651+0000) 2024-02-19T19:43:31.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:31.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:31.663894+0000) 2024-02-19T19:43:32.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:32.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:32.664049+0000) 2024-02-19T19:43:33.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:33.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:33.664282+0000) 2024-02-19T19:43:34.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:34.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:34.664427+0000) 2024-02-19T19:43:35.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:35.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:35.664672+0000) 2024-02-19T19:43:36.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:36.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:36.664907+0000) 2024-02-19T19:43:37.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:37.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:37.665147+0000) 2024-02-19T19:43:38.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:38.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:38.665358+0000) 2024-02-19T19:43:39.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:39.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:39.665493+0000) 2024-02-19T19:43:40.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:40.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:40.665754+0000) 2024-02-19T19:43:41.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:41.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:41.665967+0000) 2024-02-19T19:43:42.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:42.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:42.666130+0000) 2024-02-19T19:43:43.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:43.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:43.666304+0000) 2024-02-19T19:43:44.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:44.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:44.666493+0000) 2024-02-19T19:43:45.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:45.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:45.666739+0000) 2024-02-19T19:43:46.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:46.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:46.666952+0000) 2024-02-19T19:43:47.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:47.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:47.667217+0000) 2024-02-19T19:43:48.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:48.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:48.667473+0000) 2024-02-19T19:43:49.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:49.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:49.667732+0000) 2024-02-19T19:43:50.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:50.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:50.667878+0000) 2024-02-19T19:43:51.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:51.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:51.668105+0000) 2024-02-19T19:43:52.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:52.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:52.668341+0000) 2024-02-19T19:43:53.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:53.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:53.668560+0000) 2024-02-19T19:43:54.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:54.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:54.668886+0000) 2024-02-19T19:43:55.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:55.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:55.669060+0000) 2024-02-19T19:43:56.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:56.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:56.669250+0000) 2024-02-19T19:43:57.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:57.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:57.669504+0000) 2024-02-19T19:43:58.542 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:43:58.546 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:43:58.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:58.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:58.669772+0000) 2024-02-19T19:43:59.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:43:59.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:43:59.669971+0000) 2024-02-19T19:44:00.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:00.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:00.670114+0000) 2024-02-19T19:44:01.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:01.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:01.670234+0000) 2024-02-19T19:44:02.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:02.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:02.670399+0000) 2024-02-19T19:44:03.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:03.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:03.670615+0000) 2024-02-19T19:44:04.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:04.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:04.670768+0000) 2024-02-19T19:44:05.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:05.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:05.670894+0000) 2024-02-19T19:44:06.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:06.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:06.671029+0000) 2024-02-19T19:44:07.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:07.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:07.671159+0000) 2024-02-19T19:44:08.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:08.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:08.671330+0000) 2024-02-19T19:44:09.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:09.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:09.671507+0000) 2024-02-19T19:44:10.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:10.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:10.671649+0000) 2024-02-19T19:44:11.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:11.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:11.671836+0000) 2024-02-19T19:44:12.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:12.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:12.671985+0000) 2024-02-19T19:44:13.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:13.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:13.672212+0000) 2024-02-19T19:44:14.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:14.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:14.672472+0000) 2024-02-19T19:44:15.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:15.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:15.672693+0000) 2024-02-19T19:44:16.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:16.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:16.672909+0000) 2024-02-19T19:44:17.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:17.673+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:17.673100+0000) 2024-02-19T19:44:18.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:18.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:18.673352+0000) 2024-02-19T19:44:19.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:19.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:19.673566+0000) 2024-02-19T19:44:20.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:20.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:20.673746+0000) 2024-02-19T19:44:21.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:21.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:21.673902+0000) 2024-02-19T19:44:22.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:22.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:22.674131+0000) 2024-02-19T19:44:23.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:23.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:23.674380+0000) 2024-02-19T19:44:24.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:24.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:24.674623+0000) 2024-02-19T19:44:25.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:25.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:25.674893+0000) 2024-02-19T19:44:26.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:26.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:26.675055+0000) 2024-02-19T19:44:27.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:27.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:27.675235+0000) 2024-02-19T19:44:28.665 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:44:28.668 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:44:28.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:28.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:28.675393+0000) 2024-02-19T19:44:29.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:29.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:29.675656+0000) 2024-02-19T19:44:30.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:30.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:30.675784+0000) 2024-02-19T19:44:31.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:31.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:31.676060+0000) 2024-02-19T19:44:32.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:32.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:32.676277+0000) 2024-02-19T19:44:33.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:33.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:33.676470+0000) 2024-02-19T19:44:34.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:34.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:34.676712+0000) 2024-02-19T19:44:35.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:35.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:35.676962+0000) 2024-02-19T19:44:36.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:36.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:36.677149+0000) 2024-02-19T19:44:37.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:37.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:37.677394+0000) 2024-02-19T19:44:38.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:38.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:38.677635+0000) 2024-02-19T19:44:39.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:39.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:39.677890+0000) 2024-02-19T19:44:40.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:40.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:40.678091+0000) 2024-02-19T19:44:41.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:41.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:41.678327+0000) 2024-02-19T19:44:42.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:42.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:42.678582+0000) 2024-02-19T19:44:43.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:43.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:43.678828+0000) 2024-02-19T19:44:44.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:44.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:44.679002+0000) 2024-02-19T19:44:45.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:45.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:45.679192+0000) 2024-02-19T19:44:46.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:46.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:46.679385+0000) 2024-02-19T19:44:47.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:47.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:47.679627+0000) 2024-02-19T19:44:48.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:48.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:48.679880+0000) 2024-02-19T19:44:49.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:49.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:49.680129+0000) 2024-02-19T19:44:50.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:50.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:50.680389+0000) 2024-02-19T19:44:51.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:51.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:51.680656+0000) 2024-02-19T19:44:52.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:52.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:52.680848+0000) 2024-02-19T19:44:53.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:53.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:53.681102+0000) 2024-02-19T19:44:54.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:54.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:54.681303+0000) 2024-02-19T19:44:55.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:55.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:55.681547+0000) 2024-02-19T19:44:56.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:56.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:56.681758+0000) 2024-02-19T19:44:57.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:57.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:57.681981+0000) 2024-02-19T19:44:58.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:58.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:58.682248+0000) 2024-02-19T19:44:58.795 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:44:58.800 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:44:59.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:44:59.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:44:59.682432+0000) 2024-02-19T19:45:00.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:00.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:00.682678+0000) 2024-02-19T19:45:01.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:01.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:01.682889+0000) 2024-02-19T19:45:02.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:02.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:02.683050+0000) 2024-02-19T19:45:03.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:03.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:03.683252+0000) 2024-02-19T19:45:04.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:04.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:04.683432+0000) 2024-02-19T19:45:05.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:05.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:05.683653+0000) 2024-02-19T19:45:06.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:06.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:06.683910+0000) 2024-02-19T19:45:07.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:07.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:07.684150+0000) 2024-02-19T19:45:08.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:08.684+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:08.684341+0000) 2024-02-19T19:45:09.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:09.684+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:09.684580+0000) 2024-02-19T19:45:10.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:10.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:10.684809+0000) 2024-02-19T19:45:11.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:11.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:11.684979+0000) 2024-02-19T19:45:12.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:12.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:12.685150+0000) 2024-02-19T19:45:13.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:13.684+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:13.685395+0000) 2024-02-19T19:45:14.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:14.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:14.685667+0000) 2024-02-19T19:45:15.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:15.684+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:15.685935+0000) 2024-02-19T19:45:16.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:16.684+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:16.686186+0000) 2024-02-19T19:45:17.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:17.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:17.686432+0000) 2024-02-19T19:45:18.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:18.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:18.686661+0000) 2024-02-19T19:45:19.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:19.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:19.686883+0000) 2024-02-19T19:45:20.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:20.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:20.687140+0000) 2024-02-19T19:45:21.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:21.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:21.687352+0000) 2024-02-19T19:45:22.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:22.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:22.687586+0000) 2024-02-19T19:45:23.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:23.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:23.687811+0000) 2024-02-19T19:45:24.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:24.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:24.688064+0000) 2024-02-19T19:45:25.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:25.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:25.688332+0000) 2024-02-19T19:45:26.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:26.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:26.688561+0000) 2024-02-19T19:45:27.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:27.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:27.688803+0000) 2024-02-19T19:45:28.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:28.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:28.689033+0000) 2024-02-19T19:45:28.934 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:45:28.945 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:45:29.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:29.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:29.689270+0000) 2024-02-19T19:45:30.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:30.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:30.689562+0000) 2024-02-19T19:45:31.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:31.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:31.689764+0000) 2024-02-19T19:45:32.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:32.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:32.689911+0000) 2024-02-19T19:45:33.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:33.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:33.690074+0000) 2024-02-19T19:45:34.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:34.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:34.690314+0000) 2024-02-19T19:45:35.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:35.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:35.690572+0000) 2024-02-19T19:45:36.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:36.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:36.690860+0000) 2024-02-19T19:45:37.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:37.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:37.690990+0000) 2024-02-19T19:45:38.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:38.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:38.691168+0000) 2024-02-19T19:45:39.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:39.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:39.691415+0000) 2024-02-19T19:45:40.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:40.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:40.691682+0000) 2024-02-19T19:45:41.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:41.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:41.691888+0000) 2024-02-19T19:45:42.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:42.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:42.692020+0000) 2024-02-19T19:45:43.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:43.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:43.692246+0000) 2024-02-19T19:45:44.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:44.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:44.692453+0000) 2024-02-19T19:45:45.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:45.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:45.692676+0000) 2024-02-19T19:45:46.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:46.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:46.692860+0000) 2024-02-19T19:45:47.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:47.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:47.693050+0000) 2024-02-19T19:45:48.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:48.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:48.693292+0000) 2024-02-19T19:45:49.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:49.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:49.693513+0000) 2024-02-19T19:45:50.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:50.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:50.693739+0000) 2024-02-19T19:45:51.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:51.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:51.693994+0000) 2024-02-19T19:45:52.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:52.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:52.694247+0000) 2024-02-19T19:45:53.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:53.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:53.694430+0000) 2024-02-19T19:45:54.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:54.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:54.694598+0000) 2024-02-19T19:45:55.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:55.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:55.694836+0000) 2024-02-19T19:45:56.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:56.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:56.694989+0000) 2024-02-19T19:45:57.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:57.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:57.695171+0000) 2024-02-19T19:45:58.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:58.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:58.695442+0000) 2024-02-19T19:45:59.064 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:45:59.067 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:45:59.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:45:59.695+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:45:59.695728+0000) 2024-02-19T19:46:00.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:00.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:00.695931+0000) 2024-02-19T19:46:01.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:01.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:01.696086+0000) 2024-02-19T19:46:02.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:02.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:02.696323+0000) 2024-02-19T19:46:03.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:03.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:03.696576+0000) 2024-02-19T19:46:04.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:04.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:04.696817+0000) 2024-02-19T19:46:05.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:05.695+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:05.697074+0000) 2024-02-19T19:46:06.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:06.695+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:06.697260+0000) 2024-02-19T19:46:07.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:07.695+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:07.697502+0000) 2024-02-19T19:46:08.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:08.695+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:08.697695+0000) 2024-02-19T19:46:09.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:09.695+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:09.697952+0000) 2024-02-19T19:46:10.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:10.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:10.698167+0000) 2024-02-19T19:46:11.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:11.695+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:11.698361+0000) 2024-02-19T19:46:12.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:12.695+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:12.698625+0000) 2024-02-19T19:46:13.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:13.695+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:13.698816+0000) 2024-02-19T19:46:14.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:14.695+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:14.699051+0000) 2024-02-19T19:46:15.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:15.699+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:15.699231+0000) 2024-02-19T19:46:16.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:16.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:16.699476+0000) 2024-02-19T19:46:17.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:17.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:17.699689+0000) 2024-02-19T19:46:18.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:18.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:18.699908+0000) 2024-02-19T19:46:19.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:19.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:19.700119+0000) 2024-02-19T19:46:20.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:20.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:20.700353+0000) 2024-02-19T19:46:21.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:21.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:21.700612+0000) 2024-02-19T19:46:22.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:22.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:22.700858+0000) 2024-02-19T19:46:23.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:23.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:23.701091+0000) 2024-02-19T19:46:24.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:24.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:24.701320+0000) 2024-02-19T19:46:25.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:25.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:25.701569+0000) 2024-02-19T19:46:26.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:26.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:26.701756+0000) 2024-02-19T19:46:27.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:27.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:27.701984+0000) 2024-02-19T19:46:28.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:28.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:28.702221+0000) 2024-02-19T19:46:29.211 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:46:29.214 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:46:29.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:29.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:29.702459+0000) 2024-02-19T19:46:30.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:30.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:30.702747+0000) 2024-02-19T19:46:31.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:31.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:31.702975+0000) 2024-02-19T19:46:32.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:32.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:32.703161+0000) 2024-02-19T19:46:33.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:33.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:33.703337+0000) 2024-02-19T19:46:34.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:34.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:34.703600+0000) 2024-02-19T19:46:35.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:35.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:35.703932+0000) 2024-02-19T19:46:36.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:36.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:36.704140+0000) 2024-02-19T19:46:37.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:37.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:37.704328+0000) 2024-02-19T19:46:38.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:38.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:38.704583+0000) 2024-02-19T19:46:39.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:39.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:39.704861+0000) 2024-02-19T19:46:40.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:40.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:40.705108+0000) 2024-02-19T19:46:41.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:41.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:41.705336+0000) 2024-02-19T19:46:42.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:42.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:42.705526+0000) 2024-02-19T19:46:43.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:43.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:43.705728+0000) 2024-02-19T19:46:44.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:44.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:44.705982+0000) 2024-02-19T19:46:45.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:45.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:45.706193+0000) 2024-02-19T19:46:46.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:46.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:46.706387+0000) 2024-02-19T19:46:47.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:47.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:47.706574+0000) 2024-02-19T19:46:48.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:48.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:48.706770+0000) 2024-02-19T19:46:49.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:49.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:49.707005+0000) 2024-02-19T19:46:50.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:50.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:50.707193+0000) 2024-02-19T19:46:51.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:51.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:51.707425+0000) 2024-02-19T19:46:52.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:52.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:52.707666+0000) 2024-02-19T19:46:53.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:53.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:53.707897+0000) 2024-02-19T19:46:54.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:54.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:54.708086+0000) 2024-02-19T19:46:55.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:55.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:55.708338+0000) 2024-02-19T19:46:56.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:56.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:56.708538+0000) 2024-02-19T19:46:57.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:57.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:57.708791+0000) 2024-02-19T19:46:58.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:58.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:58.709025+0000) 2024-02-19T19:46:59.287 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:46:59.291 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:46:59.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:46:59.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:46:59.709219+0000) 2024-02-19T19:47:00.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:00.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:00.709420+0000) 2024-02-19T19:47:01.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:01.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:01.709656+0000) 2024-02-19T19:47:02.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:02.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:02.709834+0000) 2024-02-19T19:47:03.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:03.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:03.710119+0000) 2024-02-19T19:47:04.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:04.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:04.710314+0000) 2024-02-19T19:47:05.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:05.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:05.710554+0000) 2024-02-19T19:47:06.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:06.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:06.710696+0000) 2024-02-19T19:47:07.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:07.710+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:07.710912+0000) 2024-02-19T19:47:08.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:08.710+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:08.711106+0000) 2024-02-19T19:47:09.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:09.710+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:09.711305+0000) 2024-02-19T19:47:10.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:10.710+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:10.711582+0000) 2024-02-19T19:47:11.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:11.710+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:11.711840+0000) 2024-02-19T19:47:12.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:12.710+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:12.712057+0000) 2024-02-19T19:47:13.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:13.710+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:13.712325+0000) 2024-02-19T19:47:14.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:14.710+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:14.712559+0000) 2024-02-19T19:47:15.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:15.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:15.712795+0000) 2024-02-19T19:47:16.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:16.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:16.712972+0000) 2024-02-19T19:47:17.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:17.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:17.713217+0000) 2024-02-19T19:47:18.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:18.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:18.713467+0000) 2024-02-19T19:47:19.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:19.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:19.713644+0000) 2024-02-19T19:47:20.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:20.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:20.713890+0000) 2024-02-19T19:47:21.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:21.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:21.714118+0000) 2024-02-19T19:47:22.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:22.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:22.714465+0000) 2024-02-19T19:47:23.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:23.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:23.714670+0000) 2024-02-19T19:47:24.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:24.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:24.714947+0000) 2024-02-19T19:47:25.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:25.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:25.715213+0000) 2024-02-19T19:47:26.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:26.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:26.715481+0000) 2024-02-19T19:47:27.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:27.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:27.715758+0000) 2024-02-19T19:47:28.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:28.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:28.716009+0000) 2024-02-19T19:47:29.414 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:47:29.417 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:47:29.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:29.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:29.716271+0000) 2024-02-19T19:47:30.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:30.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:30.716459+0000) 2024-02-19T19:47:31.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:31.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:31.716728+0000) 2024-02-19T19:47:32.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:32.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:32.716972+0000) 2024-02-19T19:47:33.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:33.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:33.717200+0000) 2024-02-19T19:47:34.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:34.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:34.717391+0000) 2024-02-19T19:47:35.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:35.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:35.717603+0000) 2024-02-19T19:47:36.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:36.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:36.717773+0000) 2024-02-19T19:47:37.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:37.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:37.718024+0000) 2024-02-19T19:47:38.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:38.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:38.718194+0000) 2024-02-19T19:47:39.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:39.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:39.718379+0000) 2024-02-19T19:47:40.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:40.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:40.718555+0000) 2024-02-19T19:47:41.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:41.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:41.718848+0000) 2024-02-19T19:47:42.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:42.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:42.719123+0000) 2024-02-19T19:47:43.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:43.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:43.719330+0000) 2024-02-19T19:47:44.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:44.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:44.719466+0000) 2024-02-19T19:47:45.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:45.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:45.719705+0000) 2024-02-19T19:47:46.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:46.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:46.719956+0000) 2024-02-19T19:47:47.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:47.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:47.720205+0000) 2024-02-19T19:47:48.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:48.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:48.720416+0000) 2024-02-19T19:47:49.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:49.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:49.720685+0000) 2024-02-19T19:47:50.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:50.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:50.720940+0000) 2024-02-19T19:47:51.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:51.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:51.721179+0000) 2024-02-19T19:47:52.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:52.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:52.721431+0000) 2024-02-19T19:47:53.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:53.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:53.721658+0000) 2024-02-19T19:47:54.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:54.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:54.721792+0000) 2024-02-19T19:47:55.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:55.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:55.722027+0000) 2024-02-19T19:47:56.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:56.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:56.722292+0000) 2024-02-19T19:47:57.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:57.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:57.722506+0000) 2024-02-19T19:47:58.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:58.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:58.722649+0000) 2024-02-19T19:47:59.562 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:47:59.565 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:47:59.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:47:59.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:47:59.722892+0000) 2024-02-19T19:48:00.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:00.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:00.723149+0000) 2024-02-19T19:48:01.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:01.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:01.723400+0000) 2024-02-19T19:48:02.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:02.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:02.723674+0000) 2024-02-19T19:48:03.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:03.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:03.723907+0000) 2024-02-19T19:48:04.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:04.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:04.724158+0000) 2024-02-19T19:48:05.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:05.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:05.724430+0000) 2024-02-19T19:48:06.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:06.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:06.724689+0000) 2024-02-19T19:48:07.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:07.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:07.724971+0000) 2024-02-19T19:48:08.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:08.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:08.725220+0000) 2024-02-19T19:48:09.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:09.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:09.725503+0000) 2024-02-19T19:48:10.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:10.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:10.725794+0000) 2024-02-19T19:48:11.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:11.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:11.725983+0000) 2024-02-19T19:48:12.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:12.725+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:12.726227+0000) 2024-02-19T19:48:13.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:13.725+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:13.726444+0000) 2024-02-19T19:48:14.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:14.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:14.726647+0000) 2024-02-19T19:48:15.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:15.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:15.726849+0000) 2024-02-19T19:48:16.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:16.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:16.727056+0000) 2024-02-19T19:48:17.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:17.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:17.727256+0000) 2024-02-19T19:48:18.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:18.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:18.727494+0000) 2024-02-19T19:48:19.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:19.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:19.727762+0000) 2024-02-19T19:48:20.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:20.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:20.728020+0000) 2024-02-19T19:48:21.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:21.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:21.728254+0000) 2024-02-19T19:48:22.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:22.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:22.728508+0000) 2024-02-19T19:48:23.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:23.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:23.728732+0000) 2024-02-19T19:48:24.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:24.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:24.728943+0000) 2024-02-19T19:48:25.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:25.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:25.729169+0000) 2024-02-19T19:48:26.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:26.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:26.729409+0000) 2024-02-19T19:48:27.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:27.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:27.729643+0000) 2024-02-19T19:48:28.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:28.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:28.729889+0000) 2024-02-19T19:48:29.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:29.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:29.730122+0000) 2024-02-19T19:48:29.891 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:48:29.895 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:48:30.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:30.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:30.730355+0000) 2024-02-19T19:48:31.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:31.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:31.730624+0000) 2024-02-19T19:48:32.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:32.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:32.730857+0000) 2024-02-19T19:48:33.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:33.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:33.731115+0000) 2024-02-19T19:48:34.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:34.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:34.731400+0000) 2024-02-19T19:48:35.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:35.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:35.731619+0000) 2024-02-19T19:48:36.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:36.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:36.731788+0000) 2024-02-19T19:48:37.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:37.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:37.731994+0000) 2024-02-19T19:48:38.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:38.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:38.732233+0000) 2024-02-19T19:48:39.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:39.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:39.732422+0000) 2024-02-19T19:48:40.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:40.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:40.732615+0000) 2024-02-19T19:48:41.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:41.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:41.732850+0000) 2024-02-19T19:48:42.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:42.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:42.733041+0000) 2024-02-19T19:48:43.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:43.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:43.733266+0000) 2024-02-19T19:48:44.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:44.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:44.733491+0000) 2024-02-19T19:48:45.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:45.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:45.733673+0000) 2024-02-19T19:48:46.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:46.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:46.733880+0000) 2024-02-19T19:48:47.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:47.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:47.734113+0000) 2024-02-19T19:48:48.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:48.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:48.734304+0000) 2024-02-19T19:48:49.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:49.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:49.734543+0000) 2024-02-19T19:48:50.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:50.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:50.734831+0000) 2024-02-19T19:48:51.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:51.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:51.735017+0000) 2024-02-19T19:48:52.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:52.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:52.735272+0000) 2024-02-19T19:48:53.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:53.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:53.735557+0000) 2024-02-19T19:48:54.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:54.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:54.735828+0000) 2024-02-19T19:48:55.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:55.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:55.736073+0000) 2024-02-19T19:48:56.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:56.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:56.736283+0000) 2024-02-19T19:48:57.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:57.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:57.736505+0000) 2024-02-19T19:48:58.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:58.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:58.736750+0000) 2024-02-19T19:48:59.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:48:59.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:48:59.736874+0000) 2024-02-19T19:48:59.953 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:48:59.956 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:49:00.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:00.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:00.737060+0000) 2024-02-19T19:49:01.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:01.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:01.737272+0000) 2024-02-19T19:49:02.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:02.736+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:02.737509+0000) 2024-02-19T19:49:03.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:03.736+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:03.737696+0000) 2024-02-19T19:49:04.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:04.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:04.737943+0000) 2024-02-19T19:49:05.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:05.736+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:05.738211+0000) 2024-02-19T19:49:06.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:06.736+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:06.738497+0000) 2024-02-19T19:49:07.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:07.736+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:07.738791+0000) 2024-02-19T19:49:08.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:08.736+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:08.739013+0000) 2024-02-19T19:49:09.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:09.736+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:09.739198+0000) 2024-02-19T19:49:10.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:10.736+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:10.739363+0000) 2024-02-19T19:49:11.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:11.736+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:11.739591+0000) 2024-02-19T19:49:12.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:12.736+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:12.739824+0000) 2024-02-19T19:49:13.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:13.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:13.740056+0000) 2024-02-19T19:49:14.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:14.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:14.740188+0000) 2024-02-19T19:49:15.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:15.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:15.740414+0000) 2024-02-19T19:49:16.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:16.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:16.740593+0000) 2024-02-19T19:49:17.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:17.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:17.740778+0000) 2024-02-19T19:49:18.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:18.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:18.740897+0000) 2024-02-19T19:49:19.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:19.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:19.741095+0000) 2024-02-19T19:49:20.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:20.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:20.741303+0000) 2024-02-19T19:49:21.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:21.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:21.741557+0000) 2024-02-19T19:49:22.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:22.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:22.741742+0000) 2024-02-19T19:49:23.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:23.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:23.741972+0000) 2024-02-19T19:49:24.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:24.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:24.742190+0000) 2024-02-19T19:49:25.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:25.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:25.742433+0000) 2024-02-19T19:49:26.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:26.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:26.742588+0000) 2024-02-19T19:49:27.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:27.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:27.742833+0000) 2024-02-19T19:49:28.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:28.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:28.743025+0000) 2024-02-19T19:49:29.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:29.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:29.743195+0000) 2024-02-19T19:49:30.069 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:49:30.072 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:49:30.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:30.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:30.743407+0000) 2024-02-19T19:49:31.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:31.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:31.743619+0000) 2024-02-19T19:49:32.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:32.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:32.743868+0000) 2024-02-19T19:49:33.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:33.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:33.744134+0000) 2024-02-19T19:49:34.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:34.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:34.744314+0000) 2024-02-19T19:49:35.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:35.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:35.744550+0000) 2024-02-19T19:49:36.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:36.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:36.744801+0000) 2024-02-19T19:49:37.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:37.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:37.745044+0000) 2024-02-19T19:49:38.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:38.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:38.745286+0000) 2024-02-19T19:49:39.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:39.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:39.745462+0000) 2024-02-19T19:49:40.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:40.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:40.745708+0000) 2024-02-19T19:49:41.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:41.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:41.745842+0000) 2024-02-19T19:49:42.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:42.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:42.746077+0000) 2024-02-19T19:49:43.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:43.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:43.746278+0000) 2024-02-19T19:49:44.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:44.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:44.746417+0000) 2024-02-19T19:49:45.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:45.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:45.746658+0000) 2024-02-19T19:49:46.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:46.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:46.746818+0000) 2024-02-19T19:49:47.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:47.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:47.746951+0000) 2024-02-19T19:49:48.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:48.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:48.747071+0000) 2024-02-19T19:49:49.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:49.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:49.747204+0000) 2024-02-19T19:49:50.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:50.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:50.747327+0000) 2024-02-19T19:49:51.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:51.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:51.747487+0000) 2024-02-19T19:49:52.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:52.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:52.747635+0000) 2024-02-19T19:49:53.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:53.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:53.747774+0000) 2024-02-19T19:49:54.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:54.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:54.747897+0000) 2024-02-19T19:49:55.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:55.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:55.748060+0000) 2024-02-19T19:49:56.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:56.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:56.748198+0000) 2024-02-19T19:49:57.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:57.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:57.748328+0000) 2024-02-19T19:49:58.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:58.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:58.748491+0000) 2024-02-19T19:49:59.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:49:59.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:49:59.748640+0000) 2024-02-19T19:50:00.132 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:50:00.136 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:50:00.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:00.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:00.748802+0000) 2024-02-19T19:50:01.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:01.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:01.748959+0000) 2024-02-19T19:50:02.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:02.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:02.749156+0000) 2024-02-19T19:50:03.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:03.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:03.749415+0000) 2024-02-19T19:50:04.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:04.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:04.749646+0000) 2024-02-19T19:50:05.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:05.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:05.749891+0000) 2024-02-19T19:50:06.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:06.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:06.750132+0000) 2024-02-19T19:50:07.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:07.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:07.750418+0000) 2024-02-19T19:50:08.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:08.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:08.750686+0000) 2024-02-19T19:50:09.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:09.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:09.750864+0000) 2024-02-19T19:50:10.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:10.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:10.751095+0000) 2024-02-19T19:50:11.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:11.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:11.751282+0000) 2024-02-19T19:50:12.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:12.751+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:12.751505+0000) 2024-02-19T19:50:13.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:13.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:13.751749+0000) 2024-02-19T19:50:14.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:14.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:14.751942+0000) 2024-02-19T19:50:15.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:15.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:15.752180+0000) 2024-02-19T19:50:16.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:16.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:16.752385+0000) 2024-02-19T19:50:17.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:17.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:17.752629+0000) 2024-02-19T19:50:18.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:18.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:18.752899+0000) 2024-02-19T19:50:19.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:19.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:19.753133+0000) 2024-02-19T19:50:20.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:20.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:20.753318+0000) 2024-02-19T19:50:21.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:21.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:21.753573+0000) 2024-02-19T19:50:22.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:22.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:22.753809+0000) 2024-02-19T19:50:23.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:23.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:23.754039+0000) 2024-02-19T19:50:24.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:24.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:24.754313+0000) 2024-02-19T19:50:25.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:25.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:25.754578+0000) 2024-02-19T19:50:26.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:26.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:26.754855+0000) 2024-02-19T19:50:27.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:27.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:27.755086+0000) 2024-02-19T19:50:28.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:28.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:28.755276+0000) 2024-02-19T19:50:29.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:29.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:29.755527+0000) 2024-02-19T19:50:30.255 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:50:30.258 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:50:30.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:30.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:30.755762+0000) 2024-02-19T19:50:31.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:31.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:31.755908+0000) 2024-02-19T19:50:32.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:32.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:32.756157+0000) 2024-02-19T19:50:33.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:33.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:33.756388+0000) 2024-02-19T19:50:34.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:34.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:34.756632+0000) 2024-02-19T19:50:35.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:35.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:35.756834+0000) 2024-02-19T19:50:36.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:36.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:36.757092+0000) 2024-02-19T19:50:37.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:37.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:37.757300+0000) 2024-02-19T19:50:38.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:38.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:38.757507+0000) 2024-02-19T19:50:39.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:39.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:39.757694+0000) 2024-02-19T19:50:40.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:40.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:40.757907+0000) 2024-02-19T19:50:41.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:41.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:41.758164+0000) 2024-02-19T19:50:42.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:42.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:42.758363+0000) 2024-02-19T19:50:43.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:43.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:43.758633+0000) 2024-02-19T19:50:44.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:44.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:44.758814+0000) 2024-02-19T19:50:45.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:45.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:45.759067+0000) 2024-02-19T19:50:46.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:46.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:46.759341+0000) 2024-02-19T19:50:47.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:47.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:47.759613+0000) 2024-02-19T19:50:48.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:48.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:48.759851+0000) 2024-02-19T19:50:49.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:49.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:49.760083+0000) 2024-02-19T19:50:50.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:50.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:50.760295+0000) 2024-02-19T19:50:51.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:51.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:51.760555+0000) 2024-02-19T19:50:52.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:52.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:52.760824+0000) 2024-02-19T19:50:53.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:53.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:53.761077+0000) 2024-02-19T19:50:54.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:54.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:54.761269+0000) 2024-02-19T19:50:55.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:55.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:55.761522+0000) 2024-02-19T19:50:56.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:56.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:56.761778+0000) 2024-02-19T19:50:57.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:57.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:57.762021+0000) 2024-02-19T19:50:58.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:58.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:58.762255+0000) 2024-02-19T19:50:59.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:50:59.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:50:59.762489+0000) 2024-02-19T19:51:00.393 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:51:00.397 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:51:00.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:00.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:00.762689+0000) 2024-02-19T19:51:01.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:01.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:01.762914+0000) 2024-02-19T19:51:02.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:02.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:02.763107+0000) 2024-02-19T19:51:03.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:03.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:03.763293+0000) 2024-02-19T19:51:04.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:04.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:04.763526+0000) 2024-02-19T19:51:05.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:05.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:05.763757+0000) 2024-02-19T19:51:06.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:06.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:06.764000+0000) 2024-02-19T19:51:07.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:07.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:07.764201+0000) 2024-02-19T19:51:08.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:08.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:08.764409+0000) 2024-02-19T19:51:09.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:09.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:09.764651+0000) 2024-02-19T19:51:10.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:10.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:10.764892+0000) 2024-02-19T19:51:11.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:11.762+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:11.765125+0000) 2024-02-19T19:51:12.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:12.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:12.765371+0000) 2024-02-19T19:51:13.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:13.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:13.765563+0000) 2024-02-19T19:51:14.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:14.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:14.765741+0000) 2024-02-19T19:51:15.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:15.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:15.765979+0000) 2024-02-19T19:51:16.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:16.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:16.766225+0000) 2024-02-19T19:51:17.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:17.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:17.766428+0000) 2024-02-19T19:51:18.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:18.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:18.766642+0000) 2024-02-19T19:51:19.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:19.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:19.766893+0000) 2024-02-19T19:51:20.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:20.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:20.767052+0000) 2024-02-19T19:51:21.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:21.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:21.767217+0000) 2024-02-19T19:51:22.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:22.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:22.767474+0000) 2024-02-19T19:51:23.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:23.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:23.767727+0000) 2024-02-19T19:51:24.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:24.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:24.767997+0000) 2024-02-19T19:51:25.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:25.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:25.768184+0000) 2024-02-19T19:51:26.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:26.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:26.768437+0000) 2024-02-19T19:51:27.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:27.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:27.768684+0000) 2024-02-19T19:51:28.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:28.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:28.768894+0000) 2024-02-19T19:51:29.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:29.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:29.769132+0000) 2024-02-19T19:51:30.507 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:51:30.511 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:51:30.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:30.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:30.769344+0000) 2024-02-19T19:51:31.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:31.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:31.769581+0000) 2024-02-19T19:51:32.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:32.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:32.769709+0000) 2024-02-19T19:51:33.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:33.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:33.769959+0000) 2024-02-19T19:51:34.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:34.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:34.770134+0000) 2024-02-19T19:51:35.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:35.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:35.770371+0000) 2024-02-19T19:51:36.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:36.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:36.770597+0000) 2024-02-19T19:51:37.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:37.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:37.770814+0000) 2024-02-19T19:51:38.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:38.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:38.771115+0000) 2024-02-19T19:51:39.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:39.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:39.771368+0000) 2024-02-19T19:51:40.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:40.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:40.771598+0000) 2024-02-19T19:51:41.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:41.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:41.771843+0000) 2024-02-19T19:51:42.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:42.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:42.772078+0000) 2024-02-19T19:51:43.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:43.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:43.772317+0000) 2024-02-19T19:51:44.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:44.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:44.772554+0000) 2024-02-19T19:51:45.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:45.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:45.772763+0000) 2024-02-19T19:51:46.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:46.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:46.773010+0000) 2024-02-19T19:51:47.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:47.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:47.773264+0000) 2024-02-19T19:51:48.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:48.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:48.773508+0000) 2024-02-19T19:51:49.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:49.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:49.773739+0000) 2024-02-19T19:51:50.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:50.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:50.773972+0000) 2024-02-19T19:51:51.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:51.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:51.774202+0000) 2024-02-19T19:51:52.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:52.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:52.774379+0000) 2024-02-19T19:51:53.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:53.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:53.774554+0000) 2024-02-19T19:51:54.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:54.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:54.774814+0000) 2024-02-19T19:51:55.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:55.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:55.775058+0000) 2024-02-19T19:51:56.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:56.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:56.775245+0000) 2024-02-19T19:51:57.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:57.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:57.775512+0000) 2024-02-19T19:51:58.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:58.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:58.775781+0000) 2024-02-19T19:51:59.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:51:59.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:51:59.775981+0000) 2024-02-19T19:52:00.638 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:52:00.643 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:52:00.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:00.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:00.776261+0000) 2024-02-19T19:52:01.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:01.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:01.776531+0000) 2024-02-19T19:52:02.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:02.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:02.776767+0000) 2024-02-19T19:52:03.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:03.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:03.777035+0000) 2024-02-19T19:52:04.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:04.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:04.777169+0000) 2024-02-19T19:52:05.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:05.777+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:05.777345+0000) 2024-02-19T19:52:06.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:06.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:06.777522+0000) 2024-02-19T19:52:07.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:07.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:07.777767+0000) 2024-02-19T19:52:08.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:08.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:08.777938+0000) 2024-02-19T19:52:09.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:09.777+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:09.778184+0000) 2024-02-19T19:52:10.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:10.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:10.778480+0000) 2024-02-19T19:52:11.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:11.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:11.778690+0000) 2024-02-19T19:52:12.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:12.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:12.778870+0000) 2024-02-19T19:52:13.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:13.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:13.779047+0000) 2024-02-19T19:52:14.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:14.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:14.779228+0000) 2024-02-19T19:52:15.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:15.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:15.779477+0000) 2024-02-19T19:52:16.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:16.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:16.779680+0000) 2024-02-19T19:52:17.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:17.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:17.779838+0000) 2024-02-19T19:52:18.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:18.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:18.780043+0000) 2024-02-19T19:52:19.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:19.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:19.780279+0000) 2024-02-19T19:52:20.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:20.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:20.780516+0000) 2024-02-19T19:52:21.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:21.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:21.780692+0000) 2024-02-19T19:52:22.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:22.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:22.780875+0000) 2024-02-19T19:52:23.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:23.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:23.781128+0000) 2024-02-19T19:52:24.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:24.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:24.781322+0000) 2024-02-19T19:52:25.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:25.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:25.781596+0000) 2024-02-19T19:52:26.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:26.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:26.781829+0000) 2024-02-19T19:52:27.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:27.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:27.781970+0000) 2024-02-19T19:52:28.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:28.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:28.782200+0000) 2024-02-19T19:52:29.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:29.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:29.782439+0000) 2024-02-19T19:52:30.743 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:52:30.747 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:52:30.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:30.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:30.782664+0000) 2024-02-19T19:52:31.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:31.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:31.782924+0000) 2024-02-19T19:52:32.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:32.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:32.783154+0000) 2024-02-19T19:52:33.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:33.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:33.783391+0000) 2024-02-19T19:52:34.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:34.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:34.783640+0000) 2024-02-19T19:52:35.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:35.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:35.783870+0000) 2024-02-19T19:52:36.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:36.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:36.784073+0000) 2024-02-19T19:52:37.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:37.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:37.784313+0000) 2024-02-19T19:52:38.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:38.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:38.784546+0000) 2024-02-19T19:52:39.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:39.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:39.784778+0000) 2024-02-19T19:52:40.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:40.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:40.784945+0000) 2024-02-19T19:52:41.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:41.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:41.785136+0000) 2024-02-19T19:52:42.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:42.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:42.785388+0000) 2024-02-19T19:52:43.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:43.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:43.785663+0000) 2024-02-19T19:52:44.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:44.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:44.785903+0000) 2024-02-19T19:52:45.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:45.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:45.786141+0000) 2024-02-19T19:52:46.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:46.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:46.786419+0000) 2024-02-19T19:52:47.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:47.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:47.786673+0000) 2024-02-19T19:52:48.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:48.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:48.786891+0000) 2024-02-19T19:52:49.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:49.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:49.787141+0000) 2024-02-19T19:52:50.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:50.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:50.787327+0000) 2024-02-19T19:52:51.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:51.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:51.787573+0000) 2024-02-19T19:52:52.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:52.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:52.787754+0000) 2024-02-19T19:52:53.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:53.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:53.787988+0000) 2024-02-19T19:52:54.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:54.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:54.788220+0000) 2024-02-19T19:52:55.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:55.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:55.788390+0000) 2024-02-19T19:52:56.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:56.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:56.788631+0000) 2024-02-19T19:52:57.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:57.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:57.788886+0000) 2024-02-19T19:52:58.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:58.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:58.789126+0000) 2024-02-19T19:52:59.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:52:59.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:52:59.789351+0000) 2024-02-19T19:53:00.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:00.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:00.789546+0000) 2024-02-19T19:53:00.858 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:53:00.861 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:53:01.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:01.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:01.789780+0000) 2024-02-19T19:53:02.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:02.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:02.790004+0000) 2024-02-19T19:53:03.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:03.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:03.790179+0000) 2024-02-19T19:53:04.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:04.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:04.790415+0000) 2024-02-19T19:53:05.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:05.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:05.790641+0000) 2024-02-19T19:53:06.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:06.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:06.790844+0000) 2024-02-19T19:53:07.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:07.788+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:07.791058+0000) 2024-02-19T19:53:08.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:08.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:08.791306+0000) 2024-02-19T19:53:09.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:09.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:09.791498+0000) 2024-02-19T19:53:10.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:10.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:10.791697+0000) 2024-02-19T19:53:11.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:11.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:11.791932+0000) 2024-02-19T19:53:12.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:12.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:12.792166+0000) 2024-02-19T19:53:13.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:13.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:13.792390+0000) 2024-02-19T19:53:14.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:14.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:14.792650+0000) 2024-02-19T19:53:15.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:15.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:15.792891+0000) 2024-02-19T19:53:16.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:16.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:16.793146+0000) 2024-02-19T19:53:17.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:17.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:17.793417+0000) 2024-02-19T19:53:18.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:18.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:18.793617+0000) 2024-02-19T19:53:19.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:19.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:19.793794+0000) 2024-02-19T19:53:20.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:20.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:20.794033+0000) 2024-02-19T19:53:21.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:21.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:21.794269+0000) 2024-02-19T19:53:22.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:22.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:22.794527+0000) 2024-02-19T19:53:23.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:23.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:23.794832+0000) 2024-02-19T19:53:24.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:24.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:24.795034+0000) 2024-02-19T19:53:25.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:25.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:25.795246+0000) 2024-02-19T19:53:26.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:26.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:26.795526+0000) 2024-02-19T19:53:27.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:27.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:27.795755+0000) 2024-02-19T19:53:28.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:28.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:28.795999+0000) 2024-02-19T19:53:29.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:29.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:29.796233+0000) 2024-02-19T19:53:30.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:30.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:30.796486+0000) 2024-02-19T19:53:31.171 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:53:31.174 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:53:31.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:31.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:31.796718+0000) 2024-02-19T19:53:32.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:32.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:32.796959+0000) 2024-02-19T19:53:33.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:33.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:33.797155+0000) 2024-02-19T19:53:34.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:34.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:34.797367+0000) 2024-02-19T19:53:35.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:35.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:35.797602+0000) 2024-02-19T19:53:36.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:36.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:36.797863+0000) 2024-02-19T19:53:37.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:37.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:37.798136+0000) 2024-02-19T19:53:38.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:38.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:38.798411+0000) 2024-02-19T19:53:39.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:39.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:39.798645+0000) 2024-02-19T19:53:40.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:40.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:40.798882+0000) 2024-02-19T19:53:41.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:41.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:41.799089+0000) 2024-02-19T19:53:42.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:42.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:42.799282+0000) 2024-02-19T19:53:43.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:43.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:43.799525+0000) 2024-02-19T19:53:44.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:44.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:44.799792+0000) 2024-02-19T19:53:45.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:45.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:45.799956+0000) 2024-02-19T19:53:46.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:46.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:46.800268+0000) 2024-02-19T19:53:47.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:47.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:47.800510+0000) 2024-02-19T19:53:48.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:48.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:48.800720+0000) 2024-02-19T19:53:49.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:49.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:49.800926+0000) 2024-02-19T19:53:50.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:50.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:50.801113+0000) 2024-02-19T19:53:51.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:51.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:51.801340+0000) 2024-02-19T19:53:52.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:52.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:52.801592+0000) 2024-02-19T19:53:53.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:53.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:53.801843+0000) 2024-02-19T19:53:54.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:54.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:54.802083+0000) 2024-02-19T19:53:55.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:55.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:55.802339+0000) 2024-02-19T19:53:56.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:56.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:56.802599+0000) 2024-02-19T19:53:57.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:57.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:57.802876+0000) 2024-02-19T19:53:58.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:58.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:58.803152+0000) 2024-02-19T19:53:59.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:53:59.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:53:59.803335+0000) 2024-02-19T19:54:00.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:00.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:00.803581+0000) 2024-02-19T19:54:01.223 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:54:01.226 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:54:01.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:01.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:01.803835+0000) 2024-02-19T19:54:02.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:02.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:02.804050+0000) 2024-02-19T19:54:03.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:03.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:03.804329+0000) 2024-02-19T19:54:04.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:04.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:04.804545+0000) 2024-02-19T19:54:05.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:05.803+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:05.804787+0000) 2024-02-19T19:54:06.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:06.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:06.805003+0000) 2024-02-19T19:54:07.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:07.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:07.805274+0000) 2024-02-19T19:54:08.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:08.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:08.805448+0000) 2024-02-19T19:54:09.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:09.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:09.805652+0000) 2024-02-19T19:54:10.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:10.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:10.805934+0000) 2024-02-19T19:54:11.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:11.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:11.806086+0000) 2024-02-19T19:54:12.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:12.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:12.806332+0000) 2024-02-19T19:54:13.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:13.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:13.806572+0000) 2024-02-19T19:54:14.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:14.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:14.806761+0000) 2024-02-19T19:54:15.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:15.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:15.806898+0000) 2024-02-19T19:54:16.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:16.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:16.807148+0000) 2024-02-19T19:54:17.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:17.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:17.807405+0000) 2024-02-19T19:54:18.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:18.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:18.807656+0000) 2024-02-19T19:54:19.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:19.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:19.807888+0000) 2024-02-19T19:54:20.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:20.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:20.808136+0000) 2024-02-19T19:54:21.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:21.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:21.808347+0000) 2024-02-19T19:54:22.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:22.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:22.808636+0000) 2024-02-19T19:54:23.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:23.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:23.808873+0000) 2024-02-19T19:54:24.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:24.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:24.809131+0000) 2024-02-19T19:54:25.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:25.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:25.809379+0000) 2024-02-19T19:54:26.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:26.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:26.809672+0000) 2024-02-19T19:54:27.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:27.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:27.809963+0000) 2024-02-19T19:54:28.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:28.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:28.810169+0000) 2024-02-19T19:54:29.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:29.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:29.810365+0000) 2024-02-19T19:54:30.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:30.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:30.810542+0000) 2024-02-19T19:54:31.349 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:54:31.353 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:54:31.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:31.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:31.810783+0000) 2024-02-19T19:54:32.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:32.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:32.810970+0000) 2024-02-19T19:54:33.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:33.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:33.811204+0000) 2024-02-19T19:54:34.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:34.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:34.811462+0000) 2024-02-19T19:54:35.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:35.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:35.811707+0000) 2024-02-19T19:54:36.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:36.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:36.811985+0000) 2024-02-19T19:54:37.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:37.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:37.812216+0000) 2024-02-19T19:54:38.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:38.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:38.812494+0000) 2024-02-19T19:54:39.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:39.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:39.812774+0000) 2024-02-19T19:54:40.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:40.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:40.813087+0000) 2024-02-19T19:54:41.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:41.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:41.813370+0000) 2024-02-19T19:54:42.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:42.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:42.813658+0000) 2024-02-19T19:54:43.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:43.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:43.813894+0000) 2024-02-19T19:54:44.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:44.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:44.814161+0000) 2024-02-19T19:54:45.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:45.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:45.814456+0000) 2024-02-19T19:54:46.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:46.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:46.814780+0000) 2024-02-19T19:54:47.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:47.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:47.814949+0000) 2024-02-19T19:54:48.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:48.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:48.815210+0000) 2024-02-19T19:54:49.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:49.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:49.815434+0000) 2024-02-19T19:54:50.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:50.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:50.815720+0000) 2024-02-19T19:54:51.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:51.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:51.815915+0000) 2024-02-19T19:54:52.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:52.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:52.816201+0000) 2024-02-19T19:54:53.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:53.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:53.816445+0000) 2024-02-19T19:54:54.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:54.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:54.816710+0000) 2024-02-19T19:54:55.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:55.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:55.816994+0000) 2024-02-19T19:54:56.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:56.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:56.817285+0000) 2024-02-19T19:54:57.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:57.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:57.817592+0000) 2024-02-19T19:54:58.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:58.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:58.817837+0000) 2024-02-19T19:54:59.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:54:59.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:54:59.818043+0000) 2024-02-19T19:55:00.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:00.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:00.818270+0000) 2024-02-19T19:55:01.479 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:55:01.482 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:55:01.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:01.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:01.818536+0000) 2024-02-19T19:55:02.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:02.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:02.818849+0000) 2024-02-19T19:55:03.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:03.818+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:03.819021+0000) 2024-02-19T19:55:04.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:04.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:04.819259+0000) 2024-02-19T19:55:05.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:05.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:05.819508+0000) 2024-02-19T19:55:06.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:06.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:06.819782+0000) 2024-02-19T19:55:07.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:07.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:07.820018+0000) 2024-02-19T19:55:08.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:08.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:08.820217+0000) 2024-02-19T19:55:09.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:09.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:09.820434+0000) 2024-02-19T19:55:10.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:10.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:10.820694+0000) 2024-02-19T19:55:11.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:11.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:11.820971+0000) 2024-02-19T19:55:12.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:12.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:12.821232+0000) 2024-02-19T19:55:13.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:13.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:13.821491+0000) 2024-02-19T19:55:14.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:14.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:14.821736+0000) 2024-02-19T19:55:15.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:15.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:15.821921+0000) 2024-02-19T19:55:16.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:16.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:16.822127+0000) 2024-02-19T19:55:17.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:17.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:17.822393+0000) 2024-02-19T19:55:18.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:18.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:18.822597+0000) 2024-02-19T19:55:19.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:19.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:19.822852+0000) 2024-02-19T19:55:20.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:20.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:20.823099+0000) 2024-02-19T19:55:21.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:21.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:21.823342+0000) 2024-02-19T19:55:22.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:22.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:22.823519+0000) 2024-02-19T19:55:23.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:23.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:23.823679+0000) 2024-02-19T19:55:24.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:24.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:24.823900+0000) 2024-02-19T19:55:25.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:25.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:25.824151+0000) 2024-02-19T19:55:26.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:26.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:26.824433+0000) 2024-02-19T19:55:27.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:27.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:27.824680+0000) 2024-02-19T19:55:28.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:28.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:28.824914+0000) 2024-02-19T19:55:29.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:29.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:29.825108+0000) 2024-02-19T19:55:30.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:30.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:30.825354+0000) 2024-02-19T19:55:31.610 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:55:31.613 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:55:31.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:31.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:31.825584+0000) 2024-02-19T19:55:32.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:32.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:32.825833+0000) 2024-02-19T19:55:33.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:33.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:33.826027+0000) 2024-02-19T19:55:34.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:34.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:34.826276+0000) 2024-02-19T19:55:35.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:35.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:35.826522+0000) 2024-02-19T19:55:36.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:36.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:36.826790+0000) 2024-02-19T19:55:37.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:37.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:37.826977+0000) 2024-02-19T19:55:38.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:38.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:38.827154+0000) 2024-02-19T19:55:39.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:39.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:39.827315+0000) 2024-02-19T19:55:40.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:40.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:40.827504+0000) 2024-02-19T19:55:41.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:41.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:41.827726+0000) 2024-02-19T19:55:42.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:42.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:42.827923+0000) 2024-02-19T19:55:43.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:43.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:43.828154+0000) 2024-02-19T19:55:44.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:44.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:44.828405+0000) 2024-02-19T19:55:45.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:45.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:45.828607+0000) 2024-02-19T19:55:46.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:46.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:46.828854+0000) 2024-02-19T19:55:47.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:47.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:47.829102+0000) 2024-02-19T19:55:48.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:48.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:48.829293+0000) 2024-02-19T19:55:49.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:49.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:49.829564+0000) 2024-02-19T19:55:50.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:50.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:50.829771+0000) 2024-02-19T19:55:51.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:51.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:51.829940+0000) 2024-02-19T19:55:52.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:52.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:52.830186+0000) 2024-02-19T19:55:53.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:53.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:53.830362+0000) 2024-02-19T19:55:54.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:54.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:54.830545+0000) 2024-02-19T19:55:55.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:55.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:55.830754+0000) 2024-02-19T19:55:56.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:56.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:56.830917+0000) 2024-02-19T19:55:57.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:57.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:57.831148+0000) 2024-02-19T19:55:58.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:58.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:58.831333+0000) 2024-02-19T19:55:59.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:55:59.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:55:59.831502+0000) 2024-02-19T19:56:00.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:00.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:00.831695+0000) 2024-02-19T19:56:01.716 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:56:01.719 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:56:01.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:01.829+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:01.831954+0000) 2024-02-19T19:56:02.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:02.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:02.832142+0000) 2024-02-19T19:56:03.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:03.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:03.832322+0000) 2024-02-19T19:56:04.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:04.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:04.832625+0000) 2024-02-19T19:56:05.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:05.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:05.832883+0000) 2024-02-19T19:56:06.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:06.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:06.833120+0000) 2024-02-19T19:56:07.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:07.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:07.833422+0000) 2024-02-19T19:56:08.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:08.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:08.833607+0000) 2024-02-19T19:56:09.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:09.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:09.833796+0000) 2024-02-19T19:56:10.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:10.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:10.833973+0000) 2024-02-19T19:56:11.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:11.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:11.834142+0000) 2024-02-19T19:56:12.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:12.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:12.834299+0000) 2024-02-19T19:56:13.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:13.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:13.834439+0000) 2024-02-19T19:56:14.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:14.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:14.834586+0000) 2024-02-19T19:56:15.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:15.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:15.834764+0000) 2024-02-19T19:56:16.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:16.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:16.834917+0000) 2024-02-19T19:56:17.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:17.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:17.835069+0000) 2024-02-19T19:56:18.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:18.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:18.835264+0000) 2024-02-19T19:56:19.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:19.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:19.835543+0000) 2024-02-19T19:56:20.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:20.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:20.835824+0000) 2024-02-19T19:56:21.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:21.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:21.836134+0000) 2024-02-19T19:56:22.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:22.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:22.836445+0000) 2024-02-19T19:56:23.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:23.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:23.836660+0000) 2024-02-19T19:56:24.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:24.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:24.836893+0000) 2024-02-19T19:56:25.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:25.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:25.837055+0000) 2024-02-19T19:56:26.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:26.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:26.837287+0000) 2024-02-19T19:56:27.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:27.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:27.837532+0000) 2024-02-19T19:56:28.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:28.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:28.837744+0000) 2024-02-19T19:56:29.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:29.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:29.838004+0000) 2024-02-19T19:56:30.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:30.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:30.838283+0000) 2024-02-19T19:56:31.830 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:56:31.833 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:56:31.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:31.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:31.838486+0000) 2024-02-19T19:56:32.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:32.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:32.838790+0000) 2024-02-19T19:56:33.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:33.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:33.839018+0000) 2024-02-19T19:56:34.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:34.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:34.839264+0000) 2024-02-19T19:56:35.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:35.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:35.839532+0000) 2024-02-19T19:56:36.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:36.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:36.839795+0000) 2024-02-19T19:56:37.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:37.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:37.840077+0000) 2024-02-19T19:56:38.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:38.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:38.840318+0000) 2024-02-19T19:56:39.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:39.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:39.840587+0000) 2024-02-19T19:56:40.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:40.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:40.840833+0000) 2024-02-19T19:56:41.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:41.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:41.841035+0000) 2024-02-19T19:56:42.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:42.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:42.841201+0000) 2024-02-19T19:56:43.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:43.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:43.841394+0000) 2024-02-19T19:56:44.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:44.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:44.841615+0000) 2024-02-19T19:56:45.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:45.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:45.841901+0000) 2024-02-19T19:56:46.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:46.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:46.842114+0000) 2024-02-19T19:56:47.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:47.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:47.842391+0000) 2024-02-19T19:56:48.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:48.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:48.842688+0000) 2024-02-19T19:56:49.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:49.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:49.843015+0000) 2024-02-19T19:56:50.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:50.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:50.843342+0000) 2024-02-19T19:56:51.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:51.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:51.843650+0000) 2024-02-19T19:56:52.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:52.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:52.843825+0000) 2024-02-19T19:56:53.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:53.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:53.844103+0000) 2024-02-19T19:56:54.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:54.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:54.844388+0000) 2024-02-19T19:56:55.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:55.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:55.844628+0000) 2024-02-19T19:56:56.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:56.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:56.844839+0000) 2024-02-19T19:56:57.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:57.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:57.845010+0000) 2024-02-19T19:56:58.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:58.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:58.845280+0000) 2024-02-19T19:56:59.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:56:59.844+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:56:59.845568+0000) 2024-02-19T19:57:00.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:00.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:00.845858+0000) 2024-02-19T19:57:01.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:01.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:01.846134+0000) 2024-02-19T19:57:01.952 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:57:01.956 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:57:02.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:02.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:02.846358+0000) 2024-02-19T19:57:03.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:03.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:03.846581+0000) 2024-02-19T19:57:04.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:04.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:04.846877+0000) 2024-02-19T19:57:05.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:05.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:05.847161+0000) 2024-02-19T19:57:06.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:06.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:06.847364+0000) 2024-02-19T19:57:07.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:07.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:07.847642+0000) 2024-02-19T19:57:08.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:08.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:08.847852+0000) 2024-02-19T19:57:09.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:09.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:09.848097+0000) 2024-02-19T19:57:10.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:10.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:10.848394+0000) 2024-02-19T19:57:11.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:11.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:11.848625+0000) 2024-02-19T19:57:12.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:12.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:12.848865+0000) 2024-02-19T19:57:13.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:13.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:13.849128+0000) 2024-02-19T19:57:14.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:14.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:14.849355+0000) 2024-02-19T19:57:15.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:15.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:15.849614+0000) 2024-02-19T19:57:16.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:16.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:16.849897+0000) 2024-02-19T19:57:17.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:17.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:17.850190+0000) 2024-02-19T19:57:18.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:18.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:18.850475+0000) 2024-02-19T19:57:19.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:19.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:19.850816+0000) 2024-02-19T19:57:20.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:20.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:20.851021+0000) 2024-02-19T19:57:21.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:21.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:21.851261+0000) 2024-02-19T19:57:22.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:22.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:22.851435+0000) 2024-02-19T19:57:23.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:23.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:23.851632+0000) 2024-02-19T19:57:24.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:24.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:24.851894+0000) 2024-02-19T19:57:25.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:25.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:25.852064+0000) 2024-02-19T19:57:26.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:26.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:26.852246+0000) 2024-02-19T19:57:27.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:27.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:27.852502+0000) 2024-02-19T19:57:28.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:28.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:28.852679+0000) 2024-02-19T19:57:29.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:29.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:29.852945+0000) 2024-02-19T19:57:30.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:30.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:30.853175+0000) 2024-02-19T19:57:31.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:31.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:31.853418+0000) 2024-02-19T19:57:32.074 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:57:32.077 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:57:32.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:32.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:32.853616+0000) 2024-02-19T19:57:33.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:33.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:33.853855+0000) 2024-02-19T19:57:34.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:34.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:34.854094+0000) 2024-02-19T19:57:35.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:35.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:35.854336+0000) 2024-02-19T19:57:36.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:36.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:36.854574+0000) 2024-02-19T19:57:37.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:37.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:37.854786+0000) 2024-02-19T19:57:38.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:38.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:38.855025+0000) 2024-02-19T19:57:39.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:39.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:39.855230+0000) 2024-02-19T19:57:40.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:40.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:40.855412+0000) 2024-02-19T19:57:41.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:41.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:41.855638+0000) 2024-02-19T19:57:42.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:42.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:42.855834+0000) 2024-02-19T19:57:43.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:43.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:43.856024+0000) 2024-02-19T19:57:44.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:44.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:44.856279+0000) 2024-02-19T19:57:45.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:45.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:45.856516+0000) 2024-02-19T19:57:46.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:46.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:46.856692+0000) 2024-02-19T19:57:47.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:47.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:47.856938+0000) 2024-02-19T19:57:48.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:48.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:48.857150+0000) 2024-02-19T19:57:49.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:49.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:49.857410+0000) 2024-02-19T19:57:50.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:50.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:50.857671+0000) 2024-02-19T19:57:51.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:51.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:51.857865+0000) 2024-02-19T19:57:52.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:52.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:52.858114+0000) 2024-02-19T19:57:53.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:53.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:53.858349+0000) 2024-02-19T19:57:54.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:54.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:54.858604+0000) 2024-02-19T19:57:55.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:55.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:55.858857+0000) 2024-02-19T19:57:56.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:56.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:56.859116+0000) 2024-02-19T19:57:57.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:57.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:57.859365+0000) 2024-02-19T19:57:58.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:58.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:58.859498+0000) 2024-02-19T19:57:59.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:57:59.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:57:59.859740+0000) 2024-02-19T19:58:00.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:00.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:00.859967+0000) 2024-02-19T19:58:01.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:01.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:01.860220+0000) 2024-02-19T19:58:02.205 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:58:02.210 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:58:02.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:02.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:02.860402+0000) 2024-02-19T19:58:03.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:03.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:03.860641+0000) 2024-02-19T19:58:04.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:04.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:04.860821+0000) 2024-02-19T19:58:05.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:05.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:05.860995+0000) 2024-02-19T19:58:06.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:06.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:06.861238+0000) 2024-02-19T19:58:07.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:07.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:07.861425+0000) 2024-02-19T19:58:08.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:08.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:08.861671+0000) 2024-02-19T19:58:09.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:09.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:09.861902+0000) 2024-02-19T19:58:10.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:10.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:10.862142+0000) 2024-02-19T19:58:11.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:11.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:11.862371+0000) 2024-02-19T19:58:12.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:12.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:12.862604+0000) 2024-02-19T19:58:13.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:13.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:13.862855+0000) 2024-02-19T19:58:14.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:14.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:14.863043+0000) 2024-02-19T19:58:15.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:15.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:15.863192+0000) 2024-02-19T19:58:16.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:16.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:16.863366+0000) 2024-02-19T19:58:17.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:17.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:17.863606+0000) 2024-02-19T19:58:18.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:18.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:18.863868+0000) 2024-02-19T19:58:19.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:19.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:19.864041+0000) 2024-02-19T19:58:20.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:20.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:20.864286+0000) 2024-02-19T19:58:21.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:21.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:21.864527+0000) 2024-02-19T19:58:22.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:22.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:22.864785+0000) 2024-02-19T19:58:23.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:23.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:23.865045+0000) 2024-02-19T19:58:24.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:24.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:24.865251+0000) 2024-02-19T19:58:25.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:25.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:25.865492+0000) 2024-02-19T19:58:26.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:26.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:26.865742+0000) 2024-02-19T19:58:27.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:27.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:27.865947+0000) 2024-02-19T19:58:28.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:28.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:28.866135+0000) 2024-02-19T19:58:29.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:29.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:29.866318+0000) 2024-02-19T19:58:30.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:30.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:30.866497+0000) 2024-02-19T19:58:31.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:31.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:31.866767+0000) 2024-02-19T19:58:32.286 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:58:32.291 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:58:32.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:32.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:32.866958+0000) 2024-02-19T19:58:33.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:33.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:33.867201+0000) 2024-02-19T19:58:34.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:34.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:34.867450+0000) 2024-02-19T19:58:35.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:35.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:35.867690+0000) 2024-02-19T19:58:36.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:36.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:36.867918+0000) 2024-02-19T19:58:37.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:37.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:37.868171+0000) 2024-02-19T19:58:38.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:38.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:38.868428+0000) 2024-02-19T19:58:39.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:39.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:39.868678+0000) 2024-02-19T19:58:40.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:40.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:40.868867+0000) 2024-02-19T19:58:41.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:41.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:41.869059+0000) 2024-02-19T19:58:42.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:42.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:42.869281+0000) 2024-02-19T19:58:43.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:43.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:43.869528+0000) 2024-02-19T19:58:44.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:44.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:44.869785+0000) 2024-02-19T19:58:45.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:45.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:45.870038+0000) 2024-02-19T19:58:46.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:46.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:46.870325+0000) 2024-02-19T19:58:47.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:47.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:47.870580+0000) 2024-02-19T19:58:48.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:48.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:48.870848+0000) 2024-02-19T19:58:49.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:49.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:49.871117+0000) 2024-02-19T19:58:50.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:50.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:50.871355+0000) 2024-02-19T19:58:51.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:51.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:51.871584+0000) 2024-02-19T19:58:52.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:52.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:52.871838+0000) 2024-02-19T19:58:53.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:53.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:53.872098+0000) 2024-02-19T19:58:54.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:54.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:54.872282+0000) 2024-02-19T19:58:55.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:55.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:55.872528+0000) 2024-02-19T19:58:56.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:56.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:56.872716+0000) 2024-02-19T19:58:57.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:57.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:57.872953+0000) 2024-02-19T19:58:58.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:58.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:58.873203+0000) 2024-02-19T19:58:59.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:58:59.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:58:59.873457+0000) 2024-02-19T19:59:00.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:00.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:00.873715+0000) 2024-02-19T19:59:01.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:01.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:01.873939+0000) 2024-02-19T19:59:02.342 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:59:02.346 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:59:02.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:02.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:02.874202+0000) 2024-02-19T19:59:03.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:03.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:03.874385+0000) 2024-02-19T19:59:04.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:04.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:04.874624+0000) 2024-02-19T19:59:05.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:05.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:05.874927+0000) 2024-02-19T19:59:06.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:06.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:06.875179+0000) 2024-02-19T19:59:07.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:07.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:07.875357+0000) 2024-02-19T19:59:08.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:08.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:08.875514+0000) 2024-02-19T19:59:09.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:09.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:09.875768+0000) 2024-02-19T19:59:10.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:10.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:10.875956+0000) 2024-02-19T19:59:11.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:11.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:11.876130+0000) 2024-02-19T19:59:12.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:12.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:12.876394+0000) 2024-02-19T19:59:13.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:13.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:13.876627+0000) 2024-02-19T19:59:14.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:14.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:14.876801+0000) 2024-02-19T19:59:15.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:15.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:15.877079+0000) 2024-02-19T19:59:16.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:16.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:16.877337+0000) 2024-02-19T19:59:17.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:17.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:17.877648+0000) 2024-02-19T19:59:18.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:18.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:18.877909+0000) 2024-02-19T19:59:19.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:19.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:19.878077+0000) 2024-02-19T19:59:20.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:20.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:20.878256+0000) 2024-02-19T19:59:21.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:21.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:21.878497+0000) 2024-02-19T19:59:22.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:22.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:22.878681+0000) 2024-02-19T19:59:23.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:23.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:23.878979+0000) 2024-02-19T19:59:24.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:24.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:24.879200+0000) 2024-02-19T19:59:25.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:25.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:25.879434+0000) 2024-02-19T19:59:26.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:26.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:26.879622+0000) 2024-02-19T19:59:27.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:27.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:27.879872+0000) 2024-02-19T19:59:28.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:28.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:28.880100+0000) 2024-02-19T19:59:29.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:29.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:29.880327+0000) 2024-02-19T19:59:30.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:30.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:30.880513+0000) 2024-02-19T19:59:31.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:31.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:31.880740+0000) 2024-02-19T19:59:32.447 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:59:32.451 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T19:59:32.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:32.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:32.880940+0000) 2024-02-19T19:59:33.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:33.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:33.881189+0000) 2024-02-19T19:59:34.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:34.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:34.881434+0000) 2024-02-19T19:59:35.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:35.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:35.881691+0000) 2024-02-19T19:59:36.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:36.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:36.881933+0000) 2024-02-19T19:59:37.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:37.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:37.882152+0000) 2024-02-19T19:59:38.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:38.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:38.882413+0000) 2024-02-19T19:59:39.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:39.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:39.882626+0000) 2024-02-19T19:59:40.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:40.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:40.882922+0000) 2024-02-19T19:59:41.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:41.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:41.883068+0000) 2024-02-19T19:59:42.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:42.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:42.883246+0000) 2024-02-19T19:59:43.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:43.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:43.883488+0000) 2024-02-19T19:59:44.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:44.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:44.883757+0000) 2024-02-19T19:59:45.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:45.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:45.883938+0000) 2024-02-19T19:59:46.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:46.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:46.884169+0000) 2024-02-19T19:59:47.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:47.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:47.884407+0000) 2024-02-19T19:59:48.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:48.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:48.884670+0000) 2024-02-19T19:59:49.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:49.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:49.884868+0000) 2024-02-19T19:59:50.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:50.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:50.885105+0000) 2024-02-19T19:59:51.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:51.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:51.885232+0000) 2024-02-19T19:59:52.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:52.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:52.885395+0000) 2024-02-19T19:59:53.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:53.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:53.885609+0000) 2024-02-19T19:59:54.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:54.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:54.885787+0000) 2024-02-19T19:59:55.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:55.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:55.886057+0000) 2024-02-19T19:59:56.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:56.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:56.886328+0000) 2024-02-19T19:59:57.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:57.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:57.886541+0000) 2024-02-19T19:59:58.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:58.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:58.886784+0000) 2024-02-19T19:59:59.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T19:59:59.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T18:59:59.886965+0000) 2024-02-19T20:00:00.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:00.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:00.887227+0000) 2024-02-19T20:00:01.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:01.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:01.887497+0000) 2024-02-19T20:00:02.561 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:00:02.565 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:00:02.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:02.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:02.887709+0000) 2024-02-19T20:00:03.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:03.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:03.887932+0000) 2024-02-19T20:00:04.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:04.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:04.888110+0000) 2024-02-19T20:00:05.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:05.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:05.888346+0000) 2024-02-19T20:00:06.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:06.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:06.888619+0000) 2024-02-19T20:00:07.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:07.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:07.888835+0000) 2024-02-19T20:00:08.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:08.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:08.889076+0000) 2024-02-19T20:00:09.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:09.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:09.889297+0000) 2024-02-19T20:00:10.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:10.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:10.889517+0000) 2024-02-19T20:00:11.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:11.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:11.889711+0000) 2024-02-19T20:00:12.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:12.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:12.889895+0000) 2024-02-19T20:00:13.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:13.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:13.890134+0000) 2024-02-19T20:00:14.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:14.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:14.890386+0000) 2024-02-19T20:00:15.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:15.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:15.890614+0000) 2024-02-19T20:00:16.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:16.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:16.890879+0000) 2024-02-19T20:00:17.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:17.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:17.891088+0000) 2024-02-19T20:00:18.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:18.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:18.891263+0000) 2024-02-19T20:00:19.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:19.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:19.891486+0000) 2024-02-19T20:00:20.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:20.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:20.891715+0000) 2024-02-19T20:00:21.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:21.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:21.891983+0000) 2024-02-19T20:00:22.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:22.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:22.892223+0000) 2024-02-19T20:00:23.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:23.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:23.892415+0000) 2024-02-19T20:00:24.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:24.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:24.892630+0000) 2024-02-19T20:00:25.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:25.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:25.892784+0000) 2024-02-19T20:00:26.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:26.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:26.892970+0000) 2024-02-19T20:00:27.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:27.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:27.893205+0000) 2024-02-19T20:00:28.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:28.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:28.893431+0000) 2024-02-19T20:00:29.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:29.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:29.893645+0000) 2024-02-19T20:00:30.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:30.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:30.893874+0000) 2024-02-19T20:00:31.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:31.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:31.894100+0000) 2024-02-19T20:00:32.692 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:00:32.695 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:00:32.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:32.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:32.894284+0000) 2024-02-19T20:00:33.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:33.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:33.894507+0000) 2024-02-19T20:00:34.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:34.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:34.894794+0000) 2024-02-19T20:00:35.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:35.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:35.895058+0000) 2024-02-19T20:00:36.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:36.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:36.895230+0000) 2024-02-19T20:00:37.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:37.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:37.895412+0000) 2024-02-19T20:00:38.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:38.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:38.895656+0000) 2024-02-19T20:00:39.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:39.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:39.895892+0000) 2024-02-19T20:00:40.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:40.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:40.896131+0000) 2024-02-19T20:00:41.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:41.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:41.896366+0000) 2024-02-19T20:00:42.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:42.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:42.896607+0000) 2024-02-19T20:00:43.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:43.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:43.896760+0000) 2024-02-19T20:00:44.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:44.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:44.896946+0000) 2024-02-19T20:00:45.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:45.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:45.897139+0000) 2024-02-19T20:00:46.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:46.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:46.897355+0000) 2024-02-19T20:00:47.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:47.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:47.897505+0000) 2024-02-19T20:00:48.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:48.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:48.897711+0000) 2024-02-19T20:00:49.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:49.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:49.897939+0000) 2024-02-19T20:00:50.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:50.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:50.898185+0000) 2024-02-19T20:00:51.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:51.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:51.898338+0000) 2024-02-19T20:00:52.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:52.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:52.898535+0000) 2024-02-19T20:00:53.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:53.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:53.898776+0000) 2024-02-19T20:00:54.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:54.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:54.898996+0000) 2024-02-19T20:00:55.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:55.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:55.899138+0000) 2024-02-19T20:00:56.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:56.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:56.899361+0000) 2024-02-19T20:00:57.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:57.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:57.899603+0000) 2024-02-19T20:00:58.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:58.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:58.899882+0000) 2024-02-19T20:00:59.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:00:59.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:00:59.900102+0000) 2024-02-19T20:01:00.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:00.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:00.900344+0000) 2024-02-19T20:01:01.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:01.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:01.900539+0000) 2024-02-19T20:01:02.839 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:01:02.842 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:01:02.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:02.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:02.900685+0000) 2024-02-19T20:01:03.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:03.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:03.900942+0000) 2024-02-19T20:01:04.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:04.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:04.901161+0000) 2024-02-19T20:01:05.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:05.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:05.901389+0000) 2024-02-19T20:01:06.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:06.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:06.901562+0000) 2024-02-19T20:01:07.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:07.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:07.901789+0000) 2024-02-19T20:01:08.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:08.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:08.902032+0000) 2024-02-19T20:01:09.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:09.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:09.902284+0000) 2024-02-19T20:01:10.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:10.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:10.902549+0000) 2024-02-19T20:01:11.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:11.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:11.902758+0000) 2024-02-19T20:01:12.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:12.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:12.902939+0000) 2024-02-19T20:01:13.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:13.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:13.903152+0000) 2024-02-19T20:01:14.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:14.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:14.903397+0000) 2024-02-19T20:01:15.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:15.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:15.903632+0000) 2024-02-19T20:01:16.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:16.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:16.903868+0000) 2024-02-19T20:01:17.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:17.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:17.904115+0000) 2024-02-19T20:01:18.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:18.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:18.904361+0000) 2024-02-19T20:01:19.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:19.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:19.904627+0000) 2024-02-19T20:01:20.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:20.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:20.904866+0000) 2024-02-19T20:01:21.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:21.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:21.905088+0000) 2024-02-19T20:01:22.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:22.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:22.905319+0000) 2024-02-19T20:01:23.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:23.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:23.905576+0000) 2024-02-19T20:01:24.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:24.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:24.905751+0000) 2024-02-19T20:01:25.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:25.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:25.905970+0000) 2024-02-19T20:01:26.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:26.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:26.906230+0000) 2024-02-19T20:01:27.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:27.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:27.906480+0000) 2024-02-19T20:01:28.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:28.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:28.906684+0000) 2024-02-19T20:01:29.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:29.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:29.906930+0000) 2024-02-19T20:01:30.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:30.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:30.907127+0000) 2024-02-19T20:01:31.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:31.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:31.907287+0000) 2024-02-19T20:01:32.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:32.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:32.907519+0000) 2024-02-19T20:01:32.972 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:01:32.975 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:01:33.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:33.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:33.907784+0000) 2024-02-19T20:01:34.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:34.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:34.907972+0000) 2024-02-19T20:01:35.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:35.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:35.908242+0000) 2024-02-19T20:01:36.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:36.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:36.908418+0000) 2024-02-19T20:01:37.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:37.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:37.908660+0000) 2024-02-19T20:01:38.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:38.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:38.908903+0000) 2024-02-19T20:01:39.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:39.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:39.909140+0000) 2024-02-19T20:01:40.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:40.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:40.909377+0000) 2024-02-19T20:01:41.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:41.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:41.909569+0000) 2024-02-19T20:01:42.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:42.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:42.909757+0000) 2024-02-19T20:01:43.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:43.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:43.910014+0000) 2024-02-19T20:01:44.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:44.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:44.910144+0000) 2024-02-19T20:01:45.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:45.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:45.910344+0000) 2024-02-19T20:01:46.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:46.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:46.910613+0000) 2024-02-19T20:01:47.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:47.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:47.910815+0000) 2024-02-19T20:01:48.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:48.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:48.911053+0000) 2024-02-19T20:01:49.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:49.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:49.911291+0000) 2024-02-19T20:01:50.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:50.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:50.911463+0000) 2024-02-19T20:01:51.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:51.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:51.911651+0000) 2024-02-19T20:01:52.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:52.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:52.911834+0000) 2024-02-19T20:01:53.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:53.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:53.912036+0000) 2024-02-19T20:01:54.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:54.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:54.912215+0000) 2024-02-19T20:01:55.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:55.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:55.912443+0000) 2024-02-19T20:01:56.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:56.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:56.912625+0000) 2024-02-19T20:01:57.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:57.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:57.912875+0000) 2024-02-19T20:01:58.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:58.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:58.913045+0000) 2024-02-19T20:01:59.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:01:59.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:01:59.913218+0000) 2024-02-19T20:02:00.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:00.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:00.913396+0000) 2024-02-19T20:02:01.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:01.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:01.913578+0000) 2024-02-19T20:02:02.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:02.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:02.913747+0000) 2024-02-19T20:02:03.054 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:02:03.058 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:02:03.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:03.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:03.913917+0000) 2024-02-19T20:02:04.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:04.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:04.914048+0000) 2024-02-19T20:02:05.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:05.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:05.914230+0000) 2024-02-19T20:02:06.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:06.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:06.914365+0000) 2024-02-19T20:02:07.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:07.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:07.914533+0000) 2024-02-19T20:02:08.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:08.910+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:08.914679+0000) 2024-02-19T20:02:09.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:09.910+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:09.914785+0000) 2024-02-19T20:02:10.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:10.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:10.914914+0000) 2024-02-19T20:02:11.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:11.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:11.915100+0000) 2024-02-19T20:02:12.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:12.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:12.915217+0000) 2024-02-19T20:02:13.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:13.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:13.915392+0000) 2024-02-19T20:02:14.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:14.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:14.915518+0000) 2024-02-19T20:02:15.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:15.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:15.915753+0000) 2024-02-19T20:02:16.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:16.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:16.915905+0000) 2024-02-19T20:02:17.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:17.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:17.916132+0000) 2024-02-19T20:02:18.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:18.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:18.916292+0000) 2024-02-19T20:02:19.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:19.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:19.916434+0000) 2024-02-19T20:02:20.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:20.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:20.916583+0000) 2024-02-19T20:02:21.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:21.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:21.916835+0000) 2024-02-19T20:02:22.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:22.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:22.916979+0000) 2024-02-19T20:02:23.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:23.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:23.917214+0000) 2024-02-19T20:02:24.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:24.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:24.917374+0000) 2024-02-19T20:02:25.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:25.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:25.917529+0000) 2024-02-19T20:02:26.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:26.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:26.917703+0000) 2024-02-19T20:02:27.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:27.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:27.917882+0000) 2024-02-19T20:02:28.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:28.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:28.918060+0000) 2024-02-19T20:02:29.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:29.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:29.918281+0000) 2024-02-19T20:02:30.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:30.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:30.918461+0000) 2024-02-19T20:02:31.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:31.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:31.918645+0000) 2024-02-19T20:02:32.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:32.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:32.918772+0000) 2024-02-19T20:02:33.181 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:02:33.184 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:02:33.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:33.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:33.918980+0000) 2024-02-19T20:02:34.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:34.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:34.919170+0000) 2024-02-19T20:02:35.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:35.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:35.919403+0000) 2024-02-19T20:02:36.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:36.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:36.919647+0000) 2024-02-19T20:02:37.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:37.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:37.919859+0000) 2024-02-19T20:02:38.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:38.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:38.920067+0000) 2024-02-19T20:02:39.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:39.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:39.920255+0000) 2024-02-19T20:02:40.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:40.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:40.920512+0000) 2024-02-19T20:02:41.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:41.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:41.920717+0000) 2024-02-19T20:02:42.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:42.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:42.920966+0000) 2024-02-19T20:02:43.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:43.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:43.921202+0000) 2024-02-19T20:02:44.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:44.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:44.921471+0000) 2024-02-19T20:02:45.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:45.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:45.921639+0000) 2024-02-19T20:02:46.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:46.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:46.921844+0000) 2024-02-19T20:02:47.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:47.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:47.922088+0000) 2024-02-19T20:02:48.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:48.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:48.922292+0000) 2024-02-19T20:02:49.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:49.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:49.922517+0000) 2024-02-19T20:02:50.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:50.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:50.922818+0000) 2024-02-19T20:02:51.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:51.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:51.923031+0000) 2024-02-19T20:02:52.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:52.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:52.923286+0000) 2024-02-19T20:02:53.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:53.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:53.923518+0000) 2024-02-19T20:02:54.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:54.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:54.923728+0000) 2024-02-19T20:02:55.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:55.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:55.923948+0000) 2024-02-19T20:02:56.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:56.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:56.924185+0000) 2024-02-19T20:02:57.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:57.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:57.924446+0000) 2024-02-19T20:02:58.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:58.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:58.924651+0000) 2024-02-19T20:02:59.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:02:59.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:02:59.924823+0000) 2024-02-19T20:03:00.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:00.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:00.925074+0000) 2024-02-19T20:03:01.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:01.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:01.925320+0000) 2024-02-19T20:03:02.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:02.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:02.925560+0000) 2024-02-19T20:03:03.311 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:03:03.315 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:03:03.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:03.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:03.925728+0000) 2024-02-19T20:03:04.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:04.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:04.925928+0000) 2024-02-19T20:03:05.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:05.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:05.926106+0000) 2024-02-19T20:03:06.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:06.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:06.926320+0000) 2024-02-19T20:03:07.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:07.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:07.926547+0000) 2024-02-19T20:03:08.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:08.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:08.926739+0000) 2024-02-19T20:03:09.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:09.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:09.926951+0000) 2024-02-19T20:03:10.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:10.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:10.927132+0000) 2024-02-19T20:03:11.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:11.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:11.927361+0000) 2024-02-19T20:03:12.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:12.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:12.927532+0000) 2024-02-19T20:03:13.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:13.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:13.927777+0000) 2024-02-19T20:03:14.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:14.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:14.928013+0000) 2024-02-19T20:03:15.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:15.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:15.928201+0000) 2024-02-19T20:03:16.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:16.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:16.928377+0000) 2024-02-19T20:03:17.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:17.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:17.928634+0000) 2024-02-19T20:03:18.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:18.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:18.928866+0000) 2024-02-19T20:03:19.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:19.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:19.929068+0000) 2024-02-19T20:03:20.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:20.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:20.929237+0000) 2024-02-19T20:03:21.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:21.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:21.929374+0000) 2024-02-19T20:03:22.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:22.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:22.929586+0000) 2024-02-19T20:03:23.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:23.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:23.929833+0000) 2024-02-19T20:03:24.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:24.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:24.930012+0000) 2024-02-19T20:03:25.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:25.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:25.930219+0000) 2024-02-19T20:03:26.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:26.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:26.930479+0000) 2024-02-19T20:03:27.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:27.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:27.930688+0000) 2024-02-19T20:03:28.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:28.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:28.930957+0000) 2024-02-19T20:03:29.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:29.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:29.931161+0000) 2024-02-19T20:03:30.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:30.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:30.931385+0000) 2024-02-19T20:03:31.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:31.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:31.931617+0000) 2024-02-19T20:03:32.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:32.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:32.931805+0000) 2024-02-19T20:03:33.384 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:03:33.387 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:03:33.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:33.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:33.932069+0000) 2024-02-19T20:03:34.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:34.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:34.932345+0000) 2024-02-19T20:03:35.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:35.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:35.932578+0000) 2024-02-19T20:03:36.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:36.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:36.932826+0000) 2024-02-19T20:03:37.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:37.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:37.933071+0000) 2024-02-19T20:03:38.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:38.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:38.933311+0000) 2024-02-19T20:03:39.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:39.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:39.933497+0000) 2024-02-19T20:03:40.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:40.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:40.933696+0000) 2024-02-19T20:03:41.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:41.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:41.933937+0000) 2024-02-19T20:03:42.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:42.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:42.934178+0000) 2024-02-19T20:03:43.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:43.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:43.934391+0000) 2024-02-19T20:03:44.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:44.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:44.934642+0000) 2024-02-19T20:03:45.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:45.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:45.934804+0000) 2024-02-19T20:03:46.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:46.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:46.935062+0000) 2024-02-19T20:03:47.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:47.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:47.935324+0000) 2024-02-19T20:03:48.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:48.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:48.935503+0000) 2024-02-19T20:03:49.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:49.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:49.935730+0000) 2024-02-19T20:03:50.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:50.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:50.935974+0000) 2024-02-19T20:03:51.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:51.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:51.936242+0000) 2024-02-19T20:03:52.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:52.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:52.936415+0000) 2024-02-19T20:03:53.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:53.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:53.936539+0000) 2024-02-19T20:03:54.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:54.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:54.936760+0000) 2024-02-19T20:03:55.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:55.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:55.936995+0000) 2024-02-19T20:03:56.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:56.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:56.937233+0000) 2024-02-19T20:03:57.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:57.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:57.937411+0000) 2024-02-19T20:03:58.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:58.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:58.937580+0000) 2024-02-19T20:03:59.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:03:59.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:03:59.937761+0000) 2024-02-19T20:04:00.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:00.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:00.937971+0000) 2024-02-19T20:04:01.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:01.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:01.938151+0000) 2024-02-19T20:04:02.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:02.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:02.938377+0000) 2024-02-19T20:04:03.524 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:04:03.527 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:04:03.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:03.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:03.938618+0000) 2024-02-19T20:04:04.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:04.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:04.938812+0000) 2024-02-19T20:04:05.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:05.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:05.939036+0000) 2024-02-19T20:04:06.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:06.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:06.939295+0000) 2024-02-19T20:04:07.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:07.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:07.939559+0000) 2024-02-19T20:04:08.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:08.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:08.939803+0000) 2024-02-19T20:04:09.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:09.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:09.940030+0000) 2024-02-19T20:04:10.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:10.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:10.940217+0000) 2024-02-19T20:04:11.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:11.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:11.940373+0000) 2024-02-19T20:04:12.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:12.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:12.940607+0000) 2024-02-19T20:04:13.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:13.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:13.940829+0000) 2024-02-19T20:04:14.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:14.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:14.941013+0000) 2024-02-19T20:04:15.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:15.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:15.941258+0000) 2024-02-19T20:04:16.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:16.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:16.941509+0000) 2024-02-19T20:04:17.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:17.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:17.941788+0000) 2024-02-19T20:04:18.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:18.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:18.942033+0000) 2024-02-19T20:04:19.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:19.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:19.942269+0000) 2024-02-19T20:04:20.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:20.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:20.942527+0000) 2024-02-19T20:04:21.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:21.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:21.942784+0000) 2024-02-19T20:04:22.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:22.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:22.943011+0000) 2024-02-19T20:04:23.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:23.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:23.943186+0000) 2024-02-19T20:04:24.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:24.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:24.943410+0000) 2024-02-19T20:04:25.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:25.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:25.943654+0000) 2024-02-19T20:04:26.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:26.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:26.943830+0000) 2024-02-19T20:04:27.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:27.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:27.944091+0000) 2024-02-19T20:04:28.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:28.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:28.944333+0000) 2024-02-19T20:04:29.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:29.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:29.944574+0000) 2024-02-19T20:04:30.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:30.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:30.944821+0000) 2024-02-19T20:04:31.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:31.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:31.945111+0000) 2024-02-19T20:04:32.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:32.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:32.945374+0000) 2024-02-19T20:04:33.662 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:04:33.666 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:04:33.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:33.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:33.945612+0000) 2024-02-19T20:04:34.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:34.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:34.945839+0000) 2024-02-19T20:04:35.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:35.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:35.946023+0000) 2024-02-19T20:04:36.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:36.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:36.946228+0000) 2024-02-19T20:04:37.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:37.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:37.946418+0000) 2024-02-19T20:04:38.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:38.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:38.946678+0000) 2024-02-19T20:04:39.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:39.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:39.946894+0000) 2024-02-19T20:04:40.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:40.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:40.947134+0000) 2024-02-19T20:04:41.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:41.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:41.947406+0000) 2024-02-19T20:04:42.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:42.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:42.947692+0000) 2024-02-19T20:04:43.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:43.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:43.947891+0000) 2024-02-19T20:04:44.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:44.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:44.948071+0000) 2024-02-19T20:04:45.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:45.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:45.948311+0000) 2024-02-19T20:04:46.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:46.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:46.948521+0000) 2024-02-19T20:04:47.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:47.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:47.948751+0000) 2024-02-19T20:04:48.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:48.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:48.948986+0000) 2024-02-19T20:04:49.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:49.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:49.949179+0000) 2024-02-19T20:04:50.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:50.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:50.949416+0000) 2024-02-19T20:04:51.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:51.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:51.949712+0000) 2024-02-19T20:04:52.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:52.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:52.950008+0000) 2024-02-19T20:04:53.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:53.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:53.950223+0000) 2024-02-19T20:04:54.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:54.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:54.950424+0000) 2024-02-19T20:04:55.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:55.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:55.950630+0000) 2024-02-19T20:04:56.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:56.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:56.950899+0000) 2024-02-19T20:04:57.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:57.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:57.951209+0000) 2024-02-19T20:04:58.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:58.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:58.951430+0000) 2024-02-19T20:04:59.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:04:59.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:04:59.951719+0000) 2024-02-19T20:05:00.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:00.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:00.952041+0000) 2024-02-19T20:05:01.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:01.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:01.952323+0000) 2024-02-19T20:05:02.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:02.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:02.952523+0000) 2024-02-19T20:05:03.784 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:05:03.789 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:05:03.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:03.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:03.952786+0000) 2024-02-19T20:05:04.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:04.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:04.953012+0000) 2024-02-19T20:05:05.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:05.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:05.953285+0000) 2024-02-19T20:05:06.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:06.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:06.953578+0000) 2024-02-19T20:05:07.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:07.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:07.953789+0000) 2024-02-19T20:05:08.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:08.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:08.954054+0000) 2024-02-19T20:05:09.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:09.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:09.954339+0000) 2024-02-19T20:05:10.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:10.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:10.954518+0000) 2024-02-19T20:05:11.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:11.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:11.954818+0000) 2024-02-19T20:05:12.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:12.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:12.955052+0000) 2024-02-19T20:05:13.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:13.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:13.955279+0000) 2024-02-19T20:05:14.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:14.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:14.955454+0000) 2024-02-19T20:05:15.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:15.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:15.955743+0000) 2024-02-19T20:05:16.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:16.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:16.955946+0000) 2024-02-19T20:05:17.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:17.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:17.956226+0000) 2024-02-19T20:05:18.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:18.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:18.956509+0000) 2024-02-19T20:05:19.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:19.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:19.956734+0000) 2024-02-19T20:05:20.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:20.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:20.956926+0000) 2024-02-19T20:05:21.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:21.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:21.957124+0000) 2024-02-19T20:05:22.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:22.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:22.957388+0000) 2024-02-19T20:05:23.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:23.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:23.957682+0000) 2024-02-19T20:05:24.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:24.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:24.957918+0000) 2024-02-19T20:05:25.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:25.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:25.958177+0000) 2024-02-19T20:05:26.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:26.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:26.958433+0000) 2024-02-19T20:05:27.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:27.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:27.958772+0000) 2024-02-19T20:05:28.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:28.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:28.959067+0000) 2024-02-19T20:05:29.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:29.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:29.959366+0000) 2024-02-19T20:05:30.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:30.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:30.959655+0000) 2024-02-19T20:05:31.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:31.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:31.959927+0000) 2024-02-19T20:05:32.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:32.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:32.960182+0000) 2024-02-19T20:05:33.857 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:05:33.862 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:05:33.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:33.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:33.960419+0000) 2024-02-19T20:05:34.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:34.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:34.960601+0000) 2024-02-19T20:05:35.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:35.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:35.960862+0000) 2024-02-19T20:05:36.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:36.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:36.961044+0000) 2024-02-19T20:05:37.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:37.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:37.961347+0000) 2024-02-19T20:05:38.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:38.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:38.961644+0000) 2024-02-19T20:05:39.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:39.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:39.961936+0000) 2024-02-19T20:05:40.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:40.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:40.962214+0000) 2024-02-19T20:05:41.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:41.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:41.962410+0000) 2024-02-19T20:05:42.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:42.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:42.962645+0000) 2024-02-19T20:05:43.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:43.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:43.962905+0000) 2024-02-19T20:05:44.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:44.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:44.963092+0000) 2024-02-19T20:05:45.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:45.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:45.963335+0000) 2024-02-19T20:05:46.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:46.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:46.963579+0000) 2024-02-19T20:05:47.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:47.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:47.963760+0000) 2024-02-19T20:05:48.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:48.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:48.964000+0000) 2024-02-19T20:05:49.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:49.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:49.964197+0000) 2024-02-19T20:05:50.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:50.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:50.964439+0000) 2024-02-19T20:05:51.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:51.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:51.964617+0000) 2024-02-19T20:05:52.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:52.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:52.964898+0000) 2024-02-19T20:05:53.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:53.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:53.965105+0000) 2024-02-19T20:05:54.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:54.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:54.965305+0000) 2024-02-19T20:05:55.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:55.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:55.965506+0000) 2024-02-19T20:05:56.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:56.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:56.965773+0000) 2024-02-19T20:05:57.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:57.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:57.965949+0000) 2024-02-19T20:05:58.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:58.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:58.966118+0000) 2024-02-19T20:05:59.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:05:59.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:59.966388+0000) 2024-02-19T20:06:00.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:00.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:00.966670+0000) 2024-02-19T20:06:01.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:01.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:01.967000+0000) 2024-02-19T20:06:02.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:02.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:02.967302+0000) 2024-02-19T20:06:03.913 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:06:03.916 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:06:03.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:03.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:03.967448+0000) 2024-02-19T20:06:04.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:04.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:04.967635+0000) 2024-02-19T20:06:05.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:05.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:05.967788+0000) 2024-02-19T20:06:06.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:06.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:06.967956+0000) 2024-02-19T20:06:07.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:07.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:07.968185+0000) 2024-02-19T20:06:08.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:08.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:08.968439+0000) 2024-02-19T20:06:09.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:09.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:09.968641+0000) 2024-02-19T20:06:10.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:10.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:10.968785+0000) 2024-02-19T20:06:11.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:11.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:11.968974+0000) 2024-02-19T20:06:12.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:12.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:12.969174+0000) 2024-02-19T20:06:13.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:13.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:13.969324+0000) 2024-02-19T20:06:14.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:14.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:14.969509+0000) 2024-02-19T20:06:15.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:15.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:15.969732+0000) 2024-02-19T20:06:16.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:16.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:16.969914+0000) 2024-02-19T20:06:17.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:17.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:17.970104+0000) 2024-02-19T20:06:18.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:18.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:18.970307+0000) 2024-02-19T20:06:19.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:19.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:19.970471+0000) 2024-02-19T20:06:20.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:20.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:20.970639+0000) 2024-02-19T20:06:21.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:21.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:21.970796+0000) 2024-02-19T20:06:22.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:22.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:22.970924+0000) 2024-02-19T20:06:23.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:23.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:23.971062+0000) 2024-02-19T20:06:24.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:24.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:24.971243+0000) 2024-02-19T20:06:25.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:25.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:25.971401+0000) 2024-02-19T20:06:26.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:26.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:26.971556+0000) 2024-02-19T20:06:27.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:27.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:27.971751+0000) 2024-02-19T20:06:28.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:28.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:28.971979+0000) 2024-02-19T20:06:29.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:29.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:29.972189+0000) 2024-02-19T20:06:30.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:30.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:30.972329+0000) 2024-02-19T20:06:31.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:31.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:31.972557+0000) 2024-02-19T20:06:32.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:32.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:32.972777+0000) 2024-02-19T20:06:33.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:33.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:33.973019+0000) 2024-02-19T20:06:34.027 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:06:34.031 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:06:34.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:34.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:34.973240+0000) 2024-02-19T20:06:35.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:35.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:35.973501+0000) 2024-02-19T20:06:36.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:36.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:36.973708+0000) 2024-02-19T20:06:37.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:37.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:37.973911+0000) 2024-02-19T20:06:38.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:38.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:38.974093+0000) 2024-02-19T20:06:39.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:39.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:39.974357+0000) 2024-02-19T20:06:40.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:40.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:40.974583+0000) 2024-02-19T20:06:41.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:41.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:41.974821+0000) 2024-02-19T20:06:42.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:42.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:42.975084+0000) 2024-02-19T20:06:43.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:43.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:43.975320+0000) 2024-02-19T20:06:44.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:44.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:44.975454+0000) 2024-02-19T20:06:45.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:45.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:45.975628+0000) 2024-02-19T20:06:46.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:46.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:46.975754+0000) 2024-02-19T20:06:47.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:47.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:47.975918+0000) 2024-02-19T20:06:48.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:48.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:48.976111+0000) 2024-02-19T20:06:49.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:49.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:49.976361+0000) 2024-02-19T20:06:50.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:50.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:50.976581+0000) 2024-02-19T20:06:51.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:51.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:51.976773+0000) 2024-02-19T20:06:52.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:52.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:52.976900+0000) 2024-02-19T20:06:53.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:53.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:53.977027+0000) 2024-02-19T20:06:54.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:54.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:54.977213+0000) 2024-02-19T20:06:55.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:55.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:55.977388+0000) 2024-02-19T20:06:56.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:56.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:56.977543+0000) 2024-02-19T20:06:57.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:57.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:57.977764+0000) 2024-02-19T20:06:58.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:58.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:58.977956+0000) 2024-02-19T20:06:59.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:06:59.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:59.978083+0000) 2024-02-19T20:07:00.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:00.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:00.978301+0000) 2024-02-19T20:07:01.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:01.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:01.978548+0000) 2024-02-19T20:07:02.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:02.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:02.978784+0000) 2024-02-19T20:07:03.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:03.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:03.978935+0000) 2024-02-19T20:07:04.106 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:07:04.111 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:07:04.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:04.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:04.979086+0000) 2024-02-19T20:07:05.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:05.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:05.979260+0000) 2024-02-19T20:07:06.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:06.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:06.979426+0000) 2024-02-19T20:07:07.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:07.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:07.979597+0000) 2024-02-19T20:07:08.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:08.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:08.979824+0000) 2024-02-19T20:07:09.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:09.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:09.979999+0000) 2024-02-19T20:07:10.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:10.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:10.980161+0000) 2024-02-19T20:07:11.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:11.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:11.980334+0000) 2024-02-19T20:07:12.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:12.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:12.980480+0000) 2024-02-19T20:07:13.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:13.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:13.980644+0000) 2024-02-19T20:07:14.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:14.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:14.980826+0000) 2024-02-19T20:07:15.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:15.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:15.981014+0000) 2024-02-19T20:07:16.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:16.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:16.981185+0000) 2024-02-19T20:07:17.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:17.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:17.981384+0000) 2024-02-19T20:07:18.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:18.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:18.981581+0000) 2024-02-19T20:07:19.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:19.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:19.981763+0000) 2024-02-19T20:07:20.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:20.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:20.981953+0000) 2024-02-19T20:07:21.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:21.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:21.982144+0000) 2024-02-19T20:07:22.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:22.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:22.982290+0000) 2024-02-19T20:07:23.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:23.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:23.982459+0000) 2024-02-19T20:07:24.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:24.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:24.982679+0000) 2024-02-19T20:07:25.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:25.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:25.982942+0000) 2024-02-19T20:07:26.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:26.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:26.983076+0000) 2024-02-19T20:07:27.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:27.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:27.983268+0000) 2024-02-19T20:07:28.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:28.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:28.983438+0000) 2024-02-19T20:07:29.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:29.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:29.983683+0000) 2024-02-19T20:07:30.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:30.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:30.983939+0000) 2024-02-19T20:07:31.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:31.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:31.984157+0000) 2024-02-19T20:07:32.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:32.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:32.984295+0000) 2024-02-19T20:07:33.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:33.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:33.984558+0000) 2024-02-19T20:07:34.221 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:07:34.226 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:07:34.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:34.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:34.984813+0000) 2024-02-19T20:07:35.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:35.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:35.984996+0000) 2024-02-19T20:07:36.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:36.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:36.985216+0000) 2024-02-19T20:07:37.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:37.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:37.985466+0000) 2024-02-19T20:07:38.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:38.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:38.985711+0000) 2024-02-19T20:07:39.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:39.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:39.985949+0000) 2024-02-19T20:07:40.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:40.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:40.986145+0000) 2024-02-19T20:07:41.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:41.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:41.986358+0000) 2024-02-19T20:07:42.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:42.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:42.986595+0000) 2024-02-19T20:07:43.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:43.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:43.986840+0000) 2024-02-19T20:07:44.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:44.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:44.987035+0000) 2024-02-19T20:07:45.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:45.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:45.987220+0000) 2024-02-19T20:07:46.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:46.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:46.987482+0000) 2024-02-19T20:07:47.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:47.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:47.987739+0000) 2024-02-19T20:07:48.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:48.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:48.987987+0000) 2024-02-19T20:07:49.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:49.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:49.988157+0000) 2024-02-19T20:07:50.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:50.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:50.988357+0000) 2024-02-19T20:07:51.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:51.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:51.988544+0000) 2024-02-19T20:07:52.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:52.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:52.988719+0000) 2024-02-19T20:07:53.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:53.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:53.988952+0000) 2024-02-19T20:07:54.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:54.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:54.989241+0000) 2024-02-19T20:07:55.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:55.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:55.989442+0000) 2024-02-19T20:07:56.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:56.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:56.989721+0000) 2024-02-19T20:07:57.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:57.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:57.989897+0000) 2024-02-19T20:07:58.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:58.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:58.990086+0000) 2024-02-19T20:07:59.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:07:59.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:59.990309+0000) 2024-02-19T20:08:00.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:00.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:00.990569+0000) 2024-02-19T20:08:01.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:01.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:01.990837+0000) 2024-02-19T20:08:02.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:02.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:02.990960+0000) 2024-02-19T20:08:03.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:03.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:03.991115+0000) 2024-02-19T20:08:04.310 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:08:04.315 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:08:04.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:04.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:04.991309+0000) 2024-02-19T20:08:05.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:05.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:05.991562+0000) 2024-02-19T20:08:06.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:06.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:06.991778+0000) 2024-02-19T20:08:07.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:07.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:07.991980+0000) 2024-02-19T20:08:08.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:08.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:08.992153+0000) 2024-02-19T20:08:09.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:09.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:09.992363+0000) 2024-02-19T20:08:10.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:10.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:10.992609+0000) 2024-02-19T20:08:11.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:11.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:11.992838+0000) 2024-02-19T20:08:12.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:12.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:12.993075+0000) 2024-02-19T20:08:13.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:13.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:13.993265+0000) 2024-02-19T20:08:14.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:14.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:14.993465+0000) 2024-02-19T20:08:15.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:15.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:15.993738+0000) 2024-02-19T20:08:16.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:16.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:16.993976+0000) 2024-02-19T20:08:17.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:17.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:17.994230+0000) 2024-02-19T20:08:18.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:18.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:18.994456+0000) 2024-02-19T20:08:19.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:19.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:19.994697+0000) 2024-02-19T20:08:20.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:20.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:20.994878+0000) 2024-02-19T20:08:21.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:21.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:21.995049+0000) 2024-02-19T20:08:22.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:22.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:22.995201+0000) 2024-02-19T20:08:23.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:23.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:23.995337+0000) 2024-02-19T20:08:24.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:24.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:24.995597+0000) 2024-02-19T20:08:25.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:25.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:25.995851+0000) 2024-02-19T20:08:26.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:26.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:26.996152+0000) 2024-02-19T20:08:27.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:27.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:27.996394+0000) 2024-02-19T20:08:28.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:28.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:28.996628+0000) 2024-02-19T20:08:29.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:29.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:29.996863+0000) 2024-02-19T20:08:30.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:30.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:30.997098+0000) 2024-02-19T20:08:31.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:31.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:31.997332+0000) 2024-02-19T20:08:32.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:32.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:32.997565+0000) 2024-02-19T20:08:33.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:33.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:33.997791+0000) 2024-02-19T20:08:34.383 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:08:34.386 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:08:34.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:34.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:34.997987+0000) 2024-02-19T20:08:35.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:35.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:35.998164+0000) 2024-02-19T20:08:36.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:36.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:36.998352+0000) 2024-02-19T20:08:37.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:37.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:37.998601+0000) 2024-02-19T20:08:38.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:38.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:38.998840+0000) 2024-02-19T20:08:39.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:39.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:39.999015+0000) 2024-02-19T20:08:40.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:40.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:40.999205+0000) 2024-02-19T20:08:41.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:41.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:41.999354+0000) 2024-02-19T20:08:42.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:42.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:42.999618+0000) 2024-02-19T20:08:43.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:43.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:43.999807+0000) 2024-02-19T20:08:44.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:44.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:44.999999+0000) 2024-02-19T20:08:45.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:45.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:46.000232+0000) 2024-02-19T20:08:46.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:46.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:47.000461+0000) 2024-02-19T20:08:48.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:47.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:48.000714+0000) 2024-02-19T20:08:49.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:48.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:49.000971+0000) 2024-02-19T20:08:50.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:49.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:50.001203+0000) 2024-02-19T20:08:51.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:50.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:51.001330+0000) 2024-02-19T20:08:52.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:51.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:52.001487+0000) 2024-02-19T20:08:53.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:52.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:53.001721+0000) 2024-02-19T20:08:54.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:53.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:54.001894+0000) 2024-02-19T20:08:55.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:54.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:55.002070+0000) 2024-02-19T20:08:56.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:55.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:56.002320+0000) 2024-02-19T20:08:57.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:56.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:57.002522+0000) 2024-02-19T20:08:58.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:57.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:58.002742+0000) 2024-02-19T20:08:59.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:58.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:59.003002+0000) 2024-02-19T20:09:00.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:08:59.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:00.003159+0000) 2024-02-19T20:09:01.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:00.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:01.003398+0000) 2024-02-19T20:09:02.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:01.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:02.003653+0000) 2024-02-19T20:09:03.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:02.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:03.003876+0000) 2024-02-19T20:09:04.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:03.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:04.004144+0000) 2024-02-19T20:09:04.530 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:09:04.533 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:09:05.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:05.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:05.004411+0000) 2024-02-19T20:09:06.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:05.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:06.004638+0000) 2024-02-19T20:09:07.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:06.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:07.004864+0000) 2024-02-19T20:09:08.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:07.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:08.005046+0000) 2024-02-19T20:09:09.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:08.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:09.005283+0000) 2024-02-19T20:09:10.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:09.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:10.005537+0000) 2024-02-19T20:09:11.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:10.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:11.005781+0000) 2024-02-19T20:09:12.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:12.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:12.006020+0000) 2024-02-19T20:09:13.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:12.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:13.006245+0000) 2024-02-19T20:09:14.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:13.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:14.006397+0000) 2024-02-19T20:09:15.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:15.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:15.006662+0000) 2024-02-19T20:09:16.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:16.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:16.006838+0000) 2024-02-19T20:09:17.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:17.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:17.007092+0000) 2024-02-19T20:09:18.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:18.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:18.007370+0000) 2024-02-19T20:09:19.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:19.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:19.007643+0000) 2024-02-19T20:09:20.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:20.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:20.007908+0000) 2024-02-19T20:09:21.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:21.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:21.008079+0000) 2024-02-19T20:09:22.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:22.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:22.008299+0000) 2024-02-19T20:09:23.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:23.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:23.008512+0000) 2024-02-19T20:09:24.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:24.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:24.008718+0000) 2024-02-19T20:09:25.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:25.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:25.008901+0000) 2024-02-19T20:09:26.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:26.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:26.009098+0000) 2024-02-19T20:09:27.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:27.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:27.009347+0000) 2024-02-19T20:09:28.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:28.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:28.009532+0000) 2024-02-19T20:09:29.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:29.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:29.009708+0000) 2024-02-19T20:09:30.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:30.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:30.009980+0000) 2024-02-19T20:09:31.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:31.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:31.010173+0000) 2024-02-19T20:09:32.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:32.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:32.010413+0000) 2024-02-19T20:09:33.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:33.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:33.010652+0000) 2024-02-19T20:09:34.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:34.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:34.010883+0000) 2024-02-19T20:09:34.661 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:09:34.664 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:09:35.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:35.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:35.011131+0000) 2024-02-19T20:09:36.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:36.011+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:36.011327+0000) 2024-02-19T20:09:37.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:37.011+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:37.011543+0000) 2024-02-19T20:09:38.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:38.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:38.011776+0000) 2024-02-19T20:09:39.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:39.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:39.011948+0000) 2024-02-19T20:09:40.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:40.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:40.012089+0000) 2024-02-19T20:09:41.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:41.011+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:41.012322+0000) 2024-02-19T20:09:42.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:42.011+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:42.012543+0000) 2024-02-19T20:09:43.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:43.011+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:43.012787+0000) 2024-02-19T20:09:44.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:44.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:44.013025+0000) 2024-02-19T20:09:45.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:45.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:45.013278+0000) 2024-02-19T20:09:46.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:46.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:46.013518+0000) 2024-02-19T20:09:47.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:47.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:47.013835+0000) 2024-02-19T20:09:48.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:48.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:48.014094+0000) 2024-02-19T20:09:49.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:49.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:49.014329+0000) 2024-02-19T20:09:50.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:50.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:50.014515+0000) 2024-02-19T20:09:51.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:51.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:51.014810+0000) 2024-02-19T20:09:52.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:52.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:52.015001+0000) 2024-02-19T20:09:53.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:53.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:53.015229+0000) 2024-02-19T20:09:54.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:54.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:54.015483+0000) 2024-02-19T20:09:55.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:55.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:55.015721+0000) 2024-02-19T20:09:56.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:56.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:56.015974+0000) 2024-02-19T20:09:57.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:57.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:57.016174+0000) 2024-02-19T20:09:58.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:58.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:58.016424+0000) 2024-02-19T20:09:59.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:09:59.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:59.016652+0000) 2024-02-19T20:10:00.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:00.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:00.016876+0000) 2024-02-19T20:10:01.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:01.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:01.017065+0000) 2024-02-19T20:10:02.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:02.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:02.017293+0000) 2024-02-19T20:10:03.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:03.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:03.017484+0000) 2024-02-19T20:10:04.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:04.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:04.017713+0000) 2024-02-19T20:10:04.775 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:10:04.779 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:10:05.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:05.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:05.017948+0000) 2024-02-19T20:10:06.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:06.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:06.018161+0000) 2024-02-19T20:10:07.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:07.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:07.018396+0000) 2024-02-19T20:10:08.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:08.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:08.018600+0000) 2024-02-19T20:10:09.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:09.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:09.018841+0000) 2024-02-19T20:10:10.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:10.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:10.019021+0000) 2024-02-19T20:10:11.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:11.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:11.019282+0000) 2024-02-19T20:10:12.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:12.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:12.019425+0000) 2024-02-19T20:10:13.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:13.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:13.019583+0000) 2024-02-19T20:10:14.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:14.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:14.019743+0000) 2024-02-19T20:10:15.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:15.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:15.019983+0000) 2024-02-19T20:10:16.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:16.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:16.020243+0000) 2024-02-19T20:10:17.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:17.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:17.020495+0000) 2024-02-19T20:10:18.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:18.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:18.020758+0000) 2024-02-19T20:10:19.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:19.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:19.020989+0000) 2024-02-19T20:10:20.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:20.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:20.021241+0000) 2024-02-19T20:10:21.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:21.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:21.021503+0000) 2024-02-19T20:10:22.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:22.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:22.021709+0000) 2024-02-19T20:10:23.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:23.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:23.021909+0000) 2024-02-19T20:10:24.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:24.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:24.022143+0000) 2024-02-19T20:10:25.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:25.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:25.022390+0000) 2024-02-19T20:10:26.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:26.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:26.022623+0000) 2024-02-19T20:10:27.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:27.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:27.022917+0000) 2024-02-19T20:10:28.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:28.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:28.023057+0000) 2024-02-19T20:10:29.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:29.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:29.023250+0000) 2024-02-19T20:10:30.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:30.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:30.023470+0000) 2024-02-19T20:10:31.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:31.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:31.023727+0000) 2024-02-19T20:10:32.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:32.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:32.024005+0000) 2024-02-19T20:10:33.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:33.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:33.024243+0000) 2024-02-19T20:10:34.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:34.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:34.024477+0000) 2024-02-19T20:10:34.914 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:10:34.917 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:10:35.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:35.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:35.024767+0000) 2024-02-19T20:10:36.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:36.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:36.025005+0000) 2024-02-19T20:10:37.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:37.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:37.025278+0000) 2024-02-19T20:10:38.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:38.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:38.025460+0000) 2024-02-19T20:10:39.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:39.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:39.025703+0000) 2024-02-19T20:10:40.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:40.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:40.025950+0000) 2024-02-19T20:10:41.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:41.022+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:41.026204+0000) 2024-02-19T20:10:42.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:42.026+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:42.026433+0000) 2024-02-19T20:10:43.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:43.026+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:43.026679+0000) 2024-02-19T20:10:44.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:44.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:44.026878+0000) 2024-02-19T20:10:45.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:45.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:45.027001+0000) 2024-02-19T20:10:46.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:46.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:46.027179+0000) 2024-02-19T20:10:47.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:47.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:47.027407+0000) 2024-02-19T20:10:48.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:48.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:48.027657+0000) 2024-02-19T20:10:49.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:49.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:49.027917+0000) 2024-02-19T20:10:50.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:50.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:50.028104+0000) 2024-02-19T20:10:51.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:51.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:51.028338+0000) 2024-02-19T20:10:52.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:52.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:52.028587+0000) 2024-02-19T20:10:53.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:53.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:53.028843+0000) 2024-02-19T20:10:54.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:54.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:54.029053+0000) 2024-02-19T20:10:55.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:55.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:55.029263+0000) 2024-02-19T20:10:56.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:56.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:56.029472+0000) 2024-02-19T20:10:57.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:57.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:57.029732+0000) 2024-02-19T20:10:58.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:58.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:58.029968+0000) 2024-02-19T20:10:59.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:10:59.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:59.030201+0000) 2024-02-19T20:11:00.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:00.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:00.030365+0000) 2024-02-19T20:11:01.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:01.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:01.030544+0000) 2024-02-19T20:11:02.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:02.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:02.030816+0000) 2024-02-19T20:11:03.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:03.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:03.030957+0000) 2024-02-19T20:11:04.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:04.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:04.031162+0000) 2024-02-19T20:11:05.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:05.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:05.031424+0000) 2024-02-19T20:11:05.036 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:11:05.040 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:11:06.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:06.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:06.031615+0000) 2024-02-19T20:11:07.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:07.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:07.031867+0000) 2024-02-19T20:11:08.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:08.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:08.032118+0000) 2024-02-19T20:11:09.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:09.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:09.032293+0000) 2024-02-19T20:11:10.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:10.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:10.032566+0000) 2024-02-19T20:11:11.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:11.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:11.032743+0000) 2024-02-19T20:11:12.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:12.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:12.032988+0000) 2024-02-19T20:11:13.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:13.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:13.033158+0000) 2024-02-19T20:11:14.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:14.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:14.033393+0000) 2024-02-19T20:11:15.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:15.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:15.033630+0000) 2024-02-19T20:11:16.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:16.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:16.033881+0000) 2024-02-19T20:11:17.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:17.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:17.034067+0000) 2024-02-19T20:11:18.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:18.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:18.034338+0000) 2024-02-19T20:11:19.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:19.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:19.034510+0000) 2024-02-19T20:11:20.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:20.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:20.034818+0000) 2024-02-19T20:11:21.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:21.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:21.034984+0000) 2024-02-19T20:11:22.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:22.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:22.035123+0000) 2024-02-19T20:11:23.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:23.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:23.035317+0000) 2024-02-19T20:11:24.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:24.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:24.035452+0000) 2024-02-19T20:11:25.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:25.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:25.035631+0000) 2024-02-19T20:11:26.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:26.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:26.035805+0000) 2024-02-19T20:11:27.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:27.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:27.036040+0000) 2024-02-19T20:11:28.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:28.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:28.036336+0000) 2024-02-19T20:11:29.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:29.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:29.036501+0000) 2024-02-19T20:11:30.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:30.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:30.036747+0000) 2024-02-19T20:11:31.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:31.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:31.036897+0000) 2024-02-19T20:11:32.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:32.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:32.037108+0000) 2024-02-19T20:11:33.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:33.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:33.037280+0000) 2024-02-19T20:11:34.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:34.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:34.037488+0000) 2024-02-19T20:11:35.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:35.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:35.037671+0000) 2024-02-19T20:11:35.158 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:11:35.161 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:11:36.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:36.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:36.037821+0000) 2024-02-19T20:11:37.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:37.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:37.037951+0000) 2024-02-19T20:11:38.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:38.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:38.038103+0000) 2024-02-19T20:11:39.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:39.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:39.038233+0000) 2024-02-19T20:11:40.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:40.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:40.038381+0000) 2024-02-19T20:11:41.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:41.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:41.038533+0000) 2024-02-19T20:11:42.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:42.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:42.038657+0000) 2024-02-19T20:11:43.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:43.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:43.038883+0000) 2024-02-19T20:11:44.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:44.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:44.039113+0000) 2024-02-19T20:11:45.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:45.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:45.039365+0000) 2024-02-19T20:11:46.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:46.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:46.039589+0000) 2024-02-19T20:11:47.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:47.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:47.039740+0000) 2024-02-19T20:11:48.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:48.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:48.039952+0000) 2024-02-19T20:11:49.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:49.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:49.040177+0000) 2024-02-19T20:11:50.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:50.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:50.040353+0000) 2024-02-19T20:11:51.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:51.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:51.040579+0000) 2024-02-19T20:11:52.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:52.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:52.040860+0000) 2024-02-19T20:11:53.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:53.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:53.041139+0000) 2024-02-19T20:11:54.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:54.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:54.041406+0000) 2024-02-19T20:11:55.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:55.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:55.041653+0000) 2024-02-19T20:11:56.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:56.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:56.041883+0000) 2024-02-19T20:11:57.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:57.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:57.042128+0000) 2024-02-19T20:11:58.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:58.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:58.042404+0000) 2024-02-19T20:11:59.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:11:59.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:59.042577+0000) 2024-02-19T20:12:00.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:00.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:00.042823+0000) 2024-02-19T20:12:01.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:01.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:01.043037+0000) 2024-02-19T20:12:02.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:02.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:02.043278+0000) 2024-02-19T20:12:03.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:03.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:03.043538+0000) 2024-02-19T20:12:04.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:04.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:04.043789+0000) 2024-02-19T20:12:05.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:05.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:05.043949+0000) 2024-02-19T20:12:05.289 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:12:05.293 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:12:06.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:06.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:06.044200+0000) 2024-02-19T20:12:07.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:07.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:07.044398+0000) 2024-02-19T20:12:08.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:08.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:08.044627+0000) 2024-02-19T20:12:09.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:09.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:09.044850+0000) 2024-02-19T20:12:10.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:10.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:10.045117+0000) 2024-02-19T20:12:11.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:11.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:11.045380+0000) 2024-02-19T20:12:12.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:12.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:12.045628+0000) 2024-02-19T20:12:13.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:13.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:13.045886+0000) 2024-02-19T20:12:14.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:14.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:14.046125+0000) 2024-02-19T20:12:15.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:15.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:15.046376+0000) 2024-02-19T20:12:16.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:16.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:16.046562+0000) 2024-02-19T20:12:17.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:17.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:17.046784+0000) 2024-02-19T20:12:18.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:18.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:18.046941+0000) 2024-02-19T20:12:19.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:19.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:19.047067+0000) 2024-02-19T20:12:20.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:20.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:20.047294+0000) 2024-02-19T20:12:21.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:21.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:21.047574+0000) 2024-02-19T20:12:22.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:22.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:22.047771+0000) 2024-02-19T20:12:23.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:23.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:23.048024+0000) 2024-02-19T20:12:24.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:24.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:24.048255+0000) 2024-02-19T20:12:25.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:25.048+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:25.048529+0000) 2024-02-19T20:12:26.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:26.048+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:26.048771+0000) 2024-02-19T20:12:27.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:27.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:27.049008+0000) 2024-02-19T20:12:28.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:28.048+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:28.049250+0000) 2024-02-19T20:12:29.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:29.048+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:29.049518+0000) 2024-02-19T20:12:30.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:30.048+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:30.049779+0000) 2024-02-19T20:12:31.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:31.048+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:31.050021+0000) 2024-02-19T20:12:32.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:32.048+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:32.050287+0000) 2024-02-19T20:12:33.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:33.048+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:33.050533+0000) 2024-02-19T20:12:34.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:34.048+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:34.050805+0000) 2024-02-19T20:12:35.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:35.048+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:35.050992+0000) 2024-02-19T20:12:35.411 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:12:35.417 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:12:36.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:36.048+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:36.051167+0000) 2024-02-19T20:12:37.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:37.048+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:37.051341+0000) 2024-02-19T20:12:38.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:38.048+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:38.051584+0000) 2024-02-19T20:12:39.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:39.048+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:39.051770+0000) 2024-02-19T20:12:40.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:40.048+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:40.051989+0000) 2024-02-19T20:12:41.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:41.052+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:41.052191+0000) 2024-02-19T20:12:42.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:42.052+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:42.052445+0000) 2024-02-19T20:12:43.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:43.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:43.052627+0000) 2024-02-19T20:12:44.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:44.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:44.052868+0000) 2024-02-19T20:12:45.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:45.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:45.053065+0000) 2024-02-19T20:12:46.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:46.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:46.053338+0000) 2024-02-19T20:12:47.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:47.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:47.053539+0000) 2024-02-19T20:12:48.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:48.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:48.053822+0000) 2024-02-19T20:12:49.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:49.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:49.054063+0000) 2024-02-19T20:12:50.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:50.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:50.054239+0000) 2024-02-19T20:12:51.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:51.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:51.054452+0000) 2024-02-19T20:12:52.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:52.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:52.054612+0000) 2024-02-19T20:12:53.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:53.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:53.054848+0000) 2024-02-19T20:12:54.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:54.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:54.055078+0000) 2024-02-19T20:12:55.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:55.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:55.055322+0000) 2024-02-19T20:12:56.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:56.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:56.055595+0000) 2024-02-19T20:12:57.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:57.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:57.055852+0000) 2024-02-19T20:12:58.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:58.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:58.056101+0000) 2024-02-19T20:12:59.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:12:59.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:59.056337+0000) 2024-02-19T20:13:00.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:00.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:00.056579+0000) 2024-02-19T20:13:01.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:01.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:01.056778+0000) 2024-02-19T20:13:02.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:02.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:02.057052+0000) 2024-02-19T20:13:03.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:03.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:03.057299+0000) 2024-02-19T20:13:04.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:04.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:04.057582+0000) 2024-02-19T20:13:05.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:05.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:05.057837+0000) 2024-02-19T20:13:05.551 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:13:05.554 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:13:06.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:06.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:06.058012+0000) 2024-02-19T20:13:07.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:07.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:07.058254+0000) 2024-02-19T20:13:08.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:08.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:08.058483+0000) 2024-02-19T20:13:09.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:09.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:09.058802+0000) 2024-02-19T20:13:10.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:10.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:10.059051+0000) 2024-02-19T20:13:11.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:11.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:11.059315+0000) 2024-02-19T20:13:12.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:12.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:12.059576+0000) 2024-02-19T20:13:13.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:13.059+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:13.059819+0000) 2024-02-19T20:13:14.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:14.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:14.060088+0000) 2024-02-19T20:13:15.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:15.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:15.060347+0000) 2024-02-19T20:13:16.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:16.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:16.060545+0000) 2024-02-19T20:13:17.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:17.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:17.060786+0000) 2024-02-19T20:13:18.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:18.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:18.061030+0000) 2024-02-19T20:13:19.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:19.059+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:19.061279+0000) 2024-02-19T20:13:20.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:20.059+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:20.061523+0000) 2024-02-19T20:13:21.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:21.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:21.061666+0000) 2024-02-19T20:13:22.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:22.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:22.061823+0000) 2024-02-19T20:13:23.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:23.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:23.062080+0000) 2024-02-19T20:13:24.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:24.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:24.062307+0000) 2024-02-19T20:13:25.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:25.059+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:25.062556+0000) 2024-02-19T20:13:26.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:26.059+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:26.062846+0000) 2024-02-19T20:13:27.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:27.059+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:27.063083+0000) 2024-02-19T20:13:28.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:28.059+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:28.063275+0000) 2024-02-19T20:13:29.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:29.063+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:29.063515+0000) 2024-02-19T20:13:30.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:30.063+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:30.063789+0000) 2024-02-19T20:13:31.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:31.063+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:31.063994+0000) 2024-02-19T20:13:32.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:32.063+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:32.064200+0000) 2024-02-19T20:13:33.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:33.063+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:33.064455+0000) 2024-02-19T20:13:34.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:34.063+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:34.064678+0000) 2024-02-19T20:13:35.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:35.063+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:35.064937+0000) 2024-02-19T20:13:35.631 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:13:35.634 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:13:36.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:36.063+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:36.065164+0000) 2024-02-19T20:13:37.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:37.059+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:37.065399+0000) 2024-02-19T20:13:38.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:38.059+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:38.065641+0000) 2024-02-19T20:13:39.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:39.063+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:39.065906+0000) 2024-02-19T20:13:40.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:40.059+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:40.066167+0000) 2024-02-19T20:13:41.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:41.063+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:41.066302+0000) 2024-02-19T20:13:42.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:42.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:42.066507+0000) 2024-02-19T20:13:43.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:43.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:43.066794+0000) 2024-02-19T20:13:44.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:44.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:44.067005+0000) 2024-02-19T20:13:45.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:45.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:45.067249+0000) 2024-02-19T20:13:46.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:46.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:46.067501+0000) 2024-02-19T20:13:47.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:47.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:47.067705+0000) 2024-02-19T20:13:48.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:48.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:48.067955+0000) 2024-02-19T20:13:49.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:49.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:49.068218+0000) 2024-02-19T20:13:50.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:50.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:50.068390+0000) 2024-02-19T20:13:51.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:51.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:51.068581+0000) 2024-02-19T20:13:52.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:52.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:52.068762+0000) 2024-02-19T20:13:53.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:53.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:53.069013+0000) 2024-02-19T20:13:54.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:54.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:54.069242+0000) 2024-02-19T20:13:55.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:55.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:55.069487+0000) 2024-02-19T20:13:56.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:56.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:56.069680+0000) 2024-02-19T20:13:57.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:57.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:57.069917+0000) 2024-02-19T20:13:58.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:58.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:58.070162+0000) 2024-02-19T20:13:59.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:13:59.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:59.070383+0000) 2024-02-19T20:14:00.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:00.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:00.070559+0000) 2024-02-19T20:14:01.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:01.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:01.070827+0000) 2024-02-19T20:14:02.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:02.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:02.071070+0000) 2024-02-19T20:14:03.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:03.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:03.071372+0000) 2024-02-19T20:14:04.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:04.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:04.071540+0000) 2024-02-19T20:14:05.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:05.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:05.071710+0000) 2024-02-19T20:14:05.736 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:14:05.740 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:14:06.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:06.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:06.071887+0000) 2024-02-19T20:14:07.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:07.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:07.072070+0000) 2024-02-19T20:14:08.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:08.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:08.072259+0000) 2024-02-19T20:14:09.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:09.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:09.072419+0000) 2024-02-19T20:14:10.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:10.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:10.072640+0000) 2024-02-19T20:14:11.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:11.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:11.072829+0000) 2024-02-19T20:14:12.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:12.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:12.073056+0000) 2024-02-19T20:14:13.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:13.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:13.073289+0000) 2024-02-19T20:14:14.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:14.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:14.073434+0000) 2024-02-19T20:14:15.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:15.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:15.073565+0000) 2024-02-19T20:14:16.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:16.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:16.073739+0000) 2024-02-19T20:14:17.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:17.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:17.073867+0000) 2024-02-19T20:14:18.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:18.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:18.074049+0000) 2024-02-19T20:14:19.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:19.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:19.074176+0000) 2024-02-19T20:14:20.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:20.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:20.074297+0000) 2024-02-19T20:14:21.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:21.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:21.074434+0000) 2024-02-19T20:14:22.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:22.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:22.074648+0000) 2024-02-19T20:14:23.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:23.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:23.074897+0000) 2024-02-19T20:14:24.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:24.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:24.075132+0000) 2024-02-19T20:14:25.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:25.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:25.075305+0000) 2024-02-19T20:14:26.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:26.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:26.075560+0000) 2024-02-19T20:14:27.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:27.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:27.075740+0000) 2024-02-19T20:14:28.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:28.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:28.076001+0000) 2024-02-19T20:14:29.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:29.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:29.076170+0000) 2024-02-19T20:14:30.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:30.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:30.076406+0000) 2024-02-19T20:14:31.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:31.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:31.076697+0000) 2024-02-19T20:14:32.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:32.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:32.076949+0000) 2024-02-19T20:14:33.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:33.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:33.077229+0000) 2024-02-19T20:14:34.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:34.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:34.077400+0000) 2024-02-19T20:14:35.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:35.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:35.077599+0000) 2024-02-19T20:14:35.792 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:14:35.797 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:14:36.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:36.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:36.077843+0000) 2024-02-19T20:14:37.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:37.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:37.078048+0000) 2024-02-19T20:14:38.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:38.078+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:38.078340+0000) 2024-02-19T20:14:39.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:39.078+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:39.078598+0000) 2024-02-19T20:14:40.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:40.078+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:40.078814+0000) 2024-02-19T20:14:41.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:41.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:41.079076+0000) 2024-02-19T20:14:42.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:42.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:42.079318+0000) 2024-02-19T20:14:43.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:43.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:43.079507+0000) 2024-02-19T20:14:44.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:44.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:44.079761+0000) 2024-02-19T20:14:45.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:45.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:45.079963+0000) 2024-02-19T20:14:46.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:46.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:46.080205+0000) 2024-02-19T20:14:47.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:47.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:47.080450+0000) 2024-02-19T20:14:48.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:48.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:48.080681+0000) 2024-02-19T20:14:49.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:49.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:49.080944+0000) 2024-02-19T20:14:50.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:50.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:50.081088+0000) 2024-02-19T20:14:51.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:51.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:51.081309+0000) 2024-02-19T20:14:52.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:52.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:52.081566+0000) 2024-02-19T20:14:53.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:53.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:53.081775+0000) 2024-02-19T20:14:54.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:54.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:54.081959+0000) 2024-02-19T20:14:55.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:55.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:55.082182+0000) 2024-02-19T20:14:56.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:56.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:56.082394+0000) 2024-02-19T20:14:57.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:57.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:57.082608+0000) 2024-02-19T20:14:58.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:58.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:58.082777+0000) 2024-02-19T20:14:59.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:14:59.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:59.082897+0000) 2024-02-19T20:15:00.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:00.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:00.083076+0000) 2024-02-19T20:15:01.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:01.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:01.083258+0000) 2024-02-19T20:15:02.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:02.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:02.083541+0000) 2024-02-19T20:15:03.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:03.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:03.083807+0000) 2024-02-19T20:15:04.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:04.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:04.084053+0000) 2024-02-19T20:15:05.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:05.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:05.084308+0000) 2024-02-19T20:15:05.856 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:15:05.859 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:15:06.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:06.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:06.084583+0000) 2024-02-19T20:15:07.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:07.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:07.084789+0000) 2024-02-19T20:15:08.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:08.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:08.084989+0000) 2024-02-19T20:15:09.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:09.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:09.085211+0000) 2024-02-19T20:15:10.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:10.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:10.085446+0000) 2024-02-19T20:15:11.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:11.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:11.085712+0000) 2024-02-19T20:15:12.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:12.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:12.085964+0000) 2024-02-19T20:15:13.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:13.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:13.086162+0000) 2024-02-19T20:15:14.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:14.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:14.086433+0000) 2024-02-19T20:15:15.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:15.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:15.086612+0000) 2024-02-19T20:15:16.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:16.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:16.086848+0000) 2024-02-19T20:15:17.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:17.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:17.086985+0000) 2024-02-19T20:15:18.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:18.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:18.087234+0000) 2024-02-19T20:15:19.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:19.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:19.087509+0000) 2024-02-19T20:15:20.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:20.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:20.087751+0000) 2024-02-19T20:15:21.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:21.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:21.088006+0000) 2024-02-19T20:15:22.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:22.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:22.088242+0000) 2024-02-19T20:15:23.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:23.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:23.088510+0000) 2024-02-19T20:15:24.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:24.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:24.088757+0000) 2024-02-19T20:15:25.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:25.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:25.088958+0000) 2024-02-19T20:15:26.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:26.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:26.089215+0000) 2024-02-19T20:15:27.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:27.089+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:27.089458+0000) 2024-02-19T20:15:28.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:28.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:28.089710+0000) 2024-02-19T20:15:29.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:29.089+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:29.089913+0000) 2024-02-19T20:15:30.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:30.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:30.090148+0000) 2024-02-19T20:15:31.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:31.089+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:31.090434+0000) 2024-02-19T20:15:32.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:32.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:32.090666+0000) 2024-02-19T20:15:33.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:33.089+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:33.090913+0000) 2024-02-19T20:15:34.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:34.089+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:34.091184+0000) 2024-02-19T20:15:35.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:35.089+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:35.091397+0000) 2024-02-19T20:15:35.979 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:15:35.984 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:15:36.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:36.089+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:36.091646+0000) 2024-02-19T20:15:37.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:37.089+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:37.091821+0000) 2024-02-19T20:15:38.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:38.089+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:38.092119+0000) 2024-02-19T20:15:39.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:39.089+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:39.092258+0000) 2024-02-19T20:15:40.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:40.089+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:40.092459+0000) 2024-02-19T20:15:41.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:41.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:41.092728+0000) 2024-02-19T20:15:42.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:42.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:42.092904+0000) 2024-02-19T20:15:43.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:43.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:43.093132+0000) 2024-02-19T20:15:44.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:44.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:44.093347+0000) 2024-02-19T20:15:45.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:45.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:45.093621+0000) 2024-02-19T20:15:46.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:46.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:46.093852+0000) 2024-02-19T20:15:47.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:47.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:47.094025+0000) 2024-02-19T20:15:48.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:48.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:48.094186+0000) 2024-02-19T20:15:49.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:49.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:49.094442+0000) 2024-02-19T20:15:50.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:50.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:50.094773+0000) 2024-02-19T20:15:51.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:51.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:51.094958+0000) 2024-02-19T20:15:52.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:52.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:52.095179+0000) 2024-02-19T20:15:53.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:53.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:53.095419+0000) 2024-02-19T20:15:54.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:54.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:54.095649+0000) 2024-02-19T20:15:55.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:55.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:55.095890+0000) 2024-02-19T20:15:56.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:56.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:56.096094+0000) 2024-02-19T20:15:57.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:57.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:57.096371+0000) 2024-02-19T20:15:58.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:58.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:58.096571+0000) 2024-02-19T20:15:59.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:15:59.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:59.096755+0000) 2024-02-19T20:16:00.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:00.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:00.097009+0000) 2024-02-19T20:16:01.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:01.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:01.097293+0000) 2024-02-19T20:16:02.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:02.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:02.097528+0000) 2024-02-19T20:16:03.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:03.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:03.097777+0000) 2024-02-19T20:16:04.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:04.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:04.097946+0000) 2024-02-19T20:16:05.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:05.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:05.098199+0000) 2024-02-19T20:16:06.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:06.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:06.098441+0000) 2024-02-19T20:16:06.118 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:16:06.121 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:16:07.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:07.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:07.098665+0000) 2024-02-19T20:16:08.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:08.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:08.098805+0000) 2024-02-19T20:16:09.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:09.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:09.098965+0000) 2024-02-19T20:16:10.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:10.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:10.099135+0000) 2024-02-19T20:16:11.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:11.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:11.099402+0000) 2024-02-19T20:16:12.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:12.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:12.099604+0000) 2024-02-19T20:16:13.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:13.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:13.099779+0000) 2024-02-19T20:16:14.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:14.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:14.099976+0000) 2024-02-19T20:16:15.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:15.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:15.100212+0000) 2024-02-19T20:16:16.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:16.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:16.100476+0000) 2024-02-19T20:16:17.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:17.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:17.100669+0000) 2024-02-19T20:16:18.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:18.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:18.100930+0000) 2024-02-19T20:16:19.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:19.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:19.101107+0000) 2024-02-19T20:16:20.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:20.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:20.101247+0000) 2024-02-19T20:16:21.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:21.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:21.101507+0000) 2024-02-19T20:16:22.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:22.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:22.101746+0000) 2024-02-19T20:16:23.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:23.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:23.101919+0000) 2024-02-19T20:16:24.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:24.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:24.102153+0000) 2024-02-19T20:16:25.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:25.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:25.102355+0000) 2024-02-19T20:16:26.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:26.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:26.102615+0000) 2024-02-19T20:16:27.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:27.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:27.102863+0000) 2024-02-19T20:16:28.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:28.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:28.103101+0000) 2024-02-19T20:16:29.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:29.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:29.103284+0000) 2024-02-19T20:16:30.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:30.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:30.103503+0000) 2024-02-19T20:16:31.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:31.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:31.103787+0000) 2024-02-19T20:16:32.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:32.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:32.103982+0000) 2024-02-19T20:16:33.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:33.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:33.104202+0000) 2024-02-19T20:16:34.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:34.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:34.104416+0000) 2024-02-19T20:16:35.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:35.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:35.104668+0000) 2024-02-19T20:16:36.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:36.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:36.104931+0000) 2024-02-19T20:16:36.232 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:16:36.235 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:16:37.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:37.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:37.105181+0000) 2024-02-19T20:16:38.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:38.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:38.105426+0000) 2024-02-19T20:16:39.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:39.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:39.105663+0000) 2024-02-19T20:16:40.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:40.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:40.105925+0000) 2024-02-19T20:16:41.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:41.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:41.106201+0000) 2024-02-19T20:16:42.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:42.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:42.106445+0000) 2024-02-19T20:16:43.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:43.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:43.106630+0000) 2024-02-19T20:16:44.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:44.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:44.106853+0000) 2024-02-19T20:16:45.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:45.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:45.107137+0000) 2024-02-19T20:16:46.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:46.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:46.107338+0000) 2024-02-19T20:16:47.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:47.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:47.107512+0000) 2024-02-19T20:16:48.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:48.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:48.107685+0000) 2024-02-19T20:16:49.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:49.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:49.107917+0000) 2024-02-19T20:16:50.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:50.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:50.108181+0000) 2024-02-19T20:16:51.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:51.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:51.108437+0000) 2024-02-19T20:16:52.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:52.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:52.108690+0000) 2024-02-19T20:16:53.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:53.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:53.108919+0000) 2024-02-19T20:16:54.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:54.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:54.109181+0000) 2024-02-19T20:16:55.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:55.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:55.109418+0000) 2024-02-19T20:16:56.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:56.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:56.109656+0000) 2024-02-19T20:16:57.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:57.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:57.109906+0000) 2024-02-19T20:16:58.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:58.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:58.110043+0000) 2024-02-19T20:16:59.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:16:59.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:59.110292+0000) 2024-02-19T20:17:00.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:00.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:00.110530+0000) 2024-02-19T20:17:01.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:01.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:01.110754+0000) 2024-02-19T20:17:02.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:02.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:02.110877+0000) 2024-02-19T20:17:03.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:03.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:03.111087+0000) 2024-02-19T20:17:04.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:04.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:04.111337+0000) 2024-02-19T20:17:05.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:05.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:05.111589+0000) 2024-02-19T20:17:06.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:06.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:06.111850+0000) 2024-02-19T20:17:06.313 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:17:06.317 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:17:07.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:07.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:07.112153+0000) 2024-02-19T20:17:08.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:08.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:08.112460+0000) 2024-02-19T20:17:09.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:09.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:09.112678+0000) 2024-02-19T20:17:10.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:10.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:10.112961+0000) 2024-02-19T20:17:11.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:11.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:11.113183+0000) 2024-02-19T20:17:12.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:12.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:12.113431+0000) 2024-02-19T20:17:13.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:13.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:13.113715+0000) 2024-02-19T20:17:14.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:14.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:14.113950+0000) 2024-02-19T20:17:15.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:15.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:15.114136+0000) 2024-02-19T20:17:16.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:16.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:16.114359+0000) 2024-02-19T20:17:17.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:17.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:17.114611+0000) 2024-02-19T20:17:18.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:18.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:18.114916+0000) 2024-02-19T20:17:19.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:19.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:19.115149+0000) 2024-02-19T20:17:20.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:20.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:20.115405+0000) 2024-02-19T20:17:21.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:21.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:21.115580+0000) 2024-02-19T20:17:22.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:22.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:22.115839+0000) 2024-02-19T20:17:23.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:23.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:23.116015+0000) 2024-02-19T20:17:24.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:24.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:24.116252+0000) 2024-02-19T20:17:25.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:25.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:25.116504+0000) 2024-02-19T20:17:26.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:26.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:26.116764+0000) 2024-02-19T20:17:27.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:27.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:27.116941+0000) 2024-02-19T20:17:28.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:28.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:28.117109+0000) 2024-02-19T20:17:29.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:29.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:29.117342+0000) 2024-02-19T20:17:30.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:30.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:30.117595+0000) 2024-02-19T20:17:31.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:31.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:31.117848+0000) 2024-02-19T20:17:32.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:32.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:32.118102+0000) 2024-02-19T20:17:33.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:33.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:33.118301+0000) 2024-02-19T20:17:34.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:34.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:34.118557+0000) 2024-02-19T20:17:35.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:35.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:35.118771+0000) 2024-02-19T20:17:36.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:36.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:36.119007+0000) 2024-02-19T20:17:36.425 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:17:36.429 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:17:37.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:37.119+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:37.119202+0000) 2024-02-19T20:17:38.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:38.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:38.119417+0000) 2024-02-19T20:17:39.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:39.119+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:39.119602+0000) 2024-02-19T20:17:40.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:40.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:40.119854+0000) 2024-02-19T20:17:41.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:41.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:41.120092+0000) 2024-02-19T20:17:42.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:42.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:42.120290+0000) 2024-02-19T20:17:43.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:43.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:43.120513+0000) 2024-02-19T20:17:44.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:44.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:44.120750+0000) 2024-02-19T20:17:45.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:45.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:45.120988+0000) 2024-02-19T20:17:46.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:46.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:46.121157+0000) 2024-02-19T20:17:47.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:47.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:47.121392+0000) 2024-02-19T20:17:48.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:48.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:48.121571+0000) 2024-02-19T20:17:49.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:49.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:49.121728+0000) 2024-02-19T20:17:50.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:50.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:50.121939+0000) 2024-02-19T20:17:51.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:51.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:51.122174+0000) 2024-02-19T20:17:52.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:52.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:52.122416+0000) 2024-02-19T20:17:53.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:53.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:53.122672+0000) 2024-02-19T20:17:54.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:54.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:54.122812+0000) 2024-02-19T20:17:55.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:55.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:55.122983+0000) 2024-02-19T20:17:56.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:56.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:56.123184+0000) 2024-02-19T20:17:57.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:57.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:57.123381+0000) 2024-02-19T20:17:58.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:58.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:58.123625+0000) 2024-02-19T20:17:59.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:17:59.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:59.123864+0000) 2024-02-19T20:18:00.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:00.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:00.124086+0000) 2024-02-19T20:18:01.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:01.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:01.124320+0000) 2024-02-19T20:18:02.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:02.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:02.124570+0000) 2024-02-19T20:18:03.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:03.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:03.124802+0000) 2024-02-19T20:18:04.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:04.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:04.125066+0000) 2024-02-19T20:18:05.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:05.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:05.125237+0000) 2024-02-19T20:18:06.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:06.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:06.125465+0000) 2024-02-19T20:18:06.540 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:18:06.543 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:18:07.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:07.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:07.125672+0000) 2024-02-19T20:18:08.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:08.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:08.125918+0000) 2024-02-19T20:18:09.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:09.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:09.126099+0000) 2024-02-19T20:18:10.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:10.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:10.126320+0000) 2024-02-19T20:18:11.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:11.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:11.126526+0000) 2024-02-19T20:18:12.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:12.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:12.126821+0000) 2024-02-19T20:18:13.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:13.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:13.127028+0000) 2024-02-19T20:18:14.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:14.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:14.127267+0000) 2024-02-19T20:18:15.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:15.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:15.127525+0000) 2024-02-19T20:18:16.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:16.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:16.127758+0000) 2024-02-19T20:18:17.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:17.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:17.128042+0000) 2024-02-19T20:18:18.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:18.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:18.128320+0000) 2024-02-19T20:18:19.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:19.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:19.128597+0000) 2024-02-19T20:18:20.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:20.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:20.128765+0000) 2024-02-19T20:18:21.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:21.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:21.128969+0000) 2024-02-19T20:18:22.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:22.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:22.129200+0000) 2024-02-19T20:18:23.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:23.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:23.129454+0000) 2024-02-19T20:18:24.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:24.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:24.129700+0000) 2024-02-19T20:18:25.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:25.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:25.129925+0000) 2024-02-19T20:18:26.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:26.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:26.130163+0000) 2024-02-19T20:18:27.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:27.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:27.130331+0000) 2024-02-19T20:18:28.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:28.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:28.130512+0000) 2024-02-19T20:18:29.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:29.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:29.130751+0000) 2024-02-19T20:18:30.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:30.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:30.130978+0000) 2024-02-19T20:18:31.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:31.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:31.131154+0000) 2024-02-19T20:18:32.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:32.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:32.131393+0000) 2024-02-19T20:18:33.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:33.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:33.131638+0000) 2024-02-19T20:18:34.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:34.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:34.131866+0000) 2024-02-19T20:18:35.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:35.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:35.132111+0000) 2024-02-19T20:18:36.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:36.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:36.132374+0000) 2024-02-19T20:18:36.612 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:18:36.615 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:18:37.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:37.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:37.132546+0000) 2024-02-19T20:18:38.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:38.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:38.132770+0000) 2024-02-19T20:18:39.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:39.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:39.133017+0000) 2024-02-19T20:18:40.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:40.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:40.133205+0000) 2024-02-19T20:18:41.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:41.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:41.133417+0000) 2024-02-19T20:18:42.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:42.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:42.133645+0000) 2024-02-19T20:18:43.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:43.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:43.133897+0000) 2024-02-19T20:18:44.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:44.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:44.134120+0000) 2024-02-19T20:18:45.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:45.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:45.134361+0000) 2024-02-19T20:18:46.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:46.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:46.134630+0000) 2024-02-19T20:18:47.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:47.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:47.134878+0000) 2024-02-19T20:18:48.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:48.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:48.135112+0000) 2024-02-19T20:18:49.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:49.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:49.135355+0000) 2024-02-19T20:18:50.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:50.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:50.135640+0000) 2024-02-19T20:18:51.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:51.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:51.135949+0000) 2024-02-19T20:18:52.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:52.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:52.136157+0000) 2024-02-19T20:18:53.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:53.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:53.136431+0000) 2024-02-19T20:18:54.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:54.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:54.136716+0000) 2024-02-19T20:18:55.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:55.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:55.136963+0000) 2024-02-19T20:18:56.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:56.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:56.137169+0000) 2024-02-19T20:18:57.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:57.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:57.137401+0000) 2024-02-19T20:18:58.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:58.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:58.137704+0000) 2024-02-19T20:18:59.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:18:59.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:59.137879+0000) 2024-02-19T20:19:00.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:00.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:00.138136+0000) 2024-02-19T20:19:01.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:01.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:01.138404+0000) 2024-02-19T20:19:02.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:02.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:02.138557+0000) 2024-02-19T20:19:03.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:03.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:03.138695+0000) 2024-02-19T20:19:04.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:04.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:04.138867+0000) 2024-02-19T20:19:05.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:05.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:05.139008+0000) 2024-02-19T20:19:06.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:06.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:06.139217+0000) 2024-02-19T20:19:06.759 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:19:06.763 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:19:07.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:07.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:07.139359+0000) 2024-02-19T20:19:08.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:08.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:08.139534+0000) 2024-02-19T20:19:09.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:09.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:09.139769+0000) 2024-02-19T20:19:10.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:10.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:10.139993+0000) 2024-02-19T20:19:11.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:11.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:11.140233+0000) 2024-02-19T20:19:12.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:12.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:12.140540+0000) 2024-02-19T20:19:13.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:13.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:13.140802+0000) 2024-02-19T20:19:14.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:14.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:14.141038+0000) 2024-02-19T20:19:15.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:15.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:15.141249+0000) 2024-02-19T20:19:16.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:16.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:16.141498+0000) 2024-02-19T20:19:17.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:17.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:17.141774+0000) 2024-02-19T20:19:18.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:18.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:18.141994+0000) 2024-02-19T20:19:19.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:19.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:19.142204+0000) 2024-02-19T20:19:20.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:20.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:20.142433+0000) 2024-02-19T20:19:21.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:21.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:21.142643+0000) 2024-02-19T20:19:22.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:22.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:22.142870+0000) 2024-02-19T20:19:23.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:23.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:23.143025+0000) 2024-02-19T20:19:24.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:24.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:24.143197+0000) 2024-02-19T20:19:25.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:25.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:25.143458+0000) 2024-02-19T20:19:26.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:26.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:26.143669+0000) 2024-02-19T20:19:27.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:27.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:27.143837+0000) 2024-02-19T20:19:28.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:28.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:28.144028+0000) 2024-02-19T20:19:29.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:29.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:29.144239+0000) 2024-02-19T20:19:30.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:30.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:30.144472+0000) 2024-02-19T20:19:31.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:31.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:31.144711+0000) 2024-02-19T20:19:32.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:32.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:32.144945+0000) 2024-02-19T20:19:33.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:33.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:33.145196+0000) 2024-02-19T20:19:34.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:34.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:34.145450+0000) 2024-02-19T20:19:35.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:35.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:35.145711+0000) 2024-02-19T20:19:36.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:36.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:36.145937+0000) 2024-02-19T20:19:36.881 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:19:36.884 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:19:37.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:37.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:37.146169+0000) 2024-02-19T20:19:38.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:38.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:38.146392+0000) 2024-02-19T20:19:39.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:39.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:39.146684+0000) 2024-02-19T20:19:40.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:40.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:40.146966+0000) 2024-02-19T20:19:41.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:41.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:41.147209+0000) 2024-02-19T20:19:42.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:42.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:42.147473+0000) 2024-02-19T20:19:43.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:43.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:43.147710+0000) 2024-02-19T20:19:44.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:44.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:44.147957+0000) 2024-02-19T20:19:45.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:45.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:45.148146+0000) 2024-02-19T20:19:46.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:46.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:46.148399+0000) 2024-02-19T20:19:47.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:47.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:47.148579+0000) 2024-02-19T20:19:48.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:48.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:48.148802+0000) 2024-02-19T20:19:49.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:49.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:49.148953+0000) 2024-02-19T20:19:50.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:50.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:50.149194+0000) 2024-02-19T20:19:51.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:51.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:51.149397+0000) 2024-02-19T20:19:52.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:52.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:52.149629+0000) 2024-02-19T20:19:53.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:53.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:53.149870+0000) 2024-02-19T20:19:54.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:54.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:54.150064+0000) 2024-02-19T20:19:55.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:55.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:55.150288+0000) 2024-02-19T20:19:56.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:56.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:56.150520+0000) 2024-02-19T20:19:57.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:57.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:57.150698+0000) 2024-02-19T20:19:58.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:58.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:58.150899+0000) 2024-02-19T20:19:59.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:19:59.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:59.151054+0000) 2024-02-19T20:20:00.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:00.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:00.151334+0000) 2024-02-19T20:20:01.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:01.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:01.151520+0000) 2024-02-19T20:20:02.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:02.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:02.151697+0000) 2024-02-19T20:20:03.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:03.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:03.151902+0000) 2024-02-19T20:20:04.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:04.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:04.152151+0000) 2024-02-19T20:20:05.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:05.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:05.152402+0000) 2024-02-19T20:20:06.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:06.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:06.152695+0000) 2024-02-19T20:20:06.987 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:20:06.990 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:20:07.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:07.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:07.152855+0000) 2024-02-19T20:20:08.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:08.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:08.153094+0000) 2024-02-19T20:20:09.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:09.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:09.153244+0000) 2024-02-19T20:20:10.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:10.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:10.153433+0000) 2024-02-19T20:20:11.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:11.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:11.153701+0000) 2024-02-19T20:20:12.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:12.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:12.153936+0000) 2024-02-19T20:20:13.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:13.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:13.154176+0000) 2024-02-19T20:20:14.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:14.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:14.154412+0000) 2024-02-19T20:20:15.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:15.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:15.154643+0000) 2024-02-19T20:20:16.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:16.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:16.154840+0000) 2024-02-19T20:20:17.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:17.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:17.154993+0000) 2024-02-19T20:20:18.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:18.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:18.155181+0000) 2024-02-19T20:20:19.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:19.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:19.155437+0000) 2024-02-19T20:20:20.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:20.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:20.155694+0000) 2024-02-19T20:20:21.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:21.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:21.155962+0000) 2024-02-19T20:20:22.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:22.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:22.156194+0000) 2024-02-19T20:20:23.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:23.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:23.156428+0000) 2024-02-19T20:20:24.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:24.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:24.156690+0000) 2024-02-19T20:20:25.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:25.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:25.156866+0000) 2024-02-19T20:20:26.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:26.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:26.157137+0000) 2024-02-19T20:20:27.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:27.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:27.157392+0000) 2024-02-19T20:20:28.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:28.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:28.157636+0000) 2024-02-19T20:20:29.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:29.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:29.157889+0000) 2024-02-19T20:20:30.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:30.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:30.158123+0000) 2024-02-19T20:20:31.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:31.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:31.158346+0000) 2024-02-19T20:20:32.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:32.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:32.158528+0000) 2024-02-19T20:20:33.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:33.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:33.158782+0000) 2024-02-19T20:20:34.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:34.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:34.158907+0000) 2024-02-19T20:20:35.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:35.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:35.159116+0000) 2024-02-19T20:20:36.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:36.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:36.159356+0000) 2024-02-19T20:20:37.058 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:20:37.062 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:20:37.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:37.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:37.159562+0000) 2024-02-19T20:20:38.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:38.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:38.159822+0000) 2024-02-19T20:20:39.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:39.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:39.160021+0000) 2024-02-19T20:20:40.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:40.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:40.160276+0000) 2024-02-19T20:20:41.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:41.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:41.160553+0000) 2024-02-19T20:20:42.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:42.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:42.160796+0000) 2024-02-19T20:20:43.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:43.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:43.161030+0000) 2024-02-19T20:20:44.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:44.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:44.161289+0000) 2024-02-19T20:20:45.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:45.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:45.161538+0000) 2024-02-19T20:20:46.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:46.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:46.161775+0000) 2024-02-19T20:20:47.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:47.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:47.162024+0000) 2024-02-19T20:20:48.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:48.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:48.162271+0000) 2024-02-19T20:20:49.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:49.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:49.162515+0000) 2024-02-19T20:20:50.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:50.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:50.162689+0000) 2024-02-19T20:20:51.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:51.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:51.162949+0000) 2024-02-19T20:20:52.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:52.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:52.163148+0000) 2024-02-19T20:20:53.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:53.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:53.163332+0000) 2024-02-19T20:20:54.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:54.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:54.163501+0000) 2024-02-19T20:20:55.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:55.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:55.163666+0000) 2024-02-19T20:20:56.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:56.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:56.163873+0000) 2024-02-19T20:20:57.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:57.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:57.164156+0000) 2024-02-19T20:20:58.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:58.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:58.164448+0000) 2024-02-19T20:20:59.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:20:59.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:59.164699+0000) 2024-02-19T20:21:00.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:00.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:00.164941+0000) 2024-02-19T20:21:01.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:01.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:01.165190+0000) 2024-02-19T20:21:02.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:02.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:02.165483+0000) 2024-02-19T20:21:03.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:03.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:03.165733+0000) 2024-02-19T20:21:04.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:04.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:04.165987+0000) 2024-02-19T20:21:05.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:05.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:05.166208+0000) 2024-02-19T20:21:06.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:06.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:06.166391+0000) 2024-02-19T20:21:07.119 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:21:07.125 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:21:07.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:07.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:07.166631+0000) 2024-02-19T20:21:08.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:08.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:08.166871+0000) 2024-02-19T20:21:09.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:09.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:09.167073+0000) 2024-02-19T20:21:10.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:10.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:10.167248+0000) 2024-02-19T20:21:11.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:11.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:11.167523+0000) 2024-02-19T20:21:12.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:12.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:12.167795+0000) 2024-02-19T20:21:13.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:13.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:13.168044+0000) 2024-02-19T20:21:14.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:14.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:14.168295+0000) 2024-02-19T20:21:15.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:15.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:15.168511+0000) 2024-02-19T20:21:16.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:16.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:16.168736+0000) 2024-02-19T20:21:17.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:17.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:17.168992+0000) 2024-02-19T20:21:18.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:18.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:18.169246+0000) 2024-02-19T20:21:19.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:19.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:19.169509+0000) 2024-02-19T20:21:20.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:20.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:20.169771+0000) 2024-02-19T20:21:21.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:21.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:21.170018+0000) 2024-02-19T20:21:22.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:22.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:22.170266+0000) 2024-02-19T20:21:23.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:23.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:23.170435+0000) 2024-02-19T20:21:24.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:24.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:24.170692+0000) 2024-02-19T20:21:25.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:25.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:25.170947+0000) 2024-02-19T20:21:26.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:26.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:26.171095+0000) 2024-02-19T20:21:27.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:27.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:27.171338+0000) 2024-02-19T20:21:28.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:28.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:28.171630+0000) 2024-02-19T20:21:29.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:29.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:29.171866+0000) 2024-02-19T20:21:30.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:30.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:30.172100+0000) 2024-02-19T20:21:31.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:31.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:31.172232+0000) 2024-02-19T20:21:32.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:32.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:32.172441+0000) 2024-02-19T20:21:33.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:33.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:33.172585+0000) 2024-02-19T20:21:34.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:34.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:34.172764+0000) 2024-02-19T20:21:35.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:35.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:35.172981+0000) 2024-02-19T20:21:36.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:36.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:36.173230+0000) 2024-02-19T20:21:37.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:37.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:37.173517+0000) 2024-02-19T20:21:37.235 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:21:37.507 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:21:38.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:38.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:38.173794+0000) 2024-02-19T20:21:39.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:39.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:39.174025+0000) 2024-02-19T20:21:40.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:40.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:40.174273+0000) 2024-02-19T20:21:41.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:41.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:41.174457+0000) 2024-02-19T20:21:42.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:42.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:42.174805+0000) 2024-02-19T20:21:43.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:43.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:43.175086+0000) 2024-02-19T20:21:44.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:44.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:44.175338+0000) 2024-02-19T20:21:45.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:45.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:45.175587+0000) 2024-02-19T20:21:46.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:46.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:46.175819+0000) 2024-02-19T20:21:47.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:47.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:47.176064+0000) 2024-02-19T20:21:48.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:48.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:48.176305+0000) 2024-02-19T20:21:49.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:49.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:49.176480+0000) 2024-02-19T20:21:50.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:50.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:50.176652+0000) 2024-02-19T20:21:51.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:51.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:51.176896+0000) 2024-02-19T20:21:52.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:52.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:52.177135+0000) 2024-02-19T20:21:53.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:53.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:53.177403+0000) 2024-02-19T20:21:54.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:54.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:54.177652+0000) 2024-02-19T20:21:55.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:55.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:55.177907+0000) 2024-02-19T20:21:56.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:56.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:56.178182+0000) 2024-02-19T20:21:57.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:57.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:57.178366+0000) 2024-02-19T20:21:58.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:58.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:58.178647+0000) 2024-02-19T20:21:59.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:21:59.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:59.178882+0000) 2024-02-19T20:22:00.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:00.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:00.179119+0000) 2024-02-19T20:22:01.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:01.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:01.179357+0000) 2024-02-19T20:22:02.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:02.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:02.179535+0000) 2024-02-19T20:22:03.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:03.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:03.179761+0000) 2024-02-19T20:22:04.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:04.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:04.179988+0000) 2024-02-19T20:22:05.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:05.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:05.180165+0000) 2024-02-19T20:22:06.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:06.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:06.180362+0000) 2024-02-19T20:22:07.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:07.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:07.180615+0000) 2024-02-19T20:22:07.575 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:22:07.578 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:22:08.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:08.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:08.180862+0000) 2024-02-19T20:22:09.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:09.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:09.181120+0000) 2024-02-19T20:22:10.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:10.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:10.181295+0000) 2024-02-19T20:22:11.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:11.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:11.181457+0000) 2024-02-19T20:22:12.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:12.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:12.181695+0000) 2024-02-19T20:22:13.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:13.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:13.181882+0000) 2024-02-19T20:22:14.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:14.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:14.182128+0000) 2024-02-19T20:22:15.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:15.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:15.182379+0000) 2024-02-19T20:22:16.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:16.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:16.182592+0000) 2024-02-19T20:22:17.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:17.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:17.182851+0000) 2024-02-19T20:22:18.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:18.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:18.183086+0000) 2024-02-19T20:22:19.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:19.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:19.183302+0000) 2024-02-19T20:22:20.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:20.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:20.183544+0000) 2024-02-19T20:22:21.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:21.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:21.183792+0000) 2024-02-19T20:22:22.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:22.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:22.184030+0000) 2024-02-19T20:22:23.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:23.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:23.184270+0000) 2024-02-19T20:22:24.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:24.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:24.184514+0000) 2024-02-19T20:22:25.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:25.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:25.184741+0000) 2024-02-19T20:22:26.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:26.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:26.184917+0000) 2024-02-19T20:22:27.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:27.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:27.185152+0000) 2024-02-19T20:22:28.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:28.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:28.185425+0000) 2024-02-19T20:22:29.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:29.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:29.185640+0000) 2024-02-19T20:22:30.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:30.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:30.185883+0000) 2024-02-19T20:22:31.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:31.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:31.186137+0000) 2024-02-19T20:22:32.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:32.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:32.186302+0000) 2024-02-19T20:22:33.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:33.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:33.186463+0000) 2024-02-19T20:22:34.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:34.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:34.186682+0000) 2024-02-19T20:22:35.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:35.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:35.186843+0000) 2024-02-19T20:22:36.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:36.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:36.186985+0000) 2024-02-19T20:22:37.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:37.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:37.187171+0000) 2024-02-19T20:22:37.721 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:22:37.724 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:22:38.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:38.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:38.187368+0000) 2024-02-19T20:22:39.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:39.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:39.187562+0000) 2024-02-19T20:22:40.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:40.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:40.187841+0000) 2024-02-19T20:22:41.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:41.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:41.188103+0000) 2024-02-19T20:22:42.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:42.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:42.188421+0000) 2024-02-19T20:22:43.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:43.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:43.188684+0000) 2024-02-19T20:22:44.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:44.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:44.188932+0000) 2024-02-19T20:22:45.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:45.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:45.189185+0000) 2024-02-19T20:22:46.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:46.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:46.189427+0000) 2024-02-19T20:22:47.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:47.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:47.189704+0000) 2024-02-19T20:22:48.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:48.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:48.189948+0000) 2024-02-19T20:22:49.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:49.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:49.190204+0000) 2024-02-19T20:22:50.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:50.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:50.190481+0000) 2024-02-19T20:22:51.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:51.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:51.190633+0000) 2024-02-19T20:22:52.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:52.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:52.190870+0000) 2024-02-19T20:22:53.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:53.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:53.191120+0000) 2024-02-19T20:22:54.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:54.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:54.191372+0000) 2024-02-19T20:22:55.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:55.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:55.191567+0000) 2024-02-19T20:22:56.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:56.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:56.191734+0000) 2024-02-19T20:22:57.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:57.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:57.191968+0000) 2024-02-19T20:22:58.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:58.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:58.192121+0000) 2024-02-19T20:22:59.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:22:59.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:59.192359+0000) 2024-02-19T20:23:00.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:00.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:00.192519+0000) 2024-02-19T20:23:01.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:01.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:01.192757+0000) 2024-02-19T20:23:02.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:02.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:02.192979+0000) 2024-02-19T20:23:03.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:03.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:03.193168+0000) 2024-02-19T20:23:04.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:04.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:04.193362+0000) 2024-02-19T20:23:05.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:05.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:05.193646+0000) 2024-02-19T20:23:06.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:06.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:06.193876+0000) 2024-02-19T20:23:07.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:07.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:07.194049+0000) 2024-02-19T20:23:07.835 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:23:07.839 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:23:08.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:08.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:08.194296+0000) 2024-02-19T20:23:09.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:09.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:09.194491+0000) 2024-02-19T20:23:10.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:10.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:10.194613+0000) 2024-02-19T20:23:11.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:11.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:11.194836+0000) 2024-02-19T20:23:12.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:12.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:12.195017+0000) 2024-02-19T20:23:13.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:13.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:13.195221+0000) 2024-02-19T20:23:14.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:14.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:14.195402+0000) 2024-02-19T20:23:15.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:15.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:15.195640+0000) 2024-02-19T20:23:16.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:16.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:16.195896+0000) 2024-02-19T20:23:17.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:17.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:17.196126+0000) 2024-02-19T20:23:18.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:18.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:18.196343+0000) 2024-02-19T20:23:19.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:19.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:19.196576+0000) 2024-02-19T20:23:20.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:20.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:20.196824+0000) 2024-02-19T20:23:21.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:21.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:21.197084+0000) 2024-02-19T20:23:22.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:22.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:22.197267+0000) 2024-02-19T20:23:23.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:23.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:23.197494+0000) 2024-02-19T20:23:24.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:24.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:24.197756+0000) 2024-02-19T20:23:25.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:25.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:25.198021+0000) 2024-02-19T20:23:26.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:26.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:26.198275+0000) 2024-02-19T20:23:27.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:27.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:27.198506+0000) 2024-02-19T20:23:28.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:28.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:28.198856+0000) 2024-02-19T20:23:29.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:29.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:29.199035+0000) 2024-02-19T20:23:30.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:30.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:30.199178+0000) 2024-02-19T20:23:31.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:31.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:31.199358+0000) 2024-02-19T20:23:32.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:32.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:32.199525+0000) 2024-02-19T20:23:33.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:33.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:33.199769+0000) 2024-02-19T20:23:34.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:34.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:34.199991+0000) 2024-02-19T20:23:35.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:35.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:35.200230+0000) 2024-02-19T20:23:36.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:36.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:36.200496+0000) 2024-02-19T20:23:37.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:37.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:37.200769+0000) 2024-02-19T20:23:38.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:38.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:38.201034+0000) 2024-02-19T20:23:38.215 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:23:38.218 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:23:39.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:39.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:39.201290+0000) 2024-02-19T20:23:40.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:40.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:40.201485+0000) 2024-02-19T20:23:41.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:41.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:41.201742+0000) 2024-02-19T20:23:42.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:42.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:42.202026+0000) 2024-02-19T20:23:43.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:43.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:43.202308+0000) 2024-02-19T20:23:44.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:44.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:44.202585+0000) 2024-02-19T20:23:45.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:45.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:45.202828+0000) 2024-02-19T20:23:46.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:46.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:46.203105+0000) 2024-02-19T20:23:47.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:47.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:47.203366+0000) 2024-02-19T20:23:48.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:48.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:48.203609+0000) 2024-02-19T20:23:49.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:49.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:49.203837+0000) 2024-02-19T20:23:50.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:50.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:50.204012+0000) 2024-02-19T20:23:51.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:51.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:51.204242+0000) 2024-02-19T20:23:52.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:52.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:52.204388+0000) 2024-02-19T20:23:53.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:53.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:53.204613+0000) 2024-02-19T20:23:54.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:54.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:54.204760+0000) 2024-02-19T20:23:55.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:55.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:55.205034+0000) 2024-02-19T20:23:56.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:56.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:56.205314+0000) 2024-02-19T20:23:57.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:57.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:57.205586+0000) 2024-02-19T20:23:58.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:58.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:58.205766+0000) 2024-02-19T20:23:59.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:23:59.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:59.205961+0000) 2024-02-19T20:24:00.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:00.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:00.206157+0000) 2024-02-19T20:24:01.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:01.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:01.206395+0000) 2024-02-19T20:24:02.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:02.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:02.206639+0000) 2024-02-19T20:24:03.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:03.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:03.206872+0000) 2024-02-19T20:24:04.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:04.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:04.207020+0000) 2024-02-19T20:24:05.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:05.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:05.207271+0000) 2024-02-19T20:24:06.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:06.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:06.207515+0000) 2024-02-19T20:24:07.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:07.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:07.207717+0000) 2024-02-19T20:24:08.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:08.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:08.207978+0000) 2024-02-19T20:24:08.320 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:24:08.323 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:24:09.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:09.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:09.208200+0000) 2024-02-19T20:24:10.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:10.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:10.208478+0000) 2024-02-19T20:24:11.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:11.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:11.208656+0000) 2024-02-19T20:24:12.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:12.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:12.208898+0000) 2024-02-19T20:24:13.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:13.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:13.209149+0000) 2024-02-19T20:24:14.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:14.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:14.209404+0000) 2024-02-19T20:24:15.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:15.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:15.209594+0000) 2024-02-19T20:24:16.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:16.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:16.209841+0000) 2024-02-19T20:24:17.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:17.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:17.210106+0000) 2024-02-19T20:24:18.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:18.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:18.210311+0000) 2024-02-19T20:24:19.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:19.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:19.210568+0000) 2024-02-19T20:24:20.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:20.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:20.210817+0000) 2024-02-19T20:24:21.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:21.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:21.210991+0000) 2024-02-19T20:24:22.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:22.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:22.211219+0000) 2024-02-19T20:24:23.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:23.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:23.211475+0000) 2024-02-19T20:24:24.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:24.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:24.211723+0000) 2024-02-19T20:24:25.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:25.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:25.211940+0000) 2024-02-19T20:24:26.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:26.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:26.212186+0000) 2024-02-19T20:24:27.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:27.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:27.212445+0000) 2024-02-19T20:24:28.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:28.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:28.212714+0000) 2024-02-19T20:24:29.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:29.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:29.212957+0000) 2024-02-19T20:24:30.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:30.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:30.213183+0000) 2024-02-19T20:24:31.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:31.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:31.213429+0000) 2024-02-19T20:24:32.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:32.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:32.213599+0000) 2024-02-19T20:24:33.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:33.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:33.213777+0000) 2024-02-19T20:24:34.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:34.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:34.214062+0000) 2024-02-19T20:24:35.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:35.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:35.214239+0000) 2024-02-19T20:24:36.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:36.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:36.214475+0000) 2024-02-19T20:24:37.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:37.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:37.214743+0000) 2024-02-19T20:24:38.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:38.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:38.214926+0000) 2024-02-19T20:24:38.452 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:24:38.455 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:24:39.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:39.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:39.215129+0000) 2024-02-19T20:24:40.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:40.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:40.215325+0000) 2024-02-19T20:24:41.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:41.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:41.215589+0000) 2024-02-19T20:24:42.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:42.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:42.215836+0000) 2024-02-19T20:24:43.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:43.215+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:43.216015+0000) 2024-02-19T20:24:44.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:44.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:44.216284+0000) 2024-02-19T20:24:45.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:45.215+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:45.216532+0000) 2024-02-19T20:24:46.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:46.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:46.216791+0000) 2024-02-19T20:24:47.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:47.215+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:47.217023+0000) 2024-02-19T20:24:48.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:48.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:48.217285+0000) 2024-02-19T20:24:49.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:49.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:49.217549+0000) 2024-02-19T20:24:50.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:50.215+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:50.217712+0000) 2024-02-19T20:24:51.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:51.215+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:51.217960+0000) 2024-02-19T20:24:52.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:52.215+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:52.218135+0000) 2024-02-19T20:24:53.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:53.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:53.218267+0000) 2024-02-19T20:24:54.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:54.215+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:54.218427+0000) 2024-02-19T20:24:55.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:55.215+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:55.218642+0000) 2024-02-19T20:24:56.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:56.215+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:56.218955+0000) 2024-02-19T20:24:57.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:57.215+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:57.219182+0000) 2024-02-19T20:24:58.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:58.215+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:58.219427+0000) 2024-02-19T20:24:59.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:24:59.215+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:59.219680+0000) 2024-02-19T20:25:00.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:00.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:00.219929+0000) 2024-02-19T20:25:01.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:01.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:01.220207+0000) 2024-02-19T20:25:02.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:02.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:02.220443+0000) 2024-02-19T20:25:03.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:03.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:03.220690+0000) 2024-02-19T20:25:04.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:04.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:04.221009+0000) 2024-02-19T20:25:05.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:05.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:05.221221+0000) 2024-02-19T20:25:06.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:06.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:06.221417+0000) 2024-02-19T20:25:07.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:07.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:07.221669+0000) 2024-02-19T20:25:08.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:08.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:08.221880+0000) 2024-02-19T20:25:08.570 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:25:08.574 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:25:09.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:09.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:09.222084+0000) 2024-02-19T20:25:10.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:10.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:10.222328+0000) 2024-02-19T20:25:11.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:11.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:11.222585+0000) 2024-02-19T20:25:12.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:12.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:12.222855+0000) 2024-02-19T20:25:13.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:13.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:13.223097+0000) 2024-02-19T20:25:14.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:14.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:14.223288+0000) 2024-02-19T20:25:15.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:15.223+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:15.223544+0000) 2024-02-19T20:25:16.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:16.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:16.223772+0000) 2024-02-19T20:25:17.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:17.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:17.223966+0000) 2024-02-19T20:25:18.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:18.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:18.224226+0000) 2024-02-19T20:25:19.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:19.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:19.224432+0000) 2024-02-19T20:25:20.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:20.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:20.224627+0000) 2024-02-19T20:25:21.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:21.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:21.224878+0000) 2024-02-19T20:25:22.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:22.223+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:22.225128+0000) 2024-02-19T20:25:23.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:23.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:23.225398+0000) 2024-02-19T20:25:24.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:24.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:24.225641+0000) 2024-02-19T20:25:25.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:25.223+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:25.225894+0000) 2024-02-19T20:25:26.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:26.223+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:26.226146+0000) 2024-02-19T20:25:27.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:27.223+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:27.226390+0000) 2024-02-19T20:25:28.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:28.223+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:28.226643+0000) 2024-02-19T20:25:29.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:29.223+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:29.226884+0000) 2024-02-19T20:25:30.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:30.223+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:30.227086+0000) 2024-02-19T20:25:31.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:31.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:31.227328+0000) 2024-02-19T20:25:32.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:32.223+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:32.227583+0000) 2024-02-19T20:25:33.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:33.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:33.227735+0000) 2024-02-19T20:25:34.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:34.223+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:34.227992+0000) 2024-02-19T20:25:35.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:35.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:35.228239+0000) 2024-02-19T20:25:36.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:36.223+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:36.228472+0000) 2024-02-19T20:25:37.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:37.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:37.228660+0000) 2024-02-19T20:25:38.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:38.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:38.228921+0000) 2024-02-19T20:25:38.701 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:25:38.705 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:25:39.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:39.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:39.229167+0000) 2024-02-19T20:25:40.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:40.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:40.229368+0000) 2024-02-19T20:25:41.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:41.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:41.229613+0000) 2024-02-19T20:25:42.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:42.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:42.229820+0000) 2024-02-19T20:25:43.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:43.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:43.230024+0000) 2024-02-19T20:25:44.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:44.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:44.230270+0000) 2024-02-19T20:25:45.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:45.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:45.230477+0000) 2024-02-19T20:25:46.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:46.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:46.230703+0000) 2024-02-19T20:25:47.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:47.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:47.230945+0000) 2024-02-19T20:25:48.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:48.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:48.231143+0000) 2024-02-19T20:25:49.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:49.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:49.231419+0000) 2024-02-19T20:25:50.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:50.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:50.231626+0000) 2024-02-19T20:25:51.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:51.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:51.231868+0000) 2024-02-19T20:25:52.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:52.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:52.232078+0000) 2024-02-19T20:25:53.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:53.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:53.232364+0000) 2024-02-19T20:25:54.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:54.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:54.232612+0000) 2024-02-19T20:25:55.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:55.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:55.232883+0000) 2024-02-19T20:25:56.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:56.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:56.233092+0000) 2024-02-19T20:25:57.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:57.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:57.233327+0000) 2024-02-19T20:25:58.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:58.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:58.233523+0000) 2024-02-19T20:25:59.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:25:59.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:59.233752+0000) 2024-02-19T20:26:00.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:00.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:00.233980+0000) 2024-02-19T20:26:01.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:01.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:01.234153+0000) 2024-02-19T20:26:02.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:02.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:02.234387+0000) 2024-02-19T20:26:03.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:03.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:03.234605+0000) 2024-02-19T20:26:04.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:04.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:04.234813+0000) 2024-02-19T20:26:05.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:05.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:05.234976+0000) 2024-02-19T20:26:06.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:06.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:06.235208+0000) 2024-02-19T20:26:07.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:07.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:07.235390+0000) 2024-02-19T20:26:08.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:08.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:08.235691+0000) 2024-02-19T20:26:08.839 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:26:08.842 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:26:09.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:09.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:09.235973+0000) 2024-02-19T20:26:10.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:10.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:10.236261+0000) 2024-02-19T20:26:11.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:11.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:11.236451+0000) 2024-02-19T20:26:12.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:12.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:12.236657+0000) 2024-02-19T20:26:13.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:13.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:13.236920+0000) 2024-02-19T20:26:14.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:14.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:14.237163+0000) 2024-02-19T20:26:15.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:15.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:15.237402+0000) 2024-02-19T20:26:16.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:16.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:16.237652+0000) 2024-02-19T20:26:17.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:17.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:17.237936+0000) 2024-02-19T20:26:18.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:18.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:18.238139+0000) 2024-02-19T20:26:19.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:19.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:19.238378+0000) 2024-02-19T20:26:20.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:20.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:20.238615+0000) 2024-02-19T20:26:21.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:21.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:21.238855+0000) 2024-02-19T20:26:22.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:22.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:22.239065+0000) 2024-02-19T20:26:23.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:23.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:23.239267+0000) 2024-02-19T20:26:24.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:24.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:24.239474+0000) 2024-02-19T20:26:25.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:25.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:25.239721+0000) 2024-02-19T20:26:26.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:26.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:26.239916+0000) 2024-02-19T20:26:27.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:27.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:27.240159+0000) 2024-02-19T20:26:28.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:28.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:28.240401+0000) 2024-02-19T20:26:29.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:29.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:29.240645+0000) 2024-02-19T20:26:30.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:30.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:30.240917+0000) 2024-02-19T20:26:31.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:31.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:31.241118+0000) 2024-02-19T20:26:32.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:32.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:32.241319+0000) 2024-02-19T20:26:33.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:33.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:33.241450+0000) 2024-02-19T20:26:34.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:34.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:34.241683+0000) 2024-02-19T20:26:35.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:35.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:35.241892+0000) 2024-02-19T20:26:36.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:36.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:36.242156+0000) 2024-02-19T20:26:37.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:37.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:37.242416+0000) 2024-02-19T20:26:38.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:38.237+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:38.242628+0000) 2024-02-19T20:26:38.951 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:26:38.955 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:26:39.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:39.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:39.242925+0000) 2024-02-19T20:26:40.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:40.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:40.243180+0000) 2024-02-19T20:26:41.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:41.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:41.243382+0000) 2024-02-19T20:26:42.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:42.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:42.243622+0000) 2024-02-19T20:26:43.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:43.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:43.243864+0000) 2024-02-19T20:26:44.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:44.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:44.244041+0000) 2024-02-19T20:26:45.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:45.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:45.244244+0000) 2024-02-19T20:26:46.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:46.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:46.244485+0000) 2024-02-19T20:26:47.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:47.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:47.244687+0000) 2024-02-19T20:26:48.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:48.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:48.244894+0000) 2024-02-19T20:26:49.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:49.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:49.245126+0000) 2024-02-19T20:26:50.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:50.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:50.245315+0000) 2024-02-19T20:26:51.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:51.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:51.245575+0000) 2024-02-19T20:26:52.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:52.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:52.245755+0000) 2024-02-19T20:26:53.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:53.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:53.245980+0000) 2024-02-19T20:26:54.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:54.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:54.246229+0000) 2024-02-19T20:26:55.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:55.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:55.246408+0000) 2024-02-19T20:26:56.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:56.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:56.246776+0000) 2024-02-19T20:26:57.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:57.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:57.247020+0000) 2024-02-19T20:26:58.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:58.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:58.247274+0000) 2024-02-19T20:26:59.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:26:59.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:59.247518+0000) 2024-02-19T20:27:00.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:00.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:00.247758+0000) 2024-02-19T20:27:01.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:01.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:01.248006+0000) 2024-02-19T20:27:02.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:02.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:02.248242+0000) 2024-02-19T20:27:03.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:03.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:03.248443+0000) 2024-02-19T20:27:04.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:04.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:04.248664+0000) 2024-02-19T20:27:05.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:05.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:05.248899+0000) 2024-02-19T20:27:06.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:06.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:06.249176+0000) 2024-02-19T20:27:07.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:07.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:07.249413+0000) 2024-02-19T20:27:08.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:08.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:08.249661+0000) 2024-02-19T20:27:09.073 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:27:09.077 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:27:09.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:09.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:09.249916+0000) 2024-02-19T20:27:10.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:10.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:10.250072+0000) 2024-02-19T20:27:11.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:11.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:11.250277+0000) 2024-02-19T20:27:12.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:12.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:12.250514+0000) 2024-02-19T20:27:13.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:13.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:13.250799+0000) 2024-02-19T20:27:14.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:14.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:14.250995+0000) 2024-02-19T20:27:15.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:15.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:15.251235+0000) 2024-02-19T20:27:16.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:16.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:16.251484+0000) 2024-02-19T20:27:17.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:17.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:17.251664+0000) 2024-02-19T20:27:18.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:18.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:18.251866+0000) 2024-02-19T20:27:19.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:19.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:19.252089+0000) 2024-02-19T20:27:20.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:20.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:20.252258+0000) 2024-02-19T20:27:21.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:21.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:21.252458+0000) 2024-02-19T20:27:22.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:22.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:22.252674+0000) 2024-02-19T20:27:23.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:23.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:23.252932+0000) 2024-02-19T20:27:24.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:24.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:24.253209+0000) 2024-02-19T20:27:25.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:25.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:25.253359+0000) 2024-02-19T20:27:26.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:26.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:26.253553+0000) 2024-02-19T20:27:27.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:27.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:27.253815+0000) 2024-02-19T20:27:28.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:28.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:28.254080+0000) 2024-02-19T20:27:29.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:29.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:29.254357+0000) 2024-02-19T20:27:30.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:30.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:30.254563+0000) 2024-02-19T20:27:31.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:31.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:31.254739+0000) 2024-02-19T20:27:32.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:32.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:32.254908+0000) 2024-02-19T20:27:33.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:33.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:33.255081+0000) 2024-02-19T20:27:34.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:34.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:34.255264+0000) 2024-02-19T20:27:35.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:35.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:35.255522+0000) 2024-02-19T20:27:36.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:36.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:36.255704+0000) 2024-02-19T20:27:37.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:37.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:37.255931+0000) 2024-02-19T20:27:38.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:38.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:38.256194+0000) 2024-02-19T20:27:39.213 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:27:39.216 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:27:39.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:39.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:39.256396+0000) 2024-02-19T20:27:40.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:40.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:40.256628+0000) 2024-02-19T20:27:41.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:41.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:41.256834+0000) 2024-02-19T20:27:42.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:42.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:42.256996+0000) 2024-02-19T20:27:43.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:43.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:43.257231+0000) 2024-02-19T20:27:44.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:44.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:44.257470+0000) 2024-02-19T20:27:45.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:45.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:45.257723+0000) 2024-02-19T20:27:46.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:46.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:46.257962+0000) 2024-02-19T20:27:47.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:47.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:47.258133+0000) 2024-02-19T20:27:48.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:48.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:48.258305+0000) 2024-02-19T20:27:49.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:49.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:49.258497+0000) 2024-02-19T20:27:50.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:50.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:50.258755+0000) 2024-02-19T20:27:51.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:51.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:51.258930+0000) 2024-02-19T20:27:52.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:52.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:52.259083+0000) 2024-02-19T20:27:53.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:53.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:53.259259+0000) 2024-02-19T20:27:54.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:54.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:54.259439+0000) 2024-02-19T20:27:55.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:55.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:55.259675+0000) 2024-02-19T20:27:56.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:56.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:56.259927+0000) 2024-02-19T20:27:57.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:57.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:57.260168+0000) 2024-02-19T20:27:58.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:58.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:58.260432+0000) 2024-02-19T20:27:59.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:27:59.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:59.260618+0000) 2024-02-19T20:28:00.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:00.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:00.260850+0000) 2024-02-19T20:28:01.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:01.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:01.261121+0000) 2024-02-19T20:28:02.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:02.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:02.261384+0000) 2024-02-19T20:28:03.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:03.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:03.261635+0000) 2024-02-19T20:28:04.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:04.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:04.261823+0000) 2024-02-19T20:28:05.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:05.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:05.262014+0000) 2024-02-19T20:28:06.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:06.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:06.262251+0000) 2024-02-19T20:28:07.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:07.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:07.262505+0000) 2024-02-19T20:28:08.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:08.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:08.262880+0000) 2024-02-19T20:28:09.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:09.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:09.263071+0000) 2024-02-19T20:28:09.335 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:28:09.338 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:28:10.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:10.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:10.263240+0000) 2024-02-19T20:28:11.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:11.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:11.263430+0000) 2024-02-19T20:28:12.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:12.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:12.263701+0000) 2024-02-19T20:28:13.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:13.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:13.263912+0000) 2024-02-19T20:28:14.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:14.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:14.264107+0000) 2024-02-19T20:28:15.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:15.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:15.264335+0000) 2024-02-19T20:28:16.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:16.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:16.264603+0000) 2024-02-19T20:28:17.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:17.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:17.264858+0000) 2024-02-19T20:28:18.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:18.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:18.265024+0000) 2024-02-19T20:28:19.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:19.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:19.265256+0000) 2024-02-19T20:28:20.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:20.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:20.265437+0000) 2024-02-19T20:28:21.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:21.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:21.265589+0000) 2024-02-19T20:28:22.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:22.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:22.265782+0000) 2024-02-19T20:28:23.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:23.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:23.266009+0000) 2024-02-19T20:28:24.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:24.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:24.266264+0000) 2024-02-19T20:28:25.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:25.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:25.266493+0000) 2024-02-19T20:28:26.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:26.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:26.266812+0000) 2024-02-19T20:28:27.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:27.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:27.267032+0000) 2024-02-19T20:28:28.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:28.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:28.267256+0000) 2024-02-19T20:28:29.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:29.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:29.267456+0000) 2024-02-19T20:28:30.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:30.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:30.267715+0000) 2024-02-19T20:28:31.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:31.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:31.267977+0000) 2024-02-19T20:28:32.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:32.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:32.268244+0000) 2024-02-19T20:28:33.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:33.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:33.268478+0000) 2024-02-19T20:28:34.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:34.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:34.268722+0000) 2024-02-19T20:28:35.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:35.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:35.268971+0000) 2024-02-19T20:28:36.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:36.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:36.269159+0000) 2024-02-19T20:28:37.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:37.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:37.269353+0000) 2024-02-19T20:28:38.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:38.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:38.269590+0000) 2024-02-19T20:28:39.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:39.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:39.269822+0000) 2024-02-19T20:28:39.531 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:28:39.534 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:28:40.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:40.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:40.270052+0000) 2024-02-19T20:28:41.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:41.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:41.270297+0000) 2024-02-19T20:28:42.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:42.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:42.270552+0000) 2024-02-19T20:28:43.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:43.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:43.270879+0000) 2024-02-19T20:28:44.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:44.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:44.271045+0000) 2024-02-19T20:28:45.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:45.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:45.271175+0000) 2024-02-19T20:28:46.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:46.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:46.271358+0000) 2024-02-19T20:28:47.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:47.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:47.271598+0000) 2024-02-19T20:28:48.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:48.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:48.271797+0000) 2024-02-19T20:28:49.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:49.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:49.272033+0000) 2024-02-19T20:28:50.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:50.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:50.272215+0000) 2024-02-19T20:28:51.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:51.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:51.272409+0000) 2024-02-19T20:28:52.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:52.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:52.272578+0000) 2024-02-19T20:28:53.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:53.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:53.272805+0000) 2024-02-19T20:28:54.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:54.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:54.273033+0000) 2024-02-19T20:28:55.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:55.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:55.273262+0000) 2024-02-19T20:28:56.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:56.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:56.273511+0000) 2024-02-19T20:28:57.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:57.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:57.273799+0000) 2024-02-19T20:28:58.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:58.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:58.274069+0000) 2024-02-19T20:28:59.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:28:59.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:59.274304+0000) 2024-02-19T20:29:00.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:00.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:00.274491+0000) 2024-02-19T20:29:01.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:01.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:01.274660+0000) 2024-02-19T20:29:02.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:02.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:02.274810+0000) 2024-02-19T20:29:03.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:03.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:03.274942+0000) 2024-02-19T20:29:04.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:04.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:04.275129+0000) 2024-02-19T20:29:05.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:05.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:05.275275+0000) 2024-02-19T20:29:06.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:06.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:06.275460+0000) 2024-02-19T20:29:07.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:07.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:07.275745+0000) 2024-02-19T20:29:08.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:08.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:08.275974+0000) 2024-02-19T20:29:09.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:09.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:09.276198+0000) 2024-02-19T20:29:09.593 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:29:09.597 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:29:10.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:10.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:10.276416+0000) 2024-02-19T20:29:11.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:11.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:11.276658+0000) 2024-02-19T20:29:12.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:12.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:12.276933+0000) 2024-02-19T20:29:13.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:13.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:13.277245+0000) 2024-02-19T20:29:14.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:14.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:14.277513+0000) 2024-02-19T20:29:15.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:15.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:15.277771+0000) 2024-02-19T20:29:16.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:16.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:16.277981+0000) 2024-02-19T20:29:17.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:17.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:17.278148+0000) 2024-02-19T20:29:18.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:18.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:18.278469+0000) 2024-02-19T20:29:19.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:19.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:19.278773+0000) 2024-02-19T20:29:20.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:20.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:20.278940+0000) 2024-02-19T20:29:21.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:21.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:21.279148+0000) 2024-02-19T20:29:22.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:22.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:22.279380+0000) 2024-02-19T20:29:23.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:23.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:23.279707+0000) 2024-02-19T20:29:24.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:24.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:24.279973+0000) 2024-02-19T20:29:25.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:25.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:25.280192+0000) 2024-02-19T20:29:26.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:26.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:26.280479+0000) 2024-02-19T20:29:27.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:27.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:27.280692+0000) 2024-02-19T20:29:28.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:28.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:28.280941+0000) 2024-02-19T20:29:29.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:29.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:29.281175+0000) 2024-02-19T20:29:30.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:30.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:30.281451+0000) 2024-02-19T20:29:31.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:31.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:31.281709+0000) 2024-02-19T20:29:32.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:32.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:32.281966+0000) 2024-02-19T20:29:33.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:33.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:33.282221+0000) 2024-02-19T20:29:34.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:34.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:34.282399+0000) 2024-02-19T20:29:35.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:35.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:35.282527+0000) 2024-02-19T20:29:36.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:36.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:36.282912+0000) 2024-02-19T20:29:37.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:37.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:37.283112+0000) 2024-02-19T20:29:38.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:38.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:38.283385+0000) 2024-02-19T20:29:39.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:39.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:39.283640+0000) 2024-02-19T20:29:39.734 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:29:39.737 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:29:40.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:40.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:40.283903+0000) 2024-02-19T20:29:41.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:41.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:41.284196+0000) 2024-02-19T20:29:42.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:42.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:42.284478+0000) 2024-02-19T20:29:43.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:43.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:43.284786+0000) 2024-02-19T20:29:44.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:44.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:44.285052+0000) 2024-02-19T20:29:45.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:45.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:45.285288+0000) 2024-02-19T20:29:46.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:46.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:46.285501+0000) 2024-02-19T20:29:47.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:47.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:47.285738+0000) 2024-02-19T20:29:48.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:48.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:48.286020+0000) 2024-02-19T20:29:49.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:49.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:49.286216+0000) 2024-02-19T20:29:50.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:50.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:50.286492+0000) 2024-02-19T20:29:51.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:51.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:51.286693+0000) 2024-02-19T20:29:52.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:52.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:52.286906+0000) 2024-02-19T20:29:53.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:53.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:53.287051+0000) 2024-02-19T20:29:54.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:54.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:54.287188+0000) 2024-02-19T20:29:55.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:55.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:55.287441+0000) 2024-02-19T20:29:56.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:56.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:56.287739+0000) 2024-02-19T20:29:57.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:57.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:57.287920+0000) 2024-02-19T20:29:58.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:58.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:58.288098+0000) 2024-02-19T20:29:59.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:29:59.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:59.288341+0000) 2024-02-19T20:30:00.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:00.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:00.288535+0000) 2024-02-19T20:30:01.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:01.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:01.288684+0000) 2024-02-19T20:30:02.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:02.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:02.288955+0000) 2024-02-19T20:30:03.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:03.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:03.289191+0000) 2024-02-19T20:30:04.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:04.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:04.289448+0000) 2024-02-19T20:30:05.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:05.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:05.289687+0000) 2024-02-19T20:30:06.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:06.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:06.289874+0000) 2024-02-19T20:30:07.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:07.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:07.290087+0000) 2024-02-19T20:30:08.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:08.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:08.290328+0000) 2024-02-19T20:30:09.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:09.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:09.290521+0000) 2024-02-19T20:30:09.856 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:30:09.859 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:30:10.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:10.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:10.290812+0000) 2024-02-19T20:30:11.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:11.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:11.291061+0000) 2024-02-19T20:30:12.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:12.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:12.291262+0000) 2024-02-19T20:30:13.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:13.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:13.291510+0000) 2024-02-19T20:30:14.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:14.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:14.291748+0000) 2024-02-19T20:30:15.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:15.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:15.291949+0000) 2024-02-19T20:30:16.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:16.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:16.292135+0000) 2024-02-19T20:30:17.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:17.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:17.292347+0000) 2024-02-19T20:30:18.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:18.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:18.292593+0000) 2024-02-19T20:30:19.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:19.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:19.292776+0000) 2024-02-19T20:30:20.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:20.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:20.292949+0000) 2024-02-19T20:30:21.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:21.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:21.293225+0000) 2024-02-19T20:30:22.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:22.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:22.293454+0000) 2024-02-19T20:30:23.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:23.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:23.293625+0000) 2024-02-19T20:30:24.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:24.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:24.293871+0000) 2024-02-19T20:30:25.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:25.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:25.294115+0000) 2024-02-19T20:30:26.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:26.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:26.294355+0000) 2024-02-19T20:30:27.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:27.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:27.294599+0000) 2024-02-19T20:30:28.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:28.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:28.294843+0000) 2024-02-19T20:30:29.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:29.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:29.295071+0000) 2024-02-19T20:30:30.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:30.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:30.295212+0000) 2024-02-19T20:30:31.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:31.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:31.295463+0000) 2024-02-19T20:30:32.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:32.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:32.295699+0000) 2024-02-19T20:30:33.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:33.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:33.295871+0000) 2024-02-19T20:30:34.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:34.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:34.296119+0000) 2024-02-19T20:30:35.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:35.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:35.296358+0000) 2024-02-19T20:30:36.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:36.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:36.296572+0000) 2024-02-19T20:30:37.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:37.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:37.296843+0000) 2024-02-19T20:30:38.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:38.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:38.297096+0000) 2024-02-19T20:30:39.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:39.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:39.297347+0000) 2024-02-19T20:30:39.976 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:30:39.980 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:30:40.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:40.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:40.297616+0000) 2024-02-19T20:30:41.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:41.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:41.297830+0000) 2024-02-19T20:30:42.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:42.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:42.298048+0000) 2024-02-19T20:30:43.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:43.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:43.298291+0000) 2024-02-19T20:30:44.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:44.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:44.298568+0000) 2024-02-19T20:30:45.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:45.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:45.298811+0000) 2024-02-19T20:30:46.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:46.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:46.298988+0000) 2024-02-19T20:30:47.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:47.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:47.299250+0000) 2024-02-19T20:30:48.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:48.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:48.299453+0000) 2024-02-19T20:30:49.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:49.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:49.299695+0000) 2024-02-19T20:30:50.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:50.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:50.299901+0000) 2024-02-19T20:30:51.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:51.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:51.300095+0000) 2024-02-19T20:30:52.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:52.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:52.300224+0000) 2024-02-19T20:30:53.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:53.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:53.300412+0000) 2024-02-19T20:30:54.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:54.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:54.300645+0000) 2024-02-19T20:30:55.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:55.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:55.300831+0000) 2024-02-19T20:30:56.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:56.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:56.301082+0000) 2024-02-19T20:30:57.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:57.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:57.301315+0000) 2024-02-19T20:30:58.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:58.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:58.301563+0000) 2024-02-19T20:30:59.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:30:59.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:59.301815+0000) 2024-02-19T20:31:00.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:00.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:00.302100+0000) 2024-02-19T20:31:01.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:01.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:01.302348+0000) 2024-02-19T20:31:02.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:02.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:02.302542+0000) 2024-02-19T20:31:03.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:03.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:03.302811+0000) 2024-02-19T20:31:04.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:04.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:04.303059+0000) 2024-02-19T20:31:05.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:05.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:05.303300+0000) 2024-02-19T20:31:06.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:06.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:06.303571+0000) 2024-02-19T20:31:07.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:07.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:07.303826+0000) 2024-02-19T20:31:08.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:08.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:08.304002+0000) 2024-02-19T20:31:09.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:09.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:09.304175+0000) 2024-02-19T20:31:10.099 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:31:10.103 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:31:10.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:10.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:10.304348+0000) 2024-02-19T20:31:11.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:11.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:11.304637+0000) 2024-02-19T20:31:12.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:12.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:12.304940+0000) 2024-02-19T20:31:13.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:13.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:13.305144+0000) 2024-02-19T20:31:14.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:14.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:14.305274+0000) 2024-02-19T20:31:15.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:15.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:15.305523+0000) 2024-02-19T20:31:16.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:16.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:16.305771+0000) 2024-02-19T20:31:17.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:17.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:17.306024+0000) 2024-02-19T20:31:18.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:18.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:18.306273+0000) 2024-02-19T20:31:19.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:19.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:19.306514+0000) 2024-02-19T20:31:20.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:20.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:20.306815+0000) 2024-02-19T20:31:21.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:21.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:21.307061+0000) 2024-02-19T20:31:22.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:22.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:22.307304+0000) 2024-02-19T20:31:23.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:23.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:23.307558+0000) 2024-02-19T20:31:24.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:24.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:24.307756+0000) 2024-02-19T20:31:25.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:25.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:25.307986+0000) 2024-02-19T20:31:26.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:26.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:26.308195+0000) 2024-02-19T20:31:27.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:27.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:27.308428+0000) 2024-02-19T20:31:28.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:28.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:28.308676+0000) 2024-02-19T20:31:29.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:29.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:29.308958+0000) 2024-02-19T20:31:30.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:30.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:30.309193+0000) 2024-02-19T20:31:31.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:31.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:31.309475+0000) 2024-02-19T20:31:32.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:32.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:32.309715+0000) 2024-02-19T20:31:33.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:33.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:33.309948+0000) 2024-02-19T20:31:34.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:34.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:34.310201+0000) 2024-02-19T20:31:35.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:35.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:35.310450+0000) 2024-02-19T20:31:36.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:36.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:36.310691+0000) 2024-02-19T20:31:37.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:37.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:37.310876+0000) 2024-02-19T20:31:38.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:38.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:38.311065+0000) 2024-02-19T20:31:39.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:39.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:39.311252+0000) 2024-02-19T20:31:40.229 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:31:40.232 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:31:40.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:40.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:40.311488+0000) 2024-02-19T20:31:41.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:41.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:41.311747+0000) 2024-02-19T20:31:42.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:42.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:42.312013+0000) 2024-02-19T20:31:43.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:43.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:43.312261+0000) 2024-02-19T20:31:44.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:44.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:44.312497+0000) 2024-02-19T20:31:45.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:45.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:45.312677+0000) 2024-02-19T20:31:46.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:46.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:46.312851+0000) 2024-02-19T20:31:47.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:47.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:47.313147+0000) 2024-02-19T20:31:48.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:48.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:48.313316+0000) 2024-02-19T20:31:49.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:49.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:49.313573+0000) 2024-02-19T20:31:50.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:50.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:50.313840+0000) 2024-02-19T20:31:51.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:51.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:51.314095+0000) 2024-02-19T20:31:52.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:52.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:52.314312+0000) 2024-02-19T20:31:53.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:53.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:53.314492+0000) 2024-02-19T20:31:54.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:54.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:54.314685+0000) 2024-02-19T20:31:55.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:55.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:55.314858+0000) 2024-02-19T20:31:56.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:56.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:56.315075+0000) 2024-02-19T20:31:57.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:57.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:57.315313+0000) 2024-02-19T20:31:58.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:58.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:58.315533+0000) 2024-02-19T20:31:59.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:31:59.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:59.315729+0000) 2024-02-19T20:32:00.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:00.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:00.315919+0000) 2024-02-19T20:32:01.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:01.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:01.316164+0000) 2024-02-19T20:32:02.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:02.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:02.316350+0000) 2024-02-19T20:32:03.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:03.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:03.316518+0000) 2024-02-19T20:32:04.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:04.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:04.316742+0000) 2024-02-19T20:32:05.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:05.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:05.316974+0000) 2024-02-19T20:32:06.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:06.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:06.317225+0000) 2024-02-19T20:32:07.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:07.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:07.317442+0000) 2024-02-19T20:32:08.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:08.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:08.317631+0000) 2024-02-19T20:32:09.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:09.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:09.317770+0000) 2024-02-19T20:32:10.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:10.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:10.318001+0000) 2024-02-19T20:32:10.342 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:32:10.348 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:32:11.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:11.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:11.318212+0000) 2024-02-19T20:32:12.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:12.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:12.318447+0000) 2024-02-19T20:32:13.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:13.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:13.318686+0000) 2024-02-19T20:32:14.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:14.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:14.318900+0000) 2024-02-19T20:32:15.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:15.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:15.319139+0000) 2024-02-19T20:32:16.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:16.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:16.319352+0000) 2024-02-19T20:32:17.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:17.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:17.319598+0000) 2024-02-19T20:32:18.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:18.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:18.319846+0000) 2024-02-19T20:32:19.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:19.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:19.320112+0000) 2024-02-19T20:32:20.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:20.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:20.320371+0000) 2024-02-19T20:32:21.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:21.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:21.320628+0000) 2024-02-19T20:32:22.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:22.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:22.320797+0000) 2024-02-19T20:32:23.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:23.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:23.321033+0000) 2024-02-19T20:32:24.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:24.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:24.321237+0000) 2024-02-19T20:32:25.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:25.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:25.321487+0000) 2024-02-19T20:32:26.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:26.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:26.321719+0000) 2024-02-19T20:32:27.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:27.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:27.321995+0000) 2024-02-19T20:32:28.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:28.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:28.322217+0000) 2024-02-19T20:32:29.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:29.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:29.322398+0000) 2024-02-19T20:32:30.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:30.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:30.322642+0000) 2024-02-19T20:32:31.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:31.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:31.322922+0000) 2024-02-19T20:32:32.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:32.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:32.323162+0000) 2024-02-19T20:32:33.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:33.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:33.323399+0000) 2024-02-19T20:32:34.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:34.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:34.323591+0000) 2024-02-19T20:32:35.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:35.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:35.323846+0000) 2024-02-19T20:32:36.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:36.319+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:36.324120+0000) 2024-02-19T20:32:37.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:37.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:37.324408+0000) 2024-02-19T20:32:38.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:38.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:38.324672+0000) 2024-02-19T20:32:39.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:39.319+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:39.324930+0000) 2024-02-19T20:32:40.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:40.319+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:40.325143+0000) 2024-02-19T20:32:40.431 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:32:40.435 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:32:41.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:41.319+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:41.325288+0000) 2024-02-19T20:32:42.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:42.319+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:42.325486+0000) 2024-02-19T20:32:43.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:43.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:43.325755+0000) 2024-02-19T20:32:44.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:44.319+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:44.325980+0000) 2024-02-19T20:32:45.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:45.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:45.326174+0000) 2024-02-19T20:32:46.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:46.319+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:46.326359+0000) 2024-02-19T20:32:47.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:47.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:47.326607+0000) 2024-02-19T20:32:48.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:48.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:48.326840+0000) 2024-02-19T20:32:49.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:49.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:49.327100+0000) 2024-02-19T20:32:50.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:50.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:50.327305+0000) 2024-02-19T20:32:51.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:51.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:51.327597+0000) 2024-02-19T20:32:52.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:52.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:52.327786+0000) 2024-02-19T20:32:53.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:53.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:53.328009+0000) 2024-02-19T20:32:54.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:54.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:54.328172+0000) 2024-02-19T20:32:55.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:55.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:55.328367+0000) 2024-02-19T20:32:56.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:56.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:56.328595+0000) 2024-02-19T20:32:57.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:57.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:57.328847+0000) 2024-02-19T20:32:58.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:58.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:58.329166+0000) 2024-02-19T20:32:59.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:32:59.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:59.329480+0000) 2024-02-19T20:33:00.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:00.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:00.329693+0000) 2024-02-19T20:33:01.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:01.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:01.329870+0000) 2024-02-19T20:33:02.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:02.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:02.330087+0000) 2024-02-19T20:33:03.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:03.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:03.330348+0000) 2024-02-19T20:33:04.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:04.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:04.330624+0000) 2024-02-19T20:33:05.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:05.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:05.330825+0000) 2024-02-19T20:33:06.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:06.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:06.330983+0000) 2024-02-19T20:33:07.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:07.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:07.331264+0000) 2024-02-19T20:33:08.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:08.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:08.331520+0000) 2024-02-19T20:33:09.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:09.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:09.331810+0000) 2024-02-19T20:33:10.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:10.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:10.332122+0000) 2024-02-19T20:33:10.778 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:33:10.782 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:33:11.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:11.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:11.332433+0000) 2024-02-19T20:33:12.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:12.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:12.332591+0000) 2024-02-19T20:33:13.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:13.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:13.332877+0000) 2024-02-19T20:33:14.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:14.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:14.333168+0000) 2024-02-19T20:33:15.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:15.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:15.333459+0000) 2024-02-19T20:33:16.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:16.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:16.333746+0000) 2024-02-19T20:33:17.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:17.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:17.333944+0000) 2024-02-19T20:33:18.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:18.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:18.334235+0000) 2024-02-19T20:33:19.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:19.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:19.334455+0000) 2024-02-19T20:33:20.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:20.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:20.334593+0000) 2024-02-19T20:33:21.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:21.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:21.334762+0000) 2024-02-19T20:33:22.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:22.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:22.334985+0000) 2024-02-19T20:33:23.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:23.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:23.335175+0000) 2024-02-19T20:33:24.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:24.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:24.335435+0000) 2024-02-19T20:33:25.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:25.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:25.335633+0000) 2024-02-19T20:33:26.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:26.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:26.335813+0000) 2024-02-19T20:33:27.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:27.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:27.336065+0000) 2024-02-19T20:33:28.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:28.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:28.336310+0000) 2024-02-19T20:33:29.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:29.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:29.336537+0000) 2024-02-19T20:33:30.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:30.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:30.336768+0000) 2024-02-19T20:33:31.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:31.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:31.337010+0000) 2024-02-19T20:33:32.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:32.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:32.337241+0000) 2024-02-19T20:33:33.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:33.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:33.337468+0000) 2024-02-19T20:33:34.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:34.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:34.337705+0000) 2024-02-19T20:33:35.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:35.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:35.337951+0000) 2024-02-19T20:33:36.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:36.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:36.338189+0000) 2024-02-19T20:33:37.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:37.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:37.338438+0000) 2024-02-19T20:33:38.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:38.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:38.338740+0000) 2024-02-19T20:33:39.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:39.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:39.339016+0000) 2024-02-19T20:33:40.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:40.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:40.339223+0000) 2024-02-19T20:33:40.849 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:33:40.854 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:33:41.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:41.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:41.339406+0000) 2024-02-19T20:33:42.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:42.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:42.339593+0000) 2024-02-19T20:33:43.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:43.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:43.339827+0000) 2024-02-19T20:33:44.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:44.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:44.340082+0000) 2024-02-19T20:33:45.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:45.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:45.340323+0000) 2024-02-19T20:33:46.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:46.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:46.340552+0000) 2024-02-19T20:33:47.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:47.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:47.340818+0000) 2024-02-19T20:33:48.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:48.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:48.341086+0000) 2024-02-19T20:33:49.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:49.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:49.341366+0000) 2024-02-19T20:33:50.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:50.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:50.341548+0000) 2024-02-19T20:33:51.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:51.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:51.341818+0000) 2024-02-19T20:33:52.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:52.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:52.342116+0000) 2024-02-19T20:33:53.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:53.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:53.342352+0000) 2024-02-19T20:33:54.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:54.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:54.342611+0000) 2024-02-19T20:33:55.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:55.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:55.342825+0000) 2024-02-19T20:33:56.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:56.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:56.343002+0000) 2024-02-19T20:33:57.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:57.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:57.343147+0000) 2024-02-19T20:33:58.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:58.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:58.343388+0000) 2024-02-19T20:33:59.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:33:59.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:59.343630+0000) 2024-02-19T20:34:00.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:00.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:00.343871+0000) 2024-02-19T20:34:01.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:01.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:01.344065+0000) 2024-02-19T20:34:02.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:02.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:02.344206+0000) 2024-02-19T20:34:03.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:03.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:03.344390+0000) 2024-02-19T20:34:04.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:04.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:04.344651+0000) 2024-02-19T20:34:05.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:05.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:05.344877+0000) 2024-02-19T20:34:06.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:06.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:06.345059+0000) 2024-02-19T20:34:07.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:07.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:07.345333+0000) 2024-02-19T20:34:08.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:08.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:08.345621+0000) 2024-02-19T20:34:09.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:09.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:09.345866+0000) 2024-02-19T20:34:10.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:10.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:10.346002+0000) 2024-02-19T20:34:10.913 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:34:10.916 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:34:11.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:11.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:11.346186+0000) 2024-02-19T20:34:12.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:12.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:12.346395+0000) 2024-02-19T20:34:13.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:13.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:13.346578+0000) 2024-02-19T20:34:14.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:14.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:14.346824+0000) 2024-02-19T20:34:15.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:15.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:15.347072+0000) 2024-02-19T20:34:16.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:16.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:16.347245+0000) 2024-02-19T20:34:17.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:17.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:17.347457+0000) 2024-02-19T20:34:18.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:18.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:18.347668+0000) 2024-02-19T20:34:19.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:19.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:19.347925+0000) 2024-02-19T20:34:20.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:20.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:20.348160+0000) 2024-02-19T20:34:21.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:21.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:21.348385+0000) 2024-02-19T20:34:22.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:22.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:22.348622+0000) 2024-02-19T20:34:23.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:23.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:23.348801+0000) 2024-02-19T20:34:24.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:24.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:24.348941+0000) 2024-02-19T20:34:25.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:25.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:25.349188+0000) 2024-02-19T20:34:26.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:26.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:26.349427+0000) 2024-02-19T20:34:27.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:27.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:27.349680+0000) 2024-02-19T20:34:28.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:28.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:28.349923+0000) 2024-02-19T20:34:29.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:29.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:29.350115+0000) 2024-02-19T20:34:30.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:30.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:30.350354+0000) 2024-02-19T20:34:31.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:31.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:31.350525+0000) 2024-02-19T20:34:32.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:32.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:32.350769+0000) 2024-02-19T20:34:33.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:33.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:33.351016+0000) 2024-02-19T20:34:34.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:34.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:34.351253+0000) 2024-02-19T20:34:35.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:35.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:35.351525+0000) 2024-02-19T20:34:36.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:36.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:36.351733+0000) 2024-02-19T20:34:37.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:37.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:37.351985+0000) 2024-02-19T20:34:38.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:38.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:38.352158+0000) 2024-02-19T20:34:39.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:39.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:39.352384+0000) 2024-02-19T20:34:40.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:40.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:40.352558+0000) 2024-02-19T20:34:40.976 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:34:40.980 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:34:41.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:41.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:41.352777+0000) 2024-02-19T20:34:42.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:42.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:42.353030+0000) 2024-02-19T20:34:43.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:43.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:43.353265+0000) 2024-02-19T20:34:44.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:44.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:44.353436+0000) 2024-02-19T20:34:45.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:45.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:45.353693+0000) 2024-02-19T20:34:46.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:46.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:46.353936+0000) 2024-02-19T20:34:47.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:47.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:47.354223+0000) 2024-02-19T20:34:48.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:48.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:48.354472+0000) 2024-02-19T20:34:49.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:49.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:49.354657+0000) 2024-02-19T20:34:50.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:50.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:50.354943+0000) 2024-02-19T20:34:51.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:51.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:51.355138+0000) 2024-02-19T20:34:52.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:52.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:52.355341+0000) 2024-02-19T20:34:53.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:53.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:53.355574+0000) 2024-02-19T20:34:54.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:54.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:54.355768+0000) 2024-02-19T20:34:55.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:55.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:55.355947+0000) 2024-02-19T20:34:56.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:56.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:56.356191+0000) 2024-02-19T20:34:57.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:57.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:57.356438+0000) 2024-02-19T20:34:58.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:58.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:58.356683+0000) 2024-02-19T20:34:59.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:34:59.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:59.356923+0000) 2024-02-19T20:35:00.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:00.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:00.357130+0000) 2024-02-19T20:35:01.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:01.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:01.357375+0000) 2024-02-19T20:35:02.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:02.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:02.357603+0000) 2024-02-19T20:35:03.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:03.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:03.357864+0000) 2024-02-19T20:35:04.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:04.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:04.358100+0000) 2024-02-19T20:35:05.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:05.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:05.358336+0000) 2024-02-19T20:35:06.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:06.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:06.358505+0000) 2024-02-19T20:35:07.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:07.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:07.358824+0000) 2024-02-19T20:35:08.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:08.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:08.359004+0000) 2024-02-19T20:35:09.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:09.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:09.359226+0000) 2024-02-19T20:35:10.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:10.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:10.359421+0000) 2024-02-19T20:35:11.098 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:35:11.101 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:35:11.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:11.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:11.359675+0000) 2024-02-19T20:35:12.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:12.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:12.359955+0000) 2024-02-19T20:35:13.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:13.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:13.360189+0000) 2024-02-19T20:35:14.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:14.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:14.360380+0000) 2024-02-19T20:35:15.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:15.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:15.360638+0000) 2024-02-19T20:35:16.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:16.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:16.360888+0000) 2024-02-19T20:35:17.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:17.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:17.361121+0000) 2024-02-19T20:35:18.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:18.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:18.361374+0000) 2024-02-19T20:35:19.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:19.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:19.361570+0000) 2024-02-19T20:35:20.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:20.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:20.361811+0000) 2024-02-19T20:35:21.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:21.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:21.362079+0000) 2024-02-19T20:35:22.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:22.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:22.362331+0000) 2024-02-19T20:35:23.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:23.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:23.362583+0000) 2024-02-19T20:35:24.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:24.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:24.362783+0000) 2024-02-19T20:35:25.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:25.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:25.362971+0000) 2024-02-19T20:35:26.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:26.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:26.363147+0000) 2024-02-19T20:35:27.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:27.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:27.363292+0000) 2024-02-19T20:35:28.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:28.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:28.363504+0000) 2024-02-19T20:35:29.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:29.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:29.363700+0000) 2024-02-19T20:35:30.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:30.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:30.363933+0000) 2024-02-19T20:35:31.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:31.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:31.364188+0000) 2024-02-19T20:35:32.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:32.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:32.364438+0000) 2024-02-19T20:35:33.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:33.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:33.364669+0000) 2024-02-19T20:35:34.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:34.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:34.364982+0000) 2024-02-19T20:35:35.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:35.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:35.365182+0000) 2024-02-19T20:35:36.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:36.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:36.365432+0000) 2024-02-19T20:35:37.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:37.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:37.365652+0000) 2024-02-19T20:35:38.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:38.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:38.365884+0000) 2024-02-19T20:35:39.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:39.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:39.366151+0000) 2024-02-19T20:35:40.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:40.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:40.366401+0000) 2024-02-19T20:35:41.212 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:35:41.216 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:35:41.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:41.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:41.366593+0000) 2024-02-19T20:35:42.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:42.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:42.366827+0000) 2024-02-19T20:35:43.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:43.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:43.367011+0000) 2024-02-19T20:35:44.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:44.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:44.367188+0000) 2024-02-19T20:35:45.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:45.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:45.367368+0000) 2024-02-19T20:35:46.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:46.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:46.367601+0000) 2024-02-19T20:35:47.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:47.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:47.367794+0000) 2024-02-19T20:35:48.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:48.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:48.368036+0000) 2024-02-19T20:35:49.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:49.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:49.368269+0000) 2024-02-19T20:35:50.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:50.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:50.368514+0000) 2024-02-19T20:35:51.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:51.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:51.368753+0000) 2024-02-19T20:35:52.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:52.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:52.368985+0000) 2024-02-19T20:35:53.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:53.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:53.369213+0000) 2024-02-19T20:35:54.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:54.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:54.369431+0000) 2024-02-19T20:35:55.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:55.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:55.369657+0000) 2024-02-19T20:35:56.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:56.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:56.369880+0000) 2024-02-19T20:35:57.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:57.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:57.370106+0000) 2024-02-19T20:35:58.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:58.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:58.370355+0000) 2024-02-19T20:35:59.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:35:59.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:59.370557+0000) 2024-02-19T20:36:00.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:00.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:00.370830+0000) 2024-02-19T20:36:01.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:01.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:01.371057+0000) 2024-02-19T20:36:02.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:02.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:02.371334+0000) 2024-02-19T20:36:03.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:03.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:03.371595+0000) 2024-02-19T20:36:04.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:04.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:04.371789+0000) 2024-02-19T20:36:05.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:05.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:05.372017+0000) 2024-02-19T20:36:06.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:06.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:06.372232+0000) 2024-02-19T20:36:07.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:07.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:07.372482+0000) 2024-02-19T20:36:08.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:08.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:08.372705+0000) 2024-02-19T20:36:09.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:09.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:09.372885+0000) 2024-02-19T20:36:10.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:10.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:10.373109+0000) 2024-02-19T20:36:11.359 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:36:11.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:11.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:11.373362+0000) 2024-02-19T20:36:11.363 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:36:12.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:12.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:12.373559+0000) 2024-02-19T20:36:13.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:13.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:13.373741+0000) 2024-02-19T20:36:14.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:14.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:14.374008+0000) 2024-02-19T20:36:15.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:15.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:15.374270+0000) 2024-02-19T20:36:16.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:16.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:16.374534+0000) 2024-02-19T20:36:17.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:17.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:17.374852+0000) 2024-02-19T20:36:18.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:18.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:18.375062+0000) 2024-02-19T20:36:19.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:19.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:19.375329+0000) 2024-02-19T20:36:20.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:20.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:20.375547+0000) 2024-02-19T20:36:21.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:21.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:21.375748+0000) 2024-02-19T20:36:22.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:22.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:22.376011+0000) 2024-02-19T20:36:23.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:23.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:23.376323+0000) 2024-02-19T20:36:24.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:24.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:24.376605+0000) 2024-02-19T20:36:25.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:25.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:25.376947+0000) 2024-02-19T20:36:26.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:26.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:26.377165+0000) 2024-02-19T20:36:27.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:27.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:27.377470+0000) 2024-02-19T20:36:28.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:28.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:28.377777+0000) 2024-02-19T20:36:29.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:29.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:29.378100+0000) 2024-02-19T20:36:30.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:30.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:30.378382+0000) 2024-02-19T20:36:31.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:31.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:31.378584+0000) 2024-02-19T20:36:32.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:32.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:32.378889+0000) 2024-02-19T20:36:33.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:33.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:33.379172+0000) 2024-02-19T20:36:34.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:34.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:34.379408+0000) 2024-02-19T20:36:35.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:35.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:35.379678+0000) 2024-02-19T20:36:36.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:36.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:36.379979+0000) 2024-02-19T20:36:37.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:37.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:37.380230+0000) 2024-02-19T20:36:38.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:38.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:38.380494+0000) 2024-02-19T20:36:39.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:39.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:39.380727+0000) 2024-02-19T20:36:40.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:40.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:40.380985+0000) 2024-02-19T20:36:41.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:41.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:41.381203+0000) 2024-02-19T20:36:41.472 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:36:41.476 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:36:42.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:42.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:42.381390+0000) 2024-02-19T20:36:43.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:43.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:43.381578+0000) 2024-02-19T20:36:44.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:44.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:44.381832+0000) 2024-02-19T20:36:45.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:45.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:45.382072+0000) 2024-02-19T20:36:46.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:46.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:46.382305+0000) 2024-02-19T20:36:47.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:47.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:47.382538+0000) 2024-02-19T20:36:48.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:48.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:48.382697+0000) 2024-02-19T20:36:49.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:49.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:49.382999+0000) 2024-02-19T20:36:50.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:50.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:50.383189+0000) 2024-02-19T20:36:51.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:51.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:51.383429+0000) 2024-02-19T20:36:52.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:52.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:52.383621+0000) 2024-02-19T20:36:53.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:53.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:53.383832+0000) 2024-02-19T20:36:54.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:54.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:54.384070+0000) 2024-02-19T20:36:55.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:55.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:55.384349+0000) 2024-02-19T20:36:56.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:56.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:56.384585+0000) 2024-02-19T20:36:57.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:57.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:57.384758+0000) 2024-02-19T20:36:58.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:58.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:58.384959+0000) 2024-02-19T20:36:59.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:36:59.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:59.385154+0000) 2024-02-19T20:37:00.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:00.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:00.385354+0000) 2024-02-19T20:37:01.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:01.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:01.385591+0000) 2024-02-19T20:37:02.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:02.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:02.385822+0000) 2024-02-19T20:37:03.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:03.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:03.386043+0000) 2024-02-19T20:37:04.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:04.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:04.386219+0000) 2024-02-19T20:37:05.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:05.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:05.386411+0000) 2024-02-19T20:37:06.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:06.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:06.386603+0000) 2024-02-19T20:37:07.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:07.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:07.386789+0000) 2024-02-19T20:37:08.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:08.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:08.387034+0000) 2024-02-19T20:37:09.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:09.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:09.387274+0000) 2024-02-19T20:37:10.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:10.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:10.387472+0000) 2024-02-19T20:37:11.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:11.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:11.387750+0000) 2024-02-19T20:37:11.621 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:37:11.625 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:37:12.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:12.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:12.388033+0000) 2024-02-19T20:37:13.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:13.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:13.388269+0000) 2024-02-19T20:37:14.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:14.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:14.388439+0000) 2024-02-19T20:37:15.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:15.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:15.388760+0000) 2024-02-19T20:37:16.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:16.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:16.388965+0000) 2024-02-19T20:37:17.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:17.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:17.389210+0000) 2024-02-19T20:37:18.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:18.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:18.389400+0000) 2024-02-19T20:37:19.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:19.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:19.389656+0000) 2024-02-19T20:37:20.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:20.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:20.389929+0000) 2024-02-19T20:37:21.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:21.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:21.390128+0000) 2024-02-19T20:37:22.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:22.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:22.390402+0000) 2024-02-19T20:37:23.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:23.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:23.390597+0000) 2024-02-19T20:37:24.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:24.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:24.390775+0000) 2024-02-19T20:37:25.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:25.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:25.390930+0000) 2024-02-19T20:37:26.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:26.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:26.391124+0000) 2024-02-19T20:37:27.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:27.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:27.391318+0000) 2024-02-19T20:37:28.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:28.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:28.391572+0000) 2024-02-19T20:37:29.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:29.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:29.391754+0000) 2024-02-19T20:37:30.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:30.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:30.391990+0000) 2024-02-19T20:37:31.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:31.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:31.392180+0000) 2024-02-19T20:37:32.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:32.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:32.392387+0000) 2024-02-19T20:37:33.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:33.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:33.392582+0000) 2024-02-19T20:37:34.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:34.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:34.392770+0000) 2024-02-19T20:37:35.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:35.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:35.392998+0000) 2024-02-19T20:37:36.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:36.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:36.393233+0000) 2024-02-19T20:37:37.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:37.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:37.393369+0000) 2024-02-19T20:37:38.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:38.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:38.393562+0000) 2024-02-19T20:37:39.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:39.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:39.393767+0000) 2024-02-19T20:37:40.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:40.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:40.394024+0000) 2024-02-19T20:37:41.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:41.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:41.394265+0000) 2024-02-19T20:37:41.742 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:37:41.745 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:37:42.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:42.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:42.394514+0000) 2024-02-19T20:37:43.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:43.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:43.394768+0000) 2024-02-19T20:37:44.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:44.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:44.394932+0000) 2024-02-19T20:37:45.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:45.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:45.395138+0000) 2024-02-19T20:37:46.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:46.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:46.395316+0000) 2024-02-19T20:37:47.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:47.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:47.395508+0000) 2024-02-19T20:37:48.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:48.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:48.395675+0000) 2024-02-19T20:37:49.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:49.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:49.395874+0000) 2024-02-19T20:37:50.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:50.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:50.396014+0000) 2024-02-19T20:37:51.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:51.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:51.396263+0000) 2024-02-19T20:37:52.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:52.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:52.396402+0000) 2024-02-19T20:37:53.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:53.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:53.396648+0000) 2024-02-19T20:37:54.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:54.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:54.396869+0000) 2024-02-19T20:37:55.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:55.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:55.397110+0000) 2024-02-19T20:37:56.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:56.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:56.397285+0000) 2024-02-19T20:37:57.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:57.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:57.397477+0000) 2024-02-19T20:37:58.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:58.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:58.397659+0000) 2024-02-19T20:37:59.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:37:59.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:59.397873+0000) 2024-02-19T20:38:00.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:00.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:00.398018+0000) 2024-02-19T20:38:01.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:01.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:01.398248+0000) 2024-02-19T20:38:02.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:02.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:02.398396+0000) 2024-02-19T20:38:03.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:03.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:03.398630+0000) 2024-02-19T20:38:04.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:04.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:04.398783+0000) 2024-02-19T20:38:05.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:05.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:05.398965+0000) 2024-02-19T20:38:06.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:06.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:06.399175+0000) 2024-02-19T20:38:07.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:07.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:07.399413+0000) 2024-02-19T20:38:08.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:08.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:08.399611+0000) 2024-02-19T20:38:09.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:09.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:09.399773+0000) 2024-02-19T20:38:10.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:10.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:10.399982+0000) 2024-02-19T20:38:11.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:11.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:11.400262+0000) 2024-02-19T20:38:11.806 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:38:11.809 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:38:12.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:12.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:12.400432+0000) 2024-02-19T20:38:13.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:13.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:13.400615+0000) 2024-02-19T20:38:14.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:14.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:14.400815+0000) 2024-02-19T20:38:15.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:15.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:15.401063+0000) 2024-02-19T20:38:16.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:16.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:16.401301+0000) 2024-02-19T20:38:17.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:17.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:17.401494+0000) 2024-02-19T20:38:18.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:18.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:18.401735+0000) 2024-02-19T20:38:19.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:19.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:19.401979+0000) 2024-02-19T20:38:20.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:20.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:20.402219+0000) 2024-02-19T20:38:21.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:21.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:21.402478+0000) 2024-02-19T20:38:22.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:22.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:22.402691+0000) 2024-02-19T20:38:23.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:23.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:23.402887+0000) 2024-02-19T20:38:24.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:24.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:24.403047+0000) 2024-02-19T20:38:25.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:25.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:25.403244+0000) 2024-02-19T20:38:26.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:26.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:26.403492+0000) 2024-02-19T20:38:27.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:27.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:27.403670+0000) 2024-02-19T20:38:28.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:28.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:28.403887+0000) 2024-02-19T20:38:29.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:29.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:29.404041+0000) 2024-02-19T20:38:30.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:30.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:30.404318+0000) 2024-02-19T20:38:31.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:31.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:31.404519+0000) 2024-02-19T20:38:32.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:32.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:32.404750+0000) 2024-02-19T20:38:33.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:33.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:33.405003+0000) 2024-02-19T20:38:34.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:34.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:34.405225+0000) 2024-02-19T20:38:35.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:35.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:35.405427+0000) 2024-02-19T20:38:36.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:36.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:36.405703+0000) 2024-02-19T20:38:37.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:37.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:37.405942+0000) 2024-02-19T20:38:38.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:38.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:38.406187+0000) 2024-02-19T20:38:39.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:39.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:39.406439+0000) 2024-02-19T20:38:40.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:40.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:40.406739+0000) 2024-02-19T20:38:41.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:41.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:41.406987+0000) 2024-02-19T20:38:41.877 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:38:41.881 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:38:42.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:42.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:42.407240+0000) 2024-02-19T20:38:43.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:43.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:43.407461+0000) 2024-02-19T20:38:44.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:44.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:44.407719+0000) 2024-02-19T20:38:45.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:45.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:45.407913+0000) 2024-02-19T20:38:46.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:46.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:46.408173+0000) 2024-02-19T20:38:47.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:47.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:47.408355+0000) 2024-02-19T20:38:48.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:48.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:48.408557+0000) 2024-02-19T20:38:49.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:49.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:49.408780+0000) 2024-02-19T20:38:50.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:50.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:50.408959+0000) 2024-02-19T20:38:51.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:51.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:51.409174+0000) 2024-02-19T20:38:52.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:52.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:52.409426+0000) 2024-02-19T20:38:53.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:53.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:53.409671+0000) 2024-02-19T20:38:54.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:54.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:54.409920+0000) 2024-02-19T20:38:55.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:55.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:55.410302+0000) 2024-02-19T20:38:56.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:56.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:56.410658+0000) 2024-02-19T20:38:57.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:57.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:57.410908+0000) 2024-02-19T20:38:58.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:58.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:58.411104+0000) 2024-02-19T20:38:59.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:38:59.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:59.411295+0000) 2024-02-19T20:39:00.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:00.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:00.411528+0000) 2024-02-19T20:39:01.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:01.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:01.411732+0000) 2024-02-19T20:39:02.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:02.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:02.411905+0000) 2024-02-19T20:39:03.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:03.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:03.412123+0000) 2024-02-19T20:39:04.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:04.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:04.412318+0000) 2024-02-19T20:39:05.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:05.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:05.412477+0000) 2024-02-19T20:39:06.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:06.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:06.412695+0000) 2024-02-19T20:39:07.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:07.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:07.412928+0000) 2024-02-19T20:39:08.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:08.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:08.413188+0000) 2024-02-19T20:39:09.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:09.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:09.413363+0000) 2024-02-19T20:39:10.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:10.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:10.413581+0000) 2024-02-19T20:39:11.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:11.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:11.413821+0000) 2024-02-19T20:39:11.942 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:39:11.945 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:39:12.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:12.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:12.414001+0000) 2024-02-19T20:39:13.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:13.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:13.414236+0000) 2024-02-19T20:39:14.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:14.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:14.414467+0000) 2024-02-19T20:39:15.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:15.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:15.414650+0000) 2024-02-19T20:39:16.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:16.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:16.414895+0000) 2024-02-19T20:39:17.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:17.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:17.415145+0000) 2024-02-19T20:39:18.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:18.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:18.415377+0000) 2024-02-19T20:39:19.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:19.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:19.415614+0000) 2024-02-19T20:39:20.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:20.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:20.415843+0000) 2024-02-19T20:39:21.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:21.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:21.416071+0000) 2024-02-19T20:39:22.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:22.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:22.416233+0000) 2024-02-19T20:39:23.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:23.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:23.416458+0000) 2024-02-19T20:39:24.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:24.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:24.416699+0000) 2024-02-19T20:39:25.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:25.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:25.416940+0000) 2024-02-19T20:39:26.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:26.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:26.417133+0000) 2024-02-19T20:39:27.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:27.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:27.417393+0000) 2024-02-19T20:39:28.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:28.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:28.417659+0000) 2024-02-19T20:39:29.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:29.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:29.417921+0000) 2024-02-19T20:39:30.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:30.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:30.418157+0000) 2024-02-19T20:39:31.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:31.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:31.418375+0000) 2024-02-19T20:39:32.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:32.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:32.418616+0000) 2024-02-19T20:39:33.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:33.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:33.418852+0000) 2024-02-19T20:39:34.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:34.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:34.418990+0000) 2024-02-19T20:39:35.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:35.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:35.419129+0000) 2024-02-19T20:39:36.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:36.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:36.419316+0000) 2024-02-19T20:39:37.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:37.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:37.419568+0000) 2024-02-19T20:39:38.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:38.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:38.419814+0000) 2024-02-19T20:39:39.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:39.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:39.420093+0000) 2024-02-19T20:39:40.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:40.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:40.420342+0000) 2024-02-19T20:39:41.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:41.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:41.420553+0000) 2024-02-19T20:39:42.080 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:39:42.083 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:39:42.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:42.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:42.420805+0000) 2024-02-19T20:39:43.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:43.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:43.421046+0000) 2024-02-19T20:39:44.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:44.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:44.421295+0000) 2024-02-19T20:39:45.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:45.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:45.421568+0000) 2024-02-19T20:39:46.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:46.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:46.421805+0000) 2024-02-19T20:39:47.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:47.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:47.421980+0000) 2024-02-19T20:39:48.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:48.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:48.422163+0000) 2024-02-19T20:39:49.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:49.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:49.422339+0000) 2024-02-19T20:39:50.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:50.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:50.422589+0000) 2024-02-19T20:39:51.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:51.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:51.422780+0000) 2024-02-19T20:39:52.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:52.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:52.422961+0000) 2024-02-19T20:39:53.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:53.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:53.423206+0000) 2024-02-19T20:39:54.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:54.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:54.423449+0000) 2024-02-19T20:39:55.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:55.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:55.423677+0000) 2024-02-19T20:39:56.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:56.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:56.423940+0000) 2024-02-19T20:39:57.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:57.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:57.424180+0000) 2024-02-19T20:39:58.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:58.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:58.424428+0000) 2024-02-19T20:39:59.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:39:59.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:59.424607+0000) 2024-02-19T20:40:00.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:00.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:00.424871+0000) 2024-02-19T20:40:01.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:01.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:01.425078+0000) 2024-02-19T20:40:02.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:02.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:02.425333+0000) 2024-02-19T20:40:03.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:03.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:03.425612+0000) 2024-02-19T20:40:04.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:04.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:04.425850+0000) 2024-02-19T20:40:05.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:05.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:05.426090+0000) 2024-02-19T20:40:06.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:06.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:06.426330+0000) 2024-02-19T20:40:07.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:07.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:07.426566+0000) 2024-02-19T20:40:08.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:08.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:08.426771+0000) 2024-02-19T20:40:09.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:09.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:09.426944+0000) 2024-02-19T20:40:10.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:10.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:10.427148+0000) 2024-02-19T20:40:11.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:11.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:11.427402+0000) 2024-02-19T20:40:12.210 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:40:12.214 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:40:12.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:12.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:12.427641+0000) 2024-02-19T20:40:13.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:13.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:13.427828+0000) 2024-02-19T20:40:14.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:14.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:14.428067+0000) 2024-02-19T20:40:15.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:15.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:15.428238+0000) 2024-02-19T20:40:16.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:16.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:16.428476+0000) 2024-02-19T20:40:17.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:17.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:17.428663+0000) 2024-02-19T20:40:18.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:18.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:18.428916+0000) 2024-02-19T20:40:19.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:19.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:19.429087+0000) 2024-02-19T20:40:20.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:20.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:20.429288+0000) 2024-02-19T20:40:21.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:21.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:21.429533+0000) 2024-02-19T20:40:22.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:22.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:22.429793+0000) 2024-02-19T20:40:23.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:23.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:23.430036+0000) 2024-02-19T20:40:24.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:24.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:24.430262+0000) 2024-02-19T20:40:25.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:25.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:25.430508+0000) 2024-02-19T20:40:26.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:26.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:26.430736+0000) 2024-02-19T20:40:27.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:27.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:27.430934+0000) 2024-02-19T20:40:28.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:28.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:28.431209+0000) 2024-02-19T20:40:29.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:29.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:29.431452+0000) 2024-02-19T20:40:30.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:30.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:30.431687+0000) 2024-02-19T20:40:31.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:31.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:31.431828+0000) 2024-02-19T20:40:32.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:32.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:32.432058+0000) 2024-02-19T20:40:33.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:33.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:33.432262+0000) 2024-02-19T20:40:34.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:34.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:34.432514+0000) 2024-02-19T20:40:35.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:35.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:35.432756+0000) 2024-02-19T20:40:36.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:36.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:36.432990+0000) 2024-02-19T20:40:37.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:37.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:37.433227+0000) 2024-02-19T20:40:38.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:38.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:38.433416+0000) 2024-02-19T20:40:39.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:39.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:39.433661+0000) 2024-02-19T20:40:40.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:40.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:40.433840+0000) 2024-02-19T20:40:41.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:41.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:41.434076+0000) 2024-02-19T20:40:42.324 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:40:42.328 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:40:42.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:42.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:42.434274+0000) 2024-02-19T20:40:43.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:43.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:43.434581+0000) 2024-02-19T20:40:44.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:44.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:44.434792+0000) 2024-02-19T20:40:45.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:45.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:45.434932+0000) 2024-02-19T20:40:46.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:46.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:46.435180+0000) 2024-02-19T20:40:47.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:47.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:47.435430+0000) 2024-02-19T20:40:48.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:48.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:48.435709+0000) 2024-02-19T20:40:49.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:49.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:49.435944+0000) 2024-02-19T20:40:50.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:50.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:50.436216+0000) 2024-02-19T20:40:51.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:51.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:51.436482+0000) 2024-02-19T20:40:52.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:52.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:52.436655+0000) 2024-02-19T20:40:53.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:53.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:53.436831+0000) 2024-02-19T20:40:54.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:54.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:54.437005+0000) 2024-02-19T20:40:55.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:55.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:55.437264+0000) 2024-02-19T20:40:56.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:56.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:56.437479+0000) 2024-02-19T20:40:57.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:57.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:57.437677+0000) 2024-02-19T20:40:58.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:58.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:58.437955+0000) 2024-02-19T20:40:59.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:40:59.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:59.438181+0000) 2024-02-19T20:41:00.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:00.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:00.438410+0000) 2024-02-19T20:41:01.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:01.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:01.438626+0000) 2024-02-19T20:41:02.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:02.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:02.438871+0000) 2024-02-19T20:41:03.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:03.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:03.439053+0000) 2024-02-19T20:41:04.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:04.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:04.439239+0000) 2024-02-19T20:41:05.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:05.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:05.439429+0000) 2024-02-19T20:41:06.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:06.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:06.439688+0000) 2024-02-19T20:41:07.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:07.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:07.439949+0000) 2024-02-19T20:41:08.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:08.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:08.440141+0000) 2024-02-19T20:41:09.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:09.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:09.440336+0000) 2024-02-19T20:41:10.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:10.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:10.440490+0000) 2024-02-19T20:41:11.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:11.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:11.440722+0000) 2024-02-19T20:41:12.396 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:41:12.399 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:41:12.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:12.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:12.440882+0000) 2024-02-19T20:41:13.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:13.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:13.441114+0000) 2024-02-19T20:41:14.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:14.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:14.441373+0000) 2024-02-19T20:41:15.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:15.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:15.441568+0000) 2024-02-19T20:41:16.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:16.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:16.441853+0000) 2024-02-19T20:41:17.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:17.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:17.442085+0000) 2024-02-19T20:41:18.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:18.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:18.442328+0000) 2024-02-19T20:41:19.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:19.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:19.442527+0000) 2024-02-19T20:41:20.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:20.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:20.442738+0000) 2024-02-19T20:41:21.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:21.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:21.442974+0000) 2024-02-19T20:41:22.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:22.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:22.443209+0000) 2024-02-19T20:41:23.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:23.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:23.443443+0000) 2024-02-19T20:41:24.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:24.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:24.443613+0000) 2024-02-19T20:41:25.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:25.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:25.443790+0000) 2024-02-19T20:41:26.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:26.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:26.443980+0000) 2024-02-19T20:41:27.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:27.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:27.444208+0000) 2024-02-19T20:41:28.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:28.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:28.444507+0000) 2024-02-19T20:41:29.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:29.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:29.444739+0000) 2024-02-19T20:41:30.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:30.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:30.444973+0000) 2024-02-19T20:41:31.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:31.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:31.445231+0000) 2024-02-19T20:41:32.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:32.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:32.445401+0000) 2024-02-19T20:41:33.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:33.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:33.445643+0000) 2024-02-19T20:41:34.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:34.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:34.445879+0000) 2024-02-19T20:41:35.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:35.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:35.446046+0000) 2024-02-19T20:41:36.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:36.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:36.446331+0000) 2024-02-19T20:41:37.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:37.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:37.446535+0000) 2024-02-19T20:41:38.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:38.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:38.446797+0000) 2024-02-19T20:41:39.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:39.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:39.447042+0000) 2024-02-19T20:41:40.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:40.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:40.447311+0000) 2024-02-19T20:41:41.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:41.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:41.447575+0000) 2024-02-19T20:41:42.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:42.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:42.447740+0000) 2024-02-19T20:41:42.518 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:41:42.522 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:41:43.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:43.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:43.447880+0000) 2024-02-19T20:41:44.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:44.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:44.448096+0000) 2024-02-19T20:41:45.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:45.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:45.448296+0000) 2024-02-19T20:41:46.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:46.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:46.448461+0000) 2024-02-19T20:41:47.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:47.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:47.448689+0000) 2024-02-19T20:41:48.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:48.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:48.448909+0000) 2024-02-19T20:41:49.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:49.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:49.449176+0000) 2024-02-19T20:41:50.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:50.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:50.449397+0000) 2024-02-19T20:41:51.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:51.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:51.449575+0000) 2024-02-19T20:41:52.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:52.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:52.449826+0000) 2024-02-19T20:41:53.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:53.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:53.450101+0000) 2024-02-19T20:41:54.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:54.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:54.450300+0000) 2024-02-19T20:41:55.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:55.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:55.450509+0000) 2024-02-19T20:41:56.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:56.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:56.450799+0000) 2024-02-19T20:41:57.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:57.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:57.451020+0000) 2024-02-19T20:41:58.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:58.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:58.451269+0000) 2024-02-19T20:41:59.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:41:59.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:59.451503+0000) 2024-02-19T20:42:00.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:00.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:00.451753+0000) 2024-02-19T20:42:01.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:01.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:01.451958+0000) 2024-02-19T20:42:02.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:02.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:02.452174+0000) 2024-02-19T20:42:03.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:03.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:03.452352+0000) 2024-02-19T20:42:04.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:04.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:04.452599+0000) 2024-02-19T20:42:05.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:05.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:05.452825+0000) 2024-02-19T20:42:06.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:06.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:06.453037+0000) 2024-02-19T20:42:07.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:07.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:07.453175+0000) 2024-02-19T20:42:08.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:08.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:08.453418+0000) 2024-02-19T20:42:09.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:09.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:09.453632+0000) 2024-02-19T20:42:10.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:10.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:10.453893+0000) 2024-02-19T20:42:11.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:11.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:11.454154+0000) 2024-02-19T20:42:12.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:12.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:12.454368+0000) 2024-02-19T20:42:12.586 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:42:12.591 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:42:13.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:13.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:13.454617+0000) 2024-02-19T20:42:14.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:14.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:14.454901+0000) 2024-02-19T20:42:15.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:15.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:15.455056+0000) 2024-02-19T20:42:16.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:16.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:16.455205+0000) 2024-02-19T20:42:17.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:17.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:17.455502+0000) 2024-02-19T20:42:18.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:18.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:18.455723+0000) 2024-02-19T20:42:19.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:19.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:19.455966+0000) 2024-02-19T20:42:20.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:20.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:20.456161+0000) 2024-02-19T20:42:21.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:21.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:21.456410+0000) 2024-02-19T20:42:22.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:22.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:22.456638+0000) 2024-02-19T20:42:23.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:23.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:23.456807+0000) 2024-02-19T20:42:24.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:24.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:24.457035+0000) 2024-02-19T20:42:25.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:25.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:25.457248+0000) 2024-02-19T20:42:26.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:26.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:26.457497+0000) 2024-02-19T20:42:27.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:27.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:27.457776+0000) 2024-02-19T20:42:28.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:28.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:28.457996+0000) 2024-02-19T20:42:29.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:29.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:29.458192+0000) 2024-02-19T20:42:30.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:30.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:30.458453+0000) 2024-02-19T20:42:31.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:31.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:31.458741+0000) 2024-02-19T20:42:32.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:32.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:32.458963+0000) 2024-02-19T20:42:33.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:33.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:33.459185+0000) 2024-02-19T20:42:34.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:34.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:34.459433+0000) 2024-02-19T20:42:35.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:35.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:35.459672+0000) 2024-02-19T20:42:36.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:36.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:36.459882+0000) 2024-02-19T20:42:37.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:37.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:37.460147+0000) 2024-02-19T20:42:38.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:38.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:38.460394+0000) 2024-02-19T20:42:39.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:39.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:39.460633+0000) 2024-02-19T20:42:40.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:40.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:40.460907+0000) 2024-02-19T20:42:41.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:41.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:41.461154+0000) 2024-02-19T20:42:42.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:42.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:42.461388+0000) 2024-02-19T20:42:42.696 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:42:42.699 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:42:43.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:43.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:43.461629+0000) 2024-02-19T20:42:44.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:44.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:44.461872+0000) 2024-02-19T20:42:45.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:45.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:45.462043+0000) 2024-02-19T20:42:46.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:46.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:46.462279+0000) 2024-02-19T20:42:47.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:47.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:47.462540+0000) 2024-02-19T20:42:48.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:48.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:48.462822+0000) 2024-02-19T20:42:49.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:49.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:49.463073+0000) 2024-02-19T20:42:50.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:50.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:50.463294+0000) 2024-02-19T20:42:51.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:51.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:51.463545+0000) 2024-02-19T20:42:52.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:52.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:52.463716+0000) 2024-02-19T20:42:53.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:53.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:53.463938+0000) 2024-02-19T20:42:54.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:54.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:54.464178+0000) 2024-02-19T20:42:55.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:55.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:55.464411+0000) 2024-02-19T20:42:56.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:56.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:56.464663+0000) 2024-02-19T20:42:57.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:57.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:57.464936+0000) 2024-02-19T20:42:58.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:58.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:58.465167+0000) 2024-02-19T20:42:59.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:42:59.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:42:59.465397+0000) 2024-02-19T20:43:00.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:00.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:00.465638+0000) 2024-02-19T20:43:01.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:01.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:01.465920+0000) 2024-02-19T20:43:02.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:02.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:02.466158+0000) 2024-02-19T20:43:03.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:03.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:03.466392+0000) 2024-02-19T20:43:04.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:04.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:04.466635+0000) 2024-02-19T20:43:05.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:05.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:05.466823+0000) 2024-02-19T20:43:06.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:06.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:06.466990+0000) 2024-02-19T20:43:07.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:07.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:07.467217+0000) 2024-02-19T20:43:08.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:08.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:08.467460+0000) 2024-02-19T20:43:09.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:09.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:09.467707+0000) 2024-02-19T20:43:10.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:10.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:10.467921+0000) 2024-02-19T20:43:11.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:11.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:11.468153+0000) 2024-02-19T20:43:12.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:12.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:12.468383+0000) 2024-02-19T20:43:12.826 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:43:12.830 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:43:13.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:13.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:13.468609+0000) 2024-02-19T20:43:14.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:14.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:14.468887+0000) 2024-02-19T20:43:15.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:15.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:15.469106+0000) 2024-02-19T20:43:16.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:16.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:16.469329+0000) 2024-02-19T20:43:17.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:17.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:17.469571+0000) 2024-02-19T20:43:18.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:18.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:18.469832+0000) 2024-02-19T20:43:19.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:19.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:19.470050+0000) 2024-02-19T20:43:20.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:20.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:20.470263+0000) 2024-02-19T20:43:21.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:21.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:21.470514+0000) 2024-02-19T20:43:22.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:22.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:22.470811+0000) 2024-02-19T20:43:23.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:23.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:23.470970+0000) 2024-02-19T20:43:24.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:24.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:24.471162+0000) 2024-02-19T20:43:25.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:25.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:25.471330+0000) 2024-02-19T20:43:26.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:26.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:26.471557+0000) 2024-02-19T20:43:27.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:27.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:27.471812+0000) 2024-02-19T20:43:28.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:28.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:28.472094+0000) 2024-02-19T20:43:29.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:29.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:29.472273+0000) 2024-02-19T20:43:30.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:30.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:30.472445+0000) 2024-02-19T20:43:31.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:31.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:31.472635+0000) 2024-02-19T20:43:32.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:32.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:32.472893+0000) 2024-02-19T20:43:33.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:33.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:33.473090+0000) 2024-02-19T20:43:34.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:34.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:34.473329+0000) 2024-02-19T20:43:35.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:35.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:35.473529+0000) 2024-02-19T20:43:36.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:36.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:36.473753+0000) 2024-02-19T20:43:37.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:37.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:37.474031+0000) 2024-02-19T20:43:38.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:38.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:38.474276+0000) 2024-02-19T20:43:39.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:39.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:39.474485+0000) 2024-02-19T20:43:40.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:40.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:40.474752+0000) 2024-02-19T20:43:41.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:41.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:41.474911+0000) 2024-02-19T20:43:42.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:42.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:42.475086+0000) 2024-02-19T20:43:42.908 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:43:42.911 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:43:43.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:43.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:43.475282+0000) 2024-02-19T20:43:44.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:44.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:44.475515+0000) 2024-02-19T20:43:45.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:45.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:45.475752+0000) 2024-02-19T20:43:46.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:46.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:46.476017+0000) 2024-02-19T20:43:47.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:47.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:47.476205+0000) 2024-02-19T20:43:48.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:48.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:48.476397+0000) 2024-02-19T20:43:49.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:49.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:49.476564+0000) 2024-02-19T20:43:50.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:50.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:50.476705+0000) 2024-02-19T20:43:51.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:51.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:51.476930+0000) 2024-02-19T20:43:52.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:52.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:52.477192+0000) 2024-02-19T20:43:53.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:53.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:53.477450+0000) 2024-02-19T20:43:54.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:54.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:54.477650+0000) 2024-02-19T20:43:55.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:55.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:55.477885+0000) 2024-02-19T20:43:56.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:56.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:56.478116+0000) 2024-02-19T20:43:57.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:57.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:57.478351+0000) 2024-02-19T20:43:58.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:58.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:58.478479+0000) 2024-02-19T20:43:59.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:43:59.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:43:59.478696+0000) 2024-02-19T20:44:00.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:00.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:00.478951+0000) 2024-02-19T20:44:01.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:01.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:01.479194+0000) 2024-02-19T20:44:02.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:02.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:02.479446+0000) 2024-02-19T20:44:03.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:03.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:03.479702+0000) 2024-02-19T20:44:04.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:04.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:04.479888+0000) 2024-02-19T20:44:05.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:05.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:05.480126+0000) 2024-02-19T20:44:06.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:06.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:06.480271+0000) 2024-02-19T20:44:07.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:07.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:07.480507+0000) 2024-02-19T20:44:08.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:08.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:08.480751+0000) 2024-02-19T20:44:09.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:09.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:09.480992+0000) 2024-02-19T20:44:10.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:10.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:10.481189+0000) 2024-02-19T20:44:11.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:11.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:11.481393+0000) 2024-02-19T20:44:12.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:12.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:12.481621+0000) 2024-02-19T20:44:12.967 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:44:12.971 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:44:13.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:13.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:13.481859+0000) 2024-02-19T20:44:14.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:14.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:14.482039+0000) 2024-02-19T20:44:15.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:15.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:15.482281+0000) 2024-02-19T20:44:16.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:16.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:16.482523+0000) 2024-02-19T20:44:17.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:17.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:17.482806+0000) 2024-02-19T20:44:18.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:18.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:18.482939+0000) 2024-02-19T20:44:19.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:19.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:19.483161+0000) 2024-02-19T20:44:20.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:20.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:20.483344+0000) 2024-02-19T20:44:21.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:21.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:21.483583+0000) 2024-02-19T20:44:22.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:22.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:22.483777+0000) 2024-02-19T20:44:23.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:23.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:23.483989+0000) 2024-02-19T20:44:24.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:24.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:24.484228+0000) 2024-02-19T20:44:25.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:25.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:25.484465+0000) 2024-02-19T20:44:26.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:26.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:26.484718+0000) 2024-02-19T20:44:27.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:27.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:27.484859+0000) 2024-02-19T20:44:28.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:28.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:28.485077+0000) 2024-02-19T20:44:29.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:29.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:29.485321+0000) 2024-02-19T20:44:30.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:30.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:30.485460+0000) 2024-02-19T20:44:31.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:31.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:31.485684+0000) 2024-02-19T20:44:32.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:32.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:32.485961+0000) 2024-02-19T20:44:33.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:33.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:33.486201+0000) 2024-02-19T20:44:34.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:34.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:34.486419+0000) 2024-02-19T20:44:35.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:35.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:35.486660+0000) 2024-02-19T20:44:36.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:36.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:36.486916+0000) 2024-02-19T20:44:37.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:37.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:37.487105+0000) 2024-02-19T20:44:38.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:38.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:38.487253+0000) 2024-02-19T20:44:39.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:39.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:39.487512+0000) 2024-02-19T20:44:40.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:40.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:40.487731+0000) 2024-02-19T20:44:41.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:41.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:41.487998+0000) 2024-02-19T20:44:42.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:42.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:42.488244+0000) 2024-02-19T20:44:43.109 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:44:43.113 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:44:43.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:43.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:43.488472+0000) 2024-02-19T20:44:44.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:44.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:44.488719+0000) 2024-02-19T20:44:45.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:45.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:45.489000+0000) 2024-02-19T20:44:46.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:46.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:46.489140+0000) 2024-02-19T20:44:47.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:47.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:47.489361+0000) 2024-02-19T20:44:48.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:48.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:48.489602+0000) 2024-02-19T20:44:49.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:49.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:49.489837+0000) 2024-02-19T20:44:50.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:50.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:50.490093+0000) 2024-02-19T20:44:51.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:51.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:51.490307+0000) 2024-02-19T20:44:52.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:52.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:52.490529+0000) 2024-02-19T20:44:53.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:53.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:53.490747+0000) 2024-02-19T20:44:54.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:54.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:54.490904+0000) 2024-02-19T20:44:55.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:55.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:55.491052+0000) 2024-02-19T20:44:56.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:56.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:56.491222+0000) 2024-02-19T20:44:57.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:57.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:57.491422+0000) 2024-02-19T20:44:58.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:58.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:58.491672+0000) 2024-02-19T20:44:59.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:44:59.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:44:59.491909+0000) 2024-02-19T20:45:00.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:00.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:00.492168+0000) 2024-02-19T20:45:01.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:01.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:01.492405+0000) 2024-02-19T20:45:02.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:02.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:02.492595+0000) 2024-02-19T20:45:03.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:03.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:03.492831+0000) 2024-02-19T20:45:04.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:04.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:04.493095+0000) 2024-02-19T20:45:05.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:05.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:05.493356+0000) 2024-02-19T20:45:06.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:06.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:06.493608+0000) 2024-02-19T20:45:07.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:07.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:07.493810+0000) 2024-02-19T20:45:08.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:08.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:08.494039+0000) 2024-02-19T20:45:09.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:09.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:09.494239+0000) 2024-02-19T20:45:10.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:10.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:10.494499+0000) 2024-02-19T20:45:11.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:11.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:11.494911+0000) 2024-02-19T20:45:12.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:12.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:12.495078+0000) 2024-02-19T20:45:13.171 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:45:13.175 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:45:13.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:13.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:13.495324+0000) 2024-02-19T20:45:14.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:14.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:14.495597+0000) 2024-02-19T20:45:15.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:15.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:15.495859+0000) 2024-02-19T20:45:16.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:16.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:16.496098+0000) 2024-02-19T20:45:17.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:17.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:17.496343+0000) 2024-02-19T20:45:18.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:18.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:18.496580+0000) 2024-02-19T20:45:19.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:19.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:19.496762+0000) 2024-02-19T20:45:20.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:20.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:20.496946+0000) 2024-02-19T20:45:21.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:21.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:21.497190+0000) 2024-02-19T20:45:22.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:22.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:22.497384+0000) 2024-02-19T20:45:23.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:23.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:23.497610+0000) 2024-02-19T20:45:24.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:24.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:24.497856+0000) 2024-02-19T20:45:25.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:25.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:25.498040+0000) 2024-02-19T20:45:26.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:26.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:26.498200+0000) 2024-02-19T20:45:27.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:27.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:27.498357+0000) 2024-02-19T20:45:28.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:28.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:28.498553+0000) 2024-02-19T20:45:29.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:29.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:29.498826+0000) 2024-02-19T20:45:30.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:30.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:30.499003+0000) 2024-02-19T20:45:31.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:31.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:31.499169+0000) 2024-02-19T20:45:32.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:32.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:32.499373+0000) 2024-02-19T20:45:33.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:33.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:33.499602+0000) 2024-02-19T20:45:34.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:34.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:34.499854+0000) 2024-02-19T20:45:35.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:35.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:35.500021+0000) 2024-02-19T20:45:36.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:36.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:36.500215+0000) 2024-02-19T20:45:37.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:37.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:37.500453+0000) 2024-02-19T20:45:38.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:38.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:38.500669+0000) 2024-02-19T20:45:39.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:39.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:39.500870+0000) 2024-02-19T20:45:40.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:40.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:40.501105+0000) 2024-02-19T20:45:41.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:41.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:41.501356+0000) 2024-02-19T20:45:42.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:42.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:42.501508+0000) 2024-02-19T20:45:43.340 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:45:43.343 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:45:43.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:43.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:43.501744+0000) 2024-02-19T20:45:44.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:44.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:44.501933+0000) 2024-02-19T20:45:45.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:45.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:45.502167+0000) 2024-02-19T20:45:46.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:46.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:46.502403+0000) 2024-02-19T20:45:47.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:47.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:47.502651+0000) 2024-02-19T20:45:48.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:48.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:48.502851+0000) 2024-02-19T20:45:49.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:49.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:49.502984+0000) 2024-02-19T20:45:50.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:50.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:50.503230+0000) 2024-02-19T20:45:51.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:51.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:51.503422+0000) 2024-02-19T20:45:52.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:52.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:52.503666+0000) 2024-02-19T20:45:53.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:53.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:53.503877+0000) 2024-02-19T20:45:54.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:54.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:54.504153+0000) 2024-02-19T20:45:55.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:55.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:55.504331+0000) 2024-02-19T20:45:56.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:56.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:56.504543+0000) 2024-02-19T20:45:57.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:57.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:57.504727+0000) 2024-02-19T20:45:58.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:58.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:58.504916+0000) 2024-02-19T20:45:59.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:45:59.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:45:59.505104+0000) 2024-02-19T20:46:00.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:00.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:00.505304+0000) 2024-02-19T20:46:01.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:01.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:01.505574+0000) 2024-02-19T20:46:02.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:02.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:02.505796+0000) 2024-02-19T20:46:03.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:03.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:03.506017+0000) 2024-02-19T20:46:04.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:04.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:04.506273+0000) 2024-02-19T20:46:05.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:05.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:05.506452+0000) 2024-02-19T20:46:06.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:06.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:06.506692+0000) 2024-02-19T20:46:07.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:07.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:07.507008+0000) 2024-02-19T20:46:08.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:08.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:08.507273+0000) 2024-02-19T20:46:09.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:09.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:09.507488+0000) 2024-02-19T20:46:10.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:10.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:10.507734+0000) 2024-02-19T20:46:11.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:11.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:11.507953+0000) 2024-02-19T20:46:12.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:12.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:12.508184+0000) 2024-02-19T20:46:13.459 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:46:13.463 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:46:13.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:13.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:13.508401+0000) 2024-02-19T20:46:14.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:14.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:14.508622+0000) 2024-02-19T20:46:15.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:15.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:15.508839+0000) 2024-02-19T20:46:16.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:16.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:16.509071+0000) 2024-02-19T20:46:17.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:17.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:17.509306+0000) 2024-02-19T20:46:18.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:18.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:18.509565+0000) 2024-02-19T20:46:19.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:19.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:19.509707+0000) 2024-02-19T20:46:20.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:20.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:20.509910+0000) 2024-02-19T20:46:21.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:21.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:21.510184+0000) 2024-02-19T20:46:22.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:22.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:22.510394+0000) 2024-02-19T20:46:23.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:23.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:23.510628+0000) 2024-02-19T20:46:24.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:24.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:24.510820+0000) 2024-02-19T20:46:25.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:25.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:25.510995+0000) 2024-02-19T20:46:26.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:26.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:26.511179+0000) 2024-02-19T20:46:27.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:27.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:27.511376+0000) 2024-02-19T20:46:28.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:28.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:28.511634+0000) 2024-02-19T20:46:29.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:29.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:29.511866+0000) 2024-02-19T20:46:30.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:30.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:30.512102+0000) 2024-02-19T20:46:31.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:31.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:31.512301+0000) 2024-02-19T20:46:32.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:32.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:32.512560+0000) 2024-02-19T20:46:33.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:33.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:33.512722+0000) 2024-02-19T20:46:34.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:34.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:34.512928+0000) 2024-02-19T20:46:35.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:35.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:35.513192+0000) 2024-02-19T20:46:36.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:36.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:36.513451+0000) 2024-02-19T20:46:37.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:37.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:37.513678+0000) 2024-02-19T20:46:38.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:38.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:38.513950+0000) 2024-02-19T20:46:39.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:39.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:39.514179+0000) 2024-02-19T20:46:40.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:40.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:40.514416+0000) 2024-02-19T20:46:41.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:41.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:41.514569+0000) 2024-02-19T20:46:42.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:42.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:42.514790+0000) 2024-02-19T20:46:43.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:43.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:43.514999+0000) 2024-02-19T20:46:43.597 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:46:43.601 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:46:44.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:44.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:44.515166+0000) 2024-02-19T20:46:45.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:45.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:45.515353+0000) 2024-02-19T20:46:46.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:46.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:46.515650+0000) 2024-02-19T20:46:47.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:47.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:47.515844+0000) 2024-02-19T20:46:48.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:48.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:48.516031+0000) 2024-02-19T20:46:49.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:49.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:49.516172+0000) 2024-02-19T20:46:50.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:50.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:50.516432+0000) 2024-02-19T20:46:51.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:51.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:51.516694+0000) 2024-02-19T20:46:52.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:52.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:52.516946+0000) 2024-02-19T20:46:53.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:53.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:53.517176+0000) 2024-02-19T20:46:54.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:54.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:54.517305+0000) 2024-02-19T20:46:55.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:55.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:55.517574+0000) 2024-02-19T20:46:56.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:56.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:56.517897+0000) 2024-02-19T20:46:57.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:57.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:57.518197+0000) 2024-02-19T20:46:58.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:58.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:58.518517+0000) 2024-02-19T20:46:59.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:46:59.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:46:59.518676+0000) 2024-02-19T20:47:00.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:00.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:00.518908+0000) 2024-02-19T20:47:01.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:01.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:01.519118+0000) 2024-02-19T20:47:02.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:02.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:02.519404+0000) 2024-02-19T20:47:03.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:03.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:03.519622+0000) 2024-02-19T20:47:04.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:04.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:04.519810+0000) 2024-02-19T20:47:05.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:05.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:05.520057+0000) 2024-02-19T20:47:06.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:06.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:06.520256+0000) 2024-02-19T20:47:07.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:07.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:07.520519+0000) 2024-02-19T20:47:08.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:08.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:08.520833+0000) 2024-02-19T20:47:09.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:09.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:09.521063+0000) 2024-02-19T20:47:10.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:10.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:10.521316+0000) 2024-02-19T20:47:11.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:11.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:11.521590+0000) 2024-02-19T20:47:12.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:12.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:12.521899+0000) 2024-02-19T20:47:13.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:13.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:13.522226+0000) 2024-02-19T20:47:13.662 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:47:13.665 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:47:14.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:14.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:14.522486+0000) 2024-02-19T20:47:15.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:15.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:15.522699+0000) 2024-02-19T20:47:16.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:16.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:16.522941+0000) 2024-02-19T20:47:17.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:17.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:17.523218+0000) 2024-02-19T20:47:18.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:18.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:18.523441+0000) 2024-02-19T20:47:19.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:19.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:19.523706+0000) 2024-02-19T20:47:20.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:20.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:20.523930+0000) 2024-02-19T20:47:21.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:21.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:21.524184+0000) 2024-02-19T20:47:22.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:22.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:22.524497+0000) 2024-02-19T20:47:23.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:23.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:23.524748+0000) 2024-02-19T20:47:24.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:24.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:24.524938+0000) 2024-02-19T20:47:25.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:25.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:25.525194+0000) 2024-02-19T20:47:26.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:26.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:26.525506+0000) 2024-02-19T20:47:27.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:27.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:27.525822+0000) 2024-02-19T20:47:28.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:28.525+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:28.526102+0000) 2024-02-19T20:47:29.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:29.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:29.526267+0000) 2024-02-19T20:47:30.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:30.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:30.526490+0000) 2024-02-19T20:47:31.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:31.525+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:31.526671+0000) 2024-02-19T20:47:32.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:32.525+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:32.526859+0000) 2024-02-19T20:47:33.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:33.525+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:33.527105+0000) 2024-02-19T20:47:34.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:34.525+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:34.527328+0000) 2024-02-19T20:47:35.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:35.525+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:35.527626+0000) 2024-02-19T20:47:36.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:36.525+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:36.527840+0000) 2024-02-19T20:47:37.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:37.525+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:37.528030+0000) 2024-02-19T20:47:38.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:38.525+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:38.528351+0000) 2024-02-19T20:47:39.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:39.525+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:39.528644+0000) 2024-02-19T20:47:40.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:40.525+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:40.528925+0000) 2024-02-19T20:47:41.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:41.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:41.529151+0000) 2024-02-19T20:47:42.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:42.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:42.529416+0000) 2024-02-19T20:47:43.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:43.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:43.529712+0000) 2024-02-19T20:47:43.792 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:47:43.795 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:47:44.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:44.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:44.530000+0000) 2024-02-19T20:47:45.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:45.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:45.530183+0000) 2024-02-19T20:47:46.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:46.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:46.530466+0000) 2024-02-19T20:47:47.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:47.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:47.530839+0000) 2024-02-19T20:47:48.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:48.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:48.531047+0000) 2024-02-19T20:47:49.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:49.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:49.531254+0000) 2024-02-19T20:47:50.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:50.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:50.531397+0000) 2024-02-19T20:47:51.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:51.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:51.531637+0000) 2024-02-19T20:47:52.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:52.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:52.531868+0000) 2024-02-19T20:47:53.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:53.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:53.532073+0000) 2024-02-19T20:47:54.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:54.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:54.532244+0000) 2024-02-19T20:47:55.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:55.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:55.532503+0000) 2024-02-19T20:47:56.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:56.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:56.532789+0000) 2024-02-19T20:47:57.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:57.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:57.532995+0000) 2024-02-19T20:47:58.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:58.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:58.533195+0000) 2024-02-19T20:47:59.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:47:59.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:47:59.533425+0000) 2024-02-19T20:48:00.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:00.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:00.533706+0000) 2024-02-19T20:48:01.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:01.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:01.533986+0000) 2024-02-19T20:48:02.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:02.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:02.534224+0000) 2024-02-19T20:48:03.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:03.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:03.534491+0000) 2024-02-19T20:48:04.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:04.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:04.534840+0000) 2024-02-19T20:48:05.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:05.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:05.535096+0000) 2024-02-19T20:48:06.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:06.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:06.535358+0000) 2024-02-19T20:48:07.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:07.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:07.535558+0000) 2024-02-19T20:48:08.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:08.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:08.535795+0000) 2024-02-19T20:48:09.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:09.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:09.535969+0000) 2024-02-19T20:48:10.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:10.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:10.536258+0000) 2024-02-19T20:48:11.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:11.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:11.536558+0000) 2024-02-19T20:48:12.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:12.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:12.536850+0000) 2024-02-19T20:48:13.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:13.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:13.537022+0000) 2024-02-19T20:48:13.889 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:48:13.892 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:48:14.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:14.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:14.537300+0000) 2024-02-19T20:48:15.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:15.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:15.537567+0000) 2024-02-19T20:48:16.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:16.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:16.537798+0000) 2024-02-19T20:48:17.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:17.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:17.538010+0000) 2024-02-19T20:48:18.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:18.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:18.538240+0000) 2024-02-19T20:48:19.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:19.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:19.538542+0000) 2024-02-19T20:48:20.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:20.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:20.538774+0000) 2024-02-19T20:48:21.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:21.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:21.539016+0000) 2024-02-19T20:48:22.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:22.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:22.539185+0000) 2024-02-19T20:48:23.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:23.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:23.539393+0000) 2024-02-19T20:48:24.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:24.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:24.539638+0000) 2024-02-19T20:48:25.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:25.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:25.539965+0000) 2024-02-19T20:48:26.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:26.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:26.540222+0000) 2024-02-19T20:48:27.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:27.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:27.540489+0000) 2024-02-19T20:48:28.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:28.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:28.540741+0000) 2024-02-19T20:48:29.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:29.540+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:29.540939+0000) 2024-02-19T20:48:30.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:30.540+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:30.541188+0000) 2024-02-19T20:48:31.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:31.540+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:31.541472+0000) 2024-02-19T20:48:32.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:32.540+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:32.541755+0000) 2024-02-19T20:48:33.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:33.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:33.542017+0000) 2024-02-19T20:48:34.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:34.540+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:34.542318+0000) 2024-02-19T20:48:35.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:35.540+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:35.542617+0000) 2024-02-19T20:48:36.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:36.540+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:36.542903+0000) 2024-02-19T20:48:37.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:37.540+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:37.543111+0000) 2024-02-19T20:48:38.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:38.540+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:38.543295+0000) 2024-02-19T20:48:39.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:39.540+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:39.543570+0000) 2024-02-19T20:48:40.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:40.540+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:40.543762+0000) 2024-02-19T20:48:41.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:41.540+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:41.543950+0000) 2024-02-19T20:48:42.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:42.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:42.544238+0000) 2024-02-19T20:48:43.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:43.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:43.544526+0000) 2024-02-19T20:48:43.970 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:48:43.973 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:48:44.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:44.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:44.544761+0000) 2024-02-19T20:48:45.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:45.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:45.545075+0000) 2024-02-19T20:48:46.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:46.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:46.545303+0000) 2024-02-19T20:48:47.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:47.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:47.545541+0000) 2024-02-19T20:48:48.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:48.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:48.545695+0000) 2024-02-19T20:48:49.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:49.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:49.545872+0000) 2024-02-19T20:48:50.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:50.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:50.546064+0000) 2024-02-19T20:48:51.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:51.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:51.546303+0000) 2024-02-19T20:48:52.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:52.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:52.546563+0000) 2024-02-19T20:48:53.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:53.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:53.546770+0000) 2024-02-19T20:48:54.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:54.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:54.546947+0000) 2024-02-19T20:48:55.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:55.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:55.547108+0000) 2024-02-19T20:48:56.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:56.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:56.547368+0000) 2024-02-19T20:48:57.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:57.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:57.547624+0000) 2024-02-19T20:48:58.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:58.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:58.547882+0000) 2024-02-19T20:48:59.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:48:59.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:48:59.548124+0000) 2024-02-19T20:49:00.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:00.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:00.548304+0000) 2024-02-19T20:49:01.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:01.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:01.548518+0000) 2024-02-19T20:49:02.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:02.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:02.548763+0000) 2024-02-19T20:49:03.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:03.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:03.549041+0000) 2024-02-19T20:49:04.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:04.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:04.549280+0000) 2024-02-19T20:49:05.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:05.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:05.549443+0000) 2024-02-19T20:49:06.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:06.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:06.549704+0000) 2024-02-19T20:49:07.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:07.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:07.549907+0000) 2024-02-19T20:49:08.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:08.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:08.550162+0000) 2024-02-19T20:49:09.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:09.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:09.550401+0000) 2024-02-19T20:49:10.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:10.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:10.550592+0000) 2024-02-19T20:49:11.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:11.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:11.550757+0000) 2024-02-19T20:49:12.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:12.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:12.551013+0000) 2024-02-19T20:49:13.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:13.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:13.551207+0000) 2024-02-19T20:49:14.092 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:49:14.096 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:49:14.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:14.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:14.551443+0000) 2024-02-19T20:49:15.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:15.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:15.551684+0000) 2024-02-19T20:49:16.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:16.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:16.551902+0000) 2024-02-19T20:49:17.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:17.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:17.552168+0000) 2024-02-19T20:49:18.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:18.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:18.552408+0000) 2024-02-19T20:49:19.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:19.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:19.552709+0000) 2024-02-19T20:49:20.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:20.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:20.552965+0000) 2024-02-19T20:49:21.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:21.551+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:21.553221+0000) 2024-02-19T20:49:22.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:22.551+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:22.553409+0000) 2024-02-19T20:49:23.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:23.551+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:23.553641+0000) 2024-02-19T20:49:24.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:24.551+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:24.553894+0000) 2024-02-19T20:49:25.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:25.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:25.554152+0000) 2024-02-19T20:49:26.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:26.551+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:26.554434+0000) 2024-02-19T20:49:27.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:27.551+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:27.554670+0000) 2024-02-19T20:49:28.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:28.551+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:28.554866+0000) 2024-02-19T20:49:29.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:29.551+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:29.555053+0000) 2024-02-19T20:49:30.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:30.551+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:30.555289+0000) 2024-02-19T20:49:31.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:31.551+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:31.555526+0000) 2024-02-19T20:49:32.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:32.555+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:32.555758+0000) 2024-02-19T20:49:33.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:33.555+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:33.556042+0000) 2024-02-19T20:49:34.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:34.555+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:34.556237+0000) 2024-02-19T20:49:35.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:35.551+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:35.556470+0000) 2024-02-19T20:49:36.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:36.555+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:36.556692+0000) 2024-02-19T20:49:37.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:37.555+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:37.556858+0000) 2024-02-19T20:49:38.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:38.555+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:38.557127+0000) 2024-02-19T20:49:39.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:39.551+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:39.557323+0000) 2024-02-19T20:49:40.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:40.551+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:40.557494+0000) 2024-02-19T20:49:41.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:41.551+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:41.557660+0000) 2024-02-19T20:49:42.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:42.555+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:42.557785+0000) 2024-02-19T20:49:43.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:43.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:43.558051+0000) 2024-02-19T20:49:44.231 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:49:44.235 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:49:44.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:44.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:44.558309+0000) 2024-02-19T20:49:45.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:45.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:45.558517+0000) 2024-02-19T20:49:46.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:46.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:46.558757+0000) 2024-02-19T20:49:47.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:47.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:47.558958+0000) 2024-02-19T20:49:48.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:48.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:48.559203+0000) 2024-02-19T20:49:49.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:49.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:49.559473+0000) 2024-02-19T20:49:50.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:50.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:50.559740+0000) 2024-02-19T20:49:51.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:51.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:51.559909+0000) 2024-02-19T20:49:52.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:52.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:52.560166+0000) 2024-02-19T20:49:53.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:53.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:53.560366+0000) 2024-02-19T20:49:54.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:54.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:54.560607+0000) 2024-02-19T20:49:55.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:55.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:55.560843+0000) 2024-02-19T20:49:56.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:56.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:56.561056+0000) 2024-02-19T20:49:57.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:57.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:57.561297+0000) 2024-02-19T20:49:58.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:58.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:58.561523+0000) 2024-02-19T20:49:59.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:49:59.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:49:59.561782+0000) 2024-02-19T20:50:00.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:00.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:00.562000+0000) 2024-02-19T20:50:01.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:01.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:01.562251+0000) 2024-02-19T20:50:02.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:02.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:02.562480+0000) 2024-02-19T20:50:03.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:03.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:03.562674+0000) 2024-02-19T20:50:04.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:04.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:04.562933+0000) 2024-02-19T20:50:05.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:05.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:05.563174+0000) 2024-02-19T20:50:06.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:06.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:06.563415+0000) 2024-02-19T20:50:07.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:07.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:07.563628+0000) 2024-02-19T20:50:08.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:08.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:08.563942+0000) 2024-02-19T20:50:09.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:09.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:09.564134+0000) 2024-02-19T20:50:10.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:10.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:10.564373+0000) 2024-02-19T20:50:11.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:11.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:11.564559+0000) 2024-02-19T20:50:12.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:12.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:12.564847+0000) 2024-02-19T20:50:13.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:13.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:13.565044+0000) 2024-02-19T20:50:14.352 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:50:14.355 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:50:14.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:14.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:14.565282+0000) 2024-02-19T20:50:15.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:15.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:15.565486+0000) 2024-02-19T20:50:16.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:16.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:16.565748+0000) 2024-02-19T20:50:17.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:17.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:17.566004+0000) 2024-02-19T20:50:18.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:18.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:18.566246+0000) 2024-02-19T20:50:19.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:19.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:19.566493+0000) 2024-02-19T20:50:20.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:20.566+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:20.566780+0000) 2024-02-19T20:50:21.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:21.566+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:21.567037+0000) 2024-02-19T20:50:22.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:22.566+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:22.567257+0000) 2024-02-19T20:50:23.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:23.566+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:23.567503+0000) 2024-02-19T20:50:24.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:24.566+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:24.567737+0000) 2024-02-19T20:50:25.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:25.566+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:25.568014+0000) 2024-02-19T20:50:26.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:26.566+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:26.568204+0000) 2024-02-19T20:50:27.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:27.566+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:27.568381+0000) 2024-02-19T20:50:28.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:28.566+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:28.568653+0000) 2024-02-19T20:50:29.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:29.566+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:29.568829+0000) 2024-02-19T20:50:30.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:30.566+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:30.569071+0000) 2024-02-19T20:50:31.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:31.566+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:31.569298+0000) 2024-02-19T20:50:32.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:32.566+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:32.569563+0000) 2024-02-19T20:50:33.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:33.566+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:33.569795+0000) 2024-02-19T20:50:34.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:34.566+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:34.570006+0000) 2024-02-19T20:50:35.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:35.566+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:35.570201+0000) 2024-02-19T20:50:36.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:36.566+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:36.570377+0000) 2024-02-19T20:50:37.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:37.570+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:37.570634+0000) 2024-02-19T20:50:38.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:38.570+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:38.570882+0000) 2024-02-19T20:50:39.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:39.570+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:39.571131+0000) 2024-02-19T20:50:40.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:40.570+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:40.571339+0000) 2024-02-19T20:50:41.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:41.570+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:41.571589+0000) 2024-02-19T20:50:42.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:42.570+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:42.571861+0000) 2024-02-19T20:50:43.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:43.570+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:43.572082+0000) 2024-02-19T20:50:44.483 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:50:44.486 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:50:44.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:44.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:44.572339+0000) 2024-02-19T20:50:45.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:45.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:45.572598+0000) 2024-02-19T20:50:46.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:46.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:46.572850+0000) 2024-02-19T20:50:47.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:47.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:47.573053+0000) 2024-02-19T20:50:48.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:48.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:48.573340+0000) 2024-02-19T20:50:49.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:49.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:49.573582+0000) 2024-02-19T20:50:50.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:50.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:50.573820+0000) 2024-02-19T20:50:51.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:51.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:51.574075+0000) 2024-02-19T20:50:52.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:52.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:52.574317+0000) 2024-02-19T20:50:53.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:53.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:53.574555+0000) 2024-02-19T20:50:54.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:54.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:54.574690+0000) 2024-02-19T20:50:55.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:55.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:55.574954+0000) 2024-02-19T20:50:56.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:56.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:56.575161+0000) 2024-02-19T20:50:57.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:57.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:57.575365+0000) 2024-02-19T20:50:58.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:58.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:58.575548+0000) 2024-02-19T20:50:59.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:50:59.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:50:59.575831+0000) 2024-02-19T20:51:00.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:00.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:00.576082+0000) 2024-02-19T20:51:01.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:01.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:01.576337+0000) 2024-02-19T20:51:02.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:02.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:02.576581+0000) 2024-02-19T20:51:03.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:03.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:03.576777+0000) 2024-02-19T20:51:04.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:04.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:04.577048+0000) 2024-02-19T20:51:05.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:05.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:05.577293+0000) 2024-02-19T20:51:06.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:06.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:06.577540+0000) 2024-02-19T20:51:07.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:07.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:07.577832+0000) 2024-02-19T20:51:08.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:08.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:08.578029+0000) 2024-02-19T20:51:09.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:09.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:09.578302+0000) 2024-02-19T20:51:10.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:10.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:10.578545+0000) 2024-02-19T20:51:11.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:11.577+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:11.578870+0000) 2024-02-19T20:51:12.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:12.577+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:12.579093+0000) 2024-02-19T20:51:13.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:13.577+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:13.579326+0000) 2024-02-19T20:51:14.542 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:51:14.546 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:51:14.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:14.577+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:14.579523+0000) 2024-02-19T20:51:15.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:15.577+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:15.579715+0000) 2024-02-19T20:51:16.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:16.577+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:16.579964+0000) 2024-02-19T20:51:17.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:17.577+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:17.580200+0000) 2024-02-19T20:51:18.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:18.577+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:18.580447+0000) 2024-02-19T20:51:19.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:19.577+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:19.580679+0000) 2024-02-19T20:51:20.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:20.577+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:20.580817+0000) 2024-02-19T20:51:21.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:21.577+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:21.581063+0000) 2024-02-19T20:51:22.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:22.577+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:22.581311+0000) 2024-02-19T20:51:23.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:23.577+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:23.581538+0000) 2024-02-19T20:51:24.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:24.577+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:24.581751+0000) 2024-02-19T20:51:25.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:25.581+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:25.581918+0000) 2024-02-19T20:51:26.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:26.577+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:26.582144+0000) 2024-02-19T20:51:27.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:27.581+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:27.582405+0000) 2024-02-19T20:51:28.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:28.581+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:28.582691+0000) 2024-02-19T20:51:29.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:29.581+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:29.582916+0000) 2024-02-19T20:51:30.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:30.581+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:30.583093+0000) 2024-02-19T20:51:31.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:31.581+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:31.583342+0000) 2024-02-19T20:51:32.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:32.581+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:32.583580+0000) 2024-02-19T20:51:33.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:33.581+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:33.583824+0000) 2024-02-19T20:51:34.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:34.581+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:34.583983+0000) 2024-02-19T20:51:35.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:35.581+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:35.584165+0000) 2024-02-19T20:51:36.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:36.581+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:36.584346+0000) 2024-02-19T20:51:37.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:37.581+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:37.584550+0000) 2024-02-19T20:51:38.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:38.581+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:38.584795+0000) 2024-02-19T20:51:39.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:39.581+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:39.585036+0000) 2024-02-19T20:51:40.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:40.585+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:40.585275+0000) 2024-02-19T20:51:41.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:41.585+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:41.585516+0000) 2024-02-19T20:51:42.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:42.585+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:42.585741+0000) 2024-02-19T20:51:43.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:43.581+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:43.585944+0000) 2024-02-19T20:51:44.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:44.581+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:44.586142+0000) 2024-02-19T20:51:44.668 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:51:44.673 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:51:45.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:45.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:45.586345+0000) 2024-02-19T20:51:46.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:46.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:46.586584+0000) 2024-02-19T20:51:47.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:47.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:47.586788+0000) 2024-02-19T20:51:48.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:48.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:48.586996+0000) 2024-02-19T20:51:49.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:49.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:49.587184+0000) 2024-02-19T20:51:50.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:50.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:50.587332+0000) 2024-02-19T20:51:51.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:51.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:51.587524+0000) 2024-02-19T20:51:52.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:52.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:52.587735+0000) 2024-02-19T20:51:53.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:53.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:53.587976+0000) 2024-02-19T20:51:54.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:54.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:54.588215+0000) 2024-02-19T20:51:55.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:55.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:55.588479+0000) 2024-02-19T20:51:56.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:56.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:56.588714+0000) 2024-02-19T20:51:57.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:57.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:57.588890+0000) 2024-02-19T20:51:58.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:58.588+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:58.589150+0000) 2024-02-19T20:51:59.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:51:59.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:51:59.589387+0000) 2024-02-19T20:52:00.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:00.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:00.589605+0000) 2024-02-19T20:52:01.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:01.588+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:01.589869+0000) 2024-02-19T20:52:02.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:02.588+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:02.590145+0000) 2024-02-19T20:52:03.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:03.588+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:03.590392+0000) 2024-02-19T20:52:04.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:04.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:04.590591+0000) 2024-02-19T20:52:05.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:05.588+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:05.590771+0000) 2024-02-19T20:52:06.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:06.588+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:06.591070+0000) 2024-02-19T20:52:07.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:07.588+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:07.591209+0000) 2024-02-19T20:52:08.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:08.588+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:08.591413+0000) 2024-02-19T20:52:09.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:09.588+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:09.591641+0000) 2024-02-19T20:52:10.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:10.588+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:10.591802+0000) 2024-02-19T20:52:11.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:11.588+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:11.592072+0000) 2024-02-19T20:52:12.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:12.588+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:12.592382+0000) 2024-02-19T20:52:13.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:13.588+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:13.592617+0000) 2024-02-19T20:52:14.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:14.588+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:14.592788+0000) 2024-02-19T20:52:14.806 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:52:14.811 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:52:15.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:15.588+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:15.593056+0000) 2024-02-19T20:52:16.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:16.592+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:16.593324+0000) 2024-02-19T20:52:17.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:17.588+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:17.593579+0000) 2024-02-19T20:52:18.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:18.588+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:18.593875+0000) 2024-02-19T20:52:19.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:19.588+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:19.594183+0000) 2024-02-19T20:52:20.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:20.588+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:20.594460+0000) 2024-02-19T20:52:21.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:21.592+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:21.594777+0000) 2024-02-19T20:52:22.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:22.592+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:22.595023+0000) 2024-02-19T20:52:23.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:23.592+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:23.595271+0000) 2024-02-19T20:52:24.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:24.592+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:24.595547+0000) 2024-02-19T20:52:25.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:25.592+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:25.595794+0000) 2024-02-19T20:52:26.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:26.592+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:26.596039+0000) 2024-02-19T20:52:27.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:27.592+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:27.596227+0000) 2024-02-19T20:52:28.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:28.596+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:28.596527+0000) 2024-02-19T20:52:29.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:29.596+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:29.596737+0000) 2024-02-19T20:52:30.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:30.592+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:30.596984+0000) 2024-02-19T20:52:31.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:31.592+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:31.597215+0000) 2024-02-19T20:52:32.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:32.596+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:32.597491+0000) 2024-02-19T20:52:33.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:33.596+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:33.597707+0000) 2024-02-19T20:52:34.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:34.592+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:34.597910+0000) 2024-02-19T20:52:35.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:35.596+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:35.598166+0000) 2024-02-19T20:52:36.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:36.596+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:36.598366+0000) 2024-02-19T20:52:37.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:37.596+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:37.598629+0000) 2024-02-19T20:52:38.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:38.596+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:38.598885+0000) 2024-02-19T20:52:39.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:39.596+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:39.599061+0000) 2024-02-19T20:52:40.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:40.596+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:40.599202+0000) 2024-02-19T20:52:41.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:41.596+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:41.599489+0000) 2024-02-19T20:52:42.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:42.596+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:42.599767+0000) 2024-02-19T20:52:43.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:43.596+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:43.600066+0000) 2024-02-19T20:52:44.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:44.596+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:44.600363+0000) 2024-02-19T20:52:44.945 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:52:44.949 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:52:45.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:45.600+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:45.600678+0000) 2024-02-19T20:52:46.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:46.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:46.600971+0000) 2024-02-19T20:52:47.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:47.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:47.601188+0000) 2024-02-19T20:52:48.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:48.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:48.601408+0000) 2024-02-19T20:52:49.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:49.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:49.601632+0000) 2024-02-19T20:52:50.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:50.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:50.601808+0000) 2024-02-19T20:52:51.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:51.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:51.602087+0000) 2024-02-19T20:52:52.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:52.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:52.602355+0000) 2024-02-19T20:52:53.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:53.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:53.602530+0000) 2024-02-19T20:52:54.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:54.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:54.602810+0000) 2024-02-19T20:52:55.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:55.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:55.603007+0000) 2024-02-19T20:52:56.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:56.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:56.603232+0000) 2024-02-19T20:52:57.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:57.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:57.603413+0000) 2024-02-19T20:52:58.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:58.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:58.603654+0000) 2024-02-19T20:52:59.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:52:59.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:52:59.603861+0000) 2024-02-19T20:53:00.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:00.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:00.604112+0000) 2024-02-19T20:53:01.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:01.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:01.604331+0000) 2024-02-19T20:53:02.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:02.603+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:02.604603+0000) 2024-02-19T20:53:03.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:03.603+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:03.604856+0000) 2024-02-19T20:53:04.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:04.603+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:04.605028+0000) 2024-02-19T20:53:05.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:05.603+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:05.605236+0000) 2024-02-19T20:53:06.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:06.603+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:06.605427+0000) 2024-02-19T20:53:07.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:07.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:07.605706+0000) 2024-02-19T20:53:08.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:08.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:08.606026+0000) 2024-02-19T20:53:09.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:09.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:09.606336+0000) 2024-02-19T20:53:10.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:10.603+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:10.606611+0000) 2024-02-19T20:53:11.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:11.603+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:11.606926+0000) 2024-02-19T20:53:12.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:12.603+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:12.607157+0000) 2024-02-19T20:53:13.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:13.603+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:13.607455+0000) 2024-02-19T20:53:14.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:14.607+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:14.607725+0000) 2024-02-19T20:53:15.000 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:53:15.003 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:53:15.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:15.607+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:15.608028+0000) 2024-02-19T20:53:16.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:16.603+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:16.608205+0000) 2024-02-19T20:53:17.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:17.603+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:17.608474+0000) 2024-02-19T20:53:18.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:18.603+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:18.608717+0000) 2024-02-19T20:53:19.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:19.603+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:19.608976+0000) 2024-02-19T20:53:20.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:20.603+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:20.609194+0000) 2024-02-19T20:53:21.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:21.607+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:21.609374+0000) 2024-02-19T20:53:22.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:22.603+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:22.609627+0000) 2024-02-19T20:53:23.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:23.607+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:23.609916+0000) 2024-02-19T20:53:24.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:24.603+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:24.610171+0000) 2024-02-19T20:53:25.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:25.603+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:25.610441+0000) 2024-02-19T20:53:26.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:26.607+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:26.610670+0000) 2024-02-19T20:53:27.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:27.607+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:27.610897+0000) 2024-02-19T20:53:28.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:28.607+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:28.611136+0000) 2024-02-19T20:53:29.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:29.607+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:29.611367+0000) 2024-02-19T20:53:30.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:30.607+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:30.611558+0000) 2024-02-19T20:53:31.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:31.607+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:31.611816+0000) 2024-02-19T20:53:32.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:32.611+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:32.612099+0000) 2024-02-19T20:53:33.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:33.607+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:33.612403+0000) 2024-02-19T20:53:34.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:34.607+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:34.612676+0000) 2024-02-19T20:53:35.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:35.607+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:35.612934+0000) 2024-02-19T20:53:36.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:36.607+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:36.613246+0000) 2024-02-19T20:53:37.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:37.607+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:37.613551+0000) 2024-02-19T20:53:38.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:38.607+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:38.613759+0000) 2024-02-19T20:53:39.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:39.607+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:39.613942+0000) 2024-02-19T20:53:40.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:40.607+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:40.614151+0000) 2024-02-19T20:53:41.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:41.611+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:41.614402+0000) 2024-02-19T20:53:42.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:42.611+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:42.614753+0000) 2024-02-19T20:53:43.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:43.611+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:43.615031+0000) 2024-02-19T20:53:44.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:44.611+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:44.615174+0000) 2024-02-19T20:53:45.081 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:53:45.084 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:53:45.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:45.611+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:45.615394+0000) 2024-02-19T20:53:46.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:46.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:46.615649+0000) 2024-02-19T20:53:47.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:47.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:47.615905+0000) 2024-02-19T20:53:48.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:48.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:48.616190+0000) 2024-02-19T20:53:49.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:49.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:49.616439+0000) 2024-02-19T20:53:50.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:50.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:50.616711+0000) 2024-02-19T20:53:51.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:51.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:51.616880+0000) 2024-02-19T20:53:52.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:52.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:52.617108+0000) 2024-02-19T20:53:53.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:53.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:53.617357+0000) 2024-02-19T20:53:54.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:54.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:54.617525+0000) 2024-02-19T20:53:55.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:55.614+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:55.617753+0000) 2024-02-19T20:53:56.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:56.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:56.618036+0000) 2024-02-19T20:53:57.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:57.614+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:57.618330+0000) 2024-02-19T20:53:58.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:58.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:58.618626+0000) 2024-02-19T20:53:59.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:53:59.618+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:53:59.618855+0000) 2024-02-19T20:54:00.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:00.614+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:00.618986+0000) 2024-02-19T20:54:01.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:01.614+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:01.619165+0000) 2024-02-19T20:54:02.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:02.618+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:02.619369+0000) 2024-02-19T20:54:03.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:03.618+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:03.619648+0000) 2024-02-19T20:54:04.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:04.618+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:04.619904+0000) 2024-02-19T20:54:05.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:05.618+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:05.620089+0000) 2024-02-19T20:54:06.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:06.618+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:06.620314+0000) 2024-02-19T20:54:07.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:07.618+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:07.620541+0000) 2024-02-19T20:54:08.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:08.618+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:08.620736+0000) 2024-02-19T20:54:09.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:09.618+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:09.620924+0000) 2024-02-19T20:54:10.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:10.618+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:10.621081+0000) 2024-02-19T20:54:11.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:11.618+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:11.621261+0000) 2024-02-19T20:54:12.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:12.618+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:12.621478+0000) 2024-02-19T20:54:13.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:13.618+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:13.621666+0000) 2024-02-19T20:54:14.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:14.618+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:14.621894+0000) 2024-02-19T20:54:15.180 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:54:15.183 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:54:15.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:15.618+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:15.622150+0000) 2024-02-19T20:54:16.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:16.618+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:16.622351+0000) 2024-02-19T20:54:17.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:17.618+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:17.622610+0000) 2024-02-19T20:54:18.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:18.622+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:18.622852+0000) 2024-02-19T20:54:19.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:19.622+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:19.623129+0000) 2024-02-19T20:54:20.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:20.622+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:20.623383+0000) 2024-02-19T20:54:21.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:21.622+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:21.623578+0000) 2024-02-19T20:54:22.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:22.622+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:22.623792+0000) 2024-02-19T20:54:23.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:23.622+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:23.624044+0000) 2024-02-19T20:54:24.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:24.622+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:24.624278+0000) 2024-02-19T20:54:25.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:25.622+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:25.624523+0000) 2024-02-19T20:54:26.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:26.622+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:26.624782+0000) 2024-02-19T20:54:27.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:27.622+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:27.624996+0000) 2024-02-19T20:54:28.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:28.622+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:28.625229+0000) 2024-02-19T20:54:29.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:29.622+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:29.625471+0000) 2024-02-19T20:54:30.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:30.622+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:30.625640+0000) 2024-02-19T20:54:31.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:31.622+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:31.625817+0000) 2024-02-19T20:54:32.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:32.622+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:32.626033+0000) 2024-02-19T20:54:33.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:33.622+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:33.626180+0000) 2024-02-19T20:54:34.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:34.622+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:34.626302+0000) 2024-02-19T20:54:35.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:35.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:35.626511+0000) 2024-02-19T20:54:36.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:36.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:36.626702+0000) 2024-02-19T20:54:37.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:37.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:37.626968+0000) 2024-02-19T20:54:38.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:38.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:38.627108+0000) 2024-02-19T20:54:39.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:39.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:39.627384+0000) 2024-02-19T20:54:40.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:40.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:40.627560+0000) 2024-02-19T20:54:41.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:41.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:41.627772+0000) 2024-02-19T20:54:42.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:42.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:42.628025+0000) 2024-02-19T20:54:43.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:43.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:43.628295+0000) 2024-02-19T20:54:44.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:44.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:44.628492+0000) 2024-02-19T20:54:45.292 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:54:45.295 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:54:45.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:45.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:45.628672+0000) 2024-02-19T20:54:46.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:46.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:46.628859+0000) 2024-02-19T20:54:47.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:47.625+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:47.629085+0000) 2024-02-19T20:54:48.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:48.625+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:48.629320+0000) 2024-02-19T20:54:49.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:49.625+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:49.629555+0000) 2024-02-19T20:54:50.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:50.625+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:50.629792+0000) 2024-02-19T20:54:51.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:51.629+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:51.630055+0000) 2024-02-19T20:54:52.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:52.625+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:52.630275+0000) 2024-02-19T20:54:53.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:53.629+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:53.630517+0000) 2024-02-19T20:54:54.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:54.629+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:54.630703+0000) 2024-02-19T20:54:55.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:55.629+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:55.630961+0000) 2024-02-19T20:54:56.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:56.629+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:56.631168+0000) 2024-02-19T20:54:57.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:57.629+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:57.631433+0000) 2024-02-19T20:54:58.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:58.629+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:58.631672+0000) 2024-02-19T20:54:59.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:54:59.629+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:54:59.631921+0000) 2024-02-19T20:55:00.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:00.629+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:00.632111+0000) 2024-02-19T20:55:01.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:01.629+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:01.632354+0000) 2024-02-19T20:55:02.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:02.629+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:02.632552+0000) 2024-02-19T20:55:03.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:03.629+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:03.632726+0000) 2024-02-19T20:55:04.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:04.629+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:04.632861+0000) 2024-02-19T20:55:05.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:05.629+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:05.633100+0000) 2024-02-19T20:55:06.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:06.629+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:06.633329+0000) 2024-02-19T20:55:07.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:07.629+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:07.633581+0000) 2024-02-19T20:55:08.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:08.633+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:08.633800+0000) 2024-02-19T20:55:09.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:09.629+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:09.634001+0000) 2024-02-19T20:55:10.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:10.629+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:10.634181+0000) 2024-02-19T20:55:11.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:11.629+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:11.634423+0000) 2024-02-19T20:55:12.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:12.629+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:12.634612+0000) 2024-02-19T20:55:13.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:13.633+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:13.634858+0000) 2024-02-19T20:55:14.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:14.633+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:14.635096+0000) 2024-02-19T20:55:15.406 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:55:15.410 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:55:15.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:15.633+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:15.635295+0000) 2024-02-19T20:55:16.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:16.633+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:16.635547+0000) 2024-02-19T20:55:17.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:17.633+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:17.635779+0000) 2024-02-19T20:55:18.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:18.633+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:18.635988+0000) 2024-02-19T20:55:19.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:19.633+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:19.636209+0000) 2024-02-19T20:55:20.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:20.633+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:20.636445+0000) 2024-02-19T20:55:21.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:21.633+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:21.636680+0000) 2024-02-19T20:55:22.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:22.633+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:22.636878+0000) 2024-02-19T20:55:23.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:23.633+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:23.637199+0000) 2024-02-19T20:55:24.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:24.633+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:24.637475+0000) 2024-02-19T20:55:25.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:25.633+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:25.637709+0000) 2024-02-19T20:55:26.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:26.633+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:26.637948+0000) 2024-02-19T20:55:27.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:27.637+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:27.638193+0000) 2024-02-19T20:55:28.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:28.633+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:28.638442+0000) 2024-02-19T20:55:29.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:29.637+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:29.638753+0000) 2024-02-19T20:55:30.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:30.637+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:30.638925+0000) 2024-02-19T20:55:31.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:31.637+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:31.639168+0000) 2024-02-19T20:55:32.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:32.637+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:32.639330+0000) 2024-02-19T20:55:33.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:33.637+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:33.639583+0000) 2024-02-19T20:55:34.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:34.637+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:34.639840+0000) 2024-02-19T20:55:35.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:35.637+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:35.640074+0000) 2024-02-19T20:55:36.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:36.637+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:36.640320+0000) 2024-02-19T20:55:37.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:37.637+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:37.640493+0000) 2024-02-19T20:55:38.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:38.637+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:38.640729+0000) 2024-02-19T20:55:39.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:39.637+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:39.640947+0000) 2024-02-19T20:55:40.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:40.637+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:40.641136+0000) 2024-02-19T20:55:41.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:41.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:41.641341+0000) 2024-02-19T20:55:42.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:42.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:42.641522+0000) 2024-02-19T20:55:43.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:43.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:43.641765+0000) 2024-02-19T20:55:44.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:44.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:44.642023+0000) 2024-02-19T20:55:45.554 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:55:45.559 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:55:45.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:45.637+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:45.642283+0000) 2024-02-19T20:55:46.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:46.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:46.642481+0000) 2024-02-19T20:55:47.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:47.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:47.642797+0000) 2024-02-19T20:55:48.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:48.640+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:48.643080+0000) 2024-02-19T20:55:49.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:49.640+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:49.643256+0000) 2024-02-19T20:55:50.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:50.640+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:50.643456+0000) 2024-02-19T20:55:51.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:51.640+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:51.643727+0000) 2024-02-19T20:55:52.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:52.640+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:52.643984+0000) 2024-02-19T20:55:53.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:53.640+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:53.644217+0000) 2024-02-19T20:55:54.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:54.640+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:54.644412+0000) 2024-02-19T20:55:55.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:55.640+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:55.644655+0000) 2024-02-19T20:55:56.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:56.640+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:56.644896+0000) 2024-02-19T20:55:57.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:57.644+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:57.645131+0000) 2024-02-19T20:55:58.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:58.644+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:58.645394+0000) 2024-02-19T20:55:59.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:55:59.644+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:55:59.645663+0000) 2024-02-19T20:56:00.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:00.640+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:00.645827+0000) 2024-02-19T20:56:01.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:01.640+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:01.646081+0000) 2024-02-19T20:56:02.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:02.644+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:02.646342+0000) 2024-02-19T20:56:03.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:03.644+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:03.646546+0000) 2024-02-19T20:56:04.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:04.644+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:04.646818+0000) 2024-02-19T20:56:05.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:05.644+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:05.647043+0000) 2024-02-19T20:56:06.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:06.644+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:06.647236+0000) 2024-02-19T20:56:07.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:07.644+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:07.647483+0000) 2024-02-19T20:56:08.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:08.644+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:08.647714+0000) 2024-02-19T20:56:09.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:09.644+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:09.647938+0000) 2024-02-19T20:56:10.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:10.644+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:10.648192+0000) 2024-02-19T20:56:11.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:11.644+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:11.648452+0000) 2024-02-19T20:56:12.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:12.644+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:12.648703+0000) 2024-02-19T20:56:13.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:13.648+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:13.648948+0000) 2024-02-19T20:56:14.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:14.644+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:14.649192+0000) 2024-02-19T20:56:15.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:15.648+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:15.649449+0000) 2024-02-19T20:56:15.684 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:56:15.687 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:56:16.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:16.648+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:16.649715+0000) 2024-02-19T20:56:17.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:17.644+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:17.649878+0000) 2024-02-19T20:56:18.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:18.648+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:18.650119+0000) 2024-02-19T20:56:19.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:19.648+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:19.650312+0000) 2024-02-19T20:56:20.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:20.648+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:20.650538+0000) 2024-02-19T20:56:21.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:21.648+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:21.650799+0000) 2024-02-19T20:56:22.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:22.648+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:22.650929+0000) 2024-02-19T20:56:23.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:23.648+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:23.651107+0000) 2024-02-19T20:56:24.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:24.648+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:24.651400+0000) 2024-02-19T20:56:25.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:25.648+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:25.651609+0000) 2024-02-19T20:56:26.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:26.648+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:26.651851+0000) 2024-02-19T20:56:27.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:27.648+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:27.652057+0000) 2024-02-19T20:56:28.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:28.648+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:28.652225+0000) 2024-02-19T20:56:29.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:29.652+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:29.652461+0000) 2024-02-19T20:56:30.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:30.652+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:30.652677+0000) 2024-02-19T20:56:31.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:31.652+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:31.652917+0000) 2024-02-19T20:56:32.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:32.648+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:32.653168+0000) 2024-02-19T20:56:33.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:33.652+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:33.653436+0000) 2024-02-19T20:56:34.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:34.652+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:34.653674+0000) 2024-02-19T20:56:35.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:35.648+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:35.653931+0000) 2024-02-19T20:56:36.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:36.648+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:36.654130+0000) 2024-02-19T20:56:37.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:37.648+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:37.654359+0000) 2024-02-19T20:56:38.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:38.648+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:38.654618+0000) 2024-02-19T20:56:39.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:39.652+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:39.654873+0000) 2024-02-19T20:56:40.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:40.652+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:40.655115+0000) 2024-02-19T20:56:41.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:41.652+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:41.655309+0000) 2024-02-19T20:56:42.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:42.652+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:42.655549+0000) 2024-02-19T20:56:43.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:43.652+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:43.655827+0000) 2024-02-19T20:56:44.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:44.652+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:44.656109+0000) 2024-02-19T20:56:45.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:45.656+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:45.656313+0000) 2024-02-19T20:56:45.739 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:56:45.743 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:56:46.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:46.656+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:46.656527+0000) 2024-02-19T20:56:47.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:47.652+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:47.656766+0000) 2024-02-19T20:56:48.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:48.652+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:48.657003+0000) 2024-02-19T20:56:49.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:49.655+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:49.657251+0000) 2024-02-19T20:56:50.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:50.655+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:50.657440+0000) 2024-02-19T20:56:51.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:51.655+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:51.657608+0000) 2024-02-19T20:56:52.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:52.651+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:52.657833+0000) 2024-02-19T20:56:53.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:53.655+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:53.658064+0000) 2024-02-19T20:56:54.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:54.655+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:54.658234+0000) 2024-02-19T20:56:55.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:55.651+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:55.658423+0000) 2024-02-19T20:56:56.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:56.651+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:56.658604+0000) 2024-02-19T20:56:57.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:57.655+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:57.658826+0000) 2024-02-19T20:56:58.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:58.655+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:58.659020+0000) 2024-02-19T20:56:59.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:56:59.655+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:56:59.659226+0000) 2024-02-19T20:57:00.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:00.655+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:00.659475+0000) 2024-02-19T20:57:01.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:01.655+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:01.659706+0000) 2024-02-19T20:57:02.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:02.659+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:02.659971+0000) 2024-02-19T20:57:03.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:03.659+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:03.660186+0000) 2024-02-19T20:57:04.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:04.659+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:04.660433+0000) 2024-02-19T20:57:05.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:05.655+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:05.660660+0000) 2024-02-19T20:57:06.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:06.659+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:06.660918+0000) 2024-02-19T20:57:07.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:07.659+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:07.661134+0000) 2024-02-19T20:57:08.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:08.659+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:08.661334+0000) 2024-02-19T20:57:09.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:09.659+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:09.661569+0000) 2024-02-19T20:57:10.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:10.655+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:10.661820+0000) 2024-02-19T20:57:11.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:11.655+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:11.662075+0000) 2024-02-19T20:57:12.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:12.659+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:12.662320+0000) 2024-02-19T20:57:13.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:13.655+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:13.662570+0000) 2024-02-19T20:57:14.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:14.659+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:14.662856+0000) 2024-02-19T20:57:15.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:15.659+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:15.663043+0000) 2024-02-19T20:57:15.845 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:57:15.848 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:57:16.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:16.659+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:16.663316+0000) 2024-02-19T20:57:17.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:17.659+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:17.663598+0000) 2024-02-19T20:57:18.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:18.663+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:18.663835+0000) 2024-02-19T20:57:19.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:19.663+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:19.664090+0000) 2024-02-19T20:57:20.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:20.663+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:20.664351+0000) 2024-02-19T20:57:21.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:21.663+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:21.664639+0000) 2024-02-19T20:57:22.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:22.663+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:22.664888+0000) 2024-02-19T20:57:23.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:23.663+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:23.665072+0000) 2024-02-19T20:57:24.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:24.663+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:24.665270+0000) 2024-02-19T20:57:25.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:25.659+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:25.665433+0000) 2024-02-19T20:57:26.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:26.659+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:26.665695+0000) 2024-02-19T20:57:27.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:27.659+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:27.665954+0000) 2024-02-19T20:57:28.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:28.659+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:28.666206+0000) 2024-02-19T20:57:29.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:29.663+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:29.666456+0000) 2024-02-19T20:57:30.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:30.663+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:30.666766+0000) 2024-02-19T20:57:31.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:31.663+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:31.667010+0000) 2024-02-19T20:57:32.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:32.663+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:32.667124+0000) 2024-02-19T20:57:33.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:33.663+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:33.667343+0000) 2024-02-19T20:57:34.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:34.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:34.667609+0000) 2024-02-19T20:57:35.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:35.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:35.667766+0000) 2024-02-19T20:57:36.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:36.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:36.668035+0000) 2024-02-19T20:57:37.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:37.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:37.668281+0000) 2024-02-19T20:57:38.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:38.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:38.668526+0000) 2024-02-19T20:57:39.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:39.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:39.668717+0000) 2024-02-19T20:57:40.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:40.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:40.668967+0000) 2024-02-19T20:57:41.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:41.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:41.669230+0000) 2024-02-19T20:57:42.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:42.663+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:42.669390+0000) 2024-02-19T20:57:43.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:43.663+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:43.669513+0000) 2024-02-19T20:57:44.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:44.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:44.669779+0000) 2024-02-19T20:57:45.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:45.663+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:45.669973+0000) 2024-02-19T20:57:45.996 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:57:46.000 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:57:46.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:46.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:46.670203+0000) 2024-02-19T20:57:47.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:47.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:47.670406+0000) 2024-02-19T20:57:48.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:48.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:48.670640+0000) 2024-02-19T20:57:49.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:49.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:49.670881+0000) 2024-02-19T20:57:50.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:50.670+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:50.671116+0000) 2024-02-19T20:57:51.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:51.666+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:51.671289+0000) 2024-02-19T20:57:52.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:52.670+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:52.671538+0000) 2024-02-19T20:57:53.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:53.666+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:53.671769+0000) 2024-02-19T20:57:54.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:54.670+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:54.672040+0000) 2024-02-19T20:57:55.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:55.670+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:55.672286+0000) 2024-02-19T20:57:56.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:56.670+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:56.672522+0000) 2024-02-19T20:57:57.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:57.666+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:57.672695+0000) 2024-02-19T20:57:58.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:58.666+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:58.672934+0000) 2024-02-19T20:57:59.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:57:59.666+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:57:59.673172+0000) 2024-02-19T20:58:00.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:00.670+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:00.673406+0000) 2024-02-19T20:58:01.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:01.670+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:01.673652+0000) 2024-02-19T20:58:02.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:02.670+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:02.673887+0000) 2024-02-19T20:58:03.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:03.670+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:03.674168+0000) 2024-02-19T20:58:04.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:04.670+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:04.674420+0000) 2024-02-19T20:58:05.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:05.670+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:05.674614+0000) 2024-02-19T20:58:06.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:06.674+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:06.674857+0000) 2024-02-19T20:58:07.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:07.674+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:07.675105+0000) 2024-02-19T20:58:08.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:08.674+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:08.675384+0000) 2024-02-19T20:58:09.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:09.674+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:09.675585+0000) 2024-02-19T20:58:10.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:10.674+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:10.675866+0000) 2024-02-19T20:58:11.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:11.674+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:11.676138+0000) 2024-02-19T20:58:12.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:12.674+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:12.676314+0000) 2024-02-19T20:58:13.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:13.674+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:13.676555+0000) 2024-02-19T20:58:14.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:14.674+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:14.676815+0000) 2024-02-19T20:58:15.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:15.674+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:15.677051+0000) 2024-02-19T20:58:16.052 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:58:16.055 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:58:16.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:16.674+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:16.677229+0000) 2024-02-19T20:58:17.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:17.674+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:17.677410+0000) 2024-02-19T20:58:18.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:18.674+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:18.677688+0000) 2024-02-19T20:58:19.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:19.674+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:19.677932+0000) 2024-02-19T20:58:20.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:20.674+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:20.678103+0000) 2024-02-19T20:58:21.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:21.674+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:21.678349+0000) 2024-02-19T20:58:22.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:22.674+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:22.678560+0000) 2024-02-19T20:58:23.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:23.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:23.678779+0000) 2024-02-19T20:58:24.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:24.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:24.678969+0000) 2024-02-19T20:58:25.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:25.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:25.679193+0000) 2024-02-19T20:58:26.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:26.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:26.679379+0000) 2024-02-19T20:58:27.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:27.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:27.679582+0000) 2024-02-19T20:58:28.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:28.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:28.679779+0000) 2024-02-19T20:58:29.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:29.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:29.680029+0000) 2024-02-19T20:58:30.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:30.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:30.680200+0000) 2024-02-19T20:58:31.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:31.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:31.680438+0000) 2024-02-19T20:58:32.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:32.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:32.680613+0000) 2024-02-19T20:58:33.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:33.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:33.680862+0000) 2024-02-19T20:58:34.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:34.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:34.681114+0000) 2024-02-19T20:58:35.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:35.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:35.681356+0000) 2024-02-19T20:58:36.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:36.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:36.681599+0000) 2024-02-19T20:58:37.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:37.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:37.681851+0000) 2024-02-19T20:58:38.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:38.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:38.682095+0000) 2024-02-19T20:58:39.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:39.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:39.682344+0000) 2024-02-19T20:58:40.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:40.682+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:40.682550+0000) 2024-02-19T20:58:41.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:41.682+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:41.682853+0000) 2024-02-19T20:58:42.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:42.682+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:42.683100+0000) 2024-02-19T20:58:43.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:43.682+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:43.683417+0000) 2024-02-19T20:58:44.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:44.682+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:44.683673+0000) 2024-02-19T20:58:45.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:45.682+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:45.683899+0000) 2024-02-19T20:58:46.123 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:58:46.127 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:58:46.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:46.682+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:46.684145+0000) 2024-02-19T20:58:47.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:47.682+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:47.684390+0000) 2024-02-19T20:58:48.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:48.682+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:48.684599+0000) 2024-02-19T20:58:49.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:49.682+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:49.684840+0000) 2024-02-19T20:58:50.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:50.682+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:50.685024+0000) 2024-02-19T20:58:51.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:51.681+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:51.685322+0000) 2024-02-19T20:58:52.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:52.681+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:52.685529+0000) 2024-02-19T20:58:53.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:53.681+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:53.685691+0000) 2024-02-19T20:58:54.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:54.681+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:54.685876+0000) 2024-02-19T20:58:55.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:55.681+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:55.686100+0000) 2024-02-19T20:58:56.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:56.685+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:56.686346+0000) 2024-02-19T20:58:57.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:57.681+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:57.686586+0000) 2024-02-19T20:58:58.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:58.685+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:58.686834+0000) 2024-02-19T20:58:59.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:58:59.685+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:58:59.687102+0000) 2024-02-19T20:59:00.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:00.685+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:00.687356+0000) 2024-02-19T20:59:01.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:01.685+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:01.687597+0000) 2024-02-19T20:59:02.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:02.685+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:02.687780+0000) 2024-02-19T20:59:03.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:03.685+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:03.687935+0000) 2024-02-19T20:59:04.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:04.685+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:04.688179+0000) 2024-02-19T20:59:05.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:05.685+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:05.688415+0000) 2024-02-19T20:59:06.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:06.685+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:06.688692+0000) 2024-02-19T20:59:07.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:07.685+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:07.688889+0000) 2024-02-19T20:59:08.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:08.685+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:08.689136+0000) 2024-02-19T20:59:09.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:09.685+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:09.689347+0000) 2024-02-19T20:59:10.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:10.685+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:10.689594+0000) 2024-02-19T20:59:11.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:11.685+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:11.689851+0000) 2024-02-19T20:59:12.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:12.685+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:12.690086+0000) 2024-02-19T20:59:13.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:13.685+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:13.690307+0000) 2024-02-19T20:59:14.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:14.689+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:14.690510+0000) 2024-02-19T20:59:15.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:15.689+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:15.690744+0000) 2024-02-19T20:59:16.183 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:59:16.187 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:59:16.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:16.689+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:16.691009+0000) 2024-02-19T20:59:17.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:17.689+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:17.691208+0000) 2024-02-19T20:59:18.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:18.689+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:18.691432+0000) 2024-02-19T20:59:19.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:19.689+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:19.691687+0000) 2024-02-19T20:59:20.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:20.689+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:20.691925+0000) 2024-02-19T20:59:21.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:21.689+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:21.692181+0000) 2024-02-19T20:59:22.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:22.689+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:22.692425+0000) 2024-02-19T20:59:23.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:23.689+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:23.692663+0000) 2024-02-19T20:59:24.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:24.689+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:24.692918+0000) 2024-02-19T20:59:25.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:25.689+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:25.693159+0000) 2024-02-19T20:59:26.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:26.689+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:26.693432+0000) 2024-02-19T20:59:27.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:27.689+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:27.693649+0000) 2024-02-19T20:59:28.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:28.693+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:28.693874+0000) 2024-02-19T20:59:29.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:29.693+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:29.694087+0000) 2024-02-19T20:59:30.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:30.689+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:30.694326+0000) 2024-02-19T20:59:31.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:31.689+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:31.694583+0000) 2024-02-19T20:59:32.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:32.693+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:32.694874+0000) 2024-02-19T20:59:33.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:33.693+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:33.695064+0000) 2024-02-19T20:59:34.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:34.693+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:34.695205+0000) 2024-02-19T20:59:35.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:35.693+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:35.695464+0000) 2024-02-19T20:59:36.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:36.693+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:36.695690+0000) 2024-02-19T20:59:37.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:37.693+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:37.695861+0000) 2024-02-19T20:59:38.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:38.693+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:38.696041+0000) 2024-02-19T20:59:39.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:39.693+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:39.696246+0000) 2024-02-19T20:59:40.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:40.693+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:40.696425+0000) 2024-02-19T20:59:41.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:41.693+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:41.696669+0000) 2024-02-19T20:59:42.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:42.693+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:42.696926+0000) 2024-02-19T20:59:43.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:43.693+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:43.697152+0000) 2024-02-19T20:59:44.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:44.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:44.697397+0000) 2024-02-19T20:59:45.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:45.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:45.697636+0000) 2024-02-19T20:59:46.304 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:59:46.307 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T20:59:46.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:46.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:46.697921+0000) 2024-02-19T20:59:47.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:47.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:47.698146+0000) 2024-02-19T20:59:48.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:48.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:48.698396+0000) 2024-02-19T20:59:49.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:49.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:49.698574+0000) 2024-02-19T20:59:50.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:50.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:50.698833+0000) 2024-02-19T20:59:51.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:51.696+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:51.699083+0000) 2024-02-19T20:59:52.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:52.696+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:52.699257+0000) 2024-02-19T20:59:53.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:53.696+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:53.699500+0000) 2024-02-19T20:59:54.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:54.696+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:54.699723+0000) 2024-02-19T20:59:55.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:55.696+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:55.699960+0000) 2024-02-19T20:59:56.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:56.696+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:56.700201+0000) 2024-02-19T20:59:57.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:57.696+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:57.700347+0000) 2024-02-19T20:59:58.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:58.696+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:58.700604+0000) 2024-02-19T20:59:59.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T20:59:59.696+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:59:59.700770+0000) 2024-02-19T21:00:00.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:00.700+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:00.700996+0000) 2024-02-19T21:00:01.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:01.700+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:01.701246+0000) 2024-02-19T21:00:02.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:02.700+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:02.701424+0000) 2024-02-19T21:00:03.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:03.700+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:03.701678+0000) 2024-02-19T21:00:04.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:04.696+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:04.701909+0000) 2024-02-19T21:00:05.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:05.696+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:05.702170+0000) 2024-02-19T21:00:06.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:06.700+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:06.702392+0000) 2024-02-19T21:00:07.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:07.700+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:07.702659+0000) 2024-02-19T21:00:08.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:08.700+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:08.702908+0000) 2024-02-19T21:00:09.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:09.700+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:09.703081+0000) 2024-02-19T21:00:10.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:10.700+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:10.703290+0000) 2024-02-19T21:00:11.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:11.700+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:11.703527+0000) 2024-02-19T21:00:12.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:12.700+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:12.703774+0000) 2024-02-19T21:00:13.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:13.700+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:13.703967+0000) 2024-02-19T21:00:14.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:14.700+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:14.704223+0000) 2024-02-19T21:00:15.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:15.700+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:15.704472+0000) 2024-02-19T21:00:16.452 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:00:16.455 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:00:16.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:16.700+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:16.704723+0000) 2024-02-19T21:00:17.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:17.704+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:17.704968+0000) 2024-02-19T21:00:18.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:18.704+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:18.705149+0000) 2024-02-19T21:00:19.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:19.704+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:19.705399+0000) 2024-02-19T21:00:20.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:20.700+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:20.705644+0000) 2024-02-19T21:00:21.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:21.700+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:21.705881+0000) 2024-02-19T21:00:22.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:22.704+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:22.706097+0000) 2024-02-19T21:00:23.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:23.700+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:23.706329+0000) 2024-02-19T21:00:24.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:24.704+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:24.706499+0000) 2024-02-19T21:00:25.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:25.704+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:25.706700+0000) 2024-02-19T21:00:26.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:26.704+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:26.706892+0000) 2024-02-19T21:00:27.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:27.704+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:27.707091+0000) 2024-02-19T21:00:28.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:28.704+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:28.707369+0000) 2024-02-19T21:00:29.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:29.704+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:29.707626+0000) 2024-02-19T21:00:30.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:30.704+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:30.707898+0000) 2024-02-19T21:00:31.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:31.704+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:31.708107+0000) 2024-02-19T21:00:32.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:32.704+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:32.708326+0000) 2024-02-19T21:00:33.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:33.704+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:33.708520+0000) 2024-02-19T21:00:34.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:34.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:34.708765+0000) 2024-02-19T21:00:35.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:35.704+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:35.708940+0000) 2024-02-19T21:00:36.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:36.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:36.709194+0000) 2024-02-19T21:00:37.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:37.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:37.709462+0000) 2024-02-19T21:00:38.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:38.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:38.709719+0000) 2024-02-19T21:00:39.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:39.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:39.709918+0000) 2024-02-19T21:00:40.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:40.704+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:40.710169+0000) 2024-02-19T21:00:41.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:41.704+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:41.710366+0000) 2024-02-19T21:00:42.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:42.704+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:42.710615+0000) 2024-02-19T21:00:43.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:43.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:43.710800+0000) 2024-02-19T21:00:44.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:44.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:44.711048+0000) 2024-02-19T21:00:45.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:45.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:45.711189+0000) 2024-02-19T21:00:46.500 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:00:46.503 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:00:46.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:46.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:46.711410+0000) 2024-02-19T21:00:47.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:47.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:47.711621+0000) 2024-02-19T21:00:48.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:48.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:48.711903+0000) 2024-02-19T21:00:49.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:49.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:49.712143+0000) 2024-02-19T21:00:50.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:50.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:50.712319+0000) 2024-02-19T21:00:51.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:51.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:51.712519+0000) 2024-02-19T21:00:52.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:52.707+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:52.712769+0000) 2024-02-19T21:00:53.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:53.707+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:53.713004+0000) 2024-02-19T21:00:54.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:54.707+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:54.713171+0000) 2024-02-19T21:00:55.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:55.707+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:55.713445+0000) 2024-02-19T21:00:56.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:56.707+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:56.713675+0000) 2024-02-19T21:00:57.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:57.707+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:57.713916+0000) 2024-02-19T21:00:58.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:58.707+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:58.714153+0000) 2024-02-19T21:00:59.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:00:59.711+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:00:59.714387+0000) 2024-02-19T21:01:00.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:00.711+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:00.714562+0000) 2024-02-19T21:01:01.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:01.711+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:01.714855+0000) 2024-02-19T21:01:02.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:02.711+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:02.715045+0000) 2024-02-19T21:01:03.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:03.711+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:03.715220+0000) 2024-02-19T21:01:04.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:04.711+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:04.715488+0000) 2024-02-19T21:01:05.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:05.711+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:05.715689+0000) 2024-02-19T21:01:06.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:06.711+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:06.715937+0000) 2024-02-19T21:01:07.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:07.711+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:07.716153+0000) 2024-02-19T21:01:08.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:08.711+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:08.716394+0000) 2024-02-19T21:01:09.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:09.711+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:09.716628+0000) 2024-02-19T21:01:10.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:10.711+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:10.716889+0000) 2024-02-19T21:01:11.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:11.715+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:11.717145+0000) 2024-02-19T21:01:12.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:12.711+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:12.717321+0000) 2024-02-19T21:01:13.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:13.711+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:13.717557+0000) 2024-02-19T21:01:14.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:14.715+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:14.717707+0000) 2024-02-19T21:01:15.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:15.711+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:15.717951+0000) 2024-02-19T21:01:16.631 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:01:16.635 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:01:16.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:16.715+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:16.718135+0000) 2024-02-19T21:01:17.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:17.715+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:17.718412+0000) 2024-02-19T21:01:18.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:18.711+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:18.718654+0000) 2024-02-19T21:01:19.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:19.715+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:19.718898+0000) 2024-02-19T21:01:20.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:20.715+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:20.719188+0000) 2024-02-19T21:01:21.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:21.715+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:21.719459+0000) 2024-02-19T21:01:22.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:22.719+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:22.719625+0000) 2024-02-19T21:01:23.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:23.719+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:23.719861+0000) 2024-02-19T21:01:24.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:24.719+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:24.720122+0000) 2024-02-19T21:01:25.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:25.719+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:25.720367+0000) 2024-02-19T21:01:26.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:26.715+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:26.720650+0000) 2024-02-19T21:01:27.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:27.719+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:27.720893+0000) 2024-02-19T21:01:28.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:28.719+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:28.721113+0000) 2024-02-19T21:01:29.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:29.719+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:29.721374+0000) 2024-02-19T21:01:30.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:30.719+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:30.721597+0000) 2024-02-19T21:01:31.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:31.715+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:31.721777+0000) 2024-02-19T21:01:32.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:32.719+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:32.722045+0000) 2024-02-19T21:01:33.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:33.719+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:33.722292+0000) 2024-02-19T21:01:34.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:34.715+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:34.722527+0000) 2024-02-19T21:01:35.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:35.719+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:35.722738+0000) 2024-02-19T21:01:36.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:36.719+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:36.722905+0000) 2024-02-19T21:01:37.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:37.719+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:37.723150+0000) 2024-02-19T21:01:38.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:38.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:38.723344+0000) 2024-02-19T21:01:39.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:39.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:39.723551+0000) 2024-02-19T21:01:40.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:40.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:40.723800+0000) 2024-02-19T21:01:41.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:41.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:41.723973+0000) 2024-02-19T21:01:42.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:42.719+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:42.724225+0000) 2024-02-19T21:01:43.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:43.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:43.724420+0000) 2024-02-19T21:01:44.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:44.719+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:44.724608+0000) 2024-02-19T21:01:45.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:45.719+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:45.724772+0000) 2024-02-19T21:01:46.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:46.719+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:46.724967+0000) 2024-02-19T21:01:46.754 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:01:46.757 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:01:47.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:47.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:47.725162+0000) 2024-02-19T21:01:48.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:48.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:48.725409+0000) 2024-02-19T21:01:49.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:49.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:49.725648+0000) 2024-02-19T21:01:50.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:50.719+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:50.725867+0000) 2024-02-19T21:01:51.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:51.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:51.726116+0000) 2024-02-19T21:01:52.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:52.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:52.726391+0000) 2024-02-19T21:01:53.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:53.722+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:53.726627+0000) 2024-02-19T21:01:54.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:54.722+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:54.726801+0000) 2024-02-19T21:01:55.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:55.722+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:55.727000+0000) 2024-02-19T21:01:56.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:56.726+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:56.727247+0000) 2024-02-19T21:01:57.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:57.726+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:57.727521+0000) 2024-02-19T21:01:58.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:58.722+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:58.727756+0000) 2024-02-19T21:01:59.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:01:59.722+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:01:59.727994+0000) 2024-02-19T21:02:00.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:00.726+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:00.728174+0000) 2024-02-19T21:02:01.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:01.722+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:01.728355+0000) 2024-02-19T21:02:02.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:02.722+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:02.728536+0000) 2024-02-19T21:02:03.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:03.722+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:03.728783+0000) 2024-02-19T21:02:04.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:04.726+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:04.728975+0000) 2024-02-19T21:02:05.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:05.726+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:05.729118+0000) 2024-02-19T21:02:06.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:06.726+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:06.729364+0000) 2024-02-19T21:02:07.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:07.726+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:07.729601+0000) 2024-02-19T21:02:08.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:08.726+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:08.729875+0000) 2024-02-19T21:02:09.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:09.726+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:09.730081+0000) 2024-02-19T21:02:10.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:10.726+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:10.730330+0000) 2024-02-19T21:02:11.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:11.726+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:11.730592+0000) 2024-02-19T21:02:12.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:12.730+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:12.730832+0000) 2024-02-19T21:02:13.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:13.730+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:13.731058+0000) 2024-02-19T21:02:14.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:14.730+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:14.731288+0000) 2024-02-19T21:02:15.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:15.730+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:15.731477+0000) 2024-02-19T21:02:16.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:16.730+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:16.731710+0000) 2024-02-19T21:02:16.852 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:02:16.855 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:02:17.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:17.730+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:17.732010+0000) 2024-02-19T21:02:18.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:18.730+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:18.732246+0000) 2024-02-19T21:02:19.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:19.730+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:19.732593+0000) 2024-02-19T21:02:20.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:20.730+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:20.732882+0000) 2024-02-19T21:02:21.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:21.730+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:21.733164+0000) 2024-02-19T21:02:22.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:22.730+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:22.733388+0000) 2024-02-19T21:02:23.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:23.730+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:23.733582+0000) 2024-02-19T21:02:24.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:24.730+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:24.733870+0000) 2024-02-19T21:02:25.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:25.730+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:25.734146+0000) 2024-02-19T21:02:26.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:26.730+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:26.734409+0000) 2024-02-19T21:02:27.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:27.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:27.734630+0000) 2024-02-19T21:02:28.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:28.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:28.734912+0000) 2024-02-19T21:02:29.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:29.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:29.735209+0000) 2024-02-19T21:02:30.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:30.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:30.735482+0000) 2024-02-19T21:02:31.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:31.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:31.735753+0000) 2024-02-19T21:02:32.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:32.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:32.736039+0000) 2024-02-19T21:02:33.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:33.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:33.736306+0000) 2024-02-19T21:02:34.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:34.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:34.736506+0000) 2024-02-19T21:02:35.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:35.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:35.736779+0000) 2024-02-19T21:02:36.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:36.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:36.736986+0000) 2024-02-19T21:02:37.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:37.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:37.737181+0000) 2024-02-19T21:02:38.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:38.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:38.737448+0000) 2024-02-19T21:02:39.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:39.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:39.737734+0000) 2024-02-19T21:02:40.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:40.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:40.737983+0000) 2024-02-19T21:02:41.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:41.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:41.738250+0000) 2024-02-19T21:02:42.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:42.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:42.738408+0000) 2024-02-19T21:02:43.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:43.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:43.738684+0000) 2024-02-19T21:02:44.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:44.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:44.738936+0000) 2024-02-19T21:02:45.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:45.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:45.739218+0000) 2024-02-19T21:02:46.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:46.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:46.739483+0000) 2024-02-19T21:02:47.001 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:02:47.004 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:02:47.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:47.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:47.739702+0000) 2024-02-19T21:02:48.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:48.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:48.739924+0000) 2024-02-19T21:02:49.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:49.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:49.740150+0000) 2024-02-19T21:02:50.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:50.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:50.740372+0000) 2024-02-19T21:02:51.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:51.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:51.740608+0000) 2024-02-19T21:02:52.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:52.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:52.740860+0000) 2024-02-19T21:02:53.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:53.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:53.741087+0000) 2024-02-19T21:02:54.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:54.737+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:54.741377+0000) 2024-02-19T21:02:55.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:55.737+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:55.741670+0000) 2024-02-19T21:02:56.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:56.737+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:56.741874+0000) 2024-02-19T21:02:57.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:57.737+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:57.742127+0000) 2024-02-19T21:02:58.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:58.737+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:58.742358+0000) 2024-02-19T21:02:59.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:02:59.741+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:02:59.742594+0000) 2024-02-19T21:03:00.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:00.741+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:00.742863+0000) 2024-02-19T21:03:01.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:01.741+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:01.743137+0000) 2024-02-19T21:03:02.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:02.741+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:02.743401+0000) 2024-02-19T21:03:03.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:03.741+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:03.743698+0000) 2024-02-19T21:03:04.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:04.741+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:04.743890+0000) 2024-02-19T21:03:05.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:05.741+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:05.744086+0000) 2024-02-19T21:03:06.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:06.741+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:06.744370+0000) 2024-02-19T21:03:07.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:07.741+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:07.744641+0000) 2024-02-19T21:03:08.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:08.741+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:08.744895+0000) 2024-02-19T21:03:09.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:09.741+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:09.745171+0000) 2024-02-19T21:03:10.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:10.741+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:10.745475+0000) 2024-02-19T21:03:11.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:11.741+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:11.745731+0000) 2024-02-19T21:03:12.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:12.745+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:12.745997+0000) 2024-02-19T21:03:13.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:13.745+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:13.746275+0000) 2024-02-19T21:03:14.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:14.745+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:14.746475+0000) 2024-02-19T21:03:15.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:15.741+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:15.746686+0000) 2024-02-19T21:03:16.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:16.745+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:16.747003+0000) 2024-02-19T21:03:17.066 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:03:17.070 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:03:17.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:17.745+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:17.747274+0000) 2024-02-19T21:03:18.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:18.745+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:18.747568+0000) 2024-02-19T21:03:19.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:19.745+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:19.747799+0000) 2024-02-19T21:03:20.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:20.745+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:20.748007+0000) 2024-02-19T21:03:21.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:21.745+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:21.748295+0000) 2024-02-19T21:03:22.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:22.745+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:22.748493+0000) 2024-02-19T21:03:23.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:23.745+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:23.748774+0000) 2024-02-19T21:03:24.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:24.745+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:24.749056+0000) 2024-02-19T21:03:25.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:25.745+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:25.749263+0000) 2024-02-19T21:03:26.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:26.745+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:26.749548+0000) 2024-02-19T21:03:27.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:27.745+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:27.749816+0000) 2024-02-19T21:03:28.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:28.745+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:28.750021+0000) 2024-02-19T21:03:29.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:29.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:29.750225+0000) 2024-02-19T21:03:30.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:30.745+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:30.750405+0000) 2024-02-19T21:03:31.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:31.745+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:31.750648+0000) 2024-02-19T21:03:32.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:32.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:32.750850+0000) 2024-02-19T21:03:33.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:33.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:33.751064+0000) 2024-02-19T21:03:34.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:34.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:34.751294+0000) 2024-02-19T21:03:35.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:35.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:35.751521+0000) 2024-02-19T21:03:36.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:36.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:36.751820+0000) 2024-02-19T21:03:37.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:37.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:37.752020+0000) 2024-02-19T21:03:38.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:38.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:38.752281+0000) 2024-02-19T21:03:39.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:39.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:39.752530+0000) 2024-02-19T21:03:40.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:40.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:40.752830+0000) 2024-02-19T21:03:41.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:41.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:41.753151+0000) 2024-02-19T21:03:42.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:42.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:42.753466+0000) 2024-02-19T21:03:43.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:43.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:43.753713+0000) 2024-02-19T21:03:44.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:44.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:44.753971+0000) 2024-02-19T21:03:45.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:45.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:45.754232+0000) 2024-02-19T21:03:46.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:46.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:46.754418+0000) 2024-02-19T21:03:47.146 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:03:47.149 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:03:47.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:47.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:47.754640+0000) 2024-02-19T21:03:48.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:48.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:48.754915+0000) 2024-02-19T21:03:49.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:49.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:49.755192+0000) 2024-02-19T21:03:50.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:50.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:50.755505+0000) 2024-02-19T21:03:51.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:51.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:51.755685+0000) 2024-02-19T21:03:52.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:52.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:52.755970+0000) 2024-02-19T21:03:53.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:53.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:53.756250+0000) 2024-02-19T21:03:54.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:54.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:54.756451+0000) 2024-02-19T21:03:55.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:55.752+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:55.756685+0000) 2024-02-19T21:03:56.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:56.752+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:56.756958+0000) 2024-02-19T21:03:57.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:57.756+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:57.757171+0000) 2024-02-19T21:03:58.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:58.756+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:58.757356+0000) 2024-02-19T21:03:59.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:03:59.752+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:03:59.757609+0000) 2024-02-19T21:04:00.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:00.756+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:00.757848+0000) 2024-02-19T21:04:01.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:01.756+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:01.758120+0000) 2024-02-19T21:04:02.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:02.756+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:02.758328+0000) 2024-02-19T21:04:03.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:03.752+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:03.758552+0000) 2024-02-19T21:04:04.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:04.756+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:04.758855+0000) 2024-02-19T21:04:05.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:05.756+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:05.759156+0000) 2024-02-19T21:04:06.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:06.756+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:06.759377+0000) 2024-02-19T21:04:07.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:07.756+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:07.759638+0000) 2024-02-19T21:04:08.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:08.756+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:08.759956+0000) 2024-02-19T21:04:09.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:09.756+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:09.760212+0000) 2024-02-19T21:04:10.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:10.756+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:10.760418+0000) 2024-02-19T21:04:11.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:11.756+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:11.760686+0000) 2024-02-19T21:04:12.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:12.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:12.760971+0000) 2024-02-19T21:04:13.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:13.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:13.761250+0000) 2024-02-19T21:04:14.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:14.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:14.761531+0000) 2024-02-19T21:04:15.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:15.756+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:15.761765+0000) 2024-02-19T21:04:16.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:16.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:16.762042+0000) 2024-02-19T21:04:17.295 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:04:17.298 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:04:17.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:17.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:17.762363+0000) 2024-02-19T21:04:18.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:18.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:18.762601+0000) 2024-02-19T21:04:19.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:19.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:19.762859+0000) 2024-02-19T21:04:20.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:20.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:20.763078+0000) 2024-02-19T21:04:21.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:21.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:21.763346+0000) 2024-02-19T21:04:22.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:22.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:22.763585+0000) 2024-02-19T21:04:23.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:23.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:23.763756+0000) 2024-02-19T21:04:24.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:24.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:24.763956+0000) 2024-02-19T21:04:25.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:25.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:25.764254+0000) 2024-02-19T21:04:26.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:26.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:26.764391+0000) 2024-02-19T21:04:27.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:27.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:27.764685+0000) 2024-02-19T21:04:28.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:28.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:28.765028+0000) 2024-02-19T21:04:29.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:29.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:29.765284+0000) 2024-02-19T21:04:30.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:30.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:30.765619+0000) 2024-02-19T21:04:31.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:31.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:31.765825+0000) 2024-02-19T21:04:32.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:32.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:32.766056+0000) 2024-02-19T21:04:33.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:33.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:33.766317+0000) 2024-02-19T21:04:34.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:34.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:34.766511+0000) 2024-02-19T21:04:35.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:35.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:35.766741+0000) 2024-02-19T21:04:36.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:36.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:36.767022+0000) 2024-02-19T21:04:37.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:37.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:37.767265+0000) 2024-02-19T21:04:38.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:38.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:38.767514+0000) 2024-02-19T21:04:39.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:39.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:39.767744+0000) 2024-02-19T21:04:40.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:40.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:40.767929+0000) 2024-02-19T21:04:41.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:41.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:41.768150+0000) 2024-02-19T21:04:42.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:42.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:42.768394+0000) 2024-02-19T21:04:43.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:43.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:43.768571+0000) 2024-02-19T21:04:44.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:44.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:44.768821+0000) 2024-02-19T21:04:45.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:45.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:45.769118+0000) 2024-02-19T21:04:46.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:46.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:46.769397+0000) 2024-02-19T21:04:47.425 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:04:47.428 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:04:47.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:47.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:47.769702+0000) 2024-02-19T21:04:48.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:48.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:48.770021+0000) 2024-02-19T21:04:49.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:49.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:49.770311+0000) 2024-02-19T21:04:50.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:50.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:50.770545+0000) 2024-02-19T21:04:51.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:51.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:51.770842+0000) 2024-02-19T21:04:52.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:52.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:52.771095+0000) 2024-02-19T21:04:53.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:53.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:53.771340+0000) 2024-02-19T21:04:54.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:54.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:54.771620+0000) 2024-02-19T21:04:55.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:55.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:55.771841+0000) 2024-02-19T21:04:56.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:56.767+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:56.772109+0000) 2024-02-19T21:04:57.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:57.767+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:57.772392+0000) 2024-02-19T21:04:58.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:58.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:58.772664+0000) 2024-02-19T21:04:59.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:04:59.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:04:59.772947+0000) 2024-02-19T21:05:00.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:00.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:00.773228+0000) 2024-02-19T21:05:01.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:01.767+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:01.773523+0000) 2024-02-19T21:05:02.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:02.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:02.773777+0000) 2024-02-19T21:05:03.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:03.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:03.774064+0000) 2024-02-19T21:05:04.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:04.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:04.774345+0000) 2024-02-19T21:05:05.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:05.767+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:05.774620+0000) 2024-02-19T21:05:06.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:06.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:06.774884+0000) 2024-02-19T21:05:07.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:07.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:07.775150+0000) 2024-02-19T21:05:08.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:08.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:08.775362+0000) 2024-02-19T21:05:09.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:09.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:09.775560+0000) 2024-02-19T21:05:10.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:10.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:10.775794+0000) 2024-02-19T21:05:11.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:11.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:11.775971+0000) 2024-02-19T21:05:12.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:12.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:12.776232+0000) 2024-02-19T21:05:13.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:13.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:13.776511+0000) 2024-02-19T21:05:14.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:14.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:14.776763+0000) 2024-02-19T21:05:15.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:15.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:15.777038+0000) 2024-02-19T21:05:16.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:16.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:16.777313+0000) 2024-02-19T21:05:17.665 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:05:17.669 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:05:17.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:17.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:17.777606+0000) 2024-02-19T21:05:18.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:18.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:18.777947+0000) 2024-02-19T21:05:19.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:19.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:19.778207+0000) 2024-02-19T21:05:20.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:20.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:20.778486+0000) 2024-02-19T21:05:21.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:21.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:21.778821+0000) 2024-02-19T21:05:22.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:22.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:22.779115+0000) 2024-02-19T21:05:23.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:23.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:23.779352+0000) 2024-02-19T21:05:24.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:24.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:24.779632+0000) 2024-02-19T21:05:25.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:25.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:25.779847+0000) 2024-02-19T21:05:26.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:26.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:26.779983+0000) 2024-02-19T21:05:27.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:27.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:27.780237+0000) 2024-02-19T21:05:28.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:28.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:28.780540+0000) 2024-02-19T21:05:29.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:29.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:29.780814+0000) 2024-02-19T21:05:30.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:30.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:30.780958+0000) 2024-02-19T21:05:31.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:31.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:31.781261+0000) 2024-02-19T21:05:32.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:32.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:32.781539+0000) 2024-02-19T21:05:33.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:33.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:33.781805+0000) 2024-02-19T21:05:34.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:34.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:34.782060+0000) 2024-02-19T21:05:35.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:35.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:35.782315+0000) 2024-02-19T21:05:36.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:36.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:36.782612+0000) 2024-02-19T21:05:37.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:37.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:37.782813+0000) 2024-02-19T21:05:38.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:38.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:38.783111+0000) 2024-02-19T21:05:39.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:39.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:39.783389+0000) 2024-02-19T21:05:40.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:40.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:40.783640+0000) 2024-02-19T21:05:41.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:41.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:41.783816+0000) 2024-02-19T21:05:42.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:42.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:42.783954+0000) 2024-02-19T21:05:43.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:43.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:43.784228+0000) 2024-02-19T21:05:44.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:44.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:44.784448+0000) 2024-02-19T21:05:45.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:45.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:45.784638+0000) 2024-02-19T21:05:46.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:46.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:46.784782+0000) 2024-02-19T21:05:47.721 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:05:47.725 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:05:47.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:47.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:47.784969+0000) 2024-02-19T21:05:48.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:48.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:48.785303+0000) 2024-02-19T21:05:49.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:49.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:49.785506+0000) 2024-02-19T21:05:50.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:50.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:50.785759+0000) 2024-02-19T21:05:51.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:51.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:51.786049+0000) 2024-02-19T21:05:52.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:52.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:52.786323+0000) 2024-02-19T21:05:53.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:53.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:53.786548+0000) 2024-02-19T21:05:54.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:54.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:54.786858+0000) 2024-02-19T21:05:55.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:55.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:55.787062+0000) 2024-02-19T21:05:56.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:56.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:56.787270+0000) 2024-02-19T21:05:57.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:57.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:57.787531+0000) 2024-02-19T21:05:58.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:58.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:58.787771+0000) 2024-02-19T21:05:59.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:05:59.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:05:59.787976+0000) 2024-02-19T21:06:00.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:00.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:00.788109+0000) 2024-02-19T21:06:01.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:01.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:01.788286+0000) 2024-02-19T21:06:02.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:02.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:02.788484+0000) 2024-02-19T21:06:03.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:03.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:03.788748+0000) 2024-02-19T21:06:04.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:04.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:04.788909+0000) 2024-02-19T21:06:05.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:05.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:05.789063+0000) 2024-02-19T21:06:06.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:06.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:06.789319+0000) 2024-02-19T21:06:07.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:07.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:07.789551+0000) 2024-02-19T21:06:08.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:08.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:08.789818+0000) 2024-02-19T21:06:09.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:09.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:09.790093+0000) 2024-02-19T21:06:10.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:10.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:10.790264+0000) 2024-02-19T21:06:11.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:11.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:11.790403+0000) 2024-02-19T21:06:12.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:12.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:12.790600+0000) 2024-02-19T21:06:13.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:13.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:13.790909+0000) 2024-02-19T21:06:14.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:14.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:14.791113+0000) 2024-02-19T21:06:15.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:15.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:15.791285+0000) 2024-02-19T21:06:16.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:16.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:16.791540+0000) 2024-02-19T21:06:17.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:17.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:17.791770+0000) 2024-02-19T21:06:17.810 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:06:17.813 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:06:18.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:18.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:18.792033+0000) 2024-02-19T21:06:19.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:19.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:19.792225+0000) 2024-02-19T21:06:20.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:20.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:20.792456+0000) 2024-02-19T21:06:21.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:21.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:21.792731+0000) 2024-02-19T21:06:22.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:22.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:22.792899+0000) 2024-02-19T21:06:23.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:23.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:23.793155+0000) 2024-02-19T21:06:24.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:24.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:24.793406+0000) 2024-02-19T21:06:25.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:25.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:25.793629+0000) 2024-02-19T21:06:26.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:26.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:26.793858+0000) 2024-02-19T21:06:27.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:27.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:27.794053+0000) 2024-02-19T21:06:28.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:28.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:28.794282+0000) 2024-02-19T21:06:29.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:29.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:29.794524+0000) 2024-02-19T21:06:30.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:30.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:30.794808+0000) 2024-02-19T21:06:31.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:31.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:31.794981+0000) 2024-02-19T21:06:32.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:32.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:32.795194+0000) 2024-02-19T21:06:33.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:33.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:33.795456+0000) 2024-02-19T21:06:34.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:34.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:34.795626+0000) 2024-02-19T21:06:35.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:35.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:35.795866+0000) 2024-02-19T21:06:36.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:36.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:36.796104+0000) 2024-02-19T21:06:37.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:37.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:37.796276+0000) 2024-02-19T21:06:38.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:38.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:38.796485+0000) 2024-02-19T21:06:39.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:39.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:39.796730+0000) 2024-02-19T21:06:40.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:40.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:40.796977+0000) 2024-02-19T21:06:41.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:41.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:41.797220+0000) 2024-02-19T21:06:42.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:42.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:42.797475+0000) 2024-02-19T21:06:43.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:43.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:43.797713+0000) 2024-02-19T21:06:44.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:44.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:44.797948+0000) 2024-02-19T21:06:45.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:45.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:45.798224+0000) 2024-02-19T21:06:46.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:46.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:46.798465+0000) 2024-02-19T21:06:47.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:47.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:47.798662+0000) 2024-02-19T21:06:47.892 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:06:47.895 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:06:48.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:48.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:48.798819+0000) 2024-02-19T21:06:49.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:49.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:49.798978+0000) 2024-02-19T21:06:50.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:50.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:50.799220+0000) 2024-02-19T21:06:51.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:51.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:51.799493+0000) 2024-02-19T21:06:52.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:52.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:52.799771+0000) 2024-02-19T21:06:53.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:53.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:53.799978+0000) 2024-02-19T21:06:54.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:54.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:54.800235+0000) 2024-02-19T21:06:55.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:55.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:55.800466+0000) 2024-02-19T21:06:56.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:56.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:56.800734+0000) 2024-02-19T21:06:57.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:57.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:57.800987+0000) 2024-02-19T21:06:58.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:58.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:58.801266+0000) 2024-02-19T21:06:59.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:06:59.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:06:59.801555+0000) 2024-02-19T21:07:00.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:00.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:00.801810+0000) 2024-02-19T21:07:01.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:01.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:01.802042+0000) 2024-02-19T21:07:02.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:02.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:02.802322+0000) 2024-02-19T21:07:03.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:03.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:03.802615+0000) 2024-02-19T21:07:04.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:04.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:04.802816+0000) 2024-02-19T21:07:05.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:05.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:05.803033+0000) 2024-02-19T21:07:06.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:06.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:06.803233+0000) 2024-02-19T21:07:07.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:07.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:07.803505+0000) 2024-02-19T21:07:08.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:08.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:08.803813+0000) 2024-02-19T21:07:09.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:09.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:09.804077+0000) 2024-02-19T21:07:10.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:10.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:10.804348+0000) 2024-02-19T21:07:11.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:11.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:11.804627+0000) 2024-02-19T21:07:12.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:12.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:12.804863+0000) 2024-02-19T21:07:13.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:13.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:13.805116+0000) 2024-02-19T21:07:14.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:14.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:14.805333+0000) 2024-02-19T21:07:15.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:15.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:15.805581+0000) 2024-02-19T21:07:16.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:16.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:16.805867+0000) 2024-02-19T21:07:17.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:17.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:17.806109+0000) 2024-02-19T21:07:18.014 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:07:18.018 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:07:18.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:18.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:18.806407+0000) 2024-02-19T21:07:19.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:19.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:19.806607+0000) 2024-02-19T21:07:20.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:20.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:20.806862+0000) 2024-02-19T21:07:21.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:21.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:21.807139+0000) 2024-02-19T21:07:22.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:22.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:22.807317+0000) 2024-02-19T21:07:23.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:23.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:23.807632+0000) 2024-02-19T21:07:24.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:24.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:24.807920+0000) 2024-02-19T21:07:25.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:25.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:25.808151+0000) 2024-02-19T21:07:26.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:26.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:26.808420+0000) 2024-02-19T21:07:27.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:27.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:27.808721+0000) 2024-02-19T21:07:28.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:28.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:28.809009+0000) 2024-02-19T21:07:29.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:29.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:29.809300+0000) 2024-02-19T21:07:30.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:30.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:30.809555+0000) 2024-02-19T21:07:31.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:31.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:31.809690+0000) 2024-02-19T21:07:32.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:32.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:32.809930+0000) 2024-02-19T21:07:33.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:33.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:33.810135+0000) 2024-02-19T21:07:34.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:34.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:34.810388+0000) 2024-02-19T21:07:35.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:35.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:35.810616+0000) 2024-02-19T21:07:36.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:36.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:36.810805+0000) 2024-02-19T21:07:37.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:37.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:37.810966+0000) 2024-02-19T21:07:38.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:38.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:38.811205+0000) 2024-02-19T21:07:39.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:39.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:39.811407+0000) 2024-02-19T21:07:40.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:40.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:40.811639+0000) 2024-02-19T21:07:41.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:41.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:41.811848+0000) 2024-02-19T21:07:42.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:42.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:42.812114+0000) 2024-02-19T21:07:43.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:43.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:43.812299+0000) 2024-02-19T21:07:44.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:44.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:44.812533+0000) 2024-02-19T21:07:45.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:45.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:45.812788+0000) 2024-02-19T21:07:46.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:46.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:46.813044+0000) 2024-02-19T21:07:47.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:47.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:47.813298+0000) 2024-02-19T21:07:48.087 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:07:48.091 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:07:48.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:48.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:48.813561+0000) 2024-02-19T21:07:49.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:49.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:49.813794+0000) 2024-02-19T21:07:50.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:50.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:50.814042+0000) 2024-02-19T21:07:51.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:51.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:51.814222+0000) 2024-02-19T21:07:52.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:52.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:52.814454+0000) 2024-02-19T21:07:53.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:53.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:53.814641+0000) 2024-02-19T21:07:54.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:54.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:54.814837+0000) 2024-02-19T21:07:55.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:55.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:55.815080+0000) 2024-02-19T21:07:56.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:56.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:56.815308+0000) 2024-02-19T21:07:57.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:57.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:57.815559+0000) 2024-02-19T21:07:58.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:58.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:58.815811+0000) 2024-02-19T21:07:59.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:07:59.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:07:59.816074+0000) 2024-02-19T21:08:00.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:00.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:00.816309+0000) 2024-02-19T21:08:01.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:01.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:01.816563+0000) 2024-02-19T21:08:02.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:02.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:02.816758+0000) 2024-02-19T21:08:03.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:03.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:03.816946+0000) 2024-02-19T21:08:04.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:04.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:04.817148+0000) 2024-02-19T21:08:05.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:05.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:05.817347+0000) 2024-02-19T21:08:06.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:06.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:06.817654+0000) 2024-02-19T21:08:07.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:07.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:07.817940+0000) 2024-02-19T21:08:08.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:08.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:08.818135+0000) 2024-02-19T21:08:09.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:09.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:09.818327+0000) 2024-02-19T21:08:10.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:10.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:10.818482+0000) 2024-02-19T21:08:11.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:11.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:11.818676+0000) 2024-02-19T21:08:12.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:12.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:12.818911+0000) 2024-02-19T21:08:13.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:13.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:13.819127+0000) 2024-02-19T21:08:14.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:14.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:14.819367+0000) 2024-02-19T21:08:15.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:15.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:15.819542+0000) 2024-02-19T21:08:16.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:16.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:16.819786+0000) 2024-02-19T21:08:17.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:17.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:17.819996+0000) 2024-02-19T21:08:18.234 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:08:18.238 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:08:18.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:18.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:18.820136+0000) 2024-02-19T21:08:19.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:19.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:19.820315+0000) 2024-02-19T21:08:20.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:20.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:20.820561+0000) 2024-02-19T21:08:21.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:21.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:21.820748+0000) 2024-02-19T21:08:22.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:22.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:22.820929+0000) 2024-02-19T21:08:23.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:23.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:23.821142+0000) 2024-02-19T21:08:24.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:24.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:24.821366+0000) 2024-02-19T21:08:25.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:25.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:25.821603+0000) 2024-02-19T21:08:26.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:26.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:26.821777+0000) 2024-02-19T21:08:27.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:27.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:27.822029+0000) 2024-02-19T21:08:28.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:28.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:28.822271+0000) 2024-02-19T21:08:29.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:29.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:29.822513+0000) 2024-02-19T21:08:30.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:30.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:30.822816+0000) 2024-02-19T21:08:31.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:31.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:31.823078+0000) 2024-02-19T21:08:32.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:32.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:32.823317+0000) 2024-02-19T21:08:33.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:33.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:33.823584+0000) 2024-02-19T21:08:34.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:34.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:34.823808+0000) 2024-02-19T21:08:35.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:35.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:35.824031+0000) 2024-02-19T21:08:36.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:36.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:36.824287+0000) 2024-02-19T21:08:37.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:37.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:37.824563+0000) 2024-02-19T21:08:38.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:38.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:38.824764+0000) 2024-02-19T21:08:39.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:39.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:39.825054+0000) 2024-02-19T21:08:40.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:40.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:40.825266+0000) 2024-02-19T21:08:41.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:41.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:41.825500+0000) 2024-02-19T21:08:42.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:42.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:42.825717+0000) 2024-02-19T21:08:43.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:43.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:43.825939+0000) 2024-02-19T21:08:44.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:44.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:44.826154+0000) 2024-02-19T21:08:45.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:45.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:45.826417+0000) 2024-02-19T21:08:46.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:46.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:46.826579+0000) 2024-02-19T21:08:47.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:47.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:47.826768+0000) 2024-02-19T21:08:48.307 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:08:48.314 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:08:48.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:48.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:48.827018+0000) 2024-02-19T21:08:49.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:49.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:49.827148+0000) 2024-02-19T21:08:50.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:50.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:50.827416+0000) 2024-02-19T21:08:51.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:51.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:51.827641+0000) 2024-02-19T21:08:52.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:52.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:52.827881+0000) 2024-02-19T21:08:53.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:53.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:53.828092+0000) 2024-02-19T21:08:54.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:54.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:54.828352+0000) 2024-02-19T21:08:55.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:55.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:55.828530+0000) 2024-02-19T21:08:56.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:56.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:56.828752+0000) 2024-02-19T21:08:57.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:57.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:57.828943+0000) 2024-02-19T21:08:58.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:58.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:58.829169+0000) 2024-02-19T21:08:59.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:08:59.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:08:59.829339+0000) 2024-02-19T21:09:00.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:00.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:00.829580+0000) 2024-02-19T21:09:01.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:01.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:01.829821+0000) 2024-02-19T21:09:02.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:02.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:02.830047+0000) 2024-02-19T21:09:03.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:03.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:03.830234+0000) 2024-02-19T21:09:04.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:04.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:04.830420+0000) 2024-02-19T21:09:05.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:05.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:05.830584+0000) 2024-02-19T21:09:06.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:06.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:06.830775+0000) 2024-02-19T21:09:07.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:07.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:07.830971+0000) 2024-02-19T21:09:08.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:08.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:08.831143+0000) 2024-02-19T21:09:09.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:09.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:09.831371+0000) 2024-02-19T21:09:10.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:10.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:10.831657+0000) 2024-02-19T21:09:11.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:11.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:11.831837+0000) 2024-02-19T21:09:12.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:12.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:12.832054+0000) 2024-02-19T21:09:13.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:13.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:13.832297+0000) 2024-02-19T21:09:14.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:14.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:14.832505+0000) 2024-02-19T21:09:15.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:15.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:15.832746+0000) 2024-02-19T21:09:16.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:16.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:16.832950+0000) 2024-02-19T21:09:17.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:17.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:17.833195+0000) 2024-02-19T21:09:18.376 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:09:18.379 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:09:18.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:18.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:18.833452+0000) 2024-02-19T21:09:19.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:19.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:19.833690+0000) 2024-02-19T21:09:20.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:20.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:20.833934+0000) 2024-02-19T21:09:21.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:21.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:21.834166+0000) 2024-02-19T21:09:22.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:22.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:22.834366+0000) 2024-02-19T21:09:23.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:23.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:23.834587+0000) 2024-02-19T21:09:24.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:24.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:24.834828+0000) 2024-02-19T21:09:25.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:25.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:25.835069+0000) 2024-02-19T21:09:26.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:26.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:26.835290+0000) 2024-02-19T21:09:27.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:27.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:27.835497+0000) 2024-02-19T21:09:28.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:28.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:28.835674+0000) 2024-02-19T21:09:29.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:29.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:29.835896+0000) 2024-02-19T21:09:30.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:30.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:30.836129+0000) 2024-02-19T21:09:31.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:31.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:31.836393+0000) 2024-02-19T21:09:32.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:32.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:32.836610+0000) 2024-02-19T21:09:33.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:33.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:33.836836+0000) 2024-02-19T21:09:34.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:34.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:34.837098+0000) 2024-02-19T21:09:35.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:35.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:35.837292+0000) 2024-02-19T21:09:36.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:36.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:36.837524+0000) 2024-02-19T21:09:37.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:37.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:37.837734+0000) 2024-02-19T21:09:38.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:38.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:38.837900+0000) 2024-02-19T21:09:39.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:39.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:39.838076+0000) 2024-02-19T21:09:40.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:40.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:40.838308+0000) 2024-02-19T21:09:41.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:41.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:41.838433+0000) 2024-02-19T21:09:42.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:42.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:42.838575+0000) 2024-02-19T21:09:43.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:43.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:43.838779+0000) 2024-02-19T21:09:44.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:44.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:44.838957+0000) 2024-02-19T21:09:45.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:45.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:45.839133+0000) 2024-02-19T21:09:46.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:46.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:46.839276+0000) 2024-02-19T21:09:47.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:47.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:47.839517+0000) 2024-02-19T21:09:48.436 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:09:48.441 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:09:48.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:48.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:48.839699+0000) 2024-02-19T21:09:49.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:49.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:49.839944+0000) 2024-02-19T21:09:50.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:50.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:50.840213+0000) 2024-02-19T21:09:51.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:51.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:51.840380+0000) 2024-02-19T21:09:52.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:52.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:52.840610+0000) 2024-02-19T21:09:53.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:53.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:53.840801+0000) 2024-02-19T21:09:54.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:54.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:54.841017+0000) 2024-02-19T21:09:55.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:55.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:55.841197+0000) 2024-02-19T21:09:56.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:56.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:56.841427+0000) 2024-02-19T21:09:57.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:57.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:57.841666+0000) 2024-02-19T21:09:58.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:58.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:58.841923+0000) 2024-02-19T21:09:59.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:09:59.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:09:59.842190+0000) 2024-02-19T21:10:00.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:00.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:00.842417+0000) 2024-02-19T21:10:01.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:01.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:01.842609+0000) 2024-02-19T21:10:02.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:02.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:02.842832+0000) 2024-02-19T21:10:03.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:03.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:03.843077+0000) 2024-02-19T21:10:04.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:04.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:04.843334+0000) 2024-02-19T21:10:05.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:05.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:05.843538+0000) 2024-02-19T21:10:06.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:06.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:06.843796+0000) 2024-02-19T21:10:07.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:07.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:07.843997+0000) 2024-02-19T21:10:08.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:08.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:08.844161+0000) 2024-02-19T21:10:09.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:09.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:09.844327+0000) 2024-02-19T21:10:10.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:10.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:10.844505+0000) 2024-02-19T21:10:11.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:11.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:11.844753+0000) 2024-02-19T21:10:12.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:12.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:12.844989+0000) 2024-02-19T21:10:13.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:13.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:13.845223+0000) 2024-02-19T21:10:14.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:14.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:14.845430+0000) 2024-02-19T21:10:15.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:15.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:15.845690+0000) 2024-02-19T21:10:16.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:16.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:16.845937+0000) 2024-02-19T21:10:17.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:17.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:17.846098+0000) 2024-02-19T21:10:18.567 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:10:18.570 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:10:18.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:18.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:18.846278+0000) 2024-02-19T21:10:19.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:19.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:19.846398+0000) 2024-02-19T21:10:20.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:20.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:20.846634+0000) 2024-02-19T21:10:21.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:21.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:21.846857+0000) 2024-02-19T21:10:22.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:22.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:22.847092+0000) 2024-02-19T21:10:23.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:23.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:23.847213+0000) 2024-02-19T21:10:24.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:24.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:24.847467+0000) 2024-02-19T21:10:25.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:25.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:25.847663+0000) 2024-02-19T21:10:26.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:26.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:26.847845+0000) 2024-02-19T21:10:27.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:27.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:27.848074+0000) 2024-02-19T21:10:28.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:28.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:28.848331+0000) 2024-02-19T21:10:29.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:29.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:29.848586+0000) 2024-02-19T21:10:30.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:30.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:30.848765+0000) 2024-02-19T21:10:31.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:31.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:31.848999+0000) 2024-02-19T21:10:32.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:32.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:32.849223+0000) 2024-02-19T21:10:33.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:33.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:33.849414+0000) 2024-02-19T21:10:34.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:34.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:34.849637+0000) 2024-02-19T21:10:35.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:35.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:35.849869+0000) 2024-02-19T21:10:36.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:36.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:36.850045+0000) 2024-02-19T21:10:37.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:37.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:37.850265+0000) 2024-02-19T21:10:38.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:38.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:38.850507+0000) 2024-02-19T21:10:39.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:39.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:39.850702+0000) 2024-02-19T21:10:40.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:40.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:40.850953+0000) 2024-02-19T21:10:41.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:41.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:41.851141+0000) 2024-02-19T21:10:42.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:42.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:42.851376+0000) 2024-02-19T21:10:43.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:43.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:43.851615+0000) 2024-02-19T21:10:44.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:44.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:44.851874+0000) 2024-02-19T21:10:45.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:45.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:45.852077+0000) 2024-02-19T21:10:46.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:46.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:46.852302+0000) 2024-02-19T21:10:47.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:47.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:47.852531+0000) 2024-02-19T21:10:48.672 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:10:48.676 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:10:48.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:48.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:48.852723+0000) 2024-02-19T21:10:49.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:49.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:49.852880+0000) 2024-02-19T21:10:50.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:50.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:50.853096+0000) 2024-02-19T21:10:51.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:51.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:51.853350+0000) 2024-02-19T21:10:52.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:52.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:52.853580+0000) 2024-02-19T21:10:53.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:53.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:53.853819+0000) 2024-02-19T21:10:54.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:54.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:54.854062+0000) 2024-02-19T21:10:55.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:55.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:55.854322+0000) 2024-02-19T21:10:56.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:56.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:56.854569+0000) 2024-02-19T21:10:57.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:57.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:57.854775+0000) 2024-02-19T21:10:58.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:58.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:58.854966+0000) 2024-02-19T21:10:59.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:10:59.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:10:59.855209+0000) 2024-02-19T21:11:00.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:00.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:00.855477+0000) 2024-02-19T21:11:01.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:01.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:01.855710+0000) 2024-02-19T21:11:02.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:02.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:02.855870+0000) 2024-02-19T21:11:03.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:03.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:03.856087+0000) 2024-02-19T21:11:04.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:04.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:04.856354+0000) 2024-02-19T21:11:05.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:05.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:05.856586+0000) 2024-02-19T21:11:06.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:06.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:06.856827+0000) 2024-02-19T21:11:07.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:07.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:07.857111+0000) 2024-02-19T21:11:08.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:08.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:08.857354+0000) 2024-02-19T21:11:09.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:09.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:09.857523+0000) 2024-02-19T21:11:10.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:10.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:10.857682+0000) 2024-02-19T21:11:11.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:11.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:11.857905+0000) 2024-02-19T21:11:12.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:12.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:12.858078+0000) 2024-02-19T21:11:13.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:13.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:13.858340+0000) 2024-02-19T21:11:14.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:14.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:14.858603+0000) 2024-02-19T21:11:15.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:15.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:15.858777+0000) 2024-02-19T21:11:16.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:16.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:16.859001+0000) 2024-02-19T21:11:17.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:17.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:17.859168+0000) 2024-02-19T21:11:18.787 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:11:18.791 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:11:18.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:18.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:18.859432+0000) 2024-02-19T21:11:19.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:19.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:19.859693+0000) 2024-02-19T21:11:20.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:20.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:20.859870+0000) 2024-02-19T21:11:21.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:21.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:21.860112+0000) 2024-02-19T21:11:22.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:22.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:22.860361+0000) 2024-02-19T21:11:23.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:23.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:23.860605+0000) 2024-02-19T21:11:24.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:24.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:24.860791+0000) 2024-02-19T21:11:25.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:25.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:25.861036+0000) 2024-02-19T21:11:26.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:26.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:26.861273+0000) 2024-02-19T21:11:27.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:27.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:27.861513+0000) 2024-02-19T21:11:28.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:28.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:28.861768+0000) 2024-02-19T21:11:29.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:29.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:29.862030+0000) 2024-02-19T21:11:30.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:30.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:30.862264+0000) 2024-02-19T21:11:31.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:31.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:31.862506+0000) 2024-02-19T21:11:32.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:32.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:32.862688+0000) 2024-02-19T21:11:33.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:33.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:33.862944+0000) 2024-02-19T21:11:34.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:34.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:34.863206+0000) 2024-02-19T21:11:35.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:35.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:35.863456+0000) 2024-02-19T21:11:36.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:36.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:36.863727+0000) 2024-02-19T21:11:37.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:37.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:37.863969+0000) 2024-02-19T21:11:38.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:38.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:38.864204+0000) 2024-02-19T21:11:39.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:39.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:39.864451+0000) 2024-02-19T21:11:40.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:40.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:40.864698+0000) 2024-02-19T21:11:41.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:41.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:41.864940+0000) 2024-02-19T21:11:42.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:42.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:42.865176+0000) 2024-02-19T21:11:43.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:43.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:43.865460+0000) 2024-02-19T21:11:44.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:44.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:44.865727+0000) 2024-02-19T21:11:45.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:45.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:45.865921+0000) 2024-02-19T21:11:46.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:46.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:46.866060+0000) 2024-02-19T21:11:47.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:47.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:47.866310+0000) 2024-02-19T21:11:48.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:48.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:48.866560+0000) 2024-02-19T21:11:48.927 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:11:48.930 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:11:49.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:49.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:49.866817+0000) 2024-02-19T21:11:50.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:50.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:50.867076+0000) 2024-02-19T21:11:51.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:51.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:51.867345+0000) 2024-02-19T21:11:52.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:52.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:52.867517+0000) 2024-02-19T21:11:53.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:53.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:53.867720+0000) 2024-02-19T21:11:54.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:54.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:54.867953+0000) 2024-02-19T21:11:55.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:55.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:55.868212+0000) 2024-02-19T21:11:56.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:56.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:56.868451+0000) 2024-02-19T21:11:57.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:57.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:57.868699+0000) 2024-02-19T21:11:58.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:58.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:58.868878+0000) 2024-02-19T21:11:59.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:11:59.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:11:59.869039+0000) 2024-02-19T21:12:00.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:00.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:00.869300+0000) 2024-02-19T21:12:01.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:01.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:01.869467+0000) 2024-02-19T21:12:02.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:02.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:02.869729+0000) 2024-02-19T21:12:03.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:03.864+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:03.870003+0000) 2024-02-19T21:12:04.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:04.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:04.870275+0000) 2024-02-19T21:12:05.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:05.864+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:05.870507+0000) 2024-02-19T21:12:06.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:06.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:06.870756+0000) 2024-02-19T21:12:07.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:07.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:07.870902+0000) 2024-02-19T21:12:08.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:08.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:08.871131+0000) 2024-02-19T21:12:09.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:09.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:09.871332+0000) 2024-02-19T21:12:10.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:10.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:10.871581+0000) 2024-02-19T21:12:11.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:11.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:11.871824+0000) 2024-02-19T21:12:12.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:12.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:12.872112+0000) 2024-02-19T21:12:13.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:13.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:13.872330+0000) 2024-02-19T21:12:14.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:14.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:14.872563+0000) 2024-02-19T21:12:15.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:15.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:15.872711+0000) 2024-02-19T21:12:16.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:16.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:16.872951+0000) 2024-02-19T21:12:17.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:17.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:17.873209+0000) 2024-02-19T21:12:18.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:18.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:18.873476+0000) 2024-02-19T21:12:18.998 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:12:19.001 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:12:19.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:19.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:19.873742+0000) 2024-02-19T21:12:20.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:20.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:20.873959+0000) 2024-02-19T21:12:21.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:21.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:21.874199+0000) 2024-02-19T21:12:22.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:22.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:22.874491+0000) 2024-02-19T21:12:23.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:23.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:23.874624+0000) 2024-02-19T21:12:24.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:24.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:24.874772+0000) 2024-02-19T21:12:25.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:25.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:25.874919+0000) 2024-02-19T21:12:26.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:26.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:26.875191+0000) 2024-02-19T21:12:27.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:27.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:27.875433+0000) 2024-02-19T21:12:28.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:28.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:28.875676+0000) 2024-02-19T21:12:29.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:29.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:29.875924+0000) 2024-02-19T21:12:30.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:30.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:30.876182+0000) 2024-02-19T21:12:31.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:31.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:31.876367+0000) 2024-02-19T21:12:32.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:32.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:32.876608+0000) 2024-02-19T21:12:33.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:33.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:33.876818+0000) 2024-02-19T21:12:34.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:34.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:34.877043+0000) 2024-02-19T21:12:35.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:35.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:35.877331+0000) 2024-02-19T21:12:36.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:36.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:36.877588+0000) 2024-02-19T21:12:37.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:37.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:37.877846+0000) 2024-02-19T21:12:38.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:38.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:38.878110+0000) 2024-02-19T21:12:39.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:39.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:39.878254+0000) 2024-02-19T21:12:40.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:40.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:40.878486+0000) 2024-02-19T21:12:41.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:41.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:41.878784+0000) 2024-02-19T21:12:42.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:42.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:42.878966+0000) 2024-02-19T21:12:43.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:43.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:43.879140+0000) 2024-02-19T21:12:44.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:44.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:44.879384+0000) 2024-02-19T21:12:45.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:45.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:45.879630+0000) 2024-02-19T21:12:46.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:46.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:46.879869+0000) 2024-02-19T21:12:47.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:47.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:47.880046+0000) 2024-02-19T21:12:48.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:48.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:48.880234+0000) 2024-02-19T21:12:49.121 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:12:49.126 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:12:49.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:49.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:49.880482+0000) 2024-02-19T21:12:50.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:50.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:50.880736+0000) 2024-02-19T21:12:51.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:51.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:51.880991+0000) 2024-02-19T21:12:52.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:52.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:52.881197+0000) 2024-02-19T21:12:53.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:53.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:53.881374+0000) 2024-02-19T21:12:54.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:54.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:54.881623+0000) 2024-02-19T21:12:55.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:55.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:55.881763+0000) 2024-02-19T21:12:56.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:56.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:56.882010+0000) 2024-02-19T21:12:57.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:57.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:57.882262+0000) 2024-02-19T21:12:58.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:58.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:58.882518+0000) 2024-02-19T21:12:59.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:12:59.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:12:59.882819+0000) 2024-02-19T21:13:00.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:00.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:00.883065+0000) 2024-02-19T21:13:01.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:01.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:01.883293+0000) 2024-02-19T21:13:02.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:02.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:02.883539+0000) 2024-02-19T21:13:03.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:03.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:03.883774+0000) 2024-02-19T21:13:04.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:04.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:04.884012+0000) 2024-02-19T21:13:05.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:05.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:05.884187+0000) 2024-02-19T21:13:06.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:06.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:06.884429+0000) 2024-02-19T21:13:07.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:07.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:07.884668+0000) 2024-02-19T21:13:08.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:08.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:08.884945+0000) 2024-02-19T21:13:09.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:09.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:09.885128+0000) 2024-02-19T21:13:10.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:10.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:10.885347+0000) 2024-02-19T21:13:11.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:11.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:11.885534+0000) 2024-02-19T21:13:12.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:12.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:12.885695+0000) 2024-02-19T21:13:13.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:13.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:13.885901+0000) 2024-02-19T21:13:14.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:14.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:14.886072+0000) 2024-02-19T21:13:15.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:15.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:15.886319+0000) 2024-02-19T21:13:16.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:16.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:16.886601+0000) 2024-02-19T21:13:17.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:17.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:17.886851+0000) 2024-02-19T21:13:18.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:18.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:18.887096+0000) 2024-02-19T21:13:19.185 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:13:19.188 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:13:19.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:19.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:19.887233+0000) 2024-02-19T21:13:20.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:20.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:20.887391+0000) 2024-02-19T21:13:21.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:21.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:21.887581+0000) 2024-02-19T21:13:22.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:22.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:22.887766+0000) 2024-02-19T21:13:23.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:23.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:23.888043+0000) 2024-02-19T21:13:24.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:24.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:24.888270+0000) 2024-02-19T21:13:25.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:25.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:25.888432+0000) 2024-02-19T21:13:26.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:26.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:26.888621+0000) 2024-02-19T21:13:27.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:27.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:27.888853+0000) 2024-02-19T21:13:28.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:28.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:28.889112+0000) 2024-02-19T21:13:29.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:29.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:29.889368+0000) 2024-02-19T21:13:30.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:30.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:30.889559+0000) 2024-02-19T21:13:31.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:31.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:31.889788+0000) 2024-02-19T21:13:32.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:32.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:32.890037+0000) 2024-02-19T21:13:33.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:33.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:33.890244+0000) 2024-02-19T21:13:34.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:34.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:34.890416+0000) 2024-02-19T21:13:35.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:35.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:35.890661+0000) 2024-02-19T21:13:36.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:36.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:36.890836+0000) 2024-02-19T21:13:37.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:37.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:37.891086+0000) 2024-02-19T21:13:38.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:38.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:38.891229+0000) 2024-02-19T21:13:39.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:39.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:39.891421+0000) 2024-02-19T21:13:40.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:40.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:40.891687+0000) 2024-02-19T21:13:41.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:41.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:41.891941+0000) 2024-02-19T21:13:42.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:42.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:42.892146+0000) 2024-02-19T21:13:43.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:43.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:43.892400+0000) 2024-02-19T21:13:44.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:44.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:44.892633+0000) 2024-02-19T21:13:45.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:45.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:45.892873+0000) 2024-02-19T21:13:46.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:46.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:46.893107+0000) 2024-02-19T21:13:47.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:47.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:47.893350+0000) 2024-02-19T21:13:48.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:48.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:48.893627+0000) 2024-02-19T21:13:49.266 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:13:49.269 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:13:49.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:49.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:49.893824+0000) 2024-02-19T21:13:50.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:50.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:50.894071+0000) 2024-02-19T21:13:51.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:51.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:51.894214+0000) 2024-02-19T21:13:52.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:52.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:52.894385+0000) 2024-02-19T21:13:53.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:53.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:53.894614+0000) 2024-02-19T21:13:54.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:54.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:54.894827+0000) 2024-02-19T21:13:55.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:55.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:55.895073+0000) 2024-02-19T21:13:56.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:56.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:56.895305+0000) 2024-02-19T21:13:57.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:57.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:57.895454+0000) 2024-02-19T21:13:58.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:58.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:58.895714+0000) 2024-02-19T21:13:59.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:13:59.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:13:59.895939+0000) 2024-02-19T21:14:00.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:00.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:00.896085+0000) 2024-02-19T21:14:01.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:01.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:01.896255+0000) 2024-02-19T21:14:02.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:02.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:02.896490+0000) 2024-02-19T21:14:03.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:03.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:03.896760+0000) 2024-02-19T21:14:04.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:04.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:04.897008+0000) 2024-02-19T21:14:05.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:05.890+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:05.897291+0000) 2024-02-19T21:14:06.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:06.890+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:06.897507+0000) 2024-02-19T21:14:07.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:07.890+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:07.897741+0000) 2024-02-19T21:14:08.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:08.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:08.897954+0000) 2024-02-19T21:14:09.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:09.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:09.898203+0000) 2024-02-19T21:14:10.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:10.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:10.898418+0000) 2024-02-19T21:14:11.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:11.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:11.898606+0000) 2024-02-19T21:14:12.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:12.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:12.898841+0000) 2024-02-19T21:14:13.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:13.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:13.899033+0000) 2024-02-19T21:14:14.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:14.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:14.899285+0000) 2024-02-19T21:14:15.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:15.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:15.899563+0000) 2024-02-19T21:14:16.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:16.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:16.899829+0000) 2024-02-19T21:14:17.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:17.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:17.900062+0000) 2024-02-19T21:14:18.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:18.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:18.900285+0000) 2024-02-19T21:14:19.329 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:14:19.333 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:14:19.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:19.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:19.900523+0000) 2024-02-19T21:14:20.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:20.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:20.900810+0000) 2024-02-19T21:14:21.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:21.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:21.901075+0000) 2024-02-19T21:14:22.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:22.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:22.901337+0000) 2024-02-19T21:14:23.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:23.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:23.901531+0000) 2024-02-19T21:14:24.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:24.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:24.901796+0000) 2024-02-19T21:14:25.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:25.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:25.901994+0000) 2024-02-19T21:14:26.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:26.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:26.902231+0000) 2024-02-19T21:14:27.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:27.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:27.902462+0000) 2024-02-19T21:14:28.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:28.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:28.902698+0000) 2024-02-19T21:14:29.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:29.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:29.902981+0000) 2024-02-19T21:14:30.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:30.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:30.903234+0000) 2024-02-19T21:14:31.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:31.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:31.903418+0000) 2024-02-19T21:14:32.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:32.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:32.903610+0000) 2024-02-19T21:14:33.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:33.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:33.903838+0000) 2024-02-19T21:14:34.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:34.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:34.904089+0000) 2024-02-19T21:14:35.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:35.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:35.904374+0000) 2024-02-19T21:14:36.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:36.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:36.904642+0000) 2024-02-19T21:14:37.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:37.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:37.904848+0000) 2024-02-19T21:14:38.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:38.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:38.905052+0000) 2024-02-19T21:14:39.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:39.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:39.905310+0000) 2024-02-19T21:14:40.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:40.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:40.905587+0000) 2024-02-19T21:14:41.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:41.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:41.905793+0000) 2024-02-19T21:14:42.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:42.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:42.906039+0000) 2024-02-19T21:14:43.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:43.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:43.906277+0000) 2024-02-19T21:14:44.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:44.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:44.906541+0000) 2024-02-19T21:14:45.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:45.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:45.906813+0000) 2024-02-19T21:14:46.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:46.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:46.906983+0000) 2024-02-19T21:14:47.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:47.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:47.907170+0000) 2024-02-19T21:14:48.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:48.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:48.907348+0000) 2024-02-19T21:14:49.386 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:14:49.391 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:14:49.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:49.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:49.907607+0000) 2024-02-19T21:14:50.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:50.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:50.907852+0000) 2024-02-19T21:14:51.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:51.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:51.908081+0000) 2024-02-19T21:14:52.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:52.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:52.908283+0000) 2024-02-19T21:14:53.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:53.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:53.908501+0000) 2024-02-19T21:14:54.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:54.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:54.908761+0000) 2024-02-19T21:14:55.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:55.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:55.908942+0000) 2024-02-19T21:14:56.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:56.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:56.909145+0000) 2024-02-19T21:14:57.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:57.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:57.909381+0000) 2024-02-19T21:14:58.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:58.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:58.909641+0000) 2024-02-19T21:14:59.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:14:59.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:14:59.909790+0000) 2024-02-19T21:15:00.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:00.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:00.909998+0000) 2024-02-19T21:15:01.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:01.910+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:01.910227+0000) 2024-02-19T21:15:02.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:02.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:02.910377+0000) 2024-02-19T21:15:03.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:03.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:03.910533+0000) 2024-02-19T21:15:04.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:04.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:04.910844+0000) 2024-02-19T21:15:05.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:05.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:05.911032+0000) 2024-02-19T21:15:06.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:06.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:06.911196+0000) 2024-02-19T21:15:07.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:07.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:07.911339+0000) 2024-02-19T21:15:08.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:08.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:08.911567+0000) 2024-02-19T21:15:09.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:09.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:09.911800+0000) 2024-02-19T21:15:10.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:10.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:10.912056+0000) 2024-02-19T21:15:11.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:11.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:11.912288+0000) 2024-02-19T21:15:12.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:12.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:12.912514+0000) 2024-02-19T21:15:13.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:13.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:13.912700+0000) 2024-02-19T21:15:14.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:14.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:14.912929+0000) 2024-02-19T21:15:15.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:15.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:15.913171+0000) 2024-02-19T21:15:16.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:16.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:16.913310+0000) 2024-02-19T21:15:17.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:17.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:17.913539+0000) 2024-02-19T21:15:18.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:18.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:18.913718+0000) 2024-02-19T21:15:19.452 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:15:19.456 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:15:19.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:19.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:19.913942+0000) 2024-02-19T21:15:20.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:20.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:20.914078+0000) 2024-02-19T21:15:21.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:21.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:21.914306+0000) 2024-02-19T21:15:22.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:22.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:22.914541+0000) 2024-02-19T21:15:23.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:23.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:23.914746+0000) 2024-02-19T21:15:24.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:24.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:24.915011+0000) 2024-02-19T21:15:25.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:25.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:25.915241+0000) 2024-02-19T21:15:26.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:26.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:26.915412+0000) 2024-02-19T21:15:27.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:27.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:27.915641+0000) 2024-02-19T21:15:28.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:28.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:28.915856+0000) 2024-02-19T21:15:29.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:29.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:29.916035+0000) 2024-02-19T21:15:30.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:30.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:30.916250+0000) 2024-02-19T21:15:31.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:31.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:31.916495+0000) 2024-02-19T21:15:32.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:32.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:32.916745+0000) 2024-02-19T21:15:33.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:33.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:33.916930+0000) 2024-02-19T21:15:34.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:34.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:34.917146+0000) 2024-02-19T21:15:35.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:35.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:35.917380+0000) 2024-02-19T21:15:36.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:36.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:36.917568+0000) 2024-02-19T21:15:37.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:37.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:37.917746+0000) 2024-02-19T21:15:38.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:38.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:38.917976+0000) 2024-02-19T21:15:39.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:39.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:39.918270+0000) 2024-02-19T21:15:40.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:40.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:40.918515+0000) 2024-02-19T21:15:41.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:41.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:41.918772+0000) 2024-02-19T21:15:42.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:42.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:42.918896+0000) 2024-02-19T21:15:43.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:43.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:43.919029+0000) 2024-02-19T21:15:44.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:44.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:44.919273+0000) 2024-02-19T21:15:45.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:45.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:45.919537+0000) 2024-02-19T21:15:46.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:46.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:46.919761+0000) 2024-02-19T21:15:47.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:47.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:47.920035+0000) 2024-02-19T21:15:48.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:48.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:48.920309+0000) 2024-02-19T21:15:49.572 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:15:49.576 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:15:49.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:49.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:49.920552+0000) 2024-02-19T21:15:50.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:50.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:50.920738+0000) 2024-02-19T21:15:51.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:51.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:51.920996+0000) 2024-02-19T21:15:52.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:52.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:52.921203+0000) 2024-02-19T21:15:53.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:53.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:53.921426+0000) 2024-02-19T21:15:54.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:54.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:54.921618+0000) 2024-02-19T21:15:55.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:55.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:55.921827+0000) 2024-02-19T21:15:56.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:56.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:56.922069+0000) 2024-02-19T21:15:57.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:57.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:57.922234+0000) 2024-02-19T21:15:58.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:58.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:58.922467+0000) 2024-02-19T21:15:59.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:15:59.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:15:59.922641+0000) 2024-02-19T21:16:00.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:00.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:00.922859+0000) 2024-02-19T21:16:01.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:01.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:01.923040+0000) 2024-02-19T21:16:02.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:02.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:02.923213+0000) 2024-02-19T21:16:03.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:03.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:03.923448+0000) 2024-02-19T21:16:04.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:04.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:04.923602+0000) 2024-02-19T21:16:05.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:05.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:05.923799+0000) 2024-02-19T21:16:06.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:06.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:06.923957+0000) 2024-02-19T21:16:07.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:07.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:07.924178+0000) 2024-02-19T21:16:08.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:08.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:08.924425+0000) 2024-02-19T21:16:09.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:09.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:09.924656+0000) 2024-02-19T21:16:10.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:10.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:10.924898+0000) 2024-02-19T21:16:11.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:11.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:11.925137+0000) 2024-02-19T21:16:12.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:12.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:12.925363+0000) 2024-02-19T21:16:13.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:13.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:13.925565+0000) 2024-02-19T21:16:14.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:14.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:14.925734+0000) 2024-02-19T21:16:15.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:15.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:15.926019+0000) 2024-02-19T21:16:16.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:16.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:16.926259+0000) 2024-02-19T21:16:17.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:17.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:17.926494+0000) 2024-02-19T21:16:18.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:18.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:18.926790+0000) 2024-02-19T21:16:19.686 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:16:19.689 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:16:19.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:19.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:19.926986+0000) 2024-02-19T21:16:20.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:20.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:20.927210+0000) 2024-02-19T21:16:21.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:21.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:21.927380+0000) 2024-02-19T21:16:22.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:22.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:22.927616+0000) 2024-02-19T21:16:23.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:23.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:23.927817+0000) 2024-02-19T21:16:24.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:24.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:24.927936+0000) 2024-02-19T21:16:25.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:25.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:25.928133+0000) 2024-02-19T21:16:26.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:26.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:26.928382+0000) 2024-02-19T21:16:27.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:27.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:27.928615+0000) 2024-02-19T21:16:28.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:28.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:28.928783+0000) 2024-02-19T21:16:29.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:29.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:29.929002+0000) 2024-02-19T21:16:30.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:30.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:30.929195+0000) 2024-02-19T21:16:31.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:31.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:31.929436+0000) 2024-02-19T21:16:32.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:32.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:32.929632+0000) 2024-02-19T21:16:33.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:33.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:33.929851+0000) 2024-02-19T21:16:34.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:34.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:34.930030+0000) 2024-02-19T21:16:35.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:35.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:35.930254+0000) 2024-02-19T21:16:36.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:36.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:36.930469+0000) 2024-02-19T21:16:37.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:37.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:37.930647+0000) 2024-02-19T21:16:38.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:38.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:38.930800+0000) 2024-02-19T21:16:39.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:39.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:39.930955+0000) 2024-02-19T21:16:40.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:40.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:40.931086+0000) 2024-02-19T21:16:41.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:41.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:41.931263+0000) 2024-02-19T21:16:42.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:42.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:42.931490+0000) 2024-02-19T21:16:43.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:43.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:43.931703+0000) 2024-02-19T21:16:44.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:44.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:44.931924+0000) 2024-02-19T21:16:45.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:45.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:45.932207+0000) 2024-02-19T21:16:46.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:46.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:46.932466+0000) 2024-02-19T21:16:47.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:47.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:47.932707+0000) 2024-02-19T21:16:48.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:48.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:48.932958+0000) 2024-02-19T21:16:49.817 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:16:49.821 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:16:49.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:49.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:49.933192+0000) 2024-02-19T21:16:50.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:50.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:50.933445+0000) 2024-02-19T21:16:51.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:51.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:51.933668+0000) 2024-02-19T21:16:52.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:52.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:52.933911+0000) 2024-02-19T21:16:53.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:53.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:53.934148+0000) 2024-02-19T21:16:54.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:54.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:54.934388+0000) 2024-02-19T21:16:55.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:55.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:55.934626+0000) 2024-02-19T21:16:56.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:56.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:56.934766+0000) 2024-02-19T21:16:57.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:57.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:57.935011+0000) 2024-02-19T21:16:58.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:58.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:58.935217+0000) 2024-02-19T21:16:59.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:16:59.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:16:59.935479+0000) 2024-02-19T21:17:00.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:00.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:00.935728+0000) 2024-02-19T21:17:01.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:01.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:01.935985+0000) 2024-02-19T21:17:02.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:02.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:02.936191+0000) 2024-02-19T21:17:03.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:03.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:03.936417+0000) 2024-02-19T21:17:04.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:04.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:04.936610+0000) 2024-02-19T21:17:05.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:05.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:05.936840+0000) 2024-02-19T21:17:06.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:06.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:06.937016+0000) 2024-02-19T21:17:07.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:07.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:07.937261+0000) 2024-02-19T21:17:08.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:08.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:08.937508+0000) 2024-02-19T21:17:09.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:09.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:09.937780+0000) 2024-02-19T21:17:10.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:10.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:10.937979+0000) 2024-02-19T21:17:11.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:11.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:11.938155+0000) 2024-02-19T21:17:12.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:12.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:12.938342+0000) 2024-02-19T21:17:13.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:13.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:13.938594+0000) 2024-02-19T21:17:14.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:14.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:14.938829+0000) 2024-02-19T21:17:15.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:15.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:15.939109+0000) 2024-02-19T21:17:16.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:16.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:16.939300+0000) 2024-02-19T21:17:17.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:17.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:17.939552+0000) 2024-02-19T21:17:18.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:18.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:18.939803+0000) 2024-02-19T21:17:19.938 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:17:19.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:19.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:19.940034+0000) 2024-02-19T21:17:19.943 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:17:20.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:20.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:20.940298+0000) 2024-02-19T21:17:21.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:21.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:21.940523+0000) 2024-02-19T21:17:22.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:22.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:22.940700+0000) 2024-02-19T21:17:23.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:23.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:23.940934+0000) 2024-02-19T21:17:24.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:24.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:24.941163+0000) 2024-02-19T21:17:25.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:25.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:25.941408+0000) 2024-02-19T21:17:26.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:26.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:26.941667+0000) 2024-02-19T21:17:27.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:27.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:27.941915+0000) 2024-02-19T21:17:28.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:28.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:28.942160+0000) 2024-02-19T21:17:29.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:29.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:29.942337+0000) 2024-02-19T21:17:30.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:30.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:30.942558+0000) 2024-02-19T21:17:31.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:31.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:31.942830+0000) 2024-02-19T21:17:32.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:32.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:32.943072+0000) 2024-02-19T21:17:33.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:33.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:33.943315+0000) 2024-02-19T21:17:34.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:34.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:34.943550+0000) 2024-02-19T21:17:35.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:35.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:35.943753+0000) 2024-02-19T21:17:36.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:36.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:36.943935+0000) 2024-02-19T21:17:37.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:37.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:37.944163+0000) 2024-02-19T21:17:38.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:38.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:38.944417+0000) 2024-02-19T21:17:39.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:39.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:39.944664+0000) 2024-02-19T21:17:40.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:40.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:40.944901+0000) 2024-02-19T21:17:41.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:41.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:41.945099+0000) 2024-02-19T21:17:42.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:42.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:42.945279+0000) 2024-02-19T21:17:43.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:43.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:43.945479+0000) 2024-02-19T21:17:44.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:44.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:44.945680+0000) 2024-02-19T21:17:45.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:45.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:45.945935+0000) 2024-02-19T21:17:46.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:46.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:46.946143+0000) 2024-02-19T21:17:47.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:47.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:47.946393+0000) 2024-02-19T21:17:48.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:48.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:48.946606+0000) 2024-02-19T21:17:49.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:49.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:49.946809+0000) 2024-02-19T21:17:49.994 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:17:49.997 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:17:50.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:50.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:50.947039+0000) 2024-02-19T21:17:51.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:51.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:51.947214+0000) 2024-02-19T21:17:52.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:52.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:52.947382+0000) 2024-02-19T21:17:53.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:53.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:53.947574+0000) 2024-02-19T21:17:54.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:54.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:54.947738+0000) 2024-02-19T21:17:55.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:55.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:55.947917+0000) 2024-02-19T21:17:56.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:56.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:56.948100+0000) 2024-02-19T21:17:57.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:57.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:57.948315+0000) 2024-02-19T21:17:58.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:58.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:58.948558+0000) 2024-02-19T21:17:59.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:17:59.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:17:59.948732+0000) 2024-02-19T21:18:00.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:00.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:00.949004+0000) 2024-02-19T21:18:01.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:01.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:01.949170+0000) 2024-02-19T21:18:02.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:02.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:02.949421+0000) 2024-02-19T21:18:03.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:03.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:03.949618+0000) 2024-02-19T21:18:04.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:04.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:04.949878+0000) 2024-02-19T21:18:05.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:05.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:05.950053+0000) 2024-02-19T21:18:06.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:06.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:06.950301+0000) 2024-02-19T21:18:07.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:07.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:07.950524+0000) 2024-02-19T21:18:08.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:08.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:08.950783+0000) 2024-02-19T21:18:09.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:09.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:09.950925+0000) 2024-02-19T21:18:10.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:10.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:10.951097+0000) 2024-02-19T21:18:11.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:11.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:11.951288+0000) 2024-02-19T21:18:12.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:12.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:12.951521+0000) 2024-02-19T21:18:13.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:13.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:13.951770+0000) 2024-02-19T21:18:14.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:14.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:14.952034+0000) 2024-02-19T21:18:15.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:15.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:15.952274+0000) 2024-02-19T21:18:16.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:16.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:16.952558+0000) 2024-02-19T21:18:17.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:17.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:17.952794+0000) 2024-02-19T21:18:18.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:18.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:18.953076+0000) 2024-02-19T21:18:19.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:19.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:19.953313+0000) 2024-02-19T21:18:20.050 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:18:20.055 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:18:20.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:20.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:20.953553+0000) 2024-02-19T21:18:21.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:21.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:21.953810+0000) 2024-02-19T21:18:22.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:22.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:22.953988+0000) 2024-02-19T21:18:23.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:23.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:23.954242+0000) 2024-02-19T21:18:24.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:24.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:24.954488+0000) 2024-02-19T21:18:25.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:25.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:25.954678+0000) 2024-02-19T21:18:26.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:26.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:26.954897+0000) 2024-02-19T21:18:27.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:27.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:27.955056+0000) 2024-02-19T21:18:28.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:28.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:28.955260+0000) 2024-02-19T21:18:29.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:29.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:29.955497+0000) 2024-02-19T21:18:30.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:30.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:30.955753+0000) 2024-02-19T21:18:31.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:31.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:31.955964+0000) 2024-02-19T21:18:32.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:32.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:32.956208+0000) 2024-02-19T21:18:33.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:33.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:33.956450+0000) 2024-02-19T21:18:34.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:34.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:34.956698+0000) 2024-02-19T21:18:35.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:35.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:35.956888+0000) 2024-02-19T21:18:36.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:36.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:36.957107+0000) 2024-02-19T21:18:37.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:37.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:37.957313+0000) 2024-02-19T21:18:38.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:38.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:38.957566+0000) 2024-02-19T21:18:39.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:39.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:39.957829+0000) 2024-02-19T21:18:40.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:40.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:40.958083+0000) 2024-02-19T21:18:41.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:41.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:41.958328+0000) 2024-02-19T21:18:42.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:42.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:42.958564+0000) 2024-02-19T21:18:43.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:43.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:43.958855+0000) 2024-02-19T21:18:44.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:44.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:44.959036+0000) 2024-02-19T21:18:45.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:45.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:45.959160+0000) 2024-02-19T21:18:46.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:46.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:46.959359+0000) 2024-02-19T21:18:47.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:47.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:47.959609+0000) 2024-02-19T21:18:48.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:48.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:48.959861+0000) 2024-02-19T21:18:49.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:49.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:49.960023+0000) 2024-02-19T21:18:50.123 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:18:50.126 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:18:50.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:50.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:50.960286+0000) 2024-02-19T21:18:51.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:51.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:51.960576+0000) 2024-02-19T21:18:52.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:52.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:52.960850+0000) 2024-02-19T21:18:53.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:53.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:53.961162+0000) 2024-02-19T21:18:54.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:54.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:54.961451+0000) 2024-02-19T21:18:55.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:55.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:55.961730+0000) 2024-02-19T21:18:56.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:56.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:56.962068+0000) 2024-02-19T21:18:57.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:57.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:57.962360+0000) 2024-02-19T21:18:58.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:58.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:58.962667+0000) 2024-02-19T21:18:59.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:18:59.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:18:59.962951+0000) 2024-02-19T21:19:00.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:00.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:00.963151+0000) 2024-02-19T21:19:01.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:01.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:01.963358+0000) 2024-02-19T21:19:02.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:02.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:02.963576+0000) 2024-02-19T21:19:03.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:03.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:03.963867+0000) 2024-02-19T21:19:04.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:04.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:04.964033+0000) 2024-02-19T21:19:05.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:05.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:05.964330+0000) 2024-02-19T21:19:06.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:06.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:06.964613+0000) 2024-02-19T21:19:07.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:07.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:07.964853+0000) 2024-02-19T21:19:08.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:08.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:08.965157+0000) 2024-02-19T21:19:09.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:09.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:09.965442+0000) 2024-02-19T21:19:10.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:10.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:10.965747+0000) 2024-02-19T21:19:11.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:11.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:11.966026+0000) 2024-02-19T21:19:12.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:12.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:12.966321+0000) 2024-02-19T21:19:13.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:13.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:13.966593+0000) 2024-02-19T21:19:14.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:14.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:14.966893+0000) 2024-02-19T21:19:15.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:15.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:15.967141+0000) 2024-02-19T21:19:16.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:16.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:16.967419+0000) 2024-02-19T21:19:17.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:17.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:17.967650+0000) 2024-02-19T21:19:18.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:18.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:18.967839+0000) 2024-02-19T21:19:19.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:19.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:19.968097+0000) 2024-02-19T21:19:20.269 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:19:20.274 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:19:20.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:20.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:20.968298+0000) 2024-02-19T21:19:21.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:21.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:21.968577+0000) 2024-02-19T21:19:22.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:22.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:22.968864+0000) 2024-02-19T21:19:23.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:23.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:23.969122+0000) 2024-02-19T21:19:24.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:24.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:24.969401+0000) 2024-02-19T21:19:25.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:25.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:25.969694+0000) 2024-02-19T21:19:26.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:26.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:26.969968+0000) 2024-02-19T21:19:27.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:27.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:27.970184+0000) 2024-02-19T21:19:28.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:28.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:28.970481+0000) 2024-02-19T21:19:29.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:29.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:29.970841+0000) 2024-02-19T21:19:30.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:30.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:30.971075+0000) 2024-02-19T21:19:31.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:31.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:31.971323+0000) 2024-02-19T21:19:32.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:32.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:32.971566+0000) 2024-02-19T21:19:33.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:33.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:33.971829+0000) 2024-02-19T21:19:34.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:34.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:34.972020+0000) 2024-02-19T21:19:35.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:35.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:35.972285+0000) 2024-02-19T21:19:36.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:36.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:36.972477+0000) 2024-02-19T21:19:37.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:37.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:37.972691+0000) 2024-02-19T21:19:38.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:38.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:38.972897+0000) 2024-02-19T21:19:39.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:39.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:39.973142+0000) 2024-02-19T21:19:40.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:40.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:40.973377+0000) 2024-02-19T21:19:41.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:41.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:41.973652+0000) 2024-02-19T21:19:42.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:42.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:42.973897+0000) 2024-02-19T21:19:43.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:43.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:43.974092+0000) 2024-02-19T21:19:44.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:44.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:44.974282+0000) 2024-02-19T21:19:45.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:45.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:45.974483+0000) 2024-02-19T21:19:46.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:46.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:46.974760+0000) 2024-02-19T21:19:47.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:47.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:47.975035+0000) 2024-02-19T21:19:48.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:48.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:48.975288+0000) 2024-02-19T21:19:49.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:49.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:49.975540+0000) 2024-02-19T21:19:50.401 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:19:50.404 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:19:50.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:50.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:50.975996+0000) 2024-02-19T21:19:51.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:51.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:51.976253+0000) 2024-02-19T21:19:52.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:52.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:52.976478+0000) 2024-02-19T21:19:53.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:53.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:53.976738+0000) 2024-02-19T21:19:54.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:54.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:54.976975+0000) 2024-02-19T21:19:55.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:55.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:55.977203+0000) 2024-02-19T21:19:56.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:56.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:56.977408+0000) 2024-02-19T21:19:57.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:57.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:57.977637+0000) 2024-02-19T21:19:58.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:58.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:58.977908+0000) 2024-02-19T21:19:59.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:19:59.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:19:59.978151+0000) 2024-02-19T21:20:00.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:00.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:00.978380+0000) 2024-02-19T21:20:01.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:01.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:01.978629+0000) 2024-02-19T21:20:02.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:02.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:02.978877+0000) 2024-02-19T21:20:03.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:03.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:03.979157+0000) 2024-02-19T21:20:04.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:04.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:04.979361+0000) 2024-02-19T21:20:05.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:05.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:05.979571+0000) 2024-02-19T21:20:06.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:06.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:06.979757+0000) 2024-02-19T21:20:07.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:07.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:07.979892+0000) 2024-02-19T21:20:08.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:08.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:08.980093+0000) 2024-02-19T21:20:09.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:09.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:09.980315+0000) 2024-02-19T21:20:10.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:10.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:10.980453+0000) 2024-02-19T21:20:11.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:11.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:11.980668+0000) 2024-02-19T21:20:12.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:12.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:12.980904+0000) 2024-02-19T21:20:13.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:13.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:13.981139+0000) 2024-02-19T21:20:14.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:14.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:14.981364+0000) 2024-02-19T21:20:15.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:15.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:15.981619+0000) 2024-02-19T21:20:16.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:16.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:16.981839+0000) 2024-02-19T21:20:17.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:17.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:17.982111+0000) 2024-02-19T21:20:18.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:18.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:18.982358+0000) 2024-02-19T21:20:19.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:19.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:19.982613+0000) 2024-02-19T21:20:20.539 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:20:20.544 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:20:20.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:20.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:20.982790+0000) 2024-02-19T21:20:21.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:21.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:21.983027+0000) 2024-02-19T21:20:22.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:22.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:22.983269+0000) 2024-02-19T21:20:23.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:23.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:23.983477+0000) 2024-02-19T21:20:24.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:24.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:24.983710+0000) 2024-02-19T21:20:25.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:25.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:25.983943+0000) 2024-02-19T21:20:26.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:26.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:26.984184+0000) 2024-02-19T21:20:27.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:27.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:27.984364+0000) 2024-02-19T21:20:28.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:28.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:28.984566+0000) 2024-02-19T21:20:29.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:29.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:29.984807+0000) 2024-02-19T21:20:30.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:30.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:30.985040+0000) 2024-02-19T21:20:31.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:31.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:31.985336+0000) 2024-02-19T21:20:32.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:32.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:32.985587+0000) 2024-02-19T21:20:33.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:33.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:33.985789+0000) 2024-02-19T21:20:34.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:34.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:34.986024+0000) 2024-02-19T21:20:35.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:35.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:35.986268+0000) 2024-02-19T21:20:36.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:36.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:36.986530+0000) 2024-02-19T21:20:37.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:37.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:37.986760+0000) 2024-02-19T21:20:38.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:38.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:38.987030+0000) 2024-02-19T21:20:39.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:39.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:39.987298+0000) 2024-02-19T21:20:40.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:40.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:40.987553+0000) 2024-02-19T21:20:41.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:41.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:41.987793+0000) 2024-02-19T21:20:42.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:42.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:42.988046+0000) 2024-02-19T21:20:43.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:43.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:43.988224+0000) 2024-02-19T21:20:44.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:44.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:44.988425+0000) 2024-02-19T21:20:45.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:45.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:45.988668+0000) 2024-02-19T21:20:46.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:46.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:46.988949+0000) 2024-02-19T21:20:47.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:47.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:47.989186+0000) 2024-02-19T21:20:48.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:48.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:48.989458+0000) 2024-02-19T21:20:49.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:49.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:49.989711+0000) 2024-02-19T21:20:50.687 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:20:50.691 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:20:50.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:50.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:50.989916+0000) 2024-02-19T21:20:51.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:51.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:51.990153+0000) 2024-02-19T21:20:52.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:52.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:52.990346+0000) 2024-02-19T21:20:53.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:53.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:53.990583+0000) 2024-02-19T21:20:54.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:54.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:54.990821+0000) 2024-02-19T21:20:55.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:55.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:55.990993+0000) 2024-02-19T21:20:56.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:56.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:56.991156+0000) 2024-02-19T21:20:57.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:57.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:57.991346+0000) 2024-02-19T21:20:58.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:58.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:58.991533+0000) 2024-02-19T21:20:59.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:20:59.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:20:59.991786+0000) 2024-02-19T21:21:00.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:00.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:00.992040+0000) 2024-02-19T21:21:01.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:01.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:01.992216+0000) 2024-02-19T21:21:02.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:02.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:02.992440+0000) 2024-02-19T21:21:03.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:03.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:03.992693+0000) 2024-02-19T21:21:04.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:04.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:04.992927+0000) 2024-02-19T21:21:05.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:05.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:05.993157+0000) 2024-02-19T21:21:06.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:06.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:06.993370+0000) 2024-02-19T21:21:07.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:07.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:07.993613+0000) 2024-02-19T21:21:08.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:08.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:08.993850+0000) 2024-02-19T21:21:09.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:09.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:09.994041+0000) 2024-02-19T21:21:10.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:10.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:10.994295+0000) 2024-02-19T21:21:11.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:11.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:11.994546+0000) 2024-02-19T21:21:12.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:12.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:12.994746+0000) 2024-02-19T21:21:13.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:13.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:13.995018+0000) 2024-02-19T21:21:14.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:14.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:14.995211+0000) 2024-02-19T21:21:15.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:15.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:15.995470+0000) 2024-02-19T21:21:16.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:16.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:16.995690+0000) 2024-02-19T21:21:17.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:17.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:17.995954+0000) 2024-02-19T21:21:18.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:18.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:18.996121+0000) 2024-02-19T21:21:19.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:19.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:19.996368+0000) 2024-02-19T21:21:20.827 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:21:20.830 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:21:20.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:20.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:20.996578+0000) 2024-02-19T21:21:21.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:21.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:21.996786+0000) 2024-02-19T21:21:22.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:22.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:22.997042+0000) 2024-02-19T21:21:23.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:23.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:23.997243+0000) 2024-02-19T21:21:24.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:24.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:24.997488+0000) 2024-02-19T21:21:25.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:25.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:25.997729+0000) 2024-02-19T21:21:26.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:26.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:26.997943+0000) 2024-02-19T21:21:27.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:27.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:27.998183+0000) 2024-02-19T21:21:28.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:28.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:28.998444+0000) 2024-02-19T21:21:30.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:29.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:29.998682+0000) 2024-02-19T21:21:31.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:30.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:30.998951+0000) 2024-02-19T21:21:32.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:31.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:31.999178+0000) 2024-02-19T21:21:33.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:32.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:32.999396+0000) 2024-02-19T21:21:34.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:33.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:33.999638+0000) 2024-02-19T21:21:35.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:34.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:34.999877+0000) 2024-02-19T21:21:36.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:35.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:36.000122+0000) 2024-02-19T21:21:37.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:36.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:37.000361+0000) 2024-02-19T21:21:38.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:37.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:38.000626+0000) 2024-02-19T21:21:39.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:38.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:39.000834+0000) 2024-02-19T21:21:40.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:39.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:40.001087+0000) 2024-02-19T21:21:41.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:40.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:41.001320+0000) 2024-02-19T21:21:42.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:41.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:42.001573+0000) 2024-02-19T21:21:43.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:42.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:43.001796+0000) 2024-02-19T21:21:44.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:43.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:44.002055+0000) 2024-02-19T21:21:45.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:44.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:45.002243+0000) 2024-02-19T21:21:46.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:45.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:46.002437+0000) 2024-02-19T21:21:47.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:46.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:47.002659+0000) 2024-02-19T21:21:48.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:47.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:48.002871+0000) 2024-02-19T21:21:49.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:48.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:49.003113+0000) 2024-02-19T21:21:50.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:49.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:50.003267+0000) 2024-02-19T21:21:50.890 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:21:50.894 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:21:51.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:50.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:51.003505+0000) 2024-02-19T21:21:52.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:52.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:52.003790+0000) 2024-02-19T21:21:53.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:53.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:53.004016+0000) 2024-02-19T21:21:54.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:54.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:54.004275+0000) 2024-02-19T21:21:55.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:55.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:55.004470+0000) 2024-02-19T21:21:56.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:56.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:56.004734+0000) 2024-02-19T21:21:57.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:57.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:57.004998+0000) 2024-02-19T21:21:58.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:58.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:58.005182+0000) 2024-02-19T21:21:59.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:58.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:21:59.005412+0000) 2024-02-19T21:22:00.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:21:59.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:00.005585+0000) 2024-02-19T21:22:01.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:01.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:01.005833+0000) 2024-02-19T21:22:02.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:01.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:02.006072+0000) 2024-02-19T21:22:03.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:03.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:03.006311+0000) 2024-02-19T21:22:04.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:03.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:04.006479+0000) 2024-02-19T21:22:05.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:05.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:05.006764+0000) 2024-02-19T21:22:06.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:06.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:06.007008+0000) 2024-02-19T21:22:07.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:07.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:07.007241+0000) 2024-02-19T21:22:08.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:08.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:08.007384+0000) 2024-02-19T21:22:09.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:09.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:09.007588+0000) 2024-02-19T21:22:10.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:10.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:10.007756+0000) 2024-02-19T21:22:11.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:11.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:11.007941+0000) 2024-02-19T21:22:12.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:12.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:12.008136+0000) 2024-02-19T21:22:13.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:13.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:13.008323+0000) 2024-02-19T21:22:14.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:14.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:14.008563+0000) 2024-02-19T21:22:15.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:15.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:15.008681+0000) 2024-02-19T21:22:16.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:16.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:16.008915+0000) 2024-02-19T21:22:17.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:17.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:17.009138+0000) 2024-02-19T21:22:18.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:18.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:18.009384+0000) 2024-02-19T21:22:19.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:19.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:19.009628+0000) 2024-02-19T21:22:20.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:20.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:20.009953+0000) 2024-02-19T21:22:20.973 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:22:20.976 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:22:21.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:21.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:21.010237+0000) 2024-02-19T21:22:22.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:22.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:22.010457+0000) 2024-02-19T21:22:23.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:23.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:23.010744+0000) 2024-02-19T21:22:24.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:24.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:24.010916+0000) 2024-02-19T21:22:25.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:25.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:25.011092+0000) 2024-02-19T21:22:26.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:26.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:26.011282+0000) 2024-02-19T21:22:27.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:27.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:27.011452+0000) 2024-02-19T21:22:28.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:28.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:28.011625+0000) 2024-02-19T21:22:29.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:29.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:29.011803+0000) 2024-02-19T21:22:30.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:30.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:30.012048+0000) 2024-02-19T21:22:31.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:31.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:31.012273+0000) 2024-02-19T21:22:32.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:32.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:32.012496+0000) 2024-02-19T21:22:33.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:33.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:33.012727+0000) 2024-02-19T21:22:34.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:34.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:34.012947+0000) 2024-02-19T21:22:35.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:35.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:35.013195+0000) 2024-02-19T21:22:36.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:36.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:36.013434+0000) 2024-02-19T21:22:37.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:37.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:37.013664+0000) 2024-02-19T21:22:38.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:38.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:38.013909+0000) 2024-02-19T21:22:39.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:39.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:39.014147+0000) 2024-02-19T21:22:40.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:40.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:40.014414+0000) 2024-02-19T21:22:41.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:41.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:41.014669+0000) 2024-02-19T21:22:42.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:42.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:42.014877+0000) 2024-02-19T21:22:43.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:43.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:43.015061+0000) 2024-02-19T21:22:44.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:44.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:44.015285+0000) 2024-02-19T21:22:45.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:45.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:45.015539+0000) 2024-02-19T21:22:46.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:46.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:46.015802+0000) 2024-02-19T21:22:47.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:47.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:47.016024+0000) 2024-02-19T21:22:48.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:48.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:48.016160+0000) 2024-02-19T21:22:49.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:49.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:49.016340+0000) 2024-02-19T21:22:50.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:50.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:50.016517+0000) 2024-02-19T21:22:51.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:51.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:51.016692+0000) 2024-02-19T21:22:51.119 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:22:51.122 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:22:52.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:52.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:52.016939+0000) 2024-02-19T21:22:53.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:53.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:53.017127+0000) 2024-02-19T21:22:54.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:54.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:54.017346+0000) 2024-02-19T21:22:55.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:55.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:55.017532+0000) 2024-02-19T21:22:56.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:56.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:56.017693+0000) 2024-02-19T21:22:57.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:57.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:57.017923+0000) 2024-02-19T21:22:58.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:58.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:58.018149+0000) 2024-02-19T21:22:59.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:22:59.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:22:59.018385+0000) 2024-02-19T21:23:00.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:00.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:00.018615+0000) 2024-02-19T21:23:01.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:01.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:01.018845+0000) 2024-02-19T21:23:02.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:02.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:02.019034+0000) 2024-02-19T21:23:03.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:03.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:03.019210+0000) 2024-02-19T21:23:04.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:04.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:04.019481+0000) 2024-02-19T21:23:05.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:05.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:05.019661+0000) 2024-02-19T21:23:06.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:06.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:06.019842+0000) 2024-02-19T21:23:07.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:07.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:07.019998+0000) 2024-02-19T21:23:08.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:08.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:08.020276+0000) 2024-02-19T21:23:09.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:09.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:09.020549+0000) 2024-02-19T21:23:10.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:10.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:10.020771+0000) 2024-02-19T21:23:11.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:11.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:11.020996+0000) 2024-02-19T21:23:12.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:12.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:12.021255+0000) 2024-02-19T21:23:13.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:13.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:13.021471+0000) 2024-02-19T21:23:14.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:14.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:14.021641+0000) 2024-02-19T21:23:15.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:15.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:15.021870+0000) 2024-02-19T21:23:16.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:16.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:16.022010+0000) 2024-02-19T21:23:17.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:17.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:17.022171+0000) 2024-02-19T21:23:18.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:18.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:18.022333+0000) 2024-02-19T21:23:19.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:19.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:19.022552+0000) 2024-02-19T21:23:20.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:20.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:20.022835+0000) 2024-02-19T21:23:21.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:21.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:21.023025+0000) 2024-02-19T21:23:21.177 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:23:21.180 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:23:22.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:22.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:22.023250+0000) 2024-02-19T21:23:23.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:23.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:23.023415+0000) 2024-02-19T21:23:24.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:24.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:24.023640+0000) 2024-02-19T21:23:25.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:25.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:25.023843+0000) 2024-02-19T21:23:26.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:26.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:26.024019+0000) 2024-02-19T21:23:27.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:27.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:27.024263+0000) 2024-02-19T21:23:28.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:28.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:28.024519+0000) 2024-02-19T21:23:29.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:29.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:29.024718+0000) 2024-02-19T21:23:30.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:30.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:30.024943+0000) 2024-02-19T21:23:31.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:31.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:31.025189+0000) 2024-02-19T21:23:32.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:32.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:32.025386+0000) 2024-02-19T21:23:33.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:33.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:33.025592+0000) 2024-02-19T21:23:34.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:34.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:34.025847+0000) 2024-02-19T21:23:35.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:35.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:35.026044+0000) 2024-02-19T21:23:36.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:36.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:36.026301+0000) 2024-02-19T21:23:37.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:37.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:37.026528+0000) 2024-02-19T21:23:38.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:38.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:38.026827+0000) 2024-02-19T21:23:39.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:39.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:39.027020+0000) 2024-02-19T21:23:40.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:40.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:40.027187+0000) 2024-02-19T21:23:41.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:41.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:41.027412+0000) 2024-02-19T21:23:42.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:42.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:42.027635+0000) 2024-02-19T21:23:43.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:43.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:43.027837+0000) 2024-02-19T21:23:44.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:44.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:44.028018+0000) 2024-02-19T21:23:45.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:45.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:45.028249+0000) 2024-02-19T21:23:46.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:46.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:46.028506+0000) 2024-02-19T21:23:47.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:47.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:47.028673+0000) 2024-02-19T21:23:48.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:48.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:48.028896+0000) 2024-02-19T21:23:49.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:49.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:49.029069+0000) 2024-02-19T21:23:50.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:50.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:50.029243+0000) 2024-02-19T21:23:51.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:51.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:51.029466+0000) 2024-02-19T21:23:51.247 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:23:51.250 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:23:52.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:52.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:52.029743+0000) 2024-02-19T21:23:53.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:53.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:53.029949+0000) 2024-02-19T21:23:54.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:54.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:54.030145+0000) 2024-02-19T21:23:55.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:55.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:55.030396+0000) 2024-02-19T21:23:56.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:56.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:56.030636+0000) 2024-02-19T21:23:57.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:57.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:57.030820+0000) 2024-02-19T21:23:58.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:58.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:58.031058+0000) 2024-02-19T21:23:59.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:23:59.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:23:59.031289+0000) 2024-02-19T21:24:00.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:00.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:00.031479+0000) 2024-02-19T21:24:01.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:01.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:01.031690+0000) 2024-02-19T21:24:02.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:02.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:02.031870+0000) 2024-02-19T21:24:03.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:03.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:03.032064+0000) 2024-02-19T21:24:04.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:04.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:04.032257+0000) 2024-02-19T21:24:05.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:05.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:05.032420+0000) 2024-02-19T21:24:06.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:06.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:06.032684+0000) 2024-02-19T21:24:07.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:07.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:07.032957+0000) 2024-02-19T21:24:08.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:08.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:08.033154+0000) 2024-02-19T21:24:09.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:09.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:09.033304+0000) 2024-02-19T21:24:10.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:10.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:10.033568+0000) 2024-02-19T21:24:11.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:11.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:11.033807+0000) 2024-02-19T21:24:12.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:12.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:12.034005+0000) 2024-02-19T21:24:13.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:13.033+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:13.034179+0000) 2024-02-19T21:24:14.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:14.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:14.034390+0000) 2024-02-19T21:24:15.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:15.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:15.034607+0000) 2024-02-19T21:24:16.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:16.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:16.034838+0000) 2024-02-19T21:24:17.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:17.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:17.035078+0000) 2024-02-19T21:24:18.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:18.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:18.035290+0000) 2024-02-19T21:24:19.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:19.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:19.035544+0000) 2024-02-19T21:24:20.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:20.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:20.035799+0000) 2024-02-19T21:24:21.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:21.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:21.036032+0000) 2024-02-19T21:24:21.369 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:24:21.372 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:24:22.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:22.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:22.036261+0000) 2024-02-19T21:24:23.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:23.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:23.036435+0000) 2024-02-19T21:24:24.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:24.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:24.036641+0000) 2024-02-19T21:24:25.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:25.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:25.036857+0000) 2024-02-19T21:24:26.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:26.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:26.037089+0000) 2024-02-19T21:24:27.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:27.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:27.037344+0000) 2024-02-19T21:24:28.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:28.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:28.037544+0000) 2024-02-19T21:24:29.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:29.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:29.037741+0000) 2024-02-19T21:24:30.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:30.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:30.037987+0000) 2024-02-19T21:24:31.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:31.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:31.038222+0000) 2024-02-19T21:24:32.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:32.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:32.038460+0000) 2024-02-19T21:24:33.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:33.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:33.038749+0000) 2024-02-19T21:24:34.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:34.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:34.038941+0000) 2024-02-19T21:24:35.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:35.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:35.039084+0000) 2024-02-19T21:24:36.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:36.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:36.039343+0000) 2024-02-19T21:24:37.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:37.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:37.039612+0000) 2024-02-19T21:24:38.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:38.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:38.039839+0000) 2024-02-19T21:24:39.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:39.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:39.040082+0000) 2024-02-19T21:24:40.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:40.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:40.040325+0000) 2024-02-19T21:24:41.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:41.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:41.040494+0000) 2024-02-19T21:24:42.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:42.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:42.040739+0000) 2024-02-19T21:24:43.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:43.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:43.040978+0000) 2024-02-19T21:24:44.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:44.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:44.041229+0000) 2024-02-19T21:24:45.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:45.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:45.041416+0000) 2024-02-19T21:24:46.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:46.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:46.041623+0000) 2024-02-19T21:24:47.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:47.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:47.041860+0000) 2024-02-19T21:24:48.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:48.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:48.042065+0000) 2024-02-19T21:24:49.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:49.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:49.042307+0000) 2024-02-19T21:24:50.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:50.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:50.042496+0000) 2024-02-19T21:24:51.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:51.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:51.042777+0000) 2024-02-19T21:24:51.458 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:24:51.461 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:24:52.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:52.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:52.042900+0000) 2024-02-19T21:24:53.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:53.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:53.043135+0000) 2024-02-19T21:24:54.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:54.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:54.043300+0000) 2024-02-19T21:24:55.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:55.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:55.043532+0000) 2024-02-19T21:24:56.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:56.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:56.043811+0000) 2024-02-19T21:24:57.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:57.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:57.044032+0000) 2024-02-19T21:24:58.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:58.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:58.044282+0000) 2024-02-19T21:24:59.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:24:59.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:24:59.044514+0000) 2024-02-19T21:25:00.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:00.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:00.044721+0000) 2024-02-19T21:25:01.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:01.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:01.044968+0000) 2024-02-19T21:25:02.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:02.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:02.045234+0000) 2024-02-19T21:25:03.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:03.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:03.045479+0000) 2024-02-19T21:25:04.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:04.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:04.045717+0000) 2024-02-19T21:25:05.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:05.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:05.045932+0000) 2024-02-19T21:25:06.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:06.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:06.046167+0000) 2024-02-19T21:25:07.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:07.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:07.046354+0000) 2024-02-19T21:25:08.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:08.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:08.046597+0000) 2024-02-19T21:25:09.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:09.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:09.046837+0000) 2024-02-19T21:25:10.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:10.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:10.047083+0000) 2024-02-19T21:25:11.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:11.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:11.047252+0000) 2024-02-19T21:25:12.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:12.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:12.047423+0000) 2024-02-19T21:25:13.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:13.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:13.047650+0000) 2024-02-19T21:25:14.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:14.044+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:14.047788+0000) 2024-02-19T21:25:15.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:15.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:15.048014+0000) 2024-02-19T21:25:16.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:16.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:16.048235+0000) 2024-02-19T21:25:17.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:17.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:17.048469+0000) 2024-02-19T21:25:18.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:18.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:18.048710+0000) 2024-02-19T21:25:19.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:19.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:19.048988+0000) 2024-02-19T21:25:20.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:20.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:20.049222+0000) 2024-02-19T21:25:21.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:21.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:21.049476+0000) 2024-02-19T21:25:21.589 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:25:21.593 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:25:22.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:22.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:22.049719+0000) 2024-02-19T21:25:23.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:23.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:23.049990+0000) 2024-02-19T21:25:24.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:24.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:24.050260+0000) 2024-02-19T21:25:25.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:25.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:25.050495+0000) 2024-02-19T21:25:26.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:26.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:26.050693+0000) 2024-02-19T21:25:27.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:27.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:27.050953+0000) 2024-02-19T21:25:28.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:28.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:28.051186+0000) 2024-02-19T21:25:29.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:29.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:29.051460+0000) 2024-02-19T21:25:30.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:30.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:30.051708+0000) 2024-02-19T21:25:31.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:31.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:31.051944+0000) 2024-02-19T21:25:32.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:32.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:32.052218+0000) 2024-02-19T21:25:33.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:33.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:33.052460+0000) 2024-02-19T21:25:34.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:34.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:34.052681+0000) 2024-02-19T21:25:35.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:35.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:35.052907+0000) 2024-02-19T21:25:36.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:36.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:36.053166+0000) 2024-02-19T21:25:37.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:37.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:37.053405+0000) 2024-02-19T21:25:38.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:38.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:38.053652+0000) 2024-02-19T21:25:39.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:39.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:39.053902+0000) 2024-02-19T21:25:40.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:40.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:40.054110+0000) 2024-02-19T21:25:41.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:41.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:41.054300+0000) 2024-02-19T21:25:42.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:42.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:42.054571+0000) 2024-02-19T21:25:43.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:43.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:43.054972+0000) 2024-02-19T21:25:44.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:44.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:44.055212+0000) 2024-02-19T21:25:45.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:45.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:45.055454+0000) 2024-02-19T21:25:46.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:46.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:46.055674+0000) 2024-02-19T21:25:47.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:47.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:47.055896+0000) 2024-02-19T21:25:48.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:48.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:48.056142+0000) 2024-02-19T21:25:49.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:49.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:49.056385+0000) 2024-02-19T21:25:50.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:50.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:50.056567+0000) 2024-02-19T21:25:51.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:51.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:51.056789+0000) 2024-02-19T21:25:51.648 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:25:51.652 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:25:52.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:52.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:52.057011+0000) 2024-02-19T21:25:53.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:53.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:53.057177+0000) 2024-02-19T21:25:54.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:54.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:54.057430+0000) 2024-02-19T21:25:55.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:55.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:55.057683+0000) 2024-02-19T21:25:56.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:56.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:56.057855+0000) 2024-02-19T21:25:57.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:57.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:57.058048+0000) 2024-02-19T21:25:58.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:58.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:58.058294+0000) 2024-02-19T21:25:59.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:25:59.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:25:59.058483+0000) 2024-02-19T21:26:00.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:00.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:00.058675+0000) 2024-02-19T21:26:01.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:01.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:01.058968+0000) 2024-02-19T21:26:02.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:02.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:02.059203+0000) 2024-02-19T21:26:03.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:03.059+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:03.059405+0000) 2024-02-19T21:26:04.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:04.059+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:04.059677+0000) 2024-02-19T21:26:05.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:05.059+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:05.059913+0000) 2024-02-19T21:26:06.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:06.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:06.060092+0000) 2024-02-19T21:26:07.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:07.059+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:07.060274+0000) 2024-02-19T21:26:08.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:08.059+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:08.060529+0000) 2024-02-19T21:26:09.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:09.059+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:09.060745+0000) 2024-02-19T21:26:10.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:10.059+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:10.060993+0000) 2024-02-19T21:26:11.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:11.059+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:11.061187+0000) 2024-02-19T21:26:12.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:12.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:12.061414+0000) 2024-02-19T21:26:13.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:13.059+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:13.061562+0000) 2024-02-19T21:26:14.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:14.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:14.061821+0000) 2024-02-19T21:26:15.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:15.059+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:15.062012+0000) 2024-02-19T21:26:16.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:16.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:16.062239+0000) 2024-02-19T21:26:17.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:17.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:17.062419+0000) 2024-02-19T21:26:18.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:18.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:18.062642+0000) 2024-02-19T21:26:19.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:19.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:19.062817+0000) 2024-02-19T21:26:20.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:20.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:20.063065+0000) 2024-02-19T21:26:21.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:21.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:21.063302+0000) 2024-02-19T21:26:21.707 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:26:21.710 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:26:22.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:22.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:22.063484+0000) 2024-02-19T21:26:23.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:23.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:23.063739+0000) 2024-02-19T21:26:24.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:24.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:24.063915+0000) 2024-02-19T21:26:25.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:25.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:25.064091+0000) 2024-02-19T21:26:26.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:26.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:26.064269+0000) 2024-02-19T21:26:27.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:27.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:27.064528+0000) 2024-02-19T21:26:28.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:28.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:28.064722+0000) 2024-02-19T21:26:29.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:29.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:29.064880+0000) 2024-02-19T21:26:30.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:30.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:30.065105+0000) 2024-02-19T21:26:31.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:31.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:31.065346+0000) 2024-02-19T21:26:32.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:32.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:32.065585+0000) 2024-02-19T21:26:33.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:33.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:33.065773+0000) 2024-02-19T21:26:34.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:34.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:34.066049+0000) 2024-02-19T21:26:35.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:35.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:35.066221+0000) 2024-02-19T21:26:36.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:36.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:36.066459+0000) 2024-02-19T21:26:37.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:37.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:37.066693+0000) 2024-02-19T21:26:38.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:38.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:38.066969+0000) 2024-02-19T21:26:39.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:39.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:39.067158+0000) 2024-02-19T21:26:40.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:40.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:40.067399+0000) 2024-02-19T21:26:41.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:41.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:41.067629+0000) 2024-02-19T21:26:42.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:42.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:42.067854+0000) 2024-02-19T21:26:43.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:43.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:43.068096+0000) 2024-02-19T21:26:44.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:44.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:44.068287+0000) 2024-02-19T21:26:45.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:45.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:45.068494+0000) 2024-02-19T21:26:46.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:46.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:46.068659+0000) 2024-02-19T21:26:47.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:47.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:47.068906+0000) 2024-02-19T21:26:48.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:48.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:48.069110+0000) 2024-02-19T21:26:49.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:49.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:49.069350+0000) 2024-02-19T21:26:50.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:50.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:50.069531+0000) 2024-02-19T21:26:51.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:51.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:51.069712+0000) 2024-02-19T21:26:51.772 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:26:51.776 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:26:52.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:52.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:52.069993+0000) 2024-02-19T21:26:53.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:53.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:53.070268+0000) 2024-02-19T21:26:54.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:54.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:54.070505+0000) 2024-02-19T21:26:55.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:55.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:55.070758+0000) 2024-02-19T21:26:56.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:56.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:56.070988+0000) 2024-02-19T21:26:57.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:57.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:57.071208+0000) 2024-02-19T21:26:58.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:58.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:58.071480+0000) 2024-02-19T21:26:59.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:26:59.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:26:59.071655+0000) 2024-02-19T21:27:00.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:00.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:00.071888+0000) 2024-02-19T21:27:01.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:01.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:01.072092+0000) 2024-02-19T21:27:02.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:02.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:02.072297+0000) 2024-02-19T21:27:03.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:03.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:03.072567+0000) 2024-02-19T21:27:04.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:04.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:04.072804+0000) 2024-02-19T21:27:05.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:05.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:05.073064+0000) 2024-02-19T21:27:06.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:06.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:06.073303+0000) 2024-02-19T21:27:07.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:07.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:07.073536+0000) 2024-02-19T21:27:08.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:08.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:08.073735+0000) 2024-02-19T21:27:09.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:09.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:09.073964+0000) 2024-02-19T21:27:10.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:10.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:10.074216+0000) 2024-02-19T21:27:11.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:11.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:11.074446+0000) 2024-02-19T21:27:12.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:12.070+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:12.074675+0000) 2024-02-19T21:27:13.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:13.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:13.074933+0000) 2024-02-19T21:27:14.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:14.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:14.075105+0000) 2024-02-19T21:27:15.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:15.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:15.075290+0000) 2024-02-19T21:27:16.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:16.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:16.075522+0000) 2024-02-19T21:27:17.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:17.074+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:17.075816+0000) 2024-02-19T21:27:18.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:18.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:18.076055+0000) 2024-02-19T21:27:19.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:19.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:19.076387+0000) 2024-02-19T21:27:20.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:20.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:20.076563+0000) 2024-02-19T21:27:21.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:21.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:21.076761+0000) 2024-02-19T21:27:21.903 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:27:21.906 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:27:22.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:22.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:22.077011+0000) 2024-02-19T21:27:23.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:23.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:23.077278+0000) 2024-02-19T21:27:24.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:24.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:24.077552+0000) 2024-02-19T21:27:25.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:25.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:25.077822+0000) 2024-02-19T21:27:26.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:26.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:26.078085+0000) 2024-02-19T21:27:27.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:27.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:27.078325+0000) 2024-02-19T21:27:28.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:28.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:28.078554+0000) 2024-02-19T21:27:29.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:29.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:29.078786+0000) 2024-02-19T21:27:30.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:30.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:30.078908+0000) 2024-02-19T21:27:31.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:31.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:31.079049+0000) 2024-02-19T21:27:32.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:32.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:32.079246+0000) 2024-02-19T21:27:33.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:33.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:33.079518+0000) 2024-02-19T21:27:34.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:34.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:34.079731+0000) 2024-02-19T21:27:35.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:35.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:35.079948+0000) 2024-02-19T21:27:36.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:36.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:36.080176+0000) 2024-02-19T21:27:37.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:37.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:37.080408+0000) 2024-02-19T21:27:38.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:38.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:38.080653+0000) 2024-02-19T21:27:39.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:39.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:39.080931+0000) 2024-02-19T21:27:40.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:40.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:40.081216+0000) 2024-02-19T21:27:41.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:41.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:41.081506+0000) 2024-02-19T21:27:42.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:42.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:42.081789+0000) 2024-02-19T21:27:43.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:43.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:43.082075+0000) 2024-02-19T21:27:44.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:44.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:44.082356+0000) 2024-02-19T21:27:45.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:45.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:45.082551+0000) 2024-02-19T21:27:46.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:46.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:46.082777+0000) 2024-02-19T21:27:47.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:47.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:47.082965+0000) 2024-02-19T21:27:48.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:48.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:48.083156+0000) 2024-02-19T21:27:49.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:49.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:49.083422+0000) 2024-02-19T21:27:50.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:50.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:50.083609+0000) 2024-02-19T21:27:51.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:51.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:51.083840+0000) 2024-02-19T21:27:52.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:52.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:52.084072+0000) 2024-02-19T21:27:52.108 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:27:52.111 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:27:53.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:53.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:53.084323+0000) 2024-02-19T21:27:54.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:54.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:54.084613+0000) 2024-02-19T21:27:55.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:55.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:55.084890+0000) 2024-02-19T21:27:56.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:56.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:56.085120+0000) 2024-02-19T21:27:57.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:57.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:57.085348+0000) 2024-02-19T21:27:58.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:58.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:58.085524+0000) 2024-02-19T21:27:59.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:27:59.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:27:59.085805+0000) 2024-02-19T21:28:00.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:00.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:00.086123+0000) 2024-02-19T21:28:01.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:01.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:01.086387+0000) 2024-02-19T21:28:02.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:02.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:02.086646+0000) 2024-02-19T21:28:03.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:03.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:03.086868+0000) 2024-02-19T21:28:04.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:04.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:04.087118+0000) 2024-02-19T21:28:05.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:05.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:05.087334+0000) 2024-02-19T21:28:06.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:06.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:06.087658+0000) 2024-02-19T21:28:07.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:07.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:07.087928+0000) 2024-02-19T21:28:08.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:08.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:08.088193+0000) 2024-02-19T21:28:09.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:09.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:09.088497+0000) 2024-02-19T21:28:10.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:10.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:10.088676+0000) 2024-02-19T21:28:11.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:11.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:11.088878+0000) 2024-02-19T21:28:12.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:12.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:12.089141+0000) 2024-02-19T21:28:13.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:13.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:13.089327+0000) 2024-02-19T21:28:14.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:14.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:14.089527+0000) 2024-02-19T21:28:15.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:15.089+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:15.089785+0000) 2024-02-19T21:28:16.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:16.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:16.090096+0000) 2024-02-19T21:28:17.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:17.089+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:17.090317+0000) 2024-02-19T21:28:18.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:18.085+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:18.090549+0000) 2024-02-19T21:28:19.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:19.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:19.090772+0000) 2024-02-19T21:28:20.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:20.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:20.090902+0000) 2024-02-19T21:28:21.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:21.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:21.091152+0000) 2024-02-19T21:28:22.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:22.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:22.091385+0000) 2024-02-19T21:28:22.197 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:28:22.200 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:28:23.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:23.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:23.091614+0000) 2024-02-19T21:28:24.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:24.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:24.091819+0000) 2024-02-19T21:28:25.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:25.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:25.092040+0000) 2024-02-19T21:28:26.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:26.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:26.092336+0000) 2024-02-19T21:28:27.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:27.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:27.092629+0000) 2024-02-19T21:28:28.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:28.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:28.092918+0000) 2024-02-19T21:28:29.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:29.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:29.093131+0000) 2024-02-19T21:28:30.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:30.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:30.093316+0000) 2024-02-19T21:28:31.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:31.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:31.093588+0000) 2024-02-19T21:28:32.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:32.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:32.093822+0000) 2024-02-19T21:28:33.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:33.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:33.094093+0000) 2024-02-19T21:28:34.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:34.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:34.094370+0000) 2024-02-19T21:28:35.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:35.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:35.094576+0000) 2024-02-19T21:28:36.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:36.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:36.094852+0000) 2024-02-19T21:28:37.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:37.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:37.095003+0000) 2024-02-19T21:28:38.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:38.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:38.095139+0000) 2024-02-19T21:28:39.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:39.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:39.095421+0000) 2024-02-19T21:28:40.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:40.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:40.095624+0000) 2024-02-19T21:28:41.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:41.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:41.095901+0000) 2024-02-19T21:28:42.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:42.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:42.096185+0000) 2024-02-19T21:28:43.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:43.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:43.096472+0000) 2024-02-19T21:28:44.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:44.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:44.096792+0000) 2024-02-19T21:28:45.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:45.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:45.097003+0000) 2024-02-19T21:28:46.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:46.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:46.097181+0000) 2024-02-19T21:28:47.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:47.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:47.097459+0000) 2024-02-19T21:28:48.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:48.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:48.097713+0000) 2024-02-19T21:28:49.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:49.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:49.097960+0000) 2024-02-19T21:28:50.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:50.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:50.098161+0000) 2024-02-19T21:28:51.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:51.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:51.098354+0000) 2024-02-19T21:28:52.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:52.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:52.098613+0000) 2024-02-19T21:28:52.270 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:28:52.273 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:28:53.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:53.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:53.098922+0000) 2024-02-19T21:28:54.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:54.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:54.099231+0000) 2024-02-19T21:28:55.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:55.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:55.099523+0000) 2024-02-19T21:28:56.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:56.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:56.099808+0000) 2024-02-19T21:28:57.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:57.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:57.100144+0000) 2024-02-19T21:28:58.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:58.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:58.100347+0000) 2024-02-19T21:28:59.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:28:59.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:28:59.100614+0000) 2024-02-19T21:29:00.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:00.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:00.100864+0000) 2024-02-19T21:29:01.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:01.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:01.101058+0000) 2024-02-19T21:29:02.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:02.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:02.101287+0000) 2024-02-19T21:29:03.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:03.096+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:03.101530+0000) 2024-02-19T21:29:04.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:04.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:04.101768+0000) 2024-02-19T21:29:05.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:05.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:05.101930+0000) 2024-02-19T21:29:06.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:06.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:06.102188+0000) 2024-02-19T21:29:07.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:07.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:07.102433+0000) 2024-02-19T21:29:08.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:08.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:08.102660+0000) 2024-02-19T21:29:09.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:09.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:09.102884+0000) 2024-02-19T21:29:10.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:10.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:10.103043+0000) 2024-02-19T21:29:11.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:11.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:11.103274+0000) 2024-02-19T21:29:12.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:12.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:12.103486+0000) 2024-02-19T21:29:13.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:13.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:13.103691+0000) 2024-02-19T21:29:14.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:14.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:14.103919+0000) 2024-02-19T21:29:15.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:15.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:15.104161+0000) 2024-02-19T21:29:16.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:16.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:16.104423+0000) 2024-02-19T21:29:17.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:17.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:17.104672+0000) 2024-02-19T21:29:18.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:18.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:18.104927+0000) 2024-02-19T21:29:19.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:19.100+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:19.105107+0000) 2024-02-19T21:29:20.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:20.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:20.105237+0000) 2024-02-19T21:29:21.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:21.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:21.105482+0000) 2024-02-19T21:29:22.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:22.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:22.105718+0000) 2024-02-19T21:29:22.342 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:29:22.346 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:29:23.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:23.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:23.105905+0000) 2024-02-19T21:29:24.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:24.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:24.106133+0000) 2024-02-19T21:29:25.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:25.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:25.106380+0000) 2024-02-19T21:29:26.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:26.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:26.106588+0000) 2024-02-19T21:29:27.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:27.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:27.106828+0000) 2024-02-19T21:29:28.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:28.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:28.107027+0000) 2024-02-19T21:29:29.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:29.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:29.107235+0000) 2024-02-19T21:29:30.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:30.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:30.107455+0000) 2024-02-19T21:29:31.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:31.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:31.107694+0000) 2024-02-19T21:29:32.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:32.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:32.107891+0000) 2024-02-19T21:29:33.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:33.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:33.108107+0000) 2024-02-19T21:29:34.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:34.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:34.108349+0000) 2024-02-19T21:29:35.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:35.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:35.108533+0000) 2024-02-19T21:29:36.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:36.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:36.108750+0000) 2024-02-19T21:29:37.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:37.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:37.108982+0000) 2024-02-19T21:29:38.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:38.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:38.109222+0000) 2024-02-19T21:29:39.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:39.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:39.109432+0000) 2024-02-19T21:29:40.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:40.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:40.109616+0000) 2024-02-19T21:29:41.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:41.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:41.109848+0000) 2024-02-19T21:29:42.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:42.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:42.110045+0000) 2024-02-19T21:29:43.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:43.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:43.110269+0000) 2024-02-19T21:29:44.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:44.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:44.110511+0000) 2024-02-19T21:29:45.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:45.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:45.110799+0000) 2024-02-19T21:29:46.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:46.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:46.111073+0000) 2024-02-19T21:29:47.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:47.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:47.111248+0000) 2024-02-19T21:29:48.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:48.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:48.111426+0000) 2024-02-19T21:29:49.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:49.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:49.111678+0000) 2024-02-19T21:29:50.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:50.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:50.111952+0000) 2024-02-19T21:29:51.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:51.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:51.112149+0000) 2024-02-19T21:29:52.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:52.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:52.112403+0000) 2024-02-19T21:29:52.473 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:29:52.476 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:29:53.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:53.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:53.112657+0000) 2024-02-19T21:29:54.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:54.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:54.112910+0000) 2024-02-19T21:29:55.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:55.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:55.113074+0000) 2024-02-19T21:29:56.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:56.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:56.113329+0000) 2024-02-19T21:29:57.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:57.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:57.113608+0000) 2024-02-19T21:29:58.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:58.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:58.113913+0000) 2024-02-19T21:29:59.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:29:59.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:29:59.114185+0000) 2024-02-19T21:30:00.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:00.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:00.114447+0000) 2024-02-19T21:30:01.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:01.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:01.114774+0000) 2024-02-19T21:30:02.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:02.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:02.114925+0000) 2024-02-19T21:30:03.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:03.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:03.115096+0000) 2024-02-19T21:30:04.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:04.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:04.115243+0000) 2024-02-19T21:30:05.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:05.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:05.115428+0000) 2024-02-19T21:30:06.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:06.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:06.115684+0000) 2024-02-19T21:30:07.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:07.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:07.115919+0000) 2024-02-19T21:30:08.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:08.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:08.116118+0000) 2024-02-19T21:30:09.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:09.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:09.116369+0000) 2024-02-19T21:30:10.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:10.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:10.116614+0000) 2024-02-19T21:30:11.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:11.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:11.116889+0000) 2024-02-19T21:30:12.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:12.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:12.117165+0000) 2024-02-19T21:30:13.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:13.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:13.117395+0000) 2024-02-19T21:30:14.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:14.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:14.117622+0000) 2024-02-19T21:30:15.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:15.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:15.117797+0000) 2024-02-19T21:30:16.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:16.111+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:16.118037+0000) 2024-02-19T21:30:17.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:17.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:17.118285+0000) 2024-02-19T21:30:18.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:18.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:18.118527+0000) 2024-02-19T21:30:19.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:19.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:19.118836+0000) 2024-02-19T21:30:20.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:20.115+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:20.119062+0000) 2024-02-19T21:30:21.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:21.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:21.119347+0000) 2024-02-19T21:30:22.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:22.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:22.119604+0000) 2024-02-19T21:30:22.533 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:30:22.537 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:30:23.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:23.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:23.119858+0000) 2024-02-19T21:30:24.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:24.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:24.120060+0000) 2024-02-19T21:30:25.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:25.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:25.120295+0000) 2024-02-19T21:30:26.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:26.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:26.120487+0000) 2024-02-19T21:30:27.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:27.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:27.120714+0000) 2024-02-19T21:30:28.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:28.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:28.120924+0000) 2024-02-19T21:30:29.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:29.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:29.121187+0000) 2024-02-19T21:30:30.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:30.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:30.121438+0000) 2024-02-19T21:30:31.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:31.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:31.121655+0000) 2024-02-19T21:30:32.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:32.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:32.121919+0000) 2024-02-19T21:30:33.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:33.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:33.122131+0000) 2024-02-19T21:30:34.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:34.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:34.122381+0000) 2024-02-19T21:30:35.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:35.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:35.122610+0000) 2024-02-19T21:30:36.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:36.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:36.122860+0000) 2024-02-19T21:30:37.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:37.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:37.123124+0000) 2024-02-19T21:30:38.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:38.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:38.123361+0000) 2024-02-19T21:30:39.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:39.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:39.123636+0000) 2024-02-19T21:30:40.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:40.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:40.123903+0000) 2024-02-19T21:30:41.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:41.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:41.124196+0000) 2024-02-19T21:30:42.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:42.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:42.124437+0000) 2024-02-19T21:30:43.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:43.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:43.124645+0000) 2024-02-19T21:30:44.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:44.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:44.124896+0000) 2024-02-19T21:30:45.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:45.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:45.125060+0000) 2024-02-19T21:30:46.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:46.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:46.125304+0000) 2024-02-19T21:30:47.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:47.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:47.125543+0000) 2024-02-19T21:30:48.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:48.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:48.125706+0000) 2024-02-19T21:30:49.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:49.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:49.125904+0000) 2024-02-19T21:30:50.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:50.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:50.126150+0000) 2024-02-19T21:30:51.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:51.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:51.126428+0000) 2024-02-19T21:30:52.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:52.122+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:52.126674+0000) 2024-02-19T21:30:52.642 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:30:52.646 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:30:53.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:53.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:53.126851+0000) 2024-02-19T21:30:54.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:54.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:54.127131+0000) 2024-02-19T21:30:55.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:55.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:55.127296+0000) 2024-02-19T21:30:56.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:56.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:56.127462+0000) 2024-02-19T21:30:57.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:57.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:57.127694+0000) 2024-02-19T21:30:58.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:58.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:58.127904+0000) 2024-02-19T21:30:59.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:30:59.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:30:59.128158+0000) 2024-02-19T21:31:00.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:00.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:00.128414+0000) 2024-02-19T21:31:01.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:01.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:01.128598+0000) 2024-02-19T21:31:02.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:02.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:02.128781+0000) 2024-02-19T21:31:03.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:03.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:03.128972+0000) 2024-02-19T21:31:04.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:04.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:04.129245+0000) 2024-02-19T21:31:05.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:05.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:05.129467+0000) 2024-02-19T21:31:06.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:06.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:06.129706+0000) 2024-02-19T21:31:07.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:07.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:07.129877+0000) 2024-02-19T21:31:08.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:08.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:08.130125+0000) 2024-02-19T21:31:09.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:09.126+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:09.130375+0000) 2024-02-19T21:31:10.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:10.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:10.130628+0000) 2024-02-19T21:31:11.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:11.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:11.130863+0000) 2024-02-19T21:31:12.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:12.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:12.131060+0000) 2024-02-19T21:31:13.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:13.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:13.131294+0000) 2024-02-19T21:31:14.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:14.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:14.131551+0000) 2024-02-19T21:31:15.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:15.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:15.131728+0000) 2024-02-19T21:31:16.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:16.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:16.131915+0000) 2024-02-19T21:31:17.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:17.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:17.132183+0000) 2024-02-19T21:31:18.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:18.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:18.132362+0000) 2024-02-19T21:31:19.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:19.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:19.132679+0000) 2024-02-19T21:31:20.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:20.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:20.132954+0000) 2024-02-19T21:31:21.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:21.130+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:21.133204+0000) 2024-02-19T21:31:22.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:22.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:22.133435+0000) 2024-02-19T21:31:22.773 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:31:22.776 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:31:23.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:23.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:23.133717+0000) 2024-02-19T21:31:24.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:24.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:24.133943+0000) 2024-02-19T21:31:25.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:25.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:25.134113+0000) 2024-02-19T21:31:26.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:26.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:26.134399+0000) 2024-02-19T21:31:27.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:27.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:27.134760+0000) 2024-02-19T21:31:28.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:28.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:28.134948+0000) 2024-02-19T21:31:29.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:29.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:29.135283+0000) 2024-02-19T21:31:30.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:30.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:30.135459+0000) 2024-02-19T21:31:31.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:31.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:31.135628+0000) 2024-02-19T21:31:32.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:32.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:32.135804+0000) 2024-02-19T21:31:33.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:33.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:33.135970+0000) 2024-02-19T21:31:34.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:34.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:34.136220+0000) 2024-02-19T21:31:35.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:35.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:35.136484+0000) 2024-02-19T21:31:36.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:36.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:36.136686+0000) 2024-02-19T21:31:37.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:37.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:37.136978+0000) 2024-02-19T21:31:38.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:38.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:38.137290+0000) 2024-02-19T21:31:39.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:39.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:39.137594+0000) 2024-02-19T21:31:40.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:40.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:40.137797+0000) 2024-02-19T21:31:41.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:41.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:41.138101+0000) 2024-02-19T21:31:42.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:42.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:42.138380+0000) 2024-02-19T21:31:43.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:43.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:43.138627+0000) 2024-02-19T21:31:44.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:44.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:44.138851+0000) 2024-02-19T21:31:45.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:45.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:45.138964+0000) 2024-02-19T21:31:46.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:46.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:46.139174+0000) 2024-02-19T21:31:47.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:47.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:47.139422+0000) 2024-02-19T21:31:48.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:48.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:48.139694+0000) 2024-02-19T21:31:49.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:49.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:49.139950+0000) 2024-02-19T21:31:50.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:50.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:50.140133+0000) 2024-02-19T21:31:51.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:51.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:51.140296+0000) 2024-02-19T21:31:52.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:52.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:52.140523+0000) 2024-02-19T21:31:52.903 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:31:52.908 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:31:53.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:53.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:53.140747+0000) 2024-02-19T21:31:54.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:54.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:54.140988+0000) 2024-02-19T21:31:55.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:55.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:55.141227+0000) 2024-02-19T21:31:56.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:56.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:56.141468+0000) 2024-02-19T21:31:57.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:57.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:57.141642+0000) 2024-02-19T21:31:58.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:58.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:58.141886+0000) 2024-02-19T21:31:59.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:31:59.137+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:31:59.142127+0000) 2024-02-19T21:32:00.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:00.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:00.142355+0000) 2024-02-19T21:32:01.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:01.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:01.142615+0000) 2024-02-19T21:32:02.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:02.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:02.142838+0000) 2024-02-19T21:32:03.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:03.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:03.143051+0000) 2024-02-19T21:32:04.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:04.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:04.143255+0000) 2024-02-19T21:32:05.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:05.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:05.143496+0000) 2024-02-19T21:32:06.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:06.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:06.143706+0000) 2024-02-19T21:32:07.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:07.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:07.143982+0000) 2024-02-19T21:32:08.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:08.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:08.144215+0000) 2024-02-19T21:32:09.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:09.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:09.144464+0000) 2024-02-19T21:32:10.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:10.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:10.144716+0000) 2024-02-19T21:32:11.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:11.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:11.144912+0000) 2024-02-19T21:32:12.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:12.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:12.145123+0000) 2024-02-19T21:32:13.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:13.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:13.145334+0000) 2024-02-19T21:32:14.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:14.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:14.145581+0000) 2024-02-19T21:32:15.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:15.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:15.145764+0000) 2024-02-19T21:32:16.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:16.141+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:16.145939+0000) 2024-02-19T21:32:17.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:17.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:17.146121+0000) 2024-02-19T21:32:18.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:18.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:18.146353+0000) 2024-02-19T21:32:19.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:19.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:19.146598+0000) 2024-02-19T21:32:20.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:20.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:20.146810+0000) 2024-02-19T21:32:21.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:21.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:21.147043+0000) 2024-02-19T21:32:22.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:22.145+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:22.147251+0000) 2024-02-19T21:32:23.043 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:32:23.046 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:32:23.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:23.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:23.147542+0000) 2024-02-19T21:32:24.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:24.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:24.147798+0000) 2024-02-19T21:32:25.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:25.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:25.148050+0000) 2024-02-19T21:32:26.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:26.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:26.148302+0000) 2024-02-19T21:32:27.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:27.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:27.148588+0000) 2024-02-19T21:32:28.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:28.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:28.148863+0000) 2024-02-19T21:32:29.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:29.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:29.149129+0000) 2024-02-19T21:32:30.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:30.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:30.149395+0000) 2024-02-19T21:32:31.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:31.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:31.149562+0000) 2024-02-19T21:32:32.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:32.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:32.149790+0000) 2024-02-19T21:32:33.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:33.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:33.150032+0000) 2024-02-19T21:32:34.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:34.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:34.150232+0000) 2024-02-19T21:32:35.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:35.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:35.150452+0000) 2024-02-19T21:32:36.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:36.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:36.150647+0000) 2024-02-19T21:32:37.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:37.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:37.150842+0000) 2024-02-19T21:32:38.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:38.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:38.151037+0000) 2024-02-19T21:32:39.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:39.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:39.151208+0000) 2024-02-19T21:32:40.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:40.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:40.151431+0000) 2024-02-19T21:32:41.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:41.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:41.151703+0000) 2024-02-19T21:32:42.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:42.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:42.151940+0000) 2024-02-19T21:32:43.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:43.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:43.152134+0000) 2024-02-19T21:32:44.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:44.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:44.152379+0000) 2024-02-19T21:32:45.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:45.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:45.152618+0000) 2024-02-19T21:32:46.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:46.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:46.152807+0000) 2024-02-19T21:32:47.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:47.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:47.152972+0000) 2024-02-19T21:32:48.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:48.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:48.153208+0000) 2024-02-19T21:32:49.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:49.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:49.153496+0000) 2024-02-19T21:32:50.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:50.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:50.153731+0000) 2024-02-19T21:32:51.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:51.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:51.153912+0000) 2024-02-19T21:32:52.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:52.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:52.154134+0000) 2024-02-19T21:32:53.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:53.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:53.154339+0000) 2024-02-19T21:32:53.172 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:32:53.177 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:32:54.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:54.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:54.154597+0000) 2024-02-19T21:32:55.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:55.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:55.154870+0000) 2024-02-19T21:32:56.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:56.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:56.155065+0000) 2024-02-19T21:32:57.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:57.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:57.155290+0000) 2024-02-19T21:32:58.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:58.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:58.155555+0000) 2024-02-19T21:32:59.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:32:59.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:32:59.155875+0000) 2024-02-19T21:33:00.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:00.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:00.156138+0000) 2024-02-19T21:33:01.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:01.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:01.156416+0000) 2024-02-19T21:33:02.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:02.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:02.156645+0000) 2024-02-19T21:33:03.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:03.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:03.156889+0000) 2024-02-19T21:33:04.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:04.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:04.157142+0000) 2024-02-19T21:33:05.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:05.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:05.157369+0000) 2024-02-19T21:33:06.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:06.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:06.157573+0000) 2024-02-19T21:33:07.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:07.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:07.157778+0000) 2024-02-19T21:33:08.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:08.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:08.157991+0000) 2024-02-19T21:33:09.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:09.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:09.158267+0000) 2024-02-19T21:33:10.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:10.152+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:10.158469+0000) 2024-02-19T21:33:11.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:11.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:11.158823+0000) 2024-02-19T21:33:12.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:12.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:12.159125+0000) 2024-02-19T21:33:13.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:13.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:13.159381+0000) 2024-02-19T21:33:14.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:14.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:14.159577+0000) 2024-02-19T21:33:15.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:15.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:15.159779+0000) 2024-02-19T21:33:16.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:16.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:16.160065+0000) 2024-02-19T21:33:17.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:17.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:17.160391+0000) 2024-02-19T21:33:18.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:18.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:18.160641+0000) 2024-02-19T21:33:19.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:19.156+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:19.160935+0000) 2024-02-19T21:33:20.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:20.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:20.161221+0000) 2024-02-19T21:33:21.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:21.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:21.161477+0000) 2024-02-19T21:33:22.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:22.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:22.161715+0000) 2024-02-19T21:33:23.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:23.160+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:23.161897+0000) 2024-02-19T21:33:23.277 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:33:23.280 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:33:24.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:24.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:24.162120+0000) 2024-02-19T21:33:25.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:25.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:25.162305+0000) 2024-02-19T21:33:26.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:26.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:26.162540+0000) 2024-02-19T21:33:27.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:27.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:27.162791+0000) 2024-02-19T21:33:28.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:28.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:28.162998+0000) 2024-02-19T21:33:29.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:29.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:29.163236+0000) 2024-02-19T21:33:30.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:30.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:30.163410+0000) 2024-02-19T21:33:31.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:31.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:31.163657+0000) 2024-02-19T21:33:32.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:32.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:32.163921+0000) 2024-02-19T21:33:33.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:33.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:33.164196+0000) 2024-02-19T21:33:34.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:34.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:34.164426+0000) 2024-02-19T21:33:35.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:35.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:35.164597+0000) 2024-02-19T21:33:36.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:36.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:36.164837+0000) 2024-02-19T21:33:37.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:37.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:37.165078+0000) 2024-02-19T21:33:38.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:38.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:38.165289+0000) 2024-02-19T21:33:39.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:39.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:39.165500+0000) 2024-02-19T21:33:40.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:40.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:40.165775+0000) 2024-02-19T21:33:41.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:41.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:41.165946+0000) 2024-02-19T21:33:42.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:42.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:42.166214+0000) 2024-02-19T21:33:43.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:43.159+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:43.166476+0000) 2024-02-19T21:33:44.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:44.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:44.166777+0000) 2024-02-19T21:33:45.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:45.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:45.166932+0000) 2024-02-19T21:33:46.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:46.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:46.167159+0000) 2024-02-19T21:33:47.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:47.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:47.167395+0000) 2024-02-19T21:33:48.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:48.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:48.167660+0000) 2024-02-19T21:33:49.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:49.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:49.167921+0000) 2024-02-19T21:33:50.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:50.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:50.168138+0000) 2024-02-19T21:33:51.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:51.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:51.168309+0000) 2024-02-19T21:33:52.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:52.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:52.168536+0000) 2024-02-19T21:33:53.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:53.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:53.168743+0000) 2024-02-19T21:33:53.358 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:33:53.362 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:33:54.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:54.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:54.168966+0000) 2024-02-19T21:33:55.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:55.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:55.169177+0000) 2024-02-19T21:33:56.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:56.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:56.169352+0000) 2024-02-19T21:33:57.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:57.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:57.169619+0000) 2024-02-19T21:33:58.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:58.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:58.169893+0000) 2024-02-19T21:33:59.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:33:59.163+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:33:59.170178+0000) 2024-02-19T21:34:00.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:00.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:00.170403+0000) 2024-02-19T21:34:01.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:01.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:01.170606+0000) 2024-02-19T21:34:02.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:02.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:02.170879+0000) 2024-02-19T21:34:03.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:03.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:03.171083+0000) 2024-02-19T21:34:04.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:04.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:04.171282+0000) 2024-02-19T21:34:05.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:05.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:05.171490+0000) 2024-02-19T21:34:06.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:06.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:06.171770+0000) 2024-02-19T21:34:07.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:07.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:07.172040+0000) 2024-02-19T21:34:08.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:08.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:08.172214+0000) 2024-02-19T21:34:09.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:09.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:09.172411+0000) 2024-02-19T21:34:10.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:10.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:10.172618+0000) 2024-02-19T21:34:11.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:11.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:11.172792+0000) 2024-02-19T21:34:12.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:12.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:12.172970+0000) 2024-02-19T21:34:13.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:13.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:13.173229+0000) 2024-02-19T21:34:14.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:14.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:14.173438+0000) 2024-02-19T21:34:15.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:15.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:15.173652+0000) 2024-02-19T21:34:16.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:16.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:16.173918+0000) 2024-02-19T21:34:17.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:17.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:17.174140+0000) 2024-02-19T21:34:18.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:18.167+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:18.174390+0000) 2024-02-19T21:34:19.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:19.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:19.174659+0000) 2024-02-19T21:34:20.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:20.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:20.174922+0000) 2024-02-19T21:34:21.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:21.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:21.175120+0000) 2024-02-19T21:34:22.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:22.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:22.175330+0000) 2024-02-19T21:34:23.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:23.171+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:23.175490+0000) 2024-02-19T21:34:23.423 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:34:23.427 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:34:24.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:24.175+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:24.175745+0000) 2024-02-19T21:34:25.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:25.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:25.175973+0000) 2024-02-19T21:34:26.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:26.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:26.176167+0000) 2024-02-19T21:34:27.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:27.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:27.176365+0000) 2024-02-19T21:34:28.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:28.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:28.176573+0000) 2024-02-19T21:34:29.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:29.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:29.176770+0000) 2024-02-19T21:34:30.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:30.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:30.177034+0000) 2024-02-19T21:34:31.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:31.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:31.177318+0000) 2024-02-19T21:34:32.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:32.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:32.177597+0000) 2024-02-19T21:34:33.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:33.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:33.177800+0000) 2024-02-19T21:34:34.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:34.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:34.177996+0000) 2024-02-19T21:34:35.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:35.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:35.178174+0000) 2024-02-19T21:34:36.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:36.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:36.178435+0000) 2024-02-19T21:34:37.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:37.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:37.178727+0000) 2024-02-19T21:34:38.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:38.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:38.179013+0000) 2024-02-19T21:34:39.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:39.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:39.179330+0000) 2024-02-19T21:34:40.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:40.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:40.179616+0000) 2024-02-19T21:34:41.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:41.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:41.179884+0000) 2024-02-19T21:34:42.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:42.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:42.180149+0000) 2024-02-19T21:34:43.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:43.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:43.180434+0000) 2024-02-19T21:34:44.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:44.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:44.180713+0000) 2024-02-19T21:34:45.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:45.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:45.180994+0000) 2024-02-19T21:34:46.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:46.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:46.181215+0000) 2024-02-19T21:34:47.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:47.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:47.181503+0000) 2024-02-19T21:34:48.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:48.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:48.181824+0000) 2024-02-19T21:34:49.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:49.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:49.182117+0000) 2024-02-19T21:34:50.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:50.178+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:50.182398+0000) 2024-02-19T21:34:51.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:51.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:51.182764+0000) 2024-02-19T21:34:52.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:52.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:52.183084+0000) 2024-02-19T21:34:53.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:53.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:53.183398+0000) 2024-02-19T21:34:53.603 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:34:53.606 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:34:54.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:54.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:54.183642+0000) 2024-02-19T21:34:55.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:55.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:55.183879+0000) 2024-02-19T21:34:56.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:56.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:56.184140+0000) 2024-02-19T21:34:57.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:57.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:57.184380+0000) 2024-02-19T21:34:58.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:58.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:58.184623+0000) 2024-02-19T21:34:59.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:34:59.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:34:59.184912+0000) 2024-02-19T21:35:00.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:00.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:00.185191+0000) 2024-02-19T21:35:01.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:01.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:01.185477+0000) 2024-02-19T21:35:02.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:02.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:02.185734+0000) 2024-02-19T21:35:03.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:03.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:03.185980+0000) 2024-02-19T21:35:04.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:04.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:04.186161+0000) 2024-02-19T21:35:05.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:05.182+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:05.186429+0000) 2024-02-19T21:35:06.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:06.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:06.186725+0000) 2024-02-19T21:35:07.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:07.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:07.186964+0000) 2024-02-19T21:35:08.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:08.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:08.187182+0000) 2024-02-19T21:35:09.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:09.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:09.187431+0000) 2024-02-19T21:35:10.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:10.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:10.187670+0000) 2024-02-19T21:35:11.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:11.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:11.187826+0000) 2024-02-19T21:35:12.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:12.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:12.188103+0000) 2024-02-19T21:35:13.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:13.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:13.188331+0000) 2024-02-19T21:35:14.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:14.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:14.188593+0000) 2024-02-19T21:35:15.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:15.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:15.188883+0000) 2024-02-19T21:35:16.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:16.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:16.189140+0000) 2024-02-19T21:35:17.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:17.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:17.189422+0000) 2024-02-19T21:35:18.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:18.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:18.189703+0000) 2024-02-19T21:35:19.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:19.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:19.189917+0000) 2024-02-19T21:35:20.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:20.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:20.190185+0000) 2024-02-19T21:35:21.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:21.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:21.190455+0000) 2024-02-19T21:35:22.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:22.186+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:22.190656+0000) 2024-02-19T21:35:23.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:23.190+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:23.190938+0000) 2024-02-19T21:35:23.683 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:35:23.686 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:35:24.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:24.190+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:24.191180+0000) 2024-02-19T21:35:25.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:25.190+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:25.191462+0000) 2024-02-19T21:35:26.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:26.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:26.191709+0000) 2024-02-19T21:35:27.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:27.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:27.191943+0000) 2024-02-19T21:35:28.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:28.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:28.192176+0000) 2024-02-19T21:35:29.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:29.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:29.192378+0000) 2024-02-19T21:35:30.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:30.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:30.192688+0000) 2024-02-19T21:35:31.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:31.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:31.192973+0000) 2024-02-19T21:35:32.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:32.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:32.193153+0000) 2024-02-19T21:35:33.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:33.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:33.193419+0000) 2024-02-19T21:35:34.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:34.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:34.193754+0000) 2024-02-19T21:35:35.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:35.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:35.193981+0000) 2024-02-19T21:35:36.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:36.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:36.194268+0000) 2024-02-19T21:35:37.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:37.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:37.194476+0000) 2024-02-19T21:35:38.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:38.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:38.194655+0000) 2024-02-19T21:35:39.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:39.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:39.194873+0000) 2024-02-19T21:35:40.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:40.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:40.195056+0000) 2024-02-19T21:35:41.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:41.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:41.195317+0000) 2024-02-19T21:35:42.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:42.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:42.195614+0000) 2024-02-19T21:35:43.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:43.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:43.195871+0000) 2024-02-19T21:35:44.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:44.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:44.196147+0000) 2024-02-19T21:35:45.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:45.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:45.196464+0000) 2024-02-19T21:35:46.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:46.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:46.196732+0000) 2024-02-19T21:35:47.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:47.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:47.197003+0000) 2024-02-19T21:35:48.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:48.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:48.197290+0000) 2024-02-19T21:35:49.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:49.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:49.197465+0000) 2024-02-19T21:35:50.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:50.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:50.197751+0000) 2024-02-19T21:35:51.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:51.193+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:51.198058+0000) 2024-02-19T21:35:52.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:52.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:52.198326+0000) 2024-02-19T21:35:53.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:53.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:53.198551+0000) 2024-02-19T21:35:53.796 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:35:53.800 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:35:54.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:54.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:54.198877+0000) 2024-02-19T21:35:55.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:55.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:55.199162+0000) 2024-02-19T21:35:56.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:56.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:56.199357+0000) 2024-02-19T21:35:57.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:57.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:57.199535+0000) 2024-02-19T21:35:58.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:58.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:58.199796+0000) 2024-02-19T21:35:59.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:35:59.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:35:59.200063+0000) 2024-02-19T21:36:00.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:00.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:00.200338+0000) 2024-02-19T21:36:01.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:01.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:01.200604+0000) 2024-02-19T21:36:02.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:02.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:02.200898+0000) 2024-02-19T21:36:03.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:03.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:03.201125+0000) 2024-02-19T21:36:04.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:04.197+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:04.201390+0000) 2024-02-19T21:36:05.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:05.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:05.201661+0000) 2024-02-19T21:36:06.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:06.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:06.201842+0000) 2024-02-19T21:36:07.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:07.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:07.202027+0000) 2024-02-19T21:36:08.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:08.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:08.202302+0000) 2024-02-19T21:36:09.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:09.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:09.202517+0000) 2024-02-19T21:36:10.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:10.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:10.202753+0000) 2024-02-19T21:36:11.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:11.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:11.203021+0000) 2024-02-19T21:36:12.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:12.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:12.203229+0000) 2024-02-19T21:36:13.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:13.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:13.203428+0000) 2024-02-19T21:36:14.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:14.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:14.203607+0000) 2024-02-19T21:36:15.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:15.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:15.203884+0000) 2024-02-19T21:36:16.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:16.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:16.204172+0000) 2024-02-19T21:36:17.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:17.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:17.204415+0000) 2024-02-19T21:36:18.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:18.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:18.204675+0000) 2024-02-19T21:36:19.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:19.201+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:19.205005+0000) 2024-02-19T21:36:20.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:20.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:20.205265+0000) 2024-02-19T21:36:21.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:21.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:21.205528+0000) 2024-02-19T21:36:22.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:22.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:22.205843+0000) 2024-02-19T21:36:23.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:23.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:23.206078+0000) 2024-02-19T21:36:23.852 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:36:23.855 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:36:24.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:24.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:24.206292+0000) 2024-02-19T21:36:25.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:25.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:25.206468+0000) 2024-02-19T21:36:26.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:26.205+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:26.206792+0000) 2024-02-19T21:36:27.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:27.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:27.207076+0000) 2024-02-19T21:36:28.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:28.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:28.207366+0000) 2024-02-19T21:36:29.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:29.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:29.207575+0000) 2024-02-19T21:36:30.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:30.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:30.207736+0000) 2024-02-19T21:36:31.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:31.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:31.208018+0000) 2024-02-19T21:36:32.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:32.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:32.208314+0000) 2024-02-19T21:36:33.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:33.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:33.208783+0000) 2024-02-19T21:36:34.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:34.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:34.209016+0000) 2024-02-19T21:36:35.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:35.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:35.209283+0000) 2024-02-19T21:36:36.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:36.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:36.209562+0000) 2024-02-19T21:36:37.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:37.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:37.209846+0000) 2024-02-19T21:36:38.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:38.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:38.210166+0000) 2024-02-19T21:36:39.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:39.204+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:39.210440+0000) 2024-02-19T21:36:40.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:40.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:40.210670+0000) 2024-02-19T21:36:41.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:41.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:41.210948+0000) 2024-02-19T21:36:42.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:42.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:42.211220+0000) 2024-02-19T21:36:43.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:43.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:43.211485+0000) 2024-02-19T21:36:44.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:44.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:44.211674+0000) 2024-02-19T21:36:45.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:45.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:45.211862+0000) 2024-02-19T21:36:46.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:46.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:46.212143+0000) 2024-02-19T21:36:47.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:47.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:47.212431+0000) 2024-02-19T21:36:48.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:48.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:48.212632+0000) 2024-02-19T21:36:49.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:49.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:49.212840+0000) 2024-02-19T21:36:50.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:50.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:50.213012+0000) 2024-02-19T21:36:51.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:51.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:51.213277+0000) 2024-02-19T21:36:52.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:52.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:52.213534+0000) 2024-02-19T21:36:53.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:53.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:53.213741+0000) 2024-02-19T21:36:53.990 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:36:53.993 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:36:54.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:54.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:54.213998+0000) 2024-02-19T21:36:55.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:55.208+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:55.214163+0000) 2024-02-19T21:36:56.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:56.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:56.214416+0000) 2024-02-19T21:36:57.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:57.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:57.214703+0000) 2024-02-19T21:36:58.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:58.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:58.214996+0000) 2024-02-19T21:36:59.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:36:59.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:36:59.215296+0000) 2024-02-19T21:37:00.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:00.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:00.215489+0000) 2024-02-19T21:37:01.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:01.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:01.215628+0000) 2024-02-19T21:37:02.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:02.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:02.215800+0000) 2024-02-19T21:37:03.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:03.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:03.216002+0000) 2024-02-19T21:37:04.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:04.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:04.216207+0000) 2024-02-19T21:37:05.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:05.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:05.216511+0000) 2024-02-19T21:37:06.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:06.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:06.216772+0000) 2024-02-19T21:37:07.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:07.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:07.216949+0000) 2024-02-19T21:37:08.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:08.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:08.217138+0000) 2024-02-19T21:37:09.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:09.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:09.217364+0000) 2024-02-19T21:37:10.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:10.212+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:10.217575+0000) 2024-02-19T21:37:11.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:11.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:11.217835+0000) 2024-02-19T21:37:12.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:12.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:12.218124+0000) 2024-02-19T21:37:13.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:13.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:13.218355+0000) 2024-02-19T21:37:14.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:14.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:14.218532+0000) 2024-02-19T21:37:15.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:15.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:15.218837+0000) 2024-02-19T21:37:16.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:16.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:16.219069+0000) 2024-02-19T21:37:17.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:17.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:17.219349+0000) 2024-02-19T21:37:18.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:18.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:18.219608+0000) 2024-02-19T21:37:19.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:19.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:19.219907+0000) 2024-02-19T21:37:20.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:20.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:20.220166+0000) 2024-02-19T21:37:21.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:21.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:21.220510+0000) 2024-02-19T21:37:22.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:22.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:22.220777+0000) 2024-02-19T21:37:23.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:23.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:23.221071+0000) 2024-02-19T21:37:24.113 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:37:24.118 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:37:24.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:24.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:24.221352+0000) 2024-02-19T21:37:25.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:25.216+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:25.221560+0000) 2024-02-19T21:37:26.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:26.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:26.221849+0000) 2024-02-19T21:37:27.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:27.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:27.222170+0000) 2024-02-19T21:37:28.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:28.215+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:28.222460+0000) 2024-02-19T21:37:29.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:29.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:29.222747+0000) 2024-02-19T21:37:30.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:30.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:30.222968+0000) 2024-02-19T21:37:31.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:31.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:31.223172+0000) 2024-02-19T21:37:32.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:32.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:32.223373+0000) 2024-02-19T21:37:33.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:33.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:33.223607+0000) 2024-02-19T21:37:34.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:34.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:34.223745+0000) 2024-02-19T21:37:35.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:35.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:35.224004+0000) 2024-02-19T21:37:36.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:36.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:36.224228+0000) 2024-02-19T21:37:37.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:37.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:37.224502+0000) 2024-02-19T21:37:38.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:38.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:38.224737+0000) 2024-02-19T21:37:39.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:39.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:39.225024+0000) 2024-02-19T21:37:40.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:40.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:40.225233+0000) 2024-02-19T21:37:41.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:41.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:41.225432+0000) 2024-02-19T21:37:42.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:42.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:42.225609+0000) 2024-02-19T21:37:43.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:43.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:43.225781+0000) 2024-02-19T21:37:44.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:44.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:44.225923+0000) 2024-02-19T21:37:45.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:45.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:45.226127+0000) 2024-02-19T21:37:46.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:46.223+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:46.226392+0000) 2024-02-19T21:37:47.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:47.219+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:47.226635+0000) 2024-02-19T21:37:48.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:48.223+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:48.226938+0000) 2024-02-19T21:37:49.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:49.223+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:49.227137+0000) 2024-02-19T21:37:50.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:50.223+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:50.227419+0000) 2024-02-19T21:37:51.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:51.223+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:51.227717+0000) 2024-02-19T21:37:52.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:52.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:52.228011+0000) 2024-02-19T21:37:53.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:53.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:53.228215+0000) 2024-02-19T21:37:54.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:54.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:54.228449+0000) 2024-02-19T21:37:54.234 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:37:54.237 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:37:55.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:55.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:55.228655+0000) 2024-02-19T21:37:56.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:56.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:56.228870+0000) 2024-02-19T21:37:57.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:57.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:57.229132+0000) 2024-02-19T21:37:58.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:58.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:58.229374+0000) 2024-02-19T21:37:59.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:37:59.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:37:59.229676+0000) 2024-02-19T21:38:00.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:00.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:00.229878+0000) 2024-02-19T21:38:01.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:01.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:01.230175+0000) 2024-02-19T21:38:02.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:02.223+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:02.230433+0000) 2024-02-19T21:38:03.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:03.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:03.230688+0000) 2024-02-19T21:38:04.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:04.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:04.230946+0000) 2024-02-19T21:38:05.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:05.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:05.231154+0000) 2024-02-19T21:38:06.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:06.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:06.231374+0000) 2024-02-19T21:38:07.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:07.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:07.231601+0000) 2024-02-19T21:38:08.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:08.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:08.231838+0000) 2024-02-19T21:38:09.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:09.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:09.232130+0000) 2024-02-19T21:38:10.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:10.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:10.232371+0000) 2024-02-19T21:38:11.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:11.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:11.232611+0000) 2024-02-19T21:38:12.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:12.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:12.232798+0000) 2024-02-19T21:38:13.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:13.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:13.233025+0000) 2024-02-19T21:38:14.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:14.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:14.233294+0000) 2024-02-19T21:38:15.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:15.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:15.233540+0000) 2024-02-19T21:38:16.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:16.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:16.233728+0000) 2024-02-19T21:38:17.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:17.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:17.233959+0000) 2024-02-19T21:38:18.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:18.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:18.234174+0000) 2024-02-19T21:38:19.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:19.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:19.234375+0000) 2024-02-19T21:38:20.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:20.227+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:20.234646+0000) 2024-02-19T21:38:21.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:21.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:21.234867+0000) 2024-02-19T21:38:22.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:22.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:22.235048+0000) 2024-02-19T21:38:23.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:23.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:23.235293+0000) 2024-02-19T21:38:24.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:24.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:24.235534+0000) 2024-02-19T21:38:24.356 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:38:24.360 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:38:25.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:25.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:25.235800+0000) 2024-02-19T21:38:26.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:26.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:26.235999+0000) 2024-02-19T21:38:27.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:27.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:27.236273+0000) 2024-02-19T21:38:28.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:28.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:28.236527+0000) 2024-02-19T21:38:29.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:29.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:29.236677+0000) 2024-02-19T21:38:30.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:30.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:30.236892+0000) 2024-02-19T21:38:31.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:31.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:31.237123+0000) 2024-02-19T21:38:32.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:32.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:32.237323+0000) 2024-02-19T21:38:33.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:33.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:33.237461+0000) 2024-02-19T21:38:34.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:34.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:34.237638+0000) 2024-02-19T21:38:35.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:35.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:35.237869+0000) 2024-02-19T21:38:36.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:36.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:36.238117+0000) 2024-02-19T21:38:37.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:37.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:37.238322+0000) 2024-02-19T21:38:38.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:38.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:38.238596+0000) 2024-02-19T21:38:39.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:39.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:39.238833+0000) 2024-02-19T21:38:40.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:40.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:40.239050+0000) 2024-02-19T21:38:41.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:41.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:41.239331+0000) 2024-02-19T21:38:42.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:42.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:42.239632+0000) 2024-02-19T21:38:43.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:43.234+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:43.239871+0000) 2024-02-19T21:38:44.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:44.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:44.240050+0000) 2024-02-19T21:38:45.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:45.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:45.240299+0000) 2024-02-19T21:38:46.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:46.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:46.240578+0000) 2024-02-19T21:38:47.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:47.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:47.240804+0000) 2024-02-19T21:38:48.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:48.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:48.241092+0000) 2024-02-19T21:38:49.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:49.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:49.241397+0000) 2024-02-19T21:38:50.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:50.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:50.241604+0000) 2024-02-19T21:38:51.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:51.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:51.241831+0000) 2024-02-19T21:38:52.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:52.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:52.242114+0000) 2024-02-19T21:38:53.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:53.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:53.242323+0000) 2024-02-19T21:38:54.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:54.238+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:54.242552+0000) 2024-02-19T21:38:54.429 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:38:54.432 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:38:55.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:55.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:55.242770+0000) 2024-02-19T21:38:56.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:56.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:56.243082+0000) 2024-02-19T21:38:57.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:57.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:57.243375+0000) 2024-02-19T21:38:58.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:58.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:58.243649+0000) 2024-02-19T21:38:59.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:38:59.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:38:59.243863+0000) 2024-02-19T21:39:00.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:00.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:00.244087+0000) 2024-02-19T21:39:01.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:01.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:01.244379+0000) 2024-02-19T21:39:02.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:02.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:02.244582+0000) 2024-02-19T21:39:03.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:03.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:03.244900+0000) 2024-02-19T21:39:04.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:04.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:04.245139+0000) 2024-02-19T21:39:05.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:05.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:05.245403+0000) 2024-02-19T21:39:06.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:06.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:06.245624+0000) 2024-02-19T21:39:07.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:07.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:07.245778+0000) 2024-02-19T21:39:08.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:08.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:08.245984+0000) 2024-02-19T21:39:09.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:09.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:09.246183+0000) 2024-02-19T21:39:10.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:10.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:10.246437+0000) 2024-02-19T21:39:11.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:11.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:11.246746+0000) 2024-02-19T21:39:12.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:12.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:12.247017+0000) 2024-02-19T21:39:13.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:13.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:13.247245+0000) 2024-02-19T21:39:14.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:14.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:14.247411+0000) 2024-02-19T21:39:15.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:15.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:15.247632+0000) 2024-02-19T21:39:16.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:16.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:16.247825+0000) 2024-02-19T21:39:17.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:17.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:17.248057+0000) 2024-02-19T21:39:18.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:18.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:18.248336+0000) 2024-02-19T21:39:19.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:19.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:19.248607+0000) 2024-02-19T21:39:20.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:20.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:20.248842+0000) 2024-02-19T21:39:21.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:21.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:21.248981+0000) 2024-02-19T21:39:22.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:22.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:22.249283+0000) 2024-02-19T21:39:23.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:23.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:23.249482+0000) 2024-02-19T21:39:24.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:24.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:24.249705+0000) 2024-02-19T21:39:24.542 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:39:24.545 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:39:25.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:25.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:25.249879+0000) 2024-02-19T21:39:26.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:26.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:26.250122+0000) 2024-02-19T21:39:27.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:27.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:27.250416+0000) 2024-02-19T21:39:28.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:28.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:28.250750+0000) 2024-02-19T21:39:29.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:29.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:29.251054+0000) 2024-02-19T21:39:30.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:30.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:30.251349+0000) 2024-02-19T21:39:31.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:31.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:31.251597+0000) 2024-02-19T21:39:32.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:32.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:32.251897+0000) 2024-02-19T21:39:33.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:33.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:33.252117+0000) 2024-02-19T21:39:34.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:34.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:34.252342+0000) 2024-02-19T21:39:35.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:35.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:35.252576+0000) 2024-02-19T21:39:36.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:36.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:36.252781+0000) 2024-02-19T21:39:37.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:37.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:37.253009+0000) 2024-02-19T21:39:38.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:38.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:38.253261+0000) 2024-02-19T21:39:39.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:39.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:39.253509+0000) 2024-02-19T21:39:40.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:40.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:40.253777+0000) 2024-02-19T21:39:41.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:41.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:41.254032+0000) 2024-02-19T21:39:42.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:42.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:42.254183+0000) 2024-02-19T21:39:43.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:43.249+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:43.254427+0000) 2024-02-19T21:39:44.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:44.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:44.254747+0000) 2024-02-19T21:39:45.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:45.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:45.254966+0000) 2024-02-19T21:39:46.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:46.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:46.255125+0000) 2024-02-19T21:39:47.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:47.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:47.255369+0000) 2024-02-19T21:39:48.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:48.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:48.255622+0000) 2024-02-19T21:39:49.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:49.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:49.255887+0000) 2024-02-19T21:39:50.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:50.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:50.256134+0000) 2024-02-19T21:39:51.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:51.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:51.256358+0000) 2024-02-19T21:39:52.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:52.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:52.256619+0000) 2024-02-19T21:39:53.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:53.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:53.256889+0000) 2024-02-19T21:39:54.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:54.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:54.257109+0000) 2024-02-19T21:39:54.663 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:39:54.668 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:39:55.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:55.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:55.257276+0000) 2024-02-19T21:39:56.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:56.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:56.257505+0000) 2024-02-19T21:39:57.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:57.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:57.257757+0000) 2024-02-19T21:39:58.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:58.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:58.258013+0000) 2024-02-19T21:39:59.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:39:59.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:39:59.258260+0000) 2024-02-19T21:40:00.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:00.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:00.258424+0000) 2024-02-19T21:40:01.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:01.253+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:01.258656+0000) 2024-02-19T21:40:02.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:02.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:02.258901+0000) 2024-02-19T21:40:03.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:03.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:03.259082+0000) 2024-02-19T21:40:04.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:04.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:04.259286+0000) 2024-02-19T21:40:05.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:05.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:05.259520+0000) 2024-02-19T21:40:06.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:06.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:06.259731+0000) 2024-02-19T21:40:07.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:07.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:07.259953+0000) 2024-02-19T21:40:08.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:08.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:08.260173+0000) 2024-02-19T21:40:09.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:09.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:09.260425+0000) 2024-02-19T21:40:10.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:10.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:10.260652+0000) 2024-02-19T21:40:11.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:11.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:11.260932+0000) 2024-02-19T21:40:12.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:12.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:12.261189+0000) 2024-02-19T21:40:13.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:13.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:13.261351+0000) 2024-02-19T21:40:14.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:14.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:14.261534+0000) 2024-02-19T21:40:15.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:15.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:15.261732+0000) 2024-02-19T21:40:16.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:16.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:16.261955+0000) 2024-02-19T21:40:17.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:17.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:17.262228+0000) 2024-02-19T21:40:18.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:18.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:18.262443+0000) 2024-02-19T21:40:19.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:19.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:19.262683+0000) 2024-02-19T21:40:20.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:20.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:20.262979+0000) 2024-02-19T21:40:21.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:21.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:21.263260+0000) 2024-02-19T21:40:22.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:22.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:22.263494+0000) 2024-02-19T21:40:23.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:23.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:23.263724+0000) 2024-02-19T21:40:24.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:24.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:24.263977+0000) 2024-02-19T21:40:24.729 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:40:24.733 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:40:25.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:25.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:25.264241+0000) 2024-02-19T21:40:26.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:26.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:26.264515+0000) 2024-02-19T21:40:27.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:27.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:27.264733+0000) 2024-02-19T21:40:28.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:28.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:28.264952+0000) 2024-02-19T21:40:29.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:29.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:29.265087+0000) 2024-02-19T21:40:30.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:30.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:30.265330+0000) 2024-02-19T21:40:31.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:31.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:31.265560+0000) 2024-02-19T21:40:32.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:32.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:32.265814+0000) 2024-02-19T21:40:33.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:33.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:33.266067+0000) 2024-02-19T21:40:34.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:34.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:34.266293+0000) 2024-02-19T21:40:35.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:35.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:35.266419+0000) 2024-02-19T21:40:36.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:36.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:36.266678+0000) 2024-02-19T21:40:37.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:37.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:37.266896+0000) 2024-02-19T21:40:38.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:38.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:38.267090+0000) 2024-02-19T21:40:39.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:39.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:39.267409+0000) 2024-02-19T21:40:40.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:40.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:40.267686+0000) 2024-02-19T21:40:41.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:41.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:41.267935+0000) 2024-02-19T21:40:42.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:42.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:42.268158+0000) 2024-02-19T21:40:43.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:43.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:43.268411+0000) 2024-02-19T21:40:44.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:44.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:44.268696+0000) 2024-02-19T21:40:45.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:45.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:45.268944+0000) 2024-02-19T21:40:46.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:46.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:46.269077+0000) 2024-02-19T21:40:47.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:47.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:47.269260+0000) 2024-02-19T21:40:48.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:48.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:48.269516+0000) 2024-02-19T21:40:49.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:49.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:49.269793+0000) 2024-02-19T21:40:50.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:50.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:50.270038+0000) 2024-02-19T21:40:51.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:51.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:51.270227+0000) 2024-02-19T21:40:52.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:52.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:52.270398+0000) 2024-02-19T21:40:53.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:53.264+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:53.270527+0000) 2024-02-19T21:40:54.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:54.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:54.270808+0000) 2024-02-19T21:40:54.867 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:40:54.870 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:40:55.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:55.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:55.271065+0000) 2024-02-19T21:40:56.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:56.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:56.271237+0000) 2024-02-19T21:40:57.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:57.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:57.271474+0000) 2024-02-19T21:40:58.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:58.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:58.271700+0000) 2024-02-19T21:40:59.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:40:59.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:40:59.271894+0000) 2024-02-19T21:41:00.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:00.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:00.272068+0000) 2024-02-19T21:41:01.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:01.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:01.272232+0000) 2024-02-19T21:41:02.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:02.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:02.272455+0000) 2024-02-19T21:41:03.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:03.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:03.272658+0000) 2024-02-19T21:41:04.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:04.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:04.272798+0000) 2024-02-19T21:41:05.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:05.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:05.273031+0000) 2024-02-19T21:41:06.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:06.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:06.273220+0000) 2024-02-19T21:41:07.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:07.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:07.273442+0000) 2024-02-19T21:41:08.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:08.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:08.273677+0000) 2024-02-19T21:41:09.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:09.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:09.273896+0000) 2024-02-19T21:41:10.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:10.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:10.274094+0000) 2024-02-19T21:41:11.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:11.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:11.274285+0000) 2024-02-19T21:41:12.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:12.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:12.274540+0000) 2024-02-19T21:41:13.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:13.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:13.274832+0000) 2024-02-19T21:41:14.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:14.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:14.274982+0000) 2024-02-19T21:41:15.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:15.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:15.275252+0000) 2024-02-19T21:41:16.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:16.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:16.275441+0000) 2024-02-19T21:41:17.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:17.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:17.275668+0000) 2024-02-19T21:41:18.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:18.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:18.275846+0000) 2024-02-19T21:41:19.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:19.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:19.276049+0000) 2024-02-19T21:41:20.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:20.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:20.276230+0000) 2024-02-19T21:41:21.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:21.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:21.276395+0000) 2024-02-19T21:41:22.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:22.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:22.276654+0000) 2024-02-19T21:41:23.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:23.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:23.276885+0000) 2024-02-19T21:41:24.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:24.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:24.277081+0000) 2024-02-19T21:41:24.996 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:41:25.000 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:41:25.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:25.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:25.277322+0000) 2024-02-19T21:41:26.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:26.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:26.277503+0000) 2024-02-19T21:41:27.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:27.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:27.277735+0000) 2024-02-19T21:41:28.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:28.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:28.277977+0000) 2024-02-19T21:41:29.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:29.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:29.278197+0000) 2024-02-19T21:41:30.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:30.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:30.278441+0000) 2024-02-19T21:41:31.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:31.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:31.278629+0000) 2024-02-19T21:41:32.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:32.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:32.278934+0000) 2024-02-19T21:41:33.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:33.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:33.279122+0000) 2024-02-19T21:41:34.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:34.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:34.279395+0000) 2024-02-19T21:41:35.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:35.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:35.279678+0000) 2024-02-19T21:41:36.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:36.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:36.279842+0000) 2024-02-19T21:41:37.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:37.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:37.280072+0000) 2024-02-19T21:41:38.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:38.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:38.280352+0000) 2024-02-19T21:41:39.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:39.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:39.280598+0000) 2024-02-19T21:41:40.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:40.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:40.280860+0000) 2024-02-19T21:41:41.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:41.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:41.281034+0000) 2024-02-19T21:41:42.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:42.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:42.281286+0000) 2024-02-19T21:41:43.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:43.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:43.281537+0000) 2024-02-19T21:41:44.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:44.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:44.281791+0000) 2024-02-19T21:41:45.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:45.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:45.281990+0000) 2024-02-19T21:41:46.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:46.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:46.282165+0000) 2024-02-19T21:41:47.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:47.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:47.282404+0000) 2024-02-19T21:41:48.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:48.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:48.282577+0000) 2024-02-19T21:41:49.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:49.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:49.282782+0000) 2024-02-19T21:41:50.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:50.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:50.282929+0000) 2024-02-19T21:41:51.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:51.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:51.283102+0000) 2024-02-19T21:41:52.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:52.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:52.283353+0000) 2024-02-19T21:41:53.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:53.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:53.283539+0000) 2024-02-19T21:41:54.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:54.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:54.283792+0000) 2024-02-19T21:41:55.127 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:41:55.131 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:41:55.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:55.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:55.284025+0000) 2024-02-19T21:41:56.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:56.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:56.284203+0000) 2024-02-19T21:41:57.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:57.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:57.284453+0000) 2024-02-19T21:41:58.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:58.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:58.284676+0000) 2024-02-19T21:41:59.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:41:59.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:41:59.284922+0000) 2024-02-19T21:42:00.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:00.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:00.285201+0000) 2024-02-19T21:42:01.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:01.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:01.285340+0000) 2024-02-19T21:42:02.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:02.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:02.285556+0000) 2024-02-19T21:42:03.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:03.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:03.285804+0000) 2024-02-19T21:42:04.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:04.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:04.286004+0000) 2024-02-19T21:42:05.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:05.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:05.286250+0000) 2024-02-19T21:42:06.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:06.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:06.286415+0000) 2024-02-19T21:42:07.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:07.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:07.286578+0000) 2024-02-19T21:42:08.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:08.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:08.286770+0000) 2024-02-19T21:42:09.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:09.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:09.286909+0000) 2024-02-19T21:42:10.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:10.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:10.287088+0000) 2024-02-19T21:42:11.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:11.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:11.287291+0000) 2024-02-19T21:42:12.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:12.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:12.287474+0000) 2024-02-19T21:42:13.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:13.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:13.287595+0000) 2024-02-19T21:42:14.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:14.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:14.287757+0000) 2024-02-19T21:42:15.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:15.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:15.287901+0000) 2024-02-19T21:42:16.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:16.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:16.288127+0000) 2024-02-19T21:42:17.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:17.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:17.288304+0000) 2024-02-19T21:42:18.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:18.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:18.288470+0000) 2024-02-19T21:42:19.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:19.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:19.288643+0000) 2024-02-19T21:42:20.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:20.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:20.288796+0000) 2024-02-19T21:42:21.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:21.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:21.288927+0000) 2024-02-19T21:42:22.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:22.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:22.289069+0000) 2024-02-19T21:42:23.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:23.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:23.289209+0000) 2024-02-19T21:42:24.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:24.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:24.289377+0000) 2024-02-19T21:42:25.249 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:42:25.253 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:42:25.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:25.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:25.289525+0000) 2024-02-19T21:42:26.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:26.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:26.289775+0000) 2024-02-19T21:42:27.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:27.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:27.290005+0000) 2024-02-19T21:42:28.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:28.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:28.290248+0000) 2024-02-19T21:42:29.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:29.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:29.290532+0000) 2024-02-19T21:42:30.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:30.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:30.290780+0000) 2024-02-19T21:42:31.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:31.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:31.290961+0000) 2024-02-19T21:42:32.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:32.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:32.291146+0000) 2024-02-19T21:42:33.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:33.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:33.291391+0000) 2024-02-19T21:42:34.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:34.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:34.291624+0000) 2024-02-19T21:42:35.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:35.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:35.291801+0000) 2024-02-19T21:42:36.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:36.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:36.292006+0000) 2024-02-19T21:42:37.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:37.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:37.292276+0000) 2024-02-19T21:42:38.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:38.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:38.292515+0000) 2024-02-19T21:42:39.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:39.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:39.292775+0000) 2024-02-19T21:42:40.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:40.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:40.293002+0000) 2024-02-19T21:42:41.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:41.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:41.293181+0000) 2024-02-19T21:42:42.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:42.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:42.293433+0000) 2024-02-19T21:42:43.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:43.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:43.293675+0000) 2024-02-19T21:42:44.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:44.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:44.293937+0000) 2024-02-19T21:42:45.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:45.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:45.294171+0000) 2024-02-19T21:42:46.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:46.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:46.294304+0000) 2024-02-19T21:42:47.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:47.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:47.294583+0000) 2024-02-19T21:42:48.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:48.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:48.294862+0000) 2024-02-19T21:42:49.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:49.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:49.295113+0000) 2024-02-19T21:42:50.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:50.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:50.295392+0000) 2024-02-19T21:42:51.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:51.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:51.295662+0000) 2024-02-19T21:42:52.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:52.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:52.295934+0000) 2024-02-19T21:42:53.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:53.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:53.296138+0000) 2024-02-19T21:42:54.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:54.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:54.296377+0000) 2024-02-19T21:42:55.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:55.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:55.296582+0000) 2024-02-19T21:42:55.371 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:42:55.376 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:42:56.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:56.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:56.296830+0000) 2024-02-19T21:42:57.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:57.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:57.297065+0000) 2024-02-19T21:42:58.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:58.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:58.297260+0000) 2024-02-19T21:42:59.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:42:59.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:42:59.297517+0000) 2024-02-19T21:43:00.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:00.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:00.297741+0000) 2024-02-19T21:43:01.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:01.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:01.297928+0000) 2024-02-19T21:43:02.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:02.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:02.298156+0000) 2024-02-19T21:43:03.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:03.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:03.298401+0000) 2024-02-19T21:43:04.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:04.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:04.298651+0000) 2024-02-19T21:43:05.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:05.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:05.298861+0000) 2024-02-19T21:43:06.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:06.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:06.299042+0000) 2024-02-19T21:43:07.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:07.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:07.299232+0000) 2024-02-19T21:43:08.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:08.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:08.299420+0000) 2024-02-19T21:43:09.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:09.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:09.299666+0000) 2024-02-19T21:43:10.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:10.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:10.299903+0000) 2024-02-19T21:43:11.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:11.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:11.300092+0000) 2024-02-19T21:43:12.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:12.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:12.300328+0000) 2024-02-19T21:43:13.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:13.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:13.300623+0000) 2024-02-19T21:43:14.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:14.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:14.300849+0000) 2024-02-19T21:43:15.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:15.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:15.301092+0000) 2024-02-19T21:43:16.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:16.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:16.301341+0000) 2024-02-19T21:43:17.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:17.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:17.301644+0000) 2024-02-19T21:43:18.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:18.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:18.301880+0000) 2024-02-19T21:43:19.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:19.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:19.302111+0000) 2024-02-19T21:43:20.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:20.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:20.302344+0000) 2024-02-19T21:43:21.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:21.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:21.302555+0000) 2024-02-19T21:43:22.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:22.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:22.302818+0000) 2024-02-19T21:43:23.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:23.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:23.303051+0000) 2024-02-19T21:43:24.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:24.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:24.303207+0000) 2024-02-19T21:43:25.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:25.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:25.303437+0000) 2024-02-19T21:43:25.434 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:43:25.437 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:43:26.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:26.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:26.303689+0000) 2024-02-19T21:43:27.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:27.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:27.303902+0000) 2024-02-19T21:43:28.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:28.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:28.304146+0000) 2024-02-19T21:43:29.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:29.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:29.304339+0000) 2024-02-19T21:43:30.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:30.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:30.304600+0000) 2024-02-19T21:43:31.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:31.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:31.304845+0000) 2024-02-19T21:43:32.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:32.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:32.305022+0000) 2024-02-19T21:43:33.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:33.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:33.305188+0000) 2024-02-19T21:43:34.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:34.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:34.305421+0000) 2024-02-19T21:43:35.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:35.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:35.305674+0000) 2024-02-19T21:43:36.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:36.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:36.305934+0000) 2024-02-19T21:43:37.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:37.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:37.306195+0000) 2024-02-19T21:43:38.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:38.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:38.306384+0000) 2024-02-19T21:43:39.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:39.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:39.306621+0000) 2024-02-19T21:43:40.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:40.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:40.306890+0000) 2024-02-19T21:43:41.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:41.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:41.307108+0000) 2024-02-19T21:43:42.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:42.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:42.307377+0000) 2024-02-19T21:43:43.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:43.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:43.307574+0000) 2024-02-19T21:43:44.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:44.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:44.307835+0000) 2024-02-19T21:43:45.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:45.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:45.308069+0000) 2024-02-19T21:43:46.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:46.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:46.308324+0000) 2024-02-19T21:43:47.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:47.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:47.308594+0000) 2024-02-19T21:43:48.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:48.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:48.308843+0000) 2024-02-19T21:43:49.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:49.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:49.309103+0000) 2024-02-19T21:43:50.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:50.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:50.309317+0000) 2024-02-19T21:43:51.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:51.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:51.309598+0000) 2024-02-19T21:43:52.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:52.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:52.309828+0000) 2024-02-19T21:43:53.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:53.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:53.310056+0000) 2024-02-19T21:43:54.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:54.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:54.310226+0000) 2024-02-19T21:43:55.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:55.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:55.310413+0000) 2024-02-19T21:43:55.517 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:43:55.520 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:43:56.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:56.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:56.310655+0000) 2024-02-19T21:43:57.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:57.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:57.310894+0000) 2024-02-19T21:43:58.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:58.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:58.311097+0000) 2024-02-19T21:43:59.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:43:59.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:43:59.311288+0000) 2024-02-19T21:44:00.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:00.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:00.311521+0000) 2024-02-19T21:44:01.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:01.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:01.311759+0000) 2024-02-19T21:44:02.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:02.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:02.311973+0000) 2024-02-19T21:44:03.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:03.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:03.312225+0000) 2024-02-19T21:44:04.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:04.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:04.312507+0000) 2024-02-19T21:44:05.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:05.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:05.312709+0000) 2024-02-19T21:44:06.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:06.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:06.312914+0000) 2024-02-19T21:44:07.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:07.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:07.313153+0000) 2024-02-19T21:44:08.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:08.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:08.313411+0000) 2024-02-19T21:44:09.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:09.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:09.313659+0000) 2024-02-19T21:44:10.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:10.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:10.313879+0000) 2024-02-19T21:44:11.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:11.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:11.314109+0000) 2024-02-19T21:44:12.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:12.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:12.314369+0000) 2024-02-19T21:44:13.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:13.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:13.314610+0000) 2024-02-19T21:44:14.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:14.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:14.314900+0000) 2024-02-19T21:44:15.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:15.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:15.315113+0000) 2024-02-19T21:44:16.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:16.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:16.315379+0000) 2024-02-19T21:44:17.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:17.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:17.315561+0000) 2024-02-19T21:44:18.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:18.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:18.315811+0000) 2024-02-19T21:44:19.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:19.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:19.316067+0000) 2024-02-19T21:44:20.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:20.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:20.316303+0000) 2024-02-19T21:44:21.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:21.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:21.316535+0000) 2024-02-19T21:44:22.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:22.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:22.316734+0000) 2024-02-19T21:44:23.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:23.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:23.316976+0000) 2024-02-19T21:44:24.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:24.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:24.317242+0000) 2024-02-19T21:44:25.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:25.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:25.317474+0000) 2024-02-19T21:44:25.578 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:44:25.582 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:44:26.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:26.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:26.317679+0000) 2024-02-19T21:44:27.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:27.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:27.317918+0000) 2024-02-19T21:44:28.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:28.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:28.318156+0000) 2024-02-19T21:44:29.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:29.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:29.318410+0000) 2024-02-19T21:44:30.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:30.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:30.318651+0000) 2024-02-19T21:44:31.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:31.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:31.318829+0000) 2024-02-19T21:44:32.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:32.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:32.319043+0000) 2024-02-19T21:44:33.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:33.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:33.319295+0000) 2024-02-19T21:44:34.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:34.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:34.319557+0000) 2024-02-19T21:44:35.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:35.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:35.319727+0000) 2024-02-19T21:44:36.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:36.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:36.319943+0000) 2024-02-19T21:44:37.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:37.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:37.320185+0000) 2024-02-19T21:44:38.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:38.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:38.320367+0000) 2024-02-19T21:44:39.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:39.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:39.320628+0000) 2024-02-19T21:44:40.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:40.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:40.320877+0000) 2024-02-19T21:44:41.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:41.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:41.321174+0000) 2024-02-19T21:44:42.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:42.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:42.321371+0000) 2024-02-19T21:44:43.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:43.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:43.321568+0000) 2024-02-19T21:44:44.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:44.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:44.321833+0000) 2024-02-19T21:44:45.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:45.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:45.322043+0000) 2024-02-19T21:44:46.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:46.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:46.322284+0000) 2024-02-19T21:44:47.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:47.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:47.322492+0000) 2024-02-19T21:44:48.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:48.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:48.322690+0000) 2024-02-19T21:44:49.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:49.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:49.322950+0000) 2024-02-19T21:44:50.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:50.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:50.323135+0000) 2024-02-19T21:44:51.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:51.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:51.323431+0000) 2024-02-19T21:44:52.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:52.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:52.323671+0000) 2024-02-19T21:44:53.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:53.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:53.323933+0000) 2024-02-19T21:44:54.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:54.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:54.324132+0000) 2024-02-19T21:44:55.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:55.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:55.324369+0000) 2024-02-19T21:44:55.694 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:44:55.698 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:44:56.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:56.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:56.324511+0000) 2024-02-19T21:44:57.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:57.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:57.324757+0000) 2024-02-19T21:44:58.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:58.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:58.324931+0000) 2024-02-19T21:44:59.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:44:59.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:44:59.325172+0000) 2024-02-19T21:45:00.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:00.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:00.325444+0000) 2024-02-19T21:45:01.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:01.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:01.325645+0000) 2024-02-19T21:45:02.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:02.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:02.325902+0000) 2024-02-19T21:45:03.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:03.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:03.326109+0000) 2024-02-19T21:45:04.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:04.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:04.326393+0000) 2024-02-19T21:45:05.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:05.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:05.326631+0000) 2024-02-19T21:45:06.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:06.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:06.326873+0000) 2024-02-19T21:45:07.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:07.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:07.327067+0000) 2024-02-19T21:45:08.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:08.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:08.327324+0000) 2024-02-19T21:45:09.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:09.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:09.327522+0000) 2024-02-19T21:45:10.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:10.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:10.327768+0000) 2024-02-19T21:45:11.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:11.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:11.327969+0000) 2024-02-19T21:45:12.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:12.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:12.328211+0000) 2024-02-19T21:45:13.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:13.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:13.328419+0000) 2024-02-19T21:45:14.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:14.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:14.328662+0000) 2024-02-19T21:45:15.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:15.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:15.328873+0000) 2024-02-19T21:45:16.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:16.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:16.329062+0000) 2024-02-19T21:45:17.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:17.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:17.329316+0000) 2024-02-19T21:45:18.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:18.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:18.329529+0000) 2024-02-19T21:45:19.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:19.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:19.329679+0000) 2024-02-19T21:45:20.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:20.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:20.329915+0000) 2024-02-19T21:45:21.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:21.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:21.330107+0000) 2024-02-19T21:45:22.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:22.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:22.330332+0000) 2024-02-19T21:45:23.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:23.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:23.330565+0000) 2024-02-19T21:45:24.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:24.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:24.330847+0000) 2024-02-19T21:45:25.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:25.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:25.331072+0000) 2024-02-19T21:45:25.749 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:45:25.752 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:45:26.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:26.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:26.331326+0000) 2024-02-19T21:45:27.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:27.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:27.331569+0000) 2024-02-19T21:45:28.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:28.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:28.331804+0000) 2024-02-19T21:45:29.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:29.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:29.332091+0000) 2024-02-19T21:45:30.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:30.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:30.332337+0000) 2024-02-19T21:45:31.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:31.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:31.332611+0000) 2024-02-19T21:45:32.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:32.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:32.332859+0000) 2024-02-19T21:45:33.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:33.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:33.333099+0000) 2024-02-19T21:45:34.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:34.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:34.333347+0000) 2024-02-19T21:45:35.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:35.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:35.333589+0000) 2024-02-19T21:45:36.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:36.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:36.333794+0000) 2024-02-19T21:45:37.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:37.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:37.334050+0000) 2024-02-19T21:45:38.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:38.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:38.334285+0000) 2024-02-19T21:45:39.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:39.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:39.334574+0000) 2024-02-19T21:45:40.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:40.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:40.334822+0000) 2024-02-19T21:45:41.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:41.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:41.335065+0000) 2024-02-19T21:45:42.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:42.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:42.335312+0000) 2024-02-19T21:45:43.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:43.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:43.335491+0000) 2024-02-19T21:45:44.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:44.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:44.335717+0000) 2024-02-19T21:45:45.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:45.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:45.335898+0000) 2024-02-19T21:45:46.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:46.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:46.336160+0000) 2024-02-19T21:45:47.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:47.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:47.336413+0000) 2024-02-19T21:45:48.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:48.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:48.336657+0000) 2024-02-19T21:45:49.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:49.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:49.336907+0000) 2024-02-19T21:45:50.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:50.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:50.337171+0000) 2024-02-19T21:45:51.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:51.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:51.337372+0000) 2024-02-19T21:45:52.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:52.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:52.337624+0000) 2024-02-19T21:45:53.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:53.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:53.337798+0000) 2024-02-19T21:45:54.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:54.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:54.338066+0000) 2024-02-19T21:45:55.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:55.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:55.338306+0000) 2024-02-19T21:45:55.805 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:45:55.810 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:45:56.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:56.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:56.338546+0000) 2024-02-19T21:45:57.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:57.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:57.338861+0000) 2024-02-19T21:45:58.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:58.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:58.339022+0000) 2024-02-19T21:45:59.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:45:59.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:45:59.339293+0000) 2024-02-19T21:46:00.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:00.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:00.339534+0000) 2024-02-19T21:46:01.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:01.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:01.339778+0000) 2024-02-19T21:46:02.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:02.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:02.339989+0000) 2024-02-19T21:46:03.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:03.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:03.340247+0000) 2024-02-19T21:46:04.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:04.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:04.340451+0000) 2024-02-19T21:46:05.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:05.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:05.340738+0000) 2024-02-19T21:46:06.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:06.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:06.340976+0000) 2024-02-19T21:46:07.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:07.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:07.341255+0000) 2024-02-19T21:46:08.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:08.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:08.341490+0000) 2024-02-19T21:46:09.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:09.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:09.341687+0000) 2024-02-19T21:46:10.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:10.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:10.341907+0000) 2024-02-19T21:46:11.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:11.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:11.342094+0000) 2024-02-19T21:46:12.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:12.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:12.342348+0000) 2024-02-19T21:46:13.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:13.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:13.342633+0000) 2024-02-19T21:46:14.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:14.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:14.342879+0000) 2024-02-19T21:46:15.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:15.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:15.343116+0000) 2024-02-19T21:46:16.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:16.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:16.343313+0000) 2024-02-19T21:46:17.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:17.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:17.343525+0000) 2024-02-19T21:46:18.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:18.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:18.343788+0000) 2024-02-19T21:46:19.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:19.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:19.344059+0000) 2024-02-19T21:46:20.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:20.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:20.344298+0000) 2024-02-19T21:46:21.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:21.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:21.344555+0000) 2024-02-19T21:46:22.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:22.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:22.344806+0000) 2024-02-19T21:46:23.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:23.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:23.345044+0000) 2024-02-19T21:46:24.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:24.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:24.345297+0000) 2024-02-19T21:46:25.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:25.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:25.345578+0000) 2024-02-19T21:46:25.870 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:46:25.873 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:46:26.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:26.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:26.345871+0000) 2024-02-19T21:46:27.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:27.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:27.346116+0000) 2024-02-19T21:46:28.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:28.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:28.346292+0000) 2024-02-19T21:46:29.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:29.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:29.346549+0000) 2024-02-19T21:46:30.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:30.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:30.346829+0000) 2024-02-19T21:46:31.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:31.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:31.347124+0000) 2024-02-19T21:46:32.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:32.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:32.347404+0000) 2024-02-19T21:46:33.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:33.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:33.347613+0000) 2024-02-19T21:46:34.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:34.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:34.347870+0000) 2024-02-19T21:46:35.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:35.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:35.348131+0000) 2024-02-19T21:46:36.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:36.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:36.348381+0000) 2024-02-19T21:46:37.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:37.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:37.348657+0000) 2024-02-19T21:46:38.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:38.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:38.348902+0000) 2024-02-19T21:46:39.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:39.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:39.349170+0000) 2024-02-19T21:46:40.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:40.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:40.349411+0000) 2024-02-19T21:46:41.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:41.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:41.349672+0000) 2024-02-19T21:46:42.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:42.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:42.349886+0000) 2024-02-19T21:46:43.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:43.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:43.350087+0000) 2024-02-19T21:46:44.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:44.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:44.350293+0000) 2024-02-19T21:46:45.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:45.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:45.350540+0000) 2024-02-19T21:46:46.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:46.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:46.350812+0000) 2024-02-19T21:46:47.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:47.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:47.351062+0000) 2024-02-19T21:46:48.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:48.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:48.351242+0000) 2024-02-19T21:46:49.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:49.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:49.351485+0000) 2024-02-19T21:46:50.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:50.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:50.351664+0000) 2024-02-19T21:46:51.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:51.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:51.351874+0000) 2024-02-19T21:46:52.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:52.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:52.352086+0000) 2024-02-19T21:46:53.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:53.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:53.352350+0000) 2024-02-19T21:46:54.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:54.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:54.352605+0000) 2024-02-19T21:46:55.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:55.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:55.352883+0000) 2024-02-19T21:46:55.925 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:46:55.928 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:46:56.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:56.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:56.353128+0000) 2024-02-19T21:46:57.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:57.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:57.353373+0000) 2024-02-19T21:46:58.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:58.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:58.353553+0000) 2024-02-19T21:46:59.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:46:59.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:46:59.353805+0000) 2024-02-19T21:47:00.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:00.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:00.354064+0000) 2024-02-19T21:47:01.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:01.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:01.354335+0000) 2024-02-19T21:47:02.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:02.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:02.354620+0000) 2024-02-19T21:47:03.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:03.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:03.354790+0000) 2024-02-19T21:47:04.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:04.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:04.355024+0000) 2024-02-19T21:47:05.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:05.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:05.355281+0000) 2024-02-19T21:47:06.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:06.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:06.355471+0000) 2024-02-19T21:47:07.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:07.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:07.355739+0000) 2024-02-19T21:47:08.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:08.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:08.355974+0000) 2024-02-19T21:47:09.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:09.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:09.356120+0000) 2024-02-19T21:47:10.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:10.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:10.356365+0000) 2024-02-19T21:47:11.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:11.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:11.356583+0000) 2024-02-19T21:47:12.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:12.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:12.356835+0000) 2024-02-19T21:47:13.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:13.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:13.357128+0000) 2024-02-19T21:47:14.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:14.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:14.357400+0000) 2024-02-19T21:47:15.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:15.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:15.357639+0000) 2024-02-19T21:47:16.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:16.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:16.357881+0000) 2024-02-19T21:47:17.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:17.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:17.358129+0000) 2024-02-19T21:47:18.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:18.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:18.358326+0000) 2024-02-19T21:47:19.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:19.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:19.358515+0000) 2024-02-19T21:47:20.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:20.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:20.358827+0000) 2024-02-19T21:47:21.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:21.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:21.359024+0000) 2024-02-19T21:47:22.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:22.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:22.359246+0000) 2024-02-19T21:47:23.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:23.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:23.359441+0000) 2024-02-19T21:47:24.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:24.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:24.359692+0000) 2024-02-19T21:47:25.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:25.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:25.359869+0000) 2024-02-19T21:47:26.006 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:47:26.009 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:47:26.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:26.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:26.360077+0000) 2024-02-19T21:47:27.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:27.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:27.360327+0000) 2024-02-19T21:47:28.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:28.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:28.360561+0000) 2024-02-19T21:47:29.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:29.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:29.360797+0000) 2024-02-19T21:47:30.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:30.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:30.361049+0000) 2024-02-19T21:47:31.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:31.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:31.361288+0000) 2024-02-19T21:47:32.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:32.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:32.361490+0000) 2024-02-19T21:47:33.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:33.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:33.361734+0000) 2024-02-19T21:47:34.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:34.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:34.361950+0000) 2024-02-19T21:47:35.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:35.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:35.362202+0000) 2024-02-19T21:47:36.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:36.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:36.362381+0000) 2024-02-19T21:47:37.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:37.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:37.362616+0000) 2024-02-19T21:47:38.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:38.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:38.362835+0000) 2024-02-19T21:47:39.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:39.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:39.363031+0000) 2024-02-19T21:47:40.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:40.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:40.363295+0000) 2024-02-19T21:47:41.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:41.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:41.363476+0000) 2024-02-19T21:47:42.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:42.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:42.363738+0000) 2024-02-19T21:47:43.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:43.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:43.363954+0000) 2024-02-19T21:47:44.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:44.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:44.364152+0000) 2024-02-19T21:47:45.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:45.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:45.364403+0000) 2024-02-19T21:47:46.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:46.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:46.364575+0000) 2024-02-19T21:47:47.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:47.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:47.364775+0000) 2024-02-19T21:47:48.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:48.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:48.364993+0000) 2024-02-19T21:47:49.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:49.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:49.365208+0000) 2024-02-19T21:47:50.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:50.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:50.365480+0000) 2024-02-19T21:47:51.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:51.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:51.365754+0000) 2024-02-19T21:47:52.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:52.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:52.365994+0000) 2024-02-19T21:47:53.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:53.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:53.366228+0000) 2024-02-19T21:47:54.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:54.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:54.366498+0000) 2024-02-19T21:47:55.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:55.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:55.366779+0000) 2024-02-19T21:47:56.143 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:47:56.147 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:47:56.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:56.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:56.366991+0000) 2024-02-19T21:47:57.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:57.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:57.367152+0000) 2024-02-19T21:47:58.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:58.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:58.367310+0000) 2024-02-19T21:47:59.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:47:59.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:47:59.367546+0000) 2024-02-19T21:48:00.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:00.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:00.367797+0000) 2024-02-19T21:48:01.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:01.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:01.368046+0000) 2024-02-19T21:48:02.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:02.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:02.368197+0000) 2024-02-19T21:48:03.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:03.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:03.368409+0000) 2024-02-19T21:48:04.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:04.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:04.368586+0000) 2024-02-19T21:48:05.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:05.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:05.368861+0000) 2024-02-19T21:48:06.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:06.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:06.369060+0000) 2024-02-19T21:48:07.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:07.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:07.369253+0000) 2024-02-19T21:48:08.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:08.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:08.369491+0000) 2024-02-19T21:48:09.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:09.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:09.369730+0000) 2024-02-19T21:48:10.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:10.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:10.370000+0000) 2024-02-19T21:48:11.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:11.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:11.370256+0000) 2024-02-19T21:48:12.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:12.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:12.370495+0000) 2024-02-19T21:48:13.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:13.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:13.370690+0000) 2024-02-19T21:48:14.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:14.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:14.370946+0000) 2024-02-19T21:48:15.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:15.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:15.371151+0000) 2024-02-19T21:48:16.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:16.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:16.371401+0000) 2024-02-19T21:48:17.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:17.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:17.371640+0000) 2024-02-19T21:48:18.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:18.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:18.371881+0000) 2024-02-19T21:48:19.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:19.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:19.372162+0000) 2024-02-19T21:48:20.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:20.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:20.372442+0000) 2024-02-19T21:48:21.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:21.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:21.372689+0000) 2024-02-19T21:48:22.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:22.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:22.372931+0000) 2024-02-19T21:48:23.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:23.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:23.373143+0000) 2024-02-19T21:48:24.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:24.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:24.373424+0000) 2024-02-19T21:48:25.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:25.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:25.373680+0000) 2024-02-19T21:48:26.202 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:48:26.206 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:48:26.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:26.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:26.373925+0000) 2024-02-19T21:48:27.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:27.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:27.374196+0000) 2024-02-19T21:48:28.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:28.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:28.374395+0000) 2024-02-19T21:48:29.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:29.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:29.374636+0000) 2024-02-19T21:48:30.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:30.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:30.374834+0000) 2024-02-19T21:48:31.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:31.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:31.375045+0000) 2024-02-19T21:48:32.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:32.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:32.375244+0000) 2024-02-19T21:48:33.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:33.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:33.375509+0000) 2024-02-19T21:48:34.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:34.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:34.375757+0000) 2024-02-19T21:48:35.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:35.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:35.375967+0000) 2024-02-19T21:48:36.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:36.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:36.376205+0000) 2024-02-19T21:48:37.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:37.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:37.376449+0000) 2024-02-19T21:48:38.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:38.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:38.376639+0000) 2024-02-19T21:48:39.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:39.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:39.376892+0000) 2024-02-19T21:48:40.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:40.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:40.377135+0000) 2024-02-19T21:48:41.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:41.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:41.377396+0000) 2024-02-19T21:48:42.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:42.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:42.377575+0000) 2024-02-19T21:48:43.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:43.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:43.377778+0000) 2024-02-19T21:48:44.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:44.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:44.378040+0000) 2024-02-19T21:48:45.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:45.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:45.378284+0000) 2024-02-19T21:48:46.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:46.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:46.378492+0000) 2024-02-19T21:48:47.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:47.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:47.378684+0000) 2024-02-19T21:48:48.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:48.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:48.378944+0000) 2024-02-19T21:48:49.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:49.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:49.379198+0000) 2024-02-19T21:48:50.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:50.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:50.379445+0000) 2024-02-19T21:48:51.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:51.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:51.379678+0000) 2024-02-19T21:48:52.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:52.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:52.379924+0000) 2024-02-19T21:48:53.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:53.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:53.380205+0000) 2024-02-19T21:48:54.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:54.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:54.380450+0000) 2024-02-19T21:48:55.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:55.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:55.380689+0000) 2024-02-19T21:48:56.279 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:48:56.283 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:48:56.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:56.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:56.380876+0000) 2024-02-19T21:48:57.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:57.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:57.381122+0000) 2024-02-19T21:48:58.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:58.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:58.381369+0000) 2024-02-19T21:48:59.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:48:59.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:48:59.381554+0000) 2024-02-19T21:49:00.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:00.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:00.381775+0000) 2024-02-19T21:49:01.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:01.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:01.382016+0000) 2024-02-19T21:49:02.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:02.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:02.382254+0000) 2024-02-19T21:49:03.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:03.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:03.382468+0000) 2024-02-19T21:49:04.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:04.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:04.382739+0000) 2024-02-19T21:49:05.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:05.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:05.383031+0000) 2024-02-19T21:49:06.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:06.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:06.383175+0000) 2024-02-19T21:49:07.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:07.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:07.383403+0000) 2024-02-19T21:49:08.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:08.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:08.383641+0000) 2024-02-19T21:49:09.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:09.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:09.383904+0000) 2024-02-19T21:49:10.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:10.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:10.384115+0000) 2024-02-19T21:49:11.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:11.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:11.384286+0000) 2024-02-19T21:49:12.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:12.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:12.384534+0000) 2024-02-19T21:49:13.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:13.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:13.384771+0000) 2024-02-19T21:49:14.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:14.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:14.385010+0000) 2024-02-19T21:49:15.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:15.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:15.385264+0000) 2024-02-19T21:49:16.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:16.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:16.385458+0000) 2024-02-19T21:49:17.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:17.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:17.385700+0000) 2024-02-19T21:49:18.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:18.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:18.385937+0000) 2024-02-19T21:49:19.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:19.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:19.386125+0000) 2024-02-19T21:49:20.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:20.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:20.386380+0000) 2024-02-19T21:49:21.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:21.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:21.386635+0000) 2024-02-19T21:49:22.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:22.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:22.386877+0000) 2024-02-19T21:49:23.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:23.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:23.387070+0000) 2024-02-19T21:49:24.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:24.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:24.387309+0000) 2024-02-19T21:49:25.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:25.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:25.387517+0000) 2024-02-19T21:49:26.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:26.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:26.387714+0000) 2024-02-19T21:49:26.452 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:49:26.456 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:49:27.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:27.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:27.387947+0000) 2024-02-19T21:49:28.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:28.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:28.388196+0000) 2024-02-19T21:49:29.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:29.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:29.388437+0000) 2024-02-19T21:49:30.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:30.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:30.388644+0000) 2024-02-19T21:49:31.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:31.388+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:31.388892+0000) 2024-02-19T21:49:32.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:32.388+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:32.389135+0000) 2024-02-19T21:49:33.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:33.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:33.389352+0000) 2024-02-19T21:49:34.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:34.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:34.389575+0000) 2024-02-19T21:49:35.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:35.388+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:35.389755+0000) 2024-02-19T21:49:36.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:36.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:36.389957+0000) 2024-02-19T21:49:37.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:37.388+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:37.390237+0000) 2024-02-19T21:49:38.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:38.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:38.390478+0000) 2024-02-19T21:49:39.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:39.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:39.390803+0000) 2024-02-19T21:49:40.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:40.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:40.391054+0000) 2024-02-19T21:49:41.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:41.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:41.391241+0000) 2024-02-19T21:49:42.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:42.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:42.391516+0000) 2024-02-19T21:49:43.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:43.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:43.391760+0000) 2024-02-19T21:49:44.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:44.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:44.392023+0000) 2024-02-19T21:49:45.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:45.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:45.392220+0000) 2024-02-19T21:49:46.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:46.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:46.392452+0000) 2024-02-19T21:49:47.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:47.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:47.392681+0000) 2024-02-19T21:49:48.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:48.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:48.392799+0000) 2024-02-19T21:49:49.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:49.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:49.393048+0000) 2024-02-19T21:49:50.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:50.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:50.393239+0000) 2024-02-19T21:49:51.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:51.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:51.393448+0000) 2024-02-19T21:49:52.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:52.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:52.393571+0000) 2024-02-19T21:49:53.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:53.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:53.393756+0000) 2024-02-19T21:49:54.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:54.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:54.394008+0000) 2024-02-19T21:49:55.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:55.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:55.394266+0000) 2024-02-19T21:49:56.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:56.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:56.394492+0000) 2024-02-19T21:49:56.516 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:49:56.519 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:49:57.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:57.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:57.394769+0000) 2024-02-19T21:49:58.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:58.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:58.394962+0000) 2024-02-19T21:49:59.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:49:59.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:49:59.395230+0000) 2024-02-19T21:50:00.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:00.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:00.395475+0000) 2024-02-19T21:50:01.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:01.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:01.395660+0000) 2024-02-19T21:50:02.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:02.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:02.395843+0000) 2024-02-19T21:50:03.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:03.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:03.396084+0000) 2024-02-19T21:50:04.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:04.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:04.396254+0000) 2024-02-19T21:50:05.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:05.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:05.396517+0000) 2024-02-19T21:50:06.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:06.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:06.396759+0000) 2024-02-19T21:50:07.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:07.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:07.397005+0000) 2024-02-19T21:50:08.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:08.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:08.397232+0000) 2024-02-19T21:50:09.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:09.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:09.397480+0000) 2024-02-19T21:50:10.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:10.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:10.397716+0000) 2024-02-19T21:50:11.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:11.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:11.397956+0000) 2024-02-19T21:50:12.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:12.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:12.398181+0000) 2024-02-19T21:50:13.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:13.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:13.398393+0000) 2024-02-19T21:50:14.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:14.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:14.398623+0000) 2024-02-19T21:50:15.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:15.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:15.398909+0000) 2024-02-19T21:50:16.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:16.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:16.399045+0000) 2024-02-19T21:50:17.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:17.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:17.399253+0000) 2024-02-19T21:50:18.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:18.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:18.399452+0000) 2024-02-19T21:50:19.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:19.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:19.399703+0000) 2024-02-19T21:50:20.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:20.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:20.399931+0000) 2024-02-19T21:50:21.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:21.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:21.400172+0000) 2024-02-19T21:50:22.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:22.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:22.400338+0000) 2024-02-19T21:50:23.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:23.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:23.400635+0000) 2024-02-19T21:50:24.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:24.399+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:24.400827+0000) 2024-02-19T21:50:25.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:25.399+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:25.401109+0000) 2024-02-19T21:50:26.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:26.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:26.401278+0000) 2024-02-19T21:50:26.655 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:50:26.659 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:50:27.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:27.399+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:27.401488+0000) 2024-02-19T21:50:28.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:28.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:28.401663+0000) 2024-02-19T21:50:29.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:29.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:29.401846+0000) 2024-02-19T21:50:30.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:30.399+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:30.402131+0000) 2024-02-19T21:50:31.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:31.399+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:31.402386+0000) 2024-02-19T21:50:32.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:32.399+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:32.402603+0000) 2024-02-19T21:50:33.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:33.399+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:33.402877+0000) 2024-02-19T21:50:34.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:34.399+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:34.403013+0000) 2024-02-19T21:50:35.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:35.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:35.403232+0000) 2024-02-19T21:50:36.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:36.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:36.403501+0000) 2024-02-19T21:50:37.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:37.399+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:37.403662+0000) 2024-02-19T21:50:38.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:38.399+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:38.403922+0000) 2024-02-19T21:50:39.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:39.403+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:39.404189+0000) 2024-02-19T21:50:40.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:40.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:40.404470+0000) 2024-02-19T21:50:41.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:41.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:41.404726+0000) 2024-02-19T21:50:42.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:42.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:42.405024+0000) 2024-02-19T21:50:43.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:43.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:43.405318+0000) 2024-02-19T21:50:44.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:44.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:44.405527+0000) 2024-02-19T21:50:45.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:45.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:45.405778+0000) 2024-02-19T21:50:46.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:46.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:46.406013+0000) 2024-02-19T21:50:47.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:47.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:47.406187+0000) 2024-02-19T21:50:48.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:48.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:48.406454+0000) 2024-02-19T21:50:49.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:49.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:49.406700+0000) 2024-02-19T21:50:50.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:50.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:50.406994+0000) 2024-02-19T21:50:51.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:51.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:51.407229+0000) 2024-02-19T21:50:52.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:52.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:52.407458+0000) 2024-02-19T21:50:53.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:53.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:53.407650+0000) 2024-02-19T21:50:54.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:54.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:54.407869+0000) 2024-02-19T21:50:55.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:55.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:55.408144+0000) 2024-02-19T21:50:56.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:56.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:56.408431+0000) 2024-02-19T21:50:56.769 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:50:56.772 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:50:57.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:57.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:57.408635+0000) 2024-02-19T21:50:58.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:58.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:58.408835+0000) 2024-02-19T21:50:59.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:50:59.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:50:59.409112+0000) 2024-02-19T21:51:00.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:00.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:00.409394+0000) 2024-02-19T21:51:01.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:01.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:01.409716+0000) 2024-02-19T21:51:02.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:02.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:02.409937+0000) 2024-02-19T21:51:03.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:03.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:03.410233+0000) 2024-02-19T21:51:04.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:04.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:04.410522+0000) 2024-02-19T21:51:05.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:05.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:05.410792+0000) 2024-02-19T21:51:06.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:06.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:06.411044+0000) 2024-02-19T21:51:07.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:07.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:07.411338+0000) 2024-02-19T21:51:08.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:08.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:08.411627+0000) 2024-02-19T21:51:09.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:09.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:09.411914+0000) 2024-02-19T21:51:10.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:10.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:10.412190+0000) 2024-02-19T21:51:11.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:11.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:11.412484+0000) 2024-02-19T21:51:12.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:12.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:12.412784+0000) 2024-02-19T21:51:13.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:13.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:13.413089+0000) 2024-02-19T21:51:14.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:14.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:14.413309+0000) 2024-02-19T21:51:15.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:15.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:15.413496+0000) 2024-02-19T21:51:16.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:16.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:16.413748+0000) 2024-02-19T21:51:17.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:17.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:17.414054+0000) 2024-02-19T21:51:18.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:18.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:18.414360+0000) 2024-02-19T21:51:19.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:19.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:19.414617+0000) 2024-02-19T21:51:20.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:20.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:20.414857+0000) 2024-02-19T21:51:21.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:21.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:21.415061+0000) 2024-02-19T21:51:22.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:22.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:22.415276+0000) 2024-02-19T21:51:23.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:23.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:23.415536+0000) 2024-02-19T21:51:24.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:24.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:24.415795+0000) 2024-02-19T21:51:25.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:25.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:25.416003+0000) 2024-02-19T21:51:26.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:26.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:26.416240+0000) 2024-02-19T21:51:26.833 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:51:26.836 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:51:27.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:27.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:27.416494+0000) 2024-02-19T21:51:28.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:28.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:28.416750+0000) 2024-02-19T21:51:29.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:29.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:29.417031+0000) 2024-02-19T21:51:30.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:30.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:30.417260+0000) 2024-02-19T21:51:31.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:31.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:31.417482+0000) 2024-02-19T21:51:32.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:32.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:32.417682+0000) 2024-02-19T21:51:33.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:33.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:33.417954+0000) 2024-02-19T21:51:34.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:34.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:34.418189+0000) 2024-02-19T21:51:35.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:35.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:35.418389+0000) 2024-02-19T21:51:36.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:36.414+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:36.418564+0000) 2024-02-19T21:51:37.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:37.418+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:37.418846+0000) 2024-02-19T21:51:38.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:38.418+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:38.419035+0000) 2024-02-19T21:51:39.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:39.418+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:39.419211+0000) 2024-02-19T21:51:40.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:40.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:40.419463+0000) 2024-02-19T21:51:41.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:41.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:41.419700+0000) 2024-02-19T21:51:42.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:42.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:42.419893+0000) 2024-02-19T21:51:43.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:43.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:43.420091+0000) 2024-02-19T21:51:44.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:44.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:44.420365+0000) 2024-02-19T21:51:45.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:45.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:45.420617+0000) 2024-02-19T21:51:46.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:46.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:46.420866+0000) 2024-02-19T21:51:47.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:47.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:47.421039+0000) 2024-02-19T21:51:48.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:48.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:48.421218+0000) 2024-02-19T21:51:49.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:49.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:49.421414+0000) 2024-02-19T21:51:50.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:50.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:50.421649+0000) 2024-02-19T21:51:51.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:51.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:51.421876+0000) 2024-02-19T21:51:52.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:52.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:52.422084+0000) 2024-02-19T21:51:53.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:53.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:53.422322+0000) 2024-02-19T21:51:54.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:54.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:54.422552+0000) 2024-02-19T21:51:55.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:55.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:55.422853+0000) 2024-02-19T21:51:56.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:56.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:56.423083+0000) 2024-02-19T21:51:56.946 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:51:56.950 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:51:57.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:57.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:57.423279+0000) 2024-02-19T21:51:58.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:58.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:58.423546+0000) 2024-02-19T21:51:59.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:51:59.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:51:59.423826+0000) 2024-02-19T21:52:00.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:00.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:00.424069+0000) 2024-02-19T21:52:01.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:01.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:01.424338+0000) 2024-02-19T21:52:02.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:02.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:02.424574+0000) 2024-02-19T21:52:03.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:03.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:03.424778+0000) 2024-02-19T21:52:04.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:04.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:04.425051+0000) 2024-02-19T21:52:05.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:05.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:05.425259+0000) 2024-02-19T21:52:06.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:06.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:06.425537+0000) 2024-02-19T21:52:07.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:07.425+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:07.425789+0000) 2024-02-19T21:52:08.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:08.425+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:08.426058+0000) 2024-02-19T21:52:09.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:09.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:09.426283+0000) 2024-02-19T21:52:10.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:10.425+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:10.426484+0000) 2024-02-19T21:52:11.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:11.425+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:11.426790+0000) 2024-02-19T21:52:12.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:12.425+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:12.426987+0000) 2024-02-19T21:52:13.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:13.425+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:13.427185+0000) 2024-02-19T21:52:14.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:14.425+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:14.427417+0000) 2024-02-19T21:52:15.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:15.425+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:15.427675+0000) 2024-02-19T21:52:16.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:16.425+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:16.427927+0000) 2024-02-19T21:52:17.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:17.425+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:17.428113+0000) 2024-02-19T21:52:18.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:18.425+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:18.428355+0000) 2024-02-19T21:52:19.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:19.425+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:19.428528+0000) 2024-02-19T21:52:20.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:20.425+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:20.428751+0000) 2024-02-19T21:52:21.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:21.425+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:21.429015+0000) 2024-02-19T21:52:22.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:22.425+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:22.429234+0000) 2024-02-19T21:52:23.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:23.429+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:23.429512+0000) 2024-02-19T21:52:24.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:24.425+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:24.429733+0000) 2024-02-19T21:52:25.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:25.425+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:25.429961+0000) 2024-02-19T21:52:26.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:26.429+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:26.430227+0000) 2024-02-19T21:52:27.120 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:52:27.123 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:52:27.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:27.429+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:27.430478+0000) 2024-02-19T21:52:28.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:28.429+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:28.430793+0000) 2024-02-19T21:52:29.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:29.429+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:29.431046+0000) 2024-02-19T21:52:30.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:30.429+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:30.431291+0000) 2024-02-19T21:52:31.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:31.429+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:31.431568+0000) 2024-02-19T21:52:32.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:32.429+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:32.431820+0000) 2024-02-19T21:52:33.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:33.429+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:33.432001+0000) 2024-02-19T21:52:34.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:34.429+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:34.432260+0000) 2024-02-19T21:52:35.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:35.429+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:35.432478+0000) 2024-02-19T21:52:36.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:36.429+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:36.432699+0000) 2024-02-19T21:52:37.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:37.429+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:37.432947+0000) 2024-02-19T21:52:38.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:38.429+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:38.433178+0000) 2024-02-19T21:52:39.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:39.429+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:39.433406+0000) 2024-02-19T21:52:40.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:40.433+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:40.433640+0000) 2024-02-19T21:52:41.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:41.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:41.433862+0000) 2024-02-19T21:52:42.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:42.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:42.434101+0000) 2024-02-19T21:52:43.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:43.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:43.434359+0000) 2024-02-19T21:52:44.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:44.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:44.434529+0000) 2024-02-19T21:52:45.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:45.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:45.434812+0000) 2024-02-19T21:52:46.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:46.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:46.435052+0000) 2024-02-19T21:52:47.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:47.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:47.435223+0000) 2024-02-19T21:52:48.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:48.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:48.435379+0000) 2024-02-19T21:52:49.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:49.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:49.435640+0000) 2024-02-19T21:52:50.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:50.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:50.435833+0000) 2024-02-19T21:52:51.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:51.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:51.436078+0000) 2024-02-19T21:52:52.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:52.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:52.436306+0000) 2024-02-19T21:52:53.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:53.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:53.436464+0000) 2024-02-19T21:52:54.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:54.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:54.436699+0000) 2024-02-19T21:52:55.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:55.436+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:55.436883+0000) 2024-02-19T21:52:56.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:56.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:56.437103+0000) 2024-02-19T21:52:57.267 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:52:57.270 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:52:57.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:57.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:57.437343+0000) 2024-02-19T21:52:58.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:58.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:58.437559+0000) 2024-02-19T21:52:59.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:52:59.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:52:59.437804+0000) 2024-02-19T21:53:00.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:00.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:00.438081+0000) 2024-02-19T21:53:01.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:01.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:01.438353+0000) 2024-02-19T21:53:02.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:02.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:02.438519+0000) 2024-02-19T21:53:03.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:03.436+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:03.438802+0000) 2024-02-19T21:53:04.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:04.436+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:04.439021+0000) 2024-02-19T21:53:05.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:05.436+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:05.439296+0000) 2024-02-19T21:53:06.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:06.436+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:06.439472+0000) 2024-02-19T21:53:07.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:07.436+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:07.439709+0000) 2024-02-19T21:53:08.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:08.436+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:08.439931+0000) 2024-02-19T21:53:09.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:09.436+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:09.440139+0000) 2024-02-19T21:53:10.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:10.436+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:10.440347+0000) 2024-02-19T21:53:11.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:11.436+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:11.440588+0000) 2024-02-19T21:53:12.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:12.440+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:12.440805+0000) 2024-02-19T21:53:13.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:13.440+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:13.441065+0000) 2024-02-19T21:53:14.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:14.436+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:14.441288+0000) 2024-02-19T21:53:15.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:15.436+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:15.441530+0000) 2024-02-19T21:53:16.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:16.436+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:16.441808+0000) 2024-02-19T21:53:17.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:17.440+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:17.442051+0000) 2024-02-19T21:53:18.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:18.436+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:18.442285+0000) 2024-02-19T21:53:19.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:19.436+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:19.442529+0000) 2024-02-19T21:53:20.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:20.440+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:20.442802+0000) 2024-02-19T21:53:21.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:21.440+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:21.443051+0000) 2024-02-19T21:53:22.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:22.440+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:22.443230+0000) 2024-02-19T21:53:23.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:23.440+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:23.443435+0000) 2024-02-19T21:53:24.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:24.440+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:24.443687+0000) 2024-02-19T21:53:25.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:25.440+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:25.443873+0000) 2024-02-19T21:53:26.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:26.440+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:26.444128+0000) 2024-02-19T21:53:27.388 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:53:27.392 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:53:27.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:27.440+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:27.444313+0000) 2024-02-19T21:53:28.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:28.440+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:28.444583+0000) 2024-02-19T21:53:29.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:29.440+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:29.444866+0000) 2024-02-19T21:53:30.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:30.440+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:30.445093+0000) 2024-02-19T21:53:31.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:31.440+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:31.445302+0000) 2024-02-19T21:53:32.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:32.440+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:32.445492+0000) 2024-02-19T21:53:33.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:33.440+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:33.445715+0000) 2024-02-19T21:53:34.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:34.440+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:34.445947+0000) 2024-02-19T21:53:35.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:35.440+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:35.446200+0000) 2024-02-19T21:53:36.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:36.444+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:36.446440+0000) 2024-02-19T21:53:37.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:37.440+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:37.446632+0000) 2024-02-19T21:53:38.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:38.444+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:38.446897+0000) 2024-02-19T21:53:39.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:39.444+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:39.447153+0000) 2024-02-19T21:53:40.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:40.444+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:40.447339+0000) 2024-02-19T21:53:41.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:41.444+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:41.447555+0000) 2024-02-19T21:53:42.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:42.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:42.447743+0000) 2024-02-19T21:53:43.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:43.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:43.447962+0000) 2024-02-19T21:53:44.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:44.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:44.448223+0000) 2024-02-19T21:53:45.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:45.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:45.448483+0000) 2024-02-19T21:53:46.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:46.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:46.448709+0000) 2024-02-19T21:53:47.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:47.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:47.448910+0000) 2024-02-19T21:53:48.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:48.447+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:48.449149+0000) 2024-02-19T21:53:49.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:49.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:49.449334+0000) 2024-02-19T21:53:50.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:50.447+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:50.449602+0000) 2024-02-19T21:53:51.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:51.447+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:51.449773+0000) 2024-02-19T21:53:52.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:52.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:52.450018+0000) 2024-02-19T21:53:53.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:53.447+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:53.450221+0000) 2024-02-19T21:53:54.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:54.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:54.450447+0000) 2024-02-19T21:53:55.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:55.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:55.450624+0000) 2024-02-19T21:53:56.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:56.447+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:56.450800+0000) 2024-02-19T21:53:57.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:57.447+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:57.450946+0000) 2024-02-19T21:53:57.460 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:53:57.464 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:53:58.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:58.447+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:58.451186+0000) 2024-02-19T21:53:59.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:53:59.447+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:53:59.451378+0000) 2024-02-19T21:54:00.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:00.447+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:00.451570+0000) 2024-02-19T21:54:01.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:01.447+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:01.451830+0000) 2024-02-19T21:54:02.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:02.451+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:02.452003+0000) 2024-02-19T21:54:03.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:03.451+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:03.452265+0000) 2024-02-19T21:54:04.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:04.451+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:04.452478+0000) 2024-02-19T21:54:05.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:05.447+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:05.452663+0000) 2024-02-19T21:54:06.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:06.447+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:06.452882+0000) 2024-02-19T21:54:07.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:07.451+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:07.453074+0000) 2024-02-19T21:54:08.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:08.451+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:08.453356+0000) 2024-02-19T21:54:09.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:09.451+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:09.453540+0000) 2024-02-19T21:54:10.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:10.451+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:10.453779+0000) 2024-02-19T21:54:11.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:11.447+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:11.454016+0000) 2024-02-19T21:54:12.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:12.451+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:12.454254+0000) 2024-02-19T21:54:13.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:13.451+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:13.454434+0000) 2024-02-19T21:54:14.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:14.451+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:14.454665+0000) 2024-02-19T21:54:15.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:15.451+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:15.454939+0000) 2024-02-19T21:54:16.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:16.451+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:16.455136+0000) 2024-02-19T21:54:17.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:17.451+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:17.455318+0000) 2024-02-19T21:54:18.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:18.455+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:18.455518+0000) 2024-02-19T21:54:19.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:19.455+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:19.455772+0000) 2024-02-19T21:54:20.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:20.455+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:20.456007+0000) 2024-02-19T21:54:21.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:21.455+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:21.456252+0000) 2024-02-19T21:54:22.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:22.451+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:22.456489+0000) 2024-02-19T21:54:23.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:23.451+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:23.456724+0000) 2024-02-19T21:54:24.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:24.455+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:24.456957+0000) 2024-02-19T21:54:25.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:25.455+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:25.457204+0000) 2024-02-19T21:54:26.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:26.455+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:26.457427+0000) 2024-02-19T21:54:27.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:27.455+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:27.457712+0000) 2024-02-19T21:54:27.775 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:54:27.779 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:54:28.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:28.455+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:28.457945+0000) 2024-02-19T21:54:29.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:29.451+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:29.458162+0000) 2024-02-19T21:54:30.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:30.455+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:30.458375+0000) 2024-02-19T21:54:31.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:31.455+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:31.458615+0000) 2024-02-19T21:54:32.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:32.455+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:32.458883+0000) 2024-02-19T21:54:33.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:33.455+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:33.459158+0000) 2024-02-19T21:54:34.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:34.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:34.459396+0000) 2024-02-19T21:54:35.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:35.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:35.459571+0000) 2024-02-19T21:54:36.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:36.455+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:36.459845+0000) 2024-02-19T21:54:37.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:37.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:37.460101+0000) 2024-02-19T21:54:38.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:38.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:38.460339+0000) 2024-02-19T21:54:39.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:39.455+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:39.460588+0000) 2024-02-19T21:54:40.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:40.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:40.460791+0000) 2024-02-19T21:54:41.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:41.459+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:41.461097+0000) 2024-02-19T21:54:42.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:42.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:42.461230+0000) 2024-02-19T21:54:43.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:43.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:43.461416+0000) 2024-02-19T21:54:44.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:44.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:44.461658+0000) 2024-02-19T21:54:45.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:45.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:45.461857+0000) 2024-02-19T21:54:46.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:46.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:46.462098+0000) 2024-02-19T21:54:47.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:47.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:47.462345+0000) 2024-02-19T21:54:48.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:48.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:48.462540+0000) 2024-02-19T21:54:49.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:49.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:49.462827+0000) 2024-02-19T21:54:50.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:50.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:50.463043+0000) 2024-02-19T21:54:51.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:51.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:51.463289+0000) 2024-02-19T21:54:52.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:52.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:52.463530+0000) 2024-02-19T21:54:53.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:53.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:53.463709+0000) 2024-02-19T21:54:54.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:54.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:54.463847+0000) 2024-02-19T21:54:55.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:55.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:55.464093+0000) 2024-02-19T21:54:56.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:56.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:56.464324+0000) 2024-02-19T21:54:57.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:57.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:57.464528+0000) 2024-02-19T21:54:57.833 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:54:57.837 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:54:58.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:58.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:58.464772+0000) 2024-02-19T21:54:59.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:54:59.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:54:59.464962+0000) 2024-02-19T21:55:00.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:00.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:00.465223+0000) 2024-02-19T21:55:01.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:01.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:01.465401+0000) 2024-02-19T21:55:02.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:02.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:02.465643+0000) 2024-02-19T21:55:03.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:03.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:03.465884+0000) 2024-02-19T21:55:04.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:04.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:04.466132+0000) 2024-02-19T21:55:05.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:05.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:05.466331+0000) 2024-02-19T21:55:06.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:06.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:06.466586+0000) 2024-02-19T21:55:07.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:07.466+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:07.466789+0000) 2024-02-19T21:55:08.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:08.466+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:08.466988+0000) 2024-02-19T21:55:09.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:09.466+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:09.467161+0000) 2024-02-19T21:55:10.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:10.466+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:10.467381+0000) 2024-02-19T21:55:11.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:11.466+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:11.467580+0000) 2024-02-19T21:55:12.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:12.466+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:12.467813+0000) 2024-02-19T21:55:13.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:13.466+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:13.468015+0000) 2024-02-19T21:55:14.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:14.466+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:14.468248+0000) 2024-02-19T21:55:15.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:15.466+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:15.468498+0000) 2024-02-19T21:55:16.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:16.466+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:16.468732+0000) 2024-02-19T21:55:17.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:17.466+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:17.468976+0000) 2024-02-19T21:55:18.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:18.466+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:18.469244+0000) 2024-02-19T21:55:19.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:19.466+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:19.469450+0000) 2024-02-19T21:55:20.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:20.466+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:20.469695+0000) 2024-02-19T21:55:21.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:21.466+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:21.469874+0000) 2024-02-19T21:55:22.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:22.466+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:22.470113+0000) 2024-02-19T21:55:23.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:23.466+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:23.470288+0000) 2024-02-19T21:55:24.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:24.466+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:24.470540+0000) 2024-02-19T21:55:25.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:25.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:25.470831+0000) 2024-02-19T21:55:26.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:26.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:26.471020+0000) 2024-02-19T21:55:27.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:27.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:27.471257+0000) 2024-02-19T21:55:28.032 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:55:28.035 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:55:28.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:28.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:28.471434+0000) 2024-02-19T21:55:29.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:29.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:29.471679+0000) 2024-02-19T21:55:30.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:30.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:30.471909+0000) 2024-02-19T21:55:31.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:31.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:31.472160+0000) 2024-02-19T21:55:32.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:32.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:32.472434+0000) 2024-02-19T21:55:33.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:33.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:33.472655+0000) 2024-02-19T21:55:34.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:34.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:34.472890+0000) 2024-02-19T21:55:35.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:35.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:35.473099+0000) 2024-02-19T21:55:36.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:36.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:36.473300+0000) 2024-02-19T21:55:37.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:37.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:37.473545+0000) 2024-02-19T21:55:38.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:38.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:38.473737+0000) 2024-02-19T21:55:39.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:39.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:39.474014+0000) 2024-02-19T21:55:40.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:40.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:40.474275+0000) 2024-02-19T21:55:41.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:41.470+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:41.474556+0000) 2024-02-19T21:55:42.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:42.474+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:42.474782+0000) 2024-02-19T21:55:43.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:43.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:43.475003+0000) 2024-02-19T21:55:44.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:44.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:44.475251+0000) 2024-02-19T21:55:45.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:45.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:45.475500+0000) 2024-02-19T21:55:46.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:46.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:46.475697+0000) 2024-02-19T21:55:47.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:47.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:47.475944+0000) 2024-02-19T21:55:48.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:48.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:48.476187+0000) 2024-02-19T21:55:49.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:49.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:49.476454+0000) 2024-02-19T21:55:50.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:50.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:50.476694+0000) 2024-02-19T21:55:51.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:51.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:51.476973+0000) 2024-02-19T21:55:52.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:52.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:52.477193+0000) 2024-02-19T21:55:53.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:53.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:53.477379+0000) 2024-02-19T21:55:54.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:54.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:54.477609+0000) 2024-02-19T21:55:55.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:55.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:55.477799+0000) 2024-02-19T21:55:56.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:56.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:56.478030+0000) 2024-02-19T21:55:57.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:57.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:57.478274+0000) 2024-02-19T21:55:58.284 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:55:58.288 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:55:58.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:58.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:58.478509+0000) 2024-02-19T21:55:59.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:55:59.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:55:59.478700+0000) 2024-02-19T21:56:00.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:00.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:00.478949+0000) 2024-02-19T21:56:01.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:01.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:01.479177+0000) 2024-02-19T21:56:02.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:02.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:02.479453+0000) 2024-02-19T21:56:03.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:03.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:03.479700+0000) 2024-02-19T21:56:04.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:04.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:04.479883+0000) 2024-02-19T21:56:05.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:05.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:05.480126+0000) 2024-02-19T21:56:06.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:06.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:06.480358+0000) 2024-02-19T21:56:07.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:07.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:07.480612+0000) 2024-02-19T21:56:08.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:08.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:08.480846+0000) 2024-02-19T21:56:09.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:09.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:09.481115+0000) 2024-02-19T21:56:10.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:10.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:10.481291+0000) 2024-02-19T21:56:11.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:11.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:11.481495+0000) 2024-02-19T21:56:12.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:12.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:12.481761+0000) 2024-02-19T21:56:13.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:13.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:13.481970+0000) 2024-02-19T21:56:14.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:14.481+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:14.482210+0000) 2024-02-19T21:56:15.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:15.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:15.482471+0000) 2024-02-19T21:56:16.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:16.481+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:16.482742+0000) 2024-02-19T21:56:17.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:17.481+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:17.483033+0000) 2024-02-19T21:56:18.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:18.481+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:18.483273+0000) 2024-02-19T21:56:19.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:19.481+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:19.483486+0000) 2024-02-19T21:56:20.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:20.481+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:20.483725+0000) 2024-02-19T21:56:21.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:21.481+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:21.484003+0000) 2024-02-19T21:56:22.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:22.481+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:22.484239+0000) 2024-02-19T21:56:23.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:23.481+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:23.484476+0000) 2024-02-19T21:56:24.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:24.481+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:24.484713+0000) 2024-02-19T21:56:25.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:25.481+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:25.484958+0000) 2024-02-19T21:56:26.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:26.481+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:26.485207+0000) 2024-02-19T21:56:27.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:27.481+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:27.485449+0000) 2024-02-19T21:56:28.407 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:56:28.410 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:56:28.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:28.481+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:28.485684+0000) 2024-02-19T21:56:29.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:29.481+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:29.485909+0000) 2024-02-19T21:56:30.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:30.481+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:30.486161+0000) 2024-02-19T21:56:31.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:31.481+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:31.486401+0000) 2024-02-19T21:56:32.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:32.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:32.486579+0000) 2024-02-19T21:56:33.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:33.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:33.486905+0000) 2024-02-19T21:56:34.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:34.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:34.487102+0000) 2024-02-19T21:56:35.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:35.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:35.487347+0000) 2024-02-19T21:56:36.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:36.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:36.487542+0000) 2024-02-19T21:56:37.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:37.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:37.487805+0000) 2024-02-19T21:56:38.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:38.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:38.488036+0000) 2024-02-19T21:56:39.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:39.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:39.488289+0000) 2024-02-19T21:56:40.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:40.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:40.488516+0000) 2024-02-19T21:56:41.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:41.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:41.488735+0000) 2024-02-19T21:56:42.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:42.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:42.488992+0000) 2024-02-19T21:56:43.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:43.485+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:43.489244+0000) 2024-02-19T21:56:44.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:44.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:44.489474+0000) 2024-02-19T21:56:45.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:45.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:45.489715+0000) 2024-02-19T21:56:46.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:46.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:46.489962+0000) 2024-02-19T21:56:47.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:47.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:47.490208+0000) 2024-02-19T21:56:48.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:48.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:48.490442+0000) 2024-02-19T21:56:49.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:49.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:49.490662+0000) 2024-02-19T21:56:50.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:50.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:50.490901+0000) 2024-02-19T21:56:51.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:51.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:51.491135+0000) 2024-02-19T21:56:52.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:52.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:52.491345+0000) 2024-02-19T21:56:53.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:53.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:53.491602+0000) 2024-02-19T21:56:54.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:54.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:54.491778+0000) 2024-02-19T21:56:55.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:55.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:55.491961+0000) 2024-02-19T21:56:56.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:56.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:56.492199+0000) 2024-02-19T21:56:57.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:57.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:57.492393+0000) 2024-02-19T21:56:58.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:58.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:58.492631+0000) 2024-02-19T21:56:58.529 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:56:58.533 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:56:59.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:56:59.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:56:59.492819+0000) 2024-02-19T21:57:00.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:00.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:00.493090+0000) 2024-02-19T21:57:01.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:01.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:01.493304+0000) 2024-02-19T21:57:02.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:02.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:02.493543+0000) 2024-02-19T21:57:03.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:03.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:03.493717+0000) 2024-02-19T21:57:04.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:04.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:04.493964+0000) 2024-02-19T21:57:05.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:05.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:05.494205+0000) 2024-02-19T21:57:06.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:06.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:06.494473+0000) 2024-02-19T21:57:07.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:07.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:07.494701+0000) 2024-02-19T21:57:08.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:08.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:08.494960+0000) 2024-02-19T21:57:09.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:09.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:09.495223+0000) 2024-02-19T21:57:10.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:10.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:10.495468+0000) 2024-02-19T21:57:11.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:11.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:11.495735+0000) 2024-02-19T21:57:12.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:12.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:12.495921+0000) 2024-02-19T21:57:13.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:13.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:13.496163+0000) 2024-02-19T21:57:14.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:14.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:14.496422+0000) 2024-02-19T21:57:15.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:15.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:15.496688+0000) 2024-02-19T21:57:16.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:16.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:16.496951+0000) 2024-02-19T21:57:17.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:17.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:17.497125+0000) 2024-02-19T21:57:18.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:18.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:18.497330+0000) 2024-02-19T21:57:19.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:19.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:19.497550+0000) 2024-02-19T21:57:20.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:20.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:20.497740+0000) 2024-02-19T21:57:21.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:21.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:21.498054+0000) 2024-02-19T21:57:22.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:22.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:22.498251+0000) 2024-02-19T21:57:23.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:23.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:23.498453+0000) 2024-02-19T21:57:24.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:24.492+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:24.498652+0000) 2024-02-19T21:57:25.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:25.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:25.498839+0000) 2024-02-19T21:57:26.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:26.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:26.499040+0000) 2024-02-19T21:57:27.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:27.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:27.499267+0000) 2024-02-19T21:57:28.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:28.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:28.499518+0000) 2024-02-19T21:57:28.609 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:57:28.613 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:57:29.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:29.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:29.499756+0000) 2024-02-19T21:57:30.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:30.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:30.499968+0000) 2024-02-19T21:57:31.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:31.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:31.500152+0000) 2024-02-19T21:57:32.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:32.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:32.500322+0000) 2024-02-19T21:57:33.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:33.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:33.500466+0000) 2024-02-19T21:57:34.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:34.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:34.500717+0000) 2024-02-19T21:57:35.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:35.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:35.500974+0000) 2024-02-19T21:57:36.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:36.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:36.501214+0000) 2024-02-19T21:57:37.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:37.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:37.501476+0000) 2024-02-19T21:57:38.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:38.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:38.501698+0000) 2024-02-19T21:57:39.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:39.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:39.501960+0000) 2024-02-19T21:57:40.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:40.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:40.502191+0000) 2024-02-19T21:57:41.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:41.496+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:41.502448+0000) 2024-02-19T21:57:42.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:42.500+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:42.502638+0000) 2024-02-19T21:57:43.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:43.500+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:43.502815+0000) 2024-02-19T21:57:44.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:44.500+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:44.503012+0000) 2024-02-19T21:57:45.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:45.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:45.503176+0000) 2024-02-19T21:57:46.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:46.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:46.503372+0000) 2024-02-19T21:57:47.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:47.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:47.503659+0000) 2024-02-19T21:57:48.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:48.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:48.503929+0000) 2024-02-19T21:57:49.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:49.503+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:49.504257+0000) 2024-02-19T21:57:50.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:50.503+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:50.504464+0000) 2024-02-19T21:57:51.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:51.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:51.504672+0000) 2024-02-19T21:57:52.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:52.503+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:52.504912+0000) 2024-02-19T21:57:53.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:53.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:53.505169+0000) 2024-02-19T21:57:54.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:54.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:54.505406+0000) 2024-02-19T21:57:55.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:55.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:55.505593+0000) 2024-02-19T21:57:56.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:56.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:56.505760+0000) 2024-02-19T21:57:57.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:57.503+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:57.506007+0000) 2024-02-19T21:57:58.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:58.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:58.506238+0000) 2024-02-19T21:57:58.740 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:57:58.744 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:57:59.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:57:59.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:57:59.506524+0000) 2024-02-19T21:58:00.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:00.503+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:00.506817+0000) 2024-02-19T21:58:01.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:01.503+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:01.507014+0000) 2024-02-19T21:58:02.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:02.503+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:02.507254+0000) 2024-02-19T21:58:03.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:03.503+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:03.507484+0000) 2024-02-19T21:58:04.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:04.503+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:04.507706+0000) 2024-02-19T21:58:05.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:05.507+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:05.508012+0000) 2024-02-19T21:58:06.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:06.507+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:06.508232+0000) 2024-02-19T21:58:07.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:07.503+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:07.508493+0000) 2024-02-19T21:58:08.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:08.507+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:08.508690+0000) 2024-02-19T21:58:09.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:09.507+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:09.508961+0000) 2024-02-19T21:58:10.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:10.507+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:10.509228+0000) 2024-02-19T21:58:11.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:11.507+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:11.509427+0000) 2024-02-19T21:58:12.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:12.503+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:12.509695+0000) 2024-02-19T21:58:13.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:13.507+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:13.509978+0000) 2024-02-19T21:58:14.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:14.507+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:14.510283+0000) 2024-02-19T21:58:15.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:15.507+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:15.510656+0000) 2024-02-19T21:58:16.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:16.507+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:16.510890+0000) 2024-02-19T21:58:17.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:17.507+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:17.511151+0000) 2024-02-19T21:58:18.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:18.507+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:18.511462+0000) 2024-02-19T21:58:19.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:19.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:19.511770+0000) 2024-02-19T21:58:20.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:20.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:20.512049+0000) 2024-02-19T21:58:21.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:21.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:21.512362+0000) 2024-02-19T21:58:22.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:22.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:22.512649+0000) 2024-02-19T21:58:23.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:23.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:23.512948+0000) 2024-02-19T21:58:24.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:24.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:24.513205+0000) 2024-02-19T21:58:25.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:25.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:25.513408+0000) 2024-02-19T21:58:26.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:26.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:26.513666+0000) 2024-02-19T21:58:27.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:27.507+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:27.513871+0000) 2024-02-19T21:58:28.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:28.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:28.514044+0000) 2024-02-19T21:58:28.870 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:58:28.875 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:58:29.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:29.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:29.514300+0000) 2024-02-19T21:58:30.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:30.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:30.514569+0000) 2024-02-19T21:58:31.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:31.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:31.514810+0000) 2024-02-19T21:58:32.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:32.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:32.515080+0000) 2024-02-19T21:58:33.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:33.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:33.515255+0000) 2024-02-19T21:58:34.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:34.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:34.515452+0000) 2024-02-19T21:58:35.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:35.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:35.515770+0000) 2024-02-19T21:58:36.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:36.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:36.516051+0000) 2024-02-19T21:58:37.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:37.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:37.516350+0000) 2024-02-19T21:58:38.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:38.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:38.516645+0000) 2024-02-19T21:58:39.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:39.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:39.516921+0000) 2024-02-19T21:58:40.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:40.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:40.517146+0000) 2024-02-19T21:58:41.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:41.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:41.517352+0000) 2024-02-19T21:58:42.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:42.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:42.517623+0000) 2024-02-19T21:58:43.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:43.515+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:43.517854+0000) 2024-02-19T21:58:44.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:44.511+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:44.518052+0000) 2024-02-19T21:58:45.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:45.514+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:45.518306+0000) 2024-02-19T21:58:46.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:46.514+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:46.518578+0000) 2024-02-19T21:58:47.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:47.514+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:47.518856+0000) 2024-02-19T21:58:48.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:48.514+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:48.519117+0000) 2024-02-19T21:58:49.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:49.518+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:49.519407+0000) 2024-02-19T21:58:50.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:50.514+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:50.519597+0000) 2024-02-19T21:58:51.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:51.518+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:51.519867+0000) 2024-02-19T21:58:52.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:52.518+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:52.520159+0000) 2024-02-19T21:58:53.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:53.518+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:53.520438+0000) 2024-02-19T21:58:54.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:54.518+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:54.520723+0000) 2024-02-19T21:58:55.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:55.518+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:55.520998+0000) 2024-02-19T21:58:56.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:56.514+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:56.521291+0000) 2024-02-19T21:58:57.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:57.514+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:57.521516+0000) 2024-02-19T21:58:58.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:58.518+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:58.521761+0000) 2024-02-19T21:58:59.151 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:58:59.154 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:58:59.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:58:59.518+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:58:59.522049+0000) 2024-02-19T21:59:00.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:00.518+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:00.522273+0000) 2024-02-19T21:59:01.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:01.518+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:01.522476+0000) 2024-02-19T21:59:02.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:02.518+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:02.522788+0000) 2024-02-19T21:59:03.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:03.522+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:03.522959+0000) 2024-02-19T21:59:04.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:04.522+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:04.523168+0000) 2024-02-19T21:59:05.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:05.522+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:05.523392+0000) 2024-02-19T21:59:06.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:06.522+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:06.523546+0000) 2024-02-19T21:59:07.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:07.522+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:07.523819+0000) 2024-02-19T21:59:08.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:08.522+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:08.524041+0000) 2024-02-19T21:59:09.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:09.522+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:09.524302+0000) 2024-02-19T21:59:10.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:10.522+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:10.524555+0000) 2024-02-19T21:59:11.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:11.522+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:11.524865+0000) 2024-02-19T21:59:12.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:12.522+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:12.525155+0000) 2024-02-19T21:59:13.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:13.522+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:13.525444+0000) 2024-02-19T21:59:14.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:14.522+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:14.525732+0000) 2024-02-19T21:59:15.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:15.522+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:15.525950+0000) 2024-02-19T21:59:16.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:16.522+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:16.526207+0000) 2024-02-19T21:59:17.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:17.522+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:17.526450+0000) 2024-02-19T21:59:18.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:18.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:18.526727+0000) 2024-02-19T21:59:19.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:19.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:19.526967+0000) 2024-02-19T21:59:20.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:20.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:20.527121+0000) 2024-02-19T21:59:21.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:21.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:21.527342+0000) 2024-02-19T21:59:22.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:22.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:22.527550+0000) 2024-02-19T21:59:23.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:23.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:23.527769+0000) 2024-02-19T21:59:24.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:24.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:24.528026+0000) 2024-02-19T21:59:25.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:25.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:25.528315+0000) 2024-02-19T21:59:26.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:26.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:26.528604+0000) 2024-02-19T21:59:27.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:27.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:27.528835+0000) 2024-02-19T21:59:28.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:28.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:28.529123+0000) 2024-02-19T21:59:29.223 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:59:29.228 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:59:29.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:29.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:29.529396+0000) 2024-02-19T21:59:30.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:30.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:30.529652+0000) 2024-02-19T21:59:31.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:31.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:31.529856+0000) 2024-02-19T21:59:32.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:32.526+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:32.530107+0000) 2024-02-19T21:59:33.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:33.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:33.530361+0000) 2024-02-19T21:59:34.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:34.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:34.530674+0000) 2024-02-19T21:59:35.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:35.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:35.531007+0000) 2024-02-19T21:59:36.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:36.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:36.531300+0000) 2024-02-19T21:59:37.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:37.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:37.531527+0000) 2024-02-19T21:59:38.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:38.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:38.531805+0000) 2024-02-19T21:59:39.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:39.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:39.532011+0000) 2024-02-19T21:59:40.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:40.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:40.532206+0000) 2024-02-19T21:59:41.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:41.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:41.532503+0000) 2024-02-19T21:59:42.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:42.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:42.532794+0000) 2024-02-19T21:59:43.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:43.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:43.533090+0000) 2024-02-19T21:59:44.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:44.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:44.533344+0000) 2024-02-19T21:59:45.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:45.530+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:45.533634+0000) 2024-02-19T21:59:46.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:46.529+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:46.533933+0000) 2024-02-19T21:59:47.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:47.529+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:47.534158+0000) 2024-02-19T21:59:48.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:48.529+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:48.534427+0000) 2024-02-19T21:59:49.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:49.533+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:49.534766+0000) 2024-02-19T21:59:50.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:50.533+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:50.535050+0000) 2024-02-19T21:59:51.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:51.533+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:51.535345+0000) 2024-02-19T21:59:52.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:52.533+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:52.535647+0000) 2024-02-19T21:59:53.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:53.533+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:53.535945+0000) 2024-02-19T21:59:54.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:54.533+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:54.536161+0000) 2024-02-19T21:59:55.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:55.533+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:55.536335+0000) 2024-02-19T21:59:56.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:56.533+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:56.536561+0000) 2024-02-19T21:59:57.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:57.533+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:57.536799+0000) 2024-02-19T21:59:58.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:58.533+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:58.537080+0000) 2024-02-19T21:59:59.344 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:59:59.349 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T21:59:59.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T21:59:59.533+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T20:59:59.537299+0000) 2024-02-19T22:00:00.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:00.533+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:00.537457+0000) 2024-02-19T22:00:01.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:01.533+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:01.537672+0000) 2024-02-19T22:00:02.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:02.533+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:02.537945+0000) 2024-02-19T22:00:03.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:03.533+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:03.538172+0000) 2024-02-19T22:00:04.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:04.537+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:04.538366+0000) 2024-02-19T22:00:05.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:05.533+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:05.538608+0000) 2024-02-19T22:00:06.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:06.537+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:06.538848+0000) 2024-02-19T22:00:07.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:07.537+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:07.539104+0000) 2024-02-19T22:00:08.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:08.537+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:08.539327+0000) 2024-02-19T22:00:09.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:09.537+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:09.539545+0000) 2024-02-19T22:00:10.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:10.537+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:10.539796+0000) 2024-02-19T22:00:11.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:11.537+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:11.539977+0000) 2024-02-19T22:00:12.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:12.537+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:12.540218+0000) 2024-02-19T22:00:13.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:13.537+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:13.540424+0000) 2024-02-19T22:00:14.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:14.537+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:14.540654+0000) 2024-02-19T22:00:15.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:15.537+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:15.540929+0000) 2024-02-19T22:00:16.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:16.537+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:16.541057+0000) 2024-02-19T22:00:17.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:17.537+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:17.541304+0000) 2024-02-19T22:00:18.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:18.537+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:18.541554+0000) 2024-02-19T22:00:19.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:19.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:19.541817+0000) 2024-02-19T22:00:20.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:20.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:20.542021+0000) 2024-02-19T22:00:21.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:21.537+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:21.542257+0000) 2024-02-19T22:00:22.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:22.537+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:22.542453+0000) 2024-02-19T22:00:23.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:23.537+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:23.542642+0000) 2024-02-19T22:00:24.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:24.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:24.542812+0000) 2024-02-19T22:00:25.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:25.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:25.542946+0000) 2024-02-19T22:00:26.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:26.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:26.543067+0000) 2024-02-19T22:00:27.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:27.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:27.543243+0000) 2024-02-19T22:00:28.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:28.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:28.543486+0000) 2024-02-19T22:00:29.467 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:00:29.471 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:00:29.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:29.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:29.543732+0000) 2024-02-19T22:00:30.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:30.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:30.543903+0000) 2024-02-19T22:00:31.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:31.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:31.544085+0000) 2024-02-19T22:00:32.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:32.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:32.544322+0000) 2024-02-19T22:00:33.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:33.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:33.544574+0000) 2024-02-19T22:00:34.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:34.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:34.544779+0000) 2024-02-19T22:00:35.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:35.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:35.545057+0000) 2024-02-19T22:00:36.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:36.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:36.545279+0000) 2024-02-19T22:00:37.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:37.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:37.545462+0000) 2024-02-19T22:00:38.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:38.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:38.545730+0000) 2024-02-19T22:00:39.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:39.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:39.546004+0000) 2024-02-19T22:00:40.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:40.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:40.546189+0000) 2024-02-19T22:00:41.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:41.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:41.546368+0000) 2024-02-19T22:00:42.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:42.541+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:42.546549+0000) 2024-02-19T22:00:43.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:43.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:43.546776+0000) 2024-02-19T22:00:44.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:44.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:44.547036+0000) 2024-02-19T22:00:45.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:45.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:45.547264+0000) 2024-02-19T22:00:46.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:46.545+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:46.547447+0000) 2024-02-19T22:00:47.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:47.544+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:47.547729+0000) 2024-02-19T22:00:48.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:48.544+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:48.547958+0000) 2024-02-19T22:00:49.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:49.544+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:49.548119+0000) 2024-02-19T22:00:50.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:50.544+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:50.548336+0000) 2024-02-19T22:00:51.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:51.544+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:51.548584+0000) 2024-02-19T22:00:52.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:52.544+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:52.548778+0000) 2024-02-19T22:00:53.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:53.548+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:53.548986+0000) 2024-02-19T22:00:54.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:54.548+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:54.549243+0000) 2024-02-19T22:00:55.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:55.548+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:55.549447+0000) 2024-02-19T22:00:56.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:56.548+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:56.549703+0000) 2024-02-19T22:00:57.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:57.544+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:57.549952+0000) 2024-02-19T22:00:58.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:58.548+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:58.550201+0000) 2024-02-19T22:00:59.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:00:59.548+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:00:59.550408+0000) 2024-02-19T22:00:59.597 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:00:59.601 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:01:00.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:00.548+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:00.550665+0000) 2024-02-19T22:01:01.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:01.548+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:01.550907+0000) 2024-02-19T22:01:02.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:02.548+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:02.551142+0000) 2024-02-19T22:01:03.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:03.548+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:03.551395+0000) 2024-02-19T22:01:04.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:04.548+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:04.551588+0000) 2024-02-19T22:01:05.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:05.548+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:05.551864+0000) 2024-02-19T22:01:06.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:06.548+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:06.552112+0000) 2024-02-19T22:01:07.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:07.548+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:07.552361+0000) 2024-02-19T22:01:08.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:08.548+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:08.552594+0000) 2024-02-19T22:01:09.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:09.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:09.552851+0000) 2024-02-19T22:01:10.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:10.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:10.553048+0000) 2024-02-19T22:01:11.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:11.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:11.553295+0000) 2024-02-19T22:01:12.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:12.548+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:12.553491+0000) 2024-02-19T22:01:13.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:13.548+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:13.553722+0000) 2024-02-19T22:01:14.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:14.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:14.553947+0000) 2024-02-19T22:01:15.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:15.548+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:15.554190+0000) 2024-02-19T22:01:16.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:16.548+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:16.554452+0000) 2024-02-19T22:01:17.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:17.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:17.554700+0000) 2024-02-19T22:01:18.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:18.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:18.554890+0000) 2024-02-19T22:01:19.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:19.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:19.555115+0000) 2024-02-19T22:01:20.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:20.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:20.555311+0000) 2024-02-19T22:01:21.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:21.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:21.555488+0000) 2024-02-19T22:01:22.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:22.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:22.555688+0000) 2024-02-19T22:01:23.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:23.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:23.555931+0000) 2024-02-19T22:01:24.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:24.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:24.556162+0000) 2024-02-19T22:01:25.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:25.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:25.556428+0000) 2024-02-19T22:01:26.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:26.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:26.556693+0000) 2024-02-19T22:01:27.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:27.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:27.556950+0000) 2024-02-19T22:01:28.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:28.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:28.557187+0000) 2024-02-19T22:01:29.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:29.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:29.557428+0000) 2024-02-19T22:01:29.669 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:01:29.673 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:01:30.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:30.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:30.557623+0000) 2024-02-19T22:01:31.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:31.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:31.557870+0000) 2024-02-19T22:01:32.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:32.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:32.558094+0000) 2024-02-19T22:01:33.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:33.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:33.558333+0000) 2024-02-19T22:01:34.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:34.552+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:34.558567+0000) 2024-02-19T22:01:35.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:35.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:35.558845+0000) 2024-02-19T22:01:36.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:36.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:36.559078+0000) 2024-02-19T22:01:37.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:37.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:37.559359+0000) 2024-02-19T22:01:38.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:38.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:38.559551+0000) 2024-02-19T22:01:39.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:39.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:39.559802+0000) 2024-02-19T22:01:40.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:40.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:40.560074+0000) 2024-02-19T22:01:41.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:41.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:41.560244+0000) 2024-02-19T22:01:42.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:42.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:42.560511+0000) 2024-02-19T22:01:43.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:43.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:43.560738+0000) 2024-02-19T22:01:44.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:44.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:44.560961+0000) 2024-02-19T22:01:45.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:45.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:45.561198+0000) 2024-02-19T22:01:46.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:46.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:46.561427+0000) 2024-02-19T22:01:47.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:47.556+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:47.561671+0000) 2024-02-19T22:01:48.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:48.555+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:48.561915+0000) 2024-02-19T22:01:49.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:49.555+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:49.562120+0000) 2024-02-19T22:01:50.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:50.559+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:50.562354+0000) 2024-02-19T22:01:51.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:51.555+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:51.562516+0000) 2024-02-19T22:01:52.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:52.559+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:52.562704+0000) 2024-02-19T22:01:53.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:53.559+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:53.562969+0000) 2024-02-19T22:01:54.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:54.559+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:54.563101+0000) 2024-02-19T22:01:55.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:55.559+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:55.563364+0000) 2024-02-19T22:01:56.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:56.559+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:56.563522+0000) 2024-02-19T22:01:57.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:57.559+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:57.563788+0000) 2024-02-19T22:01:58.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:58.559+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:58.564022+0000) 2024-02-19T22:01:59.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:01:59.559+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:01:59.564206+0000) 2024-02-19T22:01:59.792 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:01:59.795 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:02:00.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:00.559+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:00.564452+0000) 2024-02-19T22:02:01.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:01.563+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:01.564695+0000) 2024-02-19T22:02:02.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:02.563+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:02.564930+0000) 2024-02-19T22:02:03.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:03.563+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:03.565162+0000) 2024-02-19T22:02:04.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:04.563+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:04.565393+0000) 2024-02-19T22:02:05.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:05.559+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:05.565653+0000) 2024-02-19T22:02:06.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:06.563+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:06.565887+0000) 2024-02-19T22:02:07.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:07.559+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:07.566038+0000) 2024-02-19T22:02:08.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:08.563+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:08.566274+0000) 2024-02-19T22:02:09.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:09.559+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:09.566523+0000) 2024-02-19T22:02:10.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:10.563+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:10.566809+0000) 2024-02-19T22:02:11.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:11.563+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:11.567042+0000) 2024-02-19T22:02:12.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:12.563+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:12.567320+0000) 2024-02-19T22:02:13.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:13.563+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:13.567592+0000) 2024-02-19T22:02:14.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:14.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:14.567846+0000) 2024-02-19T22:02:15.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:15.563+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:15.567992+0000) 2024-02-19T22:02:16.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:16.563+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:16.568156+0000) 2024-02-19T22:02:17.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:17.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:17.568342+0000) 2024-02-19T22:02:18.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:18.563+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:18.568590+0000) 2024-02-19T22:02:19.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:19.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:19.568796+0000) 2024-02-19T22:02:20.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:20.563+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:20.569033+0000) 2024-02-19T22:02:21.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:21.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:21.569280+0000) 2024-02-19T22:02:22.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:22.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:22.569533+0000) 2024-02-19T22:02:23.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:23.563+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:23.569725+0000) 2024-02-19T22:02:24.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:24.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:24.569959+0000) 2024-02-19T22:02:25.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:25.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:25.570221+0000) 2024-02-19T22:02:26.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:26.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:26.570453+0000) 2024-02-19T22:02:27.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:27.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:27.570663+0000) 2024-02-19T22:02:28.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:28.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:28.570908+0000) 2024-02-19T22:02:29.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:29.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:29.571142+0000) 2024-02-19T22:02:29.923 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:02:29.926 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:02:30.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:30.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:30.571332+0000) 2024-02-19T22:02:31.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:31.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:31.571474+0000) 2024-02-19T22:02:32.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:32.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:32.571728+0000) 2024-02-19T22:02:33.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:33.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:33.571978+0000) 2024-02-19T22:02:34.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:34.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:34.572216+0000) 2024-02-19T22:02:35.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:35.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:35.572458+0000) 2024-02-19T22:02:36.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:36.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:36.572657+0000) 2024-02-19T22:02:37.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:37.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:37.572897+0000) 2024-02-19T22:02:38.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:38.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:38.573135+0000) 2024-02-19T22:02:39.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:39.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:39.573378+0000) 2024-02-19T22:02:40.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:40.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:40.573613+0000) 2024-02-19T22:02:41.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:41.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:41.573811+0000) 2024-02-19T22:02:42.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:42.567+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:42.574047+0000) 2024-02-19T22:02:43.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:43.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:43.574318+0000) 2024-02-19T22:02:44.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:44.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:44.574554+0000) 2024-02-19T22:02:45.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:45.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:45.574861+0000) 2024-02-19T22:02:46.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:46.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:46.575047+0000) 2024-02-19T22:02:47.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:47.571+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:47.575256+0000) 2024-02-19T22:02:48.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:48.570+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:48.575393+0000) 2024-02-19T22:02:49.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:49.574+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:49.575620+0000) 2024-02-19T22:02:50.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:50.570+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:50.575838+0000) 2024-02-19T22:02:51.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:51.574+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:51.576046+0000) 2024-02-19T22:02:52.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:52.570+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:52.576293+0000) 2024-02-19T22:02:53.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:53.574+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:53.576541+0000) 2024-02-19T22:02:54.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:54.574+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:54.576823+0000) 2024-02-19T22:02:55.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:55.570+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:55.577064+0000) 2024-02-19T22:02:56.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:56.570+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:56.577231+0000) 2024-02-19T22:02:57.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:57.570+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:57.577483+0000) 2024-02-19T22:02:58.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:58.574+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:58.577686+0000) 2024-02-19T22:02:59.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:02:59.570+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:02:59.577945+0000) 2024-02-19T22:02:59.987 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:02:59.990 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:03:00.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:00.570+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:00.578097+0000) 2024-02-19T22:03:01.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:01.574+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:01.578326+0000) 2024-02-19T22:03:02.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:02.570+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:02.578571+0000) 2024-02-19T22:03:03.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:03.574+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:03.578841+0000) 2024-02-19T22:03:04.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:04.578+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:04.579044+0000) 2024-02-19T22:03:05.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:05.578+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:05.579305+0000) 2024-02-19T22:03:06.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:06.578+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:06.579553+0000) 2024-02-19T22:03:07.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:07.578+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:07.579800+0000) 2024-02-19T22:03:08.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:08.578+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:08.580033+0000) 2024-02-19T22:03:09.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:09.578+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:09.580282+0000) 2024-02-19T22:03:10.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:10.578+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:10.580530+0000) 2024-02-19T22:03:11.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:11.578+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:11.580772+0000) 2024-02-19T22:03:12.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:12.578+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:12.581006+0000) 2024-02-19T22:03:13.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:13.578+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:13.581189+0000) 2024-02-19T22:03:14.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:14.578+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:14.581422+0000) 2024-02-19T22:03:15.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:15.578+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:15.581711+0000) 2024-02-19T22:03:16.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:16.578+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:16.581957+0000) 2024-02-19T22:03:17.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:17.578+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:17.582197+0000) 2024-02-19T22:03:18.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:18.578+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:18.582438+0000) 2024-02-19T22:03:19.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:19.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:19.582696+0000) 2024-02-19T22:03:20.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:20.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:20.582852+0000) 2024-02-19T22:03:21.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:21.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:21.583038+0000) 2024-02-19T22:03:22.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:22.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:22.583277+0000) 2024-02-19T22:03:23.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:23.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:23.583443+0000) 2024-02-19T22:03:24.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:24.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:24.583697+0000) 2024-02-19T22:03:25.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:25.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:25.583936+0000) 2024-02-19T22:03:26.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:26.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:26.584170+0000) 2024-02-19T22:03:27.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:27.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:27.584376+0000) 2024-02-19T22:03:28.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:28.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:28.584615+0000) 2024-02-19T22:03:29.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:29.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:29.584871+0000) 2024-02-19T22:03:30.117 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:03:30.120 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:03:30.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:30.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:30.585095+0000) 2024-02-19T22:03:31.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:31.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:31.585349+0000) 2024-02-19T22:03:32.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:32.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:32.585528+0000) 2024-02-19T22:03:33.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:33.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:33.585755+0000) 2024-02-19T22:03:34.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:34.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:34.585991+0000) 2024-02-19T22:03:35.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:35.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:35.586220+0000) 2024-02-19T22:03:36.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:36.582+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:36.586451+0000) 2024-02-19T22:03:37.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:37.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:37.586701+0000) 2024-02-19T22:03:38.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:38.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:38.586950+0000) 2024-02-19T22:03:39.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:39.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:39.587180+0000) 2024-02-19T22:03:40.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:40.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:40.587452+0000) 2024-02-19T22:03:41.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:41.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:41.587685+0000) 2024-02-19T22:03:42.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:42.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:42.587945+0000) 2024-02-19T22:03:43.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:43.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:43.588173+0000) 2024-02-19T22:03:44.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:44.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:44.588439+0000) 2024-02-19T22:03:45.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:45.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:45.588697+0000) 2024-02-19T22:03:46.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:46.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:46.588951+0000) 2024-02-19T22:03:47.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:47.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:47.589206+0000) 2024-02-19T22:03:48.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:48.586+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:48.589388+0000) 2024-02-19T22:03:49.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:49.585+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:49.589628+0000) 2024-02-19T22:03:50.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:50.585+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:50.589814+0000) 2024-02-19T22:03:51.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:51.585+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:51.590045+0000) 2024-02-19T22:03:52.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:52.585+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:52.590293+0000) 2024-02-19T22:03:53.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:53.585+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:53.590564+0000) 2024-02-19T22:03:54.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:54.589+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:54.590794+0000) 2024-02-19T22:03:55.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:55.589+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:55.590959+0000) 2024-02-19T22:03:56.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:56.589+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:56.591182+0000) 2024-02-19T22:03:57.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:57.589+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:57.591359+0000) 2024-02-19T22:03:58.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:58.589+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:58.591584+0000) 2024-02-19T22:03:59.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:03:59.589+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:03:59.591821+0000) 2024-02-19T22:04:00.198 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:04:00.277 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:04:00.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:00.589+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:00.592030+0000) 2024-02-19T22:04:01.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:01.589+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:01.592265+0000) 2024-02-19T22:04:02.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:02.589+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:02.592492+0000) 2024-02-19T22:04:03.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:03.589+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:03.592713+0000) 2024-02-19T22:04:04.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:04.589+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:04.592955+0000) 2024-02-19T22:04:05.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:05.589+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:05.593141+0000) 2024-02-19T22:04:06.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:06.589+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:06.593331+0000) 2024-02-19T22:04:07.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:07.589+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:07.593508+0000) 2024-02-19T22:04:08.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:08.589+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:08.593749+0000) 2024-02-19T22:04:09.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:09.589+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:09.594019+0000) 2024-02-19T22:04:10.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:10.589+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:10.594241+0000) 2024-02-19T22:04:11.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:11.589+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:11.594423+0000) 2024-02-19T22:04:12.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:12.589+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:12.594668+0000) 2024-02-19T22:04:13.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:13.593+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:13.594876+0000) 2024-02-19T22:04:14.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:14.593+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:14.595087+0000) 2024-02-19T22:04:15.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:15.593+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:15.595309+0000) 2024-02-19T22:04:16.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:16.593+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:16.595550+0000) 2024-02-19T22:04:17.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:17.593+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:17.595786+0000) 2024-02-19T22:04:18.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:18.593+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:18.595955+0000) 2024-02-19T22:04:19.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:19.593+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:19.596220+0000) 2024-02-19T22:04:20.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:20.593+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:20.596457+0000) 2024-02-19T22:04:21.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:21.593+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:21.596696+0000) 2024-02-19T22:04:22.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:22.593+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:22.596832+0000) 2024-02-19T22:04:23.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:23.593+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:23.596995+0000) 2024-02-19T22:04:24.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:24.593+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:24.597167+0000) 2024-02-19T22:04:25.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:25.593+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:25.597394+0000) 2024-02-19T22:04:26.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:26.593+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:26.597631+0000) 2024-02-19T22:04:27.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:27.593+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:27.597867+0000) 2024-02-19T22:04:28.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:28.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:28.598142+0000) 2024-02-19T22:04:29.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:29.593+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:29.598408+0000) 2024-02-19T22:04:30.420 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:04:30.424 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:04:30.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:30.593+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:30.598597+0000) 2024-02-19T22:04:31.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:31.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:31.598830+0000) 2024-02-19T22:04:32.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:32.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:32.598999+0000) 2024-02-19T22:04:33.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:33.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:33.599191+0000) 2024-02-19T22:04:34.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:34.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:34.599442+0000) 2024-02-19T22:04:35.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:35.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:35.599680+0000) 2024-02-19T22:04:36.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:36.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:36.599855+0000) 2024-02-19T22:04:37.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:37.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:37.600049+0000) 2024-02-19T22:04:38.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:38.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:38.600167+0000) 2024-02-19T22:04:39.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:39.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:39.600429+0000) 2024-02-19T22:04:40.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:40.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:40.600697+0000) 2024-02-19T22:04:41.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:41.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:41.600935+0000) 2024-02-19T22:04:42.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:42.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:42.601163+0000) 2024-02-19T22:04:43.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:43.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:43.601405+0000) 2024-02-19T22:04:44.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:44.601+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:44.601662+0000) 2024-02-19T22:04:45.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:45.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:45.601884+0000) 2024-02-19T22:04:46.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:46.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:46.602026+0000) 2024-02-19T22:04:47.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:47.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:47.602190+0000) 2024-02-19T22:04:48.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:48.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:48.602329+0000) 2024-02-19T22:04:49.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:49.597+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:49.602547+0000) 2024-02-19T22:04:50.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:50.600+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:50.602781+0000) 2024-02-19T22:04:51.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:51.600+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:51.603008+0000) 2024-02-19T22:04:52.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:52.600+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:52.603201+0000) 2024-02-19T22:04:53.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:53.600+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:53.603379+0000) 2024-02-19T22:04:54.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:54.600+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:54.603516+0000) 2024-02-19T22:04:55.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:55.600+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:55.603650+0000) 2024-02-19T22:04:56.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:56.600+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:56.603801+0000) 2024-02-19T22:04:57.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:57.600+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:57.604027+0000) 2024-02-19T22:04:58.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:58.600+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:58.604235+0000) 2024-02-19T22:04:59.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:04:59.600+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:04:59.604408+0000) 2024-02-19T22:05:00.543 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:05:00.547 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:05:00.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:00.600+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:00.604658+0000) 2024-02-19T22:05:01.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:01.600+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:01.604902+0000) 2024-02-19T22:05:02.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:02.600+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:02.605096+0000) 2024-02-19T22:05:03.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:03.600+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:03.605309+0000) 2024-02-19T22:05:04.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:04.600+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:04.605478+0000) 2024-02-19T22:05:05.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:05.600+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:05.605659+0000) 2024-02-19T22:05:06.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:06.604+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:06.605874+0000) 2024-02-19T22:05:07.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:07.604+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:07.606157+0000) 2024-02-19T22:05:08.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:08.604+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:08.606381+0000) 2024-02-19T22:05:09.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:09.604+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:09.606568+0000) 2024-02-19T22:05:10.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:10.604+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:10.606729+0000) 2024-02-19T22:05:11.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:11.604+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:11.606857+0000) 2024-02-19T22:05:12.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:12.604+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:12.607060+0000) 2024-02-19T22:05:13.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:13.604+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:13.607314+0000) 2024-02-19T22:05:14.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:14.604+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:14.607557+0000) 2024-02-19T22:05:15.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:15.604+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:15.607803+0000) 2024-02-19T22:05:16.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:16.604+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:16.608047+0000) 2024-02-19T22:05:17.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:17.604+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:17.608303+0000) 2024-02-19T22:05:18.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:18.604+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:18.608541+0000) 2024-02-19T22:05:19.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:19.604+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:19.608741+0000) 2024-02-19T22:05:20.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:20.604+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:20.608996+0000) 2024-02-19T22:05:21.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:21.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:21.609244+0000) 2024-02-19T22:05:22.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:22.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:22.609481+0000) 2024-02-19T22:05:23.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:23.604+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:23.609760+0000) 2024-02-19T22:05:24.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:24.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:24.609961+0000) 2024-02-19T22:05:25.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:25.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:25.610164+0000) 2024-02-19T22:05:26.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:26.604+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:26.610402+0000) 2024-02-19T22:05:27.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:27.604+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:27.610634+0000) 2024-02-19T22:05:28.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:28.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:28.610826+0000) 2024-02-19T22:05:29.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:29.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:29.611005+0000) 2024-02-19T22:05:30.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:30.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:30.611187+0000) 2024-02-19T22:05:30.673 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:05:30.678 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:05:31.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:31.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:31.611467+0000) 2024-02-19T22:05:32.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:32.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:32.611719+0000) 2024-02-19T22:05:33.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:33.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:33.611931+0000) 2024-02-19T22:05:34.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:34.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:34.612088+0000) 2024-02-19T22:05:35.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:35.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:35.612253+0000) 2024-02-19T22:05:36.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:36.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:36.612483+0000) 2024-02-19T22:05:37.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:37.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:37.612707+0000) 2024-02-19T22:05:38.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:38.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:38.612890+0000) 2024-02-19T22:05:39.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:39.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:39.613128+0000) 2024-02-19T22:05:40.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:40.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:40.613372+0000) 2024-02-19T22:05:41.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:41.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:41.613658+0000) 2024-02-19T22:05:42.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:42.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:42.613938+0000) 2024-02-19T22:05:43.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:43.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:43.614254+0000) 2024-02-19T22:05:44.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:44.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:44.614481+0000) 2024-02-19T22:05:45.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:45.608+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:45.614680+0000) 2024-02-19T22:05:46.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:46.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:46.614912+0000) 2024-02-19T22:05:47.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:47.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:47.615132+0000) 2024-02-19T22:05:48.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:48.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:48.615318+0000) 2024-02-19T22:05:49.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:49.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:49.615590+0000) 2024-02-19T22:05:50.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:50.612+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:50.615788+0000) 2024-02-19T22:05:51.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:51.611+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:51.615963+0000) 2024-02-19T22:05:52.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:52.611+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:52.616209+0000) 2024-02-19T22:05:53.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:53.615+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:53.616480+0000) 2024-02-19T22:05:54.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:54.615+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:54.616720+0000) 2024-02-19T22:05:55.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:55.615+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:55.616929+0000) 2024-02-19T22:05:56.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:56.611+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:56.617151+0000) 2024-02-19T22:05:57.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:57.615+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:57.617337+0000) 2024-02-19T22:05:58.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:58.611+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:58.617457+0000) 2024-02-19T22:05:59.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:05:59.611+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:05:59.617728+0000) 2024-02-19T22:06:00.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:00.615+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:00.617912+0000) 2024-02-19T22:06:00.729 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:06:00.733 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:06:01.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:01.611+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:01.618091+0000) 2024-02-19T22:06:02.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:02.615+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:02.618310+0000) 2024-02-19T22:06:03.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:03.615+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:03.618542+0000) 2024-02-19T22:06:04.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:04.615+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:04.618807+0000) 2024-02-19T22:06:05.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:05.615+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:05.618934+0000) 2024-02-19T22:06:06.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:06.615+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:06.619151+0000) 2024-02-19T22:06:07.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:07.615+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:07.619396+0000) 2024-02-19T22:06:08.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:08.615+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:08.619586+0000) 2024-02-19T22:06:09.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:09.615+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:09.619781+0000) 2024-02-19T22:06:10.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:10.619+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:10.620010+0000) 2024-02-19T22:06:11.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:11.615+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:11.620247+0000) 2024-02-19T22:06:12.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:12.619+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:12.620481+0000) 2024-02-19T22:06:13.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:13.615+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:13.620678+0000) 2024-02-19T22:06:14.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:14.619+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:14.620887+0000) 2024-02-19T22:06:15.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:15.619+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:15.621123+0000) 2024-02-19T22:06:16.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:16.619+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:16.621340+0000) 2024-02-19T22:06:17.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:17.619+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:17.621538+0000) 2024-02-19T22:06:18.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:18.619+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:18.621729+0000) 2024-02-19T22:06:19.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:19.615+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:19.621906+0000) 2024-02-19T22:06:20.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:20.615+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:20.622132+0000) 2024-02-19T22:06:21.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:21.615+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:21.622283+0000) 2024-02-19T22:06:22.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:22.619+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:22.622538+0000) 2024-02-19T22:06:23.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:23.619+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:23.622825+0000) 2024-02-19T22:06:24.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:24.619+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:24.623062+0000) 2024-02-19T22:06:25.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:25.619+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:25.623350+0000) 2024-02-19T22:06:26.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:26.619+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:26.623491+0000) 2024-02-19T22:06:27.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:27.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:27.623742+0000) 2024-02-19T22:06:28.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:28.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:28.623942+0000) 2024-02-19T22:06:29.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:29.619+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:29.624213+0000) 2024-02-19T22:06:30.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:30.619+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:30.624461+0000) 2024-02-19T22:06:30.860 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:06:30.863 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:06:31.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:31.619+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:31.624750+0000) 2024-02-19T22:06:32.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:32.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:32.624930+0000) 2024-02-19T22:06:33.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:33.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:33.625115+0000) 2024-02-19T22:06:34.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:34.619+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:34.625358+0000) 2024-02-19T22:06:35.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:35.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:35.625563+0000) 2024-02-19T22:06:36.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:36.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:36.625750+0000) 2024-02-19T22:06:37.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:37.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:37.625983+0000) 2024-02-19T22:06:38.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:38.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:38.626177+0000) 2024-02-19T22:06:39.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:39.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:39.626391+0000) 2024-02-19T22:06:40.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:40.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:40.626623+0000) 2024-02-19T22:06:41.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:41.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:41.626860+0000) 2024-02-19T22:06:42.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:42.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:42.627026+0000) 2024-02-19T22:06:43.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:43.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:43.627245+0000) 2024-02-19T22:06:44.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:44.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:44.627427+0000) 2024-02-19T22:06:45.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:45.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:45.627628+0000) 2024-02-19T22:06:46.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:46.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:46.627850+0000) 2024-02-19T22:06:47.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:47.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:47.628095+0000) 2024-02-19T22:06:48.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:48.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:48.628386+0000) 2024-02-19T22:06:49.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:49.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:49.628618+0000) 2024-02-19T22:06:50.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:50.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:50.628834+0000) 2024-02-19T22:06:51.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:51.623+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:51.628982+0000) 2024-02-19T22:06:52.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:52.622+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:52.629196+0000) 2024-02-19T22:06:53.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:53.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:53.629433+0000) 2024-02-19T22:06:54.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:54.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:54.629644+0000) 2024-02-19T22:06:55.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:55.622+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:55.629837+0000) 2024-02-19T22:06:56.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:56.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:56.630023+0000) 2024-02-19T22:06:57.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:57.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:57.630255+0000) 2024-02-19T22:06:58.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:58.622+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:58.630492+0000) 2024-02-19T22:06:59.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:06:59.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:06:59.630762+0000) 2024-02-19T22:07:00.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:00.630+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:00.630940+0000) 2024-02-19T22:07:00.975 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:07:00.979 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:07:01.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:01.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:01.631152+0000) 2024-02-19T22:07:02.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:02.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:02.631322+0000) 2024-02-19T22:07:03.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:03.630+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:03.631571+0000) 2024-02-19T22:07:04.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:04.630+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:04.631809+0000) 2024-02-19T22:07:05.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:05.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:05.631992+0000) 2024-02-19T22:07:06.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:06.630+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:06.632161+0000) 2024-02-19T22:07:07.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:07.626+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:07.632284+0000) 2024-02-19T22:07:08.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:08.630+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:08.632509+0000) 2024-02-19T22:07:09.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:09.630+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:09.632757+0000) 2024-02-19T22:07:10.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:10.630+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:10.633014+0000) 2024-02-19T22:07:11.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:11.630+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:11.633221+0000) 2024-02-19T22:07:12.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:12.630+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:12.633444+0000) 2024-02-19T22:07:13.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:13.630+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:13.633681+0000) 2024-02-19T22:07:14.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:14.630+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:14.633932+0000) 2024-02-19T22:07:15.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:15.630+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:15.634102+0000) 2024-02-19T22:07:16.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:16.630+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:16.634336+0000) 2024-02-19T22:07:17.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:17.630+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:17.634561+0000) 2024-02-19T22:07:18.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:18.634+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:18.634832+0000) 2024-02-19T22:07:19.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:19.634+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:19.635088+0000) 2024-02-19T22:07:20.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:20.634+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:20.635359+0000) 2024-02-19T22:07:21.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:21.634+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:21.635583+0000) 2024-02-19T22:07:22.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:22.634+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:22.635741+0000) 2024-02-19T22:07:23.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:23.634+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:23.635962+0000) 2024-02-19T22:07:24.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:24.634+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:24.636185+0000) 2024-02-19T22:07:25.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:25.634+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:25.636412+0000) 2024-02-19T22:07:26.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:26.634+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:26.636651+0000) 2024-02-19T22:07:27.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:27.634+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:27.636898+0000) 2024-02-19T22:07:28.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:28.634+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:28.637132+0000) 2024-02-19T22:07:29.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:29.634+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:29.637383+0000) 2024-02-19T22:07:30.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:30.634+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:30.637619+0000) 2024-02-19T22:07:31.130 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:07:31.134 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:07:31.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:31.634+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:31.637859+0000) 2024-02-19T22:07:32.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:32.634+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:32.638103+0000) 2024-02-19T22:07:33.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:33.634+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:33.638353+0000) 2024-02-19T22:07:34.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:34.634+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:34.638592+0000) 2024-02-19T22:07:35.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:35.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:35.638785+0000) 2024-02-19T22:07:36.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:36.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:36.638979+0000) 2024-02-19T22:07:37.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:37.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:37.639233+0000) 2024-02-19T22:07:38.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:38.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:38.639471+0000) 2024-02-19T22:07:39.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:39.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:39.639723+0000) 2024-02-19T22:07:40.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:40.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:40.639956+0000) 2024-02-19T22:07:41.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:41.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:41.640210+0000) 2024-02-19T22:07:42.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:42.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:42.640480+0000) 2024-02-19T22:07:43.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:43.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:43.640750+0000) 2024-02-19T22:07:44.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:44.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:44.640920+0000) 2024-02-19T22:07:45.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:45.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:45.641087+0000) 2024-02-19T22:07:46.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:46.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:46.641343+0000) 2024-02-19T22:07:47.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:47.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:47.641506+0000) 2024-02-19T22:07:48.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:48.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:48.641688+0000) 2024-02-19T22:07:49.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:49.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:49.641923+0000) 2024-02-19T22:07:50.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:50.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:50.642129+0000) 2024-02-19T22:07:51.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:51.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:51.642330+0000) 2024-02-19T22:07:52.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:52.637+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:52.642508+0000) 2024-02-19T22:07:53.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:53.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:53.642691+0000) 2024-02-19T22:07:54.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:54.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:54.642912+0000) 2024-02-19T22:07:55.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:55.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:55.643137+0000) 2024-02-19T22:07:56.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:56.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:56.643387+0000) 2024-02-19T22:07:57.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:57.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:57.643642+0000) 2024-02-19T22:07:58.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:58.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:58.643841+0000) 2024-02-19T22:07:59.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:07:59.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:07:59.644088+0000) 2024-02-19T22:08:00.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:00.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:00.644314+0000) 2024-02-19T22:08:01.261 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:08:01.264 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:08:01.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:01.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:01.644529+0000) 2024-02-19T22:08:02.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:02.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:02.644733+0000) 2024-02-19T22:08:03.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:03.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:03.644997+0000) 2024-02-19T22:08:04.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:04.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:04.645196+0000) 2024-02-19T22:08:05.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:05.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:05.645405+0000) 2024-02-19T22:08:06.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:06.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:06.645652+0000) 2024-02-19T22:08:07.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:07.645+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:07.645842+0000) 2024-02-19T22:08:08.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:08.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:08.646100+0000) 2024-02-19T22:08:09.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:09.641+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:09.646335+0000) 2024-02-19T22:08:10.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:10.645+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:10.646613+0000) 2024-02-19T22:08:11.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:11.645+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:11.646847+0000) 2024-02-19T22:08:12.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:12.645+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:12.647023+0000) 2024-02-19T22:08:13.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:13.645+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:13.647274+0000) 2024-02-19T22:08:14.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:14.645+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:14.647445+0000) 2024-02-19T22:08:15.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:15.645+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:15.647701+0000) 2024-02-19T22:08:16.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:16.645+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:16.647914+0000) 2024-02-19T22:08:17.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:17.645+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:17.648186+0000) 2024-02-19T22:08:18.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:18.645+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:18.648428+0000) 2024-02-19T22:08:19.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:19.645+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:19.648701+0000) 2024-02-19T22:08:20.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:20.645+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:20.648941+0000) 2024-02-19T22:08:21.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:21.645+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:21.649207+0000) 2024-02-19T22:08:22.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:22.645+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:22.649374+0000) 2024-02-19T22:08:23.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:23.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:23.649621+0000) 2024-02-19T22:08:24.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:24.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:24.649862+0000) 2024-02-19T22:08:25.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:25.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:25.650102+0000) 2024-02-19T22:08:26.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:26.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:26.650342+0000) 2024-02-19T22:08:27.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:27.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:27.650586+0000) 2024-02-19T22:08:28.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:28.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:28.650821+0000) 2024-02-19T22:08:29.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:29.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:29.651061+0000) 2024-02-19T22:08:30.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:30.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:30.651254+0000) 2024-02-19T22:08:31.309 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:08:31.312 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:08:31.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:31.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:31.651446+0000) 2024-02-19T22:08:32.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:32.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:32.651674+0000) 2024-02-19T22:08:33.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:33.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:33.651951+0000) 2024-02-19T22:08:34.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:34.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:34.652121+0000) 2024-02-19T22:08:35.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:35.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:35.652394+0000) 2024-02-19T22:08:36.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:36.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:36.652670+0000) 2024-02-19T22:08:37.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:37.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:37.652871+0000) 2024-02-19T22:08:38.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:38.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:38.653027+0000) 2024-02-19T22:08:39.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:39.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:39.653260+0000) 2024-02-19T22:08:40.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:40.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:40.653501+0000) 2024-02-19T22:08:41.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:41.649+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:41.653756+0000) 2024-02-19T22:08:42.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:42.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:42.654028+0000) 2024-02-19T22:08:43.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:43.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:43.654264+0000) 2024-02-19T22:08:44.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:44.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:44.654523+0000) 2024-02-19T22:08:45.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:45.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:45.654774+0000) 2024-02-19T22:08:46.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:46.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:46.655001+0000) 2024-02-19T22:08:47.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:47.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:47.655266+0000) 2024-02-19T22:08:48.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:48.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:48.655513+0000) 2024-02-19T22:08:49.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:49.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:49.655742+0000) 2024-02-19T22:08:50.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:50.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:50.655999+0000) 2024-02-19T22:08:51.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:51.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:51.656239+0000) 2024-02-19T22:08:52.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:52.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:52.656408+0000) 2024-02-19T22:08:53.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:53.652+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:53.656652+0000) 2024-02-19T22:08:54.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:54.652+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:54.656917+0000) 2024-02-19T22:08:55.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:55.656+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:55.657162+0000) 2024-02-19T22:08:56.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:56.656+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:56.657383+0000) 2024-02-19T22:08:57.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:57.656+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:57.657611+0000) 2024-02-19T22:08:58.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:58.652+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:58.657802+0000) 2024-02-19T22:08:59.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:08:59.652+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:08:59.657964+0000) 2024-02-19T22:09:00.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:00.656+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:00.658175+0000) 2024-02-19T22:09:01.382 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:09:01.385 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:09:01.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:01.656+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:01.658456+0000) 2024-02-19T22:09:02.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:02.656+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:02.658651+0000) 2024-02-19T22:09:03.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:03.656+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:03.658842+0000) 2024-02-19T22:09:04.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:04.656+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:04.659085+0000) 2024-02-19T22:09:05.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:05.656+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:05.659262+0000) 2024-02-19T22:09:06.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:06.656+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:06.659465+0000) 2024-02-19T22:09:07.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:07.656+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:07.659674+0000) 2024-02-19T22:09:08.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:08.656+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:08.659914+0000) 2024-02-19T22:09:09.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:09.656+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:09.660092+0000) 2024-02-19T22:09:10.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:10.656+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:10.660345+0000) 2024-02-19T22:09:11.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:11.656+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:11.660527+0000) 2024-02-19T22:09:12.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:12.656+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:12.660705+0000) 2024-02-19T22:09:13.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:13.660+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:13.660979+0000) 2024-02-19T22:09:14.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:14.660+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:14.661170+0000) 2024-02-19T22:09:15.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:15.660+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:15.661445+0000) 2024-02-19T22:09:16.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:16.660+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:16.661681+0000) 2024-02-19T22:09:17.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:17.656+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:17.661898+0000) 2024-02-19T22:09:18.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:18.660+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:18.662155+0000) 2024-02-19T22:09:19.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:19.660+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:19.662405+0000) 2024-02-19T22:09:20.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:20.660+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:20.662645+0000) 2024-02-19T22:09:21.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:21.660+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:21.662924+0000) 2024-02-19T22:09:22.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:22.660+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:22.663095+0000) 2024-02-19T22:09:23.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:23.660+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:23.663284+0000) 2024-02-19T22:09:24.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:24.660+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:24.663478+0000) 2024-02-19T22:09:25.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:25.660+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:25.663736+0000) 2024-02-19T22:09:26.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:26.660+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:26.663908+0000) 2024-02-19T22:09:27.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:27.660+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:27.664091+0000) 2024-02-19T22:09:28.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:28.660+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:28.664291+0000) 2024-02-19T22:09:29.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:29.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:29.664485+0000) 2024-02-19T22:09:30.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:30.660+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:30.664723+0000) 2024-02-19T22:09:31.445 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:09:31.449 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:09:31.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:31.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:31.664978+0000) 2024-02-19T22:09:32.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:32.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:32.665260+0000) 2024-02-19T22:09:33.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:33.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:33.665518+0000) 2024-02-19T22:09:34.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:34.660+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:34.665673+0000) 2024-02-19T22:09:35.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:35.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:35.665857+0000) 2024-02-19T22:09:36.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:36.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:36.666073+0000) 2024-02-19T22:09:37.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:37.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:37.666261+0000) 2024-02-19T22:09:38.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:38.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:38.666504+0000) 2024-02-19T22:09:39.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:39.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:39.666805+0000) 2024-02-19T22:09:40.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:40.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:40.666946+0000) 2024-02-19T22:09:41.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:41.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:41.667122+0000) 2024-02-19T22:09:42.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:42.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:42.667308+0000) 2024-02-19T22:09:43.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:43.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:43.667533+0000) 2024-02-19T22:09:44.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:44.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:44.667776+0000) 2024-02-19T22:09:45.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:45.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:45.668013+0000) 2024-02-19T22:09:46.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:46.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:46.668195+0000) 2024-02-19T22:09:47.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:47.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:47.668456+0000) 2024-02-19T22:09:48.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:48.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:48.668640+0000) 2024-02-19T22:09:49.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:49.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:49.668917+0000) 2024-02-19T22:09:50.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:50.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:50.669194+0000) 2024-02-19T22:09:51.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:51.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:51.669488+0000) 2024-02-19T22:09:52.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:52.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:52.669717+0000) 2024-02-19T22:09:53.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:53.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:53.669993+0000) 2024-02-19T22:09:54.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:54.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:54.670214+0000) 2024-02-19T22:09:55.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:55.663+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:55.670390+0000) 2024-02-19T22:09:56.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:56.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:56.670609+0000) 2024-02-19T22:09:57.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:57.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:57.670863+0000) 2024-02-19T22:09:58.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:58.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:58.670992+0000) 2024-02-19T22:09:59.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:09:59.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:09:59.671222+0000) 2024-02-19T22:10:00.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:00.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:00.671470+0000) 2024-02-19T22:10:01.500 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:10:01.503 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:10:01.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:01.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:01.671723+0000) 2024-02-19T22:10:02.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:02.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:02.671958+0000) 2024-02-19T22:10:03.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:03.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:03.672190+0000) 2024-02-19T22:10:04.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:04.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:04.672439+0000) 2024-02-19T22:10:05.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:05.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:05.672635+0000) 2024-02-19T22:10:06.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:06.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:06.672851+0000) 2024-02-19T22:10:07.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:07.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:07.673092+0000) 2024-02-19T22:10:08.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:08.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:08.673269+0000) 2024-02-19T22:10:09.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:09.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:09.673503+0000) 2024-02-19T22:10:10.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:10.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:10.673692+0000) 2024-02-19T22:10:11.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:11.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:11.673917+0000) 2024-02-19T22:10:12.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:12.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:12.674087+0000) 2024-02-19T22:10:13.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:13.667+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:13.674342+0000) 2024-02-19T22:10:14.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:14.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:14.674516+0000) 2024-02-19T22:10:15.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:15.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:15.674812+0000) 2024-02-19T22:10:16.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:16.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:16.675054+0000) 2024-02-19T22:10:17.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:17.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:17.675227+0000) 2024-02-19T22:10:18.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:18.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:18.675396+0000) 2024-02-19T22:10:19.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:19.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:19.675644+0000) 2024-02-19T22:10:20.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:20.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:20.675880+0000) 2024-02-19T22:10:21.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:21.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:21.676146+0000) 2024-02-19T22:10:22.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:22.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:22.676404+0000) 2024-02-19T22:10:23.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:23.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:23.676639+0000) 2024-02-19T22:10:24.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:24.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:24.676817+0000) 2024-02-19T22:10:25.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:25.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:25.676986+0000) 2024-02-19T22:10:26.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:26.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:26.677156+0000) 2024-02-19T22:10:27.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:27.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:27.677327+0000) 2024-02-19T22:10:28.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:28.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:28.677570+0000) 2024-02-19T22:10:29.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:29.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:29.677828+0000) 2024-02-19T22:10:30.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:30.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:30.678004+0000) 2024-02-19T22:10:31.614 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:10:31.617 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:10:31.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:31.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:31.678198+0000) 2024-02-19T22:10:32.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:32.671+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:32.678405+0000) 2024-02-19T22:10:33.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:33.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:33.678645+0000) 2024-02-19T22:10:34.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:34.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:34.678921+0000) 2024-02-19T22:10:35.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:35.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:35.679137+0000) 2024-02-19T22:10:36.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:36.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:36.679352+0000) 2024-02-19T22:10:37.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:37.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:37.679589+0000) 2024-02-19T22:10:38.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:38.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:38.679810+0000) 2024-02-19T22:10:39.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:39.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:39.679951+0000) 2024-02-19T22:10:40.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:40.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:40.680191+0000) 2024-02-19T22:10:41.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:41.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:41.680386+0000) 2024-02-19T22:10:42.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:42.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:42.680604+0000) 2024-02-19T22:10:43.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:43.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:43.680796+0000) 2024-02-19T22:10:44.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:44.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:44.680975+0000) 2024-02-19T22:10:45.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:45.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:45.681210+0000) 2024-02-19T22:10:46.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:46.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:46.681392+0000) 2024-02-19T22:10:47.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:47.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:47.681626+0000) 2024-02-19T22:10:48.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:48.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:48.681837+0000) 2024-02-19T22:10:49.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:49.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:49.682121+0000) 2024-02-19T22:10:50.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:50.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:50.682329+0000) 2024-02-19T22:10:51.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:51.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:51.682536+0000) 2024-02-19T22:10:52.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:52.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:52.682814+0000) 2024-02-19T22:10:53.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:53.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:53.682983+0000) 2024-02-19T22:10:54.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:54.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:54.683147+0000) 2024-02-19T22:10:55.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:55.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:55.683337+0000) 2024-02-19T22:10:56.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:56.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:56.683548+0000) 2024-02-19T22:10:57.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:57.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:57.683771+0000) 2024-02-19T22:10:58.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:58.682+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:58.684013+0000) 2024-02-19T22:10:59.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:10:59.682+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:10:59.684223+0000) 2024-02-19T22:11:00.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:00.682+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:00.684399+0000) 2024-02-19T22:11:01.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:01.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:01.684579+0000) 2024-02-19T22:11:01.761 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:11:01.764 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:11:02.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:02.682+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:02.684768+0000) 2024-02-19T22:11:03.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:03.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:03.684998+0000) 2024-02-19T22:11:04.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:04.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:04.685234+0000) 2024-02-19T22:11:05.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:05.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:05.685483+0000) 2024-02-19T22:11:06.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:06.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:06.685679+0000) 2024-02-19T22:11:07.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:07.682+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:07.685926+0000) 2024-02-19T22:11:08.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:08.682+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:08.686172+0000) 2024-02-19T22:11:09.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:09.678+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:09.686350+0000) 2024-02-19T22:11:10.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:10.682+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:10.686531+0000) 2024-02-19T22:11:11.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:11.682+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:11.686796+0000) 2024-02-19T22:11:12.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:12.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:12.687014+0000) 2024-02-19T22:11:13.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:13.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:13.687255+0000) 2024-02-19T22:11:14.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:14.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:14.687494+0000) 2024-02-19T22:11:15.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:15.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:15.687747+0000) 2024-02-19T22:11:16.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:16.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:16.688009+0000) 2024-02-19T22:11:17.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:17.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:17.688245+0000) 2024-02-19T22:11:18.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:18.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:18.688497+0000) 2024-02-19T22:11:19.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:19.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:19.688765+0000) 2024-02-19T22:11:20.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:20.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:20.689005+0000) 2024-02-19T22:11:21.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:21.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:21.689253+0000) 2024-02-19T22:11:22.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:22.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:22.689456+0000) 2024-02-19T22:11:23.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:23.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:23.689654+0000) 2024-02-19T22:11:24.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:24.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:24.689863+0000) 2024-02-19T22:11:25.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:25.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:25.690105+0000) 2024-02-19T22:11:26.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:26.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:26.690298+0000) 2024-02-19T22:11:27.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:27.686+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:27.690544+0000) 2024-02-19T22:11:28.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:28.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:28.690814+0000) 2024-02-19T22:11:29.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:29.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:29.691054+0000) 2024-02-19T22:11:30.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:30.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:30.691245+0000) 2024-02-19T22:11:31.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:31.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:31.691443+0000) 2024-02-19T22:11:31.821 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:11:31.825 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:11:32.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:32.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:32.691703+0000) 2024-02-19T22:11:33.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:33.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:33.691939+0000) 2024-02-19T22:11:34.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:34.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:34.692202+0000) 2024-02-19T22:11:35.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:35.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:35.692491+0000) 2024-02-19T22:11:36.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:36.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:36.692729+0000) 2024-02-19T22:11:37.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:37.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:37.692940+0000) 2024-02-19T22:11:38.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:38.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:38.693178+0000) 2024-02-19T22:11:39.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:39.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:39.693404+0000) 2024-02-19T22:11:40.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:40.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:40.693663+0000) 2024-02-19T22:11:41.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:41.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:41.693836+0000) 2024-02-19T22:11:42.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:42.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:42.694054+0000) 2024-02-19T22:11:43.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:43.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:43.694286+0000) 2024-02-19T22:11:44.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:44.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:44.694493+0000) 2024-02-19T22:11:45.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:45.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:45.694682+0000) 2024-02-19T22:11:46.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:46.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:46.694951+0000) 2024-02-19T22:11:47.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:47.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:47.695212+0000) 2024-02-19T22:11:48.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:48.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:48.695399+0000) 2024-02-19T22:11:49.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:49.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:49.695592+0000) 2024-02-19T22:11:50.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:50.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:50.695803+0000) 2024-02-19T22:11:51.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:51.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:51.696078+0000) 2024-02-19T22:11:52.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:52.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:52.696314+0000) 2024-02-19T22:11:53.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:53.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:53.696532+0000) 2024-02-19T22:11:54.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:54.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:54.696686+0000) 2024-02-19T22:11:55.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:55.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:55.696934+0000) 2024-02-19T22:11:56.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:56.693+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:56.697103+0000) 2024-02-19T22:11:57.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:57.693+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:57.697362+0000) 2024-02-19T22:11:58.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:58.693+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:58.697612+0000) 2024-02-19T22:11:59.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:11:59.693+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:11:59.697814+0000) 2024-02-19T22:12:00.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:00.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:00.698053+0000) 2024-02-19T22:12:01.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:01.693+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:01.698290+0000) 2024-02-19T22:12:01.881 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:12:01.884 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:12:02.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:02.693+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:02.698514+0000) 2024-02-19T22:12:03.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:03.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:03.698742+0000) 2024-02-19T22:12:04.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:04.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:04.698916+0000) 2024-02-19T22:12:05.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:05.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:05.699101+0000) 2024-02-19T22:12:06.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:06.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:06.699342+0000) 2024-02-19T22:12:07.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:07.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:07.699583+0000) 2024-02-19T22:12:08.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:08.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:08.699832+0000) 2024-02-19T22:12:09.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:09.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:09.700052+0000) 2024-02-19T22:12:10.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:10.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:10.700250+0000) 2024-02-19T22:12:11.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:11.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:11.700485+0000) 2024-02-19T22:12:12.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:12.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:12.700620+0000) 2024-02-19T22:12:13.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:13.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:13.700812+0000) 2024-02-19T22:12:14.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:14.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:14.701095+0000) 2024-02-19T22:12:15.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:15.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:15.701295+0000) 2024-02-19T22:12:16.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:16.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:16.701524+0000) 2024-02-19T22:12:17.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:17.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:17.701768+0000) 2024-02-19T22:12:18.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:18.701+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:18.701935+0000) 2024-02-19T22:12:19.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:19.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:19.702180+0000) 2024-02-19T22:12:20.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:20.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:20.702355+0000) 2024-02-19T22:12:21.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:21.697+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:21.702555+0000) 2024-02-19T22:12:22.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:22.701+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:22.702800+0000) 2024-02-19T22:12:23.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:23.701+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:23.703023+0000) 2024-02-19T22:12:24.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:24.701+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:24.703197+0000) 2024-02-19T22:12:25.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:25.701+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:25.703420+0000) 2024-02-19T22:12:26.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:26.701+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:26.703690+0000) 2024-02-19T22:12:27.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:27.701+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:27.703934+0000) 2024-02-19T22:12:28.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:28.701+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:28.704220+0000) 2024-02-19T22:12:29.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:29.701+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:29.704483+0000) 2024-02-19T22:12:30.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:30.701+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:30.704653+0000) 2024-02-19T22:12:31.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:31.701+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:31.704914+0000) 2024-02-19T22:12:32.028 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:12:32.032 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:12:32.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:32.701+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:32.705116+0000) 2024-02-19T22:12:33.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:33.701+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:33.705361+0000) 2024-02-19T22:12:34.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:34.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:34.705545+0000) 2024-02-19T22:12:35.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:35.701+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:35.705816+0000) 2024-02-19T22:12:36.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:36.701+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:36.706009+0000) 2024-02-19T22:12:37.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:37.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:37.706286+0000) 2024-02-19T22:12:38.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:38.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:38.706542+0000) 2024-02-19T22:12:39.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:39.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:39.706771+0000) 2024-02-19T22:12:40.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:40.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:40.706942+0000) 2024-02-19T22:12:41.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:41.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:41.707104+0000) 2024-02-19T22:12:42.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:42.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:42.707267+0000) 2024-02-19T22:12:43.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:43.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:43.707522+0000) 2024-02-19T22:12:44.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:44.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:44.707753+0000) 2024-02-19T22:12:45.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:45.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:45.707993+0000) 2024-02-19T22:12:46.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:46.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:46.708225+0000) 2024-02-19T22:12:47.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:47.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:47.708495+0000) 2024-02-19T22:12:48.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:48.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:48.708665+0000) 2024-02-19T22:12:49.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:49.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:49.708923+0000) 2024-02-19T22:12:50.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:50.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:50.709155+0000) 2024-02-19T22:12:51.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:51.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:51.709393+0000) 2024-02-19T22:12:52.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:52.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:52.709605+0000) 2024-02-19T22:12:53.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:53.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:53.709831+0000) 2024-02-19T22:12:54.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:54.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:54.710085+0000) 2024-02-19T22:12:55.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:55.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:55.710324+0000) 2024-02-19T22:12:56.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:56.704+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:56.710491+0000) 2024-02-19T22:12:57.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:57.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:57.710748+0000) 2024-02-19T22:12:58.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:58.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:58.710936+0000) 2024-02-19T22:12:59.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:12:59.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:12:59.711087+0000) 2024-02-19T22:13:00.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:00.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:00.711261+0000) 2024-02-19T22:13:01.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:01.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:01.711449+0000) 2024-02-19T22:13:02.151 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:13:02.154 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:13:02.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:02.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:02.711628+0000) 2024-02-19T22:13:03.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:03.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:03.711803+0000) 2024-02-19T22:13:04.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:04.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:04.712040+0000) 2024-02-19T22:13:05.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:05.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:05.712228+0000) 2024-02-19T22:13:06.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:06.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:06.712454+0000) 2024-02-19T22:13:07.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:07.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:07.712690+0000) 2024-02-19T22:13:08.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:08.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:08.712885+0000) 2024-02-19T22:13:09.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:09.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:09.713118+0000) 2024-02-19T22:13:10.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:10.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:10.713369+0000) 2024-02-19T22:13:11.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:11.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:11.713631+0000) 2024-02-19T22:13:12.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:12.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:12.713833+0000) 2024-02-19T22:13:13.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:13.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:13.714076+0000) 2024-02-19T22:13:14.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:14.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:14.714310+0000) 2024-02-19T22:13:15.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:15.708+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:15.714509+0000) 2024-02-19T22:13:16.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:16.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:16.714766+0000) 2024-02-19T22:13:17.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:17.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:17.715015+0000) 2024-02-19T22:13:18.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:18.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:18.715219+0000) 2024-02-19T22:13:19.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:19.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:19.715444+0000) 2024-02-19T22:13:20.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:20.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:20.715697+0000) 2024-02-19T22:13:21.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:21.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:21.715954+0000) 2024-02-19T22:13:22.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:22.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:22.716201+0000) 2024-02-19T22:13:23.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:23.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:23.716398+0000) 2024-02-19T22:13:24.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:24.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:24.716618+0000) 2024-02-19T22:13:25.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:25.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:25.716859+0000) 2024-02-19T22:13:26.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:26.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:26.717097+0000) 2024-02-19T22:13:27.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:27.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:27.717271+0000) 2024-02-19T22:13:28.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:28.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:28.717461+0000) 2024-02-19T22:13:29.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:29.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:29.717690+0000) 2024-02-19T22:13:30.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:30.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:30.717953+0000) 2024-02-19T22:13:31.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:31.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:31.718207+0000) 2024-02-19T22:13:32.222 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:13:32.225 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:13:32.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:32.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:32.718460+0000) 2024-02-19T22:13:33.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:33.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:33.718741+0000) 2024-02-19T22:13:34.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:34.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:34.718989+0000) 2024-02-19T22:13:35.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:35.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:35.719171+0000) 2024-02-19T22:13:36.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:36.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:36.719422+0000) 2024-02-19T22:13:37.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:37.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:37.719601+0000) 2024-02-19T22:13:38.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:38.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:38.719836+0000) 2024-02-19T22:13:39.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:39.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:39.720071+0000) 2024-02-19T22:13:40.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:40.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:40.720307+0000) 2024-02-19T22:13:41.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:41.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:41.720541+0000) 2024-02-19T22:13:42.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:42.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:42.720770+0000) 2024-02-19T22:13:43.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:43.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:43.721025+0000) 2024-02-19T22:13:44.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:44.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:44.721245+0000) 2024-02-19T22:13:45.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:45.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:45.721472+0000) 2024-02-19T22:13:46.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:46.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:46.721711+0000) 2024-02-19T22:13:47.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:47.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:47.721956+0000) 2024-02-19T22:13:48.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:48.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:48.722231+0000) 2024-02-19T22:13:49.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:49.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:49.722482+0000) 2024-02-19T22:13:50.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:50.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:50.722743+0000) 2024-02-19T22:13:51.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:51.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:51.722923+0000) 2024-02-19T22:13:52.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:52.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:52.723079+0000) 2024-02-19T22:13:53.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:53.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:53.723392+0000) 2024-02-19T22:13:54.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:54.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:54.723647+0000) 2024-02-19T22:13:55.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:55.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:55.723918+0000) 2024-02-19T22:13:56.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:56.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:56.724185+0000) 2024-02-19T22:13:57.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:57.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:57.724424+0000) 2024-02-19T22:13:58.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:58.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:58.724701+0000) 2024-02-19T22:13:59.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:13:59.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:13:59.724920+0000) 2024-02-19T22:14:00.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:00.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:00.725160+0000) 2024-02-19T22:14:01.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:01.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:01.725416+0000) 2024-02-19T22:14:02.303 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:14:02.306 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:14:02.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:02.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:02.725648+0000) 2024-02-19T22:14:03.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:03.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:03.725864+0000) 2024-02-19T22:14:04.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:04.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:04.726141+0000) 2024-02-19T22:14:05.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:05.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:05.726318+0000) 2024-02-19T22:14:06.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:06.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:06.726525+0000) 2024-02-19T22:14:07.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:07.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:07.726806+0000) 2024-02-19T22:14:08.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:08.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:08.727016+0000) 2024-02-19T22:14:09.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:09.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:09.727251+0000) 2024-02-19T22:14:10.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:10.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:10.727512+0000) 2024-02-19T22:14:11.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:11.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:11.727709+0000) 2024-02-19T22:14:12.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:12.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:12.727891+0000) 2024-02-19T22:14:13.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:13.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:13.728118+0000) 2024-02-19T22:14:14.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:14.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:14.728366+0000) 2024-02-19T22:14:15.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:15.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:15.728634+0000) 2024-02-19T22:14:16.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:16.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:16.728828+0000) 2024-02-19T22:14:17.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:17.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:17.729058+0000) 2024-02-19T22:14:18.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:18.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:18.729254+0000) 2024-02-19T22:14:19.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:19.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:19.729513+0000) 2024-02-19T22:14:20.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:20.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:20.729785+0000) 2024-02-19T22:14:21.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:21.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:21.730026+0000) 2024-02-19T22:14:22.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:22.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:22.730264+0000) 2024-02-19T22:14:23.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:23.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:23.730501+0000) 2024-02-19T22:14:24.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:24.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:24.730681+0000) 2024-02-19T22:14:25.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:25.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:25.730911+0000) 2024-02-19T22:14:26.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:26.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:26.731148+0000) 2024-02-19T22:14:27.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:27.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:27.731339+0000) 2024-02-19T22:14:28.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:28.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:28.731570+0000) 2024-02-19T22:14:29.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:29.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:29.731816+0000) 2024-02-19T22:14:30.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:30.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:30.732070+0000) 2024-02-19T22:14:31.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:31.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:31.732285+0000) 2024-02-19T22:14:32.361 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:14:32.365 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:14:32.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:32.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:32.732509+0000) 2024-02-19T22:14:33.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:33.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:33.732680+0000) 2024-02-19T22:14:34.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:34.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:34.732879+0000) 2024-02-19T22:14:35.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:35.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:35.733109+0000) 2024-02-19T22:14:36.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:36.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:36.733389+0000) 2024-02-19T22:14:37.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:37.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:37.733616+0000) 2024-02-19T22:14:38.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:38.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:38.733856+0000) 2024-02-19T22:14:39.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:39.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:39.734097+0000) 2024-02-19T22:14:40.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:40.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:40.734285+0000) 2024-02-19T22:14:41.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:41.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:41.734491+0000) 2024-02-19T22:14:42.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:42.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:42.734679+0000) 2024-02-19T22:14:43.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:43.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:43.734818+0000) 2024-02-19T22:14:44.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:44.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:44.734984+0000) 2024-02-19T22:14:45.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:45.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:45.735229+0000) 2024-02-19T22:14:46.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:46.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:46.735462+0000) 2024-02-19T22:14:47.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:47.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:47.735645+0000) 2024-02-19T22:14:48.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:48.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:48.735905+0000) 2024-02-19T22:14:49.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:49.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:49.736153+0000) 2024-02-19T22:14:50.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:50.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:50.736398+0000) 2024-02-19T22:14:51.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:51.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:51.736598+0000) 2024-02-19T22:14:52.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:52.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:52.736898+0000) 2024-02-19T22:14:53.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:53.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:53.737171+0000) 2024-02-19T22:14:54.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:54.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:54.737441+0000) 2024-02-19T22:14:55.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:55.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:55.737695+0000) 2024-02-19T22:14:56.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:56.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:56.737886+0000) 2024-02-19T22:14:57.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:57.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:57.738136+0000) 2024-02-19T22:14:58.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:58.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:58.738379+0000) 2024-02-19T22:14:59.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:14:59.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:14:59.738665+0000) 2024-02-19T22:15:00.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:00.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:00.738863+0000) 2024-02-19T22:15:01.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:01.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:01.739109+0000) 2024-02-19T22:15:02.472 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:15:02.477 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:15:02.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:02.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:02.739377+0000) 2024-02-19T22:15:03.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:03.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:03.739641+0000) 2024-02-19T22:15:04.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:04.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:04.739915+0000) 2024-02-19T22:15:05.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:05.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:05.740195+0000) 2024-02-19T22:15:06.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:06.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:06.740364+0000) 2024-02-19T22:15:07.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:07.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:07.740626+0000) 2024-02-19T22:15:08.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:08.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:08.740867+0000) 2024-02-19T22:15:09.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:09.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:09.741148+0000) 2024-02-19T22:15:10.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:10.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:10.741417+0000) 2024-02-19T22:15:11.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:11.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:11.741685+0000) 2024-02-19T22:15:12.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:12.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:12.741933+0000) 2024-02-19T22:15:13.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:13.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:13.742149+0000) 2024-02-19T22:15:14.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:14.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:14.742349+0000) 2024-02-19T22:15:15.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:15.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:15.742584+0000) 2024-02-19T22:15:16.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:16.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:16.742818+0000) 2024-02-19T22:15:17.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:17.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:17.743000+0000) 2024-02-19T22:15:18.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:18.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:18.743264+0000) 2024-02-19T22:15:19.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:19.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:19.743533+0000) 2024-02-19T22:15:20.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:20.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:20.743795+0000) 2024-02-19T22:15:21.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:21.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:21.744041+0000) 2024-02-19T22:15:22.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:22.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:22.744212+0000) 2024-02-19T22:15:23.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:23.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:23.744450+0000) 2024-02-19T22:15:24.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:24.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:24.744731+0000) 2024-02-19T22:15:25.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:25.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:25.744966+0000) 2024-02-19T22:15:26.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:26.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:26.745196+0000) 2024-02-19T22:15:27.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:27.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:27.745457+0000) 2024-02-19T22:15:28.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:28.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:28.745643+0000) 2024-02-19T22:15:29.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:29.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:29.745850+0000) 2024-02-19T22:15:30.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:30.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:30.746058+0000) 2024-02-19T22:15:31.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:31.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:31.746275+0000) 2024-02-19T22:15:32.578 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:15:32.582 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:15:32.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:32.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:32.746486+0000) 2024-02-19T22:15:33.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:33.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:33.746784+0000) 2024-02-19T22:15:34.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:34.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:34.746941+0000) 2024-02-19T22:15:35.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:35.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:35.747120+0000) 2024-02-19T22:15:36.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:36.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:36.747249+0000) 2024-02-19T22:15:37.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:37.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:37.747498+0000) 2024-02-19T22:15:38.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:38.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:38.747726+0000) 2024-02-19T22:15:39.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:39.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:39.748009+0000) 2024-02-19T22:15:40.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:40.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:40.748206+0000) 2024-02-19T22:15:41.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:41.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:41.748452+0000) 2024-02-19T22:15:42.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:42.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:42.748686+0000) 2024-02-19T22:15:43.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:43.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:43.748934+0000) 2024-02-19T22:15:44.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:44.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:44.749169+0000) 2024-02-19T22:15:45.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:45.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:45.749474+0000) 2024-02-19T22:15:46.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:46.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:46.749660+0000) 2024-02-19T22:15:47.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:47.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:47.749886+0000) 2024-02-19T22:15:48.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:48.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:48.750083+0000) 2024-02-19T22:15:49.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:49.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:49.750241+0000) 2024-02-19T22:15:50.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:50.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:50.750482+0000) 2024-02-19T22:15:51.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:51.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:51.750751+0000) 2024-02-19T22:15:52.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:52.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:52.751005+0000) 2024-02-19T22:15:53.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:53.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:53.751274+0000) 2024-02-19T22:15:54.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:54.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:54.751513+0000) 2024-02-19T22:15:55.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:55.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:55.751760+0000) 2024-02-19T22:15:56.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:56.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:56.751989+0000) 2024-02-19T22:15:57.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:57.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:57.752234+0000) 2024-02-19T22:15:58.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:58.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:58.752500+0000) 2024-02-19T22:15:59.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:15:59.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:15:59.752749+0000) 2024-02-19T22:16:00.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:00.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:00.752981+0000) 2024-02-19T22:16:01.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:01.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:01.753228+0000) 2024-02-19T22:16:02.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:02.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:02.753464+0000) 2024-02-19T22:16:02.759 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:16:02.763 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:16:03.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:03.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:03.753635+0000) 2024-02-19T22:16:04.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:04.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:04.753868+0000) 2024-02-19T22:16:05.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:05.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:05.754021+0000) 2024-02-19T22:16:06.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:06.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:06.754218+0000) 2024-02-19T22:16:07.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:07.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:07.754458+0000) 2024-02-19T22:16:08.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:08.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:08.754699+0000) 2024-02-19T22:16:09.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:09.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:09.754988+0000) 2024-02-19T22:16:10.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:10.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:10.755228+0000) 2024-02-19T22:16:11.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:11.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:11.755398+0000) 2024-02-19T22:16:12.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:12.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:12.755631+0000) 2024-02-19T22:16:13.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:13.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:13.755808+0000) 2024-02-19T22:16:14.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:14.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:14.756063+0000) 2024-02-19T22:16:15.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:15.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:15.756259+0000) 2024-02-19T22:16:16.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:16.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:16.756476+0000) 2024-02-19T22:16:17.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:17.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:17.756662+0000) 2024-02-19T22:16:18.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:18.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:18.756850+0000) 2024-02-19T22:16:19.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:19.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:19.757039+0000) 2024-02-19T22:16:20.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:20.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:20.757240+0000) 2024-02-19T22:16:21.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:21.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:21.757414+0000) 2024-02-19T22:16:22.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:22.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:22.757650+0000) 2024-02-19T22:16:23.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:23.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:23.757897+0000) 2024-02-19T22:16:24.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:24.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:24.758153+0000) 2024-02-19T22:16:25.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:25.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:25.758333+0000) 2024-02-19T22:16:26.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:26.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:26.758570+0000) 2024-02-19T22:16:27.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:27.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:27.758825+0000) 2024-02-19T22:16:28.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:28.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:28.758983+0000) 2024-02-19T22:16:29.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:29.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:29.759207+0000) 2024-02-19T22:16:30.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:30.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:30.759428+0000) 2024-02-19T22:16:31.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:31.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:31.759555+0000) 2024-02-19T22:16:32.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:32.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:32.759711+0000) 2024-02-19T22:16:32.865 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:16:32.868 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:16:33.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:33.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:33.759942+0000) 2024-02-19T22:16:34.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:34.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:34.760148+0000) 2024-02-19T22:16:35.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:35.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:35.760311+0000) 2024-02-19T22:16:36.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:36.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:36.760466+0000) 2024-02-19T22:16:37.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:37.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:37.760623+0000) 2024-02-19T22:16:38.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:38.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:38.760783+0000) 2024-02-19T22:16:39.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:39.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:39.761031+0000) 2024-02-19T22:16:40.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:40.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:40.761236+0000) 2024-02-19T22:16:41.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:41.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:41.761405+0000) 2024-02-19T22:16:42.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:42.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:42.761575+0000) 2024-02-19T22:16:43.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:43.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:43.761742+0000) 2024-02-19T22:16:44.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:44.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:44.761964+0000) 2024-02-19T22:16:45.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:45.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:45.762138+0000) 2024-02-19T22:16:46.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:46.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:46.762282+0000) 2024-02-19T22:16:47.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:47.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:47.762445+0000) 2024-02-19T22:16:48.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:48.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:48.762603+0000) 2024-02-19T22:16:49.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:49.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:49.762768+0000) 2024-02-19T22:16:50.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:50.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:50.762934+0000) 2024-02-19T22:16:51.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:51.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:51.763131+0000) 2024-02-19T22:16:52.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:52.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:52.763354+0000) 2024-02-19T22:16:53.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:53.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:53.763519+0000) 2024-02-19T22:16:54.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:54.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:54.763654+0000) 2024-02-19T22:16:55.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:55.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:55.763799+0000) 2024-02-19T22:16:56.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:56.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:56.763974+0000) 2024-02-19T22:16:57.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:57.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:57.764125+0000) 2024-02-19T22:16:58.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:58.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:58.764329+0000) 2024-02-19T22:16:59.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:16:59.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:16:59.764513+0000) 2024-02-19T22:17:00.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:00.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:00.764746+0000) 2024-02-19T22:17:01.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:01.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:01.764882+0000) 2024-02-19T22:17:02.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:02.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:02.765117+0000) 2024-02-19T22:17:02.979 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:17:02.982 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:17:03.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:03.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:03.765324+0000) 2024-02-19T22:17:04.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:04.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:04.765555+0000) 2024-02-19T22:17:05.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:05.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:05.765704+0000) 2024-02-19T22:17:06.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:06.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:06.765829+0000) 2024-02-19T22:17:07.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:07.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:07.765954+0000) 2024-02-19T22:17:08.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:08.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:08.766071+0000) 2024-02-19T22:17:09.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:09.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:09.766246+0000) 2024-02-19T22:17:10.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:10.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:10.766423+0000) 2024-02-19T22:17:11.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:11.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:11.766548+0000) 2024-02-19T22:17:12.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:12.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:12.766773+0000) 2024-02-19T22:17:13.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:13.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:13.766979+0000) 2024-02-19T22:17:14.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:14.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:14.767196+0000) 2024-02-19T22:17:15.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:15.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:15.767350+0000) 2024-02-19T22:17:16.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:16.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:16.767539+0000) 2024-02-19T22:17:17.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:17.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:17.767694+0000) 2024-02-19T22:17:18.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:18.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:18.767820+0000) 2024-02-19T22:17:19.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:19.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:19.767959+0000) 2024-02-19T22:17:20.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:20.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:20.768088+0000) 2024-02-19T22:17:21.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:21.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:21.768274+0000) 2024-02-19T22:17:22.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:22.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:22.768409+0000) 2024-02-19T22:17:23.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:23.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:23.768629+0000) 2024-02-19T22:17:24.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:24.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:24.768780+0000) 2024-02-19T22:17:25.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:25.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:25.768917+0000) 2024-02-19T22:17:26.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:26.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:26.769076+0000) 2024-02-19T22:17:27.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:27.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:27.769257+0000) 2024-02-19T22:17:28.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:28.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:28.769394+0000) 2024-02-19T22:17:29.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:29.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:29.769551+0000) 2024-02-19T22:17:30.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:30.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:30.769713+0000) 2024-02-19T22:17:31.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:31.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:31.769911+0000) 2024-02-19T22:17:32.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:32.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:32.770070+0000) 2024-02-19T22:17:33.117 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:17:33.120 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:17:33.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:33.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:33.770319+0000) 2024-02-19T22:17:34.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:34.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:34.770474+0000) 2024-02-19T22:17:35.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:35.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:35.770757+0000) 2024-02-19T22:17:36.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:36.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:36.770996+0000) 2024-02-19T22:17:37.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:37.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:37.771200+0000) 2024-02-19T22:17:38.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:38.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:38.771337+0000) 2024-02-19T22:17:39.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:39.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:39.771469+0000) 2024-02-19T22:17:40.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:40.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:40.771669+0000) 2024-02-19T22:17:41.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:41.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:41.771831+0000) 2024-02-19T22:17:42.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:42.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:42.772005+0000) 2024-02-19T22:17:43.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:43.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:43.772209+0000) 2024-02-19T22:17:44.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:44.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:44.772363+0000) 2024-02-19T22:17:45.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:45.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:45.772537+0000) 2024-02-19T22:17:46.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:46.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:46.772798+0000) 2024-02-19T22:17:47.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:47.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:47.772973+0000) 2024-02-19T22:17:48.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:48.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:48.773150+0000) 2024-02-19T22:17:49.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:49.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:49.773317+0000) 2024-02-19T22:17:50.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:50.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:50.773581+0000) 2024-02-19T22:17:51.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:51.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:51.773765+0000) 2024-02-19T22:17:52.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:52.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:52.773935+0000) 2024-02-19T22:17:53.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:53.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:53.774165+0000) 2024-02-19T22:17:54.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:54.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:54.774405+0000) 2024-02-19T22:17:55.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:55.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:55.774556+0000) 2024-02-19T22:17:56.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:56.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:56.774739+0000) 2024-02-19T22:17:57.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:57.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:57.774983+0000) 2024-02-19T22:17:58.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:58.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:58.775153+0000) 2024-02-19T22:17:59.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:17:59.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:17:59.775320+0000) 2024-02-19T22:18:00.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:00.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:00.775464+0000) 2024-02-19T22:18:01.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:01.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:01.775667+0000) 2024-02-19T22:18:02.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:02.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:02.775826+0000) 2024-02-19T22:18:03.164 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:18:03.168 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:18:03.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:03.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:03.775953+0000) 2024-02-19T22:18:04.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:04.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:04.776170+0000) 2024-02-19T22:18:05.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:05.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:05.776366+0000) 2024-02-19T22:18:06.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:06.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:06.776526+0000) 2024-02-19T22:18:07.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:07.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:07.776692+0000) 2024-02-19T22:18:08.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:08.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:08.776926+0000) 2024-02-19T22:18:09.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:09.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:09.777067+0000) 2024-02-19T22:18:10.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:10.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:10.777299+0000) 2024-02-19T22:18:11.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:11.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:11.777476+0000) 2024-02-19T22:18:12.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:12.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:12.777650+0000) 2024-02-19T22:18:13.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:13.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:13.777824+0000) 2024-02-19T22:18:14.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:14.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:14.777981+0000) 2024-02-19T22:18:15.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:15.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:15.778180+0000) 2024-02-19T22:18:16.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:16.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:16.778414+0000) 2024-02-19T22:18:17.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:17.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:17.778662+0000) 2024-02-19T22:18:18.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:18.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:18.778911+0000) 2024-02-19T22:18:19.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:19.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:19.779077+0000) 2024-02-19T22:18:20.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:20.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:20.779308+0000) 2024-02-19T22:18:21.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:21.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:21.779565+0000) 2024-02-19T22:18:22.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:22.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:22.779784+0000) 2024-02-19T22:18:23.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:23.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:23.780010+0000) 2024-02-19T22:18:24.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:24.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:24.780238+0000) 2024-02-19T22:18:25.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:25.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:25.780459+0000) 2024-02-19T22:18:26.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:26.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:26.780626+0000) 2024-02-19T22:18:27.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:27.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:27.780826+0000) 2024-02-19T22:18:28.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:28.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:28.780991+0000) 2024-02-19T22:18:29.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:29.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:29.781184+0000) 2024-02-19T22:18:30.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:30.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:30.781440+0000) 2024-02-19T22:18:31.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:31.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:31.781676+0000) 2024-02-19T22:18:32.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:32.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:32.781928+0000) 2024-02-19T22:18:33.285 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:18:33.289 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:18:33.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:33.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:33.782197+0000) 2024-02-19T22:18:34.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:34.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:34.782412+0000) 2024-02-19T22:18:35.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:35.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:35.782651+0000) 2024-02-19T22:18:36.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:36.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:36.782818+0000) 2024-02-19T22:18:37.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:37.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:37.782948+0000) 2024-02-19T22:18:38.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:38.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:38.783102+0000) 2024-02-19T22:18:39.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:39.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:39.783270+0000) 2024-02-19T22:18:40.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:40.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:40.783413+0000) 2024-02-19T22:18:41.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:41.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:41.783555+0000) 2024-02-19T22:18:42.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:42.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:42.783702+0000) 2024-02-19T22:18:43.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:43.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:43.783845+0000) 2024-02-19T22:18:44.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:44.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:44.784043+0000) 2024-02-19T22:18:45.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:45.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:45.784255+0000) 2024-02-19T22:18:46.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:46.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:46.784452+0000) 2024-02-19T22:18:47.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:47.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:47.784679+0000) 2024-02-19T22:18:48.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:48.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:48.784837+0000) 2024-02-19T22:18:49.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:49.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:49.785078+0000) 2024-02-19T22:18:50.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:50.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:50.785247+0000) 2024-02-19T22:18:51.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:51.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:51.785417+0000) 2024-02-19T22:18:52.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:52.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:52.785608+0000) 2024-02-19T22:18:53.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:53.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:53.785792+0000) 2024-02-19T22:18:54.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:54.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:54.785988+0000) 2024-02-19T22:18:55.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:55.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:55.786158+0000) 2024-02-19T22:18:56.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:56.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:56.786319+0000) 2024-02-19T22:18:57.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:57.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:57.786449+0000) 2024-02-19T22:18:58.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:58.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:58.786572+0000) 2024-02-19T22:18:59.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:18:59.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:18:59.786758+0000) 2024-02-19T22:19:00.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:00.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:00.786922+0000) 2024-02-19T22:19:01.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:01.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:01.787107+0000) 2024-02-19T22:19:02.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:02.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:02.787251+0000) 2024-02-19T22:19:03.350 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:19:03.353 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:19:03.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:03.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:03.787450+0000) 2024-02-19T22:19:04.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:04.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:04.787676+0000) 2024-02-19T22:19:05.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:05.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:05.787870+0000) 2024-02-19T22:19:06.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:06.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:06.788061+0000) 2024-02-19T22:19:07.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:07.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:07.788232+0000) 2024-02-19T22:19:08.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:08.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:08.788429+0000) 2024-02-19T22:19:09.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:09.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:09.788565+0000) 2024-02-19T22:19:10.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:10.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:10.788735+0000) 2024-02-19T22:19:11.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:11.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:11.788906+0000) 2024-02-19T22:19:12.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:12.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:12.789043+0000) 2024-02-19T22:19:13.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:13.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:13.789238+0000) 2024-02-19T22:19:14.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:14.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:14.789419+0000) 2024-02-19T22:19:15.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:15.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:15.789536+0000) 2024-02-19T22:19:16.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:16.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:16.789693+0000) 2024-02-19T22:19:17.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:17.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:17.789931+0000) 2024-02-19T22:19:18.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:18.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:18.790162+0000) 2024-02-19T22:19:19.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:19.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:19.790362+0000) 2024-02-19T22:19:20.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:20.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:20.790520+0000) 2024-02-19T22:19:21.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:21.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:21.790800+0000) 2024-02-19T22:19:22.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:22.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:22.790959+0000) 2024-02-19T22:19:23.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:23.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:23.791117+0000) 2024-02-19T22:19:24.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:24.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:24.791354+0000) 2024-02-19T22:19:25.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:25.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:25.791516+0000) 2024-02-19T22:19:26.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:26.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:26.791739+0000) 2024-02-19T22:19:27.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:27.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:27.791902+0000) 2024-02-19T22:19:28.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:28.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:28.792216+0000) 2024-02-19T22:19:29.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:29.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:29.792410+0000) 2024-02-19T22:19:30.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:30.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:30.792525+0000) 2024-02-19T22:19:31.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:31.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:31.792661+0000) 2024-02-19T22:19:32.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:32.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:32.792776+0000) 2024-02-19T22:19:33.407 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:19:33.410 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:19:33.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:33.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:33.793028+0000) 2024-02-19T22:19:34.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:34.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:34.793217+0000) 2024-02-19T22:19:35.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:35.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:35.793436+0000) 2024-02-19T22:19:36.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:36.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:36.793601+0000) 2024-02-19T22:19:37.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:37.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:37.793845+0000) 2024-02-19T22:19:38.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:38.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:38.794079+0000) 2024-02-19T22:19:39.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:39.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:39.794319+0000) 2024-02-19T22:19:40.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:40.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:40.794496+0000) 2024-02-19T22:19:41.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:41.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:41.794748+0000) 2024-02-19T22:19:42.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:42.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:42.794921+0000) 2024-02-19T22:19:43.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:43.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:43.795138+0000) 2024-02-19T22:19:44.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:44.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:44.795266+0000) 2024-02-19T22:19:45.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:45.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:45.795486+0000) 2024-02-19T22:19:46.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:46.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:46.795696+0000) 2024-02-19T22:19:47.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:47.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:47.795824+0000) 2024-02-19T22:19:48.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:48.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:48.796037+0000) 2024-02-19T22:19:49.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:49.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:49.796280+0000) 2024-02-19T22:19:50.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:50.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:50.796512+0000) 2024-02-19T22:19:51.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:51.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:51.796735+0000) 2024-02-19T22:19:52.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:52.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:52.797019+0000) 2024-02-19T22:19:53.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:53.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:53.797222+0000) 2024-02-19T22:19:54.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:54.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:54.797376+0000) 2024-02-19T22:19:55.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:55.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:55.797530+0000) 2024-02-19T22:19:56.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:56.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:56.797797+0000) 2024-02-19T22:19:57.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:57.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:57.797989+0000) 2024-02-19T22:19:58.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:58.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:58.798145+0000) 2024-02-19T22:19:59.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:19:59.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:19:59.798311+0000) 2024-02-19T22:20:00.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:00.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:00.798564+0000) 2024-02-19T22:20:01.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:01.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:01.798790+0000) 2024-02-19T22:20:02.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:02.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:02.798937+0000) 2024-02-19T22:20:03.545 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:20:03.548 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:20:03.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:03.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:03.799102+0000) 2024-02-19T22:20:04.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:04.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:04.799278+0000) 2024-02-19T22:20:05.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:05.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:05.799434+0000) 2024-02-19T22:20:06.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:06.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:06.799590+0000) 2024-02-19T22:20:07.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:07.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:07.799812+0000) 2024-02-19T22:20:08.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:08.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:08.799983+0000) 2024-02-19T22:20:09.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:09.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:09.800147+0000) 2024-02-19T22:20:10.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:10.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:10.800274+0000) 2024-02-19T22:20:11.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:11.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:11.800438+0000) 2024-02-19T22:20:12.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:12.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:12.800603+0000) 2024-02-19T22:20:13.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:13.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:13.800826+0000) 2024-02-19T22:20:14.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:14.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:14.801063+0000) 2024-02-19T22:20:15.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:15.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:15.801243+0000) 2024-02-19T22:20:16.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:16.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:16.801398+0000) 2024-02-19T22:20:17.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:17.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:17.801576+0000) 2024-02-19T22:20:18.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:18.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:18.801741+0000) 2024-02-19T22:20:19.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:19.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:19.801912+0000) 2024-02-19T22:20:20.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:20.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:20.802071+0000) 2024-02-19T22:20:21.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:21.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:21.802290+0000) 2024-02-19T22:20:22.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:22.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:22.802518+0000) 2024-02-19T22:20:23.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:23.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:23.802679+0000) 2024-02-19T22:20:24.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:24.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:24.802892+0000) 2024-02-19T22:20:25.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:25.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:25.803061+0000) 2024-02-19T22:20:26.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:26.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:26.803216+0000) 2024-02-19T22:20:27.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:27.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:27.803426+0000) 2024-02-19T22:20:28.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:28.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:28.803660+0000) 2024-02-19T22:20:29.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:29.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:29.803856+0000) 2024-02-19T22:20:30.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:30.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:30.803988+0000) 2024-02-19T22:20:31.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:31.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:31.804191+0000) 2024-02-19T22:20:32.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:32.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:32.804360+0000) 2024-02-19T22:20:33.602 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:20:33.605 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:20:33.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:33.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:33.804532+0000) 2024-02-19T22:20:34.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:34.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:34.804689+0000) 2024-02-19T22:20:35.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:35.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:35.804888+0000) 2024-02-19T22:20:36.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:36.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:36.805049+0000) 2024-02-19T22:20:37.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:37.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:37.805228+0000) 2024-02-19T22:20:38.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:38.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:38.805385+0000) 2024-02-19T22:20:39.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:39.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:39.805558+0000) 2024-02-19T22:20:40.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:40.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:40.805716+0000) 2024-02-19T22:20:41.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:41.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:41.805871+0000) 2024-02-19T22:20:42.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:42.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:42.806083+0000) 2024-02-19T22:20:43.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:43.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:43.806243+0000) 2024-02-19T22:20:44.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:44.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:44.806415+0000) 2024-02-19T22:20:45.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:45.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:45.806541+0000) 2024-02-19T22:20:46.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:46.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:46.806662+0000) 2024-02-19T22:20:47.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:47.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:47.806783+0000) 2024-02-19T22:20:48.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:48.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:48.806910+0000) 2024-02-19T22:20:49.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:49.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:49.807114+0000) 2024-02-19T22:20:50.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:50.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:50.807347+0000) 2024-02-19T22:20:51.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:51.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:51.807508+0000) 2024-02-19T22:20:52.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:52.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:52.807745+0000) 2024-02-19T22:20:53.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:53.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:53.807904+0000) 2024-02-19T22:20:54.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:54.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:54.808048+0000) 2024-02-19T22:20:55.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:55.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:55.808311+0000) 2024-02-19T22:20:56.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:56.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:56.808554+0000) 2024-02-19T22:20:57.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:57.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:57.808810+0000) 2024-02-19T22:20:58.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:58.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:58.808969+0000) 2024-02-19T22:20:59.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:20:59.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:20:59.809192+0000) 2024-02-19T22:21:00.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:00.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:00.809448+0000) 2024-02-19T22:21:01.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:01.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:01.809646+0000) 2024-02-19T22:21:02.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:02.804+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:02.809873+0000) 2024-02-19T22:21:03.748 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:21:03.751 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:21:03.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:03.804+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:03.810034+0000) 2024-02-19T22:21:04.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:04.804+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:04.810221+0000) 2024-02-19T22:21:05.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:05.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:05.810426+0000) 2024-02-19T22:21:06.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:06.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:06.810583+0000) 2024-02-19T22:21:07.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:07.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:07.810754+0000) 2024-02-19T22:21:08.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:08.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:08.810950+0000) 2024-02-19T22:21:09.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:09.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:09.811187+0000) 2024-02-19T22:21:10.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:10.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:10.811358+0000) 2024-02-19T22:21:11.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:11.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:11.811553+0000) 2024-02-19T22:21:12.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:12.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:12.811801+0000) 2024-02-19T22:21:13.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:13.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:13.811962+0000) 2024-02-19T22:21:14.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:14.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:14.812157+0000) 2024-02-19T22:21:15.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:15.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:15.812353+0000) 2024-02-19T22:21:16.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:16.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:16.812493+0000) 2024-02-19T22:21:17.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:17.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:17.812716+0000) 2024-02-19T22:21:18.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:18.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:18.812892+0000) 2024-02-19T22:21:19.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:19.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:19.813055+0000) 2024-02-19T22:21:20.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:20.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:20.813215+0000) 2024-02-19T22:21:21.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:21.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:21.813420+0000) 2024-02-19T22:21:22.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:22.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:22.813648+0000) 2024-02-19T22:21:23.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:23.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:23.813836+0000) 2024-02-19T22:21:24.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:24.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:24.814036+0000) 2024-02-19T22:21:25.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:25.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:25.814211+0000) 2024-02-19T22:21:26.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:26.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:26.814372+0000) 2024-02-19T22:21:27.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:27.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:27.814547+0000) 2024-02-19T22:21:28.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:28.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:28.814770+0000) 2024-02-19T22:21:29.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:29.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:29.814946+0000) 2024-02-19T22:21:30.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:30.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:30.815087+0000) 2024-02-19T22:21:31.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:31.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:31.815256+0000) 2024-02-19T22:21:32.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:32.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:32.815471+0000) 2024-02-19T22:21:33.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:33.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:33.815608+0000) 2024-02-19T22:21:33.895 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:21:33.898 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:21:34.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:34.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:34.815835+0000) 2024-02-19T22:21:35.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:35.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:35.816060+0000) 2024-02-19T22:21:36.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:36.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:36.816302+0000) 2024-02-19T22:21:37.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:37.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:37.816467+0000) 2024-02-19T22:21:38.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:38.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:38.816684+0000) 2024-02-19T22:21:39.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:39.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:39.816946+0000) 2024-02-19T22:21:40.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:40.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:40.817166+0000) 2024-02-19T22:21:41.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:41.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:41.817353+0000) 2024-02-19T22:21:42.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:42.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:42.817479+0000) 2024-02-19T22:21:43.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:43.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:43.817657+0000) 2024-02-19T22:21:44.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:44.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:44.817838+0000) 2024-02-19T22:21:45.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:45.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:45.818039+0000) 2024-02-19T22:21:46.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:46.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:46.818212+0000) 2024-02-19T22:21:47.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:47.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:47.818346+0000) 2024-02-19T22:21:48.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:48.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:48.818476+0000) 2024-02-19T22:21:49.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:49.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:49.818624+0000) 2024-02-19T22:21:50.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:50.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:50.818787+0000) 2024-02-19T22:21:51.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:51.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:51.818930+0000) 2024-02-19T22:21:52.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:52.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:52.819080+0000) 2024-02-19T22:21:53.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:53.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:53.819212+0000) 2024-02-19T22:21:54.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:54.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:54.819334+0000) 2024-02-19T22:21:55.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:55.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:55.819459+0000) 2024-02-19T22:21:56.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:56.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:56.819683+0000) 2024-02-19T22:21:57.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:57.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:57.819892+0000) 2024-02-19T22:21:58.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:58.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:58.820109+0000) 2024-02-19T22:21:59.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:21:59.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:21:59.820258+0000) 2024-02-19T22:22:00.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:00.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:00.820393+0000) 2024-02-19T22:22:01.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:01.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:01.820534+0000) 2024-02-19T22:22:02.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:02.815+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:02.820747+0000) 2024-02-19T22:22:03.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:03.815+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:03.820880+0000) 2024-02-19T22:22:03.941 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:22:03.944 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:22:04.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:04.815+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:04.821012+0000) 2024-02-19T22:22:05.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:05.815+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:05.821235+0000) 2024-02-19T22:22:06.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:06.815+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:06.821409+0000) 2024-02-19T22:22:07.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:07.815+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:07.821589+0000) 2024-02-19T22:22:08.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:08.815+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:08.821711+0000) 2024-02-19T22:22:09.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:09.815+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:09.821843+0000) 2024-02-19T22:22:10.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:10.815+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:10.822061+0000) 2024-02-19T22:22:11.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:11.815+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:11.822230+0000) 2024-02-19T22:22:12.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:12.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:12.822484+0000) 2024-02-19T22:22:13.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:13.815+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:13.822673+0000) 2024-02-19T22:22:14.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:14.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:14.822881+0000) 2024-02-19T22:22:15.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:15.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:15.823076+0000) 2024-02-19T22:22:16.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:16.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:16.823214+0000) 2024-02-19T22:22:17.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:17.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:17.823458+0000) 2024-02-19T22:22:18.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:18.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:18.823585+0000) 2024-02-19T22:22:19.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:19.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:19.823719+0000) 2024-02-19T22:22:20.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:20.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:20.823844+0000) 2024-02-19T22:22:21.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:21.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:21.823970+0000) 2024-02-19T22:22:22.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:22.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:22.824098+0000) 2024-02-19T22:22:23.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:23.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:23.824332+0000) 2024-02-19T22:22:24.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:24.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:24.824452+0000) 2024-02-19T22:22:25.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:25.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:25.824580+0000) 2024-02-19T22:22:26.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:26.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:26.824756+0000) 2024-02-19T22:22:27.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:27.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:27.824930+0000) 2024-02-19T22:22:28.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:28.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:28.825189+0000) 2024-02-19T22:22:29.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:29.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:29.825335+0000) 2024-02-19T22:22:30.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:30.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:30.825471+0000) 2024-02-19T22:22:31.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:31.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:31.825617+0000) 2024-02-19T22:22:32.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:32.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:32.825767+0000) 2024-02-19T22:22:33.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:33.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:33.825897+0000) 2024-02-19T22:22:34.081 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:22:34.085 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:22:34.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:34.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:34.826082+0000) 2024-02-19T22:22:35.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:35.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:35.826343+0000) 2024-02-19T22:22:36.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:36.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:36.826489+0000) 2024-02-19T22:22:37.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:37.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:37.826619+0000) 2024-02-19T22:22:38.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:38.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:38.826805+0000) 2024-02-19T22:22:39.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:39.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:39.826958+0000) 2024-02-19T22:22:40.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:40.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:40.827174+0000) 2024-02-19T22:22:41.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:41.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:41.827314+0000) 2024-02-19T22:22:42.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:42.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:42.827456+0000) 2024-02-19T22:22:43.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:43.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:43.827613+0000) 2024-02-19T22:22:44.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:44.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:44.827849+0000) 2024-02-19T22:22:45.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:45.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:45.828056+0000) 2024-02-19T22:22:46.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:46.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:46.828272+0000) 2024-02-19T22:22:47.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:47.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:47.828511+0000) 2024-02-19T22:22:48.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:48.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:48.828729+0000) 2024-02-19T22:22:49.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:49.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:49.828901+0000) 2024-02-19T22:22:50.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:50.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:50.829089+0000) 2024-02-19T22:22:51.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:51.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:51.829327+0000) 2024-02-19T22:22:52.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:52.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:52.829469+0000) 2024-02-19T22:22:53.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:53.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:53.829692+0000) 2024-02-19T22:22:54.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:54.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:54.829871+0000) 2024-02-19T22:22:55.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:55.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:55.830035+0000) 2024-02-19T22:22:56.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:56.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:56.830191+0000) 2024-02-19T22:22:57.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:57.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:57.830347+0000) 2024-02-19T22:22:58.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:58.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:58.830676+0000) 2024-02-19T22:22:59.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:22:59.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:22:59.830931+0000) 2024-02-19T22:23:00.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:00.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:00.831173+0000) 2024-02-19T22:23:01.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:01.826+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:01.831426+0000) 2024-02-19T22:23:02.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:02.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:02.831554+0000) 2024-02-19T22:23:03.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:03.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:03.831698+0000) 2024-02-19T22:23:04.220 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:23:04.224 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:23:04.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:04.826+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:04.831917+0000) 2024-02-19T22:23:05.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:05.826+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:05.832097+0000) 2024-02-19T22:23:06.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:06.826+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:06.832376+0000) 2024-02-19T22:23:07.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:07.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:07.832638+0000) 2024-02-19T22:23:08.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:08.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:08.832864+0000) 2024-02-19T22:23:09.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:09.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:09.833042+0000) 2024-02-19T22:23:10.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:10.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:10.833257+0000) 2024-02-19T22:23:11.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:11.826+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:11.833418+0000) 2024-02-19T22:23:12.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:12.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:12.833548+0000) 2024-02-19T22:23:13.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:13.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:13.833786+0000) 2024-02-19T22:23:14.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:14.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:14.833966+0000) 2024-02-19T22:23:15.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:15.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:15.834091+0000) 2024-02-19T22:23:16.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:16.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:16.834316+0000) 2024-02-19T22:23:17.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:17.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:17.834541+0000) 2024-02-19T22:23:18.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:18.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:18.834810+0000) 2024-02-19T22:23:19.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:19.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:19.834993+0000) 2024-02-19T22:23:20.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:20.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:20.835115+0000) 2024-02-19T22:23:21.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:21.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:21.835323+0000) 2024-02-19T22:23:22.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:22.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:22.835542+0000) 2024-02-19T22:23:23.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:23.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:23.835688+0000) 2024-02-19T22:23:24.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:24.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:24.835924+0000) 2024-02-19T22:23:25.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:25.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:25.836153+0000) 2024-02-19T22:23:26.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:26.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:26.836352+0000) 2024-02-19T22:23:27.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:27.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:27.836534+0000) 2024-02-19T22:23:28.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:28.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:28.836753+0000) 2024-02-19T22:23:29.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:29.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:29.837031+0000) 2024-02-19T22:23:30.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:30.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:30.837235+0000) 2024-02-19T22:23:31.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:31.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:31.837510+0000) 2024-02-19T22:23:32.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:32.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:32.837722+0000) 2024-02-19T22:23:33.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:33.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:33.837860+0000) 2024-02-19T22:23:34.350 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:23:34.354 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:23:34.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:34.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:34.838053+0000) 2024-02-19T22:23:35.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:35.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:35.838222+0000) 2024-02-19T22:23:36.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:36.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:36.838408+0000) 2024-02-19T22:23:37.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:37.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:37.838586+0000) 2024-02-19T22:23:38.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:38.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:38.838801+0000) 2024-02-19T22:23:39.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:39.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:39.838952+0000) 2024-02-19T22:23:40.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:40.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:40.839199+0000) 2024-02-19T22:23:41.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:41.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:41.839460+0000) 2024-02-19T22:23:42.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:42.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:42.839643+0000) 2024-02-19T22:23:43.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:43.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:43.839798+0000) 2024-02-19T22:23:44.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:44.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:44.840047+0000) 2024-02-19T22:23:45.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:45.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:45.840290+0000) 2024-02-19T22:23:46.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:46.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:46.840533+0000) 2024-02-19T22:23:47.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:47.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:47.840718+0000) 2024-02-19T22:23:48.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:48.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:48.840954+0000) 2024-02-19T22:23:49.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:49.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:49.841146+0000) 2024-02-19T22:23:50.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:50.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:50.841363+0000) 2024-02-19T22:23:51.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:51.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:51.841594+0000) 2024-02-19T22:23:52.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:52.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:52.841756+0000) 2024-02-19T22:23:53.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:53.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:53.841982+0000) 2024-02-19T22:23:54.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:54.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:54.842147+0000) 2024-02-19T22:23:55.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:55.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:55.842328+0000) 2024-02-19T22:23:56.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:56.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:56.842559+0000) 2024-02-19T22:23:57.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:57.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:57.842746+0000) 2024-02-19T22:23:58.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:58.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:58.842924+0000) 2024-02-19T22:23:59.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:23:59.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:23:59.843065+0000) 2024-02-19T22:24:00.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:00.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:00.843186+0000) 2024-02-19T22:24:01.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:01.841+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:01.843421+0000) 2024-02-19T22:24:02.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:02.841+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:02.843624+0000) 2024-02-19T22:24:03.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:03.841+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:03.843794+0000) 2024-02-19T22:24:04.424 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:24:04.427 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:24:04.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:04.841+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:04.843970+0000) 2024-02-19T22:24:05.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:05.841+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:05.844125+0000) 2024-02-19T22:24:06.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:06.841+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:06.844290+0000) 2024-02-19T22:24:07.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:07.841+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:07.844471+0000) 2024-02-19T22:24:08.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:08.841+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:08.844662+0000) 2024-02-19T22:24:09.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:09.841+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:09.844823+0000) 2024-02-19T22:24:10.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:10.841+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:10.844958+0000) 2024-02-19T22:24:11.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:11.841+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:11.845090+0000) 2024-02-19T22:24:12.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:12.841+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:12.845259+0000) 2024-02-19T22:24:13.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:13.841+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:13.845516+0000) 2024-02-19T22:24:14.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:14.841+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:14.845681+0000) 2024-02-19T22:24:15.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:15.841+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:15.845886+0000) 2024-02-19T22:24:16.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:16.841+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:16.846179+0000) 2024-02-19T22:24:17.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:17.841+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:17.846361+0000) 2024-02-19T22:24:18.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:18.841+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:18.846537+0000) 2024-02-19T22:24:19.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:19.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:19.846768+0000) 2024-02-19T22:24:20.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:20.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:20.846914+0000) 2024-02-19T22:24:21.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:21.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:21.847089+0000) 2024-02-19T22:24:22.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:22.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:22.847324+0000) 2024-02-19T22:24:23.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:23.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:23.847564+0000) 2024-02-19T22:24:24.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:24.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:24.847806+0000) 2024-02-19T22:24:25.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:25.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:25.847980+0000) 2024-02-19T22:24:26.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:26.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:26.848129+0000) 2024-02-19T22:24:27.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:27.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:27.848297+0000) 2024-02-19T22:24:28.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:28.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:28.848481+0000) 2024-02-19T22:24:29.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:29.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:29.848659+0000) 2024-02-19T22:24:30.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:30.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:30.848829+0000) 2024-02-19T22:24:31.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:31.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:31.848997+0000) 2024-02-19T22:24:32.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:32.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:32.849166+0000) 2024-02-19T22:24:33.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:33.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:33.849391+0000) 2024-02-19T22:24:34.562 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:24:34.566 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:24:34.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:34.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:34.849561+0000) 2024-02-19T22:24:35.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:35.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:35.849721+0000) 2024-02-19T22:24:36.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:36.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:36.849947+0000) 2024-02-19T22:24:37.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:37.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:37.850115+0000) 2024-02-19T22:24:38.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:38.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:38.850254+0000) 2024-02-19T22:24:39.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:39.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:39.850481+0000) 2024-02-19T22:24:40.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:40.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:40.850665+0000) 2024-02-19T22:24:41.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:41.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:41.850804+0000) 2024-02-19T22:24:42.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:42.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:42.850930+0000) 2024-02-19T22:24:43.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:43.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:43.851080+0000) 2024-02-19T22:24:44.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:44.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:44.851299+0000) 2024-02-19T22:24:45.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:45.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:45.851550+0000) 2024-02-19T22:24:46.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:46.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:46.851704+0000) 2024-02-19T22:24:47.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:47.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:47.851897+0000) 2024-02-19T22:24:48.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:48.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:48.852118+0000) 2024-02-19T22:24:49.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:49.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:49.852300+0000) 2024-02-19T22:24:50.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:50.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:50.852450+0000) 2024-02-19T22:24:51.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:51.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:51.852676+0000) 2024-02-19T22:24:52.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:52.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:52.852911+0000) 2024-02-19T22:24:53.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:53.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:53.853197+0000) 2024-02-19T22:24:54.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:54.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:54.853420+0000) 2024-02-19T22:24:55.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:55.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:55.853679+0000) 2024-02-19T22:24:56.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:56.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:56.853803+0000) 2024-02-19T22:24:57.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:57.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:57.854016+0000) 2024-02-19T22:24:58.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:58.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:58.854149+0000) 2024-02-19T22:24:59.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:24:59.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:24:59.854277+0000) 2024-02-19T22:25:00.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:00.848+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:00.854442+0000) 2024-02-19T22:25:01.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:01.848+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:01.854664+0000) 2024-02-19T22:25:02.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:02.852+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:02.854916+0000) 2024-02-19T22:25:03.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:03.852+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:03.855169+0000) 2024-02-19T22:25:04.676 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:25:04.680 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:25:04.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:04.852+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:04.855397+0000) 2024-02-19T22:25:05.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:05.852+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:05.855622+0000) 2024-02-19T22:25:06.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:06.852+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:06.855848+0000) 2024-02-19T22:25:07.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:07.852+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:07.856030+0000) 2024-02-19T22:25:08.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:08.852+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:08.856245+0000) 2024-02-19T22:25:09.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:09.852+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:09.856482+0000) 2024-02-19T22:25:10.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:10.852+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:10.856786+0000) 2024-02-19T22:25:11.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:11.852+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:11.856922+0000) 2024-02-19T22:25:12.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:12.852+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:12.857088+0000) 2024-02-19T22:25:13.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:13.852+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:13.857278+0000) 2024-02-19T22:25:14.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:14.856+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:14.857451+0000) 2024-02-19T22:25:15.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:15.852+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:15.857619+0000) 2024-02-19T22:25:16.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:16.856+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:16.857870+0000) 2024-02-19T22:25:17.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:17.856+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:17.858146+0000) 2024-02-19T22:25:18.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:18.852+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:18.858377+0000) 2024-02-19T22:25:19.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:19.852+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:19.858601+0000) 2024-02-19T22:25:20.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:20.856+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:20.858876+0000) 2024-02-19T22:25:21.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:21.856+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:21.859112+0000) 2024-02-19T22:25:22.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:22.856+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:22.859307+0000) 2024-02-19T22:25:23.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:23.856+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:23.859510+0000) 2024-02-19T22:25:24.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:24.856+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:24.859641+0000) 2024-02-19T22:25:25.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:25.856+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:25.859844+0000) 2024-02-19T22:25:26.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:26.856+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:26.860064+0000) 2024-02-19T22:25:27.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:27.856+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:27.860242+0000) 2024-02-19T22:25:28.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:28.856+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:28.860398+0000) 2024-02-19T22:25:29.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:29.856+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:29.860616+0000) 2024-02-19T22:25:30.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:30.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:30.860784+0000) 2024-02-19T22:25:31.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:31.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:31.861015+0000) 2024-02-19T22:25:32.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:32.856+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:32.861259+0000) 2024-02-19T22:25:33.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:33.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:33.861427+0000) 2024-02-19T22:25:34.807 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:25:34.811 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:25:34.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:34.856+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:34.861571+0000) 2024-02-19T22:25:35.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:35.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:35.861822+0000) 2024-02-19T22:25:36.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:36.856+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:36.862078+0000) 2024-02-19T22:25:37.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:37.856+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:37.862253+0000) 2024-02-19T22:25:38.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:38.856+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:38.862471+0000) 2024-02-19T22:25:39.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:39.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:39.862625+0000) 2024-02-19T22:25:40.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:40.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:40.862781+0000) 2024-02-19T22:25:41.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:41.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:41.863004+0000) 2024-02-19T22:25:42.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:42.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:42.863227+0000) 2024-02-19T22:25:43.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:43.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:43.863420+0000) 2024-02-19T22:25:44.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:44.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:44.863556+0000) 2024-02-19T22:25:45.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:45.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:45.863686+0000) 2024-02-19T22:25:46.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:46.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:46.863810+0000) 2024-02-19T22:25:47.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:47.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:47.863940+0000) 2024-02-19T22:25:48.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:48.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:48.864065+0000) 2024-02-19T22:25:49.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:49.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:49.864198+0000) 2024-02-19T22:25:50.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:50.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:50.864325+0000) 2024-02-19T22:25:51.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:51.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:51.864462+0000) 2024-02-19T22:25:52.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:52.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:52.864618+0000) 2024-02-19T22:25:53.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:53.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:53.864861+0000) 2024-02-19T22:25:54.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:54.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:54.865014+0000) 2024-02-19T22:25:55.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:55.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:55.865146+0000) 2024-02-19T22:25:56.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:56.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:56.865343+0000) 2024-02-19T22:25:57.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:57.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:57.865474+0000) 2024-02-19T22:25:58.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:58.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:58.865732+0000) 2024-02-19T22:25:59.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:25:59.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:25:59.865898+0000) 2024-02-19T22:26:00.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:00.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:00.866027+0000) 2024-02-19T22:26:01.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:01.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:01.866166+0000) 2024-02-19T22:26:02.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:02.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:02.866298+0000) 2024-02-19T22:26:03.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:03.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:03.866421+0000) 2024-02-19T22:26:04.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:04.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:04.866547+0000) 2024-02-19T22:26:04.888 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:26:04.893 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:26:05.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:05.859+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:05.866674+0000) 2024-02-19T22:26:06.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:06.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:06.866790+0000) 2024-02-19T22:26:07.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:07.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:07.866938+0000) 2024-02-19T22:26:08.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:08.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:08.867120+0000) 2024-02-19T22:26:09.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:09.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:09.867281+0000) 2024-02-19T22:26:10.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:10.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:10.867425+0000) 2024-02-19T22:26:11.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:11.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:11.867548+0000) 2024-02-19T22:26:12.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:12.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:12.867710+0000) 2024-02-19T22:26:13.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:13.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:13.867897+0000) 2024-02-19T22:26:14.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:14.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:14.868059+0000) 2024-02-19T22:26:15.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:15.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:15.868251+0000) 2024-02-19T22:26:16.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:16.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:16.868494+0000) 2024-02-19T22:26:17.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:17.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:17.868774+0000) 2024-02-19T22:26:18.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:18.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:18.869018+0000) 2024-02-19T22:26:19.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:19.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:19.869262+0000) 2024-02-19T22:26:20.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:20.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:20.869505+0000) 2024-02-19T22:26:21.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:21.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:21.869726+0000) 2024-02-19T22:26:22.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:22.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:22.869896+0000) 2024-02-19T22:26:23.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:23.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:23.870071+0000) 2024-02-19T22:26:24.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:24.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:24.870248+0000) 2024-02-19T22:26:25.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:25.863+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:25.870475+0000) 2024-02-19T22:26:26.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:26.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:26.870786+0000) 2024-02-19T22:26:27.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:27.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:27.870932+0000) 2024-02-19T22:26:28.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:28.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:28.871062+0000) 2024-02-19T22:26:29.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:29.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:29.871186+0000) 2024-02-19T22:26:30.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:30.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:30.871364+0000) 2024-02-19T22:26:31.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:31.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:31.871580+0000) 2024-02-19T22:26:32.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:32.871+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:32.871803+0000) 2024-02-19T22:26:33.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:33.871+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:33.872039+0000) 2024-02-19T22:26:34.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:34.871+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:34.872282+0000) 2024-02-19T22:26:35.036 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:26:35.040 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:26:35.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:35.871+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:35.872494+0000) 2024-02-19T22:26:36.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:36.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:36.872701+0000) 2024-02-19T22:26:37.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:37.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:37.872900+0000) 2024-02-19T22:26:38.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:38.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:38.873154+0000) 2024-02-19T22:26:39.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:39.871+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:39.873414+0000) 2024-02-19T22:26:40.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:40.871+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:40.873571+0000) 2024-02-19T22:26:41.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:41.871+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:41.873791+0000) 2024-02-19T22:26:42.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:42.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:42.874010+0000) 2024-02-19T22:26:43.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:43.871+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:43.874279+0000) 2024-02-19T22:26:44.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:44.867+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:44.874554+0000) 2024-02-19T22:26:45.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:45.871+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:45.874736+0000) 2024-02-19T22:26:46.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:46.871+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:46.874983+0000) 2024-02-19T22:26:47.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:47.871+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:47.875182+0000) 2024-02-19T22:26:48.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:48.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:48.875420+0000) 2024-02-19T22:26:49.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:49.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:49.875680+0000) 2024-02-19T22:26:50.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:50.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:50.875935+0000) 2024-02-19T22:26:51.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:51.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:51.876214+0000) 2024-02-19T22:26:52.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:52.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:52.876453+0000) 2024-02-19T22:26:53.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:53.871+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:53.876671+0000) 2024-02-19T22:26:54.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:54.871+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:54.876815+0000) 2024-02-19T22:26:55.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:55.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:55.876978+0000) 2024-02-19T22:26:56.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:56.871+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:56.877241+0000) 2024-02-19T22:26:57.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:57.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:57.877386+0000) 2024-02-19T22:26:58.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:58.871+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:58.877610+0000) 2024-02-19T22:26:59.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:26:59.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:26:59.877881+0000) 2024-02-19T22:27:00.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:00.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:00.878035+0000) 2024-02-19T22:27:01.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:01.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:01.878311+0000) 2024-02-19T22:27:02.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:02.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:02.878537+0000) 2024-02-19T22:27:03.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:03.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:03.878777+0000) 2024-02-19T22:27:04.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:04.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:04.879012+0000) 2024-02-19T22:27:05.175 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:27:05.178 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:27:05.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:05.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:05.879261+0000) 2024-02-19T22:27:06.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:06.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:06.879492+0000) 2024-02-19T22:27:07.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:07.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:07.879620+0000) 2024-02-19T22:27:08.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:08.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:08.879750+0000) 2024-02-19T22:27:09.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:09.874+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:09.879964+0000) 2024-02-19T22:27:10.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:10.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:10.880186+0000) 2024-02-19T22:27:11.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:11.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:11.880331+0000) 2024-02-19T22:27:12.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:12.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:12.880537+0000) 2024-02-19T22:27:13.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:13.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:13.880742+0000) 2024-02-19T22:27:14.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:14.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:14.880915+0000) 2024-02-19T22:27:15.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:15.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:15.881195+0000) 2024-02-19T22:27:16.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:16.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:16.881479+0000) 2024-02-19T22:27:17.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:17.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:17.881653+0000) 2024-02-19T22:27:18.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:18.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:18.881855+0000) 2024-02-19T22:27:19.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:19.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:19.882047+0000) 2024-02-19T22:27:20.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:20.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:20.882195+0000) 2024-02-19T22:27:21.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:21.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:21.882417+0000) 2024-02-19T22:27:22.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:22.878+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:22.882621+0000) 2024-02-19T22:27:23.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:23.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:23.882800+0000) 2024-02-19T22:27:24.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:24.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:24.882957+0000) 2024-02-19T22:27:25.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:25.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:25.883089+0000) 2024-02-19T22:27:26.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:26.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:26.883251+0000) 2024-02-19T22:27:27.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:27.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:27.883380+0000) 2024-02-19T22:27:28.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:28.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:28.883541+0000) 2024-02-19T22:27:29.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:29.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:29.883675+0000) 2024-02-19T22:27:30.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:30.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:30.883812+0000) 2024-02-19T22:27:31.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:31.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:31.883944+0000) 2024-02-19T22:27:32.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:32.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:32.884073+0000) 2024-02-19T22:27:33.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:33.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:33.884195+0000) 2024-02-19T22:27:34.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:34.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:34.884321+0000) 2024-02-19T22:27:35.297 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:27:35.302 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:27:35.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:35.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:35.884465+0000) 2024-02-19T22:27:36.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:36.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:36.884622+0000) 2024-02-19T22:27:37.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:37.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:37.884753+0000) 2024-02-19T22:27:38.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:38.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:38.884872+0000) 2024-02-19T22:27:39.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:39.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:39.884998+0000) 2024-02-19T22:27:40.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:40.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:40.885139+0000) 2024-02-19T22:27:41.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:41.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:41.885367+0000) 2024-02-19T22:27:42.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:42.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:42.885604+0000) 2024-02-19T22:27:43.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:43.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:43.885747+0000) 2024-02-19T22:27:44.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:44.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:44.885888+0000) 2024-02-19T22:27:45.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:45.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:45.886036+0000) 2024-02-19T22:27:46.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:46.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:46.886179+0000) 2024-02-19T22:27:47.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:47.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:47.886326+0000) 2024-02-19T22:27:48.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:48.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:48.886466+0000) 2024-02-19T22:27:49.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:49.882+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:49.886613+0000) 2024-02-19T22:27:50.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:50.886+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:50.886769+0000) 2024-02-19T22:27:51.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:51.886+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:51.886890+0000) 2024-02-19T22:27:52.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:52.886+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:52.887026+0000) 2024-02-19T22:27:53.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:53.886+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:53.887154+0000) 2024-02-19T22:27:54.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:54.886+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:54.887338+0000) 2024-02-19T22:27:55.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:55.886+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:55.887551+0000) 2024-02-19T22:27:56.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:56.886+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:56.887702+0000) 2024-02-19T22:27:57.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:57.886+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:57.887890+0000) 2024-02-19T22:27:58.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:58.886+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:58.888024+0000) 2024-02-19T22:27:59.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:27:59.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:27:59.888227+0000) 2024-02-19T22:28:00.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:00.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:00.888372+0000) 2024-02-19T22:28:01.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:01.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:01.888579+0000) 2024-02-19T22:28:02.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:02.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:02.888720+0000) 2024-02-19T22:28:03.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:03.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:03.888839+0000) 2024-02-19T22:28:04.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:04.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:04.889061+0000) 2024-02-19T22:28:05.427 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:28:05.431 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:28:05.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:05.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:05.889289+0000) 2024-02-19T22:28:06.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:06.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:06.889441+0000) 2024-02-19T22:28:07.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:07.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:07.889624+0000) 2024-02-19T22:28:08.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:08.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:08.889831+0000) 2024-02-19T22:28:09.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:09.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:09.889992+0000) 2024-02-19T22:28:10.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:10.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:10.890166+0000) 2024-02-19T22:28:11.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:11.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:11.890416+0000) 2024-02-19T22:28:12.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:12.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:12.890637+0000) 2024-02-19T22:28:13.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:13.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:13.890810+0000) 2024-02-19T22:28:14.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:14.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:14.891028+0000) 2024-02-19T22:28:15.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:15.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:15.891230+0000) 2024-02-19T22:28:16.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:16.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:16.891481+0000) 2024-02-19T22:28:17.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:17.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:17.891734+0000) 2024-02-19T22:28:18.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:18.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:18.891955+0000) 2024-02-19T22:28:19.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:19.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:19.892117+0000) 2024-02-19T22:28:20.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:20.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:20.892292+0000) 2024-02-19T22:28:21.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:21.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:21.892428+0000) 2024-02-19T22:28:22.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:22.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:22.892657+0000) 2024-02-19T22:28:23.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:23.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:23.892793+0000) 2024-02-19T22:28:24.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:24.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:24.893021+0000) 2024-02-19T22:28:25.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:25.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:25.893198+0000) 2024-02-19T22:28:26.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:26.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:26.893376+0000) 2024-02-19T22:28:27.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:27.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:27.893602+0000) 2024-02-19T22:28:28.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:28.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:28.893775+0000) 2024-02-19T22:28:29.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:29.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:29.893939+0000) 2024-02-19T22:28:30.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:30.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:30.894101+0000) 2024-02-19T22:28:31.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:31.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:31.894263+0000) 2024-02-19T22:28:32.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:32.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:32.894400+0000) 2024-02-19T22:28:33.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:33.889+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:33.894526+0000) 2024-02-19T22:28:34.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:34.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:34.894746+0000) 2024-02-19T22:28:35.566 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:28:35.570 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:28:35.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:35.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:35.894992+0000) 2024-02-19T22:28:36.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:36.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:36.895170+0000) 2024-02-19T22:28:37.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:37.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:37.895336+0000) 2024-02-19T22:28:38.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:38.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:38.895481+0000) 2024-02-19T22:28:39.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:39.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:39.895669+0000) 2024-02-19T22:28:40.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:40.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:40.895817+0000) 2024-02-19T22:28:41.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:41.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:41.895947+0000) 2024-02-19T22:28:42.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:42.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:42.896082+0000) 2024-02-19T22:28:43.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:43.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:43.896251+0000) 2024-02-19T22:28:44.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:44.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:44.896382+0000) 2024-02-19T22:28:45.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:45.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:45.896603+0000) 2024-02-19T22:28:46.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:46.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:46.896811+0000) 2024-02-19T22:28:47.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:47.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:47.897033+0000) 2024-02-19T22:28:48.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:48.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:48.897155+0000) 2024-02-19T22:28:49.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:49.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:49.897355+0000) 2024-02-19T22:28:50.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:50.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:50.897565+0000) 2024-02-19T22:28:51.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:51.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:51.897740+0000) 2024-02-19T22:28:52.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:52.897+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:52.897942+0000) 2024-02-19T22:28:53.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:53.897+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:53.898173+0000) 2024-02-19T22:28:54.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:54.897+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:54.898415+0000) 2024-02-19T22:28:55.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:55.893+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:55.898589+0000) 2024-02-19T22:28:56.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:56.897+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:56.898770+0000) 2024-02-19T22:28:57.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:57.897+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:57.899014+0000) 2024-02-19T22:28:58.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:58.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:58.899237+0000) 2024-02-19T22:28:59.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:28:59.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:28:59.899382+0000) 2024-02-19T22:29:00.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:00.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:00.899585+0000) 2024-02-19T22:29:01.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:01.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:01.899844+0000) 2024-02-19T22:29:02.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:02.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:02.900100+0000) 2024-02-19T22:29:03.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:03.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:03.900286+0000) 2024-02-19T22:29:04.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:04.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:04.900504+0000) 2024-02-19T22:29:05.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:05.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:05.900784+0000) 2024-02-19T22:29:05.931 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:29:05.934 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:29:06.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:06.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:06.900958+0000) 2024-02-19T22:29:07.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:07.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:07.901186+0000) 2024-02-19T22:29:08.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:08.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:08.901413+0000) 2024-02-19T22:29:09.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:09.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:09.901643+0000) 2024-02-19T22:29:10.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:10.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:10.901879+0000) 2024-02-19T22:29:11.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:11.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:11.902123+0000) 2024-02-19T22:29:12.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:12.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:12.902274+0000) 2024-02-19T22:29:13.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:13.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:13.902448+0000) 2024-02-19T22:29:14.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:14.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:14.902763+0000) 2024-02-19T22:29:15.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:15.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:15.902973+0000) 2024-02-19T22:29:16.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:16.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:16.903110+0000) 2024-02-19T22:29:17.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:17.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:17.903271+0000) 2024-02-19T22:29:18.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:18.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:18.903437+0000) 2024-02-19T22:29:19.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:19.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:19.903615+0000) 2024-02-19T22:29:20.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:20.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:20.903855+0000) 2024-02-19T22:29:21.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:21.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:21.904068+0000) 2024-02-19T22:29:22.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:22.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:22.904300+0000) 2024-02-19T22:29:23.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:23.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:23.904534+0000) 2024-02-19T22:29:24.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:24.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:24.904749+0000) 2024-02-19T22:29:25.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:25.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:25.904993+0000) 2024-02-19T22:29:26.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:26.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:26.905259+0000) 2024-02-19T22:29:27.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:27.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:27.905474+0000) 2024-02-19T22:29:28.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:28.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:28.905727+0000) 2024-02-19T22:29:29.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:29.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:29.905915+0000) 2024-02-19T22:29:30.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:30.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:30.906067+0000) 2024-02-19T22:29:31.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:31.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:31.906223+0000) 2024-02-19T22:29:32.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:32.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:32.906445+0000) 2024-02-19T22:29:33.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:33.900+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:33.906678+0000) 2024-02-19T22:29:34.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:34.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:34.906932+0000) 2024-02-19T22:29:35.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:35.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:35.907159+0000) 2024-02-19T22:29:35.985 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:29:35.989 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:29:36.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:36.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:36.907295+0000) 2024-02-19T22:29:37.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:37.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:37.907492+0000) 2024-02-19T22:29:38.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:38.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:38.907685+0000) 2024-02-19T22:29:39.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:39.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:39.907830+0000) 2024-02-19T22:29:40.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:40.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:40.908040+0000) 2024-02-19T22:29:41.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:41.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:41.908199+0000) 2024-02-19T22:29:42.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:42.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:42.908392+0000) 2024-02-19T22:29:43.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:43.908+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:43.908543+0000) 2024-02-19T22:29:44.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:44.908+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:44.908757+0000) 2024-02-19T22:29:45.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:45.908+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:45.909011+0000) 2024-02-19T22:29:46.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:46.908+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:46.909245+0000) 2024-02-19T22:29:47.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:47.908+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:47.909420+0000) 2024-02-19T22:29:48.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:48.908+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:48.909623+0000) 2024-02-19T22:29:49.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:49.908+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:49.909860+0000) 2024-02-19T22:29:50.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:50.908+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:50.910032+0000) 2024-02-19T22:29:51.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:51.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:51.910173+0000) 2024-02-19T22:29:52.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:52.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:52.910306+0000) 2024-02-19T22:29:53.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:53.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:53.910441+0000) 2024-02-19T22:29:54.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:54.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:54.910556+0000) 2024-02-19T22:29:55.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:55.904+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:55.910688+0000) 2024-02-19T22:29:56.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:56.908+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:56.910822+0000) 2024-02-19T22:29:57.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:57.908+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:57.910947+0000) 2024-02-19T22:29:58.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:58.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:58.911076+0000) 2024-02-19T22:29:59.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:29:59.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:29:59.911276+0000) 2024-02-19T22:30:00.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:00.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:00.911410+0000) 2024-02-19T22:30:01.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:01.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:01.911540+0000) 2024-02-19T22:30:02.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:02.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:02.911679+0000) 2024-02-19T22:30:03.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:03.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:03.911803+0000) 2024-02-19T22:30:04.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:04.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:04.911922+0000) 2024-02-19T22:30:05.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:05.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:05.912046+0000) 2024-02-19T22:30:06.124 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:30:06.128 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:30:06.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:06.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:06.912166+0000) 2024-02-19T22:30:07.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:07.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:07.912293+0000) 2024-02-19T22:30:08.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:08.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:08.912410+0000) 2024-02-19T22:30:09.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:09.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:09.912541+0000) 2024-02-19T22:30:10.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:10.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:10.912661+0000) 2024-02-19T22:30:11.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:11.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:11.912791+0000) 2024-02-19T22:30:12.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:12.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:12.912948+0000) 2024-02-19T22:30:13.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:13.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:13.913118+0000) 2024-02-19T22:30:14.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:14.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:14.913289+0000) 2024-02-19T22:30:15.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:15.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:15.913422+0000) 2024-02-19T22:30:16.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:16.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:16.913615+0000) 2024-02-19T22:30:17.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:17.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:17.913838+0000) 2024-02-19T22:30:18.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:18.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:18.914028+0000) 2024-02-19T22:30:19.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:19.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:19.914259+0000) 2024-02-19T22:30:20.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:20.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:20.914433+0000) 2024-02-19T22:30:21.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:21.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:21.914649+0000) 2024-02-19T22:30:22.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:22.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:22.914872+0000) 2024-02-19T22:30:23.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:23.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:23.915019+0000) 2024-02-19T22:30:24.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:24.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:24.915177+0000) 2024-02-19T22:30:25.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:25.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:25.915299+0000) 2024-02-19T22:30:26.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:26.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:26.915495+0000) 2024-02-19T22:30:27.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:27.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:27.915709+0000) 2024-02-19T22:30:28.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:28.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:28.915934+0000) 2024-02-19T22:30:29.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:29.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:29.916198+0000) 2024-02-19T22:30:30.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:30.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:30.916423+0000) 2024-02-19T22:30:31.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:31.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:31.916592+0000) 2024-02-19T22:30:32.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:32.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:32.916835+0000) 2024-02-19T22:30:33.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:33.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:33.916980+0000) 2024-02-19T22:30:34.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:34.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:34.917204+0000) 2024-02-19T22:30:35.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:35.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:35.917381+0000) 2024-02-19T22:30:36.263 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:30:36.266 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:30:36.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:36.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:36.917546+0000) 2024-02-19T22:30:37.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:37.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:37.917734+0000) 2024-02-19T22:30:38.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:38.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:38.917903+0000) 2024-02-19T22:30:39.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:39.911+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:39.918143+0000) 2024-02-19T22:30:40.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:40.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:40.918380+0000) 2024-02-19T22:30:41.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:41.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:41.918623+0000) 2024-02-19T22:30:42.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:42.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:42.918863+0000) 2024-02-19T22:30:43.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:43.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:43.919066+0000) 2024-02-19T22:30:44.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:44.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:44.919328+0000) 2024-02-19T22:30:45.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:45.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:45.919599+0000) 2024-02-19T22:30:46.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:46.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:46.919828+0000) 2024-02-19T22:30:47.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:47.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:47.920100+0000) 2024-02-19T22:30:48.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:48.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:48.920297+0000) 2024-02-19T22:30:49.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:49.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:49.920479+0000) 2024-02-19T22:30:50.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:50.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:50.920601+0000) 2024-02-19T22:30:51.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:51.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:51.920823+0000) 2024-02-19T22:30:52.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:52.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:52.921051+0000) 2024-02-19T22:30:53.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:53.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:53.921271+0000) 2024-02-19T22:30:54.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:54.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:54.921409+0000) 2024-02-19T22:30:55.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:55.915+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:55.921559+0000) 2024-02-19T22:30:56.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:56.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:56.921790+0000) 2024-02-19T22:30:57.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:57.919+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:57.922025+0000) 2024-02-19T22:30:58.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:58.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:58.922298+0000) 2024-02-19T22:30:59.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:30:59.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:30:59.922448+0000) 2024-02-19T22:31:00.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:00.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:00.922662+0000) 2024-02-19T22:31:01.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:01.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:01.922822+0000) 2024-02-19T22:31:02.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:02.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:02.923015+0000) 2024-02-19T22:31:03.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:03.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:03.923168+0000) 2024-02-19T22:31:04.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:04.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:04.923385+0000) 2024-02-19T22:31:05.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:05.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:05.923536+0000) 2024-02-19T22:31:06.343 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:31:06.347 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:31:06.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:06.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:06.923712+0000) 2024-02-19T22:31:07.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:07.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:07.923857+0000) 2024-02-19T22:31:08.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:08.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:08.924005+0000) 2024-02-19T22:31:09.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:09.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:09.924266+0000) 2024-02-19T22:31:10.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:10.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:10.924484+0000) 2024-02-19T22:31:11.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:11.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:11.924637+0000) 2024-02-19T22:31:12.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:12.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:12.924858+0000) 2024-02-19T22:31:13.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:13.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:13.924999+0000) 2024-02-19T22:31:14.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:14.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:14.925206+0000) 2024-02-19T22:31:15.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:15.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:15.925344+0000) 2024-02-19T22:31:16.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:16.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:16.925477+0000) 2024-02-19T22:31:17.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:17.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:17.925688+0000) 2024-02-19T22:31:18.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:18.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:18.925826+0000) 2024-02-19T22:31:19.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:19.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:19.925980+0000) 2024-02-19T22:31:20.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:20.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:20.926188+0000) 2024-02-19T22:31:21.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:21.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:21.926358+0000) 2024-02-19T22:31:22.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:22.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:22.926567+0000) 2024-02-19T22:31:23.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:23.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:23.926828+0000) 2024-02-19T22:31:24.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:24.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:24.926995+0000) 2024-02-19T22:31:25.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:25.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:25.927166+0000) 2024-02-19T22:31:26.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:26.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:26.927285+0000) 2024-02-19T22:31:27.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:27.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:27.927456+0000) 2024-02-19T22:31:28.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:28.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:28.927591+0000) 2024-02-19T22:31:29.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:29.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:29.927724+0000) 2024-02-19T22:31:30.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:30.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:30.927917+0000) 2024-02-19T22:31:31.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:31.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:31.928073+0000) 2024-02-19T22:31:32.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:32.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:32.928301+0000) 2024-02-19T22:31:33.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:33.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:33.928444+0000) 2024-02-19T22:31:34.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:34.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:34.928568+0000) 2024-02-19T22:31:35.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:35.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:35.928710+0000) 2024-02-19T22:31:36.474 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:31:36.478 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:31:36.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:36.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:36.928932+0000) 2024-02-19T22:31:37.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:37.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:37.929189+0000) 2024-02-19T22:31:38.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:38.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:38.929404+0000) 2024-02-19T22:31:39.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:39.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:39.929646+0000) 2024-02-19T22:31:40.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:40.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:40.929918+0000) 2024-02-19T22:31:41.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:41.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:41.930116+0000) 2024-02-19T22:31:42.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:42.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:42.930317+0000) 2024-02-19T22:31:43.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:43.926+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:43.930450+0000) 2024-02-19T22:31:44.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:44.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:44.930671+0000) 2024-02-19T22:31:45.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:45.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:45.930921+0000) 2024-02-19T22:31:46.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:46.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:46.931099+0000) 2024-02-19T22:31:47.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:47.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:47.931240+0000) 2024-02-19T22:31:48.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:48.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:48.931371+0000) 2024-02-19T22:31:49.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:49.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:49.931555+0000) 2024-02-19T22:31:50.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:50.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:50.931742+0000) 2024-02-19T22:31:51.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:51.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:51.931924+0000) 2024-02-19T22:31:52.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:52.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:52.932080+0000) 2024-02-19T22:31:53.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:53.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:53.932293+0000) 2024-02-19T22:31:54.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:54.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:54.932442+0000) 2024-02-19T22:31:55.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:55.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:55.932617+0000) 2024-02-19T22:31:56.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:56.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:56.932805+0000) 2024-02-19T22:31:57.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:57.930+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:57.933031+0000) 2024-02-19T22:31:58.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:58.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:58.933182+0000) 2024-02-19T22:31:59.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:31:59.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:31:59.933361+0000) 2024-02-19T22:32:00.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:00.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:00.933562+0000) 2024-02-19T22:32:01.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:01.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:01.933791+0000) 2024-02-19T22:32:02.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:02.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:02.934004+0000) 2024-02-19T22:32:03.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:03.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:03.934129+0000) 2024-02-19T22:32:04.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:04.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:04.934300+0000) 2024-02-19T22:32:05.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:05.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:05.934491+0000) 2024-02-19T22:32:06.613 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:32:06.618 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:32:06.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:06.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:06.934628+0000) 2024-02-19T22:32:07.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:07.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:07.934839+0000) 2024-02-19T22:32:08.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:08.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:08.935030+0000) 2024-02-19T22:32:09.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:09.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:09.935225+0000) 2024-02-19T22:32:10.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:10.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:10.935361+0000) 2024-02-19T22:32:11.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:11.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:11.935500+0000) 2024-02-19T22:32:12.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:12.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:12.935709+0000) 2024-02-19T22:32:13.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:13.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:13.935864+0000) 2024-02-19T22:32:14.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:14.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:14.936080+0000) 2024-02-19T22:32:15.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:15.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:15.936317+0000) 2024-02-19T22:32:16.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:16.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:16.936463+0000) 2024-02-19T22:32:17.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:17.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:17.936640+0000) 2024-02-19T22:32:18.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:18.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:18.936897+0000) 2024-02-19T22:32:19.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:19.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:19.937055+0000) 2024-02-19T22:32:20.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:20.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:20.937222+0000) 2024-02-19T22:32:21.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:21.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:21.937398+0000) 2024-02-19T22:32:22.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:22.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:22.937572+0000) 2024-02-19T22:32:23.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:23.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:23.937832+0000) 2024-02-19T22:32:24.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:24.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:24.937996+0000) 2024-02-19T22:32:25.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:25.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:25.938172+0000) 2024-02-19T22:32:26.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:26.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:26.938416+0000) 2024-02-19T22:32:27.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:27.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:27.938541+0000) 2024-02-19T22:32:28.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:28.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:28.938814+0000) 2024-02-19T22:32:29.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:29.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:29.939051+0000) 2024-02-19T22:32:30.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:30.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:30.939185+0000) 2024-02-19T22:32:31.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:31.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:31.939332+0000) 2024-02-19T22:32:32.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:32.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:32.939452+0000) 2024-02-19T22:32:33.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:33.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:33.939662+0000) 2024-02-19T22:32:34.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:34.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:34.939848+0000) 2024-02-19T22:32:35.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:35.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:35.940005+0000) 2024-02-19T22:32:36.735 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:32:36.739 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:32:36.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:36.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:36.940233+0000) 2024-02-19T22:32:37.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:37.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:37.940480+0000) 2024-02-19T22:32:38.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:38.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:38.940655+0000) 2024-02-19T22:32:39.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:39.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:39.940890+0000) 2024-02-19T22:32:40.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:40.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:40.941100+0000) 2024-02-19T22:32:41.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:41.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:41.941317+0000) 2024-02-19T22:32:42.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:42.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:42.941480+0000) 2024-02-19T22:32:43.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:43.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:43.941733+0000) 2024-02-19T22:32:44.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:44.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:44.941959+0000) 2024-02-19T22:32:45.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:45.937+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:45.942192+0000) 2024-02-19T22:32:46.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:46.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:46.942580+0000) 2024-02-19T22:32:47.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:47.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:47.942743+0000) 2024-02-19T22:32:48.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:48.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:48.942976+0000) 2024-02-19T22:32:49.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:49.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:49.943121+0000) 2024-02-19T22:32:50.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:50.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:50.943249+0000) 2024-02-19T22:32:51.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:51.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:51.943462+0000) 2024-02-19T22:32:52.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:52.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:52.943708+0000) 2024-02-19T22:32:53.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:53.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:53.943915+0000) 2024-02-19T22:32:54.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:54.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:54.944130+0000) 2024-02-19T22:32:55.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:55.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:55.944389+0000) 2024-02-19T22:32:56.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:56.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:56.944627+0000) 2024-02-19T22:32:57.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:57.941+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:57.944768+0000) 2024-02-19T22:32:58.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:58.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:58.944973+0000) 2024-02-19T22:32:59.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:32:59.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:32:59.945115+0000) 2024-02-19T22:33:00.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:00.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:00.945320+0000) 2024-02-19T22:33:01.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:01.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:01.945453+0000) 2024-02-19T22:33:02.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:02.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:02.945624+0000) 2024-02-19T22:33:03.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:03.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:03.945785+0000) 2024-02-19T22:33:04.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:04.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:04.945935+0000) 2024-02-19T22:33:05.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:05.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:05.946188+0000) 2024-02-19T22:33:06.858 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:33:06.861 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:33:06.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:06.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:06.946434+0000) 2024-02-19T22:33:07.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:07.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:07.946651+0000) 2024-02-19T22:33:08.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:08.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:08.946836+0000) 2024-02-19T22:33:09.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:09.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:09.947061+0000) 2024-02-19T22:33:10.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:10.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:10.947196+0000) 2024-02-19T22:33:11.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:11.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:11.947363+0000) 2024-02-19T22:33:12.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:12.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:12.947618+0000) 2024-02-19T22:33:13.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:13.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:13.947883+0000) 2024-02-19T22:33:14.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:14.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:14.948031+0000) 2024-02-19T22:33:15.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:15.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:15.948222+0000) 2024-02-19T22:33:16.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:16.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:16.948387+0000) 2024-02-19T22:33:17.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:17.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:17.948529+0000) 2024-02-19T22:33:18.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:18.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:18.948729+0000) 2024-02-19T22:33:19.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:19.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:19.948905+0000) 2024-02-19T22:33:20.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:20.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:20.949151+0000) 2024-02-19T22:33:21.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:21.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:21.949337+0000) 2024-02-19T22:33:22.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:22.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:22.949591+0000) 2024-02-19T22:33:23.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:23.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:23.949815+0000) 2024-02-19T22:33:24.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:24.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:24.949953+0000) 2024-02-19T22:33:25.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:25.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:25.950168+0000) 2024-02-19T22:33:26.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:26.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:26.950393+0000) 2024-02-19T22:33:27.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:27.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:27.950637+0000) 2024-02-19T22:33:28.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:28.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:28.950860+0000) 2024-02-19T22:33:29.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:29.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:29.951095+0000) 2024-02-19T22:33:30.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:30.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:30.951323+0000) 2024-02-19T22:33:31.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:31.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:31.951502+0000) 2024-02-19T22:33:32.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:32.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:32.951656+0000) 2024-02-19T22:33:33.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:33.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:33.951897+0000) 2024-02-19T22:33:34.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:34.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:34.952097+0000) 2024-02-19T22:33:35.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:35.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:35.952313+0000) 2024-02-19T22:33:36.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:36.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:36.952463+0000) 2024-02-19T22:33:36.963 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:33:36.966 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:33:37.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:37.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:37.952691+0000) 2024-02-19T22:33:38.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:38.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:38.952850+0000) 2024-02-19T22:33:39.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:39.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:39.953046+0000) 2024-02-19T22:33:40.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:40.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:40.953285+0000) 2024-02-19T22:33:41.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:41.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:41.953544+0000) 2024-02-19T22:33:42.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:42.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:42.953771+0000) 2024-02-19T22:33:43.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:43.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:43.954030+0000) 2024-02-19T22:33:44.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:44.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:44.954266+0000) 2024-02-19T22:33:45.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:45.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:45.954418+0000) 2024-02-19T22:33:46.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:46.948+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:46.954647+0000) 2024-02-19T22:33:47.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:47.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:47.954858+0000) 2024-02-19T22:33:48.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:48.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:48.955069+0000) 2024-02-19T22:33:49.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:49.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:49.955268+0000) 2024-02-19T22:33:50.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:50.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:50.955505+0000) 2024-02-19T22:33:51.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:51.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:51.955631+0000) 2024-02-19T22:33:52.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:52.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:52.955858+0000) 2024-02-19T22:33:53.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:53.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:53.956123+0000) 2024-02-19T22:33:54.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:54.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:54.956333+0000) 2024-02-19T22:33:55.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:55.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:55.956481+0000) 2024-02-19T22:33:56.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:56.952+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:56.956726+0000) 2024-02-19T22:33:57.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:57.956+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:57.956957+0000) 2024-02-19T22:33:58.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:58.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:58.957131+0000) 2024-02-19T22:33:59.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:33:59.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:33:59.957310+0000) 2024-02-19T22:34:00.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:00.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:00.957551+0000) 2024-02-19T22:34:01.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:01.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:01.957714+0000) 2024-02-19T22:34:02.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:02.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:02.957856+0000) 2024-02-19T22:34:03.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:03.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:03.958022+0000) 2024-02-19T22:34:04.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:04.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:04.958289+0000) 2024-02-19T22:34:05.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:05.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:05.958455+0000) 2024-02-19T22:34:06.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:06.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:06.958591+0000) 2024-02-19T22:34:07.177 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:34:07.180 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:34:07.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:07.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:07.958739+0000) 2024-02-19T22:34:08.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:08.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:08.958943+0000) 2024-02-19T22:34:09.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:09.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:09.959069+0000) 2024-02-19T22:34:10.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:10.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:10.959192+0000) 2024-02-19T22:34:11.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:11.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:11.959313+0000) 2024-02-19T22:34:12.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:12.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:12.959442+0000) 2024-02-19T22:34:13.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:13.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:13.959657+0000) 2024-02-19T22:34:14.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:14.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:14.959811+0000) 2024-02-19T22:34:15.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:15.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:15.959937+0000) 2024-02-19T22:34:16.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:16.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:16.960081+0000) 2024-02-19T22:34:17.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:17.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:17.960240+0000) 2024-02-19T22:34:18.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:18.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:18.960383+0000) 2024-02-19T22:34:19.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:19.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:19.960552+0000) 2024-02-19T22:34:20.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:20.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:20.960752+0000) 2024-02-19T22:34:21.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:21.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:21.960888+0000) 2024-02-19T22:34:22.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:22.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:22.961019+0000) 2024-02-19T22:34:23.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:23.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:23.961159+0000) 2024-02-19T22:34:24.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:24.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:24.961289+0000) 2024-02-19T22:34:25.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:25.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:25.961412+0000) 2024-02-19T22:34:26.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:26.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:26.961577+0000) 2024-02-19T22:34:27.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:27.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:27.961784+0000) 2024-02-19T22:34:28.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:28.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:28.962033+0000) 2024-02-19T22:34:29.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:29.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:29.962225+0000) 2024-02-19T22:34:30.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:30.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:30.962379+0000) 2024-02-19T22:34:31.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:31.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:31.962539+0000) 2024-02-19T22:34:32.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:32.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:32.962859+0000) 2024-02-19T22:34:33.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:33.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:33.963033+0000) 2024-02-19T22:34:34.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:34.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:34.963281+0000) 2024-02-19T22:34:35.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:35.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:35.963413+0000) 2024-02-19T22:34:36.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:36.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:36.963544+0000) 2024-02-19T22:34:37.240 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:34:37.245 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:34:37.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:37.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:37.963764+0000) 2024-02-19T22:34:38.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:38.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:38.963906+0000) 2024-02-19T22:34:39.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:39.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:39.964140+0000) 2024-02-19T22:34:40.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:40.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:40.964395+0000) 2024-02-19T22:34:41.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:41.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:41.964571+0000) 2024-02-19T22:34:42.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:42.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:42.964715+0000) 2024-02-19T22:34:43.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:43.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:43.964953+0000) 2024-02-19T22:34:44.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:44.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:44.965155+0000) 2024-02-19T22:34:45.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:45.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:45.965342+0000) 2024-02-19T22:34:46.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:46.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:46.965469+0000) 2024-02-19T22:34:47.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:47.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:47.965650+0000) 2024-02-19T22:34:48.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:48.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:48.965846+0000) 2024-02-19T22:34:49.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:49.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:49.966077+0000) 2024-02-19T22:34:50.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:50.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:50.966317+0000) 2024-02-19T22:34:51.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:51.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:51.966480+0000) 2024-02-19T22:34:52.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:52.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:52.966616+0000) 2024-02-19T22:34:53.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:53.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:53.966767+0000) 2024-02-19T22:34:54.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:54.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:54.966911+0000) 2024-02-19T22:34:55.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:55.963+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:55.967085+0000) 2024-02-19T22:34:56.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:56.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:56.967276+0000) 2024-02-19T22:34:57.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:57.967+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:57.967501+0000) 2024-02-19T22:34:58.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:58.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:58.967761+0000) 2024-02-19T22:34:59.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:34:59.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:34:59.967924+0000) 2024-02-19T22:35:00.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:00.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:00.968058+0000) 2024-02-19T22:35:01.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:01.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:01.968293+0000) 2024-02-19T22:35:02.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:02.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:02.968459+0000) 2024-02-19T22:35:03.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:03.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:03.968595+0000) 2024-02-19T22:35:04.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:04.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:04.968827+0000) 2024-02-19T22:35:05.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:05.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:05.968974+0000) 2024-02-19T22:35:06.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:06.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:06.969180+0000) 2024-02-19T22:35:07.354 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:35:07.357 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:35:07.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:07.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:07.969404+0000) 2024-02-19T22:35:08.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:08.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:08.969599+0000) 2024-02-19T22:35:09.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:09.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:09.969876+0000) 2024-02-19T22:35:10.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:10.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:10.970135+0000) 2024-02-19T22:35:11.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:11.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:11.970351+0000) 2024-02-19T22:35:12.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:12.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:12.970537+0000) 2024-02-19T22:35:13.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:13.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:13.970752+0000) 2024-02-19T22:35:14.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:14.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:14.970897+0000) 2024-02-19T22:35:15.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:15.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:15.971020+0000) 2024-02-19T22:35:16.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:16.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:16.971143+0000) 2024-02-19T22:35:17.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:17.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:17.971271+0000) 2024-02-19T22:35:18.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:18.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:18.971502+0000) 2024-02-19T22:35:19.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:19.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:19.971830+0000) 2024-02-19T22:35:20.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:20.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:20.971995+0000) 2024-02-19T22:35:21.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:21.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:21.972171+0000) 2024-02-19T22:35:22.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:22.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:22.972396+0000) 2024-02-19T22:35:23.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:23.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:23.972622+0000) 2024-02-19T22:35:24.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:24.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:24.972791+0000) 2024-02-19T22:35:25.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:25.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:25.972966+0000) 2024-02-19T22:35:26.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:26.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:26.973140+0000) 2024-02-19T22:35:27.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:27.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:27.973278+0000) 2024-02-19T22:35:28.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:28.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:28.973397+0000) 2024-02-19T22:35:29.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:29.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:29.973572+0000) 2024-02-19T22:35:30.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:30.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:30.973751+0000) 2024-02-19T22:35:31.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:31.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:31.973889+0000) 2024-02-19T22:35:32.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:32.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:32.974040+0000) 2024-02-19T22:35:33.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:33.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:33.974270+0000) 2024-02-19T22:35:34.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:34.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:34.974436+0000) 2024-02-19T22:35:35.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:35.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:35.974658+0000) 2024-02-19T22:35:36.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:36.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:36.974915+0000) 2024-02-19T22:35:37.412 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:35:37.416 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:35:37.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:37.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:37.975074+0000) 2024-02-19T22:35:38.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:38.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:38.975249+0000) 2024-02-19T22:35:39.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:39.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:39.975480+0000) 2024-02-19T22:35:40.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:40.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:40.975696+0000) 2024-02-19T22:35:41.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:41.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:41.975906+0000) 2024-02-19T22:35:42.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:42.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:42.976065+0000) 2024-02-19T22:35:43.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:43.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:43.976285+0000) 2024-02-19T22:35:44.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:44.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:44.976516+0000) 2024-02-19T22:35:45.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:45.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:45.976802+0000) 2024-02-19T22:35:46.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:46.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:46.976955+0000) 2024-02-19T22:35:47.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:47.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:47.977152+0000) 2024-02-19T22:35:48.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:48.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:48.977304+0000) 2024-02-19T22:35:49.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:49.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:49.977513+0000) 2024-02-19T22:35:50.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:50.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:50.977681+0000) 2024-02-19T22:35:51.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:51.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:51.977842+0000) 2024-02-19T22:35:52.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:52.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:52.978036+0000) 2024-02-19T22:35:53.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:53.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:53.978235+0000) 2024-02-19T22:35:54.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:54.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:54.978372+0000) 2024-02-19T22:35:55.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:55.974+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:55.978506+0000) 2024-02-19T22:35:56.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:56.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:56.978806+0000) 2024-02-19T22:35:57.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:57.978+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:57.978947+0000) 2024-02-19T22:35:58.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:58.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:58.979084+0000) 2024-02-19T22:35:59.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:35:59.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:35:59.979229+0000) 2024-02-19T22:36:00.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:00.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:00.979495+0000) 2024-02-19T22:36:01.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:01.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:01.979706+0000) 2024-02-19T22:36:02.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:02.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:02.979847+0000) 2024-02-19T22:36:03.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:03.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:03.980005+0000) 2024-02-19T22:36:04.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:04.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:04.980164+0000) 2024-02-19T22:36:05.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:05.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:05.980370+0000) 2024-02-19T22:36:06.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:06.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:06.980525+0000) 2024-02-19T22:36:07.500 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:36:07.503 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:36:07.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:07.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:07.980679+0000) 2024-02-19T22:36:08.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:08.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:08.980869+0000) 2024-02-19T22:36:09.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:09.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:09.981039+0000) 2024-02-19T22:36:10.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:10.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:10.981214+0000) 2024-02-19T22:36:11.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:11.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:11.981360+0000) 2024-02-19T22:36:12.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:12.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:12.981492+0000) 2024-02-19T22:36:13.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:13.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:13.981637+0000) 2024-02-19T22:36:14.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:14.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:14.981789+0000) 2024-02-19T22:36:15.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:15.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:15.981920+0000) 2024-02-19T22:36:16.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:16.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:16.982104+0000) 2024-02-19T22:36:17.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:17.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:17.982246+0000) 2024-02-19T22:36:18.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:18.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:18.982486+0000) 2024-02-19T22:36:19.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:19.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:19.982669+0000) 2024-02-19T22:36:20.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:20.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:20.982802+0000) 2024-02-19T22:36:21.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:21.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:21.982931+0000) 2024-02-19T22:36:22.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:22.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:22.983077+0000) 2024-02-19T22:36:23.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:23.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:23.983220+0000) 2024-02-19T22:36:24.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:24.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:24.983376+0000) 2024-02-19T22:36:25.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:25.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:25.983517+0000) 2024-02-19T22:36:26.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:26.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:26.983695+0000) 2024-02-19T22:36:27.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:27.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:27.983860+0000) 2024-02-19T22:36:28.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:28.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:28.984043+0000) 2024-02-19T22:36:29.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:29.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:29.984208+0000) 2024-02-19T22:36:30.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:30.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:30.984364+0000) 2024-02-19T22:36:31.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:31.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:31.984509+0000) 2024-02-19T22:36:32.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:32.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:32.984670+0000) 2024-02-19T22:36:33.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:33.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:33.984820+0000) 2024-02-19T22:36:34.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:34.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:34.984945+0000) 2024-02-19T22:36:35.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:35.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:35.985070+0000) 2024-02-19T22:36:36.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:36.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:36.985197+0000) 2024-02-19T22:36:37.615 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:36:37.619 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:36:37.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:37.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:37.985361+0000) 2024-02-19T22:36:38.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:38.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:38.985558+0000) 2024-02-19T22:36:39.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:39.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:39.985800+0000) 2024-02-19T22:36:40.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:40.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:40.986003+0000) 2024-02-19T22:36:41.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:41.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:41.986134+0000) 2024-02-19T22:36:42.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:42.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:42.986255+0000) 2024-02-19T22:36:43.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:43.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:43.986431+0000) 2024-02-19T22:36:44.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:44.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:44.986576+0000) 2024-02-19T22:36:45.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:45.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:45.986757+0000) 2024-02-19T22:36:46.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:46.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:46.986951+0000) 2024-02-19T22:36:47.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:47.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:47.987098+0000) 2024-02-19T22:36:48.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:48.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:48.987244+0000) 2024-02-19T22:36:49.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:49.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:49.987491+0000) 2024-02-19T22:36:50.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:50.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:50.987659+0000) 2024-02-19T22:36:51.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:51.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:51.987880+0000) 2024-02-19T22:36:52.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:52.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:52.988111+0000) 2024-02-19T22:36:53.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:53.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:53.988320+0000) 2024-02-19T22:36:54.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:54.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:54.988518+0000) 2024-02-19T22:36:55.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:55.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:55.988703+0000) 2024-02-19T22:36:56.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:56.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:56.988882+0000) 2024-02-19T22:36:57.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:57.985+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:57.989063+0000) 2024-02-19T22:36:58.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:58.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:58.989276+0000) 2024-02-19T22:36:59.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:36:59.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:36:59.989459+0000) 2024-02-19T22:37:00.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:00.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:00.989596+0000) 2024-02-19T22:37:01.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:01.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:01.989806+0000) 2024-02-19T22:37:02.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:02.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:02.990026+0000) 2024-02-19T22:37:03.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:03.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:03.990178+0000) 2024-02-19T22:37:04.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:04.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:04.990315+0000) 2024-02-19T22:37:05.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:05.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:05.990488+0000) 2024-02-19T22:37:06.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:06.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:06.990758+0000) 2024-02-19T22:37:07.677 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:37:07.680 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:37:07.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:07.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:07.990904+0000) 2024-02-19T22:37:08.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:08.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:08.991115+0000) 2024-02-19T22:37:09.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:09.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:09.991340+0000) 2024-02-19T22:37:10.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:10.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:10.991477+0000) 2024-02-19T22:37:11.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:11.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:11.991710+0000) 2024-02-19T22:37:12.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:12.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:12.991952+0000) 2024-02-19T22:37:13.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:13.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:13.992180+0000) 2024-02-19T22:37:14.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:14.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:14.992329+0000) 2024-02-19T22:37:15.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:15.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:15.992462+0000) 2024-02-19T22:37:16.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:16.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:16.992683+0000) 2024-02-19T22:37:17.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:17.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:17.992817+0000) 2024-02-19T22:37:18.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:18.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:18.992976+0000) 2024-02-19T22:37:19.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:19.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:19.993199+0000) 2024-02-19T22:37:20.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:20.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:20.993387+0000) 2024-02-19T22:37:21.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:21.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:21.993576+0000) 2024-02-19T22:37:22.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:22.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:22.993814+0000) 2024-02-19T22:37:23.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:23.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:23.994037+0000) 2024-02-19T22:37:24.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:24.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:24.994275+0000) 2024-02-19T22:37:25.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:25.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:25.994452+0000) 2024-02-19T22:37:26.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:26.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:26.994627+0000) 2024-02-19T22:37:27.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:27.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:27.994801+0000) 2024-02-19T22:37:28.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:28.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:28.994968+0000) 2024-02-19T22:37:29.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:29.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:29.995097+0000) 2024-02-19T22:37:30.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:30.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:30.995234+0000) 2024-02-19T22:37:31.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:31.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:31.995393+0000) 2024-02-19T22:37:32.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:32.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:32.995564+0000) 2024-02-19T22:37:33.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:33.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:33.995747+0000) 2024-02-19T22:37:34.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:34.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:34.995977+0000) 2024-02-19T22:37:35.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:35.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:35.996206+0000) 2024-02-19T22:37:36.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:36.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:36.996364+0000) 2024-02-19T22:37:37.824 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:37:37.827 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:37:37.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:37.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:37.996501+0000) 2024-02-19T22:37:38.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:38.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:38.996642+0000) 2024-02-19T22:37:39.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:39.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:39.996879+0000) 2024-02-19T22:37:40.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:40.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:40.997080+0000) 2024-02-19T22:37:41.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:41.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:41.997288+0000) 2024-02-19T22:37:42.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:42.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:42.997441+0000) 2024-02-19T22:37:43.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:43.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:43.997674+0000) 2024-02-19T22:37:44.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:44.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:44.997829+0000) 2024-02-19T22:37:45.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:45.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:45.997962+0000) 2024-02-19T22:37:46.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:46.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:46.998183+0000) 2024-02-19T22:37:47.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:47.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:47.998349+0000) 2024-02-19T22:37:49.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:48.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:48.998538+0000) 2024-02-19T22:37:50.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:49.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:49.998737+0000) 2024-02-19T22:37:51.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:50.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:50.998894+0000) 2024-02-19T22:37:52.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:51.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:51.999117+0000) 2024-02-19T22:37:53.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:52.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:52.999281+0000) 2024-02-19T22:37:54.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:53.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:53.999404+0000) 2024-02-19T22:37:55.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:54.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:54.999568+0000) 2024-02-19T22:37:56.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:55.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:55.999853+0000) 2024-02-19T22:37:57.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:56.996+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:57.000079+0000) 2024-02-19T22:37:58.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:58.000+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:58.000230+0000) 2024-02-19T22:37:59.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:58.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:37:59.000364+0000) 2024-02-19T22:38:00.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:37:59.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:00.000575+0000) 2024-02-19T22:38:01.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:00.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:01.000798+0000) 2024-02-19T22:38:02.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:01.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:02.001042+0000) 2024-02-19T22:38:03.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:02.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:03.001228+0000) 2024-02-19T22:38:04.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:03.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:04.001473+0000) 2024-02-19T22:38:05.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:04.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:05.001674+0000) 2024-02-19T22:38:06.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:05.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:06.001922+0000) 2024-02-19T22:38:07.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:06.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:07.002147+0000) 2024-02-19T22:38:07.955 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:38:07.960 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:38:08.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:07.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:08.002319+0000) 2024-02-19T22:38:09.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:08.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:09.002551+0000) 2024-02-19T22:38:10.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:09.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:10.002756+0000) 2024-02-19T22:38:11.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:10.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:11.002880+0000) 2024-02-19T22:38:12.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:11.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:12.003009+0000) 2024-02-19T22:38:13.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:12.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:13.003132+0000) 2024-02-19T22:38:14.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:13.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:14.003319+0000) 2024-02-19T22:38:15.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:14.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:15.003560+0000) 2024-02-19T22:38:16.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:15.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:16.003795+0000) 2024-02-19T22:38:17.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:16.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:17.003951+0000) 2024-02-19T22:38:18.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:17.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:18.004078+0000) 2024-02-19T22:38:19.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:18.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:19.004204+0000) 2024-02-19T22:38:20.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:19.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:20.004349+0000) 2024-02-19T22:38:21.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:20.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:21.004474+0000) 2024-02-19T22:38:22.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:21.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:22.004595+0000) 2024-02-19T22:38:23.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:22.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:23.004723+0000) 2024-02-19T22:38:24.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:23.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:24.004929+0000) 2024-02-19T22:38:25.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:24.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:25.005067+0000) 2024-02-19T22:38:26.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:25.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:26.005193+0000) 2024-02-19T22:38:27.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:26.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:27.005320+0000) 2024-02-19T22:38:28.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:27.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:28.005446+0000) 2024-02-19T22:38:29.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:28.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:29.005569+0000) 2024-02-19T22:38:30.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:29.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:30.005705+0000) 2024-02-19T22:38:31.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:30.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:31.005830+0000) 2024-02-19T22:38:32.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:31.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:32.005969+0000) 2024-02-19T22:38:33.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:32.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:33.006094+0000) 2024-02-19T22:38:34.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:33.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:34.006215+0000) 2024-02-19T22:38:35.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:34.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:35.006340+0000) 2024-02-19T22:38:36.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:35.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:36.006452+0000) 2024-02-19T22:38:37.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:37.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:37.006576+0000) 2024-02-19T22:38:38.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:38.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:38.006735+0000) 2024-02-19T22:38:38.068 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:38:38.072 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:38:39.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:39.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:39.006891+0000) 2024-02-19T22:38:40.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:40.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:40.007035+0000) 2024-02-19T22:38:41.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:41.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:41.007167+0000) 2024-02-19T22:38:42.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:42.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:42.007363+0000) 2024-02-19T22:38:43.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:43.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:43.007506+0000) 2024-02-19T22:38:44.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:44.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:44.007677+0000) 2024-02-19T22:38:45.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:45.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:45.007899+0000) 2024-02-19T22:38:46.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:46.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:46.008056+0000) 2024-02-19T22:38:47.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:47.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:47.008229+0000) 2024-02-19T22:38:48.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:48.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:48.008503+0000) 2024-02-19T22:38:49.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:49.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:49.008724+0000) 2024-02-19T22:38:50.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:50.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:50.008893+0000) 2024-02-19T22:38:51.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:51.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:51.009051+0000) 2024-02-19T22:38:52.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:52.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:52.009289+0000) 2024-02-19T22:38:53.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:53.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:53.009570+0000) 2024-02-19T22:38:54.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:54.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:54.009808+0000) 2024-02-19T22:38:55.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:55.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:55.010010+0000) 2024-02-19T22:38:56.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:56.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:56.010177+0000) 2024-02-19T22:38:57.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:57.007+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:57.010356+0000) 2024-02-19T22:38:58.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:58.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:58.010511+0000) 2024-02-19T22:38:59.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:38:59.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:38:59.010694+0000) 2024-02-19T22:39:00.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:00.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:00.010961+0000) 2024-02-19T22:39:01.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:01.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:01.011147+0000) 2024-02-19T22:39:02.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:02.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:02.011310+0000) 2024-02-19T22:39:03.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:03.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:03.011455+0000) 2024-02-19T22:39:04.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:04.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:04.011577+0000) 2024-02-19T22:39:05.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:05.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:05.011776+0000) 2024-02-19T22:39:06.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:06.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:06.011973+0000) 2024-02-19T22:39:07.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:07.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:07.012182+0000) 2024-02-19T22:39:08.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:08.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:08.012327+0000) 2024-02-19T22:39:08.150 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:39:08.153 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:39:09.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:09.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:09.012525+0000) 2024-02-19T22:39:10.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:10.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:10.012795+0000) 2024-02-19T22:39:11.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:11.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:11.013065+0000) 2024-02-19T22:39:12.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:12.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:12.013289+0000) 2024-02-19T22:39:13.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:13.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:13.013448+0000) 2024-02-19T22:39:14.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:14.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:14.013675+0000) 2024-02-19T22:39:15.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:15.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:15.013855+0000) 2024-02-19T22:39:16.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:16.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:16.014022+0000) 2024-02-19T22:39:17.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:17.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:17.014181+0000) 2024-02-19T22:39:18.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:18.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:18.014384+0000) 2024-02-19T22:39:19.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:19.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:19.014568+0000) 2024-02-19T22:39:20.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:20.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:20.014770+0000) 2024-02-19T22:39:21.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:21.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:21.014932+0000) 2024-02-19T22:39:22.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:22.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:22.015113+0000) 2024-02-19T22:39:23.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:23.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:23.015252+0000) 2024-02-19T22:39:24.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:24.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:24.015422+0000) 2024-02-19T22:39:25.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:25.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:25.015548+0000) 2024-02-19T22:39:26.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:26.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:26.015708+0000) 2024-02-19T22:39:27.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:27.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:27.015937+0000) 2024-02-19T22:39:28.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:28.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:28.016111+0000) 2024-02-19T22:39:29.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:29.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:29.016270+0000) 2024-02-19T22:39:30.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:30.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:30.016438+0000) 2024-02-19T22:39:31.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:31.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:31.016590+0000) 2024-02-19T22:39:32.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:32.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:32.016824+0000) 2024-02-19T22:39:33.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:33.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:33.016995+0000) 2024-02-19T22:39:34.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:34.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:34.017163+0000) 2024-02-19T22:39:35.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:35.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:35.017384+0000) 2024-02-19T22:39:36.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:36.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:36.017541+0000) 2024-02-19T22:39:37.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:37.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:37.017767+0000) 2024-02-19T22:39:38.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:38.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:38.017959+0000) 2024-02-19T22:39:38.207 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:39:38.210 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:39:39.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:39.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:39.018129+0000) 2024-02-19T22:39:40.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:40.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:40.018297+0000) 2024-02-19T22:39:41.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:41.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:41.018460+0000) 2024-02-19T22:39:42.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:42.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:42.018640+0000) 2024-02-19T22:39:43.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:43.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:43.018790+0000) 2024-02-19T22:39:44.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:44.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:44.018965+0000) 2024-02-19T22:39:45.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:45.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:45.019095+0000) 2024-02-19T22:39:46.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:46.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:46.019227+0000) 2024-02-19T22:39:47.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:47.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:47.019384+0000) 2024-02-19T22:39:48.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:48.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:48.019542+0000) 2024-02-19T22:39:49.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:49.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:49.019717+0000) 2024-02-19T22:39:50.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:50.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:50.019864+0000) 2024-02-19T22:39:51.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:51.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:51.019983+0000) 2024-02-19T22:39:52.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:52.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:52.020151+0000) 2024-02-19T22:39:53.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:53.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:53.020300+0000) 2024-02-19T22:39:54.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:54.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:54.020538+0000) 2024-02-19T22:39:55.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:55.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:55.020681+0000) 2024-02-19T22:39:56.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:56.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:56.020817+0000) 2024-02-19T22:39:57.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:57.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:57.021087+0000) 2024-02-19T22:39:58.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:58.018+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:58.021247+0000) 2024-02-19T22:39:59.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:39:59.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:39:59.021465+0000) 2024-02-19T22:40:00.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:00.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:00.021601+0000) 2024-02-19T22:40:01.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:01.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:01.021753+0000) 2024-02-19T22:40:02.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:02.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:02.021990+0000) 2024-02-19T22:40:03.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:03.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:03.022148+0000) 2024-02-19T22:40:04.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:04.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:04.022281+0000) 2024-02-19T22:40:05.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:05.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:05.022490+0000) 2024-02-19T22:40:06.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:06.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:06.022679+0000) 2024-02-19T22:40:07.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:07.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:07.022891+0000) 2024-02-19T22:40:08.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:08.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:08.023014+0000) 2024-02-19T22:40:08.326 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:40:08.329 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:40:09.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:09.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:09.023190+0000) 2024-02-19T22:40:10.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:10.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:10.023425+0000) 2024-02-19T22:40:11.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:11.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:11.023608+0000) 2024-02-19T22:40:12.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:12.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:12.023795+0000) 2024-02-19T22:40:13.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:13.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:13.023997+0000) 2024-02-19T22:40:14.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:14.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:14.024122+0000) 2024-02-19T22:40:15.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:15.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:15.024246+0000) 2024-02-19T22:40:16.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:16.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:16.024473+0000) 2024-02-19T22:40:17.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:17.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:17.024709+0000) 2024-02-19T22:40:18.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:18.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:18.024939+0000) 2024-02-19T22:40:19.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:19.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:19.025097+0000) 2024-02-19T22:40:20.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:20.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:20.025316+0000) 2024-02-19T22:40:21.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:21.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:21.025500+0000) 2024-02-19T22:40:22.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:22.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:22.025650+0000) 2024-02-19T22:40:23.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:23.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:23.025790+0000) 2024-02-19T22:40:24.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:24.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:24.025928+0000) 2024-02-19T22:40:25.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:25.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:25.026142+0000) 2024-02-19T22:40:26.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:26.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:26.026323+0000) 2024-02-19T22:40:27.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:27.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:27.026545+0000) 2024-02-19T22:40:28.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:28.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:28.026860+0000) 2024-02-19T22:40:29.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:29.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:29.027056+0000) 2024-02-19T22:40:30.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:30.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:30.027291+0000) 2024-02-19T22:40:31.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:31.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:31.027531+0000) 2024-02-19T22:40:32.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:32.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:32.027675+0000) 2024-02-19T22:40:33.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:33.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:33.027820+0000) 2024-02-19T22:40:34.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:34.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:34.028055+0000) 2024-02-19T22:40:35.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:35.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:35.028287+0000) 2024-02-19T22:40:36.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:36.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:36.028444+0000) 2024-02-19T22:40:37.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:37.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:37.028660+0000) 2024-02-19T22:40:38.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:38.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:38.028863+0000) 2024-02-19T22:40:38.466 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:40:38.469 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:40:39.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:39.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:39.029077+0000) 2024-02-19T22:40:40.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:40.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:40.029224+0000) 2024-02-19T22:40:41.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:41.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:41.029386+0000) 2024-02-19T22:40:42.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:42.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:42.029623+0000) 2024-02-19T22:40:43.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:43.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:43.029845+0000) 2024-02-19T22:40:44.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:44.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:44.030005+0000) 2024-02-19T22:40:45.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:45.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:45.030231+0000) 2024-02-19T22:40:46.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:46.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:46.030408+0000) 2024-02-19T22:40:47.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:47.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:47.030571+0000) 2024-02-19T22:40:48.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:48.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:48.030750+0000) 2024-02-19T22:40:49.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:49.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:49.030978+0000) 2024-02-19T22:40:50.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:50.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:50.031136+0000) 2024-02-19T22:40:51.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:51.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:51.031260+0000) 2024-02-19T22:40:52.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:52.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:52.031387+0000) 2024-02-19T22:40:53.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:53.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:53.031511+0000) 2024-02-19T22:40:54.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:54.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:54.031637+0000) 2024-02-19T22:40:55.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:55.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:55.031765+0000) 2024-02-19T22:40:56.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:56.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:56.031894+0000) 2024-02-19T22:40:57.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:57.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:57.032030+0000) 2024-02-19T22:40:58.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:58.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:58.032236+0000) 2024-02-19T22:40:59.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:40:59.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:40:59.032450+0000) 2024-02-19T22:41:00.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:00.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:00.032617+0000) 2024-02-19T22:41:01.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:01.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:01.032815+0000) 2024-02-19T22:41:02.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:02.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:02.033046+0000) 2024-02-19T22:41:03.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:03.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:03.033227+0000) 2024-02-19T22:41:04.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:04.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:04.033457+0000) 2024-02-19T22:41:05.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:05.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:05.033710+0000) 2024-02-19T22:41:06.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:06.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:06.033872+0000) 2024-02-19T22:41:07.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:07.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:07.034031+0000) 2024-02-19T22:41:08.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:08.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:08.034275+0000) 2024-02-19T22:41:08.595 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:41:08.601 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:41:09.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:09.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:09.034494+0000) 2024-02-19T22:41:10.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:10.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:10.034765+0000) 2024-02-19T22:41:11.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:11.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:11.034932+0000) 2024-02-19T22:41:12.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:12.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:12.035061+0000) 2024-02-19T22:41:13.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:13.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:13.035275+0000) 2024-02-19T22:41:14.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:14.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:14.035497+0000) 2024-02-19T22:41:15.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:15.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:15.035637+0000) 2024-02-19T22:41:16.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:16.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:16.035762+0000) 2024-02-19T22:41:17.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:17.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:17.035989+0000) 2024-02-19T22:41:18.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:18.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:18.036174+0000) 2024-02-19T22:41:19.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:19.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:19.036414+0000) 2024-02-19T22:41:20.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:20.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:20.036582+0000) 2024-02-19T22:41:21.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:21.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:21.036827+0000) 2024-02-19T22:41:22.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:22.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:22.037087+0000) 2024-02-19T22:41:23.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:23.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:23.037273+0000) 2024-02-19T22:41:24.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:24.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:24.037480+0000) 2024-02-19T22:41:25.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:25.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:25.037635+0000) 2024-02-19T22:41:26.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:26.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:26.037884+0000) 2024-02-19T22:41:27.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:27.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:27.038115+0000) 2024-02-19T22:41:28.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:28.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:28.038385+0000) 2024-02-19T22:41:29.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:29.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:29.038530+0000) 2024-02-19T22:41:30.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:30.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:30.038742+0000) 2024-02-19T22:41:31.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:31.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:31.038973+0000) 2024-02-19T22:41:32.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:32.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:32.039145+0000) 2024-02-19T22:41:33.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:33.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:33.039296+0000) 2024-02-19T22:41:34.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:34.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:34.039457+0000) 2024-02-19T22:41:35.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:35.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:35.039617+0000) 2024-02-19T22:41:36.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:36.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:36.039849+0000) 2024-02-19T22:41:37.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:37.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:37.040057+0000) 2024-02-19T22:41:38.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:38.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:38.040230+0000) 2024-02-19T22:41:38.719 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:41:38.723 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:41:39.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:39.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:39.040463+0000) 2024-02-19T22:41:40.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:40.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:40.040698+0000) 2024-02-19T22:41:41.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:41.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:41.040830+0000) 2024-02-19T22:41:42.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:42.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:42.041002+0000) 2024-02-19T22:41:43.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:43.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:43.041147+0000) 2024-02-19T22:41:44.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:44.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:44.041371+0000) 2024-02-19T22:41:45.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:45.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:45.041498+0000) 2024-02-19T22:41:46.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:46.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:46.041727+0000) 2024-02-19T22:41:47.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:47.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:47.041887+0000) 2024-02-19T22:41:48.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:48.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:48.042047+0000) 2024-02-19T22:41:49.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:49.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:49.042224+0000) 2024-02-19T22:41:50.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:50.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:50.042396+0000) 2024-02-19T22:41:51.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:51.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:51.042628+0000) 2024-02-19T22:41:52.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:52.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:52.042818+0000) 2024-02-19T22:41:53.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:53.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:53.042973+0000) 2024-02-19T22:41:54.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:54.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:54.043153+0000) 2024-02-19T22:41:55.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:55.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:55.043388+0000) 2024-02-19T22:41:56.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:56.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:56.043619+0000) 2024-02-19T22:41:57.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:57.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:57.043798+0000) 2024-02-19T22:41:58.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:58.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:58.044041+0000) 2024-02-19T22:41:59.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:41:59.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:41:59.044251+0000) 2024-02-19T22:42:00.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:00.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:00.044512+0000) 2024-02-19T22:42:01.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:01.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:01.044763+0000) 2024-02-19T22:42:02.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:02.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:02.045008+0000) 2024-02-19T22:42:03.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:03.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:03.045249+0000) 2024-02-19T22:42:04.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:04.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:04.045440+0000) 2024-02-19T22:42:05.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:05.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:05.045655+0000) 2024-02-19T22:42:06.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:06.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:06.045832+0000) 2024-02-19T22:42:07.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:07.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:07.046049+0000) 2024-02-19T22:42:08.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:08.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:08.046271+0000) 2024-02-19T22:42:08.848 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:42:08.851 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:42:09.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:09.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:09.046462+0000) 2024-02-19T22:42:10.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:10.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:10.046694+0000) 2024-02-19T22:42:11.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:11.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:11.046928+0000) 2024-02-19T22:42:12.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:12.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:12.047071+0000) 2024-02-19T22:42:13.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:13.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:13.047295+0000) 2024-02-19T22:42:14.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:14.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:14.047529+0000) 2024-02-19T22:42:15.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:15.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:15.047678+0000) 2024-02-19T22:42:16.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:16.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:16.047911+0000) 2024-02-19T22:42:17.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:17.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:17.048121+0000) 2024-02-19T22:42:18.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:18.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:18.048344+0000) 2024-02-19T22:42:19.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:19.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:19.048503+0000) 2024-02-19T22:42:20.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:20.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:20.048733+0000) 2024-02-19T22:42:21.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:21.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:21.048937+0000) 2024-02-19T22:42:22.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:22.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:22.049074+0000) 2024-02-19T22:42:23.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:23.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:23.049212+0000) 2024-02-19T22:42:24.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:24.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:24.049374+0000) 2024-02-19T22:42:25.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:25.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:25.049564+0000) 2024-02-19T22:42:26.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:26.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:26.049728+0000) 2024-02-19T22:42:27.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:27.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:27.049868+0000) 2024-02-19T22:42:28.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:28.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:28.050083+0000) 2024-02-19T22:42:29.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:29.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:29.050253+0000) 2024-02-19T22:42:30.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:30.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:30.050482+0000) 2024-02-19T22:42:31.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:31.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:31.050646+0000) 2024-02-19T22:42:32.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:32.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:32.050795+0000) 2024-02-19T22:42:33.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:33.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:33.050961+0000) 2024-02-19T22:42:34.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:34.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:34.051087+0000) 2024-02-19T22:42:35.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:35.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:35.051223+0000) 2024-02-19T22:42:36.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:36.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:36.051377+0000) 2024-02-19T22:42:37.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:37.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:37.051612+0000) 2024-02-19T22:42:38.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:38.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:38.051870+0000) 2024-02-19T22:42:38.993 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:42:38.997 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:42:39.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:39.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:39.052105+0000) 2024-02-19T22:42:40.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:40.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:40.052307+0000) 2024-02-19T22:42:41.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:41.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:41.052501+0000) 2024-02-19T22:42:42.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:42.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:42.052675+0000) 2024-02-19T22:42:43.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:43.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:43.052923+0000) 2024-02-19T22:42:44.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:44.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:44.053060+0000) 2024-02-19T22:42:45.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:45.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:45.053233+0000) 2024-02-19T22:42:46.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:46.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:46.053433+0000) 2024-02-19T22:42:47.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:47.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:47.053570+0000) 2024-02-19T22:42:48.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:48.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:48.053710+0000) 2024-02-19T22:42:49.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:49.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:49.053834+0000) 2024-02-19T22:42:50.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:50.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:50.053963+0000) 2024-02-19T22:42:51.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:51.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:51.054105+0000) 2024-02-19T22:42:52.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:52.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:52.054219+0000) 2024-02-19T22:42:53.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:53.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:53.054365+0000) 2024-02-19T22:42:54.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:54.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:54.054507+0000) 2024-02-19T22:42:55.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:55.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:55.054762+0000) 2024-02-19T22:42:56.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:56.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:56.054933+0000) 2024-02-19T22:42:57.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:57.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:57.055084+0000) 2024-02-19T22:42:58.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:58.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:58.055248+0000) 2024-02-19T22:42:59.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:42:59.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:42:59.055387+0000) 2024-02-19T22:43:00.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:00.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:00.055562+0000) 2024-02-19T22:43:01.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:01.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:01.055733+0000) 2024-02-19T22:43:02.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:02.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:02.055940+0000) 2024-02-19T22:43:03.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:03.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:03.056088+0000) 2024-02-19T22:43:04.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:04.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:04.056304+0000) 2024-02-19T22:43:05.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:05.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:05.056525+0000) 2024-02-19T22:43:06.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:06.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:06.056775+0000) 2024-02-19T22:43:07.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:07.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:07.057031+0000) 2024-02-19T22:43:08.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:08.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:08.057283+0000) 2024-02-19T22:43:09.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:09.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:09.057410+0000) 2024-02-19T22:43:09.141 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:43:09.145 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:43:10.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:10.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:10.057659+0000) 2024-02-19T22:43:11.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:11.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:11.057787+0000) 2024-02-19T22:43:12.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:12.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:12.057967+0000) 2024-02-19T22:43:13.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:13.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:13.058137+0000) 2024-02-19T22:43:14.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:14.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:14.058450+0000) 2024-02-19T22:43:15.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:15.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:15.058693+0000) 2024-02-19T22:43:16.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:16.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:16.058883+0000) 2024-02-19T22:43:17.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:17.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:17.059069+0000) 2024-02-19T22:43:18.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:18.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:18.059209+0000) 2024-02-19T22:43:19.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:19.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:19.059383+0000) 2024-02-19T22:43:20.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:20.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:20.059607+0000) 2024-02-19T22:43:21.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:21.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:21.059874+0000) 2024-02-19T22:43:22.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:22.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:22.060070+0000) 2024-02-19T22:43:23.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:23.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:23.060293+0000) 2024-02-19T22:43:24.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:24.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:24.060527+0000) 2024-02-19T22:43:25.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:25.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:25.060774+0000) 2024-02-19T22:43:26.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:26.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:26.060952+0000) 2024-02-19T22:43:27.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:27.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:27.061164+0000) 2024-02-19T22:43:28.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:28.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:28.061417+0000) 2024-02-19T22:43:29.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:29.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:29.061621+0000) 2024-02-19T22:43:30.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:30.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:30.061851+0000) 2024-02-19T22:43:31.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:31.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:31.062080+0000) 2024-02-19T22:43:32.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:32.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:32.062352+0000) 2024-02-19T22:43:33.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:33.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:33.062553+0000) 2024-02-19T22:43:34.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:34.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:34.062832+0000) 2024-02-19T22:43:35.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:35.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:35.063022+0000) 2024-02-19T22:43:36.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:36.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:36.063167+0000) 2024-02-19T22:43:37.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:37.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:37.063368+0000) 2024-02-19T22:43:38.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:38.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:38.063621+0000) 2024-02-19T22:43:39.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:39.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:39.063809+0000) 2024-02-19T22:43:39.196 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:43:39.199 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:43:40.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:40.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:40.063998+0000) 2024-02-19T22:43:41.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:41.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:41.064233+0000) 2024-02-19T22:43:42.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:42.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:42.064468+0000) 2024-02-19T22:43:43.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:43.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:43.064634+0000) 2024-02-19T22:43:44.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:44.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:44.064822+0000) 2024-02-19T22:43:45.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:45.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:45.065018+0000) 2024-02-19T22:43:46.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:46.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:46.065235+0000) 2024-02-19T22:43:47.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:47.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:47.065470+0000) 2024-02-19T22:43:48.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:48.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:48.065701+0000) 2024-02-19T22:43:49.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:49.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:49.065935+0000) 2024-02-19T22:43:50.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:50.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:50.066120+0000) 2024-02-19T22:43:51.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:51.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:51.066277+0000) 2024-02-19T22:43:52.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:52.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:52.066496+0000) 2024-02-19T22:43:53.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:53.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:53.066670+0000) 2024-02-19T22:43:54.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:54.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:54.066887+0000) 2024-02-19T22:43:55.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:55.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:55.067029+0000) 2024-02-19T22:43:56.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:56.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:56.067216+0000) 2024-02-19T22:43:57.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:57.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:57.067426+0000) 2024-02-19T22:43:58.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:58.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:58.067632+0000) 2024-02-19T22:43:59.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:43:59.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:43:59.067873+0000) 2024-02-19T22:44:00.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:00.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:00.068074+0000) 2024-02-19T22:44:01.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:01.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:01.068305+0000) 2024-02-19T22:44:02.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:02.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:02.068517+0000) 2024-02-19T22:44:03.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:03.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:03.068719+0000) 2024-02-19T22:44:04.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:04.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:04.068986+0000) 2024-02-19T22:44:05.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:05.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:05.069221+0000) 2024-02-19T22:44:06.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:06.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:06.069393+0000) 2024-02-19T22:44:07.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:07.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:07.069605+0000) 2024-02-19T22:44:08.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:08.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:08.069904+0000) 2024-02-19T22:44:09.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:09.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:09.070141+0000) 2024-02-19T22:44:09.268 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:44:09.271 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:44:10.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:10.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:10.070387+0000) 2024-02-19T22:44:11.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:11.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:11.070555+0000) 2024-02-19T22:44:12.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:12.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:12.070827+0000) 2024-02-19T22:44:13.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:13.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:13.070969+0000) 2024-02-19T22:44:14.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:14.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:14.071143+0000) 2024-02-19T22:44:15.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:15.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:15.071337+0000) 2024-02-19T22:44:16.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:16.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:16.071531+0000) 2024-02-19T22:44:17.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:17.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:17.071783+0000) 2024-02-19T22:44:18.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:18.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:18.071963+0000) 2024-02-19T22:44:19.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:19.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:19.072162+0000) 2024-02-19T22:44:20.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:20.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:20.072403+0000) 2024-02-19T22:44:21.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:21.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:21.072623+0000) 2024-02-19T22:44:22.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:22.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:22.072856+0000) 2024-02-19T22:44:23.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:23.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:23.073049+0000) 2024-02-19T22:44:24.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:24.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:24.073261+0000) 2024-02-19T22:44:25.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:25.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:25.073456+0000) 2024-02-19T22:44:26.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:26.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:26.073681+0000) 2024-02-19T22:44:27.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:27.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:27.073865+0000) 2024-02-19T22:44:28.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:28.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:28.074102+0000) 2024-02-19T22:44:29.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:29.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:29.074322+0000) 2024-02-19T22:44:30.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:30.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:30.074613+0000) 2024-02-19T22:44:31.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:31.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:31.074820+0000) 2024-02-19T22:44:32.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:32.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:32.074940+0000) 2024-02-19T22:44:33.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:33.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:33.075063+0000) 2024-02-19T22:44:34.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:34.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:34.075203+0000) 2024-02-19T22:44:35.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:35.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:35.075379+0000) 2024-02-19T22:44:36.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:36.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:36.075573+0000) 2024-02-19T22:44:37.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:37.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:37.075799+0000) 2024-02-19T22:44:38.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:38.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:38.075973+0000) 2024-02-19T22:44:39.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:39.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:39.076175+0000) 2024-02-19T22:44:39.398 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:44:39.401 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:44:40.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:40.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:40.076422+0000) 2024-02-19T22:44:41.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:41.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:41.076698+0000) 2024-02-19T22:44:42.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:42.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:42.076935+0000) 2024-02-19T22:44:43.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:43.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:43.077115+0000) 2024-02-19T22:44:44.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:44.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:44.077379+0000) 2024-02-19T22:44:45.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:45.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:45.077538+0000) 2024-02-19T22:44:46.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:46.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:46.077739+0000) 2024-02-19T22:44:47.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:47.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:47.077916+0000) 2024-02-19T22:44:48.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:48.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:48.078182+0000) 2024-02-19T22:44:49.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:49.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:49.078355+0000) 2024-02-19T22:44:50.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:50.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:50.078530+0000) 2024-02-19T22:44:51.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:51.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:51.078811+0000) 2024-02-19T22:44:52.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:52.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:52.079003+0000) 2024-02-19T22:44:53.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:53.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:53.079194+0000) 2024-02-19T22:44:54.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:54.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:54.079389+0000) 2024-02-19T22:44:55.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:55.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:55.079649+0000) 2024-02-19T22:44:56.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:56.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:56.079886+0000) 2024-02-19T22:44:57.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:57.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:57.080121+0000) 2024-02-19T22:44:58.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:58.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:58.080302+0000) 2024-02-19T22:44:59.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:44:59.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:44:59.080571+0000) 2024-02-19T22:45:00.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:00.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:00.080870+0000) 2024-02-19T22:45:01.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:01.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:01.081102+0000) 2024-02-19T22:45:02.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:02.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:02.081282+0000) 2024-02-19T22:45:03.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:03.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:03.081448+0000) 2024-02-19T22:45:04.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:04.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:04.081588+0000) 2024-02-19T22:45:05.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:05.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:05.081725+0000) 2024-02-19T22:45:06.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:06.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:06.081886+0000) 2024-02-19T22:45:07.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:07.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:07.082030+0000) 2024-02-19T22:45:08.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:08.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:08.082166+0000) 2024-02-19T22:45:09.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:09.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:09.082305+0000) 2024-02-19T22:45:09.587 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:45:09.590 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:45:10.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:10.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:10.082464+0000) 2024-02-19T22:45:11.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:11.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:11.082668+0000) 2024-02-19T22:45:12.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:12.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:12.082895+0000) 2024-02-19T22:45:13.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:13.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:13.083073+0000) 2024-02-19T22:45:14.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:14.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:14.083267+0000) 2024-02-19T22:45:15.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:15.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:15.083422+0000) 2024-02-19T22:45:16.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:16.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:16.083607+0000) 2024-02-19T22:45:17.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:17.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:17.083722+0000) 2024-02-19T22:45:18.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:18.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:18.083972+0000) 2024-02-19T22:45:19.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:19.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:19.084192+0000) 2024-02-19T22:45:20.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:20.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:20.084390+0000) 2024-02-19T22:45:21.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:21.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:21.084655+0000) 2024-02-19T22:45:22.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:22.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:22.084912+0000) 2024-02-19T22:45:23.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:23.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:23.085164+0000) 2024-02-19T22:45:24.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:24.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:24.085382+0000) 2024-02-19T22:45:25.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:25.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:25.085577+0000) 2024-02-19T22:45:26.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:26.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:26.085799+0000) 2024-02-19T22:45:27.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:27.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:27.086060+0000) 2024-02-19T22:45:28.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:28.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:28.086275+0000) 2024-02-19T22:45:29.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:29.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:29.086502+0000) 2024-02-19T22:45:30.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:30.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:30.086803+0000) 2024-02-19T22:45:31.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:31.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:31.087017+0000) 2024-02-19T22:45:32.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:32.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:32.087296+0000) 2024-02-19T22:45:33.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:33.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:33.087514+0000) 2024-02-19T22:45:34.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:34.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:34.087748+0000) 2024-02-19T22:45:35.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:35.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:35.087918+0000) 2024-02-19T22:45:36.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:36.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:36.088038+0000) 2024-02-19T22:45:37.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:37.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:37.088227+0000) 2024-02-19T22:45:38.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:38.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:38.088459+0000) 2024-02-19T22:45:39.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:39.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:39.088692+0000) 2024-02-19T22:45:39.643 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:45:39.648 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:45:40.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:40.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:40.088877+0000) 2024-02-19T22:45:41.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:41.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:41.089124+0000) 2024-02-19T22:45:42.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:42.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:42.089288+0000) 2024-02-19T22:45:43.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:43.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:43.089452+0000) 2024-02-19T22:45:44.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:44.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:44.089702+0000) 2024-02-19T22:45:45.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:45.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:45.089939+0000) 2024-02-19T22:45:46.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:46.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:46.090163+0000) 2024-02-19T22:45:47.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:47.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:47.090411+0000) 2024-02-19T22:45:48.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:48.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:48.090603+0000) 2024-02-19T22:45:49.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:49.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:49.090825+0000) 2024-02-19T22:45:50.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:50.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:50.090998+0000) 2024-02-19T22:45:51.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:51.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:51.091220+0000) 2024-02-19T22:45:52.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:52.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:52.091471+0000) 2024-02-19T22:45:53.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:53.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:53.091723+0000) 2024-02-19T22:45:54.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:54.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:54.091900+0000) 2024-02-19T22:45:55.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:55.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:55.092118+0000) 2024-02-19T22:45:56.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:56.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:56.092362+0000) 2024-02-19T22:45:57.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:57.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:57.092575+0000) 2024-02-19T22:45:58.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:58.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:58.092804+0000) 2024-02-19T22:45:59.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:45:59.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:45:59.093040+0000) 2024-02-19T22:46:00.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:00.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:00.093284+0000) 2024-02-19T22:46:01.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:01.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:01.093518+0000) 2024-02-19T22:46:02.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:02.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:02.093693+0000) 2024-02-19T22:46:03.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:03.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:03.093919+0000) 2024-02-19T22:46:04.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:04.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:04.094054+0000) 2024-02-19T22:46:05.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:05.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:05.094236+0000) 2024-02-19T22:46:06.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:06.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:06.094416+0000) 2024-02-19T22:46:07.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:07.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:07.094629+0000) 2024-02-19T22:46:08.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:08.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:08.094809+0000) 2024-02-19T22:46:09.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:09.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:09.094982+0000) 2024-02-19T22:46:09.831 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:46:09.835 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:46:10.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:10.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:10.095187+0000) 2024-02-19T22:46:11.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:11.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:11.095321+0000) 2024-02-19T22:46:12.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:12.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:12.095575+0000) 2024-02-19T22:46:13.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:13.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:13.095758+0000) 2024-02-19T22:46:14.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:14.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:14.096000+0000) 2024-02-19T22:46:15.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:15.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:15.096262+0000) 2024-02-19T22:46:16.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:16.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:16.096506+0000) 2024-02-19T22:46:17.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:17.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:17.096774+0000) 2024-02-19T22:46:18.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:18.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:18.096974+0000) 2024-02-19T22:46:19.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:19.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:19.097190+0000) 2024-02-19T22:46:20.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:20.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:20.097440+0000) 2024-02-19T22:46:21.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:21.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:21.097679+0000) 2024-02-19T22:46:22.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:22.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:22.097945+0000) 2024-02-19T22:46:23.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:23.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:23.098177+0000) 2024-02-19T22:46:24.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:24.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:24.098438+0000) 2024-02-19T22:46:25.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:25.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:25.098626+0000) 2024-02-19T22:46:26.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:26.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:26.098833+0000) 2024-02-19T22:46:27.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:27.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:27.098994+0000) 2024-02-19T22:46:28.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:28.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:28.099219+0000) 2024-02-19T22:46:29.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:29.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:29.099459+0000) 2024-02-19T22:46:30.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:30.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:30.099719+0000) 2024-02-19T22:46:31.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:31.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:31.099918+0000) 2024-02-19T22:46:32.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:32.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:32.100112+0000) 2024-02-19T22:46:33.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:33.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:33.100332+0000) 2024-02-19T22:46:34.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:34.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:34.100510+0000) 2024-02-19T22:46:35.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:35.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:35.100686+0000) 2024-02-19T22:46:36.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:36.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:36.100933+0000) 2024-02-19T22:46:37.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:37.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:37.101207+0000) 2024-02-19T22:46:38.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:38.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:38.101392+0000) 2024-02-19T22:46:39.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:39.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:39.101615+0000) 2024-02-19T22:46:39.961 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:46:39.965 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:46:40.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:40.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:40.101864+0000) 2024-02-19T22:46:41.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:41.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:41.102077+0000) 2024-02-19T22:46:42.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:42.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:42.102323+0000) 2024-02-19T22:46:43.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:43.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:43.102561+0000) 2024-02-19T22:46:44.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:44.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:44.102764+0000) 2024-02-19T22:46:45.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:45.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:45.102901+0000) 2024-02-19T22:46:46.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:46.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:46.103043+0000) 2024-02-19T22:46:47.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:47.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:47.103315+0000) 2024-02-19T22:46:48.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:48.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:48.103589+0000) 2024-02-19T22:46:49.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:49.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:49.103771+0000) 2024-02-19T22:46:50.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:50.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:50.104013+0000) 2024-02-19T22:46:51.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:51.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:51.104223+0000) 2024-02-19T22:46:52.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:52.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:52.104451+0000) 2024-02-19T22:46:53.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:53.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:53.104707+0000) 2024-02-19T22:46:54.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:54.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:54.104860+0000) 2024-02-19T22:46:55.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:55.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:55.105085+0000) 2024-02-19T22:46:56.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:56.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:56.105306+0000) 2024-02-19T22:46:57.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:57.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:57.105564+0000) 2024-02-19T22:46:58.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:58.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:58.105758+0000) 2024-02-19T22:46:59.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:46:59.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:46:59.105960+0000) 2024-02-19T22:47:00.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:00.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:00.106203+0000) 2024-02-19T22:47:01.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:01.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:01.106429+0000) 2024-02-19T22:47:02.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:02.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:02.106606+0000) 2024-02-19T22:47:03.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:03.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:03.106888+0000) 2024-02-19T22:47:04.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:04.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:04.107044+0000) 2024-02-19T22:47:05.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:05.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:05.107255+0000) 2024-02-19T22:47:06.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:06.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:06.107495+0000) 2024-02-19T22:47:07.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:07.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:07.107752+0000) 2024-02-19T22:47:08.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:08.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:08.107993+0000) 2024-02-19T22:47:09.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:09.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:09.108220+0000) 2024-02-19T22:47:10.024 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:47:10.027 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:47:10.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:10.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:10.108452+0000) 2024-02-19T22:47:11.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:11.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:11.108645+0000) 2024-02-19T22:47:12.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:12.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:12.108854+0000) 2024-02-19T22:47:13.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:13.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:13.109108+0000) 2024-02-19T22:47:14.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:14.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:14.109362+0000) 2024-02-19T22:47:15.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:15.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:15.109598+0000) 2024-02-19T22:47:16.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:16.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:16.109824+0000) 2024-02-19T22:47:17.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:17.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:17.110058+0000) 2024-02-19T22:47:18.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:18.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:18.110201+0000) 2024-02-19T22:47:19.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:19.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:19.110410+0000) 2024-02-19T22:47:20.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:20.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:20.110602+0000) 2024-02-19T22:47:21.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:21.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:21.110820+0000) 2024-02-19T22:47:22.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:22.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:22.111060+0000) 2024-02-19T22:47:23.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:23.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:23.111244+0000) 2024-02-19T22:47:24.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:24.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:24.111410+0000) 2024-02-19T22:47:25.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:25.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:25.111663+0000) 2024-02-19T22:47:26.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:26.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:26.111879+0000) 2024-02-19T22:47:27.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:27.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:27.112097+0000) 2024-02-19T22:47:28.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:28.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:28.112280+0000) 2024-02-19T22:47:29.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:29.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:29.112486+0000) 2024-02-19T22:47:30.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:30.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:30.112697+0000) 2024-02-19T22:47:31.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:31.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:31.112914+0000) 2024-02-19T22:47:32.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:32.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:32.113129+0000) 2024-02-19T22:47:33.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:33.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:33.113365+0000) 2024-02-19T22:47:34.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:34.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:34.113557+0000) 2024-02-19T22:47:35.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:35.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:35.113784+0000) 2024-02-19T22:47:36.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:36.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:36.113967+0000) 2024-02-19T22:47:37.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:37.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:37.114190+0000) 2024-02-19T22:47:38.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:38.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:38.114426+0000) 2024-02-19T22:47:39.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:39.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:39.114657+0000) 2024-02-19T22:47:40.100 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:47:40.103 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:47:40.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:40.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:40.114818+0000) 2024-02-19T22:47:41.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:41.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:41.115002+0000) 2024-02-19T22:47:42.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:42.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:42.115234+0000) 2024-02-19T22:47:43.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:43.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:43.115369+0000) 2024-02-19T22:47:44.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:44.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:44.115550+0000) 2024-02-19T22:47:45.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:45.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:45.115759+0000) 2024-02-19T22:47:46.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:46.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:46.115917+0000) 2024-02-19T22:47:47.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:47.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:47.116077+0000) 2024-02-19T22:47:48.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:48.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:48.116259+0000) 2024-02-19T22:47:49.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:49.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:49.116421+0000) 2024-02-19T22:47:50.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:50.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:50.116594+0000) 2024-02-19T22:47:51.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:51.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:51.116794+0000) 2024-02-19T22:47:52.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:52.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:52.117060+0000) 2024-02-19T22:47:53.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:53.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:53.117229+0000) 2024-02-19T22:47:54.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:54.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:54.117472+0000) 2024-02-19T22:47:55.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:55.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:55.117690+0000) 2024-02-19T22:47:56.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:56.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:56.117936+0000) 2024-02-19T22:47:57.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:57.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:57.118140+0000) 2024-02-19T22:47:58.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:58.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:58.118323+0000) 2024-02-19T22:47:59.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:47:59.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:47:59.118496+0000) 2024-02-19T22:48:00.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:00.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:00.118803+0000) 2024-02-19T22:48:01.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:01.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:01.119055+0000) 2024-02-19T22:48:02.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:02.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:02.119262+0000) 2024-02-19T22:48:03.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:03.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:03.119454+0000) 2024-02-19T22:48:04.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:04.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:04.119689+0000) 2024-02-19T22:48:05.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:05.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:05.119895+0000) 2024-02-19T22:48:06.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:06.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:06.120143+0000) 2024-02-19T22:48:07.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:07.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:07.120397+0000) 2024-02-19T22:48:08.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:08.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:08.120604+0000) 2024-02-19T22:48:09.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:09.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:09.120835+0000) 2024-02-19T22:48:10.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:10.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:10.121092+0000) 2024-02-19T22:48:10.159 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:48:10.163 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:48:11.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:11.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:11.121328+0000) 2024-02-19T22:48:12.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:12.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:12.121567+0000) 2024-02-19T22:48:13.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:13.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:13.121792+0000) 2024-02-19T22:48:14.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:14.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:14.121978+0000) 2024-02-19T22:48:15.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:15.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:15.122249+0000) 2024-02-19T22:48:16.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:16.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:16.122485+0000) 2024-02-19T22:48:17.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:17.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:17.122687+0000) 2024-02-19T22:48:18.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:18.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:18.123007+0000) 2024-02-19T22:48:19.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:19.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:19.123177+0000) 2024-02-19T22:48:20.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:20.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:20.123377+0000) 2024-02-19T22:48:21.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:21.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:21.123590+0000) 2024-02-19T22:48:22.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:22.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:22.123833+0000) 2024-02-19T22:48:23.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:23.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:23.124026+0000) 2024-02-19T22:48:24.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:24.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:24.124172+0000) 2024-02-19T22:48:25.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:25.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:25.124403+0000) 2024-02-19T22:48:26.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:26.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:26.124641+0000) 2024-02-19T22:48:27.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:27.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:27.124822+0000) 2024-02-19T22:48:28.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:28.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:28.125067+0000) 2024-02-19T22:48:29.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:29.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:29.125304+0000) 2024-02-19T22:48:30.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:30.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:30.125561+0000) 2024-02-19T22:48:31.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:31.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:31.125737+0000) 2024-02-19T22:48:32.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:32.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:32.125885+0000) 2024-02-19T22:48:33.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:33.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:33.126067+0000) 2024-02-19T22:48:34.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:34.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:34.126308+0000) 2024-02-19T22:48:35.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:35.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:35.126578+0000) 2024-02-19T22:48:36.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:36.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:36.126787+0000) 2024-02-19T22:48:37.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:37.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:37.127012+0000) 2024-02-19T22:48:38.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:38.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:38.127250+0000) 2024-02-19T22:48:39.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:39.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:39.127501+0000) 2024-02-19T22:48:40.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:40.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:40.127654+0000) 2024-02-19T22:48:40.281 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:48:40.285 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:48:41.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:41.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:41.127849+0000) 2024-02-19T22:48:42.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:42.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:42.128085+0000) 2024-02-19T22:48:43.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:43.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:43.128326+0000) 2024-02-19T22:48:44.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:44.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:44.128597+0000) 2024-02-19T22:48:45.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:45.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:45.128822+0000) 2024-02-19T22:48:46.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:46.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:46.129067+0000) 2024-02-19T22:48:47.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:47.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:47.129342+0000) 2024-02-19T22:48:48.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:48.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:48.129487+0000) 2024-02-19T22:48:49.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:49.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:49.129695+0000) 2024-02-19T22:48:50.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:50.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:50.129968+0000) 2024-02-19T22:48:51.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:51.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:51.130204+0000) 2024-02-19T22:48:52.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:52.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:52.130447+0000) 2024-02-19T22:48:53.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:53.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:53.130693+0000) 2024-02-19T22:48:54.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:54.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:54.130894+0000) 2024-02-19T22:48:55.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:55.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:55.131139+0000) 2024-02-19T22:48:56.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:56.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:56.131286+0000) 2024-02-19T22:48:57.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:57.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:57.131535+0000) 2024-02-19T22:48:58.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:58.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:58.131780+0000) 2024-02-19T22:48:59.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:48:59.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:48:59.132011+0000) 2024-02-19T22:49:00.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:00.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:00.132207+0000) 2024-02-19T22:49:01.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:01.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:01.132440+0000) 2024-02-19T22:49:02.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:02.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:02.132630+0000) 2024-02-19T22:49:03.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:03.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:03.132868+0000) 2024-02-19T22:49:04.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:04.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:04.133060+0000) 2024-02-19T22:49:05.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:05.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:05.133280+0000) 2024-02-19T22:49:06.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:06.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:06.133520+0000) 2024-02-19T22:49:07.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:07.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:07.133781+0000) 2024-02-19T22:49:08.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:08.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:08.133990+0000) 2024-02-19T22:49:09.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:09.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:09.134232+0000) 2024-02-19T22:49:10.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:10.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:10.134447+0000) 2024-02-19T22:49:10.354 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:49:10.358 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:49:11.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:11.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:11.134690+0000) 2024-02-19T22:49:12.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:12.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:12.134879+0000) 2024-02-19T22:49:13.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:13.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:13.135142+0000) 2024-02-19T22:49:14.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:14.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:14.135398+0000) 2024-02-19T22:49:15.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:15.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:15.135702+0000) 2024-02-19T22:49:16.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:16.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:16.135981+0000) 2024-02-19T22:49:17.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:17.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:17.136253+0000) 2024-02-19T22:49:18.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:18.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:18.136471+0000) 2024-02-19T22:49:19.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:19.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:19.136751+0000) 2024-02-19T22:49:20.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:20.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:20.137004+0000) 2024-02-19T22:49:21.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:21.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:21.137241+0000) 2024-02-19T22:49:22.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:22.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:22.137475+0000) 2024-02-19T22:49:23.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:23.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:23.137729+0000) 2024-02-19T22:49:24.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:24.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:24.137897+0000) 2024-02-19T22:49:25.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:25.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:25.138128+0000) 2024-02-19T22:49:26.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:26.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:26.138389+0000) 2024-02-19T22:49:27.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:27.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:27.138644+0000) 2024-02-19T22:49:28.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:28.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:28.138791+0000) 2024-02-19T22:49:29.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:29.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:29.139054+0000) 2024-02-19T22:49:30.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:30.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:30.139245+0000) 2024-02-19T22:49:31.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:31.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:31.139495+0000) 2024-02-19T22:49:32.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:32.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:32.139696+0000) 2024-02-19T22:49:33.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:33.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:33.139874+0000) 2024-02-19T22:49:34.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:34.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:34.140148+0000) 2024-02-19T22:49:35.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:35.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:35.140348+0000) 2024-02-19T22:49:36.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:36.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:36.140567+0000) 2024-02-19T22:49:37.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:37.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:37.140747+0000) 2024-02-19T22:49:38.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:38.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:38.140927+0000) 2024-02-19T22:49:39.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:39.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:39.141115+0000) 2024-02-19T22:49:40.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:40.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:40.141351+0000) 2024-02-19T22:49:40.415 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:49:40.419 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:49:41.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:41.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:41.141524+0000) 2024-02-19T22:49:42.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:42.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:42.141771+0000) 2024-02-19T22:49:43.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:43.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:43.142012+0000) 2024-02-19T22:49:44.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:44.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:44.142149+0000) 2024-02-19T22:49:45.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:45.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:45.142394+0000) 2024-02-19T22:49:46.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:46.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:46.142584+0000) 2024-02-19T22:49:47.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:47.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:47.142864+0000) 2024-02-19T22:49:48.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:48.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:48.143119+0000) 2024-02-19T22:49:49.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:49.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:49.143376+0000) 2024-02-19T22:49:50.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:50.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:50.143614+0000) 2024-02-19T22:49:51.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:51.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:51.143792+0000) 2024-02-19T22:49:52.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:52.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:52.144022+0000) 2024-02-19T22:49:53.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:53.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:53.144286+0000) 2024-02-19T22:49:54.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:54.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:54.144525+0000) 2024-02-19T22:49:55.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:55.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:55.144657+0000) 2024-02-19T22:49:56.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:56.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:56.144821+0000) 2024-02-19T22:49:57.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:57.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:57.145010+0000) 2024-02-19T22:49:58.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:58.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:58.145245+0000) 2024-02-19T22:49:59.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:49:59.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:49:59.145460+0000) 2024-02-19T22:50:00.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:00.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:00.145741+0000) 2024-02-19T22:50:01.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:01.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:01.146003+0000) 2024-02-19T22:50:02.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:02.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:02.146258+0000) 2024-02-19T22:50:03.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:03.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:03.146450+0000) 2024-02-19T22:50:04.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:04.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:04.146687+0000) 2024-02-19T22:50:05.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:05.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:05.146956+0000) 2024-02-19T22:50:06.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:06.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:06.147148+0000) 2024-02-19T22:50:07.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:07.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:07.147383+0000) 2024-02-19T22:50:08.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:08.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:08.147611+0000) 2024-02-19T22:50:09.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:09.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:09.147873+0000) 2024-02-19T22:50:10.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:10.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:10.148133+0000) 2024-02-19T22:50:10.549 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:50:10.552 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:50:11.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:11.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:11.148360+0000) 2024-02-19T22:50:12.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:12.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:12.148587+0000) 2024-02-19T22:50:13.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:13.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:13.148839+0000) 2024-02-19T22:50:14.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:14.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:14.149043+0000) 2024-02-19T22:50:15.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:15.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:15.149211+0000) 2024-02-19T22:50:16.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:16.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:16.149451+0000) 2024-02-19T22:50:17.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:17.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:17.149666+0000) 2024-02-19T22:50:18.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:18.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:18.149921+0000) 2024-02-19T22:50:19.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:19.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:19.150141+0000) 2024-02-19T22:50:20.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:20.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:20.150437+0000) 2024-02-19T22:50:21.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:21.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:21.150645+0000) 2024-02-19T22:50:22.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:22.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:22.150905+0000) 2024-02-19T22:50:23.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:23.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:23.151191+0000) 2024-02-19T22:50:24.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:24.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:24.151421+0000) 2024-02-19T22:50:25.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:25.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:25.151586+0000) 2024-02-19T22:50:26.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:26.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:26.151773+0000) 2024-02-19T22:50:27.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:27.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:27.151970+0000) 2024-02-19T22:50:28.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:28.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:28.152241+0000) 2024-02-19T22:50:29.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:29.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:29.152473+0000) 2024-02-19T22:50:30.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:30.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:30.152758+0000) 2024-02-19T22:50:31.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:31.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:31.152947+0000) 2024-02-19T22:50:32.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:32.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:32.153234+0000) 2024-02-19T22:50:33.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:33.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:33.153481+0000) 2024-02-19T22:50:34.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:34.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:34.153700+0000) 2024-02-19T22:50:35.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:35.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:35.153929+0000) 2024-02-19T22:50:36.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:36.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:36.154118+0000) 2024-02-19T22:50:37.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:37.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:37.154366+0000) 2024-02-19T22:50:38.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:38.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:38.154564+0000) 2024-02-19T22:50:39.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:39.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:39.154833+0000) 2024-02-19T22:50:40.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:40.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:40.154975+0000) 2024-02-19T22:50:40.695 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:50:40.698 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:50:41.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:41.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:41.155169+0000) 2024-02-19T22:50:42.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:42.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:42.155473+0000) 2024-02-19T22:50:43.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:43.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:43.155647+0000) 2024-02-19T22:50:44.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:44.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:44.155904+0000) 2024-02-19T22:50:45.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:45.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:45.156118+0000) 2024-02-19T22:50:46.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:46.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:46.156300+0000) 2024-02-19T22:50:47.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:47.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:47.156572+0000) 2024-02-19T22:50:48.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:48.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:48.156752+0000) 2024-02-19T22:50:49.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:49.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:49.156995+0000) 2024-02-19T22:50:50.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:50.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:50.157237+0000) 2024-02-19T22:50:51.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:51.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:51.157423+0000) 2024-02-19T22:50:52.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:52.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:52.157679+0000) 2024-02-19T22:50:53.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:53.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:53.157936+0000) 2024-02-19T22:50:54.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:54.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:54.158137+0000) 2024-02-19T22:50:55.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:55.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:55.158343+0000) 2024-02-19T22:50:56.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:56.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:56.158587+0000) 2024-02-19T22:50:57.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:57.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:57.158854+0000) 2024-02-19T22:50:58.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:58.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:58.158986+0000) 2024-02-19T22:50:59.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:50:59.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:50:59.159176+0000) 2024-02-19T22:51:00.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:00.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:00.159429+0000) 2024-02-19T22:51:01.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:01.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:01.159691+0000) 2024-02-19T22:51:02.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:02.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:02.159880+0000) 2024-02-19T22:51:03.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:03.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:03.160094+0000) 2024-02-19T22:51:04.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:04.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:04.160305+0000) 2024-02-19T22:51:05.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:05.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:05.160473+0000) 2024-02-19T22:51:06.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:06.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:06.160673+0000) 2024-02-19T22:51:07.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:07.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:07.160903+0000) 2024-02-19T22:51:08.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:08.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:08.161061+0000) 2024-02-19T22:51:09.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:09.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:09.161235+0000) 2024-02-19T22:51:10.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:10.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:10.161411+0000) 2024-02-19T22:51:10.817 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:51:10.822 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:51:11.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:11.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:11.161643+0000) 2024-02-19T22:51:12.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:12.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:12.161813+0000) 2024-02-19T22:51:13.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:13.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:13.162004+0000) 2024-02-19T22:51:14.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:14.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:14.162171+0000) 2024-02-19T22:51:15.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:15.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:15.162401+0000) 2024-02-19T22:51:16.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:16.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:16.162527+0000) 2024-02-19T22:51:17.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:17.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:17.162694+0000) 2024-02-19T22:51:18.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:18.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:18.162844+0000) 2024-02-19T22:51:19.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:19.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:19.162972+0000) 2024-02-19T22:51:20.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:20.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:20.163108+0000) 2024-02-19T22:51:21.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:21.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:21.163232+0000) 2024-02-19T22:51:22.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:22.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:22.163373+0000) 2024-02-19T22:51:23.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:23.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:23.163499+0000) 2024-02-19T22:51:24.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:24.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:24.163626+0000) 2024-02-19T22:51:25.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:25.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:25.163774+0000) 2024-02-19T22:51:26.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:26.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:26.163934+0000) 2024-02-19T22:51:27.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:27.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:27.164077+0000) 2024-02-19T22:51:28.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:28.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:28.164297+0000) 2024-02-19T22:51:29.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:29.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:29.164532+0000) 2024-02-19T22:51:30.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:30.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:30.164789+0000) 2024-02-19T22:51:31.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:31.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:31.165012+0000) 2024-02-19T22:51:32.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:32.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:32.165240+0000) 2024-02-19T22:51:33.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:33.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:33.165463+0000) 2024-02-19T22:51:34.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:34.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:34.165697+0000) 2024-02-19T22:51:35.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:35.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:35.165867+0000) 2024-02-19T22:51:36.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:36.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:36.166078+0000) 2024-02-19T22:51:37.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:37.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:37.166236+0000) 2024-02-19T22:51:38.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:38.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:38.166463+0000) 2024-02-19T22:51:39.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:39.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:39.166662+0000) 2024-02-19T22:51:40.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:40.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:40.166888+0000) 2024-02-19T22:51:40.888 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:51:40.891 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:51:41.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:41.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:41.167109+0000) 2024-02-19T22:51:42.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:42.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:42.167281+0000) 2024-02-19T22:51:43.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:43.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:43.167442+0000) 2024-02-19T22:51:44.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:44.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:44.167619+0000) 2024-02-19T22:51:45.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:45.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:45.167819+0000) 2024-02-19T22:51:46.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:46.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:46.168083+0000) 2024-02-19T22:51:47.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:47.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:47.168305+0000) 2024-02-19T22:51:48.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:48.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:48.168543+0000) 2024-02-19T22:51:49.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:49.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:49.168775+0000) 2024-02-19T22:51:50.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:50.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:50.168978+0000) 2024-02-19T22:51:51.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:51.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:51.169195+0000) 2024-02-19T22:51:52.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:52.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:52.169404+0000) 2024-02-19T22:51:53.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:53.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:53.169625+0000) 2024-02-19T22:51:54.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:54.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:54.169819+0000) 2024-02-19T22:51:55.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:55.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:55.170041+0000) 2024-02-19T22:51:56.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:56.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:56.170212+0000) 2024-02-19T22:51:57.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:57.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:57.170439+0000) 2024-02-19T22:51:58.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:58.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:58.170635+0000) 2024-02-19T22:51:59.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:51:59.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:51:59.170811+0000) 2024-02-19T22:52:00.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:00.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:00.170946+0000) 2024-02-19T22:52:01.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:01.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:01.171168+0000) 2024-02-19T22:52:02.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:02.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:02.171346+0000) 2024-02-19T22:52:03.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:03.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:03.171596+0000) 2024-02-19T22:52:04.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:04.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:04.171775+0000) 2024-02-19T22:52:05.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:05.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:05.172017+0000) 2024-02-19T22:52:06.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:06.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:06.172205+0000) 2024-02-19T22:52:07.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:07.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:07.172391+0000) 2024-02-19T22:52:08.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:08.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:08.172588+0000) 2024-02-19T22:52:09.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:09.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:09.172813+0000) 2024-02-19T22:52:10.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:10.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:10.173010+0000) 2024-02-19T22:52:10.952 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:52:10.955 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:52:11.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:11.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:11.173207+0000) 2024-02-19T22:52:12.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:12.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:12.173413+0000) 2024-02-19T22:52:13.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:13.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:13.173639+0000) 2024-02-19T22:52:14.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:14.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:14.173818+0000) 2024-02-19T22:52:15.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:15.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:15.173976+0000) 2024-02-19T22:52:16.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:16.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:16.174237+0000) 2024-02-19T22:52:17.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:17.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:17.174424+0000) 2024-02-19T22:52:18.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:18.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:18.174780+0000) 2024-02-19T22:52:19.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:19.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:19.174951+0000) 2024-02-19T22:52:20.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:20.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:20.175130+0000) 2024-02-19T22:52:21.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:21.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:21.175311+0000) 2024-02-19T22:52:22.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:22.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:22.175558+0000) 2024-02-19T22:52:23.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:23.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:23.175768+0000) 2024-02-19T22:52:24.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:24.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:24.175933+0000) 2024-02-19T22:52:25.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:25.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:25.176132+0000) 2024-02-19T22:52:26.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:26.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:26.176367+0000) 2024-02-19T22:52:27.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:27.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:27.176633+0000) 2024-02-19T22:52:28.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:28.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:28.176838+0000) 2024-02-19T22:52:29.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:29.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:29.177065+0000) 2024-02-19T22:52:30.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:30.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:30.177317+0000) 2024-02-19T22:52:31.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:31.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:31.177571+0000) 2024-02-19T22:52:32.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:32.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:32.177805+0000) 2024-02-19T22:52:33.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:33.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:33.178034+0000) 2024-02-19T22:52:34.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:34.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:34.178293+0000) 2024-02-19T22:52:35.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:35.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:35.178570+0000) 2024-02-19T22:52:36.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:36.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:36.178869+0000) 2024-02-19T22:52:37.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:37.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:37.179019+0000) 2024-02-19T22:52:38.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:38.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:38.179202+0000) 2024-02-19T22:52:39.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:39.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:39.179382+0000) 2024-02-19T22:52:40.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:40.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:40.179595+0000) 2024-02-19T22:52:41.016 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:52:41.020 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:52:41.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:41.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:41.179875+0000) 2024-02-19T22:52:42.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:42.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:42.180099+0000) 2024-02-19T22:52:43.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:43.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:43.180346+0000) 2024-02-19T22:52:44.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:44.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:44.180580+0000) 2024-02-19T22:52:45.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:45.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:45.180729+0000) 2024-02-19T22:52:46.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:46.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:46.181002+0000) 2024-02-19T22:52:47.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:47.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:47.181227+0000) 2024-02-19T22:52:48.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:48.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:48.181423+0000) 2024-02-19T22:52:49.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:49.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:49.181678+0000) 2024-02-19T22:52:50.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:50.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:50.181944+0000) 2024-02-19T22:52:51.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:51.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:51.182229+0000) 2024-02-19T22:52:52.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:52.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:52.182408+0000) 2024-02-19T22:52:53.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:53.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:53.182635+0000) 2024-02-19T22:52:54.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:54.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:54.182883+0000) 2024-02-19T22:52:55.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:55.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:55.183068+0000) 2024-02-19T22:52:56.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:56.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:56.183308+0000) 2024-02-19T22:52:57.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:57.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:57.183574+0000) 2024-02-19T22:52:58.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:58.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:58.183858+0000) 2024-02-19T22:52:59.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:52:59.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:52:59.184040+0000) 2024-02-19T22:53:00.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:00.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:00.184309+0000) 2024-02-19T22:53:01.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:01.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:01.184583+0000) 2024-02-19T22:53:02.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:02.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:02.184846+0000) 2024-02-19T22:53:03.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:03.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:03.185027+0000) 2024-02-19T22:53:04.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:04.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:04.185287+0000) 2024-02-19T22:53:05.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:05.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:05.185496+0000) 2024-02-19T22:53:06.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:06.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:06.185760+0000) 2024-02-19T22:53:07.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:07.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:07.185951+0000) 2024-02-19T22:53:08.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:08.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:08.186198+0000) 2024-02-19T22:53:09.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:09.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:09.186391+0000) 2024-02-19T22:53:10.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:10.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:10.186628+0000) 2024-02-19T22:53:11.078 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:53:11.082 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:53:11.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:11.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:11.186879+0000) 2024-02-19T22:53:12.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:12.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:12.187123+0000) 2024-02-19T22:53:13.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:13.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:13.187372+0000) 2024-02-19T22:53:14.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:14.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:14.187647+0000) 2024-02-19T22:53:15.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:15.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:15.187898+0000) 2024-02-19T22:53:16.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:16.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:16.188143+0000) 2024-02-19T22:53:17.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:17.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:17.188394+0000) 2024-02-19T22:53:18.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:18.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:18.188566+0000) 2024-02-19T22:53:19.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:19.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:19.188825+0000) 2024-02-19T22:53:20.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:20.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:20.189103+0000) 2024-02-19T22:53:21.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:21.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:21.189305+0000) 2024-02-19T22:53:22.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:22.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:22.189469+0000) 2024-02-19T22:53:23.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:23.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:23.189692+0000) 2024-02-19T22:53:24.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:24.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:24.189877+0000) 2024-02-19T22:53:25.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:25.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:25.190051+0000) 2024-02-19T22:53:26.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:26.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:26.190293+0000) 2024-02-19T22:53:27.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:27.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:27.190528+0000) 2024-02-19T22:53:28.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:28.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:28.190758+0000) 2024-02-19T22:53:29.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:29.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:29.190928+0000) 2024-02-19T22:53:30.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:30.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:30.191076+0000) 2024-02-19T22:53:31.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:31.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:31.191294+0000) 2024-02-19T22:53:32.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:32.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:32.191545+0000) 2024-02-19T22:53:33.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:33.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:33.191763+0000) 2024-02-19T22:53:34.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:34.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:34.191986+0000) 2024-02-19T22:53:35.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:35.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:35.192154+0000) 2024-02-19T22:53:36.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:36.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:36.192356+0000) 2024-02-19T22:53:37.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:37.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:37.192625+0000) 2024-02-19T22:53:38.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:38.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:38.192899+0000) 2024-02-19T22:53:39.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:39.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:39.193132+0000) 2024-02-19T22:53:40.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:40.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:40.193413+0000) 2024-02-19T22:53:41.153 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:53:41.157 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:53:41.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:41.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:41.193621+0000) 2024-02-19T22:53:42.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:42.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:42.193874+0000) 2024-02-19T22:53:43.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:43.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:43.194127+0000) 2024-02-19T22:53:44.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:44.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:44.194312+0000) 2024-02-19T22:53:45.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:45.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:45.194552+0000) 2024-02-19T22:53:46.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:46.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:46.194727+0000) 2024-02-19T22:53:47.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:47.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:47.194864+0000) 2024-02-19T22:53:48.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:48.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:48.194986+0000) 2024-02-19T22:53:49.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:49.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:49.195115+0000) 2024-02-19T22:53:50.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:50.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:50.195303+0000) 2024-02-19T22:53:51.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:51.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:51.195559+0000) 2024-02-19T22:53:52.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:52.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:52.195735+0000) 2024-02-19T22:53:53.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:53.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:53.195866+0000) 2024-02-19T22:53:54.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:54.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:54.195984+0000) 2024-02-19T22:53:55.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:55.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:55.196156+0000) 2024-02-19T22:53:56.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:56.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:56.196328+0000) 2024-02-19T22:53:57.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:57.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:57.196460+0000) 2024-02-19T22:53:58.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:58.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:58.196635+0000) 2024-02-19T22:53:59.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:53:59.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:53:59.196763+0000) 2024-02-19T22:54:00.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:00.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:00.196901+0000) 2024-02-19T22:54:01.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:01.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:01.197060+0000) 2024-02-19T22:54:02.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:02.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:02.197198+0000) 2024-02-19T22:54:03.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:03.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:03.197338+0000) 2024-02-19T22:54:04.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:04.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:04.197462+0000) 2024-02-19T22:54:05.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:05.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:05.197610+0000) 2024-02-19T22:54:06.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:06.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:06.197740+0000) 2024-02-19T22:54:07.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:07.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:07.197962+0000) 2024-02-19T22:54:08.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:08.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:08.198233+0000) 2024-02-19T22:54:09.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:09.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:09.198515+0000) 2024-02-19T22:54:10.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:10.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:10.198885+0000) 2024-02-19T22:54:11.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:11.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:11.199053+0000) 2024-02-19T22:54:11.231 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:54:11.235 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:54:12.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:12.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:12.199226+0000) 2024-02-19T22:54:13.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:13.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:13.199462+0000) 2024-02-19T22:54:14.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:14.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:14.199720+0000) 2024-02-19T22:54:15.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:15.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:15.199916+0000) 2024-02-19T22:54:16.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:16.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:16.200182+0000) 2024-02-19T22:54:17.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:17.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:17.200466+0000) 2024-02-19T22:54:18.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:18.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:18.200727+0000) 2024-02-19T22:54:19.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:19.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:19.200954+0000) 2024-02-19T22:54:20.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:20.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:20.201199+0000) 2024-02-19T22:54:21.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:21.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:21.201470+0000) 2024-02-19T22:54:22.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:22.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:22.201703+0000) 2024-02-19T22:54:23.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:23.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:23.201941+0000) 2024-02-19T22:54:24.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:24.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:24.202176+0000) 2024-02-19T22:54:25.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:25.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:25.202393+0000) 2024-02-19T22:54:26.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:26.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:26.202645+0000) 2024-02-19T22:54:27.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:27.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:27.202877+0000) 2024-02-19T22:54:28.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:28.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:28.203159+0000) 2024-02-19T22:54:29.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:29.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:29.203330+0000) 2024-02-19T22:54:30.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:30.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:30.203588+0000) 2024-02-19T22:54:31.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:31.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:31.203806+0000) 2024-02-19T22:54:32.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:32.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:32.204044+0000) 2024-02-19T22:54:33.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:33.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:33.204228+0000) 2024-02-19T22:54:34.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:34.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:34.204403+0000) 2024-02-19T22:54:35.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:35.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:35.204540+0000) 2024-02-19T22:54:36.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:36.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:36.204671+0000) 2024-02-19T22:54:37.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:37.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:37.204900+0000) 2024-02-19T22:54:38.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:38.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:38.205091+0000) 2024-02-19T22:54:39.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:39.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:39.205391+0000) 2024-02-19T22:54:40.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:40.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:40.205636+0000) 2024-02-19T22:54:41.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:41.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:41.205913+0000) 2024-02-19T22:54:41.290 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:54:41.294 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:54:42.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:42.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:42.206236+0000) 2024-02-19T22:54:43.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:43.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:43.206526+0000) 2024-02-19T22:54:44.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:44.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:44.206771+0000) 2024-02-19T22:54:45.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:45.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:45.206971+0000) 2024-02-19T22:54:46.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:46.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:46.207186+0000) 2024-02-19T22:54:47.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:47.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:47.207372+0000) 2024-02-19T22:54:48.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:48.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:48.207581+0000) 2024-02-19T22:54:49.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:49.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:49.207751+0000) 2024-02-19T22:54:50.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:50.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:50.208052+0000) 2024-02-19T22:54:51.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:51.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:51.208304+0000) 2024-02-19T22:54:52.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:52.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:52.208564+0000) 2024-02-19T22:54:53.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:53.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:53.208802+0000) 2024-02-19T22:54:54.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:54.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:54.209068+0000) 2024-02-19T22:54:55.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:55.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:55.209353+0000) 2024-02-19T22:54:56.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:56.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:56.209572+0000) 2024-02-19T22:54:57.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:57.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:57.209762+0000) 2024-02-19T22:54:58.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:58.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:58.210104+0000) 2024-02-19T22:54:59.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:54:59.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:54:59.210397+0000) 2024-02-19T22:55:00.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:00.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:00.210699+0000) 2024-02-19T22:55:01.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:01.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:01.210992+0000) 2024-02-19T22:55:02.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:02.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:02.211232+0000) 2024-02-19T22:55:03.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:03.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:03.211450+0000) 2024-02-19T22:55:04.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:04.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:04.211705+0000) 2024-02-19T22:55:05.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:05.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:05.212007+0000) 2024-02-19T22:55:06.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:06.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:06.212222+0000) 2024-02-19T22:55:07.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:07.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:07.212455+0000) 2024-02-19T22:55:08.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:08.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:08.212646+0000) 2024-02-19T22:55:09.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:09.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:09.212819+0000) 2024-02-19T22:55:10.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:10.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:10.213046+0000) 2024-02-19T22:55:11.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:11.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:11.213183+0000) 2024-02-19T22:55:11.350 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:55:11.353 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:55:12.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:12.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:12.213422+0000) 2024-02-19T22:55:13.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:13.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:13.213681+0000) 2024-02-19T22:55:14.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:14.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:14.213990+0000) 2024-02-19T22:55:15.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:15.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:15.214238+0000) 2024-02-19T22:55:16.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:16.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:16.214425+0000) 2024-02-19T22:55:17.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:17.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:17.214600+0000) 2024-02-19T22:55:18.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:18.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:18.214778+0000) 2024-02-19T22:55:19.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:19.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:19.214958+0000) 2024-02-19T22:55:20.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:20.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:20.215204+0000) 2024-02-19T22:55:21.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:21.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:21.215398+0000) 2024-02-19T22:55:22.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:22.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:22.215679+0000) 2024-02-19T22:55:23.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:23.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:23.215932+0000) 2024-02-19T22:55:24.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:24.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:24.216167+0000) 2024-02-19T22:55:25.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:25.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:25.216409+0000) 2024-02-19T22:55:26.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:26.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:26.216616+0000) 2024-02-19T22:55:27.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:27.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:27.216861+0000) 2024-02-19T22:55:28.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:28.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:28.217127+0000) 2024-02-19T22:55:29.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:29.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:29.217349+0000) 2024-02-19T22:55:30.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:30.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:30.217625+0000) 2024-02-19T22:55:31.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:31.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:31.217891+0000) 2024-02-19T22:55:32.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:32.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:32.218126+0000) 2024-02-19T22:55:33.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:33.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:33.218339+0000) 2024-02-19T22:55:34.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:34.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:34.218588+0000) 2024-02-19T22:55:35.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:35.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:35.218858+0000) 2024-02-19T22:55:36.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:36.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:36.219104+0000) 2024-02-19T22:55:37.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:37.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:37.219358+0000) 2024-02-19T22:55:38.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:38.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:38.219581+0000) 2024-02-19T22:55:39.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:39.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:39.219759+0000) 2024-02-19T22:55:40.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:40.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:40.219997+0000) 2024-02-19T22:55:41.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:41.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:41.220165+0000) 2024-02-19T22:55:41.472 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:55:41.475 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:55:42.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:42.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:42.220417+0000) 2024-02-19T22:55:43.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:43.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:43.220616+0000) 2024-02-19T22:55:44.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:44.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:44.220849+0000) 2024-02-19T22:55:45.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:45.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:45.221025+0000) 2024-02-19T22:55:46.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:46.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:46.221282+0000) 2024-02-19T22:55:47.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:47.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:47.221567+0000) 2024-02-19T22:55:48.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:48.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:48.221855+0000) 2024-02-19T22:55:49.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:49.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:49.222036+0000) 2024-02-19T22:55:50.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:50.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:50.222289+0000) 2024-02-19T22:55:51.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:51.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:51.222533+0000) 2024-02-19T22:55:52.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:52.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:52.222849+0000) 2024-02-19T22:55:53.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:53.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:53.223037+0000) 2024-02-19T22:55:54.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:54.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:54.223193+0000) 2024-02-19T22:55:55.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:55.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:55.223371+0000) 2024-02-19T22:55:56.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:56.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:56.223579+0000) 2024-02-19T22:55:57.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:57.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:57.223826+0000) 2024-02-19T22:55:58.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:58.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:58.224075+0000) 2024-02-19T22:55:59.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:55:59.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:55:59.224308+0000) 2024-02-19T22:56:00.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:00.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:00.224585+0000) 2024-02-19T22:56:01.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:01.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:01.224832+0000) 2024-02-19T22:56:02.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:02.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:02.225014+0000) 2024-02-19T22:56:03.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:03.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:03.225258+0000) 2024-02-19T22:56:04.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:04.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:04.225506+0000) 2024-02-19T22:56:05.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:05.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:05.225743+0000) 2024-02-19T22:56:06.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:06.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:06.225980+0000) 2024-02-19T22:56:07.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:07.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:07.226234+0000) 2024-02-19T22:56:08.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:08.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:08.226401+0000) 2024-02-19T22:56:09.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:09.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:09.226623+0000) 2024-02-19T22:56:10.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:10.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:10.226810+0000) 2024-02-19T22:56:11.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:11.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:11.226956+0000) 2024-02-19T22:56:11.586 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:56:11.589 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:56:12.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:12.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:12.227200+0000) 2024-02-19T22:56:13.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:13.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:13.227408+0000) 2024-02-19T22:56:14.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:14.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:14.227627+0000) 2024-02-19T22:56:15.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:15.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:15.227863+0000) 2024-02-19T22:56:16.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:16.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:16.228060+0000) 2024-02-19T22:56:17.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:17.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:17.228312+0000) 2024-02-19T22:56:18.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:18.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:18.228459+0000) 2024-02-19T22:56:19.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:19.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:19.228710+0000) 2024-02-19T22:56:20.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:20.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:20.228968+0000) 2024-02-19T22:56:21.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:21.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:21.229209+0000) 2024-02-19T22:56:22.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:22.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:22.229459+0000) 2024-02-19T22:56:23.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:23.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:23.229675+0000) 2024-02-19T22:56:24.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:24.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:24.229973+0000) 2024-02-19T22:56:25.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:25.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:25.230193+0000) 2024-02-19T22:56:26.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:26.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:26.230332+0000) 2024-02-19T22:56:27.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:27.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:27.230629+0000) 2024-02-19T22:56:28.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:28.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:28.230893+0000) 2024-02-19T22:56:29.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:29.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:29.231139+0000) 2024-02-19T22:56:30.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:30.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:30.231411+0000) 2024-02-19T22:56:31.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:31.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:31.231701+0000) 2024-02-19T22:56:32.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:32.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:32.231979+0000) 2024-02-19T22:56:33.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:33.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:33.232265+0000) 2024-02-19T22:56:34.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:34.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:34.232574+0000) 2024-02-19T22:56:35.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:35.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:35.232783+0000) 2024-02-19T22:56:36.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:36.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:36.233077+0000) 2024-02-19T22:56:37.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:37.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:37.233306+0000) 2024-02-19T22:56:38.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:38.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:38.233552+0000) 2024-02-19T22:56:39.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:39.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:39.233824+0000) 2024-02-19T22:56:40.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:40.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:40.234120+0000) 2024-02-19T22:56:41.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:41.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:41.234319+0000) 2024-02-19T22:56:41.716 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:56:41.719 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:56:42.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:42.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:42.234553+0000) 2024-02-19T22:56:43.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:43.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:43.234826+0000) 2024-02-19T22:56:44.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:44.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:44.234967+0000) 2024-02-19T22:56:45.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:45.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:45.235145+0000) 2024-02-19T22:56:46.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:46.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:46.235345+0000) 2024-02-19T22:56:47.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:47.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:47.235536+0000) 2024-02-19T22:56:48.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:48.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:48.235781+0000) 2024-02-19T22:56:49.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:49.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:49.236035+0000) 2024-02-19T22:56:50.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:50.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:50.236188+0000) 2024-02-19T22:56:51.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:51.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:51.236408+0000) 2024-02-19T22:56:52.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:52.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:52.236679+0000) 2024-02-19T22:56:53.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:53.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:53.236876+0000) 2024-02-19T22:56:54.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:54.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:54.237139+0000) 2024-02-19T22:56:55.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:55.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:55.237311+0000) 2024-02-19T22:56:56.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:56.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:56.237564+0000) 2024-02-19T22:56:57.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:57.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:57.237762+0000) 2024-02-19T22:56:58.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:58.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:58.237957+0000) 2024-02-19T22:56:59.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:56:59.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:56:59.238166+0000) 2024-02-19T22:57:00.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:00.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:00.238367+0000) 2024-02-19T22:57:01.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:01.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:01.238575+0000) 2024-02-19T22:57:02.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:02.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:02.238813+0000) 2024-02-19T22:57:03.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:03.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:03.238981+0000) 2024-02-19T22:57:04.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:04.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:04.239265+0000) 2024-02-19T22:57:05.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:05.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:05.239565+0000) 2024-02-19T22:57:06.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:06.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:06.239827+0000) 2024-02-19T22:57:07.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:07.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:07.240065+0000) 2024-02-19T22:57:08.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:08.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:08.240281+0000) 2024-02-19T22:57:09.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:09.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:09.240517+0000) 2024-02-19T22:57:10.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:10.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:10.240730+0000) 2024-02-19T22:57:11.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:11.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:11.240971+0000) 2024-02-19T22:57:11.847 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:57:11.850 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:57:12.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:12.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:12.241166+0000) 2024-02-19T22:57:13.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:13.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:13.241370+0000) 2024-02-19T22:57:14.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:14.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:14.241510+0000) 2024-02-19T22:57:15.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:15.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:15.241763+0000) 2024-02-19T22:57:16.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:16.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:16.242024+0000) 2024-02-19T22:57:17.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:17.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:17.242257+0000) 2024-02-19T22:57:18.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:18.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:18.242501+0000) 2024-02-19T22:57:19.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:19.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:19.242757+0000) 2024-02-19T22:57:20.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:20.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:20.242951+0000) 2024-02-19T22:57:21.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:21.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:21.243109+0000) 2024-02-19T22:57:22.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:22.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:22.243367+0000) 2024-02-19T22:57:23.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:23.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:23.243613+0000) 2024-02-19T22:57:24.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:24.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:24.243827+0000) 2024-02-19T22:57:25.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:25.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:25.244102+0000) 2024-02-19T22:57:26.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:26.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:26.244277+0000) 2024-02-19T22:57:27.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:27.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:27.244513+0000) 2024-02-19T22:57:28.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:28.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:28.244780+0000) 2024-02-19T22:57:29.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:29.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:29.245030+0000) 2024-02-19T22:57:30.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:30.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:30.245283+0000) 2024-02-19T22:57:31.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:31.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:31.245512+0000) 2024-02-19T22:57:32.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:32.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:32.245764+0000) 2024-02-19T22:57:33.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:33.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:33.246039+0000) 2024-02-19T22:57:34.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:34.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:34.246256+0000) 2024-02-19T22:57:35.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:35.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:35.246426+0000) 2024-02-19T22:57:36.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:36.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:36.246552+0000) 2024-02-19T22:57:37.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:37.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:37.246763+0000) 2024-02-19T22:57:38.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:38.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:38.246990+0000) 2024-02-19T22:57:39.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:39.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:39.247250+0000) 2024-02-19T22:57:40.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:40.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:40.247434+0000) 2024-02-19T22:57:41.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:41.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:41.247617+0000) 2024-02-19T22:57:42.086 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:57:42.090 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:57:42.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:42.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:42.247829+0000) 2024-02-19T22:57:43.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:43.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:43.248068+0000) 2024-02-19T22:57:44.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:44.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:44.248243+0000) 2024-02-19T22:57:45.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:45.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:45.248481+0000) 2024-02-19T22:57:46.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:46.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:46.248727+0000) 2024-02-19T22:57:47.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:47.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:47.248982+0000) 2024-02-19T22:57:48.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:48.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:48.249228+0000) 2024-02-19T22:57:49.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:49.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:49.249497+0000) 2024-02-19T22:57:50.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:50.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:50.249708+0000) 2024-02-19T22:57:51.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:51.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:51.249909+0000) 2024-02-19T22:57:52.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:52.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:52.250123+0000) 2024-02-19T22:57:53.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:53.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:53.250363+0000) 2024-02-19T22:57:54.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:54.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:54.250601+0000) 2024-02-19T22:57:55.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:55.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:55.250893+0000) 2024-02-19T22:57:56.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:56.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:56.251082+0000) 2024-02-19T22:57:57.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:57.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:57.251294+0000) 2024-02-19T22:57:58.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:58.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:58.251564+0000) 2024-02-19T22:57:59.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:57:59.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:57:59.251731+0000) 2024-02-19T22:58:00.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:00.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:00.251980+0000) 2024-02-19T22:58:01.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:01.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:01.252234+0000) 2024-02-19T22:58:02.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:02.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:02.252492+0000) 2024-02-19T22:58:03.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:03.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:03.252711+0000) 2024-02-19T22:58:04.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:04.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:04.252950+0000) 2024-02-19T22:58:05.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:05.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:05.253196+0000) 2024-02-19T22:58:06.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:06.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:06.253381+0000) 2024-02-19T22:58:07.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:07.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:07.253614+0000) 2024-02-19T22:58:08.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:08.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:08.253848+0000) 2024-02-19T22:58:09.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:09.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:09.254091+0000) 2024-02-19T22:58:10.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:10.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:10.254345+0000) 2024-02-19T22:58:11.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:11.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:11.254529+0000) 2024-02-19T22:58:12.149 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:58:12.153 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:58:12.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:12.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:12.254757+0000) 2024-02-19T22:58:13.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:13.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:13.254952+0000) 2024-02-19T22:58:14.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:14.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:14.255168+0000) 2024-02-19T22:58:15.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:15.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:15.255335+0000) 2024-02-19T22:58:16.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:16.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:16.255526+0000) 2024-02-19T22:58:17.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:17.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:17.255650+0000) 2024-02-19T22:58:18.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:18.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:18.255834+0000) 2024-02-19T22:58:19.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:19.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:19.256059+0000) 2024-02-19T22:58:20.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:20.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:20.256231+0000) 2024-02-19T22:58:21.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:21.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:21.256510+0000) 2024-02-19T22:58:22.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:22.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:22.256731+0000) 2024-02-19T22:58:23.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:23.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:23.256939+0000) 2024-02-19T22:58:24.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:24.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:24.257161+0000) 2024-02-19T22:58:25.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:25.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:25.257343+0000) 2024-02-19T22:58:26.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:26.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:26.257579+0000) 2024-02-19T22:58:27.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:27.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:27.257821+0000) 2024-02-19T22:58:28.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:28.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:28.257958+0000) 2024-02-19T22:58:29.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:29.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:29.258182+0000) 2024-02-19T22:58:30.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:30.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:30.258465+0000) 2024-02-19T22:58:31.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:31.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:31.258689+0000) 2024-02-19T22:58:32.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:32.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:32.258907+0000) 2024-02-19T22:58:33.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:33.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:33.259078+0000) 2024-02-19T22:58:34.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:34.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:34.259273+0000) 2024-02-19T22:58:35.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:35.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:35.259466+0000) 2024-02-19T22:58:36.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:36.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:36.259708+0000) 2024-02-19T22:58:37.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:37.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:37.259891+0000) 2024-02-19T22:58:38.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:38.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:38.260095+0000) 2024-02-19T22:58:39.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:39.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:39.260345+0000) 2024-02-19T22:58:40.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:40.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:40.260602+0000) 2024-02-19T22:58:41.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:41.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:41.260863+0000) 2024-02-19T22:58:42.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:42.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:42.261097+0000) 2024-02-19T22:58:42.271 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:58:42.275 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:58:43.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:43.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:43.261304+0000) 2024-02-19T22:58:44.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:44.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:44.261532+0000) 2024-02-19T22:58:45.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:45.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:45.261774+0000) 2024-02-19T22:58:46.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:46.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:46.262037+0000) 2024-02-19T22:58:47.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:47.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:47.262297+0000) 2024-02-19T22:58:48.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:48.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:48.262569+0000) 2024-02-19T22:58:49.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:49.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:49.262817+0000) 2024-02-19T22:58:50.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:50.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:50.263054+0000) 2024-02-19T22:58:51.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:51.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:51.263229+0000) 2024-02-19T22:58:52.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:52.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:52.263460+0000) 2024-02-19T22:58:53.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:53.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:53.263633+0000) 2024-02-19T22:58:54.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:54.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:54.263821+0000) 2024-02-19T22:58:55.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:55.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:55.264049+0000) 2024-02-19T22:58:56.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:56.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:56.264187+0000) 2024-02-19T22:58:57.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:57.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:57.264448+0000) 2024-02-19T22:58:58.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:58.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:58.264647+0000) 2024-02-19T22:58:59.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:58:59.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:58:59.264835+0000) 2024-02-19T22:59:00.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:00.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:00.265089+0000) 2024-02-19T22:59:01.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:01.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:01.265295+0000) 2024-02-19T22:59:02.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:02.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:02.265522+0000) 2024-02-19T22:59:03.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:03.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:03.265771+0000) 2024-02-19T22:59:04.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:04.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:04.265974+0000) 2024-02-19T22:59:05.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:05.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:05.266136+0000) 2024-02-19T22:59:06.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:06.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:06.266368+0000) 2024-02-19T22:59:07.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:07.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:07.266595+0000) 2024-02-19T22:59:08.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:08.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:08.266842+0000) 2024-02-19T22:59:09.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:09.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:09.267091+0000) 2024-02-19T22:59:10.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:10.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:10.267290+0000) 2024-02-19T22:59:11.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:11.262+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:11.267487+0000) 2024-02-19T22:59:12.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:12.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:12.267664+0000) 2024-02-19T22:59:12.351 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:59:12.355 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:59:13.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:13.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:13.267873+0000) 2024-02-19T22:59:14.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:14.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:14.268111+0000) 2024-02-19T22:59:15.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:15.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:15.268342+0000) 2024-02-19T22:59:16.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:16.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:16.268588+0000) 2024-02-19T22:59:17.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:17.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:17.268825+0000) 2024-02-19T22:59:18.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:18.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:18.269056+0000) 2024-02-19T22:59:19.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:19.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:19.269291+0000) 2024-02-19T22:59:20.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:20.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:20.269504+0000) 2024-02-19T22:59:21.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:21.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:21.269746+0000) 2024-02-19T22:59:22.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:22.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:22.269909+0000) 2024-02-19T22:59:23.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:23.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:23.270148+0000) 2024-02-19T22:59:24.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:24.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:24.270422+0000) 2024-02-19T22:59:25.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:25.266+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:25.270692+0000) 2024-02-19T22:59:26.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:26.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:26.270814+0000) 2024-02-19T22:59:27.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:27.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:27.270946+0000) 2024-02-19T22:59:28.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:28.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:28.271113+0000) 2024-02-19T22:59:29.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:29.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:29.271312+0000) 2024-02-19T22:59:30.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:30.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:30.271581+0000) 2024-02-19T22:59:31.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:31.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:31.271780+0000) 2024-02-19T22:59:32.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:32.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:32.271997+0000) 2024-02-19T22:59:33.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:33.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:33.272253+0000) 2024-02-19T22:59:34.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:34.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:34.272522+0000) 2024-02-19T22:59:35.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:35.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:35.272734+0000) 2024-02-19T22:59:36.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:36.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:36.272999+0000) 2024-02-19T22:59:37.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:37.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:37.273201+0000) 2024-02-19T22:59:38.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:38.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:38.273500+0000) 2024-02-19T22:59:39.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:39.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:39.273752+0000) 2024-02-19T22:59:40.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:40.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:40.274030+0000) 2024-02-19T22:59:41.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:41.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:41.274212+0000) 2024-02-19T22:59:42.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:42.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:42.274458+0000) 2024-02-19T22:59:42.407 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:59:42.411 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T22:59:43.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:43.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:43.274636+0000) 2024-02-19T22:59:44.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:44.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:44.274774+0000) 2024-02-19T22:59:45.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:45.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:45.275001+0000) 2024-02-19T22:59:46.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:46.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:46.275196+0000) 2024-02-19T22:59:47.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:47.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:47.275384+0000) 2024-02-19T22:59:48.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:48.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:48.275618+0000) 2024-02-19T22:59:49.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:49.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:49.275804+0000) 2024-02-19T22:59:50.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:50.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:50.276072+0000) 2024-02-19T22:59:51.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:51.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:51.276227+0000) 2024-02-19T22:59:52.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:52.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:52.276366+0000) 2024-02-19T22:59:53.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:53.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:53.276581+0000) 2024-02-19T22:59:54.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:54.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:54.276723+0000) 2024-02-19T22:59:55.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:55.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:55.276990+0000) 2024-02-19T22:59:56.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:56.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:56.277243+0000) 2024-02-19T22:59:57.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:57.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:57.277387+0000) 2024-02-19T22:59:58.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:58.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:58.277583+0000) 2024-02-19T22:59:59.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T22:59:59.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T21:59:59.277777+0000) 2024-02-19T23:00:00.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:00.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:00.277968+0000) 2024-02-19T23:00:01.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:01.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:01.278141+0000) 2024-02-19T23:00:02.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:02.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:02.278305+0000) 2024-02-19T23:00:03.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:03.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:03.278482+0000) 2024-02-19T23:00:04.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:04.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:04.278615+0000) 2024-02-19T23:00:05.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:05.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:05.278797+0000) 2024-02-19T23:00:06.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:06.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:06.278924+0000) 2024-02-19T23:00:07.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:07.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:07.279089+0000) 2024-02-19T23:00:08.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:08.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:08.279245+0000) 2024-02-19T23:00:09.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:09.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:09.279458+0000) 2024-02-19T23:00:10.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:10.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:10.279638+0000) 2024-02-19T23:00:11.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:11.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:11.279887+0000) 2024-02-19T23:00:12.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:12.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:12.280113+0000) 2024-02-19T23:00:12.521 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:00:12.528 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:00:13.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:13.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:13.280345+0000) 2024-02-19T23:00:14.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:14.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:14.280628+0000) 2024-02-19T23:00:15.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:15.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:15.280840+0000) 2024-02-19T23:00:16.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:16.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:16.281069+0000) 2024-02-19T23:00:17.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:17.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:17.281358+0000) 2024-02-19T23:00:18.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:18.277+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:18.281557+0000) 2024-02-19T23:00:19.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:19.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:19.281783+0000) 2024-02-19T23:00:20.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:20.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:20.282054+0000) 2024-02-19T23:00:21.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:21.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:21.282334+0000) 2024-02-19T23:00:22.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:22.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:22.282563+0000) 2024-02-19T23:00:23.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:23.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:23.282756+0000) 2024-02-19T23:00:24.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:24.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:24.282984+0000) 2024-02-19T23:00:25.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:25.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:25.283160+0000) 2024-02-19T23:00:26.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:26.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:26.283413+0000) 2024-02-19T23:00:27.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:27.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:27.283590+0000) 2024-02-19T23:00:28.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:28.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:28.283737+0000) 2024-02-19T23:00:29.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:29.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:29.283989+0000) 2024-02-19T23:00:30.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:30.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:30.284270+0000) 2024-02-19T23:00:31.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:31.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:31.284492+0000) 2024-02-19T23:00:32.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:32.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:32.284684+0000) 2024-02-19T23:00:33.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:33.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:33.284892+0000) 2024-02-19T23:00:34.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:34.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:34.285120+0000) 2024-02-19T23:00:35.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:35.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:35.285321+0000) 2024-02-19T23:00:36.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:36.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:36.285563+0000) 2024-02-19T23:00:37.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:37.281+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:37.285807+0000) 2024-02-19T23:00:38.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:38.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:38.286058+0000) 2024-02-19T23:00:39.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:39.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:39.286295+0000) 2024-02-19T23:00:40.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:40.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:40.286582+0000) 2024-02-19T23:00:41.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:41.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:41.286869+0000) 2024-02-19T23:00:42.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:42.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:42.287172+0000) 2024-02-19T23:00:42.667 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:00:42.671 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:00:43.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:43.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:43.287431+0000) 2024-02-19T23:00:44.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:44.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:44.287691+0000) 2024-02-19T23:00:45.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:45.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:45.287971+0000) 2024-02-19T23:00:46.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:46.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:46.288202+0000) 2024-02-19T23:00:47.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:47.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:47.288464+0000) 2024-02-19T23:00:48.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:48.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:48.288747+0000) 2024-02-19T23:00:49.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:49.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:49.288953+0000) 2024-02-19T23:00:50.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:50.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:50.289204+0000) 2024-02-19T23:00:51.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:51.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:51.289439+0000) 2024-02-19T23:00:52.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:52.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:52.289698+0000) 2024-02-19T23:00:53.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:53.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:53.289967+0000) 2024-02-19T23:00:54.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:54.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:54.290213+0000) 2024-02-19T23:00:55.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:55.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:55.290441+0000) 2024-02-19T23:00:56.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:56.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:56.290620+0000) 2024-02-19T23:00:57.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:57.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:57.290910+0000) 2024-02-19T23:00:58.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:58.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:58.291154+0000) 2024-02-19T23:00:59.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:00:59.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:00:59.291397+0000) 2024-02-19T23:01:00.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:00.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:00.291653+0000) 2024-02-19T23:01:01.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:01.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:01.291903+0000) 2024-02-19T23:01:02.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:02.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:02.292170+0000) 2024-02-19T23:01:03.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:03.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:03.292370+0000) 2024-02-19T23:01:04.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:04.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:04.292639+0000) 2024-02-19T23:01:05.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:05.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:05.292906+0000) 2024-02-19T23:01:06.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:06.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:06.293154+0000) 2024-02-19T23:01:07.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:07.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:07.293357+0000) 2024-02-19T23:01:08.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:08.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:08.293629+0000) 2024-02-19T23:01:09.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:09.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:09.293891+0000) 2024-02-19T23:01:10.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:10.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:10.294195+0000) 2024-02-19T23:01:11.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:11.288+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:11.294463+0000) 2024-02-19T23:01:12.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:12.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:12.294701+0000) 2024-02-19T23:01:12.798 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:01:12.801 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:01:13.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:13.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:13.294899+0000) 2024-02-19T23:01:14.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:14.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:14.295125+0000) 2024-02-19T23:01:15.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:15.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:15.295333+0000) 2024-02-19T23:01:16.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:16.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:16.295590+0000) 2024-02-19T23:01:17.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:17.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:17.295839+0000) 2024-02-19T23:01:18.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:18.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:18.296088+0000) 2024-02-19T23:01:19.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:19.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:19.296271+0000) 2024-02-19T23:01:20.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:20.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:20.296520+0000) 2024-02-19T23:01:21.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:21.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:21.296770+0000) 2024-02-19T23:01:22.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:22.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:22.297040+0000) 2024-02-19T23:01:23.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:23.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:23.297300+0000) 2024-02-19T23:01:24.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:24.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:24.297550+0000) 2024-02-19T23:01:25.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:25.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:25.297808+0000) 2024-02-19T23:01:26.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:26.292+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:26.298060+0000) 2024-02-19T23:01:27.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:27.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:27.298321+0000) 2024-02-19T23:01:28.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:28.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:28.298586+0000) 2024-02-19T23:01:29.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:29.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:29.298790+0000) 2024-02-19T23:01:30.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:30.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:30.299039+0000) 2024-02-19T23:01:31.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:31.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:31.299250+0000) 2024-02-19T23:01:32.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:32.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:32.299453+0000) 2024-02-19T23:01:33.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:33.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:33.299637+0000) 2024-02-19T23:01:34.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:34.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:34.299859+0000) 2024-02-19T23:01:35.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:35.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:35.300094+0000) 2024-02-19T23:01:36.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:36.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:36.300369+0000) 2024-02-19T23:01:37.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:37.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:37.300624+0000) 2024-02-19T23:01:38.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:38.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:38.300921+0000) 2024-02-19T23:01:39.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:39.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:39.301097+0000) 2024-02-19T23:01:40.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:40.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:40.301361+0000) 2024-02-19T23:01:41.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:41.296+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:41.301590+0000) 2024-02-19T23:01:42.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:42.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:42.301880+0000) 2024-02-19T23:01:42.911 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:01:42.915 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:01:43.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:43.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:43.302135+0000) 2024-02-19T23:01:44.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:44.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:44.302350+0000) 2024-02-19T23:01:45.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:45.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:45.302631+0000) 2024-02-19T23:01:46.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:46.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:46.302874+0000) 2024-02-19T23:01:47.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:47.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:47.303147+0000) 2024-02-19T23:01:48.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:48.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:48.303370+0000) 2024-02-19T23:01:49.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:49.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:49.303551+0000) 2024-02-19T23:01:50.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:50.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:50.303795+0000) 2024-02-19T23:01:51.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:51.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:51.303998+0000) 2024-02-19T23:01:52.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:52.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:52.304250+0000) 2024-02-19T23:01:53.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:53.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:53.304451+0000) 2024-02-19T23:01:54.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:54.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:54.304701+0000) 2024-02-19T23:01:55.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:55.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:55.304960+0000) 2024-02-19T23:01:56.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:56.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:56.305217+0000) 2024-02-19T23:01:57.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:57.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:57.305462+0000) 2024-02-19T23:01:58.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:58.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:58.305680+0000) 2024-02-19T23:01:59.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:01:59.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:01:59.305926+0000) 2024-02-19T23:02:00.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:00.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:00.306137+0000) 2024-02-19T23:02:01.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:01.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:01.306376+0000) 2024-02-19T23:02:02.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:02.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:02.306645+0000) 2024-02-19T23:02:03.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:03.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:03.306907+0000) 2024-02-19T23:02:04.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:04.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:04.307179+0000) 2024-02-19T23:02:05.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:05.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:05.307438+0000) 2024-02-19T23:02:06.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:06.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:06.307688+0000) 2024-02-19T23:02:07.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:07.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:07.307958+0000) 2024-02-19T23:02:08.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:08.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:08.308209+0000) 2024-02-19T23:02:09.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:09.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:09.308442+0000) 2024-02-19T23:02:10.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:10.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:10.308716+0000) 2024-02-19T23:02:11.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:11.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:11.308893+0000) 2024-02-19T23:02:12.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:12.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:12.309174+0000) 2024-02-19T23:02:13.049 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:02:13.053 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:02:13.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:13.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:13.309415+0000) 2024-02-19T23:02:14.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:14.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:14.309592+0000) 2024-02-19T23:02:15.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:15.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:15.309769+0000) 2024-02-19T23:02:16.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:16.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:16.310022+0000) 2024-02-19T23:02:17.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:17.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:17.310229+0000) 2024-02-19T23:02:18.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:18.303+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:18.310418+0000) 2024-02-19T23:02:19.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:19.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:19.310640+0000) 2024-02-19T23:02:20.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:20.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:20.310927+0000) 2024-02-19T23:02:21.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:21.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:21.311118+0000) 2024-02-19T23:02:22.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:22.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:22.311364+0000) 2024-02-19T23:02:23.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:23.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:23.311567+0000) 2024-02-19T23:02:24.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:24.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:24.311745+0000) 2024-02-19T23:02:25.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:25.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:25.311950+0000) 2024-02-19T23:02:26.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:26.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:26.312235+0000) 2024-02-19T23:02:27.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:27.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:27.312485+0000) 2024-02-19T23:02:28.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:28.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:28.312690+0000) 2024-02-19T23:02:29.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:29.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:29.312935+0000) 2024-02-19T23:02:30.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:30.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:30.313180+0000) 2024-02-19T23:02:31.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:31.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:31.313426+0000) 2024-02-19T23:02:32.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:32.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:32.313716+0000) 2024-02-19T23:02:33.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:33.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:33.313951+0000) 2024-02-19T23:02:34.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:34.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:34.314222+0000) 2024-02-19T23:02:35.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:35.307+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:35.314427+0000) 2024-02-19T23:02:36.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:36.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:36.314647+0000) 2024-02-19T23:02:37.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:37.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:37.314896+0000) 2024-02-19T23:02:38.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:38.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:38.315048+0000) 2024-02-19T23:02:39.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:39.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:39.315241+0000) 2024-02-19T23:02:40.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:40.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:40.315505+0000) 2024-02-19T23:02:41.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:41.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:41.315792+0000) 2024-02-19T23:02:42.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:42.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:42.316030+0000) 2024-02-19T23:02:43.127 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:02:43.130 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:02:43.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:43.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:43.316170+0000) 2024-02-19T23:02:44.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:44.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:44.316445+0000) 2024-02-19T23:02:45.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:45.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:45.316688+0000) 2024-02-19T23:02:46.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:46.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:46.316933+0000) 2024-02-19T23:02:47.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:47.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:47.317178+0000) 2024-02-19T23:02:48.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:48.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:48.317350+0000) 2024-02-19T23:02:49.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:49.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:49.317606+0000) 2024-02-19T23:02:50.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:50.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:50.317823+0000) 2024-02-19T23:02:51.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:51.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:51.317961+0000) 2024-02-19T23:02:52.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:52.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:52.318192+0000) 2024-02-19T23:02:53.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:53.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:53.318471+0000) 2024-02-19T23:02:54.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:54.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:54.318671+0000) 2024-02-19T23:02:55.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:55.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:55.318962+0000) 2024-02-19T23:02:56.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:56.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:56.319197+0000) 2024-02-19T23:02:57.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:57.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:57.319454+0000) 2024-02-19T23:02:58.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:58.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:58.319588+0000) 2024-02-19T23:02:59.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:02:59.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:02:59.319768+0000) 2024-02-19T23:03:00.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:00.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:00.319918+0000) 2024-02-19T23:03:01.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:01.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:01.320159+0000) 2024-02-19T23:03:02.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:02.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:02.320333+0000) 2024-02-19T23:03:03.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:03.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:03.320578+0000) 2024-02-19T23:03:04.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:04.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:04.320760+0000) 2024-02-19T23:03:05.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:05.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:05.320883+0000) 2024-02-19T23:03:06.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:06.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:06.321062+0000) 2024-02-19T23:03:07.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:07.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:07.321307+0000) 2024-02-19T23:03:08.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:08.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:08.321487+0000) 2024-02-19T23:03:09.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:09.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:09.321717+0000) 2024-02-19T23:03:10.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:10.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:10.321908+0000) 2024-02-19T23:03:11.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:11.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:11.322072+0000) 2024-02-19T23:03:12.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:12.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:12.322229+0000) 2024-02-19T23:03:13.262 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:03:13.265 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:03:13.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:13.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:13.322392+0000) 2024-02-19T23:03:14.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:14.314+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:14.322554+0000) 2024-02-19T23:03:15.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:15.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:15.322727+0000) 2024-02-19T23:03:16.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:16.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:16.322866+0000) 2024-02-19T23:03:17.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:17.318+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:17.322998+0000) 2024-02-19T23:03:18.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:18.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:18.323133+0000) 2024-02-19T23:03:19.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:19.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:19.323259+0000) 2024-02-19T23:03:20.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:20.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:20.323520+0000) 2024-02-19T23:03:21.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:21.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:21.323750+0000) 2024-02-19T23:03:22.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:22.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:22.324041+0000) 2024-02-19T23:03:23.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:23.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:23.324249+0000) 2024-02-19T23:03:24.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:24.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:24.324451+0000) 2024-02-19T23:03:25.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:25.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:25.324626+0000) 2024-02-19T23:03:26.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:26.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:26.324832+0000) 2024-02-19T23:03:27.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:27.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:27.325127+0000) 2024-02-19T23:03:28.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:28.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:28.325370+0000) 2024-02-19T23:03:29.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:29.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:29.325668+0000) 2024-02-19T23:03:30.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:30.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:30.325942+0000) 2024-02-19T23:03:31.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:31.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:31.326241+0000) 2024-02-19T23:03:32.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:32.322+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:32.326492+0000) 2024-02-19T23:03:33.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:33.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:33.326812+0000) 2024-02-19T23:03:34.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:34.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:34.327096+0000) 2024-02-19T23:03:35.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:35.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:35.327410+0000) 2024-02-19T23:03:36.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:36.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:36.327579+0000) 2024-02-19T23:03:37.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:37.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:37.327816+0000) 2024-02-19T23:03:38.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:38.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:38.328087+0000) 2024-02-19T23:03:39.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:39.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:39.328356+0000) 2024-02-19T23:03:40.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:40.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:40.328613+0000) 2024-02-19T23:03:41.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:41.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:41.328892+0000) 2024-02-19T23:03:42.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:42.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:42.329094+0000) 2024-02-19T23:03:43.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:43.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:43.329383+0000) 2024-02-19T23:03:43.399 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:03:43.402 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:03:44.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:44.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:44.329685+0000) 2024-02-19T23:03:45.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:45.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:45.329967+0000) 2024-02-19T23:03:46.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:46.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:46.330185+0000) 2024-02-19T23:03:47.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:47.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:47.330393+0000) 2024-02-19T23:03:48.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:48.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:48.330674+0000) 2024-02-19T23:03:49.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:49.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:49.330958+0000) 2024-02-19T23:03:50.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:50.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:50.331133+0000) 2024-02-19T23:03:51.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:51.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:51.331351+0000) 2024-02-19T23:03:52.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:52.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:52.331536+0000) 2024-02-19T23:03:53.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:53.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:53.331816+0000) 2024-02-19T23:03:54.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:54.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:54.332015+0000) 2024-02-19T23:03:55.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:55.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:55.332272+0000) 2024-02-19T23:03:56.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:56.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:56.332510+0000) 2024-02-19T23:03:57.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:57.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:57.332797+0000) 2024-02-19T23:03:58.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:58.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:58.333026+0000) 2024-02-19T23:03:59.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:03:59.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:03:59.333303+0000) 2024-02-19T23:04:00.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:00.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:00.333536+0000) 2024-02-19T23:04:01.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:01.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:01.333786+0000) 2024-02-19T23:04:02.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:02.329+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:02.334013+0000) 2024-02-19T23:04:03.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:03.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:03.334206+0000) 2024-02-19T23:04:04.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:04.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:04.334472+0000) 2024-02-19T23:04:05.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:05.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:05.334699+0000) 2024-02-19T23:04:06.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:06.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:06.334937+0000) 2024-02-19T23:04:07.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:07.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:07.335121+0000) 2024-02-19T23:04:08.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:08.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:08.335440+0000) 2024-02-19T23:04:09.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:09.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:09.335711+0000) 2024-02-19T23:04:10.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:10.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:10.335994+0000) 2024-02-19T23:04:11.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:11.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:11.336197+0000) 2024-02-19T23:04:12.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:12.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:12.336454+0000) 2024-02-19T23:04:13.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:13.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:13.336767+0000) 2024-02-19T23:04:13.470 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:04:13.474 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:04:14.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:14.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:14.337083+0000) 2024-02-19T23:04:15.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:15.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:15.337312+0000) 2024-02-19T23:04:16.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:16.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:16.337606+0000) 2024-02-19T23:04:17.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:17.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:17.337810+0000) 2024-02-19T23:04:18.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:18.333+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:18.338005+0000) 2024-02-19T23:04:19.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:19.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:19.338232+0000) 2024-02-19T23:04:20.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:20.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:20.338490+0000) 2024-02-19T23:04:21.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:21.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:21.338761+0000) 2024-02-19T23:04:22.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:22.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:22.339006+0000) 2024-02-19T23:04:23.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:23.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:23.339247+0000) 2024-02-19T23:04:24.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:24.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:24.339457+0000) 2024-02-19T23:04:25.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:25.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:25.339627+0000) 2024-02-19T23:04:26.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:26.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:26.339858+0000) 2024-02-19T23:04:27.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:27.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:27.340105+0000) 2024-02-19T23:04:28.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:28.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:28.340422+0000) 2024-02-19T23:04:29.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:29.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:29.340752+0000) 2024-02-19T23:04:30.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:30.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:30.340975+0000) 2024-02-19T23:04:31.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:31.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:31.341167+0000) 2024-02-19T23:04:32.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:32.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:32.341350+0000) 2024-02-19T23:04:33.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:33.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:33.341516+0000) 2024-02-19T23:04:34.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:34.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:34.341764+0000) 2024-02-19T23:04:35.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:35.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:35.342003+0000) 2024-02-19T23:04:36.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:36.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:36.342277+0000) 2024-02-19T23:04:37.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:37.337+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:37.342522+0000) 2024-02-19T23:04:38.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:38.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:38.342804+0000) 2024-02-19T23:04:39.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:39.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:39.343005+0000) 2024-02-19T23:04:40.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:40.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:40.343201+0000) 2024-02-19T23:04:41.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:41.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:41.343462+0000) 2024-02-19T23:04:42.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:42.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:42.343746+0000) 2024-02-19T23:04:43.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:43.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:43.343926+0000) 2024-02-19T23:04:43.535 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:04:43.539 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:04:44.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:44.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:44.344198+0000) 2024-02-19T23:04:45.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:45.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:45.344429+0000) 2024-02-19T23:04:46.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:46.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:46.344633+0000) 2024-02-19T23:04:47.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:47.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:47.344846+0000) 2024-02-19T23:04:48.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:48.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:48.345120+0000) 2024-02-19T23:04:49.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:49.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:49.345328+0000) 2024-02-19T23:04:50.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:50.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:50.345585+0000) 2024-02-19T23:04:51.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:51.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:51.345835+0000) 2024-02-19T23:04:52.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:52.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:52.346063+0000) 2024-02-19T23:04:53.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:53.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:53.346314+0000) 2024-02-19T23:04:54.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:54.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:54.346549+0000) 2024-02-19T23:04:55.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:55.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:55.346773+0000) 2024-02-19T23:04:56.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:56.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:56.347054+0000) 2024-02-19T23:04:57.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:57.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:57.347319+0000) 2024-02-19T23:04:58.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:58.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:58.347596+0000) 2024-02-19T23:04:59.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:04:59.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:04:59.347868+0000) 2024-02-19T23:05:00.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:00.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:00.348127+0000) 2024-02-19T23:05:01.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:01.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:01.348372+0000) 2024-02-19T23:05:02.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:02.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:02.348555+0000) 2024-02-19T23:05:03.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:03.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:03.348814+0000) 2024-02-19T23:05:04.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:04.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:04.349043+0000) 2024-02-19T23:05:05.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:05.344+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:05.349225+0000) 2024-02-19T23:05:06.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:06.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:06.349477+0000) 2024-02-19T23:05:07.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:07.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:07.349678+0000) 2024-02-19T23:05:08.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:08.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:08.349927+0000) 2024-02-19T23:05:09.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:09.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:09.350187+0000) 2024-02-19T23:05:10.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:10.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:10.350469+0000) 2024-02-19T23:05:11.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:11.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:11.350740+0000) 2024-02-19T23:05:12.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:12.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:12.350934+0000) 2024-02-19T23:05:13.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:13.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:13.351190+0000) 2024-02-19T23:05:13.656 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:05:13.660 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:05:14.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:14.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:14.351406+0000) 2024-02-19T23:05:15.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:15.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:15.351651+0000) 2024-02-19T23:05:16.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:16.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:16.351790+0000) 2024-02-19T23:05:17.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:17.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:17.352062+0000) 2024-02-19T23:05:18.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:18.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:18.352244+0000) 2024-02-19T23:05:19.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:19.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:19.352481+0000) 2024-02-19T23:05:20.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:20.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:20.352690+0000) 2024-02-19T23:05:21.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:21.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:21.352929+0000) 2024-02-19T23:05:22.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:22.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:22.353118+0000) 2024-02-19T23:05:23.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:23.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:23.353375+0000) 2024-02-19T23:05:24.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:24.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:24.353554+0000) 2024-02-19T23:05:25.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:25.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:25.353803+0000) 2024-02-19T23:05:26.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:26.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:26.354058+0000) 2024-02-19T23:05:27.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:27.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:27.354309+0000) 2024-02-19T23:05:28.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:28.348+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:28.354495+0000) 2024-02-19T23:05:29.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:29.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:29.354743+0000) 2024-02-19T23:05:30.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:30.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:30.354947+0000) 2024-02-19T23:05:31.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:31.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:31.355133+0000) 2024-02-19T23:05:32.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:32.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:32.355334+0000) 2024-02-19T23:05:33.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:33.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:33.355592+0000) 2024-02-19T23:05:34.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:34.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:34.355869+0000) 2024-02-19T23:05:35.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:35.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:35.356086+0000) 2024-02-19T23:05:36.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:36.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:36.356264+0000) 2024-02-19T23:05:37.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:37.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:37.356454+0000) 2024-02-19T23:05:38.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:38.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:38.356665+0000) 2024-02-19T23:05:39.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:39.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:39.356834+0000) 2024-02-19T23:05:40.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:40.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:40.357112+0000) 2024-02-19T23:05:41.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:41.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:41.357371+0000) 2024-02-19T23:05:42.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:42.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:42.357605+0000) 2024-02-19T23:05:43.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:43.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:43.357821+0000) 2024-02-19T23:05:43.979 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:05:43.983 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:05:44.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:44.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:44.358059+0000) 2024-02-19T23:05:45.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:45.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:45.358303+0000) 2024-02-19T23:05:46.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:46.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:46.358474+0000) 2024-02-19T23:05:47.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:47.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:47.358696+0000) 2024-02-19T23:05:48.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:48.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:48.358927+0000) 2024-02-19T23:05:49.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:49.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:49.359143+0000) 2024-02-19T23:05:50.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:50.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:50.359345+0000) 2024-02-19T23:05:51.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:51.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:51.359596+0000) 2024-02-19T23:05:52.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:52.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:52.359735+0000) 2024-02-19T23:05:53.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:53.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:53.359986+0000) 2024-02-19T23:05:54.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:54.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:54.360232+0000) 2024-02-19T23:05:55.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:55.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:55.360420+0000) 2024-02-19T23:05:56.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:56.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:56.360667+0000) 2024-02-19T23:05:57.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:57.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:57.360922+0000) 2024-02-19T23:05:58.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:58.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:58.361141+0000) 2024-02-19T23:05:59.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:05:59.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:05:59.361391+0000) 2024-02-19T23:06:00.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:00.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:00.361688+0000) 2024-02-19T23:06:01.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:01.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:01.361873+0000) 2024-02-19T23:06:02.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:02.355+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:02.362079+0000) 2024-02-19T23:06:03.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:03.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:03.362323+0000) 2024-02-19T23:06:04.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:04.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:04.362570+0000) 2024-02-19T23:06:05.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:05.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:05.362768+0000) 2024-02-19T23:06:06.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:06.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:06.362976+0000) 2024-02-19T23:06:07.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:07.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:07.363172+0000) 2024-02-19T23:06:08.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:08.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:08.363454+0000) 2024-02-19T23:06:09.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:09.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:09.363690+0000) 2024-02-19T23:06:10.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:10.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:10.363952+0000) 2024-02-19T23:06:11.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:11.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:11.364188+0000) 2024-02-19T23:06:12.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:12.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:12.364438+0000) 2024-02-19T23:06:13.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:13.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:13.364653+0000) 2024-02-19T23:06:14.135 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:06:14.138 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:06:14.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:14.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:14.364939+0000) 2024-02-19T23:06:15.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:15.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:15.365155+0000) 2024-02-19T23:06:16.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:16.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:16.365435+0000) 2024-02-19T23:06:17.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:17.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:17.365686+0000) 2024-02-19T23:06:18.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:18.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:18.365915+0000) 2024-02-19T23:06:19.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:19.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:19.366195+0000) 2024-02-19T23:06:20.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:20.359+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:20.366467+0000) 2024-02-19T23:06:21.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:21.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:21.366780+0000) 2024-02-19T23:06:22.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:22.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:22.367071+0000) 2024-02-19T23:06:23.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:23.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:23.367320+0000) 2024-02-19T23:06:24.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:24.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:24.367528+0000) 2024-02-19T23:06:25.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:25.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:25.367801+0000) 2024-02-19T23:06:26.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:26.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:26.368094+0000) 2024-02-19T23:06:27.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:27.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:27.368308+0000) 2024-02-19T23:06:28.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:28.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:28.368541+0000) 2024-02-19T23:06:29.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:29.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:29.368762+0000) 2024-02-19T23:06:30.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:30.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:30.369022+0000) 2024-02-19T23:06:31.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:31.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:31.369272+0000) 2024-02-19T23:06:32.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:32.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:32.369522+0000) 2024-02-19T23:06:33.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:33.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:33.369763+0000) 2024-02-19T23:06:34.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:34.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:34.370037+0000) 2024-02-19T23:06:35.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:35.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:35.370291+0000) 2024-02-19T23:06:36.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:36.363+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:36.370538+0000) 2024-02-19T23:06:37.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:37.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:37.370770+0000) 2024-02-19T23:06:38.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:38.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:38.370908+0000) 2024-02-19T23:06:39.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:39.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:39.371115+0000) 2024-02-19T23:06:40.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:40.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:40.371371+0000) 2024-02-19T23:06:41.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:41.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:41.371622+0000) 2024-02-19T23:06:42.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:42.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:42.371899+0000) 2024-02-19T23:06:43.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:43.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:43.372101+0000) 2024-02-19T23:06:44.282 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:06:44.287 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:06:44.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:44.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:44.372276+0000) 2024-02-19T23:06:45.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:45.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:45.372554+0000) 2024-02-19T23:06:46.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:46.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:46.372743+0000) 2024-02-19T23:06:47.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:47.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:47.372912+0000) 2024-02-19T23:06:48.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:48.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:48.373170+0000) 2024-02-19T23:06:49.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:49.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:49.373417+0000) 2024-02-19T23:06:50.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:50.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:50.373604+0000) 2024-02-19T23:06:51.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:51.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:51.373811+0000) 2024-02-19T23:06:52.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:52.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:52.373997+0000) 2024-02-19T23:06:53.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:53.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:53.374267+0000) 2024-02-19T23:06:54.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:54.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:54.374538+0000) 2024-02-19T23:06:55.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:55.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:55.374794+0000) 2024-02-19T23:06:56.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:56.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:56.375020+0000) 2024-02-19T23:06:57.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:57.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:57.375257+0000) 2024-02-19T23:06:58.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:58.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:58.375488+0000) 2024-02-19T23:06:59.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:06:59.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:06:59.375739+0000) 2024-02-19T23:07:00.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:00.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:00.375986+0000) 2024-02-19T23:07:01.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:01.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:01.376226+0000) 2024-02-19T23:07:02.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:02.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:02.376423+0000) 2024-02-19T23:07:03.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:03.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:03.376665+0000) 2024-02-19T23:07:04.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:04.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:04.376915+0000) 2024-02-19T23:07:05.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:05.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:05.377167+0000) 2024-02-19T23:07:06.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:06.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:06.377408+0000) 2024-02-19T23:07:07.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:07.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:07.377552+0000) 2024-02-19T23:07:08.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:08.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:08.377741+0000) 2024-02-19T23:07:09.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:09.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:09.377948+0000) 2024-02-19T23:07:10.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:10.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:10.378138+0000) 2024-02-19T23:07:11.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:11.370+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:11.378380+0000) 2024-02-19T23:07:12.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:12.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:12.378556+0000) 2024-02-19T23:07:13.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:13.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:13.378827+0000) 2024-02-19T23:07:14.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:14.374+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:14.378965+0000) 2024-02-19T23:07:14.405 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:07:14.409 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:07:15.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:15.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:15.379155+0000) 2024-02-19T23:07:16.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:16.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:16.379394+0000) 2024-02-19T23:07:17.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:17.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:17.379673+0000) 2024-02-19T23:07:18.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:18.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:18.379884+0000) 2024-02-19T23:07:19.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:19.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:19.380122+0000) 2024-02-19T23:07:20.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:20.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:20.380352+0000) 2024-02-19T23:07:21.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:21.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:21.380610+0000) 2024-02-19T23:07:22.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:22.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:22.380824+0000) 2024-02-19T23:07:23.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:23.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:23.381019+0000) 2024-02-19T23:07:24.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:24.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:24.381256+0000) 2024-02-19T23:07:25.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:25.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:25.381478+0000) 2024-02-19T23:07:26.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:26.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:26.381735+0000) 2024-02-19T23:07:27.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:27.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:27.381942+0000) 2024-02-19T23:07:28.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:28.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:28.382133+0000) 2024-02-19T23:07:29.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:29.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:29.382393+0000) 2024-02-19T23:07:30.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:30.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:30.382759+0000) 2024-02-19T23:07:31.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:31.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:31.382955+0000) 2024-02-19T23:07:32.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:32.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:32.383176+0000) 2024-02-19T23:07:33.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:33.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:33.383440+0000) 2024-02-19T23:07:34.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:34.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:34.383699+0000) 2024-02-19T23:07:35.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:35.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:35.383936+0000) 2024-02-19T23:07:36.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:36.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:36.384189+0000) 2024-02-19T23:07:37.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:37.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:37.384389+0000) 2024-02-19T23:07:38.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:38.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:38.384572+0000) 2024-02-19T23:07:39.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:39.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:39.384837+0000) 2024-02-19T23:07:40.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:40.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:40.385023+0000) 2024-02-19T23:07:41.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:41.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:41.385235+0000) 2024-02-19T23:07:42.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:42.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:42.385376+0000) 2024-02-19T23:07:43.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:43.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:43.385627+0000) 2024-02-19T23:07:44.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:44.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:44.385829+0000) 2024-02-19T23:07:44.494 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:07:44.497 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:07:45.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:45.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:45.386115+0000) 2024-02-19T23:07:46.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:46.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:46.386311+0000) 2024-02-19T23:07:47.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:47.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:47.386522+0000) 2024-02-19T23:07:48.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:48.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:48.386810+0000) 2024-02-19T23:07:49.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:49.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:49.387073+0000) 2024-02-19T23:07:50.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:50.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:50.387277+0000) 2024-02-19T23:07:51.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:51.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:51.387531+0000) 2024-02-19T23:07:52.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:52.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:52.387786+0000) 2024-02-19T23:07:53.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:53.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:53.387963+0000) 2024-02-19T23:07:54.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:54.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:54.388112+0000) 2024-02-19T23:07:55.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:55.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:55.388311+0000) 2024-02-19T23:07:56.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:56.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:56.388568+0000) 2024-02-19T23:07:57.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:57.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:57.388750+0000) 2024-02-19T23:07:58.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:58.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:58.388991+0000) 2024-02-19T23:07:59.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:07:59.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:07:59.389214+0000) 2024-02-19T23:08:00.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:00.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:00.389421+0000) 2024-02-19T23:08:01.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:01.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:01.389666+0000) 2024-02-19T23:08:02.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:02.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:02.389910+0000) 2024-02-19T23:08:03.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:03.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:03.390148+0000) 2024-02-19T23:08:04.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:04.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:04.390427+0000) 2024-02-19T23:08:05.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:05.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:05.390669+0000) 2024-02-19T23:08:06.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:06.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:06.390914+0000) 2024-02-19T23:08:07.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:07.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:07.391121+0000) 2024-02-19T23:08:08.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:08.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:08.391383+0000) 2024-02-19T23:08:09.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:09.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:09.391639+0000) 2024-02-19T23:08:10.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:10.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:10.391899+0000) 2024-02-19T23:08:11.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:11.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:11.392140+0000) 2024-02-19T23:08:12.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:12.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:12.392376+0000) 2024-02-19T23:08:13.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:13.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:13.392653+0000) 2024-02-19T23:08:14.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:14.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:14.392868+0000) 2024-02-19T23:08:14.612 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:08:14.615 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:08:15.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:15.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:15.393060+0000) 2024-02-19T23:08:16.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:16.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:16.393244+0000) 2024-02-19T23:08:17.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:17.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:17.393486+0000) 2024-02-19T23:08:18.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:18.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:18.393726+0000) 2024-02-19T23:08:19.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:19.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:19.393969+0000) 2024-02-19T23:08:20.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:20.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:20.394218+0000) 2024-02-19T23:08:21.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:21.389+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:21.394446+0000) 2024-02-19T23:08:22.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:22.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:22.394748+0000) 2024-02-19T23:08:23.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:23.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:23.394936+0000) 2024-02-19T23:08:24.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:24.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:24.395075+0000) 2024-02-19T23:08:25.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:25.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:25.395249+0000) 2024-02-19T23:08:26.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:26.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:26.395516+0000) 2024-02-19T23:08:27.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:27.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:27.395753+0000) 2024-02-19T23:08:28.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:28.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:28.395990+0000) 2024-02-19T23:08:29.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:29.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:29.396193+0000) 2024-02-19T23:08:30.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:30.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:30.396357+0000) 2024-02-19T23:08:31.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:31.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:31.396551+0000) 2024-02-19T23:08:32.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:32.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:32.396729+0000) 2024-02-19T23:08:33.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:33.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:33.396913+0000) 2024-02-19T23:08:34.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:34.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:34.397095+0000) 2024-02-19T23:08:35.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:35.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:35.397254+0000) 2024-02-19T23:08:36.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:36.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:36.397432+0000) 2024-02-19T23:08:37.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:37.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:37.397653+0000) 2024-02-19T23:08:38.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:38.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:38.397843+0000) 2024-02-19T23:08:39.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:39.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:39.398020+0000) 2024-02-19T23:08:40.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:40.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:40.398177+0000) 2024-02-19T23:08:41.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:41.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:41.398342+0000) 2024-02-19T23:08:42.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:42.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:42.398468+0000) 2024-02-19T23:08:43.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:43.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:43.398610+0000) 2024-02-19T23:08:44.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:44.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:44.398773+0000) 2024-02-19T23:08:44.745 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:08:44.749 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:08:45.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:45.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:45.398897+0000) 2024-02-19T23:08:46.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:46.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:46.399060+0000) 2024-02-19T23:08:47.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:47.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:47.399309+0000) 2024-02-19T23:08:48.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:48.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:48.399657+0000) 2024-02-19T23:08:49.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:49.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:49.399917+0000) 2024-02-19T23:08:50.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:50.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:50.400135+0000) 2024-02-19T23:08:51.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:51.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:51.400412+0000) 2024-02-19T23:08:52.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:52.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:52.400619+0000) 2024-02-19T23:08:53.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:53.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:53.400855+0000) 2024-02-19T23:08:54.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:54.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:54.401104+0000) 2024-02-19T23:08:55.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:55.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:55.401274+0000) 2024-02-19T23:08:56.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:56.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:56.401522+0000) 2024-02-19T23:08:57.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:57.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:57.401746+0000) 2024-02-19T23:08:58.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:58.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:58.401972+0000) 2024-02-19T23:08:59.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:08:59.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:08:59.402223+0000) 2024-02-19T23:09:00.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:00.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:00.402472+0000) 2024-02-19T23:09:01.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:01.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:01.402752+0000) 2024-02-19T23:09:02.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:02.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:02.402970+0000) 2024-02-19T23:09:03.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:03.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:03.403189+0000) 2024-02-19T23:09:04.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:04.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:04.403450+0000) 2024-02-19T23:09:05.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:05.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:05.403641+0000) 2024-02-19T23:09:06.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:06.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:06.403860+0000) 2024-02-19T23:09:07.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:07.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:07.404121+0000) 2024-02-19T23:09:08.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:08.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:08.404382+0000) 2024-02-19T23:09:09.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:09.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:09.404642+0000) 2024-02-19T23:09:10.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:10.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:10.404900+0000) 2024-02-19T23:09:11.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:11.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:11.405124+0000) 2024-02-19T23:09:12.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:12.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:12.405377+0000) 2024-02-19T23:09:13.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:13.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:13.405652+0000) 2024-02-19T23:09:14.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:14.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:14.405896+0000) 2024-02-19T23:09:14.816 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:09:14.819 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:09:15.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:15.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:15.406094+0000) 2024-02-19T23:09:16.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:16.400+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:16.406339+0000) 2024-02-19T23:09:17.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:17.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:17.406519+0000) 2024-02-19T23:09:18.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:18.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:18.406846+0000) 2024-02-19T23:09:19.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:19.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:19.407122+0000) 2024-02-19T23:09:20.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:20.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:20.407364+0000) 2024-02-19T23:09:21.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:21.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:21.407616+0000) 2024-02-19T23:09:22.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:22.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:22.407843+0000) 2024-02-19T23:09:23.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:23.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:23.408051+0000) 2024-02-19T23:09:24.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:24.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:24.408337+0000) 2024-02-19T23:09:25.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:25.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:25.408535+0000) 2024-02-19T23:09:26.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:26.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:26.408766+0000) 2024-02-19T23:09:27.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:27.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:27.409010+0000) 2024-02-19T23:09:28.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:28.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:28.409253+0000) 2024-02-19T23:09:29.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:29.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:29.409476+0000) 2024-02-19T23:09:30.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:30.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:30.409661+0000) 2024-02-19T23:09:31.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:31.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:31.409909+0000) 2024-02-19T23:09:32.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:32.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:32.410159+0000) 2024-02-19T23:09:33.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:33.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:33.410332+0000) 2024-02-19T23:09:34.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:34.404+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:34.410585+0000) 2024-02-19T23:09:35.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:35.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:35.410775+0000) 2024-02-19T23:09:36.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:36.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:36.410981+0000) 2024-02-19T23:09:37.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:37.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:37.411120+0000) 2024-02-19T23:09:38.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:38.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:38.411282+0000) 2024-02-19T23:09:39.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:39.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:39.411539+0000) 2024-02-19T23:09:40.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:40.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:40.411772+0000) 2024-02-19T23:09:41.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:41.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:41.411949+0000) 2024-02-19T23:09:42.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:42.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:42.412112+0000) 2024-02-19T23:09:43.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:43.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:43.412371+0000) 2024-02-19T23:09:44.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:44.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:44.412634+0000) 2024-02-19T23:09:44.878 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:09:44.881 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:09:45.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:45.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:45.412849+0000) 2024-02-19T23:09:46.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:46.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:46.413101+0000) 2024-02-19T23:09:47.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:47.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:47.413341+0000) 2024-02-19T23:09:48.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:48.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:48.413542+0000) 2024-02-19T23:09:49.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:49.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:49.413722+0000) 2024-02-19T23:09:50.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:50.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:50.413951+0000) 2024-02-19T23:09:51.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:51.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:51.414129+0000) 2024-02-19T23:09:52.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:52.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:52.414381+0000) 2024-02-19T23:09:53.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:53.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:53.414619+0000) 2024-02-19T23:09:54.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:54.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:54.414826+0000) 2024-02-19T23:09:55.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:55.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:55.415065+0000) 2024-02-19T23:09:56.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:56.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:56.415262+0000) 2024-02-19T23:09:57.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:57.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:57.415475+0000) 2024-02-19T23:09:58.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:58.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:58.415717+0000) 2024-02-19T23:09:59.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:09:59.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:09:59.415957+0000) 2024-02-19T23:10:00.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:00.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:00.416211+0000) 2024-02-19T23:10:01.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:01.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:01.416485+0000) 2024-02-19T23:10:02.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:02.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:02.416733+0000) 2024-02-19T23:10:03.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:03.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:03.417011+0000) 2024-02-19T23:10:04.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:04.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:04.417264+0000) 2024-02-19T23:10:05.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:05.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:05.417506+0000) 2024-02-19T23:10:06.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:06.411+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:06.417765+0000) 2024-02-19T23:10:07.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:07.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:07.418020+0000) 2024-02-19T23:10:08.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:08.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:08.418245+0000) 2024-02-19T23:10:09.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:09.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:09.418494+0000) 2024-02-19T23:10:10.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:10.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:10.418816+0000) 2024-02-19T23:10:11.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:11.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:11.418997+0000) 2024-02-19T23:10:12.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:12.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:12.419247+0000) 2024-02-19T23:10:13.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:13.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:13.419513+0000) 2024-02-19T23:10:14.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:14.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:14.419742+0000) 2024-02-19T23:10:15.002 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:10:15.005 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:10:15.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:15.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:15.419973+0000) 2024-02-19T23:10:16.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:16.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:16.420209+0000) 2024-02-19T23:10:17.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:17.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:17.420445+0000) 2024-02-19T23:10:18.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:18.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:18.420688+0000) 2024-02-19T23:10:19.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:19.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:19.420886+0000) 2024-02-19T23:10:20.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:20.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:20.421116+0000) 2024-02-19T23:10:21.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:21.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:21.421359+0000) 2024-02-19T23:10:22.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:22.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:22.421622+0000) 2024-02-19T23:10:23.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:23.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:23.421755+0000) 2024-02-19T23:10:24.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:24.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:24.421959+0000) 2024-02-19T23:10:25.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:25.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:25.422183+0000) 2024-02-19T23:10:26.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:26.415+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:26.422444+0000) 2024-02-19T23:10:27.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:27.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:27.422694+0000) 2024-02-19T23:10:28.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:28.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:28.422907+0000) 2024-02-19T23:10:29.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:29.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:29.423087+0000) 2024-02-19T23:10:30.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:30.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:30.423360+0000) 2024-02-19T23:10:31.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:31.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:31.423550+0000) 2024-02-19T23:10:32.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:32.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:32.423788+0000) 2024-02-19T23:10:33.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:33.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:33.424020+0000) 2024-02-19T23:10:34.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:34.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:34.424217+0000) 2024-02-19T23:10:35.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:35.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:35.424458+0000) 2024-02-19T23:10:36.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:36.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:36.424697+0000) 2024-02-19T23:10:37.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:37.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:37.424895+0000) 2024-02-19T23:10:38.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:38.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:38.425119+0000) 2024-02-19T23:10:39.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:39.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:39.425292+0000) 2024-02-19T23:10:40.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:40.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:40.425554+0000) 2024-02-19T23:10:41.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:41.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:41.425787+0000) 2024-02-19T23:10:42.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:42.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:42.425961+0000) 2024-02-19T23:10:43.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:43.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:43.426177+0000) 2024-02-19T23:10:44.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:44.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:44.426409+0000) 2024-02-19T23:10:45.064 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:10:45.067 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:10:45.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:45.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:45.426597+0000) 2024-02-19T23:10:46.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:46.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:46.426874+0000) 2024-02-19T23:10:47.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:47.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:47.427070+0000) 2024-02-19T23:10:48.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:48.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:48.427358+0000) 2024-02-19T23:10:49.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:49.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:49.427590+0000) 2024-02-19T23:10:50.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:50.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:50.427800+0000) 2024-02-19T23:10:51.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:51.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:51.427982+0000) 2024-02-19T23:10:52.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:52.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:52.428190+0000) 2024-02-19T23:10:53.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:53.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:53.428396+0000) 2024-02-19T23:10:54.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:54.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:54.428593+0000) 2024-02-19T23:10:55.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:55.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:55.428840+0000) 2024-02-19T23:10:56.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:56.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:56.429065+0000) 2024-02-19T23:10:57.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:57.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:57.429321+0000) 2024-02-19T23:10:58.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:58.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:58.429529+0000) 2024-02-19T23:10:59.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:10:59.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:10:59.429778+0000) 2024-02-19T23:11:00.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:00.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:00.430030+0000) 2024-02-19T23:11:01.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:01.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:01.430239+0000) 2024-02-19T23:11:02.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:02.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:02.430489+0000) 2024-02-19T23:11:03.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:03.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:03.430700+0000) 2024-02-19T23:11:04.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:04.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:04.430907+0000) 2024-02-19T23:11:05.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:05.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:05.431153+0000) 2024-02-19T23:11:06.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:06.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:06.431334+0000) 2024-02-19T23:11:07.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:07.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:07.431622+0000) 2024-02-19T23:11:08.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:08.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:08.431834+0000) 2024-02-19T23:11:09.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:09.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:09.432107+0000) 2024-02-19T23:11:10.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:10.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:10.432287+0000) 2024-02-19T23:11:11.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:11.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:11.432516+0000) 2024-02-19T23:11:12.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:12.426+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:12.432725+0000) 2024-02-19T23:11:13.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:13.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:13.432984+0000) 2024-02-19T23:11:14.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:14.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:14.433164+0000) 2024-02-19T23:11:15.196 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:11:15.200 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:11:15.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:15.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:15.433353+0000) 2024-02-19T23:11:16.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:16.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:16.433606+0000) 2024-02-19T23:11:17.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:17.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:17.433819+0000) 2024-02-19T23:11:18.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:18.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:18.434063+0000) 2024-02-19T23:11:19.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:19.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:19.434322+0000) 2024-02-19T23:11:20.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:20.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:20.434576+0000) 2024-02-19T23:11:21.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:21.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:21.434822+0000) 2024-02-19T23:11:22.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:22.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:22.435070+0000) 2024-02-19T23:11:23.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:23.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:23.435261+0000) 2024-02-19T23:11:24.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:24.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:24.435503+0000) 2024-02-19T23:11:25.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:25.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:25.435742+0000) 2024-02-19T23:11:26.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:26.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:26.435922+0000) 2024-02-19T23:11:27.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:27.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:27.436168+0000) 2024-02-19T23:11:28.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:28.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:28.436372+0000) 2024-02-19T23:11:29.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:29.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:29.436643+0000) 2024-02-19T23:11:30.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:30.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:30.436971+0000) 2024-02-19T23:11:31.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:31.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:31.437225+0000) 2024-02-19T23:11:32.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:32.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:32.437475+0000) 2024-02-19T23:11:33.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:33.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:33.437722+0000) 2024-02-19T23:11:34.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:34.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:34.437886+0000) 2024-02-19T23:11:35.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:35.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:35.438049+0000) 2024-02-19T23:11:36.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:36.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:36.438203+0000) 2024-02-19T23:11:37.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:37.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:37.438444+0000) 2024-02-19T23:11:38.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:38.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:38.438615+0000) 2024-02-19T23:11:39.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:39.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:39.438835+0000) 2024-02-19T23:11:40.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:40.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:40.438968+0000) 2024-02-19T23:11:41.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:41.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:41.439164+0000) 2024-02-19T23:11:42.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:42.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:42.439340+0000) 2024-02-19T23:11:43.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:43.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:43.439510+0000) 2024-02-19T23:11:44.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:44.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:44.439650+0000) 2024-02-19T23:11:45.326 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:11:45.330 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:11:45.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:45.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:45.439792+0000) 2024-02-19T23:11:46.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:46.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:46.440036+0000) 2024-02-19T23:11:47.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:47.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:47.440278+0000) 2024-02-19T23:11:48.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:48.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:48.440543+0000) 2024-02-19T23:11:49.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:49.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:49.440779+0000) 2024-02-19T23:11:50.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:50.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:50.440984+0000) 2024-02-19T23:11:51.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:51.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:51.441175+0000) 2024-02-19T23:11:52.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:52.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:52.441406+0000) 2024-02-19T23:11:53.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:53.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:53.441664+0000) 2024-02-19T23:11:54.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:54.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:54.441837+0000) 2024-02-19T23:11:55.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:55.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:55.442069+0000) 2024-02-19T23:11:56.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:56.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:56.442231+0000) 2024-02-19T23:11:57.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:57.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:57.442431+0000) 2024-02-19T23:11:58.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:58.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:58.442588+0000) 2024-02-19T23:11:59.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:11:59.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:11:59.442855+0000) 2024-02-19T23:12:00.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:00.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:00.443075+0000) 2024-02-19T23:12:01.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:01.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:01.443253+0000) 2024-02-19T23:12:02.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:02.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:02.443391+0000) 2024-02-19T23:12:03.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:03.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:03.443628+0000) 2024-02-19T23:12:04.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:04.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:04.443884+0000) 2024-02-19T23:12:05.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:05.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:05.444062+0000) 2024-02-19T23:12:06.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:06.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:06.444289+0000) 2024-02-19T23:12:07.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:07.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:07.444467+0000) 2024-02-19T23:12:08.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:08.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:08.444645+0000) 2024-02-19T23:12:09.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:09.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:09.444864+0000) 2024-02-19T23:12:10.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:10.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:10.445012+0000) 2024-02-19T23:12:11.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:11.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:11.445134+0000) 2024-02-19T23:12:12.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:12.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:12.445355+0000) 2024-02-19T23:12:13.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:13.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:13.445504+0000) 2024-02-19T23:12:14.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:14.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:14.445643+0000) 2024-02-19T23:12:15.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:15.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:15.445792+0000) 2024-02-19T23:12:15.457 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:12:15.461 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:12:16.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:16.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:16.446014+0000) 2024-02-19T23:12:17.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:17.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:17.446190+0000) 2024-02-19T23:12:18.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:18.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:18.446371+0000) 2024-02-19T23:12:19.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:19.441+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:19.446544+0000) 2024-02-19T23:12:20.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:20.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:20.446824+0000) 2024-02-19T23:12:21.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:21.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:21.447048+0000) 2024-02-19T23:12:22.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:22.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:22.447262+0000) 2024-02-19T23:12:23.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:23.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:23.447500+0000) 2024-02-19T23:12:24.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:24.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:24.447679+0000) 2024-02-19T23:12:25.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:25.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:25.447907+0000) 2024-02-19T23:12:26.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:26.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:26.448153+0000) 2024-02-19T23:12:27.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:27.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:27.448424+0000) 2024-02-19T23:12:28.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:28.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:28.448665+0000) 2024-02-19T23:12:29.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:29.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:29.448952+0000) 2024-02-19T23:12:30.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:30.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:30.449197+0000) 2024-02-19T23:12:31.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:31.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:31.449439+0000) 2024-02-19T23:12:32.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:32.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:32.449643+0000) 2024-02-19T23:12:33.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:33.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:33.449829+0000) 2024-02-19T23:12:34.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:34.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:34.450052+0000) 2024-02-19T23:12:35.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:35.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:35.450286+0000) 2024-02-19T23:12:36.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:36.445+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:36.450524+0000) 2024-02-19T23:12:37.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:37.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:37.450701+0000) 2024-02-19T23:12:38.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:38.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:38.450966+0000) 2024-02-19T23:12:39.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:39.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:39.451229+0000) 2024-02-19T23:12:40.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:40.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:40.451425+0000) 2024-02-19T23:12:41.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:41.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:41.451679+0000) 2024-02-19T23:12:42.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:42.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:42.451855+0000) 2024-02-19T23:12:43.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:43.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:43.452048+0000) 2024-02-19T23:12:44.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:44.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:44.452271+0000) 2024-02-19T23:12:45.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:45.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:45.452502+0000) 2024-02-19T23:12:45.555 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:12:45.560 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:12:46.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:46.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:46.452650+0000) 2024-02-19T23:12:47.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:47.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:47.452886+0000) 2024-02-19T23:12:48.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:48.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:48.453063+0000) 2024-02-19T23:12:49.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:49.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:49.453277+0000) 2024-02-19T23:12:50.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:50.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:50.453485+0000) 2024-02-19T23:12:51.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:51.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:51.453663+0000) 2024-02-19T23:12:52.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:52.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:52.453824+0000) 2024-02-19T23:12:53.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:53.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:53.453976+0000) 2024-02-19T23:12:54.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:54.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:54.454228+0000) 2024-02-19T23:12:55.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:55.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:55.454427+0000) 2024-02-19T23:12:56.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:56.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:56.454662+0000) 2024-02-19T23:12:57.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:57.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:57.454904+0000) 2024-02-19T23:12:58.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:58.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:58.455075+0000) 2024-02-19T23:12:59.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:12:59.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:12:59.455236+0000) 2024-02-19T23:13:00.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:00.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:00.455424+0000) 2024-02-19T23:13:01.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:01.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:01.455681+0000) 2024-02-19T23:13:02.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:02.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:02.455916+0000) 2024-02-19T23:13:03.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:03.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:03.456124+0000) 2024-02-19T23:13:04.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:04.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:04.456363+0000) 2024-02-19T23:13:05.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:05.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:05.456630+0000) 2024-02-19T23:13:06.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:06.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:06.456829+0000) 2024-02-19T23:13:07.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:07.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:07.457056+0000) 2024-02-19T23:13:08.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:08.452+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:08.457200+0000) 2024-02-19T23:13:09.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:09.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:09.457359+0000) 2024-02-19T23:13:10.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:10.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:10.457608+0000) 2024-02-19T23:13:11.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:11.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:11.457798+0000) 2024-02-19T23:13:12.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:12.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:12.458042+0000) 2024-02-19T23:13:13.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:13.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:13.458298+0000) 2024-02-19T23:13:14.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:14.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:14.458477+0000) 2024-02-19T23:13:15.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:15.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:15.458645+0000) 2024-02-19T23:13:15.693 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:13:15.697 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:13:16.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:16.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:16.458907+0000) 2024-02-19T23:13:17.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:17.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:17.459084+0000) 2024-02-19T23:13:18.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:18.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:18.459252+0000) 2024-02-19T23:13:19.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:19.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:19.459489+0000) 2024-02-19T23:13:20.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:20.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:20.459776+0000) 2024-02-19T23:13:21.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:21.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:21.460031+0000) 2024-02-19T23:13:22.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:22.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:22.460288+0000) 2024-02-19T23:13:23.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:23.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:23.460469+0000) 2024-02-19T23:13:24.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:24.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:24.460656+0000) 2024-02-19T23:13:25.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:25.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:25.460908+0000) 2024-02-19T23:13:26.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:26.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:26.461159+0000) 2024-02-19T23:13:27.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:27.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:27.461337+0000) 2024-02-19T23:13:28.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:28.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:28.461585+0000) 2024-02-19T23:13:29.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:29.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:29.461792+0000) 2024-02-19T23:13:30.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:30.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:30.462018+0000) 2024-02-19T23:13:31.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:31.456+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:31.462245+0000) 2024-02-19T23:13:32.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:32.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:32.462450+0000) 2024-02-19T23:13:33.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:33.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:33.462695+0000) 2024-02-19T23:13:34.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:34.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:34.462869+0000) 2024-02-19T23:13:35.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:35.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:35.463126+0000) 2024-02-19T23:13:36.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:36.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:36.463372+0000) 2024-02-19T23:13:37.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:37.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:37.463623+0000) 2024-02-19T23:13:38.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:38.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:38.463823+0000) 2024-02-19T23:13:39.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:39.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:39.464089+0000) 2024-02-19T23:13:40.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:40.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:40.464368+0000) 2024-02-19T23:13:41.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:41.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:41.464618+0000) 2024-02-19T23:13:42.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:42.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:42.464878+0000) 2024-02-19T23:13:43.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:43.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:43.465112+0000) 2024-02-19T23:13:44.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:44.460+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:44.465290+0000) 2024-02-19T23:13:45.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:45.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:45.465557+0000) 2024-02-19T23:13:45.815 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:13:45.818 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:13:46.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:46.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:46.465802+0000) 2024-02-19T23:13:47.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:47.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:47.466037+0000) 2024-02-19T23:13:48.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:48.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:48.466288+0000) 2024-02-19T23:13:49.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:49.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:49.466493+0000) 2024-02-19T23:13:50.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:50.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:50.466774+0000) 2024-02-19T23:13:51.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:51.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:51.467037+0000) 2024-02-19T23:13:52.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:52.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:52.467282+0000) 2024-02-19T23:13:53.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:53.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:53.467533+0000) 2024-02-19T23:13:54.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:54.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:54.467745+0000) 2024-02-19T23:13:55.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:55.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:55.467929+0000) 2024-02-19T23:13:56.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:56.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:56.468114+0000) 2024-02-19T23:13:57.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:57.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:57.468299+0000) 2024-02-19T23:13:58.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:58.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:58.468580+0000) 2024-02-19T23:13:59.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:13:59.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:13:59.468833+0000) 2024-02-19T23:14:00.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:00.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:00.469087+0000) 2024-02-19T23:14:01.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:01.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:01.469288+0000) 2024-02-19T23:14:02.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:02.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:02.469568+0000) 2024-02-19T23:14:03.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:03.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:03.469783+0000) 2024-02-19T23:14:04.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:04.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:04.469965+0000) 2024-02-19T23:14:05.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:05.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:05.470293+0000) 2024-02-19T23:14:06.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:06.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:06.470473+0000) 2024-02-19T23:14:07.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:07.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:07.470660+0000) 2024-02-19T23:14:08.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:08.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:08.470868+0000) 2024-02-19T23:14:09.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:09.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:09.471052+0000) 2024-02-19T23:14:10.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:10.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:10.471335+0000) 2024-02-19T23:14:11.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:11.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:11.471613+0000) 2024-02-19T23:14:12.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:12.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:12.471807+0000) 2024-02-19T23:14:13.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:13.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:13.472019+0000) 2024-02-19T23:14:14.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:14.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:14.472268+0000) 2024-02-19T23:14:15.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:15.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:15.472526+0000) 2024-02-19T23:14:15.904 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:14:15.907 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:14:16.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:16.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:16.472764+0000) 2024-02-19T23:14:17.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:17.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:17.472954+0000) 2024-02-19T23:14:18.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:18.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:18.473201+0000) 2024-02-19T23:14:19.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:19.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:19.473412+0000) 2024-02-19T23:14:20.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:20.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:20.473696+0000) 2024-02-19T23:14:21.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:21.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:21.473880+0000) 2024-02-19T23:14:22.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:22.467+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:22.474082+0000) 2024-02-19T23:14:23.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:23.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:23.474282+0000) 2024-02-19T23:14:24.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:24.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:24.474466+0000) 2024-02-19T23:14:25.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:25.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:25.474743+0000) 2024-02-19T23:14:26.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:26.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:26.474888+0000) 2024-02-19T23:14:27.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:27.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:27.475016+0000) 2024-02-19T23:14:28.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:28.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:28.475180+0000) 2024-02-19T23:14:29.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:29.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:29.475404+0000) 2024-02-19T23:14:30.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:30.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:30.475667+0000) 2024-02-19T23:14:31.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:31.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:31.475954+0000) 2024-02-19T23:14:32.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:32.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:32.476252+0000) 2024-02-19T23:14:33.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:33.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:33.476443+0000) 2024-02-19T23:14:34.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:34.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:34.476629+0000) 2024-02-19T23:14:35.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:35.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:35.476864+0000) 2024-02-19T23:14:36.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:36.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:36.477099+0000) 2024-02-19T23:14:37.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:37.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:37.477352+0000) 2024-02-19T23:14:38.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:38.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:38.477546+0000) 2024-02-19T23:14:39.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:39.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:39.477781+0000) 2024-02-19T23:14:40.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:40.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:40.478055+0000) 2024-02-19T23:14:41.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:41.471+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:41.478308+0000) 2024-02-19T23:14:42.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:42.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:42.478531+0000) 2024-02-19T23:14:43.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:43.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:43.478780+0000) 2024-02-19T23:14:44.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:44.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:44.479000+0000) 2024-02-19T23:14:45.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:45.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:45.479260+0000) 2024-02-19T23:14:46.086 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:14:46.090 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:14:46.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:46.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:46.479524+0000) 2024-02-19T23:14:47.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:47.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:47.479801+0000) 2024-02-19T23:14:48.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:48.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:48.480004+0000) 2024-02-19T23:14:49.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:49.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:49.480182+0000) 2024-02-19T23:14:50.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:50.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:50.480414+0000) 2024-02-19T23:14:51.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:51.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:51.480608+0000) 2024-02-19T23:14:52.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:52.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:52.480797+0000) 2024-02-19T23:14:53.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:53.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:53.480945+0000) 2024-02-19T23:14:54.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:54.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:54.481110+0000) 2024-02-19T23:14:55.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:55.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:55.481355+0000) 2024-02-19T23:14:56.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:56.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:56.481612+0000) 2024-02-19T23:14:57.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:57.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:57.481908+0000) 2024-02-19T23:14:58.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:58.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:58.482134+0000) 2024-02-19T23:14:59.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:14:59.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:14:59.482304+0000) 2024-02-19T23:15:00.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:00.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:00.482587+0000) 2024-02-19T23:15:01.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:01.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:01.482839+0000) 2024-02-19T23:15:02.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:02.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:02.483104+0000) 2024-02-19T23:15:03.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:03.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:03.483309+0000) 2024-02-19T23:15:04.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:04.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:04.483525+0000) 2024-02-19T23:15:05.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:05.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:05.483814+0000) 2024-02-19T23:15:06.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:06.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:06.484064+0000) 2024-02-19T23:15:07.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:07.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:07.484318+0000) 2024-02-19T23:15:08.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:08.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:08.484538+0000) 2024-02-19T23:15:09.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:09.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:09.484674+0000) 2024-02-19T23:15:10.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:10.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:10.484915+0000) 2024-02-19T23:15:11.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:11.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:11.485172+0000) 2024-02-19T23:15:12.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:12.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:12.485371+0000) 2024-02-19T23:15:13.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:13.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:13.485592+0000) 2024-02-19T23:15:14.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:14.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:14.485839+0000) 2024-02-19T23:15:15.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:15.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:15.486076+0000) 2024-02-19T23:15:16.250 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:15:16.253 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:15:16.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:16.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:16.486350+0000) 2024-02-19T23:15:17.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:17.478+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:17.486622+0000) 2024-02-19T23:15:18.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:18.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:18.486878+0000) 2024-02-19T23:15:19.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:19.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:19.487088+0000) 2024-02-19T23:15:20.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:20.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:20.487384+0000) 2024-02-19T23:15:21.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:21.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:21.487613+0000) 2024-02-19T23:15:22.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:22.482+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:22.487865+0000) 2024-02-19T23:15:23.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:23.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:23.488094+0000) 2024-02-19T23:15:24.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:24.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:24.488376+0000) 2024-02-19T23:15:25.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:25.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:25.488600+0000) 2024-02-19T23:15:26.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:26.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:26.488781+0000) 2024-02-19T23:15:27.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:27.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:27.488992+0000) 2024-02-19T23:15:28.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:28.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:28.489163+0000) 2024-02-19T23:15:29.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:29.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:29.489356+0000) 2024-02-19T23:15:30.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:30.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:30.489598+0000) 2024-02-19T23:15:31.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:31.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:31.489790+0000) 2024-02-19T23:15:32.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:32.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:32.489995+0000) 2024-02-19T23:15:33.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:33.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:33.490230+0000) 2024-02-19T23:15:34.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:34.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:34.490442+0000) 2024-02-19T23:15:35.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:35.486+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:35.490668+0000) 2024-02-19T23:15:36.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:36.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:36.490922+0000) 2024-02-19T23:15:37.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:37.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:37.491124+0000) 2024-02-19T23:15:38.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:38.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:38.491327+0000) 2024-02-19T23:15:39.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:39.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:39.491524+0000) 2024-02-19T23:15:40.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:40.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:40.491808+0000) 2024-02-19T23:15:41.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:41.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:41.492016+0000) 2024-02-19T23:15:42.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:42.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:42.492282+0000) 2024-02-19T23:15:43.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:43.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:43.492568+0000) 2024-02-19T23:15:44.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:44.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:44.492813+0000) 2024-02-19T23:15:45.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:45.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:45.493000+0000) 2024-02-19T23:15:46.373 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:15:46.377 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:15:46.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:46.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:46.493199+0000) 2024-02-19T23:15:47.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:47.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:47.493457+0000) 2024-02-19T23:15:48.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:48.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:48.493702+0000) 2024-02-19T23:15:49.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:49.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:49.493949+0000) 2024-02-19T23:15:50.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:50.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:50.494214+0000) 2024-02-19T23:15:51.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:51.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:51.494454+0000) 2024-02-19T23:15:52.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:52.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:52.494652+0000) 2024-02-19T23:15:53.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:53.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:53.494887+0000) 2024-02-19T23:15:54.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:54.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:54.495125+0000) 2024-02-19T23:15:55.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:55.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:55.495289+0000) 2024-02-19T23:15:56.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:56.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:56.495503+0000) 2024-02-19T23:15:57.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:57.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:57.495702+0000) 2024-02-19T23:15:58.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:58.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:58.495937+0000) 2024-02-19T23:15:59.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:15:59.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:15:59.496188+0000) 2024-02-19T23:16:00.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:00.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:00.496405+0000) 2024-02-19T23:16:01.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:01.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:01.496648+0000) 2024-02-19T23:16:02.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:02.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:02.496831+0000) 2024-02-19T23:16:03.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:03.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:03.497041+0000) 2024-02-19T23:16:04.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:04.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:04.497246+0000) 2024-02-19T23:16:05.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:05.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:05.497489+0000) 2024-02-19T23:16:06.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:06.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:06.497767+0000) 2024-02-19T23:16:07.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:07.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:07.498036+0000) 2024-02-19T23:16:08.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:08.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:08.498241+0000) 2024-02-19T23:16:09.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:09.493+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:09.498446+0000) 2024-02-19T23:16:10.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:10.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:10.498649+0000) 2024-02-19T23:16:11.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:11.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:11.498912+0000) 2024-02-19T23:16:12.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:12.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:12.499057+0000) 2024-02-19T23:16:13.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:13.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:13.499325+0000) 2024-02-19T23:16:14.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:14.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:14.499572+0000) 2024-02-19T23:16:15.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:15.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:15.499846+0000) 2024-02-19T23:16:16.422 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:16:16.426 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:16:16.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:16.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:16.500128+0000) 2024-02-19T23:16:17.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:17.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:17.500307+0000) 2024-02-19T23:16:18.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:18.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:18.500528+0000) 2024-02-19T23:16:19.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:19.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:19.500784+0000) 2024-02-19T23:16:20.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:20.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:20.501031+0000) 2024-02-19T23:16:21.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:21.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:21.501266+0000) 2024-02-19T23:16:22.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:22.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:22.501466+0000) 2024-02-19T23:16:23.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:23.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:23.501691+0000) 2024-02-19T23:16:24.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:24.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:24.501957+0000) 2024-02-19T23:16:25.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:25.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:25.502154+0000) 2024-02-19T23:16:26.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:26.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:26.502392+0000) 2024-02-19T23:16:27.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:27.497+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:27.502678+0000) 2024-02-19T23:16:28.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:28.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:28.502884+0000) 2024-02-19T23:16:29.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:29.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:29.503067+0000) 2024-02-19T23:16:30.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:30.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:30.503251+0000) 2024-02-19T23:16:31.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:31.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:31.503509+0000) 2024-02-19T23:16:32.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:32.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:32.503699+0000) 2024-02-19T23:16:33.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:33.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:33.503927+0000) 2024-02-19T23:16:34.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:34.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:34.504112+0000) 2024-02-19T23:16:35.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:35.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:35.504355+0000) 2024-02-19T23:16:36.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:36.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:36.504575+0000) 2024-02-19T23:16:37.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:37.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:37.504826+0000) 2024-02-19T23:16:38.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:38.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:38.505085+0000) 2024-02-19T23:16:39.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:39.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:39.505255+0000) 2024-02-19T23:16:40.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:40.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:40.505477+0000) 2024-02-19T23:16:41.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:41.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:41.505651+0000) 2024-02-19T23:16:42.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:42.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:42.505908+0000) 2024-02-19T23:16:43.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:43.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:43.506120+0000) 2024-02-19T23:16:44.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:44.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:44.506351+0000) 2024-02-19T23:16:45.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:45.501+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:45.506616+0000) 2024-02-19T23:16:46.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:46.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:46.506927+0000) 2024-02-19T23:16:46.554 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:16:46.557 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:16:47.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:47.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:47.507164+0000) 2024-02-19T23:16:48.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:48.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:48.507439+0000) 2024-02-19T23:16:49.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:49.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:49.507754+0000) 2024-02-19T23:16:50.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:50.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:50.508041+0000) 2024-02-19T23:16:51.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:51.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:51.508323+0000) 2024-02-19T23:16:52.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:52.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:52.508612+0000) 2024-02-19T23:16:53.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:53.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:53.508839+0000) 2024-02-19T23:16:54.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:54.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:54.509131+0000) 2024-02-19T23:16:55.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:55.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:55.509358+0000) 2024-02-19T23:16:56.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:56.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:56.509653+0000) 2024-02-19T23:16:57.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:57.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:57.509933+0000) 2024-02-19T23:16:58.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:58.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:58.510215+0000) 2024-02-19T23:16:59.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:16:59.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:16:59.510514+0000) 2024-02-19T23:17:00.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:00.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:00.510789+0000) 2024-02-19T23:17:01.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:01.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:01.510978+0000) 2024-02-19T23:17:02.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:02.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:02.511130+0000) 2024-02-19T23:17:03.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:03.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:03.511376+0000) 2024-02-19T23:17:04.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:04.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:04.511577+0000) 2024-02-19T23:17:05.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:05.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:05.511851+0000) 2024-02-19T23:17:06.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:06.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:06.511986+0000) 2024-02-19T23:17:07.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:07.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:07.512235+0000) 2024-02-19T23:17:08.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:08.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:08.512476+0000) 2024-02-19T23:17:09.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:09.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:09.512675+0000) 2024-02-19T23:17:10.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:10.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:10.512843+0000) 2024-02-19T23:17:11.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:11.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:11.513017+0000) 2024-02-19T23:17:12.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:12.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:12.513188+0000) 2024-02-19T23:17:13.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:13.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:13.513445+0000) 2024-02-19T23:17:14.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:14.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:14.513642+0000) 2024-02-19T23:17:15.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:15.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:15.513811+0000) 2024-02-19T23:17:16.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:16.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:16.514001+0000) 2024-02-19T23:17:16.626 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:17:16.630 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:17:17.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:17.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:17.514153+0000) 2024-02-19T23:17:18.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:18.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:18.514284+0000) 2024-02-19T23:17:19.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:19.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:19.514444+0000) 2024-02-19T23:17:20.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:20.508+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:20.514596+0000) 2024-02-19T23:17:21.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:21.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:21.514800+0000) 2024-02-19T23:17:22.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:22.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:22.514929+0000) 2024-02-19T23:17:23.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:23.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:23.515093+0000) 2024-02-19T23:17:24.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:24.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:24.515213+0000) 2024-02-19T23:17:25.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:25.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:25.515344+0000) 2024-02-19T23:17:26.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:26.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:26.515474+0000) 2024-02-19T23:17:27.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:27.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:27.515596+0000) 2024-02-19T23:17:28.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:28.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:28.515790+0000) 2024-02-19T23:17:29.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:29.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:29.516014+0000) 2024-02-19T23:17:30.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:30.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:30.516241+0000) 2024-02-19T23:17:31.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:31.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:31.516507+0000) 2024-02-19T23:17:32.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:32.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:32.516683+0000) 2024-02-19T23:17:33.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:33.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:33.516965+0000) 2024-02-19T23:17:34.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:34.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:34.517186+0000) 2024-02-19T23:17:35.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:35.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:35.517401+0000) 2024-02-19T23:17:36.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:36.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:36.517680+0000) 2024-02-19T23:17:37.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:37.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:37.517840+0000) 2024-02-19T23:17:38.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:38.512+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:38.518063+0000) 2024-02-19T23:17:39.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:39.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:39.518235+0000) 2024-02-19T23:17:40.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:40.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:40.518519+0000) 2024-02-19T23:17:41.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:41.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:41.518821+0000) 2024-02-19T23:17:42.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:42.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:42.519064+0000) 2024-02-19T23:17:43.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:43.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:43.519296+0000) 2024-02-19T23:17:44.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:44.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:44.519541+0000) 2024-02-19T23:17:45.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:45.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:45.519780+0000) 2024-02-19T23:17:46.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:46.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:46.520019+0000) 2024-02-19T23:17:46.758 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:17:46.762 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:17:47.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:47.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:47.520298+0000) 2024-02-19T23:17:48.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:48.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:48.520558+0000) 2024-02-19T23:17:49.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:49.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:49.520749+0000) 2024-02-19T23:17:50.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:50.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:50.520979+0000) 2024-02-19T23:17:51.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:51.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:51.521120+0000) 2024-02-19T23:17:52.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:52.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:52.521336+0000) 2024-02-19T23:17:53.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:53.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:53.521532+0000) 2024-02-19T23:17:54.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:54.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:54.521739+0000) 2024-02-19T23:17:55.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:55.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:55.521916+0000) 2024-02-19T23:17:56.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:56.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:56.522145+0000) 2024-02-19T23:17:57.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:57.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:57.522363+0000) 2024-02-19T23:17:58.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:58.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:58.522567+0000) 2024-02-19T23:17:59.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:17:59.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:17:59.522846+0000) 2024-02-19T23:18:00.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:00.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:00.523068+0000) 2024-02-19T23:18:01.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:01.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:01.523321+0000) 2024-02-19T23:18:02.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:02.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:02.523553+0000) 2024-02-19T23:18:03.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:03.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:03.523814+0000) 2024-02-19T23:18:04.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:04.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:04.524049+0000) 2024-02-19T23:18:05.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:05.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:05.524309+0000) 2024-02-19T23:18:06.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:06.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:06.524567+0000) 2024-02-19T23:18:07.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:07.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:07.524798+0000) 2024-02-19T23:18:08.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:08.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:08.525056+0000) 2024-02-19T23:18:09.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:09.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:09.525298+0000) 2024-02-19T23:18:10.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:10.519+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:10.525546+0000) 2024-02-19T23:18:11.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:11.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:11.525815+0000) 2024-02-19T23:18:12.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:12.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:12.526020+0000) 2024-02-19T23:18:13.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:13.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:13.526274+0000) 2024-02-19T23:18:14.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:14.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:14.526491+0000) 2024-02-19T23:18:15.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:15.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:15.526688+0000) 2024-02-19T23:18:16.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:16.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:16.526997+0000) 2024-02-19T23:18:16.896 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:18:16.900 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:18:17.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:17.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:17.527239+0000) 2024-02-19T23:18:18.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:18.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:18.527460+0000) 2024-02-19T23:18:19.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:19.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:19.527718+0000) 2024-02-19T23:18:20.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:20.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:20.527968+0000) 2024-02-19T23:18:21.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:21.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:21.528191+0000) 2024-02-19T23:18:22.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:22.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:22.528379+0000) 2024-02-19T23:18:23.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:23.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:23.528686+0000) 2024-02-19T23:18:24.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:24.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:24.528970+0000) 2024-02-19T23:18:25.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:25.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:25.529185+0000) 2024-02-19T23:18:26.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:26.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:26.529395+0000) 2024-02-19T23:18:27.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:27.523+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:27.529639+0000) 2024-02-19T23:18:28.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:28.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:28.529883+0000) 2024-02-19T23:18:29.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:29.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:29.530133+0000) 2024-02-19T23:18:30.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:30.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:30.530404+0000) 2024-02-19T23:18:31.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:31.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:31.530614+0000) 2024-02-19T23:18:32.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:32.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:32.530891+0000) 2024-02-19T23:18:33.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:33.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:33.531089+0000) 2024-02-19T23:18:34.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:34.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:34.531290+0000) 2024-02-19T23:18:35.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:35.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:35.531513+0000) 2024-02-19T23:18:36.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:36.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:36.531750+0000) 2024-02-19T23:18:37.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:37.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:37.532018+0000) 2024-02-19T23:18:38.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:38.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:38.532261+0000) 2024-02-19T23:18:39.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:39.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:39.532448+0000) 2024-02-19T23:18:40.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:40.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:40.532704+0000) 2024-02-19T23:18:41.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:41.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:41.532884+0000) 2024-02-19T23:18:42.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:42.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:42.533130+0000) 2024-02-19T23:18:43.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:43.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:43.533312+0000) 2024-02-19T23:18:44.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:44.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:44.533525+0000) 2024-02-19T23:18:45.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:45.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:45.533778+0000) 2024-02-19T23:18:46.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:46.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:46.534025+0000) 2024-02-19T23:18:47.020 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:18:47.023 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:18:47.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:47.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:47.534276+0000) 2024-02-19T23:18:48.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:48.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:48.534538+0000) 2024-02-19T23:18:49.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:49.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:49.534816+0000) 2024-02-19T23:18:50.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:50.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:50.535079+0000) 2024-02-19T23:18:51.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:51.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:51.535268+0000) 2024-02-19T23:18:52.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:52.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:52.535532+0000) 2024-02-19T23:18:53.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:53.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:53.535791+0000) 2024-02-19T23:18:54.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:54.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:54.535990+0000) 2024-02-19T23:18:55.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:55.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:55.536219+0000) 2024-02-19T23:18:56.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:56.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:56.536477+0000) 2024-02-19T23:18:57.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:57.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:57.536727+0000) 2024-02-19T23:18:58.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:58.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:58.537049+0000) 2024-02-19T23:18:59.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:18:59.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:18:59.537286+0000) 2024-02-19T23:19:00.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:00.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:00.537536+0000) 2024-02-19T23:19:01.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:01.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:01.537792+0000) 2024-02-19T23:19:02.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:02.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:02.538031+0000) 2024-02-19T23:19:03.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:03.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:03.538273+0000) 2024-02-19T23:19:04.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:04.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:04.538523+0000) 2024-02-19T23:19:05.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:05.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:05.538823+0000) 2024-02-19T23:19:06.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:06.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:06.538958+0000) 2024-02-19T23:19:07.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:07.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:07.539144+0000) 2024-02-19T23:19:08.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:08.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:08.539395+0000) 2024-02-19T23:19:09.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:09.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:09.539667+0000) 2024-02-19T23:19:10.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:10.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:10.539876+0000) 2024-02-19T23:19:11.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:11.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:11.540138+0000) 2024-02-19T23:19:12.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:12.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:12.540392+0000) 2024-02-19T23:19:13.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:13.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:13.540631+0000) 2024-02-19T23:19:14.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:14.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:14.540882+0000) 2024-02-19T23:19:15.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:15.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:15.541132+0000) 2024-02-19T23:19:16.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:16.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:16.541376+0000) 2024-02-19T23:19:17.093 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:19:17.098 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:19:17.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:17.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:17.541654+0000) 2024-02-19T23:19:18.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:18.534+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:18.541833+0000) 2024-02-19T23:19:19.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:19.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:19.542094+0000) 2024-02-19T23:19:20.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:20.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:20.542357+0000) 2024-02-19T23:19:21.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:21.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:21.542597+0000) 2024-02-19T23:19:22.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:22.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:22.542774+0000) 2024-02-19T23:19:23.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:23.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:23.543020+0000) 2024-02-19T23:19:24.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:24.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:24.543227+0000) 2024-02-19T23:19:25.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:25.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:25.543479+0000) 2024-02-19T23:19:26.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:26.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:26.543742+0000) 2024-02-19T23:19:27.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:27.538+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:27.543997+0000) 2024-02-19T23:19:28.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:28.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:28.544254+0000) 2024-02-19T23:19:29.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:29.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:29.544457+0000) 2024-02-19T23:19:30.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:30.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:30.544649+0000) 2024-02-19T23:19:31.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:31.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:31.544934+0000) 2024-02-19T23:19:32.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:32.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:32.545175+0000) 2024-02-19T23:19:33.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:33.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:33.545413+0000) 2024-02-19T23:19:34.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:34.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:34.545649+0000) 2024-02-19T23:19:35.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:35.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:35.545888+0000) 2024-02-19T23:19:36.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:36.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:36.546075+0000) 2024-02-19T23:19:37.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:37.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:37.546339+0000) 2024-02-19T23:19:38.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:38.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:38.546602+0000) 2024-02-19T23:19:39.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:39.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:39.546799+0000) 2024-02-19T23:19:40.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:40.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:40.547104+0000) 2024-02-19T23:19:41.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:41.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:41.547384+0000) 2024-02-19T23:19:42.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:42.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:42.547600+0000) 2024-02-19T23:19:43.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:43.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:43.547835+0000) 2024-02-19T23:19:44.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:44.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:44.548080+0000) 2024-02-19T23:19:45.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:45.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:45.548350+0000) 2024-02-19T23:19:46.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:46.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:46.548588+0000) 2024-02-19T23:19:47.156 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:19:47.161 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:19:47.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:47.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:47.548874+0000) 2024-02-19T23:19:48.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:48.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:48.549072+0000) 2024-02-19T23:19:49.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:49.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:49.549344+0000) 2024-02-19T23:19:50.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:50.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:50.549556+0000) 2024-02-19T23:19:51.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:51.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:51.549810+0000) 2024-02-19T23:19:52.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:52.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:52.550047+0000) 2024-02-19T23:19:53.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:53.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:53.550248+0000) 2024-02-19T23:19:54.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:54.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:54.550426+0000) 2024-02-19T23:19:55.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:55.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:55.550633+0000) 2024-02-19T23:19:56.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:56.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:56.550867+0000) 2024-02-19T23:19:57.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:57.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:57.551172+0000) 2024-02-19T23:19:58.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:58.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:58.551354+0000) 2024-02-19T23:19:59.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:19:59.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:19:59.551553+0000) 2024-02-19T23:20:00.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:00.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:00.551726+0000) 2024-02-19T23:20:01.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:01.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:01.551992+0000) 2024-02-19T23:20:02.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:02.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:02.552260+0000) 2024-02-19T23:20:03.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:03.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:03.552508+0000) 2024-02-19T23:20:04.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:04.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:04.552689+0000) 2024-02-19T23:20:05.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:05.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:05.552902+0000) 2024-02-19T23:20:06.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:06.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:06.553139+0000) 2024-02-19T23:20:07.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:07.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:07.553365+0000) 2024-02-19T23:20:08.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:08.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:08.553562+0000) 2024-02-19T23:20:09.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:09.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:09.553817+0000) 2024-02-19T23:20:10.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:10.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:10.554029+0000) 2024-02-19T23:20:11.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:11.549+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:11.554224+0000) 2024-02-19T23:20:12.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:12.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:12.554470+0000) 2024-02-19T23:20:13.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:13.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:13.554744+0000) 2024-02-19T23:20:14.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:14.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:14.554911+0000) 2024-02-19T23:20:15.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:15.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:15.555084+0000) 2024-02-19T23:20:16.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:16.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:16.555354+0000) 2024-02-19T23:20:17.280 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:20:17.284 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:20:17.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:17.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:17.555644+0000) 2024-02-19T23:20:18.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:18.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:18.555891+0000) 2024-02-19T23:20:19.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:19.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:19.556127+0000) 2024-02-19T23:20:20.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:20.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:20.556333+0000) 2024-02-19T23:20:21.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:21.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:21.556537+0000) 2024-02-19T23:20:22.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:22.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:22.556787+0000) 2024-02-19T23:20:23.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:23.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:23.557068+0000) 2024-02-19T23:20:24.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:24.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:24.557284+0000) 2024-02-19T23:20:25.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:25.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:25.557488+0000) 2024-02-19T23:20:26.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:26.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:26.557721+0000) 2024-02-19T23:20:27.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:27.553+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:27.557879+0000) 2024-02-19T23:20:28.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:28.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:28.558062+0000) 2024-02-19T23:20:29.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:29.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:29.558306+0000) 2024-02-19T23:20:30.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:30.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:30.558536+0000) 2024-02-19T23:20:31.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:31.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:31.558760+0000) 2024-02-19T23:20:32.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:32.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:32.558993+0000) 2024-02-19T23:20:33.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:33.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:33.559140+0000) 2024-02-19T23:20:34.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:34.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:34.559431+0000) 2024-02-19T23:20:35.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:35.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:35.559729+0000) 2024-02-19T23:20:36.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:36.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:36.559910+0000) 2024-02-19T23:20:37.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:37.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:37.560170+0000) 2024-02-19T23:20:38.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:38.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:38.560405+0000) 2024-02-19T23:20:39.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:39.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:39.560612+0000) 2024-02-19T23:20:40.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:40.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:40.560840+0000) 2024-02-19T23:20:41.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:41.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:41.561102+0000) 2024-02-19T23:20:42.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:42.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:42.561400+0000) 2024-02-19T23:20:43.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:43.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:43.561715+0000) 2024-02-19T23:20:44.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:44.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:44.562019+0000) 2024-02-19T23:20:45.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:45.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:45.562245+0000) 2024-02-19T23:20:46.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:46.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:46.562508+0000) 2024-02-19T23:20:47.419 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:20:47.422 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:20:47.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:47.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:47.562800+0000) 2024-02-19T23:20:48.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:48.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:48.563046+0000) 2024-02-19T23:20:49.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:49.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:49.563324+0000) 2024-02-19T23:20:50.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:50.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:50.563689+0000) 2024-02-19T23:20:51.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:51.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:51.563985+0000) 2024-02-19T23:20:52.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:52.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:52.564274+0000) 2024-02-19T23:20:53.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:53.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:53.564566+0000) 2024-02-19T23:20:54.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:54.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:54.564876+0000) 2024-02-19T23:20:55.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:55.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:55.565118+0000) 2024-02-19T23:20:56.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:56.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:56.565382+0000) 2024-02-19T23:20:57.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:57.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:57.565607+0000) 2024-02-19T23:20:58.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:58.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:58.565812+0000) 2024-02-19T23:20:59.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:20:59.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:20:59.566056+0000) 2024-02-19T23:21:00.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:00.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:00.566250+0000) 2024-02-19T23:21:01.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:01.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:01.566514+0000) 2024-02-19T23:21:02.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:02.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:02.566869+0000) 2024-02-19T23:21:03.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:03.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:03.567146+0000) 2024-02-19T23:21:04.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:04.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:04.567393+0000) 2024-02-19T23:21:05.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:05.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:05.567606+0000) 2024-02-19T23:21:06.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:06.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:06.567875+0000) 2024-02-19T23:21:07.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:07.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:07.568139+0000) 2024-02-19T23:21:08.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:08.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:08.568398+0000) 2024-02-19T23:21:09.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:09.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:09.568592+0000) 2024-02-19T23:21:10.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:10.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:10.568902+0000) 2024-02-19T23:21:11.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:11.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:11.569205+0000) 2024-02-19T23:21:12.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:12.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:12.569493+0000) 2024-02-19T23:21:13.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:13.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:13.569739+0000) 2024-02-19T23:21:14.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:14.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:14.569903+0000) 2024-02-19T23:21:15.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:15.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:15.570179+0000) 2024-02-19T23:21:16.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:16.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:16.570416+0000) 2024-02-19T23:21:17.534 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:21:17.538 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:21:17.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:17.564+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:17.570671+0000) 2024-02-19T23:21:18.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:18.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:18.570926+0000) 2024-02-19T23:21:19.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:19.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:19.571180+0000) 2024-02-19T23:21:20.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:20.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:20.571421+0000) 2024-02-19T23:21:21.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:21.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:21.571672+0000) 2024-02-19T23:21:22.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:22.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:22.571944+0000) 2024-02-19T23:21:23.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:23.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:23.572239+0000) 2024-02-19T23:21:24.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:24.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:24.572526+0000) 2024-02-19T23:21:25.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:25.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:25.572715+0000) 2024-02-19T23:21:26.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:26.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:26.572901+0000) 2024-02-19T23:21:27.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:27.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:27.573075+0000) 2024-02-19T23:21:28.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:28.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:28.573393+0000) 2024-02-19T23:21:29.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:29.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:29.573699+0000) 2024-02-19T23:21:30.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:30.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:30.573869+0000) 2024-02-19T23:21:31.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:31.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:31.574049+0000) 2024-02-19T23:21:32.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:32.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:32.574223+0000) 2024-02-19T23:21:33.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:33.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:33.574484+0000) 2024-02-19T23:21:34.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:34.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:34.574650+0000) 2024-02-19T23:21:35.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:35.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:35.574841+0000) 2024-02-19T23:21:36.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:36.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:36.574990+0000) 2024-02-19T23:21:37.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:37.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:37.575142+0000) 2024-02-19T23:21:38.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:38.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:38.575330+0000) 2024-02-19T23:21:39.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:39.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:39.575479+0000) 2024-02-19T23:21:40.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:40.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:40.575675+0000) 2024-02-19T23:21:41.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:41.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:41.575880+0000) 2024-02-19T23:21:42.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:42.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:42.576162+0000) 2024-02-19T23:21:43.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:43.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:43.576371+0000) 2024-02-19T23:21:44.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:44.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:44.576576+0000) 2024-02-19T23:21:45.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:45.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:45.576838+0000) 2024-02-19T23:21:46.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:46.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:46.577113+0000) 2024-02-19T23:21:47.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:47.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:47.577320+0000) 2024-02-19T23:21:47.648 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:21:47.652 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:21:48.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:48.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:48.577568+0000) 2024-02-19T23:21:49.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:49.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:49.577735+0000) 2024-02-19T23:21:50.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:50.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:50.577953+0000) 2024-02-19T23:21:51.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:51.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:51.578183+0000) 2024-02-19T23:21:52.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:52.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:52.578328+0000) 2024-02-19T23:21:53.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:53.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:53.578564+0000) 2024-02-19T23:21:54.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:54.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:54.578748+0000) 2024-02-19T23:21:55.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:55.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:55.578918+0000) 2024-02-19T23:21:56.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:56.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:56.579084+0000) 2024-02-19T23:21:57.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:57.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:57.579348+0000) 2024-02-19T23:21:58.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:58.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:58.579504+0000) 2024-02-19T23:21:59.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:21:59.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:21:59.579695+0000) 2024-02-19T23:22:00.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:00.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:00.579930+0000) 2024-02-19T23:22:01.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:01.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:01.580179+0000) 2024-02-19T23:22:02.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:02.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:02.580373+0000) 2024-02-19T23:22:03.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:03.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:03.580603+0000) 2024-02-19T23:22:04.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:04.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:04.580829+0000) 2024-02-19T23:22:05.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:05.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:05.581009+0000) 2024-02-19T23:22:06.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:06.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:06.581238+0000) 2024-02-19T23:22:07.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:07.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:07.581422+0000) 2024-02-19T23:22:08.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:08.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:08.581576+0000) 2024-02-19T23:22:09.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:09.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:09.581826+0000) 2024-02-19T23:22:10.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:10.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:10.582074+0000) 2024-02-19T23:22:11.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:11.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:11.582273+0000) 2024-02-19T23:22:12.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:12.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:12.582522+0000) 2024-02-19T23:22:13.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:13.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:13.582756+0000) 2024-02-19T23:22:14.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:14.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:14.582954+0000) 2024-02-19T23:22:15.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:15.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:15.583130+0000) 2024-02-19T23:22:16.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:16.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:16.583316+0000) 2024-02-19T23:22:17.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:17.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:17.583558+0000) 2024-02-19T23:22:17.712 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:22:17.715 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:22:18.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:18.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:18.583825+0000) 2024-02-19T23:22:19.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:19.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:19.584054+0000) 2024-02-19T23:22:20.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:20.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:20.584290+0000) 2024-02-19T23:22:21.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:21.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:21.584562+0000) 2024-02-19T23:22:22.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:22.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:22.584805+0000) 2024-02-19T23:22:23.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:23.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:23.585001+0000) 2024-02-19T23:22:24.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:24.579+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:24.585239+0000) 2024-02-19T23:22:25.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:25.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:25.585436+0000) 2024-02-19T23:22:26.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:26.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:26.585653+0000) 2024-02-19T23:22:27.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:27.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:27.585896+0000) 2024-02-19T23:22:28.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:28.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:28.586094+0000) 2024-02-19T23:22:29.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:29.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:29.586331+0000) 2024-02-19T23:22:30.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:30.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:30.586560+0000) 2024-02-19T23:22:31.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:31.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:31.586844+0000) 2024-02-19T23:22:32.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:32.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:32.587150+0000) 2024-02-19T23:22:33.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:33.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:33.587330+0000) 2024-02-19T23:22:34.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:34.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:34.587517+0000) 2024-02-19T23:22:35.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:35.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:35.587691+0000) 2024-02-19T23:22:36.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:36.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:36.587904+0000) 2024-02-19T23:22:37.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:37.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:37.588069+0000) 2024-02-19T23:22:38.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:38.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:38.588238+0000) 2024-02-19T23:22:39.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:39.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:39.588404+0000) 2024-02-19T23:22:40.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:40.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:40.588632+0000) 2024-02-19T23:22:41.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:41.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:41.588868+0000) 2024-02-19T23:22:42.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:42.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:42.589075+0000) 2024-02-19T23:22:43.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:43.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:43.589240+0000) 2024-02-19T23:22:44.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:44.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:44.589440+0000) 2024-02-19T23:22:45.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:45.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:45.589659+0000) 2024-02-19T23:22:46.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:46.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:46.589853+0000) 2024-02-19T23:22:47.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:47.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:47.590015+0000) 2024-02-19T23:22:47.835 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:22:47.839 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:22:48.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:48.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:48.590241+0000) 2024-02-19T23:22:49.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:49.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:49.590440+0000) 2024-02-19T23:22:50.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:50.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:50.590665+0000) 2024-02-19T23:22:51.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:51.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:51.590906+0000) 2024-02-19T23:22:52.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:52.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:52.591164+0000) 2024-02-19T23:22:53.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:53.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:53.591353+0000) 2024-02-19T23:22:54.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:54.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:54.591607+0000) 2024-02-19T23:22:55.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:55.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:55.591838+0000) 2024-02-19T23:22:56.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:56.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:56.592074+0000) 2024-02-19T23:22:57.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:57.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:57.592266+0000) 2024-02-19T23:22:58.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:58.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:58.592529+0000) 2024-02-19T23:22:59.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:22:59.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:22:59.592706+0000) 2024-02-19T23:23:00.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:00.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:00.592892+0000) 2024-02-19T23:23:01.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:01.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:01.593116+0000) 2024-02-19T23:23:02.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:02.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:02.593385+0000) 2024-02-19T23:23:03.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:03.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:03.593593+0000) 2024-02-19T23:23:04.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:04.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:04.593845+0000) 2024-02-19T23:23:05.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:05.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:05.594085+0000) 2024-02-19T23:23:06.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:06.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:06.594285+0000) 2024-02-19T23:23:07.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:07.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:07.594462+0000) 2024-02-19T23:23:08.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:08.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:08.594645+0000) 2024-02-19T23:23:09.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:09.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:09.594907+0000) 2024-02-19T23:23:10.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:10.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:10.595097+0000) 2024-02-19T23:23:11.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:11.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:11.595246+0000) 2024-02-19T23:23:12.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:12.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:12.595473+0000) 2024-02-19T23:23:13.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:13.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:13.595676+0000) 2024-02-19T23:23:14.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:14.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:14.595907+0000) 2024-02-19T23:23:15.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:15.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:15.596149+0000) 2024-02-19T23:23:16.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:16.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:16.596342+0000) 2024-02-19T23:23:17.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:17.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:17.596507+0000) 2024-02-19T23:23:17.966 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:23:17.969 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:23:18.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:18.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:18.596725+0000) 2024-02-19T23:23:19.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:19.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:19.596991+0000) 2024-02-19T23:23:20.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:20.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:20.597240+0000) 2024-02-19T23:23:21.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:21.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:21.597436+0000) 2024-02-19T23:23:22.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:22.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:22.597608+0000) 2024-02-19T23:23:23.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:23.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:23.597785+0000) 2024-02-19T23:23:24.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:24.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:24.597974+0000) 2024-02-19T23:23:25.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:25.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:25.598169+0000) 2024-02-19T23:23:26.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:26.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:26.598415+0000) 2024-02-19T23:23:27.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:27.590+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:27.598689+0000) 2024-02-19T23:23:28.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:28.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:28.598926+0000) 2024-02-19T23:23:29.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:29.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:29.599075+0000) 2024-02-19T23:23:30.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:30.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:30.599308+0000) 2024-02-19T23:23:31.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:31.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:31.599504+0000) 2024-02-19T23:23:32.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:32.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:32.599696+0000) 2024-02-19T23:23:33.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:33.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:33.599931+0000) 2024-02-19T23:23:34.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:34.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:34.600130+0000) 2024-02-19T23:23:35.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:35.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:35.600385+0000) 2024-02-19T23:23:36.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:36.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:36.600644+0000) 2024-02-19T23:23:37.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:37.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:37.600881+0000) 2024-02-19T23:23:38.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:38.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:38.601035+0000) 2024-02-19T23:23:39.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:39.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:39.601229+0000) 2024-02-19T23:23:40.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:40.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:40.601473+0000) 2024-02-19T23:23:41.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:41.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:41.601688+0000) 2024-02-19T23:23:42.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:42.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:42.601924+0000) 2024-02-19T23:23:43.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:43.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:43.602064+0000) 2024-02-19T23:23:44.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:44.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:44.602206+0000) 2024-02-19T23:23:45.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:45.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:45.602391+0000) 2024-02-19T23:23:46.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:46.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:46.602625+0000) 2024-02-19T23:23:47.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:47.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:47.602904+0000) 2024-02-19T23:23:48.080 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:23:48.084 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:23:48.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:48.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:48.603051+0000) 2024-02-19T23:23:49.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:49.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:49.603226+0000) 2024-02-19T23:23:50.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:50.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:50.603469+0000) 2024-02-19T23:23:51.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:51.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:51.603667+0000) 2024-02-19T23:23:52.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:52.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:52.603893+0000) 2024-02-19T23:23:53.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:53.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:53.604143+0000) 2024-02-19T23:23:54.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:54.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:54.604361+0000) 2024-02-19T23:23:55.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:55.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:55.604634+0000) 2024-02-19T23:23:56.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:56.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:56.604869+0000) 2024-02-19T23:23:57.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:57.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:57.605029+0000) 2024-02-19T23:23:58.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:58.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:58.605216+0000) 2024-02-19T23:23:59.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:23:59.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:23:59.605388+0000) 2024-02-19T23:24:00.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:00.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:00.605572+0000) 2024-02-19T23:24:01.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:01.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:01.605721+0000) 2024-02-19T23:24:02.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:02.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:02.605925+0000) 2024-02-19T23:24:03.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:03.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:03.606078+0000) 2024-02-19T23:24:04.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:04.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:04.606304+0000) 2024-02-19T23:24:05.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:05.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:05.606494+0000) 2024-02-19T23:24:06.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:06.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:06.606614+0000) 2024-02-19T23:24:07.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:07.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:07.606788+0000) 2024-02-19T23:24:08.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:08.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:08.606959+0000) 2024-02-19T23:24:09.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:09.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:09.607138+0000) 2024-02-19T23:24:10.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:10.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:10.607330+0000) 2024-02-19T23:24:11.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:11.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:11.607515+0000) 2024-02-19T23:24:12.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:12.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:12.607685+0000) 2024-02-19T23:24:13.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:13.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:13.607876+0000) 2024-02-19T23:24:14.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:14.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:14.608088+0000) 2024-02-19T23:24:15.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:15.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:15.608305+0000) 2024-02-19T23:24:16.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:16.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:16.608493+0000) 2024-02-19T23:24:17.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:17.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:17.608742+0000) 2024-02-19T23:24:18.152 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:24:18.157 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:24:18.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:18.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:18.608975+0000) 2024-02-19T23:24:19.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:19.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:19.609233+0000) 2024-02-19T23:24:20.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:20.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:20.609451+0000) 2024-02-19T23:24:21.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:21.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:21.609705+0000) 2024-02-19T23:24:22.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:22.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:22.609922+0000) 2024-02-19T23:24:23.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:23.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:23.610075+0000) 2024-02-19T23:24:24.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:24.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:24.610239+0000) 2024-02-19T23:24:25.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:25.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:25.610413+0000) 2024-02-19T23:24:26.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:26.605+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:26.610617+0000) 2024-02-19T23:24:27.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:27.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:27.610775+0000) 2024-02-19T23:24:28.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:28.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:28.610959+0000) 2024-02-19T23:24:29.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:29.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:29.611151+0000) 2024-02-19T23:24:30.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:30.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:30.611389+0000) 2024-02-19T23:24:31.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:31.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:31.611594+0000) 2024-02-19T23:24:32.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:32.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:32.611767+0000) 2024-02-19T23:24:33.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:33.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:33.611996+0000) 2024-02-19T23:24:34.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:34.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:34.612222+0000) 2024-02-19T23:24:35.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:35.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:35.612407+0000) 2024-02-19T23:24:36.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:36.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:36.612650+0000) 2024-02-19T23:24:37.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:37.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:37.612828+0000) 2024-02-19T23:24:38.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:38.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:38.612965+0000) 2024-02-19T23:24:39.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:39.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:39.613141+0000) 2024-02-19T23:24:40.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:40.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:40.613335+0000) 2024-02-19T23:24:41.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:41.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:41.613505+0000) 2024-02-19T23:24:42.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:42.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:42.613668+0000) 2024-02-19T23:24:43.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:43.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:43.613892+0000) 2024-02-19T23:24:44.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:44.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:44.614150+0000) 2024-02-19T23:24:45.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:45.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:45.614356+0000) 2024-02-19T23:24:46.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:46.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:46.614618+0000) 2024-02-19T23:24:47.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:47.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:47.614841+0000) 2024-02-19T23:24:48.216 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:24:48.219 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:24:48.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:48.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:48.615001+0000) 2024-02-19T23:24:49.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:49.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:49.615207+0000) 2024-02-19T23:24:50.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:50.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:50.615439+0000) 2024-02-19T23:24:51.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:51.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:51.615636+0000) 2024-02-19T23:24:52.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:52.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:52.615874+0000) 2024-02-19T23:24:53.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:53.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:53.616100+0000) 2024-02-19T23:24:54.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:54.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:54.616346+0000) 2024-02-19T23:24:55.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:55.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:55.616620+0000) 2024-02-19T23:24:56.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:56.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:56.616889+0000) 2024-02-19T23:24:57.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:57.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:57.617081+0000) 2024-02-19T23:24:58.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:58.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:58.617298+0000) 2024-02-19T23:24:59.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:24:59.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:24:59.617533+0000) 2024-02-19T23:25:00.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:00.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:00.617756+0000) 2024-02-19T23:25:01.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:01.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:01.617967+0000) 2024-02-19T23:25:02.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:02.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:02.618133+0000) 2024-02-19T23:25:03.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:03.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:03.618312+0000) 2024-02-19T23:25:04.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:04.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:04.618537+0000) 2024-02-19T23:25:05.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:05.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:05.618725+0000) 2024-02-19T23:25:06.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:06.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:06.618943+0000) 2024-02-19T23:25:07.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:07.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:07.619133+0000) 2024-02-19T23:25:08.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:08.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:08.619401+0000) 2024-02-19T23:25:09.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:09.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:09.619700+0000) 2024-02-19T23:25:10.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:10.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:10.619925+0000) 2024-02-19T23:25:11.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:11.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:11.620128+0000) 2024-02-19T23:25:12.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:12.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:12.620303+0000) 2024-02-19T23:25:13.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:13.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:13.620429+0000) 2024-02-19T23:25:14.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:14.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:14.620635+0000) 2024-02-19T23:25:15.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:15.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:15.620803+0000) 2024-02-19T23:25:16.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:16.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:16.621037+0000) 2024-02-19T23:25:17.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:17.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:17.621282+0000) 2024-02-19T23:25:18.356 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:25:18.359 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:25:18.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:18.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:18.621462+0000) 2024-02-19T23:25:19.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:19.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:19.621681+0000) 2024-02-19T23:25:20.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:20.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:20.621897+0000) 2024-02-19T23:25:21.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:21.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:21.622060+0000) 2024-02-19T23:25:22.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:22.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:22.622246+0000) 2024-02-19T23:25:23.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:23.616+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:23.622548+0000) 2024-02-19T23:25:24.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:24.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:24.622904+0000) 2024-02-19T23:25:25.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:25.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:25.623056+0000) 2024-02-19T23:25:26.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:26.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:26.623225+0000) 2024-02-19T23:25:27.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:27.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:27.623354+0000) 2024-02-19T23:25:28.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:28.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:28.623519+0000) 2024-02-19T23:25:29.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:29.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:29.623687+0000) 2024-02-19T23:25:30.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:30.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:30.623817+0000) 2024-02-19T23:25:31.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:31.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:31.624002+0000) 2024-02-19T23:25:32.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:32.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:32.624213+0000) 2024-02-19T23:25:33.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:33.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:33.624403+0000) 2024-02-19T23:25:34.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:34.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:34.624634+0000) 2024-02-19T23:25:35.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:35.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:35.624855+0000) 2024-02-19T23:25:36.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:36.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:36.625015+0000) 2024-02-19T23:25:37.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:37.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:37.625182+0000) 2024-02-19T23:25:38.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:38.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:38.625426+0000) 2024-02-19T23:25:39.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:39.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:39.625618+0000) 2024-02-19T23:25:40.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:40.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:40.625868+0000) 2024-02-19T23:25:41.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:41.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:41.626110+0000) 2024-02-19T23:25:42.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:42.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:42.626300+0000) 2024-02-19T23:25:43.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:43.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:43.626442+0000) 2024-02-19T23:25:44.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:44.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:44.626612+0000) 2024-02-19T23:25:45.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:45.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:45.626831+0000) 2024-02-19T23:25:46.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:46.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:46.626970+0000) 2024-02-19T23:25:47.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:47.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:47.627165+0000) 2024-02-19T23:25:48.471 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:25:48.474 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:25:48.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:48.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:48.627299+0000) 2024-02-19T23:25:49.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:49.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:49.627469+0000) 2024-02-19T23:25:50.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:50.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:50.627604+0000) 2024-02-19T23:25:51.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:51.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:51.627801+0000) 2024-02-19T23:25:52.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:52.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:52.627964+0000) 2024-02-19T23:25:53.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:53.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:53.628141+0000) 2024-02-19T23:25:54.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:54.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:54.628280+0000) 2024-02-19T23:25:55.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:55.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:55.628414+0000) 2024-02-19T23:25:56.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:56.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:56.628545+0000) 2024-02-19T23:25:57.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:57.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:57.628680+0000) 2024-02-19T23:25:58.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:58.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:58.628810+0000) 2024-02-19T23:25:59.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:25:59.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:25:59.628941+0000) 2024-02-19T23:26:00.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:00.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:00.629078+0000) 2024-02-19T23:26:01.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:01.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:01.629204+0000) 2024-02-19T23:26:02.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:02.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:02.629332+0000) 2024-02-19T23:26:03.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:03.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:03.629485+0000) 2024-02-19T23:26:04.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:04.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:04.629638+0000) 2024-02-19T23:26:05.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:05.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:05.629797+0000) 2024-02-19T23:26:06.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:06.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:06.630023+0000) 2024-02-19T23:26:07.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:07.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:07.630172+0000) 2024-02-19T23:26:08.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:08.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:08.630314+0000) 2024-02-19T23:26:09.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:09.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:09.630446+0000) 2024-02-19T23:26:10.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:10.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:10.630678+0000) 2024-02-19T23:26:11.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:11.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:11.630861+0000) 2024-02-19T23:26:12.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:12.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:12.631093+0000) 2024-02-19T23:26:13.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:13.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:13.631251+0000) 2024-02-19T23:26:14.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:14.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:14.631480+0000) 2024-02-19T23:26:15.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:15.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:15.631680+0000) 2024-02-19T23:26:16.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:16.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:16.631853+0000) 2024-02-19T23:26:17.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:17.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:17.631999+0000) 2024-02-19T23:26:18.569 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:26:18.572 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:26:18.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:18.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:18.632177+0000) 2024-02-19T23:26:19.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:19.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:19.632361+0000) 2024-02-19T23:26:20.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:20.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:20.632538+0000) 2024-02-19T23:26:21.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:21.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:21.632709+0000) 2024-02-19T23:26:22.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:22.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:22.632909+0000) 2024-02-19T23:26:23.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:23.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:23.633068+0000) 2024-02-19T23:26:24.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:24.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:24.633305+0000) 2024-02-19T23:26:25.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:25.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:25.633481+0000) 2024-02-19T23:26:26.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:26.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:26.633708+0000) 2024-02-19T23:26:27.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:27.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:27.633877+0000) 2024-02-19T23:26:28.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:28.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:28.634038+0000) 2024-02-19T23:26:29.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:29.627+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:29.634271+0000) 2024-02-19T23:26:30.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:30.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:30.634490+0000) 2024-02-19T23:26:31.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:31.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:31.634755+0000) 2024-02-19T23:26:32.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:32.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:32.634991+0000) 2024-02-19T23:26:33.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:33.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:33.635173+0000) 2024-02-19T23:26:34.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:34.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:34.635349+0000) 2024-02-19T23:26:35.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:35.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:35.635560+0000) 2024-02-19T23:26:36.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:36.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:36.635754+0000) 2024-02-19T23:26:37.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:37.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:37.635980+0000) 2024-02-19T23:26:38.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:38.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:38.636208+0000) 2024-02-19T23:26:39.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:39.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:39.636430+0000) 2024-02-19T23:26:40.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:40.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:40.636660+0000) 2024-02-19T23:26:41.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:41.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:41.636836+0000) 2024-02-19T23:26:42.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:42.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:42.637128+0000) 2024-02-19T23:26:43.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:43.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:43.637312+0000) 2024-02-19T23:26:44.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:44.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:44.637484+0000) 2024-02-19T23:26:45.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:45.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:45.637624+0000) 2024-02-19T23:26:46.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:46.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:46.637758+0000) 2024-02-19T23:26:47.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:47.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:47.637938+0000) 2024-02-19T23:26:48.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:48.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:48.638123+0000) 2024-02-19T23:26:48.674 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:26:48.677 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:26:49.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:49.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:49.638401+0000) 2024-02-19T23:26:50.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:50.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:50.638594+0000) 2024-02-19T23:26:51.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:51.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:51.638786+0000) 2024-02-19T23:26:52.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:52.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:52.638947+0000) 2024-02-19T23:26:53.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:53.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:53.639122+0000) 2024-02-19T23:26:54.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:54.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:54.639315+0000) 2024-02-19T23:26:55.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:55.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:55.639477+0000) 2024-02-19T23:26:56.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:56.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:56.639625+0000) 2024-02-19T23:26:57.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:57.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:57.639876+0000) 2024-02-19T23:26:58.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:58.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:58.640034+0000) 2024-02-19T23:26:59.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:26:59.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:26:59.640269+0000) 2024-02-19T23:27:00.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:00.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:00.640515+0000) 2024-02-19T23:27:01.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:01.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:01.640698+0000) 2024-02-19T23:27:02.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:02.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:02.640927+0000) 2024-02-19T23:27:03.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:03.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:03.641101+0000) 2024-02-19T23:27:04.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:04.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:04.641317+0000) 2024-02-19T23:27:05.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:05.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:05.641497+0000) 2024-02-19T23:27:06.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:06.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:06.641738+0000) 2024-02-19T23:27:07.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:07.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:07.641981+0000) 2024-02-19T23:27:08.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:08.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:08.642212+0000) 2024-02-19T23:27:09.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:09.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:09.642382+0000) 2024-02-19T23:27:10.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:10.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:10.642556+0000) 2024-02-19T23:27:11.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:11.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:11.642817+0000) 2024-02-19T23:27:12.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:12.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:12.643007+0000) 2024-02-19T23:27:13.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:13.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:13.643167+0000) 2024-02-19T23:27:14.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:14.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:14.643355+0000) 2024-02-19T23:27:15.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:15.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:15.643531+0000) 2024-02-19T23:27:16.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:16.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:16.643714+0000) 2024-02-19T23:27:17.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:17.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:17.643946+0000) 2024-02-19T23:27:18.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:18.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:18.644179+0000) 2024-02-19T23:27:18.746 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:27:18.749 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:27:19.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:19.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:19.644386+0000) 2024-02-19T23:27:20.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:20.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:20.644557+0000) 2024-02-19T23:27:21.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:21.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:21.644756+0000) 2024-02-19T23:27:22.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:22.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:22.644931+0000) 2024-02-19T23:27:23.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:23.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:23.645157+0000) 2024-02-19T23:27:24.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:24.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:24.645342+0000) 2024-02-19T23:27:25.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:25.638+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:25.645559+0000) 2024-02-19T23:27:26.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:26.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:26.645789+0000) 2024-02-19T23:27:27.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:27.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:27.646006+0000) 2024-02-19T23:27:28.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:28.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:28.646222+0000) 2024-02-19T23:27:29.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:29.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:29.646427+0000) 2024-02-19T23:27:30.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:30.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:30.646627+0000) 2024-02-19T23:27:31.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:31.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:31.646848+0000) 2024-02-19T23:27:32.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:32.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:32.647083+0000) 2024-02-19T23:27:33.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:33.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:33.647321+0000) 2024-02-19T23:27:34.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:34.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:34.647574+0000) 2024-02-19T23:27:35.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:35.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:35.647811+0000) 2024-02-19T23:27:36.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:36.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:36.648038+0000) 2024-02-19T23:27:37.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:37.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:37.648235+0000) 2024-02-19T23:27:38.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:38.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:38.648470+0000) 2024-02-19T23:27:39.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:39.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:39.648680+0000) 2024-02-19T23:27:40.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:40.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:40.648876+0000) 2024-02-19T23:27:41.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:41.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:41.649078+0000) 2024-02-19T23:27:42.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:42.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:42.649253+0000) 2024-02-19T23:27:43.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:43.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:43.649477+0000) 2024-02-19T23:27:44.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:44.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:44.649710+0000) 2024-02-19T23:27:45.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:45.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:45.649902+0000) 2024-02-19T23:27:46.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:46.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:46.650089+0000) 2024-02-19T23:27:47.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:47.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:47.650313+0000) 2024-02-19T23:27:48.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:48.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:48.650551+0000) 2024-02-19T23:27:48.852 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:27:48.856 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:27:49.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:49.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:49.650772+0000) 2024-02-19T23:27:50.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:50.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:50.650975+0000) 2024-02-19T23:27:51.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:51.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:51.651197+0000) 2024-02-19T23:27:52.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:52.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:52.651482+0000) 2024-02-19T23:27:53.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:53.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:53.651673+0000) 2024-02-19T23:27:54.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:54.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:54.651942+0000) 2024-02-19T23:27:55.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:55.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:55.652122+0000) 2024-02-19T23:27:56.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:56.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:56.652312+0000) 2024-02-19T23:27:57.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:57.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:57.652495+0000) 2024-02-19T23:27:58.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:58.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:58.652616+0000) 2024-02-19T23:27:59.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:27:59.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:27:59.652839+0000) 2024-02-19T23:28:00.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:00.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:00.653059+0000) 2024-02-19T23:28:01.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:01.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:01.653229+0000) 2024-02-19T23:28:02.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:02.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:02.653481+0000) 2024-02-19T23:28:03.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:03.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:03.653757+0000) 2024-02-19T23:28:04.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:04.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:04.653936+0000) 2024-02-19T23:28:05.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:05.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:05.654125+0000) 2024-02-19T23:28:06.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:06.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:06.654306+0000) 2024-02-19T23:28:07.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:07.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:07.654527+0000) 2024-02-19T23:28:08.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:08.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:08.654780+0000) 2024-02-19T23:28:09.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:09.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:09.654957+0000) 2024-02-19T23:28:10.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:10.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:10.655150+0000) 2024-02-19T23:28:11.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:11.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:11.655329+0000) 2024-02-19T23:28:12.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:12.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:12.655595+0000) 2024-02-19T23:28:13.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:13.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:13.655881+0000) 2024-02-19T23:28:14.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:14.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:14.656059+0000) 2024-02-19T23:28:15.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:15.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:15.656281+0000) 2024-02-19T23:28:16.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:16.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:16.656488+0000) 2024-02-19T23:28:17.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:17.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:17.656743+0000) 2024-02-19T23:28:18.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:18.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:18.657039+0000) 2024-02-19T23:28:18.958 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:28:18.963 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:28:19.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:19.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:19.657256+0000) 2024-02-19T23:28:20.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:20.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:20.657434+0000) 2024-02-19T23:28:21.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:21.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:21.657608+0000) 2024-02-19T23:28:22.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:22.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:22.657783+0000) 2024-02-19T23:28:23.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:23.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:23.657932+0000) 2024-02-19T23:28:24.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:24.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:24.658142+0000) 2024-02-19T23:28:25.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:25.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:25.658316+0000) 2024-02-19T23:28:26.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:26.653+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:26.658478+0000) 2024-02-19T23:28:27.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:27.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:27.658619+0000) 2024-02-19T23:28:28.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:28.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:28.658854+0000) 2024-02-19T23:28:29.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:29.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:29.658990+0000) 2024-02-19T23:28:30.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:30.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:30.659128+0000) 2024-02-19T23:28:31.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:31.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:31.659298+0000) 2024-02-19T23:28:32.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:32.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:32.659532+0000) 2024-02-19T23:28:33.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:33.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:33.659713+0000) 2024-02-19T23:28:34.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:34.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:34.659936+0000) 2024-02-19T23:28:35.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:35.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:35.660132+0000) 2024-02-19T23:28:36.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:36.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:36.660321+0000) 2024-02-19T23:28:37.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:37.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:37.660592+0000) 2024-02-19T23:28:38.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:38.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:38.660757+0000) 2024-02-19T23:28:39.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:39.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:39.660942+0000) 2024-02-19T23:28:40.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:40.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:40.661176+0000) 2024-02-19T23:28:41.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:41.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:41.661407+0000) 2024-02-19T23:28:42.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:42.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:42.661544+0000) 2024-02-19T23:28:43.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:43.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:43.661753+0000) 2024-02-19T23:28:44.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:44.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:44.661976+0000) 2024-02-19T23:28:45.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:45.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:45.662184+0000) 2024-02-19T23:28:46.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:46.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:46.662409+0000) 2024-02-19T23:28:47.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:47.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:47.662641+0000) 2024-02-19T23:28:48.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:48.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:48.662797+0000) 2024-02-19T23:28:49.071 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:28:49.074 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:28:49.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:49.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:49.663027+0000) 2024-02-19T23:28:50.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:50.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:50.663297+0000) 2024-02-19T23:28:51.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:51.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:51.663459+0000) 2024-02-19T23:28:52.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:52.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:52.663627+0000) 2024-02-19T23:28:53.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:53.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:53.663854+0000) 2024-02-19T23:28:54.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:54.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:54.664081+0000) 2024-02-19T23:28:55.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:55.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:55.664349+0000) 2024-02-19T23:28:56.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:56.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:56.664579+0000) 2024-02-19T23:28:57.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:57.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:57.664800+0000) 2024-02-19T23:28:58.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:58.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:58.664988+0000) 2024-02-19T23:28:59.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:28:59.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:28:59.665254+0000) 2024-02-19T23:29:00.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:00.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:00.665481+0000) 2024-02-19T23:29:01.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:01.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:01.665702+0000) 2024-02-19T23:29:02.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:02.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:02.665873+0000) 2024-02-19T23:29:03.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:03.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:03.666033+0000) 2024-02-19T23:29:04.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:04.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:04.666198+0000) 2024-02-19T23:29:05.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:05.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:05.666374+0000) 2024-02-19T23:29:06.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:06.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:06.666579+0000) 2024-02-19T23:29:07.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:07.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:07.666784+0000) 2024-02-19T23:29:08.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:08.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:08.666958+0000) 2024-02-19T23:29:09.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:09.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:09.667172+0000) 2024-02-19T23:29:10.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:10.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:10.667405+0000) 2024-02-19T23:29:11.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:11.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:11.667584+0000) 2024-02-19T23:29:12.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:12.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:12.667779+0000) 2024-02-19T23:29:13.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:13.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:13.667994+0000) 2024-02-19T23:29:14.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:14.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:14.668186+0000) 2024-02-19T23:29:15.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:15.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:15.668436+0000) 2024-02-19T23:29:16.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:16.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:16.668625+0000) 2024-02-19T23:29:17.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:17.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:17.668775+0000) 2024-02-19T23:29:18.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:18.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:18.668951+0000) 2024-02-19T23:29:19.153 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:29:19.158 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:29:19.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:19.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:19.669162+0000) 2024-02-19T23:29:20.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:20.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:20.669346+0000) 2024-02-19T23:29:21.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:21.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:21.669574+0000) 2024-02-19T23:29:22.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:22.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:22.669739+0000) 2024-02-19T23:29:23.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:23.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:23.669915+0000) 2024-02-19T23:29:24.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:24.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:24.670081+0000) 2024-02-19T23:29:25.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:25.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:25.670252+0000) 2024-02-19T23:29:26.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:26.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:26.670432+0000) 2024-02-19T23:29:27.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:27.664+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:27.670619+0000) 2024-02-19T23:29:28.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:28.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:28.670873+0000) 2024-02-19T23:29:29.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:29.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:29.671054+0000) 2024-02-19T23:29:30.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:30.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:30.671276+0000) 2024-02-19T23:29:31.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:31.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:31.671462+0000) 2024-02-19T23:29:32.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:32.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:32.671590+0000) 2024-02-19T23:29:33.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:33.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:33.671835+0000) 2024-02-19T23:29:34.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:34.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:34.672025+0000) 2024-02-19T23:29:35.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:35.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:35.672203+0000) 2024-02-19T23:29:36.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:36.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:36.672423+0000) 2024-02-19T23:29:37.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:37.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:37.672649+0000) 2024-02-19T23:29:38.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:38.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:38.672963+0000) 2024-02-19T23:29:39.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:39.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:39.673158+0000) 2024-02-19T23:29:40.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:40.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:40.673470+0000) 2024-02-19T23:29:41.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:41.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:41.673673+0000) 2024-02-19T23:29:42.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:42.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:42.673935+0000) 2024-02-19T23:29:43.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:43.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:43.674124+0000) 2024-02-19T23:29:44.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:44.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:44.674372+0000) 2024-02-19T23:29:45.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:45.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:45.674629+0000) 2024-02-19T23:29:46.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:46.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:46.674792+0000) 2024-02-19T23:29:47.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:47.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:47.675026+0000) 2024-02-19T23:29:48.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:48.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:48.675229+0000) 2024-02-19T23:29:49.220 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:29:49.224 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:29:49.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:49.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:49.675505+0000) 2024-02-19T23:29:50.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:50.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:50.675797+0000) 2024-02-19T23:29:51.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:51.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:51.676051+0000) 2024-02-19T23:29:52.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:52.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:52.676344+0000) 2024-02-19T23:29:53.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:53.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:53.676576+0000) 2024-02-19T23:29:54.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:54.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:54.676839+0000) 2024-02-19T23:29:55.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:55.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:55.677064+0000) 2024-02-19T23:29:56.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:56.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:56.677359+0000) 2024-02-19T23:29:57.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:57.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:57.677592+0000) 2024-02-19T23:29:58.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:58.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:58.677884+0000) 2024-02-19T23:29:59.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:29:59.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:29:59.678078+0000) 2024-02-19T23:30:00.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:00.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:00.678309+0000) 2024-02-19T23:30:01.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:01.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:01.678529+0000) 2024-02-19T23:30:02.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:02.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:02.678767+0000) 2024-02-19T23:30:03.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:03.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:03.679039+0000) 2024-02-19T23:30:04.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:04.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:04.679279+0000) 2024-02-19T23:30:05.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:05.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:05.679439+0000) 2024-02-19T23:30:06.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:06.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:06.679655+0000) 2024-02-19T23:30:07.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:07.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:07.679817+0000) 2024-02-19T23:30:08.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:08.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:08.680076+0000) 2024-02-19T23:30:09.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:09.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:09.680369+0000) 2024-02-19T23:30:10.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:10.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:10.680675+0000) 2024-02-19T23:30:11.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:11.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:11.680906+0000) 2024-02-19T23:30:12.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:12.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:12.681132+0000) 2024-02-19T23:30:13.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:13.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:13.681403+0000) 2024-02-19T23:30:14.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:14.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:14.681673+0000) 2024-02-19T23:30:15.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:15.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:15.681914+0000) 2024-02-19T23:30:16.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:16.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:16.682231+0000) 2024-02-19T23:30:17.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:17.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:17.682409+0000) 2024-02-19T23:30:18.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:18.675+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:18.682617+0000) 2024-02-19T23:30:19.364 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:30:19.368 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:30:19.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:19.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:19.682824+0000) 2024-02-19T23:30:20.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:20.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:20.682990+0000) 2024-02-19T23:30:21.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:21.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:21.683171+0000) 2024-02-19T23:30:22.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:22.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:22.683389+0000) 2024-02-19T23:30:23.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:23.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:23.683598+0000) 2024-02-19T23:30:24.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:24.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:24.683847+0000) 2024-02-19T23:30:25.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:25.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:25.684078+0000) 2024-02-19T23:30:26.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:26.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:26.684310+0000) 2024-02-19T23:30:27.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:27.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:27.684443+0000) 2024-02-19T23:30:28.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:28.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:28.684619+0000) 2024-02-19T23:30:29.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:29.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:29.684827+0000) 2024-02-19T23:30:30.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:30.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:30.685078+0000) 2024-02-19T23:30:31.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:31.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:31.685278+0000) 2024-02-19T23:30:32.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:32.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:32.685516+0000) 2024-02-19T23:30:33.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:33.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:33.685773+0000) 2024-02-19T23:30:34.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:34.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:34.685965+0000) 2024-02-19T23:30:35.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:35.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:35.686141+0000) 2024-02-19T23:30:36.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:36.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:36.686380+0000) 2024-02-19T23:30:37.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:37.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:37.686569+0000) 2024-02-19T23:30:38.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:38.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:38.686828+0000) 2024-02-19T23:30:39.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:39.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:39.687008+0000) 2024-02-19T23:30:40.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:40.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:40.687180+0000) 2024-02-19T23:30:41.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:41.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:41.687434+0000) 2024-02-19T23:30:42.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:42.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:42.687654+0000) 2024-02-19T23:30:43.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:43.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:43.687828+0000) 2024-02-19T23:30:44.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:44.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:44.687988+0000) 2024-02-19T23:30:45.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:45.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:45.688241+0000) 2024-02-19T23:30:46.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:46.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:46.688436+0000) 2024-02-19T23:30:47.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:47.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:47.688591+0000) 2024-02-19T23:30:48.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:48.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:48.688772+0000) 2024-02-19T23:30:49.503 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:30:49.507 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:30:49.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:49.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:49.689042+0000) 2024-02-19T23:30:50.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:50.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:50.689342+0000) 2024-02-19T23:30:51.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:51.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:51.689580+0000) 2024-02-19T23:30:52.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:52.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:52.689752+0000) 2024-02-19T23:30:53.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:53.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:53.689991+0000) 2024-02-19T23:30:54.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:54.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:54.690235+0000) 2024-02-19T23:30:55.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:55.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:55.690386+0000) 2024-02-19T23:30:56.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:56.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:56.690589+0000) 2024-02-19T23:30:57.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:57.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:57.690856+0000) 2024-02-19T23:30:58.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:58.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:58.691069+0000) 2024-02-19T23:30:59.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:30:59.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:30:59.691222+0000) 2024-02-19T23:31:00.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:00.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:00.691422+0000) 2024-02-19T23:31:01.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:01.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:01.691643+0000) 2024-02-19T23:31:02.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:02.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:02.691920+0000) 2024-02-19T23:31:03.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:03.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:03.692061+0000) 2024-02-19T23:31:04.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:04.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:04.692336+0000) 2024-02-19T23:31:05.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:05.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:05.692499+0000) 2024-02-19T23:31:06.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:06.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:06.692656+0000) 2024-02-19T23:31:07.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:07.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:07.692843+0000) 2024-02-19T23:31:08.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:08.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:08.692993+0000) 2024-02-19T23:31:09.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:09.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:09.693161+0000) 2024-02-19T23:31:10.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:10.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:10.693353+0000) 2024-02-19T23:31:11.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:11.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:11.693491+0000) 2024-02-19T23:31:12.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:12.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:12.693650+0000) 2024-02-19T23:31:13.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:13.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:13.693775+0000) 2024-02-19T23:31:14.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:14.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:14.693908+0000) 2024-02-19T23:31:15.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:15.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:15.694132+0000) 2024-02-19T23:31:16.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:16.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:16.694311+0000) 2024-02-19T23:31:17.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:17.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:17.694463+0000) 2024-02-19T23:31:18.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:18.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:18.694738+0000) 2024-02-19T23:31:19.633 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:31:19.637 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:31:19.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:19.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:19.694882+0000) 2024-02-19T23:31:20.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:20.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:20.695026+0000) 2024-02-19T23:31:21.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:21.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:21.695184+0000) 2024-02-19T23:31:22.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:22.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:22.695381+0000) 2024-02-19T23:31:23.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:23.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:23.695616+0000) 2024-02-19T23:31:24.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:24.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:24.695755+0000) 2024-02-19T23:31:25.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:25.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:25.695901+0000) 2024-02-19T23:31:26.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:26.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:26.696039+0000) 2024-02-19T23:31:27.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:27.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:27.696183+0000) 2024-02-19T23:31:28.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:28.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:28.696353+0000) 2024-02-19T23:31:29.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:29.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:29.696529+0000) 2024-02-19T23:31:30.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:30.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:30.696747+0000) 2024-02-19T23:31:31.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:31.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:31.696894+0000) 2024-02-19T23:31:32.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:32.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:32.697136+0000) 2024-02-19T23:31:33.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:33.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:33.697347+0000) 2024-02-19T23:31:34.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:34.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:34.697585+0000) 2024-02-19T23:31:35.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:35.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:35.697743+0000) 2024-02-19T23:31:36.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:36.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:36.697888+0000) 2024-02-19T23:31:37.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:37.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:37.698091+0000) 2024-02-19T23:31:38.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:38.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:38.698274+0000) 2024-02-19T23:31:39.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:39.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:39.698531+0000) 2024-02-19T23:31:40.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:40.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:40.698743+0000) 2024-02-19T23:31:41.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:41.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:41.698906+0000) 2024-02-19T23:31:42.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:42.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:42.699068+0000) 2024-02-19T23:31:43.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:43.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:43.699223+0000) 2024-02-19T23:31:44.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:44.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:44.699360+0000) 2024-02-19T23:31:45.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:45.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:45.699612+0000) 2024-02-19T23:31:46.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:46.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:46.699764+0000) 2024-02-19T23:31:47.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:47.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:47.699920+0000) 2024-02-19T23:31:48.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:48.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:48.700109+0000) 2024-02-19T23:31:49.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:49.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:49.700292+0000) 2024-02-19T23:31:49.773 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:31:49.776 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:31:50.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:50.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:50.700462+0000) 2024-02-19T23:31:51.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:51.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:51.700621+0000) 2024-02-19T23:31:52.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:52.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:52.700854+0000) 2024-02-19T23:31:53.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:53.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:53.701005+0000) 2024-02-19T23:31:54.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:54.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:54.701173+0000) 2024-02-19T23:31:55.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:55.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:55.701381+0000) 2024-02-19T23:31:56.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:56.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:56.701632+0000) 2024-02-19T23:31:57.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:57.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:57.701866+0000) 2024-02-19T23:31:58.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:58.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:58.702087+0000) 2024-02-19T23:31:59.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:31:59.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:31:59.702322+0000) 2024-02-19T23:32:00.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:00.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:00.702577+0000) 2024-02-19T23:32:01.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:01.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:01.702771+0000) 2024-02-19T23:32:02.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:02.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:02.702904+0000) 2024-02-19T23:32:03.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:03.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:03.703124+0000) 2024-02-19T23:32:04.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:04.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:04.703259+0000) 2024-02-19T23:32:05.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:05.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:05.703424+0000) 2024-02-19T23:32:06.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:06.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:06.703657+0000) 2024-02-19T23:32:07.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:07.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:07.703903+0000) 2024-02-19T23:32:08.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:08.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:08.704149+0000) 2024-02-19T23:32:09.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:09.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:09.704375+0000) 2024-02-19T23:32:10.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:10.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:10.704636+0000) 2024-02-19T23:32:11.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:11.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:11.704874+0000) 2024-02-19T23:32:12.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:12.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:12.705047+0000) 2024-02-19T23:32:13.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:13.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:13.705235+0000) 2024-02-19T23:32:14.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:14.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:14.705398+0000) 2024-02-19T23:32:15.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:15.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:15.705583+0000) 2024-02-19T23:32:16.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:16.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:16.705805+0000) 2024-02-19T23:32:17.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:17.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:17.705996+0000) 2024-02-19T23:32:18.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:18.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:18.706225+0000) 2024-02-19T23:32:19.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:19.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:19.706385+0000) 2024-02-19T23:32:19.828 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:32:19.832 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:32:20.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:20.701+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:20.706627+0000) 2024-02-19T23:32:21.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:21.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:21.706888+0000) 2024-02-19T23:32:22.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:22.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:22.707119+0000) 2024-02-19T23:32:23.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:23.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:23.707340+0000) 2024-02-19T23:32:24.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:24.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:24.707583+0000) 2024-02-19T23:32:25.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:25.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:25.707826+0000) 2024-02-19T23:32:26.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:26.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:26.708001+0000) 2024-02-19T23:32:27.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:27.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:27.708126+0000) 2024-02-19T23:32:28.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:28.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:28.708373+0000) 2024-02-19T23:32:29.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:29.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:29.708563+0000) 2024-02-19T23:32:30.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:30.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:30.708754+0000) 2024-02-19T23:32:31.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:31.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:31.708938+0000) 2024-02-19T23:32:32.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:32.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:32.709132+0000) 2024-02-19T23:32:33.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:33.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:33.709370+0000) 2024-02-19T23:32:34.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:34.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:34.709631+0000) 2024-02-19T23:32:35.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:35.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:35.709889+0000) 2024-02-19T23:32:36.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:36.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:36.710133+0000) 2024-02-19T23:32:37.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:37.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:37.710378+0000) 2024-02-19T23:32:38.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:38.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:38.710635+0000) 2024-02-19T23:32:39.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:39.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:39.710920+0000) 2024-02-19T23:32:40.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:40.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:40.711151+0000) 2024-02-19T23:32:41.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:41.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:41.711384+0000) 2024-02-19T23:32:42.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:42.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:42.711692+0000) 2024-02-19T23:32:43.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:43.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:43.711883+0000) 2024-02-19T23:32:44.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:44.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:44.712111+0000) 2024-02-19T23:32:45.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:45.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:45.712318+0000) 2024-02-19T23:32:46.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:46.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:46.712516+0000) 2024-02-19T23:32:47.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:47.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:47.712738+0000) 2024-02-19T23:32:48.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:48.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:48.712960+0000) 2024-02-19T23:32:49.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:49.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:49.713164+0000) 2024-02-19T23:32:49.942 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:32:49.945 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:32:50.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:50.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:50.713448+0000) 2024-02-19T23:32:51.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:51.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:51.713672+0000) 2024-02-19T23:32:52.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:52.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:52.713906+0000) 2024-02-19T23:32:53.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:53.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:53.714085+0000) 2024-02-19T23:32:54.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:54.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:54.714266+0000) 2024-02-19T23:32:55.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:55.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:55.714442+0000) 2024-02-19T23:32:56.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:56.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:56.714656+0000) 2024-02-19T23:32:57.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:57.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:57.714922+0000) 2024-02-19T23:32:58.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:58.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:58.715119+0000) 2024-02-19T23:32:59.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:32:59.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:32:59.715292+0000) 2024-02-19T23:33:00.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:00.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:00.715521+0000) 2024-02-19T23:33:01.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:01.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:01.715682+0000) 2024-02-19T23:33:02.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:02.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:02.715937+0000) 2024-02-19T23:33:03.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:03.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:03.716129+0000) 2024-02-19T23:33:04.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:04.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:04.716308+0000) 2024-02-19T23:33:05.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:05.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:05.716473+0000) 2024-02-19T23:33:06.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:06.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:06.716700+0000) 2024-02-19T23:33:07.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:07.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:07.716923+0000) 2024-02-19T23:33:08.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:08.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:08.717153+0000) 2024-02-19T23:33:09.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:09.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:09.717400+0000) 2024-02-19T23:33:10.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:10.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:10.717681+0000) 2024-02-19T23:33:11.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:11.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:11.717926+0000) 2024-02-19T23:33:12.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:12.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:12.718207+0000) 2024-02-19T23:33:13.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:13.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:13.718468+0000) 2024-02-19T23:33:14.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:14.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:14.718763+0000) 2024-02-19T23:33:15.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:15.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:15.718930+0000) 2024-02-19T23:33:16.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:16.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:16.719115+0000) 2024-02-19T23:33:17.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:17.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:17.719376+0000) 2024-02-19T23:33:18.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:18.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:18.719576+0000) 2024-02-19T23:33:19.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:19.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:19.719844+0000) 2024-02-19T23:33:20.056 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:33:20.059 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:33:20.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:20.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:20.720121+0000) 2024-02-19T23:33:21.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:21.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:21.720311+0000) 2024-02-19T23:33:22.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:22.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:22.720493+0000) 2024-02-19T23:33:23.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:23.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:23.720738+0000) 2024-02-19T23:33:24.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:24.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:24.720959+0000) 2024-02-19T23:33:25.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:25.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:25.721174+0000) 2024-02-19T23:33:26.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:26.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:26.721377+0000) 2024-02-19T23:33:27.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:27.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:27.721570+0000) 2024-02-19T23:33:28.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:28.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:28.721760+0000) 2024-02-19T23:33:29.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:29.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:29.721998+0000) 2024-02-19T23:33:30.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:30.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:30.722245+0000) 2024-02-19T23:33:31.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:31.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:31.722377+0000) 2024-02-19T23:33:32.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:32.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:32.722601+0000) 2024-02-19T23:33:33.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:33.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:33.722856+0000) 2024-02-19T23:33:34.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:34.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:34.723133+0000) 2024-02-19T23:33:35.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:35.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:35.723322+0000) 2024-02-19T23:33:36.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:36.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:36.723492+0000) 2024-02-19T23:33:37.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:37.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:37.723719+0000) 2024-02-19T23:33:38.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:38.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:38.723953+0000) 2024-02-19T23:33:39.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:39.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:39.724179+0000) 2024-02-19T23:33:40.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:40.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:40.724459+0000) 2024-02-19T23:33:41.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:41.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:41.724688+0000) 2024-02-19T23:33:42.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:42.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:42.724919+0000) 2024-02-19T23:33:43.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:43.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:43.725106+0000) 2024-02-19T23:33:44.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:44.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:44.725294+0000) 2024-02-19T23:33:45.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:45.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:45.725479+0000) 2024-02-19T23:33:46.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:46.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:46.725720+0000) 2024-02-19T23:33:47.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:47.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:47.726008+0000) 2024-02-19T23:33:48.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:48.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:48.726267+0000) 2024-02-19T23:33:49.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:49.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:49.726504+0000) 2024-02-19T23:33:50.179 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:33:50.182 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:33:50.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:50.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:50.726796+0000) 2024-02-19T23:33:51.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:51.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:51.726944+0000) 2024-02-19T23:33:52.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:52.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:52.727073+0000) 2024-02-19T23:33:53.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:53.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:53.727288+0000) 2024-02-19T23:33:54.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:54.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:54.727537+0000) 2024-02-19T23:33:55.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:55.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:55.727748+0000) 2024-02-19T23:33:56.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:56.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:56.728025+0000) 2024-02-19T23:33:57.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:57.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:57.728217+0000) 2024-02-19T23:33:58.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:58.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:58.728458+0000) 2024-02-19T23:33:59.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:33:59.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:33:59.728615+0000) 2024-02-19T23:34:00.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:00.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:00.728804+0000) 2024-02-19T23:34:01.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:01.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:01.728978+0000) 2024-02-19T23:34:02.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:02.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:02.729138+0000) 2024-02-19T23:34:03.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:03.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:03.729307+0000) 2024-02-19T23:34:04.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:04.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:04.729474+0000) 2024-02-19T23:34:05.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:05.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:05.729640+0000) 2024-02-19T23:34:06.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:06.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:06.729868+0000) 2024-02-19T23:34:07.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:07.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:07.730030+0000) 2024-02-19T23:34:08.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:08.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:08.730250+0000) 2024-02-19T23:34:09.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:09.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:09.730491+0000) 2024-02-19T23:34:10.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:10.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:10.730826+0000) 2024-02-19T23:34:11.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:11.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:11.730972+0000) 2024-02-19T23:34:12.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:12.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:12.731109+0000) 2024-02-19T23:34:13.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:13.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:13.731288+0000) 2024-02-19T23:34:14.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:14.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:14.731479+0000) 2024-02-19T23:34:15.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:15.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:15.731642+0000) 2024-02-19T23:34:16.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:16.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:16.731805+0000) 2024-02-19T23:34:17.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:17.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:17.731970+0000) 2024-02-19T23:34:18.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:18.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:18.732199+0000) 2024-02-19T23:34:19.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:19.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:19.732376+0000) 2024-02-19T23:34:20.258 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:34:20.262 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:34:20.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:20.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:20.732548+0000) 2024-02-19T23:34:21.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:21.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:21.732713+0000) 2024-02-19T23:34:22.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:22.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:22.732942+0000) 2024-02-19T23:34:23.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:23.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:23.733190+0000) 2024-02-19T23:34:24.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:24.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:24.733447+0000) 2024-02-19T23:34:25.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:25.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:25.733724+0000) 2024-02-19T23:34:26.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:26.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:26.733938+0000) 2024-02-19T23:34:27.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:27.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:27.734196+0000) 2024-02-19T23:34:28.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:28.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:28.734446+0000) 2024-02-19T23:34:29.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:29.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:29.734633+0000) 2024-02-19T23:34:30.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:30.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:30.734794+0000) 2024-02-19T23:34:31.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:31.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:31.735017+0000) 2024-02-19T23:34:32.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:32.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:32.735240+0000) 2024-02-19T23:34:33.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:33.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:33.735454+0000) 2024-02-19T23:34:34.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:34.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:34.735684+0000) 2024-02-19T23:34:35.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:35.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:35.735908+0000) 2024-02-19T23:34:36.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:36.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:36.736050+0000) 2024-02-19T23:34:37.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:37.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:37.736285+0000) 2024-02-19T23:34:38.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:38.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:38.736426+0000) 2024-02-19T23:34:39.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:39.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:39.736601+0000) 2024-02-19T23:34:40.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:40.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:40.736834+0000) 2024-02-19T23:34:41.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:41.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:41.737081+0000) 2024-02-19T23:34:42.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:42.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:42.737346+0000) 2024-02-19T23:34:43.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:43.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:43.737511+0000) 2024-02-19T23:34:44.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:44.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:44.737650+0000) 2024-02-19T23:34:45.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:45.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:45.737877+0000) 2024-02-19T23:34:46.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:46.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:46.738030+0000) 2024-02-19T23:34:47.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:47.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:47.738249+0000) 2024-02-19T23:34:48.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:48.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:48.738488+0000) 2024-02-19T23:34:49.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:49.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:49.738681+0000) 2024-02-19T23:34:50.323 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:34:50.326 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:34:50.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:50.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:50.738909+0000) 2024-02-19T23:34:51.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:51.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:51.739071+0000) 2024-02-19T23:34:52.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:52.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:52.739293+0000) 2024-02-19T23:34:53.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:53.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:53.739529+0000) 2024-02-19T23:34:54.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:54.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:54.739677+0000) 2024-02-19T23:34:55.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:55.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:55.739890+0000) 2024-02-19T23:34:56.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:56.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:56.740100+0000) 2024-02-19T23:34:57.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:57.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:57.740328+0000) 2024-02-19T23:34:58.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:58.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:58.740571+0000) 2024-02-19T23:34:59.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:34:59.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:34:59.740816+0000) 2024-02-19T23:35:00.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:00.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:00.741094+0000) 2024-02-19T23:35:01.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:01.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:01.741305+0000) 2024-02-19T23:35:02.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:02.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:02.741451+0000) 2024-02-19T23:35:03.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:03.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:03.741672+0000) 2024-02-19T23:35:04.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:04.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:04.741904+0000) 2024-02-19T23:35:05.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:05.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:05.742103+0000) 2024-02-19T23:35:06.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:06.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:06.742348+0000) 2024-02-19T23:35:07.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:07.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:07.742536+0000) 2024-02-19T23:35:08.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:08.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:08.742748+0000) 2024-02-19T23:35:09.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:09.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:09.743000+0000) 2024-02-19T23:35:10.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:10.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:10.743181+0000) 2024-02-19T23:35:11.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:11.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:11.743351+0000) 2024-02-19T23:35:12.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:12.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:12.743576+0000) 2024-02-19T23:35:13.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:13.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:13.743734+0000) 2024-02-19T23:35:14.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:14.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:14.743926+0000) 2024-02-19T23:35:15.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:15.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:15.744063+0000) 2024-02-19T23:35:16.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:16.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:16.744248+0000) 2024-02-19T23:35:17.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:17.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:17.744487+0000) 2024-02-19T23:35:18.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:18.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:18.744624+0000) 2024-02-19T23:35:19.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:19.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:19.744751+0000) 2024-02-19T23:35:20.453 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:35:20.458 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:35:20.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:20.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:20.744997+0000) 2024-02-19T23:35:21.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:21.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:21.745230+0000) 2024-02-19T23:35:22.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:22.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:22.745414+0000) 2024-02-19T23:35:23.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:23.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:23.745552+0000) 2024-02-19T23:35:24.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:24.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:24.745719+0000) 2024-02-19T23:35:25.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:25.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:25.745912+0000) 2024-02-19T23:35:26.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:26.738+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:26.746125+0000) 2024-02-19T23:35:27.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:27.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:27.746368+0000) 2024-02-19T23:35:28.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:28.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:28.746592+0000) 2024-02-19T23:35:29.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:29.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:29.746795+0000) 2024-02-19T23:35:30.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:30.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:30.747033+0000) 2024-02-19T23:35:31.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:31.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:31.747259+0000) 2024-02-19T23:35:32.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:32.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:32.747466+0000) 2024-02-19T23:35:33.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:33.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:33.747640+0000) 2024-02-19T23:35:34.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:34.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:34.747821+0000) 2024-02-19T23:35:35.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:35.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:35.748065+0000) 2024-02-19T23:35:36.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:36.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:36.748301+0000) 2024-02-19T23:35:37.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:37.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:37.748549+0000) 2024-02-19T23:35:38.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:38.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:38.748826+0000) 2024-02-19T23:35:39.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:39.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:39.748963+0000) 2024-02-19T23:35:40.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:40.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:40.749200+0000) 2024-02-19T23:35:41.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:41.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:41.749379+0000) 2024-02-19T23:35:42.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:42.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:42.749539+0000) 2024-02-19T23:35:43.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:43.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:43.749719+0000) 2024-02-19T23:35:44.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:44.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:44.749959+0000) 2024-02-19T23:35:45.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:45.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:45.750358+0000) 2024-02-19T23:35:46.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:46.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:46.750557+0000) 2024-02-19T23:35:47.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:47.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:47.750686+0000) 2024-02-19T23:35:48.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:48.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:48.750818+0000) 2024-02-19T23:35:49.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:49.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:49.750957+0000) 2024-02-19T23:35:50.592 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:35:50.596 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:35:50.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:50.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:50.751087+0000) 2024-02-19T23:35:51.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:51.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:51.751267+0000) 2024-02-19T23:35:52.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:52.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:52.751424+0000) 2024-02-19T23:35:53.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:53.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:53.751672+0000) 2024-02-19T23:35:54.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:54.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:54.751874+0000) 2024-02-19T23:35:55.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:55.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:55.752033+0000) 2024-02-19T23:35:56.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:56.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:56.752250+0000) 2024-02-19T23:35:57.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:57.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:57.752440+0000) 2024-02-19T23:35:58.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:58.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:58.752691+0000) 2024-02-19T23:35:59.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:35:59.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:35:59.752925+0000) 2024-02-19T23:36:00.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:00.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:00.753171+0000) 2024-02-19T23:36:01.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:01.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:01.753434+0000) 2024-02-19T23:36:02.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:02.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:02.753710+0000) 2024-02-19T23:36:03.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:03.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:03.753937+0000) 2024-02-19T23:36:04.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:04.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:04.754189+0000) 2024-02-19T23:36:05.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:05.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:05.754422+0000) 2024-02-19T23:36:06.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:06.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:06.754753+0000) 2024-02-19T23:36:07.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:07.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:07.754987+0000) 2024-02-19T23:36:08.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:08.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:08.755168+0000) 2024-02-19T23:36:09.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:09.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:09.755379+0000) 2024-02-19T23:36:10.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:10.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:10.755596+0000) 2024-02-19T23:36:11.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:11.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:11.755826+0000) 2024-02-19T23:36:12.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:12.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:12.756044+0000) 2024-02-19T23:36:13.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:13.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:13.756271+0000) 2024-02-19T23:36:14.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:14.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:14.756486+0000) 2024-02-19T23:36:15.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:15.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:15.756718+0000) 2024-02-19T23:36:16.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:16.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:16.756944+0000) 2024-02-19T23:36:17.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:17.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:17.757153+0000) 2024-02-19T23:36:18.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:18.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:18.757401+0000) 2024-02-19T23:36:19.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:19.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:19.757604+0000) 2024-02-19T23:36:20.723 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:36:20.728 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:36:20.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:20.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:20.757875+0000) 2024-02-19T23:36:21.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:21.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:21.758070+0000) 2024-02-19T23:36:22.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:22.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:22.758349+0000) 2024-02-19T23:36:23.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:23.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:23.758577+0000) 2024-02-19T23:36:24.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:24.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:24.758890+0000) 2024-02-19T23:36:25.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:25.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:25.759027+0000) 2024-02-19T23:36:26.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:26.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:26.759219+0000) 2024-02-19T23:36:27.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:27.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:27.759402+0000) 2024-02-19T23:36:28.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:28.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:28.759599+0000) 2024-02-19T23:36:29.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:29.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:29.759823+0000) 2024-02-19T23:36:30.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:30.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:30.760065+0000) 2024-02-19T23:36:31.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:31.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:31.760210+0000) 2024-02-19T23:36:32.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:32.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:32.760377+0000) 2024-02-19T23:36:33.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:33.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:33.760514+0000) 2024-02-19T23:36:34.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:34.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:34.760632+0000) 2024-02-19T23:36:35.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:35.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:35.760839+0000) 2024-02-19T23:36:36.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:36.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:36.761069+0000) 2024-02-19T23:36:37.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:37.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:37.761295+0000) 2024-02-19T23:36:38.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:38.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:38.761480+0000) 2024-02-19T23:36:39.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:39.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:39.761689+0000) 2024-02-19T23:36:40.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:40.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:40.761842+0000) 2024-02-19T23:36:41.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:41.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:41.761992+0000) 2024-02-19T23:36:42.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:42.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:42.762115+0000) 2024-02-19T23:36:43.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:43.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:43.762240+0000) 2024-02-19T23:36:44.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:44.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:44.762402+0000) 2024-02-19T23:36:45.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:45.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:45.762582+0000) 2024-02-19T23:36:46.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:46.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:46.762732+0000) 2024-02-19T23:36:47.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:47.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:47.762860+0000) 2024-02-19T23:36:48.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:48.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:48.763038+0000) 2024-02-19T23:36:49.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:49.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:49.763160+0000) 2024-02-19T23:36:50.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:50.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:50.763355+0000) 2024-02-19T23:36:50.871 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:36:50.874 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:36:51.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:51.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:51.763595+0000) 2024-02-19T23:36:52.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:52.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:52.763857+0000) 2024-02-19T23:36:53.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:53.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:53.764090+0000) 2024-02-19T23:36:54.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:54.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:54.764323+0000) 2024-02-19T23:36:55.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:55.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:55.764593+0000) 2024-02-19T23:36:56.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:56.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:56.764873+0000) 2024-02-19T23:36:57.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:57.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:57.765108+0000) 2024-02-19T23:36:58.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:58.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:58.765334+0000) 2024-02-19T23:36:59.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:36:59.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:36:59.765558+0000) 2024-02-19T23:37:00.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:00.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:00.765790+0000) 2024-02-19T23:37:01.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:01.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:01.766025+0000) 2024-02-19T23:37:02.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:02.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:02.766252+0000) 2024-02-19T23:37:03.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:03.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:03.766443+0000) 2024-02-19T23:37:04.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:04.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:04.766611+0000) 2024-02-19T23:37:05.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:05.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:05.766791+0000) 2024-02-19T23:37:06.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:06.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:06.766990+0000) 2024-02-19T23:37:07.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:07.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:07.767142+0000) 2024-02-19T23:37:08.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:08.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:08.767356+0000) 2024-02-19T23:37:09.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:09.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:09.767513+0000) 2024-02-19T23:37:10.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:10.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:10.767683+0000) 2024-02-19T23:37:11.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:11.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:11.767847+0000) 2024-02-19T23:37:12.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:12.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:12.768083+0000) 2024-02-19T23:37:13.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:13.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:13.768344+0000) 2024-02-19T23:37:14.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:14.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:14.768532+0000) 2024-02-19T23:37:15.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:15.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:15.768755+0000) 2024-02-19T23:37:16.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:16.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:16.768959+0000) 2024-02-19T23:37:17.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:17.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:17.769151+0000) 2024-02-19T23:37:18.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:18.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:18.769291+0000) 2024-02-19T23:37:19.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:19.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:19.769519+0000) 2024-02-19T23:37:20.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:20.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:20.769709+0000) 2024-02-19T23:37:21.001 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:37:21.004 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:37:21.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:21.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:21.769897+0000) 2024-02-19T23:37:22.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:22.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:22.770036+0000) 2024-02-19T23:37:23.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:23.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:23.770200+0000) 2024-02-19T23:37:24.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:24.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:24.770415+0000) 2024-02-19T23:37:25.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:25.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:25.770645+0000) 2024-02-19T23:37:26.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:26.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:26.770922+0000) 2024-02-19T23:37:27.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:27.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:27.771151+0000) 2024-02-19T23:37:28.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:28.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:28.771432+0000) 2024-02-19T23:37:29.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:29.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:29.771631+0000) 2024-02-19T23:37:30.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:30.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:30.771865+0000) 2024-02-19T23:37:31.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:31.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:31.772123+0000) 2024-02-19T23:37:32.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:32.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:32.772371+0000) 2024-02-19T23:37:33.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:33.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:33.772620+0000) 2024-02-19T23:37:34.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:34.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:34.772852+0000) 2024-02-19T23:37:35.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:35.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:35.773027+0000) 2024-02-19T23:37:36.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:36.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:36.773215+0000) 2024-02-19T23:37:37.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:37.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:37.773393+0000) 2024-02-19T23:37:38.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:38.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:38.773569+0000) 2024-02-19T23:37:39.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:39.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:39.773806+0000) 2024-02-19T23:37:40.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:40.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:40.774027+0000) 2024-02-19T23:37:41.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:41.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:41.774245+0000) 2024-02-19T23:37:42.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:42.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:42.774506+0000) 2024-02-19T23:37:43.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:43.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:43.774728+0000) 2024-02-19T23:37:44.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:44.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:44.774889+0000) 2024-02-19T23:37:45.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:45.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:45.775031+0000) 2024-02-19T23:37:46.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:46.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:46.775209+0000) 2024-02-19T23:37:47.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:47.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:47.775389+0000) 2024-02-19T23:37:48.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:48.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:48.775558+0000) 2024-02-19T23:37:49.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:49.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:49.775766+0000) 2024-02-19T23:37:50.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:50.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:50.775935+0000) 2024-02-19T23:37:51.072 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:37:51.075 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:37:51.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:51.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:51.776167+0000) 2024-02-19T23:37:52.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:52.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:52.776407+0000) 2024-02-19T23:37:53.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:53.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:53.776629+0000) 2024-02-19T23:37:54.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:54.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:54.776818+0000) 2024-02-19T23:37:55.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:55.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:55.777019+0000) 2024-02-19T23:37:56.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:56.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:56.777283+0000) 2024-02-19T23:37:57.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:57.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:57.777535+0000) 2024-02-19T23:37:58.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:58.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:58.777817+0000) 2024-02-19T23:37:59.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:37:59.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:37:59.778071+0000) 2024-02-19T23:38:00.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:00.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:00.778328+0000) 2024-02-19T23:38:01.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:01.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:01.778525+0000) 2024-02-19T23:38:02.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:02.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:02.778758+0000) 2024-02-19T23:38:03.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:03.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:03.779011+0000) 2024-02-19T23:38:04.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:04.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:04.779230+0000) 2024-02-19T23:38:05.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:05.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:05.779467+0000) 2024-02-19T23:38:06.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:06.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:06.779681+0000) 2024-02-19T23:38:07.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:07.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:07.779901+0000) 2024-02-19T23:38:08.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:08.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:08.780134+0000) 2024-02-19T23:38:09.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:09.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:09.780386+0000) 2024-02-19T23:38:10.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:10.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:10.780706+0000) 2024-02-19T23:38:11.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:11.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:11.780892+0000) 2024-02-19T23:38:12.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:12.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:12.781109+0000) 2024-02-19T23:38:13.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:13.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:13.781247+0000) 2024-02-19T23:38:14.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:14.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:14.781453+0000) 2024-02-19T23:38:15.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:15.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:15.781675+0000) 2024-02-19T23:38:16.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:16.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:16.781909+0000) 2024-02-19T23:38:17.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:17.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:17.782106+0000) 2024-02-19T23:38:18.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:18.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:18.782228+0000) 2024-02-19T23:38:19.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:19.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:19.782356+0000) 2024-02-19T23:38:20.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:20.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:20.782583+0000) 2024-02-19T23:38:21.212 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:38:21.217 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:38:21.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:21.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:21.782874+0000) 2024-02-19T23:38:22.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:22.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:22.783015+0000) 2024-02-19T23:38:23.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:23.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:23.783234+0000) 2024-02-19T23:38:24.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:24.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:24.783494+0000) 2024-02-19T23:38:25.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:25.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:25.783731+0000) 2024-02-19T23:38:26.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:26.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:26.784006+0000) 2024-02-19T23:38:27.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:27.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:27.784177+0000) 2024-02-19T23:38:28.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:28.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:28.784341+0000) 2024-02-19T23:38:29.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:29.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:29.784543+0000) 2024-02-19T23:38:30.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:30.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:30.784773+0000) 2024-02-19T23:38:31.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:31.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:31.785009+0000) 2024-02-19T23:38:32.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:32.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:32.785336+0000) 2024-02-19T23:38:33.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:33.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:33.785545+0000) 2024-02-19T23:38:34.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:34.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:34.785766+0000) 2024-02-19T23:38:35.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:35.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:35.785995+0000) 2024-02-19T23:38:36.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:36.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:36.786220+0000) 2024-02-19T23:38:37.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:37.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:37.786381+0000) 2024-02-19T23:38:38.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:38.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:38.786540+0000) 2024-02-19T23:38:39.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:39.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:39.786852+0000) 2024-02-19T23:38:40.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:40.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:40.787071+0000) 2024-02-19T23:38:41.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:41.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:41.787243+0000) 2024-02-19T23:38:42.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:42.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:42.787488+0000) 2024-02-19T23:38:43.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:43.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:43.787727+0000) 2024-02-19T23:38:44.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:44.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:44.788033+0000) 2024-02-19T23:38:45.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:45.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:45.788224+0000) 2024-02-19T23:38:46.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:46.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:46.788472+0000) 2024-02-19T23:38:47.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:47.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:47.788683+0000) 2024-02-19T23:38:48.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:48.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:48.788959+0000) 2024-02-19T23:38:49.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:49.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:49.789229+0000) 2024-02-19T23:38:50.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:50.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:50.789497+0000) 2024-02-19T23:38:51.274 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:38:51.278 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:38:51.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:51.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:51.789667+0000) 2024-02-19T23:38:52.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:52.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:52.789934+0000) 2024-02-19T23:38:53.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:53.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:53.790119+0000) 2024-02-19T23:38:54.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:54.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:54.790277+0000) 2024-02-19T23:38:55.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:55.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:55.790496+0000) 2024-02-19T23:38:56.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:56.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:56.790759+0000) 2024-02-19T23:38:57.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:57.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:57.790930+0000) 2024-02-19T23:38:58.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:58.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:58.791172+0000) 2024-02-19T23:38:59.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:38:59.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:38:59.791385+0000) 2024-02-19T23:39:00.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:00.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:00.791674+0000) 2024-02-19T23:39:01.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:01.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:01.791920+0000) 2024-02-19T23:39:02.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:02.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:02.792200+0000) 2024-02-19T23:39:03.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:03.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:03.792369+0000) 2024-02-19T23:39:04.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:04.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:04.792607+0000) 2024-02-19T23:39:05.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:05.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:05.792767+0000) 2024-02-19T23:39:06.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:06.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:06.792921+0000) 2024-02-19T23:39:07.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:07.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:07.793166+0000) 2024-02-19T23:39:08.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:08.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:08.793423+0000) 2024-02-19T23:39:09.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:09.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:09.793654+0000) 2024-02-19T23:39:10.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:10.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:10.793906+0000) 2024-02-19T23:39:11.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:11.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:11.794110+0000) 2024-02-19T23:39:12.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:12.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:12.794354+0000) 2024-02-19T23:39:13.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:13.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:13.794550+0000) 2024-02-19T23:39:14.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:14.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:14.794873+0000) 2024-02-19T23:39:15.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:15.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:15.795055+0000) 2024-02-19T23:39:16.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:16.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:16.795202+0000) 2024-02-19T23:39:17.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:17.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:17.795445+0000) 2024-02-19T23:39:18.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:18.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:18.795639+0000) 2024-02-19T23:39:19.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:19.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:19.795906+0000) 2024-02-19T23:39:20.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:20.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:20.796196+0000) 2024-02-19T23:39:21.357 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:39:21.361 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:39:21.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:21.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:21.796360+0000) 2024-02-19T23:39:22.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:22.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:22.796567+0000) 2024-02-19T23:39:23.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:23.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:23.796693+0000) 2024-02-19T23:39:24.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:24.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:24.796930+0000) 2024-02-19T23:39:25.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:25.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:25.797158+0000) 2024-02-19T23:39:26.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:26.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:26.797378+0000) 2024-02-19T23:39:27.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:27.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:27.797579+0000) 2024-02-19T23:39:28.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:28.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:28.797866+0000) 2024-02-19T23:39:29.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:29.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:29.798114+0000) 2024-02-19T23:39:30.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:30.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:30.798276+0000) 2024-02-19T23:39:31.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:31.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:31.798515+0000) 2024-02-19T23:39:32.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:32.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:32.798813+0000) 2024-02-19T23:39:33.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:33.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:33.799089+0000) 2024-02-19T23:39:34.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:34.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:34.799280+0000) 2024-02-19T23:39:35.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:35.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:35.799441+0000) 2024-02-19T23:39:36.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:36.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:36.799601+0000) 2024-02-19T23:39:37.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:37.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:37.799760+0000) 2024-02-19T23:39:38.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:38.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:38.799920+0000) 2024-02-19T23:39:39.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:39.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:39.800107+0000) 2024-02-19T23:39:40.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:40.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:40.800340+0000) 2024-02-19T23:39:41.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:41.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:41.800596+0000) 2024-02-19T23:39:42.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:42.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:42.800799+0000) 2024-02-19T23:39:43.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:43.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:43.801025+0000) 2024-02-19T23:39:44.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:44.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:44.801171+0000) 2024-02-19T23:39:45.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:45.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:45.801342+0000) 2024-02-19T23:39:46.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:46.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:46.801495+0000) 2024-02-19T23:39:47.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:47.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:47.801730+0000) 2024-02-19T23:39:48.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:48.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:48.801983+0000) 2024-02-19T23:39:49.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:49.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:49.802199+0000) 2024-02-19T23:39:50.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:50.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:50.802361+0000) 2024-02-19T23:39:51.421 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:39:51.424 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:39:51.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:51.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:51.802653+0000) 2024-02-19T23:39:52.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:52.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:52.802863+0000) 2024-02-19T23:39:53.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:53.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:53.803007+0000) 2024-02-19T23:39:54.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:54.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:54.803163+0000) 2024-02-19T23:39:55.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:55.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:55.803393+0000) 2024-02-19T23:39:56.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:56.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:56.803664+0000) 2024-02-19T23:39:57.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:57.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:57.803815+0000) 2024-02-19T23:39:58.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:58.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:58.803954+0000) 2024-02-19T23:39:59.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:39:59.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:39:59.804106+0000) 2024-02-19T23:40:00.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:00.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:00.804273+0000) 2024-02-19T23:40:01.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:01.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:01.804424+0000) 2024-02-19T23:40:02.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:02.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:02.804616+0000) 2024-02-19T23:40:03.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:03.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:03.804780+0000) 2024-02-19T23:40:04.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:04.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:04.805010+0000) 2024-02-19T23:40:05.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:05.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:05.805256+0000) 2024-02-19T23:40:06.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:06.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:06.805530+0000) 2024-02-19T23:40:07.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:07.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:07.805695+0000) 2024-02-19T23:40:08.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:08.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:08.805939+0000) 2024-02-19T23:40:09.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:09.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:09.806226+0000) 2024-02-19T23:40:10.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:10.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:10.806477+0000) 2024-02-19T23:40:11.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:11.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:11.806659+0000) 2024-02-19T23:40:12.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:12.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:12.806790+0000) 2024-02-19T23:40:13.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:13.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:13.806940+0000) 2024-02-19T23:40:14.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:14.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:14.807089+0000) 2024-02-19T23:40:15.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:15.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:15.807222+0000) 2024-02-19T23:40:16.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:16.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:16.807462+0000) 2024-02-19T23:40:17.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:17.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:17.807652+0000) 2024-02-19T23:40:18.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:18.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:18.807796+0000) 2024-02-19T23:40:19.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:19.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:19.807965+0000) 2024-02-19T23:40:20.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:20.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:20.808233+0000) 2024-02-19T23:40:21.560 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:40:21.563 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:40:21.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:21.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:21.808403+0000) 2024-02-19T23:40:22.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:22.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:22.808651+0000) 2024-02-19T23:40:23.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:23.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:23.808883+0000) 2024-02-19T23:40:24.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:24.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:24.809036+0000) 2024-02-19T23:40:25.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:25.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:25.809268+0000) 2024-02-19T23:40:26.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:26.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:26.809516+0000) 2024-02-19T23:40:27.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:27.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:27.809728+0000) 2024-02-19T23:40:28.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:28.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:28.809943+0000) 2024-02-19T23:40:29.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:29.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:29.810085+0000) 2024-02-19T23:40:30.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:30.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:30.810237+0000) 2024-02-19T23:40:31.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:31.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:31.810447+0000) 2024-02-19T23:40:32.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:32.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:32.810616+0000) 2024-02-19T23:40:33.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:33.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:33.810807+0000) 2024-02-19T23:40:34.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:34.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:34.810992+0000) 2024-02-19T23:40:35.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:35.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:35.811181+0000) 2024-02-19T23:40:36.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:36.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:36.811441+0000) 2024-02-19T23:40:37.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:37.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:37.811707+0000) 2024-02-19T23:40:38.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:38.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:38.811926+0000) 2024-02-19T23:40:39.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:39.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:39.812101+0000) 2024-02-19T23:40:40.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:40.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:40.812305+0000) 2024-02-19T23:40:41.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:41.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:41.812530+0000) 2024-02-19T23:40:42.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:42.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:42.812756+0000) 2024-02-19T23:40:43.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:43.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:43.812896+0000) 2024-02-19T23:40:44.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:44.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:44.813032+0000) 2024-02-19T23:40:45.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:45.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:45.813220+0000) 2024-02-19T23:40:46.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:46.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:46.813366+0000) 2024-02-19T23:40:47.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:47.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:47.813603+0000) 2024-02-19T23:40:48.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:48.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:48.813856+0000) 2024-02-19T23:40:49.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:49.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:49.814067+0000) 2024-02-19T23:40:50.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:50.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:50.814331+0000) 2024-02-19T23:40:51.616 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:40:51.619 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:40:51.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:51.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:51.814560+0000) 2024-02-19T23:40:52.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:52.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:52.814887+0000) 2024-02-19T23:40:53.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:53.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:53.815130+0000) 2024-02-19T23:40:54.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:54.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:54.815424+0000) 2024-02-19T23:40:55.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:55.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:55.815682+0000) 2024-02-19T23:40:56.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:56.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:56.815891+0000) 2024-02-19T23:40:57.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:57.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:57.816112+0000) 2024-02-19T23:40:58.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:58.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:58.816361+0000) 2024-02-19T23:40:59.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:40:59.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:40:59.816519+0000) 2024-02-19T23:41:00.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:00.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:00.816783+0000) 2024-02-19T23:41:01.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:01.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:01.817070+0000) 2024-02-19T23:41:02.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:02.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:02.817300+0000) 2024-02-19T23:41:03.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:03.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:03.817474+0000) 2024-02-19T23:41:04.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:04.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:04.817731+0000) 2024-02-19T23:41:05.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:05.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:05.817988+0000) 2024-02-19T23:41:06.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:06.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:06.818252+0000) 2024-02-19T23:41:07.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:07.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:07.818404+0000) 2024-02-19T23:41:08.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:08.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:08.818560+0000) 2024-02-19T23:41:09.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:09.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:09.818869+0000) 2024-02-19T23:41:10.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:10.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:10.819077+0000) 2024-02-19T23:41:11.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:11.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:11.819233+0000) 2024-02-19T23:41:12.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:12.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:12.819386+0000) 2024-02-19T23:41:13.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:13.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:13.819534+0000) 2024-02-19T23:41:14.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:14.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:14.819762+0000) 2024-02-19T23:41:15.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:15.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:15.819959+0000) 2024-02-19T23:41:16.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:16.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:16.820109+0000) 2024-02-19T23:41:17.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:17.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:17.820257+0000) 2024-02-19T23:41:18.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:18.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:18.820496+0000) 2024-02-19T23:41:19.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:19.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:19.820786+0000) 2024-02-19T23:41:20.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:20.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:20.820969+0000) 2024-02-19T23:41:21.753 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:41:21.758 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:41:21.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:21.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:21.821190+0000) 2024-02-19T23:41:22.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:22.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:22.821367+0000) 2024-02-19T23:41:23.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:23.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:23.821542+0000) 2024-02-19T23:41:24.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:24.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:24.821705+0000) 2024-02-19T23:41:25.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:25.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:25.821859+0000) 2024-02-19T23:41:26.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:26.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:26.822013+0000) 2024-02-19T23:41:27.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:27.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:27.822189+0000) 2024-02-19T23:41:28.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:28.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:28.822417+0000) 2024-02-19T23:41:29.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:29.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:29.822577+0000) 2024-02-19T23:41:30.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:30.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:30.822771+0000) 2024-02-19T23:41:31.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:31.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:31.822931+0000) 2024-02-19T23:41:32.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:32.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:32.823116+0000) 2024-02-19T23:41:33.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:33.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:33.823292+0000) 2024-02-19T23:41:34.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:34.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:34.823429+0000) 2024-02-19T23:41:35.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:35.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:35.823671+0000) 2024-02-19T23:41:36.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:36.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:36.823834+0000) 2024-02-19T23:41:37.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:37.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:37.824033+0000) 2024-02-19T23:41:38.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:38.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:38.824221+0000) 2024-02-19T23:41:39.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:39.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:39.824376+0000) 2024-02-19T23:41:40.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:40.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:40.824527+0000) 2024-02-19T23:41:41.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:41.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:41.824671+0000) 2024-02-19T23:41:42.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:42.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:42.824814+0000) 2024-02-19T23:41:43.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:43.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:43.824956+0000) 2024-02-19T23:41:44.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:44.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:44.825135+0000) 2024-02-19T23:41:45.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:45.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:45.825296+0000) 2024-02-19T23:41:46.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:46.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:46.825473+0000) 2024-02-19T23:41:47.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:47.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:47.825638+0000) 2024-02-19T23:41:48.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:48.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:48.825783+0000) 2024-02-19T23:41:49.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:49.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:49.825928+0000) 2024-02-19T23:41:50.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:50.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:50.826065+0000) 2024-02-19T23:41:51.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:51.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:51.826302+0000) 2024-02-19T23:41:52.109 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:41:52.113 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:41:52.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:52.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:52.826462+0000) 2024-02-19T23:41:53.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:53.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:53.826678+0000) 2024-02-19T23:41:54.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:54.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:54.826849+0000) 2024-02-19T23:41:55.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:55.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:55.827059+0000) 2024-02-19T23:41:56.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:56.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:56.827281+0000) 2024-02-19T23:41:57.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:57.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:57.827524+0000) 2024-02-19T23:41:58.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:58.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:58.827760+0000) 2024-02-19T23:41:59.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:41:59.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:41:59.827974+0000) 2024-02-19T23:42:00.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:00.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:00.828258+0000) 2024-02-19T23:42:01.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:01.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:01.828560+0000) 2024-02-19T23:42:02.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:02.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:02.828721+0000) 2024-02-19T23:42:03.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:03.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:03.828951+0000) 2024-02-19T23:42:04.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:04.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:04.829128+0000) 2024-02-19T23:42:05.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:05.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:05.829304+0000) 2024-02-19T23:42:06.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:06.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:06.829602+0000) 2024-02-19T23:42:07.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:07.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:07.829846+0000) 2024-02-19T23:42:08.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:08.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:08.830038+0000) 2024-02-19T23:42:09.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:09.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:09.830283+0000) 2024-02-19T23:42:10.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:10.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:10.830577+0000) 2024-02-19T23:42:11.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:11.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:11.830857+0000) 2024-02-19T23:42:12.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:12.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:12.831106+0000) 2024-02-19T23:42:13.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:13.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:13.831277+0000) 2024-02-19T23:42:14.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:14.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:14.831542+0000) 2024-02-19T23:42:15.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:15.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:15.831768+0000) 2024-02-19T23:42:16.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:16.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:16.832060+0000) 2024-02-19T23:42:17.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:17.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:17.832264+0000) 2024-02-19T23:42:18.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:18.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:18.832512+0000) 2024-02-19T23:42:19.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:19.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:19.832777+0000) 2024-02-19T23:42:20.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:20.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:20.833014+0000) 2024-02-19T23:42:21.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:21.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:21.833242+0000) 2024-02-19T23:42:22.231 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:42:22.235 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:42:22.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:22.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:22.833505+0000) 2024-02-19T23:42:23.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:23.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:23.833707+0000) 2024-02-19T23:42:24.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:24.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:24.833996+0000) 2024-02-19T23:42:25.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:25.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:25.834185+0000) 2024-02-19T23:42:26.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:26.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:26.834481+0000) 2024-02-19T23:42:27.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:27.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:27.834673+0000) 2024-02-19T23:42:28.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:28.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:28.834925+0000) 2024-02-19T23:42:29.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:29.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:29.835155+0000) 2024-02-19T23:42:30.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:30.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:30.835404+0000) 2024-02-19T23:42:31.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:31.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:31.835651+0000) 2024-02-19T23:42:32.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:32.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:32.835948+0000) 2024-02-19T23:42:33.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:33.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:33.836123+0000) 2024-02-19T23:42:34.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:34.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:34.836375+0000) 2024-02-19T23:42:35.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:35.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:35.836517+0000) 2024-02-19T23:42:36.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:36.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:36.836733+0000) 2024-02-19T23:42:37.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:37.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:37.836944+0000) 2024-02-19T23:42:38.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:38.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:38.837216+0000) 2024-02-19T23:42:39.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:39.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:39.837399+0000) 2024-02-19T23:42:40.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:40.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:40.837649+0000) 2024-02-19T23:42:41.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:41.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:41.837872+0000) 2024-02-19T23:42:42.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:42.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:42.838101+0000) 2024-02-19T23:42:43.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:43.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:43.838344+0000) 2024-02-19T23:42:44.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:44.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:44.838629+0000) 2024-02-19T23:42:45.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:45.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:45.838861+0000) 2024-02-19T23:42:46.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:46.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:46.839092+0000) 2024-02-19T23:42:47.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:47.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:47.839303+0000) 2024-02-19T23:42:48.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:48.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:48.839502+0000) 2024-02-19T23:42:49.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:49.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:49.839711+0000) 2024-02-19T23:42:50.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:50.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:50.840006+0000) 2024-02-19T23:42:51.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:51.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:51.840261+0000) 2024-02-19T23:42:52.362 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:42:52.365 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:42:52.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:52.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:52.840458+0000) 2024-02-19T23:42:53.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:53.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:53.840632+0000) 2024-02-19T23:42:54.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:54.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:54.840861+0000) 2024-02-19T23:42:55.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:55.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:55.841087+0000) 2024-02-19T23:42:56.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:56.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:56.841384+0000) 2024-02-19T23:42:57.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:57.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:57.841571+0000) 2024-02-19T23:42:58.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:58.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:58.841818+0000) 2024-02-19T23:42:59.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:42:59.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:42:59.842090+0000) 2024-02-19T23:43:00.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:00.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:00.842360+0000) 2024-02-19T23:43:01.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:01.835+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:01.842608+0000) 2024-02-19T23:43:02.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:02.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:02.842955+0000) 2024-02-19T23:43:03.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:03.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:03.843127+0000) 2024-02-19T23:43:04.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:04.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:04.843311+0000) 2024-02-19T23:43:05.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:05.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:05.843512+0000) 2024-02-19T23:43:06.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:06.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:06.843758+0000) 2024-02-19T23:43:07.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:07.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:07.844002+0000) 2024-02-19T23:43:08.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:08.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:08.844250+0000) 2024-02-19T23:43:09.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:09.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:09.844538+0000) 2024-02-19T23:43:10.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:10.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:10.844782+0000) 2024-02-19T23:43:11.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:11.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:11.845041+0000) 2024-02-19T23:43:12.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:12.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:12.845336+0000) 2024-02-19T23:43:13.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:13.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:13.845585+0000) 2024-02-19T23:43:14.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:14.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:14.845876+0000) 2024-02-19T23:43:15.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:15.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:15.846050+0000) 2024-02-19T23:43:16.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:16.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:16.846246+0000) 2024-02-19T23:43:17.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:17.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:17.846548+0000) 2024-02-19T23:43:18.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:18.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:18.846885+0000) 2024-02-19T23:43:19.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:19.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:19.847082+0000) 2024-02-19T23:43:20.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:20.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:20.847325+0000) 2024-02-19T23:43:21.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:21.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:21.847528+0000) 2024-02-19T23:43:22.477 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:43:22.482 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:43:22.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:22.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:22.847694+0000) 2024-02-19T23:43:23.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:23.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:23.847870+0000) 2024-02-19T23:43:24.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:24.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:24.848117+0000) 2024-02-19T23:43:25.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:25.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:25.848316+0000) 2024-02-19T23:43:26.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:26.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:26.848590+0000) 2024-02-19T23:43:27.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:27.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:27.848850+0000) 2024-02-19T23:43:28.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:28.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:28.849018+0000) 2024-02-19T23:43:29.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:29.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:29.849214+0000) 2024-02-19T23:43:30.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:30.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:30.849406+0000) 2024-02-19T23:43:31.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:31.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:31.849687+0000) 2024-02-19T23:43:32.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:32.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:32.849915+0000) 2024-02-19T23:43:33.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:33.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:33.850096+0000) 2024-02-19T23:43:34.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:34.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:34.850345+0000) 2024-02-19T23:43:35.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:35.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:35.850610+0000) 2024-02-19T23:43:36.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:36.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:36.850783+0000) 2024-02-19T23:43:37.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:37.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:37.851021+0000) 2024-02-19T23:43:38.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:38.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:38.851188+0000) 2024-02-19T23:43:39.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:39.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:39.851406+0000) 2024-02-19T23:43:40.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:40.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:40.851560+0000) 2024-02-19T23:43:41.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:41.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:41.851709+0000) 2024-02-19T23:43:42.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:42.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:42.851857+0000) 2024-02-19T23:43:43.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:43.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:43.852033+0000) 2024-02-19T23:43:44.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:44.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:44.852189+0000) 2024-02-19T23:43:45.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:45.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:45.852522+0000) 2024-02-19T23:43:46.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:46.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:46.852790+0000) 2024-02-19T23:43:47.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:47.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:47.852962+0000) 2024-02-19T23:43:48.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:48.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:48.853188+0000) 2024-02-19T23:43:49.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:49.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:49.853454+0000) 2024-02-19T23:43:50.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:50.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:50.853723+0000) 2024-02-19T23:43:51.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:51.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:51.853884+0000) 2024-02-19T23:43:52.582 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:43:52.586 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:43:52.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:52.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:52.854062+0000) 2024-02-19T23:43:53.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:53.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:53.854341+0000) 2024-02-19T23:43:54.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:54.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:54.854561+0000) 2024-02-19T23:43:55.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:55.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:55.854774+0000) 2024-02-19T23:43:56.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:56.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:56.854963+0000) 2024-02-19T23:43:57.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:57.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:57.855112+0000) 2024-02-19T23:43:58.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:58.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:58.855314+0000) 2024-02-19T23:43:59.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:43:59.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:43:59.855543+0000) 2024-02-19T23:44:00.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:00.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:00.855800+0000) 2024-02-19T23:44:01.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:01.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:01.855953+0000) 2024-02-19T23:44:02.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:02.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:02.856165+0000) 2024-02-19T23:44:03.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:03.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:03.856339+0000) 2024-02-19T23:44:04.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:04.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:04.856523+0000) 2024-02-19T23:44:05.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:05.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:05.856785+0000) 2024-02-19T23:44:06.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:06.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:06.856927+0000) 2024-02-19T23:44:07.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:07.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:07.857077+0000) 2024-02-19T23:44:08.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:08.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:08.857253+0000) 2024-02-19T23:44:09.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:09.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:09.857502+0000) 2024-02-19T23:44:10.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:10.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:10.857768+0000) 2024-02-19T23:44:11.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:11.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:11.857969+0000) 2024-02-19T23:44:12.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:12.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:12.858233+0000) 2024-02-19T23:44:13.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:13.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:13.858525+0000) 2024-02-19T23:44:14.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:14.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:14.858676+0000) 2024-02-19T23:44:15.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:15.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:15.858800+0000) 2024-02-19T23:44:16.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:16.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:16.859028+0000) 2024-02-19T23:44:17.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:17.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:17.859219+0000) 2024-02-19T23:44:18.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:18.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:18.859443+0000) 2024-02-19T23:44:19.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:19.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:19.859646+0000) 2024-02-19T23:44:20.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:20.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:20.859923+0000) 2024-02-19T23:44:21.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:21.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:21.860138+0000) 2024-02-19T23:44:22.654 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:44:22.658 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:44:22.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:22.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:22.860435+0000) 2024-02-19T23:44:23.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:23.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:23.860793+0000) 2024-02-19T23:44:24.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:24.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:24.861025+0000) 2024-02-19T23:44:25.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:25.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:25.861180+0000) 2024-02-19T23:44:26.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:26.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:26.861396+0000) 2024-02-19T23:44:27.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:27.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:27.861541+0000) 2024-02-19T23:44:28.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:28.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:28.861683+0000) 2024-02-19T23:44:29.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:29.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:29.861834+0000) 2024-02-19T23:44:30.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:30.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:30.861977+0000) 2024-02-19T23:44:31.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:31.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:31.862276+0000) 2024-02-19T23:44:32.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:32.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:32.862425+0000) 2024-02-19T23:44:33.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:33.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:33.862639+0000) 2024-02-19T23:44:34.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:34.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:34.862903+0000) 2024-02-19T23:44:35.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:35.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:35.863060+0000) 2024-02-19T23:44:36.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:36.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:36.863208+0000) 2024-02-19T23:44:37.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:37.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:37.863438+0000) 2024-02-19T23:44:38.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:38.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:38.863660+0000) 2024-02-19T23:44:39.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:39.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:39.863815+0000) 2024-02-19T23:44:40.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:40.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:40.863996+0000) 2024-02-19T23:44:41.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:41.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:41.864174+0000) 2024-02-19T23:44:42.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:42.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:42.864328+0000) 2024-02-19T23:44:43.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:43.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:43.864585+0000) 2024-02-19T23:44:44.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:44.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:44.864740+0000) 2024-02-19T23:44:45.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:45.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:45.864923+0000) 2024-02-19T23:44:46.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:46.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:46.865087+0000) 2024-02-19T23:44:47.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:47.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:47.865338+0000) 2024-02-19T23:44:48.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:48.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:48.865551+0000) 2024-02-19T23:44:49.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:49.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:49.865703+0000) 2024-02-19T23:44:50.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:50.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:50.865874+0000) 2024-02-19T23:44:51.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:51.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:51.866036+0000) 2024-02-19T23:44:52.776 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:44:52.781 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:44:52.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:52.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:52.866305+0000) 2024-02-19T23:44:53.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:53.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:53.866538+0000) 2024-02-19T23:44:54.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:54.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:54.866863+0000) 2024-02-19T23:44:55.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:55.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:55.867072+0000) 2024-02-19T23:44:56.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:56.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:56.867296+0000) 2024-02-19T23:44:57.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:57.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:57.867503+0000) 2024-02-19T23:44:58.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:58.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:58.867655+0000) 2024-02-19T23:44:59.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:44:59.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:44:59.867815+0000) 2024-02-19T23:45:00.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:00.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:00.868056+0000) 2024-02-19T23:45:01.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:01.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:01.868366+0000) 2024-02-19T23:45:02.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:02.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:02.868601+0000) 2024-02-19T23:45:03.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:03.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:03.868800+0000) 2024-02-19T23:45:04.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:04.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:04.869015+0000) 2024-02-19T23:45:05.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:05.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:05.869167+0000) 2024-02-19T23:45:06.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:06.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:06.869437+0000) 2024-02-19T23:45:07.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:07.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:07.869640+0000) 2024-02-19T23:45:08.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:08.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:08.869852+0000) 2024-02-19T23:45:09.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:09.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:09.870001+0000) 2024-02-19T23:45:10.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:10.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:10.870167+0000) 2024-02-19T23:45:11.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:11.865+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:11.870348+0000) 2024-02-19T23:45:12.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:12.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:12.870547+0000) 2024-02-19T23:45:13.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:13.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:13.870837+0000) 2024-02-19T23:45:14.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:14.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:14.870998+0000) 2024-02-19T23:45:15.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:15.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:15.871253+0000) 2024-02-19T23:45:16.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:16.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:16.871510+0000) 2024-02-19T23:45:17.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:17.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:17.871687+0000) 2024-02-19T23:45:18.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:18.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:18.871876+0000) 2024-02-19T23:45:19.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:19.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:19.872037+0000) 2024-02-19T23:45:20.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:20.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:20.872203+0000) 2024-02-19T23:45:21.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:21.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:21.872380+0000) 2024-02-19T23:45:22.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:22.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:22.872526+0000) 2024-02-19T23:45:22.889 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:45:22.893 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:45:23.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:23.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:23.872843+0000) 2024-02-19T23:45:24.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:24.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:24.873039+0000) 2024-02-19T23:45:25.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:25.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:25.873200+0000) 2024-02-19T23:45:26.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:26.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:26.873352+0000) 2024-02-19T23:45:27.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:27.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:27.873509+0000) 2024-02-19T23:45:28.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:28.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:28.873672+0000) 2024-02-19T23:45:29.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:29.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:29.873895+0000) 2024-02-19T23:45:30.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:30.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:30.874081+0000) 2024-02-19T23:45:31.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:31.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:31.874231+0000) 2024-02-19T23:45:32.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:32.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:32.874451+0000) 2024-02-19T23:45:33.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:33.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:33.874620+0000) 2024-02-19T23:45:34.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:34.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:34.874800+0000) 2024-02-19T23:45:35.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:35.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:35.875045+0000) 2024-02-19T23:45:36.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:36.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:36.875332+0000) 2024-02-19T23:45:37.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:37.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:37.875532+0000) 2024-02-19T23:45:38.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:38.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:38.875775+0000) 2024-02-19T23:45:39.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:39.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:39.876054+0000) 2024-02-19T23:45:40.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:40.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:40.876295+0000) 2024-02-19T23:45:41.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:41.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:41.876474+0000) 2024-02-19T23:45:42.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:42.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:42.876616+0000) 2024-02-19T23:45:43.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:43.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:43.876860+0000) 2024-02-19T23:45:44.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:44.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:44.877035+0000) 2024-02-19T23:45:45.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:45.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:45.877177+0000) 2024-02-19T23:45:46.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:46.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:46.877416+0000) 2024-02-19T23:45:47.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:47.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:47.877630+0000) 2024-02-19T23:45:48.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:48.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:48.877906+0000) 2024-02-19T23:45:49.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:49.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:49.878095+0000) 2024-02-19T23:45:50.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:50.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:50.878289+0000) 2024-02-19T23:45:51.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:51.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:51.878566+0000) 2024-02-19T23:45:52.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:52.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:52.878786+0000) 2024-02-19T23:45:52.962 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:45:52.965 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:45:53.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:53.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:53.879036+0000) 2024-02-19T23:45:54.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:54.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:54.879222+0000) 2024-02-19T23:45:55.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:55.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:55.879373+0000) 2024-02-19T23:45:56.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:56.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:56.879622+0000) 2024-02-19T23:45:57.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:57.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:57.879825+0000) 2024-02-19T23:45:58.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:58.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:58.880104+0000) 2024-02-19T23:45:59.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:45:59.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:45:59.880416+0000) 2024-02-19T23:46:00.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:00.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:00.880624+0000) 2024-02-19T23:46:01.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:01.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:01.880940+0000) 2024-02-19T23:46:02.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:02.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:02.881335+0000) 2024-02-19T23:46:03.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:03.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:03.881589+0000) 2024-02-19T23:46:04.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:04.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:04.881748+0000) 2024-02-19T23:46:05.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:05.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:05.881991+0000) 2024-02-19T23:46:06.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:06.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:06.882146+0000) 2024-02-19T23:46:07.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:07.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:07.882485+0000) 2024-02-19T23:46:08.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:08.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:08.882642+0000) 2024-02-19T23:46:09.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:09.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:09.882795+0000) 2024-02-19T23:46:10.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:10.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:10.883034+0000) 2024-02-19T23:46:11.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:11.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:11.883205+0000) 2024-02-19T23:46:12.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:12.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:12.883341+0000) 2024-02-19T23:46:13.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:13.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:13.883499+0000) 2024-02-19T23:46:14.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:14.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:14.883658+0000) 2024-02-19T23:46:15.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:15.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:15.883816+0000) 2024-02-19T23:46:16.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:16.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:16.884021+0000) 2024-02-19T23:46:17.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:17.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:17.884191+0000) 2024-02-19T23:46:18.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:18.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:18.884337+0000) 2024-02-19T23:46:19.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:19.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:19.884495+0000) 2024-02-19T23:46:20.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:20.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:20.884771+0000) 2024-02-19T23:46:21.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:21.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:21.885101+0000) 2024-02-19T23:46:22.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:22.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:22.885249+0000) 2024-02-19T23:46:23.076 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:46:23.080 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:46:23.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:23.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:23.885435+0000) 2024-02-19T23:46:24.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:24.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:24.885622+0000) 2024-02-19T23:46:25.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:25.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:25.885848+0000) 2024-02-19T23:46:26.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:26.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:26.886036+0000) 2024-02-19T23:46:27.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:27.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:27.886220+0000) 2024-02-19T23:46:28.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:28.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:28.886387+0000) 2024-02-19T23:46:29.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:29.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:29.886593+0000) 2024-02-19T23:46:30.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:30.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:30.886840+0000) 2024-02-19T23:46:31.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:31.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:31.886997+0000) 2024-02-19T23:46:32.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:32.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:32.887199+0000) 2024-02-19T23:46:33.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:33.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:33.887396+0000) 2024-02-19T23:46:34.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:34.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:34.887633+0000) 2024-02-19T23:46:35.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:35.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:35.887834+0000) 2024-02-19T23:46:36.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:36.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:36.888040+0000) 2024-02-19T23:46:37.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:37.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:37.888264+0000) 2024-02-19T23:46:38.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:38.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:38.888564+0000) 2024-02-19T23:46:39.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:39.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:39.888859+0000) 2024-02-19T23:46:40.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:40.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:40.889068+0000) 2024-02-19T23:46:41.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:41.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:41.889218+0000) 2024-02-19T23:46:42.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:42.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:42.889379+0000) 2024-02-19T23:46:43.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:43.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:43.889624+0000) 2024-02-19T23:46:44.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:44.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:44.889780+0000) 2024-02-19T23:46:45.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:45.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:45.889939+0000) 2024-02-19T23:46:46.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:46.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:46.890134+0000) 2024-02-19T23:46:47.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:47.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:47.890291+0000) 2024-02-19T23:46:48.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:48.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:48.890445+0000) 2024-02-19T23:46:49.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:49.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:49.890594+0000) 2024-02-19T23:46:50.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:50.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:50.890788+0000) 2024-02-19T23:46:51.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:51.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:51.890940+0000) 2024-02-19T23:46:52.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:52.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:52.891090+0000) 2024-02-19T23:46:53.207 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:46:53.211 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:46:53.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:53.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:53.891267+0000) 2024-02-19T23:46:54.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:54.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:54.891433+0000) 2024-02-19T23:46:55.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:55.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:55.891604+0000) 2024-02-19T23:46:56.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:56.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:56.891742+0000) 2024-02-19T23:46:57.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:57.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:57.891881+0000) 2024-02-19T23:46:58.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:58.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:58.892043+0000) 2024-02-19T23:46:59.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:46:59.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:46:59.892190+0000) 2024-02-19T23:47:00.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:00.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:00.892336+0000) 2024-02-19T23:47:01.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:01.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:01.892473+0000) 2024-02-19T23:47:02.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:02.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:02.892666+0000) 2024-02-19T23:47:03.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:03.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:03.892840+0000) 2024-02-19T23:47:04.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:04.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:04.893017+0000) 2024-02-19T23:47:05.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:05.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:05.893193+0000) 2024-02-19T23:47:06.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:06.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:06.893444+0000) 2024-02-19T23:47:07.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:07.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:07.893598+0000) 2024-02-19T23:47:08.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:08.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:08.893764+0000) 2024-02-19T23:47:09.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:09.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:09.893934+0000) 2024-02-19T23:47:10.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:10.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:10.894144+0000) 2024-02-19T23:47:11.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:11.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:11.894287+0000) 2024-02-19T23:47:12.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:12.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:12.894552+0000) 2024-02-19T23:47:13.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:13.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:13.894733+0000) 2024-02-19T23:47:14.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:14.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:14.894903+0000) 2024-02-19T23:47:15.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:15.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:15.895045+0000) 2024-02-19T23:47:16.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:16.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:16.895201+0000) 2024-02-19T23:47:17.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:17.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:17.895384+0000) 2024-02-19T23:47:18.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:18.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:18.895584+0000) 2024-02-19T23:47:19.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:19.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:19.895729+0000) 2024-02-19T23:47:20.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:20.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:20.895936+0000) 2024-02-19T23:47:21.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:21.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:21.896073+0000) 2024-02-19T23:47:22.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:22.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:22.896227+0000) 2024-02-19T23:47:23.345 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:47:23.349 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:47:23.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:23.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:23.896422+0000) 2024-02-19T23:47:24.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:24.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:24.896585+0000) 2024-02-19T23:47:25.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:25.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:25.896810+0000) 2024-02-19T23:47:26.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:26.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:26.896997+0000) 2024-02-19T23:47:27.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:27.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:27.897158+0000) 2024-02-19T23:47:28.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:28.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:28.897352+0000) 2024-02-19T23:47:29.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:29.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:29.897514+0000) 2024-02-19T23:47:30.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:30.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:30.897689+0000) 2024-02-19T23:47:31.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:31.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:31.897871+0000) 2024-02-19T23:47:32.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:32.891+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:32.898054+0000) 2024-02-19T23:47:33.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:33.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:33.898214+0000) 2024-02-19T23:47:34.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:34.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:34.898386+0000) 2024-02-19T23:47:35.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:35.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:35.898533+0000) 2024-02-19T23:47:36.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:36.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:36.898790+0000) 2024-02-19T23:47:37.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:37.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:37.898968+0000) 2024-02-19T23:47:38.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:38.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:38.899238+0000) 2024-02-19T23:47:39.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:39.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:39.899433+0000) 2024-02-19T23:47:40.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:40.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:40.899698+0000) 2024-02-19T23:47:41.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:41.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:41.899901+0000) 2024-02-19T23:47:42.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:42.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:42.900125+0000) 2024-02-19T23:47:43.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:43.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:43.900301+0000) 2024-02-19T23:47:44.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:44.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:44.900571+0000) 2024-02-19T23:47:45.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:45.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:45.900751+0000) 2024-02-19T23:47:46.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:46.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:46.900974+0000) 2024-02-19T23:47:47.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:47.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:47.901120+0000) 2024-02-19T23:47:48.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:48.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:48.901338+0000) 2024-02-19T23:47:49.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:49.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:49.901532+0000) 2024-02-19T23:47:50.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:50.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:50.901715+0000) 2024-02-19T23:47:51.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:51.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:51.901946+0000) 2024-02-19T23:47:52.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:52.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:52.902184+0000) 2024-02-19T23:47:53.480 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:47:53.483 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:47:53.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:53.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:53.902420+0000) 2024-02-19T23:47:54.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:54.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:54.902610+0000) 2024-02-19T23:47:55.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:55.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:55.902851+0000) 2024-02-19T23:47:56.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:56.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:56.903141+0000) 2024-02-19T23:47:57.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:57.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:57.903389+0000) 2024-02-19T23:47:58.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:58.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:58.903595+0000) 2024-02-19T23:47:59.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:47:59.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:47:59.903814+0000) 2024-02-19T23:48:00.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:00.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:00.904003+0000) 2024-02-19T23:48:01.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:01.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:01.904209+0000) 2024-02-19T23:48:02.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:02.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:02.904401+0000) 2024-02-19T23:48:03.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:03.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:03.904626+0000) 2024-02-19T23:48:04.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:04.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:04.904918+0000) 2024-02-19T23:48:05.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:05.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:05.905159+0000) 2024-02-19T23:48:06.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:06.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:06.905373+0000) 2024-02-19T23:48:07.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:07.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:07.905582+0000) 2024-02-19T23:48:08.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:08.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:08.905822+0000) 2024-02-19T23:48:09.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:09.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:09.906066+0000) 2024-02-19T23:48:10.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:10.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:10.906334+0000) 2024-02-19T23:48:11.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:11.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:11.906567+0000) 2024-02-19T23:48:12.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:12.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:12.906809+0000) 2024-02-19T23:48:13.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:13.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:13.906955+0000) 2024-02-19T23:48:14.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:14.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:14.907148+0000) 2024-02-19T23:48:15.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:15.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:15.907307+0000) 2024-02-19T23:48:16.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:16.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:16.907512+0000) 2024-02-19T23:48:17.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:17.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:17.907733+0000) 2024-02-19T23:48:18.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:18.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:18.908036+0000) 2024-02-19T23:48:19.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:19.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:19.908235+0000) 2024-02-19T23:48:20.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:20.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:20.908529+0000) 2024-02-19T23:48:21.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:21.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:21.908706+0000) 2024-02-19T23:48:22.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:22.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:22.908862+0000) 2024-02-19T23:48:23.597 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:48:23.603 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:48:23.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:23.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:23.909102+0000) 2024-02-19T23:48:24.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:24.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:24.909313+0000) 2024-02-19T23:48:25.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:25.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:25.909510+0000) 2024-02-19T23:48:26.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:26.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:26.909737+0000) 2024-02-19T23:48:27.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:27.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:27.909917+0000) 2024-02-19T23:48:28.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:28.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:28.910131+0000) 2024-02-19T23:48:29.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:29.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:29.910306+0000) 2024-02-19T23:48:30.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:30.910+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:30.910487+0000) 2024-02-19T23:48:31.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:31.910+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:31.910756+0000) 2024-02-19T23:48:32.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:32.910+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:32.910958+0000) 2024-02-19T23:48:33.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:33.910+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:33.911188+0000) 2024-02-19T23:48:34.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:34.910+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:34.911411+0000) 2024-02-19T23:48:35.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:35.910+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:35.911598+0000) 2024-02-19T23:48:36.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:36.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:36.911826+0000) 2024-02-19T23:48:37.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:37.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:37.912011+0000) 2024-02-19T23:48:38.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:38.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:38.912269+0000) 2024-02-19T23:48:39.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:39.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:39.912433+0000) 2024-02-19T23:48:40.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:40.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:40.912625+0000) 2024-02-19T23:48:41.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:41.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:41.912805+0000) 2024-02-19T23:48:42.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:42.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:42.912986+0000) 2024-02-19T23:48:43.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:43.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:43.913240+0000) 2024-02-19T23:48:44.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:44.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:44.913514+0000) 2024-02-19T23:48:45.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:45.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:45.913755+0000) 2024-02-19T23:48:46.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:46.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:46.913981+0000) 2024-02-19T23:48:47.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:47.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:47.914173+0000) 2024-02-19T23:48:48.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:48.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:48.914434+0000) 2024-02-19T23:48:49.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:49.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:49.914757+0000) 2024-02-19T23:48:50.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:50.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:50.915000+0000) 2024-02-19T23:48:51.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:51.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:51.915144+0000) 2024-02-19T23:48:52.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:52.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:52.915329+0000) 2024-02-19T23:48:53.746 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:48:53.750 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:48:53.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:53.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:53.915578+0000) 2024-02-19T23:48:54.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:54.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:54.915789+0000) 2024-02-19T23:48:55.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:55.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:55.916040+0000) 2024-02-19T23:48:56.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:56.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:56.916230+0000) 2024-02-19T23:48:57.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:57.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:57.916383+0000) 2024-02-19T23:48:58.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:58.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:58.916603+0000) 2024-02-19T23:48:59.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:48:59.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:48:59.916741+0000) 2024-02-19T23:49:00.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:00.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:00.916990+0000) 2024-02-19T23:49:01.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:01.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:01.917166+0000) 2024-02-19T23:49:02.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:02.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:02.917326+0000) 2024-02-19T23:49:03.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:03.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:03.917515+0000) 2024-02-19T23:49:04.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:04.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:04.917747+0000) 2024-02-19T23:49:05.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:05.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:05.917893+0000) 2024-02-19T23:49:06.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:06.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:06.918114+0000) 2024-02-19T23:49:07.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:07.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:07.918250+0000) 2024-02-19T23:49:08.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:08.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:08.918467+0000) 2024-02-19T23:49:09.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:09.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:09.918644+0000) 2024-02-19T23:49:10.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:10.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:10.918816+0000) 2024-02-19T23:49:11.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:11.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:11.918968+0000) 2024-02-19T23:49:12.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:12.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:12.919151+0000) 2024-02-19T23:49:13.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:13.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:13.919287+0000) 2024-02-19T23:49:14.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:14.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:14.919434+0000) 2024-02-19T23:49:15.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:15.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:15.919629+0000) 2024-02-19T23:49:16.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:16.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:16.919875+0000) 2024-02-19T23:49:17.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:17.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:17.920046+0000) 2024-02-19T23:49:18.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:18.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:18.920247+0000) 2024-02-19T23:49:19.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:19.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:19.920402+0000) 2024-02-19T23:49:20.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:20.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:20.920557+0000) 2024-02-19T23:49:21.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:21.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:21.920744+0000) 2024-02-19T23:49:22.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:22.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:22.920981+0000) 2024-02-19T23:49:23.827 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:49:23.832 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:49:23.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:23.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:23.921149+0000) 2024-02-19T23:49:24.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:24.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:24.921301+0000) 2024-02-19T23:49:25.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:25.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:25.921460+0000) 2024-02-19T23:49:26.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:26.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:26.921635+0000) 2024-02-19T23:49:27.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:27.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:27.921795+0000) 2024-02-19T23:49:28.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:28.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:28.922012+0000) 2024-02-19T23:49:29.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:29.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:29.922221+0000) 2024-02-19T23:49:30.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:30.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:30.922444+0000) 2024-02-19T23:49:31.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:31.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:31.922628+0000) 2024-02-19T23:49:32.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:32.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:32.922831+0000) 2024-02-19T23:49:33.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:33.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:33.922975+0000) 2024-02-19T23:49:34.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:34.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:34.923192+0000) 2024-02-19T23:49:35.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:35.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:35.923424+0000) 2024-02-19T23:49:36.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:36.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:36.923644+0000) 2024-02-19T23:49:37.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:37.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:37.923807+0000) 2024-02-19T23:49:38.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:38.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:38.923978+0000) 2024-02-19T23:49:39.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:39.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:39.924158+0000) 2024-02-19T23:49:40.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:40.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:40.924335+0000) 2024-02-19T23:49:41.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:41.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:41.924549+0000) 2024-02-19T23:49:42.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:42.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:42.924716+0000) 2024-02-19T23:49:43.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:43.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:43.924942+0000) 2024-02-19T23:49:44.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:44.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:44.925174+0000) 2024-02-19T23:49:45.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:45.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:45.925402+0000) 2024-02-19T23:49:46.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:46.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:46.925667+0000) 2024-02-19T23:49:47.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:47.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:47.925950+0000) 2024-02-19T23:49:48.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:48.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:48.926204+0000) 2024-02-19T23:49:49.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:49.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:49.926367+0000) 2024-02-19T23:49:50.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:50.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:50.926541+0000) 2024-02-19T23:49:51.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:51.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:51.926821+0000) 2024-02-19T23:49:52.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:52.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:52.927016+0000) 2024-02-19T23:49:53.930 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:49:53.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:53.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:53.927145+0000) 2024-02-19T23:49:53.934 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:49:54.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:54.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:54.927388+0000) 2024-02-19T23:49:55.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:55.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:55.927532+0000) 2024-02-19T23:49:56.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:56.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:56.927714+0000) 2024-02-19T23:49:57.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:57.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:57.927937+0000) 2024-02-19T23:49:58.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:58.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:58.928074+0000) 2024-02-19T23:49:59.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:49:59.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:49:59.928232+0000) 2024-02-19T23:50:00.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:00.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:00.928457+0000) 2024-02-19T23:50:01.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:01.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:01.928687+0000) 2024-02-19T23:50:02.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:02.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:02.928892+0000) 2024-02-19T23:50:03.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:03.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:03.929035+0000) 2024-02-19T23:50:04.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:04.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:04.929247+0000) 2024-02-19T23:50:05.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:05.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:05.929407+0000) 2024-02-19T23:50:06.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:06.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:06.929601+0000) 2024-02-19T23:50:07.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:07.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:07.929838+0000) 2024-02-19T23:50:08.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:08.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:08.930007+0000) 2024-02-19T23:50:09.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:09.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:09.930227+0000) 2024-02-19T23:50:10.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:10.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:10.930444+0000) 2024-02-19T23:50:11.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:11.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:11.930675+0000) 2024-02-19T23:50:12.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:12.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:12.930960+0000) 2024-02-19T23:50:13.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:13.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:13.931214+0000) 2024-02-19T23:50:14.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:14.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:14.931360+0000) 2024-02-19T23:50:15.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:15.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:15.931528+0000) 2024-02-19T23:50:16.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:16.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:16.931735+0000) 2024-02-19T23:50:17.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:17.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:17.931905+0000) 2024-02-19T23:50:18.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:18.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:18.932109+0000) 2024-02-19T23:50:19.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:19.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:19.932355+0000) 2024-02-19T23:50:20.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:20.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:20.932625+0000) 2024-02-19T23:50:21.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:21.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:21.932789+0000) 2024-02-19T23:50:22.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:22.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:22.933016+0000) 2024-02-19T23:50:23.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:23.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:23.933253+0000) 2024-02-19T23:50:24.029 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:50:24.032 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:50:24.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:24.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:24.933515+0000) 2024-02-19T23:50:25.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:25.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:25.933745+0000) 2024-02-19T23:50:26.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:26.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:26.933993+0000) 2024-02-19T23:50:27.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:27.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:27.934164+0000) 2024-02-19T23:50:28.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:28.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:28.934419+0000) 2024-02-19T23:50:29.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:29.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:29.934698+0000) 2024-02-19T23:50:30.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:30.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:30.934913+0000) 2024-02-19T23:50:31.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:31.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:31.935062+0000) 2024-02-19T23:50:32.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:32.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:32.935297+0000) 2024-02-19T23:50:33.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:33.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:33.935496+0000) 2024-02-19T23:50:34.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:34.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:34.935710+0000) 2024-02-19T23:50:35.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:35.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:35.935981+0000) 2024-02-19T23:50:36.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:36.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:36.936225+0000) 2024-02-19T23:50:37.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:37.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:37.936501+0000) 2024-02-19T23:50:38.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:38.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:38.936763+0000) 2024-02-19T23:50:39.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:39.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:39.937005+0000) 2024-02-19T23:50:40.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:40.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:40.937268+0000) 2024-02-19T23:50:41.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:41.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:41.937519+0000) 2024-02-19T23:50:42.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:42.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:42.937849+0000) 2024-02-19T23:50:43.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:43.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:43.938024+0000) 2024-02-19T23:50:44.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:44.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:44.938276+0000) 2024-02-19T23:50:45.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:45.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:45.938439+0000) 2024-02-19T23:50:46.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:46.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:46.938686+0000) 2024-02-19T23:50:47.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:47.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:47.938935+0000) 2024-02-19T23:50:48.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:48.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:48.939215+0000) 2024-02-19T23:50:49.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:49.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:49.939397+0000) 2024-02-19T23:50:50.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:50.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:50.939667+0000) 2024-02-19T23:50:51.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:51.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:51.939871+0000) 2024-02-19T23:50:52.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:52.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:52.940055+0000) 2024-02-19T23:50:53.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:53.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:53.940243+0000) 2024-02-19T23:50:54.176 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:50:54.180 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:50:54.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:54.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:54.940404+0000) 2024-02-19T23:50:55.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:55.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:55.940629+0000) 2024-02-19T23:50:56.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:56.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:56.940861+0000) 2024-02-19T23:50:57.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:57.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:57.941034+0000) 2024-02-19T23:50:58.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:58.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:58.941197+0000) 2024-02-19T23:50:59.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:50:59.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:50:59.941422+0000) 2024-02-19T23:51:00.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:00.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:00.941664+0000) 2024-02-19T23:51:01.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:01.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:01.941824+0000) 2024-02-19T23:51:02.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:02.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:02.942064+0000) 2024-02-19T23:51:03.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:03.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:03.942244+0000) 2024-02-19T23:51:04.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:04.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:04.942443+0000) 2024-02-19T23:51:05.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:05.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:05.942641+0000) 2024-02-19T23:51:06.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:06.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:06.942858+0000) 2024-02-19T23:51:07.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:07.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:07.943018+0000) 2024-02-19T23:51:08.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:08.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:08.943189+0000) 2024-02-19T23:51:09.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:09.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:09.943439+0000) 2024-02-19T23:51:10.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:10.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:10.943683+0000) 2024-02-19T23:51:11.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:11.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:11.943900+0000) 2024-02-19T23:51:12.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:12.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:12.944144+0000) 2024-02-19T23:51:13.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:13.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:13.944362+0000) 2024-02-19T23:51:14.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:14.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:14.944591+0000) 2024-02-19T23:51:15.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:15.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:15.944844+0000) 2024-02-19T23:51:16.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:16.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:16.945065+0000) 2024-02-19T23:51:17.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:17.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:17.945238+0000) 2024-02-19T23:51:18.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:18.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:18.945416+0000) 2024-02-19T23:51:19.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:19.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:19.945638+0000) 2024-02-19T23:51:20.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:20.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:20.945831+0000) 2024-02-19T23:51:21.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:21.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:21.946010+0000) 2024-02-19T23:51:22.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:22.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:22.946233+0000) 2024-02-19T23:51:23.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:23.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:23.946413+0000) 2024-02-19T23:51:24.298 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:51:24.303 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:51:24.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:24.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:24.946594+0000) 2024-02-19T23:51:25.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:25.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:25.946832+0000) 2024-02-19T23:51:26.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:26.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:26.947087+0000) 2024-02-19T23:51:27.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:27.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:27.947328+0000) 2024-02-19T23:51:28.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:28.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:28.947580+0000) 2024-02-19T23:51:29.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:29.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:29.947731+0000) 2024-02-19T23:51:30.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:30.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:30.947920+0000) 2024-02-19T23:51:31.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:31.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:31.948133+0000) 2024-02-19T23:51:32.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:32.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:32.948366+0000) 2024-02-19T23:51:33.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:33.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:33.948594+0000) 2024-02-19T23:51:34.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:34.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:34.948819+0000) 2024-02-19T23:51:35.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:35.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:35.949029+0000) 2024-02-19T23:51:36.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:36.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:36.949239+0000) 2024-02-19T23:51:37.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:37.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:37.949435+0000) 2024-02-19T23:51:38.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:38.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:38.949652+0000) 2024-02-19T23:51:39.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:39.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:39.949904+0000) 2024-02-19T23:51:40.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:40.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:40.950139+0000) 2024-02-19T23:51:41.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:41.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:41.950429+0000) 2024-02-19T23:51:42.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:42.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:42.950609+0000) 2024-02-19T23:51:43.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:43.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:43.950775+0000) 2024-02-19T23:51:44.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:44.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:44.951021+0000) 2024-02-19T23:51:45.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:45.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:45.951174+0000) 2024-02-19T23:51:46.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:46.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:46.951369+0000) 2024-02-19T23:51:47.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:47.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:47.951641+0000) 2024-02-19T23:51:48.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:48.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:48.951822+0000) 2024-02-19T23:51:49.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:49.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:49.952071+0000) 2024-02-19T23:51:50.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:50.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:50.952320+0000) 2024-02-19T23:51:51.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:51.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:51.952568+0000) 2024-02-19T23:51:52.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:52.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:52.952737+0000) 2024-02-19T23:51:53.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:53.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:53.952912+0000) 2024-02-19T23:51:54.363 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:51:54.366 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:51:54.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:54.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:54.953110+0000) 2024-02-19T23:51:55.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:55.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:55.953352+0000) 2024-02-19T23:51:56.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:56.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:56.953569+0000) 2024-02-19T23:51:57.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:57.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:57.953738+0000) 2024-02-19T23:51:58.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:58.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:58.953897+0000) 2024-02-19T23:51:59.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:51:59.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:51:59.954139+0000) 2024-02-19T23:52:00.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:00.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:00.954366+0000) 2024-02-19T23:52:01.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:01.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:01.954529+0000) 2024-02-19T23:52:02.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:02.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:02.954793+0000) 2024-02-19T23:52:03.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:03.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:03.955048+0000) 2024-02-19T23:52:04.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:04.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:04.955271+0000) 2024-02-19T23:52:05.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:05.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:05.955513+0000) 2024-02-19T23:52:06.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:06.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:06.955677+0000) 2024-02-19T23:52:07.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:07.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:07.955929+0000) 2024-02-19T23:52:08.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:08.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:08.956115+0000) 2024-02-19T23:52:09.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:09.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:09.956371+0000) 2024-02-19T23:52:10.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:10.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:10.956562+0000) 2024-02-19T23:52:11.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:11.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:11.956713+0000) 2024-02-19T23:52:12.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:12.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:12.956890+0000) 2024-02-19T23:52:13.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:13.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:13.957142+0000) 2024-02-19T23:52:14.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:14.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:14.957350+0000) 2024-02-19T23:52:15.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:15.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:15.957554+0000) 2024-02-19T23:52:16.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:16.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:16.957736+0000) 2024-02-19T23:52:17.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:17.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:17.957903+0000) 2024-02-19T23:52:18.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:18.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:18.958056+0000) 2024-02-19T23:52:19.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:19.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:19.958221+0000) 2024-02-19T23:52:20.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:20.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:20.958474+0000) 2024-02-19T23:52:21.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:21.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:21.958677+0000) 2024-02-19T23:52:22.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:22.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:22.958885+0000) 2024-02-19T23:52:23.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:23.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:23.959032+0000) 2024-02-19T23:52:24.425 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:52:24.429 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:52:24.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:24.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:24.959221+0000) 2024-02-19T23:52:25.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:25.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:25.959394+0000) 2024-02-19T23:52:26.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:26.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:26.959576+0000) 2024-02-19T23:52:27.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:27.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:27.959845+0000) 2024-02-19T23:52:28.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:28.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:28.960054+0000) 2024-02-19T23:52:29.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:29.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:29.960243+0000) 2024-02-19T23:52:30.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:30.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:30.960481+0000) 2024-02-19T23:52:31.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:31.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:31.960705+0000) 2024-02-19T23:52:32.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:32.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:32.960849+0000) 2024-02-19T23:52:33.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:33.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:33.961061+0000) 2024-02-19T23:52:34.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:34.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:34.961291+0000) 2024-02-19T23:52:35.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:35.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:35.961525+0000) 2024-02-19T23:52:36.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:36.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:36.961767+0000) 2024-02-19T23:52:37.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:37.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:37.961910+0000) 2024-02-19T23:52:38.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:38.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:38.962123+0000) 2024-02-19T23:52:39.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:39.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:39.962299+0000) 2024-02-19T23:52:40.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:40.957+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:40.962514+0000) 2024-02-19T23:52:41.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:41.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:41.962794+0000) 2024-02-19T23:52:42.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:42.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:42.963008+0000) 2024-02-19T23:52:43.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:43.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:43.963229+0000) 2024-02-19T23:52:44.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:44.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:44.963429+0000) 2024-02-19T23:52:45.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:45.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:45.963641+0000) 2024-02-19T23:52:46.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:46.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:46.963780+0000) 2024-02-19T23:52:47.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:47.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:47.963996+0000) 2024-02-19T23:52:48.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:48.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:48.964186+0000) 2024-02-19T23:52:49.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:49.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:49.964306+0000) 2024-02-19T23:52:50.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:50.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:50.964531+0000) 2024-02-19T23:52:51.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:51.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:51.964784+0000) 2024-02-19T23:52:52.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:52.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:52.964983+0000) 2024-02-19T23:52:53.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:53.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:53.965185+0000) 2024-02-19T23:52:54.572 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:52:54.576 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:52:54.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:54.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:54.965391+0000) 2024-02-19T23:52:55.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:55.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:55.965609+0000) 2024-02-19T23:52:56.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:56.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:56.965888+0000) 2024-02-19T23:52:57.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:57.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:57.966148+0000) 2024-02-19T23:52:58.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:58.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:58.966360+0000) 2024-02-19T23:52:59.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:52:59.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:52:59.966554+0000) 2024-02-19T23:53:00.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:00.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:00.966837+0000) 2024-02-19T23:53:01.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:01.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:01.967052+0000) 2024-02-19T23:53:02.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:02.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:02.967237+0000) 2024-02-19T23:53:03.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:03.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:03.967479+0000) 2024-02-19T23:53:04.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:04.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:04.967695+0000) 2024-02-19T23:53:05.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:05.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:05.967914+0000) 2024-02-19T23:53:06.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:06.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:06.968109+0000) 2024-02-19T23:53:07.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:07.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:07.968329+0000) 2024-02-19T23:53:08.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:08.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:08.968500+0000) 2024-02-19T23:53:09.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:09.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:09.968715+0000) 2024-02-19T23:53:10.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:10.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:10.968945+0000) 2024-02-19T23:53:11.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:11.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:11.969161+0000) 2024-02-19T23:53:12.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:12.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:12.969358+0000) 2024-02-19T23:53:13.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:13.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:13.969571+0000) 2024-02-19T23:53:14.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:14.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:14.969821+0000) 2024-02-19T23:53:15.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:15.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:15.970007+0000) 2024-02-19T23:53:16.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:16.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:16.970227+0000) 2024-02-19T23:53:17.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:17.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:17.970402+0000) 2024-02-19T23:53:18.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:18.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:18.970622+0000) 2024-02-19T23:53:19.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:19.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:19.970862+0000) 2024-02-19T23:53:20.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:20.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:20.971120+0000) 2024-02-19T23:53:21.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:21.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:21.971317+0000) 2024-02-19T23:53:22.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:22.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:22.971547+0000) 2024-02-19T23:53:23.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:23.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:23.971757+0000) 2024-02-19T23:53:24.702 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:53:24.705 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:53:24.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:24.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:24.972013+0000) 2024-02-19T23:53:25.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:25.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:25.972224+0000) 2024-02-19T23:53:26.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:26.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:26.972456+0000) 2024-02-19T23:53:27.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:27.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:27.972714+0000) 2024-02-19T23:53:28.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:28.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:28.972980+0000) 2024-02-19T23:53:29.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:29.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:29.973171+0000) 2024-02-19T23:53:30.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:30.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:30.973325+0000) 2024-02-19T23:53:31.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:31.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:31.973550+0000) 2024-02-19T23:53:32.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:32.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:32.973737+0000) 2024-02-19T23:53:33.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:33.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:33.973901+0000) 2024-02-19T23:53:34.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:34.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:34.974139+0000) 2024-02-19T23:53:35.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:35.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:35.974371+0000) 2024-02-19T23:53:36.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:36.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:36.974557+0000) 2024-02-19T23:53:37.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:37.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:37.974737+0000) 2024-02-19T23:53:38.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:38.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:38.974940+0000) 2024-02-19T23:53:39.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:39.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:39.975131+0000) 2024-02-19T23:53:40.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:40.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:40.975339+0000) 2024-02-19T23:53:41.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:41.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:41.975510+0000) 2024-02-19T23:53:42.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:42.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:42.975710+0000) 2024-02-19T23:53:43.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:43.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:43.975873+0000) 2024-02-19T23:53:44.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:44.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:44.976069+0000) 2024-02-19T23:53:45.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:45.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:45.976228+0000) 2024-02-19T23:53:46.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:46.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:46.976394+0000) 2024-02-19T23:53:47.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:47.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:47.976672+0000) 2024-02-19T23:53:48.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:48.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:48.976870+0000) 2024-02-19T23:53:49.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:49.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:49.977109+0000) 2024-02-19T23:53:50.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:50.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:50.977358+0000) 2024-02-19T23:53:51.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:51.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:51.977564+0000) 2024-02-19T23:53:52.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:52.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:52.977792+0000) 2024-02-19T23:53:53.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:53.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:53.977937+0000) 2024-02-19T23:53:54.841 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:53:54.844 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:53:54.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:54.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:54.978190+0000) 2024-02-19T23:53:55.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:55.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:55.978364+0000) 2024-02-19T23:53:56.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:56.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:56.978539+0000) 2024-02-19T23:53:57.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:57.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:57.978680+0000) 2024-02-19T23:53:58.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:58.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:58.978886+0000) 2024-02-19T23:53:59.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:53:59.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:53:59.979081+0000) 2024-02-19T23:54:00.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:00.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:00.979270+0000) 2024-02-19T23:54:01.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:01.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:01.979449+0000) 2024-02-19T23:54:02.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:02.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:02.979630+0000) 2024-02-19T23:54:03.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:03.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:03.979840+0000) 2024-02-19T23:54:04.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:04.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:04.980088+0000) 2024-02-19T23:54:05.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:05.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:05.980329+0000) 2024-02-19T23:54:06.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:06.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:06.980526+0000) 2024-02-19T23:54:07.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:07.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:07.980788+0000) 2024-02-19T23:54:08.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:08.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:08.980997+0000) 2024-02-19T23:54:09.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:09.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:09.981203+0000) 2024-02-19T23:54:10.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:10.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:10.981457+0000) 2024-02-19T23:54:11.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:11.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:11.981730+0000) 2024-02-19T23:54:12.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:12.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:12.981930+0000) 2024-02-19T23:54:13.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:13.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:13.982160+0000) 2024-02-19T23:54:14.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:14.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:14.982422+0000) 2024-02-19T23:54:15.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:15.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:15.982640+0000) 2024-02-19T23:54:16.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:16.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:16.982886+0000) 2024-02-19T23:54:17.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:17.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:17.983102+0000) 2024-02-19T23:54:18.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:18.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:18.983301+0000) 2024-02-19T23:54:19.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:19.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:19.983546+0000) 2024-02-19T23:54:20.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:20.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:20.983776+0000) 2024-02-19T23:54:21.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:21.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:21.983959+0000) 2024-02-19T23:54:22.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:22.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:22.984181+0000) 2024-02-19T23:54:23.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:23.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:23.984412+0000) 2024-02-19T23:54:24.922 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:54:24.926 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:54:24.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:24.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:24.984646+0000) 2024-02-19T23:54:25.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:25.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:25.984881+0000) 2024-02-19T23:54:26.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:26.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:26.985088+0000) 2024-02-19T23:54:27.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:27.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:27.985270+0000) 2024-02-19T23:54:28.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:28.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:28.985443+0000) 2024-02-19T23:54:29.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:29.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:29.985662+0000) 2024-02-19T23:54:30.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:30.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:30.985916+0000) 2024-02-19T23:54:31.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:31.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:31.986097+0000) 2024-02-19T23:54:32.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:32.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:32.986360+0000) 2024-02-19T23:54:33.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:33.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:33.986554+0000) 2024-02-19T23:54:34.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:34.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:34.986750+0000) 2024-02-19T23:54:35.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:35.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:35.986941+0000) 2024-02-19T23:54:36.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:36.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:36.987116+0000) 2024-02-19T23:54:37.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:37.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:37.987386+0000) 2024-02-19T23:54:38.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:38.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:38.987595+0000) 2024-02-19T23:54:39.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:39.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:39.987768+0000) 2024-02-19T23:54:40.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:40.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:40.987950+0000) 2024-02-19T23:54:41.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:41.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:41.988086+0000) 2024-02-19T23:54:42.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:42.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:42.988270+0000) 2024-02-19T23:54:43.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:43.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:43.988436+0000) 2024-02-19T23:54:44.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:44.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:44.988631+0000) 2024-02-19T23:54:45.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:45.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:45.988786+0000) 2024-02-19T23:54:46.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:46.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:46.988966+0000) 2024-02-19T23:54:47.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:47.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:47.989111+0000) 2024-02-19T23:54:48.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:48.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:48.989261+0000) 2024-02-19T23:54:49.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:49.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:49.989418+0000) 2024-02-19T23:54:50.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:50.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:50.989557+0000) 2024-02-19T23:54:51.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:51.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:51.989686+0000) 2024-02-19T23:54:52.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:52.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:52.989807+0000) 2024-02-19T23:54:53.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:53.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:53.989971+0000) 2024-02-19T23:54:54.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:54.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:54.990086+0000) 2024-02-19T23:54:55.045 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:54:55.049 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:54:55.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:55.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:55.990220+0000) 2024-02-19T23:54:56.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:56.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:56.990437+0000) 2024-02-19T23:54:57.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:57.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:57.990603+0000) 2024-02-19T23:54:58.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:58.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:58.990836+0000) 2024-02-19T23:54:59.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:54:59.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:54:59.991001+0000) 2024-02-19T23:55:00.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:00.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:00.991175+0000) 2024-02-19T23:55:01.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:01.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:01.991406+0000) 2024-02-19T23:55:02.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:02.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:02.991612+0000) 2024-02-19T23:55:03.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:03.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:03.991776+0000) 2024-02-19T23:55:04.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:04.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:04.991988+0000) 2024-02-19T23:55:05.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:05.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:05.992206+0000) 2024-02-19T23:55:06.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:06.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:06.992416+0000) 2024-02-19T23:55:07.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:07.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:07.992641+0000) 2024-02-19T23:55:08.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:08.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:08.992875+0000) 2024-02-19T23:55:09.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:09.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:09.993111+0000) 2024-02-19T23:55:10.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:10.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:10.993301+0000) 2024-02-19T23:55:11.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:11.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:11.993525+0000) 2024-02-19T23:55:12.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:12.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:12.993713+0000) 2024-02-19T23:55:13.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:13.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:13.993930+0000) 2024-02-19T23:55:14.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:14.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:14.994165+0000) 2024-02-19T23:55:15.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:15.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:15.994392+0000) 2024-02-19T23:55:16.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:16.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:16.994617+0000) 2024-02-19T23:55:17.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:17.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:17.994861+0000) 2024-02-19T23:55:18.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:18.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:18.995050+0000) 2024-02-19T23:55:19.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:19.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:19.995187+0000) 2024-02-19T23:55:20.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:20.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:20.995354+0000) 2024-02-19T23:55:21.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:21.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:21.995579+0000) 2024-02-19T23:55:22.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:22.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:22.995743+0000) 2024-02-19T23:55:23.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:23.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:23.995937+0000) 2024-02-19T23:55:24.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:24.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:24.996091+0000) 2024-02-19T23:55:25.109 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:55:25.112 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:55:25.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:25.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:25.996280+0000) 2024-02-19T23:55:26.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:26.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:26.996477+0000) 2024-02-19T23:55:27.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:27.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:27.996701+0000) 2024-02-19T23:55:28.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:28.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:28.996917+0000) 2024-02-19T23:55:29.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:29.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:29.997166+0000) 2024-02-19T23:55:30.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:30.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:30.997430+0000) 2024-02-19T23:55:32.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:31.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:31.997649+0000) 2024-02-19T23:55:33.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:32.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:32.997905+0000) 2024-02-19T23:55:34.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:33.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:33.998087+0000) 2024-02-19T23:55:35.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:34.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:34.998290+0000) 2024-02-19T23:55:36.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:35.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:35.998496+0000) 2024-02-19T23:55:37.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:36.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:36.998756+0000) 2024-02-19T23:55:38.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:37.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:37.999001+0000) 2024-02-19T23:55:39.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:38.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:38.999246+0000) 2024-02-19T23:55:40.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:39.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:39.999498+0000) 2024-02-19T23:55:41.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:40.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:40.999705+0000) 2024-02-19T23:55:42.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:41.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:41.999907+0000) 2024-02-19T23:55:43.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:42.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:43.000118+0000) 2024-02-19T23:55:44.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:43.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:44.000297+0000) 2024-02-19T23:55:45.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:44.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:45.000460+0000) 2024-02-19T23:55:46.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:45.994+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:46.000584+0000) 2024-02-19T23:55:47.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:46.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:47.000864+0000) 2024-02-19T23:55:48.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:47.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:48.001141+0000) 2024-02-19T23:55:49.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:48.994+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:49.001407+0000) 2024-02-19T23:55:50.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:49.994+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:50.001580+0000) 2024-02-19T23:55:51.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:50.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:51.001762+0000) 2024-02-19T23:55:52.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:51.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:52.001931+0000) 2024-02-19T23:55:53.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:52.994+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:53.002111+0000) 2024-02-19T23:55:54.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:53.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:54.002277+0000) 2024-02-19T23:55:55.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:54.994+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:55.002438+0000) 2024-02-19T23:55:55.257 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:55:55.261 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:55:56.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:55.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:56.002614+0000) 2024-02-19T23:55:57.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:56.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:57.002851+0000) 2024-02-19T23:55:58.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:58.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:58.003040+0000) 2024-02-19T23:55:59.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:59.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:55:59.003175+0000) 2024-02-19T23:56:00.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:55:59.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:00.003347+0000) 2024-02-19T23:56:01.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:01.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:01.003540+0000) 2024-02-19T23:56:02.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:01.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:02.003686+0000) 2024-02-19T23:56:03.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:03.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:03.003815+0000) 2024-02-19T23:56:04.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:04.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:04.004074+0000) 2024-02-19T23:56:05.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:05.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:05.004235+0000) 2024-02-19T23:56:06.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:06.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:06.004513+0000) 2024-02-19T23:56:07.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:07.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:07.004640+0000) 2024-02-19T23:56:08.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:08.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:08.004829+0000) 2024-02-19T23:56:09.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:09.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:09.004996+0000) 2024-02-19T23:56:10.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:10.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:10.005170+0000) 2024-02-19T23:56:11.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:11.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:11.005407+0000) 2024-02-19T23:56:12.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:12.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:12.005587+0000) 2024-02-19T23:56:13.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:13.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:13.005748+0000) 2024-02-19T23:56:14.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:14.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:14.005933+0000) 2024-02-19T23:56:15.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:15.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:15.006163+0000) 2024-02-19T23:56:16.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:16.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:16.006427+0000) 2024-02-19T23:56:17.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:17.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:17.006632+0000) 2024-02-19T23:56:18.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:18.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:18.006837+0000) 2024-02-19T23:56:19.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:19.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:19.007086+0000) 2024-02-19T23:56:20.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:20.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:20.007229+0000) 2024-02-19T23:56:21.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:21.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:21.007447+0000) 2024-02-19T23:56:22.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:22.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:22.007653+0000) 2024-02-19T23:56:23.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:23.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:23.007851+0000) 2024-02-19T23:56:24.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:24.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:24.008084+0000) 2024-02-19T23:56:25.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:25.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:25.008261+0000) 2024-02-19T23:56:25.320 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:56:25.325 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:56:26.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:26.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:26.008474+0000) 2024-02-19T23:56:27.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:27.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:27.008668+0000) 2024-02-19T23:56:28.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:28.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:28.008939+0000) 2024-02-19T23:56:29.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:29.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:29.009229+0000) 2024-02-19T23:56:30.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:30.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:30.009434+0000) 2024-02-19T23:56:31.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:31.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:31.009607+0000) 2024-02-19T23:56:32.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:32.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:32.009807+0000) 2024-02-19T23:56:33.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:33.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:33.009979+0000) 2024-02-19T23:56:34.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:34.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:34.010260+0000) 2024-02-19T23:56:35.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:35.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:35.010483+0000) 2024-02-19T23:56:36.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:36.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:36.010735+0000) 2024-02-19T23:56:37.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:37.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:37.010948+0000) 2024-02-19T23:56:38.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:38.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:38.011097+0000) 2024-02-19T23:56:39.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:39.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:39.011304+0000) 2024-02-19T23:56:40.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:40.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:40.011586+0000) 2024-02-19T23:56:41.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:41.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:41.011901+0000) 2024-02-19T23:56:42.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:42.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:42.012138+0000) 2024-02-19T23:56:43.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:43.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:43.012316+0000) 2024-02-19T23:56:44.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:44.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:44.012579+0000) 2024-02-19T23:56:45.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:45.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:45.012769+0000) 2024-02-19T23:56:46.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:46.009+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:46.012910+0000) 2024-02-19T23:56:47.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:47.009+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:47.013129+0000) 2024-02-19T23:56:48.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:48.009+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:48.013306+0000) 2024-02-19T23:56:49.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:49.009+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:49.013475+0000) 2024-02-19T23:56:50.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:50.009+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:50.013648+0000) 2024-02-19T23:56:51.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:51.009+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:51.013838+0000) 2024-02-19T23:56:52.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:52.009+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:52.014027+0000) 2024-02-19T23:56:53.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:53.009+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:53.014257+0000) 2024-02-19T23:56:54.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:54.009+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:54.014405+0000) 2024-02-19T23:56:55.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:55.009+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:55.014571+0000) 2024-02-19T23:56:55.450 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:56:55.455 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:56:56.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:56.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:56.014859+0000) 2024-02-19T23:56:57.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:57.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:57.014992+0000) 2024-02-19T23:56:58.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:58.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:58.015154+0000) 2024-02-19T23:56:59.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:56:59.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:56:59.015376+0000) 2024-02-19T23:57:00.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:00.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:00.015562+0000) 2024-02-19T23:57:01.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:01.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:01.015744+0000) 2024-02-19T23:57:02.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:02.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:02.015927+0000) 2024-02-19T23:57:03.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:03.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:03.016086+0000) 2024-02-19T23:57:04.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:04.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:04.016326+0000) 2024-02-19T23:57:05.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:05.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:05.016556+0000) 2024-02-19T23:57:06.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:06.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:06.016758+0000) 2024-02-19T23:57:07.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:07.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:07.016982+0000) 2024-02-19T23:57:08.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:08.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:08.017212+0000) 2024-02-19T23:57:09.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:09.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:09.017442+0000) 2024-02-19T23:57:10.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:10.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:10.017686+0000) 2024-02-19T23:57:11.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:11.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:11.017965+0000) 2024-02-19T23:57:12.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:12.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:12.018193+0000) 2024-02-19T23:57:13.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:13.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:13.018460+0000) 2024-02-19T23:57:14.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:14.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:14.018629+0000) 2024-02-19T23:57:15.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:15.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:15.018869+0000) 2024-02-19T23:57:16.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:16.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:16.019052+0000) 2024-02-19T23:57:17.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:17.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:17.019245+0000) 2024-02-19T23:57:18.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:18.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:18.019464+0000) 2024-02-19T23:57:19.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:19.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:19.019700+0000) 2024-02-19T23:57:20.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:20.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:20.019944+0000) 2024-02-19T23:57:21.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:21.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:21.020132+0000) 2024-02-19T23:57:22.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:22.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:22.020343+0000) 2024-02-19T23:57:23.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:23.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:23.020626+0000) 2024-02-19T23:57:24.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:24.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:24.020808+0000) 2024-02-19T23:57:25.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:25.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:25.021006+0000) 2024-02-19T23:57:25.506 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:57:25.509 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:57:26.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:26.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:26.021192+0000) 2024-02-19T23:57:27.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:27.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:27.021429+0000) 2024-02-19T23:57:28.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:28.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:28.021629+0000) 2024-02-19T23:57:29.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:29.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:29.021790+0000) 2024-02-19T23:57:30.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:30.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:30.021967+0000) 2024-02-19T23:57:31.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:31.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:31.022164+0000) 2024-02-19T23:57:32.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:32.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:32.022333+0000) 2024-02-19T23:57:33.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:33.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:33.022571+0000) 2024-02-19T23:57:34.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:34.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:34.022837+0000) 2024-02-19T23:57:35.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:35.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:35.023099+0000) 2024-02-19T23:57:36.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:36.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:36.023297+0000) 2024-02-19T23:57:37.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:37.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:37.023531+0000) 2024-02-19T23:57:38.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:38.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:38.023734+0000) 2024-02-19T23:57:39.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:39.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:39.023950+0000) 2024-02-19T23:57:40.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:40.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:40.024179+0000) 2024-02-19T23:57:41.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:41.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:41.024415+0000) 2024-02-19T23:57:42.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:42.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:42.024630+0000) 2024-02-19T23:57:43.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:43.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:43.024803+0000) 2024-02-19T23:57:44.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:44.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:44.024983+0000) 2024-02-19T23:57:45.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:45.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:45.025204+0000) 2024-02-19T23:57:46.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:46.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:46.025419+0000) 2024-02-19T23:57:47.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:47.020+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:47.025660+0000) 2024-02-19T23:57:48.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:48.020+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:48.025881+0000) 2024-02-19T23:57:49.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:49.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:49.026048+0000) 2024-02-19T23:57:50.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:50.020+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:50.026226+0000) 2024-02-19T23:57:51.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:51.020+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:51.026483+0000) 2024-02-19T23:57:52.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:52.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:52.026766+0000) 2024-02-19T23:57:53.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:53.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:53.026925+0000) 2024-02-19T23:57:54.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:54.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:54.027049+0000) 2024-02-19T23:57:55.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:55.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:55.027230+0000) 2024-02-19T23:57:55.637 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:57:55.641 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:57:56.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:56.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:56.027378+0000) 2024-02-19T23:57:57.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:57.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:57.027537+0000) 2024-02-19T23:57:58.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:58.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:58.027773+0000) 2024-02-19T23:57:59.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:57:59.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:57:59.028021+0000) 2024-02-19T23:58:00.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:00.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:00.028267+0000) 2024-02-19T23:58:01.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:01.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:01.028507+0000) 2024-02-19T23:58:02.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:02.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:02.028659+0000) 2024-02-19T23:58:03.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:03.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:03.028865+0000) 2024-02-19T23:58:04.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:04.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:04.029010+0000) 2024-02-19T23:58:05.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:05.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:05.029156+0000) 2024-02-19T23:58:06.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:06.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:06.029335+0000) 2024-02-19T23:58:07.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:07.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:07.029501+0000) 2024-02-19T23:58:08.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:08.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:08.029660+0000) 2024-02-19T23:58:09.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:09.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:09.029873+0000) 2024-02-19T23:58:10.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:10.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:10.030116+0000) 2024-02-19T23:58:11.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:11.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:11.030305+0000) 2024-02-19T23:58:12.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:12.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:12.030477+0000) 2024-02-19T23:58:13.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:13.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:13.030688+0000) 2024-02-19T23:58:14.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:14.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:14.030828+0000) 2024-02-19T23:58:15.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:15.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:15.031016+0000) 2024-02-19T23:58:16.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:16.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:16.031241+0000) 2024-02-19T23:58:17.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:17.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:17.031380+0000) 2024-02-19T23:58:18.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:18.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:18.031595+0000) 2024-02-19T23:58:19.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:19.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:19.031820+0000) 2024-02-19T23:58:20.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:20.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:20.032004+0000) 2024-02-19T23:58:21.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:21.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:21.032260+0000) 2024-02-19T23:58:22.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:22.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:22.032481+0000) 2024-02-19T23:58:23.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:23.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:23.032724+0000) 2024-02-19T23:58:24.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:24.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:24.032873+0000) 2024-02-19T23:58:25.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:25.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:25.033046+0000) 2024-02-19T23:58:25.692 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:58:25.696 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:58:26.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:26.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:26.033180+0000) 2024-02-19T23:58:27.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:27.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:27.033398+0000) 2024-02-19T23:58:28.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:28.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:28.033581+0000) 2024-02-19T23:58:29.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:29.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:29.033788+0000) 2024-02-19T23:58:30.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:30.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:30.034045+0000) 2024-02-19T23:58:31.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:31.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:31.034270+0000) 2024-02-19T23:58:32.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:32.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:32.034497+0000) 2024-02-19T23:58:33.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:33.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:33.034853+0000) 2024-02-19T23:58:34.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:34.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:34.035062+0000) 2024-02-19T23:58:35.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:35.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:35.035312+0000) 2024-02-19T23:58:36.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:36.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:36.035525+0000) 2024-02-19T23:58:37.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:37.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:37.035812+0000) 2024-02-19T23:58:38.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:38.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:38.035986+0000) 2024-02-19T23:58:39.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:39.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:39.036225+0000) 2024-02-19T23:58:40.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:40.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:40.036450+0000) 2024-02-19T23:58:41.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:41.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:41.036673+0000) 2024-02-19T23:58:42.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:42.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:42.036954+0000) 2024-02-19T23:58:43.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:43.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:43.037233+0000) 2024-02-19T23:58:44.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:44.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:44.037387+0000) 2024-02-19T23:58:45.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:45.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:45.037549+0000) 2024-02-19T23:58:46.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:46.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:46.037709+0000) 2024-02-19T23:58:47.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:47.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:47.037862+0000) 2024-02-19T23:58:48.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:48.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:48.038014+0000) 2024-02-19T23:58:49.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:49.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:49.038279+0000) 2024-02-19T23:58:50.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:50.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:50.038417+0000) 2024-02-19T23:58:51.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:51.031+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:51.038601+0000) 2024-02-19T23:58:52.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:52.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:52.038787+0000) 2024-02-19T23:58:53.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:53.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:53.038922+0000) 2024-02-19T23:58:54.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:54.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:54.039062+0000) 2024-02-19T23:58:55.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:55.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:55.039195+0000) 2024-02-19T23:58:55.824 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:58:55.827 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:58:56.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:56.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:56.039339+0000) 2024-02-19T23:58:57.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:57.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:57.039470+0000) 2024-02-19T23:58:58.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:58.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:58.039612+0000) 2024-02-19T23:58:59.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:58:59.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:58:59.039767+0000) 2024-02-19T23:59:00.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:00.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:00.039915+0000) 2024-02-19T23:59:01.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:01.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:01.040092+0000) 2024-02-19T23:59:02.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:02.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:02.040308+0000) 2024-02-19T23:59:03.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:03.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:03.040537+0000) 2024-02-19T23:59:04.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:04.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:04.040691+0000) 2024-02-19T23:59:05.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:05.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:05.040911+0000) 2024-02-19T23:59:06.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:06.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:06.041055+0000) 2024-02-19T23:59:07.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:07.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:07.041298+0000) 2024-02-19T23:59:08.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:08.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:08.041534+0000) 2024-02-19T23:59:09.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:09.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:09.041755+0000) 2024-02-19T23:59:10.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:10.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:10.041919+0000) 2024-02-19T23:59:11.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:11.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:11.042072+0000) 2024-02-19T23:59:12.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:12.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:12.042259+0000) 2024-02-19T23:59:13.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:13.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:13.042390+0000) 2024-02-19T23:59:14.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:14.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:14.042597+0000) 2024-02-19T23:59:15.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:15.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:15.042979+0000) 2024-02-19T23:59:16.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:16.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:16.043147+0000) 2024-02-19T23:59:17.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:17.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:17.043361+0000) 2024-02-19T23:59:18.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:18.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:18.043580+0000) 2024-02-19T23:59:19.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:19.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:19.043760+0000) 2024-02-19T23:59:20.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:20.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:20.044005+0000) 2024-02-19T23:59:21.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:21.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:21.044269+0000) 2024-02-19T23:59:22.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:22.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:22.044432+0000) 2024-02-19T23:59:23.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:23.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:23.044685+0000) 2024-02-19T23:59:24.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:24.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:24.044907+0000) 2024-02-19T23:59:25.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:25.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:25.045118+0000) 2024-02-19T23:59:25.895 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:59:25.898 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:59:26.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:26.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:26.045303+0000) 2024-02-19T23:59:27.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:27.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:27.045562+0000) 2024-02-19T23:59:28.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:28.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:28.045776+0000) 2024-02-19T23:59:29.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:29.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:29.045983+0000) 2024-02-19T23:59:30.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:30.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:30.046235+0000) 2024-02-19T23:59:31.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:31.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:31.046516+0000) 2024-02-19T23:59:32.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:32.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:32.046730+0000) 2024-02-19T23:59:33.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:33.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:33.046999+0000) 2024-02-19T23:59:34.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:34.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:34.047233+0000) 2024-02-19T23:59:35.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:35.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:35.047527+0000) 2024-02-19T23:59:36.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:36.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:36.047761+0000) 2024-02-19T23:59:37.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:37.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:37.048016+0000) 2024-02-19T23:59:38.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:38.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:38.048197+0000) 2024-02-19T23:59:39.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:39.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:39.048459+0000) 2024-02-19T23:59:40.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:40.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:40.048735+0000) 2024-02-19T23:59:41.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:41.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:41.048948+0000) 2024-02-19T23:59:42.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:42.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:42.049085+0000) 2024-02-19T23:59:43.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:43.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:43.049289+0000) 2024-02-19T23:59:44.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:44.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:44.049529+0000) 2024-02-19T23:59:45.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:45.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:45.049814+0000) 2024-02-19T23:59:46.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:46.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:46.049982+0000) 2024-02-19T23:59:47.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:47.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:47.050213+0000) 2024-02-19T23:59:48.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:48.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:48.050435+0000) 2024-02-19T23:59:49.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:49.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:49.050759+0000) 2024-02-19T23:59:50.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:50.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:50.050935+0000) 2024-02-19T23:59:51.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:51.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:51.051183+0000) 2024-02-19T23:59:52.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:52.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:52.051404+0000) 2024-02-19T23:59:53.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:53.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:53.051738+0000) 2024-02-19T23:59:54.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:54.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:54.051935+0000) 2024-02-19T23:59:55.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:55.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:55.052207+0000) 2024-02-19T23:59:55.960 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:59:55.964 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T23:59:56.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:56.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:56.052448+0000) 2024-02-19T23:59:57.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:57.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:57.052720+0000) 2024-02-19T23:59:58.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:58.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:58.052894+0000) 2024-02-19T23:59:59.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-19T23:59:59.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T22:59:59.053066+0000) 2024-02-20T00:00:00.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:00.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:00.053316+0000) 2024-02-20T00:00:01.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:01.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:01.053560+0000) 2024-02-20T00:00:01.196 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-02-20T00:00:01.190+0000 7fc107400640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 55273) UID: 0 2024-02-20T00:00:01.196 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-02-20T00:00:01.190+0000 7fc107400640 -1 mon.a@0(leader) e1 *** Got Signal Hangup *** 2024-02-20T00:00:01.196 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-20T00:00:01.190+0000 7fad7d200640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 55273) UID: 0 2024-02-20T00:00:01.196 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-02-20T00:00:01.190+0000 7f5f3b000640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 55273) UID: 0 2024-02-20T00:00:01.197 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-02-20T00:00:01.190+0000 7f83be400640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 55273) UID: 0 2024-02-20T00:00:01.197 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-02-20T00:00:01.190+0000 7fbb53200640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 55273) UID: 0 2024-02-20T00:00:01.197 INFO:tasks.ceph.osd.3.smithi047.stderr:2024-02-20T00:00:01.190+0000 7f73ad800640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 55273) UID: 0 2024-02-20T00:00:01.198 INFO:tasks.ceph.mds.a.smithi047.stderr:2024-02-20T00:00:01.190+0000 7f923e400640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 55273) UID: 0 2024-02-20T00:00:01.235 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-02-20T00:00:01.230+0000 7f83be400640 -1 Fail to open '/proc/55278/cmdline' error = (2) No such file or directory 2024-02-20T00:00:01.235 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-02-20T00:00:01.230+0000 7f83be400640 -1 received signal: Hangup from (PID: 55278) UID: 0 2024-02-20T00:00:01.236 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-02-20T00:00:01.230+0000 7f5f3b000640 -1 Fail to open '/proc/55278/cmdline' error = (3) No such process 2024-02-20T00:00:01.236 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-02-20T00:00:01.230+0000 7f5f3b000640 -1 received signal: Hangup from (PID: 55278) UID: 0 2024-02-20T00:00:01.236 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-02-20T00:00:01.230+0000 7fbb53200640 -1 Fail to open '/proc/55278/cmdline' error = (2) No such file or directory 2024-02-20T00:00:01.236 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-02-20T00:00:01.230+0000 7fbb53200640 -1 received signal: Hangup from (PID: 55278) UID: 0 2024-02-20T00:00:01.236 INFO:tasks.ceph.osd.3.smithi047.stderr:2024-02-20T00:00:01.230+0000 7f73ad800640 -1 Fail to open '/proc/55278/cmdline' error = (2) No such file or directory 2024-02-20T00:00:01.237 INFO:tasks.ceph.osd.3.smithi047.stderr:2024-02-20T00:00:01.230+0000 7f73ad800640 -1 received signal: Hangup from (PID: 55278) UID: 0 2024-02-20T00:00:01.237 INFO:tasks.ceph.mds.a.smithi047.stderr:2024-02-20T00:00:01.230+0000 7f923e400640 -1 Fail to open '/proc/55278/cmdline' error = (2) No such file or directory 2024-02-20T00:00:01.237 INFO:tasks.ceph.mds.a.smithi047.stderr:2024-02-20T00:00:01.230+0000 7f923e400640 -1 received signal: Hangup from (PID: 55278) UID: 0 2024-02-20T00:00:01.237 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-02-20T00:00:01.230+0000 7fc107400640 -1 Fail to open '/proc/55278/cmdline' error = (2) No such file or directory 2024-02-20T00:00:01.237 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-02-20T00:00:01.230+0000 7fc107400640 -1 received signal: Hangup from (PID: 55278) UID: 0 2024-02-20T00:00:01.237 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-02-20T00:00:01.230+0000 7fc107400640 -1 mon.a@0(leader) e1 *** Got Signal Hangup *** 2024-02-20T00:00:01.238 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-20T00:00:01.230+0000 7fad7d200640 -1 Fail to open '/proc/55278/cmdline' error = (2) No such file or directory 2024-02-20T00:00:01.238 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-02-20T00:00:01.230+0000 7fad7d200640 -1 received signal: Hangup from (PID: 55278) UID: 0 2024-02-20T00:00:02.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:02.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:02.053791+0000) 2024-02-20T00:00:03.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:03.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:03.054104+0000) 2024-02-20T00:00:04.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:04.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:04.054308+0000) 2024-02-20T00:00:05.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:05.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:05.054578+0000) 2024-02-20T00:00:06.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:06.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:06.054852+0000) 2024-02-20T00:00:07.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:07.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:07.055068+0000) 2024-02-20T00:00:08.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:08.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:08.055268+0000) 2024-02-20T00:00:09.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:09.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:09.055529+0000) 2024-02-20T00:00:10.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:10.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:10.055821+0000) 2024-02-20T00:00:11.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:11.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:11.056038+0000) 2024-02-20T00:00:12.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:12.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:12.056282+0000) 2024-02-20T00:00:13.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:13.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:13.056527+0000) 2024-02-20T00:00:14.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:14.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:14.056787+0000) 2024-02-20T00:00:15.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:15.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:15.057041+0000) 2024-02-20T00:00:16.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:16.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:16.057267+0000) 2024-02-20T00:00:17.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:17.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:17.057478+0000) 2024-02-20T00:00:18.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:18.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:18.057630+0000) 2024-02-20T00:00:19.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:19.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:19.057871+0000) 2024-02-20T00:00:20.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:20.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:20.058096+0000) 2024-02-20T00:00:21.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:21.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:21.058364+0000) 2024-02-20T00:00:22.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:22.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:22.058543+0000) 2024-02-20T00:00:23.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:23.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:23.058830+0000) 2024-02-20T00:00:24.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:24.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:24.058967+0000) 2024-02-20T00:00:25.019 INFO:tasks.ceph.mon.b.smithi116.stderr:2024-02-20T00:00:25.014+0000 7f662fe00640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 36895) UID: 0 2024-02-20T00:00:25.019 INFO:tasks.ceph.mon.b.smithi116.stderr:2024-02-20T00:00:25.014+0000 7f662fe00640 -1 mon.b@1(peon) e1 *** Got Signal Hangup *** 2024-02-20T00:00:25.020 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-20T00:00:25.014+0000 7f67e5000640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 36895) UID: 0 2024-02-20T00:00:25.020 INFO:tasks.ceph.mon.c.smithi116.stderr:2024-02-20T00:00:25.014+0000 7f899f800640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 36895) UID: 0 2024-02-20T00:00:25.020 INFO:tasks.ceph.mon.c.smithi116.stderr:2024-02-20T00:00:25.014+0000 7f899f800640 -1 mon.c@2(peon) e1 *** Got Signal Hangup *** 2024-02-20T00:00:25.020 INFO:tasks.ceph.osd.4.smithi116.stderr:2024-02-20T00:00:25.014+0000 7f9094e00640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 36895) UID: 0 2024-02-20T00:00:25.021 INFO:tasks.ceph.osd.5.smithi116.stderr:2024-02-20T00:00:25.014+0000 7f0f29200640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 36895) UID: 0 2024-02-20T00:00:25.021 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-02-20T00:00:25.014+0000 7f7323a00640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 36895) UID: 0 2024-02-20T00:00:25.021 INFO:tasks.ceph.osd.7.smithi116.stderr:2024-02-20T00:00:25.014+0000 7f7364e00640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 36895) UID: 0 2024-02-20T00:00:25.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:25.018+0000 7f70ff200640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 36895) UID: 0 2024-02-20T00:00:25.044 INFO:tasks.ceph.mon.b.smithi116.stderr:2024-02-20T00:00:25.042+0000 7f662fe00640 -1 Fail to open '/proc/36896/cmdline' error = (2) No such file or directory 2024-02-20T00:00:25.044 INFO:tasks.ceph.mon.b.smithi116.stderr:2024-02-20T00:00:25.042+0000 7f662fe00640 -1 received signal: Hangup from (PID: 36896) UID: 0 2024-02-20T00:00:25.044 INFO:tasks.ceph.mon.b.smithi116.stderr:2024-02-20T00:00:25.042+0000 7f662fe00640 -1 mon.b@1(peon) e1 *** Got Signal Hangup *** 2024-02-20T00:00:25.045 INFO:tasks.ceph.mon.c.smithi116.stderr:2024-02-20T00:00:25.042+0000 7f899f800640 -1 Fail to open '/proc/36896/cmdline' error = (2) No such file or directory 2024-02-20T00:00:25.045 INFO:tasks.ceph.mon.c.smithi116.stderr:2024-02-20T00:00:25.042+0000 7f899f800640 -1 received signal: Hangup from (PID: 36896) UID: 0 2024-02-20T00:00:25.045 INFO:tasks.ceph.mon.c.smithi116.stderr:2024-02-20T00:00:25.042+0000 7f899f800640 -1 mon.c@2(peon) e1 *** Got Signal Hangup *** 2024-02-20T00:00:25.045 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-20T00:00:25.042+0000 7f67e5000640 -1 Fail to read '/proc/36896/cmdline' error = (3) No such process 2024-02-20T00:00:25.045 INFO:tasks.ceph.mgr.x.smithi116.stderr:2024-02-20T00:00:25.042+0000 7f67e5000640 -1 received signal: Hangup from (PID: 36896) UID: 0 2024-02-20T00:00:25.045 INFO:tasks.ceph.osd.4.smithi116.stderr:2024-02-20T00:00:25.042+0000 7f9094e00640 -1 Fail to read '/proc/36896/cmdline' error = (3) No such process 2024-02-20T00:00:25.046 INFO:tasks.ceph.osd.4.smithi116.stderr:2024-02-20T00:00:25.042+0000 7f9094e00640 -1 received signal: Hangup from (PID: 36896) UID: 0 2024-02-20T00:00:25.046 INFO:tasks.ceph.osd.5.smithi116.stderr:2024-02-20T00:00:25.042+0000 7f0f29200640 -1 Fail to open '/proc/36896/cmdline' error = (2) No such file or directory 2024-02-20T00:00:25.046 INFO:tasks.ceph.osd.5.smithi116.stderr:2024-02-20T00:00:25.042+0000 7f0f29200640 -1 received signal: Hangup from (PID: 36896) UID: 0 2024-02-20T00:00:25.047 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-02-20T00:00:25.042+0000 7f7323a00640 -1 Fail to read '/proc/36896/cmdline' error = (3) No such process 2024-02-20T00:00:25.047 INFO:tasks.ceph.osd.6.smithi116.stderr:2024-02-20T00:00:25.042+0000 7f7323a00640 -1 received signal: Hangup from (PID: 36896) UID: 0 2024-02-20T00:00:25.047 INFO:tasks.ceph.osd.7.smithi116.stderr:2024-02-20T00:00:25.042+0000 7f7364e00640 -1 Fail to open '/proc/36896/cmdline' error = (2) No such file or directory 2024-02-20T00:00:25.047 INFO:tasks.ceph.osd.7.smithi116.stderr:2024-02-20T00:00:25.042+0000 7f7364e00640 -1 received signal: Hangup from (PID: 36896) UID: 0 2024-02-20T00:00:25.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:25.042+0000 7f70ff200640 -1 Fail to open '/proc/36896/cmdline' error = (2) No such file or directory 2024-02-20T00:00:25.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:25.042+0000 7f70ff200640 -1 received signal: Hangup from (PID: 36896) UID: 0 2024-02-20T00:00:25.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:25.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:25.059105+0000) 2024-02-20T00:00:26.019 DEBUG:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-20T00:00:26.022 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-20T00:00:26.038 INFO:teuthology.orchestra.run.smithi047.stderr:error: state file /var/lib/logrotate/status is already locked 2024-02-20T00:00:26.039 INFO:teuthology.orchestra.run.smithi047.stderr:logrotate does not support parallel execution on the same set of logfiles. 2024-02-20T00:00:26.040 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-02-20T00:00:26.041 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/qa/tasks/ceph.py", line 189, in invoke_logrotate run.wait(procs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi047 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2024-02-20T00:00:26.071 INFO:teuthology.orchestra.run.smithi116.stderr:error: state file /var/lib/logrotate/status is already locked 2024-02-20T00:00:26.071 INFO:teuthology.orchestra.run.smithi116.stderr:logrotate does not support parallel execution on the same set of logfiles. 2024-02-20T00:00:26.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:26.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:26.059227+0000) 2024-02-20T00:00:27.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:27.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:27.059371+0000) 2024-02-20T00:00:28.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:28.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:28.059511+0000) 2024-02-20T00:00:29.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:29.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:29.059647+0000) 2024-02-20T00:00:30.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:30.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:30.059793+0000) 2024-02-20T00:00:31.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:31.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:31.059961+0000) 2024-02-20T00:00:32.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:32.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:32.060108+0000) 2024-02-20T00:00:33.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:33.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:33.060275+0000) 2024-02-20T00:00:34.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:34.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:34.060431+0000) 2024-02-20T00:00:35.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:35.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:35.060589+0000) 2024-02-20T00:00:36.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:36.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:36.060739+0000) 2024-02-20T00:00:37.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:37.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:37.060893+0000) 2024-02-20T00:00:38.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:38.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:38.061041+0000) 2024-02-20T00:00:39.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:39.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:39.061191+0000) 2024-02-20T00:00:40.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:40.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:40.061345+0000) 2024-02-20T00:00:41.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:41.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:41.061507+0000) 2024-02-20T00:00:42.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:42.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:42.061655+0000) 2024-02-20T00:00:43.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:43.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:43.061807+0000) 2024-02-20T00:00:44.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:44.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:44.061991+0000) 2024-02-20T00:00:45.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:45.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:45.062165+0000) 2024-02-20T00:00:46.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:46.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:46.062310+0000) 2024-02-20T00:00:47.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:47.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:47.062487+0000) 2024-02-20T00:00:48.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:48.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:48.062636+0000) 2024-02-20T00:00:49.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:49.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:49.062805+0000) 2024-02-20T00:00:50.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:50.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:50.062941+0000) 2024-02-20T00:00:51.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:51.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:51.063114+0000) 2024-02-20T00:00:52.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:52.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:52.063254+0000) 2024-02-20T00:00:53.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:53.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:53.063398+0000) 2024-02-20T00:00:54.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:54.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:54.063544+0000) 2024-02-20T00:00:55.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:55.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:55.063691+0000) 2024-02-20T00:00:56.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:56.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:56.063829+0000) 2024-02-20T00:00:57.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:57.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:57.063940+0000) 2024-02-20T00:00:58.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:58.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:58.064086+0000) 2024-02-20T00:00:59.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:00:59.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:00:59.064233+0000) 2024-02-20T00:01:00.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:00.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:00.064386+0000) 2024-02-20T00:01:01.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:01.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:01.064539+0000) 2024-02-20T00:01:02.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:02.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:02.064685+0000) 2024-02-20T00:01:03.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:03.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:03.064832+0000) 2024-02-20T00:01:04.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:04.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:04.064976+0000) 2024-02-20T00:01:05.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:05.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:05.065143+0000) 2024-02-20T00:01:06.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:06.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:06.065308+0000) 2024-02-20T00:01:07.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:07.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:07.065481+0000) 2024-02-20T00:01:08.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:08.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:08.065642+0000) 2024-02-20T00:01:09.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:09.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:09.065793+0000) 2024-02-20T00:01:10.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:10.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:10.065945+0000) 2024-02-20T00:01:11.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:11.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:11.066115+0000) 2024-02-20T00:01:12.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:12.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:12.066265+0000) 2024-02-20T00:01:13.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:13.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:13.066411+0000) 2024-02-20T00:01:14.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:14.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:14.066549+0000) 2024-02-20T00:01:15.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:15.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:15.066701+0000) 2024-02-20T00:01:16.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:16.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:16.066859+0000) 2024-02-20T00:01:17.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:17.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:17.067005+0000) 2024-02-20T00:01:18.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:18.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:18.067139+0000) 2024-02-20T00:01:19.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:19.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:19.067265+0000) 2024-02-20T00:01:20.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:20.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:20.067440+0000) 2024-02-20T00:01:21.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:21.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:21.067698+0000) 2024-02-20T00:01:22.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:22.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:22.067955+0000) 2024-02-20T00:01:23.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:23.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:23.068186+0000) 2024-02-20T00:01:24.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:24.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:24.068416+0000) 2024-02-20T00:01:25.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:25.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:25.068632+0000) 2024-02-20T00:01:26.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:26.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:26.068886+0000) 2024-02-20T00:01:27.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:27.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:27.069151+0000) 2024-02-20T00:01:28.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:28.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:28.069409+0000) 2024-02-20T00:01:29.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:29.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:29.069625+0000) 2024-02-20T00:01:30.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:30.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:30.069813+0000) 2024-02-20T00:01:31.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:31.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:31.070081+0000) 2024-02-20T00:01:32.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:32.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:32.070279+0000) 2024-02-20T00:01:33.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:33.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:33.070502+0000) 2024-02-20T00:01:34.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:34.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:34.070637+0000) 2024-02-20T00:01:35.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:35.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:35.070893+0000) 2024-02-20T00:01:36.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:36.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:36.071074+0000) 2024-02-20T00:01:37.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:37.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:37.071256+0000) 2024-02-20T00:01:38.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:38.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:38.071523+0000) 2024-02-20T00:01:39.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:39.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:39.071674+0000) 2024-02-20T00:01:40.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:40.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:40.071912+0000) 2024-02-20T00:01:41.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:41.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:41.072159+0000) 2024-02-20T00:01:42.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:42.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:42.072335+0000) 2024-02-20T00:01:43.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:43.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:43.072567+0000) 2024-02-20T00:01:44.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:44.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:44.072711+0000) 2024-02-20T00:01:45.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:45.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:45.072941+0000) 2024-02-20T00:01:46.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:46.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:46.073098+0000) 2024-02-20T00:01:47.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:47.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:47.073325+0000) 2024-02-20T00:01:48.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:48.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:48.073508+0000) 2024-02-20T00:01:49.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:49.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:49.073682+0000) 2024-02-20T00:01:50.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:50.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:50.073830+0000) 2024-02-20T00:01:51.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:51.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:51.073997+0000) 2024-02-20T00:01:52.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:52.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:52.074155+0000) 2024-02-20T00:01:53.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:53.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:53.074378+0000) 2024-02-20T00:01:54.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:54.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:54.074558+0000) 2024-02-20T00:01:55.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:55.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:55.074729+0000) 2024-02-20T00:01:56.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:56.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:56.074860+0000) 2024-02-20T00:01:57.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:57.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:57.075000+0000) 2024-02-20T00:01:58.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:58.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:58.075213+0000) 2024-02-20T00:01:59.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:01:59.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:01:59.075407+0000) 2024-02-20T00:02:00.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:00.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:00.075662+0000) 2024-02-20T00:02:01.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:01.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:01.075938+0000) 2024-02-20T00:02:02.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:02.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:02.076094+0000) 2024-02-20T00:02:03.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:03.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:03.076323+0000) 2024-02-20T00:02:04.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:04.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:04.076474+0000) 2024-02-20T00:02:05.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:05.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:05.076694+0000) 2024-02-20T00:02:06.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:06.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:06.076851+0000) 2024-02-20T00:02:07.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:07.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:07.077028+0000) 2024-02-20T00:02:08.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:08.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:08.077164+0000) 2024-02-20T00:02:09.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:09.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:09.077356+0000) 2024-02-20T00:02:10.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:10.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:10.077571+0000) 2024-02-20T00:02:11.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:11.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:11.077861+0000) 2024-02-20T00:02:12.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:12.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:12.078094+0000) 2024-02-20T00:02:13.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:13.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:13.078337+0000) 2024-02-20T00:02:14.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:14.072+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:14.078572+0000) 2024-02-20T00:02:15.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:15.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:15.078848+0000) 2024-02-20T00:02:16.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:16.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:16.079080+0000) 2024-02-20T00:02:17.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:17.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:17.079307+0000) 2024-02-20T00:02:18.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:18.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:18.079533+0000) 2024-02-20T00:02:19.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:19.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:19.079732+0000) 2024-02-20T00:02:20.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:20.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:20.079967+0000) 2024-02-20T00:02:21.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:21.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:21.080249+0000) 2024-02-20T00:02:22.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:22.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:22.080500+0000) 2024-02-20T00:02:23.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:23.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:23.080726+0000) 2024-02-20T00:02:24.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:24.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:24.080904+0000) 2024-02-20T00:02:25.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:25.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:25.081068+0000) 2024-02-20T00:02:26.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:26.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:26.081291+0000) 2024-02-20T00:02:27.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:27.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:27.081469+0000) 2024-02-20T00:02:28.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:28.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:28.081640+0000) 2024-02-20T00:02:29.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:29.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:29.081888+0000) 2024-02-20T00:02:30.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:30.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:30.082136+0000) 2024-02-20T00:02:31.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:31.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:31.082331+0000) 2024-02-20T00:02:32.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:32.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:32.082574+0000) 2024-02-20T00:02:33.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:33.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:33.082847+0000) 2024-02-20T00:02:34.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:34.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:34.083008+0000) 2024-02-20T00:02:35.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:35.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:35.083199+0000) 2024-02-20T00:02:36.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:36.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:36.083354+0000) 2024-02-20T00:02:37.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:37.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:37.083542+0000) 2024-02-20T00:02:38.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:38.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:38.083685+0000) 2024-02-20T00:02:39.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:39.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:39.083873+0000) 2024-02-20T00:02:40.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:40.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:40.084149+0000) 2024-02-20T00:02:41.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:41.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:41.084408+0000) 2024-02-20T00:02:42.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:42.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:42.084630+0000) 2024-02-20T00:02:43.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:43.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:43.084866+0000) 2024-02-20T00:02:44.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:44.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:44.085100+0000) 2024-02-20T00:02:45.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:45.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:45.085270+0000) 2024-02-20T00:02:46.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:46.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:46.085494+0000) 2024-02-20T00:02:47.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:47.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:47.085727+0000) 2024-02-20T00:02:48.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:48.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:48.085866+0000) 2024-02-20T00:02:49.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:49.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:49.086010+0000) 2024-02-20T00:02:50.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:50.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:50.086152+0000) 2024-02-20T00:02:51.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:51.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:51.086380+0000) 2024-02-20T00:02:52.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:52.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:52.086628+0000) 2024-02-20T00:02:53.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:53.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:53.086820+0000) 2024-02-20T00:02:54.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:54.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:54.086970+0000) 2024-02-20T00:02:55.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:55.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:55.087107+0000) 2024-02-20T00:02:56.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:56.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:56.087254+0000) 2024-02-20T00:02:57.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:57.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:57.087443+0000) 2024-02-20T00:02:58.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:58.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:58.087584+0000) 2024-02-20T00:02:59.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:02:59.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:02:59.087770+0000) 2024-02-20T00:03:00.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:00.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:00.088018+0000) 2024-02-20T00:03:01.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:01.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:01.088250+0000) 2024-02-20T00:03:02.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:02.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:02.088435+0000) 2024-02-20T00:03:03.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:03.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:03.088597+0000) 2024-02-20T00:03:04.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:04.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:04.088822+0000) 2024-02-20T00:03:05.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:05.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:05.089048+0000) 2024-02-20T00:03:06.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:06.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:06.089325+0000) 2024-02-20T00:03:07.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:07.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:07.089553+0000) 2024-02-20T00:03:08.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:08.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:08.089753+0000) 2024-02-20T00:03:09.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:09.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:09.089943+0000) 2024-02-20T00:03:10.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:10.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:10.090184+0000) 2024-02-20T00:03:11.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:11.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:11.090353+0000) 2024-02-20T00:03:12.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:12.083+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:12.090601+0000) 2024-02-20T00:03:13.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:13.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:13.090868+0000) 2024-02-20T00:03:14.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:14.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:14.091077+0000) 2024-02-20T00:03:15.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:15.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:15.091314+0000) 2024-02-20T00:03:16.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:16.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:16.091539+0000) 2024-02-20T00:03:17.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:17.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:17.091817+0000) 2024-02-20T00:03:18.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:18.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:18.092072+0000) 2024-02-20T00:03:19.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:19.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:19.092292+0000) 2024-02-20T00:03:20.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:20.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:20.092531+0000) 2024-02-20T00:03:21.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:21.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:21.092754+0000) 2024-02-20T00:03:22.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:22.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:22.092976+0000) 2024-02-20T00:03:23.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:23.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:23.093210+0000) 2024-02-20T00:03:24.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:24.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:24.093389+0000) 2024-02-20T00:03:25.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:25.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:25.093552+0000) 2024-02-20T00:03:26.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:26.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:26.093685+0000) 2024-02-20T00:03:27.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:27.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:27.093921+0000) 2024-02-20T00:03:28.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:28.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:28.094068+0000) 2024-02-20T00:03:29.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:29.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:29.094199+0000) 2024-02-20T00:03:30.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:30.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:30.094424+0000) 2024-02-20T00:03:31.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:31.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:31.094636+0000) 2024-02-20T00:03:32.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:32.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:32.094806+0000) 2024-02-20T00:03:33.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:33.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:33.094936+0000) 2024-02-20T00:03:34.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:34.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:34.095092+0000) 2024-02-20T00:03:35.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:35.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:35.095272+0000) 2024-02-20T00:03:36.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:36.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:36.095441+0000) 2024-02-20T00:03:37.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:37.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:37.095626+0000) 2024-02-20T00:03:38.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:38.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:38.095845+0000) 2024-02-20T00:03:39.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:39.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:39.096056+0000) 2024-02-20T00:03:40.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:40.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:40.096274+0000) 2024-02-20T00:03:41.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:41.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:41.096528+0000) 2024-02-20T00:03:42.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:42.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:42.096671+0000) 2024-02-20T00:03:43.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:43.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:43.096891+0000) 2024-02-20T00:03:44.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:44.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:44.097125+0000) 2024-02-20T00:03:45.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:45.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:45.097385+0000) 2024-02-20T00:03:46.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:46.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:46.097576+0000) 2024-02-20T00:03:47.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:47.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:47.097800+0000) 2024-02-20T00:03:48.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:48.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:48.098006+0000) 2024-02-20T00:03:49.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:49.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:49.098307+0000) 2024-02-20T00:03:50.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:50.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:50.098550+0000) 2024-02-20T00:03:51.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:51.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:51.098857+0000) 2024-02-20T00:03:52.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:52.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:52.099035+0000) 2024-02-20T00:03:53.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:53.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:53.099209+0000) 2024-02-20T00:03:54.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:54.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:54.099431+0000) 2024-02-20T00:03:55.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:55.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:55.099673+0000) 2024-02-20T00:03:56.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:56.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:56.099924+0000) 2024-02-20T00:03:57.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:57.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:57.100121+0000) 2024-02-20T00:03:58.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:58.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:58.100281+0000) 2024-02-20T00:03:59.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:03:59.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:03:59.100462+0000) 2024-02-20T00:04:00.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:00.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:00.100640+0000) 2024-02-20T00:04:01.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:01.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:01.100838+0000) 2024-02-20T00:04:02.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:02.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:02.101125+0000) 2024-02-20T00:04:03.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:03.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:03.101416+0000) 2024-02-20T00:04:04.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:04.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:04.101591+0000) 2024-02-20T00:04:05.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:05.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:05.101757+0000) 2024-02-20T00:04:06.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:06.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:06.101970+0000) 2024-02-20T00:04:07.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:07.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:07.102153+0000) 2024-02-20T00:04:08.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:08.094+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:08.102413+0000) 2024-02-20T00:04:09.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:09.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:09.102703+0000) 2024-02-20T00:04:10.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:10.098+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:10.102974+0000) 2024-02-20T00:04:11.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:11.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:11.103127+0000) 2024-02-20T00:04:12.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:12.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:12.103279+0000) 2024-02-20T00:04:13.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:13.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:13.103532+0000) 2024-02-20T00:04:14.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:14.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:14.103753+0000) 2024-02-20T00:04:15.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:15.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:15.103897+0000) 2024-02-20T00:04:16.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:16.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:16.104085+0000) 2024-02-20T00:04:17.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:17.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:17.104243+0000) 2024-02-20T00:04:18.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:18.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:18.104406+0000) 2024-02-20T00:04:19.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:19.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:19.104659+0000) 2024-02-20T00:04:20.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:20.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:20.104953+0000) 2024-02-20T00:04:21.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:21.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:21.105148+0000) 2024-02-20T00:04:22.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:22.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:22.105311+0000) 2024-02-20T00:04:23.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:23.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:23.105466+0000) 2024-02-20T00:04:24.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:24.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:24.105608+0000) 2024-02-20T00:04:25.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:25.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:25.105742+0000) 2024-02-20T00:04:26.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:26.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:26.105980+0000) 2024-02-20T00:04:27.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:27.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:27.106291+0000) 2024-02-20T00:04:28.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:28.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:28.106657+0000) 2024-02-20T00:04:29.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:29.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:29.106788+0000) 2024-02-20T00:04:30.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:30.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:30.106972+0000) 2024-02-20T00:04:31.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:31.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:31.107143+0000) 2024-02-20T00:04:32.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:32.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:32.107379+0000) 2024-02-20T00:04:33.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:33.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:33.107582+0000) 2024-02-20T00:04:34.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:34.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:34.107806+0000) 2024-02-20T00:04:35.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:35.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:35.107950+0000) 2024-02-20T00:04:36.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:36.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:36.108103+0000) 2024-02-20T00:04:37.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:37.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:37.108229+0000) 2024-02-20T00:04:38.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:38.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:38.108534+0000) 2024-02-20T00:04:39.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:39.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:39.108731+0000) 2024-02-20T00:04:40.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:40.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:40.108881+0000) 2024-02-20T00:04:41.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:41.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:41.109027+0000) 2024-02-20T00:04:42.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:42.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:42.109256+0000) 2024-02-20T00:04:43.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:43.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:43.109526+0000) 2024-02-20T00:04:44.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:44.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:44.109662+0000) 2024-02-20T00:04:45.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:45.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:45.109806+0000) 2024-02-20T00:04:46.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:46.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:46.110045+0000) 2024-02-20T00:04:47.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:47.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:47.110302+0000) 2024-02-20T00:04:48.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:48.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:48.110559+0000) 2024-02-20T00:04:49.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:49.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:49.110698+0000) 2024-02-20T00:04:50.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:50.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:50.110874+0000) 2024-02-20T00:04:51.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:51.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:51.111050+0000) 2024-02-20T00:04:52.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:52.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:52.111195+0000) 2024-02-20T00:04:53.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:53.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:53.111350+0000) 2024-02-20T00:04:54.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:54.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:54.111591+0000) 2024-02-20T00:04:55.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:55.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:55.111847+0000) 2024-02-20T00:04:56.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:56.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:56.112016+0000) 2024-02-20T00:04:57.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:57.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:57.112166+0000) 2024-02-20T00:04:58.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:58.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:58.112324+0000) 2024-02-20T00:04:59.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:04:59.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:04:59.112577+0000) 2024-02-20T00:05:00.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:00.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:00.112857+0000) 2024-02-20T00:05:01.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:01.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:01.113325+0000) 2024-02-20T00:05:02.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:02.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:02.113525+0000) 2024-02-20T00:05:03.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:03.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:03.113747+0000) 2024-02-20T00:05:04.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:04.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:04.113945+0000) 2024-02-20T00:05:05.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:05.109+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:05.114141+0000) 2024-02-20T00:05:06.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:06.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:06.114292+0000) 2024-02-20T00:05:07.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:07.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:07.114449+0000) 2024-02-20T00:05:08.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:08.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:08.114720+0000) 2024-02-20T00:05:09.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:09.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:09.114860+0000) 2024-02-20T00:05:10.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:10.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:10.115024+0000) 2024-02-20T00:05:11.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:11.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:11.115305+0000) 2024-02-20T00:05:12.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:12.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:12.115596+0000) 2024-02-20T00:05:13.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:13.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:13.115758+0000) 2024-02-20T00:05:14.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:14.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:14.115880+0000) 2024-02-20T00:05:15.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:15.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:15.116004+0000) 2024-02-20T00:05:16.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:16.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:16.116153+0000) 2024-02-20T00:05:17.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:17.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:17.116350+0000) 2024-02-20T00:05:18.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:18.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:18.116517+0000) 2024-02-20T00:05:19.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:19.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:19.116670+0000) 2024-02-20T00:05:20.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:20.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:20.116829+0000) 2024-02-20T00:05:21.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:21.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:21.117014+0000) 2024-02-20T00:05:22.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:22.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:22.117194+0000) 2024-02-20T00:05:23.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:23.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:23.117450+0000) 2024-02-20T00:05:24.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:24.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:24.117618+0000) 2024-02-20T00:05:25.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:25.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:25.117752+0000) 2024-02-20T00:05:26.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:26.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:26.117883+0000) 2024-02-20T00:05:27.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:27.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:27.118020+0000) 2024-02-20T00:05:28.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:28.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:28.118173+0000) 2024-02-20T00:05:29.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:29.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:29.118329+0000) 2024-02-20T00:05:30.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:30.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:30.118491+0000) 2024-02-20T00:05:31.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:31.113+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:31.118655+0000) 2024-02-20T00:05:32.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:32.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:32.118817+0000) 2024-02-20T00:05:33.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:33.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:33.119091+0000) 2024-02-20T00:05:34.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:34.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:34.119220+0000) 2024-02-20T00:05:35.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:35.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:35.119470+0000) 2024-02-20T00:05:36.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:36.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:36.119637+0000) 2024-02-20T00:05:37.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:37.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:37.119762+0000) 2024-02-20T00:05:38.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:38.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:38.119904+0000) 2024-02-20T00:05:39.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:39.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:39.120044+0000) 2024-02-20T00:05:40.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:40.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:40.120253+0000) 2024-02-20T00:05:41.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:41.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:41.120449+0000) 2024-02-20T00:05:42.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:42.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:42.120638+0000) 2024-02-20T00:05:43.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:43.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:43.120833+0000) 2024-02-20T00:05:44.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:44.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:44.121066+0000) 2024-02-20T00:05:45.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:45.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:45.121279+0000) 2024-02-20T00:05:46.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:46.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:46.121433+0000) 2024-02-20T00:05:47.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:47.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:47.121624+0000) 2024-02-20T00:05:48.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:48.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:48.121760+0000) 2024-02-20T00:05:49.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:49.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:49.121965+0000) 2024-02-20T00:05:50.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:50.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:50.122083+0000) 2024-02-20T00:05:51.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:51.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:51.122240+0000) 2024-02-20T00:05:52.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:52.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:52.122480+0000) 2024-02-20T00:05:53.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:53.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:53.122657+0000) 2024-02-20T00:05:54.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:54.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:54.122836+0000) 2024-02-20T00:05:55.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:55.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:55.122971+0000) 2024-02-20T00:05:56.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:56.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:56.123202+0000) 2024-02-20T00:05:57.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:57.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:57.123360+0000) 2024-02-20T00:05:58.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:58.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:58.123586+0000) 2024-02-20T00:05:59.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:05:59.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:05:59.123814+0000) 2024-02-20T00:06:00.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:00.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:00.124024+0000) 2024-02-20T00:06:01.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:01.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:01.124222+0000) 2024-02-20T00:06:02.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:02.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:02.124459+0000) 2024-02-20T00:06:03.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:03.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:03.124692+0000) 2024-02-20T00:06:04.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:04.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:04.124916+0000) 2024-02-20T00:06:05.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:05.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:05.125094+0000) 2024-02-20T00:06:06.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:06.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:06.125289+0000) 2024-02-20T00:06:07.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:07.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:07.125528+0000) 2024-02-20T00:06:08.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:08.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:08.125692+0000) 2024-02-20T00:06:09.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:09.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:09.125949+0000) 2024-02-20T00:06:10.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:10.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:10.126136+0000) 2024-02-20T00:06:11.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:11.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:11.126358+0000) 2024-02-20T00:06:12.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:12.120+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:12.126600+0000) 2024-02-20T00:06:13.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:13.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:13.126853+0000) 2024-02-20T00:06:14.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:14.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:14.127096+0000) 2024-02-20T00:06:15.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:15.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:15.127297+0000) 2024-02-20T00:06:16.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:16.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:16.127557+0000) 2024-02-20T00:06:17.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:17.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:17.127725+0000) 2024-02-20T00:06:18.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:18.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:18.127887+0000) 2024-02-20T00:06:19.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:19.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:19.128062+0000) 2024-02-20T00:06:20.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:20.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:20.128268+0000) 2024-02-20T00:06:21.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:21.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:21.128594+0000) 2024-02-20T00:06:22.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:22.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:22.128823+0000) 2024-02-20T00:06:23.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:23.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:23.128990+0000) 2024-02-20T00:06:24.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:24.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:24.129170+0000) 2024-02-20T00:06:25.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:25.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:25.129334+0000) 2024-02-20T00:06:26.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:26.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:26.129493+0000) 2024-02-20T00:06:27.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:27.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:27.129662+0000) 2024-02-20T00:06:28.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:28.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:28.129844+0000) 2024-02-20T00:06:29.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:29.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:29.130019+0000) 2024-02-20T00:06:30.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:30.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:30.130208+0000) 2024-02-20T00:06:31.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:31.124+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:31.130484+0000) 2024-02-20T00:06:32.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:32.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:32.130700+0000) 2024-02-20T00:06:33.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:33.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:33.130855+0000) 2024-02-20T00:06:34.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:34.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:34.131115+0000) 2024-02-20T00:06:35.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:35.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:35.131333+0000) 2024-02-20T00:06:36.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:36.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:36.131543+0000) 2024-02-20T00:06:37.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:37.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:37.131781+0000) 2024-02-20T00:06:38.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:38.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:38.131967+0000) 2024-02-20T00:06:39.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:39.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:39.132128+0000) 2024-02-20T00:06:40.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:40.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:40.132287+0000) 2024-02-20T00:06:41.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:41.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:41.132514+0000) 2024-02-20T00:06:42.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:42.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:42.132720+0000) 2024-02-20T00:06:43.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:43.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:43.132917+0000) 2024-02-20T00:06:44.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:44.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:44.133075+0000) 2024-02-20T00:06:45.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:45.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:45.133258+0000) 2024-02-20T00:06:46.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:46.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:46.133515+0000) 2024-02-20T00:06:47.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:47.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:47.133686+0000) 2024-02-20T00:06:48.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:48.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:48.133909+0000) 2024-02-20T00:06:49.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:49.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:49.134154+0000) 2024-02-20T00:06:50.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:50.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:50.134386+0000) 2024-02-20T00:06:51.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:51.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:51.134581+0000) 2024-02-20T00:06:52.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:52.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:52.134859+0000) 2024-02-20T00:06:53.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:53.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:53.135055+0000) 2024-02-20T00:06:54.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:54.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:54.135213+0000) 2024-02-20T00:06:55.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:55.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:55.135441+0000) 2024-02-20T00:06:56.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:56.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:56.135647+0000) 2024-02-20T00:06:57.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:57.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:57.135862+0000) 2024-02-20T00:06:58.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:58.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:58.136115+0000) 2024-02-20T00:06:59.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:06:59.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:06:59.136372+0000) 2024-02-20T00:07:00.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:00.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:00.136543+0000) 2024-02-20T00:07:01.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:01.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:01.136767+0000) 2024-02-20T00:07:02.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:02.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:02.136944+0000) 2024-02-20T00:07:03.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:03.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:03.137181+0000) 2024-02-20T00:07:04.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:04.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:04.137405+0000) 2024-02-20T00:07:05.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:05.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:05.137662+0000) 2024-02-20T00:07:06.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:06.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:06.137809+0000) 2024-02-20T00:07:07.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:07.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:07.137969+0000) 2024-02-20T00:07:08.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:08.131+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:08.138189+0000) 2024-02-20T00:07:09.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:09.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:09.138373+0000) 2024-02-20T00:07:10.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:10.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:10.138556+0000) 2024-02-20T00:07:11.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:11.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:11.138877+0000) 2024-02-20T00:07:12.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:12.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:12.139102+0000) 2024-02-20T00:07:13.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:13.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:13.139292+0000) 2024-02-20T00:07:14.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:14.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:14.139470+0000) 2024-02-20T00:07:15.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:15.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:15.139637+0000) 2024-02-20T00:07:16.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:16.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:16.139876+0000) 2024-02-20T00:07:17.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:17.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:17.140067+0000) 2024-02-20T00:07:18.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:18.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:18.140195+0000) 2024-02-20T00:07:19.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:19.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:19.140357+0000) 2024-02-20T00:07:20.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:20.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:20.140558+0000) 2024-02-20T00:07:21.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:21.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:21.140808+0000) 2024-02-20T00:07:22.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:22.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:22.141003+0000) 2024-02-20T00:07:23.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:23.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:23.141166+0000) 2024-02-20T00:07:24.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:24.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:24.141388+0000) 2024-02-20T00:07:25.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:25.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:25.141633+0000) 2024-02-20T00:07:26.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:26.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:26.141874+0000) 2024-02-20T00:07:27.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:27.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:27.142082+0000) 2024-02-20T00:07:28.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:28.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:28.142303+0000) 2024-02-20T00:07:29.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:29.135+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:29.142459+0000) 2024-02-20T00:07:30.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:30.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:30.142684+0000) 2024-02-20T00:07:31.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:31.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:31.142951+0000) 2024-02-20T00:07:32.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:32.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:32.143093+0000) 2024-02-20T00:07:33.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:33.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:33.143317+0000) 2024-02-20T00:07:34.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:34.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:34.143558+0000) 2024-02-20T00:07:35.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:35.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:35.143722+0000) 2024-02-20T00:07:36.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:36.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:36.143945+0000) 2024-02-20T00:07:37.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:37.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:37.144079+0000) 2024-02-20T00:07:38.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:38.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:38.144239+0000) 2024-02-20T00:07:39.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:39.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:39.144397+0000) 2024-02-20T00:07:40.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:40.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:40.144576+0000) 2024-02-20T00:07:41.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:41.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:41.144814+0000) 2024-02-20T00:07:42.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:42.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:42.145090+0000) 2024-02-20T00:07:43.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:43.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:43.145285+0000) 2024-02-20T00:07:44.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:44.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:44.145435+0000) 2024-02-20T00:07:45.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:45.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:45.145605+0000) 2024-02-20T00:07:46.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:46.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:46.145787+0000) 2024-02-20T00:07:47.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:47.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:47.145974+0000) 2024-02-20T00:07:48.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:48.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:48.146140+0000) 2024-02-20T00:07:49.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:49.139+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:49.146380+0000) 2024-02-20T00:07:50.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:50.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:50.146551+0000) 2024-02-20T00:07:51.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:51.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:51.146773+0000) 2024-02-20T00:07:52.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:52.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:52.146940+0000) 2024-02-20T00:07:53.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:53.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:53.147123+0000) 2024-02-20T00:07:54.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:54.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:54.147340+0000) 2024-02-20T00:07:55.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:55.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:55.147612+0000) 2024-02-20T00:07:56.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:56.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:56.147812+0000) 2024-02-20T00:07:57.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:57.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:57.148055+0000) 2024-02-20T00:07:58.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:58.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:58.148201+0000) 2024-02-20T00:07:59.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:07:59.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:07:59.148393+0000) 2024-02-20T00:08:00.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:00.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:00.148637+0000) 2024-02-20T00:08:01.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:01.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:01.148878+0000) 2024-02-20T00:08:02.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:02.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:02.149071+0000) 2024-02-20T00:08:03.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:03.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:03.149303+0000) 2024-02-20T00:08:04.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:04.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:04.149468+0000) 2024-02-20T00:08:05.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:05.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:05.149686+0000) 2024-02-20T00:08:06.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:06.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:06.149844+0000) 2024-02-20T00:08:07.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:07.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:07.149969+0000) 2024-02-20T00:08:08.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:08.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:08.150198+0000) 2024-02-20T00:08:09.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:09.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:09.150405+0000) 2024-02-20T00:08:10.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:10.142+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:10.150663+0000) 2024-02-20T00:08:11.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:11.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:11.150941+0000) 2024-02-20T00:08:12.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:12.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:12.151139+0000) 2024-02-20T00:08:13.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:13.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:13.151382+0000) 2024-02-20T00:08:14.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:14.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:14.151537+0000) 2024-02-20T00:08:15.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:15.146+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:15.151763+0000) 2024-02-20T00:08:16.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:16.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:16.151970+0000) 2024-02-20T00:08:17.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:17.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:17.152093+0000) 2024-02-20T00:08:18.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:18.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:18.152250+0000) 2024-02-20T00:08:19.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:19.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:19.152387+0000) 2024-02-20T00:08:20.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:20.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:20.152604+0000) 2024-02-20T00:08:21.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:21.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:21.152801+0000) 2024-02-20T00:08:22.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:22.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:22.152995+0000) 2024-02-20T00:08:23.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:23.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:23.153157+0000) 2024-02-20T00:08:24.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:24.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:24.153345+0000) 2024-02-20T00:08:25.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:25.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:25.153559+0000) 2024-02-20T00:08:26.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:26.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:26.153739+0000) 2024-02-20T00:08:27.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:27.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:27.153918+0000) 2024-02-20T00:08:28.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:28.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:28.154181+0000) 2024-02-20T00:08:29.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:29.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:29.154413+0000) 2024-02-20T00:08:30.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:30.150+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:30.154594+0000) 2024-02-20T00:08:31.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:31.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:31.154830+0000) 2024-02-20T00:08:32.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:32.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:32.155001+0000) 2024-02-20T00:08:33.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:33.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:33.155184+0000) 2024-02-20T00:08:34.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:34.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:34.155394+0000) 2024-02-20T00:08:35.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:35.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:35.155557+0000) 2024-02-20T00:08:36.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:36.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:36.155721+0000) 2024-02-20T00:08:37.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:37.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:37.155934+0000) 2024-02-20T00:08:38.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:38.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:38.156115+0000) 2024-02-20T00:08:39.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:39.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:39.156324+0000) 2024-02-20T00:08:40.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:40.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:40.156561+0000) 2024-02-20T00:08:41.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:41.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:41.156783+0000) 2024-02-20T00:08:42.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:42.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:42.157031+0000) 2024-02-20T00:08:43.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:43.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:43.157221+0000) 2024-02-20T00:08:44.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:44.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:44.157430+0000) 2024-02-20T00:08:45.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:45.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:45.157622+0000) 2024-02-20T00:08:46.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:46.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:46.157806+0000) 2024-02-20T00:08:47.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:47.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:47.158027+0000) 2024-02-20T00:08:48.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:48.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:48.158249+0000) 2024-02-20T00:08:49.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:49.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:49.158471+0000) 2024-02-20T00:08:50.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:50.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:50.158742+0000) 2024-02-20T00:08:51.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:51.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:51.158944+0000) 2024-02-20T00:08:52.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:52.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:52.159112+0000) 2024-02-20T00:08:53.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:53.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:53.159301+0000) 2024-02-20T00:08:54.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:54.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:54.159452+0000) 2024-02-20T00:08:55.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:55.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:55.159610+0000) 2024-02-20T00:08:56.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:56.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:56.159845+0000) 2024-02-20T00:08:57.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:57.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:57.160009+0000) 2024-02-20T00:08:58.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:58.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:58.160271+0000) 2024-02-20T00:08:59.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:08:59.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:08:59.160481+0000) 2024-02-20T00:09:00.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:00.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:00.160727+0000) 2024-02-20T00:09:01.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:01.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:01.161007+0000) 2024-02-20T00:09:02.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:02.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:02.161199+0000) 2024-02-20T00:09:03.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:03.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:03.161418+0000) 2024-02-20T00:09:04.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:04.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:04.161625+0000) 2024-02-20T00:09:05.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:05.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:05.161846+0000) 2024-02-20T00:09:06.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:06.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:06.162075+0000) 2024-02-20T00:09:07.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:07.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:07.162312+0000) 2024-02-20T00:09:08.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:08.157+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:08.162535+0000) 2024-02-20T00:09:09.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:09.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:09.162741+0000) 2024-02-20T00:09:10.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:10.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:10.162980+0000) 2024-02-20T00:09:11.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:11.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:11.163274+0000) 2024-02-20T00:09:12.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:12.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:12.163481+0000) 2024-02-20T00:09:13.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:13.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:13.163715+0000) 2024-02-20T00:09:14.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:14.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:14.163958+0000) 2024-02-20T00:09:15.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:15.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:15.164170+0000) 2024-02-20T00:09:16.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:16.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:16.164401+0000) 2024-02-20T00:09:17.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:17.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:17.164599+0000) 2024-02-20T00:09:18.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:18.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:18.164836+0000) 2024-02-20T00:09:19.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:19.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:19.165080+0000) 2024-02-20T00:09:20.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:20.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:20.165324+0000) 2024-02-20T00:09:21.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:21.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:21.165576+0000) 2024-02-20T00:09:22.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:22.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:22.165829+0000) 2024-02-20T00:09:23.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:23.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:23.166072+0000) 2024-02-20T00:09:24.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:24.161+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:24.166290+0000) 2024-02-20T00:09:25.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:25.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:25.166543+0000) 2024-02-20T00:09:26.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:26.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:26.166823+0000) 2024-02-20T00:09:27.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:27.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:27.167032+0000) 2024-02-20T00:09:28.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:28.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:28.167278+0000) 2024-02-20T00:09:29.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:29.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:29.167436+0000) 2024-02-20T00:09:30.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:30.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:30.167674+0000) 2024-02-20T00:09:31.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:31.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:31.167933+0000) 2024-02-20T00:09:32.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:32.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:32.168187+0000) 2024-02-20T00:09:33.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:33.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:33.168421+0000) 2024-02-20T00:09:34.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:34.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:34.168651+0000) 2024-02-20T00:09:35.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:35.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:35.168915+0000) 2024-02-20T00:09:36.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:36.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:36.169090+0000) 2024-02-20T00:09:37.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:37.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:37.169258+0000) 2024-02-20T00:09:38.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:38.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:38.169499+0000) 2024-02-20T00:09:39.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:39.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:39.169657+0000) 2024-02-20T00:09:40.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:40.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:40.169883+0000) 2024-02-20T00:09:41.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:41.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:41.170050+0000) 2024-02-20T00:09:42.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:42.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:42.170250+0000) 2024-02-20T00:09:43.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:43.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:43.170441+0000) 2024-02-20T00:09:44.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:44.165+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:44.170674+0000) 2024-02-20T00:09:45.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:45.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:45.170867+0000) 2024-02-20T00:09:46.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:46.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:46.171041+0000) 2024-02-20T00:09:47.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:47.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:47.171211+0000) 2024-02-20T00:09:48.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:48.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:48.171461+0000) 2024-02-20T00:09:49.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:49.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:49.171749+0000) 2024-02-20T00:09:50.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:50.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:50.171908+0000) 2024-02-20T00:09:51.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:51.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:51.172128+0000) 2024-02-20T00:09:52.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:52.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:52.172326+0000) 2024-02-20T00:09:53.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:53.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:53.172507+0000) 2024-02-20T00:09:54.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:54.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:54.172646+0000) 2024-02-20T00:09:55.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:55.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:55.172876+0000) 2024-02-20T00:09:56.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:56.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:56.173111+0000) 2024-02-20T00:09:57.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:57.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:57.173353+0000) 2024-02-20T00:09:58.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:58.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:58.173565+0000) 2024-02-20T00:09:59.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:09:59.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:09:59.173776+0000) 2024-02-20T00:10:00.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:00.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:00.174068+0000) 2024-02-20T00:10:01.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:01.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:01.174306+0000) 2024-02-20T00:10:02.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:02.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:02.174495+0000) 2024-02-20T00:10:03.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:03.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:03.174663+0000) 2024-02-20T00:10:04.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:04.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:04.174886+0000) 2024-02-20T00:10:05.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:05.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:05.175103+0000) 2024-02-20T00:10:06.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:06.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:06.175266+0000) 2024-02-20T00:10:07.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:07.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:07.175452+0000) 2024-02-20T00:10:08.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:08.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:08.175629+0000) 2024-02-20T00:10:09.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:09.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:09.175869+0000) 2024-02-20T00:10:10.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:10.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:10.176057+0000) 2024-02-20T00:10:11.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:11.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:11.176285+0000) 2024-02-20T00:10:12.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:12.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:12.176510+0000) 2024-02-20T00:10:13.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:13.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:13.176739+0000) 2024-02-20T00:10:14.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:14.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:14.176937+0000) 2024-02-20T00:10:15.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:15.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:15.177164+0000) 2024-02-20T00:10:16.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:16.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:16.177370+0000) 2024-02-20T00:10:17.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:17.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:17.177589+0000) 2024-02-20T00:10:18.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:18.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:18.177721+0000) 2024-02-20T00:10:19.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:19.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:19.177929+0000) 2024-02-20T00:10:20.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:20.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:20.178094+0000) 2024-02-20T00:10:21.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:21.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:21.178268+0000) 2024-02-20T00:10:22.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:22.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:22.178428+0000) 2024-02-20T00:10:23.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:23.172+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:23.178582+0000) 2024-02-20T00:10:24.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:24.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:24.178844+0000) 2024-02-20T00:10:25.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:25.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:25.179040+0000) 2024-02-20T00:10:26.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:26.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:26.179216+0000) 2024-02-20T00:10:27.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:27.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:27.179344+0000) 2024-02-20T00:10:28.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:28.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:28.179562+0000) 2024-02-20T00:10:29.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:29.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:29.179743+0000) 2024-02-20T00:10:30.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:30.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:30.179957+0000) 2024-02-20T00:10:31.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:31.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:31.180170+0000) 2024-02-20T00:10:32.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:32.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:32.180395+0000) 2024-02-20T00:10:33.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:33.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:33.180556+0000) 2024-02-20T00:10:34.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:34.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:34.180806+0000) 2024-02-20T00:10:35.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:35.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:35.181000+0000) 2024-02-20T00:10:36.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:36.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:36.181160+0000) 2024-02-20T00:10:37.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:37.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:37.181325+0000) 2024-02-20T00:10:38.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:38.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:38.181451+0000) 2024-02-20T00:10:39.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:39.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:39.181576+0000) 2024-02-20T00:10:40.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:40.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:40.181739+0000) 2024-02-20T00:10:41.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:41.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:41.181921+0000) 2024-02-20T00:10:42.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:42.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:42.182139+0000) 2024-02-20T00:10:43.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:43.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:43.182309+0000) 2024-02-20T00:10:44.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:44.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:44.182447+0000) 2024-02-20T00:10:45.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:45.176+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:45.182618+0000) 2024-02-20T00:10:46.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:46.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:46.182775+0000) 2024-02-20T00:10:47.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:47.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:47.182943+0000) 2024-02-20T00:10:48.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:48.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:48.183084+0000) 2024-02-20T00:10:49.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:49.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:49.183263+0000) 2024-02-20T00:10:50.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:50.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:50.183468+0000) 2024-02-20T00:10:51.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:51.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:51.183691+0000) 2024-02-20T00:10:52.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:52.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:52.183915+0000) 2024-02-20T00:10:53.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:53.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:53.184195+0000) 2024-02-20T00:10:54.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:54.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:54.184409+0000) 2024-02-20T00:10:55.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:55.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:55.184617+0000) 2024-02-20T00:10:56.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:56.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:56.184805+0000) 2024-02-20T00:10:57.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:57.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:57.185050+0000) 2024-02-20T00:10:58.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:58.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:58.185260+0000) 2024-02-20T00:10:59.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:10:59.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:10:59.185537+0000) 2024-02-20T00:11:00.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:00.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:00.185720+0000) 2024-02-20T00:11:01.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:01.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:01.185964+0000) 2024-02-20T00:11:02.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:02.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:02.186217+0000) 2024-02-20T00:11:03.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:03.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:03.186415+0000) 2024-02-20T00:11:04.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:04.179+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:04.186548+0000) 2024-02-20T00:11:05.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:05.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:05.186734+0000) 2024-02-20T00:11:06.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:06.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:06.186870+0000) 2024-02-20T00:11:07.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:07.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:07.187094+0000) 2024-02-20T00:11:08.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:08.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:08.187244+0000) 2024-02-20T00:11:09.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:09.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:09.187470+0000) 2024-02-20T00:11:10.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:10.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:10.187668+0000) 2024-02-20T00:11:11.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:11.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:11.187831+0000) 2024-02-20T00:11:12.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:12.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:12.188063+0000) 2024-02-20T00:11:13.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:13.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:13.188296+0000) 2024-02-20T00:11:14.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:14.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:14.188534+0000) 2024-02-20T00:11:15.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:15.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:15.188745+0000) 2024-02-20T00:11:16.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:16.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:16.188934+0000) 2024-02-20T00:11:17.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:17.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:17.189096+0000) 2024-02-20T00:11:18.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:18.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:18.189260+0000) 2024-02-20T00:11:19.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:19.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:19.189475+0000) 2024-02-20T00:11:20.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:20.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:20.189674+0000) 2024-02-20T00:11:21.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:21.183+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:21.189928+0000) 2024-02-20T00:11:22.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:22.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:22.190169+0000) 2024-02-20T00:11:23.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:23.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:23.190359+0000) 2024-02-20T00:11:24.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:24.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:24.190535+0000) 2024-02-20T00:11:25.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:25.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:25.190806+0000) 2024-02-20T00:11:26.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:26.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:26.190948+0000) 2024-02-20T00:11:27.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:27.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:27.191128+0000) 2024-02-20T00:11:28.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:28.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:28.191362+0000) 2024-02-20T00:11:29.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:29.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:29.191539+0000) 2024-02-20T00:11:30.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:30.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:30.191791+0000) 2024-02-20T00:11:31.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:31.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:31.192058+0000) 2024-02-20T00:11:32.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:32.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:32.192307+0000) 2024-02-20T00:11:33.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:33.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:33.192559+0000) 2024-02-20T00:11:34.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:34.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:34.192860+0000) 2024-02-20T00:11:35.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:35.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:35.193078+0000) 2024-02-20T00:11:36.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:36.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:36.193312+0000) 2024-02-20T00:11:37.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:37.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:37.193535+0000) 2024-02-20T00:11:38.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:38.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:38.193720+0000) 2024-02-20T00:11:39.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:39.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:39.193980+0000) 2024-02-20T00:11:40.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:40.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:40.194210+0000) 2024-02-20T00:11:41.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:41.187+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:41.194403+0000) 2024-02-20T00:11:42.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:42.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:42.194627+0000) 2024-02-20T00:11:43.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:43.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:43.194811+0000) 2024-02-20T00:11:44.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:44.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:44.195011+0000) 2024-02-20T00:11:45.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:45.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:45.195222+0000) 2024-02-20T00:11:46.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:46.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:46.195459+0000) 2024-02-20T00:11:47.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:47.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:47.195725+0000) 2024-02-20T00:11:48.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:48.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:48.195890+0000) 2024-02-20T00:11:49.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:49.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:49.196123+0000) 2024-02-20T00:11:50.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:50.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:50.196366+0000) 2024-02-20T00:11:51.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:51.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:51.196555+0000) 2024-02-20T00:11:52.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:52.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:52.196727+0000) 2024-02-20T00:11:53.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:53.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:53.196963+0000) 2024-02-20T00:11:54.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:54.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:54.197147+0000) 2024-02-20T00:11:55.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:55.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:55.197365+0000) 2024-02-20T00:11:56.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:56.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:56.197542+0000) 2024-02-20T00:11:57.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:57.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:57.197726+0000) 2024-02-20T00:11:58.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:58.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:58.197955+0000) 2024-02-20T00:11:59.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:11:59.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:11:59.198151+0000) 2024-02-20T00:12:00.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:00.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:00.198318+0000) 2024-02-20T00:12:01.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:01.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:01.198506+0000) 2024-02-20T00:12:02.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:02.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:02.198776+0000) 2024-02-20T00:12:03.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:03.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:03.199045+0000) 2024-02-20T00:12:04.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:04.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:04.199312+0000) 2024-02-20T00:12:05.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:05.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:05.199536+0000) 2024-02-20T00:12:06.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:06.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:06.199722+0000) 2024-02-20T00:12:07.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:07.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:07.199979+0000) 2024-02-20T00:12:08.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:08.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:08.200239+0000) 2024-02-20T00:12:09.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:09.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:09.200506+0000) 2024-02-20T00:12:10.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:10.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:10.200677+0000) 2024-02-20T00:12:11.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:11.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:11.200890+0000) 2024-02-20T00:12:12.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:12.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:12.201124+0000) 2024-02-20T00:12:13.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:13.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:13.201269+0000) 2024-02-20T00:12:14.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:14.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:14.201442+0000) 2024-02-20T00:12:15.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:15.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:15.201662+0000) 2024-02-20T00:12:16.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:16.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:16.201905+0000) 2024-02-20T00:12:17.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:17.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:17.202187+0000) 2024-02-20T00:12:18.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:18.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:18.202438+0000) 2024-02-20T00:12:19.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:19.194+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:19.202663+0000) 2024-02-20T00:12:20.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:20.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:20.202887+0000) 2024-02-20T00:12:21.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:21.198+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:21.203142+0000) 2024-02-20T00:12:22.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:22.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:22.203312+0000) 2024-02-20T00:12:23.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:23.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:23.203486+0000) 2024-02-20T00:12:24.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:24.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:24.203723+0000) 2024-02-20T00:12:25.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:25.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:25.203904+0000) 2024-02-20T00:12:26.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:26.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:26.204083+0000) 2024-02-20T00:12:27.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:27.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:27.204261+0000) 2024-02-20T00:12:28.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:28.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:28.204482+0000) 2024-02-20T00:12:29.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:29.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:29.204699+0000) 2024-02-20T00:12:30.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:30.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:30.204881+0000) 2024-02-20T00:12:31.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:31.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:31.205067+0000) 2024-02-20T00:12:32.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:32.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:32.205291+0000) 2024-02-20T00:12:33.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:33.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:33.205516+0000) 2024-02-20T00:12:34.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:34.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:34.205738+0000) 2024-02-20T00:12:35.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:35.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:35.206013+0000) 2024-02-20T00:12:36.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:36.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:36.206158+0000) 2024-02-20T00:12:37.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:37.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:37.206396+0000) 2024-02-20T00:12:38.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:38.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:38.206679+0000) 2024-02-20T00:12:39.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:39.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:39.206826+0000) 2024-02-20T00:12:40.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:40.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:40.206968+0000) 2024-02-20T00:12:41.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:41.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:41.207120+0000) 2024-02-20T00:12:42.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:42.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:42.207319+0000) 2024-02-20T00:12:43.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:43.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:43.207554+0000) 2024-02-20T00:12:44.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:44.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:44.207774+0000) 2024-02-20T00:12:45.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:45.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:45.208011+0000) 2024-02-20T00:12:46.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:46.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:46.208185+0000) 2024-02-20T00:12:47.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:47.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:47.208425+0000) 2024-02-20T00:12:48.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:48.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:48.208694+0000) 2024-02-20T00:12:49.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:49.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:49.208957+0000) 2024-02-20T00:12:50.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:50.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:50.209184+0000) 2024-02-20T00:12:51.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:51.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:51.209377+0000) 2024-02-20T00:12:52.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:52.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:52.209560+0000) 2024-02-20T00:12:53.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:53.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:53.209727+0000) 2024-02-20T00:12:54.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:54.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:54.209893+0000) 2024-02-20T00:12:55.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:55.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:55.210088+0000) 2024-02-20T00:12:56.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:56.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:56.210261+0000) 2024-02-20T00:12:57.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:57.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:57.210410+0000) 2024-02-20T00:12:58.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:58.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:58.210638+0000) 2024-02-20T00:12:59.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:12:59.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:12:59.210882+0000) 2024-02-20T00:13:00.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:00.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:00.211114+0000) 2024-02-20T00:13:01.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:01.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:01.211327+0000) 2024-02-20T00:13:02.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:02.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:02.211511+0000) 2024-02-20T00:13:03.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:03.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:03.211739+0000) 2024-02-20T00:13:04.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:04.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:04.211974+0000) 2024-02-20T00:13:05.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:05.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:05.212214+0000) 2024-02-20T00:13:06.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:06.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:06.212390+0000) 2024-02-20T00:13:07.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:07.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:07.212619+0000) 2024-02-20T00:13:08.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:08.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:08.212859+0000) 2024-02-20T00:13:09.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:09.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:09.213048+0000) 2024-02-20T00:13:10.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:10.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:10.213186+0000) 2024-02-20T00:13:11.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:11.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:11.213418+0000) 2024-02-20T00:13:12.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:12.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:12.213577+0000) 2024-02-20T00:13:13.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:13.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:13.213716+0000) 2024-02-20T00:13:14.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:14.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:14.213866+0000) 2024-02-20T00:13:15.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:15.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:15.214043+0000) 2024-02-20T00:13:16.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:16.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:16.214264+0000) 2024-02-20T00:13:17.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:17.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:17.214496+0000) 2024-02-20T00:13:18.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:18.209+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:18.214686+0000) 2024-02-20T00:13:19.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:19.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:19.214909+0000) 2024-02-20T00:13:20.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:20.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:20.215073+0000) 2024-02-20T00:13:21.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:21.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:21.215212+0000) 2024-02-20T00:13:22.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:22.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:22.215368+0000) 2024-02-20T00:13:23.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:23.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:23.215544+0000) 2024-02-20T00:13:24.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:24.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:24.215682+0000) 2024-02-20T00:13:25.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:25.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:25.215825+0000) 2024-02-20T00:13:26.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:26.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:26.216060+0000) 2024-02-20T00:13:27.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:27.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:27.216280+0000) 2024-02-20T00:13:28.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:28.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:28.216515+0000) 2024-02-20T00:13:29.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:29.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:29.216706+0000) 2024-02-20T00:13:30.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:30.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:30.216887+0000) 2024-02-20T00:13:31.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:31.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:31.217018+0000) 2024-02-20T00:13:32.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:32.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:32.217242+0000) 2024-02-20T00:13:33.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:33.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:33.217403+0000) 2024-02-20T00:13:34.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:34.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:34.217589+0000) 2024-02-20T00:13:35.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:35.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:35.217820+0000) 2024-02-20T00:13:36.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:36.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:36.218080+0000) 2024-02-20T00:13:37.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:37.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:37.218263+0000) 2024-02-20T00:13:38.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:38.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:38.218455+0000) 2024-02-20T00:13:39.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:39.213+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:39.218628+0000) 2024-02-20T00:13:40.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:40.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:40.218839+0000) 2024-02-20T00:13:41.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:41.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:41.219011+0000) 2024-02-20T00:13:42.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:42.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:42.219193+0000) 2024-02-20T00:13:43.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:43.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:43.219323+0000) 2024-02-20T00:13:44.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:44.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:44.219536+0000) 2024-02-20T00:13:45.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:45.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:45.219703+0000) 2024-02-20T00:13:46.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:46.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:46.219951+0000) 2024-02-20T00:13:47.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:47.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:47.220105+0000) 2024-02-20T00:13:48.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:48.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:48.220335+0000) 2024-02-20T00:13:49.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:49.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:49.220534+0000) 2024-02-20T00:13:50.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:50.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:50.220668+0000) 2024-02-20T00:13:51.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:51.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:51.220816+0000) 2024-02-20T00:13:52.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:52.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:52.220977+0000) 2024-02-20T00:13:53.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:53.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:53.221141+0000) 2024-02-20T00:13:54.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:54.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:54.221286+0000) 2024-02-20T00:13:55.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:55.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:55.221428+0000) 2024-02-20T00:13:56.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:56.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:56.221629+0000) 2024-02-20T00:13:57.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:57.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:57.221803+0000) 2024-02-20T00:13:58.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:58.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:58.222067+0000) 2024-02-20T00:13:59.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:13:59.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:13:59.222301+0000) 2024-02-20T00:14:00.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:00.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:00.222496+0000) 2024-02-20T00:14:01.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:01.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:01.222776+0000) 2024-02-20T00:14:02.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:02.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:02.222999+0000) 2024-02-20T00:14:03.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:03.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:03.223171+0000) 2024-02-20T00:14:04.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:04.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:04.223363+0000) 2024-02-20T00:14:05.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:05.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:05.223566+0000) 2024-02-20T00:14:06.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:06.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:06.223734+0000) 2024-02-20T00:14:07.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:07.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:07.223970+0000) 2024-02-20T00:14:08.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:08.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:08.224127+0000) 2024-02-20T00:14:09.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:09.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:09.224316+0000) 2024-02-20T00:14:10.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:10.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:10.224542+0000) 2024-02-20T00:14:11.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:11.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:11.224787+0000) 2024-02-20T00:14:12.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:12.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:12.224993+0000) 2024-02-20T00:14:13.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:13.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:13.225194+0000) 2024-02-20T00:14:14.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:14.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:14.225418+0000) 2024-02-20T00:14:15.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:15.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:15.225670+0000) 2024-02-20T00:14:16.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:16.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:16.225899+0000) 2024-02-20T00:14:17.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:17.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:17.226108+0000) 2024-02-20T00:14:18.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:18.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:18.226259+0000) 2024-02-20T00:14:19.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:19.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:19.226387+0000) 2024-02-20T00:14:20.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:20.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:20.226525+0000) 2024-02-20T00:14:21.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:21.220+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:21.226667+0000) 2024-02-20T00:14:22.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:22.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:22.226907+0000) 2024-02-20T00:14:23.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:23.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:23.227089+0000) 2024-02-20T00:14:24.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:24.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:24.227226+0000) 2024-02-20T00:14:25.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:25.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:25.227450+0000) 2024-02-20T00:14:26.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:26.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:26.227684+0000) 2024-02-20T00:14:27.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:27.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:27.227859+0000) 2024-02-20T00:14:28.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:28.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:28.228101+0000) 2024-02-20T00:14:29.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:29.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:29.228309+0000) 2024-02-20T00:14:30.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:30.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:30.228469+0000) 2024-02-20T00:14:31.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:31.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:31.228608+0000) 2024-02-20T00:14:32.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:32.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:32.228780+0000) 2024-02-20T00:14:33.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:33.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:33.228996+0000) 2024-02-20T00:14:34.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:34.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:34.229159+0000) 2024-02-20T00:14:35.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:35.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:35.229317+0000) 2024-02-20T00:14:36.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:36.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:36.229446+0000) 2024-02-20T00:14:37.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:37.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:37.229613+0000) 2024-02-20T00:14:38.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:38.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:38.229797+0000) 2024-02-20T00:14:39.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:39.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:39.230003+0000) 2024-02-20T00:14:40.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:40.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:40.230206+0000) 2024-02-20T00:14:41.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:41.224+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:41.230479+0000) 2024-02-20T00:14:42.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:42.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:42.230660+0000) 2024-02-20T00:14:43.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:43.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:43.230854+0000) 2024-02-20T00:14:44.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:44.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:44.231035+0000) 2024-02-20T00:14:45.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:45.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:45.231240+0000) 2024-02-20T00:14:46.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:46.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:46.231388+0000) 2024-02-20T00:14:47.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:47.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:47.231556+0000) 2024-02-20T00:14:48.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:48.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:48.231766+0000) 2024-02-20T00:14:49.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:49.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:49.231980+0000) 2024-02-20T00:14:50.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:50.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:50.232107+0000) 2024-02-20T00:14:51.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:51.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:51.232338+0000) 2024-02-20T00:14:52.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:52.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:52.232523+0000) 2024-02-20T00:14:53.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:53.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:53.232668+0000) 2024-02-20T00:14:54.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:54.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:54.232931+0000) 2024-02-20T00:14:55.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:55.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:55.233117+0000) 2024-02-20T00:14:56.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:56.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:56.233348+0000) 2024-02-20T00:14:57.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:57.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:57.233578+0000) 2024-02-20T00:14:58.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:58.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:58.233761+0000) 2024-02-20T00:14:59.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:14:59.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:14:59.234032+0000) 2024-02-20T00:15:00.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:00.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:00.234291+0000) 2024-02-20T00:15:01.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:01.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:01.234498+0000) 2024-02-20T00:15:02.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:02.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:02.234674+0000) 2024-02-20T00:15:03.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:03.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:03.234882+0000) 2024-02-20T00:15:04.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:04.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:04.235033+0000) 2024-02-20T00:15:05.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:05.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:05.235225+0000) 2024-02-20T00:15:06.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:06.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:06.235403+0000) 2024-02-20T00:15:07.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:07.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:07.235583+0000) 2024-02-20T00:15:08.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:08.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:08.235798+0000) 2024-02-20T00:15:09.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:09.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:09.236009+0000) 2024-02-20T00:15:10.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:10.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:10.236176+0000) 2024-02-20T00:15:11.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:11.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:11.236378+0000) 2024-02-20T00:15:12.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:12.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:12.236574+0000) 2024-02-20T00:15:13.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:13.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:13.236804+0000) 2024-02-20T00:15:14.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:14.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:14.236956+0000) 2024-02-20T00:15:15.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:15.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:15.237119+0000) 2024-02-20T00:15:16.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:16.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:16.237361+0000) 2024-02-20T00:15:17.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:17.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:17.237566+0000) 2024-02-20T00:15:18.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:18.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:18.237760+0000) 2024-02-20T00:15:19.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:19.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:19.238003+0000) 2024-02-20T00:15:20.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:20.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:20.238248+0000) 2024-02-20T00:15:21.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:21.231+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:21.238448+0000) 2024-02-20T00:15:22.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:22.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:22.238679+0000) 2024-02-20T00:15:23.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:23.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:23.238932+0000) 2024-02-20T00:15:24.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:24.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:24.239222+0000) 2024-02-20T00:15:25.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:25.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:25.239423+0000) 2024-02-20T00:15:26.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:26.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:26.239592+0000) 2024-02-20T00:15:27.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:27.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:27.239763+0000) 2024-02-20T00:15:28.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:28.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:28.239969+0000) 2024-02-20T00:15:29.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:29.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:29.240109+0000) 2024-02-20T00:15:30.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:30.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:30.240374+0000) 2024-02-20T00:15:31.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:31.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:31.240598+0000) 2024-02-20T00:15:32.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:32.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:32.240745+0000) 2024-02-20T00:15:33.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:33.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:33.240945+0000) 2024-02-20T00:15:34.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:34.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:34.241208+0000) 2024-02-20T00:15:35.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:35.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:35.241476+0000) 2024-02-20T00:15:36.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:36.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:36.241632+0000) 2024-02-20T00:15:37.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:37.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:37.241849+0000) 2024-02-20T00:15:38.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:38.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:38.242008+0000) 2024-02-20T00:15:39.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:39.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:39.242166+0000) 2024-02-20T00:15:40.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:40.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:40.242307+0000) 2024-02-20T00:15:41.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:41.235+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:41.242573+0000) 2024-02-20T00:15:42.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:42.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:42.242792+0000) 2024-02-20T00:15:43.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:43.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:43.242934+0000) 2024-02-20T00:15:44.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:44.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:44.243110+0000) 2024-02-20T00:15:45.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:45.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:45.243300+0000) 2024-02-20T00:15:46.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:46.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:46.243489+0000) 2024-02-20T00:15:47.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:47.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:47.243662+0000) 2024-02-20T00:15:48.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:48.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:48.243838+0000) 2024-02-20T00:15:49.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:49.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:49.244018+0000) 2024-02-20T00:15:50.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:50.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:50.244248+0000) 2024-02-20T00:15:51.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:51.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:51.244415+0000) 2024-02-20T00:15:52.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:52.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:52.244689+0000) 2024-02-20T00:15:53.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:53.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:53.244901+0000) 2024-02-20T00:15:54.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:54.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:54.245159+0000) 2024-02-20T00:15:55.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:55.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:55.245387+0000) 2024-02-20T00:15:56.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:56.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:56.245606+0000) 2024-02-20T00:15:57.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:57.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:57.245810+0000) 2024-02-20T00:15:58.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:58.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:58.245987+0000) 2024-02-20T00:15:59.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:15:59.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:15:59.246148+0000) 2024-02-20T00:16:00.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:00.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:00.246268+0000) 2024-02-20T00:16:01.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:01.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:01.246412+0000) 2024-02-20T00:16:02.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:02.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:02.246572+0000) 2024-02-20T00:16:03.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:03.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:03.246798+0000) 2024-02-20T00:16:04.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:04.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:04.246956+0000) 2024-02-20T00:16:05.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:05.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:05.247100+0000) 2024-02-20T00:16:06.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:06.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:06.247323+0000) 2024-02-20T00:16:07.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:07.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:07.247580+0000) 2024-02-20T00:16:08.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:08.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:08.247816+0000) 2024-02-20T00:16:09.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:09.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:09.248066+0000) 2024-02-20T00:16:10.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:10.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:10.248274+0000) 2024-02-20T00:16:11.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:11.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:11.248481+0000) 2024-02-20T00:16:12.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:12.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:12.248730+0000) 2024-02-20T00:16:13.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:13.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:13.248904+0000) 2024-02-20T00:16:14.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:14.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:14.249151+0000) 2024-02-20T00:16:15.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:15.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:15.249415+0000) 2024-02-20T00:16:16.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:16.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:16.249620+0000) 2024-02-20T00:16:17.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:17.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:17.249861+0000) 2024-02-20T00:16:18.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:18.242+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:18.250008+0000) 2024-02-20T00:16:19.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:19.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:19.250170+0000) 2024-02-20T00:16:20.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:20.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:20.250321+0000) 2024-02-20T00:16:21.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:21.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:21.250487+0000) 2024-02-20T00:16:22.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:22.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:22.250756+0000) 2024-02-20T00:16:23.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:23.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:23.250948+0000) 2024-02-20T00:16:24.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:24.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:24.251153+0000) 2024-02-20T00:16:25.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:25.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:25.251350+0000) 2024-02-20T00:16:26.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:26.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:26.251544+0000) 2024-02-20T00:16:27.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:27.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:27.251740+0000) 2024-02-20T00:16:28.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:28.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:28.251944+0000) 2024-02-20T00:16:29.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:29.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:29.252228+0000) 2024-02-20T00:16:30.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:30.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:30.252465+0000) 2024-02-20T00:16:31.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:31.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:31.252649+0000) 2024-02-20T00:16:32.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:32.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:32.252898+0000) 2024-02-20T00:16:33.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:33.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:33.253157+0000) 2024-02-20T00:16:34.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:34.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:34.253362+0000) 2024-02-20T00:16:35.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:35.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:35.253572+0000) 2024-02-20T00:16:36.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:36.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:36.253740+0000) 2024-02-20T00:16:37.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:37.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:37.253960+0000) 2024-02-20T00:16:38.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:38.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:38.254179+0000) 2024-02-20T00:16:39.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:39.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:39.254378+0000) 2024-02-20T00:16:40.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:40.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:40.254601+0000) 2024-02-20T00:16:41.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:41.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:41.254849+0000) 2024-02-20T00:16:42.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:42.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:42.255083+0000) 2024-02-20T00:16:43.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:43.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:43.255316+0000) 2024-02-20T00:16:44.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:44.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:44.255558+0000) 2024-02-20T00:16:45.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:45.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:45.255737+0000) 2024-02-20T00:16:46.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:46.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:46.255904+0000) 2024-02-20T00:16:47.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:47.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:47.256144+0000) 2024-02-20T00:16:48.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:48.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:48.256309+0000) 2024-02-20T00:16:49.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:49.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:49.256495+0000) 2024-02-20T00:16:50.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:50.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:50.256742+0000) 2024-02-20T00:16:51.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:51.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:51.256998+0000) 2024-02-20T00:16:52.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:52.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:52.257243+0000) 2024-02-20T00:16:53.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:53.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:53.257470+0000) 2024-02-20T00:16:54.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:54.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:54.257705+0000) 2024-02-20T00:16:55.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:55.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:55.257897+0000) 2024-02-20T00:16:56.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:56.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:56.258135+0000) 2024-02-20T00:16:57.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:57.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:57.258364+0000) 2024-02-20T00:16:58.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:58.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:58.258597+0000) 2024-02-20T00:16:59.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:16:59.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:16:59.258827+0000) 2024-02-20T00:17:00.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:00.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:00.259101+0000) 2024-02-20T00:17:01.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:01.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:01.259280+0000) 2024-02-20T00:17:02.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:02.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:02.259517+0000) 2024-02-20T00:17:03.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:03.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:03.259771+0000) 2024-02-20T00:17:04.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:04.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:04.259903+0000) 2024-02-20T00:17:05.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:05.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:05.260134+0000) 2024-02-20T00:17:06.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:06.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:06.260389+0000) 2024-02-20T00:17:07.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:07.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:07.260634+0000) 2024-02-20T00:17:08.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:08.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:08.260836+0000) 2024-02-20T00:17:09.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:09.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:09.261081+0000) 2024-02-20T00:17:10.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:10.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:10.261347+0000) 2024-02-20T00:17:11.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:11.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:11.261609+0000) 2024-02-20T00:17:12.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:12.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:12.261842+0000) 2024-02-20T00:17:13.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:13.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:13.262051+0000) 2024-02-20T00:17:14.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:14.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:14.262283+0000) 2024-02-20T00:17:15.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:15.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:15.262546+0000) 2024-02-20T00:17:16.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:16.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:16.262703+0000) 2024-02-20T00:17:17.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:17.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:17.262868+0000) 2024-02-20T00:17:18.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:18.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:18.263093+0000) 2024-02-20T00:17:19.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:19.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:19.263338+0000) 2024-02-20T00:17:20.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:20.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:20.263517+0000) 2024-02-20T00:17:21.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:21.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:21.263773+0000) 2024-02-20T00:17:22.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:22.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:22.263954+0000) 2024-02-20T00:17:23.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:23.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:23.264165+0000) 2024-02-20T00:17:24.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:24.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:24.264396+0000) 2024-02-20T00:17:25.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:25.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:25.264691+0000) 2024-02-20T00:17:26.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:26.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:26.264928+0000) 2024-02-20T00:17:27.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:27.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:27.265113+0000) 2024-02-20T00:17:28.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:28.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:28.265344+0000) 2024-02-20T00:17:29.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:29.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:29.265636+0000) 2024-02-20T00:17:30.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:30.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:30.265839+0000) 2024-02-20T00:17:31.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:31.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:31.266111+0000) 2024-02-20T00:17:32.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:32.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:32.266339+0000) 2024-02-20T00:17:33.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:33.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:33.266671+0000) 2024-02-20T00:17:34.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:34.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:34.266812+0000) 2024-02-20T00:17:35.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:35.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:35.266972+0000) 2024-02-20T00:17:36.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:36.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:36.267125+0000) 2024-02-20T00:17:37.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:37.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:37.267276+0000) 2024-02-20T00:17:38.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:38.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:38.267509+0000) 2024-02-20T00:17:39.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:39.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:39.267799+0000) 2024-02-20T00:17:40.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:40.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:40.268070+0000) 2024-02-20T00:17:41.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:41.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:41.268364+0000) 2024-02-20T00:17:42.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:42.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:42.268612+0000) 2024-02-20T00:17:43.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:43.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:43.268831+0000) 2024-02-20T00:17:44.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:44.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:44.269086+0000) 2024-02-20T00:17:45.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:45.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:45.269380+0000) 2024-02-20T00:17:46.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:46.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:46.269593+0000) 2024-02-20T00:17:47.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:47.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:47.269890+0000) 2024-02-20T00:17:48.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:48.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:48.270089+0000) 2024-02-20T00:17:49.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:49.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:49.270338+0000) 2024-02-20T00:17:50.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:50.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:50.270582+0000) 2024-02-20T00:17:51.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:51.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:51.270796+0000) 2024-02-20T00:17:52.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:52.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:52.270993+0000) 2024-02-20T00:17:53.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:53.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:53.271161+0000) 2024-02-20T00:17:54.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:54.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:54.271410+0000) 2024-02-20T00:17:55.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:55.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:55.271624+0000) 2024-02-20T00:17:56.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:56.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:56.271857+0000) 2024-02-20T00:17:57.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:57.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:57.272107+0000) 2024-02-20T00:17:58.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:58.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:58.272339+0000) 2024-02-20T00:17:59.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:17:59.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:17:59.272629+0000) 2024-02-20T00:18:00.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:00.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:00.272897+0000) 2024-02-20T00:18:01.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:01.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:01.273193+0000) 2024-02-20T00:18:02.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:02.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:02.273422+0000) 2024-02-20T00:18:03.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:03.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:03.273663+0000) 2024-02-20T00:18:04.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:04.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:04.273897+0000) 2024-02-20T00:18:05.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:05.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:05.274152+0000) 2024-02-20T00:18:06.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:06.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:06.274344+0000) 2024-02-20T00:18:07.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:07.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:07.274566+0000) 2024-02-20T00:18:08.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:08.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:08.274746+0000) 2024-02-20T00:18:09.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:09.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:09.274905+0000) 2024-02-20T00:18:10.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:10.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:10.275044+0000) 2024-02-20T00:18:11.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:11.273+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:11.275180+0000) 2024-02-20T00:18:12.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:12.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:12.275391+0000) 2024-02-20T00:18:13.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:13.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:13.275688+0000) 2024-02-20T00:18:14.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:14.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:14.275905+0000) 2024-02-20T00:18:15.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:15.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:15.276074+0000) 2024-02-20T00:18:16.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:16.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:16.276282+0000) 2024-02-20T00:18:17.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:17.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:17.276509+0000) 2024-02-20T00:18:18.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:18.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:18.276736+0000) 2024-02-20T00:18:19.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:19.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:19.277036+0000) 2024-02-20T00:18:20.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:20.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:20.277254+0000) 2024-02-20T00:18:21.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:21.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:21.277433+0000) 2024-02-20T00:18:22.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:22.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:22.277686+0000) 2024-02-20T00:18:23.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:23.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:23.278007+0000) 2024-02-20T00:18:24.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:24.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:24.278177+0000) 2024-02-20T00:18:25.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:25.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:25.278323+0000) 2024-02-20T00:18:26.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:26.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:26.278460+0000) 2024-02-20T00:18:27.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:27.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:27.278647+0000) 2024-02-20T00:18:28.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:28.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:28.278790+0000) 2024-02-20T00:18:29.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:29.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:29.278935+0000) 2024-02-20T00:18:30.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:30.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:30.279067+0000) 2024-02-20T00:18:31.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:31.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:31.279205+0000) 2024-02-20T00:18:32.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:32.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:32.279348+0000) 2024-02-20T00:18:33.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:33.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:33.279519+0000) 2024-02-20T00:18:34.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:34.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:34.279687+0000) 2024-02-20T00:18:35.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:35.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:35.279851+0000) 2024-02-20T00:18:36.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:36.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:36.280002+0000) 2024-02-20T00:18:37.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:37.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:37.280234+0000) 2024-02-20T00:18:38.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:38.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:38.280379+0000) 2024-02-20T00:18:39.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:39.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:39.280606+0000) 2024-02-20T00:18:40.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:40.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:40.280760+0000) 2024-02-20T00:18:41.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:41.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:41.280947+0000) 2024-02-20T00:18:42.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:42.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:42.281184+0000) 2024-02-20T00:18:43.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:43.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:43.281446+0000) 2024-02-20T00:18:44.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:44.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:44.281584+0000) 2024-02-20T00:18:45.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:45.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:45.281802+0000) 2024-02-20T00:18:46.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:46.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:46.281951+0000) 2024-02-20T00:18:47.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:47.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:47.282105+0000) 2024-02-20T00:18:48.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:48.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:48.282339+0000) 2024-02-20T00:18:49.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:49.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:49.282557+0000) 2024-02-20T00:18:50.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:50.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:50.282854+0000) 2024-02-20T00:18:51.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:51.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:51.283136+0000) 2024-02-20T00:18:52.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:52.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:52.283366+0000) 2024-02-20T00:18:53.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:53.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:53.283588+0000) 2024-02-20T00:18:54.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:54.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:54.283756+0000) 2024-02-20T00:18:55.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:55.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:55.283892+0000) 2024-02-20T00:18:56.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:56.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:56.284106+0000) 2024-02-20T00:18:57.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:57.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:57.284329+0000) 2024-02-20T00:18:58.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:58.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:58.284537+0000) 2024-02-20T00:18:59.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:18:59.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:18:59.284807+0000) 2024-02-20T00:19:00.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:00.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:00.285045+0000) 2024-02-20T00:19:01.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:01.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:01.285293+0000) 2024-02-20T00:19:02.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:02.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:02.285509+0000) 2024-02-20T00:19:03.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:03.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:03.285672+0000) 2024-02-20T00:19:04.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:04.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:04.285842+0000) 2024-02-20T00:19:05.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:05.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:05.286005+0000) 2024-02-20T00:19:06.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:06.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:06.286266+0000) 2024-02-20T00:19:07.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:07.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:07.286557+0000) 2024-02-20T00:19:08.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:08.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:08.286701+0000) 2024-02-20T00:19:09.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:09.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:09.287010+0000) 2024-02-20T00:19:10.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:10.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:10.287232+0000) 2024-02-20T00:19:11.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:11.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:11.287445+0000) 2024-02-20T00:19:12.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:12.284+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:12.287679+0000) 2024-02-20T00:19:13.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:13.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:13.287981+0000) 2024-02-20T00:19:14.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:14.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:14.288189+0000) 2024-02-20T00:19:15.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:15.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:15.288474+0000) 2024-02-20T00:19:16.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:16.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:16.288690+0000) 2024-02-20T00:19:17.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:17.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:17.288941+0000) 2024-02-20T00:19:18.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:18.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:18.289168+0000) 2024-02-20T00:19:19.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:19.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:19.289399+0000) 2024-02-20T00:19:20.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:20.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:20.289589+0000) 2024-02-20T00:19:21.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:21.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:21.289843+0000) 2024-02-20T00:19:22.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:22.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:22.290085+0000) 2024-02-20T00:19:23.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:23.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:23.290384+0000) 2024-02-20T00:19:24.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:24.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:24.290613+0000) 2024-02-20T00:19:25.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:25.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:25.290874+0000) 2024-02-20T00:19:26.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:26.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:26.291028+0000) 2024-02-20T00:19:27.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:27.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:27.291183+0000) 2024-02-20T00:19:28.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:28.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:28.291375+0000) 2024-02-20T00:19:29.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:29.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:29.291543+0000) 2024-02-20T00:19:30.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:30.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:30.291708+0000) 2024-02-20T00:19:31.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:31.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:31.291966+0000) 2024-02-20T00:19:32.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:32.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:32.292145+0000) 2024-02-20T00:19:33.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:33.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:33.292361+0000) 2024-02-20T00:19:34.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:34.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:34.292620+0000) 2024-02-20T00:19:35.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:35.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:35.292819+0000) 2024-02-20T00:19:36.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:36.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:36.293069+0000) 2024-02-20T00:19:37.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:37.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:37.293297+0000) 2024-02-20T00:19:38.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:38.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:38.293512+0000) 2024-02-20T00:19:39.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:39.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:39.293720+0000) 2024-02-20T00:19:40.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:40.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:40.293913+0000) 2024-02-20T00:19:41.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:41.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:41.294145+0000) 2024-02-20T00:19:42.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:42.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:42.294351+0000) 2024-02-20T00:19:43.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:43.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:43.294612+0000) 2024-02-20T00:19:44.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:44.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:44.294827+0000) 2024-02-20T00:19:45.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:45.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:45.295084+0000) 2024-02-20T00:19:46.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:46.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:46.295299+0000) 2024-02-20T00:19:47.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:47.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:47.295566+0000) 2024-02-20T00:19:48.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:48.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:48.295807+0000) 2024-02-20T00:19:49.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:49.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:49.296086+0000) 2024-02-20T00:19:50.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:50.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:50.296321+0000) 2024-02-20T00:19:51.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:51.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:51.296598+0000) 2024-02-20T00:19:52.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:52.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:52.296862+0000) 2024-02-20T00:19:53.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:53.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:53.297135+0000) 2024-02-20T00:19:54.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:54.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:54.297327+0000) 2024-02-20T00:19:55.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:55.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:55.297517+0000) 2024-02-20T00:19:56.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:56.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:56.297698+0000) 2024-02-20T00:19:57.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:57.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:57.297927+0000) 2024-02-20T00:19:58.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:58.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:58.298135+0000) 2024-02-20T00:19:59.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:19:59.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:19:59.298322+0000) 2024-02-20T00:20:00.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:00.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:00.298551+0000) 2024-02-20T00:20:01.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:01.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:01.298857+0000) 2024-02-20T00:20:02.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:02.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:02.299085+0000) 2024-02-20T00:20:03.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:03.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:03.299381+0000) 2024-02-20T00:20:04.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:04.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:04.299592+0000) 2024-02-20T00:20:05.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:05.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:05.299788+0000) 2024-02-20T00:20:06.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:06.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:06.300019+0000) 2024-02-20T00:20:07.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:07.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:07.300248+0000) 2024-02-20T00:20:08.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:08.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:08.300473+0000) 2024-02-20T00:20:09.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:09.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:09.300747+0000) 2024-02-20T00:20:10.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:10.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:10.300988+0000) 2024-02-20T00:20:11.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:11.299+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:11.301245+0000) 2024-02-20T00:20:12.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:12.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:12.301427+0000) 2024-02-20T00:20:13.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:13.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:13.301623+0000) 2024-02-20T00:20:14.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:14.295+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:14.301853+0000) 2024-02-20T00:20:15.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:15.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:15.302106+0000) 2024-02-20T00:20:16.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:16.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:16.302268+0000) 2024-02-20T00:20:17.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:17.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:17.302425+0000) 2024-02-20T00:20:18.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:18.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:18.302665+0000) 2024-02-20T00:20:19.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:19.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:19.302982+0000) 2024-02-20T00:20:20.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:20.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:20.303198+0000) 2024-02-20T00:20:21.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:21.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:21.303519+0000) 2024-02-20T00:20:22.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:22.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:22.303735+0000) 2024-02-20T00:20:23.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:23.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:23.304039+0000) 2024-02-20T00:20:24.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:24.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:24.304298+0000) 2024-02-20T00:20:25.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:25.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:25.304504+0000) 2024-02-20T00:20:26.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:26.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:26.304751+0000) 2024-02-20T00:20:27.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:27.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:27.304964+0000) 2024-02-20T00:20:28.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:28.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:28.305134+0000) 2024-02-20T00:20:29.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:29.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:29.305391+0000) 2024-02-20T00:20:30.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:30.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:30.305678+0000) 2024-02-20T00:20:31.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:31.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:31.305972+0000) 2024-02-20T00:20:32.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:32.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:32.306221+0000) 2024-02-20T00:20:33.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:33.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:33.306390+0000) 2024-02-20T00:20:34.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:34.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:34.306599+0000) 2024-02-20T00:20:35.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:35.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:35.306883+0000) 2024-02-20T00:20:36.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:36.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:36.307098+0000) 2024-02-20T00:20:37.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:37.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:37.307258+0000) 2024-02-20T00:20:38.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:38.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:38.307454+0000) 2024-02-20T00:20:39.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:39.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:39.307681+0000) 2024-02-20T00:20:40.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:40.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:40.307914+0000) 2024-02-20T00:20:41.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:41.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:41.308136+0000) 2024-02-20T00:20:42.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:42.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:42.308394+0000) 2024-02-20T00:20:43.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:43.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:43.308614+0000) 2024-02-20T00:20:44.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:44.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:44.308828+0000) 2024-02-20T00:20:45.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:45.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:45.308991+0000) 2024-02-20T00:20:46.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:46.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:46.309218+0000) 2024-02-20T00:20:47.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:47.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:47.309491+0000) 2024-02-20T00:20:48.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:48.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:48.309674+0000) 2024-02-20T00:20:49.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:49.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:49.309839+0000) 2024-02-20T00:20:50.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:50.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:50.310017+0000) 2024-02-20T00:20:51.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:51.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:51.310247+0000) 2024-02-20T00:20:52.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:52.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:52.310502+0000) 2024-02-20T00:20:53.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:53.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:53.310683+0000) 2024-02-20T00:20:54.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:54.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:54.310884+0000) 2024-02-20T00:20:55.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:55.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:55.311076+0000) 2024-02-20T00:20:56.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:56.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:56.311311+0000) 2024-02-20T00:20:57.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:57.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:57.311536+0000) 2024-02-20T00:20:58.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:58.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:58.311787+0000) 2024-02-20T00:20:59.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:20:59.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:20:59.312001+0000) 2024-02-20T00:21:00.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:00.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:00.312152+0000) 2024-02-20T00:21:01.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:01.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:01.312331+0000) 2024-02-20T00:21:02.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:02.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:02.312491+0000) 2024-02-20T00:21:03.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:03.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:03.312624+0000) 2024-02-20T00:21:04.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:04.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:04.312753+0000) 2024-02-20T00:21:05.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:05.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:05.312875+0000) 2024-02-20T00:21:06.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:06.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:06.313008+0000) 2024-02-20T00:21:07.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:07.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:07.313128+0000) 2024-02-20T00:21:08.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:08.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:08.313323+0000) 2024-02-20T00:21:09.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:09.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:09.313447+0000) 2024-02-20T00:21:10.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:10.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:10.313580+0000) 2024-02-20T00:21:11.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:11.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:11.313720+0000) 2024-02-20T00:21:12.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:12.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:12.313849+0000) 2024-02-20T00:21:13.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:13.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:13.313974+0000) 2024-02-20T00:21:14.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:14.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:14.314125+0000) 2024-02-20T00:21:15.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:15.310+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:15.314252+0000) 2024-02-20T00:21:16.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:16.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:16.314375+0000) 2024-02-20T00:21:17.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:17.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:17.314569+0000) 2024-02-20T00:21:18.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:18.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:18.314743+0000) 2024-02-20T00:21:19.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:19.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:19.314902+0000) 2024-02-20T00:21:20.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:20.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:20.315053+0000) 2024-02-20T00:21:21.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:21.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:21.315242+0000) 2024-02-20T00:21:22.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:22.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:22.315436+0000) 2024-02-20T00:21:23.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:23.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:23.315644+0000) 2024-02-20T00:21:24.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:24.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:24.315767+0000) 2024-02-20T00:21:25.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:25.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:25.315942+0000) 2024-02-20T00:21:26.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:26.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:26.316184+0000) 2024-02-20T00:21:27.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:27.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:27.316448+0000) 2024-02-20T00:21:28.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:28.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:28.316635+0000) 2024-02-20T00:21:29.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:29.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:29.316876+0000) 2024-02-20T00:21:30.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:30.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:30.317037+0000) 2024-02-20T00:21:31.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:31.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:31.317238+0000) 2024-02-20T00:21:32.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:32.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:32.317394+0000) 2024-02-20T00:21:33.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:33.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:33.317554+0000) 2024-02-20T00:21:34.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:34.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:34.317772+0000) 2024-02-20T00:21:35.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:35.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:35.317950+0000) 2024-02-20T00:21:36.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:36.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:36.318170+0000) 2024-02-20T00:21:37.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:37.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:37.318415+0000) 2024-02-20T00:21:38.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:38.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:38.318572+0000) 2024-02-20T00:21:39.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:39.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:39.318826+0000) 2024-02-20T00:21:40.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:40.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:40.319106+0000) 2024-02-20T00:21:41.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:41.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:41.319308+0000) 2024-02-20T00:21:42.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:42.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:42.319527+0000) 2024-02-20T00:21:43.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:43.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:43.319719+0000) 2024-02-20T00:21:44.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:44.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:44.320022+0000) 2024-02-20T00:21:45.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:45.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:45.320239+0000) 2024-02-20T00:21:46.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:46.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:46.320399+0000) 2024-02-20T00:21:47.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:47.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:47.320585+0000) 2024-02-20T00:21:48.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:48.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:48.320735+0000) 2024-02-20T00:21:49.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:49.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:49.320958+0000) 2024-02-20T00:21:50.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:50.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:50.321106+0000) 2024-02-20T00:21:51.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:51.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:51.321322+0000) 2024-02-20T00:21:52.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:52.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:52.321463+0000) 2024-02-20T00:21:53.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:53.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:53.321592+0000) 2024-02-20T00:21:54.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:54.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:54.321722+0000) 2024-02-20T00:21:55.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:55.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:55.321882+0000) 2024-02-20T00:21:56.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:56.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:56.322030+0000) 2024-02-20T00:21:57.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:57.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:57.322198+0000) 2024-02-20T00:21:58.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:58.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:58.322362+0000) 2024-02-20T00:21:59.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:21:59.317+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:21:59.322552+0000) 2024-02-20T00:22:00.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:00.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:00.322759+0000) 2024-02-20T00:22:01.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:01.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:01.322988+0000) 2024-02-20T00:22:02.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:02.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:02.323212+0000) 2024-02-20T00:22:03.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:03.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:03.323416+0000) 2024-02-20T00:22:04.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:04.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:04.323660+0000) 2024-02-20T00:22:05.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:05.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:05.323793+0000) 2024-02-20T00:22:06.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:06.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:06.323927+0000) 2024-02-20T00:22:07.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:07.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:07.324166+0000) 2024-02-20T00:22:08.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:08.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:08.324335+0000) 2024-02-20T00:22:09.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:09.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:09.324537+0000) 2024-02-20T00:22:10.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:10.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:10.324781+0000) 2024-02-20T00:22:11.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:11.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:11.324954+0000) 2024-02-20T00:22:12.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:12.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:12.325132+0000) 2024-02-20T00:22:13.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:13.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:13.325340+0000) 2024-02-20T00:22:14.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:14.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:14.325560+0000) 2024-02-20T00:22:15.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:15.321+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:15.325817+0000) 2024-02-20T00:22:16.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:16.325+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:16.326092+0000) 2024-02-20T00:22:17.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:17.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:17.326267+0000) 2024-02-20T00:22:18.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:18.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:18.326513+0000) 2024-02-20T00:22:19.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:19.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:19.326791+0000) 2024-02-20T00:22:20.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:20.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:20.327002+0000) 2024-02-20T00:22:21.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:21.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:21.327177+0000) 2024-02-20T00:22:22.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:22.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:22.327380+0000) 2024-02-20T00:22:23.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:23.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:23.327582+0000) 2024-02-20T00:22:24.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:24.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:24.327759+0000) 2024-02-20T00:22:25.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:25.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:25.327950+0000) 2024-02-20T00:22:26.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:26.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:26.328133+0000) 2024-02-20T00:22:27.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:27.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:27.328303+0000) 2024-02-20T00:22:28.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:28.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:28.328554+0000) 2024-02-20T00:22:29.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:29.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:29.328691+0000) 2024-02-20T00:22:30.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:30.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:30.328970+0000) 2024-02-20T00:22:31.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:31.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:31.329207+0000) 2024-02-20T00:22:32.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:32.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:32.329459+0000) 2024-02-20T00:22:33.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:33.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:33.329632+0000) 2024-02-20T00:22:34.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:34.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:34.329891+0000) 2024-02-20T00:22:35.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:35.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:35.330108+0000) 2024-02-20T00:22:36.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:36.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:36.330304+0000) 2024-02-20T00:22:37.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:37.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:37.330487+0000) 2024-02-20T00:22:38.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:38.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:38.330653+0000) 2024-02-20T00:22:39.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:39.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:39.330869+0000) 2024-02-20T00:22:40.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:40.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:40.331048+0000) 2024-02-20T00:22:41.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:41.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:41.331203+0000) 2024-02-20T00:22:42.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:42.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:42.331348+0000) 2024-02-20T00:22:43.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:43.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:43.331535+0000) 2024-02-20T00:22:44.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:44.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:44.331680+0000) 2024-02-20T00:22:45.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:45.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:45.331810+0000) 2024-02-20T00:22:46.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:46.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:46.331998+0000) 2024-02-20T00:22:47.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:47.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:47.332186+0000) 2024-02-20T00:22:48.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:48.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:48.332386+0000) 2024-02-20T00:22:49.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:49.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:49.332568+0000) 2024-02-20T00:22:50.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:50.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:50.332732+0000) 2024-02-20T00:22:51.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:51.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:51.332938+0000) 2024-02-20T00:22:52.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:52.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:52.333130+0000) 2024-02-20T00:22:53.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:53.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:53.333269+0000) 2024-02-20T00:22:54.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:54.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:54.333488+0000) 2024-02-20T00:22:55.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:55.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:55.333695+0000) 2024-02-20T00:22:56.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:56.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:56.333922+0000) 2024-02-20T00:22:57.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:57.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:57.334084+0000) 2024-02-20T00:22:58.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:58.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:58.334313+0000) 2024-02-20T00:22:59.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:22:59.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:22:59.334499+0000) 2024-02-20T00:23:00.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:00.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:00.334704+0000) 2024-02-20T00:23:01.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:01.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:01.334939+0000) 2024-02-20T00:23:02.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:02.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:02.335154+0000) 2024-02-20T00:23:03.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:03.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:03.335310+0000) 2024-02-20T00:23:04.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:04.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:04.335491+0000) 2024-02-20T00:23:05.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:05.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:05.335695+0000) 2024-02-20T00:23:06.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:06.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:06.335941+0000) 2024-02-20T00:23:07.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:07.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:07.336147+0000) 2024-02-20T00:23:08.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:08.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:08.336473+0000) 2024-02-20T00:23:09.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:09.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:09.336594+0000) 2024-02-20T00:23:10.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:10.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:10.336827+0000) 2024-02-20T00:23:11.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:11.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:11.336976+0000) 2024-02-20T00:23:12.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:12.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:12.337197+0000) 2024-02-20T00:23:13.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:13.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:13.337370+0000) 2024-02-20T00:23:14.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:14.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:14.337524+0000) 2024-02-20T00:23:15.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:15.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:15.337674+0000) 2024-02-20T00:23:16.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:16.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:16.337837+0000) 2024-02-20T00:23:17.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:17.332+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:17.338002+0000) 2024-02-20T00:23:18.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:18.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:18.338163+0000) 2024-02-20T00:23:19.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:19.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:19.338390+0000) 2024-02-20T00:23:20.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:20.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:20.338597+0000) 2024-02-20T00:23:21.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:21.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:21.338825+0000) 2024-02-20T00:23:22.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:22.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:22.338975+0000) 2024-02-20T00:23:23.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:23.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:23.339094+0000) 2024-02-20T00:23:24.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:24.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:24.339297+0000) 2024-02-20T00:23:25.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:25.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:25.339542+0000) 2024-02-20T00:23:26.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:26.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:26.339733+0000) 2024-02-20T00:23:27.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:27.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:27.339916+0000) 2024-02-20T00:23:28.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:28.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:28.340099+0000) 2024-02-20T00:23:29.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:29.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:29.340296+0000) 2024-02-20T00:23:30.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:30.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:30.340533+0000) 2024-02-20T00:23:31.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:31.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:31.340751+0000) 2024-02-20T00:23:32.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:32.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:32.341031+0000) 2024-02-20T00:23:33.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:33.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:33.341194+0000) 2024-02-20T00:23:34.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:34.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:34.341357+0000) 2024-02-20T00:23:35.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:35.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:35.341574+0000) 2024-02-20T00:23:36.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:36.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:36.341717+0000) 2024-02-20T00:23:37.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:37.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:37.341922+0000) 2024-02-20T00:23:38.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:38.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:38.342074+0000) 2024-02-20T00:23:39.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:39.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:39.342198+0000) 2024-02-20T00:23:40.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:40.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:40.342346+0000) 2024-02-20T00:23:41.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:41.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:41.342490+0000) 2024-02-20T00:23:42.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:42.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:42.342683+0000) 2024-02-20T00:23:43.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:43.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:43.342819+0000) 2024-02-20T00:23:44.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:44.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:44.342965+0000) 2024-02-20T00:23:45.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:45.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:45.343100+0000) 2024-02-20T00:23:46.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:46.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:46.343242+0000) 2024-02-20T00:23:47.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:47.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:47.343373+0000) 2024-02-20T00:23:48.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:48.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:48.343513+0000) 2024-02-20T00:23:49.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:49.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:49.343650+0000) 2024-02-20T00:23:50.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:50.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:50.343783+0000) 2024-02-20T00:23:51.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:51.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:51.343919+0000) 2024-02-20T00:23:52.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:52.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:52.344099+0000) 2024-02-20T00:23:53.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:53.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:53.344257+0000) 2024-02-20T00:23:54.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:54.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:54.344420+0000) 2024-02-20T00:23:55.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:55.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:55.344641+0000) 2024-02-20T00:23:56.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:56.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:56.344891+0000) 2024-02-20T00:23:57.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:57.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:57.345057+0000) 2024-02-20T00:23:58.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:58.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:58.345265+0000) 2024-02-20T00:23:59.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:23:59.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:23:59.345421+0000) 2024-02-20T00:24:00.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:00.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:00.345592+0000) 2024-02-20T00:24:01.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:01.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:01.345819+0000) 2024-02-20T00:24:02.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:02.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:02.346066+0000) 2024-02-20T00:24:03.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:03.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:03.346359+0000) 2024-02-20T00:24:04.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:04.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:04.346577+0000) 2024-02-20T00:24:05.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:05.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:05.346838+0000) 2024-02-20T00:24:06.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:06.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:06.346964+0000) 2024-02-20T00:24:07.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:07.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:07.347171+0000) 2024-02-20T00:24:08.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:08.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:08.347350+0000) 2024-02-20T00:24:09.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:09.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:09.347532+0000) 2024-02-20T00:24:10.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:10.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:10.347755+0000) 2024-02-20T00:24:11.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:11.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:11.347923+0000) 2024-02-20T00:24:12.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:12.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:12.348098+0000) 2024-02-20T00:24:13.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:13.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:13.348321+0000) 2024-02-20T00:24:14.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:14.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:14.348473+0000) 2024-02-20T00:24:15.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:15.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:15.348636+0000) 2024-02-20T00:24:16.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:16.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:16.348826+0000) 2024-02-20T00:24:17.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:17.347+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:17.349007+0000) 2024-02-20T00:24:18.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:18.343+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:18.349243+0000) 2024-02-20T00:24:19.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:19.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:19.349434+0000) 2024-02-20T00:24:20.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:20.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:20.349629+0000) 2024-02-20T00:24:21.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:21.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:21.349854+0000) 2024-02-20T00:24:22.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:22.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:22.350054+0000) 2024-02-20T00:24:23.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:23.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:23.350287+0000) 2024-02-20T00:24:24.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:24.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:24.350543+0000) 2024-02-20T00:24:25.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:25.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:25.350761+0000) 2024-02-20T00:24:26.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:26.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:26.350899+0000) 2024-02-20T00:24:27.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:27.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:27.351032+0000) 2024-02-20T00:24:28.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:28.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:28.351236+0000) 2024-02-20T00:24:29.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:29.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:29.351507+0000) 2024-02-20T00:24:30.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:30.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:30.351738+0000) 2024-02-20T00:24:31.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:31.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:31.351985+0000) 2024-02-20T00:24:32.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:32.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:32.352223+0000) 2024-02-20T00:24:33.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:33.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:33.352488+0000) 2024-02-20T00:24:34.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:34.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:34.352728+0000) 2024-02-20T00:24:35.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:35.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:35.353013+0000) 2024-02-20T00:24:36.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:36.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:36.353285+0000) 2024-02-20T00:24:37.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:37.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:37.353550+0000) 2024-02-20T00:24:38.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:38.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:38.353783+0000) 2024-02-20T00:24:39.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:39.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:39.354037+0000) 2024-02-20T00:24:40.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:40.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:40.354310+0000) 2024-02-20T00:24:41.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:41.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:41.354559+0000) 2024-02-20T00:24:42.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:42.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:42.354866+0000) 2024-02-20T00:24:43.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:43.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:43.355084+0000) 2024-02-20T00:24:44.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:44.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:44.355257+0000) 2024-02-20T00:24:45.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:45.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:45.355469+0000) 2024-02-20T00:24:46.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:46.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:46.355663+0000) 2024-02-20T00:24:47.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:47.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:47.355928+0000) 2024-02-20T00:24:48.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:48.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:48.356184+0000) 2024-02-20T00:24:49.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:49.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:49.356389+0000) 2024-02-20T00:24:50.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:50.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:50.356632+0000) 2024-02-20T00:24:51.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:51.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:51.356925+0000) 2024-02-20T00:24:52.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:52.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:52.357157+0000) 2024-02-20T00:24:53.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:53.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:53.357447+0000) 2024-02-20T00:24:54.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:54.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:54.357619+0000) 2024-02-20T00:24:55.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:55.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:55.357880+0000) 2024-02-20T00:24:56.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:56.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:56.358125+0000) 2024-02-20T00:24:57.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:57.354+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:57.358415+0000) 2024-02-20T00:24:58.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:58.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:58.358593+0000) 2024-02-20T00:24:59.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:24:59.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:24:59.358851+0000) 2024-02-20T00:25:00.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:00.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:00.359117+0000) 2024-02-20T00:25:01.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:01.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:01.359411+0000) 2024-02-20T00:25:02.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:02.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:02.359622+0000) 2024-02-20T00:25:03.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:03.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:03.359915+0000) 2024-02-20T00:25:04.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:04.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:04.360175+0000) 2024-02-20T00:25:05.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:05.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:05.360464+0000) 2024-02-20T00:25:06.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:06.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:06.360697+0000) 2024-02-20T00:25:07.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:07.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:07.360964+0000) 2024-02-20T00:25:08.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:08.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:08.361174+0000) 2024-02-20T00:25:09.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:09.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:09.361399+0000) 2024-02-20T00:25:10.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:10.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:10.361681+0000) 2024-02-20T00:25:11.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:11.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:11.361941+0000) 2024-02-20T00:25:12.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:12.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:12.362168+0000) 2024-02-20T00:25:13.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:13.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:13.362355+0000) 2024-02-20T00:25:14.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:14.358+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:14.362516+0000) 2024-02-20T00:25:15.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:15.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:15.362722+0000) 2024-02-20T00:25:16.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:16.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:16.362871+0000) 2024-02-20T00:25:17.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:17.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:17.363118+0000) 2024-02-20T00:25:18.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:18.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:18.363294+0000) 2024-02-20T00:25:19.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:19.362+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:19.363553+0000) 2024-02-20T00:25:20.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:20.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:20.363837+0000) 2024-02-20T00:25:21.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:21.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:21.364062+0000) 2024-02-20T00:25:22.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:22.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:22.364302+0000) 2024-02-20T00:25:23.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:23.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:23.364580+0000) 2024-02-20T00:25:24.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:24.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:24.364821+0000) 2024-02-20T00:25:25.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:25.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:25.365054+0000) 2024-02-20T00:25:26.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:26.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:26.365320+0000) 2024-02-20T00:25:27.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:27.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:27.365562+0000) 2024-02-20T00:25:28.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:28.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:28.365819+0000) 2024-02-20T00:25:29.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:29.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:29.366049+0000) 2024-02-20T00:25:30.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:30.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:30.366333+0000) 2024-02-20T00:25:31.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:31.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:31.366531+0000) 2024-02-20T00:25:32.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:32.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:32.366821+0000) 2024-02-20T00:25:33.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:33.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:33.367034+0000) 2024-02-20T00:25:34.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:34.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:34.367229+0000) 2024-02-20T00:25:35.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:35.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:35.367484+0000) 2024-02-20T00:25:36.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:36.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:36.367710+0000) 2024-02-20T00:25:37.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:37.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:37.367933+0000) 2024-02-20T00:25:38.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:38.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:38.368101+0000) 2024-02-20T00:25:39.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:39.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:39.368324+0000) 2024-02-20T00:25:40.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:40.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:40.368527+0000) 2024-02-20T00:25:41.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:41.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:41.368686+0000) 2024-02-20T00:25:42.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:42.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:42.368924+0000) 2024-02-20T00:25:43.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:43.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:43.369134+0000) 2024-02-20T00:25:44.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:44.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:44.369393+0000) 2024-02-20T00:25:45.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:45.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:45.369613+0000) 2024-02-20T00:25:46.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:46.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:46.369856+0000) 2024-02-20T00:25:47.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:47.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:47.370167+0000) 2024-02-20T00:25:48.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:48.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:48.370338+0000) 2024-02-20T00:25:49.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:49.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:49.370572+0000) 2024-02-20T00:25:50.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:50.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:50.370810+0000) 2024-02-20T00:25:51.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:51.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:51.370987+0000) 2024-02-20T00:25:52.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:52.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:52.371150+0000) 2024-02-20T00:25:53.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:53.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:53.371362+0000) 2024-02-20T00:25:54.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:54.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:54.371593+0000) 2024-02-20T00:25:55.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:55.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:55.371846+0000) 2024-02-20T00:25:56.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:56.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:56.372063+0000) 2024-02-20T00:25:57.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:57.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:57.372375+0000) 2024-02-20T00:25:58.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:58.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:58.372544+0000) 2024-02-20T00:25:59.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:25:59.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:25:59.372683+0000) 2024-02-20T00:26:00.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:00.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:00.372829+0000) 2024-02-20T00:26:01.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:01.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:01.373082+0000) 2024-02-20T00:26:02.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:02.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:02.373317+0000) 2024-02-20T00:26:03.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:03.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:03.373524+0000) 2024-02-20T00:26:04.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:04.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:04.373750+0000) 2024-02-20T00:26:05.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:05.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:05.373994+0000) 2024-02-20T00:26:06.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:06.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:06.374169+0000) 2024-02-20T00:26:07.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:07.369+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:07.374445+0000) 2024-02-20T00:26:08.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:08.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:08.374671+0000) 2024-02-20T00:26:09.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:09.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:09.374958+0000) 2024-02-20T00:26:10.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:10.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:10.375236+0000) 2024-02-20T00:26:11.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:11.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:11.375489+0000) 2024-02-20T00:26:12.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:12.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:12.375667+0000) 2024-02-20T00:26:13.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:13.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:13.375865+0000) 2024-02-20T00:26:14.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:14.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:14.376096+0000) 2024-02-20T00:26:15.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:15.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:15.376336+0000) 2024-02-20T00:26:16.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:16.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:16.376517+0000) 2024-02-20T00:26:17.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:17.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:17.376664+0000) 2024-02-20T00:26:18.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:18.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:18.376803+0000) 2024-02-20T00:26:19.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:19.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:19.376973+0000) 2024-02-20T00:26:20.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:20.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:20.377153+0000) 2024-02-20T00:26:21.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:21.373+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:21.377322+0000) 2024-02-20T00:26:22.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:22.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:22.377492+0000) 2024-02-20T00:26:23.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:23.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:23.377629+0000) 2024-02-20T00:26:24.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:24.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:24.377777+0000) 2024-02-20T00:26:25.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:25.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:25.377921+0000) 2024-02-20T00:26:26.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:26.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:26.378064+0000) 2024-02-20T00:26:27.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:27.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:27.378210+0000) 2024-02-20T00:26:28.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:28.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:28.378347+0000) 2024-02-20T00:26:29.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:29.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:29.378468+0000) 2024-02-20T00:26:30.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:30.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:30.378603+0000) 2024-02-20T00:26:31.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:31.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:31.378778+0000) 2024-02-20T00:26:32.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:32.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:32.378929+0000) 2024-02-20T00:26:33.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:33.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:33.379065+0000) 2024-02-20T00:26:34.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:34.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:34.379190+0000) 2024-02-20T00:26:35.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:35.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:35.379366+0000) 2024-02-20T00:26:36.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:36.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:36.379605+0000) 2024-02-20T00:26:37.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:37.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:37.379859+0000) 2024-02-20T00:26:38.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:38.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:38.380101+0000) 2024-02-20T00:26:39.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:39.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:39.380251+0000) 2024-02-20T00:26:40.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:40.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:40.380507+0000) 2024-02-20T00:26:41.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:41.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:41.380698+0000) 2024-02-20T00:26:42.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:42.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:42.380888+0000) 2024-02-20T00:26:43.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:43.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:43.381138+0000) 2024-02-20T00:26:44.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:44.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:44.381352+0000) 2024-02-20T00:26:45.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:45.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:45.381597+0000) 2024-02-20T00:26:46.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:46.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:46.381794+0000) 2024-02-20T00:26:47.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:47.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:47.381917+0000) 2024-02-20T00:26:48.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:48.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:48.382161+0000) 2024-02-20T00:26:49.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:49.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:49.382330+0000) 2024-02-20T00:26:50.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:50.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:50.382468+0000) 2024-02-20T00:26:51.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:51.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:51.382698+0000) 2024-02-20T00:26:52.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:52.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:52.382927+0000) 2024-02-20T00:26:53.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:53.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:53.383086+0000) 2024-02-20T00:26:54.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:54.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:54.383267+0000) 2024-02-20T00:26:55.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:55.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:55.383450+0000) 2024-02-20T00:26:56.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:56.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:56.383626+0000) 2024-02-20T00:26:57.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:57.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:57.383883+0000) 2024-02-20T00:26:58.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:58.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:58.384048+0000) 2024-02-20T00:26:59.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:26:59.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:26:59.384210+0000) 2024-02-20T00:27:00.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:00.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:00.384465+0000) 2024-02-20T00:27:01.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:01.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:01.384733+0000) 2024-02-20T00:27:02.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:02.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:02.384898+0000) 2024-02-20T00:27:03.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:03.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:03.385060+0000) 2024-02-20T00:27:04.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:04.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:04.385354+0000) 2024-02-20T00:27:05.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:05.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:05.385602+0000) 2024-02-20T00:27:06.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:06.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:06.385734+0000) 2024-02-20T00:27:07.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:07.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:07.385889+0000) 2024-02-20T00:27:08.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:08.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:08.386025+0000) 2024-02-20T00:27:09.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:09.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:09.386187+0000) 2024-02-20T00:27:10.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:10.380+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:10.386389+0000) 2024-02-20T00:27:11.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:11.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:11.386610+0000) 2024-02-20T00:27:12.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:12.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:12.386952+0000) 2024-02-20T00:27:13.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:13.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:13.387199+0000) 2024-02-20T00:27:14.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:14.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:14.387404+0000) 2024-02-20T00:27:15.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:15.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:15.387660+0000) 2024-02-20T00:27:16.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:16.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:16.387908+0000) 2024-02-20T00:27:17.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:17.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:17.388167+0000) 2024-02-20T00:27:18.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:18.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:18.388322+0000) 2024-02-20T00:27:19.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:19.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:19.388470+0000) 2024-02-20T00:27:20.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:20.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:20.388657+0000) 2024-02-20T00:27:21.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:21.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:21.388832+0000) 2024-02-20T00:27:22.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:22.384+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:22.388982+0000) 2024-02-20T00:27:23.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:23.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:23.389177+0000) 2024-02-20T00:27:24.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:24.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:24.389353+0000) 2024-02-20T00:27:25.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:25.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:25.389594+0000) 2024-02-20T00:27:26.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:26.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:26.389884+0000) 2024-02-20T00:27:27.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:27.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:27.390089+0000) 2024-02-20T00:27:28.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:28.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:28.390219+0000) 2024-02-20T00:27:29.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:29.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:29.390371+0000) 2024-02-20T00:27:30.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:30.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:30.390518+0000) 2024-02-20T00:27:31.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:31.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:31.390718+0000) 2024-02-20T00:27:32.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:32.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:32.390882+0000) 2024-02-20T00:27:33.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:33.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:33.391026+0000) 2024-02-20T00:27:34.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:34.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:34.391254+0000) 2024-02-20T00:27:35.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:35.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:35.391467+0000) 2024-02-20T00:27:36.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:36.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:36.391793+0000) 2024-02-20T00:27:37.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:37.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:37.392026+0000) 2024-02-20T00:27:38.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:38.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:38.392214+0000) 2024-02-20T00:27:39.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:39.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:39.392390+0000) 2024-02-20T00:27:40.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:40.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:40.392551+0000) 2024-02-20T00:27:41.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:41.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:41.392794+0000) 2024-02-20T00:27:42.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:42.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:42.393002+0000) 2024-02-20T00:27:43.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:43.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:43.393222+0000) 2024-02-20T00:27:44.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:44.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:44.393361+0000) 2024-02-20T00:27:45.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:45.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:45.393516+0000) 2024-02-20T00:27:46.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:46.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:46.393660+0000) 2024-02-20T00:27:47.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:47.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:47.393785+0000) 2024-02-20T00:27:48.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:48.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:48.393983+0000) 2024-02-20T00:27:49.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:49.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:49.394193+0000) 2024-02-20T00:27:50.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:50.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:50.394374+0000) 2024-02-20T00:27:51.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:51.387+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:51.394570+0000) 2024-02-20T00:27:52.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:52.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:52.394776+0000) 2024-02-20T00:27:53.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:53.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:53.394988+0000) 2024-02-20T00:27:54.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:54.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:54.395157+0000) 2024-02-20T00:27:55.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:55.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:55.395370+0000) 2024-02-20T00:27:56.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:56.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:56.395558+0000) 2024-02-20T00:27:57.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:57.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:57.395782+0000) 2024-02-20T00:27:58.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:58.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:58.395981+0000) 2024-02-20T00:27:59.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:27:59.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:27:59.396198+0000) 2024-02-20T00:28:00.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:00.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:00.396370+0000) 2024-02-20T00:28:01.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:01.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:01.396590+0000) 2024-02-20T00:28:02.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:02.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:02.396820+0000) 2024-02-20T00:28:03.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:03.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:03.397055+0000) 2024-02-20T00:28:04.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:04.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:04.397215+0000) 2024-02-20T00:28:05.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:05.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:05.397386+0000) 2024-02-20T00:28:06.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:06.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:06.397552+0000) 2024-02-20T00:28:07.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:07.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:07.397673+0000) 2024-02-20T00:28:08.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:08.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:08.397847+0000) 2024-02-20T00:28:09.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:09.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:09.398014+0000) 2024-02-20T00:28:10.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:10.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:10.398204+0000) 2024-02-20T00:28:11.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:11.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:11.398425+0000) 2024-02-20T00:28:12.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:12.391+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:12.398681+0000) 2024-02-20T00:28:13.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:13.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:13.398853+0000) 2024-02-20T00:28:14.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:14.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:14.398987+0000) 2024-02-20T00:28:15.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:15.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:15.399161+0000) 2024-02-20T00:28:16.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:16.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:16.399331+0000) 2024-02-20T00:28:17.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:17.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:17.399500+0000) 2024-02-20T00:28:18.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:18.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:18.399703+0000) 2024-02-20T00:28:19.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:19.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:19.399937+0000) 2024-02-20T00:28:20.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:20.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:20.400125+0000) 2024-02-20T00:28:21.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:21.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:21.400367+0000) 2024-02-20T00:28:22.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:22.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:22.400539+0000) 2024-02-20T00:28:23.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:23.395+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:23.400682+0000) 2024-02-20T00:28:24.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:24.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:24.400899+0000) 2024-02-20T00:28:25.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:25.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:25.401065+0000) 2024-02-20T00:28:26.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:26.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:26.401228+0000) 2024-02-20T00:28:27.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:27.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:27.401405+0000) 2024-02-20T00:28:28.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:28.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:28.401577+0000) 2024-02-20T00:28:29.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:29.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:29.401720+0000) 2024-02-20T00:28:30.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:30.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:30.401903+0000) 2024-02-20T00:28:31.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:31.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:31.402124+0000) 2024-02-20T00:28:32.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:32.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:32.402274+0000) 2024-02-20T00:28:33.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:33.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:33.402475+0000) 2024-02-20T00:28:34.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:34.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:34.402684+0000) 2024-02-20T00:28:35.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:35.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:35.402869+0000) 2024-02-20T00:28:36.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:36.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:36.403076+0000) 2024-02-20T00:28:37.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:37.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:37.403272+0000) 2024-02-20T00:28:38.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:38.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:38.403502+0000) 2024-02-20T00:28:39.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:39.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:39.403653+0000) 2024-02-20T00:28:40.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:40.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:40.403810+0000) 2024-02-20T00:28:41.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:41.398+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:41.403948+0000) 2024-02-20T00:28:42.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:42.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:42.404155+0000) 2024-02-20T00:28:43.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:43.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:43.404323+0000) 2024-02-20T00:28:44.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:44.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:44.404488+0000) 2024-02-20T00:28:45.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:45.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:45.404659+0000) 2024-02-20T00:28:46.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:46.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:46.404807+0000) 2024-02-20T00:28:47.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:47.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:47.404942+0000) 2024-02-20T00:28:48.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:48.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:48.405075+0000) 2024-02-20T00:28:49.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:49.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:49.405190+0000) 2024-02-20T00:28:50.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:50.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:50.405313+0000) 2024-02-20T00:28:51.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:51.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:51.405576+0000) 2024-02-20T00:28:52.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:52.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:52.405848+0000) 2024-02-20T00:28:53.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:53.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:53.406071+0000) 2024-02-20T00:28:54.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:54.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:54.406286+0000) 2024-02-20T00:28:55.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:55.402+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:55.406502+0000) 2024-02-20T00:28:56.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:56.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:56.406735+0000) 2024-02-20T00:28:57.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:57.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:57.406961+0000) 2024-02-20T00:28:58.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:58.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:58.407119+0000) 2024-02-20T00:28:59.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:28:59.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:28:59.407298+0000) 2024-02-20T00:29:00.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:00.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:00.407469+0000) 2024-02-20T00:29:01.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:01.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:01.407695+0000) 2024-02-20T00:29:02.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:02.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:02.407920+0000) 2024-02-20T00:29:03.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:03.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:03.408169+0000) 2024-02-20T00:29:04.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:04.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:04.408361+0000) 2024-02-20T00:29:05.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:05.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:05.408584+0000) 2024-02-20T00:29:06.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:06.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:06.408808+0000) 2024-02-20T00:29:07.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:07.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:07.409101+0000) 2024-02-20T00:29:08.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:08.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:08.409284+0000) 2024-02-20T00:29:09.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:09.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:09.409515+0000) 2024-02-20T00:29:10.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:10.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:10.409729+0000) 2024-02-20T00:29:11.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:11.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:11.409978+0000) 2024-02-20T00:29:12.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:12.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:12.410269+0000) 2024-02-20T00:29:13.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:13.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:13.410481+0000) 2024-02-20T00:29:14.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:14.406+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:14.410661+0000) 2024-02-20T00:29:15.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:15.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:15.410790+0000) 2024-02-20T00:29:16.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:16.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:16.410970+0000) 2024-02-20T00:29:17.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:17.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:17.411192+0000) 2024-02-20T00:29:18.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:18.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:18.411427+0000) 2024-02-20T00:29:19.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:19.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:19.411667+0000) 2024-02-20T00:29:20.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:20.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:20.411927+0000) 2024-02-20T00:29:21.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:21.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:21.412138+0000) 2024-02-20T00:29:22.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:22.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:22.412401+0000) 2024-02-20T00:29:23.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:23.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:23.412588+0000) 2024-02-20T00:29:24.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:24.410+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:24.412804+0000) 2024-02-20T00:29:25.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:25.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:25.413048+0000) 2024-02-20T00:29:26.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:26.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:26.413208+0000) 2024-02-20T00:29:27.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:27.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:27.413363+0000) 2024-02-20T00:29:28.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:28.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:28.413514+0000) 2024-02-20T00:29:29.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:29.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:29.413718+0000) 2024-02-20T00:29:30.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:30.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:30.413872+0000) 2024-02-20T00:29:31.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:31.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:31.414074+0000) 2024-02-20T00:29:32.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:32.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:32.414228+0000) 2024-02-20T00:29:33.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:33.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:33.414431+0000) 2024-02-20T00:29:34.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:34.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:34.414617+0000) 2024-02-20T00:29:35.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:35.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:35.414821+0000) 2024-02-20T00:29:36.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:36.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:36.415023+0000) 2024-02-20T00:29:37.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:37.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:37.415194+0000) 2024-02-20T00:29:38.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:38.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:38.415425+0000) 2024-02-20T00:29:39.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:39.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:39.415668+0000) 2024-02-20T00:29:40.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:40.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:40.415881+0000) 2024-02-20T00:29:41.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:41.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:41.416113+0000) 2024-02-20T00:29:42.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:42.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:42.416377+0000) 2024-02-20T00:29:43.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:43.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:43.416667+0000) 2024-02-20T00:29:44.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:44.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:44.416842+0000) 2024-02-20T00:29:45.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:45.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:45.417028+0000) 2024-02-20T00:29:46.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:46.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:46.417177+0000) 2024-02-20T00:29:47.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:47.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:47.417358+0000) 2024-02-20T00:29:48.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:48.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:48.417570+0000) 2024-02-20T00:29:49.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:49.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:49.417723+0000) 2024-02-20T00:29:50.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:50.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:50.417955+0000) 2024-02-20T00:29:51.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:51.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:51.418167+0000) 2024-02-20T00:29:52.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:52.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:52.418306+0000) 2024-02-20T00:29:53.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:53.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:53.418491+0000) 2024-02-20T00:29:54.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:54.413+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:54.418625+0000) 2024-02-20T00:29:55.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:55.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:55.418780+0000) 2024-02-20T00:29:56.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:56.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:56.418907+0000) 2024-02-20T00:29:57.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:57.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:57.419085+0000) 2024-02-20T00:29:58.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:58.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:58.419208+0000) 2024-02-20T00:29:59.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:29:59.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:29:59.419391+0000) 2024-02-20T00:30:00.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:00.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:00.419568+0000) 2024-02-20T00:30:01.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:01.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:01.419855+0000) 2024-02-20T00:30:02.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:02.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:02.420061+0000) 2024-02-20T00:30:03.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:03.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:03.420244+0000) 2024-02-20T00:30:04.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:04.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:04.420460+0000) 2024-02-20T00:30:05.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:05.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:05.420628+0000) 2024-02-20T00:30:06.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:06.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:06.420841+0000) 2024-02-20T00:30:07.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:07.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:07.421107+0000) 2024-02-20T00:30:08.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:08.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:08.421356+0000) 2024-02-20T00:30:09.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:09.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:09.421604+0000) 2024-02-20T00:30:10.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:10.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:10.421863+0000) 2024-02-20T00:30:11.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:11.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:11.422037+0000) 2024-02-20T00:30:12.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:12.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:12.422317+0000) 2024-02-20T00:30:13.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:13.417+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:13.422607+0000) 2024-02-20T00:30:14.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:14.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:14.422807+0000) 2024-02-20T00:30:15.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:15.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:15.422935+0000) 2024-02-20T00:30:16.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:16.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:16.423188+0000) 2024-02-20T00:30:17.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:17.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:17.423384+0000) 2024-02-20T00:30:18.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:18.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:18.423604+0000) 2024-02-20T00:30:19.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:19.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:19.423816+0000) 2024-02-20T00:30:20.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:20.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:20.424050+0000) 2024-02-20T00:30:21.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:21.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:21.424324+0000) 2024-02-20T00:30:22.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:22.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:22.424529+0000) 2024-02-20T00:30:23.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:23.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:23.424796+0000) 2024-02-20T00:30:24.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:24.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:24.425049+0000) 2024-02-20T00:30:25.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:25.421+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:25.425280+0000) 2024-02-20T00:30:26.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:26.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:26.425546+0000) 2024-02-20T00:30:27.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:27.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:27.425791+0000) 2024-02-20T00:30:28.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:28.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:28.425999+0000) 2024-02-20T00:30:29.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:29.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:29.426212+0000) 2024-02-20T00:30:30.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:30.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:30.426443+0000) 2024-02-20T00:30:31.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:31.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:31.426641+0000) 2024-02-20T00:30:32.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:32.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:32.426927+0000) 2024-02-20T00:30:33.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:33.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:33.427163+0000) 2024-02-20T00:30:34.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:34.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:34.427347+0000) 2024-02-20T00:30:35.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:35.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:35.427596+0000) 2024-02-20T00:30:36.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:36.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:36.427832+0000) 2024-02-20T00:30:37.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:37.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:37.428015+0000) 2024-02-20T00:30:38.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:38.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:38.428210+0000) 2024-02-20T00:30:39.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:39.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:39.428365+0000) 2024-02-20T00:30:40.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:40.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:40.428592+0000) 2024-02-20T00:30:41.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:41.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:41.428779+0000) 2024-02-20T00:30:42.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:42.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:42.428970+0000) 2024-02-20T00:30:43.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:43.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:43.429107+0000) 2024-02-20T00:30:44.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:44.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:44.429324+0000) 2024-02-20T00:30:45.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:45.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:45.429498+0000) 2024-02-20T00:30:46.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:46.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:46.429668+0000) 2024-02-20T00:30:47.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:47.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:47.429842+0000) 2024-02-20T00:30:48.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:48.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:48.430001+0000) 2024-02-20T00:30:49.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:49.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:49.430157+0000) 2024-02-20T00:30:50.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:50.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:50.430377+0000) 2024-02-20T00:30:51.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:51.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:51.430520+0000) 2024-02-20T00:30:52.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:52.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:52.430679+0000) 2024-02-20T00:30:53.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:53.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:53.430826+0000) 2024-02-20T00:30:54.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:54.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:54.431063+0000) 2024-02-20T00:30:55.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:55.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:55.431217+0000) 2024-02-20T00:30:56.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:56.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:56.431366+0000) 2024-02-20T00:30:57.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:57.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:57.431588+0000) 2024-02-20T00:30:58.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:58.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:58.431742+0000) 2024-02-20T00:30:59.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:30:59.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:30:59.431866+0000) 2024-02-20T00:31:00.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:00.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:00.432005+0000) 2024-02-20T00:31:01.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:01.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:01.432161+0000) 2024-02-20T00:31:02.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:02.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:02.432353+0000) 2024-02-20T00:31:03.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:03.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:03.432535+0000) 2024-02-20T00:31:04.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:04.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:04.432719+0000) 2024-02-20T00:31:05.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:05.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:05.432871+0000) 2024-02-20T00:31:06.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:06.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:06.433153+0000) 2024-02-20T00:31:07.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:07.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:07.433430+0000) 2024-02-20T00:31:08.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:08.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:08.433678+0000) 2024-02-20T00:31:09.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:09.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:09.433972+0000) 2024-02-20T00:31:10.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:10.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:10.434162+0000) 2024-02-20T00:31:11.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:11.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:11.434332+0000) 2024-02-20T00:31:12.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:12.428+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:12.434526+0000) 2024-02-20T00:31:13.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:13.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:13.434722+0000) 2024-02-20T00:31:14.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:14.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:14.434870+0000) 2024-02-20T00:31:15.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:15.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:15.435068+0000) 2024-02-20T00:31:16.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:16.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:16.435266+0000) 2024-02-20T00:31:17.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:17.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:17.435480+0000) 2024-02-20T00:31:18.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:18.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:18.435728+0000) 2024-02-20T00:31:19.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:19.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:19.435927+0000) 2024-02-20T00:31:20.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:20.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:20.436207+0000) 2024-02-20T00:31:21.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:21.436+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:21.436476+0000) 2024-02-20T00:31:22.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:22.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:22.436684+0000) 2024-02-20T00:31:23.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:23.436+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:23.436943+0000) 2024-02-20T00:31:24.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:24.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:24.437196+0000) 2024-02-20T00:31:25.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:25.436+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:25.437434+0000) 2024-02-20T00:31:26.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:26.432+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:26.437660+0000) 2024-02-20T00:31:27.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:27.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:27.437956+0000) 2024-02-20T00:31:28.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:28.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:28.438139+0000) 2024-02-20T00:31:29.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:29.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:29.438391+0000) 2024-02-20T00:31:30.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:30.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:30.438637+0000) 2024-02-20T00:31:31.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:31.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:31.438834+0000) 2024-02-20T00:31:32.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:32.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:32.439063+0000) 2024-02-20T00:31:33.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:33.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:33.439372+0000) 2024-02-20T00:31:34.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:34.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:34.439542+0000) 2024-02-20T00:31:35.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:35.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:35.439775+0000) 2024-02-20T00:31:36.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:36.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:36.439915+0000) 2024-02-20T00:31:37.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:37.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:37.440166+0000) 2024-02-20T00:31:38.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:38.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:38.440398+0000) 2024-02-20T00:31:39.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:39.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:39.440630+0000) 2024-02-20T00:31:40.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:40.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:40.440890+0000) 2024-02-20T00:31:41.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:41.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:41.441155+0000) 2024-02-20T00:31:42.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:42.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:42.441356+0000) 2024-02-20T00:31:43.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:43.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:43.441595+0000) 2024-02-20T00:31:44.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:44.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:44.441822+0000) 2024-02-20T00:31:45.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:45.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:45.442135+0000) 2024-02-20T00:31:46.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:46.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:46.442310+0000) 2024-02-20T00:31:47.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:47.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:47.442519+0000) 2024-02-20T00:31:48.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:48.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:48.442779+0000) 2024-02-20T00:31:49.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:49.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:49.443055+0000) 2024-02-20T00:31:50.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:50.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:50.443341+0000) 2024-02-20T00:31:51.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:51.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:51.443696+0000) 2024-02-20T00:31:52.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:52.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:52.443898+0000) 2024-02-20T00:31:53.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:53.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:53.444102+0000) 2024-02-20T00:31:54.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:54.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:54.444318+0000) 2024-02-20T00:31:55.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:55.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:55.444495+0000) 2024-02-20T00:31:56.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:56.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:56.444703+0000) 2024-02-20T00:31:57.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:57.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:57.444939+0000) 2024-02-20T00:31:58.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:58.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:58.445152+0000) 2024-02-20T00:31:59.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:31:59.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:31:59.445367+0000) 2024-02-20T00:32:00.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:00.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:00.445639+0000) 2024-02-20T00:32:01.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:01.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:01.445861+0000) 2024-02-20T00:32:02.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:02.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:02.446013+0000) 2024-02-20T00:32:03.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:03.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:03.446163+0000) 2024-02-20T00:32:04.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:04.439+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:04.446395+0000) 2024-02-20T00:32:05.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:05.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:05.446700+0000) 2024-02-20T00:32:06.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:06.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:06.446939+0000) 2024-02-20T00:32:07.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:07.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:07.447155+0000) 2024-02-20T00:32:08.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:08.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:08.447351+0000) 2024-02-20T00:32:09.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:09.447+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:09.447579+0000) 2024-02-20T00:32:10.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:10.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:10.447835+0000) 2024-02-20T00:32:11.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:11.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:11.448114+0000) 2024-02-20T00:32:12.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:12.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:12.448314+0000) 2024-02-20T00:32:13.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:13.447+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:13.448536+0000) 2024-02-20T00:32:14.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:14.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:14.448702+0000) 2024-02-20T00:32:15.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:15.447+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:15.448968+0000) 2024-02-20T00:32:16.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:16.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:16.449170+0000) 2024-02-20T00:32:17.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:17.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:17.449441+0000) 2024-02-20T00:32:18.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:18.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:18.449585+0000) 2024-02-20T00:32:19.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:19.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:19.449736+0000) 2024-02-20T00:32:20.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:20.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:20.449927+0000) 2024-02-20T00:32:21.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:21.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:21.450166+0000) 2024-02-20T00:32:22.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:22.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:22.450397+0000) 2024-02-20T00:32:23.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:23.443+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:23.450676+0000) 2024-02-20T00:32:24.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:24.447+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:24.450872+0000) 2024-02-20T00:32:25.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:25.447+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:25.451104+0000) 2024-02-20T00:32:26.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:26.447+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:26.451334+0000) 2024-02-20T00:32:27.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:27.447+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:27.451593+0000) 2024-02-20T00:32:28.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:28.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:28.451842+0000) 2024-02-20T00:32:29.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:29.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:29.452059+0000) 2024-02-20T00:32:30.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:30.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:30.452309+0000) 2024-02-20T00:32:31.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:31.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:31.452533+0000) 2024-02-20T00:32:32.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:32.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:32.452811+0000) 2024-02-20T00:32:33.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:33.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:33.453077+0000) 2024-02-20T00:32:34.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:34.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:34.453291+0000) 2024-02-20T00:32:35.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:35.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:35.453575+0000) 2024-02-20T00:32:36.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:36.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:36.453806+0000) 2024-02-20T00:32:37.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:37.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:37.454109+0000) 2024-02-20T00:32:38.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:38.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:38.454370+0000) 2024-02-20T00:32:39.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:39.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:39.454558+0000) 2024-02-20T00:32:40.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:40.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:40.454855+0000) 2024-02-20T00:32:41.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:41.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:41.455057+0000) 2024-02-20T00:32:42.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:42.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:42.455239+0000) 2024-02-20T00:32:43.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:43.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:43.455406+0000) 2024-02-20T00:32:44.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:44.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:44.455618+0000) 2024-02-20T00:32:45.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:45.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:45.455897+0000) 2024-02-20T00:32:46.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:46.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:46.456099+0000) 2024-02-20T00:32:47.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:47.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:47.456391+0000) 2024-02-20T00:32:48.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:48.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:48.456567+0000) 2024-02-20T00:32:49.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:49.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:49.456794+0000) 2024-02-20T00:32:50.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:50.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:50.457005+0000) 2024-02-20T00:32:51.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:51.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:51.457282+0000) 2024-02-20T00:32:52.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:52.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:52.457512+0000) 2024-02-20T00:32:53.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:53.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:53.457786+0000) 2024-02-20T00:32:54.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:54.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:54.458011+0000) 2024-02-20T00:32:55.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:55.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:55.458313+0000) 2024-02-20T00:32:56.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:56.454+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:56.458492+0000) 2024-02-20T00:32:57.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:57.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:57.458734+0000) 2024-02-20T00:32:58.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:58.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:58.458924+0000) 2024-02-20T00:32:59.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:32:59.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:32:59.459171+0000) 2024-02-20T00:33:00.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:00.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:00.459389+0000) 2024-02-20T00:33:01.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:01.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:01.459684+0000) 2024-02-20T00:33:02.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:02.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:02.459902+0000) 2024-02-20T00:33:03.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:03.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:03.460145+0000) 2024-02-20T00:33:04.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:04.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:04.460361+0000) 2024-02-20T00:33:05.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:05.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:05.460613+0000) 2024-02-20T00:33:06.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:06.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:06.460858+0000) 2024-02-20T00:33:07.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:07.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:07.461152+0000) 2024-02-20T00:33:08.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:08.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:08.461414+0000) 2024-02-20T00:33:09.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:09.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:09.461679+0000) 2024-02-20T00:33:10.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:10.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:10.461942+0000) 2024-02-20T00:33:11.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:11.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:11.462261+0000) 2024-02-20T00:33:12.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:12.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:12.462439+0000) 2024-02-20T00:33:13.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:13.458+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:13.462610+0000) 2024-02-20T00:33:14.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:14.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:14.462888+0000) 2024-02-20T00:33:15.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:15.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:15.463113+0000) 2024-02-20T00:33:16.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:16.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:16.463276+0000) 2024-02-20T00:33:17.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:17.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:17.463503+0000) 2024-02-20T00:33:18.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:18.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:18.463656+0000) 2024-02-20T00:33:19.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:19.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:19.463804+0000) 2024-02-20T00:33:20.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:20.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:20.464092+0000) 2024-02-20T00:33:21.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:21.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:21.464323+0000) 2024-02-20T00:33:22.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:22.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:22.464587+0000) 2024-02-20T00:33:23.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:23.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:23.464934+0000) 2024-02-20T00:33:24.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:24.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:24.465141+0000) 2024-02-20T00:33:25.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:25.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:25.465462+0000) 2024-02-20T00:33:26.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:26.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:26.465704+0000) 2024-02-20T00:33:27.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:27.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:27.465952+0000) 2024-02-20T00:33:28.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:28.462+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:28.466133+0000) 2024-02-20T00:33:29.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:29.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:29.466383+0000) 2024-02-20T00:33:30.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:30.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:30.466563+0000) 2024-02-20T00:33:31.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:31.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:31.466876+0000) 2024-02-20T00:33:32.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:32.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:32.467109+0000) 2024-02-20T00:33:33.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:33.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:33.467389+0000) 2024-02-20T00:33:34.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:34.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:34.467615+0000) 2024-02-20T00:33:35.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:35.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:35.467933+0000) 2024-02-20T00:33:36.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:36.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:36.468147+0000) 2024-02-20T00:33:37.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:37.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:37.468348+0000) 2024-02-20T00:33:38.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:38.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:38.468581+0000) 2024-02-20T00:33:39.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:39.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:39.468827+0000) 2024-02-20T00:33:40.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:40.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:40.469070+0000) 2024-02-20T00:33:41.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:41.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:41.469340+0000) 2024-02-20T00:33:42.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:42.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:42.469571+0000) 2024-02-20T00:33:43.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:43.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:43.469762+0000) 2024-02-20T00:33:44.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:44.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:44.469911+0000) 2024-02-20T00:33:45.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:45.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:45.470100+0000) 2024-02-20T00:33:46.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:46.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:46.470326+0000) 2024-02-20T00:33:47.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:47.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:47.470533+0000) 2024-02-20T00:33:48.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:48.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:48.470842+0000) 2024-02-20T00:33:49.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:49.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:49.471077+0000) 2024-02-20T00:33:50.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:50.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:50.471354+0000) 2024-02-20T00:33:51.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:51.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:51.471575+0000) 2024-02-20T00:33:52.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:52.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:52.471799+0000) 2024-02-20T00:33:53.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:53.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:53.472119+0000) 2024-02-20T00:33:54.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:54.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:54.472350+0000) 2024-02-20T00:33:55.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:55.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:55.472663+0000) 2024-02-20T00:33:56.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:56.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:56.472863+0000) 2024-02-20T00:33:57.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:57.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:57.473116+0000) 2024-02-20T00:33:58.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:58.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:58.473340+0000) 2024-02-20T00:33:59.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:33:59.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:33:59.473527+0000) 2024-02-20T00:34:00.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:00.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:00.473745+0000) 2024-02-20T00:34:01.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:01.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:01.474023+0000) 2024-02-20T00:34:02.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:02.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:02.474196+0000) 2024-02-20T00:34:03.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:03.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:03.474328+0000) 2024-02-20T00:34:04.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:04.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:04.474453+0000) 2024-02-20T00:34:05.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:05.469+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:05.474611+0000) 2024-02-20T00:34:06.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:06.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:06.474761+0000) 2024-02-20T00:34:07.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:07.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:07.474881+0000) 2024-02-20T00:34:08.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:08.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:08.475006+0000) 2024-02-20T00:34:09.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:09.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:09.475146+0000) 2024-02-20T00:34:10.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:10.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:10.475370+0000) 2024-02-20T00:34:11.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:11.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:11.475667+0000) 2024-02-20T00:34:12.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:12.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:12.475814+0000) 2024-02-20T00:34:13.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:13.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:13.475961+0000) 2024-02-20T00:34:14.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:14.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:14.476081+0000) 2024-02-20T00:34:15.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:15.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:15.476208+0000) 2024-02-20T00:34:16.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:16.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:16.476421+0000) 2024-02-20T00:34:17.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:17.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:17.476621+0000) 2024-02-20T00:34:18.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:18.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:18.476762+0000) 2024-02-20T00:34:19.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:19.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:19.476897+0000) 2024-02-20T00:34:20.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:20.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:20.477036+0000) 2024-02-20T00:34:21.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:21.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:21.477184+0000) 2024-02-20T00:34:22.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:22.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:22.477350+0000) 2024-02-20T00:34:23.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:23.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:23.477488+0000) 2024-02-20T00:34:24.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:24.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:24.477608+0000) 2024-02-20T00:34:25.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:25.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:25.477860+0000) 2024-02-20T00:34:26.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:26.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:26.478134+0000) 2024-02-20T00:34:27.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:27.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:27.478380+0000) 2024-02-20T00:34:28.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:28.473+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:28.478576+0000) 2024-02-20T00:34:29.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:29.477+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:29.478773+0000) 2024-02-20T00:34:30.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:30.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:30.479009+0000) 2024-02-20T00:34:31.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:31.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:31.479220+0000) 2024-02-20T00:34:32.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:32.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:32.479456+0000) 2024-02-20T00:34:33.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:33.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:33.479697+0000) 2024-02-20T00:34:34.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:34.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:34.479934+0000) 2024-02-20T00:34:35.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:35.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:35.480176+0000) 2024-02-20T00:34:36.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:36.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:36.480383+0000) 2024-02-20T00:34:37.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:37.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:37.480651+0000) 2024-02-20T00:34:38.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:38.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:38.480922+0000) 2024-02-20T00:34:39.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:39.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:39.481218+0000) 2024-02-20T00:34:40.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:40.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:40.481373+0000) 2024-02-20T00:34:41.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:41.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:41.481542+0000) 2024-02-20T00:34:42.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:42.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:42.481697+0000) 2024-02-20T00:34:43.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:43.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:43.481878+0000) 2024-02-20T00:34:44.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:44.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:44.482049+0000) 2024-02-20T00:34:45.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:45.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:45.482269+0000) 2024-02-20T00:34:46.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:46.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:46.482538+0000) 2024-02-20T00:34:47.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:47.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:47.482736+0000) 2024-02-20T00:34:48.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:48.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:48.482987+0000) 2024-02-20T00:34:49.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:49.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:49.483229+0000) 2024-02-20T00:34:50.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:50.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:50.483479+0000) 2024-02-20T00:34:51.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:51.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:51.483690+0000) 2024-02-20T00:34:52.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:52.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:52.483866+0000) 2024-02-20T00:34:53.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:53.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:53.484086+0000) 2024-02-20T00:34:54.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:54.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:54.484303+0000) 2024-02-20T00:34:55.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:55.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:55.484551+0000) 2024-02-20T00:34:56.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:56.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:56.484762+0000) 2024-02-20T00:34:57.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:57.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:57.485024+0000) 2024-02-20T00:34:58.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:58.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:58.485232+0000) 2024-02-20T00:34:59.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:34:59.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:34:59.485420+0000) 2024-02-20T00:35:00.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:00.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:00.485641+0000) 2024-02-20T00:35:01.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:01.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:01.485840+0000) 2024-02-20T00:35:02.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:02.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:02.486025+0000) 2024-02-20T00:35:03.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:03.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:03.486213+0000) 2024-02-20T00:35:04.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:04.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:04.486483+0000) 2024-02-20T00:35:05.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:05.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:05.486655+0000) 2024-02-20T00:35:06.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:06.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:06.486934+0000) 2024-02-20T00:35:07.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:07.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:07.487216+0000) 2024-02-20T00:35:08.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:08.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:08.487478+0000) 2024-02-20T00:35:09.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:09.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:09.487658+0000) 2024-02-20T00:35:10.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:10.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:10.487872+0000) 2024-02-20T00:35:11.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:11.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:11.488049+0000) 2024-02-20T00:35:12.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:12.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:12.488208+0000) 2024-02-20T00:35:13.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:13.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:13.488432+0000) 2024-02-20T00:35:14.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:14.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:14.488592+0000) 2024-02-20T00:35:15.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:15.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:15.488761+0000) 2024-02-20T00:35:16.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:16.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:16.488911+0000) 2024-02-20T00:35:17.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:17.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:17.489124+0000) 2024-02-20T00:35:18.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:18.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:18.489400+0000) 2024-02-20T00:35:19.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:19.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:19.489554+0000) 2024-02-20T00:35:20.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:20.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:20.489808+0000) 2024-02-20T00:35:21.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:21.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:21.489997+0000) 2024-02-20T00:35:22.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:22.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:22.490215+0000) 2024-02-20T00:35:23.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:23.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:23.490450+0000) 2024-02-20T00:35:24.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:24.484+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:24.490678+0000) 2024-02-20T00:35:25.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:25.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:25.490820+0000) 2024-02-20T00:35:26.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:26.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:26.491054+0000) 2024-02-20T00:35:27.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:27.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:27.491208+0000) 2024-02-20T00:35:28.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:28.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:28.491425+0000) 2024-02-20T00:35:29.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:29.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:29.491596+0000) 2024-02-20T00:35:30.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:30.488+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:30.491816+0000) 2024-02-20T00:35:31.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:31.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:31.492027+0000) 2024-02-20T00:35:32.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:32.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:32.492261+0000) 2024-02-20T00:35:33.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:33.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:33.492517+0000) 2024-02-20T00:35:34.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:34.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:34.492738+0000) 2024-02-20T00:35:35.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:35.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:35.493001+0000) 2024-02-20T00:35:36.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:36.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:36.493164+0000) 2024-02-20T00:35:37.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:37.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:37.493382+0000) 2024-02-20T00:35:38.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:38.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:38.493533+0000) 2024-02-20T00:35:39.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:39.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:39.493678+0000) 2024-02-20T00:35:40.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:40.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:40.493813+0000) 2024-02-20T00:35:41.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:41.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:41.493998+0000) 2024-02-20T00:35:42.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:42.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:42.494136+0000) 2024-02-20T00:35:43.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:43.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:43.494348+0000) 2024-02-20T00:35:44.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:44.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:44.494503+0000) 2024-02-20T00:35:45.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:45.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:45.494739+0000) 2024-02-20T00:35:46.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:46.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:46.494943+0000) 2024-02-20T00:35:47.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:47.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:47.495169+0000) 2024-02-20T00:35:48.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:48.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:48.495396+0000) 2024-02-20T00:35:49.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:49.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:49.495553+0000) 2024-02-20T00:35:50.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:50.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:50.495702+0000) 2024-02-20T00:35:51.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:51.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:51.495891+0000) 2024-02-20T00:35:52.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:52.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:52.496110+0000) 2024-02-20T00:35:53.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:53.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:53.496280+0000) 2024-02-20T00:35:54.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:54.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:54.496451+0000) 2024-02-20T00:35:55.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:55.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:55.496666+0000) 2024-02-20T00:35:56.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:56.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:56.496903+0000) 2024-02-20T00:35:57.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:57.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:57.497099+0000) 2024-02-20T00:35:58.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:58.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:58.497259+0000) 2024-02-20T00:35:59.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:35:59.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:35:59.497469+0000) 2024-02-20T00:36:00.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:00.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:00.497707+0000) 2024-02-20T00:36:01.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:01.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:01.497834+0000) 2024-02-20T00:36:02.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:02.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:02.497975+0000) 2024-02-20T00:36:03.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:03.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:03.498124+0000) 2024-02-20T00:36:04.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:04.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:04.498348+0000) 2024-02-20T00:36:05.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:05.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:05.498538+0000) 2024-02-20T00:36:06.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:06.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:06.498701+0000) 2024-02-20T00:36:07.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:07.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:07.498907+0000) 2024-02-20T00:36:08.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:08.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:08.499085+0000) 2024-02-20T00:36:09.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:09.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:09.499241+0000) 2024-02-20T00:36:10.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:10.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:10.499381+0000) 2024-02-20T00:36:11.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:11.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:11.499510+0000) 2024-02-20T00:36:12.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:12.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:12.499655+0000) 2024-02-20T00:36:13.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:13.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:13.499785+0000) 2024-02-20T00:36:14.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:14.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:14.499947+0000) 2024-02-20T00:36:15.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:15.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:15.500067+0000) 2024-02-20T00:36:16.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:16.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:16.500196+0000) 2024-02-20T00:36:17.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:17.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:17.500343+0000) 2024-02-20T00:36:18.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:18.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:18.500581+0000) 2024-02-20T00:36:19.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:19.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:19.500841+0000) 2024-02-20T00:36:20.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:20.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:20.501127+0000) 2024-02-20T00:36:21.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:21.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:21.501389+0000) 2024-02-20T00:36:22.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:22.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:22.501675+0000) 2024-02-20T00:36:23.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:23.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:23.501879+0000) 2024-02-20T00:36:24.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:24.495+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:24.502120+0000) 2024-02-20T00:36:25.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:25.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:25.502314+0000) 2024-02-20T00:36:26.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:26.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:26.502545+0000) 2024-02-20T00:36:27.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:27.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:27.502758+0000) 2024-02-20T00:36:28.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:28.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:28.502903+0000) 2024-02-20T00:36:29.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:29.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:29.503135+0000) 2024-02-20T00:36:30.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:30.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:30.503373+0000) 2024-02-20T00:36:31.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:31.499+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:31.503576+0000) 2024-02-20T00:36:32.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:32.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:32.503822+0000) 2024-02-20T00:36:33.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:33.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:33.504069+0000) 2024-02-20T00:36:34.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:34.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:34.504238+0000) 2024-02-20T00:36:35.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:35.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:35.504466+0000) 2024-02-20T00:36:36.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:36.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:36.504626+0000) 2024-02-20T00:36:37.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:37.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:37.504781+0000) 2024-02-20T00:36:38.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:38.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:38.504934+0000) 2024-02-20T00:36:39.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:39.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:39.505130+0000) 2024-02-20T00:36:40.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:40.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:40.505281+0000) 2024-02-20T00:36:41.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:41.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:41.505470+0000) 2024-02-20T00:36:42.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:42.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:42.505648+0000) 2024-02-20T00:36:43.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:43.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:43.505899+0000) 2024-02-20T00:36:44.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:44.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:44.506185+0000) 2024-02-20T00:36:45.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:45.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:45.506410+0000) 2024-02-20T00:36:46.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:46.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:46.506569+0000) 2024-02-20T00:36:47.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:47.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:47.506754+0000) 2024-02-20T00:36:48.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:48.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:48.506976+0000) 2024-02-20T00:36:49.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:49.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:49.507157+0000) 2024-02-20T00:36:50.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:50.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:50.507419+0000) 2024-02-20T00:36:51.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:51.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:51.507655+0000) 2024-02-20T00:36:52.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:52.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:52.507882+0000) 2024-02-20T00:36:53.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:53.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:53.508065+0000) 2024-02-20T00:36:54.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:54.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:54.508245+0000) 2024-02-20T00:36:55.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:55.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:55.508424+0000) 2024-02-20T00:36:56.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:56.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:56.508645+0000) 2024-02-20T00:36:57.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:57.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:57.508863+0000) 2024-02-20T00:36:58.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:58.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:58.509059+0000) 2024-02-20T00:36:59.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:36:59.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:36:59.509272+0000) 2024-02-20T00:37:00.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:00.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:00.509459+0000) 2024-02-20T00:37:01.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:01.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:01.509714+0000) 2024-02-20T00:37:02.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:02.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:02.509892+0000) 2024-02-20T00:37:03.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:03.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:03.510112+0000) 2024-02-20T00:37:04.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:04.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:04.510347+0000) 2024-02-20T00:37:05.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:05.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:05.510540+0000) 2024-02-20T00:37:06.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:06.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:06.510770+0000) 2024-02-20T00:37:07.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:07.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:07.510927+0000) 2024-02-20T00:37:08.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:08.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:08.511104+0000) 2024-02-20T00:37:09.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:09.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:09.511315+0000) 2024-02-20T00:37:10.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:10.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:10.511562+0000) 2024-02-20T00:37:11.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:11.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:11.511824+0000) 2024-02-20T00:37:12.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:12.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:12.512030+0000) 2024-02-20T00:37:13.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:13.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:13.512294+0000) 2024-02-20T00:37:14.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:14.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:14.512484+0000) 2024-02-20T00:37:15.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:15.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:15.512706+0000) 2024-02-20T00:37:16.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:16.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:16.512892+0000) 2024-02-20T00:37:17.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:17.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:17.513103+0000) 2024-02-20T00:37:18.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:18.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:18.513241+0000) 2024-02-20T00:37:19.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:19.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:19.513432+0000) 2024-02-20T00:37:20.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:20.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:20.513704+0000) 2024-02-20T00:37:21.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:21.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:21.514011+0000) 2024-02-20T00:37:22.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:22.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:22.514179+0000) 2024-02-20T00:37:23.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:23.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:23.514386+0000) 2024-02-20T00:37:24.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:24.510+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:24.514572+0000) 2024-02-20T00:37:25.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:25.514+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:25.514749+0000) 2024-02-20T00:37:26.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:26.514+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:26.514919+0000) 2024-02-20T00:37:27.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:27.514+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:27.515102+0000) 2024-02-20T00:37:28.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:28.514+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:28.515307+0000) 2024-02-20T00:37:29.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:29.514+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:29.515580+0000) 2024-02-20T00:37:30.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:30.514+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:30.515807+0000) 2024-02-20T00:37:31.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:31.514+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:31.516044+0000) 2024-02-20T00:37:32.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:32.514+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:32.516202+0000) 2024-02-20T00:37:33.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:33.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:33.516397+0000) 2024-02-20T00:37:34.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:34.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:34.516558+0000) 2024-02-20T00:37:35.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:35.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:35.516787+0000) 2024-02-20T00:37:36.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:36.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:36.516909+0000) 2024-02-20T00:37:37.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:37.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:37.517103+0000) 2024-02-20T00:37:38.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:38.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:38.517241+0000) 2024-02-20T00:37:39.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:39.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:39.517366+0000) 2024-02-20T00:37:40.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:40.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:40.517542+0000) 2024-02-20T00:37:41.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:41.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:41.517742+0000) 2024-02-20T00:37:42.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:42.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:42.517903+0000) 2024-02-20T00:37:43.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:43.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:43.518044+0000) 2024-02-20T00:37:44.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:44.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:44.518272+0000) 2024-02-20T00:37:45.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:45.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:45.518425+0000) 2024-02-20T00:37:46.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:46.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:46.518649+0000) 2024-02-20T00:37:47.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:47.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:47.518827+0000) 2024-02-20T00:37:48.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:48.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:48.519051+0000) 2024-02-20T00:37:49.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:49.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:49.519201+0000) 2024-02-20T00:37:50.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:50.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:50.519377+0000) 2024-02-20T00:37:51.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:51.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:51.519548+0000) 2024-02-20T00:37:52.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:52.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:52.519694+0000) 2024-02-20T00:37:53.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:53.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:53.519847+0000) 2024-02-20T00:37:54.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:54.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:54.519992+0000) 2024-02-20T00:37:55.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:55.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:55.520134+0000) 2024-02-20T00:37:56.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:56.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:56.520329+0000) 2024-02-20T00:37:57.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:57.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:57.520469+0000) 2024-02-20T00:37:58.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:58.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:58.520635+0000) 2024-02-20T00:37:59.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:37:59.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:37:59.520795+0000) 2024-02-20T00:38:00.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:00.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:00.520931+0000) 2024-02-20T00:38:01.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:01.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:01.521065+0000) 2024-02-20T00:38:02.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:02.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:02.521203+0000) 2024-02-20T00:38:03.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:03.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:03.521335+0000) 2024-02-20T00:38:04.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:04.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:04.521470+0000) 2024-02-20T00:38:05.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:05.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:05.521598+0000) 2024-02-20T00:38:06.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:06.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:06.521735+0000) 2024-02-20T00:38:07.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:07.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:07.521862+0000) 2024-02-20T00:38:08.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:08.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:08.522021+0000) 2024-02-20T00:38:09.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:09.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:09.522154+0000) 2024-02-20T00:38:10.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:10.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:10.522278+0000) 2024-02-20T00:38:11.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:11.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:11.522458+0000) 2024-02-20T00:38:12.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:12.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:12.522631+0000) 2024-02-20T00:38:13.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:13.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:13.522862+0000) 2024-02-20T00:38:14.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:14.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:14.523036+0000) 2024-02-20T00:38:15.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:15.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:15.523175+0000) 2024-02-20T00:38:16.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:16.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:16.523412+0000) 2024-02-20T00:38:17.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:17.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:17.523595+0000) 2024-02-20T00:38:18.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:18.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:18.523773+0000) 2024-02-20T00:38:19.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:19.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:19.523949+0000) 2024-02-20T00:38:20.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:20.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:20.524181+0000) 2024-02-20T00:38:21.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:21.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:21.524473+0000) 2024-02-20T00:38:22.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:22.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:22.524698+0000) 2024-02-20T00:38:23.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:23.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:23.524863+0000) 2024-02-20T00:38:24.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:24.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:24.525087+0000) 2024-02-20T00:38:25.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:25.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:25.525270+0000) 2024-02-20T00:38:26.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:26.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:26.525438+0000) 2024-02-20T00:38:27.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:27.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:27.525656+0000) 2024-02-20T00:38:28.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:28.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:28.525876+0000) 2024-02-20T00:38:29.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:29.525+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:29.526107+0000) 2024-02-20T00:38:30.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:30.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:30.526354+0000) 2024-02-20T00:38:31.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:31.521+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:31.526598+0000) 2024-02-20T00:38:32.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:32.525+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:32.526855+0000) 2024-02-20T00:38:33.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:33.525+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:33.527010+0000) 2024-02-20T00:38:34.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:34.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:34.527180+0000) 2024-02-20T00:38:35.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:35.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:35.527413+0000) 2024-02-20T00:38:36.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:36.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:36.527604+0000) 2024-02-20T00:38:37.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:37.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:37.527829+0000) 2024-02-20T00:38:38.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:38.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:38.528024+0000) 2024-02-20T00:38:39.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:39.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:39.528222+0000) 2024-02-20T00:38:40.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:40.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:40.528421+0000) 2024-02-20T00:38:41.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:41.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:41.528639+0000) 2024-02-20T00:38:42.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:42.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:42.528863+0000) 2024-02-20T00:38:43.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:43.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:43.529086+0000) 2024-02-20T00:38:44.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:44.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:44.529320+0000) 2024-02-20T00:38:45.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:45.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:45.529607+0000) 2024-02-20T00:38:46.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:46.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:46.529730+0000) 2024-02-20T00:38:47.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:47.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:47.530021+0000) 2024-02-20T00:38:48.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:48.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:48.530245+0000) 2024-02-20T00:38:49.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:49.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:49.530462+0000) 2024-02-20T00:38:50.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:50.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:50.530772+0000) 2024-02-20T00:38:51.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:51.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:51.531022+0000) 2024-02-20T00:38:52.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:52.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:52.531236+0000) 2024-02-20T00:38:53.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:53.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:53.531469+0000) 2024-02-20T00:38:54.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:54.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:54.531681+0000) 2024-02-20T00:38:55.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:55.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:55.531902+0000) 2024-02-20T00:38:56.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:56.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:56.532154+0000) 2024-02-20T00:38:57.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:57.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:57.532406+0000) 2024-02-20T00:38:58.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:58.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:58.532691+0000) 2024-02-20T00:38:59.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:38:59.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:38:59.532970+0000) 2024-02-20T00:39:00.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:00.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:00.533213+0000) 2024-02-20T00:39:01.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:01.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:01.533456+0000) 2024-02-20T00:39:02.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:02.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:02.533665+0000) 2024-02-20T00:39:03.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:03.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:03.533916+0000) 2024-02-20T00:39:04.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:04.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:04.534083+0000) 2024-02-20T00:39:05.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:05.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:05.534218+0000) 2024-02-20T00:39:06.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:06.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:06.534452+0000) 2024-02-20T00:39:07.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:07.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:07.534753+0000) 2024-02-20T00:39:08.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:08.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:08.535006+0000) 2024-02-20T00:39:09.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:09.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:09.535203+0000) 2024-02-20T00:39:10.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:10.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:10.535337+0000) 2024-02-20T00:39:11.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:11.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:11.535577+0000) 2024-02-20T00:39:12.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:12.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:12.535806+0000) 2024-02-20T00:39:13.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:13.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:13.536084+0000) 2024-02-20T00:39:14.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:14.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:14.536356+0000) 2024-02-20T00:39:15.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:15.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:15.536599+0000) 2024-02-20T00:39:16.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:16.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:16.536803+0000) 2024-02-20T00:39:17.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:17.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:17.536971+0000) 2024-02-20T00:39:18.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:18.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:18.537248+0000) 2024-02-20T00:39:19.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:19.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:19.537415+0000) 2024-02-20T00:39:20.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:20.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:20.537591+0000) 2024-02-20T00:39:21.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:21.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:21.537774+0000) 2024-02-20T00:39:22.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:22.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:22.537905+0000) 2024-02-20T00:39:23.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:23.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:23.538039+0000) 2024-02-20T00:39:24.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:24.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:24.538186+0000) 2024-02-20T00:39:25.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:25.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:25.538412+0000) 2024-02-20T00:39:26.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:26.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:26.538804+0000) 2024-02-20T00:39:27.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:27.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:27.538947+0000) 2024-02-20T00:39:28.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:28.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:28.539091+0000) 2024-02-20T00:39:29.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:29.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:29.539338+0000) 2024-02-20T00:39:30.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:30.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:30.539502+0000) 2024-02-20T00:39:31.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:31.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:31.539637+0000) 2024-02-20T00:39:32.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:32.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:32.539824+0000) 2024-02-20T00:39:33.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:33.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:33.540008+0000) 2024-02-20T00:39:34.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:34.536+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:34.540170+0000) 2024-02-20T00:39:35.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:35.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:35.540316+0000) 2024-02-20T00:39:36.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:36.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:36.540454+0000) 2024-02-20T00:39:37.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:37.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:37.540595+0000) 2024-02-20T00:39:38.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:38.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:38.540802+0000) 2024-02-20T00:39:39.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:39.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:39.540958+0000) 2024-02-20T00:39:40.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:40.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:40.541091+0000) 2024-02-20T00:39:41.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:41.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:41.541230+0000) 2024-02-20T00:39:42.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:42.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:42.541364+0000) 2024-02-20T00:39:43.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:43.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:43.541527+0000) 2024-02-20T00:39:44.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:44.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:44.541669+0000) 2024-02-20T00:39:45.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:45.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:45.541804+0000) 2024-02-20T00:39:46.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:46.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:46.541938+0000) 2024-02-20T00:39:47.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:47.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:47.542116+0000) 2024-02-20T00:39:48.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:48.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:48.542261+0000) 2024-02-20T00:39:49.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:49.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:49.542402+0000) 2024-02-20T00:39:50.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:50.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:50.542583+0000) 2024-02-20T00:39:51.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:51.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:51.542786+0000) 2024-02-20T00:39:52.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:52.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:52.543032+0000) 2024-02-20T00:39:53.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:53.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:53.543207+0000) 2024-02-20T00:39:54.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:54.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:54.543378+0000) 2024-02-20T00:39:55.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:55.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:55.543541+0000) 2024-02-20T00:39:56.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:56.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:56.543736+0000) 2024-02-20T00:39:57.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:57.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:57.543952+0000) 2024-02-20T00:39:58.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:58.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:58.544150+0000) 2024-02-20T00:39:59.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:39:59.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:39:59.544310+0000) 2024-02-20T00:40:00.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:00.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:00.544498+0000) 2024-02-20T00:40:01.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:01.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:01.544729+0000) 2024-02-20T00:40:02.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:02.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:02.544929+0000) 2024-02-20T00:40:03.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:03.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:03.545171+0000) 2024-02-20T00:40:04.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:04.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:04.545423+0000) 2024-02-20T00:40:05.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:05.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:05.545653+0000) 2024-02-20T00:40:06.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:06.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:06.545839+0000) 2024-02-20T00:40:07.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:07.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:07.545999+0000) 2024-02-20T00:40:08.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:08.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:08.546189+0000) 2024-02-20T00:40:09.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:09.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:09.546456+0000) 2024-02-20T00:40:10.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:10.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:10.546740+0000) 2024-02-20T00:40:11.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:11.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:11.547020+0000) 2024-02-20T00:40:12.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:12.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:12.547170+0000) 2024-02-20T00:40:13.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:13.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:13.547395+0000) 2024-02-20T00:40:14.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:14.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:14.547593+0000) 2024-02-20T00:40:15.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:15.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:15.547769+0000) 2024-02-20T00:40:16.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:16.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:16.547965+0000) 2024-02-20T00:40:17.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:17.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:17.548246+0000) 2024-02-20T00:40:18.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:18.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:18.548479+0000) 2024-02-20T00:40:19.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:19.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:19.548657+0000) 2024-02-20T00:40:20.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:20.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:20.548904+0000) 2024-02-20T00:40:21.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:21.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:21.549088+0000) 2024-02-20T00:40:22.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:22.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:22.549290+0000) 2024-02-20T00:40:23.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:23.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:23.549417+0000) 2024-02-20T00:40:24.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:24.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:24.549644+0000) 2024-02-20T00:40:25.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:25.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:25.549896+0000) 2024-02-20T00:40:26.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:26.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:26.550142+0000) 2024-02-20T00:40:27.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:27.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:27.550418+0000) 2024-02-20T00:40:28.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:28.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:28.550660+0000) 2024-02-20T00:40:29.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:29.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:29.550887+0000) 2024-02-20T00:40:30.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:30.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:30.551065+0000) 2024-02-20T00:40:31.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:31.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:31.551268+0000) 2024-02-20T00:40:32.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:32.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:32.551512+0000) 2024-02-20T00:40:33.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:33.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:33.551694+0000) 2024-02-20T00:40:34.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:34.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:34.551913+0000) 2024-02-20T00:40:35.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:35.547+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:35.552156+0000) 2024-02-20T00:40:36.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:36.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:36.552458+0000) 2024-02-20T00:40:37.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:37.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:37.552598+0000) 2024-02-20T00:40:38.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:38.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:38.552787+0000) 2024-02-20T00:40:39.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:39.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:39.553016+0000) 2024-02-20T00:40:40.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:40.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:40.553270+0000) 2024-02-20T00:40:41.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:41.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:41.553473+0000) 2024-02-20T00:40:42.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:42.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:42.553695+0000) 2024-02-20T00:40:43.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:43.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:43.553941+0000) 2024-02-20T00:40:44.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:44.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:44.554104+0000) 2024-02-20T00:40:45.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:45.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:45.554341+0000) 2024-02-20T00:40:46.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:46.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:46.554518+0000) 2024-02-20T00:40:47.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:47.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:47.554694+0000) 2024-02-20T00:40:48.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:48.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:48.554907+0000) 2024-02-20T00:40:49.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:49.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:49.555113+0000) 2024-02-20T00:40:50.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:50.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:50.555323+0000) 2024-02-20T00:40:51.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:51.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:51.555544+0000) 2024-02-20T00:40:52.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:52.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:52.555724+0000) 2024-02-20T00:40:53.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:53.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:53.555949+0000) 2024-02-20T00:40:54.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:54.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:54.556129+0000) 2024-02-20T00:40:55.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:55.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:55.556358+0000) 2024-02-20T00:40:56.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:56.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:56.556626+0000) 2024-02-20T00:40:57.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:57.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:57.556788+0000) 2024-02-20T00:40:58.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:58.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:58.557061+0000) 2024-02-20T00:40:59.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:40:59.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:40:59.557359+0000) 2024-02-20T00:41:00.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:00.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:00.557574+0000) 2024-02-20T00:41:01.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:01.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:01.557840+0000) 2024-02-20T00:41:02.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:02.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:02.558081+0000) 2024-02-20T00:41:03.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:03.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:03.558289+0000) 2024-02-20T00:41:04.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:04.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:04.558474+0000) 2024-02-20T00:41:05.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:05.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:05.558684+0000) 2024-02-20T00:41:06.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:06.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:06.558929+0000) 2024-02-20T00:41:07.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:07.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:07.559185+0000) 2024-02-20T00:41:08.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:08.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:08.559410+0000) 2024-02-20T00:41:09.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:09.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:09.559580+0000) 2024-02-20T00:41:10.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:10.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:10.559822+0000) 2024-02-20T00:41:11.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:11.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:11.560022+0000) 2024-02-20T00:41:12.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:12.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:12.560188+0000) 2024-02-20T00:41:13.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:13.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:13.560349+0000) 2024-02-20T00:41:14.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:14.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:14.560486+0000) 2024-02-20T00:41:15.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:15.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:15.560697+0000) 2024-02-20T00:41:16.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:16.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:16.560856+0000) 2024-02-20T00:41:17.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:17.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:17.561038+0000) 2024-02-20T00:41:18.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:18.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:18.561198+0000) 2024-02-20T00:41:19.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:19.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:19.561386+0000) 2024-02-20T00:41:20.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:20.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:20.561653+0000) 2024-02-20T00:41:21.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:21.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:21.561922+0000) 2024-02-20T00:41:22.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:22.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:22.562166+0000) 2024-02-20T00:41:23.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:23.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:23.562436+0000) 2024-02-20T00:41:24.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:24.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:24.562652+0000) 2024-02-20T00:41:25.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:25.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:25.562856+0000) 2024-02-20T00:41:26.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:26.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:26.563020+0000) 2024-02-20T00:41:27.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:27.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:27.563223+0000) 2024-02-20T00:41:28.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:28.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:28.563411+0000) 2024-02-20T00:41:29.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:29.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:29.563640+0000) 2024-02-20T00:41:30.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:30.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:30.563909+0000) 2024-02-20T00:41:31.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:31.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:31.564172+0000) 2024-02-20T00:41:32.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:32.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:32.564339+0000) 2024-02-20T00:41:33.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:33.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:33.564576+0000) 2024-02-20T00:41:34.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:34.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:34.564787+0000) 2024-02-20T00:41:35.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:35.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:35.565079+0000) 2024-02-20T00:41:36.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:36.562+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:36.565261+0000) 2024-02-20T00:41:37.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:37.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:37.565537+0000) 2024-02-20T00:41:38.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:38.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:38.565797+0000) 2024-02-20T00:41:39.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:39.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:39.566046+0000) 2024-02-20T00:41:40.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:40.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:40.566219+0000) 2024-02-20T00:41:41.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:41.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:41.566442+0000) 2024-02-20T00:41:42.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:42.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:42.566611+0000) 2024-02-20T00:41:43.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:43.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:43.566789+0000) 2024-02-20T00:41:44.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:44.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:44.566920+0000) 2024-02-20T00:41:45.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:45.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:45.567046+0000) 2024-02-20T00:41:46.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:46.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:46.567193+0000) 2024-02-20T00:41:47.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:47.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:47.567364+0000) 2024-02-20T00:41:48.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:48.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:48.567527+0000) 2024-02-20T00:41:49.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:49.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:49.567722+0000) 2024-02-20T00:41:50.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:50.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:50.567885+0000) 2024-02-20T00:41:51.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:51.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:51.568051+0000) 2024-02-20T00:41:52.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:52.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:52.568225+0000) 2024-02-20T00:41:53.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:53.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:53.568451+0000) 2024-02-20T00:41:54.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:54.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:54.568634+0000) 2024-02-20T00:41:55.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:55.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:55.568831+0000) 2024-02-20T00:41:56.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:56.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:56.569088+0000) 2024-02-20T00:41:57.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:57.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:57.569322+0000) 2024-02-20T00:41:58.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:58.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:58.569504+0000) 2024-02-20T00:41:59.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:41:59.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:41:59.569653+0000) 2024-02-20T00:42:00.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:00.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:00.569797+0000) 2024-02-20T00:42:01.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:01.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:01.569923+0000) 2024-02-20T00:42:02.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:02.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:02.570046+0000) 2024-02-20T00:42:03.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:03.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:03.570188+0000) 2024-02-20T00:42:04.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:04.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:04.570356+0000) 2024-02-20T00:42:05.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:05.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:05.570517+0000) 2024-02-20T00:42:06.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:06.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:06.570795+0000) 2024-02-20T00:42:07.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:07.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:07.570966+0000) 2024-02-20T00:42:08.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:08.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:08.571092+0000) 2024-02-20T00:42:09.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:09.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:09.571218+0000) 2024-02-20T00:42:10.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:10.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:10.571349+0000) 2024-02-20T00:42:11.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:11.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:11.571516+0000) 2024-02-20T00:42:12.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:12.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:12.571675+0000) 2024-02-20T00:42:13.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:13.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:13.571803+0000) 2024-02-20T00:42:14.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:14.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:14.571931+0000) 2024-02-20T00:42:15.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:15.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:15.572059+0000) 2024-02-20T00:42:16.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:16.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:16.572197+0000) 2024-02-20T00:42:17.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:17.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:17.572338+0000) 2024-02-20T00:42:18.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:18.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:18.572488+0000) 2024-02-20T00:42:19.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:19.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:19.572619+0000) 2024-02-20T00:42:20.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:20.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:20.572758+0000) 2024-02-20T00:42:21.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:21.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:21.572949+0000) 2024-02-20T00:42:22.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:22.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:22.573117+0000) 2024-02-20T00:42:23.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:23.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:23.573321+0000) 2024-02-20T00:42:24.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:24.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:24.573554+0000) 2024-02-20T00:42:25.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:25.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:25.573698+0000) 2024-02-20T00:42:26.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:26.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:26.573891+0000) 2024-02-20T00:42:27.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:27.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:27.574089+0000) 2024-02-20T00:42:28.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:28.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:28.574252+0000) 2024-02-20T00:42:29.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:29.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:29.574463+0000) 2024-02-20T00:42:30.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:30.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:30.574773+0000) 2024-02-20T00:42:31.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:31.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:31.575079+0000) 2024-02-20T00:42:32.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:32.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:32.575261+0000) 2024-02-20T00:42:33.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:33.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:33.575435+0000) 2024-02-20T00:42:34.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:34.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:34.575664+0000) 2024-02-20T00:42:35.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:35.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:35.575931+0000) 2024-02-20T00:42:36.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:36.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:36.576178+0000) 2024-02-20T00:42:37.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:37.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:37.576488+0000) 2024-02-20T00:42:38.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:38.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:38.576714+0000) 2024-02-20T00:42:39.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:39.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:39.576995+0000) 2024-02-20T00:42:40.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:40.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:40.577207+0000) 2024-02-20T00:42:41.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:41.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:41.577395+0000) 2024-02-20T00:42:42.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:42.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:42.577662+0000) 2024-02-20T00:42:43.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:43.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:43.577956+0000) 2024-02-20T00:42:44.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:44.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:44.578180+0000) 2024-02-20T00:42:45.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:45.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:45.578401+0000) 2024-02-20T00:42:46.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:46.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:46.578582+0000) 2024-02-20T00:42:47.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:47.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:47.578865+0000) 2024-02-20T00:42:48.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:48.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:48.579080+0000) 2024-02-20T00:42:49.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:49.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:49.579305+0000) 2024-02-20T00:42:50.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:50.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:50.579523+0000) 2024-02-20T00:42:51.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:51.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:51.579721+0000) 2024-02-20T00:42:52.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:52.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:52.579917+0000) 2024-02-20T00:42:53.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:53.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:53.580109+0000) 2024-02-20T00:42:54.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:54.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:54.580358+0000) 2024-02-20T00:42:55.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:55.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:55.580607+0000) 2024-02-20T00:42:56.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:56.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:56.580864+0000) 2024-02-20T00:42:57.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:57.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:57.581087+0000) 2024-02-20T00:42:58.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:58.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:58.581317+0000) 2024-02-20T00:42:59.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:42:59.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:42:59.581561+0000) 2024-02-20T00:43:00.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:00.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:00.581766+0000) 2024-02-20T00:43:01.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:01.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:01.582010+0000) 2024-02-20T00:43:02.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:02.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:02.582264+0000) 2024-02-20T00:43:03.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:03.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:03.582523+0000) 2024-02-20T00:43:04.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:04.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:04.582833+0000) 2024-02-20T00:43:05.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:05.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:05.583027+0000) 2024-02-20T00:43:06.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:06.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:06.583228+0000) 2024-02-20T00:43:07.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:07.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:07.583495+0000) 2024-02-20T00:43:08.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:08.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:08.583640+0000) 2024-02-20T00:43:09.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:09.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:09.583778+0000) 2024-02-20T00:43:10.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:10.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:10.584006+0000) 2024-02-20T00:43:11.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:11.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:11.584197+0000) 2024-02-20T00:43:12.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:12.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:12.584440+0000) 2024-02-20T00:43:13.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:13.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:13.584677+0000) 2024-02-20T00:43:14.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:14.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:14.584929+0000) 2024-02-20T00:43:15.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:15.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:15.585186+0000) 2024-02-20T00:43:16.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:16.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:16.585406+0000) 2024-02-20T00:43:17.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:17.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:17.585523+0000) 2024-02-20T00:43:18.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:18.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:18.585649+0000) 2024-02-20T00:43:19.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:19.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:19.585882+0000) 2024-02-20T00:43:20.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:20.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:20.586104+0000) 2024-02-20T00:43:21.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:21.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:21.586288+0000) 2024-02-20T00:43:22.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:22.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:22.586497+0000) 2024-02-20T00:43:23.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:23.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:23.586743+0000) 2024-02-20T00:43:24.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:24.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:24.586955+0000) 2024-02-20T00:43:25.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:25.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:25.587085+0000) 2024-02-20T00:43:26.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:26.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:26.587338+0000) 2024-02-20T00:43:27.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:27.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:27.587515+0000) 2024-02-20T00:43:28.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:28.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:28.587745+0000) 2024-02-20T00:43:29.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:29.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:29.588008+0000) 2024-02-20T00:43:30.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:30.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:30.588192+0000) 2024-02-20T00:43:31.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:31.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:31.588354+0000) 2024-02-20T00:43:32.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:32.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:32.588494+0000) 2024-02-20T00:43:33.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:33.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:33.588618+0000) 2024-02-20T00:43:34.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:34.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:34.588749+0000) 2024-02-20T00:43:35.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:35.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:35.588883+0000) 2024-02-20T00:43:36.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:36.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:36.589027+0000) 2024-02-20T00:43:37.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:37.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:37.589179+0000) 2024-02-20T00:43:38.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:38.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:38.589322+0000) 2024-02-20T00:43:39.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:39.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:39.589454+0000) 2024-02-20T00:43:40.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:40.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:40.589594+0000) 2024-02-20T00:43:41.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:41.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:41.589726+0000) 2024-02-20T00:43:42.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:42.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:42.589869+0000) 2024-02-20T00:43:43.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:43.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:43.590011+0000) 2024-02-20T00:43:44.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:44.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:44.590166+0000) 2024-02-20T00:43:45.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:45.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:45.590293+0000) 2024-02-20T00:43:46.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:46.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:46.590422+0000) 2024-02-20T00:43:47.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:47.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:47.590592+0000) 2024-02-20T00:43:48.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:48.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:48.590849+0000) 2024-02-20T00:43:49.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:49.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:49.590986+0000) 2024-02-20T00:43:50.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:50.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:50.591166+0000) 2024-02-20T00:43:51.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:51.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:51.591350+0000) 2024-02-20T00:43:52.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:52.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:52.591607+0000) 2024-02-20T00:43:53.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:53.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:53.591856+0000) 2024-02-20T00:43:54.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:54.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:54.592096+0000) 2024-02-20T00:43:55.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:55.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:55.592284+0000) 2024-02-20T00:43:56.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:56.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:56.592459+0000) 2024-02-20T00:43:57.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:57.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:57.592698+0000) 2024-02-20T00:43:58.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:58.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:58.592952+0000) 2024-02-20T00:43:59.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:43:59.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:43:59.593181+0000) 2024-02-20T00:44:00.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:00.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:00.593432+0000) 2024-02-20T00:44:01.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:01.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:01.593663+0000) 2024-02-20T00:44:02.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:02.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:02.593906+0000) 2024-02-20T00:44:03.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:03.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:03.594170+0000) 2024-02-20T00:44:04.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:04.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:04.594436+0000) 2024-02-20T00:44:05.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:05.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:05.594594+0000) 2024-02-20T00:44:06.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:06.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:06.594725+0000) 2024-02-20T00:44:07.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:07.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:07.594867+0000) 2024-02-20T00:44:08.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:08.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:08.595017+0000) 2024-02-20T00:44:09.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:09.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:09.595151+0000) 2024-02-20T00:44:10.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:10.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:10.595389+0000) 2024-02-20T00:44:11.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:11.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:11.595555+0000) 2024-02-20T00:44:12.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:12.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:12.595754+0000) 2024-02-20T00:44:13.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:13.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:13.596040+0000) 2024-02-20T00:44:14.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:14.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:14.596237+0000) 2024-02-20T00:44:15.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:15.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:15.596485+0000) 2024-02-20T00:44:16.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:16.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:16.596719+0000) 2024-02-20T00:44:17.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:17.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:17.596977+0000) 2024-02-20T00:44:18.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:18.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:18.597195+0000) 2024-02-20T00:44:19.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:19.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:19.597438+0000) 2024-02-20T00:44:20.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:20.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:20.597705+0000) 2024-02-20T00:44:21.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:21.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:21.598021+0000) 2024-02-20T00:44:22.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:22.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:22.598251+0000) 2024-02-20T00:44:23.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:23.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:23.598447+0000) 2024-02-20T00:44:24.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:24.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:24.598672+0000) 2024-02-20T00:44:25.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:25.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:25.598810+0000) 2024-02-20T00:44:26.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:26.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:26.598967+0000) 2024-02-20T00:44:27.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:27.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:27.599172+0000) 2024-02-20T00:44:28.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:28.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:28.599435+0000) 2024-02-20T00:44:29.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:29.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:29.599694+0000) 2024-02-20T00:44:30.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:30.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:30.599945+0000) 2024-02-20T00:44:31.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:31.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:31.600103+0000) 2024-02-20T00:44:32.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:32.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:32.600281+0000) 2024-02-20T00:44:33.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:33.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:33.600542+0000) 2024-02-20T00:44:34.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:34.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:34.600742+0000) 2024-02-20T00:44:35.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:35.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:35.600925+0000) 2024-02-20T00:44:36.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:36.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:36.601158+0000) 2024-02-20T00:44:37.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:37.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:37.601425+0000) 2024-02-20T00:44:38.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:38.599+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:38.601691+0000) 2024-02-20T00:44:39.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:39.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:39.601887+0000) 2024-02-20T00:44:40.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:40.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:40.602061+0000) 2024-02-20T00:44:41.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:41.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:41.602236+0000) 2024-02-20T00:44:42.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:42.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:42.602486+0000) 2024-02-20T00:44:43.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:43.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:43.602667+0000) 2024-02-20T00:44:44.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:44.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:44.602836+0000) 2024-02-20T00:44:45.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:45.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:45.603007+0000) 2024-02-20T00:44:46.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:46.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:46.603179+0000) 2024-02-20T00:44:47.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:47.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:47.603366+0000) 2024-02-20T00:44:48.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:48.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:48.603614+0000) 2024-02-20T00:44:49.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:49.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:49.603786+0000) 2024-02-20T00:44:50.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:50.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:50.603946+0000) 2024-02-20T00:44:51.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:51.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:51.604069+0000) 2024-02-20T00:44:52.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:52.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:52.604198+0000) 2024-02-20T00:44:53.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:53.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:53.604361+0000) 2024-02-20T00:44:54.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:54.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:54.604500+0000) 2024-02-20T00:44:55.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:55.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:55.604634+0000) 2024-02-20T00:44:56.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:56.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:56.604774+0000) 2024-02-20T00:44:57.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:57.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:57.604904+0000) 2024-02-20T00:44:58.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:58.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:58.605039+0000) 2024-02-20T00:44:59.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:44:59.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:44:59.605217+0000) 2024-02-20T00:45:00.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:00.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:00.605361+0000) 2024-02-20T00:45:01.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:01.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:01.605529+0000) 2024-02-20T00:45:02.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:02.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:02.605667+0000) 2024-02-20T00:45:03.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:03.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:03.605807+0000) 2024-02-20T00:45:04.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:04.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:04.605953+0000) 2024-02-20T00:45:05.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:05.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:05.606093+0000) 2024-02-20T00:45:06.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:06.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:06.606243+0000) 2024-02-20T00:45:07.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:07.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:07.606407+0000) 2024-02-20T00:45:08.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:08.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:08.606534+0000) 2024-02-20T00:45:09.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:09.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:09.606655+0000) 2024-02-20T00:45:10.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:10.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:10.606777+0000) 2024-02-20T00:45:11.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:11.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:11.606969+0000) 2024-02-20T00:45:12.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:12.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:12.607152+0000) 2024-02-20T00:45:13.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:13.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:13.607329+0000) 2024-02-20T00:45:14.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:14.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:14.607596+0000) 2024-02-20T00:45:15.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:15.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:15.607775+0000) 2024-02-20T00:45:16.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:16.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:16.607959+0000) 2024-02-20T00:45:17.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:17.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:17.608131+0000) 2024-02-20T00:45:18.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:18.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:18.608305+0000) 2024-02-20T00:45:19.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:19.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:19.608479+0000) 2024-02-20T00:45:20.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:20.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:20.608740+0000) 2024-02-20T00:45:21.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:21.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:21.608972+0000) 2024-02-20T00:45:22.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:22.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:22.609243+0000) 2024-02-20T00:45:23.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:23.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:23.609472+0000) 2024-02-20T00:45:24.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:24.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:24.609668+0000) 2024-02-20T00:45:25.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:25.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:25.609907+0000) 2024-02-20T00:45:26.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:26.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:26.610054+0000) 2024-02-20T00:45:27.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:27.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:27.610223+0000) 2024-02-20T00:45:28.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:28.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:28.610452+0000) 2024-02-20T00:45:29.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:29.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:29.610649+0000) 2024-02-20T00:45:30.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:30.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:30.610841+0000) 2024-02-20T00:45:31.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:31.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:31.610994+0000) 2024-02-20T00:45:32.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:32.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:32.611195+0000) 2024-02-20T00:45:33.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:33.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:33.611471+0000) 2024-02-20T00:45:34.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:34.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:34.611680+0000) 2024-02-20T00:45:35.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:35.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:35.611910+0000) 2024-02-20T00:45:36.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:36.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:36.612072+0000) 2024-02-20T00:45:37.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:37.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:37.612294+0000) 2024-02-20T00:45:38.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:38.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:38.612454+0000) 2024-02-20T00:45:39.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:39.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:39.612692+0000) 2024-02-20T00:45:40.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:40.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:40.612867+0000) 2024-02-20T00:45:41.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:41.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:41.613085+0000) 2024-02-20T00:45:42.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:42.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:42.613247+0000) 2024-02-20T00:45:43.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:43.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:43.613461+0000) 2024-02-20T00:45:44.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:44.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:44.613614+0000) 2024-02-20T00:45:45.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:45.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:45.613789+0000) 2024-02-20T00:45:46.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:46.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:46.613944+0000) 2024-02-20T00:45:47.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:47.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:47.614174+0000) 2024-02-20T00:45:48.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:48.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:48.614319+0000) 2024-02-20T00:45:49.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:49.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:49.614502+0000) 2024-02-20T00:45:50.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:50.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:50.614777+0000) 2024-02-20T00:45:51.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:51.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:51.615023+0000) 2024-02-20T00:45:52.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:52.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:52.615167+0000) 2024-02-20T00:45:53.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:53.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:53.615407+0000) 2024-02-20T00:45:54.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:54.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:54.615591+0000) 2024-02-20T00:45:55.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:55.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:55.615876+0000) 2024-02-20T00:45:56.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:56.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:56.616052+0000) 2024-02-20T00:45:57.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:57.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:57.616218+0000) 2024-02-20T00:45:58.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:58.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:58.616342+0000) 2024-02-20T00:45:59.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:45:59.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:45:59.616481+0000) 2024-02-20T00:46:00.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:00.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:00.616639+0000) 2024-02-20T00:46:01.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:01.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:01.616866+0000) 2024-02-20T00:46:02.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:02.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:02.617118+0000) 2024-02-20T00:46:03.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:03.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:03.617371+0000) 2024-02-20T00:46:04.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:04.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:04.617624+0000) 2024-02-20T00:46:05.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:05.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:05.617881+0000) 2024-02-20T00:46:06.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:06.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:06.618140+0000) 2024-02-20T00:46:07.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:07.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:07.618270+0000) 2024-02-20T00:46:08.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:08.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:08.618443+0000) 2024-02-20T00:46:09.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:09.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:09.618624+0000) 2024-02-20T00:46:10.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:10.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:10.618870+0000) 2024-02-20T00:46:11.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:11.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:11.619116+0000) 2024-02-20T00:46:12.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:12.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:12.619275+0000) 2024-02-20T00:46:13.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:13.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:13.619469+0000) 2024-02-20T00:46:14.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:14.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:14.619617+0000) 2024-02-20T00:46:15.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:15.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:15.619801+0000) 2024-02-20T00:46:16.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:16.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:16.620028+0000) 2024-02-20T00:46:17.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:17.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:17.620182+0000) 2024-02-20T00:46:18.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:18.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:18.620312+0000) 2024-02-20T00:46:19.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:19.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:19.620458+0000) 2024-02-20T00:46:20.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:20.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:20.620636+0000) 2024-02-20T00:46:21.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:21.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:21.620785+0000) 2024-02-20T00:46:22.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:22.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:22.620908+0000) 2024-02-20T00:46:23.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:23.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:23.621037+0000) 2024-02-20T00:46:24.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:24.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:24.621189+0000) 2024-02-20T00:46:25.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:25.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:25.621328+0000) 2024-02-20T00:46:26.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:26.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:26.621460+0000) 2024-02-20T00:46:27.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:27.621+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:27.621584+0000) 2024-02-20T00:46:28.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:28.621+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:28.621715+0000) 2024-02-20T00:46:29.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:29.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:29.621942+0000) 2024-02-20T00:46:30.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:30.621+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:30.622142+0000) 2024-02-20T00:46:31.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:31.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:31.622391+0000) 2024-02-20T00:46:32.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:32.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:32.622607+0000) 2024-02-20T00:46:33.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:33.621+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:33.622814+0000) 2024-02-20T00:46:34.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:34.621+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:34.622975+0000) 2024-02-20T00:46:35.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:35.621+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:35.623126+0000) 2024-02-20T00:46:36.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:36.621+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:36.623376+0000) 2024-02-20T00:46:37.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:37.621+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:37.623650+0000) 2024-02-20T00:46:38.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:38.621+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:38.623806+0000) 2024-02-20T00:46:39.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:39.621+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:39.623954+0000) 2024-02-20T00:46:40.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:40.621+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:40.624205+0000) 2024-02-20T00:46:41.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:41.621+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:41.624480+0000) 2024-02-20T00:46:42.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:42.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:42.624638+0000) 2024-02-20T00:46:43.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:43.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:43.624844+0000) 2024-02-20T00:46:44.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:44.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:44.625008+0000) 2024-02-20T00:46:45.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:45.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:45.625270+0000) 2024-02-20T00:46:46.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:46.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:46.625491+0000) 2024-02-20T00:46:47.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:47.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:47.625679+0000) 2024-02-20T00:46:48.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:48.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:48.625837+0000) 2024-02-20T00:46:49.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:49.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:49.625992+0000) 2024-02-20T00:46:50.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:50.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:50.626247+0000) 2024-02-20T00:46:51.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:51.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:51.626535+0000) 2024-02-20T00:46:52.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:52.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:52.626684+0000) 2024-02-20T00:46:53.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:53.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:53.626935+0000) 2024-02-20T00:46:54.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:54.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:54.627089+0000) 2024-02-20T00:46:55.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:55.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:55.627269+0000) 2024-02-20T00:46:56.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:56.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:56.627494+0000) 2024-02-20T00:46:57.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:57.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:57.627746+0000) 2024-02-20T00:46:58.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:58.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:58.627901+0000) 2024-02-20T00:46:59.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:46:59.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:46:59.628150+0000) 2024-02-20T00:47:00.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:00.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:00.628333+0000) 2024-02-20T00:47:01.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:01.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:01.628482+0000) 2024-02-20T00:47:02.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:02.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:02.628704+0000) 2024-02-20T00:47:03.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:03.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:03.628837+0000) 2024-02-20T00:47:04.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:04.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:04.629064+0000) 2024-02-20T00:47:05.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:05.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:05.629205+0000) 2024-02-20T00:47:06.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:06.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:06.629466+0000) 2024-02-20T00:47:07.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:07.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:07.629635+0000) 2024-02-20T00:47:08.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:08.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:08.629798+0000) 2024-02-20T00:47:09.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:09.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:09.629939+0000) 2024-02-20T00:47:10.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:10.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:10.630152+0000) 2024-02-20T00:47:11.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:11.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:11.630445+0000) 2024-02-20T00:47:12.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:12.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:12.630630+0000) 2024-02-20T00:47:13.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:13.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:13.630847+0000) 2024-02-20T00:47:14.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:14.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:14.630998+0000) 2024-02-20T00:47:15.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:15.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:15.631190+0000) 2024-02-20T00:47:16.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:16.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:16.631332+0000) 2024-02-20T00:47:17.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:17.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:17.631580+0000) 2024-02-20T00:47:18.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:18.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:18.631746+0000) 2024-02-20T00:47:19.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:19.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:19.631990+0000) 2024-02-20T00:47:20.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:20.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:20.632248+0000) 2024-02-20T00:47:21.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:21.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:21.632418+0000) 2024-02-20T00:47:22.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:22.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:22.632575+0000) 2024-02-20T00:47:23.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:23.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:23.632733+0000) 2024-02-20T00:47:24.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:24.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:24.632963+0000) 2024-02-20T00:47:25.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:25.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:25.633158+0000) 2024-02-20T00:47:26.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:26.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:26.633289+0000) 2024-02-20T00:47:27.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:27.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:27.633432+0000) 2024-02-20T00:47:28.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:28.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:28.633573+0000) 2024-02-20T00:47:29.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:29.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:29.633697+0000) 2024-02-20T00:47:30.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:30.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:30.633843+0000) 2024-02-20T00:47:31.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:31.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:31.633987+0000) 2024-02-20T00:47:32.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:32.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:32.634107+0000) 2024-02-20T00:47:33.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:33.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:33.634233+0000) 2024-02-20T00:47:34.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:34.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:34.634354+0000) 2024-02-20T00:47:35.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:35.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:35.634478+0000) 2024-02-20T00:47:36.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:36.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:36.634616+0000) 2024-02-20T00:47:37.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:37.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:37.634768+0000) 2024-02-20T00:47:38.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:38.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:38.634889+0000) 2024-02-20T00:47:39.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:39.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:39.635005+0000) 2024-02-20T00:47:40.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:40.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:40.635196+0000) 2024-02-20T00:47:41.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:41.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:41.635373+0000) 2024-02-20T00:47:42.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:42.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:42.635562+0000) 2024-02-20T00:47:43.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:43.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:43.635750+0000) 2024-02-20T00:47:44.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:44.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:44.635984+0000) 2024-02-20T00:47:45.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:45.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:45.636147+0000) 2024-02-20T00:47:46.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:46.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:46.636321+0000) 2024-02-20T00:47:47.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:47.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:47.636523+0000) 2024-02-20T00:47:48.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:48.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:48.636719+0000) 2024-02-20T00:47:49.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:49.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:49.636905+0000) 2024-02-20T00:47:50.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:50.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:50.637163+0000) 2024-02-20T00:47:51.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:51.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:51.637430+0000) 2024-02-20T00:47:52.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:52.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:52.637673+0000) 2024-02-20T00:47:53.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:53.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:53.637876+0000) 2024-02-20T00:47:54.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:54.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:54.638038+0000) 2024-02-20T00:47:55.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:55.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:55.638254+0000) 2024-02-20T00:47:56.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:56.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:56.638522+0000) 2024-02-20T00:47:57.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:57.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:57.638868+0000) 2024-02-20T00:47:58.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:58.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:58.639092+0000) 2024-02-20T00:47:59.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:47:59.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:47:59.639220+0000) 2024-02-20T00:48:00.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:00.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:00.639371+0000) 2024-02-20T00:48:01.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:01.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:01.639620+0000) 2024-02-20T00:48:02.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:02.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:02.639787+0000) 2024-02-20T00:48:03.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:03.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:03.639980+0000) 2024-02-20T00:48:04.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:04.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:04.640200+0000) 2024-02-20T00:48:05.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:05.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:05.640497+0000) 2024-02-20T00:48:06.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:06.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:06.640672+0000) 2024-02-20T00:48:07.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:07.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:07.640882+0000) 2024-02-20T00:48:08.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:08.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:08.641085+0000) 2024-02-20T00:48:09.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:09.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:09.641326+0000) 2024-02-20T00:48:10.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:10.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:10.641575+0000) 2024-02-20T00:48:11.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:11.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:11.641853+0000) 2024-02-20T00:48:12.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:12.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:12.642021+0000) 2024-02-20T00:48:13.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:13.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:13.642290+0000) 2024-02-20T00:48:14.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:14.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:14.642496+0000) 2024-02-20T00:48:15.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:15.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:15.642829+0000) 2024-02-20T00:48:16.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:16.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:16.642975+0000) 2024-02-20T00:48:17.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:17.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:17.643102+0000) 2024-02-20T00:48:18.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:18.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:18.643291+0000) 2024-02-20T00:48:19.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:19.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:19.643492+0000) 2024-02-20T00:48:20.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:20.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:20.643656+0000) 2024-02-20T00:48:21.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:21.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:21.643876+0000) 2024-02-20T00:48:22.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:22.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:22.644100+0000) 2024-02-20T00:48:23.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:23.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:23.644368+0000) 2024-02-20T00:48:24.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:24.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:24.644690+0000) 2024-02-20T00:48:25.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:25.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:25.644905+0000) 2024-02-20T00:48:26.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:26.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:26.645060+0000) 2024-02-20T00:48:27.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:27.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:27.645233+0000) 2024-02-20T00:48:28.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:28.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:28.645385+0000) 2024-02-20T00:48:29.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:29.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:29.645562+0000) 2024-02-20T00:48:30.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:30.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:30.645851+0000) 2024-02-20T00:48:31.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:31.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:31.646137+0000) 2024-02-20T00:48:32.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:32.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:32.646304+0000) 2024-02-20T00:48:33.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:33.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:33.646478+0000) 2024-02-20T00:48:34.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:34.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:34.646702+0000) 2024-02-20T00:48:35.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:35.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:35.646918+0000) 2024-02-20T00:48:36.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:36.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:36.647056+0000) 2024-02-20T00:48:37.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:37.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:37.647256+0000) 2024-02-20T00:48:38.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:38.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:38.647414+0000) 2024-02-20T00:48:39.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:39.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:39.647555+0000) 2024-02-20T00:48:40.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:40.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:40.647762+0000) 2024-02-20T00:48:41.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:41.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:41.647921+0000) 2024-02-20T00:48:42.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:42.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:42.648052+0000) 2024-02-20T00:48:43.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:43.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:43.648167+0000) 2024-02-20T00:48:44.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:44.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:44.648296+0000) 2024-02-20T00:48:45.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:45.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:45.648470+0000) 2024-02-20T00:48:46.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:46.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:46.648617+0000) 2024-02-20T00:48:47.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:47.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:47.648750+0000) 2024-02-20T00:48:48.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:48.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:48.649007+0000) 2024-02-20T00:48:49.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:49.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:49.649178+0000) 2024-02-20T00:48:50.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:50.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:50.649369+0000) 2024-02-20T00:48:51.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:51.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:51.649559+0000) 2024-02-20T00:48:52.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:52.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:52.649792+0000) 2024-02-20T00:48:53.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:53.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:53.649982+0000) 2024-02-20T00:48:54.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:54.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:54.650218+0000) 2024-02-20T00:48:55.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:55.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:55.650495+0000) 2024-02-20T00:48:56.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:56.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:56.650676+0000) 2024-02-20T00:48:57.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:57.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:57.650876+0000) 2024-02-20T00:48:58.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:58.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:58.651092+0000) 2024-02-20T00:48:59.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:48:59.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:48:59.651245+0000) 2024-02-20T00:49:00.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:00.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:00.651502+0000) 2024-02-20T00:49:01.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:01.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:01.651761+0000) 2024-02-20T00:49:02.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:02.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:02.651972+0000) 2024-02-20T00:49:03.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:03.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:03.652238+0000) 2024-02-20T00:49:04.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:04.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:04.652472+0000) 2024-02-20T00:49:05.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:05.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:05.652739+0000) 2024-02-20T00:49:06.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:06.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:06.652965+0000) 2024-02-20T00:49:07.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:07.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:07.653177+0000) 2024-02-20T00:49:08.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:08.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:08.653423+0000) 2024-02-20T00:49:09.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:09.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:09.653659+0000) 2024-02-20T00:49:10.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:10.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:10.653904+0000) 2024-02-20T00:49:11.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:11.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:11.654047+0000) 2024-02-20T00:49:12.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:12.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:12.654257+0000) 2024-02-20T00:49:13.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:13.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:13.654513+0000) 2024-02-20T00:49:14.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:14.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:14.654733+0000) 2024-02-20T00:49:15.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:15.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:15.654976+0000) 2024-02-20T00:49:16.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:16.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:16.655207+0000) 2024-02-20T00:49:17.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:17.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:17.655482+0000) 2024-02-20T00:49:18.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:18.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:18.655748+0000) 2024-02-20T00:49:19.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:19.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:19.655924+0000) 2024-02-20T00:49:20.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:20.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:20.656175+0000) 2024-02-20T00:49:21.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:21.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:21.656376+0000) 2024-02-20T00:49:22.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:22.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:22.656617+0000) 2024-02-20T00:49:23.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:23.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:23.656873+0000) 2024-02-20T00:49:24.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:24.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:24.657049+0000) 2024-02-20T00:49:25.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:25.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:25.657274+0000) 2024-02-20T00:49:26.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:26.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:26.657478+0000) 2024-02-20T00:49:27.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:27.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:27.657666+0000) 2024-02-20T00:49:28.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:28.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:28.657897+0000) 2024-02-20T00:49:29.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:29.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:29.658120+0000) 2024-02-20T00:49:30.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:30.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:30.658335+0000) 2024-02-20T00:49:31.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:31.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:31.658603+0000) 2024-02-20T00:49:32.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:32.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:32.658929+0000) 2024-02-20T00:49:33.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:33.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:33.659132+0000) 2024-02-20T00:49:34.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:34.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:34.659395+0000) 2024-02-20T00:49:35.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:35.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:35.659660+0000) 2024-02-20T00:49:36.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:36.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:36.659901+0000) 2024-02-20T00:49:37.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:37.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:37.660098+0000) 2024-02-20T00:49:38.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:38.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:38.660319+0000) 2024-02-20T00:49:39.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:39.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:39.660516+0000) 2024-02-20T00:49:40.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:40.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:40.660746+0000) 2024-02-20T00:49:41.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:41.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:41.661008+0000) 2024-02-20T00:49:42.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:42.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:42.661197+0000) 2024-02-20T00:49:43.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:43.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:43.661427+0000) 2024-02-20T00:49:44.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:44.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:44.661685+0000) 2024-02-20T00:49:45.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:45.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:45.661929+0000) 2024-02-20T00:49:46.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:46.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:46.662128+0000) 2024-02-20T00:49:47.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:47.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:47.662313+0000) 2024-02-20T00:49:48.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:48.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:48.662481+0000) 2024-02-20T00:49:49.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:49.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:49.662650+0000) 2024-02-20T00:49:50.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:50.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:50.662814+0000) 2024-02-20T00:49:51.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:51.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:51.662985+0000) 2024-02-20T00:49:52.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:52.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:52.663135+0000) 2024-02-20T00:49:53.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:53.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:53.663289+0000) 2024-02-20T00:49:54.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:54.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:54.663451+0000) 2024-02-20T00:49:55.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:55.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:55.663585+0000) 2024-02-20T00:49:56.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:56.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:56.663715+0000) 2024-02-20T00:49:57.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:57.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:57.663838+0000) 2024-02-20T00:49:58.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:58.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:58.663978+0000) 2024-02-20T00:49:59.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:49:59.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:49:59.664107+0000) 2024-02-20T00:50:00.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:00.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:00.664244+0000) 2024-02-20T00:50:01.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:01.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:01.664399+0000) 2024-02-20T00:50:02.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:02.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:02.664528+0000) 2024-02-20T00:50:03.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:03.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:03.664687+0000) 2024-02-20T00:50:04.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:04.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:04.664812+0000) 2024-02-20T00:50:05.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:05.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:05.664996+0000) 2024-02-20T00:50:06.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:06.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:06.665202+0000) 2024-02-20T00:50:07.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:07.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:07.665369+0000) 2024-02-20T00:50:08.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:08.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:08.665550+0000) 2024-02-20T00:50:09.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:09.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:09.665713+0000) 2024-02-20T00:50:10.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:10.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:10.665916+0000) 2024-02-20T00:50:11.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:11.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:11.666084+0000) 2024-02-20T00:50:12.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:12.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:12.666278+0000) 2024-02-20T00:50:13.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:13.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:13.666462+0000) 2024-02-20T00:50:14.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:14.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:14.666690+0000) 2024-02-20T00:50:15.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:15.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:15.666905+0000) 2024-02-20T00:50:16.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:16.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:16.667042+0000) 2024-02-20T00:50:17.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:17.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:17.667172+0000) 2024-02-20T00:50:18.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:18.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:18.667292+0000) 2024-02-20T00:50:19.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:19.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:19.667518+0000) 2024-02-20T00:50:20.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:20.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:20.667739+0000) 2024-02-20T00:50:21.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:21.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:21.667960+0000) 2024-02-20T00:50:22.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:22.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:22.668195+0000) 2024-02-20T00:50:23.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:23.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:23.668426+0000) 2024-02-20T00:50:24.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:24.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:24.668597+0000) 2024-02-20T00:50:25.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:25.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:25.668758+0000) 2024-02-20T00:50:26.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:26.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:26.668981+0000) 2024-02-20T00:50:27.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:27.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:27.669209+0000) 2024-02-20T00:50:28.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:28.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:28.669451+0000) 2024-02-20T00:50:29.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:29.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:29.669687+0000) 2024-02-20T00:50:30.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:30.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:30.669863+0000) 2024-02-20T00:50:31.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:31.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:31.670026+0000) 2024-02-20T00:50:32.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:32.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:32.670194+0000) 2024-02-20T00:50:33.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:33.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:33.670354+0000) 2024-02-20T00:50:34.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:34.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:34.670513+0000) 2024-02-20T00:50:35.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:35.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:35.670761+0000) 2024-02-20T00:50:36.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:36.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:36.670957+0000) 2024-02-20T00:50:37.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:37.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:37.671185+0000) 2024-02-20T00:50:38.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:38.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:38.671366+0000) 2024-02-20T00:50:39.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:39.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:39.671541+0000) 2024-02-20T00:50:40.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:40.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:40.671739+0000) 2024-02-20T00:50:41.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:41.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:41.671948+0000) 2024-02-20T00:50:42.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:42.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:42.672137+0000) 2024-02-20T00:50:43.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:43.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:43.672370+0000) 2024-02-20T00:50:44.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:44.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:44.672600+0000) 2024-02-20T00:50:45.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:45.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:45.672830+0000) 2024-02-20T00:50:46.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:46.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:46.673067+0000) 2024-02-20T00:50:47.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:47.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:47.673239+0000) 2024-02-20T00:50:48.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:48.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:48.673466+0000) 2024-02-20T00:50:49.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:49.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:49.673704+0000) 2024-02-20T00:50:50.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:50.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:50.673903+0000) 2024-02-20T00:50:51.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:51.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:51.674054+0000) 2024-02-20T00:50:52.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:52.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:52.674225+0000) 2024-02-20T00:50:53.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:53.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:53.674360+0000) 2024-02-20T00:50:54.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:54.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:54.674535+0000) 2024-02-20T00:50:55.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:55.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:55.674657+0000) 2024-02-20T00:50:56.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:56.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:56.674780+0000) 2024-02-20T00:50:57.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:57.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:57.674899+0000) 2024-02-20T00:50:58.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:58.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:58.675019+0000) 2024-02-20T00:50:59.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:50:59.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:50:59.675162+0000) 2024-02-20T00:51:00.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:00.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:00.675296+0000) 2024-02-20T00:51:01.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:01.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:01.675441+0000) 2024-02-20T00:51:02.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:02.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:02.675569+0000) 2024-02-20T00:51:03.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:03.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:03.675705+0000) 2024-02-20T00:51:04.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:04.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:04.675933+0000) 2024-02-20T00:51:05.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:05.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:05.676119+0000) 2024-02-20T00:51:06.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:06.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:06.676301+0000) 2024-02-20T00:51:07.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:07.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:07.676588+0000) 2024-02-20T00:51:08.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:08.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:08.676822+0000) 2024-02-20T00:51:09.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:09.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:09.677055+0000) 2024-02-20T00:51:10.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:10.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:10.677292+0000) 2024-02-20T00:51:11.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:11.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:11.677511+0000) 2024-02-20T00:51:12.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:12.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:12.677698+0000) 2024-02-20T00:51:13.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:13.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:13.677919+0000) 2024-02-20T00:51:14.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:14.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:14.678172+0000) 2024-02-20T00:51:15.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:15.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:15.678421+0000) 2024-02-20T00:51:16.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:16.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:16.678668+0000) 2024-02-20T00:51:17.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:17.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:17.678808+0000) 2024-02-20T00:51:18.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:18.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:18.678999+0000) 2024-02-20T00:51:19.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:19.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:19.679217+0000) 2024-02-20T00:51:20.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:20.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:20.679466+0000) 2024-02-20T00:51:21.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:21.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:21.679683+0000) 2024-02-20T00:51:22.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:22.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:22.679916+0000) 2024-02-20T00:51:23.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:23.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:23.680093+0000) 2024-02-20T00:51:24.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:24.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:24.680247+0000) 2024-02-20T00:51:25.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:25.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:25.680471+0000) 2024-02-20T00:51:26.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:26.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:26.680667+0000) 2024-02-20T00:51:27.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:27.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:27.680849+0000) 2024-02-20T00:51:28.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:28.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:28.681036+0000) 2024-02-20T00:51:29.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:29.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:29.681262+0000) 2024-02-20T00:51:30.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:30.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:30.681509+0000) 2024-02-20T00:51:31.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:31.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:31.681720+0000) 2024-02-20T00:51:32.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:32.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:32.681919+0000) 2024-02-20T00:51:33.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:33.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:33.682093+0000) 2024-02-20T00:51:34.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:34.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:34.682255+0000) 2024-02-20T00:51:35.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:35.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:35.682557+0000) 2024-02-20T00:51:36.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:36.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:36.682747+0000) 2024-02-20T00:51:37.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:37.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:37.682963+0000) 2024-02-20T00:51:38.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:38.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:38.683099+0000) 2024-02-20T00:51:39.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:39.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:39.683261+0000) 2024-02-20T00:51:40.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:40.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:40.683411+0000) 2024-02-20T00:51:41.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:41.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:41.683587+0000) 2024-02-20T00:51:42.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:42.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:42.683812+0000) 2024-02-20T00:51:43.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:43.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:43.683992+0000) 2024-02-20T00:51:44.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:44.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:44.684206+0000) 2024-02-20T00:51:45.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:45.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:45.684372+0000) 2024-02-20T00:51:46.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:46.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:46.684535+0000) 2024-02-20T00:51:47.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:47.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:47.684667+0000) 2024-02-20T00:51:48.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:48.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:48.684811+0000) 2024-02-20T00:51:49.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:49.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:49.684946+0000) 2024-02-20T00:51:50.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:50.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:50.685166+0000) 2024-02-20T00:51:51.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:51.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:51.685312+0000) 2024-02-20T00:51:52.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:52.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:52.685471+0000) 2024-02-20T00:51:53.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:53.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:53.685592+0000) 2024-02-20T00:51:54.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:54.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:54.685729+0000) 2024-02-20T00:51:55.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:55.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:55.685859+0000) 2024-02-20T00:51:56.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:56.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:56.686004+0000) 2024-02-20T00:51:57.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:57.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:57.686133+0000) 2024-02-20T00:51:58.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:58.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:58.686291+0000) 2024-02-20T00:51:59.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:51:59.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:51:59.686471+0000) 2024-02-20T00:52:00.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:00.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:00.686653+0000) 2024-02-20T00:52:01.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:01.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:01.686866+0000) 2024-02-20T00:52:02.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:02.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:02.687040+0000) 2024-02-20T00:52:03.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:03.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:03.687185+0000) 2024-02-20T00:52:04.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:04.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:04.687349+0000) 2024-02-20T00:52:05.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:05.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:05.687564+0000) 2024-02-20T00:52:06.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:06.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:06.687793+0000) 2024-02-20T00:52:07.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:07.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:07.687969+0000) 2024-02-20T00:52:08.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:08.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:08.688095+0000) 2024-02-20T00:52:09.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:09.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:09.688217+0000) 2024-02-20T00:52:10.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:10.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:10.688474+0000) 2024-02-20T00:52:11.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:11.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:11.688722+0000) 2024-02-20T00:52:12.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:12.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:12.688945+0000) 2024-02-20T00:52:13.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:13.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:13.689139+0000) 2024-02-20T00:52:14.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:14.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:14.689373+0000) 2024-02-20T00:52:15.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:15.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:15.689583+0000) 2024-02-20T00:52:16.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:16.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:16.689802+0000) 2024-02-20T00:52:17.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:17.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:17.690059+0000) 2024-02-20T00:52:18.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:18.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:18.690293+0000) 2024-02-20T00:52:19.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:19.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:19.690577+0000) 2024-02-20T00:52:20.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:20.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:20.690845+0000) 2024-02-20T00:52:21.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:21.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:21.691033+0000) 2024-02-20T00:52:22.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:22.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:22.691182+0000) 2024-02-20T00:52:23.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:23.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:23.691352+0000) 2024-02-20T00:52:24.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:24.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:24.691549+0000) 2024-02-20T00:52:25.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:25.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:25.691697+0000) 2024-02-20T00:52:26.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:26.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:26.691857+0000) 2024-02-20T00:52:27.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:27.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:27.692018+0000) 2024-02-20T00:52:28.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:28.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:28.692170+0000) 2024-02-20T00:52:29.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:29.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:29.692302+0000) 2024-02-20T00:52:30.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:30.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:30.692450+0000) 2024-02-20T00:52:31.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:31.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:31.692580+0000) 2024-02-20T00:52:32.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:32.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:32.692718+0000) 2024-02-20T00:52:33.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:33.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:33.692861+0000) 2024-02-20T00:52:34.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:34.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:34.692991+0000) 2024-02-20T00:52:35.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:35.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:35.693200+0000) 2024-02-20T00:52:36.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:36.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:36.693348+0000) 2024-02-20T00:52:37.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:37.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:37.693485+0000) 2024-02-20T00:52:38.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:38.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:38.693631+0000) 2024-02-20T00:52:39.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:39.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:39.693755+0000) 2024-02-20T00:52:40.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:40.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:40.693898+0000) 2024-02-20T00:52:41.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:41.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:41.694029+0000) 2024-02-20T00:52:42.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:42.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:42.694167+0000) 2024-02-20T00:52:43.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:43.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:43.694289+0000) 2024-02-20T00:52:44.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:44.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:44.694436+0000) 2024-02-20T00:52:45.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:45.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:45.694578+0000) 2024-02-20T00:52:46.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:46.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:46.694726+0000) 2024-02-20T00:52:47.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:47.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:47.694848+0000) 2024-02-20T00:52:48.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:48.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:48.695111+0000) 2024-02-20T00:52:49.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:49.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:49.695271+0000) 2024-02-20T00:52:50.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:50.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:50.695500+0000) 2024-02-20T00:52:51.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:51.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:51.695662+0000) 2024-02-20T00:52:52.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:52.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:52.695879+0000) 2024-02-20T00:52:53.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:53.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:53.696044+0000) 2024-02-20T00:52:54.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:54.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:54.696191+0000) 2024-02-20T00:52:55.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:55.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:55.696398+0000) 2024-02-20T00:52:56.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:56.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:56.696564+0000) 2024-02-20T00:52:57.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:57.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:57.696744+0000) 2024-02-20T00:52:58.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:58.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:58.696879+0000) 2024-02-20T00:52:59.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:52:59.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:52:59.697001+0000) 2024-02-20T00:53:00.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:00.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:00.697220+0000) 2024-02-20T00:53:01.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:01.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:01.697404+0000) 2024-02-20T00:53:02.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:02.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:02.697626+0000) 2024-02-20T00:53:03.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:03.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:03.697765+0000) 2024-02-20T00:53:04.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:04.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:04.697947+0000) 2024-02-20T00:53:05.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:05.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:05.698155+0000) 2024-02-20T00:53:06.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:06.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:06.698402+0000) 2024-02-20T00:53:07.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:07.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:07.698589+0000) 2024-02-20T00:53:08.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:08.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:08.698785+0000) 2024-02-20T00:53:09.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:09.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:09.698942+0000) 2024-02-20T00:53:10.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:10.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:10.699161+0000) 2024-02-20T00:53:11.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:11.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:11.699342+0000) 2024-02-20T00:53:12.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:12.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:12.699520+0000) 2024-02-20T00:53:13.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:13.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:13.699658+0000) 2024-02-20T00:53:14.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:14.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:14.699825+0000) 2024-02-20T00:53:15.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:15.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:15.699960+0000) 2024-02-20T00:53:16.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:16.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:16.700222+0000) 2024-02-20T00:53:17.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:17.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:17.700441+0000) 2024-02-20T00:53:18.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:18.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:18.700613+0000) 2024-02-20T00:53:19.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:19.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:19.700803+0000) 2024-02-20T00:53:20.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:20.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:20.701043+0000) 2024-02-20T00:53:21.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:21.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:21.701223+0000) 2024-02-20T00:53:22.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:22.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:22.701404+0000) 2024-02-20T00:53:23.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:23.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:23.701591+0000) 2024-02-20T00:53:24.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:24.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:24.701788+0000) 2024-02-20T00:53:25.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:25.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:25.701998+0000) 2024-02-20T00:53:26.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:26.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:26.702138+0000) 2024-02-20T00:53:27.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:27.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:27.702264+0000) 2024-02-20T00:53:28.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:28.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:28.702411+0000) 2024-02-20T00:53:29.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:29.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:29.702549+0000) 2024-02-20T00:53:30.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:30.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:30.702725+0000) 2024-02-20T00:53:31.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:31.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:31.702876+0000) 2024-02-20T00:53:32.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:32.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:32.703012+0000) 2024-02-20T00:53:33.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:33.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:33.703231+0000) 2024-02-20T00:53:34.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:34.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:34.703485+0000) 2024-02-20T00:53:35.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:35.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:35.703658+0000) 2024-02-20T00:53:36.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:36.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:36.703805+0000) 2024-02-20T00:53:37.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:37.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:37.703984+0000) 2024-02-20T00:53:38.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:38.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:38.704108+0000) 2024-02-20T00:53:39.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:39.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:39.704330+0000) 2024-02-20T00:53:40.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:40.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:40.704573+0000) 2024-02-20T00:53:41.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:41.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:41.704786+0000) 2024-02-20T00:53:42.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:42.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:42.704990+0000) 2024-02-20T00:53:43.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:43.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:43.705238+0000) 2024-02-20T00:53:44.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:44.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:44.705469+0000) 2024-02-20T00:53:45.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:45.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:45.705727+0000) 2024-02-20T00:53:46.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:46.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:46.705981+0000) 2024-02-20T00:53:47.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:47.701+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:47.706248+0000) 2024-02-20T00:53:48.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:48.701+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:48.706452+0000) 2024-02-20T00:53:49.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:49.701+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:49.706651+0000) 2024-02-20T00:53:50.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:50.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:50.706867+0000) 2024-02-20T00:53:51.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:51.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:51.707050+0000) 2024-02-20T00:53:52.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:52.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:52.707272+0000) 2024-02-20T00:53:53.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:53.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:53.707526+0000) 2024-02-20T00:53:54.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:54.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:54.707705+0000) 2024-02-20T00:53:55.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:55.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:55.707855+0000) 2024-02-20T00:53:56.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:56.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:56.708107+0000) 2024-02-20T00:53:57.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:57.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:57.708325+0000) 2024-02-20T00:53:58.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:58.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:58.708461+0000) 2024-02-20T00:53:59.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:53:59.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:53:59.708606+0000) 2024-02-20T00:54:00.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:00.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:00.708725+0000) 2024-02-20T00:54:01.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:01.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:01.708901+0000) 2024-02-20T00:54:02.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:02.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:02.709063+0000) 2024-02-20T00:54:03.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:03.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:03.709215+0000) 2024-02-20T00:54:04.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:04.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:04.709368+0000) 2024-02-20T00:54:05.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:05.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:05.709515+0000) 2024-02-20T00:54:06.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:06.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:06.709696+0000) 2024-02-20T00:54:07.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:07.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:07.709862+0000) 2024-02-20T00:54:08.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:08.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:08.710039+0000) 2024-02-20T00:54:09.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:09.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:09.710179+0000) 2024-02-20T00:54:10.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:10.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:10.710344+0000) 2024-02-20T00:54:11.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:11.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:11.710534+0000) 2024-02-20T00:54:12.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:12.705+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:12.710683+0000) 2024-02-20T00:54:13.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:13.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:13.710789+0000) 2024-02-20T00:54:14.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:14.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:14.710917+0000) 2024-02-20T00:54:15.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:15.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:15.711058+0000) 2024-02-20T00:54:16.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:16.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:16.711229+0000) 2024-02-20T00:54:17.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:17.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:17.711366+0000) 2024-02-20T00:54:18.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:18.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:18.711516+0000) 2024-02-20T00:54:19.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:19.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:19.711656+0000) 2024-02-20T00:54:20.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:20.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:20.711868+0000) 2024-02-20T00:54:21.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:21.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:21.712119+0000) 2024-02-20T00:54:22.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:22.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:22.712395+0000) 2024-02-20T00:54:23.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:23.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:23.712592+0000) 2024-02-20T00:54:24.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:24.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:24.712751+0000) 2024-02-20T00:54:25.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:25.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:25.713035+0000) 2024-02-20T00:54:26.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:26.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:26.713174+0000) 2024-02-20T00:54:27.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:27.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:27.713313+0000) 2024-02-20T00:54:28.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:28.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:28.713438+0000) 2024-02-20T00:54:29.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:29.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:29.713561+0000) 2024-02-20T00:54:30.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:30.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:30.713758+0000) 2024-02-20T00:54:31.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:31.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:31.713992+0000) 2024-02-20T00:54:32.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:32.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:32.714162+0000) 2024-02-20T00:54:33.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:33.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:33.714358+0000) 2024-02-20T00:54:34.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:34.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:34.714483+0000) 2024-02-20T00:54:35.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:35.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:35.714675+0000) 2024-02-20T00:54:36.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:36.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:36.714777+0000) 2024-02-20T00:54:37.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:37.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:37.714907+0000) 2024-02-20T00:54:38.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:38.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:38.715057+0000) 2024-02-20T00:54:39.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:39.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:39.715249+0000) 2024-02-20T00:54:40.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:40.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:40.715458+0000) 2024-02-20T00:54:41.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:41.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:41.715636+0000) 2024-02-20T00:54:42.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:42.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:42.715812+0000) 2024-02-20T00:54:43.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:43.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:43.715958+0000) 2024-02-20T00:54:44.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:44.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:44.716177+0000) 2024-02-20T00:54:45.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:45.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:45.716350+0000) 2024-02-20T00:54:46.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:46.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:46.716474+0000) 2024-02-20T00:54:47.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:47.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:47.716601+0000) 2024-02-20T00:54:48.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:48.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:48.716743+0000) 2024-02-20T00:54:49.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:49.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:49.716886+0000) 2024-02-20T00:54:50.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:50.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:50.717032+0000) 2024-02-20T00:54:51.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:51.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:51.717199+0000) 2024-02-20T00:54:52.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:52.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:52.717335+0000) 2024-02-20T00:54:53.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:53.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:53.717459+0000) 2024-02-20T00:54:54.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:54.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:54.717586+0000) 2024-02-20T00:54:55.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:55.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:55.717836+0000) 2024-02-20T00:54:56.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:56.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:56.717982+0000) 2024-02-20T00:54:57.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:57.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:57.718121+0000) 2024-02-20T00:54:58.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:58.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:58.718265+0000) 2024-02-20T00:54:59.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:54:59.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:54:59.718407+0000) 2024-02-20T00:55:00.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:00.712+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:00.718547+0000) 2024-02-20T00:55:01.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:01.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:01.718844+0000) 2024-02-20T00:55:02.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:02.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:02.719024+0000) 2024-02-20T00:55:03.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:03.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:03.719241+0000) 2024-02-20T00:55:04.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:04.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:04.719426+0000) 2024-02-20T00:55:05.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:05.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:05.719691+0000) 2024-02-20T00:55:06.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:06.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:06.719871+0000) 2024-02-20T00:55:07.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:07.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:07.720124+0000) 2024-02-20T00:55:08.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:08.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:08.720361+0000) 2024-02-20T00:55:09.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:09.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:09.720528+0000) 2024-02-20T00:55:10.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:10.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:10.720720+0000) 2024-02-20T00:55:11.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:11.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:11.720979+0000) 2024-02-20T00:55:12.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:12.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:12.721243+0000) 2024-02-20T00:55:13.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:13.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:13.721412+0000) 2024-02-20T00:55:14.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:14.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:14.721634+0000) 2024-02-20T00:55:15.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:15.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:15.721878+0000) 2024-02-20T00:55:16.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:16.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:16.722145+0000) 2024-02-20T00:55:17.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:17.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:17.722379+0000) 2024-02-20T00:55:18.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:18.716+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:18.722603+0000) 2024-02-20T00:55:19.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:19.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:19.722867+0000) 2024-02-20T00:55:20.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:20.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:20.723009+0000) 2024-02-20T00:55:21.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:21.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:21.723154+0000) 2024-02-20T00:55:22.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:22.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:22.723331+0000) 2024-02-20T00:55:23.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:23.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:23.723508+0000) 2024-02-20T00:55:24.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:24.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:24.723758+0000) 2024-02-20T00:55:25.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:25.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:25.723979+0000) 2024-02-20T00:55:26.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:26.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:26.724220+0000) 2024-02-20T00:55:27.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:27.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:27.724419+0000) 2024-02-20T00:55:28.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:28.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:28.724609+0000) 2024-02-20T00:55:29.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:29.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:29.724773+0000) 2024-02-20T00:55:30.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:30.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:30.724990+0000) 2024-02-20T00:55:31.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:31.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:31.725218+0000) 2024-02-20T00:55:32.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:32.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:32.725344+0000) 2024-02-20T00:55:33.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:33.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:33.725515+0000) 2024-02-20T00:55:34.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:34.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:34.725684+0000) 2024-02-20T00:55:35.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:35.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:35.725862+0000) 2024-02-20T00:55:36.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:36.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:36.726011+0000) 2024-02-20T00:55:37.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:37.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:37.726140+0000) 2024-02-20T00:55:38.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:38.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:38.726295+0000) 2024-02-20T00:55:39.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:39.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:39.726458+0000) 2024-02-20T00:55:40.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:40.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:40.726658+0000) 2024-02-20T00:55:41.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:41.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:41.726893+0000) 2024-02-20T00:55:42.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:42.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:42.727090+0000) 2024-02-20T00:55:43.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:43.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:43.727292+0000) 2024-02-20T00:55:44.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:44.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:44.727473+0000) 2024-02-20T00:55:45.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:45.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:45.727685+0000) 2024-02-20T00:55:46.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:46.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:46.727926+0000) 2024-02-20T00:55:47.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:47.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:47.728161+0000) 2024-02-20T00:55:48.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:48.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:48.728411+0000) 2024-02-20T00:55:49.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:49.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:49.728615+0000) 2024-02-20T00:55:50.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:50.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:50.728804+0000) 2024-02-20T00:55:51.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:51.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:51.729135+0000) 2024-02-20T00:55:52.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:52.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:52.729429+0000) 2024-02-20T00:55:53.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:53.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:53.729599+0000) 2024-02-20T00:55:54.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:54.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:54.729733+0000) 2024-02-20T00:55:55.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:55.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:55.729855+0000) 2024-02-20T00:55:56.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:56.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:56.729982+0000) 2024-02-20T00:55:57.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:57.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:57.730184+0000) 2024-02-20T00:55:58.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:58.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:58.730323+0000) 2024-02-20T00:55:59.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:55:59.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:55:59.730442+0000) 2024-02-20T00:56:00.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:00.723+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:00.730565+0000) 2024-02-20T00:56:01.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:01.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:01.730746+0000) 2024-02-20T00:56:02.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:02.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:02.730870+0000) 2024-02-20T00:56:03.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:03.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:03.730996+0000) 2024-02-20T00:56:04.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:04.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:04.731113+0000) 2024-02-20T00:56:05.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:05.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:05.731305+0000) 2024-02-20T00:56:06.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:06.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:06.731465+0000) 2024-02-20T00:56:07.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:07.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:07.731611+0000) 2024-02-20T00:56:08.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:08.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:08.731772+0000) 2024-02-20T00:56:09.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:09.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:09.731900+0000) 2024-02-20T00:56:10.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:10.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:10.732040+0000) 2024-02-20T00:56:11.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:11.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:11.732188+0000) 2024-02-20T00:56:12.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:12.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:12.732334+0000) 2024-02-20T00:56:13.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:13.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:13.732581+0000) 2024-02-20T00:56:14.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:14.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:14.732792+0000) 2024-02-20T00:56:15.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:15.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:15.733027+0000) 2024-02-20T00:56:16.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:16.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:16.733297+0000) 2024-02-20T00:56:17.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:17.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:17.733528+0000) 2024-02-20T00:56:18.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:18.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:18.733728+0000) 2024-02-20T00:56:19.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:19.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:19.733848+0000) 2024-02-20T00:56:20.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:20.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:20.734152+0000) 2024-02-20T00:56:21.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:21.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:21.734381+0000) 2024-02-20T00:56:22.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:22.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:22.734528+0000) 2024-02-20T00:56:23.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:23.727+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:23.734685+0000) 2024-02-20T00:56:24.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:24.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:24.734785+0000) 2024-02-20T00:56:25.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:25.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:25.734943+0000) 2024-02-20T00:56:26.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:26.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:26.735146+0000) 2024-02-20T00:56:27.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:27.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:27.735290+0000) 2024-02-20T00:56:28.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:28.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:28.735433+0000) 2024-02-20T00:56:29.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:29.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:29.735622+0000) 2024-02-20T00:56:30.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:30.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:30.735764+0000) 2024-02-20T00:56:31.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:31.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:31.735898+0000) 2024-02-20T00:56:32.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:32.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:32.736023+0000) 2024-02-20T00:56:33.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:33.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:33.736164+0000) 2024-02-20T00:56:34.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:34.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:34.736285+0000) 2024-02-20T00:56:35.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:35.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:35.736406+0000) 2024-02-20T00:56:36.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:36.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:36.736533+0000) 2024-02-20T00:56:37.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:37.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:37.736661+0000) 2024-02-20T00:56:38.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:38.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:38.736785+0000) 2024-02-20T00:56:39.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:39.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:39.737017+0000) 2024-02-20T00:56:40.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:40.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:40.737263+0000) 2024-02-20T00:56:41.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:41.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:41.737438+0000) 2024-02-20T00:56:42.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:42.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:42.737662+0000) 2024-02-20T00:56:43.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:43.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:43.737862+0000) 2024-02-20T00:56:44.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:44.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:44.738034+0000) 2024-02-20T00:56:45.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:45.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:45.738331+0000) 2024-02-20T00:56:46.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:46.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:46.738554+0000) 2024-02-20T00:56:47.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:47.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:47.738843+0000) 2024-02-20T00:56:48.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:48.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:48.738990+0000) 2024-02-20T00:56:49.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:49.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:49.739188+0000) 2024-02-20T00:56:50.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:50.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:50.739383+0000) 2024-02-20T00:56:51.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:51.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:51.739644+0000) 2024-02-20T00:56:52.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:52.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:52.739898+0000) 2024-02-20T00:56:53.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:53.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:53.740120+0000) 2024-02-20T00:56:54.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:54.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:54.740354+0000) 2024-02-20T00:56:55.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:55.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:55.740565+0000) 2024-02-20T00:56:56.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:56.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:56.740820+0000) 2024-02-20T00:56:57.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:57.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:57.741077+0000) 2024-02-20T00:56:58.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:58.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:58.741319+0000) 2024-02-20T00:56:59.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:56:59.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:56:59.741543+0000) 2024-02-20T00:57:00.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:00.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:00.741724+0000) 2024-02-20T00:57:01.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:01.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:01.741857+0000) 2024-02-20T00:57:02.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:02.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:02.742013+0000) 2024-02-20T00:57:03.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:03.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:03.742143+0000) 2024-02-20T00:57:04.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:04.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:04.742287+0000) 2024-02-20T00:57:05.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:05.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:05.742412+0000) 2024-02-20T00:57:06.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:06.734+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:06.742543+0000) 2024-02-20T00:57:07.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:07.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:07.742812+0000) 2024-02-20T00:57:08.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:08.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:08.743049+0000) 2024-02-20T00:57:09.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:09.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:09.743293+0000) 2024-02-20T00:57:10.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:10.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:10.743497+0000) 2024-02-20T00:57:11.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:11.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:11.743643+0000) 2024-02-20T00:57:12.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:12.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:12.743818+0000) 2024-02-20T00:57:13.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:13.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:13.744088+0000) 2024-02-20T00:57:14.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:14.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:14.744315+0000) 2024-02-20T00:57:15.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:15.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:15.744527+0000) 2024-02-20T00:57:16.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:16.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:16.744739+0000) 2024-02-20T00:57:17.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:17.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:17.744920+0000) 2024-02-20T00:57:18.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:18.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:18.745159+0000) 2024-02-20T00:57:19.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:19.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:19.745396+0000) 2024-02-20T00:57:20.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:20.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:20.745635+0000) 2024-02-20T00:57:21.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:21.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:21.745779+0000) 2024-02-20T00:57:22.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:22.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:22.745974+0000) 2024-02-20T00:57:23.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:23.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:23.746128+0000) 2024-02-20T00:57:24.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:24.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:24.746316+0000) 2024-02-20T00:57:25.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:25.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:25.746564+0000) 2024-02-20T00:57:26.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:26.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:26.746826+0000) 2024-02-20T00:57:27.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:27.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:27.747024+0000) 2024-02-20T00:57:28.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:28.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:28.747224+0000) 2024-02-20T00:57:29.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:29.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:29.747411+0000) 2024-02-20T00:57:30.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:30.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:30.747668+0000) 2024-02-20T00:57:31.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:31.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:31.747851+0000) 2024-02-20T00:57:32.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:32.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:32.748033+0000) 2024-02-20T00:57:33.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:33.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:33.748205+0000) 2024-02-20T00:57:34.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:34.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:34.748411+0000) 2024-02-20T00:57:35.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:35.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:35.748626+0000) 2024-02-20T00:57:36.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:36.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:36.748852+0000) 2024-02-20T00:57:37.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:37.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:37.749047+0000) 2024-02-20T00:57:38.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:38.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:38.749221+0000) 2024-02-20T00:57:39.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:39.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:39.749396+0000) 2024-02-20T00:57:40.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:40.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:40.749627+0000) 2024-02-20T00:57:41.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:41.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:41.749821+0000) 2024-02-20T00:57:42.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:42.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:42.750046+0000) 2024-02-20T00:57:43.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:43.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:43.750245+0000) 2024-02-20T00:57:44.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:44.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:44.750463+0000) 2024-02-20T00:57:45.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:45.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:45.750621+0000) 2024-02-20T00:57:46.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:46.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:46.750802+0000) 2024-02-20T00:57:47.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:47.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:47.750971+0000) 2024-02-20T00:57:48.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:48.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:48.751105+0000) 2024-02-20T00:57:49.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:49.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:49.751242+0000) 2024-02-20T00:57:50.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:50.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:50.751417+0000) 2024-02-20T00:57:51.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:51.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:51.751554+0000) 2024-02-20T00:57:52.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:52.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:52.751695+0000) 2024-02-20T00:57:53.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:53.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:53.751877+0000) 2024-02-20T00:57:54.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:54.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:54.752105+0000) 2024-02-20T00:57:55.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:55.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:55.752341+0000) 2024-02-20T00:57:56.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:56.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:56.752581+0000) 2024-02-20T00:57:57.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:57.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:57.752766+0000) 2024-02-20T00:57:58.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:58.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:58.752986+0000) 2024-02-20T00:57:59.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:57:59.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:57:59.753151+0000) 2024-02-20T00:58:00.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:00.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:00.753318+0000) 2024-02-20T00:58:01.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:01.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:01.753517+0000) 2024-02-20T00:58:02.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:02.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:02.753742+0000) 2024-02-20T00:58:03.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:03.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:03.753905+0000) 2024-02-20T00:58:04.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:04.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:04.754102+0000) 2024-02-20T00:58:05.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:05.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:05.754341+0000) 2024-02-20T00:58:06.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:06.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:06.754526+0000) 2024-02-20T00:58:07.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:07.749+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:07.754651+0000) 2024-02-20T00:58:08.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:08.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:08.754861+0000) 2024-02-20T00:58:09.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:09.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:09.755065+0000) 2024-02-20T00:58:10.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:10.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:10.755236+0000) 2024-02-20T00:58:11.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:11.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:11.755454+0000) 2024-02-20T00:58:12.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:12.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:12.755634+0000) 2024-02-20T00:58:13.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:13.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:13.755899+0000) 2024-02-20T00:58:14.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:14.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:14.756105+0000) 2024-02-20T00:58:15.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:15.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:15.756324+0000) 2024-02-20T00:58:16.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:16.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:16.756556+0000) 2024-02-20T00:58:17.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:17.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:17.756790+0000) 2024-02-20T00:58:18.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:18.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:18.757027+0000) 2024-02-20T00:58:19.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:19.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:19.757196+0000) 2024-02-20T00:58:20.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:20.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:20.757356+0000) 2024-02-20T00:58:21.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:21.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:21.757593+0000) 2024-02-20T00:58:22.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:22.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:22.757818+0000) 2024-02-20T00:58:23.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:23.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:23.758010+0000) 2024-02-20T00:58:24.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:24.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:24.758270+0000) 2024-02-20T00:58:25.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:25.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:25.758500+0000) 2024-02-20T00:58:26.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:26.753+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:26.758687+0000) 2024-02-20T00:58:27.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:27.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:27.758815+0000) 2024-02-20T00:58:28.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:28.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:28.758960+0000) 2024-02-20T00:58:29.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:29.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:29.759190+0000) 2024-02-20T00:58:30.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:30.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:30.759441+0000) 2024-02-20T00:58:31.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:31.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:31.759637+0000) 2024-02-20T00:58:32.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:32.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:32.759908+0000) 2024-02-20T00:58:33.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:33.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:33.760076+0000) 2024-02-20T00:58:34.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:34.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:34.760342+0000) 2024-02-20T00:58:35.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:35.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:35.760574+0000) 2024-02-20T00:58:36.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:36.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:36.760755+0000) 2024-02-20T00:58:37.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:37.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:37.760932+0000) 2024-02-20T00:58:38.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:38.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:38.761163+0000) 2024-02-20T00:58:39.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:39.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:39.761373+0000) 2024-02-20T00:58:40.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:40.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:40.761611+0000) 2024-02-20T00:58:41.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:41.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:41.761788+0000) 2024-02-20T00:58:42.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:42.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:42.762052+0000) 2024-02-20T00:58:43.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:43.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:43.762243+0000) 2024-02-20T00:58:44.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:44.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:44.762483+0000) 2024-02-20T00:58:45.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:45.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:45.762626+0000) 2024-02-20T00:58:46.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:46.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:46.762777+0000) 2024-02-20T00:58:47.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:47.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:47.762949+0000) 2024-02-20T00:58:48.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:48.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:48.763181+0000) 2024-02-20T00:58:49.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:49.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:49.763317+0000) 2024-02-20T00:58:50.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:50.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:50.763573+0000) 2024-02-20T00:58:51.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:51.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:51.763815+0000) 2024-02-20T00:58:52.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:52.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:52.764049+0000) 2024-02-20T00:58:53.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:53.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:53.764218+0000) 2024-02-20T00:58:54.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:54.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:54.764389+0000) 2024-02-20T00:58:55.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:55.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:55.764564+0000) 2024-02-20T00:58:56.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:56.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:56.764804+0000) 2024-02-20T00:58:57.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:57.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:57.764973+0000) 2024-02-20T00:58:58.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:58.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:58.765236+0000) 2024-02-20T00:58:59.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:58:59.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:58:59.765484+0000) 2024-02-20T00:59:00.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:00.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:00.765618+0000) 2024-02-20T00:59:01.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:01.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:01.765782+0000) 2024-02-20T00:59:02.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:02.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:02.765931+0000) 2024-02-20T00:59:03.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:03.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:03.766176+0000) 2024-02-20T00:59:04.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:04.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:04.766367+0000) 2024-02-20T00:59:05.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:05.760+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:05.766516+0000) 2024-02-20T00:59:06.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:06.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:06.766739+0000) 2024-02-20T00:59:07.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:07.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:07.766867+0000) 2024-02-20T00:59:08.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:08.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:08.767062+0000) 2024-02-20T00:59:09.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:09.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:09.767210+0000) 2024-02-20T00:59:10.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:10.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:10.767346+0000) 2024-02-20T00:59:11.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:11.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:11.767545+0000) 2024-02-20T00:59:12.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:12.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:12.767784+0000) 2024-02-20T00:59:13.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:13.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:13.768040+0000) 2024-02-20T00:59:14.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:14.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:14.768179+0000) 2024-02-20T00:59:15.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:15.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:15.768407+0000) 2024-02-20T00:59:16.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:16.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:16.768548+0000) 2024-02-20T00:59:17.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:17.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:17.768788+0000) 2024-02-20T00:59:18.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:18.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:18.769025+0000) 2024-02-20T00:59:19.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:19.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:19.769191+0000) 2024-02-20T00:59:20.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:20.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:20.769370+0000) 2024-02-20T00:59:21.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:21.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:21.769551+0000) 2024-02-20T00:59:22.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:22.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:22.769736+0000) 2024-02-20T00:59:23.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:23.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:23.769910+0000) 2024-02-20T00:59:24.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:24.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:24.770117+0000) 2024-02-20T00:59:25.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:25.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:25.770294+0000) 2024-02-20T00:59:26.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:26.764+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:26.770543+0000) 2024-02-20T00:59:27.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:27.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:27.770740+0000) 2024-02-20T00:59:28.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:28.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:28.770868+0000) 2024-02-20T00:59:29.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:29.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:29.771025+0000) 2024-02-20T00:59:30.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:30.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:30.771158+0000) 2024-02-20T00:59:31.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:31.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:31.771355+0000) 2024-02-20T00:59:32.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:32.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:32.771615+0000) 2024-02-20T00:59:33.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:33.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:33.771855+0000) 2024-02-20T00:59:34.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:34.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:34.772053+0000) 2024-02-20T00:59:35.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:35.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:35.772241+0000) 2024-02-20T00:59:36.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:36.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:36.772407+0000) 2024-02-20T00:59:37.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:37.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:37.772672+0000) 2024-02-20T00:59:38.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:38.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:38.772922+0000) 2024-02-20T00:59:39.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:39.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:39.773108+0000) 2024-02-20T00:59:40.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:40.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:40.773338+0000) 2024-02-20T00:59:41.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:41.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:41.773516+0000) 2024-02-20T00:59:42.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:42.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:42.773772+0000) 2024-02-20T00:59:43.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:43.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:43.773960+0000) 2024-02-20T00:59:44.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:44.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:44.774089+0000) 2024-02-20T00:59:45.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:45.768+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:45.774315+0000) 2024-02-20T00:59:46.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:46.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:46.774570+0000) 2024-02-20T00:59:47.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:47.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:47.774792+0000) 2024-02-20T00:59:48.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:48.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:48.774923+0000) 2024-02-20T00:59:49.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:49.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:49.775062+0000) 2024-02-20T00:59:50.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:50.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:50.775312+0000) 2024-02-20T00:59:51.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:51.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:51.775491+0000) 2024-02-20T00:59:52.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:52.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:52.775753+0000) 2024-02-20T00:59:53.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:53.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:53.775930+0000) 2024-02-20T00:59:54.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:54.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:54.776056+0000) 2024-02-20T00:59:55.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:55.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:55.776234+0000) 2024-02-20T00:59:56.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:56.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:56.776473+0000) 2024-02-20T00:59:57.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:57.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:57.776709+0000) 2024-02-20T00:59:58.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:58.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:58.776928+0000) 2024-02-20T00:59:59.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T00:59:59.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T23:59:59.777162+0000) 2024-02-20T01:00:00.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:00.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:00.777306+0000) 2024-02-20T01:00:01.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:01.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:01.777451+0000) 2024-02-20T01:00:02.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:02.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:02.777584+0000) 2024-02-20T01:00:03.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:03.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:03.777833+0000) 2024-02-20T01:00:04.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:04.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:04.778039+0000) 2024-02-20T01:00:05.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:05.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:05.778297+0000) 2024-02-20T01:00:06.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:06.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:06.778474+0000) 2024-02-20T01:00:07.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:07.771+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:07.778657+0000) 2024-02-20T01:00:08.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:08.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:08.778913+0000) 2024-02-20T01:00:09.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:09.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:09.779189+0000) 2024-02-20T01:00:10.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:10.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:10.779366+0000) 2024-02-20T01:00:11.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:11.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:11.779537+0000) 2024-02-20T01:00:12.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:12.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:12.779678+0000) 2024-02-20T01:00:13.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:13.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:13.779896+0000) 2024-02-20T01:00:14.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:14.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:14.780078+0000) 2024-02-20T01:00:15.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:15.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:15.780315+0000) 2024-02-20T01:00:16.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:16.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:16.780562+0000) 2024-02-20T01:00:17.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:17.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:17.780733+0000) 2024-02-20T01:00:18.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:18.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:18.780979+0000) 2024-02-20T01:00:19.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:19.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:19.781145+0000) 2024-02-20T01:00:20.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:20.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:20.781288+0000) 2024-02-20T01:00:21.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:21.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:21.781441+0000) 2024-02-20T01:00:22.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:22.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:22.781626+0000) 2024-02-20T01:00:23.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:23.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:23.781827+0000) 2024-02-20T01:00:24.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:24.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:24.782001+0000) 2024-02-20T01:00:25.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:25.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:25.782175+0000) 2024-02-20T01:00:26.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:26.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:26.782346+0000) 2024-02-20T01:00:27.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:27.775+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:27.782566+0000) 2024-02-20T01:00:28.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:28.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:28.782739+0000) 2024-02-20T01:00:29.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:29.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:29.783005+0000) 2024-02-20T01:00:30.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:30.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:30.783252+0000) 2024-02-20T01:00:31.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:31.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:31.783446+0000) 2024-02-20T01:00:32.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:32.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:32.783703+0000) 2024-02-20T01:00:33.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:33.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:33.783905+0000) 2024-02-20T01:00:34.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:34.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:34.784147+0000) 2024-02-20T01:00:35.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:35.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:35.784391+0000) 2024-02-20T01:00:36.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:36.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:36.784545+0000) 2024-02-20T01:00:37.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:37.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:37.784661+0000) 2024-02-20T01:00:38.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:38.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:38.784835+0000) 2024-02-20T01:00:39.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:39.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:39.785011+0000) 2024-02-20T01:00:40.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:40.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:40.785172+0000) 2024-02-20T01:00:41.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:41.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:41.785399+0000) 2024-02-20T01:00:42.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:42.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:42.785652+0000) 2024-02-20T01:00:43.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:43.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:43.785872+0000) 2024-02-20T01:00:44.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:44.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:44.786142+0000) 2024-02-20T01:00:45.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:45.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:45.786329+0000) 2024-02-20T01:00:46.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:46.779+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:46.786456+0000) 2024-02-20T01:00:47.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:47.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:47.786695+0000) 2024-02-20T01:00:48.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:48.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:48.786917+0000) 2024-02-20T01:00:49.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:49.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:49.787129+0000) 2024-02-20T01:00:50.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:50.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:50.787352+0000) 2024-02-20T01:00:51.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:51.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:51.787595+0000) 2024-02-20T01:00:52.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:52.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:52.787857+0000) 2024-02-20T01:00:53.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:53.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:53.788081+0000) 2024-02-20T01:00:54.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:54.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:54.788316+0000) 2024-02-20T01:00:55.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:55.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:55.788587+0000) 2024-02-20T01:00:56.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:56.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:56.788847+0000) 2024-02-20T01:00:57.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:57.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:57.789085+0000) 2024-02-20T01:00:58.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:58.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:58.789265+0000) 2024-02-20T01:00:59.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:00:59.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:00:59.789432+0000) 2024-02-20T01:01:00.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:00.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:00.789612+0000) 2024-02-20T01:01:01.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:01.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:01.789845+0000) 2024-02-20T01:01:02.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:02.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:02.790073+0000) 2024-02-20T01:01:03.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:03.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:03.790297+0000) 2024-02-20T01:01:04.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:04.782+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:04.790496+0000) 2024-02-20T01:01:05.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:05.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:05.790688+0000) 2024-02-20T01:01:06.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:06.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:06.790917+0000) 2024-02-20T01:01:07.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:07.786+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:07.791118+0000) 2024-02-20T01:01:08.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:08.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:08.791320+0000) 2024-02-20T01:01:09.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:09.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:09.791556+0000) 2024-02-20T01:01:10.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:10.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:10.791753+0000) 2024-02-20T01:01:11.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:11.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:11.791979+0000) 2024-02-20T01:01:12.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:12.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:12.792168+0000) 2024-02-20T01:01:13.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:13.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:13.792390+0000) 2024-02-20T01:01:14.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:14.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:14.792625+0000) 2024-02-20T01:01:15.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:15.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:15.792838+0000) 2024-02-20T01:01:16.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:16.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:16.792981+0000) 2024-02-20T01:01:17.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:17.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:17.793186+0000) 2024-02-20T01:01:18.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:18.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:18.793326+0000) 2024-02-20T01:01:19.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:19.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:19.793526+0000) 2024-02-20T01:01:20.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:20.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:20.793758+0000) 2024-02-20T01:01:21.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:21.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:21.793995+0000) 2024-02-20T01:01:22.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:22.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:22.794227+0000) 2024-02-20T01:01:23.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:23.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:23.794474+0000) 2024-02-20T01:01:24.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:24.790+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:24.794622+0000) 2024-02-20T01:01:25.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:25.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:25.794892+0000) 2024-02-20T01:01:26.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:26.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:26.795120+0000) 2024-02-20T01:01:27.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:27.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:27.795376+0000) 2024-02-20T01:01:28.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:28.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:28.795625+0000) 2024-02-20T01:01:29.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:29.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:29.795806+0000) 2024-02-20T01:01:30.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:30.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:30.796078+0000) 2024-02-20T01:01:31.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:31.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:31.796343+0000) 2024-02-20T01:01:32.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:32.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:32.796585+0000) 2024-02-20T01:01:33.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:33.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:33.796819+0000) 2024-02-20T01:01:34.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:34.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:34.797076+0000) 2024-02-20T01:01:35.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:35.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:35.797270+0000) 2024-02-20T01:01:36.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:36.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:36.797486+0000) 2024-02-20T01:01:37.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:37.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:37.797725+0000) 2024-02-20T01:01:38.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:38.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:38.797905+0000) 2024-02-20T01:01:39.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:39.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:39.798089+0000) 2024-02-20T01:01:40.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:40.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:40.798294+0000) 2024-02-20T01:01:41.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:41.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:41.798475+0000) 2024-02-20T01:01:42.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:42.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:42.798675+0000) 2024-02-20T01:01:43.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:43.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:43.798940+0000) 2024-02-20T01:01:44.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:44.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:44.799132+0000) 2024-02-20T01:01:45.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:45.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:45.799319+0000) 2024-02-20T01:01:46.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:46.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:46.799458+0000) 2024-02-20T01:01:47.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:47.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:47.799693+0000) 2024-02-20T01:01:48.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:48.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:48.799975+0000) 2024-02-20T01:01:49.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:49.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:49.800216+0000) 2024-02-20T01:01:50.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:50.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:50.800486+0000) 2024-02-20T01:01:51.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:51.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:51.800745+0000) 2024-02-20T01:01:52.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:52.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:52.800946+0000) 2024-02-20T01:01:53.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:53.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:53.801163+0000) 2024-02-20T01:01:54.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:54.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:54.801423+0000) 2024-02-20T01:01:55.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:55.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:55.801676+0000) 2024-02-20T01:01:56.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:56.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:56.801812+0000) 2024-02-20T01:01:57.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:57.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:57.801990+0000) 2024-02-20T01:01:58.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:58.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:58.802176+0000) 2024-02-20T01:01:59.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:01:59.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:01:59.802438+0000) 2024-02-20T01:02:00.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:00.797+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:00.802653+0000) 2024-02-20T01:02:01.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:01.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:01.802907+0000) 2024-02-20T01:02:02.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:02.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:02.803035+0000) 2024-02-20T01:02:03.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:03.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:03.803242+0000) 2024-02-20T01:02:04.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:04.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:04.803496+0000) 2024-02-20T01:02:05.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:05.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:05.803712+0000) 2024-02-20T01:02:06.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:06.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:06.803850+0000) 2024-02-20T01:02:07.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:07.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:07.804121+0000) 2024-02-20T01:02:08.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:08.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:08.804261+0000) 2024-02-20T01:02:09.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:09.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:09.804503+0000) 2024-02-20T01:02:10.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:10.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:10.804681+0000) 2024-02-20T01:02:11.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:11.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:11.804894+0000) 2024-02-20T01:02:12.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:12.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:12.805050+0000) 2024-02-20T01:02:13.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:13.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:13.805279+0000) 2024-02-20T01:02:14.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:14.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:14.805438+0000) 2024-02-20T01:02:15.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:15.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:15.805594+0000) 2024-02-20T01:02:16.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:16.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:16.805791+0000) 2024-02-20T01:02:17.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:17.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:17.805977+0000) 2024-02-20T01:02:18.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:18.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:18.806118+0000) 2024-02-20T01:02:19.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:19.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:19.806286+0000) 2024-02-20T01:02:20.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:20.801+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:20.806490+0000) 2024-02-20T01:02:21.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:21.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:21.806772+0000) 2024-02-20T01:02:22.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:22.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:22.806902+0000) 2024-02-20T01:02:23.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:23.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:23.807144+0000) 2024-02-20T01:02:24.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:24.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:24.807325+0000) 2024-02-20T01:02:25.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:25.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:25.807530+0000) 2024-02-20T01:02:26.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:26.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:26.807735+0000) 2024-02-20T01:02:27.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:27.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:27.808024+0000) 2024-02-20T01:02:28.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:28.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:28.808199+0000) 2024-02-20T01:02:29.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:29.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:29.808382+0000) 2024-02-20T01:02:30.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:30.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:30.808533+0000) 2024-02-20T01:02:31.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:31.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:31.808731+0000) 2024-02-20T01:02:32.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:32.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:32.808912+0000) 2024-02-20T01:02:33.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:33.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:33.809066+0000) 2024-02-20T01:02:34.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:34.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:34.809205+0000) 2024-02-20T01:02:35.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:35.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:35.809444+0000) 2024-02-20T01:02:36.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:36.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:36.809583+0000) 2024-02-20T01:02:37.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:37.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:37.809788+0000) 2024-02-20T01:02:38.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:38.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:38.809956+0000) 2024-02-20T01:02:39.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:39.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:39.810196+0000) 2024-02-20T01:02:40.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:40.805+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:40.810461+0000) 2024-02-20T01:02:41.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:41.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:41.810751+0000) 2024-02-20T01:02:42.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:42.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:42.810927+0000) 2024-02-20T01:02:43.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:43.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:43.811130+0000) 2024-02-20T01:02:44.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:44.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:44.811262+0000) 2024-02-20T01:02:45.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:45.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:45.811513+0000) 2024-02-20T01:02:46.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:46.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:46.811783+0000) 2024-02-20T01:02:47.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:47.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:47.811911+0000) 2024-02-20T01:02:48.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:48.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:48.812103+0000) 2024-02-20T01:02:49.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:49.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:49.812221+0000) 2024-02-20T01:02:50.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:50.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:50.812383+0000) 2024-02-20T01:02:51.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:51.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:51.812584+0000) 2024-02-20T01:02:52.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:52.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:52.812860+0000) 2024-02-20T01:02:53.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:53.809+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:53.813101+0000) 2024-02-20T01:02:54.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:54.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:54.813274+0000) 2024-02-20T01:02:55.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:55.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:55.813528+0000) 2024-02-20T01:02:56.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:56.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:56.813721+0000) 2024-02-20T01:02:57.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:57.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:57.813941+0000) 2024-02-20T01:02:58.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:58.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:58.814085+0000) 2024-02-20T01:02:59.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:02:59.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:02:59.814291+0000) 2024-02-20T01:03:00.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:00.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:00.814477+0000) 2024-02-20T01:03:01.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:01.808+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:01.814606+0000) 2024-02-20T01:03:02.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:02.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:02.814809+0000) 2024-02-20T01:03:03.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:03.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:03.814938+0000) 2024-02-20T01:03:04.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:04.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:04.815065+0000) 2024-02-20T01:03:05.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:05.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:05.815291+0000) 2024-02-20T01:03:06.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:06.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:06.815441+0000) 2024-02-20T01:03:07.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:07.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:07.815694+0000) 2024-02-20T01:03:08.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:08.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:08.815880+0000) 2024-02-20T01:03:09.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:09.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:09.816103+0000) 2024-02-20T01:03:10.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:10.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:10.816278+0000) 2024-02-20T01:03:11.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:11.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:11.816526+0000) 2024-02-20T01:03:12.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:12.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:12.816670+0000) 2024-02-20T01:03:13.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:13.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:13.816912+0000) 2024-02-20T01:03:14.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:14.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:14.817089+0000) 2024-02-20T01:03:15.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:15.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:15.817320+0000) 2024-02-20T01:03:16.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:16.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:16.817471+0000) 2024-02-20T01:03:17.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:17.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:17.817698+0000) 2024-02-20T01:03:18.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:18.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:18.817937+0000) 2024-02-20T01:03:19.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:19.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:19.818169+0000) 2024-02-20T01:03:20.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:20.812+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:20.818321+0000) 2024-02-20T01:03:21.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:21.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:21.818580+0000) 2024-02-20T01:03:22.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:22.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:22.818748+0000) 2024-02-20T01:03:23.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:23.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:23.818990+0000) 2024-02-20T01:03:24.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:24.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:24.819174+0000) 2024-02-20T01:03:25.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:25.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:25.819451+0000) 2024-02-20T01:03:26.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:26.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:26.819628+0000) 2024-02-20T01:03:27.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:27.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:27.819823+0000) 2024-02-20T01:03:28.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:28.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:28.820063+0000) 2024-02-20T01:03:29.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:29.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:29.820296+0000) 2024-02-20T01:03:30.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:30.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:30.820486+0000) 2024-02-20T01:03:31.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:31.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:31.820713+0000) 2024-02-20T01:03:32.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:32.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:32.820883+0000) 2024-02-20T01:03:33.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:33.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:33.821120+0000) 2024-02-20T01:03:34.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:34.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:34.821278+0000) 2024-02-20T01:03:35.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:35.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:35.821409+0000) 2024-02-20T01:03:36.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:36.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:36.821556+0000) 2024-02-20T01:03:37.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:37.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:37.821775+0000) 2024-02-20T01:03:38.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:38.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:38.821923+0000) 2024-02-20T01:03:39.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:39.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:39.822148+0000) 2024-02-20T01:03:40.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:40.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:40.822292+0000) 2024-02-20T01:03:41.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:41.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:41.822491+0000) 2024-02-20T01:03:42.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:42.816+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:42.822638+0000) 2024-02-20T01:03:43.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:43.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:43.822774+0000) 2024-02-20T01:03:44.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:44.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:44.822949+0000) 2024-02-20T01:03:45.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:45.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:45.823104+0000) 2024-02-20T01:03:46.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:46.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:46.823260+0000) 2024-02-20T01:03:47.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:47.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:47.823507+0000) 2024-02-20T01:03:48.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:48.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:48.823664+0000) 2024-02-20T01:03:49.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:49.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:49.823876+0000) 2024-02-20T01:03:50.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:50.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:50.824021+0000) 2024-02-20T01:03:51.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:51.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:51.824147+0000) 2024-02-20T01:03:52.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:52.820+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:52.824285+0000) 2024-02-20T01:03:53.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:53.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:53.824510+0000) 2024-02-20T01:03:54.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:54.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:54.824655+0000) 2024-02-20T01:03:55.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:55.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:55.824826+0000) 2024-02-20T01:03:56.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:56.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:56.825009+0000) 2024-02-20T01:03:57.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:57.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:57.825189+0000) 2024-02-20T01:03:58.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:58.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:58.825372+0000) 2024-02-20T01:03:59.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:03:59.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:03:59.825493+0000) 2024-02-20T01:04:00.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:00.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:00.825650+0000) 2024-02-20T01:04:01.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:01.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:01.825812+0000) 2024-02-20T01:04:02.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:02.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:02.825988+0000) 2024-02-20T01:04:03.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:03.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:03.826208+0000) 2024-02-20T01:04:04.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:04.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:04.826397+0000) 2024-02-20T01:04:05.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:05.819+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:05.826590+0000) 2024-02-20T01:04:06.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:06.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:06.826764+0000) 2024-02-20T01:04:07.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:07.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:07.826886+0000) 2024-02-20T01:04:08.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:08.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:08.827019+0000) 2024-02-20T01:04:09.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:09.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:09.827191+0000) 2024-02-20T01:04:10.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:10.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:10.827379+0000) 2024-02-20T01:04:11.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:11.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:11.827540+0000) 2024-02-20T01:04:12.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:12.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:12.827686+0000) 2024-02-20T01:04:13.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:13.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:13.827910+0000) 2024-02-20T01:04:14.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:14.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:14.828061+0000) 2024-02-20T01:04:15.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:15.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:15.828182+0000) 2024-02-20T01:04:16.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:16.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:16.828338+0000) 2024-02-20T01:04:17.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:17.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:17.828555+0000) 2024-02-20T01:04:18.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:18.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:18.828696+0000) 2024-02-20T01:04:19.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:19.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:19.828964+0000) 2024-02-20T01:04:20.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:20.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:20.829109+0000) 2024-02-20T01:04:21.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:21.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:21.829383+0000) 2024-02-20T01:04:22.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:22.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:22.829525+0000) 2024-02-20T01:04:23.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:23.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:23.829670+0000) 2024-02-20T01:04:24.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:24.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:24.829815+0000) 2024-02-20T01:04:25.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:25.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:25.829965+0000) 2024-02-20T01:04:26.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:26.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:26.830112+0000) 2024-02-20T01:04:27.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:27.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:27.830288+0000) 2024-02-20T01:04:28.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:28.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:28.830436+0000) 2024-02-20T01:04:29.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:29.823+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:29.830673+0000) 2024-02-20T01:04:30.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:30.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:30.830776+0000) 2024-02-20T01:04:31.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:31.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:31.830950+0000) 2024-02-20T01:04:32.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:32.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:32.831080+0000) 2024-02-20T01:04:33.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:33.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:33.831251+0000) 2024-02-20T01:04:34.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:34.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:34.831412+0000) 2024-02-20T01:04:35.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:35.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:35.831632+0000) 2024-02-20T01:04:36.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:36.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:36.831816+0000) 2024-02-20T01:04:37.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:37.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:37.832009+0000) 2024-02-20T01:04:38.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:38.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:38.832207+0000) 2024-02-20T01:04:39.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:39.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:39.832384+0000) 2024-02-20T01:04:40.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:40.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:40.832535+0000) 2024-02-20T01:04:41.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:41.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:41.832684+0000) 2024-02-20T01:04:42.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:42.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:42.832830+0000) 2024-02-20T01:04:43.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:43.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:43.833062+0000) 2024-02-20T01:04:44.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:44.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:44.833249+0000) 2024-02-20T01:04:45.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:45.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:45.833446+0000) 2024-02-20T01:04:46.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:46.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:46.833641+0000) 2024-02-20T01:04:47.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:47.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:47.833855+0000) 2024-02-20T01:04:48.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:48.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:48.833999+0000) 2024-02-20T01:04:49.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:49.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:49.834135+0000) 2024-02-20T01:04:50.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:50.827+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:50.834333+0000) 2024-02-20T01:04:51.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:51.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:51.834601+0000) 2024-02-20T01:04:52.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:52.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:52.834801+0000) 2024-02-20T01:04:53.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:53.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:53.835028+0000) 2024-02-20T01:04:54.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:54.831+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:54.835183+0000) 2024-02-20T01:04:55.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:55.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:55.835356+0000) 2024-02-20T01:04:56.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:56.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:56.835549+0000) 2024-02-20T01:04:57.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:57.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:57.835820+0000) 2024-02-20T01:04:58.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:58.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:58.836026+0000) 2024-02-20T01:04:59.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:04:59.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:04:59.836202+0000) 2024-02-20T01:05:00.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:00.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:00.836340+0000) 2024-02-20T01:05:01.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:01.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:01.836611+0000) 2024-02-20T01:05:02.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:02.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:02.836804+0000) 2024-02-20T01:05:03.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:03.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:03.837054+0000) 2024-02-20T01:05:04.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:04.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:04.837229+0000) 2024-02-20T01:05:05.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:05.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:05.837503+0000) 2024-02-20T01:05:06.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:06.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:06.837639+0000) 2024-02-20T01:05:07.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:07.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:07.837844+0000) 2024-02-20T01:05:08.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:08.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:08.837988+0000) 2024-02-20T01:05:09.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:09.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:09.838219+0000) 2024-02-20T01:05:10.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:10.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:10.838372+0000) 2024-02-20T01:05:11.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:11.830+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:11.838605+0000) 2024-02-20T01:05:12.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:12.834+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:12.838874+0000) 2024-02-20T01:05:13.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:13.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:13.839119+0000) 2024-02-20T01:05:14.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:14.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:14.839283+0000) 2024-02-20T01:05:15.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:15.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:15.839533+0000) 2024-02-20T01:05:16.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:16.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:16.839767+0000) 2024-02-20T01:05:17.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:17.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:17.840030+0000) 2024-02-20T01:05:18.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:18.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:18.840226+0000) 2024-02-20T01:05:19.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:19.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:19.840501+0000) 2024-02-20T01:05:20.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:20.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:20.840684+0000) 2024-02-20T01:05:21.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:21.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:21.840950+0000) 2024-02-20T01:05:22.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:22.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:22.841093+0000) 2024-02-20T01:05:23.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:23.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:23.841324+0000) 2024-02-20T01:05:24.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:24.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:24.841513+0000) 2024-02-20T01:05:25.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:25.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:25.841732+0000) 2024-02-20T01:05:26.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:26.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:26.841876+0000) 2024-02-20T01:05:27.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:27.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:27.842059+0000) 2024-02-20T01:05:28.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:28.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:28.842226+0000) 2024-02-20T01:05:29.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:29.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:29.842433+0000) 2024-02-20T01:05:30.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:30.838+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:30.842558+0000) 2024-02-20T01:05:31.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:31.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:31.842822+0000) 2024-02-20T01:05:32.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:32.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:32.843062+0000) 2024-02-20T01:05:33.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:33.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:33.843249+0000) 2024-02-20T01:05:34.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:34.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:34.843510+0000) 2024-02-20T01:05:35.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:35.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:35.843742+0000) 2024-02-20T01:05:36.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:36.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:36.843940+0000) 2024-02-20T01:05:37.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:37.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:37.844102+0000) 2024-02-20T01:05:38.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:38.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:38.844324+0000) 2024-02-20T01:05:39.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:39.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:39.844579+0000) 2024-02-20T01:05:40.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:40.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:40.844823+0000) 2024-02-20T01:05:41.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:41.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:41.845092+0000) 2024-02-20T01:05:42.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:42.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:42.845325+0000) 2024-02-20T01:05:43.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:43.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:43.845594+0000) 2024-02-20T01:05:44.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:44.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:44.845817+0000) 2024-02-20T01:05:45.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:45.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:45.846043+0000) 2024-02-20T01:05:46.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:46.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:46.846285+0000) 2024-02-20T01:05:47.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:47.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:47.846506+0000) 2024-02-20T01:05:48.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:48.842+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:48.846694+0000) 2024-02-20T01:05:49.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:49.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:49.846880+0000) 2024-02-20T01:05:50.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:50.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:50.847022+0000) 2024-02-20T01:05:51.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:51.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:51.847163+0000) 2024-02-20T01:05:52.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:52.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:52.847334+0000) 2024-02-20T01:05:53.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:53.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:53.847602+0000) 2024-02-20T01:05:54.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:54.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:54.847847+0000) 2024-02-20T01:05:55.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:55.846+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:55.848102+0000) 2024-02-20T01:05:56.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:56.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:56.848378+0000) 2024-02-20T01:05:57.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:57.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:57.848604+0000) 2024-02-20T01:05:58.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:58.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:58.848858+0000) 2024-02-20T01:05:59.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:05:59.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:05:59.849040+0000) 2024-02-20T01:06:00.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:00.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:00.849209+0000) 2024-02-20T01:06:01.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:01.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:01.849333+0000) 2024-02-20T01:06:02.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:02.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:02.849532+0000) 2024-02-20T01:06:03.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:03.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:03.849751+0000) 2024-02-20T01:06:04.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:04.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:04.849946+0000) 2024-02-20T01:06:05.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:05.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:05.850115+0000) 2024-02-20T01:06:06.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:06.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:06.850275+0000) 2024-02-20T01:06:07.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:07.845+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:07.850509+0000) 2024-02-20T01:06:08.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:08.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:08.850801+0000) 2024-02-20T01:06:09.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:09.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:09.851052+0000) 2024-02-20T01:06:10.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:10.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:10.851168+0000) 2024-02-20T01:06:11.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:11.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:11.851463+0000) 2024-02-20T01:06:12.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:12.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:12.851754+0000) 2024-02-20T01:06:13.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:13.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:13.851997+0000) 2024-02-20T01:06:14.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:14.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:14.852274+0000) 2024-02-20T01:06:15.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:15.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:15.852552+0000) 2024-02-20T01:06:16.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:16.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:16.852774+0000) 2024-02-20T01:06:17.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:17.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:17.853036+0000) 2024-02-20T01:06:18.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:18.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:18.853230+0000) 2024-02-20T01:06:19.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:19.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:19.853483+0000) 2024-02-20T01:06:20.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:20.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:20.853775+0000) 2024-02-20T01:06:21.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:21.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:21.854036+0000) 2024-02-20T01:06:22.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:22.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:22.854307+0000) 2024-02-20T01:06:23.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:23.849+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:23.854630+0000) 2024-02-20T01:06:24.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:24.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:24.854913+0000) 2024-02-20T01:06:25.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:25.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:25.855196+0000) 2024-02-20T01:06:26.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:26.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:26.855439+0000) 2024-02-20T01:06:27.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:27.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:27.855594+0000) 2024-02-20T01:06:28.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:28.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:28.855860+0000) 2024-02-20T01:06:29.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:29.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:29.856115+0000) 2024-02-20T01:06:30.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:30.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:30.856310+0000) 2024-02-20T01:06:31.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:31.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:31.856600+0000) 2024-02-20T01:06:32.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:32.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:32.856851+0000) 2024-02-20T01:06:33.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:33.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:33.857126+0000) 2024-02-20T01:06:34.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:34.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:34.857420+0000) 2024-02-20T01:06:35.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:35.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:35.857706+0000) 2024-02-20T01:06:36.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:36.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:36.857890+0000) 2024-02-20T01:06:37.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:37.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:37.858125+0000) 2024-02-20T01:06:38.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:38.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:38.858396+0000) 2024-02-20T01:06:39.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:39.853+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:39.858679+0000) 2024-02-20T01:06:40.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:40.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:40.858966+0000) 2024-02-20T01:06:41.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:41.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:41.859228+0000) 2024-02-20T01:06:42.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:42.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:42.859529+0000) 2024-02-20T01:06:43.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:43.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:43.859741+0000) 2024-02-20T01:06:44.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:44.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:44.859997+0000) 2024-02-20T01:06:45.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:45.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:45.860190+0000) 2024-02-20T01:06:46.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:46.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:46.860398+0000) 2024-02-20T01:06:47.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:47.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:47.860626+0000) 2024-02-20T01:06:48.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:48.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:48.860798+0000) 2024-02-20T01:06:49.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:49.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:49.861001+0000) 2024-02-20T01:06:50.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:50.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:50.861257+0000) 2024-02-20T01:06:51.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:51.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:51.861565+0000) 2024-02-20T01:06:52.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:52.861+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:52.861813+0000) 2024-02-20T01:06:53.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:53.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:53.862081+0000) 2024-02-20T01:06:54.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:54.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:54.862361+0000) 2024-02-20T01:06:55.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:55.857+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:55.862669+0000) 2024-02-20T01:06:56.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:56.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:56.862906+0000) 2024-02-20T01:06:57.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:57.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:57.863081+0000) 2024-02-20T01:06:58.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:58.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:58.863288+0000) 2024-02-20T01:06:59.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:06:59.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:06:59.863584+0000) 2024-02-20T01:07:00.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:00.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:00.863860+0000) 2024-02-20T01:07:01.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:01.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:01.864145+0000) 2024-02-20T01:07:02.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:02.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:02.864381+0000) 2024-02-20T01:07:03.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:03.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:03.864616+0000) 2024-02-20T01:07:04.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:04.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:04.864875+0000) 2024-02-20T01:07:05.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:05.864+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:05.865194+0000) 2024-02-20T01:07:06.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:06.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:06.865408+0000) 2024-02-20T01:07:07.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:07.864+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:07.865667+0000) 2024-02-20T01:07:08.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:08.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:08.865880+0000) 2024-02-20T01:07:09.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:09.864+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:09.866188+0000) 2024-02-20T01:07:10.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:10.860+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:10.866508+0000) 2024-02-20T01:07:11.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:11.864+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:11.866849+0000) 2024-02-20T01:07:12.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:12.864+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:12.867121+0000) 2024-02-20T01:07:13.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:13.864+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:13.867407+0000) 2024-02-20T01:07:14.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:14.864+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:14.867705+0000) 2024-02-20T01:07:15.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:15.864+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:15.867978+0000) 2024-02-20T01:07:16.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:16.864+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:16.868289+0000) 2024-02-20T01:07:17.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:17.864+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:17.868611+0000) 2024-02-20T01:07:18.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:18.864+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:18.868898+0000) 2024-02-20T01:07:19.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:19.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:19.869194+0000) 2024-02-20T01:07:20.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:20.864+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:20.869457+0000) 2024-02-20T01:07:21.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:21.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:21.869743+0000) 2024-02-20T01:07:22.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:22.864+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:22.870046+0000) 2024-02-20T01:07:23.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:23.864+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:23.870290+0000) 2024-02-20T01:07:24.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:24.864+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:24.870543+0000) 2024-02-20T01:07:25.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:25.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:25.870865+0000) 2024-02-20T01:07:26.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:26.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:26.871081+0000) 2024-02-20T01:07:27.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:27.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:27.871214+0000) 2024-02-20T01:07:28.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:28.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:28.871470+0000) 2024-02-20T01:07:29.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:29.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:29.871751+0000) 2024-02-20T01:07:30.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:30.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:30.872032+0000) 2024-02-20T01:07:31.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:31.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:31.872277+0000) 2024-02-20T01:07:32.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:32.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:32.872467+0000) 2024-02-20T01:07:33.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:33.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:33.872660+0000) 2024-02-20T01:07:34.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:34.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:34.872880+0000) 2024-02-20T01:07:35.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:35.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:35.873079+0000) 2024-02-20T01:07:36.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:36.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:36.873420+0000) 2024-02-20T01:07:37.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:37.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:37.873650+0000) 2024-02-20T01:07:38.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:38.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:38.873920+0000) 2024-02-20T01:07:39.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:39.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:39.874206+0000) 2024-02-20T01:07:40.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:40.868+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:40.874481+0000) 2024-02-20T01:07:41.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:41.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:41.874683+0000) 2024-02-20T01:07:42.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:42.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:42.874808+0000) 2024-02-20T01:07:43.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:43.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:43.874986+0000) 2024-02-20T01:07:44.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:44.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:44.875182+0000) 2024-02-20T01:07:45.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:45.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:45.875373+0000) 2024-02-20T01:07:46.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:46.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:46.875600+0000) 2024-02-20T01:07:47.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:47.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:47.875802+0000) 2024-02-20T01:07:48.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:48.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:48.876062+0000) 2024-02-20T01:07:49.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:49.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:49.876289+0000) 2024-02-20T01:07:50.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:50.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:50.876441+0000) 2024-02-20T01:07:51.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:51.876+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:51.876674+0000) 2024-02-20T01:07:52.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:52.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:52.876873+0000) 2024-02-20T01:07:53.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:53.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:53.877118+0000) 2024-02-20T01:07:54.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:54.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:54.877369+0000) 2024-02-20T01:07:55.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:55.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:55.877618+0000) 2024-02-20T01:07:56.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:56.872+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:56.877859+0000) 2024-02-20T01:07:57.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:57.871+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:57.878099+0000) 2024-02-20T01:07:58.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:58.871+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:58.878229+0000) 2024-02-20T01:07:59.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:07:59.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:07:59.878456+0000) 2024-02-20T01:08:00.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:00.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:00.878747+0000) 2024-02-20T01:08:01.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:01.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:01.878894+0000) 2024-02-20T01:08:02.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:02.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:02.879054+0000) 2024-02-20T01:08:03.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:03.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:03.879297+0000) 2024-02-20T01:08:04.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:04.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:04.879538+0000) 2024-02-20T01:08:05.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:05.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:05.879783+0000) 2024-02-20T01:08:06.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:06.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:06.879964+0000) 2024-02-20T01:08:07.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:07.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:07.880156+0000) 2024-02-20T01:08:08.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:08.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:08.880393+0000) 2024-02-20T01:08:09.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:09.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:09.880632+0000) 2024-02-20T01:08:10.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:10.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:10.880869+0000) 2024-02-20T01:08:11.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:11.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:11.881110+0000) 2024-02-20T01:08:12.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:12.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:12.881335+0000) 2024-02-20T01:08:13.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:13.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:13.881577+0000) 2024-02-20T01:08:14.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:14.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:14.881768+0000) 2024-02-20T01:08:15.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:15.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:15.882006+0000) 2024-02-20T01:08:16.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:16.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:16.882167+0000) 2024-02-20T01:08:17.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:17.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:17.882416+0000) 2024-02-20T01:08:18.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:18.875+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:18.882640+0000) 2024-02-20T01:08:19.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:19.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:19.882809+0000) 2024-02-20T01:08:20.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:20.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:20.883056+0000) 2024-02-20T01:08:21.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:21.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:21.883300+0000) 2024-02-20T01:08:22.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:22.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:22.883479+0000) 2024-02-20T01:08:23.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:23.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:23.883722+0000) 2024-02-20T01:08:24.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:24.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:24.883961+0000) 2024-02-20T01:08:25.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:25.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:25.884194+0000) 2024-02-20T01:08:26.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:26.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:26.884441+0000) 2024-02-20T01:08:27.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:27.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:27.884697+0000) 2024-02-20T01:08:28.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:28.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:28.884864+0000) 2024-02-20T01:08:29.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:29.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:29.885171+0000) 2024-02-20T01:08:30.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:30.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:30.885399+0000) 2024-02-20T01:08:31.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:31.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:31.885653+0000) 2024-02-20T01:08:32.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:32.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:32.885810+0000) 2024-02-20T01:08:33.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:33.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:33.885954+0000) 2024-02-20T01:08:34.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:34.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:34.886113+0000) 2024-02-20T01:08:35.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:35.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:35.886352+0000) 2024-02-20T01:08:36.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:36.879+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:36.886608+0000) 2024-02-20T01:08:37.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:37.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:37.886875+0000) 2024-02-20T01:08:38.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:38.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:38.887055+0000) 2024-02-20T01:08:39.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:39.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:39.887224+0000) 2024-02-20T01:08:40.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:40.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:40.887482+0000) 2024-02-20T01:08:41.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:41.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:41.887770+0000) 2024-02-20T01:08:42.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:42.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:42.887959+0000) 2024-02-20T01:08:43.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:43.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:43.888165+0000) 2024-02-20T01:08:44.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:44.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:44.888399+0000) 2024-02-20T01:08:45.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:45.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:45.888559+0000) 2024-02-20T01:08:46.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:46.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:46.888779+0000) 2024-02-20T01:08:47.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:47.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:47.888956+0000) 2024-02-20T01:08:48.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:48.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:48.889204+0000) 2024-02-20T01:08:49.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:49.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:49.889457+0000) 2024-02-20T01:08:50.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:50.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:50.889660+0000) 2024-02-20T01:08:51.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:51.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:51.889839+0000) 2024-02-20T01:08:52.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:52.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:52.890047+0000) 2024-02-20T01:08:53.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:53.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:53.890200+0000) 2024-02-20T01:08:54.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:54.883+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:54.890337+0000) 2024-02-20T01:08:55.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:55.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:55.890547+0000) 2024-02-20T01:08:56.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:56.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:56.890768+0000) 2024-02-20T01:08:57.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:57.887+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:57.890990+0000) 2024-02-20T01:08:58.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:58.886+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:58.891169+0000) 2024-02-20T01:08:59.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:08:59.890+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:08:59.891358+0000) 2024-02-20T01:09:00.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:00.886+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:00.891583+0000) 2024-02-20T01:09:01.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:01.886+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:01.891784+0000) 2024-02-20T01:09:02.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:02.886+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:02.891999+0000) 2024-02-20T01:09:03.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:03.890+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:03.892188+0000) 2024-02-20T01:09:04.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:04.886+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:04.892437+0000) 2024-02-20T01:09:05.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:05.886+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:05.892671+0000) 2024-02-20T01:09:06.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:06.886+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:06.892836+0000) 2024-02-20T01:09:07.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:07.886+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:07.893075+0000) 2024-02-20T01:09:08.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:08.886+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:08.893334+0000) 2024-02-20T01:09:09.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:09.890+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:09.893501+0000) 2024-02-20T01:09:10.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:10.886+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:10.893714+0000) 2024-02-20T01:09:11.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:11.890+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:11.893963+0000) 2024-02-20T01:09:12.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:12.886+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:12.894155+0000) 2024-02-20T01:09:13.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:13.890+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:13.894363+0000) 2024-02-20T01:09:14.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:14.890+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:14.894597+0000) 2024-02-20T01:09:15.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:15.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:15.894819+0000) 2024-02-20T01:09:16.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:16.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:16.895057+0000) 2024-02-20T01:09:17.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:17.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:17.895362+0000) 2024-02-20T01:09:18.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:18.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:18.895609+0000) 2024-02-20T01:09:19.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:19.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:19.895840+0000) 2024-02-20T01:09:20.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:20.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:20.896102+0000) 2024-02-20T01:09:21.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:21.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:21.896345+0000) 2024-02-20T01:09:22.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:22.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:22.896557+0000) 2024-02-20T01:09:23.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:23.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:23.896809+0000) 2024-02-20T01:09:24.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:24.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:24.897045+0000) 2024-02-20T01:09:25.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:25.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:25.897263+0000) 2024-02-20T01:09:26.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:26.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:26.897528+0000) 2024-02-20T01:09:27.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:27.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:27.897767+0000) 2024-02-20T01:09:28.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:28.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:28.897958+0000) 2024-02-20T01:09:29.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:29.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:29.898123+0000) 2024-02-20T01:09:30.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:30.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:30.898346+0000) 2024-02-20T01:09:31.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:31.894+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:31.898602+0000) 2024-02-20T01:09:32.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:32.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:32.898831+0000) 2024-02-20T01:09:33.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:33.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:33.899084+0000) 2024-02-20T01:09:34.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:34.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:34.899212+0000) 2024-02-20T01:09:35.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:35.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:35.899457+0000) 2024-02-20T01:09:36.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:36.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:36.899678+0000) 2024-02-20T01:09:37.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:37.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:37.899856+0000) 2024-02-20T01:09:38.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:38.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:38.900009+0000) 2024-02-20T01:09:39.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:39.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:39.900169+0000) 2024-02-20T01:09:40.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:40.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:40.900401+0000) 2024-02-20T01:09:41.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:41.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:41.900597+0000) 2024-02-20T01:09:42.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:42.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:42.900761+0000) 2024-02-20T01:09:43.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:43.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:43.900990+0000) 2024-02-20T01:09:44.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:44.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:44.901255+0000) 2024-02-20T01:09:45.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:45.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:45.901479+0000) 2024-02-20T01:09:46.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:46.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:46.901725+0000) 2024-02-20T01:09:47.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:47.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:47.901978+0000) 2024-02-20T01:09:48.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:48.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:48.902225+0000) 2024-02-20T01:09:49.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:49.898+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:49.902479+0000) 2024-02-20T01:09:50.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:50.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:50.902749+0000) 2024-02-20T01:09:51.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:51.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:51.902999+0000) 2024-02-20T01:09:52.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:52.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:52.903175+0000) 2024-02-20T01:09:53.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:53.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:53.903433+0000) 2024-02-20T01:09:54.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:54.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:54.903636+0000) 2024-02-20T01:09:55.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:55.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:55.903869+0000) 2024-02-20T01:09:56.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:56.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:56.904114+0000) 2024-02-20T01:09:57.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:57.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:57.904393+0000) 2024-02-20T01:09:58.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:58.902+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:58.904578+0000) 2024-02-20T01:09:59.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:09:59.901+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:09:59.904802+0000) 2024-02-20T01:10:00.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:00.901+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:00.905045+0000) 2024-02-20T01:10:01.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:01.901+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:01.905205+0000) 2024-02-20T01:10:02.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:02.901+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:02.905450+0000) 2024-02-20T01:10:03.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:03.901+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:03.905635+0000) 2024-02-20T01:10:04.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:04.901+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:04.905831+0000) 2024-02-20T01:10:05.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:05.901+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:05.906112+0000) 2024-02-20T01:10:06.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:06.901+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:06.906336+0000) 2024-02-20T01:10:07.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:07.901+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:07.906587+0000) 2024-02-20T01:10:08.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:08.905+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:08.906867+0000) 2024-02-20T01:10:09.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:09.905+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:09.907034+0000) 2024-02-20T01:10:10.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:10.905+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:10.907223+0000) 2024-02-20T01:10:11.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:11.905+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:11.907420+0000) 2024-02-20T01:10:12.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:12.905+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:12.907603+0000) 2024-02-20T01:10:13.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:13.905+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:13.907875+0000) 2024-02-20T01:10:14.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:14.905+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:14.908150+0000) 2024-02-20T01:10:15.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:15.905+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:15.908317+0000) 2024-02-20T01:10:16.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:16.905+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:16.908554+0000) 2024-02-20T01:10:17.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:17.905+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:17.908792+0000) 2024-02-20T01:10:18.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:18.905+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:18.909038+0000) 2024-02-20T01:10:19.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:19.905+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:19.909261+0000) 2024-02-20T01:10:20.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:20.905+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:20.909502+0000) 2024-02-20T01:10:21.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:21.905+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:21.909723+0000) 2024-02-20T01:10:22.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:22.905+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:22.909899+0000) 2024-02-20T01:10:23.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:23.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:23.910143+0000) 2024-02-20T01:10:24.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:24.905+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:24.910374+0000) 2024-02-20T01:10:25.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:25.905+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:25.910570+0000) 2024-02-20T01:10:26.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:26.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:26.910833+0000) 2024-02-20T01:10:27.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:27.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:27.910958+0000) 2024-02-20T01:10:28.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:28.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:28.911149+0000) 2024-02-20T01:10:29.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:29.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:29.911398+0000) 2024-02-20T01:10:30.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:30.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:30.911588+0000) 2024-02-20T01:10:31.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:31.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:31.911824+0000) 2024-02-20T01:10:32.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:32.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:32.912060+0000) 2024-02-20T01:10:33.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:33.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:33.912297+0000) 2024-02-20T01:10:34.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:34.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:34.912552+0000) 2024-02-20T01:10:35.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:35.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:35.912795+0000) 2024-02-20T01:10:36.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:36.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:36.912984+0000) 2024-02-20T01:10:37.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:37.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:37.913228+0000) 2024-02-20T01:10:38.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:38.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:38.913474+0000) 2024-02-20T01:10:39.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:39.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:39.913708+0000) 2024-02-20T01:10:40.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:40.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:40.913941+0000) 2024-02-20T01:10:41.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:41.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:41.914218+0000) 2024-02-20T01:10:42.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:42.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:42.914409+0000) 2024-02-20T01:10:43.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:43.909+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:43.914654+0000) 2024-02-20T01:10:44.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:44.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:44.914892+0000) 2024-02-20T01:10:45.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:45.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:45.915120+0000) 2024-02-20T01:10:46.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:46.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:46.915242+0000) 2024-02-20T01:10:47.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:47.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:47.915482+0000) 2024-02-20T01:10:48.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:48.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:48.915677+0000) 2024-02-20T01:10:49.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:49.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:49.915876+0000) 2024-02-20T01:10:50.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:50.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:50.916078+0000) 2024-02-20T01:10:51.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:51.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:51.916214+0000) 2024-02-20T01:10:52.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:52.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:52.916473+0000) 2024-02-20T01:10:53.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:53.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:53.916701+0000) 2024-02-20T01:10:54.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:54.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:54.916937+0000) 2024-02-20T01:10:55.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:55.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:55.917162+0000) 2024-02-20T01:10:56.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:56.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:56.917409+0000) 2024-02-20T01:10:57.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:57.913+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:57.917661+0000) 2024-02-20T01:10:58.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:58.917+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:58.917897+0000) 2024-02-20T01:10:59.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:10:59.916+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:10:59.918135+0000) 2024-02-20T01:11:00.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:00.912+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:00.918309+0000) 2024-02-20T01:11:01.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:01.912+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:01.918569+0000) 2024-02-20T01:11:02.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:02.916+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:02.918863+0000) 2024-02-20T01:11:03.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:03.916+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:03.919120+0000) 2024-02-20T01:11:04.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:04.916+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:04.919331+0000) 2024-02-20T01:11:05.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:05.916+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:05.919499+0000) 2024-02-20T01:11:06.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:06.916+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:06.919726+0000) 2024-02-20T01:11:07.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:07.916+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:07.919937+0000) 2024-02-20T01:11:08.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:08.916+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:08.920157+0000) 2024-02-20T01:11:09.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:09.916+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:09.920395+0000) 2024-02-20T01:11:10.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:10.916+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:10.920635+0000) 2024-02-20T01:11:11.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:11.916+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:11.920883+0000) 2024-02-20T01:11:12.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:12.916+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:12.921089+0000) 2024-02-20T01:11:13.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:13.916+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:13.921284+0000) 2024-02-20T01:11:14.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:14.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:14.921531+0000) 2024-02-20T01:11:15.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:15.916+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:15.921719+0000) 2024-02-20T01:11:16.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:16.916+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:16.921936+0000) 2024-02-20T01:11:17.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:17.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:17.922177+0000) 2024-02-20T01:11:18.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:18.916+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:18.922407+0000) 2024-02-20T01:11:19.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:19.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:19.922677+0000) 2024-02-20T01:11:20.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:20.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:20.922896+0000) 2024-02-20T01:11:21.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:21.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:21.923066+0000) 2024-02-20T01:11:22.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:22.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:22.923243+0000) 2024-02-20T01:11:23.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:23.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:23.923512+0000) 2024-02-20T01:11:24.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:24.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:24.923700+0000) 2024-02-20T01:11:25.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:25.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:25.923937+0000) 2024-02-20T01:11:26.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:26.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:26.924207+0000) 2024-02-20T01:11:27.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:27.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:27.924463+0000) 2024-02-20T01:11:28.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:28.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:28.924643+0000) 2024-02-20T01:11:29.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:29.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:29.924834+0000) 2024-02-20T01:11:30.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:30.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:30.925044+0000) 2024-02-20T01:11:31.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:31.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:31.925279+0000) 2024-02-20T01:11:32.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:32.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:32.925464+0000) 2024-02-20T01:11:33.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:33.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:33.925701+0000) 2024-02-20T01:11:34.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:34.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:34.925945+0000) 2024-02-20T01:11:35.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:35.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:35.926164+0000) 2024-02-20T01:11:36.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:36.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:36.926391+0000) 2024-02-20T01:11:37.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:37.920+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:37.926640+0000) 2024-02-20T01:11:38.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:38.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:38.926919+0000) 2024-02-20T01:11:39.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:39.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:39.927153+0000) 2024-02-20T01:11:40.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:40.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:40.927359+0000) 2024-02-20T01:11:41.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:41.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:41.927641+0000) 2024-02-20T01:11:42.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:42.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:42.927848+0000) 2024-02-20T01:11:43.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:43.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:43.928015+0000) 2024-02-20T01:11:44.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:44.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:44.928299+0000) 2024-02-20T01:11:45.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:45.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:45.928527+0000) 2024-02-20T01:11:46.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:46.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:46.928774+0000) 2024-02-20T01:11:47.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:47.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:47.929088+0000) 2024-02-20T01:11:48.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:48.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:48.929370+0000) 2024-02-20T01:11:49.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:49.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:49.929644+0000) 2024-02-20T01:11:50.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:50.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:50.929902+0000) 2024-02-20T01:11:51.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:51.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:51.930196+0000) 2024-02-20T01:11:52.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:52.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:52.930422+0000) 2024-02-20T01:11:53.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:53.924+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:53.930601+0000) 2024-02-20T01:11:54.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:54.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:54.930780+0000) 2024-02-20T01:11:55.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:55.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:55.931016+0000) 2024-02-20T01:11:56.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:56.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:56.931225+0000) 2024-02-20T01:11:57.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:57.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:57.931444+0000) 2024-02-20T01:11:58.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:58.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:58.931669+0000) 2024-02-20T01:11:59.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:11:59.928+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:11:59.931885+0000) 2024-02-20T01:12:00.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:00.927+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:00.932111+0000) 2024-02-20T01:12:01.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:01.927+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:01.932367+0000) 2024-02-20T01:12:02.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:02.927+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:02.932560+0000) 2024-02-20T01:12:03.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:03.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:03.932791+0000) 2024-02-20T01:12:04.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:04.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:04.932994+0000) 2024-02-20T01:12:05.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:05.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:05.933227+0000) 2024-02-20T01:12:06.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:06.927+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:06.933443+0000) 2024-02-20T01:12:07.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:07.927+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:07.933701+0000) 2024-02-20T01:12:08.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:08.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:08.933919+0000) 2024-02-20T01:12:09.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:09.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:09.934107+0000) 2024-02-20T01:12:10.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:10.927+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:10.934351+0000) 2024-02-20T01:12:11.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:11.927+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:11.934594+0000) 2024-02-20T01:12:12.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:12.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:12.934865+0000) 2024-02-20T01:12:13.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:13.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:13.935116+0000) 2024-02-20T01:12:14.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:14.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:14.935290+0000) 2024-02-20T01:12:15.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:15.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:15.935543+0000) 2024-02-20T01:12:16.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:16.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:16.935741+0000) 2024-02-20T01:12:17.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:17.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:17.935964+0000) 2024-02-20T01:12:18.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:18.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:18.936205+0000) 2024-02-20T01:12:19.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:19.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:19.936485+0000) 2024-02-20T01:12:20.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:20.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:20.936694+0000) 2024-02-20T01:12:21.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:21.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:21.936946+0000) 2024-02-20T01:12:22.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:22.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:22.937203+0000) 2024-02-20T01:12:23.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:23.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:23.937391+0000) 2024-02-20T01:12:24.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:24.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:24.937561+0000) 2024-02-20T01:12:25.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:25.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:25.937787+0000) 2024-02-20T01:12:26.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:26.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:26.938010+0000) 2024-02-20T01:12:27.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:27.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:27.938249+0000) 2024-02-20T01:12:28.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:28.931+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:28.938513+0000) 2024-02-20T01:12:29.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:29.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:29.938746+0000) 2024-02-20T01:12:30.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:30.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:30.938903+0000) 2024-02-20T01:12:31.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:31.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:31.939066+0000) 2024-02-20T01:12:32.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:32.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:32.939256+0000) 2024-02-20T01:12:33.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:33.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:33.939500+0000) 2024-02-20T01:12:34.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:34.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:34.939734+0000) 2024-02-20T01:12:35.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:35.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:35.940016+0000) 2024-02-20T01:12:36.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:36.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:36.940222+0000) 2024-02-20T01:12:37.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:37.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:37.940459+0000) 2024-02-20T01:12:38.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:38.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:38.940698+0000) 2024-02-20T01:12:39.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:39.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:39.940891+0000) 2024-02-20T01:12:40.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:40.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:40.941138+0000) 2024-02-20T01:12:41.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:41.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:41.941409+0000) 2024-02-20T01:12:42.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:42.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:42.941604+0000) 2024-02-20T01:12:43.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:43.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:43.941815+0000) 2024-02-20T01:12:44.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:44.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:44.942036+0000) 2024-02-20T01:12:45.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:45.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:45.942222+0000) 2024-02-20T01:12:46.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:46.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:46.942409+0000) 2024-02-20T01:12:47.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:47.935+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:47.942639+0000) 2024-02-20T01:12:48.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:48.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:48.942866+0000) 2024-02-20T01:12:49.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:49.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:49.943059+0000) 2024-02-20T01:12:50.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:50.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:50.943214+0000) 2024-02-20T01:12:51.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:51.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:51.943417+0000) 2024-02-20T01:12:52.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:52.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:52.943627+0000) 2024-02-20T01:12:53.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:53.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:53.943863+0000) 2024-02-20T01:12:54.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:54.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:54.944113+0000) 2024-02-20T01:12:55.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:55.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:55.944356+0000) 2024-02-20T01:12:56.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:56.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:56.944623+0000) 2024-02-20T01:12:57.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:57.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:57.944813+0000) 2024-02-20T01:12:58.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:58.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:58.945036+0000) 2024-02-20T01:12:59.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:12:59.939+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:12:59.945278+0000) 2024-02-20T01:13:00.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:00.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:00.945480+0000) 2024-02-20T01:13:01.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:01.942+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:01.945766+0000) 2024-02-20T01:13:02.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:02.938+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:02.945935+0000) 2024-02-20T01:13:03.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:03.938+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:03.946134+0000) 2024-02-20T01:13:04.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:04.938+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:04.946307+0000) 2024-02-20T01:13:05.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:05.942+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:05.946489+0000) 2024-02-20T01:13:06.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:06.942+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:06.946770+0000) 2024-02-20T01:13:07.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:07.942+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:07.947010+0000) 2024-02-20T01:13:08.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:08.942+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:08.947205+0000) 2024-02-20T01:13:09.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:09.942+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:09.947476+0000) 2024-02-20T01:13:10.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:10.942+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:10.947727+0000) 2024-02-20T01:13:11.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:11.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:11.947975+0000) 2024-02-20T01:13:12.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:12.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:12.948233+0000) 2024-02-20T01:13:13.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:13.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:13.948449+0000) 2024-02-20T01:13:14.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:14.942+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:14.948723+0000) 2024-02-20T01:13:15.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:15.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:15.948910+0000) 2024-02-20T01:13:16.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:16.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:16.949169+0000) 2024-02-20T01:13:17.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:17.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:17.949405+0000) 2024-02-20T01:13:18.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:18.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:18.949591+0000) 2024-02-20T01:13:19.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:19.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:19.949747+0000) 2024-02-20T01:13:20.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:20.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:20.949978+0000) 2024-02-20T01:13:21.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:21.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:21.950223+0000) 2024-02-20T01:13:22.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:22.946+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:22.950437+0000) 2024-02-20T01:13:23.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:23.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:23.950691+0000) 2024-02-20T01:13:24.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:24.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:24.950983+0000) 2024-02-20T01:13:25.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:25.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:25.951218+0000) 2024-02-20T01:13:26.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:26.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:26.951460+0000) 2024-02-20T01:13:27.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:27.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:27.951687+0000) 2024-02-20T01:13:28.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:28.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:28.952008+0000) 2024-02-20T01:13:29.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:29.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:29.952291+0000) 2024-02-20T01:13:30.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:30.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:30.952529+0000) 2024-02-20T01:13:31.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:31.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:31.952785+0000) 2024-02-20T01:13:32.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:32.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:32.953018+0000) 2024-02-20T01:13:33.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:33.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:33.953259+0000) 2024-02-20T01:13:34.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:34.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:34.953458+0000) 2024-02-20T01:13:35.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:35.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:35.953631+0000) 2024-02-20T01:13:36.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:36.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:36.953869+0000) 2024-02-20T01:13:37.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:37.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:37.954073+0000) 2024-02-20T01:13:38.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:38.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:38.954298+0000) 2024-02-20T01:13:39.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:39.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:39.954493+0000) 2024-02-20T01:13:40.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:40.950+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:40.954649+0000) 2024-02-20T01:13:41.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:41.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:41.954875+0000) 2024-02-20T01:13:42.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:42.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:42.955113+0000) 2024-02-20T01:13:43.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:43.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:43.955371+0000) 2024-02-20T01:13:44.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:44.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:44.955559+0000) 2024-02-20T01:13:45.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:45.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:45.955816+0000) 2024-02-20T01:13:46.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:46.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:46.956063+0000) 2024-02-20T01:13:47.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:47.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:47.956268+0000) 2024-02-20T01:13:48.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:48.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:48.956427+0000) 2024-02-20T01:13:49.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:49.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:49.956722+0000) 2024-02-20T01:13:50.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:50.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:50.956936+0000) 2024-02-20T01:13:51.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:51.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:51.957186+0000) 2024-02-20T01:13:52.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:52.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:52.957382+0000) 2024-02-20T01:13:53.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:53.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:53.957638+0000) 2024-02-20T01:13:54.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:54.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:54.957854+0000) 2024-02-20T01:13:55.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:55.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:55.958082+0000) 2024-02-20T01:13:56.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:56.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:56.958304+0000) 2024-02-20T01:13:57.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:57.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:57.958571+0000) 2024-02-20T01:13:58.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:58.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:58.958814+0000) 2024-02-20T01:13:59.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:13:59.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:13:59.959041+0000) 2024-02-20T01:14:00.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:00.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:00.959292+0000) 2024-02-20T01:14:01.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:01.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:01.959549+0000) 2024-02-20T01:14:02.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:02.957+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:02.959792+0000) 2024-02-20T01:14:03.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:03.957+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:03.960055+0000) 2024-02-20T01:14:04.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:04.957+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:04.960263+0000) 2024-02-20T01:14:05.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:05.957+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:05.960380+0000) 2024-02-20T01:14:06.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:06.957+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:06.960630+0000) 2024-02-20T01:14:07.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:07.957+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:07.960891+0000) 2024-02-20T01:14:08.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:08.957+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:08.961048+0000) 2024-02-20T01:14:09.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:09.957+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:09.961184+0000) 2024-02-20T01:14:10.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:10.957+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:10.961421+0000) 2024-02-20T01:14:11.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:11.957+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:11.961758+0000) 2024-02-20T01:14:12.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:12.957+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:12.961933+0000) 2024-02-20T01:14:13.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:13.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:13.962183+0000) 2024-02-20T01:14:14.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:14.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:14.962430+0000) 2024-02-20T01:14:15.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:15.957+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:15.962635+0000) 2024-02-20T01:14:16.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:16.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:16.962812+0000) 2024-02-20T01:14:17.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:17.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:17.962993+0000) 2024-02-20T01:14:18.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:18.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:18.963208+0000) 2024-02-20T01:14:19.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:19.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:19.963429+0000) 2024-02-20T01:14:20.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:20.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:20.963698+0000) 2024-02-20T01:14:21.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:21.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:21.963894+0000) 2024-02-20T01:14:22.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:22.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:22.964147+0000) 2024-02-20T01:14:23.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:23.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:23.964341+0000) 2024-02-20T01:14:24.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:24.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:24.964584+0000) 2024-02-20T01:14:25.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:25.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:25.964776+0000) 2024-02-20T01:14:26.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:26.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:26.964990+0000) 2024-02-20T01:14:27.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:27.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:27.965234+0000) 2024-02-20T01:14:28.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:28.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:28.965469+0000) 2024-02-20T01:14:29.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:29.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:29.965732+0000) 2024-02-20T01:14:30.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:30.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:30.965998+0000) 2024-02-20T01:14:31.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:31.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:31.966249+0000) 2024-02-20T01:14:32.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:32.961+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:32.966493+0000) 2024-02-20T01:14:33.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:33.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:33.966750+0000) 2024-02-20T01:14:34.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:34.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:34.966964+0000) 2024-02-20T01:14:35.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:35.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:35.967192+0000) 2024-02-20T01:14:36.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:36.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:36.967444+0000) 2024-02-20T01:14:37.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:37.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:37.967714+0000) 2024-02-20T01:14:38.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:38.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:38.967948+0000) 2024-02-20T01:14:39.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:39.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:39.968116+0000) 2024-02-20T01:14:40.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:40.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:40.968374+0000) 2024-02-20T01:14:41.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:41.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:41.968645+0000) 2024-02-20T01:14:42.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:42.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:42.968875+0000) 2024-02-20T01:14:43.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:43.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:43.969050+0000) 2024-02-20T01:14:44.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:44.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:44.969297+0000) 2024-02-20T01:14:45.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:45.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:45.969551+0000) 2024-02-20T01:14:46.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:46.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:46.969737+0000) 2024-02-20T01:14:47.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:47.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:47.969897+0000) 2024-02-20T01:14:48.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:48.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:48.970064+0000) 2024-02-20T01:14:49.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:49.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:49.970291+0000) 2024-02-20T01:14:50.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:50.965+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:50.970520+0000) 2024-02-20T01:14:51.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:51.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:51.970797+0000) 2024-02-20T01:14:52.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:52.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:52.970984+0000) 2024-02-20T01:14:53.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:53.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:53.971174+0000) 2024-02-20T01:14:54.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:54.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:54.971356+0000) 2024-02-20T01:14:55.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:55.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:55.971519+0000) 2024-02-20T01:14:56.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:56.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:56.971752+0000) 2024-02-20T01:14:57.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:57.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:57.972002+0000) 2024-02-20T01:14:58.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:58.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:58.972261+0000) 2024-02-20T01:14:59.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:14:59.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:14:59.972498+0000) 2024-02-20T01:15:00.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:00.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:00.972787+0000) 2024-02-20T01:15:01.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:01.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:01.973051+0000) 2024-02-20T01:15:02.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:02.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:02.973318+0000) 2024-02-20T01:15:03.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:03.968+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:03.973548+0000) 2024-02-20T01:15:04.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:04.968+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:04.973738+0000) 2024-02-20T01:15:05.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:05.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:05.973905+0000) 2024-02-20T01:15:06.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:06.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:06.974154+0000) 2024-02-20T01:15:07.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:07.968+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:07.974282+0000) 2024-02-20T01:15:08.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:08.968+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:08.974457+0000) 2024-02-20T01:15:09.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:09.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:09.974679+0000) 2024-02-20T01:15:10.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:10.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:10.974956+0000) 2024-02-20T01:15:11.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:11.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:11.975126+0000) 2024-02-20T01:15:12.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:12.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:12.975367+0000) 2024-02-20T01:15:13.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:13.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:13.975641+0000) 2024-02-20T01:15:14.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:14.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:14.975841+0000) 2024-02-20T01:15:15.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:15.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:15.976065+0000) 2024-02-20T01:15:16.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:16.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:16.976295+0000) 2024-02-20T01:15:17.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:17.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:17.976546+0000) 2024-02-20T01:15:18.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:18.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:18.976765+0000) 2024-02-20T01:15:19.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:19.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:19.976978+0000) 2024-02-20T01:15:20.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:20.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:20.977168+0000) 2024-02-20T01:15:21.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:21.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:21.977438+0000) 2024-02-20T01:15:22.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:22.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:22.977697+0000) 2024-02-20T01:15:23.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:23.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:23.977914+0000) 2024-02-20T01:15:24.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:24.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:24.978176+0000) 2024-02-20T01:15:25.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:25.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:25.978437+0000) 2024-02-20T01:15:26.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:26.972+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:26.978640+0000) 2024-02-20T01:15:27.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:27.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:27.978818+0000) 2024-02-20T01:15:28.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:28.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:28.978944+0000) 2024-02-20T01:15:29.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:29.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:29.979071+0000) 2024-02-20T01:15:30.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:30.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:30.979255+0000) 2024-02-20T01:15:31.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:31.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:31.979519+0000) 2024-02-20T01:15:32.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:32.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:32.979730+0000) 2024-02-20T01:15:33.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:33.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:33.979973+0000) 2024-02-20T01:15:34.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:34.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:34.980188+0000) 2024-02-20T01:15:35.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:35.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:35.980369+0000) 2024-02-20T01:15:36.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:36.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:36.980609+0000) 2024-02-20T01:15:37.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:37.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:37.980802+0000) 2024-02-20T01:15:38.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:38.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:38.981033+0000) 2024-02-20T01:15:39.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:39.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:39.981283+0000) 2024-02-20T01:15:40.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:40.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:40.981524+0000) 2024-02-20T01:15:41.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:41.976+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:41.981775+0000) 2024-02-20T01:15:42.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:42.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:42.981985+0000) 2024-02-20T01:15:43.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:43.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:43.982144+0000) 2024-02-20T01:15:44.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:44.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:44.982378+0000) 2024-02-20T01:15:45.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:45.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:45.982636+0000) 2024-02-20T01:15:46.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:46.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:46.982867+0000) 2024-02-20T01:15:47.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:47.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:47.983120+0000) 2024-02-20T01:15:48.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:48.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:48.983350+0000) 2024-02-20T01:15:49.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:49.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:49.983538+0000) 2024-02-20T01:15:50.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:50.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:50.983797+0000) 2024-02-20T01:15:51.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:51.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:51.984064+0000) 2024-02-20T01:15:52.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:52.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:52.984347+0000) 2024-02-20T01:15:53.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:53.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:53.984625+0000) 2024-02-20T01:15:54.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:54.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:54.984884+0000) 2024-02-20T01:15:55.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:55.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:55.985116+0000) 2024-02-20T01:15:56.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:56.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:56.985382+0000) 2024-02-20T01:15:57.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:57.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:57.985589+0000) 2024-02-20T01:15:58.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:58.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:58.985817+0000) 2024-02-20T01:15:59.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:15:59.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:15:59.985981+0000) 2024-02-20T01:16:00.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:00.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:00.986210+0000) 2024-02-20T01:16:01.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:01.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:01.986476+0000) 2024-02-20T01:16:02.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:02.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:02.986748+0000) 2024-02-20T01:16:03.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:03.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:03.987015+0000) 2024-02-20T01:16:04.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:04.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:04.987218+0000) 2024-02-20T01:16:05.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:05.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:05.987485+0000) 2024-02-20T01:16:06.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:06.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:06.987714+0000) 2024-02-20T01:16:07.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:07.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:07.987900+0000) 2024-02-20T01:16:08.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:08.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:08.988151+0000) 2024-02-20T01:16:09.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:09.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:09.988394+0000) 2024-02-20T01:16:10.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:10.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:10.988620+0000) 2024-02-20T01:16:11.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:11.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:11.988851+0000) 2024-02-20T01:16:12.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:12.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:12.989082+0000) 2024-02-20T01:16:13.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:13.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:13.989272+0000) 2024-02-20T01:16:14.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:14.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:14.989431+0000) 2024-02-20T01:16:15.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:15.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:15.989557+0000) 2024-02-20T01:16:16.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:16.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:16.989759+0000) 2024-02-20T01:16:17.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:17.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:17.989998+0000) 2024-02-20T01:16:18.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:18.983+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:18.990175+0000) 2024-02-20T01:16:19.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:19.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:19.990420+0000) 2024-02-20T01:16:20.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:20.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:20.990563+0000) 2024-02-20T01:16:21.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:21.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:21.990790+0000) 2024-02-20T01:16:22.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:22.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:22.990936+0000) 2024-02-20T01:16:23.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:23.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:23.991210+0000) 2024-02-20T01:16:24.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:24.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:24.991387+0000) 2024-02-20T01:16:25.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:25.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:25.991643+0000) 2024-02-20T01:16:26.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:26.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:26.991882+0000) 2024-02-20T01:16:27.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:27.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:27.992124+0000) 2024-02-20T01:16:28.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:28.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:28.992319+0000) 2024-02-20T01:16:29.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:29.987+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:29.992561+0000) 2024-02-20T01:16:30.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:30.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:30.992815+0000) 2024-02-20T01:16:31.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:31.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:31.992997+0000) 2024-02-20T01:16:32.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:32.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:32.993246+0000) 2024-02-20T01:16:33.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:33.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:33.993483+0000) 2024-02-20T01:16:34.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:34.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:34.993723+0000) 2024-02-20T01:16:35.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:35.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:35.993963+0000) 2024-02-20T01:16:36.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:36.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:36.994199+0000) 2024-02-20T01:16:37.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:37.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:37.994466+0000) 2024-02-20T01:16:38.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:38.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:38.994743+0000) 2024-02-20T01:16:39.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:39.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:39.994936+0000) 2024-02-20T01:16:40.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:40.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:40.995121+0000) 2024-02-20T01:16:41.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:41.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:41.995383+0000) 2024-02-20T01:16:42.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:42.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:42.995599+0000) 2024-02-20T01:16:43.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:43.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:43.995774+0000) 2024-02-20T01:16:44.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:44.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:44.996020+0000) 2024-02-20T01:16:45.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:45.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:45.996316+0000) 2024-02-20T01:16:46.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:46.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:46.996528+0000) 2024-02-20T01:16:47.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:47.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:47.996772+0000) 2024-02-20T01:16:48.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:48.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:48.997022+0000) 2024-02-20T01:16:49.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:49.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:49.997243+0000) 2024-02-20T01:16:50.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:50.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:50.997478+0000) 2024-02-20T01:16:51.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:51.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:51.997707+0000) 2024-02-20T01:16:52.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:52.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:52.997897+0000) 2024-02-20T01:16:53.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:53.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:53.998131+0000) 2024-02-20T01:16:54.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:54.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:54.998332+0000) 2024-02-20T01:16:55.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:55.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:55.998513+0000) 2024-02-20T01:16:56.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:56.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:56.998693+0000) 2024-02-20T01:16:57.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:57.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:57.998908+0000) 2024-02-20T01:16:58.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:58.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:58.999094+0000) 2024-02-20T01:16:59.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:16:59.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:16:59.999348+0000) 2024-02-20T01:17:00.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:00.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:00.999615+0000) 2024-02-20T01:17:01.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:01.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:01.999809+0000) 2024-02-20T01:17:02.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:02.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:02.999969+0000) 2024-02-20T01:17:04.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:03.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:04.000170+0000) 2024-02-20T01:17:05.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:04.994+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:05.000324+0000) 2024-02-20T01:17:06.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:05.994+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:06.000536+0000) 2024-02-20T01:17:07.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:06.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:07.000788+0000) 2024-02-20T01:17:08.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:07.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:08.001015+0000) 2024-02-20T01:17:09.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:08.994+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:09.001264+0000) 2024-02-20T01:17:10.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:09.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:10.001448+0000) 2024-02-20T01:17:11.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:10.994+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:11.001627+0000) 2024-02-20T01:17:12.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:11.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:12.001796+0000) 2024-02-20T01:17:13.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:12.994+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:13.001978+0000) 2024-02-20T01:17:14.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:13.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:14.002193+0000) 2024-02-20T01:17:15.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:14.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:15.002369+0000) 2024-02-20T01:17:16.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:15.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:16.002622+0000) 2024-02-20T01:17:17.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:16.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:17.002821+0000) 2024-02-20T01:17:18.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:17.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:18.002962+0000) 2024-02-20T01:17:19.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:18.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:19.003129+0000) 2024-02-20T01:17:20.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:19.998+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:20.003300+0000) 2024-02-20T01:17:21.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:21.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:21.003527+0000) 2024-02-20T01:17:22.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:22.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:22.003669+0000) 2024-02-20T01:17:23.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:23.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:23.003871+0000) 2024-02-20T01:17:24.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:24.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:24.004073+0000) 2024-02-20T01:17:25.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:25.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:25.004313+0000) 2024-02-20T01:17:26.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:26.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:26.004535+0000) 2024-02-20T01:17:27.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:27.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:27.004774+0000) 2024-02-20T01:17:28.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:28.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:28.004968+0000) 2024-02-20T01:17:29.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:29.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:29.005210+0000) 2024-02-20T01:17:30.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:30.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:30.005462+0000) 2024-02-20T01:17:31.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:31.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:31.005635+0000) 2024-02-20T01:17:32.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:32.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:32.005884+0000) 2024-02-20T01:17:33.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:33.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:33.006083+0000) 2024-02-20T01:17:34.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:34.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:34.006234+0000) 2024-02-20T01:17:35.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:35.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:35.006398+0000) 2024-02-20T01:17:36.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:36.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:36.006669+0000) 2024-02-20T01:17:37.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:37.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:37.006905+0000) 2024-02-20T01:17:38.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:38.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:38.007076+0000) 2024-02-20T01:17:39.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:39.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:39.007255+0000) 2024-02-20T01:17:40.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:40.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:40.007577+0000) 2024-02-20T01:17:41.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:41.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:41.007755+0000) 2024-02-20T01:17:42.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:42.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:42.007974+0000) 2024-02-20T01:17:43.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:43.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:43.008176+0000) 2024-02-20T01:17:44.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:44.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:44.008358+0000) 2024-02-20T01:17:45.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:45.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:45.008590+0000) 2024-02-20T01:17:46.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:46.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:46.008753+0000) 2024-02-20T01:17:47.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:47.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:47.008985+0000) 2024-02-20T01:17:48.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:48.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:48.009196+0000) 2024-02-20T01:17:49.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:49.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:49.009358+0000) 2024-02-20T01:17:50.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:50.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:50.009503+0000) 2024-02-20T01:17:51.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:51.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:51.009704+0000) 2024-02-20T01:17:52.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:52.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:52.009928+0000) 2024-02-20T01:17:53.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:53.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:53.010158+0000) 2024-02-20T01:17:54.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:54.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:54.010393+0000) 2024-02-20T01:17:55.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:55.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:55.010628+0000) 2024-02-20T01:17:56.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:56.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:56.010869+0000) 2024-02-20T01:17:57.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:57.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:57.011126+0000) 2024-02-20T01:17:58.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:58.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:58.011368+0000) 2024-02-20T01:17:59.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:17:59.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:17:59.011507+0000) 2024-02-20T01:18:00.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:00.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:00.011763+0000) 2024-02-20T01:18:01.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:01.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:01.012004+0000) 2024-02-20T01:18:02.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:02.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:02.012253+0000) 2024-02-20T01:18:03.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:03.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:03.012450+0000) 2024-02-20T01:18:04.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:04.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:04.012690+0000) 2024-02-20T01:18:05.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:05.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:05.012912+0000) 2024-02-20T01:18:06.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:06.009+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:06.013166+0000) 2024-02-20T01:18:07.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:07.009+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:07.013307+0000) 2024-02-20T01:18:08.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:08.009+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:08.013548+0000) 2024-02-20T01:18:09.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:09.009+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:09.013712+0000) 2024-02-20T01:18:10.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:10.009+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:10.013947+0000) 2024-02-20T01:18:11.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:11.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:11.014129+0000) 2024-02-20T01:18:12.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:12.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:12.014367+0000) 2024-02-20T01:18:13.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:13.009+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:13.014600+0000) 2024-02-20T01:18:14.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:14.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:14.014855+0000) 2024-02-20T01:18:15.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:15.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:15.014989+0000) 2024-02-20T01:18:16.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:16.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:16.015163+0000) 2024-02-20T01:18:17.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:17.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:17.015366+0000) 2024-02-20T01:18:18.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:18.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:18.015608+0000) 2024-02-20T01:18:19.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:19.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:19.015851+0000) 2024-02-20T01:18:20.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:20.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:20.016089+0000) 2024-02-20T01:18:21.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:21.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:21.016295+0000) 2024-02-20T01:18:22.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:22.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:22.016517+0000) 2024-02-20T01:18:23.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:23.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:23.016761+0000) 2024-02-20T01:18:24.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:24.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:24.017005+0000) 2024-02-20T01:18:25.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:25.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:25.017180+0000) 2024-02-20T01:18:26.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:26.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:26.017421+0000) 2024-02-20T01:18:27.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:27.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:27.017654+0000) 2024-02-20T01:18:28.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:28.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:28.017852+0000) 2024-02-20T01:18:29.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:29.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:29.018049+0000) 2024-02-20T01:18:30.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:30.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:30.018243+0000) 2024-02-20T01:18:31.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:31.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:31.018444+0000) 2024-02-20T01:18:32.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:32.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:32.018589+0000) 2024-02-20T01:18:33.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:33.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:33.018839+0000) 2024-02-20T01:18:34.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:34.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:34.019086+0000) 2024-02-20T01:18:35.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:35.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:35.019363+0000) 2024-02-20T01:18:36.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:36.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:36.019573+0000) 2024-02-20T01:18:37.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:37.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:37.019822+0000) 2024-02-20T01:18:38.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:38.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:38.020061+0000) 2024-02-20T01:18:39.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:39.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:39.020265+0000) 2024-02-20T01:18:40.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:40.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:40.020515+0000) 2024-02-20T01:18:41.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:41.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:41.020757+0000) 2024-02-20T01:18:42.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:42.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:42.021010+0000) 2024-02-20T01:18:43.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:43.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:43.021208+0000) 2024-02-20T01:18:44.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:44.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:44.021440+0000) 2024-02-20T01:18:45.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:45.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:45.021645+0000) 2024-02-20T01:18:46.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:46.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:46.021887+0000) 2024-02-20T01:18:47.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:47.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:47.022117+0000) 2024-02-20T01:18:48.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:48.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:48.022370+0000) 2024-02-20T01:18:49.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:49.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:49.022633+0000) 2024-02-20T01:18:50.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:50.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:50.022811+0000) 2024-02-20T01:18:51.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:51.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:51.022956+0000) 2024-02-20T01:18:52.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:52.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:52.023139+0000) 2024-02-20T01:18:53.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:53.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:53.023386+0000) 2024-02-20T01:18:54.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:54.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:54.023634+0000) 2024-02-20T01:18:55.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:55.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:55.023852+0000) 2024-02-20T01:18:56.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:56.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:56.024105+0000) 2024-02-20T01:18:57.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:57.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:57.024342+0000) 2024-02-20T01:18:58.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:58.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:58.024592+0000) 2024-02-20T01:18:59.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:18:59.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:18:59.024788+0000) 2024-02-20T01:19:00.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:00.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:00.025000+0000) 2024-02-20T01:19:01.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:01.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:01.025244+0000) 2024-02-20T01:19:02.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:02.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:02.025526+0000) 2024-02-20T01:19:03.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:03.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:03.025725+0000) 2024-02-20T01:19:04.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:04.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:04.025965+0000) 2024-02-20T01:19:05.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:05.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:05.026206+0000) 2024-02-20T01:19:06.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:06.020+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:06.026457+0000) 2024-02-20T01:19:07.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:07.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:07.026752+0000) 2024-02-20T01:19:08.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:08.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:08.026992+0000) 2024-02-20T01:19:09.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:09.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:09.027242+0000) 2024-02-20T01:19:10.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:10.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:10.027520+0000) 2024-02-20T01:19:11.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:11.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:11.027772+0000) 2024-02-20T01:19:12.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:12.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:12.027966+0000) 2024-02-20T01:19:13.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:13.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:13.028202+0000) 2024-02-20T01:19:14.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:14.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:14.028442+0000) 2024-02-20T01:19:15.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:15.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:15.028632+0000) 2024-02-20T01:19:16.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:16.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:16.028858+0000) 2024-02-20T01:19:17.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:17.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:17.029083+0000) 2024-02-20T01:19:18.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:18.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:18.029252+0000) 2024-02-20T01:19:19.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:19.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:19.029482+0000) 2024-02-20T01:19:20.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:20.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:20.029730+0000) 2024-02-20T01:19:21.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:21.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:21.029929+0000) 2024-02-20T01:19:22.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:22.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:22.030104+0000) 2024-02-20T01:19:23.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:23.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:23.030245+0000) 2024-02-20T01:19:24.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:24.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:24.030448+0000) 2024-02-20T01:19:25.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:25.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:25.030628+0000) 2024-02-20T01:19:26.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:26.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:26.030838+0000) 2024-02-20T01:19:27.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:27.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:27.030989+0000) 2024-02-20T01:19:28.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:28.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:28.031154+0000) 2024-02-20T01:19:29.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:29.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:29.031290+0000) 2024-02-20T01:19:30.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:30.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:30.031547+0000) 2024-02-20T01:19:31.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:31.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:31.031782+0000) 2024-02-20T01:19:32.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:32.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:32.032040+0000) 2024-02-20T01:19:33.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:33.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:33.032287+0000) 2024-02-20T01:19:34.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:34.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:34.032523+0000) 2024-02-20T01:19:35.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:35.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:35.032750+0000) 2024-02-20T01:19:36.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:36.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:36.032976+0000) 2024-02-20T01:19:37.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:37.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:37.033209+0000) 2024-02-20T01:19:38.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:38.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:38.033446+0000) 2024-02-20T01:19:39.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:39.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:39.033618+0000) 2024-02-20T01:19:40.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:40.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:40.033873+0000) 2024-02-20T01:19:41.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:41.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:41.034061+0000) 2024-02-20T01:19:42.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:42.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:42.034305+0000) 2024-02-20T01:19:43.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:43.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:43.034439+0000) 2024-02-20T01:19:44.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:44.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:44.034681+0000) 2024-02-20T01:19:45.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:45.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:45.034973+0000) 2024-02-20T01:19:46.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:46.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:46.035183+0000) 2024-02-20T01:19:47.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:47.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:47.035413+0000) 2024-02-20T01:19:48.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:48.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:48.035585+0000) 2024-02-20T01:19:49.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:49.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:49.035780+0000) 2024-02-20T01:19:50.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:50.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:50.035917+0000) 2024-02-20T01:19:51.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:51.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:51.036133+0000) 2024-02-20T01:19:52.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:52.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:52.036360+0000) 2024-02-20T01:19:53.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:53.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:53.036519+0000) 2024-02-20T01:19:54.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:54.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:54.036737+0000) 2024-02-20T01:19:55.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:55.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:55.036933+0000) 2024-02-20T01:19:56.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:56.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:56.037154+0000) 2024-02-20T01:19:57.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:57.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:57.037414+0000) 2024-02-20T01:19:58.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:58.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:58.037655+0000) 2024-02-20T01:19:59.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:19:59.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:19:59.037809+0000) 2024-02-20T01:20:00.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:00.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:00.038045+0000) 2024-02-20T01:20:01.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:01.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:01.038309+0000) 2024-02-20T01:20:02.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:02.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:02.038565+0000) 2024-02-20T01:20:03.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:03.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:03.038839+0000) 2024-02-20T01:20:04.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:04.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:04.039017+0000) 2024-02-20T01:20:05.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:05.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:05.039258+0000) 2024-02-20T01:20:06.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:06.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:06.039510+0000) 2024-02-20T01:20:07.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:07.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:07.039766+0000) 2024-02-20T01:20:08.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:08.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:08.040031+0000) 2024-02-20T01:20:09.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:09.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:09.040208+0000) 2024-02-20T01:20:10.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:10.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:10.040431+0000) 2024-02-20T01:20:11.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:11.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:11.040670+0000) 2024-02-20T01:20:12.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:12.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:12.040920+0000) 2024-02-20T01:20:13.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:13.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:13.041129+0000) 2024-02-20T01:20:14.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:14.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:14.041371+0000) 2024-02-20T01:20:15.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:15.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:15.041641+0000) 2024-02-20T01:20:16.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:16.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:16.041849+0000) 2024-02-20T01:20:17.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:17.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:17.042069+0000) 2024-02-20T01:20:18.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:18.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:18.042318+0000) 2024-02-20T01:20:19.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:19.035+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:19.042586+0000) 2024-02-20T01:20:20.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:20.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:20.042796+0000) 2024-02-20T01:20:21.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:21.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:21.042918+0000) 2024-02-20T01:20:22.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:22.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:22.043063+0000) 2024-02-20T01:20:23.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:23.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:23.043259+0000) 2024-02-20T01:20:24.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:24.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:24.043516+0000) 2024-02-20T01:20:25.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:25.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:25.043754+0000) 2024-02-20T01:20:26.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:26.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:26.044000+0000) 2024-02-20T01:20:27.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:27.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:27.044169+0000) 2024-02-20T01:20:28.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:28.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:28.044393+0000) 2024-02-20T01:20:29.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:29.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:29.044643+0000) 2024-02-20T01:20:30.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:30.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:30.044818+0000) 2024-02-20T01:20:31.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:31.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:31.045033+0000) 2024-02-20T01:20:32.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:32.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:32.045282+0000) 2024-02-20T01:20:33.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:33.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:33.045534+0000) 2024-02-20T01:20:34.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:34.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:34.045787+0000) 2024-02-20T01:20:35.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:35.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:35.046016+0000) 2024-02-20T01:20:36.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:36.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:36.046300+0000) 2024-02-20T01:20:37.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:37.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:37.046512+0000) 2024-02-20T01:20:38.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:38.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:38.046805+0000) 2024-02-20T01:20:39.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:39.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:39.046944+0000) 2024-02-20T01:20:40.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:40.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:40.047190+0000) 2024-02-20T01:20:41.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:41.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:41.047327+0000) 2024-02-20T01:20:42.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:42.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:42.047541+0000) 2024-02-20T01:20:43.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:43.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:43.047758+0000) 2024-02-20T01:20:44.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:44.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:44.047954+0000) 2024-02-20T01:20:45.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:45.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:45.048237+0000) 2024-02-20T01:20:46.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:46.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:46.048474+0000) 2024-02-20T01:20:47.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:47.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:47.048712+0000) 2024-02-20T01:20:48.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:48.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:48.048937+0000) 2024-02-20T01:20:49.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:49.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:49.049195+0000) 2024-02-20T01:20:50.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:50.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:50.049358+0000) 2024-02-20T01:20:51.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:51.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:51.049633+0000) 2024-02-20T01:20:52.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:52.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:52.049819+0000) 2024-02-20T01:20:53.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:53.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:53.050064+0000) 2024-02-20T01:20:54.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:54.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:54.050304+0000) 2024-02-20T01:20:55.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:55.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:55.050533+0000) 2024-02-20T01:20:56.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:56.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:56.050805+0000) 2024-02-20T01:20:57.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:57.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:57.051015+0000) 2024-02-20T01:20:58.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:58.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:58.051160+0000) 2024-02-20T01:20:59.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:20:59.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:20:59.051339+0000) 2024-02-20T01:21:00.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:00.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:00.051592+0000) 2024-02-20T01:21:01.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:01.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:01.051761+0000) 2024-02-20T01:21:02.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:02.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:02.052006+0000) 2024-02-20T01:21:03.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:03.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:03.052126+0000) 2024-02-20T01:21:04.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:04.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:04.052347+0000) 2024-02-20T01:21:05.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:05.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:05.052580+0000) 2024-02-20T01:21:06.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:06.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:06.052781+0000) 2024-02-20T01:21:07.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:07.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:07.052986+0000) 2024-02-20T01:21:08.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:08.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:08.053145+0000) 2024-02-20T01:21:09.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:09.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:09.053300+0000) 2024-02-20T01:21:10.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:10.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:10.053482+0000) 2024-02-20T01:21:11.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:11.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:11.053642+0000) 2024-02-20T01:21:12.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:12.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:12.053858+0000) 2024-02-20T01:21:13.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:13.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:13.054084+0000) 2024-02-20T01:21:14.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:14.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:14.054263+0000) 2024-02-20T01:21:15.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:15.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:15.054435+0000) 2024-02-20T01:21:16.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:16.046+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:16.054617+0000) 2024-02-20T01:21:17.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:17.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:17.054805+0000) 2024-02-20T01:21:18.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:18.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:18.054970+0000) 2024-02-20T01:21:19.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:19.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:19.055108+0000) 2024-02-20T01:21:20.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:20.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:20.055231+0000) 2024-02-20T01:21:21.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:21.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:21.055407+0000) 2024-02-20T01:21:22.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:22.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:22.055582+0000) 2024-02-20T01:21:23.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:23.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:23.055754+0000) 2024-02-20T01:21:24.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:24.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:24.056014+0000) 2024-02-20T01:21:25.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:25.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:25.056267+0000) 2024-02-20T01:21:26.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:26.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:26.056498+0000) 2024-02-20T01:21:27.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:27.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:27.056729+0000) 2024-02-20T01:21:28.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:28.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:28.056899+0000) 2024-02-20T01:21:29.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:29.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:29.057075+0000) 2024-02-20T01:21:30.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:30.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:30.057329+0000) 2024-02-20T01:21:31.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:31.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:31.057565+0000) 2024-02-20T01:21:32.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:32.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:32.057779+0000) 2024-02-20T01:21:33.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:33.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:33.057992+0000) 2024-02-20T01:21:34.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:34.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:34.058192+0000) 2024-02-20T01:21:35.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:35.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:35.058430+0000) 2024-02-20T01:21:36.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:36.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:36.058685+0000) 2024-02-20T01:21:37.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:37.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:37.058916+0000) 2024-02-20T01:21:38.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:38.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:38.059044+0000) 2024-02-20T01:21:39.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:39.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:39.059216+0000) 2024-02-20T01:21:40.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:40.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:40.059427+0000) 2024-02-20T01:21:41.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:41.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:41.059634+0000) 2024-02-20T01:21:42.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:42.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:42.059890+0000) 2024-02-20T01:21:43.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:43.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:43.060028+0000) 2024-02-20T01:21:44.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:44.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:44.060266+0000) 2024-02-20T01:21:45.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:45.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:45.060405+0000) 2024-02-20T01:21:46.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:46.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:46.060620+0000) 2024-02-20T01:21:47.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:47.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:47.060771+0000) 2024-02-20T01:21:48.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:48.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:48.060917+0000) 2024-02-20T01:21:49.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:49.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:49.061120+0000) 2024-02-20T01:21:50.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:50.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:50.061289+0000) 2024-02-20T01:21:51.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:51.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:51.061526+0000) 2024-02-20T01:21:52.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:52.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:52.061795+0000) 2024-02-20T01:21:53.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:53.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:53.062001+0000) 2024-02-20T01:21:54.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:54.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:54.062276+0000) 2024-02-20T01:21:55.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:55.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:55.062561+0000) 2024-02-20T01:21:56.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:56.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:56.062826+0000) 2024-02-20T01:21:57.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:57.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:57.063119+0000) 2024-02-20T01:21:58.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:58.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:58.063362+0000) 2024-02-20T01:21:59.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:21:59.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:21:59.063609+0000) 2024-02-20T01:22:00.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:00.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:00.063906+0000) 2024-02-20T01:22:01.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:01.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:01.064106+0000) 2024-02-20T01:22:02.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:02.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:02.064408+0000) 2024-02-20T01:22:03.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:03.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:03.064618+0000) 2024-02-20T01:22:04.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:04.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:04.064790+0000) 2024-02-20T01:22:05.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:05.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:05.065090+0000) 2024-02-20T01:22:06.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:06.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:06.065321+0000) 2024-02-20T01:22:07.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:07.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:07.065581+0000) 2024-02-20T01:22:08.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:08.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:08.065868+0000) 2024-02-20T01:22:09.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:09.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:09.066082+0000) 2024-02-20T01:22:10.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:10.061+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:10.066356+0000) 2024-02-20T01:22:11.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:11.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:11.066588+0000) 2024-02-20T01:22:12.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:12.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:12.066774+0000) 2024-02-20T01:22:13.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:13.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:13.067011+0000) 2024-02-20T01:22:14.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:14.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:14.067251+0000) 2024-02-20T01:22:15.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:15.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:15.067521+0000) 2024-02-20T01:22:16.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:16.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:16.067783+0000) 2024-02-20T01:22:17.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:17.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:17.068066+0000) 2024-02-20T01:22:18.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:18.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:18.068354+0000) 2024-02-20T01:22:19.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:19.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:19.068559+0000) 2024-02-20T01:22:20.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:20.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:20.068756+0000) 2024-02-20T01:22:21.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:21.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:21.069051+0000) 2024-02-20T01:22:22.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:22.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:22.069342+0000) 2024-02-20T01:22:23.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:23.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:23.069574+0000) 2024-02-20T01:22:24.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:24.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:24.069751+0000) 2024-02-20T01:22:25.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:25.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:25.070029+0000) 2024-02-20T01:22:26.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:26.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:26.070278+0000) 2024-02-20T01:22:27.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:27.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:27.070545+0000) 2024-02-20T01:22:28.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:28.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:28.070847+0000) 2024-02-20T01:22:29.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:29.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:29.070980+0000) 2024-02-20T01:22:30.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:30.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:30.071155+0000) 2024-02-20T01:22:31.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:31.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:31.071401+0000) 2024-02-20T01:22:32.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:32.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:32.071654+0000) 2024-02-20T01:22:33.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:33.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:33.071937+0000) 2024-02-20T01:22:34.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:34.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:34.072212+0000) 2024-02-20T01:22:35.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:35.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:35.072421+0000) 2024-02-20T01:22:36.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:36.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:36.072638+0000) 2024-02-20T01:22:37.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:37.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:37.072879+0000) 2024-02-20T01:22:38.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:38.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:38.073213+0000) 2024-02-20T01:22:39.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:39.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:39.073392+0000) 2024-02-20T01:22:40.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:40.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:40.073619+0000) 2024-02-20T01:22:41.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:41.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:41.073860+0000) 2024-02-20T01:22:42.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:42.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:42.074113+0000) 2024-02-20T01:22:43.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:43.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:43.074389+0000) 2024-02-20T01:22:44.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:44.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:44.074660+0000) 2024-02-20T01:22:45.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:45.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:45.074852+0000) 2024-02-20T01:22:46.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:46.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:46.075111+0000) 2024-02-20T01:22:47.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:47.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:47.075368+0000) 2024-02-20T01:22:48.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:48.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:48.075611+0000) 2024-02-20T01:22:49.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:49.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:49.075859+0000) 2024-02-20T01:22:50.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:50.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:50.076011+0000) 2024-02-20T01:22:51.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:51.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:51.076149+0000) 2024-02-20T01:22:52.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:52.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:52.076314+0000) 2024-02-20T01:22:53.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:53.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:53.076475+0000) 2024-02-20T01:22:54.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:54.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:54.076602+0000) 2024-02-20T01:22:55.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:55.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:55.076816+0000) 2024-02-20T01:22:56.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:56.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:56.076994+0000) 2024-02-20T01:22:57.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:57.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:57.077177+0000) 2024-02-20T01:22:58.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:58.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:58.077401+0000) 2024-02-20T01:22:59.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:22:59.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:22:59.077620+0000) 2024-02-20T01:23:00.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:00.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:00.077801+0000) 2024-02-20T01:23:01.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:01.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:01.077986+0000) 2024-02-20T01:23:02.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:02.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:02.078241+0000) 2024-02-20T01:23:03.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:03.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:03.078406+0000) 2024-02-20T01:23:04.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:04.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:04.078563+0000) 2024-02-20T01:23:05.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:05.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:05.078864+0000) 2024-02-20T01:23:06.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:06.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:06.079035+0000) 2024-02-20T01:23:07.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:07.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:07.079174+0000) 2024-02-20T01:23:08.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:08.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:08.079377+0000) 2024-02-20T01:23:09.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:09.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:09.079561+0000) 2024-02-20T01:23:10.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:10.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:10.079817+0000) 2024-02-20T01:23:11.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:11.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:11.079989+0000) 2024-02-20T01:23:12.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:12.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:12.080260+0000) 2024-02-20T01:23:13.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:13.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:13.080452+0000) 2024-02-20T01:23:14.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:14.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:14.080699+0000) 2024-02-20T01:23:15.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:15.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:15.080936+0000) 2024-02-20T01:23:16.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:16.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:16.081174+0000) 2024-02-20T01:23:17.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:17.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:17.081417+0000) 2024-02-20T01:23:18.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:18.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:18.081691+0000) 2024-02-20T01:23:19.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:19.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:19.081864+0000) 2024-02-20T01:23:20.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:20.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:20.082069+0000) 2024-02-20T01:23:21.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:21.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:21.082312+0000) 2024-02-20T01:23:22.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:22.076+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:22.082553+0000) 2024-02-20T01:23:23.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:23.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:23.082849+0000) 2024-02-20T01:23:24.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:24.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:24.082984+0000) 2024-02-20T01:23:25.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:25.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:25.083183+0000) 2024-02-20T01:23:26.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:26.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:26.083368+0000) 2024-02-20T01:23:27.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:27.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:27.083564+0000) 2024-02-20T01:23:28.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:28.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:28.083834+0000) 2024-02-20T01:23:29.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:29.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:29.084010+0000) 2024-02-20T01:23:30.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:30.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:30.084217+0000) 2024-02-20T01:23:31.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:31.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:31.084422+0000) 2024-02-20T01:23:32.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:32.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:32.084689+0000) 2024-02-20T01:23:33.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:33.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:33.084938+0000) 2024-02-20T01:23:34.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:34.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:34.085168+0000) 2024-02-20T01:23:35.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:35.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:35.085408+0000) 2024-02-20T01:23:36.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:36.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:36.085621+0000) 2024-02-20T01:23:37.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:37.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:37.085877+0000) 2024-02-20T01:23:38.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:38.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:38.086074+0000) 2024-02-20T01:23:39.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:39.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:39.086338+0000) 2024-02-20T01:23:40.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:40.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:40.086618+0000) 2024-02-20T01:23:41.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:41.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:41.086853+0000) 2024-02-20T01:23:42.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:42.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:42.087032+0000) 2024-02-20T01:23:43.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:43.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:43.087176+0000) 2024-02-20T01:23:44.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:44.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:44.087374+0000) 2024-02-20T01:23:45.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:45.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:45.087566+0000) 2024-02-20T01:23:46.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:46.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:46.087783+0000) 2024-02-20T01:23:47.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:47.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:47.087961+0000) 2024-02-20T01:23:48.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:48.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:48.088244+0000) 2024-02-20T01:23:49.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:49.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:49.088483+0000) 2024-02-20T01:23:50.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:50.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:50.088709+0000) 2024-02-20T01:23:51.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:51.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:51.088882+0000) 2024-02-20T01:23:52.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:52.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:52.089180+0000) 2024-02-20T01:23:53.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:53.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:53.089374+0000) 2024-02-20T01:23:54.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:54.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:54.089649+0000) 2024-02-20T01:23:55.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:55.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:55.089869+0000) 2024-02-20T01:23:56.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:56.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:56.090059+0000) 2024-02-20T01:23:57.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:57.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:57.090295+0000) 2024-02-20T01:23:58.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:58.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:58.090466+0000) 2024-02-20T01:23:59.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:23:59.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:23:59.090742+0000) 2024-02-20T01:24:00.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:00.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:00.090864+0000) 2024-02-20T01:24:01.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:01.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:01.091106+0000) 2024-02-20T01:24:02.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:02.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:02.091362+0000) 2024-02-20T01:24:03.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:03.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:03.091595+0000) 2024-02-20T01:24:04.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:04.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:04.091865+0000) 2024-02-20T01:24:05.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:05.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:05.092028+0000) 2024-02-20T01:24:06.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:06.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:06.092230+0000) 2024-02-20T01:24:07.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:07.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:07.092433+0000) 2024-02-20T01:24:08.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:08.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:08.092672+0000) 2024-02-20T01:24:09.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:09.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:09.092919+0000) 2024-02-20T01:24:10.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:10.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:10.093146+0000) 2024-02-20T01:24:11.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:11.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:11.093322+0000) 2024-02-20T01:24:12.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:12.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:12.093478+0000) 2024-02-20T01:24:13.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:13.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:13.093697+0000) 2024-02-20T01:24:14.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:14.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:14.093987+0000) 2024-02-20T01:24:15.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:15.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:15.094228+0000) 2024-02-20T01:24:16.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:16.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:16.094419+0000) 2024-02-20T01:24:17.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:17.087+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:17.094681+0000) 2024-02-20T01:24:18.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:18.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:18.094903+0000) 2024-02-20T01:24:19.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:19.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:19.095100+0000) 2024-02-20T01:24:20.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:20.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:20.095294+0000) 2024-02-20T01:24:21.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:21.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:21.095542+0000) 2024-02-20T01:24:22.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:22.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:22.095805+0000) 2024-02-20T01:24:23.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:23.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:23.096077+0000) 2024-02-20T01:24:24.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:24.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:24.096218+0000) 2024-02-20T01:24:25.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:25.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:25.096492+0000) 2024-02-20T01:24:26.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:26.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:26.096664+0000) 2024-02-20T01:24:27.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:27.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:27.096924+0000) 2024-02-20T01:24:28.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:28.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:28.097151+0000) 2024-02-20T01:24:29.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:29.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:29.097348+0000) 2024-02-20T01:24:30.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:30.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:30.097584+0000) 2024-02-20T01:24:31.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:31.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:31.097754+0000) 2024-02-20T01:24:32.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:32.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:32.097954+0000) 2024-02-20T01:24:33.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:33.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:33.098206+0000) 2024-02-20T01:24:34.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:34.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:34.098445+0000) 2024-02-20T01:24:35.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:35.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:35.098613+0000) 2024-02-20T01:24:36.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:36.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:36.098830+0000) 2024-02-20T01:24:37.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:37.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:37.098971+0000) 2024-02-20T01:24:38.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:38.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:38.099170+0000) 2024-02-20T01:24:39.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:39.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:39.099359+0000) 2024-02-20T01:24:40.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:40.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:40.099607+0000) 2024-02-20T01:24:41.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:41.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:41.099884+0000) 2024-02-20T01:24:42.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:42.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:42.100088+0000) 2024-02-20T01:24:43.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:43.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:43.100337+0000) 2024-02-20T01:24:44.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:44.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:44.100546+0000) 2024-02-20T01:24:45.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:45.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:45.100792+0000) 2024-02-20T01:24:46.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:46.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:46.101008+0000) 2024-02-20T01:24:47.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:47.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:47.101204+0000) 2024-02-20T01:24:48.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:48.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:48.101400+0000) 2024-02-20T01:24:49.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:49.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:49.101620+0000) 2024-02-20T01:24:50.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:50.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:50.101813+0000) 2024-02-20T01:24:51.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:51.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:51.102059+0000) 2024-02-20T01:24:52.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:52.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:52.102302+0000) 2024-02-20T01:24:53.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:53.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:53.102507+0000) 2024-02-20T01:24:54.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:54.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:54.102793+0000) 2024-02-20T01:24:55.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:55.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:55.102993+0000) 2024-02-20T01:24:56.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:56.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:56.103179+0000) 2024-02-20T01:24:57.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:57.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:57.103425+0000) 2024-02-20T01:24:58.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:58.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:58.103665+0000) 2024-02-20T01:24:59.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:24:59.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:24:59.103922+0000) 2024-02-20T01:25:00.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:00.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:00.104100+0000) 2024-02-20T01:25:01.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:01.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:01.104270+0000) 2024-02-20T01:25:02.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:02.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:02.104539+0000) 2024-02-20T01:25:03.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:03.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:03.104786+0000) 2024-02-20T01:25:04.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:04.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:04.105029+0000) 2024-02-20T01:25:05.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:05.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:05.105229+0000) 2024-02-20T01:25:06.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:06.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:06.105472+0000) 2024-02-20T01:25:07.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:07.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:07.105713+0000) 2024-02-20T01:25:08.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:08.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:08.106007+0000) 2024-02-20T01:25:09.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:09.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:09.106280+0000) 2024-02-20T01:25:10.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:10.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:10.106425+0000) 2024-02-20T01:25:11.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:11.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:11.106635+0000) 2024-02-20T01:25:12.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:12.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:12.106887+0000) 2024-02-20T01:25:13.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:13.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:13.107037+0000) 2024-02-20T01:25:14.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:14.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:14.107301+0000) 2024-02-20T01:25:15.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:15.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:15.107537+0000) 2024-02-20T01:25:16.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:16.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:16.107713+0000) 2024-02-20T01:25:17.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:17.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:17.107939+0000) 2024-02-20T01:25:18.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:18.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:18.108173+0000) 2024-02-20T01:25:19.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:19.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:19.108350+0000) 2024-02-20T01:25:20.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:20.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:20.108569+0000) 2024-02-20T01:25:21.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:21.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:21.108793+0000) 2024-02-20T01:25:22.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:22.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:22.108972+0000) 2024-02-20T01:25:23.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:23.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:23.109240+0000) 2024-02-20T01:25:24.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:24.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:24.109505+0000) 2024-02-20T01:25:25.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:25.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:25.109758+0000) 2024-02-20T01:25:26.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:26.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:26.109962+0000) 2024-02-20T01:25:27.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:27.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:27.110203+0000) 2024-02-20T01:25:28.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:28.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:28.110405+0000) 2024-02-20T01:25:29.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:29.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:29.110610+0000) 2024-02-20T01:25:30.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:30.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:30.110843+0000) 2024-02-20T01:25:31.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:31.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:31.111056+0000) 2024-02-20T01:25:32.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:32.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:32.111245+0000) 2024-02-20T01:25:33.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:33.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:33.111520+0000) 2024-02-20T01:25:34.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:34.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:34.111794+0000) 2024-02-20T01:25:35.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:35.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:35.112062+0000) 2024-02-20T01:25:36.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:36.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:36.112323+0000) 2024-02-20T01:25:37.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:37.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:37.112608+0000) 2024-02-20T01:25:38.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:38.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:38.112823+0000) 2024-02-20T01:25:39.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:39.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:39.113094+0000) 2024-02-20T01:25:40.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:40.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:40.113387+0000) 2024-02-20T01:25:41.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:41.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:41.113573+0000) 2024-02-20T01:25:42.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:42.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:42.113818+0000) 2024-02-20T01:25:43.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:43.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:43.114007+0000) 2024-02-20T01:25:44.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:44.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:44.114236+0000) 2024-02-20T01:25:45.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:45.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:45.114453+0000) 2024-02-20T01:25:46.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:46.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:46.114669+0000) 2024-02-20T01:25:47.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:47.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:47.114914+0000) 2024-02-20T01:25:48.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:48.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:48.115103+0000) 2024-02-20T01:25:49.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:49.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:49.115347+0000) 2024-02-20T01:25:50.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:50.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:50.115576+0000) 2024-02-20T01:25:51.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:51.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:51.115776+0000) 2024-02-20T01:25:52.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:52.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:52.115987+0000) 2024-02-20T01:25:53.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:53.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:53.116234+0000) 2024-02-20T01:25:54.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:54.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:54.116504+0000) 2024-02-20T01:25:55.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:55.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:55.116726+0000) 2024-02-20T01:25:56.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:56.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:56.116951+0000) 2024-02-20T01:25:57.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:57.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:57.117211+0000) 2024-02-20T01:25:58.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:58.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:58.117436+0000) 2024-02-20T01:25:59.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:25:59.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:25:59.117603+0000) 2024-02-20T01:26:00.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:00.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:00.117831+0000) 2024-02-20T01:26:01.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:01.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:01.118082+0000) 2024-02-20T01:26:02.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:02.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:02.118404+0000) 2024-02-20T01:26:03.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:03.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:03.118639+0000) 2024-02-20T01:26:04.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:04.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:04.118850+0000) 2024-02-20T01:26:05.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:05.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:05.119073+0000) 2024-02-20T01:26:06.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:06.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:06.119272+0000) 2024-02-20T01:26:07.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:07.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:07.119413+0000) 2024-02-20T01:26:08.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:08.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:08.119659+0000) 2024-02-20T01:26:09.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:09.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:09.119877+0000) 2024-02-20T01:26:10.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:10.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:10.120085+0000) 2024-02-20T01:26:11.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:11.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:11.120314+0000) 2024-02-20T01:26:12.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:12.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:12.120577+0000) 2024-02-20T01:26:13.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:13.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:13.120814+0000) 2024-02-20T01:26:14.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:14.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:14.120983+0000) 2024-02-20T01:26:15.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:15.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:15.121210+0000) 2024-02-20T01:26:16.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:16.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:16.121486+0000) 2024-02-20T01:26:17.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:17.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:17.121748+0000) 2024-02-20T01:26:18.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:18.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:18.122018+0000) 2024-02-20T01:26:19.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:19.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:19.122249+0000) 2024-02-20T01:26:20.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:20.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:20.122408+0000) 2024-02-20T01:26:21.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:21.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:21.122613+0000) 2024-02-20T01:26:22.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:22.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:22.122877+0000) 2024-02-20T01:26:23.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:23.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:23.123065+0000) 2024-02-20T01:26:24.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:24.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:24.123296+0000) 2024-02-20T01:26:25.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:25.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:25.123494+0000) 2024-02-20T01:26:26.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:26.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:26.123702+0000) 2024-02-20T01:26:27.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:27.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:27.123933+0000) 2024-02-20T01:26:28.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:28.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:28.124206+0000) 2024-02-20T01:26:29.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:29.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:29.124434+0000) 2024-02-20T01:26:30.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:30.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:30.124672+0000) 2024-02-20T01:26:31.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:31.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:31.124927+0000) 2024-02-20T01:26:32.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:32.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:32.125198+0000) 2024-02-20T01:26:33.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:33.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:33.125458+0000) 2024-02-20T01:26:34.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:34.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:34.125722+0000) 2024-02-20T01:26:35.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:35.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:35.126006+0000) 2024-02-20T01:26:36.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:36.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:36.126265+0000) 2024-02-20T01:26:37.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:37.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:37.126515+0000) 2024-02-20T01:26:38.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:38.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:38.126809+0000) 2024-02-20T01:26:39.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:39.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:39.127030+0000) 2024-02-20T01:26:40.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:40.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:40.127177+0000) 2024-02-20T01:26:41.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:41.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:41.127417+0000) 2024-02-20T01:26:42.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:42.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:42.127671+0000) 2024-02-20T01:26:43.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:43.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:43.127953+0000) 2024-02-20T01:26:44.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:44.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:44.128102+0000) 2024-02-20T01:26:45.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:45.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:45.128295+0000) 2024-02-20T01:26:46.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:46.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:46.128487+0000) 2024-02-20T01:26:47.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:47.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:47.128730+0000) 2024-02-20T01:26:48.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:48.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:48.128987+0000) 2024-02-20T01:26:49.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:49.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:49.129301+0000) 2024-02-20T01:26:50.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:50.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:50.129549+0000) 2024-02-20T01:26:51.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:51.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:51.129803+0000) 2024-02-20T01:26:52.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:52.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:52.130068+0000) 2024-02-20T01:26:53.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:53.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:53.130328+0000) 2024-02-20T01:26:54.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:54.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:54.130559+0000) 2024-02-20T01:26:55.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:55.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:55.130825+0000) 2024-02-20T01:26:56.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:56.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:56.131004+0000) 2024-02-20T01:26:57.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:57.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:57.131203+0000) 2024-02-20T01:26:58.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:58.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:58.131372+0000) 2024-02-20T01:26:59.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:26:59.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:26:59.131607+0000) 2024-02-20T01:27:00.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:00.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:00.131813+0000) 2024-02-20T01:27:01.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:01.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:01.132072+0000) 2024-02-20T01:27:02.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:02.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:02.132276+0000) 2024-02-20T01:27:03.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:03.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:03.132487+0000) 2024-02-20T01:27:04.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:04.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:04.132732+0000) 2024-02-20T01:27:05.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:05.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:05.132987+0000) 2024-02-20T01:27:06.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:06.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:06.133250+0000) 2024-02-20T01:27:07.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:07.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:07.133441+0000) 2024-02-20T01:27:08.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:08.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:08.133672+0000) 2024-02-20T01:27:09.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:09.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:09.133936+0000) 2024-02-20T01:27:10.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:10.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:10.134163+0000) 2024-02-20T01:27:11.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:11.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:11.134360+0000) 2024-02-20T01:27:12.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:12.128+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:12.134592+0000) 2024-02-20T01:27:13.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:13.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:13.134849+0000) 2024-02-20T01:27:14.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:14.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:14.135049+0000) 2024-02-20T01:27:15.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:15.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:15.135210+0000) 2024-02-20T01:27:16.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:16.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:16.135390+0000) 2024-02-20T01:27:17.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:17.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:17.135645+0000) 2024-02-20T01:27:18.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:18.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:18.135890+0000) 2024-02-20T01:27:19.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:19.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:19.136150+0000) 2024-02-20T01:27:20.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:20.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:20.136422+0000) 2024-02-20T01:27:21.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:21.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:21.136670+0000) 2024-02-20T01:27:22.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:22.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:22.136936+0000) 2024-02-20T01:27:23.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:23.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:23.137119+0000) 2024-02-20T01:27:24.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:24.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:24.137348+0000) 2024-02-20T01:27:25.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:25.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:25.137542+0000) 2024-02-20T01:27:26.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:26.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:26.137717+0000) 2024-02-20T01:27:27.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:27.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:27.137918+0000) 2024-02-20T01:27:28.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:28.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:28.138095+0000) 2024-02-20T01:27:29.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:29.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:29.138293+0000) 2024-02-20T01:27:30.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:30.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:30.138475+0000) 2024-02-20T01:27:31.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:31.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:31.138762+0000) 2024-02-20T01:27:32.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:32.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:32.139029+0000) 2024-02-20T01:27:33.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:33.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:33.139298+0000) 2024-02-20T01:27:34.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:34.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:34.139533+0000) 2024-02-20T01:27:35.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:35.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:35.139784+0000) 2024-02-20T01:27:36.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:36.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:36.139976+0000) 2024-02-20T01:27:37.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:37.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:37.140235+0000) 2024-02-20T01:27:38.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:38.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:38.140502+0000) 2024-02-20T01:27:39.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:39.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:39.140767+0000) 2024-02-20T01:27:40.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:40.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:40.141051+0000) 2024-02-20T01:27:41.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:41.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:41.141337+0000) 2024-02-20T01:27:42.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:42.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:42.141650+0000) 2024-02-20T01:27:43.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:43.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:43.141922+0000) 2024-02-20T01:27:44.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:44.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:44.142123+0000) 2024-02-20T01:27:45.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:45.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:45.142355+0000) 2024-02-20T01:27:46.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:46.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:46.142640+0000) 2024-02-20T01:27:47.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:47.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:47.142899+0000) 2024-02-20T01:27:48.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:48.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:48.143077+0000) 2024-02-20T01:27:49.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:49.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:49.143331+0000) 2024-02-20T01:27:50.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:50.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:50.143499+0000) 2024-02-20T01:27:51.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:51.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:51.143747+0000) 2024-02-20T01:27:52.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:52.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:52.144004+0000) 2024-02-20T01:27:53.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:53.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:53.144215+0000) 2024-02-20T01:27:54.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:54.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:54.144393+0000) 2024-02-20T01:27:55.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:55.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:55.144583+0000) 2024-02-20T01:27:56.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:56.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:56.144813+0000) 2024-02-20T01:27:57.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:57.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:57.145062+0000) 2024-02-20T01:27:58.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:58.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:58.145287+0000) 2024-02-20T01:27:59.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:27:59.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:27:59.145522+0000) 2024-02-20T01:28:00.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:00.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:00.145776+0000) 2024-02-20T01:28:01.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:01.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:01.146076+0000) 2024-02-20T01:28:02.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:02.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:02.146304+0000) 2024-02-20T01:28:03.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:03.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:03.146519+0000) 2024-02-20T01:28:04.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:04.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:04.146788+0000) 2024-02-20T01:28:05.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:05.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:05.147002+0000) 2024-02-20T01:28:06.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:06.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:06.147235+0000) 2024-02-20T01:28:07.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:07.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:07.147409+0000) 2024-02-20T01:28:08.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:08.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:08.147640+0000) 2024-02-20T01:28:09.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:09.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:09.147895+0000) 2024-02-20T01:28:10.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:10.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:10.148038+0000) 2024-02-20T01:28:11.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:11.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:11.148268+0000) 2024-02-20T01:28:12.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:12.148+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:12.148531+0000) 2024-02-20T01:28:13.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:13.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:13.148771+0000) 2024-02-20T01:28:14.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:14.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:14.149031+0000) 2024-02-20T01:28:15.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:15.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:15.149277+0000) 2024-02-20T01:28:16.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:16.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:16.149537+0000) 2024-02-20T01:28:17.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:17.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:17.149818+0000) 2024-02-20T01:28:18.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:18.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:18.150089+0000) 2024-02-20T01:28:19.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:19.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:19.150280+0000) 2024-02-20T01:28:20.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:20.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:20.150472+0000) 2024-02-20T01:28:21.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:21.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:21.150601+0000) 2024-02-20T01:28:22.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:22.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:22.150867+0000) 2024-02-20T01:28:23.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:23.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:23.151047+0000) 2024-02-20T01:28:24.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:24.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:24.151218+0000) 2024-02-20T01:28:25.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:25.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:25.151471+0000) 2024-02-20T01:28:26.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:26.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:26.151732+0000) 2024-02-20T01:28:27.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:27.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:27.151981+0000) 2024-02-20T01:28:28.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:28.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:28.152257+0000) 2024-02-20T01:28:29.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:29.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:29.152496+0000) 2024-02-20T01:28:30.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:30.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:30.152788+0000) 2024-02-20T01:28:31.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:31.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:31.153021+0000) 2024-02-20T01:28:32.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:32.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:32.153239+0000) 2024-02-20T01:28:33.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:33.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:33.153480+0000) 2024-02-20T01:28:34.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:34.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:34.153737+0000) 2024-02-20T01:28:35.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:35.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:35.153998+0000) 2024-02-20T01:28:36.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:36.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:36.154218+0000) 2024-02-20T01:28:37.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:37.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:37.154438+0000) 2024-02-20T01:28:38.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:38.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:38.154648+0000) 2024-02-20T01:28:39.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:39.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:39.154838+0000) 2024-02-20T01:28:40.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:40.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:40.155029+0000) 2024-02-20T01:28:41.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:41.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:41.155208+0000) 2024-02-20T01:28:42.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:42.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:42.155448+0000) 2024-02-20T01:28:43.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:43.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:43.155625+0000) 2024-02-20T01:28:44.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:44.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:44.155859+0000) 2024-02-20T01:28:45.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:45.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:45.156087+0000) 2024-02-20T01:28:46.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:46.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:46.156364+0000) 2024-02-20T01:28:47.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:47.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:47.156564+0000) 2024-02-20T01:28:48.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:48.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:48.156813+0000) 2024-02-20T01:28:49.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:49.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:49.157020+0000) 2024-02-20T01:28:50.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:50.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:50.157213+0000) 2024-02-20T01:28:51.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:51.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:51.157489+0000) 2024-02-20T01:28:52.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:52.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:52.157731+0000) 2024-02-20T01:28:53.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:53.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:53.157969+0000) 2024-02-20T01:28:54.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:54.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:54.158151+0000) 2024-02-20T01:28:55.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:55.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:55.158391+0000) 2024-02-20T01:28:56.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:56.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:56.158623+0000) 2024-02-20T01:28:57.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:57.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:57.158811+0000) 2024-02-20T01:28:58.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:58.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:58.159010+0000) 2024-02-20T01:28:59.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:28:59.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:28:59.159244+0000) 2024-02-20T01:29:00.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:00.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:00.159498+0000) 2024-02-20T01:29:01.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:01.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:01.159747+0000) 2024-02-20T01:29:02.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:02.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:02.160029+0000) 2024-02-20T01:29:03.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:03.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:03.160272+0000) 2024-02-20T01:29:04.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:04.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:04.160484+0000) 2024-02-20T01:29:05.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:05.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:05.160666+0000) 2024-02-20T01:29:06.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:06.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:06.160933+0000) 2024-02-20T01:29:07.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:07.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:07.161213+0000) 2024-02-20T01:29:08.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:08.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:08.161484+0000) 2024-02-20T01:29:09.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:09.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:09.161713+0000) 2024-02-20T01:29:10.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:10.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:10.161914+0000) 2024-02-20T01:29:11.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:11.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:11.162154+0000) 2024-02-20T01:29:12.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:12.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:12.162434+0000) 2024-02-20T01:29:13.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:13.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:13.162673+0000) 2024-02-20T01:29:14.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:14.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:14.162882+0000) 2024-02-20T01:29:15.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:15.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:15.163129+0000) 2024-02-20T01:29:16.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:16.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:16.163313+0000) 2024-02-20T01:29:17.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:17.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:17.163456+0000) 2024-02-20T01:29:18.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:18.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:18.163719+0000) 2024-02-20T01:29:19.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:19.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:19.163979+0000) 2024-02-20T01:29:20.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:20.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:20.164218+0000) 2024-02-20T01:29:21.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:21.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:21.164455+0000) 2024-02-20T01:29:22.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:22.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:22.164728+0000) 2024-02-20T01:29:23.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:23.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:23.164939+0000) 2024-02-20T01:29:24.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:24.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:24.165077+0000) 2024-02-20T01:29:25.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:25.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:25.165260+0000) 2024-02-20T01:29:26.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:26.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:26.165503+0000) 2024-02-20T01:29:27.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:27.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:27.165720+0000) 2024-02-20T01:29:28.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:28.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:28.165997+0000) 2024-02-20T01:29:29.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:29.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:29.166173+0000) 2024-02-20T01:29:30.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:30.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:30.166376+0000) 2024-02-20T01:29:31.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:31.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:31.166593+0000) 2024-02-20T01:29:32.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:32.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:32.166775+0000) 2024-02-20T01:29:33.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:33.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:33.167034+0000) 2024-02-20T01:29:34.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:34.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:34.167267+0000) 2024-02-20T01:29:35.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:35.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:35.167521+0000) 2024-02-20T01:29:36.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:36.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:36.167764+0000) 2024-02-20T01:29:37.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:37.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:37.167899+0000) 2024-02-20T01:29:38.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:38.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:38.168136+0000) 2024-02-20T01:29:39.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:39.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:39.168388+0000) 2024-02-20T01:29:40.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:40.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:40.168588+0000) 2024-02-20T01:29:41.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:41.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:41.168735+0000) 2024-02-20T01:29:42.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:42.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:42.168964+0000) 2024-02-20T01:29:43.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:43.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:43.169216+0000) 2024-02-20T01:29:44.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:44.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:44.169455+0000) 2024-02-20T01:29:45.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:45.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:45.169648+0000) 2024-02-20T01:29:46.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:46.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:46.169863+0000) 2024-02-20T01:29:47.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:47.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:47.170136+0000) 2024-02-20T01:29:48.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:48.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:48.170376+0000) 2024-02-20T01:29:49.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:49.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:49.170535+0000) 2024-02-20T01:29:50.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:50.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:50.170693+0000) 2024-02-20T01:29:51.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:51.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:51.170939+0000) 2024-02-20T01:29:52.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:52.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:52.171190+0000) 2024-02-20T01:29:53.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:53.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:53.171329+0000) 2024-02-20T01:29:54.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:54.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:54.171583+0000) 2024-02-20T01:29:55.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:55.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:55.171836+0000) 2024-02-20T01:29:56.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:56.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:56.172071+0000) 2024-02-20T01:29:57.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:57.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:57.172306+0000) 2024-02-20T01:29:58.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:58.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:58.172553+0000) 2024-02-20T01:29:59.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:29:59.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:29:59.172808+0000) 2024-02-20T01:30:00.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:00.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:00.173049+0000) 2024-02-20T01:30:01.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:01.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:01.173273+0000) 2024-02-20T01:30:02.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:02.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:02.173450+0000) 2024-02-20T01:30:03.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:03.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:03.173691+0000) 2024-02-20T01:30:04.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:04.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:04.173935+0000) 2024-02-20T01:30:05.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:05.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:05.174188+0000) 2024-02-20T01:30:06.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:06.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:06.174490+0000) 2024-02-20T01:30:07.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:07.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:07.174775+0000) 2024-02-20T01:30:08.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:08.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:08.174965+0000) 2024-02-20T01:30:09.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:09.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:09.175173+0000) 2024-02-20T01:30:10.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:10.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:10.175347+0000) 2024-02-20T01:30:11.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:11.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:11.175586+0000) 2024-02-20T01:30:12.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:12.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:12.175774+0000) 2024-02-20T01:30:13.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:13.174+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:13.175970+0000) 2024-02-20T01:30:14.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:14.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:14.176197+0000) 2024-02-20T01:30:15.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:15.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:15.176441+0000) 2024-02-20T01:30:16.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:16.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:16.176697+0000) 2024-02-20T01:30:17.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:17.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:17.176900+0000) 2024-02-20T01:30:18.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:18.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:18.177159+0000) 2024-02-20T01:30:19.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:19.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:19.177413+0000) 2024-02-20T01:30:20.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:20.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:20.177620+0000) 2024-02-20T01:30:21.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:21.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:21.177887+0000) 2024-02-20T01:30:22.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:22.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:22.178132+0000) 2024-02-20T01:30:23.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:23.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:23.178302+0000) 2024-02-20T01:30:24.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:24.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:24.178565+0000) 2024-02-20T01:30:25.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:25.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:25.178794+0000) 2024-02-20T01:30:26.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:26.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:26.178963+0000) 2024-02-20T01:30:27.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:27.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:27.179140+0000) 2024-02-20T01:30:28.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:28.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:28.179417+0000) 2024-02-20T01:30:29.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:29.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:29.179675+0000) 2024-02-20T01:30:30.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:30.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:30.179921+0000) 2024-02-20T01:30:31.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:31.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:31.180102+0000) 2024-02-20T01:30:32.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:32.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:32.180299+0000) 2024-02-20T01:30:33.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:33.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:33.180494+0000) 2024-02-20T01:30:34.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:34.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:34.180727+0000) 2024-02-20T01:30:35.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:35.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:35.180986+0000) 2024-02-20T01:30:36.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:36.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:36.181168+0000) 2024-02-20T01:30:37.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:37.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:37.181353+0000) 2024-02-20T01:30:38.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:38.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:38.181614+0000) 2024-02-20T01:30:39.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:39.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:39.181858+0000) 2024-02-20T01:30:40.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:40.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:40.182096+0000) 2024-02-20T01:30:41.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:41.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:41.182330+0000) 2024-02-20T01:30:42.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:42.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:42.182616+0000) 2024-02-20T01:30:43.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:43.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:43.182795+0000) 2024-02-20T01:30:44.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:44.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:44.182981+0000) 2024-02-20T01:30:45.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:45.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:45.183224+0000) 2024-02-20T01:30:46.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:46.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:46.183471+0000) 2024-02-20T01:30:47.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:47.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:47.183643+0000) 2024-02-20T01:30:48.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:48.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:48.183881+0000) 2024-02-20T01:30:49.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:49.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:49.184150+0000) 2024-02-20T01:30:50.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:50.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:50.184324+0000) 2024-02-20T01:30:51.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:51.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:51.184506+0000) 2024-02-20T01:30:52.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:52.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:52.184740+0000) 2024-02-20T01:30:53.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:53.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:53.184919+0000) 2024-02-20T01:30:54.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:54.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:54.185075+0000) 2024-02-20T01:30:55.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:55.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:55.185266+0000) 2024-02-20T01:30:56.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:56.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:56.185507+0000) 2024-02-20T01:30:57.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:57.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:57.185781+0000) 2024-02-20T01:30:58.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:58.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:58.185982+0000) 2024-02-20T01:30:59.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:30:59.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:30:59.186227+0000) 2024-02-20T01:31:00.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:00.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:00.186542+0000) 2024-02-20T01:31:01.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:01.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:01.186834+0000) 2024-02-20T01:31:02.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:02.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:02.186995+0000) 2024-02-20T01:31:03.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:03.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:03.187242+0000) 2024-02-20T01:31:04.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:04.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:04.187479+0000) 2024-02-20T01:31:05.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:05.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:05.187719+0000) 2024-02-20T01:31:06.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:06.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:06.187887+0000) 2024-02-20T01:31:07.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:07.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:07.188131+0000) 2024-02-20T01:31:08.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:08.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:08.188346+0000) 2024-02-20T01:31:09.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:09.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:09.188594+0000) 2024-02-20T01:31:10.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:10.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:10.188766+0000) 2024-02-20T01:31:11.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:11.185+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:11.188943+0000) 2024-02-20T01:31:12.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:12.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:12.189152+0000) 2024-02-20T01:31:13.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:13.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:13.189411+0000) 2024-02-20T01:31:14.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:14.189+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:14.189664+0000) 2024-02-20T01:31:15.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:15.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:15.189907+0000) 2024-02-20T01:31:16.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:16.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:16.190176+0000) 2024-02-20T01:31:17.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:17.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:17.190364+0000) 2024-02-20T01:31:18.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:18.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:18.190596+0000) 2024-02-20T01:31:19.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:19.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:19.190785+0000) 2024-02-20T01:31:20.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:20.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:20.191044+0000) 2024-02-20T01:31:21.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:21.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:21.191240+0000) 2024-02-20T01:31:22.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:22.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:22.191487+0000) 2024-02-20T01:31:23.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:23.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:23.191664+0000) 2024-02-20T01:31:24.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:24.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:24.191924+0000) 2024-02-20T01:31:25.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:25.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:25.192121+0000) 2024-02-20T01:31:26.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:26.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:26.192298+0000) 2024-02-20T01:31:27.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:27.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:27.192574+0000) 2024-02-20T01:31:28.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:28.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:28.192745+0000) 2024-02-20T01:31:29.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:29.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:29.192956+0000) 2024-02-20T01:31:30.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:30.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:30.193242+0000) 2024-02-20T01:31:31.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:31.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:31.193438+0000) 2024-02-20T01:31:32.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:32.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:32.193580+0000) 2024-02-20T01:31:33.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:33.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:33.193836+0000) 2024-02-20T01:31:34.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:34.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:34.194085+0000) 2024-02-20T01:31:35.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:35.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:35.194335+0000) 2024-02-20T01:31:36.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:36.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:36.194602+0000) 2024-02-20T01:31:37.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:37.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:37.194860+0000) 2024-02-20T01:31:38.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:38.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:38.195100+0000) 2024-02-20T01:31:39.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:39.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:39.195361+0000) 2024-02-20T01:31:40.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:40.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:40.195617+0000) 2024-02-20T01:31:41.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:41.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:41.195856+0000) 2024-02-20T01:31:42.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:42.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:42.196103+0000) 2024-02-20T01:31:43.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:43.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:43.196357+0000) 2024-02-20T01:31:44.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:44.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:44.196572+0000) 2024-02-20T01:31:45.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:45.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:45.196828+0000) 2024-02-20T01:31:46.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:46.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:46.197113+0000) 2024-02-20T01:31:47.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:47.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:47.197371+0000) 2024-02-20T01:31:48.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:48.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:48.197581+0000) 2024-02-20T01:31:49.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:49.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:49.197833+0000) 2024-02-20T01:31:50.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:50.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:50.198089+0000) 2024-02-20T01:31:51.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:51.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:51.198297+0000) 2024-02-20T01:31:52.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:52.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:52.198504+0000) 2024-02-20T01:31:53.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:53.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:53.198777+0000) 2024-02-20T01:31:54.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:54.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:54.199035+0000) 2024-02-20T01:31:55.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:55.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:55.199307+0000) 2024-02-20T01:31:56.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:56.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:56.199513+0000) 2024-02-20T01:31:57.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:57.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:57.199760+0000) 2024-02-20T01:31:58.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:58.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:58.200060+0000) 2024-02-20T01:31:59.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:31:59.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:31:59.200257+0000) 2024-02-20T01:32:00.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:00.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:00.200473+0000) 2024-02-20T01:32:01.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:01.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:01.200669+0000) 2024-02-20T01:32:02.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:02.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:02.200915+0000) 2024-02-20T01:32:03.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:03.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:03.201102+0000) 2024-02-20T01:32:04.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:04.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:04.201397+0000) 2024-02-20T01:32:05.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:05.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:05.201564+0000) 2024-02-20T01:32:06.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:06.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:06.201797+0000) 2024-02-20T01:32:07.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:07.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:07.202053+0000) 2024-02-20T01:32:08.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:08.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:08.202240+0000) 2024-02-20T01:32:09.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:09.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:09.202430+0000) 2024-02-20T01:32:10.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:10.196+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:10.202674+0000) 2024-02-20T01:32:11.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:11.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:11.202914+0000) 2024-02-20T01:32:12.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:12.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:12.203110+0000) 2024-02-20T01:32:13.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:13.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:13.203311+0000) 2024-02-20T01:32:14.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:14.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:14.203565+0000) 2024-02-20T01:32:15.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:15.200+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:15.203893+0000) 2024-02-20T01:32:16.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:16.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:16.204128+0000) 2024-02-20T01:32:17.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:17.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:17.204332+0000) 2024-02-20T01:32:18.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:18.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:18.204556+0000) 2024-02-20T01:32:19.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:19.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:19.204790+0000) 2024-02-20T01:32:20.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:20.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:20.204990+0000) 2024-02-20T01:32:21.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:21.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:21.205147+0000) 2024-02-20T01:32:22.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:22.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:22.205396+0000) 2024-02-20T01:32:23.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:23.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:23.205641+0000) 2024-02-20T01:32:24.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:24.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:24.205866+0000) 2024-02-20T01:32:25.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:25.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:25.206069+0000) 2024-02-20T01:32:26.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:26.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:26.206325+0000) 2024-02-20T01:32:27.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:27.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:27.206562+0000) 2024-02-20T01:32:28.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:28.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:28.206792+0000) 2024-02-20T01:32:29.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:29.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:29.207038+0000) 2024-02-20T01:32:30.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:30.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:30.207242+0000) 2024-02-20T01:32:31.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:31.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:31.207504+0000) 2024-02-20T01:32:32.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:32.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:32.207778+0000) 2024-02-20T01:32:33.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:33.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:33.208018+0000) 2024-02-20T01:32:34.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:34.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:34.208172+0000) 2024-02-20T01:32:35.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:35.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:35.208356+0000) 2024-02-20T01:32:36.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:36.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:36.208554+0000) 2024-02-20T01:32:37.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:37.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:37.208739+0000) 2024-02-20T01:32:38.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:38.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:38.208980+0000) 2024-02-20T01:32:39.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:39.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:39.209257+0000) 2024-02-20T01:32:40.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:40.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:40.209440+0000) 2024-02-20T01:32:41.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:41.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:41.209664+0000) 2024-02-20T01:32:42.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:42.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:42.209911+0000) 2024-02-20T01:32:43.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:43.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:43.210145+0000) 2024-02-20T01:32:44.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:44.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:44.210323+0000) 2024-02-20T01:32:45.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:45.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:45.210536+0000) 2024-02-20T01:32:46.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:46.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:46.210817+0000) 2024-02-20T01:32:47.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:47.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:47.211050+0000) 2024-02-20T01:32:48.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:48.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:48.211214+0000) 2024-02-20T01:32:49.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:49.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:49.211354+0000) 2024-02-20T01:32:50.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:50.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:50.211550+0000) 2024-02-20T01:32:51.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:51.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:51.211745+0000) 2024-02-20T01:32:52.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:52.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:52.211968+0000) 2024-02-20T01:32:53.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:53.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:53.212210+0000) 2024-02-20T01:32:54.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:54.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:54.212439+0000) 2024-02-20T01:32:55.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:55.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:55.212638+0000) 2024-02-20T01:32:56.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:56.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:56.212842+0000) 2024-02-20T01:32:57.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:57.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:57.213023+0000) 2024-02-20T01:32:58.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:58.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:58.213266+0000) 2024-02-20T01:32:59.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:32:59.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:32:59.213540+0000) 2024-02-20T01:33:00.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:00.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:00.213827+0000) 2024-02-20T01:33:01.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:01.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:01.214100+0000) 2024-02-20T01:33:02.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:02.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:02.214383+0000) 2024-02-20T01:33:03.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:03.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:03.214587+0000) 2024-02-20T01:33:04.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:04.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:04.214735+0000) 2024-02-20T01:33:05.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:05.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:05.214933+0000) 2024-02-20T01:33:06.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:06.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:06.215219+0000) 2024-02-20T01:33:07.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:07.215+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:07.215498+0000) 2024-02-20T01:33:08.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:08.215+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:08.215784+0000) 2024-02-20T01:33:09.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:09.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:09.215963+0000) 2024-02-20T01:33:10.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:10.215+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:10.216226+0000) 2024-02-20T01:33:11.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:11.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:11.216518+0000) 2024-02-20T01:33:12.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:12.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:12.216728+0000) 2024-02-20T01:33:13.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:13.211+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:13.216995+0000) 2024-02-20T01:33:14.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:14.215+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:14.217286+0000) 2024-02-20T01:33:15.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:15.215+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:15.217460+0000) 2024-02-20T01:33:16.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:16.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:16.217713+0000) 2024-02-20T01:33:17.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:17.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:17.217916+0000) 2024-02-20T01:33:18.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:18.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:18.218075+0000) 2024-02-20T01:33:19.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:19.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:19.218275+0000) 2024-02-20T01:33:20.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:20.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:20.218580+0000) 2024-02-20T01:33:21.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:21.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:21.218866+0000) 2024-02-20T01:33:22.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:22.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:22.219194+0000) 2024-02-20T01:33:23.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:23.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:23.219490+0000) 2024-02-20T01:33:24.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:24.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:24.219692+0000) 2024-02-20T01:33:25.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:25.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:25.219958+0000) 2024-02-20T01:33:26.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:26.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:26.220245+0000) 2024-02-20T01:33:27.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:27.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:27.220528+0000) 2024-02-20T01:33:28.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:28.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:28.220823+0000) 2024-02-20T01:33:29.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:29.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:29.221127+0000) 2024-02-20T01:33:30.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:30.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:30.221417+0000) 2024-02-20T01:33:31.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:31.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:31.221644+0000) 2024-02-20T01:33:32.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:32.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:32.221907+0000) 2024-02-20T01:33:33.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:33.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:33.222146+0000) 2024-02-20T01:33:34.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:34.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:34.222275+0000) 2024-02-20T01:33:35.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:35.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:35.222458+0000) 2024-02-20T01:33:36.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:36.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:36.222649+0000) 2024-02-20T01:33:37.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:37.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:37.222854+0000) 2024-02-20T01:33:38.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:38.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:38.223087+0000) 2024-02-20T01:33:39.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:39.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:39.223223+0000) 2024-02-20T01:33:40.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:40.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:40.223510+0000) 2024-02-20T01:33:41.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:41.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:41.223705+0000) 2024-02-20T01:33:42.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:42.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:42.224025+0000) 2024-02-20T01:33:43.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:43.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:43.224298+0000) 2024-02-20T01:33:44.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:44.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:44.224463+0000) 2024-02-20T01:33:45.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:45.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:45.224765+0000) 2024-02-20T01:33:46.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:46.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:46.225087+0000) 2024-02-20T01:33:47.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:47.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:47.225311+0000) 2024-02-20T01:33:48.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:48.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:48.225578+0000) 2024-02-20T01:33:49.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:49.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:49.225769+0000) 2024-02-20T01:33:50.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:50.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:50.226031+0000) 2024-02-20T01:33:51.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:51.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:51.226322+0000) 2024-02-20T01:33:52.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:52.222+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:52.226563+0000) 2024-02-20T01:33:53.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:53.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:53.226866+0000) 2024-02-20T01:33:54.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:54.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:54.227153+0000) 2024-02-20T01:33:55.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:55.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:55.227320+0000) 2024-02-20T01:33:56.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:56.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:56.227493+0000) 2024-02-20T01:33:57.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:57.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:57.227709+0000) 2024-02-20T01:33:58.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:58.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:58.227958+0000) 2024-02-20T01:33:59.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:33:59.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:33:59.228175+0000) 2024-02-20T01:34:00.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:00.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:00.228460+0000) 2024-02-20T01:34:01.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:01.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:01.228703+0000) 2024-02-20T01:34:02.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:02.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:02.228991+0000) 2024-02-20T01:34:03.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:03.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:03.229282+0000) 2024-02-20T01:34:04.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:04.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:04.229507+0000) 2024-02-20T01:34:05.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:05.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:05.229765+0000) 2024-02-20T01:34:06.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:06.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:06.230020+0000) 2024-02-20T01:34:07.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:07.226+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:07.230225+0000) 2024-02-20T01:34:08.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:08.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:08.230480+0000) 2024-02-20T01:34:09.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:09.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:09.230822+0000) 2024-02-20T01:34:10.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:10.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:10.231111+0000) 2024-02-20T01:34:11.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:11.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:11.231249+0000) 2024-02-20T01:34:12.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:12.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:12.231526+0000) 2024-02-20T01:34:13.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:13.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:13.231791+0000) 2024-02-20T01:34:14.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:14.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:14.232019+0000) 2024-02-20T01:34:15.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:15.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:15.232274+0000) 2024-02-20T01:34:16.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:16.230+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:16.232467+0000) 2024-02-20T01:34:17.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:17.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:17.232673+0000) 2024-02-20T01:34:18.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:18.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:18.232956+0000) 2024-02-20T01:34:19.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:19.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:19.233151+0000) 2024-02-20T01:34:20.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:20.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:20.233418+0000) 2024-02-20T01:34:21.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:21.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:21.233577+0000) 2024-02-20T01:34:22.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:22.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:22.233805+0000) 2024-02-20T01:34:23.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:23.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:23.234024+0000) 2024-02-20T01:34:24.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:24.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:24.234313+0000) 2024-02-20T01:34:25.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:25.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:25.234602+0000) 2024-02-20T01:34:26.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:26.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:26.234920+0000) 2024-02-20T01:34:27.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:27.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:27.235159+0000) 2024-02-20T01:34:28.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:28.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:28.235437+0000) 2024-02-20T01:34:29.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:29.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:29.235690+0000) 2024-02-20T01:34:30.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:30.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:30.235885+0000) 2024-02-20T01:34:31.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:31.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:31.236080+0000) 2024-02-20T01:34:32.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:32.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:32.236336+0000) 2024-02-20T01:34:33.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:33.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:33.236608+0000) 2024-02-20T01:34:34.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:34.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:34.236833+0000) 2024-02-20T01:34:35.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:35.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:35.237096+0000) 2024-02-20T01:34:36.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:36.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:36.237311+0000) 2024-02-20T01:34:37.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:37.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:37.237568+0000) 2024-02-20T01:34:38.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:38.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:38.237875+0000) 2024-02-20T01:34:39.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:39.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:39.238169+0000) 2024-02-20T01:34:40.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:40.233+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:40.238407+0000) 2024-02-20T01:34:41.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:41.237+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:41.238744+0000) 2024-02-20T01:34:42.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:42.237+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:42.238983+0000) 2024-02-20T01:34:43.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:43.237+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:43.239216+0000) 2024-02-20T01:34:44.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:44.237+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:44.239486+0000) 2024-02-20T01:34:45.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:45.237+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:45.239733+0000) 2024-02-20T01:34:46.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:46.237+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:46.239981+0000) 2024-02-20T01:34:47.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:47.237+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:47.240220+0000) 2024-02-20T01:34:48.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:48.237+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:48.240456+0000) 2024-02-20T01:34:49.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:49.237+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:49.240699+0000) 2024-02-20T01:34:50.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:50.237+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:50.240835+0000) 2024-02-20T01:34:51.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:51.237+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:51.241007+0000) 2024-02-20T01:34:52.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:52.237+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:52.241228+0000) 2024-02-20T01:34:53.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:53.237+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:53.241465+0000) 2024-02-20T01:34:54.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:54.237+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:54.241739+0000) 2024-02-20T01:34:55.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:55.237+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:55.241976+0000) 2024-02-20T01:34:56.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:56.237+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:56.242190+0000) 2024-02-20T01:34:57.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:57.237+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:57.242434+0000) 2024-02-20T01:34:58.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:58.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:58.242607+0000) 2024-02-20T01:34:59.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:34:59.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:34:59.242852+0000) 2024-02-20T01:35:00.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:00.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:00.243049+0000) 2024-02-20T01:35:01.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:01.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:01.243226+0000) 2024-02-20T01:35:02.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:02.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:02.243526+0000) 2024-02-20T01:35:03.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:03.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:03.243813+0000) 2024-02-20T01:35:04.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:04.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:04.244000+0000) 2024-02-20T01:35:05.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:05.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:05.244240+0000) 2024-02-20T01:35:06.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:06.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:06.244493+0000) 2024-02-20T01:35:07.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:07.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:07.244736+0000) 2024-02-20T01:35:08.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:08.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:08.244921+0000) 2024-02-20T01:35:09.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:09.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:09.245179+0000) 2024-02-20T01:35:10.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:10.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:10.245448+0000) 2024-02-20T01:35:11.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:11.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:11.245685+0000) 2024-02-20T01:35:12.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:12.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:12.245934+0000) 2024-02-20T01:35:13.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:13.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:13.246100+0000) 2024-02-20T01:35:14.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:14.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:14.246330+0000) 2024-02-20T01:35:15.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:15.241+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:15.246591+0000) 2024-02-20T01:35:16.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:16.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:16.246862+0000) 2024-02-20T01:35:17.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:17.245+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:17.247083+0000) 2024-02-20T01:35:18.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:18.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:18.247264+0000) 2024-02-20T01:35:19.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:19.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:19.247437+0000) 2024-02-20T01:35:20.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:20.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:20.247671+0000) 2024-02-20T01:35:21.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:21.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:21.247927+0000) 2024-02-20T01:35:22.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:22.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:22.248169+0000) 2024-02-20T01:35:23.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:23.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:23.248398+0000) 2024-02-20T01:35:24.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:24.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:24.248642+0000) 2024-02-20T01:35:25.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:25.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:25.248860+0000) 2024-02-20T01:35:26.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:26.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:26.249102+0000) 2024-02-20T01:35:27.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:27.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:27.249331+0000) 2024-02-20T01:35:28.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:28.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:28.249563+0000) 2024-02-20T01:35:29.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:29.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:29.249760+0000) 2024-02-20T01:35:30.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:30.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:30.250006+0000) 2024-02-20T01:35:31.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:31.244+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:31.250271+0000) 2024-02-20T01:35:32.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:32.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:32.250518+0000) 2024-02-20T01:35:33.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:33.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:33.250819+0000) 2024-02-20T01:35:34.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:34.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:34.250993+0000) 2024-02-20T01:35:35.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:35.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:35.251223+0000) 2024-02-20T01:35:36.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:36.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:36.251432+0000) 2024-02-20T01:35:37.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:37.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:37.251694+0000) 2024-02-20T01:35:38.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:38.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:38.251928+0000) 2024-02-20T01:35:39.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:39.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:39.252128+0000) 2024-02-20T01:35:40.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:40.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:40.252375+0000) 2024-02-20T01:35:41.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:41.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:41.252643+0000) 2024-02-20T01:35:42.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:42.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:42.252892+0000) 2024-02-20T01:35:43.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:43.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:43.253123+0000) 2024-02-20T01:35:44.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:44.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:44.253281+0000) 2024-02-20T01:35:45.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:45.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:45.253519+0000) 2024-02-20T01:35:46.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:46.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:46.253791+0000) 2024-02-20T01:35:47.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:47.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:47.253976+0000) 2024-02-20T01:35:48.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:48.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:48.254153+0000) 2024-02-20T01:35:49.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:49.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:49.254394+0000) 2024-02-20T01:35:50.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:50.248+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:50.254560+0000) 2024-02-20T01:35:51.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:51.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:51.254775+0000) 2024-02-20T01:35:52.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:52.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:52.254964+0000) 2024-02-20T01:35:53.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:53.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:53.255187+0000) 2024-02-20T01:35:54.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:54.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:54.255414+0000) 2024-02-20T01:35:55.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:55.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:55.255602+0000) 2024-02-20T01:35:56.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:56.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:56.255809+0000) 2024-02-20T01:35:57.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:57.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:57.256010+0000) 2024-02-20T01:35:58.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:58.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:58.256262+0000) 2024-02-20T01:35:59.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:35:59.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:35:59.256541+0000) 2024-02-20T01:36:00.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:00.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:00.256784+0000) 2024-02-20T01:36:01.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:01.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:01.257030+0000) 2024-02-20T01:36:02.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:02.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:02.257277+0000) 2024-02-20T01:36:03.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:03.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:03.257509+0000) 2024-02-20T01:36:04.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:04.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:04.257702+0000) 2024-02-20T01:36:05.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:05.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:05.257931+0000) 2024-02-20T01:36:06.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:06.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:06.258160+0000) 2024-02-20T01:36:07.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:07.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:07.258347+0000) 2024-02-20T01:36:08.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:08.252+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:08.258601+0000) 2024-02-20T01:36:09.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:09.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:09.258801+0000) 2024-02-20T01:36:10.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:10.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:10.259044+0000) 2024-02-20T01:36:11.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:11.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:11.259222+0000) 2024-02-20T01:36:12.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:12.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:12.259475+0000) 2024-02-20T01:36:13.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:13.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:13.259746+0000) 2024-02-20T01:36:14.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:14.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:14.260022+0000) 2024-02-20T01:36:15.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:15.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:15.260269+0000) 2024-02-20T01:36:16.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:16.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:16.260508+0000) 2024-02-20T01:36:17.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:17.256+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:17.260776+0000) 2024-02-20T01:36:18.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:18.260+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:18.260952+0000) 2024-02-20T01:36:19.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:19.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:19.261098+0000) 2024-02-20T01:36:20.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:20.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:20.261290+0000) 2024-02-20T01:36:21.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:21.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:21.261509+0000) 2024-02-20T01:36:22.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:22.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:22.261722+0000) 2024-02-20T01:36:23.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:23.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:23.261915+0000) 2024-02-20T01:36:24.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:24.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:24.262126+0000) 2024-02-20T01:36:25.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:25.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:25.262364+0000) 2024-02-20T01:36:26.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:26.255+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:26.262588+0000) 2024-02-20T01:36:27.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:27.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:27.262838+0000) 2024-02-20T01:36:28.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:28.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:28.263022+0000) 2024-02-20T01:36:29.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:29.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:29.263188+0000) 2024-02-20T01:36:30.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:30.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:30.263456+0000) 2024-02-20T01:36:31.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:31.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:31.263736+0000) 2024-02-20T01:36:32.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:32.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:32.263933+0000) 2024-02-20T01:36:33.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:33.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:33.264169+0000) 2024-02-20T01:36:34.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:34.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:34.264354+0000) 2024-02-20T01:36:35.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:35.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:35.264608+0000) 2024-02-20T01:36:36.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:36.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:36.264867+0000) 2024-02-20T01:36:37.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:37.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:37.265052+0000) 2024-02-20T01:36:38.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:38.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:38.265273+0000) 2024-02-20T01:36:39.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:39.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:39.265520+0000) 2024-02-20T01:36:40.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:40.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:40.265798+0000) 2024-02-20T01:36:41.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:41.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:41.265999+0000) 2024-02-20T01:36:42.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:42.259+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:42.266194+0000) 2024-02-20T01:36:43.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:43.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:43.266361+0000) 2024-02-20T01:36:44.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:44.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:44.266556+0000) 2024-02-20T01:36:45.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:45.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:45.266755+0000) 2024-02-20T01:36:46.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:46.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:46.266950+0000) 2024-02-20T01:36:47.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:47.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:47.267238+0000) 2024-02-20T01:36:48.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:48.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:48.267426+0000) 2024-02-20T01:36:49.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:49.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:49.267616+0000) 2024-02-20T01:36:50.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:50.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:50.267841+0000) 2024-02-20T01:36:51.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:51.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:51.268111+0000) 2024-02-20T01:36:52.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:52.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:52.268442+0000) 2024-02-20T01:36:53.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:53.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:53.268693+0000) 2024-02-20T01:36:54.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:54.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:54.268903+0000) 2024-02-20T01:36:55.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:55.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:55.269178+0000) 2024-02-20T01:36:56.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:56.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:56.269425+0000) 2024-02-20T01:36:57.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:57.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:57.269664+0000) 2024-02-20T01:36:58.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:58.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:58.269928+0000) 2024-02-20T01:36:59.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:36:59.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:36:59.270197+0000) 2024-02-20T01:37:00.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:00.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:00.270400+0000) 2024-02-20T01:37:01.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:01.263+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:01.270641+0000) 2024-02-20T01:37:02.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:02.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:02.270843+0000) 2024-02-20T01:37:03.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:03.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:03.271088+0000) 2024-02-20T01:37:04.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:04.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:04.271362+0000) 2024-02-20T01:37:05.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:05.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:05.271640+0000) 2024-02-20T01:37:06.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:06.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:06.271835+0000) 2024-02-20T01:37:07.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:07.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:07.272115+0000) 2024-02-20T01:37:08.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:08.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:08.272285+0000) 2024-02-20T01:37:09.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:09.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:09.272481+0000) 2024-02-20T01:37:10.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:10.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:10.272635+0000) 2024-02-20T01:37:11.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:11.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:11.272764+0000) 2024-02-20T01:37:12.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:12.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:12.272975+0000) 2024-02-20T01:37:13.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:13.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:13.273218+0000) 2024-02-20T01:37:14.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:14.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:14.273479+0000) 2024-02-20T01:37:15.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:15.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:15.273719+0000) 2024-02-20T01:37:16.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:16.267+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:16.273909+0000) 2024-02-20T01:37:17.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:17.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:17.274156+0000) 2024-02-20T01:37:18.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:18.271+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:18.274426+0000) 2024-02-20T01:37:19.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:19.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:19.274653+0000) 2024-02-20T01:37:20.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:20.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:20.274897+0000) 2024-02-20T01:37:21.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:21.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:21.275084+0000) 2024-02-20T01:37:22.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:22.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:22.275343+0000) 2024-02-20T01:37:23.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:23.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:23.275583+0000) 2024-02-20T01:37:24.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:24.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:24.275858+0000) 2024-02-20T01:37:25.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:25.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:25.276040+0000) 2024-02-20T01:37:26.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:26.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:26.276299+0000) 2024-02-20T01:37:27.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:27.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:27.276458+0000) 2024-02-20T01:37:28.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:28.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:28.276728+0000) 2024-02-20T01:37:29.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:29.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:29.276989+0000) 2024-02-20T01:37:30.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:30.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:30.277185+0000) 2024-02-20T01:37:31.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:31.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:31.277426+0000) 2024-02-20T01:37:32.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:32.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:32.277672+0000) 2024-02-20T01:37:33.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:33.270+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:33.277907+0000) 2024-02-20T01:37:34.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:34.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:34.278186+0000) 2024-02-20T01:37:35.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:35.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:35.278461+0000) 2024-02-20T01:37:36.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:36.274+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:36.278808+0000) 2024-02-20T01:37:37.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:37.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:37.279064+0000) 2024-02-20T01:37:38.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:38.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:38.279344+0000) 2024-02-20T01:37:39.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:39.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:39.279614+0000) 2024-02-20T01:37:40.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:40.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:40.279895+0000) 2024-02-20T01:37:41.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:41.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:41.280110+0000) 2024-02-20T01:37:42.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:42.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:42.280333+0000) 2024-02-20T01:37:43.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:43.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:43.280571+0000) 2024-02-20T01:37:44.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:44.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:44.280798+0000) 2024-02-20T01:37:45.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:45.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:45.281021+0000) 2024-02-20T01:37:46.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:46.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:46.281275+0000) 2024-02-20T01:37:47.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:47.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:47.281525+0000) 2024-02-20T01:37:48.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:48.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:48.281811+0000) 2024-02-20T01:37:49.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:49.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:49.282007+0000) 2024-02-20T01:37:50.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:50.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:50.282250+0000) 2024-02-20T01:37:51.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:51.278+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:51.282453+0000) 2024-02-20T01:37:52.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:52.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:52.282652+0000) 2024-02-20T01:37:53.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:53.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:53.282830+0000) 2024-02-20T01:37:54.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:54.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:54.282955+0000) 2024-02-20T01:37:55.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:55.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:55.283134+0000) 2024-02-20T01:37:56.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:56.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:56.283329+0000) 2024-02-20T01:37:57.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:57.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:57.283573+0000) 2024-02-20T01:37:58.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:58.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:58.283751+0000) 2024-02-20T01:37:59.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:37:59.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:37:59.283981+0000) 2024-02-20T01:38:00.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:00.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:00.284231+0000) 2024-02-20T01:38:01.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:01.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:01.284426+0000) 2024-02-20T01:38:02.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:02.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:02.284563+0000) 2024-02-20T01:38:03.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:03.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:03.284777+0000) 2024-02-20T01:38:04.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:04.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:04.285011+0000) 2024-02-20T01:38:05.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:05.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:05.285277+0000) 2024-02-20T01:38:06.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:06.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:06.285526+0000) 2024-02-20T01:38:07.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:07.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:07.285789+0000) 2024-02-20T01:38:08.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:08.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:08.286037+0000) 2024-02-20T01:38:09.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:09.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:09.286281+0000) 2024-02-20T01:38:10.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:10.282+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:10.286472+0000) 2024-02-20T01:38:11.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:11.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:11.286805+0000) 2024-02-20T01:38:12.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:12.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:12.287019+0000) 2024-02-20T01:38:13.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:13.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:13.287211+0000) 2024-02-20T01:38:14.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:14.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:14.287413+0000) 2024-02-20T01:38:15.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:15.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:15.287656+0000) 2024-02-20T01:38:16.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:16.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:16.287899+0000) 2024-02-20T01:38:17.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:17.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:17.288115+0000) 2024-02-20T01:38:18.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:18.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:18.288373+0000) 2024-02-20T01:38:19.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:19.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:19.288643+0000) 2024-02-20T01:38:20.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:20.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:20.288879+0000) 2024-02-20T01:38:21.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:21.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:21.289122+0000) 2024-02-20T01:38:22.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:22.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:22.289349+0000) 2024-02-20T01:38:23.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:23.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:23.289614+0000) 2024-02-20T01:38:24.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:24.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:24.289858+0000) 2024-02-20T01:38:25.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:25.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:25.290101+0000) 2024-02-20T01:38:26.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:26.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:26.290302+0000) 2024-02-20T01:38:27.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:27.285+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:27.290571+0000) 2024-02-20T01:38:28.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:28.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:28.290833+0000) 2024-02-20T01:38:29.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:29.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:29.291042+0000) 2024-02-20T01:38:30.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:30.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:30.291271+0000) 2024-02-20T01:38:31.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:31.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:31.291537+0000) 2024-02-20T01:38:32.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:32.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:32.291843+0000) 2024-02-20T01:38:33.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:33.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:33.292033+0000) 2024-02-20T01:38:34.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:34.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:34.292260+0000) 2024-02-20T01:38:35.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:35.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:35.292495+0000) 2024-02-20T01:38:36.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:36.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:36.292764+0000) 2024-02-20T01:38:37.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:37.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:37.292969+0000) 2024-02-20T01:38:38.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:38.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:38.293195+0000) 2024-02-20T01:38:39.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:39.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:39.293451+0000) 2024-02-20T01:38:40.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:40.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:40.293637+0000) 2024-02-20T01:38:41.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:41.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:41.293909+0000) 2024-02-20T01:38:42.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:42.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:42.294177+0000) 2024-02-20T01:38:43.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:43.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:43.294394+0000) 2024-02-20T01:38:44.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:44.289+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:44.294648+0000) 2024-02-20T01:38:45.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:45.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:45.294901+0000) 2024-02-20T01:38:46.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:46.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:46.295207+0000) 2024-02-20T01:38:47.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:47.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:47.295394+0000) 2024-02-20T01:38:48.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:48.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:48.295627+0000) 2024-02-20T01:38:49.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:49.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:49.295872+0000) 2024-02-20T01:38:50.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:50.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:50.296078+0000) 2024-02-20T01:38:51.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:51.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:51.296307+0000) 2024-02-20T01:38:52.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:52.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:52.296544+0000) 2024-02-20T01:38:53.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:53.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:53.296828+0000) 2024-02-20T01:38:54.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:54.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:54.297072+0000) 2024-02-20T01:38:55.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:55.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:55.297339+0000) 2024-02-20T01:38:56.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:56.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:56.297580+0000) 2024-02-20T01:38:57.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:57.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:57.297828+0000) 2024-02-20T01:38:58.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:58.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:58.298051+0000) 2024-02-20T01:38:59.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:38:59.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:38:59.298301+0000) 2024-02-20T01:39:00.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:00.293+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:00.298564+0000) 2024-02-20T01:39:01.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:01.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:01.298857+0000) 2024-02-20T01:39:02.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:02.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:02.299122+0000) 2024-02-20T01:39:03.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:03.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:03.299392+0000) 2024-02-20T01:39:04.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:04.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:04.299674+0000) 2024-02-20T01:39:05.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:05.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:05.299867+0000) 2024-02-20T01:39:06.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:06.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:06.300120+0000) 2024-02-20T01:39:07.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:07.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:07.300343+0000) 2024-02-20T01:39:08.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:08.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:08.300574+0000) 2024-02-20T01:39:09.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:09.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:09.300815+0000) 2024-02-20T01:39:10.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:10.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:10.301070+0000) 2024-02-20T01:39:11.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:11.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:11.301332+0000) 2024-02-20T01:39:12.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:12.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:12.301611+0000) 2024-02-20T01:39:13.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:13.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:13.301897+0000) 2024-02-20T01:39:14.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:14.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:14.302158+0000) 2024-02-20T01:39:15.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:15.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:15.302406+0000) 2024-02-20T01:39:16.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:16.297+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:16.302694+0000) 2024-02-20T01:39:17.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:17.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:17.302883+0000) 2024-02-20T01:39:18.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:18.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:18.303118+0000) 2024-02-20T01:39:19.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:19.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:19.303366+0000) 2024-02-20T01:39:20.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:20.301+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:20.303613+0000) 2024-02-20T01:39:21.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:21.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:21.303877+0000) 2024-02-20T01:39:22.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:22.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:22.304099+0000) 2024-02-20T01:39:23.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:23.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:23.304326+0000) 2024-02-20T01:39:24.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:24.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:24.304511+0000) 2024-02-20T01:39:25.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:25.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:25.304676+0000) 2024-02-20T01:39:26.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:26.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:26.304811+0000) 2024-02-20T01:39:27.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:27.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:27.305045+0000) 2024-02-20T01:39:28.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:28.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:28.305271+0000) 2024-02-20T01:39:29.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:29.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:29.305457+0000) 2024-02-20T01:39:30.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:30.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:30.305678+0000) 2024-02-20T01:39:31.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:31.300+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:31.305939+0000) 2024-02-20T01:39:32.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:32.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:32.306104+0000) 2024-02-20T01:39:33.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:33.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:33.306260+0000) 2024-02-20T01:39:34.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:34.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:34.306491+0000) 2024-02-20T01:39:35.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:35.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:35.306733+0000) 2024-02-20T01:39:36.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:36.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:36.306937+0000) 2024-02-20T01:39:37.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:37.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:37.307240+0000) 2024-02-20T01:39:38.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:38.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:38.307507+0000) 2024-02-20T01:39:39.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:39.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:39.307782+0000) 2024-02-20T01:39:40.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:40.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:40.308065+0000) 2024-02-20T01:39:41.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:41.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:41.308259+0000) 2024-02-20T01:39:42.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:42.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:42.308508+0000) 2024-02-20T01:39:43.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:43.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:43.308704+0000) 2024-02-20T01:39:44.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:44.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:44.308890+0000) 2024-02-20T01:39:45.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:45.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:45.309122+0000) 2024-02-20T01:39:46.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:46.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:46.309319+0000) 2024-02-20T01:39:47.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:47.304+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:47.309545+0000) 2024-02-20T01:39:48.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:48.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:48.309784+0000) 2024-02-20T01:39:49.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:49.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:49.310054+0000) 2024-02-20T01:39:50.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:50.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:50.310304+0000) 2024-02-20T01:39:51.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:51.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:51.310501+0000) 2024-02-20T01:39:52.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:52.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:52.310790+0000) 2024-02-20T01:39:53.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:53.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:53.311054+0000) 2024-02-20T01:39:54.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:54.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:54.311300+0000) 2024-02-20T01:39:55.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:55.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:55.311484+0000) 2024-02-20T01:39:56.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:56.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:56.311738+0000) 2024-02-20T01:39:57.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:57.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:57.311976+0000) 2024-02-20T01:39:58.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:58.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:58.312210+0000) 2024-02-20T01:39:59.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:39:59.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:39:59.312459+0000) 2024-02-20T01:40:00.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:00.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:00.312658+0000) 2024-02-20T01:40:01.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:01.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:01.312878+0000) 2024-02-20T01:40:02.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:02.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:02.313127+0000) 2024-02-20T01:40:03.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:03.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:03.313359+0000) 2024-02-20T01:40:04.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:04.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:04.313597+0000) 2024-02-20T01:40:05.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:05.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:05.313794+0000) 2024-02-20T01:40:06.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:06.308+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:06.314037+0000) 2024-02-20T01:40:07.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:07.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:07.314277+0000) 2024-02-20T01:40:08.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:08.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:08.314520+0000) 2024-02-20T01:40:09.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:09.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:09.314779+0000) 2024-02-20T01:40:10.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:10.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:10.314969+0000) 2024-02-20T01:40:11.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:11.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:11.315219+0000) 2024-02-20T01:40:12.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:12.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:12.315410+0000) 2024-02-20T01:40:13.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:13.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:13.315556+0000) 2024-02-20T01:40:14.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:14.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:14.315810+0000) 2024-02-20T01:40:15.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:15.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:15.316036+0000) 2024-02-20T01:40:16.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:16.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:16.316271+0000) 2024-02-20T01:40:17.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:17.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:17.316522+0000) 2024-02-20T01:40:18.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:18.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:18.316757+0000) 2024-02-20T01:40:19.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:19.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:19.316965+0000) 2024-02-20T01:40:20.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:20.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:20.317171+0000) 2024-02-20T01:40:21.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:21.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:21.317423+0000) 2024-02-20T01:40:22.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:22.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:22.317695+0000) 2024-02-20T01:40:23.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:23.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:23.317887+0000) 2024-02-20T01:40:24.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:24.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:24.318062+0000) 2024-02-20T01:40:25.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:25.311+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:25.318281+0000) 2024-02-20T01:40:26.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:26.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:26.318553+0000) 2024-02-20T01:40:27.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:27.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:27.318833+0000) 2024-02-20T01:40:28.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:28.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:28.319035+0000) 2024-02-20T01:40:29.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:29.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:29.319275+0000) 2024-02-20T01:40:30.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:30.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:30.319540+0000) 2024-02-20T01:40:31.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:31.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:31.319783+0000) 2024-02-20T01:40:32.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:32.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:32.320019+0000) 2024-02-20T01:40:33.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:33.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:33.320243+0000) 2024-02-20T01:40:34.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:34.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:34.320423+0000) 2024-02-20T01:40:35.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:35.319+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:35.320652+0000) 2024-02-20T01:40:36.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:36.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:36.320909+0000) 2024-02-20T01:40:37.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:37.319+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:37.321153+0000) 2024-02-20T01:40:38.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:38.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:38.321384+0000) 2024-02-20T01:40:39.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:39.319+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:39.321625+0000) 2024-02-20T01:40:40.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:40.319+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:40.321827+0000) 2024-02-20T01:40:41.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:41.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:41.322072+0000) 2024-02-20T01:40:42.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:42.319+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:42.322261+0000) 2024-02-20T01:40:43.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:43.315+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:43.322477+0000) 2024-02-20T01:40:44.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:44.319+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:44.322755+0000) 2024-02-20T01:40:45.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:45.319+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:45.322919+0000) 2024-02-20T01:40:46.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:46.319+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:46.323085+0000) 2024-02-20T01:40:47.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:47.319+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:47.323332+0000) 2024-02-20T01:40:48.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:48.319+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:48.323567+0000) 2024-02-20T01:40:49.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:49.319+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:49.323822+0000) 2024-02-20T01:40:50.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:50.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:50.324049+0000) 2024-02-20T01:40:51.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:51.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:51.324296+0000) 2024-02-20T01:40:52.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:52.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:52.324549+0000) 2024-02-20T01:40:53.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:53.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:53.324771+0000) 2024-02-20T01:40:54.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:54.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:54.325003+0000) 2024-02-20T01:40:55.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:55.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:55.325268+0000) 2024-02-20T01:40:56.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:56.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:56.325567+0000) 2024-02-20T01:40:57.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:57.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:57.325841+0000) 2024-02-20T01:40:58.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:58.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:58.326107+0000) 2024-02-20T01:40:59.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:40:59.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:40:59.326294+0000) 2024-02-20T01:41:00.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:00.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:00.326540+0000) 2024-02-20T01:41:01.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:01.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:01.326745+0000) 2024-02-20T01:41:02.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:02.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:02.326924+0000) 2024-02-20T01:41:03.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:03.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:03.327119+0000) 2024-02-20T01:41:04.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:04.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:04.327314+0000) 2024-02-20T01:41:05.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:05.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:05.327491+0000) 2024-02-20T01:41:06.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:06.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:06.327746+0000) 2024-02-20T01:41:07.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:07.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:07.327987+0000) 2024-02-20T01:41:08.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:08.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:08.328227+0000) 2024-02-20T01:41:09.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:09.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:09.328504+0000) 2024-02-20T01:41:10.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:10.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:10.328738+0000) 2024-02-20T01:41:11.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:11.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:11.328985+0000) 2024-02-20T01:41:12.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:12.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:12.329230+0000) 2024-02-20T01:41:13.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:13.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:13.329498+0000) 2024-02-20T01:41:14.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:14.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:14.329678+0000) 2024-02-20T01:41:15.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:15.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:15.329937+0000) 2024-02-20T01:41:16.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:16.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:16.330181+0000) 2024-02-20T01:41:17.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:17.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:17.330414+0000) 2024-02-20T01:41:18.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:18.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:18.330622+0000) 2024-02-20T01:41:19.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:19.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:19.330781+0000) 2024-02-20T01:41:20.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:20.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:20.331031+0000) 2024-02-20T01:41:21.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:21.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:21.331236+0000) 2024-02-20T01:41:22.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:22.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:22.331443+0000) 2024-02-20T01:41:23.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:23.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:23.331666+0000) 2024-02-20T01:41:24.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:24.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:24.331915+0000) 2024-02-20T01:41:25.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:25.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:25.332154+0000) 2024-02-20T01:41:26.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:26.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:26.332355+0000) 2024-02-20T01:41:27.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:27.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:27.332569+0000) 2024-02-20T01:41:28.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:28.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:28.332779+0000) 2024-02-20T01:41:29.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:29.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:29.333037+0000) 2024-02-20T01:41:30.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:30.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:30.333257+0000) 2024-02-20T01:41:31.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:31.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:31.333485+0000) 2024-02-20T01:41:32.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:32.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:32.333734+0000) 2024-02-20T01:41:33.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:33.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:33.333911+0000) 2024-02-20T01:41:34.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:34.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:34.334088+0000) 2024-02-20T01:41:35.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:35.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:35.334318+0000) 2024-02-20T01:41:36.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:36.326+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:36.334462+0000) 2024-02-20T01:41:37.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:37.330+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:37.334676+0000) 2024-02-20T01:41:38.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:38.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:38.334909+0000) 2024-02-20T01:41:39.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:39.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:39.335101+0000) 2024-02-20T01:41:40.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:40.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:40.335336+0000) 2024-02-20T01:41:41.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:41.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:41.335598+0000) 2024-02-20T01:41:42.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:42.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:42.335822+0000) 2024-02-20T01:41:43.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:43.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:43.336065+0000) 2024-02-20T01:41:44.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:44.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:44.336295+0000) 2024-02-20T01:41:45.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:45.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:45.336547+0000) 2024-02-20T01:41:46.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:46.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:46.336771+0000) 2024-02-20T01:41:47.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:47.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:47.337016+0000) 2024-02-20T01:41:48.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:48.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:48.337253+0000) 2024-02-20T01:41:49.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:49.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:49.337499+0000) 2024-02-20T01:41:50.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:50.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:50.337818+0000) 2024-02-20T01:41:51.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:51.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:51.338025+0000) 2024-02-20T01:41:52.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:52.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:52.338295+0000) 2024-02-20T01:41:53.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:53.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:53.338449+0000) 2024-02-20T01:41:54.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:54.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:54.338622+0000) 2024-02-20T01:41:55.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:55.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:55.338875+0000) 2024-02-20T01:41:56.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:56.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:56.339077+0000) 2024-02-20T01:41:57.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:57.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:57.339313+0000) 2024-02-20T01:41:58.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:58.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:58.339561+0000) 2024-02-20T01:41:59.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:41:59.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:41:59.339721+0000) 2024-02-20T01:42:00.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:00.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:00.339945+0000) 2024-02-20T01:42:01.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:01.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:01.340159+0000) 2024-02-20T01:42:02.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:02.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:02.340400+0000) 2024-02-20T01:42:03.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:03.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:03.340622+0000) 2024-02-20T01:42:04.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:04.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:04.340862+0000) 2024-02-20T01:42:05.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:05.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:05.341018+0000) 2024-02-20T01:42:06.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:06.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:06.341199+0000) 2024-02-20T01:42:07.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:07.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:07.341419+0000) 2024-02-20T01:42:08.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:08.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:08.341656+0000) 2024-02-20T01:42:09.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:09.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:09.341792+0000) 2024-02-20T01:42:10.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:10.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:10.342049+0000) 2024-02-20T01:42:11.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:11.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:11.342322+0000) 2024-02-20T01:42:12.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:12.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:12.342553+0000) 2024-02-20T01:42:13.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:13.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:13.342759+0000) 2024-02-20T01:42:14.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:14.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:14.342938+0000) 2024-02-20T01:42:15.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:15.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:15.343163+0000) 2024-02-20T01:42:16.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:16.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:16.343410+0000) 2024-02-20T01:42:17.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:17.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:17.343663+0000) 2024-02-20T01:42:18.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:18.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:18.343895+0000) 2024-02-20T01:42:19.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:19.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:19.344119+0000) 2024-02-20T01:42:20.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:20.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:20.344355+0000) 2024-02-20T01:42:21.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:21.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:21.344567+0000) 2024-02-20T01:42:22.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:22.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:22.344791+0000) 2024-02-20T01:42:23.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:23.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:23.344980+0000) 2024-02-20T01:42:24.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:24.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:24.345134+0000) 2024-02-20T01:42:25.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:25.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:25.345375+0000) 2024-02-20T01:42:26.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:26.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:26.345622+0000) 2024-02-20T01:42:27.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:27.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:27.345859+0000) 2024-02-20T01:42:28.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:28.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:28.346096+0000) 2024-02-20T01:42:29.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:29.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:29.346328+0000) 2024-02-20T01:42:30.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:30.341+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:30.346496+0000) 2024-02-20T01:42:31.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:31.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:31.346791+0000) 2024-02-20T01:42:32.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:32.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:32.346957+0000) 2024-02-20T01:42:33.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:33.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:33.347097+0000) 2024-02-20T01:42:34.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:34.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:34.347295+0000) 2024-02-20T01:42:35.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:35.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:35.347511+0000) 2024-02-20T01:42:36.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:36.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:36.347683+0000) 2024-02-20T01:42:37.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:37.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:37.347909+0000) 2024-02-20T01:42:38.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:38.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:38.348099+0000) 2024-02-20T01:42:39.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:39.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:39.348312+0000) 2024-02-20T01:42:40.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:40.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:40.348564+0000) 2024-02-20T01:42:41.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:41.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:41.348789+0000) 2024-02-20T01:42:42.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:42.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:42.348968+0000) 2024-02-20T01:42:43.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:43.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:43.349203+0000) 2024-02-20T01:42:44.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:44.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:44.349388+0000) 2024-02-20T01:42:45.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:45.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:45.349572+0000) 2024-02-20T01:42:46.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:46.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:46.349823+0000) 2024-02-20T01:42:47.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:47.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:47.350042+0000) 2024-02-20T01:42:48.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:48.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:48.350209+0000) 2024-02-20T01:42:49.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:49.345+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:49.350382+0000) 2024-02-20T01:42:50.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:50.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:50.350614+0000) 2024-02-20T01:42:51.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:51.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:51.350778+0000) 2024-02-20T01:42:52.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:52.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:52.351012+0000) 2024-02-20T01:42:53.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:53.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:53.351255+0000) 2024-02-20T01:42:54.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:54.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:54.351449+0000) 2024-02-20T01:42:55.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:55.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:55.351616+0000) 2024-02-20T01:42:56.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:56.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:56.351815+0000) 2024-02-20T01:42:57.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:57.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:57.352056+0000) 2024-02-20T01:42:58.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:58.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:58.352236+0000) 2024-02-20T01:42:59.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:42:59.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:42:59.352458+0000) 2024-02-20T01:43:00.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:00.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:00.352703+0000) 2024-02-20T01:43:01.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:01.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:01.352945+0000) 2024-02-20T01:43:02.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:02.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:02.353090+0000) 2024-02-20T01:43:03.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:03.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:03.353248+0000) 2024-02-20T01:43:04.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:04.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:04.353486+0000) 2024-02-20T01:43:05.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:05.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:05.353738+0000) 2024-02-20T01:43:06.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:06.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:06.353941+0000) 2024-02-20T01:43:07.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:07.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:07.354190+0000) 2024-02-20T01:43:08.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:08.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:08.354431+0000) 2024-02-20T01:43:09.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:09.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:09.354666+0000) 2024-02-20T01:43:10.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:10.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:10.354830+0000) 2024-02-20T01:43:11.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:11.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:11.355027+0000) 2024-02-20T01:43:12.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:12.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:12.355215+0000) 2024-02-20T01:43:13.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:13.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:13.355466+0000) 2024-02-20T01:43:14.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:14.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:14.355712+0000) 2024-02-20T01:43:15.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:15.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:15.355969+0000) 2024-02-20T01:43:16.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:16.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:16.356208+0000) 2024-02-20T01:43:17.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:17.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:17.356357+0000) 2024-02-20T01:43:18.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:18.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:18.356563+0000) 2024-02-20T01:43:19.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:19.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:19.356790+0000) 2024-02-20T01:43:20.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:20.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:20.357023+0000) 2024-02-20T01:43:21.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:21.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:21.357279+0000) 2024-02-20T01:43:22.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:22.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:22.357450+0000) 2024-02-20T01:43:23.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:23.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:23.357676+0000) 2024-02-20T01:43:24.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:24.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:24.357897+0000) 2024-02-20T01:43:25.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:25.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:25.358105+0000) 2024-02-20T01:43:26.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:26.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:26.358361+0000) 2024-02-20T01:43:27.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:27.352+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:27.358522+0000) 2024-02-20T01:43:28.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:28.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:28.358815+0000) 2024-02-20T01:43:29.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:29.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:29.359008+0000) 2024-02-20T01:43:30.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:30.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:30.359234+0000) 2024-02-20T01:43:31.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:31.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:31.359374+0000) 2024-02-20T01:43:32.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:32.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:32.359580+0000) 2024-02-20T01:43:33.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:33.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:33.359785+0000) 2024-02-20T01:43:34.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:34.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:34.360047+0000) 2024-02-20T01:43:35.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:35.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:35.360293+0000) 2024-02-20T01:43:36.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:36.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:36.360532+0000) 2024-02-20T01:43:37.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:37.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:37.360769+0000) 2024-02-20T01:43:38.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:38.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:38.361006+0000) 2024-02-20T01:43:39.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:39.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:39.361216+0000) 2024-02-20T01:43:40.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:40.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:40.361456+0000) 2024-02-20T01:43:41.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:41.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:41.361714+0000) 2024-02-20T01:43:42.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:42.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:42.361982+0000) 2024-02-20T01:43:43.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:43.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:43.362222+0000) 2024-02-20T01:43:44.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:44.356+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:44.362453+0000) 2024-02-20T01:43:45.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:45.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:45.362623+0000) 2024-02-20T01:43:46.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:46.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:46.362807+0000) 2024-02-20T01:43:47.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:47.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:47.363010+0000) 2024-02-20T01:43:48.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:48.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:48.363286+0000) 2024-02-20T01:43:49.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:49.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:49.363479+0000) 2024-02-20T01:43:50.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:50.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:50.363719+0000) 2024-02-20T01:43:51.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:51.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:51.363966+0000) 2024-02-20T01:43:52.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:52.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:52.364215+0000) 2024-02-20T01:43:53.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:53.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:53.364470+0000) 2024-02-20T01:43:54.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:54.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:54.364662+0000) 2024-02-20T01:43:55.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:55.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:55.364911+0000) 2024-02-20T01:43:56.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:56.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:56.365113+0000) 2024-02-20T01:43:57.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:57.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:57.365391+0000) 2024-02-20T01:43:58.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:58.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:58.365569+0000) 2024-02-20T01:43:59.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:43:59.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:43:59.365839+0000) 2024-02-20T01:44:00.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:00.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:00.366081+0000) 2024-02-20T01:44:01.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:01.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:01.366253+0000) 2024-02-20T01:44:02.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:02.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:02.366420+0000) 2024-02-20T01:44:03.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:03.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:03.366621+0000) 2024-02-20T01:44:04.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:04.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:04.366783+0000) 2024-02-20T01:44:05.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:05.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:05.367023+0000) 2024-02-20T01:44:06.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:06.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:06.367189+0000) 2024-02-20T01:44:07.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:07.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:07.367355+0000) 2024-02-20T01:44:08.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:08.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:08.367614+0000) 2024-02-20T01:44:09.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:09.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:09.367857+0000) 2024-02-20T01:44:10.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:10.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:10.368076+0000) 2024-02-20T01:44:11.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:11.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:11.368255+0000) 2024-02-20T01:44:12.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:12.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:12.368443+0000) 2024-02-20T01:44:13.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:13.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:13.368627+0000) 2024-02-20T01:44:14.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:14.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:14.368876+0000) 2024-02-20T01:44:15.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:15.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:15.369156+0000) 2024-02-20T01:44:16.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:16.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:16.369399+0000) 2024-02-20T01:44:17.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:17.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:17.369617+0000) 2024-02-20T01:44:18.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:18.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:18.369793+0000) 2024-02-20T01:44:19.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:19.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:19.369993+0000) 2024-02-20T01:44:20.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:20.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:20.370288+0000) 2024-02-20T01:44:21.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:21.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:21.370539+0000) 2024-02-20T01:44:22.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:22.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:22.370787+0000) 2024-02-20T01:44:23.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:23.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:23.370947+0000) 2024-02-20T01:44:24.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:24.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:24.371203+0000) 2024-02-20T01:44:25.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:25.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:25.371376+0000) 2024-02-20T01:44:26.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:26.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:26.371609+0000) 2024-02-20T01:44:27.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:27.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:27.371872+0000) 2024-02-20T01:44:28.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:28.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:28.372099+0000) 2024-02-20T01:44:29.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:29.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:29.372307+0000) 2024-02-20T01:44:30.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:30.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:30.372505+0000) 2024-02-20T01:44:31.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:31.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:31.372784+0000) 2024-02-20T01:44:32.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:32.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:32.373045+0000) 2024-02-20T01:44:33.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:33.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:33.373217+0000) 2024-02-20T01:44:34.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:34.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:34.373418+0000) 2024-02-20T01:44:35.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:35.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:35.373660+0000) 2024-02-20T01:44:36.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:36.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:36.373912+0000) 2024-02-20T01:44:37.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:37.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:37.374167+0000) 2024-02-20T01:44:38.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:38.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:38.374457+0000) 2024-02-20T01:44:39.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:39.367+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:39.374669+0000) 2024-02-20T01:44:40.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:40.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:40.374906+0000) 2024-02-20T01:44:41.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:41.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:41.375177+0000) 2024-02-20T01:44:42.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:42.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:42.375421+0000) 2024-02-20T01:44:43.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:43.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:43.375669+0000) 2024-02-20T01:44:44.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:44.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:44.375903+0000) 2024-02-20T01:44:45.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:45.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:45.376134+0000) 2024-02-20T01:44:46.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:46.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:46.376388+0000) 2024-02-20T01:44:47.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:47.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:47.376584+0000) 2024-02-20T01:44:48.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:48.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:48.376809+0000) 2024-02-20T01:44:49.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:49.371+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:49.377060+0000) 2024-02-20T01:44:50.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:50.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:50.377185+0000) 2024-02-20T01:44:51.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:51.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:51.377425+0000) 2024-02-20T01:44:52.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:52.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:52.377697+0000) 2024-02-20T01:44:53.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:53.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:53.377974+0000) 2024-02-20T01:44:54.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:54.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:54.378241+0000) 2024-02-20T01:44:55.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:55.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:55.378492+0000) 2024-02-20T01:44:56.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:56.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:56.378820+0000) 2024-02-20T01:44:57.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:57.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:57.379020+0000) 2024-02-20T01:44:58.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:58.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:58.379216+0000) 2024-02-20T01:44:59.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:44:59.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:44:59.379445+0000) 2024-02-20T01:45:00.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:00.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:00.379697+0000) 2024-02-20T01:45:01.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:01.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:01.379981+0000) 2024-02-20T01:45:02.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:02.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:02.380256+0000) 2024-02-20T01:45:03.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:03.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:03.380530+0000) 2024-02-20T01:45:04.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:04.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:04.380780+0000) 2024-02-20T01:45:05.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:05.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:05.381018+0000) 2024-02-20T01:45:06.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:06.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:06.381159+0000) 2024-02-20T01:45:07.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:07.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:07.381391+0000) 2024-02-20T01:45:08.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:08.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:08.381599+0000) 2024-02-20T01:45:09.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:09.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:09.381833+0000) 2024-02-20T01:45:10.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:10.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:10.382119+0000) 2024-02-20T01:45:11.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:11.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:11.382377+0000) 2024-02-20T01:45:12.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:12.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:12.382615+0000) 2024-02-20T01:45:13.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:13.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:13.382857+0000) 2024-02-20T01:45:14.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:14.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:14.383092+0000) 2024-02-20T01:45:15.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:15.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:15.383286+0000) 2024-02-20T01:45:16.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:16.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:16.383477+0000) 2024-02-20T01:45:17.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:17.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:17.383711+0000) 2024-02-20T01:45:18.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:18.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:18.383885+0000) 2024-02-20T01:45:19.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:19.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:19.384112+0000) 2024-02-20T01:45:20.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:20.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:20.384299+0000) 2024-02-20T01:45:21.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:21.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:21.384467+0000) 2024-02-20T01:45:22.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:22.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:22.384713+0000) 2024-02-20T01:45:23.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:23.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:23.384975+0000) 2024-02-20T01:45:24.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:24.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:24.385205+0000) 2024-02-20T01:45:25.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:25.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:25.385401+0000) 2024-02-20T01:45:26.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:26.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:26.385654+0000) 2024-02-20T01:45:27.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:27.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:27.385926+0000) 2024-02-20T01:45:28.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:28.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:28.386166+0000) 2024-02-20T01:45:29.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:29.378+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:29.386395+0000) 2024-02-20T01:45:30.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:30.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:30.386540+0000) 2024-02-20T01:45:31.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:31.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:31.386831+0000) 2024-02-20T01:45:32.385 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:32.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:32.387054+0000) 2024-02-20T01:45:33.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:33.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:33.387256+0000) 2024-02-20T01:45:34.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:34.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:34.387403+0000) 2024-02-20T01:45:35.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:35.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:35.387601+0000) 2024-02-20T01:45:36.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:36.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:36.387845+0000) 2024-02-20T01:45:37.386 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:37.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:37.388072+0000) 2024-02-20T01:45:38.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:38.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:38.388311+0000) 2024-02-20T01:45:39.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:39.382+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:39.388506+0000) 2024-02-20T01:45:40.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:40.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:40.388687+0000) 2024-02-20T01:45:41.387 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:41.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:41.388942+0000) 2024-02-20T01:45:42.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:42.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:42.389198+0000) 2024-02-20T01:45:43.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:43.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:43.389379+0000) 2024-02-20T01:45:44.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:44.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:44.389621+0000) 2024-02-20T01:45:45.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:45.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:45.389865+0000) 2024-02-20T01:45:46.388 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:46.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:46.390103+0000) 2024-02-20T01:45:47.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:47.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:47.390371+0000) 2024-02-20T01:45:48.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:48.386+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:48.390561+0000) 2024-02-20T01:45:49.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:49.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:49.390799+0000) 2024-02-20T01:45:50.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:50.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:50.390981+0000) 2024-02-20T01:45:51.389 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:51.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:51.391125+0000) 2024-02-20T01:45:52.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:52.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:52.391334+0000) 2024-02-20T01:45:53.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:53.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:53.391540+0000) 2024-02-20T01:45:54.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:54.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:54.391739+0000) 2024-02-20T01:45:55.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:55.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:55.391918+0000) 2024-02-20T01:45:56.390 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:56.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:56.392153+0000) 2024-02-20T01:45:57.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:57.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:57.392400+0000) 2024-02-20T01:45:58.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:58.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:58.392574+0000) 2024-02-20T01:45:59.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:45:59.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:45:59.392796+0000) 2024-02-20T01:46:00.391 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:00.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:00.393071+0000) 2024-02-20T01:46:01.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:01.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:01.393345+0000) 2024-02-20T01:46:02.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:02.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:02.393590+0000) 2024-02-20T01:46:03.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:03.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:03.393814+0000) 2024-02-20T01:46:04.392 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:04.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:04.394037+0000) 2024-02-20T01:46:05.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:05.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:05.394204+0000) 2024-02-20T01:46:06.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:06.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:06.394402+0000) 2024-02-20T01:46:07.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:07.390+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:07.394594+0000) 2024-02-20T01:46:08.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:08.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:08.394824+0000) 2024-02-20T01:46:09.393 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:09.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:09.395053+0000) 2024-02-20T01:46:10.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:10.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:10.395286+0000) 2024-02-20T01:46:11.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:11.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:11.395512+0000) 2024-02-20T01:46:12.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:12.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:12.395749+0000) 2024-02-20T01:46:13.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:13.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:13.396005+0000) 2024-02-20T01:46:14.394 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:14.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:14.396224+0000) 2024-02-20T01:46:15.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:15.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:15.396387+0000) 2024-02-20T01:46:16.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:16.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:16.396613+0000) 2024-02-20T01:46:17.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:17.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:17.396882+0000) 2024-02-20T01:46:18.395 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:18.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:18.397109+0000) 2024-02-20T01:46:19.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:19.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:19.397348+0000) 2024-02-20T01:46:20.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:20.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:20.397625+0000) 2024-02-20T01:46:21.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:21.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:21.397871+0000) 2024-02-20T01:46:22.396 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:22.394+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:22.398128+0000) 2024-02-20T01:46:23.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:23.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:23.398387+0000) 2024-02-20T01:46:24.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:24.393+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:24.398577+0000) 2024-02-20T01:46:25.397 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:25.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:25.399117+0000) 2024-02-20T01:46:26.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:26.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:26.399331+0000) 2024-02-20T01:46:27.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:27.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:27.399588+0000) 2024-02-20T01:46:28.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:28.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:28.399755+0000) 2024-02-20T01:46:29.398 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:29.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:29.399967+0000) 2024-02-20T01:46:30.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:30.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:30.400191+0000) 2024-02-20T01:46:31.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:31.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:31.400429+0000) 2024-02-20T01:46:32.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:32.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:32.400624+0000) 2024-02-20T01:46:33.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:33.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:33.400869+0000) 2024-02-20T01:46:34.399 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:34.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:34.401101+0000) 2024-02-20T01:46:35.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:35.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:35.401336+0000) 2024-02-20T01:46:36.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:36.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:36.401578+0000) 2024-02-20T01:46:37.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:37.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:37.401824+0000) 2024-02-20T01:46:38.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:38.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:38.402038+0000) 2024-02-20T01:46:39.400 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:39.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:39.402247+0000) 2024-02-20T01:46:40.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:40.397+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:40.402427+0000) 2024-02-20T01:46:41.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:41.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:41.402592+0000) 2024-02-20T01:46:42.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:42.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:42.402811+0000) 2024-02-20T01:46:43.401 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:43.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:43.403066+0000) 2024-02-20T01:46:44.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:44.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:44.403304+0000) 2024-02-20T01:46:45.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:45.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:45.403549+0000) 2024-02-20T01:46:46.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:46.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:46.403792+0000) 2024-02-20T01:46:47.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:47.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:47.403983+0000) 2024-02-20T01:46:48.402 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:48.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:48.404164+0000) 2024-02-20T01:46:49.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:49.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:49.404389+0000) 2024-02-20T01:46:50.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:50.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:50.404604+0000) 2024-02-20T01:46:51.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:51.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:51.404772+0000) 2024-02-20T01:46:52.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:52.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:52.404947+0000) 2024-02-20T01:46:53.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:53.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:53.405111+0000) 2024-02-20T01:46:54.403 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:54.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:54.405331+0000) 2024-02-20T01:46:55.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:55.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:55.405455+0000) 2024-02-20T01:46:56.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:56.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:56.405577+0000) 2024-02-20T01:46:57.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:57.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:57.405837+0000) 2024-02-20T01:46:58.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:58.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:58.406071+0000) 2024-02-20T01:46:59.404 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:46:59.401+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:46:59.406271+0000) 2024-02-20T01:47:00.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:00.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:00.406515+0000) 2024-02-20T01:47:01.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:01.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:01.406783+0000) 2024-02-20T01:47:02.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:02.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:02.406955+0000) 2024-02-20T01:47:03.405 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:03.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:03.407085+0000) 2024-02-20T01:47:04.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:04.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:04.407298+0000) 2024-02-20T01:47:05.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:05.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:05.407503+0000) 2024-02-20T01:47:06.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:06.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:06.407664+0000) 2024-02-20T01:47:07.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:07.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:07.407924+0000) 2024-02-20T01:47:08.406 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:08.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:08.408180+0000) 2024-02-20T01:47:09.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:09.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:09.408429+0000) 2024-02-20T01:47:10.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:10.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:10.408628+0000) 2024-02-20T01:47:11.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:11.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:11.408810+0000) 2024-02-20T01:47:12.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:12.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:12.409075+0000) 2024-02-20T01:47:13.407 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:13.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:13.409277+0000) 2024-02-20T01:47:14.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:14.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:14.409539+0000) 2024-02-20T01:47:15.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:15.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:15.409804+0000) 2024-02-20T01:47:16.408 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:16.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:16.410040+0000) 2024-02-20T01:47:17.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:17.405+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:17.410319+0000) 2024-02-20T01:47:18.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:18.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:18.410563+0000) 2024-02-20T01:47:19.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:19.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:19.410855+0000) 2024-02-20T01:47:20.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:20.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:20.410990+0000) 2024-02-20T01:47:21.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:21.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:21.411139+0000) 2024-02-20T01:47:22.409 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:22.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:22.411345+0000) 2024-02-20T01:47:23.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:23.409+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:23.411590+0000) 2024-02-20T01:47:24.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:24.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:24.411851+0000) 2024-02-20T01:47:25.410 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:25.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:25.412087+0000) 2024-02-20T01:47:26.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:26.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:26.412278+0000) 2024-02-20T01:47:27.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:27.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:27.412515+0000) 2024-02-20T01:47:28.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:28.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:28.412766+0000) 2024-02-20T01:47:29.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:29.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:29.412961+0000) 2024-02-20T01:47:30.411 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:30.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:30.413209+0000) 2024-02-20T01:47:31.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:31.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:31.413447+0000) 2024-02-20T01:47:32.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:32.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:32.413670+0000) 2024-02-20T01:47:33.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:33.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:33.413849+0000) 2024-02-20T01:47:34.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:34.408+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:34.414050+0000) 2024-02-20T01:47:35.412 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:35.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:35.414193+0000) 2024-02-20T01:47:36.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:36.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:36.414423+0000) 2024-02-20T01:47:37.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:37.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:37.414668+0000) 2024-02-20T01:47:38.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:38.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:38.414915+0000) 2024-02-20T01:47:39.413 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:39.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:39.415084+0000) 2024-02-20T01:47:40.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:40.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:40.415332+0000) 2024-02-20T01:47:41.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:41.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:41.415590+0000) 2024-02-20T01:47:42.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:42.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:42.415740+0000) 2024-02-20T01:47:43.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:43.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:43.415930+0000) 2024-02-20T01:47:44.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:44.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:44.416151+0000) 2024-02-20T01:47:45.414 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:45.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:45.416354+0000) 2024-02-20T01:47:46.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:46.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:46.416595+0000) 2024-02-20T01:47:47.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:47.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:47.416804+0000) 2024-02-20T01:47:48.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:48.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:48.417053+0000) 2024-02-20T01:47:49.415 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:49.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:49.417298+0000) 2024-02-20T01:47:50.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:50.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:50.417508+0000) 2024-02-20T01:47:51.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:51.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:51.417744+0000) 2024-02-20T01:47:52.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:52.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:52.417972+0000) 2024-02-20T01:47:53.416 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:53.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:53.418213+0000) 2024-02-20T01:47:54.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:54.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:54.418410+0000) 2024-02-20T01:47:55.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:55.412+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:55.418653+0000) 2024-02-20T01:47:56.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:56.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:56.418772+0000) 2024-02-20T01:47:57.417 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:57.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:57.418975+0000) 2024-02-20T01:47:58.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:58.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:58.419238+0000) 2024-02-20T01:47:59.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:47:59.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:47:59.419446+0000) 2024-02-20T01:48:00.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:00.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:00.419705+0000) 2024-02-20T01:48:01.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:01.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:01.419902+0000) 2024-02-20T01:48:02.418 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:02.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:02.420088+0000) 2024-02-20T01:48:03.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:03.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:03.420334+0000) 2024-02-20T01:48:04.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:04.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:04.420551+0000) 2024-02-20T01:48:05.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:05.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:05.420747+0000) 2024-02-20T01:48:06.419 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:06.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:06.420982+0000) 2024-02-20T01:48:07.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:07.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:07.421220+0000) 2024-02-20T01:48:08.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:08.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:08.421457+0000) 2024-02-20T01:48:09.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:09.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:09.421685+0000) 2024-02-20T01:48:10.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:10.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:10.421914+0000) 2024-02-20T01:48:11.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:11.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:11.422109+0000) 2024-02-20T01:48:12.420 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:12.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:12.422270+0000) 2024-02-20T01:48:13.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:13.416+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:13.422503+0000) 2024-02-20T01:48:14.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:14.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:14.422757+0000) 2024-02-20T01:48:15.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:15.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:15.422971+0000) 2024-02-20T01:48:16.421 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:16.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:16.423168+0000) 2024-02-20T01:48:17.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:17.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:17.423401+0000) 2024-02-20T01:48:18.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:18.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:18.423658+0000) 2024-02-20T01:48:19.422 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:19.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:19.423927+0000) 2024-02-20T01:48:20.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:20.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:20.424227+0000) 2024-02-20T01:48:21.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:21.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:21.424475+0000) 2024-02-20T01:48:22.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:22.420+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:22.424706+0000) 2024-02-20T01:48:23.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:23.424+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:23.424912+0000) 2024-02-20T01:48:24.423 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:24.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:24.425075+0000) 2024-02-20T01:48:25.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:25.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:25.425299+0000) 2024-02-20T01:48:26.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:26.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:26.425484+0000) 2024-02-20T01:48:27.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:27.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:27.425756+0000) 2024-02-20T01:48:28.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:28.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:28.425933+0000) 2024-02-20T01:48:29.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:29.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:29.426165+0000) 2024-02-20T01:48:30.424 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:30.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:30.426371+0000) 2024-02-20T01:48:31.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:31.419+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:31.426644+0000) 2024-02-20T01:48:32.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:32.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:32.426896+0000) 2024-02-20T01:48:33.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:33.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:33.427068+0000) 2024-02-20T01:48:34.425 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:34.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:34.427324+0000) 2024-02-20T01:48:35.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:35.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:35.427521+0000) 2024-02-20T01:48:36.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:36.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:36.427714+0000) 2024-02-20T01:48:37.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:37.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:37.427951+0000) 2024-02-20T01:48:38.426 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:38.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:38.428181+0000) 2024-02-20T01:48:39.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:39.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:39.428405+0000) 2024-02-20T01:48:40.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:40.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:40.428634+0000) 2024-02-20T01:48:41.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:41.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:41.428837+0000) 2024-02-20T01:48:42.427 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:42.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:42.429044+0000) 2024-02-20T01:48:43.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:43.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:43.429284+0000) 2024-02-20T01:48:44.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:44.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:44.429516+0000) 2024-02-20T01:48:45.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:45.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:45.429770+0000) 2024-02-20T01:48:46.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:46.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:46.429977+0000) 2024-02-20T01:48:47.428 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:47.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:47.430222+0000) 2024-02-20T01:48:48.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:48.423+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:48.430413+0000) 2024-02-20T01:48:49.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:49.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:49.430667+0000) 2024-02-20T01:48:50.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:50.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:50.430910+0000) 2024-02-20T01:48:51.429 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:51.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:51.431191+0000) 2024-02-20T01:48:52.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:52.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:52.431405+0000) 2024-02-20T01:48:53.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:53.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:53.431659+0000) 2024-02-20T01:48:54.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:54.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:54.431911+0000) 2024-02-20T01:48:55.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:55.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:55.432126+0000) 2024-02-20T01:48:56.430 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:56.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:56.432258+0000) 2024-02-20T01:48:57.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:57.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:57.432497+0000) 2024-02-20T01:48:58.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:58.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:58.432691+0000) 2024-02-20T01:48:59.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:48:59.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:48:59.432911+0000) 2024-02-20T01:49:00.431 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:00.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:00.433081+0000) 2024-02-20T01:49:01.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:01.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:01.433267+0000) 2024-02-20T01:49:02.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:02.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:02.433507+0000) 2024-02-20T01:49:03.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:03.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:03.433747+0000) 2024-02-20T01:49:04.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:04.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:04.433940+0000) 2024-02-20T01:49:05.432 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:05.427+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:05.434078+0000) 2024-02-20T01:49:06.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:06.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:06.434299+0000) 2024-02-20T01:49:07.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:07.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:07.434498+0000) 2024-02-20T01:49:08.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:08.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:08.434798+0000) 2024-02-20T01:49:09.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:09.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:09.434974+0000) 2024-02-20T01:49:10.433 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:10.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:10.435153+0000) 2024-02-20T01:49:11.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:11.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:11.435387+0000) 2024-02-20T01:49:12.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:12.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:12.435654+0000) 2024-02-20T01:49:13.434 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:13.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:13.435927+0000) 2024-02-20T01:49:14.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:14.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:14.436180+0000) 2024-02-20T01:49:15.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:15.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:15.436348+0000) 2024-02-20T01:49:16.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:16.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:16.436575+0000) 2024-02-20T01:49:17.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:17.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:17.436816+0000) 2024-02-20T01:49:18.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:18.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:18.437026+0000) 2024-02-20T01:49:19.435 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:19.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:19.437252+0000) 2024-02-20T01:49:20.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:20.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:20.437382+0000) 2024-02-20T01:49:21.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:21.431+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:21.437614+0000) 2024-02-20T01:49:22.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:22.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:22.437879+0000) 2024-02-20T01:49:23.436 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:23.435+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:23.438077+0000) 2024-02-20T01:49:24.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:24.430+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:24.438333+0000) 2024-02-20T01:49:25.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:25.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:25.438588+0000) 2024-02-20T01:49:26.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:26.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:26.438856+0000) 2024-02-20T01:49:27.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:27.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:27.439087+0000) 2024-02-20T01:49:28.437 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:28.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:28.439278+0000) 2024-02-20T01:49:29.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:29.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:29.439528+0000) 2024-02-20T01:49:30.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:30.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:30.439787+0000) 2024-02-20T01:49:31.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:31.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:31.440051+0000) 2024-02-20T01:49:32.438 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:32.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:32.440283+0000) 2024-02-20T01:49:33.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:33.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:33.440519+0000) 2024-02-20T01:49:34.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:34.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:34.440751+0000) 2024-02-20T01:49:35.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:35.434+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:35.440981+0000) 2024-02-20T01:49:36.439 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:36.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:36.441273+0000) 2024-02-20T01:49:37.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:37.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:37.441472+0000) 2024-02-20T01:49:38.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:38.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:38.441675+0000) 2024-02-20T01:49:39.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:39.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:39.441896+0000) 2024-02-20T01:49:40.440 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:40.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:40.442155+0000) 2024-02-20T01:49:41.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:41.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:41.442434+0000) 2024-02-20T01:49:42.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:42.438+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:42.442670+0000) 2024-02-20T01:49:43.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:43.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:43.442962+0000) 2024-02-20T01:49:44.441 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:44.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:44.443267+0000) 2024-02-20T01:49:45.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:45.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:45.443556+0000) 2024-02-20T01:49:46.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:46.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:46.443864+0000) 2024-02-20T01:49:47.442 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:47.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:47.444162+0000) 2024-02-20T01:49:48.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:48.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:48.444456+0000) 2024-02-20T01:49:49.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:49.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:49.444748+0000) 2024-02-20T01:49:50.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:50.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:50.444945+0000) 2024-02-20T01:49:51.443 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:51.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:51.445214+0000) 2024-02-20T01:49:52.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:52.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:52.445493+0000) 2024-02-20T01:49:53.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:53.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:53.445808+0000) 2024-02-20T01:49:54.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:54.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:54.446005+0000) 2024-02-20T01:49:55.444 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:55.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:55.446270+0000) 2024-02-20T01:49:56.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:56.442+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:56.446563+0000) 2024-02-20T01:49:57.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:57.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:57.446868+0000) 2024-02-20T01:49:58.445 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:58.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:58.447105+0000) 2024-02-20T01:49:59.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:49:59.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:49:59.447386+0000) 2024-02-20T01:50:00.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:00.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:00.447618+0000) 2024-02-20T01:50:01.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:01.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:01.447864+0000) 2024-02-20T01:50:02.446 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:02.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:02.448101+0000) 2024-02-20T01:50:03.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:03.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:03.448374+0000) 2024-02-20T01:50:04.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:04.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:04.448664+0000) 2024-02-20T01:50:05.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:05.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:05.448973+0000) 2024-02-20T01:50:06.447 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:06.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:06.449264+0000) 2024-02-20T01:50:07.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:07.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:07.449466+0000) 2024-02-20T01:50:08.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:08.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:08.449752+0000) 2024-02-20T01:50:09.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:09.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:09.450028+0000) 2024-02-20T01:50:10.448 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:10.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:10.450231+0000) 2024-02-20T01:50:11.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:11.446+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:11.450501+0000) 2024-02-20T01:50:12.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:12.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:12.450902+0000) 2024-02-20T01:50:13.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:13.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:13.451117+0000) 2024-02-20T01:50:14.449 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:14.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:14.451312+0000) 2024-02-20T01:50:15.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:15.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:15.451509+0000) 2024-02-20T01:50:16.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:16.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:16.451724+0000) 2024-02-20T01:50:17.450 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:17.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:17.451990+0000) 2024-02-20T01:50:18.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:18.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:18.452277+0000) 2024-02-20T01:50:19.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:19.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:19.452551+0000) 2024-02-20T01:50:20.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:20.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:20.452790+0000) 2024-02-20T01:50:21.451 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:21.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:21.453063+0000) 2024-02-20T01:50:22.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:22.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:22.453344+0000) 2024-02-20T01:50:23.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:23.450+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:23.453527+0000) 2024-02-20T01:50:24.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:24.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:24.453695+0000) 2024-02-20T01:50:25.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:25.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:25.453963+0000) 2024-02-20T01:50:26.452 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:26.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:26.454182+0000) 2024-02-20T01:50:27.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:27.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:27.454352+0000) 2024-02-20T01:50:28.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:28.449+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:28.454648+0000) 2024-02-20T01:50:29.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:29.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:29.454886+0000) 2024-02-20T01:50:30.453 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:30.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:30.455120+0000) 2024-02-20T01:50:31.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:31.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:31.455398+0000) 2024-02-20T01:50:32.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:32.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:32.455669+0000) 2024-02-20T01:50:33.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:33.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:33.455982+0000) 2024-02-20T01:50:34.454 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:34.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:34.456254+0000) 2024-02-20T01:50:35.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:35.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:35.456546+0000) 2024-02-20T01:50:36.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:36.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:36.456771+0000) 2024-02-20T01:50:37.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:37.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:37.457053+0000) 2024-02-20T01:50:38.455 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:38.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:38.457284+0000) 2024-02-20T01:50:39.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:39.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:39.457476+0000) 2024-02-20T01:50:40.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:40.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:40.457697+0000) 2024-02-20T01:50:41.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:41.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:41.457889+0000) 2024-02-20T01:50:42.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:42.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:42.458189+0000) 2024-02-20T01:50:43.456 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:43.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:43.458405+0000) 2024-02-20T01:50:44.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:44.453+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:44.458541+0000) 2024-02-20T01:50:45.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:45.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:45.458855+0000) 2024-02-20T01:50:46.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:46.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:46.459015+0000) 2024-02-20T01:50:47.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:47.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:47.459161+0000) 2024-02-20T01:50:48.457 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:48.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:48.459354+0000) 2024-02-20T01:50:49.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:49.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:49.459557+0000) 2024-02-20T01:50:50.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:50.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:50.459830+0000) 2024-02-20T01:50:51.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:51.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:51.460005+0000) 2024-02-20T01:50:52.458 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:52.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:52.460347+0000) 2024-02-20T01:50:53.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:53.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:53.460501+0000) 2024-02-20T01:50:54.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:54.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:54.460698+0000) 2024-02-20T01:50:55.459 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:55.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:55.460955+0000) 2024-02-20T01:50:56.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:56.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:56.461194+0000) 2024-02-20T01:50:57.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:57.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:57.461403+0000) 2024-02-20T01:50:58.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:58.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:58.461725+0000) 2024-02-20T01:50:59.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:50:59.457+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:50:59.461937+0000) 2024-02-20T01:51:00.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:00.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:00.462131+0000) 2024-02-20T01:51:01.460 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:01.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:01.462397+0000) 2024-02-20T01:51:02.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:02.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:02.462598+0000) 2024-02-20T01:51:03.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:03.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:03.462808+0000) 2024-02-20T01:51:04.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:04.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:04.463022+0000) 2024-02-20T01:51:05.461 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:05.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:05.463296+0000) 2024-02-20T01:51:06.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:06.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:06.463519+0000) 2024-02-20T01:51:07.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:07.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:07.463767+0000) 2024-02-20T01:51:08.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:08.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:08.463942+0000) 2024-02-20T01:51:09.462 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:09.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:09.464163+0000) 2024-02-20T01:51:10.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:10.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:10.464436+0000) 2024-02-20T01:51:11.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:11.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:11.464680+0000) 2024-02-20T01:51:12.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:12.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:12.464917+0000) 2024-02-20T01:51:13.463 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:13.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:13.465161+0000) 2024-02-20T01:51:14.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:14.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:14.465384+0000) 2024-02-20T01:51:15.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:15.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:15.465646+0000) 2024-02-20T01:51:16.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:16.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:16.465891+0000) 2024-02-20T01:51:17.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:17.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:17.466105+0000) 2024-02-20T01:51:18.464 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:18.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:18.466364+0000) 2024-02-20T01:51:19.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:19.461+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:19.466588+0000) 2024-02-20T01:51:20.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:20.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:20.466901+0000) 2024-02-20T01:51:21.465 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:21.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:21.467173+0000) 2024-02-20T01:51:22.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:22.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:22.467470+0000) 2024-02-20T01:51:23.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:23.465+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:23.467678+0000) 2024-02-20T01:51:24.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:24.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:24.467873+0000) 2024-02-20T01:51:25.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:25.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:25.468120+0000) 2024-02-20T01:51:26.466 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:26.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:26.468362+0000) 2024-02-20T01:51:27.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:27.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:27.468564+0000) 2024-02-20T01:51:28.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:28.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:28.468855+0000) 2024-02-20T01:51:29.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:29.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:29.469135+0000) 2024-02-20T01:51:30.467 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:30.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:30.469288+0000) 2024-02-20T01:51:31.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:31.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:31.469496+0000) 2024-02-20T01:51:32.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:32.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:32.469794+0000) 2024-02-20T01:51:33.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:33.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:33.470089+0000) 2024-02-20T01:51:34.468 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:34.464+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:34.470354+0000) 2024-02-20T01:51:35.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:35.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:35.470556+0000) 2024-02-20T01:51:36.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:36.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:36.470807+0000) 2024-02-20T01:51:37.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:37.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:37.470982+0000) 2024-02-20T01:51:38.469 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:38.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:38.471250+0000) 2024-02-20T01:51:39.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:39.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:39.471510+0000) 2024-02-20T01:51:40.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:40.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:40.471791+0000) 2024-02-20T01:51:41.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:41.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:41.472039+0000) 2024-02-20T01:51:42.470 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:42.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:42.472276+0000) 2024-02-20T01:51:43.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:43.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:43.472459+0000) 2024-02-20T01:51:44.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:44.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:44.472680+0000) 2024-02-20T01:51:45.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:45.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:45.472920+0000) 2024-02-20T01:51:46.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:46.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:46.473175+0000) 2024-02-20T01:51:47.471 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:47.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:47.473339+0000) 2024-02-20T01:51:48.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:48.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:48.473598+0000) 2024-02-20T01:51:49.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:49.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:49.473783+0000) 2024-02-20T01:51:50.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:50.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:50.473963+0000) 2024-02-20T01:51:51.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:51.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:51.474151+0000) 2024-02-20T01:51:52.472 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:52.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:52.474353+0000) 2024-02-20T01:51:53.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:53.468+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:53.474696+0000) 2024-02-20T01:51:54.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:54.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:54.474953+0000) 2024-02-20T01:51:55.473 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:55.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:55.475231+0000) 2024-02-20T01:51:56.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:56.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:56.475510+0000) 2024-02-20T01:51:57.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:57.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:57.475778+0000) 2024-02-20T01:51:58.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:58.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:58.476054+0000) 2024-02-20T01:51:59.474 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:51:59.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:51:59.476280+0000) 2024-02-20T01:52:00.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:00.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:00.476439+0000) 2024-02-20T01:52:01.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:01.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:01.476604+0000) 2024-02-20T01:52:02.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:02.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:02.476902+0000) 2024-02-20T01:52:03.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:03.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:03.477164+0000) 2024-02-20T01:52:04.475 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:04.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:04.477410+0000) 2024-02-20T01:52:05.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:05.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:05.477557+0000) 2024-02-20T01:52:06.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:06.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:06.477830+0000) 2024-02-20T01:52:07.476 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:07.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:07.478121+0000) 2024-02-20T01:52:08.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:08.472+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:08.478326+0000) 2024-02-20T01:52:09.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:09.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:09.478604+0000) 2024-02-20T01:52:10.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:10.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:10.478800+0000) 2024-02-20T01:52:11.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:11.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:11.479028+0000) 2024-02-20T01:52:12.477 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:12.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:12.479288+0000) 2024-02-20T01:52:13.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:13.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:13.479626+0000) 2024-02-20T01:52:14.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:14.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:14.479781+0000) 2024-02-20T01:52:15.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:15.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:15.479927+0000) 2024-02-20T01:52:16.478 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:16.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:16.480204+0000) 2024-02-20T01:52:17.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:17.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:17.480508+0000) 2024-02-20T01:52:18.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:18.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:18.480651+0000) 2024-02-20T01:52:19.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:19.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:19.480892+0000) 2024-02-20T01:52:20.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:20.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:20.481076+0000) 2024-02-20T01:52:21.479 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:21.480+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:21.481270+0000) 2024-02-20T01:52:22.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:22.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:22.481529+0000) 2024-02-20T01:52:23.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:23.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:23.481823+0000) 2024-02-20T01:52:24.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:24.476+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:24.482027+0000) 2024-02-20T01:52:25.480 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:25.475+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:25.482264+0000) 2024-02-20T01:52:26.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:26.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:26.482520+0000) 2024-02-20T01:52:27.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:27.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:27.482839+0000) 2024-02-20T01:52:28.481 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:28.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:28.482988+0000) 2024-02-20T01:52:29.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:29.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:29.483174+0000) 2024-02-20T01:52:30.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:30.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:30.483424+0000) 2024-02-20T01:52:31.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:31.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:31.483754+0000) 2024-02-20T01:52:32.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:32.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:32.483980+0000) 2024-02-20T01:52:33.482 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:33.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:33.484301+0000) 2024-02-20T01:52:34.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:34.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:34.484561+0000) 2024-02-20T01:52:35.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:35.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:35.484726+0000) 2024-02-20T01:52:36.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:36.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:36.484905+0000) 2024-02-20T01:52:37.483 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:37.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:37.485073+0000) 2024-02-20T01:52:38.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:38.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:38.485324+0000) 2024-02-20T01:52:39.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:39.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:39.485564+0000) 2024-02-20T01:52:40.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:40.479+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:40.485832+0000) 2024-02-20T01:52:41.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:41.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:41.486148+0000) 2024-02-20T01:52:42.484 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:42.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:42.486354+0000) 2024-02-20T01:52:43.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:43.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:43.486595+0000) 2024-02-20T01:52:44.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:44.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:44.486774+0000) 2024-02-20T01:52:45.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:45.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:45.486929+0000) 2024-02-20T01:52:46.485 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:46.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:46.487123+0000) 2024-02-20T01:52:47.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:47.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:47.487374+0000) 2024-02-20T01:52:48.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:48.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:48.487544+0000) 2024-02-20T01:52:49.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:49.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:49.487781+0000) 2024-02-20T01:52:50.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:50.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:50.488075+0000) 2024-02-20T01:52:51.486 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:51.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:51.488365+0000) 2024-02-20T01:52:52.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:52.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:52.488543+0000) 2024-02-20T01:52:53.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:53.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:53.488687+0000) 2024-02-20T01:52:54.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:54.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:54.488873+0000) 2024-02-20T01:52:55.487 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:55.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:55.489099+0000) 2024-02-20T01:52:56.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:56.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:56.489399+0000) 2024-02-20T01:52:57.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:57.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:57.489603+0000) 2024-02-20T01:52:58.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:58.483+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:58.489873+0000) 2024-02-20T01:52:59.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:52:59.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:52:59.490072+0000) 2024-02-20T01:53:00.488 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:00.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:00.490339+0000) 2024-02-20T01:53:01.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:01.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:01.490587+0000) 2024-02-20T01:53:02.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:02.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:02.490838+0000) 2024-02-20T01:53:03.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:03.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:03.491025+0000) 2024-02-20T01:53:04.489 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:04.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:04.491264+0000) 2024-02-20T01:53:05.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:05.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:05.491516+0000) 2024-02-20T01:53:06.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:06.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:06.491742+0000) 2024-02-20T01:53:07.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:07.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:07.491961+0000) 2024-02-20T01:53:08.490 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:08.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:08.492204+0000) 2024-02-20T01:53:09.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:09.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:09.492451+0000) 2024-02-20T01:53:10.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:10.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:10.492693+0000) 2024-02-20T01:53:11.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:11.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:11.492904+0000) 2024-02-20T01:53:12.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:12.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:12.493065+0000) 2024-02-20T01:53:13.491 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:13.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:13.493331+0000) 2024-02-20T01:53:14.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:14.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:14.493521+0000) 2024-02-20T01:53:15.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:15.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:15.493801+0000) 2024-02-20T01:53:16.492 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:16.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:16.494144+0000) 2024-02-20T01:53:17.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:17.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:17.494384+0000) 2024-02-20T01:53:18.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:18.487+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:18.494532+0000) 2024-02-20T01:53:19.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:19.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:19.494660+0000) 2024-02-20T01:53:20.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:20.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:20.494835+0000) 2024-02-20T01:53:21.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:21.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:21.494992+0000) 2024-02-20T01:53:22.493 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:22.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:22.495292+0000) 2024-02-20T01:53:23.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:23.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:23.495558+0000) 2024-02-20T01:53:24.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:24.491+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:24.495814+0000) 2024-02-20T01:53:25.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:25.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:25.495997+0000) 2024-02-20T01:53:26.494 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:26.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:26.496195+0000) 2024-02-20T01:53:27.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:27.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:27.496494+0000) 2024-02-20T01:53:28.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:28.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:28.496730+0000) 2024-02-20T01:53:29.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:29.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:29.496930+0000) 2024-02-20T01:53:30.495 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:30.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:30.497198+0000) 2024-02-20T01:53:31.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:31.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:31.497460+0000) 2024-02-20T01:53:32.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:32.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:32.497688+0000) 2024-02-20T01:53:33.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:33.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:33.497966+0000) 2024-02-20T01:53:34.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:34.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:34.498135+0000) 2024-02-20T01:53:35.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:35.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:35.498275+0000) 2024-02-20T01:53:36.496 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:36.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:36.498419+0000) 2024-02-20T01:53:37.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:37.490+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:37.498629+0000) 2024-02-20T01:53:38.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:38.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:38.498863+0000) 2024-02-20T01:53:39.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:39.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:39.499023+0000) 2024-02-20T01:53:40.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:40.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:40.499218+0000) 2024-02-20T01:53:41.497 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:41.494+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:41.499378+0000) 2024-02-20T01:53:42.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:42.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:42.499849+0000) 2024-02-20T01:53:43.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:43.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:43.500047+0000) 2024-02-20T01:53:44.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:44.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:44.500183+0000) 2024-02-20T01:53:45.498 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:45.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:45.500350+0000) 2024-02-20T01:53:46.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:46.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:46.500505+0000) 2024-02-20T01:53:47.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:47.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:47.500725+0000) 2024-02-20T01:53:48.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:48.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:48.500975+0000) 2024-02-20T01:53:49.499 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:49.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:49.501239+0000) 2024-02-20T01:53:50.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:50.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:50.501540+0000) 2024-02-20T01:53:51.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:51.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:51.501727+0000) 2024-02-20T01:53:52.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:52.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:52.501894+0000) 2024-02-20T01:53:53.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:53.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:53.502050+0000) 2024-02-20T01:53:54.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:54.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:54.502253+0000) 2024-02-20T01:53:55.500 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:55.498+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:55.502517+0000) 2024-02-20T01:53:56.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:56.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:56.502739+0000) 2024-02-20T01:53:57.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:57.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:57.503010+0000) 2024-02-20T01:53:58.501 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:58.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:58.503274+0000) 2024-02-20T01:53:59.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:53:59.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:53:59.503430+0000) 2024-02-20T01:54:00.502 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:00.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:00.503694+0000) 2024-02-20T01:54:01.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:01.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:01.503947+0000) 2024-02-20T01:54:02.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:02.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:02.504124+0000) 2024-02-20T01:54:03.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:03.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:03.504413+0000) 2024-02-20T01:54:04.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:04.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:04.504602+0000) 2024-02-20T01:54:05.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:05.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:05.504766+0000) 2024-02-20T01:54:06.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:06.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:06.504909+0000) 2024-02-20T01:54:07.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:07.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:07.505061+0000) 2024-02-20T01:54:08.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:08.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:08.505214+0000) 2024-02-20T01:54:09.503 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:09.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:09.505433+0000) 2024-02-20T01:54:10.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:10.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:10.505595+0000) 2024-02-20T01:54:11.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:11.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:11.505809+0000) 2024-02-20T01:54:12.504 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:12.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:12.506054+0000) 2024-02-20T01:54:13.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:13.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:13.506299+0000) 2024-02-20T01:54:14.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:14.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:14.506471+0000) 2024-02-20T01:54:15.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:15.502+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:15.506641+0000) 2024-02-20T01:54:16.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:16.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:16.506891+0000) 2024-02-20T01:54:17.505 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:17.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:17.507158+0000) 2024-02-20T01:54:18.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:18.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:18.507404+0000) 2024-02-20T01:54:19.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:19.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:19.507616+0000) 2024-02-20T01:54:20.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:20.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:20.507810+0000) 2024-02-20T01:54:21.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:21.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:21.507973+0000) 2024-02-20T01:54:22.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:22.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:22.508215+0000) 2024-02-20T01:54:23.506 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:23.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:23.508378+0000) 2024-02-20T01:54:24.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:24.506+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:24.508603+0000) 2024-02-20T01:54:25.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:25.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:25.508808+0000) 2024-02-20T01:54:26.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:26.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:26.509058+0000) 2024-02-20T01:54:27.507 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:27.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:27.509231+0000) 2024-02-20T01:54:28.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:28.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:28.509484+0000) 2024-02-20T01:54:29.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:29.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:29.509737+0000) 2024-02-20T01:54:30.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:30.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:30.509913+0000) 2024-02-20T01:54:31.508 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:31.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:31.510159+0000) 2024-02-20T01:54:32.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:32.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:32.510393+0000) 2024-02-20T01:54:33.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:33.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:33.510535+0000) 2024-02-20T01:54:34.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:34.505+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:34.510682+0000) 2024-02-20T01:54:35.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:35.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:35.510845+0000) 2024-02-20T01:54:36.509 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:36.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:36.511077+0000) 2024-02-20T01:54:37.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:37.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:37.511254+0000) 2024-02-20T01:54:38.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:38.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:38.511479+0000) 2024-02-20T01:54:39.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:39.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:39.511726+0000) 2024-02-20T01:54:40.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:40.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:40.511971+0000) 2024-02-20T01:54:41.510 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:41.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:41.512131+0000) 2024-02-20T01:54:42.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:42.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:42.512389+0000) 2024-02-20T01:54:43.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:43.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:43.512638+0000) 2024-02-20T01:54:44.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:44.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:44.512876+0000) 2024-02-20T01:54:45.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:45.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:45.513050+0000) 2024-02-20T01:54:46.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:46.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:46.513200+0000) 2024-02-20T01:54:47.511 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:47.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:47.513358+0000) 2024-02-20T01:54:48.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:48.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:48.513592+0000) 2024-02-20T01:54:49.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:49.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:49.513841+0000) 2024-02-20T01:54:50.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:50.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:50.514025+0000) 2024-02-20T01:54:51.512 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:51.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:51.514277+0000) 2024-02-20T01:54:52.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:52.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:52.514489+0000) 2024-02-20T01:54:53.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:53.509+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:53.514645+0000) 2024-02-20T01:54:54.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:54.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:54.514893+0000) 2024-02-20T01:54:55.513 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:55.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:55.515234+0000) 2024-02-20T01:54:56.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:56.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:56.515445+0000) 2024-02-20T01:54:57.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:57.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:57.515581+0000) 2024-02-20T01:54:58.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:58.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:58.515805+0000) 2024-02-20T01:54:59.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:54:59.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:54:59.515970+0000) 2024-02-20T01:55:00.514 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:00.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:00.516204+0000) 2024-02-20T01:55:01.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:01.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:01.516516+0000) 2024-02-20T01:55:02.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:02.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:02.516713+0000) 2024-02-20T01:55:03.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:03.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:03.516879+0000) 2024-02-20T01:55:04.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:04.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:04.517037+0000) 2024-02-20T01:55:05.515 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:05.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:05.517321+0000) 2024-02-20T01:55:06.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:06.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:06.517502+0000) 2024-02-20T01:55:07.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:07.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:07.517683+0000) 2024-02-20T01:55:08.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:08.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:08.517837+0000) 2024-02-20T01:55:09.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:09.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:09.517996+0000) 2024-02-20T01:55:10.516 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:10.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:10.518297+0000) 2024-02-20T01:55:11.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:11.513+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:11.518464+0000) 2024-02-20T01:55:12.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:12.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:12.518727+0000) 2024-02-20T01:55:13.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:13.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:13.518883+0000) 2024-02-20T01:55:14.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:14.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:14.519063+0000) 2024-02-20T01:55:15.517 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:15.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:15.519338+0000) 2024-02-20T01:55:16.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:16.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:16.519511+0000) 2024-02-20T01:55:17.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:17.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:17.519665+0000) 2024-02-20T01:55:18.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:18.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:18.519969+0000) 2024-02-20T01:55:19.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:19.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:19.520158+0000) 2024-02-20T01:55:20.518 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:20.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:20.520355+0000) 2024-02-20T01:55:21.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:21.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:21.520606+0000) 2024-02-20T01:55:22.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:22.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:22.520848+0000) 2024-02-20T01:55:23.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:23.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:23.521003+0000) 2024-02-20T01:55:24.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:24.517+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:24.521216+0000) 2024-02-20T01:55:25.519 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:25.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:25.521370+0000) 2024-02-20T01:55:26.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:26.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:26.521509+0000) 2024-02-20T01:55:27.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:27.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:27.521644+0000) 2024-02-20T01:55:28.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:28.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:28.521775+0000) 2024-02-20T01:55:29.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:29.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:29.521978+0000) 2024-02-20T01:55:30.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:30.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:30.522136+0000) 2024-02-20T01:55:31.520 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:31.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:31.522324+0000) 2024-02-20T01:55:32.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:32.516+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:32.522526+0000) 2024-02-20T01:55:33.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:33.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:33.522730+0000) 2024-02-20T01:55:34.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:34.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:34.522951+0000) 2024-02-20T01:55:35.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:35.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:35.523230+0000) 2024-02-20T01:55:36.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:36.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:36.523421+0000) 2024-02-20T01:55:37.521 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:37.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:37.523566+0000) 2024-02-20T01:55:38.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:38.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:38.523845+0000) 2024-02-20T01:55:39.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:39.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:39.524065+0000) 2024-02-20T01:55:40.522 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:40.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:40.524264+0000) 2024-02-20T01:55:41.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:41.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:41.524453+0000) 2024-02-20T01:55:42.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:42.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:42.524680+0000) 2024-02-20T01:55:43.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:43.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:43.524837+0000) 2024-02-20T01:55:44.523 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:44.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:44.525077+0000) 2024-02-20T01:55:45.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:45.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:45.525351+0000) 2024-02-20T01:55:46.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:46.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:46.525694+0000) 2024-02-20T01:55:47.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:47.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:47.525840+0000) 2024-02-20T01:55:48.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:48.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:48.526032+0000) 2024-02-20T01:55:49.524 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:49.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:49.526182+0000) 2024-02-20T01:55:50.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:50.520+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:50.526467+0000) 2024-02-20T01:55:51.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:51.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:51.526762+0000) 2024-02-20T01:55:52.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:52.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:52.526909+0000) 2024-02-20T01:55:53.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:53.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:53.527117+0000) 2024-02-20T01:55:54.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:54.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:54.527249+0000) 2024-02-20T01:55:55.525 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:55.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:55.527374+0000) 2024-02-20T01:55:56.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:56.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:56.527585+0000) 2024-02-20T01:55:57.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:57.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:57.527720+0000) 2024-02-20T01:55:58.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:58.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:58.527934+0000) 2024-02-20T01:55:59.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:55:59.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:55:59.528110+0000) 2024-02-20T01:56:00.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:00.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:00.528303+0000) 2024-02-20T01:56:01.526 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:01.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:01.528436+0000) 2024-02-20T01:56:02.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:02.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:02.528590+0000) 2024-02-20T01:56:03.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:03.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:03.528832+0000) 2024-02-20T01:56:04.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:04.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:04.529028+0000) 2024-02-20T01:56:05.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:05.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:05.529223+0000) 2024-02-20T01:56:06.527 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:06.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:06.529393+0000) 2024-02-20T01:56:07.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:07.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:07.529633+0000) 2024-02-20T01:56:08.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:08.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:08.529850+0000) 2024-02-20T01:56:09.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:09.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:09.530094+0000) 2024-02-20T01:56:10.528 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:10.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:10.530297+0000) 2024-02-20T01:56:11.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:11.524+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:11.530556+0000) 2024-02-20T01:56:12.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:12.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:12.530761+0000) 2024-02-20T01:56:13.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:13.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:13.531015+0000) 2024-02-20T01:56:14.529 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:14.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:14.531178+0000) 2024-02-20T01:56:15.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:15.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:15.531477+0000) 2024-02-20T01:56:16.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:16.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:16.531676+0000) 2024-02-20T01:56:17.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:17.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:17.531813+0000) 2024-02-20T01:56:18.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:18.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:18.531999+0000) 2024-02-20T01:56:19.530 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:19.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:19.532200+0000) 2024-02-20T01:56:20.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:20.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:20.532474+0000) 2024-02-20T01:56:21.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:21.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:21.532670+0000) 2024-02-20T01:56:22.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:22.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:22.532908+0000) 2024-02-20T01:56:23.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:23.528+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:23.533079+0000) 2024-02-20T01:56:24.531 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:24.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:24.533227+0000) 2024-02-20T01:56:25.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:25.532+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:25.533490+0000) 2024-02-20T01:56:26.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:26.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:26.533637+0000) 2024-02-20T01:56:27.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:27.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:27.533784+0000) 2024-02-20T01:56:28.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:28.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:28.534063+0000) 2024-02-20T01:56:29.532 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:29.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:29.534228+0000) 2024-02-20T01:56:30.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:30.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:30.534379+0000) 2024-02-20T01:56:31.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:31.527+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:31.534605+0000) 2024-02-20T01:56:32.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:32.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:32.534776+0000) 2024-02-20T01:56:33.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:33.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:33.534919+0000) 2024-02-20T01:56:34.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:34.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:34.535130+0000) 2024-02-20T01:56:35.533 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:35.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:35.535393+0000) 2024-02-20T01:56:36.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:36.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:36.535617+0000) 2024-02-20T01:56:37.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:37.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:37.535836+0000) 2024-02-20T01:56:38.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:38.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:38.536026+0000) 2024-02-20T01:56:39.534 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:39.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:39.536171+0000) 2024-02-20T01:56:40.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:40.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:40.536442+0000) 2024-02-20T01:56:41.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:41.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:41.536633+0000) 2024-02-20T01:56:42.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:42.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:42.536836+0000) 2024-02-20T01:56:43.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:43.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:43.537050+0000) 2024-02-20T01:56:44.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:44.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:44.537295+0000) 2024-02-20T01:56:45.535 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:45.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:45.537501+0000) 2024-02-20T01:56:46.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:46.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:46.537799+0000) 2024-02-20T01:56:47.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:47.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:47.538018+0000) 2024-02-20T01:56:48.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:48.531+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:48.538169+0000) 2024-02-20T01:56:49.536 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:49.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:49.538398+0000) 2024-02-20T01:56:50.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:50.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:50.538597+0000) 2024-02-20T01:56:51.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:51.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:51.538864+0000) 2024-02-20T01:56:52.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:52.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:52.539104+0000) 2024-02-20T01:56:53.537 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:53.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:53.539258+0000) 2024-02-20T01:56:54.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:54.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:54.539485+0000) 2024-02-20T01:56:55.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:55.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:55.539759+0000) 2024-02-20T01:56:56.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:56.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:56.539951+0000) 2024-02-20T01:56:57.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:57.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:57.540090+0000) 2024-02-20T01:56:58.538 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:58.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:58.540243+0000) 2024-02-20T01:56:59.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:56:59.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:56:59.540429+0000) 2024-02-20T01:57:00.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:00.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:00.540689+0000) 2024-02-20T01:57:01.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:01.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:01.540897+0000) 2024-02-20T01:57:02.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:02.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:02.541075+0000) 2024-02-20T01:57:03.539 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:03.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:03.541320+0000) 2024-02-20T01:57:04.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:04.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:04.541601+0000) 2024-02-20T01:57:05.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:05.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:05.541877+0000) 2024-02-20T01:57:06.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:06.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:06.542089+0000) 2024-02-20T01:57:07.540 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:07.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:07.542276+0000) 2024-02-20T01:57:08.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:08.535+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:08.542548+0000) 2024-02-20T01:57:09.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:09.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:09.542749+0000) 2024-02-20T01:57:10.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:10.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:10.542951+0000) 2024-02-20T01:57:11.541 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:11.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:11.543166+0000) 2024-02-20T01:57:12.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:12.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:12.543408+0000) 2024-02-20T01:57:13.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:13.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:13.543687+0000) 2024-02-20T01:57:14.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:14.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:14.543996+0000) 2024-02-20T01:57:15.542 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:15.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:15.544194+0000) 2024-02-20T01:57:16.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:16.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:16.544423+0000) 2024-02-20T01:57:17.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:17.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:17.544706+0000) 2024-02-20T01:57:18.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:18.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:18.544971+0000) 2024-02-20T01:57:19.543 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:19.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:19.545234+0000) 2024-02-20T01:57:20.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:20.539+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:20.545418+0000) 2024-02-20T01:57:21.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:21.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:21.545721+0000) 2024-02-20T01:57:22.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:22.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:22.545928+0000) 2024-02-20T01:57:23.544 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:23.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:23.546209+0000) 2024-02-20T01:57:24.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:24.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:24.546560+0000) 2024-02-20T01:57:25.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:25.543+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:25.546767+0000) 2024-02-20T01:57:26.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:26.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:26.547079+0000) 2024-02-20T01:57:27.545 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:27.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:27.547235+0000) 2024-02-20T01:57:28.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:28.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:28.547489+0000) 2024-02-20T01:57:29.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:29.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:29.547817+0000) 2024-02-20T01:57:30.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:30.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:30.548016+0000) 2024-02-20T01:57:31.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:31.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:31.548245+0000) 2024-02-20T01:57:32.546 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:32.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:32.548452+0000) 2024-02-20T01:57:33.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:33.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:33.548659+0000) 2024-02-20T01:57:34.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:34.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:34.548892+0000) 2024-02-20T01:57:35.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:35.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:35.549164+0000) 2024-02-20T01:57:36.547 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:36.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:36.549450+0000) 2024-02-20T01:57:37.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:37.542+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:37.549650+0000) 2024-02-20T01:57:38.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:38.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:38.549903+0000) 2024-02-20T01:57:39.548 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:39.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:39.550208+0000) 2024-02-20T01:57:40.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:40.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:40.550522+0000) 2024-02-20T01:57:41.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:41.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:41.550773+0000) 2024-02-20T01:57:42.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:42.546+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:42.551015+0000) 2024-02-20T01:57:43.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:43.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:43.551167+0000) 2024-02-20T01:57:44.549 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:44.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:44.551312+0000) 2024-02-20T01:57:45.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:45.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:45.551543+0000) 2024-02-20T01:57:46.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:46.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:46.551949+0000) 2024-02-20T01:57:47.550 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:47.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:47.552139+0000) 2024-02-20T01:57:48.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:48.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:48.552397+0000) 2024-02-20T01:57:49.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:49.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:49.552655+0000) 2024-02-20T01:57:50.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:50.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:50.552912+0000) 2024-02-20T01:57:51.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:51.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:51.553084+0000) 2024-02-20T01:57:52.551 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:52.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:52.553323+0000) 2024-02-20T01:57:53.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:53.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:53.553567+0000) 2024-02-20T01:57:54.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:54.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:54.553809+0000) 2024-02-20T01:57:55.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:55.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:55.554080+0000) 2024-02-20T01:57:56.552 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:56.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:56.554382+0000) 2024-02-20T01:57:57.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:57.550+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:57.554624+0000) 2024-02-20T01:57:58.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:58.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:58.554845+0000) 2024-02-20T01:57:59.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:57:59.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:57:59.555071+0000) 2024-02-20T01:58:00.553 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:00.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:00.555329+0000) 2024-02-20T01:58:01.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:01.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:01.555653+0000) 2024-02-20T01:58:02.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:02.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:02.555875+0000) 2024-02-20T01:58:03.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:03.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:03.556104+0000) 2024-02-20T01:58:04.554 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:04.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:04.556383+0000) 2024-02-20T01:58:05.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:05.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:05.556718+0000) 2024-02-20T01:58:06.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:06.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:06.557027+0000) 2024-02-20T01:58:07.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:07.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:07.557308+0000) 2024-02-20T01:58:08.555 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:08.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:08.557504+0000) 2024-02-20T01:58:09.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:09.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:09.557791+0000) 2024-02-20T01:58:10.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:10.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:10.557974+0000) 2024-02-20T01:58:11.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:11.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:11.558247+0000) 2024-02-20T01:58:12.556 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:12.554+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:12.558494+0000) 2024-02-20T01:58:13.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:13.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:13.558667+0000) 2024-02-20T01:58:14.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:14.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:14.558914+0000) 2024-02-20T01:58:15.557 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:15.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:15.559145+0000) 2024-02-20T01:58:16.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:16.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:16.559427+0000) 2024-02-20T01:58:17.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:17.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:17.559603+0000) 2024-02-20T01:58:18.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:18.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:18.559788+0000) 2024-02-20T01:58:19.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:19.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:19.559974+0000) 2024-02-20T01:58:20.558 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:20.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:20.560246+0000) 2024-02-20T01:58:21.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:21.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:21.560454+0000) 2024-02-20T01:58:22.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:22.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:22.560699+0000) 2024-02-20T01:58:23.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:23.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:23.560866+0000) 2024-02-20T01:58:24.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:24.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:24.561031+0000) 2024-02-20T01:58:25.559 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:25.558+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:25.561278+0000) 2024-02-20T01:58:26.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:26.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:26.561436+0000) 2024-02-20T01:58:27.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:27.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:27.561583+0000) 2024-02-20T01:58:28.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:28.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:28.561839+0000) 2024-02-20T01:58:29.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:29.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:29.562035+0000) 2024-02-20T01:58:30.560 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:30.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:30.562313+0000) 2024-02-20T01:58:31.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:31.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:31.562479+0000) 2024-02-20T01:58:32.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:32.557+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:32.562636+0000) 2024-02-20T01:58:33.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:33.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:33.562846+0000) 2024-02-20T01:58:34.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:34.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:34.563006+0000) 2024-02-20T01:58:35.561 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:35.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:35.563178+0000) 2024-02-20T01:58:36.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:36.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:36.563380+0000) 2024-02-20T01:58:37.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:37.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:37.563616+0000) 2024-02-20T01:58:38.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:38.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:38.563868+0000) 2024-02-20T01:58:39.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:39.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:39.564061+0000) 2024-02-20T01:58:40.562 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:40.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:40.564258+0000) 2024-02-20T01:58:41.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:41.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:41.564493+0000) 2024-02-20T01:58:42.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:42.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:42.564764+0000) 2024-02-20T01:58:43.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:43.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:43.564998+0000) 2024-02-20T01:58:44.563 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:44.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:44.565243+0000) 2024-02-20T01:58:45.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:45.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:45.565495+0000) 2024-02-20T01:58:46.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:46.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:46.565696+0000) 2024-02-20T01:58:47.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:47.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:47.565851+0000) 2024-02-20T01:58:48.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:48.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:48.566019+0000) 2024-02-20T01:58:49.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:49.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:49.566234+0000) 2024-02-20T01:58:50.564 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:50.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:50.566439+0000) 2024-02-20T01:58:51.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:51.561+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:51.566611+0000) 2024-02-20T01:58:52.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:52.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:52.566867+0000) 2024-02-20T01:58:53.565 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:53.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:53.567032+0000) 2024-02-20T01:58:54.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:54.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:54.567276+0000) 2024-02-20T01:58:55.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:55.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:55.567512+0000) 2024-02-20T01:58:56.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:56.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:56.567766+0000) 2024-02-20T01:58:57.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:57.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:57.567972+0000) 2024-02-20T01:58:58.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:58.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:58.568202+0000) 2024-02-20T01:58:59.566 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:58:59.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:58:59.568434+0000) 2024-02-20T01:59:00.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:00.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:00.568676+0000) 2024-02-20T01:59:01.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:01.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:01.568876+0000) 2024-02-20T01:59:02.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:02.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:02.569039+0000) 2024-02-20T01:59:03.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:03.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:03.569222+0000) 2024-02-20T01:59:04.567 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:04.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:04.569418+0000) 2024-02-20T01:59:05.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:05.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:05.569588+0000) 2024-02-20T01:59:06.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:06.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:06.569822+0000) 2024-02-20T01:59:07.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:07.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:07.570080+0000) 2024-02-20T01:59:08.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:08.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:08.570326+0000) 2024-02-20T01:59:09.568 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:09.565+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:09.570499+0000) 2024-02-20T01:59:10.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:10.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:10.570730+0000) 2024-02-20T01:59:11.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:11.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:11.570879+0000) 2024-02-20T01:59:12.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:12.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:12.571036+0000) 2024-02-20T01:59:13.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:13.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:13.571277+0000) 2024-02-20T01:59:14.569 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:14.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:14.571437+0000) 2024-02-20T01:59:15.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:15.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:15.571644+0000) 2024-02-20T01:59:16.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:16.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:16.571803+0000) 2024-02-20T01:59:17.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:17.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:17.571944+0000) 2024-02-20T01:59:18.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:18.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:18.572085+0000) 2024-02-20T01:59:19.570 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:19.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:19.572240+0000) 2024-02-20T01:59:20.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:20.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:20.572404+0000) 2024-02-20T01:59:21.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:21.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:21.572654+0000) 2024-02-20T01:59:22.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:22.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:22.572857+0000) 2024-02-20T01:59:23.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:23.569+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:23.573079+0000) 2024-02-20T01:59:24.571 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:24.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:24.573305+0000) 2024-02-20T01:59:25.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:25.573+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:25.573551+0000) 2024-02-20T01:59:26.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:26.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:26.573829+0000) 2024-02-20T01:59:27.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:27.568+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:27.573984+0000) 2024-02-20T01:59:28.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:28.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:28.574166+0000) 2024-02-20T01:59:29.572 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:29.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:29.574387+0000) 2024-02-20T01:59:30.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:30.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:30.574577+0000) 2024-02-20T01:59:31.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:31.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:31.574883+0000) 2024-02-20T01:59:32.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:32.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:32.575094+0000) 2024-02-20T01:59:33.573 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:33.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:33.575229+0000) 2024-02-20T01:59:34.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:34.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:34.575469+0000) 2024-02-20T01:59:35.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:35.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:35.575652+0000) 2024-02-20T01:59:36.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:36.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:36.575806+0000) 2024-02-20T01:59:37.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:37.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:37.576020+0000) 2024-02-20T01:59:38.574 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:38.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:38.576190+0000) 2024-02-20T01:59:39.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:39.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:39.576395+0000) 2024-02-20T01:59:40.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:40.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:40.576662+0000) 2024-02-20T01:59:41.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:41.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:41.576918+0000) 2024-02-20T01:59:42.575 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:42.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:42.577160+0000) 2024-02-20T01:59:43.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:43.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:43.577364+0000) 2024-02-20T01:59:44.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:44.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:44.577518+0000) 2024-02-20T01:59:45.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:45.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:45.577758+0000) 2024-02-20T01:59:46.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:46.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:46.577913+0000) 2024-02-20T01:59:47.576 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:47.572+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:47.578216+0000) 2024-02-20T01:59:48.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:48.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:48.578452+0000) 2024-02-20T01:59:49.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:49.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:49.578684+0000) 2024-02-20T01:59:50.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:50.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:50.578894+0000) 2024-02-20T01:59:51.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:51.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:51.579047+0000) 2024-02-20T01:59:52.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:52.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:52.579197+0000) 2024-02-20T01:59:53.577 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:53.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:53.579403+0000) 2024-02-20T01:59:54.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:54.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:54.579596+0000) 2024-02-20T01:59:55.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:55.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:55.579788+0000) 2024-02-20T01:59:56.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:56.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:56.580009+0000) 2024-02-20T01:59:57.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:57.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:57.580148+0000) 2024-02-20T01:59:58.578 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:58.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:58.580288+0000) 2024-02-20T01:59:59.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T01:59:59.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T00:59:59.580536+0000) 2024-02-20T02:00:00.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:00.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:00.580794+0000) 2024-02-20T02:00:01.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:01.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:01.581009+0000) 2024-02-20T02:00:02.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:02.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:02.581291+0000) 2024-02-20T02:00:03.579 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:03.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:03.581413+0000) 2024-02-20T02:00:04.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:04.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:04.581695+0000) 2024-02-20T02:00:05.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:05.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:05.581871+0000) 2024-02-20T02:00:06.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:06.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:06.582033+0000) 2024-02-20T02:00:07.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:07.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:07.582194+0000) 2024-02-20T02:00:08.580 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:08.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:08.582366+0000) 2024-02-20T02:00:09.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:09.576+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:09.582561+0000) 2024-02-20T02:00:10.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:10.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:10.582750+0000) 2024-02-20T02:00:11.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:11.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:11.582993+0000) 2024-02-20T02:00:12.581 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:12.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:12.583233+0000) 2024-02-20T02:00:13.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:13.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:13.583513+0000) 2024-02-20T02:00:14.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:14.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:14.583712+0000) 2024-02-20T02:00:15.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:15.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:15.583922+0000) 2024-02-20T02:00:16.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:16.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:16.584119+0000) 2024-02-20T02:00:17.582 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:17.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:17.584280+0000) 2024-02-20T02:00:18.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:18.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:18.584500+0000) 2024-02-20T02:00:19.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:19.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:19.584695+0000) 2024-02-20T02:00:20.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:20.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:20.584897+0000) 2024-02-20T02:00:21.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:21.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:21.585170+0000) 2024-02-20T02:00:22.583 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:22.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:22.585373+0000) 2024-02-20T02:00:23.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:23.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:23.585633+0000) 2024-02-20T02:00:24.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:24.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:24.585843+0000) 2024-02-20T02:00:25.584 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:25.580+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:25.586039+0000) 2024-02-20T02:00:26.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:26.584+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:26.586324+0000) 2024-02-20T02:00:27.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:27.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:27.586525+0000) 2024-02-20T02:00:28.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:28.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:28.586758+0000) 2024-02-20T02:00:29.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:29.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:29.586920+0000) 2024-02-20T02:00:30.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:30.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:30.587084+0000) 2024-02-20T02:00:31.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:31.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:31.587245+0000) 2024-02-20T02:00:32.585 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:32.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:32.587410+0000) 2024-02-20T02:00:33.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:33.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:33.587572+0000) 2024-02-20T02:00:34.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:34.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:34.587812+0000) 2024-02-20T02:00:35.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:35.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:35.587968+0000) 2024-02-20T02:00:36.586 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:36.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:36.588211+0000) 2024-02-20T02:00:37.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:37.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:37.588398+0000) 2024-02-20T02:00:38.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:38.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:38.588583+0000) 2024-02-20T02:00:39.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:39.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:39.588796+0000) 2024-02-20T02:00:40.587 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:40.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:40.589079+0000) 2024-02-20T02:00:41.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:41.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:41.589367+0000) 2024-02-20T02:00:42.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:42.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:42.589614+0000) 2024-02-20T02:00:43.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:43.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:43.589841+0000) 2024-02-20T02:00:44.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:44.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:44.590062+0000) 2024-02-20T02:00:45.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:45.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:45.590203+0000) 2024-02-20T02:00:46.588 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:46.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:46.590380+0000) 2024-02-20T02:00:47.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:47.583+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:47.590523+0000) 2024-02-20T02:00:48.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:48.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:48.590691+0000) 2024-02-20T02:00:49.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:49.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:49.590884+0000) 2024-02-20T02:00:50.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:50.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:50.591094+0000) 2024-02-20T02:00:51.589 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:51.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:51.591292+0000) 2024-02-20T02:00:52.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:52.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:52.591538+0000) 2024-02-20T02:00:53.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:53.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:53.591789+0000) 2024-02-20T02:00:54.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:54.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:54.591956+0000) 2024-02-20T02:00:55.590 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:55.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:55.592139+0000) 2024-02-20T02:00:56.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:56.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:56.592391+0000) 2024-02-20T02:00:57.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:57.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:57.592584+0000) 2024-02-20T02:00:58.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:58.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:58.592768+0000) 2024-02-20T02:00:59.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:00:59.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:00:59.592947+0000) 2024-02-20T02:01:00.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:00.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:00.593166+0000) 2024-02-20T02:01:01.591 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:01.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:01.593369+0000) 2024-02-20T02:01:02.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:02.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:02.593694+0000) 2024-02-20T02:01:03.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:03.587+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:03.593868+0000) 2024-02-20T02:01:04.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:04.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:04.594045+0000) 2024-02-20T02:01:05.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:05.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:05.594256+0000) 2024-02-20T02:01:06.592 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:06.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:06.594461+0000) 2024-02-20T02:01:07.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:07.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:07.594784+0000) 2024-02-20T02:01:08.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:08.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:08.594917+0000) 2024-02-20T02:01:09.593 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:09.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:09.595086+0000) 2024-02-20T02:01:10.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:10.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:10.595297+0000) 2024-02-20T02:01:11.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:11.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:11.595583+0000) 2024-02-20T02:01:12.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:12.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:12.595765+0000) 2024-02-20T02:01:13.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:13.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:13.596008+0000) 2024-02-20T02:01:14.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:14.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:14.596167+0000) 2024-02-20T02:01:15.594 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:15.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:15.596390+0000) 2024-02-20T02:01:16.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:16.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:16.596671+0000) 2024-02-20T02:01:17.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:17.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:17.596879+0000) 2024-02-20T02:01:18.595 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:18.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:18.597040+0000) 2024-02-20T02:01:19.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:19.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:19.597254+0000) 2024-02-20T02:01:20.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:20.591+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:20.597450+0000) 2024-02-20T02:01:21.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:21.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:21.597626+0000) 2024-02-20T02:01:22.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:22.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:22.597877+0000) 2024-02-20T02:01:23.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:23.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:23.598027+0000) 2024-02-20T02:01:24.596 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:24.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:24.598301+0000) 2024-02-20T02:01:25.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:25.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:25.598577+0000) 2024-02-20T02:01:26.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:26.595+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:26.598789+0000) 2024-02-20T02:01:27.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:27.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:27.599006+0000) 2024-02-20T02:01:28.597 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:28.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:28.599210+0000) 2024-02-20T02:01:29.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:29.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:29.599356+0000) 2024-02-20T02:01:30.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:30.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:30.599514+0000) 2024-02-20T02:01:31.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:31.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:31.599728+0000) 2024-02-20T02:01:32.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:32.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:32.599886+0000) 2024-02-20T02:01:33.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:33.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:33.600115+0000) 2024-02-20T02:01:34.598 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:34.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:34.600276+0000) 2024-02-20T02:01:35.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:35.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:35.600469+0000) 2024-02-20T02:01:36.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:36.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:36.600672+0000) 2024-02-20T02:01:37.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:37.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:37.600863+0000) 2024-02-20T02:01:38.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:38.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:38.601056+0000) 2024-02-20T02:01:39.599 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:39.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:39.601275+0000) 2024-02-20T02:01:40.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:40.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:40.601492+0000) 2024-02-20T02:01:41.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:41.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:41.601732+0000) 2024-02-20T02:01:42.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:42.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:42.601921+0000) 2024-02-20T02:01:43.600 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:43.594+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:43.602113+0000) 2024-02-20T02:01:44.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:44.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:44.602345+0000) 2024-02-20T02:01:45.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:45.598+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:45.602583+0000) 2024-02-20T02:01:46.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:46.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:46.602882+0000) 2024-02-20T02:01:47.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:47.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:47.603111+0000) 2024-02-20T02:01:48.601 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:48.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:48.603336+0000) 2024-02-20T02:01:49.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:49.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:49.603495+0000) 2024-02-20T02:01:50.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:50.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:50.603794+0000) 2024-02-20T02:01:51.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:51.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:51.603967+0000) 2024-02-20T02:01:52.602 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:52.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:52.604159+0000) 2024-02-20T02:01:53.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:53.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:53.604380+0000) 2024-02-20T02:01:54.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:54.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:54.604532+0000) 2024-02-20T02:01:55.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:55.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:55.604668+0000) 2024-02-20T02:01:56.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:56.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:56.604812+0000) 2024-02-20T02:01:57.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:57.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:57.604951+0000) 2024-02-20T02:01:58.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:58.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:58.605103+0000) 2024-02-20T02:01:59.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:01:59.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:01:59.605254+0000) 2024-02-20T02:02:00.603 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:00.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:00.605457+0000) 2024-02-20T02:02:01.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:01.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:01.605733+0000) 2024-02-20T02:02:02.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:02.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:02.605907+0000) 2024-02-20T02:02:03.604 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:03.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:03.606186+0000) 2024-02-20T02:02:04.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:04.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:04.606330+0000) 2024-02-20T02:02:05.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:05.602+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:05.606524+0000) 2024-02-20T02:02:06.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:06.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:06.606749+0000) 2024-02-20T02:02:07.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:07.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:07.606911+0000) 2024-02-20T02:02:08.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:08.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:08.607079+0000) 2024-02-20T02:02:09.605 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:09.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:09.607370+0000) 2024-02-20T02:02:10.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:10.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:10.607590+0000) 2024-02-20T02:02:11.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:11.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:11.607882+0000) 2024-02-20T02:02:12.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:12.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:12.608123+0000) 2024-02-20T02:02:13.606 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:13.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:13.608309+0000) 2024-02-20T02:02:14.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:14.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:14.608464+0000) 2024-02-20T02:02:15.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:15.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:15.608677+0000) 2024-02-20T02:02:16.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:16.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:16.608893+0000) 2024-02-20T02:02:17.607 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:17.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:17.609142+0000) 2024-02-20T02:02:18.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:18.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:18.609396+0000) 2024-02-20T02:02:19.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:19.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:19.609558+0000) 2024-02-20T02:02:20.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:20.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:20.609812+0000) 2024-02-20T02:02:21.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:21.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:21.610041+0000) 2024-02-20T02:02:22.608 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:22.606+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:22.610317+0000) 2024-02-20T02:02:23.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:23.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:23.610615+0000) 2024-02-20T02:02:24.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:24.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:24.610785+0000) 2024-02-20T02:02:25.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:25.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:25.611014+0000) 2024-02-20T02:02:26.609 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:26.610+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:26.611181+0000) 2024-02-20T02:02:27.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:27.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:27.611408+0000) 2024-02-20T02:02:28.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:28.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:28.611601+0000) 2024-02-20T02:02:29.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:29.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:29.611746+0000) 2024-02-20T02:02:30.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:30.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:30.611894+0000) 2024-02-20T02:02:31.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:31.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:31.612038+0000) 2024-02-20T02:02:32.610 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:32.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:32.612273+0000) 2024-02-20T02:02:33.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:33.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:33.612425+0000) 2024-02-20T02:02:34.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:34.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:34.612565+0000) 2024-02-20T02:02:35.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:35.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:35.612742+0000) 2024-02-20T02:02:36.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:36.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:36.613060+0000) 2024-02-20T02:02:37.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:37.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:37.613202+0000) 2024-02-20T02:02:38.611 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:38.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:38.613370+0000) 2024-02-20T02:02:39.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:39.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:39.613509+0000) 2024-02-20T02:02:40.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:40.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:40.613746+0000) 2024-02-20T02:02:41.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:41.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:41.613928+0000) 2024-02-20T02:02:42.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:42.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:42.614090+0000) 2024-02-20T02:02:43.612 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:43.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:43.614330+0000) 2024-02-20T02:02:44.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:44.609+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:44.614495+0000) 2024-02-20T02:02:45.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:45.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:45.614666+0000) 2024-02-20T02:02:46.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:46.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:46.614873+0000) 2024-02-20T02:02:47.613 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:47.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:47.615128+0000) 2024-02-20T02:02:48.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:48.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:48.615401+0000) 2024-02-20T02:02:49.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:49.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:49.615612+0000) 2024-02-20T02:02:50.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:50.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:50.615780+0000) 2024-02-20T02:02:51.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:51.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:51.616014+0000) 2024-02-20T02:02:52.614 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:52.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:52.616211+0000) 2024-02-20T02:02:53.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:53.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:53.616449+0000) 2024-02-20T02:02:54.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:54.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:54.616633+0000) 2024-02-20T02:02:55.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:55.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:55.616801+0000) 2024-02-20T02:02:56.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:56.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:56.616982+0000) 2024-02-20T02:02:57.615 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:57.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:57.617236+0000) 2024-02-20T02:02:58.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:58.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:58.617488+0000) 2024-02-20T02:02:59.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:02:59.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:02:59.617723+0000) 2024-02-20T02:03:00.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:00.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:00.617961+0000) 2024-02-20T02:03:01.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:01.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:01.618191+0000) 2024-02-20T02:03:02.616 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:02.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:02.618362+0000) 2024-02-20T02:03:03.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:03.613+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:03.618591+0000) 2024-02-20T02:03:04.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:04.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:04.618872+0000) 2024-02-20T02:03:05.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:05.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:05.619044+0000) 2024-02-20T02:03:06.617 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:06.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:06.619242+0000) 2024-02-20T02:03:07.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:07.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:07.619468+0000) 2024-02-20T02:03:08.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:08.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:08.619630+0000) 2024-02-20T02:03:09.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:09.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:09.619776+0000) 2024-02-20T02:03:10.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:10.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:10.620040+0000) 2024-02-20T02:03:11.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:11.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:11.620217+0000) 2024-02-20T02:03:12.618 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:12.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:12.620410+0000) 2024-02-20T02:03:13.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:13.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:13.620689+0000) 2024-02-20T02:03:14.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:14.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:14.620869+0000) 2024-02-20T02:03:15.619 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:15.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:15.621135+0000) 2024-02-20T02:03:16.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:16.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:16.621398+0000) 2024-02-20T02:03:17.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:17.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:17.621559+0000) 2024-02-20T02:03:18.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:18.621+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:18.621801+0000) 2024-02-20T02:03:19.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:19.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:19.622026+0000) 2024-02-20T02:03:20.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:20.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:20.622190+0000) 2024-02-20T02:03:21.620 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:21.617+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:21.622339+0000) 2024-02-20T02:03:22.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:22.621+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:22.622534+0000) 2024-02-20T02:03:23.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:23.621+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:23.622913+0000) 2024-02-20T02:03:24.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:24.621+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:24.623048+0000) 2024-02-20T02:03:25.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:25.621+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:25.623217+0000) 2024-02-20T02:03:26.621 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:26.621+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:26.623354+0000) 2024-02-20T02:03:27.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:27.621+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:27.623504+0000) 2024-02-20T02:03:28.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:28.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:28.623714+0000) 2024-02-20T02:03:29.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:29.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:29.623984+0000) 2024-02-20T02:03:30.622 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:30.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:30.624191+0000) 2024-02-20T02:03:31.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:31.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:31.624434+0000) 2024-02-20T02:03:32.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:32.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:32.624660+0000) 2024-02-20T02:03:33.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:33.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:33.624902+0000) 2024-02-20T02:03:34.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:34.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:34.625134+0000) 2024-02-20T02:03:35.623 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:35.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:35.625341+0000) 2024-02-20T02:03:36.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:36.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:36.625492+0000) 2024-02-20T02:03:37.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:37.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:37.625668+0000) 2024-02-20T02:03:38.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:38.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:38.625886+0000) 2024-02-20T02:03:39.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:39.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:39.626062+0000) 2024-02-20T02:03:40.624 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:40.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:40.626216+0000) 2024-02-20T02:03:41.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:41.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:41.626482+0000) 2024-02-20T02:03:42.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:42.620+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:42.626671+0000) 2024-02-20T02:03:43.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:43.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:43.626828+0000) 2024-02-20T02:03:44.625 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:44.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:44.627004+0000) 2024-02-20T02:03:45.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:45.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:45.627275+0000) 2024-02-20T02:03:46.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:46.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:46.627437+0000) 2024-02-20T02:03:47.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:47.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:47.627675+0000) 2024-02-20T02:03:48.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:48.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:48.627846+0000) 2024-02-20T02:03:49.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:49.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:49.628092+0000) 2024-02-20T02:03:50.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:50.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:50.628246+0000) 2024-02-20T02:03:51.626 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:51.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:51.628386+0000) 2024-02-20T02:03:52.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:52.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:52.628555+0000) 2024-02-20T02:03:53.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:53.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:53.628744+0000) 2024-02-20T02:03:54.627 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:54.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:54.629020+0000) 2024-02-20T02:03:55.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:55.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:55.629271+0000) 2024-02-20T02:03:56.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:56.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:56.629442+0000) 2024-02-20T02:03:57.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:57.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:57.629687+0000) 2024-02-20T02:03:58.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:58.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:58.629886+0000) 2024-02-20T02:03:59.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:03:59.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:03:59.630076+0000) 2024-02-20T02:04:00.628 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:00.624+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:00.630335+0000) 2024-02-20T02:04:01.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:01.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:01.630511+0000) 2024-02-20T02:04:02.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:02.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:02.630769+0000) 2024-02-20T02:04:03.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:03.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:03.630920+0000) 2024-02-20T02:04:04.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:04.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:04.631208+0000) 2024-02-20T02:04:05.629 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:05.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:05.631349+0000) 2024-02-20T02:04:06.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:06.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:06.631490+0000) 2024-02-20T02:04:07.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:07.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:07.631719+0000) 2024-02-20T02:04:08.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:08.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:08.631948+0000) 2024-02-20T02:04:09.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:09.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:09.632086+0000) 2024-02-20T02:04:10.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:10.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:10.632271+0000) 2024-02-20T02:04:11.630 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:11.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:11.632463+0000) 2024-02-20T02:04:12.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:12.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:12.632762+0000) 2024-02-20T02:04:13.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:13.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:13.632952+0000) 2024-02-20T02:04:14.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:14.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:14.633177+0000) 2024-02-20T02:04:15.631 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:15.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:15.633353+0000) 2024-02-20T02:04:16.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:16.628+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:16.633543+0000) 2024-02-20T02:04:17.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:17.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:17.633847+0000) 2024-02-20T02:04:18.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:18.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:18.634010+0000) 2024-02-20T02:04:19.632 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:19.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:19.634188+0000) 2024-02-20T02:04:20.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:20.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:20.634436+0000) 2024-02-20T02:04:21.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:21.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:21.634753+0000) 2024-02-20T02:04:22.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:22.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:22.634992+0000) 2024-02-20T02:04:23.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:23.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:23.635159+0000) 2024-02-20T02:04:24.633 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:24.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:24.635333+0000) 2024-02-20T02:04:25.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:25.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:25.635486+0000) 2024-02-20T02:04:26.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:26.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:26.635654+0000) 2024-02-20T02:04:27.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:27.632+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:27.635933+0000) 2024-02-20T02:04:28.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:28.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:28.636097+0000) 2024-02-20T02:04:29.634 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:29.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:29.636282+0000) 2024-02-20T02:04:30.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:30.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:30.636508+0000) 2024-02-20T02:04:31.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:31.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:31.636729+0000) 2024-02-20T02:04:32.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:32.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:32.636901+0000) 2024-02-20T02:04:33.635 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:33.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:33.637083+0000) 2024-02-20T02:04:34.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:34.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:34.637293+0000) 2024-02-20T02:04:35.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:35.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:35.637517+0000) 2024-02-20T02:04:36.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:36.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:36.637709+0000) 2024-02-20T02:04:37.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:37.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:37.637941+0000) 2024-02-20T02:04:38.636 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:38.631+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:38.638112+0000) 2024-02-20T02:04:39.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:39.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:39.638351+0000) 2024-02-20T02:04:40.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:40.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:40.638527+0000) 2024-02-20T02:04:41.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:41.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:41.638775+0000) 2024-02-20T02:04:42.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:42.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:42.638924+0000) 2024-02-20T02:04:43.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:43.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:43.639112+0000) 2024-02-20T02:04:44.637 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:44.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:44.639349+0000) 2024-02-20T02:04:45.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:45.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:45.639603+0000) 2024-02-20T02:04:46.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:46.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:46.639884+0000) 2024-02-20T02:04:47.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:47.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:47.640020+0000) 2024-02-20T02:04:48.638 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:48.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:48.640192+0000) 2024-02-20T02:04:49.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:49.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:49.640468+0000) 2024-02-20T02:04:50.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:50.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:50.640730+0000) 2024-02-20T02:04:51.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:51.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:51.640888+0000) 2024-02-20T02:04:52.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:52.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:52.641062+0000) 2024-02-20T02:04:53.639 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:53.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:53.641337+0000) 2024-02-20T02:04:54.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:54.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:54.641604+0000) 2024-02-20T02:04:55.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:55.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:55.641765+0000) 2024-02-20T02:04:56.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:56.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:56.641960+0000) 2024-02-20T02:04:57.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:57.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:57.642122+0000) 2024-02-20T02:04:58.640 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:58.635+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:58.642317+0000) 2024-02-20T02:04:59.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:04:59.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:04:59.642494+0000) 2024-02-20T02:05:00.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:00.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:00.642829+0000) 2024-02-20T02:05:01.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:01.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:01.643008+0000) 2024-02-20T02:05:02.641 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:02.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:02.643168+0000) 2024-02-20T02:05:03.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:03.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:03.643443+0000) 2024-02-20T02:05:04.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:04.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:04.643629+0000) 2024-02-20T02:05:05.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:05.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:05.643824+0000) 2024-02-20T02:05:06.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:06.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:06.643999+0000) 2024-02-20T02:05:07.642 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:07.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:07.644193+0000) 2024-02-20T02:05:08.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:08.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:08.644425+0000) 2024-02-20T02:05:09.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:09.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:09.644590+0000) 2024-02-20T02:05:10.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:10.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:10.644830+0000) 2024-02-20T02:05:11.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:11.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:11.645137+0000) 2024-02-20T02:05:12.643 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:12.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:12.645310+0000) 2024-02-20T02:05:13.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:13.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:13.645645+0000) 2024-02-20T02:05:14.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:14.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:14.645880+0000) 2024-02-20T02:05:15.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:15.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:15.646071+0000) 2024-02-20T02:05:16.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:16.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:16.646223+0000) 2024-02-20T02:05:17.644 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:17.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:17.646378+0000) 2024-02-20T02:05:18.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:18.639+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:18.646598+0000) 2024-02-20T02:05:19.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:19.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:19.646788+0000) 2024-02-20T02:05:20.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:20.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:20.646983+0000) 2024-02-20T02:05:21.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:21.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:21.647185+0000) 2024-02-20T02:05:22.645 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:22.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:22.647410+0000) 2024-02-20T02:05:23.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:23.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:23.647660+0000) 2024-02-20T02:05:24.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:24.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:24.647808+0000) 2024-02-20T02:05:25.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:25.647+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:25.647956+0000) 2024-02-20T02:05:26.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:26.643+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:26.648262+0000) 2024-02-20T02:05:27.646 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:27.647+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:27.648477+0000) 2024-02-20T02:05:28.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:28.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:28.648726+0000) 2024-02-20T02:05:29.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:29.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:29.648872+0000) 2024-02-20T02:05:30.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:30.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:30.649012+0000) 2024-02-20T02:05:31.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:31.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:31.649196+0000) 2024-02-20T02:05:32.647 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:32.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:32.649455+0000) 2024-02-20T02:05:33.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:33.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:33.649650+0000) 2024-02-20T02:05:34.648 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:34.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:34.649871+0000) 2024-02-20T02:05:35.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:35.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:35.650004+0000) 2024-02-20T02:05:36.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:36.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:36.650322+0000) 2024-02-20T02:05:37.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:37.642+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:37.650571+0000) 2024-02-20T02:05:38.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:38.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:38.650748+0000) 2024-02-20T02:05:39.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:39.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:39.650896+0000) 2024-02-20T02:05:40.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:40.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:40.651039+0000) 2024-02-20T02:05:41.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:41.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:41.651193+0000) 2024-02-20T02:05:42.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:42.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:42.651360+0000) 2024-02-20T02:05:43.649 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:43.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:43.651536+0000) 2024-02-20T02:05:44.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:44.646+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:44.651795+0000) 2024-02-20T02:05:45.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:45.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:45.651943+0000) 2024-02-20T02:05:46.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:46.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:46.652090+0000) 2024-02-20T02:05:47.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:47.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:47.652240+0000) 2024-02-20T02:05:48.650 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:48.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:48.652410+0000) 2024-02-20T02:05:49.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:49.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:49.652615+0000) 2024-02-20T02:05:50.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:50.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:50.652825+0000) 2024-02-20T02:05:51.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:51.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:51.653066+0000) 2024-02-20T02:05:52.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:52.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:52.653269+0000) 2024-02-20T02:05:53.651 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:53.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:53.653414+0000) 2024-02-20T02:05:54.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:54.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:54.653605+0000) 2024-02-20T02:05:55.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:55.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:55.653751+0000) 2024-02-20T02:05:56.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:56.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:56.654017+0000) 2024-02-20T02:05:57.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:57.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:57.654223+0000) 2024-02-20T02:05:58.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:58.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:58.654379+0000) 2024-02-20T02:05:59.652 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:05:59.650+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:05:59.654605+0000) 2024-02-20T02:06:00.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:00.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:00.654759+0000) 2024-02-20T02:06:01.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:01.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:01.654942+0000) 2024-02-20T02:06:02.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:02.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:02.655080+0000) 2024-02-20T02:06:03.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:03.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:03.655224+0000) 2024-02-20T02:06:04.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:04.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:04.655362+0000) 2024-02-20T02:06:05.653 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:05.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:05.655503+0000) 2024-02-20T02:06:06.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:06.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:06.655776+0000) 2024-02-20T02:06:07.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:07.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:07.655948+0000) 2024-02-20T02:06:08.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:08.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:08.656097+0000) 2024-02-20T02:06:09.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:09.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:09.656243+0000) 2024-02-20T02:06:10.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:10.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:10.656445+0000) 2024-02-20T02:06:11.654 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:11.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:11.656674+0000) 2024-02-20T02:06:12.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:12.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:12.656972+0000) 2024-02-20T02:06:13.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:13.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:13.657139+0000) 2024-02-20T02:06:14.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:14.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:14.657276+0000) 2024-02-20T02:06:15.655 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:15.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:15.657532+0000) 2024-02-20T02:06:16.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:16.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:16.657698+0000) 2024-02-20T02:06:17.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:17.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:17.657845+0000) 2024-02-20T02:06:18.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:18.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:18.658034+0000) 2024-02-20T02:06:19.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:19.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:19.658224+0000) 2024-02-20T02:06:20.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:20.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:20.658413+0000) 2024-02-20T02:06:21.656 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:21.654+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:21.658628+0000) 2024-02-20T02:06:22.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:22.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:22.658832+0000) 2024-02-20T02:06:23.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:23.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:23.659071+0000) 2024-02-20T02:06:24.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:24.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:24.659198+0000) 2024-02-20T02:06:25.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:25.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:25.659443+0000) 2024-02-20T02:06:26.657 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:26.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:26.659705+0000) 2024-02-20T02:06:27.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:27.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:27.659856+0000) 2024-02-20T02:06:28.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:28.658+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:28.660015+0000) 2024-02-20T02:06:29.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:29.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:29.660167+0000) 2024-02-20T02:06:30.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:30.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:30.660323+0000) 2024-02-20T02:06:31.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:31.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:31.660518+0000) 2024-02-20T02:06:32.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:32.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:32.660683+0000) 2024-02-20T02:06:33.658 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:33.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:33.660839+0000) 2024-02-20T02:06:34.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:34.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:34.661087+0000) 2024-02-20T02:06:35.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:35.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:35.661361+0000) 2024-02-20T02:06:36.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:36.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:36.661553+0000) 2024-02-20T02:06:37.659 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:37.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:37.661711+0000) 2024-02-20T02:06:38.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:38.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:38.661865+0000) 2024-02-20T02:06:39.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:39.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:39.662109+0000) 2024-02-20T02:06:40.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:40.657+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:40.662298+0000) 2024-02-20T02:06:41.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:41.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:41.662446+0000) 2024-02-20T02:06:42.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:42.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:42.662602+0000) 2024-02-20T02:06:43.660 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:43.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:43.662818+0000) 2024-02-20T02:06:44.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:44.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:44.662969+0000) 2024-02-20T02:06:45.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:45.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:45.663193+0000) 2024-02-20T02:06:46.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:46.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:46.663352+0000) 2024-02-20T02:06:47.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:47.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:47.663534+0000) 2024-02-20T02:06:48.661 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:48.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:48.663712+0000) 2024-02-20T02:06:49.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:49.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:49.664035+0000) 2024-02-20T02:06:50.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:50.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:50.664241+0000) 2024-02-20T02:06:51.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:51.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:51.664405+0000) 2024-02-20T02:06:52.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:52.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:52.664570+0000) 2024-02-20T02:06:53.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:53.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:53.664795+0000) 2024-02-20T02:06:54.662 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:54.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:54.664992+0000) 2024-02-20T02:06:55.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:55.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:55.665149+0000) 2024-02-20T02:06:56.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:56.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:56.665360+0000) 2024-02-20T02:06:57.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:57.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:57.665593+0000) 2024-02-20T02:06:58.663 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:58.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:58.665766+0000) 2024-02-20T02:06:59.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:06:59.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:06:59.665967+0000) 2024-02-20T02:07:00.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:00.661+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:00.666249+0000) 2024-02-20T02:07:01.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:01.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:01.666406+0000) 2024-02-20T02:07:02.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:02.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:02.666581+0000) 2024-02-20T02:07:03.664 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:03.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:03.666781+0000) 2024-02-20T02:07:04.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:04.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:04.666931+0000) 2024-02-20T02:07:05.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:05.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:05.667158+0000) 2024-02-20T02:07:06.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:06.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:06.667368+0000) 2024-02-20T02:07:07.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:07.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:07.667626+0000) 2024-02-20T02:07:08.665 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:08.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:08.667824+0000) 2024-02-20T02:07:09.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:09.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:09.668025+0000) 2024-02-20T02:07:10.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:10.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:10.668169+0000) 2024-02-20T02:07:11.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:11.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:11.668326+0000) 2024-02-20T02:07:12.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:12.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:12.668470+0000) 2024-02-20T02:07:13.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:13.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:13.668638+0000) 2024-02-20T02:07:14.666 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:14.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:14.668911+0000) 2024-02-20T02:07:15.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:15.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:15.669081+0000) 2024-02-20T02:07:16.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:16.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:16.669269+0000) 2024-02-20T02:07:17.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:17.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:17.669431+0000) 2024-02-20T02:07:18.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:18.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:18.669591+0000) 2024-02-20T02:07:19.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:19.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:19.669763+0000) 2024-02-20T02:07:20.667 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:20.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:20.669957+0000) 2024-02-20T02:07:21.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:21.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:21.670148+0000) 2024-02-20T02:07:22.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:22.665+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:22.670429+0000) 2024-02-20T02:07:23.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:23.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:23.670669+0000) 2024-02-20T02:07:24.668 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:24.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:24.670820+0000) 2024-02-20T02:07:25.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:25.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:25.671078+0000) 2024-02-20T02:07:26.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:26.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:26.671251+0000) 2024-02-20T02:07:27.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:27.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:27.671399+0000) 2024-02-20T02:07:28.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:28.669+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:28.671633+0000) 2024-02-20T02:07:29.669 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:29.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:29.671888+0000) 2024-02-20T02:07:30.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:30.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:30.672100+0000) 2024-02-20T02:07:31.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:31.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:31.672255+0000) 2024-02-20T02:07:32.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:32.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:32.672490+0000) 2024-02-20T02:07:33.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:33.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:33.672727+0000) 2024-02-20T02:07:34.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:34.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:34.672903+0000) 2024-02-20T02:07:35.670 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:35.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:35.673141+0000) 2024-02-20T02:07:36.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:36.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:36.673317+0000) 2024-02-20T02:07:37.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:37.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:37.673561+0000) 2024-02-20T02:07:38.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:38.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:38.673798+0000) 2024-02-20T02:07:39.671 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:39.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:39.673954+0000) 2024-02-20T02:07:40.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:40.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:40.674167+0000) 2024-02-20T02:07:41.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:41.668+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:41.674405+0000) 2024-02-20T02:07:42.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:42.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:42.674754+0000) 2024-02-20T02:07:43.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:43.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:43.674901+0000) 2024-02-20T02:07:44.672 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:44.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:44.675047+0000) 2024-02-20T02:07:45.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:45.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:45.675196+0000) 2024-02-20T02:07:46.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:46.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:46.675344+0000) 2024-02-20T02:07:47.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:47.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:47.675477+0000) 2024-02-20T02:07:48.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:48.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:48.675602+0000) 2024-02-20T02:07:49.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:49.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:49.675790+0000) 2024-02-20T02:07:50.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:50.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:50.675925+0000) 2024-02-20T02:07:51.673 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:51.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:51.676162+0000) 2024-02-20T02:07:52.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:52.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:52.676305+0000) 2024-02-20T02:07:53.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:53.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:53.676497+0000) 2024-02-20T02:07:54.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:54.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:54.676701+0000) 2024-02-20T02:07:55.674 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:55.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:55.676942+0000) 2024-02-20T02:07:56.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:56.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:56.677229+0000) 2024-02-20T02:07:57.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:57.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:57.677447+0000) 2024-02-20T02:07:58.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:58.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:58.677691+0000) 2024-02-20T02:07:59.675 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:07:59.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:07:59.677929+0000) 2024-02-20T02:08:00.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:00.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:00.678136+0000) 2024-02-20T02:08:01.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:01.672+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:01.678397+0000) 2024-02-20T02:08:02.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:02.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:02.678611+0000) 2024-02-20T02:08:03.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:03.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:03.678771+0000) 2024-02-20T02:08:04.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:04.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:04.678920+0000) 2024-02-20T02:08:05.676 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:05.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:05.679073+0000) 2024-02-20T02:08:06.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:06.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:06.679227+0000) 2024-02-20T02:08:07.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:07.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:07.679459+0000) 2024-02-20T02:08:08.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:08.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:08.679707+0000) 2024-02-20T02:08:09.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:09.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:09.679895+0000) 2024-02-20T02:08:10.677 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:10.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:10.680078+0000) 2024-02-20T02:08:11.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:11.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:11.680380+0000) 2024-02-20T02:08:12.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:12.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:12.680663+0000) 2024-02-20T02:08:13.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:13.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:13.680905+0000) 2024-02-20T02:08:14.678 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:14.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:14.681074+0000) 2024-02-20T02:08:15.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:15.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:15.681370+0000) 2024-02-20T02:08:16.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:16.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:16.681542+0000) 2024-02-20T02:08:17.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:17.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:17.681701+0000) 2024-02-20T02:08:18.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:18.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:18.681887+0000) 2024-02-20T02:08:19.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:19.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:19.682143+0000) 2024-02-20T02:08:20.679 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:20.676+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:20.682357+0000) 2024-02-20T02:08:21.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:21.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:21.682615+0000) 2024-02-20T02:08:22.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:22.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:22.682851+0000) 2024-02-20T02:08:23.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:23.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:23.683156+0000) 2024-02-20T02:08:24.680 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:24.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:24.683367+0000) 2024-02-20T02:08:25.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:25.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:25.683616+0000) 2024-02-20T02:08:26.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:26.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:26.683838+0000) 2024-02-20T02:08:27.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:27.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:27.683999+0000) 2024-02-20T02:08:28.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:28.680+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:28.684153+0000) 2024-02-20T02:08:29.681 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:29.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:29.684303+0000) 2024-02-20T02:08:30.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:30.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:30.684568+0000) 2024-02-20T02:08:31.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:31.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:31.684758+0000) 2024-02-20T02:08:32.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:32.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:32.684895+0000) 2024-02-20T02:08:33.682 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:33.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:33.685083+0000) 2024-02-20T02:08:34.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:34.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:34.685320+0000) 2024-02-20T02:08:35.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:35.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:35.685554+0000) 2024-02-20T02:08:36.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:36.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:36.685804+0000) 2024-02-20T02:08:37.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:37.679+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:37.686062+0000) 2024-02-20T02:08:38.683 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:38.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:38.686297+0000) 2024-02-20T02:08:39.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:39.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:39.686495+0000) 2024-02-20T02:08:40.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:40.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:40.686804+0000) 2024-02-20T02:08:41.684 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:41.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:41.687057+0000) 2024-02-20T02:08:42.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:42.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:42.687326+0000) 2024-02-20T02:08:43.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:43.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:43.687575+0000) 2024-02-20T02:08:44.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:44.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:44.687835+0000) 2024-02-20T02:08:45.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:45.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:45.688082+0000) 2024-02-20T02:08:46.685 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:46.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:46.688344+0000) 2024-02-20T02:08:47.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:47.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:47.688541+0000) 2024-02-20T02:08:48.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:48.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:48.688692+0000) 2024-02-20T02:08:49.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:49.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:49.688914+0000) 2024-02-20T02:08:50.686 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:50.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:50.689201+0000) 2024-02-20T02:08:51.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:51.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:51.689462+0000) 2024-02-20T02:08:52.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:52.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:52.689733+0000) 2024-02-20T02:08:53.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:53.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:53.689964+0000) 2024-02-20T02:08:54.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:54.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:54.690184+0000) 2024-02-20T02:08:55.687 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:55.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:55.690398+0000) 2024-02-20T02:08:56.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:56.683+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:56.690619+0000) 2024-02-20T02:08:57.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:57.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:57.690875+0000) 2024-02-20T02:08:58.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:58.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:58.691219+0000) 2024-02-20T02:08:59.688 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:08:59.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:08:59.691396+0000) 2024-02-20T02:09:00.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:00.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:00.691656+0000) 2024-02-20T02:09:01.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:01.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:01.691923+0000) 2024-02-20T02:09:02.689 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:02.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:02.692100+0000) 2024-02-20T02:09:03.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:03.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:03.692252+0000) 2024-02-20T02:09:04.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:04.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:04.692458+0000) 2024-02-20T02:09:05.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:05.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:05.692704+0000) 2024-02-20T02:09:06.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:06.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:06.692839+0000) 2024-02-20T02:09:07.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:07.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:07.693095+0000) 2024-02-20T02:09:08.690 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:08.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:08.693378+0000) 2024-02-20T02:09:09.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:09.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:09.693622+0000) 2024-02-20T02:09:10.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:10.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:10.693892+0000) 2024-02-20T02:09:11.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:11.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:11.694031+0000) 2024-02-20T02:09:12.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:12.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:12.694200+0000) 2024-02-20T02:09:13.691 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:13.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:13.694386+0000) 2024-02-20T02:09:14.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:14.687+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:14.694600+0000) 2024-02-20T02:09:15.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:15.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:15.694790+0000) 2024-02-20T02:09:16.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:16.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:16.695047+0000) 2024-02-20T02:09:17.692 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:17.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:17.695316+0000) 2024-02-20T02:09:18.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:18.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:18.695490+0000) 2024-02-20T02:09:19.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:19.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:19.695643+0000) 2024-02-20T02:09:20.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:20.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:20.695797+0000) 2024-02-20T02:09:21.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:21.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:21.695933+0000) 2024-02-20T02:09:22.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:22.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:22.696194+0000) 2024-02-20T02:09:23.693 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:23.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:23.696365+0000) 2024-02-20T02:09:24.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:24.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:24.696576+0000) 2024-02-20T02:09:25.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:25.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:25.696858+0000) 2024-02-20T02:09:26.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:26.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:26.697084+0000) 2024-02-20T02:09:27.694 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:27.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:27.697357+0000) 2024-02-20T02:09:28.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:28.691+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:28.697622+0000) 2024-02-20T02:09:29.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:29.695+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:29.697817+0000) 2024-02-20T02:09:30.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:30.690+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:30.698038+0000) 2024-02-20T02:09:31.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:31.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:31.698306+0000) 2024-02-20T02:09:32.695 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:32.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:32.698501+0000) 2024-02-20T02:09:33.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:33.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:33.698766+0000) 2024-02-20T02:09:34.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:34.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:34.699038+0000) 2024-02-20T02:09:35.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:35.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:35.699279+0000) 2024-02-20T02:09:36.696 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:36.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:36.699488+0000) 2024-02-20T02:09:37.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:37.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:37.699739+0000) 2024-02-20T02:09:38.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:38.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:38.699891+0000) 2024-02-20T02:09:39.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:39.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:39.700155+0000) 2024-02-20T02:09:40.697 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:40.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:40.700303+0000) 2024-02-20T02:09:41.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:41.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:41.700529+0000) 2024-02-20T02:09:42.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:42.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:42.700837+0000) 2024-02-20T02:09:43.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:43.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:43.701067+0000) 2024-02-20T02:09:44.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:44.694+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:44.701330+0000) 2024-02-20T02:09:45.698 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:45.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:45.701521+0000) 2024-02-20T02:09:46.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:46.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:46.701803+0000) 2024-02-20T02:09:47.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:47.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:47.702003+0000) 2024-02-20T02:09:48.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:48.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:48.702196+0000) 2024-02-20T02:09:49.699 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:49.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:49.702421+0000) 2024-02-20T02:09:50.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:50.698+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:50.702667+0000) 2024-02-20T02:09:51.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:51.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:51.702906+0000) 2024-02-20T02:09:52.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:52.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:52.703066+0000) 2024-02-20T02:09:53.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:53.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:53.703228+0000) 2024-02-20T02:09:54.700 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:54.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:54.703379+0000) 2024-02-20T02:09:55.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:55.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:55.703654+0000) 2024-02-20T02:09:56.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:56.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:56.703840+0000) 2024-02-20T02:09:57.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:57.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:57.704040+0000) 2024-02-20T02:09:58.701 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:58.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:58.704269+0000) 2024-02-20T02:09:59.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:09:59.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:09:59.704533+0000) 2024-02-20T02:10:00.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:00.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:00.704842+0000) 2024-02-20T02:10:01.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:01.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:01.705125+0000) 2024-02-20T02:10:02.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:02.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:02.705347+0000) 2024-02-20T02:10:03.702 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:03.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:03.705617+0000) 2024-02-20T02:10:04.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:04.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:04.705802+0000) 2024-02-20T02:10:05.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:05.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:05.706018+0000) 2024-02-20T02:10:06.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:06.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:06.706234+0000) 2024-02-20T02:10:07.703 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:07.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:07.706473+0000) 2024-02-20T02:10:08.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:08.702+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:08.706667+0000) 2024-02-20T02:10:09.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:09.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:09.706847+0000) 2024-02-20T02:10:10.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:10.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:10.707074+0000) 2024-02-20T02:10:11.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:11.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:11.707286+0000) 2024-02-20T02:10:12.704 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:12.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:12.707581+0000) 2024-02-20T02:10:13.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:13.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:13.707853+0000) 2024-02-20T02:10:14.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:14.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:14.708109+0000) 2024-02-20T02:10:15.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:15.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:15.708282+0000) 2024-02-20T02:10:16.705 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:16.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:16.708509+0000) 2024-02-20T02:10:17.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:17.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:17.708765+0000) 2024-02-20T02:10:18.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:18.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:18.708986+0000) 2024-02-20T02:10:19.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:19.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:19.709121+0000) 2024-02-20T02:10:20.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:20.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:20.709313+0000) 2024-02-20T02:10:21.706 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:21.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:21.709566+0000) 2024-02-20T02:10:22.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:22.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:22.709734+0000) 2024-02-20T02:10:23.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:23.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:23.709939+0000) 2024-02-20T02:10:24.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:24.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:24.710078+0000) 2024-02-20T02:10:25.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:25.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:25.710221+0000) 2024-02-20T02:10:26.707 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:26.710+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:26.710429+0000) 2024-02-20T02:10:27.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:27.706+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:27.710633+0000) 2024-02-20T02:10:28.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:28.710+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:28.710911+0000) 2024-02-20T02:10:29.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:29.710+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:29.711118+0000) 2024-02-20T02:10:30.708 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:30.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:30.711398+0000) 2024-02-20T02:10:31.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:31.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:31.711659+0000) 2024-02-20T02:10:32.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:32.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:32.711842+0000) 2024-02-20T02:10:33.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:33.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:33.712013+0000) 2024-02-20T02:10:34.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:34.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:34.712209+0000) 2024-02-20T02:10:35.709 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:35.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:35.712457+0000) 2024-02-20T02:10:36.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:36.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:36.712647+0000) 2024-02-20T02:10:37.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:37.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:37.712940+0000) 2024-02-20T02:10:38.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:38.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:38.713111+0000) 2024-02-20T02:10:39.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:39.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:39.713411+0000) 2024-02-20T02:10:40.710 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:40.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:40.713669+0000) 2024-02-20T02:10:41.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:41.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:41.713889+0000) 2024-02-20T02:10:42.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:42.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:42.714054+0000) 2024-02-20T02:10:43.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:43.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:43.714238+0000) 2024-02-20T02:10:44.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:44.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:44.714505+0000) 2024-02-20T02:10:45.711 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:45.709+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:45.714681+0000) 2024-02-20T02:10:46.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:46.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:46.714941+0000) 2024-02-20T02:10:47.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:47.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:47.715109+0000) 2024-02-20T02:10:48.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:48.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:48.715245+0000) 2024-02-20T02:10:49.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:49.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:49.715381+0000) 2024-02-20T02:10:50.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:50.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:50.715571+0000) 2024-02-20T02:10:51.712 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:51.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:51.715736+0000) 2024-02-20T02:10:52.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:52.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:52.715938+0000) 2024-02-20T02:10:53.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:53.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:53.716190+0000) 2024-02-20T02:10:54.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:54.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:54.716451+0000) 2024-02-20T02:10:55.713 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:55.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:55.716702+0000) 2024-02-20T02:10:56.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:56.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:56.717010+0000) 2024-02-20T02:10:57.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:57.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:57.717282+0000) 2024-02-20T02:10:58.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:58.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:58.717447+0000) 2024-02-20T02:10:59.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:10:59.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:10:59.717656+0000) 2024-02-20T02:11:00.714 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:00.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:00.717981+0000) 2024-02-20T02:11:01.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:01.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:01.718127+0000) 2024-02-20T02:11:02.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:02.713+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:02.718318+0000) 2024-02-20T02:11:03.715 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:03.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:03.718517+0000) 2024-02-20T02:11:04.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:04.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:04.718790+0000) 2024-02-20T02:11:05.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:05.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:05.718972+0000) 2024-02-20T02:11:06.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:06.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:06.719142+0000) 2024-02-20T02:11:07.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:07.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:07.719383+0000) 2024-02-20T02:11:08.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:08.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:08.719583+0000) 2024-02-20T02:11:09.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:09.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:09.719747+0000) 2024-02-20T02:11:10.716 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:10.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:10.719910+0000) 2024-02-20T02:11:11.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:11.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:11.720082+0000) 2024-02-20T02:11:12.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:12.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:12.720321+0000) 2024-02-20T02:11:13.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:13.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:13.720585+0000) 2024-02-20T02:11:14.717 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:14.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:14.720762+0000) 2024-02-20T02:11:15.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:15.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:15.720993+0000) 2024-02-20T02:11:16.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:16.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:16.721182+0000) 2024-02-20T02:11:17.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:17.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:17.721418+0000) 2024-02-20T02:11:18.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:18.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:18.721580+0000) 2024-02-20T02:11:19.718 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:19.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:19.721759+0000) 2024-02-20T02:11:20.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:20.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:20.722018+0000) 2024-02-20T02:11:21.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:21.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:21.722259+0000) 2024-02-20T02:11:22.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:22.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:22.722443+0000) 2024-02-20T02:11:23.719 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:23.717+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:23.722660+0000) 2024-02-20T02:11:24.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:24.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:24.722844+0000) 2024-02-20T02:11:25.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:25.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:25.723106+0000) 2024-02-20T02:11:26.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:26.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:26.723316+0000) 2024-02-20T02:11:27.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:27.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:27.723566+0000) 2024-02-20T02:11:28.720 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:28.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:28.723816+0000) 2024-02-20T02:11:29.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:29.721+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:29.724044+0000) 2024-02-20T02:11:30.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:30.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:30.724177+0000) 2024-02-20T02:11:31.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:31.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:31.724424+0000) 2024-02-20T02:11:32.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:32.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:32.724594+0000) 2024-02-20T02:11:33.721 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:33.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:33.724749+0000) 2024-02-20T02:11:34.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:34.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:34.724936+0000) 2024-02-20T02:11:35.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:35.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:35.725163+0000) 2024-02-20T02:11:36.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:36.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:36.725311+0000) 2024-02-20T02:11:37.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:37.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:37.725567+0000) 2024-02-20T02:11:38.722 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:38.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:38.725850+0000) 2024-02-20T02:11:39.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:39.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:39.726068+0000) 2024-02-20T02:11:40.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:40.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:40.726237+0000) 2024-02-20T02:11:41.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:41.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:41.726376+0000) 2024-02-20T02:11:42.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:42.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:42.726540+0000) 2024-02-20T02:11:43.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:43.720+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:43.726682+0000) 2024-02-20T02:11:44.723 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:44.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:44.726859+0000) 2024-02-20T02:11:45.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:45.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:45.727095+0000) 2024-02-20T02:11:46.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:46.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:46.727333+0000) 2024-02-20T02:11:47.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:47.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:47.727598+0000) 2024-02-20T02:11:48.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:48.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:48.727778+0000) 2024-02-20T02:11:49.724 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:49.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:49.727973+0000) 2024-02-20T02:11:50.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:50.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:50.728192+0000) 2024-02-20T02:11:51.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:51.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:51.728445+0000) 2024-02-20T02:11:52.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:52.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:52.728692+0000) 2024-02-20T02:11:53.725 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:53.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:53.728870+0000) 2024-02-20T02:11:54.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:54.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:54.729098+0000) 2024-02-20T02:11:55.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:55.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:55.729266+0000) 2024-02-20T02:11:56.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:56.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:56.729414+0000) 2024-02-20T02:11:57.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:57.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:57.729556+0000) 2024-02-20T02:11:58.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:58.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:58.729714+0000) 2024-02-20T02:11:59.726 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:11:59.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:11:59.729842+0000) 2024-02-20T02:12:00.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:00.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:00.729979+0000) 2024-02-20T02:12:01.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:01.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:01.730211+0000) 2024-02-20T02:12:02.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:02.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:02.730417+0000) 2024-02-20T02:12:03.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:03.724+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:03.730559+0000) 2024-02-20T02:12:04.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:04.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:04.730725+0000) 2024-02-20T02:12:05.727 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:05.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:05.730938+0000) 2024-02-20T02:12:06.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:06.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:06.731135+0000) 2024-02-20T02:12:07.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:07.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:07.731345+0000) 2024-02-20T02:12:08.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:08.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:08.731503+0000) 2024-02-20T02:12:09.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:09.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:09.731741+0000) 2024-02-20T02:12:10.728 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:10.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:10.731959+0000) 2024-02-20T02:12:11.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:11.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:11.732134+0000) 2024-02-20T02:12:12.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:12.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:12.732368+0000) 2024-02-20T02:12:13.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:13.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:13.732510+0000) 2024-02-20T02:12:14.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:14.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:14.732729+0000) 2024-02-20T02:12:15.729 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:15.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:15.732877+0000) 2024-02-20T02:12:16.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:16.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:16.733168+0000) 2024-02-20T02:12:17.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:17.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:17.733337+0000) 2024-02-20T02:12:18.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:18.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:18.733562+0000) 2024-02-20T02:12:19.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:19.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:19.733792+0000) 2024-02-20T02:12:20.730 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:20.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:20.734009+0000) 2024-02-20T02:12:21.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:21.728+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:21.734169+0000) 2024-02-20T02:12:22.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:22.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:22.734339+0000) 2024-02-20T02:12:23.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:23.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:23.734572+0000) 2024-02-20T02:12:24.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:24.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:24.734798+0000) 2024-02-20T02:12:25.731 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:25.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:25.735035+0000) 2024-02-20T02:12:26.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:26.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:26.735271+0000) 2024-02-20T02:12:27.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:27.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:27.735536+0000) 2024-02-20T02:12:28.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:28.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:28.735791+0000) 2024-02-20T02:12:29.732 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:29.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:29.736016+0000) 2024-02-20T02:12:30.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:30.732+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:30.736248+0000) 2024-02-20T02:12:31.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:31.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:31.736499+0000) 2024-02-20T02:12:32.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:32.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:32.736772+0000) 2024-02-20T02:12:33.733 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:33.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:33.737077+0000) 2024-02-20T02:12:34.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:34.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:34.737228+0000) 2024-02-20T02:12:35.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:35.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:35.737363+0000) 2024-02-20T02:12:36.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:36.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:36.737511+0000) 2024-02-20T02:12:37.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:37.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:37.737688+0000) 2024-02-20T02:12:38.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:38.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:38.737927+0000) 2024-02-20T02:12:39.734 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:39.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:39.738091+0000) 2024-02-20T02:12:40.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:40.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:40.738329+0000) 2024-02-20T02:12:41.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:41.731+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:41.738570+0000) 2024-02-20T02:12:42.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:42.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:42.738807+0000) 2024-02-20T02:12:43.735 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:43.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:43.739096+0000) 2024-02-20T02:12:44.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:44.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:44.739266+0000) 2024-02-20T02:12:45.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:45.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:45.739462+0000) 2024-02-20T02:12:46.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:46.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:46.739730+0000) 2024-02-20T02:12:47.736 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:47.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:47.739994+0000) 2024-02-20T02:12:48.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:48.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:48.740213+0000) 2024-02-20T02:12:49.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:49.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:49.740413+0000) 2024-02-20T02:12:50.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:50.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:50.740662+0000) 2024-02-20T02:12:51.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:51.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:51.740857+0000) 2024-02-20T02:12:52.737 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:52.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:52.741181+0000) 2024-02-20T02:12:53.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:53.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:53.741327+0000) 2024-02-20T02:12:54.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:54.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:54.741464+0000) 2024-02-20T02:12:55.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:55.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:55.741609+0000) 2024-02-20T02:12:56.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:56.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:56.741756+0000) 2024-02-20T02:12:57.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:57.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:57.741903+0000) 2024-02-20T02:12:58.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:58.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:58.742028+0000) 2024-02-20T02:12:59.738 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:12:59.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:12:59.742186+0000) 2024-02-20T02:13:00.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:00.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:00.742406+0000) 2024-02-20T02:13:01.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:01.735+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:01.742592+0000) 2024-02-20T02:13:02.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:02.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:02.742810+0000) 2024-02-20T02:13:03.739 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:03.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:03.742951+0000) 2024-02-20T02:13:04.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:04.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:04.743179+0000) 2024-02-20T02:13:05.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:05.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:05.743350+0000) 2024-02-20T02:13:06.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:06.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:06.743596+0000) 2024-02-20T02:13:07.740 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:07.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:07.743787+0000) 2024-02-20T02:13:08.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:08.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:08.744019+0000) 2024-02-20T02:13:09.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:09.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:09.744302+0000) 2024-02-20T02:13:10.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:10.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:10.744501+0000) 2024-02-20T02:13:11.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:11.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:11.744707+0000) 2024-02-20T02:13:12.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:12.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:12.744868+0000) 2024-02-20T02:13:13.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:13.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:13.745092+0000) 2024-02-20T02:13:14.741 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:14.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:14.745296+0000) 2024-02-20T02:13:15.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:15.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:15.745566+0000) 2024-02-20T02:13:16.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:16.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:16.745747+0000) 2024-02-20T02:13:17.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:17.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:17.745917+0000) 2024-02-20T02:13:18.742 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:18.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:18.746071+0000) 2024-02-20T02:13:19.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:19.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:19.746419+0000) 2024-02-20T02:13:20.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:20.739+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:20.746597+0000) 2024-02-20T02:13:21.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:21.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:21.746824+0000) 2024-02-20T02:13:22.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:22.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:22.747044+0000) 2024-02-20T02:13:23.743 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:23.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:23.747195+0000) 2024-02-20T02:13:24.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:24.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:24.747423+0000) 2024-02-20T02:13:25.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:25.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:25.747659+0000) 2024-02-20T02:13:26.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:26.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:26.747818+0000) 2024-02-20T02:13:27.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:27.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:27.747965+0000) 2024-02-20T02:13:28.744 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:28.743+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:28.748164+0000) 2024-02-20T02:13:29.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:29.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:29.748327+0000) 2024-02-20T02:13:30.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:30.747+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:30.748456+0000) 2024-02-20T02:13:31.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:31.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:31.748609+0000) 2024-02-20T02:13:32.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:32.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:32.748756+0000) 2024-02-20T02:13:33.745 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:33.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:33.748981+0000) 2024-02-20T02:13:34.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:34.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:34.749199+0000) 2024-02-20T02:13:35.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:35.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:35.749436+0000) 2024-02-20T02:13:36.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:36.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:36.749704+0000) 2024-02-20T02:13:37.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:37.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:37.749872+0000) 2024-02-20T02:13:38.746 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:38.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:38.750111+0000) 2024-02-20T02:13:39.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:39.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:39.750292+0000) 2024-02-20T02:13:40.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:40.742+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:40.750461+0000) 2024-02-20T02:13:41.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:41.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:41.750695+0000) 2024-02-20T02:13:42.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:42.746+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:42.751013+0000) 2024-02-20T02:13:43.747 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:43.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:43.751212+0000) 2024-02-20T02:13:44.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:44.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:44.751475+0000) 2024-02-20T02:13:45.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:45.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:45.751645+0000) 2024-02-20T02:13:46.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:46.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:46.751798+0000) 2024-02-20T02:13:47.748 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:47.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:47.752027+0000) 2024-02-20T02:13:48.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:48.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:48.752281+0000) 2024-02-20T02:13:49.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:49.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:49.752441+0000) 2024-02-20T02:13:50.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:50.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:50.752658+0000) 2024-02-20T02:13:51.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:51.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:51.752805+0000) 2024-02-20T02:13:52.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:52.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:52.752987+0000) 2024-02-20T02:13:53.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:53.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:53.753174+0000) 2024-02-20T02:13:54.749 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:54.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:54.753380+0000) 2024-02-20T02:13:55.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:55.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:55.753604+0000) 2024-02-20T02:13:56.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:56.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:56.753862+0000) 2024-02-20T02:13:57.750 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:57.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:57.754101+0000) 2024-02-20T02:13:58.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:58.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:58.754308+0000) 2024-02-20T02:13:59.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:13:59.750+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:13:59.754559+0000) 2024-02-20T02:14:00.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:00.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:00.754843+0000) 2024-02-20T02:14:01.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:01.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:01.755069+0000) 2024-02-20T02:14:02.751 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:02.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:02.755378+0000) 2024-02-20T02:14:03.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:03.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:03.755588+0000) 2024-02-20T02:14:04.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:04.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:04.755845+0000) 2024-02-20T02:14:05.752 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:05.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:05.756151+0000) 2024-02-20T02:14:06.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:06.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:06.756467+0000) 2024-02-20T02:14:07.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:07.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:07.756623+0000) 2024-02-20T02:14:08.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:08.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:08.756781+0000) 2024-02-20T02:14:09.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:09.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:09.756950+0000) 2024-02-20T02:14:10.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:10.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:10.757106+0000) 2024-02-20T02:14:11.753 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:11.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:11.757267+0000) 2024-02-20T02:14:12.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:12.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:12.757432+0000) 2024-02-20T02:14:13.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:13.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:13.757642+0000) 2024-02-20T02:14:14.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:14.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:14.757835+0000) 2024-02-20T02:14:15.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:15.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:15.757988+0000) 2024-02-20T02:14:16.754 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:16.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:16.758221+0000) 2024-02-20T02:14:17.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:17.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:17.758481+0000) 2024-02-20T02:14:18.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:18.754+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:18.758673+0000) 2024-02-20T02:14:19.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:19.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:19.758847+0000) 2024-02-20T02:14:20.755 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:20.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:20.759083+0000) 2024-02-20T02:14:21.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:21.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:21.759335+0000) 2024-02-20T02:14:22.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:22.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:22.759583+0000) 2024-02-20T02:14:23.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:23.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:23.759773+0000) 2024-02-20T02:14:24.756 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:24.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:24.760033+0000) 2024-02-20T02:14:25.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:25.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:25.760271+0000) 2024-02-20T02:14:26.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:26.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:26.760529+0000) 2024-02-20T02:14:27.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:27.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:27.760784+0000) 2024-02-20T02:14:28.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:28.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:28.760971+0000) 2024-02-20T02:14:29.757 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:29.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:29.761218+0000) 2024-02-20T02:14:30.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:30.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:30.761461+0000) 2024-02-20T02:14:31.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:31.758+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:31.761641+0000) 2024-02-20T02:14:32.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:32.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:32.761891+0000) 2024-02-20T02:14:33.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:33.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:33.762089+0000) 2024-02-20T02:14:34.758 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:34.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:34.762271+0000) 2024-02-20T02:14:35.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:35.757+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:35.762454+0000) 2024-02-20T02:14:36.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:36.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:36.762681+0000) 2024-02-20T02:14:37.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:37.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:37.762939+0000) 2024-02-20T02:14:38.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:38.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:38.763180+0000) 2024-02-20T02:14:39.759 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:39.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:39.763406+0000) 2024-02-20T02:14:40.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:40.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:40.763592+0000) 2024-02-20T02:14:41.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:41.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:41.763818+0000) 2024-02-20T02:14:42.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:42.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:42.763997+0000) 2024-02-20T02:14:43.760 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:43.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:43.764153+0000) 2024-02-20T02:14:44.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:44.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:44.764391+0000) 2024-02-20T02:14:45.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:45.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:45.764606+0000) 2024-02-20T02:14:46.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:46.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:46.764787+0000) 2024-02-20T02:14:47.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:47.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:47.764940+0000) 2024-02-20T02:14:48.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:48.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:48.765104+0000) 2024-02-20T02:14:49.761 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:49.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:49.765250+0000) 2024-02-20T02:14:50.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:50.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:50.765502+0000) 2024-02-20T02:14:51.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:51.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:51.765686+0000) 2024-02-20T02:14:52.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:52.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:52.765922+0000) 2024-02-20T02:14:53.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:53.761+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:53.766117+0000) 2024-02-20T02:14:54.762 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:54.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:54.766325+0000) 2024-02-20T02:14:55.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:55.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:55.766491+0000) 2024-02-20T02:14:56.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:56.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:56.766766+0000) 2024-02-20T02:14:57.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:57.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:57.766920+0000) 2024-02-20T02:14:58.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:58.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:58.767154+0000) 2024-02-20T02:14:59.763 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:14:59.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:14:59.767334+0000) 2024-02-20T02:15:00.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:00.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:00.767571+0000) 2024-02-20T02:15:01.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:01.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:01.767797+0000) 2024-02-20T02:15:02.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:02.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:02.767977+0000) 2024-02-20T02:15:03.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:03.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:03.768139+0000) 2024-02-20T02:15:04.764 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:04.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:04.768307+0000) 2024-02-20T02:15:05.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:05.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:05.768501+0000) 2024-02-20T02:15:06.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:06.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:06.768691+0000) 2024-02-20T02:15:07.765 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:07.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:07.768949+0000) 2024-02-20T02:15:08.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:08.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:08.769212+0000) 2024-02-20T02:15:09.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:09.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:09.769449+0000) 2024-02-20T02:15:10.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:10.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:10.769619+0000) 2024-02-20T02:15:11.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:11.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:11.769786+0000) 2024-02-20T02:15:12.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:12.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:12.769981+0000) 2024-02-20T02:15:13.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:13.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:13.770168+0000) 2024-02-20T02:15:14.766 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:14.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:14.770402+0000) 2024-02-20T02:15:15.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:15.765+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:15.770605+0000) 2024-02-20T02:15:16.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:16.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:16.770880+0000) 2024-02-20T02:15:17.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:17.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:17.771058+0000) 2024-02-20T02:15:18.767 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:18.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:18.771309+0000) 2024-02-20T02:15:19.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:19.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:19.771619+0000) 2024-02-20T02:15:20.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:20.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:20.771900+0000) 2024-02-20T02:15:21.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:21.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:21.772068+0000) 2024-02-20T02:15:22.768 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:22.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:22.772293+0000) 2024-02-20T02:15:23.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:23.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:23.772593+0000) 2024-02-20T02:15:24.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:24.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:24.772758+0000) 2024-02-20T02:15:25.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:25.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:25.772991+0000) 2024-02-20T02:15:26.769 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:26.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:26.773126+0000) 2024-02-20T02:15:27.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:27.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:27.773340+0000) 2024-02-20T02:15:28.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:28.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:28.773628+0000) 2024-02-20T02:15:29.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:29.773+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:29.773802+0000) 2024-02-20T02:15:30.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:30.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:30.773985+0000) 2024-02-20T02:15:31.770 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:31.769+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:31.774302+0000) 2024-02-20T02:15:32.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:32.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:32.774536+0000) 2024-02-20T02:15:33.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:33.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:33.774773+0000) 2024-02-20T02:15:34.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:34.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:34.774932+0000) 2024-02-20T02:15:35.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:35.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:35.775074+0000) 2024-02-20T02:15:36.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:36.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:36.775245+0000) 2024-02-20T02:15:37.771 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:37.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:37.775461+0000) 2024-02-20T02:15:38.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:38.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:38.775755+0000) 2024-02-20T02:15:39.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:39.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:39.776082+0000) 2024-02-20T02:15:40.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:40.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:40.776244+0000) 2024-02-20T02:15:41.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:41.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:41.776426+0000) 2024-02-20T02:15:42.772 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:42.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:42.776583+0000) 2024-02-20T02:15:43.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:43.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:43.776773+0000) 2024-02-20T02:15:44.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:44.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:44.777020+0000) 2024-02-20T02:15:45.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:45.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:45.777174+0000) 2024-02-20T02:15:46.773 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:46.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:46.777442+0000) 2024-02-20T02:15:47.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:47.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:47.777626+0000) 2024-02-20T02:15:48.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:48.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:48.777986+0000) 2024-02-20T02:15:49.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:49.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:49.778163+0000) 2024-02-20T02:15:50.774 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:50.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:50.778364+0000) 2024-02-20T02:15:51.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:51.772+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:51.778672+0000) 2024-02-20T02:15:52.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:52.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:52.778909+0000) 2024-02-20T02:15:53.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:53.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:53.779067+0000) 2024-02-20T02:15:54.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:54.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:54.779242+0000) 2024-02-20T02:15:55.775 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:55.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:55.779386+0000) 2024-02-20T02:15:56.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:56.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:56.779682+0000) 2024-02-20T02:15:57.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:57.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:57.779964+0000) 2024-02-20T02:15:58.776 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:58.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:58.780320+0000) 2024-02-20T02:15:59.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:15:59.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:15:59.780568+0000) 2024-02-20T02:16:00.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:00.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:00.780762+0000) 2024-02-20T02:16:01.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:01.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:01.780896+0000) 2024-02-20T02:16:02.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:02.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:02.781050+0000) 2024-02-20T02:16:03.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:03.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:03.781327+0000) 2024-02-20T02:16:04.777 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:04.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:04.781520+0000) 2024-02-20T02:16:05.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:05.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:05.781777+0000) 2024-02-20T02:16:06.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:06.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:06.782043+0000) 2024-02-20T02:16:07.778 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:07.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:07.782222+0000) 2024-02-20T02:16:08.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:08.776+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:08.782454+0000) 2024-02-20T02:16:09.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:09.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:09.782797+0000) 2024-02-20T02:16:10.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:10.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:10.783080+0000) 2024-02-20T02:16:11.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:11.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:11.783210+0000) 2024-02-20T02:16:12.779 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:12.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:12.783479+0000) 2024-02-20T02:16:13.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:13.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:13.783638+0000) 2024-02-20T02:16:14.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:14.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:14.783845+0000) 2024-02-20T02:16:15.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:15.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:15.784035+0000) 2024-02-20T02:16:16.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:16.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:16.784303+0000) 2024-02-20T02:16:17.780 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:17.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:17.784474+0000) 2024-02-20T02:16:18.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:18.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:18.784726+0000) 2024-02-20T02:16:19.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:19.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:19.784964+0000) 2024-02-20T02:16:20.781 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:20.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:20.785230+0000) 2024-02-20T02:16:21.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:21.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:21.785445+0000) 2024-02-20T02:16:22.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:22.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:22.785727+0000) 2024-02-20T02:16:23.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:23.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:23.786041+0000) 2024-02-20T02:16:24.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:24.780+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:24.786263+0000) 2024-02-20T02:16:25.782 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:25.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:25.786453+0000) 2024-02-20T02:16:26.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:26.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:26.786678+0000) 2024-02-20T02:16:27.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:27.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:27.786885+0000) 2024-02-20T02:16:28.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:28.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:28.787139+0000) 2024-02-20T02:16:29.783 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:29.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:29.787376+0000) 2024-02-20T02:16:30.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:30.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:30.787652+0000) 2024-02-20T02:16:31.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:31.784+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:31.787837+0000) 2024-02-20T02:16:32.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:32.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:32.788010+0000) 2024-02-20T02:16:33.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:33.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:33.788173+0000) 2024-02-20T02:16:34.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:34.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:34.788471+0000) 2024-02-20T02:16:35.784 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:35.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:35.788696+0000) 2024-02-20T02:16:36.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:36.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:36.788986+0000) 2024-02-20T02:16:37.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:37.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:37.789155+0000) 2024-02-20T02:16:38.785 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:38.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:38.789405+0000) 2024-02-20T02:16:39.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:39.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:39.789676+0000) 2024-02-20T02:16:40.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:40.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:40.790041+0000) 2024-02-20T02:16:41.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:41.783+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:41.790178+0000) 2024-02-20T02:16:42.786 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:42.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:42.790378+0000) 2024-02-20T02:16:43.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:43.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:43.790647+0000) 2024-02-20T02:16:44.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:44.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:44.790859+0000) 2024-02-20T02:16:45.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:45.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:45.791058+0000) 2024-02-20T02:16:46.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:46.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:46.791329+0000) 2024-02-20T02:16:47.787 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:47.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:47.791630+0000) 2024-02-20T02:16:48.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:48.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:48.791786+0000) 2024-02-20T02:16:49.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:49.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:49.791919+0000) 2024-02-20T02:16:50.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:50.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:50.792066+0000) 2024-02-20T02:16:51.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:51.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:51.792247+0000) 2024-02-20T02:16:52.788 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:52.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:52.792480+0000) 2024-02-20T02:16:53.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:53.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:53.792680+0000) 2024-02-20T02:16:54.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:54.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:54.792975+0000) 2024-02-20T02:16:55.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:55.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:55.793170+0000) 2024-02-20T02:16:56.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:56.787+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:56.793323+0000) 2024-02-20T02:16:57.789 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:57.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:57.793590+0000) 2024-02-20T02:16:58.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:58.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:58.793873+0000) 2024-02-20T02:16:59.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:16:59.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:16:59.794112+0000) 2024-02-20T02:17:00.790 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:00.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:00.794377+0000) 2024-02-20T02:17:01.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:01.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:01.794679+0000) 2024-02-20T02:17:02.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:02.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:02.794939+0000) 2024-02-20T02:17:03.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:03.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:03.795093+0000) 2024-02-20T02:17:04.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:04.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:04.795231+0000) 2024-02-20T02:17:05.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:05.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:05.795501+0000) 2024-02-20T02:17:06.791 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:06.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:06.795737+0000) 2024-02-20T02:17:07.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:07.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:07.795989+0000) 2024-02-20T02:17:08.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:08.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:08.796173+0000) 2024-02-20T02:17:09.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:09.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:09.796367+0000) 2024-02-20T02:17:10.792 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:10.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:10.796555+0000) 2024-02-20T02:17:11.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:11.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:11.796785+0000) 2024-02-20T02:17:12.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:12.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:12.796998+0000) 2024-02-20T02:17:13.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:13.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:13.797181+0000) 2024-02-20T02:17:14.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:14.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:14.797443+0000) 2024-02-20T02:17:15.793 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:15.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:15.797684+0000) 2024-02-20T02:17:16.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:16.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:16.797950+0000) 2024-02-20T02:17:17.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:17.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:17.798230+0000) 2024-02-20T02:17:18.794 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:18.791+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:18.798576+0000) 2024-02-20T02:17:19.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:19.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:19.798829+0000) 2024-02-20T02:17:20.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:20.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:20.799030+0000) 2024-02-20T02:17:21.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:21.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:21.799289+0000) 2024-02-20T02:17:22.795 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:22.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:22.799631+0000) 2024-02-20T02:17:23.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:23.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:23.799884+0000) 2024-02-20T02:17:24.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:24.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:24.800029+0000) 2024-02-20T02:17:25.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:25.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:25.800192+0000) 2024-02-20T02:17:26.796 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:26.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:26.800396+0000) 2024-02-20T02:17:27.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:27.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:27.800681+0000) 2024-02-20T02:17:28.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:28.795+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:28.800903+0000) 2024-02-20T02:17:29.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:29.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:29.801154+0000) 2024-02-20T02:17:30.797 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:30.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:30.801399+0000) 2024-02-20T02:17:31.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:31.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:31.801648+0000) 2024-02-20T02:17:32.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:32.799+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:32.801964+0000) 2024-02-20T02:17:33.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:33.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:33.802255+0000) 2024-02-20T02:17:34.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:34.794+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:34.802491+0000) 2024-02-20T02:17:35.798 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:35.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:35.802729+0000) 2024-02-20T02:17:36.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:36.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:36.802933+0000) 2024-02-20T02:17:37.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:37.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:37.803193+0000) 2024-02-20T02:17:38.799 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:38.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:38.803554+0000) 2024-02-20T02:17:39.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:39.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:39.803821+0000) 2024-02-20T02:17:40.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:40.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:40.804016+0000) 2024-02-20T02:17:41.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:41.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:41.804285+0000) 2024-02-20T02:17:42.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:42.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:42.804470+0000) 2024-02-20T02:17:43.800 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:43.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:43.804719+0000) 2024-02-20T02:17:44.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:44.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:44.804936+0000) 2024-02-20T02:17:45.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:45.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:45.805156+0000) 2024-02-20T02:17:46.801 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:46.798+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:46.805394+0000) 2024-02-20T02:17:47.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:47.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:47.805671+0000) 2024-02-20T02:17:48.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:48.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:48.805933+0000) 2024-02-20T02:17:49.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:49.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:49.806101+0000) 2024-02-20T02:17:50.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:50.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:50.806378+0000) 2024-02-20T02:17:51.802 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:51.802+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:51.806689+0000) 2024-02-20T02:17:52.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:52.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:52.806864+0000) 2024-02-20T02:17:53.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:53.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:53.807137+0000) 2024-02-20T02:17:54.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:54.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:54.807319+0000) 2024-02-20T02:17:55.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:55.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:55.807542+0000) 2024-02-20T02:17:56.803 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:56.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:56.807766+0000) 2024-02-20T02:17:57.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:57.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:57.808048+0000) 2024-02-20T02:17:58.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:58.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:58.808321+0000) 2024-02-20T02:17:59.804 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:17:59.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:17:59.808600+0000) 2024-02-20T02:18:00.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:00.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:00.808875+0000) 2024-02-20T02:18:01.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:01.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:01.809175+0000) 2024-02-20T02:18:02.805 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:02.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:02.809476+0000) 2024-02-20T02:18:03.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:03.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:03.809708+0000) 2024-02-20T02:18:04.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:04.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:04.809904+0000) 2024-02-20T02:18:05.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:05.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:05.810156+0000) 2024-02-20T02:18:06.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:06.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:06.810486+0000) 2024-02-20T02:18:07.806 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:07.806+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:07.810642+0000) 2024-02-20T02:18:08.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:08.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:08.810880+0000) 2024-02-20T02:18:09.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:09.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:09.811021+0000) 2024-02-20T02:18:10.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:10.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:10.811180+0000) 2024-02-20T02:18:11.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:11.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:11.811423+0000) 2024-02-20T02:18:12.807 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:12.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:12.811607+0000) 2024-02-20T02:18:13.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:13.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:13.811887+0000) 2024-02-20T02:18:14.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:14.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:14.812174+0000) 2024-02-20T02:18:15.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:15.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:15.812318+0000) 2024-02-20T02:18:16.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:16.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:16.812474+0000) 2024-02-20T02:18:17.808 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:17.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:17.812700+0000) 2024-02-20T02:18:18.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:18.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:18.812964+0000) 2024-02-20T02:18:19.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:19.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:19.813226+0000) 2024-02-20T02:18:20.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:20.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:20.813464+0000) 2024-02-20T02:18:21.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:21.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:21.813705+0000) 2024-02-20T02:18:22.809 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:22.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:22.813908+0000) 2024-02-20T02:18:23.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:23.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:23.814058+0000) 2024-02-20T02:18:24.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:24.810+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:24.814269+0000) 2024-02-20T02:18:25.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:25.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:25.814510+0000) 2024-02-20T02:18:26.810 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:26.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:26.814794+0000) 2024-02-20T02:18:27.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:27.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:27.814983+0000) 2024-02-20T02:18:28.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:28.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:28.815207+0000) 2024-02-20T02:18:29.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:29.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:29.815378+0000) 2024-02-20T02:18:30.811 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:30.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:30.815613+0000) 2024-02-20T02:18:31.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:31.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:31.815865+0000) 2024-02-20T02:18:32.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:32.814+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:32.816041+0000) 2024-02-20T02:18:33.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:33.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:33.816263+0000) 2024-02-20T02:18:34.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:34.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:34.816400+0000) 2024-02-20T02:18:35.812 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:35.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:35.816543+0000) 2024-02-20T02:18:36.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:36.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:36.816828+0000) 2024-02-20T02:18:37.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:37.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:37.817028+0000) 2024-02-20T02:18:38.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:38.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:38.817269+0000) 2024-02-20T02:18:39.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:39.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:39.817437+0000) 2024-02-20T02:18:40.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:40.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:40.817633+0000) 2024-02-20T02:18:41.813 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:41.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:41.817774+0000) 2024-02-20T02:18:42.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:42.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:42.817924+0000) 2024-02-20T02:18:43.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:43.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:43.818146+0000) 2024-02-20T02:18:44.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:44.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:44.818372+0000) 2024-02-20T02:18:45.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:45.813+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:45.818503+0000) 2024-02-20T02:18:46.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:46.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:46.818649+0000) 2024-02-20T02:18:47.814 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:47.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:47.818829+0000) 2024-02-20T02:18:48.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:48.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:48.819066+0000) 2024-02-20T02:18:49.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:49.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:49.819334+0000) 2024-02-20T02:18:50.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:50.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:50.819508+0000) 2024-02-20T02:18:51.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:51.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:51.819714+0000) 2024-02-20T02:18:52.815 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:52.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:52.819905+0000) 2024-02-20T02:18:53.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:53.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:53.820143+0000) 2024-02-20T02:18:54.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:54.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:54.820302+0000) 2024-02-20T02:18:55.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:55.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:55.820543+0000) 2024-02-20T02:18:56.816 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:56.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:56.820795+0000) 2024-02-20T02:18:57.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:57.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:57.821062+0000) 2024-02-20T02:18:58.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:58.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:58.821328+0000) 2024-02-20T02:18:59.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:18:59.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:18:59.821512+0000) 2024-02-20T02:19:00.817 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:00.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:00.821666+0000) 2024-02-20T02:19:01.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:01.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:01.821936+0000) 2024-02-20T02:19:02.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:02.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:02.822185+0000) 2024-02-20T02:19:03.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:03.817+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:03.822361+0000) 2024-02-20T02:19:04.818 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:04.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:04.822529+0000) 2024-02-20T02:19:05.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:05.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:05.822795+0000) 2024-02-20T02:19:06.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:06.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:06.822930+0000) 2024-02-20T02:19:07.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:07.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:07.823135+0000) 2024-02-20T02:19:08.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:08.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:08.823379+0000) 2024-02-20T02:19:09.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:09.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:09.823657+0000) 2024-02-20T02:19:10.819 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:10.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:10.823874+0000) 2024-02-20T02:19:11.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:11.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:11.824118+0000) 2024-02-20T02:19:12.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:12.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:12.824385+0000) 2024-02-20T02:19:13.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:13.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:13.824582+0000) 2024-02-20T02:19:14.820 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:14.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:14.824819+0000) 2024-02-20T02:19:15.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:15.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:15.825033+0000) 2024-02-20T02:19:16.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:16.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:16.825372+0000) 2024-02-20T02:19:17.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:17.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:17.825602+0000) 2024-02-20T02:19:18.821 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:18.821+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:18.825815+0000) 2024-02-20T02:19:19.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:19.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:19.826099+0000) 2024-02-20T02:19:20.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:20.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:20.826367+0000) 2024-02-20T02:19:21.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:21.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:21.826551+0000) 2024-02-20T02:19:22.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:22.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:22.826703+0000) 2024-02-20T02:19:23.822 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:23.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:23.826889+0000) 2024-02-20T02:19:24.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:24.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:24.827143+0000) 2024-02-20T02:19:25.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:25.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:25.827348+0000) 2024-02-20T02:19:26.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:26.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:26.827550+0000) 2024-02-20T02:19:27.823 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:27.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:27.827786+0000) 2024-02-20T02:19:28.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:28.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:28.828052+0000) 2024-02-20T02:19:29.824 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:29.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:29.828353+0000) 2024-02-20T02:19:30.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:30.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:30.828618+0000) 2024-02-20T02:19:31.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:31.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:31.828948+0000) 2024-02-20T02:19:32.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:32.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:32.829222+0000) 2024-02-20T02:19:33.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:33.825+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:33.829512+0000) 2024-02-20T02:19:34.825 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:34.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:34.829712+0000) 2024-02-20T02:19:35.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:35.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:35.829869+0000) 2024-02-20T02:19:36.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:36.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:36.830160+0000) 2024-02-20T02:19:37.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:37.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:37.830345+0000) 2024-02-20T02:19:38.826 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:38.824+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:38.830543+0000) 2024-02-20T02:19:39.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:39.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:39.830890+0000) 2024-02-20T02:19:40.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:40.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:40.831053+0000) 2024-02-20T02:19:41.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:41.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:41.831251+0000) 2024-02-20T02:19:42.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:42.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:42.831547+0000) 2024-02-20T02:19:43.827 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:43.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:43.831884+0000) 2024-02-20T02:19:44.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:44.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:44.832052+0000) 2024-02-20T02:19:45.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:45.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:45.832206+0000) 2024-02-20T02:19:46.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:46.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:46.832360+0000) 2024-02-20T02:19:47.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:47.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:47.832592+0000) 2024-02-20T02:19:48.828 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:48.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:48.832853+0000) 2024-02-20T02:19:49.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:49.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:49.833163+0000) 2024-02-20T02:19:50.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:50.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:50.833351+0000) 2024-02-20T02:19:51.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:51.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:51.833596+0000) 2024-02-20T02:19:52.829 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:52.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:52.833855+0000) 2024-02-20T02:19:53.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:53.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:53.834049+0000) 2024-02-20T02:19:54.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:54.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:54.834275+0000) 2024-02-20T02:19:55.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:55.828+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:55.834544+0000) 2024-02-20T02:19:56.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:56.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:56.834723+0000) 2024-02-20T02:19:57.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:57.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:57.834884+0000) 2024-02-20T02:19:58.830 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:58.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:58.835096+0000) 2024-02-20T02:19:59.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:19:59.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:19:59.835343+0000) 2024-02-20T02:20:00.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:00.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:00.835501+0000) 2024-02-20T02:20:01.831 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:01.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:01.835652+0000) 2024-02-20T02:20:02.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:02.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:02.835972+0000) 2024-02-20T02:20:03.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:03.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:03.836258+0000) 2024-02-20T02:20:04.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:04.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:04.836428+0000) 2024-02-20T02:20:05.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:05.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:05.836574+0000) 2024-02-20T02:20:06.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:06.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:06.836788+0000) 2024-02-20T02:20:07.832 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:07.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:07.836973+0000) 2024-02-20T02:20:08.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:08.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:08.837216+0000) 2024-02-20T02:20:09.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:09.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:09.837428+0000) 2024-02-20T02:20:10.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:10.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:10.837614+0000) 2024-02-20T02:20:11.833 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:11.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:11.837782+0000) 2024-02-20T02:20:12.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:12.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:12.838048+0000) 2024-02-20T02:20:13.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:13.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:13.838263+0000) 2024-02-20T02:20:14.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:14.832+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:14.838509+0000) 2024-02-20T02:20:15.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:15.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:15.838672+0000) 2024-02-20T02:20:16.834 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:16.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:16.838936+0000) 2024-02-20T02:20:17.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:17.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:17.839227+0000) 2024-02-20T02:20:18.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:18.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:18.839366+0000) 2024-02-20T02:20:19.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:19.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:19.839523+0000) 2024-02-20T02:20:20.835 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:20.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:20.839669+0000) 2024-02-20T02:20:21.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:21.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:21.839941+0000) 2024-02-20T02:20:22.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:22.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:22.840286+0000) 2024-02-20T02:20:23.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:23.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:23.840460+0000) 2024-02-20T02:20:24.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:24.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:24.840755+0000) 2024-02-20T02:20:25.836 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:25.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:25.840932+0000) 2024-02-20T02:20:26.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:26.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:26.841126+0000) 2024-02-20T02:20:27.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:27.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:27.841336+0000) 2024-02-20T02:20:28.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:28.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:28.841715+0000) 2024-02-20T02:20:29.837 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:29.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:29.841953+0000) 2024-02-20T02:20:30.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:30.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:30.842194+0000) 2024-02-20T02:20:31.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:31.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:31.842405+0000) 2024-02-20T02:20:32.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:32.836+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:32.842624+0000) 2024-02-20T02:20:33.838 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:33.840+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:33.842903+0000) 2024-02-20T02:20:34.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:34.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:34.843131+0000) 2024-02-20T02:20:35.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:35.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:35.843402+0000) 2024-02-20T02:20:36.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:36.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:36.843649+0000) 2024-02-20T02:20:37.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:37.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:37.843889+0000) 2024-02-20T02:20:38.839 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:38.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:38.844129+0000) 2024-02-20T02:20:39.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:39.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:39.844311+0000) 2024-02-20T02:20:40.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:40.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:40.844564+0000) 2024-02-20T02:20:41.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:41.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:41.844757+0000) 2024-02-20T02:20:42.840 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:42.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:42.845059+0000) 2024-02-20T02:20:43.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:43.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:43.845304+0000) 2024-02-20T02:20:44.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:44.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:44.845468+0000) 2024-02-20T02:20:45.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:45.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:45.845744+0000) 2024-02-20T02:20:46.841 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:46.839+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:46.845922+0000) 2024-02-20T02:20:47.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:47.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:47.846132+0000) 2024-02-20T02:20:48.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:48.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:48.846367+0000) 2024-02-20T02:20:49.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:49.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:49.846565+0000) 2024-02-20T02:20:50.842 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:50.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:50.846848+0000) 2024-02-20T02:20:51.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:51.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:51.847095+0000) 2024-02-20T02:20:52.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:52.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:52.847391+0000) 2024-02-20T02:20:53.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:53.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:53.847595+0000) 2024-02-20T02:20:54.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:54.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:54.847818+0000) 2024-02-20T02:20:55.843 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:55.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:55.848063+0000) 2024-02-20T02:20:56.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:56.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:56.848313+0000) 2024-02-20T02:20:57.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:57.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:57.848573+0000) 2024-02-20T02:20:58.844 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:58.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:58.848841+0000) 2024-02-20T02:20:59.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:20:59.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:20:59.849100+0000) 2024-02-20T02:21:00.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:00.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:00.849347+0000) 2024-02-20T02:21:01.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:01.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:01.849600+0000) 2024-02-20T02:21:02.845 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:02.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:02.849948+0000) 2024-02-20T02:21:03.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:03.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:03.850214+0000) 2024-02-20T02:21:04.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:04.843+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:04.850462+0000) 2024-02-20T02:21:05.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:05.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:05.850649+0000) 2024-02-20T02:21:06.846 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:06.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:06.850907+0000) 2024-02-20T02:21:07.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:07.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:07.851085+0000) 2024-02-20T02:21:08.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:08.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:08.851252+0000) 2024-02-20T02:21:09.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:09.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:09.851510+0000) 2024-02-20T02:21:10.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:10.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:10.851748+0000) 2024-02-20T02:21:11.847 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:11.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:11.851980+0000) 2024-02-20T02:21:12.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:12.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:12.852242+0000) 2024-02-20T02:21:13.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:13.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:13.852498+0000) 2024-02-20T02:21:14.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:14.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:14.852769+0000) 2024-02-20T02:21:15.848 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:15.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:15.853012+0000) 2024-02-20T02:21:16.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:16.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:16.853248+0000) 2024-02-20T02:21:17.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:17.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:17.853476+0000) 2024-02-20T02:21:18.849 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:18.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:18.853704+0000) 2024-02-20T02:21:19.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:19.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:19.853943+0000) 2024-02-20T02:21:20.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:20.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:20.854160+0000) 2024-02-20T02:21:21.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:21.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:21.854413+0000) 2024-02-20T02:21:22.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:22.847+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:22.854678+0000) 2024-02-20T02:21:23.850 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:23.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:23.854899+0000) 2024-02-20T02:21:24.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:24.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:24.855170+0000) 2024-02-20T02:21:25.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:25.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:25.855435+0000) 2024-02-20T02:21:26.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:26.851+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:26.855717+0000) 2024-02-20T02:21:27.851 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:27.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:27.855945+0000) 2024-02-20T02:21:28.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:28.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:28.856156+0000) 2024-02-20T02:21:29.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:29.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:29.856367+0000) 2024-02-20T02:21:30.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:30.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:30.856647+0000) 2024-02-20T02:21:31.852 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:31.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:31.856927+0000) 2024-02-20T02:21:32.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:32.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:32.857119+0000) 2024-02-20T02:21:33.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:33.855+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:33.857277+0000) 2024-02-20T02:21:34.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:34.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:34.857471+0000) 2024-02-20T02:21:35.853 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:35.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:35.857725+0000) 2024-02-20T02:21:36.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:36.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:36.857966+0000) 2024-02-20T02:21:37.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:37.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:37.858209+0000) 2024-02-20T02:21:38.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:38.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:38.858482+0000) 2024-02-20T02:21:39.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:39.850+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:39.858684+0000) 2024-02-20T02:21:40.854 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:40.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:40.858939+0000) 2024-02-20T02:21:41.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:41.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:41.859235+0000) 2024-02-20T02:21:42.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:42.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:42.859436+0000) 2024-02-20T02:21:43.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:43.854+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:43.859622+0000) 2024-02-20T02:21:44.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:44.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:44.859831+0000) 2024-02-20T02:21:45.855 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:45.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:45.860107+0000) 2024-02-20T02:21:46.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:46.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:46.860419+0000) 2024-02-20T02:21:47.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:47.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:47.860720+0000) 2024-02-20T02:21:48.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:48.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:48.860902+0000) 2024-02-20T02:21:49.856 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:49.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:49.861104+0000) 2024-02-20T02:21:50.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:50.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:50.861333+0000) 2024-02-20T02:21:51.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:51.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:51.861569+0000) 2024-02-20T02:21:52.857 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:52.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:52.861858+0000) 2024-02-20T02:21:53.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:53.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:53.862125+0000) 2024-02-20T02:21:54.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:54.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:54.862462+0000) 2024-02-20T02:21:55.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:55.858+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:55.862666+0000) 2024-02-20T02:21:56.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:56.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:56.862883+0000) 2024-02-20T02:21:57.858 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:57.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:57.863128+0000) 2024-02-20T02:21:58.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:58.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:58.863384+0000) 2024-02-20T02:21:59.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:21:59.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:21:59.863640+0000) 2024-02-20T02:22:00.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:00.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:00.863885+0000) 2024-02-20T02:22:01.859 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:01.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:01.864130+0000) 2024-02-20T02:22:02.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:02.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:02.864330+0000) 2024-02-20T02:22:03.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:03.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:03.864551+0000) 2024-02-20T02:22:04.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:04.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:04.864778+0000) 2024-02-20T02:22:05.860 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:05.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:05.865017+0000) 2024-02-20T02:22:06.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:06.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:06.865251+0000) 2024-02-20T02:22:07.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:07.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:07.865486+0000) 2024-02-20T02:22:08.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:08.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:08.865738+0000) 2024-02-20T02:22:09.861 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:09.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:09.865984+0000) 2024-02-20T02:22:10.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:10.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:10.866232+0000) 2024-02-20T02:22:11.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:11.862+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:11.866512+0000) 2024-02-20T02:22:12.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:12.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:12.866821+0000) 2024-02-20T02:22:13.862 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:13.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:13.867064+0000) 2024-02-20T02:22:14.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:14.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:14.867304+0000) 2024-02-20T02:22:15.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:15.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:15.867563+0000) 2024-02-20T02:22:16.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:16.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:16.867859+0000) 2024-02-20T02:22:17.863 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:17.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:17.868095+0000) 2024-02-20T02:22:18.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:18.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:18.868241+0000) 2024-02-20T02:22:19.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:19.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:19.868443+0000) 2024-02-20T02:22:20.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:20.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:20.868718+0000) 2024-02-20T02:22:21.864 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:21.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:21.868985+0000) 2024-02-20T02:22:22.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:22.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:22.869225+0000) 2024-02-20T02:22:23.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:23.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:23.869499+0000) 2024-02-20T02:22:24.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:24.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:24.869801+0000) 2024-02-20T02:22:25.865 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:25.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:25.870035+0000) 2024-02-20T02:22:26.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:26.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:26.870333+0000) 2024-02-20T02:22:27.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:27.866+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:27.870629+0000) 2024-02-20T02:22:28.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:28.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:28.870794+0000) 2024-02-20T02:22:29.866 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:29.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:29.871043+0000) 2024-02-20T02:22:30.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:30.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:30.871284+0000) 2024-02-20T02:22:31.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:31.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:31.871489+0000) 2024-02-20T02:22:32.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:32.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:32.871709+0000) 2024-02-20T02:22:33.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:33.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:33.871920+0000) 2024-02-20T02:22:34.867 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:34.870+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:34.872126+0000) 2024-02-20T02:22:35.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:35.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:35.872338+0000) 2024-02-20T02:22:36.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:36.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:36.872606+0000) 2024-02-20T02:22:37.868 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:37.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:37.872831+0000) 2024-02-20T02:22:38.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:38.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:38.873105+0000) 2024-02-20T02:22:39.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:39.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:39.873390+0000) 2024-02-20T02:22:40.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:40.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:40.873656+0000) 2024-02-20T02:22:41.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:41.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:41.873960+0000) 2024-02-20T02:22:42.869 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:42.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:42.874163+0000) 2024-02-20T02:22:43.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:43.869+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:43.874423+0000) 2024-02-20T02:22:44.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:44.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:44.874774+0000) 2024-02-20T02:22:45.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:45.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:45.874949+0000) 2024-02-20T02:22:46.870 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:46.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:46.875173+0000) 2024-02-20T02:22:47.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:47.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:47.875430+0000) 2024-02-20T02:22:48.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:48.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:48.875611+0000) 2024-02-20T02:22:49.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:49.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:49.875814+0000) 2024-02-20T02:22:50.871 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:50.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:50.876002+0000) 2024-02-20T02:22:51.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:51.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:51.876284+0000) 2024-02-20T02:22:52.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:52.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:52.876458+0000) 2024-02-20T02:22:53.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:53.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:53.876672+0000) 2024-02-20T02:22:54.872 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:54.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:54.876859+0000) 2024-02-20T02:22:55.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:55.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:55.877119+0000) 2024-02-20T02:22:56.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:56.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:56.877305+0000) 2024-02-20T02:22:57.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:57.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:57.877512+0000) 2024-02-20T02:22:58.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:58.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:58.877733+0000) 2024-02-20T02:22:59.873 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:22:59.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:22:59.877937+0000) 2024-02-20T02:23:00.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:00.873+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:00.878220+0000) 2024-02-20T02:23:01.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:01.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:01.878507+0000) 2024-02-20T02:23:02.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:02.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:02.878750+0000) 2024-02-20T02:23:03.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:03.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:03.878971+0000) 2024-02-20T02:23:04.874 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:04.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:04.879163+0000) 2024-02-20T02:23:05.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:05.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:05.879391+0000) 2024-02-20T02:23:06.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:06.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:06.879730+0000) 2024-02-20T02:23:07.875 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:07.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:07.879902+0000) 2024-02-20T02:23:08.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:08.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:08.880140+0000) 2024-02-20T02:23:09.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:09.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:09.880339+0000) 2024-02-20T02:23:10.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:10.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:10.880612+0000) 2024-02-20T02:23:11.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:11.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:11.880844+0000) 2024-02-20T02:23:12.876 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:12.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:12.881038+0000) 2024-02-20T02:23:13.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:13.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:13.881263+0000) 2024-02-20T02:23:14.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:14.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:14.881476+0000) 2024-02-20T02:23:15.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:15.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:15.881744+0000) 2024-02-20T02:23:16.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:16.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:16.881946+0000) 2024-02-20T02:23:17.877 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:17.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:17.882237+0000) 2024-02-20T02:23:18.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:18.877+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:18.882489+0000) 2024-02-20T02:23:19.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:19.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:19.882740+0000) 2024-02-20T02:23:20.878 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:20.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:20.882970+0000) 2024-02-20T02:23:21.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:21.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:21.883175+0000) 2024-02-20T02:23:22.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:22.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:22.883497+0000) 2024-02-20T02:23:23.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:23.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:23.883789+0000) 2024-02-20T02:23:24.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:24.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:24.884064+0000) 2024-02-20T02:23:25.879 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:25.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:25.884217+0000) 2024-02-20T02:23:26.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:26.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:26.884366+0000) 2024-02-20T02:23:27.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:27.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:27.884519+0000) 2024-02-20T02:23:28.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:28.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:28.884674+0000) 2024-02-20T02:23:29.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:29.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:29.884878+0000) 2024-02-20T02:23:30.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:30.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:30.885032+0000) 2024-02-20T02:23:31.880 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:31.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:31.885184+0000) 2024-02-20T02:23:32.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:32.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:32.885393+0000) 2024-02-20T02:23:33.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:33.885+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:33.885536+0000) 2024-02-20T02:23:34.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:34.881+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:34.885783+0000) 2024-02-20T02:23:35.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:35.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:35.885939+0000) 2024-02-20T02:23:36.881 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:36.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:36.886186+0000) 2024-02-20T02:23:37.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:37.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:37.886338+0000) 2024-02-20T02:23:38.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:38.880+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:38.886519+0000) 2024-02-20T02:23:39.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:39.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:39.886784+0000) 2024-02-20T02:23:40.882 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:40.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:40.887081+0000) 2024-02-20T02:23:41.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:41.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:41.887322+0000) 2024-02-20T02:23:42.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:42.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:42.887474+0000) 2024-02-20T02:23:43.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:43.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:43.887720+0000) 2024-02-20T02:23:44.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:44.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:44.887920+0000) 2024-02-20T02:23:45.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:45.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:45.888085+0000) 2024-02-20T02:23:46.883 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:46.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:46.888330+0000) 2024-02-20T02:23:47.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:47.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:47.888558+0000) 2024-02-20T02:23:48.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:48.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:48.888846+0000) 2024-02-20T02:23:49.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:49.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:49.888986+0000) 2024-02-20T02:23:50.884 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:50.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:50.889167+0000) 2024-02-20T02:23:51.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:51.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:51.889437+0000) 2024-02-20T02:23:52.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:52.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:52.889752+0000) 2024-02-20T02:23:53.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:53.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:53.889902+0000) 2024-02-20T02:23:54.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:54.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:54.890087+0000) 2024-02-20T02:23:55.885 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:55.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:55.890246+0000) 2024-02-20T02:23:56.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:56.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:56.890433+0000) 2024-02-20T02:23:57.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:57.884+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:57.890628+0000) 2024-02-20T02:23:58.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:58.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:58.890788+0000) 2024-02-20T02:23:59.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:23:59.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:23:59.891039+0000) 2024-02-20T02:24:00.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:00.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:00.891206+0000) 2024-02-20T02:24:01.886 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:01.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:01.891370+0000) 2024-02-20T02:24:02.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:02.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:02.891628+0000) 2024-02-20T02:24:03.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:03.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:03.891813+0000) 2024-02-20T02:24:04.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:04.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:04.891964+0000) 2024-02-20T02:24:05.887 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:05.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:05.892182+0000) 2024-02-20T02:24:06.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:06.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:06.892418+0000) 2024-02-20T02:24:07.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:07.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:07.892673+0000) 2024-02-20T02:24:08.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:08.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:08.892881+0000) 2024-02-20T02:24:09.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:09.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:09.893122+0000) 2024-02-20T02:24:10.888 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:10.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:10.893314+0000) 2024-02-20T02:24:11.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:11.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:11.893562+0000) 2024-02-20T02:24:12.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:12.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:12.893825+0000) 2024-02-20T02:24:13.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:13.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:13.894004+0000) 2024-02-20T02:24:14.889 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:14.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:14.894232+0000) 2024-02-20T02:24:15.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:15.888+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:15.894455+0000) 2024-02-20T02:24:16.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:16.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:16.894639+0000) 2024-02-20T02:24:17.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:17.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:17.894823+0000) 2024-02-20T02:24:18.890 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:18.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:18.895100+0000) 2024-02-20T02:24:19.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:19.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:19.895325+0000) 2024-02-20T02:24:20.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:20.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:20.895602+0000) 2024-02-20T02:24:21.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:21.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:21.895775+0000) 2024-02-20T02:24:22.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:22.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:22.895931+0000) 2024-02-20T02:24:23.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:23.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:23.896112+0000) 2024-02-20T02:24:24.891 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:24.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:24.896301+0000) 2024-02-20T02:24:25.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:25.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:25.896449+0000) 2024-02-20T02:24:26.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:26.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:26.896678+0000) 2024-02-20T02:24:27.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:27.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:27.896935+0000) 2024-02-20T02:24:28.892 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:28.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:28.897192+0000) 2024-02-20T02:24:29.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:29.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:29.897409+0000) 2024-02-20T02:24:30.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:30.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:30.897661+0000) 2024-02-20T02:24:31.893 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:31.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:31.897930+0000) 2024-02-20T02:24:32.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:32.892+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:32.898161+0000) 2024-02-20T02:24:33.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:33.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:33.898391+0000) 2024-02-20T02:24:34.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:34.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:34.898660+0000) 2024-02-20T02:24:35.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:35.896+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:35.898903+0000) 2024-02-20T02:24:36.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:36.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:36.899079+0000) 2024-02-20T02:24:37.894 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:37.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:37.899352+0000) 2024-02-20T02:24:38.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:38.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:38.899524+0000) 2024-02-20T02:24:39.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:39.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:39.899760+0000) 2024-02-20T02:24:40.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:40.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:40.899950+0000) 2024-02-20T02:24:41.895 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:41.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:41.900138+0000) 2024-02-20T02:24:42.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:42.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:42.900408+0000) 2024-02-20T02:24:43.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:43.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:43.900549+0000) 2024-02-20T02:24:44.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:44.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:44.900735+0000) 2024-02-20T02:24:45.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:45.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:45.900878+0000) 2024-02-20T02:24:46.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:46.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:46.901010+0000) 2024-02-20T02:24:47.896 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:47.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:47.901157+0000) 2024-02-20T02:24:48.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:48.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:48.901431+0000) 2024-02-20T02:24:49.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:49.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:49.901665+0000) 2024-02-20T02:24:50.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:50.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:50.901830+0000) 2024-02-20T02:24:51.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:51.895+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:51.901959+0000) 2024-02-20T02:24:52.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:52.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:52.902110+0000) 2024-02-20T02:24:53.897 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:53.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:53.902300+0000) 2024-02-20T02:24:54.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:54.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:54.902509+0000) 2024-02-20T02:24:55.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:55.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:55.902780+0000) 2024-02-20T02:24:56.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:56.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:56.903040+0000) 2024-02-20T02:24:57.898 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:57.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:57.903260+0000) 2024-02-20T02:24:58.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:58.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:58.903417+0000) 2024-02-20T02:24:59.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:24:59.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:24:59.903650+0000) 2024-02-20T02:25:00.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:00.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:00.903831+0000) 2024-02-20T02:25:01.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:01.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:01.903991+0000) 2024-02-20T02:25:02.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:02.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:02.904210+0000) 2024-02-20T02:25:03.899 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:03.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:03.904390+0000) 2024-02-20T02:25:04.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:04.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:04.904624+0000) 2024-02-20T02:25:05.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:05.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:05.904793+0000) 2024-02-20T02:25:06.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:06.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:06.904925+0000) 2024-02-20T02:25:07.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:07.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:07.905081+0000) 2024-02-20T02:25:08.900 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:08.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:08.905261+0000) 2024-02-20T02:25:09.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:09.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:09.905444+0000) 2024-02-20T02:25:10.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:10.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:10.905705+0000) 2024-02-20T02:25:11.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:11.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:11.905894+0000) 2024-02-20T02:25:12.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:12.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:12.906090+0000) 2024-02-20T02:25:13.901 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:13.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:13.906338+0000) 2024-02-20T02:25:14.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:14.899+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:14.906575+0000) 2024-02-20T02:25:15.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:15.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:15.906749+0000) 2024-02-20T02:25:16.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:16.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:16.906912+0000) 2024-02-20T02:25:17.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:17.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:17.907053+0000) 2024-02-20T02:25:18.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:18.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:18.907216+0000) 2024-02-20T02:25:19.902 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:19.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:19.907388+0000) 2024-02-20T02:25:20.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:20.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:20.907616+0000) 2024-02-20T02:25:21.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:21.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:21.907786+0000) 2024-02-20T02:25:22.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:22.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:22.907937+0000) 2024-02-20T02:25:23.903 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:23.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:23.908165+0000) 2024-02-20T02:25:24.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:24.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:24.908403+0000) 2024-02-20T02:25:25.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:25.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:25.908655+0000) 2024-02-20T02:25:26.904 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:26.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:26.908903+0000) 2024-02-20T02:25:27.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:27.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:27.909149+0000) 2024-02-20T02:25:28.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:28.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:28.909401+0000) 2024-02-20T02:25:29.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:29.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:29.909667+0000) 2024-02-20T02:25:30.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:30.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:30.909837+0000) 2024-02-20T02:25:31.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:31.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:31.910046+0000) 2024-02-20T02:25:32.905 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:32.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:32.910253+0000) 2024-02-20T02:25:33.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:33.903+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:33.910415+0000) 2024-02-20T02:25:34.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:34.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:34.910639+0000) 2024-02-20T02:25:35.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:35.907+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:35.910818+0000) 2024-02-20T02:25:36.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:36.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:36.910963+0000) 2024-02-20T02:25:37.906 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:37.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:37.911190+0000) 2024-02-20T02:25:38.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:38.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:38.911426+0000) 2024-02-20T02:25:39.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:39.910+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:39.911648+0000) 2024-02-20T02:25:40.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:40.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:40.911899+0000) 2024-02-20T02:25:41.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:41.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:41.912076+0000) 2024-02-20T02:25:42.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:42.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:42.912230+0000) 2024-02-20T02:25:43.907 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:43.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:43.912382+0000) 2024-02-20T02:25:44.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:44.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:44.912590+0000) 2024-02-20T02:25:45.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:45.910+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:45.912840+0000) 2024-02-20T02:25:46.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:46.910+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:46.913011+0000) 2024-02-20T02:25:47.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:47.910+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:47.913162+0000) 2024-02-20T02:25:48.908 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:48.906+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:48.913332+0000) 2024-02-20T02:25:49.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:49.910+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:49.913521+0000) 2024-02-20T02:25:50.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:50.910+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:50.913821+0000) 2024-02-20T02:25:51.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:51.910+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:51.913958+0000) 2024-02-20T02:25:52.909 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:52.910+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:52.914143+0000) 2024-02-20T02:25:53.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:53.910+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:53.914340+0000) 2024-02-20T02:25:54.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:54.910+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:54.914595+0000) 2024-02-20T02:25:55.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:55.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:55.914854+0000) 2024-02-20T02:25:56.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:56.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:56.915088+0000) 2024-02-20T02:25:57.910 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:57.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:57.915301+0000) 2024-02-20T02:25:58.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:58.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:58.915558+0000) 2024-02-20T02:25:59.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:25:59.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:25:59.915770+0000) 2024-02-20T02:26:00.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:00.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:00.915916+0000) 2024-02-20T02:26:01.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:01.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:01.916102+0000) 2024-02-20T02:26:02.911 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:02.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:02.916400+0000) 2024-02-20T02:26:03.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:03.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:03.916678+0000) 2024-02-20T02:26:04.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:04.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:04.916848+0000) 2024-02-20T02:26:05.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:05.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:05.917007+0000) 2024-02-20T02:26:06.912 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:06.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:06.917251+0000) 2024-02-20T02:26:07.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:07.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:07.917424+0000) 2024-02-20T02:26:08.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:08.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:08.917582+0000) 2024-02-20T02:26:09.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:09.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:09.917730+0000) 2024-02-20T02:26:10.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:10.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:10.917954+0000) 2024-02-20T02:26:11.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:11.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:11.918175+0000) 2024-02-20T02:26:12.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:12.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:12.918330+0000) 2024-02-20T02:26:13.913 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:13.914+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:13.918494+0000) 2024-02-20T02:26:14.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:14.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:14.918685+0000) 2024-02-20T02:26:15.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:15.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:15.918932+0000) 2024-02-20T02:26:16.914 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:16.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:16.919103+0000) 2024-02-20T02:26:17.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:17.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:17.919308+0000) 2024-02-20T02:26:18.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:18.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:18.919570+0000) 2024-02-20T02:26:19.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:19.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:19.919794+0000) 2024-02-20T02:26:20.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:20.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:20.919979+0000) 2024-02-20T02:26:21.915 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:21.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:21.920212+0000) 2024-02-20T02:26:22.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:22.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:22.920459+0000) 2024-02-20T02:26:23.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:23.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:23.920695+0000) 2024-02-20T02:26:24.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:24.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:24.920947+0000) 2024-02-20T02:26:25.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:25.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:25.921111+0000) 2024-02-20T02:26:26.916 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:26.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:26.921271+0000) 2024-02-20T02:26:27.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:27.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:27.921542+0000) 2024-02-20T02:26:28.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:28.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:28.921812+0000) 2024-02-20T02:26:29.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:29.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:29.922050+0000) 2024-02-20T02:26:30.917 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:30.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:30.922257+0000) 2024-02-20T02:26:31.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:31.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:31.922456+0000) 2024-02-20T02:26:32.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:32.918+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:32.922620+0000) 2024-02-20T02:26:33.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:33.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:33.922760+0000) 2024-02-20T02:26:34.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:34.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:34.923037+0000) 2024-02-20T02:26:35.918 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:35.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:35.923258+0000) 2024-02-20T02:26:36.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:36.922+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:36.923394+0000) 2024-02-20T02:26:37.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:37.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:37.923541+0000) 2024-02-20T02:26:38.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:38.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:38.923771+0000) 2024-02-20T02:26:39.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:39.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:39.923939+0000) 2024-02-20T02:26:40.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:40.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:40.924083+0000) 2024-02-20T02:26:41.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:41.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:41.924255+0000) 2024-02-20T02:26:42.919 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:42.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:42.924499+0000) 2024-02-20T02:26:43.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:43.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:43.924747+0000) 2024-02-20T02:26:44.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:44.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:44.925084+0000) 2024-02-20T02:26:45.920 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:45.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:45.925284+0000) 2024-02-20T02:26:46.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:46.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:46.925513+0000) 2024-02-20T02:26:47.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:47.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:47.925667+0000) 2024-02-20T02:26:48.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:48.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:48.925918+0000) 2024-02-20T02:26:49.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:49.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:49.926148+0000) 2024-02-20T02:26:50.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:50.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:50.926285+0000) 2024-02-20T02:26:51.921 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:51.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:51.926520+0000) 2024-02-20T02:26:52.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:52.921+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:52.926684+0000) 2024-02-20T02:26:53.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:53.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:53.926903+0000) 2024-02-20T02:26:54.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:54.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:54.927078+0000) 2024-02-20T02:26:55.922 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:55.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:55.927250+0000) 2024-02-20T02:26:56.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:56.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:56.927506+0000) 2024-02-20T02:26:57.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:57.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:57.927743+0000) 2024-02-20T02:26:58.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:58.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:58.927900+0000) 2024-02-20T02:26:59.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:26:59.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:26:59.928035+0000) 2024-02-20T02:27:00.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:00.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:00.928177+0000) 2024-02-20T02:27:01.923 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:01.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:01.928359+0000) 2024-02-20T02:27:02.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:02.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:02.928528+0000) 2024-02-20T02:27:03.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:03.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:03.928850+0000) 2024-02-20T02:27:04.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:04.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:04.929034+0000) 2024-02-20T02:27:05.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:05.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:05.929227+0000) 2024-02-20T02:27:06.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:06.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:06.929373+0000) 2024-02-20T02:27:07.924 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:07.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:07.929544+0000) 2024-02-20T02:27:08.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:08.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:08.929730+0000) 2024-02-20T02:27:09.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:09.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:09.929963+0000) 2024-02-20T02:27:10.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:10.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:10.930113+0000) 2024-02-20T02:27:11.925 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:11.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:11.930266+0000) 2024-02-20T02:27:12.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:12.925+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:12.930490+0000) 2024-02-20T02:27:13.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:13.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:13.930675+0000) 2024-02-20T02:27:14.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:14.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:14.930924+0000) 2024-02-20T02:27:15.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:15.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:15.931197+0000) 2024-02-20T02:27:16.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:16.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:16.931460+0000) 2024-02-20T02:27:17.926 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:17.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:17.931630+0000) 2024-02-20T02:27:18.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:18.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:18.931858+0000) 2024-02-20T02:27:19.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:19.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:19.932116+0000) 2024-02-20T02:27:20.927 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:20.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:20.932372+0000) 2024-02-20T02:27:21.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:21.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:21.932622+0000) 2024-02-20T02:27:22.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:22.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:22.932894+0000) 2024-02-20T02:27:23.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:23.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:23.933065+0000) 2024-02-20T02:27:24.928 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:24.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:24.933291+0000) 2024-02-20T02:27:25.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:25.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:25.933551+0000) 2024-02-20T02:27:26.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:26.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:26.933747+0000) 2024-02-20T02:27:27.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:27.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:27.933962+0000) 2024-02-20T02:27:28.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:28.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:28.934102+0000) 2024-02-20T02:27:29.929 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:29.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:29.934325+0000) 2024-02-20T02:27:30.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:30.929+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:30.934586+0000) 2024-02-20T02:27:31.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:31.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:31.934784+0000) 2024-02-20T02:27:32.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:32.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:32.934946+0000) 2024-02-20T02:27:33.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:33.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:33.935085+0000) 2024-02-20T02:27:34.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:34.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:34.935278+0000) 2024-02-20T02:27:35.930 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:35.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:35.935526+0000) 2024-02-20T02:27:36.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:36.933+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:36.935722+0000) 2024-02-20T02:27:37.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:37.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:37.936001+0000) 2024-02-20T02:27:38.931 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:38.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:38.936216+0000) 2024-02-20T02:27:39.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:39.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:39.936457+0000) 2024-02-20T02:27:40.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:40.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:40.936638+0000) 2024-02-20T02:27:41.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:41.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:41.936864+0000) 2024-02-20T02:27:42.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:42.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:42.937047+0000) 2024-02-20T02:27:43.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:43.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:43.937198+0000) 2024-02-20T02:27:44.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:44.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:44.937425+0000) 2024-02-20T02:27:45.932 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:45.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:45.937583+0000) 2024-02-20T02:27:46.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:46.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:46.937731+0000) 2024-02-20T02:27:47.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:47.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:47.937913+0000) 2024-02-20T02:27:48.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:48.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:48.938143+0000) 2024-02-20T02:27:49.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:49.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:49.938289+0000) 2024-02-20T02:27:50.933 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:50.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:50.938430+0000) 2024-02-20T02:27:51.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:51.932+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:51.938585+0000) 2024-02-20T02:27:52.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:52.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:52.938754+0000) 2024-02-20T02:27:53.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:53.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:53.938950+0000) 2024-02-20T02:27:54.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:54.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:54.939094+0000) 2024-02-20T02:27:55.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:55.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:55.939239+0000) 2024-02-20T02:27:56.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:56.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:56.939393+0000) 2024-02-20T02:27:57.934 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:57.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:57.939559+0000) 2024-02-20T02:27:58.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:58.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:58.939733+0000) 2024-02-20T02:27:59.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:27:59.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:27:59.939877+0000) 2024-02-20T02:28:00.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:00.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:00.940100+0000) 2024-02-20T02:28:01.935 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:01.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:01.940305+0000) 2024-02-20T02:28:02.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:02.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:02.940499+0000) 2024-02-20T02:28:03.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:03.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:03.940736+0000) 2024-02-20T02:28:04.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:04.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:04.940964+0000) 2024-02-20T02:28:05.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:05.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:05.941188+0000) 2024-02-20T02:28:06.936 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:06.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:06.941427+0000) 2024-02-20T02:28:07.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:07.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:07.941602+0000) 2024-02-20T02:28:08.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:08.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:08.941819+0000) 2024-02-20T02:28:09.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:09.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:09.941991+0000) 2024-02-20T02:28:10.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:10.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:10.942167+0000) 2024-02-20T02:28:11.937 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:11.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:11.942324+0000) 2024-02-20T02:28:12.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:12.936+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:12.942604+0000) 2024-02-20T02:28:13.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:13.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:13.942764+0000) 2024-02-20T02:28:14.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:14.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:14.943005+0000) 2024-02-20T02:28:15.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:15.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:15.943184+0000) 2024-02-20T02:28:16.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:16.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:16.943336+0000) 2024-02-20T02:28:17.938 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:17.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:17.943559+0000) 2024-02-20T02:28:18.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:18.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:18.943776+0000) 2024-02-20T02:28:19.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:19.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:19.943928+0000) 2024-02-20T02:28:20.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:20.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:20.944081+0000) 2024-02-20T02:28:21.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:21.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:21.944332+0000) 2024-02-20T02:28:22.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:22.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:22.944489+0000) 2024-02-20T02:28:23.939 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:23.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:23.944617+0000) 2024-02-20T02:28:24.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:24.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:24.944772+0000) 2024-02-20T02:28:25.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:25.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:25.944939+0000) 2024-02-20T02:28:26.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:26.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:26.945209+0000) 2024-02-20T02:28:27.940 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:27.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:27.945420+0000) 2024-02-20T02:28:28.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:28.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:28.945589+0000) 2024-02-20T02:28:29.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:29.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:29.945903+0000) 2024-02-20T02:28:30.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:30.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:30.946095+0000) 2024-02-20T02:28:31.941 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:31.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:31.946253+0000) 2024-02-20T02:28:32.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:32.940+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:32.946500+0000) 2024-02-20T02:28:33.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:33.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:33.946675+0000) 2024-02-20T02:28:34.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:34.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:34.946946+0000) 2024-02-20T02:28:35.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:35.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:35.947220+0000) 2024-02-20T02:28:36.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:36.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:36.947529+0000) 2024-02-20T02:28:37.942 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:37.944+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:37.947703+0000) 2024-02-20T02:28:38.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:38.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:38.948027+0000) 2024-02-20T02:28:39.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:39.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:39.948236+0000) 2024-02-20T02:28:40.943 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:40.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:40.948399+0000) 2024-02-20T02:28:41.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:41.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:41.948555+0000) 2024-02-20T02:28:42.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:42.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:42.948727+0000) 2024-02-20T02:28:43.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:43.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:43.948875+0000) 2024-02-20T02:28:44.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:44.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:44.949036+0000) 2024-02-20T02:28:45.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:45.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:45.949171+0000) 2024-02-20T02:28:46.944 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:46.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:46.949318+0000) 2024-02-20T02:28:47.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:47.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:47.949531+0000) 2024-02-20T02:28:48.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:48.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:48.949717+0000) 2024-02-20T02:28:49.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:49.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:49.949868+0000) 2024-02-20T02:28:50.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:50.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:50.950055+0000) 2024-02-20T02:28:51.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:51.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:51.950302+0000) 2024-02-20T02:28:52.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:52.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:52.950501+0000) 2024-02-20T02:28:53.945 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:53.943+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:53.950696+0000) 2024-02-20T02:28:54.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:54.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:54.950879+0000) 2024-02-20T02:28:55.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:55.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:55.951019+0000) 2024-02-20T02:28:56.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:56.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:56.951223+0000) 2024-02-20T02:28:57.946 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:57.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:57.951470+0000) 2024-02-20T02:28:58.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:58.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:58.951710+0000) 2024-02-20T02:28:59.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:28:59.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:28:59.951900+0000) 2024-02-20T02:29:00.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:00.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:00.952145+0000) 2024-02-20T02:29:01.947 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:01.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:01.952403+0000) 2024-02-20T02:29:02.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:02.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:02.952641+0000) 2024-02-20T02:29:03.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:03.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:03.952942+0000) 2024-02-20T02:29:04.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:04.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:04.953209+0000) 2024-02-20T02:29:05.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:05.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:05.953374+0000) 2024-02-20T02:29:06.948 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:06.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:06.953645+0000) 2024-02-20T02:29:07.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:07.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:07.953917+0000) 2024-02-20T02:29:08.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:08.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:08.954127+0000) 2024-02-20T02:29:09.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:09.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:09.954320+0000) 2024-02-20T02:29:10.949 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:10.947+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:10.954546+0000) 2024-02-20T02:29:11.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:11.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:11.954758+0000) 2024-02-20T02:29:12.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:12.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:12.954912+0000) 2024-02-20T02:29:13.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:13.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:13.955067+0000) 2024-02-20T02:29:14.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:14.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:14.955269+0000) 2024-02-20T02:29:15.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:15.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:15.955411+0000) 2024-02-20T02:29:16.950 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:16.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:16.955550+0000) 2024-02-20T02:29:17.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:17.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:17.955691+0000) 2024-02-20T02:29:18.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:18.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:18.955869+0000) 2024-02-20T02:29:19.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:19.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:19.956105+0000) 2024-02-20T02:29:20.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:20.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:20.956264+0000) 2024-02-20T02:29:21.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:21.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:21.956415+0000) 2024-02-20T02:29:22.951 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:22.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:22.956593+0000) 2024-02-20T02:29:23.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:23.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:23.956816+0000) 2024-02-20T02:29:24.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:24.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:24.957056+0000) 2024-02-20T02:29:25.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:25.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:25.957307+0000) 2024-02-20T02:29:26.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:26.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:26.957492+0000) 2024-02-20T02:29:27.952 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:27.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:27.957731+0000) 2024-02-20T02:29:28.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:28.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:28.957976+0000) 2024-02-20T02:29:29.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:29.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:29.958120+0000) 2024-02-20T02:29:30.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:30.951+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:30.958303+0000) 2024-02-20T02:29:31.953 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:31.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:31.958504+0000) 2024-02-20T02:29:32.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:32.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:32.958819+0000) 2024-02-20T02:29:33.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:33.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:33.959100+0000) 2024-02-20T02:29:34.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:34.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:34.959239+0000) 2024-02-20T02:29:35.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:35.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:35.959366+0000) 2024-02-20T02:29:36.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:36.959+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:36.959516+0000) 2024-02-20T02:29:37.954 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:37.955+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:37.959713+0000) 2024-02-20T02:29:38.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:38.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:38.959893+0000) 2024-02-20T02:29:39.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:39.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:39.960050+0000) 2024-02-20T02:29:40.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:40.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:40.960333+0000) 2024-02-20T02:29:41.955 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:41.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:41.960481+0000) 2024-02-20T02:29:42.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:42.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:42.960744+0000) 2024-02-20T02:29:43.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:43.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:43.960978+0000) 2024-02-20T02:29:44.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:44.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:44.961124+0000) 2024-02-20T02:29:45.956 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:45.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:45.961378+0000) 2024-02-20T02:29:46.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:46.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:46.961674+0000) 2024-02-20T02:29:47.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:47.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:47.961856+0000) 2024-02-20T02:29:48.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:48.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:48.962117+0000) 2024-02-20T02:29:49.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:49.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:49.962336+0000) 2024-02-20T02:29:50.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:50.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:50.962509+0000) 2024-02-20T02:29:51.957 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:51.954+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:51.962662+0000) 2024-02-20T02:29:52.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:52.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:52.962838+0000) 2024-02-20T02:29:53.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:53.958+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:53.962995+0000) 2024-02-20T02:29:54.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:54.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:54.963174+0000) 2024-02-20T02:29:55.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:55.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:55.963384+0000) 2024-02-20T02:29:56.958 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:56.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:56.963595+0000) 2024-02-20T02:29:57.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:57.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:57.963807+0000) 2024-02-20T02:29:58.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:58.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:58.963956+0000) 2024-02-20T02:29:59.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:29:59.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:29:59.964095+0000) 2024-02-20T02:30:00.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:00.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:00.964250+0000) 2024-02-20T02:30:01.959 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:01.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:01.964488+0000) 2024-02-20T02:30:02.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:02.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:02.964676+0000) 2024-02-20T02:30:03.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:03.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:03.964907+0000) 2024-02-20T02:30:04.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:04.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:04.965072+0000) 2024-02-20T02:30:05.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:05.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:05.965242+0000) 2024-02-20T02:30:06.960 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:06.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:06.965523+0000) 2024-02-20T02:30:07.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:07.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:07.965731+0000) 2024-02-20T02:30:08.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:08.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:08.965917+0000) 2024-02-20T02:30:09.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:09.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:09.966068+0000) 2024-02-20T02:30:10.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:10.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:10.966299+0000) 2024-02-20T02:30:11.961 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:11.962+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:11.966538+0000) 2024-02-20T02:30:12.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:12.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:12.966936+0000) 2024-02-20T02:30:13.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:13.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:13.967086+0000) 2024-02-20T02:30:14.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:14.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:14.967242+0000) 2024-02-20T02:30:15.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:15.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:15.967424+0000) 2024-02-20T02:30:16.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:16.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:16.967662+0000) 2024-02-20T02:30:17.962 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:17.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:17.967848+0000) 2024-02-20T02:30:18.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:18.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:18.967996+0000) 2024-02-20T02:30:19.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:19.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:19.968166+0000) 2024-02-20T02:30:20.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:20.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:20.968310+0000) 2024-02-20T02:30:21.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:21.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:21.968456+0000) 2024-02-20T02:30:22.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:22.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:22.968594+0000) 2024-02-20T02:30:23.963 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:23.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:23.968723+0000) 2024-02-20T02:30:24.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:24.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:24.968875+0000) 2024-02-20T02:30:25.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:25.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:25.969051+0000) 2024-02-20T02:30:26.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:26.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:26.969277+0000) 2024-02-20T02:30:27.964 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:27.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:27.969580+0000) 2024-02-20T02:30:28.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:28.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:28.969837+0000) 2024-02-20T02:30:29.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:29.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:29.970022+0000) 2024-02-20T02:30:30.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:30.966+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:30.970308+0000) 2024-02-20T02:30:31.965 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:31.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:31.970501+0000) 2024-02-20T02:30:32.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:32.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:32.970782+0000) 2024-02-20T02:30:33.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:33.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:33.971021+0000) 2024-02-20T02:30:34.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:34.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:34.971302+0000) 2024-02-20T02:30:35.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:35.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:35.971462+0000) 2024-02-20T02:30:36.966 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:36.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:36.971652+0000) 2024-02-20T02:30:37.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:37.970+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:37.971893+0000) 2024-02-20T02:30:38.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:38.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:38.972015+0000) 2024-02-20T02:30:39.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:39.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:39.972258+0000) 2024-02-20T02:30:40.967 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:40.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:40.972533+0000) 2024-02-20T02:30:41.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:41.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:41.972717+0000) 2024-02-20T02:30:42.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:42.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:42.972892+0000) 2024-02-20T02:30:43.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:43.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:43.973022+0000) 2024-02-20T02:30:44.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:44.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:44.973220+0000) 2024-02-20T02:30:45.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:45.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:45.973421+0000) 2024-02-20T02:30:46.968 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:46.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:46.973635+0000) 2024-02-20T02:30:47.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:47.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:47.973854+0000) 2024-02-20T02:30:48.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:48.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:48.973993+0000) 2024-02-20T02:30:49.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:49.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:49.974147+0000) 2024-02-20T02:30:50.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:50.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:50.974401+0000) 2024-02-20T02:30:51.969 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:51.969+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:51.974558+0000) 2024-02-20T02:30:52.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:52.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:52.974770+0000) 2024-02-20T02:30:53.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:53.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:53.974931+0000) 2024-02-20T02:30:54.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:54.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:54.975121+0000) 2024-02-20T02:30:55.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:55.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:55.975354+0000) 2024-02-20T02:30:56.970 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:56.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:56.975612+0000) 2024-02-20T02:30:57.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:57.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:57.975903+0000) 2024-02-20T02:30:58.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:58.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:58.976090+0000) 2024-02-20T02:30:59.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:30:59.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:30:59.976312+0000) 2024-02-20T02:31:00.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:00.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:00.976467+0000) 2024-02-20T02:31:01.971 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:01.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:01.976620+0000) 2024-02-20T02:31:02.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:02.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:02.976884+0000) 2024-02-20T02:31:03.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:03.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:03.977035+0000) 2024-02-20T02:31:04.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:04.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:04.977247+0000) 2024-02-20T02:31:05.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:05.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:05.977492+0000) 2024-02-20T02:31:06.972 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:06.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:06.977634+0000) 2024-02-20T02:31:07.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:07.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:07.977820+0000) 2024-02-20T02:31:08.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:08.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:08.978042+0000) 2024-02-20T02:31:09.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:09.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:09.978202+0000) 2024-02-20T02:31:10.973 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:10.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:10.978423+0000) 2024-02-20T02:31:11.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:11.973+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:11.978604+0000) 2024-02-20T02:31:12.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:12.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:12.978861+0000) 2024-02-20T02:31:13.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:13.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:13.979084+0000) 2024-02-20T02:31:14.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:14.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:14.979223+0000) 2024-02-20T02:31:15.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:15.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:15.979437+0000) 2024-02-20T02:31:16.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:16.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:16.979607+0000) 2024-02-20T02:31:17.974 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:17.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:17.979760+0000) 2024-02-20T02:31:18.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:18.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:18.979902+0000) 2024-02-20T02:31:19.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:19.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:19.980049+0000) 2024-02-20T02:31:20.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:20.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:20.980251+0000) 2024-02-20T02:31:21.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:21.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:21.980393+0000) 2024-02-20T02:31:22.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:22.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:22.980620+0000) 2024-02-20T02:31:23.975 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:23.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:23.980766+0000) 2024-02-20T02:31:24.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:24.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:24.980909+0000) 2024-02-20T02:31:25.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:25.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:25.981040+0000) 2024-02-20T02:31:26.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:26.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:26.981215+0000) 2024-02-20T02:31:27.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:27.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:27.981504+0000) 2024-02-20T02:31:28.976 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:28.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:28.981692+0000) 2024-02-20T02:31:29.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:29.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:29.981849+0000) 2024-02-20T02:31:30.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:30.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:30.982053+0000) 2024-02-20T02:31:31.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:31.977+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:31.982230+0000) 2024-02-20T02:31:32.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:32.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:32.982515+0000) 2024-02-20T02:31:33.977 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:33.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:33.982822+0000) 2024-02-20T02:31:34.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:34.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:34.983058+0000) 2024-02-20T02:31:35.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:35.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:35.983276+0000) 2024-02-20T02:31:36.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:36.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:36.983425+0000) 2024-02-20T02:31:37.978 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:37.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:37.983559+0000) 2024-02-20T02:31:38.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:38.981+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:38.983802+0000) 2024-02-20T02:31:39.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:39.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:39.983966+0000) 2024-02-20T02:31:40.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:40.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:40.984106+0000) 2024-02-20T02:31:41.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:41.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:41.984273+0000) 2024-02-20T02:31:42.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:42.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:42.984427+0000) 2024-02-20T02:31:43.979 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:43.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:43.984641+0000) 2024-02-20T02:31:44.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:44.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:44.984796+0000) 2024-02-20T02:31:45.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:45.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:45.984982+0000) 2024-02-20T02:31:46.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:46.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:46.985223+0000) 2024-02-20T02:31:47.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:47.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:47.985487+0000) 2024-02-20T02:31:48.980 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:48.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:48.985736+0000) 2024-02-20T02:31:49.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:49.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:49.985910+0000) 2024-02-20T02:31:50.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:50.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:50.986086+0000) 2024-02-20T02:31:51.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:51.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:51.986277+0000) 2024-02-20T02:31:52.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:52.980+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:52.986473+0000) 2024-02-20T02:31:53.981 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:53.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:53.986777+0000) 2024-02-20T02:31:54.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:54.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:54.986934+0000) 2024-02-20T02:31:55.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:55.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:55.987110+0000) 2024-02-20T02:31:56.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:56.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:56.987261+0000) 2024-02-20T02:31:57.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:57.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:57.987526+0000) 2024-02-20T02:31:58.982 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:58.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:58.987830+0000) 2024-02-20T02:31:59.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:31:59.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:31:59.988018+0000) 2024-02-20T02:32:00.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:00.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:00.988162+0000) 2024-02-20T02:32:01.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:01.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:01.988399+0000) 2024-02-20T02:32:02.983 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:02.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:02.988570+0000) 2024-02-20T02:32:03.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:03.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:03.988828+0000) 2024-02-20T02:32:04.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:04.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:04.989002+0000) 2024-02-20T02:32:05.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:05.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:05.989133+0000) 2024-02-20T02:32:06.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:06.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:06.989267+0000) 2024-02-20T02:32:07.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:07.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:07.989464+0000) 2024-02-20T02:32:08.984 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:08.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:08.989654+0000) 2024-02-20T02:32:09.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:09.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:09.989875+0000) 2024-02-20T02:32:10.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:10.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:10.990086+0000) 2024-02-20T02:32:11.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:11.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:11.990223+0000) 2024-02-20T02:32:12.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:12.984+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:12.990375+0000) 2024-02-20T02:32:13.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:13.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:13.990515+0000) 2024-02-20T02:32:14.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:14.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:14.990661+0000) 2024-02-20T02:32:15.985 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:15.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:15.990787+0000) 2024-02-20T02:32:16.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:16.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:16.991020+0000) 2024-02-20T02:32:17.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:17.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:17.991246+0000) 2024-02-20T02:32:18.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:18.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:18.991424+0000) 2024-02-20T02:32:19.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:19.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:19.991574+0000) 2024-02-20T02:32:20.986 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:20.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:20.991865+0000) 2024-02-20T02:32:21.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:21.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:21.992074+0000) 2024-02-20T02:32:22.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:22.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:22.992338+0000) 2024-02-20T02:32:23.987 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:23.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:23.992521+0000) 2024-02-20T02:32:24.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:24.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:24.992783+0000) 2024-02-20T02:32:25.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:25.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:25.992948+0000) 2024-02-20T02:32:26.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:26.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:26.993094+0000) 2024-02-20T02:32:27.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:27.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:27.993327+0000) 2024-02-20T02:32:28.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:28.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:28.993527+0000) 2024-02-20T02:32:29.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:29.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:29.993671+0000) 2024-02-20T02:32:30.988 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:30.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:30.993814+0000) 2024-02-20T02:32:31.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:31.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:31.994037+0000) 2024-02-20T02:32:32.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:32.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:32.994262+0000) 2024-02-20T02:32:33.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:33.988+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:33.994491+0000) 2024-02-20T02:32:34.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:34.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:34.994628+0000) 2024-02-20T02:32:35.989 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:35.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:35.994834+0000) 2024-02-20T02:32:36.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:36.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:36.995040+0000) 2024-02-20T02:32:37.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:37.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:37.995188+0000) 2024-02-20T02:32:38.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:38.992+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:38.995331+0000) 2024-02-20T02:32:39.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:39.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:39.995477+0000) 2024-02-20T02:32:40.990 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:40.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:40.995622+0000) 2024-02-20T02:32:41.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:41.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:41.995851+0000) 2024-02-20T02:32:42.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:42.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:42.996042+0000) 2024-02-20T02:32:43.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:43.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:43.996269+0000) 2024-02-20T02:32:44.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:44.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:44.996422+0000) 2024-02-20T02:32:45.991 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:45.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:45.996591+0000) 2024-02-20T02:32:46.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:46.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:46.996837+0000) 2024-02-20T02:32:47.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:47.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:47.997012+0000) 2024-02-20T02:32:48.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:48.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:48.997200+0000) 2024-02-20T02:32:49.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:49.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:49.997482+0000) 2024-02-20T02:32:50.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:50.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:50.997659+0000) 2024-02-20T02:32:51.992 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:51.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:51.997846+0000) 2024-02-20T02:32:52.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:52.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:52.998068+0000) 2024-02-20T02:32:53.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:53.991+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:53.998265+0000) 2024-02-20T02:32:54.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:54.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:54.998524+0000) 2024-02-20T02:32:55.993 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:55.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:55.998862+0000) 2024-02-20T02:32:56.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:56.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:56.999012+0000) 2024-02-20T02:32:57.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:57.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:57.999178+0000) 2024-02-20T02:32:58.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:58.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:58.999324+0000) 2024-02-20T02:32:59.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:32:59.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:32:59.999455+0000) 2024-02-20T02:33:00.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:00.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:00.999708+0000) 2024-02-20T02:33:01.994 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:01.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:01.999887+0000) 2024-02-20T02:33:02.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:02.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:03.000026+0000) 2024-02-20T02:33:03.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:03.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:04.000163+0000) 2024-02-20T02:33:04.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:04.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:05.000305+0000) 2024-02-20T02:33:05.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:05.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:06.000447+0000) 2024-02-20T02:33:06.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:06.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:07.000607+0000) 2024-02-20T02:33:07.995 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:07.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:08.000777+0000) 2024-02-20T02:33:08.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:08.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:09.001024+0000) 2024-02-20T02:33:09.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:09.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:10.001224+0000) 2024-02-20T02:33:10.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:10.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:11.001442+0000) 2024-02-20T02:33:11.996 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:11.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:12.001660+0000) 2024-02-20T02:33:12.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:12.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:13.001898+0000) 2024-02-20T02:33:13.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:13.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:14.002157+0000) 2024-02-20T02:33:14.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:14.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:15.002368+0000) 2024-02-20T02:33:15.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:15.995+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:16.002590+0000) 2024-02-20T02:33:16.997 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:16.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:17.002851+0000) 2024-02-20T02:33:17.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:17.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:18.003014+0000) 2024-02-20T02:33:18.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:18.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:19.003339+0000) 2024-02-20T02:33:19.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:19.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:20.003529+0000) 2024-02-20T02:33:20.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:20.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:21.003752+0000) 2024-02-20T02:33:21.998 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:21.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:22.003904+0000) 2024-02-20T02:33:22.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:22.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:23.004068+0000) 2024-02-20T02:33:23.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:24.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:24.004251+0000) 2024-02-20T02:33:24.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:24.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:25.004444+0000) 2024-02-20T02:33:25.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:26.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:26.004647+0000) 2024-02-20T02:33:26.999 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:26.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:27.004853+0000) 2024-02-20T02:33:28.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:27.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:28.005033+0000) 2024-02-20T02:33:29.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:28.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:29.005200+0000) 2024-02-20T02:33:30.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:29.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:30.005345+0000) 2024-02-20T02:33:31.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:30.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:31.005493+0000) 2024-02-20T02:33:32.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:31.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:32.005649+0000) 2024-02-20T02:33:33.000 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:32.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:33.005819+0000) 2024-02-20T02:33:34.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:34.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:34.005992+0000) 2024-02-20T02:33:35.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:34.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:35.006180+0000) 2024-02-20T02:33:36.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:35.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:36.006360+0000) 2024-02-20T02:33:37.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:36.999+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:37.006556+0000) 2024-02-20T02:33:38.001 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:38.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:38.006700+0000) 2024-02-20T02:33:39.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:39.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:39.006850+0000) 2024-02-20T02:33:40.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:40.003+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:40.007031+0000) 2024-02-20T02:33:41.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:41.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:41.007264+0000) 2024-02-20T02:33:42.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:42.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:42.007528+0000) 2024-02-20T02:33:43.002 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:43.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:43.007704+0000) 2024-02-20T02:33:44.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:44.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:44.007859+0000) 2024-02-20T02:33:45.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:45.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:45.008090+0000) 2024-02-20T02:33:46.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:46.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:46.008244+0000) 2024-02-20T02:33:47.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:47.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:47.008485+0000) 2024-02-20T02:33:48.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:48.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:48.008748+0000) 2024-02-20T02:33:49.003 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:49.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:49.008937+0000) 2024-02-20T02:33:50.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:50.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:50.009083+0000) 2024-02-20T02:33:51.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:51.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:51.009218+0000) 2024-02-20T02:33:52.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:52.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:52.009346+0000) 2024-02-20T02:33:53.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:53.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:53.009488+0000) 2024-02-20T02:33:54.004 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:54.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:54.009729+0000) 2024-02-20T02:33:55.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:55.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:55.009900+0000) 2024-02-20T02:33:56.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:56.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:56.010086+0000) 2024-02-20T02:33:57.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:57.002+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:57.010215+0000) 2024-02-20T02:33:58.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:58.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:58.010361+0000) 2024-02-20T02:33:59.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:33:59.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:33:59.010505+0000) 2024-02-20T02:34:00.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:00.006+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:00.010652+0000) 2024-02-20T02:34:01.005 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:01.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:01.010787+0000) 2024-02-20T02:34:02.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:02.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:02.011008+0000) 2024-02-20T02:34:03.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:03.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:03.011167+0000) 2024-02-20T02:34:04.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:04.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:04.011425+0000) 2024-02-20T02:34:05.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:05.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:05.011638+0000) 2024-02-20T02:34:06.006 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:06.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:06.011819+0000) 2024-02-20T02:34:07.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:07.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:07.012012+0000) 2024-02-20T02:34:08.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:08.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:08.012167+0000) 2024-02-20T02:34:09.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:09.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:09.012385+0000) 2024-02-20T02:34:10.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:10.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:10.012655+0000) 2024-02-20T02:34:11.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:11.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:11.012905+0000) 2024-02-20T02:34:12.007 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:12.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:12.013041+0000) 2024-02-20T02:34:13.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:13.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:13.013259+0000) 2024-02-20T02:34:14.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:14.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:14.013427+0000) 2024-02-20T02:34:15.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:15.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:15.013651+0000) 2024-02-20T02:34:16.008 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:16.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:16.013913+0000) 2024-02-20T02:34:17.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:17.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:17.014069+0000) 2024-02-20T02:34:18.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:18.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:18.014223+0000) 2024-02-20T02:34:19.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:19.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:19.014371+0000) 2024-02-20T02:34:20.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:20.010+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:20.014613+0000) 2024-02-20T02:34:21.009 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:21.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:21.014782+0000) 2024-02-20T02:34:22.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:22.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:22.015074+0000) 2024-02-20T02:34:23.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:23.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:23.015243+0000) 2024-02-20T02:34:24.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:24.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:24.015399+0000) 2024-02-20T02:34:25.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:25.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:25.015547+0000) 2024-02-20T02:34:26.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:26.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:26.015693+0000) 2024-02-20T02:34:27.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:27.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:27.015853+0000) 2024-02-20T02:34:28.010 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:28.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:28.016008+0000) 2024-02-20T02:34:29.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:29.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:29.016159+0000) 2024-02-20T02:34:30.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:30.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:30.016398+0000) 2024-02-20T02:34:31.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:31.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:31.016586+0000) 2024-02-20T02:34:32.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:32.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:32.016730+0000) 2024-02-20T02:34:33.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:33.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:33.016888+0000) 2024-02-20T02:34:34.011 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:34.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:34.017025+0000) 2024-02-20T02:34:35.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:35.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:35.017166+0000) 2024-02-20T02:34:36.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:36.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:36.017420+0000) 2024-02-20T02:34:37.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:37.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:37.017586+0000) 2024-02-20T02:34:38.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:38.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:38.017720+0000) 2024-02-20T02:34:39.012 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:39.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:39.017880+0000) 2024-02-20T02:34:40.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:40.014+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:40.018123+0000) 2024-02-20T02:34:41.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:41.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:41.018297+0000) 2024-02-20T02:34:42.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:42.013+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:42.018502+0000) 2024-02-20T02:34:43.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:43.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:43.018771+0000) 2024-02-20T02:34:44.013 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:44.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:44.018910+0000) 2024-02-20T02:34:45.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:45.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:45.019118+0000) 2024-02-20T02:34:46.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:46.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:46.019332+0000) 2024-02-20T02:34:47.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:47.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:47.019565+0000) 2024-02-20T02:34:48.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:48.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:48.019698+0000) 2024-02-20T02:34:49.014 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:49.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:49.019871+0000) 2024-02-20T02:34:50.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:50.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:50.020013+0000) 2024-02-20T02:34:51.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:51.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:51.020174+0000) 2024-02-20T02:34:52.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:52.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:52.020343+0000) 2024-02-20T02:34:53.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:53.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:53.020550+0000) 2024-02-20T02:34:54.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:54.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:54.020744+0000) 2024-02-20T02:34:55.015 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:55.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:55.020907+0000) 2024-02-20T02:34:56.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:56.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:56.021097+0000) 2024-02-20T02:34:57.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:57.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:57.021386+0000) 2024-02-20T02:34:58.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:58.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:58.021579+0000) 2024-02-20T02:34:59.016 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:34:59.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:34:59.021837+0000) 2024-02-20T02:35:00.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:00.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:00.021997+0000) 2024-02-20T02:35:01.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:01.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:01.022194+0000) 2024-02-20T02:35:02.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:02.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:02.022401+0000) 2024-02-20T02:35:03.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:03.017+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:03.022644+0000) 2024-02-20T02:35:04.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:04.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:04.022804+0000) 2024-02-20T02:35:05.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:05.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:05.022962+0000) 2024-02-20T02:35:06.017 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:06.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:06.023135+0000) 2024-02-20T02:35:07.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:07.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:07.023359+0000) 2024-02-20T02:35:08.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:08.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:08.023529+0000) 2024-02-20T02:35:09.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:09.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:09.023668+0000) 2024-02-20T02:35:10.018 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:10.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:10.023943+0000) 2024-02-20T02:35:11.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:11.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:11.024130+0000) 2024-02-20T02:35:12.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:12.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:12.024388+0000) 2024-02-20T02:35:13.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:13.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:13.024558+0000) 2024-02-20T02:35:14.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:14.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:14.024788+0000) 2024-02-20T02:35:15.019 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:15.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:15.024954+0000) 2024-02-20T02:35:16.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:16.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:16.025080+0000) 2024-02-20T02:35:17.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:17.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:17.025235+0000) 2024-02-20T02:35:18.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:18.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:18.025510+0000) 2024-02-20T02:35:19.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:19.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:19.025782+0000) 2024-02-20T02:35:20.020 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:20.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:20.025935+0000) 2024-02-20T02:35:21.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:21.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:21.026169+0000) 2024-02-20T02:35:22.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:22.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:22.026356+0000) 2024-02-20T02:35:23.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:23.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:23.026516+0000) 2024-02-20T02:35:24.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:24.021+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:24.026671+0000) 2024-02-20T02:35:25.021 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:25.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:25.026795+0000) 2024-02-20T02:35:26.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:26.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:26.026939+0000) 2024-02-20T02:35:27.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:27.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:27.027145+0000) 2024-02-20T02:35:28.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:28.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:28.027371+0000) 2024-02-20T02:35:29.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:29.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:29.027525+0000) 2024-02-20T02:35:30.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:30.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:30.027789+0000) 2024-02-20T02:35:31.022 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:31.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:31.027924+0000) 2024-02-20T02:35:32.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:32.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:32.028229+0000) 2024-02-20T02:35:33.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:33.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:33.028525+0000) 2024-02-20T02:35:34.023 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:34.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:34.028731+0000) 2024-02-20T02:35:35.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:35.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:35.028981+0000) 2024-02-20T02:35:36.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:36.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:36.029142+0000) 2024-02-20T02:35:37.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:37.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:37.029470+0000) 2024-02-20T02:35:38.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:38.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:38.029609+0000) 2024-02-20T02:35:39.024 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:39.029+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:39.029792+0000) 2024-02-20T02:35:40.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:40.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:40.029975+0000) 2024-02-20T02:35:41.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:41.025+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:41.030101+0000) 2024-02-20T02:35:42.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:42.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:42.030238+0000) 2024-02-20T02:35:43.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:43.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:43.030412+0000) 2024-02-20T02:35:44.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:44.024+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:44.030557+0000) 2024-02-20T02:35:45.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:45.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:45.030846+0000) 2024-02-20T02:35:46.025 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:46.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:46.030989+0000) 2024-02-20T02:35:47.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:47.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:47.031125+0000) 2024-02-20T02:35:48.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:48.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:48.031364+0000) 2024-02-20T02:35:49.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:49.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:49.031602+0000) 2024-02-20T02:35:50.026 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:50.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:50.031746+0000) 2024-02-20T02:35:51.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:51.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:51.032021+0000) 2024-02-20T02:35:52.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:52.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:52.032192+0000) 2024-02-20T02:35:53.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:53.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:53.032344+0000) 2024-02-20T02:35:54.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:54.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:54.032481+0000) 2024-02-20T02:35:55.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:55.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:55.032686+0000) 2024-02-20T02:35:56.027 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:56.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:56.032912+0000) 2024-02-20T02:35:57.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:57.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:57.033091+0000) 2024-02-20T02:35:58.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:58.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:58.033302+0000) 2024-02-20T02:35:59.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:35:59.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:35:59.033535+0000) 2024-02-20T02:36:00.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:00.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:00.033720+0000) 2024-02-20T02:36:01.028 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:01.028+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:01.033958+0000) 2024-02-20T02:36:02.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:02.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:02.034195+0000) 2024-02-20T02:36:03.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:03.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:03.034464+0000) 2024-02-20T02:36:04.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:04.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:04.034701+0000) 2024-02-20T02:36:05.029 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:05.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:05.034954+0000) 2024-02-20T02:36:06.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:06.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:06.035161+0000) 2024-02-20T02:36:07.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:07.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:07.035418+0000) 2024-02-20T02:36:08.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:08.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:08.035669+0000) 2024-02-20T02:36:09.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:09.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:09.035814+0000) 2024-02-20T02:36:10.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:10.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:10.035966+0000) 2024-02-20T02:36:11.030 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:11.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:11.036112+0000) 2024-02-20T02:36:12.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:12.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:12.036353+0000) 2024-02-20T02:36:13.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:13.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:13.036610+0000) 2024-02-20T02:36:14.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:14.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:14.036770+0000) 2024-02-20T02:36:15.031 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:15.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:15.036906+0000) 2024-02-20T02:36:16.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:16.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:16.037140+0000) 2024-02-20T02:36:17.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:17.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:17.037422+0000) 2024-02-20T02:36:18.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:18.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:18.037611+0000) 2024-02-20T02:36:19.032 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:19.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:19.037754+0000) 2024-02-20T02:36:20.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:20.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:20.038063+0000) 2024-02-20T02:36:21.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:21.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:21.038293+0000) 2024-02-20T02:36:22.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:22.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:22.038420+0000) 2024-02-20T02:36:23.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:23.032+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:23.038678+0000) 2024-02-20T02:36:24.033 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:24.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:24.038874+0000) 2024-02-20T02:36:25.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:25.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:25.039000+0000) 2024-02-20T02:36:26.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:26.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:26.039222+0000) 2024-02-20T02:36:27.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:27.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:27.039345+0000) 2024-02-20T02:36:28.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:28.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:28.039504+0000) 2024-02-20T02:36:29.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:29.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:29.039693+0000) 2024-02-20T02:36:30.034 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:30.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:30.039927+0000) 2024-02-20T02:36:31.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:31.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:31.040204+0000) 2024-02-20T02:36:32.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:32.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:32.040469+0000) 2024-02-20T02:36:33.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:33.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:33.040741+0000) 2024-02-20T02:36:34.035 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:34.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:34.040927+0000) 2024-02-20T02:36:35.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:35.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:35.041096+0000) 2024-02-20T02:36:36.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:36.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:36.041323+0000) 2024-02-20T02:36:37.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:37.040+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:37.041587+0000) 2024-02-20T02:36:38.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:38.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:38.041759+0000) 2024-02-20T02:36:39.036 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:39.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:39.041975+0000) 2024-02-20T02:36:40.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:40.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:40.042148+0000) 2024-02-20T02:36:41.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:41.036+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:41.042362+0000) 2024-02-20T02:36:42.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:42.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:42.042605+0000) 2024-02-20T02:36:43.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:43.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:43.042780+0000) 2024-02-20T02:36:44.037 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:44.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:44.042920+0000) 2024-02-20T02:36:45.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:45.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:45.043095+0000) 2024-02-20T02:36:46.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:46.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:46.043281+0000) 2024-02-20T02:36:47.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:47.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:47.043521+0000) 2024-02-20T02:36:48.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:48.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:48.043714+0000) 2024-02-20T02:36:49.038 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:49.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:49.043991+0000) 2024-02-20T02:36:50.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:50.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:50.044152+0000) 2024-02-20T02:36:51.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:51.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:51.044413+0000) 2024-02-20T02:36:52.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:52.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:52.044595+0000) 2024-02-20T02:36:53.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:53.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:53.044841+0000) 2024-02-20T02:36:54.039 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:54.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:54.045002+0000) 2024-02-20T02:36:55.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:55.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:55.045232+0000) 2024-02-20T02:36:56.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:56.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:56.045479+0000) 2024-02-20T02:36:57.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:57.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:57.045701+0000) 2024-02-20T02:36:58.040 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:58.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:58.045938+0000) 2024-02-20T02:36:59.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:36:59.039+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:36:59.046115+0000) 2024-02-20T02:37:00.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:00.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:00.046238+0000) 2024-02-20T02:37:01.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:01.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:01.046501+0000) 2024-02-20T02:37:02.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:02.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:02.046771+0000) 2024-02-20T02:37:03.041 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:03.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:03.047044+0000) 2024-02-20T02:37:04.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:04.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:04.047259+0000) 2024-02-20T02:37:05.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:05.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:05.047411+0000) 2024-02-20T02:37:06.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:06.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:06.047574+0000) 2024-02-20T02:37:07.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:07.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:07.047796+0000) 2024-02-20T02:37:08.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:08.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:08.047987+0000) 2024-02-20T02:37:09.042 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:09.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:09.048133+0000) 2024-02-20T02:37:10.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:10.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:10.048282+0000) 2024-02-20T02:37:11.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:11.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:11.048522+0000) 2024-02-20T02:37:12.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:12.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:12.048699+0000) 2024-02-20T02:37:13.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:13.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:13.048933+0000) 2024-02-20T02:37:14.043 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:14.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:14.049128+0000) 2024-02-20T02:37:15.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:15.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:15.049378+0000) 2024-02-20T02:37:16.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:16.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:16.049654+0000) 2024-02-20T02:37:17.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:17.043+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:17.049793+0000) 2024-02-20T02:37:18.044 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:18.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:18.050024+0000) 2024-02-20T02:37:19.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:19.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:19.050318+0000) 2024-02-20T02:37:20.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:20.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:20.050571+0000) 2024-02-20T02:37:21.045 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:21.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:21.050817+0000) 2024-02-20T02:37:22.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:22.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:22.051053+0000) 2024-02-20T02:37:23.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:23.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:23.051355+0000) 2024-02-20T02:37:24.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:24.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:24.051647+0000) 2024-02-20T02:37:25.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:25.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:25.051880+0000) 2024-02-20T02:37:26.046 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:26.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:26.052110+0000) 2024-02-20T02:37:27.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:27.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:27.052348+0000) 2024-02-20T02:37:28.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:28.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:28.052589+0000) 2024-02-20T02:37:29.047 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:29.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:29.052840+0000) 2024-02-20T02:37:30.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:30.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:30.053090+0000) 2024-02-20T02:37:31.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:31.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:31.053337+0000) 2024-02-20T02:37:32.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:32.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:32.053601+0000) 2024-02-20T02:37:33.048 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:33.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:33.053898+0000) 2024-02-20T02:37:34.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:34.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:34.054219+0000) 2024-02-20T02:37:35.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:35.047+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:35.054424+0000) 2024-02-20T02:37:36.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:36.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:36.054638+0000) 2024-02-20T02:37:37.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:37.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:37.054817+0000) 2024-02-20T02:37:38.049 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:38.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:38.055061+0000) 2024-02-20T02:37:39.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:39.055+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:39.055335+0000) 2024-02-20T02:37:40.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:40.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:40.055619+0000) 2024-02-20T02:37:41.050 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:41.051+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:41.055905+0000) 2024-02-20T02:37:42.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:42.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:42.056215+0000) 2024-02-20T02:37:43.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:43.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:43.056450+0000) 2024-02-20T02:37:44.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:44.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:44.056630+0000) 2024-02-20T02:37:45.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:45.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:45.056842+0000) 2024-02-20T02:37:46.051 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:46.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:46.057197+0000) 2024-02-20T02:37:47.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:47.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:47.057389+0000) 2024-02-20T02:37:48.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:48.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:48.057590+0000) 2024-02-20T02:37:49.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:49.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:49.057851+0000) 2024-02-20T02:37:50.052 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:50.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:50.058027+0000) 2024-02-20T02:37:51.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:51.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:51.058267+0000) 2024-02-20T02:37:52.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:52.050+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:52.058530+0000) 2024-02-20T02:37:53.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:53.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:53.058810+0000) 2024-02-20T02:37:54.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:54.054+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:54.058992+0000) 2024-02-20T02:37:55.053 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:55.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:55.059188+0000) 2024-02-20T02:37:56.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:56.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:56.059389+0000) 2024-02-20T02:37:57.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:57.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:57.059669+0000) 2024-02-20T02:37:58.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:58.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:58.059915+0000) 2024-02-20T02:37:59.054 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:37:59.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:37:59.060184+0000) 2024-02-20T02:38:00.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:00.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:00.060449+0000) 2024-02-20T02:38:01.055 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:01.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:01.060720+0000) 2024-02-20T02:38:02.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:02.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:02.061003+0000) 2024-02-20T02:38:03.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:03.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:03.061201+0000) 2024-02-20T02:38:04.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:04.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:04.061383+0000) 2024-02-20T02:38:05.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:05.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:05.061604+0000) 2024-02-20T02:38:06.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:06.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:06.061855+0000) 2024-02-20T02:38:07.056 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:07.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:07.062036+0000) 2024-02-20T02:38:08.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:08.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:08.062326+0000) 2024-02-20T02:38:09.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:09.058+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:09.062618+0000) 2024-02-20T02:38:10.057 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:10.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:10.062913+0000) 2024-02-20T02:38:11.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:11.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:11.063139+0000) 2024-02-20T02:38:12.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:12.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:12.063297+0000) 2024-02-20T02:38:13.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:13.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:13.063525+0000) 2024-02-20T02:38:14.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:14.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:14.063664+0000) 2024-02-20T02:38:15.058 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:15.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:15.063932+0000) 2024-02-20T02:38:16.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:16.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:16.064116+0000) 2024-02-20T02:38:17.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:17.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:17.064258+0000) 2024-02-20T02:38:18.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:18.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:18.064429+0000) 2024-02-20T02:38:19.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:19.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:19.064694+0000) 2024-02-20T02:38:20.059 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:20.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:20.064875+0000) 2024-02-20T02:38:21.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:21.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:21.065110+0000) 2024-02-20T02:38:22.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:22.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:22.065400+0000) 2024-02-20T02:38:23.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:23.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:23.065672+0000) 2024-02-20T02:38:24.060 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:24.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:24.066023+0000) 2024-02-20T02:38:25.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:25.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:25.066274+0000) 2024-02-20T02:38:26.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:26.062+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:26.066512+0000) 2024-02-20T02:38:27.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:27.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:27.066785+0000) 2024-02-20T02:38:28.061 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:28.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:28.067047+0000) 2024-02-20T02:38:29.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:29.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:29.067222+0000) 2024-02-20T02:38:30.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:30.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:30.067415+0000) 2024-02-20T02:38:31.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:31.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:31.067699+0000) 2024-02-20T02:38:32.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:32.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:32.067929+0000) 2024-02-20T02:38:33.062 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:33.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:33.068146+0000) 2024-02-20T02:38:34.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:34.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:34.068348+0000) 2024-02-20T02:38:35.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:35.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:35.068568+0000) 2024-02-20T02:38:36.063 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:36.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:36.068849+0000) 2024-02-20T02:38:37.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:37.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:37.069138+0000) 2024-02-20T02:38:38.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:38.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:38.069428+0000) 2024-02-20T02:38:39.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:39.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:39.069629+0000) 2024-02-20T02:38:40.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:40.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:40.069777+0000) 2024-02-20T02:38:41.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:41.066+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:41.069929+0000) 2024-02-20T02:38:42.064 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:42.065+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:42.070066+0000) 2024-02-20T02:38:43.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:43.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:43.070335+0000) 2024-02-20T02:38:44.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:44.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:44.070636+0000) 2024-02-20T02:38:45.065 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:45.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:45.070908+0000) 2024-02-20T02:38:46.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:46.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:46.071212+0000) 2024-02-20T02:38:47.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:47.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:47.071397+0000) 2024-02-20T02:38:48.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:48.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:48.071663+0000) 2024-02-20T02:38:49.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:49.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:49.071841+0000) 2024-02-20T02:38:50.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:50.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:50.071982+0000) 2024-02-20T02:38:51.066 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:51.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:51.072186+0000) 2024-02-20T02:38:52.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:52.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:52.072497+0000) 2024-02-20T02:38:53.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:53.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:53.072701+0000) 2024-02-20T02:38:54.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:54.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:54.072838+0000) 2024-02-20T02:38:55.067 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:55.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:55.073017+0000) 2024-02-20T02:38:56.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:56.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:56.073168+0000) 2024-02-20T02:38:57.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:57.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:57.073329+0000) 2024-02-20T02:38:58.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:58.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:58.073580+0000) 2024-02-20T02:38:59.068 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:38:59.069+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:38:59.073838+0000) 2024-02-20T02:39:00.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:00.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:00.074104+0000) 2024-02-20T02:39:01.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:01.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:01.074347+0000) 2024-02-20T02:39:02.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:02.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:02.074615+0000) 2024-02-20T02:39:03.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:03.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:03.074848+0000) 2024-02-20T02:39:04.069 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:04.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:04.075079+0000) 2024-02-20T02:39:05.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:05.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:05.075277+0000) 2024-02-20T02:39:06.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:06.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:06.075493+0000) 2024-02-20T02:39:07.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:07.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:07.075712+0000) 2024-02-20T02:39:08.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:08.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:08.075933+0000) 2024-02-20T02:39:09.070 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:09.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:09.076145+0000) 2024-02-20T02:39:10.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:10.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:10.076417+0000) 2024-02-20T02:39:11.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:11.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:11.076658+0000) 2024-02-20T02:39:12.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:12.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:12.076893+0000) 2024-02-20T02:39:13.071 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:13.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:13.077137+0000) 2024-02-20T02:39:14.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:14.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:14.077404+0000) 2024-02-20T02:39:15.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:15.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:15.077698+0000) 2024-02-20T02:39:16.072 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:16.073+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:16.077930+0000) 2024-02-20T02:39:17.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:17.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:17.078229+0000) 2024-02-20T02:39:18.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:18.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:18.078528+0000) 2024-02-20T02:39:19.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:19.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:19.078845+0000) 2024-02-20T02:39:20.073 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:20.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:20.079046+0000) 2024-02-20T02:39:21.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:21.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:21.079296+0000) 2024-02-20T02:39:22.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:22.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:22.079519+0000) 2024-02-20T02:39:23.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:23.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:23.079724+0000) 2024-02-20T02:39:24.074 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:24.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:24.079925+0000) 2024-02-20T02:39:25.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:25.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:25.080159+0000) 2024-02-20T02:39:26.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:26.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:26.080335+0000) 2024-02-20T02:39:27.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:27.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:27.080480+0000) 2024-02-20T02:39:28.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:28.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:28.080666+0000) 2024-02-20T02:39:29.075 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:29.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:29.080899+0000) 2024-02-20T02:39:30.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:30.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:30.081147+0000) 2024-02-20T02:39:31.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:31.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:31.081347+0000) 2024-02-20T02:39:32.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:32.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:32.081577+0000) 2024-02-20T02:39:33.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:33.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:33.081817+0000) 2024-02-20T02:39:34.076 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:34.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:34.082065+0000) 2024-02-20T02:39:35.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:35.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:35.082296+0000) 2024-02-20T02:39:36.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:36.077+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:36.082569+0000) 2024-02-20T02:39:37.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:37.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:37.082830+0000) 2024-02-20T02:39:38.077 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:38.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:38.083040+0000) 2024-02-20T02:39:39.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:39.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:39.083313+0000) 2024-02-20T02:39:40.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:40.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:40.083605+0000) 2024-02-20T02:39:41.078 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:41.081+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:41.083904+0000) 2024-02-20T02:39:42.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:42.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:42.084103+0000) 2024-02-20T02:39:43.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:43.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:43.084253+0000) 2024-02-20T02:39:44.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:44.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:44.084412+0000) 2024-02-20T02:39:45.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:45.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:45.084561+0000) 2024-02-20T02:39:46.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:46.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:46.084751+0000) 2024-02-20T02:39:47.079 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:47.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:47.084930+0000) 2024-02-20T02:39:48.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:48.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:48.085203+0000) 2024-02-20T02:39:49.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:49.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:49.085530+0000) 2024-02-20T02:39:50.080 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:50.080+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:50.085715+0000) 2024-02-20T02:39:51.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:51.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:51.086127+0000) 2024-02-20T02:39:52.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:52.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:52.086294+0000) 2024-02-20T02:39:53.081 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:53.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:53.086523+0000) 2024-02-20T02:39:54.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:54.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:54.086755+0000) 2024-02-20T02:39:55.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:55.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:55.086920+0000) 2024-02-20T02:39:56.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:56.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:56.087076+0000) 2024-02-20T02:39:57.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:57.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:57.087241+0000) 2024-02-20T02:39:58.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:58.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:58.087396+0000) 2024-02-20T02:39:59.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:39:59.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:39:59.087548+0000) 2024-02-20T02:40:00.082 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:00.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:00.087760+0000) 2024-02-20T02:40:01.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:01.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:01.087932+0000) 2024-02-20T02:40:02.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:02.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:02.088201+0000) 2024-02-20T02:40:03.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:03.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:03.088369+0000) 2024-02-20T02:40:04.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:04.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:04.088526+0000) 2024-02-20T02:40:05.083 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:05.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:05.088742+0000) 2024-02-20T02:40:06.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:06.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:06.088955+0000) 2024-02-20T02:40:07.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:07.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:07.089182+0000) 2024-02-20T02:40:08.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:08.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:08.089381+0000) 2024-02-20T02:40:09.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:09.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:09.089513+0000) 2024-02-20T02:40:10.084 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:10.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:10.089751+0000) 2024-02-20T02:40:11.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:11.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:11.089901+0000) 2024-02-20T02:40:12.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:12.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:12.090039+0000) 2024-02-20T02:40:13.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:13.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:13.090193+0000) 2024-02-20T02:40:14.085 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:14.084+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:14.090452+0000) 2024-02-20T02:40:15.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:15.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:15.090693+0000) 2024-02-20T02:40:16.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:16.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:16.090851+0000) 2024-02-20T02:40:17.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:17.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:17.091069+0000) 2024-02-20T02:40:18.086 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:18.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:18.091317+0000) 2024-02-20T02:40:19.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:19.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:19.091477+0000) 2024-02-20T02:40:20.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:20.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:20.091685+0000) 2024-02-20T02:40:21.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:21.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:21.091834+0000) 2024-02-20T02:40:22.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:22.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:22.092000+0000) 2024-02-20T02:40:23.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:23.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:23.092165+0000) 2024-02-20T02:40:24.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:24.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:24.092356+0000) 2024-02-20T02:40:25.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:25.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:25.092540+0000) 2024-02-20T02:40:26.087 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:26.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:26.092711+0000) 2024-02-20T02:40:27.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:27.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:27.092910+0000) 2024-02-20T02:40:28.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:28.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:28.093118+0000) 2024-02-20T02:40:29.088 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:29.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:29.093379+0000) 2024-02-20T02:40:30.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:30.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:30.093648+0000) 2024-02-20T02:40:31.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:31.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:31.093835+0000) 2024-02-20T02:40:32.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:32.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:32.093998+0000) 2024-02-20T02:40:33.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:33.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:33.094212+0000) 2024-02-20T02:40:34.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:34.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:34.094449+0000) 2024-02-20T02:40:35.089 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:35.088+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:35.094612+0000) 2024-02-20T02:40:36.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:36.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:36.094842+0000) 2024-02-20T02:40:37.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:37.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:37.095029+0000) 2024-02-20T02:40:38.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:38.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:38.095269+0000) 2024-02-20T02:40:39.090 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:39.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:39.095529+0000) 2024-02-20T02:40:40.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:40.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:40.095762+0000) 2024-02-20T02:40:41.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:41.092+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:41.095936+0000) 2024-02-20T02:40:42.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:42.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:42.096175+0000) 2024-02-20T02:40:43.091 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:43.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:43.096362+0000) 2024-02-20T02:40:44.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:44.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:44.096523+0000) 2024-02-20T02:40:45.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:45.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:45.096656+0000) 2024-02-20T02:40:46.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:46.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:46.096796+0000) 2024-02-20T02:40:47.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:47.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:47.097002+0000) 2024-02-20T02:40:48.092 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:48.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:48.097170+0000) 2024-02-20T02:40:49.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:49.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:49.097460+0000) 2024-02-20T02:40:50.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:50.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:50.097629+0000) 2024-02-20T02:40:51.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:51.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:51.097846+0000) 2024-02-20T02:40:52.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:52.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:52.098047+0000) 2024-02-20T02:40:53.093 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:53.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:53.098300+0000) 2024-02-20T02:40:54.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:54.091+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:54.098616+0000) 2024-02-20T02:40:55.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:55.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:55.098783+0000) 2024-02-20T02:40:56.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:56.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:56.098920+0000) 2024-02-20T02:40:57.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:57.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:57.099059+0000) 2024-02-20T02:40:58.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:58.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:58.099211+0000) 2024-02-20T02:40:59.094 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:40:59.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:40:59.099365+0000) 2024-02-20T02:41:00.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:00.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:00.099591+0000) 2024-02-20T02:41:01.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:01.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:01.099794+0000) 2024-02-20T02:41:02.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:02.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:02.100038+0000) 2024-02-20T02:41:03.095 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:03.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:03.100329+0000) 2024-02-20T02:41:04.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:04.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:04.100471+0000) 2024-02-20T02:41:05.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:05.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:05.100617+0000) 2024-02-20T02:41:06.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:06.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:06.100874+0000) 2024-02-20T02:41:07.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:07.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:07.101123+0000) 2024-02-20T02:41:08.096 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:08.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:08.101270+0000) 2024-02-20T02:41:09.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:09.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:09.101518+0000) 2024-02-20T02:41:10.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:10.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:10.101648+0000) 2024-02-20T02:41:11.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:11.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:11.101797+0000) 2024-02-20T02:41:12.097 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:12.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:12.102022+0000) 2024-02-20T02:41:13.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:13.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:13.102231+0000) 2024-02-20T02:41:14.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:14.095+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:14.102493+0000) 2024-02-20T02:41:15.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:15.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:15.102807+0000) 2024-02-20T02:41:16.098 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:16.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:16.103048+0000) 2024-02-20T02:41:17.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:17.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:17.103434+0000) 2024-02-20T02:41:18.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:18.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:18.103574+0000) 2024-02-20T02:41:19.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:19.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:19.103702+0000) 2024-02-20T02:41:20.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:20.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:20.103954+0000) 2024-02-20T02:41:21.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:21.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:21.104107+0000) 2024-02-20T02:41:22.099 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:22.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:22.104298+0000) 2024-02-20T02:41:23.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:23.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:23.104556+0000) 2024-02-20T02:41:24.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:24.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:24.104726+0000) 2024-02-20T02:41:25.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:25.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:25.104860+0000) 2024-02-20T02:41:26.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:26.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:26.104996+0000) 2024-02-20T02:41:27.100 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:27.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:27.105139+0000) 2024-02-20T02:41:28.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:28.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:28.105279+0000) 2024-02-20T02:41:29.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:29.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:29.105474+0000) 2024-02-20T02:41:30.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:30.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:30.105637+0000) 2024-02-20T02:41:31.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:31.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:31.105785+0000) 2024-02-20T02:41:32.101 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:32.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:32.105937+0000) 2024-02-20T02:41:33.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:33.099+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:33.106109+0000) 2024-02-20T02:41:34.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:34.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:34.106364+0000) 2024-02-20T02:41:35.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:35.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:35.106555+0000) 2024-02-20T02:41:36.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:36.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:36.106771+0000) 2024-02-20T02:41:37.102 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:37.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:37.106926+0000) 2024-02-20T02:41:38.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:38.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:38.107168+0000) 2024-02-20T02:41:39.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:39.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:39.107403+0000) 2024-02-20T02:41:40.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:40.107+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:40.107637+0000) 2024-02-20T02:41:41.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:41.103+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:41.107874+0000) 2024-02-20T02:41:42.103 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:42.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:42.108049+0000) 2024-02-20T02:41:43.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:43.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:43.108279+0000) 2024-02-20T02:41:44.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:44.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:44.108535+0000) 2024-02-20T02:41:45.104 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:45.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:45.108838+0000) 2024-02-20T02:41:46.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:46.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:46.109097+0000) 2024-02-20T02:41:47.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:47.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:47.109241+0000) 2024-02-20T02:41:48.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:48.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:48.109443+0000) 2024-02-20T02:41:49.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:49.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:49.109597+0000) 2024-02-20T02:41:50.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:50.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:50.109736+0000) 2024-02-20T02:41:51.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:51.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:51.109885+0000) 2024-02-20T02:41:52.105 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:52.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:52.110021+0000) 2024-02-20T02:41:53.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:53.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:53.110175+0000) 2024-02-20T02:41:54.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:54.102+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:54.110469+0000) 2024-02-20T02:41:55.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:55.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:55.110743+0000) 2024-02-20T02:41:56.106 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:56.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:56.110979+0000) 2024-02-20T02:41:57.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:57.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:57.111239+0000) 2024-02-20T02:41:58.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:58.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:58.111513+0000) 2024-02-20T02:41:59.107 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:41:59.106+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:41:59.111753+0000) 2024-02-20T02:42:00.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:00.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:00.112054+0000) 2024-02-20T02:42:01.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:01.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:01.112280+0000) 2024-02-20T02:42:02.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:02.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:02.112558+0000) 2024-02-20T02:42:03.108 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:03.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:03.112771+0000) 2024-02-20T02:42:04.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:04.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:04.112974+0000) 2024-02-20T02:42:05.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:05.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:05.113258+0000) 2024-02-20T02:42:06.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:06.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:06.113459+0000) 2024-02-20T02:42:07.109 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:07.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:07.113679+0000) 2024-02-20T02:42:08.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:08.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:08.113943+0000) 2024-02-20T02:42:09.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:09.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:09.114252+0000) 2024-02-20T02:42:10.110 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:10.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:10.114486+0000) 2024-02-20T02:42:11.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:11.110+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:11.114694+0000) 2024-02-20T02:42:12.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:12.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:12.114892+0000) 2024-02-20T02:42:13.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:13.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:13.115194+0000) 2024-02-20T02:42:14.111 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:14.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:14.115476+0000) 2024-02-20T02:42:15.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:15.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:15.115741+0000) 2024-02-20T02:42:16.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:16.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:16.115977+0000) 2024-02-20T02:42:17.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:17.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:17.116182+0000) 2024-02-20T02:42:18.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:18.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:18.116399+0000) 2024-02-20T02:42:19.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:19.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:19.116611+0000) 2024-02-20T02:42:20.112 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:20.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:20.116839+0000) 2024-02-20T02:42:21.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:21.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:21.117085+0000) 2024-02-20T02:42:22.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:22.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:22.117244+0000) 2024-02-20T02:42:23.113 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:23.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:23.117521+0000) 2024-02-20T02:42:24.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:24.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:24.117799+0000) 2024-02-20T02:42:25.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:25.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:25.118057+0000) 2024-02-20T02:42:26.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:26.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:26.118249+0000) 2024-02-20T02:42:27.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:27.114+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:27.118475+0000) 2024-02-20T02:42:28.114 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:28.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:28.118700+0000) 2024-02-20T02:42:29.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:29.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:29.118960+0000) 2024-02-20T02:42:30.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:30.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:30.119226+0000) 2024-02-20T02:42:31.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:31.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:31.119472+0000) 2024-02-20T02:42:32.115 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:32.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:32.119639+0000) 2024-02-20T02:42:33.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:33.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:33.119849+0000) 2024-02-20T02:42:34.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:34.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:34.120107+0000) 2024-02-20T02:42:35.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:35.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:35.120347+0000) 2024-02-20T02:42:36.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:36.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:36.120560+0000) 2024-02-20T02:42:37.116 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:37.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:37.120711+0000) 2024-02-20T02:42:38.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:38.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:38.120917+0000) 2024-02-20T02:42:39.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:39.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:39.121188+0000) 2024-02-20T02:42:40.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:40.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:40.121379+0000) 2024-02-20T02:42:41.117 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:41.118+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:41.121523+0000) 2024-02-20T02:42:42.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:42.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:42.121736+0000) 2024-02-20T02:42:43.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:43.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:43.121902+0000) 2024-02-20T02:42:44.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:44.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:44.122064+0000) 2024-02-20T02:42:45.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:45.117+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:45.122217+0000) 2024-02-20T02:42:46.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:46.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:46.122382+0000) 2024-02-20T02:42:47.118 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:47.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:47.122619+0000) 2024-02-20T02:42:48.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:48.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:48.122855+0000) 2024-02-20T02:42:49.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:49.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:49.123066+0000) 2024-02-20T02:42:50.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:50.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:50.123296+0000) 2024-02-20T02:42:51.119 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:51.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:51.123555+0000) 2024-02-20T02:42:52.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:52.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:52.123824+0000) 2024-02-20T02:42:53.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:53.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:53.124080+0000) 2024-02-20T02:42:54.120 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:54.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:54.124350+0000) 2024-02-20T02:42:55.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:55.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:55.124622+0000) 2024-02-20T02:42:56.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:56.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:56.124792+0000) 2024-02-20T02:42:57.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:57.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:57.125022+0000) 2024-02-20T02:42:58.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:58.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:58.125272+0000) 2024-02-20T02:42:59.121 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:42:59.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:42:59.125565+0000) 2024-02-20T02:43:00.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:00.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:00.125823+0000) 2024-02-20T02:43:01.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:01.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:01.125982+0000) 2024-02-20T02:43:02.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:02.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:02.126214+0000) 2024-02-20T02:43:03.122 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:03.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:03.126410+0000) 2024-02-20T02:43:04.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:04.121+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:04.126585+0000) 2024-02-20T02:43:05.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:05.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:05.126845+0000) 2024-02-20T02:43:06.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:06.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:06.127021+0000) 2024-02-20T02:43:07.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:07.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:07.127180+0000) 2024-02-20T02:43:08.123 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:08.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:08.127350+0000) 2024-02-20T02:43:09.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:09.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:09.127589+0000) 2024-02-20T02:43:10.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:10.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:10.127775+0000) 2024-02-20T02:43:11.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:11.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:11.128046+0000) 2024-02-20T02:43:12.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:12.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:12.128303+0000) 2024-02-20T02:43:13.124 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:13.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:13.128504+0000) 2024-02-20T02:43:14.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:14.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:14.128674+0000) 2024-02-20T02:43:15.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:15.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:15.128986+0000) 2024-02-20T02:43:16.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:16.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:16.129266+0000) 2024-02-20T02:43:17.125 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:17.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:17.129484+0000) 2024-02-20T02:43:18.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:18.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:18.129644+0000) 2024-02-20T02:43:19.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:19.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:19.129789+0000) 2024-02-20T02:43:20.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:20.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:20.129937+0000) 2024-02-20T02:43:21.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:21.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:21.130091+0000) 2024-02-20T02:43:22.126 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:22.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:22.130287+0000) 2024-02-20T02:43:23.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:23.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:23.130444+0000) 2024-02-20T02:43:24.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:24.125+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:24.130607+0000) 2024-02-20T02:43:25.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:25.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:25.130797+0000) 2024-02-20T02:43:26.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:26.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:26.130995+0000) 2024-02-20T02:43:27.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:27.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:27.131216+0000) 2024-02-20T02:43:28.127 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:28.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:28.131421+0000) 2024-02-20T02:43:29.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:29.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:29.131648+0000) 2024-02-20T02:43:30.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:30.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:30.131904+0000) 2024-02-20T02:43:31.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:31.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:31.132121+0000) 2024-02-20T02:43:32.128 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:32.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:32.132340+0000) 2024-02-20T02:43:33.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:33.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:33.132612+0000) 2024-02-20T02:43:34.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:34.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:34.132848+0000) 2024-02-20T02:43:35.129 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:35.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:35.133009+0000) 2024-02-20T02:43:36.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:36.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:36.133289+0000) 2024-02-20T02:43:37.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:37.133+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:37.133558+0000) 2024-02-20T02:43:38.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:38.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:38.133725+0000) 2024-02-20T02:43:39.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:39.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:39.133857+0000) 2024-02-20T02:43:40.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:40.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:40.134016+0000) 2024-02-20T02:43:41.130 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:41.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:41.134200+0000) 2024-02-20T02:43:42.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:42.129+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:42.134523+0000) 2024-02-20T02:43:43.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:43.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:43.134726+0000) 2024-02-20T02:43:44.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:44.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:44.134880+0000) 2024-02-20T02:43:45.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:45.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:45.135126+0000) 2024-02-20T02:43:46.131 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:46.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:46.135341+0000) 2024-02-20T02:43:47.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:47.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:47.135522+0000) 2024-02-20T02:43:48.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:48.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:48.135681+0000) 2024-02-20T02:43:49.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:49.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:49.135857+0000) 2024-02-20T02:43:50.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:50.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:50.136032+0000) 2024-02-20T02:43:51.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:51.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:51.136264+0000) 2024-02-20T02:43:52.132 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:52.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:52.136453+0000) 2024-02-20T02:43:53.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:53.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:53.136623+0000) 2024-02-20T02:43:54.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:54.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:54.136839+0000) 2024-02-20T02:43:55.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:55.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:55.137096+0000) 2024-02-20T02:43:56.133 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:56.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:56.137359+0000) 2024-02-20T02:43:57.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:57.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:57.137594+0000) 2024-02-20T02:43:58.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:58.132+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:58.137774+0000) 2024-02-20T02:43:59.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:43:59.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:43:59.137933+0000) 2024-02-20T02:44:00.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:00.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:00.138099+0000) 2024-02-20T02:44:01.134 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:01.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:01.138298+0000) 2024-02-20T02:44:02.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:02.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:02.138531+0000) 2024-02-20T02:44:03.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:03.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:03.138856+0000) 2024-02-20T02:44:04.135 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:04.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:04.139073+0000) 2024-02-20T02:44:05.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:05.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:05.139369+0000) 2024-02-20T02:44:06.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:06.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:06.139621+0000) 2024-02-20T02:44:07.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:07.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:07.139778+0000) 2024-02-20T02:44:08.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:08.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:08.139940+0000) 2024-02-20T02:44:09.136 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:09.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:09.140100+0000) 2024-02-20T02:44:10.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:10.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:10.140264+0000) 2024-02-20T02:44:11.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:11.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:11.140502+0000) 2024-02-20T02:44:12.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:12.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:12.140655+0000) 2024-02-20T02:44:13.137 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:13.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:13.140949+0000) 2024-02-20T02:44:14.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:14.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:14.141153+0000) 2024-02-20T02:44:15.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:15.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:15.141324+0000) 2024-02-20T02:44:16.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:16.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:16.141597+0000) 2024-02-20T02:44:17.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:17.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:17.141936+0000) 2024-02-20T02:44:18.138 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:18.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:18.142131+0000) 2024-02-20T02:44:19.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:19.136+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:19.142329+0000) 2024-02-20T02:44:20.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:20.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:20.142481+0000) 2024-02-20T02:44:21.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:21.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:21.142683+0000) 2024-02-20T02:44:22.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:22.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:22.142874+0000) 2024-02-20T02:44:23.139 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:23.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:23.143097+0000) 2024-02-20T02:44:24.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:24.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:24.143333+0000) 2024-02-20T02:44:25.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:25.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:25.143551+0000) 2024-02-20T02:44:26.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:26.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:26.143802+0000) 2024-02-20T02:44:27.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:27.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:27.143990+0000) 2024-02-20T02:44:28.140 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:28.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:28.144150+0000) 2024-02-20T02:44:29.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:29.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:29.144314+0000) 2024-02-20T02:44:30.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:30.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:30.144580+0000) 2024-02-20T02:44:31.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:31.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:31.144768+0000) 2024-02-20T02:44:32.141 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:32.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:32.144959+0000) 2024-02-20T02:44:33.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:33.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:33.145217+0000) 2024-02-20T02:44:34.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:34.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:34.145576+0000) 2024-02-20T02:44:35.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:35.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:35.145751+0000) 2024-02-20T02:44:36.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:36.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:36.145920+0000) 2024-02-20T02:44:37.142 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:37.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:37.146080+0000) 2024-02-20T02:44:38.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:38.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:38.146237+0000) 2024-02-20T02:44:39.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:39.140+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:39.146465+0000) 2024-02-20T02:44:40.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:40.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:40.146617+0000) 2024-02-20T02:44:41.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:41.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:41.146827+0000) 2024-02-20T02:44:42.143 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:42.144+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:42.146971+0000) 2024-02-20T02:44:43.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:43.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:43.147110+0000) 2024-02-20T02:44:44.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:44.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:44.147253+0000) 2024-02-20T02:44:45.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:45.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:45.147382+0000) 2024-02-20T02:44:46.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:46.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:46.147538+0000) 2024-02-20T02:44:47.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:47.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:47.147771+0000) 2024-02-20T02:44:48.144 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:48.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:48.148014+0000) 2024-02-20T02:44:49.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:49.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:49.148152+0000) 2024-02-20T02:44:50.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:50.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:50.148408+0000) 2024-02-20T02:44:51.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:51.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:51.148597+0000) 2024-02-20T02:44:52.145 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:52.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:52.148738+0000) 2024-02-20T02:44:53.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:53.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:53.148965+0000) 2024-02-20T02:44:54.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:54.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:54.149222+0000) 2024-02-20T02:44:55.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:55.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:55.149412+0000) 2024-02-20T02:44:56.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:56.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:56.149574+0000) 2024-02-20T02:44:57.146 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:57.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:57.149718+0000) 2024-02-20T02:44:58.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:58.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:58.149944+0000) 2024-02-20T02:44:59.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:44:59.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:44:59.150210+0000) 2024-02-20T02:45:00.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:00.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:00.150373+0000) 2024-02-20T02:45:01.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:01.143+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:01.150608+0000) 2024-02-20T02:45:02.147 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:02.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:02.150894+0000) 2024-02-20T02:45:03.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:03.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:03.151105+0000) 2024-02-20T02:45:04.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:04.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:04.151347+0000) 2024-02-20T02:45:05.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:05.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:05.151543+0000) 2024-02-20T02:45:06.148 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:06.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:06.151822+0000) 2024-02-20T02:45:07.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:07.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:07.152100+0000) 2024-02-20T02:45:08.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:08.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:08.152253+0000) 2024-02-20T02:45:09.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:09.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:09.152391+0000) 2024-02-20T02:45:10.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:10.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:10.152541+0000) 2024-02-20T02:45:11.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:11.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:11.152677+0000) 2024-02-20T02:45:12.149 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:12.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:12.152867+0000) 2024-02-20T02:45:13.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:13.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:13.153098+0000) 2024-02-20T02:45:14.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:14.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:14.153259+0000) 2024-02-20T02:45:15.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:15.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:15.153423+0000) 2024-02-20T02:45:16.150 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:16.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:16.153663+0000) 2024-02-20T02:45:17.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:17.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:17.154030+0000) 2024-02-20T02:45:18.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:18.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:18.154199+0000) 2024-02-20T02:45:19.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:19.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:19.154359+0000) 2024-02-20T02:45:20.151 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:20.147+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:20.154504+0000) 2024-02-20T02:45:21.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:21.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:21.154815+0000) 2024-02-20T02:45:22.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:22.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:22.154954+0000) 2024-02-20T02:45:23.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:23.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:23.155212+0000) 2024-02-20T02:45:24.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:24.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:24.155397+0000) 2024-02-20T02:45:25.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:25.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:25.155535+0000) 2024-02-20T02:45:26.152 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:26.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:26.155708+0000) 2024-02-20T02:45:27.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:27.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:27.155858+0000) 2024-02-20T02:45:28.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:28.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:28.156127+0000) 2024-02-20T02:45:29.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:29.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:29.156359+0000) 2024-02-20T02:45:30.153 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:30.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:30.156591+0000) 2024-02-20T02:45:31.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:31.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:31.156808+0000) 2024-02-20T02:45:32.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:32.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:32.157002+0000) 2024-02-20T02:45:33.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:33.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:33.157210+0000) 2024-02-20T02:45:34.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:34.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:34.157373+0000) 2024-02-20T02:45:35.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:35.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:35.157525+0000) 2024-02-20T02:45:36.154 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:36.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:36.157752+0000) 2024-02-20T02:45:37.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:37.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:37.157941+0000) 2024-02-20T02:45:38.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:38.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:38.158086+0000) 2024-02-20T02:45:39.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:39.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:39.158259+0000) 2024-02-20T02:45:40.155 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:40.151+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:40.158429+0000) 2024-02-20T02:45:41.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:41.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:41.158703+0000) 2024-02-20T02:45:42.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:42.155+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:42.158837+0000) 2024-02-20T02:45:43.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:43.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:43.159003+0000) 2024-02-20T02:45:44.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:44.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:44.159240+0000) 2024-02-20T02:45:45.156 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:45.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:45.159426+0000) 2024-02-20T02:45:46.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:46.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:46.159720+0000) 2024-02-20T02:45:47.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:47.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:47.159884+0000) 2024-02-20T02:45:48.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:48.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:48.160116+0000) 2024-02-20T02:45:49.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:49.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:49.160307+0000) 2024-02-20T02:45:50.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:50.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:50.160534+0000) 2024-02-20T02:45:51.157 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:51.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:51.160712+0000) 2024-02-20T02:45:52.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:52.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:52.160862+0000) 2024-02-20T02:45:53.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:53.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:53.161077+0000) 2024-02-20T02:45:54.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:54.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:54.161323+0000) 2024-02-20T02:45:55.158 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:55.154+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:55.161517+0000) 2024-02-20T02:45:56.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:56.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:56.161740+0000) 2024-02-20T02:45:57.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:57.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:57.161994+0000) 2024-02-20T02:45:58.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:58.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:58.162214+0000) 2024-02-20T02:45:59.159 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:45:59.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:45:59.162387+0000) 2024-02-20T02:46:00.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:00.158+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:00.162617+0000) 2024-02-20T02:46:01.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:01.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:01.162885+0000) 2024-02-20T02:46:02.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:02.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:02.163136+0000) 2024-02-20T02:46:03.160 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:03.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:03.163440+0000) 2024-02-20T02:46:04.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:04.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:04.163636+0000) 2024-02-20T02:46:05.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:05.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:05.163903+0000) 2024-02-20T02:46:06.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:06.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:06.164064+0000) 2024-02-20T02:46:07.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:07.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:07.164347+0000) 2024-02-20T02:46:08.161 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:08.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:08.164580+0000) 2024-02-20T02:46:09.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:09.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:09.164744+0000) 2024-02-20T02:46:10.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:10.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:10.164908+0000) 2024-02-20T02:46:11.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:11.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:11.165084+0000) 2024-02-20T02:46:12.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:12.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:12.165258+0000) 2024-02-20T02:46:13.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:13.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:13.165465+0000) 2024-02-20T02:46:14.162 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:14.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:14.165624+0000) 2024-02-20T02:46:15.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:15.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:15.165782+0000) 2024-02-20T02:46:16.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:16.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:16.165945+0000) 2024-02-20T02:46:17.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:17.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:17.166097+0000) 2024-02-20T02:46:18.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:18.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:18.166372+0000) 2024-02-20T02:46:19.163 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:19.162+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:19.166517+0000) 2024-02-20T02:46:20.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:20.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:20.166840+0000) 2024-02-20T02:46:21.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:21.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:21.167047+0000) 2024-02-20T02:46:22.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:22.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:22.167205+0000) 2024-02-20T02:46:23.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:23.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:23.167368+0000) 2024-02-20T02:46:24.164 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:24.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:24.167568+0000) 2024-02-20T02:46:25.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:25.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:25.167718+0000) 2024-02-20T02:46:26.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:26.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:26.167944+0000) 2024-02-20T02:46:27.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:27.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:27.168179+0000) 2024-02-20T02:46:28.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:28.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:28.168334+0000) 2024-02-20T02:46:29.165 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:29.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:29.168539+0000) 2024-02-20T02:46:30.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:30.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:30.168702+0000) 2024-02-20T02:46:31.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:31.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:31.168882+0000) 2024-02-20T02:46:32.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:32.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:32.169180+0000) 2024-02-20T02:46:33.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:33.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:33.169435+0000) 2024-02-20T02:46:34.166 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:34.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:34.169585+0000) 2024-02-20T02:46:35.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:35.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:35.169832+0000) 2024-02-20T02:46:36.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:36.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:36.169980+0000) 2024-02-20T02:46:37.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:37.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:37.170132+0000) 2024-02-20T02:46:38.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:38.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:38.170407+0000) 2024-02-20T02:46:39.167 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:39.166+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:39.170570+0000) 2024-02-20T02:46:40.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:40.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:40.170884+0000) 2024-02-20T02:46:41.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:41.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:41.171088+0000) 2024-02-20T02:46:42.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:42.170+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:42.171276+0000) 2024-02-20T02:46:43.168 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:43.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:43.171517+0000) 2024-02-20T02:46:44.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:44.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:44.171697+0000) 2024-02-20T02:46:45.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:45.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:45.171894+0000) 2024-02-20T02:46:46.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:46.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:46.172125+0000) 2024-02-20T02:46:47.169 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:47.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:47.172412+0000) 2024-02-20T02:46:48.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:48.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:48.172596+0000) 2024-02-20T02:46:49.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:49.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:49.172758+0000) 2024-02-20T02:46:50.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:50.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:50.173017+0000) 2024-02-20T02:46:51.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:51.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:51.173177+0000) 2024-02-20T02:46:52.170 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:52.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:52.173337+0000) 2024-02-20T02:46:53.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:53.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:53.173503+0000) 2024-02-20T02:46:54.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:54.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:54.173660+0000) 2024-02-20T02:46:55.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:55.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:55.173985+0000) 2024-02-20T02:46:56.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:56.169+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:56.174157+0000) 2024-02-20T02:46:57.171 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:57.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:57.174415+0000) 2024-02-20T02:46:58.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:58.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:58.174632+0000) 2024-02-20T02:46:59.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:46:59.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:46:59.174880+0000) 2024-02-20T02:47:00.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:00.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:00.175162+0000) 2024-02-20T02:47:01.172 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:01.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:01.175318+0000) 2024-02-20T02:47:02.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:02.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:02.175478+0000) 2024-02-20T02:47:03.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:03.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:03.175658+0000) 2024-02-20T02:47:04.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:04.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:04.175811+0000) 2024-02-20T02:47:05.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:05.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:05.176091+0000) 2024-02-20T02:47:06.173 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:06.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:06.176311+0000) 2024-02-20T02:47:07.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:07.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:07.176466+0000) 2024-02-20T02:47:08.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:08.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:08.176713+0000) 2024-02-20T02:47:09.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:09.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:09.176882+0000) 2024-02-20T02:47:10.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:10.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:10.177142+0000) 2024-02-20T02:47:11.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:11.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:11.177335+0000) 2024-02-20T02:47:12.174 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:12.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:12.177493+0000) 2024-02-20T02:47:13.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:13.173+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:13.177674+0000) 2024-02-20T02:47:14.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:14.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:14.177920+0000) 2024-02-20T02:47:15.175 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:15.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:15.178099+0000) 2024-02-20T02:47:16.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:16.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:16.178334+0000) 2024-02-20T02:47:17.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:17.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:17.178576+0000) 2024-02-20T02:47:18.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:18.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:18.178797+0000) 2024-02-20T02:47:19.176 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:19.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:19.179079+0000) 2024-02-20T02:47:20.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:20.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:20.179258+0000) 2024-02-20T02:47:21.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:21.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:21.179414+0000) 2024-02-20T02:47:22.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:22.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:22.179694+0000) 2024-02-20T02:47:23.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:23.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:23.179894+0000) 2024-02-20T02:47:24.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:24.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:24.180030+0000) 2024-02-20T02:47:25.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:25.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:25.180181+0000) 2024-02-20T02:47:26.177 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:26.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:26.180339+0000) 2024-02-20T02:47:27.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:27.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:27.180583+0000) 2024-02-20T02:47:28.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:28.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:28.180776+0000) 2024-02-20T02:47:29.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:29.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:29.180932+0000) 2024-02-20T02:47:30.178 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:30.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:30.181188+0000) 2024-02-20T02:47:31.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:31.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:31.181457+0000) 2024-02-20T02:47:32.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:32.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:32.181629+0000) 2024-02-20T02:47:33.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:33.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:33.181843+0000) 2024-02-20T02:47:34.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:34.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:34.182075+0000) 2024-02-20T02:47:35.179 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:35.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:35.182248+0000) 2024-02-20T02:47:36.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:36.177+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:36.182491+0000) 2024-02-20T02:47:37.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:37.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:37.182793+0000) 2024-02-20T02:47:38.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:38.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:38.182981+0000) 2024-02-20T02:47:39.180 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:39.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:39.183156+0000) 2024-02-20T02:47:40.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:40.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:40.183413+0000) 2024-02-20T02:47:41.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:41.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:41.183645+0000) 2024-02-20T02:47:42.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:42.181+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:42.183868+0000) 2024-02-20T02:47:43.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:43.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:43.184077+0000) 2024-02-20T02:47:44.181 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:44.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:44.184232+0000) 2024-02-20T02:47:45.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:45.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:45.184446+0000) 2024-02-20T02:47:46.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:46.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:46.184683+0000) 2024-02-20T02:47:47.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:47.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:47.184964+0000) 2024-02-20T02:47:48.182 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:48.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:48.185239+0000) 2024-02-20T02:47:49.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:49.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:49.185498+0000) 2024-02-20T02:47:50.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:50.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:50.185667+0000) 2024-02-20T02:47:51.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:51.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:51.185818+0000) 2024-02-20T02:47:52.183 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:52.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:52.185984+0000) 2024-02-20T02:47:53.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:53.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:53.186281+0000) 2024-02-20T02:47:54.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:54.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:54.186463+0000) 2024-02-20T02:47:55.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:55.180+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:55.186672+0000) 2024-02-20T02:47:56.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:56.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:56.186811+0000) 2024-02-20T02:47:57.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:57.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:57.186961+0000) 2024-02-20T02:47:58.184 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:58.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:58.187161+0000) 2024-02-20T02:47:59.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:47:59.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:47:59.187464+0000) 2024-02-20T02:48:00.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:00.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:00.187670+0000) 2024-02-20T02:48:01.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:01.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:01.187854+0000) 2024-02-20T02:48:02.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:02.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:02.188014+0000) 2024-02-20T02:48:03.185 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:03.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:03.188223+0000) 2024-02-20T02:48:04.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:04.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:04.188414+0000) 2024-02-20T02:48:05.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:05.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:05.188628+0000) 2024-02-20T02:48:06.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:06.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:06.188867+0000) 2024-02-20T02:48:07.186 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:07.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:07.189029+0000) 2024-02-20T02:48:08.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:08.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:08.189320+0000) 2024-02-20T02:48:09.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:09.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:09.189510+0000) 2024-02-20T02:48:10.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:10.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:10.189761+0000) 2024-02-20T02:48:11.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:11.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:11.189918+0000) 2024-02-20T02:48:12.187 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:12.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:12.190109+0000) 2024-02-20T02:48:13.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:13.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:13.190424+0000) 2024-02-20T02:48:14.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:14.184+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:14.190607+0000) 2024-02-20T02:48:15.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:15.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:15.190766+0000) 2024-02-20T02:48:16.188 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:16.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:16.190928+0000) 2024-02-20T02:48:17.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:17.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:17.191096+0000) 2024-02-20T02:48:18.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:18.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:18.191255+0000) 2024-02-20T02:48:19.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:19.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:19.191410+0000) 2024-02-20T02:48:20.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:20.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:20.191549+0000) 2024-02-20T02:48:21.189 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:21.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:21.191684+0000) 2024-02-20T02:48:22.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:22.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:22.191902+0000) 2024-02-20T02:48:23.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:23.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:23.192071+0000) 2024-02-20T02:48:24.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:24.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:24.192324+0000) 2024-02-20T02:48:25.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:25.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:25.192500+0000) 2024-02-20T02:48:26.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:26.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:26.192718+0000) 2024-02-20T02:48:27.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:27.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:27.192923+0000) 2024-02-20T02:48:28.190 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:28.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:28.193064+0000) 2024-02-20T02:48:29.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:29.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:29.193295+0000) 2024-02-20T02:48:30.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:30.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:30.193549+0000) 2024-02-20T02:48:31.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:31.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:31.193881+0000) 2024-02-20T02:48:32.191 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:32.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:32.194019+0000) 2024-02-20T02:48:33.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:33.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:33.194190+0000) 2024-02-20T02:48:34.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:34.188+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:34.194361+0000) 2024-02-20T02:48:35.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:35.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:35.194541+0000) 2024-02-20T02:48:36.192 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:36.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:36.194774+0000) 2024-02-20T02:48:37.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:37.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:37.195082+0000) 2024-02-20T02:48:38.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:38.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:38.195223+0000) 2024-02-20T02:48:39.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:39.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:39.195377+0000) 2024-02-20T02:48:40.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:40.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:40.195612+0000) 2024-02-20T02:48:41.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:41.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:41.195850+0000) 2024-02-20T02:48:42.193 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:42.192+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:42.196034+0000) 2024-02-20T02:48:43.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:43.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:43.196171+0000) 2024-02-20T02:48:44.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:44.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:44.196314+0000) 2024-02-20T02:48:45.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:45.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:45.196580+0000) 2024-02-20T02:48:46.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:46.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:46.196814+0000) 2024-02-20T02:48:47.194 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:47.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:47.196964+0000) 2024-02-20T02:48:48.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:48.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:48.197227+0000) 2024-02-20T02:48:49.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:49.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:49.197526+0000) 2024-02-20T02:48:50.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:50.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:50.197704+0000) 2024-02-20T02:48:51.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:51.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:51.197905+0000) 2024-02-20T02:48:52.195 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:52.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:52.198041+0000) 2024-02-20T02:48:53.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:53.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:53.198233+0000) 2024-02-20T02:48:54.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:54.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:54.198428+0000) 2024-02-20T02:48:55.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:55.191+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:55.198664+0000) 2024-02-20T02:48:56.196 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:56.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:56.198878+0000) 2024-02-20T02:48:57.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:57.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:57.199109+0000) 2024-02-20T02:48:58.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:58.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:58.199357+0000) 2024-02-20T02:48:59.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:48:59.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:48:59.199607+0000) 2024-02-20T02:49:00.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:00.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:00.199836+0000) 2024-02-20T02:49:01.197 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:01.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:01.199999+0000) 2024-02-20T02:49:02.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:02.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:02.200222+0000) 2024-02-20T02:49:03.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:03.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:03.200442+0000) 2024-02-20T02:49:04.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:04.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:04.200673+0000) 2024-02-20T02:49:05.198 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:05.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:05.200888+0000) 2024-02-20T02:49:06.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:06.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:06.201131+0000) 2024-02-20T02:49:07.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:07.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:07.201363+0000) 2024-02-20T02:49:08.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:08.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:08.201566+0000) 2024-02-20T02:49:09.199 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:09.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:09.201816+0000) 2024-02-20T02:49:10.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:10.195+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:10.201983+0000) 2024-02-20T02:49:11.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:11.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:11.202211+0000) 2024-02-20T02:49:12.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:12.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:12.202501+0000) 2024-02-20T02:49:13.200 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:13.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:13.202791+0000) 2024-02-20T02:49:14.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:14.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:14.202959+0000) 2024-02-20T02:49:15.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:15.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:15.203202+0000) 2024-02-20T02:49:16.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:16.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:16.203370+0000) 2024-02-20T02:49:17.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:17.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:17.203514+0000) 2024-02-20T02:49:18.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:18.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:18.203709+0000) 2024-02-20T02:49:19.201 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:19.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:19.203876+0000) 2024-02-20T02:49:20.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:20.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:20.204138+0000) 2024-02-20T02:49:21.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:21.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:21.204304+0000) 2024-02-20T02:49:22.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:22.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:22.204461+0000) 2024-02-20T02:49:23.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:23.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:23.204713+0000) 2024-02-20T02:49:24.202 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:24.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:24.204931+0000) 2024-02-20T02:49:25.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:25.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:25.205105+0000) 2024-02-20T02:49:26.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:26.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:26.205308+0000) 2024-02-20T02:49:27.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:27.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:27.205623+0000) 2024-02-20T02:49:28.203 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:28.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:28.205786+0000) 2024-02-20T02:49:29.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:29.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:29.205935+0000) 2024-02-20T02:49:30.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:30.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:30.206141+0000) 2024-02-20T02:49:31.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:31.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:31.206272+0000) 2024-02-20T02:49:32.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:32.199+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:32.206438+0000) 2024-02-20T02:49:33.204 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:33.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:33.206682+0000) 2024-02-20T02:49:34.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:34.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:34.206944+0000) 2024-02-20T02:49:35.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:35.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:35.207211+0000) 2024-02-20T02:49:36.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:36.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:36.207371+0000) 2024-02-20T02:49:37.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:37.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:37.207535+0000) 2024-02-20T02:49:38.205 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:38.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:38.207770+0000) 2024-02-20T02:49:39.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:39.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:39.207937+0000) 2024-02-20T02:49:40.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:40.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:40.208106+0000) 2024-02-20T02:49:41.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:41.207+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:41.208263+0000) 2024-02-20T02:49:42.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:42.203+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:42.208417+0000) 2024-02-20T02:49:43.206 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:43.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:43.208630+0000) 2024-02-20T02:49:44.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:44.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:44.208912+0000) 2024-02-20T02:49:45.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:45.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:45.209133+0000) 2024-02-20T02:49:46.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:46.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:46.209311+0000) 2024-02-20T02:49:47.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:47.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:47.209543+0000) 2024-02-20T02:49:48.207 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:48.202+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:48.209838+0000) 2024-02-20T02:49:49.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:49.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:49.210222+0000) 2024-02-20T02:49:50.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:50.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:50.210362+0000) 2024-02-20T02:49:51.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:51.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:51.210558+0000) 2024-02-20T02:49:52.208 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:52.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:52.210697+0000) 2024-02-20T02:49:53.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:53.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:53.210875+0000) 2024-02-20T02:49:54.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:54.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:54.211014+0000) 2024-02-20T02:49:55.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:55.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:55.211205+0000) 2024-02-20T02:49:56.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:56.206+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:56.211371+0000) 2024-02-20T02:49:57.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:57.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:57.211626+0000) 2024-02-20T02:49:58.209 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:58.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:58.211823+0000) 2024-02-20T02:49:59.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:49:59.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:49:59.212057+0000) 2024-02-20T02:50:00.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:00.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:00.212339+0000) 2024-02-20T02:50:01.210 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:01.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:01.212491+0000) 2024-02-20T02:50:02.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:02.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:02.212649+0000) 2024-02-20T02:50:03.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:03.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:03.212896+0000) 2024-02-20T02:50:04.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:04.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:04.213176+0000) 2024-02-20T02:50:05.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:05.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:05.213403+0000) 2024-02-20T02:50:06.211 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:06.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:06.213622+0000) 2024-02-20T02:50:07.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:07.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:07.213874+0000) 2024-02-20T02:50:08.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:08.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:08.214075+0000) 2024-02-20T02:50:09.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:09.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:09.214280+0000) 2024-02-20T02:50:10.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:10.210+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:10.214509+0000) 2024-02-20T02:50:11.212 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:11.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:11.214770+0000) 2024-02-20T02:50:12.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:12.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:12.214976+0000) 2024-02-20T02:50:13.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:13.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:13.215282+0000) 2024-02-20T02:50:14.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:14.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:14.215448+0000) 2024-02-20T02:50:15.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:15.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:15.215606+0000) 2024-02-20T02:50:16.213 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:16.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:16.215817+0000) 2024-02-20T02:50:17.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:17.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:17.216057+0000) 2024-02-20T02:50:18.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:18.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:18.216284+0000) 2024-02-20T02:50:19.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:19.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:19.216462+0000) 2024-02-20T02:50:20.214 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:20.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:20.216613+0000) 2024-02-20T02:50:21.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:21.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:21.216764+0000) 2024-02-20T02:50:22.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:22.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:22.216910+0000) 2024-02-20T02:50:23.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:23.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:23.217047+0000) 2024-02-20T02:50:24.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:24.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:24.217207+0000) 2024-02-20T02:50:25.215 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:25.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:25.217471+0000) 2024-02-20T02:50:26.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:26.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:26.217748+0000) 2024-02-20T02:50:27.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:27.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:27.217999+0000) 2024-02-20T02:50:28.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:28.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:28.218212+0000) 2024-02-20T02:50:29.216 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:29.214+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:29.218533+0000) 2024-02-20T02:50:30.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:30.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:30.218694+0000) 2024-02-20T02:50:31.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:31.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:31.218898+0000) 2024-02-20T02:50:32.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:32.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:32.219086+0000) 2024-02-20T02:50:33.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:33.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:33.219248+0000) 2024-02-20T02:50:34.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:34.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:34.219465+0000) 2024-02-20T02:50:35.217 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:35.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:35.219613+0000) 2024-02-20T02:50:36.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:36.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:36.219835+0000) 2024-02-20T02:50:37.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:37.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:37.219997+0000) 2024-02-20T02:50:38.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:38.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:38.220361+0000) 2024-02-20T02:50:39.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:39.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:39.220570+0000) 2024-02-20T02:50:40.218 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:40.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:40.220727+0000) 2024-02-20T02:50:41.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:41.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:41.220958+0000) 2024-02-20T02:50:42.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:42.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:42.221187+0000) 2024-02-20T02:50:43.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:43.218+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:43.221378+0000) 2024-02-20T02:50:44.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:44.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:44.221613+0000) 2024-02-20T02:50:45.219 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:45.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:45.221760+0000) 2024-02-20T02:50:46.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:46.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:46.222005+0000) 2024-02-20T02:50:47.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:47.217+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:47.222168+0000) 2024-02-20T02:50:48.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:48.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:48.222324+0000) 2024-02-20T02:50:49.220 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:49.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:49.222562+0000) 2024-02-20T02:50:50.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:50.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:50.222700+0000) 2024-02-20T02:50:51.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:51.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:51.223040+0000) 2024-02-20T02:50:52.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:52.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:52.223254+0000) 2024-02-20T02:50:53.221 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:53.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:53.223544+0000) 2024-02-20T02:50:54.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:54.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:54.223689+0000) 2024-02-20T02:50:55.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:55.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:55.223836+0000) 2024-02-20T02:50:56.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:56.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:56.224030+0000) 2024-02-20T02:50:57.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:57.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:57.224249+0000) 2024-02-20T02:50:58.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:58.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:58.224440+0000) 2024-02-20T02:50:59.222 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:50:59.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:50:59.224583+0000) 2024-02-20T02:51:00.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:00.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:00.224729+0000) 2024-02-20T02:51:01.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:01.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:01.224931+0000) 2024-02-20T02:51:02.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:02.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:02.225088+0000) 2024-02-20T02:51:03.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:03.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:03.225246+0000) 2024-02-20T02:51:04.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:04.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:04.225469+0000) 2024-02-20T02:51:05.223 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:05.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:05.225619+0000) 2024-02-20T02:51:06.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:06.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:06.225803+0000) 2024-02-20T02:51:07.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:07.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:07.226041+0000) 2024-02-20T02:51:08.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:08.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:08.226185+0000) 2024-02-20T02:51:09.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:09.221+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:09.226316+0000) 2024-02-20T02:51:10.224 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:10.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:10.226495+0000) 2024-02-20T02:51:11.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:11.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:11.226658+0000) 2024-02-20T02:51:12.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:12.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:12.226894+0000) 2024-02-20T02:51:13.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:13.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:13.227137+0000) 2024-02-20T02:51:14.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:14.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:14.227337+0000) 2024-02-20T02:51:15.225 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:15.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:15.227492+0000) 2024-02-20T02:51:16.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:16.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:16.227740+0000) 2024-02-20T02:51:17.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:17.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:17.227904+0000) 2024-02-20T02:51:18.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:18.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:18.228044+0000) 2024-02-20T02:51:19.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:19.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:19.228242+0000) 2024-02-20T02:51:20.226 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:20.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:20.228392+0000) 2024-02-20T02:51:21.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:21.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:21.228562+0000) 2024-02-20T02:51:22.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:22.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:22.228823+0000) 2024-02-20T02:51:23.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:23.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:23.229003+0000) 2024-02-20T02:51:24.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:24.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:24.229163+0000) 2024-02-20T02:51:25.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:25.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:25.229314+0000) 2024-02-20T02:51:26.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:26.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:26.229456+0000) 2024-02-20T02:51:27.227 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:27.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:27.229609+0000) 2024-02-20T02:51:28.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:28.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:28.229757+0000) 2024-02-20T02:51:29.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:29.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:29.229914+0000) 2024-02-20T02:51:30.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:30.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:30.230071+0000) 2024-02-20T02:51:31.228 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:31.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:31.230314+0000) 2024-02-20T02:51:32.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:32.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:32.230496+0000) 2024-02-20T02:51:33.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:33.225+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:33.230663+0000) 2024-02-20T02:51:34.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:34.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:34.230788+0000) 2024-02-20T02:51:35.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:35.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:35.230957+0000) 2024-02-20T02:51:36.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:36.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:36.231095+0000) 2024-02-20T02:51:37.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:37.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:37.231257+0000) 2024-02-20T02:51:38.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:38.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:38.231443+0000) 2024-02-20T02:51:39.229 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:39.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:39.231573+0000) 2024-02-20T02:51:40.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:40.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:40.231813+0000) 2024-02-20T02:51:41.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:41.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:41.232035+0000) 2024-02-20T02:51:42.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:42.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:42.232170+0000) 2024-02-20T02:51:43.230 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:43.229+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:43.232316+0000) 2024-02-20T02:51:44.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:44.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:44.232544+0000) 2024-02-20T02:51:45.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:45.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:45.232687+0000) 2024-02-20T02:51:46.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:46.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:46.232912+0000) 2024-02-20T02:51:47.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:47.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:47.233060+0000) 2024-02-20T02:51:48.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:48.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:48.233207+0000) 2024-02-20T02:51:49.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:49.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:49.233370+0000) 2024-02-20T02:51:50.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:50.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:50.233494+0000) 2024-02-20T02:51:51.231 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:51.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:51.233642+0000) 2024-02-20T02:51:52.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:52.228+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:52.233797+0000) 2024-02-20T02:51:53.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:53.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:53.233957+0000) 2024-02-20T02:51:54.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:54.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:54.234118+0000) 2024-02-20T02:51:55.232 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:55.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:55.234327+0000) 2024-02-20T02:51:56.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:56.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:56.234534+0000) 2024-02-20T02:51:57.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:57.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:57.234822+0000) 2024-02-20T02:51:58.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:58.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:58.235036+0000) 2024-02-20T02:51:59.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:51:59.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:51:59.235201+0000) 2024-02-20T02:52:00.233 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:00.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:00.235425+0000) 2024-02-20T02:52:01.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:01.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:01.235573+0000) 2024-02-20T02:52:02.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:02.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:02.235729+0000) 2024-02-20T02:52:03.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:03.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:03.235887+0000) 2024-02-20T02:52:04.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:04.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:04.236042+0000) 2024-02-20T02:52:05.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:05.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:05.236205+0000) 2024-02-20T02:52:06.234 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:06.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:06.236423+0000) 2024-02-20T02:52:07.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:07.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:07.236571+0000) 2024-02-20T02:52:08.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:08.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:08.236760+0000) 2024-02-20T02:52:09.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:09.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:09.236942+0000) 2024-02-20T02:52:10.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:10.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:10.237106+0000) 2024-02-20T02:52:11.235 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:11.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:11.237262+0000) 2024-02-20T02:52:12.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:12.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:12.237500+0000) 2024-02-20T02:52:13.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:13.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:13.237747+0000) 2024-02-20T02:52:14.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:14.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:14.237910+0000) 2024-02-20T02:52:15.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:15.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:15.238065+0000) 2024-02-20T02:52:16.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:16.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:16.238217+0000) 2024-02-20T02:52:17.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:17.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:17.238367+0000) 2024-02-20T02:52:18.236 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:18.232+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:18.238515+0000) 2024-02-20T02:52:19.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:19.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:19.238668+0000) 2024-02-20T02:52:20.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:20.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:20.238825+0000) 2024-02-20T02:52:21.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:21.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:21.238989+0000) 2024-02-20T02:52:22.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:22.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:22.239192+0000) 2024-02-20T02:52:23.237 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:23.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:23.239329+0000) 2024-02-20T02:52:24.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:24.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:24.239475+0000) 2024-02-20T02:52:25.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:25.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:25.239655+0000) 2024-02-20T02:52:26.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:26.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:26.239805+0000) 2024-02-20T02:52:27.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:27.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:27.239963+0000) 2024-02-20T02:52:28.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:28.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:28.240113+0000) 2024-02-20T02:52:29.238 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:29.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:29.240283+0000) 2024-02-20T02:52:30.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:30.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:30.240437+0000) 2024-02-20T02:52:31.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:31.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:31.240592+0000) 2024-02-20T02:52:32.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:32.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:32.240778+0000) 2024-02-20T02:52:33.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:33.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:33.240978+0000) 2024-02-20T02:52:34.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:34.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:34.241212+0000) 2024-02-20T02:52:35.239 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:35.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:35.241375+0000) 2024-02-20T02:52:36.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:36.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:36.241534+0000) 2024-02-20T02:52:37.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:37.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:37.241694+0000) 2024-02-20T02:52:38.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:38.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:38.241920+0000) 2024-02-20T02:52:39.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:39.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:39.242066+0000) 2024-02-20T02:52:40.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:40.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:40.242227+0000) 2024-02-20T02:52:41.240 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:41.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:41.242369+0000) 2024-02-20T02:52:42.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:42.236+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:42.242608+0000) 2024-02-20T02:52:43.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:43.240+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:43.242782+0000) 2024-02-20T02:52:44.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:44.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:44.242933+0000) 2024-02-20T02:52:45.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:45.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:45.243085+0000) 2024-02-20T02:52:46.241 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:46.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:46.243253+0000) 2024-02-20T02:52:47.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:47.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:47.243403+0000) 2024-02-20T02:52:48.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:48.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:48.243536+0000) 2024-02-20T02:52:49.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:49.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:49.243695+0000) 2024-02-20T02:52:50.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:50.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:50.243829+0000) 2024-02-20T02:52:51.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:51.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:51.244001+0000) 2024-02-20T02:52:52.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:52.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:52.244139+0000) 2024-02-20T02:52:53.242 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:53.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:53.244337+0000) 2024-02-20T02:52:54.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:54.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:54.244494+0000) 2024-02-20T02:52:55.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:55.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:55.244637+0000) 2024-02-20T02:52:56.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:56.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:56.244866+0000) 2024-02-20T02:52:57.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:57.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:57.245132+0000) 2024-02-20T02:52:58.243 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:58.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:58.245291+0000) 2024-02-20T02:52:59.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:52:59.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:52:59.245445+0000) 2024-02-20T02:53:00.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:00.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:00.245592+0000) 2024-02-20T02:53:01.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:01.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:01.245747+0000) 2024-02-20T02:53:02.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:02.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:02.245904+0000) 2024-02-20T02:53:03.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:03.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:03.246072+0000) 2024-02-20T02:53:04.244 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:04.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:04.246219+0000) 2024-02-20T02:53:05.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:05.239+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:05.246481+0000) 2024-02-20T02:53:06.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:06.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:06.246665+0000) 2024-02-20T02:53:07.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:07.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:07.246801+0000) 2024-02-20T02:53:08.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:08.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:08.246949+0000) 2024-02-20T02:53:09.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:09.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:09.247149+0000) 2024-02-20T02:53:10.245 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:10.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:10.247387+0000) 2024-02-20T02:53:11.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:11.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:11.247535+0000) 2024-02-20T02:53:12.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:12.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:12.247717+0000) 2024-02-20T02:53:13.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:13.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:13.247877+0000) 2024-02-20T02:53:14.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:14.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:14.248033+0000) 2024-02-20T02:53:15.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:15.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:15.248178+0000) 2024-02-20T02:53:16.246 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:16.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:16.248323+0000) 2024-02-20T02:53:17.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:17.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:17.248471+0000) 2024-02-20T02:53:18.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:18.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:18.248621+0000) 2024-02-20T02:53:19.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:19.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:19.248797+0000) 2024-02-20T02:53:20.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:20.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:20.248971+0000) 2024-02-20T02:53:21.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:21.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:21.249131+0000) 2024-02-20T02:53:22.247 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:22.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:22.249271+0000) 2024-02-20T02:53:23.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:23.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:23.249498+0000) 2024-02-20T02:53:24.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:24.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:24.249650+0000) 2024-02-20T02:53:25.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:25.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:25.249818+0000) 2024-02-20T02:53:26.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:26.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:26.249966+0000) 2024-02-20T02:53:27.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:27.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:27.250115+0000) 2024-02-20T02:53:28.248 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:28.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:28.250266+0000) 2024-02-20T02:53:29.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:29.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:29.250432+0000) 2024-02-20T02:53:30.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:30.243+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:30.250599+0000) 2024-02-20T02:53:31.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:31.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:31.250788+0000) 2024-02-20T02:53:32.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:32.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:32.251002+0000) 2024-02-20T02:53:33.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:33.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:33.251268+0000) 2024-02-20T02:53:34.249 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:34.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:34.251454+0000) 2024-02-20T02:53:35.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:35.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:35.251619+0000) 2024-02-20T02:53:36.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:36.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:36.251766+0000) 2024-02-20T02:53:37.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:37.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:37.251944+0000) 2024-02-20T02:53:38.250 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:38.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:38.252087+0000) 2024-02-20T02:53:39.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:39.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:39.252247+0000) 2024-02-20T02:53:40.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:40.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:40.252408+0000) 2024-02-20T02:53:41.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:41.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:41.252555+0000) 2024-02-20T02:53:42.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:42.251+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:42.252754+0000) 2024-02-20T02:53:43.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:43.247+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:43.252956+0000) 2024-02-20T02:53:44.251 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:44.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:44.253219+0000) 2024-02-20T02:53:45.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:45.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:45.253379+0000) 2024-02-20T02:53:46.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:46.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:46.253527+0000) 2024-02-20T02:53:47.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:47.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:47.253675+0000) 2024-02-20T02:53:48.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:48.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:48.253817+0000) 2024-02-20T02:53:49.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:49.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:49.253953+0000) 2024-02-20T02:53:50.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:50.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:50.254151+0000) 2024-02-20T02:53:51.252 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:51.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:51.254311+0000) 2024-02-20T02:53:52.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:52.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:52.254447+0000) 2024-02-20T02:53:53.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:53.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:53.254560+0000) 2024-02-20T02:53:54.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:54.246+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:54.254692+0000) 2024-02-20T02:53:55.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:55.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:55.254873+0000) 2024-02-20T02:53:56.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:56.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:56.255084+0000) 2024-02-20T02:53:57.253 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:57.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:57.255242+0000) 2024-02-20T02:53:58.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:58.250+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:58.255386+0000) 2024-02-20T02:53:59.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:53:59.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:53:59.255541+0000) 2024-02-20T02:54:00.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:00.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:00.255688+0000) 2024-02-20T02:54:01.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:01.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:01.255841+0000) 2024-02-20T02:54:02.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:02.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:02.255999+0000) 2024-02-20T02:54:03.254 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:03.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:03.256153+0000) 2024-02-20T02:54:04.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:04.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:04.256295+0000) 2024-02-20T02:54:05.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:05.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:05.256459+0000) 2024-02-20T02:54:06.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:06.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:06.256617+0000) 2024-02-20T02:54:07.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:07.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:07.256772+0000) 2024-02-20T02:54:08.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:08.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:08.256929+0000) 2024-02-20T02:54:09.255 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:09.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:09.257054+0000) 2024-02-20T02:54:10.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:10.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:10.257198+0000) 2024-02-20T02:54:11.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:11.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:11.257335+0000) 2024-02-20T02:54:12.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:12.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:12.257457+0000) 2024-02-20T02:54:13.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:13.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:13.257599+0000) 2024-02-20T02:54:14.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:14.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:14.257725+0000) 2024-02-20T02:54:15.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:15.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:15.257881+0000) 2024-02-20T02:54:16.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:16.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:16.258043+0000) 2024-02-20T02:54:17.256 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:17.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:17.258192+0000) 2024-02-20T02:54:18.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:18.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:18.258363+0000) 2024-02-20T02:54:19.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:19.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:19.258516+0000) 2024-02-20T02:54:20.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:20.254+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:20.258645+0000) 2024-02-20T02:54:21.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:21.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:21.258771+0000) 2024-02-20T02:54:22.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:22.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:22.258901+0000) 2024-02-20T02:54:23.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:23.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:23.259074+0000) 2024-02-20T02:54:24.257 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:24.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:24.259199+0000) 2024-02-20T02:54:25.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:25.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:25.259342+0000) 2024-02-20T02:54:26.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:26.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:26.259480+0000) 2024-02-20T02:54:27.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:27.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:27.259623+0000) 2024-02-20T02:54:28.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:28.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:28.259757+0000) 2024-02-20T02:54:29.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:29.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:29.259887+0000) 2024-02-20T02:54:30.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:30.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:30.260010+0000) 2024-02-20T02:54:31.258 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:31.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:31.260144+0000) 2024-02-20T02:54:32.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:32.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:32.260275+0000) 2024-02-20T02:54:33.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:33.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:33.260405+0000) 2024-02-20T02:54:34.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:34.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:34.260543+0000) 2024-02-20T02:54:35.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:35.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:35.260692+0000) 2024-02-20T02:54:36.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:36.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:36.260819+0000) 2024-02-20T02:54:37.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:37.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:37.260942+0000) 2024-02-20T02:54:38.259 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:38.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:38.261087+0000) 2024-02-20T02:54:39.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:39.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:39.261254+0000) 2024-02-20T02:54:40.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:40.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:40.261433+0000) 2024-02-20T02:54:41.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:41.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:41.261583+0000) 2024-02-20T02:54:42.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:42.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:42.261740+0000) 2024-02-20T02:54:43.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:43.258+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:43.261901+0000) 2024-02-20T02:54:44.260 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:44.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:44.262050+0000) 2024-02-20T02:54:45.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:45.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:45.262213+0000) 2024-02-20T02:54:46.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:46.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:46.262370+0000) 2024-02-20T02:54:47.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:47.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:47.262528+0000) 2024-02-20T02:54:48.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:48.257+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:48.262684+0000) 2024-02-20T02:54:49.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:49.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:49.262850+0000) 2024-02-20T02:54:50.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:50.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:50.262981+0000) 2024-02-20T02:54:51.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:51.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:51.263133+0000) 2024-02-20T02:54:52.261 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:52.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:52.263279+0000) 2024-02-20T02:54:53.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:53.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:53.263436+0000) 2024-02-20T02:54:54.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:54.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:54.263576+0000) 2024-02-20T02:54:55.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:55.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:55.263727+0000) 2024-02-20T02:54:56.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:56.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:56.263873+0000) 2024-02-20T02:54:57.262 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:57.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:57.264025+0000) 2024-02-20T02:54:58.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:58.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:58.264163+0000) 2024-02-20T02:54:59.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:54:59.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:54:59.264356+0000) 2024-02-20T02:55:00.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:00.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:00.264506+0000) 2024-02-20T02:55:01.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:01.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:01.264663+0000) 2024-02-20T02:55:02.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:02.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:02.264814+0000) 2024-02-20T02:55:03.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:03.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:03.264976+0000) 2024-02-20T02:55:04.263 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:04.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:04.265147+0000) 2024-02-20T02:55:05.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:05.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:05.265305+0000) 2024-02-20T02:55:06.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:06.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:06.265459+0000) 2024-02-20T02:55:07.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:07.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:07.265630+0000) 2024-02-20T02:55:08.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:08.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:08.265775+0000) 2024-02-20T02:55:09.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:09.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:09.265937+0000) 2024-02-20T02:55:10.264 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:10.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:10.266082+0000) 2024-02-20T02:55:11.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:11.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:11.266318+0000) 2024-02-20T02:55:12.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:12.261+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:12.266482+0000) 2024-02-20T02:55:13.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:13.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:13.266810+0000) 2024-02-20T02:55:14.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:14.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:14.266958+0000) 2024-02-20T02:55:15.265 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:15.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:15.267107+0000) 2024-02-20T02:55:16.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:16.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:16.267278+0000) 2024-02-20T02:55:17.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:17.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:17.267466+0000) 2024-02-20T02:55:18.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:18.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:18.267595+0000) 2024-02-20T02:55:19.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:19.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:19.267747+0000) 2024-02-20T02:55:20.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:20.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:20.267876+0000) 2024-02-20T02:55:21.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:21.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:21.268009+0000) 2024-02-20T02:55:22.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:22.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:22.268140+0000) 2024-02-20T02:55:23.266 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:23.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:23.268296+0000) 2024-02-20T02:55:24.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:24.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:24.268443+0000) 2024-02-20T02:55:25.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:25.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:25.268589+0000) 2024-02-20T02:55:26.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:26.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:26.268742+0000) 2024-02-20T02:55:27.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:27.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:27.268879+0000) 2024-02-20T02:55:28.267 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:28.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:28.269071+0000) 2024-02-20T02:55:29.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:29.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:29.269218+0000) 2024-02-20T02:55:30.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:30.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:30.269360+0000) 2024-02-20T02:55:31.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:31.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:31.269550+0000) 2024-02-20T02:55:32.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:32.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:32.269691+0000) 2024-02-20T02:55:33.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:33.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:33.269915+0000) 2024-02-20T02:55:34.268 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:34.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:34.270098+0000) 2024-02-20T02:55:35.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:35.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:35.270361+0000) 2024-02-20T02:55:36.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:36.265+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:36.270522+0000) 2024-02-20T02:55:37.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:37.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:37.270681+0000) 2024-02-20T02:55:38.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:38.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:38.270802+0000) 2024-02-20T02:55:39.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:39.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:39.270972+0000) 2024-02-20T02:55:40.269 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:40.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:40.271116+0000) 2024-02-20T02:55:41.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:41.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:41.271277+0000) 2024-02-20T02:55:42.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:42.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:42.271426+0000) 2024-02-20T02:55:43.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:43.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:43.271574+0000) 2024-02-20T02:55:44.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:44.269+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:44.271713+0000) 2024-02-20T02:55:45.270 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:45.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:45.272022+0000) 2024-02-20T02:55:46.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:46.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:46.272166+0000) 2024-02-20T02:55:47.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:47.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:47.272319+0000) 2024-02-20T02:55:48.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:48.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:48.272456+0000) 2024-02-20T02:55:49.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:49.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:49.272614+0000) 2024-02-20T02:55:50.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:50.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:50.272797+0000) 2024-02-20T02:55:51.271 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:51.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:51.272950+0000) 2024-02-20T02:55:52.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:52.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:52.273089+0000) 2024-02-20T02:55:53.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:53.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:53.273278+0000) 2024-02-20T02:55:54.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:54.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:54.273426+0000) 2024-02-20T02:55:55.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:55.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:55.273572+0000) 2024-02-20T02:55:56.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:56.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:56.273721+0000) 2024-02-20T02:55:57.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:57.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:57.273879+0000) 2024-02-20T02:55:58.272 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:58.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:58.274028+0000) 2024-02-20T02:55:59.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:55:59.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:55:59.274180+0000) 2024-02-20T02:56:00.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:00.268+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:00.274316+0000) 2024-02-20T02:56:01.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:01.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:01.274476+0000) 2024-02-20T02:56:02.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:02.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:02.274680+0000) 2024-02-20T02:56:03.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:03.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:03.274828+0000) 2024-02-20T02:56:04.273 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:04.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:04.274951+0000) 2024-02-20T02:56:05.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:05.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:05.275216+0000) 2024-02-20T02:56:06.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:06.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:06.275484+0000) 2024-02-20T02:56:07.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:07.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:07.275658+0000) 2024-02-20T02:56:08.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:08.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:08.275813+0000) 2024-02-20T02:56:09.274 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:09.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:09.275966+0000) 2024-02-20T02:56:10.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:10.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:10.276152+0000) 2024-02-20T02:56:11.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:11.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:11.276401+0000) 2024-02-20T02:56:12.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:12.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:12.276583+0000) 2024-02-20T02:56:13.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:13.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:13.276721+0000) 2024-02-20T02:56:14.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:14.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:14.276864+0000) 2024-02-20T02:56:15.275 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:15.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:15.277018+0000) 2024-02-20T02:56:16.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:16.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:16.277179+0000) 2024-02-20T02:56:17.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:17.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:17.277354+0000) 2024-02-20T02:56:18.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:18.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:18.277494+0000) 2024-02-20T02:56:19.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:19.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:19.277651+0000) 2024-02-20T02:56:20.276 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:20.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:20.277808+0000) 2024-02-20T02:56:21.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:21.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:21.277949+0000) 2024-02-20T02:56:22.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:22.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:22.278144+0000) 2024-02-20T02:56:23.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:23.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:23.278288+0000) 2024-02-20T02:56:24.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:24.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:24.278497+0000) 2024-02-20T02:56:25.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:25.272+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:25.278647+0000) 2024-02-20T02:56:26.277 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:26.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:26.278784+0000) 2024-02-20T02:56:27.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:27.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:27.279098+0000) 2024-02-20T02:56:28.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:28.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:28.279248+0000) 2024-02-20T02:56:29.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:29.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:29.279447+0000) 2024-02-20T02:56:30.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:30.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:30.279602+0000) 2024-02-20T02:56:31.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:31.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:31.279746+0000) 2024-02-20T02:56:32.278 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:32.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:32.279880+0000) 2024-02-20T02:56:33.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:33.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:33.280035+0000) 2024-02-20T02:56:34.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:34.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:34.280182+0000) 2024-02-20T02:56:35.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:35.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:35.280329+0000) 2024-02-20T02:56:36.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:36.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:36.280463+0000) 2024-02-20T02:56:37.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:37.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:37.280623+0000) 2024-02-20T02:56:38.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:38.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:38.280784+0000) 2024-02-20T02:56:39.279 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:39.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:39.280941+0000) 2024-02-20T02:56:40.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:40.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:40.281071+0000) 2024-02-20T02:56:41.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:41.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:41.281213+0000) 2024-02-20T02:56:42.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:42.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:42.281463+0000) 2024-02-20T02:56:43.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:43.280+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:43.281651+0000) 2024-02-20T02:56:44.280 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:44.276+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:44.281818+0000) 2024-02-20T02:56:45.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:45.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:45.282058+0000) 2024-02-20T02:56:46.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:46.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:46.282231+0000) 2024-02-20T02:56:47.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:47.275+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:47.282391+0000) 2024-02-20T02:56:48.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:48.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:48.282548+0000) 2024-02-20T02:56:49.281 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:49.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:49.282739+0000) 2024-02-20T02:56:50.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:50.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:50.282958+0000) 2024-02-20T02:56:51.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:51.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:51.283153+0000) 2024-02-20T02:56:52.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:52.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:52.283369+0000) 2024-02-20T02:56:53.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:53.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:53.283526+0000) 2024-02-20T02:56:54.282 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:54.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:54.283729+0000) 2024-02-20T02:56:55.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:55.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:55.283991+0000) 2024-02-20T02:56:56.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:56.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:56.284196+0000) 2024-02-20T02:56:57.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:57.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:57.284373+0000) 2024-02-20T02:56:58.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:58.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:58.284571+0000) 2024-02-20T02:56:59.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:56:59.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:56:59.284723+0000) 2024-02-20T02:57:00.283 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:00.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:00.284878+0000) 2024-02-20T02:57:01.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:01.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:01.285034+0000) 2024-02-20T02:57:02.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:02.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:02.285178+0000) 2024-02-20T02:57:03.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:03.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:03.285338+0000) 2024-02-20T02:57:04.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:04.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:04.285518+0000) 2024-02-20T02:57:05.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:05.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:05.285718+0000) 2024-02-20T02:57:06.284 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:06.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:06.285912+0000) 2024-02-20T02:57:07.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:07.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:07.286070+0000) 2024-02-20T02:57:08.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:08.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:08.286221+0000) 2024-02-20T02:57:09.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:09.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:09.286373+0000) 2024-02-20T02:57:10.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:10.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:10.286526+0000) 2024-02-20T02:57:11.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:11.279+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:11.286680+0000) 2024-02-20T02:57:12.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:12.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:12.286772+0000) 2024-02-20T02:57:13.285 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:13.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:13.286927+0000) 2024-02-20T02:57:14.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:14.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:14.287059+0000) 2024-02-20T02:57:15.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:15.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:15.287312+0000) 2024-02-20T02:57:16.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:16.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:16.287455+0000) 2024-02-20T02:57:17.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:17.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:17.287610+0000) 2024-02-20T02:57:18.286 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:18.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:18.287875+0000) 2024-02-20T02:57:19.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:19.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:19.288071+0000) 2024-02-20T02:57:20.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:20.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:20.288217+0000) 2024-02-20T02:57:21.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:21.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:21.288367+0000) 2024-02-20T02:57:22.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:22.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:22.288538+0000) 2024-02-20T02:57:23.287 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:23.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:23.288737+0000) 2024-02-20T02:57:24.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:24.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:24.288974+0000) 2024-02-20T02:57:25.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:25.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:25.289197+0000) 2024-02-20T02:57:26.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:26.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:26.289376+0000) 2024-02-20T02:57:27.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:27.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:27.289563+0000) 2024-02-20T02:57:28.288 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:28.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:28.289763+0000) 2024-02-20T02:57:29.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:29.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:29.289973+0000) 2024-02-20T02:57:30.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:30.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:30.290260+0000) 2024-02-20T02:57:31.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:31.283+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:31.290483+0000) 2024-02-20T02:57:32.289 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:32.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:32.290805+0000) 2024-02-20T02:57:33.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:33.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:33.291018+0000) 2024-02-20T02:57:34.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:34.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:34.291170+0000) 2024-02-20T02:57:35.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:35.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:35.291446+0000) 2024-02-20T02:57:36.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:36.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:36.291636+0000) 2024-02-20T02:57:37.290 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:37.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:37.291864+0000) 2024-02-20T02:57:38.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:38.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:38.292045+0000) 2024-02-20T02:57:39.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:39.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:39.292240+0000) 2024-02-20T02:57:40.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:40.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:40.292462+0000) 2024-02-20T02:57:41.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:41.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:41.292667+0000) 2024-02-20T02:57:42.291 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:42.287+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:42.292901+0000) 2024-02-20T02:57:43.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:43.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:43.293110+0000) 2024-02-20T02:57:44.292 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:44.291+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:44.293407+0000) 2024-02-20T02:57:45.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:45.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:45.293675+0000) 2024-02-20T02:57:46.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:46.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:46.293993+0000) 2024-02-20T02:57:47.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:47.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:47.294240+0000) 2024-02-20T02:57:48.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:48.286+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:48.294453+0000) 2024-02-20T02:57:49.293 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:49.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:49.294608+0000) 2024-02-20T02:57:50.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:50.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:50.294851+0000) 2024-02-20T02:57:51.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:51.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:51.295192+0000) 2024-02-20T02:57:52.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:52.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:52.295365+0000) 2024-02-20T02:57:53.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:53.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:53.295651+0000) 2024-02-20T02:57:54.294 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:54.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:54.295837+0000) 2024-02-20T02:57:55.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:55.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:55.296120+0000) 2024-02-20T02:57:56.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:56.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:56.296293+0000) 2024-02-20T02:57:57.295 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:57.290+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:57.296571+0000) 2024-02-20T02:57:58.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:58.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:58.296814+0000) 2024-02-20T02:57:59.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:57:59.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:57:59.297094+0000) 2024-02-20T02:58:00.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:00.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:00.297306+0000) 2024-02-20T02:58:01.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:01.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:01.297649+0000) 2024-02-20T02:58:02.296 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:02.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:02.297784+0000) 2024-02-20T02:58:03.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:03.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:03.297936+0000) 2024-02-20T02:58:04.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:04.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:04.298198+0000) 2024-02-20T02:58:05.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:05.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:05.298368+0000) 2024-02-20T02:58:06.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:06.294+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:06.298565+0000) 2024-02-20T02:58:07.297 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:07.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:07.298796+0000) 2024-02-20T02:58:08.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:08.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:08.298915+0000) 2024-02-20T02:58:09.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:09.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:09.299156+0000) 2024-02-20T02:58:10.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:10.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:10.299318+0000) 2024-02-20T02:58:11.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:11.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:11.299454+0000) 2024-02-20T02:58:12.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:12.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:12.299723+0000) 2024-02-20T02:58:13.298 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:13.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:13.299932+0000) 2024-02-20T02:58:14.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:14.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:14.300183+0000) 2024-02-20T02:58:15.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:15.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:15.300423+0000) 2024-02-20T02:58:16.299 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:16.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:16.300722+0000) 2024-02-20T02:58:17.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:17.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:17.301010+0000) 2024-02-20T02:58:18.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:18.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:18.301193+0000) 2024-02-20T02:58:19.300 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:19.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:19.301352+0000) 2024-02-20T02:58:20.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:20.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:20.301608+0000) 2024-02-20T02:58:21.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:21.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:21.301862+0000) 2024-02-20T02:58:22.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:22.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:22.302118+0000) 2024-02-20T02:58:23.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:23.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:23.302317+0000) 2024-02-20T02:58:24.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:24.298+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:24.302487+0000) 2024-02-20T02:58:25.301 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:25.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:25.302791+0000) 2024-02-20T02:58:26.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:26.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:26.302941+0000) 2024-02-20T02:58:27.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:27.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:27.303183+0000) 2024-02-20T02:58:28.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:28.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:28.303339+0000) 2024-02-20T02:58:29.302 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:29.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:29.303643+0000) 2024-02-20T02:58:30.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:30.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:30.303835+0000) 2024-02-20T02:58:31.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:31.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:31.304016+0000) 2024-02-20T02:58:32.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:32.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:32.304268+0000) 2024-02-20T02:58:33.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:33.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:33.304515+0000) 2024-02-20T02:58:34.303 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:34.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:34.304740+0000) 2024-02-20T02:58:35.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:35.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:35.304998+0000) 2024-02-20T02:58:36.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:36.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:36.305165+0000) 2024-02-20T02:58:37.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:37.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:37.305380+0000) 2024-02-20T02:58:38.304 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:38.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:38.305579+0000) 2024-02-20T02:58:39.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:39.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:39.305778+0000) 2024-02-20T02:58:40.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:40.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:40.305949+0000) 2024-02-20T02:58:41.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:41.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:41.306218+0000) 2024-02-20T02:58:42.305 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:42.302+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:42.306459+0000) 2024-02-20T02:58:43.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:43.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:43.306768+0000) 2024-02-20T02:58:44.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:44.306+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:44.307084+0000) 2024-02-20T02:58:45.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:45.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:45.307313+0000) 2024-02-20T02:58:46.306 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:46.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:46.307580+0000) 2024-02-20T02:58:47.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:47.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:47.307870+0000) 2024-02-20T02:58:48.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:48.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:48.308110+0000) 2024-02-20T02:58:49.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:49.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:49.308399+0000) 2024-02-20T02:58:50.307 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:50.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:50.308567+0000) 2024-02-20T02:58:51.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:51.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:51.308793+0000) 2024-02-20T02:58:52.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:52.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:52.309055+0000) 2024-02-20T02:58:53.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:53.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:53.309248+0000) 2024-02-20T02:58:54.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:54.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:54.309391+0000) 2024-02-20T02:58:55.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:55.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:55.309627+0000) 2024-02-20T02:58:56.308 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:56.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:56.309773+0000) 2024-02-20T02:58:57.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:57.305+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:57.309963+0000) 2024-02-20T02:58:58.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:58.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:58.310209+0000) 2024-02-20T02:58:59.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:58:59.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:58:59.310483+0000) 2024-02-20T02:59:00.309 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:00.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:00.310690+0000) 2024-02-20T02:59:01.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:01.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:01.310952+0000) 2024-02-20T02:59:02.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:02.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:02.311127+0000) 2024-02-20T02:59:03.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:03.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:03.311370+0000) 2024-02-20T02:59:04.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:04.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:04.311685+0000) 2024-02-20T02:59:05.310 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:05.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:05.311829+0000) 2024-02-20T02:59:06.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:06.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:06.311964+0000) 2024-02-20T02:59:07.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:07.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:07.312263+0000) 2024-02-20T02:59:08.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:08.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:08.312410+0000) 2024-02-20T02:59:09.311 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:09.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:09.312599+0000) 2024-02-20T02:59:10.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:10.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:10.312855+0000) 2024-02-20T02:59:11.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:11.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:11.313005+0000) 2024-02-20T02:59:12.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:12.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:12.313251+0000) 2024-02-20T02:59:13.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:13.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:13.313425+0000) 2024-02-20T02:59:14.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:14.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:14.313579+0000) 2024-02-20T02:59:15.312 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:15.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:15.313758+0000) 2024-02-20T02:59:16.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:16.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:16.313933+0000) 2024-02-20T02:59:17.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:17.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:17.314133+0000) 2024-02-20T02:59:18.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:18.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:18.314281+0000) 2024-02-20T02:59:19.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:19.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:19.314440+0000) 2024-02-20T02:59:20.313 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:20.309+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:20.314593+0000) 2024-02-20T02:59:21.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:21.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:21.314860+0000) 2024-02-20T02:59:22.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:22.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:22.314990+0000) 2024-02-20T02:59:23.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:23.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:23.315121+0000) 2024-02-20T02:59:24.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:24.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:24.315240+0000) 2024-02-20T02:59:25.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:25.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:25.315366+0000) 2024-02-20T02:59:26.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:26.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:26.315511+0000) 2024-02-20T02:59:27.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:27.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:27.315666+0000) 2024-02-20T02:59:28.314 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:28.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:28.315797+0000) 2024-02-20T02:59:29.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:29.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:29.315956+0000) 2024-02-20T02:59:30.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:30.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:30.316079+0000) 2024-02-20T02:59:31.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:31.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:31.316220+0000) 2024-02-20T02:59:32.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:32.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:32.316342+0000) 2024-02-20T02:59:33.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:33.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:33.316496+0000) 2024-02-20T02:59:34.315 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:34.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:34.316629+0000) 2024-02-20T02:59:35.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:35.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:35.316759+0000) 2024-02-20T02:59:36.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:36.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:36.316938+0000) 2024-02-20T02:59:37.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:37.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:37.317102+0000) 2024-02-20T02:59:38.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:38.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:38.317250+0000) 2024-02-20T02:59:39.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:39.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:39.317381+0000) 2024-02-20T02:59:40.316 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:40.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:40.317518+0000) 2024-02-20T02:59:41.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:41.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:41.317694+0000) 2024-02-20T02:59:42.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:42.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:42.317837+0000) 2024-02-20T02:59:43.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:43.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:43.317993+0000) 2024-02-20T02:59:44.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:44.313+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:44.318141+0000) 2024-02-20T02:59:45.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:45.312+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:45.318342+0000) 2024-02-20T02:59:46.317 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:46.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:46.318497+0000) 2024-02-20T02:59:47.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:47.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:47.318694+0000) 2024-02-20T02:59:48.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:48.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:48.318897+0000) 2024-02-20T02:59:49.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:49.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:49.319100+0000) 2024-02-20T02:59:50.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:50.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:50.319262+0000) 2024-02-20T02:59:51.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:51.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:51.319485+0000) 2024-02-20T02:59:52.318 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:52.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:52.319628+0000) 2024-02-20T02:59:53.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:53.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:53.319875+0000) 2024-02-20T02:59:54.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:54.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:54.320075+0000) 2024-02-20T02:59:55.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:55.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:55.320227+0000) 2024-02-20T02:59:56.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:56.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:56.320456+0000) 2024-02-20T02:59:57.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:57.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:57.320637+0000) 2024-02-20T02:59:58.319 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:58.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:58.320775+0000) 2024-02-20T02:59:59.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T02:59:59.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T01:59:59.320970+0000) 2024-02-20T03:00:00.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:00.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:00.321162+0000) 2024-02-20T03:00:01.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:01.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:01.321321+0000) 2024-02-20T03:00:02.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:02.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:02.321479+0000) 2024-02-20T03:00:03.320 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:03.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:03.321637+0000) 2024-02-20T03:00:04.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:04.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:04.321807+0000) 2024-02-20T03:00:05.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:05.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:05.322056+0000) 2024-02-20T03:00:06.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:06.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:06.322243+0000) 2024-02-20T03:00:07.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:07.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:07.322406+0000) 2024-02-20T03:00:08.321 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:08.316+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:08.322574+0000) 2024-02-20T03:00:09.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:09.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:09.322849+0000) 2024-02-20T03:00:10.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:10.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:10.323047+0000) 2024-02-20T03:00:11.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:11.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:11.323223+0000) 2024-02-20T03:00:12.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:12.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:12.323435+0000) 2024-02-20T03:00:13.322 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:13.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:13.323668+0000) 2024-02-20T03:00:14.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:14.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:14.323808+0000) 2024-02-20T03:00:15.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:15.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:15.324104+0000) 2024-02-20T03:00:16.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:16.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:16.324279+0000) 2024-02-20T03:00:17.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:17.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:17.324422+0000) 2024-02-20T03:00:18.323 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:18.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:18.324573+0000) 2024-02-20T03:00:19.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:19.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:19.324772+0000) 2024-02-20T03:00:20.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:20.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:20.324995+0000) 2024-02-20T03:00:21.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:21.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:21.325205+0000) 2024-02-20T03:00:22.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:22.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:22.325400+0000) 2024-02-20T03:00:23.324 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:23.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:23.325574+0000) 2024-02-20T03:00:24.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:24.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:24.325831+0000) 2024-02-20T03:00:25.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:25.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:25.326002+0000) 2024-02-20T03:00:26.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:26.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:26.326137+0000) 2024-02-20T03:00:27.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:27.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:27.326319+0000) 2024-02-20T03:00:28.325 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:28.320+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:28.326489+0000) 2024-02-20T03:00:29.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:29.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:29.326862+0000) 2024-02-20T03:00:30.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:30.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:30.327036+0000) 2024-02-20T03:00:31.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:31.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:31.327172+0000) 2024-02-20T03:00:32.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:32.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:32.327300+0000) 2024-02-20T03:00:33.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:33.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:33.327488+0000) 2024-02-20T03:00:34.326 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:34.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:34.327624+0000) 2024-02-20T03:00:35.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:35.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:35.327852+0000) 2024-02-20T03:00:36.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:36.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:36.328027+0000) 2024-02-20T03:00:37.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:37.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:37.328241+0000) 2024-02-20T03:00:38.327 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:38.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:38.328370+0000) 2024-02-20T03:00:39.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:39.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:39.328655+0000) 2024-02-20T03:00:40.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:40.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:40.328811+0000) 2024-02-20T03:00:41.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:41.324+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:41.328976+0000) 2024-02-20T03:00:42.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:42.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:42.329260+0000) 2024-02-20T03:00:43.328 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:43.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:43.329399+0000) 2024-02-20T03:00:44.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:44.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:44.329637+0000) 2024-02-20T03:00:45.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:45.328+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:45.329836+0000) 2024-02-20T03:00:46.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:46.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:46.330108+0000) 2024-02-20T03:00:47.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:47.323+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:47.330290+0000) 2024-02-20T03:00:48.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:48.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:48.330461+0000) 2024-02-20T03:00:49.329 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:49.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:49.330647+0000) 2024-02-20T03:00:50.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:50.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:50.330896+0000) 2024-02-20T03:00:51.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:51.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:51.331082+0000) 2024-02-20T03:00:52.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:52.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:52.331306+0000) 2024-02-20T03:00:53.330 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:53.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:53.331500+0000) 2024-02-20T03:00:54.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:54.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:54.331660+0000) 2024-02-20T03:00:55.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:55.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:55.331839+0000) 2024-02-20T03:00:56.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:56.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:56.332081+0000) 2024-02-20T03:00:57.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:57.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:57.332263+0000) 2024-02-20T03:00:58.331 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:58.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:58.332500+0000) 2024-02-20T03:00:59.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:00:59.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:00:59.332751+0000) 2024-02-20T03:01:00.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:00.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:00.333011+0000) 2024-02-20T03:01:01.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:01.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:01.333248+0000) 2024-02-20T03:01:02.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:02.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:02.333419+0000) 2024-02-20T03:01:03.332 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:03.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:03.333584+0000) 2024-02-20T03:01:04.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:04.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:04.333737+0000) 2024-02-20T03:01:05.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:05.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:05.333896+0000) 2024-02-20T03:01:06.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:06.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:06.334049+0000) 2024-02-20T03:01:07.333 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:07.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:07.334285+0000) 2024-02-20T03:01:08.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:08.327+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:08.334572+0000) 2024-02-20T03:01:09.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:09.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:09.334888+0000) 2024-02-20T03:01:10.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:10.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:10.335013+0000) 2024-02-20T03:01:11.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:11.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:11.335184+0000) 2024-02-20T03:01:12.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:12.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:12.335333+0000) 2024-02-20T03:01:13.334 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:13.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:13.335495+0000) 2024-02-20T03:01:14.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:14.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:14.335786+0000) 2024-02-20T03:01:15.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:15.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:15.335963+0000) 2024-02-20T03:01:16.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:16.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:16.336089+0000) 2024-02-20T03:01:17.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:17.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:17.336321+0000) 2024-02-20T03:01:18.335 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:18.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:18.336539+0000) 2024-02-20T03:01:19.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:19.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:19.336723+0000) 2024-02-20T03:01:20.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:20.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:20.336934+0000) 2024-02-20T03:01:21.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:21.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:21.337178+0000) 2024-02-20T03:01:22.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:22.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:22.337309+0000) 2024-02-20T03:01:23.336 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:23.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:23.337562+0000) 2024-02-20T03:01:24.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:24.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:24.337721+0000) 2024-02-20T03:01:25.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:25.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:25.337878+0000) 2024-02-20T03:01:26.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:26.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:26.338038+0000) 2024-02-20T03:01:27.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:27.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:27.338266+0000) 2024-02-20T03:01:28.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:28.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:28.338430+0000) 2024-02-20T03:01:29.337 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:29.331+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:29.338633+0000) 2024-02-20T03:01:30.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:30.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:30.338870+0000) 2024-02-20T03:01:31.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:31.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:31.339049+0000) 2024-02-20T03:01:32.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:32.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:32.339209+0000) 2024-02-20T03:01:33.338 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:33.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:33.339382+0000) 2024-02-20T03:01:34.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:34.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:34.339543+0000) 2024-02-20T03:01:35.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:35.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:35.339806+0000) 2024-02-20T03:01:36.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:36.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:36.339992+0000) 2024-02-20T03:01:37.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:37.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:37.340247+0000) 2024-02-20T03:01:38.339 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:38.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:38.340500+0000) 2024-02-20T03:01:39.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:39.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:39.340745+0000) 2024-02-20T03:01:40.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:40.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:40.340900+0000) 2024-02-20T03:01:41.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:41.335+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:41.341049+0000) 2024-02-20T03:01:42.340 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:42.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:42.341315+0000) 2024-02-20T03:01:43.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:43.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:43.341478+0000) 2024-02-20T03:01:44.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:44.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:44.341706+0000) 2024-02-20T03:01:45.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:45.339+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:45.341872+0000) 2024-02-20T03:01:46.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:46.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:46.342111+0000) 2024-02-20T03:01:47.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:47.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:47.342293+0000) 2024-02-20T03:01:48.341 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:48.334+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:48.342446+0000) 2024-02-20T03:01:49.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:49.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:49.342767+0000) 2024-02-20T03:01:50.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:50.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:50.342936+0000) 2024-02-20T03:01:51.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:51.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:51.343191+0000) 2024-02-20T03:01:52.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:52.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:52.343343+0000) 2024-02-20T03:01:53.342 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:53.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:53.343511+0000) 2024-02-20T03:01:54.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:54.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:54.343639+0000) 2024-02-20T03:01:55.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:55.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:55.343794+0000) 2024-02-20T03:01:56.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:56.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:56.344067+0000) 2024-02-20T03:01:57.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:57.338+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:57.344294+0000) 2024-02-20T03:01:58.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:58.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:58.344446+0000) 2024-02-20T03:01:59.343 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:01:59.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:01:59.344583+0000) 2024-02-20T03:02:00.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:00.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:00.344858+0000) 2024-02-20T03:02:01.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:01.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:01.345122+0000) 2024-02-20T03:02:02.344 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:02.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:02.345296+0000) 2024-02-20T03:02:03.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:03.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:03.345597+0000) 2024-02-20T03:02:04.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:04.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:04.345773+0000) 2024-02-20T03:02:05.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:05.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:05.345946+0000) 2024-02-20T03:02:06.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:06.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:06.346068+0000) 2024-02-20T03:02:07.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:07.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:07.346299+0000) 2024-02-20T03:02:08.345 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:08.342+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:08.346474+0000) 2024-02-20T03:02:09.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:09.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:09.346725+0000) 2024-02-20T03:02:10.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:10.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:10.346958+0000) 2024-02-20T03:02:11.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:11.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:11.347139+0000) 2024-02-20T03:02:12.346 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:12.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:12.347381+0000) 2024-02-20T03:02:13.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:13.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:13.347568+0000) 2024-02-20T03:02:14.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:14.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:14.347789+0000) 2024-02-20T03:02:15.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:15.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:15.347945+0000) 2024-02-20T03:02:16.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:16.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:16.348078+0000) 2024-02-20T03:02:17.347 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:17.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:17.348347+0000) 2024-02-20T03:02:18.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:18.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:18.348486+0000) 2024-02-20T03:02:19.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:19.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:19.348794+0000) 2024-02-20T03:02:20.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:20.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:20.349040+0000) 2024-02-20T03:02:21.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:21.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:21.349267+0000) 2024-02-20T03:02:22.348 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:22.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:22.349407+0000) 2024-02-20T03:02:23.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:23.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:23.349738+0000) 2024-02-20T03:02:24.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:24.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:24.349876+0000) 2024-02-20T03:02:25.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:25.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:25.350158+0000) 2024-02-20T03:02:26.349 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:26.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:26.350299+0000) 2024-02-20T03:02:27.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:27.346+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:27.350446+0000) 2024-02-20T03:02:28.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:28.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:28.350749+0000) 2024-02-20T03:02:29.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:29.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:29.350943+0000) 2024-02-20T03:02:30.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:30.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:30.351072+0000) 2024-02-20T03:02:31.350 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:31.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:31.351318+0000) 2024-02-20T03:02:32.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:32.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:32.351487+0000) 2024-02-20T03:02:33.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:33.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:33.351668+0000) 2024-02-20T03:02:34.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:34.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:34.351811+0000) 2024-02-20T03:02:35.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:35.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:35.351955+0000) 2024-02-20T03:02:36.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:36.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:36.352139+0000) 2024-02-20T03:02:37.351 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:37.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:37.352381+0000) 2024-02-20T03:02:38.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:38.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:38.352633+0000) 2024-02-20T03:02:39.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:39.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:39.352932+0000) 2024-02-20T03:02:40.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:40.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:40.353115+0000) 2024-02-20T03:02:41.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:41.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:41.353262+0000) 2024-02-20T03:02:42.352 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:42.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:42.353432+0000) 2024-02-20T03:02:43.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:43.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:43.353581+0000) 2024-02-20T03:02:44.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:44.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:44.353757+0000) 2024-02-20T03:02:45.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:45.350+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:45.353951+0000) 2024-02-20T03:02:46.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:46.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:46.354105+0000) 2024-02-20T03:02:47.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:47.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:47.354259+0000) 2024-02-20T03:02:48.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:48.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:48.354399+0000) 2024-02-20T03:02:49.353 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:49.349+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:49.354531+0000) 2024-02-20T03:02:50.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:50.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:50.354648+0000) 2024-02-20T03:02:51.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:51.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:51.354962+0000) 2024-02-20T03:02:52.354 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:52.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:52.355285+0000) 2024-02-20T03:02:53.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:53.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:53.355563+0000) 2024-02-20T03:02:54.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:54.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:54.355720+0000) 2024-02-20T03:02:55.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:55.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:55.355881+0000) 2024-02-20T03:02:56.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:56.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:56.356035+0000) 2024-02-20T03:02:57.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:57.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:57.356199+0000) 2024-02-20T03:02:58.355 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:58.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:58.356408+0000) 2024-02-20T03:02:59.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:02:59.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:02:59.356569+0000) 2024-02-20T03:03:00.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:00.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:00.356713+0000) 2024-02-20T03:03:01.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:01.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:01.356937+0000) 2024-02-20T03:03:02.356 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:02.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:02.357168+0000) 2024-02-20T03:03:03.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:03.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:03.357461+0000) 2024-02-20T03:03:04.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:04.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:04.357851+0000) 2024-02-20T03:03:05.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:05.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:05.358062+0000) 2024-02-20T03:03:06.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:06.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:06.358309+0000) 2024-02-20T03:03:07.357 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:07.353+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:07.358462+0000) 2024-02-20T03:03:08.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:08.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:08.358704+0000) 2024-02-20T03:03:09.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:09.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:09.358913+0000) 2024-02-20T03:03:10.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:10.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:10.359116+0000) 2024-02-20T03:03:11.358 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:11.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:11.359268+0000) 2024-02-20T03:03:12.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:12.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:12.359491+0000) 2024-02-20T03:03:13.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:13.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:13.359651+0000) 2024-02-20T03:03:14.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:14.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:14.359804+0000) 2024-02-20T03:03:15.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:15.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:15.360086+0000) 2024-02-20T03:03:16.359 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:16.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:16.360246+0000) 2024-02-20T03:03:17.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:17.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:17.360534+0000) 2024-02-20T03:03:18.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:18.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:18.360739+0000) 2024-02-20T03:03:19.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:19.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:19.361038+0000) 2024-02-20T03:03:20.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:20.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:20.361233+0000) 2024-02-20T03:03:21.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:21.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:21.361378+0000) 2024-02-20T03:03:22.360 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:22.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:22.361505+0000) 2024-02-20T03:03:23.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:23.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:23.361803+0000) 2024-02-20T03:03:24.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:24.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:24.362058+0000) 2024-02-20T03:03:25.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:25.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:25.362278+0000) 2024-02-20T03:03:26.361 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:26.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:26.362475+0000) 2024-02-20T03:03:27.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:27.357+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:27.362679+0000) 2024-02-20T03:03:28.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:28.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:28.362818+0000) 2024-02-20T03:03:29.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:29.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:29.363075+0000) 2024-02-20T03:03:30.362 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:30.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:30.363376+0000) 2024-02-20T03:03:31.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:31.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:31.363597+0000) 2024-02-20T03:03:32.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:32.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:32.363755+0000) 2024-02-20T03:03:33.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:33.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:33.363951+0000) 2024-02-20T03:03:34.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:34.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:34.364093+0000) 2024-02-20T03:03:35.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:35.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:35.364280+0000) 2024-02-20T03:03:36.363 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:36.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:36.364425+0000) 2024-02-20T03:03:37.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:37.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:37.364712+0000) 2024-02-20T03:03:38.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:38.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:38.364976+0000) 2024-02-20T03:03:39.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:39.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:39.365204+0000) 2024-02-20T03:03:40.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:40.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:40.365328+0000) 2024-02-20T03:03:41.364 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:41.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:41.365481+0000) 2024-02-20T03:03:42.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:42.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:42.365732+0000) 2024-02-20T03:03:43.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:43.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:43.365944+0000) 2024-02-20T03:03:44.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:44.365+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:44.366081+0000) 2024-02-20T03:03:45.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:45.361+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:45.366325+0000) 2024-02-20T03:03:46.365 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:46.360+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:46.366455+0000) 2024-02-20T03:03:47.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:47.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:47.366771+0000) 2024-02-20T03:03:48.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:48.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:48.366939+0000) 2024-02-20T03:03:49.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:49.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:49.367217+0000) 2024-02-20T03:03:50.366 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:50.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:50.367394+0000) 2024-02-20T03:03:51.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:51.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:51.367642+0000) 2024-02-20T03:03:52.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:52.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:52.367792+0000) 2024-02-20T03:03:53.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:53.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:53.367950+0000) 2024-02-20T03:03:54.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:54.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:54.368199+0000) 2024-02-20T03:03:55.367 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:55.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:55.368447+0000) 2024-02-20T03:03:56.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:56.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:56.368685+0000) 2024-02-20T03:03:57.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:57.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:57.368861+0000) 2024-02-20T03:03:58.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:58.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:58.369131+0000) 2024-02-20T03:03:59.368 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:03:59.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:03:59.369403+0000) 2024-02-20T03:04:00.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:00.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:00.369552+0000) 2024-02-20T03:04:01.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:01.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:01.369803+0000) 2024-02-20T03:04:02.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:02.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:02.369987+0000) 2024-02-20T03:04:03.369 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:03.364+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:03.370244+0000) 2024-02-20T03:04:04.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:04.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:04.370515+0000) 2024-02-20T03:04:05.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:05.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:05.370744+0000) 2024-02-20T03:04:06.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:06.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:06.370935+0000) 2024-02-20T03:04:07.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:07.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:07.371169+0000) 2024-02-20T03:04:08.370 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:08.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:08.371320+0000) 2024-02-20T03:04:09.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:09.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:09.371545+0000) 2024-02-20T03:04:10.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:10.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:10.371771+0000) 2024-02-20T03:04:11.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:11.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:11.372024+0000) 2024-02-20T03:04:12.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:12.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:12.372241+0000) 2024-02-20T03:04:13.371 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:13.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:13.372406+0000) 2024-02-20T03:04:14.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:14.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:14.372671+0000) 2024-02-20T03:04:15.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:15.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:15.372881+0000) 2024-02-20T03:04:16.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:16.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:16.373032+0000) 2024-02-20T03:04:17.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:17.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:17.373239+0000) 2024-02-20T03:04:18.372 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:18.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:18.373400+0000) 2024-02-20T03:04:19.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:19.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:19.373593+0000) 2024-02-20T03:04:20.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:20.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:20.373838+0000) 2024-02-20T03:04:21.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:21.368+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:21.374067+0000) 2024-02-20T03:04:22.373 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:22.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:22.374319+0000) 2024-02-20T03:04:23.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:23.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:23.374582+0000) 2024-02-20T03:04:24.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:24.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:24.374763+0000) 2024-02-20T03:04:25.374 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:25.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:25.374913+0000) 2024-02-20T03:04:26.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:26.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:26.375311+0000) 2024-02-20T03:04:27.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:27.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:27.375503+0000) 2024-02-20T03:04:28.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:28.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:28.375672+0000) 2024-02-20T03:04:29.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:29.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:29.375823+0000) 2024-02-20T03:04:30.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:30.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:30.375957+0000) 2024-02-20T03:04:31.375 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:31.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:31.376194+0000) 2024-02-20T03:04:32.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:32.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:32.376348+0000) 2024-02-20T03:04:33.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:33.372+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:33.376569+0000) 2024-02-20T03:04:34.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:34.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:34.376722+0000) 2024-02-20T03:04:35.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:35.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:35.376885+0000) 2024-02-20T03:04:36.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:36.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:36.377124+0000) 2024-02-20T03:04:37.376 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:37.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:37.377296+0000) 2024-02-20T03:04:38.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:38.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:38.377440+0000) 2024-02-20T03:04:39.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:39.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:39.377651+0000) 2024-02-20T03:04:40.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:40.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:40.377796+0000) 2024-02-20T03:04:41.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:41.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:41.377951+0000) 2024-02-20T03:04:42.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:42.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:42.378119+0000) 2024-02-20T03:04:43.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:43.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:43.378286+0000) 2024-02-20T03:04:44.377 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:44.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:44.378449+0000) 2024-02-20T03:04:45.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:45.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:45.378775+0000) 2024-02-20T03:04:46.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:46.376+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:46.379001+0000) 2024-02-20T03:04:47.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:47.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:47.379245+0000) 2024-02-20T03:04:48.378 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:48.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:48.379416+0000) 2024-02-20T03:04:49.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:49.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:49.379554+0000) 2024-02-20T03:04:50.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:50.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:50.379787+0000) 2024-02-20T03:04:51.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:51.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:51.379942+0000) 2024-02-20T03:04:52.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:52.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:52.380139+0000) 2024-02-20T03:04:53.379 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:53.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:53.380365+0000) 2024-02-20T03:04:54.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:54.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:54.380612+0000) 2024-02-20T03:04:55.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:55.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:55.380925+0000) 2024-02-20T03:04:56.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:56.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:56.381191+0000) 2024-02-20T03:04:57.380 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:57.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:57.381440+0000) 2024-02-20T03:04:58.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:58.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:58.381592+0000) 2024-02-20T03:04:59.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:04:59.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:04:59.381747+0000) 2024-02-20T03:05:00.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:05:00.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:05:00.381973+0000) 2024-02-20T03:05:01.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:05:01.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:05:01.382155+0000) 2024-02-20T03:05:02.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:05:02.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:05:02.382288+0000) 2024-02-20T03:05:03.381 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:05:03.375+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:05:03.382431+0000) 2024-02-20T03:05:04.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:05:04.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:05:04.382597+0000) 2024-02-20T03:05:05.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:05:05.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:05:05.382785+0000) 2024-02-20T03:05:06.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:05:06.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:05:06.382925+0000) 2024-02-20T03:05:07.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:05:07.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:05:07.383093+0000) 2024-02-20T03:05:08.382 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:05:08.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:05:08.383343+0000) 2024-02-20T03:05:09.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:05:09.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:05:09.383700+0000) 2024-02-20T03:05:10.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:05:10.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:05:10.383947+0000) 2024-02-20T03:05:11.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:05:11.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:05:11.384130+0000) 2024-02-20T03:05:12.383 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:05:12.383+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:05:12.384284+0000) 2024-02-20T03:05:13.384 INFO:tasks.ceph.mds.b.smithi116.stderr:2024-02-20T03:05:13.379+0000 7f70fc000640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T02:05:13.384528+0000) 2024-02-20T03:05:13.775 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-02-20T03:05:13.775 DEBUG:teuthology.task.console_log:Killing console logger for smithi047 2024-02-20T03:05:13.776 DEBUG:teuthology.task.console_log:Killing console logger for smithi116 2024-02-20T03:05:13.776 DEBUG:teuthology.exit:Finished running handlers